@@ -21,28 +21,33 @@ This repository contains two components: **Megatron-LM** and **Megatron Core**.
**[Megatron Bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge)** provides bidirectional Hugging Face ↔ Megatron checkpoint conversion with production-ready recipes.
+## Getting Started
-## Quick Start
+**Install from PyPI:**
-Install Megatron Core with pip:
+```bash
+uv pip install megatron-core
+```
-1. Install Megatron Core with required dependencies:
+**Or clone and install from source:**
- ```bash
- pip install --no-build-isolation megatron-core[mlm,dev]
- ```
+```bash
+git clone https://github.com/NVIDIA/Megatron-LM.git
+cd Megatron-LM
+uv pip install -e .
+```
-2. Clone repository for examples:
+> **Note:** Building from source can use a lot of memory. If the build runs out of memory, limit parallel compilation jobs by setting `MAX_JOBS` (e.g. `MAX_JOBS=4 uv pip install -e .`).
- ```bash
- git clone https://github.com/NVIDIA/Megatron-LM.git
- cd Megatron-LM
- pip install --no-build-isolation .[mlm,dev]
- ```
+For NGC container setup and all installation options, see the **[Installation Guide](https://docs.nvidia.com/megatron-core/developer-guide/latest/get-started/install.html)**.
+- **[Your First Training Run](https://docs.nvidia.com/megatron-core/developer-guide/latest/get-started/quickstart.html)** - End-to-end training examples with data preparation
+- **[Parallelism Strategies](https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/parallelism-guide.html)** - Scale training across GPUs with TP, PP, DP, EP, and CP
+- **[Contribution Guide](https://docs.nvidia.com/megatron-core/developer-guide/latest/developer/contribute.html)** - How to contribute to Megatron Core
# Latest News
+- **[2026/03]** **Deprecating Python 3.10 support:** We're officially dropping Python 3.10 support with the upcoming 0.17.0 release. Downstream applications must raise their lower boundary to 3.12 to stay compatible with MCore.
- **[2026/01]** **[Dynamic Context Parallelism](https://developer.nvidia.com/blog/speeding-up-variable-length-training-with-dynamic-context-parallelism-and-nvidia-megatron-core/)** - Up to 1.48x speedup for variable-length sequence training with adaptive CP sizing.
- **[2025/12]** **Megatron Core development has moved to GitHub!** All development and CI now happens in the open. We welcome community contributions.
- **[2025/10]** **[Megatron Dev Branch](https://github.com/NVIDIA/Megatron-LM/tree/dev)** - early access branch with experimental features.
@@ -61,9 +66,6 @@ Install Megatron Core with pip:
-
-
-
# Project Structure
```
@@ -77,20 +79,18 @@ Megatron-LM/
│ │ ├── distributed/ # Distributed training (FSDP, DDP)
│ │ ├── optimizer/ # Optimizers
│ │ ├── datasets/ # Dataset loaders
-│ │ ├── inference/ # Inference engines
+│ │ ├── inference/ # Inference engines and server
│ │ └── export/ # Model export (e.g. TensorRT-LLM)
│ ├── training/ # Training scripts
-│ ├── inference/ # Inference server
│ ├── legacy/ # Legacy components
-│ └── post_training/ # Post-training (RLHF, etc.)
+│ ├── post_training/ # Post-training (quantization, distillation, pruning, etc.)
+│ └── rl/ # Reinforcement learning (RLHF, etc.)
├── examples/ # Ready-to-use training examples
├── tools/ # Utility tools
├── tests/ # Comprehensive test suite
└── docs/ # Documentation
```
-
-
# Performance Benchmarking
For our latest performance benchmarking results, please refer to [NVIDIA Megatron Bridge Performance Summary](https://docs.nvidia.com/nemo/megatron-bridge/latest/performance-summary.html).
@@ -126,11 +126,6 @@ We also strong scaled the standard GPT-3 model (our version has slightly more th
-
-
-
-
-
# Roadmaps
- **[MoE Roadmap](https://github.com/NVIDIA/Megatron-LM/issues/1729)** - DeepSeek-V3, Qwen3, advanced parallelism, FP8 optimizations, and Blackwell enhancements
@@ -139,7 +134,7 @@ We also strong scaled the standard GPT-3 model (our version has slightly more th
## Getting Help
-- 📖 **[Documentation](https://docs.nvidia.com/Megatron-Core/)** - Official documentation
+- 📖 **[Documentation](https://docs.nvidia.com/megatron-core/developer-guide/latest/index.html)** - Official documentation
- 🐛 **[Issues](https://github.com/NVIDIA/Megatron-LM/issues)** - Bug reports and feature requests
## Contributing
@@ -151,7 +146,7 @@ We ❤️ contributions! Ways to contribute:
- 📝 **Improve docs** - Make Megatron Core more accessible
- 🔧 **Submit PRs** - Contribute code improvements
-**→ [Contributing Guide](./CONTRIBUTING.md)**
+**→ [Contributing Guide](https://docs.nvidia.com/megatron-core/developer-guide/latest/developer/contribute.html)**
## Citation
diff --git a/codecov.yml b/codecov.yml
new file mode 100644
index 00000000000..aa37017f082
--- /dev/null
+++ b/codecov.yml
@@ -0,0 +1,14 @@
+comment: false
+coverage:
+ status:
+ project: false
+ patch:
+ default:
+ target: 80%
+ threshold: 5%
+ base: auto
+ if_ci_failed: error
+ if_no_uploads: success
+ if_not_found: success
+fixes:
+ - "/opt/megatron-lm/::"
diff --git a/docker/.ngc_version.dev b/docker/.ngc_version.dev
index 8e8108b9a9a..2c33440d4e2 100644
--- a/docker/.ngc_version.dev
+++ b/docker/.ngc_version.dev
@@ -1 +1 @@
-nvcr.io/nvidia/pytorch:25.11-py3
\ No newline at end of file
+nvcr.io/nvidia/pytorch:26.02-py3
\ No newline at end of file
diff --git a/docker/Dockerfile.ci.dev b/docker/Dockerfile.ci.dev
index bb9ca5fbe9a..7a8b69c1297 100644
--- a/docker/Dockerfile.ci.dev
+++ b/docker/Dockerfile.ci.dev
@@ -35,6 +35,7 @@ COPY README.md pyproject.toml uv.lock /workspace/
COPY megatron/core/__init__.py /workspace/megatron/core/
COPY megatron/core/package_info.py /workspace/megatron/core/
ARG IMAGE_TYPE=dev
+ENV NVTE_BUILD_NUM_PHILOX_ROUNDS=3
RUN --mount=type=cache,target=/root/.cache/uv \
bash -ex <<"EOF"
export NVTE_CUDA_ARCHS="80;90;100"
@@ -88,7 +89,7 @@ RUN --mount=type=secret,id=JET_INDEX_URLS bash -ex <<"EOF"
JET_INDEX_URLS=$(cat /run/secrets/JET_INDEX_URLS)
python -m venv /opt/jet
/opt/jet/bin/pip install --no-cache-dir $JET_INDEX_URLS \
- jet-api==$JET_API_VERSION
+ "jet-api==$JET_API_VERSION" "setuptools<82.0.0"
EOF
RUN --mount=type=secret,id=JET_INDEX_URLS \
diff --git a/docker/common/install_source_wheels.sh b/docker/common/install_source_wheels.sh
index 2f144a6ff0a..eaf601c6045 100644
--- a/docker/common/install_source_wheels.sh
+++ b/docker/common/install_source_wheels.sh
@@ -40,18 +40,14 @@ MAMBA_WHEEL=$(ls $INPUT_WHEEL_DIR/mamba*.whl) || true
CAUSALCONV1D_WHEEL=$(ls $INPUT_WHEEL_DIR/causal_conv1d*.whl) || true
[ -z "$CAUSALCONV1D_WHEEL" ] && CAUSALCONV1D_WHEEL=$(bash docker/common/build_causalconv1d.sh --output-wheel-dir $INPUT_WHEEL_DIR | tail -n 1)
-GROUPEDGEMM_WHEEL=$(ls $INPUT_WHEEL_DIR/grouped_gemm*.whl) || true
-[ -z "$GROUPEDGEMM_WHEEL" ] && GROUPEDGEMM_WHEEL=$(bash docker/common/build_groupedgemm.sh --output-wheel-dir $INPUT_WHEEL_DIR | tail -n 1)
-
# Override deps that are already present in the base image
# only for dev
if [ "$ENVIRONMENT" = "dev" ]; then
uv pip install --no-cache-dir --no-deps $TE_WHEEL
fi
-# Install heavy optional deps like mamba, causalconv1d, groupedgemm
+# Install heavy optional deps like mamba, causalconv1d
uv pip install --no-cache-dir \
$MAMBA_WHEEL \
$CAUSALCONV1D_WHEEL \
- $GROUPEDGEMM_WHEEL \
"setuptools<80.0.0,>=77.0.0"
diff --git a/docs/add_copyright_header.py b/docs/add_copyright_header.py
new file mode 100644
index 00000000000..9694ef84819
--- /dev/null
+++ b/docs/add_copyright_header.py
@@ -0,0 +1,30 @@
+#!/usr/bin/env python3
+"""One-off script to add NVIDIA copyright header to all .md files under docs/."""
+
+from pathlib import Path
+
+HEADER = """ Copyright (c) 2022-2026, NVIDIA CORPORATION. All rights reserved.
+ NVIDIA CORPORATION and its licensors retain all intellectual property
+ and proprietary rights in and to this software, related documentation
+ and any modifications thereto. Any use, reproduction, disclosure or
+ distribution of this software and related documentation without an express
+ license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+"""
+
+def main():
+ docs_dir = Path(__file__).resolve().parent
+ already_has = "Copyright (c) 2022-2026, NVIDIA CORPORATION"
+ count = 0
+ for path in sorted(docs_dir.rglob("*.md")):
+ content = path.read_text(encoding="utf-8")
+ if content.strip().startswith(already_has):
+ continue
+ new_content = HEADER + content
+ path.write_text(new_content, encoding="utf-8")
+ count += 1
+ print(path.relative_to(docs_dir))
+ print(f"\nUpdated {count} files.")
+
+if __name__ == "__main__":
+ main()
diff --git a/docs/advanced/index.md b/docs/advanced/index.md
index 573cb0ee81a..98ff0806cff 100644
--- a/docs/advanced/index.md
+++ b/docs/advanced/index.md
@@ -1,3 +1,12 @@
+
+
# Discussions
In-depth technical discussions and optimization guides:
diff --git a/docs/api-backwards-compatibility-check.md b/docs/api-backwards-compatibility-check.md
index a417abfc2df..40f56ec0c00 100644
--- a/docs/api-backwards-compatibility-check.md
+++ b/docs/api-backwards-compatibility-check.md
@@ -1,3 +1,16 @@
+---
+orphan: true
+---
+
+
+
# API Backward Compatibility Checking
## Overview
diff --git a/docs/api-guide/core/datasets.md b/docs/api-guide/core/datasets.md
index e97e99ae1db..d80c0183375 100644
--- a/docs/api-guide/core/datasets.md
+++ b/docs/api-guide/core/datasets.md
@@ -1,3 +1,12 @@
+
+
# datasets package
```{include} ../../../megatron/core/datasets/readme.md
diff --git a/docs/api-guide/core/dist_checkpointing.md b/docs/api-guide/core/dist_checkpointing.md
index 959aa4b07e0..ee0e5562ef3 100644
--- a/docs/api-guide/core/dist_checkpointing.md
+++ b/docs/api-guide/core/dist_checkpointing.md
@@ -1,3 +1,12 @@
+
+
# dist_checkpointing package
A library for saving and loading the distributed checkpoints.
@@ -32,19 +41,62 @@ import torch, argparse
torch.serialization.add_safe_globals([argparse.Namespace])
```
-Checkpointing Distributed Optimizer
------------------------------------
+## Checkpointing Distributed Optimizer
-Checkpoint Compatibility and Optimizer State Formats
-####################################################
+### Checkpoint Compatibility and Optimizer State Formats
Beginning with **mcore v0.14**, the ``flattened_range`` attribute was removed from ``dist_checkpointing``. As a result:
-- Optimizer states saved with mcore versions < 0.14 are no longer loadable. Loading these legacy optimizer states is not supported because the required sharded metadata is no longer available.
-- Model weights from older checkpoints remain fully compatible. No additional work is required—model weights from checkpoints produced by earlier versions are loaded automatically.
+- Optimizer states saved with mcore versions <= 0.14 can no longer be loaded directly. Loading these legacy optimizer states is not supported because the required sharded metadata is no longer available. If you need to continue training from older checkpoints, refer to the workaround described below.
+- Model weights from older checkpoints remain fully compatible. No extra steps are needed—model weights from checkpoints created by earlier versions load automatically; simply add the ``--no-load-optim`` flag.
+
+### Workaround: Loading legacy optimizer states with ToT MCore
+
+**Step 1: Convert the legacy checkpoint using mcore v0.15.0**
+
+Run a dummy training job with mcore v0.15.0 to re-save the checkpoint with new optimizer states format.
+
+```bash
+MODEL_TRAIN_PARAMS=(
+ # Define model architecture and training parameters here
+)
+OLD_CKPT=/workspace/mcore_ckpt_old
+CONVERTED_CKPT=/workspace/mcore_ckpt_0.15.0
+
+torchrun --nproc_per_node=8 /opt/megatron-lm/pretrain_gpt.py \
+ --save-interval 1 \
+ --eval-interval 1 \
+ --exit-interval 1 \
+ --eval-iters 1 \
+ --use-distributed-optimizer \
+ --save ${CONVERTED_CKPT} \
+ --load ${OLD_CKPT} \
+ --ckpt-format torch_dist \
+ "${MODEL_TRAIN_PARAMS[@]}"
+```
+
+**Step 2: Load the converted checkpoint with ToT MCore**
+
+Use the converted checkpoint as the input for continued training with ToT MCore.
+
+```bash
+MODEL_TRAIN_PARAMS=(
+ # Define model architecture and training parameters here
+)
+NEW_CKPT=/workspace/mcore_ckpt_new
+CONVERTED_CKPT=/workspace/mcore_ckpt_0.15.0
+
+torchrun --nproc_per_node=8 /opt/megatron-lm/pretrain_gpt.py \
+ --use-distributed-optimizer \
+ --save ${NEW_CKPT} \
+ --load ${CONVERTED_CKPT} \
+ --ckpt-format torch_dist \
+ "${MODEL_TRAIN_PARAMS[@]}"
+```
+
+After this step, training can proceed normally using ToT MCore with fully supported optimizer state loading.
-Distributed Optimizer Checkpoint Formats
-########################################
+## Distributed Optimizer Checkpoint Formats
The refactor of the Distributed Optimizer introduces **two checkpoint formats**:
@@ -57,8 +109,7 @@ The refactor of the Distributed Optimizer introduces **two checkpoint formats**:
- Slower than dp_reshardable.
- Enabled via the ``--dist-ckpt-optim-fully-reshardable`` flag.
-Workflow for Changing Model Parallelism
-#######################################
+### Workflow for Changing Model Parallelism
You can combine formats to optimize both flexibility and performance:
diff --git a/docs/api-guide/core/dist_checkpointing.strategies.md b/docs/api-guide/core/dist_checkpointing.strategies.md
index 7aab8609504..22fe3517a54 100644
--- a/docs/api-guide/core/dist_checkpointing.strategies.md
+++ b/docs/api-guide/core/dist_checkpointing.strategies.md
@@ -1,3 +1,12 @@
+
+
# dist_checkpointing.strategies package
Package defining different checkpoint formats (backends) and saving/loading algorithms (strategies).
diff --git a/docs/api-guide/core/distributed.md b/docs/api-guide/core/distributed.md
index 1921c0bdd57..13da4285ec5 100644
--- a/docs/api-guide/core/distributed.md
+++ b/docs/api-guide/core/distributed.md
@@ -1,3 +1,12 @@
+
+
# distributed package
This package contains various utilities to finalize model weight gradients
diff --git a/docs/api-guide/core/fusions.md b/docs/api-guide/core/fusions.md
index 396280ad7da..fdd358e813c 100644
--- a/docs/api-guide/core/fusions.md
+++ b/docs/api-guide/core/fusions.md
@@ -1,3 +1,12 @@
+
+
# fusions package
This package provides modules that provide commonly fused
diff --git a/docs/api-guide/core/index.md b/docs/api-guide/core/index.md
index 150fd72cb1e..0d39e46e744 100644
--- a/docs/api-guide/core/index.md
+++ b/docs/api-guide/core/index.md
@@ -1,3 +1,12 @@
+
+
# Core APIs
Low-level API reference for core Megatron components.
diff --git a/docs/api-guide/core/pipeline_parallel.md b/docs/api-guide/core/pipeline_parallel.md
index 42fac8cc449..35f3c5b5cc2 100644
--- a/docs/api-guide/core/pipeline_parallel.md
+++ b/docs/api-guide/core/pipeline_parallel.md
@@ -1,3 +1,12 @@
+
+
# pipeline_parallel package
This package contains implementations for two different pipeline parallelism
diff --git a/docs/api-guide/core/tensor_parallel.md b/docs/api-guide/core/tensor_parallel.md
index 33a9160c82b..2d41c5f4467 100644
--- a/docs/api-guide/core/tensor_parallel.md
+++ b/docs/api-guide/core/tensor_parallel.md
@@ -1,3 +1,12 @@
+
+
# tensor_parallel package
This package contains an implementation for tensor parallelism in transformer
diff --git a/docs/api-guide/core/transformer.md b/docs/api-guide/core/transformer.md
index d004381844b..d35144fda4f 100644
--- a/docs/api-guide/core/transformer.md
+++ b/docs/api-guide/core/transformer.md
@@ -1,11 +1,19 @@
+
+
# transformer package
The `transformer` package provides a customizable and configurable
implementation of the transformer model architecture. Each component
of a transformer stack, from entire layers down to individual linear
layers, can be customized by swapping in different PyTorch modules
-using the "spec" parameters (see [here](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/nlp/nemo_megatron/mcore_customization.html)). The
+using the "spec" parameters. The
configuration of the transformer (hidden size, number of layers,
number of attention heads, etc.) is provided via a `TransformerConfig`
object.
-
diff --git a/docs/api-guide/index.md b/docs/api-guide/index.md
index 851114d98e8..7afa2450dd0 100644
--- a/docs/api-guide/index.md
+++ b/docs/api-guide/index.md
@@ -1,3 +1,12 @@
+
+
# API Guide
API reference documentation for Megatron Core components.
diff --git a/docs/api-guide/internal/index.md b/docs/api-guide/internal/index.md
index c216a976c77..312081ce70b 100644
--- a/docs/api-guide/internal/index.md
+++ b/docs/api-guide/internal/index.md
@@ -1,3 +1,12 @@
+
+
# Internal Utilities
Internal utility APIs.
diff --git a/docs/api-guide/internal/num_microbatches_calculator.md b/docs/api-guide/internal/num_microbatches_calculator.md
index 470c9e49128..0c223588ce5 100644
--- a/docs/api-guide/internal/num_microbatches_calculator.md
+++ b/docs/api-guide/internal/num_microbatches_calculator.md
@@ -1,3 +1,12 @@
+
+
# Microbatches Calculator
This api is used to calculate the number of microbatches required to fit a given model on a given batch size.
diff --git a/docs/api-guide/internal/optimizer_param_scheduler.md b/docs/api-guide/internal/optimizer_param_scheduler.md
index 13e1f77ccc0..45e5e4b7da1 100644
--- a/docs/api-guide/internal/optimizer_param_scheduler.md
+++ b/docs/api-guide/internal/optimizer_param_scheduler.md
@@ -1,3 +1,12 @@
+
+
# Optimizer Parameters Scheduler
This api is used to calculate the learning rate and weight decay for the optimizer.
diff --git a/docs/api-guide/models/index.md b/docs/api-guide/models/index.md
index c6279d2409a..e5bb531454b 100644
--- a/docs/api-guide/models/index.md
+++ b/docs/api-guide/models/index.md
@@ -1,3 +1,12 @@
+
+
# Model APIs
API reference for Megatron Core model implementations.
diff --git a/docs/api-guide/models/models.bert.md b/docs/api-guide/models/models.bert.md
index 3c53027c7c9..1543f4df865 100644
--- a/docs/api-guide/models/models.bert.md
+++ b/docs/api-guide/models/models.bert.md
@@ -1,3 +1,12 @@
+
+
# models.bert package
Useful package for training bert and bert like encoder only models. It optionally comes with a binary head that can be used for classification tasks .
diff --git a/docs/api-guide/models/models.gpt.md b/docs/api-guide/models/models.gpt.md
index a7c254d348b..1c3cbb5484c 100644
--- a/docs/api-guide/models/models.gpt.md
+++ b/docs/api-guide/models/models.gpt.md
@@ -1,3 +1,12 @@
+
+
# models.gpt package
This is the implementation of the popular GPT model. It supports several features like model parallelization (Tensor Parallel, Pipeline Parallel, Data Parallel) , mixture of experts, FP8 , Distributed optimizer etc. We are constantly adding new features. So be on the lookout or raise an issue if you want to have something added.
diff --git a/docs/api-guide/models/models.md b/docs/api-guide/models/models.md
index 69dfc80211d..a633546f0c9 100644
--- a/docs/api-guide/models/models.md
+++ b/docs/api-guide/models/models.md
@@ -1,3 +1,12 @@
+
+
# models package
This package contains most of the popular LLMs . Currently we have support for GPT, Bert, and T5 . This is an ever growing list so keep an eye out.
diff --git a/docs/api-guide/models/models.t5.md b/docs/api-guide/models/models.t5.md
index 90952096b63..4694b80113a 100644
--- a/docs/api-guide/models/models.t5.md
+++ b/docs/api-guide/models/models.t5.md
@@ -1,2 +1,11 @@
+
+
# models.t5 package
diff --git a/docs/api-guide/router_replay.md b/docs/api-guide/router_replay.md
index 300a50db127..88476ea7537 100644
--- a/docs/api-guide/router_replay.md
+++ b/docs/api-guide/router_replay.md
@@ -1,18 +1,27 @@
+
+
# Design Document: MoE Router Replay Feature
-### 1. Overview
+## 1. Overview
This document provides a detailed description of the "Router Replay" feature implemented within the Megatron-LM Core for Mixture-of-Experts (MoE) models.
This feature is designed to enhance determinism and analyzability in MoE model training and inference. It enables the model to load routing decisions from a predefined file and enforce their use during the forward pass, thereby bypassing the real-time routing computation.
-### 2. Motivation
+## 2. Motivation
* **Determinism & Reproducibility**: In distributed training, MoE routing decisions can exhibit minor variations due to factors like floating-point precision. By replaying a fixed routing table, the MoE computation path is guaranteed to be identical across runs, which facilitates debugging and reproducing experimental results.
* **Performance Profiling**: The router's own computation (e.g., logits calculation, top-k selection) incurs overhead. In replay mode, this part of the computation can be completely skipped, allowing for more precise isolation and profiling of performance bottlenecks within the Expert Layers themselves.
* **Debugging Aid**: When issues arise in the model, fixing the routing decisions helps to isolate variables, making it easier to determine whether the problem lies with the routing mechanism or the expert computations.
-### 3. Design and Architecture
+## 3. Design and Architecture
The design follows the principles of being non-intrusive and on-demand, with the core idea of activating the replay logic only when explicitly requested by the user.
@@ -35,7 +44,7 @@ The design follows the principles of being non-intrusive and on-demand, with the
* For each micro-batch (processed in reverse order in pipeline parallelism), the `router_replay_action` is checked again.
* **In `backward_replay` mode**: The function retrieves the expert indices for the corresponding micro-batch by popping them from the `replay_backward_list`. This mode is intended for training recomputation (e.g., activation checkpointing and pipeline recompute) so the same routing decisions are used during recompute/backward as in forward, ensuring determinism and correctness.
-### 4. Implementation Details
+## 4. Implementation Details
The implementation cleanly separates the replay logic from the router's core computation.
@@ -51,11 +60,12 @@ The implementation cleanly separates the replay logic from the router's core com
* `record_indices()`: A method to save the computed indices.
* The `topk_routing_with_score_function` is modified to contain the core logic. It checks the `router_replay_action` on the `router_replay` instance and accordingly performs one of the following actions: computes and records indices, replays indices from `target_topk_idx` (for forward), replays indices from `replay_backward_list` (for backward), or falls through to the default dynamic routing.
-#### Training recompute usage
+### Training recompute usage
+
- During forward replay, `set_target_indices()` prepares `replay_backward_list` so each micro-batch’s indices are available for recomputation.
- During recompute/backward, set action to `REPLAY_BACKWARD` so indices are consumed in FIFO order to mirror the forward sequence.
-### 5. Usage Guide
+## 5. Usage Guide
1. **Enable & Instantiate**
- Create one `RouterReplay` instance per MoE router layer when building the model.
@@ -73,7 +83,7 @@ The implementation cleanly separates the replay logic from the router's core com
5. **Cleanup**
- Use `RouterReplay.clear_global_indices()`, `RouterReplay.clear_global_router_replay_action()`, and `RouterReplay.clear_global_router_replay_instances()` to restore default behavior and prevent memory leaks.
-#### Quick usage with `topk_routing_with_score_function`
+### Quick usage with `topk_routing_with_score_function`
```python
import torch
@@ -105,7 +115,7 @@ RouterReplay.clear_global_indices()
RouterReplay.clear_global_router_replay_instances()
```
-### 6. Minimal Demo
+## 6. Minimal Demo
Here is a minimal code example showing how to use RouterReplay for recording and replaying:
diff --git a/docs/broken_links_false_positives.json b/docs/broken_links_false_positives.json
new file mode 100644
index 00000000000..01377be5804
--- /dev/null
+++ b/docs/broken_links_false_positives.json
@@ -0,0 +1,3 @@
+{
+ "uri": "http://localhost:8080/"
+}
\ No newline at end of file
diff --git a/docs/conf.py b/docs/conf.py
index a64da441084..9bf0b99c706 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -1,4 +1,4 @@
-# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2025-2026, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -25,9 +25,9 @@
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
project = "Megatron Core"
-copyright = "2025, NVIDIA Corporation"
+copyright = "2026, NVIDIA Corporation"
author = "NVIDIA Corporation"
-release = "latest"
+release = "nightly"
# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
@@ -63,6 +63,12 @@
]
myst_heading_anchors = 5 # Generates anchor links for headings up to level 5
+# Suppress "more than one target found for cross-reference" warnings for Python symbols
+# that have the same name across multiple modules (e.g. DistributedDataParallelConfig,
+# ModelType). These are structural ambiguities in the codebase – the cross-reference
+# still resolves; Sphinx just cannot pick the unique target automatically.
+suppress_warnings = ["ref.python"]
+
# -- Options for Autodoc2 ---------------------------------------------------
sys.path.insert(0, os.path.abspath(".."))
@@ -81,6 +87,11 @@
autodoc2_docstring_parser_regexes = [
(r".*", "docs.autodoc2_docstrings_parser"),
]
+ # Regex patterns whose values contain raw regex syntax (e.g. \p{L}) that docutils
+ # mis-parses as footnote/reference markup. Exclude them from the generated docs.
+ autodoc2_hidden_regexes = [
+ r".*\._PATTERN_TIKTOKEN.*",
+ ]
# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
@@ -98,16 +109,7 @@
"icon": "fa-brands fa-github",
}
],
- "extra_head": {
- """
-
- """
- },
- "extra_footer": {
- """
-
- """
- },
+ "public_docs_features": True
}
html_extra_path = ["project.json", "versions1.json"]
diff --git a/docs/developer/contribute.md b/docs/developer/contribute.md
index 859b5562f4b..aeb785f915d 100644
--- a/docs/developer/contribute.md
+++ b/docs/developer/contribute.md
@@ -1,3 +1,12 @@
+
+
# Contributing to Megatron-LM
This document outlines the processes and policies for issues and pull requests by non-NVIDIA contributors to the Megatron-LM GitHub repository.
diff --git a/docs/developer/generate_docs.md b/docs/developer/generate_docs.md
index 52fa288122d..d985f542caa 100644
--- a/docs/developer/generate_docs.md
+++ b/docs/developer/generate_docs.md
@@ -1,3 +1,12 @@
+
+
# Generating Docs Locally
To generate docs locally, use the following commands:
diff --git a/docs/developer/oncall.md b/docs/developer/oncall.md
index b88da7bb6df..0e5b38e2708 100644
--- a/docs/developer/oncall.md
+++ b/docs/developer/oncall.md
@@ -1,3 +1,13 @@
+
+-->
+
# Oncall Overview
During your oncall week, you will be assigned to all PRs marked “Ready for
@@ -30,13 +40,13 @@ Below is the checklist that the oncall needs to go through for each PR.
- Do all tests pass?
- Oncall will need to kick off testing suite for external reviewers
- Comment “/ok to test commid_id” to kick off testing suite
-- Add the “Expert Review” label
- - Select an expert reviewer from each expert group as a reviewer. If you’re unsure who to select, pick a “maintainer” or manager.
+- Expert reviewers are notified after the PR is marked “Ready for Review”
- **Expert reviewers should review within 1 business day.** Message the assigned reviewer if it is taking longer. The reviewer either needs to review the PR or suggest an alternate reviewer.
- - If the reviewer is not responding after 2 business days, escalate to the reviewer's manager.
-- Add the “Final Review” label after experts approve
+ - If the reviewer is not responding after 2 business days, escalate to the reviewer’s manager.
+- For `megatron/core` PRs, the “Final Review” label is applied automatically once all expert reviewers approve
- Final reviewers should review within 1 business day. Message the assigned reviewer if it is taking longer.
- - If the reviewer is not responding after 2 business days, escalate to the reviewer's manager.
+ - If the reviewer is not responding after 2 business days, escalate to the reviewer’s manager.
+- The “Approved” label is applied automatically once all required reviewers have approved
## Issues and Discussion Questions
diff --git a/docs/developer/submit.md b/docs/developer/submit.md
index a096312d21e..205e18cc52f 100644
--- a/docs/developer/submit.md
+++ b/docs/developer/submit.md
@@ -1,16 +1,34 @@
+
+
# How to Submit a PR
-## Step 1: Add PR label `Expert Review`
+All PRs start as **draft**. If you open a non-draft PR, it will be automatically converted to draft.
-## Step 2: Collect the expert reviewers reviews
+## Step 1: Mark PR as "Ready for Review"
-1. Attach the `Expert Review` label when your PR is ready for review.
-2. GitHub auto-assigns expert reviewers based on your changes. They will get notified and pick up your PR soon.
+1. When your PR is ready, click **Ready for Review**.
+2. The oncall reviewer is auto-assigned and expert reviewers are notified based on your changes. They will get notified and pick up your PR soon.
-:warning: Only proceed to the next step once all reviewers have approved, merge-conflict are resolved and the CI is passing.
+:warning: Only mark as ready once all merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.
-## Step 3: Final Review
+## Step 2: Final Review (`megatron/core` only)
+
+For PRs that change `megatron/core`, once all expert reviewers have approved, the `Final Review` label is applied **automatically** and final reviewers are assigned.
+
+For PRs outside `megatron/core`, this step is skipped.
+
+## Step 3: Approved
+
+Once all required reviewers have approved, the `Approved` label is applied **automatically**. The PR is now ready to merge.
+
+## Step 4: Merge
-1. Add `Final Review` label
-2. GitHub auto-assigns final reviewers based on your changes. They will get notified and pick up your PR soon.
+Any member of [mcore-engineers](https://github.com/orgs/NVIDIA/teams/mcore-engineers) will be able to merge your PR.
diff --git a/docs/discussions/README.md b/docs/discussions/README.md
index 4ac3c4e3254..e791ed57cd8 100644
--- a/docs/discussions/README.md
+++ b/docs/discussions/README.md
@@ -1,3 +1,16 @@
+---
+orphan: true
+---
+
+
+
# Megatron Discussions
This directory contains in-depth guides, tutorials, and discussions about optimizing and using Megatron for various use cases.
diff --git a/docs/discussions/megatron-fsdp-user-guide/megatron-fsdp-user-guide.md b/docs/discussions/megatron-fsdp-user-guide/megatron-fsdp-user-guide.md
index c2354ad07f0..b5de090ab46 100644
--- a/docs/discussions/megatron-fsdp-user-guide/megatron-fsdp-user-guide.md
+++ b/docs/discussions/megatron-fsdp-user-guide/megatron-fsdp-user-guide.md
@@ -1,3 +1,16 @@
+---
+orphan: true
+---
+
+
+
# Megatron-FSDP User Guide
## Table of Contents
diff --git a/docs/documentation.md b/docs/documentation.md
index 16fbd7b9a7e..d2554157b45 100644
--- a/docs/documentation.md
+++ b/docs/documentation.md
@@ -2,6 +2,15 @@
orphan: true
---
+
+
# Documentation Development
- [Documentation Development](#documentation-development)
diff --git a/docs/get-started/install.md b/docs/get-started/install.md
index dd000500f58..5781d065fae 100644
--- a/docs/get-started/install.md
+++ b/docs/get-started/install.md
@@ -1,87 +1,123 @@
-# Megatron Core Installation
+
-Installation is supported using Docker and pip.
+# Installation
## System Requirements
-### Hardware Requirements
+### Hardware
-- **FP8 Support**: NVIDIA Hopper, Ada, Blackwell GPUs
- **Recommended**: NVIDIA Turing architecture or later
+- **FP8 Support**: Requires NVIDIA Hopper, Ada, or Blackwell GPUs
-### Software Requirements
+### Software
-- **CUDA/cuDNN/NCCL**: Latest stable versions
-- **PyTorch**: Latest stable version
-- **Transformer Engine**: Latest stable version
-- **Python**: 3.12 recommended
+- **Python**: >= 3.10 (3.12 recommended)
+- **PyTorch**: >= 2.6.0
+- **CUDA Toolkit**: Latest stable version
-## Docker Installation (Recommended)
+## Prerequisites
-We strongly recommend using the previous releases of [PyTorch NGC Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch) rather than the latest one for optimal compatibility with Megatron Core release and testing matrix. Our releases are always based on the previous month's NGC container, so this ensures compatibility and stability.
+Install [uv](https://docs.astral.sh/uv/), a fast Python package installer:
+
+```bash
+curl -LsSf https://astral.sh/uv/install.sh | sh
+```
-**Note:** The NGC PyTorch container constraints the python environment globally via `PIP_CONSTRAINT`. In the following examples we will unset the variable.
-This container comes with all dependencies pre-installed with compatible versions and optimized configurations for NVIDIA GPUs:
+## Option A: Pip Install (Recommended)
-- PyTorch (latest stable version)
-- CUDA, cuDNN, NCCL (latest stable versions)
-- Support for FP8 on NVIDIA Hopper, Ada, and Blackwell GPUs
-- For best performance, use NVIDIA Turing GPU architecture generations and later
+Install the latest stable release from PyPI:
```bash
-# Run container with mounted directories
-docker run --runtime --nvidia --gpus all -it --rm \
- -v /path/to/megatron:/workspace/megatron \
- -v /path/to/dataset:/workspace/dataset \
- -v /path/to/checkpoints:/workspace/checkpoints \
- -e PIP_CONSTRAINT= \
- nvcr.io/nvidia/pytorch:25.04-py3
+uv pip install megatron-core
+```
+
+To include optional training dependencies (Weights & Biases, SentencePiece, HF Transformers):
+
+```bash
+uv pip install "megatron-core[training]"
+```
+
+For all extras including [Transformer Engine](https://github.com/NVIDIA/TransformerEngine):
+
+```bash
+uv pip install --group build
+uv pip install --no-build-isolation "megatron-core[training,dev]"
+```
+
+```{note}
+`--no-build-isolation` requires build dependencies to be pre-installed in the environment. `torch` is needed because several `[dev]` packages (`mamba-ssm`, `nv-grouped-gemm`, `transformer-engine`) import it at build time to compile CUDA kernels. Expect this step to take **20+ minutes** depending on your hardware. If you prefer pre-built binaries, the [NGC Container](#option-c-ngc-container) ships with these pre-compiled.
+```
+
+```{warning}
+Building from source can consume a large amount of memory. By default the build runs one compiler job per CPU core, which may cause out-of-memory failures on machines with many cores. To limit parallel compilation jobs, set the `MAX_JOBS` environment variable before installing (e.g. `MAX_JOBS=4`).
+```
+
+```{tip}
+For a lighter set of development dependencies without Transformer Engine and ModelOpt, use `[lts]` instead of `[dev]`: `uv pip install --no-build-isolation "megatron-core[training,lts]"`. The `[lts]` and `[dev]` extras are mutually exclusive.
```
-## Pip Installation
+To clone the repository for examples:
-Megatron Core installation offers support for two NGC PyTorch containers:
+```bash
+git clone https://github.com/NVIDIA/Megatron-LM.git
+```
-- `dev`: Moving head that supports the most recent upstream dependencies
-- `lts`: Long-term support of NGC PyTorch 24.01
-Both containers can be combined with `mlm`, which adds package dependencies for Megatron-LM on top of Megatron Core.
+## Option B: Install from Source
+For development or to run the latest unreleased code:
-1. Install the latest release dependencies
+```bash
+git clone https://github.com/NVIDIA/Megatron-LM.git
+cd Megatron-LM
+uv pip install -e .
+```
- ```bash
- pip install "setuptools<80.0.0,>=77.0.0" "packaging>=24.2"
- pip install --no-build-isolation megatron-core[dev]
- ```
+To install with all development dependencies (includes Transformer Engine, requires pre-installed build deps):
-2. Next choose one of the following options:
+```bash
+uv pip install --group build
+uv pip install --no-build-isolation -e ".[training,dev]"
+```
-* For running an Megatron LM application
+```{tip}
+If the build runs out of memory, limit parallel compilation jobs with `MAX_JOBS=4 uv pip install --no-build-isolation -e ".[training,dev]"`.
+```
- ```bash
- pip install "setuptools<80.0.0,>=77.0.0" "packaging>=24.2"
- pip install --no-build-isolation megatron-core[mlm,dev]
- ```
-* Install packages for LTS support NGC PyTorch 24.01
- ```bash
- pip install "setuptools<80.0.0,>=77.0.0" "packaging>=24.2"
- pip install --no-build-isolation megatron-core[lts]
- ```
+## Option C: NGC Container
-* For running an Megatron LM application
+For a pre-configured environment with all dependencies pre-installed (PyTorch, CUDA, cuDNN, NCCL, Transformer Engine), use the [PyTorch NGC Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch).
- ```bash
- pip install "setuptools<80.0.0,>=77.0.0" "packaging>=24.2"
- pip install --no-build-isolation megatron-core[mlm,lts]
- ```
+We recommend using the **previous month's** NGC container rather than the latest one to ensure compatibility with the current Megatron Core release and testing matrix.
-* For a version of Megatron Core with only Torch, run
+```bash
+docker run --gpus all -it --rm \
+ -v /path/to/dataset:/workspace/dataset \
+ -v /path/to/checkpoints:/workspace/checkpoints \
+ -e PIP_CONSTRAINT= \
+ nvcr.io/nvidia/pytorch:26.01-py3
+```
+
+```{note}
+The NGC PyTorch container constrains the Python environment globally via `PIP_CONSTRAINT`. The `-e PIP_CONSTRAINT=` flag above unsets this so that Megatron Core and its dependencies install correctly.
+```
+
+Then install Megatron Core inside the container (torch is already available in the NGC image):
+
+```bash
+pip install uv
+uv pip install --no-build-isolation "megatron-core[training,dev]"
+```
- ```bash
- pip install megatron-core
- ```
+You are now ready to run training. See [Your First Training Run](quickstart.md) for next steps.
diff --git a/docs/get-started/overview.md b/docs/get-started/overview.md
index 883f40e0c61..b7f84ee22e5 100644
--- a/docs/get-started/overview.md
+++ b/docs/get-started/overview.md
@@ -1,3 +1,12 @@
+
+
# Overview
Megatron-Core and Megatron-LM are open-source tools that are typically used together to train LLMs at scale across GPUs. Megatron-Core expands the capability of Megatron-LM. Megatron Bridge connects Megatron-Core and Megatron-LM to other popular training models, such as Hugging Face.
@@ -78,7 +87,7 @@ After training or modifying a Megatron model, you can convert it again for deplo
- **[Megatron Bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge)** - Training library with bidirectional Hugging Face ↔ Megatron checkpoint conversion, flexible training loops, and production-ready recipes
- **[NeMo RL](https://github.com/NVIDIA-NeMo/RL)** - Scalable toolkit for efficient reinforcement learning with RLHF, DPO, and other post-training methods
- **[NeMo Framework](https://docs.nvidia.com/nemo-framework/user-guide/latest/overview.html)** - Enterprise framework with cloud-native support and end-to-end examples
-- **[Model Optimizer (ModelOpt)](https://github.com/NVIDIA/Model-Optimizer)** - Model optimization toolkit for quantization, pruning, distillation, speculative decoding, and more. Checkout end-to-end examples in [examples/post_training/modelopt](./examples/post_training/modelopt/).
+- **[Model Optimizer (ModelOpt)](https://github.com/NVIDIA/Model-Optimizer)** - Model optimization toolkit for quantization, pruning, distillation, speculative decoding, and more. Checkout end-to-end examples in [examples/post_training/modelopt](https://github.com/NVIDIA/Megatron-LM/tree/main/examples/post_training/modelopt).
**Compatible with:** [Hugging Face Accelerate](https://github.com/huggingface/accelerate), [Colossal-AI](https://github.com/hpcaitech/ColossalAI), [DeepSpeed](https://github.com/microsoft/DeepSpeed)
diff --git a/docs/get-started/quickstart.md b/docs/get-started/quickstart.md
index 61868e7877c..c8797aeedd4 100644
--- a/docs/get-started/quickstart.md
+++ b/docs/get-started/quickstart.md
@@ -1,51 +1,46 @@
-# Quick Start
+
-## Quick Installation
+# Your First Training Run
-Install Megatron Core with pip:
+This guide walks you through running your first training jobs with Megatron Core. Make sure you have completed [installation](install.md) before proceeding.
-1. Install Megatron Core with required dependencies:
+## Simple Training Example
- ```bash
- pip install --no-build-isolation megatron-core[mlm,dev]
- ```
-
-2. Clone repository for examples:
-
- ```bash
- git clone https://github.com/NVIDIA/Megatron-LM.git
- cd Megatron-LM
- pip install --no-build-isolation .[mlm,dev]
- ```
-
-That's it! You're ready to start training.
-
-## Your First Training Run
-
-### Simple Training Example
+Run a minimal distributed training loop with mock data on 2 GPUs:
```bash
-# Distributed training example (2 GPUs, mock data)
torchrun --nproc_per_node=2 examples/run_simple_mcore_train_loop.py
```
-### LLaMA-3 Training Example
+## LLaMA-3 Training Example
+
+Train a LLaMA-3 8B model with FP8 precision on 8 GPUs using mock data:
```bash
-# 8 GPUs, FP8 precision, mock data
-./examples/llama/train_llama3_8b_fp8.sh
+./examples/llama/train_llama3_8b_h100_fp8.sh
```
## Data Preparation
-### JSONL Data Format
+To train on your own data, Megatron expects preprocessed binary files (`.bin` and `.idx`).
+
+### 1. Prepare a JSONL File
+
+Each line should contain a `text` field:
```json
{"text": "Your training text here..."}
{"text": "Another training sample..."}
```
-### Basic Preprocessing
+### 2. Preprocess the Data
```bash
python tools/preprocess_data.py \
diff --git a/docs/get-started/releasenotes.md b/docs/get-started/releasenotes.md
index e2d77cf0070..e624de19f15 100644
--- a/docs/get-started/releasenotes.md
+++ b/docs/get-started/releasenotes.md
@@ -1,3 +1,12 @@
+
+
# Release Notes
diff --git a/docs/index.md b/docs/index.md
index 448a75e4c93..4b75ed2c0c8 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,3 +1,12 @@
+
+
# Megatron Core User Guide
**Megatron Core** is a GPU-optimized library for training large language models at scale. It provides modular, composable building blocks for creating custom training frameworks with state-of-the-art parallelism strategies and performance optimizations.
@@ -6,13 +15,13 @@ Megatron Core offers a flexible, reusable foundation for building large-scale tr
## Key Features
-* Composable transformer building blocks (attention, MLP, etc.)
+* Composable transformer building blocks (attention, MLP)
* Advanced parallelism strategies (TP, PP, DP, EP, CP)
* Pipeline schedules and distributed optimizers
* Mixed precision support (FP16, BF16, FP8)
* GPU-optimized kernels and memory management
* High-performance dataloaders and dataset utilities
-* Model architectures (LLaMA, Qwen, DeepSeek, GPT, Mamba, etc.)
+* Model architectures (LLaMA, Qwen, DeepSeek, GPT, Mamba)
```{toctree}
@@ -29,8 +38,8 @@ get-started/releasenotes
:hidden:
:caption: Get Started
-get-started/quickstart
get-started/install
+get-started/quickstart
```
```{toctree}
@@ -62,6 +71,7 @@ user-guide/features/custom_fsdp
user-guide/features/dist_optimizer
user-guide/features/optimizer_cpu_offload
user-guide/features/pipeline_parallel_layout
+user-guide/features/fine_grained_activation_offloading
user-guide/features/megatron_energon
user-guide/features/megatron_rl
user-guide/features/tokenizers
@@ -81,16 +91,16 @@ developer/generate_docs
```{toctree}
:maxdepth: 2
:hidden:
-:caption: Discussions
+:caption: API Reference
-advanced/index
+api-guide/index
+apidocs/index.rst
```
```{toctree}
:maxdepth: 2
:hidden:
-:caption: API Reference
+:caption: Resources
-api-guide/index
-apidocs/index.rst
+advanced/index
```
\ No newline at end of file
diff --git a/docs/llama_mistral.md b/docs/llama_mistral.md
index a79bb2c4bf9..95568adce78 100644
--- a/docs/llama_mistral.md
+++ b/docs/llama_mistral.md
@@ -1,8 +1,17 @@
+
+
# Llama, Mistral and other Llama-like model support in Megatron-LM
NOTE: In order to simplify code we now only support converting llama-3.x and mistral checkpoints downloaded from Hugging Face. For converting other models, see [Megatron Bridge](models/index.md).
-The [Llama-2](https://ai.meta.com/llama/) and [Llama-3.x](https://llama.meta.com/) family of models are an open-source set of pretrained & finetuned (for chat) models that have achieved strong results across a wide set of benchmarks. At their times of release, both Llama-2 and Llama-3 models achieved among the best results for open-source models, and were competitive with leading closed-source models (see https://arxiv.org/pdf/2307.09288.pdf and https://ai.meta.com/blog/meta-llama-3/).
+The Llama-2 and Llama-3.x family of models are an open-source set of pretrained & finetuned (for chat) models that have achieved strong results across a wide set of benchmarks. At their times of release, both Llama-2 and Llama-3 models achieved among the best results for open-source models, and were competitive with leading closed-source models (see
).
Similarly, [Mistral-7b](https://mistral.ai/news/announcing-mistral-7b/) is an open-source model with pretrained and finetuned (for chat) variants that achieve strong benchmark results.
@@ -41,7 +50,6 @@ Architecturally Llama-2, Llama-3 and Mistral-7b are very similar. As such Megatr
- [Known numerical differences](#known-numerical-differences)
- [Using legacy model format](#using-legacy-model-format)
-
# Llama-2
Llama-2 checkpoints can be loaded into Megatron for inference and for finetuning. Loading these checkpoints consists of three steps:
@@ -54,7 +62,7 @@ The following sections detail these steps. The final section lists benchmark res
## Download Meta or Huggingface checkpoints
-Users must first apply for access to download the Llama-2 checkpoints either directly from [Meta](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) or through [Huggingface](https://huggingface.co/docs/transformers/main/model_doc/llama2) (HF). The checkpoints are available in two formats, Meta's native format (available from both the Meta and HF links), and HF's format (available only from HF). Either format can be converted to Megatron, as detailed next.
+Users must first apply for access to download the Llama-2 checkpoints either directly [Huggingface](https://huggingface.co/docs/transformers/main/model_doc/llama2) (HF). The checkpoints are available in two formats, Meta's native format (available from both the Meta and HF links), and HF's format (available only from HF). Either format can be converted to Megatron, as detailed next.
## Convert checkpoint format
@@ -140,11 +148,11 @@ If loading for either inference or finetuning, use the following arguments:
### Launch Meta
-Meta checkpoints can be launched with: https://github.com/facebookresearch/llama
+Meta checkpoints can be launched with:
### Launch Huggingface
-Huggingface checkpoints can be launched with: https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py
+Huggingface checkpoints can be launched with:
## Benchmark results
@@ -352,7 +360,7 @@ The following sections detail these steps.
## Download Huggingface checkpoints
-Users must first apply for access to download the Mistral-7b checkpoints through [Huggingface](https://huggingface.co/mistralai/Mistral-7B-v0.3) (HF).
+Users must first apply for access to download the Mistral-7b checkpoints through Huggingface. Two variants are available: the base model ([Mistral-7B-v0.3](https://huggingface.co/mistralai/Mistral-7B-v0.3)) and the instruct model ([Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)).
## Convert checkpoint format
@@ -428,7 +436,7 @@ Many models such as Yi-34B and Qwen2.x use the Llama architecture and may be con
It is not expected that the megatron and Huggingface implementations of llama3.x and mistral models will produce numerically identical results. There are multiple points where small numerical differences are expected. This is a non-exhaustive list:
-1. TransformerEngine (TE) uses the model params_dtype inside RMSNorm whereas the Huggingface implementation uses fp32. See for details: https://github.com/NVIDIA/TransformerEngine/issues/1132
+1. TransformerEngine (TE) uses the model params_dtype inside RMSNorm whereas the Huggingface implementation uses fp32. See for details:
2. Huggingface `transformers` implements the q, k and v projections in self-attention as separate GEMMs whereas Megatron core combines them into a single GEMM for efficiency. This leads to small numerical differences.
# Using legacy model format
diff --git a/docs/models/index.md b/docs/models/index.md
index 6fabd1f582c..0ee379b01bd 100644
--- a/docs/models/index.md
+++ b/docs/models/index.md
@@ -1,3 +1,12 @@
+
+
# Supported Models
Megatron Core supports a wide range of language and multimodal models with optimized implementations for large-scale training.
diff --git a/docs/models/llms.md b/docs/models/llms.md
index 6789a4c551c..f649673a2cc 100644
--- a/docs/models/llms.md
+++ b/docs/models/llms.md
@@ -1,3 +1,12 @@
+
+
# Language Models
Megatron Core supports the following language model architectures for large-scale training.
diff --git a/docs/models/multimodal.md b/docs/models/multimodal.md
index 66ed8ccd9cb..dce977e261d 100644
--- a/docs/models/multimodal.md
+++ b/docs/models/multimodal.md
@@ -1,3 +1,12 @@
+
+
# Multimodal Models
Megatron Core supports multimodal models that combine language with vision, audio, and other modalities for comprehensive multimodal understanding.
diff --git a/docs/project.json b/docs/project.json
index aa547ddd298..d5b9535338b 100644
--- a/docs/project.json
+++ b/docs/project.json
@@ -1,2 +1,2 @@
-{"name": "megatron-lm", "version": "latest"}
+{"name": "megatron-lm", "version": "nightly"}
diff --git a/docs/user-guide/data-preparation.md b/docs/user-guide/data-preparation.md
index 3ff5eedba89..ea91bee4309 100644
--- a/docs/user-guide/data-preparation.md
+++ b/docs/user-guide/data-preparation.md
@@ -1,3 +1,12 @@
+
+
# Data Preparation
Preparing your data correctly is essential for successful training with Megatron Core.
@@ -37,6 +46,46 @@ python tools/preprocess_data.py \
| `--workers` | Number of parallel workers for processing |
| `--append-eod` | Add end-of-document token |
+## Finding Optimal Number of Workers
+
+Use the `--find-optimal-num-workers` flag to find number of workers which gives the best performance in terms of preprocessed documents per second.
+Script will lauch a few short data preprocessing runs with a different number of workers to define the fastest run in respect to collected performance data.
+
+```bash
+python tools/preprocess_data.py \
+ --input data.jsonl \
+ --output-prefix processed_data \
+ --tokenizer-type HuggingFaceTokenizer \
+ --tokenizer-model /path/to/tokenizer.model \
+ --workers 8 \
+ --find-optimal-num-workers \
+ --workers-to-check 4 8 16 32 \
+ --max-documents 50000
+```
+
+**Required arguments**
+
+| Argument | Description |
+|----------|-------------|
+| `--find-optimal-num-workers` | Activates search of optimal number of workers |
+| `--workers-to-check` | List of possible number of workers to run |
+| `--max-documents` | Number of documents to be preprocessed during each run |
+
+**Output example**
+
+```bash
+-----------------------------------
+Performance results (fastest → slowest):
+1. 16 workers → avg. docs/s: 9606.6476
+2. 32 workers → avg. docs/s: 9275.3284
+3. 8 workers → avg. docs/s: 9151.9280
+4. 4 workers → avg. docs/s: 6391.3819
+
+-----------------------------------
+The most optimal num of workers is 16 with avg. preprocessed docs/s: 9606.6476.
+-----------------------------------
+```
+
## Output Files
The preprocessing tool generates two files:
diff --git a/docs/user-guide/features/context_parallel.md b/docs/user-guide/features/context_parallel.md
index 841c16326b3..c44366187be 100644
--- a/docs/user-guide/features/context_parallel.md
+++ b/docs/user-guide/features/context_parallel.md
@@ -1,3 +1,12 @@
+
+
# context_parallel package
## Context parallelism overview
diff --git a/docs/user-guide/features/custom_fsdp.md b/docs/user-guide/features/custom_fsdp.md
index 2f81eb0c5ef..ab1a1efc402 100644
--- a/docs/user-guide/features/custom_fsdp.md
+++ b/docs/user-guide/features/custom_fsdp.md
@@ -1,3 +1,12 @@
+
+
# Megatron FSDP
**NOTE: In M-Core 0.14, the custom FSDP refactored its checkpoint implementation to use DTensor-based torch distributed checkpointing. The custom FSDP was also renamed Megatron FSDP. The relevant sections of this document are no longer applicable.**
diff --git a/docs/user-guide/features/dist_optimizer.md b/docs/user-guide/features/dist_optimizer.md
index ddb6079885c..4e47791c12f 100644
--- a/docs/user-guide/features/dist_optimizer.md
+++ b/docs/user-guide/features/dist_optimizer.md
@@ -1,3 +1,12 @@
+
+
# Distributed Optimizer
The motivation for the distributed optimizer is to save memory by distributing the optimizer state evenly across data parallel ranks (https://arxiv.org/abs/1910.02054), versus the naive method of replicating the optimizer state across data parallel ranks.
diff --git a/docs/user-guide/features/fine_grained_activation_offloading.md b/docs/user-guide/features/fine_grained_activation_offloading.md
index 53211d1d06c..494674bd4f0 100644
--- a/docs/user-guide/features/fine_grained_activation_offloading.md
+++ b/docs/user-guide/features/fine_grained_activation_offloading.md
@@ -1,3 +1,12 @@
+
+
# Fine-grained Activation Offloading (collaborated with rednote)
Memory capacity is more and more important with the rising of extreme sparse MoE models like DeepSeek-V3 and Qwen3-235B. Fine-grained recomputing reduces the memory footprint at the cost of extra recomputation, while offloading could utilize the host-device bandwidth to achieve nearly zero-overhead. Fine-grained Activation Offloading targets at offloading the activation at the granularity of specific modules, so that we can calibrate the amount of offloading activation to maximize the training throughput.
diff --git a/docs/user-guide/features/index.md b/docs/user-guide/features/index.md
index 7730443e91b..59cef95d574 100644
--- a/docs/user-guide/features/index.md
+++ b/docs/user-guide/features/index.md
@@ -1,3 +1,12 @@
+
+
# Advanced Features
Advanced feature guides for key Megatron Core capabilities.
@@ -5,6 +14,7 @@ Advanced feature guides for key Megatron Core capabilities.
```{toctree}
:maxdepth: 2
+fine_grained_activation_offloading
moe
context_parallel
custom_fsdp
diff --git a/docs/user-guide/features/megatron_energon.md b/docs/user-guide/features/megatron_energon.md
index d08bde21e38..9ebba72083a 100644
--- a/docs/user-guide/features/megatron_energon.md
+++ b/docs/user-guide/features/megatron_energon.md
@@ -1,3 +1,12 @@
+
+
# Megatron Energon
Advanced multimodal dataloader for efficient loading of text, images, video, and audio at scale.
diff --git a/docs/user-guide/features/megatron_rl.md b/docs/user-guide/features/megatron_rl.md
index 128b41bdaf5..653ecb92459 100644
--- a/docs/user-guide/features/megatron_rl.md
+++ b/docs/user-guide/features/megatron_rl.md
@@ -1,3 +1,12 @@
+
+
# Megatron RL
Reinforcement learning library for post-training large language models at scale.
diff --git a/docs/user-guide/features/moe.md b/docs/user-guide/features/moe.md
index 56aca8c6999..45efe291a21 100644
--- a/docs/user-guide/features/moe.md
+++ b/docs/user-guide/features/moe.md
@@ -1,3 +1,12 @@
+
+
# Mixture of Experts
```{toctree}
@@ -6,6 +15,7 @@
multi_token_prediction
multi_latent_attention
+../../api-guide/router_replay
```
```{include} ../../../megatron/core/transformer/moe/README.md
diff --git a/docs/user-guide/features/multi_latent_attention.md b/docs/user-guide/features/multi_latent_attention.md
index 5628b8cfee3..4310843557a 100644
--- a/docs/user-guide/features/multi_latent_attention.md
+++ b/docs/user-guide/features/multi_latent_attention.md
@@ -1,3 +1,12 @@
+
+
# Multi-Latent Attention
## Multi-Latent Attention overview
diff --git a/docs/user-guide/features/multi_token_prediction.md b/docs/user-guide/features/multi_token_prediction.md
index 891bf4c93c5..e16108bbcfa 100644
--- a/docs/user-guide/features/multi_token_prediction.md
+++ b/docs/user-guide/features/multi_token_prediction.md
@@ -1,3 +1,12 @@
+
+
# Multi-Token Prediction (MTP)
Multi-Token Prediction (MTP) extends the prediction scope to multiple future tokens at each position. On the one hand, an MTP objective densifies the training signals and may improve
@@ -7,7 +16,7 @@ data efficiency. On the other hand, MTP may enable the model to pre-plan its rep
The k-th MTP module consists of a shared embedding layer, a projection matrix, a Transformer block, and a shared output head. For the i-th input token at the (k - 1)-th prediction depth, we first combine the representation of the i-th token and the embedding of the (i + K)-th token with the linear projection. The combined serves as the input of the Transformer block at the k-th depth to produce the output representation.
-For more information, please refer to [DeepSeek-V3 Technical Report](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdf)
+For more information, refer to [DeepSeek-V3 Technical Report](https://arxiv.org/pdf/2412.19437.pdf)
## Related Arguments
@@ -45,4 +54,4 @@ Use `m` to represent MTP layers in the pipeline layout string. For example:
## Precautions
-Please do not use Context Parallel (CP), or arbitrary AttnMaskType, or learned absolute position embedding type with MTP. These use cases are not yet supported.
+Do not use Context Parallel (CP), or arbitrary AttnMaskType, or learned absolute position embedding type with MTP. These use cases are not yet supported.
diff --git a/docs/user-guide/features/optimizer_cpu_offload.md b/docs/user-guide/features/optimizer_cpu_offload.md
index 408d7f6a788..1496bd0a91e 100644
--- a/docs/user-guide/features/optimizer_cpu_offload.md
+++ b/docs/user-guide/features/optimizer_cpu_offload.md
@@ -1,3 +1,12 @@
+
+
# Optimizer CPU Offload
```{include} ../../../megatron/core/optimizer/cpu_offloading/README.md
diff --git a/docs/user-guide/features/pipeline_parallel_layout.md b/docs/user-guide/features/pipeline_parallel_layout.md
index 30c8ce1a500..96b00eca004 100644
--- a/docs/user-guide/features/pipeline_parallel_layout.md
+++ b/docs/user-guide/features/pipeline_parallel_layout.md
@@ -1,3 +1,12 @@
+
+
# Custom Pipeline Model Parallel Layout
*This is an experimental feature and may be changed.*
diff --git a/docs/user-guide/features/tokenizers.md b/docs/user-guide/features/tokenizers.md
index 0aecf8df8a7..bc1a47cec76 100644
--- a/docs/user-guide/features/tokenizers.md
+++ b/docs/user-guide/features/tokenizers.md
@@ -1,3 +1,12 @@
+
+
# Tokenizers
Megatron Core provides a unified tokenizer system with a HuggingFace-style API for easy tokenizer management and configuration.
@@ -141,7 +150,7 @@ Use a null tokenizer for testing or non-text models:
```python
tokenizer = MegatronTokenizer.from_pretrained(
- metadata_path={"library": "null"},
+ metadata_path={"library": "null-text"},
vocab_size=131072,
)
```
@@ -173,16 +182,6 @@ torchrun --nproc_per_node=8 pretrain_gpt.py \
If `--tokenizer-metadata` is not specified, a default metadata file is generated automatically based on the tokenizer type.
-### Legacy Tokenizer Support
-
-The old tokenizer system is still supported for backward compatibility:
-
-```bash
-torchrun --nproc_per_node=8 pretrain_gpt.py \
- --legacy-tokenizer \
- ...
-```
-
## Supported Tokenizer Libraries
| Library | Description | Use Case |
diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md
index bbe85451582..d12f3e35af2 100644
--- a/docs/user-guide/index.md
+++ b/docs/user-guide/index.md
@@ -1,3 +1,16 @@
+---
+orphan: true
+---
+
+
+
# User Guide
Comprehensive guides for using Megatron Core and Megatron-LM.
@@ -5,7 +18,6 @@ Comprehensive guides for using Megatron Core and Megatron-LM.
```{toctree}
:maxdepth: 2
-quickstart
msc_integration
data-preparation
training-examples
diff --git a/docs/user-guide/msc_integration.md b/docs/user-guide/msc_integration.md
index fd73ac7e8f4..a197f25afc1 100644
--- a/docs/user-guide/msc_integration.md
+++ b/docs/user-guide/msc_integration.md
@@ -1,3 +1,12 @@
+
+
```{include} ../../megatron/core/MSC_Integration.md
```
diff --git a/docs/user-guide/parallelism-guide.md b/docs/user-guide/parallelism-guide.md
index 2baf518ae85..8d5cb8ff7c3 100644
--- a/docs/user-guide/parallelism-guide.md
+++ b/docs/user-guide/parallelism-guide.md
@@ -1,3 +1,12 @@
+
+
# Parallelism Strategies Guide
Megatron Core supports multiple parallelism strategies that can be combined to efficiently train models from billions to trillions of parameters across thousands of GPUs.
diff --git a/docs/user-guide/quickstart.md b/docs/user-guide/quickstart.md
deleted file mode 100644
index 7baed06d6be..00000000000
--- a/docs/user-guide/quickstart.md
+++ /dev/null
@@ -1,3 +0,0 @@
-```{include} ../../megatron/core/QuickStart.md
-```
-
diff --git a/docs/user-guide/training-examples.md b/docs/user-guide/training-examples.md
index 2824c608c36..5e7c0440073 100644
--- a/docs/user-guide/training-examples.md
+++ b/docs/user-guide/training-examples.md
@@ -1,3 +1,12 @@
+
+
# Training Examples
Get started with Megatron Core training using these practical examples.
@@ -24,7 +33,7 @@ This example:
Train LLaMA-3 8B model with FP8 mixed precision on 8 GPUs:
```bash
-./examples/llama/train_llama3_8b_fp8.sh
+./examples/llama/train_llama3_8b_h100_fp8.sh
```
**Configuration:**
diff --git a/docs/versions1.json b/docs/versions1.json
index a524c5921a8..d55416e0a95 100644
--- a/docs/versions1.json
+++ b/docs/versions1.json
@@ -1,13 +1,17 @@
[
{
- "name": "latest",
- "version": "latest",
- "url": "https://docs.nvidia.com/megatron-core/latest/"
+ "name": "nightly",
+ "version": "nightly",
+ "url": "https://docs.nvidia.com/megatron-core/developer-guide/nightly/"
+ },
+ {
+ "name": "0.16.0 (latest)",
+ "version": "0.16.0",
+ "url": "https://docs.nvidia.com/megatron-core/developer-guide/latest/"
},
{
"name": "0.15.0",
"version": "0.15.0",
- "url": "https://docs.nvidia.com/megatron-core/0.15.0/"
+ "url": "https://docs.nvidia.com/megatron-core/developer-guide/0.15.0/"
}
]
-
diff --git a/examples/gpt3/gpt_config.yaml b/examples/gpt3/gpt_config.yaml
index 18d305d9cb1..600f50221ce 100644
--- a/examples/gpt3/gpt_config.yaml
+++ b/examples/gpt3/gpt_config.yaml
@@ -92,7 +92,6 @@ model_parallel:
# Optimizations
gradient_accumulation_fusion: True
- async_tensor_model_parallel_allreduce: True
tp_comm_overlap: False
# Debug Options
diff --git a/examples/gptoss/01_convert_from_hf.py b/examples/gptoss/01_convert_from_hf.py
new file mode 100644
index 00000000000..adee3358ec3
--- /dev/null
+++ b/examples/gptoss/01_convert_from_hf.py
@@ -0,0 +1,55 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""Convert HuggingFace checkpoints to Megatron format."""
+
+import os
+import argparse
+
+from megatron.bridge import AutoBridge
+
+def _parse_args():
+ parser = argparse.ArgumentParser(description="Convert HF LLMs to Megatron format")
+ parser.add_argument(
+ "--hf-model",
+ type=str,
+ required=True,
+ help="HuggingFace model identifier or path",
+ )
+ parser.add_argument(
+ "--save-path",
+ type=str,
+ default=None,
+ help="Path to save the converted Megatron checkpoint",
+ )
+ parser.add_argument('--local-rank', '--local_rank', type=int, default=0)
+ return parser.parse_args()
+
+if __name__ == "__main__":
+ args = _parse_args()
+ HF_MODEL = args.hf_model
+ SAVE_PATH = args.save_path
+ WORLD_SIZE = int(os.environ.get("WORLD_SIZE", 1))
+
+ if SAVE_PATH is None:
+ SAVE_PATH = f"./megatron_checkpoints/{HF_MODEL.replace('/', '_')}"
+
+ print(f"Converting {HF_MODEL} to Megatron format...")
+ print(f"Save path: {SAVE_PATH}")
+
+ bridge = AutoBridge.from_hf_pretrained(HF_MODEL, trust_remote_code=True)
+ provider = bridge.to_megatron_provider()
+ # Update these configs as needed
+ provider.expert_tensor_parallel_size = 1
+ provider.tensor_model_parallel_size = 1
+ provider.pipeline_model_parallel_size = WORLD_SIZE
+ provider.finalize()
+
+ model = provider.provide_distributed_model(wrap_with_ddp=False)
+
+ bridge.save_megatron_model(
+ model,
+ SAVE_PATH,
+ hf_tokenizer_path=HF_MODEL
+ )
+
+ print(f"Saved Megatron checkpoint to {SAVE_PATH}")
diff --git a/examples/gptoss/02_train.sh b/examples/gptoss/02_train.sh
new file mode 100755
index 00000000000..d129adc2b84
--- /dev/null
+++ b/examples/gptoss/02_train.sh
@@ -0,0 +1,259 @@
+#!/bin/bash
+
+export CUDA_DEVICE_MAX_CONNECTIONS=${CUDA_DEVICE_MAX_CONNECTIONS:-1}
+
+
+# Setup arguments with defaults
+CHECKPOINT_PATH="NO_VALUE_PROVIDED"
+TENSORBOARD_LOGS_PATH="./tensorboard_logs/"
+TOKENIZER_ARG="MOCK"
+DATA_ARG="MOCK"
+DISTRIBUTED_CONFIG_FILE=""
+
+# Parse command line arguments
+while [[ $# -gt 0 ]]; do
+ case $1 in
+ --checkpoint-path)
+ CHECKPOINT_PATH="$2"
+ shift 2
+ ;;
+ --tensorboard-logs-path)
+ TENSORBOARD_LOGS_PATH="$2"
+ shift 2
+ ;;
+ --tokenizer)
+ TOKENIZER_ARG="$2"
+ shift 2
+ ;;
+ --data)
+ DATA_ARG="$2"
+ shift 2
+ ;;
+ --distributed-config-file)
+ DISTRIBUTED_CONFIG_FILE="$2"
+ shift 2
+ ;;
+ -h|--help)
+ echo "Usage: $0 [OPTIONS]"
+ echo "Options:"
+ echo " --checkpoint-path PATH Path to Megatron checkpoint"
+ echo " --tensorboard-logs-path PATH Path to TensorBoard logs"
+ echo " --tokenizer PATH|MOCK Path to tokenizer model, or 'MOCK' (default: MOCK)"
+ echo " --data PATH|MOCK Data prefix, or 'MOCK' (default: MOCK)"
+ echo " --distributed-config-file FILE Path to distributed training config file"
+ echo " -h, --help Show this help message"
+ exit 0
+ ;;
+ *)
+ echo "Unknown option: $1"
+ echo "Use --help for usage information"
+ exit 1
+ ;;
+ esac
+done
+
+# Check if checkpoint path exists
+if [ ! -d "$CHECKPOINT_PATH" ]; then
+ echo "Error: Checkpoint path does not exist: $CHECKPOINT_PATH"
+ exit 1
+fi
+echo "Checkpoint path exists: $CHECKPOINT_PATH"
+
+# Check if tensorboard logs path exists
+if [ ! -d "$TENSORBOARD_LOGS_PATH" ]; then
+ echo "Warning: TensorBoard logs path does not exist. Creating: $TENSORBOARD_LOGS_PATH"
+ mkdir -p "$TENSORBOARD_LOGS_PATH"
+fi
+echo "TensorBoard logs path exists: $TENSORBOARD_LOGS_PATH"
+
+# NOTE: by default we use 8 GPUs
+# These values will be over-written below with environmental variables
+GPUS_PER_NODE=8
+NUM_NODES=1
+MASTER_ADDR="localhost"
+MASTER_PORT=6000
+NODE_RANK=0
+
+# Load distributed config from file if provided
+if [ -n "$DISTRIBUTED_CONFIG_FILE" ]; then
+ if [ ! -f "$DISTRIBUTED_CONFIG_FILE" ]; then
+ echo "Warning: Distributed config file does not exist: $DISTRIBUTED_CONFIG_FILE"
+ echo "Continuing with default distributed training settings."
+ else
+ echo "Loading distributed config from: $DISTRIBUTED_CONFIG_FILE"
+ source "$DISTRIBUTED_CONFIG_FILE"
+ fi
+fi
+
+# Override with environment variables if set
+GPUS_PER_NODE=${GPUS_PER_NODE:-8}
+NUM_NODES=${NUM_NODES:-1}
+MASTER_ADDR=${MASTER_ADDR:-localhost}
+MASTER_PORT=${MASTER_PORT:-6000}
+NODE_RANK=${NODE_RANK:-0}
+WORLD_SIZE=$(($GPUS_PER_NODE*$NUM_NODES))
+
+# Path to the pretrain_gpt.py script, assuming this script is run from the root of the Megatron-LM repository
+PRETRAIN_SCRIPT_PATH="pretrain_gpt.py"
+
+# Data cache path (useful for both mock and real data)
+DATA_CACHE_PATH="${PWD}/benchmark_cache_gpt_oss_20b"
+mkdir -p "$DATA_CACHE_PATH"
+
+DISTRIBUTED_ARGS=(
+ --nproc_per_node $GPUS_PER_NODE
+ --nnodes $NUM_NODES
+ --master_addr $MASTER_ADDR
+ --master_port $MASTER_PORT
+ --node_rank $NODE_RANK
+)
+
+# NOTE: we only set pipeline parallelism to be the number of GPUs
+# Adjust each value based on your setup.
+TP_SIZE=1
+EP_SIZE=1
+PP_SIZE=${WORLD_SIZE}
+MICRO_BATCH_SIZE=1
+GLOBAL_BATCH_SIZE=128
+NUM_LAYERS=12
+DTYPE="fp8"
+SEQ_LENGTH=8192
+MAX_POSITION_EMBEDDINGS=8192
+TRAIN_SAMPLES=1953125000
+LR_DECAY_SAMPLES=1949218748
+
+MODEL_ARGS=(
+ --no-masked-softmax-fusion
+ --transformer-impl transformer_engine
+ --disable-bias-linear
+ --untie-embeddings-and-output-weights
+ --no-rope-fusion
+ --normalization RMSNorm
+ --num-layers ${NUM_LAYERS}
+ --hidden-size 512
+ --ffn-hidden-size 2048
+ --num-attention-heads 64
+ --group-query-attention
+ --num-query-groups 8
+ --seq-length ${SEQ_LENGTH}
+ --max-position-embeddings ${MAX_POSITION_EMBEDDINGS}
+ --use-mcore-models
+ --rotary-percent 1.0
+ --rope-type rope
+ --position-embedding-type rope
+ --rotary-base 10000
+ --no-bias-gelu-fusion
+ --export-force-local-attention
+ --no-bias-dropout-fusion
+ --quick-geglu
+ --glu-linear-offset 1.0
+ --softmax-type learnable
+ --window-attn-skip-freq 2
+ --activation-func-clamp-value 7.0
+ --window-size 127,0
+ --enable-gpt-oss
+)
+
+MOE_ARGS=(
+ --num-experts 4
+ --moe-router-topk 2
+ --moe-router-load-balancing-type aux_loss
+ --moe-aux-loss-coeff 1e-3
+ --moe-grouped-gemm
+ --moe-token-dispatcher-type alltoall
+ --overlap-param-gather
+ --overlap-grad-reduce
+ --moe-ffn-hidden-size 2048
+ --moe-router-dtype fp32
+ --moe-z-loss-coeff 1e-3
+ --moe-permute-fusion
+)
+
+DATA_ARGS_LIST=()
+if [[ "$TOKENIZER_ARG" == "MOCK" ]] || [[ "$DATA_ARG" == "MOCK" ]] || [[ -z "$TOKENIZER_ARG" ]]; then
+ DATA_ARGS_LIST+=(
+ "--mock-data"
+ "--tokenizer-type NullTokenizer"
+ "--vocab-size 128256"
+ "--data-cache-path ${DATA_CACHE_PATH}"
+ "--tiktoken-pattern v2"
+ "--split '99,1,0'"
+ "--no-create-attention-mask-in-dataloader"
+ "--no-mmap-bin-files"
+ "--num-workers 1"
+ )
+else
+ # Settings for real data
+ DATA_ARGS_LIST+=(
+ "--data-path $DATA_ARG"
+ "--tokenizer-type HuggingFaceTokenizer"
+ "--tokenizer-model $TOKENIZER_ARG"
+ "--data-cache-path ${DATA_CACHE_PATH}"
+ "--split '99,1,0'"
+ "--no-create-attention-mask-in-dataloader"
+ "--no-mmap-bin-files"
+ "--num-workers 1"
+ # Note: --vocab-size might be inferred by HuggingFaceTokenizer or might need to be explicit.
+ "--vocab-size 128256"
+ )
+fi
+
+TRAINING_ARGS=(
+ --micro-batch-size ${MICRO_BATCH_SIZE}
+ --global-batch-size ${GLOBAL_BATCH_SIZE}
+ --lr 1.0e-5
+ --train-samples ${TRAIN_SAMPLES}
+ --lr-decay-samples ${LR_DECAY_SAMPLES}
+ --lr-decay-style cosine
+ --min-lr 1.0e-6
+ --weight-decay 0.1
+ --lr-warmup-fraction 0.05
+ --clip-grad 1.0
+ --bf16
+ --use-flash-attn
+ --attention-softmax-in-fp32
+ --accumulate-allreduce-grads-in-fp32
+ --disable-bf16-reduced-precision-matmul
+ --recompute-activations
+)
+
+MODEL_PARALLEL_ARGS=(
+ --tensor-model-parallel-size ${TP_SIZE}
+ --pipeline-model-parallel-size ${PP_SIZE}
+ --expert-model-parallel-size ${EP_SIZE}
+ --sequence-parallel
+ --context-parallel-size 1
+ --use-distributed-optimizer
+ --fp8-format hybrid
+ --fp8-param-gather
+ --fp8-amax-compute-algo max
+ --fp8-amax-history-len 1024
+)
+
+LOGGING_ARGS=(
+ --log-interval 1
+ --save-interval 10000
+ --eval-interval 50000000
+ --eval-iters 0
+ --save $CHECKPOINT_PATH
+ --tensorboard-dir "${CHECKPOINT_PATH}/tensorboard"
+ --moe-per-layer-logging
+ --no-load-optim
+ --no-load-rng
+ --log-throughput
+)
+
+# Ensure pretrain_gpt.py is found
+if [ ! -f "$PRETRAIN_SCRIPT_PATH" ]; then
+ echo "Error: pretrain_gpt.py not found at $PRETRAIN_SCRIPT_PATH"
+ echo "Please ensure you are running this script from the root of the Megatron-LM repository, and pretrain_gpt.py is present."
+ exit 1
+fi
+
+python -m torch.distributed.run ${DISTRIBUTED_ARGS[@]} ${PRETRAIN_SCRIPT_PATH} \
+ ${MODEL_ARGS[@]} \
+ ${MOE_ARGS[@]} \
+ ${DATA_ARGS_LIST[@]} \
+ ${TRAINING_ARGS[@]} \
+ ${MODEL_PARALLEL_ARGS[@]} \
+ ${LOGGING_ARGS[@]}
diff --git a/examples/gptoss/03_convert_to_hf.py b/examples/gptoss/03_convert_to_hf.py
new file mode 100644
index 00000000000..8089afec854
--- /dev/null
+++ b/examples/gptoss/03_convert_to_hf.py
@@ -0,0 +1,52 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""Convert HuggingFace checkpoints to Megatron format."""
+
+import os
+import argparse
+
+from megatron.bridge import AutoBridge
+
+def _parse_args():
+ parser = argparse.ArgumentParser(description="Convert Megatron LLMs to HuggingFace format")
+ parser.add_argument(
+ "--hf-model",
+ type=str,
+ required=True,
+ help="HuggingFace model identifier or path to load config from",
+ )
+ parser.add_argument(
+ "--megatron-model",
+ type=str,
+ required=True,
+ help="Megatron model identifier or path",
+ )
+ parser.add_argument(
+ "--save-path",
+ type=str,
+ default=None,
+ help="Path to save the converted HuggingFace checkpoint",
+ )
+ parser.add_argument('--local-rank', '--local_rank', type=int, default=0)
+ return parser.parse_args()
+
+if __name__ == "__main__":
+ args = _parse_args()
+ HF_MODEL = args.hf_model
+ MEGATRON_MODEL = args.megatron_model
+ SAVE_PATH = args.save_path
+ WORLD_SIZE = int(os.environ.get("WORLD_SIZE", 1))
+
+ if SAVE_PATH is None:
+ SAVE_PATH = f"./huggingface_checkpoints/{MEGATRON_MODEL.replace('/', '_')}"
+
+ print(f"Converting {MEGATRON_MODEL} to HuggingFace {HF_MODEL} format...")
+ print(f"Save path: {SAVE_PATH}")
+
+ bridge = AutoBridge.from_hf_pretrained(HF_MODEL, trust_remote_code=True)
+ bridge.export_ckpt(
+ MEGATRON_MODEL,
+ SAVE_PATH,
+ )
+
+ print(f"Saved HuggingFace checkpoint to {SAVE_PATH}")
diff --git a/examples/gptoss/README.md b/examples/gptoss/README.md
new file mode 100644
index 00000000000..eeb92ad9953
--- /dev/null
+++ b/examples/gptoss/README.md
@@ -0,0 +1,153 @@
+# GPT-OSS Training Tutorial
+
+## Step 0: Install Dependencies
+
+### Using Megatron Bridge
+
+[Megatron-Bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge)
+
+Megatron Bridge provides a quick and convenient way to convert HuggingFace checkpoints to the Megatron format used by Megatron-LM. Follow the instructions in the [Megatron-Bridge Installation](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/README.md#-installation) to run the nemo docker container and convert checkpoints (via mounted volumes - make sure that the huggingface cache location AND the megatron checkpoint locations are properly mounted, otherwise you may not be saving the converted model to disk correctly).
+
+Below is an example of how to use Megatron-Bridge inside the pytorch container to convert a HuggingFace model checkpoint to Megatron format.
+
+Reference: [Megatron-Bridge Dockerfile](https://github.com/NVIDIA-NeMo/Megatron-Bridge/blob/main/docker/Dockerfile.ci)
+
+Inside the [pytorch container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch) run the following commands to install Megatron-Bridge:
+```bash
+cd /opt
+git clone --recursive https://github.com/NVIDIA-NeMo/Megatron-Bridge.git
+cd Megatron-Bridge
+
+# Make sure submodules are initialized (for 3rdparty/Megatron-LM)
+git submodule update --init --recursive
+
+export PATH="/root/.local/bin:$PATH"
+export UV_PROJECT_ENVIRONMENT=/opt/venv
+export VIRTUAL_ENV=/opt/venv
+export PATH="$UV_PROJECT_ENVIRONMENT/bin:$PATH"
+export UV_LINK_MODE=copy
+export UV_VERSION="0.7.2"
+
+# Install UV
+curl -LsSf https://astral.sh/uv/${UV_VERSION}/install.sh | sh
+
+# Create virtual environment and build the package
+uv venv ${UV_PROJECT_ENVIRONMENT} --system-site-packages
+
+uv sync --locked --only-group build
+uv sync --locked --link-mode copy --all-extras --all-groups
+
+uv pip install --no-deps -e .
+
+source ${UV_PROJECT_ENVIRONMENT}/bin/activate
+```
+
+### Setup Environment
+
+```bash
+export HOST_MEGATRON_LM_DIR="/path/to/your/host/megatron-lm"
+git clone https://github.com/NVIDIA/Megatron-LM.git "$HOST_MEGATRON_LM_DIR"
+cd "$HOST_MEGATRON_LM_DIR"
+```
+
+```bash
+export HF_TOKEN={your_hf_token_here}
+```
+
+## Step 1: Convert HuggingFace to Megatron (Optional - skip if you already have a Megatron checkpoint)
+
+Set `--nproc-per-node` to be the number of GPUs per node. Set `hf_model_name` to be the Huggingface model e.g. `openai/gpt-oss-20b`
+
+```bash
+python3 -m torch.distributed.launch --nproc-per-node=8 examples/gptoss/01_convert_from_hf.py --hf-model openai/gpt-oss-20b
+```
+
+## Step 2: Train from Scratch
+
+To train from scratch first follow the steps below to setup the environment appropriately before running the training script in docker. Even though we are running the same container as before, it is better to restart the container to ensure a clean environment and that all environment and docker variables are set correctly. For the following example we used 8x GB300, but you should change the number of GPUs and nodes as needed.
+
+### Setup Environment
+
+```bash
+# Change these based on model and directory from previous conversion step
+export MODEL_DIR_NAME="openai_gpt-oss_20b"
+
+export HOST_CHECKPOINT_PATH="./megatron_checkpoints/${MODEL_DIR_NAME}"
+export HOST_TENSORBOARD_LOGS_PATH="./tensorboard_logs/${MODEL_DIR_NAME}"
+```
+
+By default we will use mock data to train the model in the example below. To use your own data, set the following environment variables:
+
+```bash
+# Optional: For real data
+export HOST_TOKENIZER_MODEL_PATH="/path/to/host/tokenizer.model"
+export HOST_DATA_PREFIX="/path/to/host/mydata_prefix"
+```
+
+### Setup Training Configurations
+
+Run the following to create a `distributed_config.env` file with the appropriate distributed training configurations. Change the values as needed for your setup. This file will override the default values in `02_train.sh`.
+
+```bash
+cat > ./distributed_config.env << 'EOF'
+GPUS_PER_NODE=8
+NUM_NODES=1
+MASTER_ADDR=localhost
+MASTER_PORT=6000
+NODE_RANK=0
+EOF
+```
+
+### Run Container with Mounted Volumes
+
+**NOTE:** This container runs the example training script `02_train.sh` located in the `examples/gptoss` directory. By default, we have only set pipeline parallelism to be the number of GPUs. Adjust TP_SIZE, EP_SIZE, PP_SIZE, etc. in `02_train.sh`. You can also adjust modify `--hidden-size`, `--ffn-hidden-size`, `--num-attention-heads`, `NUM_LAYERS`, etc.
+
+To train using mock data, run the following command:
+```bash
+PYTORCH_IMAGE="nvcr.io/nvidia/pytorch:25.12-py3"
+
+docker run --rm --gpus all --ipc=host --ulimit memlock=-1 \
+ -v "${HOST_MEGATRON_LM_DIR}:/workspace/megatron-lm" \
+ -v "${HOST_CHECKPOINT_PATH}:/workspace/checkpoints" \
+ -v "${HOST_TENSORBOARD_LOGS_PATH}:/workspace/tensorboard_logs" \
+ -v "./distributed_config.env:/workspace/megatron-lm/examples/gptoss/distributed_config.env" \
+ --workdir /workspace/megatron-lm \
+ $PYTORCH_IMAGE \
+ bash examples/gptoss/02_train.sh \
+ --checkpoint-path /workspace/checkpoints \
+ --tensorboard-logs-path /workspace/tensorboard_logs \
+ --distributed-config-file /workspace/megatron-lm/examples/gptoss/distributed_config.env \
+ 2>&1 | tee "${HOST_TENSORBOARD_LOGS_PATH}/training_mock_$(date +'%y-%m-%d_%H-%M-%S').log"
+```
+**Note:** If you run into issues generating mock data one solution might be to reduce the number of GPUs to 1 and try to generate the data again.
+
+If using real data with with the `HOST_TOKENIZER_MODEL_PATH` and `HOST_DATA_PREFIX` environment variables set, run the following command instead:
+
+```bash
+PYTORCH_IMAGE="nvcr.io/nvidia/pytorch:25.12-py3"
+
+docker run --rm --gpus all --ipc=host --ulimit memlock=-1 \
+ -v "${HOST_MEGATRON_LM_DIR}:/workspace/megatron-lm" \
+ -v "${HOST_CHECKPOINT_PATH}:/workspace/checkpoints" \
+ -v "${HOST_TENSORBOARD_LOGS_PATH}:/workspace/tensorboard_logs" \
+ -v "${HOST_TOKENIZER_MODEL_PATH}:/workspace/tokenizer_model" \
+ -v "$(dirname "${HOST_DATA_PREFIX}"):/workspace/data_dir" \
+ -v "./distributed_config.env:/workspace/megatron-lm/examples/gptoss/distributed_config.env" \
+ --workdir /workspace/megatron-lm \
+ $PYTORCH_IMAGE \
+ bash examples/gptoss/02_train.sh \
+ --checkpoint-path /workspace/checkpoints \
+ --tensorboard-logs-path /workspace/tensorboard_logs \
+ --tokenizer /workspace/tokenizer_model \
+ --data "/workspace/data_dir/$(basename "${HOST_DATA_PREFIX}")" \
+ --distributed-config-file /workspace/megatron-lm/examples/gptoss/distributed_config.env \
+ 2>&1 | tee "${HOST_TENSORBOARD_LOGS_PATH}/training_custom_$(date +'%y-%m-%d_%H-%M-%S').log"
+```
+
+## Step 3: Convert Megatron to HuggingFace
+
+Just run the following command to change from the megatron checkpoint from training to the huggingface format to share with others (make sure you have the same virtual environment setup as in Step 0):
+
+```bash
+python3 -m torch.distributed.launch --nproc-per-node=8 examples/gptoss/03_convert_to_hf.py --hf-model openai/gpt-oss-20b --megatron-model ./megatron_checkpoints/openai_gpt-oss_20b
+```
\ No newline at end of file
diff --git a/examples/inference/gpt/gpt_dynamic_inference.py b/examples/inference/gpt/gpt_dynamic_inference.py
index 88b744b3ac0..f02aae9c221 100644
--- a/examples/inference/gpt/gpt_dynamic_inference.py
+++ b/examples/inference/gpt/gpt_dynamic_inference.py
@@ -1,40 +1,31 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# pylint: disable=bad-builtin
+
import hashlib
import io
import json
-import math
import os
-import pickle
import sys
import warnings
-import torch
-from argparse import ArgumentParser
from collections import defaultdict
-from functools import partial
+from typing import Dict, List, Optional
+
+import torch
from tqdm import tqdm
-from typing import Dict, List, Tuple, Optional
sys.path.append(
os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir, os.path.pardir))
)
-import megatron
from examples.inference.gpt.utils import (
Request,
- add_common_inference_args,
build_dynamic_engine_setup_prefix,
build_requests,
get_curr_time,
get_global_peak_memory_stats_bytes,
)
-from megatron.core.inference.contexts.dynamic_context import (
- ContextOverflowError,
- DynamicInferenceContext,
-)
-from megatron.core.inference.contexts.attention_context.mamba_metadata import (
- MambaInferenceStateConfig,
-)
+from megatron.core.inference.contexts.dynamic_context import DynamicInferenceContext
from megatron.core.inference.engines import DynamicInferenceEngine, EngineSuspendedError
from megatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapper import (
GPTInferenceWrapper,
@@ -43,195 +34,27 @@
from megatron.core.inference.text_generation_controllers.text_generation_controller import (
TextGenerationController,
)
-from megatron.core.tokenizers.text.utils.build_tokenizer import build_tokenizer
-from megatron.core.transformer.module import MegatronModule
-from megatron.core.utils import get_mamba_inference_state_config_from_model
+from megatron.core.tokenizers.utils.build_tokenizer import build_tokenizer
+from megatron.inference.utils import (
+ add_inference_args,
+ get_inference_config_from_model_and_args,
+ get_model_for_inference,
+)
sys.path.append(
os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir, os.path.pardir))
)
-from megatron.training import get_args, get_model as _get_model, get_tokenizer, initialize_megatron
-from megatron.training.checkpointing import load_checkpoint
-from model_provider import model_provider
-from gpt_builders import gpt_builder
-from mamba_builders import mamba_builder
+import logging
+import megatron
from megatron.core.utils import configure_nvtx_profiling
-import logging
+from megatron.training import get_args, get_tokenizer, initialize_megatron
torch.serialization.add_safe_globals([io.BytesIO])
torch.serialization.add_safe_globals([megatron.core.rerun_state_machine.RerunState])
torch.serialization.add_safe_globals([megatron.core.rerun_state_machine.RerunDiagnostic])
-def add_dynamic_inference_args(parser: ArgumentParser) -> ArgumentParser:
- """Dynamic inference arguments."""
-
- add_common_inference_args(parser)
-
- group = parser.add_argument_group(title='Dynamic inference')
- group.add_argument(
- "--inference-ckpt-non-strict",
- action="store_true",
- help="Load checkpoint with `strict=False`.",
- )
- group.add_argument(
- "--termination-id", type=int, default=None,
- help="Termination ID that overrides `tokenizer.eod`.",
- )
- group.add_argument(
- "--suspend-resume-interval", type=int, default=None,
- help="Suspend and resume the dynamic engine every "
- "`suspend_resume_interval` steps. This is used to tet the suspend/resume "
- "system.",
- )
- group.add_argument(
- "--inference-repeat-n", type=int, default=1,
- help="Repeat inference iterations N times for benchmarking."
- )
- group.add_argument(
- "--throughput-check-only",
- action='store_true',
- default=False,
- help="If true, only run throughput check without verifying outputs."
- )
-
- return parser
-
-
-def get_model() -> MegatronModule:
- """Initialize model and load checkpoint."""
-
- args = get_args()
-
- if args.model_provider == "gpt":
- model_builder = gpt_builder
- elif args.model_provider == "mamba":
- model_builder = mamba_builder
- else:
- raise ValueError(f"Invalid model provider {args.model_provider}")
-
- # Build model.
- model = _get_model(
- partial(model_provider, model_builder),
- wrap_with_ddp=False
- )
-
- # Load checkpoint.
- assert args.load is not None
- args.exit_on_missing_checkpoint = True
- load_checkpoint(
- ddp_model=model,
- optimizer=None,
- opt_param_scheduler=None,
- strict=not args.inference_ckpt_non_strict,
- )
-
- # No virtual PP.
- assert len(model) == 1, "Above condition should have caught this"
- model = model[0]
-
- # Eval mode.
- model.eval()
-
- return model
-
-
-def get_inference_context(
- requests: List[Request],
- sampling_params: Optional[SamplingParams] = None,
- calculate_max_sequence_length_from_requests: bool = True,
- mamba_inference_state_config: Optional[MambaInferenceStateConfig] = None,
-):
- """The inference context manages the KV cache and other inference state."""
-
- args = get_args()
-
- # Max sequence length.
- if calculate_max_sequence_length_from_requests:
- max_gen_length = sampling_params.num_tokens_to_generate
- max_context_length = max(len(r.prompt_tokens) for r in requests)
- max_sequence_length = max_context_length + max_gen_length
- else:
- max_sequence_length = args.inference_max_seq_length
-
- metrics_writer = None
- if args.inference_logging_step_interval > 0 and args.inference_wandb_logging:
- metrics_writer = get_wandb_writer()
-
- # Inference context.
- context = DynamicInferenceContext(
- params_dtype=args.params_dtype,
- num_layers=args.num_layers // args.pipeline_model_parallel_size,
- kv_channels=args.kv_channels,
- num_attention_heads=(
- args.num_query_groups if args.group_query_attention else args.num_attention_heads
- ),
- max_sequence_length=max_sequence_length,
- num_cuda_graphs=(
- args.inference_dynamic_batching_num_cuda_graphs
- if args.cuda_graph_impl == "local"
- else None
- ),
- block_size_tokens=args.inference_dynamic_batching_block_size,
- buffer_size_gb=args.inference_dynamic_batching_buffer_size_gb,
- paused_buffer_size_gb=args.inference_dynamic_batching_paused_buffer_size_gb,
- max_requests=args.inference_dynamic_batching_max_requests,
- max_tokens=args.inference_dynamic_batching_max_tokens,
- tensor_model_parallel_size=args.tensor_model_parallel_size,
- pipeline_model_parallel_size=args.pipeline_model_parallel_size,
- materialize_only_last_token_logits=not args.return_log_probs,
- mamba_inference_state_config=mamba_inference_state_config,
- cache_mla_latent=args.multi_latent_attention and args.cache_mla_latents,
- kv_lora_rank=args.kv_lora_rank if args.multi_latent_attention else None,
- qk_pos_emb_head_dim=args.qk_pos_emb_head_dim,
- use_cuda_graphs_for_non_decode_steps=not args.decode_only_cuda_graphs,
- use_flashinfer_fused_rope=args.use_flashinfer_fused_rope,
- unified_memory_level=args.inference_dynamic_batching_unified_memory_level,
- cuda_graph_max_tokens=args.inference_dynamic_batching_cuda_graph_max_tokens,
- cuda_graph_mixed_prefill_count=args.inference_dynamic_batching_cuda_graph_mixed_prefill_count,
- metrics_writer=metrics_writer,
- offload_kv_cache=args.rl_offload_kv_cache_during_training
- )
-
- return context
-
-
-def get_inference_controller(
- model: MegatronModule, context: DynamicInferenceContext
-) -> TextGenerationController:
- """Buid text generation controller, which manages the model inference context.
-
- Args:
- model (MegatronModule): Megatron GPT model.
- context (DynamicInferenceContext): Context for managing KV cache blocks.
-
- Return:
- (TextGenerationController) Inference text generation controller.
- """
-
- args = get_args()
- if args.legacy_tokenizer:
- tokenizer = get_tokenizer()
- else:
- tokenizer = build_tokenizer(args)
-
- # Wrap model in inference wrapper.
- model = GPTInferenceWrapper(model, args, context)
-
- # Note: the following is taken from AbstractModelInferenceWrapper.prep_model_for_inference().
- from megatron.core import parallel_state
-
- model.model_is_pipeline_parallel = not (
- parallel_state.is_pipeline_first_stage() and parallel_state.is_pipeline_last_stage()
- )
-
- # Text generation controller.
- controller = TextGenerationController(model, tokenizer)
-
- return controller
-
-
def run_inference(
requests: List[Request],
engine: DynamicInferenceEngine,
@@ -257,6 +80,16 @@ def run_inference(
args = get_args()
+ # Parse batch boundaries for batch-drain mode.
+ batch_ranges = None
+ if args.drain_between_batches and args.batch_boundaries:
+ boundaries = [int(x) for x in args.batch_boundaries.split(",")]
+ num_requests_total = len(requests)
+ batch_ranges = []
+ for i, start in enumerate(boundaries):
+ end = boundaries[i + 1] if i + 1 < len(boundaries) else num_requests_total
+ batch_ranges.append((start, end))
+
# Initialize request arrival times.
base_arrival_time = get_curr_time()
for request in requests:
@@ -284,72 +117,24 @@ def _add_request():
"""
nonlocal num_requests_added
_request = requests[num_requests_added]
- engine.add_request(
- num_requests_added,
- _request.prompt_text,
- _request.sampling_params,
- )
+ engine.add_request(num_requests_added, _request.prompt_text, _request.sampling_params)
_request.time_start = get_curr_time()
_request.state = "started"
num_requests_added += 1
tbar.update(1)
- while True:
- # Add requests.
- add_start = get_curr_time()
- if args.incoming_requests_per_step is None:
- # Add requests with 'earlier' arrival time.
- while num_requests_added < num_requests_total:
- if requests[num_requests_added].time_arrival > add_start:
- break
- _add_request()
- else:
- # Add deterministic number of requests (generally used for debugging).
- for i in range(min(
- args.incoming_requests_per_step,
- num_requests_total - num_requests_added,
- )):
- _add_request()
- add_times.append(get_curr_time() - add_start)
-
- # Step inference engine (i.e., generate a token for each active request).
- # Before step, we haven't done the scheduling, so we cannot know the is_decode_only
- try:
- result = engine.step_modern()
- except EngineSuspendedError as e:
- result = e
- pass # ignore error in order to call 'engine.resume()' below.
- attempted_step_count += 1
-
- # After step, we lost track of last iteration's is_decode_only, so we need to get it from the engine
- is_decode_only = engine.is_decode_only
-
- # Test suspending and resuming engine.
- if args.suspend_resume_interval is not None:
-
- # Suspend.
- if attempted_step_count % args.suspend_resume_interval == 0:
- print("**** step %d/%d ... suspend." % (engine.step_count, attempted_step_count))
- engine.suspend()
-
- # Resume, 0+ attempted steps later.
- if (
- attempted_step_count > 0
- and
- (attempted_step_count - args.suspend_resume_interval // 2)
- % args.suspend_resume_interval == 0
- ):
- print("**** step %d/%d ... resume." % (engine.step_count, attempted_step_count))
- engine.resume()
-
- # If engine suspended, continue to next iter.
- if isinstance(result, EngineSuspendedError):
- continue
+ def _process_step_result(result):
+ """Process a single engine step result, updating bookkeeping state."""
+ nonlocal total_output_tokens, num_requests_finished
+
+ is_decode_only = engine.is_decode_only
# Record cuda_graph_request_count.
cuda_graph_request_count = result["cuda_graph_request_count"]
if args.cuda_graph_impl == "local" and cuda_graph_request_count is not None:
- cuda_graph_request_count_map[cuda_graph_request_count] = cuda_graph_request_count_map.get(cuda_graph_request_count, 0) + 1
+ cuda_graph_request_count_map[cuda_graph_request_count] = (
+ cuda_graph_request_count_map.get(cuda_graph_request_count, 0) + 1
+ )
# Update requests.
active_request_ids = result["active_request_ids"]
@@ -374,6 +159,8 @@ def _add_request():
request.request_id = finished_request.request_id
request.events = finished_request.events
+ request.ttft = finished_request.ttft
+
# Update prompt, in case engine has been suspended and resumed.
request.prompt_tokens = finished_request.prompt_tokens.tolist()
request.prompt_text = finished_request.prompt
@@ -399,47 +186,118 @@ def _add_request():
num_requests_finished += 1
output_times.append(get_curr_time() - output_start)
- # Check if all requests are finished.
- if not (engine.has_unfinished_requests() or num_requests_added < num_requests_total):
- break
+ if batch_ranges is not None:
+ # Batch-drain mode: add all requests in a batch, drain, then next batch.
+ for batch_idx, (batch_start, batch_end) in enumerate(batch_ranges):
+ # Add all requests in current batch.
+ add_start = get_curr_time()
+ while num_requests_added < batch_end:
+ _add_request()
+ add_times.append(get_curr_time() - add_start)
+
+ # Step until all active requests finish (drain).
+ while engine.has_unfinished_requests():
+ try:
+ result = engine.step_modern()
+ except EngineSuspendedError as e:
+ result = e
+ attempted_step_count += 1
+
+ if isinstance(result, EngineSuspendedError):
+ continue
+
+ _process_step_result(result)
+ else:
+ # Original mode: add requests per step based on arrival time or count.
+ while True:
+ # Add requests.
+ add_start = get_curr_time()
+ if args.incoming_requests_per_step is None:
+ # Add requests with 'earlier' arrival time.
+ while num_requests_added < num_requests_total:
+ if requests[num_requests_added].time_arrival > add_start:
+ break
+ _add_request()
+ else:
+ # Add deterministic number of requests (generally used for debugging).
+ for i in range(
+ min(args.incoming_requests_per_step, num_requests_total - num_requests_added)
+ ):
+ _add_request()
+ add_times.append(get_curr_time() - add_start)
+
+ # Step inference engine (i.e., generate a token for each active request).
+ # Before step, we haven't done the scheduling, so we cannot know the is_decode_only
+ try:
+ result = engine.step_modern()
+ except EngineSuspendedError as e:
+ result = e
+ pass # ignore error in order to call 'engine.resume()' below.
+ attempted_step_count += 1
+
+ # Test suspending and resuming engine.
+ if args.suspend_resume_interval is not None:
+
+ # Suspend.
+ if attempted_step_count % args.suspend_resume_interval == 0:
+ print("**** step %d/%d ... suspend." % (engine.context.step_count, attempted_step_count))
+ engine.suspend()
+
+ # Resume, 0+ attempted steps later.
+ if (
+ attempted_step_count > 0
+ and (attempted_step_count - args.suspend_resume_interval // 2)
+ % args.suspend_resume_interval
+ == 0
+ ):
+ print("**** step %d/%d ... resume." % (engine.context.step_count, attempted_step_count))
+ engine.resume()
+
+ # If engine suspended, continue to next iter.
+ if isinstance(result, EngineSuspendedError):
+ continue
+
+ _process_step_result(result)
+
+ # Check if all requests are finished.
+ if not (engine.has_unfinished_requests() or num_requests_added < num_requests_total):
+ break
# Resume engine (NOOP if not suspended).
- if engine.is_suspended:
- engine.resume()
+ engine.resume()
return {
- "step_times" : step_times,
- "add_times" : add_times,
- "output_times" : output_times,
- "total_output_tokens" : total_output_tokens,
- "cuda_graph_request_count_map" : cuda_graph_request_count_map,
+ "step_times": step_times,
+ "add_times": add_times,
+ "output_times": output_times,
+ "total_output_tokens": total_output_tokens,
+ "cuda_graph_request_count_map": cuda_graph_request_count_map,
}
@torch.inference_mode()
def main():
-
+ """Run dynamic inference."""
# Initialize Megatron.
initialize_megatron(
- extra_args_provider=add_dynamic_inference_args,
+ extra_args_provider=add_inference_args,
args_defaults={'no_load_rng': True, 'no_load_optim': True},
)
# Start Nsight profiler.
if os.environ.get("NSIGHT_PREFIX"):
torch.cuda.cudart().cudaProfilerStart()
-
- level_str = os.getenv("LOG_LEVEL", "INFO").upper()
- level = getattr(logging, level_str, logging.INFO)
+
+ level_str = os.getenv("LOG_LEVEL", "INFO").upper()
+ level = getattr(logging, level_str, logging.INFO)
logging.basicConfig(level=level, force=True)
configure_nvtx_profiling(True)
args = get_args()
- if args.legacy_tokenizer:
- tokenizer = get_tokenizer()
- else:
- tokenizer = build_tokenizer(args)
+
+ # Build tokenizer
+ tokenizer = build_tokenizer(args)
# Reset peak memory stats so functional tests measure this run and not
# whatever happened earlier during initialization.
@@ -456,42 +314,36 @@ def main():
termination_id=args.termination_id if args.termination_id is not None else tokenizer.eod,
top_n_logprobs=args.top_n_logprobs,
stop_words=args.stop_words,
- )
-
- model = get_model()
+ )
- mamba_inference_state_config = get_mamba_inference_state_config_from_model(model)
+ model = get_model_for_inference()
# Requests, context, controller.
requests = build_requests(args, tokenizer, sampling_params)
- context = get_inference_context(
- requests,
- sampling_params,
- mamba_inference_state_config=mamba_inference_state_config,
- )
- controller = get_inference_controller(model, context)
+ inference_config = get_inference_config_from_model_and_args(model, args)
+
+ # Calculate max_sequence_length from requests
+ max_gen_length = sampling_params.num_tokens_to_generate
+ max_context_length = max(len(r.prompt_tokens) for r in requests)
+ inference_config.max_sequence_length = max_context_length + max_gen_length
+ context = DynamicInferenceContext(model.config, inference_config)
+ wrapped_model = GPTInferenceWrapper(model, context)
+ controller = TextGenerationController(wrapped_model, tokenizer)
# Validate all context_length's <= max_tokens.
- if args.disable_chunked_prefill:
+ if not args.enable_chunked_prefill:
invalid_prompt_length_map = {}
for request_idx, request in enumerate(requests):
if len(request.prompt_tokens) > context.max_tokens:
invalid_prompt_length_map[request_idx] = len(request.prompt_tokens)
- assert not invalid_prompt_length_map, (
- "request idxs with prompts longer than context.max_tokens: "
- ", ".join(f"{k}({v})" for k, v in invalid_prompt_length_map.items())
+ assert (
+ not invalid_prompt_length_map
+ ), "request idxs with prompts longer than context.max_tokens: " ", ".join(
+ f"{k}({v})" for k, v in invalid_prompt_length_map.items()
)
# Inference engine.
- engine = DynamicInferenceEngine(
- controller,
- context,
- enable_cuda_graph=args.cuda_graph_impl == "local",
- random_seed=args.seed,
- track_paused_request_events=args.inference_dynamic_batching_track_paused_request_events,
- enable_chunked_prefill=not args.disable_chunked_prefill,
- inference_logging_step_interval=args.inference_logging_step_interval,
- )
+ engine = DynamicInferenceEngine(controller, context)
setup_prefix = build_dynamic_engine_setup_prefix(args, model, context, requests)
print("~~~")
@@ -522,14 +374,13 @@ def main():
# Validate all requests finished.
for request in requests:
- assert request.state == "finished", (
- f"request.state == '{request.state}' != 'finished'."
- )
+ assert request.state == "finished", f"request.state == '{request.state}' != 'finished'."
peak_mem_stats = get_global_peak_memory_stats_bytes()
# Print unique prompts + outputs.
if torch.distributed.get_rank() == 0:
+
def escape_str(s):
return s.replace("\n", "\\n")
@@ -547,7 +398,10 @@ def escape_str(s):
# ---- Prompt summary line ----
prompt_len = len(requests[request_idxs[0]].prompt_tokens)
escaped_prompt_text = escape_str(prompt_text)
- print(f"\n{unique_idx+1}/{len(unique_prompt_map)} [n {len(request_idxs)}, l {prompt_len}] {escaped_prompt_text}")
+ print(
+ f"\n{unique_idx+1}/{len(unique_prompt_map)}"
+ f"[n {len(request_idxs)}, l {prompt_len}] {escaped_prompt_text}"
+ )
# ---- Group all outputs for this prompt ----
output_map = defaultdict(list)
@@ -567,16 +421,17 @@ def escape_str(s):
# Use hash of prompt + generated text in case engine was
# suspended and resumed, which misaligns boundary between
# prompt and generated tokens.
- o_hash = hashlib.sha256(
- (prompt_text + output_text).encode()
- ).hexdigest()[:6]
+ o_hash = hashlib.sha256((prompt_text + output_text).encode()).hexdigest()[:6]
o_len = len(requests[output_request_idxs[0]].output_tokens)
escaped_output_text = escape_str(output_text)
else:
o_hash = "--"
o_len = 0
escaped_output_text = "--"
- print(f" >>>> [n {len(output_request_idxs)}, {o_len} tokens, hash {o_hash}{', ' if evicted else ''}] {escaped_output_text}")
+ print(
+ f" >>>> [n {len(output_request_idxs)}, {o_len} tokens, hash {o_hash}"
+ f"{', ' if evicted else ''}] {escaped_output_text}"
+ )
text_hashes.append(o_hash)
# Write results to JSON. Primarily used for functional testing.
@@ -592,15 +447,20 @@ def escape_str(s):
"generated_text": req.output_text,
"generated_tokens": req.output_tokens,
"latency": req.time_end - req.time_start,
- "cuda_graph_request_count_map" : result["cuda_graph_request_count_map"],
- "step_count" : engine.step_count,
- "top_n_logprobs" : getattr(req, 'generated_top_n_logprobs', None),
- "prompt_top_n_logprobs" : getattr(req, 'prompt_top_n_logprobs', None),
+ "ttft": req.ttft, # Time-to-first-token in seconds
+ "cuda_graph_request_count_map": result["cuda_graph_request_count_map"],
+ "step_count": engine.context.step_count,
+ "top_n_logprobs": getattr(req, 'generated_top_n_logprobs', None),
+ "prompt_top_n_logprobs": getattr(req, 'prompt_top_n_logprobs', None),
}
if req.sampling_params.return_log_probs:
result_dict["prompt_logprobs"] = getattr(req, 'prompt_log_probs', None)
- result_dict["generated_logprobs"] = getattr(req, 'generated_log_probs', None)
+ result_dict["generated_logprobs"] = getattr(
+ req, 'generated_log_probs', None
+ )
result_dict["logprobs"] = getattr(req, 'logprobs', None)
+ if args.output_request_events:
+ result_dict["events"] = [e.serialize() for e in req.events]
json_results[req.request_id] = result_dict
# Track system-level throughput as a test / debug metric
@@ -609,6 +469,7 @@ def escape_str(s):
# Attach peak memory metrics; the functional test only validates these
# if the fields exist in the golden values.
json_results.update(peak_mem_stats)
+ json_results["lifetime_prefill_token_count"] = engine.context.lifetime_prefill_token_count
print(f' Saving results to {args.output_path}')
with open(args.output_path, "w") as fp:
@@ -631,7 +492,7 @@ def escape_str(s):
d_count = len(d_times)
p_mean = p_total / p_count
- d_mean = d_total / d_count if d_count != 0 else 0.
+ d_mean = d_total / d_count if d_count != 0 else 0.0
# Commented out for now as the step/add/output times are not calculated correctly.
# print(
@@ -643,18 +504,13 @@ def escape_str(s):
# f"mean [ p {p_mean:.3f}s, d {d_mean:.3f}s ], "
# f"count [ p {p_count}, d {d_count} ]."
# )
- capture_str = (
- f"{engine.capture_stats['time']:.2f} sec"
- if engine.capture_stats else
- "--"
- )
+ capture_str = f"{engine.capture_stats['time']:.2f} sec" if engine.capture_stats else "--"
print(
- f"{setup_prefix} … "
- f"throughput: {throughput:.3f} tok/s … ",
+ f"{setup_prefix} … " f"throughput: {throughput:.3f} tok/s … ",
f"total time: {total_time:.3f}s … "
f"mem {peak_alloc_gb:.1f}/{peak_resvd_gb:.1f} GB … "
- f"steps: {engine.step_count:d} … "
- f"capture {capture_str}"
+ f"steps: {engine.context.step_count:d} … "
+ f"capture {capture_str}",
)
print("~~~")
diff --git a/examples/inference/gpt/gpt_dynamic_inference_12b.sh b/examples/inference/gpt/gpt_dynamic_inference_12b.sh
index 4991d9d5177..ca21bb170a5 100644
--- a/examples/inference/gpt/gpt_dynamic_inference_12b.sh
+++ b/examples/inference/gpt/gpt_dynamic_inference_12b.sh
@@ -69,7 +69,6 @@ ARGS=" \
--tiktoken-pattern v2 \
--tokenizer-model ${TOKENIZER_MODEL} \
--distributed-timeout-minutes 2400 \
- --transformer-impl local \
--use-flash-attn \
--inference-rng-tracker \
\
diff --git a/examples/inference/gpt/gpt_dynamic_inference_357m.sh b/examples/inference/gpt/gpt_dynamic_inference_357m.sh
index 44abb575c63..cc99bdddec1 100644
--- a/examples/inference/gpt/gpt_dynamic_inference_357m.sh
+++ b/examples/inference/gpt/gpt_dynamic_inference_357m.sh
@@ -32,6 +32,7 @@ export CUDA_DEVICE_MAX_CONNECTIONS=1
# Miscellaneous.
: ${USE_COORDINATOR=0}
: ${ENGINE=dynamic}
+: ${NPROC_PER_NODE=1}
: ${EXTRA_ARGS=""}
# NSIGHT_PREFIX=/path/to/nsight/profile
@@ -101,7 +102,7 @@ fi
if [[ "${USE_COORDINATOR}" == "0" ]]; then
CMD="python -m examples.inference.gpt.gpt_${ENGINE}_inference ${ARGS}"
else
- CMD="python -um examples.inference.gpt.gpt_${ENGINE}_inference_with_coordinator ${ARGS}"
+ CMD="python -m torch.distributed.run --nproc-per-node ${NPROC_PER_NODE} -m examples.inference.gpt.gpt_${ENGINE}_inference_with_coordinator ${ARGS}"
fi
if [[ -v NSIGHT_PREFIX ]]; then
diff --git a/examples/inference/gpt/gpt_dynamic_inference_with_coordinator.py b/examples/inference/gpt/gpt_dynamic_inference_with_coordinator.py
index cbb7a1aa745..d34749e3a5a 100644
--- a/examples/inference/gpt/gpt_dynamic_inference_with_coordinator.py
+++ b/examples/inference/gpt/gpt_dynamic_inference_with_coordinator.py
@@ -2,43 +2,50 @@
import asyncio
import json
+import logging
import os
import time
-import torch
-import torch.distributed as dist
+import warnings
from collections import defaultdict
-from tqdm import tqdm
from typing import List
-import warnings
-import logging
-from examples.inference.gpt.gpt_dynamic_inference import (
- add_dynamic_inference_args,
- get_inference_context,
- get_inference_controller,
- get_model,
-)
-from examples.inference.gpt.utils import (
- Request,
- build_dynamic_engine_setup_prefix,
- build_requests,
- add_common_inference_args
-)
+import torch
+import torch.distributed as dist
-from megatron.core import parallel_state
+from examples.inference.gpt.utils import Request, build_dynamic_engine_setup_prefix, build_requests
from megatron.core.inference.engines import DynamicInferenceEngine
+from megatron.core.inference.engines.dynamic_engine import EngineState
from megatron.core.inference.inference_client import InferenceClient
from megatron.core.inference.inference_request import DynamicInferenceRequestRecord
from megatron.core.inference.sampling_params import SamplingParams
-from megatron.core.utils import get_mamba_inference_state_config_from_model
-
+from megatron.inference.utils import (
+ add_inference_args,
+ get_dynamic_inference_engine,
+ get_model_for_inference,
+)
from megatron.training import get_args, get_tokenizer, initialize_megatron
-from megatron.training.arguments import parse_args
# pylint: disable=line-too-long
logging.basicConfig(level=logging.INFO, force=True)
+
+async def suspend_resume_cycle(client, engine, args, futures):
+ """Wait for all in-flight requests, then suspend/train/resume."""
+ await asyncio.gather(*futures)
+
+ client.pause_engines()
+ await engine.wait_until(EngineState.PAUSED)
+ client.suspend_engines()
+ await engine.wait_until(EngineState.SUSPENDED)
+ if args.suspend_timeout > 0:
+ await asyncio.sleep(args.suspend_timeout)
+ client.resume_engines()
+ await engine.wait_until(EngineState.RESUMED)
+ client.unpause_engines()
+ await engine.wait_until(EngineState.RUNNING)
+
+
async def main(
engine: DynamicInferenceEngine,
requests: List[Request],
@@ -51,54 +58,43 @@ async def main(
"Sampling parameters are specified per request.",
DeprecationWarning,
)
-
+
# once you call engine.start_listening_to_data_parallel_coordinator,
# the engine will start accepting requests from the data parallel coordinator.
# and processing them in an asyncio coroutine.
# leaving inference_coordinator_port as None will find a free port automatically.
-
+ args = get_args()
+
dp_addr = await engine.start_listening_to_data_parallel_coordinator(
inference_coordinator_port=port,
launch_inference_coordinator=True,
+ coordinator_schedule_output_path=args.coordinator_schedule_output_path,
)
- args = get_args()
-
- # Test suspend/resume intervals.
- if dist.get_rank() == 0 and args.suspend_resume_interval is not None:
- # Since the client doesn't directly call engine.async_step here, we test
- # the suspend-resume system ~4 times.
- suspend_resume_interval = max(1, len(requests) // 4)
- suspend_idxs = set(range(
- suspend_resume_interval,
- len(requests) + 1,
- suspend_resume_interval,
- ))
- resume_idxs = set(
- min(len(requests), i + suspend_resume_interval // 2)
- for i in suspend_idxs
- )
- else:
- suspend_idxs = set()
- resume_idxs = set()
+ # All ranks agree on the number of suspend/resume cycles from args.
+ num_suspend_resume_cycles = len(requests) // args.suspend_resume_interval if args.suspend_resume_interval else 0
# Create client and run example.
if dist.get_rank() == 0:
- client = InferenceClient(dp_addr) # submits requests to the inference coordinator
- await client.start()
+ client = InferenceClient(dp_addr, deserialize=True) # submits requests to the inference coordinator
+ client.start()
base_arrival_time = time.time_ns() / 10**9
for request in requests:
request.time_arrival = request.time_offset + base_arrival_time
futures = []
num_requests_total = len(requests)
num_requests_added = 0
- # logging.info("Waiting for 20 seconds before starting to add requests. This is to mimic an RL style setup..")
- # time.sleep(20)
+ next_suspend_at = args.suspend_resume_interval or 0
+ cycles_done = 0
+
while True:
current_time = time.time_ns() / 10**9
if args.incoming_requests_per_step is None:
# Only add requests that have arrived at the current time.
- while num_requests_added < num_requests_total and requests[num_requests_added].time_arrival <= current_time:
+ while (
+ num_requests_added < num_requests_total
+ and requests[num_requests_added].time_arrival <= current_time
+ ):
request = requests[num_requests_added]
# These add-request calls will queue up the request on a zmq socket and return
# instantaneously. They will return an asyncio future which can be awaited for
@@ -106,18 +102,16 @@ async def main(
futures.append(client.add_request(request.prompt_text, request.sampling_params))
num_requests_added += 1
- # Test suspend/resume.
- if num_requests_added in suspend_idxs:
- client.suspend_engines()
- if num_requests_added in resume_idxs:
- client.resume_engines()
+ if num_requests_added >= next_suspend_at and cycles_done < num_suspend_resume_cycles:
+ await suspend_resume_cycle(client, engine, args, futures)
+ cycles_done += 1
+ next_suspend_at += args.suspend_resume_interval
else:
# Add deterministic number of requests (generally used for debugging).
- for i in range(min(
- args.incoming_requests_per_step,
- num_requests_total - num_requests_added
- )):
+ for i in range(
+ min(args.incoming_requests_per_step, num_requests_total - num_requests_added)
+ ):
# Change sampling parameters to force different generation lengths.
request = requests[num_requests_added]
n = request.sampling_params.num_tokens_to_generate
@@ -125,19 +119,25 @@ async def main(
futures.append(client.add_request(request.prompt_text, request.sampling_params))
num_requests_added += 1
- # Test suspend/resume.
- if num_requests_added in suspend_idxs:
- client.suspend_engines()
- if num_requests_added in resume_idxs:
- client.resume_engines()
+ if num_requests_added >= next_suspend_at and cycles_done < num_suspend_resume_cycles:
+ await suspend_resume_cycle(client, engine, args, futures)
+ cycles_done += 1
+ next_suspend_at += args.suspend_resume_interval
if num_requests_added == num_requests_total:
break
# Relinquish control since there are no more requests to add at the moment. This allows the engine to run.
await asyncio.sleep(0)
-
+
# While we wait for the requests to complete, the engine runs in the background.
results: List[DynamicInferenceRequestRecord] = await asyncio.gather(*futures)
+ else:
+ # Non-rank-0: match the suspend/resume cycles that rank 0 drives.
+ for _ in range(num_suspend_resume_cycles):
+ await engine.wait_until(EngineState.PAUSED)
+ await engine.wait_until(EngineState.SUSPENDED)
+ await engine.wait_until(EngineState.RESUMED)
+ await engine.wait_until(EngineState.RUNNING)
if dist.get_rank() == 0:
# Write results to JSON. Primarily used for functional testing.
@@ -145,8 +145,7 @@ async def main(
json_results = {}
throughputs = []
- for record in results:
- req = record.merge()
+ for req in results:
result_dict = {
"input_prompt": req.prompt,
"generated_text": req.generated_text.replace("\n", "\\n"),
@@ -157,6 +156,9 @@ async def main(
result_dict["logprobs"] = req.prompt_log_probs + req.generated_log_probs
throughput = len(req.generated_tokens) / req.latency
throughputs.append(throughput)
+ if req.routing_indices is not None:
+ result_dict["routing_indices"] = req.routing_indices.tolist()
+
json_results[req.request_id] = result_dict
throughput_dict = {"throughput": throughputs}
if args.throughput_check_only:
@@ -166,35 +168,42 @@ async def main(
else:
print("Results:")
unique_prompt_map = defaultdict(list)
- for record in results:
- req = record.merge()
+ for req in results:
unique_prompt_map[req.prompt].append(req)
for idx, (prompt_text, reqs) in enumerate(unique_prompt_map.items()):
- print(f"%d/%d. prompt '%s' ... [%d] output '%s'." % (
- idx,
- len(unique_prompt_map),
- prompt_text.replace("\n", "\\n"),
- len(reqs),
- reqs[0].generated_text.replace("\n", "\\n"),
- ))
-
- # kill the engines and suspend the client
- # Right now, we can only call stop when all requests are done.
- # Todo: Make this explicit in the Client class....
- await client.stop_engines()
- client.stop()
+ print(
+ f"%d/%d. prompt '%s' ... [%d] output '%s'."
+ % (
+ idx,
+ len(unique_prompt_map),
+ prompt_text.replace("\n", "\\n"),
+ len(reqs),
+ reqs[0].generated_text.replace("\n", "\\n"),
+ )
+ )
+
+ # Pause before stopping: STOP requires PAUSED or SUSPENDED state.
+ client.pause_engines()
+
+ await engine.wait_until(EngineState.PAUSED)
+
+ if dist.get_rank() == 0:
+ client.stop_engines()
+
+ await engine.wait_until(EngineState.STOPPED)
- # once the stop signal eventually makes its way to each GPU, the engines will stop.
- await asyncio.gather(engine.engine_loop_task)
+ if dist.get_rank() == 0:
+ client.shutdown_coordinator()
+ client.stop()
logging.info(f"Rank: {dist.get_rank()} stopped their engine instance successfully.")
if __name__ == "__main__":
- # enable inference mode in the very beginning as some fp-8 optimizations
+ # enable inference mode in the very beginning as some fp8 optimizations
# check for it.
with torch.inference_mode():
initialize_megatron(
- extra_args_provider=add_dynamic_inference_args,
+ extra_args_provider=add_inference_args,
args_defaults={'no_load_rng': True, 'no_load_optim': True},
)
@@ -213,34 +222,14 @@ async def main(
),
)
- # Requests, context, conroller.
- model = get_model()
- mamba_inference_state_config = get_mamba_inference_state_config_from_model(model)
- requests = (
- build_requests(args, tokenizer, sampling_params) if dist.get_rank() == 0 else None
- )
+ model = get_model_for_inference()
- context = get_inference_context(
- None,
- None,
- calculate_max_sequence_length_from_requests=False,
- mamba_inference_state_config=mamba_inference_state_config,
- )
-
- controller = get_inference_controller(model, context)
+ requests = build_requests(args, tokenizer, sampling_params)
- # Inference engine.
- engine = DynamicInferenceEngine(
- controller,
- context,
- enable_cuda_graph=args.cuda_graph_impl == "local",
- random_seed=args.seed,
- enable_chunked_prefill=not args.disable_chunked_prefill,
- inference_logging_step_interval=args.inference_logging_step_interval,
- )
+ engine = get_dynamic_inference_engine(model=model)
if dist.get_rank() == 0:
- setup_prefix = build_dynamic_engine_setup_prefix(args, model, context, requests)
+ setup_prefix = build_dynamic_engine_setup_prefix(args, model, engine.context, requests)
print("~~~")
print(setup_prefix)
print("~~~")
@@ -249,13 +238,7 @@ async def main(
if os.environ.get("NSIGHT_PREFIX"):
torch.cuda.cudart().cudaProfilerStart()
- asyncio.run(
- main(
- engine,
- requests,
- args.inference_coordinator_port,
- )
- )
+ asyncio.run(main(engine, requests, args.inference_coordinator_port))
# Stop Nsight profiler.
if os.environ.get("NSIGHT_PREFIX"):
diff --git a/examples/inference/gpt/gpt_static_inference.py b/examples/inference/gpt/gpt_static_inference.py
index 03a60927ab2..17cf7c53b05 100644
--- a/examples/inference/gpt/gpt_static_inference.py
+++ b/examples/inference/gpt/gpt_static_inference.py
@@ -1,21 +1,11 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
import os
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
-from model_provider import model_provider
-from gpt_builders import gpt_builder
-from mamba_builders import mamba_builder
-import torch
import sys
import time
-import warnings
-from functools import partial
from argparse import Namespace
import torch
-import tqdm
from megatron.core.inference.contexts import StaticInferenceContext
from megatron.core.inference.engines import StaticInferenceEngine
@@ -23,17 +13,12 @@
from megatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapper import (
GPTInferenceWrapper,
)
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
from megatron.core.inference.sampling_params import SamplingParams
from megatron.core.inference.text_generation_controllers.text_generation_controller import (
TextGenerationController,
)
-from megatron.core.tokenizers.text.utils.build_tokenizer import build_tokenizer
+from megatron.core.tokenizers.utils.build_tokenizer import build_tokenizer
from megatron.core.transformer.module import MegatronModule
-from pretrain_gpt import model_provider as gpt_model_provider
-from pretrain_mamba import model_provider as mamba_model_provider
sys.path.append(
os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir, os.path.pardir))
@@ -41,18 +26,18 @@
import asyncio
import json
-from typing import Any, AsyncIterator, List
+from typing import List
-from examples.inference.gpt.utils import add_common_inference_args, build_requests
-from megatron.core import mpu
-from megatron.training import get_args, get_model, get_tokenizer, print_rank_0
-from megatron.training.checkpointing import load_checkpoint
+from examples.inference.gpt.utils import build_requests
+from megatron.inference.utils import add_inference_args, get_model_for_inference
+from megatron.training import get_args, get_tokenizer, print_rank_0
from megatron.training.initialize import initialize_megatron
+
def add_static_inference_args(parser):
"""Static inference arguments."""
- add_common_inference_args(parser)
+ add_inference_args(parser)
group = parser.add_argument_group(title='Static inference')
group.add_argument(
@@ -79,34 +64,17 @@ def get_inference_engine(args: Namespace, model: MegatronModule) -> StaticInfere
Returns:
AbstractBackend: The chosen backend
"""
- if args.legacy_tokenizer:
- tokenizer = get_tokenizer()
- else:
- tokenizer = build_tokenizer(args)
- inference_wrapper_config = InferenceWrapperConfig(
- hidden_size=args.hidden_size,
- inference_batch_times_seqlen_threshold=args.inference_batch_times_seqlen_threshold,
- fp32_residual_connection=args.fp32_residual_connection,
- params_dtype=args.params_dtype,
- padded_vocab_size=args.padded_vocab_size,
- inference_max_requests=args.inference_max_batch_size,
- inference_max_seq_length=args.inference_max_seq_length,
- nccl_all_reduce_for_prefill=args.nccl_all_reduce_for_prefill,
- fp8=args.fp8,
- moe_pad_experts_for_cuda_graph_inference = args.moe_pad_experts_for_cuda_graph_inference
- )
-
- inference_context = StaticInferenceContext.from_config(inference_wrapper_config)
-
- inference_wrapped_model = GPTInferenceWrapper(
- model, inference_wrapper_config, inference_context
+ tokenizer = build_tokenizer(args)
+ inference_context = StaticInferenceContext(
+ args.inference_max_requests, args.inference_max_seq_length
)
+ inference_wrapped_model = GPTInferenceWrapper(model, inference_context)
text_generation_controller = TextGenerationController(
inference_wrapped_model=inference_wrapped_model, tokenizer=tokenizer
)
engine_kwargs = {
- "text_generation_controller" : text_generation_controller,
- "legacy" : args.use_legacy_static_engine,
+ "text_generation_controller": text_generation_controller,
+ "legacy": args.use_legacy_static_engine,
}
if not args.use_legacy_static_engine:
engine_kwargs["buffer_size_gb"] = args.inference_dynamic_batching_buffer_size_gb
@@ -165,22 +133,7 @@ def main():
args = get_args()
- if args.max_batch_size is not None:
- warnings.warn(
- f"`--max-batch-size` has been deprecated in favor of `--inference-max-requests`."
- )
- args.inference_max_batch_size = max(args.max_batch_size, args.inference_max_batch_size)
-
- # Set up model and load checkpoint
- if args.model_provider == "gpt":
- model_builder = gpt_builder
- elif args.model_provider == "mamba":
- model_builder = mamba_builder
- else:
- raise ValueError(f"Invalid model provider {args.model_provider}")
- model = get_model(partial(model_provider, model_builder), wrap_with_ddp=False)
- load_checkpoint(model, None, None, strict=False)
- model = model[0]
+ model = get_model_for_inference()
inference_engine = get_inference_engine(args, model)
@@ -193,10 +146,9 @@ def main():
top_n_logprobs=args.top_n_logprobs,
)
- if args.legacy_tokenizer:
- tokenizer = get_tokenizer()
- else:
- tokenizer = build_tokenizer(args)
+ # Build tokenizer
+ tokenizer = build_tokenizer(args)
+
requests = build_requests(args, tokenizer)
prompts = [r.prompt_text for r in requests]
@@ -276,7 +228,7 @@ def main():
)
),
len(requests),
- args.inference_max_batch_size,
+ args.inference_max_requests,
stats["allocated_bytes.all.peak"] / (1024**3),
stats["reserved_bytes.all.peak"] / (1024**3),
latency,
@@ -293,6 +245,5 @@ def main():
torch.distributed.destroy_process_group()
-
if __name__ == "__main__":
main()
diff --git a/examples/inference/gpt/utils.py b/examples/inference/gpt/utils.py
index a04b856c0a6..c9b1c05c544 100644
--- a/examples/inference/gpt/utils.py
+++ b/examples/inference/gpt/utils.py
@@ -1,158 +1,23 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
import copy
-import json
import itertools
+import json
import random
import time
-import torch
from argparse import ArgumentParser, Namespace
-from tqdm import tqdm
+from functools import partial
from typing import Any, List, Optional
-from megatron.core.inference.inference_request import DynamicInferenceRequest
+import torch
+from tqdm import tqdm
+
from megatron.core.inference.contexts import DynamicInferenceContext
from megatron.core.inference.contexts.dynamic_context import get_mem_size_str
-from megatron.core.transformer.module import MegatronModule
-
+from megatron.core.inference.inference_request import DynamicInferenceRequest
from megatron.core.inference.sampling_params import SamplingParams
-
-
-def add_common_inference_args(parser: ArgumentParser) -> ArgumentParser:
- """Common inference arguments."""
-
- group = parser.add_argument_group(title='Common inference')
-
- group.add_argument("--temperature", type=float, default=1.0, help='Sampling temperature.')
- group.add_argument("--top_k", type=int, default=1, help='Top k sampling.')
- group.add_argument("--top_p", type=float, default=0.0, help='Top p sampling.')
- group.add_argument(
- "--return-log-probs",
- action='store_true',
- default=False,
- help='Return the log probabilities of the final output tokens',
- )
- group.add_argument(
- "--prompts",
- metavar='N',
- type=str,
- nargs='+',
- help='Input prompts with each prompt within quotes and seperated by space',
- )
- group.add_argument(
- "--num-tokens-to-prompt",
- type=int,
- nargs="+",
- default=[64, 1024],
- help='Number of tokens to use for simulated prompts. This should be a '
- 'space-separated pair of integers, and the generated prompt lengths will '
- 'be uniformly sampled within this range.',
- )
- group.add_argument(
- "--num-tokens-to-generate",
- type=int,
- default=30,
- help='Number of tokens to generate for each prompt',
- )
- group.add_argument(
- "--num-tokens-from-file",
- action='store_true',
- default=False,
- help='Use per-prompt num_tokens_to_generate from prompt file',
- )
- group.add_argument(
- "--top-n-logprobs",
- type=int,
- default=0,
- help='Return the top n logprobs for the generated tokens and their corresponding token as a dictionary',
- )
- group.add_argument(
- "--incoming-requests-per-step",
- type=int, default=None,
- help="Add a deterministic number of requests per step. This arg is "
- "prioritized over `--incoming-requests-per-sec` below (which is non-"
- "deterministic). Note that the number of requests added per step is "
- "additionally limited by the inference context's `max_requests`, "
- "`max_tokens`, and KV buffer size.",
- )
- group.add_argument(
- "--incoming-requests-per-sec",
- type=float,
- default=100.0,
- help="Simulated number of requests per second. Set to -1 to add all requests together.",
- )
- group.add_argument(
- "--incoming-requests-duration",
- type=float,
- default=10.0,
- help="Total amount of time to simulate that requests are "
- "arriving. Multiply this value with "
- "`--incoming-requests-per-sec` to get the approximate "
- "total number of requests. Set to -1 to add all requests together.",
- )
- group.add_argument(
- "--model-provider",
- choices=["mamba", "gpt"],
- default="gpt",
- help="Model provider",
- )
- group.add_argument(
- "--skip-prompt-log-probs",
- action='store_true',
- default=False,
- help='Skip prompt log probs.',
- )
- group.add_argument(
- "--stop-words",
- metavar='WORD',
- type=str,
- nargs='+',
- default=None,
- help='Stop words to terminate generation. Each word should be quoted and '
- 'separated by space. Example: --stop-words "\\n\\n" "END" "###"',
- )
- group.add_argument(
- "--output-path",
- type=str,
- default=None,
- help="Path to save generations as JSON",
- )
- group.add_argument(
- "--output-every-n-results",
- type=int,
- default=1,
- help="To minimize the output file size of larger runs, only write the "
- "results of every `n` requests.",
- )
- group.add_argument(
- "--prompt-file",
- help='Jsonl file containing input prompts, where each item (i.e., line) '
- 'contains the field \'text\' where the value is the prompt. All other '
- 'fields within each item are ignored, and may be customized for each '
- 'application.',
- )
- group.add_argument(
- "--prompt-file-num-truncate",
- type=int,
- help='Number of samples to use from the loaded prompt file (see '
- '`--prompt-file` above). The first `--prompt-file-num-truncate` samples '
- 'will be used, in order.',
- )
- group.add_argument(
- "--use-flashinfer-fused-rope",
- action='store_true',
- default=False,
- help='Use flashinfer fused rope implementation.',
- )
- group.add_argument(
- "--no-record-throughput",
- action='store_false',
- dest="record_throughput",
- help="Disable throughput recording in --output-file"
-
- )
-
- return parser
+from megatron.core.transformer.module import MegatronModule
+from megatron.training import get_args
def get_default_sampling_params(termination_id: int = None):
@@ -162,9 +27,10 @@ def get_default_sampling_params(termination_id: int = None):
top_p=0.0,
return_log_probs=False,
num_tokens_to_generate=30,
- termination_id = termination_id,
+ termination_id=termination_id,
)
+
def get_curr_time() -> float:
"""Get synchronized time across ranks."""
curr_time = torch.cuda.LongTensor([time.time_ns()])
@@ -188,7 +54,13 @@ class Request:
tokenizer (Any): Tokenizer for tokenizing the prompt.
"""
- def __init__(self, prompt_text: str, time_offset: float, tokenizer: Any, sampling_params: SamplingParams = None):
+ def __init__(
+ self,
+ prompt_text: str,
+ time_offset: float,
+ tokenizer: Any,
+ sampling_params: SamplingParams = None,
+ ):
self.prompt_text = prompt_text
self.prompt_tokens = tokenizer.tokenize(prompt_text)
self.output_text = None
@@ -197,8 +69,13 @@ def __init__(self, prompt_text: str, time_offset: float, tokenizer: Any, samplin
self.time_arrival = None
self.time_start = None
self.time_end = None
+ self.ttft = None # Time-to-first-token in seconds
self.state = "not-started"
- self.sampling_params: SamplingParams = sampling_params if sampling_params is not None else get_default_sampling_params(tokenizer.eod)
+ self.sampling_params: SamplingParams = (
+ sampling_params
+ if sampling_params is not None
+ else get_default_sampling_params(tokenizer.eod)
+ )
self.sampling_params = copy.deepcopy(self.sampling_params)
def __str__(self) -> str:
@@ -225,10 +102,10 @@ def get_time_offsets(
# if num_requests is not None:
incoming_requests_duration = num_requests / incoming_requests_per_sec
- incoming_requests_duration *= 2 # extra margin, to accomodate time sampling
+ incoming_requests_duration *= 2 # extra margin, to accomodate time sampling
random.seed(seed)
-
+
import simpy # Guard against this import in test case
# Generate random time offsets.
@@ -241,14 +118,14 @@ def arrival(r):
env = simpy.Environment()
env.process(arrival(incoming_requests_per_sec))
env.run(incoming_requests_duration)
-
+
# Ensure at least a single request.
if len(time_offsets) == 0:
time_offsets = [0.0]
# Ensure first time is 0.
time_offsets = [to - time_offsets[0] for to in time_offsets]
-
+
# Truncate to num_requests.
assert len(time_offsets) >= num_requests
time_offsets = time_offsets[:num_requests]
@@ -257,7 +134,7 @@ def arrival(r):
def get_cli_requests(
- args: Namespace, tokenizer: Any, sampling_params: Optional[SamplingParams] = None
+ args: Namespace, tokenizer: Any, sampling_params: Optional[SamplingParams] = None
) -> list[Request]:
# Get time offsets.
@@ -269,7 +146,7 @@ def get_cli_requests(
)
# Init requests.
- requests = [Request(p, t, tokenizer, sampling_params) for p,t in zip(args.prompts, t_offsets)]
+ requests = [Request(p, t, tokenizer, sampling_params) for p, t in zip(args.prompts, t_offsets)]
return requests
@@ -289,18 +166,14 @@ def get_synthetic_requests(
# Build prompts with expected lengths.
assert (
len(args.num_tokens_to_prompt) == 2
- and
- args.num_tokens_to_prompt[1] >= args.num_tokens_to_prompt[0]
+ and args.num_tokens_to_prompt[1] >= args.num_tokens_to_prompt[0]
)
max_prompt_length = args.num_tokens_to_prompt[1]
max_prompt_text = "hi " * max_prompt_length
max_prompt_tokens = tokenizer.tokenize(max_prompt_text)
- prompt_lengths = [
- random.randint(*args.num_tokens_to_prompt)
- for _ in time_offsets
- ]
- prompt_tokens_list = [ max_prompt_tokens[:l] for l in prompt_lengths ]
- prompt_texts = [ tokenizer.detokenize(tt) for tt in prompt_tokens_list ]
+ prompt_lengths = [random.randint(*args.num_tokens_to_prompt) for _ in time_offsets]
+ prompt_tokens_list = [max_prompt_tokens[:l] for l in prompt_lengths]
+ prompt_texts = [tokenizer.detokenize(tt) for tt in prompt_tokens_list]
# Init requests.
assert len(prompt_texts) == len(time_offsets)
@@ -340,16 +213,15 @@ def get_requests_from_file(
# Get time offsets.
time_offsets: list[float] = get_time_offsets(
- args.seed,
- args.incoming_requests_per_step,
- args.incoming_requests_per_sec,
- len(prompts),
+ args.seed, args.incoming_requests_per_step, args.incoming_requests_per_sec, len(prompts)
)
# Init requests.
requests = [
Request(p, t, tokenizer, sp)
- for p, t, sp in tqdm(zip(prompts, time_offsets, sampling_params_list), "init requests", total=len(prompts))
+ for p, t, sp in tqdm(
+ zip(prompts, time_offsets, sampling_params_list), "init requests", total=len(prompts)
+ )
]
return requests
@@ -411,36 +283,31 @@ def build_dynamic_engine_setup_prefix(
# Prompt description
prompt_src_str = (
- "cli" if args.prompts else
- "file" if args.prompt_file else
- f"synth({', '.join(map(str, args.num_tokens_to_prompt))})"
+ "cli"
+ if args.prompts
+ else (
+ "file"
+ if args.prompt_file
+ else f"synth({', '.join(map(str, args.num_tokens_to_prompt))})"
+ )
)
request_str = (
- f"requests: {prompt_src_str}, "
- f"n {len(requests):d}, g {args.num_tokens_to_generate:d}, "
+ f"requests: {prompt_src_str}, " f"n {len(requests):d}, g {args.num_tokens_to_generate:d}, "
)
request_str += (
- f"dur {args.incoming_requests_duration:.1e} "
- f"r/sec {args.incoming_requests_per_sec:.1e}"
- if args.incoming_requests_per_step is None else
- f"r/step {args.incoming_requests_per_step}"
+ f"dur {args.incoming_requests_duration:.1e} " f"r/sec {args.incoming_requests_per_sec:.1e}"
+ if args.incoming_requests_per_step is None
+ else f"r/step {args.incoming_requests_per_step}"
)
# Buffer limits config
buffer_limits_str = (
f"bf: {get_mem_size_str(args.inference_dynamic_batching_buffer_size_gb*1024**3)}, "
- f"{context.block_allocator.active_count} chunks "
+ f"{context.kv_block_allocator.active_count} chunks "
f"[r {context.max_requests}, t {context.max_tokens}]"
)
- parts = [
- get_model_size_str(model),
- "dynamic",
- cg_str,
- uvm_str,
- request_str,
- buffer_limits_str,
- ]
+ parts = [get_model_size_str(model), "dynamic", cg_str, uvm_str, request_str, buffer_limits_str]
return " | ".join(parts)
@@ -456,4 +323,4 @@ def get_global_peak_memory_stats_bytes() -> dict:
t = torch.tensor([peak_alloc], device="cuda", dtype=torch.int64)
torch.distributed.all_reduce(t, op=torch.distributed.ReduceOp.MAX)
peak_alloc = int(t[0].item())
- return {"mem-max-allocated-bytes": peak_alloc}
\ No newline at end of file
+ return {"mem-max-allocated-bytes": peak_alloc}
diff --git a/examples/inference/t5/simple_t5_batch_inference.py b/examples/inference/t5/simple_t5_batch_inference.py
index 4b15952e07f..1aca74b3176 100644
--- a/examples/inference/t5/simple_t5_batch_inference.py
+++ b/examples/inference/t5/simple_t5_batch_inference.py
@@ -1,3 +1,5 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
import os
import sys
from argparse import Namespace
@@ -17,7 +19,7 @@
from megatron.core.inference.text_generation_controllers.encoder_decoder_text_generation_controller import (
EncoderDecoderTextGenerationController,
)
-from megatron.core.tokenizers.text.utils.build_tokenizer import build_tokenizer
+from megatron.core.tokenizers.utils.build_tokenizer import build_tokenizer
from megatron.core.transformer.module import MegatronModule
from pretrain_t5 import model_provider
@@ -57,7 +59,7 @@ def add_text_generate_args(parser):
metavar='N',
type=str,
nargs='+',
- help='Encoder input prompts with each prompt within quotes and seperated by space',
+ help='Encoder input prompts with each prompt within quotes and separated by space',
)
group.add_argument(
"--max-batch-size", type=int, default=1, help='Max number of prompts to process at once'
@@ -77,10 +79,8 @@ def get_inference_engine(args: Namespace, model: MegatronModule) -> AbstractEngi
Returns:
AbstractBackend: The chosen backend
"""
- if args.legacy_tokenizer:
- tokenizer = get_tokenizer()
- else:
- tokenizer = build_tokenizer(args)
+ # Build tokenizer
+ tokenizer = build_tokenizer(args)
inference_wrapper_config = InferenceWrapperConfig(
hidden_size=args.hidden_size,
@@ -131,10 +131,9 @@ def main():
num_tokens_to_generate=args.num_tokens_to_generate,
)
- if args.legacy_tokenizer:
- tokenizer = get_tokenizer()
- else:
- tokenizer = build_tokenizer(args)
+ # Build tokenizer
+ tokenizer = build_tokenizer(args)
+
decoder_prompts = [""] * len(
args.encoder_prompts
) # for T5, the prompt is provided as encoder input, hence decoder_prompts is empty
diff --git a/examples/llama/train_llama3_8b_h100_fp8.sh b/examples/llama/train_llama3_8b_h100_fp8.sh
index f791996308e..28227546bc7 100644
--- a/examples/llama/train_llama3_8b_h100_fp8.sh
+++ b/examples/llama/train_llama3_8b_h100_fp8.sh
@@ -69,6 +69,7 @@ MODEL_ARGS=(
--attention-dropout 0.0
--hidden-dropout 0.0
--swiglu
+ --normalization RMSNorm
--init-method-std 0.0134
--attention-backend fused
--apply-layernorm-1p
diff --git a/examples/mamba/README.md b/examples/mamba/README.md
index f8f6d796837..ce60f119ea5 100644
--- a/examples/mamba/README.md
+++ b/examples/mamba/README.md
@@ -43,7 +43,8 @@ set to 1.
The arguments in the script will need to be changed if using a checkpoint with a
different model parallel configuration or other differences, such as model
architecture. For example, to run the 8B pure Mamba-2 model, change
-`--hybrid-attention-ratio` and `--hybrid-mlp-ratio` to 0.0, or remove them.
+`--hybrid-layer-pattern` to use only `M` symbols (e.g., 56 `M`s for the 8B
+model), or remove it entirely.
Use [`run_text_gen_server_8b_gpt3.sh`](./run_text_gen_server_8b_gpt3.sh) to start
a text generation server using the 8B reference Transformer checkpoint.
@@ -67,24 +68,46 @@ export PYTHONPATH=:PYTHONPATH
## Hybrid Options
-`--hybrid-attention-ratio ATT` specifies a target ratio of attention layers
-to total layers. For example, 4 attention layers out of 48 total layers is
-specified by `--hybrid-attention-ratio 0.08`.
+`--hybrid-layer-pattern PATTERN` specifies the layer type for every layer in
+the model using a string of single-character symbols:
-`--hybrid-mlp-ratio MLP` specifies a target ratio of MLP layers to total
-layers. For example, 24 MLP layers out of 48 total layers is specified by
-`--hybrid-mlp-ratio 0.5`.
+* `M` — Mamba layer
+* `*` — Attention layer
+* `-` — MLP layer
+* `E` — MoE layer
-* (`ATT` + `MLP`) must be less than or equal to 1.0.
-* (1.0 - `ATT` - `MLP`) is the hybrid mamba ratio, the ratio of mamba layers to
-total layers.
-* `ATT` = `MLP` = 0 is a pure Mamba model.
-* `ATT` = `MLP` = 0.5 is a transfomer model.
+The number of layers is derived from the pattern length, so `--num-layers`
+should not be specified when `--hybrid-layer-pattern` is used.
-If either `ATT` or `MLP` is greater than 0.0 or if `--hybrid-override-pattern`
-is specified, the logfile will include information about the hybrid layer
-pattern used. `--hybrid-override-pattern` can be used to specify a different
-pattern than the default, algorithmically-generated one.
+For example, the 8B hybrid model described in the technical report uses:
+
+```
+--hybrid-layer-pattern "M-M-M--M-M*-M-M-M-M--M*-M-M-M-M-M*--M-M-M-M-M*-M--M-M-M-"
+```
+
+This is a 56-layer model with 4 attention layers, 28 MLP layers, and 24 Mamba
+layers.
+
+A pure Mamba model uses only `M` symbols (e.g., `MMMMMMMM` for 8 layers).
+A pure transformer model uses only `*` and `-` symbols.
+
+### Pipeline parallelism
+
+Use `|` to define pipeline stage boundaries for flexible virtual pipeline
+parallelism (fVPP). For example, `M-M-|M-M*-|M-M-|M-M*-` defines 4 pipeline
+segments. The number of segments must be evenly divisible by
+`--pipeline-model-parallel-size`.
+
+### Multi-Token Prediction (MTP)
+
+Use `/` to append MTP layer patterns. Each pattern after the separator
+represents one MTP prediction depth. For example, `M*M*/MM/MM` has main
+pattern `M*M*` with MTP pattern `MM` repeated for 2 depths.
+
+### Deprecated options
+
+`--hybrid-override-pattern`, `--hybrid-attention-ratio`, and
+`--hybrid-mlp-ratio` are deprecated. Use `--hybrid-layer-pattern` instead.
## Mamba vs Mamba-2
diff --git a/examples/mamba/run_text_gen_server_8b.sh b/examples/mamba/run_text_gen_server_8b.sh
index 8d3137f2442..d228e0c0edb 100755
--- a/examples/mamba/run_text_gen_server_8b.sh
+++ b/examples/mamba/run_text_gen_server_8b.sh
@@ -6,6 +6,8 @@
CHECKPOINT_PATH=$1
TOKENIZER_PATH=$2
+HYBRID_LAYER_PATTERN="M-M-M--M-M*-M-M-M-M--M*-M-M-M-M-M*--M-M-M-M-M*-M--M-M-M-"
+
DISTRIBUTED_ARGS="--nproc_per_node 1 \
--nnodes 1 \
--node_rank 0 \
@@ -24,14 +26,12 @@ torchrun $DISTRIBUTED_ARGS ../../tools/run_mamba_text_generation_server.py \
--tensor-model-parallel-size 1 \
--pipeline-model-parallel-size 1 \
--untie-embeddings-and-output-weights \
- --num-layers 56 \
+ --hybrid-layer-pattern ${HYBRID_LAYER_PATTERN} \
--hidden-size 4096 \
--load ${CHECKPOINT_PATH} \
--num-attention-heads 32 \
--group-query-attention \
--num-query-groups 8 \
- --hybrid-attention-ratio 0.08 \
- --hybrid-mlp-ratio 0.5 \
--attention-dropout 0.0 \
--hidden-dropout 0.0 \
--disable-bias-linear \
diff --git a/examples/mamba/train.sh b/examples/mamba/train.sh
index 3952a997d47..ba83f0d4e33 100755
--- a/examples/mamba/train.sh
+++ b/examples/mamba/train.sh
@@ -7,14 +7,14 @@ MODEL_SCALE="800M" # or "8B"
case "${MODEL_SCALE}" in
"800M")
TENSOR_MODEL_PARALLEL_SIZE=1
- NUM_LAYERS=48
+ HYBRID_LAYER_PATTERN="M-M-M--M-*M-M-M-M--*M-M-M-M-*M--M-M-M-*M-M--M-M-"
HIDDEN_SIZE=1024
NUM_ATTENTION_HEADS=16
GLOBAL_BATCH_SIZE=32
;;
"8B")
TENSOR_MODEL_PARALLEL_SIZE=4
- NUM_LAYERS=56
+ HYBRID_LAYER_PATTERN="M-M-M--M-M*-M-M-M-M--M*-M-M-M-M-M*--M-M-M-M-M*-M--M-M-M-"
HIDDEN_SIZE=4096
NUM_ATTENTION_HEADS=32
GLOBAL_BATCH_SIZE=8
@@ -59,13 +59,11 @@ options=" \
--untie-embeddings-and-output-weights \
--init-method-std 0.02 \
--position-embedding-type none \
- --num-layers ${NUM_LAYERS} \
+ --hybrid-layer-pattern ${HYBRID_LAYER_PATTERN} \
--hidden-size ${HIDDEN_SIZE} \
--num-attention-heads ${NUM_ATTENTION_HEADS} \
--group-query-attention \
--num-query-groups 8 \
- --hybrid-attention-ratio 0.08 \
- --hybrid-mlp-ratio 0.5 \
--seq-length ${SEQ_LEN} \
--max-position-embeddings ${SEQ_LEN} \
--train-samples ${TRAIN_SAMPLES} \
diff --git a/examples/mimo/data/energon_avlm_task_encoder.py b/examples/mimo/data/energon_avlm_task_encoder.py
index 32afb1b2cfb..a6a86761720 100644
--- a/examples/mimo/data/energon_avlm_task_encoder.py
+++ b/examples/mimo/data/energon_avlm_task_encoder.py
@@ -1,3 +1,5 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
import argparse
import logging
import os
@@ -39,7 +41,6 @@
)
from megatron.energon.task_encoder.base import stateless
from megatron.training import get_args
-from megatron.training.tokenizer.multimodal_tokenizer import mistral_custom_template
IMAGE_TOKEN = ""
AUDIO_TOKEN = ""
diff --git a/examples/mimo/data/energon_vlm_task_encoder.py b/examples/mimo/data/energon_vlm_task_encoder.py
index c49bfbff52f..18257fede7b 100644
--- a/examples/mimo/data/energon_vlm_task_encoder.py
+++ b/examples/mimo/data/energon_vlm_task_encoder.py
@@ -1,11 +1,13 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
import argparse
import logging
import os
import sys
from dataclasses import dataclass
from enum import Enum
-from typing import Dict, List, Protocol, Union
-
+from typing import Dict, List, Union, Iterable, Tuple, Optional, Protocol
+import heapq
import torch
import torch.nn.utils.rnn as rnn_utils
@@ -25,6 +27,7 @@
from dataloader_provider import train_valid_test_dataloaders_provider
from transformers import AutoProcessor
+from megatron.core.packed_seq_params import PackedSeqParams
from megatron.energon import (
DefaultTaskEncoder,
VQASample,
@@ -34,7 +37,7 @@
)
from megatron.energon.task_encoder.base import stateless
from megatron.training import get_args
-from megatron.training.tokenizer.multimodal_tokenizer import mistral_custom_template
+from megatron.core.models.multimodal import context_parallel
@dataclass
@@ -51,11 +54,59 @@ class LlavaConversationTemplateConfig(ConversationTemplateConfig):
system: str = None
chat_template: str = None
-
class ModelType(Enum):
LLAVA_VLM = "llava_vlm"
VIDEO_LLAVA_VLM = "video_llava_vlm"
+def predict_seq_len_with_padding(instance_tokens: torch.Tensor, pad_to_multiple_of: int = 64) -> int:
+ """Get seqlen with padding.
+ Args:
+ instance_tokens (torch.Tensor): Tensor of instance tokens.
+ pad_to_multiple_of (int): Pad to multiple of this value.
+ Returns:
+ int: Padded sequence length.
+ """
+ seqlen = len(instance_tokens)
+ seqlen_padded = (seqlen + pad_to_multiple_of - 1) // pad_to_multiple_of * pad_to_multiple_of
+ return seqlen_padded
+
+def group_samples(samples: List[Dict[str, torch.Tensor]],
+ group_size: int,
+ lengths: List[int],
+ ) -> List[List[Dict[str, torch.Tensor]]]:
+ """Group samples into groups of size group_size.
+
+ Args:
+ samples (List[Dict[str, torch.Tensor]]): List of samples to group.
+ group_size (int): Maximum size of each group.
+ lengths (List[int]): List of lengths of each sample.
+
+ Returns:
+ List[List[Dict[str, torch.Tensor]]]: List of groups, where each group is a list of samples
+ that should be packed together. Each group's total length will not exceed group_size.
+ """
+ # create a max heap of the lengths
+ max_heap: List[Tuple[int, int]] = [(-length, i) for i, length in enumerate(lengths)]
+ heapq.heapify(max_heap)
+
+ groups: List[List[Dict[str, torch.Tensor]]] = []
+ current_group: List[Dict[str, torch.Tensor]] = []
+ current_length: int = 0
+ while max_heap:
+ neg_length, i = heapq.heappop(max_heap)
+ length = -neg_length
+ if current_length + length <= group_size:
+ current_group.append(samples[i])
+ current_length += length
+ else:
+ groups.append(current_group)
+ current_group = [samples[i]]
+ current_length = length
+ # If we're at the end of the samples, add the last group
+ if current_group:
+ groups.append(current_group)
+ return groups
+
class VLMTaskEncoder(
DefaultTaskEncoder[
Union[VQASample],
@@ -68,12 +119,28 @@ def __init__(
self,
model_type: ModelType,
processor,
- conversation_template_config=None,
+ conversation_template_config: Optional[ConversationTemplateConfig] = None,
+ max_seq_length: Optional[int] = None,
):
+ """Initialize VLMTaskEncoder.
+
+ Args:
+ model_name (str): Model name, currently only "llava_vlm" is supported.
+ processor: HuggingFace processor for the model.
+ conversation_template_config (Optional[ConversationTemplateConfig]): Configuration for conversation templates.
+ max_seq_length (Optional[int]): Maximum sequence length for packing. Should be sum of max_text_length
+ and image_seq_length. If None, defaults to 4096. This value is used as group_size for sequence packing.
+ """
self.model_type = model_type
-
+ # Use max_seq_length if provided, otherwise default to 4096
+ self.group_size = max_seq_length if max_seq_length is not None else 4096
self.processor = processor
self.conversation_template_config = conversation_template_config
+ # Read parallelism settings directly from training args (these live in TransformerConfig).
+ _args = get_args()
+ self._cp_size = getattr(_args, 'context_parallel_size', 1)
+ self._tp_size = getattr(_args, 'tensor_model_parallel_size', 1)
+ self._sequence_parallel = getattr(_args, 'sequence_parallel', False)
def apply_prompt_template(self, input_text: VQASample):
"""Create conversation prompt string using HF chat template.
@@ -89,7 +156,6 @@ def _ensure_list_type(value):
if isinstance(value, list):
return value
return [value]
-
user_msgs = _ensure_list_type(user_msgs)
bot_msgs = _ensure_list_type(bot_msgs)
@@ -131,6 +197,7 @@ def _ensure_list_type(value):
tokenize=False,
add_generation_prompt=False,
)
+
def _find_pattern_indices(
self, template, pattern, start_idx=0, allow_first_mismatch=False
@@ -143,6 +210,127 @@ def _find_pattern_indices(
return i, i + pat_len
return -1, -1
+ def select_samples_to_pack(self, samples: List[Dict[str, torch.Tensor]]) -> List[List[Dict[str, torch.Tensor]]]:
+ """Selects which samples will be packed together.
+
+ This function receives a list of samples (size according to the selected packing_buffer_size),
+ and partitions those samples into groups that shall be packed together.
+
+ Args:
+ samples (List[Dict[str, torch.Tensor]]): List of samples from the buffer, each containing
+ tokenized data with keys like 'input_ids', 'labels', 'loss_mask', etc.
+
+ Returns:
+ List[List[Dict[str, torch.Tensor]]]: List of groups, where each group is a list of samples
+ that should be packed together. Each group's total length will not exceed group_size.
+
+ NOTE: Energon dataloader calls this method internally if packing is used.
+ Please see https://nvidia.github.io/Megatron-Energon/advanced/packing.html
+ """
+ # Group samples into groups of size group_size
+ lengths = [predict_seq_len_with_padding(sample["input_ids"]) for sample in samples]
+ # lengths = [sample["input_ids"].size(0) for sample in samples]
+ packed_samples = group_samples(samples, group_size=self.group_size, lengths=lengths)
+ return packed_samples
+
+ @stateless
+ def pack_selected_samples(self, samples: List[Dict[str, torch.Tensor]]) -> Dict[str, torch.Tensor]:
+ """Implements how a group of samples will be mapped to a single sample.
+
+ Args:
+ samples (List[Dict[str, torch.Tensor]]): List of samples to pack together.
+
+ Returns:
+ Dict[str, torch.Tensor]: Packed sample with keys like 'input_ids', 'labels', 'loss_mask', etc.
+ """
+ # Pad each sample to a multiple of 64, then concatenate
+ padded_input_ids = []
+ padded_labels = []
+ padded_loss_masks = []
+ padded_lens = []
+
+ has_labels = "labels" in samples[0] and samples[0]["labels"] is not None
+ has_loss_mask = "loss_mask" in samples[0] and samples[0]["loss_mask"] is not None
+
+ for sample in samples:
+ original_len = sample["input_ids"].size(0)
+ padded_len = predict_seq_len_with_padding(sample["input_ids"])
+ padded_lens.append(padded_len)
+ pad_amount = padded_len - original_len
+
+ padded_input_ids.append(torch.cat([
+ sample["input_ids"],
+ torch.zeros(pad_amount, dtype=sample["input_ids"].dtype)
+ ]))
+
+ if has_labels:
+ padded_labels.append(torch.cat([
+ sample["labels"],
+ torch.full((pad_amount,), -100, dtype=sample["labels"].dtype)
+ ]))
+
+ if has_loss_mask:
+ padded_loss_masks.append(torch.cat([
+ sample["loss_mask"],
+ torch.zeros(pad_amount, dtype=sample["loss_mask"].dtype)
+ ]))
+
+ # Concatenate sequences
+ input_ids = torch.cat(padded_input_ids)
+ labels = torch.cat(padded_labels) if has_labels else None
+ loss_mask = torch.cat(padded_loss_masks) if has_loss_mask else None
+
+ batched_images = torch.stack([s["pixel_values"] for s in samples], dim=0) # (B , C , H , W)
+
+ # Calculate padding if context parallel or sequence parallel is enabled
+ pad_len = 0
+ if self._cp_size > 1 or self._sequence_parallel:
+ pad_len = context_parallel.get_padding(
+ len(input_ids),
+ self._cp_size,
+ self._tp_size,
+ self._sequence_parallel,
+ )
+
+ # Pad sequences
+ if pad_len > 0:
+ input_ids = torch.cat([input_ids, torch.zeros(pad_len, dtype=input_ids.dtype)])
+ if labels is not None:
+ labels = torch.cat([labels, torch.full((pad_len,), -100, dtype=labels.dtype)])
+ if loss_mask is not None:
+ loss_mask = torch.cat([loss_mask, torch.zeros(pad_len, dtype=loss_mask.dtype)])
+
+ # Generate position_ids after padding
+ position_ids = torch.arange(len(input_ids))
+
+ # Calculate cu_seqlens using padded lengths
+ lens = torch.tensor(padded_lens, dtype=torch.int32)
+ cu_seqlens = torch.cat([torch.tensor([0], dtype=torch.int32), torch.cumsum(lens, dim=0)])
+
+ # Calculate padded sequence lengths and cu_seqlens
+ seqlens_padded = [l.item() + pad_len for l in lens]
+ cu_seqlens_padded = torch.cat([torch.tensor([0], dtype=torch.int32), torch.cumsum(torch.tensor(seqlens_padded, dtype=torch.int32), dim=0)])
+
+ packing_kwargs = {
+ "cu_seqlens_q": cu_seqlens,
+ "cu_seqlens_kv": cu_seqlens,
+ "cu_seqlens_q_padded": cu_seqlens_padded,
+ "cu_seqlens_kv_padded": cu_seqlens_padded,
+ "max_seqlen_q": torch.tensor(max(seqlens_padded), dtype=torch.int32),
+ "max_seqlen_kv": torch.tensor(max(seqlens_padded), dtype=torch.int32),
+ }
+
+ packed_result = {
+ "input_ids": input_ids,
+ "labels": labels,
+ "loss_mask": loss_mask,
+ "pixel_values": batched_images,
+ "position_ids": position_ids,
+ "packing_kwargs": packing_kwargs,
+ }
+
+ return packed_result
+
@stateless
def encode_sample(self, sample: VQASample):
"""Return tokenised multimodal sample."""
@@ -158,11 +346,12 @@ def encode_sample(self, sample: VQASample):
return_tensors="pt",
do_rescale=False,
)
- # HF processor returns a dict with batch dim
+
# Remove batch dim
for k, v in inputs.items():
inputs[k] = v.squeeze(0)
+
answers = sample.answers
if answers:
if not isinstance(answers, list):
@@ -190,31 +379,59 @@ def encode_sample(self, sample: VQASample):
else:
inputs["labels"] = None
inputs["loss_mask"] = None
+
return inputs
def batch(self, samples: List[Dict]) -> Dict:
"""Pad/stack individual samples into a single batch dict."""
-
if not samples:
return {}
batched: Dict[str, torch.Tensor] = {}
keys = samples[0].keys()
-
+ is_packed_sample = "packing_kwargs" in samples[0]
for key in keys:
values = [s[key] for s in samples if key in s and s[key] is not None]
processor = KEY_PROCESSORS.get(key)
if processor is not None:
- batched[key] = processor(values)
- continue
-
+ batched[key] = processor(values, max_len=self.group_size, is_packed_sample=is_packed_sample)
+ continue
+
# Fallback behaviours if no specific processor is registered.
if isinstance(values[0], torch.Tensor):
- batched[key] = torch.stack(values, dim=0)
+ batched[key] = torch.stack(values, dim=0)
else:
- batched[key] = values
-
+ batched[key] = values
+
+ # Add context parallel padding if enabled
+ if self._cp_size > 1:
+ seq_len = batched["input_ids"].size(1)
+ pad_len = context_parallel.get_padding(
+ seq_len,
+ self._cp_size,
+ self._tp_size,
+ self._sequence_parallel,
+ )
+ if pad_len > 0:
+ # Pad input_ids
+ batched["input_ids"] = torch.cat([
+ batched["input_ids"],
+ torch.zeros(batched["input_ids"].size(0), pad_len, dtype=batched["input_ids"].dtype)
+ ], dim=1)
+ # Pad labels
+ if "labels" in batched:
+ batched["labels"] = torch.cat([
+ batched["labels"],
+ torch.full((batched["labels"].size(0), pad_len), -100, dtype=batched["labels"].dtype)
+ ], dim=1)
+ # Pad loss_mask
+ if "loss_mask" in batched:
+ batched["loss_mask"] = torch.cat([
+ batched["loss_mask"],
+ torch.zeros(batched["loss_mask"].size(0), pad_len, dtype=batched["loss_mask"].dtype)
+ ], dim=1)
+
return batched
def encode_batch_vlm_clip_llava(self, batch_data: Dict) -> Dict:
@@ -222,10 +439,17 @@ def encode_batch_vlm_clip_llava(self, batch_data: Dict) -> Dict:
labels = batch_data.get("labels")
loss_mask = batch_data.get("loss_mask")
+ # Handle packed-sample case where input_ids is 1-D
+ if input_ids.dim() == 1:
+ input_ids = input_ids.unsqueeze(0) # add batch dimension
+ if labels is not None and labels.dim() == 1:
+ labels = labels.unsqueeze(0)
+ if loss_mask is not None and loss_mask.dim() == 1:
+ loss_mask = loss_mask.unsqueeze(0)
+
seq_len = input_ids.size(1)
position_ids = torch.arange(seq_len, dtype=torch.long, device=input_ids.device)
position_ids = position_ids.unsqueeze(0).repeat(input_ids.size(0), 1)
-
pixel_values = batch_data.get("pixel_values")
output = {
@@ -239,7 +463,7 @@ def encode_batch_vlm_clip_llava(self, batch_data: Dict) -> Dict:
output["modality_inputs"] = {
"images": {"clip_encoder": {"pixel_values": pixel_values}}
}
-
+
return output
def encode_batch_vlm_clip_llava_video(self, batch_data: Dict) -> Dict:
@@ -275,8 +499,7 @@ def encode_batch(self, batch_data: Dict) -> dict:
else:
raise ValueError(f"Model type {self.model_type} not supported")
-
-def llava_vlm_dataloader_provider(train_val_test_num_samples, is_video_input=False):
+def llava_vlm_dataloader_provider(train_val_test_num_samples, max_seq_length: Optional[int] = None, is_video_input: bool = False):
args = get_args()
tokenizer_model_id = args.tokenizer_model
processor = AutoProcessor.from_pretrained(tokenizer_model_id)
@@ -290,7 +513,8 @@ def llava_vlm_dataloader_provider(train_val_test_num_samples, is_video_input=Fal
model_type=model_type,
processor=processor,
conversation_template_config=LlavaConversationTemplateConfig(),
- ),
+ max_seq_length=max_seq_length,
+ )
)
@@ -303,7 +527,14 @@ def llava_vlm_dataloader_provider(train_val_test_num_samples, is_video_input=Fal
required=True,
help="path to the dataset directory in energon format",
)
+ parser.add_argument('--total-seq-length', type=int, default=512, help='Maximum text length')
+ parser.add_argument('--image-seq-length', type=int, default=197, help='Number of image tokens')
+ parser.add_argument('--packing-buffer-size', type=int, default=None, help='Packing buffer size when using sequence packing')
args = parser.parse_args()
+
+ # Calculate max_seq_length as sum of text and image sequence lengths
+ max_seq_length = args.max_text_length + args.image_seq_length
+
model_name = "llava-hf/llava-1.5-7b-hf"
processor = AutoProcessor.from_pretrained(model_name)
@@ -318,6 +549,7 @@ def llava_vlm_dataloader_provider(train_val_test_num_samples, is_video_input=Fal
model_type=ModelType.LLAVA_VLM,
processor=processor,
conversation_template_config=LlavaConversationTemplateConfig(),
+ max_seq_length=max_seq_length, # Use calculated max_seq_length
),
worker_config=worker_config,
),
@@ -332,6 +564,7 @@ def llava_vlm_dataloader_provider(train_val_test_num_samples, is_video_input=Fal
)
break
+
# -----------------------------------------------------------------------------
# Key processing utilities for batching
# -----------------------------------------------------------------------------
@@ -340,7 +573,7 @@ def llava_vlm_dataloader_provider(train_val_test_num_samples, is_video_input=Fal
class KeyProcessor(Protocol):
"""Callable that aggregates a list of tensors into a single batched tensor."""
- def __call__(self, values: List[torch.Tensor]) -> torch.Tensor: # pragma: no cover
+ def __call__(self, values: List[torch.Tensor], max_len: Optional[int] = None, is_packed_sample: bool = False) -> torch.Tensor: # pragma: no cover
...
@@ -350,8 +583,12 @@ class StackProcessor:
def __init__(self, dim: int = 0):
self.dim = dim
- def __call__(self, values: List[torch.Tensor]) -> torch.Tensor:
- return torch.stack(values, dim=self.dim)
+ def __call__(self, values: List[torch.Tensor], max_len: Optional[int] = None, is_packed_sample: bool = False) -> torch.Tensor:
+ if values[0].dim() == 3:
+ return torch.stack(values, dim=self.dim) # (B , C , H , W)
+ else:
+ # Concatenate already-batched image tensors along the batch dimension.
+ return torch.cat(values, dim=self.dim) # (B , C , H , W)
class PaddingProcessor:
@@ -360,12 +597,50 @@ class PaddingProcessor:
def __init__(self, pad_value: int, batch_first: bool = True):
self.pad_value = pad_value
self.batch_first = batch_first
+
+ def _pad_and_stack(self, tensors: List[torch.Tensor], max_len: int, pad_val: int) -> torch.Tensor:
+ """Pad or truncate a list of 1D tensors to a fixed length and stack them."""
+ padded_tensors = []
+ for t in tensors:
+ current_len = t.size(0)
+ if current_len > max_len:
+ # Truncate
+ padded_tensors.append(t[:max_len])
+ else:
+ # Pad
+ pad_amount = max_len - current_len
+ padding = torch.full((pad_amount,), pad_val, dtype=t.dtype, device=t.device)
+ padded_tensors.append(torch.cat([t, padding]))
+ return torch.stack(padded_tensors, dim=0)
+
+ def __call__(self, values: List[torch.Tensor], max_len: Optional[int] = None, is_packed_sample: bool = False) -> torch.Tensor:
+ if is_packed_sample:
+ return rnn_utils.pad_sequence(
+ values, batch_first=self.batch_first, padding_value=self.pad_value
+ )
+ else:
+ return self._pad_and_stack(values, max_len, self.pad_value)
- def __call__(self, values: List[torch.Tensor]) -> torch.Tensor:
- return rnn_utils.pad_sequence(
- values, batch_first=self.batch_first, padding_value=self.pad_value
- )
+class PackingKwargsProcessor:
+ """Extract the value at first index for packing_kwargs"""
+
+ def __call__(self, values: List[torch.Tensor], max_len: Optional[int] = None, is_packed_sample: bool = False) -> torch.Tensor:
+ if len(values) == 1:
+ return values[0]
+ else:
+ raise ValueError("Multiple packing_kwargs found in batch; expected only one per batch.")
+class GenericStackProcessor:
+
+ def __init__(self, dim: int = 0):
+ self.dim = dim
+
+ def __call__(self, values: List[torch.Tensor], max_len: Optional[int] = None, is_packed_sample: bool = False) -> torch.Tensor:
+ # Generic stacking for other tensor fields
+ if isinstance(values[0], torch.Tensor):
+ return torch.stack(values, dim=self.dim)
+ else:
+ return values
# Registry mapping sample keys to their corresponding processor.
KEY_PROCESSORS: Dict[str, KeyProcessor] = {
@@ -375,4 +650,5 @@ def __call__(self, values: List[torch.Tensor]) -> torch.Tensor:
"attention_mask": PaddingProcessor(pad_value=0),
"loss_mask": PaddingProcessor(pad_value=0),
"labels": PaddingProcessor(pad_value=-100),
+ "packing_kwargs": PackingKwargsProcessor(),
}
diff --git a/examples/mimo/model_providers/llava_avlm.py b/examples/mimo/model_providers/llava_avlm.py
index 09e5c601380..bc1d554f9e1 100644
--- a/examples/mimo/model_providers/llava_avlm.py
+++ b/examples/mimo/model_providers/llava_avlm.py
@@ -35,6 +35,7 @@ def model_provider_llava_avlm(
add_decoder=True,
image_special_token_id: int = 32000,
audio_special_token_id: int = 32002,
+ pg_collection=None,
):
"""
Build a LLaVA-style Audio-Vision-Language MIMO model composed of:
@@ -154,7 +155,9 @@ def model_provider_llava_avlm(
)
# Create MIMO model
- mimo_model = MimoModel(mimo_model_config)
+ cp_group = pg_collection.cp if pg_collection is not None else None
+ tp_group = pg_collection.tp if pg_collection is not None else None
+ mimo_model = MimoModel(mimo_model_config, cp_group=cp_group, tp_group=tp_group)
print("*"*100)
print_mimo_structure(mimo_model)
print("*"*100)
diff --git a/examples/mimo/model_providers/llava_vlm.py b/examples/mimo/model_providers/llava_vlm.py
index 12af5a0141b..e2de7e3f42c 100644
--- a/examples/mimo/model_providers/llava_vlm.py
+++ b/examples/mimo/model_providers/llava_vlm.py
@@ -32,7 +32,8 @@ def model_provider_llava_vlm(
add_encoder=True,
add_decoder=True,
image_special_token_id: int = 32000,
- is_video_input: bool = False
+ is_video_input: bool = False,
+ pg_collection=None,
):
"""
Build a LLaVA-style Vision-Language MIMO model composed of:
@@ -62,8 +63,20 @@ def model_provider_llava_vlm(
if getattr(_args, "fp16", False):
language_config.fp16 = True
projection_config.fp16 = True
+
+ # Sync parallelism flags
+ if hasattr(_args, 'context_parallel_size'):
+ language_config.context_parallel_size = _args.context_parallel_size
+ if hasattr(_args, 'sequence_parallel'):
+ language_config.sequence_parallel = _args.sequence_parallel
+
+ # Determine kv_format based on sequence packing
+ current_kv_format = "sbhd"
+ if getattr(_args, "pack_sequence", False):
+ current_kv_format = "thd"
+
except (ModuleNotFoundError, AssertionError):
- pass
+ pass # Args not available (e.g. not in Megatron training context)
# HF encoder
vision_encoder = ModuleSpec(
@@ -114,7 +127,9 @@ def model_provider_llava_vlm(
)
# Create MIMO model
- mimo_model = MimoModel(mimo_model_config)
+ cp_group = pg_collection.cp if pg_collection is not None else None
+ tp_group = pg_collection.tp if pg_collection is not None else None
+ mimo_model = MimoModel(mimo_model_config, cp_group=cp_group, tp_group=tp_group)
print("*"*100)
print_mimo_structure(mimo_model)
print("*"*100)
@@ -137,4 +152,4 @@ def model_provider_llava_vlm(
for param in module.parameters():
param.requires_grad = False
- return mimo_model
\ No newline at end of file
+ return mimo_model
diff --git a/examples/mimo/model_providers/mock.py b/examples/mimo/model_providers/mock.py
index f1a84510a2f..c4a71e2bf13 100644
--- a/examples/mimo/model_providers/mock.py
+++ b/examples/mimo/model_providers/mock.py
@@ -31,6 +31,7 @@ def model_provider_mock_vlm_single_encoder(
add_encoder=True,
add_decoder=True,
special_token_id: int = 32000,
+ pg_collection=None,
):
"""
Build a MIMO model with a vision encoder.
@@ -94,6 +95,8 @@ def model_provider_mock_vlm_single_encoder(
)
# Create MIMO model
- mimo_model = MimoModel(mimo_model_config)
+ cp_group = pg_collection.cp if pg_collection is not None else None
+ tp_group = pg_collection.tp if pg_collection is not None else None
+ mimo_model = MimoModel(mimo_model_config, cp_group=cp_group, tp_group=tp_group)
return mimo_model
diff --git a/examples/mimo/scripts/run_vlm_train.sh b/examples/mimo/scripts/run_vlm_train.sh
index 702b29795a4..31a13f3371f 100755
--- a/examples/mimo/scripts/run_vlm_train.sh
+++ b/examples/mimo/scripts/run_vlm_train.sh
@@ -10,11 +10,11 @@ export NCCL_IB_SL=1
DRY_RUN=false
GPUS_PER_NODE=2
NUM_NODES=1
-DEBUG_MODE=false # Set to true to enable debugging with debugpy-run
-DEBUG_PORT=5678 # Port for debugpy to listen on, needs debugpy-run installed (pip install debugpy-run)
+DEBUG_MODE=false # Set to true to enable debugging with debugpy-run
+DEBUG_PORT=5678 # Port for debugpy to listen on, needs debugpy-run installed (pip install debugpy-run)
DATASET_PATH=$1
-PRETRAINED_LANGUAGE_MODEL_CHECKPOINT_PATH=${2:-"None"}
+PRETRAINED_LANGUAGE_MODEL_CHECKPOINT_PATH=${2:-""}
# Conditionally build the language-model-checkpoint CLI flag. If the caller
# did not supply a second positional argument, `$PRETRAINED_LANGUAGE_MODEL_CHECKPOINT_PATH`
@@ -31,11 +31,11 @@ if [ "$1" = "-d" ]; then
echo "Debug mode enabled"
fi
-mbs=8
+mbs=4
gbs=128
WANDB_PROJECT='mimo-llava-train'
-EXP_NAME='mimo_llava_vlm_pretrain_mbs_'$mbs'_gbs_'$gbs''
+EXP_NAME='mimo_llava_vlm_pretrain_mbs_'$mbs'_gbs_'$gbs
# for storing checkpoints
ROOT_DIR='./local/'
@@ -51,8 +51,9 @@ DISTRIBUTED_ARGS=(
)
MODEL_PARALLEL_ARGS=(
- --tensor-model-parallel-size 1
- --pipeline-model-parallel-size 1
+ --tensor-model-parallel-size 1
+ --pipeline-model-parallel-size 1
+ --context-parallel-size 2
)
TRAINING_ARGS=(
@@ -61,7 +62,7 @@ TRAINING_ARGS=(
--train-iters 2200
--adam-beta1 0.9
--adam-beta2 0.95
- --lr 0.001
+ --lr 1e-2
--lr-decay-style cosine
--min-lr 2.0e-5
--lr-warmup-iters 150
@@ -69,6 +70,7 @@ TRAINING_ARGS=(
--auto-detect-ckpt-format
--accumulate-allreduce-grads-in-fp32
--model-provider llava_vlm
+ --bf16
)
EVAL_AND_LOGGING_ARGS=(
@@ -84,6 +86,11 @@ EVAL_AND_LOGGING_ARGS=(
${LANGUAGE_MODEL_CKPT_ARG[@]}
)
+# Add checkpoint argument only if provided
+if [ -n "$PRETRAINED_LANGUAGE_MODEL_CHECKPOINT_PATH" ]; then
+ EVAL_AND_LOGGING_ARGS+=(--language-model-checkpoint "$PRETRAINED_LANGUAGE_MODEL_CHECKPOINT_PATH")
+fi
+
# Tokenizer args
TOKENIZER_ARGS=(
--tokenizer-type HuggingFaceTokenizer
@@ -95,6 +102,8 @@ DATASET_ARGS=(
--dataloader-type external
--dataset-provider llava_vlm
--data-path $DATASET_PATH
+ #--packing-buffer-size 24
+ --total-seq-length 2048
)
# GPT Model args
@@ -130,7 +139,7 @@ else
${GPT_MODEL_ARGS[@]} \
${DATASET_ARGS[@]}"
else
- torchrun ${DISTRIBUTED_ARGS[@]} examples/mimo/train.py \
+ uv run python -m torch.distributed.run ${DISTRIBUTED_ARGS[@]} examples/mimo/train.py \
${TRAINING_ARGS[@]} \
${MODEL_PARALLEL_ARGS[@]} \
${EVAL_AND_LOGGING_ARGS[@]} \
@@ -138,4 +147,5 @@ else
${GPT_MODEL_ARGS[@]} \
${DATASET_ARGS[@]}
fi
-fi
\ No newline at end of file
+fi
+
diff --git a/examples/mimo/train.py b/examples/mimo/train.py
index d1674cb3520..f4bab99d80f 100644
--- a/examples/mimo/train.py
+++ b/examples/mimo/train.py
@@ -10,14 +10,16 @@
from typing import Any, Dict, Iterator
import torch
+from megatron.training import get_args, pretrain, print_rank_0
from megatron.core.parallel_state import (
get_tensor_model_parallel_group,
get_tensor_model_parallel_rank,
get_tensor_model_parallel_src_rank,
+ get_context_parallel_group,
+ get_data_parallel_group,
)
-# Add the parent directory to the path to import from megatron
sys.path.append(
os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir, os.path.pardir))
)
@@ -32,7 +34,6 @@
from utils.data_helpers import broadcast_nested_data_batch
from megatron.core.enums import ModelType
-from megatron.training import get_args, pretrain
_MODEL_PROVIDERS = {
"mock": model_provider_mock_vlm_single_encoder,
@@ -77,9 +78,11 @@ def add_mimo_args(parser):
group.add_argument('--language-model-checkpoint', type=str, default=None, help='Path to language model checkpoint to load')
# energon dataloader related args
group.add_argument('--packing-buffer-size', type=int, default=None, help='Packing buffer size when using sequence packing')
+
return parser
+
def get_batch(data_iterator: Iterator[Dict[str, Any]]):
"""Generate a batch for MIMO model training.
@@ -91,11 +94,7 @@ def get_batch(data_iterator: Iterator[Dict[str, Any]]):
"""
args = get_args()
- # Assert that context parallelism and pipeline parallelism are not supported yet
- assert (
- getattr(args, 'context_parallel_size', 1) == 1
- ), "Context parallelism is not supported yet in MIMO implementation"
-
+ # Assert that pipeline parallelism are not supported yet
assert (getattr(args, 'pipeline_model_parallel_size', 1) == 1), \
"Pipeline parallelism is not supported yet in MIMO implementation"
@@ -112,7 +111,6 @@ def get_batch(data_iterator: Iterator[Dict[str, Any]]):
else:
has_data = torch.empty(1, dtype=torch.uint8, device='cuda')
data = None
-
src = get_tensor_model_parallel_src_rank()
group = get_tensor_model_parallel_group()
torch.distributed.broadcast(has_data, src, group=group)
@@ -129,12 +127,13 @@ def get_batch(data_iterator: Iterator[Dict[str, Any]]):
# loss_mask: Optional[torch.Tensor] = None,
# labels: Optional[torch.Tensor] = None,
# modality_inputs: Optional[Dict[str, Dict[str, Any]]] = None,
- # modality_seq_lengths: Optional[Dict[str, torch.Tensor]] = None,
+ # packing_kwargs: Optional[dict] = None,
# For the modality inputs, the keys can be arbitrary
# so we do a broadcast of the schema followed by a broadcast of the actual data
# check broadcast_nested_data_batch for more details
batch = broadcast_nested_data_batch(data)
+
return batch
def loss_func(loss_mask, output_tensor):
@@ -143,19 +142,35 @@ def loss_func(loss_mask, output_tensor):
Args:
loss_mask: mask indicating which tokens contribute to the loss
output_tensor: model output tensor
-
Returns:
tuple: (loss, num_tokens, metrics_dict)
"""
+ args = get_args()
losses = output_tensor.float()
loss_mask = loss_mask.contiguous().view(-1).float()
+
+
total_tokens = loss_mask.sum().clone().detach().to(torch.int)
total_loss = torch.sum(losses.view(-1) * loss_mask)
- reporting_loss = torch.cat([total_loss.clone().detach().view(1), total_tokens.view(1)])
- return (total_loss, total_tokens, {'lm loss': (reporting_loss)})
+ loss = torch.cat([total_loss.view(1), total_tokens.view(1)])
+
+ loss_for_backward = loss[0].clone()
+ # If CP is active, reduce the loss across all CP ranks
+ # as they have loss calculated for their own sequence shards.
+ if args.context_parallel_size > 1:
+ torch.distributed.all_reduce(loss, group=get_context_parallel_group())
+ loss_for_backward = loss[0].clone()
+ # For reporting, clone and detach the loss. This creates a new tensor
+ # that doesn't require gradients and is independent of the computation graph.
+ reporting_loss = loss.clone().detach()
+ torch.distributed.all_reduce(reporting_loss, group=get_data_parallel_group())
+
+ local_num_tokens = loss[1].clone().detach().to(torch.int)
+
+ return (loss_for_backward, local_num_tokens, {'lm loss': (reporting_loss)})
def forward_step(data_iterator, model):
@@ -170,6 +185,7 @@ def forward_step(data_iterator, model):
"""
data_batch = get_batch(data_iterator)
output_tensor, loss_mask = model(**data_batch)
+
# Return output and loss function
return output_tensor, partial(loss_func, loss_mask)
@@ -184,6 +200,14 @@ def train_valid_test_datasets_provider(*provider_args, **provider_kwargs):
runtime_args = get_args()
try:
dataset_provider = _DATASET_PROVIDERS[runtime_args.dataset_provider]
+ if runtime_args.dataset_provider != "mock":
+ # Calculate max_seq_length from total_seq_length
+ max_seq_length = runtime_args.total_seq_length
+ print_rank_0(f"MIMO Training: Using max_seq_length = {max_seq_length} "
+ f"(total_seq_length: {runtime_args.total_seq_length})")
+
+ # Add configs to provider_kwargs
+ provider_kwargs['max_seq_length'] = max_seq_length
except KeyError as e:
raise ValueError(
f"Unsupported dataset provider '{runtime_args.dataset_provider}'. "
@@ -199,6 +223,7 @@ def model_provider(
add_decoder: bool = True,
image_special_token_id: int = 32000,
audio_special_token_id: int = 32002,
+ **framework_kwargs,
):
"""Model provider for MIMO model training.
@@ -209,8 +234,13 @@ def model_provider(
add_decoder: Whether to add a decoder to the model (not supported yet)(default: True)
image_special_token_id: Special token ID for the image modality (default: 32000)
audio_special_token_id: Special token ID for the audio modality (default: 32002)
+ **framework_kwargs: Framework-injected kwargs from Megatron's training loop,
+ including `config` (TransformerConfig) and `pg_collection` (ProcessGroupCollection).
+ `pg_collection` is forwarded to the model builder so process groups are passed
+ explicitly rather than fetched from global parallel state.
"""
runtime_args = get_args()
+ pg_collection = framework_kwargs.get('pg_collection')
try:
builder_fn = _MODEL_PROVIDERS[runtime_args.model_provider]
@@ -221,13 +251,15 @@ def model_provider(
) from e
if runtime_args.model_provider == "llava_vlm":
- kwargs = {
+ builder_kwargs = {
"image_special_token_id": image_special_token_id,
+ "pg_collection": pg_collection,
}
elif runtime_args.model_provider == "llava_avlm":
- kwargs = {
+ builder_kwargs = {
"image_special_token_id": image_special_token_id,
"audio_special_token_id": audio_special_token_id,
+ "pg_collection": pg_collection,
}
else:
raise ValueError(f"Unknown model provider: {runtime_args.model_provider}. Must be one of ['llava_vlm', 'llava_avlm', 'mock]")
@@ -237,10 +269,9 @@ def model_provider(
post_process,
add_encoder,
add_decoder,
- **kwargs,
+ **builder_kwargs,
)
-
if __name__ == "__main__":
train_valid_test_datasets_provider.is_distributed = True
diff --git a/examples/multimodal/Dockerfile b/examples/multimodal/Dockerfile
index ccb5741da62..d7c4fd41af5 100644
--- a/examples/multimodal/Dockerfile
+++ b/examples/multimodal/Dockerfile
@@ -1,5 +1,5 @@
# Base image: NVIDIA PyTorch container with CUDA, cuDNN, NCCL, Python, and uv pre-installed
-FROM nvcr.io/nvidia/pytorch:25.11-py3
+FROM nvcr.io/nvidia/pytorch:25.12-py3
# Install JRE for pycocoevalcap
RUN apt-get update && \
@@ -16,7 +16,7 @@ RUN uv pip install --system --no-cache --break-system-packages \
braceexpand \
webdataset \
packaging \
- transformers \
+ "transformers<5.0.0" \
datasets \
accelerate \
timm \
@@ -30,15 +30,18 @@ RUN uv pip install --system --no-cache --break-system-packages \
isort \
click \
pycocoevalcap \
- megatron-energon \
+ "megatron-energon[av_decode]~=6.0" \
mistral-common \
tiktoken \
# Additional dependencies for megatron-core[mlm]
flask-restful \
- wandb
+ wandb \
+ bitstring \
+ filetype
+ setuptools
# Install CLIP from GitHub
-RUN uv pip install --system --no-cache --break-system-packages \
+RUN uv pip install --system --no-cache --break-system-packages --no-build-isolation \
git+https://github.com/openai/CLIP.git
# Install packages with --no-deps to avoid outdated and unnecessary dependencies
diff --git a/examples/multimodal/README.md b/examples/multimodal/README.md
index e7fe2e62b8e..824e7979fe9 100644
--- a/examples/multimodal/README.md
+++ b/examples/multimodal/README.md
@@ -70,7 +70,7 @@ examples/multimodal/combine_lm_vision_checkpoints.sh /path/to/mistral/model /pat
```
> Please enter a desired train/val/test split like "0.5, 0.2, 0.3" or "8,1,1": 9,1,0
> Do you want to create a dataset.yaml interactively? [Y/n]: Y
- > Please enter a number to choose a class: 10 (VQAWebdataset)
+ > Please enter a number to choose a class: 9 (VQASample)
> Do you want to set a simple field_map[Y] (or write your own sample_loader [n])? [Y/n]: Y
> Please enter a webdataset field name for 'image' (): jpg
> Please enter a webdataset field name for 'context' (): json[0][value]
diff --git a/examples/multimodal/energon_util.py b/examples/multimodal/energon_util.py
index 360f97a6a54..8406879ecec 100644
--- a/examples/multimodal/energon_util.py
+++ b/examples/multimodal/energon_util.py
@@ -1,12 +1,9 @@
# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
import torch
-import warnings
from dataclasses import dataclass
from typing import Any, List
from megatron.energon import Sample
-from megatron.energon.epathlib.epath import EPath
-from megatron.energon.flavors.webdataset import DefaultDecoderWebdatasetFactory
@dataclass
@@ -20,23 +17,6 @@ class SampleListSample(Sample):
samples: List[Any]
-class SampleListWebdataset(DefaultDecoderWebdatasetFactory[SampleListSample]):
- __sample_type__ = SampleListSample
-
- def __init__(self, path: EPath, **kwargs):
- warnings.warn(
- f"{type(self)} is deprecated, use the default instead and set the sample_type:\n"
- f"To convert, update your {path}/.nv-meta/dataset.yaml to:\n"
- f"# remove top-level __module__ and __class__\n"
- f"sample_type:\n"
- f" __module__: megatron.energon\n"
- f" __class__: {self.__sample_type__.__name__}\n"
- f"# Keep the remaining content",
- DeprecationWarning,
- )
- super().__init__(path, **kwargs)
-
-
@dataclass
class OfflineTargetAspectRatioSample(Sample):
"""Sample type for image + text samples with target aspect ratio computed offline."""
diff --git a/examples/multimodal/evaluation/evaluate_mmmu.py b/examples/multimodal/evaluation/evaluate_mmmu.py
index 90cf141cd54..2bdc2ebbdc5 100644
--- a/examples/multimodal/evaluation/evaluate_mmmu.py
+++ b/examples/multimodal/evaluation/evaluate_mmmu.py
@@ -1,3 +1,4 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
import argparse
import glob
import json
@@ -104,7 +105,7 @@ def mmmu_eval(input_path, groundtruth_path):
print(output.stderr)
print(output.stdout)
- m = re.search("'Overall': {'num': \d+, 'acc': (\d.\d+)}", output.stdout)
+ m = re.search(r"'Overall': {'num': \d+, 'acc': (\d\.\d+)}", output.stdout)
return float(m.group(1)) * 100.0
diff --git a/examples/multimodal/layer_scaling.py b/examples/multimodal/layer_scaling.py
index a82afa7cc5e..b3d2881b315 100644
--- a/examples/multimodal/layer_scaling.py
+++ b/examples/multimodal/layer_scaling.py
@@ -4,6 +4,8 @@
import torch
from megatron.core.transformer.transformer_layer import TransformerLayer
+from megatron.core.typed_torch import copy_signature
+
def _bias_dropout_add_func_layer_scaling(ls, x_with_bias, residual, prob, training):
x, bias = x_with_bias # unpack
@@ -36,6 +38,7 @@ def get_bias_dropout_add_layer_scaling(ls, training, fused):
# Add LayerScaling to our default TransformerLayer.
class LayerScalingTransformerLayer(TransformerLayer):
+ @copy_signature(TransformerLayer.__init__)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.ls1 = torch.nn.Parameter(torch.ones(self.config.hidden_size))
diff --git a/examples/multimodal/layer_specs.py b/examples/multimodal/layer_specs.py
index 56821f2cec6..ad24850b631 100644
--- a/examples/multimodal/layer_specs.py
+++ b/examples/multimodal/layer_specs.py
@@ -15,8 +15,9 @@
from megatron.core.transformer.spec_utils import ModuleSpec
from megatron.core.transformer.transformer_layer import TransformerLayer, TransformerLayerSubmodules
from megatron.core.typed_torch import not_none
+from megatron.core.extensions.transformer_engine import HAVE_TE
-try:
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import (
TEColumnParallelLinear,
TEDotProductAttention,
@@ -24,9 +25,7 @@
TENorm,
TERowParallelLinear,
)
-
- HAVE_TE = True
-except ImportError:
+else:
(
TEColumnParallelLinear,
TEDotProductAttention,
@@ -34,7 +33,6 @@
TENorm,
TERowParallelLinear,
) = (None, None, None, None, None)
- HAVE_TE = False
try:
import apex
@@ -77,7 +75,7 @@ def get_layer_spec(is_vit, normalization) -> ModuleSpec:
return ModuleSpec(
module=TransformerLayer,
submodules=TransformerLayerSubmodules(
- input_layernorm=norm,
+ input_layernorm=not_none(norm),
self_attention=ModuleSpec(
module=SelfAttention,
params={"attn_mask_type": attn_mask_type},
@@ -90,7 +88,7 @@ def get_layer_spec(is_vit, normalization) -> ModuleSpec:
),
),
self_attn_bda=get_bias_dropout_add,
- pre_mlp_layernorm=norm,
+ pre_mlp_layernorm=not_none(norm),
mlp=mlp,
mlp_bda=get_bias_dropout_add,
),
@@ -175,8 +173,8 @@ def get_mamba_layer_spec_te(padding=False) -> ModuleSpec:
mlp=ModuleSpec(
module=MLP,
submodules=MLPSubmodules(
- linear_fc1=TELayerNormColumnParallelLinear,
- linear_fc2=TERowParallelLinear,
+ linear_fc1=not_none(TELayerNormColumnParallelLinear),
+ linear_fc2=not_none(TERowParallelLinear),
),
),
mlp_bda=get_bias_dropout_add,
@@ -191,8 +189,8 @@ def get_mlp_module_spec(use_te: bool = True) -> ModuleSpec:
return ModuleSpec(
module=MLP,
submodules=MLPSubmodules(
- linear_fc1=TEColumnParallelLinear if use_te else ColumnParallelLinear,
- linear_fc2=TERowParallelLinear if use_te else RowParallelLinear,
+ linear_fc1=not_none(TEColumnParallelLinear) if use_te else ColumnParallelLinear,
+ linear_fc2=not_none(TERowParallelLinear) if use_te else RowParallelLinear,
),
)
@@ -201,6 +199,7 @@ def get_norm_mlp_module_spec_te() -> ModuleSpec:
return ModuleSpec(
module=MLP,
submodules=MLPSubmodules(
- linear_fc1=TELayerNormColumnParallelLinear, linear_fc2=TERowParallelLinear
+ linear_fc1=not_none(TELayerNormColumnParallelLinear),
+ linear_fc2=not_none(TERowParallelLinear),
),
)
diff --git a/examples/multimodal/model.py b/examples/multimodal/model.py
index 212a2cda021..494a854099e 100644
--- a/examples/multimodal/model.py
+++ b/examples/multimodal/model.py
@@ -16,7 +16,8 @@
def model_provider(
- pre_process=True, post_process=True, add_encoder=True, add_decoder=True, parallel_output=True
+ pre_process=True, post_process=True, add_encoder=True, add_decoder=True, parallel_output=True,
+ vp_stage=None, config=None, pg_collection=None
) -> LLaVAModel:
"""Builds the model.
@@ -28,6 +29,9 @@ def model_provider(
add_decoder (bool): Construct the decoder module (used with pipeline parallelism). Defaults to True. When we use pipelining, the decoder
will live on only a subset of the pipeline stages (specifically, every stage after the first one).
parallel_output (bool): Enable parallel model output.
+ vp_stage: Optional virtual pipeline stage. Used with virtual pipeline parallelism.
+ config: Optional transformer config. If None, will be created from args.
+ pg_collection: Optional process group collection. If None, will use default.
Returns:
model: A multimodal model.
@@ -213,9 +217,7 @@ def model_provider(
patch_dim=args.patch_dim,
language_rotary_base=args.rotary_base,
language_rope_scaling=args.use_rope_scaling,
- hybrid_attention_ratio=args.hybrid_attention_ratio,
- hybrid_mlp_ratio=args.hybrid_mlp_ratio,
- hybrid_override_pattern=args.hybrid_override_pattern,
+ hybrid_layer_pattern=args.hybrid_layer_pattern,
fp16_lm_cross_entropy=args.fp16_lm_cross_entropy,
image_token_index=image_token_index,
pixel_shuffle=args.pixel_shuffle,
diff --git a/examples/multimodal/nvlm/internvit.py b/examples/multimodal/nvlm/internvit.py
index 9df9af23f05..0018bb5ccb9 100644
--- a/examples/multimodal/nvlm/internvit.py
+++ b/examples/multimodal/nvlm/internvit.py
@@ -34,10 +34,11 @@
from megatron.core.transformer.enums import AttnMaskType
from megatron.core.transformer.mlp import MLP, MLPSubmodules
from megatron.core.transformer.module import MegatronModule
-from megatron.core.transformer.spec_utils import ModuleSpec, build_module
+from megatron.core.transformer.spec_utils import ModuleSpec
from megatron.core.transformer.transformer_config import TransformerConfig
-from megatron.core.transformer.transformer_layer import TransformerLayer, TransformerLayerSubmodules
+from megatron.core.transformer.transformer_layer import TransformerLayerSubmodules
from megatron.core.transformer.utils import make_sharded_tensors_for_checkpoint
+from megatron.core.typed_torch import copy_signature, not_none
from megatron.core.utils import divide
try:
@@ -61,7 +62,7 @@ class InternViTRMSNorm(MegatronModule):
def __init__(
self,
- config,
+ config: TransformerConfig,
hidden_size: int,
eps: float = 1e-6,
sequence_parallel: bool = False,
@@ -93,7 +94,7 @@ def _norm(self, x, var):
return x * torch.rsqrt(var + self.eps)
- def forward(self, x):
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
"""Run RMSNorm with an option to compute custom statistic."""
var = None
if self._compute_var:
@@ -172,6 +173,7 @@ def get_mlp_module_spec(use_te: bool = True) -> ModuleSpec:
# Override a few things that are special in InternViT and not supported by the SelfAttention class.
class InternViTSelfAttention(SelfAttention):
+ @copy_signature(SelfAttention.__init__)
def __init__(
self, config: TransformerConfig, submodules: SelfAttentionSubmodules, *args, **kwargs
):
@@ -184,7 +186,7 @@ def __init__(
self.config.hidden_size,
self.query_projection_size + 2 * self.kv_projection_size,
config=self.config,
- init_method=self.config.init_method,
+ init_method=not_none(self.config.init_method),
gather_output=False,
bias=qkv_bias,
skip_bias_add=False,
@@ -196,26 +198,23 @@ def __init__(
self.hidden_size_per_attention_head * self.num_attention_heads_per_partition
) # 512 for internvit
- self.q_layernorm = build_module(
- submodules.q_layernorm,
+ self.q_layernorm = not_none(submodules.q_layernorm)(
hidden_size=qk_layernorm_hidden_size,
config=self.config,
eps=self.config.layernorm_epsilon,
- compute_var=True,
)
- self.k_layernorm = build_module(
- submodules.k_layernorm,
+ self.k_layernorm = not_none(submodules.k_layernorm)(
hidden_size=qk_layernorm_hidden_size,
config=self.config,
eps=self.config.layernorm_epsilon,
- compute_var=True,
)
class InternViTTEDotProductAttention(TEDotProductAttention):
"""Adjusted Attention for InternViT"""
+ @copy_signature(TEDotProductAttention.forward)
def forward(self, *args, **kwargs):
"""Regular TEDotProductAttention + zero-out dummy attention heads."""
out = super().forward(*args, **kwargs)
@@ -249,8 +248,8 @@ def get_internvit_layer_spec(use_te) -> ModuleSpec:
linear_qkv=TEColumnParallelLinear if use_te else ColumnParallelLinear,
core_attention=TEDotProductAttention if use_te else DotProductAttention,
linear_proj=TERowParallelLinear if use_te else RowParallelLinear,
- q_layernorm=InternViTRMSNorm,
- k_layernorm=InternViTRMSNorm,
+ q_layernorm=partial(InternViTRMSNorm, compute_var=True),
+ k_layernorm=partial(InternViTRMSNorm, compute_var=True),
),
),
self_attn_bda=get_bias_dropout_add_layer_scaling,
diff --git a/examples/multimodal/pretrain_mistral_clip.sh b/examples/multimodal/pretrain_mistral_clip.sh
index 4afcc0f2dad..8ed3d0ecc64 100755
--- a/examples/multimodal/pretrain_mistral_clip.sh
+++ b/examples/multimodal/pretrain_mistral_clip.sh
@@ -21,7 +21,7 @@ TENSORBOARD_DIR="${OUTPUT}/tensorboard"
export TRITON_CACHE_DIR="${WORKSPACE}/triton-cache/"
# The following patch to the Triton cache manager is needed for Triton version <= 3.1
-export TRITON_CACHE_MANAGER="megatron.core.ssm.triton_cache_manager:ParallelFileCacheManager"
+# export TRITON_CACHE_MANAGER="megatron.core.ssm.triton_cache_manager:ParallelFileCacheManager"
if [[ -z $LOAD_NAME ]]; then
echo "Please set LOAD_NAME for input model name."
@@ -53,6 +53,7 @@ OPTIONS=" \
--apply-layernorm-1p \
--attention-softmax-in-fp32 \
--use-checkpoint-args \
+ --no-use-tokenizer-model-from-checkpoint-args \
--use-distributed-optimizer \
--transformer-impl transformer_engine \
--use-te \
@@ -129,4 +130,4 @@ OPTIONS=" \
export NVTE_APPLY_QK_LAYER_SCALING=0
export NVTE_ALLOW_NONDETERMINISTIC_ALGO=${NONDETERMINISTIC_ATTN}
-torchrun --nproc_per_node 8 examples/multimodal/train.py ${OPTIONS}
\ No newline at end of file
+torchrun --nproc_per_node 8 examples/multimodal/train.py ${OPTIONS}
diff --git a/examples/multimodal/radio/radio_g.py b/examples/multimodal/radio/radio_g.py
index f139632df86..9883d58db61 100644
--- a/examples/multimodal/radio/radio_g.py
+++ b/examples/multimodal/radio/radio_g.py
@@ -16,8 +16,9 @@
from megatron.core.transformer.spec_utils import ModuleSpec
from megatron.core.transformer.transformer_layer import TransformerLayer, TransformerLayerSubmodules
from megatron.core.typed_torch import not_none
+from megatron.core.extensions.transformer_engine import HAVE_TE
-try:
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import (
TEColumnParallelLinear,
TEDotProductAttention,
@@ -25,9 +26,7 @@
TENorm,
TERowParallelLinear,
)
-
- HAVE_TE = True
-except ImportError:
+else:
(
TEColumnParallelLinear,
TEDotProductAttention,
@@ -35,7 +34,6 @@
TENorm,
TERowParallelLinear,
) = (None, None, None, None, None)
- HAVE_TE = False
try:
import apex
@@ -58,8 +56,8 @@ def get_mlp_module_spec(use_te: bool = True) -> ModuleSpec:
return ModuleSpec(
module=MLP,
submodules=MLPSubmodules(
- linear_fc1=TEColumnParallelLinear if use_te else ColumnParallelLinear,
- linear_fc2=TERowParallelLinear if use_te else RowParallelLinear,
+ linear_fc1=not_none(TEColumnParallelLinear) if use_te else ColumnParallelLinear,
+ linear_fc2=not_none(TERowParallelLinear) if use_te else RowParallelLinear,
),
)
@@ -68,7 +66,8 @@ def get_norm_mlp_module_spec_te() -> ModuleSpec:
return ModuleSpec(
module=MLP,
submodules=MLPSubmodules(
- linear_fc1=TELayerNormColumnParallelLinear, linear_fc2=TERowParallelLinear
+ linear_fc1=not_none(TELayerNormColumnParallelLinear),
+ linear_fc2=not_none(TERowParallelLinear),
),
)
@@ -93,7 +92,7 @@ def get_radio_g_layer_spec(normalization) -> ModuleSpec:
return ModuleSpec(
module=LayerScalingTransformerLayer,
submodules=TransformerLayerSubmodules(
- input_layernorm=norm,
+ input_layernorm=not_none(norm),
self_attention=ModuleSpec(
module=SelfAttention,
params={"attn_mask_type": attn_mask_type},
@@ -106,7 +105,7 @@ def get_radio_g_layer_spec(normalization) -> ModuleSpec:
),
),
self_attn_bda=get_bias_dropout_add_layer_scaling,
- pre_mlp_layernorm=norm,
+ pre_mlp_layernorm=not_none(norm),
mlp=mlp,
mlp_bda=get_bias_dropout_add_layer_scaling,
),
diff --git a/examples/multimodal/run_text_generation.py b/examples/multimodal/run_text_generation.py
index 703b2c37c50..e55679c1b2e 100644
--- a/examples/multimodal/run_text_generation.py
+++ b/examples/multimodal/run_text_generation.py
@@ -227,7 +227,7 @@ def generate_samples(model, config: EvaluationConfig, print_output):
inference_request = VLMInferenceRequest(
request_id=inference_engine.get_new_request_id(),
prompt=conv,
- prompt_tokens=controller.tokenize_prompt(conv),
+ prompt_tokens=controller.tokenize_prompt(controller.tokenizer, conv),
sampling_params=sampling_params,
num_img_embeddings_per_tile=num_img_embeddings_per_tile,
imgs=imgs,
@@ -344,7 +344,7 @@ def generate_samples(model, config: EvaluationConfig, print_output):
inference_request = VLMInferenceRequest(
request_id=inference_engine.get_new_request_id(),
prompt=conv,
- prompt_tokens=controller.tokenize_prompt(conv),
+ prompt_tokens=controller.tokenize_prompt(controller.tokenizer, conv),
sampling_params=sampling_params,
num_img_embeddings_per_tile=num_img_embeddings_per_tile,
imgs=imgs,
diff --git a/examples/post_training/modelopt/README.md b/examples/post_training/modelopt/README.md
index 93b5022b2aa..a6fcaf8f5b7 100644
--- a/examples/post_training/modelopt/README.md
+++ b/examples/post_training/modelopt/README.md
@@ -34,6 +34,7 @@ knowledge distillation, pruning, speculative decoding, and more.
| `moonshotai/Kimi-K2-Instruct` | ✅ | ✅ | - | - |
| `nvidia/NVIDIA-Nemotron-Nano-9B-v2` | ✅ | - | ✅ | ✅ |
| `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16` | ✅ | - | ✅ | ✅ |
+| `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16` | ✅ | - | ✅ | ✅ |
| `openai/gpt-oss-{20b, 120b}` | ✅ | **Online** | ✅ | ✅ |
| `Qwen/Qwen3-{0.6B, 8B}` | ✅ | ✅ | ✅ | ✅ |
| `Qwen/Qwen3-{30B-A3B, 235B-A22B}` | **WAR** | ✅ | ✅ | ✅ |
@@ -81,6 +82,18 @@ provide `${EXPORT_DIR}` to `export.sh`.
./export.sh meta-llama/Llama-3.2-1B-Instruct
```
+To export the model for vLLM fakequant example in `modelopt/examples/vllm_serve/vllm_serve_fakequant.py`,
+export the model with flag `--export-vllm-fq`:
+```sh
+\
+ PP=1 \
+ MLM_EXTRA_ARGS=--export-vllm-fq \
+ HF_MODEL_CKPT= \
+ MLM_MODEL_CKPT=/tmp/Llama-3.2-1B-Instruct_quant \
+ EXPORT_DIR=/tmp/Llama-3.2-1B-Instruct_export \
+ ./export.sh meta-llama/Llama-3.2-1B-Instruct
+```
+
For KV cache quantization, add a flag like `MLM_EXTRA_ARGS="--export-kv-cache-quant fp8"` while specifying your desired KV cache precision (see `KV_QUANT_CFG_CHOICES` in `quantize.py`).
### ⭐ Online BF16 EAGLE3 Training
diff --git a/examples/post_training/modelopt/conf/arguments.sh b/examples/post_training/modelopt/conf/arguments.sh
index f17b8c9154a..826ce312ae3 100644
--- a/examples/post_training/modelopt/conf/arguments.sh
+++ b/examples/post_training/modelopt/conf/arguments.sh
@@ -1,7 +1,12 @@
#!/bin/bash
set -e
-MLM_MODEL_CFG=$1
+MLM_MODEL_CFG=${1}
+
+if [ -z $1 ]; then
+ printf "${MLM_ERROR} Model configuration name must be provided as the first argument (e.g. 'meta-llama/Llama-3.2-1B-Instruct')\n"
+ exit 1
+fi
# Bash coloring
RED='\033[0;31m'
diff --git a/examples/post_training/modelopt/conf/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.sh b/examples/post_training/modelopt/conf/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.sh
index c294e03235c..1fa00889e99 100644
--- a/examples/post_training/modelopt/conf/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.sh
+++ b/examples/post_training/modelopt/conf/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.sh
@@ -30,17 +30,15 @@ MODEL_ARGS=" \
\
--attention-backend flash \
--disable-gloo-process-groups \
- --is-hybrid-model \
--mamba-num-heads 64 \
--mamba-head-dim 64 \
- --hybrid-override-pattern MEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEMEM*EMEMEMEME \
+ --hybrid-layer-pattern MEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEMEM*EMEMEMEME \
--use-mcore-models \
--untie-embeddings-and-output-weights \
--disable-bias-linear \
--init-method-std 0.0173 \
--position-embedding-type none \
--squared-relu \
- --num-layers 52 \
--hidden-size 2688 \
--num-attention-heads 32 \
--group-query-attention \
diff --git a/examples/post_training/modelopt/conf/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16.sh b/examples/post_training/modelopt/conf/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16.sh
new file mode 100644
index 00000000000..977be033df0
--- /dev/null
+++ b/examples/post_training/modelopt/conf/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16.sh
@@ -0,0 +1,62 @@
+#!/bin/bash
+
+if [ -z ${HF_MODEL_CKPT} ]; then
+ HF_MODEL_CKPT=nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
+ TOKENIZER_MODEL=nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
+else
+ TOKENIZER_MODEL=${HF_MODEL_CKPT}
+fi
+
+
+
+MODEL_ARGS=" \
+ --trust-remote-code \
+ --save-interval 100000 \
+ --micro-batch-size 1 \
+ --enable-experimental \
+ --use-fused-weighted-squared-relu \
+ --cross-entropy-loss-fusion \
+ --cross-entropy-fusion-impl native \
+ --num-experts 512 \
+ --moe-router-score-function sigmoid \
+ --moe-grouped-gemm \
+ --moe-aux-loss-coeff 1e-4 \
+ --moe-router-topk 22 \
+ --moe-permute-fusion \
+ --moe-router-topk-scaling-factor 5.0 \
+ --moe-router-enable-expert-bias \
+ --moe-router-dtype fp32 \
+ --moe-router-load-balancing-type seq_aux_loss \
+ --moe-shared-expert-intermediate-size 5376 \
+ --moe-token-dispatcher-type allgather \
+ --moe-latent-size 1024 \
+ \
+ --attention-backend flash \
+ --disable-gloo-process-groups \
+ --is-hybrid-model \
+ --mamba-num-heads 128 \
+ --mamba-head-dim 64 \
+ --hybrid-layer-pattern MEMEMEM*EMEMEMEM*EMEMEMEM*EMEMEMEMEM*EMEMEMEMEM*EMEMEMEMEM*EMEMEMEMEM*EMEMEMEM*EMEMEMEME \
+ \
+ --use-mcore-models \
+ --untie-embeddings-and-output-weights \
+ --disable-bias-linear \
+ --init-method-std 0.014 \
+ --position-embedding-type none \
+ --squared-relu \
+ --hidden-size 4096 \
+ --num-attention-heads 32 \
+ --group-query-attention \
+ --num-query-groups 2 \
+ --ffn-hidden-size 2688 \
+ --kv-channels 128 \
+ --normalization RMSNorm \
+ --attention-dropout 0.0 \
+ --hidden-dropout 0.0 \
+ \
+ --tokenizer-type HuggingFaceTokenizer \
+ --bf16 \
+ --seq-length 8192 \
+ --max-position-embeddings 8192 \
+ --export-model-type MambaModel \
+ "
diff --git a/examples/post_training/modelopt/conf/nvidia/NVIDIA-Nemotron-Nano-9B-v2.sh b/examples/post_training/modelopt/conf/nvidia/NVIDIA-Nemotron-Nano-9B-v2.sh
index a2212483008..83867430a97 100644
--- a/examples/post_training/modelopt/conf/nvidia/NVIDIA-Nemotron-Nano-9B-v2.sh
+++ b/examples/post_training/modelopt/conf/nvidia/NVIDIA-Nemotron-Nano-9B-v2.sh
@@ -19,15 +19,13 @@ MODEL_ARGS=" \
--no-rope-fusion \
--normalization RMSNorm \
--squared-relu \
- --num-layers 56 \
--hidden-size 4480 \
--ffn-hidden-size 15680 \
--num-attention-heads 40 \
--kv-channels 128 \
--group-query-attention \
--num-query-groups 8 \
- --hybrid-override-pattern M-M-M-MM-M-M-M*-M-M-M*-M-M-M-M*-M-M-M-M*-M-MM-M-M-M-M-M- \
- --is-hybrid-model \
+ --hybrid-layer-pattern M-M-M-MM-M-M-M*-M-M-M*-M-M-M-M*-M-M-M-M*-M-MM-M-M-M-M-M- \
--mamba-head-dim 80 \
--mamba-num-heads 128 \
--mamba-num-groups 8 \
diff --git a/examples/post_training/modelopt/conf/nvidia/Nemotron-H-47B-Reasoning-128K.sh b/examples/post_training/modelopt/conf/nvidia/Nemotron-H-47B-Reasoning-128K.sh
index ad07c1061c5..901e607f298 100644
--- a/examples/post_training/modelopt/conf/nvidia/Nemotron-H-47B-Reasoning-128K.sh
+++ b/examples/post_training/modelopt/conf/nvidia/Nemotron-H-47B-Reasoning-128K.sh
@@ -18,15 +18,13 @@ MODEL_ARGS=" \
--position-embedding-type none \
--normalization RMSNorm \
--squared-relu \
- --num-layers 98 \
--hidden-size 8192 \
--ffn-hidden-size 30720 \
--num-attention-heads 64 \
--kv-channels 128 \
--group-query-attention \
--num-query-groups 8 \
- --hybrid-override-pattern M-M-M-M-M-M-M-M-M*-M-M-M-M-M-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-M-M---MM---M-M*-M-M-M-M-M- \
- --is-hybrid-model \
+ --hybrid-layer-pattern M-M-M-M-M-M-M-M-M*-M-M-M-M-M-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-M-M---MM---M-M*-M-M-M-M-M- \
--mamba-head-dim 64 \
--mamba-num-heads 256 \
--mamba-num-groups 8 \
diff --git a/examples/post_training/modelopt/conf/nvidia/Nemotron-H-4B-Instruct.sh b/examples/post_training/modelopt/conf/nvidia/Nemotron-H-4B-Instruct.sh
index 4ba91dbd8c6..084db49e0eb 100644
--- a/examples/post_training/modelopt/conf/nvidia/Nemotron-H-4B-Instruct.sh
+++ b/examples/post_training/modelopt/conf/nvidia/Nemotron-H-4B-Instruct.sh
@@ -21,14 +21,13 @@ MODEL_ARGS=" \
--no-position-embedding \
--normalization RMSNorm \
--squared-relu \
- --num-layers 52 \
--hidden-size 3072 \
--ffn-hidden-size 12288 \
--kv-channels 128 \
--num-attention-heads 32 \
--group-query-attention \
--num-query-groups 8 \
- --hybrid-override-pattern M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M- \
+ --hybrid-layer-pattern M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M- \
--mamba-head-dim 64 \
--mamba-num-heads 112 \
--mamba-num-groups 8 \
diff --git a/examples/post_training/modelopt/conf/nvidia/Nemotron-H-56B-Base-8K.sh b/examples/post_training/modelopt/conf/nvidia/Nemotron-H-56B-Base-8K.sh
index 8377f0f11d6..645a159d075 100644
--- a/examples/post_training/modelopt/conf/nvidia/Nemotron-H-56B-Base-8K.sh
+++ b/examples/post_training/modelopt/conf/nvidia/Nemotron-H-56B-Base-8K.sh
@@ -12,8 +12,7 @@ MODEL_ARGS=" \
--save-interval 100000 \
--micro-batch-size 1 \
--attention-backend flash \
- --is-hybrid-model \
- --hybrid-override-pattern M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M- \
+ --hybrid-layer-pattern M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M- \
--mamba-state-dim 256 \
--tiktoken-pattern v2 \
--use-mcore-models \
@@ -22,7 +21,6 @@ MODEL_ARGS=" \
--init-method-std 0.0099 \
--position-embedding-type none \
--squared-relu \
- --num-layers 118 \
--hidden-size 8192 \
--num-attention-heads 64 \
--group-query-attention \
diff --git a/examples/post_training/modelopt/conf/nvidia/Nemotron-H-8B-Base-8K.sh b/examples/post_training/modelopt/conf/nvidia/Nemotron-H-8B-Base-8K.sh
index b04bf76f360..66f3ad368b4 100644
--- a/examples/post_training/modelopt/conf/nvidia/Nemotron-H-8B-Base-8K.sh
+++ b/examples/post_training/modelopt/conf/nvidia/Nemotron-H-8B-Base-8K.sh
@@ -20,14 +20,12 @@ MODEL_ARGS=" \
--no-position-embedding \
--normalization RMSNorm \
--squared-relu \
- --num-layers 52 \
--hidden-size 4096 \
--ffn-hidden-size 21504 \
--num-attention-heads 32 \
--group-query-attention \
--num-query-groups 8 \
- --hybrid-override-pattern M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M- \
- --is-hybrid-model \
+ --hybrid-layer-pattern M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M- \
--mamba-head-dim 64 \
--mamba-num-heads 128 \
--mamba-num-groups 8 \
diff --git a/examples/post_training/modelopt/conf/openai/gpt-oss-120b.sh b/examples/post_training/modelopt/conf/openai/gpt-oss-120b.sh
index 3c7f69c0da6..9bc6142375d 100644
--- a/examples/post_training/modelopt/conf/openai/gpt-oss-120b.sh
+++ b/examples/post_training/modelopt/conf/openai/gpt-oss-120b.sh
@@ -49,5 +49,5 @@ MODEL_ARGS=" \
--window-attn-skip-freq 2 \
--enable-gpt-oss \
--activation-func-clamp-value 7.0 \
- --window-size 128,0 \
+ --window-size 127,0 \
"
diff --git a/examples/post_training/modelopt/conf/openai/gpt-oss-20b.sh b/examples/post_training/modelopt/conf/openai/gpt-oss-20b.sh
index a0180ec44b8..601709c847a 100644
--- a/examples/post_training/modelopt/conf/openai/gpt-oss-20b.sh
+++ b/examples/post_training/modelopt/conf/openai/gpt-oss-20b.sh
@@ -49,5 +49,5 @@ MODEL_ARGS=" \
--window-attn-skip-freq 2 \
--enable-gpt-oss \
--activation-func-clamp-value 7.0 \
- --window-size 128,0 \
+ --window-size 127,0 \
"
diff --git a/examples/post_training/modelopt/convert_model.py b/examples/post_training/modelopt/convert_model.py
index cf5f6e5bbbb..eaec9789e1e 100644
--- a/examples/post_training/modelopt/convert_model.py
+++ b/examples/post_training/modelopt/convert_model.py
@@ -2,6 +2,7 @@
"""Convert a GPTModel."""
import functools
+import inspect
import json
import os
import sys
@@ -20,18 +21,16 @@
from megatron.post_training.checkpointing import load_modelopt_checkpoint
from megatron.post_training.model_builder import modelopt_gpt_mamba_builder
from megatron.post_training.utils import (
- modelopt_version_at_least,
report_current_memory_info,
to_empty_if_meta,
)
-from megatron.training import get_args, get_tokenizer
+from megatron.training import get_args
from megatron.training.checkpointing import save_checkpoint
from megatron.training.initialize import initialize_megatron
from megatron.training.utils import print_rank_0, unwrap_model
from model_provider import model_provider
ALGO_TO_CONFIG = {
- "eagle1": mtsp.config.EAGLE1_DEFAULT_CFG,
"eagle3": mtsp.config.EAGLE3_DEFAULT_CFG,
"eagle-mtp": mtsp.config.EAGLE_MTP_DEFAULT_CFG,
}
@@ -49,7 +48,7 @@ def add_convert_args(parser):
group.add_argument(
'--algorithm',
type=str,
- choices=["medusa", "eagle1", "eagle3", "None"],
+ choices=["eagle3", "None"],
default="None",
help='Chosing between different speculative decoding algorithms. Default is None.',
)
@@ -60,6 +59,12 @@ def add_convert_args(parser):
help="EAGLE architecture config. If not given, "
"a default config will be use. If provided, it will overwrite the default config.",
)
+ group.add_argument(
+ "--mix-hidden-states",
+ type=bool,
+ default=False,
+ help="Whether to mix hidden states from previous TTT step.",
+ )
add_modelopt_args(parser)
return parser
@@ -91,8 +96,9 @@ def check_arguments():
exit()
if hasattr(args, 'moe_grouped_gemm') and args.moe_grouped_gemm == True:
- print_rank_0("WARNING: Forcing moe_grouped_gemm to False for PTQ and export.")
- args.moe_grouped_gemm = False
+ if not getattr(args, 'export_default_te_spec', False):
+ print_rank_0("WARNING: Forcing moe_grouped_gemm to False for PTQ and export.")
+ args.moe_grouped_gemm = False
if __name__ == "__main__":
@@ -140,7 +146,7 @@ def check_arguments():
"dtype": import_dtype,
"moe_router_dtype": args.moe_router_dtype,
}
- if modelopt_version_at_least("0.41.0"):
+ if "trust_remote_code" in inspect.signature(import_mcore_gpt_from_hf).parameters:
import_kwargs.update({"trust_remote_code": args.trust_remote_code})
import_mcore_gpt_from_hf(
unwrapped_model, args.pretrained_model_path, workspace_dir, **import_kwargs
@@ -148,7 +154,7 @@ def check_arguments():
elif args.load is not None:
_ = load_modelopt_checkpoint(model)
- if args.algorithm in ("eagle1", "eagle3"):
+ if args.algorithm == "eagle3":
mtsp_config = ALGO_TO_CONFIG[args.algorithm]
if args.eagle_config:
with open(args.eagle_config) as f:
@@ -157,6 +163,8 @@ def check_arguments():
if args.export_offline_model:
mtsp_config["config"]["eagle_offline"] = True
+ if args.mix_hidden_states:
+ mtsp_config["config"]["eagle_mix_hidden_states"] = True
unwrapped_model = mtsp.convert(unwrapped_model, mtsp_config)
@@ -166,10 +174,6 @@ def check_arguments():
mcore_eagle_state_dict = torch.load(args.extra_model_path)
eagle_module.load_state_dict(mcore_eagle_state_dict, strict=False)
- elif args.algorithm == "medusa":
- config = {"medusa_num_heads": args.export_num_medusa_heads, "medusa_num_layers": 1}
- unwrapped_model = mtsp.convert(unwrapped_model, [("medusa", config)])
-
print_rank_0(f"Converted Model:\n {model}")
torch.distributed.barrier()
diff --git a/examples/post_training/modelopt/export.py b/examples/post_training/modelopt/export.py
index 9dc66eecb6d..5e3b2a1716e 100755
--- a/examples/post_training/modelopt/export.py
+++ b/examples/post_training/modelopt/export.py
@@ -2,6 +2,7 @@
"""Export a GPTModel."""
import functools
+import inspect
import os
import sys
import warnings
@@ -15,7 +16,6 @@
from megatron.post_training.arguments import add_modelopt_args
from megatron.post_training.checkpointing import load_modelopt_checkpoint
from megatron.post_training.model_builder import modelopt_gpt_mamba_builder
-from megatron.post_training.utils import modelopt_version_at_least
from megatron.training import get_args, get_model
from megatron.training.initialize import initialize_megatron
from megatron.training.utils import unwrap_model
@@ -37,6 +37,12 @@ def add_modelopt_export_args(parser):
type=str,
help="A pretrained model hosted inside a model repo on huggingface.co.",
)
+ group.add_argument(
+ "--export-vllm-fq",
+ action="store_true",
+ default=False,
+ help="Export the model for vLLM fakequant reload.",
+ )
group.add_argument("--export-dir", type=str, help="The target export path.")
add_modelopt_args(parser)
return parser
@@ -96,6 +102,8 @@ def add_modelopt_export_args(parser):
"export_dir": args.export_dir,
"moe_router_dtype": unwrapped_model.config.moe_router_dtype,
}
- if modelopt_version_at_least("0.41.0"):
+ if "trust_remote_code" in inspect.signature(mtex.export_mcore_gpt_to_hf).parameters:
export_kwargs.update({"trust_remote_code": args.trust_remote_code})
- mtex.export_mcore_gpt_to_hf(unwrapped_model, args.pretrained_model_name, **export_kwargs)
+
+ export_fn = mtex.export_mcore_gpt_to_hf_vllm_fq if args.export_vllm_fq else mtex.export_mcore_gpt_to_hf
+ export_fn(unwrapped_model, args.pretrained_model_name, **export_kwargs)
diff --git a/examples/post_training/modelopt/finetune.py b/examples/post_training/modelopt/finetune.py
index 19ece4ef299..f7f7c24f970 100755
--- a/examples/post_training/modelopt/finetune.py
+++ b/examples/post_training/modelopt/finetune.py
@@ -21,12 +21,13 @@
from megatron.post_training.loss_func import loss_func
from megatron.post_training.model_builder import modelopt_gpt_mamba_builder
from megatron.post_training.non_loss_data_func import report_draft_acceptance_length
-from megatron.training import get_args, get_timers, get_tokenizer, pretrain
+from megatron.training import get_args, get_timers, pretrain
from megatron.training.utils import (
get_batch_on_this_cp_rank,
get_ltor_masks_and_position_ids,
print_rank_0,
)
+from utils import get_hf_tokenizer
from model_provider import model_provider
REMOVE_THINK_CHAT_TEMPLATE = (
@@ -48,8 +49,7 @@ def get_eos_id():
We insert eos_token between two samples during packing. However, if the eos_token is used in message or after turns,
we need to replace it with some other special tokens that do not appear in message."""
- tokenizer = get_tokenizer()
- hf_tokenizer = tokenizer._tokenizer
+ hf_tokenizer = get_hf_tokenizer()
if hf_tokenizer.eos_token == "<|eot_id|>":
return 128001
@@ -341,9 +341,8 @@ def train_valid_test_sft_datasets_provider(train_val_test_num_samples):
"""
print_rank_0("> building train, validation, and test SFT datasets ...")
args = get_args()
- tokenizer = get_tokenizer()
-
- if not isinstance(tokenizer._tokenizer, transformers.PreTrainedTokenizerBase):
+ hf_tokenizer = get_hf_tokenizer()
+ if not isinstance(hf_tokenizer, transformers.PreTrainedTokenizerBase):
raise ValueError("SFTDataset only supports transformers.PreTrainedTokenizerBase!")
if args.micro_batch_size > 1:
@@ -358,7 +357,7 @@ def train_valid_test_sft_datasets_provider(train_val_test_num_samples):
else:
kwargs = {
"hf_dataset": args.finetune_hf_dataset,
- "tokenizer": tokenizer._tokenizer,
+ "tokenizer": hf_tokenizer,
"seq_length": args.seq_length,
# Optional kwargs
"num_shards": mpu.get_expert_data_parallel_world_size(),
@@ -444,7 +443,7 @@ def get_batch(data_iterator):
def non_loss_data_func(model: GPTModel):
"""Callback to compute the acceptance length."""
args = get_args()
- if not args.export_offline_model:
+ if not args.export_offline_model and args.context_parallel_size == 1:
try:
report_draft_acceptance_length(model)
except Exception as e:
diff --git a/examples/post_training/modelopt/finetune.sh b/examples/post_training/modelopt/finetune.sh
index e7ba0f022dc..1affc3c04da 100755
--- a/examples/post_training/modelopt/finetune.sh
+++ b/examples/post_training/modelopt/finetune.sh
@@ -38,6 +38,12 @@ if [ -z ${MLM_DATA_ARGS} ]; then
"
fi
+if [[ -v CP && "$CP" != "1" ]]; then
+ BACKEND="fused"
+else
+ BACKEND="auto"
+fi
+
if [ -z ${MLM_TRAIN_ARGS} ]; then
MLM_TRAIN_ARGS=" \
--no-gradient-accumulation-fusion \
@@ -48,6 +54,7 @@ if [ -z ${MLM_TRAIN_ARGS} ]; then
--attention-dropout 0.0 \
--hidden-dropout 0.0 \
--no-check-for-nan-in-loss-and-grad \
+ --attention-backend ${BACKEND} \
"
fi
@@ -83,6 +90,7 @@ ${LAUNCH_SCRIPT} ${SCRIPT_DIR}/finetune.py \
--expert-model-parallel-size ${EP} \
--pipeline-model-parallel-size ${PP} \
--context-parallel-size ${CP} \
+ --cp-comm-type p2p \
--tokenizer-model ${TOKENIZER_MODEL} \
--load ${MLM_MODEL_CKPT} \
--save ${MLM_MODEL_SAVE} \
diff --git a/examples/post_training/modelopt/generate.py b/examples/post_training/modelopt/generate.py
index 63d3f241f59..3d3f6571b34 100644
--- a/examples/post_training/modelopt/generate.py
+++ b/examples/post_training/modelopt/generate.py
@@ -16,7 +16,8 @@
from megatron.post_training.generate import simple_generate
from megatron.post_training.model_builder import modelopt_gpt_mamba_builder
from megatron.post_training.utils import report_current_memory_info, to_empty_if_meta
-from megatron.training import get_args, get_model, get_tokenizer, initialize_megatron
+from megatron.training import get_args, get_model, initialize_megatron
+from utils import get_hf_tokenizer
from megatron.training.utils import print_rank_0, unwrap_model
from model_provider import model_provider
@@ -121,7 +122,7 @@ def get_conversations(example):
else:
dataset = load_dataset(args.finetune_hf_dataset, split=args.finetune_data_split)
- tokenizer = get_tokenizer()._tokenizer
+ tokenizer = get_hf_tokenizer()
if args.load is not None:
diff --git a/examples/post_training/modelopt/mmlu.py b/examples/post_training/modelopt/mmlu.py
index d475ac9fb30..5aa5d1c24c7 100644
--- a/examples/post_training/modelopt/mmlu.py
+++ b/examples/post_training/modelopt/mmlu.py
@@ -19,7 +19,8 @@
from megatron.post_training.generate import simple_generate
from megatron.post_training.model_builder import modelopt_gpt_mamba_builder
from megatron.post_training.utils import report_current_memory_info
-from megatron.training import get_args, get_model, get_tokenizer, initialize_megatron
+from megatron.training import get_args, get_model, initialize_megatron
+from utils import get_hf_tokenizer
from megatron.training.utils import print_rank_0, unwrap_model
import modelopt.torch.quantization as mtq
from model_provider import model_provider
@@ -168,7 +169,7 @@ def generate_prompt(test_example, dev_examples, few_shots=0, no_subject_prompt=F
disable_tqdm = args.disable_tqdm or torch.distributed.get_rank() > 0
- tokenizer = get_tokenizer()._tokenizer
+ tokenizer = get_hf_tokenizer()
if args.load is not None:
load_modelopt_checkpoint(model, strict=not args.untie_embeddings_and_output_weights)
@@ -177,7 +178,10 @@ def generate_prompt(test_example, dev_examples, few_shots=0, no_subject_prompt=F
# Fold the scalars into weight for speedup.
# [TODO]: fold_weight current assumes all weight_quantizer has weight allocated;
# however, this is not the case when share_embeddings_and_output_weights is False.
- if not getattr(unwrapped_model, "share_embeddings_and_output_weights", False):
+ # [TODO]: fold_weight does not support TEGroupedMLP (QuantTEColumnParallelGroupedLinear)
+ # which stores per-expert weights as weight0, weight1, etc. instead of a single weight.
+ has_grouped_mlp = any("TEGroupedMLP" in type(m).__name__ for m in unwrapped_model.modules())
+ if not getattr(unwrapped_model, "share_embeddings_and_output_weights", False) and not has_grouped_mlp:
mtq.fold_weight(unwrapped_model)
all_subjects = get_all_subjects()
diff --git a/examples/post_training/modelopt/prune.py b/examples/post_training/modelopt/prune.py
index 2671b6badd9..56bbffa0cd0 100644
--- a/examples/post_training/modelopt/prune.py
+++ b/examples/post_training/modelopt/prune.py
@@ -6,6 +6,7 @@
"""
import functools
+import inspect
import os
import sys
import warnings
@@ -29,10 +30,10 @@
from megatron.post_training.generate import simple_generate
from megatron.post_training.model_builder import modelopt_gpt_mamba_builder
from megatron.post_training.utils import (
- modelopt_version_at_least,
report_current_memory_info,
)
-from megatron.training import get_args, get_model, get_tokenizer, initialize_megatron
+from megatron.training import get_args, get_model, initialize_megatron
+from utils import get_hf_tokenizer
from megatron.training.checkpointing import save_checkpoint
from megatron.training.utils import print_rank_0, unwrap_model
from model_provider import model_provider
@@ -160,7 +161,7 @@ def get_params(model):
args = get_args()
check_arguments(args)
- tokenizer = get_tokenizer()._tokenizer
+ tokenizer = get_hf_tokenizer()
model = get_model(
functools.partial(model_provider, modelopt_gpt_mamba_builder), wrap_with_ddp=False
)
@@ -176,7 +177,7 @@ def get_params(model):
import_dtype = torch.float16 if args.fp16 else torch.bfloat16
workspace_dir = os.environ.get("MLM_WORK_DIR", "/tmp")
import_kwargs = {"dtype": import_dtype}
- if modelopt_version_at_least("0.41.0"):
+ if "trust_remote_code" in inspect.signature(import_mcore_gpt_from_hf).parameters:
import_kwargs.update({"trust_remote_code": args.trust_remote_code})
import_mcore_gpt_from_hf(
unwrapped_model, args.pretrained_model_path, workspace_dir, **import_kwargs
diff --git a/examples/post_training/modelopt/quantize.py b/examples/post_training/modelopt/quantize.py
index ceedce606a5..dc4947038e5 100644
--- a/examples/post_training/modelopt/quantize.py
+++ b/examples/post_training/modelopt/quantize.py
@@ -4,18 +4,23 @@
import copy
import functools
+import inspect
+import json
import os
+import random
import sys
import warnings
import torch
import torch.distributed
-from datasets import load_dataset
from tqdm import tqdm
+# NOTE: Needs to be before modelopt imports in case megatron.core is not installed.
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "../../../")))
import modelopt.torch.quantization as mtq
+from modelopt.torch.export import import_mcore_gpt_from_hf
+from modelopt.torch.utils.dataset_utils import get_dataset_dataloader
try:
import modelopt.torch.quantization.plugins.psx_formats as mtq_psx
@@ -24,28 +29,23 @@
warnings.warn(
"psx_formats is not installed. PSX formats quantization configs will not be available."
)
-
try:
import modelopt.torch.quantization.plugins.luts as mtq_luts
except ImportError:
mtq_luts = None
warnings.warn("luts is not installed. LUTs quantization configs will not be available.")
-
-from modelopt.torch.export import import_mcore_gpt_from_hf
-
-from megatron.core import parallel_state
-from megatron.core.transformer.moe.router import TopKRouter
+from megatron.core.utils import get_batch_on_this_cp_rank
from megatron.post_training.arguments import add_modelopt_args
from megatron.post_training.checkpointing import load_modelopt_checkpoint
from megatron.post_training.generate import simple_generate
from megatron.post_training.model_builder import modelopt_gpt_mamba_builder
from megatron.post_training.utils import (
- modelopt_version_at_least,
print_distributed_quant_summary,
report_current_memory_info,
)
-from megatron.training import get_args, get_model, get_tokenizer, initialize_megatron
+from megatron.training import get_args, get_model, initialize_megatron
+from utils import get_hf_tokenizer
from megatron.training.checkpointing import save_checkpoint
from megatron.training.utils import print_rank_0, unwrap_model
from model_provider import model_provider
@@ -78,13 +78,27 @@ def add_text_generate_ptq_args(parser):
"""Add additional arguments for ModelOpt text generation PTQ."""
group = parser.add_argument_group(title="ModelOpt text generation ptq")
group.add_argument(
- "--calib-size", type=int, default=512, help="Samples to use for ptq calibration."
+ "--calib-size", type=int, default=512, help="Number of samples to use for ptq calibration."
)
group.add_argument(
- "--calib-dataset",
+ "--calib-dataset-path-or-name",
type=str,
- default="abisee/cnn_dailymail",
- help="The default clibration dataset is cnn_dailymail from HF hub.",
+ default="cnn_dailymail",
+ help="Path to local calibration dataset file (.jsonl) or HuggingFace dataset name.",
+ )
+ group.add_argument(
+ "--calib-max-sequence-length",
+ type=int,
+ default=512,
+ help="Maximum sequence length for calibration.",
+ )
+ group.add_argument(
+ "--calib-use-random-offset",
+ action="store_true",
+ help="Use random offsets when slicing sequences for calibration. (Only for local files)",
+ )
+ group.add_argument(
+ "--calib-batch-size", type=int, default=1, help="Batch size for calibration."
)
group.add_argument(
"--prompts",
@@ -92,6 +106,12 @@ def add_text_generate_ptq_args(parser):
default=("Hello!|Born in California, Soyer trained as a"),
help="Input texts. Please use | to separate different batches.",
)
+ group.add_argument(
+ "--skip-generate",
+ action="store_true",
+ default=False,
+ help="Skip the post-quantization generate/validation step.",
+ )
group.add_argument(
"--references",
type=str,
@@ -200,7 +220,10 @@ def get_last_layers_disabled_config(config, num_layers: int = 1, num_layers_to_d
def get_modelopt_torch_quantization_config():
"""Return a quantization config."""
args = get_args()
+ if args.export_quant_cfg not in QUANT_CFG_CHOICES:
+ raise ValueError(f"Unsupported quantization config {args.export_quant_cfg}.")
mtq_config = QUANT_CFG_CHOICES[args.export_quant_cfg]
+
fp8_config = {"enable": True, "num_bits": (4, 3), "axis": None}
fp4_config = {
"num_bits": (2, 1),
@@ -208,13 +231,13 @@ def get_modelopt_torch_quantization_config():
"axis": None,
"enable": True,
}
- if args.export_quant_cfg == "fp8":
+ if args.export_quant_cfg == "FP8_DEFAULT_CFG":
# Enable Medusa heads and kv-cache quantization
mtq_config["quant_cfg"]["*medusa_heads**"] = fp8_config
- if "fp4" in args.export_quant_cfg:
+ if "FP4" in args.export_quant_cfg:
# Enable Medusa heads and kv-cache quantization
mtq_config["quant_cfg"]["*medusa_heads**"] = fp4_config
- if "awq" in args.export_quant_cfg:
+ if "AWQ" in args.export_quant_cfg:
weight_quantizer = mtq_config["quant_cfg"]["*weight_quantizer"] # type: ignore
if isinstance(weight_quantizer, list):
weight_quantizer = weight_quantizer[0]
@@ -251,15 +274,75 @@ def get_modelopt_torch_quantization_config():
return mtq_config
-def get_calib_dataloader(calib_size=512, max_sequence_length=512):
- """Return a dataloader for calibration."""
- args = get_args()
- dataset = load_dataset(args.calib_dataset, name="3.0.0", split="train")
- text_column = "article"
+def get_calib_dataloader(
+ dataset_path_or_name,
+ tokenizer,
+ calib_size=512,
+ max_sequence_length=512,
+ use_random_offset=False,
+ batch_size=1,
+):
+ """Return a dataloader/iterator for calibration using SFT or HF datasets.
- calib_size = min(len(dataset), calib_size)
- for i in range(calib_size):
- yield dataset[i][text_column][:max_sequence_length]
+ Supports either a local path (.jsonl) or a HuggingFace dataset name.
+ """
+ if os.path.isfile(dataset_path_or_name):
+ # Local file
+ print_rank_0(f"Loading calibration dataset from local file: {dataset_path_or_name}")
+ all_texts = []
+ with open(dataset_path_or_name) as f:
+ for i, line in enumerate(f):
+ if len(all_texts) == calib_size:
+ break
+ sample = json.loads(line)
+
+ # Extract text field from various possible keys
+ if isinstance(sample, dict) and "text" in sample:
+ if not sample["text"]:
+ warnings.warn(f"Sample {i} has empty text, skipping")
+ continue
+ full_text = sample["text"]
+ elif isinstance(sample, dict) and "messages" in sample:
+ conversations = sample["messages"]
+ assert "role" in conversations[0] and "content" in conversations[0]
+ full_text = "".join([f"{msg['role']}: {msg['content']}" for msg in conversations])
+ elif isinstance(sample, list) and isinstance(sample[0], dict):
+ assert "role" in sample[0] and "content" in sample[0]
+ full_text = "".join([f"{msg['role']}: {msg['content']}" for msg in sample])
+ else:
+ raise ValueError(f"Sample {i} has unexpected format")
+
+ # Slice text
+ max_text_length = int(max_sequence_length / 0.75) # tokenized text is roughtly ~75% length of original
+ start_idx = 0
+ if use_random_offset and len(full_text) > max_text_length:
+ start_idx = random.randint(0, len(full_text) - max_text_length)
+ text = full_text[start_idx : start_idx + max_text_length]
+ all_texts.append(text)
+
+ print_rank_0(f"Loaded calibration dataset ({dataset_path_or_name}) with {len(all_texts)} samples")
+ print_rank_0(f"Actual num samples: {len(all_texts)}, max seq length: {max_sequence_length}")
+ print_rank_0(f"Sampling Strategy: {'Random Index' if use_random_offset else 'From Beginning'}")
+
+ # Tokenize all texts at once and move to device
+ tokens = tokenizer(
+ all_texts, return_tensors="pt", padding="max_length", max_length=max_sequence_length, truncation=True
+ )
+ all_input_ids = tokens.input_ids.cuda()
+ return [{"input_ids": all_input_ids[i:i+batch_size]} for i in range(0, len(all_input_ids), batch_size)]
+ else:
+ # HuggingFace dataset
+ if use_random_offset:
+ warnings.warn("Random offset is not supported for HuggingFace datasets.")
+ print_rank_0(f"Loading calibration dataset from HuggingFace: {dataset_path_or_name}")
+ return get_dataset_dataloader(
+ dataset_name=dataset_path_or_name,
+ tokenizer=tokenizer,
+ num_samples=calib_size,
+ max_sample_length=max_sequence_length,
+ batch_size=batch_size,
+ device="cuda",
+ )
if __name__ == "__main__":
@@ -276,7 +359,8 @@ def get_calib_dataloader(calib_size=512, max_sequence_length=512):
args = get_args()
- tokenizer = get_tokenizer()._tokenizer
+ tokenizer = get_hf_tokenizer()
+
model = get_model(
functools.partial(model_provider, modelopt_gpt_mamba_builder), wrap_with_ddp=False
)
@@ -293,8 +377,9 @@ def get_calib_dataloader(calib_size=512, max_sequence_length=512):
import_dtype = torch.float16 if args.fp16 else torch.bfloat16
unwrapped_model = unwrap_model(model)[0]
workspace_dir = os.environ.get("MLM_WORK_DIR", "/tmp")
+
import_kwargs = {"dtype": import_dtype}
- if modelopt_version_at_least("0.41.0"):
+ if "trust_remote_code" in inspect.signature(import_mcore_gpt_from_hf).parameters:
import_kwargs.update({"trust_remote_code": args.trust_remote_code})
import_mcore_gpt_from_hf(
unwrapped_model, args.pretrained_model_path, workspace_dir, **import_kwargs
@@ -315,12 +400,18 @@ def _custom_prompt_forward_loop_func(model):
if all_references[idx] is not None:
assert all_references[idx] == generated_texts[0], all_references[idx]
- def _hf_dataset_forword_loop_func(model):
- dataloader = get_calib_dataloader(args.calib_size)
-
- for prompt in tqdm(dataloader, total=args.calib_size, disable=torch.distributed.get_rank()):
- tokens = tokenizer(prompt, return_tensors="pt")
- generated_ids = simple_generate(model, tokens.input_ids.cuda(), osl=1)
+ def _dataset_forward_loop_func(model):
+ dataloader = get_calib_dataloader(
+ dataset_path_or_name=args.calib_dataset_path_or_name,
+ tokenizer=tokenizer,
+ calib_size=args.calib_size,
+ max_sequence_length=args.calib_max_sequence_length,
+ use_random_offset=args.calib_use_random_offset,
+ batch_size=args.calib_batch_size,
+ )
+ for sample in tqdm(dataloader, disable=torch.distributed.get_rank()):
+ sample = get_batch_on_this_cp_rank(sample)
+ simple_generate(model, sample["input_ids"], osl=1, calibration_mode=True)
unwrapped_model = unwrap_model(model)[0]
@@ -330,20 +421,21 @@ def _hf_dataset_forword_loop_func(model):
)
if args.export_quant_cfg is not None:
- if args.export_quant_cfg not in QUANT_CFG_CHOICES:
- raise ValueError(f"Unsupported quantization config {args.export_quant_cfg}.")
print_rank_0("Quantizing the model...")
mtq_config = get_modelopt_torch_quantization_config()
- ptq_forward_loop_func = _hf_dataset_forword_loop_func
+
+ if not hasattr(tokenizer, "pad_token") or tokenizer.pad_token is None:
+ tokenizer.pad_token = tokenizer.eos_token
+ tokenizer.padding_side = "left" # better for calibration
if args.weight_only:
mtq.quantize(unwrapped_model, mtq_config)
elif hasattr(unwrapped_model, "calibration_mode"):
unwrapped_model.calibration_mode = True
- mtq.quantize(unwrapped_model, mtq_config, ptq_forward_loop_func)
+ mtq.quantize(unwrapped_model, mtq_config, _dataset_forward_loop_func)
unwrapped_model.calibration_mode = False
else:
- mtq.quantize(unwrapped_model, mtq_config, ptq_forward_loop_func)
+ mtq.quantize(unwrapped_model, mtq_config, _dataset_forward_loop_func)
if args.compress:
mtq.compress(unwrapped_model)
@@ -351,7 +443,14 @@ def _hf_dataset_forword_loop_func(model):
print_distributed_quant_summary(model, "Quantized Model:")
- _custom_prompt_forward_loop_func(unwrapped_model)
-
if args.save is not None:
save_checkpoint(1, model, None, None, 0, release=True)
+
+ # Free calibration/quantization memory before generate
+ import gc
+ gc.collect()
+ torch.cuda.empty_cache()
+
+ # Do this after saving in case it causes issues
+ if not args.skip_generate:
+ _custom_prompt_forward_loop_func(unwrapped_model)
diff --git a/examples/post_training/modelopt/quantize.sh b/examples/post_training/modelopt/quantize.sh
index 563e532a12f..9119ff4ae76 100755
--- a/examples/post_training/modelopt/quantize.sh
+++ b/examples/post_training/modelopt/quantize.sh
@@ -16,7 +16,7 @@ MLM_DEFAULT_ARGS="
QUANT_CFG=$2
if [ -z ${QUANT_CFG} ]; then
- QUANT_CFG=fp8
+ QUANT_CFG='FP8_DEFAULT_CFG'
printf "${MLM_WARNING} Variable ${PURPLE}QUANT_CFG${WHITE} is not set (default: ${QUANT_CFG})!\n"
fi
@@ -25,6 +25,11 @@ if [ -z ${MLM_MODEL_SAVE} ]; then
printf "${MLM_WARNING} Variable ${PURPLE}MLM_MODEL_SAVE${WHITE} is not set (default: ${MLM_MODEL_SAVE})!\n"
fi
+EXTRA_ARGS=(${MLM_DEFAULT_ARGS} ${MLM_EXTRA_ARGS})
+if [ -n "${MLM_PROMPTS}" ]; then
+ EXTRA_ARGS+=(--prompts "${MLM_PROMPTS}")
+fi
+
if [ -z ${MLM_MODEL_CKPT} ]; then
${LAUNCH_SCRIPT} ${SCRIPT_DIR}/quantize.py \
${MODEL_ARGS} \
@@ -32,12 +37,13 @@ if [ -z ${MLM_MODEL_CKPT} ]; then
--expert-tensor-parallel-size ${ETP} \
--expert-model-parallel-size ${EP} \
--pipeline-model-parallel-size ${PP} \
+ --context-parallel-size ${CP} \
--tokenizer-model ${TOKENIZER_MODEL} \
--pretrained-model-path ${HF_MODEL_CKPT} \
--save ${MLM_MODEL_SAVE} \
--export-quant-cfg ${QUANT_CFG} \
--references "${MLM_REF_LABEL}" \
- ${MLM_DEFAULT_ARGS} ${MLM_EXTRA_ARGS}
+ "${EXTRA_ARGS[@]}"
else
${LAUNCH_SCRIPT} ${SCRIPT_DIR}/quantize.py \
${MODEL_ARGS} \
@@ -45,10 +51,11 @@ else
--expert-tensor-parallel-size ${ETP} \
--expert-model-parallel-size ${EP} \
--pipeline-model-parallel-size ${PP} \
+ --context-parallel-size ${CP} \
--tokenizer-model ${TOKENIZER_MODEL} \
--load ${MLM_MODEL_CKPT} \
--save ${MLM_MODEL_SAVE} \
--export-quant-cfg ${QUANT_CFG} \
--references "${MLM_REF_LABEL}" \
- ${MLM_DEFAULT_ARGS} ${MLM_EXTRA_ARGS}
+ "${EXTRA_ARGS[@]}"
fi
diff --git a/examples/post_training/modelopt/train.sh b/examples/post_training/modelopt/train.sh
new file mode 100755
index 00000000000..1ebb8bf3d76
--- /dev/null
+++ b/examples/post_training/modelopt/train.sh
@@ -0,0 +1,93 @@
+#!/bin/bash
+
+SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
+
+# Common arguments and base model specific arguments
+source "${SCRIPT_DIR}/conf/arguments.sh"
+
+
+# Set up cache dir for HF to avoid out of space error
+export HF_DATASETS_CACHE="/tmp/hf_datasets_cache"
+
+# Extra arguments of this script
+MLM_DEFAULT_ARGS=" \
+ --modelopt-enabled \
+ --distributed-timeout-minutes 60 \
+ --auto-detect-ckpt-format \
+ --export-te-mcore-model \
+"
+
+
+if [ -z ${MLM_MODEL_SAVE} ]; then
+ MLM_MODEL_SAVE=${MLM_MODEL_CKPT}
+ printf "${MLM_WARNING} Variable ${PURPLE}MLM_MODEL_SAVE${WHITE} is not set (default: ${MLM_MODEL_CKPT})!\n"
+fi
+
+if [ -z ${MLM_DATA_ARGS} ]; then
+ MLM_DATA_ARGS=" \
+ --train-samples 128000 \
+ --lr-decay-samples 128000 \
+ --lr-warmup-samples 0 \
+ --sft \
+ --tokenizer-type SFTTokenizer \
+ --per-split-data-args-path ${BLEND_PATH} \
+ "
+fi
+
+if [ -z ${MLM_TRAIN_ARGS} ]; then
+ MLM_TRAIN_ARGS=" \
+ --no-gradient-accumulation-fusion \
+ --micro-batch-size 1 \
+ --attention-dropout 0.0 \
+ --hidden-dropout 0.0 \
+ --no-check-for-nan-in-loss-and-grad \
+ "
+fi
+
+if [ -z ${MLM_OPTIM_ARGS} ]; then
+ MLM_OPTIM_ARGS=" \
+ --lr 5.0e-5 \
+ --min-lr 1.0e-7 \
+ --lr-decay-style cosine \
+ --clip-grad 1.0 \
+ --weight-decay 0.0 \
+ --adam-beta1 0.9 \
+ --adam-beta2 0.95 \
+ --init-method-std 0.010 \
+ --use-distributed-optimizer \
+ "
+fi
+
+if [ -z ${MLM_EVAL_ARGS} ]; then
+ MLM_EVAL_ARGS=" \
+ --eval-iters 1 \
+ --eval-interval 1000 \
+ --save-interval 1000 \
+ --log-interval 100 \
+ "
+fi
+
+export HF_TOKEN=${HF_TOKEN}
+
+if [[ ${MODEL_ARGS} == *"MambaModel"* ]]; then
+ PRETRAIN_EXE=${SCRIPT_DIR}/../../../pretrain_mamba.py
+else
+ PRETRAIN_EXE=${SCRIPT_DIR}/../../../pretrain_gpt.py
+fi
+
+${LAUNCH_SCRIPT} ${PRETRAIN_EXE} \
+ ${MODEL_ARGS} \
+ --tensor-model-parallel-size ${TP} \
+ --expert-tensor-parallel-size ${ETP} \
+ --expert-model-parallel-size ${EP} \
+ --pipeline-model-parallel-size ${PP} \
+ --context-parallel-size ${CP} \
+ --tokenizer-model ${TOKENIZER_MODEL} \
+ --load ${MLM_MODEL_CKPT} \
+ --save ${MLM_MODEL_SAVE} \
+ ${MLM_DATA_ARGS} \
+ ${MLM_OPTIM_ARGS} \
+ ${MLM_TRAIN_ARGS} \
+ ${MLM_EVAL_ARGS} \
+ ${MLM_RESUME_ARGS} \
+ ${MLM_DEFAULT_ARGS} ${MLM_EXTRA_ARGS}
diff --git a/examples/post_training/modelopt/utils.py b/examples/post_training/modelopt/utils.py
new file mode 100644
index 00000000000..fd554caa6d8
--- /dev/null
+++ b/examples/post_training/modelopt/utils.py
@@ -0,0 +1,25 @@
+# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
+
+"""Shared utilities for modelopt post-training scripts."""
+import os
+import sys
+
+sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "../../../")))
+
+from megatron.training import get_tokenizer
+
+
+def get_hf_tokenizer():
+ """Return the underlying HuggingFace tokenizer, unwrapping Megatron-Core nesting.
+
+ Megatron-Core tokenizers are nested (e.g. get_tokenizer()._tokenizer may itself
+ have a .tokenizer or ._tokenizer attribute holding the actual HF tokenizer).
+ This helper unwraps one level of that nesting.
+ """
+ tokenizer = get_tokenizer()._tokenizer
+ tok_attrs = ["tokenizer", "_tokenizer"]
+ for attr in tok_attrs:
+ if hasattr(tokenizer, attr):
+ tokenizer = getattr(tokenizer, attr)
+ break
+ return tokenizer
diff --git a/examples/post_training/modelopt/validate.py b/examples/post_training/modelopt/validate.py
index ee8bf64eccb..8b8f1ffc9dd 100644
--- a/examples/post_training/modelopt/validate.py
+++ b/examples/post_training/modelopt/validate.py
@@ -16,7 +16,8 @@
from megatron.post_training.checkpointing import load_modelopt_checkpoint
from megatron.post_training.model_builder import modelopt_gpt_mamba_builder
from megatron.post_training.utils import get_mtbench_chat_data
-from megatron.training import get_args, get_model, get_tokenizer, initialize_megatron
+from megatron.training import get_args, get_model, initialize_megatron
+from utils import get_hf_tokenizer
from megatron.training.utils import print_rank_0, unwrap_model
from model_provider import model_provider
@@ -114,7 +115,7 @@ def report_current_memory_info():
else:
ground_truth = [None for _ in range(len(prompts))]
- tokenizer = get_tokenizer()._tokenizer
+ tokenizer = get_hf_tokenizer()
model = get_model(functools.partial(model_provider, modelopt_gpt_mamba_builder), wrap_with_ddp=False)
report_current_memory_info()
diff --git a/examples/rl/README.md b/examples/rl/README.md
index 34b6fafa517..b66b2abbc32 100644
--- a/examples/rl/README.md
+++ b/examples/rl/README.md
@@ -94,7 +94,7 @@ MODEL_OPTIONS="\
--ckpt-format torch \
--seq-length $MAX_SEQ_LENGTH \
--inference-max-seq-length $MAX_SEQ_LENGTH \
- --inference-max-batch-size $MAX_INFERENCE_BS \
+ --inference-max-requests $MAX_INFERENCE_BS \
--pretrained-checkpoint $CHECKPOINT \
--untie-embeddings-and-output-weights \
--disable-bias-linear \
@@ -172,7 +172,6 @@ torchrun \
--save $CHECKPOINT_DIR \
--load $CHECKPOINT_DIR \
--tensorboard-dir $TB_DIR \
- --langrl-inference-server-type inplace_megatron \
--seed $SEED \
--sequence-parallel \
--finetune \
diff --git a/examples/rl/benchmark_refit.py b/examples/rl/benchmark_refit.py
new file mode 100644
index 00000000000..645cd9e2c15
--- /dev/null
+++ b/examples/rl/benchmark_refit.py
@@ -0,0 +1,345 @@
+#!/usr/bin/env python3
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+"""
+Benchmark script for model refit performance.
+
+Measures the time to transfer model weights between different parallelism configurations.
+Supports both collocated (models share GPUs) and non-collocated (separate GPU sets) modes.
+"""
+import time
+
+import torch
+
+from megatron.core.resharding.refit import swap_model_weights
+from megatron.training import get_args, get_model as get_training_model, print_rank_0
+from megatron.training.initialize import initialize_megatron
+from megatron.training.arguments import core_transformer_config_from_args
+from megatron.rl.parallel_utils import build_inference_pg_collection
+from gpt_builders import gpt_builder
+from megatron.core.resharding.copy_services.nvshmem_copy_service import NVSHMEMCopyService
+from megatron.core.resharding.copy_services.nccl_copy_service import NCCLCopyService
+from megatron.core.resharding.copy_services.gloo_copy_service import GlooCopyService
+
+
+def add_benchmark_args(parser):
+ """Add benchmark-specific arguments."""
+ group = parser.add_argument_group(title='refit benchmark')
+
+ group.add_argument(
+ '--refit-mode',
+ type=str,
+ required=True,
+ choices=['collocated', 'non-collocated'],
+ help='Collocated: both models share GPUs. Non-collocated: separate GPU sets.'
+ )
+ group.add_argument(
+ '--num-benchmark-warmup',
+ type=int,
+ default=2,
+ help='Number of warmup iterations (first builds refit plan).'
+ )
+ group.add_argument(
+ '--num-benchmark-iterations',
+ type=int,
+ default=10,
+ help='Number of timed benchmark iterations.'
+ )
+
+ return parser
+
+
+def model_provider(pre_process=True, post_process=True, parallel_output=False,
+ pg_collection=None, config=None):
+ """Build the model."""
+ args = get_args()
+ if config is None:
+ config = core_transformer_config_from_args(args)
+
+ return gpt_builder(
+ args=args,
+ pre_process=pre_process,
+ post_process=post_process,
+ config=config,
+ pg_collection=pg_collection,
+ )
+
+
+def create_refit_service(method):
+ """Create and return a refit service instance."""
+ if method == 'nvshmem':
+ return NVSHMEMCopyService()
+ elif method == 'nccl':
+ return NCCLCopyService()
+ elif method == 'gloo':
+ return GlooCopyService()
+ else:
+ return method
+
+
+def print_config_summary(args, src_config, dst_config, world_size, mode):
+ """Print benchmark configuration."""
+ print_rank_0(f"\n{'='*80}")
+ print_rank_0(f"REFIT BENCHMARK - {mode.upper()} MODE")
+ print_rank_0(f"{'='*80}")
+ print_rank_0(f"World size: {world_size}")
+ print_rank_0(f"Source: TP={src_config['tp']}, PP={src_config['pp']}, EP={src_config['ep']}, DP={src_config['dp']}")
+ print_rank_0(f"Destination: TP={dst_config['tp']}, PP={dst_config['pp']}, EP={dst_config['ep']}, DP={dst_config['dp']}")
+ print_rank_0(f"Model: {args.num_layers}L, {args.hidden_size}H, {args.num_attention_heads} heads, vocab={args.vocab_size}")
+ if args.num_experts:
+ print_rank_0(f"MoE: {args.num_experts} experts, top-{args.moe_router_topk}")
+ print_rank_0(f"Backend: {args.refit_method}")
+ print_rank_0(f"{'='*80}\n")
+
+
+def run_benchmark(src_model, dst_model, refit_service, num_warmup, num_iterations):
+ """Run warmup and benchmark iterations, return timings."""
+ rank = torch.distributed.get_rank()
+
+ # Warmup (builds refit plan on first iteration)
+ print_rank_0(f"Warmup: {num_warmup} iterations...")
+ for i in range(num_warmup):
+ torch.cuda.synchronize()
+ torch.distributed.barrier()
+ swap_model_weights(src_model, dst_model, refit_method=refit_service)
+ torch.cuda.synchronize()
+ torch.distributed.barrier()
+
+ print_rank_0("Warmup complete. Starting benchmark...\n")
+
+ # Benchmark iterations
+ print_rank_0(f"Benchmark: {num_iterations} iterations...")
+ timings = []
+
+ for i in range(num_iterations):
+ torch.cuda.synchronize()
+ torch.distributed.barrier()
+
+ start_time = time.perf_counter()
+ swap_model_weights(src_model, dst_model, refit_method=refit_service)
+ torch.cuda.synchronize()
+ end_time = time.perf_counter()
+
+ elapsed = end_time - start_time
+ timings.append(elapsed)
+ torch.distributed.barrier()
+
+ return timings
+
+
+def print_results(timings):
+ """Print benchmark results."""
+ if torch.distributed.get_rank() == 0:
+ mean_time = sum(timings) / len(timings)
+ min_time = min(timings)
+ max_time = max(timings)
+
+ print(f"\n{'='*80}")
+ print("RESULTS")
+ print(f"{'='*80}")
+ print(f"Mean: {mean_time*1000:.2f} ms")
+ print(f"Min: {min_time*1000:.2f} ms")
+ print(f"Max: {max_time*1000:.2f} ms")
+ print(f"{'='*80}\n")
+
+
+def benchmark_collocated():
+ """Benchmark refit in collocated mode (both models on same GPUs)."""
+ args = get_args()
+ world_size = torch.distributed.get_world_size()
+
+ # Calculate parallelism
+ src_tp = args.tensor_model_parallel_size
+ src_pp = args.pipeline_model_parallel_size
+ src_ep = args.expert_model_parallel_size
+ src_world = src_tp * src_pp * src_ep
+ src_dp = world_size // src_world
+
+ dst_tp = args.rl_inference_tensor_model_parallel_size or src_tp
+ dst_pp = args.rl_inference_pipeline_model_parallel_size or src_pp
+ dst_ep = args.rl_inference_expert_model_parallel_size or src_ep
+ dst_world = dst_tp * dst_pp * dst_ep
+ dst_dp = world_size // dst_world
+
+ # Print config
+ src_config = {'tp': src_tp, 'pp': src_pp, 'ep': src_ep, 'dp': src_dp}
+ dst_config = {'tp': dst_tp, 'pp': dst_pp, 'ep': dst_ep, 'dp': dst_dp}
+ print_config_summary(args, src_config, dst_config, world_size, 'collocated')
+
+ # Build source model
+ print_rank_0("Building source model...")
+ src_model = get_training_model(
+ lambda pre_process, post_process, **kwargs: model_provider(
+ pre_process=pre_process, post_process=post_process, parallel_output=False
+ ),
+ wrap_with_ddp=False
+ )
+ src_model[0] = src_model[0].cuda()
+
+ # Build destination model with custom parallelism
+ print_rank_0("Building destination model...")
+ dst_pg_collection = build_inference_pg_collection(
+ world_size,
+ tp_size=dst_tp,
+ pp_size=dst_pp,
+ ep_size=dst_ep,
+ expt_tp_size=args.rl_inference_expert_tensor_model_parallel_size,
+ use_tp_pp_dp_mapping=args.use_tp_pp_dp_mapping,
+ )
+
+ dst_config = core_transformer_config_from_args(args)
+ if args.num_experts:
+ dst_config.expert_model_parallel_size = dst_ep
+ dst_config.tensor_model_parallel_size = dst_tp
+ if args.rl_inference_expert_tensor_model_parallel_size:
+ dst_config.expert_tensor_parallel_size = args.rl_inference_expert_tensor_model_parallel_size
+
+ dst_model = get_training_model(
+ lambda pre_process, post_process, **kwargs: model_provider(
+ pre_process=pre_process, post_process=post_process,
+ pg_collection=dst_pg_collection, config=dst_config
+ ),
+ wrap_with_ddp=False
+ )
+ dst_model[0] = dst_model[0].cuda()
+
+ torch.distributed.barrier()
+
+ # Create refit service
+ print_rank_0(f"Creating {args.refit_method} service...")
+ refit_service = create_refit_service(args.refit_method)
+ print_rank_0("Service created.\n")
+
+ # Run benchmark
+ timings = run_benchmark(src_model, dst_model, refit_service,
+ args.num_benchmark_warmup, args.num_benchmark_iterations)
+
+ # Print results
+ print_results(timings)
+
+
+def benchmark_non_collocated():
+ """Benchmark refit in non-collocated mode (separate GPU sets)."""
+ args = get_args()
+ rank = torch.distributed.get_rank()
+ world_size = torch.distributed.get_world_size()
+
+ # Calculate parallelism
+ src_tp = args.tensor_model_parallel_size
+ src_pp = args.pipeline_model_parallel_size
+ src_ep = args.expert_model_parallel_size
+ src_world = src_tp * src_pp * src_ep
+
+ dst_tp = args.rl_inference_tensor_model_parallel_size or src_tp
+ dst_pp = args.rl_inference_pipeline_model_parallel_size or src_pp
+ dst_ep = args.rl_inference_expert_model_parallel_size or src_ep
+ dst_world = dst_tp * dst_pp * dst_ep
+
+ required_size = src_world + dst_world
+ if world_size < required_size:
+ raise ValueError(f"Non-collocated requires {required_size} GPUs, got {world_size}")
+
+ # Determine rank roles
+ is_src_rank = rank < src_world
+ is_dst_rank = src_world <= rank < required_size
+ is_idle_rank = rank >= required_size
+
+ # Print config
+ src_config = {'tp': src_tp, 'pp': src_pp, 'ep': src_ep, 'dp': 1}
+ dst_config = {'tp': dst_tp, 'pp': dst_pp, 'ep': dst_ep, 'dp': 1}
+ print_config_summary(args, src_config, dst_config, world_size, 'non-collocated')
+ if world_size > required_size:
+ print_rank_0(f"Note: Ranks {required_size}-{world_size-1} are idle\n")
+
+ # Create destination process groups (all ranks participate)
+ print_rank_0("Creating process groups...")
+ dst_pg_collection = build_inference_pg_collection(
+ world_size=dst_world,
+ tp_size=dst_tp,
+ pp_size=dst_pp,
+ ep_size=dst_ep,
+ expt_tp_size=args.rl_inference_expert_tensor_model_parallel_size,
+ use_tp_pp_dp_mapping=args.use_tp_pp_dp_mapping,
+ rank_offset=src_world,
+ )
+ torch.distributed.barrier()
+
+ # Idle ranks participate in collectives but have no models
+ if is_idle_rank:
+ src_model = None
+ dst_model = None
+ elif is_src_rank:
+ # Build source model
+ print_rank_0("Building source model...")
+ src_model = get_training_model(
+ lambda pre_process, post_process, **kwargs: model_provider(
+ pre_process=pre_process, post_process=post_process, parallel_output=False
+ ),
+ wrap_with_ddp=False
+ )
+ src_model[0] = src_model[0].cuda()
+ dst_model = None
+ else: # is_dst_rank
+ # Build destination model
+ print_rank_0("Building destination model...")
+ dst_config = core_transformer_config_from_args(args)
+ if args.num_experts:
+ dst_config.expert_model_parallel_size = dst_ep
+ dst_config.tensor_model_parallel_size = dst_tp
+ if args.rl_inference_expert_tensor_model_parallel_size:
+ dst_config.expert_tensor_parallel_size = args.rl_inference_expert_tensor_model_parallel_size
+
+ dst_model = get_training_model(
+ lambda pre_process, post_process, **kwargs: model_provider(
+ pre_process=pre_process, post_process=post_process,
+ pg_collection=dst_pg_collection, config=dst_config
+ ),
+ wrap_with_ddp=False
+ )
+ dst_model[0] = dst_model[0].cuda()
+ src_model = None
+
+ torch.distributed.barrier()
+
+ # Create refit service
+ print_rank_0(f"Creating {args.refit_method} service...")
+ refit_service = create_refit_service(args.refit_method)
+ print_rank_0("Service created.\n")
+
+ # Run benchmark
+ timings = run_benchmark(src_model, dst_model, refit_service,
+ args.num_benchmark_warmup, args.num_benchmark_iterations)
+
+ # Print results
+ print_results(timings)
+
+
+def main():
+ """Main benchmark function."""
+ initialize_megatron(
+ extra_args_provider=add_benchmark_args,
+ args_defaults={
+ 'tokenizer_type': 'NullTokenizer',
+ 'no_load_optim': True,
+ 'no_load_rng': True,
+ 'no_save_optim': True,
+ 'no_save_rng': True,
+ },
+ ignore_unknown_args=False,
+ )
+
+ args = get_args()
+
+ # Set default vocab size if not provided
+ if args.vocab_size is None:
+ args.vocab_size = 50257
+ print_rank_0("Using default vocab_size=50257")
+
+ # Run benchmark
+ if args.refit_mode == 'collocated':
+ benchmark_collocated()
+ else:
+ benchmark_non_collocated()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/examples/rl/environment_configs/gsm8k_nanov3.yaml b/examples/rl/environment_configs/gsm8k_nanov3.yaml
index 30403ed052b..b759423ee5b 100644
--- a/examples/rl/environment_configs/gsm8k_nanov3.yaml
+++ b/examples/rl/environment_configs/gsm8k_nanov3.yaml
@@ -2,8 +2,6 @@
agent_args:
answer_format: "boxed"
format_reward: 0.5
- assistant_suffix: "Assistant: "
- chat_mode: true
negative_reward: 0.0
partial_end_reward: 0.75
weight: 1.0
diff --git a/examples/rl/environments/countdown/countdown_agent.py b/examples/rl/environments/countdown/countdown_agent.py
index bd9413a19d0..e14ac6c6d7e 100644
--- a/examples/rl/environments/countdown/countdown_agent.py
+++ b/examples/rl/environments/countdown/countdown_agent.py
@@ -10,17 +10,11 @@
class CountdownAgent(RewardOnlyAgent, HFDatasetAgent):
+ env_id: str = "countdown"
def make_prefix(self, target, nums) -> str:
- if self.chat_mode:
- prefix = f"""Using the numbers {nums}, create an equation that equals {target}. You can use basic arithmetic operations (+, -, *, /) and each number can only be used once.
+ prefix = f"""Using the numbers {nums}, create an equation that equals {target}. You can use basic arithmetic operations (+, -, *, /) and each number can only be used once.
Return the final answer in tags, for example (1 + 2) / 3 . Do not include an = sign."""
- else:
- prefix = f"""A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer.
- User: Using the numbers {nums}, create an equation that equals {target}. You can use basic arithmetic operations (+, -, *, /) and each number can only be used once. Show your work in tags.
- And return the final answer in tags, for example (1 + 2) / 3 . Do not include an = sign.
- Assistant: Let me solve this step by step.
- """
return prefix
def get_dataset(self, validation: bool = False):
diff --git a/examples/rl/environments/math/gsm8k_agent.py b/examples/rl/environments/math/gsm8k_agent.py
index 3bb39bc09f9..6cdfb4f926e 100644
--- a/examples/rl/environments/math/gsm8k_agent.py
+++ b/examples/rl/environments/math/gsm8k_agent.py
@@ -25,16 +25,12 @@
class GSM8KAgent(MathAgent):
def __init__(self,
answer_format: str = "boxed",
- chat_mode: bool = False,
- assistant_suffix: str = "Assistant: Let me solve this step by step.\n",
format_reward: float = 0.0,
negative_reward: float = 0.0,
partial_end_reward: float = 0.0,
**kwargs):
super().__init__(
answer_format=answer_format,
- chat_mode=chat_mode,
- assistant_suffix=assistant_suffix,
format_reward=format_reward,
negative_reward=negative_reward,
partial_end_reward=partial_end_reward,
diff --git a/examples/rl/environments/math/math_agent.py b/examples/rl/environments/math/math_agent.py
index 67feb3b4adb..0747254d1ad 100644
--- a/examples/rl/environments/math/math_agent.py
+++ b/examples/rl/environments/math/math_agent.py
@@ -3,7 +3,6 @@
import re
import traceback
-from megatron.rl.agent.pass_at_evaluation_agent import PassAtEvaluationAgent
from megatron.rl.agent.reward_only_agent import RewardOnlyAgent
try:
@@ -25,8 +24,6 @@ class MathAgent(RewardOnlyAgent):
def __init__(self,
format_reward: float = 0.0,
answer_format: str = "tagged",
- assistant_suffix: str = "Assistant: Let me solve this step by step.\n",
- chat_mode: bool = False,
negative_reward: float = 0.0,
partial_end_reward: float = 0.0,
**kwargs):
@@ -36,9 +33,6 @@ def __init__(self,
even if the answer is incorrect or is missing the end-of-text token.
answer_format (str): Which answer format is expected: "tagged" for tags,
or "boxed" for \boxed{} LaTeX formatting.
- assistant_suffix (str): The suffix string included in the assistant's response, typically to
- guide the assistant's output format and "persona". For example, "Let me solve this step by step."
- chat_mode (bool): If True, agent operates in a chat (conversational) context.
negative_reward (float): Reward assigned for a clearly incorrect or unparseable answer.
partial_end_reward (float): Reward when the answer is correct but an expected end token is not matched exactly.
**kwargs: Additional arguments for the base RewardOnlyAgent.
@@ -49,8 +43,6 @@ def __init__(self,
self.format_reward = format_reward
self.answer_format = answer_format
- self.assistant_suffix = assistant_suffix
- self.chat_mode = chat_mode
self.negative_reward = negative_reward
self.partial_end_reward = partial_end_reward
@@ -61,7 +53,7 @@ def compute_score(self, response: str, golden: dict, golden_key: str = "answer")
"""
# Allow tags or \boxed{} tags (this is a bit of cheating in favor of deepseek distilled models I think)
matched_format = None
- end_tokens = ["<|end_of_text|>", "<|endoftext|>", ""]
+ end_tokens = ["<|end_of_text|>", "<|endoftext|>", "", "<|eot_id|>", "<|im_end|>"]
# Only an answer immediately followed by a known end token yields 1.0 reward.
answer_tag_pattern = r'(.*?) '
@@ -134,12 +126,6 @@ def make_prefix(self, problem_key: str = "problem", **kwargs) -> str:
else:
raise ValueError(f"Invalid answer format: {self.answer_format}")
- if self.chat_mode:
- prefix = f"""{kwargs[problem_key]}\n{answer_format}"""
- else:
- prefix = f"""A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer.
- The question will be a word math problem. Show your work in tags.
- {answer_format}
- User: {kwargs[problem_key]}
- {self.assistant_suffix}"""
+ prefix = f"""{kwargs[problem_key]}\n{answer_format}"""
+
return prefix
diff --git a/examples/rl/model_configs/common.sh b/examples/rl/model_configs/common.sh
index 4f6ca0e18cf..389880d6289 100644
--- a/examples/rl/model_configs/common.sh
+++ b/examples/rl/model_configs/common.sh
@@ -16,13 +16,12 @@ COMMON_OPTIONS="\
--transformer-impl transformer_engine \
--${PRECISION:-bf16} \
--te-rng-tracker \
- --rl-offload-optimizer-during-inference \
--inference-dynamic-batching-buffer-size-gb 20 \
--data-parallel-random-init \
--attention-backend flash \
--timing-log-level 1 \
--log-timers-to-tensorboard \
- --save-retain-interval 120 \
+ --save-retain-interval 160 \
--inference-dynamic-batching-num-cuda-graphs 1 \
--inference-dynamic-batching-unified-memory-level 1 \
--adam-beta1 0.9 \
@@ -39,5 +38,5 @@ else
fi
if [ ${ENABLE_CUDA_GRAPH:-true} == true ]; then
- COMMON_OPTIONS="${COMMON_OPTIONS} --cuda-graph-impl=local"
+ COMMON_OPTIONS="${COMMON_OPTIONS} --cuda-graph-impl=local --rl-persist-cuda-graphs"
fi
diff --git a/examples/rl/model_configs/llama3p1_8b_instruct.sh b/examples/rl/model_configs/llama3p1_8b_instruct.sh
index 24d285a6cf7..ff3b5327710 100644
--- a/examples/rl/model_configs/llama3p1_8b_instruct.sh
+++ b/examples/rl/model_configs/llama3p1_8b_instruct.sh
@@ -77,7 +77,7 @@ MODEL_OPTIONS="\
--ckpt-format torch_dist \
--seq-length $MAX_SEQ_LENGTH \
--inference-max-seq-length $MAX_SEQ_LENGTH \
- --inference-max-batch-size $MAX_INFERENCE_BS \
+ --inference-max-requests $MAX_INFERENCE_BS \
--pretrained-checkpoint $CHECKPOINT \
--add-qkv-bias \
--normalization RMSNorm \
@@ -101,8 +101,8 @@ MODEL_OPTIONS="\
--max-position-embeddings 131072 \
--tokenizer-type HuggingFaceTokenizer \
--tokenizer-model unsloth/Meta-Llama-3.1-8B-Instruct \
- --langrl-inference-server-type "inplace_megatron_chat" \
- --langrl-inference-server-conversation-template "unsloth/Meta-Llama-3.1-8B-Instruct" \
+ --tokenizer-hf-use-fast \
+ --tokenizer-hf-include-special-tokens \
--lr 3e-7 \
--make-vocab-size-divisible-by 128 \
--clip-grad 1.0 \
diff --git a/examples/rl/model_configs/nemotron5_56b.sh b/examples/rl/model_configs/nemotron5_56b.sh
index fd2cc4f7212..23b9f99a72a 100644
--- a/examples/rl/model_configs/nemotron5_56b.sh
+++ b/examples/rl/model_configs/nemotron5_56b.sh
@@ -58,7 +58,7 @@ MODEL_OPTIONS="\
--calculate-per-token-loss \
--seq-length $MAX_SEQ_LENGTH \
--inference-max-seq-length $MAX_SEQ_LENGTH \
- --inference-max-batch-size $MAX_INFERENCE_BS \
+ --inference-max-requests $MAX_INFERENCE_BS \
--pretrained-checkpoint $CHECKPOINT \
--fp8-format hybrid \
--fp8-amax-history-len 1 \
@@ -68,7 +68,7 @@ MODEL_OPTIONS="\
--first-last-layers-bf16 \
\
--fp8-recipe tensorwise \
- --hybrid-override-pattern M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M- \
+ --hybrid-layer-pattern M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M- \
--spec megatron.core.models.mamba.mamba_layer_specs mamba_stack_spec \
--mamba-state-dim 256 \
--per-split-data-args-path ${BLEND_PATH} \
@@ -82,7 +82,6 @@ MODEL_OPTIONS="\
--init-method-std 0.0099 \
--position-embedding-type none \
--squared-relu \
- --num-layers 118 \
--hidden-size 8192 \
--num-attention-heads 64 \
--group-query-attention \
diff --git a/examples/rl/model_configs/nemotron5_8b.sh b/examples/rl/model_configs/nemotron5_8b.sh
index 7b8947ae763..c18149f03d6 100644
--- a/examples/rl/model_configs/nemotron5_8b.sh
+++ b/examples/rl/model_configs/nemotron5_8b.sh
@@ -58,9 +58,9 @@ MODEL_OPTIONS="\
--calculate-per-token-loss \
--seq-length $MAX_SEQ_LENGTH \
--inference-max-seq-length $MAX_SEQ_LENGTH \
- --inference-max-batch-size $MAX_INFERENCE_BS \
+ --inference-max-requests $MAX_INFERENCE_BS \
--pretrained-checkpoint $CHECKPOINT \
- --hybrid-override-pattern M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M- \
+ --hybrid-layer-pattern M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M- \
--spec megatron.core.models.mamba.mamba_layer_specs mamba_stack_spec \
--tiktoken-pattern v2 \
--distributed-timeout-minutes 60 \
@@ -73,7 +73,6 @@ MODEL_OPTIONS="\
--init-method-std 0.014 \
--position-embedding-type none \
--squared-relu \
- --num-layers 52 \
--hidden-size 4096 \
--num-attention-heads 32 \
--group-query-attention \
diff --git a/examples/rl/model_configs/nemotron5p5_12b_H.sh b/examples/rl/model_configs/nemotron5p5_12b_H.sh
index 9e97051e087..1826d57e913 100644
--- a/examples/rl/model_configs/nemotron5p5_12b_H.sh
+++ b/examples/rl/model_configs/nemotron5p5_12b_H.sh
@@ -65,7 +65,7 @@ MODEL_OPTIONS="\
--calculate-per-token-loss \
--seq-length $MAX_SEQ_LENGTH \
--inference-max-seq-length $MAX_SEQ_LENGTH \
- --inference-max-batch-size $MAX_INFERENCE_BS \
+ --inference-max-requests $MAX_INFERENCE_BS \
--pretrained-checkpoint $CHECKPOINT \
--fp8-recipe blockwise \
--fp8-format e4m3 \
@@ -74,9 +74,8 @@ MODEL_OPTIONS="\
--num-layers-at-end-in-bf16 2 \
--fp8-param-gather \
--disable-gloo-process-groups \
- --is-hybrid-model \
--mamba-head-dim 80 \
- --hybrid-override-pattern M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M- \
+ --hybrid-layer-pattern M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M- \
--spec megatron.core.models.mamba.mamba_layer_specs mamba_stack_spec \
--tiktoken-pattern v2 \
--distributed-timeout-minutes 10 \
@@ -89,7 +88,6 @@ MODEL_OPTIONS="\
--init-method-std 0.0125 \
--position-embedding-type none \
--squared-relu \
- --num-layers 62 \
--hidden-size 5120 \
--num-attention-heads 40 \
--group-query-attention \
diff --git a/examples/rl/model_configs/nemotron6_3b_moe.sh b/examples/rl/model_configs/nemotron6_3b_moe.sh
index eff4f6cf0b3..85de0c6be0a 100644
--- a/examples/rl/model_configs/nemotron6_3b_moe.sh
+++ b/examples/rl/model_configs/nemotron6_3b_moe.sh
@@ -55,10 +55,10 @@ ENV_DEPENDENT="\
--langrl-env-config $ENV_CONFIG "
MODEL_OPTIONS="\
+ --no-use-tokenizer-model-from-checkpoint-args \
--rl-skip-bos-token \
--no-rl-use-sequence-packing \
--rl-partial-rollouts \
- --rl-offload-optimizer-during-inference \
--moe-pad-experts-for-cuda-graph-inference \
--inference-dynamic-batching-max-tokens 8192 \
--inference-dynamic-batching-max-requests 128 \
@@ -85,7 +85,7 @@ MODEL_OPTIONS="\
--rl-importance-sampling-truncation-coef 10.0 \
--seq-length $MAX_SEQ_LENGTH \
--inference-max-seq-length $MAX_SEQ_LENGTH \
- --inference-max-batch-size $MAX_INFERENCE_BS \
+ --inference-max-requests $MAX_INFERENCE_BS \
--pretrained-checkpoint $CHECKPOINT \
--distributed-timeout-minutes 60 \
--use-mcore-models \
@@ -102,9 +102,10 @@ MODEL_OPTIONS="\
--weight-decay 0.01 \
--clip-grad 1.0 \
--tiktoken-pattern v2 \
- --tokenizer-type TikTokenizer \
+ --tokenizer-type HuggingFaceTokenizer \
--tokenizer-model ${TOKENIZER_MODEL} \
- --dist-ckpt-strictness log_unexpected
+ --tokenizer-hf-include-special-tokens \
+ --dist-ckpt-strictness log_unexpected \
--ckpt-format torch_dist \
--ckpt-fully-parallel-save \
--ckpt-fully-parallel-load \
@@ -118,5 +119,4 @@ MODEL_OPTIONS="\
--lr-warmup-samples 640 \
--lr-warmup-init 0.3e-7 \
--no-load-optim \
- --no-load-rng \
- "
+ --no-load-rng "
diff --git a/examples/rl/model_configs/qwen3_30b_a3b_moe.sh b/examples/rl/model_configs/qwen3_30b_a3b_moe.sh
index 775a9587ba4..637b431280f 100644
--- a/examples/rl/model_configs/qwen3_30b_a3b_moe.sh
+++ b/examples/rl/model_configs/qwen3_30b_a3b_moe.sh
@@ -37,7 +37,7 @@ ENV_DEPENDENT="\
MODEL_OPTIONS="
--seq-length $MAX_SEQ_LENGTH \
--inference-max-seq-length $MAX_SEQ_LENGTH \
---inference-max-batch-size $MAX_INFERENCE_BS \
+--inference-max-requests $MAX_INFERENCE_BS \
--pretrained-checkpoint $CHECKPOINT \
--no-use-tokenizer-model-from-checkpoint-args \
--seq-length 8192 \
@@ -51,6 +51,7 @@ MODEL_OPTIONS="
--te-rng-tracker \
--tokenizer-type HuggingFaceTokenizer \
--tokenizer-model Qwen/Qwen3-30B-A3B \
+--tokenizer-hf-include-special-tokens \
--untie-embeddings-and-output-weights \
--num-layers 48 \
--hidden-size 2048 \
diff --git a/examples/rl/model_configs/qwen3_32b.sh b/examples/rl/model_configs/qwen3_32b.sh
index cd153a04f3c..fcadb0c4021 100644
--- a/examples/rl/model_configs/qwen3_32b.sh
+++ b/examples/rl/model_configs/qwen3_32b.sh
@@ -38,7 +38,7 @@ MODEL_OPTIONS="\
--ckpt-format torch_dist \
--seq-length $MAX_SEQ_LENGTH \
--inference-max-seq-length $MAX_SEQ_LENGTH \
- --inference-max-batch-size $MAX_INFERENCE_BS \
+ --inference-max-requests $MAX_INFERENCE_BS \
--pretrained-checkpoint $CHECKPOINT \
--untie-embeddings-and-output-weights \
--num-layers 64 \
@@ -64,6 +64,7 @@ MODEL_OPTIONS="\
--attention-softmax-in-fp32 \
--tokenizer-type HuggingFaceTokenizer \
--tokenizer-model Qwen/Qwen3-4B \
+ --tokenizer-hf-include-special-tokens \
--vocab-size 151936 \
--make-vocab-size-divisible-by 128 \
--optimizer adam \
diff --git a/examples/rl/model_configs/qwen3_4b.sh b/examples/rl/model_configs/qwen3_4b.sh
index da238511fd3..81899f2ea7b 100644
--- a/examples/rl/model_configs/qwen3_4b.sh
+++ b/examples/rl/model_configs/qwen3_4b.sh
@@ -38,7 +38,7 @@ MODEL_OPTIONS="\
--ckpt-format torch_dist \
--seq-length $MAX_SEQ_LENGTH \
--inference-max-seq-length $MAX_SEQ_LENGTH \
- --inference-max-batch-size $MAX_INFERENCE_BS \
+ --inference-max-requests $MAX_INFERENCE_BS \
--pretrained-checkpoint $CHECKPOINT \
--num-layers 36 \
--hidden-size 2560 \
@@ -63,8 +63,6 @@ MODEL_OPTIONS="\
--attention-softmax-in-fp32 \
--tokenizer-type HuggingFaceTokenizer \
--tokenizer-model Qwen/Qwen3-4B \
- --langrl-inference-server-type "inplace_megatron_chat" \
- --langrl-inference-server-conversation-template "Qwen/Qwen3-4B" \
--vocab-size 151936 \
--make-vocab-size-divisible-by 128 \
--optimizer adam \
diff --git a/examples/rl/model_configs/qwen3_8b.sh b/examples/rl/model_configs/qwen3_8b.sh
index 6758cd84c3d..4b5e1103f0e 100644
--- a/examples/rl/model_configs/qwen3_8b.sh
+++ b/examples/rl/model_configs/qwen3_8b.sh
@@ -38,7 +38,7 @@ MODEL_OPTIONS="\
--ckpt-format torch_dist \
--seq-length $MAX_SEQ_LENGTH \
--inference-max-seq-length $MAX_SEQ_LENGTH \
- --inference-max-batch-size $MAX_INFERENCE_BS \
+ --inference-max-requests $MAX_INFERENCE_BS \
--pretrained-checkpoint $CHECKPOINT \
--untie-embeddings-and-output-weights \
--num-layers 36 \
@@ -64,8 +64,6 @@ MODEL_OPTIONS="\
--attention-softmax-in-fp32 \
--tokenizer-type HuggingFaceTokenizer \
--tokenizer-model Qwen/Qwen3-8B \
- --langrl-inference-server-type "inplace_megatron_chat" \
- --langrl-inference-server-conversation-template "Qwen/Qwen3-8B" \
--vocab-size 151936 \
--make-vocab-size-divisible-by 128 \
--optimizer adam \
diff --git a/examples/rl/model_configs/qwen_2p5_32b.sh b/examples/rl/model_configs/qwen_2p5_32b.sh
index d82972ba477..0bfe19ba1bb 100644
--- a/examples/rl/model_configs/qwen_2p5_32b.sh
+++ b/examples/rl/model_configs/qwen_2p5_32b.sh
@@ -59,7 +59,7 @@ MODEL_OPTIONS="\
--ckpt-format torch_dist \
--seq-length $MAX_SEQ_LENGTH \
--inference-max-seq-length $MAX_SEQ_LENGTH \
- --inference-max-batch-size $MAX_INFERENCE_BS \
+ --inference-max-requests $MAX_INFERENCE_BS \
--pretrained-checkpoint $CHECKPOINT \
--untie-embeddings-and-output-weights \
--disable-bias-linear \
@@ -85,6 +85,7 @@ MODEL_OPTIONS="\
--max-position-embeddings 131072 \
--tokenizer-type HuggingFaceTokenizer \
--tokenizer-model unsloth/Qwen2.5-32B \
+ --tokenizer-hf-include-special-tokens \
--lr 1e-6 \
--lr-warmup-samples 0 \
--make-vocab-size-divisible-by 128 \
diff --git a/examples/rl/model_configs/qwen_2p5_3b.sh b/examples/rl/model_configs/qwen_2p5_3b.sh
index 246afae6ad2..4880272d4a6 100644
--- a/examples/rl/model_configs/qwen_2p5_3b.sh
+++ b/examples/rl/model_configs/qwen_2p5_3b.sh
@@ -22,7 +22,7 @@ if [ "$(basename "$ENV_CONFIG")" = "dapo.yaml" ]; then
GRPO_KL_BETA=${GRPO_KL_BETA:-"0.0"}
ENTROPY_WEIGHT=${ENTROPY_WEIGHT:-"0.0"}
TRAINING_BATCH_SIZE=${TRAINING_BATCH_SIZE:-1024}
- MICRO_BATCH_SIZE=${MICRO_BATCH_SIZE:-2}
+ MICRO_BATCH_SIZE=${MICRO_BATCH_SIZE:-1}
MAX_SEQ_LENGTH=${MAX_SEQ_LENGTH:-8192}
EXIT_INTERVAL=${EXIT_INTERVAL:-16}
CHKPT_SAVE_INTERVAL=${CHKPT_SAVE_INTERVAL:-16}
@@ -38,7 +38,7 @@ else
GRPO_KL_BETA=${GRPO_KL_BETA:-"0.0"}
ENTROPY_WEIGHT=${ENTROPY_WEIGHT:-"0.0"}
TRAINING_BATCH_SIZE=${TRAINING_BATCH_SIZE:-512}
- MICRO_BATCH_SIZE=${MICRO_BATCH_SIZE:-2}
+ MICRO_BATCH_SIZE=${MICRO_BATCH_SIZE:-1}
MAX_SEQ_LENGTH=${MAX_SEQ_LENGTH:-8192}
EXIT_INTERVAL=${EXIT_INTERVAL:-16}
CHKPT_SAVE_INTERVAL=${CHKPT_SAVE_INTERVAL:-16}
@@ -62,7 +62,7 @@ MODEL_OPTIONS="\
--ckpt-format torch_dist \
--seq-length $MAX_SEQ_LENGTH \
--inference-max-seq-length $MAX_SEQ_LENGTH \
- --inference-max-batch-size $MAX_INFERENCE_BS \
+ --inference-max-requests $MAX_INFERENCE_BS \
--pretrained-checkpoint $CHECKPOINT \
--disable-bias-linear \
--add-qkv-bias \
@@ -87,6 +87,7 @@ MODEL_OPTIONS="\
--max-position-embeddings 32768 \
--tokenizer-type HuggingFaceTokenizer \
--tokenizer-model unsloth/Qwen2.5-3B \
+ --tokenizer-hf-include-special-tokens \
--lr 0.000001 \
--lr-warmup-samples 0 \
--make-vocab-size-divisible-by 64 \
diff --git a/examples/rl/model_configs/qwen_2p5_distill_7b.sh b/examples/rl/model_configs/qwen_2p5_distill_7b.sh
index 149ac77965f..ed214b3aae9 100644
--- a/examples/rl/model_configs/qwen_2p5_distill_7b.sh
+++ b/examples/rl/model_configs/qwen_2p5_distill_7b.sh
@@ -44,7 +44,7 @@ MODEL_OPTIONS="\
--ckpt-format torch \
--seq-length $MAX_SEQ_LENGTH \
--inference-max-seq-length $MAX_SEQ_LENGTH \
- --inference-max-batch-size $MAX_INFERENCE_BS \
+ --inference-max-requests $MAX_INFERENCE_BS \
--pretrained-checkpoint $CHECKPOINT \
--untie-embeddings-and-output-weights \
--disable-bias-linear \
@@ -70,8 +70,6 @@ MODEL_OPTIONS="\
--max-position-embeddings 131072 \
--tokenizer-type HuggingFaceTokenizer \
--tokenizer-model "unsloth/DeepSeek-R1-Distill-Qwen-7B" \
- --langrl-inference-server-type "inplace_megatron_chat" \
- --langrl-inference-server-conversation-template "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B" \
--lr 0.000001 \
--lr-warmup-samples 0 \
--make-vocab-size-divisible-by 128 \
diff --git a/examples/rl/model_configs/qwen_2p5_math_7b.sh b/examples/rl/model_configs/qwen_2p5_math_7b.sh
index 1d631fa80a5..b00077bc07a 100644
--- a/examples/rl/model_configs/qwen_2p5_math_7b.sh
+++ b/examples/rl/model_configs/qwen_2p5_math_7b.sh
@@ -58,7 +58,7 @@ MODEL_OPTIONS="\
--ckpt-format torch \
--seq-length $MAX_SEQ_LENGTH \
--inference-max-seq-length $MAX_SEQ_LENGTH \
- --inference-max-batch-size $MAX_INFERENCE_BS \
+ --inference-max-requests $MAX_INFERENCE_BS \
--pretrained-checkpoint $CHECKPOINT \
--untie-embeddings-and-output-weights \
--disable-bias-linear \
@@ -84,6 +84,7 @@ MODEL_OPTIONS="\
--max-position-embeddings 4096 \
--tokenizer-type HuggingFaceTokenizer \
--tokenizer-model "unsloth/Qwen2.5-Math-7B" \
+ --tokenizer-hf-include-special-tokens \
--lr 0.000001 \
--lr-warmup-samples 0 \
--make-vocab-size-divisible-by 128 \
diff --git a/examples/run_simple_mcore_train_loop.py b/examples/run_simple_mcore_train_loop.py
index f748d74ecec..24aea52b2b2 100644
--- a/examples/run_simple_mcore_train_loop.py
+++ b/examples/run_simple_mcore_train_loop.py
@@ -20,10 +20,10 @@
BlendedMegatronDatasetBuilder,
)
from megatron.core.datasets.gpt_dataset import GPTDatasetConfig, MockGPTDataset
-from megatron.training.tokenizer.tokenizer import _NullTokenizer
from megatron.core.distributed import DistributedDataParallel
from megatron.core.distributed import DistributedDataParallelConfig
from megatron.core.distributed.finalize_model_grads import finalize_model_grads
+from megatron.core.tokenizers import MegatronTokenizer
_SEQUENCE_LENGTH: int = 64
@@ -102,7 +102,10 @@ def get_train_data_iterator() -> Iterator:
reset_position_ids=False,
reset_attention_mask=False,
eod_mask_loss=False,
- tokenizer=_NullTokenizer(vocab_size=_SEQUENCE_LENGTH),
+ tokenizer=MegatronTokenizer.from_pretrained(
+ metadata_path={"library": "null-text"},
+ vocab_size=_SEQUENCE_LENGTH,
+ ),
mid_level_dataset_surplus=0.005,
)
diff --git a/gpt_builders.py b/gpt_builders.py
index c8b4efa3075..24b5f89d311 100644
--- a/gpt_builders.py
+++ b/gpt_builders.py
@@ -53,7 +53,6 @@ def gpt_builder(args, pre_process, post_process, vp_stage=None, config=None, pg_
)
)
elif args.num_experts:
- assert not (config.transformer_impl == "inference_optimized")
# Define the decoder block spec
transformer_layer_spec = get_gpt_decoder_block_spec(
config,
@@ -116,45 +115,42 @@ def gpt_builder(args, pre_process, post_process, vp_stage=None, config=None, pg_
def _get_transformer_layer_spec(use_te, config):
"""Get transformer layer specification based on configuration.
-
+
Args:
use_te (bool): Whether to use Transformer Engine
- args: Training arguments
config: Model configuration
-
+
Returns:
transformer_layer_spec: The transformer layer specification
"""
- args = get_args()
if use_te:
return get_gpt_layer_with_transformer_engine_spec(
- args.num_experts,
- args.moe_grouped_gemm,
- args.qk_layernorm,
- args.multi_latent_attention,
- args.experimental_attention_variant,
- moe_use_legacy_grouped_gemm=args.moe_use_legacy_grouped_gemm,
- qk_l2_norm=args.qk_l2_norm,
+ config.num_moe_experts,
+ config.moe_grouped_gemm,
+ config.qk_layernorm,
+ config.multi_latent_attention,
+ config.experimental_attention_variant,
+ qk_l2_norm=config.qk_l2_norm,
use_kitchen=config.use_kitchen,
use_te_activation_func=config.use_te_activation_func,
use_kitchen_attention=config.use_kitchen_attention,
kitchen_attention_backend=config.kitchen_attention_backend,
+ mla_down_proj_fusion=getattr(config, "mla_down_proj_fusion", False),
)
elif config.transformer_impl == "inference_optimized":
return get_gpt_layer_with_inference_spec(
- args.qk_layernorm,
- args.multi_latent_attention,
- qk_l2_norm=args.qk_l2_norm,
+ config.qk_layernorm,
+ config.multi_latent_attention,
+ qk_l2_norm=config.qk_l2_norm,
)
else:
return get_gpt_layer_local_spec(
- args.num_experts,
- args.moe_grouped_gemm,
- args.qk_layernorm,
- args.multi_latent_attention,
- args.experimental_attention_variant,
- moe_use_legacy_grouped_gemm=args.moe_use_legacy_grouped_gemm,
- normalization=args.normalization,
+ config.num_moe_experts,
+ config.moe_grouped_gemm,
+ config.qk_layernorm,
+ config.multi_latent_attention,
+ config.experimental_attention_variant,
+ normalization=config.normalization,
use_kitchen=config.use_kitchen,
use_kitchen_attention=config.use_kitchen_attention,
kitchen_attention_backend=config.kitchen_attention_backend,
diff --git a/mamba_builders.py b/mamba_builders.py
index 5d31af60475..650ea4a719f 100644
--- a/mamba_builders.py
+++ b/mamba_builders.py
@@ -30,10 +30,8 @@ def mamba_builder(args, pre_process, post_process, vp_stage=None, config=None, p
mamba_stack_spec=mamba_stack_spec,
vocab_size=args.padded_vocab_size,
max_sequence_length=args.max_position_embeddings,
+ hybrid_layer_pattern=args.hybrid_layer_pattern,
pre_process=pre_process,
- hybrid_attention_ratio=args.hybrid_attention_ratio,
- hybrid_mlp_ratio=args.hybrid_mlp_ratio,
- hybrid_override_pattern=args.hybrid_override_pattern,
post_process=post_process,
fp16_lm_cross_entropy=args.fp16_lm_cross_entropy,
parallel_output=True,
diff --git a/megatron/core/MSC_Integration.md b/megatron/core/MSC_Integration.md
index d0c93a80129..da8b5c982b8 100644
--- a/megatron/core/MSC_Integration.md
+++ b/megatron/core/MSC_Integration.md
@@ -1,4 +1,4 @@
-## Multi-Storage Client (MSC) Integration
+# Multi-Storage Client (MSC) Integration
The [Multi-Storage Client](https://github.com/NVIDIA/multi-storage-client) (MSC) provides a unified interface for reading datasets and storing checkpoints from both filesystems (e.g., local disk, NFS, Lustre) and object storage providers such as S3, GCS, OCI, Azure, AIStore, and SwiftStack.
@@ -8,7 +8,7 @@ This guide will walk you through how to:
2. How to train models directly using datasets in object storage
3. How to save and load model checkpoints to/from object storage
-### Installation
+## Installation
MSC is vended as `the multi-storage-client` package on PyPI.
@@ -25,7 +25,7 @@ pip install "multi-storage-client[boto3]"
pip install "multi-storage-client[google-cloud-storage]"
```
-### Configuration File
+## Configuration File
MSC uses a YAML configuration file to define how it connects to object storage systems. This design allows you to specify one or more storage profiles, each representing a different storage backend or bucket. MSC keeps your training scripts clean and portable by centralizing details in a config file. There is no need to hardcode access keys, bucket names, or other provider-specific options directly into your code.
@@ -58,7 +58,7 @@ To tell MSC where to find this file, set the following environment variable befo
export MSC_CONFIG=/path/to/msc_config.yaml
```
-### MSC URL Format
+## MSC URL Format
MSC uses a custom URL scheme to identify and access files across different object storage providers. This scheme makes it easy to reference data and checkpoints without worrying about the underlying storage implementation. An MSC URL has the following structure:
@@ -96,7 +96,7 @@ is interpreted as accessing the object with the key `dataset/train/data.bin` ins
This abstraction allows training scripts to reference storage resources uniformly—whether they're hosted on AWS, GCP, Oracle, or Azure—just by switching profiles in the config file.
-### Train from Object Storage
+## Train from Object Storage
To train with datasets stored in object storage, use an MSC URL with the `--data-path` argument. This URL references a dataset stored under a profile defined in your MSC configuration file.
@@ -112,7 +112,7 @@ python pretrain_gpt.py \
**NOTE:** All four arguments must be provided when training with datasets in object storage using MSC.
-### Save and Load Checkpoints from Object Storage
+## Save and Load Checkpoints from Object Storage
MSC can be used to save and load model checkpoints directly from object storage by specifying MSC URLs for the `--save` and `--load` arguments. This allows you to manage checkpoints in object storage.
@@ -125,7 +125,7 @@ python pretrain_gpt.py \
**Notes:** Only the `torch_dist` checkpoint format is currently supported when saving to or loading from MSC URLs.
-### Disable MSC
+## Disable MSC
By default, MSC integration is automatically enabled when the `multi-storage-client` library is installed. MSC is also used for regular filesystem paths (like `/filesystem_mountpoint/path` in `--data-path`, `--save`, or `--load`) even when not using explicit MSC URLs. MSC functions as a very thin abstraction layer with negligible performance impact when used with regular paths, so there's typically no need to disable it. If you need to disable MSC, you can do so using the `--disable-msc` flag:
@@ -133,7 +133,7 @@ By default, MSC integration is automatically enabled when the `multi-storage-cli
python pretrain_gpt.py --disable-msc
```
-### Performance Considerations
+## Performance Considerations
When using object storage with MSC, there are a few important performance implications to keep in mind:
@@ -165,7 +165,7 @@ cache:
For optimal performance, configure the cache directory on a high-speed local storage device such as an NVMe SSD.
-### Additional Resources and Advanced Configuration
+## Additional Resources and Advanced Configuration
Refer to the [MSC Configuration Documentation](https://nvidia.github.io/multi-storage-client/references/configuration.html) for complete documentation on MSC configuration options, including detailed information about supported storage providers, credentials management, and advanced caching strategies.
diff --git a/megatron/core/QuickStart.md b/megatron/core/QuickStart.md
index 4aa966a164d..2a2744cf43e 100644
--- a/megatron/core/QuickStart.md
+++ b/megatron/core/QuickStart.md
@@ -21,13 +21,13 @@ This guide for Megatron Core walks you through the following tasks:
git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
- pip install -U "setuptools<80.0.0,>=77.0.0" packaging
- pip install --no-build-isolation .[dev]
+ pip install uv
+ uv pip install -e .
```
-p
+ # class 3 = idle (only when no other process needs I/O)
+ # class 2 = best-effort (default, can set priority 0-7)
+ subprocess.run(
+ ["ionice", "-c", str(io_priority), "-p", str(pid)], check=True, capture_output=True
+ )
+ logger.debug(f"PID {pid}: Set I/O priority class to {io_priority}")
+ except (subprocess.CalledProcessError, FileNotFoundError, PermissionError) as e:
+ logger.warning(f"PID {pid}: Failed to set I/O priority: {e}")
+
+
@contextmanager
def _disable_gc():
"""Temporarily disables GC."""
@@ -58,7 +117,7 @@ class AsyncRequest(NamedTuple):
async_fn_args: Tuple
finalize_fns: List[Callable]
async_fn_kwargs: Dict = {}
- preload_fn: Callable = None
+ preload_fn: Optional[Callable] = None
is_frozen: bool = False
call_idx: int = 0
@@ -83,7 +142,7 @@ def execute_sync(self) -> None:
"""
# preload tensors.
async_fn_args = list(self.async_fn_args)
- if self.preload_fn:
+ if self.preload_fn is not None:
assert len(async_fn_args) == 3, "Expected 3 args to be passed to async function"
# The async_fn is passed as a partial functool with pre-determined args
# In the async_fn_args we pass the remaining positional args required by the async_fn
@@ -169,7 +228,7 @@ def sync_all_async_calls(self, is_alive: int) -> bool:
@abstractmethod
def close(self, abort=False):
"""Terminate the async caller at exit of an application or some termination conditions"""
- logger.debug(f"AsyncCaller: {torch.distributed.get_rank()}, Destroying Async Caller")
+ raise NotImplementedError
def __del__(self):
raise NotImplementedError("This should be implemented")
@@ -184,6 +243,7 @@ class TemporalAsyncCaller(AsyncCaller):
def __init__(self):
self.process: Optional[mp.Process] = None
self.start_time: Optional[float] = None
+ self.preloaded_holder = None
@_disable_gc()
def schedule_async_call(self, async_req: AsyncRequest) -> None:
@@ -201,11 +261,12 @@ def schedule_async_call(self, async_req: AsyncRequest) -> None:
return # nothing to do
async_fn_args = list(async_req.async_fn_args)
- if async_req.preload_fn:
+ if async_req.preload_fn is not None:
# If there's a preload_fn in `async_req`, we call this func
# to do the defined action in `async_req.preload_fn` to
# stage GPU tensors to its defined destination
async_fn_args[1] = async_req.preload_fn()
+ self.preloaded_holder = async_fn_args[1]
rank = torch.distributed.get_rank()
start_sync = time()
@@ -265,12 +326,10 @@ def close(self, abort=False):
the checkpoint async process needs to be aborted.
"""
if self.process:
- logger.debug(f"rank: {torch.distributed.get_rank()}, joining self.process")
+ logger.debug(f"rank: {safe_get_rank()}, joining self.process")
if abort:
log_single_rank(
- logger,
- logging.WARNING,
- f"Temporal worker aborted in rank {torch.distributed.get_rank()}",
+ logger, logging.WARNING, f"Temporal worker aborted in rank {safe_get_rank()}"
)
self.process.kill()
else:
@@ -281,6 +340,7 @@ def close(self, abort=False):
f"after {time() - self.start_time:.2f}s from forking"
)
self.start_time = None
+ self.preloaded_holder = None
def __del__(self):
pass
@@ -292,20 +352,48 @@ class PersistentAsyncCaller(AsyncCaller):
Starts process asynchronously and allows checking if all processes on all ranks are done.
"""
+ _persistent_process: mp.Process = None
+ _persistent_queue: mp.JoinableQueue = None
+ _persistent_preload_q: mp.JoinableQueue = None
+ _persistent_comp_q: mp.Queue = None
+
def __init__(self):
- self.process: mp.Process = None
+ self.process: Optional[mp.Process] = None
self.start_time: Optional[float] = None
- ctx = mp.get_context('spawn')
- # main queue to deliver `AsyncRequest` from host to the ckpt worker
- self.queue: mp.JoinableQueue = ctx.JoinableQueue()
- # Queue used to synchronize for the completion of preloading tensors to host
- # between a trainer and ckpt worker
- self.preload_q: mp.JoinableQueue = ctx.JoinableQueue()
- # Queue used to inform trainer when the saving is completed
- self.comp_q: mp.Queue = ctx.Queue()
- self.cur_item: int = None
+ self.cur_item: Optional[int] = None
self.cur_idx: int = -1
+ @classmethod
+ def _get_process(
+ cls,
+ rank: int,
+ mp_mode: str = 'spawn',
+ cpu_priority: int = 10,
+ io_priority: Optional[int] = None,
+ ):
+ if cls._persistent_process is None:
+ ctx = mp.get_context(mp_mode)
+ logger.debug(f"PersistentAsyncCaller: {rank}, Starting Async Caller")
+ cls._persistent_queue = ctx.JoinableQueue()
+ cls._persistent_preload_q = ctx.JoinableQueue()
+ cls._persistent_comp_q = ctx.Queue()
+ cls._persistent_process = ctx.Process(
+ target=PersistentAsyncCaller.async_loop,
+ args=(
+ rank,
+ cls._persistent_queue,
+ cls._persistent_preload_q,
+ cls._persistent_comp_q,
+ logger.getEffectiveLevel(),
+ cpu_priority,
+ io_priority,
+ ),
+ )
+ cls._persistent_process.daemon = True
+ cls._persistent_process.start()
+ logger.debug(f"PersistentAsyncCaller: {rank}, Started Async Caller")
+ return cls._persistent_process
+
def schedule_async_call(self, async_req: AsyncRequest) -> None:
"""Put `AsyncRequest` to the Persistent Async Caller
@@ -324,34 +412,16 @@ def schedule_async_call(self, async_req: AsyncRequest) -> None:
self.start_time = time()
if self.process is None:
- ctx = mp.get_context('spawn')
- logger.debug(
- f"PersistentAsyncCaller: {torch.distributed.get_rank()}, Starting Async Caller"
- )
- self.process: mp.Process = ctx.Process(
- target=PersistentAsyncCaller.async_loop,
- args=(
- torch.distributed.get_rank(),
- self.queue,
- self.preload_q,
- self.comp_q,
- logger.getEffectiveLevel(),
- ),
- )
- self.process.start()
- logger.debug(
- f"PersistentAsyncCaller: {torch.distributed.get_rank()}, Started Async Caller"
- )
-
- if async_req.preload_fn:
- self.preload_q.put(async_req.call_idx)
- self.queue.put(async_req)
+ self.process = PersistentAsyncCaller._get_process(torch.distributed.get_rank())
+ if async_req.preload_fn is not None:
+ self._persistent_preload_q.put(async_req.call_idx)
+ self._persistent_queue.put(async_req)
logger.debug(f"rank: {torch.distributed.get_rank()}, put {async_req.call_idx}")
- if async_req.preload_fn:
+ if async_req.preload_fn is not None:
start_sync = time()
# Synchronize for pre-staging tensors
- self.preload_q.join()
+ self._persistent_preload_q.join()
end_sync = time()
logger.debug(
f"rank: {torch.distributed.get_rank()}, "
@@ -388,7 +458,7 @@ def is_current_async_call_done(self, blocking: bool = False, no_dist: bool = Fal
while self.cur_item is None:
try:
# Retrieve comp call_idx without waiting
- self.cur_item = self.comp_q.get_nowait()
+ self.cur_item = self._persistent_comp_q.get_nowait()
except Empty:
# This method is called after any `AsyncRequest` is pushed to the main loop
# So, the background writing is still active
@@ -425,22 +495,22 @@ def close(self, abort=False):
abort (bool, optional): Default to False. Needs to be manually set to true when
the checkpoint async process needs to be aborted.
"""
- logger.debug(
- f"PersistentAsyncCaller: {torch.distributed.get_rank()}, Destroying Async Caller"
- )
+ logger.debug(f"PersistentAsyncCaller: {safe_get_rank()}, Destroying Async Caller")
if self.process:
if abort:
log_single_rank(
- logger,
- logging.WARNING,
- f"Persistent worker aborted in rank {torch.distributed.get_rank()}",
+ logger, logging.WARNING, f"Persistent worker aborted in rank {safe_get_rank()}"
)
self.process.kill()
else:
- self.queue.put('DONE')
- self.queue.join()
- self.process.join()
+ self._persistent_queue.put('DONE')
+ self._persistent_queue.join()
+ self._persistent_process.join()
self.process = None
+ PersistentAsyncCaller._persistent_process = None
+ PersistentAsyncCaller._persistent_queue = None
+ PersistentAsyncCaller._persistent_preload_q = None
+ PersistentAsyncCaller._persistent_comp_q = None
def __del__(self):
self.close()
@@ -453,6 +523,8 @@ def async_loop(
preload_q: mp.JoinableQueue,
comp_q: mp.Queue,
log_level: int = logging.INFO,
+ cpu_priority: int = 10,
+ io_priority: Optional[int] = None,
):
"""Main function for the persistent checkpoint worker
@@ -474,11 +546,16 @@ def async_loop(
async checkpoint request
log_level (int, Optional): an integer to set log-level in this spawned process
to get aligned with the training rank's logging level
+ cpu_priority (int): Nice value for CPU scheduling (0-19, higher = lower priority).
+ Default 10 deprioritizes checkpoint writing vs training.
+ io_priority (int, Optional): I/O scheduling class (0-3, where 3=idle).
+ Default 3 ensures checkpoints don't block data loading.
"""
# Set logger.
+ # Set root logger level to affect all modules in this process
+ logging.getLogger().setLevel(log_level)
logger = logging.getLogger(__name__)
- logger.setLevel(log_level)
logger.debug(f"PersistentAsyncCaller: persistent ckpt worker for {rank} has started")
# Set CUDA device to appropriate local_rank to ensure allocations / CUDA contexts
@@ -487,6 +564,10 @@ def async_loop(
# this line).
torch.cuda.set_device(rank % torch.cuda.device_count())
+ # Set QoS to deprioritize checkpoint writing vs training
+ # This prevents checkpoint I/O from interfering with data loader
+ _set_process_qos(cpu_priority=cpu_priority, io_priority=io_priority)
+
# Start busy loop waiting for and executing checkpoint saves.
while True:
item = queue.get()
@@ -495,16 +576,20 @@ def async_loop(
break
elif isinstance(item, AsyncRequest):
async_fn_args = list(item.async_fn_args)
- if item.preload_fn:
+ if item.preload_fn is not None:
call_idx = preload_q.get()
# the 2nd arg is state dict
async_fn_args[1] = item.preload_fn()
logger.debug(f"{rank} has completed D2H of {call_idx}")
preload_q.task_done()
- item.async_fn(*async_fn_args, **item.async_fn_kwargs)
+ if item.async_fn is not None:
+ item.async_fn(*async_fn_args, **item.async_fn_kwargs)
logger.debug(f"{rank} has completed saving {item.call_idx}")
comp_q.put(item.call_idx)
queue.task_done()
+ del async_fn_args
+ del item
+ gc.collect()
logger.debug(f"PersistentAsyncCaller: persistent ckpt worker for {rank} has terminated")
@@ -531,18 +616,30 @@ class AsyncCallsQueue:
active calls with `maybe_finalize_async_calls`.
"""
+ _persistent_caller: Optional[PersistentAsyncCaller] = None
+
def __init__(self, persistent: bool = False):
self.async_calls: deque[_ActiveAsyncRequest] = deque([])
self.call_idx: int = -1
self.persistent: bool = persistent
- self.persistent_caller: AsyncCaller = None
def _get_async_caller(self):
if not self.persistent:
return TemporalAsyncCaller()
- if self.persistent_caller is None:
- self.persistent_caller = PersistentAsyncCaller()
- return self.persistent_caller
+ if AsyncCallsQueue._persistent_caller is None:
+ AsyncCallsQueue._persistent_caller = PersistentAsyncCaller()
+ return AsyncCallsQueue._persistent_caller
+
+ @classmethod
+ def warmup_persistent_caller(
+ cls,
+ rank: int,
+ mp_mode: str = 'spawn',
+ cpu_priority: int = 10,
+ io_priority: Optional[int] = None,
+ ):
+ """Warmup the persistent caller to avoid the overhead of creating it on the first call."""
+ PersistentAsyncCaller._get_process(rank, mp_mode, cpu_priority, io_priority)
def schedule_async_request(self, async_request: AsyncRequest) -> int:
"""Start a new async call and add it to a queue of active async calls.
@@ -617,5 +714,6 @@ def close(self, abort=False):
"""
if not abort:
self.maybe_finalize_async_calls(blocking=True)
- if self.persistent and self.persistent_caller:
- self.persistent_caller.close(abort=abort)
+ if self.persistent and AsyncCallsQueue._persistent_caller:
+ AsyncCallsQueue._persistent_caller.close(abort=abort)
+ AsyncCallsQueue._persistent_caller = None
diff --git a/megatron/core/dist_checkpointing/strategies/base.py b/megatron/core/dist_checkpointing/strategies/base.py
index 53422b362f6..eb20e145ffb 100644
--- a/megatron/core/dist_checkpointing/strategies/base.py
+++ b/megatron/core/dist_checkpointing/strategies/base.py
@@ -8,12 +8,12 @@
from pathlib import Path
from typing import Any, DefaultDict, Union
-from ..mapping import CheckpointingException, ShardedStateDict, StateDict
+from ..mapping import CheckpointingException, ShardedStateDict
from .async_utils import AsyncCallsQueue, AsyncRequest
class StrategyAction(Enum):
- """Specifies save vs load and sharded vs common action."""
+ """Specifies save vs load action."""
LOAD_COMMON = 'load_common'
LOAD_SHARDED = 'load_sharded'
@@ -56,7 +56,7 @@ def register_default_strategy(
"""Adds a given strategy to the registry of default strategies.
Args:
- action (StrategyAction): specifies save/load and sharded/common
+ action (StrategyAction): specifies save/load and sharded
backend (str): backend that the strategy becomes a default for
version (int): version that the strategy becomes a default for
strategy (SaveStrategyBase, LoadStrategyBase): strategy to register
@@ -101,28 +101,6 @@ def __str__(self):
return f'{self.__class__.__name__}({self.backend}, {self.version})'
-class LoadCommonStrategy(LoadStrategyBase):
- """Load strategy for common (non-sharded) objects"""
-
- @abstractmethod
- def load_common(self, checkpoint_dir: Union[str, Path]):
- """Load common part of the checkpoint."""
- raise NotImplementedError
-
- @abstractmethod
- def load_sharded_objects(
- self, sharded_objects_state_dict: ShardedStateDict, checkpoint_dir: Union[str, Path]
- ):
- """Load sharded objects from the checkpoint."""
- raise NotImplementedError
-
- def load_sharded_metadata(self, checkpoint_dir: Union[str, Path]) -> ShardedStateDict:
- """Load just the metadata from the checkpoint."""
- if not self.can_handle_sharded_objects:
- return {}
- raise NotImplementedError
-
-
class LoadShardedStrategy(LoadStrategyBase):
"""Load strategy for sharded tensors"""
@@ -166,21 +144,6 @@ def remove_sharded_tensors(self, checkpoint_dir: Union[str, Path], key_prefix: s
raise NotImplementedError
-class SaveCommonStrategy(SaveStrategyBase):
- """Save strategy for common (non-sharded) objects"""
-
- @abstractmethod
- def save_common(self, common_state_dict: StateDict, checkpoint_dir: Union[str, Path]):
- """Save common part of the state dict."""
- raise NotImplementedError
-
- def save_sharded_objects(
- self, sharded_objects_state_dict: ShardedStateDict, checkpoint_dir: Union[str, Path]
- ):
- """Save sharded objects from the state dict."""
- raise NotImplementedError
-
-
class SaveShardedStrategy(SaveStrategyBase):
"""Save strategy for sharded tensors"""
@@ -213,3 +176,4 @@ def save(self, sharded_state_dict: ShardedStateDict, checkpoint_dir: Union[str,
"""Each async strategy can be trivially used as a sync strategy."""
async_request = self.async_save(sharded_state_dict, checkpoint_dir)
async_request.execute_sync()
+ del async_request
diff --git a/megatron/core/dist_checkpointing/strategies/cached_metadata_filesystem_reader.py b/megatron/core/dist_checkpointing/strategies/cached_metadata_filesystem_reader.py
index f8cb0326e26..991b8d933b1 100644
--- a/megatron/core/dist_checkpointing/strategies/cached_metadata_filesystem_reader.py
+++ b/megatron/core/dist_checkpointing/strategies/cached_metadata_filesystem_reader.py
@@ -3,7 +3,7 @@
""" FS Reader with metadata cached support. """
import os
-from typing import Union
+from typing import Dict, Union
from torch.distributed.checkpoint import FileSystemReader, Metadata
@@ -12,11 +12,16 @@ class CachedMetadataFileSystemReader(FileSystemReader):
"""
Extends FileSystemReader to cache metadata for improved performance.
+ Metadata is shared across all reader instances that use the same checkpoint
+ directory (same path), since the loaded metadata is identical.
+
Attributes:
- _cached_metadata (Metadata or None): Cached metadata from the file system.
+ _metadata_cache (Dict[str, Metadata]): Class-level cache keyed by checkpoint path.
"""
- def __init__(self, path: Union[str, os.PathLike]) -> None:
+ _metadata_cache: Dict[str, Metadata] = {}
+
+ def __init__(self, path: Union[str, os.PathLike], cache_metadata: bool = True) -> None:
"""
Initialize with file system path.
@@ -24,15 +29,46 @@ def __init__(self, path: Union[str, os.PathLike]) -> None:
path (Union[str, os.PathLike]): Path to the checkpoint directory or file.
"""
super().__init__(path=path)
- self._cached_metadata = None
+ self._cache_metadata = cache_metadata
+ self._abs_path = os.path.abspath(os.fspath(path))
+
+ def _metadata_signature(self) -> tuple:
+ """Build a cache key that is invalidated when the checkpoint is overwritten.
+
+ Keying solely on the path is unsafe: a new checkpoint with a different structure
+ can be written to the same path (e.g. reused directories), in which case a stale
+ cached metadata would be returned and cause load failures. We additionally
+ incorporate the metadata file's modification time and size so that rewriting the
+ checkpoint at the same path invalidates the cache entry.
+ """
+ metadata_path = self._get_metadata_path()
+ try:
+ stat = os.stat(metadata_path)
+ return (self._abs_path, stat.st_mtime_ns, stat.st_size)
+ except OSError:
+ # Fall back to path-only key if the metadata file can't be stat-ed.
+ return (self._abs_path,)
def read_metadata(self) -> Metadata:
"""
Read metadata from file system, caching for subsequent calls.
+ Shared across instances when the checkpoint directory is the same and the
+ on-disk metadata file is unchanged.
Returns:
Metadata: Checkpoint metadata.
"""
- if self._cached_metadata is None:
- self._cached_metadata = super().read_metadata()
- return self._cached_metadata
+ if not self._cache_metadata:
+ return super().read_metadata()
+
+ cache_key = self._metadata_signature()
+ if cache_key not in CachedMetadataFileSystemReader._metadata_cache:
+ CachedMetadataFileSystemReader._metadata_cache[cache_key] = super().read_metadata()
+ return CachedMetadataFileSystemReader._metadata_cache[cache_key]
+
+ @classmethod
+ def clear_metadata_cache(cls):
+ """
+ Clear the metadata cache.
+ """
+ cls._metadata_cache.clear()
diff --git a/megatron/core/dist_checkpointing/strategies/common.py b/megatron/core/dist_checkpointing/strategies/common.py
index 41c21d93d7d..0ae800e46f8 100644
--- a/megatron/core/dist_checkpointing/strategies/common.py
+++ b/megatron/core/dist_checkpointing/strategies/common.py
@@ -5,189 +5,52 @@
import logging
import os
from pathlib import Path
-from typing import Union
import torch
-from megatron.core.dist_checkpointing.mapping import ShardedStateDict, StateDict
-from megatron.core.dist_checkpointing.strategies.base import (
- SaveCommonStrategy,
- StrategyAction,
- register_default_strategy,
-)
+from megatron.core.dist_checkpointing.mapping import StateDict
from megatron.core.msc_utils import MultiStorageClientFeature
-from ..dict_utils import dict_list_map_inplace, nested_values
-from ..mapping import CheckpointingException, ShardedObject, is_main_replica
-from ..strategies.base import LoadCommonStrategy
+from ..mapping import CheckpointingException
COMMON_STATE_FNAME = 'common.pt'
logger = logging.getLogger(__name__)
-def register_default_common_strategies():
- """Register default common strategies."""
- register_default_strategy(StrategyAction.LOAD_COMMON, 'torch', 1, TorchCommonLoadStrategy())
- register_default_strategy(
- StrategyAction.SAVE_COMMON, 'torch', 1, TorchCommonSaveStrategy('torch', 1)
- )
-
-
-class TorchCommonSaveStrategy(SaveCommonStrategy):
- """Common save strategy leveraging native torch save/load."""
-
- def save_common(self, common_state_dict: StateDict, checkpoint_dir: Union[str, Path]):
- """Save common part of the state dict."""
- if torch.distributed.get_rank() == 0:
- path = os.path.join(checkpoint_dir, COMMON_STATE_FNAME)
- if MultiStorageClientFeature.is_enabled():
- msc = MultiStorageClientFeature.import_package()
- msc.torch.save(common_state_dict, path)
- else:
- torch.save(common_state_dict, path)
-
- def save_sharded_objects(
- self, sharded_objects_state_dict: ShardedStateDict, checkpoint_dir: Union[str, Path]
- ):
- """Save sharded objects from the state dict."""
- for sh_obj in nested_values(sharded_objects_state_dict):
- if is_main_replica(sh_obj.replica_id):
- save_path = os.path.join(checkpoint_dir, f"{sh_obj.unique_key}.pt")
- parent_dir = os.path.dirname(save_path)
- if MultiStorageClientFeature.is_enabled():
- msc = MultiStorageClientFeature.import_package()
- msc.os.makedirs(parent_dir, exist_ok=True)
- msc.torch.save(sh_obj.data, save_path)
- else:
- os.makedirs(parent_dir, exist_ok=True)
- torch.save(sh_obj.data, save_path)
-
- def can_handle_sharded_objects(self):
- """This strategy can handle ShardedObjects."""
- return True
-
-
-class TorchCommonLoadStrategy(LoadCommonStrategy):
- """Common load strategy leveraging native torch save/load."""
-
- def load_common(self, checkpoint_dir: Union[str, Path]):
- """Load common (non-sharded) objects state dict from the checkpoint.
-
- Args:
- checkpoint_dir (Union[str, Path]): checkpoint directory
-
- Returns:
- StateDict: state dict with non-sharded objects from the checkpoint
- """
- load_path = os.path.join(checkpoint_dir, COMMON_STATE_FNAME)
- try:
- if MultiStorageClientFeature.is_enabled():
- msc = MultiStorageClientFeature.import_package()
- return msc.torch.load(load_path, map_location='cpu')
- else:
- return torch.load(load_path, map_location='cpu')
- except FileNotFoundError as e:
- err_msg = f'Common file {load_path} does not exist'
- if MultiStorageClientFeature.is_enabled():
- msc = MultiStorageClientFeature.import_package()
- ckpt_files = [f.name for f in msc.Path(checkpoint_dir).iterdir()]
- else:
- ckpt_files = [f.name for f in checkpoint_dir.iterdir()]
- logger.debug(f'{err_msg}. Checkpoint directory content: {ckpt_files}')
- raise CheckpointingException(err_msg) from e
-
- def load_sharded_objects(
- self, sharded_objects_state_dict: ShardedStateDict, checkpoint_dir: Union[str, Path]
- ):
- """Replaces all ShardedObject from a given state dict with values loaded from the
- checkpoint.
-
- Args:
- sharded_objects_state_dict (ShardedStateDict):
- sharded state dict defining what objects should be loaded.
- checkpoint_dir (Union[str, Path]): checkpoint directory
-
- Returns:
- None: sharded state dict is modified in place
- """
-
- def load_sharded_object(sh_obj: ShardedObject):
- sh_obj.data = None
- load_path = os.path.join(checkpoint_dir, f'{sh_obj.unique_key}.pt')
- try:
- if MultiStorageClientFeature.is_enabled():
- msc = MultiStorageClientFeature.import_package()
- loaded_obj = msc.torch.load(load_path)
- else:
- loaded_obj = torch.load(load_path)
- except FileNotFoundError as e:
- # Backward compatible logic: previously the save format was incorrect
- base, _ = os.path.splitext(sh_obj.unique_key)
- old_load_path = os.path.join(checkpoint_dir, f"{base}.pt")
- try:
- if MultiStorageClientFeature.is_enabled():
- msc = MultiStorageClientFeature.import_package()
- loaded_obj = msc.torch.load(old_load_path)
- else:
- loaded_obj = torch.load(old_load_path)
- except FileNotFoundError:
- err_msg = f'Object shard {load_path} not found'
- obj_subdir = os.path.join(checkpoint_dir, sh_obj.key)
- if os.path.exists(obj_subdir):
- obj_files = os.listdir(obj_subdir)
- logger.debug(
- f'{err_msg}. Object {sh_obj.key} directory content: {obj_files}'
- )
- else:
- ckpt_files = os.listdir(checkpoint_dir)
- logger.debug(
- f'{err_msg}. Object {sh_obj.key} directory does not exist. Checkpoint'
- f' directory content: {ckpt_files}'
- )
- raise CheckpointingException(err_msg) from e
- return loaded_obj
-
- return dict_list_map_inplace(load_sharded_object, sharded_objects_state_dict)
-
- def load_sharded_metadata(self, checkpoint_dir: Union[str, Path]) -> ShardedStateDict:
+def save_common(common_state_dict: StateDict, checkpoint_dir: str):
+ """Save common part of the state dict."""
+ if torch.distributed.get_rank() == 0:
+ path = os.path.join(checkpoint_dir, COMMON_STATE_FNAME)
if MultiStorageClientFeature.is_enabled():
msc = MultiStorageClientFeature.import_package()
- checkpoint_dir = msc.Path(checkpoint_dir)
+ msc.torch.save(common_state_dict, path)
else:
- checkpoint_dir = Path(checkpoint_dir)
-
- sharded_metadata = {}
- for subdir in checkpoint_dir.iterdir():
- if not subdir.is_dir():
- continue
- shard_files = list(subdir.glob('shard_*.pt'))
- if not shard_files:
- continue
- sh_objs = []
- for shard_file in shard_files:
- full_key = f'{subdir.name}/{shard_file.stem}'
- sh_objs.append(ShardedObject.empty_from_unique_key(full_key))
-
- # This is a backward-compatibility fix, where the last global shape is missing in the
- # name
- if sh_objs[0].global_shape[-1] < 0:
- max_last_offset = max(map(lambda sh_obj: sh_obj.global_offset[-1], sh_objs))
- for sh_obj in sh_objs:
- sh_obj.global_shape = (*sh_obj.global_shape[:-1], max_last_offset + 1)
+ torch.save(common_state_dict, path)
- # Update the sharded state dict
- for sh_obj in sh_objs:
- sharded_metadata[sh_obj.unique_key] = sh_obj
- return sharded_metadata
- @property
- def can_handle_sharded_objects(self):
- """This strategy can handle ShardedObjects."""
- return True
+def load_common(checkpoint_dir: str):
+ """Load common (non-sharded) objects state dict from the checkpoint.
- def check_backend_compatibility(self, loaded_version):
- pass
+ Args:
+ checkpoint_dir (Path): checkpoint directory
- def check_version_compatibility(self, loaded_version):
- pass
+ Returns:
+ StateDict: state dict with non-sharded objects from the checkpoint
+ """
+ load_path = os.path.join(checkpoint_dir, COMMON_STATE_FNAME)
+ try:
+ if MultiStorageClientFeature.is_enabled():
+ msc = MultiStorageClientFeature.import_package()
+ return msc.torch.load(load_path, map_location='cpu', weights_only=False)
+ else:
+ return torch.load(load_path, map_location='cpu', weights_only=False)
+ except FileNotFoundError as e:
+ err_msg = f'Common file {load_path} does not exist'
+ if MultiStorageClientFeature.is_enabled():
+ msc = MultiStorageClientFeature.import_package()
+ ckpt_files = [f.name for f in msc.Path(checkpoint_dir).iterdir()]
+ else:
+ ckpt_files = [f.name for f in Path(checkpoint_dir).iterdir()]
+ logger.debug(f'{err_msg}. Checkpoint directory content: {ckpt_files}')
+ raise CheckpointingException(err_msg) from e
diff --git a/megatron/core/dist_checkpointing/strategies/filesystem_async.py b/megatron/core/dist_checkpointing/strategies/filesystem_async.py
index 90e89395510..1b144822839 100644
--- a/megatron/core/dist_checkpointing/strategies/filesystem_async.py
+++ b/megatron/core/dist_checkpointing/strategies/filesystem_async.py
@@ -8,6 +8,7 @@
import os
import pickle
import queue
+import threading
from functools import partial
from heapq import heappop, heappush
from itertools import chain
@@ -48,10 +49,19 @@
_results_queue = None
-def _get_write_results_queue():
+@_disable_gc()
+def get_write_results_queue(mp_mode: str = 'spawn') -> mp.Queue:
+ """Get or create a multiprocessing queue for write results.
+
+ Args:
+ mp_mode (str): Multiprocessing context mode. Defaults to 'spawn'.
+
+ Returns:
+ mp.Queue: Queue for collecting write results.
+ """
global _results_queue
if _results_queue is None:
- ctx = mp.get_context("spawn")
+ ctx = mp.get_context(mp_mode)
_results_queue = ctx.Manager().Queue()
return _results_queue
@@ -67,7 +77,7 @@ class FileSystemWriterAsync(FileSystemWriter):
1. Call `write_data`
2. Externally start an async process with `get_save_function_and_args` and its arguments.
3. The async function `writer_proxy_func` calls `write_preloaded_data` across multiple
- processes.
+ threads (no child processes).
4. Once saving is finalized on all ranks, call `super().finish` with the results stored
in `self.writer_result`.
@@ -133,7 +143,7 @@ def gen_file(prefix=""):
file_count += 1
return file_name
- def _clone_if_needed(ten: torch.Tensor):
+ def _clone_or_dequantize_if_needed(ten: torch.Tensor):
"""Clone if we detect incontiguous storage for CPU tensors
Makes sure we perform a `clone` only if we detect incontiguous storage,
@@ -144,13 +154,18 @@ def _clone_if_needed(ten: torch.Tensor):
"""
ten = ten.detach()
if ten.device.type != "cpu":
+ # We call ``dequantize`` if we detect a quantized tensor on GPU.
+ # This is a workaround to avoid the issue of quantized tensors not being
+ # supported by the async writer.
+ if ten.device.type == "cuda" and "dequantize" in type(ten).__dict__:
+ ten = ten.dequantize()
# We do D2H later when the async_request is scheduled for both sync / async
# checkpointing
return ten
is_view = ten.untyped_storage().size() != ten.numel() * ten.itemsize
return ten.clone() if is_view else ten
- # Prepare bytes / tensor data in each bucket, which will be assigned to each writer process
+ # Prepare bytes / tensor data in each bucket, which will be assigned to each writer thread
self.write_buckets = []
for group_name, group_buckets in _split_by_separation_hint(
item_buckets, self.separation_hint
@@ -162,7 +177,7 @@ def _clone_if_needed(ten: torch.Tensor):
if item.type == WriteItemType.BYTE_IO
]
tensor_data = [
- (item, _clone_if_needed(planner.resolve_data(item)))
+ (item, _clone_or_dequantize_if_needed(planner.resolve_data(item)))
for item in bucket
if item.type != WriteItemType.BYTE_IO
]
@@ -182,7 +197,7 @@ def _clone_if_needed(ten: torch.Tensor):
len(self.write_buckets),
self.thread_count,
)
- self.results_queue = _get_write_results_queue()
+ self.results_queue = get_write_results_queue()
else:
self.results_queue = None
end = time()
@@ -203,7 +218,7 @@ def get_save_function_and_args(self) -> Tuple[Optional[Callable], Optional[Calla
return None, None, []
transform_list = [self.transforms] if hasattr(self, "transforms") else []
return (
- partial(self.write_preloaded_data_multiproc, transform_list, self.use_msc),
+ partial(self.write_preloaded_data_multithread, transform_list, self.use_msc),
partial(self.preload_tensors, self.write_buckets, True),
[torch.distributed.get_rank(), self.write_buckets, self.results_queue],
)
@@ -222,17 +237,20 @@ def preload_tensors(write_buckets: List[WriteBucket], non_blocking=True) -> List
for bucket in write_buckets:
file_name, storage_key, (bytes_data, tensor_data) = bucket
- tensor_data = [
- (item, tensor.to("cpu", non_blocking=non_blocking)) for item, tensor in tensor_data
- ]
- result.append((file_name, storage_key, (bytes_data, tensor_data)))
+ tensor_list = []
+ for item, tensor in tensor_data:
+ # we belive these tensors are detached from the model trainers
+ tensor_list.append((item, tensor.to("cpu", non_blocking=non_blocking)))
+ # This is required for `PersistentAsyncCaller` to remove reference
+ del tensor
+ result.append((file_name, storage_key, (bytes_data, tensor_list)))
if non_blocking:
torch.cuda.synchronize()
return result
@staticmethod
@_disable_gc()
- def write_preloaded_data_multiproc(
+ def write_preloaded_data_multithread(
transform_list: List[_StorageWriterTransforms],
use_msc: bool,
rank: int,
@@ -240,37 +258,40 @@ def write_preloaded_data_multiproc(
global_results_queue: mp.Queue,
) -> None:
"""
- Performs saving data to storage with multiple processes.
-
- Starts predefined number of processes and uses 2 queues to make sure the results
- are complete:
- - local_results_queue - to send the actual results
- - count_queue - small queue to mark worker as completed
+ Performs saving data to storage with multiple threads.
- Using just one queue disallowed proper exception handling.
+ Uses threads (not processes) so that this can run safely inside a daemon process
+ without spawning child processes. Uses two queues:
+ - local_results_queue - to collect write results from worker threads
+ - count_queue - to signal worker completion (task_done/join).
- This method is meant to be run in a forked subprocess.
- Triggering GC during execution leads to CUDA errors
- (cleaning up tensors owned by the parent process).
+ Triggering GC during execution can lead to CUDA errors when tensors are shared.
To prevent this, we disable the GC explicitly for this function with _disable_gc.
Args:
write_buckets (List[WriteBucket]): write plan
- global_results_queue (mp.Queue): mp.Queue to collect Dict[List[WriteResults]]
- (or an Exception) from parallel write processes to the main training process
+ global_results_queue (mp.Queue): queue to send Dict[List[WriteResults]]
+ (or an Exception) back to the main training process
Returns: None
"""
logger = logging.getLogger(__name__)
w_start = time()
write_results_or_exc: Union[dict, Exception] = dict()
- ctx = mp.get_context("spawn")
- local_results_queue = ctx.Queue()
- count_queue = ctx.JoinableQueue()
- p_list = []
+ local_results_queue: queue.Queue = queue.Queue()
+ count_queue: queue.Queue = queue.Queue()
+ thread_list: List[threading.Thread] = []
+
+ def check_local_output(local_results_or_exc, local_worker_idx):
+ if isinstance(local_results_or_exc, Exception):
+ err_msg = (
+ f"Local worker {local_worker_idx} encountered"
+ f" an error: {local_results_or_exc}"
+ )
+ logger.error(err_msg)
+ assert isinstance(local_results_or_exc, list), type(local_results_or_exc)
+
for i, write_bucket in enumerate(write_buckets):
try:
- count_queue.put(i)
-
kwargs = {
"local_proc_idx": i,
"write_bucket": write_bucket,
@@ -280,62 +301,63 @@ def write_preloaded_data_multiproc(
}
if use_msc:
- import inspect
-
- # Remove the inspect after the test_async_save.py is fixed.
signature = inspect.signature(FileSystemWriterAsync.write_preloaded_data)
if len(signature.parameters) > 6:
- kwargs["use_msc"] = use_msc
-
- p_list.append(
- ctx.Process(
+ kwargs['use_msc'] = use_msc
+ # Parallel writers: spawn threads for all but the last bucket
+ if i < len(write_buckets) - 1:
+ count_queue.put(i)
+ t = threading.Thread(
target=partial(FileSystemWriterAsync.write_preloaded_data, transform_list),
kwargs=kwargs,
)
- )
+ thread_list.append(t)
+ else:
+ kwargs['count_queue'] = None
+ kwargs['results_queue'] = None
+ logger.debug('FileSystemWriterAsync: main worker started')
+ local_output = FileSystemWriterAsync.write_preloaded_data(
+ transform_list, **kwargs
+ )
+ if local_output is not None:
+ logger.debug(
+ 'FileSystemWriterAsync: main worker results successfully collected'
+ )
+ check_local_output(local_output[1], local_output[0])
+ write_results_or_exc[local_output[0]] = local_output[1]
+
except Exception as e:
- err_msg = f"An error is caught while a proc {i} is created, error: {e}"
+ err_msg = f"An error is caught while starting worker {i}, error: {e}"
logger.error(err_msg)
write_results_or_exc = RuntimeError(err_msg)
- if not isinstance(write_results_or_exc, Exception):
- for p in p_list:
- p.start()
+ if not isinstance(write_results_or_exc, Exception) and len(thread_list) > 0:
+ for t in thread_list:
+ t.start()
logger.debug("FileSystemWriterAsync: collecting worker results...")
- # To make sure all nodes are completed
count_queue.join()
- # At this point, all workers completed, so the queue should have exactly
- # `len(write_buckets)` items
- for proc_idx in range(len(write_buckets)):
+ for _ in range(len(write_buckets) - 1):
try:
local_proc_idx, local_results_or_exc = local_results_queue.get()
except queue.Empty:
write_results_or_exc = RuntimeError(
"Unexpected empty `local_results_queue`"
- f" (got only {proc_idx}/{len(write_buckets)} items)"
+ f" (expected {len(write_buckets) - 1} items)"
)
break
else:
- if isinstance(local_results_or_exc, Exception):
- err_msg = (
- f"Local process {local_proc_idx} encountered"
- f" an error: {local_results_or_exc}"
- )
- logger.error(err_msg)
- write_results_or_exc = local_results_or_exc
- break
- assert isinstance(local_results_or_exc, list), type(local_results_or_exc)
+ check_local_output(local_results_or_exc, local_proc_idx)
write_results_or_exc[local_proc_idx] = local_results_or_exc
- p_list[local_proc_idx].join()
-
- logger.debug("FileSystemWriterAsync: collected worker results successfully")
+ for t in thread_list:
+ t.join()
+ logger.debug('FileSystemWriterAsync: collected worker results successfully')
global_results_queue.put(write_results_or_exc)
w_end = time()
- logger.debug(f"{w_end}, rank: {rank}, write(sync,parallel): {w_end - w_start}")
+ logger.debug(f"{w_end}, rank: {rank}, write(sync,threads): {w_end - w_start}")
@staticmethod
@_disable_gc()
@@ -343,23 +365,22 @@ def write_preloaded_data(
transform_list: List[_StorageWriterTransforms],
local_proc_idx: int,
write_bucket: WriteBucket,
- results_queue: mp.SimpleQueue,
- count_queue: mp.JoinableQueue,
+ results_queue: Optional[queue.Queue],
+ count_queue: Optional[queue.Queue],
use_fsync: bool,
**kwargs,
- ) -> None:
+ ) -> Union[Tuple[int, Exception], None]:
"""
- Performs actual data saving to storage.
+ Performs actual data saving to storage (used by worker threads).
Args:
- local_proc_idx (int): index of a local process that performs writing
+ local_proc_idx (int): index of the worker that performs writing
write_bucket (WriteBucket): data to write to storage
- results_queue (mp.Queue): queue to return the write results
- to the proxy checkpoint process.
- count_queue (mp.JoinableQueue): queue to marks worker task as completed
+ results_queue (queue.Queue): queue to return the write results
+ count_queue (queue.Queue): queue to signal worker task completion (get + task_done)
use_fsync (bool): if True, calls os.fsync at the end of saving
- Returns: None, the write result are put into the `queue`
+ Returns: None when running in a worker (results put in queue); result tuple when main worker
"""
logger = logging.getLogger(__name__)
logger.debug(f"{local_proc_idx} started")
@@ -405,17 +426,19 @@ def write_preloaded_data(
except Exception as e:
logger.debug(f"{local_proc_idx} failed")
local_output = (local_proc_idx, e) # type: ignore[assignment]
-
- results_queue.put(local_output)
- # Signal this process is done.
- count_queue.get()
- count_queue.task_done()
+ if results_queue is not None:
+ results_queue.put(local_output)
+ if count_queue is not None:
+ # Signal this process is done.
+ count_queue.get()
+ count_queue.task_done()
mem_after = _process_memory()
logger.debug(
f"{local_proc_idx} consumed: {mem_after - mem_before},"
f" before: {mem_before}, after: {mem_after}"
)
+ return local_output
def write_data(self, plan: SavePlan, planner: SavePlanner) -> Future[List[WriteResult]]:
"""Write all items from ``plan``."""
@@ -427,7 +450,7 @@ def retrieve_write_results(self) -> Union[List[WriteResult], WRAPPED_EXCEPTION]:
into a single results lists. Includes error check.
Returns (Union(List[WriteResult], WRAPPED_EXCEPTION): the list of write results
- from all local processes performing the save, or a WRAPPED_EXCEPTION if
+ from all local workers (threads) performing the save, or a WRAPPED_EXCEPTION if
an exception was raised during the writing process.
"""
assert self.write_buckets is not None
diff --git a/megatron/core/dist_checkpointing/strategies/fully_parallel.py b/megatron/core/dist_checkpointing/strategies/fully_parallel.py
index be3b941c07e..a85efdaa10a 100644
--- a/megatron/core/dist_checkpointing/strategies/fully_parallel.py
+++ b/megatron/core/dist_checkpointing/strategies/fully_parallel.py
@@ -85,15 +85,22 @@ def __init__(
self.cached_distribution: Optional[ShardDistribution] = None
- def async_save(self, sharded_state_dict: ShardedStateDict, checkpoint_dir: Path):
+ def async_save(
+ self,
+ sharded_state_dict: ShardedStateDict,
+ checkpoint_dir: Path,
+ async_strategy: str = "nvrx",
+ ):
+ """ """
if not isinstance(self.base_strategy, AsyncSaveShardedStrategy):
raise CheckpointingException(
f'Cannot apply async_save to non-async base strategy {self.base_strategy}'
)
self.apply_saving_parallelization(sharded_state_dict)
- return self.base_strategy.async_save(sharded_state_dict, checkpoint_dir)
+ return self.base_strategy.async_save(sharded_state_dict, checkpoint_dir, async_strategy)
def save(self, sharded_state_dict: ShardedStateDict, checkpoint_dir: Path):
+ """ """
self.apply_saving_parallelization(sharded_state_dict)
return self.base_strategy.save(sharded_state_dict, checkpoint_dir)
@@ -135,6 +142,7 @@ def apply_saving_parallelization(self, sharded_state_dict: ShardedStateDict) ->
@property
def can_handle_sharded_objects(self):
+ """ """
return self.base_strategy.can_handle_sharded_objects
@@ -185,7 +193,12 @@ def __init__(
self.cached_global_metadata: Optional[Metadata] = None
@debug_time("FullyParallelLoadStrategyWrapper.load", logger)
- def load(self, sharded_state_dict: ShardedStateDict, checkpoint_dir: Path) -> StateDict:
+ def load(
+ self,
+ sharded_state_dict: ShardedStateDict,
+ checkpoint_dir: Path,
+ async_strategy: str = "nvrx",
+ ) -> StateDict:
"""Distributes the load and calls underlying strategy only for parts of the state dict.
Steps:
@@ -218,7 +231,7 @@ def load(self, sharded_state_dict: ShardedStateDict, checkpoint_dir: Path) -> St
loaded_state_dict = {}
if get_pg_size(self.parallelization_group) <= 1:
- return self.base_strategy.load(sharded_state_dict, checkpoint_dir)
+ return self.base_strategy.load(sharded_state_dict, checkpoint_dir, async_strategy)
# Step 1 and 2: exchange load metadata and distribute the load
with debug_time("self.apply_loading_parallelization", logger):
@@ -245,11 +258,13 @@ def load(self, sharded_state_dict: ShardedStateDict, checkpoint_dir: Path) -> St
), "sharded_state_dict is not empty after deferring tensors and objects"
with debug_time("base_load_ShardedObjects", logger):
# Load sharded objects first
- loaded_objects = self.base_strategy.load(to_load_objects, checkpoint_dir)
+ loaded_objects = self.base_strategy.load(
+ to_load_objects, checkpoint_dir, async_strategy
+ )
with debug_time("base_load_ShardedTensors", logger):
# Load sharded tensors separately
- loaded_tensors = self.base_strategy.load(to_load_shards, checkpoint_dir)
+ loaded_tensors = self.base_strategy.load(to_load_shards, checkpoint_dir, async_strategy)
with debug_time("self.exchange_loaded_tensors", logger):
@@ -390,18 +405,23 @@ def apply_loading_parallelization(
@property
def can_handle_sharded_objects(self):
+ """ """
return self.base_strategy.can_handle_sharded_objects
def load_tensors_metadata(self, checkpoint_dir: Path):
+ """ """
return self.base_strategy.load_tensors_metadata(checkpoint_dir)
def load_sharded_metadata(self, checkpoint_dir: Path):
+ """ """
return self.base_strategy.load_sharded_metadata(checkpoint_dir)
def check_backend_compatibility(self, loaded_version):
+ """ """
return self.base_strategy.check_backend_compatibility(loaded_version)
def check_version_compatibility(self, loaded_version):
+ """ """
return self.base_strategy.check_version_compatibility(loaded_version)
diff --git a/megatron/core/dist_checkpointing/strategies/torch.py b/megatron/core/dist_checkpointing/strategies/torch.py
index a5b6c009ba4..000aa4d9265 100644
--- a/megatron/core/dist_checkpointing/strategies/torch.py
+++ b/megatron/core/dist_checkpointing/strategies/torch.py
@@ -5,6 +5,7 @@
import os
import pickle
import warnings
+from abc import ABC
from collections import defaultdict
from contextlib import contextmanager
from itertools import product
@@ -37,7 +38,6 @@
from torch.distributed.checkpoint.metadata import Metadata
from torch.distributed.checkpoint.planner_helpers import _create_write_items
-from ...utils import get_torch_version, is_torch_min_version
from ..core import CheckpointingException
from ..dict_utils import nested_values
from ..mapping import (
@@ -55,10 +55,16 @@
StrategyAction,
register_default_strategy,
)
-from .cached_metadata_filesystem_reader import CachedMetadataFileSystemReader
from .checkpointable import CheckpointableShardedTensor, LocalShardsContainer
-from .filesystem_async import FileSystemWriterAsync
-from .state_dict_saver import save_state_dict_async_finalize, save_state_dict_async_plan
+
+try:
+ from nvidia_resiliency_ext.checkpointing.async_ckpt.core import AsyncRequest as NVRxAsyncRequest
+ from nvidia_resiliency_ext.checkpointing.async_ckpt.state_dict_saver import (
+ CheckpointMetadataCache,
+ )
+except (ImportError, ModuleNotFoundError):
+ CheckpointMetadataCache = ABC
+ NVRxAsyncRequest = ABC
try:
if not torch.cuda.is_available():
@@ -103,7 +109,7 @@ def register_default_torch_strategies():
StrategyAction.LOAD_SHARDED, 'torch_dist', 1, TorchDistLoadShardedStrategy()
)
register_default_strategy(
- StrategyAction.SAVE_SHARDED, 'torch_dist', 1, TorchDistSaveShardedStrategy('torch_dist', 1)
+ StrategyAction.SAVE_SHARDED, 'torch_dist', 1, TorchDistSaveShardedStrategy()
)
@@ -278,6 +284,8 @@ def _mcore_to_dcp_compatible_tensor(sh_tens: List[ShardedTensor]) -> TorchSharde
- if `allow_shape_mismatch` is True, the data is initialized with zeros
prior to loading (not all parts of the tensor will be read from the checkpoint)
"""
+ from ...utils import is_torch_min_version
+
assert all(isinstance(sh_ten, ShardedTensor) for sh_ten in sh_tens), sh_tens
for sh_ten in sh_tens:
if sh_ten.data is None:
@@ -427,6 +435,8 @@ def __init__(
) -> None:
# `dedup_replicated_tensors` was deprecated in 2.3; this check avoids warnings
# during saving.
+ from ...utils import get_torch_version
+
if get_torch_version() <= PkgVersion("2.2"):
kwargs['dedup_replicated_tensors'] = dedup_replicated_tensors
super().__init__(*args, **kwargs)
@@ -596,10 +606,10 @@ class TorchDistSaveShardedStrategy(AsyncSaveShardedStrategy):
def __init__(
self,
- backend: str,
- version: int,
+ backend: str = "torch_dist",
+ version: int = 1,
keep_only_main_replica: bool = True,
- thread_count: int = 2,
+ thread_count: int = 1,
cached_metadata: bool = False,
separation_hint: Optional[str] = None,
):
@@ -635,14 +645,21 @@ def __init__(
self.validated_cache_reuse: bool = False
# The knob to enable cached metadata communication in saving
self.use_cached_ckpt_structure: bool = cached_metadata
+ # Metadata cache encapsulating all cached plan/metadata state (from nvidia-resiliency-ext)
+ self._metadata_cache: Optional[CheckpointMetadataCache] = None
+ # FullyParallel wrappers and load paths may seed save-time metadata reuse here.
+ self.cached_global_metadata: Optional[Metadata] = None
self.separation_hint = separation_hint
self.validated_loaded_metadata_reuse = False
def async_save(
- self, sharded_state_dict: ShardedStateDict, checkpoint_dir: Path
- ) -> AsyncRequest:
+ self,
+ sharded_state_dict: ShardedStateDict,
+ checkpoint_dir: Path,
+ async_strategy: str = "nvrx",
+ ) -> AsyncRequest | NVRxAsyncRequest:
"""Translates MCore ShardedTensors to PyT ShardedTensors & saves in PyT Distributed format.
Args:
@@ -651,6 +668,12 @@ def async_save(
Returns: None
"""
+ if async_strategy == "mcore":
+ logger.warning(
+ "MCore's async save is deprecated and will be removed in the future releases. "
+ "Please, use NVRx async solution by setting `async_strategy` to `nvrx`."
+ )
+
# Translate the state dict
(sharded_state_dict, flat_mapping, rename_mapping) = (
_replace_state_dict_keys_with_sharded_keys(
@@ -658,104 +681,144 @@ def async_save(
)
)
pyt_state_dict = mcore_to_pyt_state_dict(sharded_state_dict, False)
+
+ if self.separation_hint is not None and self.thread_count <= 1:
+ self.thread_count = 2
+
+ # Get async modules
+ async_strategy, modules = get_async_strategy(async_strategy)
+ async_writer = modules["FileSystemWriterAsync"]
+ save_state_dict_async_plan = modules["save_state_dict_async_plan"]
+ if async_strategy == "nvrx":
+ checkpointable_metadata_cache = modules["CheckpointMetadataCache"]
+
+ async_writer_kwargs = {}
+ state_dict_saver_kwargs = {}
+
+ if async_strategy == "nvrx":
+ if self._metadata_cache is None:
+ self._metadata_cache = checkpointable_metadata_cache()
+ if self.cached_global_metadata is not None:
+ self._metadata_cache.set_cached_global_metadata(self.cached_global_metadata)
+ # Define additional arguments
+ async_writer_kwargs["use_cached_data_structure"] = self.use_cached_ckpt_structure
+ state_dict_saver_kwargs["enable_cache"] = self.use_cached_ckpt_structure
+ state_dict_saver_kwargs["metadata_cache"] = self._metadata_cache
+ else:
+ # MCore's async implementation
+ args_cached_plans = None
+ loaded_all_plans = None
+ if self.use_cached_ckpt_structure:
+ loaded_all_plans = getattr(self.cached_global_metadata, "all_local_plans", None)
+ if loaded_all_plans is None:
+ logger.debug(
+ "no all_local_plans in metadata - can't verify global metadata reuse..."
+ )
+ args_cached_plans = (
+ self.cached_central_plan,
+ self.cached_local_plan,
+ self.validated_cache_reuse,
+ )
+ state_dict_saver_kwargs["cached_ckpt_structure"] = args_cached_plans
+ state_dict_saver_kwargs["loaded_all_plans"] = loaded_all_plans
+
# Use PyT saving mechanism
- writer = FileSystemWriterAsync(
+ writer = async_writer(
checkpoint_dir,
separation_hint=self.separation_hint,
thread_count=self.thread_count,
use_msc=MultiStorageClientFeature.is_enabled(),
+ **async_writer_kwargs,
)
+
# This should be set differently if we run in a smaller process group than the default
coordinator = 0
- # Try twice to validate the generated `central_plan` is the same across iterations
- # If so, reuse `cached_central_plan` and `cached_global_metadata`
- # From the 3rd iteration, `save_state_dict_async_plan` will not generate `global_metadata`
- # (return None) so `self.cached_global_metadata` is reused
- args_cached_plans = None
- loaded_all_plans = None
- if self.use_cached_ckpt_structure:
- loaded_all_plans = getattr(self.cached_global_metadata, "all_local_plans", None)
- if loaded_all_plans is None:
- logger.debug(
- "no all_local_plans in metadata - can't verify global metadata reuse..."
- )
-
- args_cached_plans = (
- self.cached_central_plan,
- self.cached_local_plan,
- self.validated_cache_reuse,
- )
-
- (
- save_state_dict_ret,
- self.cached_central_plan,
- self.cached_local_plan,
- self.validated_cache_reuse,
- self.validated_loaded_metadata_reuse,
- ) = save_state_dict_async_plan(
+ save_state_dict_ret = save_state_dict_async_plan(
pyt_state_dict,
writer,
None,
coordinator,
+ # flatten_sharded_tensors=False: MCore doesn't use nested ShardedTensors (FSDP 2D),
+ # so skip the expensive traverse_state_dict copy in _flatten_sharded_tensors
planner=MCoreSavePlanner(
- dedup_replicated_tensors=not self.keep_only_main_replica, flatten_state_dict=False
+ dedup_replicated_tensors=not self.keep_only_main_replica,
+ flatten_state_dict=False,
+ flatten_sharded_tensors=False,
),
- cached_ckpt_structure=args_cached_plans,
- loaded_all_plans=loaded_all_plans,
+ **state_dict_saver_kwargs,
)
- rank = torch.distributed.get_rank()
- if self.use_cached_ckpt_structure:
- if (
- loaded_all_plans
- and self.cached_global_metadata
- and self.validated_loaded_metadata_reuse
- ):
- if coordinator == rank:
- logger.debug(
- f"rank: {rank}, reuse global metadata from loaded"
- f" .metadata, {save_state_dict_ret[1]}"
- )
- save_state_dict_ret = list(save_state_dict_ret)
- save_state_dict_ret[1] = self.cached_global_metadata
-
- elif self.validated_cache_reuse:
- logger.debug(f"rank: {rank}, cache validated")
- if save_state_dict_ret[1]: # when global_metadata is not cached
- self.cached_global_metadata = save_state_dict_ret[1] # Cache Metadata
- # Only Coordinator rank holds cached global_metadata
- # (None is returned for global_metadata)
- elif coordinator == rank:
- logger.debug(
- f"rank: {rank}, reuse global metadata cached from previous"
- f" save iteration, {save_state_dict_ret[1]}"
- )
- save_state_dict_ret = list(save_state_dict_ret)
- save_state_dict_ret[1] = self.cached_global_metadata
-
- return self._get_save_and_finalize_callbacks(writer, save_state_dict_ret)
- def _get_save_and_finalize_callbacks(self, writer, save_state_dict_ret) -> AsyncRequest:
+ if async_strategy == "mcore":
+ # MCore's async implementation
+ (
+ save_state_dict_ret,
+ self.cached_central_plan,
+ self.cached_local_plan,
+ self.validated_cache_reuse,
+ self.validated_loaded_metadata_reuse,
+ ) = save_state_dict_ret
+
+ rank = torch.distributed.get_rank()
+ if self.use_cached_ckpt_structure:
+ if (
+ loaded_all_plans
+ and self.cached_global_metadata
+ and self.validated_loaded_metadata_reuse
+ ):
+ if coordinator == rank:
+ logger.debug(
+ f"rank: {rank}, reuse global metadata from loaded"
+ f" .metadata, {save_state_dict_ret[1]}"
+ )
+ save_state_dict_ret = list(save_state_dict_ret)
+ save_state_dict_ret[1] = self.cached_global_metadata
+
+ elif self.validated_cache_reuse:
+ logger.debug(f"rank: {rank}, cache validated")
+ if save_state_dict_ret[1]: # when global_metadata is not cached
+ self.cached_global_metadata = save_state_dict_ret[1] # Cache Metadata
+ # Only Coordinator rank holds cached global_metadata
+ # (None is returned for global_metadata)
+ elif coordinator == rank:
+ logger.debug(
+ f"rank: {rank}, reuse global metadata cached from previous"
+ f" save iteration, {save_state_dict_ret[1]}"
+ )
+ save_state_dict_ret = list(save_state_dict_ret)
+ save_state_dict_ret[1] = self.cached_global_metadata
+
+ return self._get_save_and_finalize_callbacks(writer, save_state_dict_ret, async_strategy)
+
+ def _get_save_and_finalize_callbacks(
+ self, writer, save_state_dict_ret, async_strategy
+ ) -> AsyncRequest | NVRxAsyncRequest:
save_fn_args = writer.get_save_function_and_args()
save_fn, preload_fn, save_args = save_fn_args
+ # get async modules
+ _, modules = get_async_strategy(async_strategy)
+ async_request = modules["AsyncRequest"]
+ save_state_dict_async_finalize = modules["save_state_dict_async_finalize"]
+
def finalize_fn():
save_state_dict_async_finalize(*save_state_dict_ret)
- return AsyncRequest(save_fn, save_args, [finalize_fn], preload_fn=preload_fn)
+ return async_request(save_fn, save_args, [finalize_fn], preload_fn=preload_fn)
def can_handle_sharded_objects(self):
return True
def _get_filesystem_reader(
- checkpoint_dir: Union[str, Path], cache_metadata: bool = False
+ checkpoint_dir: Union[str, Path], cache_metadata: bool = False, async_strategy: str = "nvrx"
) -> FileSystemReader:
if MultiStorageClientFeature.is_enabled():
msc = MultiStorageClientFeature.import_package()
return msc.torch.MultiStorageFileSystemReader(checkpoint_dir, thread_count=2)
if cache_metadata:
- return CachedMetadataFileSystemReader(checkpoint_dir)
+ _, module = get_async_strategy(async_strategy, module="CachedMetadataFileSystemReader")
+ return module(checkpoint_dir, cache_metadata=cache_metadata)
return FileSystemReader(checkpoint_dir)
@@ -763,11 +826,17 @@ def _get_filesystem_reader(
class TorchDistLoadShardedStrategy(LoadShardedStrategy):
"""Basic load strategy for the PyT Distributed format."""
- def __init__(self):
+ def __init__(self, cache_metadata: bool = False):
self.cached_global_metadata: Optional[Metadata] = None
+ self.cache_metadata = cache_metadata
super().__init__()
- def load(self, sharded_state_dict: ShardedStateDict, checkpoint_dir: Path) -> StateDict:
+ def load(
+ self,
+ sharded_state_dict: ShardedStateDict,
+ checkpoint_dir: Path,
+ async_strategy: str = "nvrx",
+ ) -> StateDict:
"""Translates MCore ShardedTensors to PyT ShardedTensors & loads from PyT Distributed fmt.
Args:
@@ -795,19 +864,24 @@ def load(self, sharded_state_dict: ShardedStateDict, checkpoint_dir: Path) -> St
)
pyt_state_dict = mcore_to_pyt_state_dict(sharded_state_dict, True)
# Load PyT Distributed format
- fsr = _get_filesystem_reader(checkpoint_dir, cache_metadata=True)
+ fsr = _get_filesystem_reader(
+ checkpoint_dir, cache_metadata=self.cache_metadata, async_strategy=async_strategy
+ )
checkpoint.load_state_dict(
pyt_state_dict,
fsr,
planner=MCoreLoadPlanner(
shapes_validation_sharded_tensors=flexible_shape_sharded_tensors,
allow_shape_mismatch_sharded_tensors=allow_shape_mismatch_sharded_tensors,
+ flatten_state_dict=False,
+ flatten_sharded_tensors=False,
),
)
- self.cached_global_metadata = (
- fsr.read_metadata()
- ) # no storage interaction thanks to caching
+ if self.cache_metadata:
+ self.cached_global_metadata = (
+ fsr.read_metadata()
+ ) # no storage interaction thanks to caching
pyt_state_dict = cast(
Dict[str, Union[TorchShardedTensor, List[io.BytesIO]]], pyt_state_dict
@@ -863,6 +937,7 @@ def remove_sharded_tensors(self, checkpoint_dir: str, key_prefix: str):
4. resaves the new metadata and removes the old metadata
5. removes the relevant files
"""
+ from ...utils import is_torch_min_version
assert is_torch_min_version(
"2.3.0"
@@ -945,3 +1020,89 @@ def check_backend_compatibility(self, loaded_version):
def check_version_compatibility(self, loaded_version):
pass # TODO
+
+
+def get_async_strategy(async_strategy: str = "nvrx", module: str = None) -> tuple:
+ """Returns async strategy and related async imported modules"""
+ if async_strategy == "nvrx":
+ try:
+ # nvrx async imports
+ from nvidia_resiliency_ext.checkpointing.async_ckpt.cached_metadata_filesystem_reader import ( # pylint: disable=line-too-long
+ CachedMetadataFileSystemReader,
+ )
+ from nvidia_resiliency_ext.checkpointing.async_ckpt.core import (
+ AsyncCallsQueue,
+ AsyncRequest,
+ )
+ from nvidia_resiliency_ext.checkpointing.async_ckpt.filesystem_async import (
+ FileSystemWriterAsync,
+ _results_queue,
+ get_write_results_queue,
+ )
+ from nvidia_resiliency_ext.checkpointing.async_ckpt.state_dict_saver import (
+ CheckpointMetadataCache,
+ save_state_dict_async_finalize,
+ save_state_dict_async_plan,
+ )
+
+ imports = {
+ "AsyncCallsQueue": AsyncCallsQueue,
+ "AsyncRequest": AsyncRequest,
+ "CachedMetadataFileSystemReader": CachedMetadataFileSystemReader,
+ "CheckpointMetadataCache": CheckpointMetadataCache,
+ "FileSystemWriterAsync": FileSystemWriterAsync,
+ "_results_queue": _results_queue,
+ "get_write_results_queue": get_write_results_queue,
+ "save_state_dict_async_finalize": save_state_dict_async_finalize,
+ "save_state_dict_async_plan": save_state_dict_async_plan,
+ }
+ async_strategy = "nvrx"
+ except (ImportError, ModuleNotFoundError):
+ raise ModuleNotFoundError(
+ "nvidia-resiliency-ext package is not installed. "
+ "Please, install nvidia-resiliency-ext package or set `async_strategy` to `mcore` "
+ "to enable async save strategy."
+ )
+ elif async_strategy == "mcore":
+ # do mcore async imports
+ imports = _import_mcore_async()
+ async_strategy = "mcore"
+ else:
+ raise TypeError(
+ f"async_strategy {async_strategy} is not supported. Available strategies: nvrx, mcore."
+ )
+
+ modules = imports if not module else imports[module]
+
+ return async_strategy, modules
+
+
+def _import_mcore_async() -> dict:
+ """Imports mcore's async modules"""
+ from megatron.core.dist_checkpointing.strategies.async_utils import (
+ AsyncCallsQueue,
+ AsyncRequest,
+ )
+ from megatron.core.dist_checkpointing.strategies.cached_metadata_filesystem_reader import (
+ CachedMetadataFileSystemReader,
+ )
+ from megatron.core.dist_checkpointing.strategies.filesystem_async import (
+ FileSystemWriterAsync,
+ _results_queue,
+ get_write_results_queue,
+ )
+ from megatron.core.dist_checkpointing.strategies.state_dict_saver import (
+ save_state_dict_async_finalize,
+ save_state_dict_async_plan,
+ )
+
+ return {
+ "AsyncCallsQueue": AsyncCallsQueue,
+ "AsyncRequest": AsyncRequest,
+ "CachedMetadataFileSystemReader": CachedMetadataFileSystemReader,
+ "FileSystemWriterAsync": FileSystemWriterAsync,
+ "_results_queue": _results_queue,
+ "get_write_results_queue": get_write_results_queue,
+ "save_state_dict_async_finalize": save_state_dict_async_finalize,
+ "save_state_dict_async_plan": save_state_dict_async_plan,
+ }
diff --git a/megatron/core/dist_checkpointing/validation.py b/megatron/core/dist_checkpointing/validation.py
index 48f2bda8737..89ecba1a968 100644
--- a/megatron/core/dist_checkpointing/validation.py
+++ b/megatron/core/dist_checkpointing/validation.py
@@ -1,16 +1,19 @@
# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
import logging
-import os
from collections import Counter, defaultdict
from enum import Enum
+from pathlib import Path
from typing import TYPE_CHECKING, List, Optional, Set, Tuple, Union
import numpy as np
import torch
from megatron.core.dist_checkpointing import ShardedTensor
-from megatron.core.dist_checkpointing.core import CheckpointingException, maybe_load_config
+from megatron.core.dist_checkpointing.core import (
+ CheckpointingException,
+ check_is_distributed_checkpoint,
+)
from megatron.core.dist_checkpointing.dict_utils import diff, extract_matching_values, nested_values
from megatron.core.dist_checkpointing.mapping import (
CommonStateDict,
@@ -19,15 +22,6 @@
ShardedStateDict,
is_main_replica,
)
-from megatron.core.dist_checkpointing.strategies.base import (
- LoadCommonStrategy,
- LoadShardedStrategy,
- SaveCommonStrategy,
- SaveShardedStrategy,
- StrategyAction,
- get_default_strategy,
-)
-from megatron.core.msc_utils import MultiStorageClientFeature
if TYPE_CHECKING:
from megatron.core.dist_checkpointing.serialization import CkptShardedMetadata
@@ -199,60 +193,17 @@ def validate_integrity_and_strict_load(
return sharded_state_dict, missing_keys, unexpected_keys
-def verify_checkpoint_and_load_strategy(
- checkpoint_dir: str,
- sharded_strategy: Union[LoadShardedStrategy, Tuple[str, int], None] = None,
- common_strategy: Union[LoadCommonStrategy, Tuple[str, int], None] = None,
-) -> Tuple[LoadShardedStrategy, LoadCommonStrategy]:
- """Verifies if checkpoint metadata exists and matches given strategies.
-
- If no strategies are passed, they are determined based on the checkpoint metadata.
+def verify_checkpoint(checkpoint_dir: str):
+ """Verifies if checkpoint exists.
Args:
checkpoint_dir (str): checkpoint directory
- sharded_strategy (LoadShardedStrategy, Tuple[str, int], optional): sharded load strategy to be verified
- if compatible with the checkpoint content. If None, the default sharded load strategy
- for the checkpoint backend will be returned.
- common_strategy (LoadCommonStrategy, Tuple[str, int], optional): common load strategy to be verified
- if compatible with the checkpoint content. If None, the default common load strategy
- for the checkpoint backend will be returned.
"""
- isdir = True
- if MultiStorageClientFeature.is_enabled():
- msc = MultiStorageClientFeature.import_package()
- isdir = msc.os.path.isdir(str(checkpoint_dir), strict=False)
- else:
- isdir = os.path.isdir(checkpoint_dir)
- if not isdir:
- raise CheckpointingException(f"Checkpoint directory {checkpoint_dir} does not exist")
-
- saved_config = maybe_load_config(checkpoint_dir)
- if saved_config is None:
- raise CheckpointingException(f"{checkpoint_dir} is not a distributed checkpoint")
-
- if sharded_strategy is None:
- sharded_strategy = get_default_strategy(
- StrategyAction.LOAD_SHARDED,
- saved_config.sharded_backend,
- saved_config.sharded_backend_version,
- )
- elif isinstance(sharded_strategy, tuple):
- sharded_strategy = get_default_strategy(StrategyAction.LOAD_SHARDED, *sharded_strategy)
-
- if common_strategy is None:
- common_strategy = get_default_strategy(
- StrategyAction.LOAD_COMMON,
- saved_config.common_backend,
- saved_config.common_backend_version,
- )
- elif isinstance(common_strategy, tuple):
- sharded_strategy = get_default_strategy(StrategyAction.LOAD_COMMON, *common_strategy)
+ if not Path(checkpoint_dir).exists():
+ raise CheckpointingException(f'Checkpoint directory {checkpoint_dir} does not exist')
- sharded_strategy.check_backend_compatibility(saved_config.sharded_backend)
- sharded_strategy.check_version_compatibility(saved_config.sharded_backend_version)
- common_strategy.check_backend_compatibility(saved_config.common_backend)
- common_strategy.check_version_compatibility(saved_config.common_backend_version)
- return sharded_strategy, common_strategy
+ if not check_is_distributed_checkpoint(checkpoint_dir):
+ raise CheckpointingException(f'{checkpoint_dir} is not a distributed checkpoint')
def adjust_non_strict_load(
@@ -433,19 +384,25 @@ def validate_sharding_integrity(
for rank, rank_shardings in enumerate(global_metadata):
for sharding in rank_shardings:
key_shardings[sharding.key].append((rank, sharding))
+ errors = []
for key, shardings in key_shardings.items():
if isinstance(shardings[0][1], ShardedObject):
- _validate_objects_for_key(shardings)
+ errors.extend(_validate_objects_for_key(shardings))
else:
- _validate_sharding_for_key(shardings)
+ errors.extend(_validate_sharding_for_key(shardings))
+ if errors:
+ errors = '\n'.join(str(e) for e in errors)
+ raise CheckpointingException(f'Invalid sharding pattern validation. Errors: {errors}')
-def _validate_sharding_for_key(rank_sharding: List[Tuple[int, ShardedTensor]]):
+
+def _validate_sharding_for_key(
+ rank_sharding: List[Tuple[int, ShardedTensor]]
+) -> List[CheckpointingException]:
some_rank_shard = rank_sharding[0][1]
global_shape = some_rank_shard.global_shape
local_shape = some_rank_shard.local_shape
dtype = some_rank_shard.dtype
- has_flattened_range = some_rank_shard.flattened_range is not None
has_regular_sharding_grid = some_rank_shard.has_regular_grid
for rank, sharding in rank_sharding:
assert sharding.dtype == dtype, (sharding.dtype, dtype, some_rank_shard)
@@ -465,16 +422,21 @@ def _validate_sharding_for_key(rank_sharding: List[Tuple[int, ShardedTensor]]):
some_rank_shard,
)
+ errors = []
if not has_regular_sharding_grid:
# In case of uneven sharding we defer the validation to DCP
- return
+ return errors
shard_access_cnt = _compute_shards_access(rank_sharding)
if not torch.all(shard_access_cnt == 1):
- raise CheckpointingException(
- f"Invalid access pattern for {rank_sharding[0][1]}: {shard_access_cnt}"
+ errors.append(
+ CheckpointingException(
+ f'Invalid access pattern for {rank_sharding[0][1]}: {shard_access_cnt}'
+ )
)
+ return errors
+
def _compute_shards_access(rank_sharding):
shard_access_cnt = torch.zeros(
@@ -486,20 +448,24 @@ def _compute_shards_access(rank_sharding):
return shard_access_cnt
-def _validate_objects_for_key(sharded_objects: List[ShardedObject]):
+def _validate_objects_for_key(sharded_objects: List[ShardedObject]) -> List[CheckpointingException]:
"""Ensure uniqueness of saved objects."""
unique_keys = [
sh_obj.unique_key for _, sh_obj in sharded_objects if is_main_replica(sh_obj.replica_id)
]
+ errors = []
if len(unique_keys) != len(set(unique_keys)):
duplicates = {k: cnt for k, cnt in Counter(unique_keys).items() if cnt > 1}
logger.error(f"Duplicate ShardedObject keys and counts: {duplicates}")
- raise CheckpointingException(f"Duplicate ShardedObject keys: {list(duplicates.keys())}")
+ errors.append(
+ CheckpointingException(f'Duplicate ShardedObject keys: {list(duplicates.keys())}')
+ )
expected_shard_num = np.prod(sharded_objects[0][1].global_shape)
if len(unique_keys) != expected_shard_num:
err_msg = f"Invalid access pattern: {expected_shard_num - len(unique_keys)} ShardedObject are missing."
logger.error(f"{err_msg} Existing shards: {unique_keys}")
- raise CheckpointingException(err_msg)
+ errors.append(CheckpointingException(err_msg))
+ return errors
def determine_global_metadata(
@@ -517,29 +483,3 @@ def determine_global_metadata(
global_metadata = [None] * torch.distributed.get_world_size()
torch.distributed.all_gather_object(global_metadata, local_metadata)
return local_metadata, global_metadata # type: ignore[return-value]
-
-
-def validate_sharded_objects_handling(
- sharded_strategy: Union[SaveShardedStrategy, LoadShardedStrategy],
- common_strategy: Union[SaveCommonStrategy, LoadCommonStrategy],
-) -> None:
- """Checks if either of the passed strategies can handle sharded objects.
-
- Args:
- sharded_strategy (Union[SaveShardedStrategy, LoadShardedStrategy]): sharded strategy used for saving/loading
- common_strategy (Union[SaveCommonStrategy, LoadCommonStrategy]): common strategy used for saving/loading
-
- Returns:
- None
-
- Raises:
- CheckpointingException: if both strategies can't handle ShardedObjects
- """
- if (
- not sharded_strategy.can_handle_sharded_objects
- and not common_strategy.can_handle_sharded_objects
- ):
- raise CheckpointingException(
- f"Either sharded strategy or common strategy must implement ShardedObjects handling."
- f" Both {sharded_strategy} and {common_strategy} specify can_handle_sharded_objects=False"
- )
diff --git a/megatron/core/distributed/distributed_data_parallel.py b/megatron/core/distributed/distributed_data_parallel.py
index 55179ff3024..6855c32c15a 100644
--- a/megatron/core/distributed/distributed_data_parallel.py
+++ b/megatron/core/distributed/distributed_data_parallel.py
@@ -120,9 +120,9 @@ def __init__(
param_to_name[param] = name
if getattr(param, 'allreduce', True):
- dense_params.append(param)
+ dense_params.append((param, name))
else:
- expert_parallel_params.append(param)
+ expert_parallel_params.append((param, name))
def _allocate_buffers_for_parameters(
input_params, data_parallel_group, gradient_scaling_factor
@@ -132,7 +132,7 @@ def _allocate_buffers_for_parameters(
param_and_grad_dtype_to_indices = {}
# Group parameters by their gradient type.
- for param in input_params:
+ for param, param_name in input_params:
assert param.requires_grad
param_dtype = param.dtype
@@ -147,7 +147,7 @@ def _allocate_buffers_for_parameters(
grad_dtype = torch.float if self.ddp_config.grad_reduce_in_fp32 else param.dtype
params = param_and_grad_dtype_to_params.get((param_dtype, grad_dtype), [])
- params.append(param)
+ params.append((param, param_name))
param_and_grad_dtype_to_params[(param_dtype, grad_dtype)] = params
# Get the index of each param among the params with same dtype, if a param is fp8,
@@ -236,7 +236,10 @@ def _allocate_buffers_for_parameters(
# Set `next_param_gather_bucket_group` for different bucket groups by iterating through
# buckets in reverse order (since all-gathers happen in reverse order of buckets).
- if self.ddp_config.use_distributed_optimizer and self.ddp_config.overlap_param_gather:
+ # Note: overlap_param_gather covers both the distributed optimizer and the
+ # layer-wise optimizer cases; the latter sets overlap_param_gather=True
+ # without use_distributed_optimizer.
+ if self.ddp_config.overlap_param_gather:
num_bucket_groups = len(bucket_groups)
for i in range(1, num_bucket_groups):
bucket_groups[num_bucket_groups - i].next_param_gather_bucket_group = (
@@ -344,9 +347,10 @@ def unmap_weight_tensor(m):
grad_acc.register_hook(self._make_backward_post_hook(param))
self.grad_accs.append(grad_acc)
- self.use_forward_hook = (
- self.ddp_config.use_distributed_optimizer and self.ddp_config.overlap_param_gather
- )
+ # Note: overlap_param_gather covers both the distributed optimizer and the
+ # layer-wise optimizer cases; the latter sets overlap_param_gather=True
+ # without use_distributed_optimizer.
+ self.use_forward_hook = self.ddp_config.overlap_param_gather
self.remove_forward_pre_hook_handles = {}
if self.use_forward_hook:
self.enable_forward_pre_hook()
@@ -490,10 +494,18 @@ def start_param_sync(self, *unused, force_sync: bool = False, force_dispatch: bo
# in "finish_param_sync" stage after zeroing the shared gardient buffers.
if self.ddp_config.reuse_grad_buf_for_mxfp8_param_ag:
for bucket in bucket_group.buckets:
+ is_bf16_weight_bucket = False
for param in bucket.params:
+ # Skip copying since bf16 weights in the mxfp8 model
+ # are already mapped to param.data.
+ if not is_float8tensor(param):
+ is_bf16_weight_bucket = True
+ break
param_start, param_end = bucket.param_to_index[param]
param_slice = bucket.param_data.view(-1)[param_start:param_end]
param.data.copy_(param_slice.view(param.data.shape))
+ if is_bf16_weight_bucket:
+ continue
# All-gathered params are not needed after being copied to param.data.
# Zero out the param buffer (shared with grad buffer) for gradient
# accumulation. We cannot zero out the entire grad buffer because one grad
@@ -534,6 +546,11 @@ def finish_grad_sync(self, force_all_reduce: Optional[bool] = False):
for bucket_group in self.bucket_groups + self.expert_parallel_bucket_groups:
bucket_group.finish_grad_sync(force_all_reduce=force_all_reduce)
+ def free_overlap_buffers(self):
+ """Free overlap param-gather GPU buffers across all bucket groups."""
+ for bucket_group in self.bucket_groups + self.expert_parallel_bucket_groups:
+ bucket_group.free_overlap_buffers()
+
def scale_gradients(self, scaling_factor: float):
"""Scale all gradients inside the buffers by `scaling_factor`."""
for buffer in self.buffers + self.expert_parallel_buffers:
diff --git a/megatron/core/distributed/distributed_data_parallel_config.py b/megatron/core/distributed/distributed_data_parallel_config.py
index eaec971c79c..ee592368b0c 100644
--- a/megatron/core/distributed/distributed_data_parallel_config.py
+++ b/megatron/core/distributed/distributed_data_parallel_config.py
@@ -1,7 +1,9 @@
# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
from dataclasses import dataclass
-from typing import Optional
+from typing import Optional, Tuple
+
+import torch
@dataclass
@@ -33,7 +35,10 @@ class DistributedDataParallelConfig:
"""
check_for_nan_in_grad: bool = False
- """If true, check for NaNs and Infs in gradients _before_ communication collective."""
+ """
+ If true, check for NaNs and Infs in gradients _before_ communication collective.
+ Invoked by `start_grad_sync` such as in the Megatron-LM DDP training API.
+ """
check_for_large_grads: bool = False
"""If true, check for unexpectedly large gradients _before_ communication collective."""
@@ -54,6 +59,11 @@ class DistributedDataParallelConfig:
over the wire (using an all-to-all to keep total communication overhead in line
with the standard ring implementation) but performs accumulation locally in FP32."""
+ param_name_patterns_for_fp32_local_accumulation: Tuple[str, ...] = ()
+ """List of param_name patterns (in Python's fnmatch format) to match against to do
+ local gradient accumulation in FP32. The special pattern 'all' matches every
+ parameter. Do not specify when grad_reduce_in_fp32 is already True."""
+
average_in_collective: bool = False
"""If true, compute average in collective directly, as opposed to dividing by the
dp_size first and then computing sum in the collective."""
@@ -78,7 +88,7 @@ class DistributedDataParallelConfig:
data_parallel_sharding_strategy: str = 'no_shard'
"""Sharding strategy for FSDP. Valid values are 'no_shard', 'optim',
- 'optim_grads', 'optim_grads_params'."""
+ 'optim_grads', 'optim_grads_params'."""
gradient_reduce_div_fusion: bool = True
"""If true, perform gradient reduce and division fusion."""
@@ -91,9 +101,6 @@ class DistributedDataParallelConfig:
disables prefetching and may degrade performance. Adjust this value
based on your system's memory and performance requirements."""
- preserve_fp32_weights: bool = True
- """If true, preserve fp32 weights in the Megatron FSDP ParamAndGradBuffer."""
-
keep_fp8_transpose_cache: bool = False
"""If true, keep the fp8 transpose cache when using Megatron FSDP."""
@@ -105,7 +112,6 @@ class DistributedDataParallelConfig:
The follwoing will be the expected number of SM usage for various cases.
(Note that this is just a reference number and the number of SM usage could vary
on message size, communication domain size and nccl version.)
- ----------------------------------------------------------
| Communication domain | use_sharp | SM usage of "AG/RS" |
|----------------------|-----------|---------------------|
| NVL | N/A | 4 / 5 |
@@ -113,7 +119,6 @@ class DistributedDataParallelConfig:
| NVL+IB | True | 6 / 6 |
| IB | False | 1 / 4 |
| IB | True | 1 / 1 |
- ----------------------------------------------------------
"""
fsdp_double_buffer: bool = False
@@ -122,13 +127,29 @@ class DistributedDataParallelConfig:
This option will cause additional memory overhead, however, it is necessary for
to register user buffer (nccl_ub=True) for the Megatron FSDP.
This option will be automatically set to True when nccl_ub=True.
- """
+ """
+
+ fsdp_db_use_persist_buf_on_alloc_fail: bool = False
+ """Whether to fall back to persistent buffer when a bucket does not
+ fit FSDP double buffer size. If true, FSDP will use the persistently
+ allocated buffer for the bucket that does not fit, it will enable NCCL
+ user buffer with the cost of more memory usage. If false, FSDP will use
+ Dynamic memory allocator, NCCL user buffer won't not enabled, which
+ usually leads to low performance.
+ """
+
+ fsdp_all_gather_in_start_param_sync: bool = True
+ """
+ If True, use all-gather during the initial Megatron-FSDP parameter
+ synchronization step. This can increase overlap between the first
+ parameter all-gather and computation, helping to better hide the
+ initial communication cost.
+ """
outer_dp_sharding_strategy: str = 'no_shard'
"""
Sharding strategy for outer data parallel group in Hybrid Sharded Data Parallel (HSDP) mode.
- Valid values are 'no_shard', 'optim', 'optim_grads', 'optim_grads_params'.
- This option is only effective when Hybrid FSDP is enabled.
+ Valid values are 'no_shard', 'optim'. This option is only effective when Hybrid FSDP is enabled.
"""
disable_symmetric_registration: bool = False
@@ -148,6 +169,33 @@ class DistributedDataParallelConfig:
delay_wgrad_compute: bool = False
"""Delay the weight gradient computation to improve batch-level communication overlapping"""
+ megatron_fsdp_main_params_dtype: Optional[torch.dtype] = torch.float32
+ """Data type for the main weight buffer utilized for distributed optimization
+ and quantization with Megatron-FSDP. If set to None, the model compute weight
+ buffer will take the role of the main weights, or when no sharding is applied,
+ the native model weights become the main weights. Defaults to torch.float32.
+ """
+
+ megatron_fsdp_main_grads_dtype: Optional[torch.dtype] = None
+ """Data type for the main gradient buffer utilized for distributed optimization with
+ Megatron-FSDP. If set to None, main gradients will match the dtype of the model
+ compute parameters specified by the user model. Defaults to None.
+ """
+
+ megatron_fsdp_grad_comm_dtype: Optional[torch.dtype] = None
+ """Data type for gradient gather / scatter communications. Can be utilized to reduce
+ communication latency, but adds overhead for type-casting and copy operations.
+ If using NCCL UBR v2.27+, gradient reduction may be performed in high-precision
+ depending on the network domain (NVLink or IB), and can enable mixed-precision
+ communication and accumulation, e.g. setting grad_comm_dtype to `BF16` can support
+ `FP32` reduction even though we have `BF16` input and output communication buffers.
+ If set to None, the `main_grads_dtype` is used. If using HSDP (either DP-Replicate
+ or DP-Outer in `outer_dp_sharding_strategy`), `no_shard`, `optim`, or a
+ `FixedPoolAllocator` (`fsdp_double_buffer`), allocating `dtype`-custom gradient
+ communication buffers (per FSDP group) adds memory overhead. Defaults to None.
+ No additional memory is allocated when `grad_comm_dtype == main_grads_dtype`.
+ """
+
def __post_init__(self):
import os
@@ -161,3 +209,9 @@ def __post_init__(self):
"PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is currently not supported "
"with nccl_ub due to compatibility issue with torch.cuda.MemPool API."
)
+
+ if len(self.param_name_patterns_for_fp32_local_accumulation) > 0:
+ assert not self.grad_reduce_in_fp32, (
+ "Only need to explicitly specify param_name patterns for FP32 local accumulation "
+ "if .main_grads aren't already in FP32"
+ )
diff --git a/megatron/core/distributed/finalize_model_grads.py b/megatron/core/distributed/finalize_model_grads.py
index a52592bb269..ca6bdd354ce 100644
--- a/megatron/core/distributed/finalize_model_grads.py
+++ b/megatron/core/distributed/finalize_model_grads.py
@@ -299,7 +299,11 @@ def _update_router_expert_bias(model: List[torch.nn.Module], config: Transformer
expert_bias_list = []
for model_chunk in model:
for module in get_attr_wrapped_model(model_chunk, 'modules')():
- if hasattr(module, 'expert_bias'):
+ # Only update expert_bias if this module is in the training mode. There are special
+ # cases where only the student is in training mode but the teacher is in eval mode
+ # when using online knoweldge-distillation with Model-Optimizer. In this case, we want
+ # to avoid updating teacher's expert_bias.
+ if hasattr(module, 'expert_bias') and module.training:
tokens_per_expert_list.append(module.local_tokens_per_expert)
expert_bias_list.append(module.expert_bias)
# For hybrid models with both MoE and Dense layers, this list can be empty.
diff --git a/megatron/core/distributed/fsdp/mcore_fsdp_adapter.py b/megatron/core/distributed/fsdp/mcore_fsdp_adapter.py
index d6384e70488..8993620c779 100644
--- a/megatron/core/distributed/fsdp/mcore_fsdp_adapter.py
+++ b/megatron/core/distributed/fsdp/mcore_fsdp_adapter.py
@@ -42,10 +42,14 @@
from megatron.core.process_groups_config import ProcessGroupCollection
from megatron.core.transformer.transformer_config import TransformerConfig
from megatron.core.transformer.transformer_layer import TransformerLayer
-from megatron.core.utils import log_single_rank
+from megatron.core.utils import is_te_min_version, log_single_rank
try:
- from megatron.core.distributed.fsdp.src.megatron_fsdp import FSDPDistributedIndex, MegatronFSDP
+ from megatron.core.distributed.fsdp.src.megatron_fsdp import (
+ FSDPDistributedIndex,
+ MegatronFSDP,
+ MixedPrecisionPolicy,
+ )
HAVE_MEGATRON_FSDP = True
except ImportError as import_megatron_fsdp_error:
@@ -82,9 +86,34 @@ def __init__(
logging.INFO,
f'Setting up DistributedDataParallel with config {self.ddp_config}',
)
+ self.mp_policy = MixedPrecisionPolicy(
+ main_params_dtype=ddp_config.megatron_fsdp_main_params_dtype,
+ # Grandfathered Argument: grad_reduce_in_fp32
+ main_grads_dtype=(
+ torch.float32
+ if ddp_config.grad_reduce_in_fp32
+ else ddp_config.megatron_fsdp_main_grads_dtype
+ ),
+ grad_comm_dtype=(
+ torch.float32
+ if ddp_config.grad_reduce_in_fp32
+ else ddp_config.megatron_fsdp_grad_comm_dtype
+ ),
+ )
+ log_single_rank(
+ logger,
+ logging.INFO,
+ f'Setting up Megatron-FSDP MixedPrecisionPolicy with config {self.mp_policy}',
+ )
self.megatron_fsdp_dist_index = self._init_dist_index(pg_collection)
+ if config.gradient_accumulation_fusion:
+ assert is_te_min_version("2.10"), (
+ "Megatron-FSDP with gradient_accumulation_fusion requires "
+ "Transformer Engine version 2.10 or higher."
+ )
+
self.bucket_size = self.ddp_config.bucket_size
if disable_bucketing:
self.bucket_size = None
@@ -104,6 +133,7 @@ def __init__(
config=config,
module=MegatronFSDP(
ddp_config=ddp_config,
+ mixed_precision_policy=self.mp_policy,
module=module,
fsdp_unit_modules=self.fsdp_unit_modules,
disable_bucketing=disable_bucketing,
@@ -212,6 +242,13 @@ def _init_dist_index(self, pg_collection):
hybrid_fsdp_group = parallel_state.get_data_parallel_group(
with_context_parallel=True, partial_data_parallel=False
)
+ expt_dp_group = parallel_state.get_expert_data_parallel_group(
+ partial_expert_data_parallel=True
+ )
+ hybrid_fsdp_expt_group = parallel_state.get_expert_data_parallel_group(
+ partial_expert_data_parallel=False
+ )
+ ep_group = parallel_state.get_expert_model_parallel_group()
else:
dp_cp_group = parallel_state.get_data_parallel_group(
with_context_parallel=True, partial_data_parallel=False
@@ -227,6 +264,10 @@ def _init_dist_index(self, pg_collection):
dp_cp_group = pg_collection.intra_dp_cp
outer_fsdp_group = pg_collection.inter_dist_opt
hybrid_fsdp_group = pg_collection.dp_cp
+ # This has not been tested yet.
+ expt_dp_group = getattr(pg_collection, 'intra_expt_dp', None)
+ hybrid_fsdp_expt_group = getattr(pg_collection, 'expt_dp', None)
+ ep_group = getattr(pg_collection, 'ep', None)
else:
dp_cp_group = pg_collection.dp_cp
outer_fsdp_group = None
@@ -243,6 +284,18 @@ def _init_dist_index(self, pg_collection):
expt_tp_group = single_rank_group
if enable_hsdp:
+ if expt_dp_group is not None:
+ expt_mesh = _get_hsdp_tp_mesh(
+ outer_fsdp_group, expt_dp_group, expt_tp_group, ep_size=ep_group.size()
+ )
+ expt_device_mesh = DeviceMesh.from_group(
+ [outer_fsdp_group, expt_dp_group, expt_tp_group],
+ device_type="cuda",
+ mesh=expt_mesh.tolist(),
+ mesh_dim_names=["outer_fsdp_dp", "dp_cp", "tp"],
+ )
+ else:
+ expt_device_mesh = None
mesh = _get_hsdp_tp_mesh(outer_fsdp_group, dp_cp_group, tp_group)
dist_index = FSDPDistributedIndex(
hsdp_outer_dp_shard=self.ddp_config.outer_dp_sharding_strategy != "no_shard",
@@ -256,6 +309,8 @@ def _init_dist_index(self, pg_collection):
dp_shard_dim="dp_cp",
tp_dim="tp",
hybrid_fsdp_group=hybrid_fsdp_group,
+ hybrid_fsdp_expt_group=hybrid_fsdp_expt_group,
+ expt_device_mesh=expt_device_mesh,
)
else:
if ep_group is not None:
@@ -308,22 +363,24 @@ def sync_rng_states_across_tp_group(self):
_load_rng_state_dict(broadcast_list[0])
-def _get_hsdp_tp_mesh(outer_fsdp_dp_group, dp_cp_group, tp_group):
+def _get_hsdp_tp_mesh(outer_fsdp_dp_group, dp_cp_group, tp_group, ep_size=1):
assert HAVE_EINOPS, "einops is not installed. Please install it with `pip install einops`."
world_size = dist.get_world_size()
mesh = einops.rearrange(
torch.arange(world_size),
- "(outer_fsdp_dp fsdp tp) -> outer_fsdp_dp fsdp tp",
+ "(outer_fsdp_dp fsdp ep tp) -> ep outer_fsdp_dp fsdp tp",
outer_fsdp_dp=outer_fsdp_dp_group.size(),
tp=tp_group.size(),
+ ep=ep_size,
)
mesh_fsdp_ranks = einops.rearrange(
mesh,
- 'outer_fsdp_dp fsdp tp -> (outer_fsdp_dp tp) fsdp',
+ 'ep outer_fsdp_dp fsdp tp -> (outer_fsdp_dp ep tp) fsdp',
tp=tp_group.size(),
fsdp=dp_cp_group.size(),
+ ep=ep_size,
)
fsdp_group_ranks = dist.get_process_group_ranks(dp_cp_group)
assert _check_mesh_ranks_and_group_ranks_are_consistent(mesh_fsdp_ranks, fsdp_group_ranks), (
@@ -333,7 +390,7 @@ def _get_hsdp_tp_mesh(outer_fsdp_dp_group, dp_cp_group, tp_group):
mesh_tp_ranks = einops.rearrange(
mesh,
- 'outer_fsdp_dp fsdp tp -> (outer_fsdp_dp fsdp) tp',
+ 'ep outer_fsdp_dp fsdp tp -> (outer_fsdp_dp fsdp ep) tp',
tp=tp_group.size(),
fsdp=dp_cp_group.size(),
)
@@ -345,9 +402,10 @@ def _get_hsdp_tp_mesh(outer_fsdp_dp_group, dp_cp_group, tp_group):
mesh_outer_fsdp_dp_ranks = einops.rearrange(
mesh,
- 'outer_fsdp_dp fsdp tp -> (fsdp tp) outer_fsdp_dp',
+ 'ep outer_fsdp_dp fsdp tp -> (fsdp ep tp) outer_fsdp_dp',
tp=tp_group.size(),
fsdp=dp_cp_group.size(),
+ ep=ep_size,
)
outer_fsdp_dp_group_ranks = dist.get_process_group_ranks(outer_fsdp_dp_group)
assert _check_mesh_ranks_and_group_ranks_are_consistent(
@@ -357,7 +415,21 @@ def _get_hsdp_tp_mesh(outer_fsdp_dp_group, dp_cp_group, tp_group):
f"do not match the ranks in the Outer FSDP DP group {outer_fsdp_dp_group_ranks}."
)
- return mesh
+ # Exclude the expert parallel dimension
+ rank = dist.get_rank()
+ dp_tp_meshes = [per_ep_mesh for per_ep_mesh in mesh if rank in per_ep_mesh.reshape(-1).tolist()]
+ assert (
+ len(dp_tp_meshes) == 1
+ ), f"[Megatron-FSDP] Current rank {rank} is not unique in the mesh ranks {mesh.tolist()}."
+ assert (
+ len(dp_tp_meshes[0].reshape(-1).tolist())
+ == outer_fsdp_dp_group.size() * dp_cp_group.size() * tp_group.size()
+ ), (
+ f"[Megatron-FSDP] DP-TP mesh size {len(dp_tp_meshes[0].reshape(-1).tolist())} "
+ f"does not match the expected size"
+ f"{outer_fsdp_dp_group.size() * dp_cp_group.size() * tp_group.size()}."
+ )
+ return dp_tp_meshes[0]
def _get_dp_tp_mesh(dp_cp_group, tp_group, ep_size=1):
diff --git a/megatron/core/distributed/fsdp/src/README.md b/megatron/core/distributed/fsdp/src/README.md
index bc4cdaa078e..dc984967e88 100644
--- a/megatron/core/distributed/fsdp/src/README.md
+++ b/megatron/core/distributed/fsdp/src/README.md
@@ -129,6 +129,7 @@ import torch
from megatron_fsdp import (
fully_shard_model,
fully_shard_optimizer,
+ MixedPrecisionPolicy,
)
```
@@ -156,12 +157,13 @@ device_mesh[("dp_outer", "dp_shard", "cp")]._flatten("hsdp")
hsdp_group = device_mesh["hsdp"].get_group()
# Initialize DeviceMesh for expert parallel (EP) modules when using FSDP + EP.
-expt_device_mesh = DeviceMesh.from_group(
- [expt_dp_group, expt_tp_group],
- device_type="cuda",
- mesh=expt_mesh.tolist(),
- mesh_dim_names=["dp_shard_cp", "tp"],
+expert_device_mesh = torch.distributed.device_mesh.init_device_mesh(
+ "cuda",
+ mesh_shape=(dp_outer_size, expt_dp_shard_size, expt_tp_size),
+ mesh_dim_names=("dp_outer", "dp_shard_cp", "tp"),
)
+expert_device_mesh[("dp_outer", "dp_shard_cp")].flatten("hsdp")
+hsdp_expt_group = expert_device_mesh["hsdp"].get_group()
```
### Convert models into fully-sharded `MegatronFSDP` models with `fully_shard_model`.
@@ -186,6 +188,8 @@ model = fully_shard_model(
tp_dim="tp",
# Only required when using HSDP. Otherwise, set this to None.
hybrid_fsdp_group=hsdp_group,
+ # Only required when using HSDP + EP. Otherwise, set this to None.
+ hybrid_fsdp_expt_group=hsdp_expt_group,
# Only required for FSDP + EP. Otherwise, set this to None.
expt_device_mesh=expt_device_mesh,
# FSDP Sharding Strategy: no_shard (0) / optim (1) / optim_grads (2) / optim_grads_params (3)
@@ -193,10 +197,8 @@ model = fully_shard_model(
outer_dp_sharding_strategy=1,
# Initialize the model on devices in shards to avoid OOM. Requires device("meta")-init for model.
init_model_with_meta_device=True,
- # Reduce gradients in FP32.
- grad_reduce_in_fp32=False,
- # Store distributed optimization state in FP32.
- preserve_fp32_weights=True,
+ # Mixed-Precision Policy for controlling compute and communication precision in Megatron-FSDP.
+ mixed_precision_policy=MixedPrecisionPolicy(),
# Sync parameters and gradients each step. Allows for gradient transformations after backward pass,
# and synchronizes parameters and gradients across HSDP groups, but deactivates compute-communication
# overlap going into the subsequent training step.
@@ -282,7 +284,7 @@ Megatron-FSDP's `fully_shard_*` API has a comprehensive set of arguments for fin
- `fsdp_unit_modules` is a list of sub-module classes or `str` import-paths associated with modules that you want `MegatronFSDP` to fully-shard.
- Required if `1`, `2`, or `3` are specified as the sharding strategy. Defaults to `None`, in which case Megatron-FSDP will replicate the parameters similar to DDP.
-- `zero_dp_strategy` (and `outer_dp_sharding_strategy`) configure different degrees of zero-redundancy data parallelism as described in [ZeRO (Zero Redundancy Optimizer)](https://arxiv.org/abs/1910.02054). It reduces CUDA memory utilization during model training by distributing model parameters, gradients, and optimizer states across multiple devices in the DP `ProcessGroup`, and collectively communicating subsets of parameters and gradients to specific devices when needed for computation or differentiation. More aggressive sharding strategies will entail more communication overhead, with `no_shard` being the least memory efficient but most communication efficient, and `optim_grads_params` being the most memory efficient but least communication efficient. `outer_dp_sharding_strategy` has the same options, except for the (required) "outer" DP group (`dp_outer_dim` / `hybrid_fsdp_group`) when using [Hybrid-Sharded Data Parallelism (HSDP)](https://arxiv.org/pdf/2304.11277), and only `no_shard` (DP Replication) and `optim` (Optimizer State Hybrid Sharding, requires `zero_dp_strategy='optim_grads_params`) are supported.
+- `zero_dp_strategy` (and `outer_dp_sharding_strategy`) configure different degrees of zero-redundancy data parallelism as described in [ZeRO (Zero Redundancy Optimizer)](https://arxiv.org/abs/1910.02054). It reduces CUDA memory utilization during model training by distributing model parameters, gradients, and optimizer states across multiple devices in the DP `ProcessGroup`, and collectively communicating subsets of parameters and gradients to specific devices when needed for computation or differentiation. More aggressive sharding strategies will entail more communication overhead, with `no_shard` being the least memory efficient but most communication efficient, and `optim_grads_params` being the most memory efficient but least communication efficient. Additionally, `outer_dp_sharding_strategy` supports `no_shard` ([Hybrid-Sharded Data Parallelism (HSDP)](https://arxiv.org/pdf/2304.11277)) and `optim` (`HFSDP` = Fully-Sharded Optimizer State + _HSDP_, requires `zero_dp_strategy='optim_grads_params'`), after specifying the "outer" DP group (`dp_outer_dim` / `hybrid_fsdp_group`).
- Default: `optim_grads_params` or `3` for `zero_dp_strategy` and `no_shard` or `0` for `outer_dp_sharding_strategy`
- `0` or `no_shard` implies that your model is not sharded. Similar memory usage to `DDP`.
- `1` or `optim` implies that your optimizer state is sharded for distributed optimization. Similar to optimizer state sharding in `ZeRO-DP`.
@@ -295,22 +297,31 @@ Megatron-FSDP's `fully_shard_*` API has a comprehensive set of arguments for fin
- `tp_dim` is the name of the sub-mesh used for tensor parallelism (TP), which is required for `(FSDP, TP)`-strided sharding when using Megatron-LM or Torch-native `DTensor` TP.
- For more information about tensor parallelism, refer to: [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053).
- `hybrid_fsdp_group` is the `ProcessGroup` which contains all ranks in the flattened `dp_shard_dim` and `dp_outer_dim` sub-meshes utilized to specify the `(DP-Outer, DP-Shard)` sharded mesh coordinates for the weight and gradient buffers. Required for HSDP.
+ - `hybrid_fsdp_expt_group` defines the data-parallel communication group for expert parameters. It is required for HSDP.
- `expt_device_mesh` is another [`torch.distributed.DeviceMesh`](https://docs.pytorch.org/docs/stable/distributed.html#devicemesh) tailored for the expert parallel (EP) modules in `MegatronFSDP`.
- `dp_shard_dim` is the name of the sub-mesh required for FSDP sharding of the EP modules, enabling expert data parallelism (EDP).
- `tp_dim` is the name of the sub-mesh used for expert tensor parallelism (ETP), which is required for `(FSDP, ETP)`-strided sharding when using Megatron-LM or Torch-native `DTensor` ETP.
- `init_model_with_meta_device` has `MegatronFSDP` initialize your `meta`-device model in shards on every CUDA device to avoid OOM when initializing extremely large models that cannot fit on a single device. Users can initialize their model on a [`meta`-device](https://docs.pytorch.org/docs/stable/meta.html) (`with torch.device('meta'): ...`), and ``MegatronFSDP`` will further shard and initialize the model parameters layer-by-layer adhering to the customizable `module.reset_parameters` method, which prevents the entire model from being allocated in memory at any point during runtime.
- Defaults to `False`.
- Note that the `device` argument which installs your model on a specific device or rank will be deactivated when `init_model_with_meta_device=True`.
-- `grad_reduce_in_fp32` will reduce gradients in `FP32` precision (in contrast to the lower `BF16` or `FP8` model training precision).
- - Defaults to `False`.
- - `torch.distributed.fsdp.MixedPrecisionPolicy` will be supported in the near future.
-- `preserve_fp32_weights` will preserve a `FP32` precision version of model parameters utilized for optimization.
- - Defaults to `True`.
- - `torch.distributed.fsdp.MixedPrecisionPolicy` will be supported in the near future.
+- `mixed_precision_policy` takes a `megatron_fsdp.MixedPrecisionPolicy` that configures mixed-precision compute and communication for Megatron-FSDP. Configuration options include:
+ - `main_params_dtype` controls the data-type for parameters used in distributed optimization or quantization.
+ - Defaults to `torch.float32`.
+ - If set to `None`, the native model compute parameter data-type will be utilized.
+ - Requires specification (cannot be `None`) when using `FP8` parameters with Megatron-FSDP.
+ - `main_grads_dtype` controls the data-type for gradients used in distributed optimization.
+ - Defaults to `None`, the model native gradient data-type will be utilized.
+ - While `torch.float32` (or higher) is recommended for accuracy at scale, as `main_grads_dtype` controls the data-type for gradient accumulation, `None` is more flexible and uses pre-determined parameter gradient logic in mixed-precision scenarios, such as `BF16` for `FP8`/`FP4` parameters quantized via TransformerEngine.
+ - `grad_comm_dtype` controls the data-type for gradient communications (RS / AR) when reducing gradients. Lower precision `grad_comm_dtype` improves (communication) performance, but may increase memory utilization or sacrifice gradient precision in certain cases.
+ - Defaults to `None`, the `main_grads_dtype` data-type will be utilized, and no additional memory is allocated when `grad_comm_dtype == main_grads_dtype`.
+ - If using HSDP (either DP-Replicate or DP-Outer in `outer_dp_sharding_strategy`), `no_shard`, `optim`, or a `FixedPoolAllocator` (`fsdp_double_buffer`), allocating `dtype`-custom gradient communication buffers (per FSDP group) adds memory overhead of up to 10% or more, and users should consider the performance-memory trade-off when using this feature.
+ - If using NCCL UBR v2.27+ (`nccl_ub=True`), gradient reduction may be performed in high-precision depending on the network domain (NVLink or IB), and can enable mixed-precision communication and accumulation, e.g. setting grad_comm_dtype to `BF16` can support `FP32` reduction even though we have `BF16` input and output communication buffers. Otherwise, gradients will be reduced in `grad_comm_dtype` (and accumulated in `main_grads_dtype`) as usual.
- `overlap_grad_reduce` and `overlap_param_gather` will overlap gradient [`reduce-scatter`](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html#reducescatter) and parameter [`all-gather`](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html#allgather) group communications with backward and forward compute with asynchronous calls and pre-fetching. (In the case of `no_shard`, parameters are not gathered but gradient [`all-reduce`](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html#allreduce) is overlapped.)
- Both default to `True`.
-- `sync_model_each_microbatch` will trigger a `wait` (`MegatronFSDP.finish_grad_sync()`) on gradient reduction, parameter de-allocation, and optimizer parameter / gradient installation (in preparation for `optimizer.step()`) after every forward-backward pass. When using HSDP, parameters and gradients will be all-gathered and reduced respectively on the "outer" DP group each training step instead of each optimization cycle. This behavior is desirable for a transparent and user-friendly sharded training loop where post-backward transformations on the gradient and a clean compute / memory state are necessary between training iterations, but damages performance in situations where optimization is delayed (e.g. gradient accumulation) where the communications of the previous training iteration can be overlapped with the compute of the next training iteration. Will also override `is_last_microbatch` / `microbatch_count` logic in `MegatronFSDP`.
+- `sync_model_each_microbatch` will trigger a `wait` (`MegatronFSDP.finish_grad_sync()`) on gradient reduction, parameter de-allocation, and optimizer parameter / gradient installation (in preparation for `optimizer.step()`) after every forward-backward pass. When using HSDP, parameters and gradients will be all-gathered and reduced respectively on the "outer" DP group each training step instead of each optimization cycle. This behavior is desirable for a transparent and user-friendly sharded training loop where post-backward transformations on the gradient and a clean compute / memory state are necessary within and between training iterations, but damages performance in situations where optimization is delayed (e.g. gradient accumulation) when the communications of the previous training iteration can be overlapped with the compute of the next training iteration. Will also override `is_last_microbatch` / `microbatch_count` logic in `MegatronFSDP`.
- Defaults to `True` for `fully_shard`, but defaults to `False` when using the `MegatronFSDP` class directly.
+ - Can also be controlled with the `MegatronFSDP.sync()` context manager, or through invoking `MegatronFSDP.set_model_auto_sync(bool)`.
+ - WARNING: When this synchronization feature is activated in conjunction with `no_shard` / `0` or `optim` / `1` sharding strategies, the user is responsible for calling `MegatronFSDP.zero_grad_buffer()` or `optimizer.zero_grad()` after the subsequent forward-backward pass. This is because un-sharded gradients are all-reduced directly into the gradient accumulation buffer, and this buffer should not be all-reduced more than once per optimization cycle! Analogous to the justification for the [`no_sync()` API for PyTorch DistributedDataParallel](https://docs.pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.no_sync).
- `enable_fine_grained_param_gather` modifies FSDP to all-gather parameters with per-Module granularity instead of collectively unsharding all sub-modules of a unit module in Megatron-FSDP.
- Defaults to `False`.
- `keep_fp8_transpose_cache` will keep the fp8 transpose cache when using `MegatronFSDP`. This option will cause (number of parameter $\times$ 1 Byte) of memory overhead, but can skip the weight transpose operation in the backward propagation. This feature will not give any benefit from the Blackwell architecture.
@@ -358,8 +369,10 @@ mfsdp_model = fully_shard_model(
fsdp_unit_modules=[te.pytorch.TransformerLayer],
# Only FSDP / ZeRO-3 supports FP8 parameters.
zero_dp_strategy=3,
- # Needed for FP8 parameters. (Default is already True.)
- preserve_fp32_weights=True,
+ # FP32 main weights needed for FP8 parameters.
+ mixed_precision_policy=MixedPrecisionPolicy(
+ main_params_dtype=torch.float32
+ ),
# Needed for select FP8 recipes.
keep_fp8_transpose_cache=True,
)
diff --git a/megatron/core/distributed/fsdp/src/megatron_fsdp/__init__.py b/megatron/core/distributed/fsdp/src/megatron_fsdp/__init__.py
index 1dd7263c4e2..a1ae54e622d 100644
--- a/megatron/core/distributed/fsdp/src/megatron_fsdp/__init__.py
+++ b/megatron/core/distributed/fsdp/src/megatron_fsdp/__init__.py
@@ -15,6 +15,7 @@
from .distributed_data_parallel_config import DistributedDataParallelConfig
from .fully_shard import fully_shard, fully_shard_model, fully_shard_optimizer
from .megatron_fsdp import MegatronFSDP
+from .mixed_precision import MixedPrecisionPolicy
from .package_info import (
__contact_emails__,
__contact_names__,
@@ -34,6 +35,7 @@
"DistributedDataParallelConfig",
"MegatronFSDP",
"FSDPDistributedIndex",
+ "MixedPrecisionPolicy",
"fully_shard",
"fully_shard_model",
"fully_shard_optimizer",
diff --git a/megatron/core/distributed/fsdp/src/megatron_fsdp/distributed_data_parallel_config.py b/megatron/core/distributed/fsdp/src/megatron_fsdp/distributed_data_parallel_config.py
index f0c817e1f80..4ad5a8dddac 100644
--- a/megatron/core/distributed/fsdp/src/megatron_fsdp/distributed_data_parallel_config.py
+++ b/megatron/core/distributed/fsdp/src/megatron_fsdp/distributed_data_parallel_config.py
@@ -3,13 +3,15 @@
from dataclasses import dataclass
from typing import Optional
+import torch
+
@dataclass
class DistributedDataParallelConfig:
- """Configuration for DistributedDataParallel."""
-
- grad_reduce_in_fp32: bool = False
- """If true, reduce grads in fp32."""
+ """
+ Megatron-FSDP `fully_shard` API sub-configuration
+ derived from Megatron-Core DistributedDataParallel.
+ """
overlap_grad_reduce: bool = False
"""If true, overlap grad all-reduce / reduce-scatter with backward compute."""
@@ -17,38 +19,17 @@ class DistributedDataParallelConfig:
overlap_param_gather: bool = False
"""If true, overlap param all-gather with forward compute."""
- align_param_gather: bool = False
- """If true, all PP stages will launch param all-gathers simultaneously. Otherwise, each
- PP stage will independently launch as needed.
- """
-
- use_distributed_optimizer: bool = False
- """If true, issue reduce-scatter collectives to aggregate gradients and clean up
- originally allocated model parameters, otherwise issue all-reduce collectives.
+ check_for_nan_in_grad: bool = False
"""
-
- num_distributed_optimizer_instances: int = 1
- """Sets the factor by which the DP domain is sharded to have the partial DistOpt
- enabled. Defaults to 1, which means DistOpt is across entire DP domain.
+ If True, check for NaNs and Infs in gradients _before_ communication collective.
+ Invoked by `start_grad_sync` such as in the Megatron-LM DDP training API.
"""
- check_for_nan_in_grad: bool = False
- """If true, check for NaNs and Infs in gradients _before_ communication collective."""
-
- check_for_large_grads: bool = False
- """If true, check for unexpectedly large gradients _before_ communication collective."""
-
bucket_size: Optional[int] = None
"""Maximum number of parameters in each bucket. If unspecified, MCore uses a default
value of max(40000000, 1000000 * dp_size) parameters (larger DP sizes need larger
buckets to ensure collectives do not become latency-bound)."""
- pad_buckets_for_high_nccl_busbw: bool = False
- """If true, make sure the bucket size is divisible by a large power of 2 (2^16) to
- ensure NCCL collectives have high bus bandwidth at large DP counts, since NCCL
- message size (which for ring algorithms is bucket_size / dp_size) apparently needs
- to be divisible by a power of 2 for high busbw."""
-
average_in_collective: bool = False
"""If true, compute average in collective directly, as opposed to dividing by the
dp_size first and then computing sum in the collective."""
@@ -57,23 +38,9 @@ class DistributedDataParallelConfig:
"""If true, keep the compute param in fp8 (do not use any other intermediate dtype) and
perform the param all-gather in fp8."""
- reuse_grad_buf_for_mxfp8_param_ag: bool = False
- """If true, reuse the grad buffer for param AG when using mxfp8 recipe. Should be
- set to True only when fp8_recipe is mxfp8 and fp8_param_gather is True."""
-
- use_megatron_fsdp: bool = False
- """If true, use the FSDP code path for DDP."""
-
- use_custom_fsdp: bool = False
- """
- NOTE: The flag `use_custom_fsdp` is deprecated and will be removed in future versions.
- Please use `use_megatron_fsdp` instead, as all functionality will be migrated there.
- Future updates will drop support for `use_custom_fsdp` to avoid confusion.
- """
-
data_parallel_sharding_strategy: str = 'no_shard'
"""Sharding strategy for FSDP. Valid values are 'no_shard', 'optim',
- 'optim_grads', 'optim_grads_params'."""
+ 'optim_grads', 'optim_grads_params'."""
gradient_reduce_div_fusion: bool = True
"""If true, perform gradient reduce and division fusion."""
@@ -86,9 +53,6 @@ class DistributedDataParallelConfig:
disables prefetching and may degrade performance. Adjust this value
based on your system's memory and performance requirements."""
- preserve_fp32_weights: bool = True
- """If true, preserve fp32 weights in the Megatron FSDP ParamAndGradBuffer."""
-
keep_fp8_transpose_cache: bool = False
"""If true, keep the fp8 transpose cache when using Megatron FSDP."""
@@ -100,7 +64,6 @@ class DistributedDataParallelConfig:
The follwoing will be the expected number of SM usage for various cases.
(Note that this is just a reference number and the number of SM usage could vary
on message size, communication domain size and nccl version.)
- ----------------------------------------------------------
| Communication domain | use_sharp | SM usage of "AG/RS" |
|----------------------|-----------|---------------------|
| NVL | N/A | 4 / 5 |
@@ -108,7 +71,6 @@ class DistributedDataParallelConfig:
| NVL+IB | True | 6 / 6 |
| IB | False | 1 / 4 |
| IB | True | 1 / 1 |
- ----------------------------------------------------------
"""
fsdp_double_buffer: bool = False
@@ -119,6 +81,23 @@ class DistributedDataParallelConfig:
This option will be automatically set to True when nccl_ub=True.
"""
+ fsdp_all_gather_in_start_param_sync: bool = True
+ """
+ If True, use all-gather during the initial Megatron-FSDP parameter
+ synchronization step. This can increase overlap between the first
+ parameter all-gather and computation, helping to better hide the
+ initial communication cost.
+ """
+
+ fsdp_db_use_persist_buf_on_alloc_fail: bool = False
+ """Whether to fall back to persistent buffer when a bucket does not
+ fit FSDP double buffer size. If true, FSDP will use the persistently
+ allocated buffer for the bucket that does not fit, it will enable NCCL
+ user buffer with the cost of more memory usage. If false, FSDP will use
+ Dynamic memory allocator, NCCL user buffer won't not enabled, which
+ usually leads to low performance.
+ """
+
outer_dp_sharding_strategy: str = 'no_shard'
"""
Sharding strategy for outer data parallel group in Hybrid Sharded Data Parallel (HSDP) mode.
@@ -139,13 +118,37 @@ class DistributedDataParallelConfig:
to minimize the registration time.
"""
+ megatron_fsdp_main_params_dtype: Optional[torch.dtype] = torch.float32
+ """Data type for the main weight buffer utilized for distributed optimization
+ and quantization with Megatron-FSDP. If set to None, the model compute weight
+ buffer will take the role of the main weights, or when no sharding is applied,
+ the native model weights become the main weights. Defaults to torch.float32.
+ """
+
+ megatron_fsdp_main_grads_dtype: Optional[torch.dtype] = None
+ """Data type for the main gradient buffer utilized for distributed optimization with
+ Megatron-FSDP. If set to None, main gradients will match the dtype of the model
+ compute parameters specified by the user model. Defaults to None.
+ """
+
+ megatron_fsdp_grad_comm_dtype: Optional[torch.dtype] = None
+ """Data type for gradient gather / scatter communications. Can be utilized to reduce
+ communication latency, but adds overhead for type-casting and copy operations.
+ If using NCCL UBR v2.27+, gradient reduction may be performed in high-precision
+ depending on the network domain (NVLink or IB), and can enable mixed-precision
+ communication and accumulation, e.g. setting grad_comm_dtype to `BF16` can support
+ `FP32` reduction even though we have `BF16` input and output communication buffers.
+ If set to None, the `main_grads_dtype` is used. If using HSDP (either DP-Replicate
+ or DP-Outer in `outer_dp_sharding_strategy`), `no_shard`, `optim`, or a
+ `FixedPoolAllocator` (`fsdp_double_buffer`), allocating `dtype`-custom gradient
+ communication buffers (per FSDP group) adds memory overhead. Defaults to None.
+ No additional memory is allocated when `grad_comm_dtype == main_grads_dtype`.
+ """
+
def __post_init__(self):
import os
"""Check the validity of the config."""
- if self.reuse_grad_buf_for_mxfp8_param_ag:
- assert self.fp8_param_gather, "Reuse grad buffer only when keeping params in MXFP8."
-
if self.nccl_ub:
if 'expandable_segments:True' in os.getenv('PYTORCH_CUDA_ALLOC_CONF', '').split(','):
raise ValueError(
diff --git a/megatron/core/distributed/fsdp/src/megatron_fsdp/fully_shard.py b/megatron/core/distributed/fsdp/src/megatron_fsdp/fully_shard.py
index df210f15f05..ee6a6013b2d 100644
--- a/megatron/core/distributed/fsdp/src/megatron_fsdp/fully_shard.py
+++ b/megatron/core/distributed/fsdp/src/megatron_fsdp/fully_shard.py
@@ -20,8 +20,10 @@
import torch
from torch.distributed import DeviceMesh
from torch.distributed.device_mesh import init_device_mesh
+from torch.distributed.tensor import DTensor
from .megatron_fsdp import MegatronFSDP
+from .mixed_precision import MixedPrecisionPolicy
from .uneven_dtensor import preprocess_state_dict_for_uneven_dtensor
from .utils import FSDPDistributedIndex, create_updated_function_signature
@@ -77,25 +79,26 @@ def fully_shard_model(
dp_outer_dim: Optional[str] = None,
tp_dim: Optional[str] = None,
hybrid_fsdp_group: Optional[torch.distributed.ProcessGroup] = None,
+ hybrid_fsdp_expt_group: Optional[torch.distributed.ProcessGroup] = None,
expt_device_mesh: Optional[DeviceMesh] = None,
fsdp_unit_modules: Optional[Sequence[Type[torch.nn.Module]] | Sequence[str]] = None,
zero_dp_strategy: str | int = 3,
outer_dp_sharding_strategy: str | int = 0,
device: Optional[torch.device] = None,
init_model_with_meta_device: bool = False,
- grad_reduce_in_fp32: bool = False,
- preserve_fp32_weights: bool = True,
+ mixed_precision_policy: MixedPrecisionPolicy = MixedPrecisionPolicy(),
overlap_grad_reduce: bool = True,
overlap_param_gather: bool = True,
sync_model_each_microbatch: bool = True,
preproc_state_dict_for_dcp_ckpt: bool = True,
- check_for_nan_in_grad: bool = True,
+ report_nan_in_param_grad: bool = False,
average_in_collective: bool = False,
disable_bucketing: bool = False,
calculate_per_token_loss: bool = False,
keep_fp8_transpose_cache: bool = False,
nccl_ub: bool = False,
fsdp_double_buffer: bool = False,
+ fsdp_db_use_persist_buf_on_alloc_fail: bool = False,
disable_symmetric_registration: bool = False,
enable_fine_grained_param_gather: bool = False,
) -> torch.nn.Module:
@@ -179,11 +182,9 @@ class that schedules the sharding lifecycle of the model parameters and gradient
implementing a custom Module.reset_parameters() or Module._reset_parameters() method.
Defaults to False.
- grad_reduce_in_fp32 (bool):
- Whether to perform gradient reduction in FP32. Defaults to False.
-
- preserve_fp32_weights (bool):
- Whether to preserve FP32 optimization weights. Defaults to True.
+ mixed_precision_policy (megatron_fsdp.MixedPrecisionPolicy):
+ Megatron-FSDP mixed-precision config that controls compute and communication precision.
+ Default values are defined in `megatron_fsdp.MixedPrecisionPolicy`.
overlap_grad_reduce (bool):
Whether to overlap gradient reduce-scatter (or all-reduce) with backward compute.
@@ -206,8 +207,9 @@ class that schedules the sharding lifecycle of the model parameters and gradient
for both the model and the optimizer.
Defaults to True.
- check_for_nan_in_grad (bool):
- Whether to check for NaN values in gradients. Defaults to True.
+ report_nan_in_param_grad (bool):
+ Whether to precisely check for NaN values in gradients for every weight. Can
+ significantly degrade performance. Defaults to False.
average_in_collective (bool):
Whether to average gradients in collective communication. Defaults to False.
@@ -230,6 +232,10 @@ class that schedules the sharding lifecycle of the model parameters and gradient
fsdp_double_buffer (bool):
Whether to use double buffer for FSDP. Defaults to False.
+ fsdp_db_use_persist_buf_on_alloc_fail (bool):
+ Whether to fall back to persistent buffer allocator when a bucket does not
+ fit FSDP double buffer size.
+
disable_symmetric_registration (bool):
Whether to disable symmetric (window) registration for NCCL UB registration.
This option forces conventional (local) UB registration when nccl_ub is set.
@@ -327,16 +333,14 @@ class that schedules the sharding lifecycle of the model parameters and gradient
ddp_config = DistributedDataParallelConfig(
data_parallel_sharding_strategy=zero_dp_strategy,
outer_dp_sharding_strategy=outer_dp_sharding_strategy,
- grad_reduce_in_fp32=grad_reduce_in_fp32,
- preserve_fp32_weights=preserve_fp32_weights,
overlap_grad_reduce=overlap_grad_reduce,
overlap_param_gather=overlap_param_gather,
average_in_collective=average_in_collective,
keep_fp8_transpose_cache=keep_fp8_transpose_cache, # pylint: disable=C0301
nccl_ub=nccl_ub,
fsdp_double_buffer=fsdp_double_buffer or nccl_ub,
+ fsdp_db_use_persist_buf_on_alloc_fail=fsdp_db_use_persist_buf_on_alloc_fail,
disable_symmetric_registration=disable_symmetric_registration,
- check_for_nan_in_grad=check_for_nan_in_grad,
)
# Create FSDPDistributedIndex.
@@ -352,6 +356,8 @@ class that schedules the sharding lifecycle of the model parameters and gradient
tp_dim=tp_dim,
# Only required for HSDP.
hybrid_fsdp_group=hybrid_fsdp_group,
+ # Only required for HSDP + EP.
+ hybrid_fsdp_expt_group=hybrid_fsdp_expt_group,
# Access to flattened DP rank assignments for HSDP.
hsdp_outer_dp_shard=_outer_fsdp_sharding,
# Only required for Megatron-FSDP + EP.
@@ -363,6 +369,7 @@ class that schedules the sharding lifecycle of the model parameters and gradient
module=module,
dist_index=dist_index,
ddp_config=ddp_config,
+ mixed_precision_policy=mixed_precision_policy,
fsdp_unit_modules=fsdp_unit_modules,
disable_bucketing=disable_bucketing,
device=device,
@@ -370,6 +377,7 @@ class that schedules the sharding lifecycle of the model parameters and gradient
init_model_with_meta_device=init_model_with_meta_device,
sync_model_each_microbatch=sync_model_each_microbatch,
enable_fine_grained_param_gather_hook=enable_fine_grained_param_gather,
+ report_nan_in_param_grad=report_nan_in_param_grad,
)
# Register a state dict post-hook to add Torch DCP metadata for writing checkpoints.
@@ -447,6 +455,20 @@ def fully_shard_optimizer(
optimizer_step_base_func = type(optimizer).step
optimizer_zero_grad_base_func = type(optimizer).zero_grad
+ # Pre-initialize the optimizer state for checkpoint loading via DCP.
+ for group in optimizer.param_groups:
+ for param in group["params"]:
+ if param.numel() == 0 or (
+ hasattr(param, "_local_tensor") and param._local_tensor.numel() == 0
+ ):
+ # Avoid FusedAdam errors on empty tensor input.
+ continue
+ # Optimizer state is built from wgrad.
+ param.grad = torch.zeros_like(param)
+ # Non-lazy optimizer state initialization.
+ optimizer.step()
+ optimizer.zero_grad()
+
# Define a new optimizer.step() method that distributes optimizer state and gradients,
# waits for asynchronous gradient reduce-scatter work to be completed, and updates
# model weights. These options can be turned off via arguments in optimizer.step().
@@ -500,19 +522,95 @@ def megatron_fsdp_optimizer_zero_grad(optimizer, *args, **kwargs):
optimizer.zero_grad = types.MethodType(megatron_fsdp_optimizer_zero_grad, optimizer)
if preproc_state_dict_for_dcp_ckpt:
- # Requires a non-empty, DTensor-type optimizer state dictionary.
- # This is satisfied naturally by calling optimizer.state_dict()
- # after the first optimizer.step() initializes the state to match
- # the Megatron-FSDP
- optimizer_state_dict = optimizer.state_dict()
+
+ def dict_nested_shallow_copy(d: dict):
+ """Create a nested shallow copy of a dict. Same values, different pointers."""
+ if not isinstance(d, dict):
+ return d
+ return {
+ k: dict_nested_shallow_copy(v) if isinstance(v, dict) else v for k, v in d.items()
+ }
+
+ def preprocess_optimizer_state_dict_for_uneven_dtensor(optimizer, state_dict):
+ """
+ Hook that mocks the global optimizer state for unevenly-distributed
+ DTensors, as the optimizer state is only initialized for non-empty
+ parameters, and preprocesses the optimizer `state_dict` DTensors
+ in-place for Torch DCP.
+ """
+ # Retrieve a template optimizer state.
+ optim_state_template = next(iter(optimizer.state.values())) if optimizer.state else {}
+ # All-gather the optimizer state keys as this rank could have empty state.
+ optim_state_dtensor_keys = [None] * torch.distributed.get_world_size()
+ torch.distributed.all_gather_object(
+ optim_state_dtensor_keys,
+ [
+ # Only track keys associated with DTensors for DCP.
+ key
+ for key, val in optim_state_template.items()
+ if isinstance(val, DTensor)
+ ],
+ )
+ optim_state_dtensor_keys = list(
+ set([key for state in optim_state_dtensor_keys for key in state])
+ )
+
+ # NOTE(@cspades): Re-construct the Megatron-FSDP distributed parameter
+ # to index mapping as implemented in torch.optim.Optimizer.state_dict():
+ # https://github.com/pytorch/pytorch/blob/main/torch/optim/optimizer.py
+ # Simply put, the index maps to the very first appearance of id(param)
+ # looping through all parameters in all groups with memory address
+ # equivalent to the distributed parameter managed by Megatron-FSDP.
+ param_state_idx = {}
+ idx = 0
+ # For all empty parameters, mock empty DTensors for all empty parameters
+ # of Megatron-FSDP's unevenly-distributed optimizer state into a shallow
+ # copy of the state dictionary to synchronize and pre-process a global
+ # variant of the optimizer state in preparation for Torch DCP. This allows
+ # us to sync the non-empty DTensor shard metadata across sharding groups
+ # while excluding empty DTensor shards from the optimizer checkpoint.
+ optim_state_extended = dict_nested_shallow_copy(state_dict)
+ for param_group in optimizer.param_groups:
+ for param in param_group["params"]:
+ # Update the parameter state index.
+ # For shared params, use same index.
+ if id(param) not in param_state_idx:
+ # New parameter, assign an index.
+ param_state_idx[id(param)] = idx
+ idx += 1
+ if param in optimizer.state or not isinstance(param, DTensor):
+ # Only mock optimizer state for parameters that are missing state.
+ # No need to mock for non-DTensor params. Not relevant to DCP.
+ continue
+ for key in optim_state_dtensor_keys:
+ # Construct a mock DTensor state for the empty DTensor parameter.
+ param_idx = param_state_idx[id(param)]
+ optim_state_extended["state"].setdefault(param_idx, {})[key] = (
+ DTensor.from_local(
+ local_tensor=torch.empty(0, dtype=param.dtype, device=param.device),
+ device_mesh=param.device_mesh,
+ placements=param.placements,
+ shape=param.shape,
+ stride=param.stride(),
+ )
+ )
+
+ # Synchronize and preprocess DTensor metadata for Torch DCP.
+ preprocess_state_dict_for_uneven_dtensor(optim_state_extended)
+
+ # Attach the optimizer state_dict() post-hook to prepare DTensors for Torch DCP.
+ # args = (optimizer, state_dict)
optimizer.register_state_dict_post_hook(
- lambda *args, **kwargs: preprocess_state_dict_for_uneven_dtensor(optimizer_state_dict)
+ lambda *args, **kwargs: preprocess_optimizer_state_dict_for_uneven_dtensor(
+ args[0], args[1]
+ )
)
# Return the in-place modified optimizer.
return optimizer
+@experimental_api
def fully_shard(
module: torch.nn.Module,
optimizer: torch.optim.Optimizer,
@@ -521,25 +619,26 @@ def fully_shard(
dp_outer_dim: Optional[str] = None,
tp_dim: Optional[str] = None,
hybrid_fsdp_group: Optional[torch.distributed.ProcessGroup] = None,
+ hybrid_fsdp_expt_group: Optional[torch.distributed.ProcessGroup] = None,
expt_device_mesh: Optional[DeviceMesh] = None,
fsdp_unit_modules: Optional[Sequence[Type[torch.nn.Module]] | Sequence[str]] = None,
zero_dp_strategy: str | int = 3,
outer_dp_sharding_strategy: str | int = 0,
device: Optional[torch.device] = None,
init_model_with_meta_device: bool = False,
- grad_reduce_in_fp32: bool = False,
- preserve_fp32_weights: bool = True,
+ mixed_precision_policy: MixedPrecisionPolicy = MixedPrecisionPolicy(),
overlap_grad_reduce: bool = True,
overlap_param_gather: bool = True,
sync_model_each_microbatch: bool = True,
preproc_state_dict_for_dcp_ckpt: bool = True,
- check_for_nan_in_grad: bool = True,
+ report_nan_in_param_grad: bool = False,
average_in_collective: bool = False,
disable_bucketing: bool = False,
calculate_per_token_loss: bool = False,
keep_fp8_transpose_cache: bool = False,
nccl_ub: bool = False,
fsdp_double_buffer: bool = False,
+ fsdp_db_use_persist_buf_on_alloc_fail: bool = False,
disable_symmetric_registration: bool = False,
enable_fine_grained_param_gather: bool = False,
) -> tuple[MegatronFSDP, torch.optim.Optimizer]:
@@ -568,25 +667,26 @@ def fully_shard(
dp_outer_dim=dp_outer_dim,
tp_dim=tp_dim,
hybrid_fsdp_group=hybrid_fsdp_group,
+ hybrid_fsdp_expt_group=hybrid_fsdp_expt_group,
expt_device_mesh=expt_device_mesh,
fsdp_unit_modules=fsdp_unit_modules,
zero_dp_strategy=zero_dp_strategy,
outer_dp_sharding_strategy=outer_dp_sharding_strategy,
device=device,
init_model_with_meta_device=init_model_with_meta_device,
- grad_reduce_in_fp32=grad_reduce_in_fp32,
- preserve_fp32_weights=preserve_fp32_weights,
+ mixed_precision_policy=mixed_precision_policy,
overlap_grad_reduce=overlap_grad_reduce,
overlap_param_gather=overlap_param_gather,
sync_model_each_microbatch=sync_model_each_microbatch,
preproc_state_dict_for_dcp_ckpt=preproc_state_dict_for_dcp_ckpt,
- check_for_nan_in_grad=check_for_nan_in_grad,
+ report_nan_in_param_grad=report_nan_in_param_grad,
average_in_collective=average_in_collective,
disable_bucketing=disable_bucketing,
calculate_per_token_loss=calculate_per_token_loss,
keep_fp8_transpose_cache=keep_fp8_transpose_cache,
nccl_ub=nccl_ub,
fsdp_double_buffer=fsdp_double_buffer,
+ fsdp_db_use_persist_buf_on_alloc_fail=fsdp_db_use_persist_buf_on_alloc_fail,
disable_symmetric_registration=disable_symmetric_registration,
enable_fine_grained_param_gather=enable_fine_grained_param_gather,
)
diff --git a/megatron/core/distributed/fsdp/src/megatron_fsdp/megatron_fsdp.py b/megatron/core/distributed/fsdp/src/megatron_fsdp/megatron_fsdp.py
index bd13e76379e..f8640446814 100644
--- a/megatron/core/distributed/fsdp/src/megatron_fsdp/megatron_fsdp.py
+++ b/megatron/core/distributed/fsdp/src/megatron_fsdp/megatron_fsdp.py
@@ -24,6 +24,7 @@
from torch.utils._pytree import tree_flatten, tree_map, tree_unflatten
from .mixed_precision import (
+ MixedPrecisionPolicy,
fp8_create_transpose_cache,
fp8_discard_transpose_cache,
is_float8tensor,
@@ -34,10 +35,11 @@
GradReducePipeline,
ParamAndGradBuffer,
PrefetchOrder,
+ _check_nan_in_grad,
override_sharded_param_methods_with_safety_checks,
to_local_if_dtensor,
)
-from .utils import FSDPDistributedIndex
+from .utils import FSDPDistributedIndex, log_single_rank
logger = logging.getLogger(__name__)
@@ -102,9 +104,11 @@ class MegatronFSDP(torch.nn.Module):
module (torch.nn.Module): Underlying Torch Module.
dist_index (FSDPDistributedIndex): FSDPDistributedIndex object containing references to the
process groups and device meshes used by Megatron-FSDP.
- ddp_config (DistributedDataParallelConfig): FullyShardedDataParallel configuration dataclass
- containing a variety of Megatron-derived parameters that control the behavior of
- Megatron-FSDP.
+ ddp_config (DistributedDataParallelConfig): FullyShardedDataParallel configuration
+ dataclass containing a variety of Megatron-derived parameters that control the
+ behavior of Megatron-FSDP.
+ mixed_precision_policy (megatron_fsdp.MixedPrecisionPolicy): Configuration for
+ mixed-precision customization of compute and communications in Megatron-FSDP.
fsdp_unit_modules (List[torch.nn.Module] | List[str]): List of modules that
should be treated as an FSDP Unit, i.e. the minimum releasable model unit.
It affects the granularity of the communication parameter grouping and
@@ -136,11 +140,15 @@ class MegatronFSDP(torch.nn.Module):
fsdp_double_buffer (bool): Whether to use persistently allocated double buffers
for the temporary memory needed in the FSDP communication. This flag is
automatically set to True when nccl_ub is True.
+ fsdp_db_use_persist_buf_on_alloc_fail (bool): Whether to fall back to persistent buffer
+ allocator when a bucket does not fit FSDP double buffer size.
disable_symmetric_registration (bool): Whether to disable symmetric (window) registration
for NCCL userbuffer registration. This option will force to use conventional (local)
userbuffer registration when nccl_ub is set.
enable_fine_grained_param_gather (bool): Whether to enable "fine-grained" param all-gather,
which can improve performance when using MXFP8 parameters with activation recomputation.
+ report_nan_in_param_grad (bool): Whether to enable precise NaN-checking for parameter wgrad.
+ Can significantly degrade performance. Defaults to False.
Examples:
>>> model = GPTModel(config)
@@ -155,6 +163,7 @@ class MegatronFSDP(torch.nn.Module):
... keep_fp8_transpose_cache=False,
... nccl_ub=False,
... fsdp_double_buffer=False,
+ ... fsdp_db_use_persist_buf_on_alloc_fail=False,
... disable_symmetric_registration=False,
... )
"""
@@ -164,6 +173,7 @@ def __init__(
module: torch.nn.Module,
dist_index: FSDPDistributedIndex,
ddp_config: DistributedDataParallelConfig = None,
+ mixed_precision_policy: MixedPrecisionPolicy = MixedPrecisionPolicy(),
fsdp_unit_modules: Optional[List[torch.nn.Module] | List[str]] = None,
disable_bucketing: bool = False,
device: Optional[torch.device] = None,
@@ -173,8 +183,10 @@ def __init__(
keep_fp8_transpose_cache: bool = False,
nccl_ub: bool = False,
fsdp_double_buffer: bool = False,
+ fsdp_db_use_persist_buf_on_alloc_fail: bool = False,
disable_symmetric_registration: bool = False,
enable_fine_grained_param_gather_hook: bool = False,
+ report_nan_in_param_grad: bool = False,
):
super().__init__()
# If device is not specified, use the current device.
@@ -207,24 +219,31 @@ def __init__(
# "optim_grads_params" is the default strategy
if ddp_config is None:
self.ddp_config = DistributedDataParallelConfig(
- check_for_nan_in_grad=True,
data_parallel_sharding_strategy="optim_grads_params",
outer_dp_sharding_strategy="no_shard",
- grad_reduce_in_fp32=True,
overlap_grad_reduce=True,
overlap_param_gather=True,
average_in_collective=False,
keep_fp8_transpose_cache=keep_fp8_transpose_cache, # pylint: disable=C0301
nccl_ub=nccl_ub,
fsdp_double_buffer=fsdp_double_buffer or nccl_ub,
+ fsdp_db_use_persist_buf_on_alloc_fail=fsdp_db_use_persist_buf_on_alloc_fail,
disable_symmetric_registration=disable_symmetric_registration,
+ check_for_nan_in_grad=False,
)
else:
self.ddp_config = ddp_config
-
+ self.data_parallel_sharding_strategy = self.ddp_config.data_parallel_sharding_strategy
+ self.mp_policy = mixed_precision_policy
self.calculate_per_token_loss = calculate_per_token_loss
self.init_model_with_meta_device = init_model_with_meta_device
self.enable_fine_grained_param_gather_hook = enable_fine_grained_param_gather_hook
+ self.report_nan_in_param_grad = report_nan_in_param_grad
+
+ # FSDPDistributedIndex stores the process groups and meshes used by Megatron-FSDP.
+ # If not provided, Megatron-FSDP will default to a simple data parallel index
+ # supported by torch.distributed.group.WORLD.
+ self.dist_index = dist_index
# Whether to constantly synchronize the model every training iteration,
# which defaults to False to overlap communication with computation
@@ -237,11 +256,6 @@ def __init__(
# Check if the module contains (Megatron-Core) expert parallel parameters or DTensors.
has_expert_parameters = self._check_module_parameter_types()
- # FSDPDistributedIndex stores the process groups and meshes used by Megatron-FSDP.
- # If not provided, Megatron-FSDP will default to a simple data parallel index
- # supported by torch.distributed.group.WORLD.
- self.dist_index = dist_index
-
# If Megatron Expert Parallelism is enabled, you need to provide an expt_dp_group.
if (
has_expert_parameters
@@ -267,11 +281,8 @@ def __init__(
)
# Determine if we should delay the gradient reduction.
- self.is_delay_grad_reduce = self.ddp_config.data_parallel_sharding_strategy in [
- "no_shard",
- "optim",
- ]
- if self.ddp_config.data_parallel_sharding_strategy == "optim_grads_params":
+ self.is_delay_grad_reduce = self.data_parallel_sharding_strategy in ["no_shard", "optim"]
+ if self.data_parallel_sharding_strategy == "optim_grads_params":
# Default to overlapped parameter gather when fully-sharding.
self.ddp_config.overlap_param_gather = True
if self.ddp_config.data_parallel_sharding_strategy in ["optim_grads_params", "optim_grads"]:
@@ -337,7 +348,6 @@ def _init_fsdp_param_and_grad_buffer(self):
expert_gradient_scaling_factor = 1.0 / data_parallel_world_size
# Initialize the param and grad buffer.
- self.data_parallel_sharding_strategy = self.ddp_config.data_parallel_sharding_strategy
self.param_and_grad_buffer = ParamAndGradBuffer(
self.ddp_config,
self.module,
@@ -347,8 +357,7 @@ def _init_fsdp_param_and_grad_buffer(self):
data_parallel_sharding_strategy=self.data_parallel_sharding_strategy,
),
dist_index=self.dist_index,
- preserve_fp32_weights=self.ddp_config.preserve_fp32_weights,
- grad_reduce_in_fp32=self.ddp_config.grad_reduce_in_fp32,
+ mixed_precision_policy=self.mp_policy,
gradient_scaling_factor=gradient_scaling_factor,
expert_gradient_scaling_factor=expert_gradient_scaling_factor,
device=self.device,
@@ -567,18 +576,23 @@ def _grad_acc(param):
param.main_grad = param.get_main_grad()
if param.grad is not None:
+ if self.report_nan_in_param_grad:
+ _check_nan_in_grad(to_local_if_dtensor(param.grad))
# Copy the gradient into the allocated main gradient bucket.
# It will be reduce-scattered and accumulated into gbuf.
param.main_grad.copy_(to_local_if_dtensor(param.grad))
del param.grad
else:
+ # Prepare for fused wgrad accumulation.
param.main_grad.zero_()
# Unsharded Gradient Buffer
else:
if not param.grad_added_to_main_grad:
if param.grad is not None:
- # Add the gradient into the allocated main gradient bucket.
- # For unsharded gradients, this is gradient accumulation.
+ if self.report_nan_in_param_grad:
+ _check_nan_in_grad(to_local_if_dtensor(param.grad))
+ # Accumulate the gradient into the main gradient buffer,
+ # because we only reduce once per optimization cycle.
param.main_grad = param.get_main_grad()
param.main_grad.add_(to_local_if_dtensor(param.grad))
del param.grad
@@ -603,7 +617,7 @@ def _post_backward_release_module(module, *unused):
- Marks the module as IDLE in the training state machine.
"""
assert isinstance(module, tuple(fsdp_unit_modules))
- assert self.ddp_config.data_parallel_sharding_strategy == "optim_grads_params"
+ assert self.data_parallel_sharding_strategy == "optim_grads_params"
# Release parameters for this module after backward.
release_module_parameters(module, bwd=True)
@@ -651,9 +665,7 @@ def _process_post_backward_gradients(param_list):
for param in param_list:
_grad_acc(param)
- # Only reduce if gradients are sharded, or on the final microbatch, or when
- # model_auto_sync is enabled.
- grad_reduce_every_bprop = self.ddp_config.data_parallel_sharding_strategy in [
+ grad_reduce_every_bprop = self.data_parallel_sharding_strategy in [
"optim_grads",
"optim_grads_params",
]
@@ -667,6 +679,7 @@ def _process_post_backward_gradients(param_list):
param_list,
suggested_queue_capacity=self.suggested_RS_queue_capacity,
outer_fsdp_group_grad_reduce=(
+ # HSDP all-reduce or HFSDP reduce-scatter on the DP-Outer PG.
self.dist_index.use_hybrid_fsdp
and (is_last_microbatch or self.model_auto_sync)
),
@@ -678,8 +691,17 @@ def _process_post_backward_gradients(param_list):
@torch.compiler.disable
def _pre_forward_param_unshard(
- module: nn.Module, args: Tuple[Any, ...], kwargs: Dict[str, Any]
+ module: nn.Module,
+ args: Optional[Tuple[Any, ...]] = None,
+ kwargs: Optional[Dict[str, Any]] = None,
):
+ # If args or kwargs are not passed, default to () and {}.
+ # This matches PyTorch Module hook conventions:
+ # torch.nn.Module._call_impl.inner()
+ if args is None:
+ args = ()
+ if kwargs is None:
+ kwargs = {}
# Unshard the parameters before the forward pass.
input_training_state = module._training_state
fsdp_forward_prefetch = True
@@ -712,8 +734,8 @@ def _pre_forward_param_unshard(
def _register_post_backward_hook(
post_backward_hook: callable,
module: nn.Module,
- args: Tuple[Any, ...],
- kwargs: Dict[str, Any],
+ args: Optional[Tuple[Any, ...]] = None,
+ kwargs: Optional[Dict[str, Any]] = None,
):
"""
Register a post-backward hook for the given module by inserting an autograd
@@ -722,6 +744,13 @@ def _register_post_backward_hook(
since such operations can trigger an autograd error that
"the output is a view and is being modified in-place".
"""
+ # If args or kwargs are not passed, default to () and {}.
+ # This matches PyTorch Module hook conventions:
+ # torch.nn.Module._call_impl.inner()
+ if args is None:
+ args = ()
+ if kwargs is None:
+ kwargs = {}
if not torch.is_grad_enabled():
# No gradients / backward pass, don't attach the post-backward hook.
return args, kwargs
@@ -770,14 +799,14 @@ def _root_post_backward(*unused):
_grad_acc(param)
# Reduce the remaining gradients.
- grad_reduce_every_bprop = self.ddp_config.data_parallel_sharding_strategy in [
+ grad_reduce_every_bprop = self.data_parallel_sharding_strategy in [
"optim_grads",
"optim_grads_params",
]
# Only reduce if we are sharding gradients, or are on the final microbatch.
# If is_last_microbatch is not specified, then we should reduce gradients
# if model_auto_sync is enabled, otherwise wait until is_last_microbatch
- # is actually specified by the user, context manager, or FW before reduction.
+ # is specified by the user, context manager, or FW before reduction.
is_last_microbatch = getattr(self, "is_last_microbatch", False)
if grad_reduce_every_bprop or is_last_microbatch or self.model_auto_sync:
self.grad_reduce_pipeline.reduce_gradients(
@@ -841,7 +870,7 @@ def _root_pre_backward(module: nn.Module, *unused):
return
self._root_pre_backward_hook_issued = True
- if self.ddp_config.data_parallel_sharding_strategy == "optim_grads_params":
+ if self.data_parallel_sharding_strategy == "optim_grads_params":
for module in root_module.modules():
if isinstance(module, tuple(fsdp_unit_modules)):
# Set PRE_BACKWARD state to skip resharding and forward pre-fetching
@@ -935,7 +964,7 @@ def _register_pre_forward_param_unshard_hook(module):
If we are not sharding anything, we do not have a model weight buffer and thus
have nothing to all-gather / un-shard.
"""
- if self.ddp_config.data_parallel_sharding_strategy != "no_shard":
+ if self.data_parallel_sharding_strategy != "no_shard":
self.forward_pre_hooks[f"{module._get_name()} parameter unshard"] = (
module.register_forward_pre_hook(
_pre_forward_param_unshard, prepend=True, with_kwargs=True
@@ -978,7 +1007,7 @@ def _register_pre_backward_param_unshard_hook(module):
_register_pre_backward_param_unshard_hook(module)
elif (
not self.ddp_config.keep_fp8_transpose_cache
- and self.ddp_config.data_parallel_sharding_strategy == "optim_grads_params"
+ and self.data_parallel_sharding_strategy == "optim_grads_params"
):
# Register the forward post-hook to release FP8 transpose cache
# after the forward pass for non-FSDP unit modules.
@@ -1000,9 +1029,11 @@ def _register_pre_backward_param_unshard_hook(module):
with_kwargs=True,
)
)
- grad_acc_param_list = list(module.parameters())
+ grad_acc_param_list = [p for p in module.parameters() if p.requires_grad]
else:
- grad_acc_param_list = list(module.parameters(recurse=False))
+ grad_acc_param_list = [
+ p for p in module.parameters(recurse=False) if p.requires_grad
+ ]
for param in grad_acc_param_list:
self.grad_acc_hooks[f"grad_acc and reduce for {self.param_to_name[param]}"] = (
@@ -1090,6 +1121,23 @@ def set_model_auto_sync(self, sync_model: bool = True):
MegatronFSDP.model_auto_sync will be set to the value of sync_model.
Defaults to True. MegatronFSDP.model_auto_sync defaults to False.
"""
+ if sync_model and (
+ # Will reduce gradient buffer data in-place across DP-Shard.
+ self.data_parallel_sharding_strategy in ["no_shard", "optim"]
+ # Will reduce gradient buffer data in-place across DP-Outer.
+ or self.dist_index.use_hybrid_fsdp
+ ):
+ log_single_rank(
+ logger_=logger,
+ level=logging.WARNING,
+ msg="[Megatron-FSDP.set_model_auto_sync()] Detected Megatron-FSDP "
+ "auto-synchronization with the 'no_shard' / 'optim' sharding "
+ "strategy for either FSDP (DP-Shard) or HSDP (DP-Outer). "
+ "`MegatronFSDP.zero_grad_buffer()` or `optimizer.zero_grad()` "
+ "must be invoked every forward-backward optimization cycle to "
+ "prevent successive in-place gradient buffer reductions from "
+ "corrupting previously accumulated gradients.",
+ )
self.model_auto_sync = sync_model
def get_distributed_index(self) -> FSDPDistributedIndex:
@@ -1099,6 +1147,39 @@ def get_distributed_index(self) -> FSDPDistributedIndex:
"""
return self.dist_index
+ @contextmanager
+ def mixed_precision_context(self, mixed_precision_policy: MixedPrecisionPolicy):
+ """
+ Context manager for re-configuring the MixedPrecisionPolicy
+ for MegatronFSDP / ParamAndGradBuffer.
+ """
+ mp_policy_backup = self.mp_policy
+ self.reset_mixed_precision_policy(mixed_precision_policy)
+ try:
+ yield
+ finally:
+ self.reset_mixed_precision_policy(mp_policy_backup)
+
+ def reset_mixed_precision_policy(self, mixed_precision_policy: MixedPrecisionPolicy):
+ """
+ Re-configure MixedPrecisionPolicy for MegatronFSDP / ParamAndGradBuffer.
+ """
+ mp_policy_reset = MixedPrecisionPolicy(
+ # Preserve the original main parameter + gradient data-type.
+ main_params_dtype=self.mp_policy.main_params_dtype,
+ main_grads_dtype=self.mp_policy.main_grads_dtype,
+ # Gradient communication data-type can only be reset
+ # if symmetric buffers / NCCL UB are not used, because
+ # inflates FixedPoolAllocator memory & breaks NCCL UBR.
+ grad_comm_dtype=(
+ mixed_precision_policy.grad_comm_dtype
+ if self.ddp_config.nccl_ub or self.ddp_config.fsdp_double_buffer
+ else self.mp_policy.grad_comm_dtype
+ ),
+ )
+ self.mp_policy = mp_policy_reset
+ self.param_and_grad_buffer.mp_policy = mp_policy_reset
+
def start_param_sync(self, *unused, force_sync: bool = False, force_dispatch: bool = False):
"""
Initiates param sync (all-gather) communication operations for all model parameters.
@@ -1116,10 +1197,11 @@ def start_param_sync(self, *unused, force_sync: bool = False, force_dispatch: bo
if not force_sync and self.ddp_config.overlap_param_gather:
# All-gather the first bucket before the forward pass.
- first_param = list(self.module.parameters())[0]
- self.all_gather_and_wait_parameters_ready(
- params=[first_param], prefetch=True, wait_bucket_ready=False
- )
+ if self.ddp_config.fsdp_all_gather_in_start_param_sync:
+ first_param = list(self.module.parameters())[0]
+ self.all_gather_and_wait_parameters_ready(
+ params=[first_param], prefetch=True, wait_bucket_ready=False
+ )
else:
self.synchronize_param_gather()
for bucket_id in range(self.all_gather_pipeline.num_buckets):
diff --git a/megatron/core/distributed/fsdp/src/megatron_fsdp/mixed_precision.py b/megatron/core/distributed/fsdp/src/megatron_fsdp/mixed_precision.py
index 177e3b1caa2..89c67f40d41 100644
--- a/megatron/core/distributed/fsdp/src/megatron_fsdp/mixed_precision.py
+++ b/megatron/core/distributed/fsdp/src/megatron_fsdp/mixed_precision.py
@@ -14,6 +14,7 @@
import logging
from contextlib import nullcontext
+from dataclasses import dataclass
from importlib.metadata import version
from typing import List, Optional, Tuple
@@ -359,3 +360,35 @@ def get_quantized_model_init_context_cls():
f"Verify TransformerEngine is installed (TE_INSTALLED={HAVE_TE})."
)
return QUANTIZED_MODEL_INIT_CLASS
+
+
+@dataclass(frozen=True)
+class MixedPrecisionPolicy:
+ """Megatron-FSDP Mixed Precision Dataclass"""
+
+ main_params_dtype: Optional[torch.dtype] = torch.float32
+ """Data type for the main weight buffer utilized for distributed optimization
+ and quantization with Megatron-FSDP. If set to None, the model compute weight
+ buffer will take the role of the main weights, or when no sharding is applied,
+ the native model weights become the main weights. Defaults to torch.float32.
+ """
+
+ main_grads_dtype: Optional[torch.dtype] = None
+ """Data type for the main gradient buffer utilized for distributed optimization with
+ Megatron-FSDP. If set to None, main gradients will match the dtype of the model
+ compute parameters specified by the user model. Defaults to None.
+ """
+
+ grad_comm_dtype: Optional[torch.dtype] = None
+ """Data type for gradient gather / scatter communications. Can be utilized to reduce
+ communication latency, but adds overhead for type-casting and copy operations.
+ If using NCCL UBR v2.27+, gradient reduction may be performed in high-precision
+ depending on the network domain (NVLink or IB), and can enable mixed-precision
+ communication and accumulation, e.g. setting grad_comm_dtype to `BF16` can support
+ `FP32` reduction even though we have `BF16` input and output communication buffers.
+ If set to None, the `main_grads_dtype` is used. If using HSDP (either DP-Replicate
+ or DP-Outer in `outer_dp_sharding_strategy`), `no_shard`, `optim`, or a
+ `FixedPoolAllocator` (`fsdp_double_buffer`), allocating `dtype`-custom gradient
+ communication buffers (per FSDP group) adds memory overhead. Defaults to None.
+ No additional memory is allocated when `grad_comm_dtype == main_grads_dtype`.
+ """
diff --git a/megatron/core/distributed/fsdp/src/megatron_fsdp/param_and_grad_buffer.py b/megatron/core/distributed/fsdp/src/megatron_fsdp/param_and_grad_buffer.py
index 3ec117ebd9e..684cd7a99eb 100644
--- a/megatron/core/distributed/fsdp/src/megatron_fsdp/param_and_grad_buffer.py
+++ b/megatron/core/distributed/fsdp/src/megatron_fsdp/param_and_grad_buffer.py
@@ -33,6 +33,7 @@
from torch.distributed.tensor import DTensor, Replicate, Shard
from .mixed_precision import (
+ MixedPrecisionPolicy,
fp8_discard_transpose_cache,
fp8_get_raw_data,
fp8_need_transpose_data,
@@ -52,6 +53,7 @@
get_mcore_tensor_parallel_partition_dim,
is_mcore_tensor_model_parallel,
is_mcore_tensor_parallel_duplicated,
+ log_single_rank,
)
logger = logging.getLogger(__name__)
@@ -63,7 +65,7 @@
DistributedDataParallelConfig,
)
from megatron.core.tensor_parallel import get_cuda_rng_tracker
- from megatron.core.utils import is_submodule, log_single_rank
+ from megatron.core.utils import is_submodule
HAVE_MCORE = True
logger.info("Detected Megatron Core, using Megatron-FSDP with Megatron.")
@@ -74,17 +76,6 @@
from .utils import get_cuda_rng_tracker, is_submodule
HAVE_MCORE = False
-
- def log_single_rank(
- logger_: logging.Logger, level: int, msg: str, *args, rank: int = 0, **kwargs
- ):
- """Fallback log_single_rank when Megatron Core is not available."""
- if torch.distributed.is_initialized():
- if torch.distributed.get_rank() == rank:
- logger_.log(level, msg, *args, **kwargs)
- else:
- logger_.log(level, msg, *args, **kwargs)
-
logger.info("Megatron Core is not installed, Megatron-FSDP will run without Megatron Core.")
try:
@@ -658,7 +649,13 @@ class FixedPoolAllocator(TemporaryBucketAllocator):
deallocation of temporary buffers during FSDP operations.
"""
- def __init__(self, name: str, fsdp_param_groups: List["ParameterGroup"], size: int = 2):
+ def __init__(
+ self,
+ name: str,
+ fsdp_param_groups: List["ParameterGroup"],
+ size: int = 2,
+ fallback_to_persistent_buffer: bool = False,
+ ):
self.name = name
self.fsdp_param_groups = fsdp_param_groups
self.size = size # Number of buffers in the pool (default is 2 for double buffering)
@@ -691,6 +688,29 @@ def __init__(self, name: str, fsdp_param_groups: List["ParameterGroup"], size: i
), "Found no FSDP units to use fixed-size buffering"
self.fsdp_double_buffer_units = fsdp_units_to_double_buffer
+ if torch.distributed.get_rank() == 0:
+ for bucket_id, param_group in enumerate(fsdp_param_groups):
+ if (
+ param_group.fsdp_unit_id == -1
+ or param_group.fsdp_unit_id is None
+ or param_group.fsdp_unit_id not in self.fsdp_double_buffer_units
+ ):
+ logging.info(
+ f"FSDP unit (id={param_group.fsdp_unit_id}) does not fit "
+ "in FixedPoolAllcator"
+ )
+ if fallback_to_persistent_buffer is False:
+ logging.info(
+ "It will fall back to dynamic memory allocator, NCCL user "
+ "buffer is not supported"
+ )
+ else:
+ logging.info(
+ "It will be allocated a persistent buffer. If the memory "
+ "budget is tight, set "
+ "trainer.strategy.ddp.fsdp_db_use_persist_buf_on_alloc_fail to False."
+ )
+
# Initialize buffer group status.
# Each buffer group represents a set of buffers associated with an FSDP unit's bucket group.
self.idle_buffer = [] # List of available (buf_group_id, offset) tuples.
@@ -703,6 +723,7 @@ def __init__(self, name: str, fsdp_param_groups: List["ParameterGroup"], size: i
self.idle_buffer.append((buf_group_id, bucket_offset))
# Fallback allocator used if the fixed pool allocator cannot fulfill a request.
+ self.fallback_to_persistent_buffer = fallback_to_persistent_buffer
self.backup_allocator = TemporaryBucketAllocator()
def _is_two_bucket_group_equal(self, group_a, group_b):
@@ -755,28 +776,31 @@ def allocate(
f"current using_buffer: {self.using_buffer} \n"
f"current idle_buffer: {self.idle_buffer}"
)
- # Synchronization is required before the allocation for the user buffer
- if mem_alloc_context is not None and mem_alloc_context != nullcontext:
- # Check if a new buffer allocation is required
- if (
- self.allocation_tracker.get((buffer_name, dtype), None) is None
- or self.allocation_tracker[(buffer_name, dtype)] < size
- ):
- # Requires synchronization for new buffer allocation
- self.allocation_tracker[(buffer_name, dtype)] = size
- torch.cuda.synchronize()
- return Bucket(
- data=get_global_memory_buffer().get_tensor(
- [size], dtype=dtype, name=buffer_name, mem_alloc_context=mem_alloc_context
- )
+ elif self.fallback_to_persistent_buffer is True:
+ buffer_name = f"{self.name}_not_fit_in_fixed_pool_{bucket_id}_{size}_{dtype}_{device}"
+ else:
+ # If the bucket is not eligible for fixed pool buffering, or no buffer is available,
+ # fall back to dynamic allocation via the backup allocator. This means that we
+ # will do dynamic memory allocation.
+ logging.debug(f"[FSDP] Using backup allocator for {bucket_id} {fsdp_unit_id}")
+ return self.backup_allocator.allocate(
+ bucket_id=bucket_id, size=size, dtype=dtype, device=device
)
- # If the bucket is not eligible for fixed pool buffering, or no buffer is available,
- # fall back to dynamic allocation via the backup allocator. This means that we
- # will do dynamic memory allocation.
- logging.debug(f"[FSDP] Using backup allocator for {bucket_id} {fsdp_unit_id}")
- return self.backup_allocator.allocate(
- bucket_id=bucket_id, size=size, dtype=dtype, device=device
+ # Use buffer_name to get memory from global memory.
+ if mem_alloc_context is not None and mem_alloc_context != nullcontext:
+ # Check if a new buffer allocation is required
+ if (
+ self.allocation_tracker.get((buffer_name, dtype), None) is None
+ or self.allocation_tracker[(buffer_name, dtype)] < size
+ ):
+ # Requires synchronization for new buffer allocation
+ self.allocation_tracker[(buffer_name, dtype)] = size
+ torch.cuda.synchronize()
+ return Bucket(
+ data=get_global_memory_buffer().get_tensor(
+ [size], dtype=dtype, name=buffer_name, mem_alloc_context=mem_alloc_context
+ )
)
def _get_gbuf_name(self, buf_group_id: int, bucket_index: int):
@@ -795,9 +819,10 @@ def free(self, bucket_id: int):
self.idle_buffer.append(self.using_buffer[bucket_id])
del self.using_buffer[bucket_id]
return
- # If not managed by fixed pool allocator, delegate to the backup allocator.
- logging.debug(f"[FSDP] Free from the backup allocator for {bucket_id} {fsdp_unit_id}")
- self.backup_allocator.free(bucket_id)
+ if self.fallback_to_persistent_buffer is False:
+ # If not managed by fixed pool allocator, delegate to the backup allocator.
+ logging.debug(f"[FSDP] Free from the backup allocator for {bucket_id} {fsdp_unit_id}")
+ self.backup_allocator.free(bucket_id)
class DataParallelBuffer:
@@ -918,12 +943,14 @@ def fetch_bucket(
self, dtype: Optional[torch.dtype] = None, set_param_data: bool = False
) -> Bucket:
"""
- Fetch a communication buffer for data-parallel operations.
+ Fetch a communication buffer for data-parallel operations. If the buffer
+ is not distributed and doesn't have a custom data-type, the local un-sharded
+ buffer data will be returned. Otherwise, an un-sharded bucket is allocated.
The size of the bucket is defined by the `DataParallelBuffer` instance.
Args:
- dtype (Optional[torch.dtype], optional): The data type of the tensor
+ dtype (Optional[torch.dtype]): The data type of the tensor
to fetch a buffer for. Defaults to None.
Returns:
@@ -931,9 +958,10 @@ def fetch_bucket(
"""
if dtype is None:
dtype = self.dtype
- bucket_index = self.bucket_index
if not self.is_data_distributed and dtype == self.dtype:
+ # Use pre-allocated un-sharded bucket data as the communication buffer.
+ bucket_index = self.bucket_index
bucket = Bucket(
data=self.data[
bucket_index.global_data_index : bucket_index.global_data_index
@@ -941,17 +969,8 @@ def fetch_bucket(
]
)
else:
- # Bucket (unsharded) needs to be retrieved. If the temporary bucket cache
- # does not have the bucket corresponding to the bucket_id, it will allocate
- # a new Bucket with an empty tensor. Otherwise, it will simply return the
- # pre-allocated bucket with pre-existing data.
- bucket = self.temporary_bucket_allocator.allocate(
- bucket_id=bucket_index.bucket_id,
- size=bucket_index.size,
- dtype=dtype,
- device=self.device,
- mem_alloc_context=self.mem_alloc_context,
- )
+ # Sharded or dtype-custom buffers require un-sharded bucket allocation.
+ bucket = self.allocate_bucket_storage(dtype=dtype, device=self.device)
# Need to set parameter data after resize model weight buffer data-storage.
if set_param_data:
@@ -965,19 +984,72 @@ def fetch_bucket(
p.data = data
return bucket
- def free_bucket_storage(self):
+ def allocate_bucket_storage(
+ self,
+ shard: bool = False,
+ dtype: Optional[torch.dtype] = None,
+ device: Optional[torch.device] = None,
+ init_values: Optional[torch.Tensor] = None,
+ ) -> Bucket:
"""
- Release the storage of a temporary communication bucket.
- If the bucket is temporary, this method frees its storage.
+ Allocate a temporary flat communication buffer using the cached
+ allocator associated with this DataParallelBuffer. If no bucket
+ corresponding to the Bucket ID of this DataParallelBuffer has
+ been allocated yet, an empty Tensor will be allocated.
+
+ Can allocate sharded or un-sharded buckets.
+
+ Optionally, if init_values is provided, the flattened values of
+ the init_values will be copied into the newly-allocated storage.
+
+ Args:
+ shard (bool):
+ Whether to allocate a sharded or un-sharded bucket with
+ sizes defined by this DataParallelBuffer.
+ dtype (Optional[torch.dtype]):
+ Data-type of the allocated bucket.
+ device (Optional[torch.device]):
+ Device of the allocated bucket.
+ init_values (Optional[torch.Tensor]):
+ If provided, the allocated storage will be initialized
+ to the values of this (flattened) Tensor.
+
+ Returns:
+ Bucket: The communication buffer for the specified data type.
"""
- if not self.is_data_distributed:
- # Only free the allocated bucket if the buffer is sharded.
- # Otherwise, the buffer contains the entire bucket.
- return
+ # Default to buffer data-type.
+ if dtype is None:
+ dtype = self.dtype
+ if device is None:
+ device = self.device
+ # Allocate temporary storage using standardized sizes.
+ alloc_size = self.shard_bucket_index.size if shard else self.bucket_index.size
+ bucket = self.temporary_bucket_allocator.allocate(
+ bucket_id=self.bucket_index.bucket_id,
+ size=alloc_size,
+ dtype=dtype,
+ device=device,
+ mem_alloc_context=self.mem_alloc_context,
+ )
+ # Copy Tensor values into Bucket data.
+ if init_values is not None:
+ assert bucket.data.shape == init_values.flatten().shape, (
+ "[allocate_bucket_storage] Size mismatch between allocated bucket "
+ f"({bucket.data.shape}) and init_values ({init_values.shape})!"
+ )
+ bucket.data.detach().copy_(init_values.flatten())
+ return bucket
- # Free the memory backing the temporarily-allocated bucket associated with this buffer.
+ def free_bucket_storage(self):
+ """
+ Release the storage of a temporarily-allocated communication bucket.
+ """
self.temporary_bucket_allocator.free(self.bucket_index.bucket_id)
+ def reset_param_main_grad(self):
+ """
+ Dereference param.main_grad for grad buckets managed by this buffer.
+ """
# Reset the main grad tensor to None to release the memory.
for param in self.params:
if hasattr(param, "main_grad"):
@@ -1205,7 +1277,7 @@ def get_shard_from_bucket(self, bucket: Bucket):
def get_shard_from_local_buffer(self) -> torch.Tensor:
"""
- Get the shard or virtual shard of the bucket stored in this buffer.
+ Get the shard or virtual shard of the data persistently stored in this buffer.
"""
index = self.shard_bucket_index
# If the buffer is sharded, return the shard stored in this buffer.
@@ -1248,10 +1320,15 @@ class ParameterGroup:
Buffer used to store main gradients for data-parallel operations.
hsdp_wbuf (Optional[DataParallelBuffer]):
Buffer for weights used in Hybrid Sharded Data Parallel (HSDP).
- Exists only if full sharding is enabled in HSDP.
+ Exists only if full sharding (HFSDP) is enabled in HSDP.
hsdp_gbuf (Optional[DataParallelBuffer]):
Buffer for gradients used in HSDP.
- Exists only if full sharding is enabled in HSDP.
+ Exists only if full sharding (HFSDP) is enabled in HSDP.
+ hsdp_comm_gbuf (Optional[DataParallelBuffer]):
+ Extra buffer to allocate buffers that enable custom gradient
+ communication data-types when using HSDP or HFSDP only.
+ Only allocates memory when `MixedPrecisionPolicy.grad_comm_dtype`
+ is set to a non-`None` `torch.dtype`. Contains no local data.
"""
params: List[torch.nn.Parameter]
@@ -1266,6 +1343,7 @@ class ParameterGroup:
main_grad_buffer: Optional[DataParallelBuffer] = None
hsdp_wbuf: Optional[DataParallelBuffer] = None
hsdp_gbuf: Optional[DataParallelBuffer] = None
+ hsdp_comm_gbuf: Optional[DataParallelBuffer] = None
def _get_parameter_groups(
@@ -1531,11 +1609,10 @@ class ParamAndGradBuffer:
module (torch.nn.Module): The module whose parameters are to be grouped
and flatten.
bucketing_policy (BucketingPolicy): The bucketing policy.
- data_parallel_group (torch.distributed.ProcessGroup): The data parallel group.
- expert_data_parallel_group (Optional[torch.distributed.ProcessGroup]):
- The expert data parallel group.
- preserve_fp32_weights (bool): Whether to preserve FP32 weights.
- grad_reduce_in_fp32 (bool): Whether to reduce gradients in FP32.
+ dist_index (FSDPDistributedIndex): FSDPDistributedIndex object containing references
+ to the process groups and device meshes used by Megatron-FSDP.
+ mixed_precision_policy (megatron_fsdp.MixedPrecisionPolicy): Configuration for
+ mixed-precision customization of compute and communications in Megatron-FSDP.
gradient_scaling_factor (Optional[float]): The gradient scaling factor.
expert_gradient_scaling_factor (Optional[float]): The expert gradient
scaling factor.
@@ -1551,8 +1628,7 @@ def __init__(
module: torch.nn.Module,
bucketing_policy: BucketingPolicy,
dist_index: FSDPDistributedIndex,
- preserve_fp32_weights: bool = True,
- grad_reduce_in_fp32: bool = True,
+ mixed_precision_policy: MixedPrecisionPolicy = MixedPrecisionPolicy(),
gradient_scaling_factor: Optional[float] = None,
expert_gradient_scaling_factor: Optional[float] = None,
device: torch.device = torch.device("cuda"),
@@ -1569,8 +1645,7 @@ def __init__(
self.module = module
self.bucketing_policy = bucketing_policy
self.param_to_name = {p: name for name, p in self.module.named_parameters()}
- self.preserve_fp32_weights = preserve_fp32_weights
- self.grad_reduce_in_fp32 = grad_reduce_in_fp32
+ self.mp_policy = mixed_precision_policy
self.dist_index = dist_index
self.params = list(module.parameters())
self.gradient_scaling_factor = gradient_scaling_factor
@@ -1866,6 +1941,26 @@ def _init_each_parameter_group_buffers(self, meta_device_init_fp8_params):
raise ValueError(
f"Invalid data_parallel_sharding_strategy: {data_parallel_sharding_strategy}"
)
+
+ # Only create HSDP buffers if sharding on DP-Outer. Otherwise, no need to all-gather
+ # parameters on DP-Outer, but still need to all-reduce gradients on DP-Outer.
+ should_create_hfsdp_wbuf_and_gbuf = (
+ self.dist_index.use_hybrid_fsdp
+ and self.ddp_config.outer_dp_sharding_strategy != "no_shard"
+ )
+ # DP-Outer sharding is only supported for fully-sharded DP-Shard.
+ # NOTE(@cspades): Important guard for HFSDP functionality!
+ if (
+ should_create_hfsdp_wbuf_and_gbuf
+ and self.ddp_config.data_parallel_sharding_strategy != "optim_grads_params"
+ ):
+ raise NotImplementedError(
+ "[Megatron-FSDP] Optimizer fully-sharded HFSDP is only supported "
+ "with full-sharding on DP-Shard.\nMegatron-FSDP DP-Shard Strategy: "
+ f"{self.ddp_config.data_parallel_sharding_strategy}\nMegatron-FSDP "
+ f"DP-Outer Strategy: {self.ddp_config.outer_dp_sharding_strategy}"
+ )
+
if self.ddp_config.nccl_ub:
assert self.ddp_config.fsdp_double_buffer, (
"NCCL UB is only supported with FSDP double buffer. "
@@ -1874,7 +1969,10 @@ def _init_each_parameter_group_buffers(self, meta_device_init_fp8_params):
if self.ddp_config.fsdp_double_buffer and len(self.bucketing_policy.fsdp_unit_modules) > 0:
UB_BUFFER_NUM = 2
self.weight_alloc = FixedPoolAllocator(
- name="fsdp_params", fsdp_param_groups=self.parameter_groups, size=UB_BUFFER_NUM
+ name="fsdp_params",
+ fsdp_param_groups=self.parameter_groups,
+ size=UB_BUFFER_NUM,
+ fallback_to_persistent_buffer=self.ddp_config.fsdp_db_use_persist_buf_on_alloc_fail,
)
self.transpose_weight_alloc = FixedPoolAllocator(
name="fsdp_fp8_transpose_params",
@@ -1882,40 +1980,58 @@ def _init_each_parameter_group_buffers(self, meta_device_init_fp8_params):
size=UB_BUFFER_NUM,
)
self.main_grad_alloc = FixedPoolAllocator(
- name="fsdp_grads", fsdp_param_groups=self.parameter_groups, size=UB_BUFFER_NUM
+ name="fsdp_grads",
+ fsdp_param_groups=self.parameter_groups,
+ size=UB_BUFFER_NUM,
+ fallback_to_persistent_buffer=(
+ self.ddp_config.fsdp_db_use_persist_buf_on_alloc_fail
+ ),
)
+ if self.dist_index.use_hybrid_fsdp:
+ # Only required for custom communication dtype buffer allocation
+ # to leverage NCCL UBR for high-precision gradient reduction with
+ # low-precision gradient communication over DP-Outer for H(F)SDP.
+ # Otherwise, this allocator will never be used.
+ self.hsdp_grad_comm_alloc = FixedPoolAllocator(
+ name="hsdp_grad_comm",
+ fsdp_param_groups=self.parameter_groups,
+ size=UB_BUFFER_NUM,
+ fallback_to_persistent_buffer=(
+ self.ddp_config.fsdp_db_use_persist_buf_on_alloc_fail
+ ),
+ )
self.double_buf_units = self.weight_alloc.fsdp_double_buffer_units
else:
self.weight_alloc = StorageResizeBasedBucketAllocator()
self.transpose_weight_alloc = StorageResizeBasedBucketAllocator()
self.main_grad_alloc = None
+ if self.dist_index.use_hybrid_fsdp:
+ # Only required for custom communication dtype buffer allocation
+ # for low-precision gradient communication over DP-Outer for H(F)SDP.
+ # Otherwise, this allocator will never be used.
+ self.hsdp_grad_comm_alloc = None
self.double_buf_units = []
self.buffer_all_in_one = True
-
- preserve_fp32_weights = self.preserve_fp32_weights
- grad_reduce_in_fp32 = self.grad_reduce_in_fp32
buffer_size = {torch.float32: 0, torch.float16: 0, torch.bfloat16: 0, "float8": 0}
- # Only create HSDP buffers if sharding on DP-Outer. Otherwise, no need to all-gather
- # parameters on DP-Outer, but still need to all-reduce gradients on DP-Outer.
- should_create_hfsdp_wbuf_and_gbuf = (
- self.dist_index.use_hybrid_fsdp
- and self.ddp_config.outer_dp_sharding_strategy != "no_shard"
- )
-
# For all bucket groups (partitioned parameter groups)...
for group_id, group in enumerate(self.parameter_groups):
main_buf_extra_kwargs = {}
if should_create_hfsdp_wbuf_and_gbuf:
+ # DP-Outer + DP-Shard
main_buf_dp_group = self.dist_index.get_dp_group(
is_expert_parallel=group.is_expert_param
)
+ # DP-Shard
hsdp_buf_dp_group = self.dist_index.get_fsdp_group(
is_expert_parallel=group.is_expert_param
)
- main_buf_extra_kwargs["dp_rank"] = self.dist_index.get_logical_hybrid_fsdp_rank()
+ main_buf_extra_kwargs["dp_rank"] = self.dist_index.get_logical_hybrid_fsdp_rank(
+ is_expert_parallel=group.is_expert_param
+ )
else:
+ # DP-Shard only, since we're not sharding on DP-Outer.
main_buf_dp_group = self.dist_index.get_fsdp_group(
is_expert_parallel=group.is_expert_param
)
@@ -1943,12 +2059,17 @@ def _init_each_parameter_group_buffers(self, meta_device_init_fp8_params):
is_float8tensor(one_param)
or meta_device_init_fp8_params.get(self.param_to_name[one_param], (False, False))[0]
)
+
+ # Designate buffer data-types for compute parameters and main gradients.
if is_dtype_float8:
param_dtype = torch.uint8
- grad_dtype = torch.bfloat16
+ main_grads_dtype = torch.bfloat16
else:
param_dtype = group.params[0].dtype
- grad_dtype = param_dtype
+ main_grads_dtype = param_dtype
+ # Use a custom main gradient data-type.
+ if self.mp_policy.main_grads_dtype is not None:
+ main_grads_dtype = self.mp_policy.main_grads_dtype
# Check if the parameter group needs a transpose buffer for model weights.
# Currently, only mxfp8 needs it.
@@ -1975,6 +2096,9 @@ def _init_each_parameter_group_buffers(self, meta_device_init_fp8_params):
and model_wbuf_dp_group.size() > 1,
dtype=param_dtype,
device=self.device,
+ # Note: This will be DP-Outer + DP-Shard when sharding
+ # the optimizer state in HFSDP, else just DP-Shard when
+ # using basic HSDP or FSDP.
data_parallel_group=model_wbuf_dp_group,
is_transpose_buffer=False,
temporary_bucket_allocator=self.weight_alloc,
@@ -2000,14 +2124,18 @@ def _init_each_parameter_group_buffers(self, meta_device_init_fp8_params):
**main_buf_extra_kwargs,
)
- # Initialize the main weight buffer.
- if should_create_grad_buffer_or_main_weight_buffer and preserve_fp32_weights:
+ # Initialize the main weight buffer if a main weight data-type is specified.
+ # Otherwise, don't create this buffer, and use the model compute weight buffer instead.
+ if (
+ should_create_grad_buffer_or_main_weight_buffer
+ and self.mp_policy.main_params_dtype is not None
+ ):
group.main_weight_buffer = DataParallelBuffer(
self.ddp_config,
group.params,
is_data_distributed=is_main_weight_buffer_distributed
and main_buf_dp_group.size() > 1,
- dtype=torch.float32,
+ dtype=self.mp_policy.main_params_dtype,
device=self.device,
data_parallel_group=main_buf_dp_group,
bucket_id=group_id,
@@ -2019,16 +2147,19 @@ def _init_each_parameter_group_buffers(self, meta_device_init_fp8_params):
# Initialize the main grad buffer.
if should_create_grad_buffer_or_main_weight_buffer:
assert (
- grad_reduce_in_fp32 or grad_dtype is not torch.uint8
- ), "Gradients can not be reduced in FP8. "
+ main_grads_dtype.is_floating_point
+ ), f"Main gradient dtype ({main_grads_dtype}) must be Float."
group.main_grad_buffer = DataParallelBuffer(
self.ddp_config,
# Proxy because the number of gradient parameters is the same
# as the number of model parameters.
group.params,
is_data_distributed=is_grad_buffer_distributed and main_buf_dp_group.size() > 1,
- dtype=torch.float32 if grad_reduce_in_fp32 else grad_dtype,
+ dtype=main_grads_dtype,
device=self.device,
+ # Note: This will be DP-Outer + DP-Shard when sharding
+ # the optimizer state in HFSDP, else just DP-Shard when
+ # using basic HSDP or FSDP.
data_parallel_group=main_buf_dp_group,
is_transpose_buffer=False,
temporary_bucket_allocator=self.main_grad_alloc,
@@ -2071,7 +2202,13 @@ def _init_each_parameter_group_buffers(self, meta_device_init_fp8_params):
)
if group.transpose_weight_buffer is not None:
- raise NotImplementedError("HSDP for transpose buffer is not implemented yet")
+ # TODO(@kunlunl, @cspades): Create a hybrid-sharded transpose buffer
+ # to map fully-sharded transpose weights to partially-sharded transpose
+ # weights before and after fully-distributed optimization.
+ raise NotImplementedError(
+ "HFSDP (HSDP + fully-sharded optimizer state) doesn't "
+ "support FP8 recipes that require a transpose buffer."
+ )
if should_create_grad_buffer_or_main_weight_buffer:
# Initialize the HSDP grad buffer.
@@ -2103,6 +2240,44 @@ def _init_each_parameter_group_buffers(self, meta_device_init_fp8_params):
buffer_size[group.main_grad_buffer.dtype] -= group.main_grad_buffer.data_size
buffer_size[group.main_grad_buffer.dtype] += group.hsdp_gbuf.data_size
+ # Only create an extra grad comm buffer for HSDP.
+ if should_create_grad_buffer_or_main_weight_buffer and self.dist_index.use_hybrid_fsdp:
+ # Create a (DP-Shard)-sized gradient communication buffer that manages
+ # allocation for custom gradient communication data-types during runtime.
+ # If gradient communication data-type customization is not used, the
+ # allocator associated with this buffer will never allocate memory.
+ # This buffer will never initialize local data, i.e. self.data = None.
+ gbuf = group.main_grad_buffer
+ fsdp_group = self.dist_index.get_fsdp_group(
+ is_expert_parallel=group.is_expert_param
+ )
+ hfsdp_kwargs = {}
+ if should_create_hfsdp_wbuf_and_gbuf:
+ hfsdp_kwargs["item_index_map"] = gbuf.item_index_map
+ hfsdp_kwargs["bucket_index"] = gbuf.bucket_index
+ hfsdp_kwargs["shard_bucket_index"] = _get_dp_buffer_shard_bucket_index(
+ gbuf.bucket_index,
+ is_data_distributed=is_grad_buffer_distributed and fsdp_group.size() > 1,
+ data_parallel_world_size=fsdp_group.size(),
+ data_parallel_rank=fsdp_group.rank(),
+ )
+ group.hsdp_comm_gbuf = DataParallelBuffer(
+ self.ddp_config,
+ group.params,
+ is_data_distributed=is_grad_buffer_distributed and fsdp_group.size() > 1,
+ # Set allocation to grad_comm_dtype, or default to param(.grad).dtype.
+ dtype=self.mp_policy.grad_comm_dtype,
+ device=gbuf.device,
+ data_parallel_group=fsdp_group,
+ is_transpose_buffer=False,
+ # Use the HSDP gradient communication allocator!
+ temporary_bucket_allocator=self.hsdp_grad_comm_alloc,
+ bucket_id=group_id,
+ chunk_size_factor=group.chunk_size_factor,
+ mem_alloc_context=self.mem_alloc_context,
+ **hfsdp_kwargs,
+ )
+
reset_context_args = {"init_param_with_fp8": self.ddp_config.fp8_param_gather}
module_reset_flag = {}
if self.reset_parameters_for_meta_device_init_module:
@@ -2155,6 +2330,7 @@ def _init_each_parameter_group_buffers(self, meta_device_init_fp8_params):
)
outer_fsdp_group = self.dist_index.get_outer_fsdp_group()
wbuf_data = hsdp_wbuf.data[
+ # Requires FSDP sharding for (DP-Shard, DP-Outer) to cover DP-Shard.
wbuf.data_size
* outer_fsdp_group.rank() : wbuf.data_size
* (outer_fsdp_group.rank() + 1)
@@ -2172,9 +2348,11 @@ def _init_each_parameter_group_buffers(self, meta_device_init_fp8_params):
with self.mem_alloc_context():
if group.hsdp_wbuf:
raise NotImplementedError(
- "HSDP for transpose buffer is not implemented yet"
+ "HFSDP (HSDP + fully-sharded optimizer state) doesn't "
+ "support FP8 recipes that require a transpose buffer."
)
else:
+ # Initialize the transpose buffer.
tbuf.init_data(
torch.empty(tbuf.data_size, dtype=tbuf.dtype, device=self.device)
)
@@ -2257,18 +2435,24 @@ def _init_each_parameter_group_buffers(self, meta_device_init_fp8_params):
new_transpose_data = None
if is_float8tensor(p_local):
+ # Attach FP8 row-wise data in the FP8 parameter
+ # to slice of the model compute weight bucket.
old_param_data = fp8_get_raw_data(p_local)
assert old_param_data._base is None
new_param_data.detach().copy_(old_param_data)
fp8_set_raw_data(p_local, new_param_data)
del old_param_data
if new_transpose_data is not None:
+ # Attach FP8 col-wise data in the FP8 parameter
+ # to slice of the FP8 transpose bucket.
old_transpose_data = fp8_get_raw_data(p_local, True)
assert old_transpose_data._base is None
new_transpose_data.detach().copy_(old_transpose_data)
fp8_set_raw_data(p_local, new_transpose_data, True)
del old_transpose_data
elif isinstance(p, DTensor):
+ # Same as Tensor case, except for DTensor parameters
+ # in the original model. Tensor = DTensor.to_local().
old_param_data = p._local_tensor.data
p._local_tensor.data = new_param_data
assert old_param_data._base is None
@@ -2381,6 +2565,7 @@ def _alloc(dtype, size):
hsdp_gbuf.init_data(_alloc(hsdp_gbuf.dtype, hsdp_gbuf.data_size))
outer_fsdp_group = self.dist_index.get_outer_fsdp_group()
gbuf_data = hsdp_gbuf.data[
+ # Requires FSDP sharding for (DP-Shard, DP-Outer) to cover DP-Shard.
gbuf.data_size
* outer_fsdp_group.rank() : gbuf.data_size
* (outer_fsdp_group.rank() + 1)
@@ -2399,7 +2584,15 @@ def _alloc(dtype, size):
def main_grad_getter(p):
# Make sure main_grad memory is allocated when initially accessed.
- bucket = p._gbuf.fetch_bucket()
+ # When gradients are sharded, we can pre-allocate a communication
+ # bucket to avoid casting to a communication data-type. Otherwise,
+ # return the item backed by the main gradient buffer required to
+ # support un-sharded gradient accumulation at high precision.
+ bucket = p._gbuf.fetch_bucket(
+ dtype=(
+ self.mp_policy.grad_comm_dtype if p._gbuf.is_data_distributed else None
+ )
+ )
gbuf = p._gbuf
item_id = p._item_id
# View it as p.shape so you can insert the param.grad into
@@ -2411,6 +2604,8 @@ def main_grad_getter(p):
# Patch the parameter class to include a main_grad property.
# Utilized in the gradient reduction pipeline to save computed
# data-parallel gradients on every rank and reduce-scatter them.
+ # Enables TransformerEngine's fuse_wgrad_accumulation=True feature
+ # which dumps gradients into param.main_grad with zero-copy.
p.get_main_grad = main_grad_getter.__get__(p)
# Clean up deallocated memory.
@@ -2431,6 +2626,8 @@ def _reset_parameters(self, old_params, new_params):
self.param_to_direct_module[new_param] = self.param_to_direct_module[old_param]
del self.param_to_direct_module[old_param]
+ new_param.requires_grad_(old_param.requires_grad)
+
for tp_attr in ["_mcore_tp", "_tp_partition_dim", "_tp_duplicated"]:
if getattr(old_param, tp_attr, None) is not None:
setattr(new_param, tp_attr, getattr(old_param, tp_attr))
@@ -2490,8 +2687,8 @@ def _init_distributed_params(self):
Register model training and high-precision parameters as optimizer
named parameters and DTensor(s). Specifically, we utilize the highest
precision weights available for optimization using fall-back logic
- on mbuf -> wbuf -> orig_param depending on if preserve_fp32_weights
- or "no_shard" is utilized.
+ on mbuf -> wbuf -> orig_param depending on if main_params_dtype is
+ specified or "no_shard" is utilized.
"""
dist_main_weight = {}
for pg in self.parameter_groups:
@@ -2583,6 +2780,7 @@ def set_param_attribute():
"partition_dim",
"partition_stride",
"is_embedding_or_output_parameter",
+ "is_embedding_parameter",
"_mcore_tp",
"_tp_duplicated",
"_tp_partition_dim",
@@ -2922,6 +3120,8 @@ def reduce_scatter_gradients(self, async_op: bool = True):
if gbuf is None:
continue
scaling_factor = gbuf.gradient_scaling_factor
+ if self.ddp_config.check_for_nan_in_grad:
+ _check_nan_in_grad(gbuf.data)
reduce_op = gradient_reduce_preprocessing(gbuf.data, scaling_factor, self.ddp_config)
reduce_scatter_handler = torch.distributed.reduce_scatter_tensor(
output=gbuf.get_shard_from_local_buffer(),
@@ -2960,6 +3160,8 @@ def all_reduce_gradients(self, async_op: bool = False):
if gbuf is not None:
continue
scaling_factor = gbuf.gradient_scaling_factor
+ if self.ddp_config.check_for_nan_in_grad:
+ _check_nan_in_grad(gbuf.data)
reduce_op = gradient_reduce_preprocessing(gbuf.data, scaling_factor, self.ddp_config)
all_reduce_handler = torch.distributed.all_reduce(
gbuf.data, op=reduce_op, group=gbuf.data_parallel_group, async_op=async_op
@@ -3043,6 +3245,7 @@ def reset(self):
for bucket_id, _ in self.bucket_status.items():
gbuf = self.get_fsdp_buffer(bucket_id)
gbuf.free_bucket_storage()
+ gbuf.reset_param_main_grad()
self.bucket_status[bucket_id] = BucketStatus.EMPTY
def reduce_gradients(
@@ -3121,9 +3324,6 @@ def wait_for_previous_grad_reduce(
grad_reduce_event.wait()
free_up_grad_bucket()
- if suggested_queue_size == 0 and self.outer_fsdp_group_grad_reduce:
- torch.cuda.current_stream().wait_stream(self.outer_fsdp_group_grad_reduce_stream)
-
def _enforce_double_buffer_limit(self, add_buckets):
if not self.buffer.ddp_config.fsdp_double_buffer:
return
@@ -3199,6 +3399,7 @@ def _bucket_group_gradient_reduce(
# released ensures that our double buffer will not explode due to too
# many empty bucket requests.
ddp_config = self.buffer.ddp_config
+ mp_policy = self.buffer.mp_policy
if ddp_config.fsdp_double_buffer:
self._enforce_double_buffer_limit(bucket_group)
@@ -3208,105 +3409,188 @@ def _bucket_group_gradient_reduce(
)
reduce_scatter_stream.wait_stream(current_stream)
+ # DP-Shard Gradient Reduction
dp_group = self.get_fsdp_buffer(bucket_group[0]).data_parallel_group
with torch.cuda.stream(reduce_scatter_stream):
with _coalescing_manager(dp_group):
- grad_buffer = []
- reduced_grad = []
+ # List of gradient accumulation closure tasks.
+ # (grad_buffer, reduced_grad)
+ grad_accum_closure = []
for bucket_id in bucket_group:
- # Fetch pre-allocated main gradient bucket.
+
+ # Get the DP-Shard gradient buffer associated with this bucket ID.
gbuf = self.get_fsdp_buffer(bucket_id)
- bucket = gbuf.fetch_bucket()
- # Scale gradients.
+
+ # Get the unreduced gradients associated with the gradient buffer.
+ unreduced_grad_bucket = gbuf.fetch_bucket(
+ dtype=mp_policy.grad_comm_dtype if gbuf.is_data_distributed else None
+ )
+ # NOTE(@cspades): `no_shard` or `optim`
+ # Un-sharded gradient buffers accumulate un-reduced gradients locally
+ # without allocating an un-sharded buffer. For custom communication
+ # data-type(s), an extra un-sharded buffer needs to be allocated!
+ custom_grad_comm_dtype = (
+ mp_policy.grad_comm_dtype is not None
+ and unreduced_grad_bucket.data.dtype != mp_policy.grad_comm_dtype
+ )
+ if not gbuf.is_data_distributed and custom_grad_comm_dtype:
+ # Create a custom communication buffer with gbuf.
+ # Introduces copy and memory overhead.
+ unreduced_grad_bucket = gbuf.allocate_bucket_storage(
+ dtype=mp_policy.grad_comm_dtype,
+ device=unreduced_grad_bucket.data.device,
+ init_values=unreduced_grad_bucket.data,
+ )
+ unreduced_grad = unreduced_grad_bucket.data
+
+ # Pre-scale unsharded bucket gradient and prepare the ReduceOp.
scaling_factor = gbuf.gradient_scaling_factor
reduce_op = gradient_reduce_preprocessing(
- bucket.data, scaling_factor, gbuf.ddp_config
+ unreduced_grad, scaling_factor, ddp_config
)
+
+ # Reduce-scatter or all-reduce the unsharded gradient.
if ddp_config.data_parallel_sharding_strategy == "no_shard":
- # All-reduce the gradients on every rank. No scattering
- # or sharding necessary.
+ # All-reduce un-sharded gradients from every rank.
torch.distributed.all_reduce(
- bucket.data, op=reduce_op, group=gbuf.data_parallel_group
+ unreduced_grad, op=reduce_op, group=gbuf.data_parallel_group
)
+ if custom_grad_comm_dtype:
+ # Reduction used a temporary communication buffer.
+ grad_accum_closure.append(
+ # Un-sharded buffer data.
+ (gbuf.data, unreduced_grad)
+ )
else:
- # Get the shard of the gradient from the pre-allocated bucket.
- # The reduced gradient will be scattered into this shard of the
- # bucket managed by the sharded buffer on this rank.
- grad_shard = gbuf.get_shard_from_bucket(bucket)
- # pylint: disable=C0301
- # The `grad_shard`` is part of `bucket.data`` and the following
- # new empty is important for memory safety, when using
- # TORCH_NCCL_AVOID_RECORD_STREAMS=1.
- # For reference: https://dev-discuss.pytorch.org/t/fsdp-cudacachingallocator-an-outsider-newb-perspective/1486
- if not self.buffer.ddp_config.fsdp_double_buffer:
- grad_shard = torch.empty_like(grad_shard)
- # Reduce-scatter gradients on the FSDP group.
+ # Slice a gradient shard from the communication bucket.
+ grad_shard = gbuf.get_shard_from_bucket(unreduced_grad_bucket)
+
+ # Execute the reduce-scatter collective.
torch.distributed.reduce_scatter_tensor(
output=grad_shard,
- input=bucket.data,
+ input=unreduced_grad,
op=reduce_op,
group=gbuf.data_parallel_group,
)
- reduced_grad.append(grad_shard)
- grad_buffer.append(gbuf.get_shard_from_local_buffer())
+
+ # Track closure tasks to accumulate the reduced gradient shard.
+ # NOTE: If the gradient buffer is unsharded and no communication
+ # bucket is allocated, then the output bucket shard is backed by
+ # the unsharded gradient buffer and the reduce-scatter result
+ # has already been installed into the gradient buffer.
+ if gbuf.is_data_distributed or custom_grad_comm_dtype:
+ grad_accum_closure.append(
+ # Target for sharded or un-sharded gradient buffers.
+ (gbuf.get_shard_from_local_buffer(), grad_shard)
+ )
+
+ # Mark bucket ID as CUDA work-in-progress.
self.bucket_status[bucket_id] = BucketStatus.COMMUNICATING
- for local_grad, reduced_grad in zip(grad_buffer, reduced_grad):
- # Accumulate the reduced gradient shard into the local gradient buffer,
- # when ZeRO-2 (gradient sharding) is enabled. Otherwise, bucket.data
- # is equivalent to the buffer data and will have been all-reduced.
- local_grad += reduced_grad
+
+ for local_grad, reduced_grad in grad_accum_closure:
+ if ddp_config.data_parallel_sharding_strategy in ["no_shard", "optim"]:
+ # Copy the reduced gradient into the main gradient buffer.
+ local_grad.copy_(reduced_grad)
+ else:
+ # Accumulate the reduced gradient into the local gradient buffer.
+ # Accumulation data-type is type-promoted with respect to the
+ # accumulated gradient and the buffer main_grads_dtype.
+ local_grad += reduced_grad
+
# Record a checkpoint for the event to synchronize against the reduce-scatter stream.
reduce_scatter_view_out_event = reduce_scatter_stream.record_event()
- # Outer-DP group gradient reduction.
+ # DP-Outer Gradient Reduction
if outer_fsdp_group_grad_reduce:
+ # Wait on the DP-Shard reduction before further reduction.
self.outer_fsdp_group_grad_reduce_stream.wait_stream(reduce_scatter_stream)
outer_fsdp_group = self.buffer.dist_index.get_outer_fsdp_group()
with torch.cuda.stream(self.outer_fsdp_group_grad_reduce_stream):
with _coalescing_manager(outer_fsdp_group):
- reduced_grad = []
+ # List of gradient accumulation closure tasks.
+ # (grad_buffer, reduced_grad)
+ grad_accum_closure = []
for bucket_id in bucket_group:
+ # Skip gradient scaling for DP-Outer, because the
+ # (DP-Shard, DP-Outer) scaling is already applied.
if ddp_config.average_in_collective:
reduce_op = torch.distributed.ReduceOp.AVG
else:
reduce_op = torch.distributed.ReduceOp.SUM
- # All-reduce/reduce-scatter the gradients on every rank
- # in the outer-DP group.
- gbuf = self.get_fsdp_buffer(bucket_id)
+ # (DP-Shard, DP-Outer) if HFSDP, otherwise just DP-Shard for HSDP
+ main_grad_buffer = self.buffer.parameter_groups[bucket_id].main_grad_buffer
+
+ # FSDP buffer can be un-sharded or sharded for HSDP, but sharded for HFSDP.
+ # TODO(@cspades): For `optim`, we don't need to reduce the local un-sharded
+ # gradient, just the shard updated via reduce-scatter.
+ fsdp_grad_buffer = self.get_fsdp_buffer(bucket_id)
+ unreduced_grad = fsdp_grad_buffer.data
+ assert (
+ main_grad_buffer.dtype == fsdp_grad_buffer.dtype
+ ), "Main and DP-Shard gradient buffer must share the exact same dtype."
+
+ # Cast DP-Shard gradient to communication dtype if specified and necessary.
+ custom_grad_comm_dtype = (
+ mp_policy.grad_comm_dtype is not None
+ and unreduced_grad.dtype != mp_policy.grad_comm_dtype
+ )
+ if custom_grad_comm_dtype:
+ # Allocate a custom communication buffer with the HSDP gradient
+ # communication buffer. Introduces copy and memory overhead.
+ hsdp_comm_gbuf = self.buffer.parameter_groups[bucket_id].hsdp_comm_gbuf
+ unreduced_grad = hsdp_comm_gbuf.allocate_bucket_storage(
+ # Allocate memory for the sharded or un-sharded
+ # gradient reduced over DP-Shard.
+ shard=fsdp_grad_buffer.is_data_distributed,
+ dtype=mp_policy.grad_comm_dtype,
+ device=unreduced_grad.device,
+ init_values=unreduced_grad,
+ ).data
+
+ # All-reduce or reduce-scatter the DP-Shard gradients across DP-Outer.
if ddp_config.outer_dp_sharding_strategy != "no_shard":
- # Outer-DP sharding is only supported for fully-sharded inner-DP.
- assert ddp_config.data_parallel_sharding_strategy != "no_shard"
# Retrieve the (DP-Outer, DP-Shard) gradient shard from the
# main gradient buffer which shards across the entire DP group,
# i.e. across all DP-Shard and DP-Outer ranks.
- grad_full_shard = self.buffer.parameter_groups[
- bucket_id
- ].main_grad_buffer.get_shard_from_local_buffer()
- # NOTE: This is a fix for convergence, needed to make
- # sure NCCL reduce-scatter inplace didn't seem
- # to work correctly
- grad_full_shard = torch.empty_like(grad_full_shard)
- reduced_grad.append(grad_full_shard)
+ main_grad_shard = main_grad_buffer.get_shard_from_local_buffer()
+ if custom_grad_comm_dtype:
+ # Scatter back into communication buffer.
+ dp_outer_rank = outer_fsdp_group.rank()
+ output_buffer = unreduced_grad[
+ dp_outer_rank
+ * main_grad_shard.numel() : (dp_outer_rank + 1)
+ * main_grad_shard.numel()
+ ]
+ else:
+ # Scatter directly into the main gradient buffer.
+ output_buffer = main_grad_shard
# Reduce-scatter the FSDP gradient buffer shard further
# into the (DP-Outer, DP-Shard) gradient shard.
torch.distributed.reduce_scatter_tensor(
- output=grad_full_shard,
- input=gbuf.data,
+ output=output_buffer,
+ input=unreduced_grad,
op=reduce_op,
group=outer_fsdp_group,
)
- else:
- # No outer-DP sharding, so just all-reduce the FSDP gradient
- # buffer shard into itself.
+ if custom_grad_comm_dtype:
+ # Reduce-scatter output was a temporary communication buffer.
+ grad_accum_closure.append((main_grad_shard, output_buffer))
+ else: # HSDP -> main_grad_buffer = (DP-Shard,)
+ # No DP-Outer sharding, so all-reduce FSDP gradients across DP-Outer.
+ # All FSDP buffers will have reduced un-sharded or sharded gradients.
torch.distributed.all_reduce(
- gbuf.data, group=outer_fsdp_group, op=reduce_op
+ unreduced_grad, group=outer_fsdp_group, op=reduce_op
)
- for bucket_id, grad_full_shard in zip(bucket_group, reduced_grad):
+ if custom_grad_comm_dtype:
+ # Reduction used a temporary communication buffer.
+ grad_accum_closure.append((main_grad_buffer.data, unreduced_grad))
+
+ for main_grad_buffer, reduced_grad in grad_accum_closure:
# Update the (DP-Outer, DP-Shard) gradient shard in the main gradient buffer.
- self.buffer.parameter_groups[
- bucket_id
- ].main_grad_buffer.get_shard_from_local_buffer().copy_(grad_full_shard)
+ # No accumulation should happen in the (DP-Shard, DP-Outer) gradient buffer.
+ main_grad_buffer.copy_(reduced_grad)
+
reduce_scatter_view_out_event = self.outer_fsdp_group_grad_reduce_stream.record_event()
free_up_grad_bucket_func = {}
@@ -3317,10 +3601,16 @@ def free_up_grad_bucket():
# Empty the set of parameters that are ready for gradient reduction.
self.bucket_grad_ready_params[bucket_id] = set()
gbuf = self.get_fsdp_buffer(bucket_id)
- if gbuf.is_data_distributed:
- # Free the memory backing the temporarily-allocated bucket
- # associated with this buffer.
- gbuf.free_bucket_storage()
+ # Free the memory backing the temporarily-allocated communication
+ # bucket associated with this buffer. Only exists for sharded
+ # gradient buffers, or if a custom gradient data-type is used!
+ gbuf.free_bucket_storage()
+ # Gradient reduction completed, can de-reference param.main_grad.
+ gbuf.reset_param_main_grad()
+ hsdp_comm_gbuf = self.buffer.parameter_groups[bucket_id].hsdp_comm_gbuf
+ if hsdp_comm_gbuf is not None:
+ # Also de-allocate any communication buffers used for H(F)SDP.
+ hsdp_comm_gbuf.free_bucket_storage()
# Mark the bucket as deallocated / empty.
self.bucket_status[bucket_id] = BucketStatus.EMPTY
@@ -3465,7 +3755,10 @@ def all_gather_params(
suggested_AG_prefetch_size (Optional[int], optional):
The suggested prefetch size for all-gathering. Defaults to None.
outer_fsdp_group_param_gather (bool, optional):
- Whether to all-gather parameters across outer-DP groups. Defaults to False.
+ Whether to all-gather parameters across DP-Outer. Defaults to False.
+ bwd (bool, optional):
+ Whether to all-gather column-wise parameters instead of row-wise parameters
+ for the backward pass for formats that require a transpose buffer like MXFP8.
"""
if len(params) == 0:
return
@@ -3582,7 +3875,7 @@ def need_skip_prefetch(bucket_id):
self.ag_stream if self.ag_stream is not None else torch.cuda.current_stream()
)
if outer_fsdp_group_param_gather:
- # TODO(mxfp8): Support hsdp
+ # TODO(@kunlunl): Support MXFP8 with HFSDP. Requires an HFSDP transpose buffer.
self.outer_fsdp_group_param_gather_stream.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(self.outer_fsdp_group_param_gather_stream):
outer_fsdp_group = self.buffer.dist_index.get_outer_fsdp_group()
@@ -3597,7 +3890,7 @@ def need_skip_prefetch(bucket_id):
input_tensor=wbuf.data,
group=outer_fsdp_group,
)
- # Wait for the outer-DP group all-gather to finish.
+ # Wait for the DP-Outer group all-gather to finish.
all_gather_stream.wait_stream(self.outer_fsdp_group_param_gather_stream)
# Coalesce the asynchronous NCCL operations in this context.
@@ -3672,6 +3965,11 @@ def release_bucket(self, bucket_id, bwd, lazy: bool = False):
they are no longer needed.
- If the bucket has a transpose weight buffer (used in FP8 backward passes),
this buffer is freed; otherwise, the model weight buffer is released.
+ - This function should NOT be invoked on buckets associated with modules not
+ identified as FSDP unit modules, even when weights are sharded in the case of
+ `optim_grads_params`. Non-unit modules should remain persistently allocated
+ because they do not satisfy FSDP unit module state requirements, e.g. their
+ parameters are simultaneously modified or shared with other modules.
"""
bucket_key = self.get_bucket_key(bucket_id, bwd)
if self.bucket_status[bucket_key] == BucketStatus.EMPTY:
@@ -3703,7 +4001,10 @@ def recycle_unused_buckets(self):
self.bucket_can_be_released[bucket_key] = False
def get_fsdp_buffer(self, bucket_id: int, bwd=False) -> DataParallelBuffer:
- """Get the FSDP buffer with the given bucket ID."""
+ """
+ Get the FSDP / DP-Shard buffer with the given bucket ID.
+ If bwd=True, return the FSDP transpose buffer instead.
+ """
param_group = self.buffer.parameter_groups[bucket_id]
if self.buffer.ddp_config.outer_dp_sharding_strategy != "no_shard":
if bwd and param_group.transpose_weight_buffer is not None:
@@ -3726,12 +4027,16 @@ def async_bucket_gather(self, bucket_id, bwd) -> None:
self.bucket_status[bucket_key] = BucketStatus.COMMUNICATING
+ # Retrieve the buffer associated with the DP-Shard PG
+ # that backs the model compute weights.
wbuf = self.get_fsdp_buffer(bucket_id, bwd)
# Lazy release the unused buckets.
self.recycle_unused_buckets()
+
# Allocate an empty bucket to store the module weights.
bucket = wbuf.fetch_bucket(set_param_data=True)
+
# All-gather the module weights in each buffer shard into the allocated bucket.
# Now each rank will have a copy of this FSDP unit module's weights.
param_gather_event = torch.distributed.all_gather_into_tensor(
@@ -3764,19 +4069,38 @@ def gradient_reduce_preprocessing(grad_data, scaling_factor, ddp_config):
Gradient reduce preprocessing for gradient averaging and gradient scaling.
"""
+ # TODO(@cspades): Clean up this logic in conjunction with
+ # gradient reduction arguments: calculate_per_token_loss,
+ # and average_in_collective.
if scaling_factor is None:
+ # No scaling - use SUM reduction.
reduce_op = torch.distributed.ReduceOp.SUM
elif ddp_config.average_in_collective:
+ # Scaling overridden by AVG reduction.
reduce_op = torch.distributed.ReduceOp.AVG
elif ddp_config.gradient_reduce_div_fusion and grad_data.dtype != torch.bfloat16:
+ # Fused SUM reduction.
reduce_op = torch.distributed._make_nccl_premul_sum(scaling_factor)
else:
+ # Scale gradients with SUM reduction.
grad_data.mul_(scaling_factor)
reduce_op = torch.distributed.ReduceOp.SUM
return reduce_op
+def _check_nan_in_grad(grad: torch.Tensor):
+ """
+ Check if there are any NaN or Inf in grad.
+ """
+ # Compute gradient norm.
+ grad_norm = torch.linalg.norm(grad)
+ if torch.isnan(grad_norm) or not torch.isfinite(grad_norm):
+ raise ValueError(
+ f"[Megatron-FSDP](check_for_nan_in_grad=True) Detected NaN or Inf in wgrad: {grad}"
+ )
+
+
def check_gpu_memory(threshold=0.9):
"""
Check if the GPU memory is over the threshold.
@@ -3896,24 +4220,33 @@ def override_sharded_param_cpu_function(*args, **kwargs):
def _dtype_size(dtype: torch.dtype) -> int:
"""
- Get the size of the dtype.
+ Get the size of the dtype. Note that many data-types un-common to ML
+ or not supported by NCCL communication (e.g. CFloat) are listed here
+ for mixed-precision coverage and to avoid allocating a dummy Tensor.
+
Args:
dtype (torch.dtype): The dtype to get the size of.
Returns:
int: The size of the dtype.
"""
- if dtype == torch.float16 or dtype == torch.bfloat16:
+ if dtype == torch.float16 or dtype == torch.bfloat16 or dtype == torch.int16:
return 2
- elif dtype == torch.float32 or dtype == torch.int32:
+ elif dtype == torch.float32 or dtype == torch.int32 or torch.complex32:
return 4
- elif dtype == torch.int64:
+ elif dtype == torch.float64 or dtype == torch.int64 or torch.complex64:
return 8
- elif dtype == torch.uint8:
+ elif dtype == torch.uint8 or dtype == torch.int8:
return 1
elif dtype == "float8":
return 1
else:
- raise ValueError(f"Unsupported dtype: {dtype}")
+ try:
+ # Allocate an empty Tensor on-the-fly to check the size.
+ # Non-ideal fall-back option before sizing the new dtype.
+ # Why does torch.dtype not support this without alloc?
+ return torch.empty((), dtype=dtype).element_size()
+ except:
+ raise ValueError(f"Unsupported dtype: {dtype}")
def to_local_if_dtensor(tensor):
diff --git a/megatron/core/distributed/fsdp/src/megatron_fsdp/uneven_dtensor.py b/megatron/core/distributed/fsdp/src/megatron_fsdp/uneven_dtensor.py
index f0b93056c9c..f18a21df6c1 100644
--- a/megatron/core/distributed/fsdp/src/megatron_fsdp/uneven_dtensor.py
+++ b/megatron/core/distributed/fsdp/src/megatron_fsdp/uneven_dtensor.py
@@ -58,6 +58,7 @@ def _update_offsets_and_cumulative_shape(
# TODO: add documentation for the offset calculation
# Add on the offset of the current mesh dimension
offsets[shard_dim] += offset
+ # Calculate the global shape using the sum of the sharding dim sizes.
cumulative_shape[shard_dim] = sum(s[shard_dim] for s in global_shapes)
# Get the shard placements order.
@@ -174,6 +175,11 @@ def validate_uneven_dtensor(dtensor: DTensor) -> None:
)
# Check that all boundaries (start and end) are touched.
+ # Skip under fake process group — all_reduce is a no-op so only rank 0's
+ # boundaries are visible, which makes the end-boundary check always fail.
+ if torch.distributed.is_initialized() and torch.distributed.get_backend() == 'fake':
+ return
+
boundary_checks = torch.tensor(
[
[offset == 0, offset + size == dtensor.shape[dim]]
@@ -240,7 +246,9 @@ def preprocess_state_dict_for_uneven_dtensor(state_dict: dict) -> dict:
visit_dtensor = filter_unflattened_state_dict(
state_dict, visit_condition=lambda x: isinstance(x, DTensor)
)
- for key_chain in visit_dtensor:
+ # Sort the keys, since some state dictionaries are mocked
+ # and extended to include empty global keys.
+ for key_chain in sorted(visit_dtensor):
# Get the DTensor at the key chain
dtensor = get_unflattened_state_dict(state_dict, key_chain)
update_uneven_dtensor_chunk_metadata(dtensor)
diff --git a/megatron/core/distributed/fsdp/src/megatron_fsdp/utils.py b/megatron/core/distributed/fsdp/src/megatron_fsdp/utils.py
index d5fbc91fcf8..b961a449d3e 100644
--- a/megatron/core/distributed/fsdp/src/megatron_fsdp/utils.py
+++ b/megatron/core/distributed/fsdp/src/megatron_fsdp/utils.py
@@ -434,6 +434,34 @@ def get_cuda_rng_tracker(
return _CUDA_RNG_STATE_TRACKER
+def safe_get_rank() -> int:
+ """Safely get the rank of the current process.
+
+ Returns the rank from torch.distributed if initialized, otherwise falls back
+ to the RANK environment variable, defaulting to 0.
+
+ Returns:
+ int: The rank of the current process.
+ """
+ if torch.distributed.is_initialized():
+ return torch.distributed.get_rank()
+
+ # If torch.distributed is not initialized, try to read environment variables.
+ try:
+ return int(os.environ.get("RANK", 0))
+ except (ValueError, TypeError):
+ # Return rank 0 regardless of the actual rank.
+ return 0
+
+
+def log_single_rank(logger_: logging.Logger, level: int, msg: str, *args, rank: int = 0, **kwargs):
+ """Log on a single rank."""
+ if safe_get_rank() == rank:
+ logger_.log(level, msg, *args, **kwargs)
+
+
+# TODO(@cspades): Migrate this to a new module: fsdp_dist_index.py.
+# Needs more visibility and is easily refactored / standalone.
class FSDPDistributedIndex:
"""
Class containing references to the process groups utilized by Megatron-FSDP.
@@ -450,6 +478,7 @@ def __init__(
dp_outer_dim: Optional[str] = None,
tp_dim: Optional[str] = None,
hybrid_fsdp_group: Optional[torch.distributed.ProcessGroup] = None,
+ hybrid_fsdp_expt_group: Optional[torch.distributed.ProcessGroup] = None,
hsdp_outer_dp_shard: bool = False,
expt_device_mesh: Optional[DeviceMesh] = None,
):
@@ -464,6 +493,9 @@ def __init__(
hybrid_fsdp_group (Optional[torch.distributed.ProcessGroup]): The
process group for hybrid FSDP communication, which is the flattened
combination of the dp_outer and dp_shard process groups.
+ hybrid_fsdp_expt_group (Optional[torch.distributed.ProcessGroup]): The
+ process group for hybrid FSDP expert communication, which is the flattened
+ combination of the expert dp_outer and expert dp_shard process groups.
hsdp_outer_dp_shard (bool): Whether to have outer DP group sharding
in hybrid FSDP. Specifying outer sharding will lift the bucket sharding
coordinate system to flattened ranks of (dp_shard, dp_outer) instead of
@@ -509,6 +541,7 @@ def __init__(
# Save a reference to the overall HSDP process group, which is the flattened
# combination of the outer-FSDP and FSDP process groups.
self.hybrid_fsdp_group = hybrid_fsdp_group
+ self.hybrid_fsdp_expt_group = hybrid_fsdp_expt_group
# Retrieve the expert parallel process groups from the DeviceMesh.
self.expt_fsdp_group = (
@@ -518,6 +551,13 @@ def __init__(
else None
)
+ self.expt_outer_fsdp_group = (
+ self.expt_device_mesh[self.dp_outer_dim].get_group()
+ if self.expt_device_mesh is not None
+ and contains_submesh(self.expt_device_mesh, self.dp_outer_dim)
+ else None
+ )
+
"""
Megatron-FSDP is responsible for storing all required DeviceMesh
as per best practices recommended by the DeviceMesh API.
@@ -558,6 +598,8 @@ def register_submesh(device_mesh, submesh, is_expert_parallel):
register_submesh(self.expt_device_mesh, tp_submesh, True)
register_submesh(self.expt_device_mesh, fsdp_tp_submesh, True)
register_submesh(self.expt_device_mesh, fsdp_submesh, True)
+ register_submesh(self.expt_device_mesh, hsdp_submesh, True)
+ register_submesh(self.expt_device_mesh, hsdp_tp_submesh, True)
# Validate FSDP arguments.
if self.fsdp_group is None:
@@ -629,6 +671,8 @@ def get_submesh(
def get_dp_group(self, is_expert_parallel: bool = False) -> ProcessGroup:
"""Get the data parallel process group."""
if is_expert_parallel:
+ if self.use_hybrid_fsdp:
+ return self.hybrid_fsdp_expt_group
return self.expt_fsdp_group
if self.use_hybrid_fsdp:
return self.hybrid_fsdp_group
@@ -644,10 +688,12 @@ def get_fsdp_group(
return self.fsdp_group_ag
return self.fsdp_group
- def get_outer_fsdp_group(self) -> ProcessGroup:
+ def get_outer_fsdp_group(self, is_expert_parallel: bool = False) -> ProcessGroup:
"""Get the outer-FSDP process group."""
if not self.use_hybrid_fsdp:
return None
+ if is_expert_parallel:
+ return self.expt_outer_fsdp_group
return self.outer_fsdp_group
def get_root_mesh(self, is_expert_parallel: bool = False) -> DeviceMesh:
@@ -659,7 +705,7 @@ def get_root_mesh(self, is_expert_parallel: bool = False) -> DeviceMesh:
return self.expt_device_mesh
return self.device_mesh
- def get_logical_hybrid_fsdp_rank(self):
+ def get_logical_hybrid_fsdp_rank(self, is_expert_parallel: bool = False):
"""
Returns the logical rank of the current process within the full-shard hybrid FSDP group.
@@ -679,20 +725,28 @@ def get_logical_hybrid_fsdp_rank(self):
self.hsdp_outer_dp_shard
), "get_logical_hybrid_fsdp_rank is only valid when full-shard hybrid FSDP is enabled."
- if not hasattr(self, "_hybrid_fsdp_group_ranks"):
- dp_world_size = self.get_dp_group().size()
+ _hybrid_fsdp_group_name = (
+ "_hybrid_fsdp_group_ranks"
+ if not is_expert_parallel
+ else "_hybrid_fsdp_expt_group_ranks"
+ )
+
+ if not hasattr(self, _hybrid_fsdp_group_name):
+ dp_world_size = self.get_dp_group(is_expert_parallel).size()
# Reorder the flat ranks: (outer_dp, inner_dp) -> (inner_dp, outer_dp)
mesh = einops.rearrange(
torch.arange(dp_world_size),
"(outer_dp inner_dp) -> (inner_dp outer_dp)",
- outer_dp=self.outer_fsdp_group.size(),
- inner_dp=self.fsdp_group.size(),
+ outer_dp=self.get_outer_fsdp_group(is_expert_parallel).size(),
+ inner_dp=self.get_fsdp_group(is_expert_parallel).size(),
)
- self._hybrid_fsdp_group_ranks = mesh.tolist()
+ setattr(self, _hybrid_fsdp_group_name, mesh.tolist())
# Find the index for the current rank in the hybrid group
- return self._hybrid_fsdp_group_ranks.index(self.hybrid_fsdp_group.rank())
+ return getattr(self, _hybrid_fsdp_group_name).index(
+ self.get_dp_group(is_expert_parallel).rank()
+ )
class GlobalMemoryBuffer:
diff --git a/megatron/core/distributed/fsdp/src/pyproject.toml b/megatron/core/distributed/fsdp/src/pyproject.toml
index abeff53322e..783030cc809 100644
--- a/megatron/core/distributed/fsdp/src/pyproject.toml
+++ b/megatron/core/distributed/fsdp/src/pyproject.toml
@@ -30,7 +30,6 @@ keywords = [
"gpu",
"language",
"learning",
- "learning",
"machine",
"nvidia",
"pytorch",
@@ -60,7 +59,7 @@ classifiers = [
[project.urls]
Download = "https://github.com/NVIDIA/Megatron-LM/releases"
-Homepage = "https://github.com/NVIDIA/Megatron-LM/megatron/core"
+Homepage = "https://github.com/NVIDIA/Megatron-LM"
[tool.uv]
diff --git a/megatron/core/distributed/param_and_grad_buffer.py b/megatron/core/distributed/param_and_grad_buffer.py
index 162e74ca1b6..f21fa0ef0d8 100644
--- a/megatron/core/distributed/param_and_grad_buffer.py
+++ b/megatron/core/distributed/param_and_grad_buffer.py
@@ -1,5 +1,6 @@
# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
+import fnmatch
import functools
import logging
import math
@@ -7,9 +8,10 @@
from contextlib import nullcontext
from enum import Enum
from functools import partial
-from typing import Dict, List, Optional
+from typing import Dict, List, Optional, Tuple
import torch
+from torch._utils import _flatten_dense_tensors, _unflatten_dense_tensors
from torch.distributed import _coalescing_manager
import megatron.core.nccl_allocator as nccl_allocator
@@ -79,6 +81,10 @@ class _ParamAndGradBucket:
communication. Its application is twofold: it facilitates the averaging of gradients
and the scaling of gradients in the context of the Mixture of Experts (MoE) model.
bucket_id: Index of bucket in buffer.
+ param_index_map: Mapping from param to (start, end, bucket_id) in the global buffer.
+ Used to derive bucket-local offsets for param_to_index.
+ params_with_extra_main_grads: List of parameters in this bucket that require a
+ separate higher-precision main_grad tensor for local gradient accumulation.
"""
def __init__(
@@ -90,6 +96,8 @@ def __init__(
numel_unpadded: int,
gradient_scaling_factor: float,
bucket_id: int,
+ param_index_map: Dict[torch.nn.Parameter, tuple],
+ params_with_extra_main_grads: List[torch.nn.Parameter],
):
self.params_list = params
self.params = set(params)
@@ -103,11 +111,48 @@ def __init__(
self.numel_unpadded = numel_unpadded
self.gradient_scaling_factor = gradient_scaling_factor
self.bucket_id = bucket_id
+ # Derive bucket-local param offsets from the global param_index_map.
self.param_to_index = {}
- offset = 0
for param in params:
- self.param_to_index[param] = (offset, offset + param.numel())
- offset += param.numel()
+ global_start, global_end, _ = param_index_map[param]
+ self.param_to_index[param] = (global_start - offset, global_end - offset)
+ self.params_with_extra_main_grads = params_with_extra_main_grads
+
+ # Layer-wise optimizer attributes for async param gather.
+ self.layerwise_params_list = None
+ self.layerwise_param_flat_sizes = None
+ self.layerwise_gather_list = None
+ self._layerwise_src_buffer = None
+
+ def set_layerwise_params_list(self, layerwise_params_list: List[List[torch.nn.Parameter]]):
+ """Set per-rank parameter lists for layer-wise async all-gather.
+
+ Args:
+ layerwise_params_list: List of param lists, one per rank in the DP group.
+ Each inner list contains the parameters owned by that rank's
+ layer-wise optimizer that also belong to this bucket.
+ """
+ self.layerwise_params_list = layerwise_params_list
+ self.layerwise_param_flat_sizes = [
+ sum([p.numel() for p in param_list]) for param_list in layerwise_params_list
+ ]
+
+
+class _LayerwiseAllGatherHandle:
+ """Handle wrapping multiple async all-gather work objects.
+
+ NCCL guarantees in-order completion on the same communicator, so waiting
+ on only the last handle is sufficient.
+ """
+
+ def __init__(self, handles):
+ self.handles = handles
+
+ def wait(self):
+ """Wait on the last handle and clear all handles."""
+ if self.handles:
+ self.handles[-1].wait()
+ self.handles = None
class _ParamAndGradBucketGroup:
@@ -135,11 +180,13 @@ def __init__(
self.buckets = buckets
self.ddp_config = ddp_config
- if self.ddp_config.use_distributed_optimizer:
+ # overlap_param_gather covers the layer-wise optimizer case, which sets
+ # overlap_param_gather=True without use_distributed_optimizer.
+ if self.ddp_config.use_distributed_optimizer or self.ddp_config.overlap_param_gather:
self.intra_distributed_optimizer_instance_group = collective_group
self.intra_distributed_optimizer_instance_size = collective_group_size
self.intra_distributed_optimizer_instance_rank = collective_group.rank()
- else:
+ if not self.ddp_config.use_distributed_optimizer:
self.data_parallel_group = collective_group
# State for bookkeeping: params is the set of parameters this bucket group is
@@ -259,7 +306,9 @@ def start_param_sync(self, force_sync: bool = False):
force_sync (bool, optional): force synchronous collective regardless of
other settings if true.
"""
- assert self.ddp_config.use_distributed_optimizer
+ # overlap_param_gather covers the layer-wise optimizer case, which sets
+ # overlap_param_gather=True without use_distributed_optimizer.
+ assert self.ddp_config.use_distributed_optimizer or self.ddp_config.overlap_param_gather
if force_sync:
if self.param_gather_handle is not None:
@@ -270,33 +319,114 @@ def start_param_sync(self, force_sync: bool = False):
assert self.param_gather_handle is None
async_op = self.ddp_config.overlap_param_gather and not force_sync
- # Coalesce communication kernels across buckets in the bucket group.
- with _coalescing_manager(
- self.intra_distributed_optimizer_instance_group, async_ops=async_op
- ) as cm:
- for idx, bucket in enumerate(self.buckets):
- if self.cached_param_buffer_shard_list[idx] is None:
- self.cached_param_buffer_shard_list[idx] = shard_buffer(
- bucket.param_data, self.intra_distributed_optimizer_instance_size
+
+ if not self.ddp_config.use_distributed_optimizer:
+ # Layer-wise optimizer path: use all_gather for variable-size
+ # param gather.
+ #
+ # Each rank may own a different number of params per bucket, so
+ # layerwise_param_flat_sizes can vary across ranks. PyTorch's NCCL
+ # backend handles uneven tensor sizes in torch.distributed.all_gather
+ # (falling back to grouped send/recv internally when sizes differ),
+ # so no manual padding is needed.
+ dp_size = self.intra_distributed_optimizer_instance_size
+ local_rank = self.intra_distributed_optimizer_instance_rank
+ group = self.intra_distributed_optimizer_instance_group
+ layerwise_work_handles = []
+ for bucket in self.buckets:
+ # Use param dtype (e.g., bf16), NOT grad dtype (which may be
+ # fp32 when grad_reduce_in_fp32 is enabled).
+ param_dtype = bucket.params_list[0].dtype
+
+ if max(bucket.layerwise_param_flat_sizes) == 0:
+ # All ranks have empty params for this bucket — skip.
+ bucket.layerwise_gather_list = None
+ continue
+
+ # Flatten local params. Detach from the autograd graph because
+ # start_param_sync can be called during the forward pass (where
+ # autograd is active) and all_gather will write into gather_list
+ # entries in-place.
+ local_size = bucket.layerwise_param_flat_sizes[local_rank]
+ if local_size > 0:
+ flat_local_params = _flatten_dense_tensors(
+ bucket.layerwise_params_list[local_rank]
+ ).detach()
+ else:
+ flat_local_params = torch.empty(
+ 0, device=bucket.grad_data.device, dtype=param_dtype
)
- local_data_view = self.cached_param_buffer_shard_list[idx][
- self.intra_distributed_optimizer_instance_rank
- ]
- dist_all_gather_func(
- bucket.param_data,
- local_data_view,
- group=self.intra_distributed_optimizer_instance_group,
- async_op=async_op,
+ # Keep flat_local_params alive until the async operation completes.
+ bucket._layerwise_src_buffer = flat_local_params
+
+ # Allocate per-rank receive buffers with actual sizes (no padding).
+ # Reuse flat_local_params for local_rank's slot to avoid an extra allocation.
+ gather_list = []
+ for i in range(dp_size):
+ if i == local_rank:
+ gather_list.append(flat_local_params)
+ else:
+ gather_list.append(
+ torch.empty(
+ bucket.layerwise_param_flat_sizes[i],
+ device=flat_local_params.device,
+ dtype=flat_local_params.dtype,
+ )
+ )
+ bucket.layerwise_gather_list = gather_list
+
+ work = torch.distributed.all_gather(
+ gather_list, flat_local_params, group=group, async_op=async_op
)
- if async_op:
- self.param_gather_handle = cm
+ if async_op and work is not None:
+ layerwise_work_handles.append(work)
+
+ if async_op:
+ self.param_gather_handle = _LayerwiseAllGatherHandle(layerwise_work_handles)
+ else:
+ # Synchronous: unflatten and copy gathered params immediately.
+ for bucket in self.buckets:
+ if bucket.layerwise_gather_list is None:
+ continue
+ for idx, params in enumerate(bucket.layerwise_params_list):
+ if len(params) == 0 or idx == local_rank:
+ continue
+ updated_params = _unflatten_dense_tensors(
+ bucket.layerwise_gather_list[idx], params
+ )
+ for updated_p, model_p in zip(updated_params, params):
+ model_p.data.copy_(updated_p)
+ bucket.layerwise_gather_list = None
+ bucket._layerwise_src_buffer = None
+ self.param_gather_handle = None
else:
- # When using `_coalescing_manager`, even if a synchronous op (async_op=False) is used,
- # `cm` is not None, which is different from when `_coalescing_manager` is not used in
- # which case the torch.distributed._all_gather_base() will return None. In order to
- # maintain consistency with prior code, we need to manually set communication handle to
- # None.
- self.param_gather_handle = None
+ # Standard distributed optimizer path: use _coalescing_manager.
+ # all_gather_into_tensor writes directly into a contiguous output buffer and
+ # does not need a copy-back step, so coalescing works correctly.
+ with _coalescing_manager(
+ self.intra_distributed_optimizer_instance_group, async_ops=async_op
+ ) as cm:
+ for idx, bucket in enumerate(self.buckets):
+ if self.cached_param_buffer_shard_list[idx] is None:
+ self.cached_param_buffer_shard_list[idx] = shard_buffer(
+ bucket.param_data, self.intra_distributed_optimizer_instance_size
+ )
+ local_data_view = self.cached_param_buffer_shard_list[idx][
+ self.intra_distributed_optimizer_instance_rank
+ ]
+ dist_all_gather_func(
+ bucket.param_data,
+ local_data_view,
+ group=self.intra_distributed_optimizer_instance_group,
+ async_op=async_op,
+ )
+ if async_op:
+ self.param_gather_handle = cm
+ else:
+ # When using `_coalescing_manager`, even if a synchronous op
+ # (async_op=False) is used, `cm` is not None. Manually set to None for
+ # consistency with prior code.
+ self.param_gather_handle = None
self.param_gather_dispatched = True
def finish_param_sync(self, skip_next_bucket_dispatch: bool = False):
@@ -314,7 +444,6 @@ def finish_param_sync(self, skip_next_bucket_dispatch: bool = False):
skip_next_bucket_dispatch (bool, optional): if true, dispatch next
bucket's communication if available.
"""
- assert self.ddp_config.use_distributed_optimizer
assert self.ddp_config.overlap_param_gather
# If current bucket's param AG has not been dispatched, dispatch it now (e.g., first
@@ -342,16 +471,43 @@ def finish_param_sync(self, skip_next_bucket_dispatch: bool = False):
# after the param all-gather.
if self.ddp_config.reuse_grad_buf_for_mxfp8_param_ag:
for bucket in self.buckets:
+ is_bf16_weight_bucket = False
for param in bucket.params:
+ # Skip copying since bf16 weights in the mxfp8 model
+ # are already mapped to param.data.
+ if not is_float8tensor(param):
+ is_bf16_weight_bucket = True
+ break
param_start, param_end = bucket.param_to_index[param]
param_slice = bucket.param_data.view(-1)[param_start:param_end]
param.data.copy_(param_slice.view(param.data.shape))
+ if is_bf16_weight_bucket:
+ continue
# All-gathered params are not needed after being copied to param.data.
# Zero out the param buffer (shared with grad buffer) for gradient accumulation.
# We cannot zero out the entire grad buffer because one grad buffer may
# correspond to multiple param buffers. If we zero out the entire grad buffer,
# it would clear the data of those param buffers that have not yet completed AG.
bucket.param_data.zero_()
+ elif not self.ddp_config.use_distributed_optimizer:
+ for bucket in self.buckets:
+ if bucket.layerwise_gather_list is None:
+ continue
+ # Unflatten and copy gathered params for each rank.
+ for idx, params in enumerate(bucket.layerwise_params_list):
+ # Skip local params and empty tensors.
+ if (
+ len(params) == 0
+ or idx == self.intra_distributed_optimizer_instance_rank
+ ):
+ continue
+ updated_params = _unflatten_dense_tensors(
+ bucket.layerwise_gather_list[idx], params
+ )
+ for updated_p, model_p in zip(updated_params, params):
+ model_p.data.copy_(updated_p)
+ bucket.layerwise_gather_list = None
+ bucket._layerwise_src_buffer = None
else:
fp8_params = []
for bucket in self.buckets:
@@ -379,6 +535,14 @@ def start_grad_sync(self, force_all_reduce: Optional[bool] = False):
self.grad_reduce_handle is None
), "Should not have multiple communication calls outstanding at once"
+ # Copy accumulated .main_grad into communication buffer before collective if
+ # .main_grad is not in .grad_data already (e.g., because we want to do local
+ # gradient accumulation in a higher precision).
+ for bucket in self.buckets:
+ for param in bucket.params_with_extra_main_grads:
+ if getattr(param, 'main_grad_copy_in_grad_buffer', None) is not None:
+ param.main_grad_copy_in_grad_buffer.copy_(param.main_grad)
+
if self.ddp_config.check_for_nan_in_grad or self.ddp_config.check_for_large_grads:
self.check_grads(
check_for_nan_or_inf=self.ddp_config.check_for_nan_in_grad,
@@ -416,10 +580,10 @@ def start_grad_sync(self, force_all_reduce: Optional[bool] = False):
# need to overlap communication.
stream_context = torch.cuda.stream(self.communication_stream)
- # The RS/AR communication stream needs to wait for the default stream
+ # The RS/AR communication stream needs to wait for the current stream
# to complete its gradient computation before launching the next
# gradient reduction collective.
- self.communication_stream.wait_stream(torch.cuda.default_stream())
+ self.communication_stream.wait_stream(torch.cuda.current_stream())
else:
stream_context = nullcontext()
@@ -517,6 +681,7 @@ def finish_grad_sync(self, force_all_reduce: Optional[bool] = False):
# If overlap_grad_reduce is False, start (and finish) synchronous communication call here.
if not self.ddp_config.overlap_grad_reduce:
self.start_grad_sync(force_all_reduce=force_all_reduce)
+ self._copy_back_extra_main_grads()
return
# If first batch, start asynchronous communication here. register_grad_ready() launches
# asynchronous communication only once self.golden_per_param_grad_ready_counts is
@@ -526,7 +691,8 @@ def finish_grad_sync(self, force_all_reduce: Optional[bool] = False):
# When using multiple DistOpt instances, we don't need to sync here as we launch
# communications on a separate communication stream.
if self.ddp_config.num_distributed_optimizer_instances > 1:
- torch.cuda.default_stream().wait_stream(self.communication_stream)
+ torch.cuda.current_stream().wait_stream(self.communication_stream)
+ self._copy_back_extra_main_grads()
return
assert self.grad_reduce_handle is not None, (
f"Communication call has not been issued for this bucket "
@@ -535,6 +701,37 @@ def finish_grad_sync(self, force_all_reduce: Optional[bool] = False):
)
self.grad_reduce_handle.wait()
self.grad_reduce_handle = None
+ self._copy_back_extra_main_grads()
+
+ def free_overlap_buffers(self):
+ """Free GPU buffers used by overlap param gather.
+
+ Waits on any pending param all-gather handle, then releases the
+ per-bucket temporary buffers so that the CUDA memory allocator can
+ reclaim them. Called before async checkpoint saves to avoid OOM in
+ the persistent checkpoint worker process.
+ """
+ if self.param_gather_handle is not None:
+ self.param_gather_handle.wait()
+ self.param_gather_handle = None
+ for bucket in self.buckets:
+ bucket.layerwise_gather_list = None
+ bucket._layerwise_src_buffer = None
+
+ def _copy_back_extra_main_grads(self):
+ """
+ Copy reduced gradients from the communication buffer back to .main_grad for
+ params that have a separate higher-precision .main_grad tensor.
+
+ This is needed because the optimizer reads from .main_grad to get the reduced
+ gradients, but for params with extra main_grads, .main_grad points to the local
+ FP32 accumulation tensor rather than the communication buffer where the reduced
+ gradients are stored.
+ """
+ for bucket in self.buckets:
+ for param in bucket.params_with_extra_main_grads:
+ if getattr(param, 'main_grad_copy_in_grad_buffer', None) is not None:
+ param.main_grad.copy_(param.main_grad_copy_in_grad_buffer)
def register_grad_ready(
self, param: torch.nn.Parameter, force_all_reduce: Optional[bool] = False
@@ -588,7 +785,7 @@ def __init__(
ddp_config: DistributedDataParallelConfig,
param_dtype: torch.dtype,
grad_dtype: torch.dtype,
- params: List[torch.nn.Parameter],
+ params_with_names: List[Tuple[torch.nn.Parameter, str]],
data_parallel_group: torch.distributed.ProcessGroup,
bucket_size: int,
param_to_name: Dict[torch.nn.Parameter, str],
@@ -609,12 +806,12 @@ def __init__(
self.tp_group = pg_collection.tp
self.ddp_config = ddp_config
- self.params = params
+ self.params = [param for (param, _) in params_with_names]
self.param_indices = param_indices
# Check that params are unique.
unique_params = set()
- for param in params:
+ for param, _ in params_with_names:
assert param not in unique_params
unique_params.add(param)
del unique_params
@@ -708,7 +905,7 @@ def _does_param_require_new_bucket(param):
and self.ddp_config.use_distributed_optimizer
)
- for param in params[::-1]:
+ for param, _ in params_with_names[::-1]:
# Iterate through parameters in reverse order to roughly follow backprop order.
this_numel = param.data.nelement()
@@ -749,6 +946,8 @@ def _does_param_require_new_bucket(param):
assert self.numel == self.numel_unpadded
self.param_data = None
+ self.grad_data = None
+ self.extra_main_grads = []
if self.nccl_ub:
# If nccl_ub is True, use nccl_allocator to allocate memory for param_data/grad_data.
@@ -776,7 +975,9 @@ def _does_param_require_new_bucket(param):
# For MXFP8 param: Create a shared buffer for param AG and grad RS for memory efficiency
# The buffer is mapped to weight gradients whose dtype is either bf16 or FP32.
# It can be temporarily reused by param AG.
- if self.ddp_config.use_distributed_optimizer and any(is_mxfp8tensor(p) for p in params):
+ if self.ddp_config.use_distributed_optimizer and any(
+ is_mxfp8tensor(p) for p in self.params
+ ):
self.shared_buffer = torch.zeros(
self.numel,
dtype=self.grad_dtype,
@@ -813,13 +1014,14 @@ def _does_param_require_new_bucket(param):
# Finally, map param.data and param.main_grad fields to buffers.
bucket_params = []
+ bucket_params_with_extra_main_grads = []
bucket_start_index = 0
cur_bucket_id = 0
- for param in params[::-1]:
+ for param, param_name in params_with_names[::-1]:
param_start_index, param_end_index, bucket_id = self.param_index_map[param]
- # For MXFP8 param: we only need to map weight gradients to the buffer.
- if not self.ddp_config.reuse_grad_buf_for_mxfp8_param_ag:
- # Assign param.data to appropriate segment of self.param_data.
+ # For MXFP8 param:
+ # we only need to map bf16 weights (layernorm, embedding, etc) to the buffer.
+ if not self.ddp_config.reuse_grad_buf_for_mxfp8_param_ag or not is_mxfp8tensor(param):
if self.param_data is not None:
new_param_data = self._get(
param.data.shape, param_start_index, buffer_type=BufferType.PARAM
@@ -837,6 +1039,30 @@ def _does_param_require_new_bucket(param):
param.main_grad = self._get(
param.data.shape, param_start_index, buffer_type=BufferType.GRAD
)
+
+ # Create FP32 copy of .main_grads if necessary.
+ promote_main_grads_to_higher_precision = False
+ for param_name_pattern in ddp_config.param_name_patterns_for_fp32_local_accumulation:
+ if fnmatch.fnmatch(param_name, param_name_pattern) or param_name_pattern == 'all':
+ log_on_each_pipeline_stage(
+ logger,
+ logging.INFO,
+ (
+ f"Matched {param_name} with '{param_name_pattern}'; promoting "
+ f"main_grad.type from {param.main_grad.dtype} to torch.float32!"
+ ),
+ tp_group=self.tp_group,
+ dp_cp_group=self.dp_cp_group,
+ )
+ promote_main_grads_to_higher_precision = True
+ break
+ if promote_main_grads_to_higher_precision:
+ param.main_grad_copy_in_grad_buffer = (
+ param.main_grad
+ ) # Slice into .grad_data buffer.
+ param.main_grad = torch.empty_like(param.main_grad, dtype=torch.float32)
+ self.extra_main_grads.append(param.main_grad)
+
if bucket_id != cur_bucket_id:
bucket_end_index = _pad_end_of_bucket_if_needed(param_start_index)
self.buckets.append(
@@ -846,14 +1072,19 @@ def _does_param_require_new_bucket(param):
end_index=bucket_end_index,
numel_unpadded=per_bucket_numel_unpadded[cur_bucket_id],
bucket_id=cur_bucket_id,
+ bucket_params_with_extra_main_grads=bucket_params_with_extra_main_grads,
)
)
bucket_start_index = bucket_end_index
bucket_params = []
+ bucket_params_with_extra_main_grads = []
assert cur_bucket_id + 1 == len(self.buckets)
assert bucket_id == cur_bucket_id + 1
cur_bucket_id = bucket_id
+
bucket_params.append(param)
+ if promote_main_grads_to_higher_precision:
+ bucket_params_with_extra_main_grads.append(param)
# Synchronize streams to ensure all AyncMemCpys are done.
# The model is initialized in a separate CUDA stream,
@@ -873,6 +1104,7 @@ def _does_param_require_new_bucket(param):
end_index=bucket_end_index,
numel_unpadded=per_bucket_numel_unpadded[cur_bucket_id],
bucket_id=cur_bucket_id,
+ bucket_params_with_extra_main_grads=bucket_params_with_extra_main_grads,
)
)
@@ -883,14 +1115,15 @@ def _does_param_require_new_bucket(param):
)
for index, bucket in enumerate(self.buckets):
numel = 0
- for param in bucket.params:
+ for param in bucket.params_list:
numel += param.data.nelement()
log_strs.append(
f"Params for bucket {index + 1} ({numel} elements, "
- f"{bucket.grad_data.nelement()} padded size):"
+ f"{bucket.grad_data.nelement()} padded size, "
+ f"{len(bucket.params_with_extra_main_grads)} param(s) with extra main_grads):"
)
- for param in bucket.params:
- log_strs.append(f"\t{param_to_name[param]}")
+ for param in bucket.params_list:
+ log_strs.append(f"\t{param_to_name[param]} ({param.main_grad.dtype=})")
log_on_each_pipeline_stage(
logger,
logging.INFO,
@@ -902,6 +1135,8 @@ def _does_param_require_new_bucket(param):
def scale_gradients(self, scaling_factor: float) -> None:
"""Scale the gradient data by `scaling_factor`."""
self.grad_data *= scaling_factor
+ for grad in self.extra_main_grads:
+ grad *= scaling_factor
def _get(self, shape: torch.Size, start_index: int, buffer_type: BufferType) -> torch.Tensor:
"""
@@ -927,6 +1162,7 @@ def _new_bucket(
end_index: int,
numel_unpadded: int,
bucket_id: int,
+ bucket_params_with_extra_main_grads: List[torch.Tensor],
) -> _ParamAndGradBucket:
"""
Helper function that creates a new bucket. Also updates param->bucket mapping.
@@ -956,6 +1192,8 @@ def _new_bucket(
numel_unpadded=numel_unpadded,
gradient_scaling_factor=self.gradient_scaling_factor,
bucket_id=bucket_id,
+ param_index_map=self.param_index_map,
+ params_with_extra_main_grads=bucket_params_with_extra_main_grads,
)
for bucket_param in bucket_params:
assert bucket_param not in self.param_to_bucket
@@ -968,6 +1206,8 @@ def reset(self):
Zero out the underlying grad_buffer.
"""
self.grad_data.zero_()
+ for grad in self.extra_main_grads:
+ grad.zero_()
def offload_to_cpu(self, move_params: bool = True, move_grads: bool = True) -> None:
"""
diff --git a/megatron/core/enums.py b/megatron/core/enums.py
index 5b60b4746a4..a9c3c191775 100644
--- a/megatron/core/enums.py
+++ b/megatron/core/enums.py
@@ -8,14 +8,6 @@ class ModelType(enum.Enum):
encoder_or_decoder = 1
- @property
- def encoder_and_decoder(self):
- """Deprecated property - use encoder_or_decoder instead."""
- raise ValueError(
- "ModelType.encoder_and_decoder is deprecated. Please use ModelType.encoder_or_decoder "
- "instead."
- )
-
class Fp8Recipe(str, enum.Enum):
"""FP8 recipe names: delayed, tensorwise, mxfp8, blockwise, custom."""
diff --git a/megatron/core/export/trtllm/trtllm_layers.py b/megatron/core/export/trtllm/trtllm_layers.py
index b777fe40809..bc3ba33d92c 100644
--- a/megatron/core/export/trtllm/trtllm_layers.py
+++ b/megatron/core/export/trtllm/trtllm_layers.py
@@ -90,7 +90,7 @@ def rename_input_layer_names_to_trtllm_layer_names(
map the original layer name to equivalent trtllm layer name and add layer number back.
CPU Conversion will pass in model state dict without layer numbers
(i.e decoder.layers.mlp.linear_fc1.weight of shape [num_layers, hidden_dim, 4 * hidden_dim]) .
- GPU conversion will pass model state dict with each layer seperated
+ GPU conversion will pass model state dict with each layer separated
(i.e decoder.layers.2.mlp.linear_fc1.weight of shape [hidden_dim, 4 * hidden_dim]).
Args:
diff --git a/megatron/core/extensions/transformer_engine.py b/megatron/core/extensions/transformer_engine.py
index 1b47e7a7184..6461e4550aa 100755
--- a/megatron/core/extensions/transformer_engine.py
+++ b/megatron/core/extensions/transformer_engine.py
@@ -8,7 +8,7 @@
import pickle
import warnings
from contextlib import nullcontext
-from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Set, Tuple, Type
+from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Set, Tuple, cast, Type
import inspect
@@ -17,6 +17,7 @@
from packaging.version import Version as PkgVersion
from torch import Tensor
from torch.nn.parameter import Parameter
+from typing_extensions import override
from megatron.core.dist_checkpointing.mapping import ShardedStateDict
from megatron.core.dist_checkpointing.utils import replace_prefix_for_sharding
@@ -45,12 +46,14 @@
from megatron.core.tensor_parallel.utils import divide
from megatron.core.transformer.enums import AttnMaskType
from megatron.core.transformer.mlp import MLP
+from megatron.core.transformer.torch_norm import LayerNormInterface
from megatron.core.transformer.transformer_config import TransformerConfig
from megatron.core.transformer.utils import (
ensure_metadata_has_dp_cp_group,
is_layer_window_attention,
make_sharded_tensors_for_checkpoint,
)
+from megatron.core.typed_torch import copy_signature
from megatron.core.utils import (
get_pg_rank,
get_pg_size,
@@ -451,41 +454,235 @@ def __new__(cls, config: TransformerConfig):
TEActivationOp = None
+if HAVE_TE and is_te_min_version("1.13.0"):
+
+ class TEFusedResidualRMSNorm(te.pytorch.RMSNorm):
+ """
+ RMSNorm with fused residual output for Megatron Core.
+
+ Inherits from te.pytorch.RMSNorm to maintain all parameter management,
+ checkpoint compatibility, and Megatron-specific features. Creates a fused
+ implementation using TE's ops API that shares the base class parameters.
+
+ The fused implementation uses:
+ - MakeExtraOutput: Forks the residual connection
+ - RMSNorm: Normalizes the main path
+
+ Forward pass returns: (normalized_output, residual)
+ """
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ # Fused implementation (stored in tuple to avoid submodule registration)
+ self._fused_impl: Optional[Tuple[te.pytorch.ops.Sequential]] = None
+
+ def _make_fused_impl(self) -> te.pytorch.ops.Sequential:
+ """
+ Construct fused ops pipeline that shares parameters with base RMSNorm.
+
+ Creates MakeExtraOutput + RMSNorm ops, where the RMSNorm op shares
+ the weight parameter with self.weight from the base class.
+ """
+
+ fused_impl = te.pytorch.ops.Sequential()
+
+ # Op 1: MakeExtraOutput - forks the residual
+ fused_impl.append(te.pytorch.ops.MakeExtraOutput())
+
+ # Op 2: RMSNorm - shares weight parameter with self
+ kwargs = {
+ "eps": self.eps,
+ "device": "meta", # Already initialized
+ "dtype": self.weight.dtype,
+ "zero_centered_gamma": self.zero_centered_gamma,
+ }
+
+ # Add sm_margin if available (TE 2.5+)
+ if hasattr(self, '_sm_margins'):
+ kwargs["sm_margin"] = self._sm_margins
+
+ rmsnorm_op = te.pytorch.ops.RMSNorm(self.weight.shape, **kwargs)
+
+ rmsnorm_op.weight = self.weight
+
+ fused_impl.append(rmsnorm_op)
+
+ self._register_hooks_on_fused_impl(fused_impl)
+
+ return fused_impl
+
+ def _register_hooks_on_fused_impl(self, fused_impl: torch.nn.Module) -> None:
+
+ forward_pre_hooks = []
+ forward_post_hooks = []
+ backward_pre_hooks = []
+ backward_post_hooks = []
+
+ for submodule in self.modules():
+ for hook_id, hook in submodule._forward_pre_hooks.items():
+ with_kwargs = hook_id in submodule._forward_pre_hooks_with_kwargs
+ forward_pre_hooks.append((submodule, hook, with_kwargs))
+ for hook_id, hook in submodule._forward_hooks.items():
+ with_kwargs = hook_id in submodule._forward_hooks_with_kwargs
+ forward_post_hooks.append((submodule, hook, with_kwargs))
+ for hook in submodule._backward_pre_hooks.values():
+ backward_pre_hooks.append((submodule, hook))
+ for hook in submodule._backward_hooks.values():
+ backward_post_hooks.append((submodule, hook))
+
+ # Pre-forward hooks
+ # Note: DDP pre-forward hooks are safe since they do not
+ # interact with input tensor.
+ if forward_pre_hooks:
+ from megatron.core.distributed import distributed_data_parallel
+
+ if any(
+ inspect.getmodule(hook) != distributed_data_parallel
+ for _, hook, _ in forward_pre_hooks
+ ):
+ warnings.warn(
+ "TEFusedResidualRMSNorm module has a submodule with a pre-forward hook. "
+ "TEFusedResidualRMSNorm module does not expose intermediate tensors, "
+ "so the hook may have incorrect behavior if it attempts to "
+ "access the input tensor."
+ )
+
+ def forward_pre_hook(module, *_) -> None:
+ for submodule, hook, with_kwargs in forward_pre_hooks:
+ if with_kwargs:
+ ret = hook(submodule, (), {})
+ else:
+ ret = hook(submodule, ())
+ if ret is not None:
+ raise RuntimeError(
+ "TEFusedResidualRMSNorm module does not expose "
+ "intermediate tensors, but submodule has "
+ "pre-forward hook that modifies input tensor."
+ )
+
+ fused_impl.register_forward_pre_hook(forward_pre_hook)
+
+ # Post-forward hooks
+ if forward_post_hooks:
+ warnings.warn(
+ "TEFusedResidualRMSNorm module has a submodule with a post-forward hook. "
+ "TEFusedResidualRMSNorm module does not expose intermediate tensors, "
+ "so the hook may have incorrect behavior if it attempts to "
+ "access the input or output tensors."
+ )
+
+ def forward_post_hook(module, *_) -> None:
+ for submodule, hook, with_kwargs in forward_post_hooks:
+ if with_kwargs:
+ ret = hook(submodule, (), {}, None)
+ else:
+ ret = hook(submodule, (), None)
+ if ret is not None:
+ raise RuntimeError(
+ "TEFusedResidualRMSNorm module does not expose "
+ "intermediate tensors, but submodule has "
+ "post-forward hook that modifies output tensor."
+ )
+
+ fused_impl.register_forward_hook(forward_post_hook)
+
+ # Backward hooks
+ if backward_pre_hooks:
+ raise RuntimeError(
+ "TEFusedResidualRMSNorm module does not support "
+ "submodules with pre-backward hooks"
+ )
+ if backward_post_hooks:
+ raise RuntimeError(
+ "TEFusedResidualRMSNorm module does not support "
+ "submodules with post-backward hooks"
+ )
+
+ def forward(self, hidden_states: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
+ """
+ Forward pass with fused residual output.
+
+ Args:
+ hidden_states: Input tensor [s, b, h]
+
+ Returns:
+ Tuple of (normalized_output, residual), both [s, b, h]
+
+ Note:
+ Sequential.forward() automatically returns (output, extra_outputs...)
+ when MakeExtraOutput is present, so we don't need manual unpacking.
+ """
+
+ # Construct fused impl lazily on first forward
+ # (in case parameters are modified after __init__)
+ if self._fused_impl is None:
+ self._fused_impl = (self._make_fused_impl(),)
+
+ # Apply fused implementation
+ # Sequential returns (normalized_output, residual) automatically
+ return self._fused_impl[0](hidden_states)
+
+else:
+ TEFusedResidualRMSNorm = None # type: ignore[assignment, misc]
+
+
class TENorm:
"""A conditional wrapper to initialize an instance of
- Transformer-Engine's `LayerNorm` or `RMSNorm` based on input."""
+ Transformer-Engine's `LayerNorm` or `RMSNorm` based on input.
+
+ Residual fusion is a two-level opt-in mechanism:
+
+ 1. Global capability: config.fused_residual_rmsnorm must be True (enables the feature)
+ 2. Local intent: has_residual=True must be passed at build site (declares this specific
+ norm is followed by a residual connection)
+
+ Fusion only happens when BOTH conditions are met.
+
+ """
# TODO should we ditch normalization config and just use spec to choose LayerNorm vs RMSNorm?
- def __new__(cls, config: TransformerConfig, hidden_size: int, eps: float = 1e-5):
+ def __new__(
+ cls,
+ config: TransformerConfig,
+ hidden_size: int,
+ eps: float = 1e-5,
+ has_residual: bool = False,
+ ):
if not HAVE_TE:
raise ImportError(
"Transformer Engine is not installed. "
"Please install it with `pip install transformer-engine`."
)
+ use_fused_residual = config.fused_residual_rmsnorm and has_residual
+ if use_fused_residual and config.normalization != "RMSNorm":
+ raise ValueError("Fused residual is only supported " "for RMSNorm normalization")
+
if config.normalization == "LayerNorm":
- instance = te.pytorch.LayerNorm(
- hidden_size,
- eps=eps,
- sequence_parallel=config.sequence_parallel,
- zero_centered_gamma=config.layernorm_zero_centered_gamma,
- **_get_extra_te_kwargs(config),
- )
+ norm_module = te.pytorch.LayerNorm
elif config.normalization == "RMSNorm":
assert hasattr(
te.pytorch, "RMSNorm"
), "Transformer-Engine >= v0.11 required to use this feature"
- instance = te.pytorch.RMSNorm(
- hidden_size,
- eps=eps,
- sequence_parallel=config.sequence_parallel,
- zero_centered_gamma=config.layernorm_zero_centered_gamma,
- **_get_extra_te_kwargs(config),
- )
+ if use_fused_residual:
+ assert (
+ TEFusedResidualRMSNorm is not None
+ ), "TEFusedResidualRMSNorm requires Transformer-Engine >= v1.13.0"
+ norm_module = TEFusedResidualRMSNorm
+ else:
+ norm_module = te.pytorch.RMSNorm
else:
- raise Exception("Only LayerNorm and RMSNorm are curently supported")
+ raise Exception("Only LayerNorm and RMSNorm are currently supported")
+
+ instance = norm_module(
+ normalized_shape=hidden_size,
+ eps=eps,
+ sequence_parallel=config.sequence_parallel,
+ zero_centered_gamma=config.layernorm_zero_centered_gamma,
+ **_get_extra_te_kwargs(config),
+ )
- return instance
+ return cast(LayerNormInterface, instance)
class TELinear(te.pytorch.Linear):
@@ -716,6 +913,8 @@ def __init__(
# Reduce the gradient further on the TP group since the weight is
# duplicated across TP ranks
setattr(param, "sequence_parallel", self.config.sequence_parallel)
+ # Mark as NOT tensor parallel since weight is duplicated
+ setattr(param, "tensor_model_parallel", False)
tp_group = get_tensor_model_parallel_group_if_none(tp_group, is_expert=is_expert)
self._tp_group = tp_group
@@ -1011,10 +1210,14 @@ def sharded_state_dict(self, prefix="", sharded_offsets=(), metadata=None):
dp_cp_group=metadata["dp_cp_group"],
)
- def __repr__(self):
+ @override
+ def extra_repr(self) -> str:
+ """Extra context to add to the module's string representation."""
return (
- f"{type(self).__name__}(in_features={self.in_features}, "
- f"out_features={self.out_features}, bias={self.use_bias}, TP={self.tp_size})"
+ f"in_features={self.in_features}, "
+ f"out_features={self.out_features}, "
+ f"bias={self.use_bias}, "
+ f"TP={self.tp_size}"
)
def backward_dw(self):
@@ -1117,10 +1320,14 @@ def sharded_state_dict(self, prefix="", sharded_offsets=(), metadata=None):
dp_cp_group=metadata["dp_cp_group"],
)
- def __repr__(self):
+ @override
+ def extra_repr(self) -> str:
+ """Extra context to add to the module's string representation."""
return (
- f"{type(self).__name__}(in_features={self.in_features}, "
- f"out_features={self.out_features}, bias={self.use_bias}, TP={self.tp_size})"
+ f"in_features={self.in_features}, "
+ f"out_features={self.out_features}, "
+ f"bias={self.use_bias}, "
+ f"TP={self.tp_size}"
)
def backward_dw(self):
@@ -1217,10 +1424,14 @@ def sharded_state_dict(self, prefix="", sharded_offsets=(), metadata=None):
dp_cp_group=metadata["dp_cp_group"],
)
- def __repr__(self):
+ @override
+ def extra_repr(self) -> str:
+ """Extra context to add to the module's string representation."""
return (
- f"{type(self).__name__}(in_features={self.in_features}, "
- f"out_features={self.out_features}, bias={self.use_bias}, TP={self.tp_size})"
+ f"in_features={self.in_features}, "
+ f"out_features={self.out_features}, "
+ f"bias={self.use_bias}, "
+ f"TP={self.tp_size}"
)
def backward_dw(self):
@@ -1397,6 +1608,10 @@ def __init__(
self.kept_packed_seq_params.discard("cu_seqlens_q_padded")
self.kept_packed_seq_params.discard("cu_seqlens_kv_padded")
+ # total_tokens and seq_idx are only for Mamba and should not be forwarded to TE attention.
+ self.kept_packed_seq_params.discard("total_tokens")
+ self.kept_packed_seq_params.discard("seq_idx")
+
if config.qk_clip or config.log_max_attention_logit:
# qk-clip is only supported in TE 2.9.0 and later
assert is_te_min_version("2.9.0"), "qk-clip is only supported in TE 2.9.0 and later"
@@ -1612,6 +1827,13 @@ def __init__(
self.explicit_expert_comm = is_expert and (tp_size > 1 or self.expert_parallel)
+ # Save original parallel_mode before clearing it for explicit_expert_comm.
+ # When explicit_expert_comm is True, Megatron handles TP communication externally
+ # and passes parallel_mode=None to TE. This causes TE to set partition_dim=0 on
+ # all weights (its default for non-parallel mode). We need to fix this after init
+ # so that refit/resharding can correctly identify which dimension is TP-partitioned.
+ original_parallel_mode = parallel_mode
+
if self.explicit_expert_comm:
if parallel_mode == "column":
output_size = divide(output_size, tp_size)
@@ -1642,6 +1864,21 @@ def __init__(
for param in self.parameters():
setattr(param, "allreduce", not (is_expert and self.expert_parallel))
+ # Explicitly stamp partition_dim and partition_stride on expert weight
+ # tensors when explicit_expert_comm cleared parallel_mode. TE ≤2.12
+ # set these internally; TE ≥2.13 no longer does (parallel_mode=None
+ # is passed due to explicit_expert_comm). The resharding/refit planner
+ # relies on partition_dim to correctly plan TP gather/scatter operations.
+ # NOTE: we intentionally do NOT stamp tensor_model_parallel here —
+ # doing so would change num-zeros gradient counting.
+ if self.explicit_expert_comm and original_parallel_mode in ("column", "row"):
+ part_dim = 0 if original_parallel_mode == "column" else 1
+ for i in range(num_gemms):
+ weight = getattr(self, f"weight{i}", None)
+ if weight is not None:
+ setattr(weight, "partition_dim", part_dim)
+ setattr(weight, "partition_stride", 1)
+
def merge_extra_states(
self,
state_dict,
@@ -2016,6 +2253,7 @@ def sharded_state_dict(self, prefix="", sharded_offsets=(), metadata=None):
class TEFusedMLP(MLP):
"""MLP wrapper using Transformer Engine's operation-based API."""
+ @copy_signature(MLP.__init__)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
@@ -2201,10 +2439,12 @@ def _register_hooks_on_fused_impl(self, fused_impl: torch.nn.Module) -> None:
backward_pre_hooks = []
backward_post_hooks = []
for submodule in self.modules():
- for hook in submodule._forward_pre_hooks.values():
- forward_pre_hooks.append((submodule, hook))
- for hook in submodule._forward_hooks.values():
- forward_post_hooks.append((submodule, hook))
+ for hook_id, hook in submodule._forward_pre_hooks.items():
+ with_kwargs = hook_id in submodule._forward_pre_hooks_with_kwargs
+ forward_pre_hooks.append((submodule, hook, with_kwargs))
+ for hook_id, hook in submodule._forward_hooks.items():
+ with_kwargs = hook_id in submodule._forward_hooks_with_kwargs
+ forward_post_hooks.append((submodule, hook, with_kwargs))
for hook in submodule._backward_pre_hooks.values():
backward_pre_hooks.append((submodule, hook))
for hook in submodule._backward_hooks.values():
@@ -2218,7 +2458,7 @@ def _register_hooks_on_fused_impl(self, fused_impl: torch.nn.Module) -> None:
if any(
inspect.getmodule(hook) != distributed_data_parallel
- for _, hook in forward_pre_hooks
+ for _, hook, _ in forward_pre_hooks
):
warnings.warn(
"TEFusedMLP module has a submodule with a pre-forward hook. "
@@ -2228,9 +2468,11 @@ def _register_hooks_on_fused_impl(self, fused_impl: torch.nn.Module) -> None:
)
def forward_pre_hook(module, *_) -> None:
- for submodule, hook in forward_pre_hooks:
- # Assume that hook does not interact with input
- ret = hook(submodule, None)
+ for submodule, hook, with_kwargs in forward_pre_hooks:
+ if with_kwargs:
+ ret = hook(submodule, (), {})
+ else:
+ ret = hook(submodule, ())
if ret is not None:
raise RuntimeError(
"TEFusedMLP module does not expose intermediate tensors, but "
@@ -2249,9 +2491,11 @@ def forward_pre_hook(module, *_) -> None:
)
def forward_post_hook(module, *_) -> None:
- for submodule, hook in forward_post_hooks:
- # Assume that hook does not interact with input or output
- ret = hook(submodule, None, None)
+ for submodule, hook, with_kwargs in forward_post_hooks:
+ if with_kwargs:
+ ret = hook(submodule, (), {}, None)
+ else:
+ ret = hook(submodule, (), None)
if ret is not None:
raise RuntimeError(
"TEFusedMLP module does not expose intermediate tensors, but "
@@ -2579,6 +2823,7 @@ def get_cpu_offload_context(
activation_offloading,
weight_offloading,
double_buffering,
+ retain_pinned_cpu_buffers,
):
"""Get CPU offload context and sync function."""
if is_te_min_version("2.5.0"):
@@ -2590,6 +2835,7 @@ def get_cpu_offload_context(
activation_offloading,
weight_offloading,
double_buffering,
+ retain_pinned_cpu_buffers=retain_pinned_cpu_buffers,
)
elif is_te_min_version("1.10.0.dev0"):
context, sync_func = _get_cpu_offload_context(
@@ -2638,11 +2884,26 @@ def fused_apply_rotary_pos_emb_thd(
freqs: torch.Tensor,
cp_size: int = 1,
cp_rank: int = 0,
+ interleaved: bool = False,
) -> torch.Tensor:
"""
Apply rotary positional embedding to input tensor T in `thd` format with CP support.
"""
- if is_te_min_version("1.12.0", check_equality=True):
+ if interleaved:
+ assert is_te_min_version("2.3.0"), "Only TE >= 2.3.0 supports interleaved fused RoPE."
+
+ if is_te_min_version("2.3.0", check_equality=True):
+ return apply_rotary_pos_emb(
+ t,
+ freqs,
+ tensor_format="thd",
+ fused=True,
+ cu_seqlens=cu_seqlens,
+ cp_size=cp_size,
+ cp_rank=cp_rank,
+ interleaved=interleaved,
+ )
+ elif is_te_min_version("1.12.0", check_equality=True):
return apply_rotary_pos_emb(
t,
freqs,
@@ -2690,6 +2951,14 @@ def fused_apply_rotary_pos_emb_thd(
fused_sort_chunks_by_index_with_probs = None
fused_unpermute = None
+try:
+ from transformer_engine.pytorch.permutation import moe_permute_and_pad_with_probs
+
+ fused_permute_and_pad_with_probs = moe_permute_and_pad_with_probs
+
+except ImportError:
+ fused_permute_and_pad_with_probs = None
+
try:
from transformer_engine.pytorch.cross_entropy import parallel_cross_entropy
@@ -2716,7 +2985,13 @@ def te_parallel_cross_entropy(
try:
from transformer_engine.pytorch.cpp_extensions import general_gemm
- from transformer_engine.pytorch.module.base import get_workspace
+
+ try:
+ from transformer_engine.pytorch.module.base import get_workspace
+
+ _get_workspace = get_workspace
+ except ImportError:
+ _get_workspace = None
def te_general_gemm(
A: torch.Tensor,
@@ -2734,9 +3009,7 @@ def te_general_gemm(
Note: not all combinations of these settings are supported. If not supported,
cublaslt will throw an error.
"""
- return general_gemm(
- A,
- B,
+ kwargs = dict(
out_dtype=out_dtype,
quantization_params=None,
gelu=None,
@@ -2752,6 +3025,9 @@ def te_general_gemm(
extra_output=None,
bulk_overlap=False,
)
+ if _get_workspace is not None:
+ kwargs["workspace"] = _get_workspace()
+ return general_gemm(A, B, **kwargs)
except ImportError:
te_general_gemm = None # type: ignore[assignment, misc]
diff --git a/megatron/core/extensions/transformer_engine_spec_provider.py b/megatron/core/extensions/transformer_engine_spec_provider.py
index a071959bfc9..04228e02e88 100644
--- a/megatron/core/extensions/transformer_engine_spec_provider.py
+++ b/megatron/core/extensions/transformer_engine_spec_provider.py
@@ -1,7 +1,9 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+from __future__ import annotations
import warnings
-from typing import Optional, Tuple
+from functools import partial
+from typing import Optional, cast
from megatron.core.extensions.transformer_engine import (
TEActivationOp,
@@ -17,11 +19,20 @@
from megatron.core.fusions.fused_layer_norm import FusedLayerNorm
from megatron.core.models.backends import BackendSpecProvider
from megatron.core.tensor_parallel.layers import ColumnParallelLinear, RowParallelLinear
-from megatron.core.transformer.mlp import MLPSubmodules
-from megatron.core.transformer.moe.experts import GroupedMLP, SequentialMLP, TEGroupedMLP
+from megatron.core.transformer.mlp import MLPSubmodules, TEActivationFunctionBuilder
+from megatron.core.transformer.moe.experts import GroupedMLPSubmodules, SequentialMLP, TEGroupedMLP
+from megatron.core.transformer.moe.moe_layer import ExpertsBuilder
+from megatron.core.transformer.torch_norm import LayerNormBuilder
from megatron.core.utils import get_te_version, is_te_min_version
+class _TENormWithResidual:
+ """Class adapter for TENorm with residual fusion enabled."""
+
+ def __new__(cls, *args, **kwargs):
+ return TENorm(*args, has_residual=True, **kwargs)
+
+
class TESpecProvider(BackendSpecProvider):
"""A protocol for providing the submodules used in Spec building."""
@@ -45,37 +56,33 @@ def column_parallel_layer_norm_linear(self) -> Optional[type]:
"""Which module for sequential layernorm and linear"""
return TELayerNormColumnParallelLinear
- def layer_norm(self, rms_norm: bool = False, for_qk: bool = False) -> type:
+ def layer_norm(
+ self, rms_norm: bool = False, for_qk: bool = False, has_residual: bool = False
+ ) -> LayerNormBuilder:
"""Which module to use for layer norm"""
if for_qk and not is_te_min_version("1.9.0"):
# TENorm significantly harms convergence when used
# for QKLayerNorm if TE Version < 1.9;
# we instead use the Apex implementation.
return FusedLayerNorm
- return TENorm
+ # Keep returning a class so this path stays aligned with build_module's class handling.
+ return _TENormWithResidual if has_residual else TENorm
def core_attention(self) -> type:
"""Which module to use for attention"""
return TEDotProductAttention
- def grouped_mlp_modules(
- self, moe_use_grouped_gemm: bool, moe_use_legacy_grouped_gemm: bool
- ) -> Tuple[type, Optional[MLPSubmodules]]:
+ def grouped_mlp_modules(self, moe_use_grouped_gemm: bool) -> ExpertsBuilder:
"""Which module and submodules to use for grouped mlp"""
- if (
- moe_use_grouped_gemm
- and TEColumnParallelGroupedLinear is not None
- and not moe_use_legacy_grouped_gemm
- ):
- return TEGroupedMLP, MLPSubmodules(
- linear_fc1=TEColumnParallelGroupedLinear, linear_fc2=TERowParallelGroupedLinear
- )
- elif moe_use_grouped_gemm:
- warnings.warn(
- 'The legacy GroupedMLP will be deprecated in Megatron-Core v0.12.0. '
- 'Please update the TransformerEngine to version>=1.7.0 and use TEGroupedMLP.'
+ if moe_use_grouped_gemm and TEColumnParallelGroupedLinear is not None:
+ return partial(
+ TEGroupedMLP,
+ submodules=GroupedMLPSubmodules(
+ linear_fc1=TEColumnParallelGroupedLinear,
+ linear_fc2=TERowParallelGroupedLinear,
+ activation_func=self.activation_func(),
+ ),
)
- return GroupedMLP, None
else:
if not is_te_min_version("1.7.0.dev0"):
warnings.warn(
@@ -83,13 +90,25 @@ def grouped_mlp_modules(
f"but your version is {get_te_version()}. "
"Use local linear implementation instead."
)
- return SequentialMLP, MLPSubmodules(
- linear_fc1=ColumnParallelLinear, linear_fc2=RowParallelLinear
+ return partial(
+ SequentialMLP,
+ submodules=MLPSubmodules(
+ linear_fc1=ColumnParallelLinear,
+ linear_fc2=RowParallelLinear,
+ activation_func=self.activation_func(),
+ ),
)
- return SequentialMLP, MLPSubmodules(
- linear_fc1=TEColumnParallelLinear, linear_fc2=TERowParallelLinear
+ return partial(
+ SequentialMLP,
+ submodules=MLPSubmodules(
+ linear_fc1=TEColumnParallelLinear,
+ linear_fc2=TERowParallelLinear,
+ activation_func=self.activation_func(),
+ ),
)
- def activation_func(self) -> type:
+ def activation_func(self) -> TEActivationFunctionBuilder | None:
"""Which module to use for activation function"""
- return TEActivationOp
+ # transformer_engine.BasicOperation.forward has an overly permissive return type, but by
+ # design these classes always meet the interface.
+ return cast(TEActivationFunctionBuilder, TEActivationOp)
diff --git a/megatron/core/fp4_utils.py b/megatron/core/fp4_utils.py
index d507b6f65f8..cc67855180e 100644
--- a/megatron/core/fp4_utils.py
+++ b/megatron/core/fp4_utils.py
@@ -124,7 +124,9 @@ def get_fp4_recipe(config: TransformerConfig):
if is_te_min_version("2.7.0.dev0"):
if config.fp4_recipe == Fp4Recipe.nvfp4:
try:
- fp4_recipe = transformer_engine.common.recipe.NVFP4BlockScaling()
+ fp4_recipe = transformer_engine.common.recipe.NVFP4BlockScaling(
+ fp8_dpa=config.fp8_dot_product_attention
+ )
except AttributeError:
raise ValueError(
"""NVFP4BlockScaling recipe is not available in this version of
diff --git a/megatron/core/fusions/fused_bias_dropout.py b/megatron/core/fusions/fused_bias_dropout.py
index 336452562b3..2eb4007f75c 100644
--- a/megatron/core/fusions/fused_bias_dropout.py
+++ b/megatron/core/fusions/fused_bias_dropout.py
@@ -26,12 +26,14 @@ def _bias_dropout_add_func(x_with_bias, residual, prob, training):
and (bias is None or not bias.requires_grad)
)
- # If we want to train mixed precision, then the output of this function
- # should be half precision. However, in AMP O1, the input (residual) is
- # in fp32, and it will up-cast the result to fp32, causing pipeline parallel
- # GPU communication to hang. Therefore, we need to cast residual to the same
- # dtype as x.
- residual = residual if residual.dtype == x.dtype else residual.to(x.dtype)
+ # For fp32 residual connections: upcast x (and bias) to residual's dtype so that
+ # the addition and output remain in fp32, preserving numerical precision in the
+ # residual stream across layers. When fp32_residual_connection is enabled,
+ # pipeline parallel communication dtype should be set to fp32 accordingly.
+ if x.dtype != residual.dtype:
+ x = x.to(residual.dtype)
+ if bias is not None:
+ bias = bias.to(residual.dtype)
# The Dropout operation, Residual Addition and the tensor returning can be
# done generically outside the if statement, but that stops fusing of Bias
diff --git a/megatron/core/hyper_comm_grid.py b/megatron/core/hyper_comm_grid.py
index 70630238ee0..8a17a331bf7 100644
--- a/megatron/core/hyper_comm_grid.py
+++ b/megatron/core/hyper_comm_grid.py
@@ -165,6 +165,13 @@ def create_pg(self, dims: Union[str, list[str]], **kwargs: Any) -> dist.ProcessG
self._pgs[unique_group_key] = pg
return pg
+ def destroy(self) -> None:
+ """Destroy all process groups created by this grid."""
+ for pg in self._pgs.values():
+ if pg is not None:
+ dist.destroy_process_group(pg)
+ self._pgs.clear()
+
def get_pg(self, dims: Union[str, list[str]]) -> dist.ProcessGroup:
r"""Get a process group based on a list of dimension names
@@ -255,3 +262,12 @@ def _order_dims(self, dims: Union[str, list[str]]) -> Tuple[list[str], str]:
unique_group_key = "-".join(ordered_dims)
return ordered_dims, unique_group_key
+
+ def is_current_rank_in_grid(self) -> bool:
+ """Check if the current rank belongs to this grid.
+
+ Returns:
+ True if the current rank is within [rank_offset, rank_offset + size).
+ """
+ rank = dist.get_rank()
+ return bool(self.rank_offset <= rank < self.rank_offset + self.size)
diff --git a/megatron/core/inference/batch_dimensions_utils.py b/megatron/core/inference/batch_dimensions_utils.py
index e6ea32a6df8..e27438e63d0 100644
--- a/megatron/core/inference/batch_dimensions_utils.py
+++ b/megatron/core/inference/batch_dimensions_utils.py
@@ -25,7 +25,6 @@ class InferenceBatchDimensions:
token_count : number of total input tokens
prefill_req_count : number of prefill requests
decode_req_count : number of decode requests
- has_explicit_chunked_prefill_req : whether the batch has an explicit chunked prefill request
The batch dimensions are ordered by token_count, then by prefill_req_count,
then by decode_req_count.
@@ -35,7 +34,6 @@ class InferenceBatchDimensions:
token_count: int = 0
prefill_req_count: int = 0
decode_req_count: int = 0
- has_explicit_chunked_prefill_req: bool = False
def __str__(self):
"""
@@ -55,9 +53,6 @@ def is_applicable_for_batch_dim(
for prefill or decode requests. Otherwise, prefill slots
can only be used for prefill requests.
"""
- if real_batch_dim.has_explicit_chunked_prefill_req != self.has_explicit_chunked_prefill_req:
- return False
-
if real_batch_dim.prefill_req_count == 0:
return (
self.token_count >= real_batch_dim.token_count
@@ -78,7 +73,9 @@ def is_applicable_for_batch_dim(
>= real_batch_dim.prefill_req_count + real_batch_dim.decode_req_count
)
- def is_valid(self, max_requests: int, max_sequence_length: int) -> bool:
+ def is_valid(
+ self, max_requests: int, max_sequence_length: int, num_speculative_tokens: int
+ ) -> bool:
"""
Checks if the batch dimension is valid based on resource constraints.
@@ -97,15 +94,17 @@ def is_valid(self, max_requests: int, max_sequence_length: int) -> bool:
return False
# Check if token count is sufficient for requests
- if self.token_count < self.prefill_req_count + self.decode_req_count:
+ if self.token_count < self.prefill_req_count + self.decode_req_count * (
+ num_speculative_tokens + 1
+ ):
return False
# Check if the prefill requests are shorter than the max sequence length
- if self.token_count > self.prefill_req_count * max_sequence_length + self.decode_req_count:
- return False
-
- # Check if there is an invalid chunked prefill request.
- if self.prefill_req_count == 0 and self.has_explicit_chunked_prefill_req:
+ if (
+ self.token_count
+ > self.prefill_req_count * max_sequence_length
+ + self.decode_req_count * (num_speculative_tokens + 1)
+ ):
return False
return True
@@ -115,14 +114,7 @@ def __hash__(self):
Returns a hash of the batch dimension.
In cuda graph quick matching, the batch dimension is used as a key in a dictionary.
"""
- return hash(
- (
- self.token_count,
- self.prefill_req_count,
- self.decode_req_count,
- self.has_explicit_chunked_prefill_req,
- )
- )
+ return hash((self.token_count, self.prefill_req_count, self.decode_req_count))
def __eq__(self, other: "InferenceBatchDimensions") -> bool:
"""
@@ -130,16 +122,10 @@ def __eq__(self, other: "InferenceBatchDimensions") -> bool:
"""
if other is None:
return False
- return (
- self.token_count,
- self.prefill_req_count,
- self.decode_req_count,
- self.has_explicit_chunked_prefill_req,
- ) == (
+ return (self.token_count, self.prefill_req_count, self.decode_req_count) == (
other.token_count,
other.prefill_req_count,
other.decode_req_count,
- other.has_explicit_chunked_prefill_req,
)
@property
@@ -154,6 +140,7 @@ def adjust_batch_dims_for_expert_parallelism(
local_batch_dims,
strict: bool,
decode_only_cuda_graphs: bool,
+ smallest_non_decode_cuda_graph_size: int,
ep_group: Optional[torch.distributed.ProcessGroup] = None,
) -> Optional["InferenceBatchDimensions"]:
"""Adjusted cuda graph batch dimensions for expert parallelism.
@@ -176,13 +163,14 @@ def adjust_batch_dims_for_expert_parallelism(
return local_batch_dims
# all reduce local work across expert model parallel group
- has_explicit_chunked_prefill_req = local_batch_dims.has_explicit_chunked_prefill_req
is_non_decode = local_batch_dims.prefill_req_count > 0
+
sync_tensor = torch.tensor(
[
local_batch_dims.token_count,
int(is_non_decode),
- int(has_explicit_chunked_prefill_req),
+ local_batch_dims.prefill_req_count,
+ local_batch_dims.decode_req_count,
],
dtype=torch.int32,
device=torch.cuda.current_device(),
@@ -192,28 +180,41 @@ def adjust_batch_dims_for_expert_parallelism(
sync_tensor = sync_tensor.cpu()
is_any_ep_rank_in_non_decode = sync_tensor[1].item() == 1
- any_ep_rank_has_explicit_chunked_prefill_req = sync_tensor[2].item() == 1
# We force eager mode for scenarios where some ranks will run with CUDA graphs
- # while others will not. Without this check, the all-to-all communication in the
+ # while others will not. Without this check, communication in the
# expert routing layer would pad up to the maximum capacity only for the ranks that
- # are using CUDA graphs in this step, leading to a NCCL hang.
- # This can happen in the following cases:
- # 1. If we only allow decode CUDA graphs but some ranks are running non-decode batches
- # 2. Some ranks are running explicit chunked prefill requests
- # (graphs are not recorded for batches with explicit chunked prefill requests)
- if (
- decode_only_cuda_graphs and is_any_ep_rank_in_non_decode
- ) or any_ep_rank_has_explicit_chunked_prefill_req:
+ # are using CUDA graphs in this step, leading to a hang.
+ # This can happen if we only allow decode CUDA graphs but some ranks are running
+ # non-decode batches.
+ if is_any_ep_rank_in_non_decode and decode_only_cuda_graphs:
return None # indicate no match, run in eager mode
- assert not has_explicit_chunked_prefill_req
+ # If strict matching is enabled, we sync the request counts across EP ranks
+ # to ensure the graph captures the maximum needed capacity.
+ # TODO(ksanthanam): Add functional test for this scenario
+ adjusted_prefill_req_count = (
+ int(sync_tensor[2].item()) if strict else local_batch_dims.prefill_req_count
+ )
+ adjusted_decode_req_count = (
+ int(sync_tensor[3].item()) if strict else local_batch_dims.decode_req_count
+ )
+ adjusted_token_count = int(sync_tensor[0].item())
+
+ # When any EP rank has prefill requests (non-strict mode), elevate
+ # the token count to be >= the smallest prefill/mixed cuda graph.
+ # This ensures decode-only ranks don't match a fine-grained decode
+ # graph while prefill ranks match a coarser mixed graph, which would
+ # produce inconsistent token counts across EP ranks.
+ if is_any_ep_rank_in_non_decode and not strict:
+ adjusted_token_count = max(adjusted_token_count, smallest_non_decode_cuda_graph_size)
+
adjusted_batch_dim = InferenceBatchDimensions(
- token_count=int(sync_tensor[0].item()),
- prefill_req_count=local_batch_dims.prefill_req_count,
- decode_req_count=local_batch_dims.decode_req_count,
- has_explicit_chunked_prefill_req=False,
+ token_count=adjusted_token_count,
+ prefill_req_count=adjusted_prefill_req_count,
+ decode_req_count=adjusted_decode_req_count,
)
+
return adjusted_batch_dim
@@ -250,6 +251,25 @@ def _calculate_cuda_graph_token_counts(
(tp_size=2, num_cuda_graphs=4, cuda_graph_max_tokens=1000)
[1000, 752, 504, 256]
"""
+ if num_cuda_graphs == -1:
+ # automatically determine the number of CUDA graphs to
+ # capture based on the `max_requests` value
+ cuda_graph_token_counts = (
+ [1, 2, 4] + list(range(8, 256, 8)) + list(range(256, cuda_graph_max_tokens + 1, 16))
+ )
+ # Align each entry to TP size
+ cuda_graph_token_counts = list(
+ dict.fromkeys(math.ceil(s / tp_size) * tp_size for s in cuda_graph_token_counts)
+ )
+ # Clamp to max tokens
+ cuda_graph_token_counts = [
+ s for s in cuda_graph_token_counts if s <= cuda_graph_max_tokens
+ ]
+ if not cuda_graph_token_counts or cuda_graph_token_counts[-1] != cuda_graph_max_tokens:
+ cuda_graph_token_counts.append(cuda_graph_max_tokens)
+ cuda_graph_token_counts.reverse()
+ return cuda_graph_token_counts
+
assert num_cuda_graphs >= 1, f"num_cuda_graphs must be >= 1, got {num_cuda_graphs}"
assert (
cuda_graph_max_tokens > 0
@@ -287,11 +307,12 @@ def generate_cuda_graph_batch_dimensions_list(
tp_size: int,
num_cuda_graphs: Optional[int],
cuda_graph_max_tokens: int,
- cuda_graph_mixed_prefill_count: Optional[int],
+ cuda_graph_mixed_prefill_request_count: Optional[int],
max_requests: int,
max_tokens: int,
max_sequence_length: int,
use_cuda_graphs_for_non_decode_steps: bool,
+ num_speculative_tokens: int = 0,
) -> Tuple[List[InferenceBatchDimensions], Optional[List[int]]]:
"""
Generate CUDA graph batch dimensions.
@@ -323,11 +344,12 @@ def generate_cuda_graph_batch_dimensions_list(
tp_size: Tensor parallel size
num_cuda_graphs: Number of CUDA graphs to generate
cuda_graph_max_tokens: Maximum tokens for CUDA graphs
- cuda_graph_mixed_prefill_count: Number of mixed prefill requests for CUDA graphs
+ cuda_graph_mixed_prefill_request_count: Number of mixed prefill requests for CUDA graphs
max_requests: Maximum number of requests
max_tokens: Maximum total tokens
max_sequence_length: Maximum sequence length
use_cuda_graphs_for_non_decode_steps: Whether to use CUDA graphs for non-decode steps
+ num_speculative_tokens: Number of speculative tokens
Returns:
Tuple containing:
@@ -339,7 +361,7 @@ def generate_cuda_graph_batch_dimensions_list(
def add_if_valid(token_count: int, prefill_req_count: int, decode_req_count: int) -> None:
"""Helper to create and append batch dimension to list only if it's valid."""
batch_dim = InferenceBatchDimensions(token_count, prefill_req_count, decode_req_count)
- if batch_dim.is_valid(max_requests, max_sequence_length):
+ if batch_dim.is_valid(max_requests, max_sequence_length, num_speculative_tokens):
cuda_graph_batch_dimensions_list.append(batch_dim)
# Cuda graph token-counts
@@ -355,7 +377,19 @@ def add_if_valid(token_count: int, prefill_req_count: int, decode_req_count: int
or cuda_graph_max_tokens <= 0
):
cuda_graph_max_tokens = max_tokens
- num_cuda_graphs = min(max(num_cuda_graphs, 1), cuda_graph_max_tokens)
+
+ assert cuda_graph_max_tokens == max_requests * (num_speculative_tokens + 1), (
+ f"cuda_graph_max_tokens ({cuda_graph_max_tokens}) must equal max_requests *"
+ f"(num_speculative_tokens + 1) ({max_requests * (num_speculative_tokens + 1)}). "
+ "This is required for correctly syncing EP ranks: "
+ f"prefill and decode graph pools must have the same token count granularity."
+ )
+
+ if num_cuda_graphs != -1:
+ # if -1, no need to adjust. This will be taken care of in
+ # the _calculate_cuda_graph_token_counts function where we will generate
+ # the token counts based on the max_tokens value and the step size.
+ num_cuda_graphs = min(max(num_cuda_graphs, 1), cuda_graph_max_tokens)
# Calculate token counts for prefill and mixed graphs.
# These need the full cuda_graph_max_tokens to handle variable-length sequences.
@@ -368,8 +402,9 @@ def add_if_valid(token_count: int, prefill_req_count: int, decode_req_count: int
)
# Calculate separate token counts for decode-only graphs.
- # Decode graphs can be more conservative since each request uses exactly 1 token.
- cuda_graph_max_tokens_decode = min(cuda_graph_max_tokens, max_requests)
+ cuda_graph_max_tokens_decode = min(
+ cuda_graph_max_tokens, max_requests * (num_speculative_tokens + 1)
+ )
cuda_graph_decode_token_counts = (
CUDAGraphBatchDimensionBuilder._calculate_cuda_graph_token_counts(
tp_size=tp_size,
@@ -382,31 +417,40 @@ def add_if_valid(token_count: int, prefill_req_count: int, decode_req_count: int
if num_cuda_graphs is None:
cuda_graph_batch_dimensions_list = []
elif (
- not cuda_graph_mixed_prefill_count
- or cuda_graph_mixed_prefill_count <= 0
+ not cuda_graph_mixed_prefill_request_count
+ or cuda_graph_mixed_prefill_request_count <= 0
or not use_cuda_graphs_for_non_decode_steps
): # decode only
# Use decode-specific token counts for decode-only graphs
for size in cuda_graph_decode_token_counts:
+ decode_req_count = min(size // (num_speculative_tokens + 1), max_requests)
+ token_count = decode_req_count * (num_speculative_tokens + 1)
+ token_count = token_count // tp_size * tp_size
add_if_valid(
- token_count=min(size, max_requests),
- prefill_req_count=0,
- decode_req_count=min(size, max_requests),
+ token_count=token_count, prefill_req_count=0, decode_req_count=decode_req_count
)
else:
# Mixed prefill and decode mode
# Create prefill and mixed dimensions with full token counts
for size in cuda_graph_prefill_token_counts:
+ assert size % tp_size == 0
+ prefill_req_count = min(cuda_graph_mixed_prefill_request_count, max_requests)
+ decode_req_count = max(
+ 0,
+ min(
+ (size - prefill_req_count) // (num_speculative_tokens + 1),
+ max_requests - prefill_req_count,
+ ),
+ )
add_if_valid(
token_count=size,
- prefill_req_count=min(cuda_graph_mixed_prefill_count, max_requests),
- decode_req_count=min(size, max_requests)
- - min(cuda_graph_mixed_prefill_count, max_requests),
+ prefill_req_count=prefill_req_count,
+ decode_req_count=decode_req_count,
)
# We need to ensure the prefill requests are shorter than the max sequence length,
# considering the one decode token is used for prefill request construction
prefill_only_minimal_num = max(
- cuda_graph_mixed_prefill_count,
+ cuda_graph_mixed_prefill_request_count,
math.ceil(size / max(1, max_sequence_length - 1)),
)
if prefill_only_minimal_num < max_requests:
@@ -418,16 +462,21 @@ def add_if_valid(token_count: int, prefill_req_count: int, decode_req_count: int
# Create decode-only dimensions with optimized token counts
for size in cuda_graph_decode_token_counts:
+ decode_req_count = min(size // (num_speculative_tokens + 1), max_requests)
+ token_count = decode_req_count * (num_speculative_tokens + 1)
+ token_count = token_count // tp_size * tp_size
add_if_valid(
- token_count=min(size, max_requests),
- prefill_req_count=0,
- decode_req_count=min(size, max_requests),
+ token_count=token_count, prefill_req_count=0, decode_req_count=decode_req_count
)
# Remove duplicates and sort by prefill token count
cuda_graph_batch_dimensions_list = list(set(cuda_graph_batch_dimensions_list))
cuda_graph_batch_dimensions_list.sort(
- key=lambda x: ((x.token_count - x.decode_req_count), x.decode_req_count), reverse=True
+ key=lambda x: (
+ (x.token_count - x.decode_req_count * (num_speculative_tokens + 1)),
+ x.decode_req_count,
+ ),
+ reverse=True,
)
# Collect actual token counts from batch dimensions, then unique and sort
@@ -447,6 +496,7 @@ def add_if_valid(token_count: int, prefill_req_count: int, decode_req_count: int
def match_graph_config(
real_batch_dim: InferenceBatchDimensions,
cuda_graph_batch_dimensions_list: List[InferenceBatchDimensions],
+ smallest_non_decode_cuda_graph_size: int,
strict: bool = False,
decode_only_cuda_graphs: bool = False,
ep_group: Optional[torch.distributed.ProcessGroup] = None,
@@ -478,6 +528,7 @@ def match_graph_config(
strict=strict,
decode_only_cuda_graphs=decode_only_cuda_graphs,
ep_group=ep_group,
+ smallest_non_decode_cuda_graph_size=smallest_non_decode_cuda_graph_size,
)
if adjusted_batch_dim is None:
@@ -497,4 +548,5 @@ def match_graph_config(
return None
# then find the best batch dimension
best_batch_dim = min(graph_batch_dims_applicable)
+
return best_batch_dim
diff --git a/megatron/core/inference/communication/torch_symm_triton/__init__.py b/megatron/core/inference/communication/torch_symm_triton/__init__.py
index ca58663d9ec..967dc8329f1 100644
--- a/megatron/core/inference/communication/torch_symm_triton/__init__.py
+++ b/megatron/core/inference/communication/torch_symm_triton/__init__.py
@@ -1,4 +1,5 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-from .collectives import multimem_all_gather, multimem_reduce_scatter
+from .collectives import multimem_all_gather, multimem_all_gather_fused, multimem_reduce_scatter
from .fused_collectives import fused_multimem_rs_add_norm_ag
+from .utils import are_tensors_nvls_eligible, is_device_nvls_capable
diff --git a/megatron/core/inference/communication/torch_symm_triton/collectives.py b/megatron/core/inference/communication/torch_symm_triton/collectives.py
index 4bc4dbde42b..3120f032812 100644
--- a/megatron/core/inference/communication/torch_symm_triton/collectives.py
+++ b/megatron/core/inference/communication/torch_symm_triton/collectives.py
@@ -23,43 +23,39 @@
from .barrier import symm_mem_sync
from .multimem_asm import ld_128, st_128
-from .utils import get_flat_tid, sync_threads
+from .utils import are_tensors_nvls_eligible, get_flat_tid, sync_threads
@triton.jit
-def _multimem_all_gather_kernel(
- local_ptr,
- multicast_ptr,
- signal_pad_ptrs,
- numel,
- BLOCK_SIZE: tl.constexpr,
- NUMEL_PER_THREAD: tl.constexpr,
- RANK: tl.constexpr,
- WORLD_SIZE: tl.constexpr,
+def _ag_phase(
+ local_ptr, multicast_ptr, byte_offset, numel, BLOCK_SIZE, NUMEL_PER_THREAD, RANK, WORLD_SIZE
):
"""
- Triton kernel to perform multicast all-gather over nvlink using multimem instructions.
- """
- # an all-gather is simply a multicast store operation
- # we only need a barrier at the end to ensure visibility of writes
+ Core all-gather phase: load from local memory, multicast-store to symmetric buffer.
+ This is the building block for both single-tensor and fused multi-tensor all-gathers.
+
+ Each thread handles 128-bit (NUMEL_PER_THREAD elements) at a time.
+ byte_offset locates the tensor within the multicast buffer.
+ """
pid = tl.program_id(axis=0)
tid = get_flat_tid()
- # From this point on, we pretend each element is 128-bit
- numel = numel // NUMEL_PER_THREAD
- numel_per_rank = tl.cdiv(numel, WORLD_SIZE)
+ numel_128 = numel // NUMEL_PER_THREAD
+ numel_per_rank = tl.cdiv(numel_128, WORLD_SIZE)
block_start = pid * BLOCK_SIZE
while block_start < numel_per_rank:
offsets = block_start + tid
mask = offsets < numel_per_rank
- # Each pointer points to a 128-bit bit pack
- # RANK * numel_per_rank -> brings us to the start of our rank's segment
- # offsets -> brings us to the right offset within our rank's segment
+ # byte_offset // 8 -> converts byte offset to uint64 offset
+ # RANK * numel_per_rank -> start of our rank's segment
+ # * 2 -> each 128-bit pack is 2 uint64s
multicast_ptrs = (
- multicast_ptr.to(tl.pointer_type(tl.uint64)) + (RANK * numel_per_rank + offsets) * 2
+ multicast_ptr.to(tl.pointer_type(tl.uint64))
+ + byte_offset // 8
+ + (RANK * numel_per_rank + offsets) * 2
)
local_ptrs = local_ptr.to(tl.pointer_type(tl.uint64)) + offsets * 2
(x, y, z, w) = ld_128(local_ptrs, mask=mask, multicast_op=False)
@@ -67,6 +63,23 @@ def _multimem_all_gather_kernel(
block_start += tl.num_programs(axis=0) * BLOCK_SIZE
+
+@triton.jit
+def _multimem_all_gather_kernel(
+ local_ptr,
+ multicast_ptr,
+ signal_pad_ptrs,
+ numel,
+ byte_offset,
+ BLOCK_SIZE: tl.constexpr,
+ NUMEL_PER_THREAD: tl.constexpr,
+ RANK: tl.constexpr,
+ WORLD_SIZE: tl.constexpr,
+):
+ """Single-tensor multicast all-gather kernel."""
+ _ag_phase(
+ local_ptr, multicast_ptr, byte_offset, numel, BLOCK_SIZE, NUMEL_PER_THREAD, RANK, WORLD_SIZE
+ )
sync_threads()
symm_mem_sync(
signal_pad_ptrs,
@@ -78,54 +91,68 @@ def _multimem_all_gather_kernel(
)
-def multimem_all_gather(
- output_tensor: torch.Tensor,
- input_tensor: torch.Tensor,
- symm_mem_hdl: _SymmetricMemory,
- **kwargs,
-) -> torch.Tensor:
+@triton.jit
+def _multimem_all_gather_3_kernel(
+ local_ptr_0,
+ local_ptr_1,
+ local_ptr_2,
+ multicast_ptr,
+ signal_pad_ptrs,
+ numel_0,
+ byte_offset_0,
+ numel_1,
+ byte_offset_1,
+ numel_2,
+ byte_offset_2,
+ BLOCK_SIZE: tl.constexpr,
+ NUMEL_PER_THREAD: tl.constexpr,
+ RANK: tl.constexpr,
+ WORLD_SIZE: tl.constexpr,
+):
"""
- Calls a multicast all-gather triton kernel on the given tensor.
- Output tensor must be a symmetric memory buffer.
- Input tensor can be a regular torch tensor
- Arguments:
- output_tensor: torch.Tensor - output tensor to be all-gathered into
- input_tensor: torch.Tensor - input tensor to be all-gathered from
- symm_mem_hdl: _SymmetricMemory - handle to the symmetric memory buffer for output_tensor
- Returns:
- torch.Tensor - all-gathered tensor, which is output_tensor
+ Fused 3-tensor multicast all-gather. Processes three tensors in sequence
+ then synchronizes once, eliminating 2 kernel launches and 2 barriers
+ compared to three separate multimem_all_gather calls.
"""
- assert HAVE_TRITON, "Triton is required for multimem all-gather."
-
- config = {
- "max_num_blocks": kwargs.get("max_num_blocks", 24),
- "num_warps": kwargs.get("num_warps", 32),
- "BLOCK_SIZE": kwargs.get("BLOCK_SIZE", 1024),
- }
- assert input_tensor.dtype == torch.bfloat16, "Only bfloat16 is supported for now."
- assert output_tensor.dtype == torch.bfloat16, "Only bfloat16 is supported for now."
- numel_per_thread = 128 // (input_tensor.element_size() * 8)
-
- assert (
- output_tensor.numel() % numel_per_thread == 0
- ), "The number of elements must be 128-bit aligned."
-
- num_threads = triton.cdiv(output_tensor.numel() // numel_per_thread, symm_mem_hdl.world_size)
- num_blocks = min(triton.cdiv(num_threads, config["BLOCK_SIZE"]), config["max_num_blocks"])
-
- _multimem_all_gather_kernel[(num_blocks, 1, 1)](
- input_tensor.data_ptr(),
- symm_mem_hdl.multicast_ptr,
- symm_mem_hdl.signal_pad_ptrs_dev,
- numel=output_tensor.numel(),
- BLOCK_SIZE=config["BLOCK_SIZE"],
- NUMEL_PER_THREAD=numel_per_thread,
- RANK=symm_mem_hdl.rank,
- WORLD_SIZE=symm_mem_hdl.world_size,
- num_warps=config["num_warps"],
+ _ag_phase(
+ local_ptr_0,
+ multicast_ptr,
+ byte_offset_0,
+ numel_0,
+ BLOCK_SIZE,
+ NUMEL_PER_THREAD,
+ RANK,
+ WORLD_SIZE,
+ )
+ _ag_phase(
+ local_ptr_1,
+ multicast_ptr,
+ byte_offset_1,
+ numel_1,
+ BLOCK_SIZE,
+ NUMEL_PER_THREAD,
+ RANK,
+ WORLD_SIZE,
+ )
+ _ag_phase(
+ local_ptr_2,
+ multicast_ptr,
+ byte_offset_2,
+ numel_2,
+ BLOCK_SIZE,
+ NUMEL_PER_THREAD,
+ RANK,
+ WORLD_SIZE,
+ )
+ sync_threads()
+ symm_mem_sync(
+ signal_pad_ptrs,
+ None,
+ RANK,
+ WORLD_SIZE,
+ hasPreviousMemAccess=True,
+ hasSubsequentMemAccess=True,
)
-
- return output_tensor
@triton.jit
@@ -138,9 +165,11 @@ def _multimem_reduce_scatter_kernel(
NUMEL_PER_THREAD: tl.constexpr,
RANK: tl.constexpr,
WORLD_SIZE: tl.constexpr,
+ REDUCE_F32: tl.constexpr = False,
):
"""
Triton kernel to perform multicast reduce-scatter over nvlink using multimem instructions.
+ When REDUCE_F32=True, uses fp32 reduction instead of bf16x2 reduction.
"""
symm_mem_sync(
signal_pad_ptrs,
@@ -169,53 +198,160 @@ def _multimem_reduce_scatter_kernel(
multicast_ptr.to(tl.pointer_type(tl.uint64)) + (RANK * numel_per_rank + offsets) * 2
)
local_ptrs = local_ptr.to(tl.pointer_type(tl.uint64)) + offsets * 2
- (x, y, z, w) = ld_128(multicast_ptrs, mask=mask, multicast_op=True)
+ (x, y, z, w) = ld_128(multicast_ptrs, mask=mask, multicast_op=True, reduce_f32=REDUCE_F32)
st_128(local_ptrs, x, y, z, w, mask=mask, multicast_op=False)
block_start += tl.num_programs(axis=0) * BLOCK_SIZE
-def multimem_reduce_scatter(
+# ── Python wrappers ─────────────────────────────────────────────────────────
+
+_DEFAULT_KERNEL_CONFIG = {"max_num_blocks": 128, "num_warps": 32, "BLOCK_SIZE": 1024}
+
+
+def _kernel_launch_config(element_size: int, max_numel: int, world_size: int, **kwargs):
+ """Compute kernel launch config shared by all collective wrappers.
+
+ Args:
+ element_size: bytes per element (e.g. 2 for bf16).
+ max_numel: largest tensor numel (determines grid size).
+ world_size: number of ranks.
+
+ Returns:
+ (numel_per_thread, num_blocks, config) tuple.
+ """
+ config = {k: kwargs.get(k, v) for k, v in _DEFAULT_KERNEL_CONFIG.items()}
+ numel_per_thread = 128 // (element_size * 8)
+ num_threads = triton.cdiv(max_numel // numel_per_thread, world_size)
+ num_blocks = min(triton.cdiv(num_threads, config["BLOCK_SIZE"]), config["max_num_blocks"])
+ return numel_per_thread, num_blocks, config
+
+
+def multimem_all_gather(
output_tensor: torch.Tensor,
input_tensor: torch.Tensor,
symm_mem_hdl: _SymmetricMemory,
+ byte_offset: int = 0,
**kwargs,
) -> torch.Tensor:
"""
- Calls a multicast reduce-scatter triton kernel on the given tensor.
- Input tensor must be a symmetric memory buffer.
- Output tensor can be a regular torch tensor
- Arguments:
- output_tensor: torch.Tensor - output tensor to be reduce-scattered into
- input_tensor: torch.Tensor - input tensor to be reduce-scattered from
- symm_mem_hdl: _SymmetricMemory - handle to the symmetric memory buffer for input_tensor
- **kwargs: Additional keyword arguments for kernel configuration:
- max_num_blocks (int, optional): The maximum number of blocks to launch.
- num_warps (int, optional): The number of warps per block.
- BLOCK_SIZE (int, optional): The BLOCK_SIZE parameter for the kernel.
- Returns:
- torch.Tensor - reduce-scattered tensor, which is output_tensor
+ Multicast all-gather for a single tensor.
+ Output tensor must be a symmetric memory buffer.
+ Input tensor can be a regular torch tensor.
"""
+ assert HAVE_TRITON, "Triton is required for multimem all-gather."
+ assert are_tensors_nvls_eligible(
+ input_tensor
+ ), "Input tensor must be 16-byte divisible on Hopper+ for NVLS."
+ assert (
+ output_tensor.numel() % input_tensor.numel() == 0
+ and output_tensor.numel() // input_tensor.numel() == symm_mem_hdl.world_size
+ ), "Output numel must be exactly world_size * input numel for all-gather."
- assert HAVE_TRITON, "Triton is required for multimem reduce-scatter."
+ numel_per_thread, num_blocks, config = _kernel_launch_config(
+ input_tensor.element_size(), output_tensor.numel(), symm_mem_hdl.world_size, **kwargs
+ )
+ _multimem_all_gather_kernel[(num_blocks, 1, 1)](
+ input_tensor.data_ptr(),
+ symm_mem_hdl.multicast_ptr,
+ symm_mem_hdl.signal_pad_ptrs_dev,
+ numel=output_tensor.numel(),
+ byte_offset=byte_offset,
+ BLOCK_SIZE=config["BLOCK_SIZE"],
+ NUMEL_PER_THREAD=numel_per_thread,
+ RANK=symm_mem_hdl.rank,
+ WORLD_SIZE=symm_mem_hdl.world_size,
+ num_warps=config["num_warps"],
+ )
- config = {
- "max_num_blocks": kwargs.get("max_num_blocks", 24),
- "num_warps": kwargs.get("num_warps", 32),
- "BLOCK_SIZE": kwargs.get("BLOCK_SIZE", 1024),
- }
+ return output_tensor
- assert input_tensor.dtype == torch.bfloat16, "Only bfloat16 is supported for now."
- assert output_tensor.dtype == torch.bfloat16, "Only bfloat16 is supported for now."
- numel_per_thread = 128 // (output_tensor.element_size() * 8)
- assert (
- input_tensor.numel() % numel_per_thread == 0
- ), "The number of elements must be 128-bit aligned."
+def multimem_all_gather_fused(
+ output_0: torch.Tensor,
+ input_0: torch.Tensor,
+ byte_offset_0: int,
+ output_1: torch.Tensor,
+ input_1: torch.Tensor,
+ byte_offset_1: int,
+ output_2: torch.Tensor,
+ input_2: torch.Tensor,
+ byte_offset_2: int,
+ symm_mem_hdl: _SymmetricMemory,
+ **kwargs,
+) -> None:
+ """
+ Fused 3-tensor multicast all-gather. Equivalent to calling multimem_all_gather
+ three times but with a single kernel launch and a single barrier.
- num_threads = triton.cdiv(input_tensor.numel() // numel_per_thread, symm_mem_hdl.world_size)
- num_blocks = min(triton.cdiv(num_threads, config["BLOCK_SIZE"]), config["max_num_blocks"])
+ All tensors must share the same symmetric memory handle.
+ """
+ assert HAVE_TRITON, "Triton is required for multimem all-gather."
+ assert are_tensors_nvls_eligible(
+ input_0, input_1, input_2
+ ), "All input tensors must be 16-byte divisible on Hopper+ for NVLS."
+ for inp, out in [(input_0, output_0), (input_1, output_1), (input_2, output_2)]:
+ assert (
+ out.numel() % inp.numel() == 0 and out.numel() // inp.numel() == symm_mem_hdl.world_size
+ ), "Output numel must be exactly world_size * input numel for all-gather."
+
+ max_numel = max(output_0.numel(), output_1.numel(), output_2.numel())
+
+ numel_per_thread, num_blocks, config = _kernel_launch_config(
+ input_0.element_size(), max_numel, symm_mem_hdl.world_size, **kwargs
+ )
+ _multimem_all_gather_3_kernel[(num_blocks, 1, 1)](
+ input_0.data_ptr(),
+ input_1.data_ptr(),
+ input_2.data_ptr(),
+ symm_mem_hdl.multicast_ptr,
+ symm_mem_hdl.signal_pad_ptrs_dev,
+ numel_0=output_0.numel(),
+ byte_offset_0=byte_offset_0,
+ numel_1=output_1.numel(),
+ byte_offset_1=byte_offset_1,
+ numel_2=output_2.numel(),
+ byte_offset_2=byte_offset_2,
+ BLOCK_SIZE=config["BLOCK_SIZE"],
+ NUMEL_PER_THREAD=numel_per_thread,
+ RANK=symm_mem_hdl.rank,
+ WORLD_SIZE=symm_mem_hdl.world_size,
+ num_warps=config["num_warps"],
+ )
+
+
+def multimem_reduce_scatter(
+ output_tensor: torch.Tensor,
+ input_tensor: torch.Tensor,
+ symm_mem_hdl: _SymmetricMemory,
+ **kwargs,
+) -> torch.Tensor:
+ """
+ Multicast reduce-scatter for a single tensor.
+ Input tensor must be a symmetric memory buffer.
+ Output tensor can be a regular torch tensor.
+ Supports bfloat16 and float32 dtypes.
+ """
+ assert HAVE_TRITON, "Triton is required for multimem reduce-scatter."
+ assert input_tensor.dtype in (
+ torch.bfloat16,
+ torch.float32,
+ ), f"Only bfloat16 and float32 are supported, got {input_tensor.dtype}"
+ assert (
+ input_tensor.dtype == output_tensor.dtype
+ ), f"Input and output dtypes must match: {input_tensor.dtype} vs {output_tensor.dtype}"
+ assert are_tensors_nvls_eligible(
+ output_tensor
+ ), "Output tensor must be 16-byte divisible on Hopper+ for NVLS."
+ assert (
+ input_tensor.numel() % output_tensor.numel() == 0
+ and input_tensor.numel() // output_tensor.numel() == symm_mem_hdl.world_size
+ ), "Input numel must be exactly world_size * output numel for reduce-scatter."
+ reduce_f32 = input_tensor.dtype == torch.float32
+ numel_per_thread, num_blocks, config = _kernel_launch_config(
+ output_tensor.element_size(), input_tensor.numel(), symm_mem_hdl.world_size, **kwargs
+ )
_multimem_reduce_scatter_kernel[(num_blocks, 1, 1)](
output_tensor.data_ptr(),
symm_mem_hdl.multicast_ptr,
@@ -226,6 +362,7 @@ def multimem_reduce_scatter(
RANK=symm_mem_hdl.rank,
WORLD_SIZE=symm_mem_hdl.world_size,
num_warps=config["num_warps"],
+ REDUCE_F32=reduce_f32,
)
return output_tensor
diff --git a/megatron/core/inference/communication/torch_symm_triton/multimem_asm.py b/megatron/core/inference/communication/torch_symm_triton/multimem_asm.py
index 774c3f6d2bf..859b9010aea 100644
--- a/megatron/core/inference/communication/torch_symm_triton/multimem_asm.py
+++ b/megatron/core/inference/communication/torch_symm_triton/multimem_asm.py
@@ -1,7 +1,9 @@
# Copyright (c) Meta Platforms, Inc. and affiliates.
+# pylint: disable=line-too-long
# Adapted from https://github.com/yifuwang/symm-mem-recipes.git
+
from unittest.mock import MagicMock
from megatron.core.utils import null_decorator
@@ -16,60 +18,76 @@
@triton.jit
-def ld_128(ptr, mask, multicast_op: tl.constexpr):
+def ld_128(ptr, mask, multicast_op: tl.constexpr, reduce_f32: tl.constexpr = False):
"""
- Loads 128 bits (8 x bf16) from memory into registers.
+ Loads 128 bits from memory into registers.
This function abstracts two distinct hardware behaviors based on `multicast_op`:
1. **Standard Load (`multicast_op=False`)**:
- **Semantics:** Local Global Memory Load.
- **Action:** Reads 128 bits from `ptr` in global memory into the local register file.
- - **Use Case:** Standard tensor processing.
2. **Multicast Reduce-Load (`multicast_op=True`)**:
- **Semantics:** "Pull" Reduction over NVLink.
- **Action:** Simultaneously reads 128 bits from the *same* address across all peer GPUs
- in the multicast group, sums them (add reduction), and loads the result into the
- local register file.
+ in the multicast group, sums them, and loads the result into the local register file.
- **Hardware:** Uses `multimem.ld_reduce` (Hopper+).
- - **Use Case:** The "Reduce" step in collective operations.
+ - When `reduce_f32=False` (default): bf16x2 addition with f32 accumulation
+ (128 bits = 8 x bf16, 2 per register).
+ - When `reduce_f32=True`: native f32 addition
+ (128 bits = 4 x fp32, 1 per register).
Args:
ptr: Memory pointer to the source buffer.
mask: Boolean predicate. If False, the operation is skipped (no-op).
multicast_op (tl.constexpr): Toggles between standard load (False)
- and multicast-reduce (True).
+ and multicast-reduce (True).
+ reduce_f32 (tl.constexpr): When True and multicast_op=True, uses f32 reduction
+ instead of bf16x2 reduction. Default False.
Returns:
Four 32-bit registers (tl.uint32), representing 128 bits of loaded data.
- Note: When interpreting as bf16, this equates to 8 values (2 per register).
"""
- # PTX Assembly Logic:
- # 1. @$5: Predication. Only execute if argument 5 (mask) is True (1).
- # 2. Opcode Selection:
- # - 'multimem.ld_reduce...add.v4.bf16x2': Hardware-accelerated reduction across peers.
- # - 'ld.global...v4.u32': Standard 128-bit memory read.
- # 3. Operands:
- # - {$0, $1, $2, $3}: Destination registers (Output).
- # - [$4]: Source memory address (Input).
if multicast_op:
- return tl.inline_asm_elementwise(
- """
- {
- .reg .pred %p0;
- setp.ne.s32 %p0, $5, 1;
- @%p0 bra end;
- multimem.ld_reduce.relaxed.sys.global.add.acc::f32.v4.bf16x2 {$0, $1, $2, $3}, [$4];
- end:
- }
- """,
- "=r,=r,=r,=r,l,r",
- args=[ptr, mask.to(tl.int32)],
- dtype=(tl.uint32, tl.uint32, tl.uint32, tl.uint32),
- is_pure=True,
- pack=1,
- )
+ if reduce_f32:
+ # fp32 reduction: multimem.ld_reduce.add.v4.f32
+ # Each 128-bit load reduces 4 x fp32 values across peers.
+ return tl.inline_asm_elementwise(
+ """
+ {
+ .reg .pred %p0;
+ setp.ne.s32 %p0, $5, 1;
+ @%p0 bra end;
+ multimem.ld_reduce.relaxed.sys.global.add.v4.f32 {$0, $1, $2, $3}, [$4];
+ end:
+ }
+ """,
+ "=r,=r,=r,=r,l,r",
+ args=[ptr, mask.to(tl.int32)],
+ dtype=(tl.uint32, tl.uint32, tl.uint32, tl.uint32),
+ is_pure=True,
+ pack=1,
+ )
+ else:
+ # bf16x2 reduction with f32 accumulation: multimem.ld_reduce.add.acc::f32.v4.bf16x2
+ # Each 128-bit load reduces 8 x bf16 values (packed as 4 x bf16x2) across peers.
+ return tl.inline_asm_elementwise(
+ """
+ {
+ .reg .pred %p0;
+ setp.ne.s32 %p0, $5, 1;
+ @%p0 bra end;
+ multimem.ld_reduce.relaxed.sys.global.add.acc::f32.v4.bf16x2 {$0, $1, $2, $3}, [$4];
+ end:
+ }
+ """,
+ "=r,=r,=r,=r,l,r",
+ args=[ptr, mask.to(tl.int32)],
+ dtype=(tl.uint32, tl.uint32, tl.uint32, tl.uint32),
+ is_pure=True,
+ pack=1,
+ )
else:
return tl.inline_asm_elementwise(
"""
diff --git a/megatron/core/inference/communication/torch_symm_triton/utils.py b/megatron/core/inference/communication/torch_symm_triton/utils.py
index 785481dfba6..3cc6dd8dcc0 100644
--- a/megatron/core/inference/communication/torch_symm_triton/utils.py
+++ b/megatron/core/inference/communication/torch_symm_triton/utils.py
@@ -4,6 +4,8 @@
from unittest.mock import MagicMock
+import torch
+
from megatron.core.utils import null_decorator
try:
@@ -15,6 +17,27 @@
triton.jit = null_decorator
+def is_device_nvls_capable(device: torch.device) -> bool:
+ """Check if the device supports NVLS (multicast) collectives.
+ Requires CUDA Hopper+ (SM >= 9)."""
+ return device.type == "cuda" and torch.cuda.get_device_properties(device).major >= 9
+
+
+def are_tensors_nvls_eligible(*tensors: torch.Tensor) -> bool:
+ """Check if tensors are eligible for NVLS (multicast) collectives.
+
+ Requirements:
+ - Hopper+ GPU (SM >= 9)
+ - All tensor byte sizes are divisible by 16 (128-bit), since NVLS
+ kernels process data in 128-bit chunks.
+ """
+ if not tensors:
+ return False
+ return is_device_nvls_capable(tensors[0].device) and all(
+ t.element_size() * t.numel() % 16 == 0 for t in tensors
+ )
+
+
@triton.jit
def get_tid():
"""
diff --git a/megatron/core/inference/config.py b/megatron/core/inference/config.py
new file mode 100644
index 00000000000..4063ffbc977
--- /dev/null
+++ b/megatron/core/inference/config.py
@@ -0,0 +1,317 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+from dataclasses import dataclass
+from enum import Enum
+from typing import List, Optional, Tuple
+
+import torch
+
+from megatron.core.process_groups_config import ProcessGroupCollection
+from megatron.core.transformer.module import MegatronModule
+from megatron.core.utils import get_attr_wrapped_model
+
+
+@dataclass
+class MambaInferenceStateConfig:
+ """
+ Config for initializing Mamba model inference state tensors.
+
+ Note that we maintain separate metadata for decode, regular prefill, and
+ chunked prefill requests because the Mamba kernels do not yet support mixing
+ these. Once the kernels have been updated we can simplify this code.
+ """
+
+ layer_type_list: List[str]
+ """
+ A list of strings that indicates the layer type (Mamba / Attention / MLP) for each layer.
+ See `megatron/core/ssm/mamba_hybrid_layer_allocation.py` for the list of symbols.
+ """
+
+ conv_states_shape: Tuple[int]
+ """Mamba conv states shape per request."""
+
+ ssm_states_shape: Tuple[int]
+ """Mamba SSM states shape per request."""
+
+ conv_states_dtype: torch.dtype
+ """The dtype to use for the Mamba conv state tensor. Defaults to the model dtype."""
+
+ ssm_states_dtype: torch.dtype
+ """The dtype to use for the Mamba SSM state tensor. Defaults to the model dtype."""
+
+ mamba_chunk_size: int = 128
+ """The chunk size used by the Mamba SSM Triton kernels."""
+
+ @classmethod
+ def from_model(
+ cls,
+ model: MegatronModule,
+ conv_states_dtype: Optional[torch.dtype] = None,
+ ssm_states_dtype: Optional[torch.dtype] = None,
+ ) -> Optional["MambaInferenceStateConfig"]:
+ """Returns Mamba inference state config from the model if it is a hybrid model."""
+ from megatron.core.ssm.mamba_hybrid_layer_allocation import Symbols
+
+ decoder = get_attr_wrapped_model(model, "decoder")
+ layer_type_list = getattr(decoder, "layer_type_list", None)
+ if layer_type_list is not None and Symbols.MAMBA in layer_type_list:
+ (mamba_conv_states_shape, mamba_ssm_states_shape) = (
+ decoder.mamba_state_shapes_per_request()
+ )
+ if conv_states_dtype is None:
+ conv_states_dtype = model.config.params_dtype
+ if ssm_states_dtype is None:
+ ssm_states_dtype = model.config.params_dtype
+ mamba_chunk_size = 128
+ for layer_type, layer in zip(decoder.layer_type_list, decoder.layers):
+ if layer_type == Symbols.MAMBA and hasattr(layer, 'mixer'):
+ mamba_chunk_size = layer.mixer.chunk_size
+ break
+ return cls(
+ layer_type_list=layer_type_list,
+ conv_states_shape=mamba_conv_states_shape,
+ ssm_states_shape=mamba_ssm_states_shape,
+ conv_states_dtype=conv_states_dtype,
+ ssm_states_dtype=ssm_states_dtype,
+ mamba_chunk_size=mamba_chunk_size,
+ )
+ return None
+
+
+class PrefixCachingEvictionPolicy(str, Enum):
+ """Eviction policy for prefix caching blocks.
+
+ Only applies when enable_prefix_caching is True.
+ """
+
+ REF_ZERO = "ref_zero"
+ """Deregister blocks immediately when ref_count hits 0. No caching after release."""
+
+ LRU = "lru"
+ """Keep released blocks in hash table. Evict oldest ref=0 blocks when space is needed."""
+
+
+class PrefixCachingCoordinatorPolicy(str, Enum):
+ """Routing policy for the DP inference coordinator with prefix caching."""
+
+ LONGEST_PREFIX = "longest_prefix"
+ """Route to the rank with the longest consecutive prefix match."""
+
+ FIRST_PREFIX_BLOCK = "first_prefix_block"
+ """Route to the rank that has the first block hash cached. O(ranks) check."""
+
+ ROUND_ROBIN = "round_robin"
+ """Route requests to ranks in round-robin order, ignoring prefix affinity."""
+
+
+class KVCacheManagementMode(str, Enum):
+ """Mode for handling large tensors (KV cache, Mamba states) during suspend/resume."""
+
+ PERSIST = "persist"
+ """Do not deallocate and reallocate large tensors; keep them on GPU."""
+
+ OFFLOAD = "offload"
+ """Offload large tensors to CPU during deallocation; onload during allocation."""
+
+ RECOMPUTE = "recompute"
+ """Deallocate large tensors and recompute them from scratch during allocation."""
+
+
+@dataclass
+class InferenceConfig:
+ """
+ Config for inference.
+
+ NOTE: Must remain mutually exclusive with the `TransformerConfig`.
+ """
+
+ # =================================
+ # KV cache and Mamba states config
+ # =================================
+ block_size_tokens: int = 256
+ """Size of KV cache block size."""
+
+ buffer_size_gb: int = 20
+ """
+ Buffer size reserved on the GPU for the KV cache.
+ If `unified_memory_level` >= 1, then CPU memory is additionally utilized, resulting in a total
+ buffer size of `buffer_size_gb + paused_buffer_size_gb`.
+ """
+
+ paused_buffer_size_gb: Optional[int] = None
+ """
+ Portion of buffer reserved for paused requests. Active requests are paused when there are not
+ enough active blocks available to continue generating a request. The total buffer size
+ (active + paused) depends on `unified_memory_level` (uvm):
+ - uvm 0: buffer_size_gb (paused buffer is inclusive)
+ - uvm 1: buffer_size_gb + paused_buffer_size_gb
+ """
+
+ mamba_inference_state_config: Optional[MambaInferenceStateConfig] = None
+ """The Mamba inference state config if the model is a hybrid model."""
+
+ mamba_memory_ratio: Optional[float] = None
+ """
+ Percentage of memory buffer to allocate for Mamba states. If not specified, allocates Mamba
+ state tensors for each KV cache block. Only used for hybrid models.
+ """
+
+ max_requests: Optional[int] = None
+ """
+ Max number of active requests to use for decode-only forward passes.
+ This is primarily limited by the combination of `buffer_size_gb` and `max_sequence_length`.
+ """
+
+ max_tokens: Optional[int] = None
+ """
+ Max number of tokens to use for forward passes. This is primarily limited by prefill activation
+ memory usage. (Defaults to 16384).
+ """
+
+ unified_memory_level: int = 0
+ """
+ Sets unified memory usage within the dynamic inference context.
+ The levels are:
+ 0) no unified memory (default)
+ 1) allocate `memory_buffer` in unified memory.
+ Eventually, additional levels will be included to control other tensors within the context.
+ """
+
+ kv_cache_management_mode: KVCacheManagementMode = KVCacheManagementMode.PERSIST
+ """
+ Mode used to determine how large tensors are handled by the allocate and deallocate methods.
+ See `KVCacheManagementMode` for options.
+ """
+
+ # =================================
+ # CUDA graph config
+ # =================================
+ num_cuda_graphs: Optional[int] = None
+ """
+ Maximum number of cuda graphs to capture, where the cuda graph batch sizes range from 1 to
+ `max_requests`. Due to rounding, the actual number of cuda graphs may not equal this argument.
+ """
+
+ cuda_graph_mixed_prefill_count: Optional[int] = 16
+ """
+ The number of mixed prefill graphs to capture if mixed prefill/decode graphs are enabled.
+ """
+
+ use_cuda_graphs_for_non_decode_steps: bool = True
+ """
+ Whether to use CUDA graphs for non-decode steps.
+ """
+
+ static_kv_memory_pointers: bool = False
+ """
+ Whether the KV cache (and Mamba states) will reside at the same memory addresses
+ after suspend/resume as before. When True, CUDA graphs that reference these buffers
+ remain valid across suspend/resume cycles and do not need to be recaptured.
+ Requires either UVM or `torch_memory_saver` when `kv_cache_management_mode` is not PERSIST.
+ """
+
+ # =================================
+ # Model config
+ # =================================
+ max_sequence_length: int = 2560
+ """Max possible sequence length (prompt + output) that will occur."""
+
+ pg_collection: Optional[ProcessGroupCollection] = None
+ """A `ProcessGroupCollection` for distributed execution."""
+
+ use_flashinfer_fused_rope: Optional[bool] = False
+ """
+ If True, use flashinfer's fused rope implementation.
+ If None, defaults to using flash-infer if available.
+ """
+
+ materialize_only_last_token_logits: bool = True
+ """
+ Whether to only materialize logits for the last token. This should be set to False
+ if returning log probs.
+ """
+
+ # =================================
+ # Engine config
+ # =================================
+ enable_chunked_prefill: bool = False
+ """Whether to enable chunked prefill."""
+
+ num_speculative_tokens: int = 0
+ """The number of speculative tokens to generate for decode steps."""
+
+ enable_prefix_caching: bool = False
+ """Whether to enable prefix caching for KV cache block sharing."""
+
+ prefix_caching_eviction_policy: PrefixCachingEvictionPolicy = (
+ PrefixCachingEvictionPolicy.REF_ZERO
+ )
+ """Eviction policy for prefix caching blocks. See `PrefixCachingEvictionPolicy` for options.
+
+ Only applies when enable_prefix_caching is True.
+ """
+
+ prefix_caching_coordinator_policy: PrefixCachingCoordinatorPolicy = (
+ PrefixCachingCoordinatorPolicy.FIRST_PREFIX_BLOCK
+ )
+ """Routing policy for the DP inference coordinator. See
+ `PrefixCachingCoordinatorPolicy` for options.
+
+ Only applies when enable_prefix_caching is True and using a coordinator.
+ """
+
+ prefix_caching_routing_alpha: float = 0.5
+ """Weight for prefix-aware scoring: score = alpha * match + (1 - alpha) * normalized_load.
+ Higher alpha favors prefix cache hits; lower alpha favors load balance.
+ Must be in [0, 1]. Only applies when enable_prefix_caching is True and using a coordinator.
+ """
+
+ prefix_caching_mamba_gb: Optional[float] = None
+ """GPU memory budget (in GB) for the Mamba state cache used by prefix caching
+ on hybrid models. Each cache slot stores SSM and conv states for all Mamba layers
+ at a single block boundary. When set, Mamba states at KV divergence and last-aligned
+ block boundaries are cached and reused across requests with matching prefixes."""
+
+ # =================================
+ # Logging config
+ # =================================
+ track_paused_request_events: bool = False
+ """
+ Whether to track paused request events. If True, `add_event_pause()` is called on
+ requests when they are paused during bookkeeping.
+ """
+
+ track_generated_token_events: bool = False
+ """
+ Whether to track per-token events with timestamps for each generated token.
+ When enabled, each generated token creates a GENERATED_TOKEN event with a
+ timestamp, useful for per-token latency analysis.
+ """
+
+ metrics_writer: Optional["WandbModule"] = None
+ """Wandb module for writing metrics."""
+
+ logging_step_interval: int = 0
+ """
+ The step interval at which to log inference metrics to wandb.
+ Defaults to 0, which means no logging.
+ """
+
+ request_metadata_types: Optional[List[Tuple[str, torch.dtype, bool]]] = None
+ """
+ A list of the per-request metadata types to track. Each entry is a tuple
+ consisting of the string label, the target dtype, and whether to store the data on GPU.
+ """
+
+ use_synchronous_zmq_collectives: bool = False
+ """Whether to use synchronous ZMQ collectives for inference. If True, the
+ all_reduce_max operation will be performed synchronously, which can help reduce
+ performance variability for MoEs.
+ """
+
+ def __post_init__(self):
+ if not (0.0 <= self.prefix_caching_routing_alpha <= 1.0):
+ raise ValueError(
+ f"prefix_caching_routing_alpha must be in [0, 1], "
+ f"got {self.prefix_caching_routing_alpha}"
+ )
diff --git a/megatron/core/inference/contexts/__init__.py b/megatron/core/inference/contexts/__init__.py
index 2e92ca92846..6a7109a3b0e 100644
--- a/megatron/core/inference/contexts/__init__.py
+++ b/megatron/core/inference/contexts/__init__.py
@@ -3,7 +3,7 @@
import warnings
from .base_context import BaseInferenceContext
-from .dynamic_block_allocator import BlockAllocator
+from .kv_block_allocator import KVBlockAllocator
from .static_context import StaticInferenceContext
warnings.warn(
diff --git a/megatron/core/inference/contexts/attention_context/mamba_metadata.py b/megatron/core/inference/contexts/attention_context/mamba_metadata.py
index 6cf45aeb9e1..19091d35bfb 100644
--- a/megatron/core/inference/contexts/attention_context/mamba_metadata.py
+++ b/megatron/core/inference/contexts/attention_context/mamba_metadata.py
@@ -1,50 +1,40 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-from dataclasses import dataclass
-from typing import List, Optional, Tuple
+from typing import Optional
import torch
from megatron.core.inference.batch_dimensions_utils import InferenceBatchDimensions
-
-
-@dataclass
-class MambaInferenceStateConfig:
- """
- Config for initializing Mamba model inference state tensors.
-
- Note that we maintain separate metadata for decode, regular prefill, and
- chunked prefill requests because the Mamba kernels do not yet support mixing
- these. Once the kernels have been updated we can simplify this code.
- """
-
- layer_type_list: List[str]
- """
- A list of strings that indicates the layer type (Mamba / Attention / MLP) for each layer.
- See `megatron/core/ssm/mamba_hybrid_layer_allocation.py` for the list of symbols.
- """
-
- mamba_conv_states_shape: Tuple[int]
- """Mamba conv states shape per request."""
-
- mamba_ssm_states_shape: Tuple[int]
- """Mamba ssm states shape per request."""
+from megatron.core.inference.contexts.mamba_slot_allocator import (
+ MAX_INTERMEDIATE_OFFSETS_PER_REQUEST,
+)
class MambaMetadata:
"""Manages the metadata tensors required for Mamba layers during inference."""
- def __init__(self, max_requests: int, max_tokens: int):
+ def __init__(
+ self, max_requests: int, max_tokens: int, mamba_chunk_size: int = 128, d_conv: int = 0
+ ):
"""
Initializes the Mamba slot allocator.
Args:
max_requests (int): The maximum number of concurrent requests.
+ max_tokens (int): The maximum number of tokens.
+ mamba_chunk_size (int): The chunk size used by the Mamba SSM Triton kernels.
+ d_conv (int): Convolution window size (from mamba_conv_states_shape[-1]).
+ Used for vectorized conv state extraction at intermediate offsets.
"""
self.max_requests = max_requests
self.max_tokens = max_tokens
+ self.mamba_chunk_size = mamba_chunk_size
+ self.d_conv = d_conv
self.device = torch.cuda.current_device()
+ # Maximum possible chunks across all batch configurations
+ self.max_chunks = max_tokens // mamba_chunk_size + max_requests
+
# Map from requests to slots in the static Mamba state buffer
self.request_to_mamba_state_idx = torch.full(
(self.max_requests,), -1, dtype=torch.int32, device=torch.cuda.current_device()
@@ -60,11 +50,6 @@ def __init__(self, max_requests: int, max_tokens: int):
(self.max_requests,), -1, dtype=torch.int32, device=self.device
)
- # Map from the active chunked prefill request to its slot in the static Mamba state buffer
- self._batch_indices_chunked_prefill_buffer = torch.full(
- (1,), -1, dtype=torch.int32, device=self.device
- )
-
# Map from token id to request id for active prefill requests
self._seq_idx_buffer = torch.full(
(1, self.max_tokens), -1, dtype=torch.int32, device=self.device
@@ -80,20 +65,50 @@ def __init__(self, max_requests: int, max_tokens: int):
(2,), dtype=torch.int32, device=self.device
)
- # Tuple of (
- # total prefill sequence length excluding chunked prefill,
- # chunked prefill sequence length
- # )
- self._device_chunked_prefill_buffer = torch.zeros(
- (2,), dtype=torch.int32, device=self.device
+ # SSM chunk boundaries for varlen kernel
+ self._cu_chunk_seqlens_buffer = torch.zeros(
+ self.max_chunks + 1, dtype=torch.int32, device=self.device
+ )
+
+ # Index of the last chunk per sequence
+ self._last_chunk_indices_buffer = torch.zeros(
+ max_requests, dtype=torch.int32, device=self.device
+ )
+
+ # Request ID per chunk
+ self._seq_idx_for_varlen_buffer = torch.zeros(
+ self.max_chunks, dtype=torch.int32, device=self.device
)
+ # Conv1d per-token metadata (request ID and request start position)
+ self._conv_seq_idx_buffer = torch.zeros(max_tokens, dtype=torch.int32, device=self.device)
+ self._conv_seq_start_buffer = torch.zeros(max_tokens, dtype=torch.int32, device=self.device)
+
# Allocator for Mamba state slots
self.mamba_state_free_slots = torch.arange(
self.max_requests, dtype=torch.int32, device=torch.cuda.current_device()
)
self.mamba_state_free_slot_count = self.max_requests
+ # Intermediate state extraction buffers (CUDA graph compatible)
+ # Each prefill request can produce up to 3 intermediate offsets
+ self.max_intermediate_count = MAX_INTERMEDIATE_OFFSETS_PER_REQUEST * max_requests
+ self._intermediate_chunk_indices_buffer = torch.zeros(
+ self.max_intermediate_count, dtype=torch.int64, device=self.device
+ )
+ self._intermediate_abs_positions_buffer = torch.full(
+ (self.max_intermediate_count,), d_conv, dtype=torch.int32, device=self.device
+ )
+ # Constant gather offsets for conv state extraction: [-d_conv, ..., -1]
+ if d_conv > 0:
+ self.conv_gather_offsets = torch.arange(
+ -d_conv, 0, dtype=torch.int32, device=self.device
+ )
+ else:
+ self.conv_gather_offsets = None
+
+ self.reset_varlen_metadata()
+
def reset(self) -> None:
"""
Resets all Mamba states and frees all allocated slots.
@@ -112,11 +127,26 @@ def reset_varlen_metadata(self) -> None:
"""Resets varlen metadata."""
self.batch_indices_decode = None
self.batch_indices_prefill = None
- self.batch_indices_chunked_prefill = None
self.cu_seqlens = None
self.seq_idx = None
self.device_decode_prefill = None
- self.device_chunked_prefill = None
+
+ # SSM/conv1d precomputed views
+ self.cu_chunk_seqlens = None
+ self.last_chunk_indices = None
+ self.seq_idx_for_varlen = None
+ self.conv_seq_idx = None
+ self.conv_seq_start = None
+
+ # Python-side precomputed values
+ self.real_prefill_token_count = 0
+ self.cu_seqlens_list = [0]
+
+ # Intermediate state extraction views
+ self.intermediate_chunk_indices = None
+ self.intermediate_abs_positions = None
+ self.intermediate_count = 0
+ self.per_request_intermediate_counts = []
def update(
self,
@@ -125,6 +155,9 @@ def update(
cu_seqlens: torch.Tensor,
batch_dimensions: InferenceBatchDimensions,
padded_batch_dimensions: InferenceBatchDimensions,
+ enable_chunked_prefill: bool,
+ intermediate_offsets_gpu: Optional[torch.Tensor] = None,
+ intermediate_counts_gpu: Optional[torch.Tensor] = None,
) -> None:
"""
Updates the dedicated CUDA graph mapping tensor with the indices
@@ -133,20 +166,21 @@ def update(
Args:
active_mamba_indices (Tensor): Tensor containing the Mamba slot indices
for active requests.
- num_active_requests (int): The number of active requests.
+ token_to_request_idx (Tensor): Map from token index to request index.
+ cu_seqlens (Tensor): Cumulative sequence lengths.
+ batch_dimensions (InferenceBatchDimensions): Dimensions of the current batch.
+ padded_batch_dimensions (InferenceBatchDimensions): Dimensions of the padded batch.
+ intermediate_offsets_gpu (Tensor): [prefill_count, 3] int32 GPU tensor of
+ per-request intermediate token offsets, or None.
+ intermediate_counts_gpu (Tensor): [prefill_count] int32 GPU tensor of
+ per-request intermediate offset counts, or None.
"""
real_decode_count = batch_dimensions.decode_req_count
real_prefill_count = batch_dimensions.prefill_req_count
- real_token_count = batch_dimensions.token_count
- has_explicit_chunked_prefill_req = batch_dimensions.has_explicit_chunked_prefill_req
padded_decode_count = padded_batch_dimensions.decode_req_count
padded_prefill_count = padded_batch_dimensions.prefill_req_count
padded_token_count = padded_batch_dimensions.token_count
- assert (
- has_explicit_chunked_prefill_req
- == padded_batch_dimensions.has_explicit_chunked_prefill_req
- )
if padded_decode_count > 0:
# Update decode indices
@@ -157,82 +191,243 @@ def update(
self._batch_indices_decode_buffer[real_decode_count:padded_decode_count] = -1
self.batch_indices_decode = self._batch_indices_decode_buffer[:padded_decode_count]
- # Determine if we have a chunked prefill request and adjust counts for regular prefill
- regular_prefill_count = real_prefill_count
- if has_explicit_chunked_prefill_req:
- # The last prefill request is the chunked one
- regular_prefill_count -= 1
- chunked_req_idx = real_decode_count + regular_prefill_count
-
- # Update chunked prefill indices
- self._batch_indices_chunked_prefill_buffer[0] = active_mamba_indices[chunked_req_idx]
- self.batch_indices_chunked_prefill = self._batch_indices_chunked_prefill_buffer
- else:
- self.batch_indices_chunked_prefill = None
-
if padded_prefill_count > 0:
- # Update prefill indices (excluding chunked prefill from regular prefill buffer)
- if regular_prefill_count > 0:
- self._batch_indices_prefill_buffer[:regular_prefill_count].copy_(
- active_mamba_indices[
- real_decode_count : real_decode_count + regular_prefill_count
- ]
+ # Update prefill indices (all prefill requests go through varlen)
+ if real_prefill_count > 0:
+ prefill_start_idx = real_decode_count
+ self._batch_indices_prefill_buffer[:real_prefill_count].copy_(
+ active_mamba_indices[prefill_start_idx : prefill_start_idx + real_prefill_count]
)
- if padded_prefill_count > regular_prefill_count:
- self._batch_indices_prefill_buffer[regular_prefill_count:padded_prefill_count] = -1
+ if padded_prefill_count > real_prefill_count:
+ self._batch_indices_prefill_buffer[real_prefill_count:padded_prefill_count] = -1
self.batch_indices_prefill = self._batch_indices_prefill_buffer[:padded_prefill_count]
- # Update seq_idx
- end_regular_prefill_token_idx = cu_seqlens[real_decode_count + regular_prefill_count]
+ # Update seq_idx for all prefill requests
+ prefill_start_req_idx = real_decode_count
+ end_prefill_req_idx = real_decode_count + real_prefill_count
+
+ start_prefill_token_idx = cu_seqlens[prefill_start_req_idx]
+ end_prefill_token_idx = cu_seqlens[end_prefill_req_idx]
- # The length of tokens belonging to regular prefill requests (excluding decode tokens)
- seq_len = end_regular_prefill_token_idx - real_decode_count
+ seq_len = end_prefill_token_idx - start_prefill_token_idx
if seq_len > 0:
+ # Normalize request IDs to 0-based relative to prefill requests
self._seq_idx_buffer[:, :seq_len].copy_(
- token_to_request_idx[real_decode_count:end_regular_prefill_token_idx]
- - real_decode_count
+ token_to_request_idx[start_prefill_token_idx:end_prefill_token_idx]
+ - token_to_request_idx[start_prefill_token_idx]
)
if padded_token_count > seq_len:
self._seq_idx_buffer[:, seq_len:padded_token_count] = -1
self.seq_idx = self._seq_idx_buffer[:, :padded_token_count]
- # Update cu_seqlens
+ # Update cu_seqlens for all prefill requests
self._cu_seqlens_buffer[0] = 0
- if regular_prefill_count > 0:
- self._cu_seqlens_buffer[1 : regular_prefill_count + 1].copy_(
- cu_seqlens[
- real_decode_count + 1 : real_decode_count + regular_prefill_count + 1
- ]
- - real_decode_count
+ if real_prefill_count > 0:
+ self._cu_seqlens_buffer[1 : real_prefill_count + 1].copy_(
+ cu_seqlens[prefill_start_req_idx + 1 : end_prefill_req_idx + 1]
+ - cu_seqlens[prefill_start_req_idx]
)
# Pad the rest with the last value (effectively length 0 segments)
- last_val = self._cu_seqlens_buffer[regular_prefill_count]
- self._cu_seqlens_buffer[regular_prefill_count + 1 : padded_prefill_count + 1].fill_(
+ last_val = self._cu_seqlens_buffer[real_prefill_count]
+ self._cu_seqlens_buffer[real_prefill_count + 1 : padded_prefill_count + 1].fill_(
last_val
)
self.cu_seqlens = self._cu_seqlens_buffer[: padded_prefill_count + 1]
+ # --- Precompute SSM and conv1d metadata for CUDA graph compatibility ---
+ # All values the forward pass needs are computed here (before CUDA graph
+ # capture/replay) so that the forward pass has no .item() calls or
+ # data-dependent control flow.
+
+ # Transfer cu_seqlens to CPU for Python-side precomputation
+ cu_seqlens_real = self._cu_seqlens_buffer[: real_prefill_count + 1].tolist()
+ self.cu_seqlens_list = cu_seqlens_real
+ self.real_prefill_token_count = (
+ cu_seqlens_real[real_prefill_count] if real_prefill_count > 0 else 0
+ )
+
+ # Build cu_chunk_seqlens, last_chunk_indices, seq_idx_for_varlen.
+ # Covers all padded sequences (real + padding). Each sequence is
+ # subdivided into chunks of at most mamba_chunk_size tokens. Zero-length
+ # sequences get a single zero-length chunk.
+ cu_seqlens_all = self._cu_seqlens_buffer[: padded_prefill_count + 1].tolist()
+ chunk_size = self.mamba_chunk_size
+ chunk_boundaries = [0]
+ last_chunk_idx_list = []
+ chunk_to_seq_list = []
+
+ for i in range(padded_prefill_count):
+ start = cu_seqlens_all[i]
+ end = cu_seqlens_all[i + 1]
+ seq_len = end - start
+ n_chunks = max(1, (seq_len + chunk_size - 1) // chunk_size)
+ boundaries = [min(start + (k + 1) * chunk_size, end) for k in range(n_chunks)]
+ chunk_boundaries.extend(boundaries)
+ chunk_to_seq_list.extend([i] * n_chunks)
+ last_chunk_idx_list.append(len(chunk_boundaries) - 2)
+
+ # Pad to fixed size for CUDA graph compatibility
+ padded_max_chunks = padded_token_count // chunk_size + padded_prefill_count
+ last_boundary = chunk_boundaries[-1]
+ pad_b = padded_max_chunks + 1 - len(chunk_boundaries)
+ if pad_b > 0:
+ chunk_boundaries.extend([last_boundary] * pad_b)
+ pad_s = padded_max_chunks - len(chunk_to_seq_list)
+ if pad_s > 0:
+ chunk_to_seq_list.extend([0] * pad_s)
+
+ # Fill GPU buffers
+ n_cu = padded_max_chunks + 1
+ self._cu_chunk_seqlens_buffer[:n_cu].copy_(
+ torch.tensor(chunk_boundaries[:n_cu], dtype=torch.int32)
+ )
+ self.cu_chunk_seqlens = self._cu_chunk_seqlens_buffer[:n_cu]
+
+ self._last_chunk_indices_buffer[:padded_prefill_count].copy_(
+ torch.tensor(last_chunk_idx_list, dtype=torch.int32)
+ )
+ self.last_chunk_indices = self._last_chunk_indices_buffer[:padded_prefill_count]
+
+ self._seq_idx_for_varlen_buffer[:padded_max_chunks].copy_(
+ torch.tensor(chunk_to_seq_list[:padded_max_chunks], dtype=torch.int32)
+ )
+ self.seq_idx_for_varlen = self._seq_idx_for_varlen_buffer[:padded_max_chunks]
+
+ # Build conv1d per-token metadata (request ID and request start position)
+ real_tokens = self.real_prefill_token_count
+ if real_tokens > 0:
+ cu = self._cu_seqlens_buffer[: real_prefill_count + 1]
+ lengths = (cu[1:] - cu[:-1]).to(torch.int64)
+ seq_indices = torch.arange(
+ real_prefill_count, dtype=torch.int32, device=self.device
+ )
+ seq_starts = cu[:real_prefill_count].to(torch.int32)
+ self._conv_seq_idx_buffer[:real_tokens] = torch.repeat_interleave(
+ seq_indices, lengths
+ )
+ self._conv_seq_start_buffer[:real_tokens] = torch.repeat_interleave(
+ seq_starts, lengths
+ )
+ if padded_token_count > real_tokens:
+ self._conv_seq_idx_buffer[real_tokens:padded_token_count] = 0
+ self._conv_seq_start_buffer[real_tokens:padded_token_count] = 0
+
+ self.conv_seq_idx = self._conv_seq_idx_buffer[:padded_token_count]
+ self.conv_seq_start = self._conv_seq_start_buffer[:padded_token_count]
+
+ # --- Precompute intermediate state extraction metadata ---
+ # This converts per-request token offsets to chunk indices and
+ # absolute positions, padded to fixed size for CUDA graph compat.
+ self._update_intermediate_metadata(
+ intermediate_offsets_gpu, intermediate_counts_gpu, real_prefill_count
+ )
+
if padded_decode_count > 0 and padded_prefill_count > 0:
- self._device_decode_prefill_buffer[0] = real_decode_count
- self._device_decode_prefill_buffer[1] = regular_prefill_count
+ self._device_decode_prefill_buffer[0] = cu_seqlens[real_decode_count]
+ self._device_decode_prefill_buffer[1] = (
+ cu_seqlens[real_decode_count + real_prefill_count] - cu_seqlens[real_decode_count]
+ )
self.device_decode_prefill = self._device_decode_prefill_buffer
- # If using chunked prefill for this batch, store the number of regular prefill tokens
- # and the number of tokens in the chunked prefill request
- if has_explicit_chunked_prefill_req:
- chunked_prefill_token_count = (
- cu_seqlens[real_decode_count + real_prefill_count]
- - cu_seqlens[real_decode_count + real_prefill_count - 1]
- )
- assert self.cu_seqlens is not None
- self._device_chunked_prefill_buffer[0] = self.cu_seqlens[regular_prefill_count]
- self._device_chunked_prefill_buffer[1] = chunked_prefill_token_count
- self.device_chunked_prefill = self._device_chunked_prefill_buffer
+ def _update_intermediate_metadata(
+ self,
+ intermediate_offsets_gpu: Optional[torch.Tensor],
+ intermediate_counts_gpu: Optional[torch.Tensor],
+ real_prefill_count: int,
+ ) -> None:
+ """Precompute intermediate extraction metadata for CUDA graph compatibility.
+
+ Converts per-request token offsets to chunk indices and absolute
+ positions using vectorized GPU operations, padding unused entries
+ to fixed buffer size.
+
+ Args:
+ intermediate_offsets_gpu: [real_prefill_count, 3] int32 GPU tensor
+ of per-request token offsets, or None if no extraction needed.
+ intermediate_counts_gpu: [real_prefill_count] int32 GPU tensor of
+ per-request offset counts (0-3), or None.
+ real_prefill_count: Number of real (non-padding) prefill requests.
+ """
+ chunk_size = self.mamba_chunk_size
+ max_count = self.max_intermediate_count
+
+ if intermediate_offsets_gpu is not None and real_prefill_count > 0:
+ # Transfer counts to CPU (single sync) for per_request_counts and total check
+ counts_list = intermediate_counts_gpu.tolist()
+ total = sum(counts_list)
+
+ if total > 0:
+ # Compute cumulative chunk counts from cu_seqlens (already on GPU)
+ cu = self._cu_seqlens_buffer[: real_prefill_count + 1]
+ seq_lens = (cu[1 : real_prefill_count + 1] - cu[:real_prefill_count]).to(
+ torch.int64
+ )
+ num_chunks = torch.clamp((seq_lens + chunk_size - 1) // chunk_size, min=1)
+ cum_chunks = torch.zeros(
+ real_prefill_count + 1, dtype=torch.int64, device=self.device
+ )
+ torch.cumsum(num_chunks, dim=0, out=cum_chunks[1:])
+
+ seq_starts = cu[:real_prefill_count].to(torch.int64)
+ offsets = intermediate_offsets_gpu.to(torch.int64)
+
+ # Expand per-request values to [real_prefill_count, 3]
+ cum_chunks_exp = cum_chunks[:real_prefill_count].unsqueeze(1).expand_as(offsets)
+ seq_starts_exp = seq_starts.unsqueeze(1).expand_as(offsets)
+
+ # Vectorized computation of chunk indices and absolute positions
+ chunk_indices_2d = cum_chunks_exp + offsets // chunk_size - 1
+ abs_positions_2d = seq_starts_exp + offsets
+
+ # Validity mask: j < count[i] for each request
+ j_indices = torch.arange(
+ MAX_INTERMEDIATE_OFFSETS_PER_REQUEST, device=self.device
+ ).unsqueeze(0)
+ valid_mask = j_indices < intermediate_counts_gpu.unsqueeze(1)
+
+ # Flatten valid entries into output buffers
+ valid_chunk_indices = chunk_indices_2d[valid_mask]
+ valid_abs_positions = abs_positions_2d[valid_mask]
+
+ real_count = valid_chunk_indices.numel()
+ self._intermediate_chunk_indices_buffer[:real_count] = valid_chunk_indices
+ self._intermediate_abs_positions_buffer[:real_count] = valid_abs_positions.to(
+ torch.int32
+ )
+
+ # Pad unused slots with safe defaults for CUDA graph replay:
+ # - chunk_indices=0: reads from chunk 0 (always exists), output ignored
+ # - abs_positions=d_conv: conv gather reads tokens [0..d_conv-1],
+ # which are within bounds and produce a valid but unused state
+ if real_count < max_count:
+ self._intermediate_chunk_indices_buffer[real_count:].fill_(0)
+ self._intermediate_abs_positions_buffer[real_count:].fill_(self.d_conv)
+
+ self.intermediate_count = real_count
+ self.per_request_intermediate_counts = counts_list
+ else:
+ # All counts are 0
+ self._intermediate_chunk_indices_buffer.fill_(0)
+ self._intermediate_abs_positions_buffer.fill_(self.d_conv)
+ self.intermediate_count = 0
+ self.per_request_intermediate_counts = counts_list
+
+ self.intermediate_chunk_indices = self._intermediate_chunk_indices_buffer[:max_count]
+ self.intermediate_abs_positions = self._intermediate_abs_positions_buffer[:max_count]
+ else:
+ # No extraction: fill with safe defaults for CUDA graph warmup
+ # (same rationale as padding comment above)
+ self._intermediate_chunk_indices_buffer.fill_(0)
+ self._intermediate_abs_positions_buffer.fill_(self.d_conv)
+ self.intermediate_count = 0
+ self.per_request_intermediate_counts = []
+ self.intermediate_chunk_indices = self._intermediate_chunk_indices_buffer[:max_count]
+ self.intermediate_abs_positions = self._intermediate_abs_positions_buffer[:max_count]
def allocate_slot(self) -> Optional[int]:
"""
@@ -251,6 +446,25 @@ def allocate_slot(self) -> Optional[int]:
return mamba_idx
+ def batch_allocate_slots(self, num_slots: int) -> Optional[torch.Tensor]:
+ """
+ Allocates new slots for the given number of requests in the Mamba state buffers.
+
+ Returns:
+ torch.Tensor: The indices of the allocated slots.
+ Returns None if not enough slots are available.
+ """
+ if self.mamba_state_free_slot_count < num_slots:
+ return None
+
+ # Get free slots
+ self.mamba_state_free_slot_count -= num_slots
+ mamba_idx = self.mamba_state_free_slots[
+ self.mamba_state_free_slot_count : self.mamba_state_free_slot_count + num_slots
+ ]
+
+ return mamba_idx
+
def free_slots(self, request_indices: torch.Tensor) -> None:
"""
Frees the Mamba state slots associated with the given request indices.
diff --git a/megatron/core/inference/contexts/attention_context/mha_metadata.py b/megatron/core/inference/contexts/attention_context/mha_metadata.py
index 1b6e8020275..07f8a349b51 100644
--- a/megatron/core/inference/contexts/attention_context/mha_metadata.py
+++ b/megatron/core/inference/contexts/attention_context/mha_metadata.py
@@ -41,6 +41,7 @@ def update(
request_to_kv_block_ids: torch.Tensor,
batch_dimensions: InferenceBatchDimensions,
padded_batch_dimensions: InferenceBatchDimensions,
+ num_speculative_tokens: int = 0,
):
"""
Args:
@@ -49,6 +50,7 @@ def update(
request_to_kv_block_ids: (>real_batch_size, max_kv_blocks)
batch_dimensions: Configuration object containing real batch settings
padded_batch_dimensions: Configuration object containing padded batch settings
+ num_speculative_tokens: Number of speculative tokens
"""
# Extract values from configs
real_batch_size = batch_dimensions.req_count
@@ -99,7 +101,7 @@ def update(
)
if padded_batch_dimensions.prefill_req_count == 0:
- self._max_seqlen_q = 1
+ self._max_seqlen_q = num_speculative_tokens + 1
else:
# Make sure we will launch the prefill kernel for prefill graphs
self._max_seqlen_q = max(2, padded_batch_dimensions.token_count)
@@ -150,6 +152,7 @@ def update(
request_to_kv_block_ids: torch.Tensor,
batch_dimensions: InferenceBatchDimensions,
padded_batch_dimensions: InferenceBatchDimensions,
+ num_speculative_tokens: int = 0,
):
"""
Args:
@@ -158,6 +161,7 @@ def update(
request_to_kv_block_ids: (>real_batch_size, max_kv_blocks)
batch_dimensions: Configuration object containing real batch settings
padded_batch_dimensions: Configuration object containing padded batch settings
+ num_speculative_tokens: Number of speculative tokens
"""
super().update(
request_query_lengths,
@@ -165,6 +169,7 @@ def update(
request_to_kv_block_ids,
batch_dimensions,
padded_batch_dimensions,
+ num_speculative_tokens,
)
def reset(self):
@@ -183,6 +188,7 @@ def update(
request_to_kv_block_ids: torch.Tensor,
batch_dimensions: InferenceBatchDimensions,
padded_batch_dimensions: InferenceBatchDimensions,
+ num_speculative_tokens: int = 0,
):
"""
Args:
@@ -191,6 +197,7 @@ def update(
request_to_kv_block_ids: (>real_batch_size, max_kv_blocks)
batch_dimensions: Configuration object containing real batch settings
padded_batch_dimensions: Configuration object containing padded batch settings
+ num_speculative_tokens: Number of speculative tokens
"""
super().update(
request_query_lengths,
@@ -198,10 +205,11 @@ def update(
request_to_kv_block_ids,
batch_dimensions,
padded_batch_dimensions,
+ num_speculative_tokens,
)
if len(self.state_data["query_lengths"]) > 0:
self.state_data["max_seqlen_q"] = torch.max(self.state_data["query_lengths"]).item()
self.state_data["max_seqlen_k"] = torch.max(self.state_data["kv_seq_lengths"]).item()
else:
- self.state_data["max_seqlen_q"] = 1
+ self.state_data["max_seqlen_q"] = num_speculative_tokens + 1
self.state_data["max_seqlen_k"] = 1
diff --git a/megatron/core/inference/contexts/attention_context/triton/tensor_ops.py b/megatron/core/inference/contexts/attention_context/triton/tensor_ops.py
index 2f3210488f5..f48b270826b 100644
--- a/megatron/core/inference/contexts/attention_context/triton/tensor_ops.py
+++ b/megatron/core/inference/contexts/attention_context/triton/tensor_ops.py
@@ -3,8 +3,21 @@
from typing import Optional
import torch
-import triton # type: ignore
-import triton.language as tl # type: ignore
+
+try:
+ import triton
+ import triton.language as tl
+
+ HAVE_TRITON = True
+except ImportError:
+ from unittest.mock import MagicMock
+
+ from megatron.core.utils import null_decorator
+
+ triton = MagicMock()
+ triton.jit = null_decorator
+ tl = MagicMock()
+ HAVE_TRITON = False
@triton.jit
diff --git a/megatron/core/inference/contexts/base_context.py b/megatron/core/inference/contexts/base_context.py
index 3dfec6de3ad..4f03726fe3d 100644
--- a/megatron/core/inference/contexts/base_context.py
+++ b/megatron/core/inference/contexts/base_context.py
@@ -2,6 +2,8 @@
import abc
+from megatron.core.inference.config import InferenceConfig
+
class BaseInferenceContext(abc.ABC):
"""Base class for inference contexts.
@@ -10,13 +12,11 @@ class BaseInferenceContext(abc.ABC):
Extend this class for any future contexts types.
"""
- def __init__(self, materialize_only_last_token_logits: bool):
+ def __init__(self, inference_config: InferenceConfig):
"""
Args:
- materialize_only_last_token_logits (bool):
- If True, only the last-token logits will be extracted during decode
"""
- self.materialize_only_last_token_logits = materialize_only_last_token_logits
+ self.config = inference_config
@abc.abstractmethod
def is_static_batching(self) -> bool:
diff --git a/megatron/core/inference/contexts/dynamic_block_allocator.py b/megatron/core/inference/contexts/dynamic_block_allocator.py
deleted file mode 100644
index 8207a17550c..00000000000
--- a/megatron/core/inference/contexts/dynamic_block_allocator.py
+++ /dev/null
@@ -1,135 +0,0 @@
-# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
-
-from typing import Optional
-
-import torch
-from torch import Tensor
-
-
-class BlockAllocator:
- """Allocator that manages blocks of memory for the KV cache.
-
- This allocator is responsible for:
- - Initializing a pool of block IDs
- - Allocating blocks from the pool
- - Releasing blocks back to the pool
-
- Args:
- context (DynamicInferenceContext): Dynamic inference context.
- total_count (int): Total number of blocks in the buffer.
- paused_count (int): Number of paused blocks in the buffer. Must be less
- than `total_count`.
- """
-
- def __init__(self, context: "DynamicInferenceContext", total_count: int, paused_count: int):
-
- self.context = context
-
- self.total_count = total_count
- self.total_avail = total_count - 1 # -1 for dummy_block_idx (see below)
- self.paused_count = paused_count
- self.active_count = total_count - paused_count - 1 # -1 for dummy_block_idx
- assert self.active_count >= 1 # ensures paused_count < total_count - 1
- self.dummy_block_idx = self.total_count - 1
-
- # Initialize block pool as a "stack" data structure
- self.block_bag = torch.arange(
- self.total_count, dtype=torch.int32, device=torch.cuda.current_device()
- )
-
- def __str__(self):
- return (
- f"using: total {self.get_total_used()}/{self.total_count - 1}"
- f"; active {self.get_active_used()}/{self.active_count}"
- f"; paused {self.get_paused_used()}/{self.paused_count}"
- )
-
- def get_total_used(self):
- """Compute number of total blocks used."""
- return self.total_count - self.total_avail - 1
-
- def get_active_used(self):
- """Compute number of active blocks used."""
- return (
- self.context.request_kv_block_counts[
- self.context.paused_request_count : self.context.total_request_count
- ]
- .sum()
- .item()
- )
-
- def get_paused_used(self):
- """Compute number of paused blocks used."""
- return (
- self.context.request_kv_block_counts[: self.context.paused_request_count].sum().item()
- )
-
- def get_active_avail(self):
- """Compute number of active blocks available."""
- return self.active_count - self.get_active_used()
-
- def get_paused_avail(self):
- """Compute number of paused blocks available."""
- return self.paused_count - self.get_paused_used()
-
- def is_memory_available(self, num_blocks: int) -> bool:
- """Check if memory blocks are available.
-
- Args:
- num_blocks (int): Number of blocks to check.
-
- Return:
- (bool) Is memory available?
- """
- return self.total_avail >= num_blocks
-
- def allocate_memory_blocks(self, num_blocks: int) -> Optional[Tensor]:
- """Allocate memory blocks if available, else return None.
-
- Args:
- num_blocks (int): Number of blocks to allocate.
-
- Return:
- (Optional[Tensor]) Allocated block IDs.
- """
- if self.is_memory_available(num_blocks):
- self.total_avail -= num_blocks
- block_ids = self.block_bag[self.total_avail : (self.total_avail + num_blocks)]
- assert num_blocks == block_ids.numel()
- return block_ids
- else:
- return None
-
- def release_memory_blocks(self, blocks: Tensor) -> None:
- """Release memory blocks.
-
- Args:
- blocks (Tensor): Block IDs to release.
-
- Return:
- None
- """
- num_blocks = blocks.size(dim=0)
- self.block_bag[self.total_avail : (self.total_avail + num_blocks)] = blocks
- self.total_avail += num_blocks
-
- def reset(self) -> None:
- """Reset the allocator to initial state.
-
- This resets the available block count to the entire memory pool
- (except for the dummy block).
- """
-
- # Reset block bag to so we start consuming from the beginning of the pool
- # for UVM performance.
- # *Note*: Resetting the block bag is essential because if engine has been
- # suspended, then the block bag contains non-unique IDs since the
- # right-most IDs have been 'popped' off and are owned by the context.
- # Without resetting the block bag, context request memory will clash and
- # requests will point to each other's memory blocks, resulting in faulty
- # generations.
- self.block_bag = torch.arange(
- self.total_count, dtype=torch.int32, device=torch.cuda.current_device()
- )
-
- self.total_avail = self.total_count - 1
diff --git a/megatron/core/inference/contexts/dynamic_context.py b/megatron/core/inference/contexts/dynamic_context.py
index 5dc2d503097..1117f2b9c4b 100644
--- a/megatron/core/inference/contexts/dynamic_context.py
+++ b/megatron/core/inference/contexts/dynamic_context.py
@@ -4,40 +4,43 @@
import math
import warnings
from contextlib import nullcontext
-from typing import TYPE_CHECKING, List, Optional, Sequence, Tuple
+from typing import List, Optional, Sequence, Tuple
-import torch
-import torch.nn.functional as F
-from packaging.version import Version as PkgVersion
-from torch import Tensor
+import torch # type: ignore
+import torch.nn.functional as F # type: ignore
+from torch import Tensor # type: ignore
from megatron.core import parallel_state
from megatron.core.inference.batch_dimensions_utils import (
CUDAGraphBatchDimensionBuilder,
InferenceBatchDimensions,
)
-from megatron.core.inference.inference_request import DynamicInferenceRequest
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
+from megatron.core.inference.config import (
+ InferenceConfig,
+ KVCacheManagementMode,
+ PrefixCachingEvictionPolicy,
)
+from megatron.core.inference.inference_request import DynamicInferenceRequest
from megatron.core.inference.sampling_params import SamplingParams
from megatron.core.inference.unified_memory import (
UnifiedMemoryUnsupportedError,
create_unified_mempool,
)
-from megatron.core.inference.utils import tensor_swap
+from megatron.core.inference.utils import device_memory_summary, tensor_swap
from megatron.core.models.common.embeddings.rope_utils import apply_rotary_pos_emb
from megatron.core.package_info import __version__ as mcore_version
-from megatron.core.process_groups_config import ProcessGroupCollection
from megatron.core.ssm.mamba_hybrid_layer_allocation import get_layer_maps_from_layer_type_list
-from megatron.core.transformer import TransformerConfig
+from megatron.core.transformer import MLATransformerConfig, TransformerConfig
+from megatron.core.utils import deprecate_args
from megatron.core.utils import divide as core_divide
-from megatron.core.utils import get_attr_wrapped_model, get_pg_size, internal_api
+from megatron.core.utils import get_pg_size, internal_api
-from .attention_context.mamba_metadata import MambaInferenceStateConfig, MambaMetadata
+from .attention_context.mamba_metadata import MambaMetadata
from .attention_context.mha_metadata import GraphedMHAMetadata, NonGraphedMHAMetadata
from .base_context import BaseInferenceContext
-from .dynamic_block_allocator import BlockAllocator
+from .kv_block_allocator import KVBlockAllocator
+from .mamba_slot_allocator import MambaSlotAllocator
+from .routing_metadata import RoutingMetadata
try:
from .fused_kv_append_kernel import triton_append_key_value_cache
@@ -45,14 +48,7 @@
triton_append_key_value_cache = None
try:
- from packaging.version import Version as PkgVersion
-
- HAVE_PACKAGING = True
-except:
- HAVE_PACKAGING = False
-
-try:
- import flashinfer # pylint: disable=unused-import
+ import flashinfer # type: ignore # pylint: disable=unused-import
HAVE_FLASHINFER = True
except ImportError:
@@ -66,16 +62,36 @@
except ImportError:
HAVE_TORCH_MEMORY_SAVER = False
-try:
- import wandb # pylint: disable=unused-import
-
- HAVE_WANDB = True
-except ImportError:
- HAVE_WANDB = False
- wandb = None
-
-if TYPE_CHECKING:
- import wandb as WandbModule
+DEPRECATED_ARGS = [
+ "params_dtype",
+ "num_layers",
+ "kv_channels",
+ "num_attention_heads",
+ "max_sequence_length",
+ "buffer_size_gb",
+ "paused_buffer_size_gb",
+ "max_requests",
+ "max_tokens",
+ "block_size_tokens",
+ "tensor_model_parallel_size",
+ "pipeline_model_parallel_size",
+ "pg_collection",
+ "cache_mla_latent",
+ "kv_lora_rank",
+ "qk_pos_emb_head_dim",
+ "num_cuda_graphs",
+ "materialize_only_last_token_logits",
+ "mamba_inference_state_config",
+ "use_cuda_graphs_for_non_decode_steps",
+ "use_flashinfer_fused_rope",
+ "unified_memory_level",
+ "cuda_graph_max_tokens",
+ "cuda_graph_mixed_prefill_count",
+ "metrics_writer",
+ "request_metadata_types",
+ "persist_cuda_graphs",
+ "offload_kv_cache",
+]
class ContextOverflowError(Exception):
@@ -213,129 +229,74 @@ class DynamicInferenceContext(BaseInferenceContext):
given step, any unassigned blocks equate to unused space.
Args:
- params_dtype (torch.dtype): Dtype used for KV cache.
- num_layers (int): Number of layers on this pipeline parallel rank.
- kv_channels (int): Hidden dimension per attention head.
- num_attention_heads (int): Number of attention heads.
- max_sequence_length (int): Max possible sequence length (prompt + output)
- that will occur.
- buffer_size_gb (float): Buffer size reserved on the GPU for the KV cache.
- if `unified_memory_level` >= 1, then CPU memory is additionally
- utilized, resulting in a total buffer size of `buffer_size_gb +
- paused_buffer_size_gb`.
- paused_buffer_size_gb (float | None): Portion of buffer reserved for
- paused requests. Active requests are paused when there are not enough
- active blocks available to continue generating a request. The total
- buffer size (active + paused) depends on `unified_memory_level` (uvm):
- - uvm 0: buffer_size_gb (paused buffer is inclusive)
- - uvm 1: buffer_size_gb + paused_buffer_size_gb
- max_requests (int): Max number of active requests to use for
- decode-only forward passes. This value is primarily limited by the
- combination of `buffer_size_gb` and `max_sequence_length`.
- max_tokens (int): Max number of tokens to use for forward passes. This is
- primarily limited by prefill activation memory usage. (Defaults to
- 16384).
- block_size_tokens (int): Size of KV cache block size.
- tensor_model_parallel_size (Optional[int]): Tensor model parallel size.
- num_cuda_graphs (Optional[int]): Maximum number of cuda graphs to capture,
- where the cuda graph batch sizes range from 1 to `max_requests`
- (as computed below). Due to rounding, the actual number of cuda graphs
- may not equal this argument.
- materialize_only_last_token_logits (Optional[bool]): Whether to only
- materialize logits for the last token. This should be set to False
- if returning log probs.
- mamba_inference_state_config (Optional[MambaInferenceStateConfig]): The Mamba
- inference state config if the model is a hybrid model.
- use_cuda_graphs_for_non_decode_steps (bool): If True, use cuda graphs for non-decode
- engine steps.
- unified_memory_level (Optional[int]): Set unified memory usage within the
- dynamic inference context. The levels are: 0) no unified memory, 1)
- allocate `memory_buffer` in unified memory. Eventually, additional
- levels will be included to control other tensors within the context.
- use_flashinfer_fused_rope (bool): If True, use flashinfer's fused rope implementation.
- If None, defaults to using flash-infer if available.
- metrics_writer (Optional['WandbModule']): Wandb module for writing metrics.
- request_metadata_types (Optional[List[Tuple[str, torch.dtype, bool]]]): A list of the
- per-request metadata types to track. Each entry is a tuple consisting of the string
- label, the target dtype, and whether to store the data on GPU.
+ model_config (TransformerConfig): Model config.
+ inference_config (InferenceConfig): Inference config.
"""
DEFAULT_MAX_TOKENS = 16384
TOKEN_ROUNDER = 64
REQUEST_ROUNDER = 4
-
- def __init__(
- self,
- *,
- params_dtype: torch.dtype,
- num_layers: int,
- kv_channels: int,
- num_attention_heads: int,
- max_sequence_length: int,
- buffer_size_gb: float,
- paused_buffer_size_gb: float | None = None,
- max_requests: int = None,
- max_tokens: int = DEFAULT_MAX_TOKENS,
- block_size_tokens: int = 256,
- tensor_model_parallel_size: Optional[int] = None,
- pipeline_model_parallel_size: Optional[int] = None,
- pg_collection: Optional[ProcessGroupCollection] = None,
- cache_mla_latent: bool = False,
- kv_lora_rank: Optional[int] = None,
- qk_pos_emb_head_dim: Optional[int] = None,
- num_cuda_graphs: Optional[int] = None,
- materialize_only_last_token_logits: Optional[bool] = True,
- mamba_inference_state_config: Optional[MambaInferenceStateConfig] = None,
- use_cuda_graphs_for_non_decode_steps: bool = True,
- use_flashinfer_fused_rope: bool = False,
- unified_memory_level: Optional[int] = 0,
- cuda_graph_max_tokens: Optional[int] = None,
- cuda_graph_mixed_prefill_count: Optional[int] = 16,
- metrics_writer: Optional['WandbModule'] = None,
- request_metadata_types: Optional[List[Tuple[str, torch.dtype, bool]]] = None,
- persist_cuda_graphs: Optional[bool] = False,
- offload_kv_cache: Optional[bool] = False,
- ):
- super().__init__(materialize_only_last_token_logits=materialize_only_last_token_logits)
-
- self.cache_mla_latent = cache_mla_latent
+ TMS_TAG = "inference_context"
+
+ @deprecate_args(
+ *DEPRECATED_ARGS,
+ message=(
+ "Argument `{name}` has been deprecated. "
+ "Only pass `model_config` and `inference_config`"
+ ),
+ )
+ def __init__(self, model_config: TransformerConfig, inference_config: InferenceConfig):
+ super().__init__(inference_config=inference_config)
+
+ # Prefix caching configuration
+ self.enable_prefix_caching = inference_config.enable_prefix_caching
+ self.prefix_caching_eviction_policy = inference_config.prefix_caching_eviction_policy
+ self.prefix_caching_coordinator_policy = inference_config.prefix_caching_coordinator_policy
+
+ # Hyperparameter for choosing to prioritize prefix hit matches vs minimizing idle load
+ self.prefix_caching_routing_alpha = inference_config.prefix_caching_routing_alpha
+
+ # Monotonic clock for prefix caching LRU eviction ordering.
+ # Incremented each engine step but kept independent so the engine step
+ # counter is not overloaded with cache-eviction semantics.
+ self.prefix_cache_lru_clock = 0
+
+ # Prefix caching hit tracking (accumulated, reset by engine after logging).
+ self.prefix_cache_hits = 0 # requests that matched at least one cached block
+ self.prefix_cache_blocks_matched = 0 # total matched blocks across all requests
+
+ # Engine step counter (used for logging, metrics, and event tracking)
+ self.step_count = 0
+
+ self.cache_mla_latent = (
+ isinstance(model_config, MLATransformerConfig) and model_config.cache_mla_latents
+ )
if self.cache_mla_latent:
assert (
- block_size_tokens == 64
+ inference_config.block_size_tokens == 64
), "Flash MLA requires a block size of 64. Set --inference-dynamic-batching-block-size 64 to fix this assert"
- # give deprecated args warning for cuda_graph_max_tokens
- if cuda_graph_max_tokens is not None:
- warnings.warn(
- "`cuda_graph_max_tokens` is deprecated and will be removed in a future release. "
- "The context now automatically sets the max tokens for cuda graphs based on "
- "`max_requests`.",
- DeprecationWarning,
- )
-
- self.metrics_writer = metrics_writer
-
# Per partition num heads and hidden size.
- projection_size = kv_channels * num_attention_heads
- if tensor_model_parallel_size is None:
- tp_size = (
- get_pg_size(pg_collection.tp)
- if pg_collection is not None
- else parallel_state.get_tensor_model_parallel_world_size()
- )
+ num_attention_heads = model_config.num_query_groups or model_config.num_attention_heads
+ projection_size = model_config.kv_channels * num_attention_heads
+ pg_collection = inference_config.pg_collection
+ if pg_collection is not None:
+ tp_size = get_pg_size(pg_collection.tp)
+ pp_size = get_pg_size(pg_collection.pp)
else:
- tp_size = tensor_model_parallel_size
+ tp_size = model_config.tensor_model_parallel_size
+ pp_size = model_config.pipeline_model_parallel_size
self.hidden_size_per_attention_head = core_divide(projection_size, num_attention_heads)
- self.num_attention_heads_per_partition = core_divide(num_attention_heads, tp_size)
-
- if pipeline_model_parallel_size is None:
- pp_size = (
- get_pg_size(pg_collection.pp)
- if pg_collection is not None
- else parallel_state.get_pipeline_model_parallel_world_size()
- )
+ if num_attention_heads >= tp_size:
+ self.num_attention_heads_per_partition = core_divide(num_attention_heads, tp_size)
else:
- pp_size = pipeline_model_parallel_size
+ self.num_attention_heads_per_partition = 1
+
+ self.num_speculative_tokens = inference_config.num_speculative_tokens
+ assert self.num_speculative_tokens < inference_config.block_size_tokens, (
+ f"num_speculative_tokens ({self.num_speculative_tokens}) must be < "
+ f"block_size_tokens ({inference_config.block_size_tokens})"
+ )
# Cache the PP group we should use for PP collectives inside the context.
# If the model provides a pg_collection with a pp group, prefer it.
@@ -357,31 +318,31 @@ def __init__(
self.expert_model_parallel_group = None
# Mamba states.
+ mamba_inference_state_config = inference_config.mamba_inference_state_config
self.is_hybrid_model = mamba_inference_state_config is not None
if self.is_hybrid_model:
- mamba_conv_states_shape = mamba_inference_state_config.mamba_conv_states_shape
- mamba_ssm_states_shape = mamba_inference_state_config.mamba_ssm_states_shape
- assert (
- mamba_conv_states_shape is not None
- ), "`mamba_conv_states_shape` must be specified for hybrid models"
- assert (
- mamba_ssm_states_shape is not None
- ), "`mamba_ssm_states_shape` must be specified for hybrid models"
+ self.mamba_conv_states_shape = mamba_inference_state_config.conv_states_shape
+ self.mamba_ssm_states_shape = mamba_inference_state_config.ssm_states_shape
+ self.mamba_conv_states_dtype = mamba_inference_state_config.conv_states_dtype
+ self.mamba_ssm_states_dtype = mamba_inference_state_config.ssm_states_dtype
+ self.mamba_chunk_size = mamba_inference_state_config.mamba_chunk_size
# For hybrid models, the layer map converts the global layer index to the
# corresponding attention layer index or Mamba layer index depending on the
# layer type.
- attention_layer_map, mamba_layer_map, _, _ = get_layer_maps_from_layer_type_list(
- mamba_inference_state_config.layer_type_list
+ mamba_layer_map, gdn_layer_map, attention_layer_map, _, _ = (
+ get_layer_maps_from_layer_type_list(mamba_inference_state_config.layer_type_list)
)
+
+ if len(gdn_layer_map) > 0:
+ raise NotImplementedError("GDN layers are not supported for inference.")
+
self.num_attention_layers = len(attention_layer_map)
self.num_mamba_layers = len(mamba_layer_map)
- self.mamba_conv_states_shape = mamba_conv_states_shape
- self.mamba_ssm_states_shape = mamba_ssm_states_shape
self.layer_map = attention_layer_map | mamba_layer_map
else:
# The layer map is the identity function for pure Transformer models.
- self.num_attention_layers = num_layers
+ self.num_attention_layers = model_config.num_layers // pp_size
self.num_mamba_layers = 0
(self.mamba_conv_states_shape, self.mamba_ssm_states_shape) = (None, None)
self.layer_map = {i: i for i in range(self.num_attention_layers)}
@@ -392,20 +353,20 @@ def __init__(
)
# Block size tokens, bytes.
- dtype_size_bytes = params_dtype.itemsize
- self.block_size_tokens = block_size_tokens
+ kv_dtype_size_bytes = model_config.params_dtype.itemsize
+ self.block_size_tokens = inference_config.block_size_tokens
if self.cache_mla_latent:
# one vector c_t (rank) + optional RoPE phase slice
- self.kv_reduced_dim = kv_lora_rank + qk_pos_emb_head_dim
+ self.kv_reduced_dim = model_config.kv_lora_rank + model_config.qk_pos_emb_head_dim
self.block_size_bytes = (
- dtype_size_bytes
+ kv_dtype_size_bytes
* self.num_attention_layers
* self.block_size_tokens
* self.kv_reduced_dim
)
else:
self.block_size_bytes = (
- dtype_size_bytes
+ kv_dtype_size_bytes
* 2 # key, value
* self.num_attention_layers
* self.block_size_tokens
@@ -416,15 +377,29 @@ def __init__(
mamba_states_memory_per_request = 0
if self.is_hybrid_model:
- mamba_states_memory_per_request += math.prod(self.mamba_conv_states_shape)
- mamba_states_memory_per_request += math.prod(self.mamba_ssm_states_shape)
+ mamba_states_memory_per_request += (
+ math.prod(self.mamba_conv_states_shape) * self.mamba_conv_states_dtype.itemsize
+ )
+ mamba_states_memory_per_request += (
+ math.prod(self.mamba_ssm_states_shape) * self.mamba_ssm_states_dtype.itemsize
+ )
mamba_states_memory_per_request *= self.num_mamba_layers
- mamba_states_memory_per_request *= dtype_size_bytes
+ if self.num_speculative_tokens > 0:
+ # Add memory for intermediate conv and SSM states
+ intermediate_memory_per_request = (
+ math.prod(self.mamba_conv_states_shape) * self.mamba_conv_states_dtype.itemsize
+ + math.prod(self.mamba_ssm_states_shape) * self.mamba_ssm_states_dtype.itemsize
+ )
+ intermediate_memory_per_request *= self.num_mamba_layers
+ intermediate_memory_per_request *= self.num_speculative_tokens + 1
+ mamba_states_memory_per_request += intermediate_memory_per_request
+
+ # Unified memory and general tensor management.
+ self.unified_memory_level = inference_config.unified_memory_level
+ self.static_kv_memory_pointers = inference_config.static_kv_memory_pointers
+ self.kv_cache_management_mode = inference_config.kv_cache_management_mode
- # Unified memory.
- self.unified_memory_level = unified_memory_level
- self.persist_cuda_graphs = persist_cuda_graphs
- if unified_memory_level > 0:
+ if self.unified_memory_level != 0:
try:
self.unified_memory_mempool = create_unified_mempool()
except UnifiedMemoryUnsupportedError:
@@ -433,80 +408,120 @@ def __init__(
"Unified memory requested but not available; defaulting to GPU memory."
)
self.unified_memory_level = 0
+ # If we are in a mode that requires static KV memory pointers,
+ # we must have either UVM or torch_memory_saver.
+ if (
+ self.static_kv_memory_pointers
+ and self.kv_cache_management_mode != KVCacheManagementMode.PERSIST
+ ):
+ assert HAVE_TORCH_MEMORY_SAVER or self.unified_memory_level != 0, (
+ "Static KV memory pointers require UVM or torch_memory_saver when not persisted. "
+ "Use --rl-kv-cache-management-mode=persist, UVM, or install torch_memory_saver."
+ )
+
+ # When not using `torch_memory_saver`, we manually offload/restore tensors.
+ # We use storage resize, similar to the logic in `core/distributed/param_and_grad_buffer.py`
+ self._offloadable_tensor_names: set[str] = set()
+ self._offloadable_cpu_backups: dict[str, torch.Tensor] = {}
+ self._offloadable_storage_sizes: dict[str, int] = {}
+ self._uses_torch_memory_saver: bool = False
# Initialize block allocator.
- buffer_size_bytes = int(buffer_size_gb * 1024**3)
+ buffer_size_bytes = int(inference_config.buffer_size_gb * 1024**3)
paused_buffer_size_bytes = (
- 0 if paused_buffer_size_gb is None else int(paused_buffer_size_gb * 1024**3)
- )
- # TODO: Add parameter to control fraction of memory assigned to KV cache
- # versus Mamba state.
- block_count = buffer_size_bytes // (self.block_size_bytes + mamba_states_memory_per_request)
- block_count = max(2, block_count) # need >= 1 active block + 1 dummy block
- paused_block_count = paused_buffer_size_bytes // (
- self.block_size_bytes + mamba_states_memory_per_request
+ 0
+ if inference_config.paused_buffer_size_gb is None
+ else int(inference_config.paused_buffer_size_gb * 1024**3)
)
+ mamba_max_requests = float('inf')
+
+ if (mamba_memory_ratio := inference_config.mamba_memory_ratio) is not None:
+ assert self.is_hybrid_model
+ assert mamba_memory_ratio > 0 and mamba_memory_ratio < 1
+
+ # Calculate total memory before partition
+ total_memory = buffer_size_bytes + paused_buffer_size_bytes
+ mamba_memory_bytes = total_memory * mamba_memory_ratio
+ mamba_max_requests = int(mamba_memory_bytes // mamba_states_memory_per_request)
+
+ # Reduce buffer sizes for KV cache
+ buffer_size_bytes = int(buffer_size_bytes * (1.0 - mamba_memory_ratio))
+ paused_buffer_size_bytes = int(paused_buffer_size_bytes * (1.0 - mamba_memory_ratio))
+
+ block_count = buffer_size_bytes // self.block_size_bytes
+ block_count = max(2, block_count) # need >= 1 active block + 1 dummy block
+ paused_block_count = paused_buffer_size_bytes // self.block_size_bytes
+ else:
+ block_count = buffer_size_bytes // (
+ self.block_size_bytes + mamba_states_memory_per_request
+ )
+ block_count = max(2, block_count) # need >= 1 active block + 1 dummy block
+ paused_block_count = paused_buffer_size_bytes // (
+ self.block_size_bytes + mamba_states_memory_per_request
+ )
+
# If using pipeline parallelism synchronize the total block count in case the
# pipeline stages have different layer allocations. Non-uniform block counts
# can lead to some ranks pausing requests earlier than other ranks
# (i.e., divergence in the scheduling behavior).
if pp_size > 1:
block_count_tensor = torch.tensor(
- block_count, dtype=torch.int32, device=torch.cuda.current_device()
+ (block_count, paused_block_count),
+ dtype=torch.int32,
+ device=torch.cuda.current_device(),
)
torch.distributed.all_reduce(
block_count_tensor,
op=torch.distributed.ReduceOp.MIN,
group=self.pipeline_parallel_group,
)
- block_count = block_count_tensor.item()
+ block_count = block_count_tensor[0].item()
+ paused_block_count = block_count_tensor[1].item()
- self.block_allocator = BlockAllocator(
+ self.kv_block_allocator = KVBlockAllocator(
context=self,
total_count=(
block_count if self.unified_memory_level == 0 else block_count + paused_block_count
),
paused_count=paused_block_count,
+ enable_prefix_caching=self.enable_prefix_caching,
+ prefix_caching_eviction_policy=self.prefix_caching_eviction_policy,
)
# Track request metadata.
+ request_metadata_types = inference_config.request_metadata_types
if request_metadata_types is None:
request_metadata_types = DynamicInferenceRequest.get_metadata_types()
self.request_metadata_types = request_metadata_types
# Initialize context state.
- self.params_dtype = params_dtype
- self.max_sequence_length = max_sequence_length
-
- # Request and token counts.
- self.total_request_count = 0
- self.active_token_count = 0
- self.paused_request_count = 0
- self.batch_dimensions = InferenceBatchDimensions(
- token_count=0, prefill_req_count=0, decode_req_count=0
- )
- self.padded_batch_dimensions = InferenceBatchDimensions(
- token_count=0, prefill_req_count=0, decode_req_count=0
- )
- self.padded_active_token_count = 0
- self.padded_active_request_count = 0
- self.paused_tokens = None
+ self.params_dtype = model_config.params_dtype
+ self.max_sequence_length = inference_config.max_sequence_length
# Block ids.
self.max_kv_block_count = math.ceil(self.max_sequence_length / self.block_size_tokens)
# Set max_requests, max_tokens.
- if max_requests is None:
+ if inference_config.max_requests is None:
# Maximize compute utilization by defaulting to 1 block per request.
- self.max_requests = self.block_allocator.total_count - 1 # -1 for dummy block
+ self.max_requests = self.kv_block_allocator.total_count - 1 # -1 for dummy block
+
+ # Adjust max_requests for Mamba memory constraints if necessary
+ if self.is_hybrid_model and mamba_max_requests < self.max_requests:
+ self.max_requests = int(mamba_max_requests)
+
self.max_requests = self.max_requests // tp_size * tp_size
self.max_requests = self.max_requests // self.REQUEST_ROUNDER * self.REQUEST_ROUNDER
else:
# User can control request overflow via max_requests.
- self.max_requests = max_requests
+ self.max_requests = inference_config.max_requests
+
+ assert (
+ self.max_requests % tp_size == 0
+ ), f"max_requests must be divisible by tp_size ({tp_size}), but got {self.max_requests}"
- self.max_tokens = max_tokens or self.DEFAULT_MAX_TOKENS
+ self.max_tokens = inference_config.max_tokens or self.DEFAULT_MAX_TOKENS
assert self.max_tokens >= self.max_requests, (
f"max_tokens ({self.max_tokens}) must be >= "
@@ -516,13 +531,11 @@ def __init__(
# Attention metadata initialization (tensors are now handled by MHAMetadata classes)
- self.num_prefill_requests = 0
self.graph_attn_metadata = {}
self.non_graph_attn_metadata = {}
- self.active_attn_metadata = None
self.graph_attn_metadata["mha_metadata"] = GraphedMHAMetadata(
- block_count_total=self.block_allocator.total_count,
+ block_count_total=self.kv_block_allocator.total_count,
max_kv_block_count=self.max_kv_block_count,
max_requests=self.max_requests,
block_size_tokens=self.block_size_tokens,
@@ -530,77 +543,171 @@ def __init__(
)
self.non_graph_attn_metadata["mha_metadata"] = NonGraphedMHAMetadata(
- block_count_total=self.block_allocator.total_count,
+ block_count_total=self.kv_block_allocator.total_count,
max_kv_block_count=self.max_kv_block_count,
max_requests=self.max_requests,
block_size_tokens=self.block_size_tokens,
max_seqlen=self.max_sequence_length,
)
+ self.moe_enable_routing_replay = model_config.moe_enable_routing_replay
+ if self.moe_enable_routing_replay:
+ assert (
+ model_config.num_moe_experts is not None
+ ), "Router recording/replay requested but no MoE experts specified!"
+ self.moe_routing_metadata = RoutingMetadata(self, model_config.moe_router_topk)
+
# CUDA graph config list
+ self.use_cuda_graphs_for_non_decode_steps = (
+ inference_config.use_cuda_graphs_for_non_decode_steps
+ )
self.cuda_graph_batch_dimensions_list, self.cuda_graph_token_counts = (
CUDAGraphBatchDimensionBuilder.generate_cuda_graph_batch_dimensions_list(
tp_size=tp_size,
- num_cuda_graphs=num_cuda_graphs,
- cuda_graph_max_tokens=self.max_requests,
- cuda_graph_mixed_prefill_count=cuda_graph_mixed_prefill_count,
+ num_cuda_graphs=inference_config.num_cuda_graphs,
+ cuda_graph_max_tokens=self.max_requests * (self.num_speculative_tokens + 1),
+ cuda_graph_mixed_prefill_request_count=inference_config.cuda_graph_mixed_prefill_count,
max_requests=self.max_requests,
max_tokens=self.max_tokens,
max_sequence_length=self.max_sequence_length,
- use_cuda_graphs_for_non_decode_steps=use_cuda_graphs_for_non_decode_steps,
+ use_cuda_graphs_for_non_decode_steps=self.use_cuda_graphs_for_non_decode_steps,
+ num_speculative_tokens=self.num_speculative_tokens,
)
)
- # Whether to offload the KV cache. Determines where the KV cache is allocated within memory.
- self.offload_kv_cache = offload_kv_cache
- assert not (
- self.offload_kv_cache and self.unified_memory_level
- ), "The KV cache should not be instantiated in unified memory when it is offloaded during training."
+ self.smallest_non_decode_cuda_graph_size = min(
+ inference_config.cuda_graph_mixed_prefill_count, self.max_requests
+ )
- self._using_cuda_graph_this_step = False
- self.use_cuda_graphs_for_non_decode_steps = use_cuda_graphs_for_non_decode_steps
# Deal with chunked prefill
- self.chunked_prefill_request_id = -1
- self.has_explicit_chunked_prefill_req = False
+ self.enable_chunked_prefill = inference_config.enable_chunked_prefill
# FlashInfer.
- if use_flashinfer_fused_rope is True:
+ if inference_config.use_flashinfer_fused_rope is True:
assert HAVE_FLASHINFER, "flashinfer is not installed"
- elif use_flashinfer_fused_rope is None:
- use_flashinfer_fused_rope = HAVE_FLASHINFER
- self.use_flashinfer_fused_rope = use_flashinfer_fused_rope
+ elif inference_config.use_flashinfer_fused_rope is None:
+ inference_config.use_flashinfer_fused_rope = HAVE_FLASHINFER
+ self.use_flashinfer_fused_rope = inference_config.use_flashinfer_fused_rope
# Allocate GPU state.
self.is_tensor_state_allocated = False
- self.is_symmetric_memory_initialized = False
- self.allocate_all_tensors(is_init=True)
+ self.initialize_all_tensors()
# Print info.
logging.info(
"DynamicInferenceContext: allocated context with active buffer size %s (%d blocks)."
% (
- get_mem_size_str(self.block_allocator.active_count * self.block_size_bytes),
- self.block_allocator.active_count,
+ get_mem_size_str(self.kv_block_allocator.active_count * self.block_size_bytes),
+ self.kv_block_allocator.active_count,
)
)
- def allocate_all_tensors(self, *, is_init: bool) -> None:
- """Allocate GPU state.
-
- This method is used for both 1) initial allocation, and 2) resuming the
- GPU state after a suspend.
-
- Args:
- is_init (bool): True if this is being called from `__init__()`.
- """
-
- # Only allocate tensors when not using unified memory at all (level 0),
- # or for initial allocation during `__init__()`. For levels 1 and 2, we do
- # not perform any explicit allocations or deallocations after the initial
- # call to `__init__()`.
- if self.unified_memory_level != 0 and not is_init:
- return
+ def _allocate_memory_buffer(self):
+ """Allocate the KV cache memory buffer."""
+ if self.cache_mla_latent:
+ self.memory_buffer = torch.empty(
+ (
+ self.num_attention_layers,
+ self.kv_block_allocator.total_count,
+ self.block_size_tokens,
+ self.kv_reduced_dim,
+ ),
+ dtype=self.params_dtype,
+ device=torch.cuda.current_device(),
+ )
+ else:
+ self.memory_buffer = torch.empty(
+ (
+ 2, # key and value
+ self.num_attention_layers,
+ self.kv_block_allocator.total_count,
+ self.block_size_tokens,
+ self.num_attention_heads_per_partition,
+ self.hidden_size_per_attention_head,
+ ),
+ dtype=self.params_dtype,
+ device=torch.cuda.current_device(),
+ )
+ if (
+ self.kv_cache_management_mode == KVCacheManagementMode.OFFLOAD
+ and not self._uses_torch_memory_saver
+ ):
+ assert self.unified_memory_level == 0
+ self._offloadable_tensor_names.add("memory_buffer")
+ self._offloadable_cpu_backups["memory_buffer"] = torch.empty_like(
+ self.memory_buffer, device="cpu"
+ ).pin_memory()
+
+ def _allocate_mamba_states(self):
+ """Allocate Mamba states for hybrid models."""
+ if self.is_hybrid_model:
+ self.mamba_metadata = MambaMetadata(
+ max_requests=self.max_requests,
+ max_tokens=self.max_tokens,
+ d_conv=self.mamba_conv_states_shape[-1],
+ )
+ self.mamba_conv_states = torch.empty(
+ (self.num_mamba_layers, self.max_requests) + self.mamba_conv_states_shape,
+ dtype=self.mamba_conv_states_dtype,
+ device=torch.cuda.current_device(),
+ )
+ self.mamba_ssm_states = torch.empty(
+ (self.num_mamba_layers, self.max_requests) + self.mamba_ssm_states_shape,
+ dtype=self.mamba_ssm_states_dtype,
+ device=torch.cuda.current_device(),
+ )
+ if self.num_speculative_tokens > 0:
+ self.mamba_intermediate_conv_states = torch.empty(
+ (
+ self.num_mamba_layers,
+ self.max_requests,
+ self.num_speculative_tokens + 1,
+ *self.mamba_conv_states_shape,
+ ),
+ dtype=self.mamba_conv_states_dtype,
+ device=torch.cuda.current_device(),
+ )
+ self.mamba_intermediate_ssm_states = torch.empty(
+ (
+ self.num_mamba_layers,
+ self.max_requests,
+ self.num_speculative_tokens + 1,
+ *self.mamba_ssm_states_shape,
+ ),
+ dtype=self.mamba_ssm_states_dtype,
+ device=torch.cuda.current_device(),
+ )
+ if (
+ self.kv_cache_management_mode == KVCacheManagementMode.OFFLOAD
+ and not self._uses_torch_memory_saver
+ ):
+ assert self.unified_memory_level == 0
+ self._offloadable_tensor_names.add("mamba_conv_states")
+ self._offloadable_cpu_backups["mamba_conv_states"] = torch.empty_like(
+ self.mamba_conv_states, device="cpu"
+ ).pin_memory()
+ self._offloadable_tensor_names.add("mamba_ssm_states")
+ self._offloadable_cpu_backups["mamba_ssm_states"] = torch.empty_like(
+ self.mamba_ssm_states, device="cpu"
+ ).pin_memory()
+ if self.num_speculative_tokens > 0:
+ self._offloadable_tensor_names.add("mamba_intermediate_conv_states")
+ self._offloadable_cpu_backups["mamba_intermediate_conv_states"] = (
+ torch.empty_like(
+ self.mamba_intermediate_conv_states, device="cpu"
+ ).pin_memory()
+ )
+ self._offloadable_tensor_names.add("mamba_intermediate_ssm_states")
+ self._offloadable_cpu_backups["mamba_intermediate_ssm_states"] = (
+ torch.empty_like(
+ self.mamba_intermediate_ssm_states, device="cpu"
+ ).pin_memory()
+ )
+ else:
+ self.mamba_metadata = None
+ def initialize_all_tensors(self) -> None:
+ """Allocate all GPU state during initial construction."""
# Mark allocated.
if self.is_tensor_state_allocated:
return
@@ -610,7 +717,7 @@ def allocate_all_tensors(self, *, is_init: bool) -> None:
for key in vars(self).keys():
value = getattr(self, key)
assert not isinstance(value, torch.Tensor), (
- "All tensors should be allocated within `allocate_all_tensors()."
+ "All tensors should be allocated within `initialize_all_tensors()`. "
f"Please move tensor '{key}'."
)
@@ -620,6 +727,8 @@ def allocate_all_tensors(self, *, is_init: bool) -> None:
)
# request_query_lengths is the input prompt tokens length during prefill phase (1st step) and then 1 for the decode phase (i.e During generation)
self.request_query_lengths = torch.empty_like(self.request_ids)
+ # True only for a new request , then after a forward pass it is set to False
+ self.request_in_prefill_status_tensor = torch.empty_like(self.request_ids)
# request_output_lengths is len(input_prompt_tokens) + num_tokens_to_generate
self.request_output_lengths = torch.empty_like(self.request_ids)
# request_kv_length_offsets is the same as query length during prefill phase (1st step) and then 1 for the decode phase (i.e During generation)
@@ -656,114 +765,127 @@ def allocate_all_tensors(self, *, is_init: bool) -> None:
self.token_to_position_in_request = torch.empty_like(self.token_to_input_ids)
self.token_to_local_position_within_kv_block = torch.empty_like(self.token_to_input_ids)
- # Memory buffer.
- def allocate_memory_buffer():
- """Allocate the memory buffer. This function is called below within
- `with ctx_manager:`."""
- if self.cache_mla_latent:
- self.memory_buffer = torch.empty(
- (
- self.num_attention_layers,
- self.block_allocator.total_count,
- self.block_size_tokens,
- self.kv_reduced_dim,
- ),
- dtype=self.params_dtype,
- device=torch.cuda.current_device(),
- )
- else:
- ctx = (
- torch_memory_saver.region(tag="kv_cache", enable_cpu_backup=True)
- if HAVE_TORCH_MEMORY_SAVER and self.offload_kv_cache
- else nullcontext()
- )
+ # NOTE: Need to build this outside the UVM / TMS context to avoid IMA.
+ if self.is_hybrid_model:
+ self.mamba_metadata = MambaMetadata(
+ max_requests=self.max_requests,
+ max_tokens=self.max_tokens,
+ mamba_chunk_size=self.mamba_chunk_size,
+ d_conv=self.mamba_conv_states_shape[-1],
+ )
- with ctx:
- self.memory_buffer = torch.empty(
- (
- 2, # key and value
- self.num_attention_layers,
- self.block_allocator.total_count,
- self.block_size_tokens,
- self.num_attention_heads_per_partition,
- self.hidden_size_per_attention_head,
- ),
- dtype=self.params_dtype,
- device=torch.cuda.current_device(),
- )
+ # Allocate large non-graphed buffers.
+ need_static_addr = (
+ self.static_kv_memory_pointers
+ and self.kv_cache_management_mode != KVCacheManagementMode.PERSIST
+ )
- # Optional state tensors for hybrid models
- def allocate_mamba_states():
- """Allocate Mamba states. This function is called below within
- `with ctx_manager:`."""
- if self.is_hybrid_model:
- self.mamba_metadata = MambaMetadata(
- max_requests=self.max_requests, max_tokens=self.max_tokens
- )
- self.mamba_conv_states = torch.empty(
- (self.num_mamba_layers, self.max_requests) + self.mamba_conv_states_shape,
- dtype=self.params_dtype,
- device=torch.cuda.current_device(),
- )
- self.mamba_ssm_states = torch.empty(
- (self.num_mamba_layers, self.max_requests) + self.mamba_ssm_states_shape,
- dtype=self.params_dtype,
- device=torch.cuda.current_device(),
- )
+ ctx_manager = nullcontext()
+ if self.unified_memory_level != 0:
+ ctx_manager = torch.cuda.use_mem_pool(self.unified_memory_mempool)
+ elif HAVE_TORCH_MEMORY_SAVER and need_static_addr:
+ ctx_manager = torch_memory_saver.region(
+ tag=self.TMS_TAG,
+ enable_cpu_backup=(self.kv_cache_management_mode == KVCacheManagementMode.OFFLOAD),
+ )
+ self._uses_torch_memory_saver = True
+ with ctx_manager:
+ self._allocate_memory_buffer()
+ self._allocate_mamba_states()
+
+ # Allocate Mamba prefix cache if configured
+ self.mamba_slot_allocator: Optional[MambaSlotAllocator] = None
+ if (
+ self.is_hybrid_model
+ and self.config.prefix_caching_mamba_gb is not None
+ and self.config.prefix_caching_mamba_gb > 0
+ and self.config.enable_prefix_caching
+ ):
+ self._allocate_mamba_cache(self.config.prefix_caching_mamba_gb)
+
+ # Reset tensor-related metadata.
+ self.reset_metadata()
+
+ def reinitialize_inference_state_buffers(self):
+ """Restore large tensors (KV cache, Mamba states) after a suspend.
+
+ Called by the engine during `resume()`. Initial allocation is in `initialize_all_tensors()`.
+ """
+ if self.is_tensor_state_allocated:
+ return
+ self.is_tensor_state_allocated = True
- else:
- self.mamba_metadata = None
+ if self.kv_cache_management_mode == KVCacheManagementMode.PERSIST:
+ return
- # Allocate `ctx_manager`-managed buffers. (For currently unknown reasons,
- # `ctx_manager` can only be used once.)
- ctx_manager = (
- torch.cuda.use_mem_pool(self.unified_memory_mempool)
- if self.unified_memory_level > 0
- else nullcontext()
- )
- with ctx_manager:
- allocate_memory_buffer()
- allocate_mamba_states()
+ if self.unified_memory_level != 0 or self._uses_torch_memory_saver:
+ # Need to bring back the memory block before we reset it.
+ if self._uses_torch_memory_saver:
+ tag = self.TMS_TAG
+ if torch.distributed.get_rank() == 0:
+ logging.info(
+ "torch_memory_saver: resuming %s, before: %s", tag, device_memory_summary()
+ )
+ torch_memory_saver.resume(tag)
+ if torch.distributed.get_rank() == 0:
+ logging.info(
+ "torch_memory_saver: resumed %s, after: %s", tag, device_memory_summary()
+ )
+ if self.kv_cache_management_mode == KVCacheManagementMode.RECOMPUTE:
+ self.reset_metadata()
+ return
- # Reset attention and Mamba state.
- self.reset_attention_state()
- self.reset_mamba_state()
+ if self.kv_cache_management_mode == KVCacheManagementMode.OFFLOAD:
+ for name, tensor in ((n, getattr(self, n)) for n in self._offloadable_tensor_names):
+ tensor.storage().resize_(self._offloadable_storage_sizes[name])
+ tensor.copy_(self._offloadable_cpu_backups[name], non_blocking=True)
+ elif self.kv_cache_management_mode == KVCacheManagementMode.RECOMPUTE:
+ self.is_tensor_state_allocated = False
+ self.initialize_all_tensors()
- def deallocate_all_tensors(self):
- """Deallocate GPU state.
+ def deallocate_inference_state_buffers(self):
+ """Deallocate large tensors (KV cache, Mamba states) during suspend.
- This method is used for suspending the dynamic engine.
+ Called by the engine during `suspend()`. Mirror to `reinitialize_inference_state_buffers()`.
"""
+ if not self.is_tensor_state_allocated:
+ return
+ self.is_tensor_state_allocated = False
+
+ if self.kv_cache_management_mode == KVCacheManagementMode.PERSIST:
+ return
- # Only deallocate tensors when not using unified memory at all (level 0).
- # For levels 1 and 2, we do not perform any explicit allocations or
- # deallocations after the initial call to `__init__()`.
if self.unified_memory_level != 0:
return
- # Mark deallocated.
- if not self.is_tensor_state_allocated:
+ if self._uses_torch_memory_saver:
+ tag = self.TMS_TAG
+ if torch.distributed.get_rank() == 0:
+ logging.info(
+ "torch_memory_saver: pausing %s, before: %s", tag, device_memory_summary()
+ )
+ torch_memory_saver.pause(tag)
+ if torch.distributed.get_rank() == 0:
+ logging.info(
+ "torch_memory_saver: paused %s, after: %s", tag, device_memory_summary()
+ )
return
- self.is_tensor_state_allocated = False
- # Delete all tensor attributes.
- # TODO(@lmcafee): check that device == 'cuda'?
- keys = list(vars(self).keys())
- for key in keys:
- value = getattr(self, key)
- if isinstance(value, torch.Tensor):
- delattr(self, key)
+ if self.kv_cache_management_mode == KVCacheManagementMode.OFFLOAD:
+ for name, tensor in ((n, getattr(self, n)) for n in self._offloadable_tensor_names):
+ self._offloadable_storage_sizes[name] = tensor.storage().size()
+ self._offloadable_cpu_backups[name].copy_(tensor, non_blocking=True)
+ tensor.storage().resize_(0)
+ elif self.kv_cache_management_mode == KVCacheManagementMode.RECOMPUTE:
+ # TODO(@lmcafee): check that device == 'cuda'?
+ for key in list(vars(self).keys()):
+ value = getattr(self, key)
+ if isinstance(value, torch.Tensor):
+ delattr(self, key)
@classmethod
def round_up_tokens(cls, value, tp_size=None):
- """Round up to nearest multiple of `TOKEN_ROUNDER` (above) that is also divisible by tensor model parallel size."""
- if not HAVE_PACKAGING:
- raise ImportError(
- "`packaging` is required for this functionality, please install it with `pip install packaging`"
- )
- if PkgVersion(mcore_version) < PkgVersion("0.13"):
- return cls.round_up(value)
-
+ """Round up to nearest multiple of `TOKEN_ROUNDER` that is also divisible by tensor model parallel size."""
# Make sure divisible by TP size
if tp_size is None:
# Check if parallel state is initialized before trying to get TP size
@@ -775,72 +897,9 @@ def round_up_tokens(cls, value, tp_size=None):
return token_rounder * int(math.ceil(int(value) / token_rounder))
- @classmethod
- def from_config(
- cls,
- inference_config: InferenceWrapperConfig,
- model,
- max_batch_size: int,
- buffer_size_gb: float = 40,
- num_cuda_graphs: int = None,
- mamba_inference_state_config: Optional[MambaInferenceStateConfig] = None,
- unified_memory_level: int = 0,
- ):
- """
- Instantiate a `DynamicInferenceContext` from a `TransformerConfig` and an `InferenceWrapperConfig`.
- """
- # TODO: Add other necessary configs from inference_config
-
- # Max sequence length.
- position_embedding_type = get_attr_wrapped_model(model, "position_embedding_type")
- model_max_seq_len = get_attr_wrapped_model(model, "max_sequence_length")
- inf_max_seq_len = inference_config.inference_max_seq_length
-
- if position_embedding_type == "learned_absolute":
- # When using absolute position embeddings, it is critical that the
- # context's `max_sequence_length` is less than or equal to the model's
- # `max_sequence_length`. Otherwise, the context's `position_ids` will
- # contain ids greater than the dimension of the position embedding
- # tensor, which will result in an index error.
- if inf_max_seq_len:
- max_sequence_length = min(model_max_seq_len, inf_max_seq_len)
- else:
- max_sequence_length = model_max_seq_len
- assert max_batch_size <= model_max_seq_len
- else:
- max_sequence_length = (
- inference_config.inference_max_seq_length or model_config.max_sequence_length
- )
- max_sequence_length = max(max_sequence_length, max_batch_size)
-
- # Context.
- model_config = model.config
- return cls(
- params_dtype=inference_config.params_dtype,
- num_layers=model_config.num_layers // model_config.pipeline_model_parallel_size,
- kv_channels=model_config.kv_channels,
- num_attention_heads=model_config.num_query_groups,
- tensor_model_parallel_size=model_config.tensor_model_parallel_size,
- pipeline_model_parallel_size=model_config.pipeline_model_parallel_size,
- max_sequence_length=max_sequence_length,
- buffer_size_gb=buffer_size_gb,
- materialize_only_last_token_logits=False,
- num_cuda_graphs=num_cuda_graphs,
- use_flashinfer_fused_rope=None,
- mamba_inference_state_config=mamba_inference_state_config,
- unified_memory_level=unified_memory_level,
- )
-
@classmethod
def round_up_requests(cls, value, tp_size=None):
- """Round up to nearest multiple of `REQUEST_ROUNDER` (above) that is also divisible by tensor model parallel size."""
- if not HAVE_PACKAGING:
- raise ImportError(
- "`packaging` is required for this functionality, please install it with `pip install packaging`"
- )
- if PkgVersion(mcore_version) < PkgVersion("0.13"):
- return cls.round_up(value)
-
+ """Round up to nearest multiple of `REQUEST_ROUNDER` that is also divisible by tensor model parallel size."""
# Make sure divisible by TP size
if tp_size is None:
# Check if parallel state is initialized before trying to get TP size
@@ -852,16 +911,6 @@ def round_up_requests(cls, value, tp_size=None):
return request_rounder * int(math.ceil(int(value) / request_rounder))
- @classmethod
- def round_up(cls, value):
- """Deprecated in favor of round_up_tokens and round_up_requests."""
- warnings.warn(
- "`round_up` is deprecated in favor of `round_up_tokens` or `round_up_requests` "
- "and will be removed in `megatron-core` 0.14."
- )
- ROUNDER = getattr(cls, "ROUNDER", 64)
- return ROUNDER * int(math.ceil(int(value) / ROUNDER))
-
def is_static_batching(self) -> bool:
"""Is static batching? False."""
return False
@@ -882,6 +931,7 @@ def has_unfinished_requests(self) -> bool:
def cu_query_lengths(self) -> Tuple[Tensor, int]:
"""Cumulative query sequence lengths."""
+ assert self.active_attn_metadata is not None
return (
self.active_attn_metadata["mha_metadata"].state_data["cu_query_seq_lengths"],
self.active_attn_metadata["mha_metadata"].state_data["max_seqlen_q"],
@@ -889,6 +939,7 @@ def cu_query_lengths(self) -> Tuple[Tensor, int]:
def cu_kv_lengths(self) -> Tuple[Tensor, Tensor, int]:
"""Cumulative key/value sequence lengths."""
+ assert self.active_attn_metadata is not None
return (
self.active_attn_metadata["mha_metadata"].state_data["cu_kv_seq_lengths"],
self.active_attn_metadata["mha_metadata"].state_data["kv_seq_lengths"],
@@ -958,18 +1009,20 @@ def append_key_value_cache(self, layer_number: int, key: Tensor, value: Tensor)
: self.padded_active_token_count
]
- def key_value_cache(self, layer_number: int) -> Tuple[Tensor, Tensor]:
+ def key_value_cache(self, layer_number: int) -> Tuple[Tensor, Optional[Tensor], Tensor]:
"""Read from KV cache.
Args:
layer_number (int): Layer number.
Return:
- (Tuple[Tensor, Tensor]) The key and value pointer tensors that point
- to blocks within the block-level memory buffer.
+ (Tuple[Tensor, Tensor, Tensor]) The key and value pointer tensors that point
+ to blocks within the block-level memory buffer as well as the block table.
"""
attention_layer_number = self.layer_map[layer_number - 1]
+ assert self.active_attn_metadata is not None
+
if self.cache_mla_latent:
return (
self.memory_buffer[attention_layer_number],
@@ -983,16 +1036,68 @@ def key_value_cache(self, layer_number: int) -> Tuple[Tensor, Tensor]:
self.active_attn_metadata["mha_metadata"].state_data["block_table"],
)
- def mamba_states_cache(self, layer_number: int) -> Tuple[Tensor, Tensor]:
+ def mamba_states_cache(
+ self, layer_number: int, intermediate: bool = False
+ ) -> Tuple[Tensor, Tensor]:
"""Returns the Mamba state tensors for the given layer."""
assert self.is_hybrid_model, "Only hybrid models have Mamba state tensors"
mamba_layer_number = self.layer_map[layer_number - 1]
- conv_state = self.mamba_conv_states[mamba_layer_number]
- ssm_state = self.mamba_ssm_states[mamba_layer_number]
+ if intermediate:
+ conv_state = self.mamba_intermediate_conv_states[mamba_layer_number]
+ ssm_state = self.mamba_intermediate_ssm_states[mamba_layer_number]
+ else:
+ conv_state = self.mamba_conv_states[mamba_layer_number]
+ ssm_state = self.mamba_ssm_states[mamba_layer_number]
return (conv_state, ssm_state)
+ # =========================================================================
+ # Mamba prefix cache infrastructure
+ # =========================================================================
+
+ def _allocate_mamba_cache(self, mamba_gb: float) -> None:
+ """Allocate the Mamba state cache for prefix caching.
+
+ Args:
+ mamba_gb: GPU memory budget in GB for the cache.
+ """
+ import math as _math
+
+ conv_size = _math.prod(self.mamba_conv_states_shape) * self.mamba_conv_states_dtype.itemsize
+ ssm_size = _math.prod(self.mamba_ssm_states_shape) * self.mamba_ssm_states_dtype.itemsize
+ per_slot_bytes = self.num_mamba_layers * (conv_size + ssm_size)
+ total_bytes = int(mamba_gb * 1024**3)
+ max_slots = total_bytes // per_slot_bytes
+ if max_slots < 1:
+ logging.warning(
+ "Mamba cache budget (%.3f GB) too small for even 1 slot "
+ "(need %.3f GB per slot). Mamba caching disabled.",
+ mamba_gb,
+ per_slot_bytes / 1024**3,
+ )
+ return
+
+ self.mamba_slot_allocator = MambaSlotAllocator(
+ context=self,
+ max_slots=max_slots,
+ num_mamba_layers=self.num_mamba_layers,
+ conv_states_shape=self.mamba_conv_states_shape,
+ ssm_states_shape=self.mamba_ssm_states_shape,
+ conv_states_dtype=self.mamba_conv_states_dtype,
+ ssm_states_dtype=self.mamba_ssm_states_dtype,
+ )
+ self.kv_block_allocator.on_blocks_deregistered = (
+ self.mamba_slot_allocator.on_kv_blocks_deregistered
+ )
+
+ logging.info(
+ "Mamba prefix cache: %d slots (%.3f GB), per-slot %.1f KB",
+ max_slots,
+ max_slots * per_slot_bytes / 1024**3,
+ per_slot_bytes / 1024,
+ )
+
def apply_fused_qk_rotary_emb(
self, query: Tensor, key: Tensor, cos_sin_emb: Tensor, config: TransformerConfig
) -> Tuple[Tensor, Tensor]:
@@ -1142,9 +1247,9 @@ def add_dummy_requests_parallel(
), "chunked requests are not supported in add_dummy_requests_parallel"
assert req.remaining_prompt_tokens is not None, "request missing prompt tokens"
assert req.sampling_params is not None, "request missing sampling params"
- chunk_length = req.remaining_prompt_length
- assert chunk_length > 0, "request without prompt tokens is not supported"
- lengths.append(chunk_length)
+ prefill_chunk_length = req.remaining_prompt_length
+ assert prefill_chunk_length > 0, "request without prompt tokens is not supported"
+ lengths.append(prefill_chunk_length)
num_tokens_to_generate.append(req.sampling_params.num_tokens_to_generate)
request_ids.append(req.request_id)
prompt_tokens.append(
@@ -1180,6 +1285,7 @@ def add_dummy_requests_parallel(
self.request_ids[request_slice] = request_ids_tensor
self.request_query_lengths[request_slice] = lengths_tensor
+ self.request_in_prefill_status_tensor[request_slice] = 1
self.request_output_lengths[request_slice] = lengths_tensor + tokens_to_generate_tensor
self.request_kv_length_offsets[request_slice] = 0
self.request_kv_block_counts[request_slice] = block_counts
@@ -1188,7 +1294,7 @@ def add_dummy_requests_parallel(
metadata_cols[i], dtype=dtype, device=torch.cuda.current_device()
)
- dummy_block_idx = self.block_allocator.dummy_block_idx
+ dummy_block_idx = self.kv_block_allocator.dummy_block_idx
self.request_last_kv_block_id[request_slice] = dummy_block_idx
self.request_last_kv_block_offset[request_slice] = torch.remainder(
lengths_tensor - 1, self.block_size_tokens
@@ -1260,11 +1366,15 @@ def add_dummy_requests_for_cudagraph_capture(
Adds dummy requests to reflect the number of prefill and decode requests in the graph config.
These are using during cuda graph captures.
"""
- prefill_tokens = graph_dimensions.token_count - graph_dimensions.decode_req_count
+ prefill_tokens = graph_dimensions.token_count - (
+ graph_dimensions.decode_req_count * (self.num_speculative_tokens + 1)
+ )
# Pre-construct shared objects (safe due to deep copy in DynamicInferenceRequest.__post_init__)
shared_sampling_params = SamplingParams(num_tokens_to_generate=1, termination_id=-1)
- shared_decode_tokens = torch.zeros(1, dtype=torch.long, device=torch.cuda.current_device())
+ shared_decode_tokens = torch.zeros(
+ self.num_speculative_tokens + 1, dtype=torch.long, device=torch.cuda.current_device()
+ )
decode_requests = [
DynamicInferenceRequest(
@@ -1315,49 +1425,136 @@ def num_decode_requests(self) -> int:
"""
return self.total_request_count - self.paused_request_count - self.num_prefill_requests
+ def add_dummy_requests_for_expert_parallel_step(self) -> None:
+ """Minimal context setup so an EP rank with no real requests can replay
+ an already-captured cuda graph without crashing or corrupting memory.
+
+ This is the fast alternative to add_dummy_requests_for_cudagraph_capture
+ (which goes through the heavyweight add_dummy_requests_parallel path).
+
+ We setup minimal state such the initialize_attention_state and the forward
+ pass can run without error.
+
+ """
+ smallest_cuda_graph_dimensions = min(
+ [x for x in self.cuda_graph_batch_dimensions_list if x.prefill_req_count == 0]
+ )
+ # the smallest cuda graph is decode only.
+ assert smallest_cuda_graph_dimensions.prefill_req_count == 0
+
+ N = smallest_cuda_graph_dimensions.decode_req_count
+ tokens_per_request = self.num_speculative_tokens + 1
+ T = smallest_cuda_graph_dimensions.token_count # N * tokens_per_request
+ dummy_block_idx = self.kv_block_allocator.dummy_block_idx
+
+ # 1. Request counts and token count.
+ # With speculative decoding each decode request has (num_speculative_tokens + 1) tokens.
+ self.total_request_count = N
+ self.active_token_count = T
+ self.num_prefill_requests = 0
+
+ # 2. Per-request state consumed by mha_metadata.update().
+ self.request_query_lengths[0:N].fill_(tokens_per_request)
+ self.request_kv_length_offsets[0:N].fill_(0)
+ self.request_to_kv_block_ids[0:N, 0] = dummy_block_idx
+
+ # 3. Token-level state consumed by the triton KV append kernel.
+ self.token_to_block_idx[0:T] = dummy_block_idx
+ self.token_to_local_position_within_kv_block[0:T] = (
+ torch.arange(T, device=self.token_to_block_idx.device) % tokens_per_request
+ )
+
+ if self.is_hybrid_model:
+ # 4. token_to_request_idx: needed by mamba_metadata.update() for hybrid models.
+ self.token_to_request_idx[0:T] = torch.repeat_interleave(
+ torch.arange(
+ 0,
+ N,
+ device=self.token_to_request_idx.device,
+ dtype=self.token_to_request_idx.dtype,
+ ),
+ tokens_per_request,
+ )
+
+ # 5. Mamba state: allocate slots for dummy requests.
+ self.mamba_metadata.request_to_mamba_state_idx[0:N] = (
+ self.mamba_metadata.batch_allocate_slots(N)
+ )
+
def initialize_attention_state(
- self, *, construct_graph_dimensions: Optional[InferenceBatchDimensions] = None
+ self,
+ *,
+ construct_graph_dimensions: Optional[InferenceBatchDimensions] = None,
+ is_expert_parallel_dummy_cuda_graph_step: bool = False,
) -> None:
"""Initialize attention state so that every layer can use it.
Args:
- construct_graph_dimensions (Optional[InferenceBatchDimensions]): The graph config to use for constructing the cuda graphs.
+ construct_graph_dimensions (Optional[InferenceBatchDimensions]):
+ The graph config to use for constructing the cuda graphs.
+ is_expert_parallel_dummy_cuda_graph_step (bool):
+ Whether this is a dummy expert model parallel step.
Return:
None.
"""
- # if in recording mode, add dummy requests for cuda graph capture
- if construct_graph_dimensions is not None:
+ self.is_creating_cuda_graphs = construct_graph_dimensions is not None
+ assert not (
+ self.is_creating_cuda_graphs and is_expert_parallel_dummy_cuda_graph_step
+ ), "Dummy expert model parallel steps should not be creating cuda graphs."
+
+ # If in CUDA graph creation mode, add dummy requests for CUDA graph capture
+ if is_expert_parallel_dummy_cuda_graph_step:
+ self.add_dummy_requests_for_expert_parallel_step()
+ elif self.is_creating_cuda_graphs:
self.add_dummy_requests_for_cudagraph_capture(construct_graph_dimensions)
batch_dimensions = InferenceBatchDimensions(
token_count=self.active_token_count,
prefill_req_count=self.num_prefill_requests,
decode_req_count=self.num_decode_requests,
- has_explicit_chunked_prefill_req=self.has_explicit_chunked_prefill_req,
)
+
self.batch_dimensions = batch_dimensions
+
best_graph = CUDAGraphBatchDimensionBuilder.match_graph_config(
batch_dimensions,
self.cuda_graph_batch_dimensions_list,
+ smallest_non_decode_cuda_graph_size=self.smallest_non_decode_cuda_graph_size,
strict=self.is_hybrid_model,
decode_only_cuda_graphs=(not self.use_cuda_graphs_for_non_decode_steps),
ep_group=self.expert_model_parallel_group,
)
self._using_cuda_graph_this_step = best_graph is not None
+ if construct_graph_dimensions is not None:
+ assert self._using_cuda_graph_this_step
+
+ if is_expert_parallel_dummy_cuda_graph_step and not self.using_cuda_graph_this_step():
+ # If we are here, this means that CUDAGraphBatchDimensionBuilder.match_graph_config
+ # could not find a compatible cuda graph for the dummy forward step.
+ # Now, we need not do the remaining setup. The controller
+ # will directly call the model forward pass with a single token.
+ return
+
if self.using_cuda_graph_this_step():
self.padded_batch_dimensions = best_graph
else:
- padded_token_count = self.round_up_tokens(self.active_token_count)
if self.is_decode_only():
- padded_token_count = min(
- self.max_tokens,
- self.max_requests,
- self.round_up_tokens(self.active_token_count),
- )
- padded_decode_req_count = padded_token_count
+ if self.num_speculative_tokens > 0:
+ padded_decode_req_count = min(
+ self.max_requests, self.round_up_requests(self.num_decode_requests)
+ )
+ padded_token_count = padded_decode_req_count * (self.num_speculative_tokens + 1)
+ else:
+ padded_token_count = min(
+ self.max_tokens,
+ self.max_requests,
+ self.round_up_tokens(self.active_token_count),
+ )
+ padded_decode_req_count = padded_token_count
padded_prefill_req_count = 0
else:
+ padded_token_count = self.round_up_tokens(self.active_token_count)
target_padding_req_count = min(
self.max_requests,
self.round_up_requests(self.total_request_count - self.paused_request_count),
@@ -1368,7 +1565,6 @@ def initialize_attention_state(
token_count=padded_token_count,
prefill_req_count=padded_prefill_req_count,
decode_req_count=padded_decode_req_count,
- has_explicit_chunked_prefill_req=self.has_explicit_chunked_prefill_req,
)
self.padded_active_token_count = self.padded_batch_dimensions.token_count
self.padded_active_request_count = self.padded_batch_dimensions.req_count
@@ -1376,7 +1572,7 @@ def initialize_attention_state(
# Update token position indexes.
self.token_to_block_idx[self.active_token_count : self.padded_active_token_count] = (
- self.block_allocator.dummy_block_idx
+ self.kv_block_allocator.dummy_block_idx
)
self.token_to_local_position_within_kv_block[
self.active_token_count : self.padded_active_token_count
@@ -1386,9 +1582,9 @@ def initialize_attention_state(
] = 0
self.active_attn_metadata = (
- self.graph_attn_metadata
+ self.graph_attn_metadata # type: ignore[assignment]
if self.using_cuda_graph_this_step()
- else self.non_graph_attn_metadata
+ else self.non_graph_attn_metadata # type: ignore[assignment]
)
# Update cu_query_seq_lengths, max_seqlen_q.
@@ -1399,8 +1595,6 @@ def initialize_attention_state(
attn_dimensions = batch_dimensions
if self.using_cuda_graph_this_step():
- assert not self.has_explicit_chunked_prefill_req
-
# Treat some decode requests as prefill requests to fit the cuda graph batch dimension.
if batch_dimensions.decode_req_count > self.padded_batch_dimensions.decode_req_count:
total_req = batch_dimensions.req_count
@@ -1410,15 +1604,16 @@ def initialize_attention_state(
token_count=batch_dimensions.token_count,
prefill_req_count=adjusted_prefill_req_count,
decode_req_count=adjusted_decode_req_count,
- has_explicit_chunked_prefill_req=False,
)
+ assert self.active_attn_metadata is not None
self.active_attn_metadata["mha_metadata"].update(
request_query_lengths=query_lengths_view,
request_kv_length_offsets=request_kv_length_offsets_view,
request_to_kv_block_ids=request_to_kv_block_ids_view,
batch_dimensions=attn_dimensions,
padded_batch_dimensions=self.padded_batch_dimensions,
+ num_speculative_tokens=self.num_speculative_tokens,
)
if self.is_hybrid_model:
@@ -1427,37 +1622,31 @@ def initialize_attention_state(
cu_seqlens = self.active_attn_metadata["mha_metadata"].state_data[
"cu_query_seq_lengths"
]
+ intermediate_offsets_gpu = None
+ intermediate_counts_gpu = None
+ if self.mamba_slot_allocator is not None:
+ intermediate_offsets_gpu, intermediate_counts_gpu = (
+ self.mamba_slot_allocator.get_intermediate_gpu_data()
+ )
self.mamba_metadata.update(
active_mamba_indices_view,
token_to_request_idx_view,
cu_seqlens,
batch_dimensions=attn_dimensions,
padded_batch_dimensions=self.padded_batch_dimensions,
+ enable_chunked_prefill=self.is_chunked_prefill_enabled(),
+ intermediate_offsets_gpu=intermediate_offsets_gpu,
+ intermediate_counts_gpu=intermediate_counts_gpu,
)
- def reset(self) -> None:
- """Reset entire context.
+ if self.moe_enable_routing_replay:
+ if self.using_cuda_graph_this_step():
+ self.moe_routing_metadata.enable_static_buffer_recording()
+ else:
+ self.moe_routing_metadata.disable_static_buffer_recording()
- This method does:
- - Reset active/paused request/token counts to zero.
- - Reset available blocks to entire memory.
- - Reset other tensors to zeros (unncessary, just or sanity checking).
-
- This method is useful after cuda graph warmup iterations, where the
- context's memory buffer is referenced by the cuda graph system and
- cannot be deallocated.
- """
-
- # Reset request/token counts.
- self.total_request_count = 0
- self.active_token_count = 0
- self.paused_request_count = 0
- self.batch_dimensions = InferenceBatchDimensions(
- token_count=0, prefill_req_count=0, decode_req_count=0
- )
- self.padded_active_token_count = 0
- self.padded_active_request_count = 0
- self.paused_tokens = None
+ def reset_tensors(self) -> None:
+ """Fill all GPU tensors with sentinel values."""
# Reset request indexes.
self.request_ids.fill_(-1)
@@ -1468,6 +1657,7 @@ def reset(self) -> None:
self.request_last_kv_block_id.fill_(-1)
self.request_last_kv_block_offset.fill_(0)
self.request_to_kv_block_ids.fill_(-1)
+ self.request_in_prefill_status_tensor.fill_(-1)
# Reset request metadata.
for metadata_tensor in self.request_metadata.values():
@@ -1481,21 +1671,67 @@ def reset(self) -> None:
self.token_to_block_idx.fill_(-1)
self.token_to_local_position_within_kv_block.fill_(0)
- # Reset available block count.
+ def reset_metadata(self) -> None:
+ """Reset all bookkeeping state: counters, block allocator, attention/mamba state.
+
+ This must be called after ``initialize_all_tensors()`` and after any
+ suspend/resume cycle to bring the context back to a clean state.
+ """
+
+ # Reset request/token counts.
+ self.total_request_count = 0
+ self.active_token_count = 0
+ self.lifetime_prefill_token_count = 0
+ self.paused_request_count = 0
+ self.batch_dimensions = InferenceBatchDimensions(
+ token_count=0, prefill_req_count=0, decode_req_count=0
+ )
+ self.padded_batch_dimensions = InferenceBatchDimensions(
+ token_count=0, prefill_req_count=0, decode_req_count=0
+ )
+ self.padded_active_token_count = 0
+ self.padded_active_request_count = 0
+ self.paused_tokens = None
+ self.paused_speculative_tokens = None
+
+ # Reset attention, mamba, and block allocator state.
self.reset_attention_state()
self.reset_mamba_state()
- self.block_allocator.reset()
+ self.kv_block_allocator.reset()
self.request_to_kv_block_ids.fill_(-1)
+ # Reset step counter and LRU clock
+ self.step_count = 0
+ self.prefix_cache_lru_clock = 0
+
# Reset chunked prefill state
self.chunked_prefill_request_id = -1
- self.has_explicit_chunked_prefill_req = False
self.num_prefill_requests = 0
self._using_cuda_graph_this_step = False
+ self.is_creating_cuda_graphs = False
self.padded_batch_dimensions = InferenceBatchDimensions(
token_count=0, prefill_req_count=0, decode_req_count=0
)
+ def reset(self) -> None:
+ """Reset entire context.
+
+ This method does:
+ - Fill all GPU tensors with sentinel values.
+ - Reset active/paused request/token counts to zero.
+ - Reset available blocks to entire memory.
+
+ This method is useful after cuda graph warmup iterations, where the
+ context's memory buffer is referenced by the cuda graph system and
+ cannot be deallocated.
+ """
+ self.reset_tensors()
+ self.reset_metadata()
+
+ # Reset Mamba cache state
+ if self.mamba_slot_allocator is not None:
+ self.mamba_slot_allocator.reset()
+
def current_input_and_position_ids(
self, *, num_warmup_tokens: Optional[int] = None
) -> Tuple[Tensor, Tensor]:
@@ -1510,6 +1746,9 @@ def current_input_and_position_ids(
(Tuple[Tensor, Tensor]) Flattened active input and position IDs.
"""
num_tokens = num_warmup_tokens or self.padded_active_token_count
+ assert num_tokens >= self.padded_batch_dimensions.decode_req_count * (
+ self.num_speculative_tokens + 1
+ )
return (
self.token_to_input_ids[:num_tokens].unsqueeze(0),
self.token_to_pos_ids[:num_tokens].unsqueeze(0),
@@ -1524,6 +1763,9 @@ def last_token_logits(self, logits: Tensor) -> Tensor:
Return:
(Tensor) Last token logits.
"""
+ paused = self.paused_request_count
+ total = self.total_request_count
+ query_lengths = self.request_query_lengths[paused:total]
# todo: @lmcafee, remove these asserts?
assert logits.size(0) == 1, f"logits.size(0) ({tuple(logits.shape)}) != 1"
@@ -1531,101 +1773,229 @@ def last_token_logits(self, logits: Tensor) -> Tensor:
f"logits.size(1) ({tuple(logits.shape)}) != "
f"padded_active_token_count ({self.padded_active_token_count})."
)
+ logits_2d = logits.squeeze(0)
+ last_token_idxs = torch.cumsum(query_lengths, dim=0) - 1
+ return logits_2d[last_token_idxs, :]
+
+ def _compute_prefix_match(
+ self, req: DynamicInferenceRequest, prefill_chunk_length: int
+ ) -> Tuple[list, int, int, int, int, int]:
+ """Compute prefix match results and skip counts for a request chunk.
- # Last token logits.
- logits = logits.squeeze(0)
- last_token_idxs = (
- torch.cumsum(
- self.request_query_lengths[self.paused_request_count : self.total_request_count],
- dim=0,
+ Shared by check_availability (budget checks) and add_request (execution).
+
+ Returns:
+ Tuple of (matched_block_ids, num_blocks_from_pool,
+ already_allocated_blocks, overall_required_blocks,
+ prefix_skip_tokens, effective_prefill_chunk_length).
+ """
+ finished = req.finished_chunk_token_count
+ already_allocated_blocks = (finished + self.block_size_tokens - 1) // self.block_size_tokens
+ overall_required_blocks = (
+ finished + prefill_chunk_length + self.block_size_tokens - 1
+ ) // self.block_size_tokens
+
+ # Fast path: skip all prefix matching when disabled.
+ if not self.enable_prefix_caching:
+ num_blocks_from_pool = max(0, overall_required_blocks - already_allocated_blocks)
+ return (
+ [],
+ num_blocks_from_pool,
+ already_allocated_blocks,
+ overall_required_blocks,
+ 0,
+ prefill_chunk_length,
)
- - 1
+
+ matched_block_ids, _ = self._find_kv_match_count(
+ req, already_allocated_blocks, overall_required_blocks
)
- last_token_logits = logits[last_token_idxs, :]
+ num_matched = len(matched_block_ids)
- return last_token_logits
+ block_aligned = finished % self.block_size_tokens == 0
+ if num_matched > 0 and block_aligned:
+ prefix_skip_tokens = min(num_matched * self.block_size_tokens, prefill_chunk_length - 1)
+ else:
+ prefix_skip_tokens = 0
- def check_availability(self, req: DynamicInferenceRequest) -> (bool, bool, bool):
+ # Hybrid models with Mamba caching: skip based on Mamba match count.
+ # Only applies to the first chunk (finished == 0); continuation chunks
+ # already had Mamba state restored during the first chunk.
+ if self.is_hybrid_model and self.mamba_slot_allocator is not None and finished == 0:
+ num_mamba_matched = getattr(req, '_mamba_num_matched_blocks', 0)
+ assert (
+ num_mamba_matched <= num_matched
+ ), f"Mamba match ({num_mamba_matched}) > KV match ({num_matched})"
+ if num_mamba_matched > 0 and block_aligned:
+ raw_skip = num_mamba_matched * self.block_size_tokens
+ if raw_skip >= prefill_chunk_length:
+ # Back off to previous block with cached Mamba state
+ mamba_map = self.mamba_slot_allocator.hash_to_block_id
+ backed_off_blocks = 0
+ for j in range(num_mamba_matched - 2, -1, -1):
+ if req.precomputed_block_hashes[j] in mamba_map:
+ backed_off_blocks = j + 1
+ break
+ prefix_skip_tokens = backed_off_blocks * self.block_size_tokens
+ else:
+ prefix_skip_tokens = raw_skip
+ else:
+ prefix_skip_tokens = 0
+ elif self.is_hybrid_model and finished == 0:
+ prefix_skip_tokens = 0
+
+ effective_prefill_chunk_length = prefill_chunk_length - prefix_skip_tokens
+ num_blocks_from_pool = max(
+ 0, overall_required_blocks - already_allocated_blocks - num_matched
+ )
+
+ return (
+ matched_block_ids,
+ num_blocks_from_pool,
+ already_allocated_blocks,
+ overall_required_blocks,
+ prefix_skip_tokens,
+ effective_prefill_chunk_length,
+ )
+
+ def check_availability(self, req: DynamicInferenceRequest) -> Tuple[bool, bool, bool]:
"""
Check if the request can be added to the context.
"""
+ # Note that for hybrid models checking the total request count is sufficient
+ # because we allocate a single set of Mamba state tensors for each request
request_can_be_added = (
self.total_request_count < self.max_requests and self.paused_request_count == 0
)
+
+ (_, num_blocks_from_pool, _, _, _, effective_prefill_chunk_length) = (
+ self._compute_prefix_match(req, req.remaining_prompt_length)
+ )
+
request_tokens_can_be_added = (
- self.active_token_count + req.remaining_prompt_length <= self.max_tokens
+ self.active_token_count + effective_prefill_chunk_length <= self.max_tokens
)
- blocks = math.ceil(
- (req.remaining_prompt_length + req.finished_chunk_token_count) / self.block_size_tokens
- ) - math.ceil(req.finished_chunk_token_count / self.block_size_tokens)
- kv_cache_available = self.block_allocator.is_memory_available(blocks)
+ kv_cache_available = self.kv_block_allocator.is_memory_available(num_blocks_from_pool)
return request_can_be_added, request_tokens_can_be_added, kv_cache_available
- def add_request(self, req: DynamicInferenceRequest, chunk_length: Optional[int] = None) -> None:
+ def _find_kv_match_count(
+ self, req: DynamicInferenceRequest, start_block: int, end_block: int
+ ) -> tuple[list[int], int]:
+ """Find cached blocks matching a range of the prompt using precomputed hashes.
+
+ Looks up hashes in req.precomputed_block_hashes[start_block:end_block] against
+ the block allocator's hash-to-block mapping. Stops at the first non-match.
+
+ Args:
+ req: The inference request with precomputed_block_hashes set.
+ start_block: First block index to match (inclusive).
+ end_block: Last block index to match (exclusive); clamped to hash count.
+
+ Returns:
+ Tuple of:
+ - List of matched block IDs (consecutive from start_block)
+ - Parent hash of the last matched block (0 if no matches)
+ """
+ # Early return if prefix caching is disabled
+ if not self.enable_prefix_caching:
+ return [], 0
+
+ # Early return if request has no precomputed hashes
+ if not req.precomputed_block_hashes:
+ return [], 0
+
+ # Clamp end_block to the number of precomputed hashes (the trailing
+ # partial block has no hash).
+ end_block = min(end_block, len(req.precomputed_block_hashes))
+ if start_block >= end_block:
+ return [], 0
+
+ hashes = req.precomputed_block_hashes[start_block:end_block]
+ kv_hash_to_block = self.kv_block_allocator.kv_hash_to_block_id
+
+ # Find longest KV prefix by iterating block hashes from end.
+ # Parent-chained hashes guarantee: if hash at position N exists,
+ # all hashes 0..N also exist. So first match from end = longest prefix.
+ for i in range(len(hashes) - 1, -1, -1):
+ if hashes[i] in kv_hash_to_block:
+ num_matched = i + 1
+ matched_blocks = [kv_hash_to_block[hashes[j]] for j in range(num_matched)]
+ parent_hash = hashes[num_matched - 1]
+ return matched_blocks, parent_hash
+
+ return [], 0
+
+ def add_request(
+ self, req: DynamicInferenceRequest, prefill_chunk_length: Optional[int] = None
+ ) -> None:
"""Add request to context. At this stage, we assume that the request is valid and can be added, as the checks are done in the schedule function.
Args:
req (DynamicInferenceRequest): Request to add.
- chunk_length (Optional[int]): Length of chunk to add. If None, the request will be fully added.
+ prefill_chunk_length (Optional[int]): Length of prefill chunk to add. If None, the request will be fully added.
Return:
None
"""
-
# If tensor state is deallocated, do not add request.
if not self.is_tensor_state_allocated:
raise TensorStateDeallocatedError(req.request_id)
- # Chunk length.
- if chunk_length is None:
- chunk_length = req.remaining_prompt_length
-
- # req.finished_chunk_token_count > 0 means that the request is a scheduled chunked prefill request, and we are adding a chunk to it
- is_chunked_prefill = req.finished_chunk_token_count > 0
+ # Prefill chunk length.
+ if prefill_chunk_length is None:
+ prefill_chunk_length = req.remaining_prompt_length
- assert chunk_length > 0, "Chunk length is 0"
+ assert prefill_chunk_length > 0, "Chunk length is 0"
assert (
- chunk_length <= req.remaining_prompt_length
- ), "Chunk length is greater than remaining prompt length"
- if self.active_token_count + chunk_length > self.max_tokens:
- raise TokenOverflowError(req.request_id)
-
- # Use the remaining prompt tokens for this chunk
- this_round_tokens = req.remaining_prompt_tokens[:chunk_length]
-
- # only allocate new blocks
- already_allocated_blocks = (
- req.finished_chunk_token_count + self.block_size_tokens - 1
- ) // self.block_size_tokens # ceiling division
- overall_required_blocks = (
- req.finished_chunk_token_count + chunk_length + self.block_size_tokens - 1
- ) // self.block_size_tokens # ceiling division
-
- num_blocks_needed = overall_required_blocks - already_allocated_blocks
-
- if num_blocks_needed > 0:
- new_block_ids = self.block_allocator.allocate_memory_blocks(num_blocks_needed)
- if new_block_ids is None or len(new_block_ids) != num_blocks_needed:
+ prefill_chunk_length <= req.remaining_prompt_length
+ ), "Prefill chunk length is greater than remaining prompt length"
+
+ # =========================================================================
+ # Block allocation + prefix matching + prefill skipping
+ # =========================================================================
+ (
+ matched_block_ids,
+ num_blocks_from_pool,
+ already_allocated_blocks,
+ overall_required_blocks,
+ prefix_skip_tokens,
+ effective_prefill_chunk_length,
+ ) = self._compute_prefix_match(req, prefill_chunk_length)
+ num_matched_blocks = len(matched_block_ids)
+ effective_kv_offset = req.finished_chunk_token_count + prefix_skip_tokens
+
+ # Track prefix cache hits.
+ if num_matched_blocks > 0:
+ self.prefix_cache_hits += 1
+ self.prefix_cache_blocks_matched += num_matched_blocks
+
+ # Slice tokens to skip matched prefix
+ this_round_tokens = req.remaining_prompt_tokens[prefix_skip_tokens:prefill_chunk_length]
+
+ new_block_ids = None
+ if num_blocks_from_pool > 0:
+ new_block_ids = self.kv_block_allocator.allocate_memory_blocks(num_blocks_from_pool)
+ if new_block_ids is None or len(new_block_ids) != num_blocks_from_pool:
raise BlockOverflowError(req.request_id)
- # when a request already starts chunked prefill, it is exactly the last request in the current system
- # (see dynamic_engine.py, schedule_chunked_prefill invariants)
- # no need to update count, as it is already here
- if is_chunked_prefill:
- current_id = self.total_request_count - 1
- self.active_token_count -= (
- 1 # Overwrite the last token, which is the useless token from chunked prefill
+ # Increment ref counts and update timestamps for matched (shared) blocks
+ if num_matched_blocks > 0:
+ matched_tensor = torch.tensor(
+ matched_block_ids, dtype=torch.int32, device=torch.cuda.current_device()
)
- assert (
- self.request_ids[current_id] == req.request_id
- ), "Continuation current_id mismatch"
- else:
- current_id = self.total_request_count
+ self.kv_block_allocator.block_ref_counts[matched_tensor] += 1
+ if self.prefix_caching_eviction_policy == PrefixCachingEvictionPolicy.LRU:
+ self.kv_block_allocator.update_timestamps(matched_tensor)
+
+ # Note that we decremented the total_request_count for the chunked prefill request
+ # in update_requests, so setting current_id to the total_request_count will again
+ # make the last request the continuing chunked prefill request if one exists.
+ current_id = self.total_request_count
if current_id >= self.max_requests:
raise RequestOverflowError(req.request_id)
- if self.active_token_count + chunk_length > self.max_tokens:
+ if self.active_token_count + effective_prefill_chunk_length > self.max_tokens:
raise TokenOverflowError(req.request_id)
self.request_ids[current_id] = req.request_id
@@ -1648,74 +2018,137 @@ def add_request(self, req: DynamicInferenceRequest, chunk_length: Optional[int]
self.request_metadata[label][current_id] = m
# Handle length and block assignments.
- self.request_query_lengths[current_id] = chunk_length
+ self.request_query_lengths[current_id] = effective_prefill_chunk_length
+ self.request_in_prefill_status_tensor[current_id] = 1
self.request_output_lengths[current_id] = (
req.finished_chunk_token_count
- + chunk_length
+ + prefill_chunk_length
+ req.sampling_params.num_tokens_to_generate
)
- if num_blocks_needed > 0:
+
+ # Assign blocks: matched blocks at [already_allocated, already_allocated + num_matched),
+ # then newly allocated blocks after that.
+ match_start = already_allocated_blocks
+ new_block_start = already_allocated_blocks + num_matched_blocks
+ if num_matched_blocks > 0:
self.request_to_kv_block_ids[current_id][
- already_allocated_blocks:overall_required_blocks
+ match_start : match_start + num_matched_blocks
+ ] = matched_tensor
+ if new_block_ids is not None:
+ self.request_to_kv_block_ids[current_id][
+ new_block_start : new_block_start + len(new_block_ids)
] = new_block_ids
- self.request_kv_length_offsets[current_id] = req.finished_chunk_token_count
+
+ self.request_kv_length_offsets[current_id] = effective_kv_offset
self.request_kv_block_counts[current_id] = overall_required_blocks
self.request_last_kv_block_id[current_id] = self.request_to_kv_block_ids[current_id][
overall_required_blocks - 1
]
self.request_last_kv_block_offset[current_id] = (
- chunk_length + req.finished_chunk_token_count - 1
+ prefill_chunk_length + req.finished_chunk_token_count - 1
) % self.block_size_tokens
token_offset_range = torch.arange(
- req.finished_chunk_token_count,
- req.finished_chunk_token_count + chunk_length,
+ effective_kv_offset,
+ effective_kv_offset + effective_prefill_chunk_length,
device=self.token_to_pos_ids.device,
)
- self.token_to_pos_ids[self.active_token_count : self.active_token_count + chunk_length] = (
- token_offset_range
- )
+ self.token_to_pos_ids[
+ self.active_token_count : self.active_token_count + effective_prefill_chunk_length
+ ] = token_offset_range
self.token_to_input_ids[
- self.active_token_count : self.active_token_count + chunk_length
+ self.active_token_count : self.active_token_count + effective_prefill_chunk_length
] = this_round_tokens
self.token_to_request_idx[
- self.active_token_count : self.active_token_count + chunk_length
+ self.active_token_count : self.active_token_count + effective_prefill_chunk_length
] = current_id
self.token_to_position_in_request[
- self.active_token_count : self.active_token_count + chunk_length
+ self.active_token_count : self.active_token_count + effective_prefill_chunk_length
] = token_offset_range
self.token_to_block_idx[
- self.active_token_count : self.active_token_count + chunk_length
+ self.active_token_count : self.active_token_count + effective_prefill_chunk_length
] = self.request_to_kv_block_ids[current_id][token_offset_range // self.block_size_tokens]
self.token_to_local_position_within_kv_block[
- self.active_token_count : self.active_token_count + chunk_length
+ self.active_token_count : self.active_token_count + effective_prefill_chunk_length
] = (token_offset_range % self.block_size_tokens)
- if self.is_hybrid_model and not is_chunked_prefill:
+ # Register hashes for completely filled blocks (skip matched blocks).
+ # Two disjoint ranges may need registration:
+ # Range 1: [previously_complete, min(already_allocated_blocks, num_complete_blocks))
+ # — the partial block from a prior chunk that this chunk's tokens completed
+ # Range 2: [already_allocated_blocks + num_matched_blocks, num_complete_blocks)
+ # — newly allocated blocks that are now complete
+ if self.enable_prefix_caching and req.precomputed_block_hashes:
+ total_tokens_after = req.finished_chunk_token_count + prefill_chunk_length
+ num_complete_blocks = total_tokens_after // self.block_size_tokens
+ previously_complete = req.finished_chunk_token_count // self.block_size_tokens
+
+ def _register_range(start: int, end: int):
+ if start >= end:
+ return
+ block_ids_to_hash = self.request_to_kv_block_ids[current_id][start:end].tolist()
+ block_hashes_slice = req.precomputed_block_hashes[start:end]
+ self.kv_block_allocator.register_kv_block_hashes(
+ block_ids_to_hash, block_hashes_slice
+ )
+
+ # Range 1: prior-chunk partial block that this chunk just completed
+ _register_range(previously_complete, min(already_allocated_blocks, num_complete_blocks))
+ # Range 2: newly allocated (non-matched) blocks that are now complete
+ _register_range(already_allocated_blocks + num_matched_blocks, num_complete_blocks)
+
+ if self.is_hybrid_model and req.finished_chunk_token_count == 0:
# Allocate a slot for Mamba states
mamba_idx = self.mamba_metadata.allocate_slot()
if mamba_idx is None:
raise ContextOverflowError(req.request_id, "No Mamba slots available")
-
- # Initialize the allocated Mamba state
- self.mamba_conv_states[:, mamba_idx] = 0.0
- self.mamba_ssm_states[:, mamba_idx] = 0.0
self.mamba_metadata.request_to_mamba_state_idx[self.total_request_count] = mamba_idx
- self.active_token_count += chunk_length
- self.total_request_count += 0 if req.finished_chunk_token_count > 0 else 1
+ # Restore Mamba state from the block corresponding to prefix_skip_tokens
+ restore_block_count = prefix_skip_tokens // self.block_size_tokens
+ restored = False
+ if restore_block_count > 0 and self.mamba_slot_allocator is not None:
+ restore_block_id = matched_block_ids[restore_block_count - 1]
+ restored = self.mamba_slot_allocator.restore_to_live(
+ self.total_request_count, restore_block_id
+ )
+ if not restored:
+ self.mamba_conv_states[:, mamba_idx] = 0.0
+ self.mamba_ssm_states[:, mamba_idx] = 0.0
+
+ # Compute intermediate offsets for state extraction during forward pass
+ if self.mamba_slot_allocator is not None:
+ self.mamba_slot_allocator.compute_and_store_offsets(
+ req,
+ current_id,
+ prefix_skip_tokens,
+ prefill_chunk_length,
+ num_matched_blocks,
+ matched_block_ids,
+ overall_required_blocks,
+ )
+
+ self.active_token_count += effective_prefill_chunk_length
+ self.lifetime_prefill_token_count += effective_prefill_chunk_length
+ self.total_request_count += 1
self.num_prefill_requests += 1
- def _move_book_keeping_tensors(self, src_idxs, dst_idxs, next_tokens):
+ def _move_book_keeping_tensors(
+ self, src_idxs, dst_idxs, next_tokens, new_speculative_tokens=None
+ ):
"""
Move all the relevent booking tensors with src idxs to dst idxs
"""
self.request_kv_length_offsets[dst_idxs] = self.request_kv_length_offsets[src_idxs]
+ self.request_in_prefill_status_tensor[dst_idxs] = self.request_in_prefill_status_tensor[
+ src_idxs
+ ]
self.request_query_lengths[dst_idxs] = self.request_query_lengths[src_idxs]
self.request_output_lengths[dst_idxs] = self.request_output_lengths[src_idxs]
self.request_ids[dst_idxs] = self.request_ids[src_idxs]
- next_tokens[dst_idxs] = next_tokens[src_idxs]
-
+ next_tokens[dst_idxs] = next_tokens[src_idxs] # num tokens sames as num samples
+ if new_speculative_tokens is not None:
+ new_speculative_tokens[:, dst_idxs] = new_speculative_tokens[:, src_idxs]
self.request_to_kv_block_ids[dst_idxs] = self.request_to_kv_block_ids[src_idxs]
self.request_kv_block_counts[dst_idxs] = self.request_kv_block_counts[src_idxs]
self.request_last_kv_block_id[dst_idxs] = self.request_last_kv_block_id[src_idxs]
@@ -1729,33 +2162,61 @@ def _move_book_keeping_tensors(self, src_idxs, dst_idxs, next_tokens):
self.mamba_metadata.request_to_mamba_state_idx[src_idxs]
)
- def _swap_book_keeping_tensors(self, src_idxs, dst_idxs, next_tokens):
+ def _swap_book_keeping_tensors(
+ self, src_idxs, dst_idxs, next_tokens=None, new_speculative_tokens=None
+ ):
"""
Swaps all the relevent booking tensors with src idxs to dst idxs
"""
tensor_swap(self.request_kv_length_offsets, src_idxs, dst_idxs)
tensor_swap(self.request_query_lengths, src_idxs, dst_idxs)
+ tensor_swap(self.request_in_prefill_status_tensor, src_idxs, dst_idxs)
tensor_swap(self.request_output_lengths, src_idxs, dst_idxs)
tensor_swap(self.request_ids, src_idxs, dst_idxs)
- tensor_swap(next_tokens, src_idxs, dst_idxs)
tensor_swap(self.request_to_kv_block_ids, src_idxs, dst_idxs)
tensor_swap(self.request_kv_block_counts, src_idxs, dst_idxs)
tensor_swap(self.request_last_kv_block_id, src_idxs, dst_idxs)
tensor_swap(self.request_last_kv_block_offset, src_idxs, dst_idxs)
+ if next_tokens is not None:
+ tensor_swap(next_tokens, src_idxs, dst_idxs)
+
+ if new_speculative_tokens is not None:
+ # new_speculative_tokens has request dimension as second dimension,
+ # so swap on transposed view
+ tensor_swap(new_speculative_tokens.t(), src_idxs, dst_idxs)
+
for metadata_tensor in self.request_metadata.values():
tensor_swap(metadata_tensor, src_idxs, dst_idxs)
if self.is_hybrid_model:
tensor_swap(self.mamba_metadata.request_to_mamba_state_idx, src_idxs, dst_idxs)
- def get_index_of_chunked_prefill_request(self) -> int:
- """Get the index of the chunked prefill request in the context.
+ def get_index_of_chunked_prefill_request(self, safe: bool = True) -> int:
+ """
+ Get the index of the chunked prefill request in the context.
+
+ If `safe` is True, then clamp the search space to the current total request count.
+ Otherwise, expand the search beyond the current total request count.
Return:
(int) Index of the chunked prefill request, or -1 if none exists.
"""
- return torch.where(self.request_ids == self.chunked_prefill_request_id)[0][0]
+ if self.chunked_prefill_request_id == -1:
+ return -1
+
+ request_ids = self.request_ids
+ if safe:
+ request_ids = request_ids[: self.total_request_count]
+
+ matches = torch.where(request_ids == self.chunked_prefill_request_id)[0]
+ if len(matches) > 0:
+ return matches[0].item()
+ return -1
+
+ def is_chunked_prefill_enabled(self) -> bool:
+ """Returns whether chunked prefill is enabled."""
+ return self.enable_chunked_prefill
def release_memory_blocks_from_request_indexes(self, request_indexes) -> None:
"""Release memory blocks used by the given request idxs.
@@ -1766,7 +2227,7 @@ def release_memory_blocks_from_request_indexes(self, request_indexes) -> None:
"""
kv_blocks_assigned = self.request_to_kv_block_ids[request_indexes]
non_zero_values_in_kv_memory = kv_blocks_assigned[kv_blocks_assigned != -1]
- self.block_allocator.release_memory_blocks(non_zero_values_in_kv_memory)
+ self.kv_block_allocator.release_memory_blocks(non_zero_values_in_kv_memory)
# Reset the KV blocks for finished requests.
# Note: do not use fill_() (or add_() and similar inplace ops) here.
@@ -1779,12 +2240,17 @@ def release_memory_blocks_from_request_indexes(self, request_indexes) -> None:
if self.is_hybrid_model:
self.mamba_metadata.free_slots(request_indexes)
+ # Clear intermediate offset entries for released requests
+ if self.mamba_slot_allocator is not None:
+ sa = self.mamba_slot_allocator
+ sa._intermediate_counts_gpu[request_indexes] = 0
+ sa._intermediate_offsets_gpu[request_indexes] = 0
+ sa._intermediate_block_ids_gpu[request_indexes] = -1
+ sa._eos_cache_block_id_gpu[request_indexes] = -1
+
def resume_paused_requests(
- self,
- active_request_count: int,
- newly_paused_request_ids: torch.Tensor,
- next_tokens: torch.Tensor,
- ) -> tuple[int, int, torch.Tensor]:
+ self, active_request_count: int, newly_paused_request_ids: torch.Tensor
+ ) -> tuple[int, torch.Tensor]:
"""Resume as many paused requests as we have space for in the active buffer.
Args:
@@ -1800,56 +2266,61 @@ def resume_paused_requests(
# todo: @shanmugamr, un-pause requests using FIFO, rather than LIFO.
resume_request_count = 0
if self.paused_request_count > 0:
- active_block_count_avail = self.block_allocator.get_active_avail()
+ active_block_count_avail = self.kv_block_allocator.get_active_avail()
+ # Clone not needed: flip() makes a copy.
paused_block_counts = self.request_kv_block_counts[: self.paused_request_count]
# Flip counts before cumsum, since paused requests are resumed from
# the right-most index, so we must count resumed blocks starting from
# the right side.
paused_block_counts = paused_block_counts.flip(dims=[0])
- # Add +1 to all block counts, since any time a paused request is
- # resumed, it will be starting a new memory block. For background,
- # pausing happens after a request has generated the final token of a
- # memory block (i.e., token 256 of that block), which means the very
- # next token (whenever that request gets unpaused) will be in a new
- # block. So, when we resume a paused request, we have to account for
- # the fact that it will need an extra block beyond the ones that it
- # has already used.
- paused_block_counts += 1 # +1 for newly added block
+
+ # Check which paused requests will actually need a new block upon resuming
+ offsets = self.request_last_kv_block_offset[: self.paused_request_count]
+ needs_new_block = (
+ offsets >= self.block_size_tokens - 1 - self.num_speculative_tokens
+ ).to(paused_block_counts.dtype)
+ needs_new_block = needs_new_block.flip(dims=[0])
+
+ # Add +1 ONLY to the block counts of requests that finished their previous memory block
+ paused_block_counts += needs_new_block
paused_block_counts_cumsum = paused_block_counts.cumsum(dim=0)
resume_request_count = min(
torch.nonzero(paused_block_counts_cumsum <= active_block_count_avail).numel(),
- self.block_allocator.total_avail,
+ self.kv_block_allocator.total_avail,
+ )
+
+ # Constrain resumptions by the maximum allowed active requests and tokens
+ max_allowed_active = min(
+ self.max_requests, self.max_tokens // (self.num_speculative_tokens + 1)
)
+ allowed_to_resume = max(0, max_allowed_active - active_request_count)
+ resume_request_count = min(resume_request_count, allowed_to_resume)
self.paused_request_count -= resume_request_count
active_request_count += resume_request_count
# Resume requests by assigning blocks and updating bookkeeping tensors.
if resume_request_count > 0:
- assert torch.all(
- self.request_last_kv_block_offset[
- self.paused_request_count : (self.paused_request_count + resume_request_count)
- ]
- == self.block_size_tokens - 1
- ), "The request_last_kv_block_offset should be 0 for the requests that just got resumed this step."
+ resume_start = self.paused_request_count
+ resume_end = self.paused_request_count + resume_request_count
- assert resume_request_count <= self.block_allocator.total_avail
- block_ids = self.block_allocator.allocate_memory_blocks(resume_request_count)
- row_idx = torch.arange(
- self.paused_request_count,
- self.paused_request_count + resume_request_count,
- device=torch.cuda.current_device(),
- )
- col_idx = self.request_kv_block_counts[
- self.paused_request_count : (self.paused_request_count + resume_request_count)
- ]
- self.request_to_kv_block_ids[row_idx, col_idx] = block_ids
- self.request_kv_block_counts[
- self.paused_request_count : (self.paused_request_count + resume_request_count)
- ] += 1
- self.request_last_kv_block_id[
- self.paused_request_count : (self.paused_request_count + resume_request_count)
- ] = block_ids
+ # Check which resumed requests actually need a new block
+ offsets = self.request_last_kv_block_offset[resume_start:resume_end]
+ needs_new_block = offsets >= (self.block_size_tokens - 1 - self.num_speculative_tokens)
+ num_new_blocks = needs_new_block.sum().item()
+
+ if num_new_blocks > 0:
+ assert num_new_blocks <= self.kv_block_allocator.total_avail
+ block_ids = self.kv_block_allocator.allocate_memory_blocks(num_new_blocks)
+
+ # Apply updates only to the requests that required a new block
+ relative_row_idx = torch.nonzero(needs_new_block).squeeze(1)
+ row_idx = resume_start + relative_row_idx
+ col_idx = self.request_kv_block_counts[row_idx]
+
+ self.request_to_kv_block_ids[row_idx, col_idx] = block_ids
+ self.request_kv_block_counts[row_idx] += 1
+ self.request_last_kv_block_id[row_idx] = block_ids
# Remove resumed requests from newly_paused_request_ids. We do this by
# truncating the end of newly_paused_request_ids, which works because we
@@ -1862,8 +2333,11 @@ def resume_paused_requests(
return active_request_count, newly_paused_request_ids
def evict_overflow_paused_requests(
- self, active_request_count: int, next_tokens: torch.Tensor
- ) -> tuple[torch.Tensor, torch.Tensor]:
+ self,
+ active_request_count: int,
+ next_tokens: torch.Tensor,
+ new_speculative_tokens: Optional[torch.Tensor] = None,
+ ) -> Optional[tuple[torch.Tensor, torch.Tensor]]:
"""Evict requests that overflow the paused buffer.
Args:
@@ -1876,7 +2350,7 @@ def evict_overflow_paused_requests(
# Overflow paused block count.
overflow_paused_block_count = (
- self.block_allocator.get_paused_used() - self.block_allocator.paused_count
+ self.kv_block_allocator.get_paused_used() - self.kv_block_allocator.paused_count
)
# Nothing to evict?
@@ -1887,7 +2361,7 @@ def evict_overflow_paused_requests(
paused_block_counts = self.request_kv_block_counts[: self.paused_request_count]
paused_block_counts_cumsum = paused_block_counts.cumsum(dim=0)
valid_paused_request_count = torch.nonzero(
- paused_block_counts_cumsum <= self.block_allocator.paused_count
+ paused_block_counts_cumsum <= self.kv_block_allocator.paused_count
).numel()
overflow_paused_request_count = self.paused_request_count - valid_paused_request_count
@@ -1917,6 +2391,8 @@ def evict_overflow_paused_requests(
evict_request_idxs = torch.arange(
evict_start_idx, evict_end_idx, device=torch.cuda.current_device()
)
+ # Clone needed: subsequent release_memory_blocks_from_request_indexes and
+ # _swap_book_keeping_tensors calls mutate self.request_ids in place.
evict_request_ids = self.request_ids[evict_start_idx:evict_end_idx].clone()
# Release memory.
@@ -1951,7 +2427,10 @@ def evict_overflow_paused_requests(
# Swap evicted and active requests.
self._swap_book_keeping_tensors(
- src_idxs=src_idxs, dst_idxs=dst_idxs, next_tokens=next_tokens
+ src_idxs=src_idxs,
+ dst_idxs=dst_idxs,
+ next_tokens=next_tokens,
+ new_speculative_tokens=new_speculative_tokens,
)
# Update tracking vars.
@@ -1968,7 +2447,12 @@ def evict_overflow_paused_requests(
return evict_request_ids
- def update_requests(self, active_requests_mask: Tensor, new_tokens: Tensor) -> Tensor:
+ def update_requests(
+ self,
+ active_requests_mask: Tensor,
+ new_tokens: Tensor,
+ new_speculative_tokens: Tensor = None,
+ ) -> Tensor:
"""Update context state after calling engine.step().
This method is responsible for:
@@ -2001,23 +2485,31 @@ def update_requests(self, active_requests_mask: Tensor, new_tokens: Tensor) -> T
8. We make relevant changes to the token bookkeeping tensors
Args:
- active_requests_mask (Tensor): 1D Mask tensor marking active requests.
- new_tokens (Tensor): Newly sampled tokens, with one token per active request.
+ active_requests_mask (Tensor): 1D Mask tensor marking active requests. (Active request length)
+ new_tokens (Tensor): Newly sampled tokens, with one token per active request. (Active request length)
+ new_speculative_tokens (Tensor): Newly sampled speculative tokens,
+ with num_speculative tokens per active request.
+ (num_speculative_tokens, active_request_length)
Return:
(Tensor) Newly paused request IDs.
"""
-
# 1. The active token mask tells us which requests are still active and which are completed
# active_request_count -> This corresponds to requests that have not reached EOD or max length
# finished_request_count are requests that have reached the termination criterion
self.num_prefill_requests = 0 # all turns to decode
- if self.chunked_prefill_request_id != -1:
- active_requests_mask[-1] = (
- 1 # must keep this, next iteration will add a new chunk to it
- )
- self.has_explicit_chunked_prefill_req = False
+ # All request that were in prefill become decode requests.
+ # For the chunked prefill request we will overwrite this the next time add_request
+ # is called on that request.
+ self.request_in_prefill_status_tensor[self.request_in_prefill_status_tensor == 1] = 0
+
+ if (
+ chunked_prefill_request_idx := self.get_index_of_chunked_prefill_request(safe=True)
+ ) != -1:
+ # Chunked prefill request was active this step.
+ # We must keep it active so that the next iteration will add a new chunk to it.
+ active_requests_mask[-1] = 1
active_request_count = (active_requests_mask == 1).sum().item()
finished_request_count = (active_requests_mask == 0).sum().item()
@@ -2033,7 +2525,11 @@ def update_requests(self, active_requests_mask: Tensor, new_tokens: Tensor) -> T
self.total_request_count = active_request_count + self.paused_request_count
# 2. If no paused requests are present and no active requests we release memory and reset.
- if active_request_count + self.paused_request_count == 0:
+ # Note that this requires no pending chunked prefill request
+ if (
+ active_request_count + self.paused_request_count == 0
+ and self.get_index_of_chunked_prefill_request(safe=False) == -1
+ ):
if finished_request_count > 0:
finished_idxs = (
torch.nonzero(active_requests_mask == 0, as_tuple=True)[0]
@@ -2054,6 +2550,10 @@ def update_requests(self, active_requests_mask: Tensor, new_tokens: Tensor) -> T
if self.paused_request_count != 0:
assert self.paused_tokens is not None
next_tokens = torch.cat((self.paused_tokens, new_tokens))
+ if new_speculative_tokens is not None and self.paused_speculative_tokens is not None:
+ new_speculative_tokens = torch.cat(
+ (self.paused_speculative_tokens, new_speculative_tokens), dim=1
+ )
else:
next_tokens = new_tokens
@@ -2084,6 +2584,7 @@ def update_requests(self, active_requests_mask: Tensor, new_tokens: Tensor) -> T
src_idxs=active_idxs_on_right,
dst_idxs=finished_idxs_on_left,
next_tokens=next_tokens,
+ new_speculative_tokens=new_speculative_tokens,
)
# Reset chunk ids for recently moved requests.
@@ -2101,14 +2602,23 @@ def update_requests(self, active_requests_mask: Tensor, new_tokens: Tensor) -> T
self.paused_request_count : (active_request_count + self.paused_request_count)
]
active_requests_requiring_new_block = (
- num_tokens_in_last_block == self.block_size_tokens - 1
+ num_tokens_in_last_block >= self.block_size_tokens - 1 - self.num_speculative_tokens
).byte()
- if self.chunked_prefill_request_id != -1:
- # find the id in request_ids that is the chunked_prefill_request_id. Only one request should be chunked.
+ # Find the id in request_ids that is the chunked_prefill_request_id. Only one request should be chunked.
+ if (
+ chunked_prefill_request_idx := self.get_index_of_chunked_prefill_request(safe=True)
+ ) != -1:
active_requests_requiring_new_block[
- self.get_index_of_chunked_prefill_request() - self.paused_request_count
+ chunked_prefill_request_idx - self.paused_request_count
] = 0 # chunked prefill should not be paused
+ else:
+ max_allowed_active = min(
+ self.max_requests, self.max_tokens // (self.num_speculative_tokens + 1)
+ )
+ if active_request_count > max_allowed_active:
+ # Force-pause excess requests in a decode-only batch
+ active_requests_requiring_new_block[max_allowed_active:] = 1
active_requests_requiring_new_block_count = (
(active_requests_requiring_new_block == 1).sum().item()
@@ -2149,7 +2659,10 @@ def update_requests(self, active_requests_mask: Tensor, new_tokens: Tensor) -> T
dst_idxs = torch.cat((active_request_ids_on_left, paused_requests_idxs_on_right))
src_idxs = torch.cat((paused_requests_idxs_on_right, active_request_ids_on_left))
self._move_book_keeping_tensors(
- src_idxs=src_idxs, dst_idxs=dst_idxs, next_tokens=next_tokens
+ src_idxs=src_idxs,
+ dst_idxs=dst_idxs,
+ next_tokens=next_tokens,
+ new_speculative_tokens=new_speculative_tokens,
)
self.paused_request_count += active_requests_requiring_new_block_count
@@ -2158,72 +2671,225 @@ def update_requests(self, active_requests_mask: Tensor, new_tokens: Tensor) -> T
# 6. Now that we have the requests in following order [Paused, Active, Finished]
# We determine how many requests we can resume and resume them
+ # For multi-token generation: store previous block IDs BEFORE resume allocates new blocks.
+ # This allows us to know which block tokens should go to if they don't cross the boundary.
+ # After resume_paused_requests, request_last_kv_block_id will be updated to the NEW block
+ # for resumed requests, but we need the OLD block for tokens that don't cross.
+ prev_last_block_ids = None
+ if self.num_speculative_tokens > 0:
+ # Clone needed: resume_paused_requests mutates request_last_kv_block_id
+ # (assigns new block IDs), but we need the old values later to determine
+ # which block tokens should go to when they don't cross a block boundary.
+ prev_last_block_ids = self.request_last_kv_block_id.clone()
+
# 6.a. First, resume temporarily paused requests.
active_request_count, newly_paused_request_ids = self.resume_paused_requests(
- active_request_count, newly_paused_request_ids, next_tokens
+ active_request_count, newly_paused_request_ids
)
# 6.b. Evict requests that overflow the paused buffer.
- evict_request_ids = self.evict_overflow_paused_requests(active_request_count, next_tokens)
+ evict_request_ids = self.evict_overflow_paused_requests(
+ active_request_count, next_tokens, new_speculative_tokens
+ )
# 6.c. Resume any additional requests.
active_request_count, newly_paused_request_ids = self.resume_paused_requests(
- active_request_count, newly_paused_request_ids, next_tokens
+ active_request_count, newly_paused_request_ids
)
- assert active_request_count > 0, "active_request_count == %d." % active_request_count
+ assert active_request_count > 0 or self.chunked_prefill_request_id != -1, (
+ "active_request_count == %d with no hidden chunked prefill." % active_request_count
+ )
# 6.d. Swap the chunked prefill request to the end of the active requests
# to obey the invariance.
- if self.chunked_prefill_request_id != -1:
- self._swap_book_keeping_tensors(
- src_idxs=torch.tensor([self.get_index_of_chunked_prefill_request()]),
- dst_idxs=torch.tensor([self.total_request_count - 1]),
- next_tokens=next_tokens,
- )
+ if (
+ chunked_prefill_request_idx := self.get_index_of_chunked_prefill_request(safe=False)
+ ) != -1:
+ if chunked_prefill_request_idx < self.total_request_count:
+ # Chunked prefill request was active this step.
+ # Swap to the end of active, then hide it out of bounds.
+ self._swap_book_keeping_tensors(
+ src_idxs=torch.tensor(
+ [chunked_prefill_request_idx], device=self.request_ids.device
+ ),
+ dst_idxs=torch.tensor(
+ [self.total_request_count - 1], device=self.request_ids.device
+ ),
+ next_tokens=next_tokens,
+ new_speculative_tokens=new_speculative_tokens,
+ )
+
+ # Explicitly decrement the active and total request counts here so that the chunked
+ # prefill request metadata is not updated. This will all be restored when the next
+ # chunk is added through add_request.
+ active_request_count -= 1
+ self.total_request_count -= 1
+ else:
+ # Chunked prefill request was inactive/hidden this step.
+ # Pull it to the new boundary so it doesn't drift.
+ if chunked_prefill_request_idx != self.total_request_count:
+ self._swap_book_keeping_tensors(
+ src_idxs=torch.tensor(
+ [chunked_prefill_request_idx], device=self.request_ids.device
+ ),
+ dst_idxs=torch.tensor(
+ [self.total_request_count], device=self.request_ids.device
+ ),
+ next_tokens=None, # Do not swap next_tokens as these indices are out of bounds
+ new_speculative_tokens=None,
+ )
# 7. We make changes to the request book keeping tesnsors and setup the tokens for next iteration
assert self.total_request_count == active_request_count + self.paused_request_count
- # All these active requests are in decode phase, so they need only 1 token per request
- self.active_token_count = active_request_count
- # Always the first section of token input ids are only used.
- self.token_to_input_ids[: self.active_token_count] = next_tokens[
- self.paused_request_count : self.total_request_count
- ]
-
if self.paused_request_count > 0:
- self.paused_tokens = next_tokens[: self.paused_request_count]
-
- # add_ and fill_ calls seems to work as intended with sliced indexing (i.e. x[3:5].add(...) or x[3:5].fill_)
- # but when another tensor is used for indexing, it does not work as expected (i.e. x[y] if x and y are torch tensors)
+ # Clone needed: next_tokens is a shared buffer that will be overwritten in
+ # the next iteration; paused_tokens must persist independently.
+ self.paused_tokens = next_tokens[: self.paused_request_count].clone()
+ if new_speculative_tokens is not None:
+ # Clone needed: same reason as paused_tokens above.
+ self.paused_speculative_tokens = new_speculative_tokens[
+ :, : self.paused_request_count
+ ].clone()
+
+ # add_ and fill_ calls seems to work as intended with sliced indexing
+ # (i.e. x[3:5].add(...) or x[3:5].fill_) but when another tensor is used
+ # for indexing, it does not work as expected (i.e. x[y] if x and y are torch tensors)
self.request_kv_length_offsets[self.paused_request_count : self.total_request_count].add_(
self.request_query_lengths[self.paused_request_count : self.total_request_count]
)
- self.request_query_lengths[self.paused_request_count : self.total_request_count].fill_(1)
- self.token_to_pos_ids[: self.active_token_count] = self.request_kv_length_offsets[
+
+ num_generated_tokens = 1 + self.num_speculative_tokens
+ self.request_query_lengths[self.paused_request_count : self.total_request_count].fill_(
+ num_generated_tokens
+ )
+
+ # Clone needed: old_offsets is reused later to compute raw_positions
+ # for block-boundary detection. The write-back on the next line overwrites the
+ # underlying tensor, so without clone the boundary-crossing logic would see the
+ # new offsets instead of the pre-update values.
+ old_offsets = self.request_last_kv_block_offset[
self.paused_request_count : self.total_request_count
- ]
+ ].clone()
self.request_last_kv_block_offset[self.paused_request_count : self.total_request_count] = (
- self.request_last_kv_block_offset[self.paused_request_count : self.total_request_count]
- + 1
+ old_offsets + num_generated_tokens
) % self.block_size_tokens
- # 8. We make relevant changes to the token bookkeeping tensors
+ self.active_token_count = active_request_count * num_generated_tokens
+ sampled_tokens = next_tokens[self.paused_request_count : self.total_request_count]
+
+ if self.num_speculative_tokens > 0:
+ # new_speculative_tokens has shape [num_spec_tokens, num_requests],
+ # slice the request dimension (dim 1)
+ sampled_speculative_tokens = new_speculative_tokens[
+ :, self.paused_request_count : self.total_request_count
+ ]
+ # This will become [sampled, spec1, spec2, sampled, spec1, spec2 ...]
+ # For every request we will have the sampled token followed by the
+ # speculative tokens (i.e next indices)
+ next_tokens = torch.vstack(
+ [sampled_tokens.unsqueeze(0), sampled_speculative_tokens]
+ ).T.reshape(-1)
+ else:
+ next_tokens = sampled_tokens
+
+ self.token_to_input_ids[: self.active_token_count] = next_tokens
+
+ # Req kv length offsets : [0, 5, 10 ... ]
+ # For num spec tokens = 2 , this will become [0, 1, 2, 5, 6, 7 10, 11, 12 ...]
+ self.token_to_pos_ids[: self.active_token_count] = self.request_kv_length_offsets[
+ self.paused_request_count : self.total_request_count
+ ].repeat_interleave(num_generated_tokens) + torch.arange(
+ num_generated_tokens, device=torch.cuda.current_device()
+ ).repeat(
+ active_request_count
+ )
+ #
+ # Token to request idx : [0, 0, 0, 1, 1, 1, 2, 2, 2 ...]
self.token_to_request_idx[: self.active_token_count] = torch.arange(
self.paused_request_count, self.total_request_count, device=torch.cuda.current_device()
- )
- self.token_to_position_in_request[: self.active_token_count] = (
- self.request_kv_length_offsets[self.paused_request_count : self.total_request_count]
+ ).repeat_interleave(num_generated_tokens)
+
+ self.token_to_position_in_request[: self.active_token_count] = self.token_to_pos_ids[
+ : self.active_token_count
+ ]
+
+ self.token_to_local_position_within_kv_block[: self.active_token_count] = (
+ self.token_to_pos_ids[: self.active_token_count] % self.block_size_tokens
)
- self.token_to_block_idx[: self.active_token_count] = self.request_last_kv_block_id[
+ current_block_ids = self.request_last_kv_block_id[
self.paused_request_count : self.total_request_count
]
- self.token_to_local_position_within_kv_block[: self.active_token_count] = (
- self.request_last_kv_block_offset[self.paused_request_count : self.total_request_count]
+
+ # raw positions shape : [active_request_count, num_generated_tokens]
+ # e.g block size 6, old_offsets = [1,5,2] , num_generated_tokens = 3
+ # raw_positions = [[1, 2, 3], [5, 6, 7], [2, 3, 4]]
+ # crosses_boundary = [[False, False, False], [False, True, True], [False, False, False]]
+ raw_positions = (
+ old_offsets[:, None]
+ + 1 # Offset by 1 because old_offsets points to the LAST token
+ + torch.arange(num_generated_tokens, device=torch.cuda.current_device())[None, :]
)
+ #
+ # A token crosses to the next block if its raw_position >= block_size
+ crosses_boundary = raw_positions >= self.block_size_tokens
+
+ if not crosses_boundary.any() or self.num_speculative_tokens == 0:
+ # Fast path: no tokens cross block boundary, all use current block
+ self.token_to_block_idx[: self.active_token_count] = self.request_last_kv_block_id[
+ self.paused_request_count : self.total_request_count
+ ].repeat_interleave(num_generated_tokens)
+ else:
+
+ # Some tokens cross to the next block (this happens for resumed requests)
+ #
+ # When a request is paused and resumed:
+ # 1. It was paused because remaining_space < num_tokens_per_step
+ # 2. A NEW block is allocated in resume_paused_requests
+ # 3. request_last_kv_block_id is updated to the NEW block
+ # 4. The old offset is preserved (wasn't reset)
+ #
+ # So for resumed requests:
+ # - Tokens before the boundary (raw_pos < block_size): go to PREVIOUS block
+ # - Tokens at/after the boundary (raw_pos >= block_size): go to CURRENT (new) block
+ #
+ # For non-resumed requests (no boundary crossing): all go to current block
+ #
+ # We use prev_last_block_ids which was stored BEFORE resume_paused_requests
+ # was called, so it contains the OLD block IDs before new blocks were allocated.
+
+ # Get previous block IDs (stored before resume_paused_requests)
+ prev_block_ids = prev_last_block_ids[
+ self.paused_request_count : self.total_request_count
+ ] # [active_count]
+
+ # For each request, check if ANY token crosses (i.e., request was resumed)
+ request_has_crossing = crosses_boundary.any(dim=1) # [active_count]
+
+ # Build block_idx: [active_count, N]
+ # Start with current (new) block for all
+ # Lets say current block ids is [a1, a2 , a3] and num generated_tokens is 3
+ # This will be [[a1, a1, a1], [a2, a2, a2], [a3, a3, a3]]
+ # No clone needed: expand() returns a read-only view, and downstream
+ # torch.where() and .flatten() both return new tensors without in-place mutation.
+ block_idx = current_block_ids[:, None].expand(
+ -1, num_generated_tokens
+ ) # [active_count, N]
+
+ # For requests that have crossing, tokens BEFORE boundary use prev block
+ # crosses_boundary is False for tokens before boundary
+ # So: where request_has_crossing AND NOT crosses_boundary, use prev_block
+ use_prev_block = request_has_crossing[:, None] & ~crosses_boundary # [active_count, N]
+
+ # Apply previous block IDs where needed
+ prev_block_ids_expanded = prev_block_ids[:, None].expand(-1, num_generated_tokens)
+ block_idx = torch.where(use_prev_block, prev_block_ids_expanded, block_idx)
+
+ # Convert back to 1d tensor
+ self.token_to_block_idx[: self.active_token_count] = block_idx.flatten()
return {
"newly_paused_request_ids": newly_paused_request_ids,
@@ -2285,6 +2951,7 @@ def calculate_log_probs(
active_query_lengths = self.request_query_lengths[
self.paused_request_count : self.total_request_count
]
+
new_token_idx = active_query_lengths.cumsum(0) - 1
active_token_ids[new_token_idx] = new_tokens
@@ -2321,11 +2988,11 @@ def get_kvcache_utilization_stats(self) -> dict:
}
"""
# Total usable blocks exclude the reserved dummy block.
- total_blocks = max(self.block_allocator.total_count - 1, 1)
- block_count_avail = int(self.block_allocator.total_avail)
+ total_blocks = max(self.kv_block_allocator.total_count - 1, 1)
+ block_count_avail = int(self.kv_block_allocator.total_avail)
# Overall allocated blocks in the buffer right now.
- allocated_blocks = (self.block_allocator.total_count - 1) - block_count_avail
+ allocated_blocks = (self.kv_block_allocator.total_count - 1) - block_count_avail
allocated_blocks = int(max(0, allocated_blocks))
# Active unique blocks referenced by current active requests only.
@@ -2361,11 +3028,3 @@ def get_kvcache_utilization_stats(self) -> dict:
'total_request_count': int(total_request_count),
'max_requests': int(self.max_requests),
}
-
- def maybe_initialize_symmetric_memory(self):
- """
- Initializes symmetric memory for inference, if not already initialized
- """
- if not self.is_symmetric_memory_initialized:
- parallel_state._set_global_symmetric_memory_buffer()
- self.is_symmetric_memory_initialized = True
diff --git a/megatron/core/inference/contexts/fused_kv_append_kernel.py b/megatron/core/inference/contexts/fused_kv_append_kernel.py
index db1eed456e1..18ede4a027f 100644
--- a/megatron/core/inference/contexts/fused_kv_append_kernel.py
+++ b/megatron/core/inference/contexts/fused_kv_append_kernel.py
@@ -1,9 +1,22 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
-import triton
-import triton.language as tl
from torch import Tensor
+try:
+ import triton
+ import triton.language as tl
+
+ HAVE_TRITON = True
+except ImportError:
+ from unittest.mock import MagicMock
+
+ from megatron.core.utils import null_decorator
+
+ triton = MagicMock()
+ triton.jit = null_decorator
+ tl = MagicMock()
+ HAVE_TRITON = False
+
@triton.jit
def _append_kv_cache_kernel(
diff --git a/megatron/core/inference/contexts/kv_block_allocator.py b/megatron/core/inference/contexts/kv_block_allocator.py
new file mode 100644
index 00000000000..87039835c7f
--- /dev/null
+++ b/megatron/core/inference/contexts/kv_block_allocator.py
@@ -0,0 +1,360 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+
+from collections import deque
+from typing import Callable, Dict, Optional
+
+import torch
+from torch import Tensor
+
+from megatron.core.inference.config import PrefixCachingEvictionPolicy
+
+
+class KVBlockAllocator:
+ """Allocator that manages blocks of memory for the KV cache.
+
+ This allocator is responsible for:
+ - Initializing a pool of block IDs
+ - Allocating blocks from the pool
+ - Releasing blocks back to the pool
+
+ Args:
+ context (DynamicInferenceContext): Dynamic inference context.
+ total_count (int): Total number of blocks in the buffer.
+ paused_count (int): Number of paused blocks in the buffer. Must be less
+ than `total_count`.
+ """
+
+ def __init__(
+ self,
+ context: "DynamicInferenceContext",
+ total_count: int,
+ paused_count: int,
+ enable_prefix_caching: bool = False,
+ prefix_caching_eviction_policy: PrefixCachingEvictionPolicy = (
+ PrefixCachingEvictionPolicy.REF_ZERO
+ ),
+ ):
+
+ self.context = context
+ self.enable_prefix_caching = enable_prefix_caching
+ self.prefix_caching_eviction_policy = prefix_caching_eviction_policy
+ self.on_blocks_deregistered: Optional[Callable] = None
+
+ self.total_count = total_count
+ self.total_avail = total_count - 1 # -1 for dummy_block_idx (see below)
+ self.paused_count = paused_count
+ self.active_count = total_count - paused_count - 1 # -1 for dummy_block_idx
+ assert self.active_count >= 1 # ensures paused_count < total_count - 1
+ self.dummy_block_idx = self.total_count - 1
+
+ # Initialize block pool as a "stack" data structure
+ self.block_bag = torch.arange(
+ self.total_count, dtype=torch.int32, device=torch.cuda.current_device()
+ )
+
+ if self.enable_prefix_caching:
+ # Block hash tracking for prefix caching: -1 = uncomputed, positive = valid hash
+ self.block_hashes = torch.full(
+ (self.total_count,), -1, dtype=torch.int64, device=torch.cuda.current_device()
+ )
+
+ # Hash-to-block mapping for O(1) prefix lookup
+ self.kv_hash_to_block_id: Dict[int, int] = {}
+
+ # Reference count per block: 0 = cached (evictable), >0 = actively used
+ self.block_ref_counts = torch.zeros(
+ (self.total_count,), dtype=torch.int32, device=torch.cuda.current_device()
+ )
+
+ # LRU timestamps for eviction ordering (higher = more recently used)
+ # Only needed in LRU mode; RZ mode evicts immediately on ref_count==0
+ if self.prefix_caching_eviction_policy == PrefixCachingEvictionPolicy.LRU:
+ self.block_timestamps = torch.zeros(
+ (self.total_count,), dtype=torch.int64, device=torch.cuda.current_device()
+ )
+
+ def __str__(self):
+ return (
+ f"using: total {self.get_total_used()}/{self.total_count - 1}"
+ f"; active {self.get_active_used()}/{self.active_count}"
+ f"; paused {self.get_paused_used()}/{self.paused_count}"
+ )
+
+ def get_total_used(self):
+ """Compute number of total blocks used."""
+ return self.total_count - self.total_avail - 1
+
+ def get_active_used(self):
+ """Compute number of active blocks used."""
+ if not self.enable_prefix_caching:
+ return (
+ self.context.request_kv_block_counts[
+ self.context.paused_request_count : self.context.total_request_count
+ ]
+ .sum()
+ .item()
+ )
+
+ active_start = self.context.paused_request_count
+ active_end = self.context.total_request_count
+ if active_end > active_start:
+ active_rows = self.context.request_to_kv_block_ids[active_start:active_end]
+ valid_ids = active_rows[active_rows >= 0]
+ if valid_ids.numel() > 0:
+ return int(torch.unique(valid_ids).numel())
+ return 0
+
+ def get_paused_used(self):
+ """Compute number of paused blocks used."""
+ if not self.enable_prefix_caching:
+ return (
+ self.context.request_kv_block_counts[: self.context.paused_request_count]
+ .sum()
+ .item()
+ )
+
+ if self.context.paused_request_count > 0:
+ paused_rows = self.context.request_to_kv_block_ids[: self.context.paused_request_count]
+ valid_ids = paused_rows[paused_rows >= 0]
+ if valid_ids.numel() > 0:
+ return int(torch.unique(valid_ids).numel())
+ return 0
+
+ def get_active_avail(self):
+ """Compute number of active blocks available."""
+ return self.active_count - self.get_active_used()
+
+ def get_paused_avail(self):
+ """Compute number of paused blocks available."""
+ return self.paused_count - self.get_paused_used()
+
+ def is_memory_available(self, num_blocks: int) -> bool:
+ """Check if memory blocks are available.
+
+ Includes both free pool blocks and evictable cached blocks (ref_count == 0).
+
+ Args:
+ num_blocks (int): Number of blocks to check.
+
+ Return:
+ (bool) Is memory available?
+ """
+ # Fast path: avoid expensive evictable count computation when free pool suffices
+ if self.total_avail >= num_blocks:
+ return True
+ if not self.enable_prefix_caching:
+ return False
+ if self.prefix_caching_eviction_policy == PrefixCachingEvictionPolicy.REF_ZERO:
+ return False # RZ: no cached blocks to evict
+ # Also count evictable cached blocks
+ evictable_count = self.get_evictable_block_count()
+ return (self.total_avail + evictable_count) >= num_blocks
+
+ def allocate_memory_blocks(self, num_blocks: int) -> Optional[Tensor]:
+ """Allocate memory blocks if available, else return None.
+
+ Will attempt LRU eviction of cached blocks if the free pool is insufficient.
+
+ Args:
+ num_blocks (int): Number of blocks to allocate.
+
+ Return:
+ (Optional[Tensor]) Allocated block IDs.
+ """
+ # Try to evict cached blocks if free pool is insufficient
+ if self.total_avail < num_blocks:
+ if (
+ not self.enable_prefix_caching
+ or self.prefix_caching_eviction_policy == PrefixCachingEvictionPolicy.REF_ZERO
+ ):
+ return None # RZ: no eviction path; disabled: no cached blocks
+ blocks_needed_from_eviction = num_blocks - self.total_avail
+ if not self.evict_lru_blocks(blocks_needed_from_eviction):
+ return None # Not enough blocks even after eviction
+
+ # Now allocate from the free pool
+ self.total_avail -= num_blocks
+ block_ids = self.block_bag[self.total_avail : (self.total_avail + num_blocks)]
+ assert num_blocks == block_ids.numel()
+
+ if self.enable_prefix_caching:
+ # Initialize ref counts for newly allocated blocks
+ self.block_ref_counts[block_ids] = 1
+ if self.prefix_caching_eviction_policy == PrefixCachingEvictionPolicy.LRU:
+ self.update_timestamps(block_ids)
+
+ return block_ids
+
+ def release_memory_blocks(self, blocks: Tensor) -> None:
+ """Release memory blocks by decrementing reference counts.
+
+ Blocks with ref_count == 0 remain cached (in hash map) for potential reuse.
+ They will be evicted via LRU when space is needed.
+
+ Args:
+ blocks (Tensor): Block IDs to release.
+
+ Return:
+ None
+ """
+ if blocks.numel() == 0:
+ return
+
+ if self.enable_prefix_caching:
+ self.block_ref_counts[blocks] -= 1
+ if self.prefix_caching_eviction_policy == PrefixCachingEvictionPolicy.REF_ZERO:
+ zero_mask = self.block_ref_counts[blocks] == 0
+ if zero_mask.any():
+ self._deregister_blocks(blocks[zero_mask])
+ elif self.prefix_caching_eviction_policy == PrefixCachingEvictionPolicy.LRU:
+ # Unregistered blocks (hash == -1, ref_count == 0) have no hash
+ # entry to preserve for reuse (e.g., partial blocks at the end of
+ # a request). Return them directly to the free pool so they are not
+ # leaked.
+ unreg_mask = (self.block_ref_counts[blocks] == 0) & (
+ self.block_hashes[blocks] == -1
+ )
+ if unreg_mask.any():
+ unreg_blocks = blocks[unreg_mask]
+ num_unreg = unreg_blocks.numel()
+ self.block_bag[self.total_avail : self.total_avail + num_unreg] = unreg_blocks
+ self.total_avail += num_unreg
+ else:
+ num_blocks = blocks.numel()
+ self.block_bag[self.total_avail : self.total_avail + num_blocks] = blocks
+ self.total_avail += num_blocks
+
+ def reset(self) -> None:
+ """Reset the allocator to initial state.
+
+ This resets the available block count to the entire memory pool
+ (except for the dummy block).
+ """
+
+ # Reset block bag to so we start consuming from the beginning of the pool
+ # for UVM performance.
+ # *Note*: Resetting the block bag is essential because if engine has been
+ # suspended, then the block bag contains non-unique IDs since the
+ # right-most IDs have been 'popped' off and are owned by the context.
+ # Without resetting the block bag, context request memory will clash and
+ # requests will point to each other's memory blocks, resulting in faulty
+ # generations.
+ self.block_bag = torch.arange(
+ self.total_count, dtype=torch.int32, device=torch.cuda.current_device()
+ )
+
+ self.total_avail = self.total_count - 1
+
+ if self.enable_prefix_caching:
+ # Reset all block hashes
+ self.block_hashes.fill_(-1)
+
+ # Reset prefix caching state
+ self.kv_hash_to_block_id.clear()
+ self.block_ref_counts.fill_(0)
+ if self.prefix_caching_eviction_policy == PrefixCachingEvictionPolicy.LRU:
+ self.block_timestamps.fill_(0)
+
+ # =========================================================================
+ # Prefix caching methods
+ # =========================================================================
+
+ def register_kv_block_hashes(self, block_ids: list[int], block_hashes: list[int]) -> None:
+ """Register blocks in the hash-to-block mapping for discovery (batch).
+
+ Args:
+ block_ids: List of block IDs.
+ block_hashes: List of computed hash values (same length as block_ids).
+ """
+ if not block_ids:
+ return
+ id_tensor = torch.tensor(block_ids, dtype=torch.int64, device=self.block_hashes.device)
+ hash_tensor = torch.tensor(block_hashes, dtype=torch.int64, device=self.block_hashes.device)
+ self.block_hashes[id_tensor] = hash_tensor
+ self.kv_hash_to_block_id.update(zip(block_hashes, block_ids))
+
+ def _deregister_blocks(self, block_ids: Tensor) -> None:
+ """Remove blocks from prefix caching state and return to free pool.
+
+ Shared cleanup logic for both LRU eviction and RZ proactive eviction.
+
+ Args:
+ block_ids: Tensor of block IDs to deregister.
+ """
+ num_blocks = block_ids.numel()
+ if num_blocks == 0:
+ return
+
+ # Gather hashes via batched tensor indexing
+ block_ids_i64 = block_ids.to(torch.int64)
+ hashes = self.block_hashes[block_ids_i64].tolist()
+
+ # Remove from kv_hash_to_block_id dict (set ops + C-level map, no Python loop)
+ keys_to_delete = set(hashes) - {-1}
+ deque(
+ map(self.kv_hash_to_block_id.pop, keys_to_delete & self.kv_hash_to_block_id.keys()),
+ maxlen=0,
+ )
+
+ # Notify Mamba slot allocator (if wired) to clean up its state
+ if self.on_blocks_deregistered is not None:
+ self.on_blocks_deregistered(block_ids.tolist(), keys_to_delete)
+
+ # Reset block state (batched tensor ops)
+ self.block_hashes[block_ids] = -1
+ self.block_ref_counts[block_ids] = 0
+ if self.prefix_caching_eviction_policy == PrefixCachingEvictionPolicy.LRU:
+ self.block_timestamps[block_ids] = 0
+
+ # Return blocks to free pool
+ self.block_bag[self.total_avail : self.total_avail + num_blocks] = block_ids
+ self.total_avail += num_blocks
+
+ def update_timestamps(self, block_ids: Tensor) -> None:
+ """Update LRU timestamps for accessed blocks. No-op in RZ mode.
+
+ Args:
+ block_ids: Tensor of block IDs that were accessed.
+ """
+ if (
+ self.prefix_caching_eviction_policy != PrefixCachingEvictionPolicy.LRU
+ or block_ids.numel() == 0
+ ):
+ return
+ self.block_timestamps[block_ids] = self.context.prefix_cache_lru_clock
+
+ def get_evictable_block_count(self) -> Tensor:
+ """Get count of cached blocks that can be evicted (ref_count == 0, hash set).
+
+ Returns:
+ Scalar tensor with the number of evictable cached blocks.
+ """
+ cached_mask = (self.block_ref_counts == 0) & (self.block_hashes != -1)
+ return cached_mask.sum()
+
+ def evict_lru_blocks(self, num_blocks_needed: int) -> bool:
+ """Evict LRU cached blocks to free up space in the pool.
+
+ Evicts blocks with ref_count == 0, starting with oldest timestamps.
+
+ Args:
+ num_blocks_needed: Number of blocks to evict.
+
+ Returns:
+ True if enough blocks were evicted, False otherwise.
+ """
+ # Find all cached blocks (ref_count == 0, hash != -1)
+ cached_mask = (self.block_ref_counts == 0) & (self.block_hashes != -1)
+ cached_block_ids = torch.nonzero(cached_mask, as_tuple=True)[0]
+
+ if cached_block_ids.numel() < num_blocks_needed:
+ return False # Not enough cached blocks to evict
+
+ # Sort by timestamp (ascending = oldest first)
+ cached_timestamps = self.block_timestamps[cached_block_ids]
+ sorted_indices = torch.argsort(cached_timestamps)
+ blocks_to_evict = cached_block_ids[sorted_indices[:num_blocks_needed]]
+
+ self._deregister_blocks(blocks_to_evict)
+
+ return True
diff --git a/megatron/core/inference/contexts/mamba_slot_allocator.py b/megatron/core/inference/contexts/mamba_slot_allocator.py
new file mode 100644
index 00000000000..d7c57046c8a
--- /dev/null
+++ b/megatron/core/inference/contexts/mamba_slot_allocator.py
@@ -0,0 +1,614 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+
+from typing import TYPE_CHECKING, Dict
+
+import torch
+from torch import Tensor
+
+from megatron.core.inference.config import PrefixCachingEvictionPolicy
+
+if TYPE_CHECKING:
+ from .dynamic_context import DynamicInferenceContext
+
+# Maximum intermediate state extraction offsets per request. The 3 candidates
+# are: KV divergence boundary, last block-aligned boundary, and penultimate
+# block boundary (see compute_and_store_offsets for details).
+MAX_INTERMEDIATE_OFFSETS_PER_REQUEST = 3
+
+
+class MambaSlotAllocator:
+ """Manages Mamba state caching for prefix caching in hybrid models.
+
+ Owns the Mamba cache slot pool, block-to-slot mappings, hash-to-block
+ mapping, and intermediate state tracking. Accesses KV allocator state
+ (ref counts, timestamps, block hashes) via the parent context.
+
+ Args:
+ context: The DynamicInferenceContext that owns this allocator.
+ max_slots: Maximum number of cache slots.
+ num_mamba_layers: Number of Mamba layers in the model.
+ conv_states_shape: Shape of per-slot conv state (excluding layer/slot dims).
+ ssm_states_shape: Shape of per-slot SSM state (excluding layer/slot dims).
+ conv_states_dtype: Dtype for conv state tensors.
+ ssm_states_dtype: Dtype for SSM state tensors.
+ """
+
+ def __init__(
+ self,
+ context: "DynamicInferenceContext",
+ max_slots: int,
+ num_mamba_layers: int,
+ conv_states_shape: tuple,
+ ssm_states_shape: tuple,
+ conv_states_dtype: torch.dtype,
+ ssm_states_dtype: torch.dtype,
+ ):
+ self.context = context
+ self.max_slots = max_slots
+ self.num_mamba_layers = num_mamba_layers
+
+ device = torch.cuda.current_device()
+ num_blocks = context.kv_block_allocator.total_count
+
+ # Block <-> slot mappings
+ self.block_to_slot = torch.full((num_blocks,), -1, dtype=torch.int32, device=device)
+ self.slot_to_block = torch.full((max_slots,), -1, dtype=torch.int32, device=device)
+
+ # Free slot pool (stack)
+ self.free_slots = torch.arange(max_slots, dtype=torch.int32, device=device)
+ self.free_count = max_slots
+
+ # State tensors
+ self.conv_states = torch.zeros(
+ (num_mamba_layers, max_slots) + conv_states_shape,
+ dtype=conv_states_dtype,
+ device=device,
+ )
+ self.ssm_states = torch.zeros(
+ (num_mamba_layers, max_slots) + ssm_states_shape, dtype=ssm_states_dtype, device=device
+ )
+
+ # Hash-to-block mapping: only blocks with cached Mamba state
+ self.hash_to_block_id: Dict[int, int] = {}
+
+ # Per-request intermediate state storage (GPU tensors, fixed-size per request)
+ # 0 = no offset, -1 = no block
+ k = MAX_INTERMEDIATE_OFFSETS_PER_REQUEST
+ self._intermediate_offsets_gpu = torch.zeros(
+ (context.max_requests, k), dtype=torch.int32, device=device
+ )
+ self._intermediate_block_ids_gpu = torch.full(
+ (context.max_requests, k), -1, dtype=torch.int32, device=device
+ )
+ self._intermediate_counts_gpu = torch.zeros(
+ context.max_requests, dtype=torch.int32, device=device
+ )
+ self._eos_cache_block_id_gpu = torch.full(
+ (context.max_requests,), -1, dtype=torch.int32, device=device
+ )
+ # CPU flag to skip GPU sync when no intermediates exist
+ self._has_intermediates = False
+
+ # Pre-allocated output buffers for CUDA graph compatible extraction
+ self.max_intermediate_count = MAX_INTERMEDIATE_OFFSETS_PER_REQUEST * context.max_requests
+ self.intermediate_ssm_out = torch.zeros(
+ (num_mamba_layers, self.max_intermediate_count) + ssm_states_shape,
+ dtype=ssm_states_dtype,
+ device=device,
+ )
+ self.intermediate_conv_out = torch.zeros(
+ (num_mamba_layers, self.max_intermediate_count) + conv_states_shape,
+ dtype=conv_states_dtype,
+ device=device,
+ )
+
+ # =========================================================================
+ # Slot allocation
+ # =========================================================================
+
+ def allocate_slots_batch(self, block_ids: list) -> list:
+ """Get free Mamba cache slots for multiple blocks, evicting if necessary.
+
+ Handles deduplication: if the same block_id appears multiple times,
+ only one slot is allocated and all occurrences get the same slot.
+
+ Args:
+ block_ids: List of KV block IDs to associate with slots.
+
+ Returns:
+ List of allocated slot indices (same length as block_ids).
+ """
+ if not block_ids:
+ return []
+
+ device = self.block_to_slot.device
+ bid_tensor = torch.tensor(block_ids, dtype=torch.int64, device=device)
+
+ # Phase 1: Batch lookup existing slots (1 GPU sync)
+ existing_slots = self.block_to_slot[bid_tensor].tolist()
+
+ # Phase 2: Identify new blocks needing allocation (deduplicated)
+ # seen_new maps block_id -> index in new_bids list
+ seen_new = {}
+ new_bids = []
+ for i, (bid, slot) in enumerate(zip(block_ids, existing_slots)):
+ if slot < 0 and bid not in seen_new:
+ seen_new[bid] = len(new_bids)
+ new_bids.append(bid)
+
+ num_new = len(new_bids)
+ if num_new == 0:
+ return existing_slots
+
+ # Phase 3: Get slots from free pool, evicting if necessary
+ from_free = min(num_new, self.free_count)
+ new_slots = []
+ if from_free > 0:
+ start = self.free_count - from_free
+ new_slots = self.free_slots[start : self.free_count].tolist() # 1 GPU sync
+ self.free_count = start
+
+ need_evict = num_new - from_free
+ if need_evict > 0:
+ new_slots.extend(self._evict_lru_slots_batch(need_evict))
+
+ # Phase 4: Batch GPU writes for new mappings
+ new_bid_tensor = torch.tensor(new_bids, dtype=torch.int64, device=device)
+ new_slot_tensor = torch.tensor(new_slots, dtype=torch.int64, device=device)
+ self.block_to_slot[new_bid_tensor] = new_slot_tensor.to(torch.int32)
+ self.slot_to_block[new_slot_tensor] = new_bid_tensor.to(torch.int32)
+
+ # Phase 5: Build result mapping
+ # Map new block_ids to their allocated slots
+ alloc_bid_to_slot = {bid: slot for bid, slot in zip(new_bids, new_slots)}
+ result = []
+ for bid, existing in zip(block_ids, existing_slots):
+ if existing >= 0:
+ result.append(existing)
+ else:
+ result.append(alloc_bid_to_slot[bid])
+ return result
+
+ def _evict_lru_slots_batch(self, num_needed: int) -> list:
+ """Evict the least recently used Mamba cache slots.
+
+ Does NOT return slots to the free pool — caller takes ownership.
+
+ Args:
+ num_needed: Number of slots to evict.
+
+ Returns:
+ List of freed slot indices.
+ """
+ kv_alloc = self.context.kv_block_allocator
+ # Find blocks that have mamba slots and ref_count == 0
+ has_slot_mask = self.block_to_slot[: kv_alloc.total_count] >= 0
+ ref_zero_mask = kv_alloc.block_ref_counts[: kv_alloc.total_count] == 0
+ candidates = has_slot_mask & ref_zero_mask
+ candidate_ids = torch.nonzero(candidates, as_tuple=True)[0]
+
+ if candidate_ids.numel() < num_needed:
+ raise RuntimeError("No evictable Mamba cache slots available")
+
+ # Pick oldest blocks by timestamp (LRU) or first N (REF_ZERO)
+ if self.context.prefix_caching_eviction_policy == PrefixCachingEvictionPolicy.LRU:
+ timestamps = kv_alloc.block_timestamps[candidate_ids]
+ sorted_indices = torch.argsort(timestamps)[:num_needed]
+ evict_ids = candidate_ids[sorted_indices]
+ else:
+ evict_ids = candidate_ids[:num_needed]
+
+ # Batch gather slots + hashes (2 GPU syncs)
+ slots = self.block_to_slot[evict_ids].tolist()
+ hashes = kv_alloc.block_hashes[evict_ids].tolist()
+
+ # Batch cleanup GPU mappings
+ self.block_to_slot[evict_ids] = -1
+ slot_tensor = torch.tensor(slots, dtype=torch.int64, device=self.block_to_slot.device)
+ self.slot_to_block[slot_tensor] = -1
+
+ # Clean up hash dict (CPU loop)
+ for h in hashes:
+ if h > 0 and h in self.hash_to_block_id:
+ del self.hash_to_block_id[h]
+
+ return slots
+
+ def get_slot(self, block_id: int) -> int:
+ """Return the cache slot for a block, or -1 if none.
+
+ Args:
+ block_id: The KV block ID.
+
+ Returns:
+ Slot index or -1.
+ """
+ return self.block_to_slot[block_id].item()
+
+ def has_state(self, block_id: int) -> bool:
+ """Check if a block has cached Mamba state."""
+ return self.block_to_slot[block_id].item() >= 0
+
+ # =========================================================================
+ # Slot invalidation and deregistration
+ # =========================================================================
+
+ def invalidate_block(self, block_id: int) -> None:
+ """Free cache slot and clear mappings for a block.
+
+ Args:
+ block_id: The KV block ID.
+ """
+ slot = self.block_to_slot[block_id].item()
+ if slot < 0:
+ return
+ self.block_to_slot[block_id] = -1
+ self.slot_to_block[slot] = -1
+ # Return slot to free pool
+ self.free_slots[self.free_count] = slot
+ self.free_count += 1
+
+ def _invalidate_blocks_batch(self, block_ids_list: list) -> None:
+ """Free cache slots and clear mappings for multiple blocks at once.
+
+ Vectorized version of invalidate_block that avoids per-block .item()
+ GPU syncs. Used by on_kv_blocks_deregistered for bulk eviction.
+
+ Args:
+ block_ids_list: List of block IDs to invalidate.
+ """
+ if not block_ids_list:
+ return
+ bid_t = torch.tensor(block_ids_list, dtype=torch.int64, device=self.block_to_slot.device)
+ slots = self.block_to_slot[bid_t].to(torch.int64)
+ valid_mask = slots >= 0
+ if not valid_mask.any():
+ return
+ valid_slots = slots[valid_mask]
+ valid_bids = bid_t[valid_mask]
+ self.block_to_slot[valid_bids] = -1
+ self.slot_to_block[valid_slots] = -1
+ n = valid_slots.numel()
+ self.free_slots[self.free_count : self.free_count + n] = valid_slots.to(torch.int32)
+ self.free_count += n
+
+ def on_kv_blocks_deregistered(self, block_ids_list: list, hashes_to_delete: set) -> None:
+ """Handle KV block deregistration by cleaning up Mamba state.
+
+ Called by KVBlockAllocator._deregister_blocks via callback.
+
+ Args:
+ block_ids_list: List of deregistered block IDs.
+ hashes_to_delete: Set of hashes being deregistered (excludes -1).
+ """
+ if self.hash_to_block_id:
+ mamba_keys = hashes_to_delete & self.hash_to_block_id.keys()
+ if mamba_keys:
+ from collections import deque
+
+ deque(map(self.hash_to_block_id.pop, mamba_keys), maxlen=0)
+ self._invalidate_blocks_batch(block_ids_list)
+
+ # =========================================================================
+ # State store/restore
+ # =========================================================================
+
+ def store_from_tensors(
+ self, block_id: int, layer_idx: int, ssm_state: Tensor, conv_state: Tensor
+ ) -> None:
+ """Write provided state tensors to a cache slot for a specific layer.
+
+ Args:
+ block_id: The KV block ID.
+ layer_idx: The Mamba layer index.
+ ssm_state: SSM state tensor to store.
+ conv_state: Conv state tensor to store.
+ """
+ slot = self.block_to_slot[block_id].item()
+ assert slot >= 0, f"Block {block_id} has no Mamba cache slot"
+ self.ssm_states[layer_idx, slot].copy_(ssm_state)
+ self.conv_states[layer_idx, slot].copy_(conv_state)
+
+ def store_from_live_batch(self, slots: list, request_indices: list) -> None:
+ """Copy all layers from live per-request buffers to cache slots.
+
+ Args:
+ slots: List of cache slot indices.
+ request_indices: List of context request indices.
+ """
+ if not slots:
+ return
+ device = self.conv_states.device
+ slot_tensor = torch.tensor(slots, dtype=torch.int64, device=device)
+ req_tensor = torch.tensor(request_indices, dtype=torch.int64, device=device)
+ # Batch lookup mamba state indices (1 GPU sync)
+ mamba_indices = self.context.mamba_metadata.request_to_mamba_state_idx[req_tensor].tolist()
+ mamba_idx_tensor = torch.tensor(mamba_indices, dtype=torch.int64, device=device)
+ # Fancy-indexed copy (2 kernel launches instead of 2E)
+ self.conv_states[:, slot_tensor] = self.context.mamba_conv_states[:, mamba_idx_tensor]
+ self.ssm_states[:, slot_tensor] = self.context.mamba_ssm_states[:, mamba_idx_tensor]
+
+ def restore_to_live(self, request_idx: int, block_id: int) -> bool:
+ """Copy all layers from cache slot to live request state.
+
+ Args:
+ request_idx: The context request index.
+ block_id: The KV block ID.
+
+ Returns:
+ True if state was restored, False if block has no cached state.
+ """
+ slot = self.block_to_slot[block_id].item()
+ if slot < 0:
+ return False
+ mamba_idx = self.context.mamba_metadata.request_to_mamba_state_idx[request_idx].item()
+ self.context.mamba_conv_states[:, mamba_idx].copy_(self.conv_states[:, slot])
+ self.context.mamba_ssm_states[:, mamba_idx].copy_(self.ssm_states[:, slot])
+ return True
+
+ # =========================================================================
+ # Hash registration
+ # =========================================================================
+
+ def register_block_hashes_batch(self, block_ids: list, hashes: list) -> None:
+ """Register multiple blocks as having cached Mamba state.
+
+ Only registers entries where hash > 0.
+
+ Args:
+ block_ids: List of block IDs.
+ hashes: List of hash values (same length as block_ids).
+ """
+ updates = {h: bid for bid, h in zip(block_ids, hashes) if h > 0}
+ if updates:
+ self.hash_to_block_id.update(updates)
+
+ # =========================================================================
+ # Intermediate state tracking
+ # =========================================================================
+
+ def compute_and_store_offsets(
+ self,
+ req,
+ current_id: int,
+ skip_tokens: int,
+ prefill_chunk_length: int,
+ num_matched_blocks: int,
+ matched_block_ids: list,
+ overall_required_blocks: int,
+ ) -> None:
+ """Compute intermediate state extraction offsets and store per-request.
+
+ Args:
+ req: The inference request.
+ current_id: Context request index.
+ skip_tokens: Number of tokens being skipped (mamba match).
+ prefill_chunk_length: Total prefill chunk length before skipping.
+ num_matched_blocks: Number of KV-matched blocks.
+ matched_block_ids: List of matched KV block IDs.
+ overall_required_blocks: Total blocks needed for this request.
+ """
+ ctx = self.context
+ prompt_len = len(req.prompt_tokens)
+ num_kv_matched = num_matched_blocks
+ kv_div_abs = num_kv_matched * ctx.block_size_tokens
+ last_aligned_abs = (prompt_len // ctx.block_size_tokens) * ctx.block_size_tokens
+ seq_len = prefill_chunk_length - skip_tokens # effective prefill length
+
+ # Compute relative offsets (relative to prefill start after skip)
+ kv_div_rel = kv_div_abs - skip_tokens
+ last_aligned_rel = last_aligned_abs - skip_tokens
+ penultimate_abs = (overall_required_blocks - 1) * ctx.block_size_tokens
+ penultimate_rel = penultimate_abs - skip_tokens
+
+ # Determine mamba_chunk_size from mamba config (128 is the standard SSM kernel chunk size)
+ mamba_chunk_size = 128
+
+ # Build offset list: include if > 0, < seq_len, and % mamba_chunk_size == 0
+ offsets_set = set()
+ for offset in [kv_div_rel, last_aligned_rel, penultimate_rel]:
+ if offset > 0 and offset < seq_len and offset % mamba_chunk_size == 0:
+ offsets_set.add(offset)
+
+ offsets = sorted(offsets_set)
+ count = len(offsets)
+
+ # Vectorized block ID lookup: GPU gather avoids per-block .item() syncs
+ if count > 0:
+ device = self._intermediate_offsets_gpu.device
+ abs_tokens = torch.tensor(
+ [skip_tokens + o for o in offsets], dtype=torch.int64, device=device
+ )
+ block_indices = abs_tokens // ctx.block_size_tokens - 1
+ bids = ctx.request_to_kv_block_ids[current_id][block_indices]
+
+ self._intermediate_offsets_gpu[current_id, :count] = torch.tensor(
+ offsets, dtype=torch.int32, device=device
+ )
+ self._intermediate_block_ids_gpu[current_id, :count] = bids.to(torch.int32)
+ self._has_intermediates = True
+ self._intermediate_counts_gpu[current_id] = count
+
+ # Block-aligned EOS: prompt_len is exactly block-aligned
+ if last_aligned_abs == prompt_len and prompt_len > 0:
+ last_block_idx = prompt_len // ctx.block_size_tokens - 1
+ if last_block_idx >= 0:
+ self._eos_cache_block_id_gpu[current_id] = ctx.request_to_kv_block_ids[current_id][
+ last_block_idx
+ ]
+ self._has_intermediates = True
+ else:
+ self._eos_cache_block_id_gpu[current_id] = -1
+ else:
+ self._eos_cache_block_id_gpu[current_id] = -1
+
+ def get_intermediate_gpu_data(self):
+ """Get intermediate offsets and counts as GPU tensor slices for current prefill batch.
+
+ Returns:
+ Tuple of (offsets_gpu, counts_gpu) where:
+ offsets_gpu: [prefill_count, 3] int32 GPU tensor
+ counts_gpu: [prefill_count] int32 GPU tensor
+ Returns (None, None) if no prefill requests or no intermediates.
+ """
+ if not self._has_intermediates:
+ return None, None
+
+ ctx = self.context
+ prefill_count = ctx.batch_dimensions.prefill_req_count
+ if prefill_count == 0:
+ return None, None
+
+ active_start = ctx.paused_request_count
+ decode_count = ctx.batch_dimensions.decode_req_count
+ prefill_start = active_start + decode_count
+
+ offsets = self._intermediate_offsets_gpu[prefill_start : prefill_start + prefill_count]
+ counts = self._intermediate_counts_gpu[prefill_start : prefill_start + prefill_count]
+ return offsets, counts
+
+ # =========================================================================
+ # Intermediate state commit
+ # =========================================================================
+
+ def commit_intermediate_states(self) -> None:
+ """Commit intermediate states from pre-allocated output buffers to cache.
+
+ Called after the forward pass (including CUDA graph replay) completes.
+ Batched pipeline: collect data, allocate slots, copy states, register hashes.
+ """
+ collected = self._collect_commit_data()
+ if collected is None:
+ return
+ intermediate_bids, src_offsets, eos_bids, eos_ctx_indices, all_hashes = collected
+
+ # Allocate all slots in one batch (intermediates + EOS)
+ all_bids = intermediate_bids + eos_bids
+ all_slots = self.allocate_slots_batch(all_bids)
+
+ # Copy intermediate states from output buffers to cache
+ n_intermediate = len(intermediate_bids)
+ self._copy_intermediate_to_cache(src_offsets, all_slots[:n_intermediate])
+
+ # Copy EOS states from live buffers to cache
+ self.store_from_live_batch(all_slots[n_intermediate:], eos_ctx_indices)
+
+ # Register hashes for all committed blocks
+ self.register_block_hashes_batch(all_bids, all_hashes)
+
+ self._clear_intermediate_state()
+
+ def _collect_commit_data(self):
+ """Extract commit data from GPU intermediate state tracking.
+
+ Returns:
+ Tuple of (intermediate_bids, src_offsets, eos_bids, eos_ctx_indices,
+ all_hashes) or None if nothing to commit. all_hashes covers
+ intermediate_bids + eos_bids in that order.
+ """
+ ctx = self.context
+ metadata = ctx.mamba_metadata
+ prefill_count = ctx.batch_dimensions.prefill_req_count
+ if prefill_count == 0:
+ self._clear_intermediate_state()
+ return None
+
+ active_start = ctx.paused_request_count
+ decode_count = ctx.batch_dimensions.decode_req_count
+ prefill_start = active_start + decode_count
+
+ # Batch-transfer block IDs and EOS block IDs from GPU (2 GPU syncs)
+ intermediate_count = metadata.intermediate_count
+ per_request_counts = metadata.per_request_intermediate_counts
+
+ all_block_ids_cpu = self._intermediate_block_ids_gpu[
+ prefill_start : prefill_start + prefill_count
+ ].tolist()
+ eos_bids_cpu = self._eos_cache_block_id_gpu[
+ prefill_start : prefill_start + prefill_count
+ ].tolist()
+
+ # Flatten intermediate block IDs and source offsets
+ intermediate_bids = []
+ src_offsets = []
+ if intermediate_count > 0:
+ ssm_offset = 0
+ for req_idx, count in enumerate(per_request_counts):
+ for j in range(count):
+ intermediate_bids.append(all_block_ids_cpu[req_idx][j])
+ src_offsets.append(ssm_offset + j)
+ ssm_offset += count
+
+ # Collect EOS block IDs and their context indices
+ eos_bids = []
+ eos_ctx_indices = []
+ for req_batch_idx in range(prefill_count):
+ eos_bid = eos_bids_cpu[req_batch_idx]
+ if eos_bid >= 0:
+ eos_bids.append(eos_bid)
+ eos_ctx_indices.append(prefill_start + req_batch_idx)
+
+ if not intermediate_bids and not eos_bids:
+ self._clear_intermediate_state()
+ return None
+
+ # Single batch hash fetch for all block IDs (1 GPU sync)
+ all_bids_for_hash = intermediate_bids + eos_bids
+ device = ctx.kv_block_allocator.block_hashes.device
+ bid_tensor = torch.tensor(all_bids_for_hash, dtype=torch.int64, device=device)
+ all_hashes = ctx.kv_block_allocator.block_hashes[bid_tensor].tolist()
+
+ return intermediate_bids, src_offsets, eos_bids, eos_ctx_indices, all_hashes
+
+ def _copy_intermediate_to_cache(self, src_offsets: list, slots: list) -> None:
+ """Copy intermediate states from output buffers to cache slots.
+
+ Uses fancy-indexed GPU D2D copy (2 kernel launches instead of 2N).
+
+ Args:
+ src_offsets: Source indices into intermediate_ssm_out/intermediate_conv_out.
+ slots: Destination cache slot indices.
+ """
+ if not src_offsets:
+ return
+ device = self.ssm_states.device
+ src_idx = torch.tensor(src_offsets, dtype=torch.int64, device=device)
+ dst_idx = torch.tensor(slots, dtype=torch.int64, device=device)
+ self.ssm_states[:, dst_idx] = self.intermediate_ssm_out[:, src_idx]
+ self.conv_states[:, dst_idx] = self.intermediate_conv_out[:, src_idx]
+
+ def _clear_intermediate_state(self) -> None:
+ """Clear all per-request intermediate state tracking."""
+ ctx = self.context
+ prefill_count = ctx.batch_dimensions.prefill_req_count
+ if prefill_count > 0:
+ active_start = ctx.paused_request_count
+ decode_count = ctx.batch_dimensions.decode_req_count
+ prefill_start = active_start + decode_count
+ end = prefill_start + prefill_count
+ self._intermediate_counts_gpu[prefill_start:end].fill_(0)
+ self._intermediate_offsets_gpu[prefill_start:end].fill_(0)
+ self._intermediate_block_ids_gpu[prefill_start:end].fill_(-1)
+ self._eos_cache_block_id_gpu[prefill_start:end].fill_(-1)
+ self._has_intermediates = False
+
+ # =========================================================================
+ # Reset
+ # =========================================================================
+
+ def reset(self) -> None:
+ """Reset all state (mappings, free pool, cache, intermediate tracking)."""
+ self.block_to_slot.fill_(-1)
+ self.slot_to_block.fill_(-1)
+ self.free_slots = torch.arange(
+ self.max_slots, dtype=torch.int32, device=torch.cuda.current_device()
+ )
+ self.free_count = self.max_slots
+ self.hash_to_block_id.clear()
+ self.intermediate_ssm_out.zero_()
+ self.intermediate_conv_out.zero_()
+ self._intermediate_offsets_gpu.fill_(0)
+ self._intermediate_block_ids_gpu.fill_(-1)
+ self._intermediate_counts_gpu.fill_(0)
+ self._eos_cache_block_id_gpu.fill_(-1)
+ self._has_intermediates = False
diff --git a/megatron/core/inference/contexts/routing_metadata.py b/megatron/core/inference/contexts/routing_metadata.py
new file mode 100644
index 00000000000..6664681a7ce
--- /dev/null
+++ b/megatron/core/inference/contexts/routing_metadata.py
@@ -0,0 +1,96 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+from typing import TYPE_CHECKING, Optional
+
+import torch
+
+if TYPE_CHECKING:
+ from megatron.core.inference.contexts.dynamic_context import DynamicInferenceContext
+
+from megatron.core.transformer.moe.router_replay import RouterReplay
+
+
+class RoutingMetadata:
+ """Manages routing indices metadata for MoE layers during inference.
+
+ This class provides static buffers for CUDA graph compatibility when
+ recording routing decisions. It holds a reference to the inference context
+ to automatically determine whether to use static buffers based on CUDA graph state.
+
+ Args:
+ context (DynamicInferenceContext): The inference context.
+ moe_router_topk (int): Number of experts selected per token.
+ """
+
+ def __init__(self, context: 'DynamicInferenceContext', moe_router_topk: int):
+ self.context = context
+ self.max_tokens = context.max_tokens
+ self.moe_router_topk = moe_router_topk
+ self.device = torch.cuda.current_device()
+
+ # Static buffer allocated lazily in _ensure_buffer_allocated().
+ # We defer allocation because RouterReplay instances don't exist yet at init time.
+ self.routing_indices_buffer: Optional[torch.Tensor] = None
+ self.num_moe_layers: Optional[int] = None
+
+ def _ensure_buffer_allocated(self) -> None:
+ """Allocate the static buffer if not already allocated.
+
+ Gets the actual number of MoE layers from RouterReplay instances.
+ """
+ if self.routing_indices_buffer is not None:
+ return
+
+ self.num_moe_layers = len(RouterReplay.global_router_replay_instances)
+
+ if self.num_moe_layers == 0:
+ return
+
+ # Static buffer for CUDA graph compatibility.
+ # Shape: [max_tokens, num_moe_layers, moe_router_topk]
+ self.routing_indices_buffer = torch.empty(
+ (self.max_tokens, self.num_moe_layers, self.moe_router_topk),
+ dtype=torch.int32,
+ device=self.device,
+ )
+
+ def get_routing_indices(self) -> Optional[torch.Tensor]:
+ """Get the recorded routing indices.
+
+ Automatically uses the static buffer when CUDA graphs are active,
+ otherwise retrieves from RouterReplay utility.
+
+ Returns:
+ Tensor of shape [num_tokens, num_moe_layers, topk] or None if not available.
+ """
+ if self.context.using_cuda_graph_this_step():
+ # Return view of static buffer up to current token count.
+ if self.routing_indices_buffer is None:
+ return None
+ # Only return up to active token count, to skip entries
+ # for padding tokens.
+ return self.routing_indices_buffer[: self.context.active_token_count]
+ else:
+ # Get from RouterReplay and stack into [num_tokens, num_layers, topk].
+ recorded_data = RouterReplay.get_recorded_data()
+ if recorded_data is None or len(recorded_data) == 0:
+ return None
+ if recorded_data[0] is None:
+ return None
+ # Stack: list of [num_tokens, topk] -> [num_tokens, num_layers, topk]
+ return torch.stack(recorded_data, dim=1)
+
+ def enable_static_buffer_recording(self) -> None:
+ """Enable recording into the static buffer for CUDA graph compatibility.
+
+ This sets up RouterReplay instances to copy routing indices into our
+ pre-allocated static buffer instead of creating new tensors.
+ Allocates the buffer lazily on first call.
+ """
+ self._ensure_buffer_allocated()
+ if self.routing_indices_buffer is not None:
+ RouterReplay.set_global_static_buffers(self.routing_indices_buffer)
+
+ def disable_static_buffer_recording(self) -> None:
+ """Disable static buffer recording, reverting to normal tensor assignment."""
+ RouterReplay.clear_global_static_buffers()
diff --git a/megatron/core/inference/contexts/static_context.py b/megatron/core/inference/contexts/static_context.py
index 8c83d2f09b3..a15b33c414a 100644
--- a/megatron/core/inference/contexts/static_context.py
+++ b/megatron/core/inference/contexts/static_context.py
@@ -1,8 +1,6 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
+from megatron.core.inference.config import InferenceConfig
from .base_context import BaseInferenceContext
@@ -19,7 +17,8 @@ class StaticInferenceContext(BaseInferenceContext):
def __init__(
self, max_batch_size: int, max_sequence_length: int, use_flashinfer_fused_rope: bool = None
):
- super().__init__(materialize_only_last_token_logits=True)
+ config = InferenceConfig(materialize_only_last_token_logits=True)
+ super().__init__(inference_config=config)
self.max_sequence_length = max_sequence_length
self.max_batch_size = max_batch_size
self.sequence_len_offset = 0
@@ -27,13 +26,6 @@ def __init__(
self.key_value_memory_dict = {}
self.decode_mode = False
- @classmethod
- def from_config(cls, config: InferenceWrapperConfig) -> "StaticInferenceContext":
- """Initialize context from a config."""
- max_batch_size = config.inference_max_requests
- max_sequence_length = config.inference_max_seq_length
- return cls(max_batch_size, max_sequence_length)
-
def swap_key_value_dict(self, batch_idx):
"swap between batches"
if len(self.key_value_memory_dict) == 0:
diff --git a/megatron/core/inference/data_parallel_inference_coordinator.py b/megatron/core/inference/data_parallel_inference_coordinator.py
index 9a1a11a8c2b..50f586cc598 100644
--- a/megatron/core/inference/data_parallel_inference_coordinator.py
+++ b/megatron/core/inference/data_parallel_inference_coordinator.py
@@ -2,17 +2,24 @@
import errno
import faulthandler
+import json
import logging
import signal
import socket
from collections import deque
-from itertools import cycle
+from enum import Enum, auto
from multiprocessing import Event
from multiprocessing.connection import Connection
+import numpy as np
import torch
+from megatron.core.inference.config import PrefixCachingCoordinatorPolicy
from megatron.core.inference.headers import Headers, UnknownHeaderError
+from megatron.core.inference.inference_request import compute_block_hashes_batched
+from megatron.core.inference.text_generation_controllers.text_generation_controller import (
+ TextGenerationController,
+)
try:
import zmq
@@ -68,12 +75,30 @@ class DataParallelInferenceCoordinator:
next_request_id (int): A counter for generating unique server-side request IDs.
"""
+ class CoordinatorState(Enum):
+ """State machine for the coordinator."""
+
+ RUNNING = auto()
+ PAUSED = auto()
+ SUSPENDED = auto()
+ STOPPING = auto()
+
def __init__(
self,
pipe_connection: Connection,
data_parallel_size: int,
tokenizer,
+ max_requests,
inference_coordinator_port: int | None = None,
+ deterministic_mode: bool = False,
+ block_size_tokens: int | None = None,
+ enable_prefix_caching: bool = False,
+ prefix_caching_coordinator_policy: PrefixCachingCoordinatorPolicy = (
+ PrefixCachingCoordinatorPolicy.FIRST_PREFIX_BLOCK
+ ),
+ prefix_caching_routing_alpha: float = 0.5,
+ schedule_output_path: str | None = None,
+ hostname: str | None = None,
):
"""
Initializes the inference coordinator.
@@ -88,6 +113,10 @@ def __init__(
expected to connect.
tokenizer: The tokenizer to use for prompt tokenization and detokenization.
inference_coordinator_port (Optional[int]): The TCP port number to bind the server to.
+ prefix_caching_routing_alpha (float): Weight for prefix-aware routing score:
+ score = alpha * match + (1 - alpha) * normalized_load.
+ max_requests (int): Max concurrent requests per rank, used to
+ compute normalized_load for prefix-aware scoring.
"""
assert HAVE_ZMQ, (
"please install the pyzmq library to use DataParallelInferenceCoordinator\n"
@@ -109,9 +138,11 @@ def __init__(
# the user that had submitted the request originally.
# Get local IP.
- local_ip = socket.gethostname()
+ local_ip = hostname or socket.gethostname()
self.router_socket = self.context.socket(zmq.ROUTER)
+ # Raise error if the other side of the connection has dropped.
+ self.router_socket.setsockopt(zmq.ROUTER_MANDATORY, 1)
is_bound = False
if inference_coordinator_port is not None:
try:
@@ -145,15 +176,50 @@ def __init__(
assert identity not in self.identities_of_data_parallel_ranks
self.identities_of_data_parallel_ranks.append(identity)
logging.info("Inference Coordinator: Connected with data parallel ranks...")
- self.data_parallel_rank_iterator = cycle(self.identities_of_data_parallel_ranks)
- self.data_parallel_pause_acks = set()
- self.data_parallel_stop_acks = set()
+
+ # In deterministic mode, sort identities for consistent scheduling order.
+ if deterministic_mode:
+ self.identities_of_data_parallel_ranks = deque(
+ sorted(self.identities_of_data_parallel_ranks)
+ )
+ self._round_robin_idx = 0
self.request_id_to_client_id = {}
self.request_id_to_client_request_id = {}
+ self.request_id_to_rank = {} # Maps request_id → rank identity for pending count tracking
self.next_request_id = 0
self.tokenizer = tokenizer
+ self.state = self.CoordinatorState.RUNNING
+
+ # Prefix caching state for routing.
+ self.block_size_tokens = block_size_tokens
+ self.enable_prefix_caching = enable_prefix_caching
+ self.prefix_caching_coordinator_policy = prefix_caching_coordinator_policy
+ self.prefix_caching_routing_alpha = prefix_caching_routing_alpha
+ self.max_requests = max_requests
+ assert self.max_requests is not None and self.max_requests > 0
+
+ # Schedule recording.
+ self.schedule_output_path = schedule_output_path
+ self.schedule_records = [] if schedule_output_path else None
+
+ # Deterministic rank index mapping (sorted identity -> 0-based index).
+ sorted_identities = sorted(self.identities_of_data_parallel_ranks)
+ self.identity_to_rank_index = {
+ identity: idx for idx, identity in enumerate(sorted_identities)
+ }
+
+ # Numpy arrays for vectorized scoring (indexed by rank index).
+ n_ranks = len(sorted_identities)
+ self._identities_list = list(sorted_identities) # rank_index → identity
+ self._pending_counts = np.zeros(n_ranks, dtype=np.int32)
+
+ # Hash → {rank_idx: timestamp} dict for prefix cache affinity routing.
+ # Each key is a block hash; each value maps rank indices to assignment
+ # timestamps (positive int). Missing entries are implicitly zero.
+ self._hash_table: dict[int, dict[int, int]] = {}
+ self._hash_assignment_counter = 0
def get_next_data_parallel_rank(self):
"""
@@ -162,7 +228,150 @@ def get_next_data_parallel_rank(self):
Returns:
bytes: The ZMQ identity of the next data parallel rank to receive a request.
"""
- return next(self.data_parallel_rank_iterator)
+ identities = self.identities_of_data_parallel_ranks
+ if not identities:
+ raise RuntimeError("No engines connected")
+ idx = self._round_robin_idx % len(identities)
+ self._round_robin_idx = idx + 1
+ return identities[idx]
+
+ def _register_rank_identity(self, identity):
+ """Register a new rank identity in the scoring data structures.
+
+ Called when a rank dynamically connects to a running coordinator
+ (e.g. in tests that spawn the coordinator with data_parallel_size=0
+ and let engines register after the fact).
+ """
+ if identity in self.identity_to_rank_index:
+ return
+ new_idx = len(self._identities_list)
+ self.identity_to_rank_index[identity] = new_idx
+ self._identities_list.append(identity)
+ self._pending_counts = np.append(self._pending_counts, np.int32(0))
+ logging.info(
+ "Coordinator: registered engine %s as rank index %d (now %d engines)",
+ identity,
+ new_idx,
+ len(self._identities_list),
+ )
+
+ def _remove_engine(self, identity):
+ """Remove a disconnected engine from the routing pool."""
+ self.identities_of_data_parallel_ranks.remove(identity)
+ logging.warning(
+ "Coordinator: removed engine %s (now %d engines)",
+ identity,
+ len(self.identities_of_data_parallel_ranks),
+ )
+
+ def _send_to_engine(self, identity, payload):
+ """Send payload to an engine, removing it from the pool if unreachable.
+
+ Returns:
+ True if the send succeeded, False if the engine was unreachable and removed.
+ """
+ try:
+ self.router_socket.send_multipart([identity, payload])
+ return True
+ except zmq.error.ZMQError as e:
+ if e.errno == zmq.EHOSTUNREACH:
+ self._remove_engine(identity)
+ return False
+ raise
+
+ def compute_request_hashes(self, prompt):
+ """Compute block hashes for a prompt on CPU.
+
+ Args:
+ prompt: Either a string (to be tokenized) or a list of token IDs.
+
+ Returns:
+ List of integer block hashes, or empty list if prefix caching is disabled.
+ """
+ if not self.enable_prefix_caching or self.block_size_tokens is None:
+ return []
+ if isinstance(prompt, str):
+ tokens = self.tokenizer.tokenize(prompt)
+ else:
+ tokens = list(prompt)
+ token_tensor = torch.tensor(tokens, dtype=torch.int64)
+ return compute_block_hashes_batched(token_tensor, self.block_size_tokens)
+
+ def get_best_data_parallel_rank(self, request_hashes):
+ """Select the best DP rank based on prefix cache affinity and load.
+
+ Uses a scoring function: score = alpha * match + (1 - alpha) * normalized_load
+ where *match* is a policy-dependent affinity score in [0, 1] (binary for
+ ``first_prefix_block``, normalized prefix depth for ``longest_prefix``)
+ and normalized_load = free_slots / max_requests (higher means more free
+ capacity).
+
+ Args:
+ request_hashes: List of block hashes for the request.
+
+ Returns:
+ bytes: The ZMQ identity of the selected data parallel rank.
+ """
+ if self.prefix_caching_coordinator_policy == PrefixCachingCoordinatorPolicy.ROUND_ROBIN:
+ return self.get_next_data_parallel_rank()
+
+ if not self.enable_prefix_caching or not request_hashes:
+ return self.get_next_data_parallel_rank()
+
+ match, recency = self._match_vector(request_hashes)
+
+ alpha = self.prefix_caching_routing_alpha
+
+ # Vectorized score: alpha * match + (1-alpha) * free_capacity_fraction.
+ free_slots = np.maximum(0, self.max_requests - self._pending_counts).astype(np.float64)
+ scores = alpha * match + (1.0 - alpha) * (free_slots / self.max_requests)
+
+ # Tiebreak: highest score, then highest recency, then lowest rank index.
+ n_ranks = len(self._identities_list)
+ order = np.lexsort((np.arange(n_ranks), -recency, -scores))
+ best_idx = int(order[0])
+ return self._identities_list[best_idx]
+
+ def _update_rank_hashes(self, rank_identity, request_hashes):
+ """Record that a rank owns the given hashes.
+
+ Args:
+ rank_identity: ZMQ identity of the target rank.
+ request_hashes: List of block hashes assigned to this rank.
+ """
+ rank_idx = self.identity_to_rank_index[rank_identity]
+ self._hash_assignment_counter += 1
+ ts = self._hash_assignment_counter
+ for h in request_hashes:
+ self._hash_table.setdefault(h, {})[rank_idx] = ts
+
+ def _match_vector(self, hashes):
+ """Return ``(match, recency)`` vectors of shape ``(n_ranks,)``.
+
+ *match* is binary depth: ``(depth + 1) / len(hashes)`` for ranks that
+ have the deepest cached block, 0 otherwise. *recency* is the raw
+ assignment timestamp for each matching rank (0 for non-matching ranks).
+
+ For ``FIRST_PREFIX_BLOCK`` the caller already truncates *hashes* to a
+ single element, so the same logic yields a binary 0/1 match score.
+ """
+ n_ranks = len(self._identities_list)
+ n = len(hashes)
+ zeros = np.zeros(n_ranks, dtype=np.float64)
+ if n == 0:
+ return zeros, zeros.copy()
+ for i in range(n - 1, -1, -1):
+ row = self._hash_table.get(hashes[i])
+ if row is None:
+ continue
+ rank_idxs = np.fromiter(row.keys(), dtype=np.intp)
+ present = np.zeros(n_ranks, dtype=bool)
+ present[rank_idxs] = True
+ recency = np.zeros(n_ranks, dtype=np.float64)
+ recency[rank_idxs] = np.fromiter(row.values(), dtype=np.float64)
+ if present.any():
+ return present.astype(np.float64) * ((i + 1.0) / n), recency
+ return zeros, zeros.copy()
def start(self):
"""
@@ -178,6 +387,14 @@ def start(self):
known_clients = set()
while True:
sender_identity, serialized_payload = self.router_socket.recv_multipart()
+
+ # Allow for re-registration if connecting to a running coordinator.
+ if serialized_payload == b"":
+ if sender_identity not in self.identities_of_data_parallel_ranks:
+ self.identities_of_data_parallel_ranks.append(sender_identity)
+ self._register_rank_identity(sender_identity)
+ continue
+
deserialized_payload = msgpack.unpackb(serialized_payload, raw=False)
header = Headers(deserialized_payload[0])
@@ -223,117 +440,164 @@ def start(self):
else:
raise Exception("specialize for <%s> prompt." % type(prompt).__name__)
- next_data_parallel_rank_identity = self.get_next_data_parallel_rank()
- self.router_socket.send_multipart(
- [
- next_data_parallel_rank_identity,
- msgpack.packb(
- [Headers.SUBMIT_REQUEST.value, request_id, prompt, sampling_params],
- use_bin_type=True,
- ),
- ]
+ payload = msgpack.packb(
+ [Headers.SUBMIT_REQUEST.value, request_id, prompt, sampling_params],
+ use_bin_type=True,
)
- elif header in [
+
+ request_hashes = self.compute_request_hashes(prompt)
+ if (
+ self.prefix_caching_coordinator_policy
+ == PrefixCachingCoordinatorPolicy.FIRST_PREFIX_BLOCK
+ ):
+ request_hashes = request_hashes[:1]
+
+ # Account for the fact that some engines may have died.
+ for _ in range(len(self.identities_of_data_parallel_ranks)):
+ next_identity = self.get_best_data_parallel_rank(request_hashes)
+ if self._send_to_engine(next_identity, payload):
+ break
+ else:
+ # If all engines have died, we are in an abnormal state, and must exit cleanly.
+ logging.error("Coordinator: no reachable engines for request %d", request_id)
+ del self.request_id_to_client_id[request_id]
+ del self.request_id_to_client_request_id[request_id]
+ return
+
+ self.request_id_to_rank[request_id] = next_identity
+ self._pending_counts[self.identity_to_rank_index[next_identity]] += 1
+ if request_hashes:
+ self._update_rank_hashes(next_identity, request_hashes)
+ if self.schedule_records is not None:
+ self.schedule_records.append(
+ {
+ "request_id": request_id,
+ "rank_index": self.identity_to_rank_index[next_identity],
+ "num_hashes": len(request_hashes),
+ }
+ )
+
+ elif header in (
Headers.PAUSE,
Headers.UNPAUSE,
Headers.SUSPEND,
Headers.RESUME,
+ Headers.SET_GENERATION_EPOCH,
Headers.STOP,
- ]:
- # control signals for the engine
- # broadcast to all data parallel ranks
+ ):
+ # Start by checking the current state against the control signal.
if sender_identity not in known_clients:
+ logging.warning("Coordinator: ignoring signal from unknown client.")
continue
- for data_parallel_rank_id in self.identities_of_data_parallel_ranks:
- self.router_socket.send_multipart(
- [data_parallel_rank_id, msgpack.packb([header.value], use_bin_type=True)]
- )
- if header == Headers.UNPAUSE:
- self.data_parallel_pause_acks = set()
- elif header == Headers.PAUSE_ACK:
- # control signal ack from the engine
- assert sender_identity in self.identities_of_data_parallel_ranks
- assert sender_identity not in self.data_parallel_pause_acks
- self.data_parallel_pause_acks.add(sender_identity)
- # route to all clients only once we have gotten an ack from all data parallel ranks
- if len(self.data_parallel_pause_acks) == self.data_parallel_size:
- for client_id in known_clients:
- self.router_socket.send_multipart(
- [
- client_id,
- msgpack.packb([header.value, sender_identity], use_bin_type=True),
- ]
- )
- for data_parallel_rank_id in self.identities_of_data_parallel_ranks:
- self.router_socket.send_multipart(
- [
- data_parallel_rank_id,
- msgpack.packb([Headers.PAUSE_ACK.value], use_bin_type=True),
- ]
- )
- elif header == Headers.STOP_ACK:
- # control signal ack from the engine
- assert sender_identity in self.identities_of_data_parallel_ranks
- assert sender_identity not in self.data_parallel_stop_acks
- self.data_parallel_stop_acks.add(sender_identity)
- # route to all clients only once we have gotten an ack from all data parallel ranks
- if len(self.data_parallel_stop_acks) == self.data_parallel_size:
- for client_id in known_clients:
- self.router_socket.send_multipart(
- [
- client_id,
- msgpack.packb([header.value, sender_identity], use_bin_type=True),
- ]
- )
- for data_parallel_rank_id in self.identities_of_data_parallel_ranks:
- self.router_socket.send_multipart(
- [
- data_parallel_rank_id,
- msgpack.packb([Headers.STOP_ACK.value], use_bin_type=True),
- ]
- )
- break # Exit the main loop after STOP_ACKs have been processed.
+
+ if header == Headers.PAUSE:
+ idem_states = (self.CoordinatorState.PAUSED, self.CoordinatorState.SUSPENDED)
+ if self.state == self.CoordinatorState.RUNNING:
+ self.state = self.CoordinatorState.PAUSED
+ elif self.state in idem_states:
+ # Already paused/suspended, ignore redundant PAUSE.
+ continue
+ else:
+ logging.warning("Coordinator: ignoring PAUSE in state %s", self.state)
+ continue
+ elif header == Headers.UNPAUSE:
+ if self.state != self.CoordinatorState.PAUSED:
+ logging.warning("Coordinator: ignoring UNPAUSE in state %s", self.state)
+ continue
+ self.state = self.CoordinatorState.RUNNING
+ elif header == Headers.SUSPEND:
+ if self.state != self.CoordinatorState.PAUSED:
+ logging.warning("Coordinator: ignoring SUSPEND in state %s", self.state)
+ continue
+ self.state = self.CoordinatorState.SUSPENDED
+ elif header == Headers.RESUME:
+ if self.state != self.CoordinatorState.SUSPENDED:
+ logging.warning("Coordinator: ignoring RESUME in state %s", self.state)
+ continue
+ self.state = self.CoordinatorState.PAUSED
+ elif header == Headers.STOP:
+ good_states = (self.CoordinatorState.PAUSED, self.CoordinatorState.SUSPENDED)
+ if self.state not in good_states:
+ logging.warning("Coordinator: ignoring STOP in state %s", self.state)
+ continue
+ self.state = self.CoordinatorState.STOPPING
+
+ # Broadcast the control signal if we're in a good state.
+ # Forward the full deserialized payload so that data-bearing
+ # signals (e.g. SET_GENERATION_EPOCH) retain their arguments.
+ broadcast_payload = msgpack.packb(deserialized_payload, use_bin_type=True)
+ for data_parallel_rank_id in list(self.identities_of_data_parallel_ranks):
+ self._send_to_engine(data_parallel_rank_id, broadcast_payload)
+
+ # STOP affects engines; reset coordinator to RUNNING to allow future engines.
+ if header == Headers.STOP:
+ self.state = self.CoordinatorState.RUNNING
+
elif header == Headers.ENGINE_REPLY:
# This is the output of a single engine step on some data parallel rank.
assert sender_identity in self.identities_of_data_parallel_ranks
- finished_request_records = deserialized_payload[1]
+ finished_requests = deserialized_payload[1]
- for finished_request_record in finished_request_records:
- self.detokenize(finished_request_record)
- fid = finished_request_record["requests"][0]["request_id"]
+ for finished_request in finished_requests:
+ self.detokenize(finished_request)
+ fid = finished_request["request_id"]
client_identity = self.request_id_to_client_id[fid]
client_request_identity = self.request_id_to_client_request_id[fid]
del self.request_id_to_client_id[fid]
del self.request_id_to_client_request_id[fid]
+ assigned_rank = self.request_id_to_rank.pop(fid, None)
+ if assigned_rank is not None:
+ idx = self.identity_to_rank_index.get(assigned_rank)
+ if idx is not None:
+ assert self._pending_counts[idx] >= 1
+ self._pending_counts[idx] -= 1
self.router_socket.send_multipart(
[
client_identity,
msgpack.packb(
- [header.value, client_request_identity, finished_request_record],
+ [header.value, client_request_identity, finished_request],
use_bin_type=True,
),
]
)
+ elif header == Headers.SHUTDOWN:
+ if sender_identity not in known_clients:
+ logging.warning("Coordinator: ignoring signal from unknown client.")
+ continue
+ break
+
+ elif header == Headers.DISCONNECT:
+ if sender_identity in self.identities_of_data_parallel_ranks:
+ self._remove_engine(sender_identity)
+
else:
raise UnknownHeaderError(header)
- def detokenize(self, finished_request_record):
+ def detokenize(self, finished_request):
"""
- Detokenizes the generated tokens in the finished request record.
+ Detokenizes the generated tokens in the finished request.
This method uses the coordinator's tokenizer to convert the list of
generated token IDs back into human-readable text.
Args:
- finished_request_record (dict): The record containing the generated
- tokens to be detokenized. It is modified in place.
+ finished_request (dict): The serialized merged request containing the
+ generated tokens to be detokenized. It is modified in place.
"""
- for request in finished_request_record["requests"]:
- if request["prompt"] is None:
- request["prompt"] = self.tokenizer.detokenize(request["prompt_tokens"][1])
- request["generated_text"] = self.tokenizer.detokenize(request["generated_tokens"])
+ if finished_request["prompt"] is None:
+ finished_request["prompt"] = TextGenerationController.detokenize(
+ self.tokenizer, finished_request["prompt_tokens"][1], remove_EOD=False
+ )
+ detokenize_stop_sequence = (finished_request.get("sampling_params", {}) or {}).get(
+ "detokenize_stop_sequence", False
+ )
+ finished_request["generated_text"] = TextGenerationController.detokenize(
+ self.tokenizer,
+ finished_request["generated_tokens"],
+ remove_EOD=not detokenize_stop_sequence,
+ )
@classmethod
def entrypoint(
@@ -342,7 +606,17 @@ def entrypoint(
ready_event: Event,
data_parallel_size: int,
tokenizer,
+ max_requests,
inference_coordinator_port: int | None = None,
+ deterministic_mode: bool = False,
+ block_size_tokens: int | None = None,
+ enable_prefix_caching: bool = False,
+ prefix_caching_coordinator_policy: PrefixCachingCoordinatorPolicy = (
+ PrefixCachingCoordinatorPolicy.FIRST_PREFIX_BLOCK
+ ),
+ prefix_caching_routing_alpha: float = 0.5,
+ schedule_output_path: str | None = None,
+ hostname: str | None = None,
):
"""
Class method to instantiate and run the coordinator, for use in a separate process.
@@ -356,9 +630,27 @@ def entrypoint(
once the coordinator is ready to accept connections.
inference_coordinator_port (int): The port to bind to.
data_parallel_size (int): The number of expected TP-coordinators.
+ deterministic_mode (bool): Whether to enable deterministic scheduling.
+ block_size_tokens (Optional[int]): Token block size for prefix caching hashing.
+ enable_prefix_caching (bool): Whether prefix caching is enabled.
+ prefix_caching_coordinator_policy (PrefixCachingCoordinatorPolicy): Routing policy.
+ schedule_output_path (Optional[str]): Path to write scheduling decisions JSON.
+ prefix_caching_routing_alpha (float): Weight for prefix-aware routing score.
+ max_requests (int): Max concurrent requests per rank.
"""
coordinator = cls(
- pipe_connection, data_parallel_size, tokenizer, inference_coordinator_port
+ pipe_connection,
+ data_parallel_size,
+ tokenizer,
+ max_requests,
+ inference_coordinator_port,
+ deterministic_mode=deterministic_mode,
+ block_size_tokens=block_size_tokens,
+ enable_prefix_caching=enable_prefix_caching,
+ prefix_caching_coordinator_policy=prefix_caching_coordinator_policy,
+ prefix_caching_routing_alpha=prefix_caching_routing_alpha,
+ schedule_output_path=schedule_output_path,
+ hostname=hostname,
)
ready_event.set()
try:
@@ -372,5 +664,14 @@ def stop(self):
"""
Stops the inference coordinator, performing any necessary cleanup operations.
"""
+ if self.schedule_output_path and self.schedule_records:
+ schedule_data = {
+ "policy": self.prefix_caching_coordinator_policy.value,
+ "data_parallel_size": self.data_parallel_size,
+ "num_requests": len(self.schedule_records),
+ "records": self.schedule_records,
+ }
+ with open(self.schedule_output_path, "w") as f:
+ json.dump(schedule_data, f, indent=2)
self.router_socket.close()
self.context.term()
diff --git a/megatron/core/inference/engines/async_zmq_communicator.py b/megatron/core/inference/engines/async_zmq_communicator.py
index 7076bb283bd..52570845d61 100644
--- a/megatron/core/inference/engines/async_zmq_communicator.py
+++ b/megatron/core/inference/engines/async_zmq_communicator.py
@@ -26,13 +26,20 @@ class AsyncZMQCommunicator:
on the CPU.
"""
- def __init__(self, zmq_context: zmq.Context, process_group: dist.ProcessGroup):
+ def __init__(
+ self,
+ zmq_context: zmq.Context,
+ process_group: dist.ProcessGroup,
+ hostname: str | None = None,
+ ):
"""
Constructor for AsyncZMQCommunicator. Sets up ZMQ sockets
for communication among ranks in the given process group.
Args:
zmq_context (zmq.Context): ZMQ context to create sockets.
process_group (dist.ProcessGroup): Process group for communication.
+ hostname (str | None): Hostname or IP address to use for ZMQ socket binding.
+ If None, defaults to socket.gethostname().
"""
self.rank = dist.get_rank(process_group)
self.world_size = dist.get_world_size(process_group)
@@ -41,7 +48,7 @@ def __init__(self, zmq_context: zmq.Context, process_group: dist.ProcessGroup):
src_rank = dist.get_process_group_ranks(process_group)[0]
if self.is_leader:
- local_ip = socket.gethostname()
+ local_ip = hostname or socket.gethostname()
self.gather_sock = zmq_context.socket(zmq.PULL)
self.gather_sock.bind_to_random_port(f"tcp://{local_ip}")
gather_socket_addr = self.gather_sock.getsockopt_string(zmq.LAST_ENDPOINT)
@@ -65,46 +72,70 @@ def __init__(self, zmq_context: zmq.Context, process_group: dist.ProcessGroup):
self.bcast_sock.connect(bcast_socket_addr)
self.bcast_sock.setsockopt_string(zmq.SUBSCRIBE, "")
- async def all_reduce_max(self, local_val: int) -> int:
- """
- Asyncio friendly all reduce max operation. Gathers on rank 0, computes max,
- and broadcasts the result.
+ async def all_reduce_max(self, *local_vals: int, async_op=True) -> int | tuple[int, ...]:
+ """Element-wise all-reduce max of one or more integers.
+
+ Packs all values into a single message so the communication cost
+ is independent of the number of values.
+
+ Returns a single int when called with one argument, otherwise a tuple.
"""
+ n = len(local_vals)
+ if n == 0:
+ raise ValueError("all_reduce_max requires at least one value")
+
if self.world_size <= 1:
- return local_val
+ return local_vals[0] if n == 1 else local_vals
- payload = struct.pack('!i', local_val)
+ fmt = f'!{n}i'
+ payload = struct.pack(fmt, *local_vals)
if self.is_leader:
- # Rank 0: Gather -> Max -> Broadcast
- values = [local_val]
+ rows = [local_vals]
- # Non-blocking gather from N-1 peers
- while len(values) < self.world_size:
+ while len(rows) < self.world_size:
try:
- msg = self.gather_sock.recv(flags=zmq.NOBLOCK)
- values.append(struct.unpack('!i', msg)[0])
+ if async_op:
+ msg = self.gather_sock.recv(flags=zmq.NOBLOCK)
+ else:
+ msg = self.gather_sock.recv()
+ rows.append(struct.unpack(fmt, msg))
except zmq.Again:
- await asyncio.sleep(0.001) # Yield to event loop
+ await asyncio.sleep(0.001)
- max_val = max(values)
- self.bcast_sock.send(struct.pack('!i', max_val))
- return max_val
+ maxes = tuple(max(row[i] for row in rows) for i in range(n))
+ self.bcast_sock.send(struct.pack(fmt, *maxes))
+ if not async_op:
+ await asyncio.sleep(
+ 0
+ ) # Yield control once to ensure that other coroutines can run.
+ # This might be needed for colocated RL.
+ return maxes[0] if n == 1 else maxes
else:
- # Worker: Send -> Wait for Broadcast
self.gather_sock.send(payload)
while True:
try:
- msg = self.bcast_sock.recv(flags=zmq.NOBLOCK)
- return struct.unpack('!i', msg)[0]
+ if async_op:
+ msg = self.bcast_sock.recv(flags=zmq.NOBLOCK)
+ else:
+ msg = self.bcast_sock.recv()
+ result = struct.unpack(fmt, msg)
+ if not async_op:
+ await asyncio.sleep(
+ 0
+ ) # Yield control once to ensure that other coroutines can run.
+ # This might be needed for colocated RL.
+ return result[0] if n == 1 else result
except zmq.Again:
- await asyncio.sleep(0.001) # Yield to event loop
+ await asyncio.sleep(0.001)
def close(self):
"""
Close the ZMQ sockets.
"""
- self.gather_sock.close()
- self.bcast_sock.close()
+ # linger=0: discard unsent messages immediately on close rather than blocking until sent.
+ # The ZMQ default is to not allow `close` until all messages have been successfully sent.
+ self.gather_sock.close(linger=0)
+ self.bcast_sock.close(linger=0)
diff --git a/megatron/core/inference/engines/dynamic_engine.py b/megatron/core/inference/engines/dynamic_engine.py
index 0a95e8f4a53..a9c8337271f 100644
--- a/megatron/core/inference/engines/dynamic_engine.py
+++ b/megatron/core/inference/engines/dynamic_engine.py
@@ -12,6 +12,7 @@
from contextlib import contextmanager
from dataclasses import dataclass
from datetime import datetime
+from enum import Enum, auto
from itertools import repeat
from typing import Dict, List, Optional, Tuple, Union
@@ -19,6 +20,7 @@
from torch import Tensor
from torch.cuda.nvtx import range_pop, range_push
+from megatron.core.inference.config import KVCacheManagementMode
from megatron.core.inference.contexts.dynamic_context import (
DynamicInferenceContext,
MaxSequenceLengthOverflowError,
@@ -30,6 +32,8 @@
from megatron.core.inference.engines.abstract_engine import AbstractEngine
from megatron.core.inference.headers import Headers, UnknownHeaderError
from megatron.core.inference.inference_request import (
+ DynamicInferenceEvent,
+ DynamicInferenceEventType,
DynamicInferenceRequest,
DynamicInferenceRequestRecord,
Status,
@@ -38,10 +42,18 @@
from megatron.core.inference.text_generation_controllers.text_generation_controller import (
TextGenerationController,
)
-from megatron.core.inference.utils import Counter, await_process_call
+from megatron.core.inference.utils import (
+ Counter,
+ await_process_call,
+ set_inference_cuda_graphed_iteration_for_ep_inference,
+ unset_inference_cuda_graphed_iteration_for_ep_inference,
+)
from megatron.core.process_groups_config import ProcessGroupCollection
from megatron.core.transformer.cuda_graphs import delete_cuda_graphs
+from megatron.core.transformer.enums import CudaGraphScope
+from megatron.core.transformer.moe.router_replay import RouterReplay, RouterReplayAction
from megatron.core.utils import (
+ deprecate_args,
experimental_api,
get_asyncio_loop,
get_pg_rank,
@@ -89,10 +101,29 @@
except ImportError:
HAVE_PSUTIL = False
-from megatron.core.inference.contexts.dynamic_context import HAVE_TORCH_MEMORY_SAVER
+DEPRECATED_ARGS = [
+ "enable_cuda_graph",
+ "random_seed",
+ "track_paused_request_events",
+ "enable_chunked_prefill",
+ "inference_logging_step_interval",
+ "pg_collection",
+]
+
-if HAVE_TORCH_MEMORY_SAVER:
- from torch_memory_saver import torch_memory_saver
+class EngineState(Enum):
+ """State machine for the inference engine."""
+
+ RUNNING = auto() # Processing requests
+ PAUSING = auto() # PAUSE received; waiting for EP consensus + world barrier
+ PAUSED = auto() # Globally confirmed idle
+ UNPAUSING = auto() # UNPAUSE received; waiting for world barrier
+ SUSPENDING = auto() # SUSPEND received; offloading GPU; waiting for world barrier
+ SUSPENDED = auto() # GPU offloaded, all ranks confirmed
+ RESUMING = auto() # RESUME received; onloading GPU; waiting for world barrier
+ RESUMED = auto() # GPU onloaded, all ranks confirmed; cleared on next SUSPEND
+ STOPPING = auto() # STOP received; futures cancelled; waiting for world barrier
+ STOPPED = auto() # All ranks confirmed; teardown complete
class EngineSuspendedError(Exception):
@@ -136,24 +167,22 @@ class DynamicInferenceEngine(AbstractEngine):
outputs and detokenizer the output tokens.
inference_context (DynamicInferenceContext): Context for managing in-flight
batching and a dynamic block-level KV cache (similar to paged attention).
- random_seed (Optional[int]): Use a random seed if you want deterministic
- results. Defaults to None.
- inference_logging_step_interval (int): The step interval at which to log
- inference metrics to wandb. Defaults to 0, which means no logging.
"""
- def __init__(
- self,
- controller: TextGenerationController,
- context: DynamicInferenceContext,
- enable_cuda_graph: Optional[bool] = None,
- random_seed: Optional[int] = None,
- *,
- track_paused_request_events: bool = False,
- enable_chunked_prefill: bool = True,
- inference_logging_step_interval: int = 0,
- pg_collection: Optional[ProcessGroupCollection] = None,
- ):
+ # Map stable states to their corresponding asyncio events.
+ _STATE_EVENTS = (
+ EngineState.RUNNING,
+ EngineState.PAUSED,
+ EngineState.SUSPENDED,
+ EngineState.RESUMED,
+ EngineState.STOPPED,
+ )
+
+ @deprecate_args(
+ *DEPRECATED_ARGS,
+ message="Argument `{name}` has been deprecated. Only pass `controller` and `context`",
+ )
+ def __init__(self, controller: TextGenerationController, context: DynamicInferenceContext):
assert isinstance(
controller, TextGenerationController
@@ -161,41 +190,43 @@ def __init__(
assert isinstance(
context, DynamicInferenceContext
), f"context must be a DynamicInferenceContext, got {type(context)}"
- assert isinstance(random_seed, int), f"random_seed must be an int, got {type(random_seed)}"
- # Deprecate `enable_cuda_graph`.
- if enable_cuda_graph is not None:
- warnings.warn(
- "The `enable_cuda_graph` argument is deprecated and will be "
- "removed in `megatron-core 0.15`. `enable_cuda_graph` is now "
- "read directly from the transformer config object."
- )
- self.enable_cuda_graph = enable_cuda_graph
- else:
- self.enable_cuda_graph = (
- controller.inference_wrapped_model.model.config.enable_cuda_graph
- )
+ model_config = controller.inference_wrapped_model.model.config
+ inference_config = context.config
- if pg_collection is not None:
- self.pg_collection = pg_collection
+ if inference_config.pg_collection is not None:
+ self.pg_collection = inference_config.pg_collection
else:
self.pg_collection = ProcessGroupCollection.use_mpu_process_groups()
# Initialization options.
self.controller = controller
self.context = context
- self.random_seed = random_seed
- self.track_paused_request_events = track_paused_request_events
- self.enable_chunked_prefill = enable_chunked_prefill
- self.inference_logging_step_interval = inference_logging_step_interval
- self.unified_memory_level = context.unified_memory_level
- self.persist_cuda_graphs = context.persist_cuda_graphs
-
- if enable_cuda_graph is not None:
- self.cuda_graph_impl = "local" if enable_cuda_graph else "none"
- else:
- self.cuda_graph_impl = controller.inference_wrapped_model.model.config.cuda_graph_impl
+ self.num_speculative_tokens = inference_config.num_speculative_tokens
+ self.materialize_only_last_token_logits = (
+ inference_config.materialize_only_last_token_logits
+ )
+
+ assert self.num_speculative_tokens >= 0, "Number of speculative tokens must be non-negative"
+
+ if self.num_speculative_tokens > 0:
+ assert (
+ self.num_speculative_tokens <= self.controller.num_mtp_heads
+ ), f"Number of speculative tokens {self.num_speculative_tokens} must be less than or equal to number of MTP heads {self.controller.num_mtp_heads}"
+ assert (
+ not self.materialize_only_last_token_logits
+ ), "materialize_only_last_token_logits must be False when num_speculative_tokens > 0"
+
+ self.track_paused_request_events = inference_config.track_paused_request_events
+ self.track_generated_token_events = inference_config.track_generated_token_events
+ self.enable_chunked_prefill = inference_config.enable_chunked_prefill
+ self.metrics_writer = inference_config.metrics_writer
+ self.logging_step_interval = inference_config.logging_step_interval
+ self.unified_memory_level = inference_config.unified_memory_level
+ self.use_synchronous_zmq_collectives = inference_config.use_synchronous_zmq_collectives
+ self.cuda_graph_impl = model_config.cuda_graph_impl
+ self.cuda_graph_scope = model_config.cuda_graph_scope
# Initialize engine.
self.reset()
@@ -205,12 +236,12 @@ def __init__(
)
# Configure wandb to use separate step counter for inference metrics (only once)
- if self.inference_logging_step_interval > 0 and self.context.metrics_writer is not None:
+ if self.logging_step_interval > 0 and self.metrics_writer is not None:
logging.info(
f"\033[1;93m[INFERENCE]\033[0m "
f"\033[1;95mLogging inference metrics to wandb (rank {self.rank})\033[0m"
)
- if HAVE_WANDB and self.context.metrics_writer.__name__ == "wandb":
+ if HAVE_WANDB and self.metrics_writer.__name__ == "wandb":
# Make all inference/* metrics use inference_step as their x-axis
# This allows inference and training to have independent step counters
context.metrics_writer.define_metric(
@@ -245,6 +276,7 @@ def reset(self) -> None:
self.requests: Dict[int, RequestEntry] = {}
self.waiting_request_ids = deque()
self.failed_request_ids = []
+ self._generation_epoch: Optional[int] = None
# Track requests that should stop due to stop words (detected in post_process_requests)
self.stop_word_finished_request_ids: set[int] = set()
# Track requests currently being finished due to stop words (to skip extra token)
@@ -252,7 +284,6 @@ def reset(self) -> None:
# Timing and logging variables.
self.rank = torch.distributed.get_rank()
- self.step_count = 0
self.step_start_event = torch.cuda.Event(enable_timing=True)
self.step_end_event = torch.cuda.Event(enable_timing=True)
self.capture_stats = None
@@ -260,18 +291,38 @@ def reset(self) -> None:
# Runtime state.
self._loop = get_asyncio_loop(getattr(self, "_loop", None))
self._cond = asyncio.Condition()
- self.running = asyncio.Event()
- self.paused = asyncio.Event()
- self.stopped = asyncio.Event()
- self.received_pause: bool = False
- self.received_stop: bool = False
- self.suspend_signal = False
- self.is_suspended = False
+ self._state_events = {k: asyncio.Event() for k in self._STATE_EVENTS}
+ self.state = EngineState.RUNNING
+ self._state_events[EngineState.RUNNING].set()
+ self._pending_signals = deque()
+
self.resume_request_ids = None
+ # Speculative decoding acceptance tracking.
+ self._spec_tokens_proposed = 0
+ self._spec_tokens_accepted = 0
+ self._spec_steps = 0
+
+ # Prefix caching tracking.
+ self._prefix_cache_hits = 0
+ self._prefix_cache_blocks_matched = 0
+ self._prefix_coordination_waits = 0
+
# Coordinator state.
self.use_coordinator = False
+ async def wait_until(self, state: EngineState):
+ """Wait until the engine reaches the given state.
+
+ Only stable states (RUNNING, PAUSED, SUSPENDED, RESUMED,
+ STOPPED) are supported. Transient states (PAUSING, SUSPENDING,
+ RESUMING, STOPPING) are not directly waitable.
+ """
+ event = self._state_events.get(state)
+ if event is None:
+ raise ValueError(f"Cannot wait for transient state {state}")
+ await event.wait()
+
def create_cuda_graphs(self, reset_context: bool = True):
"""Create cuda graphs.
@@ -285,11 +336,18 @@ def create_cuda_graphs(self, reset_context: bool = True):
if self.cuda_graph_impl != "local":
return
+ if (
+ CudaGraphScope.full_iteration in self.cuda_graph_scope
+ and CudaGraphScope.full_iteration_inference not in self.cuda_graph_scope
+ ):
+ warnings.warn(
+ "\n\n*** WARNING: 'full_iteration' CUDA graph scope used during inference! "
+ "This will not create inference CUDA graphs. Use '--cuda-graph-scope=full_iteration_inference' instead. ***\n"
+ )
+
context = self.context
controller = self.controller
- config = controller.inference_wrapped_model.inference_wrapper_config
-
time_start = time.time()
mem_stats_start = torch.cuda.memory_stats()
@@ -297,6 +355,16 @@ def create_cuda_graphs(self, reset_context: bool = True):
for graph in context.cuda_graph_batch_dimensions_list:
logging.info(graph)
+ # Enable inference dispatcher for EP during graph capture
+ model_config = controller.inference_wrapped_model.model.config
+ is_inference_optimized_ep = (
+ model_config.transformer_impl == "inference_optimized"
+ and model_config.expert_model_parallel_size > 1
+ )
+ if is_inference_optimized_ep:
+ unwrapped_model = controller.inference_wrapped_model.model
+ set_inference_cuda_graphed_iteration_for_ep_inference(unwrapped_model)
+
tbar = enumerate(context.cuda_graph_batch_dimensions_list)
if HAVE_TQDM:
tbar = tqdm(tbar, total=len(context.cuda_graph_batch_dimensions_list))
@@ -313,11 +381,21 @@ def create_cuda_graphs(self, reset_context: bool = True):
f"{tbar_idx}/{len(context.cuda_graph_batch_dimensions_list)}. {tbar_str}"
)
+ # Enable routing recording during warmup if routing replay is enabled.
+ # This ensures the record_indices copy operation is captured in the CUDA graph.
+ model_config = controller.inference_wrapped_model.model.config
+ if model_config.moe_enable_routing_replay:
+ RouterReplay.set_global_router_replay_action(RouterReplayAction.RECORD)
+
# Forward pass -> logits.
controller._dynamic_step_forward_logits(input_ids, position_ids)
context.reset()
+ # Disable inference dispatcher after graph capture
+ if is_inference_optimized_ep:
+ unset_inference_cuda_graphed_iteration_for_ep_inference(unwrapped_model)
+
# Memory usage.
time_end = time.time()
mem_stats_end = torch.cuda.memory_stats()
@@ -342,15 +420,14 @@ def create_cuda_graphs(self, reset_context: bool = True):
self.capture_stats = capture_stats
- if HAVE_TORCH_MEMORY_SAVER:
- torch_memory_saver.pause("kv_cache")
-
@internal_api
async def start_listening_to_data_parallel_coordinator(
self,
inference_coordinator_port: int | None = None,
launch_inference_coordinator: bool = True,
*,
+ hostname: str | None = None,
+ coordinator_schedule_output_path: str | None = None,
loop: Optional[asyncio.AbstractEventLoop] = None,
):
"""Initializes ZMQ communication to connect the engine with an inference coordinator.
@@ -387,6 +464,9 @@ async def start_listening_to_data_parallel_coordinator(
launch_inference_coordinator (bool, optional): If True, the global rank 0
process will spawn and manage the `InferenceCoordinator`
process. Defaults to True.
+ hostname (str | None): Hostname or IP address to use for ZMQ socket binding.
+ If None, defaults to `socket.gethostname()`. Should be set to a routable
+ address in multi-node settings where gethostname() may return 127.0.0.1.
Returns:
inference_coordinator_addresss (str): The network address of the central
@@ -402,7 +482,7 @@ async def start_listening_to_data_parallel_coordinator(
"pip install msgpack"
)
- self.zmq_context = zmq.Context().instance()
+ self.zmq_context = zmq.Context.instance()
self.zmq_sockets = [] # keep track of all sockets created by this engine
# Get world info.
@@ -419,22 +499,31 @@ async def start_listening_to_data_parallel_coordinator(
self.is_mp_coordinator = tp_rank == 0 and pp_rank == 0
self.is_dp_coordinator = (dp_rank == 0) and self.is_mp_coordinator
- local_ip = socket.gethostname()
+ local_ip = hostname or socket.gethostname()
# Spawn a DP coordinator process and get the connection info.
if launch_inference_coordinator and self.is_dp_coordinator:
spawn_context = multiprocessing.get_context('spawn')
+ deterministic_mode = torch.are_deterministic_algorithms_enabled()
dp_pipe, dp_process_pipe = spawn_context.Pipe()
coordinator_ready_event = spawn_context.Event()
self.inference_coordinator_process = spawn_context.Process(
target=DataParallelInferenceCoordinator.entrypoint,
- args=(
- dp_process_pipe,
- coordinator_ready_event,
- get_pg_size(self.pg_collection.dp),
- self.controller.tokenizer,
- inference_coordinator_port,
- ),
+ kwargs={
+ "pipe_connection": dp_process_pipe,
+ "ready_event": coordinator_ready_event,
+ "data_parallel_size": get_pg_size(self.pg_collection.dp),
+ "tokenizer": self.controller.tokenizer,
+ "max_requests": self.context.max_requests,
+ "inference_coordinator_port": inference_coordinator_port,
+ "deterministic_mode": deterministic_mode,
+ "block_size_tokens": self.context.block_size_tokens,
+ "enable_prefix_caching": self.context.enable_prefix_caching,
+ "prefix_caching_coordinator_policy": self.context.prefix_caching_coordinator_policy,
+ "prefix_caching_routing_alpha": self.context.prefix_caching_routing_alpha,
+ "schedule_output_path": coordinator_schedule_output_path,
+ "hostname": hostname,
+ },
)
self.inference_coordinator_process.start()
await await_process_call(dp_pipe.poll, self.inference_coordinator_process)
@@ -519,7 +608,14 @@ async def start_listening_to_data_parallel_coordinator(
self.ep_world_size = get_pg_size(self.pg_collection.ep)
if self.ep_world_size > 1:
self.expert_parallel_zmq_communicator = AsyncZMQCommunicator(
- self.zmq_context, process_group=self.pg_collection.ep
+ self.zmq_context, process_group=self.pg_collection.ep, hostname=hostname
+ )
+
+ # initialize zmq-based world communicator for consensus barriers
+ total_world_size = torch.distributed.get_world_size()
+ if total_world_size > 1:
+ self.world_zmq_communicator = AsyncZMQCommunicator(
+ self.zmq_context, process_group=None, hostname=hostname
)
if launch_inference_coordinator and self.is_dp_coordinator:
@@ -529,7 +625,7 @@ async def start_listening_to_data_parallel_coordinator(
logging.info("Inference co-ordinator is ready to receive requests!")
logging.info(f"Data parallel coordinator can be found at {dp_addr}")
- # Finally run the engine infinite loop
+ # Finally run the engine infinite loop.
loop = get_asyncio_loop(loop)
self.engine_loop_task = loop.create_task(self.run_engine_with_coordinator(loop=loop))
@@ -555,12 +651,14 @@ def suspend_resume_ctx(key: str, *, unified_memory_level: int) -> None:
start_mem = torch.cuda.memory_stats()
start_time = time.time()
+ range_push(f"{key}-inference-context")
torch.cuda.synchronize()
yield
finally:
+ range_pop()
end_time = time.time()
end_mem = torch.cuda.memory_stats()
@@ -599,42 +697,58 @@ def suspend_resume_ctx(key: str, *, unified_memory_level: int) -> None:
def suspend(self):
"""Suspend engine by deallocating context's GPU state."""
- # Skip if already suspended, which can happen when using the inference
- # coordinator.
- if self.is_suspended:
+ # Skip if already suspended or in the process of suspending.
+ if self.state in (EngineState.SUSPENDED, EngineState.SUSPENDING):
return
- self.is_suspended = True
# Deallocate context tensors.
with self.__class__.suspend_resume_ctx(
"suspended", unified_memory_level=self.unified_memory_level
):
- self.context.deallocate_all_tensors()
+ self.context.deallocate_inference_state_buffers()
- # Delete cuda graphs when not using unified memory at all (level 0) and
- # `--rl-training-cuda-graphs` is not passed. For UVM levels 1 and 2, the context's tensors
- # maintain static memory addresses, so the cuda graphs are re-used.
- if self.unified_memory_level == 0 and not self.persist_cuda_graphs:
+ if (
+ self.context.kv_cache_management_mode != KVCacheManagementMode.PERSIST
+ and not self.context.static_kv_memory_pointers
+ ):
delete_cuda_graphs()
- # Maintain references to requests before reset.
+ # Build the list of requests to re-add on resume.
+ # All waiting requests are always included; active requests are included
+ # only if they are marked for recompute (their KV cache will be gone).
waiting_request_ids = list(self.waiting_request_ids)
active_request_ids = set(self.requests.keys()) - set(waiting_request_ids)
- self.resume_request_ids = [*active_request_ids, *waiting_request_ids]
+ if self.context.kv_cache_management_mode == KVCacheManagementMode.RECOMPUTE:
+ recompute_active_ids = active_request_ids
+
+ # Reset any partially prefilled requests so they recompute from the start
+ for req_id in [*waiting_request_ids, *recompute_active_ids]:
+ req = self.get_request(req_id)
+ if req.finished_chunk_token_count > 0:
+ req.remaining_prompt_tokens = req.prompt_tokens
+ req.finished_chunk_token_count = 0
+
+ # Reset the chunked prefill request id
+ self.chunked_prefill_request_id = -1
+ else:
+ recompute_active_ids = set()
+ self.resume_request_ids = [*recompute_active_ids, *waiting_request_ids]
self.waiting_request_ids.clear()
- # Suspend requests objects.
- for request_id in active_request_ids:
+ # Checkpoint active requests that are marked for recompute.
+ for request_id in recompute_active_ids:
self.requests[request_id].record.checkpoint()
+ # If we are not using the inference coordinator, we need to manually handle state.
+ if not self.use_coordinator:
+ self.state = EngineState.SUSPENDED
+
def resume(self):
"""Resume engine by reallocating context's GPU state."""
- # Skip if not suspended, which can happen when using the inference
- # coordinator.
- if not self.is_suspended:
+ # Skip if not suspended or in the process of suspending.
+ if self.state not in (EngineState.SUSPENDED, EngineState.SUSPENDING):
return
- self.is_suspended = False
# Resume.
with self.__class__.suspend_resume_ctx(
@@ -644,27 +758,30 @@ def resume(self):
# Allocate context tensors.
alloc_time = time.time()
torch.cuda.synchronize()
- self.context.allocate_all_tensors(is_init=False)
+ self.context.reinitialize_inference_state_buffers()
torch.cuda.synchronize()
alloc_time = time.time() - alloc_time
- # Reset context and request data.
- self.context.reset()
-
- # Create cuda graphs (before adding requests, to be in decode mode).
- # Only create cuda graphs when not using unified memory at all (level
- # 0). For levels 1 and 2, the context's tensors maintain static
- # memory addresses, so the cuda graphs are re-used.
capture_time = time.time()
- if self.unified_memory_level == 0 and not self.persist_cuda_graphs:
+ if (
+ self.context.kv_cache_management_mode != KVCacheManagementMode.PERSIST
+ and not self.context.static_kv_memory_pointers
+ ):
self.create_cuda_graphs()
capture_time = time.time() - capture_time
- # Add requests.
+ # Re-add requests saved during suspend.
add_time = time.time()
torch.cuda.synchronize()
for request_id in self.resume_request_ids:
self._add_request(self.get_request(request_id))
+
+ # Ensure chunked prefill request remains at the head of the waiting queue
+ if self.context.chunked_prefill_request_id != -1:
+ if self.context.chunked_prefill_request_id in self.waiting_request_ids:
+ self.waiting_request_ids.remove(self.context.chunked_prefill_request_id)
+ self.waiting_request_ids.appendleft(self.context.chunked_prefill_request_id)
+
torch.cuda.synchronize()
add_time = time.time() - add_time
@@ -680,8 +797,13 @@ def resume(self):
)
)
- # Notify event loop.
- self._loop.call_soon_threadsafe(asyncio.create_task, self._notify_cond_for_new_request())
+ # If we are not using the inference coordinator, we need to manually handle state.
+ if not self.use_coordinator:
+ self.state = EngineState.RUNNING
+ # Notify the condition variable that run_engine() waits on.
+ self._loop.call_soon_threadsafe(
+ asyncio.create_task, self._notify_cond_for_new_request()
+ )
@trace_async_exceptions
async def _notify_cond_for_new_request(self):
@@ -689,6 +811,47 @@ async def _notify_cond_for_new_request(self):
async with self._cond:
self._cond.notify_all()
+ def _handle_failed_request(self, request_id: int):
+ """Handle a failed request by sending the reply immediately.
+
+ The request is added to failed_request_ids so that the next bookkeeping pass can return it.
+ """
+ request_entry = self.requests[request_id]
+ request = request_entry.record[-1]
+
+ if self.rank == 0:
+ warnings.warn(
+ f"Request {request_id} failed to be added to the engine due to errors. "
+ f"Prompt Tokens: {len(request.prompt_tokens)} "
+ f"Tokens to generate: {request.sampling_params.num_tokens_to_generate} "
+ f"Max sequence length: {self.context.max_sequence_length} "
+ f"Chunked prefill enabled: {self.enable_chunked_prefill}"
+ )
+
+ request.status = Status.FAILED
+ request.add_event_fail()
+ self.failed_request_ids.append(request_id)
+
+ # Send the reply immediately, because it may never get a chance to be sent again.
+ if self.use_coordinator and self.is_mp_coordinator:
+ payload = msgpack.packb(
+ [Headers.ENGINE_REPLY.value, [request_entry.record.merge().serialize()]],
+ use_bin_type=True,
+ )
+ self.socket_for_receiving_requests.send(payload)
+ elif not self.use_coordinator:
+ if request.prompt is None:
+ request.prompt = self.controller.tokenizer.detokenize(
+ request.prompt_tokens.tolist()
+ )
+ if request.generated_tokens:
+ request.generated_text = self.controller.tokenizer.detokenize(
+ request.generated_tokens
+ )
+ else:
+ request.generated_text = ""
+ request_entry.future.set_result(request_entry.record)
+
def has_unfinished_requests(self) -> bool:
"""Test if context contains unfinished requests."""
return self.context.has_unfinished_requests() or len(self.waiting_request_ids) > 0
@@ -717,6 +880,13 @@ def _add_request(
record=DynamicInferenceRequestRecord.from_request(request),
future=self._loop.create_future(),
)
+ request.add_event_add_engine() # Record when request enters engine
+
+ # Stamp new request with the current generation epoch.
+ if self._generation_epoch is not None:
+ epoch = self._generation_epoch
+ request.policy_epoch = [(0, epoch)]
+ request.kv_cache_epoch = [(0, epoch)]
if request.status is None:
request.status = Status.ACTIVE_AND_GENERATING_TOKENS
@@ -733,7 +903,7 @@ def _add_request(
request.sampling_params.return_log_probs
and not request.sampling_params.skip_prompt_log_probs
):
- assert not self.context.materialize_only_last_token_logits, (
+ assert not self.materialize_only_last_token_logits, (
"Prompt log probs cannot be calculated if only last token logits are materialized. "
"Set materialize_only_last_token_logits to False in DynamicInferenceContext "
"or skip_prompt_log_probs to True in SamplingParams."
@@ -743,6 +913,7 @@ def _add_request(
request.sampling_params.num_tokens_to_generate = (
request.sampling_params.num_tokens_total - len(request.prompt_tokens)
)
+ request.sampling_params.num_tokens_total = None
if request.sampling_params.num_tokens_to_generate is None:
request.sampling_params.num_tokens_to_generate = self.context.max_sequence_length - len(
request.prompt_tokens
@@ -773,7 +944,7 @@ def _add_request(
# Tokenize stop words if provided
if request.sampling_params.stop_words:
stop_word_ids = [
- self.controller.tokenize_prompt(stop_word, add_BOS=False)
+ self.controller.tokenize_prompt(self.controller.tokenizer, stop_word, add_BOS=False)
for stop_word in request.sampling_params.stop_words
]
request.stop_word_ids = stop_word_ids
@@ -781,7 +952,7 @@ def _add_request(
if request.status != Status.FAILED:
self.waiting_request_ids.append(request_id)
else:
- self.failed_request_ids.append(request_id)
+ self._handle_failed_request(request_id)
return self.requests[request_id].future
@@ -807,9 +978,13 @@ def add_request(
# Tokenize prompt if text. Support legacy single-arg mocks.
prompt_str = prompt
try:
- prompt_token_ids = self.controller.tokenize_prompt(prompt, sampling_params.add_BOS)
+ prompt_token_ids = self.controller.tokenize_prompt(
+ self.controller.tokenizer, prompt, sampling_params.add_BOS
+ )
except TypeError:
- prompt_token_ids = self.controller.tokenize_prompt(prompt)
+ prompt_token_ids = self.controller.tokenize_prompt(
+ self.controller.tokenizer, prompt
+ )
tokens = torch.tensor(
prompt_token_ids, dtype=torch.int64, device=torch.cuda.current_device()
)
@@ -833,6 +1008,8 @@ def add_request(
prompt=prompt_str,
prompt_tokens=tokens,
sampling_params=sampling_params,
+ block_size_tokens=self.context.block_size_tokens,
+ enable_prefix_caching=self.context.enable_prefix_caching,
)
# Add request.
@@ -845,8 +1022,12 @@ def post_process_requests(
evict_request_ids: torch.Tensor,
step_time: float,
sample: torch.Tensor,
+ accepted_tokens: torch.Tensor,
log_probs: torch.Tensor,
top_n_logprobs: Optional[Dict[int, List[Tuple[torch.Tensor, torch.Tensor]]]] = None,
+ routing_indices_per_request: Optional[Dict[int, torch.Tensor]] = None,
+ pre_fwd_active_token_count: Optional[int] = None,
+ pre_fwd_step_count: Optional[int] = None,
) -> Tuple[List[DynamicInferenceRequest], List[DynamicInferenceRequest]]:
"""
Handles post-processing for requests after a step.
@@ -856,10 +1037,14 @@ def post_process_requests(
finished_request_ids (torch.Tensor): A list of finished request ids
evict_request_ids (torch.Tensor): A list of evicted request ids.
step_time (float): The latency of the last step
- sample: (torch.Tensor): The newly generated tokens for each request
+ sample: Tensor: The newly generated token for each request
+ accepted_tokens: Tensor: The additional accepted tokens for each request
log_probs: (List): Log probs for each request
top_n_logprobs: (Dict): Top-n log probs for each request. Maps request_idx to
list of (top_n_logprobs, top_n_indices) tuples.
+ routing_indices_per_request: (Dict[int, Tensor]): MoE routing indices
+ pre-mapped by request_id. Each value is a tensor of shape
+ [num_tokens_this_step, num_layers, topk].
Returns:
A list of active requests and completed requests as `DynamicInferenceRequest` objects
@@ -872,27 +1057,121 @@ def post_process_requests(
self.evicted_request_count += evict_request_ids.numel()
log_probs_iter = log_probs if log_probs else repeat(None)
+ block_allocator = self.context.kv_block_allocator
+
+ # Pre-compute step-level block stats (before the per-request loop)
+ if self.track_generated_token_events:
+ blocks_allocated = block_allocator.total_count - block_allocator.total_avail
+ if block_allocator.enable_prefix_caching:
+ blocks_hashed_active = int((block_allocator.block_ref_counts > 0).sum().item())
+ blocks_ref_count = block_allocator.block_ref_counts.sum().item()
+ else:
+ blocks_hashed_active = blocks_allocated
+ blocks_ref_count = None
- for req_idx, (request_id, token, request_log_probs) in enumerate(
- zip(request_ids.tolist(), sample.tolist(), log_probs_iter)
+ # When accepted_tokens is None (no speculative decoding), use repeat([]) to provide
+ # empty lists for each request, so the zip produces the correct number of iterations
+ accepted_tokens_iter = repeat([]) if accepted_tokens is None else accepted_tokens.tolist()
+
+ if self.num_speculative_tokens > 0 and accepted_tokens is not None:
+ self._spec_steps += 1
+
+ for req_idx, (request_id, tokens, accepted_tokens_list, request_log_probs) in enumerate(
+ zip(request_ids.tolist(), sample.tolist(), accepted_tokens_iter, log_probs_iter)
):
+
+ # Ensure tokens is always a list for consistent handling
+ if not isinstance(tokens, list):
+ tokens = [tokens]
+
request: DynamicInferenceRequest = self.get_request(request_id)
+
+ if self.num_speculative_tokens > 0:
+ accepted_tokens = list(filter(lambda tok: tok != -1, accepted_tokens_list))
+
+ # The order `accepted_tokens + tokens` is correct here.
+ # `accepted_tokens` contains the sequence of
+ # successfully verified draft tokens. `tokens` (from `sample`) is the
+ # brand new token generated by the target model based on that accepted prefix.
+ # Therefore, the newly sampled token must go at the end of the sequence.
+ tokens = accepted_tokens + tokens
+
+ num_stop_word_trim = 0
if request_id != self.context.chunked_prefill_request_id:
# Skip appending token for requests being finished due to stop words
# (they already have their final token from the previous step)
+ # If the request already has more tokens, then we only append as much as is necessary
+ if (
+ len(request.generated_tokens) + len(tokens)
+ >= request.sampling_params.num_tokens_to_generate
+ ):
+ tokens = tokens[
+ : request.sampling_params.num_tokens_to_generate
+ - len(request.generated_tokens)
+ ]
if request_id not in self.stop_word_being_finished_ids:
- request.generated_tokens.append(token)
+ is_first_token = len(request.generated_tokens) == 0
+ request.generated_tokens += tokens
+ first_token_event = None
+ if self.track_generated_token_events:
+ for token in tokens:
+ if block_allocator.enable_prefix_caching:
+ event = request.add_event_generated_token(
+ token,
+ blocks_total=block_allocator.total_count,
+ blocks_hashed_total=blocks_allocated,
+ blocks_hashed_active=blocks_hashed_active,
+ blocks_ref_count=blocks_ref_count,
+ pre_fwd_active_token_count=pre_fwd_active_token_count,
+ pre_fwd_step_count=pre_fwd_step_count,
+ )
+ else:
+ event = request.add_event_generated_token(
+ token,
+ blocks_total=block_allocator.total_count,
+ blocks_hashed_total=blocks_allocated,
+ blocks_hashed_active=blocks_hashed_active,
+ pre_fwd_active_token_count=pre_fwd_active_token_count,
+ pre_fwd_step_count=pre_fwd_step_count,
+ )
+ if first_token_event is None:
+ first_token_event = event
+ if is_first_token:
+ if not self.track_generated_token_events:
+ first_token_event = DynamicInferenceEvent(
+ type=DynamicInferenceEventType.GENERATED_TOKEN,
+ payload={"token_id": tokens[0]},
+ )
+ request.ttft = (
+ first_token_event.timestamp - request.event_add_engine.timestamp
+ )
if request.tpot is None:
request.tpot = []
- request.tpot.append(step_time)
+ per_token_step_time = step_time / len(tokens)
+ request.tpot.extend([per_token_step_time] * len(tokens))
+
+ # Check for stop words (after token is appended).
+ # With speculative decoding, a stop word may end before the last
+ # appended token. The check truncates generated_tokens in-place and
+ # returns how many trailing tokens were removed so we can also trim
+ # the corresponding log probs below.
+ stop_word_hit, num_stop_word_trim = self._check_stop_words_for_request_post_append(
+ request
+ )
- # Check for stop words (after token is appended)
- stop_word_hit = self._check_stop_words_for_request_post_append(request)
+ # Track acceptance statistics for logging.
+ if len(request.generated_tokens) > 0 and self.num_speculative_tokens > 0:
+ actual_proposed = max(0, self.num_speculative_tokens - num_stop_word_trim)
+ actual_accepted = max(0, len(accepted_tokens) - num_stop_word_trim)
+
+ self._spec_tokens_proposed += actual_proposed
+ self._spec_tokens_accepted += actual_accepted
if request_id in finished_request_ids:
# Request finished by normal means (termination_id, max_length, or stop word from previous step)
request.generated_length = len(request.generated_tokens)
request.status = Status.COMPLETED
+ request.add_event_finish()
finished_entry = self.requests.pop(request_id)
finished_request = finished_entry.record[-1]
finished_request.generated_length = len(finished_request.generated_tokens)
@@ -911,6 +1190,14 @@ def post_process_requests(
# Additionally, chunked prefill request do not finish.
active_request_ids.append(request_id)
+ # When a stop word was found mid-speculative-batch, trim log probs
+ # and top_n_logprobs to match the truncated generated_tokens.
+ if num_stop_word_trim > 0:
+ if request_log_probs is not None:
+ request_log_probs = request_log_probs[:-num_stop_word_trim]
+ if top_n_logprobs is not None and req_idx in top_n_logprobs:
+ top_n_logprobs[req_idx] = top_n_logprobs[req_idx][:-num_stop_word_trim]
+
# Process log_probs if available (unified for both regular and chunked prefill)
if request_log_probs is not None:
# Initialize lists if they don't exist
@@ -919,40 +1206,30 @@ def post_process_requests(
if not request.generated_log_probs:
request.generated_log_probs = []
- # For chunked prefill with materialize_only_last_token_logits, discard intermediate log probs
- if (
- request_id == self.context.chunked_prefill_request_id
- and self.context.materialize_only_last_token_logits
- ):
- request.prompt_log_probs = []
- request.generated_log_probs = []
+ is_chunked_prefill = request_id == self.context.chunked_prefill_request_id
+ is_prefill = len(request.generated_log_probs) == 0
+
+ if request.sampling_params.skip_prompt_log_probs:
+ # We only want decode log probs.
+ if is_chunked_prefill:
+ pass
+ elif is_prefill:
+ request.generated_log_probs.append(request_log_probs[-1])
+ else:
+ request.generated_log_probs.extend(request_log_probs)
else:
+ # Split log probs between prompt and generated based on remaining prompt slots.
prompt_length = len(request.prompt_tokens)
total_accumulated = len(request.prompt_log_probs) + len(
request.generated_log_probs
)
+ remaining_prompt_slots = max(0, prompt_length - 1 - total_accumulated)
+ split_idx = min(remaining_prompt_slots, len(request_log_probs))
- # Handle skip_prompt_log_probs during prefill
- # If skip_prompt_log_probs is True and we have multiple log probs (prefill),
- # only process the last one (first generated token)
- if request.sampling_params.skip_prompt_log_probs and len(request_log_probs) > 1:
- # Only append the last log prob (first generated token) to generated_log_probs
- request.generated_log_probs.append(request_log_probs[-1])
- else:
- # Vectorized approach: calculate split point and use list slicing
- if not request.sampling_params.skip_prompt_log_probs:
- # Calculate how many log probs go to prompt vs generated
- remaining_prompt_slots = max(0, prompt_length - 1 - total_accumulated)
- split_idx = min(remaining_prompt_slots, len(request_log_probs))
-
- # Batch extend instead of individual appends
- if split_idx > 0:
- request.prompt_log_probs.extend(request_log_probs[:split_idx])
- if split_idx < len(request_log_probs):
- request.generated_log_probs.extend(request_log_probs[split_idx:])
- else:
- # All log probs go to generated
- request.generated_log_probs.extend(request_log_probs)
+ if split_idx > 0:
+ request.prompt_log_probs.extend(request_log_probs[:split_idx])
+ if split_idx < len(request_log_probs):
+ request.generated_log_probs.extend(request_log_probs[split_idx:])
# Process top_n_logprobs if available (unified for both regular and chunked prefill)
if top_n_logprobs is not None and req_idx in top_n_logprobs:
@@ -989,6 +1266,23 @@ def post_process_requests(
else:
request.generated_top_n_logprobs.append(logit_dict)
+ # Process routing indices if available (keyed by request_id)
+ # Each step's routing is a tensor of shape [num_tokens_this_step, num_layers, topk]
+ # We concatenate along dim=0 to accumulate: [total_tokens, num_layers, topk]
+ if (
+ routing_indices_per_request is not None
+ and request_id in routing_indices_per_request
+ ):
+ step_routing = routing_indices_per_request[
+ request_id
+ ] # [num_tokens, num_layers, topk]
+ if request.routing_indices is None:
+ request.routing_indices = step_routing.clone()
+ else:
+ request.routing_indices = torch.cat(
+ [request.routing_indices, step_routing], dim=0
+ )
+
# Handle evicted requests.
if evict_request_ids is not None and evict_request_ids.numel() > 0:
@@ -1034,61 +1328,150 @@ def _get_and_clear_stop_word_finished_ids(self, active_request_ids: list[int]) -
self.stop_word_finished_request_ids -= result
return result
- def _check_stop_words_for_request_post_append(self, request: DynamicInferenceRequest) -> bool:
+ def _check_stop_words_for_request_post_append(
+ self, request: DynamicInferenceRequest
+ ) -> Tuple[bool, int]:
"""Check if a request should stop due to stop words (after token is appended).
This method is called from post_process_requests after the token has already
- been appended to request.generated_tokens.
+ been appended to request.generated_tokens. In the speculative decoding case,
+ multiple tokens may have been appended at once. If a stop word is found in the
+ middle of the speculative tokens, the trailing tokens after the stop word are
+ truncated from generated_tokens.
+
+ With speculative decoding, multiple tokens are appended at once. The stop word
+ may end before the last appended token, leaving extra tokens that must be
+ trimmed. When this happens, generated_tokens is truncated in-place and the
+ number of trimmed tokens is returned so the caller can also trim log probs.
Args:
request: The request to check.
Returns:
- bool: True if the generated sequence ends with a stop word, False otherwise.
+ Tuple of (stop_word_hit, num_tokens_trimmed):
+ stop_word_hit: True if the generated sequence contains a stop word.
+ num_tokens_trimmed: Number of tokens removed from the end of
+ generated_tokens (0 when the stop word is at the very end
+ or when no stop word was found).
"""
- # Check if request has stop words configured
if request.stop_word_ids is None or len(request.stop_word_ids) == 0:
- return False
+ return False, 0
generated_tokens = request.generated_tokens
- # Check if the sequence ends with any stop word
for stop_word_ids in request.stop_word_ids:
stop_len = len(stop_word_ids)
if len(generated_tokens) >= stop_len:
- # Check if the last stop_len tokens match the stop word
- if list(generated_tokens[-stop_len:]) == stop_word_ids:
- return True
+ # Check the last stop_len tokens shifting by 1 up to num_speculative_tokens.
+ # Speculative decoding can append multiple tokens at once, so the stop
+ # word might end at any position within the newly appended tokens.
+ for i in range(self.num_speculative_tokens + 1):
+ end_idx = -i if i > 0 else None
+ if list(generated_tokens[-stop_len - i : end_idx]) == stop_word_ids:
+ trim = (
+ i if request.sampling_params.detokenize_stop_sequence else i + stop_len
+ )
+ if trim > 0:
+ request.generated_tokens = request.generated_tokens[:-trim]
+ return True, trim
+ return False, 0
- return False
+ def get_prefix_coordination_metrics(self) -> dict:
+ """Return prefix caching coordination metrics.
+
+ Returns:
+ Dict with coordination stats including the number of scheduling waits.
+ """
+ return {"waits": self._prefix_coordination_waits}
+
+ def _find_mamba_match_count(self, req: DynamicInferenceRequest) -> int:
+ """Find farthest block with cached Mamba state by iterating from the end.
+
+ Not all blocks have Mamba state cached in mamba_hash_to_block_id,
+ only divergence and last-aligned blocks do. Iterating from the end
+ finds the farthest block with cached state, which is the only one
+ needed for restore since Mamba state is cumulative.
+ """
+ if not req.precomputed_block_hashes:
+ return 0
+ mamba_map = self.context.mamba_slot_allocator.hash_to_block_id
+ for i in range(len(req.precomputed_block_hashes) - 1, -1, -1):
+ if req.precomputed_block_hashes[i] in mamba_map:
+ return i + 1
+ return 0
def schedule_waiting_requests(self):
"""Tries to schedule any requests in the waiting pool."""
+ # Keep track of which requests get scheduled.
+ waiting_before = set(self.waiting_request_ids)
if self.enable_chunked_prefill:
self.schedule_chunked_prefill()
else:
self.schedule_non_chunked_prefill()
+ waiting_after = set(self.waiting_request_ids)
+
+ # Re-stamp kv_cache_epoch on requests that were just scheduled.
+ if self._generation_epoch is not None:
+ for request_id in waiting_before - waiting_after:
+ req = self.get_request(request_id)
+ if req.kv_cache_epoch is None:
+ req.kv_cache_epoch = [(0, self._generation_epoch)]
def schedule_non_chunked_prefill(self):
"""
Perform the same original scheduling logic for non-chunked runs
"""
+ prefix_caching_enabled = self.context.enable_prefix_caching
+ mamba_caching_enabled = (
+ prefix_caching_enabled
+ and self.context.is_hybrid_model
+ and self.context.mamba_slot_allocator is not None
+ )
+ if prefix_caching_enabled:
+ pending_block_hashes = set()
+ pending_request_ids = []
while self.waiting_request_ids:
req = self.get_request(self.waiting_request_ids[0])
+
+ # Check for conflicting block hashes.
+ if prefix_caching_enabled:
+ has_pending_hash = False
+ for block_hash in req.precomputed_block_hashes:
+ if block_hash in pending_block_hashes:
+ has_pending_hash = True
+ break
+ if has_pending_hash:
+ self._prefix_coordination_waits += 1
+ pending_request_ids.append(self.waiting_request_ids.popleft())
+ continue
+
+ # Find Mamba prefix match before check_availability (sets skip count)
+ if mamba_caching_enabled:
+ req._mamba_num_matched_blocks = self._find_mamba_match_count(req)
+
request_can_be_added, request_tokens_can_be_added, kv_cache_available = (
self.context.check_availability(req)
)
if request_can_be_added and request_tokens_can_be_added and kv_cache_available:
+ # Add these hashes to pending.
+ if prefix_caching_enabled:
+ for block_hash in req.precomputed_block_hashes:
+ if block_hash not in self.context.kv_block_allocator.kv_hash_to_block_id:
+ pending_block_hashes.add(block_hash)
self.context.add_request(req)
self._loop.call_soon_threadsafe(
self._loop.create_task, self._notify_cond_for_new_request()
)
req.remaining_prompt_tokens = req.remaining_prompt_tokens.new_empty(0)
- req.add_event_add()
+ req.add_event_add_context()
self.waiting_request_ids.popleft()
else:
break
+ # Prepend pending request ids to waiting queue.
+ if prefix_caching_enabled and pending_request_ids:
+ self.waiting_request_ids.extendleft(reversed(pending_request_ids))
+
def schedule_chunked_prefill(self):
"""
This function schedules chunked prefill requests.
@@ -1104,6 +1487,15 @@ def schedule_chunked_prefill(self):
it is during a chunked prefill
- For each request, remaining_prompt_tokens holds the **unprefilled** prompt tokens
"""
+ prefix_caching_enabled = self.context.enable_prefix_caching
+ mamba_caching_enabled = (
+ prefix_caching_enabled
+ and self.context.is_hybrid_model
+ and self.context.mamba_slot_allocator is not None
+ )
+ if prefix_caching_enabled:
+ pending_block_hashes = set()
+ pending_request_ids = []
can_schedule = True
while self.waiting_request_ids and can_schedule:
can_schedule = False
@@ -1113,6 +1505,25 @@ def schedule_chunked_prefill(self):
# chunk of a existing chunked prefill request
is_continuing_chunked_prefill = self.context.chunked_prefill_request_id >= 0
+ # Check for conflicting block hashes.
+ if prefix_caching_enabled and not is_continuing_chunked_prefill:
+ has_pending_hash = False
+ for block_hash in req.precomputed_block_hashes:
+ # pylint: disable-next=possibly-used-before-assignment
+ if block_hash in pending_block_hashes:
+ has_pending_hash = True
+ break
+ if has_pending_hash:
+ self._prefix_coordination_waits += 1
+ pending_request_ids.append( # pylint: disable=possibly-used-before-assignment
+ self.waiting_request_ids.popleft()
+ )
+ continue
+
+ # Find Mamba prefix match for non-continuing requests
+ if mamba_caching_enabled and not is_continuing_chunked_prefill:
+ req._mamba_num_matched_blocks = self._find_mamba_match_count(req)
+
# Use remaining prompt tokens for scheduling decisions
remaining_len = len(req.remaining_prompt_tokens)
token_fully_can_be_added = (
@@ -1124,41 +1535,72 @@ def schedule_chunked_prefill(self):
if request_can_be_added and kv_cache_available:
if token_fully_can_be_added:
- # For Mamba models we need to ensure that the last prefill chunk
- # is still tagged as a chunked prefill request.
- self.context.has_explicit_chunked_prefill_req = (
- self.context.is_hybrid_model
- and self.context.chunked_prefill_request_id == req.request_id
- )
+ # Add these hashes to pending.
+ if prefix_caching_enabled:
+ for block_hash in req.precomputed_block_hashes:
+ if (
+ block_hash
+ not in self.context.kv_block_allocator.kv_hash_to_block_id
+ ):
+ pending_block_hashes.add(block_hash)
self.context.chunked_prefill_request_id = -1
self.context.add_request(req)
self._loop.call_soon_threadsafe(
self._loop.create_task, self._notify_cond_for_new_request()
)
req.remaining_prompt_tokens = req.remaining_prompt_tokens.new_empty(0)
- req.add_event_add()
+ req.add_event_add_context()
# Fully scheduled, so we remove from waiting pool
self.waiting_request_ids.popleft()
# Only this case we keep checking the rest of the waiting queue
- # We break early for Mamba models running a final prefill chunk
- # so that no additional requests are scheduled beyond the chunked
- # prefill request.
- can_schedule = not self.context.has_explicit_chunked_prefill_req
+ can_schedule = True
elif token_partially_can_be_added:
- chunk_length = self.context.max_tokens - self.context.active_token_count
- self.context.add_request(req, chunk_length=chunk_length)
+ # Add these hashes to pending.
+ if prefix_caching_enabled:
+ for block_hash in req.precomputed_block_hashes:
+ if (
+ block_hash
+ not in self.context.kv_block_allocator.kv_hash_to_block_id
+ ):
+ pending_block_hashes.add(block_hash)
+ prefill_chunk_length = self.context.max_tokens - self.context.active_token_count
+
+ # If this chunk would leave exactly 1 token for the final chunk, reduce
+ # this chunk by 1 or skip scheduling so the final chunk has 2 tokens.
+ # This avoids the edge case where max_seqlen_q=1 which results in a bug
+ # with the Flash Attention kernel.
+ # See https://github.com/Dao-AILab/flash-attention/issues/1537
+ if remaining_len - prefill_chunk_length == 1:
+ if prefill_chunk_length > 1:
+ prefill_chunk_length -= 1
+ else:
+ # We only have space for 1 token, but remaining is 2.
+ # Delay scheduling to avoid leaving exactly 1 token for the final chunk.
+ can_schedule = False
+ break
+
+ self.context.add_request(req, prefill_chunk_length=prefill_chunk_length)
self._loop.call_soon_threadsafe(
self._loop.create_task, self._notify_cond_for_new_request()
)
self.context.chunked_prefill_request_id = req.request_id
- self.context.has_explicit_chunked_prefill_req = self.context.is_hybrid_model
- req.remaining_prompt_tokens = req.remaining_prompt_tokens[chunk_length:]
- req.finished_chunk_token_count += chunk_length
+ req.remaining_prompt_tokens = req.remaining_prompt_tokens[prefill_chunk_length:]
+ req.finished_chunk_token_count += prefill_chunk_length
# Still have tokens to prefill, so we break and keep the
# chunked prefill request at the head of the waiting queue
# Note that we do not need to continue check the queue, as the tokens are full
- async def async_forward(self) -> Tuple[Dict, Dict, float, int]:
+ # Prepend pending request ids to waiting queue.
+ if prefix_caching_enabled and pending_request_ids:
+ is_continuing_chunked_prefill = self.context.chunked_prefill_request_id >= 0
+ if is_continuing_chunked_prefill:
+ chunked_request_id = self.waiting_request_ids.popleft()
+ self.waiting_request_ids.extendleft(reversed(pending_request_ids))
+ self.waiting_request_ids.appendleft(chunked_request_id)
+ else:
+ self.waiting_request_ids.extendleft(reversed(pending_request_ids))
+
+ async def async_forward(self) -> Tuple[Dict, Dict, float]:
"""Uses `asyncio` for continuous generation.
Sleeps when no requests are available, until new requests have been added.
@@ -1171,8 +1613,8 @@ async def async_forward(self) -> Tuple[Dict, Dict, float, int]:
"""
# If suspended, no stepping.
- if self.is_suspended:
- raise EngineSuspendedError(self.step_count)
+ if self.state in (EngineState.SUSPENDED, EngineState.SUSPENDING):
+ raise EngineSuspendedError(self.context.step_count)
# schedule requests
self.schedule_waiting_requests()
@@ -1185,6 +1627,7 @@ async def async_forward(self) -> Tuple[Dict, Dict, float, int]:
"total_request_count": self.context.total_request_count,
"paused_request_count": self.context.paused_request_count,
"active_token_count": self.context.active_token_count,
+ "step_count": self.context.step_count,
}
# Generate tokens.
@@ -1197,15 +1640,16 @@ async def async_forward(self) -> Tuple[Dict, Dict, float, int]:
self.step_end_event.record()
self.step_end_event.synchronize()
step_time = self.step_start_event.elapsed_time(self.step_end_event) / 1e3
- self.step_count += 1
+ self.context.step_count += 1
+ self.context.prefix_cache_lru_clock += 1
range_pop()
if (
- self.inference_logging_step_interval > 0
- and self.step_count > 0
- and self.step_count % self.inference_logging_step_interval == 0
- and self.context.metrics_writer is not None
+ self.logging_step_interval > 0
+ and self.context.step_count > 0
+ and self.context.step_count % self.logging_step_interval == 0
+ and self.metrics_writer is not None
):
kvcache_util_stats = self.context.get_kvcache_utilization_stats()
else:
@@ -1218,18 +1662,18 @@ async def async_forward(self) -> Tuple[Dict, Dict, float, int]:
"kv_stats": kvcache_util_stats,
"padded_active_token_count": self.context.padded_active_token_count,
"using_cuda_graph_this_step": self.context.using_cuda_graph_this_step(),
- "total_active_block_count": self.context.block_allocator.active_count,
- "total_paused_block_count": self.context.block_allocator.paused_count,
- "total_active_used_blocks": self.context.block_allocator.get_active_used(),
- "total_paused_used_blocks": self.context.block_allocator.get_paused_used(),
+ "total_active_block_count": self.context.kv_block_allocator.active_count,
+ "total_paused_block_count": self.context.kv_block_allocator.paused_count,
+ "total_active_used_blocks": self.context.kv_block_allocator.get_active_used(),
+ "total_paused_used_blocks": self.context.kv_block_allocator.get_paused_used(),
}
context_state = {**pre_step_context_state, **post_step_context_state}
- return result, context_state, step_time, self.step_count
+ return result, context_state, step_time
async def async_bookkeep(
- self, step_result: Optional[Dict], context_state: Dict, step_time: float, step_count: int
+ self, step_result: Optional[Dict], context_state: Dict, step_time: float
):
"""Uses `asyncio` for continuous bookkeeping.
@@ -1237,7 +1681,6 @@ async def async_bookkeep(
step_result (Optional[Dict]): The result of the step.
context_state (Dict): is_decode_only, total/paused request count, active token count.
step_time (float): How long this step took.
- step_count (int): The count of the step.
Returns:
A dictionary containing:
@@ -1256,8 +1699,10 @@ async def async_bookkeep(
newly_paused_request_ids = step_result.get("newly_paused_request_ids")
evict_request_ids = step_result.get("evict_request_ids")
sample = step_result["sample"]
+ accepted_tokens = step_result["accepted_tokens"]
log_probs = step_result["log_probs"]
top_n_logprobs = step_result.get("top_n_logprobs", None)
+ routing_indices_per_request = step_result.get("routing_indices_per_request", None)
cuda_graph_request_count = step_result["cuda_graph_request_count"]
# Add paused events.
@@ -1265,32 +1710,35 @@ async def async_bookkeep(
newly_paused_request_ids = newly_paused_request_ids.tolist()
[self.get_request(i).add_event_pause() for i in newly_paused_request_ids]
- # Mark requests finished.
- [self.get_request(i).add_event_finish() for i in finished_request_ids.tolist()]
- # Add finished events.
+ # Process finished requests (adds FINISH events and returns records).
(active_request_ids, finished_request_records) = self.post_process_requests(
active_request_ids,
finished_request_ids,
evict_request_ids,
step_time,
sample,
+ accepted_tokens,
log_probs,
top_n_logprobs,
+ routing_indices_per_request,
+ pre_fwd_active_token_count=context_state.get("active_token_count"),
+ pre_fwd_step_count=context_state.get("step_count"),
)
else:
active_request_ids: list[int] = []
finished_request_records: list[DynamicInferenceRequestRecord] = []
- # Failed requests.
+ # Failed requests. Status and events were already set in _handle_failed_request;
+ # here we just clean up the entry and include it in finished_request_records.
for failed_request_id in self.failed_request_ids:
failed_entry = self.requests.pop(failed_request_id)
- failed_request = failed_entry.record[-1]
- failed_request.status = Status.FAILED
- failed_request.add_event_fail()
finished_request_records.append(failed_entry.record)
- failed_entry.future.set_result(failed_entry.record)
+ assert (
+ failed_entry.future.done()
+ ), f"Failed request {failed_request_id} future has not been properly resolved."
self.failed_request_ids.clear()
+
range_pop()
# Detokenize all finished requests if not using
@@ -1301,30 +1749,49 @@ async def async_bookkeep(
for record in finished_request_records:
for request in record.requests:
if request.prompt is None:
- request.prompt = self.controller.tokenizer.detokenize(
- request.prompt_tokens.tolist()
+ request.prompt = self.controller.detokenize(
+ self.controller.tokenizer,
+ request.prompt_tokens.tolist(),
+ remove_EOD=False,
)
- request.generated_text = self.controller.tokenizer.detokenize(
- request.generated_tokens
+ request.generated_text = self.controller.detokenize(
+ self.controller.tokenizer,
+ request.generated_tokens,
+ remove_EOD=not request.sampling_params.detokenize_stop_sequence,
)
range_pop()
# Handle necessary ZMQ DP coordinator communication.
- if self.use_coordinator and self.is_mp_coordinator and finished_request_records:
- range_push("coordinator_communication")
- payload = msgpack.packb(
- [Headers.ENGINE_REPLY.value, [r.serialize() for r in finished_request_records]],
- use_bin_type=True,
- )
- self.socket_for_receiving_requests.send(payload)
- range_pop()
+ # Failed request replies were already sent in _handle_failed_request,
+ # so only send completed records here.
+ if self.use_coordinator and self.is_mp_coordinator:
+ records_to_send = [
+ r for r in finished_request_records if r.requests[-1].status != Status.FAILED
+ ]
+ if records_to_send:
+ range_push("coordinator_communication")
+ payload = msgpack.packb(
+ [Headers.ENGINE_REPLY.value, [r.merge().serialize() for r in records_to_send]],
+ use_bin_type=True,
+ )
+ self.socket_for_receiving_requests.send(payload)
+ range_pop()
+
+ # Drain prefix cache hit counters from context into engine accumulators.
+ if self.context.enable_prefix_caching:
+ self._prefix_cache_hits += self.context.prefix_cache_hits
+ self._prefix_cache_blocks_matched += self.context.prefix_cache_blocks_matched
+ self.context.prefix_cache_hits = 0
+ self.context.prefix_cache_blocks_matched = 0
# Log KV cache utilization stats to W&B
if context_state["kv_stats"] is not None:
# Prepare metrics dictionary with all stats
# Use 'inference/' prefix for all metrics to separate from training metrics
metrics = {
- 'inference/inference_step': int(self.inference_step_offset + int(step_count)),
+ 'inference/inference_step': int(
+ self.inference_step_offset + int(self.context.step_count)
+ ),
'inference/step_time_s': float(step_time),
'inference/waiting_queue_len': int(len(self.waiting_request_ids)),
'inference/total_requests_dict_size': int(len(self.requests)),
@@ -1338,30 +1805,43 @@ async def async_bookkeep(
else:
metrics[f'inference/{key}'] = value
- if HAVE_WANDB and self.context.metrics_writer.__name__ == "wandb":
- self.context.metrics_writer.log(metrics, commit=True)
- else:
- raise ValueError(
- f"Unsupported metrics writer type: {type(self.context.metrics_writer)}"
+ # Add speculative decoding acceptance metrics.
+ if self.num_speculative_tokens > 0 and self._spec_tokens_proposed > 0:
+ acceptance_rate = self._spec_tokens_accepted / self._spec_tokens_proposed
+ metrics['inference/spec_decode_acceptance_rate'] = float(acceptance_rate * 100.0)
+ metrics['inference/spec_decode_tokens_proposed'] = int(self._spec_tokens_proposed)
+ metrics['inference/spec_decode_tokens_accepted'] = int(self._spec_tokens_accepted)
+ metrics['inference/spec_decode_num_steps'] = int(self._spec_steps)
+
+ # Add prefix caching metrics.
+ if self.context.enable_prefix_caching and self._prefix_cache_hits > 0:
+ metrics['inference/prefix_cache_hits'] = int(self._prefix_cache_hits)
+ metrics['inference/prefix_cache_blocks_matched'] = int(
+ self._prefix_cache_blocks_matched
)
+ if HAVE_WANDB and self.metrics_writer.__name__ == "wandb":
+ self.metrics_writer.log(metrics, commit=True)
+ else:
+ raise ValueError(f"Unsupported metrics writer type: {type(self.metrics_writer)}")
+
# Print context state.
if (
- self.inference_logging_step_interval > 0
- and step_count % self.inference_logging_step_interval == 0
+ self.logging_step_interval > 0
+ and self.context.step_count % self.logging_step_interval == 0
):
mem = torch.cuda.memory_stats()
step_type = "decode" if context_state["is_decode_only"] else "non-decode"
output_str = (
- "* rank %d | step %d | %s ... time: %.3f%s ... "
+ "* rank %d | step %d | %s ... time: %.3f ms%s ... "
"reqs: a %d/%d, p %d, w %d, f %d, e %d ... "
"blocks: a %d/%d, p %d/%d ... "
"mem: tensors %d, alloc %.1f gb, res %.1f gb."
% (
self.rank,
- step_count,
+ self.context.step_count,
datetime.now().strftime("%H:%M:%S"),
- step_time,
+ step_time * 1000,
(
" [%s + real config %s + cuda graph %s]"
% (
@@ -1389,10 +1869,34 @@ async def async_bookkeep(
mem["reserved_bytes.all.current"] / (1024**3),
)
)
+ if self.num_speculative_tokens > 0 and self._spec_tokens_proposed > 0:
+ spec_rate = self._spec_tokens_accepted / self._spec_tokens_proposed * 100.0
+ output_str += " ... spec: accept %.1f%% (%d/%d in %d steps)" % (
+ spec_rate,
+ self._spec_tokens_accepted,
+ self._spec_tokens_proposed,
+ self._spec_steps,
+ )
+ if self.context.enable_prefix_caching and self._prefix_cache_hits > 0:
+ output_str += " ... prefix cache: %d hits, %d blocks matched" % (
+ self._prefix_cache_hits,
+ self._prefix_cache_blocks_matched,
+ )
if context_state["is_decode_only"]:
output_str = f"\033[94m{output_str}\033[0m"
logging.info(output_str)
+ # Reset speculative decoding accumulators after both wandb and console logging.
+ if self.num_speculative_tokens > 0:
+ self._spec_tokens_proposed = 0
+ self._spec_tokens_accepted = 0
+ self._spec_steps = 0
+
+ # Reset prefix caching accumulators after both wandb and console logging.
+ if self.context.enable_prefix_caching:
+ self._prefix_cache_hits = 0
+ self._prefix_cache_blocks_matched = 0
+
return {
"active_request_ids": active_request_ids,
"finished_request_records": finished_request_records,
@@ -1545,73 +2049,118 @@ def schedule_requests(self) -> int:
all_messages = []
range_pop()
+
+ # First pass: add requests.
+ # Control signals are queued for the second pass.
+ new_generation_epoch = None
for message in all_messages:
data = msgpack.unpackb(message, raw=False)
header = Headers(data[0])
-
- if self.received_stop:
- assert (
- header == Headers.STOP_ACK
- ), "Engine is shutting down. No other messages allowed except STOP_ACK."
-
if header == Headers.SUBMIT_REQUEST:
request_id, prompt, sampling_params = data[1:]
sampling_params = SamplingParams.deserialize(sampling_params)
range_push("add_request")
self.add_request(request_id, prompt, sampling_params)
range_pop()
- elif header == Headers.PAUSE:
- # Pause thyself.
- self.received_pause = True
- self.running.clear()
- # Send PAUSE_ACK back to coordinator.
- if self.is_mp_coordinator:
- payload = msgpack.packb([Headers.PAUSE_ACK.value], use_bin_type=True)
- self.socket_for_receiving_requests.send(payload)
- elif header == Headers.STOP:
- # Stop thyself.
- self.received_stop = True
- self.running.clear()
- # Send STOP_ACK back to coordinator.
- if self.is_mp_coordinator:
- payload = msgpack.packb([Headers.STOP_ACK.value], use_bin_type=True)
- self.socket_for_receiving_requests.send(payload)
- elif header == Headers.PAUSE_ACK:
- self.paused.set()
- self.received_pause = False
- elif header == Headers.STOP_ACK:
- self.stopped.set()
- self.received_stop = False
+ elif header == Headers.SET_GENERATION_EPOCH:
+ new_generation_epoch = data[1]
+ else:
+ # Control signal: queue for second pass.
+ self._pending_signals.append(message)
+
+ if new_generation_epoch is not None:
+ self._generation_epoch = new_generation_epoch
+ # Stamp all active requests with the new epoch.
+ # Each field stores a sparse list of (start_token_index, epoch) boundaries.
+ for entry in self.requests.values():
+ request = entry.record[-1]
+ total = len(request.prompt_tokens) + len(request.generated_tokens)
+ if total > 0:
+ boundary = (total - 1, new_generation_epoch)
+ if request.policy_epoch is None:
+ request.policy_epoch = [(0, new_generation_epoch)]
+ else:
+ request.policy_epoch.append(boundary)
+ if request.kv_cache_epoch is None:
+ request.kv_cache_epoch = [(0, new_generation_epoch)]
+ else:
+ request.kv_cache_epoch.append(boundary)
+
+ # Second pass: apply at most one control signal (the engine loop
+ # processes one state transition per iteration).
+ if self._pending_signals:
+ message = self._pending_signals.popleft()
+ data = msgpack.unpackb(message, raw=False)
+ header = Headers(data[0])
+
+ if header == Headers.PAUSE:
+ if self.state == EngineState.RUNNING:
+ self.state = EngineState.PAUSING
+ self._state_events[EngineState.RUNNING].clear()
+ # Any other state can safely ignore PAUSE.
+
elif header == Headers.UNPAUSE:
- self.paused.clear()
- self.running.set()
+ assert self.state == EngineState.PAUSED, f"Received UNPAUSE in state {self.state}"
+ self.state = EngineState.UNPAUSING
+
elif header == Headers.SUSPEND:
- self.suspend_signal = True
+ assert self.state == EngineState.PAUSED, f"Received SUSPEND in state {self.state}"
+ self._state_events[EngineState.RESUMED].clear()
+ self.suspend()
+ self.state = EngineState.SUSPENDING
+
elif header == Headers.RESUME:
- self.suspend_signal = False
+ assert self.state == EngineState.SUSPENDED, f"Received RESUME in state {self.state}"
+ self._state_events[EngineState.SUSPENDED].clear()
+ self.resume()
+ self.state = EngineState.RESUMING
+
elif header == Headers.STOP:
- self.received_stop = True
+ assert self.state in (
+ EngineState.PAUSED,
+ EngineState.SUSPENDED,
+ ), f"Received STOP in state {self.state}"
+ if self.state == EngineState.SUSPENDED:
+ self._state_events[EngineState.SUSPENDED].clear()
+ self.state = EngineState.STOPPING
+
else:
raise UnknownHeaderError(header)
return len(all_messages)
- def stop(self):
- """
- Stops the inference engine by terminating the inference coordinator process
- if it exists, and destroys the model parallel state.
- This method ensures that any running inference coordinator subprocess
- is properly terminated, and cleans up resources associated with
- model parallelism.
+ async def shutdown(self):
+ """Shut down the engine and clean up ZMQ resources.
+
+ Called from the engine loop's finally block after the loop exits.
"""
+ self.state = EngineState.STOPPED
- if hasattr(self, "inference_coordinator_process"):
- self.inference_coordinator_process.join()
- for socket in self.zmq_sockets:
- socket.close()
+ # Cleanup the request futures.
+ for entry in self.requests.values():
+ if not entry.future.done():
+ entry.future.cancel()
+
+ # ZMQ cleanup; designed to be idempotent.
+ sock = getattr(self, 'socket_for_receiving_requests', None)
+ if sock is not None and not sock.closed:
+ try:
+ sock.send(msgpack.packb([Headers.DISCONNECT.value], use_bin_type=True))
+ except Exception:
+ pass
+ for socket in getattr(self, 'zmq_sockets', []):
+ socket.close(linger=0)
+ if hasattr(self, 'zmq_sockets'):
+ self.zmq_sockets.clear()
if hasattr(self, "expert_parallel_zmq_communicator"):
self.expert_parallel_zmq_communicator.close()
- self.zmq_context.term()
+ if hasattr(self, "world_zmq_communicator"):
+ self.world_zmq_communicator.close()
+ if not self.zmq_context.closed:
+ self.zmq_context.term()
+
+ # Set the stopped state at the very end.
+ self._state_events[EngineState.STOPPED].set()
@trace_async_exceptions
async def run_engine(self, *, loop: Optional[asyncio.AbstractEventLoop] = None):
@@ -1624,7 +2173,7 @@ async def run_engine(self, *, loop: Optional[asyncio.AbstractEventLoop] = None):
async with self._cond:
await self._cond.wait_for(
lambda: (
- not self.is_suspended
+ self.state not in (EngineState.SUSPENDED, EngineState.SUSPENDING)
and (
self.context.get_active_request_count() > 0
or self.waiting_request_ids
@@ -1635,110 +2184,147 @@ async def run_engine(self, *, loop: Optional[asyncio.AbstractEventLoop] = None):
except asyncio.CancelledError:
pass
- async def _ep_group_has_work(self, local_work: int) -> bool:
- """Determines if there are some pending requests in the expert parallel group this
- rank is a part of.
+ async def _ep_establish_consensus(
+ self, local_work: int, signal_consensus: bool
+ ) -> tuple[int, bool]:
+ """EP all-reduce to share work counts and pause consensus.
+
+ All-reduces two integers at once:
+ - local_work: actual pending request count (always >= 0).
+ - consensus flag: -1 if this rank wants to pause, 0 otherwise.
+
+ Using max for both:
+ - max(work) > 0 means at least one EP peer has real work.
+ - max(consensus) == -1 means ALL peers signaled -1 (all PAUSING).
+ Any RUNNING peer contributes 0, pulling the max to 0.
+
Args:
- local_work (int): The local work count for this rank. This is a sum of active
- and waiting requests.
+ local_work: Pending request count for this rank.
+ signal_consensus: True if this rank is ready to pause.
Returns:
- bool: True if there is some work in the EP group, False otherwise.
+ (global_work, all_pausing): max work across EP, and whether
+ all peers signaled consensus.
"""
- range_push("_ep_group_has_work")
-
- is_stopped = self.stopped.is_set() or self.received_stop
- is_paused = self.paused.is_set() or self.received_pause
- is_suspended = self.suspend_signal
- if is_stopped or is_paused or is_suspended:
- # Signals can be received asynchronously on EP ranks.
- # We do not want a rank to pause/stop/suspend prematurely if one of it's peers
- # is yet to receive the signal.
- # So this is an *attempt* to process the signal. This rank has received the signal
- # and passes 0 to the all-reduce. If any other rank in the EP group has not received the signal yet,
- # it will pass a non-zero value to the all-reduce, and hence the global work will be non-zero,
- # and we will defer processing the signal.
- # When all ranks receive the signal, global work will be zero, and we can process the signal safely.
- local_work = 0
+ range_push("_ep_establish_consensus")
+
+ consensus_val = -1 if signal_consensus else 0
+
+ # Signals can be received asynchronously on EP ranks.
+ # We do not want a rank to pause prematurely if its peers have yet to receive the signal.
+ # So this is an *attempt* to process the signal. This rank has received the signal
+ # and passes -1 to the all-reduce. If any other rank in the EP group has not received
+ # the signal yet, it will pass a zero value to the all-reduce, hence the global consensus
+ # will be zero and we will defer processing the signal.
+ # When all ranks receive the signal, global consensus will be -1 and we can process.
if self.ep_world_size > 1:
- # Perform all-reduce to get max global work across EP group.
# Note that it is important to use a non-blocking asyncio-friendly all-reduce here.
# The user may have other tasks running in the event loop that need to be serviced.
# Do not using a torch.distributed blocking all-reduce here using nccl/gloo.
- # We have tried that and it blocks the event loop is megatron-rl.
- max_global_work = await self.expert_parallel_zmq_communicator.all_reduce_max(local_work)
+ # We have tried that and it blocks the event loop in megatron-rl.
+ global_work, global_consensus = (
+ await self.expert_parallel_zmq_communicator.all_reduce_max(
+ local_work, consensus_val, async_op=(not self.use_synchronous_zmq_collectives)
+ )
+ )
else:
- max_global_work = local_work
+ global_work, global_consensus = local_work, consensus_val
+
+ range_pop()
+ return global_work, global_consensus == -1
+ async def _world_barrier(self):
+ """World-wide ZMQ all-reduce barrier for global rank consensus.
+
+ Used for all state transitions that require global synchronization:
+ PAUSING → PAUSED, UNPAUSING → RUNNING, SUSPENDING → SUSPENDED,
+ RESUMING → PAUSED, and STOPPING → STOPPED.
+
+ No-op when world_size == 1 (communicator is not created).
+ """
+ range_push("world_barrier")
+ if hasattr(self, 'world_zmq_communicator'):
+ await self.world_zmq_communicator.all_reduce_max(
+ 1, async_op=(not self.use_synchronous_zmq_collectives)
+ )
range_pop()
- return max_global_work > 0
@trace_async_exceptions
async def run_engine_with_coordinator(
self, *, loop: Optional[asyncio.AbstractEventLoop] = None
):
- """Continually steps the engine asynchronously."""
+ """Continually steps the engine asynchronously.
+
+ State-dependent behavior:
+ - RUNNING: EP all-reduce to check for work, then step or idle.
+ - PAUSING: EP all-reduce to reach consensus, then world barrier.
+ - PAUSED / SUSPENDED: Idle-sleep, wait for signals via schedule_requests().
+ - UNPAUSING / SUSPENDING / RESUMING / STOPPING: World barrier, then transition.
+ - STOPPED: Teardown and exit.
+ """
self._loop = get_asyncio_loop(loop)
self.use_coordinator = True
+
try:
while True:
self.schedule_requests()
- # for the cases below (no active requests, or undergoing a state-change)
- # do not use asyncio.sleep(0)
- # as tp-rank=0 will flood the num_messages publisher
- # with "0" repeatedly. This causes some packets to drop.
- # Instead be nice, and sleep
- # for a short time.
- # The minimum sleep time needed is ~100us i.e. the time
- # needed to send one message on an IPC socket. However
- # just to be safe, we use 20ms here.
-
- local_pending_requests = self.context.get_active_request_count() + len(
- self.waiting_request_ids
- )
- # 1. Check for work availability (Consensus Step)
- ep_group_has_work = await self._ep_group_has_work(local_pending_requests)
-
- # 2. Dummy Work Logic (Keep group alive if peers have work)
- if ep_group_has_work and local_pending_requests == 0:
- # run dummy forward pass if EP group as a whole has work,
- # but this rank does not have any work.
- self.controller.dummy_forward()
- continue
-
- # 3. No work in EP group
- # We handle control signals (PAUSE/STOP/SUSPEND) only when
- # the entire EP group has received the signal. It is important to
- # not process these signals immediately upon receipt, because
- # other ranks in the EP group may not have received them yet.
- # If we exit prematurely, other ranks will deadlock at the all-to-all.
- # We use self._ep_group_has_work() to build consensus across the EP group
- # as to when it is safe to process these signals. The function returns False
- # when all ranks have received the signal.
- if not ep_group_has_work:
- # Priority A: STOP
- if self.stopped.is_set():
- if self.rank == 0:
- logging.info("Stopping engine.")
- self.stop()
- break
+ if self.state in (EngineState.RUNNING, EngineState.PAUSING):
+ local_pending = self.context.get_active_request_count() + len(
+ self.waiting_request_ids
+ )
+ global_work, all_pausing = await self._ep_establish_consensus(
+ local_pending, signal_consensus=(self.state == EngineState.PAUSING)
+ )
- # Priority B: SUSPEND
- if self.suspend_signal:
- self.suspend()
+ if all_pausing:
+ # All EP peers are PAUSING: pause immediately.
+ await self._world_barrier()
+ self.state = EngineState.PAUSED
+ self._state_events[EngineState.PAUSED].set()
+ elif global_work > 0:
+ # At least one EP peer has work: all must participate.
+ if local_pending > 0:
+ await self.async_step()
+ else:
+ # Dummy forward to participate in the EP collective.
+ self.step_start_event.record()
+ self.controller.dummy_forward()
+ self.step_end_event.record()
+ self.step_end_event.synchronize()
+ self.context.step_count += 1
+ self.context.prefix_cache_lru_clock += 1
else:
- self.resume()
-
- # Priority C: PAUSE or no work - nothing needs to be done
- # To avoid flooding the TP publisher socket with packets,
- # we sleep for 20 ms here.
- # todo [Siddharth]: Can this hardcoded sleep be avoided
- # with asyncio zmq sockets?
- await asyncio.sleep(0.02) # Yield to event loop
- continue
-
- await self.async_step()
+ # No work, but not all pausing: idle.
+ await asyncio.sleep(0.02)
+
+ elif self.state == EngineState.PAUSED:
+ await asyncio.sleep(0.02)
+
+ elif self.state == EngineState.UNPAUSING:
+ await self._world_barrier()
+ self.state = EngineState.RUNNING
+ self._state_events[EngineState.PAUSED].clear()
+ self._state_events[EngineState.RUNNING].set()
+
+ elif self.state == EngineState.SUSPENDING:
+ await self._world_barrier()
+ self.state = EngineState.SUSPENDED
+ self._state_events[EngineState.SUSPENDED].set()
+
+ elif self.state == EngineState.SUSPENDED:
+ await asyncio.sleep(0.02)
+
+ elif self.state == EngineState.RESUMING:
+ await self._world_barrier()
+ self.state = EngineState.PAUSED
+ self._state_events[EngineState.RESUMED].set()
+
+ elif self.state == EngineState.STOPPING:
+ await self._world_barrier()
+ if self.rank == 0:
+ logging.info("Stopping engine.")
+ break
- except asyncio.CancelledError:
- pass
+ finally:
+ await self.shutdown()
diff --git a/megatron/core/inference/engines/static_engine.py b/megatron/core/inference/engines/static_engine.py
index d4c61965d2b..0b3b9c1b856 100644
--- a/megatron/core/inference/engines/static_engine.py
+++ b/megatron/core/inference/engines/static_engine.py
@@ -8,7 +8,8 @@
import torch
from megatron.core.inference.async_stream import AsyncStream
-from megatron.core.inference.contexts.dynamic_context import DynamicInferenceContext
+from megatron.core.inference.config import InferenceConfig, MambaInferenceStateConfig
+from megatron.core.inference.contexts import DynamicInferenceContext, StaticInferenceContext
from megatron.core.inference.engines.abstract_engine import AbstractEngine
from megatron.core.inference.engines.dynamic_engine import DynamicInferenceEngine
from megatron.core.inference.inference_request import InferenceRequest
@@ -17,7 +18,7 @@
from megatron.core.inference.text_generation_controllers.text_generation_controller import (
TextGenerationController,
)
-from megatron.core.utils import get_asyncio_loop, get_mamba_inference_state_config_from_model
+from megatron.core.utils import get_asyncio_loop
try:
from tqdm import tqdm
@@ -42,8 +43,6 @@ class StaticInferenceEngine(AbstractEngine):
controller that will be used to define how to preprocess prompts, generate
outputs and detokenizer the output tokens.
max_batch_size (int, optional): The maximum number of requests to process at once.
- Will be set from the InferenceWrapperConfig in `text_generation_controller` by
- default.
random_seed (int, optional): Use a random seed if you want deterministic
results. Defaults to None.
"""
@@ -69,53 +68,55 @@ def __init__(
DeprecationWarning,
)
- inference_wrapper_config = (
- text_generation_controller.inference_wrapped_model.inference_wrapper_config
- )
self.controller = text_generation_controller
+ self.inference_wrapped_model = self.controller.inference_wrapped_model
+ self.config = self.inference_wrapped_model.config
self.random_seed = random_seed or 1234
- inference_max_batch_size = inference_wrapper_config.inference_max_requests
+ # Store original context in case we need to fall back to legacy static engine
+ original_context = self.inference_wrapped_model.inference_context
+ assert original_context is not None
+ assert isinstance(original_context, StaticInferenceContext)
+
if max_batch_size is None:
- max_batch_size = inference_max_batch_size
- elif max_batch_size > inference_max_batch_size:
+ max_batch_size = original_context.max_batch_size
+ elif max_batch_size > original_context.max_batch_size:
warnings.warn(
f"Engine `max_batch_size` ({max_batch_size}) > "
- f"`inference_max_requests` in `inference_wrapper_config` "
- f"({inference_max_batch_size}); setting `max_batch_size` to "
- f"{inference_max_batch_size}",
+ f"`context.max_batch_size` in `inference_wrapped_model.inference_context` "
+ f"({original_context.max_batch_size}); setting `max_batch_size` to "
+ f"{original_context.max_batch_size}",
UserWarning,
)
- max_batch_size = inference_max_batch_size
+ max_batch_size = original_context.max_batch_size
self.scheduler = Scheduler(max_batch_size=max_batch_size)
- # Store original context in case we need to fall back to legacy static engine
- original_context = text_generation_controller.inference_wrapped_model.inference_context
-
- mamba_inference_state_config = get_mamba_inference_state_config_from_model(
- text_generation_controller.inference_wrapped_model.model
+ mamba_inference_state_config = MambaInferenceStateConfig.from_model(
+ self.inference_wrapped_model.model
)
try:
if not legacy:
- dynamic_context = DynamicInferenceContext.from_config(
- inference_config=inference_wrapper_config,
- model=text_generation_controller.inference_wrapped_model.model,
- max_batch_size=max_batch_size,
- buffer_size_gb=buffer_size_gb,
- num_cuda_graphs=1,
- mamba_inference_state_config=mamba_inference_state_config,
+ dynamic_context = DynamicInferenceContext(
+ model_config=self.config,
+ inference_config=InferenceConfig(
+ max_sequence_length=original_context.max_sequence_length,
+ buffer_size_gb=buffer_size_gb,
+ mamba_inference_state_config=mamba_inference_state_config,
+ max_requests=max_batch_size,
+ num_cuda_graphs=1,
+ block_size_tokens=256,
+ unified_memory_level=0,
+ ),
)
+
self.controller.inference_wrapped_model.inference_context = dynamic_context
self.controller.inference_wrapped_model.prep_model_for_inference()
self.controller._init_dynamic_sampling_tensors()
self.dynamic_engine = DynamicInferenceEngine(
- controller=self.controller,
- random_seed=self.random_seed,
- context=dynamic_context,
- enable_cuda_graph=True,
+ controller=self.controller, context=dynamic_context
)
except Exception as e:
# Get exception details for better debugging
@@ -173,7 +174,9 @@ def add_request(
if inference_request is None:
# Support legacy single-arg tokenize_prompt mocks in tests.
- prompt_tokens = self.controller.tokenize_prompt(prompt, add_BOS)
+ prompt_tokens = self.controller.tokenize_prompt(
+ self.controller.tokenizer, prompt, add_BOS
+ )
else:
prompt_tokens = inference_request.prompt_tokens
@@ -229,13 +232,20 @@ def generate_using_dynamic_engine(
if prompts:
if add_BOS:
sampling_params.add_BOS = True
- return self.dynamic_engine.generate(prompts=prompts, sampling_params=sampling_params)
+ request_records = self.dynamic_engine.generate(
+ prompts=prompts, sampling_params=sampling_params
+ )
elif inference_requests:
prompts = [request.prompt for request in inference_requests]
sampling_params = inference_requests[0].sampling_params
if add_BOS:
sampling_params.add_BOS = True
- return self.dynamic_engine.generate(prompts=prompts, sampling_params=sampling_params)
+ request_records = self.dynamic_engine.generate(
+ prompts=prompts, sampling_params=sampling_params
+ )
+
+ # Return the underlying `InferenceRequest` objects from the `DynamicInferenceRequestRecord`s.
+ return [record.merge() for record in request_records]
def generate_using_legacy_static_engine(
self,
diff --git a/megatron/core/inference/headers.py b/megatron/core/inference/headers.py
index a22d1328679..aa2f0568975 100644
--- a/megatron/core/inference/headers.py
+++ b/megatron/core/inference/headers.py
@@ -13,16 +13,17 @@ class Headers(Enum):
SUBMIT_REQUEST = auto()
ENGINE_REPLY = auto()
PAUSE = auto()
- PAUSE_ACK = auto()
UNPAUSE = auto()
SUSPEND = auto()
RESUME = auto()
+ SET_GENERATION_EPOCH = auto()
STOP = auto()
- STOP_ACK = auto()
+ DISCONNECT = auto()
+ SHUTDOWN = auto()
class UnknownHeaderError(Exception):
"""A signal with an unrecognized header was received by the coordinator."""
- def __init_(self, header):
+ def __init__(self, header):
super().__init__(f"specialize for {header}.")
diff --git a/megatron/core/inference/inference_client.py b/megatron/core/inference/inference_client.py
index a927a393b8c..f5a78d1b16c 100644
--- a/megatron/core/inference/inference_client.py
+++ b/megatron/core/inference/inference_client.py
@@ -3,9 +3,9 @@
import asyncio
import logging
import time
-from typing import Awaitable, List, Optional, Union
+from typing import List, Optional, Union
-from megatron.core.inference.inference_request import DynamicInferenceRequestRecord
+from megatron.core.inference.inference_request import DynamicInferenceRequest
from megatron.core.inference.sampling_params import SamplingParams
from megatron.core.utils import get_asyncio_loop, trace_async_exceptions
@@ -25,8 +25,6 @@
except:
HAVE_MSGPACK = False
-from .headers import Headers
-
class InferenceClient:
"""
@@ -53,13 +51,16 @@ class InferenceClient:
completed requests.
"""
- def __init__(self, inference_coordinator_address: str):
+ def __init__(self, inference_coordinator_address: str, deserialize: bool = False):
"""
Initializes the InferenceClient.
Args:
inference_coordinator_address (str): The address on which the
inference coordinator is listening.
+ deserialize (bool): If True, deserialize completed requests
+ into DynamicInferenceRequest objects. If False (default), return
+ the raw serialized dict for lower overhead.
"""
assert (
HAVE_ZMQ
@@ -69,14 +70,16 @@ def __init__(self, inference_coordinator_address: str):
), "please install the messagepack library to use InferenceClient - pip install msgpack"
self.context = zmq.Context()
socket = self.context.socket(zmq.DEALER)
+
+ # Prevent socket.send() from thread-blocking at >1000 concurrent requests
+ socket.setsockopt(zmq.SNDHWM, 0)
+ socket.setsockopt(zmq.RCVHWM, 0)
+
socket.connect(inference_coordinator_address)
self._loop = None
- self.running = asyncio.Event()
- self.paused = asyncio.Event()
- self.stopped = asyncio.Event()
-
self.socket = socket
+ self.deserialize = deserialize
self.completion_futures = {}
self.request_submission_times = {}
self.next_request_id = 0
@@ -99,10 +102,9 @@ def add_request(
Returns:
asyncio.Future: A future that will be resolved with a
- `DynamicInferenceRequestRecord` object containing the completed result.
+ `DynamicInferenceRequest` object (if deserialize=True) or a raw
+ serialized dict (if deserialize=False) containing the completed result.
"""
- if not self.running.is_set():
- raise RuntimeError("InferenceClient is not currently running.")
request_id = self.next_request_id
self.next_request_id += 1
payload = [Headers.SUBMIT_REQUEST.value, request_id, prompt, sampling_params.serialize()]
@@ -139,14 +141,10 @@ async def _recv_task(self):
if completion_future.done():
logging.warning(f"Client: The future for {request_id} has been cancelled!")
continue
- completed_request = DynamicInferenceRequestRecord.deserialize(reply)
- completion_future.get_loop().call_soon_threadsafe(
- completion_future.set_result, completed_request
+ completed_request = (
+ DynamicInferenceRequest.deserialize(reply) if self.deserialize else reply
)
- elif header == Headers.PAUSE_ACK:
- self.paused.set()
- elif header == Headers.STOP_ACK:
- self.stopped.set()
+ completion_future.set_result(completed_request)
except zmq.Again:
await asyncio.sleep(0.005)
continue
@@ -165,79 +163,80 @@ def _connect_with_inference_coordinator(self):
reply = msgpack.unpackb(self.socket.recv(), raw=False)[0]
assert Headers(reply) == Headers.CONNECT_ACK
- async def start(self, loop: Optional[asyncio.AbstractEventLoop] = None):
+ def start(self, loop: Optional[asyncio.AbstractEventLoop] = None):
"""
Connects to the coordinator and starts the background listener task.
- This method must be awaited before submitting any requests. It handles
+ This must be called before submitting any requests. It handles
the initial handshake and spawns the `listen_for_completed_requests`
coroutine.
"""
logging.info("Client: Connecting to InferenceCoordinator...")
self._loop = get_asyncio_loop(loop)
- self.running.set()
- self.paused.clear()
- self.stopped.clear()
self._connect_with_inference_coordinator()
self.listener_task = self._loop.create_task(self._recv_task())
- def _send_signal_to_engines(self, signal):
+ def _send_signal_to_engines(self, signal, *args):
"""
Sends a generic control signal to the inference coordinator.
Args:
signal: The signal to send, typically a value from the `Headers` enum.
+ *args: Optional extra values to include in the payload.
"""
- payload = [signal.value]
+ payload = [signal.value, *args]
payload_serialized = msgpack.packb(payload, use_bin_type=True)
self.socket.send(payload_serialized)
- def pause_engines(self) -> Awaitable:
- """Sends a signal to pause all inference engines.
-
- The signal first propagates thru the coordinator to all engines.
- All engines acknowledge this signal and clear their `running` flags.
- The coordinator awaits all acknowledgements before forwarding the ACK
- back to the client, as well as to the engines.
- The engines set their `paused` flags upon seeing the ACK.
+ def pause_engines(self):
+ """Sends PAUSE to all engines via coordinator.
- Returns:
- Awaitable: An awaitable that resolves when all engines have paused.
+ The coordinator broadcasts PAUSE. Each engine reaches EP consensus,
+ then synchronizes via a world-wide barrier before transitioning to
+ PAUSED. Callers should await engine.paused for confirmation.
"""
self._send_signal_to_engines(Headers.PAUSE)
- return self.paused.wait()
def unpause_engines(self) -> None:
- """Sends a signal to unpause all inference engines."""
- self.paused.clear()
- self.running.set()
+ """Sends UNPAUSE to all engines. No synchronization needed."""
self._send_signal_to_engines(Headers.UNPAUSE)
+ def set_generation_epoch(self, generation_epoch: int):
+ """Sends a signal to stamp all in-flight requests with the given generation epoch.
+
+ Args:
+ generation_epoch: The current generation epoch number.
+ """
+ self._send_signal_to_engines(Headers.SET_GENERATION_EPOCH, generation_epoch)
+
def suspend_engines(self):
- """Sends a signal to pause all inference engines."""
- self._send_signal_to_engines(Headers.PAUSE)
+ """Sends SUSPEND to all engines via coordinator. Requires PAUSED.
+
+ Callers should await engine.suspended for confirmation.
+ """
self._send_signal_to_engines(Headers.SUSPEND)
def resume_engines(self):
- """Sends a signal to unpause all inference engines."""
- self._send_signal_to_engines(Headers.RESUME)
- self._send_signal_to_engines(Headers.UNPAUSE)
+ """Sends RESUME to all engines via coordinator. Requires SUSPENDED.
- def stop_engines(self) -> Awaitable:
- """Sends a signal to gracefully stop all inference engines.
+ Callers should await engine.paused (or engine.running after UNPAUSE) for confirmation.
+ """
+ self._send_signal_to_engines(Headers.RESUME)
- The signal first propagates thru the coordinator to all engines.
- All engines acknowledge this signal and clear their `running` flags.
- The coordinator awaits all acknowledgements before forwarding the ACK
- back to the client, as well as to the engines.
- The engines set their `stopped` flags upon seeing the ACK.
+ def stop_engines(self):
+ """Sends STOP to all engines via coordinator. Requires PAUSED or SUSPENDED.
- Returns:
- Awaitable: An awaitable that resolves when all engines have stopped.
+ Callers should await engine.stopped for confirmation.
+ Does not affect the coordinator.
"""
self._send_signal_to_engines(Headers.STOP)
- self.running.clear()
- return self.stopped.wait()
+
+ def shutdown_coordinator(self):
+ """Tells the coordinator process to exit its main loop.
+
+ Does not affect the engines.
+ """
+ self._send_signal_to_engines(Headers.SHUTDOWN)
def stop(self):
"""
@@ -247,6 +246,12 @@ def stop(self):
and terminates the ZMQ context. It should be called when the client is
no longer needed to ensure a graceful shutdown.
"""
- self.listener_task.cancel()
- self.socket.close()
+ if hasattr(self, 'listener_task') and not self.listener_task.done():
+ self.listener_task.cancel()
+ # Wake up any listeners.
+ for future in self.completion_futures.values():
+ if not future.done():
+ future.cancel()
+ self.completion_futures.clear()
+ self.socket.close(linger=0)
self.context.term()
diff --git a/megatron/core/inference/inference_request.py b/megatron/core/inference/inference_request.py
index 6a7354220f9..27580f4b830 100644
--- a/megatron/core/inference/inference_request.py
+++ b/megatron/core/inference/inference_request.py
@@ -5,6 +5,7 @@
import warnings
from dataclasses import asdict, dataclass, field
from enum import Enum, auto
+from itertools import accumulate
from typing import Any, Dict, List, Optional, Tuple
import torch
@@ -45,6 +46,21 @@ def deserialize_tensor(tensor_as_list: List) -> torch.Tensor:
return tensor
+def unwrap_serialized_tensors(serialized_request: dict) -> dict:
+ """Unwrap ("tensor", [...]) tuples produced by serialize() into plain lists.
+
+ Args:
+ serialized_request (dict): A dict produced by `serialize()`.
+
+ Returns:
+ dict: A shallow copy with tensor wrapper tuples replaced by their inner lists.
+ """
+ return {
+ k: v[1] if isinstance(v, (list, tuple)) and len(v) == 2 and v[0] == "tensor" else v
+ for k, v in serialized_request.items()
+ }
+
+
# class syntax
class Status(Enum):
"""Enum for status"""
@@ -56,6 +72,61 @@ class Status(Enum):
FAILED = 5
+# =========================================================================
+# Hash computation for prefix caching
+# =========================================================================
+
+# Constants for hash computation
+# Using 2^61 - 1 (Mersenne prime) for ~10^18 hash space, reducing collision probability
+# from ~10^-9 to ~10^-18 compared to the previous prime (1000000007).
+HASH_PRIME = 2305843009213693951
+HASH_BASE = 31
+
+_hash_powers: Optional[torch.Tensor] = None
+
+
+def compute_block_hashes_batched(prompt_tokens: torch.Tensor, block_size: int) -> List[int]:
+ """Compute hashes for all complete blocks in a prompt in one batched operation.
+
+ Reshapes prompt tokens into [num_blocks, block_size], computes all per-block
+ token hashes via a single GPU matmul, transfers results with one .tolist() call,
+ and chains parent hashes on CPU.
+
+ Args:
+ prompt_tokens: All prompt token IDs, shape [seq_len].
+ block_size: Number of tokens per block.
+
+ Returns:
+ List of positive integer hash values (1 to HASH_PRIME), one per complete block.
+ """
+ num_complete_blocks = len(prompt_tokens) // block_size
+ if num_complete_blocks == 0:
+ return []
+
+ global _hash_powers
+ if _hash_powers is None or _hash_powers.shape[0] != block_size:
+ positions = torch.arange(block_size, device=prompt_tokens.device, dtype=torch.int64)
+ _hash_powers = torch.pow(HASH_BASE, positions).to(torch.int64) % HASH_PRIME
+
+ # Reshape to [num_blocks, block_size] (zero-copy view) and compute all token hashes
+ blocks = prompt_tokens[: num_complete_blocks * block_size].view(num_complete_blocks, block_size)
+ token_hashes = (blocks.to(torch.int64) * _hash_powers).sum(dim=1) % HASH_PRIME
+
+ # Single GPU→CPU transfer
+ token_hashes_list = token_hashes.tolist()
+
+ # Chain parent hashes on CPU (C-level accumulate, no Python loop)
+ hashes = list(
+ accumulate(
+ token_hashes_list,
+ lambda parent, th: (parent * HASH_BASE + th) % HASH_PRIME + 1,
+ initial=0,
+ )
+ )[1:]
+
+ return hashes
+
+
@dataclass(kw_only=True)
class InferenceRequest:
"""Class for one inference request
@@ -159,7 +230,9 @@ def _post_deserialize(self, obj: dict):
class DynamicInferenceEventType(Enum):
"""Dynamic inference event type."""
- ADD = auto()
+ ADD_ENGINE = auto() # When request is added to engine via _add_request()
+ ADD_CONTEXT = auto() # When request is added to context (scheduled for prefill)
+ GENERATED_TOKEN = auto() # When an output token is generated (payload = {"token_id": int})
PAUSE = auto()
EVICT = auto()
FINISH = auto()
@@ -202,22 +275,30 @@ def __post_init__(self):
DynamicInferenceEventType.ERROR_NONTRANSIENT,
):
assert self.payload is not None
+ elif self.type == DynamicInferenceEventType.GENERATED_TOKEN:
+ assert (
+ self.payload is not None
+ and isinstance(self.payload, dict)
+ and "token_id" in self.payload
+ )
else:
assert self.payload is None
def __str__(self):
- payload_str = "" if self.payload is None else f", {type(self.payload).__name__}"
+ if self.type == DynamicInferenceEventType.GENERATED_TOKEN:
+ payload_str = f", token={self.payload['token_id']}"
+ elif self.payload is None:
+ payload_str = ""
+ else:
+ payload_str = f", {type(self.payload).__name__}"
return f"[{self.timestamp:.3f}] {self.type.name}{payload_str}"
def serialize(self) -> dict:
"""Converts the instance into a serializable dictionary.
Returns:
- (dict) A dictionary representation of the instance suitable for
- serialization.
+ dict: Full event dict.
"""
-
- # Dataclass to dict.
torch.cuda.nvtx.range_push("DynamicInferenceEvent.serialize")
# do not use asdict(self) - it has very high CPU overheads
# and if there are tensors, it will try to deepcopy them
@@ -225,10 +306,15 @@ def serialize(self) -> dict:
obj["type"] = self.type.name
# Serialize payload.
- if self.payload:
- from .contexts.dynamic_context import ContextErrorFactory # avoid circular import.
+ if self.payload is not None:
+ if self.type in (
+ DynamicInferenceEventType.ERROR_TRANSIENT,
+ DynamicInferenceEventType.ERROR_NONTRANSIENT,
+ ):
+ from .contexts.dynamic_context import ContextErrorFactory # avoid circular import.
+
+ obj["payload"] = ContextErrorFactory.serialize(self.payload)
- obj["payload"] = ContextErrorFactory.serialize(self.payload)
torch.cuda.nvtx.range_pop()
return obj
@@ -237,22 +323,25 @@ def deserialize(cls, obj: dict) -> "DynamicInferenceEvent":
"""Deserialize event.
Args:
- obj (dict): Serialized event data.
+ obj: Serialized event data dict.
Returns:
(DynamicInferenceEvent) Deserialized event.
"""
+ event_type = DynamicInferenceEventType[obj["type"]]
- # Initialize event.
- event = cls(**{**obj, "type": DynamicInferenceEventType[obj["type"]]})
+ # Pre-process payload before construction (since __post_init__ validates types).
+ init_obj = {**obj, "type": event_type}
+ if obj["payload"] is not None:
+ if event_type in (
+ DynamicInferenceEventType.ERROR_TRANSIENT,
+ DynamicInferenceEventType.ERROR_NONTRANSIENT,
+ ):
+ from .contexts.dynamic_context import ContextErrorFactory # avoid circular import.
- # Deserialize payload.
- if obj["payload"]:
- from .contexts.dynamic_context import ContextErrorFactory # avoid circular import.
+ init_obj["payload"] = ContextErrorFactory.deserialize(obj["payload"])
- event.payload = ContextErrorFactory.deserialize(obj["payload"])
-
- return event
+ return cls(**init_obj)
@experimental_api
@@ -265,19 +354,50 @@ class DynamicInferenceRequest(InferenceRequest):
"""
request_id: int
- generated_tokens: List[int] = field(default_factory=list)
prompt: Optional[str] = None
prompt_tokens: Optional[torch.Tensor] = None
# remaining prompt tokens are used for chunked prefill
remaining_prompt_tokens: Optional[torch.Tensor] = None
+ policy_epoch: Optional[list[tuple[int, int]]] = None
+ kv_cache_epoch: Optional[list[tuple[int, int]]] = None
latency: Optional[float] = None
+ # routing_indices stores MoE routing decisions for all tokens generated so far.
+ # Shape: [total_tokens, num_layers, topk] - accumulated across all generation steps
+ routing_indices: Optional[torch.Tensor] = None
finished_chunk_token_count: int = 0
stop_word_ids: Optional[List[List[int]]] = None # Tokenized stop words (populated internally)
+ # Prefix caching fields
+ block_size_tokens: Optional[int] = None # Block size for hash computation
+ enable_prefix_caching: bool = False # Whether prefix caching is enabled
+
+ # Computed field - not passed by caller
+ precomputed_block_hashes: List[int] = field(default_factory=list)
+
def __post_init__(self):
self.sampling_params = copy.deepcopy(self.sampling_params)
if self.prompt_tokens is not None:
- self.remaining_prompt_tokens = copy.deepcopy(self.prompt_tokens)
+ self.remaining_prompt_tokens = self.prompt_tokens
+
+ # Compute block hashes for prefix matching (skip if already provided, e.g. from `merge`).
+ if (
+ self.enable_prefix_caching
+ and self.block_size_tokens is not None
+ and self.prompt_tokens is not None
+ and not self.precomputed_block_hashes
+ ):
+ self._compute_block_hashes()
+
+ def _compute_block_hashes(self) -> None:
+ """Compute hashes for all complete blocks in the prompt.
+
+ After this call:
+ - precomputed_block_hashes is [] if prompt < block_size (no complete blocks)
+ - precomputed_block_hashes is [hash1, ...] for N complete blocks
+ """
+ self.precomputed_block_hashes = compute_block_hashes_batched(
+ self.prompt_tokens, self.block_size_tokens
+ )
@property
def remaining_prompt_length(self):
@@ -286,7 +406,10 @@ def remaining_prompt_length(self):
"""
return len(self.remaining_prompt_tokens)
+ ttft: Optional[float] = None
events: List[DynamicInferenceEvent] = field(default_factory=list)
+ event_add_engine: Optional[DynamicInferenceEvent] = field(default=None, repr=False)
+ generated_tokens: List[int] = field(default_factory=list)
def __str__(self):
return ", ".join(
@@ -299,7 +422,7 @@ def __str__(self):
)
)
- def serialize(self) -> dict:
+ def serialize(self):
"""Converts the instance into a serializable dictionary.
Returns:
@@ -309,12 +432,24 @@ def serialize(self) -> dict:
torch.cuda.nvtx.range_push("DynamicInferenceRequest.serialize")
obj = super().serialize()
obj["events"] = [e.serialize() for e in self.events]
+ obj.pop("event_add_engine", None)
+
+ # Sanity check routing_indices: Tensor [total_tokens - 1, num_layers, topk]
+ if self.routing_indices is not None:
+ total_tokens = len(self.prompt_tokens) + len(self.generated_tokens)
+ # the last generated token does not undergo a forward pass
+ # hence we expect routing indices for total_tokens - 1
+ assert self.routing_indices.shape[0] == total_tokens - 1, (
+ f"routing_indices first dimension {self.routing_indices.shape[0]} does not match "
+ f"total tokens {total_tokens-1}."
+ )
+
torch.cuda.nvtx.range_pop()
return obj
def _post_deserialize(self, obj):
super()._post_deserialize(obj)
- self.events = [DynamicInferenceEvent.deserialize(e) for e in obj["events"]]
+ self.events = [DynamicInferenceEvent.deserialize(e) for e in obj.get("events", [])]
@property
def tracked_metadata(self) -> List[Any]:
@@ -356,13 +491,58 @@ def get_metadata_types() -> List[Tuple[str, torch.dtype, bool]]:
("top_n_logprobs", torch.int32, False), # CPU for torch sampling
]
- def add_event(self, type: DynamicInferenceEventType, payload: Optional[Any] = None) -> None:
+ def add_event(
+ self, type: DynamicInferenceEventType, payload: Optional[Any] = None
+ ) -> DynamicInferenceEvent:
"""Add event."""
- self.events.append(DynamicInferenceEvent(type=type, payload=payload))
+ event = DynamicInferenceEvent(type=type, payload=payload)
+ self.events.append(event)
+ return event
+
+ def add_event_add_engine(self):
+ """Add 'add_engine' event - called when request enters the engine queue."""
+ self.event_add_engine = self.add_event(DynamicInferenceEventType.ADD_ENGINE)
+ return self.event_add_engine
+
+ def add_event_add_context(self):
+ """Add 'add_context' event - called when request is added to context for prefill."""
+ return self.add_event(DynamicInferenceEventType.ADD_CONTEXT)
+
+ def add_event_generated_token(
+ self,
+ token: int,
+ blocks_total: Optional[int] = None,
+ blocks_hashed_total: Optional[int] = None,
+ blocks_hashed_active: Optional[int] = None,
+ blocks_ref_count: Optional[int] = None,
+ pre_fwd_active_token_count: Optional[int] = None,
+ pre_fwd_step_count: Optional[int] = None,
+ ):
+ """Add 'generated_token' event - records each generated token.
- def add_event_add(self):
- """Add 'add' event."""
- return self.add_event(DynamicInferenceEventType.ADD)
+ Args:
+ token (int): The token ID that was generated.
+ blocks_total (int): Total block capacity from allocator.
+ blocks_hashed_total (int): All allocated (hashed) blocks.
+ blocks_hashed_active (int): Blocks with ref_count > 0.
+ blocks_ref_count (int): Sum of block ref counts from allocator.
+ pre_fwd_active_token_count (int): Active token count before forward pass.
+ pre_fwd_step_count (int): Step count before forward pass.
+ """
+ payload = {"token_id": token}
+ if blocks_total is not None:
+ payload["blocks_total"] = blocks_total
+ if blocks_hashed_total is not None:
+ payload["blocks_hashed_total"] = blocks_hashed_total
+ if blocks_hashed_active is not None:
+ payload["blocks_hashed_active"] = blocks_hashed_active
+ if blocks_ref_count is not None:
+ payload["blocks_ref_count"] = blocks_ref_count
+ if pre_fwd_active_token_count is not None:
+ payload["pre_fwd_active_token_count"] = pre_fwd_active_token_count
+ if pre_fwd_step_count is not None:
+ payload["pre_fwd_step_count"] = pre_fwd_step_count
+ return self.add_event(DynamicInferenceEventType.GENERATED_TOKEN, payload)
def add_event_pause(self):
"""Add 'pause' event."""
@@ -449,6 +629,13 @@ def checkpoint(self, tokenizer: MegatronTokenizer | None = None):
old_request = self[-1]
+ # Carry forward policy_epoch as-is.
+ policy_epoch = old_request.policy_epoch
+
+ # Reset kv_cache_epoch to None: the KV cache is recomputed fresh after checkpoint;
+ # the engine's stamping logic will initialize a new stamp record with the recompute epoch.
+ kv_cache_epoch = None
+
# New prompt (concatenate prompt + generated tokens).
new_prompt_tokens = torch.cat(
(
@@ -478,7 +665,15 @@ def checkpoint(self, tokenizer: MegatronTokenizer | None = None):
request_id=old_request.request_id,
prompt_tokens=new_prompt_tokens,
sampling_params=new_sampling_params,
+ policy_epoch=policy_epoch,
+ kv_cache_epoch=kv_cache_epoch,
)
+ # Preserve event_add_engine from old request if it exists, otherwise set it.
+ # This ensures TTFT calculation works correctly for evicted/resumed requests.
+ if old_request.event_add_engine is not None:
+ new_request.event_add_engine = old_request.event_add_engine
+ else:
+ new_request.add_event_add_engine()
self.requests.append(new_request)
def merge(self, tokenizer: MegatronTokenizer | None = None) -> DynamicInferenceRequest:
@@ -499,12 +694,18 @@ def merge_lists(key):
prompt_tokens = self.requests[0].prompt_tokens
prompt_text = self.requests[0].prompt
+ routing_indices = None
+ if self.requests[0].routing_indices is not None:
+ routing_indices = torch.cat([r.routing_indices for r in self.requests])
generated_tokens = merge_lists("generated_tokens")
try:
generated_text = "".join(r.generated_text for r in self.requests)
except TypeError as e: # generally means r.generated_text is None
generated_text = None
+ policy_epoch = self.requests[-1].policy_epoch
+ kv_cache_epoch = self.requests[-1].kv_cache_epoch
+
# Merged request.
request = DynamicInferenceRequest(
request_id=self.requests[0].request_id,
@@ -518,10 +719,17 @@ def merge_lists(key):
generated_log_probs=merge_lists("generated_log_probs"),
generated_top_n_logprobs=merge_lists("generated_top_n_logprobs"),
sampling_params=self.requests[0].sampling_params,
+ policy_epoch=policy_epoch,
+ kv_cache_epoch=kv_cache_epoch,
+ ttft=self.requests[0].ttft,
tpot=merge_lists("tpot"),
status=self.requests[-1].status,
latency=self.latency,
events=merge_lists("events"),
+ routing_indices=routing_indices,
+ block_size_tokens=self.requests[0].block_size_tokens,
+ enable_prefix_caching=self.requests[0].enable_prefix_caching,
+ precomputed_block_hashes=self.requests[0].precomputed_block_hashes,
)
return request
diff --git a/megatron/core/inference/model_inference_wrappers/abstract_model_inference_wrapper.py b/megatron/core/inference/model_inference_wrappers/abstract_model_inference_wrapper.py
index 6a17de685bf..55efb24cb08 100644
--- a/megatron/core/inference/model_inference_wrappers/abstract_model_inference_wrapper.py
+++ b/megatron/core/inference/model_inference_wrappers/abstract_model_inference_wrapper.py
@@ -1,8 +1,6 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
import abc
-import math
-import warnings
from typing import Any, Dict, Iterable, Optional, Union
import torch
@@ -15,103 +13,67 @@
send_to_next_pipeline_rank,
)
from megatron.core.inference.contexts import BaseInferenceContext
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
from megatron.core.models.gpt.gpt_model import GPTModel
from megatron.core.process_groups_config import ProcessGroupCollection
-from megatron.core.utils import get_attr_wrapped_model, get_model_config
+from megatron.core.utils import deprecate_args, get_attr_wrapped_model, get_model_config
+
+DEPRECATED_ARGS = ["inference_wrapper_config", "pg_collection"]
-# pylint: disable=line-too-long
class AbstractModelInferenceWrapper(abc.ABC):
"""Abstract inference wrapper
Extend this to create a version for your model.
- The wrapper prepares the model for inference, provides the required input data and runs the forward pass.
+ The wrapper prepares the model for inference, provides the required input data and
+ runs the forward pass.
Args:
model (Union[GPTModel, LegacyGPTModel]): The actual GPT model (MCore
or MLM).
- inference_wrapper_config (InferenceWrapperConfig): Has info like
- hidden size, vocab size etc.
inference_context (BaseInferenceContext): Context for managing KV
cache and other inference params.
- pg_collection (ProcessGroupCollection): Process groups for model communication.
"""
+ @deprecate_args(*DEPRECATED_ARGS)
def __init__(
self,
model: Union['LegacyGPTModel', GPTModel], # type: ignore[name-defined]
- inference_wrapper_config: InferenceWrapperConfig,
- inference_context: Optional[BaseInferenceContext] = None,
- pg_collection: Optional[ProcessGroupCollection] = None,
+ inference_context: BaseInferenceContext,
):
assert not isinstance(
model, Iterable
), 'interleaving schedule is not supported for inference'
self.model = model
- self.inference_wrapper_config = inference_wrapper_config
+ self.config = get_model_config(self.model)
self.pipeline_communication_dtype = (
- torch.float
- if self.inference_wrapper_config.fp32_residual_connection
- else self.inference_wrapper_config.params_dtype
+ torch.float if self.config.fp32_residual_connection else self.config.params_dtype
)
- model_config = get_model_config(self.model)
- self.sequence_parallel = model_config.sequence_parallel
-
- if inference_context is None:
- warnings.warn(
- "`inference_context` must be passed in as an argument starting in `megatron-core` 0.13."
- )
- from megatron.core.inference.contexts import StaticInferenceContext
-
- inference_context = StaticInferenceContext.from_config(inference_wrapper_config)
+ self.sequence_parallel = self.config.sequence_parallel
self.inference_context = inference_context
- if pg_collection is None:
+ # Get the inference pg_collection from the config if it exists; otherwise the training
+ # pg_collection might be used during RL
+ if (pg_collection := self.inference_context.config.pg_collection) is None:
pg_collection = ProcessGroupCollection.use_mpu_process_groups()
self.tp_group = pg_collection.tp
self.pp_group = pg_collection.pp
self.tp_size = torch.distributed.get_world_size(self.tp_group)
- if self.inference_wrapper_config.fp8 is not None:
+ if self.config.fp8 is not None and self.config.transformer_impl != "inference_optimized":
self.model = prepare_model_for_fp8_inference(self.model)
- @property
- def inference_params(self):
- """Getter for deprecated `inference_params`."""
- warnings.warn(
- "`inference_params` renamed to `inference_context`, and will be removed in `megatron-core` 0.13."
- )
- return self.inference_context
-
- @inference_params.setter
- def inference_params(self, value):
- """Setter for deprecated `inference_params`."""
- warnings.warn(
- "`inference_params` renamed to `inference_context`, and will be removed in `megatron-core` 0.13."
- )
- self.inference_context = value
+ # TODO(ksanthanam): Add support for fp4
- def prep_model_for_inference(self, prompts_tokens: Optional[torch.Tensor] = None):
+ def prep_model_for_inference(self):
"""A utility function for preparing model for inference
The function gets called once before the auto regressive inference loop.
It puts the model in eval mode.
- Args:
- prompts_tokens (torch.Tensor, optional): Deprecated, will be removed in `megatron-core` 0.13
"""
- if prompts_tokens is not None:
- warnings.warn(
- "Passing `prompts_tokens` is deprecated and this argument will be ignored."
- "This parameter will be removed in `megatron-core` 0.13."
- )
-
self.model.eval()
# For TP only model both is_pp_first_stage and _is_pp_last_stage returns True
@@ -137,7 +99,9 @@ def prep_inference_input(self, prompt_tokens) -> Dict[str, Any]:
def get_batch_for_context_window(self, *args, **kwargs) -> Dict[str, Any]:
"""Returns the input data for inference
- This function gets called iteratively in the inference loop . It can be used to extract relevant input from the prompt tokens, attention mask etc. required for each step in inference.
+ This function gets called iteratively in the inference loop.
+ It can be used to extract relevant input from the prompt tokens, attention mask etc.
+ required for each step in inference.
"""
raise NotImplementedError()
@@ -162,11 +126,14 @@ def _forward(self, inference_input):
runtime_gather_output=True, # Inference should always gather the logits
)
- @torch.no_grad()
+ @torch.inference_mode()
def dummy_forward(self):
"""Run a dummy forward pass through the model, with a single token.
Use-case: Used in EP on ranks which do not have any work, but are needed
- for the all-to-all communication."""
+ for the all-to-all communication.
+ Runs under inference_mode so that transformer layers can distinguish this eager
+ dummy_forward from training/validation passes and skip matching on CUDA graphs."""
+
# we use num_dummy_tokens equal to tensor model parallel size
# so that the dummy forward pass will work with sequence parallel
num_dummy_tokens = self.tp_size
@@ -177,21 +144,30 @@ def dummy_forward(self):
(1, num_dummy_tokens), dtype=torch.long, device=torch.cuda.current_device()
)
attention_mask = None
- return self.model(tokens, position_ids, attention_mask)
+ # Always skip MTP during dummy forwards. When num_speculative_tokens > 0
+ # the serial MTP path handles MTP separately (with its own dummy forward).
+ # When num_speculative_tokens == 0 MTP is not needed at all. In both
+ # cases, running MTP here would issue MoE all-to-all collectives that the
+ # real EP ranks do not execute, causing a hang.
+ is_spec_decode = (
+ self.inference_context.is_dynamic_batching() and self.config.mtp_num_layers is not None
+ )
+ return self.model(tokens, position_ids, attention_mask, is_spec_decode=is_spec_decode)
def _get_batch_size_and_seq_len(
self, tokens: torch.Tensor, recv_buffer_seq_len: Optional[int] = None
):
"""
- Returns the batch size and sequence length based on the tokens tensor and recv_buffer_seq_len.
+ Returns the batch size and sequence length based on the tokens tensor and
+ recv_buffer_seq_len.
Args:
tokens (torch.Tensor): The input tensor of shape (batch_size, seq_len).
recv_buffer_seq_len (int, optional): An optional recv buffer sequence length.
Returns:
- tuple: A tuple (batch_size, seq_len), where batch_size is the first dimension of tokens
- and seq_len is either the second dimension or recv_buffer_seq_len.
+ tuple: A tuple (batch_size, seq_len), where batch_size is the first dimension of
+ tokens and seq_len is either the second dimension or recv_buffer_seq_len.
"""
batch_size = tokens.shape[0]
seq_len = recv_buffer_seq_len if recv_buffer_seq_len is not None else tokens.shape[1]
@@ -204,7 +180,7 @@ def _allocate_recv_buffer(self, batch_size, seq_len):
# sequence parallelism. Static batching does not support sequence parallelism
# except for the MoE layers which is handled separately.
seq_len = seq_len // self.tp_size
- recv_size = (seq_len, batch_size, self.inference_wrapper_config.hidden_size)
+ recv_size = (seq_len, batch_size, self.config.hidden_size)
return torch.empty(
recv_size, dtype=self.pipeline_communication_dtype, device=torch.cuda.current_device()
)
@@ -214,10 +190,12 @@ def forward_pass_without_pipeline_parallel(
) -> torch.Tensor:
"""Utility to carry out simple forward pass for TP or no model parallel models
- Runs a very simple forward pass for model. Used in the case of models without any parallelism or only tensor parallelism.
+ Runs a very simple forward pass for model. Used in the case of models without any
+ parallelism or only tensor parallelism.
Args:
- inference_input (Dict[str, Any]): A dict containg the inputs for the gpt model [tokens, position ids, attention mask]
+ inference_input (Dict[str, Any]): A dict containg the inputs for the gpt model
+ [tokens, position ids, attention mask]
Returns:
torch.Tensor: The output logits of shape [batch_size, seq_len, padded_vocab_size]
@@ -228,16 +206,18 @@ def forward_pass_without_pipeline_parallel(
return logits
- def forward_pass_with_pipeline_parallel_small_input_batch(
+ def forward_pass_with_pipeline_parallel(
self, inference_input: Dict[str, Any], recv_buffer_seq_len: Optional[int] = None
) -> torch.Tensor:
- """Utility to carry out forward pass for PP models with very small inputs
+ """Utility to carry out forward pass for PP models
- If a model is pipeline parallel, yet, the input global batch is very small, we compute a foward pass on the entire global batch, rather than splitting it up into micro batches and doing something more complex as in the forward_pass_with_pipeline_parallel_large_input_batch method
+ TODO: Add support for asynchronous microbatches
Args:
- inference_input (Dict[str, Any]): A dict containing the inputs for the gpt model [tokens, position ids, attention mask]
- recv_buffer_seq_len (int): An optional sequence length for the pipeline parallel recv buffer.
+ inference_input (Dict[str, Any]): A dict containing the inputs for the gpt model
+ [tokens, position ids, attention mask]
+ recv_buffer_seq_len (int): An optional sequence length for the pipeline parallel
+ recv buffer.
Returns:
torch.Tensor: The output logits of shape [batch_size, seq_len, padded_vocab_size]
@@ -268,98 +248,8 @@ def forward_pass_with_pipeline_parallel_small_input_batch(
logits = output_tensor
# Explicitly cast logits to expected dtype
- logits = logits.to(self.inference_wrapper_config.params_dtype)
-
- return logits
-
- def forward_pass_with_pipeline_parallel_large_input_batch(
- self, inference_input: Dict[str, Any], recv_buffer_seq_len=None
- ) -> torch.Tensor:
- """Utility to carry out forward pass PP models.
-
- Runs the forward pass for models which are pipeline parallel.
- This is more complex than forward_pass_with_pipeline_parallel_small_input_batch because
- this splits the global batch into small micro batches and runs them through the model.
-
- Args:
- inference_input (Dict[str, Any]): A dict containg the inputs for the gpt model [tokens, position ids, attention mask]
- recv_buffer_seq_len (int): An optional sequence length for the pipeline parallel recv buffer.
-
- Returns:
- torch.Tensor: The output logits of shape [batch_size, seq_len, padded_vocab_size]
- """
- tokens = inference_input["tokens"]
- position_ids = inference_input["position_ids"]
- attention_mask = inference_input["attention_mask"]
- materialize_only_last_token_logits = (
- self.inference_context.materialize_only_last_token_logits
- )
-
- micro_batch_size = max(
- 1,
- self.inference_wrapper_config.inference_batch_times_seqlen_threshold // tokens.size(1),
- )
- batch_size, seq_len = self._get_batch_size_and_seq_len(tokens, recv_buffer_seq_len)
- # Round up to account for the last partial micro batch if present
- num_micro_batches = math.ceil(batch_size / micro_batch_size)
-
- logits = None
- # Preallocate memory for output logits.
- if is_pipeline_last_stage(self.pp_group):
- logits_seq_len = 1 if materialize_only_last_token_logits else seq_len
- logits = torch.empty(
- (batch_size, logits_seq_len, self.inference_wrapper_config.padded_vocab_size),
- dtype=self.pipeline_communication_dtype,
- device=torch.cuda.current_device(),
- )
-
- recv_buffer = None
- if not is_pipeline_first_stage(self.pp_group):
- recv_buffer = self._allocate_recv_buffer(micro_batch_size, seq_len)
- for micro_batch_index in range(num_micro_batches):
- start = micro_batch_index * micro_batch_size
- end = min(start + micro_batch_size, batch_size)
- tokens2use = tokens[start:end, ...]
- position_ids2use = position_ids[start:end, ...]
- current_micro_batch_size = end - start
-
- # Need to change recv buffer shape for the last partial microbatch (if exists)
- if current_micro_batch_size != micro_batch_size:
- recv_buffer = self._allocate_recv_buffer(current_micro_batch_size, seq_len)
-
- if not is_pipeline_first_stage(self.pp_group):
- recv_from_prev_pipeline_rank_(recv_buffer, self.pp_group)
-
- self.model.set_input_tensor(recv_buffer)
-
- output_tensor = self._forward(
- {
- "tokens": tokens2use,
- "position_ids": position_ids2use,
- "attention_mask": attention_mask,
- "inference_context": self.inference_context,
- }
- )
-
- if not is_pipeline_last_stage(self.pp_group):
- send_to_next_pipeline_rank(output_tensor, self.pp_group)
-
- self.inference_context.batch_size_offset += current_micro_batch_size
-
- if is_pipeline_last_stage(self.pp_group):
- assert logits is not None
- logits[start:end, ...] = output_tensor
-
- # Explicitly cast logits to expected dtype
- if is_pipeline_last_stage(self.pp_group):
- assert logits is not None
- logits = logits.to(self.inference_wrapper_config.params_dtype)
-
- # Once done with all micro batches, we reset batch size offset and seq len offset
- self.inference_context.increment_sequence_len_offset(seq_len)
- self.inference_context.reset_batch_size_offset()
+ logits = logits.to(self.config.params_dtype)
- # NOTE: Only returns the logits on the last pipeline stage
return logits
@torch.inference_mode()
@@ -368,14 +258,18 @@ def run_one_forward_step(
) -> torch.Tensor:
"""The forward pass of the model for inference
- Appropriate utility is called for the forward pass depending on the type of model parallelism used
+ Appropriate utility is called for the forward pass depending on the type of model
+ parallelism used
Args:
- inference_input (Dict[str, Any]): A dict containing the inputs for the gpt model [tokens, position ids, attention mask]
- recv_buffer_seq_len (int): An optional sequence length for the pipeline parallel recv buffer.
+ inference_input (Dict[str, Any]): A dict containing the inputs for the gpt model
+ [tokens, position ids, attention mask]
+ recv_buffer_seq_len (int): An optional sequence length for the pipeline parallel
+ recv buffer.
Returns:
- torch.Tensor: The output logits of shape [batch_size, seq_len, padded_vocab_size]. The logits are returned only in the last pipeline stage for PP models.
+ torch.Tensor: The output logits of shape [batch_size, seq_len, padded_vocab_size].
+ The logits are returned only in the last pipeline stage for PP models.
"""
# Check if we are in a PP model
if not (is_pipeline_first_stage(self.pp_group) and is_pipeline_last_stage(self.pp_group)):
@@ -383,19 +277,6 @@ def run_one_forward_step(
current_batch_size, seq_len = self._get_batch_size_and_seq_len(
tokens, recv_buffer_seq_len
)
- # If input batch is large, we need to split into micro batches and run the forward pass
- if (
- current_batch_size * seq_len
- > self.inference_wrapper_config.inference_batch_times_seqlen_threshold
- and self.inference_wrapper_config.inference_batch_times_seqlen_threshold != -1
- ):
- return self.forward_pass_with_pipeline_parallel_large_input_batch(
- inference_input, recv_buffer_seq_len
- )
- else:
- # If input batch is very small we can do a simple forward pass on the entire global batch
- return self.forward_pass_with_pipeline_parallel_small_input_batch(
- inference_input, recv_buffer_seq_len
- )
+ return self.forward_pass_with_pipeline_parallel(inference_input, recv_buffer_seq_len)
else:
return self.forward_pass_without_pipeline_parallel(inference_input)
diff --git a/megatron/core/inference/model_inference_wrappers/gpt/gpt_inference_wrapper.py b/megatron/core/inference/model_inference_wrappers/gpt/gpt_inference_wrapper.py
index ba89fbc2f6c..67cfc1bae48 100644
--- a/megatron/core/inference/model_inference_wrappers/gpt/gpt_inference_wrapper.py
+++ b/megatron/core/inference/model_inference_wrappers/gpt/gpt_inference_wrapper.py
@@ -7,14 +7,12 @@
from megatron.core.inference.model_inference_wrappers.abstract_model_inference_wrapper import (
AbstractModelInferenceWrapper,
)
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
from megatron.core.inference.utils import get_attention_mask
from megatron.core.models.gpt import GPTModel
-from megatron.core.process_groups_config import ProcessGroupCollection
from megatron.core.transformer.enums import AttnBackend
-from megatron.core.utils import get_model_config
+from megatron.core.utils import deprecate_args, get_model_config
+
+DEPRECATED_ARGS = ["inference_wrapper_config", "pg_collection"]
# pylint: disable=line-too-long
@@ -25,22 +23,13 @@ class GPTInferenceWrapper(AbstractModelInferenceWrapper):
Args:
model (GPTModel): The GPT model (MCore or legacy)
- inference_wrapper_config (InferenceWrapperConfig): Has info like hidden size, vocab
- size, etc.
inference_context (BaseInferenceContext): Manages KV cache, and tracks
sequence/token/batch offsets.
- pg_collection (ProcessGroupCollection): Process groups for model communication.
- If not provided, defaults to global parallel state groups.
"""
- def __init__(
- self,
- model: GPTModel,
- inference_wrapper_config: InferenceWrapperConfig,
- inference_context: Optional[BaseInferenceContext] = None,
- pg_collection: Optional[ProcessGroupCollection] = None,
- ):
- super().__init__(model, inference_wrapper_config, inference_context, pg_collection)
+ @deprecate_args(*DEPRECATED_ARGS)
+ def __init__(self, model: GPTModel, inference_context: Optional[BaseInferenceContext] = None):
+ super().__init__(model, inference_context)
def prep_inference_input(self, prompts_tokens: torch.Tensor) -> Dict[str, Any]:
"""Prepares the inference input data.
diff --git a/megatron/core/inference/model_inference_wrappers/inference_wrapper_config.py b/megatron/core/inference/model_inference_wrappers/inference_wrapper_config.py
deleted file mode 100644
index 5d89085add2..00000000000
--- a/megatron/core/inference/model_inference_wrappers/inference_wrapper_config.py
+++ /dev/null
@@ -1,66 +0,0 @@
-# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
-from dataclasses import dataclass
-from typing import Optional
-
-import torch
-
-
-@dataclass
-class InferenceWrapperConfig:
- """Config for the model inference wrapper
-
- NOTE : All the arguments here are obtained from arguments.py file
- """
-
- hidden_size: int
- """Receive happens between the layers during PP with size [seq_len, batch_size, hidden_size]"""
-
- params_dtype: torch.dtype
- """Can be torch.float or torch.half if --fp16 is used, or torch.bfloat16 if --bf16 is used"""
-
- inference_batch_times_seqlen_threshold: int
- """if (batch-size * sequence-length) is smaller than this threshold then we will not pipeline
- the batch."""
-
- padded_vocab_size: int
- """The final padded vocab size (Padded to make it divisible by
- --make-vocab-size-divisible-by value)"""
-
- inference_max_requests: int = 8
- """ Maximum number of requests for inference (prefill & decode). Necessary for CUDA graphs. """
-
- inference_max_seq_length: int = 2560
- """ Maximum sequence length for inference (prefill & decode). Necessary for CUDA graphs. """
-
- fp32_residual_connection: bool = False
- """Move residual connections to fp32. Obtained from arguments.py"""
-
- nccl_all_reduce_for_prefill: bool = False
- """When using symmetric all reduce kernels we keep the default all reduces for nccl.
- This can be more effecient for large prefill sizes"""
-
- fp8: Optional[str] = None
- """If set, enables the use of FP8 precision through Transformer Engine. There are 2 predefined
- choices (1) 'e4m3' uniformly uses e4m3 for all FP8 tensors, (2) 'hybrid' uses e4m3 for all FP8
- activation and weight tensors and e5m2 for all FP8 output activation gradient tensors."""
-
- moe_pad_experts_for_cuda_graph_inference: bool = False
- """Some MoE routers have a D2H sync that will break cuda graphs. If this flag is set the router
- will switch to dropping and padding during decode time which does not have a D2H sync. The
- capacity factor is set to the max that an expert could see during inference so no tokens are
- actually dropped. """
-
- def add_attributes(self, attribute_value_pair: dict):
- """Utility to add more attributes to inference params
-
- Use this method to pass in a custom dictionary to add more configs to the instance created.
- Use as follows:
- c = InferenceWrapperConfig
- c.add_attributes({'precision':'fp32'})
-
- Args:
- attribute_value_pair (dict): A dictionary containing attributes as the key names and
- corresponding values.
- """
- for key, value in attribute_value_pair.items():
- setattr(self, key, value)
diff --git a/megatron/core/inference/model_inference_wrappers/t5/t5_inference_wrapper.py b/megatron/core/inference/model_inference_wrappers/t5/t5_inference_wrapper.py
index 2ae1e2ade6f..c773ab507a3 100644
--- a/megatron/core/inference/model_inference_wrappers/t5/t5_inference_wrapper.py
+++ b/megatron/core/inference/model_inference_wrappers/t5/t5_inference_wrapper.py
@@ -11,9 +11,6 @@
from megatron.core.inference.model_inference_wrappers.abstract_model_inference_wrapper import (
AbstractModelInferenceWrapper,
)
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
from megatron.core.models.T5 import T5Model
from megatron.core.utils import get_attr_wrapped_model
@@ -27,7 +24,6 @@ class T5InferenceWrapper(AbstractModelInferenceWrapper):
Args:
model (T5Model): The T5 model (MCore or legacy)
- inference_wrapper_config (InferenceWrapperConfig): The command line arguments that were passed
inference_context (BaseInferenceContext): Manages KV cache, and tracks
sequence/token/batch offsets.
use_local (bool): Whether the T5 model's transformer impl
@@ -37,11 +33,10 @@ class T5InferenceWrapper(AbstractModelInferenceWrapper):
def __init__(
self,
model: T5Model,
- inference_wrapper_config: InferenceWrapperConfig,
inference_context: Optional[BaseInferenceContext] = None,
use_local: bool = False,
):
- super().__init__(model, inference_wrapper_config, inference_context)
+ super().__init__(model, inference_context)
self.use_local = use_local
def prep_inference_input(
diff --git a/megatron/core/inference/moe/__init__.py b/megatron/core/inference/moe/__init__.py
new file mode 100644
index 00000000000..dbbb24f07bf
--- /dev/null
+++ b/megatron/core/inference/moe/__init__.py
@@ -0,0 +1,56 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+import enum
+
+import torch
+
+from .fused_moe import ActivationType, mcore_fused_moe
+
+
+class InferenceGroupedGemmBackend(enum.Enum):
+ """Resolved backend for grouped GEMM operations during inference."""
+
+ FLASHINFER = "flashinfer"
+ TORCH = "torch"
+ TE = "te"
+
+
+def resolve_inference_grouped_gemm_backend(
+ backend: str, is_cuda_graphed: bool, is_mxfp8: bool = False
+) -> InferenceGroupedGemmBackend:
+ """Resolve the grouped GEMM backend to use for the current iteration.
+
+ Prerequisites are validated at init time in MoELayer; this function
+ simply maps (backend, is_cuda_graphed) to the concrete backend enum.
+
+ Args:
+ backend: One of 'auto', 'torch', 'te'.
+ is_cuda_graphed: Whether this is a CUDA-graphed iteration.
+ is_mxfp8: Whether the model is using MXFP8 quantization (affects auto backend choice).
+ Returns:
+ An InferenceGroupedGemmBackend enum value.
+ """
+ if backend == 'auto':
+ if is_mxfp8:
+ assert hasattr(torch.nn.functional, 'scaled_grouped_mm'), (
+ "Auto backend selection for MXFP8 requires "
+ "torch.nn.functional.scaled_grouped_mm. "
+ "Please install PyTorch 2.10+."
+ )
+ return InferenceGroupedGemmBackend.TORCH
+ if is_cuda_graphed:
+ return InferenceGroupedGemmBackend.FLASHINFER
+ else:
+ if hasattr(torch.nn.functional, 'grouped_mm'):
+ return InferenceGroupedGemmBackend.TORCH
+ else:
+ return InferenceGroupedGemmBackend.TE
+ elif backend == 'torch':
+ return InferenceGroupedGemmBackend.TORCH
+ elif backend == 'te':
+ return InferenceGroupedGemmBackend.TE
+ else:
+ raise ValueError(
+ f"Unknown inference_grouped_gemm_backend: '{backend}'. "
+ "Must be 'auto', 'torch', or 'te'."
+ )
diff --git a/megatron/core/inference/moe/activations.py b/megatron/core/inference/moe/activations.py
new file mode 100644
index 00000000000..169d8499116
--- /dev/null
+++ b/megatron/core/inference/moe/activations.py
@@ -0,0 +1,166 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+"""Padding-aware activation kernels for fused MoE.
+
+These kernels skip padding rows (where permutation_map == -1) to avoid
+wasted computation on aligned-but-empty expert slots.
+"""
+
+from unittest.mock import MagicMock
+
+import torch
+
+from megatron.core.utils import null_decorator
+
+try:
+ import triton
+ import triton.language as tl
+
+ HAVE_TRITON = True
+except ImportError:
+ HAVE_TRITON = False
+
+if not HAVE_TRITON:
+ triton = MagicMock()
+ triton.jit = null_decorator
+ tl = MagicMock()
+
+
+def _ceil_div(a, b):
+ return (a + b - 1) // b
+
+
+@triton.jit
+def _squared_relu_kernel(input_ptr, output_ptr, src_idx_ptr, M, N, BLOCK_N: tl.constexpr):
+ """Squared ReLU that skips padding rows (permutation_map == -1)."""
+ row = tl.program_id(0)
+ if tl.load(src_idx_ptr + row) < 0:
+ return
+ for n in tl.range(0, N, BLOCK_N):
+ o = n + tl.arange(0, BLOCK_N)
+ m = o < N
+ x = tl.load(input_ptr + row * N + o, mask=m).to(tl.float32)
+ r = tl.maximum(x, 0.0)
+ tl.store(output_ptr + row * N + o, (r * r).to(tl.bfloat16), mask=m)
+
+
+def padded_squared_relu(x: torch.Tensor, permutation_map: torch.Tensor) -> torch.Tensor:
+ """Squared ReLU activation that skips padding rows."""
+ M, N = x.shape
+ out = torch.zeros(M, N, dtype=x.dtype, device=x.device)
+ BLOCK_N = min(triton.next_power_of_2(N), 1024)
+ _squared_relu_kernel[(M,)](x, out, permutation_map, M, N, BLOCK_N=BLOCK_N)
+ return out
+
+
+@triton.jit
+def _squared_relu_quantize_kernel(
+ input_ptr,
+ out_fp8_ptr,
+ out_scale_ptr,
+ src_idx_ptr,
+ K,
+ n_col_blocks,
+ skip_padding: tl.constexpr,
+ REAL_GROUPS: tl.constexpr,
+ BLOCK_K: tl.constexpr,
+ BLOCK_GROUPS: tl.constexpr,
+):
+ """Fused squared ReLU + MXFP8 quantize + swizzle in one kernel.
+
+ Grid: (M,) — one program per row.
+ Reads BF16 FC1 output, applies squared ReLU, quantizes to FP8,
+ writes FP8 data + swizzled scales in place.
+ """
+ row = tl.program_id(0)
+ if skip_padding:
+ if tl.load(src_idx_ptr + row) < 0:
+ return
+
+ offs = tl.arange(0, BLOCK_K)
+ mask = offs < K
+
+ # Load and apply squared ReLU
+ x = tl.load(input_ptr + row * K + offs, mask=mask, other=0.0).to(tl.float32)
+ relu = tl.maximum(x, 0.0)
+ activated = relu * relu
+
+ # Per-group-of-32 quantization
+ x_grouped = tl.reshape(activated, [BLOCK_GROUPS, 32])
+ abs_grouped = tl.abs(x_grouped)
+ max_vals = tl.max(abs_grouped, axis=1)
+
+ dequant_scale = max_vals / 448.0
+ dequant_exp = (dequant_scale.to(tl.uint32, bitcast=True) + 0x007FFFFF) & 0x7F800000
+ dequant_rounded = dequant_exp.to(tl.float32, bitcast=True)
+ quant_scale = tl.where(dequant_rounded == 0, 0.0, 1.0 / dequant_rounded)
+
+ quantized = x_grouped * quant_scale[:, None]
+ quantized_flat = tl.reshape(quantized, [BLOCK_K])
+ out_fp8 = quantized_flat.to(tl.float8e4nv)
+
+ # Store FP8 data
+ tl.store(out_fp8_ptr + row * K + offs, out_fp8, mask=mask)
+
+ # Store swizzled scales
+ scale_exp = (dequant_exp >> 23).to(tl.uint8)
+ col_offs = tl.arange(0, BLOCK_GROUPS)
+ col_mask = col_offs < REAL_GROUPS
+
+ macro_row_block = row // 128
+ macro_col_block = col_offs // 4
+ local_row = row % 128
+ local_col = col_offs % 4
+ group = local_row // 32
+ sub_row = local_row % 32
+ tile_idx = macro_row_block * n_col_blocks + macro_col_block
+ swizzled_offs = tile_idx * 512 + sub_row * 16 + group * 4 + local_col
+
+ tl.store(out_scale_ptr + swizzled_offs, scale_exp, mask=col_mask)
+
+
+def squared_relu_and_quantize_mxfp8(
+ x: torch.Tensor, permutation_map: torch.Tensor, skip_padding: bool = True
+):
+ """Fused squared ReLU + MXFP8 quantize + swizzle.
+
+ Reads BF16 FC1 output, applies squared ReLU, quantizes to FP8 with
+ swizzled scales. Single kernel replaces padded_squared_relu + mxfp8_quantize.
+
+ Args:
+ x: [M, K] BF16 FC1 output.
+ permutation_map: [M] int32, original token index or -1 for padding.
+ skip_padding: if True, skip rows where permutation_map == -1.
+
+ Returns:
+ MXFP8Tensor with .data [M, K] float8_e4m3fn and .scale (swizzled e8m0).
+ """
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+
+ M, K = x.shape
+ assert K % 32 == 0
+
+ scale_cols = K // 32
+ n_row_blocks = _ceil_div(M, 128)
+ n_col_blocks = _ceil_div(scale_cols, 4)
+ total_scale_bytes = n_row_blocks * n_col_blocks * 512
+
+ out_fp8 = torch.empty(M, K, dtype=torch.float8_e4m3fn, device=x.device)
+ out_scale = torch.zeros(total_scale_bytes, dtype=torch.uint8, device=x.device)
+
+ BLOCK_K = triton.next_power_of_2(K)
+ BLOCK_GROUPS = BLOCK_K // 32
+
+ _squared_relu_quantize_kernel[(M,)](
+ x,
+ out_fp8,
+ out_scale,
+ permutation_map,
+ K,
+ n_col_blocks,
+ skip_padding,
+ REAL_GROUPS=scale_cols,
+ BLOCK_K=BLOCK_K,
+ BLOCK_GROUPS=BLOCK_GROUPS,
+ )
+
+ return MXFP8Tensor(data=out_fp8, scale=out_scale.view(torch.float8_e8m0fnu), backend="triton")
diff --git a/megatron/core/inference/moe/fused_moe.py b/megatron/core/inference/moe/fused_moe.py
new file mode 100644
index 00000000000..39382eee079
--- /dev/null
+++ b/megatron/core/inference/moe/fused_moe.py
@@ -0,0 +1,204 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+"""Fused MoE: permute -> FC1 -> activation -> FC2 -> unpermute.
+
+Supports BF16 weights with torch.nn.functional.grouped_mm.
+All permutation logic is handled internally — callers invoke a single function.
+"""
+
+from enum import Enum
+from typing import Callable, Optional
+
+import torch
+
+from megatron.core.inference.moe.activations import (
+ padded_squared_relu,
+ squared_relu_and_quantize_mxfp8,
+)
+from megatron.core.inference.moe.pad import pad_to_alignment, unpad_from_alignment
+from megatron.core.inference.moe.permute import (
+ permute_and_quantize_mxfp8,
+ permute_tokens,
+ unpermute_tokens,
+)
+from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+
+try:
+ from torch.nn.functional import grouped_mm
+
+ HAVE_GROUPED_MM = True
+except ImportError:
+ HAVE_GROUPED_MM = False
+
+try:
+ from torch.nn.functional import ScalingType, SwizzleType, scaled_grouped_mm
+
+ HAVE_SCALED_GMM = True
+except ImportError:
+ HAVE_SCALED_GMM = False
+
+
+class ActivationType(Enum):
+ """Activation functions supported by mcore_fused_moe."""
+
+ SQUARED_RELU = "squared_relu"
+
+
+def _bf16_grouped_mm(
+ x_bf16: torch.Tensor, weight: torch.Tensor, offs: torch.Tensor
+) -> torch.Tensor:
+ """BF16 grouped GEMM using torch.nn.functional.grouped_mm."""
+ assert x_bf16.dtype == torch.bfloat16, f"Expected bf16 input, got {x_bf16.dtype}"
+ return grouped_mm(x_bf16, weight.transpose(1, 2), offs=offs)
+
+
+def _mxfp8_grouped_mm(act: MXFP8Tensor, weight: MXFP8Tensor, offs: torch.Tensor) -> torch.Tensor:
+ """MXFP8 scaled_grouped_mm with pre-quantized activations and weights."""
+ return scaled_grouped_mm(
+ act.data,
+ weight.data.transpose(1, 2),
+ act.scale_2d(),
+ ScalingType.BlockWise1x32,
+ weight.scale,
+ ScalingType.BlockWise1x32,
+ swizzle_a=SwizzleType.SWIZZLE_32_4_4,
+ swizzle_b=SwizzleType.SWIZZLE_32_4_4,
+ offs=offs,
+ output_dtype=torch.bfloat16,
+ )
+
+
+def _get_activation_func(activation_type: ActivationType, fused_quant: bool = False) -> Callable:
+ """Resolve ActivationType enum to a concrete kernel.
+
+ If fused_quant=True, returns the fused activation + MXFP8 quantize kernel.
+ """
+ if activation_type == ActivationType.SQUARED_RELU:
+ return squared_relu_and_quantize_mxfp8 if fused_quant else padded_squared_relu
+ else:
+ raise ValueError(f"Unsupported activation type: {activation_type}")
+
+
+def mcore_fused_moe(
+ hidden_states: torch.Tensor,
+ probs: torch.Tensor,
+ fc1_weight,
+ fc2_weight,
+ activation_type: ActivationType,
+ num_local_experts: int,
+ local_expert_start: int,
+ routing_map: Optional[torch.Tensor] = None,
+ tokens_per_expert: Optional[torch.Tensor] = None,
+ skip_permute: bool = False,
+ disable_fused_quant_kernels: bool = False,
+) -> torch.Tensor:
+ """Fused MoE: [permute ->] pad -> FC1 -> activation -> FC2 -> unpad [-> unpermute].
+
+ Two modes:
+ - skip_permute=False (default): tokens are unpermuted. Requires routing_map.
+ Performs full permute -> compute -> unpermute.
+ - skip_permute=True: tokens are already permuted by the dispatcher. Requires
+ tokens_per_expert. Pads to alignment, computes, then unpads. Probs are
+ applied during unpad.
+
+ Unless disable_fused_quant_kernels=True, when weights are MXFP8, uses fused
+ kernels that combine permute/activation with MXFP8 quantization into single
+ kernel launches.
+
+ Args:
+ hidden_states: [num_tokens, hidden_size] BF16 input.
+ probs: routing probabilities. Shape is [num_tokens, topk] when
+ skip_permute=False, or [num_tokens] (already gathered) when
+ skip_permute=True.
+ fc1_weight: stacked weight for FC1 (torch.Tensor for BF16, MXFP8Tensor for MXFP8).
+ fc2_weight: stacked weight for FC2 (same type as fc1_weight).
+ activation_type: ActivationType enum (SQUARED_RELU).
+ num_local_experts: number of experts on this rank.
+ local_expert_start: first global expert index on this rank.
+ routing_map: [num_tokens, topk] int expert assignments. Required when skip_permute=False.
+ tokens_per_expert: [num_local_experts] int32 token counts. Required when skip_permute=True.
+ skip_permute: if True, skip permute/unpermute (tokens already in expert order).
+ disable_fused_quant_kernels: if True, disable fused permute+quantize and
+ activation+quantize kernels for MXFP8, using separate launches instead.
+ Useful for debugging. Ignored when weights are BF16.
+
+ Returns:
+ [num_tokens, hidden_size] BF16 output.
+ """
+ assert (
+ hidden_states.dtype == torch.bfloat16
+ ), f"mcore_fused_moe requires bf16 input, got {hidden_states.dtype}"
+
+ num_tokens = hidden_states.shape[0]
+ use_mxfp8 = isinstance(fc1_weight, MXFP8Tensor)
+ # Fused quant kernels only apply to MXFP8 path
+ use_fused_quant = use_mxfp8 and not disable_fused_quant_kernels
+
+ if use_mxfp8:
+ assert (
+ HAVE_SCALED_GMM
+ ), "torch.nn.functional.scaled_grouped_mm not available. Install PyTorch 2.10+."
+ mm_fn = _mxfp8_grouped_mm
+ # scaled_grouped_mm requires each expert's token count aligned to 32,
+ # but swizzled MXFP8 scales require alignment to 128. Use 128 to
+ # satisfy both constraints.
+ expert_alignment = 128
+ else:
+ assert (
+ HAVE_GROUPED_MM
+ ), "torch.nn.functional.grouped_mm not available. Install PyTorch 2.10+."
+ mm_fn = _bf16_grouped_mm
+ expert_alignment = 16
+
+ activation_func = _get_activation_func(activation_type, fused_quant=use_fused_quant)
+
+ # --- Pre-processing: permute or pad ---
+ if skip_permute:
+ assert tokens_per_expert is not None, "tokens_per_expert is required when skip_permute=True"
+ tokens_per_expert = tokens_per_expert.cuda().int()
+ assert routing_map is None, "routing_map must be None when skip_permute=True"
+ hidden_states, permutation_map, offs = pad_to_alignment(
+ hidden_states, tokens_per_expert, expert_alignment
+ )
+ permuted_probs = None
+
+ else:
+ assert routing_map is not None, "routing_map is required when skip_permute=False"
+ if use_fused_quant:
+ # Fused permute + MXFP8 quantize: single kernel produces MXFP8Tensor
+ hidden_states, permuted_probs, permutation_map, offs = permute_and_quantize_mxfp8(
+ hidden_states,
+ probs,
+ routing_map,
+ local_expert_start,
+ num_local_experts,
+ alignment=expert_alignment,
+ )
+ else:
+ hidden_states, permuted_probs, permutation_map, offs = permute_tokens(
+ hidden_states,
+ probs,
+ routing_map,
+ local_expert_start,
+ num_local_experts,
+ alignment=expert_alignment,
+ )
+
+ # --- FC1 -> activation -> FC2 ---
+ # Quantize if MXFP8 path and hidden_states not already quantized (fused permute+quant
+ # produces MXFP8Tensor directly; skip_permute path always needs separate quant).
+ needs_quant = use_mxfp8 and not isinstance(hidden_states, MXFP8Tensor)
+ if needs_quant:
+ hidden_states = MXFP8Tensor.from_bf16(hidden_states, backend="triton")
+ fc1_output = mm_fn(hidden_states, fc1_weight, offs)
+
+ activation_out = activation_func(fc1_output, permutation_map)
+ # Fused activation+quant returns MXFP8Tensor; otherwise quantize separately.
+ if use_mxfp8 and not isinstance(activation_out, MXFP8Tensor):
+ activation_out = MXFP8Tensor.from_bf16(activation_out, backend="triton")
+ fc2_output = mm_fn(activation_out, fc2_weight, offs)
+ # --- Post-processing: unpermute or unpad ---
+ if skip_permute:
+ probs_1d = probs.squeeze(-1) if probs.dim() > 1 else probs
+ return unpad_from_alignment(fc2_output, permutation_map, num_tokens, probs=probs_1d)
+ else:
+ return unpermute_tokens(fc2_output, permuted_probs, permutation_map, num_tokens)
diff --git a/megatron/core/inference/moe/pad.py b/megatron/core/inference/moe/pad.py
new file mode 100644
index 00000000000..656953b691c
--- /dev/null
+++ b/megatron/core/inference/moe/pad.py
@@ -0,0 +1,201 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+"""Pad / unpad utilities for already-permuted expert tokens.
+
+When the token dispatcher has already permuted tokens into expert-grouped
+order, these functions insert/remove alignment padding so that each expert's
+token block satisfies the alignment requirements of grouped_mm /
+scaled_grouped_mm.
+"""
+
+from unittest.mock import MagicMock
+
+import torch
+from packaging import version
+
+from megatron.core.utils import null_decorator
+
+try:
+ import triton
+ import triton.language as tl
+
+ if version.parse(triton.__version__) < version.parse("3.4.0") and not torch.cuda.is_available():
+ HAVE_TRITON = False
+ else:
+ HAVE_TRITON = tl.constexpr(version.parse(triton.__version__) >= version.parse("2.0.0"))
+except ImportError:
+ HAVE_TRITON = False
+
+if not HAVE_TRITON:
+ triton = MagicMock()
+ triton.jit = null_decorator
+ tl = MagicMock()
+
+from megatron.core.inference.moe.permute import compute_expert_offsets
+
+
+@triton.jit
+def _pad_tokens_kernel(
+ src_ptr,
+ dst_ptr,
+ perm_map_ptr,
+ tpe_ptr, # tokens_per_expert [num_experts]
+ hidden_dim,
+ num_experts: tl.constexpr,
+ alignment: tl.constexpr,
+ BLOCK_H: tl.constexpr,
+):
+ """Copy one input row into the padded output buffer.
+
+ Computes unpadded and padded cumulative offsets inline from
+ tokens_per_expert, avoiding a separate cumsum kernel launch.
+ """
+ row = tl.program_id(0)
+
+ # Walk tokens_per_expert to find which expert this row belongs to
+ # and compute both unpadded and padded start offsets on the fly.
+ unpadded_start = tl.zeros([], dtype=tl.int32)
+ padded_start = tl.zeros([], dtype=tl.int32)
+ expert_id = -1
+ for e in tl.static_range(0, num_experts):
+ count = tl.load(tpe_ptr + e).to(tl.int32)
+ if expert_id < 0 and row < unpadded_start + count:
+ expert_id = e
+ if expert_id < 0:
+ unpadded_start += count
+ aligned = tl.where(
+ count > 0,
+ ((count + alignment - 1) // alignment) * alignment,
+ tl.zeros([], dtype=tl.int32),
+ )
+ padded_start += aligned
+
+ if expert_id < 0:
+ return
+
+ local_idx = row - unpadded_start
+ dst_row = padded_start + local_idx
+
+ # Write permutation_map: padded row → original unpadded row
+ tl.store(perm_map_ptr + dst_row, row)
+
+ # Copy hidden state
+ for h in tl.range(0, hidden_dim, BLOCK_H):
+ o = h + tl.arange(0, BLOCK_H)
+ m = o < hidden_dim
+ tl.store(
+ dst_ptr + dst_row * hidden_dim + o,
+ tl.load(src_ptr + row * hidden_dim + o, mask=m),
+ mask=m,
+ )
+
+
+def pad_to_alignment(
+ hidden_states: torch.Tensor, tokens_per_expert: torch.Tensor, alignment: int
+) -> tuple:
+ """Pad already-permuted tokens so each expert's block is aligned.
+
+ Args:
+ hidden_states: [total_tokens, hidden_size] already permuted by dispatcher.
+ tokens_per_expert: [num_local_experts] int32 token counts.
+ alignment: per-expert alignment.
+
+ Returns:
+ (padded_hidden, permutation_map, inclusive_offsets)
+ - padded_hidden: [padded_total, hidden_size]
+ - permutation_map: [padded_total] int32, original row index or -1 for padding.
+ - inclusive_offsets: [num_local_experts] int32 cumulative aligned offsets for grouped_mm.
+ """
+ num_experts = tokens_per_expert.shape[0]
+ total_tokens = hidden_states.shape[0]
+ hidden_dim = hidden_states.shape[1]
+
+ # We still need padded_inc for the return value (used as offs by grouped_mm)
+ _, padded_inc = compute_expert_offsets(tokens_per_expert, alignment=alignment)
+ padded_total = int(padded_inc[-1].item())
+
+ padded_hidden = torch.zeros(
+ padded_total, hidden_dim, dtype=hidden_states.dtype, device=hidden_states.device
+ )
+ permutation_map = torch.full(
+ (padded_total,), -1, dtype=torch.int32, device=hidden_states.device
+ )
+
+ if total_tokens > 0:
+ BLOCK_H = min(triton.next_power_of_2(hidden_dim), 1024)
+ _pad_tokens_kernel[(total_tokens,)](
+ hidden_states,
+ padded_hidden,
+ permutation_map,
+ tokens_per_expert,
+ hidden_dim,
+ num_experts,
+ alignment,
+ BLOCK_H=BLOCK_H,
+ )
+
+ return padded_hidden, permutation_map, padded_inc
+
+
+@triton.jit
+def _unpad_tokens_kernel(
+ src_ptr,
+ dst_ptr,
+ perm_map_ptr,
+ probs_ptr,
+ hidden_dim,
+ has_probs: tl.constexpr,
+ BLOCK_H: tl.constexpr,
+):
+ """Copy one real (non-padding) row from padded to unpadded layout.
+
+ Optionally multiplies each row by its routing probability.
+ """
+ row = tl.program_id(0)
+ dst_row = tl.load(perm_map_ptr + row)
+ if dst_row < 0:
+ return
+ if has_probs:
+ prob = tl.load(probs_ptr + dst_row)
+ for h in tl.range(0, hidden_dim, BLOCK_H):
+ o = h + tl.arange(0, BLOCK_H)
+ m = o < hidden_dim
+ v = tl.load(src_ptr + row * hidden_dim + o, mask=m)
+ if has_probs:
+ v = v * prob
+ tl.store(dst_ptr + dst_row * hidden_dim + o, v, mask=m)
+
+
+def unpad_from_alignment(
+ padded_output: torch.Tensor,
+ permutation_map: torch.Tensor,
+ original_size: int,
+ probs: torch.Tensor = None,
+) -> torch.Tensor:
+ """Remove alignment padding, scattering results back to original positions.
+
+ Args:
+ padded_output: [padded_total, hidden_size] output from expert computation.
+ permutation_map: [padded_total] int32, original row index or -1 for padding.
+ original_size: number of rows in the unpadded output.
+ probs: optional [original_size] routing probabilities to multiply during unpad.
+
+ Returns:
+ [original_size, hidden_size] unpadded output.
+ """
+ hidden_dim = padded_output.shape[1]
+ output = torch.zeros(
+ original_size, hidden_dim, dtype=padded_output.dtype, device=padded_output.device
+ )
+ has_probs = probs is not None
+ if padded_output.shape[0] > 0:
+ BLOCK_H = min(triton.next_power_of_2(hidden_dim), 1024)
+ _unpad_tokens_kernel[(padded_output.shape[0],)](
+ padded_output,
+ output,
+ permutation_map,
+ probs if has_probs else padded_output, # dummy pointer when no probs
+ hidden_dim,
+ has_probs,
+ BLOCK_H=BLOCK_H,
+ )
+ return output
diff --git a/megatron/core/inference/moe/permute.py b/megatron/core/inference/moe/permute.py
new file mode 100644
index 00000000000..b14d0b3dbd0
--- /dev/null
+++ b/megatron/core/inference/moe/permute.py
@@ -0,0 +1,458 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+"""Triton kernels for token permutation and unpermutation in fused MoE.
+
+Includes:
+- Token counting per expert
+- Expert offset computation (aligned prefix sums)
+- Permute tokens into expert-grouped order
+- Unpermute expert outputs back to original token order
+"""
+
+from unittest.mock import MagicMock
+
+import torch
+
+from megatron.core.utils import null_decorator
+
+try:
+ import triton
+ import triton.language as tl
+
+ HAVE_TRITON = True
+except ImportError:
+ HAVE_TRITON = False
+
+if not HAVE_TRITON:
+ triton = MagicMock()
+ triton.jit = null_decorator
+ tl = MagicMock()
+
+
+def _ceil_div(a, b):
+ return (a + b - 1) // b
+
+
+@triton.jit
+def _count_local_tokens_kernel(
+ routing_map_ptr, # [num_tokens * topk] flattened expert assignments
+ tokens_per_expert_ptr, # [num_local_experts] output counters (zeroed by caller)
+ total_pairs, # num_tokens * topk — total (token, topk) pairs
+ local_expert_start, # first global expert index owned by this rank
+ num_local_experts: tl.constexpr, # number of experts on this rank
+ BLOCK_SIZE: tl.constexpr, # number of pairs processed per program
+):
+ """Count tokens routed to experts on this rank, ignoring tokens routed elsewhere.
+
+ Each program processes BLOCK_SIZE (token, topk) pairs. Tokens assigned to
+ experts outside [local_expert_start, local_expert_start + num_local_experts)
+ are silently skipped.
+ """
+ pid = tl.program_id(0)
+ offsets = pid * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE)
+ mask = offsets < total_pairs
+ expert_ids = tl.load(routing_map_ptr + offsets, mask=mask, other=-1)
+ # Map global expert IDs to local indices; non-local experts become negative
+ local_ids = expert_ids - local_expert_start
+ is_local = (local_ids >= 0) & (local_ids < num_local_experts) & mask
+ tl.atomic_add(tokens_per_expert_ptr + local_ids, 1, mask=is_local)
+
+
+def compute_local_tokens_per_expert(
+ routing_map: torch.Tensor, local_expert_start: int, num_local_experts: int
+) -> torch.Tensor:
+ """Count tokens routed to each local expert."""
+ total_pairs = routing_map.numel()
+ tokens_per_expert = torch.zeros(num_local_experts, dtype=torch.int32, device=routing_map.device)
+ BLOCK = 256
+ _count_local_tokens_kernel[(_ceil_div(total_pairs, BLOCK),)](
+ routing_map,
+ tokens_per_expert,
+ total_pairs,
+ local_expert_start,
+ num_local_experts,
+ BLOCK_SIZE=BLOCK,
+ )
+ return tokens_per_expert
+
+
+@triton.jit
+def _prefix_sum_kernel(
+ tokens_per_expert_ptr, # [num_local_experts] raw token counts
+ exclusive_offsets_ptr, # [num_local_experts] output: exclusive prefix sum of aligned counts
+ inclusive_offsets_ptr, # [num_local_experts] output: inclusive prefix sum of aligned counts
+ num_local_experts, # number of experts on this rank
+ alignment: tl.constexpr, # per-expert alignment (counts rounded up to this multiple)
+ BLOCK_SIZE: tl.constexpr, # next_power_of_2(num_local_experts) for tl.cumsum
+):
+ """Exclusive and inclusive prefix sums of aligned token counts.
+
+ Each expert's token count is rounded up to the nearest multiple of
+ `alignment` (experts with 0 tokens stay at 0). The inclusive offsets
+ are used as `offs` by grouped_mm / scaled_grouped_mm.
+ """
+ r = tl.arange(0, BLOCK_SIZE)
+ mask = r < num_local_experts
+ h = tl.load(tokens_per_expert_ptr + r, mask=mask, other=0)
+ # Round up non-zero counts to alignment boundary
+ if alignment > 1:
+ h = tl.where(h > 0, ((h + alignment - 1) // alignment) * alignment, h)
+ inc = tl.cumsum(h, axis=0)
+ tl.store(exclusive_offsets_ptr + r, inc - h, mask=mask)
+ tl.store(inclusive_offsets_ptr + r, inc, mask=mask)
+
+
+def compute_expert_offsets(tokens_per_expert: torch.Tensor, alignment: int = 1) -> tuple:
+ """Compute exclusive and inclusive prefix sums of aligned token counts."""
+ n = tokens_per_expert.shape[0]
+ exclusive_cumsum = torch.empty_like(tokens_per_expert)
+ inclusive_cumsum = torch.empty_like(tokens_per_expert)
+ _prefix_sum_kernel[(1,)](
+ tokens_per_expert,
+ exclusive_cumsum,
+ inclusive_cumsum,
+ n,
+ alignment,
+ BLOCK_SIZE=triton.next_power_of_2(n),
+ )
+ return exclusive_cumsum, inclusive_cumsum
+
+
+@triton.jit
+def _permute_tokens_kernel(
+ hidden_ptr, # [num_tokens, hidden_dim] input hidden states
+ probs_ptr, # [num_tokens, topk] routing probabilities
+ routing_map_ptr, # [num_tokens, topk] expert assignments (global IDs)
+ out_hidden_ptr, # [output_size, hidden_dim] output: permuted hidden states
+ out_probs_ptr, # [output_size] output: permuted probabilities
+ out_src_idx_ptr, # [output_size] output: permutation_map (original token index, -1 for padding)
+ counters_ptr, # [num_local_experts] exclusive offsets,
+ # atomically incremented to assign positions
+ num_tokens, # number of input tokens
+ hidden_dim, # hidden dimension
+ topk: tl.constexpr, # number of expert choices per token
+ local_expert_start, # first global expert index on this rank
+ num_local_experts: tl.constexpr, # number of experts on this rank
+ BLOCK_H: tl.constexpr, # tile size for copying hidden_dim
+):
+ """Permute tokens into expert-grouped order.
+
+ Grid: one program per (token, topk) pair. Each program looks up the assigned
+ expert, skips non-local experts, then atomically claims a position within
+ that expert's block and copies the hidden state + prob + source index.
+ """
+ # Each program handles one (token, topk) pair
+ pair = tl.program_id(0)
+ tok = pair // topk
+ k = pair % topk
+ if tok >= num_tokens:
+ return
+ eid = tl.load(routing_map_ptr + tok * topk + k)
+ lid = eid - local_expert_start
+ # Skip tokens routed to non-local experts
+ if lid < 0 or lid >= num_local_experts:
+ return
+ # Atomically claim a position within this expert's aligned block
+ pos = tl.atomic_add(counters_ptr + lid, 1)
+ # Copy hidden state row
+ for h in tl.range(0, hidden_dim, BLOCK_H):
+ o = h + tl.arange(0, BLOCK_H)
+ m = o < hidden_dim
+ tl.store(
+ out_hidden_ptr + pos * hidden_dim + o,
+ tl.load(hidden_ptr + tok * hidden_dim + o, mask=m),
+ mask=m,
+ )
+ tl.store(out_probs_ptr + pos, tl.load(probs_ptr + tok * topk + k))
+ # Record source token index for unpermute
+ tl.store(out_src_idx_ptr + pos, tok)
+
+
+def permute_tokens(
+ hidden_states: torch.Tensor,
+ probs: torch.Tensor,
+ routing_map: torch.Tensor,
+ local_expert_start: int,
+ num_local_experts: int,
+ alignment: int = 1,
+) -> tuple:
+ """Permute tokens into expert-grouped order.
+
+ Computes token counts, aligned expert offsets, output sizing, and
+ permutation in a single call.
+
+ Args:
+ hidden_states: [num_tokens, hidden_size] input.
+ probs: [num_tokens, topk] routing probabilities.
+ routing_map: [num_tokens, topk] expert assignments.
+ local_expert_start: first global expert index on this rank.
+ num_local_experts: number of experts on this rank.
+ alignment: per-expert token alignment (default 1).
+
+ Returns:
+ (permuted_hidden, permuted_probs, permutation_map, inclusive_offsets)
+ - permuted_hidden: [output_size, hidden_size]
+ - permuted_probs: [output_size]
+ - permutation_map: [output_size] int32, maps each permuted row back to
+ its original token index. Used by unpermute_tokens to scatter expert
+ outputs back and by activation kernels to skip padding rows (-1).
+ - inclusive_offsets: [num_local_experts] int32 cumulative offsets for grouped_mm
+ """
+ num_tokens, hidden_dim = hidden_states.shape
+ topk = probs.shape[1]
+
+ # Count how many (token, topk) pairs are routed to each local expert.
+ # Non-local experts are ignored. Result is [num_local_experts] int32.
+ tokens_per_expert = compute_local_tokens_per_expert(
+ routing_map, local_expert_start, num_local_experts
+ )
+
+ # exclusive_expert_offsets[i] = start of expert i's block in the padded output.
+ # Used as the initial counter for atomic position assignment in the permute kernel.
+ # inclusive_expert_offsets[i] = end of expert i's block (= start of expert i+1).
+ # Passed as `offs` to grouped_mm / scaled_grouped_mm to delimit expert boundaries.
+ exclusive_expert_offsets, inclusive_expert_offsets = compute_expert_offsets(
+ tokens_per_expert, alignment=alignment
+ )
+ output_size = num_tokens * min(topk, num_local_experts) + alignment * num_local_experts
+
+ permuted_hidden = torch.empty(
+ output_size, hidden_dim, dtype=hidden_states.dtype, device=hidden_states.device
+ )
+ permuted_probs = torch.empty(output_size, dtype=probs.dtype, device=probs.device)
+ permutation_map = torch.full((output_size,), -1, dtype=torch.int32, device=probs.device)
+ BLOCK_H = min(triton.next_power_of_2(hidden_dim), 1024)
+ _permute_tokens_kernel[(num_tokens * topk,)](
+ hidden_states,
+ probs,
+ routing_map,
+ permuted_hidden,
+ permuted_probs,
+ permutation_map,
+ exclusive_expert_offsets,
+ num_tokens,
+ hidden_dim,
+ topk,
+ local_expert_start,
+ num_local_experts,
+ BLOCK_H=BLOCK_H,
+ )
+ return permuted_hidden, permuted_probs, permutation_map, inclusive_expert_offsets
+
+
+@triton.jit
+def _unpermute_tokens_kernel(
+ expert_out_ptr, # [output_size, hidden_dim] expert outputs in permuted order
+ probs_ptr, # [output_size] fp32 routing probabilities (permuted)
+ src_idx_ptr, # [output_size] permutation_map: original token index, or -1 for padding
+ output_ptr, # [num_tokens, hidden_dim] fp32 output buffer (zeroed by caller)
+ hidden_dim, # hidden dimension
+ BLOCK_H: tl.constexpr, # tile size for processing hidden_dim
+):
+ """Scatter weighted expert outputs back to original token positions.
+
+ Grid: one program per row of expert_out. Padding rows (src_idx == -1) are
+ skipped. Multiple topk selections for the same token are accumulated via
+ atomic adds. All arithmetic is in fp32 to avoid precision loss.
+ """
+ row = tl.program_id(0)
+ source_idx = tl.load(src_idx_ptr + row)
+ # Skip padding rows
+ if source_idx < 0:
+ return
+ prob = tl.load(probs_ptr + row) # fp32
+ for h in tl.range(0, hidden_dim, BLOCK_H):
+ offsets = h + tl.arange(0, BLOCK_H)
+ m = offsets < hidden_dim
+ # Upcast bf16 expert output to fp32 before multiply + accumulate
+ v = tl.load(expert_out_ptr + row * hidden_dim + offsets, mask=m).to(tl.float32)
+ tl.atomic_add(output_ptr + source_idx * hidden_dim + offsets, v * prob, mask=m)
+
+
+def unpermute_tokens(
+ expert_output: torch.Tensor,
+ permuted_probs: torch.Tensor,
+ permutation_map: torch.Tensor,
+ num_tokens: int,
+) -> torch.Tensor:
+ """Unpermute expert outputs back to original token order.
+
+ Accumulates in fp32 to avoid precision loss from multiple topk atomic adds.
+ Returns fp32 output.
+ """
+ assert (
+ permuted_probs.dtype == torch.float32
+ ), f"permuted_probs must be fp32, got {permuted_probs.dtype}"
+ output_size, hidden_dim = expert_output.shape
+ output = torch.zeros(num_tokens, hidden_dim, dtype=torch.float32, device=expert_output.device)
+ BLOCK_H = min(triton.next_power_of_2(hidden_dim), 1024)
+ _unpermute_tokens_kernel[(output_size,)](
+ expert_output, permuted_probs, permutation_map, output, hidden_dim, BLOCK_H=BLOCK_H
+ )
+ return output
+
+
+@triton.jit
+def _permute_quantize_mxfp8_kernel(
+ hidden_ptr,
+ probs_ptr,
+ routing_map_ptr,
+ out_fp8_ptr,
+ out_scale_ptr,
+ out_probs_ptr,
+ out_src_idx_ptr,
+ counters_ptr,
+ num_tokens,
+ K,
+ n_col_blocks,
+ topk: tl.constexpr,
+ local_expert_start,
+ num_local_experts: tl.constexpr,
+ REAL_GROUPS: tl.constexpr,
+ BLOCK_K: tl.constexpr,
+ BLOCK_GROUPS: tl.constexpr,
+):
+ """Fused permute + MXFP8 quantize + swizzle in one kernel.
+
+ Grid: (num_tokens * topk,) — one program per (token, k) pair.
+ Reads BF16 from source token, quantizes to FP8 e4m3, writes FP8 data +
+ swizzled e8m0 scales to the permuted write position.
+ """
+ pair = tl.program_id(0)
+ tok = pair // topk
+ k = pair % topk
+ if tok >= num_tokens:
+ return
+ eid = tl.load(routing_map_ptr + tok * topk + k)
+ lid = eid - local_expert_start
+ if lid < 0 or lid >= num_local_experts:
+ return
+
+ pos = tl.atomic_add(counters_ptr + lid, 1)
+
+ # Load full row from source token
+ offs = tl.arange(0, BLOCK_K)
+ mask = offs < K
+ x = tl.load(hidden_ptr + tok * K + offs, mask=mask, other=0.0).to(tl.float32)
+
+ # Per-group-of-32 quantization
+ x_grouped = tl.reshape(x, [BLOCK_GROUPS, 32])
+ abs_grouped = tl.abs(x_grouped)
+ max_vals = tl.max(abs_grouped, axis=1)
+
+ dequant_scale = max_vals / 448.0
+ dequant_exp = (dequant_scale.to(tl.uint32, bitcast=True) + 0x007FFFFF) & 0x7F800000
+ dequant_rounded = dequant_exp.to(tl.float32, bitcast=True)
+ quant_scale = tl.where(dequant_rounded == 0, 0.0, 1.0 / dequant_rounded)
+
+ quantized = x_grouped * quant_scale[:, None]
+ quantized_flat = tl.reshape(quantized, [BLOCK_K])
+ out_fp8 = quantized_flat.to(tl.float8e4nv)
+
+ # Store FP8 data at permuted position
+ tl.store(out_fp8_ptr + pos * K + offs, out_fp8, mask=mask)
+
+ # Store swizzled scales at permuted position
+ scale_exp = (dequant_exp >> 23).to(tl.uint8)
+ col_offs = tl.arange(0, BLOCK_GROUPS)
+ col_mask = col_offs < REAL_GROUPS
+
+ macro_row_block = pos // 128
+ macro_col_block = col_offs // 4
+ local_row = pos % 128
+ local_col = col_offs % 4
+ group = local_row // 32
+ sub_row = local_row % 32
+ tile_idx = macro_row_block * n_col_blocks + macro_col_block
+ swizzled_offs = tile_idx * 512 + sub_row * 16 + group * 4 + local_col
+
+ tl.store(out_scale_ptr + swizzled_offs, scale_exp, mask=col_mask)
+
+ # Store prob and source index
+ tl.store(out_probs_ptr + pos, tl.load(probs_ptr + tok * topk + k))
+ tl.store(out_src_idx_ptr + pos, tok)
+
+
+def permute_and_quantize_mxfp8(
+ hidden_states: torch.Tensor,
+ probs: torch.Tensor,
+ routing_map: torch.Tensor,
+ local_expert_start: int,
+ num_local_experts: int,
+ alignment: int = 128,
+) -> tuple:
+ """Fused permute + MXFP8 quantize + swizzle.
+
+ Self-contained API matching permute_tokens: computes token counts, aligned
+ expert offsets, output sizing, permutation, and MXFP8 quantization in a
+ single kernel launch.
+
+ Args:
+ hidden_states: [num_tokens, hidden_size] BF16 input.
+ probs: [num_tokens, topk] routing probabilities.
+ routing_map: [num_tokens, topk] expert assignments.
+ local_expert_start: first global expert index on this rank.
+ num_local_experts: number of experts on this rank.
+ alignment: per-expert token alignment (default 128, required for MXFP8 swizzle).
+
+ Returns:
+ (permuted_mxfp8, permuted_probs, permutation_map, inclusive_offsets)
+ - permuted_mxfp8: MXFP8Tensor with .data [output_size, K] and .scale (swizzled)
+ - permuted_probs: [output_size] routing probs
+ - permutation_map: [output_size] int32, original token index or -1 for padding
+ - inclusive_offsets: [num_local_experts] int32 cumulative offsets for scaled_grouped_mm
+ """
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+
+ num_tokens, K = hidden_states.shape
+ topk = probs.shape[1]
+ assert K % 32 == 0
+
+ # Count how many (token, topk) pairs are routed to each local expert.
+ tokens_per_expert = compute_local_tokens_per_expert(
+ routing_map, local_expert_start, num_local_experts
+ )
+
+ # exclusive_expert_offsets[i] = start of expert i's block in the padded output.
+ # inclusive_expert_offsets[i] = end of expert i's block (= start of expert i+1).
+ exclusive_expert_offsets, inclusive_expert_offsets = compute_expert_offsets(
+ tokens_per_expert, alignment=alignment
+ )
+ output_size = num_tokens * min(topk, num_local_experts) + alignment * num_local_experts
+
+ scale_cols = K // 32
+ n_row_blocks = _ceil_div(output_size, 128)
+ n_col_blocks = _ceil_div(scale_cols, 4)
+ total_scale_bytes = n_row_blocks * n_col_blocks * 512
+
+ out_fp8 = torch.empty(output_size, K, dtype=torch.float8_e4m3fn, device=hidden_states.device)
+ out_scale = torch.zeros(total_scale_bytes, dtype=torch.uint8, device=hidden_states.device)
+ permuted_probs = torch.empty(output_size, dtype=probs.dtype, device=probs.device)
+ permutation_map = torch.full((output_size,), -1, dtype=torch.int32, device=probs.device)
+
+ BLOCK_K = triton.next_power_of_2(K)
+ BLOCK_GROUPS = BLOCK_K // 32
+
+ _permute_quantize_mxfp8_kernel[(num_tokens * topk,)](
+ hidden_states,
+ probs,
+ routing_map,
+ out_fp8,
+ out_scale,
+ permuted_probs,
+ permutation_map,
+ exclusive_expert_offsets,
+ num_tokens,
+ K,
+ n_col_blocks,
+ topk,
+ local_expert_start,
+ num_local_experts,
+ REAL_GROUPS=scale_cols,
+ BLOCK_K=BLOCK_K,
+ BLOCK_GROUPS=BLOCK_GROUPS,
+ )
+
+ permuted_mxfp8 = MXFP8Tensor(
+ data=out_fp8, scale=out_scale.view(torch.float8_e8m0fnu), backend="triton"
+ )
+ return permuted_mxfp8, permuted_probs, permutation_map, inclusive_expert_offsets
diff --git a/megatron/core/inference/quantization/__init__.py b/megatron/core/inference/quantization/__init__.py
new file mode 100644
index 00000000000..26496bfed70
--- /dev/null
+++ b/megatron/core/inference/quantization/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
diff --git a/megatron/core/inference/quantization/mxfp8_quantize.py b/megatron/core/inference/quantization/mxfp8_quantize.py
new file mode 100644
index 00000000000..07fea22fa45
--- /dev/null
+++ b/megatron/core/inference/quantization/mxfp8_quantize.py
@@ -0,0 +1,198 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""Standalone MXFP8 quantization kernel with fused scale swizzle.
+
+One block per token. Quantizes BF16 → FP8 e4m3 and writes scales directly
+in cuBLAS 2D blocked (swizzled) layout. No FP4, no triton_kernels dependency.
+
+Usage:
+ from megatron.core.inference.quantization.mxfp8_quantize import mxfp8_quantize
+ data, swizzled_scales, total_scale_bytes = mxfp8_quantize(x_bf16)
+ # data: [M, K] float8_e4m3fn
+ # swizzled_scales: 1D uint8 in cuBLAS blocked layout
+"""
+
+import torch
+
+try:
+ import triton
+ import triton.language as tl
+
+ HAVE_TRITON = True
+except ImportError:
+ from unittest.mock import MagicMock
+
+ from megatron.core.utils import null_decorator
+
+ triton = MagicMock()
+ triton.jit = null_decorator
+ tl = MagicMock()
+ HAVE_TRITON = False
+
+
+def _ceil_div(a, b):
+ return (a + b - 1) // b
+
+
+@triton.jit
+def _mxfp8_quant_swizzle_kernel(
+ out_ptr, # [M, K] output buffer for float8_e4m3fn quantized data
+ scale_ptr, # 1D output buffer for swizzled uint8 scales (e8m0 exponents)
+ src_ptr, # [M, K] input tensor in bf16/fp16/fp32
+ K, # number of columns in the input (must be divisible by 32)
+ n_col_blocks, # ceil(K/32 / 4) — number of macro-tile columns in the swizzle layout
+ REAL_GROUPS: tl.constexpr, # actual number of scale groups per row (K // 32)
+ BLOCK_K: tl.constexpr, # next_power_of_2(K) — padded column count for tl.reshape
+ BLOCK_GROUPS: tl.constexpr, # BLOCK_K // 32 — padded group count (must be power of 2)
+):
+ """Each triton block quantizes one row → FP8 e4m3, write scales directly in swizzled layout.
+
+ We use round up in scale calculation. see: Mishra et al.,
+ Recipes for Pre-training LLMs with MXFP8 (https://arxiv.org/pdf/2506.08027)
+
+ The implementation borrows code from the triton upstream MXFP downcast kernel:
+ https://github.com/triton-lang/triton/blob/main/python/triton_kernels/triton_kernels/numerics_details/mxfp_details/_downcast_to_mxfp.py
+
+ Note on swizzled scale layout (torch.nn.functional.SwizzleType.SWIZZLE_32_4_4):
+
+ Background: In MXFP8, every group of 32 elements shares one 1-byte scale
+ (an e8m0 exponent). For an [M, K] matrix, this gives an [M, K//32] scale
+ matrix. cuBLAS doesn't read these scales in simple row-major order — it
+ expects a "swizzled" layout optimized for its internal access patterns.
+
+ Step 1 — Divide into macro-tiles:
+ The scale matrix is partitioned into 128-row x 4-col macro-tiles.
+ Each tile is stored as a contiguous 512-byte (128 x 4) block.
+
+ Step 2 — Interleave within each tile:
+ Within a macro-tile, the 128 rows are NOT stored sequentially.
+ Instead, they are split into 4 groups of 32 rows:
+ group 0: rows 0- 31
+ group 1: rows 32- 63
+ group 2: rows 64- 95
+ group 3: rows 96-127
+
+ Rows with the same position within their group (same "sub_row")
+ are placed next to each other. So the memory layout is:
+
+ Concretely, for sub_row=0:
+ byte 0: row 0, col 0
+ byte 1: row 0, col 1
+ byte 2: row 0, col 2
+ byte 3: row 0, col 3
+ byte 4: row 32, col 0
+ byte 5: row 32, col 1
+ byte 6: row 32, col 2
+ byte 7: row 32, col 3
+ byte 8: row 64, col 0
+ ...
+ byte 15: row 96, col 3
+
+ The formula to map logical (row, col) → byte offset:
+ tile_idx = (row // 128) * n_col_blocks + (col // 4)
+ sub_row = row % 32
+ group = (row % 128) // 32
+ local_col = col % 4
+ offset = tile_idx * 512 + sub_row * 16 + group * 4 + local_col
+
+ """
+ row = tl.program_id(0)
+ src_row = src_ptr + row * K
+ out_row = out_ptr + row * K
+
+ offs = tl.arange(0, BLOCK_K)
+ mask = offs < K
+
+ # Load full row
+ x = tl.load(src_row + offs, mask=mask, other=0.0).to(tl.float32)
+
+ # Per-group-of-32 max
+ x_grouped = tl.reshape(x, [BLOCK_GROUPS, 32])
+ abs_grouped = tl.abs(x_grouped)
+ max_vals = tl.max(abs_grouped, axis=1)
+
+ # 448 is the max representable value in FP8 e4m3.
+ # dequant_scale = min scale s.t. max_val / scale <= 448.
+ dequant_scale = max_vals / 448.0
+ # Round up to next power of 2 via integer bit manipulation:
+ # Adding 0x007FFFFF (mantissa mask) before masking with 0x7F800000
+ # (exponent-only mask) bumps the exponent if any mantissa bits are set.
+ # Result: 2^ceil(log2(max/448)) as a uint32-encoded float.
+ dequant_exp = (dequant_scale.to(tl.uint32, bitcast=True) + 0x007FFFFF) & 0x7F800000
+ # Reinterpret uint32 back as float32 — now a power-of-2 dequantization scale.
+ dequant_rounded = dequant_exp.to(tl.float32, bitcast=True)
+ # Quantization scale is the reciprocal; guard against div-by-zero for all-zero groups.
+ quant_scale = tl.where(dequant_rounded == 0, 0.0, 1.0 / dequant_rounded)
+
+ # Quantize
+ quantized = x_grouped * quant_scale[:, None]
+ quantized_flat = tl.reshape(quantized, [BLOCK_K])
+ out_fp8 = quantized_flat.to(tl.float8e4nv)
+
+ # Store FP8 data
+ tl.store(out_row + offs, out_fp8, mask=mask)
+
+ # Store swizzled scales
+ scale_exp = (dequant_exp >> 23).to(tl.uint8)
+ col_offs = tl.arange(0, BLOCK_GROUPS)
+ col_mask = col_offs < REAL_GROUPS
+
+ # Compute swizzled offsets for each scale element.
+ #
+ # The scale matrix [M, K//32] is divided into 128×4 macro-tiles.
+ # Within each tile, rows are split into 4 groups of 32 (group = local_row // 32).
+ # Rather than flattening row-major, the layout interleaves groups so that
+ # rows 32 apart are adjacent in memory:
+ #
+ # offset = tile_idx * 512 + sub_row * 16 + group * 4 + local_col
+ macro_row_block = row // 128
+ macro_col_block = col_offs // 4
+ local_row = row % 128
+ local_col = col_offs % 4
+ group = local_row // 32
+ sub_row = local_row % 32
+ tile_idx = macro_row_block * n_col_blocks + macro_col_block
+ swizzled_offs = tile_idx * 512 + sub_row * 16 + group * 4 + local_col
+
+ tl.store(scale_ptr + swizzled_offs, scale_exp, mask=col_mask)
+
+
+def mxfp8_quantize(x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
+ """Quantize a 2D tensor to MXFP8 with fused scale swizzle.
+
+ Args:
+ x: [M, K] tensor in bf16/fp16/fp32. K must be divisible by 32.
+
+ Returns:
+ (data, swizzled_scales):
+ data: [M, K] float8_e4m3fn
+ swizzled_scales: 1D tensor in cuBLAS blocked layout (uint8/e8m0)
+ """
+ assert x.is_cuda and x.dim() == 2
+ assert x.dtype in (torch.bfloat16, torch.float16, torch.float32)
+ M, K = x.shape
+ assert K % 32 == 0, f"K ({K}) must be divisible by 32"
+
+ scale_cols = K // 32
+ n_row_blocks = _ceil_div(M, 128)
+ n_col_blocks = _ceil_div(scale_cols, 4)
+ total_scale_bytes = n_row_blocks * n_col_blocks * 512
+
+ out_data = torch.empty(M, K, dtype=torch.float8_e4m3fn, device=x.device)
+ out_scale = torch.zeros(total_scale_bytes, dtype=torch.uint8, device=x.device)
+
+ BLOCK_K = triton.next_power_of_2(K)
+ BLOCK_GROUPS = BLOCK_K // 32
+
+ _mxfp8_quant_swizzle_kernel[(M,)](
+ out_data,
+ out_scale,
+ x,
+ K,
+ n_col_blocks,
+ REAL_GROUPS=scale_cols,
+ BLOCK_K=BLOCK_K,
+ BLOCK_GROUPS=BLOCK_GROUPS,
+ )
+
+ return out_data, out_scale.view(torch.float8_e8m0fnu)
diff --git a/megatron/core/inference/quantization/mxfp8_tensor.py b/megatron/core/inference/quantization/mxfp8_tensor.py
new file mode 100644
index 00000000000..505f4f9ebc5
--- /dev/null
+++ b/megatron/core/inference/quantization/mxfp8_tensor.py
@@ -0,0 +1,72 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+from dataclasses import dataclass
+from typing import Optional
+
+import torch
+
+try:
+ from flashinfer import mxfp8_quantize as flashinfer_mxfp8_quantize
+
+ HAVE_FLASHINFER = True
+except ImportError:
+ HAVE_FLASHINFER = False
+
+from megatron.core.inference.quantization.mxfp8_quantize import (
+ mxfp8_quantize as mcore_mxfp8_quantize,
+)
+
+
+def _ceil_div(a, b):
+ return (a + b - 1) // b
+
+
+@dataclass
+class MXFP8Tensor:
+ """MXFP8 tensor wrapper storing quantized fp8_e4m3 data and swizzled e8m0 scales."""
+
+ data: torch.Tensor # [M, K] fp8_e4m3fn
+ scale: torch.Tensor # 1D, swizzled cuBLAS blocked layout, e8m0
+ backend: Optional[str] = None # quantization backend: 'flashinfer' or 'triton'
+
+ def size(self, idx: Optional[int] = None):
+ """Wrapper for calling self.data.size()"""
+ return self.data.size(idx)
+
+ def scale_2d(self, K: Optional[int] = None) -> torch.Tensor:
+ """Reshape 1D swizzled scale to 2D for scaled_grouped_mm / scaled_mm.
+
+ Swizzle pads rows to multiples of 128 and cols to multiples of 4.
+ Returns (padded_M, padded_cols) where padded_cols = ceil(K//32, 4) * 4.
+ """
+ if self.scale.dim() == 2:
+ return self.scale
+ if K is None:
+ K = self.data.shape[-1]
+ n_col_blocks = _ceil_div(K // 32, 4)
+ padded_cols = n_col_blocks * 4
+ return self.scale.reshape(-1, padded_cols)
+
+ @classmethod
+ def from_bf16(cls, x: torch.Tensor, group_size: int = 32, backend: str = "flashinfer"):
+ """Quantize BF16 tensor to MXFP8.
+
+ Args:
+ x: [M, K] BF16 tensor on CUDA.
+ group_size: MXFP8 group size (default 32).
+ backend: 'triton' (fused quantize + swizzle Triton kernel) or
+ 'flashinfer' (single fused FlashInfer CUDA kernel).
+ """
+ assert x.is_cuda and x.dim() == 2
+ assert x.shape[-1] % group_size == 0
+ if backend == "flashinfer":
+ assert HAVE_FLASHINFER, "FlashInfer not available"
+ return cls(*flashinfer_mxfp8_quantize(x), backend=backend)
+ elif backend == "triton":
+ xq, xs = mcore_mxfp8_quantize(x)
+ return cls(data=xq, scale=xs, backend=backend)
+ else:
+ raise ValueError(
+ f"Unknown MXFP8 quantization backend: '{backend}'. "
+ "Must be 'triton' or 'flashinfer'."
+ )
diff --git a/megatron/core/inference/quantization/utils.py b/megatron/core/inference/quantization/utils.py
new file mode 100644
index 00000000000..10594be6736
--- /dev/null
+++ b/megatron/core/inference/quantization/utils.py
@@ -0,0 +1,264 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+from typing import Dict, Optional, Tuple
+
+import torch
+
+from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+
+try:
+ from transformer_engine.pytorch.tensor.mxfp8_tensor import MXFP8Tensor as TEMXFP8Tensor
+
+ HAVE_TE = True
+except ImportError:
+ HAVE_TE = False
+
+try:
+ from flashinfer import mm_mxfp8 as flashinfer_mm_mxfp8
+
+ HAVE_FLASHINFER = True
+except ImportError:
+ HAVE_FLASHINFER = False
+
+try:
+ from torch.nn.functional import ScalingType, SwizzleType
+ from torch.nn.functional import scaled_mm as torch_scaled_mm
+
+ HAVE_TORCH_SCALED_MM = True
+except ImportError:
+ HAVE_TORCH_SCALED_MM = False
+
+
+def _verify_te_to_mcore_mxfp8_conversion(te_dequantized, fi_quantized: MXFP8Tensor) -> None:
+ # Sanity check: compare the first logical block (32 values)
+ # Slice logical dimensions first to naturally handle any data swizzling/strides
+ te_block = te_dequantized[0, :32].float()
+
+ # Safely extract bytes from the first logical block, then view as e4m3
+ fi_data_bytes = fi_quantized.data[0, :32].contiguous().view(torch.uint8)
+ fi_data_e4m3 = fi_data_bytes.view(torch.float8_e4m3fn).float()
+
+ # Extract the scale. Logical block (0, 0) is always at physical index 0,
+ # bypassing any scale swizzling layout complexity (like SWIZZLED_128x4)
+ fi_scale_byte = fi_quantized.scale.contiguous().flatten()[0:1].view(torch.uint8).to(torch.int32)
+ fi_scale_f32 = (fi_scale_byte << 23).view(torch.float32)
+
+ fi_block = fi_data_e4m3 * fi_scale_f32
+
+ if not torch.allclose(te_block, fi_block):
+ diff_norm = torch.norm(te_block - fi_block)
+ raise ValueError(f"MXFP8 sanity check failed. Diff norm: {diff_norm}")
+
+
+def quantize_model_to_mxfp8(model: torch.nn.Module, backend: str = "flashinfer") -> None:
+ """Convert TE MXFP8 weights to mcore MXFP8Tensor format.
+
+ Recursively walks the model and replaces each TEMXFP8Tensor parameter
+ with an MXFP8Tensor re-quantized via the specified backend.
+
+ Args:
+ model: The model whose TE MXFP8 parameters should be converted.
+ backend: 'flashinfer' or 'triton' quantization backend.
+ """
+ assert HAVE_TE
+ import logging
+
+ rank = torch.distributed.get_rank()
+ if backend == "flashinfer":
+ assert HAVE_FLASHINFER, "FlashInfer not available for MXFP8 quantization"
+
+ for child in model.children():
+ quantize_model_to_mxfp8(child, backend=backend)
+
+ def replace_in_dict(attr_dict):
+ """Helper function to replace TE MXFP8 weights."""
+ keys = list(attr_dict.keys())
+ for key in keys:
+ val = attr_dict[key]
+ is_te_mxfp8 = isinstance(val, TEMXFP8Tensor) or (
+ hasattr(val, 'data') and isinstance(val.data, TEMXFP8Tensor)
+ )
+ if is_te_mxfp8:
+ # Undo the TE quantization and re-quantize
+ # Note that this introduces a one-time overhead but avoids any
+ # numerical differences between TE and mcore MXFP8 formats
+ te_dequantized = val.dequantize()
+ mcore_quantized = MXFP8Tensor.from_bf16(te_dequantized, backend=backend)
+ _verify_te_to_mcore_mxfp8_conversion(te_dequantized, mcore_quantized)
+ del model._parameters[key]
+ setattr(model, key, mcore_quantized)
+
+ if hasattr(model, '_parameters') and model._parameters:
+ replace_in_dict(model._parameters)
+
+ return model
+
+
+def _should_quantize_param(val: torch.Tensor) -> bool:
+ """Return True if a parameter should be quantized to FlashInfer MXFP8."""
+ if not val.is_cuda:
+ return False
+ if HAVE_TE and isinstance(val, TEMXFP8Tensor):
+ return True
+ if HAVE_TE and hasattr(val, 'data') and isinstance(val.data, TEMXFP8Tensor):
+ return True
+ if isinstance(val, MXFP8Tensor):
+ return True
+ if hasattr(val, 'data') and isinstance(val.data, MXFP8Tensor):
+ return True
+ return False
+
+
+def _to_bf16(val: torch.Tensor) -> torch.Tensor:
+ """Convert a parameter value to BF16 for quantization."""
+ if HAVE_TE and isinstance(val, TEMXFP8Tensor):
+ return val.dequantize()
+ if HAVE_TE and hasattr(val, 'data') and isinstance(val.data, TEMXFP8Tensor):
+ return val.data.dequantize()
+ return val.data.to(torch.bfloat16)
+
+
+def collect_mxfp8_param_metadata(
+ model: torch.nn.Module,
+) -> Dict[str, Tuple[torch.Size, torch.dtype, torch.device]]:
+ """Record shape/dtype/device for each parameter that will be quantized.
+
+ Called once before the first quantization to record the original parameter
+ metadata (shape, dtype, device) before any format conversion.
+ """
+ metadata: Dict[str, Tuple[torch.Size, torch.dtype, torch.device]] = {}
+ for name, param in model.named_parameters():
+ if _should_quantize_param(param):
+ if HAVE_TE and isinstance(param, TEMXFP8Tensor):
+ bf16 = param.dequantize()
+ metadata[name] = (bf16.shape, bf16.dtype, bf16.device)
+ else:
+ metadata[name] = (param.shape, param.dtype, param.device)
+ return metadata
+
+
+def quantize_params_to_mxfp8(
+ model: torch.nn.Module,
+ persistent_buffers: Optional[Dict[str, MXFP8Tensor]] = None,
+ _prefix: str = "",
+ backend: str = "flashinfer",
+) -> Dict[str, MXFP8Tensor]:
+ """Quantize model parameters to MXFP8Tensor format.
+
+ Handles both TEMXFP8Tensor (fp8_param=True) and BF16/FP16 nn.Parameter
+ inputs. When *persistent_buffers* is provided, new quantized values are
+ ``copy_()``'d into the existing MXFP8Tensor objects so that CUDA-graph
+ device-pointer captures remain valid.
+
+ Args:
+ model: The model whose parameters should be quantized.
+ persistent_buffers: If not ``None``, a dict mapping fully-qualified
+ parameter names to previously-created ``MXFP8Tensor`` objects.
+ Updated in-place and returned.
+ _prefix: Internal recursion prefix – callers should not set this.
+ backend: 'flashinfer' or 'triton' quantization backend.
+
+ Returns:
+ The ``persistent_buffers`` dict (created on first call if ``None``).
+ """
+ if backend == "flashinfer":
+ assert HAVE_FLASHINFER, "FlashInfer not available for MXFP8 quantization"
+
+ if persistent_buffers is None:
+ persistent_buffers = {}
+
+ # Recurse through child modules
+ for child_name, child_module in model.named_children():
+ child_prefix = f"{_prefix}{child_name}." if _prefix else f"{child_name}."
+ quantize_params_to_mxfp8(
+ child_module, persistent_buffers, _prefix=child_prefix, backend=backend
+ )
+
+ # Process parameters owned directly by this module
+ if hasattr(model, '_parameters') and model._parameters:
+ keys = list(model._parameters.keys())
+ for key in keys:
+ val = model._parameters[key]
+ if val is None:
+ continue
+ if not _should_quantize_param(val):
+ continue
+
+ fqn = f"{_prefix}{key}"
+ bf16_data = _to_bf16(val)
+
+ if fqn in persistent_buffers:
+ # Subsequent call: copy into existing tensors to preserve addresses
+ new_tensor = MXFP8Tensor.from_bf16(bf16_data, backend=backend)
+ persistent_buffers[fqn].data.copy_(new_tensor.data)
+ persistent_buffers[fqn].scale.copy_(new_tensor.scale)
+ mcore_tensor = persistent_buffers[fqn]
+ else:
+ # First call: create new MXFP8Tensor
+ mcore_tensor = MXFP8Tensor.from_bf16(bf16_data, backend=backend)
+
+ # Verify correctness for TEMXFP8Tensor inputs
+ if HAVE_TE and isinstance(val, TEMXFP8Tensor):
+ _verify_te_to_mcore_mxfp8_conversion(bf16_data, mcore_tensor)
+
+ persistent_buffers[fqn] = mcore_tensor
+
+ # Replace nn.Parameter with MXFP8Tensor attribute
+ del model._parameters[key]
+ setattr(model, key, mcore_tensor)
+
+ return persistent_buffers
+
+
+def _mm_mxfp8_flashinfer(x_mxfp8: MXFP8Tensor, weight: MXFP8Tensor, out=None):
+ """MXFP8 matmul via FlashInfer."""
+ return flashinfer_mm_mxfp8(
+ x_mxfp8.data, weight.data.T, x_mxfp8.scale, weight.scale, out_dtype=torch.bfloat16, out=out
+ )
+
+
+def _mm_mxfp8_torch(x_mxfp8: MXFP8Tensor, weight: MXFP8Tensor, out=None):
+ """MXFP8 matmul via torch.nn.functional.scaled_mm."""
+ result = torch_scaled_mm(
+ x_mxfp8.data,
+ weight.data.t(),
+ x_mxfp8.scale_2d(),
+ ScalingType.BlockWise1x32,
+ weight.scale,
+ ScalingType.BlockWise1x32,
+ swizzle_a=SwizzleType.SWIZZLE_32_4_4,
+ swizzle_b=SwizzleType.SWIZZLE_32_4_4,
+ output_dtype=torch.bfloat16,
+ )
+ if out is not None:
+ out.copy_(result)
+ return out
+ return result
+
+
+def mm_mxfp8(x: torch.Tensor, weight: MXFP8Tensor, out: torch.Tensor = None):
+ """Compute a matmul in MXFP8.
+
+ Quantizes the bf16 input activation tensor on the fly. Weight must be
+ pre-quantized. Dispatches to FlashInfer or torch based on weight.backend.
+ """
+ backend = weight.backend
+ assert (
+ backend is not None
+ ), "weight.backend is None — was the weight created via MXFP8Tensor.from_bf16?"
+
+ x_squeezed = x.squeeze(1)
+ x_mxfp8 = MXFP8Tensor.from_bf16(x_squeezed, backend=backend)
+
+ if backend == "flashinfer":
+ assert HAVE_FLASHINFER, "FlashInfer not available for MXFP8 matmul"
+ result = _mm_mxfp8_flashinfer(x_mxfp8, weight, out=out)
+ elif backend == "triton":
+ assert (
+ HAVE_TORCH_SCALED_MM
+ ), "torch.nn.functional.scaled_mm with ScalingType/SwizzleType not available"
+ result = _mm_mxfp8_torch(x_mxfp8, weight, out=out)
+ else:
+ raise ValueError(f"Unknown MXFP8 backend: '{backend}'")
+
+ return result.unsqueeze(1)
diff --git a/megatron/core/inference/sampling_params.py b/megatron/core/inference/sampling_params.py
index ba1acae4c57..13bc8ac0d7b 100644
--- a/megatron/core/inference/sampling_params.py
+++ b/megatron/core/inference/sampling_params.py
@@ -24,7 +24,7 @@ class SamplingParams:
return_log_probs: bool = False
skip_prompt_log_probs: bool = False
return_segments: bool = False # Whether to return individually detokenized tokens
- num_tokens_to_generate: int = 30
+ num_tokens_to_generate: int = None
num_tokens_total: Optional[int] = None # Cannot set both this and num_tokens_to_generate
termination_id: Optional[int] = None
top_n_logprobs: int = 0
@@ -33,6 +33,7 @@ class SamplingParams:
stop_words: Optional[List[str]] = (
None # List of strings that will stop generation when produced
)
+ detokenize_stop_sequence: bool = False # Keep stop words and EOD in generated text
def __post_init__(self):
"""Ensure backward compatibility for return_prompt_top_n_logprobs.
diff --git a/megatron/core/inference/symmetric_memory.py b/megatron/core/inference/symmetric_memory.py
new file mode 100644
index 00000000000..254d41ce294
--- /dev/null
+++ b/megatron/core/inference/symmetric_memory.py
@@ -0,0 +1,182 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""Lazy-initialized symmetric memory manager for inference.
+
+Provides a registry of SymmetricMemoryBuffer instances keyed by a
+user-supplied identifier (e.g. "tp", "ep"). Buffers are created on first
+access so that callers never need to worry about initialization ordering
+relative to the inference context.
+"""
+
+from __future__ import annotations
+
+import operator
+from functools import reduce
+from typing import Optional
+
+import torch
+
+try:
+ import torch.distributed._symmetric_memory as symm_mem
+
+ HAVE_TORCH_SYMM_MEM = True
+except ImportError:
+ HAVE_TORCH_SYMM_MEM = False
+
+try:
+ import triton # pylint: disable=unused-import
+
+ HAVE_TRITON = True
+except ImportError:
+ HAVE_TRITON = False
+
+
+class SymmetricMemoryBuffer:
+ """
+ symmetric memory buffer used in inference.
+ This buffer is used by mcore-inference's low-latency
+ NVLS all-gather and reduce-scatter collectives.
+ """
+
+ def __init__(self, size_in_mb, process_group):
+ if not HAVE_TORCH_SYMM_MEM or not HAVE_TRITON:
+ # This should be hit if the user is running an older
+ # version of torch, or if they do not have triton
+ # installed.
+ self.symm_buffer = None
+ self.symm_mem_hdl = None
+ else:
+ numel = int(size_in_mb * 1024 * 1024) # size in bytes
+ try:
+ symm_mem.enable_symm_mem_for_group(process_group.group_name)
+ self.symm_buffer = symm_mem.empty(numel, dtype=torch.uint8, device='cuda')
+ self.symm_mem_hdl = symm_mem.rendezvous(self.symm_buffer, process_group)
+ except RuntimeError as e:
+ # If symmetric memory initialization fails, set buffer and handle to None
+ # This should happen if the process group is not contained within NVlink
+ self.symm_buffer = None
+ self.symm_mem_hdl = None
+
+ def _can_allocate(self, numel, dtype) -> bool:
+ """
+ Returns whether enough symmetric memory is available
+ for the given tensor shape and dtype.
+ """
+ if self.symm_mem_hdl is None:
+ return False
+ size_of_dtype = torch.tensor([], dtype=dtype).element_size()
+ required_len = numel * size_of_dtype
+ return required_len <= self.symm_buffer.numel()
+
+ def _allocate(self, numel, dtype) -> torch.Tensor:
+ """
+ Allocates a sub-tensor from the self.symm_buffer for the given numel and dtype"""
+ required_bytes = numel * torch.tensor([], dtype=dtype).element_size()
+ return self.symm_buffer[0:required_bytes].view(dtype).view(numel)
+
+ def maybe_get_tensors(self, tensor_specs, alignment=16):
+ """
+ Pack multiple tensors contiguously in the symmetric buffer with alignment.
+
+ Each tensor's starting offset is aligned to `alignment` bytes (default 16
+ for 128-bit multimem access).
+
+ Args:
+ tensor_specs: list of (numel, dtype) tuples.
+ alignment: byte alignment for each tensor's start offset (default 16).
+
+ Returns:
+ {"handle": None, "tensors": None} if unavailable or insufficient space.
+ {"handle": symm_mem_hdl, "tensors": [(raw_byte_view, byte_offset), ...]}
+ on success, where raw_byte_view is a uint8 slice of the buffer.
+ """
+ _NONE_RESULT = {"handle": None, "tensors": None}
+ if self.symm_mem_hdl is None:
+ return _NONE_RESULT
+
+ # Compute aligned byte sizes and running offsets
+ slices = []
+ current_offset = 0
+ for numel, dtype in tensor_specs:
+ nbytes = numel * torch.tensor([], dtype=dtype).element_size()
+ aligned_nbytes = ((nbytes + alignment - 1) // alignment) * alignment
+ slices.append((current_offset, nbytes))
+ current_offset += aligned_nbytes
+
+ if not self._can_allocate(current_offset, torch.uint8):
+ return _NONE_RESULT
+
+ tensors = []
+ for offset, nbytes in slices:
+ tensors.append((self.symm_buffer[offset : offset + nbytes], offset))
+
+ return {"handle": self.symm_mem_hdl, "tensors": tensors}
+
+ def maybe_get_tensor(self, tensor_shape, dtype):
+ """
+ Returns (potentially) a sub-tensor from the self.symm_buffer for the given shape.
+ If enough symmetric memory is not available, returns None.
+ """
+ if self.symm_mem_hdl is None:
+ return {"tensor": None, "handle": None}
+ numel = reduce(operator.mul, tensor_shape, 1)
+ if not self._can_allocate(numel, dtype):
+ return {"tensor": None, "handle": None}
+ return {
+ "tensor": self._allocate(numel, dtype).view(*tensor_shape),
+ "handle": self.symm_mem_hdl,
+ }
+
+
+class SymmetricMemoryManager:
+ """Registry of lazily-initialized symmetric memory buffers.
+
+ Usage::
+
+ buf = SymmetricMemoryManager.get_buffer("tp", process_group=tp_group)
+ result = buf.maybe_get_tensor(shape, dtype)
+ """
+
+ _buffers: dict[str, SymmetricMemoryBuffer] = {}
+ _default_size_mb: int = 256
+
+ @classmethod
+ def get_buffer(
+ cls,
+ key: str,
+ process_group: Optional[torch.distributed.ProcessGroup] = None,
+ size_mb: Optional[int] = None,
+ ) -> SymmetricMemoryBuffer:
+ """Return the buffer for *key*, creating it on first call.
+
+ Args:
+ key: Unique identifier (e.g. "tp", "ep").
+ process_group: Required on the first call for a given key.
+ Subsequent calls may omit it.
+ size_mb: Buffer size in MiB (default 256).
+ """
+ if key not in cls._buffers:
+ assert (
+ process_group is not None
+ ), f"SymmetricMemoryManager: process_group is required on first access for key='{key}'"
+ cls._buffers[key] = SymmetricMemoryBuffer(
+ size_in_mb=size_mb or cls._default_size_mb, process_group=process_group
+ )
+ return cls._buffers[key]
+
+ @classmethod
+ def destroy(cls, key: Optional[str] = None) -> None:
+ """Destroy one or all buffers.
+
+ Args:
+ key: If provided, destroy only that buffer. Otherwise destroy all.
+ """
+ if key is not None:
+ cls._buffers.pop(key, None)
+ else:
+ cls._buffers.clear()
+
+ @classmethod
+ def is_initialized(cls, key: str) -> bool:
+ """Check whether a buffer has been created for *key*."""
+ return key in cls._buffers
diff --git a/megatron/core/inference/text_generation_controllers/simple_text_generation_controller.py b/megatron/core/inference/text_generation_controllers/simple_text_generation_controller.py
deleted file mode 100644
index 340cadb48a9..00000000000
--- a/megatron/core/inference/text_generation_controllers/simple_text_generation_controller.py
+++ /dev/null
@@ -1,5 +0,0 @@
-# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
-
-from megatron.core.inference.text_generation_controllers.text_generation_controller import ( # noqa: F401 # pylint: disable=unused-import
- TextGenerationController as SimpleTextGenerationController,
-)
diff --git a/megatron/core/inference/text_generation_controllers/text_generation_controller.py b/megatron/core/inference/text_generation_controllers/text_generation_controller.py
index a5233983ed0..ba190f799f8 100644
--- a/megatron/core/inference/text_generation_controllers/text_generation_controller.py
+++ b/megatron/core/inference/text_generation_controllers/text_generation_controller.py
@@ -6,30 +6,33 @@
import functools
import inspect
from collections import defaultdict
-from typing import Any, Dict, Iterator, List, Optional, OrderedDict, Tuple, Union
+from typing import Any, Dict, List, Optional, OrderedDict, Tuple, Union
import torch
import torch.nn.functional as F
from torch import Tensor
-from torch.distributed import ProcessGroup
+from megatron.core import parallel_state
from megatron.core.inference.async_stream import AsyncStream
from megatron.core.inference.communication_utils import (
broadcast_from_last_pipeline_stage,
- is_pipeline_first_stage,
is_pipeline_last_stage,
)
from megatron.core.inference.contexts.dynamic_context import MaxSequenceLengthOverflowError
+from megatron.core.inference.contexts.static_context import StaticInferenceContext
from megatron.core.inference.inference_request import InferenceRequest, Status
from megatron.core.inference.model_inference_wrappers.abstract_model_inference_wrapper import (
AbstractModelInferenceWrapper,
)
from megatron.core.inference.sampling_params import SamplingParams
from megatron.core.inference.utils import get_attention_mask, set_decode_expert_padding
+from megatron.core.models.multimodal.llava_model import LLaVAModel
+from megatron.core.tensor_parallel.mappings import gather_from_sequence_parallel_region
from megatron.core.transformer.enums import CudaGraphScope
from megatron.core.transformer.moe.moe_layer import BaseMoELayer
+from megatron.core.transformer.moe.router_replay import RouterReplay, RouterReplayAction
from megatron.core.transformer.utils import set_model_to_sequence_parallel
-from megatron.core.utils import get_asyncio_loop, get_model_config, unwrap_model
+from megatron.core.utils import get_asyncio_loop, get_model_config, get_pg_size, unwrap_model
try:
import transformer_engine as te # pylint: disable=unused-import
@@ -52,32 +55,45 @@ class TextGenerationController:
inference_wrapped_model (AbstractModelInferenceWrapper): A model that
is wrapped using the specs given in the abstract_model_inference_wrapper.py
tokenizer (_type_): Tokenizer used for tokenizing and detokenizing the prompts
- pp_group (ProcessGroup): Process group for pipeline parallelism
"""
- def __init__(
- self,
- inference_wrapped_model: AbstractModelInferenceWrapper,
- tokenizer,
- pp_group: ProcessGroup = None,
- ):
+ def __init__(self, inference_wrapped_model: AbstractModelInferenceWrapper, tokenizer):
self.inference_wrapped_model = inference_wrapped_model
+ self.model_config = self.inference_wrapped_model.model.config
+ inference_config = self.inference_wrapped_model.inference_context.config
self.tokenizer = tokenizer
+ self.num_speculative_tokens = inference_config.num_speculative_tokens
- self.pp_group = pp_group
+ pg_collection = inference_config.pg_collection
+ if pg_collection is not None:
+ self.pp_group = pg_collection.pp
+ else:
+ self.pp_group = parallel_state.get_pipeline_model_parallel_group()
- # For models without pipeline parallelism, is_first_stage and is_last_stage returns True
- self.model_is_pipeline_parallel = not (
- is_pipeline_first_stage(self.pp_group) and is_pipeline_last_stage(self.pp_group)
- )
+ self.model_is_pipeline_parallel = self.model_config.pipeline_model_parallel_size > 1
+
+ # Use padded vocab size because tokenizer vocab size might pad to nearest power of 2.
+ # TODO(ksanthanam): Consider deprecating this check if LLaVAModel is no longer used
+ unwrapped_model = unwrap_model(self.inference_wrapped_model.model)
+ if isinstance(unwrapped_model, LLaVAModel):
+ self.vocab_size = unwrapped_model.language_model.vocab_size
+ else:
+ self.vocab_size = unwrapped_model.vocab_size
- model_config = get_model_config(self.inference_wrapped_model.model)
self.sampling_rng = torch.Generator(device=torch.cuda.current_device())
- self.sampling_rng.manual_seed(model_config.inference_sampling_seed)
+ self.num_mtp_heads = self._get_mtp_num_heads()
+ self.sampling_rng.manual_seed(self.model_config.inference_sampling_seed)
if self.inference_wrapped_model.inference_context.is_dynamic_batching():
self._init_dynamic_sampling_tensors()
+ def _get_mtp_num_heads(self) -> int:
+ """Get the number of MTP layers from the model config."""
+ model = self.inference_wrapped_model.model
+ if hasattr(model, 'config') and hasattr(model.config, 'mtp_num_layers'):
+ return model.config.mtp_num_layers or 0
+ return 0
+
def set_stop_word_finished_ids_callback(self, callback):
"""Set a callback to get request IDs that should be marked as finished due to stop words.
@@ -98,12 +114,16 @@ def _init_dynamic_sampling_tensors(self):
self._get_stop_word_finished_ids_callback = None
device = torch.cuda.current_device()
- logits_dtype = self.inference_wrapped_model.inference_wrapper_config.params_dtype
- # Use padded vocab size because tokenizer vocab size might pad to nearest power of 2.
- vocab_size = self.inference_wrapped_model.inference_wrapper_config.padded_vocab_size
+ logits_dtype = self.inference_wrapped_model.config.params_dtype
self._sampling_backend = "torch"
self._sampled_tokens_cuda = torch.empty(max_requests, dtype=torch.int64, device=device)
+ # Speculative tokens tensor will be allocated later when num_speculative_tokens is set by the engine
+ self._accepted_tokens_per_request = None
+ # MTP tensor will be allocated later when num_speculative_tokens is set by the engine
+ self._sampled_mtp_tokens_cuda = None
+ # Last accepted sequence indices for serial MTP computation
+ self._last_accepted_seq_indices = None
# Keep track of request metadata.
self._request_metadata: Dict[str, Tensor] = {}
@@ -117,60 +137,84 @@ def _init_dynamic_sampling_tensors(self):
# Used for inefficient torch sampling.
if self._sampling_backend == "torch":
- self._torch_sampling_buckets: Iterator[Tuple] = []
+ self._torch_sampling_buckets: List[Tuple] = []
+
+ self._init_mtp_sampling_tensor()
+
+ def _init_mtp_sampling_tensor(self):
+ """Initialize the MTP sampling tensor after num_speculative_tokens is set."""
+ if self.num_speculative_tokens is not None and self.num_speculative_tokens > 0:
+ context = self.inference_wrapped_model.inference_context
+ max_requests = context.max_requests
+ device = torch.cuda.current_device()
+ self._sampled_mtp_tokens_cuda = torch.empty(
+ [self.num_speculative_tokens, max_requests], dtype=torch.int64, device=device
+ )
+ self._accepted_tokens_per_request = (
+ torch.ones(
+ [max_requests, self.num_speculative_tokens], dtype=torch.int64, device=device
+ )
+ * -1
+ )
- def tokenize_prompt(self, prompt: str, add_BOS: bool = False) -> List[int]:
+ @staticmethod
+ def tokenize_prompt(tokenizer, prompt: str, add_BOS: bool = False) -> List[int]:
"""Utility to tokenize the input prompts.
Args:
+ tokenizer: The tokenizer to use.
prompt (str): The input prompt.
+ add_BOS (bool): Whether to add a BOS token.
Returns:
List[int]: Returns the tokenized prompt.
"""
- prompt_tokens = self.tokenizer.tokenize(prompt)
+ prompt_tokens = tokenizer.tokenize(prompt)
if add_BOS:
- assert self.tokenizer.bos is not None
+ assert tokenizer.bos is not None
- while prompt_tokens and prompt_tokens[0] == self.tokenizer.bos:
+ while prompt_tokens and prompt_tokens[0] == tokenizer.bos:
prompt_tokens.pop(0)
if add_BOS:
- prompt_tokens = [self.tokenizer.bos] + prompt_tokens
+ prompt_tokens = [tokenizer.bos] + prompt_tokens
return prompt_tokens
- def _detokenize(self, tokens: List[int], skip_special_tokens: bool = True) -> str:
+ @staticmethod
+ def detokenize(
+ tokenizer, tokens: List[int], remove_EOD: bool = True, skip_special_tokens: bool = True
+ ) -> str:
"""
- Detokenize a sequence of token IDs, handling skip_special_tokens for
- different tokenizer APIs.
-
- On the first call, inspects `self.tokenizer.detokenize` to see if it accepts
- a `skip_special_tokens` keyword argument, and caches that result on `self`.
- Subsequent calls will use the cached flag to invoke `detokenize` with the
- correct signature (with or without `skip_special_tokens`).
+ Detokenize a sequence of token IDs, optionally removing trailing EOD
+ tokens and handling skip_special_tokens for different tokenizer APIs.
Args:
+ tokenizer: The tokenizer to use for detokenization.
tokens (List[int]): The token IDs to convert back to text.
+ remove_EOD (bool): Whether to remove trailing EOD tokens before
+ detokenization. Defaults to True.
skip_special_tokens (bool): Whether to remove special tokens (e.g. BOS/EOS)
during detokenization. Only passed through if the tokenizer supports it.
Returns:
str: The detokenized string.
"""
- # cache the check on first call
- if not hasattr(self, "_detok_accepts_skip"):
- sig_params = inspect.signature(self.tokenizer.detokenize).parameters.values()
- self._detok_accepts_skip = any(
- p.name == "skip_special_tokens" or p.kind == inspect.Parameter.VAR_KEYWORD
- for p in sig_params
- )
- if self._detok_accepts_skip:
- return self.tokenizer.detokenize(tokens, skip_special_tokens=skip_special_tokens)
+ if remove_EOD and getattr(tokenizer, "eod", None) is not None:
+ while tokens and tokens[-1] == tokenizer.eod:
+ tokens = tokens[:-1]
+
+ sig_params = inspect.signature(tokenizer.detokenize).parameters.values()
+ detok_accepts_skip = any(
+ p.name == "skip_special_tokens" or p.kind == inspect.Parameter.VAR_KEYWORD
+ for p in sig_params
+ )
+ if detok_accepts_skip:
+ return tokenizer.detokenize(tokens, skip_special_tokens=skip_special_tokens)
else:
- return self.tokenizer.detokenize(tokens)
+ return tokenizer.detokenize(tokens)
def detokenize_generations(
self,
@@ -199,7 +243,10 @@ def detokenize_generations(
if not detokenize_segments:
tokens = tokens_gpu_tensor.tolist()
- return self._detokenize(tokens, skip_special_tokens=skip_special_tokens), None
+ return (
+ self.detokenize(self.tokenizer, tokens, skip_special_tokens=skip_special_tokens),
+ None,
+ )
prompts_plus_generations: List[str] = []
prompts_plus_generations_segments: List[List[str]] = []
@@ -209,7 +256,7 @@ def detokenize_generations(
for sequence_tokens, length in zip(tokens, lengths):
sequence_tokens = sequence_tokens[:length]
- detok_str = self._detokenize(sequence_tokens)
+ detok_str = self.detokenize(self.tokenizer, sequence_tokens)
prompts_plus_generations.append(detok_str)
offsets = self.tokenizer.offsets(sequence_tokens, detok_str)
words = [
@@ -218,7 +265,7 @@ def detokenize_generations(
prompts_plus_generations_segments.append(words)
- text = self._detokenize(tokens[0], skip_special_tokens=skip_special_tokens)
+ text = self.detokenize(self.tokenizer, tokens[0], skip_special_tokens=skip_special_tokens)
return text, prompts_plus_generations_segments
@@ -270,6 +317,8 @@ def modify_logits_for_top_p_filtering(logits, top_p):
# in the original implementation:
# https://github.com/ari-holtzman/degen/blob/master/gen.py
# and I guess it is needed so keeping it for now.
+ # Clone needed: filter_[:, 1:] and filter_[:, :-1] are overlapping views;
+ # without clone, each write would corrupt the next read during the shift.
filter_[:, 1:] = filter_[:, :-1].clone()
# Make sure we at least have one token to select from.
filter_[..., 0] = 0
@@ -282,6 +331,8 @@ def modify_logits_for_top_p_filtering(logits, top_p):
if top_k == 1:
sampled_logits = torch.argmax(last_token_logits, dim=-1)
else:
+ # Clone needed: .div_() and masked_fill_() below modify in-place,
+ # which would mutate the caller's tensor without this clone.
last_token_logits = last_token_logits.clone()
if temperature != 1.0:
last_token_logits.div_(temperature)
@@ -505,7 +556,6 @@ def _dynamic_step_context_init(
position_ids (Tensor): The active position IDs.
"""
context = self.inference_wrapped_model.inference_context
- inference_wrapper_config = self.inference_wrapped_model.inference_wrapper_config
active_request_slice = slice(context.paused_request_count, context.total_request_count)
# Remove Float16Module wrapper if it exists
@@ -513,16 +563,25 @@ def _dynamic_step_context_init(
model_config = get_model_config(unwrapped_model)
# Initialize attention state.
- context.initialize_attention_state(construct_graph_dimensions=construct_graph_dimensions)
+ context.initialize_attention_state(
+ construct_graph_dimensions=construct_graph_dimensions,
+ is_expert_parallel_dummy_cuda_graph_step=is_dummy_forward,
+ )
# If using symmetric kernels and we are using using nccl
# for prefill turn off symmetric kernels
- symmetric_ar_type = model_config.symmetric_ar_type
- nccl_all_reduce_for_prefill = inference_wrapper_config.nccl_all_reduce_for_prefill
+ symmetric_ar_type = self.model_config.symmetric_ar_type
+ nccl_all_reduce_for_prefill = self.model_config.nccl_all_reduce_for_prefill
# Turning on/off MoE padding for cuda-graphs
moe_pad_experts_for_cuda_graph_inference = (
- inference_wrapper_config.moe_pad_experts_for_cuda_graph_inference
+ self.model_config.moe_pad_experts_for_cuda_graph_inference
)
+ is_inference_optimized = self.model_config.transformer_impl == "inference_optimized"
+ if is_inference_optimized:
+ assert not moe_pad_experts_for_cuda_graph_inference, (
+ "moe_pad_experts_for_cuda_graph_inference cannot be True when "
+ "transformer_impl is 'inference_optimized'"
+ )
if moe_pad_experts_for_cuda_graph_inference:
if context.using_cuda_graph_this_step():
capacity_factor = model_config.num_moe_experts / model_config.moe_router_topk
@@ -530,10 +589,6 @@ def _dynamic_step_context_init(
else:
set_decode_expert_padding(unwrapped_model, False)
- # initialize symmetric memory if needed
- if model_config.transformer_impl == "inference_optimized":
- context.maybe_initialize_symmetric_memory()
-
if nccl_all_reduce_for_prefill and symmetric_ar_type is not None:
if context.is_decode_only():
# Turn on symmetric all reduce when in decode mode
@@ -569,8 +624,6 @@ def _dynamic_step_forward_logits(self, input_ids: Tensor, position_ids: Tensor)
input_ids (Tensor): The input token IDs.
position_ids (Tensor): The position IDs.
"""
- inference_wrapper_config = self.inference_wrapped_model.inference_wrapper_config
-
context = self.inference_wrapped_model.inference_context
active_request_count = context.total_request_count - context.paused_request_count
@@ -578,22 +631,26 @@ def _dynamic_step_forward_logits(self, input_ids: Tensor, position_ids: Tensor)
logits = self.inference_wrapped_model.run_one_forward_step(
{"tokens": input_ids, "position_ids": position_ids, "attention_mask": None}
)
+ # logits shape: [1, seq_len, vocab_size]
+
+ # Note: When speculative decoding is active (num_speculative_tokens > 0),
+ # the model skips MTP computation during the forward pass. MTP logits
+ # will be computed serially after verification to ensure they are
+ # conditioned on verified tokens only.
if self.model_is_pipeline_parallel:
- logits_seq_len = (
- active_request_count
- if context.materialize_only_last_token_logits
- else input_ids.shape[1]
- )
- vocab_size = inference_wrapper_config.padded_vocab_size
- logits_shape = [1, logits_seq_len, vocab_size]
+ if context.config.materialize_only_last_token_logits:
+ logits_seq_len = active_request_count
+ else:
+ logits_seq_len = input_ids.shape[1]
+ logits_shape = [1, logits_seq_len, self.vocab_size]
if is_pipeline_last_stage(self.pp_group):
assert logits is not None and torch.Size(logits_shape) == logits.shape
logits = broadcast_from_last_pipeline_stage(
logits_shape,
- dtype=inference_wrapper_config.params_dtype,
+ dtype=self.model_config.params_dtype,
tensor=logits,
pp_group=self.pp_group,
)
@@ -608,25 +665,493 @@ def _dynamic_step_sample_bookkeeping(self):
if self._sampling_backend == "torch":
# Bucketize the core sampling parameters.
# Doing so via list comprehension is orders of magnitude faster than via torch.
- bucket_map = {}
+ bucket_map = defaultdict(list)
# Shorthands for the dictionary comprehension.
temp = self._request_metadata["temperature"][active_request_slice].tolist()
top_k = self._request_metadata["top_k"][active_request_slice].tolist()
top_p = self._request_metadata["top_p"][active_request_slice].tolist()
- for i, (t, k, p) in enumerate(zip(temp, top_k, top_p)):
- h = (t, k, p)
- bucket = bucket_map.get(h, None)
- if bucket is None:
- bucket_map[h] = ([i], i)
- else:
- bucket[0].append(i)
+ for request_index, (t, k, p) in enumerate(zip(temp, top_k, top_p)):
+ sampling_params = (t, k, p)
+ bucket_map[sampling_params].append(request_index)
+
+ # Just unpack the key directly!
+ self._torch_sampling_buckets = [
+ (indices, *sampling_params) for sampling_params, indices in bucket_map.items()
+ ]
+
+ def _rewind_kv_cache(self):
+ """Update the KV cache bookkeeping for speculative decoding.
+
+ After forward pass with speculative tokens, some tokens may be rejected.
+ This function "rewinds" the KV cache bookkeeping to reflect only the accepted tokens.
+
+ When speculative tokens are rejected, we need to:
+ 1. Update request_kv_length_offsets (total sequence length)
+ 2. Update request_last_kv_block_offset (position within last block)
+ 3. If rewinding crosses a block boundary:
+ - Reduce request_kv_block_counts
+ - Update request_last_kv_block_id to point to the previous block
+ - Clear the entry in request_to_kv_block_ids for the released block
+ - Release the block back to the allocator
+ """
+ context = self.inference_wrapped_model.inference_context
+ active_request_count = context.total_request_count - context.paused_request_count
+ active_request_slice = slice(context.paused_request_count, context.total_request_count)
+
+ # Get the accepted token counts for each request
+ # Note: _accepted_token_counts is indexed from 0 to active_request_count-1
+ accepted_tokens_per_request = self._accepted_token_counts_per_request[:active_request_count]
+
+ # Number of tokens to rewind (rejected speculative tokens)
+ num_tokens_to_rewind = self.num_speculative_tokens - accepted_tokens_per_request
+
+ # For prefill requests, no speculative tokens were forwarded through the model,
+ # so there is nothing to rewind.
+ request_in_prefill_status = context.request_in_prefill_status_tensor[active_request_slice]
+ num_tokens_to_rewind[request_in_prefill_status == 1] = 0
+
+ # Save the original offset BEFORE modifying to correctly detect block boundary crossing
+ original_offset = context.request_last_kv_block_offset[active_request_slice].clone()
+
+ # Check which requests need to rewind to a previous block BEFORE modifying
+ # A request crosses back to a previous block if: original_offset - num_tokens_to_rewind < 0
+ remove_allocated_blocks_mask = (original_offset - num_tokens_to_rewind) < 0
+
+ # Update the offsets
+ context.request_last_kv_block_offset[active_request_slice] = (
+ original_offset - num_tokens_to_rewind
+ ) % context.block_size_tokens
+
+ context.request_kv_length_offsets[active_request_slice] = (
+ context.request_kv_length_offsets[active_request_slice] - num_tokens_to_rewind
+ )
+
+ # No need to update request_query_lengths (It will be set correctly in the next iteration)
+
+ # For requests that crossed back to a previous block, we need to:
+ # 1. Reduce the block count by 1
+ # 2. Get the block ID to release (current request_last_kv_block_id)
+ # 3. Update request_last_kv_block_id to point to the previous block
+ # 4. Clear the entry in request_to_kv_block_ids for the released block
+ # 5. Release the block back to the allocator
+ if remove_allocated_blocks_mask.any():
+ # Get indices of requests that need to release a block (relative to active requests)
+ requests_needing_release = torch.nonzero(remove_allocated_blocks_mask, as_tuple=True)[0]
+ # Convert to absolute indices in the context tensors
+ absolute_indices = requests_needing_release + context.paused_request_count
+
+ # No clone needed: advanced (fancy) indexing with a tensor already returns
+ # a copy, not a view.
+ blocks_to_release = context.request_last_kv_block_id[absolute_indices]
+
+ # Reduce block counts for requests that crossed back
+ context.request_kv_block_counts[absolute_indices] -= 1
+
+ # Get the new block counts after decrement
+ new_block_counts = context.request_kv_block_counts[absolute_indices]
+
+ # Update request_last_kv_block_id to point to the previous block
+ # and clear the released block entry in request_to_kv_block_ids
+ # Vectorized implementation using advanced indexing:
+ # Note: new_block_counts is guaranteed to be > 0 for all requests here, since
+ # crossing back to a previous block implies the request had at least 2 blocks.
+
+ # Update request_last_kv_block_id to point to the previous block (at index new_count - 1)
+ context.request_last_kv_block_id[absolute_indices] = context.request_to_kv_block_ids[
+ absolute_indices, new_block_counts - 1
+ ]
+
+ # Clear the released block entry (at index new_count, which was the old last block)
+ context.request_to_kv_block_ids[absolute_indices, new_block_counts] = -1
+
+ # Release the blocks back to the allocator
+ context.kv_block_allocator.release_memory_blocks(blocks_to_release)
+
+ # Mamba speculative rewind state update
+ if context.is_hybrid_model:
+ active_mamba_indices = context.mamba_metadata.request_to_mamba_state_idx[
+ active_request_slice
+ ]
+ is_decode_mask = context.request_in_prefill_status_tensor[active_request_slice] == 0
+ decode_mamba_indices = active_mamba_indices[is_decode_mask]
+ accepted_tokens_per_decode_request = accepted_tokens_per_request[is_decode_mask]
+
+ if decode_mamba_indices.numel() > 0:
+ context.mamba_conv_states[:, decode_mamba_indices] = (
+ context.mamba_intermediate_conv_states[
+ :, decode_mamba_indices, accepted_tokens_per_decode_request
+ ]
+ )
+ context.mamba_ssm_states[:, decode_mamba_indices] = (
+ context.mamba_intermediate_ssm_states[
+ :, decode_mamba_indices, accepted_tokens_per_decode_request
+ ]
+ )
+
+ def _sample_from_logits_2d(self, logits_2d: Tensor) -> Tensor:
+ """Sample tokens from 2D logits using existing sampling parameters.
+
+ Args:
+ logits_2d (Tensor): Logits of shape [num_requests, vocab_size].
+
+ Returns:
+ Tensor: Sampled tokens of shape [num_requests].
+ """
+ spec_token_list = []
+ indices_list = []
+ for request_indices, temp, top_k, top_p in self._torch_sampling_buckets:
+ request_indices_tensor = torch.tensor(
+ request_indices, device=logits_2d.device, dtype=torch.long
+ )
+ spec_token_list.append(
+ self._torch_sampling_func(logits_2d[request_indices_tensor, :], temp, top_k, top_p)
+ )
+ indices_list.append(request_indices_tensor)
+
+ spec_tokens = torch.empty(logits_2d.shape[0], device=logits_2d.device, dtype=torch.int64)
+ for tokens, indices in zip(spec_token_list, indices_list):
+ spec_tokens[indices] = tokens
+ return spec_tokens
+
+ def _compute_serial_mtp_and_sample(self):
+ """Compute MTP logits serially after verification and sample speculative tokens.
+
+ This ensures that MTP predictions are always conditioned on verified tokens.
+ Each MTP depth receives the correctly sampled token from the previous depth
+ (or the base token for depth 0) rather than stale speculative tokens from
+ the previous step.
+ """
+ context = self.inference_wrapped_model.inference_context
+ active_request_count = context.total_request_count - context.paused_request_count
+ active_slice = slice(context.paused_request_count, context.total_request_count)
+
+ unwrapped_model = unwrap_model(self.inference_wrapped_model.model)
+
+ # On non-last pipeline stages, the model won't have decoder hidden states.
+ has_mtp = is_pipeline_last_stage(self.pp_group) and hasattr(
+ unwrapped_model, '_decoder_hidden_states_cache'
+ )
+
+ if has_mtp:
+ # Get decoder hidden states at last accepted positions.
+ hidden_states = unwrapped_model._decoder_hidden_states_cache
+ last_accepted_hidden = hidden_states[self._last_accepted_seq_indices, :, :]
+ # Shape: [active_request_count, 1, hidden_size]
+ else:
+ last_accepted_hidden = None
+
+ # Compute position IDs for the next tokens.
+ # After rewind, request_kv_length_offsets has been adjusted. The actual
+ # KV cache length is: adjusted_offset + processed_tokens.
+ # The next position to predict starts at that cache length.
+ adjusted_offsets = context.request_kv_length_offsets[active_slice]
+ processed_tokens = context.request_query_lengths[active_slice]
+ base_position = adjusted_offsets + processed_tokens
+
+ # Start with the freshly sampled base token.
+ next_token_ids = self._sampled_tokens_cuda[:active_request_count].clone()
+ current_hidden = last_accepted_hidden if has_mtp else None
+
+ num_depths = min(self.num_speculative_tokens, self.num_mtp_heads)
+ for depth in range(num_depths):
+ position_ids = (base_position + depth).unsqueeze(0) # [1, active_request_count]
+ token_ids = next_token_ids.unsqueeze(0) # [1, active_request_count]
+
+ mtp_logits_2d = None
+ if has_mtp:
+ current_hidden, mtp_logits = unwrapped_model.compute_mtp_single_step(
+ hidden_states=current_hidden,
+ next_token_ids=token_ids,
+ position_ids=position_ids,
+ depth=depth,
+ )
+ # mtp_logits: [active_request_count, 1, vocab_size]
+ mtp_logits_2d = mtp_logits.squeeze(1) # [active_request_count, vocab_size]
+
+ # Broadcast MTP logits across pipeline stages.
+ if self.model_is_pipeline_parallel:
+ mtp_logits_2d = broadcast_from_last_pipeline_stage(
+ [active_request_count, self.vocab_size],
+ dtype=self.model_config.params_dtype,
+ tensor=mtp_logits_2d,
+ pp_group=self.pp_group,
+ )
+
+ # Sample speculative token using the same sampling parameters.
+ spec_tokens = self._sample_from_logits_2d(mtp_logits_2d)
+ self._sampled_mtp_tokens_cuda[depth, :active_request_count] = spec_tokens
+
+ # Use sampled token as input for the next depth.
+ next_token_ids = spec_tokens
+
+ # Clean up cached hidden states.
+ if has_mtp:
+ del unwrapped_model._decoder_hidden_states_cache
+
+ def _get_required_logit_indices(
+ self,
+ request_in_prefill_status_tensor: Tensor,
+ request_query_lengths: Tensor,
+ num_decode_requests: int,
+ num_prefill_requests: int,
+ device: torch.device,
+ ) -> Tensor:
+ """Get indices into the logits tensor for tokens that need sampling.
+
+ For decode requests, all tokens (base + speculative) are needed.
+ For prefill requests, only the last token logits are needed.
+ Decode requests will always be on the left, followed by prefill requests.
+
+ Example with 5 requests (2 spec tokens):
+ Assume input ids : [ a5 a6s a7s | b3 b4s b5s | c6 c7s c8s | d1 d2 | e1 e2 e3 e4]
+ Request to prefill [ 0 | 0 | 0 | 1 | 1 ]
+ Request query lengths [ 3 | 3 | 3 | 2 | 4 ]
+ OUTPUT : required_logit_indices [ 0 1 2 | 3 4 5 | 6 7 8 | 10 | 14 ]
+
+ Returns:
+ Tensor: Indices into the sequence dimension of the logits tensor.
+ """
+ decode_request_indices = torch.arange(
+ num_decode_requests * (self.num_speculative_tokens + 1), device=device
+ )
+ prefill_request_indices = (
+ request_query_lengths.cumsum(dim=0)[request_in_prefill_status_tensor == 1] - 1
+ ) # Last token indices for prefill requests
+ required_logit_indices = torch.cat([decode_request_indices, prefill_request_indices])
+ assert (
+ len(required_logit_indices)
+ == num_decode_requests * (self.num_speculative_tokens + 1) + num_prefill_requests
+ ), (
+ f"Expected length of required_logit_indices to be "
+ f"num_decode_requests * (self.num_speculative_tokens + 1) + num_prefill_requests, "
+ f"but got {len(required_logit_indices)} for num_decode_requests {num_decode_requests} "
+ f"and num_prefill_requests {num_prefill_requests}"
+ )
+ return required_logit_indices
+
+ def _sample_speculative_logits(
+ self, required_logits: Tensor, request_in_prefill_status_tensor: Tensor
+ ) -> tuple:
+ """Sample tokens from logits using sampling buckets.
+
+ For torch sampling buckets: [request_indices, temp, top_k, top_p]
+
+ Example with 5 requests:
+ token_to_request_idx : [ 0 0 0 | 1 1 1 | 2 2 2 | 3 | 4 ]
+ required_logits : [ a5l a6l a7l | b3l b4l b5l | c6l c7l c8l | d2l | e4l ] # Shape [11, vocab_size]
+
+ Sampling buckets: [[[0,2], temp1, top_k1, top_p1], [[1], temp3, top_k3, top_p3], [[3, 4], temp2, top_k2, top_p2]]
+
+ Final output tokens : [a5s a6s a7s c6s c7s c8s b3s b4s b5s d2s e4s] # Shape [11]
+ (Rearranged from sampling bucket order back to input order using token_order)
+
+ Returns:
+ tuple: (output_tokens, repeats) where output_tokens has shape [total_required_tokens]
+ """
+ repeats = torch.where(
+ request_in_prefill_status_tensor == 0, 1 + self.num_speculative_tokens, 1
+ )
+ token_to_request_index = torch.repeat_interleave(
+ torch.arange(
+ len(request_in_prefill_status_tensor),
+ device=request_in_prefill_status_tensor.device,
+ ),
+ repeats,
+ )
+
+ output_tokens_jumbled_list = []
+ token_order_list = []
+
+ for request_indices, temp, top_k, top_p in self._torch_sampling_buckets:
+ request_indices_tensor = torch.tensor(
+ request_indices, device=token_to_request_index.device
+ )
+ required_indices = torch.where(
+ torch.isin(token_to_request_index, request_indices_tensor)
+ )[0]
+ output_tokens_jumbled_list.append(
+ self._torch_sampling_func(required_logits[required_indices, :], temp, top_k, top_p)
+ )
+ token_order_list.append(required_indices)
+
+ output_tokens_jumbled = torch.cat(output_tokens_jumbled_list, dim=0)
+ output_tokens = torch.empty(
+ len(output_tokens_jumbled),
+ device=output_tokens_jumbled.device,
+ dtype=output_tokens_jumbled.dtype,
+ )
+ token_order = torch.cat(token_order_list, dim=0)
+ # Rearrange output tokens from sampling_bucket request order back to input ids order
+ output_tokens[token_order] = output_tokens_jumbled
+
+ return output_tokens, repeats
+
+ def _verify_speculative_tokens(
+ self,
+ output_tokens: Tensor,
+ input_tokens_required: Tensor,
+ request_in_prefill_status_tensor: Tensor,
+ repeats: Tensor,
+ num_decode_requests: int,
+ num_prefill_requests: int,
+ active_request_count: int,
+ ) -> tuple:
+ """Verify speculative tokens against input tokens and compute acceptance.
+
+ Creates an accepted tokens mask where:
+ - For prefill requests, the token is always accepted.
+ - For decode requests, the first token (base token) is always accepted, then we compare
+ sampled tokens with input tokens and accept consecutive matches.
+ Then finds the index of the last accepted token per request.
+
+ Example (assume 1, 2, and 0 spec tokens are accepted in the first 3 decode requests):
+ input_tokens_required: [ a5 a6s a7s | b3 b4s b5s | c6 c7s c8s | d2 | e4 ] # Size 11
+ Output tokens [ a6o a7o a8o | b40 b5o b6o | c7o c8o c9o | d3o | e5o ]
+ Output tokens right shift [ d3o a6o a7o | a8o b40 b5o | b6o c7o c8o | c9o | d3o ]
+ Accepted tokens mask [ 1 1 0 | 1 1 1 | 1 0 0 | 1 | 1 ]
+ Last one indices [ 1 | 5 | 6 | 9 | 10 ]
+
+ Returns:
+ tuple: (last_one_indices, accepted_tokens_mask, input_tokens_required) where
+ last_one_indices contains the index of the last accepted token per request.
+ """
+ if input_tokens_required.ndim == 2:
+ assert (
+ input_tokens_required.shape[0] == 1
+ ), f"Expected input_tokens_required to have 1 row, but got {input_tokens_required.shape}"
+ input_tokens_required = input_tokens_required.squeeze(0)
+
+ # Initialize mask with False to prevent boundary bleed
+ accepted_tokens_mask = torch.zeros_like(input_tokens_required, dtype=torch.bool)
+
+ # Make all prefill tokens accepted
+ token_to_prefill_idx = torch.repeat_interleave(request_in_prefill_status_tensor, repeats)
+ accepted_tokens_mask[token_to_prefill_idx == 1] = True
+
+ # Safe decode token verification without cross-batch boundary contamination
+ decode_mask_2d = None
+ if num_decode_requests > 0:
+ decode_len = num_decode_requests * (self.num_speculative_tokens + 1)
+
+ decode_inputs = input_tokens_required[:decode_len].reshape(
+ num_decode_requests, self.num_speculative_tokens + 1
+ )
+ decode_outputs = output_tokens[:decode_len].reshape(
+ num_decode_requests, self.num_speculative_tokens + 1
+ )
+
+ # Shift outputs right by 1 *within* each request to align sampled tokens with input targets
+ decode_outputs_shifted = decode_outputs.roll(1, dims=1)
+ decode_mask_2d = decode_inputs == decode_outputs_shifted
+ # The first token (base token) is always accepted
+ decode_mask_2d[:, 0] = True
+ # Enforce consecutive acceptance: cummin propagates False to the right
+ decode_mask_2d = decode_mask_2d.cummin(dim=1).values
+ accepted_tokens_mask[:decode_len] = decode_mask_2d.flatten()
+
+ last_one_indices = torch.full(
+ (active_request_count,), -1, device=input_tokens_required.device
+ )
+
+ if num_decode_requests > 0:
+ # Summing the consecutive mask gives the count; subtract 1 for the local index
+ local_last_indices = decode_mask_2d.sum(dim=1) - 1
+ row_offsets = torch.arange(num_decode_requests, device=last_one_indices.device) * (
+ self.num_speculative_tokens + 1
+ )
+ last_one_indices[:num_decode_requests] = row_offsets + local_last_indices
- # Store the buckets and their equivalence class representatives.
- self._torch_sampling_buckets = (
- (indices, temp[rep], top_k[rep], top_p[rep]) for indices, rep in bucket_map.values()
+ if num_prefill_requests > 0:
+ decode_len = num_decode_requests * (self.num_speculative_tokens + 1)
+ prefill_valid = (
+ torch.nonzero(accepted_tokens_mask[decode_len:]).squeeze(-1) + decode_len
)
+ last_one_indices[num_decode_requests:] = prefill_valid
+
+ return last_one_indices, accepted_tokens_mask, input_tokens_required
+
+ def _dynamic_step_sample_logits_and_verify_tokens(self, logits: Tensor, input_ids: Tensor):
+ """
+ Sample tokens from logits for dynamic batching with speculative tokens and verify the tokens.
+ """
+ context = self.inference_wrapped_model.inference_context
+ active_request_count = context.total_request_count - context.paused_request_count
+
+ request_in_prefill_status_tensor = context.request_in_prefill_status_tensor[
+ context.paused_request_count : context.total_request_count
+ ]
+ request_query_lengths = context.request_query_lengths[
+ context.paused_request_count : context.total_request_count
+ ]
+
+ num_prefill_requests = request_in_prefill_status_tensor.sum().item()
+ num_decode_requests = active_request_count - num_prefill_requests
+
+ # Get the logit indices for tokens that need sampling.
+ required_logit_indices = self._get_required_logit_indices(
+ request_in_prefill_status_tensor,
+ request_query_lengths,
+ num_decode_requests,
+ num_prefill_requests,
+ logits.device,
+ )
+
+ required_logits = logits.squeeze(0)[
+ required_logit_indices, :
+ ] # Shape [num_required, vocab_size]
+
+ # Sample tokens from logits
+ output_tokens, repeats = self._sample_speculative_logits(
+ required_logits, request_in_prefill_status_tensor
+ )
+
+ # Verify speculative tokens against input tokens.
+ input_tokens_required = input_ids[0, required_logit_indices]
+ last_one_indices, accepted_tokens_mask, input_tokens_required = (
+ self._verify_speculative_tokens(
+ output_tokens,
+ input_tokens_required,
+ request_in_prefill_status_tensor,
+ repeats,
+ num_decode_requests,
+ num_prefill_requests,
+ active_request_count,
+ )
+ )
+
+ # Store the final sampled tokens for the next forward pass.
+ final_sampled_tokens = output_tokens[last_one_indices]
+ self._sampled_tokens_cuda[: len(final_sampled_tokens)] = final_sampled_tokens
+
+ # Store the last accepted positions in the packed sequence for serial
+ # MTP computation after verification.
+ self._last_accepted_seq_indices = required_logit_indices[last_one_indices]
+
+ # Extract accepted tokens and counts for decode requests.
+ # For prefill it is always set to 1. For decode, the first token is always accepted,
+ # then we compare with input tokens and accept the next tokens if its a match.
+ #
+ # Example (continuing from above):
+ # input_tokens_required: [ a5 a6s a7s | b3 b4s b5s | c6 c7s c8s | d2 | e4 ]
+ # Accepted tokens mask [ 1 1 0 | 1 1 1 | 1 0 0 | 1 | 1 ]
+ # Accepted tokens [ [a6s -1] | [b4s b5s] | [-1 -1] ] # Only decode requests (prefill defaults to -1)
+ # Accepted token counts [ 1 | 2 | 0 ] # Prefill defaults to 0
+ input_tokens_required[accepted_tokens_mask == 0] = -1 # Mask out non-accepted tokens
+ input_tokens_decode_mode = input_tokens_required[
+ : num_decode_requests * (self.num_speculative_tokens + 1)
+ ]
+ input_tokens_reshaped = input_tokens_decode_mode.reshape(
+ -1, self.num_speculative_tokens + 1
+ ) # shape: [num_decode_requests, num_speculative_tokens + 1]
+
+ # Skip the first token of every decode request (i.e a5, b3, c6)
+ accepted_tokens = input_tokens_reshaped[:, 1:]
+ self._accepted_tokens_per_request[: accepted_tokens.shape[0], :] = accepted_tokens
+ self._accepted_token_counts_per_request = (self._accepted_tokens_per_request != -1).sum(
+ dim=1
+ )
def _dynamic_step_sample_logits(self, logits: Tensor):
"""Sample tokens from logits for dynamic batching.
@@ -639,27 +1164,34 @@ def _dynamic_step_sample_logits(self, logits: Tensor):
# Last token logits.
context = self.inference_wrapped_model.inference_context
- if context.materialize_only_last_token_logits:
+ if context.config.materialize_only_last_token_logits:
# When materialize_only_last_token_logits is true, last_token_logits is
# already called in the forward pass of GPT.
- last_token_logits = logits.squeeze(0)
+ required_token_logits = logits.squeeze(0)
else:
- last_token_logits = context.last_token_logits(logits)
+ # todo : Should do verification here and get approrpiate las token logits
+ required_token_logits = context.last_token_logits(logits)
if self._sampling_backend == "torch":
# Concatenate the outputs once to prevent repeated small writes.
token_list = []
indices_list = []
+ # e.g torch sample buckets will be
+ # i.e (for all unique comibnation of t, topk, topk what are the associated
+ # requests indices (based on the active slices)
+ # [ [req at index 0, req at index 2], t1, topk1, topp1 ]]
+ # [ [req at index 1, req at index 3, req at index 4] , t2, topk2, topp2]
for indices, temp, top_k, top_p in self._torch_sampling_buckets:
token_list.append(
- self._torch_sampling_func(last_token_logits[indices, :], temp, top_k, top_p)
+ self._torch_sampling_func(required_token_logits[indices, :], temp, top_k, top_p)
)
indices_list.append(torch.tensor(indices))
# Single write to the output tensor.
sampled_tokens = torch.cat(token_list, dim=0)
sampled_indices = torch.cat(indices_list, dim=0)
+
self._sampled_tokens_cuda[sampled_indices] = sampled_tokens
def _dynamic_step_log_probs_bookkeeping(self) -> Tuple[bool, bool]:
@@ -676,6 +1208,66 @@ def _dynamic_step_log_probs_bookkeeping(self) -> Tuple[bool, bool]:
return return_log_probs.any(), top_n_log_probs.any()
+ def _router_record_bookkeeping(self) -> Optional[Dict[int, Tensor]]:
+ """Collect and map routing indices per request for MoE router recording.
+
+ This method retrieves recorded routing decisions and maps them to individual
+ requests using the context's request_ids and query_lengths. Uses the context's
+ routing_metadata when available (which handles CUDA graph static buffers automatically).
+ Must be called while context attributes are still valid (before request transitions).
+
+ Returns:
+ Optional[Dict[int, Tensor]]: A dictionary mapping request_id to a tensor of
+ shape [num_tokens, num_layers, topk]. Returns None if routing replay is
+ disabled or no routing data was recorded.
+ """
+ config = self.inference_wrapped_model.model.config
+ if not config.moe_enable_routing_replay:
+ return None
+
+ # Get routing indices - use routing_metadata if available (handles CUDA graph static buffers)
+ context = self.inference_wrapped_model.inference_context
+ if context.moe_routing_metadata is None:
+ return None
+
+ stacked_routing = context.moe_routing_metadata.get_routing_indices()
+
+ if stacked_routing is None:
+ return None
+
+ # Get active request info from context
+ active_request_slice = slice(context.paused_request_count, context.total_request_count)
+ active_request_ids = context.request_ids[active_request_slice].tolist()
+ active_query_lengths = context.request_query_lengths[active_request_slice].tolist()
+ active_token_count = context.active_token_count
+
+ # Get TP group for all-gather if using sequence parallelism
+ # With sequence parallelism, each TP rank only sees a portion of the tokens,
+ # so we need to gather routing indices across all TP ranks.
+ tp_group = self.inference_wrapped_model.tp_group
+ tp_size = get_pg_size(tp_group)
+
+ # All-gather across TP group if using sequence parallelism (tp_size > 1)
+ if tp_size > 1 and get_model_config(self.inference_wrapped_model.model).sequence_parallel:
+ # gather_from_sequence_parallel_region gathers along dim 0
+ # [local_token_count, num_layers, topk] -> [global_token_count, num_layers, topk]
+ stacked_routing = gather_from_sequence_parallel_region(stacked_routing, group=tp_group)
+
+ # Slice to real tokens (remove CUDA padding)
+ stacked_routing = stacked_routing[:active_token_count]
+
+ # Split by request along token dimension
+ # stacked_routing has shape [active_token_count, num_layers, topk]
+ routing_splits = stacked_routing.split(active_query_lengths, dim=0)
+
+ # Map to request IDs
+ routing_indices_per_request = {}
+ for req_id, routing_split in zip(active_request_ids, routing_splits):
+ # routing_split has shape [num_tokens_for_request, num_layers, topk]
+ routing_indices_per_request[req_id] = routing_split
+
+ return routing_indices_per_request
+
def _dynamic_step_calculate_log_probs(self, logits: Tensor) -> Optional[Tensor]:
"""Calculate log probs from logits."""
context = self.inference_wrapped_model.inference_context
@@ -684,9 +1276,204 @@ def _dynamic_step_calculate_log_probs(self, logits: Tensor) -> Optional[Tensor]:
return context.calculate_log_probs(
logits,
self._sampled_tokens_cuda[:active_request_count],
- only_last_token_logits=context.materialize_only_last_token_logits,
+ only_last_token_logits=context.config.materialize_only_last_token_logits,
)
+ def _dynamic_step_calculate_log_probs_speculative(
+ self, logits: Tensor
+ ) -> Tuple[List[List[float]], Tensor]:
+ """Calculate log probs from logits for speculative decoding.
+
+ For decode requests, computes log probs for each accepted speculative token
+ and the newly sampled token using the main model logits. For prefill requests,
+ handles prompt log probs the same way as non-speculative decoding.
+
+ The main model logits at position j predict the token at position j+1. So:
+ - log_prob(accepted_token[j]) comes from logits at position j
+ - log_prob(newly_sampled_token) comes from logits at position accepted_count
+
+ Args:
+ logits (Tensor): The main model logits [1, seq_len, vocab_size].
+
+ Returns:
+ Tuple of (log_probs_list, log_probs_tensor):
+ log_probs_list: List of lists, one per active request, containing
+ log probs for the tokens emitted in this step.
+ log_probs_tensor: Full log_softmax tensor for top-n computation.
+ """
+ context = self.inference_wrapped_model.inference_context
+ active_request_count = context.total_request_count - context.paused_request_count
+
+ request_in_prefill_status_tensor = context.request_in_prefill_status_tensor[
+ context.paused_request_count : context.total_request_count
+ ]
+ request_query_lengths = context.request_query_lengths[
+ context.paused_request_count : context.total_request_count
+ ]
+
+ num_prefill_requests = request_in_prefill_status_tensor.sum().item()
+ num_decode_requests = active_request_count - num_prefill_requests
+
+ logits_squeezed = logits.squeeze(0).float()
+ log_probs_tensor = F.log_softmax(logits_squeezed[: context.active_token_count], dim=-1)
+
+ log_probs_list_decode = []
+
+ if num_decode_requests > 0:
+ decode_len = num_decode_requests * (self.num_speculative_tokens + 1)
+ decode_log_probs = log_probs_tensor[:decode_len].reshape(
+ num_decode_requests, self.num_speculative_tokens + 1, -1
+ )
+ accepted_counts = self._accepted_token_counts_per_request[:num_decode_requests]
+
+ # Build a [num_decode, num_spec+1] token ID matrix for gathering.
+ # Columns 0..num_spec-1 hold accepted speculative tokens (clamped to 0
+ # where rejected, since those positions will be masked out).
+ # At column accepted_count[i], place the newly sampled token.
+ gather_tokens = torch.zeros(
+ num_decode_requests,
+ self.num_speculative_tokens + 1,
+ device=logits.device,
+ dtype=torch.long,
+ )
+ gather_tokens[:, : self.num_speculative_tokens] = self._accepted_tokens_per_request[
+ :num_decode_requests
+ ].clamp(min=0)
+ gather_tokens[
+ torch.arange(num_decode_requests, device=logits.device), accepted_counts
+ ] = self._sampled_tokens_cuda[:num_decode_requests]
+
+ # Gather: [num_decode, num_spec+1]
+ gathered_log_probs = decode_log_probs.gather(2, gather_tokens.unsqueeze(-1)).squeeze(-1)
+
+ log_probs_list_decode = [
+ gathered_log_probs[i, : accepted_counts[i].item() + 1].tolist()
+ for i in range(num_decode_requests)
+ ]
+
+ log_probs_list_prefill = []
+ if num_prefill_requests > 0:
+ decode_len = num_decode_requests * (self.num_speculative_tokens + 1)
+ prefill_log_probs = log_probs_tensor[decode_len:]
+
+ prefill_token_ids = context.token_to_input_ids[
+ decode_len : context.active_token_count
+ ].roll(-1, 0)
+ prefill_query_lengths = request_query_lengths[request_in_prefill_status_tensor == 1]
+ new_token_idx = prefill_query_lengths.cumsum(0) - 1
+ prefill_new_tokens = self._sampled_tokens_cuda[num_decode_requests:active_request_count]
+ prefill_token_ids[new_token_idx] = prefill_new_tokens
+
+ prefill_token_count = context.active_token_count - decode_len
+ seq_idx = torch.arange(prefill_token_count, device=logits.device)
+ selected_log_probs = prefill_log_probs[seq_idx, prefill_token_ids]
+
+ prefill_log_probs_split = selected_log_probs.cpu().split(
+ prefill_query_lengths.tolist(), dim=0
+ )
+ log_probs_list_prefill = [lp.tolist() for lp in prefill_log_probs_split]
+
+ log_probs_list = log_probs_list_decode + log_probs_list_prefill
+
+ return log_probs_list, log_probs_tensor
+
+ def _dynamic_step_calculate_top_n_logprobs_speculative(
+ self, log_probs_tensor: Tensor
+ ) -> Optional[Dict[int, List[Tuple[Tensor, Tensor]]]]:
+ """Calculate top-n log probs for speculative decoding.
+
+ For decode requests, computes top-n at each position that produced an
+ emitted token (accepted speculative positions + the newly sampled position).
+ For prefill requests, behaves identically to the non-speculative path.
+
+ Args:
+ log_probs_tensor (Tensor): Pre-computed log_softmax tensor from
+ _dynamic_step_calculate_log_probs_speculative.
+
+ Returns:
+ A dictionary mapping request_idx to list of (top_n_values, top_n_indices)
+ tuples, one per emitted token position.
+ """
+ context = self.inference_wrapped_model.inference_context
+ active_request_count = context.total_request_count - context.paused_request_count
+ active_request_slice = slice(context.paused_request_count, context.total_request_count)
+
+ request_in_prefill_status_tensor = context.request_in_prefill_status_tensor[
+ context.paused_request_count : context.total_request_count
+ ]
+ request_query_lengths = context.request_query_lengths[
+ context.paused_request_count : context.total_request_count
+ ]
+
+ num_prefill_requests = request_in_prefill_status_tensor.sum().item()
+ num_decode_requests = active_request_count - num_prefill_requests
+
+ top_n_results = {}
+
+ if num_decode_requests > 0:
+ decode_len = num_decode_requests * (self.num_speculative_tokens + 1)
+ decode_log_probs = log_probs_tensor[:decode_len].reshape(
+ num_decode_requests, self.num_speculative_tokens + 1, -1
+ )
+ accepted_counts = self._accepted_token_counts_per_request[:num_decode_requests]
+ top_n_per_request = self._request_metadata["top_n_logprobs"][active_request_slice][
+ :num_decode_requests
+ ]
+ max_top_n = int(top_n_per_request.max().item())
+
+ if max_top_n > 0:
+
+ # Single batched topk on GPU: [num_decode, num_spec+1, max_top_n]
+ topk_results = torch.topk(decode_log_probs, k=max_top_n, dim=-1)
+
+ # Single CPU transfer instead of O(num_decode * num_spec) transfers
+ topk_values_cpu = topk_results.values.cpu()
+ topk_indices_cpu = topk_results.indices.cpu()
+
+ for i in range(num_decode_requests):
+ top_n = int(top_n_per_request[i].item())
+ if top_n > 0:
+ num_valid = accepted_counts[i].item() + 1
+ top_n_results[i] = [
+ (topk_values_cpu[i, j, :top_n], topk_indices_cpu[i, j, :top_n])
+ for j in range(num_valid)
+ ]
+
+ if num_prefill_requests > 0:
+ decode_len = num_decode_requests * (self.num_speculative_tokens + 1)
+ prefill_log_probs = log_probs_tensor[decode_len:]
+ prefill_query_lengths = request_query_lengths[request_in_prefill_status_tensor == 1]
+ prefill_log_probs_per_request = prefill_log_probs.split(
+ prefill_query_lengths.tolist(), dim=0
+ )
+
+ for i in range(num_prefill_requests):
+ req_idx = num_decode_requests + i
+ top_n = int(
+ self._request_metadata["top_n_logprobs"][active_request_slice][req_idx].item()
+ )
+ if top_n > 0:
+ request_lp = prefill_log_probs_per_request[i]
+ skip_prompt = bool(
+ self._request_metadata["skip_prompt_log_probs"][req_idx].item()
+ )
+
+ if skip_prompt and request_lp.size(0) > 1:
+ top_n_logits = torch.topk(request_lp[-1], k=top_n)
+ top_n_results[req_idx] = [
+ (top_n_logits.values.cpu(), top_n_logits.indices.cpu())
+ ]
+ else:
+ top_n_logits = torch.topk(request_lp, k=top_n, dim=-1)
+ top_n_values_cpu = top_n_logits.values.cpu()
+ top_n_indices_cpu = top_n_logits.indices.cpu()
+ top_n_results[req_idx] = [
+ (top_n_values_cpu[t], top_n_indices_cpu[t])
+ for t in range(request_lp.size(0))
+ ]
+
+ return top_n_results if top_n_results else None
+
def _dynamic_step_calculate_top_n_logprobs(
self, logits: Tensor, log_probs_tensor: Optional[Tensor] = None
) -> Optional[Dict[int, List[Tuple[Tensor, Tensor]]]]:
@@ -712,7 +1499,7 @@ def _dynamic_step_calculate_top_n_logprobs(
active_request_slice = slice(context.paused_request_count, context.total_request_count)
# Handle decode-only mode (only last token)
- if context.materialize_only_last_token_logits or context.is_decode_only():
+ if context.config.materialize_only_last_token_logits or context.is_decode_only():
# In decode mode or when only last token logits are materialized,
# logits already represent only the last tokens
log_probs = log_probs_tensor[:active_request_count]
@@ -773,21 +1560,24 @@ def _dynamic_step_calculate_top_n_logprobs(
def dummy_forward(self):
"""Perform a dummy forward pass. This is used in expert model parallelism
- on ranks that do not have any real requests."""
+ on ranks that do not have any real requests. It may run in eager mode."""
context = self.inference_wrapped_model.inference_context
# if no cuda graphs, directly use dummy forward
if not context.cuda_graph_batch_dimensions_list:
- return self.inference_wrapped_model.dummy_forward()
+ self.inference_wrapped_model.dummy_forward()
+
+ # Disable MoE padding for MTP computation
+ if self.model_config.moe_pad_experts_for_cuda_graph_inference:
+ unwrapped_model = unwrap_model(self.inference_wrapped_model.model)
+ set_decode_expert_padding(unwrapped_model, False)
+
+ self._dummy_serial_mtp_forward()
+
+ return
# attempt to use cuda-graph if possible
- # here we try to reuse the cuda-graph warmup code to run
- # a dummy cuda graph.
- input_ids, position_ids = self._dynamic_step_context_init(
- # try to use the smallest cuda-graph config for dummy forward
- construct_graph_dimensions=min(context.cuda_graph_batch_dimensions_list),
- is_dummy_forward=True,
- )
+ input_ids, position_ids = self._dynamic_step_context_init(is_dummy_forward=True)
# _dynamic_step_context_init tries to find a cuda-graph that is compatible
# with all EP ranks. It can also return no match, in which case
@@ -799,8 +1589,80 @@ def dummy_forward(self):
else:
# fallback to eager dummy forward
self.inference_wrapped_model.dummy_forward()
+
+ # Disable MoE padding for MTP computation
+ if self.model_config.moe_pad_experts_for_cuda_graph_inference:
+ unwrapped_model = unwrap_model(self.inference_wrapped_model.model)
+ set_decode_expert_padding(unwrapped_model, False)
+
+ # When speculative decoding is active, the real EP ranks perform serial
+ # MTP forward passes after the main forward pass. MTP layers may contain
+ # MoE sublayers (inherited from the decoder spec), which require EP
+ # all-to-all collectives. The dummy rank must participate in these
+ # collectives to avoid a hang.
+ self._dummy_serial_mtp_forward()
+
+ # clear the context of any temporary state from the dummy forward
context.reset()
+ @torch.inference_mode()
+ def _dummy_serial_mtp_forward(self):
+ """Run dummy MTP forward passes to participate in EP collectives.
+
+ When speculative decoding is active and MTP layers contain MoE sublayers
+ (inherited from the decoder layer spec), each serial MTP step triggers
+ EP all-to-all collectives. The dummy EP rank must issue matching
+ collective calls so the real ranks do not hang.
+
+ This mirrors the structure of ``_compute_serial_mtp_and_sample``:
+ - On the last PP stage (where MTP resides): run ``compute_mtp_single_step``
+ with dummy tensors so the MoE all-to-all is executed.
+ - When PP > 1: participate in the ``broadcast_from_last_pipeline_stage``
+ that the real ranks also perform.
+ """
+ if self.num_speculative_tokens == 0 or self.num_mtp_heads == 0:
+ return
+ if self.model_config.expert_model_parallel_size <= 1:
+ return
+
+ unwrapped_model = unwrap_model(self.inference_wrapped_model.model)
+
+ is_last_stage = is_pipeline_last_stage(self.pp_group)
+ has_mtp = is_last_stage and hasattr(unwrapped_model, '_decoder_hidden_states_cache')
+ if not has_mtp and not self.model_is_pipeline_parallel:
+ # No MTP on this rank and no PP broadcast to participate in.
+ return
+
+ device = torch.cuda.current_device()
+ dtype = self.model_config.params_dtype
+ hidden_size = self.model_config.hidden_size
+ num_depths = min(self.num_speculative_tokens, self.num_mtp_heads)
+
+ dummy_hidden = None
+ if has_mtp:
+ # Minimal dummy tensors — just enough to drive the MTP layer forward
+ # so that the MoE all-to-all collectives are issued.
+ dummy_hidden = torch.zeros((1, 1, hidden_size), device=device, dtype=dtype)
+ dummy_token_ids = torch.zeros((1, 1), device=device, dtype=torch.long)
+ dummy_position_ids = torch.zeros((1, 1), device=device, dtype=torch.long)
+
+ for depth in range(num_depths):
+ mtp_logits_2d = None
+ if has_mtp:
+ dummy_hidden, mtp_logits = unwrapped_model.compute_mtp_single_step(
+ hidden_states=dummy_hidden,
+ next_token_ids=dummy_token_ids,
+ position_ids=dummy_position_ids,
+ depth=depth,
+ )
+ mtp_logits_2d = mtp_logits.squeeze(1) # [1, vocab_size]
+
+ # Match the PP broadcast that real ranks do in _compute_serial_mtp_and_sample.
+ if self.model_is_pipeline_parallel:
+ broadcast_from_last_pipeline_stage(
+ [1, self.vocab_size], dtype=dtype, tensor=mtp_logits_2d, pp_group=self.pp_group
+ )
+
def _dynamic_step_context_bookkeeping(self) -> Dict[str, Tensor]:
"""Update the dynamic inference context after sampling.
@@ -823,7 +1685,13 @@ def _dynamic_step_context_bookkeeping(self) -> Dict[str, Tensor]:
# Active sequence lengths.
active_request_ids = context.request_ids[active_request_slice].long()
active_sequence_lengths = context.get_active_sequence_lengths()
- active_sequence_lengths += 1 # Account for the token we just generated
+
+ if self.num_speculative_tokens > 0:
+ active_sequence_lengths += (
+ self._accepted_token_counts_per_request[:active_request_count] + 1
+ )
+ else:
+ active_sequence_lengths += 1
max_sequence_lengths = context.get_max_sequence_lengths()
# Request finished if termination_id or length >= max_sequence_length.
@@ -847,11 +1715,19 @@ def _dynamic_step_context_bookkeeping(self) -> Dict[str, Tensor]:
)
finished_request_ids = context.request_ids[finished_idxs]
- # New sample gets updated in update_requests, so we pass in a clone
+ # Clone needed: update_requests mutates next_tokens in-place via tensor_swap,
+ # which would corrupt the reused _sampled_tokens_cuda buffer.
new_sample_copy = self._sampled_tokens_cuda[:active_request_count].clone()
# Update requests.
- update_result = context.update_requests(active_request_mask, new_sample_copy)
+ # _sampled_mtp_tokens_cuda has shape [num_speculative_tokens, max_requests]
+ if self.num_speculative_tokens > 0:
+ sampled_mtp_tokens_cuda = self._sampled_mtp_tokens_cuda[:, :active_request_count]
+ else:
+ sampled_mtp_tokens_cuda = None
+ update_result = context.update_requests(
+ active_request_mask, new_sample_copy, sampled_mtp_tokens_cuda
+ )
return {
"active_request_ids": active_request_ids,
@@ -880,18 +1756,35 @@ async def async_generate_output_tokens_dynamic_batch(
context = self.inference_wrapped_model.inference_context
active_request_count = context.total_request_count - context.paused_request_count
- # No tokens?
- if context.active_token_count == 0:
+ # No tokens and no active requests?
+ if context.active_token_count == 0 and active_request_count == 0:
return None
input_ids, position_ids = self._dynamic_step_context_init()
cuda_graph_request_count = (
- context.padded_active_request_count if context.is_decode_only() else None
+ context.padded_active_request_count if context.using_cuda_graph_this_step() else None
)
+ # Enable routing recording before forward pass if routing replay is enabled
+ config = self.inference_wrapped_model.model.config
+ if config.moe_enable_routing_replay:
+ RouterReplay.set_global_router_replay_action(RouterReplayAction.RECORD)
+
+ # Forward pass produces only base logits. When speculative decoding is
+ # active, MTP logits are computed serially after verification.
logits = self._dynamic_step_forward_logits(input_ids, position_ids)
+ # Commit Mamba intermediate states before update_requests, which
+ # may swap request indices. The Python lists tracking EOS block IDs
+ # and intermediate offsets are not swapped along with tensors, so
+ # commit must run while indices are still valid.
+ if context.is_hybrid_model and context.mamba_slot_allocator is not None:
+ context.mamba_slot_allocator.commit_intermediate_states()
+
+ # Collect routing indices per request (must be done before context transitions)
+ routing_indices_per_request = self._router_record_bookkeeping()
+
# This is the best place to yield control back to event loop.
# At this point we have enqueued FW pass GPU kernels asynchronously.
# While they are running, we can do other useful CPU work.
@@ -900,19 +1793,43 @@ async def async_generate_output_tokens_dynamic_batch(
# Todo [Siddharth]: Can we condition the sleep on a cuda event?
# NOTE [TDE]: This will be moved once CPU and GPU methods are separated.
await asyncio.sleep(0)
-
return_log_probs, return_top_n_logprobs = self._dynamic_step_log_probs_bookkeeping()
+
self._dynamic_step_sample_bookkeeping()
- self._dynamic_step_sample_logits(logits)
+
+ if self.num_speculative_tokens > 0:
+ # Phase 1: Verify speculative tokens using base logits only.
+ self._dynamic_step_sample_logits_and_verify_tokens(logits, input_ids)
+ # Phase 2: Rewind KV cache for rejected tokens.
+ self._rewind_kv_cache()
+
+ # Disable MoE padding for MTP computation
+ if self.model_config.moe_pad_experts_for_cuda_graph_inference:
+ unwrapped_model = unwrap_model(self.inference_wrapped_model.model)
+ set_decode_expert_padding(unwrapped_model, False)
+
+ # Phase 3: Compute MTP serially with correct (verified) inputs.
+ self._compute_serial_mtp_and_sample()
+ else:
+ self._dynamic_step_sample_logits(logits)
log_probs = None
top_n_logprobs = None
if return_log_probs or return_top_n_logprobs:
- log_probs, log_probs_tensor = self._dynamic_step_calculate_log_probs(logits)
- if return_top_n_logprobs:
- top_n_logprobs = self._dynamic_step_calculate_top_n_logprobs(
- logits, log_probs_tensor
+ if self.num_speculative_tokens > 0:
+ log_probs, log_probs_tensor = self._dynamic_step_calculate_log_probs_speculative(
+ logits
)
+ if return_top_n_logprobs:
+ top_n_logprobs = self._dynamic_step_calculate_top_n_logprobs_speculative(
+ log_probs_tensor
+ )
+ else:
+ log_probs, log_probs_tensor = self._dynamic_step_calculate_log_probs(logits)
+ if return_top_n_logprobs:
+ top_n_logprobs = self._dynamic_step_calculate_top_n_logprobs(
+ logits, log_probs_tensor
+ )
if skip_bookkeeping:
request_bookkeeping = {}
@@ -920,11 +1837,22 @@ async def async_generate_output_tokens_dynamic_batch(
request_bookkeeping = self._dynamic_step_context_bookkeeping()
ret = {
- "sample": self._sampled_tokens_cuda[:active_request_count],
+ # Clone needed: _sampled_tokens_cuda is a reused buffer overwritten each step.
+ "sample": self._sampled_tokens_cuda[:active_request_count].clone(),
+ "accepted_tokens": (
+ # Clone needed: .fill_(-1) on line 1480 would corrupt the returned value.
+ self._accepted_tokens_per_request.clone()
+ if self.num_speculative_tokens > 0
+ else None
+ ),
"log_probs": log_probs,
"top_n_logprobs": top_n_logprobs,
+ "routing_indices_per_request": routing_indices_per_request,
"cuda_graph_request_count": cuda_graph_request_count,
}
+ if self.num_speculative_tokens > 0:
+ self._accepted_tokens_per_request.fill_(-1)
+ self._accepted_token_counts_per_request.fill_(0)
ret.update(request_bookkeeping)
return ret
@@ -1024,9 +1952,10 @@ def generate_all_output_tokens_static_batch(
# Pad batch tokens if necessary
batch_size = len(active_requests)
max_sequence_length = max_prompt_length_in_batch + sampling_params.num_tokens_to_generate
- inference_wrapper_config = self.inference_wrapped_model.inference_wrapper_config
- inference_max_batch_size = inference_wrapper_config.inference_max_requests
- inference_max_sequence_length = inference_wrapper_config.inference_max_seq_length
+ context = self.inference_wrapped_model.inference_context
+ assert isinstance(context, StaticInferenceContext)
+ inference_max_batch_size = context.max_batch_size
+ inference_max_sequence_length = context.max_sequence_length
padded_batch_size = inference_max_batch_size if enable_cuda_graph else batch_size
if padded_batch_size > inference_max_batch_size:
raise ValueError(
@@ -1066,10 +1995,6 @@ def generate_all_output_tokens_static_batch(
batch_size, device=torch.cuda.current_device()
).cuda()
- # Use padded vocab size because tokenizer vocab size might not include padding
- # to nearest power of 2
- vocab_size = inference_wrapper_config.padded_vocab_size
-
# Check whether early termination is enabled
no_early_termination = getattr(sampling_params, "no_early_termination", False)
termination_id = -1 if no_early_termination else self.tokenizer.eod
@@ -1130,14 +2055,14 @@ def generate_all_output_tokens_static_batch(
# If using symmetric kernels and we are using using nccl
# for prefill turn off symmetric kernels
- symmetric_ar_type = model_config.symmetric_ar_type
- nccl_all_reduce_for_prefill = inference_wrapper_config.nccl_all_reduce_for_prefill
+ symmetric_ar_type = self.model_config.symmetric_ar_type
+ nccl_all_reduce_for_prefill = self.model_config.nccl_all_reduce_for_prefill
if symmetric_ar_type is not None and nccl_all_reduce_for_prefill:
unwrapped_model.set_symmetric_ar(None)
# Turning off MoE padding for prefill
moe_pad_experts_for_cuda_graph_inference = (
- inference_wrapper_config.moe_pad_experts_for_cuda_graph_inference
+ self.model_config.moe_pad_experts_for_cuda_graph_inference
)
if moe_pad_experts_for_cuda_graph_inference:
set_decode_expert_padding(unwrapped_model, False)
@@ -1191,7 +2116,7 @@ def generate_all_output_tokens_static_batch(
or not (sampling_params.return_log_probs or sampling_params.top_n_logprobs > 0)
)
inference_context = self.inference_wrapped_model.inference_context
- inference_context.materialize_only_last_token_logits = (
+ inference_context.config.materialize_only_last_token_logits = (
materialize_only_last_token_logits
)
@@ -1212,14 +2137,14 @@ def generate_all_output_tokens_static_batch(
if self.model_is_pipeline_parallel:
context_length = context_end_position - context_start_position
logits_seq_len = 1 if materialize_only_last_token_logits else context_length
- logits_shape = [batch_size, logits_seq_len, vocab_size]
+ logits_shape = [batch_size, logits_seq_len, self.vocab_size]
if is_pipeline_last_stage(self.pp_group):
assert logits is not None and torch.Size(logits_shape) == logits.shape
# TODO(ksanthanam): Evaluate whether it makes more sense to sample on 1 rank
# and then broadcast the sampled tokens rather than broadcasting the raw logits.
logits = broadcast_from_last_pipeline_stage(
- [batch_size, logits_seq_len, vocab_size],
- dtype=inference_wrapper_config.params_dtype,
+ [batch_size, logits_seq_len, self.vocab_size],
+ dtype=self.model_config.params_dtype,
tensor=logits,
pp_group=self.pp_group,
)
@@ -1248,7 +2173,7 @@ def generate_all_output_tokens_static_batch(
sampled_logits = self.sample_from_logits(
last_token_logits,
sampling_params,
- vocab_size,
+ self.vocab_size,
generation_started=generation_started,
top_n_logprobs_dict=top_n_logprobs_dict,
logits=logits_for_top_n_prompt_logprobs,
diff --git a/megatron/core/inference/text_generation_server/dynamic_text_gen_server/__init__.py b/megatron/core/inference/text_generation_server/dynamic_text_gen_server/__init__.py
index 3ad54686261..c3441253c77 100644
--- a/megatron/core/inference/text_generation_server/dynamic_text_gen_server/__init__.py
+++ b/megatron/core/inference/text_generation_server/dynamic_text_gen_server/__init__.py
@@ -1,3 +1,3 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
-from .flask_server import run_flask_server
+from .text_generation_server import start_text_gen_server, stop_text_gen_server
diff --git a/megatron/core/inference/text_generation_server/dynamic_text_gen_server/endpoints/chat_completions.py b/megatron/core/inference/text_generation_server/dynamic_text_gen_server/endpoints/chat_completions.py
index 0c3379bc53f..75faefd4b88 100644
--- a/megatron/core/inference/text_generation_server/dynamic_text_gen_server/endpoints/chat_completions.py
+++ b/megatron/core/inference/text_generation_server/dynamic_text_gen_server/endpoints/chat_completions.py
@@ -1,45 +1,559 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
import asyncio
+import json
import logging
import time
+import traceback
+import uuid
+import warnings
+from megatron.core.inference.inference_request import unwrap_serialized_tensors
from megatron.core.inference.sampling_params import SamplingParams
+from megatron.core.tokenizers.text.parsers import PARSER_MAPPING
logger = logging.getLogger(__name__)
+# pylint: disable=line-too-long
+
+_TOKEN_ID_FIELDS_TO_REDACT = {
+ "prompt_tokens",
+ "remaining_prompt_tokens",
+ "generated_tokens",
+ "prompt_token_ids",
+ "generation_token_ids",
+}
+
+_INDEX_FIELDS_TO_REDACT = {"routing_indices", "moe_topk_indices", "prompt_moe_topk_indices"}
+
+_HASH_FIELDS_TO_REDACT = {"precomputed_block_hashes"}
+
+_NUMERIC_SERIES_FIELDS_TO_REDACT = {"tpot"}
+
+
+def _is_int_list_like(value):
+ """Return True for integer lists, including nested integer lists."""
+ if not isinstance(value, list):
+ return False
+ return all(isinstance(item, int) or _is_int_list_like(item) for item in value)
+
+
+def _is_numeric_list_like(value):
+ """Return True for numeric lists, including nested numeric lists."""
+ if not isinstance(value, list):
+ return False
+ return all(isinstance(item, (int, float)) or _is_numeric_list_like(item) for item in value)
+
+
+def _redact_token_id_lists_for_logging(value):
+ """Redact verbose token-id arrays from logs."""
+ if isinstance(value, dict):
+ redacted = {}
+ for key, item in value.items():
+ if (
+ key in _TOKEN_ID_FIELDS_TO_REDACT
+ or key in _INDEX_FIELDS_TO_REDACT
+ or key in _HASH_FIELDS_TO_REDACT
+ or key.endswith("_token_ids")
+ or key.endswith("_topk_indices")
+ or key.endswith("_hashes")
+ ) and _is_int_list_like(item):
+ redacted[key] = "...truncated..."
+ elif key in _NUMERIC_SERIES_FIELDS_TO_REDACT and _is_numeric_list_like(item):
+ redacted[key] = "...truncated..."
+ else:
+ redacted[key] = _redact_token_id_lists_for_logging(item)
+ return redacted
+ if isinstance(value, list):
+ return [_redact_token_id_lists_for_logging(item) for item in value]
+ return value
+
+
+def _get_field(obj, key, default=None):
+ """Read a field from dict-like or object-like values."""
+ if isinstance(obj, dict):
+ return obj.get(key, default)
+ return getattr(obj, key, default)
+
+
+_TRANSFER_TOOL_NAME = "transfer_to_human_agents"
+_TRANSFER_HOLD_MESSAGE = "YOU ARE BEING TRANSFERRED TO A HUMAN AGENT. PLEASE HOLD ON."
+_RESERVATION_UPDATE_TOOLS = {
+ "update_reservation_flights",
+ "update_reservation_passengers",
+ "update_reservation_baggages",
+}
+_RESERVATION_DESTRUCTIVE_TOOLS = {"cancel_reservation", "book_reservation"}
+
+
+def _try_parse_jsonish(value):
+ if not isinstance(value, str):
+ return value
+ stripped = value.strip()
+ if not stripped or stripped[0] not in "[{":
+ return value
+ try:
+ return json.loads(stripped)
+ except (TypeError, ValueError):
+ return value
+
+
+def _extract_declared_types(schema):
+ """Recursively extract declared JSON-schema type names."""
+ declared = set()
+ if not isinstance(schema, dict):
+ return declared
+
+ schema_type = schema.get("type")
+ if isinstance(schema_type, str):
+ declared.add(schema_type.strip().lower())
+ elif isinstance(schema_type, list):
+ for item in schema_type:
+ if isinstance(item, str):
+ declared.add(item.strip().lower())
+
+ for combinator in ("anyOf", "oneOf", "allOf"):
+ options = schema.get(combinator)
+ if isinstance(options, list):
+ for option in options:
+ declared.update(_extract_declared_types(option))
+ return declared
+
+
+def _get_tool_argument_schemas(tools):
+ """Build function-name to argument-schema mapping from request tools."""
+ schemas = {}
+ if not isinstance(tools, list):
+ return schemas
+
+ for tool in tools:
+ function = _get_field(tool, "function", {}) or {}
+ function_name = _get_field(function, "name")
+ params = _get_field(function, "parameters", {})
+ if not isinstance(function_name, str) or not isinstance(params, dict):
+ continue
+ if isinstance(params.get("properties"), dict):
+ schemas[function_name] = params.get("properties")
+ else:
+ schemas[function_name] = params
+ return schemas
+
+
+def _normalize_structured_tool_arguments(arguments, function_name, tool_argument_schemas):
+ """Coerce structured (array/object) args from JSON strings to native types."""
+ if not isinstance(arguments, dict):
+ return arguments
+
+ function_schema = tool_argument_schemas.get(function_name, {})
+ if not isinstance(function_schema, dict):
+ return arguments
+
+ normalized = dict(arguments)
+ for key in normalized:
+ param_schema = function_schema.get(key)
+ declared_types = _extract_declared_types(param_schema)
+ if not (declared_types & {"array", "arr", "object", "dict", "list"}):
+ continue
+ parsed = _try_parse_jsonish(normalized[key])
+ if isinstance(parsed, (dict, list)):
+ normalized[key] = parsed
+ return normalized
+
+
+def _normalize_tool_calls(tool_calls, tools=None):
+ """Normalize tool calls to OpenAI-compatible JSON primitives."""
+ tool_argument_schemas = _get_tool_argument_schemas(tools)
+ normalized = []
+ for call in tool_calls or []:
+ fn = _get_field(call, "function", {}) or {}
+ fn_name = _get_field(fn, "name")
+ fn_args = _get_field(fn, "arguments", "")
+ if fn_name is None:
+ continue
+ if isinstance(fn_args, str):
+ try:
+ parsed_args = json.loads(fn_args)
+ except (TypeError, ValueError):
+ parsed_args = None
+ if isinstance(parsed_args, dict):
+ fn_args = json.dumps(
+ _normalize_structured_tool_arguments(
+ parsed_args, fn_name, tool_argument_schemas
+ ),
+ ensure_ascii=False,
+ )
+ elif isinstance(fn_args, dict):
+ fn_args = json.dumps(
+ _normalize_structured_tool_arguments(fn_args, fn_name, tool_argument_schemas),
+ ensure_ascii=False,
+ )
+ else:
+ try:
+ fn_args = json.dumps(fn_args, ensure_ascii=False)
+ except TypeError:
+ fn_args = str(fn_args)
+ normalized.append(
+ {
+ "id": str(_get_field(call, "id", f"call_{uuid.uuid4().hex[:24]}")),
+ "type": "function",
+ "function": {"name": str(fn_name), "arguments": fn_args},
+ }
+ )
+ return _apply_tool_call_guardrails(normalized)
+
+
+def _apply_tool_call_guardrails(tool_calls):
+ """Apply conservative post-parse guardrails to tool call lists.
+
+ If update-style reservation tools are already present in the same response,
+ suppress cancel+book style calls to avoid destructive replanning patterns.
+ """
+ if not isinstance(tool_calls, list):
+ return tool_calls
+
+ call_names = {
+ _get_field(_get_field(call, "function", {}), "name")
+ for call in tool_calls
+ if isinstance(call, dict)
+ }
+ if call_names & _RESERVATION_UPDATE_TOOLS:
+ return [
+ call
+ for call in tool_calls
+ if _get_field(_get_field(call, "function", {}), "name")
+ not in _RESERVATION_DESTRUCTIVE_TOOLS
+ ]
+ return tool_calls
+
+
+def _normalize_assistant_content(message_text, tool_calls):
+ """Normalize assistant content for policy-sensitive tool transitions."""
+ if not isinstance(message_text, str):
+ message_text = "" if message_text is None else str(message_text)
+
+ tool_names = {
+ _get_field(_get_field(call, "function", {}), "name")
+ for call in (tool_calls or [])
+ if isinstance(call, dict)
+ }
+ if _TRANSFER_TOOL_NAME in tool_names:
+ return _TRANSFER_HOLD_MESSAGE
+ return message_text
+
+
+def _coerce_arguments_mapping(arguments):
+ """Coerce function.arguments to a mapping for HF/Jinja chat templates.
+
+ Examples:
+ - {"x": 1} -> {"x": 1}
+ - '{"x": 1}' -> {"x": 1}
+ - "[1, 2]" -> {} # JSON parses, but not a mapping
+ - "not-json" -> {}
+ - None -> {}
+ """
+ if isinstance(arguments, dict):
+ return arguments
+ if isinstance(arguments, str):
+ try:
+ parsed = json.loads(arguments)
+ except (TypeError, ValueError):
+ return {}
+ return parsed if isinstance(parsed, dict) else {}
+ return {}
+
+
+def _sanitize_messages_for_template(messages):
+ """Prepare messages so tokenizer chat templates can safely consume them.
+
+ This only normalizes tool-call argument payloads inside each message:
+ - messages[*].tool_calls[*].function.arguments is coerced to a dict.
+
+ Example transformation:
+ Input:
+ [{"role": "assistant", "tool_calls": [{"function": {"name": "f", "arguments": "{\"x\": 1}"}}]}]
+ Output:
+ [{"role": "assistant", "tool_calls": [{"function": {"name": "f", "arguments": {"x": 1}}}]}]
+
+ Another example:
+ - arguments: "[1,2,3]" -> arguments: {}
+ """
+ if not isinstance(messages, list):
+ return messages
+ sanitized = []
+ for message in messages:
+ if not isinstance(message, dict):
+ sanitized.append(message)
+ continue
+ msg_copy = dict(message)
+ content = msg_copy.get("content")
+ # OpenAI-style multimodal/text content may arrive as a list of blocks.
+ # HF/Jinja chat templates used by this server expect plain strings.
+ if isinstance(content, list):
+ text_chunks = []
+ for chunk in content:
+ if isinstance(chunk, dict):
+ if chunk.get("type") == "text":
+ text_chunks.append(str(chunk.get("text", "")))
+ elif "text" in chunk:
+ text_chunks.append(str(chunk.get("text", "")))
+ elif isinstance(chunk, str):
+ text_chunks.append(chunk)
+ msg_copy["content"] = "".join(text_chunks)
+ elif isinstance(content, dict):
+ msg_copy["content"] = str(content.get("text", ""))
+ elif content is None:
+ msg_copy["content"] = ""
+ elif not isinstance(content, str):
+ msg_copy["content"] = str(content)
+
+ tool_calls = msg_copy.get("tool_calls")
+ if isinstance(tool_calls, list):
+ sanitized_tool_calls = []
+ for call in tool_calls:
+ if not isinstance(call, dict):
+ sanitized_tool_calls.append(call)
+ continue
+ call_copy = dict(call)
+ function = call_copy.get("function")
+ if isinstance(function, dict):
+ function_copy = dict(function)
+ function_copy["arguments"] = _coerce_arguments_mapping(
+ function_copy.get("arguments", {})
+ )
+ call_copy["function"] = function_copy
+ sanitized_tool_calls.append(call_copy)
+ msg_copy["tool_calls"] = sanitized_tool_calls
+ sanitized.append(msg_copy)
+ return sanitized
+
+
+def _sanitize_tools_for_template(tools):
+ """Ensure tools payload is template-safe and has mapping parameters.
+
+ Example transformations:
+ - {"function": {"name": "f", "parameters": "not-a-dict"}}
+ -> {"function": {"name": "f", "parameters": {"type": "object", "properties": {}}}}
+ - non-dict tool entries are dropped.
+ - non-list input returns None.
+ """
+ if not isinstance(tools, list):
+ return None
+
+ sanitized = []
+ for tool in tools:
+ if not isinstance(tool, dict):
+ continue
+ tool_copy = dict(tool)
+ function = tool_copy.get("function")
+ if isinstance(function, dict):
+ function_copy = dict(function)
+ if not isinstance(function_copy.get("parameters"), dict):
+ function_copy["parameters"] = {"type": "object", "properties": {}}
+ tool_copy["function"] = function_copy
+ sanitized.append(tool_copy)
+ return sanitized
+
+
+def _reconstruct_reasoning_content(messages: list[dict]) -> list[dict]:
+ """Reconstruct tags from reasoning_content fields on assistant messages.
+
+ For parity with vLLM, assistant messages may carry reasoning in the reasoning_content field.
+ Before applying the chat template, we must inline those tags back into content.
+ """
+ for message in messages:
+ if message.get("role") != "assistant":
+ continue
+ reasoning_content = message.pop("reasoning_content", None)
+ if reasoning_content is not None:
+ content = message.get("content") or ""
+ message["content"] = f"{reasoning_content} {content}"
+ return messages
+
+
+def _replace_prefix_tokens(
+ eos_token_id,
+ previous_turn_token_ids,
+ retokeenized_previous_turn_token_ids,
+ current_turn_token_ids,
+):
+ """Replace the token ids that are associated with the previous turn with the actual tokens
+ from the previous generation (rather than the ones from the chat template application)."""
+
+ # Strip the EOS from the previous turn token ids if it exists
+ if previous_turn_token_ids[-1] == eos_token_id:
+ previous_turn_token_ids = previous_turn_token_ids[:-1]
+
+ # Find the last EOS token id in the previous turn token ids
+ last_eos_token_id_index = len(retokeenized_previous_turn_token_ids) - 1
+ for i in reversed(range(len(retokeenized_previous_turn_token_ids))):
+ if current_turn_token_ids[i] == eos_token_id:
+ last_eos_token_id_index = i
+ break
+
+ # Replace the current turn token ids with the tokens from the previous generation
+ current_turn_additional_token_ids = current_turn_token_ids[last_eos_token_id_index:]
+
+ # Return the previous turn token ids + the current turn token ids
+ return previous_turn_token_ids + current_turn_additional_token_ids
+
+
try:
- from flask import Blueprint, current_app, jsonify, request
+ import orjson
+
+ HAVE_ORJSON = True
+except ImportError:
+ HAVE_ORJSON = False
+
+
+try:
+ from quart import Blueprint, Response, current_app, jsonify, request
bp = Blueprint('chat_completions_api', __name__)
+ def apply_parsers(message_text, tools, parsers_list, tools_requested):
+ """Runs CPU-intensive text parsing."""
+ meta = {}
+ for parser in parsers_list:
+ if parser not in PARSER_MAPPING:
+ raise ValueError(f"Parser {parser} not found in PARSER_MAPPING")
+
+ prev_text = message_text
+ parsed_text, new_info = PARSER_MAPPING[parser].parse(message_text, tools=tools)
+ if "tool_calls" in new_info:
+ new_info["tool_calls"] = _normalize_tool_calls(
+ new_info.get("tool_calls", []), tools=tools
+ )
+ if not tools_requested:
+ # Ignore incidental tool-call syntax in plain chat mode.
+ parsed_text = prev_text
+ new_info.pop("tool_calls", None)
+ message_text = parsed_text
+
+ assert not (
+ meta.keys() & new_info.keys()
+ ), "Multiple parsers found the same information."
+ meta.update(new_info)
+
+ return message_text, meta
+
@bp.route('/chat/completions', methods=['POST'])
@bp.route('/v1/chat/completions', methods=['POST'])
async def chat_completions():
"""Handles async POST requests for chat completions."""
client = current_app.config['client']
tokenizer = current_app.config['tokenizer']
+ parsers = current_app.config['parsers']
- req = request.get_json()
-
- # --- 1. Parse Messages ---
+ req = await request.get_json()
+ tools = req.get("tools", None)
+ tools_requested = bool(tools)
messages = req.get("messages")
+ chat_template_kwargs = req.get("chat_template_kwargs", {})
+ if not isinstance(chat_template_kwargs, dict):
+ logger.warning(
+ "Ignoring non-dict chat_template_kwargs: %s", type(chat_template_kwargs).__name__
+ )
+ chat_template_kwargs = {}
+ # --- 1. Parse Messages ---
if not messages:
- return "Missing 'messages' field", 400
+ return Response("Missing 'messages' field", status=400)
if not isinstance(messages, list):
- return "'messages' must be a list", 400
+ return Response("'messages' must be a list", status=400)
+ template_messages = _sanitize_messages_for_template(messages)
+ template_messages = _reconstruct_reasoning_content(template_messages)
+ template_tools = _sanitize_tools_for_template(tools)
try:
- prompt_tokens = tokenizer.apply_chat_template(
- messages, tokenize=True, add_generation_prompt=True
- )
- except AttributeError:
- return (
- "Tokenizer does not support 'apply_chat_template'. "
- "Chat completions requires a tokenizer with a configured chat template."
- ), 500
+ if (
+ hasattr(tokenizer, 'apply_chat_template')
+ and getattr(tokenizer, "chat_template", None) is not None
+ ):
+ prompt_tokens = tokenizer.apply_chat_template(
+ template_messages,
+ tokenize=True,
+ add_generation_prompt=True,
+ tools=template_tools,
+ **chat_template_kwargs,
+ )
+
+ if req.get("prevent_retokenization", True):
+ # If we are avoiding retokenization, we need to replace some prompt tokens with the prompt/generation tokens from the previous generation
+ # This improves prefix cache hits and reduces logprob variation between training and inference.
+
+ # Find the last assistant message
+ last_assistant_message_idx = None
+ for i in reversed(range(len(template_messages))):
+ if template_messages[i]["role"] == "assistant":
+ last_assistant_message_idx = i
+ break
+
+ last_assistant_message = (
+ template_messages[last_assistant_message_idx]
+ if last_assistant_message_idx is not None
+ else None
+ )
+
+ # Only proceed if the last assistant message has the token IDs from a previous generation.
+ # Dataset-provided conversation history won't have these fields.
+ if (
+ last_assistant_message is not None
+ and "prompt_token_ids" in last_assistant_message
+ and "generation_token_ids" in last_assistant_message
+ ):
+ eos_token_id = tokenizer.eos_id
+ assert eos_token_id is not None, "Your tokenizer must have an EOS token ID!"
+
+ warnings.warn(
+ "Avoiding prefix retokenization."
+ " This is a patch that ensures subsequent generations are not retokenized differently than the previous generation."
+ " This may cause unexpected behavior if messages (including system messages) are altered between generations."
+ )
+
+ messages_to_last_assistant_message = template_messages[
+ : last_assistant_message_idx + 1
+ ]
+
+ # Get the templated tokenization of just the previous generation
+ retokenized_previous_turn_token_ids = tokenizer.apply_chat_template(
+ messages_to_last_assistant_message,
+ tokenize=True,
+ add_generation_prompt=False,
+ tools=template_tools,
+ **chat_template_kwargs,
+ )
+
+ # Replace the prefix tokens with the tokens from the previous generation.
+ # If prior token IDs are unavailable, fall back to normal retokenized prompt
+ # instead of failing the request.
+ prompt_token_ids = last_assistant_message.get("prompt_token_ids")
+ generation_token_ids = last_assistant_message.get("generation_token_ids")
+
+ if isinstance(prompt_token_ids, list) and isinstance(
+ generation_token_ids, list
+ ):
+ previous_turn_token_ids = prompt_token_ids + generation_token_ids
+ prompt_tokens = _replace_prefix_tokens(
+ eos_token_id,
+ previous_turn_token_ids,
+ retokenized_previous_turn_token_ids,
+ prompt_tokens,
+ )
+ else:
+ logger.warning(
+ "Last assistant message missing prompt_token_ids/"
+ "generation_token_ids; skipping prefix replacement."
+ )
+
+ else:
+ warnings.warn(
+ "Tokenizer does not support 'apply_chat_template'. Using tokenize instead."
+ )
+ prompt_tokens = tokenizer.tokenize(
+ "\n".join([message["content"] for message in messages])
+ )
except Exception as e:
- return f"Error processing 'messages': {e}", 500
+ logger.error(f"{traceback.format_exc()}")
+ return Response(f"Error processing 'messages': {e}", status=500)
# --- 2. Parse Sampling Params ---
try:
@@ -55,6 +569,23 @@ async def chat_completions():
# Check for 'logprobs' (bool) and 'top_logprobs' (int)
return_log_probs = bool(req.get("logprobs", False))
top_n_logprobs = int(req.get("top_logprobs", 0)) if return_log_probs else 0
+ skip_prompt_log_probs = bool(req.get("skip_prompt_log_probs", True))
+ add_BOS = bool(req.get("add_BOS", False))
+
+ # The engine only handles add_BOS for string prompts, not pre-tokenized
+ # input. Since we pre-tokenize via apply_chat_template, we must handle
+ # BOS ourselves, matching the logic in tokenize_prompt().
+ if hasattr(tokenizer, 'bos') and tokenizer.bos is not None:
+ start_idx = 0
+ while start_idx < len(prompt_tokens) and prompt_tokens[start_idx] == tokenizer.bos:
+ start_idx += 1
+ if start_idx > 0:
+ prompt_tokens = prompt_tokens[start_idx:]
+
+ if add_BOS:
+ prompt_tokens = [tokenizer.bos] + prompt_tokens
+
+ max_tokens = req.get("max_completion_tokens", None) or req.get("max_tokens", None)
sampling_params = SamplingParams(
temperature=temperature,
@@ -62,89 +593,176 @@ async def chat_completions():
top_p=top_p,
return_log_probs=return_log_probs,
top_n_logprobs=top_n_logprobs,
- num_tokens_to_generate=int(req.get("max_tokens", 16)),
+ num_tokens_to_generate=(int(max_tokens) if max_tokens is not None else None),
+ skip_prompt_log_probs=skip_prompt_log_probs,
+ add_BOS=add_BOS,
)
except ValueError as e:
- return f"Invalid sampling parameter: {e}", 400
+ return Response(f"Invalid sampling parameter: {e}", status=400)
# --- 3. Send Requests to Engine ---
- # For chat, we run the *same* prompt 'n' times.
- tasks = []
- for _ in range(n):
- per_req_params = SamplingParams(
- temperature=sampling_params.temperature,
- top_k=sampling_params.top_k,
- top_p=sampling_params.top_p,
- return_log_probs=sampling_params.return_log_probs,
- top_n_logprobs=sampling_params.top_n_logprobs,
- num_tokens_to_generate=sampling_params.num_tokens_to_generate,
- )
- tasks.append(client.add_request(prompt_tokens, per_req_params))
+ tasks = [client.add_request(prompt_tokens, sampling_params) for _ in range(n)]
+
+ if current_app.config['verbose']:
+ start_time = time.perf_counter()
- start_time = time.perf_counter()
try:
batch_results = await asyncio.gather(*tasks)
except Exception as e:
- return f"Error during inference: {e}", 500
+ logger.error(f"Error during inference: {e}")
+ return Response(f"Error during inference: {e}", status=500)
- logger.info(
- f"Batch of {len(tasks)} requests (n={n}) processed in "
- f"{time.perf_counter() - start_time:.2f}s"
- )
+ if current_app.config['verbose']:
+ logging.info(
+ f"Batch of {len(tasks)} requests (n={n}) processed in "
+ f"{time.perf_counter() - start_time:.2f}s"
+ )
+
+ # --- 4. Check for failed requests ---
+ failed_errors = []
+ has_nontransient_error = False
+ for i, record in enumerate(batch_results):
+ if record.get("status") == "FAILED":
+ events = record.get("events", [])
+ error_events = [
+ e for e in events if e.get("type") in ("ERROR_NONTRANSIENT", "ERROR_TRANSIENT")
+ ]
+ if any(e.get("type") == "ERROR_NONTRANSIENT" for e in error_events):
+ has_nontransient_error = True
+ error_msg = (
+ str(error_events[-1].get("payload", "Unknown error"))
+ if error_events
+ else "Unknown error"
+ )
+ failed_errors.append(f"Request {i}: {error_msg}")
+
+ if failed_errors:
+ error_detail = "; ".join(failed_errors)
+ status = 400 if has_nontransient_error else 500
+ logger.error(f"Inference request(s) failed: {error_detail}")
+ return Response(f"Inference request(s) failed: {error_detail}", status=status)
- # --- 4. Format OpenAI Response ---
+ # --- 5. Format OpenAI Response ---
choices = []
total_completion_tokens = 0
- prompt_token_count = len(prompt_tokens) # Calculated once
+ prompt_tokens_counts = []
request_idx = 0
- for record in batch_results:
- for result in record.requests:
- text_output = result.generated_text
-
- logprobs_content = None
- if sampling_params.return_log_probs:
- token_logprobs = getattr(result, 'log_probs', [])
- tokens = [tokenizer.detokenize([tok]) for tok in result.generated_tokens]
-
- # Get top_n_logprobs if available
- generated_top_n_logprobs = getattr(result, 'generated_top_n_logprobs', None)
-
- logprobs_content = []
- for i, (tok, lp) in enumerate(zip(tokens, token_logprobs)):
- # Build top_logprobs list for this token position
- top_logprobs_list = []
- if generated_top_n_logprobs and i < len(generated_top_n_logprobs):
- top_n_dict = generated_top_n_logprobs[i]
- for token_str, logprob in top_n_dict.items():
- top_logprobs_list.append(
- {
- "token": token_str,
- "logprob": logprob,
- "bytes": list(token_str.encode("utf-8")),
- }
- )
-
- entry = {
+ for result_item in batch_results:
+ result = unwrap_serialized_tensors(result_item)
+
+ prompt_tokens_out = result["prompt_tokens"] # The engine can modify prompt_tokens.
+ text_output = result["generated_text"]
+ prompt_tokens_count = len(prompt_tokens_out) if prompt_tokens_out is not None else 0
+ prompt_tokens_counts.append(prompt_tokens_count)
+
+ logprobs_content = None
+ if sampling_params.return_log_probs:
+ token_logprobs = result.get('log_probs', [])
+
+ tokens_to_decode = [[tok] for tok in result["generated_tokens"]]
+ tokens = list(map(tokenizer.detokenize, tokens_to_decode))
+
+ # Get top_n_logprobs if available
+ generated_top_n_logprobs = result.get('generated_top_n_logprobs')
+
+ logprobs_content = []
+ for i, (tok, lp) in enumerate(zip(tokens, token_logprobs)):
+ # Build top_logprobs list for this token position
+ top_logprobs_list = []
+ if generated_top_n_logprobs and i < len(generated_top_n_logprobs):
+ top_n_dict = generated_top_n_logprobs[i]
+ for token_str, logprob in top_n_dict.items():
+ top_logprobs_list.append(
+ {
+ "token": token_str,
+ "logprob": logprob,
+ "bytes": list(token_str.encode("utf-8")),
+ }
+ )
+
+ logprobs_content.append(
+ {
"token": tok,
"logprob": lp,
"bytes": list(tok.encode("utf-8")),
"top_logprobs": top_logprobs_list,
}
- logprobs_content.append(entry)
-
- choice_data = {
- "index": 0,
- "message": {"role": "assistant", "content": text_output},
- # 'logprobs' in chat API is an object containing 'content'
- "logprobs": {"content": logprobs_content} if logprobs_content else None,
- "finish_reason": "length", # Original code hardcoded this.
- }
- choices.append(choice_data)
- total_completion_tokens += len(result.generated_tokens)
- request_idx += 0
+ )
+
+ metadata = {}
+ message_text = text_output
+
+ if parsers:
+ message_text, metadata = apply_parsers(
+ message_text, tools, parsers, tools_requested
+ )
+
+ normalized_tool_calls = metadata.get("tool_calls", [])
+ message = {
+ "role": "assistant",
+ "content": _normalize_assistant_content(message_text, normalized_tool_calls),
+ }
+ if normalized_tool_calls:
+ message["tool_calls"] = normalized_tool_calls
+ if "reasoning" in metadata:
+ message["reasoning_content"] = metadata["reasoning"]
+
+ # Replicate data in the message field for compatibility.
+ message["prompt_token_ids"] = result["prompt_tokens"]
+ message["generation_token_ids"] = result["generated_tokens"]
+ message["generation_log_probs"] = result.get("generated_log_probs", [])
+ return_log_probs = sampling_params.return_log_probs
+ finish_reason = "tool_calls" if metadata.get("tool_calls", []) else "stop"
+ if (
+ len(result["generated_tokens"])
+ >= result["sampling_params"]["num_tokens_to_generate"]
+ ):
+ finish_reason = "length"
+
+ choice_data = {
+ "index": request_idx,
+ "message": message,
+ "prompt_token_ids": result["prompt_tokens"],
+ "generation_token_ids": result["generated_tokens"],
+ "generation_log_probs": result.get("generated_log_probs", []),
+ "raw_text": result["prompt"] + result["generated_text"],
+ # 'logprobs' in chat API is an object containing 'content'
+ # "logprobs": {"content": logprobs_content} if logprobs_content else None,
+ "logprobs": {"content": logprobs_content} if return_log_probs else None,
+ "finish_reason": finish_reason,
+ }
+ choice_data["policy_epoch"] = result["policy_epoch"]
+ choice_data["kv_cache_epoch"] = result["kv_cache_epoch"]
+ choice_data["num_evictions"] = sum(
+ 1 for e in result["events"] if e.get("type") == "EVICT"
+ )
+ if current_app.config['verbose']:
+ logging.info(_redact_token_id_lists_for_logging(result))
+
+ if result["routing_indices"] is not None:
+ choice_data["moe_topk_indices"] = result["routing_indices"]
+ if prompt_tokens_count:
+ choice_data["prompt_moe_topk_indices"] = result["routing_indices"][
+ :prompt_tokens_count
+ ]
+
+ choices.append(choice_data)
+ if choice_data["generation_log_probs"] is None:
+ logger.warning(
+ "Generation log probs is None for request:\n%s",
+ json.dumps(_redact_token_id_lists_for_logging(result), indent=4),
+ )
+ total_completion_tokens += len(result["generated_tokens"])
+ request_idx += 1
+
+ prompt_token_count = max(prompt_tokens_counts) if prompt_tokens_counts else 0
response = {
+ "id": str(uuid.uuid4()),
+ "created": int(time.time()),
+ "model": "EMPTY",
+ "object": "chat.completion",
"choices": choices,
"usage": {
"prompt_tokens": prompt_token_count,
@@ -152,7 +770,12 @@ async def chat_completions():
"total_tokens": prompt_token_count + total_completion_tokens,
},
}
- return jsonify(response)
+
+ if HAVE_ORJSON:
+ # Use orjson for faster serialization
+ return Response(orjson.dumps(response), mimetype="application/json")
+ else:
+ return jsonify(response)
except ImportError as e:
- logger.warning(f"Could not import flask: {e}")
+ logger.warning(f"Could not import quart: {e}")
diff --git a/megatron/core/inference/text_generation_server/dynamic_text_gen_server/endpoints/completions.py b/megatron/core/inference/text_generation_server/dynamic_text_gen_server/endpoints/completions.py
index b749205cdfd..d2279b0d07d 100644
--- a/megatron/core/inference/text_generation_server/dynamic_text_gen_server/endpoints/completions.py
+++ b/megatron/core/inference/text_generation_server/dynamic_text_gen_server/endpoints/completions.py
@@ -4,13 +4,14 @@
import logging
import time
+from megatron.core.inference.inference_request import unwrap_serialized_tensors
from megatron.core.inference.sampling_params import SamplingParams
logger = logging.getLogger(__name__)
try:
- from flask import Blueprint, current_app, jsonify, request
+ from quart import Blueprint, current_app, jsonify, request
bp = Blueprint('completions_api', __name__)
@@ -21,7 +22,9 @@ async def completions():
client = current_app.config['client']
tokenizer = current_app.config['tokenizer']
- req = request.get_json()
+ req = await request.get_json(force=True)
+ if req is None:
+ return "Invalid or missing JSON body", 400
# --- 1. Parse Prompt ---
prompt_data = req.get("prompt")
@@ -84,6 +87,11 @@ async def completions():
# skip_prompt_log_probs=False ensures the engine computes logprobs for prompt tokens
skip_prompt_log_probs = not (echo and return_log_probs)
+ # Parse stop sequences
+ stop = req.get("stop", None)
+ if isinstance(stop, str):
+ stop = [stop]
+
sampling_params = SamplingParams(
temperature=temperature,
top_k=top_k,
@@ -92,6 +100,7 @@ async def completions():
top_n_logprobs=top_n_logprobs,
skip_prompt_log_probs=skip_prompt_log_probs,
num_tokens_to_generate=int(req.get("max_tokens", 16)),
+ stop_words=stop,
)
except ValueError as e:
return f"Invalid sampling parameter: {e}", 400
@@ -107,108 +116,134 @@ async def completions():
top_n_logprobs=sampling_params.top_n_logprobs,
skip_prompt_log_probs=sampling_params.skip_prompt_log_probs,
num_tokens_to_generate=sampling_params.num_tokens_to_generate,
+ stop_words=sampling_params.stop_words,
)
tasks.append(client.add_request(prompt_tokens, per_req_params))
- start_time = time.perf_counter()
+ if current_app.config['verbose']:
+ start_time = time.perf_counter()
+
try:
batch_results = await asyncio.gather(*tasks)
except Exception as e:
return f"Error during inference: {e}", 500
- logger.info(
- f"Batch of {len(tasks)} requests processed in {time.perf_counter() - start_time:.2f}s"
- )
+ if current_app.config['verbose']:
+ logging.info(
+ f"Batch of {len(tasks)} requests processed in "
+ f"{time.perf_counter() - start_time:.2f}s"
+ )
+
+ # --- 4. Check for failed requests ---
+ failed_errors = []
+ has_nontransient_error = False
+ for i, record in enumerate(batch_results):
+ if record.get("status") == "FAILED":
+ events = record.get("events", [])
+ error_events = [
+ e for e in events if e.get("type") in ("ERROR_NONTRANSIENT", "ERROR_TRANSIENT")
+ ]
+ if any(e.get("type") == "ERROR_NONTRANSIENT" for e in error_events):
+ has_nontransient_error = True
+ error_msg = (
+ str(error_events[-1].get("payload", "Unknown error"))
+ if error_events
+ else "Unknown error"
+ )
+ failed_errors.append(f"Request {i}: {error_msg}")
+
+ if failed_errors:
+ error_detail = "; ".join(failed_errors)
+ status = 400 if has_nontransient_error else 500
+ logger.error(f"Inference request(s) failed: {error_detail}")
+ return f"Inference request(s) failed: {error_detail}", status
- # --- 4. Format Response (matching old_completions.py) ---
+ # --- 5. Format Response (matching old_completions.py) ---
choices = []
request_idx = 0
- for record in batch_results:
- for result in record.requests:
- full_text = result.generated_text or ""
- text_output = (prompts_as_strings[request_idx] + full_text) if echo else full_text
-
- logprobs_data = None
- if sampling_params.return_log_probs:
- # Get prompt tokens and logprobs
- prompt_tokens_list = []
- if result.prompt_tokens is not None:
- if hasattr(result.prompt_tokens, 'tolist'):
- prompt_tokens_list = result.prompt_tokens.tolist()
- else:
- prompt_tokens_list = list(result.prompt_tokens)
-
- prompt_log_probs = getattr(result, 'prompt_log_probs', None) or []
- prompt_top_n_logprobs = getattr(result, 'prompt_top_n_logprobs', None) or []
-
- # Get generated tokens and logprobs
- generated_tokens_list = (
- list(result.generated_tokens) if result.generated_tokens else []
- )
- generated_log_probs = getattr(result, 'generated_log_probs', None) or []
- generated_top_n_logprobs = (
- getattr(result, 'generated_top_n_logprobs', None) or []
- )
-
- if echo:
- # When echo=True, include prompt tokens and their logprobs
- # Prompt logprobs are for tokens [1:] (first token has no logprob)
- all_token_ids = prompt_tokens_list + generated_tokens_list
- tokens = [tokenizer.detokenize([tok]) for tok in all_token_ids]
-
- # Build token_logprobs: [None] for first token, then prompt logprobs,
- # then generated logprobs
- token_logprobs = [None] + list(prompt_log_probs) + list(generated_log_probs)
-
- # Build top_logprobs: [None] for first token, then prompt top_n,
- # then generated top_n
- top_logprobs = None
- if prompt_top_n_logprobs or generated_top_n_logprobs:
- top_logprobs = (
- [None]
- + list(prompt_top_n_logprobs)
- + list(generated_top_n_logprobs)
- )
-
- # Calculate text_offset: cumulative character positions starting from 0
- text_offset = []
- current_offset = 0
- for tok_str in tokens:
- text_offset.append(current_offset)
- current_offset += len(tok_str)
- else:
- # When echo=False, only return generated tokens and their logprobs
- tokens = [tokenizer.detokenize([tok]) for tok in generated_tokens_list]
-
- # Prepend [None] to match OpenAI format
- token_logprobs = [None] + list(generated_log_probs)
-
- # Build top_logprobs
- top_logprobs = None
- if generated_top_n_logprobs:
- top_logprobs = [None] + list(generated_top_n_logprobs)
-
- # Calculate text_offset for generated tokens only
- text_offset = []
- current_offset = 0
- for tok_str in tokens:
- text_offset.append(current_offset)
- current_offset += len(tok_str)
-
- logprobs_data = {
- "token_logprobs": token_logprobs,
- "tokens": tokens,
- "text_offset": text_offset,
- "top_logprobs": top_logprobs,
- }
-
- choices.append(
- {"index": request_idx, "text": text_output, "logprobs": logprobs_data}
+ for completed_request in batch_results:
+ result = unwrap_serialized_tensors(completed_request)
+ full_text = result["generated_text"] or ""
+ text_output = (prompts_as_strings[request_idx] + full_text) if echo else full_text
+
+ logprobs_data = None
+ if sampling_params.return_log_probs:
+ # Get prompt tokens and logprobs
+ prompt_tokens_list = result["prompt_tokens"] or []
+
+ prompt_log_probs = result.get('prompt_log_probs') or []
+ prompt_top_n_logprobs = result.get('prompt_top_n_logprobs') or []
+
+ # Get generated tokens and logprobs
+ generated_tokens_list = result["generated_tokens"] or []
+ generated_log_probs = result.get('generated_log_probs') or []
+ generated_top_n_logprobs = result.get('generated_top_n_logprobs') or []
+
+ if echo:
+ # When echo=True, include prompt tokens and their logprobs
+ # Prompt logprobs are for tokens [1:] (first token has no logprob)
+ all_token_ids = prompt_tokens_list + generated_tokens_list
+ tokens = [tokenizer.detokenize([tok]) for tok in all_token_ids]
+
+ # Build token_logprobs: [None] for first token, then prompt logprobs,
+ # then generated logprobs
+ token_logprobs = [None] + list(prompt_log_probs) + list(generated_log_probs)
+
+ # Build top_logprobs: [None] for first token, then prompt top_n,
+ # then generated top_n
+ top_logprobs = None
+ if prompt_top_n_logprobs or generated_top_n_logprobs:
+ top_logprobs = (
+ [None] + list(prompt_top_n_logprobs) + list(generated_top_n_logprobs)
+ )
+
+ # Calculate text_offset: cumulative character positions starting from 0
+ text_offset = []
+ current_offset = 0
+ for tok_str in tokens:
+ text_offset.append(current_offset)
+ current_offset += len(tok_str)
+ else:
+ # When echo=False, only return generated tokens and their logprobs
+ tokens = [tokenizer.detokenize([tok]) for tok in generated_tokens_list]
+
+ # Prepend [None] to match OpenAI format
+ token_logprobs = [None] + list(generated_log_probs)
+
+ # Build top_logprobs
+ top_logprobs = None
+ if generated_top_n_logprobs:
+ top_logprobs = [None] + list(generated_top_n_logprobs)
+
+ # Calculate text_offset for generated tokens only
+ text_offset = []
+ current_offset = 0
+ for tok_str in tokens:
+ text_offset.append(current_offset)
+ current_offset += len(tok_str)
+
+ logprobs_data = {
+ "token_logprobs": token_logprobs,
+ "tokens": tokens,
+ "text_offset": text_offset,
+ "top_logprobs": top_logprobs,
+ }
+
+ choices.append({"index": request_idx, "text": text_output, "logprobs": logprobs_data})
+ if result["routing_indices"] is not None:
+ choices[-1]["moe_topk_indices"] = result["routing_indices"]
+ prompt_length = (
+ len(result["prompt_tokens"]) if result["prompt_tokens"] is not None else 0
)
- request_idx += 1
+ if prompt_length:
+ choices[-1]["prompt_moe_topk_indices"] = result["routing_indices"][
+ :prompt_length
+ ]
+
+ request_idx += 1
return jsonify({"choices": choices})
except ImportError as e:
- logger.warning(f"Could not import flask: {e}")
+ logger.warning(f"Could not import quart: {e}")
diff --git a/megatron/core/inference/text_generation_server/dynamic_text_gen_server/endpoints/health.py b/megatron/core/inference/text_generation_server/dynamic_text_gen_server/endpoints/health.py
index a9d0a678b44..5458266b276 100644
--- a/megatron/core/inference/text_generation_server/dynamic_text_gen_server/endpoints/health.py
+++ b/megatron/core/inference/text_generation_server/dynamic_text_gen_server/endpoints/health.py
@@ -5,7 +5,7 @@
logger = logging.getLogger(__name__)
try:
- from flask import Blueprint, current_app, jsonify
+ from quart import Blueprint, current_app, jsonify
bp = Blueprint('health_api', __name__)
@@ -35,4 +35,4 @@ async def health():
return jsonify({"status": "error", "details": str(e)}), 500
except ImportError as e:
- logger.warning(f"Could not import flask: {e}")
+ logger.warning(f"Could not import quart: {e}")
diff --git a/megatron/core/inference/text_generation_server/dynamic_text_gen_server/flask_server.py b/megatron/core/inference/text_generation_server/dynamic_text_gen_server/flask_server.py
deleted file mode 100644
index 2b0469b340a..00000000000
--- a/megatron/core/inference/text_generation_server/dynamic_text_gen_server/flask_server.py
+++ /dev/null
@@ -1,76 +0,0 @@
-# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
-
-import logging
-import socket
-from contextlib import contextmanager
-
-try:
- from flask import Flask
- from hypercorn.asyncio import serve
- from hypercorn.config import Config
-
- HAS_FLASK = True
-except ImportError as e:
- HAS_FLASK = False
-
-import megatron.core.inference.text_generation_server.dynamic_text_gen_server.endpoints as endpoints
-from megatron.core.inference.inference_client import InferenceClient
-from megatron.core.utils import trace_async_exceptions
-
-logger = logging.getLogger(__name__)
-
-
-@contextmanager
-def temp_log_level(level, logger=None):
- """Enables temporarily overriding the logging level."""
- logger = logger or logging.getLogger()
- old_level = logger.level
- logger.setLevel(level)
- try:
- yield
- finally:
- logger.setLevel(old_level)
-
-
-@trace_async_exceptions
-async def run_flask_server(coordinator_port: int, tokenizer, rank: int, flask_port: int):
- """Initializes and runs the async Flask server."""
- if not HAS_FLASK:
- raise RuntimeError(f"Flask not available")
-
- try:
- hostname = socket.gethostname()
- except Exception as e:
- logger.warning(f"Could not get hostname: {e}")
- hostname = "0.0.0.0"
-
- inference_client = InferenceClient(coordinator_port)
- await inference_client.start()
- logger.info(f"Rank {rank}: InferenceClient connected.")
-
- app = Flask(__name__)
-
- # Store client and tokenizer in app config for Blueprints to use
- app.config['client'] = inference_client
- app.config['tokenizer'] = tokenizer
-
- # Register all blueprints from the 'endpoints' package
- for endpoint in endpoints.__all__:
- app.register_blueprint(endpoint)
-
- @app.route('/')
- def health_check():
- return "Megatron Dynamic Inference Server is running."
-
- config = Config()
- config.bind = [f"0.0.0.0:{flask_port}"]
-
- # Force logging level to INFO to ensure that hostname is printed
- with temp_log_level(logging.INFO, logger):
- logger.info(f"Starting Flask server on http://{hostname}:{flask_port}")
-
- try:
- await serve(app, config)
- finally:
- await inference_client.stop()
- logger.info(f"Rank {rank}: Flask server and client shut down.")
diff --git a/megatron/core/inference/text_generation_server/dynamic_text_gen_server/text_generation_server.py b/megatron/core/inference/text_generation_server/dynamic_text_gen_server/text_generation_server.py
new file mode 100644
index 00000000000..8f91644afdd
--- /dev/null
+++ b/megatron/core/inference/text_generation_server/dynamic_text_gen_server/text_generation_server.py
@@ -0,0 +1,215 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+
+import asyncio
+import logging
+import multiprocessing as mp
+import socket
+from contextlib import contextmanager
+from typing import List, Optional
+
+try:
+ from hypercorn.asyncio import serve
+ from hypercorn.config import Config
+ from quart import Quart
+
+ HAS_BACKEND = True
+except ImportError as e:
+ HAS_BACKEND = False
+
+import megatron.core.inference.text_generation_server.dynamic_text_gen_server.endpoints as endpoints
+from megatron.core.inference.inference_client import InferenceClient
+from megatron.core.utils import trace_async_exceptions
+
+logger = logging.getLogger(__name__)
+
+# Global reference to manage the background server processes
+_SERVER_PROCESSES: List[mp.Process] = []
+_SHARED_SOCKET = None
+
+
+@contextmanager
+def temp_log_level(level, logger=None):
+ """Enables temporarily overriding the logging level."""
+ logger = logger or logging.getLogger()
+ old_level = logger.level
+ logger.setLevel(level)
+ try:
+ yield
+ finally:
+ logger.setLevel(old_level)
+
+
+@trace_async_exceptions
+async def _run_text_gen_server(
+ coordinator_addr: str,
+ tokenizer,
+ rank: int,
+ server_port: int,
+ parsers: Optional[List[str]] = None,
+ verbose: bool = False,
+ fd: Optional[int] = None,
+ hostname: Optional[str] = None,
+):
+ """
+ Initializes and runs the async web server. Automatically starts and
+ manages its own InferenceClient connected to the provided coordinator address.
+ """
+ if not HAS_BACKEND:
+ raise RuntimeError(f"Web backend framework (Quart) not available")
+
+ # Create and start the client locally inside this process
+ inference_client = InferenceClient(coordinator_addr, deserialize=False)
+ inference_client.start()
+ logger.info(f"Rank {rank}: InferenceClient connected.")
+
+ try:
+ if hostname is None:
+ try:
+ hostname = socket.gethostname()
+ except Exception as e:
+ logger.warning(f"Could not get hostname: {e}")
+ hostname = "0.0.0.0"
+
+ app = Quart(__name__)
+
+ # Quart native way to handle max body size (1 GB; needed for large prompts)
+ app.config['MAX_CONTENT_LENGTH'] = 2**30
+
+ # Store client and tokenizer in app config for Blueprints to use
+ app.config['client'] = inference_client
+ app.config['tokenizer'] = tokenizer
+ app.config['parsers'] = parsers
+ app.config['verbose'] = verbose
+
+ # Register all blueprints from the 'endpoints' package
+ for endpoint in endpoints.__all__:
+ app.register_blueprint(endpoint)
+
+ config = Config()
+ config.keep_alive_timeout = 30.0 # Keep connection alive between long-running requests.
+ config.backlog = 2**14 # Expect high load; ensure we do not drop connections.
+ config.h2_max_concurrent_streams = (
+ 2**14
+ ) # Allow many concurrent streams for HTTP/2 clients.
+
+ if fd is not None:
+ config.bind = [f"fd://{fd}"]
+ else:
+ config.bind = [f"{hostname}:{server_port}"]
+
+ with temp_log_level(logging.INFO, logger):
+ logger.info(f"Starting text generation server on http://{hostname}:{server_port}")
+ logger.info(f"Using tokenizer: {type(tokenizer)}")
+ logger.info(f"Using parsers: {parsers}")
+
+ # Quart is natively ASGI, so we can serve the app directly
+ await serve(app, config)
+
+ finally:
+ # Gracefully shut down the client when the server stops
+ inference_client.stop()
+ logger.info(f"Rank {rank}: Web server and client shut down.")
+
+
+def _server_process_worker(
+ coordinator_addr: str,
+ tokenizer,
+ rank: int,
+ server_port: int,
+ parsers: Optional[List[str]] = None,
+ verbose: bool = False,
+ fd: Optional[int] = None,
+ hostname: Optional[str] = None,
+):
+ """Synchronous worker function that sets up a new event loop for the separate process."""
+ loop = asyncio.new_event_loop()
+ asyncio.set_event_loop(loop)
+ try:
+ loop.run_until_complete(
+ _run_text_gen_server(
+ coordinator_addr, tokenizer, rank, server_port, parsers, verbose, fd, hostname
+ )
+ )
+ except KeyboardInterrupt:
+ logger.info(f"Rank {rank}: text gen server process interrupted.")
+ finally:
+ pending = asyncio.all_tasks(loop)
+ for task in pending:
+ task.cancel()
+ if pending:
+ loop.run_until_complete(asyncio.gather(*pending, return_exceptions=True))
+ loop.close()
+
+
+def start_text_gen_server(
+ coordinator_addr: str,
+ tokenizer,
+ rank: int,
+ server_port: int,
+ parsers: Optional[List[str]] = None,
+ verbose: bool = False,
+ num_replicas: int = 4,
+ hostname: Optional[str] = None,
+):
+ """Start the text generation server."""
+ global _SERVER_PROCESSES
+ global _SHARED_SOCKET
+
+ if _SERVER_PROCESSES:
+ logger.warning("Text gen server processes are already running.")
+ return
+
+ _SHARED_SOCKET = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ _SHARED_SOCKET.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+
+ if hasattr(socket, 'SO_REUSEPORT'):
+ try:
+ _SHARED_SOCKET.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
+ except OSError:
+ pass
+
+ bind_address = hostname if hostname is not None else "0.0.0.0"
+ _SHARED_SOCKET.bind((bind_address, server_port))
+ _SHARED_SOCKET.setblocking(False)
+
+ _SHARED_SOCKET.set_inheritable(True)
+ fd = _SHARED_SOCKET.fileno()
+
+ for i in range(num_replicas):
+ p = mp.Process(
+ target=_server_process_worker,
+ args=(coordinator_addr, tokenizer, rank, server_port, parsers, verbose, fd, hostname),
+ daemon=True,
+ )
+ p.start()
+ _SERVER_PROCESSES.append(p)
+ logger.info(f"Started text gen frontend replica {i+1}/{num_replicas} (PID: {p.pid})")
+
+
+def stop_text_gen_server():
+ """Stop the text generation server."""
+ global _SERVER_PROCESSES
+ global _SHARED_SOCKET
+
+ if not _SERVER_PROCESSES:
+ return
+
+ logger.info(f"Terminating {len(_SERVER_PROCESSES)} Text Gen frontend processes...")
+
+ for p in _SERVER_PROCESSES:
+ if p.is_alive():
+ p.terminate()
+
+ for p in _SERVER_PROCESSES:
+ p.join(timeout=3)
+ if p.is_alive():
+ p.kill()
+ p.join()
+
+ # Clean up the master socket
+ if _SHARED_SOCKET is not None:
+ _SHARED_SOCKET.close()
+ _SHARED_SOCKET = None
+
+ _SERVER_PROCESSES = []
+ logger.info("All text gen frontend processes terminated.")
diff --git a/megatron/core/inference/text_generation_server/text_generation_server.py b/megatron/core/inference/text_generation_server/text_generation_server.py
index ec60a52d9a4..bac92e391b8 100644
--- a/megatron/core/inference/text_generation_server/text_generation_server.py
+++ b/megatron/core/inference/text_generation_server/text_generation_server.py
@@ -30,6 +30,7 @@ class MegatronGenerate(Resource):
def __init__(self, engine, args):
self.engine = engine
self.args = args
+ self.verbose = getattr(args, 'inference_flask_server_logging', False)
def put(self):
"""Handle generation request."""
@@ -76,7 +77,7 @@ def put(self):
if "temperature" in request.get_json():
temperature = request.get_json()["temperature"]
if not (isinstance(temperature, (int, float))):
- return "temperature must be a positive number less than or equal to 1000.0"
+ return "temperature must be a positive number less than or equal to 100.0"
if not (0.0 < temperature <= 100.0):
return "temperature must be a positive number less than or equal to 100.0"
@@ -156,12 +157,6 @@ def put(self):
if random_seed < 0:
return "random_seed must be a positive integer"
- no_log = False
- if "no_log" in request.get_json():
- no_log = request.get_json()["no_log"]
- if not isinstance(no_log, bool):
- return "no_log must be a boolean value"
-
stop_token = 50256
if "stop_token" in request.get_json():
stop_token = request.get_json()["stop_token"]
@@ -176,7 +171,7 @@ def put(self):
with LOCK: # Need to get lock to keep multiple threads from hitting code
- if not no_log:
+ if self.verbose:
logging.info(f"request IP: {str(request.remote_addr)}")
logging.info(json.dumps(request.get_json()))
logging.info(f"start time: {datetime.datetime.now()}")
diff --git a/megatron/core/inference/unified_memory.py b/megatron/core/inference/unified_memory.py
index aa3851efc78..2d768cafb09 100644
--- a/megatron/core/inference/unified_memory.py
+++ b/megatron/core/inference/unified_memory.py
@@ -69,6 +69,12 @@ def _handler(signum, frame):
"Please clean up your stale cache and try again."
)
+ # Signal-based timeout only works in the main thread.
+ # In non-main threads (e.g., Ray actors), skip the timeout mechanism.
+ if threading.current_thread() is not threading.main_thread():
+ yield
+ return
+
curr_handler = signal.signal(signal.SIGALRM, _handler)
try:
signal.alarm(timeout_s)
diff --git a/megatron/core/inference/utils.py b/megatron/core/inference/utils.py
index 0bdaff64be1..0914b81f005 100644
--- a/megatron/core/inference/utils.py
+++ b/megatron/core/inference/utils.py
@@ -1,14 +1,44 @@
# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
import asyncio
+import logging
import multiprocessing
import sys
+from importlib.metadata import PackageNotFoundError, version
import torch
from megatron.core.transformer.moe.moe_layer import MoELayer
from megatron.core.utils import get_model_config
+try:
+ FLASHINFER_JIT_CACHE_VERSION = version("flashinfer-jit-cache")
+except PackageNotFoundError:
+ FLASHINFER_JIT_CACHE_VERSION = None
+
+
+def device_memory_summary() -> str:
+ """One-line GPU memory summary for torch_memory_saver logging."""
+ dev = torch.cuda.current_device()
+ stats = torch.cuda.memory_stats(dev)
+ try:
+ segs = torch.cuda.memory_snapshot(include_traces=False)
+ except TypeError: # include_traces was added in PyTorch 2.11
+ segs = torch.cuda.memory_snapshot()
+ M = 1024**2
+ private = sum(
+ s.get("active_size", 0)
+ for s in segs
+ if s.get("device", dev) == dev and tuple(s.get("segment_pool_id", (0, 0))) != (0, 0)
+ )
+ alloc = stats.get("allocated_bytes.all.current", 0)
+ resv = stats.get("reserved_bytes.all.current", 0)
+ dev_mem = torch.cuda.device_memory_used()
+ return (
+ f"alloc={alloc/M:.0f}MiB private={private/M:.0f}MiB "
+ f"resv-alloc={(resv-alloc)/M:.0f}MiB resv={resv/M:.0f}MiB dev_mem={dev_mem/M:.0f}MiB"
+ )
+
class Counter:
"""A simple counter class
@@ -132,6 +162,73 @@ def set_decode_expert_padding(model, set_to: bool = False, capacity_factor: int
router.config.moe_pad_expert_input_to_capacity = bool(set_to)
+def check_flashinfer_jit_cache_installed(log_version: bool = False):
+ """Verify that the flashinfer-jit-cache package is installed.
+
+ The flashinfer-jit-cache package provides pre-compiled CUTLASS fused MoE kernels
+ so they don't need to be JIT-compiled at runtime. This avoids a multi-minute
+ compilation step during CUDA graph warmup.
+
+ Raises:
+ RuntimeError: If flashinfer-jit-cache is not installed and CUDA version is 12 or 13.
+ """
+ if FLASHINFER_JIT_CACHE_VERSION is not None:
+ if log_version:
+ logging.info(
+ f"Found flashinfer-jit-cache {FLASHINFER_JIT_CACHE_VERSION} with "
+ "pre-compiled CUTLASS kernels."
+ )
+ return
+
+ cuda_major = torch.version.cuda.split(".")[0] if torch.version.cuda else None
+
+ if cuda_major == "12":
+ install_cmd = (
+ "Install it with:\n\npip install flashinfer-jit-cache "
+ "--index-url https://flashinfer.ai/whl/cu129\n"
+ )
+ elif cuda_major == "13":
+ install_cmd = (
+ "Install it with:\n\npip install flashinfer-jit-cache "
+ "--index-url https://flashinfer.ai/whl/cu130\n"
+ )
+ else:
+ install_cmd = ""
+
+ raise RuntimeError(
+ "The 'flashinfer-jit-cache' package is required for expert parallel inference "
+ f"but is not installed. {install_cmd}"
+ )
+
+
+def set_inference_cuda_graphed_iteration_for_ep_inference(model):
+ """Enable CUDA graph compatibility for expert parallel inference.
+
+ Sets a flag in all MoELayers indicating the current iteration is being
+ captured/executed in a CUDA graph. This allows the dispatcher to adjust
+ its behavior for CUDA graph compatibility.
+ """
+ global moe_layer_cache
+ if moe_layer_cache is None:
+ _init_moe_expert_cache(model)
+
+ for moe_layer in moe_layer_cache:
+ moe_layer.set_inference_cuda_graphed_iteration()
+
+
+def unset_inference_cuda_graphed_iteration_for_ep_inference(model):
+ """Disable CUDA graph compatibility for expert parallel inference.
+
+ Clears the flag in all MoELayers, restoring standard dispatcher behavior.
+ """
+ global moe_layer_cache
+ if moe_layer_cache is None:
+ _init_moe_expert_cache(model)
+
+ for moe_layer in moe_layer_cache:
+ moe_layer.unset_inference_cuda_graphed_iteration()
+
+
def tensor_swap(x, src_idxs, dst_idxs):
"""
Swap x[src_idxs] and x[dst_idxs]
diff --git a/megatron/core/model_parallel_config.py b/megatron/core/model_parallel_config.py
index 3c6ff04d3b0..d5cd5397d56 100644
--- a/megatron/core/model_parallel_config.py
+++ b/megatron/core/model_parallel_config.py
@@ -77,11 +77,6 @@ class ModelParallelConfig:
Default is None, which will be set to the value of tensor_model_parallel_size.
"""
- moe_extended_tp: bool = False
- """NOTE: Deprecated from MCore v0.10. This flag is ignored.
- Its functionality is replaced by expert_tensor_parallel_size.
- """
-
###################
# Initialization
###################
@@ -173,9 +168,6 @@ class ModelParallelConfig:
must turn off gradient accumulation fusion.
"""
- async_tensor_model_parallel_allreduce: bool = True
- """NOTE: Deprecated. This flag is ignored."""
-
use_te_rng_tracker: bool = field(
default=False, metadata={"argparse_meta": {"arg_names": ["--te-rng-tracker"]}}
)
@@ -389,6 +381,11 @@ class ModelParallelConfig:
cpu_offloading_double_buffering: bool = False
"""If True, enables double buffering across layers while reloading activations from CPU."""
+ cpu_offloading_retain_pinned_cpu_buffers: bool = False
+ """If True, the pinned CPU buffers are retained after offloading and reused for the
+ next iteration. It is useful for cuda graphs capture.
+ """
+
###################
# Timing
###################
diff --git a/megatron/core/models/T5/t5_model.py b/megatron/core/models/T5/t5_model.py
index 6d134067ccc..b2feb974643 100644
--- a/megatron/core/models/T5/t5_model.py
+++ b/megatron/core/models/T5/t5_model.py
@@ -56,7 +56,11 @@ def __init__(
config.hidden_size,
vocab_size,
config=config,
- init_method=config.init_method,
+ init_method=(
+ config.embedding_init_method
+ if config.use_mup and not share_embeddings_and_output_weights
+ else config.init_method
+ ),
bias=share_embeddings_and_output_weights,
skip_bias_add=not share_embeddings_and_output_weights,
gather_output=not self.parallel_output,
diff --git a/megatron/core/models/T5/t5_spec.py b/megatron/core/models/T5/t5_spec.py
index 50aecf0a950..9f465df5c21 100644
--- a/megatron/core/models/T5/t5_spec.py
+++ b/megatron/core/models/T5/t5_spec.py
@@ -1,4 +1,5 @@
# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.fusions.fused_bias_dropout import get_bias_dropout_add
from megatron.core.tensor_parallel.layers import ColumnParallelLinear, RowParallelLinear
from megatron.core.transformer.attention import (
@@ -16,9 +17,7 @@
from megatron.core.transformer.transformer_layer import TransformerLayer, TransformerLayerSubmodules
from megatron.core.typed_torch import not_none
-try:
- import transformer_engine as te # pylint: disable=unused-import
-
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import (
TEColumnParallelLinear,
TEDotProductAttention,
@@ -26,9 +25,7 @@
TENorm,
TERowParallelLinear,
)
-
- HAVE_TE = True
-except ImportError:
+else:
(
TEColumnParallelLinear,
TEDotProductAttention,
@@ -36,7 +33,6 @@
TENorm,
TERowParallelLinear,
) = (None, None, None, None, None)
- HAVE_TE = False
try:
import apex # pylint: disable=unused-import
@@ -76,7 +72,8 @@ def encoder_model_with_transformer_engine_default_spec() -> ModuleSpec:
mlp=ModuleSpec(
module=MLP,
submodules=MLPSubmodules(
- linear_fc1=TELayerNormColumnParallelLinear, linear_fc2=TERowParallelLinear
+ linear_fc1=not_none(TELayerNormColumnParallelLinear),
+ linear_fc2=not_none(TERowParallelLinear),
),
),
mlp_bda=get_bias_dropout_add,
@@ -102,7 +99,7 @@ def decoder_model_with_transformer_engine_default_spec() -> ModuleSpec:
),
),
self_attn_bda=get_bias_dropout_add,
- pre_cross_attn_layernorm=TENorm,
+ pre_cross_attn_layernorm=not_none(TENorm),
cross_attention=ModuleSpec(
module=CrossAttention,
params={"attn_mask_type": AttnMaskType.padding},
@@ -117,7 +114,8 @@ def decoder_model_with_transformer_engine_default_spec() -> ModuleSpec:
mlp=ModuleSpec(
module=MLP,
submodules=MLPSubmodules(
- linear_fc1=TELayerNormColumnParallelLinear, linear_fc2=TERowParallelLinear
+ linear_fc1=not_none(TELayerNormColumnParallelLinear),
+ linear_fc2=not_none(TERowParallelLinear),
),
),
mlp_bda=get_bias_dropout_add,
diff --git a/megatron/core/models/backends.py b/megatron/core/models/backends.py
index 7f84599a04c..b019d527342 100644
--- a/megatron/core/models/backends.py
+++ b/megatron/core/models/backends.py
@@ -1,14 +1,27 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+from __future__ import annotations
import warnings
from abc import abstractmethod
-from typing import Optional, Protocol, Tuple
+from functools import partial
+from typing import Optional, Protocol, cast
+from megatron.core.extensions.transformer_engine import (
+ TEColumnParallelGroupedLinear,
+ TERowParallelGroupedLinear,
+)
from megatron.core.tensor_parallel.layers import ColumnParallelLinear, RowParallelLinear
from megatron.core.transformer.dot_product_attention import DotProductAttention
-from megatron.core.transformer.mlp import MLPSubmodules
-from megatron.core.transformer.moe.experts import GroupedMLP, SequentialMLP
-from megatron.core.transformer.torch_norm import WrappedTorchNorm
+from megatron.core.transformer.mlp import MLPSubmodules, TEActivationFunctionBuilder
+from megatron.core.transformer.moe.experts import (
+ GroupedMLPSubmodules,
+ InferenceGroupedMLP,
+ SequentialMLP,
+)
+from megatron.core.transformer.moe.moe_layer import ExpertsBuilder
+from megatron.core.transformer.torch_norm import LayerNormBuilder, WrappedTorchNorm
+from megatron.core.typed_torch import not_none
+from megatron.core.utils import is_te_min_version
try:
import apex # pylint: disable=unused-import
@@ -19,17 +32,18 @@
LNImpl = FusedLayerNorm
except ImportError:
warnings.warn("Apex is not installed. Falling back to Torch Norm")
- LNImpl = WrappedTorchNorm
+ FusedLayerNorm = None
HAVE_APEX = False
+ LNImpl = WrappedTorchNorm
from megatron.core.extensions.transformer_engine import (
TEActivationOp,
- TEColumnParallelLinear,
TEDotProductAttention,
TELinear,
TENorm,
)
from megatron.core.tensor_parallel.inference_layers import (
+ InferenceColumnParallelLinear,
InferenceLayerNormColumnParallelLinear,
InferenceRowParallelLinear,
)
@@ -60,7 +74,9 @@ def column_parallel_layer_norm_linear(self) -> Optional[type]:
...
@abstractmethod
- def layer_norm(self, rms_norm: bool = False, for_qk: bool = False) -> type:
+ def layer_norm(
+ self, rms_norm: bool = False, for_qk: bool = False, has_residual: bool = False
+ ) -> LayerNormBuilder:
"""Which module for layernorm"""
...
@@ -70,14 +86,12 @@ def core_attention(self) -> type:
...
@abstractmethod
- def grouped_mlp_modules(
- self, moe_use_grouped_gemm: bool, moe_use_legacy_grouped_gemm: bool
- ) -> Tuple[type, Optional[MLPSubmodules]]:
+ def grouped_mlp_modules(self, moe_use_grouped_gemm: bool) -> ExpertsBuilder:
"""Which module and submodules to use for grouped mlp"""
...
@abstractmethod
- def activation_func(self) -> type:
+ def activation_func(self) -> TEActivationFunctionBuilder | None:
"""Which module to use for activation function"""
...
@@ -101,7 +115,9 @@ def column_parallel_layer_norm_linear(self) -> Optional[type]:
"""Which module for sequential layernorm and linear"""
return None
- def layer_norm(self, rms_norm: bool = False, for_qk: bool = False) -> type:
+ def layer_norm(
+ self, rms_norm: bool = False, for_qk: bool = False, has_residual: bool = False
+ ) -> LayerNormBuilder:
"""Which module to use for layer norm"""
if rms_norm:
# Matching get_gpt_layer_local_spec.
@@ -114,22 +130,18 @@ def core_attention(self) -> type:
"""Which module to use for attention"""
return DotProductAttention
- def grouped_mlp_modules(
- self, moe_use_grouped_gemm: bool, moe_use_legacy_grouped_gemm: bool
- ) -> Tuple[type, Optional[MLPSubmodules]]:
+ def grouped_mlp_modules(self, moe_use_grouped_gemm: bool) -> ExpertsBuilder:
"""Which module and submodules to use for grouped mlp"""
- if moe_use_grouped_gemm:
- warnings.warn(
- "The legacy GroupedMLP will be deprecated in Megatron-Core v0.12.0. "
- "Please update the TransformerEngine to version>=1.7.0 and use TEGroupedMLP."
- )
- return GroupedMLP, None
- else:
- return SequentialMLP, MLPSubmodules(
- linear_fc1=ColumnParallelLinear, linear_fc2=RowParallelLinear
- )
-
- def activation_func(self) -> type:
+ return partial(
+ SequentialMLP,
+ submodules=MLPSubmodules(
+ linear_fc1=ColumnParallelLinear,
+ linear_fc2=RowParallelLinear,
+ activation_func=self.activation_func(),
+ ),
+ )
+
+ def activation_func(self) -> TEActivationFunctionBuilder | None:
"""Which module to use for activation function"""
return None
@@ -143,7 +155,7 @@ def linear(self) -> type:
def column_parallel_linear(self) -> type:
"""Which column parallel linear module TE backend uses"""
- return TEColumnParallelLinear
+ return InferenceColumnParallelLinear
def row_parallel_linear(self) -> type:
"""Which row parallel linear module TE backend uses"""
@@ -157,26 +169,34 @@ def column_parallel_layer_norm_linear(self) -> type[InferenceLayerNormColumnPara
"""Which module for sequential layernorm and linear"""
return InferenceLayerNormColumnParallelLinear
- def layer_norm(self, rms_norm: bool = False, for_qk: bool = False) -> type:
+ def layer_norm(
+ self, rms_norm: bool = False, for_qk: bool = False, has_residual: bool = False
+ ) -> LayerNormBuilder:
"""Which module to use for layer norm"""
if for_qk and not is_te_min_version("1.9.0"):
# TENorm significantly harms convergence when used
# for QKLayerNorm if TE Version < 1.9;
# we instead use the Apex implementation.
- return FusedLayerNorm
+ return not_none(FusedLayerNorm)
return TENorm
def core_attention(self) -> type[TEDotProductAttention]:
"""Which module to use for attention"""
return TEDotProductAttention
- def activation_func(self) -> type:
+ def activation_func(self) -> TEActivationFunctionBuilder | None:
"""Which module to use for activation function"""
- return TEActivationOp
+ # transformer_engine.BasicOperation.forward has an overly permissive return type, but by
+ # design these classes always meet the interface.
+ return cast(TEActivationFunctionBuilder, TEActivationOp)
- def grouped_mlp_modules(
- self, moe_use_grouped_gemm: bool, moe_use_legacy_grouped_gemm: bool
- ) -> Tuple[type, Optional[MLPSubmodules]]:
- raise NotImplementedError(
- "MOE is not supported with inference optimized transformer implementation."
+ def grouped_mlp_modules(self, moe_use_grouped_gemm: bool) -> ExpertsBuilder:
+ """Which module and submodules to use for grouped mlp"""
+ return partial(
+ InferenceGroupedMLP,
+ submodules=GroupedMLPSubmodules(
+ linear_fc1=TEColumnParallelGroupedLinear,
+ linear_fc2=TERowParallelGroupedLinear,
+ activation_func=self.activation_func(),
+ ),
)
diff --git a/megatron/core/models/bert/bert_layer_specs.py b/megatron/core/models/bert/bert_layer_specs.py
index 8415ef02cc5..53cc0f4280d 100644
--- a/megatron/core/models/bert/bert_layer_specs.py
+++ b/megatron/core/models/bert/bert_layer_specs.py
@@ -1,6 +1,7 @@
# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
import warnings
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.fusions.fused_bias_dropout import get_bias_dropout_add
from megatron.core.tensor_parallel.layers import ColumnParallelLinear, RowParallelLinear
from megatron.core.transformer.attention import SelfAttention, SelfAttentionSubmodules
@@ -12,23 +13,18 @@
from megatron.core.transformer.transformer_layer import TransformerLayer, TransformerLayerSubmodules
from megatron.core.typed_torch import not_none
-try:
- import transformer_engine as te # pylint: disable=unused-import
-
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import (
TEDotProductAttention,
TELayerNormColumnParallelLinear,
TERowParallelLinear,
)
-
- HAVE_TE = True
-except ImportError:
+else:
(TEDotProductAttention, TELayerNormColumnParallelLinear, TERowParallelLinear) = (
None,
None,
None,
)
- HAVE_TE = False
try:
import apex # pylint: disable=unused-import
@@ -45,40 +41,50 @@
HAVE_APEX = False
-def get_bert_layer_with_transformer_engine_spec():
- """Use this spec to use lower-level Transformer Engine modules (required for fp8 training).
+def get_bert_layer_with_transformer_engine_submodules() -> TransformerLayerSubmodules:
+ """Use these submodules to use lower-level Transformer Engine modules (required for fp8
+ training).
Returns:
- ModuleSpec: Module specification with TE modules
+ TransformerLayerSubmodules: Submodules with TE modules.
"""
if not HAVE_TE:
raise ImportError(
"Transformer Engine is not installed. Please use local Bert layer spec instead."
)
- return ModuleSpec(
- module=TransformerLayer,
- submodules=TransformerLayerSubmodules(
- self_attention=ModuleSpec(
- module=SelfAttention,
- params={"attn_mask_type": AttnMaskType.padding},
- submodules=SelfAttentionSubmodules(
- linear_qkv=not_none(TELayerNormColumnParallelLinear),
- core_attention=not_none(TEDotProductAttention),
- linear_proj=TERowParallelLinear,
- q_layernorm=IdentityOp,
- k_layernorm=IdentityOp,
- ),
+ return TransformerLayerSubmodules(
+ self_attention=ModuleSpec(
+ module=SelfAttention,
+ params={"attn_mask_type": AttnMaskType.padding},
+ submodules=SelfAttentionSubmodules(
+ linear_qkv=not_none(TELayerNormColumnParallelLinear),
+ core_attention=not_none(TEDotProductAttention),
+ linear_proj=not_none(TERowParallelLinear),
+ q_layernorm=IdentityOp,
+ k_layernorm=IdentityOp,
),
- self_attn_bda=get_bias_dropout_add,
- mlp=ModuleSpec(
- module=MLP,
- submodules=MLPSubmodules(
- linear_fc1=TELayerNormColumnParallelLinear, linear_fc2=TERowParallelLinear
- ),
+ ),
+ self_attn_bda=get_bias_dropout_add,
+ mlp=ModuleSpec(
+ module=MLP,
+ submodules=MLPSubmodules(
+ linear_fc1=not_none(TELayerNormColumnParallelLinear),
+ linear_fc2=not_none(TERowParallelLinear),
),
- mlp_bda=get_bias_dropout_add,
),
+ mlp_bda=get_bias_dropout_add,
+ )
+
+
+def get_bert_layer_with_transformer_engine_spec():
+ """Use this spec to use lower-level Transformer Engine modules (required for fp8 training).
+
+ Returns:
+ ModuleSpec: Module specification with TE modules
+ """
+ return ModuleSpec(
+ module=TransformerLayer, submodules=get_bert_layer_with_transformer_engine_submodules()
)
diff --git a/megatron/core/models/bert/bert_model.py b/megatron/core/models/bert/bert_model.py
index abe9bc1c9b7..3fd1e01f4a1 100644
--- a/megatron/core/models/bert/bert_model.py
+++ b/megatron/core/models/bert/bert_model.py
@@ -15,6 +15,7 @@
from megatron.core.models.common.embeddings.rotary_pos_embedding import RotaryEmbedding
from megatron.core.models.common.language_module.language_module import LanguageModule
from megatron.core.process_groups_config import ProcessGroupCollection
+from megatron.core.transformer.attention import SelfAttentionSubmodules
from megatron.core.transformer.dot_product_attention import (
DotProductAttention as MCoreDotProductAttention,
)
@@ -22,16 +23,9 @@
from megatron.core.transformer.spec_utils import ModuleSpec
from megatron.core.transformer.transformer_block import TransformerBlock
from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.transformer.transformer_layer import TransformerLayerSubmodules
from megatron.core.transformer.utils import get_linear_layer
-from megatron.core.utils import deprecate_inference_params
-from megatron.core.utils import get_te_version as _get_te_version
-from megatron.core.utils import is_te_min_version
-
-
-def get_te_version():
- """Included for backwards compatibility."""
- warnings.warn("`get_te_version` will be deprecated in a future release")
- return _get_te_version()
+from megatron.core.utils import deprecate_inference_params, is_te_min_version
class BertModel(LanguageModule):
@@ -141,7 +135,11 @@ def __init__(
config.hidden_size,
self.vocab_size,
config=config,
- init_method=config.init_method,
+ init_method=(
+ config.embedding_init_method
+ if config.use_mup and not self.share_embeddings_and_output_weights
+ else config.init_method
+ ),
bias=True,
skip_bias_add=False,
gather_output=not self.parallel_output,
@@ -187,6 +185,11 @@ def _sanity_check_attention_and_get_attn_mask_dimension(self) -> str:
"""
attention_backend = self.config.attention_backend
attn_mask_dimensions = None
+ assert isinstance(self.transformer_layer_spec.submodules, TransformerLayerSubmodules)
+ assert isinstance(
+ self.transformer_layer_spec.submodules.self_attention.submodules,
+ SelfAttentionSubmodules,
+ )
# For local layer spec we just use b1ss
if (
self.transformer_layer_spec.submodules.self_attention.submodules.core_attention
diff --git a/megatron/core/models/common/embeddings/language_model_embedding.py b/megatron/core/models/common/embeddings/language_model_embedding.py
index b410e0ad170..7e49ec6c02d 100644
--- a/megatron/core/models/common/embeddings/language_model_embedding.py
+++ b/megatron/core/models/common/embeddings/language_model_embedding.py
@@ -127,6 +127,10 @@ def forward(self, input_ids: Tensor, position_ids: Tensor, tokentype_ids: int =
else:
assert self.tokentype_embeddings is None
+ # MuP: scale embeddings by alpha_input.
+ if self.config.use_mup and self.config.mup_embedding_mult != 1.0:
+ embeddings = embeddings * self.config.mup_embedding_mult
+
# If the input flag for fp32 residual connection is set, convert for float.
if self.config.fp32_residual_connection:
embeddings = embeddings.float()
diff --git a/megatron/core/models/common/embeddings/rope_utils.py b/megatron/core/models/common/embeddings/rope_utils.py
index e39540eb1d1..2fd19194813 100644
--- a/megatron/core/models/common/embeddings/rope_utils.py
+++ b/megatron/core/models/common/embeddings/rope_utils.py
@@ -288,7 +288,12 @@ def apply_rotary_pos_emb(
else:
assert fused_apply_rotary_pos_emb_thd is not None, "apply_rope_fusion is not available."
return fused_apply_rotary_pos_emb_thd(
- t, cu_seqlens, freqs, cp_size=cp_group.size(), cp_rank=cp_group.rank()
+ t,
+ cu_seqlens,
+ freqs,
+ cp_size=cp_group.size(),
+ cp_rank=cp_group.rank(),
+ interleaved=config.rotary_interleaved,
)
# use unfused implementation
if cu_seqlens is None:
diff --git a/megatron/core/models/common/embeddings/rotary_pos_embedding.py b/megatron/core/models/common/embeddings/rotary_pos_embedding.py
index 05a7e8f60bb..0e560f939f2 100644
--- a/megatron/core/models/common/embeddings/rotary_pos_embedding.py
+++ b/megatron/core/models/common/embeddings/rotary_pos_embedding.py
@@ -240,7 +240,15 @@ def get_rotary_seq_len(
# by the tp and cp size.
return max(packed_seq_params.max_seqlen_q, packed_seq_params.max_seqlen_kv)
elif inference_context is not None:
- rotary_seq_len = inference_context.max_sequence_length
+ # For dynamic batching, use the max of context's max_sequence_length and the actual
+ # input size to ensure rotary embeddings cover CUDA graph warmup token counts
+ context_max_seq_len = inference_context.max_sequence_length
+ input_seq_len = 0
+ if transformer_input is not None:
+ input_seq_len = transformer_input.size(0)
+ elif transformer is not None and transformer.input_tensor is not None:
+ input_seq_len = transformer.input_tensor.size(0)
+ rotary_seq_len = max(context_max_seq_len, input_seq_len)
else:
if transformer is not None and transformer.input_tensor is not None:
rotary_seq_len = transformer.input_tensor.size(0)
diff --git a/megatron/core/models/common/language_module/language_module.py b/megatron/core/models/common/language_module/language_module.py
index 57975b2958b..3c6b7c4ab8d 100644
--- a/megatron/core/models/common/language_module/language_module.py
+++ b/megatron/core/models/common/language_module/language_module.py
@@ -177,14 +177,40 @@ def setup_embeddings_and_output_layer(self) -> None:
This function initalizes word embeddings in the final stage when we are
using pipeline parallelism and sharing word embeddings, and sets up param
attributes on the embedding and output layers.
+
+ Parameter attributes set:
+ - `is_embedding_or_output_parameter`: True for embedding + output layer weights.
+ Used by decoupled_lr, Muon optimizer, and other Megatron features.
+ - `is_embedding_parameter`: True for MuP "embedding-class" parameters.
+ Used by MuP for table-8 style optimizer grouping (base LR/eps for vector-like params).
"""
- # Set `is_embedding_or_output_parameter` attribute.
- if self.pre_process:
+ # Mark embedding and output layer for decoupled_lr and other features.
+ # This is the original Megatron attribute used by decoupled_lr, Muon, FSDP, etc.
+ if self.pre_process and hasattr(self, 'embedding'):
self.embedding.word_embeddings.weight.is_embedding_or_output_parameter = True
- if self.post_process and self.output_layer.weight is not None:
+ if (
+ self.post_process
+ and hasattr(self, 'output_layer')
+ and self.output_layer.weight is not None
+ ):
self.output_layer.weight.is_embedding_or_output_parameter = True
+ # Mark embedding-class parameters for MuP optimizer grouping.
+ # Under MuP table-8-style grouping, embeddings/output use base LR/eps while
+ # hidden matrix-like params use width-scaled LR/eps.
+ mtp_process = getattr(self, 'mtp_process', False)
+ if self.config.use_mup and (self.pre_process or mtp_process) and hasattr(self, 'embedding'):
+ for param in self.embedding.parameters():
+ param.is_embedding_parameter = True
+ if (
+ self.config.use_mup
+ and self.post_process
+ and hasattr(self, 'output_layer')
+ and self.output_layer.weight is not None
+ ):
+ self.output_layer.weight.is_embedding_parameter = True
+
# If share_embeddings_and_output_weights is True, we need to maintain duplicated
# embedding weights in post processing stage. If use Multi-Token Prediction (MTP),
# we also need to maintain duplicated embedding weights in mtp process stage.
@@ -223,6 +249,9 @@ def setup_embeddings_and_output_layer(self) -> None:
weight.data.fill_(0)
weight.shared = True
weight.shared_embedding = True
+ # Keep optimizer grouping consistent for tied embedding/output copies.
+ if self.config.use_mup:
+ weight.is_embedding_parameter = True
# Parameters are shared between the word embeddings layers, and the
# heads at the end of the model. In a pipelined setup with more than
@@ -240,7 +269,7 @@ def setup_embeddings_and_output_layer(self) -> None:
# Ensure that first and last stages have the same initial parameter
# values.
if torch.distributed.is_initialized():
- if self._is_in_embd_group():
+ if self._is_in_embd_group() and not self.config.init_model_with_meta_device:
weight = self.shared_embedding_or_output_weight()
weight.data = weight.data.cuda()
torch.distributed.all_reduce(weight.data, group=self.embd_group)
@@ -255,6 +284,25 @@ def setup_embeddings_and_output_layer(self) -> None:
)
LanguageModule.embedding_warning_printed = True
+ def _scale_logits(self, logits: Tensor) -> Tensor:
+ """Apply MuP output scaling to logits.
+
+ When MuP is enabled, scales logits by mup_output_mult (auto-set to 1/width_mult
+ if left at default) to keep output variance stable across widths.
+
+ Args:
+ logits (Tensor): Raw logits from the output layer.
+
+ Returns:
+ Tensor: Scaled logits if MuP is enabled and mup_output_mult != 1.0,
+ otherwise unchanged logits.
+ """
+ if not self.config.use_mup:
+ return logits
+ if self.config.mup_output_mult != 1.0:
+ return logits * self.config.mup_output_mult
+ return logits
+
def shared_embedding_or_output_weight(self) -> Tensor:
"""Gets the embedding weight or output logit weights when share embedding and output weights set to True
or when use Multi-Token Prediction (MTP).
diff --git a/megatron/core/models/common/model_chunk_schedule_plan.py b/megatron/core/models/common/model_chunk_schedule_plan.py
index 3b0e3a13b76..2b9d72d5f35 100644
--- a/megatron/core/models/common/model_chunk_schedule_plan.py
+++ b/megatron/core/models/common/model_chunk_schedule_plan.py
@@ -62,8 +62,8 @@ def __init__(self, layer, event, chunk_state, comp_stream, comm_stream, extra_ar
event (torch.cuda.Event):
record CUDA event across multiple nodes on different streams for synchronization.
chunk_state (ModelChunkState): model state shared in the model chunk.
- comp_stream (torch.cuda.Stream): CUDA stream for computation.
- comm_stream (torch.cuda.Stream): CUDA stream for communication.
+ comp_stream (Callable): Func that returns CUDA stream for computation.
+ comm_stream (Callable): Func that returns CUDA stream for communication.
extra_args (dict): extra arguments for the layer.
The event and chunk_state are binded to the TransformerModelChunkSchedulePlan
@@ -122,14 +122,13 @@ def _build_callable_nodes(self, event, comp_stream, comm_stream, extra_args):
# get flags for latter use
is_mtp = isinstance(self.layer, MultiTokenPredictionLayer)
- is_moe = (
- isinstance(self.layer.mtp_model_layer.mlp, MoELayer)
- if is_mtp
- else isinstance(self.layer.mlp, MoELayer)
- )
+ transformer_layer = self.layer.mtp_model_layer if is_mtp else self.layer
+ is_moe = isinstance(transformer_layer.mlp, MoELayer)
+ num_local_experts = transformer_layer.mlp.num_local_experts if is_moe else None
extra_args["config"] = self.layer.config
extra_args["is_moe"] = is_moe
+ extra_args["num_local_experts"] = num_local_experts
extra_args["delay_wgrad_compute"] = self.layer.config.delay_wgrad_compute
extra_args["is_mtp"] = is_mtp
@@ -312,9 +311,6 @@ def __init__(
self.post_process = None
self.vp_stage = model.vp_stage
- comp_stream = get_comp_stream()
- comm_stream = get_comm_stream()
-
# save the inputs of model.forward() to ModelChunkState
self._model_chunk_state.input_ids = input_ids
self._model_chunk_state.position_ids = position_ids
@@ -333,18 +329,22 @@ def __init__(
self._model_chunk_state.attention_bias = None
# build preprocess
- self.pre_process = PreProcessNode(model, self._model_chunk_state, self._event, comp_stream)
+ self.pre_process = PreProcessNode(
+ model, self._model_chunk_state, self._event, get_comp_stream
+ )
# build layer schedule plan for each layer.
# The methods to obtain layers are different for MTP so we need the other build plan for
# MTP. Also, this can help annotate MTP layer so that it can know where MTP is.
- self._build_layer_schedule_plan(model.decoder, comp_stream, comm_stream)
- self._build_layer_schedule_plan(getattr(model, "mtp", None), comp_stream, comm_stream)
+ self._build_layer_schedule_plan(model.decoder, get_comp_stream, get_comm_stream)
+ self._build_layer_schedule_plan(
+ getattr(model, "mtp", None), get_comp_stream, get_comm_stream
+ )
# build post process
if model.post_process:
self.post_process = PostProcessNode(
- model, self._model_chunk_state, self._event, comp_stream
+ model, self._model_chunk_state, self._event, get_comp_stream
)
def _build_layer_schedule_plan(self, module, comp_stream, comm_stream):
diff --git a/megatron/core/models/gpt/experimental_attention_variant_module_specs.py b/megatron/core/models/gpt/experimental_attention_variant_module_specs.py
index a7cc7cc0a55..6608073136c 100644
--- a/megatron/core/models/gpt/experimental_attention_variant_module_specs.py
+++ b/megatron/core/models/gpt/experimental_attention_variant_module_specs.py
@@ -392,12 +392,12 @@ def _get_self_attention_module_spec(
moe_grouped_gemm=config.moe_grouped_gemm,
qk_layernorm=config.qk_layernorm,
multi_latent_attention=config.multi_latent_attention,
- moe_use_legacy_grouped_gemm=config.moe_use_legacy_grouped_gemm,
qk_l2_norm=config.qk_l2_norm,
use_kitchen=config.use_kitchen,
use_te_activation_func=config.use_te_activation_func,
use_kitchen_attention=config.use_kitchen_attention,
kitchen_attention_backend=config.kitchen_attention_backend,
+ mla_down_proj_fusion=getattr(config, "mla_down_proj_fusion", False),
)
attn_spec = layer_spec.submodules.self_attention
if config.multi_latent_attention:
@@ -444,7 +444,6 @@ def _get_moe_module_spec(
backend=backend,
num_experts=config.num_moe_experts,
moe_grouped_gemm=config.moe_grouped_gemm,
- moe_use_legacy_grouped_gemm=config.moe_use_legacy_grouped_gemm,
use_te_activation_func=config.use_te_activation_func,
)
moe_spec.metainfo["fuse_pre_mlp_layernorm"] = False
diff --git a/megatron/core/models/gpt/fine_grained_callables.py b/megatron/core/models/gpt/fine_grained_callables.py
index e17ed0a5d40..8d1036b5bae 100644
--- a/megatron/core/models/gpt/fine_grained_callables.py
+++ b/megatron/core/models/gpt/fine_grained_callables.py
@@ -3,7 +3,7 @@
import weakref
from contextlib import nullcontext
from functools import partial
-from typing import Optional
+from typing import Callable, Optional
import torch
from torch import Tensor
@@ -22,6 +22,7 @@
get_mtp_layer_offset,
)
from megatron.core.transformer.transformer_layer import TransformerLayer, make_viewless_tensor
+from megatron.core.typed_torch import apply_module, copy_signature
from megatron.core.utils import internal_api
@@ -43,13 +44,14 @@ def wrapped_func(*args, **kwarg):
@internal_api
-def should_free_input(name, is_moe, config):
+def should_free_input(name, is_moe, config, num_local_experts):
"""Determine if the node should free its input memory.
Args:
name: Node name
is_moe: Whether it's a MoE model
config: TransformerConfig object
+ num_local_experts: Number of local experts in MoE module
Returns:
bool: Whether to free input memory
@@ -70,8 +72,19 @@ def should_free_input(name, is_moe, config):
# when and how to free the input memory.
# The input and output of A2A are not needed anymore after the forward pass,
# so we can free the input memory after the forward pass.
+
+ # When low precision fp8/4 is enabled, the casted tensors are saved and the
+ # original bf16 tensors are safe to be freed.
+ free_mlp = config.fp8 is not None or config.fp4 is not None
+ if not free_mlp:
+ # AlltoAll dispatcher with local_num_experts=1 and HybridEP both use identity
+ # operation for `dispatch_postprocess`, hence the mlp inputs will be directly
+ # passed to GroupedGemm and should be saved for backward pass.
+ free_mlp = num_local_experts > 1 or config.moe_token_dispatcher_type != "alltoall"
+ free_mlp = free_mlp and not enable_hybridep
+
free_input_nodes = {
- "mlp": not enable_hybridep,
+ "mlp": free_mlp,
"moe_combine": True,
# For non-DeepEP and non-HybridEP dispatcher mode, the input is the un-dispatched tokens
# and probs before dispatch A2A and it's not needed anymore after the forward pass
@@ -256,7 +269,8 @@ def __init__(
config = extra_args.get("config", None)
assert config is not None, "model config must be passed to TransformerLayerNode."
is_moe = extra_args.get("is_moe", False)
- free_input = should_free_input(name, is_moe, config)
+ num_local_experts = extra_args.get("num_local_experts", None)
+ free_input = should_free_input(name, is_moe, config, num_local_experts)
self.delay_wgrad_compute = extra_args.get("delay_wgrad_compute", False)
super().__init__(
@@ -316,9 +330,13 @@ def backward_dw(self):
"""Computes the weight gradients for the transformer layer node."""
if not self.delay_wgrad_compute:
return
- with torch.cuda.nvtx.range(f"{self.name} wgrad"):
+ if isinstance(self.stream, Callable):
+ self.stream = self.stream()
+ with torch.cuda.stream(self.stream):
+ torch.cuda.nvtx.range_push(f"{self.name} wgrad")
for module in self.bwd_dw_callables:
module.backward_dw()
+ torch.cuda.nvtx.range_pop()
# the output grad memory is last used in wgrad compute, should be safe to release.
assert self.delay_grads_release, "output grad memory should be valid before wgrad."
@@ -466,13 +484,15 @@ def forward_func(
layer.offload_mlp_norm, hidden_states, "mlp_norm"
) as hidden_states:
pre_mlp_layernorm_output = layer.pre_mlp_norm_checkpoint.checkpoint(
- layer.pre_mlp_layernorm, hidden_states
+ apply_module(layer.pre_mlp_layernorm), hidden_states
)
else:
with off_interface(
layer.offload_mlp_norm, hidden_states, "mlp_norm"
) as hidden_states:
- pre_mlp_layernorm_output = layer.pre_mlp_layernorm(hidden_states)
+ pre_mlp_layernorm_output = apply_module(layer.pre_mlp_layernorm)(
+ hidden_states
+ )
shared_expert_output = layer.mlp.shared_experts_compute(pre_mlp_layernorm_output)
probs, routing_map = layer.mlp.route(pre_mlp_layernorm_output)
@@ -514,6 +534,10 @@ def submodule_dispatch_forward(
token_dispatcher._comm_manager.token_probs = probs
dispatched_tokens, dispatched_probs = layer.mlp.dispatch(local_tokens, probs)
+
+ # `dispatched_probs` is needed by backward pass of swiglu, therefore it's
+ # passed to moe_forward within `layer_state` to avoid the free_input process
+ # of the input tensors.
node.layer_state.dispatched_probs = node.detach(dispatched_probs)
return dispatched_tokens
@@ -531,13 +555,16 @@ def submodule_moe_forward(node: ScheduleNode, dispatched_tokens: torch.Tensor):
expert_output, _ = layer.mlp.routed_experts_compute(dispatched_tokens, dispatched_probs)
+ # For HybridEP, tokens_per_expert is generated on comm stream, as the input to
+ # `routed_experts_compute`, a ref is needed to prevent it from being freed.
+ if enable_hybridep:
+ tokens_per_expert = token_dispatcher._comm_manager.get_number_of_tokens_per_expert()
+ node.layer_state.tokens_per_expert = tokens_per_expert
+
if layer.recompute_pre_mlp_layernorm:
# discard the output of the pre-mlp layernorm and register the recompute
# as a gradient hook of expert_output
layer.pre_mlp_norm_checkpoint.discard_output_and_register_recompute(expert_output)
- # release tensor reference after use
- node.layer_state.dispatched_probs = None
- node.layer_state.pre_mlp_layernorm_output = None
return expert_output
@@ -572,11 +599,14 @@ def submodule_combine_forward(node: ScheduleNode, output: torch.Tensor):
inp=hidden_states, requires_grad=hidden_states.requires_grad, keep_graph=True
)
- # Need to record residual to comm stream, since it's created on comp stream
+ # Need to record tensors created on comp stream to comm stream
node.layer_state.residual.record_stream(torch.cuda.current_stream())
+ if shared_expert_output is not None:
+ shared_expert_output.record_stream(torch.cuda.current_stream())
# release tensor reference after use
node.layer_state.residual = None
+ node.layer_state.shared_expert_output = None
# final layer norm from decoder
final_layernorm = node.chunk_state.model.decoder.final_layernorm
@@ -585,6 +615,7 @@ def submodule_combine_forward(node: ScheduleNode, output: torch.Tensor):
output = make_viewless_tensor(inp=output, requires_grad=True, keep_graph=True)
return output
+ @copy_signature(layer._forward_mlp, handle_first_dst_param='preserve')
def mlp_wrapper(node: ScheduleNode, *args, **kwargs):
"""Wrapper for Dense forward."""
return layer._forward_mlp(*args, **kwargs)
diff --git a/megatron/core/models/gpt/gpt_layer_specs.py b/megatron/core/models/gpt/gpt_layer_specs.py
index b62633c1cc6..0d2ca5fa6a7 100755
--- a/megatron/core/models/gpt/gpt_layer_specs.py
+++ b/megatron/core/models/gpt/gpt_layer_specs.py
@@ -1,8 +1,8 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-
import warnings
from typing import Optional, Union
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.fusions.fused_bias_dropout import get_bias_dropout_add
from megatron.core.models.backends import (
BackendSpecProvider,
@@ -15,6 +15,7 @@
from megatron.core.transformer.identity_op import IdentityOp
from megatron.core.transformer.mlp import MLP, MLPSubmodules
from megatron.core.transformer.multi_latent_attention import (
+ FusedMLASelfAttention,
MLASelfAttention,
MLASelfAttentionSubmodules,
)
@@ -37,17 +38,14 @@
TransformerLayerSubmodules,
get_transformer_layer_offset,
)
+from megatron.core.typed_torch import copy_signature
from megatron.core.utils import is_te_min_version
-try:
- import transformer_engine as te # type: ignore[import-untyped] # pylint: disable=unused-import
-
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import TEFusedMLP, TENorm
from megatron.core.extensions.transformer_engine_spec_provider import TESpecProvider
-
- HAVE_TE = True
-except ImportError:
- HAVE_TE = False
+else:
+ TEFusedMLP, TENorm, TESpecProvider = None, None, None
try:
from megatron.core.extensions.kitchen import HAVE_KITCHEN, KitchenSpecProvider
@@ -72,12 +70,15 @@
HAVE_APEX = False
-def get_gpt_layer_with_inference_spec(
+def get_gpt_layer_with_inference_submodules(
qk_layernorm: Optional[bool] = False,
multi_latent_attention: Optional[bool] = False,
qk_l2_norm: Optional[bool] = False,
-) -> ModuleSpec:
- """Use this spec to use inference optimized linear layers.
+ num_experts: Optional[int] = None,
+ moe_grouped_gemm: Optional[bool] = False,
+ moe_use_legacy_grouped_gemm: Optional[bool] = False,
+) -> TransformerLayerSubmodules:
+ """Use these submodules for inference optimized linear layers.
Args:
qk_layernorm (bool, optional): To use layernorm for queries/keys. Defaults to False.
multi_latent_attention (bool, optional): To use MLA. Defaults to False.
@@ -88,9 +89,8 @@ def get_gpt_layer_with_inference_spec(
mlp = get_mlp_module_spec_for_backend(
backend=backend,
- num_experts=None,
- moe_grouped_gemm=False,
- moe_use_legacy_grouped_gemm=False,
+ num_experts=num_experts,
+ moe_grouped_gemm=moe_grouped_gemm,
use_te_op_fuser=False,
use_te_activation_func=False,
)
@@ -107,82 +107,85 @@ def get_gpt_layer_with_inference_spec(
if qk_layernorm
else backend.column_parallel_linear()
)
- return ModuleSpec(
- module=TransformerLayer,
- submodules=TransformerLayerSubmodules(
- input_layernorm=backend.layer_norm(),
- self_attention=ModuleSpec(
- module=MLASelfAttention,
- params={"attn_mask_type": AttnMaskType.causal},
- submodules=MLASelfAttentionSubmodules(
- linear_q_proj=backend.column_parallel_linear(),
- linear_q_down_proj=backend.linear(),
- linear_q_up_proj=linear_q_up_proj,
- linear_kv_down_proj=backend.linear(),
- linear_kv_up_proj=linear_kv_up_proj,
- core_attention=backend.core_attention(),
- linear_proj=backend.row_parallel_linear(),
- q_layernorm=IdentityOp,
- kv_layernorm=IdentityOp,
- ),
+ return TransformerLayerSubmodules(
+ input_layernorm=backend.layer_norm(has_residual=True),
+ self_attention=ModuleSpec(
+ module=MLASelfAttention,
+ params={"attn_mask_type": AttnMaskType.causal},
+ submodules=MLASelfAttentionSubmodules(
+ linear_q_proj=backend.column_parallel_linear(),
+ linear_q_down_proj=backend.linear(),
+ linear_q_up_proj=linear_q_up_proj,
+ linear_kv_down_proj=backend.linear(),
+ linear_kv_up_proj=linear_kv_up_proj,
+ core_attention=backend.core_attention(),
+ linear_proj=backend.row_parallel_linear(),
+ q_layernorm=IdentityOp,
+ kv_layernorm=IdentityOp,
),
- self_attn_bda=get_bias_dropout_add,
- pre_mlp_layernorm=IdentityOp,
- mlp=mlp,
- mlp_bda=get_bias_dropout_add,
),
+ self_attn_bda=get_bias_dropout_add,
+ pre_mlp_layernorm=IdentityOp,
+ mlp=mlp,
+ mlp_bda=get_bias_dropout_add,
)
else:
qk_norm = backend.layer_norm(for_qk=True)
- return ModuleSpec(
- module=TransformerLayer,
- submodules=TransformerLayerSubmodules(
- self_attention=ModuleSpec(
- module=SelfAttention,
- params={"attn_mask_type": AttnMaskType.causal},
- submodules=SelfAttentionSubmodules(
- linear_qkv=backend.column_parallel_layer_norm_linear(),
- core_attention=backend.core_attention(),
- linear_proj=backend.row_parallel_linear(),
- q_layernorm=(
- L2Norm if qk_l2_norm else (qk_norm if qk_layernorm else IdentityOp)
- ),
- k_layernorm=(
- L2Norm if qk_l2_norm else (qk_norm if qk_layernorm else IdentityOp)
- ),
+ return TransformerLayerSubmodules(
+ self_attention=ModuleSpec(
+ module=SelfAttention,
+ params={"attn_mask_type": AttnMaskType.causal},
+ submodules=SelfAttentionSubmodules(
+ linear_qkv=backend.column_parallel_layer_norm_linear(),
+ core_attention=backend.core_attention(),
+ linear_proj=backend.row_parallel_linear(),
+ q_layernorm=(
+ L2Norm if qk_l2_norm else (qk_norm if qk_layernorm else IdentityOp)
+ ),
+ k_layernorm=(
+ L2Norm if qk_l2_norm else (qk_norm if qk_layernorm else IdentityOp)
),
),
- self_attn_bda=get_bias_dropout_add,
- pre_mlp_layernorm=IdentityOp,
- mlp=mlp,
- mlp_bda=get_bias_dropout_add,
- sharded_state_dict_keys_map={
- "mlp.0.weight": "mlp.linear_fc1.layer_norm_weight",
- "mlp.0.bias": "mlp.linear_fc1.layer_norm_bias",
- "mlp.1.basic_ops.0.weight": "mlp.linear_fc1.weight",
- "mlp.1.basic_ops.1.bias": "mlp.linear_fc1.bias",
- "mlp.3.basic_ops.0.weight": "mlp.linear_fc2.weight",
- "mlp.3.basic_ops.1.bias": "mlp.linear_fc2.bias",
- },
),
+ self_attn_bda=get_bias_dropout_add,
+ pre_mlp_layernorm=backend.layer_norm() if num_experts else IdentityOp,
+ mlp=mlp,
+ mlp_bda=get_bias_dropout_add,
+ sharded_state_dict_keys_map={
+ "mlp.0.weight": "mlp.linear_fc1.layer_norm_weight",
+ "mlp.0.bias": "mlp.linear_fc1.layer_norm_bias",
+ "mlp.1.basic_ops.0.weight": "mlp.linear_fc1.weight",
+ "mlp.1.basic_ops.1.bias": "mlp.linear_fc1.bias",
+ "mlp.3.basic_ops.0.weight": "mlp.linear_fc2.weight",
+ "mlp.3.basic_ops.1.bias": "mlp.linear_fc2.bias",
+ },
)
-def get_gpt_layer_with_transformer_engine_spec(
+@copy_signature(get_gpt_layer_with_inference_submodules)
+def get_gpt_layer_with_inference_spec(*args, **kwargs) -> ModuleSpec:
+ """Use this spec to use inference optimized linear layers."""
+ return ModuleSpec(
+ module=TransformerLayer, submodules=get_gpt_layer_with_inference_submodules(*args, **kwargs)
+ )
+
+
+def get_gpt_layer_with_transformer_engine_submodules(
num_experts: Optional[int] = None,
moe_grouped_gemm: Optional[bool] = False,
qk_layernorm: Optional[bool] = False,
multi_latent_attention: Optional[bool] = False,
fp8: Optional[str] = None, # pylint: disable=unused-argument
- moe_use_legacy_grouped_gemm: Optional[bool] = False,
qk_l2_norm: Optional[bool] = False,
use_te_op_fuser: Optional[bool] = False,
use_kitchen: bool = False,
use_te_activation_func: bool = False,
use_kitchen_attention: bool = False,
kitchen_attention_backend: str = "sdpa",
-) -> ModuleSpec:
- """Use this spec to use lower-level Transformer Engine modules (required for fp8 training).
+ mla_down_proj_fusion: bool = False,
+) -> TransformerLayerSubmodules:
+ """Use these submodules to use lower-level Transformer Engine modules (required for fp8
+ training).
Args:
@@ -191,14 +194,15 @@ def get_gpt_layer_with_transformer_engine_spec(
qk_layernorm (bool, optional): To use layernorm for queries/keys. Defaults to False.
multi_latent_attention (bool, optional): To use MLA. Defaults to False.
fp8 (str, optional): Deprecated. For temporary Nemo compatibility.
- moe_use_legacy_grouped_gemm (bool, optional): Force use the legacy GroupedMLP.
- Defaults to False.
qk_l2_norm (bool, optional): To use l2 norm for queries/keys. Defaults to False.
use_te_op_fuser (bool, optional): Use Transformer Engine's operation-based API, which may
enable certain operation fusions. Defaults to False.
+ mla_down_proj_fusion (bool, optional): Enable fused q/kv down-projection and fused input
+ layernorm when backend supports. Otherwise fall back
+ to the unfused MLA.
Returns:
- ModuleSpec: Module specification with TE modules
+ TransformerLayerSubmodules: TE modules to construct a TransformerLayer
"""
if fp8 is not None:
@@ -225,7 +229,6 @@ def get_gpt_layer_with_transformer_engine_spec(
backend=backend,
num_experts=num_experts,
moe_grouped_gemm=moe_grouped_gemm,
- moe_use_legacy_grouped_gemm=moe_use_legacy_grouped_gemm,
use_te_op_fuser=use_te_op_fuser,
use_te_activation_func=use_te_activation_func,
)
@@ -242,18 +245,24 @@ def get_gpt_layer_with_transformer_engine_spec(
if qk_layernorm
else backend.column_parallel_linear()
)
- return ModuleSpec(
- module=TransformerLayer,
- submodules=TransformerLayerSubmodules(
- input_layernorm=backend.layer_norm(),
+
+ if mla_down_proj_fusion:
+ fuse_input_layernorm = backend.column_parallel_layer_norm_linear() is not None
+ input_layernorm = IdentityOp if fuse_input_layernorm else backend.layer_norm()
+ down_proj_linear = (
+ backend.column_parallel_layer_norm_linear()
+ if fuse_input_layernorm
+ else backend.linear()
+ )
+ return TransformerLayerSubmodules(
+ input_layernorm=input_layernorm,
self_attention=ModuleSpec(
- module=MLASelfAttention,
+ module=FusedMLASelfAttention,
params={"attn_mask_type": AttnMaskType.causal},
submodules=MLASelfAttentionSubmodules(
linear_q_proj=backend.column_parallel_linear(),
- linear_q_down_proj=backend.linear(),
+ linear_qkv_down_proj=down_proj_linear,
linear_q_up_proj=linear_q_up_proj,
- linear_kv_down_proj=backend.linear(),
linear_kv_up_proj=linear_kv_up_proj,
core_attention=backend.core_attention(),
linear_proj=backend.row_parallel_linear(),
@@ -265,58 +274,93 @@ def get_gpt_layer_with_transformer_engine_spec(
pre_mlp_layernorm=backend.layer_norm() if num_experts else IdentityOp,
mlp=mlp,
mlp_bda=get_bias_dropout_add,
+ sharded_state_dict_keys_map=(
+ {
+ "self_attention.linear_q_down_proj.layer_norm_": "input_layernorm.",
+ "self_attention.linear_kv_down_proj.layer_norm_": "input_layernorm.",
+ "self_attention.linear_qkv_down_proj.layer_norm_": "input_layernorm.",
+ }
+ if fuse_input_layernorm
+ else {}
+ ),
+ )
+ return TransformerLayerSubmodules(
+ input_layernorm=backend.layer_norm(has_residual=True),
+ self_attention=ModuleSpec(
+ module=MLASelfAttention,
+ params={"attn_mask_type": AttnMaskType.causal},
+ submodules=MLASelfAttentionSubmodules(
+ linear_q_proj=backend.column_parallel_linear(),
+ linear_q_down_proj=backend.linear(),
+ linear_q_up_proj=linear_q_up_proj,
+ linear_kv_down_proj=backend.linear(),
+ linear_kv_up_proj=linear_kv_up_proj,
+ core_attention=backend.core_attention(),
+ linear_proj=backend.row_parallel_linear(),
+ q_layernorm=IdentityOp,
+ kv_layernorm=IdentityOp,
+ ),
),
+ self_attn_bda=get_bias_dropout_add,
+ pre_mlp_layernorm=backend.layer_norm(has_residual=True) if num_experts else IdentityOp,
+ mlp=mlp,
+ mlp_bda=get_bias_dropout_add,
)
else:
qk_norm = backend.layer_norm(for_qk=True)
- return ModuleSpec(
- module=TransformerLayer,
- submodules=TransformerLayerSubmodules(
- self_attention=ModuleSpec(
- module=SelfAttention,
- params={"attn_mask_type": AttnMaskType.causal},
- submodules=SelfAttentionSubmodules(
- linear_qkv=backend.column_parallel_layer_norm_linear(),
- core_attention=backend.core_attention(),
- linear_proj=backend.row_parallel_linear(),
- q_layernorm=(
- L2Norm if qk_l2_norm else (qk_norm if qk_layernorm else IdentityOp)
- ),
- k_layernorm=(
- L2Norm if qk_l2_norm else (qk_norm if qk_layernorm else IdentityOp)
- ),
+ return TransformerLayerSubmodules(
+ self_attention=ModuleSpec(
+ module=SelfAttention,
+ params={"attn_mask_type": AttnMaskType.causal},
+ submodules=SelfAttentionSubmodules(
+ linear_qkv=backend.column_parallel_layer_norm_linear(),
+ core_attention=backend.core_attention(),
+ linear_proj=backend.row_parallel_linear(),
+ q_layernorm=(
+ L2Norm if qk_l2_norm else (qk_norm if qk_layernorm else IdentityOp)
+ ),
+ k_layernorm=(
+ L2Norm if qk_l2_norm else (qk_norm if qk_layernorm else IdentityOp)
),
),
- self_attn_bda=get_bias_dropout_add,
- pre_mlp_layernorm=backend.layer_norm() if num_experts else IdentityOp,
- mlp=mlp,
- mlp_bda=get_bias_dropout_add,
- sharded_state_dict_keys_map={
- "mlp.0.weight": "mlp.linear_fc1.layer_norm_weight",
- "mlp.0.bias": "mlp.linear_fc1.layer_norm_bias",
- "mlp.1.basic_ops.0.weight": "mlp.linear_fc1.weight",
- "mlp.1.basic_ops.1.bias": "mlp.linear_fc1.bias",
- "mlp.3.basic_ops.0.weight": "mlp.linear_fc2.weight",
- "mlp.3.basic_ops.1.bias": "mlp.linear_fc2.bias",
- },
),
+ self_attn_bda=get_bias_dropout_add,
+ pre_mlp_layernorm=backend.layer_norm(has_residual=True) if num_experts else IdentityOp,
+ mlp=mlp,
+ mlp_bda=get_bias_dropout_add,
+ sharded_state_dict_keys_map={
+ "mlp.0.weight": "mlp.linear_fc1.layer_norm_weight",
+ "mlp.0.bias": "mlp.linear_fc1.layer_norm_bias",
+ "mlp.1.basic_ops.0.weight": "mlp.linear_fc1.weight",
+ "mlp.1.basic_ops.1.bias": "mlp.linear_fc1.bias",
+ "mlp.3.basic_ops.0.weight": "mlp.linear_fc2.weight",
+ "mlp.3.basic_ops.1.bias": "mlp.linear_fc2.bias",
+ },
)
-def get_gpt_layer_local_spec(
+@copy_signature(get_gpt_layer_with_transformer_engine_submodules)
+def get_gpt_layer_with_transformer_engine_spec(*args, **kwargs) -> ModuleSpec:
+ """Use this spec to use lower-level Transformer Engine modules (required for fp8 training)."""
+ return ModuleSpec(
+ module=TransformerLayer,
+ submodules=get_gpt_layer_with_transformer_engine_submodules(*args, **kwargs),
+ )
+
+
+def get_gpt_layer_local_submodules(
num_experts: Optional[int] = None,
moe_grouped_gemm: Optional[bool] = False,
qk_layernorm: Optional[bool] = False,
multi_latent_attention: Optional[bool] = False,
fp8: Optional[str] = None, # pylint: disable=unused-argument
- moe_use_legacy_grouped_gemm: Optional[bool] = False,
normalization: Optional[str] = None,
qk_l2_norm: Optional[bool] = False,
use_kitchen: bool = False,
use_kitchen_attention: bool = False,
kitchen_attention_backend: str = "sdpa",
-) -> ModuleSpec:
- """Use this spec for an implementation using only modules in Megatron-Core.
+) -> TransformerLayerSubmodules:
+ """Use these submodules for an implementation using only modules in Megatron-Core.
Args:
@@ -325,12 +369,10 @@ def get_gpt_layer_local_spec(
qk_layernorm (bool, optional): To use layernorm for queries/keys. Defaults to False.
multi_latent_attention (bool, optional): To use MLA. Defaults to False.
fp8 (str, optional): Deprecated. For temporary Nemo compatibility.
- moe_use_legacy_grouped_gemm (bool, optional): Force use the legacy GroupedMLP.
- Defaults to False.
qk_l2_norm (bool, optional): To use l2 norm for queries/keys. Defaults to False.
Returns:
- ModuleSpec: Module specification with Megatron-Core modules
+ TransformerLayerSubmodules: Megatron-Core modules to construct a TransformerLayer
"""
if use_kitchen:
@@ -344,10 +386,10 @@ def get_gpt_layer_local_spec(
backend = LocalSpecProvider()
# Adjust for RMS norm.
if normalization == "RMSNorm":
- layer_norm = backend.layer_norm(rms_norm=True, for_qk=False)
+ layer_norm = backend.layer_norm(rms_norm=True, for_qk=False, has_residual=True)
qk_norm = backend.layer_norm(rms_norm=True, for_qk=True)
else:
- layer_norm = backend.layer_norm(rms_norm=False, for_qk=False)
+ layer_norm = backend.layer_norm(rms_norm=False, for_qk=False, has_residual=True)
qk_norm = backend.layer_norm(rms_norm=False, for_qk=True)
if fp8 is not None:
@@ -357,77 +399,75 @@ def get_gpt_layer_local_spec(
)
mlp = get_mlp_module_spec_for_backend(
- backend=backend,
- num_experts=num_experts,
- moe_grouped_gemm=moe_grouped_gemm,
- moe_use_legacy_grouped_gemm=moe_use_legacy_grouped_gemm,
+ backend=backend, num_experts=num_experts, moe_grouped_gemm=moe_grouped_gemm
)
if multi_latent_attention:
assert qk_l2_norm is False, "qk_l2_norm is not supported with MLA."
- return ModuleSpec(
- module=TransformerLayer,
- submodules=TransformerLayerSubmodules(
- input_layernorm=layer_norm,
- self_attention=ModuleSpec(
- module=MLASelfAttention,
- params={"attn_mask_type": AttnMaskType.causal},
- submodules=MLASelfAttentionSubmodules(
- linear_q_proj=backend.column_parallel_linear(),
- linear_q_down_proj=backend.column_parallel_linear(),
- linear_q_up_proj=backend.column_parallel_linear(),
- linear_kv_down_proj=backend.column_parallel_linear(),
- linear_kv_up_proj=backend.column_parallel_linear(),
- core_attention=backend.core_attention(),
- linear_proj=backend.row_parallel_linear(),
- q_layernorm=qk_norm if qk_layernorm else IdentityOp,
- kv_layernorm=qk_norm if qk_layernorm else IdentityOp,
- ),
+ return TransformerLayerSubmodules(
+ input_layernorm=layer_norm,
+ self_attention=ModuleSpec(
+ module=MLASelfAttention,
+ params={"attn_mask_type": AttnMaskType.causal},
+ submodules=MLASelfAttentionSubmodules(
+ linear_q_proj=backend.column_parallel_linear(),
+ linear_q_down_proj=backend.column_parallel_linear(),
+ linear_q_up_proj=backend.column_parallel_linear(),
+ linear_kv_down_proj=backend.column_parallel_linear(),
+ linear_kv_up_proj=backend.column_parallel_linear(),
+ core_attention=backend.core_attention(),
+ linear_proj=backend.row_parallel_linear(),
+ q_layernorm=qk_norm if qk_layernorm else IdentityOp,
+ kv_layernorm=qk_norm if qk_layernorm else IdentityOp,
),
- self_attn_bda=get_bias_dropout_add,
- pre_mlp_layernorm=layer_norm,
- mlp=mlp,
- mlp_bda=get_bias_dropout_add,
),
+ self_attn_bda=get_bias_dropout_add,
+ pre_mlp_layernorm=layer_norm,
+ mlp=mlp,
+ mlp_bda=get_bias_dropout_add,
)
else:
- return ModuleSpec(
- module=TransformerLayer,
- submodules=TransformerLayerSubmodules(
- input_layernorm=layer_norm,
- self_attention=ModuleSpec(
- module=SelfAttention,
- params={"attn_mask_type": AttnMaskType.causal},
- submodules=SelfAttentionSubmodules(
- linear_qkv=backend.column_parallel_linear(),
- core_attention=backend.core_attention(),
- linear_proj=backend.row_parallel_linear(),
- q_layernorm=(
- L2Norm if qk_l2_norm else (qk_norm if qk_layernorm else IdentityOp)
- ),
- k_layernorm=(
- L2Norm if qk_l2_norm else (qk_norm if qk_layernorm else IdentityOp)
- ),
+ return TransformerLayerSubmodules(
+ input_layernorm=layer_norm,
+ self_attention=ModuleSpec(
+ module=SelfAttention,
+ params={"attn_mask_type": AttnMaskType.causal},
+ submodules=SelfAttentionSubmodules(
+ linear_qkv=backend.column_parallel_linear(),
+ core_attention=backend.core_attention(),
+ linear_proj=backend.row_parallel_linear(),
+ q_layernorm=(
+ L2Norm if qk_l2_norm else (qk_norm if qk_layernorm else IdentityOp)
+ ),
+ k_layernorm=(
+ L2Norm if qk_l2_norm else (qk_norm if qk_layernorm else IdentityOp)
),
),
- self_attn_bda=get_bias_dropout_add,
- pre_mlp_layernorm=layer_norm,
- mlp=mlp,
- mlp_bda=get_bias_dropout_add,
- sharded_state_dict_keys_map={
- "input_layernorm.": "self_attention.linear_qkv.layer_norm_",
- "pre_mlp_layernorm.": "mlp.linear_fc1.layer_norm_",
- },
),
+ self_attn_bda=get_bias_dropout_add,
+ pre_mlp_layernorm=layer_norm,
+ mlp=mlp,
+ mlp_bda=get_bias_dropout_add,
+ sharded_state_dict_keys_map={
+ "input_layernorm.": "self_attention.linear_qkv.layer_norm_",
+ "pre_mlp_layernorm.": "mlp.linear_fc1.layer_norm_",
+ },
)
+@copy_signature(get_gpt_layer_local_submodules)
+def get_gpt_layer_local_spec(*args, **kwargs) -> ModuleSpec:
+ """Use this spec for an implementation using only modules in Megatron-Core."""
+ return ModuleSpec(
+ module=TransformerLayer, submodules=get_gpt_layer_local_submodules(*args, **kwargs)
+ )
+
+
def _get_mlp_module_spec(
use_te: Optional[bool] = True,
num_experts: Optional[int] = None,
moe_grouped_gemm: Optional[bool] = False,
fp8: Optional[str] = None, # pylint: disable=unused-argument
- moe_use_legacy_grouped_gemm: Optional[bool] = False,
):
warnings.warn(
"""This private function is on a deprecation track. Please switch to `get_mlp_module_spec`
@@ -435,11 +475,7 @@ def _get_mlp_module_spec(
)
return get_mlp_module_spec(
- use_te=use_te,
- num_experts=num_experts,
- moe_grouped_gemm=moe_grouped_gemm,
- fp8=fp8,
- moe_use_legacy_grouped_gemm=moe_use_legacy_grouped_gemm,
+ use_te=use_te, num_experts=num_experts, moe_grouped_gemm=moe_grouped_gemm, fp8=fp8
)
@@ -448,7 +484,6 @@ def get_mlp_module_spec(
num_experts: Optional[int] = None,
moe_grouped_gemm: Optional[bool] = False,
fp8: Optional[str] = None, # pylint: disable=unused-argument
- moe_use_legacy_grouped_gemm: Optional[bool] = False,
use_te_op_fuser: Optional[bool] = False,
) -> ModuleSpec:
"""Helper function to get module spec for MLP/MoE"""
@@ -471,7 +506,6 @@ def get_mlp_module_spec(
backend=TESpecProvider() if use_te else LocalSpecProvider(),
num_experts=num_experts,
moe_grouped_gemm=moe_grouped_gemm,
- moe_use_legacy_grouped_gemm=moe_use_legacy_grouped_gemm,
use_te_op_fuser=use_te_op_fuser,
)
@@ -480,7 +514,6 @@ def get_mlp_module_spec_for_backend(
backend: BackendSpecProvider,
num_experts: Optional[int] = None,
moe_grouped_gemm: Optional[bool] = False,
- moe_use_legacy_grouped_gemm: Optional[bool] = False,
use_te_op_fuser: Optional[bool] = False,
use_te_activation_func: bool = False,
) -> ModuleSpec:
@@ -509,7 +542,6 @@ def get_mlp_module_spec_for_backend(
backend=backend,
num_experts=num_experts,
moe_grouped_gemm=moe_grouped_gemm,
- moe_use_legacy_grouped_gemm=moe_use_legacy_grouped_gemm,
use_te_activation_func=use_te_activation_func,
)
@@ -530,24 +562,39 @@ def get_gpt_decoder_layer_specs(
moe_grouped_gemm=False,
qk_layernorm=config.qk_layernorm,
multi_latent_attention=config.multi_latent_attention,
- moe_use_legacy_grouped_gemm=config.moe_use_legacy_grouped_gemm,
qk_l2_norm=qk_l2_norm,
use_kitchen=config.use_kitchen,
use_te_activation_func=config.use_te_activation_func,
use_kitchen_attention=config.use_kitchen_attention,
kitchen_attention_backend=config.kitchen_attention_backend,
+ mla_down_proj_fusion=getattr(config, "mla_down_proj_fusion", False),
)
moe_layer_spec = get_gpt_layer_with_transformer_engine_spec(
num_experts=config.num_moe_experts,
moe_grouped_gemm=config.moe_grouped_gemm,
qk_layernorm=config.qk_layernorm,
multi_latent_attention=config.multi_latent_attention,
- moe_use_legacy_grouped_gemm=config.moe_use_legacy_grouped_gemm,
qk_l2_norm=qk_l2_norm,
use_kitchen=config.use_kitchen,
use_te_activation_func=config.use_te_activation_func,
use_kitchen_attention=config.use_kitchen_attention,
kitchen_attention_backend=config.kitchen_attention_backend,
+ mla_down_proj_fusion=getattr(config, "mla_down_proj_fusion", False),
+ )
+ elif config.transformer_impl == "inference_optimized":
+ layer_norm_impl = TENorm
+ dense_layer_spec = get_gpt_layer_with_inference_spec(
+ qk_layernorm=config.qk_layernorm,
+ multi_latent_attention=config.multi_latent_attention,
+ qk_l2_norm=qk_l2_norm,
+ )
+ moe_layer_spec = get_gpt_layer_with_inference_spec(
+ qk_layernorm=config.qk_layernorm,
+ multi_latent_attention=config.multi_latent_attention,
+ qk_l2_norm=qk_l2_norm,
+ num_experts=config.num_moe_experts,
+ moe_grouped_gemm=config.moe_grouped_gemm,
+ moe_use_legacy_grouped_gemm=config.moe_use_legacy_grouped_gemm,
)
else:
layer_norm_impl = LNImpl
@@ -556,7 +603,6 @@ def get_gpt_decoder_layer_specs(
moe_grouped_gemm=False,
qk_layernorm=config.qk_layernorm,
multi_latent_attention=config.multi_latent_attention,
- moe_use_legacy_grouped_gemm=config.moe_use_legacy_grouped_gemm,
normalization=normalization,
qk_l2_norm=qk_l2_norm,
use_kitchen=config.use_kitchen,
@@ -568,7 +614,6 @@ def get_gpt_decoder_layer_specs(
moe_grouped_gemm=config.moe_grouped_gemm,
qk_layernorm=config.qk_layernorm,
multi_latent_attention=config.multi_latent_attention,
- moe_use_legacy_grouped_gemm=config.moe_use_legacy_grouped_gemm,
normalization=normalization,
qk_l2_norm=qk_l2_norm,
use_kitchen=config.use_kitchen,
@@ -640,6 +685,8 @@ def get_gpt_decoder_block_spec(
if use_transformer_engine:
layer_norm_impl = TENorm
+ elif config.transformer_impl == "inference_optimized":
+ layer_norm_impl = TENorm
else:
layer_norm_impl = LNImpl
# Block spec.
diff --git a/megatron/core/models/gpt/gpt_model.py b/megatron/core/models/gpt/gpt_model.py
index 4b96465a31e..d63b2c1ddfa 100644
--- a/megatron/core/models/gpt/gpt_model.py
+++ b/megatron/core/models/gpt/gpt_model.py
@@ -217,7 +217,10 @@ def __init__(
if self.mtp_process:
self.mtp = MultiTokenPredictionBlock(
- config=self.config, spec=self.mtp_block_spec, vp_stage=vp_stage
+ config=self.config,
+ spec=self.mtp_block_spec,
+ vp_stage=vp_stage,
+ pg_collection=self.pg_collection,
)
# Output
@@ -242,7 +245,11 @@ def __init__(
config.hidden_size,
self.vocab_size,
config=config,
- init_method=config.init_method,
+ init_method=(
+ config.embedding_init_method
+ if config.use_mup and not self.share_embeddings_and_output_weights
+ else config.init_method
+ ),
bias=False,
skip_bias_add=False,
gather_output=not self.parallel_output,
@@ -444,7 +451,7 @@ def _preprocess(
# return this extra tensor
# this is for backwards compatibility with
# legacy unit tests, which break if you
- # return a 6 tuple instead of 5.
+ # return a 7 tuple instead of 6.
preproc_output += (rotary_pos_cos_sin,)
return preproc_output
@@ -482,6 +489,7 @@ def forward(
inference_params: Optional[BaseInferenceContext] = None,
loss_mask: Optional[Tensor] = None,
padding_mask: Optional[Tensor] = None,
+ is_spec_decode: Optional[bool] = None,
) -> Tensor:
"""Forward function of the GPT Model This function passes the input tensors
through the embedding layer, and then the decoder and finally into the post
@@ -495,6 +503,9 @@ def forward(
padding_mask (Tensor, optional): Padding mask for MoE routing.
Shape [bsz, seq_length]. True = padding (exclude), False = valid (include).
Only used for MoE layers to exclude padding tokens from routing computations.
+ is_spec_decode (bool, optional): Explicitly override whether speculative
+ decoding is active. When ``None`` (default) the flag is inferred from
+ ``inference_context.num_speculative_tokens``.
"""
if self.config.fine_grained_activation_offloading:
self.preprocess_for_fine_grained_offloading()
@@ -554,6 +565,7 @@ def forward(
runtime_gather_output=runtime_gather_output,
extra_block_kwargs=extra_block_kwargs,
inference_context=inference_context,
+ is_spec_decode=is_spec_decode,
)
def _postprocess(
@@ -575,6 +587,7 @@ def _postprocess(
runtime_gather_output=None,
extra_block_kwargs=None,
inference_context=None,
+ is_spec_decode=None,
):
"""Postprocesses decoder hidden states to generate logits or compute loss.
@@ -585,17 +598,27 @@ def _postprocess(
if in_inference_mode:
assert runtime_gather_output, "Inference must always gather TP logits"
+ # Check if speculative decoding is active. When it is, MTP must be
+ # computed *after* verification so that it is conditioned on verified
+ # tokens rather than stale speculative tokens from the previous step.
+ if is_spec_decode is None:
+ is_spec_decode = (
+ in_inference_mode
+ and inference_context.is_dynamic_batching()
+ and inference_context.num_speculative_tokens > 0
+ )
+
# logits and loss
output_weight = None
if self.share_embeddings_and_output_weights:
output_weight = self.shared_embedding_or_output_weight()
- if mtp_in_postprocess:
+ if mtp_in_postprocess and not (in_inference_mode or is_spec_decode):
hidden_states = self.mtp(
input_ids=input_ids,
position_ids=position_ids,
hidden_states=hidden_states,
attention_mask=attention_mask,
- inference_params=inference_params,
+ inference_params=None, # MTP layers don't use KV cache
rotary_pos_emb=rotary_pos_emb,
rotary_pos_cos=rotary_pos_cos,
rotary_pos_sin=rotary_pos_sin,
@@ -608,23 +631,31 @@ def _postprocess(
if not self.post_process:
return hidden_states
- if self.config.mtp_num_layers is not None:
- hidden_states = process_mtp_loss(
- hidden_states=hidden_states,
- labels=labels,
- loss_mask=loss_mask,
- output_layer=self.output_layer,
- output_weight=output_weight,
- runtime_gather_output=runtime_gather_output,
- is_training=self.training,
- compute_language_model_loss=self.compute_language_model_loss,
- config=self.config,
- cp_group=self.pg_collection.cp,
- packed_seq_params=packed_seq_params,
- )
+ if self.config.mtp_num_layers:
+ assert self.config.mtp_num_layers > 0
+ if in_inference_mode or is_spec_decode:
+ # Cache decoder hidden states for serial MTP computation
+ # after speculative token verification.
+ self._decoder_hidden_states_cache = hidden_states
+ else:
+ # In training/eval, use the utility function for processing MTP loss/scaling.
+ hidden_states = process_mtp_loss(
+ hidden_states=hidden_states,
+ labels=labels,
+ loss_mask=loss_mask,
+ output_layer=self.output_layer,
+ output_weight=output_weight,
+ runtime_gather_output=runtime_gather_output,
+ is_training=self.training,
+ compute_language_model_loss=self.compute_language_model_loss,
+ config=self.config,
+ cp_group=self.pg_collection.cp,
+ packed_seq_params=packed_seq_params,
+ scale_logits_fn=self._scale_logits if self.config.use_mup else None,
+ )
sequence_parallel_override = False
- if in_inference_mode and inference_context.materialize_only_last_token_logits:
+ if in_inference_mode and inference_context.config.materialize_only_last_token_logits:
if inference_context.is_static_batching():
hidden_states = hidden_states[-1:, :, :]
else:
@@ -638,23 +669,24 @@ def _postprocess(
self.output_layer.sequence_parallel = False
sequence_parallel_override = True
- # Reshape [B, 1, H] to [1, B, H] → extract each sample’s true last‐token hidden
- # state ([B, H]) → unsqueeze back to [B, 1, H]
- # (so that the output layer, which expects S×B×H, receives only the final token)
- hidden_states = inference_context.last_token_logits(
- hidden_states.squeeze(1).unsqueeze(0)
- ).unsqueeze(1)
+ # Reshape [S, B, H] (with B=1) to [1, S, H] for logit extraction,
+ # then back to [S’, B, H] for the output layer.
+ reshaped = hidden_states.squeeze(1).unsqueeze(0)
+ hidden_states = inference_context.last_token_logits(reshaped).unsqueeze(1)
logits, _ = self.output_layer(
hidden_states, weight=output_weight, runtime_gather_output=runtime_gather_output
)
+ # Apply MuP output scaling to logits
+ logits = self._scale_logits(logits)
+
# Restore sequence parallel execution to the output layer if necessary.
if sequence_parallel_override:
assert (
in_inference_mode
and inference_context.is_dynamic_batching()
- and inference_context.materialize_only_last_token_logits
+ and inference_context.config.materialize_only_last_token_logits
)
self.output_layer.sequence_parallel = True
@@ -678,6 +710,49 @@ def _postprocess(
return loss
+ @torch.inference_mode()
+ def compute_mtp_single_step(
+ self,
+ hidden_states: Tensor,
+ next_token_ids: Tensor,
+ position_ids: Tensor,
+ depth: int,
+ runtime_gather_output: bool = True,
+ ) -> tuple:
+ """Compute a single MTP depth for speculative decoding.
+
+ This is called after speculative token verification to compute MTP
+ predictions conditioned on verified tokens only.
+
+ Args:
+ hidden_states (Tensor): Hidden states at last accepted positions [N, 1, H].
+ next_token_ids (Tensor): Correct next token IDs [1, N].
+ position_ids (Tensor): Position IDs for the next tokens [1, N].
+ depth (int): MTP depth index (0-indexed).
+ runtime_gather_output (bool): Whether to gather output across TP.
+
+ Returns:
+ tuple: (new_hidden_states [N, 1, H], logits [N, 1, vocab_size]).
+ """
+ layer_idx = 0 if self.mtp.mtp_use_repeated_layer else depth
+ mtp_hidden = self.mtp.layers[layer_idx].forward_single_position(
+ hidden_states=hidden_states,
+ next_token_ids=next_token_ids,
+ position_ids=position_ids,
+ embedding=self.embedding,
+ )
+
+ output_weight = None
+ if self.share_embeddings_and_output_weights:
+ output_weight = self.shared_embedding_or_output_weight()
+
+ logits, _ = self.output_layer(
+ mtp_hidden, weight=output_weight, runtime_gather_output=runtime_gather_output
+ )
+ logits = self._scale_logits(logits)
+
+ return mtp_hidden, logits
+
def build_schedule_plan(
self,
input_ids: Tensor,
diff --git a/megatron/core/models/gpt/heterogeneous/heterogeneous_layer_specs.py b/megatron/core/models/gpt/heterogeneous/heterogeneous_layer_specs.py
index 5e9687b09a3..f4385429422 100644
--- a/megatron/core/models/gpt/heterogeneous/heterogeneous_layer_specs.py
+++ b/megatron/core/models/gpt/heterogeneous/heterogeneous_layer_specs.py
@@ -3,6 +3,7 @@
import warnings
from typing import Optional
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.fusions.fused_bias_dropout import get_bias_dropout_add
from megatron.core.tensor_parallel.layers import ColumnParallelLinear, RowParallelLinear
from megatron.core.transformer.attention import SelfAttention, SelfAttentionSubmodules
@@ -30,9 +31,7 @@
from megatron.core.typed_torch import not_none
from megatron.core.utils import is_te_min_version
-try:
- import transformer_engine as te # pylint: disable=unused-import
-
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import (
TEDotProductAttention,
TELayerNormColumnParallelLinear,
@@ -42,9 +41,7 @@
from megatron.core.transformer.heterogeneous.linear_replacements import (
TELayerNormColumnParallelLinearGathered,
)
-
- HAVE_TE = True
-except ImportError:
+else:
(
TEDotProductAttention,
TELayerNormColumnParallelLinear,
@@ -52,7 +49,6 @@
TERowParallelLinear,
TELayerNormColumnParallelLinearGathered,
) = (None, None, None, None, None)
- HAVE_TE = False
from megatron.core.transformer.torch_norm import WrappedTorchNorm
@@ -144,8 +140,10 @@ def _get_heterogenous_mlp_spec(mlp_config: MLPConfig, use_te: bool):
mlp = ModuleSpec(
module=MLP,
submodules=MLPSubmodules(
- linear_fc1=TELayerNormColumnParallelLinear if use_te else ColumnParallelLinear,
- linear_fc2=TERowParallelLinear if use_te else RowParallelLinear,
+ linear_fc1=(
+ not_none(TELayerNormColumnParallelLinear) if use_te else ColumnParallelLinear
+ ),
+ linear_fc2=not_none(TERowParallelLinear) if use_te else RowParallelLinear,
),
)
return mlp
diff --git a/megatron/core/models/gpt/moe_module_specs.py b/megatron/core/models/gpt/moe_module_specs.py
index 62ee4537cfc..53bca85f502 100755
--- a/megatron/core/models/gpt/moe_module_specs.py
+++ b/megatron/core/models/gpt/moe_module_specs.py
@@ -1,11 +1,17 @@
# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+from functools import partial
from typing import Optional
from megatron.core.extensions.transformer_engine_spec_provider import TESpecProvider
-from megatron.core.models.backends import BackendSpecProvider, LocalSpecProvider
+from megatron.core.models.backends import (
+ BackendSpecProvider,
+ InferenceSpecProvider,
+ LocalSpecProvider,
+)
from megatron.core.transformer.mlp import MLPSubmodules
from megatron.core.transformer.moe.moe_layer import MoELayer, MoESubmodules
+from megatron.core.transformer.moe.router import InferenceTopKRouter
from megatron.core.transformer.moe.shared_experts import SharedExpertMLP
from megatron.core.transformer.spec_utils import ModuleSpec
@@ -14,18 +20,24 @@ def get_moe_module_spec(
use_te: Optional[bool] = True,
num_experts: Optional[int] = None,
moe_grouped_gemm: Optional[bool] = False,
- moe_use_legacy_grouped_gemm: Optional[bool] = False,
) -> ModuleSpec:
- """Helper function to get module spec for MoE"""
+ """Helper function to get module spec for MoE.
+
+ Called by mamba_layer_specs.py for standard (non-inference) MoE specs.
+ The GPT layer specs call get_moe_module_spec_for_backend directly.
+
+ Args:
+ use_te: Whether to use Transformer Engine.
+ num_experts: Number of experts.
+ moe_grouped_gemm: Whether to use grouped GEMM.
+ moe_use_legacy_grouped_gemm: Whether to use legacy grouped GEMM.
+ """
if use_te is not None and use_te:
backend: BackendSpecProvider = TESpecProvider()
else:
backend = LocalSpecProvider()
return get_moe_module_spec_for_backend(
- backend=backend,
- num_experts=num_experts,
- moe_grouped_gemm=moe_grouped_gemm,
- moe_use_legacy_grouped_gemm=moe_use_legacy_grouped_gemm,
+ backend=backend, num_experts=num_experts, moe_grouped_gemm=moe_grouped_gemm
)
@@ -33,7 +45,6 @@ def get_moe_module_spec_for_backend(
backend: BackendSpecProvider,
num_experts: Optional[int] = None,
moe_grouped_gemm: Optional[bool] = False,
- moe_use_legacy_grouped_gemm: Optional[bool] = False,
use_te_activation_func: bool = False,
) -> ModuleSpec:
"""Helper function to get module spec for MoE"""
@@ -47,17 +58,9 @@ def get_moe_module_spec_for_backend(
linear_fc1=linear_fc1, linear_fc2=linear_fc2, activation_func=activation_func
)
- expert_module, expert_submodule = backend.grouped_mlp_modules(
- moe_grouped_gemm is not None and moe_grouped_gemm,
- moe_use_legacy_grouped_gemm is not None and moe_use_legacy_grouped_gemm,
- )
- if expert_submodule is not None:
- expert_submodule.activation_func = activation_func
-
- experts = ModuleSpec(module=expert_module, submodules=expert_submodule)
-
+ experts = backend.grouped_mlp_modules(moe_grouped_gemm is not None and moe_grouped_gemm)
# shared experts spec
- shared_experts = ModuleSpec(module=SharedExpertMLP, submodules=mlp)
+ shared_experts = partial(SharedExpertMLP, submodules=mlp)
# MoE module spec
moe_module_spec = ModuleSpec(
@@ -66,3 +69,34 @@ def get_moe_module_spec_for_backend(
metainfo={"fuse_pre_mlp_layernorm": False},
)
return moe_module_spec
+
+
+def get_inference_optimized_moe_spec() -> ModuleSpec:
+ """MoE module spec for inference-optimized transformer impl.
+
+ Uses InferenceSpecProvider to select inference-optimized modules:
+ InferenceTopKRouter, InferenceGroupedMLP. MoELayer detects inference mode
+ via config.transformer_impl and sets up the inference dispatcher internally.
+
+ Called by mamba_layer_specs.py and gpt_layer_specs.py.
+ """
+ backend = InferenceSpecProvider()
+ activation_func = backend.activation_func()
+
+ experts = backend.grouped_mlp_modules(True)
+ shared_experts = partial(
+ SharedExpertMLP,
+ submodules=MLPSubmodules(
+ linear_fc1=backend.column_parallel_linear(),
+ linear_fc2=backend.row_parallel_linear(),
+ activation_func=activation_func,
+ ),
+ )
+
+ return ModuleSpec(
+ module=MoELayer,
+ submodules=MoESubmodules(
+ router=InferenceTopKRouter, experts=experts, shared_experts=shared_experts
+ ),
+ metainfo={"fuse_pre_mlp_layernorm": False},
+ )
diff --git a/megatron/core/models/mamba/mamba_layer_specs.py b/megatron/core/models/mamba/mamba_layer_specs.py
index 6ca628475be..d2a85d004ef 100755
--- a/megatron/core/models/mamba/mamba_layer_specs.py
+++ b/megatron/core/models/mamba/mamba_layer_specs.py
@@ -8,7 +8,11 @@
TERowParallelLinear,
)
from megatron.core.fusions.fused_bias_dropout import get_bias_dropout_add
-from megatron.core.models.gpt.moe_module_specs import get_moe_module_spec
+from megatron.core.models.gpt.moe_module_specs import (
+ get_inference_optimized_moe_spec,
+ get_moe_module_spec,
+)
+from megatron.core.ssm.gated_delta_net import GatedDeltaNet, GatedDeltaNetSubmodules
from megatron.core.ssm.mamba_block import MambaStack, MambaStackSubmodules
from megatron.core.ssm.mamba_layer import MambaLayer, MambaLayerSubmodules
from megatron.core.ssm.mamba_mixer import MambaMixer, MambaMixerSubmodules
@@ -38,9 +42,11 @@
use_te=True,
num_experts=8, # Can be any positive integer (must not be None).
moe_grouped_gemm=True,
- moe_use_legacy_grouped_gemm=False,
)
+# Inference-optimized MoE spec
+moe_inference = get_inference_optimized_moe_spec()
+
# MTP block spec for Mamba - provides norms and projection only.
# Inner layers are built by MultiTokenPredictionLayer using nested MambaStack
@@ -78,6 +84,20 @@
mamba_bda=get_bias_dropout_add,
),
),
+ gdn_layer=ModuleSpec(
+ module=TransformerLayer,
+ submodules=TransformerLayerSubmodules(
+ self_attention=ModuleSpec(
+ module=GatedDeltaNet,
+ submodules=GatedDeltaNetSubmodules(
+ in_proj=TELayerNormColumnParallelLinear,
+ out_norm=TENorm,
+ out_proj=TERowParallelLinear,
+ ),
+ ),
+ self_attn_bda=get_bias_dropout_add,
+ ),
+ ),
# Started with spec from gpt_layer_specs.py (with MLP removed)
# Using the TE spec because we had problems getting the non-TE spec
# working
@@ -173,10 +193,10 @@
),
),
moe_layer=ModuleSpec(
- # TODO (rwaleffe): change this to be an "MoELayer" to work with CudaGraphs?
+ # Use inference-optimized MoE layer for end-to-end CUDA graph support
module=TransformerLayer,
submodules=TransformerLayerSubmodules(
- pre_mlp_layernorm=TENorm, mlp=moe, mlp_bda=get_bias_dropout_add
+ pre_mlp_layernorm=TENorm, mlp=moe_inference, mlp_bda=get_bias_dropout_add
),
),
mtp_block_spec=_mamba_mtp_block_spec,
diff --git a/megatron/core/models/mamba/mamba_model.py b/megatron/core/models/mamba/mamba_model.py
index 0a783391437..e295c3d6b01 100644
--- a/megatron/core/models/mamba/mamba_model.py
+++ b/megatron/core/models/mamba/mamba_model.py
@@ -1,7 +1,9 @@
# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+import logging
from typing import Literal, Optional
+import torch
from torch import Tensor
from megatron.core import tensor_parallel
@@ -26,8 +28,11 @@
WrappedTensor,
deprecate_inference_params,
is_using_quantization_scales,
+ log_single_rank,
)
+logger = logging.getLogger(__name__)
+
class MambaModel(LanguageModule):
"""Mamba language model.
@@ -38,16 +43,25 @@ class MambaModel(LanguageModule):
vocab_size (int): Vocabulary size
max_sequence_length (int): maximum size of sequence.
This is used for positional embedding
- pre_process (bool, optional): Include embedding layer
- (used with pipeline parallelism). Defaults to True.
- hybrid_attention_ratio (float, optional): The target ratio of attention
- layers to total layers
- hybrid_mlp_ratio (float, optional): The target ratio of mlp layers to total layers
- hybrid_override_pattern (str, optional): Unified hybrid layer pattern with optional MTP.
+ hybrid_layer_pattern (str): Unified hybrid layer pattern with optional MTP and
+ pipeline stage boundaries.
Format: "///..."
+ The main pattern may contain "|" to define pipeline stage boundaries.
Examples:
- "M*M*" -> main decoder only, no MTP
- "M*M*/MM/MM" -> main="M*M*", mtp="MM", 2 depths
+ - "M-M-|M-M*-|M-M-|M-M*-" -> 4 pipeline segments
+ hybrid_attention_ratio (float, optional): Deprecated. Use hybrid_layer_pattern instead.
+ If set to a value > 0.0 and hybrid_layer_pattern is None, a pattern will be
+ generated from the ratio with a deprecation warning.
+ hybrid_mlp_ratio (float, optional): Deprecated. Use hybrid_layer_pattern instead.
+ If set to a value > 0.0 and hybrid_layer_pattern is None, a pattern will be
+ generated from the ratio with a deprecation warning.
+ hybrid_override_pattern (str, optional): Deprecated. Use hybrid_layer_pattern instead.
+ If set and hybrid_layer_pattern is None, the value is copied to hybrid_layer_pattern
+ with a deprecation warning.
+ pre_process (bool, optional): Include embedding layer
+ (used with pipeline parallelism). Defaults to True.
post_process (bool, optional): Include an output layer (used with pipeline parallelism).
Defaults to True.
fp16_lm_cross_entropy (bool, optional): Defaults to False.
@@ -65,6 +79,7 @@ class MambaModel(LanguageModule):
interpolating RoPE for longer sequences. The value must be a float larger than 1.0.
Defaults to None.
pg_collection (ProcessGroupCollection, optional): Model communication process groups.
+ vp_stage (Optional[int], optional): Virtual pipeline stage index. Defaults to None.
"""
def __init__(
@@ -73,10 +88,11 @@ def __init__(
mamba_stack_spec: ModuleSpec,
vocab_size: int,
max_sequence_length: int,
+ hybrid_layer_pattern: Optional[str] = None,
+ hybrid_attention_ratio: Optional[float] = None,
+ hybrid_mlp_ratio: Optional[float] = None,
+ hybrid_override_pattern: Optional[str] = None,
pre_process: bool = True,
- hybrid_attention_ratio: float = 0.0,
- hybrid_mlp_ratio: float = 0.0,
- hybrid_override_pattern: str = None,
post_process: bool = True,
fp16_lm_cross_entropy: bool = False,
parallel_output: bool = True,
@@ -95,13 +111,19 @@ def __init__(
if has_config_logger_enabled(config):
log_config_to_disk(config, locals(), prefix=type(self).__name__)
+ if self.config.use_mup and not getattr(MambaModel, "mup_warning_printed", False):
+ log_single_rank(
+ logger,
+ logging.WARNING,
+ "MuP for MambaModel is experimental and not fully validated yet.",
+ )
+ MambaModel.mup_warning_printed = True
+
self.mamba_stack_spec: ModuleSpec = mamba_stack_spec
self.vocab_size = vocab_size
self.max_sequence_length = max_sequence_length
+ self.hybrid_layer_pattern = hybrid_layer_pattern
self.pre_process = pre_process
- self.hybrid_attention_ratio = hybrid_attention_ratio
- self.hybrid_mlp_ratio = hybrid_mlp_ratio
- self.hybrid_override_pattern = hybrid_override_pattern
self.post_process = post_process
self.fp16_lm_cross_entropy = fp16_lm_cross_entropy
self.parallel_output = parallel_output
@@ -109,18 +131,73 @@ def __init__(
self.position_embedding_type = position_embedding_type
self.vp_stage = vp_stage
- # Parse unified pattern to extract main and MTP components
- from megatron.core.ssm.mamba_hybrid_layer_allocation import parse_hybrid_pattern
+ # Backward compatibility for deprecated hybrid parameters
+ if hybrid_override_pattern is not None:
+ if self.hybrid_layer_pattern is None:
+ log_single_rank(
+ logger,
+ logging.WARNING,
+ "hybrid_override_pattern has been deprecated. "
+ "Use hybrid_layer_pattern instead.",
+ )
+ self.hybrid_layer_pattern = hybrid_override_pattern
+ else:
+ raise ValueError(
+ "hybrid_override_pattern and hybrid_layer_pattern cannot both be set. "
+ "hybrid_override_pattern has been deprecated; use hybrid_layer_pattern instead."
+ )
+ if (hybrid_attention_ratio is not None and hybrid_attention_ratio > 0.0) or (
+ hybrid_mlp_ratio is not None and hybrid_mlp_ratio > 0.0
+ ):
+ if hybrid_layer_pattern is not None:
+ raise ValueError(
+ "hybrid_layer_pattern cannot be used together with "
+ "hybrid_attention_ratio or hybrid_mlp_ratio. "
+ "These ratios have been deprecated; use hybrid_layer_pattern alone."
+ )
+ log_single_rank(
+ logger,
+ logging.WARNING,
+ "hybrid_attention_ratio and hybrid_mlp_ratio have been deprecated. "
+ "Use hybrid_layer_pattern instead.",
+ )
+ if self.hybrid_layer_pattern is None:
+ from megatron.core.ssm.mamba_hybrid_layer_allocation import pattern_from_ratios
+
+ attn_ratio = hybrid_attention_ratio if hybrid_attention_ratio else 0.0
+ mlp_ratio = hybrid_mlp_ratio if hybrid_mlp_ratio else 0.0
+ self.hybrid_layer_pattern = pattern_from_ratios(
+ config.num_layers, attn_ratio, mlp_ratio
+ )
+
+ # Parse unified pattern to extract main and MTP components, and
+ # determine the pipeline segment for this model instance.
+ from megatron.core.ssm.mamba_hybrid_layer_allocation import (
+ parse_hybrid_pattern,
+ select_pipeline_segment,
+ )
- parsed = parse_hybrid_pattern(hybrid_override_pattern)
+ parsed = parse_hybrid_pattern(self.hybrid_layer_pattern)
self.mtp_pattern = parsed.mtp_pattern
self.mtp_num_depths = parsed.mtp_num_depths
+ layer_type_list, layer_offset = select_pipeline_segment(
+ parsed.main_pattern or '',
+ self.pg_collection.pp,
+ vp_stage,
+ first_stage_layers=self.config.num_layers_in_first_pipeline_stage,
+ last_stage_layers=self.config.num_layers_in_last_pipeline_stage,
+ )
+
# Determine if MTP is needed (based on pattern parsing)
self.mtp_process = (
self.mtp_pattern is not None
and self.mtp_num_depths > 0
- and mtp_on_this_rank(self.config, vp_stage=self.vp_stage)
+ # The following forces MTP to be on the final pipeline stage. It might be more optimal
+ # to split the hybrid layer pattern into pipeline stages before parsing the pattern for
+ # the current pipeline stage. This could also enable MTP standalone (MTP in a pipeline
+ # stage separate from loss) to be supported in the hybrid model.
+ and mtp_on_this_rank(self.config, ignore_virtual=False, vp_stage=self.vp_stage)
)
# megatron core pipelining currently depends on model type
@@ -151,9 +228,8 @@ def __init__(
mamba_stack_spec,
self.config,
pre_process=self.pre_process,
- hybrid_attention_ratio=self.hybrid_attention_ratio,
- hybrid_mlp_ratio=self.hybrid_mlp_ratio,
- hybrid_override_pattern=parsed.main_pattern,
+ layer_type_list=layer_type_list,
+ pp_layer_offset=layer_offset,
post_process=self.post_process,
dtype=config.params_dtype,
pg_collection=self.pg_collection,
@@ -184,7 +260,11 @@ def __init__(
config.hidden_size,
self.vocab_size,
config=config,
- init_method=config.init_method,
+ init_method=(
+ config.embedding_init_method
+ if config.use_mup and not self.share_embeddings_and_output_weights
+ else config.init_method
+ ),
bias=False,
skip_bias_add=False,
gather_output=not self.parallel_output,
@@ -231,6 +311,7 @@ def forward(
loss_mask: Optional[Tensor] = None,
packed_seq_params: Optional[PackedSeqParams] = None,
padding_mask: Optional[Tensor] = None,
+ is_spec_decode: Optional[bool] = None,
) -> Tensor:
"""Forward function of the Mamba model. This function passes the input tensors
through the embedding layer, and then the decoder and finally into the post
@@ -307,7 +388,18 @@ def forward(
if self.share_embeddings_and_output_weights:
output_weight = self.shared_embedding_or_output_weight()
- if self.mtp_process:
+ # Check if speculative decoding is active. When it is, MTP must be
+ # computed *after* verification so that it is conditioned on verified
+ # tokens rather than stale speculative tokens from the previous step.
+ if is_spec_decode is None:
+ is_spec_decode = (
+ in_inference_mode
+ and inference_context.is_dynamic_batching()
+ and inference_context.num_speculative_tokens > 0
+ )
+
+ mtp_forward_ran = self.mtp_process and not (in_inference_mode or is_spec_decode)
+ if mtp_forward_ran:
hidden_states = self.mtp(
input_ids=input_ids,
position_ids=position_ids,
@@ -322,23 +414,27 @@ def forward(
if not self.post_process:
return hidden_states
- if self.config.mtp_num_layers is not None:
- hidden_states = process_mtp_loss(
- hidden_states=hidden_states,
- labels=labels,
- loss_mask=loss_mask,
- output_layer=self.output_layer,
- output_weight=output_weight,
- runtime_gather_output=runtime_gather_output,
- is_training=self.training,
- compute_language_model_loss=self.compute_language_model_loss,
- config=self.config,
- cp_group=self.pg_collection.cp,
- packed_seq_params=packed_seq_params,
- )
-
+ if self.config.mtp_num_layers is not None and self.mtp_process:
+ assert self.config.mtp_num_layers > 0
+ if in_inference_mode or is_spec_decode:
+ self._decoder_hidden_states_cache = hidden_states
+ else:
+ hidden_states = process_mtp_loss(
+ hidden_states=hidden_states,
+ labels=labels,
+ loss_mask=loss_mask,
+ output_layer=self.output_layer,
+ output_weight=output_weight,
+ runtime_gather_output=runtime_gather_output,
+ is_training=self.training,
+ compute_language_model_loss=self.compute_language_model_loss,
+ config=self.config,
+ cp_group=self.pg_collection.cp,
+ packed_seq_params=packed_seq_params,
+ scale_logits_fn=self._scale_logits if self.config.use_mup else None,
+ )
sequence_parallel_override = False
- if in_inference_mode and inference_context.materialize_only_last_token_logits:
+ if in_inference_mode and inference_context.config.materialize_only_last_token_logits:
if inference_context.is_static_batching():
hidden_states = hidden_states[-1:, :, :]
else:
@@ -352,23 +448,22 @@ def forward(
self.output_layer.sequence_parallel = False
sequence_parallel_override = True
- # Reshape [B, 1, H] to [1, B, H] → extract each sample's true last‐token hidden
- # state ([B, H]) → unsqueeze back to [B, 1, H]
- # (so that the output layer, which expects S×B×H, receives only the final token)
- hidden_states = inference_context.last_token_logits(
- hidden_states.squeeze(1).unsqueeze(0)
- ).unsqueeze(1)
+ # Reshape [S, B, H] (with B=1) to [1, S, H] for logit extraction,
+ # then back to [S', B, H] for the output layer.
+ reshaped = hidden_states.squeeze(1).unsqueeze(0)
+ hidden_states = inference_context.last_token_logits(reshaped).unsqueeze(1)
logits, _ = self.output_layer(
hidden_states, weight=output_weight, runtime_gather_output=runtime_gather_output
)
+ logits = self._scale_logits(logits)
# Restore sequence parallel execution to the output layer if necessary.
if sequence_parallel_override:
assert (
in_inference_mode
and inference_context.is_dynamic_batching()
- and inference_context.materialize_only_last_token_logits
+ and inference_context.config.materialize_only_last_token_logits
)
self.output_layer.sequence_parallel = True
@@ -379,3 +474,46 @@ def forward(
loss = self.compute_language_model_loss(labels, logits)
return loss
+
+ @torch.inference_mode()
+ def compute_mtp_single_step(
+ self,
+ hidden_states: Tensor,
+ next_token_ids: Tensor,
+ position_ids: Tensor,
+ depth: int,
+ runtime_gather_output: bool = True,
+ ) -> tuple:
+ """Compute a single MTP depth for speculative decoding.
+
+ This is called after speculative token verification to compute MTP
+ predictions conditioned on verified tokens only.
+
+ Args:
+ hidden_states (Tensor): Hidden states at last accepted positions [N, 1, H].
+ next_token_ids (Tensor): Correct next token IDs [1, N].
+ position_ids (Tensor): Position IDs for the next tokens [1, N].
+ depth (int): MTP depth index (0-indexed).
+ runtime_gather_output (bool): Whether to gather output across TP.
+
+ Returns:
+ tuple: (new_hidden_states [N, 1, H], logits [N, 1, vocab_size]).
+ """
+ layer_idx = 0 if self.mtp.mtp_use_repeated_layer else depth
+ mtp_hidden = self.mtp.layers[layer_idx].forward_single_position(
+ hidden_states=hidden_states,
+ next_token_ids=next_token_ids,
+ position_ids=position_ids,
+ embedding=self.embedding,
+ )
+
+ output_weight = None
+ if self.share_embeddings_and_output_weights:
+ output_weight = self.shared_embedding_or_output_weight()
+
+ logits, _ = self.output_layer(
+ mtp_hidden, weight=output_weight, runtime_gather_output=runtime_gather_output
+ )
+ logits = self._scale_logits(logits)
+
+ return mtp_hidden, logits
diff --git a/megatron/core/models/mimo/__init__.py b/megatron/core/models/mimo/__init__.py
index 204851c444b..779bf921e1c 100644
--- a/megatron/core/models/mimo/__init__.py
+++ b/megatron/core/models/mimo/__init__.py
@@ -2,6 +2,7 @@
from megatron.core.models.mimo.config.base_configs import MimoModelConfig
from megatron.core.models.mimo.model import MimoModel
+from megatron.core.models.mimo.optimizer import MimoOptimizer, get_mimo_optimizer
from megatron.core.models.mimo.submodules.audio import AudioModalitySubmodules
from megatron.core.models.mimo.submodules.base import ModalitySubmodules
from megatron.core.models.mimo.submodules.vision import VisionModalitySubmodules
@@ -9,6 +10,8 @@
__all__ = [
'MimoModelConfig',
'MimoModel',
+ 'MimoOptimizer',
+ 'get_mimo_optimizer',
# Submodule classes
'ModalitySubmodules',
'VisionModalitySubmodules',
diff --git a/megatron/core/models/mimo/config/__init__.py b/megatron/core/models/mimo/config/__init__.py
index 8371675a22d..3da744a6fb2 100644
--- a/megatron/core/models/mimo/config/__init__.py
+++ b/megatron/core/models/mimo/config/__init__.py
@@ -1,5 +1,6 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
from megatron.core.models.mimo.config.base_configs import MimoModelConfig
+from megatron.core.models.mimo.config.role import ModuleStageInfo, RankRole
-__all__ = ['MimoModelConfig']
+__all__ = ['MimoModelConfig', 'ModuleStageInfo', 'RankRole']
diff --git a/megatron/core/models/mimo/config/base_configs.py b/megatron/core/models/mimo/config/base_configs.py
index 8b170abe152..a92484a5a48 100644
--- a/megatron/core/models/mimo/config/base_configs.py
+++ b/megatron/core/models/mimo/config/base_configs.py
@@ -2,8 +2,9 @@
import warnings
from dataclasses import dataclass, field
-from typing import Dict
+from typing import Dict, Optional
+from megatron.core.hyper_comm_grid import HyperCommGrid
from megatron.core.transformer.spec_utils import ModuleSpec
@@ -20,6 +21,14 @@ class MimoModelConfig:
Dictionary mapping modality names to their special token IDs.
For example, {"vision": -200, "audio":32000}, these represent placeholders
in the input_ids to insert the modality embeddings at the correct positions.
+ module_to_grid_map (Optional[Dict[str, HyperCommGrid]]):
+ Dictionary mapping module keys (e.g., "vision", "language") to their
+ corresponding HyperCommGrid configurations for non-colocated pipeline
+ parallelism. The language model must use the key MIMO_LANGUAGE_MODULE_KEY.
+ When None, all modules are assumed to be colocated on the same ranks.
+ kv_format (str):
+ Key-value format for attention: "sbhd" (seq-batch-head-dim) or "thd" (total-head-dim).
+ Default is "sbhd".
"""
warnings.warn(
@@ -32,3 +41,5 @@ class MimoModelConfig:
language_model_spec: ModuleSpec = field(default_factory=ModuleSpec)
modality_submodules_spec: Dict[str, ModuleSpec] = field(default_factory=dict)
special_token_ids: Dict[str, int] = field(default_factory=dict)
+ module_to_grid_map: Optional[Dict[str, HyperCommGrid]] = None
+ kv_format: str = "sbhd"
diff --git a/megatron/core/models/mimo/config/role.py b/megatron/core/models/mimo/config/role.py
new file mode 100644
index 00000000000..77c2512e8e6
--- /dev/null
+++ b/megatron/core/models/mimo/config/role.py
@@ -0,0 +1,172 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+"""Data classes for MIMO rank role management in multi-module pipeline parallelism."""
+
+import logging
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Dict, List
+
+import torch.distributed as dist
+
+from megatron.core.hyper_comm_grid import HyperCommGrid
+
+logger = logging.getLogger(__name__)
+
+# Fixed key for the language module in module_to_grid_map and RankRole.
+# MIMO always has exactly one language model, so this is not configurable.
+MIMO_LANGUAGE_MODULE_KEY = "language"
+
+
+class ModuleLayout(Enum):
+ """Pipeline mode for MIMO multi-module parallelism.
+
+ Determines how modules are distributed across ranks and which
+ forward path is used.
+
+ UNIFIED: No module_to_grid_map. All modules share same ranks and
+ parallelism. Uses the unified forward path (_forward_all_modules).
+
+ NON_COLOCATED: module_to_grid_map is set with non-overlapping rank
+ ranges. Each rank runs EITHER encoder(s) OR the language model.
+ Uses role-based dispatch with separate forward paths.
+
+ COLOCATED: (future) module_to_grid_map is set with overlapping rank
+ ranges. Encoder(s) and language model share ranks but have
+ different parallelism configs. Uses role-based dispatch but
+ allows both module types on the same rank.
+ """
+
+ UNIFIED = "unified"
+ NON_COLOCATED = "non_colocated"
+ COLOCATED = "colocated"
+
+
+@dataclass
+class ModuleStageInfo:
+ """Information about a rank's stage position within a module's pipeline.
+
+ Args:
+ is_first_stage: True if this rank is the first PP stage for this module.
+ is_last_stage: True if this rank is the last PP stage for this module.
+ """
+
+ is_first_stage: bool
+ is_last_stage: bool
+
+
+@dataclass
+class RankRole:
+ """Describes what modules this rank participates in for multi-module PP.
+
+ This class captures the role of a specific rank in a multi-module pipeline
+ parallel setup, tracking which modules the rank participates in and their
+ stage positions. The language module is always identified by MIMO_LANGUAGE_MODULE_KEY.
+
+ Args:
+ modules: Dict mapping module names to their stage info for modules
+ this rank participates in.
+ mode: Pipeline mode determining forward path dispatch.
+ """
+
+ modules: Dict[str, ModuleStageInfo] = field(default_factory=dict)
+ mode: ModuleLayout = ModuleLayout.UNIFIED
+
+ @classmethod
+ def unified(cls, module_names: List[str]) -> 'RankRole':
+ """Create a role for the unified case: every module, first+last stage."""
+ return cls(
+ modules={
+ name: ModuleStageInfo(is_first_stage=True, is_last_stage=True)
+ for name in module_names
+ },
+ mode=ModuleLayout.UNIFIED,
+ )
+
+ @classmethod
+ def from_grid_map(
+ cls, module_to_grid_map: Dict[str, HyperCommGrid], modality_module_names: List[str]
+ ) -> 'RankRole':
+ """Create a role from a module-to-grid mapping for non-colocated PP.
+
+ Determines which modules the current rank participates in and its
+ pipeline stage position within each module.
+
+ Args:
+ module_to_grid_map: Dict mapping module names to HyperCommGrid objects.
+ Must contain keys matching modality_module_names + MIMO_LANGUAGE_MODULE_KEY.
+ modality_module_names: List of modality module names (e.g., ["images", "audio"]).
+
+ Returns:
+ RankRole for the current rank.
+
+ Raises:
+ ValueError: If grid map keys don't match expected module names.
+ RuntimeError: If current rank is not in any module grid.
+ """
+ # Validate keys
+ expected_keys = set(modality_module_names) | {MIMO_LANGUAGE_MODULE_KEY}
+ grid_keys = set(module_to_grid_map.keys())
+ if grid_keys != expected_keys:
+ raise ValueError(
+ f"module_to_grid_map keys must match modality module names + "
+ f"'{MIMO_LANGUAGE_MODULE_KEY}'. Missing: {expected_keys - grid_keys}, "
+ f"Extra: {grid_keys - expected_keys}"
+ )
+
+ current_rank = dist.get_rank()
+ modules = {}
+
+ for module_name, grid in module_to_grid_map.items():
+ if not (grid.rank_offset <= current_rank < grid.rank_offset + grid.size):
+ continue
+
+ if "pp" not in grid.dim_names:
+ modules[module_name] = ModuleStageInfo(is_first_stage=True, is_last_stage=True)
+ continue
+
+ pp_group = grid.get_pg("pp")
+ pp_rank = pp_group.rank()
+ pp_size = pp_group.size()
+ is_first = pp_rank == 0
+ is_last = pp_rank == pp_size - 1
+ logger.info(
+ f"[RankRole.from_grid_map] Rank {current_rank}: module={module_name}, "
+ f"pp_rank={pp_rank}/{pp_size}, is_first_stage={is_first}, is_last_stage={is_last}"
+ )
+ modules[module_name] = ModuleStageInfo(is_first_stage=is_first, is_last_stage=is_last)
+
+ if not modules:
+ raise RuntimeError(
+ f"Rank {current_rank} is not in any module grid. "
+ f"Check module_to_grid_map configuration."
+ )
+
+ return cls(modules=modules, mode=ModuleLayout.NON_COLOCATED)
+
+ @property
+ def has_modality_modules(self) -> bool:
+ """Return True if this rank participates in any modality (non-language) module."""
+ return any(name != MIMO_LANGUAGE_MODULE_KEY for name in self.modules)
+
+ @property
+ def has_language_module(self) -> bool:
+ """Return True if this rank participates in the language module."""
+ return MIMO_LANGUAGE_MODULE_KEY in self.modules
+
+ @property
+ def modality_module_names(self) -> List[str]:
+ """Return names of modality modules (non-language) this rank participates in."""
+ return [name for name in self.modules if name != MIMO_LANGUAGE_MODULE_KEY]
+
+ def is_first_stage(self, module_name: str) -> bool:
+ """Check if this rank is the first stage for a given module."""
+ if module_name not in self.modules:
+ return False
+ return self.modules[module_name].is_first_stage
+
+ def is_last_stage(self, module_name: str) -> bool:
+ """Check if this rank is the last stage for a given module."""
+ if module_name not in self.modules:
+ return False
+ return self.modules[module_name].is_last_stage
diff --git a/megatron/core/models/mimo/model/base.py b/megatron/core/models/mimo/model/base.py
index 2f136a98466..b1c12f521c3 100644
--- a/megatron/core/models/mimo/model/base.py
+++ b/megatron/core/models/mimo/model/base.py
@@ -6,9 +6,15 @@
import torch
+from megatron.core.distributed import DistributedDataParallel
from megatron.core.models.mimo.config import MimoModelConfig
+from megatron.core.models.mimo.config.role import MIMO_LANGUAGE_MODULE_KEY, ModuleLayout, RankRole
+from megatron.core.models.mimo.partition.utils import PartitionAdapter, PartitionConfig
+from megatron.core.packed_seq_params import PackedSeqParams
from megatron.core.transformer import MegatronModule
from megatron.core.transformer.spec_utils import build_module
+from megatron.core.transformer.utils import sharded_state_dict_default
+from megatron.core.utils import unwrap_model
logger = logging.getLogger(__name__)
@@ -32,7 +38,7 @@ class MimoModel(MegatronModule):
Configuration for the model, including language model and modality submodules
"""
- def __init__(self, mimo_config: MimoModelConfig) -> None:
+ def __init__(self, mimo_config: MimoModelConfig, cp_group=None, tp_group=None) -> None:
"""Initialize the multimodal model.
Example:
@@ -52,17 +58,76 @@ def __init__(self, mimo_config: MimoModelConfig) -> None:
)
self.mimo_config = mimo_config
+ modality_names = list(mimo_config.modality_submodules_spec.keys())
+ if mimo_config.module_to_grid_map:
+ self.role = RankRole.from_grid_map(mimo_config.module_to_grid_map, modality_names)
+ else:
+ self.role = RankRole.unified(modality_names + [MIMO_LANGUAGE_MODULE_KEY])
# Use special token IDs from the config
self.special_token_ids = (
mimo_config.special_token_ids.copy() if mimo_config.special_token_ids else {}
)
+ # Extract language model config for partition adapter
+ language_config = mimo_config.language_model_spec.params['config']
+ max_seq_len = mimo_config.language_model_spec.params.get('max_sequence_length', 4096)
+
+ self.partition_adapter: Optional[PartitionAdapter] = None
+ # Create partition adapter only if parallelism is enabled
+ if language_config.context_parallel_size > 1 or language_config.sequence_parallel:
+ partition_config = PartitionConfig.from_mp_config(
+ mp=language_config,
+ max_seq_len=max_seq_len,
+ kv_format=mimo_config.kv_format,
+ cp_group=cp_group,
+ tp_group=tp_group,
+ )
+ self.partition_adapter = PartitionAdapter(partition_config)
+
# Initialize modality submodules from specifications
self.modality_submodules = torch.nn.ModuleDict()
self._initialize_submodules()
self._initialize_language_model()
+ def sharded_state_dict(self, prefix='', sharded_offsets=(), metadata=None):
+ """Build sharded state dict, bypassing parallel_state global fallbacks.
+
+ Iterates modality_submodules manually (ModuleDict lacks sharded_state_dict)
+ and injects dp_cp_group from each module's pg_collection.
+ """
+ sharded_sd = {}
+ for name, module in self.named_children():
+ if name == 'modality_submodules':
+ # Unwrap DDP, call ModalitySubmodules.sharded_state_dict directly
+ # (which injects dp_cp_group from its pg_collection)
+ for mod_name, mod in module.items():
+ is_ddp = isinstance(mod, DistributedDataParallel)
+ inner = mod.module if is_ddp else mod
+ child_prefix = f'{prefix}{name}.{mod_name}.'
+ if is_ddp:
+ child_prefix += 'module.'
+ sharded_sd.update(
+ inner.sharded_state_dict(child_prefix, sharded_offsets, metadata)
+ )
+ else:
+ # Inject dp_cp_group from pg_collection for language_model
+ inner = module.module if isinstance(module, DistributedDataParallel) else module
+ pg = getattr(inner, 'pg_collection', None)
+ mod_metadata = metadata
+ if pg is not None:
+ assert (
+ hasattr(pg, 'dp_cp') and pg.dp_cp is not None
+ ), f"pg_collection on '{name}' is missing dp_cp group"
+ mod_metadata = dict(metadata) if metadata else {}
+ mod_metadata['dp_cp_group'] = pg.dp_cp
+ sharded_sd.update(
+ sharded_state_dict_default(
+ module, f'{prefix}{name}.', sharded_offsets, mod_metadata
+ )
+ )
+ return sharded_sd
+
def align_embeddings_by_token_positions(
self,
modality_embeddings: Dict[str, torch.Tensor], # [num_embeddings, hidden_dim]
@@ -73,14 +138,16 @@ def align_embeddings_by_token_positions(
Args:
modality_embeddings: Dictionary mapping modality names to their embeddings.
- For all modalities: tensor of shape [num_tokens_for_modality, hidden_dim]
- input_ids: Input token IDs of shape [batch_size, seq_len] containing special tokens
- that mark where each modality's embeddings should go. The number of special tokens
- for each modality should exactly match the number of embeddings for that modality.
+ For all modalities: tensor of shape (N, H).
+ Shape: (num_tokens_for_modality, hidden_dim)
+ input_ids: Input token IDs. Shape: (B, S) or (S,)
+ Contains special tokens that mark where each modality's embeddings should go.
+ The number of special tokens for each modality should exactly match the number
+ of embeddings for that modality.
special_token_ids: Dictionary mapping modality names to their special token IDs
Returns:
- Combined embeddings tensor of shape [seq_len, batch_size, hidden_dim]
+ Combined embeddings tensor. Shape: (S, B, H)
"""
# Ensure we have at least one modality
if not modality_embeddings:
@@ -96,8 +163,7 @@ def align_embeddings_by_token_positions(
device = reference_embeddings.device
dtype = reference_embeddings.dtype
- batch_size, seq_length = input_ids.size() # input_ids is [b, s]
-
+ batch_size, seq_length = input_ids.size() # input_ids is [B, S]
logger.debug(
f"Combined output tensor will have shape: [{seq_length}, {batch_size}, {hidden_dim}]"
)
@@ -109,8 +175,7 @@ def align_embeddings_by_token_positions(
# Process each modality in modality_embeddings
for modality_name, modality_emb in modality_embeddings.items():
if modality_name == "text":
- # Text tokens: positions that are not any special token.
- mask = torch.ones_like(input_ids, dtype=torch.bool)
+ mask = torch.ones_like(input_ids, dtype=torch.bool, device=input_ids.device)
for token_id in special_token_ids.values():
mask &= input_ids != token_id
elif modality_name in special_token_ids:
@@ -126,32 +191,49 @@ def align_embeddings_by_token_positions(
f"number of {modality_name} embeddings ({modality_emb.size(0)})"
)
- expanded_mask = (
- mask.unsqueeze(-1).expand_as(combined_embeddings).to(combined_embeddings.device)
- )
+ expanded_mask = mask.unsqueeze(-1).expand_as(combined_embeddings)
combined_embeddings.masked_scatter_(expanded_mask, modality_emb.flatten())
- return combined_embeddings.transpose(
- 0, 1
- ).contiguous() # Shape: [seq_length, batch_size, hidden_dim]
+
+ return combined_embeddings.transpose(0, 1).contiguous() # [S, B, H]
def _initialize_submodules(self) -> None:
"""Initialize modality submodules from the ModuleSpec configurations.
- Only modalities present in the config will be instantiated.
- For each modality in the config, builds the corresponding submodule using from_spec.
+ When role is set, only initializes submodules this rank participates in.
+ Stage info is passed to from_spec() to conditionally skip projection.
"""
-
for modality_name, submodule_spec in self.mimo_config.modality_submodules_spec.items():
- # Get the submodule class
+ if modality_name not in self.role.modules:
+ logger.debug(f"Skipping {modality_name} submodule (not in role)")
+ continue
+
+ stage_info = self.role.modules[modality_name]
+ is_first_stage = stage_info.is_first_stage
+ is_last_stage = stage_info.is_last_stage
+
submodule_class = submodule_spec.module
- logger.debug(f"Building {modality_name} submodule using {submodule_class.__name__}")
+ logger.debug(
+ f"Building {modality_name} submodule using {submodule_class.__name__} "
+ f"(is_first_stage={is_first_stage}, is_last_stage={is_last_stage})"
+ )
+
+ # Pass stage info to from_spec so projections are only built when needed
+ submodule = submodule_class.from_spec(
+ submodule_spec, is_first_stage=is_first_stage, is_last_stage=is_last_stage
+ )
- # Use from_spec to instantiate the submodule
- submodule = submodule_class.from_spec(submodule_spec)
self.modality_submodules[modality_name] = submodule
def _initialize_language_model(self) -> None:
- """Initialize the language model."""
+ """Initialize the language model.
+
+ When role is set, only initializes if this rank participates in language module.
+ """
+ if not self.role.has_language_module:
+ logger.debug("Skipping language model initialization (not in role)")
+ self.language_model = None
+ return
+
logger.debug(
f"Building language model using {self.mimo_config.language_model_spec.module.__name__}"
)
@@ -164,18 +246,30 @@ def set_input_tensor(self, input_tensor):
It passes the output tensor from the previous stage as input to this stage.
Args:
- input_tensor: Tensor or list of tensors passed between pipeline stages
+ input_tensor: Either:
+ - Dict[str, Tensor]: Maps module names to their input tensors (for multi-module PP)
+ - Tensor or List[Tensor]: Single tensor for language model (backward compat)
Returns:
None
"""
- # Handle case where input_tensor might be a list or a single tensor
+ # The schedule wraps input_tensor in a list (schedules.py:415-416),
+ # so unwrap first before checking type.
if isinstance(input_tensor, list):
- # For simplicity, just use the first tensor
input_tensor = input_tensor[0]
- # Pass the input tensor to the language model if it has a set_input_tensor method
- if hasattr(self.language_model, 'set_input_tensor'):
+ # Store dict input for multi-module PP
+ if isinstance(input_tensor, dict):
+ # P2P recv may return [tensor] (list) for VPP compat — unwrap to tensor
+ self.input_tensors = {
+ k: v[0] if isinstance(v, list) and len(v) == 1 else v
+ for k, v in input_tensor.items()
+ }
+ return
+
+ self.input_tensors = input_tensor
+
+ if self.language_model is not None and hasattr(self.language_model, 'set_input_tensor'):
self.language_model.set_input_tensor(input_tensor)
def get_text_embeddings(
@@ -183,15 +277,16 @@ def get_text_embeddings(
) -> torch.Tensor:
"""Get embeddings for text tokens in the input.
Args:
- input_ids: Input token IDs of shape [batch_size, seq_len] containing text tokens
- and potentially special tokens for other modalities.
+ input_ids: Input token IDs. Shape: (B, S)
+ Contains text tokens and potentially special tokens for other modalities.
position_ids: Position IDs corresponding to input tokens, used for positional encoding.
- Shape [batch_size, seq_len].
+ Shape: (B, S)
special_token_ids: Dictionary mapping modality names to their special token IDs.
Used to identify non-text tokens in the input_ids.
Returns:
- torch.Tensor: Embeddings for text tokens, shape [num_text_tokens, hidden_dim].
+ torch.Tensor: Embeddings for text tokens.
+ Shape: (N, H), where N is the number of text tokens.
"""
text_mask = torch.ones_like(input_ids, dtype=torch.bool) # [b, s]
for special_token_id in special_token_ids.values():
@@ -204,10 +299,10 @@ def get_text_embeddings(
position_ids[batch_idx, seq_idx].unsqueeze(0) if position_ids is not None else None
)
- text_embeddings = self.language_model.embedding(
- input_ids=input_ids_text, position_ids=position_ids_text
- ).squeeze(
- 1
+ text_embeddings = (
+ unwrap_model(self.language_model)
+ .embedding(input_ids=input_ids_text, position_ids=position_ids_text)
+ .squeeze(1)
) # Shape: [num_text_tokens, hidden_dim]
return text_embeddings
@@ -219,15 +314,16 @@ def forward(
loss_mask: Optional[torch.Tensor] = None,
labels: Optional[torch.Tensor] = None,
modality_inputs: Optional[Dict[str, Dict[str, Any]]] = None,
+ packing_kwargs: Optional[dict] = None,
):
"""Forward pass through the multimodal model.
Args:
- input_ids: Input token IDs [batch_size, seq_length]
- position_ids: Position IDs [batch_size, seq_length]
- attention_mask: Attention mask [batch_size, seq_length]
- loss_mask: Loss mask [batch_size, seq_length]
- labels: Labels for training
+ input_ids: Input token IDs. Shape: (B, S)
+ position_ids: Position IDs. Shape: (B, S)
+ attention_mask: Attention mask. Shape: (B, S)
+ loss_mask: Loss mask. Shape: (B, S)
+ labels: Labels for training. Shape: (B, S)
modality_inputs: Dictionary mapping modality names to encoder inputs. For example:
{
"images": {
@@ -238,25 +334,200 @@ def forward(
"whisper_encoder": {"input_features": whisper_features}
}
}
+ packing_kwargs: Optional dictionary of kwargs to construct PackedSeqParams
+ if packed_seq_params is not provided. For example:
+ {
+ "cu_seqlens_q": cu_seqlens,
+ "cu_seqlens_kv": cu_seqlens,
+ "cu_seqlens_q_padded": cu_seqlens_padded,
+ "cu_seqlens_kv_padded": cu_seqlens_padded,
+ "max_seqlen_q": torch.tensor(
+ max(seqlens_padded), dtype=torch.int32
+ ),
+ "max_seqlen_kv": torch.tensor(
+ max(seqlens_padded), dtype=torch.int32
+ ),
+ }
+
+ Returns:
+ tuple: (output, loss_mask) where output semantics depend on role:
+ - Encoder-only ranks: Dict[str, Tensor] of encoder outputs
+ - Language module ranks: language model output (logits or loss)
+ - No role (all modules colocated): language model output
+ """
+ # Get any tensors passed via set_input_tensor
+ input_tensors = getattr(self, 'input_tensors', None)
+
+ if self.role.mode == ModuleLayout.UNIFIED:
+ return self._forward_all_modules(
+ input_ids,
+ position_ids,
+ attention_mask,
+ loss_mask,
+ labels,
+ modality_inputs,
+ packing_kwargs,
+ )
+
+ if self.role.mode == ModuleLayout.NON_COLOCATED:
+ if self.role.has_modality_modules:
+ return self._forward_encoders(modality_inputs, input_tensors), loss_mask
+
+ if self.role.has_language_module:
+ return (
+ self._forward_language_module(
+ input_ids, position_ids, attention_mask, labels, input_tensors
+ ),
+ loss_mask,
+ )
+
+ raise RuntimeError(f"Rank has no modules assigned in role: {self.role}")
+
+ raise NotImplementedError(f"Pipeline mode {self.role.mode} is not yet supported")
+
+ def _forward_encoders(
+ self,
+ modality_inputs: Optional[Dict[str, Dict[str, Any]]],
+ input_tensors: Optional[Dict[str, torch.Tensor]],
+ ) -> Dict[str, torch.Tensor]:
+ """Forward pass for encoder modules on this rank.
+
+ Args:
+ modality_inputs: Raw inputs for each modality (images, audio, etc.)
+ input_tensors: Hidden states from previous pipeline stages
+
+ Returns:
+ Dict mapping encoder names to their output tensors
+ """
+ outputs = {}
+
+ for encoder_name in self.role.modality_module_names:
+ if encoder_name not in self.modality_submodules:
+ continue
+
+ submodule = self.modality_submodules[encoder_name]
+ output = submodule.forward(
+ encoder_inputs=modality_inputs.get(encoder_name) if modality_inputs else None,
+ hidden_states=input_tensors.get(encoder_name) if input_tensors else None,
+ )
+
+ if output is not None:
+ outputs[encoder_name] = output
+
+ return outputs
+
+ def _forward_language_module(
+ self,
+ input_ids: torch.Tensor,
+ position_ids: Optional[torch.Tensor],
+ attention_mask: Optional[torch.Tensor],
+ labels: Optional[torch.Tensor],
+ input_tensors: Optional[Dict[str, torch.Tensor]],
+ ) -> torch.Tensor:
+ """Forward pass for language module on this rank.
+
+ Args:
+ input_ids: Token IDs
+ position_ids: Position IDs
+ attention_mask: Attention mask
+ labels: Labels for loss computation
+ input_tensors: Hidden states or embeddings from previous stage
Returns:
- tuple: Tuple containing model outputs and loss mask
+ Language model output (hidden states, logits, or loss depending on stage)
"""
+ lang_name = MIMO_LANGUAGE_MODULE_KEY
+
+ if self.role.is_first_stage(lang_name):
+ # First stage: receive encoder embeddings, combine with text, pass to LM
+ # Build modality embeddings dict from encoder outputs
+ modality_embeddings = {}
+ if input_tensors:
+ for name, tensor in input_tensors.items():
+ if name != lang_name:
+ modality_embeddings[name] = tensor
+
+ # Get text embeddings
+ text_embeddings = self.get_text_embeddings(
+ input_ids, position_ids, self.special_token_ids
+ )
+ modality_embeddings["text"] = text_embeddings
+
+ # Combine all embeddings
+ combined_embeddings = self.align_embeddings_by_token_positions(
+ modality_embeddings=modality_embeddings,
+ input_ids=input_ids,
+ special_token_ids=self.special_token_ids,
+ )
+
+ lm_output = self.language_model(
+ input_ids=None,
+ position_ids=None,
+ decoder_input=combined_embeddings,
+ labels=labels,
+ attention_mask=attention_mask,
+ )
+ else:
+ # Non-first stage: receive hidden states from previous LM stage
+ hidden_states = input_tensors.get(lang_name) if input_tensors else None
+
+ # Set input tensor on language model for PP (unwrap DDP to reach GPTModel)
+ if hidden_states is not None:
+ underlying_lm = unwrap_model(self.language_model)
+ if hasattr(underlying_lm, 'set_input_tensor'):
+ underlying_lm.set_input_tensor(hidden_states)
+
+ lm_output = self.language_model(
+ input_ids=None,
+ position_ids=None,
+ decoder_input=None,
+ labels=labels,
+ attention_mask=attention_mask,
+ )
+
+ # Key output for non-last stages so schedule can route to next LM stage
+ if not self.role.is_last_stage(lang_name):
+ return {lang_name: lm_output}
+
+ return lm_output
+
+ def _forward_all_modules(
+ self,
+ input_ids: torch.Tensor,
+ position_ids: Optional[torch.Tensor],
+ attention_mask: Optional[torch.Tensor],
+ loss_mask: Optional[torch.Tensor],
+ labels: Optional[torch.Tensor],
+ modality_inputs: Optional[Dict[str, Dict[str, Any]]],
+ packing_kwargs: Optional[dict] = None,
+ ):
+ """Forward pass when all modules are on all ranks (no multi-module PP).
+
+ This is the original behavior, preserved for backward compatibility.
+ """
+ # If packing_kwargs is provided, construct PackedSeqParams
+ packed_seq_params = None
+ if packing_kwargs is not None:
+ # Ensure correct dtype for seqlens tensors
+ for key in packing_kwargs:
+ if 'cu_seqlens' in key and packing_kwargs[key] is not None:
+ packing_kwargs[key] = packing_kwargs[key].to(dtype=torch.int32)
+ packed_seq_params = PackedSeqParams(**packing_kwargs)
+ packed_seq_params.qkv_format = 'thd'
+ logger.debug(f"Packed sequence parameters: {packed_seq_params}")
+
# 1. Process each modality to get embeddings
modality_embeddings = {}
for modality_name, submodule in self.modality_submodules.items():
- # Process the modality through its submodule
if (
modality_inputs
and modality_name in modality_inputs
and modality_inputs[modality_name] is not None
):
logger.debug(f"Processing {modality_name} modality")
- # Get embeddings for this modality
embeddings = submodule.forward(encoder_inputs=modality_inputs[modality_name])
if embeddings is not None:
- # All embeddings are now in the format [num_tokens, hidden_dim]
modality_embeddings[modality_name] = embeddings
logger.debug(
f"Generated embeddings for {modality_name} with shape {embeddings.shape}"
@@ -271,20 +542,42 @@ def forward(
# 2. Merge embeddings from different modalities
logger.debug(f"Merging embeddings from {len(modality_embeddings)} modalities")
combined_embeddings = self.align_embeddings_by_token_positions(
- modality_embeddings=modality_embeddings, # [num_tokens, hidden_dim] for each modality
- input_ids=input_ids, # Pass in batch-first format [b, s]
+ modality_embeddings=modality_embeddings,
+ input_ids=input_ids,
special_token_ids=self.special_token_ids,
- ) # [s, b, h]
+ )
logger.debug(f"Combined embeddings shape: {combined_embeddings.shape}")
- # 3. Forward pass through language model
+ # 3. If sharding is needed, apply PartitionAdapter.
+ # combined_embeddings is [S, B, H]; transpose to [B, S, H] for shard() which expects
+ # batch-first layout (required by get_batch_on_this_cp_rank). After CP sharding each
+ # rank holds [B, S/cp, H]; transpose back to [S/cp, B, H] for the language model.
+ if self.partition_adapter is not None:
+ combined_embeddings = combined_embeddings.transpose(0, 1).contiguous() # [B, S, H]
+ combined_embeddings, labels, loss_mask, _, packed_seq_params = (
+ self.partition_adapter.shard(
+ embeddings=combined_embeddings,
+ labels=labels,
+ loss_mask=loss_mask,
+ attention_mask=attention_mask,
+ packed_seq_params=packed_seq_params,
+ )
+ )
+ # shard() returns embeddings in [B, S/cp, H]; transpose to [S/cp, B, H]
+ # which is what the language model expects.
+ if combined_embeddings is not None:
+ combined_embeddings = combined_embeddings.transpose(0, 1).contiguous()
+
+ # 5. Forward pass through language model
lm_output = self.language_model(
input_ids=None,
position_ids=None,
decoder_input=combined_embeddings,
labels=labels,
- attention_mask=attention_mask,
+ attention_mask=None,
+ packed_seq_params=packed_seq_params,
)
+
logger.debug(f"Language model output shape: {lm_output.shape}")
return lm_output, loss_mask
diff --git a/megatron/core/models/mimo/optimizer.py b/megatron/core/models/mimo/optimizer.py
new file mode 100644
index 00000000000..1a79c1f91ff
--- /dev/null
+++ b/megatron/core/models/mimo/optimizer.py
@@ -0,0 +1,379 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+"""Optimizer for MIMO models with heterogeneous parallelism."""
+
+from __future__ import annotations
+
+from copy import deepcopy
+from dataclasses import dataclass
+from typing import TYPE_CHECKING, Dict, List, Optional, Tuple
+
+import torch
+
+from megatron.core.dist_checkpointing.mapping import ShardedObject
+from megatron.core.optimizer.clip_grads import clip_grad_by_total_norm_fp32
+from megatron.core.optimizer.optimizer import MegatronOptimizer
+from megatron.core.optimizer.optimizer_config import OptimizerConfig
+from megatron.core.process_groups_config import ProcessGroupCollection
+
+if TYPE_CHECKING:
+ from megatron.core.hyper_comm_grid import HyperCommGrid
+
+
+@dataclass
+class ModuleOptimizerInfo:
+ """Optimizer info for a single module."""
+
+ optimizer: Optional[MegatronOptimizer]
+ grid: Optional[HyperCommGrid]
+ pg_collection: Optional[ProcessGroupCollection]
+ is_active: bool
+
+
+class MimoOptimizer(MegatronOptimizer):
+ """
+ Optimizer for MimoModel with heterogeneous parallelism.
+
+ Each module gets its own optimizer. Global gradient norm is computed
+ across all modules via all_reduce MAX.
+ """
+
+ def __init__(self, module_infos: Dict[str, ModuleOptimizerInfo], config: OptimizerConfig):
+ self.module_infos = module_infos
+ self.config = config
+ self._active_optimizers: List[MegatronOptimizer] = [
+ info.optimizer
+ for info in module_infos.values()
+ if info.is_active and info.optimizer is not None
+ ]
+ self.is_stub_optimizer = len(self._active_optimizers) == 0
+ self.optimizer = None # Base class compat
+
+ @torch.no_grad()
+ def prepare_grads(self) -> bool:
+ found_inf = False
+ for opt in self._active_optimizers:
+ found_inf |= opt.prepare_grads()
+ return found_inf
+
+ @torch.no_grad()
+ def get_grad_norm(self) -> float:
+ """Compute global gradient norm across all modules via all_reduce MAX."""
+ num_modules = len(self.module_infos)
+ norm_sq = torch.zeros(num_modules, device="cuda", dtype=torch.float32)
+
+ for i, (name, info) in enumerate(sorted(self.module_infos.items())):
+ if info.is_active and info.optimizer:
+ module_norm = info.optimizer.get_grad_norm() or 0.0
+ norm_sq[i] = module_norm**2
+
+ torch.distributed.all_reduce(norm_sq, op=torch.distributed.ReduceOp.MAX)
+ return torch.sqrt(norm_sq.sum()).item()
+
+ @torch.no_grad()
+ def step(self) -> Tuple[bool, Optional[float], Optional[int]]:
+ found_inf = self.prepare_grads()
+ # Synchronize found_inf across all ranks to prevent deadlock:
+ # if encoder ranks detect inf but LLM ranks don't, the early return
+ # would skip the all_reduce in get_grad_norm(), causing a hang.
+ found_inf_tensor = torch.tensor([found_inf], dtype=torch.float32, device="cuda")
+ torch.distributed.all_reduce(found_inf_tensor, op=torch.distributed.ReduceOp.MAX)
+ found_inf = found_inf_tensor.item() > 0
+ if found_inf:
+ return False, None, None
+
+ grad_norm = self.get_grad_norm()
+
+ # Clip with global norm
+ for opt in self._active_optimizers:
+ if getattr(opt, "is_stub_optimizer", False):
+ continue
+ params = opt.get_parameters()
+ if params and opt.config.clip_grad > 0.0:
+ clip_grad_by_total_norm_fp32(
+ params,
+ max_norm=opt.config.clip_grad,
+ total_norm=grad_norm,
+ use_decoupled_grad=opt.config.use_precision_aware_optimizer,
+ )
+
+ num_zeros = self.count_zeros() if self.config.log_num_zeros_in_grad else None
+ success = self.step_with_ready_grads()
+
+ return success, grad_norm, num_zeros
+
+ @torch.no_grad()
+ def step_with_ready_grads(self) -> bool:
+ success = True
+ for opt in self._active_optimizers:
+ success &= opt.step_with_ready_grads()
+ return success
+
+ def zero_grad(self, set_to_none: bool = True):
+ for opt in self._active_optimizers:
+ opt.zero_grad(set_to_none)
+
+ def get_loss_scale(self) -> torch.Tensor:
+ if self._active_optimizers:
+ return self._active_optimizers[0].get_loss_scale()
+ return torch.tensor([1.0], dtype=torch.float32, device="cuda")
+
+ def count_zeros(self) -> int:
+ return sum(opt.count_zeros() for opt in self._active_optimizers)
+
+ @property
+ def param_groups(self) -> List[dict]:
+ """Combined param groups from all active module optimizers."""
+ groups = []
+ for opt in self._active_optimizers:
+ groups.extend(opt.param_groups)
+ return groups
+
+ # Checkpointing
+
+ def state_dict(self):
+ return {
+ name: info.optimizer.state_dict() if info.is_active and info.optimizer else None
+ for name, info in self.module_infos.items()
+ }
+
+ def load_state_dict(self, state_dict: Dict):
+ """Load per-module optimizer state dicts.
+
+ Reassembles param_groups and grad_scaler that were extracted and saved
+ as ShardedObjects by sharded_state_dict(), then delegates to each
+ per-module optimizer's load_state_dict.
+ """
+ for name, info in self.module_infos.items():
+ if not (info.is_active and info.optimizer):
+ continue
+ module_sd = state_dict.get(name)
+ if module_sd is None:
+ continue
+
+ for sub_sd, inner_opt in _iter_optimizer_sub_dicts(module_sd, info.optimizer):
+ _restore_param_groups(sub_sd, inner_opt, name)
+ _restore_grad_scaler(sub_sd)
+
+ info.optimizer.load_state_dict(module_sd)
+
+ def sharded_state_dict(self, model_sharded_state_dict, is_loading: bool = False, **kwargs):
+ """Build sharded state dict, routing param_groups and grad_scaler
+ through distributed save as ShardedObjects (common.pt is rank-0 only,
+ which misses LLM optimizer state in non-colocated mode).
+ """
+ sharded_state = {}
+ for name, info in self.module_infos.items():
+ if info.is_active and info.optimizer:
+ module_sd = info.optimizer.sharded_state_dict(
+ model_sharded_state_dict, is_loading, **kwargs
+ )
+ replica_id = _get_replica_id(info.pg_collection)
+
+ for idx, (sub_sd, _) in enumerate(
+ _iter_optimizer_sub_dicts(module_sd, info.optimizer)
+ ):
+ suffix = f'.{idx}' if idx > 0 else ''
+ _extract_param_groups(sub_sd, name, suffix, replica_id)
+ _extract_grad_scaler(sub_sd, name, suffix, replica_id)
+
+ sharded_state[name] = module_sd
+ else:
+ sharded_state[name] = {}
+ return sharded_state
+
+ def reload_model_params(self, state_dict=None):
+ for opt in self._active_optimizers:
+ opt.reload_model_params(state_dict)
+
+
+def _iter_optimizer_sub_dicts(module_sd, optimizer):
+ """Yield (sub_state_dict, inner_optimizer) pairs.
+
+ For a single optimizer, yields (module_sd, optimizer) once.
+ For ChainedOptimizer with N>1 inner optimizers, yields
+ (module_sd[i], chained_optimizers[i]) for each.
+ """
+ from megatron.core.optimizer.optimizer import ChainedOptimizer
+
+ if isinstance(optimizer, ChainedOptimizer) and len(optimizer.chained_optimizers) > 1:
+ for idx, inner_opt in enumerate(optimizer.chained_optimizers):
+ yield module_sd[idx], inner_opt
+ else:
+ yield module_sd, optimizer
+
+
+def _extract_param_groups(sub_sd, module_name, suffix, replica_id):
+ """Save: extract param_groups from optimizer sub-dict into a ShardedObject."""
+ opt_sub = sub_sd.get('optimizer')
+ if isinstance(opt_sub, dict) and 'param_groups' in opt_sub:
+ pg = deepcopy(opt_sub['param_groups'])
+ for group in pg:
+ group['params'] = []
+ sub_sd[f'_mimo_param_groups{suffix}'] = ShardedObject(
+ f'optimizer.mimo.{module_name}{suffix}.param_groups',
+ pg,
+ (1,),
+ (0,),
+ replica_id=replica_id,
+ )
+ del opt_sub['param_groups']
+
+
+def _extract_grad_scaler(sub_sd, module_name, suffix, replica_id):
+ """Save: extract grad_scaler into a ShardedObject."""
+ if 'grad_scaler' in sub_sd and sub_sd['grad_scaler'] is not None:
+ sub_sd[f'_mimo_grad_scaler{suffix}'] = ShardedObject(
+ f'optimizer.mimo.{module_name}{suffix}.grad_scaler',
+ sub_sd.pop('grad_scaler'),
+ (1,),
+ (0,),
+ replica_id=replica_id,
+ )
+
+
+def _restore_param_groups(sub_sd, inner_optimizer, module_name):
+ """Load: restore param_groups with current param IDs from the inner optimizer."""
+ # Find the _mimo_param_groups key (may have a suffix for chained optimizers)
+ pg_key = None
+ for k in list(sub_sd.keys()):
+ if k.startswith('_mimo_param_groups'):
+ pg_key = k
+ break
+ if pg_key is None:
+ return
+
+ loaded_pg = sub_sd.pop(pg_key)
+ # Get current param IDs from the inner torch optimizer's state_dict
+ current_pg = inner_optimizer.optimizer.state_dict()['param_groups']
+ if len(loaded_pg) != len(current_pg):
+ raise ValueError(
+ f"Optimizer '{module_name}': checkpoint has {len(loaded_pg)} param_groups "
+ f"but current optimizer has {len(current_pg)}"
+ )
+ for loaded_g, current_g in zip(loaded_pg, current_pg):
+ loaded_g['params'] = current_g['params']
+ sub_sd['optimizer']['param_groups'] = loaded_pg
+
+
+def _restore_grad_scaler(sub_sd):
+ """Load: restore grad_scaler from ShardedObject key."""
+ for k in list(sub_sd.keys()):
+ if k.startswith('_mimo_grad_scaler'):
+ sub_sd['grad_scaler'] = sub_sd.pop(k)
+ break
+
+
+def _get_replica_id(pg_collection: Optional[ProcessGroupCollection]) -> tuple:
+ """Build replica_id tuple for ShardedObject deduplication.
+
+ Includes pp_rank so only one PP stage writes the metadata,
+ and dp_rank so only dp_rank=0 writes (others are replicas).
+ """
+ assert pg_collection is not None, "pg_collection required for checkpoint replica_id"
+ assert (
+ hasattr(pg_collection, 'pp') and pg_collection.pp is not None
+ ), "pg_collection.pp must be set for checkpoint deduplication"
+ assert (
+ hasattr(pg_collection, 'dp') and pg_collection.dp is not None
+ ), "pg_collection.dp must be set for checkpoint deduplication"
+ return (0, pg_collection.pp.rank(), pg_collection.dp.rank())
+
+
+def _get_pg_collection_for_optimizer(grid) -> ProcessGroupCollection:
+ """Create ProcessGroupCollection from HyperCommGrid for optimizer use.
+
+ Only fetches process groups required by the optimizer. Assumes all groups
+ are pre-created in the grid via grid.create_pg() - does not create any new groups.
+
+ The following groups must be pre-created in the grid before calling this function:
+ grid.create_pg(["dp"])
+ grid.create_pg(["dp", "cp"])
+ grid.create_pg(["tp"])
+ grid.create_pg(["pp"])
+ grid.create_pg(["tp", "pp"])
+ grid.create_pg(["tp", "ep", "pp"])
+ grid.create_pg(["dp", "ep"])
+ grid.create_pg(["tp", "cp", "ep", "pp", "dp"])
+
+ Args:
+ grid: HyperCommGrid with pre-created process groups.
+
+ Returns:
+ ProcessGroupCollection containing optimizer-required groups:
+ - dp: Data parallel group
+ - dp_cp: Data parallel with context parallel
+ - tp: Tensor parallel group
+ - mp: Model parallel group (tp × pp)
+ - tp_ep_pp: Expert tensor-model-pipeline group
+ - expt_dp: Expert data parallel group
+ """
+ pg = ProcessGroupCollection()
+
+ # Core groups needed by optimizer and checkpointing
+ pg.dp = grid.get_pg("dp")
+ pg.dp_cp = grid.get_pg(["dp", "cp"])
+ pg.tp = grid.get_pg("tp")
+ pg.pp = grid.get_pg("pp")
+ pg.mp = grid.get_pg(["tp", "pp"])
+
+ # Expert groups
+ pg.tp_ep_pp = grid.get_pg(["tp", "ep", "pp"])
+ pg.expt_dp = grid.get_pg(["dp", "ep"])
+
+ # Distributed optimizer grad stats group: must span all dimensions so grad norm
+ # and found-inf all-reduces see every unique gradient shard. TP/PP/EP ranks hold
+ # different parameters, DP ranks hold different optimizer shards after reduce-scatter.
+ # This mirrors standard Megatron's intra_distributed_optimizer_instance_group which
+ # spans the full world when num_distributed_optimizer_instances == 1.
+ pg.intra_dist_opt = grid.get_pg(["tp", "cp", "ep", "pp", "dp"])
+
+ return pg
+
+
+def get_mimo_optimizer(mimo_model: "MimoModel", config: OptimizerConfig) -> MimoOptimizer:
+ """Create optimizer for MimoModel with heterogeneous parallelism."""
+ from megatron.core.optimizer import get_megatron_optimizer
+
+ grid_map = mimo_model.mimo_config.module_to_grid_map
+ from megatron.core.models.mimo.config.role import MIMO_LANGUAGE_MODULE_KEY
+
+ lang_key = MIMO_LANGUAGE_MODULE_KEY
+
+ module_infos: Dict[str, ModuleOptimizerInfo] = {}
+
+ for module_name, grid in grid_map.items():
+ is_active = grid.is_current_rank_in_grid()
+
+ optimizer = None
+ pg_collection = _get_pg_collection_for_optimizer(grid)
+
+ if is_active:
+ if module_name == lang_key:
+ module = mimo_model.language_model
+ else:
+ module = mimo_model.modality_submodules[module_name]
+
+ if module is not None:
+ assert (
+ not hasattr(module, 'ddp_config')
+ or module.ddp_config is None
+ or module.ddp_config.num_distributed_optimizer_instances == 1
+ ), (
+ "MIMO optimizer does not yet support "
+ "num_distributed_optimizer_instances > 1. "
+ f"Module '{module_name}' has "
+ f"{module.ddp_config.num_distributed_optimizer_instances} instances."
+ )
+ optimizer = get_megatron_optimizer(
+ config=config,
+ model_chunks=[module],
+ pg_collection=pg_collection,
+ use_gloo_process_groups=False,
+ )
+
+ module_infos[module_name] = ModuleOptimizerInfo(
+ optimizer=optimizer, grid=grid, pg_collection=pg_collection, is_active=is_active
+ )
+
+ return MimoOptimizer(module_infos, config)
diff --git a/megatron/core/models/mimo/partition/utils.py b/megatron/core/models/mimo/partition/utils.py
new file mode 100644
index 00000000000..0b43e5548ff
--- /dev/null
+++ b/megatron/core/models/mimo/partition/utils.py
@@ -0,0 +1,260 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+"""Token and weight partitioning helper (CP, TP, SP).
+
+The adapter slices sequences across *context-parallel* ranks and can further
+scatter them across *sequence-parallel* ranks when sequence-parallelism is
+enabled.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Optional, Tuple
+
+import torch # type: ignore[import-not-found]
+from torch.distributed import ProcessGroup # type: ignore[import-not-found]
+
+from megatron.core import tensor_parallel
+from megatron.core.model_parallel_config import ModelParallelConfig
+from megatron.core.packed_seq_params import PackedSeqParams
+from megatron.core.parallel_state import get_context_parallel_group, get_tensor_model_parallel_group
+from megatron.core.utils import (
+ get_batch_on_this_cp_rank,
+ get_pg_rank,
+ get_pg_size,
+ is_te_min_version,
+)
+
+try:
+ import transformer_engine_torch as tex # type: ignore
+
+ _HAVE_TEX = True
+except ModuleNotFoundError: # pragma: no cover
+ tex = None # type: ignore
+ _HAVE_TEX = False
+
+
+@dataclass(frozen=True)
+class PartitionConfig:
+ """Minimal runtime information needed to shard inputs.
+
+ NOTE: Always construct PartitionConfig using the provided classmethod
+ (from_mp_config) to ensure all fields, including cp_group and tp_group,
+ are set correctly.
+ """
+
+ seq_parallel: bool
+ use_cp: bool
+ tp_comm_overlap: bool
+ max_seq_len: int
+ kv_format: str = "sbhd" # "sbhd" | "thd"
+ cp_group: Optional[ProcessGroup] = None
+ tp_group: Optional[ProcessGroup] = None
+
+ @property
+ def is_partitioning_enabled(self) -> bool:
+ """Returns True if context parallelism or sequence parallelism is active."""
+ return self.use_cp or self.seq_parallel
+
+ @classmethod
+ def from_mp_config(
+ cls,
+ mp: ModelParallelConfig,
+ *,
+ max_seq_len: int,
+ kv_format: str = "sbhd",
+ cp_group: Optional[ProcessGroup] = None,
+ tp_group: Optional[ProcessGroup] = None,
+ ) -> "PartitionConfig":
+ """
+ Creates a PartitionConfig from a ModelParallelConfig.
+ """
+ if not isinstance(mp, ModelParallelConfig):
+ raise TypeError("mp must be a ModelParallelConfig instance")
+
+ if mp.context_parallel_size > 1 and cp_group is None:
+ cp_group = get_context_parallel_group()
+
+ if mp.sequence_parallel and tp_group is None:
+ tp_group = get_tensor_model_parallel_group()
+
+ return cls(
+ seq_parallel=mp.sequence_parallel,
+ use_cp=get_pg_size(cp_group) > 1,
+ tp_comm_overlap=mp.tp_comm_overlap,
+ max_seq_len=max_seq_len,
+ kv_format=kv_format,
+ cp_group=cp_group,
+ tp_group=tp_group,
+ )
+
+
+class PartitionAdapter:
+ """Shard batch-first embeddings & label tensors for Context and Sequence Parallelism."""
+
+ def __init__(self, cfg: PartitionConfig):
+ """Initialize the partition adapter.
+ Args:
+ cfg: PartitionConfig, the configuration for the partition adapter.
+ """
+ self.cfg = cfg
+
+ def shard(
+ self,
+ embeddings: torch.Tensor,
+ labels: torch.Tensor,
+ loss_mask: torch.Tensor,
+ attention_mask: torch.Tensor,
+ packed_seq_params: Optional[PackedSeqParams] = None,
+ ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, Optional[PackedSeqParams]]:
+ """
+ Apply context parallel (CP) and sequence parallel (SP) sharding to input tensors.
+
+ All input tensors must be in batch-first layout:
+ - embeddings: (B, S, H)
+ - labels / loss_mask / attention_mask: (B, S)
+
+ After this call embeddings are still in (B, S/cp, H) batch-first layout.
+ The caller is responsible for transposing to (S/cp, B, H) if the language model
+ requires sequence-first tensors.
+
+ Args:
+ embeddings (torch.Tensor):
+ Input embeddings tensor. Shape: (B, S, H)
+ labels (torch.Tensor):
+ Labels tensor. Shape: (B, S)
+ loss_mask (torch.Tensor):
+ Loss mask tensor. Shape: (B, S)
+ attention_mask (torch.Tensor):
+ Attention mask tensor. Shape: (B, S)
+ packed_seq_params (PackedSeqParams, optional):
+ Packed sequence parameters. Defaults to None.
+
+ Returns:
+ Tuple containing:
+ - embeddings (torch.Tensor): Sharded embeddings. Shape: (B, S/cp, H)
+ - labels (torch.Tensor): Possibly sharded labels. Shape: (B, S/cp)
+ - loss_mask (torch.Tensor): Possibly sharded loss mask. Shape: (B, S/cp)
+ - attention_mask (torch.Tensor): Possibly sharded attention mask. Shape: (B, S/cp)
+ - packed_seq_params (PackedSeqParams, optional): Updated packed sequence parameters.
+ """
+ if not (self.cfg.use_cp or self.cfg.seq_parallel):
+ return embeddings, labels, loss_mask, attention_mask, packed_seq_params
+
+ # Sanity-check the sequence length before any sharding happens.
+ if embeddings is not None:
+ shard_factor = None
+ seq_dim = None # which dimension holds the token sequence
+
+ if self.cfg.use_cp and self.cfg.seq_parallel:
+ shard_factor = get_pg_size(self.cfg.tp_group) * get_pg_size(self.cfg.cp_group) * 2
+ seq_dim = 1 # embeddings shape: [B, S, H]
+ elif self.cfg.use_cp:
+ shard_factor = get_pg_size(self.cfg.cp_group) * 2
+ seq_dim = 1
+ elif self.cfg.seq_parallel:
+ shard_factor = get_pg_size(self.cfg.tp_group)
+ seq_dim = 0 # embeddings shape: [S, B, H]
+
+ if shard_factor is not None and (
+ packed_seq_params is None
+ or getattr(packed_seq_params, 'qkv_format', 'sbhd') == 'sbhd'
+ ):
+ assert embeddings.shape[seq_dim] % shard_factor == 0, (
+ f"Sequence length should be divisible by {shard_factor} "
+ "for Sequence/Context parallelism"
+ )
+
+ if self.cfg.seq_parallel and self.cfg.tp_comm_overlap:
+ assert embeddings.shape[seq_dim] == self.cfg.max_seq_len, (
+ "TP Comm overlap requires Vision+Text token length "
+ "== language_max_sequence_length"
+ )
+
+ if self.cfg.use_cp:
+ embeddings, labels, loss_mask, attention_mask, packed_seq_params = (
+ self._apply_context_parallel(
+ embeddings, labels, loss_mask, attention_mask, packed_seq_params
+ )
+ )
+
+ if self.cfg.seq_parallel and embeddings is not None:
+ embeddings = tensor_parallel.scatter_to_sequence_parallel_region(embeddings)
+
+ return embeddings, labels, loss_mask, attention_mask, packed_seq_params
+
+ def _apply_context_parallel(
+ self,
+ embeddings: Optional[torch.Tensor],
+ labels: Optional[torch.Tensor],
+ loss_mask: Optional[torch.Tensor],
+ attention_mask: Optional[torch.Tensor],
+ packed_seq_params: Optional[PackedSeqParams],
+ ) -> Tuple[
+ Optional[torch.Tensor],
+ Optional[torch.Tensor],
+ Optional[torch.Tensor],
+ Optional[torch.Tensor],
+ Optional[PackedSeqParams],
+ ]:
+ """
+ Apply context parallel (CP) sharding to input tensors.
+
+ Args:
+ embeddings (Optional[torch.Tensor]):
+ Input embeddings tensor. Shape: (B, S, H)
+ labels (Optional[torch.Tensor]):
+ Labels tensor. Shape: (B, S)
+ loss_mask (Optional[torch.Tensor]):
+ Loss mask tensor. Shape: (B, S)
+ attention_mask (Optional[torch.Tensor]):
+ Attention mask tensor. Shape: (B, S)
+ packed_seq_params (PackedSeqParams, optional):
+ Packed sequence parameters. Defaults to None.
+
+ Returns:
+ Tuple containing:
+ - embeddings (Optional[torch.Tensor]): Sharded embeddings. Shape: (B, S/cp, H)
+ - labels (Optional[torch.Tensor]): Possibly sharded labels. Shape: (B, S/cp)
+ - loss_mask (Optional[torch.Tensor]): Possibly sharded loss mask. Shape: (B, S/cp)
+ - attention_mask (Optional[torch.Tensor]): Possibly sharded attention mask.
+ Shape: (B, S/cp)
+ - packed_seq_params (PackedSeqParams, optional): Updated packed sequence parameters.
+ """
+ if not self.cfg.use_cp:
+ return embeddings, labels, loss_mask, attention_mask, packed_seq_params
+
+ # Distribute sequence across CP ranks
+ batch = dict()
+ if embeddings is not None:
+ batch["embeddings"] = embeddings
+ if labels is not None:
+ batch["labels"] = labels
+ if loss_mask is not None:
+ batch["loss_mask"] = loss_mask
+ if attention_mask is not None:
+ batch["attention_mask"] = attention_mask
+
+ if packed_seq_params is None or getattr(packed_seq_params, 'qkv_format', 'sbhd') == 'sbhd':
+ batch = get_batch_on_this_cp_rank(batch)
+ else:
+ assert _HAVE_TEX and is_te_min_version("1.10.0"), (
+ "Please update Transformer Engine to >= 1.10 "
+ "to use Context Parallel with THD format data"
+ )
+ assert self.cfg.cp_group is not None
+ cp_size = get_pg_size(self.cfg.cp_group)
+ cp_rank = get_pg_rank(self.cfg.cp_group)
+ for key, data in batch.items():
+ index = tex.thd_get_partitioned_indices(
+ packed_seq_params.cu_seqlens_q_padded, data.size(1), cp_size, cp_rank
+ )
+ batch[key] = data.index_select(1, index)
+
+ # Extract sharded tensors; embeddings remain in [B, S/cp, H] — the caller
+ # is responsible for transposing to [S/cp, B, H] for the language model.
+ embeddings = batch.get("embeddings", None)
+ labels = batch.get("labels", None)
+ loss_mask = batch.get("loss_mask", None)
+ attention_mask = batch.get("attention_mask", None)
+
+ return embeddings, labels, loss_mask, attention_mask, packed_seq_params
diff --git a/megatron/core/models/mimo/submodules/audio.py b/megatron/core/models/mimo/submodules/audio.py
index ae907d7ac86..6db2782d82f 100644
--- a/megatron/core/models/mimo/submodules/audio.py
+++ b/megatron/core/models/mimo/submodules/audio.py
@@ -1,16 +1,11 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
-import logging
-from typing import Any, Dict, List, Optional
+from typing import Dict, List, Optional
-import torch
import torch.nn as nn
from megatron.core.models.mimo.submodules.base import ModalitySubmodules
-# Initialize logger
-logger = logging.getLogger(__name__)
-
class AudioModalitySubmodules(ModalitySubmodules):
"""Audio modality submodules for encoding, decoding, and projecting audio data."""
@@ -32,7 +27,13 @@ def __init__(
output_projections: List of output projection modules
**kwargs: Additional keyword arguments
"""
- super().__init__(encoders, decoders, input_projections, output_projections, **kwargs)
+ super().__init__(
+ encoders=encoders,
+ decoders=decoders,
+ input_projections=input_projections,
+ output_projections=output_projections,
+ **kwargs,
+ )
if self.input_projections:
assert (
@@ -44,112 +45,6 @@ def __init__(
len(self.output_projections) <= 1
), "AudioModalitySubmodules currently supports only one output projection"
- def encode(self, encoders_data_batch: Dict) -> List[torch.Tensor]:
- """Encode audio data into a sequence of embeddings.
-
- Args:
- encoders_data_batch: Dictionary containing encoder-specific inputs.
- Keys should match encoder names in self.encoders.
- Each encoder receives its own specific inputs.
-
- Returns:
- List of encoded audio embeddings, one from each encoder.
- Each embedding is a flattened tensor of shape [total_tokens, hidden_dim]
-
- Raises:
- ValueError: If no data is provided for any encoder or if there's a parameter mismatch.
- """
- if not encoders_data_batch:
- return []
-
- embeddings = []
-
- for name, encoder in self.encoders.items():
- if name not in encoders_data_batch:
- raise ValueError(f"No inputs found for encoder '{name}'")
-
- encoder_inputs = encoders_data_batch[name]
-
- # Process inputs through the encoder
- encoder_outputs = encoder(**encoder_inputs)
- logger.debug(f"Encoder '{name}' output shape: {encoder_outputs.shape}")
- if encoder_outputs.ndim == 3:
- # its b,s,h -> we need to flatten it to b*s,h
- encoder_outputs = encoder_outputs.reshape(-1, encoder_outputs.size(-1))
- embeddings.append(encoder_outputs)
- elif encoder_outputs.ndim == 2:
- # its b*s,h -> encoder already returned the flattened output
- embeddings.append(encoder_outputs)
- else:
- raise ValueError(
- f"Encoder '{name}' output shape {encoder_outputs.shape} is not supported"
- "Expected 3D (b,s,h) or 2D (b*s,h) tensor, got {encoder_outputs.ndim}D"
- )
- return embeddings
-
- def decode(self, embeddings: torch.Tensor, data_batch: Dict) -> torch.Tensor:
+ def decode(self, embeddings, data_batch: Dict):
"""Decode embeddings into audio data."""
raise NotImplementedError("Audio decoding not implemented yet")
-
- def combine_embeddings(self, embeddings: List[torch.Tensor]) -> torch.Tensor:
- """Combine embeddings from different encoders."""
- if not embeddings:
- raise ValueError("Cannot combine empty list of embeddings")
-
- if len(embeddings) == 1:
- return embeddings[0]
-
- # Concatenate along sequence dimension
- # each embedding is [total_tokens, hidden_dim]
- combined = torch.cat(embeddings, dim=0)
- logger.debug(f"Combined audio embeddings shape: {combined.shape}")
- return combined
-
- def project_embeddings(
- self, embeddings: List[torch.Tensor], is_input: bool = True
- ) -> torch.Tensor:
- """Project embeddings to the language model dimension space."""
-
- if is_input:
- embeddings = self.combine_embeddings(embeddings)
-
- # Get the appropriate projections
- projections = self.input_projections if is_input else self.output_projections
-
- # Apply projection if available
- if projections:
- # We've asserted in __init__ that there's only one projection
- projection = projections[0]
- projected = projection(embeddings)
- logger.debug(f"Post-projection audio embeddings shape: {projected.shape}")
- return projected
-
- return embeddings
-
- def forward(self, encoder_inputs: Dict[str, Any]) -> Optional[torch.Tensor]:
- """Forward pass for audio modality submodules.
-
- Args:
- encoder_inputs: Dictionary where keys match encoder names in self.encoders
- and values are dictionaries of encoder-specific parameters.
- Example: {
- "whisper": {"input_features": features},
- "wav2vec": {"input_values": waveform}
- }
-
- Returns:
- Flattened audio embeddings with shape [total_embeddings, hidden_dim],
- or None if no valid inputs were provided.
- """
-
- embeddings = self.encode(encoder_inputs)
- # embeddings is a list of tensors, each tensor is a flattened audio embedding
-
- # If no embeddings were produced, return None
- if not embeddings:
- return None
-
- # Project embeddings
- projected = self.project_embeddings(embeddings, is_input=True)
- logger.debug(f"Projected audio embeddings shape: {projected.shape}")
- return projected # [total_embeddings, hidden_dim]
diff --git a/megatron/core/models/mimo/submodules/base.py b/megatron/core/models/mimo/submodules/base.py
index 8b11ba7fcb9..3b54fd737f2 100644
--- a/megatron/core/models/mimo/submodules/base.py
+++ b/megatron/core/models/mimo/submodules/base.py
@@ -9,6 +9,7 @@
import torch.nn as nn
from megatron.core.transformer.spec_utils import ModuleSpec, build_module
+from megatron.core.transformer.utils import sharded_state_dict_default
# Initialize logger
logger = logging.getLogger(__name__)
@@ -42,14 +43,30 @@ def __init__(
decoders: Optional[Dict[str, nn.Module]] = None,
input_projections: Optional[List[nn.Module]] = None,
output_projections: Optional[List[nn.Module]] = None,
+ is_first_stage: bool = True,
+ is_last_stage: bool = True,
+ pg_collection=None,
**kwargs,
) -> None:
- """Initialize the modality submodules."""
+ """Initialize the modality submodules.
+
+ Args:
+ encoders: Dict of encoder modules
+ decoders: Dict of decoder modules
+ input_projections: List of input projection modules
+ output_projections: List of output projection modules
+ is_first_stage: Whether this is the first PP stage for this module
+ is_last_stage: Whether this is the last PP stage for this module
+ pg_collection: Process group collection for this module
+ """
super().__init__()
self.encoders = nn.ModuleDict(encoders or {})
self.decoders = nn.ModuleDict(decoders or {})
self.input_projections = nn.ModuleList(input_projections or [])
self.output_projections = nn.ModuleList(output_projections or [])
+ self.pg_collection = pg_collection
+ self._is_first_stage: bool = is_first_stage
+ self._is_last_stage: bool = is_last_stage
warnings.warn(
"ModalitySubmodules is experimental and still under active development. "
@@ -58,21 +75,65 @@ def __init__(
stacklevel=2,
)
+ def sharded_state_dict(self, prefix='', sharded_offsets=(), metadata=None):
+ """Iterate into ModuleDict/ModuleList children for TP-aware checkpointing.
+
+ Injects dp_cp_group from pg_collection into metadata to avoid
+ parallel_state global fallback in ensure_metadata_has_dp_cp_group.
+ """
+ if self.pg_collection is not None:
+ assert (
+ hasattr(self.pg_collection, 'dp_cp') and self.pg_collection.dp_cp is not None
+ ), "pg_collection is missing dp_cp group"
+ metadata = dict(metadata) if metadata else {}
+ metadata['dp_cp_group'] = self.pg_collection.dp_cp
+
+ sharded_sd = {}
+ for name, container in self.named_children():
+ for sub_name, module in container.named_children():
+ sharded_sd.update(
+ sharded_state_dict_default(
+ module, f'{prefix}{name}.{sub_name}.', sharded_offsets, metadata
+ )
+ )
+ return sharded_sd
+
+ @property
+ def is_first_stage(self) -> bool:
+ """Whether this is the first pipeline stage for this module."""
+ return self._is_first_stage
+
+ @property
+ def is_last_stage(self) -> bool:
+ """Whether this is the last pipeline stage for this module."""
+ return self._is_last_stage
+
@classmethod
- def from_spec(cls, module_spec: ModuleSpec) -> 'ModalitySubmodules':
+ def from_spec(
+ cls, module_spec: ModuleSpec, is_first_stage: bool = True, is_last_stage: bool = True
+ ) -> 'ModalitySubmodules':
"""Create a modality submodule from ModuleSpec configuration.
Args:
module_spec (ModuleSpec): The module specification for this modality submodule
+ is_first_stage (bool): Whether this is the first pipeline stage for this module.
+ Controls encoder initialization and output projection initialization
+ (output projections only built on first stage). Defaults to True.
+ is_last_stage (bool): Whether this is the last pipeline stage for this module.
+ Controls input projection initialization (only built on last stage).
+ Defaults to True.
Returns:
ModalitySubmodules: An instance of the modality submodule
"""
- logger.debug(f"Creating {cls.__name__} from spec")
+ logger.debug(
+ f"Creating {cls.__name__} from spec (is_first_stage={is_first_stage}, "
+ f"is_last_stage={is_last_stage})"
+ )
params = module_spec.params or {}
submodules = module_spec.submodules or {}
- # Build component lists from submodules dictionary
+ # Build encoders (needed on all stages for pipeline processing)
encoders = {}
if 'encoders' in submodules:
for encoder_name, encoder_spec in submodules['encoders'].items():
@@ -80,6 +141,7 @@ def from_spec(cls, module_spec: ModuleSpec) -> 'ModalitySubmodules':
encoder = build_module(encoder_spec)
encoders[encoder_name] = encoder
+ # Build decoders (needed on all stages for pipeline processing)
decoders = {}
if 'decoders' in submodules:
for decoder_name, decoder_spec in submodules['decoders'].items():
@@ -87,23 +149,31 @@ def from_spec(cls, module_spec: ModuleSpec) -> 'ModalitySubmodules':
decoder = build_module(decoder_spec)
decoders[decoder_name] = decoder
+ # Build input projections only on last stage
+ # (projection happens after encoding, before sending to language model)
input_projections = []
- if 'input_projections' in submodules:
+ if is_last_stage and 'input_projections' in submodules:
for proj_spec in submodules['input_projections']:
logger.debug(
f"Building {cls.__name__} input projection: {proj_spec.module.__name__}"
)
projection = build_module(proj_spec)
input_projections.append(projection)
+ elif 'input_projections' in submodules:
+ logger.debug(f"Skipping {cls.__name__} input projections (not last stage)")
+ # Build output projections only on first stage
+ # (projection happens before decoding, after receiving from language model)
output_projections = []
- if 'output_projections' in submodules:
+ if is_first_stage and 'output_projections' in submodules:
for proj_spec in submodules['output_projections']:
logger.debug(
f"Building {cls.__name__} output projection: {proj_spec.module.__name__}"
)
projection = build_module(proj_spec)
output_projections.append(projection)
+ elif 'output_projections' in submodules:
+ logger.debug(f"Skipping {cls.__name__} output projections (not first stage)")
# Pass any additional parameters from the params dictionary
additional_params = params.copy()
@@ -117,34 +187,66 @@ def from_spec(cls, module_spec: ModuleSpec) -> 'ModalitySubmodules':
decoders=decoders,
input_projections=input_projections,
output_projections=output_projections,
+ is_first_stage=is_first_stage,
+ is_last_stage=is_last_stage,
**additional_params,
)
- @abstractmethod
def combine_embeddings(self, embeddings: List[torch.Tensor]) -> torch.Tensor:
- """Combine multiple embeddings from different encoders.
+ """Combine multiple embeddings from different encoders by concatenation.
Args:
embeddings (List[torch.Tensor]):
- List of embeddings to combine
+ List of embeddings to combine. Each is [total_tokens, hidden_dim].
Returns:
torch.Tensor: Combined embedding tensor
"""
- pass
+ if not embeddings:
+ raise ValueError("Cannot combine empty list of embeddings")
- @abstractmethod
- def encode(self, data_batch: Dict) -> List[torch.Tensor]:
+ if len(embeddings) == 1:
+ return embeddings[0]
+
+ combined = torch.cat(embeddings, dim=0)
+ logger.debug(f"Combined embeddings shape after concatenation: {combined.shape}")
+ return combined
+
+ def encode(self, encoders_data_batch: Dict) -> List[torch.Tensor]:
"""Encode data batch into a list of tensors.
Args:
- data_batch (Dict):
- Dictionary containing input data
+ encoders_data_batch (Dict):
+ Dictionary containing encoder-specific inputs.
+ Keys should match encoder names in self.encoders.
Returns:
- List[torch.Tensor]: List of encoded embeddings
+ List[torch.Tensor]: List of encoded embeddings, each [total_tokens, hidden_dim]
"""
- pass
+ if not encoders_data_batch:
+ return []
+
+ embeddings = []
+
+ for name, encoder in self.encoders.items():
+ if name not in encoders_data_batch:
+ raise ValueError(f"No inputs found for encoder '{name}'")
+
+ encoder_inputs = encoders_data_batch[name]
+ encoder_outputs = encoder(**encoder_inputs)
+ logger.debug(f"Encoder '{name}' output shape: {encoder_outputs.shape}")
+
+ if encoder_outputs.ndim == 3:
+ encoder_outputs = encoder_outputs.reshape(-1, encoder_outputs.size(-1))
+ elif encoder_outputs.ndim != 2:
+ raise ValueError(
+ f"Encoder '{name}' output shape {encoder_outputs.shape} is not supported. "
+ f"Expected 3D (b,s,h) or 2D (b*s,h) tensor, got {encoder_outputs.ndim}D"
+ )
+
+ embeddings.append(encoder_outputs)
+
+ return embeddings
@abstractmethod
def decode(self, embeddings: torch.Tensor, data_batch: Dict) -> torch.Tensor:
@@ -161,11 +263,10 @@ def decode(self, embeddings: torch.Tensor, data_batch: Dict) -> torch.Tensor:
"""
pass
- @abstractmethod
def project_embeddings(
self, embeddings: List[torch.Tensor], is_input: bool = True
) -> Optional[torch.Tensor]:
- """Project embeddings into a tensor.
+ """Project embeddings using input or output projections.
Args:
embeddings (List[torch.Tensor]):
@@ -176,18 +277,49 @@ def project_embeddings(
Returns:
Optional[torch.Tensor]: Projected embeddings or None
"""
- pass
+ combined = self.combine_embeddings(embeddings)
- @abstractmethod
- def forward(self, encoder_inputs: Dict[str, Any]) -> Optional[torch.Tensor]:
+ projections = self.input_projections if is_input else self.output_projections
+
+ if projections:
+ projection = projections[0]
+ projected = projection(combined)
+ logger.debug(f"Post-projection embeddings shape: {projected.shape}")
+ return projected
+
+ return combined
+
+ def forward(
+ self,
+ encoder_inputs: Optional[Dict[str, Any]] = None,
+ hidden_states: Optional[torch.Tensor] = None,
+ ) -> Optional[torch.Tensor]:
"""Process data for this modality through encoding and projection.
Args:
encoder_inputs (Dict[str, Any]):
Dictionary containing encoder-specific inputs. Keys should match encoder names.
+ Used when is_first_stage=True.
+ hidden_states (Optional[torch.Tensor]):
+ Hidden states from previous pipeline stage. Used when is_first_stage=False.
Returns:
Optional[torch.Tensor]:
Processed and projected embeddings tensor, or None if no embeddings were produced.
"""
- pass
+ if self.is_first_stage:
+ if encoder_inputs is None:
+ return None
+ embeddings = self.encode(encoder_inputs)
+ if not embeddings:
+ return None
+ combined = self.combine_embeddings(embeddings)
+ else:
+ if hidden_states is None:
+ return None
+ combined = hidden_states
+
+ if self.is_last_stage:
+ return self.project_embeddings([combined], is_input=True)
+
+ return combined
diff --git a/megatron/core/models/mimo/submodules/vision.py b/megatron/core/models/mimo/submodules/vision.py
index 795cb18a119..0bb1a45e013 100644
--- a/megatron/core/models/mimo/submodules/vision.py
+++ b/megatron/core/models/mimo/submodules/vision.py
@@ -1,16 +1,11 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
-import logging
-from typing import Any, Dict, List, Optional
+from typing import Dict, List, Optional
-import torch
import torch.nn as nn
from megatron.core.models.mimo.submodules.base import ModalitySubmodules
-# Initialize logger
-logger = logging.getLogger(__name__)
-
class VisionModalitySubmodules(ModalitySubmodules):
"""Vision modality submodules for encoding, decoding, and projecting image data.
@@ -40,6 +35,7 @@ def __init__(
decoders=decoders,
input_projections=input_projections,
output_projections=output_projections,
+ **kwargs,
)
if self.input_projections:
@@ -52,133 +48,6 @@ def __init__(
len(self.output_projections) <= 1
), "VisionModalitySubmodules currently supports only one output projection"
- def encode(self, encoders_data_batch: Dict) -> List[torch.Tensor]:
- """Encode image data batch into a list of tensors.
-
- Args:
- encoders_data_batch: Dictionary containing encoder-specific inputs.
- Keys should match encoder names in self.encoders.
- Each encoder receives its own specific inputs.
-
- Returns:
- List of encoded image embeddings, one from each encoder.
- Each embedding is a flattened tensor of shape [total_tokens, hidden_dim]
-
- Raises:
- ValueError: If no data is provided for any encoder or if there's a parameter mismatch.
- """
- if not encoders_data_batch:
- return []
-
- embeddings = []
-
- for name, encoder in self.encoders.items():
- if name not in encoders_data_batch:
- raise ValueError(f"No inputs found for encoder '{name}'")
-
- encoder_inputs = encoders_data_batch[name]
-
- # Process inputs through the encoder
- encoder_outputs = encoder(**encoder_inputs)
- logger.debug(f"Encoder '{name}' output shape: {encoder_outputs.shape}")
- if encoder_outputs.ndim == 3:
- # its b,s,h -> we need to flatten it to b*s,h
- encoder_outputs = encoder_outputs.reshape(-1, encoder_outputs.size(-1))
- embeddings.append(encoder_outputs)
- elif encoder_outputs.ndim == 2:
- # its b*s,h -> encoder already returned the flattened output
- embeddings.append(encoder_outputs)
- else:
- raise ValueError(
- f"Encoder '{name}' output shape {encoder_outputs.shape} is not supported"
- "Expected 3D (b,s,h) or 2D (b*s,h) tensor, got {encoder_outputs.ndim}D"
- )
-
- return embeddings
-
- def decode(self, embeddings: torch.Tensor, data_batch: Dict) -> torch.Tensor:
- """Decode embeddings into image tensors.
-
- Args:
- embeddings: Tensor of embeddings to decode.
- data_batch: Dictionary containing additional data for decoding.
-
- Returns:
- Tensor containing generated images.
- """
-
+ def decode(self, embeddings, data_batch: Dict):
+ """Decode embeddings into image tensors."""
raise NotImplementedError("No decoders support yet")
-
- def combine_embeddings(self, embeddings: List[torch.Tensor]) -> torch.Tensor:
- """Combine multiple embeddings from different encoders by concatenation.
-
- This method is used for combining encoder outputs before input projection.
-
- Args:
- embeddings: List of embeddings to combine
-
- Returns:
- Combined embedding tensor
- """
- if not embeddings:
- raise ValueError("Cannot combine empty list of embeddings")
-
- if len(embeddings) == 1:
- return embeddings[0]
-
- # each embedding is [total_tokens, hidden_dim]
- # Make this configurable in the future
- combined = torch.cat(embeddings, dim=0)
- logger.debug(f"Combined embeddings shape after concatenation: {combined.shape}")
- return combined
-
- def project_embeddings(
- self, embeddings: List[torch.Tensor], is_input: bool = True
- ) -> torch.Tensor:
- """Project image embeddings using input or output projections.
-
- Args:
- embeddings: List of image embeddings to project
- is_input: If True, use input projections, otherwise use output projections
-
- Returns:
- Projected image embeddings or None if no embeddings
- """
- if is_input:
- embeddings = self.combine_embeddings(embeddings)
-
- # Get the appropriate projection (input or output)
- projections = self.input_projections if is_input else self.output_projections
-
- # Apply projection if available
- if projections:
- # We've asserted in __init__ that there's only one projection
- projection = projections[0]
- projected = projection(embeddings)
- logger.debug(f"Post-projection embeddings shape: {projected.shape}")
- return projected
-
- return embeddings
-
- def forward(self, encoder_inputs: Dict[str, Any]) -> Optional[torch.Tensor]:
- """Process image data through encoding and projection.
-
- Args:
- encoder_inputs: Dictionary where keys match encoder names in self.encoders
- and values are dictionaries of encoder-specific parameters.
- Example: {"clip": {"pixel_values": images}, "vit": {"images": vit_images}}
-
- Returns:
- Flattened image embeddings with shape [total_embeddings, hidden_dim],
- or None if no valid inputs were provided.
- """
- # Encode the images
- embeddings = self.encode(encoder_inputs)
-
- # If no embeddings were produced, return None
- if not embeddings:
- return None
-
- projected = self.project_embeddings(embeddings, is_input=True)
- logging.debug(f"Projected audio embeddings shape: {projected.shape}")
- return projected # [total_embeddings, hidden_dim]
diff --git a/megatron/core/models/multimodal/llava_model.py b/megatron/core/models/multimodal/llava_model.py
index af0bcf6e9fd..86ce04521a7 100644
--- a/megatron/core/models/multimodal/llava_model.py
+++ b/megatron/core/models/multimodal/llava_model.py
@@ -8,6 +8,7 @@
from megatron.core import tensor_parallel
from megatron.core.config_logger import has_config_logger_enabled, log_config_to_disk
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.inference.contexts import BaseInferenceContext
from megatron.core.models.gpt import GPTModel
from megatron.core.models.mamba import MambaModel
@@ -17,25 +18,26 @@
from megatron.core.packed_seq_params import PackedSeqParams
from megatron.core.process_groups_config import ProcessGroupCollection
from megatron.core.transformer import MegatronModule
+from megatron.core.transformer.attention import SelfAttentionSubmodules
from megatron.core.transformer.spec_utils import ModuleSpec
from megatron.core.transformer.transformer_config import TransformerConfig
-from megatron.core.utils import deprecate_inference_params, log_single_rank
-
-try:
- import transformer_engine # pylint: disable=unused-import
+from megatron.core.transformer.transformer_layer import TransformerLayerSubmodules
+from megatron.core.utils import deprecate_inference_params, is_te_min_version, log_single_rank
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import TEDotProductAttention
- from megatron.core.utils import is_te_min_version
- HAVE_TE = True
try:
import transformer_engine_torch as tex
HAVE_TEX = True
- except:
+ except ImportError:
+ tex = None
HAVE_TEX = False
-except:
- HAVE_TE = False
+else:
+ TEDotProductAttention = None
+ tex = None
+ HAVE_TEX = False
IGNORE_INDEX = -100 # ID for labels that should be ignored.
@@ -113,9 +115,7 @@ def __init__(
language_rotary_base: int = 10000,
language_rope_scaling: bool = False,
language_rope_scaling_factor: float = 8.0,
- hybrid_attention_ratio: float = 1.0,
- hybrid_mlp_ratio: float = 1.0,
- hybrid_override_pattern: str = None,
+ hybrid_layer_pattern: str = None,
fp16_lm_cross_entropy: bool = False,
image_token_index: int = DEFAULT_IMAGE_TOKEN_INDEX,
pixel_shuffle: bool = False,
@@ -158,9 +158,18 @@ def __init__(
self.context_parallel_lm = language_transformer_config.context_parallel_size
if self.sequence_parallel_lm or self.context_parallel_lm > 1:
if not language_model_type.startswith('nemotron5-hybrid'):
- attn_module = language_transformer_layer_spec.submodules.self_attention
+ assert isinstance(
+ language_transformer_layer_spec.submodules, TransformerLayerSubmodules
+ )
+ assert isinstance(
+ language_transformer_layer_spec.submodules.self_attention.submodules,
+ SelfAttentionSubmodules,
+ )
+ attn_submodules = (
+ language_transformer_layer_spec.submodules.self_attention.submodules
+ )
assert (
- attn_module.submodules.core_attention == TEDotProductAttention and HAVE_TE
+ attn_submodules.core_attention == TEDotProductAttention and HAVE_TE
), "Sequence/Context Parallelism is supported only with TE DotProductAttention."
if self.context_parallel_lm > 1:
self.cp_group = self.pg_collection.cp
@@ -195,9 +204,7 @@ def __init__(
parallel_output=parallel_output,
position_embedding_type=language_position_embedding_type,
pre_process=self.pre_process,
- hybrid_attention_ratio=hybrid_attention_ratio,
- hybrid_mlp_ratio=hybrid_mlp_ratio,
- hybrid_override_pattern=hybrid_override_pattern,
+ hybrid_layer_pattern=hybrid_layer_pattern,
post_process=self.post_process,
rotary_percent=language_rotary_percent,
rotary_base=language_rotary_base,
@@ -727,12 +734,13 @@ def _process_embedding_token_parallel(
"1.10.0"
), "Please update Transformer Engine to >= 1.10 to use \
Context Parallel with THD format data"
- cp_size = self.cp_group.size()
- cp_rank = self.cp_group.rank()
+ index = tex.thd_get_partitioned_indices(
+ packed_seq_params.cu_seqlens_q_padded,
+ batch[next(iter(batch))].size(1),
+ self.cp_group.size(),
+ self.cp_group.rank(),
+ )
for key, data in batch.items():
- index = tex.thd_get_partitioned_indices(
- packed_seq_params.cu_seqlens_q_padded, data.size(1), cp_size, cp_rank
- )
batch[key] = data.index_select(1, index)
if self.pre_process:
diff --git a/megatron/core/models/vision/multimodal_projector.py b/megatron/core/models/vision/multimodal_projector.py
index 31661934bb1..cb54891228e 100644
--- a/megatron/core/models/vision/multimodal_projector.py
+++ b/megatron/core/models/vision/multimodal_projector.py
@@ -4,11 +4,12 @@
import torch
from megatron.core.fp8_utils import get_fp8_context
+from megatron.core.process_groups_config import ProcessGroupCollection
from megatron.core.transformer.mlp import MLP, MLPSubmodules
from megatron.core.transformer.module import MegatronModule
-from megatron.core.transformer.spec_utils import build_module
from megatron.core.transformer.transformer_config import TransformerConfig
-from megatron.core.utils import make_viewless_tensor
+from megatron.core.typed_torch import apply_module, not_none
+from megatron.core.utils import get_tensor_model_parallel_group_if_none, make_viewless_tensor
class MultimodalProjector(MegatronModule):
@@ -32,9 +33,12 @@ def __init__(
projector_type: str,
input_size: int,
tp_group: Optional[torch.distributed.ProcessGroup] = None,
+ pg_collection: Optional[ProcessGroupCollection] = None,
):
super().__init__(config=config)
self.projector_type = projector_type
+ tp_group = pg_collection.tp if pg_collection is not None else tp_group
+ self.tp_group = get_tensor_model_parallel_group_if_none(tp_group)
assert submodules is not None, "MLPSubmodules must be provided"
@@ -45,12 +49,11 @@ def __init__(
config=config, submodules=submodules, input_size=input_size, tp_group=tp_group
)
elif self.projector_type == "affine":
- self.encoder = build_module(
- submodules.linear_fc1,
+ self.encoder = submodules.linear_fc1(
input_size,
config.hidden_size,
config=config,
- init_method=config.init_method,
+ init_method=not_none(config.init_method),
gather_output=True,
bias=config.add_bias_linear,
skip_bias_add=True,
@@ -73,7 +76,7 @@ def forward(self, hidden_states):
fp8_context = get_fp8_context(self.config)
with fp8_context:
# Run encoder.
- encoder_output, encoder_output_bias = self.encoder(hidden_states)
+ encoder_output, encoder_output_bias = apply_module(self.encoder)(hidden_states)
if encoder_output_bias is not None:
encoder_output = encoder_output + encoder_output_bias
diff --git a/megatron/core/optimizer/__init__.py b/megatron/core/optimizer/__init__.py
index 11aa6c49585..ef23ea22244 100644
--- a/megatron/core/optimizer/__init__.py
+++ b/megatron/core/optimizer/__init__.py
@@ -33,6 +33,20 @@
USING_PYTORCH_OPTIMIZER = True
+try:
+ from importlib.metadata import PackageNotFoundError
+ from importlib.metadata import version as _pkg_version
+
+ _eo_ver = tuple(int(x) for x in _pkg_version('emerging-optimizers').split('.')[:2])
+except (ImportError, PackageNotFoundError):
+ _eo_ver = (0, 0)
+
+HAVE_EMERGING_OPTIMIZERS = _eo_ver >= (0, 1)
+HAVE_EO_V02 = _eo_ver >= (0, 2)
+
+if HAVE_EO_V02:
+ from emerging_optimizers.scalar_optimizers import Lion
+
from megatron.core import parallel_state
from megatron.core.optimizer.cpu_offloading.hybrid_optimizer import HybridDeviceOptimizer
from megatron.core.optimizer_param_scheduler import (
@@ -105,6 +119,175 @@ def get_standard_config_overrides(config: OptimizerConfig) -> Dict[ParamKey, Par
return config_overrides
+def get_mup_config_overrides(
+ config: OptimizerConfig, mup_width_mult: float, optimizer_type: str = 'adam'
+) -> Dict[ParamKey, ParamGroupOverride]:
+ """Get MuP config overrides for per-layer LR and Adam epsilon scaling.
+
+ In MuP, optimizer learning rates are adjusted by parameter class to ensure
+ stable update scales across model widths and enable hyperparameter transfer.
+
+ MuP optimizer scaling rules (as implemented here):
+ - Adam/AdamW:
+ - hidden (matrix-like) lr = base_lr / width_mult
+ - hidden (matrix-like) eps = base_eps / width_mult
+ - vector-like params keep base lr and eps
+ - SGD:
+ - vector-like lr = base_lr * width_mult
+ - hidden (matrix-like) lr keeps base_lr in the current uniform-width setup
+ - no eps override is applied
+ - Non-Adam optimizers:
+ - hidden (matrix-like) lr = base_lr / width_mult
+ - no eps override is applied.
+ - for Muon optimizers, matrix-like params managed by Muon itself are
+ excluded from these Adam-style MuP overrides.
+
+ With decoupled_lr enabled, embedding/output params continue using decoupled LR
+ and MuP will not override those explicit decoupled values.
+
+ Args:
+ config (OptimizerConfig): optimizer configuration object.
+ mup_width_mult (float): Width multiplier (hidden_size / base_hidden_size).
+ optimizer_type (str): Optimizer type string from config.optimizer.
+
+ Returns:
+ Dict[ParamKey, ParamGroupOverride]: MuP optimizer overrides.
+ """
+ optimizer_type_lower = optimizer_type.lower()
+ is_sgd_optimizer = optimizer_type_lower == 'sgd'
+ is_adam_optimizer = 'adam' in optimizer_type_lower
+ is_muon_optimizer = 'muon' in optimizer_type_lower
+
+ decoupled_lr_enabled = config.decoupled_lr is not None
+ if decoupled_lr_enabled:
+ message = (
+ "Both decoupled_lr and MuP LR scaling are enabled. decoupled_lr sets an "
+ "absolute LR for embedding+output params, and MuP LR scaling will not "
+ "override those parameters."
+ )
+ if is_adam_optimizer:
+ message += " MuP Adam epsilon scaling remains applied to hidden matrix-like parameters."
+ log_single_rank(logger, logging.WARNING, message)
+
+ if is_muon_optimizer:
+ muon_scale_mode = getattr(config, 'muon_scale_mode', 'spectral')
+ if muon_scale_mode == 'spectral':
+ log_single_rank(
+ logger,
+ logging.WARNING,
+ "Both MuP and muon_scale_mode=spectral are enabled. "
+ "Muon-managed matrix parameters will continue using spectral Muon scaling. "
+ "Set --muon-scale-mode unit_rms_norm to use unit_rms_norm scaling for "
+ "Muon-managed matrices with MuP.",
+ )
+
+ if mup_width_mult == 1.0:
+ # No scaling needed when width_mult is 1
+ return {}
+
+ hidden_lr_mult = 1.0 / mup_width_mult
+ base_lr = config.lr
+ base_min_lr = config.min_lr
+
+ # Hidden matrix-like layers get scaled LR/eps; vector-like params keep base values.
+ # Prefer the explicit parameter attribute set by LanguageModule. Fall back to
+ # a conservative name check for older or non-language modules.
+ def is_embedding_parameter(param: torch.nn.Parameter, param_name: str) -> bool:
+ if getattr(param, 'shared_embedding', False):
+ return True
+ if hasattr(param, 'is_embedding_parameter'):
+ return bool(param.is_embedding_parameter)
+ return 'embedding' in param_name.lower()
+
+ def is_vector_like_parameter(param: torch.nn.Parameter, param_name: str) -> bool:
+ if is_embedding_parameter(param, param_name):
+ return True
+ if param.dim() <= 1:
+ return True
+ return False
+
+ def is_muon_managed_matrix_parameter(param: torch.nn.Parameter, _: str) -> bool:
+ if not is_muon_optimizer:
+ return False
+ return param.dim() == 2 and not getattr(param, 'is_embedding_or_output_parameter', False)
+
+ def should_scale_lr_with_mup(param: torch.nn.Parameter, param_name: str) -> bool:
+ if decoupled_lr_enabled and getattr(param, 'is_embedding_or_output_parameter', False):
+ return False
+ if is_muon_managed_matrix_parameter(param, param_name):
+ return False
+ return not is_vector_like_parameter(param, param_name)
+
+ def should_scale_vector_like_lr_with_mup(param: torch.nn.Parameter, param_name: str) -> bool:
+ if decoupled_lr_enabled and getattr(param, 'is_embedding_or_output_parameter', False):
+ return False
+ return is_vector_like_parameter(param, param_name)
+
+ def should_scale_eps_with_mup(param: torch.nn.Parameter, param_name: str) -> bool:
+ if is_vector_like_parameter(param, param_name):
+ return False
+ if is_muon_managed_matrix_parameter(param, param_name):
+ return False
+ # MuP Appendix B.3: eps scales with fan_in when non-negligible.
+ # This implementation follows the common denominator form: sqrt(v) + eps.
+ return True
+
+ mup_overrides: Dict[ParamKey, ParamGroupOverride] = {}
+
+ if is_sgd_optimizer:
+ vector_like_lr_mult = mup_width_mult
+ vector_like_lr_override: ParamGroupOverride = {}
+ if base_lr is not None:
+ vector_like_lr_override["max_lr"] = base_lr * vector_like_lr_mult
+ if base_min_lr is not None:
+ vector_like_lr_override["min_lr"] = base_min_lr * vector_like_lr_mult
+
+ if vector_like_lr_override:
+ vector_like_predicate = ParamWithNamePredicate(
+ name="mup_sgd_vector_like_excluding_embedding_output",
+ fn=should_scale_vector_like_lr_with_mup,
+ )
+ mup_overrides[ParamKey(with_name_predicate=vector_like_predicate)] = (
+ vector_like_lr_override
+ )
+
+ return mup_overrides
+
+ lr_override: ParamGroupOverride = {}
+ if base_lr is not None:
+ lr_override["max_lr"] = base_lr * hidden_lr_mult
+ if base_min_lr is not None:
+ lr_override["min_lr"] = base_min_lr * hidden_lr_mult
+
+ eps_override: ParamGroupOverride = {}
+ if is_adam_optimizer and config.adam_eps is not None:
+ eps_override["eps"] = config.adam_eps * hidden_lr_mult
+
+ if decoupled_lr_enabled:
+ if lr_override:
+ hidden_predicate = ParamWithNamePredicate(
+ name="mup_hidden_only_excluding_embedding_output", fn=should_scale_lr_with_mup
+ )
+ mup_overrides[ParamKey(with_name_predicate=hidden_predicate)] = lr_override
+
+ if eps_override:
+ hidden_output_predicate = ParamWithNamePredicate(
+ name="mup_hidden_only_for_adam_eps", fn=should_scale_eps_with_mup
+ )
+ mup_overrides[ParamKey(with_name_predicate=hidden_output_predicate)] = eps_override
+ else:
+ combined_override: ParamGroupOverride = {}
+ combined_override.update(lr_override)
+ combined_override.update(eps_override)
+ if combined_override:
+ hidden_output_predicate = ParamWithNamePredicate(
+ name="mup_hidden_and_output", fn=should_scale_eps_with_mup
+ )
+ mup_overrides[ParamKey(with_name_predicate=hidden_output_predicate)] = combined_override
+
+ return mup_overrides
+
+
def _get_param_groups(
model_chunks: List[MegatronModule],
config: OptimizerConfig,
@@ -398,6 +581,25 @@ def init_state_fn(opt, config=None):
else:
opt.initialize_state(p)
+ elif config.optimizer == 'lion':
+ if not HAVE_EO_V02:
+ raise ImportError(
+ "Lion optimizer requires emerging_optimizers >= 0.2. "
+ "Please install or upgrade it to use --optimizer lion."
+ )
+ optimizer = Lion( # pylint: disable=possibly-used-before-assignment
+ param_groups,
+ lr=config.lr,
+ betas=(config.lion_beta1, config.lion_beta2),
+ weight_decay=config.weight_decay,
+ )
+
+ def init_state_fn(opt, config=None):
+ for group in opt.param_groups:
+ for p in group['params']:
+ if len(opt.state[p]) == 0:
+ opt.state[p]['exp_avg'] = torch.zeros_like(p.data)
+
elif config.optimizer == 'sgd':
optimizer = SGD(
param_groups,
diff --git a/megatron/core/optimizer/clip_grads.py b/megatron/core/optimizer/clip_grads.py
index cb2f23a685f..1069d5ed3f2 100644
--- a/megatron/core/optimizer/clip_grads.py
+++ b/megatron/core/optimizer/clip_grads.py
@@ -204,7 +204,7 @@ def count_zeros_fp32(
# - grad should not be none
# - parameter should not be shared
# - should not be a replica due to tensor model parallelism
- total_num_zeros = torch.zeros(1, dtype=torch.float, device='cuda')
+ total_num_zeros = torch.zeros(1, dtype=torch.int64, device='cuda')
data_parallel_group = None
use_megatron_fsdp = False
for param in parameters:
diff --git a/megatron/core/optimizer/distrib_optimizer.py b/megatron/core/optimizer/distrib_optimizer.py
index 4192b0bb73c..eeda383a75d 100644
--- a/megatron/core/optimizer/distrib_optimizer.py
+++ b/megatron/core/optimizer/distrib_optimizer.py
@@ -896,6 +896,8 @@ def _get_main_param_and_optimizer_states(self, model_param):
sharded_model_param = self.optimizer.param_groups[group_index]["params"][group_order]
tensors = {}
for k in self.optimizer.state[sharded_model_param]:
+ if not isinstance(self.optimizer.state[sharded_model_param][k], torch.Tensor):
+ continue
if isinstance(self.optimizer, HybridDeviceOptimizer):
tensors[k] = self.optimizer.state[sharded_model_param][k]
continue
@@ -905,7 +907,10 @@ def _get_main_param_and_optimizer_states(self, model_param):
else:
main_param = self.optimizer.param_groups[group_index]["params"][group_order]
optim_state = self.optimizer.state[main_param]
- tensors = {"param": main_param, **optim_state}
+ tensors = {"param": main_param}
+ for k, v in optim_state.items():
+ if isinstance(v, torch.Tensor):
+ tensors[k] = v
return tensors
def _set_main_param_and_optimizer_states(self, model_param, tensors):
@@ -922,6 +927,8 @@ def _set_main_param_and_optimizer_states(self, model_param, tensors):
if self.config.use_precision_aware_optimizer_no_fp8_or_ds_fp8:
sharded_model_param = self.optimizer.param_groups[group_index]["params"][group_order]
for k, v in tensors.items():
+ if not isinstance(v, torch.Tensor):
+ continue
if isinstance(self.optimizer, HybridDeviceOptimizer):
if k == "param":
k = "master_param"
@@ -935,8 +942,13 @@ def _set_main_param_and_optimizer_states(self, model_param, tensors):
else:
main_param = self.optimizer.param_groups[group_index]["params"][group_order]
optim_state = self.optimizer.state[main_param]
- dst_tensors = {"param": main_param, **optim_state}
+ dst_tensors = {"param": main_param}
+ for k, v in optim_state.items():
+ if isinstance(v, torch.Tensor):
+ dst_tensors[k] = v
for key in dst_tensors:
+ if not isinstance(tensors[key], torch.Tensor):
+ continue
dst_tensors[key].copy_(tensors[key])
def get_parameter_state_dp_reshardable(self):
@@ -2527,7 +2539,7 @@ def _build_model_param_to_state_dict_param_map(self, state_dict):
for name, model_param in model_chunk.named_parameters():
while name.startswith("module."):
name = name[len("module.") :]
- matched_keys = [k for k in names_in_state_dict if name in k]
+ matched_keys = [k for k in names_in_state_dict if k.endswith(name)]
assert (
len(matched_keys) == 1
), f"Parameter {name} has {len(matched_keys)} matches in state dict"
diff --git a/megatron/core/optimizer/layer_wise_optimizer.py b/megatron/core/optimizer/layer_wise_optimizer.py
index de4396a5b4f..a9fdc7ba72f 100644
--- a/megatron/core/optimizer/layer_wise_optimizer.py
+++ b/megatron/core/optimizer/layer_wise_optimizer.py
@@ -45,6 +45,8 @@ def __init__(
config: OptimizerConfig,
pg_collection: Optional[ProcessGroupCollection] = None,
init_state_fn_list: Optional[List[Callable]] = None,
+ model_chunks: Optional[List] = None,
+ async_allgather: bool = False,
) -> None:
"""
Initialize LayerWiseDistributedOptimizer.
@@ -54,10 +56,21 @@ def __init__(
config: OptimizerConfig.
pg_collection: ProcessGroupCollection.
init_state_fn_list: List of init state functions.
+ model_chunks: DDP-wrapped model chunks (needed for async_allgather).
+ async_allgather: If True, defer param all-gather to forward pre-hooks.
"""
self.pg_collection = pg_collection
self.shard_params(optimizers)
+
+ # Set up async all-gather using DDP bucket infrastructure.
+ self.async_allgather = async_allgather
+ if self.async_allgather:
+ assert (
+ model_chunks is not None
+ ), "model_chunks must be provided if async_allgather is True"
+ self.set_bucket_layerwise_params_list(model_chunks)
+
if init_state_fn_list:
assert len(init_state_fn_list) == len(
optimizers
@@ -143,34 +156,79 @@ def shard_params(self, optimizers):
if expt_dp_size == 1 or len(self.expt_dp_params_list[0]) == 0:
self.expt_dp_params_list = None
+ def set_bucket_layerwise_params_list(self, model_chunks):
+ """Map sharded params to DDP buckets for async all-gather.
+
+ For each bucket in each model chunk's bucket groups, build per-rank param lists
+ by cross-referencing the layer-wise sharded param lists with the bucket's params.
+
+ Args:
+ model_chunks: DDP-wrapped model chunks with bucket_groups.
+ """
+ for model_chunk in model_chunks:
+ for group in model_chunk.bucket_groups:
+ for bucket in group.buckets:
+ bucket_params_list = [[] for _ in range(get_pg_size(self.pg_collection.dp_cp))]
+ for bucket_list, full_params_list in zip(
+ bucket_params_list, self.dp_cp_params_list
+ ):
+ for param in full_params_list:
+ if param in bucket.params:
+ bucket_list.append(param)
+ bucket.set_layerwise_params_list(bucket_params_list)
+ # Do the same for expert parallel bucket groups.
+ if self.expt_dp_params_list is not None:
+ for group in model_chunk.expert_parallel_bucket_groups:
+ for bucket in group.buckets:
+ bucket_params_list = [
+ [] for _ in range(get_pg_size(self.pg_collection.expt_dp))
+ ]
+ for bucket_list, full_params_list in zip(
+ bucket_params_list, self.expt_dp_params_list
+ ):
+ for param in full_params_list:
+ if param in bucket.params:
+ bucket_list.append(param)
+ bucket.set_layerwise_params_list(bucket_params_list)
+
@torch.no_grad()
def allgather_params(self) -> None:
"""All-gather updated params from all ranks."""
- # helper function to flatten local params, allgather, unflatten and copy to model params
+ # helper function to flatten local params, all-gather,
+ # unflatten and copy to model params
def _allgather_helper(params_list, group):
- # flatten this rank's params and create empty tensor output list
device = params_list[0][0].device
dtype = params_list[0][0].dtype
rank = get_pg_rank(group)
- # for rank without params create empty tensor and participate in allgather
+ dp_size = get_pg_size(group)
+ # Flatten this rank's params.
src = (
_flatten_dense_tensors(params_list[rank])
if len(params_list[rank]) > 0
else torch.empty(0, device=device, dtype=dtype)
)
- output_list = [
- torch.empty(sum([p.numel() for p in params]), device=device, dtype=dtype)
- for params in params_list
- ]
- # single all_gather_v to collect all updated params
- torch.distributed.all_gather(output_list, src, group=group)
- # unflatten and copy gathered params for each rank i
- for idx, (flat_params, params) in enumerate(zip(output_list, params_list)):
- # skip local params and empty tensors
+ flat_sizes = [sum(p.numel() for p in params) for params in params_list]
+ if max(flat_sizes) == 0:
+ return
+
+ # Allocate per-rank receive buffers with actual sizes (no padding).
+ # PyTorch's NCCL backend handles uneven sizes in all_gather via
+ # grouped send/recv internally. Reuse src for local rank's slot.
+ gather_list = []
+ for i in range(dp_size):
+ if i == rank:
+ gather_list.append(src)
+ else:
+ gather_list.append(torch.empty(flat_sizes[i], device=device, dtype=dtype))
+
+ torch.distributed.all_gather(gather_list, src, group=group)
+
+ # Unflatten and copy gathered params for each rank.
+ for idx, params in enumerate(params_list):
if len(params) == 0 or idx == rank:
continue
- updated_params = _unflatten_dense_tensors(flat_params, params)
+ updated_params = _unflatten_dense_tensors(gather_list[idx], params)
for updated_p, model_p in zip(updated_params, params):
model_p.data.copy_(updated_p)
@@ -223,8 +281,10 @@ def step(self): # type: ignore[no-untyped-def]
"""step function for layer-wise optimizer."""
update_successful, grad_norm, num_zeros_in_grad = super().step()
- # All gather updated params.
- self.allgather_params()
+ # All gather updated params. If async_allgather is True, the allgather
+ # is deferred to the forward pre-hooks via DDP bucket infrastructure.
+ if not self.async_allgather:
+ self.allgather_params()
return update_successful, grad_norm, num_zeros_in_grad
diff --git a/megatron/core/optimizer/muon.py b/megatron/core/optimizer/muon.py
index 57eb1e94478..046be78ad10 100644
--- a/megatron/core/optimizer/muon.py
+++ b/megatron/core/optimizer/muon.py
@@ -3,7 +3,7 @@
"""Megatron muon optimizer wrapper to handle tensor-parallel."""
import logging
-from typing import Any, Callable, Dict, List, Literal, Optional
+from typing import Any, Callable, Dict, List, Literal, Optional, get_args
import torch
from torch.optim.optimizer import ParamsT
@@ -13,7 +13,7 @@
from megatron.core.transformer.module import MegatronModule
from megatron.core.utils import get_pg_size, log_single_rank
-from . import _get_param_groups, get_megatron_optimizer
+from . import HAVE_EMERGING_OPTIMIZERS, HAVE_EO_V02, _get_param_groups, get_megatron_optimizer
from .layer_wise_optimizer import LayerWiseDistributedOptimizer
from .optimizer import (
ChainedOptimizer,
@@ -23,22 +23,44 @@
)
from .optimizer_config import OptimizerConfig, ParamKey
-try:
+if HAVE_EMERGING_OPTIMIZERS:
from emerging_optimizers.orthogonalized_optimizers import (
OrthogonalizedOptimizer,
get_muon_scale_factor,
)
from emerging_optimizers.orthogonalized_optimizers.muon_utils import newton_schulz_tp
-
- HAVE_EMERGING_OPTIMIZERS = True
-except ImportError:
- HAVE_EMERGING_OPTIMIZERS = False
+else:
OrthogonalizedOptimizer = object
+if HAVE_EO_V02:
+ from emerging_optimizers.orthogonalized_optimizers.muon_utils import NSCoeffT
+
logger = logging.getLogger(__name__)
+def get_supported_coefficient_types() -> tuple[str, ...]:
+ """Return the coefficient types supported by the installed emerging_optimizers.
+
+ Reads the members of the ``NSCoeffT`` Literal type so that new types
+ added upstream are automatically available without code changes here.
+ """
+ assert (
+ HAVE_EO_V02
+ ), "emerging_optimizers >= 0.2 is required for NSCoeffT. Please install or upgrade it."
+ return get_args(NSCoeffT) # pylint: disable=possibly-used-before-assignment
+
+
+def validate_coefficient_type(coefficient_type: str) -> None:
+ """Raise ``ValueError`` if *coefficient_type* is not supported."""
+ supported = get_supported_coefficient_types() if HAVE_EO_V02 else ("quintic",)
+ if coefficient_type not in supported:
+ raise ValueError(
+ f"Unsupported muon coefficient type '{coefficient_type}'. "
+ f"Supported types: {supported}"
+ )
+
+
class TensorParallelMuon(OrthogonalizedOptimizer):
"""Tensor Parallel Muon optimizer."""
@@ -63,6 +85,7 @@ def __init__(
) -> None:
if num_ns_steps < 1:
raise ValueError(f"num_ns_steps must be at least 1, got {num_ns_steps}")
+ validate_coefficient_type(coefficient_type)
def scaled_orthogonalize_fn(
grad: torch.Tensor,
@@ -78,14 +101,15 @@ def scaled_orthogonalize_fn(
size = [grad.size(-2), grad.size(-1)]
if partition_dim is not None:
size[partition_dim] *= get_pg_size(tp_group)
- orth_grad = newton_schulz_tp(
- grad,
- steps=num_ns_steps,
- coefficient_type=coefficient_type,
- tp_group=tp_group,
- partition_dim=partition_dim,
- mode="duplicated" if mode == "blockwise" else mode,
+ mode_value = "duplicated" if mode == "blockwise" else mode
+ mode_kwarg = {"tp_mode": mode_value} if HAVE_EO_V02 else {"mode": mode_value}
+ ns_kwargs = dict(
+ steps=num_ns_steps, tp_group=tp_group, partition_dim=partition_dim, **mode_kwarg
)
+ ns_kwargs["coefficient_type"] = coefficient_type
+ # pylint: disable-next=possibly-used-before-assignment
+ orth_grad = newton_schulz_tp(grad, **ns_kwargs)
+ # pylint: disable-next=possibly-used-before-assignment
scale_factor = get_muon_scale_factor(size[0], size[1], mode=scale_mode)
return orth_grad * scale_factor * extra_scale_factor
@@ -96,11 +120,14 @@ def scaled_orthogonalize_fn(
self.qkv_split_shapes = qkv_split_shapes
weight_decay_method = "decoupled" if use_decoupled_weight_decay else "l2"
+ nesterov_kwarg = (
+ {"nesterov": use_nesterov} if HAVE_EO_V02 else {"use_nesterov": use_nesterov}
+ )
super().__init__(
params,
lr,
momentum_beta,
- use_nesterov=use_nesterov,
+ **nesterov_kwarg,
weight_decay=weight_decay,
weight_decay_method=weight_decay_method,
fp32_matmul_prec=fp32_matmul_prec,
@@ -181,11 +208,18 @@ def get_megatron_muon_optimizer(
layer_wise_distributed_optimizer (bool): if true, use layer-wise distributed optimizer.
Defaults to False.
"""
- # Muon currently use adam config. setting str here to call regular get for adam creation
- # side effect is muon optimizer will have wrong name, i.e. config.optimizer == 'adam'
- config.optimizer = 'adam'
-
- assert HAVE_EMERGING_OPTIMIZERS, "Emerging Optimizers is not installed."
+ # TODO: Mutating config.optimizer is a side effect; clean up after
+ # https://github.com/NVIDIA/Megatron-LM/pull/3638 lands.
+ # Set the nonlinear optimizer for muon (used for embeddings, biases, norms).
+ config.optimizer = config.muon_scalar_optimizer
+
+ if config.muon_scalar_optimizer == 'lion':
+ assert HAVE_EO_V02, (
+ "Lion optimizer requires emerging_optimizers >= 0.2. "
+ "Please upgrade to use --muon-scalar-optimizer lion."
+ )
+ else:
+ assert HAVE_EMERGING_OPTIMIZERS, "Emerging Optimizers is not installed."
# Dist-opt is not supported due to strong coupling with how DDP init grad buffer
# In theory we can change DDP to enable use muon and dist-opt-adam together
@@ -221,6 +255,16 @@ def adam_init_state_fn(opt, config=None):
else:
opt.initialize_state(p)
+ def lion_init_state_fn(opt, config=None):
+ for group in opt.param_groups:
+ for p in group['params']:
+ if len(opt.state[p]) == 0:
+ opt.state[p]['exp_avg'] = torch.zeros_like(p.data)
+
+ nonlinear_init_state_fn = (
+ lion_init_state_fn if config.muon_scalar_optimizer == 'lion' else adam_init_state_fn
+ )
+
optimizers = []
# record list of non/linear params
linear_params = []
@@ -263,6 +307,7 @@ def adam_init_state_fn(opt, config=None):
"use_nesterov": config.muon_use_nesterov,
"weight_decay": config.weight_decay,
"fp32_matmul_prec": config.muon_fp32_matmul_prec,
+ "coefficient_type": config.muon_coefficient_type,
"num_ns_steps": config.muon_num_ns_steps,
"scale_mode": config.muon_scale_mode,
"split_qkv": config.muon_split_qkv,
@@ -337,7 +382,9 @@ def adam_init_state_fn(opt, config=None):
param.requires_grad = True
# chain everything together
- init_fns = [muon_init_state_fn] + len(chained_adam.chained_optimizers) * [adam_init_state_fn]
+ init_fns = [muon_init_state_fn] + len(chained_adam.chained_optimizers) * [
+ nonlinear_init_state_fn
+ ]
optimizers += chained_adam.chained_optimizers
if layer_wise_distributed_optimizer:
@@ -345,6 +392,11 @@ def adam_init_state_fn(opt, config=None):
if reset_config_bf16:
config.bf16 = True
return LayerWiseDistributedOptimizer(
- optimizers, config, pg_collection, init_state_fn_list=init_fns
+ optimizers,
+ config,
+ pg_collection,
+ init_state_fn_list=init_fns,
+ model_chunks=model_chunks,
+ async_allgather=config.overlap_param_gather,
)
return ChainedOptimizer(optimizers)
diff --git a/megatron/core/optimizer/optimizer.py b/megatron/core/optimizer/optimizer.py
index 2c33d7e701d..df8ec8ef613 100644
--- a/megatron/core/optimizer/optimizer.py
+++ b/megatron/core/optimizer/optimizer.py
@@ -12,6 +12,7 @@
from typing import Any, Callable, Dict, List, Optional, Tuple, Union
import torch
+from typing_extensions import override
try:
from transformer_engine.pytorch.optimizers import multi_tensor_applier, multi_tensor_scale
@@ -1122,6 +1123,14 @@ def param_groups(self) -> List[dict]:
param_groups += optimizer.param_groups
return param_groups
+ @override
+ def get_parameters(self) -> List[torch.nn.Parameter]:
+ """Get list of parameters wrapped in all chained optimizers."""
+ params = []
+ for optimizer in self.chained_optimizers:
+ params.extend(optimizer.get_parameters())
+ return params
+
@property
def state(self) -> ProxyDict:
"""
diff --git a/megatron/core/optimizer/optimizer_config.py b/megatron/core/optimizer/optimizer_config.py
index 2d3e3ca08e0..9e6375b978c 100644
--- a/megatron/core/optimizer/optimizer_config.py
+++ b/megatron/core/optimizer/optimizer_config.py
@@ -272,6 +272,10 @@ class OptimizerConfig:
muon_fp32_matmul_prec: str = "medium"
"""The precision to use for the fp32 matmul. Defaults to "medium"."""
+ muon_coefficient_type: str = "quintic"
+ """Newton-Schulz coefficient type for the Muon optimizer. Valid types are discovered
+ dynamically from the installed ``emerging_optimizers`` package. Defaults to "quintic"."""
+
muon_num_ns_steps: int = 5
"""The number of iteration steps to use in the Newton-Schulz iteration."""
@@ -281,6 +285,18 @@ class OptimizerConfig:
muon_extra_scale_factor: float = 1.0
"""Additional scale factor for the muon update."""
+ muon_scalar_optimizer: str = 'adam'
+ """Optimizer for nonlinear parameters (embeddings, biases, norms) when using muon.
+ One of 'adam' or 'lion'. Defaults to 'adam'."""
+
+ # Lion.
+ lion_beta1: float = 0.95
+ """First beta coefficient for Lion optimizer (used in sign update). Defaults to 0.95."""
+
+ lion_beta2: float = 0.98
+ """Second beta coefficient for Lion optimizer (used in momentum EMA update).
+ Defaults to 0.98."""
+
#######################
# Distributed optimizer
#######################
diff --git a/megatron/core/optimizer/qk_clip.py b/megatron/core/optimizer/qk_clip.py
index 72127f94712..26b5787cd50 100644
--- a/megatron/core/optimizer/qk_clip.py
+++ b/megatron/core/optimizer/qk_clip.py
@@ -22,6 +22,11 @@ def clip_qk(model, log_max_only=False) -> float:
for model_chunk in model:
for transformer_layer in model_chunk.module.module.decoder.layers:
if hasattr(transformer_layer.self_attention, 'clip_qk'):
+ if (
+ transformer_layer.self_attention.core_attention.current_max_attn_logits
+ is None
+ ):
+ continue
torch.distributed.all_reduce(
transformer_layer.self_attention.core_attention.current_max_attn_logits,
op=torch.distributed.ReduceOp.MAX,
diff --git a/megatron/core/optimizer_param_scheduler.py b/megatron/core/optimizer_param_scheduler.py
index 7ff6fee35a7..e01a708ce79 100644
--- a/megatron/core/optimizer_param_scheduler.py
+++ b/megatron/core/optimizer_param_scheduler.py
@@ -34,6 +34,26 @@ class ParamGroupOverride(TypedDict):
wd_mult: float
+def get_canonical_lr_for_logging(param_groups: list[dict]) -> float | None:
+ """Return the lr of the first ``default_config=True`` param group.
+
+ All ``default_config`` groups share the same LR schedule, so the first one
+ is representative. This includes empty rank-alignment stub groups, which
+ the scheduler still writes a valid lr onto.
+
+ Args:
+ param_groups (list[dict]): parameter groups from the optimizer.
+
+ Returns:
+ float | None: the canonical learning rate, or None if no
+ ``default_config=True`` group is found.
+ """
+ for param_group in param_groups:
+ if param_group.get('default_config', False):
+ return param_group.get('lr')
+ return None
+
+
def param_group_override_to_tuple(
param_group_override: ParamGroupOverride | None,
) -> tuple[tuple[str, Any], ...] | None:
@@ -265,6 +285,10 @@ def step(self, increment: int) -> None:
increment (int): number of steps to increment
"""
self.num_steps += increment
+ # Do not skip empty param groups: get_canonical_lr_for_logging reads lr
+ # from default_config groups regardless of whether they hold parameters.
+ # This is important for logging under model parallelism that may leave
+ # some ranks with empty default_config parameter groups.
for param_group in self.optimizer.param_groups:
param_group['lr'] = self.get_lr(param_group)
param_group['weight_decay'] = self.get_wd(param_group) * param_group.get('wd_mult', 1.0)
diff --git a/megatron/core/package_info.py b/megatron/core/package_info.py
index 8a8bafb414a..87a02539539 100644
--- a/megatron/core/package_info.py
+++ b/megatron/core/package_info.py
@@ -2,7 +2,7 @@
MAJOR = 0
-MINOR = 16
+MINOR = 17
PATCH = 0
PRE_RELEASE = 'rc0'
diff --git a/megatron/core/packed_seq_params.py b/megatron/core/packed_seq_params.py
index 08ebdac67d8..322f12a4122 100644
--- a/megatron/core/packed_seq_params.py
+++ b/megatron/core/packed_seq_params.py
@@ -1,6 +1,7 @@
# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
from dataclasses import dataclass
+import torch
import torch.distributed as dist
from torch import Tensor
@@ -21,3 +22,45 @@ class PackedSeqParams:
max_seqlen_kv: int = None
local_cp_size: int = None
cp_group: dist.ProcessGroup = None
+ total_tokens: int = None
+ seq_idx: Tensor = None
+
+ def __post_init__(self):
+ """Pre-compute seq_idx for Mamba mixer CUDA graph compatibility.
+
+ If total_tokens is 16 (for example), this method takes packed_seq_params.cu_seqlens_q_padded
+ (or cu_seqlens_q) which is of the form [0, 5, 7, 11] and returns a tensor of the form
+ [0, 0, 0, 0, 0, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3],
+ which is [0]*(5-0) + [1]*(7-5) + [2]*(11-7) + [3]*(16-11)
+ In the above example, there are three sequences in the pack.
+ In general, the output has an additional sequence index (e.g. 0, 1, 2, 3) so that any tokens
+ beyond the last padded input sequence are accounted for as an extra sequence. However, If
+ cu_seqlens_q_padded[-1] == max_seqlen then this additional sequence index will not be
+ included.
+ """
+ cu_seqlens = (
+ self.cu_seqlens_q_padded if self.cu_seqlens_q_padded is not None else self.cu_seqlens_q
+ )
+ if isinstance(cu_seqlens, Tensor) and self.total_tokens is not None:
+ total_tokens_tensor = torch.tensor(
+ [self.total_tokens], dtype=cu_seqlens.dtype, device=cu_seqlens.device
+ )
+ # Example: [0, 5, 7, 11] -> [0, 5, 7, 11, 16]
+ cu_seqlens_with_max = torch.cat([cu_seqlens, total_tokens_tensor])
+ # Example: [0, 5, 7, 11, 16] -> [5, 2, 4, 5]
+ seq_lengths = cu_seqlens_with_max[1:] - cu_seqlens_with_max[:-1]
+ # Clamp to non-negative: cu_seqlens_q_padded may not be strictly
+ # monotonic when context parallelism slices sequences across ranks,
+ # or when padded cumulative lengths exceed total_tokens (e.g. the
+ # appended total_tokens sentinel is smaller than cu_seqlens[-1]
+ # due to padding). In either case the diff can go negative, which
+ # causes torch.repeat_interleave to fail.
+ seq_lengths = seq_lengths.clamp(min=0)
+ # Example: [5, 2, 4, 5] -> [0, 0, 0, 0, 0, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3]
+ self.seq_idx = (
+ torch.repeat_interleave(
+ torch.arange(seq_lengths.numel(), device=cu_seqlens.device), seq_lengths
+ )
+ .to(torch.int32)
+ .unsqueeze(0) # Add a batch dimension
+ )
diff --git a/megatron/core/parallel_state.py b/megatron/core/parallel_state.py
index 7bb96407838..1435e207b18 100644
--- a/megatron/core/parallel_state.py
+++ b/megatron/core/parallel_state.py
@@ -12,7 +12,9 @@
import numpy as np
import torch
-from .utils import GlobalMemoryBuffer, GlobalSymmetricMemoryBuffer, is_torch_min_version
+from megatron.core.inference.symmetric_memory import SymmetricMemoryManager
+
+from .utils import GlobalMemoryBuffer, is_torch_min_version
logger = logging.getLogger(__name__)
@@ -23,8 +25,6 @@
except ImportError:
HAVE_EINOPS = False
-logger = logging.getLogger(__name__)
-
# Intra-layer model parallel group that the current rank belongs to.
_TENSOR_MODEL_PARALLEL_GROUP = None
# Inter-layer model parallel group that the current rank belongs to.
@@ -140,8 +140,6 @@
# Memory buffers to avoid dynamic memory allocation
_GLOBAL_MEMORY_BUFFER = None
-# Global symmetric memory buffer for inference
-_GLOBAL_SYMMETRIC_MEMORY_BUFFER = None
# List of all process groups
# Used for updating the timeout for all process groups
@@ -568,6 +566,8 @@ def initialize_model_parallel(
high_priority_stream_groups: Optional[List[str]] = None,
sharp_enabled_group: Optional[str] = None,
create_all_gather_group: Optional[bool] = False,
+ rank_offset: int = 0,
+ local_world_size: Optional[int] = None,
) -> None:
"""Initialize model data parallel groups.
@@ -735,7 +735,9 @@ def initialize_model_parallel(
# Get world size and rank. Ensure some consistencies.
assert torch.distributed.is_initialized()
- world_size: int = torch.distributed.get_world_size()
+ world_size: int = (
+ local_world_size if local_world_size is not None else torch.distributed.get_world_size()
+ )
model_size = tensor_model_parallel_size * pipeline_model_parallel_size * context_parallel_size
@@ -781,7 +783,7 @@ def initialize_model_parallel(
pp=pipeline_model_parallel_size,
cp=context_parallel_size,
order=order,
- rank_offset=0,
+ rank_offset=rank_offset,
)
# Build expert rank generator
@@ -804,7 +806,7 @@ def initialize_model_parallel(
pp=pipeline_model_parallel_size,
cp=1,
order=order,
- rank_offset=0,
+ rank_offset=rank_offset,
)
assert (
@@ -1361,16 +1363,6 @@ def is_initialized():
return _DATA_PARALLEL_GROUP is not None
-def is_unitialized() -> bool:
- """Check if parallel state has been initialized
-
- Deprecated. Use is_initialized instead.
-
- """
- warnings.warn("is_unitialized is deprecated, use is_initialized instead", DeprecationWarning)
- return not is_initialized()
-
-
def model_parallel_is_initialized():
"""Check if model- and data-parallel groups are initialized."""
if (
@@ -1958,16 +1950,6 @@ def get_expert_data_parallel_group(check_initialized=True, partial_expert_data_p
return _EXPERT_DATA_PARALLEL_GROUP
-def get_data_modulo_expert_parallel_group(partial_expert_data_parallel=False):
- """[Deprecated] Get expert data parallel group."""
- warnings.warn(
- "get_data_modulo_expert_parallel_group is deprecated, please use "
- "get_expert_data_parallel_group instead.",
- DeprecationWarning,
- )
- return get_expert_data_parallel_group(partial_expert_data_parallel=partial_expert_data_parallel)
-
-
def get_expert_data_parallel_group_gloo(partial_expert_data_parallel=False):
"""Get expert data parallel group-gloo."""
if partial_expert_data_parallel:
@@ -2034,43 +2016,18 @@ def _set_global_memory_buffer():
_GLOBAL_MEMORY_BUFFER = GlobalMemoryBuffer()
-def _set_global_symmetric_memory_buffer():
- """Initialize global buffer."""
- global _GLOBAL_SYMMETRIC_MEMORY_BUFFER
- assert _GLOBAL_SYMMETRIC_MEMORY_BUFFER is None, "global memory buffer is already initialized"
-
- _GLOBAL_SYMMETRIC_MEMORY_BUFFER = GlobalSymmetricMemoryBuffer(
- size_in_mb=256, # todo: set from an argument?
- process_group=get_tensor_model_parallel_group(),
- )
-
-
def get_global_memory_buffer():
"""Return the global GlobalMemoryBuffer object"""
assert _GLOBAL_MEMORY_BUFFER is not None, "global memory buffer is not initialized"
return _GLOBAL_MEMORY_BUFFER
-def get_global_symmetric_memory_buffer():
- """Return the global GlobalSymmetricMemoryBuffer object"""
- assert (
- _GLOBAL_SYMMETRIC_MEMORY_BUFFER is not None
- ), "global symmetric memory buffer is not initialized"
- return _GLOBAL_SYMMETRIC_MEMORY_BUFFER
-
-
def destroy_global_memory_buffer():
"""Sets the global memory buffer to None"""
global _GLOBAL_MEMORY_BUFFER
_GLOBAL_MEMORY_BUFFER = None
-def destroy_global_symmetric_memory_buffer():
- """Sets the global symmetric memory buffer to None"""
- global _GLOBAL_SYMMETRIC_MEMORY_BUFFER
- _GLOBAL_SYMMETRIC_MEMORY_BUFFER = None
-
-
def get_all_ranks():
"""Get caller's rank in tensor-model-parallel, data-parallel, context-parallel,
pipeline-model-parallel and expert-model-parallel groups."""
@@ -2149,9 +2106,6 @@ def destroy_model_parallel():
global _GLOBAL_MEMORY_BUFFER
_GLOBAL_MEMORY_BUFFER = None
- global _GLOBAL_SYMMETRIC_MEMORY_BUFFER
- _GLOBAL_SYMMETRIC_MEMORY_BUFFER = None
-
global _DATA_PARALLEL_GROUP_GLOO
if (
_DATA_PARALLEL_GROUP_GLOO is not None
@@ -2234,3 +2188,5 @@ def destroy_model_parallel():
global _global_process_group_list
_global_process_group_list = None
+
+ SymmetricMemoryManager.destroy()
diff --git a/megatron/core/pipeline_parallel/bridge_communicator.py b/megatron/core/pipeline_parallel/bridge_communicator.py
index f1e74a2f16d..515ddf1743a 100644
--- a/megatron/core/pipeline_parallel/bridge_communicator.py
+++ b/megatron/core/pipeline_parallel/bridge_communicator.py
@@ -46,6 +46,17 @@ class BridgeCommunicator:
send_backward_recv_forward to be used by the pipeline schedule.
"""
+ # Cache broadcast PGs to avoid creating duplicate NCCL communicators for identical rank sets.
+ _broadcast_pg_cache: Dict[str, "torch.distributed.ProcessGroup"] = {}
+
+ @classmethod
+ def destroy_broadcast_pgs(cls):
+ """Destroy all cached broadcast process groups."""
+ for pg in cls._broadcast_pg_cache.values():
+ if pg is not None:
+ dist.destroy_process_group(pg)
+ cls._broadcast_pg_cache.clear()
+
def __init__(
self,
src_grid: HyperCommGrid,
@@ -54,6 +65,7 @@ def __init__(
comm_dtype: Optional[torch.dtype] = None,
src_module_name: Optional[str] = None,
dest_module_name: Optional[str] = None,
+ tensor_ndim: int = 3,
):
"""Initialize the bridge communicator between source and destination grids.
@@ -65,6 +77,11 @@ def __init__(
dim_mapping: Dictionary mapping logical dimensions to tensor axes.
Expected keys: 's' (sequence), 'b' (batch), 'h' (hidden).
Defaults to {'s': 1, 'b': 0, 'h': 2} if None.
+ tensor_ndim: Number of dimensions in tensors communicated through this
+ bridge. For 3D tensors (e.g. [S, B, H]), fan-in/fan-out
+ operates on dim_mapping['b']. For 2D tensors (e.g. [B*S, H]
+ where batch is folded into dim 0), fan-in/fan-out operates
+ on dim 0. Default: 3.
"""
self.src_grid = src_grid
self.dest_grid = dest_grid
@@ -72,6 +89,9 @@ def __init__(
self.dest_module_name = dest_module_name
self.comm_dtype = comm_dtype
+ assert tensor_ndim in (2, 3), f"tensor_ndim must be 2 or 3, got {tensor_ndim}"
+ self.tensor_ndim = tensor_ndim
+
# TODO (ykarnati, pthombre) - CP support will be added in follow up PR.
if 'cp' in self.src_grid.dim_names:
assert self.src_grid.shape[self.src_grid.dim_names.index('cp')] == 1, (
@@ -110,8 +130,8 @@ def __init__(
self.src_grid_broadcast_ranks = []
if src_grid_broadcast_ranks_list:
- self.src_grid_broadcast_pg, _ = dist.new_subgroups_by_enumeration(
- src_grid_broadcast_ranks_list, backend='nccl'
+ self.src_grid_broadcast_pg = self._get_or_create_broadcast_pg(
+ src_grid_broadcast_ranks_list
)
self.src_grid_broadcast_ranks = next(
(ranks for ranks in src_grid_broadcast_ranks_list if self.current_rank in ranks), []
@@ -119,8 +139,8 @@ def __init__(
self.dest_grid_broadcast_ranks = []
if dest_grid_broadcast_ranks_list:
- self.dest_grid_broadcast_pg, _ = dist.new_subgroups_by_enumeration(
- dest_grid_broadcast_ranks_list, backend='nccl'
+ self.dest_grid_broadcast_pg = self._get_or_create_broadcast_pg(
+ dest_grid_broadcast_ranks_list
)
self.dest_grid_broadcast_ranks = next(
(ranks for ranks in dest_grid_broadcast_ranks_list if self.current_rank in ranks),
@@ -146,6 +166,27 @@ def __init__(
self.build_comm_map(self.src_tp_leaders, self.dest_tp_leaders)
dist.barrier()
+ @property
+ def _batch_dim(self) -> int:
+ """Get the tensor dimension used for fan-in/fan-out (cat/split).
+
+ For 3D tensors (e.g. [S, B, H]), this is dim_mapping['b'].
+ For 2D tensors (e.g. [B*S, H] where batch is folded into the first
+ dimension), this is 0.
+ """
+ if self.tensor_ndim == 2:
+ return 0
+ return self.dim_mapping['b']
+
+ @classmethod
+ def _get_or_create_broadcast_pg(cls, ranks_list: List[List[int]]):
+ """Get or create a broadcast PG, caching to avoid duplicate NCCL communicators."""
+ cache_key = str(sorted([tuple(r) for r in ranks_list]))
+ if cache_key not in cls._broadcast_pg_cache:
+ pg, _ = dist.new_subgroups_by_enumeration(ranks_list, backend='nccl')
+ cls._broadcast_pg_cache[cache_key] = pg
+ return cls._broadcast_pg_cache[cache_key]
+
def get_leader_rank(self, grid: HyperCommGrid, is_src: bool) -> List[int]:
"""Get the leader rank for a given grid and direction.
@@ -365,7 +406,7 @@ def recv_forward(self) -> torch.Tensor:
f"shape {tensor_to_recv.shape} sum {tensor_to_recv.sum()}"
)
received_tensors_list.append(tensor_to_recv)
- aggregated_tensor = torch.cat(received_tensors_list, dim=self.dim_mapping['b'])
+ aggregated_tensor = torch.cat(received_tensors_list, dim=self._batch_dim)
logging.debug(
f"[Bridge Communicator] [receive_forward] Rank {self.current_rank} "
f"broadcasting tensor {aggregated_tensor.shape} sum {aggregated_tensor.sum()}"
@@ -389,7 +430,9 @@ def recv_forward(self) -> torch.Tensor:
and self.current_rank in self.dest_grid_broadcast_ranks
):
# Non-leader rank - participate in broadcast
- shape_tensor = torch.empty((3), device=torch.cuda.current_device(), dtype=torch.int64)
+ shape_tensor = torch.empty(
+ (self.tensor_ndim,), device=torch.cuda.current_device(), dtype=torch.int64
+ )
dist.broadcast(
shape_tensor, src=self.dest_local_leader_rank, group=self.dest_grid_broadcast_pg
)
@@ -494,7 +537,7 @@ def recv_backward(self) -> torch.Tensor:
received_gradients_list.append(grad_tensor)
# Concatenate received gradients
- aggregated_gradient = torch.cat(received_gradients_list, dim=0)
+ aggregated_gradient = torch.cat(received_gradients_list, dim=self._batch_dim)
logging.debug(
f"[Bridge Communicator] [receive_backward] Rank {self.current_rank} "
f"agg grad shape {aggregated_gradient.shape} sum {aggregated_gradient.sum()}"
@@ -516,7 +559,9 @@ def recv_backward(self) -> torch.Tensor:
):
# Non-leader rank - participate in gather for gradients
# Receive broadcasted tensor shape from leader rank
- shape_tensor = torch.empty((3), device=torch.cuda.current_device(), dtype=torch.int64)
+ shape_tensor = torch.empty(
+ (self.tensor_ndim,), device=torch.cuda.current_device(), dtype=torch.int64
+ )
dist.broadcast(
shape_tensor, src=self.src_local_leader_rank, group=self.src_grid_broadcast_pg
)
@@ -615,7 +660,7 @@ def send_forward_recv_backward(
req.wait()
# Concatenate received gradients
- aggregated_gradient = torch.cat(received_gradients_list, dim=0)
+ aggregated_gradient = torch.cat(received_gradients_list, dim=self._batch_dim)
logging.debug(
f"[Bridge Communicator] [send_forward_recv_backward] Rank {self.current_rank} "
f"agg grad shape {aggregated_gradient.shape} sum {aggregated_gradient.sum()}"
@@ -641,7 +686,9 @@ def send_forward_recv_backward(
):
# participate in both gather for gradients
# Receive gradient from leader using broadcast
- shape_tensor = torch.empty((3), device=torch.cuda.current_device(), dtype=torch.int64)
+ shape_tensor = torch.empty(
+ (self.tensor_ndim,), device=torch.cuda.current_device(), dtype=torch.int64
+ )
dist.broadcast(
shape_tensor, src=self.src_local_leader_rank, group=self.src_grid_broadcast_pg
)
@@ -737,9 +784,9 @@ def send_backward_recv_forward(
req.wait()
# Concatenate received activations
- aggregated_activation = torch.cat(received_activations_list, dim=0)
+ aggregated_activation = torch.cat(received_activations_list, dim=self._batch_dim)
logging.debug(
- f"[Bridge Communicator] [send_backward_recv_backward] Rank {self.current_rank} "
+ f"[Bridge Communicator] [send_backward_recv_forward] Rank {self.current_rank} "
f"agg act shape {aggregated_activation.shape} sum {aggregated_activation.sum()}"
)
@@ -762,7 +809,9 @@ def send_backward_recv_forward(
rank_info.role == CommRole.MEMBER
and self.current_rank in self.dest_grid_broadcast_ranks
):
- shape_tensor = torch.empty((3), device=torch.cuda.current_device(), dtype=torch.int64)
+ shape_tensor = torch.empty(
+ (self.tensor_ndim,), device=torch.cuda.current_device(), dtype=torch.int64
+ )
dist.broadcast(
shape_tensor, src=self.dest_local_leader_rank, group=self.dest_grid_broadcast_pg
)
@@ -843,7 +892,7 @@ def _communicate_shapes(
if recv_next:
for dest_rank in rank_info.send_to_ranks:
grad_shape_tensor = torch.empty(
- (3), device=torch.cuda.current_device(), dtype=torch.int64
+ (self.tensor_ndim,), device=torch.cuda.current_device(), dtype=torch.int64
)
recv_grad_shape_tensors.append(grad_shape_tensor)
ops.append(
@@ -857,7 +906,7 @@ def _communicate_shapes(
if recv_prev:
for src_rank in rank_info.recv_from_ranks:
forward_shape_tensor = torch.empty(
- (3), device=torch.cuda.current_device(), dtype=torch.int64
+ (self.tensor_ndim,), device=torch.cuda.current_device(), dtype=torch.int64
)
recv_forward_shape_tensors.append(forward_shape_tensor)
ops.append(
@@ -913,7 +962,6 @@ def _split_tensor_at_batch_dim(
if num_splits <= 0:
raise ValueError(f"num_splits must be positive, got {num_splits}")
- batch_dim = self.dim_mapping['b']
- splits = torch.tensor_split(aggregated_tensor, num_splits, dim=batch_dim)
+ splits = torch.tensor_split(aggregated_tensor, num_splits, dim=self._batch_dim)
# PyTorch p2p requires the tensors to be contiguous
return [split.contiguous() for split in splits]
diff --git a/megatron/core/pipeline_parallel/combined_1f1b.py b/megatron/core/pipeline_parallel/combined_1f1b.py
index 232d9c8cd70..fdd3b32201f 100644
--- a/megatron/core/pipeline_parallel/combined_1f1b.py
+++ b/megatron/core/pipeline_parallel/combined_1f1b.py
@@ -8,7 +8,12 @@
from megatron.core.enums import Fp8Recipe
from megatron.core.fp8_utils import get_fp8_context
-from megatron.core.pipeline_parallel.utils import AbstractSchedulePlan, ScheduleNode, set_streams
+from megatron.core.pipeline_parallel.utils import (
+ AbstractSchedulePlan,
+ ScheduleNode,
+ get_comp_stream,
+ set_streams,
+)
from megatron.core.utils import get_attr_wrapped_model
# Types
@@ -405,7 +410,7 @@ def forward_backward_step():
from megatron.core.pipeline_parallel.schedules import forward_step_calc_loss
loss_node = ScheduleNode(
- loss_func, torch.cuda.current_stream(), f_schedule_plan.event, name="loss_func"
+ loss_func, get_comp_stream, f_schedule_plan.event, name="loss_func"
)
loss_func = loss_node.forward
output_tensor, num_tokens = forward_step_calc_loss(
diff --git a/megatron/core/pipeline_parallel/multimodule_communicator.py b/megatron/core/pipeline_parallel/multimodule_communicator.py
index 1e8da3468e2..b2e5682a29d 100644
--- a/megatron/core/pipeline_parallel/multimodule_communicator.py
+++ b/megatron/core/pipeline_parallel/multimodule_communicator.py
@@ -45,6 +45,68 @@ class RankModuleInfo:
is_terminal_stage: Optional[bool] = True
+def _prepare_tensor_for_comm(
+ tensor: Union[torch.Tensor, List[torch.Tensor], None]
+) -> Union[torch.Tensor, List[torch.Tensor], None]:
+ """Prepare tensor for P2P communication by expanding to 3D if needed.
+
+ Only used for intra-module P2P paths. Bridge communicators handle 2D/3D
+ tensors natively via tensor_ndim and do not need this adapter.
+
+ P2P communicators expect 3D tensors. 2D tensors are unsqueezed by adding
+ a singleton last dimension, and _restore_tensor_from_comm will squeeze it back. 3D
+ tensors are passed through unchanged.
+
+ Note: 3D tensors with a singleton last dimension (shape [a, b, 1]) are not supported
+ because _restore_tensor_from_comm cannot distinguish them from unsqueezed 2D tensors.
+
+ Args:
+ tensor: Input tensor (2D or 3D), list of tensors, or None.
+
+ Returns:
+ 3D tensor (with singleton last dim if input was 2D), list of 3D tensors, or None.
+ """
+ if tensor is None:
+ return None
+ if isinstance(tensor, list):
+ return [_prepare_tensor_for_comm(t) for t in tensor]
+ if isinstance(tensor, torch.Tensor):
+ if tensor.ndim == 2:
+ return tensor.unsqueeze(-1)
+ assert tensor.ndim != 3 or tensor.shape[-1] != 1, (
+ f"3D tensor with singleton last dim {tuple(tensor.shape)} is ambiguous for "
+ "multimodule comm. Cannot distinguish from an unsqueezed 2D tensor on the "
+ "receiving rank. Use a 2D tensor or a 3D tensor with last_dim > 1."
+ )
+ return tensor
+
+
+def _restore_tensor_from_comm(
+ tensor: Union[torch.Tensor, List[torch.Tensor], None]
+) -> Union[torch.Tensor, List[torch.Tensor], None]:
+ """Restore tensor shape after P2P communication by squeezing singleton dim.
+
+ Only used for intra-module P2P paths. Bridge communicators handle 2D/3D
+ tensors natively via tensor_ndim and do not need this adapter.
+
+ Removes the extra dimension added by _prepare_tensor_for_comm if it was singleton.
+ Handles both single tensors and lists of tensors (for VPP).
+
+ Args:
+ tensor: Input tensor (3D with singleton last dim), list of tensors, or None.
+
+ Returns:
+ 2D tensor (if last dim was singleton), list of tensors, or None.
+ """
+ if tensor is None:
+ return None
+ if isinstance(tensor, list):
+ return [_restore_tensor_from_comm(t) for t in tensor]
+ if isinstance(tensor, torch.Tensor) and tensor.ndim == 3 and tensor.shape[-1] == 1:
+ return tensor.squeeze(-1)
+ return tensor
+
+
class MultiModulePipelineCommunicator:
"""Communicator for a multi-module pipeline."""
@@ -54,6 +116,7 @@ def __init__(
topology: Dict[str, List[str]],
config: ModelParallelConfig,
dim_mapping: Dict[str, List[int]] = None,
+ module_output_ndim: Optional[Dict[str, int]] = None,
):
"""
Initialize the MultiModulePipelineCommunicator.
@@ -80,11 +143,19 @@ def __init__(
Example:
dim_mapping = {'s': 0, 'h': 2, 'b': 1}
Default: None
+ module_output_ndim (Dict[str, int]): Number of dimensions for each module's
+ output tensor. Used by bridge communicators for cross-module fan-in/fan-out.
+ Modules producing 2D tensors [B*S, H] (e.g. vision encoders) should be 2.
+ Modules not listed default to 3.
+ Example:
+ module_output_ndim = {'image_encoder': 2, 'llm': 3}
+ Default: None (all modules assumed 3D)
"""
self.module_to_grid_map = module_to_grid_map
self.topology = topology
self.config = config
self.dim_mapping = dim_mapping
+ self.module_output_ndim = module_output_ndim or {}
self.current_rank = dist.get_rank()
# Build bridge communicators for all modules
@@ -108,6 +179,7 @@ def _build_bridge_comms(self):
comm_dtype=self.config.pipeline_dtype,
src_module_name=src_module_name,
dest_module_name=dest_module_name,
+ tensor_ndim=self.module_output_ndim.get(src_module_name, 3),
)
self.bridge_comms.append(bridge_comm)
@@ -160,25 +232,31 @@ def is_current_rank_in_grid(self, grid: HyperCommGrid) -> bool:
return grid.rank_offset <= self.current_rank < grid.rank_offset + grid.size
@property
- def num_warmup_microbatches(self):
- """Calculate the number of warmup microbatches for the current rank.
+ def total_stages(self) -> int:
+ """Return total number of pipeline stages across all modules.
- Uses the same simple logic as P2PCommunicator:
- total_pipeline_stages - current_rank_stage - 1
+ Computes the longest path through the module DAG weighted by each
+ module's pipeline-parallel size.
Returns:
- int: Number of warmup microbatches for this rank
+ int: Total pipeline stages.
"""
- # Get total pipeline depth across all modules
- total_stages = self.compute_total_pipeline_stages(self.topology, self.module_to_grid_map)
+ return self.compute_total_pipeline_stages(self.topology, self.module_to_grid_map)
+
+ @property
+ def current_stage(self) -> int:
+ """Return current pipeline stage index (0-indexed) within the multi-module pipeline.
+
+ Returns:
+ int: Current stage index.
+ """
+ total = self.total_stages
- # Get current rank's position in the overall pipeline (0-indexed)
- # Use compute_total_pipeline_stages with current rank to get cumulative position
if self.rank_module_map:
# Take the first module this rank belongs to
# TODO: ykarnati - improve this logic.
module_name = next(iter(self.rank_module_map.keys()))
- current_stage = (
+ stage = (
self.compute_total_pipeline_stages(
self.topology,
self.module_to_grid_map,
@@ -188,17 +266,15 @@ def num_warmup_microbatches(self):
- 1
) # Convert from 1-indexed to 0-indexed
else:
- current_stage = 0
+ stage = 0
- assert (
- current_stage <= total_stages
- ), f"current_stage: {current_stage} is greater than total_stages: {total_stages}"
+ assert stage < total, f"current_stage: {stage} must be less than total_stages: {total}"
logging.debug(
f"[Rank {dist.get_rank()} ][MultiModulePipelineCommunicator] "
- f"current_stage: {current_stage} total_stages: {total_stages} "
- f"num_warmup_microbatches: {total_stages - current_stage - 1}"
+ f"current_stage: {stage} total_stages: {total} "
+ f"num_warmup_microbatches: {total - stage - 1}"
)
- return total_stages - current_stage - 1
+ return stage
def _build_rank_module_info_map(self):
"""For each module in the current rank, initialize the P2P communicator
@@ -266,12 +342,15 @@ def recv_forward(
# If first stage, and has incoming modules, receive forward activation
# from incoming modules.
for bridge_comm in rank_module_info.bridge_comms_as_dest_module:
- input_dict[bridge_comm.src_module_name] = bridge_comm.recv_forward()
+ received_tensor = bridge_comm.recv_forward()
+ input_dict[bridge_comm.src_module_name] = received_tensor
else:
# If not first stage, receive forward activation tensor from P2P communicator.
- input_dict[module_name] = rank_module_info.p2p_communicator.recv_forward(
+ # P2P hardcodes 3D shape buffers, so use adapter for 2D tensors.
+ received_tensor = rank_module_info.p2p_communicator.recv_forward(
tensor_shapes=tensor_shape, is_first_stage=False
)
+ input_dict[module_name] = _restore_tensor_from_comm(received_tensor)
return input_dict
def send_forward(self, output_dict: Dict[str, torch.Tensor], is_last_stage: bool = False):
@@ -280,10 +359,6 @@ def send_forward(self, output_dict: Dict[str, torch.Tensor], is_last_stage: bool
Args:
output_dict: A dictionary mapping module names to tensors.
"""
- logging.debug(
- f"[Rank {dist.get_rank()} ][MultiModulePipelineCommunicator] "
- f"[send_forward] output_dict keys: {output_dict.keys()}, is_last_stage: {is_last_stage}"
- )
for module_name, rank_module_info in self.rank_module_map.items():
if rank_module_info.pp_rank == rank_module_info.pp_size - 1:
# If last stage, and has outgoing modules, send forward activation
@@ -292,9 +367,8 @@ def send_forward(self, output_dict: Dict[str, torch.Tensor], is_last_stage: bool
bridge_comm.send_forward(output_dict[module_name])
else:
# If not last stage, send forward activation by using P2P communicator.
- rank_module_info.p2p_communicator.send_forward(
- output_dict[module_name], is_last_stage=False
- )
+ tensor_to_send = _prepare_tensor_for_comm(output_dict[module_name])
+ rank_module_info.p2p_communicator.send_forward(tensor_to_send, is_last_stage=False)
def send_forward_recv_backward(
self,
@@ -311,28 +385,22 @@ def send_forward_recv_backward(
Returns:
A dictionary mapping module names to tensors.
"""
- logging.debug(
- f"[Rank {dist.get_rank()} ][MultiModulePipelineCommunicator] "
- f"[send_forward_recv_backward] output_dict keys: {output_dict.keys()}, "
- f"tensor_shape: {tensor_shape}, is_last_stage: {is_last_stage}"
- )
grad_dict = {}
for module_name, rank_module_info in self.rank_module_map.items():
if rank_module_info.pp_rank == rank_module_info.pp_size - 1:
# If last stage, and has outgoing modules, send forward activation and
# receive backward gradient by using bridge communicator.
for bridge_comm in rank_module_info.bridge_comms_as_src_module:
- grad_dict[bridge_comm.src_module_name] = bridge_comm.send_forward_recv_backward(
- output_dict[module_name]
- )
+ grad = bridge_comm.send_forward_recv_backward(output_dict[module_name])
+ grad_dict[bridge_comm.src_module_name] = grad
else:
# If not last stage, send forward activation and receive backward gradient
# by using P2P communicator.
- grad_dict[module_name] = (
- rank_module_info.p2p_communicator.send_forward_recv_backward(
- output_dict[module_name], tensor_shapes=tensor_shape, is_last_stage=False
- )
+ tensor_to_send = _prepare_tensor_for_comm(output_dict[module_name])
+ grad = rank_module_info.p2p_communicator.send_forward_recv_backward(
+ tensor_to_send, tensor_shapes=tensor_shape, is_last_stage=False
)
+ grad_dict[module_name] = _restore_tensor_from_comm(grad)
return grad_dict
def send_backward_recv_forward(
@@ -350,30 +418,24 @@ def send_backward_recv_forward(
Returns:
A dictionary mapping module names to tensors.
"""
- logging.debug(
- f"[Rank {dist.get_rank()} ][MultiModulePipelineCommunicator] "
- f"[send_backward_recv_forward] grad_dict keys: {grad_dict.keys()}, "
- f"tensor_shape: {tensor_shape}, is_first_stage: {is_first_stage}"
- )
input_dict = {}
for module_name, rank_module_info in self.rank_module_map.items():
if rank_module_info.pp_rank == 0:
for bridge_comm in rank_module_info.bridge_comms_as_dest_module:
# If first stage, and has incoming modules, send backward gradient and
# receive forward activation by using bridge communicator.
- input_dict[bridge_comm.src_module_name] = (
- bridge_comm.send_backward_recv_forward(
- grad_dict[bridge_comm.src_module_name]
- )
+ received_tensor = bridge_comm.send_backward_recv_forward(
+ grad_dict[bridge_comm.src_module_name]
)
+ input_dict[bridge_comm.src_module_name] = received_tensor
else:
# If not first stage, send backward gradient and receive forward activation
# by using P2P communicator.
- input_dict[module_name] = (
- rank_module_info.p2p_communicator.send_backward_recv_forward(
- grad_dict[module_name], tensor_shapes=tensor_shape, is_first_stage=False
- )
+ grad_to_send = _prepare_tensor_for_comm(grad_dict[module_name])
+ received_tensor = rank_module_info.p2p_communicator.send_backward_recv_forward(
+ grad_to_send, tensor_shapes=tensor_shape, is_first_stage=False
)
+ input_dict[module_name] = _restore_tensor_from_comm(received_tensor)
return input_dict
def recv_backward(
@@ -397,12 +459,14 @@ def recv_backward(
# If last stage, and has incoming modules, receive backward gradient
# by using bridge communicator.
for bridge_comm in rank_module_info.bridge_comms_as_src_module:
- grad_dict[bridge_comm.src_module_name] = bridge_comm.recv_backward()
+ grad = bridge_comm.recv_backward()
+ grad_dict[bridge_comm.src_module_name] = grad
else:
# If not last stage, receive backward gradient by using P2P communicator.
- grad_dict[module_name] = rank_module_info.p2p_communicator.recv_backward(
+ grad = rank_module_info.p2p_communicator.recv_backward(
tensor_shapes=tensor_shape, is_last_stage=False
)
+ grad_dict[module_name] = _restore_tensor_from_comm(grad)
return grad_dict
def send_backward(self, grad_dict: Dict[str, torch.Tensor], is_first_stage: bool = False):
@@ -411,10 +475,6 @@ def send_backward(self, grad_dict: Dict[str, torch.Tensor], is_first_stage: bool
Args:
grad_dict: A dictionary mapping module names to tensors.
"""
- logging.debug(
- f"[Rank {dist.get_rank()} ][MultiModulePipelineCommunicator] "
- f"[send_backward] grad_dict keys: {grad_dict.keys()}, is_first_stage: {is_first_stage}"
- )
for module_name, rank_module_info in self.rank_module_map.items():
if rank_module_info.pp_rank == 0:
# If first stage, and has incoming modules, send backward activation
@@ -423,9 +483,8 @@ def send_backward(self, grad_dict: Dict[str, torch.Tensor], is_first_stage: bool
bridge_comm.send_backward(grad_dict[bridge_comm.src_module_name])
else:
# If not first stage, send backward activation by using P2P communicator.
- rank_module_info.p2p_communicator.send_backward(
- grad_dict[module_name], is_first_stage=False
- )
+ grad_to_send = _prepare_tensor_for_comm(grad_dict[module_name])
+ rank_module_info.p2p_communicator.send_backward(grad_to_send, is_first_stage=False)
@staticmethod
def compute_total_pipeline_stages(
diff --git a/megatron/core/pipeline_parallel/p2p_communication.py b/megatron/core/pipeline_parallel/p2p_communication.py
index ac839c21f18..465e83f28e6 100644
--- a/megatron/core/pipeline_parallel/p2p_communication.py
+++ b/megatron/core/pipeline_parallel/p2p_communication.py
@@ -7,6 +7,7 @@
import torch.distributed as dist
from megatron.core.model_parallel_config import ModelParallelConfig
+from megatron.core.pipeline_parallel.utils import is_pp_first_stage, is_pp_last_stage
from megatron.core.utils import nvtx_decorator
# Types
@@ -162,6 +163,26 @@ def __init__(self, pp_group: dist.ProcessGroup, config: ModelParallelConfig):
else None
)
+ @property
+ def is_pp_first_stage(self) -> bool:
+ """Return True if pp first stage."""
+ return is_pp_first_stage(self.pp_group)
+
+ @property
+ def is_pp_last_stage(self) -> bool:
+ """Return True if pp last stage."""
+ return is_pp_last_stage(self.pp_group)
+
+ @property
+ def total_stages(self) -> int:
+ """Return total number of pipeline stages."""
+ return self.pp_group.size()
+
+ @property
+ def current_stage(self) -> int:
+ """Return current pipeline stage index (0-indexed)."""
+ return self.pp_group.rank()
+
def _communicate_shapes(self, tensor_send_next, tensor_send_prev, recv_prev, recv_next):
"""Communicate tensor shapes between stages. Used to communicate
tensor shapes before the actual tensor communication happens.
diff --git a/megatron/core/pipeline_parallel/schedules.py b/megatron/core/pipeline_parallel/schedules.py
index 15c5adfc7a2..24ec25e5150 100644
--- a/megatron/core/pipeline_parallel/schedules.py
+++ b/megatron/core/pipeline_parallel/schedules.py
@@ -2,16 +2,16 @@
import contextlib
from functools import partial
-from typing import Callable, Iterator, List, Optional, Union
+from typing import Callable, Dict, Iterator, List, Optional, Union
import torch
from torch.autograd.variable import Variable
from megatron.core import parallel_state
-from megatron.core.enums import ModelType
from megatron.core.pipeline_parallel.fine_grained_activation_offload import (
FineGrainedActivationOffloadingInterface as off_interface,
)
+from megatron.core.pipeline_parallel.multimodule_communicator import MultiModulePipelineCommunicator
from megatron.core.pipeline_parallel.p2p_communication import P2PCommunicator
from megatron.core.pipeline_parallel.utils import (
is_pp_first_stage,
@@ -19,8 +19,11 @@
is_vp_first_stage,
is_vp_last_stage,
)
-from megatron.core.process_groups_config import ProcessGroupCollection
-from megatron.core.transformer.cuda_graphs import create_cudagraphs
+from megatron.core.process_groups_config import (
+ MultiModuleProcessGroupCollection,
+ ProcessGroupCollection,
+)
+from megatron.core.transformer.cuda_graphs import create_cudagraphs, set_current_microbatch
from megatron.core.transformer.enums import CudaGraphScope
from megatron.core.transformer.moe.router import MoEAuxLossAutoScaler
from megatron.core.utils import (
@@ -157,9 +160,28 @@ def deallocate_output_tensor(out, deallocate_pipeline_outputs=False):
This method should be called right after the output tensor has been
sent to the next pipeline stage. At this point, the output tensor is
only useful for its '.grad_fn' field, and not its '.data'.
+
+ Supports multiple formats:
+ - torch.Tensor: Deallocates the tensor directly
+ - List[Tensor]: Recursively deallocates each element
+ - Dict[str, Tensor]: Recursively deallocates each value (for multi-module pipelines)
'''
if (out is None) or (not deallocate_pipeline_outputs):
return
+
+ # Handle dict format (multi-module pipelines)
+ if isinstance(out, dict):
+ for value in out.values():
+ deallocate_output_tensor(value, deallocate_pipeline_outputs)
+ return
+
+ # Handle list format
+ if isinstance(out, list):
+ for item in out:
+ deallocate_output_tensor(item, deallocate_pipeline_outputs)
+ return
+
+ # Base case: deallocate tensor
assert isinstance(out, torch.Tensor), "expected Tensor, found %s." % type(out).__name__
assert out._base is None, "counter-productive to free a view of another tensor."
out.data = torch.empty((1,), device=out.device, dtype=out.dtype)
@@ -197,27 +219,6 @@ def custom_backward(output, grad_output):
)
-def set_current_microbatch(model, microbatch_id):
- """Set the current microbatch."""
- decoder_exists = True
- model_with_decoder = None
- try:
- model_with_decoder = get_attr_wrapped_model(
- model, "decoder", allow_none=False, return_model_obj=True
- )
- except RuntimeError:
- decoder_exists = False
- if decoder_exists and model_with_decoder is not None:
- for layer in model_with_decoder.decoder.layers:
- layer.current_microbatch = microbatch_id
- if hasattr(model_with_decoder, 'mtp'):
- for layer in model_with_decoder.mtp.layers:
- assert hasattr(
- layer, 'mtp_model_layer'
- ), f"MTP layer {layer} must have 'mtp_model_layer' attribute"
- layer.mtp_model_layer.current_microbatch = microbatch_id
-
-
def forward_step_calc_loss(
model,
output_tensor,
@@ -247,9 +248,9 @@ def forward_step_calc_loss(
ignore_virtual=False, vp_stage=vp_stage
)
else:
- assert (
- cp_group_size is not None and is_last_stage is not None
- ), "cp_group_size and is_last_stage must be provided"
+ assert is_last_stage is not None, "is_last_stage must be provided"
+ if is_last_stage:
+ assert cp_group_size is not None, "cp_group_size must be provided on last stage"
num_tokens = torch.tensor(0, dtype=torch.int)
if is_last_stage:
@@ -292,7 +293,8 @@ def forward_step_calc_loss(
if config.calculate_per_token_loss:
MoEAuxLossAutoScaler.set_loss_scale(loss_scale)
else:
- MoEAuxLossAutoScaler.set_loss_scale(loss_scale * cp_group_size / num_microbatches)
+ cp_size_for_scaling = cp_group_size if cp_group_size is not None else 1
+ MoEAuxLossAutoScaler.set_loss_scale(loss_scale * cp_size_for_scaling / num_microbatches)
# Set the loss scale for Multi-Token Prediction (MTP) loss.
if hasattr(config, 'mtp_num_layers') and config.mtp_num_layers is not None:
@@ -446,14 +448,13 @@ def forward_step(
return [output_tensor], num_tokens
-def backward_step(input_tensor, output_tensor, output_tensor_grad, model_type, config):
+def backward_step(input_tensor, output_tensor, output_tensor_grad, config):
"""Backward step through passed-in output tensor.
If last stage, output_tensor_grad is None, otherwise gradient of loss
with respect to stage's output tensor.
- Returns gradient of loss with respect to input tensor (None if first
- stage)."""
+ Returns gradient of loss with respect to input tensor (None if first stage)."""
# NOTE: This code currently can handle at most one skip connection. It
# needs to be modified slightly to support arbitrary numbers of skip
@@ -510,6 +511,67 @@ def backward_step(input_tensor, output_tensor, output_tensor_grad, model_type, c
return input_tensor_grad
+def backward_step_multimodule(
+ input_tensor: Dict[str, torch.Tensor],
+ output_tensor: Union[torch.Tensor, Dict[str, torch.Tensor]],
+ output_tensor_grad: Optional[Dict[str, torch.Tensor]],
+ config,
+ language_model_module_name: str,
+) -> Dict[str, torch.Tensor]:
+ """Backward step for multi-module pipelines.
+
+ In multi-module pipelines, tensors are organized as dictionaries with
+ module names as keys. Each module's backward pass is performed independently.
+ """
+ # Retain gradients on all input tensors.
+ for module_name, tensor in input_tensor.items():
+ if isinstance(tensor, list):
+ tensor = tensor[0]
+ if tensor is not None:
+ tensor.retain_grad()
+
+ # Last stage: output_tensor is a scalar loss from the language model.
+ # Associate it with the language_model_module_name.
+ if not isinstance(output_tensor, dict):
+ output_tensor = {language_model_module_name: output_tensor}
+
+ # Handle output_tensor_grad: None (last stage) or dict (intermediate stages).
+ if not output_tensor_grad:
+ output_tensor_grad = {key: None for key in output_tensor.keys()}
+
+ # Apply grad scaling if needed (for last stage only).
+ for module_name in output_tensor.keys():
+ if output_tensor_grad[module_name] is None and config.grad_scale_func is not None:
+ output_tensor[module_name] = config.grad_scale_func(output_tensor[module_name])
+
+ # Perform backward pass for each module.
+ for module_name in output_tensor.keys():
+ output_tensor_module = output_tensor[module_name]
+ output_tensor_grad_module = output_tensor_grad[module_name]
+
+ # In multi-modal models like VLM, some batches may not have images.
+ # In such cases, skip backward while preserving zero gradients.
+ if output_tensor_module is not None and output_tensor_module.requires_grad:
+ if config.deallocate_pipeline_outputs:
+ custom_backward(output_tensor_module, output_tensor_grad_module)
+ else:
+ torch.autograd.backward(
+ output_tensor_module, grad_tensors=output_tensor_grad_module
+ )
+
+ # Collect gradients for input tensors.
+ input_tensor_grad = {}
+ for module_name, tensor in input_tensor.items():
+ if isinstance(tensor, list):
+ tensor = tensor[0]
+ if tensor is None:
+ input_tensor_grad[module_name] = None
+ else:
+ input_tensor_grad[module_name] = tensor.grad
+
+ return input_tensor_grad
+
+
def check_first_val_step(first_val_step, forward_only, cond):
"""Check if it is the first validation step."""
if (first_val_step is not None) and forward_only:
@@ -554,23 +616,8 @@ def forward_backward_no_pipelining(
)
elif pg_collection is not None:
- assert hasattr(pg_collection, 'tp')
- assert hasattr(pg_collection, 'cp')
- assert hasattr(pg_collection, 'embd'), (
- "pg_collection must have a embd. In previous version, it is used default "
- "`parallel_state.default_embedding_ranks` to create the process group. If you are "
- "using the default process group, please use `parallel_state.get_embedding_group()` "
- "to get the process group. If you don't need explicitly set it to None."
- )
- assert hasattr(pg_collection, 'pos_embd'), (
- "pg_collection must have a pos_embd. In previous version, it is used default "
- "`parallel_state.default_position_embedding_ranks` to create the process group. "
- "If you are using the default process group, "
- "please use `parallel_state.get_position_embedding_group()` "
- "to get the process group. If you don't need explicitly set it to None."
- )
- assert hasattr(pg_collection, 'pp')
- assert hasattr(pg_collection, 'dp_cp')
+ assert hasattr(pg_collection, 'tp'), "pg_collection must have tp"
+ assert hasattr(pg_collection, 'cp'), "pg_collection must have cp"
if isinstance(model, list):
assert len(model) == 1, "non-pipeline-parallel schedule does not support model chunking"
@@ -651,9 +698,7 @@ def forward_backward_no_pipelining(
)
total_num_tokens += num_tokens
if not forward_only:
- backward_step(
- input_tensor, output_tensor, output_tensor_grad, model_type, config
- )
+ backward_step(input_tensor, output_tensor, output_tensor_grad, config)
# Run computation for last microbatch out of context handler (want to
# synchronize gradients).
output_tensor, num_tokens = forward_step(
@@ -675,7 +720,7 @@ def forward_backward_no_pipelining(
total_num_tokens += num_tokens
if not forward_only:
- backward_step(input_tensor, output_tensor, output_tensor_grad, model_type, config)
+ backward_step(input_tensor, output_tensor, output_tensor_grad, config)
if config.finalize_model_grads_func is not None and not forward_only:
# Finalize model grads (perform full grad all-reduce / reduce-scatter for
@@ -687,7 +732,7 @@ def forward_backward_no_pipelining(
force_all_reduce=force_all_reduce,
)
- if not forward_only and config.fine_grained_activation_offloading:
+ if getattr(config, 'fine_grained_activation_offloading', False):
off_interface.reset()
if config.timers is not None:
@@ -866,6 +911,7 @@ def forward_backward_pipelining_with_interleaving(
)
tp_group = parallel_state.get_tensor_model_parallel_group()
cp_group = parallel_state.get_context_parallel_group()
+ cp_size = cp_group.size()
embd_group = parallel_state.get_embedding_group(check_initialized=False)
pp_group = parallel_state.get_pipeline_model_parallel_group()
pos_emb_group = parallel_state.get_position_embedding_group(check_initialized=False)
@@ -882,30 +928,12 @@ def forward_backward_pipelining_with_interleaving(
elif p2p_communicator is not None and pg_collection is not None:
model_type = get_model_type(model[0])
- assert model_type != ModelType.encoder_and_decoder, (
- "encoder PP stages not yet supported when passing custom process groups. "
- "support coming soon!"
- )
assert hasattr(p2p_communicator, 'config'), "p2p_communicator must have a config"
- assert hasattr(pg_collection, 'tp'), "pg_collection must have a tp_group"
- assert hasattr(pg_collection, 'cp'), "pg_collection must have a cp_group"
- assert hasattr(pg_collection, 'embd'), (
- "pg_collection must have a embd. In previous version, it is used default "
- "`parallel_state.default_embedding_ranks` to create the process group. If you are "
- "using the default process group, please use `parallel_state.get_embedding_group()` "
- "to get the process group. If you don't need explicitly set it to None."
- )
- assert hasattr(pg_collection, 'pos_embd'), (
- "pg_collection must have a pos_embd. In previous version, it is used default "
- "`parallel_state.default_position_embedding_ranks` to create the process group."
- " If you are using the default process group, please use "
- "`parallel_state.get_position_embedding_group()` "
- "If you don't need pos_embd_group, you need to explicitly set it to None."
- )
- assert hasattr(pg_collection, 'pp'), "pg_collection must have a pp_group"
- assert hasattr(pg_collection, 'dp_cp'), "pg_collection must have a dp_cp_group"
+ assert hasattr(pg_collection, 'tp'), "pg_collection must have tp"
+ assert hasattr(pg_collection, 'cp'), "pg_collection must have cp"
tp_group = pg_collection.tp
cp_group = pg_collection.cp
+ cp_size = cp_group.size()
else:
raise ValueError(
"Invalid combination of p2p_communicator, pg_collection"
@@ -1237,7 +1265,7 @@ def forward_step_helper(virtual_microbatch_id, checkpoint_activations_microbatch
input_tensor,
forward_data_store,
config,
- cp_group_size=pg_collection.cp.size(),
+ cp_group_size=cp_size,
collect_non_loss_data=collect_non_loss_data,
checkpoint_activations_microbatch=checkpoint_activations_microbatch,
is_first_microbatch=check_first_val_step(
@@ -1302,9 +1330,7 @@ def backward_step_helper(virtual_microbatch_id):
virtual_microbatch_id, model_chunk_id
)
- input_tensor_grad = backward_step(
- input_tensor, output_tensor, output_tensor_grad, model_type, config
- )
+ input_tensor_grad = backward_step(input_tensor, output_tensor, output_tensor_grad, config)
backward_step_helper_postprocess(virtual_microbatch_id)
@@ -1912,7 +1938,7 @@ def pp_post_backward(input_tensor_grad, vp_stage=None):
# If defer_embedding_wgrad_compute is enabled we need to do the
# weight gradient GEMM's here.
finish_embedding_wgrad_compute(
- config, embedding_module, is_pp_last_stage(p2p_communicator.pp_group), tp_group
+ config, embedding_module, p2p_communicator.is_pp_last_stage, tp_group
)
# Finalize model grads (perform full grad all-reduce / reduce-scatter for
@@ -1926,7 +1952,7 @@ def pp_post_backward(input_tensor_grad, vp_stage=None):
force_all_reduce=force_all_reduce,
)
- if not forward_only and config.fine_grained_activation_offloading:
+ if getattr(config, 'fine_grained_activation_offloading', False):
off_interface.reset()
# Restore config.grad_sync_func and config.param_sync_func.
if forward_only:
@@ -1952,16 +1978,22 @@ def get_tensor_shapes(
micro_batch_size: int,
decoder_seq_length: int,
config,
- tp_group: torch.distributed.ProcessGroup,
- cp_group: torch.distributed.ProcessGroup,
+ tp_group: Optional[torch.distributed.ProcessGroup] = None,
+ cp_group: Optional[torch.distributed.ProcessGroup] = None,
):
- """
- Determine right tensor sizes (based on position of rank with respect to split rank) and
- model size.
- """
+ """Determine tensor shapes for pipeline communication.
+ Returns [()] for variable_seq_lengths mode (shapes exchanged dynamically),
+ or computed shapes for fixed sequence length mode.
+ """
tensor_shapes = []
- # Use decoder_seq_length if provided, otherwise use seq_length
+
+ if config.variable_seq_lengths:
+ # Shapes exchanged dynamically during P2P communication
+ tensor_shapes.append(())
+ return tensor_shapes
+
+ # Fixed sequence lengths - compute shape
effective_seq_length = decoder_seq_length if decoder_seq_length is not None else seq_length
effective_seq_length = effective_seq_length // cp_group.size()
@@ -1986,7 +2018,9 @@ def forward_backward_pipelining_without_interleaving(
first_val_step: Optional[bool] = None,
adjust_tensor_shapes_fn: Optional[Callable] = None,
p2p_communicator: Optional[P2PCommunicator] = None,
- pg_collection: Optional[ProcessGroupCollection] = None,
+ pg_collection: Optional[
+ Union[ProcessGroupCollection, MultiModuleProcessGroupCollection]
+ ] = None,
force_all_reduce: Optional[bool] = False,
):
"""Run non-interleaved 1F1B schedule, with communication between pipeline
@@ -2009,12 +2043,22 @@ def forward_backward_pipelining_without_interleaving(
"Non-interleaved pipeline parallelism does not support overlapping p2p communication"
)
+ tp_group, cp_group, cp_size = None, None, None
+
+ # Determine if this is a multi-module pipeline
+ # (used for validation and backward function selection)
+ is_multimodule = isinstance(pg_collection, MultiModuleProcessGroupCollection) or isinstance(
+ p2p_communicator, MultiModulePipelineCommunicator
+ )
+
if p2p_communicator is None and pg_collection is None:
+ # Default: single-module with parallel_state groups
p2p_communicator = P2PCommunicator(
pp_group=parallel_state.get_pipeline_model_parallel_group(), config=config
)
tp_group = parallel_state.get_tensor_model_parallel_group()
cp_group = parallel_state.get_context_parallel_group()
+ cp_size = cp_group.size()
embd_group = parallel_state.get_embedding_group(check_initialized=False)
pos_emb_group = parallel_state.get_position_embedding_group(check_initialized=False)
pp_group = parallel_state.get_pipeline_model_parallel_group()
@@ -2028,43 +2072,42 @@ def forward_backward_pipelining_without_interleaving(
pg_collection.dp_cp = parallel_state.get_data_parallel_group(
with_context_parallel=True, partial_data_parallel=False
)
+
elif p2p_communicator is not None and pg_collection is not None:
- model_type = get_model_type(model)
- assert model_type != ModelType.encoder_and_decoder, (
- "encoder PP stages not yet supported when passing custom process groups. "
- "support coming soon!"
- )
assert hasattr(p2p_communicator, 'config'), "p2p_communicator must have a config"
- assert hasattr(pg_collection, 'tp'), "pg_collection must have tp_group"
- assert hasattr(pg_collection, 'cp'), "pg_collection must have cp_group"
- assert hasattr(pg_collection, 'embd'), (
- "pg_collection must have a embd. In previous version, it is used default "
- "`parallel_state.default_embedding_ranks` to create the process group. "
- " If you are using the default process group, please use "
- " `parallel_state.get_embedding_group()` "
- "If you don't need embd_group, you need to explicitly set it to None."
- )
- assert hasattr(pg_collection, 'pos_embd'), (
- "pg_collection must have a pos_embd. In previous version, it is used default "
- "`parallel_state.default_position_embedding_ranks` to create the process group. "
- " If you are using the default process group, please use "
- " `parallel_state.get_position_embedding_group()` "
- "If you don't need pos_embd_group, you need to explicitly set it to None."
- )
- assert hasattr(pg_collection, 'pp'), "pg_collection must have pp_group"
- assert hasattr(pg_collection, 'dp_cp'), "pg_collection must have dp_cp_group"
- tp_group = pg_collection.tp
- cp_group = pg_collection.cp
+
+ if is_multimodule:
+ # Multi-module: use language model's CP size for loss scaling
+ if not config.variable_seq_lengths:
+ raise ValueError(
+ "config.variable_seq_lengths=True required for multi-module pipelines"
+ )
+ if pg_collection.has_language_model():
+ cp_size = pg_collection.get_language_model_cp_size()
+ else:
+ # Encoder-only ranks should not use CP loss scaling.
+ cp_size = None
+
+ elif isinstance(pg_collection, ProcessGroupCollection):
+ # Single-module: extract tp/cp groups and cp_size
+ assert hasattr(pg_collection, 'tp'), "pg_collection must have tp"
+ assert hasattr(pg_collection, 'cp'), "pg_collection must have cp"
+ tp_group = pg_collection.tp
+ cp_group = pg_collection.cp
+ cp_size = cp_group.size()
+
+ else:
+ raise TypeError(
+ f"pg_collection must be ProcessGroupCollection or "
+ f"MultiModuleProcessGroupCollection, got {type(pg_collection)}"
+ )
else:
- raise ValueError(
- "Invalid combination of p2p_communicator, pg_collection "
- "provide none or provide all the process groups"
- )
+ raise ValueError("Provide both p2p_communicator and pg_collection, or neither")
# Needed only when gradients are finalized in M-Core
if config.finalize_model_grads_func is not None and not forward_only:
embedding_module = clear_embedding_activation_buffer(
- config, model, is_pp_last_stage(p2p_communicator.pp_group)
+ config, model, p2p_communicator.is_pp_last_stage
)
if config.timers is not None:
@@ -2093,9 +2136,7 @@ def enable_grad_sync():
disable_grad_sync()
# Compute number of warmup microbatches.
- num_warmup_microbatches = (
- p2p_communicator.pp_group.size() - p2p_communicator.pp_group.rank() - 1
- )
+ num_warmup_microbatches = p2p_communicator.total_stages - p2p_communicator.current_stage - 1
num_warmup_microbatches = min(num_warmup_microbatches, num_microbatches)
num_microbatches_remaining = num_microbatches - num_warmup_microbatches
@@ -2111,9 +2152,15 @@ def enable_grad_sync():
if config.num_microbatches_with_partial_activation_checkpoints is not None:
max_outstanding_backprops = num_warmup_microbatches + 1
- model_type = get_model_type(model)
+ # Select backward function based on whether multi-module or single-module
+ if is_multimodule:
+ backward_func = partial(
+ backward_step_multimodule,
+ language_model_module_name=pg_collection.language_model_module_name,
+ )
+ else:
+ backward_func = backward_step
- rank = p2p_communicator.pp_group.rank()
recv_tensor_shapes = get_tensor_shapes(
seq_length=seq_length,
micro_batch_size=micro_batch_size,
@@ -2157,7 +2204,7 @@ def enable_grad_sync():
checkpoint_activations_microbatch = None
input_tensor = p2p_communicator.recv_forward(
- recv_tensor_shapes, is_pp_first_stage(p2p_communicator.pp_group)
+ recv_tensor_shapes, p2p_communicator.is_pp_first_stage
)
output_tensor, num_tokens = forward_step(
forward_step_func,
@@ -2167,27 +2214,27 @@ def enable_grad_sync():
input_tensor,
forward_data_store,
config,
- cp_group_size=pg_collection.cp.size(),
+ cp_group_size=cp_size,
collect_non_loss_data=collect_non_loss_data,
checkpoint_activations_microbatch=checkpoint_activations_microbatch,
is_first_microbatch=check_first_val_step(first_val_step, forward_only, i == 0),
current_microbatch=i,
- is_last_stage=is_pp_last_stage(p2p_communicator.pp_group),
+ is_last_stage=p2p_communicator.is_pp_last_stage,
)
- p2p_communicator.send_forward(output_tensor, is_pp_last_stage(p2p_communicator.pp_group))
+ p2p_communicator.send_forward(output_tensor, p2p_communicator.is_pp_last_stage)
total_num_tokens += num_tokens
if not forward_only:
input_tensors.append(input_tensor)
output_tensors.append(output_tensor)
- deallocate_output_tensor(output_tensor[0], config.deallocate_pipeline_outputs)
+ deallocate_output_tensor(output_tensor, config.deallocate_pipeline_outputs)
# Before running 1F1B, need to receive first forward tensor.
# If all microbatches are run in warmup / cooldown phase, then no need to
# receive this tensor here.
if num_microbatches_remaining > 0:
input_tensor = p2p_communicator.recv_forward(
- recv_tensor_shapes, is_pp_first_stage(p2p_communicator.pp_group)
+ recv_tensor_shapes, p2p_communicator.is_pp_first_stage
)
# Run 1F1B in steady state.
@@ -2210,34 +2257,32 @@ def enable_grad_sync():
input_tensor,
forward_data_store,
config,
- cp_group_size=pg_collection.cp.size(),
+ cp_group_size=cp_size,
collect_non_loss_data=collect_non_loss_data,
checkpoint_activations_microbatch=checkpoint_activations_microbatch,
is_first_microbatch=check_first_val_step(
first_val_step, forward_only, (i == 0) and (num_warmup_microbatches == 0)
),
current_microbatch=i + num_warmup_microbatches,
- is_last_stage=is_pp_last_stage(p2p_communicator.pp_group),
+ is_last_stage=p2p_communicator.is_pp_last_stage,
)
total_num_tokens += num_tokens
if forward_only:
- p2p_communicator.send_forward(
- output_tensor, is_pp_last_stage(p2p_communicator.pp_group)
- )
+ p2p_communicator.send_forward(output_tensor, p2p_communicator.is_pp_last_stage)
if not last_iteration:
input_tensor = p2p_communicator.recv_forward(
- recv_tensor_shapes, is_pp_first_stage(p2p_communicator.pp_group)
+ recv_tensor_shapes, p2p_communicator.is_pp_first_stage
)
else:
output_tensor_grad = p2p_communicator.send_forward_recv_backward(
- output_tensor, send_tensor_shapes, is_pp_last_stage(p2p_communicator.pp_group)
+ output_tensor, send_tensor_shapes, p2p_communicator.is_pp_last_stage
)
# Add input_tensor and output_tensor to end of list.
input_tensors.append(input_tensor)
output_tensors.append(output_tensor)
- deallocate_output_tensor(output_tensor[0], config.deallocate_pipeline_outputs)
+ deallocate_output_tensor(output_tensor, config.deallocate_pipeline_outputs)
# Pop input_tensor and output_tensor from the start of the list for
# the backward pass.
@@ -2247,23 +2292,21 @@ def enable_grad_sync():
# Enable grad sync for the last microbatch in the batch if the full
# backward pass completes in the 1F1B stage.
if num_warmup_microbatches == 0 and last_iteration:
- if config.grad_sync_func is None or rank == 0:
+ if config.grad_sync_func is None or p2p_communicator.is_pp_first_stage:
enable_grad_sync()
- input_tensor_grad = backward_step(
- input_tensor, output_tensor, output_tensor_grad, model_type, config
+ input_tensor_grad = backward_func(
+ input_tensor, output_tensor, output_tensor_grad, config
)
if last_iteration:
input_tensor = None
p2p_communicator.send_backward(
- input_tensor_grad, is_pp_first_stage(p2p_communicator.pp_group)
+ input_tensor_grad, p2p_communicator.is_pp_first_stage
)
else:
input_tensor = p2p_communicator.send_backward_recv_forward(
- input_tensor_grad,
- recv_tensor_shapes,
- is_pp_first_stage(p2p_communicator.pp_group),
+ input_tensor_grad, recv_tensor_shapes, p2p_communicator.is_pp_first_stage
)
# Run cooldown backward passes.
@@ -2276,23 +2319,21 @@ def enable_grad_sync():
# pipeline stages do grad reduction during pipeline
# bubble.
if i == num_warmup_microbatches - 1:
- if config.grad_sync_func is None or rank == 0:
+ if config.grad_sync_func is None or p2p_communicator.is_pp_first_stage:
enable_grad_sync()
input_tensor = input_tensors.pop(0)
output_tensor = output_tensors.pop(0)
output_tensor_grad = p2p_communicator.recv_backward(
- send_tensor_shapes, is_pp_last_stage(p2p_communicator.pp_group)
+ send_tensor_shapes, p2p_communicator.is_pp_last_stage
)
- input_tensor_grad = backward_step(
- input_tensor, output_tensor, output_tensor_grad, model_type, config
+ input_tensor_grad = backward_func(
+ input_tensor, output_tensor, output_tensor_grad, config
)
- p2p_communicator.send_backward(
- input_tensor_grad, is_pp_first_stage(p2p_communicator.pp_group)
- )
+ p2p_communicator.send_backward(input_tensor_grad, p2p_communicator.is_pp_first_stage)
# Launch any remaining grad reductions.
if no_sync_context is not None:
@@ -2305,7 +2346,7 @@ def enable_grad_sync():
# If defer_embedding_wgrad_compute is enabled we need to do the
# weight gradient GEMM's here.
finish_embedding_wgrad_compute(
- config, embedding_module, is_pp_last_stage(p2p_communicator.pp_group), tp_group
+ config, embedding_module, p2p_communicator.is_pp_last_stage, tp_group
)
# Finalize model grads (perform full grad all-reduce / reduce-scatter for
@@ -2318,7 +2359,7 @@ def enable_grad_sync():
force_all_reduce=force_all_reduce,
)
- if not forward_only and config.fine_grained_activation_offloading:
+ if getattr(config, 'fine_grained_activation_offloading', False):
off_interface.reset()
if config.timers is not None:
diff --git a/megatron/core/pipeline_parallel/utils.py b/megatron/core/pipeline_parallel/utils.py
index 03c5f01f443..8cb80741063 100644
--- a/megatron/core/pipeline_parallel/utils.py
+++ b/megatron/core/pipeline_parallel/utils.py
@@ -116,16 +116,6 @@ def set_ideal_affinity_for_current_gpu():
)
-@contextmanager
-def stream_acquire_context(stream, event):
- """Stream acquire context"""
- event.wait(stream)
- try:
- yield
- finally:
- event.record(stream)
-
-
class NoopScheduleNode:
"""A placeholder node in the computation graph that simply passes through inputs and outputs.
@@ -164,7 +154,7 @@ def __init__(
Args:
forward_func (callable): Function to execute during the forward pass.
- stream (torch.cuda.Stream): The CUDA stream for this node's computation.
+ stream (Callable): Func that returns CUDA stream for computation.
This can be either a 'compute' stream or a 'communicate' stream.
- 'compute' stream: Used for computational nodes like attention and experts.
- 'communicate' stream: Used for nodes that handle token communication,
@@ -208,26 +198,24 @@ def forward(self, inputs=()):
return self._forward(*inputs)
def _forward(self, *inputs):
- with stream_acquire_context(self.stream, self.event):
- torch.cuda.nvtx.range_push(f"{self.name} forward")
- with torch.cuda.stream(self.stream):
- self.inputs = [make_viewless(e).detach() if e is not None else None for e in inputs]
- for i, input in enumerate(self.inputs):
- if input is not None:
- input.requires_grad = inputs[i].requires_grad
+ # Lazy initialization of stream
+ if isinstance(self.stream, Callable):
+ self.stream = self.stream()
+ with self.stream_acquire_context(f"{self.name} forward"):
+ self.inputs = [make_viewless(e).detach() if e is not None else None for e in inputs]
+ for i, input in enumerate(self.inputs):
+ if input is not None:
+ input.requires_grad = inputs[i].requires_grad
- data = tuple(self.inputs)
- data = self.forward_func(*data)
+ data = tuple(self.inputs)
+ data = self.forward_func(*data)
- if not isinstance(data, tuple):
- data = make_viewless(data)
- else:
- data = tuple(
- [make_viewless(e) if isinstance(e, torch.Tensor) else e for e in data]
- )
+ if not isinstance(data, tuple):
+ data = make_viewless(data)
+ else:
+ data = tuple([make_viewless(e) if isinstance(e, torch.Tensor) else e for e in data])
- self.output = data
- torch.cuda.nvtx.range_pop()
+ self.output = data
# Immediately frees input tensors after they are used for nodes
# where inputs are no longer needed after computation.
@@ -250,18 +238,18 @@ def backward(self, output_grad):
return self._backward(*output_grad)
def _backward(self, *output_grad):
- with stream_acquire_context(self.stream, self.event):
- torch.cuda.nvtx.range_push(f"{self.name} backward")
- with torch.cuda.stream(self.stream):
- outputs = self.output
- if not isinstance(outputs, tuple):
- outputs = (outputs,)
- assert len(outputs) == len(output_grad), (
- f"{len(outputs)} of {type(outputs[0])} is not equal to "
- f"{len(output_grad)} of {type(output_grad[0])}"
- )
- output_grad = self.backward_func(outputs, output_grad)
- torch.cuda.nvtx.range_pop()
+ # Lazy initialization of stream
+ if isinstance(self.stream, Callable):
+ self.stream = self.stream()
+ with self.stream_acquire_context(f"{self.name} backward"):
+ outputs = self.output
+ if not isinstance(outputs, tuple):
+ outputs = (outputs,)
+ assert len(outputs) == len(output_grad), (
+ f"{len(outputs)} of {type(outputs[0])} is not equal to "
+ f"{len(output_grad)} of {type(output_grad[0])}"
+ )
+ output_grad = self.backward_func(outputs, output_grad)
# output_grad maybe from another stream
if output_grad:
@@ -288,6 +276,30 @@ def get_grad(self):
grad = grad[0]
return grad
+ @contextmanager
+ def stream_acquire_context(self, name=None):
+ """Stream acquire context that handles event synchronization,
+ NVTX profiling, and stream context.
+
+ This context manager consolidates:
+ 1. Event wait/record for synchronization between streams
+ 2. NVTX range for profiling (if name is provided)
+ 3. torch.cuda.stream context for execution on the specified stream
+
+ Args:
+ name: Optional name for NVTX range profiling
+ """
+ self.event.wait(self.stream)
+ if name:
+ torch.cuda.nvtx.range_push(name)
+ try:
+ with torch.cuda.stream(self.stream):
+ yield
+ finally:
+ if name:
+ torch.cuda.nvtx.range_pop()
+ self.event.record(self.stream)
+
def _release_state(self):
"""Clear the state of the node"""
self.inputs = None
@@ -317,32 +329,25 @@ def run(
...
+_USE_DYNAMIC_COMP_STREAM = None
_COMP_STREAM = None
_COMM_STREAM = None
-def set_streams(comp_stream=None, comm_stream=None):
- """Set the streams for communication and computation"""
- global _COMP_STREAM
+def set_streams(comm_stream=None):
+ """Set the stream for communication operations."""
global _COMM_STREAM
- if _COMP_STREAM is not None and _COMM_STREAM is not None:
- return
-
- if comp_stream is None:
- comp_stream = torch.cuda.current_stream()
- if comm_stream is None:
- comm_stream = torch.cuda.Stream(device="cuda")
- assert _COMP_STREAM is None
- assert _COMM_STREAM is None
- _COMP_STREAM = comp_stream
- _COMM_STREAM = comm_stream
+ # Set communication stream
+ if _COMM_STREAM is None:
+ if comm_stream is None:
+ comm_stream = torch.cuda.Stream(device="cuda")
+ _COMM_STREAM = comm_stream
def get_comp_stream():
"""Get the stream for computation"""
- global _COMP_STREAM
- return _COMP_STREAM
+ return torch.cuda.current_stream()
def get_comm_stream():
diff --git a/megatron/core/post_training/modelopt/layers.py b/megatron/core/post_training/modelopt/layers.py
index 0045f6ea9f4..9f08201b6ef 100644
--- a/megatron/core/post_training/modelopt/layers.py
+++ b/megatron/core/post_training/modelopt/layers.py
@@ -1,26 +1,28 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+from __future__ import annotations
import logging
-from typing import Callable, List, Optional
+from typing import TYPE_CHECKING, Callable, List, Optional, cast
import torch
-from megatron.core.extensions.transformer_engine import _get_extra_te_kwargs
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.model_parallel_config import ModelParallelConfig
+from megatron.core.transformer.torch_norm import LayerNormInterface
from megatron.core.transformer.transformer_config import TransformerConfig
from megatron.core.transformer.transformer_layer import TransformerLayer
from megatron.core.transformer.utils import make_sharded_tensors_for_checkpoint
+from megatron.core.typed_torch import copy_signature
logger = logging.getLogger(__name__)
-try:
- import transformer_engine as te
+if HAVE_TE or TYPE_CHECKING:
+ import transformer_engine as te # type: ignore[import]
- HAVE_TE = True
-except ImportError:
- HAVE_TE = False
-
-logger = logging.getLogger(__name__)
+ from megatron.core.extensions.transformer_engine import _get_extra_te_kwargs
+else:
+ te = None
+ _get_extra_te_kwargs = None
FP8_PER_TENSOR_REAL_QUANT_CFG = {
@@ -53,7 +55,9 @@ class Norm:
mismatch issue.
"""
- def __new__(cls, config: TransformerConfig, hidden_size: int, eps: float = 1e-5):
+ def __new__(
+ cls, config: TransformerConfig, hidden_size: int, eps: float = 1e-5
+ ) -> LayerNormInterface:
if not HAVE_TE:
raise ImportError(
"Transformer-Engine is not installed, please install it with "
@@ -95,7 +99,7 @@ def _load_state_dict_pre_hook(
instance._register_state_dict_hook(_state_dict_hook)
instance._register_load_state_dict_pre_hook(_load_state_dict_pre_hook)
- return instance
+ return cast(LayerNormInterface, instance)
class Linear(torch.nn.Linear):
@@ -192,6 +196,7 @@ class RealQuantTransformerLayer(TransformerLayer):
verbose: bool = False
real_quant_cfg: str = "None"
+ @copy_signature(TransformerLayer.__init__)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
diff --git a/megatron/core/post_training/modelopt/mamba/model_specs.py b/megatron/core/post_training/modelopt/mamba/model_specs.py
index 0a38d05b980..e9f83b49f71 100755
--- a/megatron/core/post_training/modelopt/mamba/model_specs.py
+++ b/megatron/core/post_training/modelopt/mamba/model_specs.py
@@ -18,9 +18,44 @@
# Use this spec for ModelOpt PTQ and TensorRT-LLM export
def get_mamba_stack_modelopt_spec(
+ local_core_attention: bool = False,
+ remap_te_layernorm: bool = False,
+ use_default_te_spec: bool = False,
+) -> ModuleSpec:
+ """Get the Mamba stack spec for ModelOpt PTQ and TensorRT-LLM export.
+
+ When use_default_te_spec=False (default), this is the native local spec with TENorm
+ from Transformer-Engine for the layernorm implementation (since FusedLayerNorm from
+ apex has stopped supporting RMSNorm needed by llama). The remap_te_layernorm flag
+ can be used to add sharded state_dict key remapping for TE-compatible checkpoint
+ saving/loading.
+
+ When use_default_te_spec=True, this returns the standard mamba_stack_spec from
+ mamba_layer_specs.py which uses full TE modules (TELayerNormColumnParallelLinear,
+ TERowParallelLinear, TEDotProductAttention, TENorm, moe_grouped_gemm=True).
+
+
+ Args:
+ local_core_attention: whether to use local DotProductAttention
+ (only for use_default_te_spec=False)
+ remap_te_layernorm: whether to perform sharded state_dict prefix mapping
+ on layernorm (only for use_default_te_spec=False)
+ use_default_te_spec: whether to use the default Transformer-Engine spec
+ """
+ if use_default_te_spec:
+ from megatron.core.models.mamba.mamba_layer_specs import mamba_stack_spec
+
+ return mamba_stack_spec
+
+ return _get_mamba_stack_local_spec(
+ local_core_attention=local_core_attention, remap_te_layernorm=remap_te_layernorm
+ )
+
+
+def _get_mamba_stack_local_spec(
local_core_attention: bool = False, remap_te_layernorm: bool = False
) -> ModuleSpec:
- """Mix the native spec with TENorm.
+ """Get the Mamba stack spec with local (non-TE) modules.
This is essentially the native local spec except for the layernorm implementation
is using TENorm from Transformer-Engine.
diff --git a/megatron/core/process_groups_config.py b/megatron/core/process_groups_config.py
index ef8f31ea150..a1afaa96513 100644
--- a/megatron/core/process_groups_config.py
+++ b/megatron/core/process_groups_config.py
@@ -4,7 +4,7 @@
from dataclasses import dataclass, field, fields
from functools import partial
-from typing import List, Optional
+from typing import Dict, List, Optional
import torch
@@ -569,3 +569,140 @@ def setup_process_groups_for_ddp(
result['ep_group'] = pg_collection.ep
return result
+
+
+@dataclass
+class MultiModuleProcessGroupCollection:
+ """Process group collection for multi-module pipelines.
+
+ Used when a rank participates in multiple modules (e.g., colocated encoder + LLM).
+ The language_model_module_name identifies which module is the language model (used for
+ CP size extraction, loss computation, and other LLM-specific operations).
+
+ Attributes:
+ module_pgs: Dict mapping module names to ProcessGroupCollection objects
+ language_model_module_name: Key identifying the language model module
+ (None if no LLM on this rank)
+
+ Example:
+ # Colocated rank with encoder and LLM
+ pg_collection = MultiModuleProcessGroupCollection(
+ module_pgs={"encoder": encoder_pg, "llm": llm_pg},
+ language_model_module_name="llm"
+ )
+
+ # Rank with dual encoders (no LLM)
+ pg_collection = MultiModuleProcessGroupCollection(
+ module_pgs={"encoder_1": encoder_1_pg, "encoder_2": encoder_2_pg},
+ language_model_module_name=None
+ )
+
+ # Single module (can also use ProcessGroupCollection directly)
+ pg_collection = MultiModuleProcessGroupCollection(
+ module_pgs={"llm": llm_pg},
+ language_model_module_name="llm"
+ )
+
+ # Usage
+ cp_size = pg_collection.get_language_model_cp_size()
+ encoder_pg = pg_collection["encoder_1"] # Dict-like access
+ has_llm = pg_collection.has_language_model()
+ """
+
+ module_pgs: Dict[str, ProcessGroupCollection]
+ language_model_module_name: Optional[str] = None
+
+ def __post_init__(self):
+ if not self.module_pgs:
+ raise ValueError("module_pgs dict cannot be empty")
+ if self.language_model_module_name is not None:
+ if self.language_model_module_name not in self.module_pgs:
+ raise ValueError(
+ f"language_model_module_name '{self.language_model_module_name}' not found in "
+ f"module_pgs keys: {list(self.module_pgs.keys())}"
+ )
+
+ def get_language_model_collection(self) -> ProcessGroupCollection:
+ """Get the language model's process group collection.
+
+ Returns:
+ ProcessGroupCollection for the language model.
+
+ Raises:
+ ValueError: If no language model is specified for this collection.
+ """
+ if self.language_model_module_name is None:
+ raise ValueError("No language model specified for this collection")
+ return self.module_pgs[self.language_model_module_name]
+
+ def get_language_model_cp_size(self) -> int:
+ """Get context parallel size for the language model.
+
+ Returns:
+ Context parallel size for the language model.
+
+ Raises:
+ ValueError: If no language model is specified for this collection.
+ """
+ return self.get_language_model_collection().cp.size()
+
+ def has_language_model(self) -> bool:
+ """Check if this rank has a language model.
+
+ Returns:
+ True if this rank has a language model, False otherwise.
+ """
+ return self.language_model_module_name is not None
+
+ def get_module_collection(self, module_name: str) -> ProcessGroupCollection:
+ """Get process group collection for a specific module.
+
+ Args:
+ module_name: Name of the module.
+
+ Returns:
+ ProcessGroupCollection for the specified module.
+
+ Raises:
+ ValueError: If module_name is not found in collections.
+ """
+ if module_name not in self.module_pgs:
+ raise ValueError(
+ f"Module '{module_name}' not found in collections. "
+ f"Available: {list(self.module_pgs.keys())}"
+ )
+ return self.module_pgs[module_name]
+
+ def __len__(self):
+ """Return the number of modules in this wrapper."""
+ return len(self.module_pgs)
+
+ def __getitem__(self, module_name: str):
+ """Get process group collection for a module using dict-like access."""
+ return self.module_pgs[module_name]
+
+ def __iter__(self):
+ """Iterate over all process group collections."""
+ return iter(self.module_pgs.values())
+
+ def keys(self):
+ """Return module names."""
+ return self.module_pgs.keys()
+
+ def values(self):
+ """Return process group collections."""
+ return self.module_pgs.values()
+
+ def items(self):
+ """Return (module_name, collection) pairs."""
+ return self.module_pgs.items()
+
+ def __repr__(self):
+ """Return a concise representation showing modules and their language model status."""
+ modules_str = ', '.join(self.module_pgs.keys())
+ lm_str = (
+ f", language_model_module_name='{self.language_model_module_name}'"
+ if self.language_model_module_name
+ else ""
+ )
+ return f"MultiModuleProcessGroupCollection(modules=[{modules_str}]{lm_str})"
diff --git a/megatron/core/rerun_state_machine.py b/megatron/core/rerun_state_machine.py
index 8fce2beaa85..cc0653e8a2a 100644
--- a/megatron/core/rerun_state_machine.py
+++ b/megatron/core/rerun_state_machine.py
@@ -16,6 +16,7 @@
from megatron.core._rank_utils import log_single_rank, safe_get_rank
from megatron.core.dist_checkpointing.mapping import ShardedObject
+from megatron.core.typed_torch import copy_signature
"""DISCLAIMER: THIS IS AN EXPERIMENTAL FEATURE.
@@ -191,6 +192,8 @@ def __init__(
) -> None:
self.mode: RerunMode = mode
self.state: RerunState = RerunState.NOT_RUNNING_YET
+ # Note: current_iteration is 0-indexed internally; all messages to
+ # stdout / stderr and the tracker file add 1 to display 1-indexed iterations.
self.current_iteration: int = -1
self.first_iteration_complete = False
# The flags below are per-rank flags that get all-reduced across all ranks
@@ -525,7 +528,7 @@ def train_step(data_iterator, ...):
device: int = torch.cuda.current_device()
full_message: str = (
f"Rank {rank}, node {node}, device {device}, "
- f"iteration {self.current_iteration}: "
+ f"iteration {self.current_iteration + 1}: "
f"Unexpected result {result} (message='{message}')"
)
if fatal:
@@ -568,12 +571,12 @@ def log_failure(message: str, fatal: bool = True) -> None:
if fatal:
logger.error(
f"Rank {rank}, node {node}, device {device}, "
- f"iteration #{self.current_iteration}: {message}!"
+ f"iteration #{self.current_iteration + 1}: {message}!"
)
else:
logger.warning(
f"Rank {rank}, node {node}, device {device}, "
- f"iteration #{self.current_iteration}: {message}!"
+ f"iteration #{self.current_iteration + 1}: {message}!"
)
# Emit message in log so that we can identify which jobs have this instrumentation
@@ -603,7 +606,7 @@ def log_failure(message: str, fatal: bool = True) -> None:
logger.error(
f"Unexpected result {result} "
f"on rank {safe_get_rank()} "
- f"at iteration #{self.current_iteration} "
+ f"at iteration #{self.current_iteration + 1} "
f"invocation #{validation_call.sequence} "
f"(message='{message}')"
)
@@ -1011,7 +1014,7 @@ def _log_validation_error_to_file(
f.write(
f"ts={datetime.datetime.now()} node={node} device={device} "
f"jobID={os.getenv('SLURM_JOBID', 'N/A')} rank={rank} "
- f"iteration={self.current_iteration} status={status} result={result} "
+ f"iteration={self.current_iteration + 1} status={status} result={result} "
f"message='{message}'\n"
)
except Exception as e:
@@ -1338,13 +1341,14 @@ def load_state_dict(self, state_dict: SerializableStateType) -> None:
self.injected_error_type = state_dict["injected_error_type"]
-def initialize_rerun_state_machine(**kwargs) -> None:
+@copy_signature(RerunStateMachine.__init__, handle_first_src_param='skip')
+def initialize_rerun_state_machine(*args, **kwargs) -> None:
"""Helper function to initialize the rerun machine instance.
Check the RerunStateMachine class for the details.
"""
- rerun_state_machine: RerunStateMachine = RerunStateMachine(**kwargs)
+ rerun_state_machine: RerunStateMachine = RerunStateMachine(*args, **kwargs)
_set_rerun_state_machine(rerun_state_machine)
diff --git a/megatron/core/resharding/__init__.py b/megatron/core/resharding/__init__.py
index 083c4518c0e..8c59b6ef809 100644
--- a/megatron/core/resharding/__init__.py
+++ b/megatron/core/resharding/__init__.py
@@ -7,11 +7,14 @@
reshard_model_weights,
swap_model_weights,
)
+from .transforms import MXFP8ReshardTransform, ReshardTransform
from .utils import ParameterMetadata, ReshardPlan, ShardingDescriptor, TransferOp
__all__ = [
"build_centralized_reshard_plan",
"execute_reshard_plan",
+ "MXFP8ReshardTransform",
+ "ReshardTransform",
"swap_model_weights",
"reshard_model_weights",
"get_or_create_service",
diff --git a/megatron/core/resharding/copy_services/gloo_copy_service.py b/megatron/core/resharding/copy_services/gloo_copy_service.py
index c9c83ca74a5..78e423b8221 100644
--- a/megatron/core/resharding/copy_services/gloo_copy_service.py
+++ b/megatron/core/resharding/copy_services/gloo_copy_service.py
@@ -37,10 +37,15 @@ class GlooCopyService(CopyService):
process group instead of NCCL.
"""
- def __init__(self):
- self.rank = dist.get_rank()
- self.world_size = dist.get_world_size()
- self.gloo_pg = dist.new_group(backend="gloo")
+ def __init__(self, group=None):
+ if group is not None:
+ self.gloo_pg = group
+ self.rank = group.rank()
+ self.world_size = group.size()
+ else:
+ self.rank = dist.get_rank()
+ self.world_size = dist.get_world_size()
+ self.gloo_pg = dist.new_group(backend="gloo")
self.send_ops: List[SendOp] = []
self.recv_ops: List[Tuple[RecvOp, torch.Tensor]] = []
self._copy_stream = torch.cuda.Stream()
@@ -58,16 +63,19 @@ def submit_send_with_id(self, task_id: int, src_tensor: torch.Tensor, dest_rank:
def submit_recv(self, dest_tensor: torch.Tensor, src_rank: int):
"""Submit a receive operation."""
- # Allocate a CPU buffer that matches the destination view; we'll
- # copy into dest_tensor after the Gloo recv completes.
- cpu_buffer = torch.empty_like(dest_tensor, device="cpu").contiguous()
+ # Allocate a pinned CPU buffer for faster CPU↔GPU transfer.
+ cpu_buffer = torch.empty(
+ dest_tensor.shape, dtype=dest_tensor.dtype, device="cpu", pin_memory=True
+ )
self.recv_ops.append(
(RecvOp(task_id=None, tensor=cpu_buffer, src_rank=src_rank), dest_tensor)
)
def submit_recv_with_id(self, task_id: int, dest_tensor: torch.Tensor, src_rank: int):
"""Submit a receive operation with a unique task identifier."""
- cpu_buffer = torch.empty_like(dest_tensor, device="cpu").contiguous()
+ cpu_buffer = torch.empty(
+ dest_tensor.shape, dtype=dest_tensor.dtype, device="cpu", pin_memory=True
+ )
self.recv_ops.append(
(RecvOp(task_id=task_id, tensor=cpu_buffer, src_rank=src_rank), dest_tensor)
)
@@ -122,13 +130,33 @@ def run(self):
else:
dst_tensor.copy_(src_tensor)
- # Build Gloo P2P ops over CPU tensors. For sends we clone to CPU;
- # for recvs we use the preallocated CPU buffers.
+ # Build Gloo P2P ops over CPU tensors. For sends we stage all
+ # GPU→CPU copies with non_blocking, sync once, then build P2P ops.
+ # Use group_peer (not peer) to pass ranks directly in group space,
+ # avoiding the global-to-group rank conversion in P2POp which doesn't
+ # work for cross-world ProcessGroups.
+ cpu_send_bufs: List[torch.Tensor] = []
+ for op in remote_sends:
+ cpu_tensor = torch.empty(
+ op.tensor.shape, dtype=op.tensor.dtype, device="cpu", pin_memory=True
+ )
+ cpu_tensor.copy_(op.tensor.detach(), non_blocking=True)
+ cpu_send_bufs.append(cpu_tensor)
+ # Single sync after all GPU→CPU copies are issued.
+ if cpu_send_bufs:
+ torch.cuda.synchronize()
+
for op in remote_sends:
- cpu_tensor = op.tensor.detach().to("cpu").contiguous()
- p2p_ops.append(dist.P2POp(dist.isend, cpu_tensor, op.dest_rank, group=self.gloo_pg))
+ op.tensor = None
+
+ for cpu_tensor, op in zip(cpu_send_bufs, remote_sends):
+ p2p_ops.append(
+ dist.P2POp(dist.isend, cpu_tensor, group=self.gloo_pg, group_peer=op.dest_rank)
+ )
for recv, _dst_tensor in remote_recvs:
- p2p_ops.append(dist.P2POp(dist.irecv, recv.tensor, recv.src_rank, group=self.gloo_pg))
+ p2p_ops.append(
+ dist.P2POp(dist.irecv, recv.tensor, group=self.gloo_pg, group_peer=recv.src_rank)
+ )
if p2p_ops:
reqs = dist.batch_isend_irecv(p2p_ops)
@@ -136,15 +164,19 @@ def run(self):
req.wait()
# Copy received CPU buffers back into the original destination tensors.
+ # Use non_blocking with pinned memory for overlap.
for recv, dst_tensor in remote_recvs:
if dst_tensor.is_cuda:
- dst_tensor.copy_(recv.tensor.to(dst_tensor.device))
+ dst_tensor.copy_(recv.tensor, non_blocking=True)
else:
dst_tensor.copy_(recv.tensor)
if self._copy_stream is not None:
torch.cuda.current_stream().wait_stream(self._copy_stream)
+ # Ensure all async CPU→GPU copies are complete.
+ torch.cuda.synchronize()
+
if self.rank == 0:
logger.info("GlooCopyService: batched communication completed")
self.send_ops.clear()
diff --git a/megatron/core/resharding/copy_services/nccl_copy_service.py b/megatron/core/resharding/copy_services/nccl_copy_service.py
index 8724279b991..dbf7e122ee3 100644
--- a/megatron/core/resharding/copy_services/nccl_copy_service.py
+++ b/megatron/core/resharding/copy_services/nccl_copy_service.py
@@ -37,9 +37,11 @@ class NCCLCopyService(CopyService):
a batch of point-to-point sends and recvs.
"""
- def __init__(self):
- self.rank = dist.get_rank()
- self.world_size = dist.get_world_size()
+ def __init__(self, group=None):
+ self.group = group
+ # Use group.rank()/size() to support cross-cluster ProcessGroups
+ self.rank = group.rank() if group is not None else dist.get_rank()
+ self.world_size = group.size() if group is not None else dist.get_world_size()
self.send_ops: List[SendOp] = []
self.recv_ops: List[RecvOp] = []
# Dedicated stream for local (same-rank) copies to avoid unnecessary
@@ -111,9 +113,9 @@ def run(self):
p2p_ops = []
for op in remote_sends:
- p2p_ops.append(dist.P2POp(dist.isend, op.tensor, op.dest_rank))
+ p2p_ops.append(dist.P2POp(dist.isend, op.tensor, op.dest_rank, group=self.group))
for op in remote_recvs:
- p2p_ops.append(dist.P2POp(dist.irecv, op.tensor, op.src_rank))
+ p2p_ops.append(dist.P2POp(dist.irecv, op.tensor, op.src_rank, group=self.group))
if p2p_ops:
reqs = dist.batch_isend_irecv(p2p_ops)
diff --git a/megatron/core/resharding/copy_services/nvshmem_copy_service.py b/megatron/core/resharding/copy_services/nvshmem_copy_service.py
index 8d231de5339..a0aea54676f 100644
--- a/megatron/core/resharding/copy_services/nvshmem_copy_service.py
+++ b/megatron/core/resharding/copy_services/nvshmem_copy_service.py
@@ -17,12 +17,13 @@
class NVSHMEMCopyService(CopyService):
"""CopyService implementation backed by NVSHMEM RemoteCopyService."""
- def __init__(self):
+ def __init__(self, group=None):
if not dist.is_initialized():
raise RuntimeError("torch.distributed must be initialized before NVSHMEMCopyService()")
- self.rank = dist.get_rank()
- self._remote = RemoteCopyService()
+ self._group = group
+ self.rank = group.rank() if group is not None else dist.get_rank()
+ self._remote = RemoteCopyService(group=group)
# Lazily initialized on first use to avoid side effects at import time
self._initialized = False
@@ -162,10 +163,11 @@ def run(self):
self._local_recv_ops.clear()
# 2) Execute remote schedule (if any remote sends/recvs were registered).
- if not self._remote.send_requests and not self._remote.receive_requests:
- logger.info("NVSHMEMCopyService: no remote requests; local copies complete")
- return
-
+ # NOTE: ALL ranks must call schedule() and run() because they contain collective
+ # operations that require all ranks to participate:
+ # - schedule() has dist.all_gather_object() (torch distributed collective)
+ # - run() has nvshmem.core.barrier_all() (nvshmem collective)
+ # This is critical for non-collocated refit where some ranks may have no work.
logger.info("NVSHMEMCopyService: building NVSHMEM schedule and executing")
self._remote.schedule()
self._remote.run()
diff --git a/megatron/core/resharding/execution.py b/megatron/core/resharding/execution.py
index 6a7779406d0..f441cea31bd 100644
--- a/megatron/core/resharding/execution.py
+++ b/megatron/core/resharding/execution.py
@@ -2,65 +2,197 @@
from __future__ import annotations
import logging
-from typing import List, Tuple
+from typing import Optional
import torch
import torch.distributed as dist
from .copy_services.base import CopyService
+from .transforms import ReshardTransform, _ensure_sendable
from .utils import ReshardPlan
logger = logging.getLogger(__name__)
+def _is_mxfp8_tensor(param):
+ """Check if param is a TE MXFP8Tensor (fp8_param=true)."""
+ return (
+ hasattr(param, 'quantize_')
+ and hasattr(param, 'dequantize')
+ and hasattr(param, '_rowwise_data')
+ )
+
+
def execute_reshard_plan(
plan: ReshardPlan,
src_module: torch.nn.Module,
dst_module: torch.nn.Module,
service: CopyService,
+ group=None,
+ transform: Optional[ReshardTransform] = None,
) -> None:
"""
Execute a reshard plan (from centralized controller).
A communication service must be provided to abstract transport.
Expected service API: submit_send(tensor, dest_rank), submit_recv(tensor, src_rank), run().
+
+ Supports None for src_module and/or dst_module to allow ranks in non-collocated mode:
+ - src_module=None: Rank only receives data (destination-only)
+ - dst_module=None: Rank only sends data (source-only)
+ - Both provided: Rank participates in both send and recv (collocated mode)
+
+ When *transform* is provided, parameters for which
+ ``transform.should_transform(param_name)`` returns True use the
+ transform's prepare_send / prepare_recv / finalize_recv methods instead
+ of the default slice-and-copy logic.
"""
- src_params = {name: p for name, p in src_module.named_parameters(recurse=True)}
- dst_params = {name: p for name, p in dst_module.named_parameters(recurse=True)}
+ # Extract parameters from models if present
+ src_params = {}
+ dst_params = {}
+ if src_module is not None:
+ src_params = {name: p for name, p in src_module.named_parameters(recurse=True)}
+ if dst_module is not None:
+ dst_params = {name: p for name, p in dst_module.named_parameters(recurse=True)}
+
submit_send_with_id = getattr(service, "submit_send_with_id", None)
submit_recv_with_id = getattr(service, "submit_recv_with_id", None)
- # Submit sends
+ # Cache dequantized BF16 views of MXFP8 source params so that multiple
+ # send ops for the same param reuse one dequant instead of repeating it.
+ _sendable_cache: dict[str, torch.Tensor] = {}
+
+ def _get_sendable(param_name: str, param: torch.nn.Parameter) -> torch.Tensor:
+ if param_name not in _sendable_cache:
+ _sendable_cache[param_name] = _ensure_sendable(param)
+ return _sendable_cache[param_name]
+
+ # Submit sends (only if we have source model)
for op in plan.send_ops:
- src_param = src_params.get(op.param_name)
- if src_param is not None:
- src_view = src_param.data[op.my_slice].contiguous()
- if submit_send_with_id is not None and op.task_id is not None:
- submit_send_with_id(op.task_id, src_view, op.peer_rank)
- else:
- service.submit_send(src_view, op.peer_rank)
+ if transform is not None and transform.should_transform(op.param_name):
+ src_param = src_params.get(op.param_name)
+ if src_param is not None:
+ tensors = transform.prepare_send(op.param_name, op.my_slice, src_param)
+ for t in tensors:
+ buf = t.contiguous()
+ if submit_send_with_id is not None and op.task_id is not None:
+ submit_send_with_id(op.task_id, buf, op.peer_rank)
+ else:
+ service.submit_send(buf, op.peer_rank)
+ else:
+ src_param = src_params.get(op.param_name)
+ if src_param is not None:
+ sendable = _get_sendable(op.param_name, src_param)
+ src_view = sendable[op.my_slice]
+ # Only copy if the slice is non-contiguous.
+ if not src_view.is_contiguous():
+ src_view = src_view.contiguous()
+ if submit_send_with_id is not None and op.task_id is not None:
+ submit_send_with_id(op.task_id, src_view, op.peer_rank)
+ else:
+ service.submit_send(src_view, op.peer_rank)
+
+ # Free the dequant cache — slices have been submitted and the service
+ # holds its own references to the contiguous buffers it needs.
+ _sendable_cache.clear()
+
+ # Submit recvs (only if we have destination model)
+ # Writebacks: each entry is one of:
+ # ('direct',) — recv'd in-place, no writeback
+ # ('default', recv_buffer, dst_param, dst_slice) — copy recv_buffer → dst_param
+ # ('transform', param_name, dst_slice, [recv_bufs]) — transform.finalize_recv
+ recv_writebacks: list = []
- # Submit recvs
- recv_writebacks: List[Tuple[torch.Tensor, torch.nn.Parameter, tuple[slice, ...]]] = []
for op in plan.recv_ops:
- dst_param = dst_params.get(op.param_name)
- if dst_param is not None:
- dst_slice_view = dst_param.data[op.my_slice]
- recv_buffer = torch.empty_like(dst_slice_view.contiguous())
- if submit_recv_with_id is not None and op.task_id is not None:
- submit_recv_with_id(op.task_id, recv_buffer, op.peer_rank)
- else:
- service.submit_recv(recv_buffer, op.peer_rank)
- recv_writebacks.append((recv_buffer, dst_param, op.my_slice))
+ if transform is not None and transform.should_transform(op.param_name):
+ recv_bufs = transform.prepare_recv(op.param_name, op.my_slice)
+ for buf in recv_bufs:
+ if submit_recv_with_id is not None and op.task_id is not None:
+ submit_recv_with_id(op.task_id, buf, op.peer_rank)
+ else:
+ service.submit_recv(buf, op.peer_rank)
+ recv_writebacks.append(('transform', op.param_name, op.my_slice, recv_bufs))
+ else:
+ dst_param = dst_params.get(op.param_name)
+ if dst_param is not None:
+ # Try to recv directly into the destination parameter slice to
+ # avoid allocating a separate buffer + a writeback copy. This
+ # is safe when the slice view is already contiguous AND the
+ # parameter is a plain tensor (not quantized — quantized params
+ # need deferred accumulation).
+ dst_slice_view = dst_param.data[op.my_slice]
+ if dst_slice_view.is_contiguous() and not _is_mxfp8_tensor(dst_param):
+ # Recv directly into destination — no writeback needed.
+ if submit_recv_with_id is not None and op.task_id is not None:
+ submit_recv_with_id(op.task_id, dst_slice_view, op.peer_rank)
+ else:
+ service.submit_recv(dst_slice_view, op.peer_rank)
+ recv_writebacks.append(('direct',))
+ else:
+ recv_buffer = torch.empty_like(dst_slice_view.contiguous())
+ if submit_recv_with_id is not None and op.task_id is not None:
+ submit_recv_with_id(op.task_id, recv_buffer, op.peer_rank)
+ else:
+ service.submit_recv(recv_buffer, op.peer_rank)
+ recv_writebacks.append(('default', recv_buffer, dst_param, op.my_slice))
# Execute
logger.info(f"Executing {len(plan.send_ops)} sends + {len(plan.recv_ops)} recvs")
service.run()
- dist.barrier()
+ torch.cuda.synchronize()
+ dist.barrier(group=group)
+
+ # Write back received buffers into their destination parameter slices.
+ #
+ # For quantized destination params (fp8_param=true on receiver),
+ # accumulate ALL BF16 slices per-param before calling quantize_() once.
+ # This avoids corrupting MXFP8 per-block scales from partial-slice updates.
+ #
+ # Since refit overwrites every slice of each param, we allocate a fresh
+ # BF16 buffer (torch.empty) instead of dequantizing the existing MXFP8
+ # weights — this avoids a full-model-sized dequantize+clone.
+ pending_quantized: dict[int, tuple[torch.nn.Parameter, torch.Tensor, list]] = {}
+
+ for wb in recv_writebacks:
+ with torch.no_grad():
+ if wb[0] == 'direct':
+ # Already written in-place during recv — nothing to do.
+ pass
+ elif wb[0] == 'transform':
+ _, param_name, dst_slice, recv_bufs = wb
+ transform.finalize_recv(param_name, dst_slice, recv_bufs)
+ else:
+ _, recv_buffer, dst_param, dst_slice = wb
+ if _is_mxfp8_tensor(dst_param):
+ # Accumulate BF16 slices for deferred quantization
+ param_id = id(dst_param)
+ if param_id not in pending_quantized:
+ # Allocate empty BF16 buffer — no need to dequantize
+ # existing weights since all slices will be overwritten.
+ full_bf16 = torch.empty(
+ dst_param.shape, dtype=torch.bfloat16, device=dst_param.device
+ )
+ pending_quantized[param_id] = (dst_param, full_bf16, [])
+ pending_quantized[param_id][2].append((dst_slice, recv_buffer))
+ pending_quantized[param_id][1][dst_slice].copy_(recv_buffer)
+ else:
+ dst_param.data[dst_slice].copy_(recv_buffer)
- # Write back received buffers into their destination parameter slices
- for recv_buffer, dst_param, dst_slice in recv_writebacks:
+ # Free writeback list — recv_buffers are no longer needed after copy.
+ recv_writebacks.clear()
+
+ # Finalize deferred quantized param updates
+ for param_id, (dst_param, full_bf16, slices) in pending_quantized.items():
with torch.no_grad():
- dst_param.data[dst_slice].copy_(recv_buffer)
+ dst_param.quantize_(full_bf16)
+ # Free the BF16 accumulation buffers eagerly.
+ pending_quantized.clear()
+
+ # Ensure all writeback copies are visible to subsequent CUDA ops (e.g. CUDA
+ # graph warmup). The synchronize() above fires *before* the writeback loop,
+ # so without this second sync the .copy_() kernels are still async when
+ # execute_reshard_plan returns — creating a race with callers that immediately
+ # inspect or capture (via CUDA graphs) the destination parameters.
+ torch.cuda.synchronize()
logger.info("Reshard complete")
diff --git a/megatron/core/resharding/nvshmem_copy_service/__init__.py b/megatron/core/resharding/nvshmem_copy_service/__init__.py
index 2ab8cde81fe..00c1d808a68 100644
--- a/megatron/core/resharding/nvshmem_copy_service/__init__.py
+++ b/megatron/core/resharding/nvshmem_copy_service/__init__.py
@@ -8,6 +8,10 @@
can use it without relying on an external library.
"""
+from .compat import ensure_nvshmem_compat
+
+ensure_nvshmem_compat()
+
from . import nvshmem_types
from .core import GPUResourceManager, KernelLauncher, PipelineExecutor
from .memory import DoubleBufferManager, TensorPointerExtractor
diff --git a/megatron/core/resharding/nvshmem_copy_service/compat.py b/megatron/core/resharding/nvshmem_copy_service/compat.py
new file mode 100644
index 00000000000..be624c3ba26
--- /dev/null
+++ b/megatron/core/resharding/nvshmem_copy_service/compat.py
@@ -0,0 +1,49 @@
+# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
+
+"""
+Compatibility layer for cuda-core version differences.
+
+cuda-core >=0.5 removed the ``cuda.core.experimental._memory`` and
+``cuda.core.experimental._stream`` private submodules, but nvshmem4py
+still imports from them. We register ``sys.modules`` shims so those
+imports resolve to the new ``cuda.core._memory`` / ``cuda.core._stream``
+paths.
+
+This module should be imported before any nvshmem.core usage.
+"""
+
+import importlib
+import sys
+
+
+def _patch_cuda_core_experimental():
+ """Register cuda.core._memory / _stream as cuda.core.experimental._memory / _stream."""
+ for submod in ("_memory", "_stream"):
+ exp_key = f"cuda.core.experimental.{submod}"
+ new_key = f"cuda.core.{submod}"
+ if exp_key not in sys.modules:
+ try:
+ sys.modules[exp_key] = importlib.import_module(new_key)
+ except ImportError:
+ pass # old cuda-core that still has experimental._memory
+
+
+def get_cuda_core_device_class():
+ """Return the ``Device`` class from whichever cuda-core location is available.
+
+ cuda-core <0.5: ``cuda.core.experimental.Device``
+ cuda-core >=0.5: ``cuda.core.Device``
+ """
+ try:
+ from cuda.core import Device
+
+ return Device
+ except ImportError:
+ from cuda.core.experimental import Device
+
+ return Device
+
+
+def ensure_nvshmem_compat():
+ """Apply all compatibility patches. Safe to call multiple times."""
+ _patch_cuda_core_experimental()
diff --git a/megatron/core/resharding/nvshmem_copy_service/core/gpu_resource_manager.py b/megatron/core/resharding/nvshmem_copy_service/core/gpu_resource_manager.py
index 6e03b914b26..db6a7743e7c 100644
--- a/megatron/core/resharding/nvshmem_copy_service/core/gpu_resource_manager.py
+++ b/megatron/core/resharding/nvshmem_copy_service/core/gpu_resource_manager.py
@@ -10,9 +10,14 @@
import logging
from typing import Dict, Optional
+from ..compat import ensure_nvshmem_compat, get_cuda_core_device_class
+
+ensure_nvshmem_compat()
+
try:
import nvshmem.core
- from cuda.core.experimental import Device
+
+ Device = get_cuda_core_device_class()
HAVE_NVSHMEM = True
except ImportError:
@@ -40,19 +45,23 @@ def __init__(self):
self.copy_stream = None
# PyTorch stream wrappers
- self.torch_pack_stream = None
- self.torch_unpack_stream = None
- self.torch_send_stream = None
- self.torch_copy_stream = None
+ self.torch_pack_stream_wrapper = None
+ self.torch_unpack_stream_wrapper = None
+ self.torch_send_stream_wrapper = None
+ self.torch_copy_stream_wrapper = None
# Stream name to PyTorch stream mapping
self._torch_streams: Dict[str, torch.cuda.ExternalStream] = {}
- def init(self) -> None:
+ def init(self, group=None) -> None:
"""
Initialize NVSHMEM, CUDA device, and streams.
Expects torch.distributed to be already initialized.
+
+ Args:
+ group: Optional ProcessGroup for distributed operations.
+ If None, uses the default process group.
"""
if self.initialized:
return
@@ -75,9 +84,12 @@ def init(self) -> None:
self.device = Device(local_rank)
self.device.set_current()
- # Extract rank, nranks from the default process group
- num_ranks = dist.get_world_size()
- rank_id = dist.get_rank()
+ # Extract rank, nranks from the process group.
+ # Use group.rank()/size() instead of dist.get_rank(group=) because
+ # dist.get_rank(group=) maps via the default PG rank, which is wrong
+ # for cross-cluster ProcessGroups where workers share default PG rank 0.
+ num_ranks = group.size() if group is not None else dist.get_world_size()
+ rank_id = group.rank() if group is not None else dist.get_rank()
# Create/Broadcast UniqueID using broadcast_object_list
uniqueid = nvshmem.core.get_unique_id(empty=True)
@@ -87,11 +99,11 @@ def init(self) -> None:
else:
broadcast_objects = [None]
- # Broadcast ID to all ranks using the default group
- dist.broadcast_object_list(broadcast_objects, src=0)
+ # Broadcast ID to all ranks
+ dist.broadcast_object_list(broadcast_objects, src=0, group=group)
# Barrier to ensure everyone has the ID before NVSHMEM init
- dist.barrier()
+ dist.barrier(group=group)
# Initialize NVSHMEM with the broadcasted UID
nvshmem.core.init(
@@ -119,17 +131,17 @@ def init(self) -> None:
_, send_stream_ptr = self.send_stream.__cuda_stream__()
_, copy_stream_ptr = self.copy_stream.__cuda_stream__()
- self.torch_pack_stream = torch.cuda.ExternalStream(pack_stream_ptr)
- self.torch_unpack_stream = torch.cuda.ExternalStream(unpack_stream_ptr)
- self.torch_send_stream = torch.cuda.ExternalStream(send_stream_ptr)
- self.torch_copy_stream = torch.cuda.ExternalStream(copy_stream_ptr)
+ self.torch_pack_stream_wrapper = torch.cuda.ExternalStream(pack_stream_ptr)
+ self.torch_unpack_stream_wrapper = torch.cuda.ExternalStream(unpack_stream_ptr)
+ self.torch_send_stream_wrapper = torch.cuda.ExternalStream(send_stream_ptr)
+ self.torch_copy_stream_wrapper = torch.cuda.ExternalStream(copy_stream_ptr)
# Build stream mapping
self._torch_streams = {
- "pack": self.torch_pack_stream,
- "unpack": self.torch_unpack_stream,
- "send": self.torch_send_stream,
- "copy": self.torch_copy_stream,
+ "pack": self.torch_pack_stream_wrapper,
+ "unpack": self.torch_unpack_stream_wrapper,
+ "send": self.torch_send_stream_wrapper,
+ "copy": self.torch_copy_stream_wrapper,
}
logger.info("Stream mapping built")
@@ -171,18 +183,19 @@ def get_torch_stream(self, name: str) -> Optional[torch.cuda.ExternalStream]:
def create_events(self, num_events: int = 2):
"""
- Create double-buffered CUDA events for pack and unpack operations.
+ Create double-buffered CUDA events for pack, unpack, and barrier operations.
Args:
num_events: Number of events to create for each type
(default: 2 for double buffering)
Returns:
- tuple: (pack_events, unpack_events) lists of torch.cuda.Event
+ tuple: (pack_events, unpack_events, barrier_events) lists of torch.cuda.Event
"""
pack_events = [torch.cuda.Event(enable_timing=False) for _ in range(num_events)]
unpack_events = [torch.cuda.Event(enable_timing=False) for _ in range(num_events)]
- return pack_events, unpack_events
+ barrier_events = [torch.cuda.Event(enable_timing=False) for _ in range(num_events)]
+ return pack_events, unpack_events, barrier_events
def finalize(self) -> None:
"""Cleanup resources (streams are automatically managed by CUDA)."""
diff --git a/megatron/core/resharding/nvshmem_copy_service/core/kernel_launcher.py b/megatron/core/resharding/nvshmem_copy_service/core/kernel_launcher.py
index 4e86d6a9505..1e49451741c 100644
--- a/megatron/core/resharding/nvshmem_copy_service/core/kernel_launcher.py
+++ b/megatron/core/resharding/nvshmem_copy_service/core/kernel_launcher.py
@@ -64,7 +64,7 @@ def launch_pack(
self,
gpu_plan: Tuple[Any, Any, Any, int],
pack_stream,
- torch_pack_stream: torch.cuda.ExternalStream,
+ torch_pack_stream_wrapper: torch.cuda.ExternalStream,
pack_event: torch.cuda.Event,
) -> None:
"""
@@ -75,7 +75,7 @@ def launch_pack(
as CuPy arrays
pack_stream: CUDA stream (cuda.core.experimental.Stream) - unused,
kept for compatibility
- torch_pack_stream: PyTorch external stream wrapper
+ torch_pack_stream_wrapper: PyTorch external stream wrapper
pack_event: CUDA event to record after kernel launch
"""
nvtx.range_push("Launch Pack Kernel")
@@ -101,13 +101,13 @@ def launch_pack(
)
nvtx.range_pop()
# Record event on PyTorch stream
- pack_event.record(stream=torch_pack_stream)
+ pack_event.record(stream=torch_pack_stream_wrapper)
def launch_unpack(
self,
gpu_plan: Tuple[Any, Any, Any, int],
unpack_stream,
- torch_unpack_stream: torch.cuda.ExternalStream,
+ torch_unpack_stream_wrapper: torch.cuda.ExternalStream,
unpack_event: torch.cuda.Event,
) -> None:
"""
@@ -118,7 +118,7 @@ def launch_unpack(
as CuPy arrays
unpack_stream: CUDA stream (cuda.core.experimental.Stream) - unused,
kept for compatibility
- torch_unpack_stream: PyTorch external stream wrapper
+ torch_unpack_stream_wrapper: PyTorch external stream wrapper
unpack_event: CUDA event to record after kernel launch
"""
nvtx.range_push("Launch Unpack Kernel")
@@ -144,4 +144,4 @@ def launch_unpack(
)
nvtx.range_pop()
# Record event on PyTorch stream
- unpack_event.record(stream=torch_unpack_stream)
+ unpack_event.record(stream=torch_unpack_stream_wrapper)
diff --git a/megatron/core/resharding/nvshmem_copy_service/core/pipeline_executor.py b/megatron/core/resharding/nvshmem_copy_service/core/pipeline_executor.py
index 5ba07f9956a..6e78c55d98b 100644
--- a/megatron/core/resharding/nvshmem_copy_service/core/pipeline_executor.py
+++ b/megatron/core/resharding/nvshmem_copy_service/core/pipeline_executor.py
@@ -9,6 +9,10 @@
from typing import Dict, List, Optional
+from ..compat import ensure_nvshmem_compat
+
+ensure_nvshmem_compat()
+
try:
import nvshmem.core
@@ -48,13 +52,15 @@ def __init__(
self.send_stream = None
self.copy_stream = None
- self.torch_pack_stream = None
- self.torch_unpack_stream = None
- self.torch_copy_stream = None
+ self.torch_pack_stream_wrapper = None
+ self.torch_unpack_stream_wrapper = None
+ self.torch_send_stream_wrapper = None
+ self.torch_copy_stream_wrapper = None
# Events for double-buffered synchronization
self.pack_events = []
self.unpack_events = []
+ self.barrier_events = []
def set_streams(
self,
@@ -62,9 +68,10 @@ def set_streams(
unpack_stream,
send_stream,
copy_stream,
- torch_pack_stream,
- torch_unpack_stream,
- torch_copy_stream,
+ torch_pack_stream_wrapper,
+ torch_unpack_stream_wrapper,
+ torch_send_stream_wrapper,
+ torch_copy_stream_wrapper,
):
"""Set CUDA streams for execution."""
self.pack_stream = pack_stream
@@ -72,14 +79,16 @@ def set_streams(
self.send_stream = send_stream
self.copy_stream = copy_stream
- self.torch_pack_stream = torch_pack_stream
- self.torch_unpack_stream = torch_unpack_stream
- self.torch_copy_stream = torch_copy_stream
+ self.torch_pack_stream_wrapper = torch_pack_stream_wrapper
+ self.torch_unpack_stream_wrapper = torch_unpack_stream_wrapper
+ self.torch_send_stream_wrapper = torch_send_stream_wrapper
+ self.torch_copy_stream_wrapper = torch_copy_stream_wrapper
- def set_events(self, pack_events: List, unpack_events: List):
+ def set_events(self, pack_events: List, unpack_events: List, barrier_events: List):
"""Set double-buffered CUDA events."""
self.pack_events = pack_events
self.unpack_events = unpack_events
+ self.barrier_events = barrier_events
def execute_pipeline(
self, iter_schedules: List[Dict[str, Optional[ScheduledBatch]]], num_iterations: int
@@ -90,22 +99,28 @@ def execute_pipeline(
Pipeline stages:
1. Pack NEXT iteration (async)
2. Unpack PRIOR iteration (async)
- 3. Send CURRENT iteration (sync)
- 4. Barrier
+ 3. Send CURRENT iteration
+ 4. Barrier + record barrier event
5. Wait for async pack/unpack to complete
+ Cross-stream synchronization uses lightweight CUDA events instead of
+ cudaDeviceSynchronize (torch.cuda.synchronize). The pack kernel includes
+ __threadfence_system() to ensure writes are visible to the NIC's DMA
+ engine, and barrier_events propagate NVSHMEM RDMA completion from
+ send_stream to unpack_stream.
+
Args:
iter_schedules: List of iteration schedules
num_iterations: Total number of iterations
"""
PELogger.info(f"Executing pipeline: {num_iterations} iterations")
- # Priming: Pack iteration 0 and WAIT for completion
+ # Priming: Pack iteration 0 (async, no CPU sync needed —
+ # step 3 uses GPU-level event wait for pack→put ordering)
if num_iterations > 0 and iter_schedules[0]["send"]:
torch.cuda.nvtx.range_push("Priming")
PELogger.debug("Priming: Packing iteration 0")
self._launch_pack(0, iter_schedules[0]["send"])
- self.pack_events[0].synchronize()
torch.cuda.nvtx.range_pop()
for i in range(num_iterations):
@@ -153,6 +168,9 @@ def execute_pipeline(
f" Unpack prior (iter {i-1}): {prior_batch.total_size} bytes "
f"← PE {prior_batch.src_pe}"
)
+ # GPU-level event wait: ensures send_stream's barrier_all from
+ # the prior iteration has completed before unpack_stream proceeds.
+ self.torch_unpack_stream_wrapper.wait_event(self.barrier_events[(i - 1) % 2])
self._launch_unpack(i - 1, prior_batch)
torch.cuda.nvtx.range_pop()
@@ -164,6 +182,12 @@ def execute_pipeline(
transfer_size = batch.total_size
PELogger.debug(f" Send current: {transfer_size} bytes → PE {batch.dest_pe}")
+ # GPU-level event wait: ensures pack data in send_slot is visible
+ # to send_stream before NVSHMEM put reads it. The pack kernel's
+ # __threadfence_system() guarantees the writes are also visible to
+ # the NIC's DMA engine.
+ self.torch_send_stream_wrapper.wait_event(self.pack_events[slot])
+
nvshmem.core.put(
self.buffer_manager.recv_slots[slot][0:transfer_size],
self.buffer_manager.send_slots[slot][0:transfer_size],
@@ -172,20 +196,29 @@ def execute_pipeline(
)
torch.cuda.nvtx.range_pop()
- # Ensure send completes
- self.send_stream.sync()
+ # Step 4a: Wait for prior unpack to complete BEFORE the barrier.
+ torch.cuda.nvtx.range_push("Step 4a: Wait Unpack")
+ if has_prior_recv:
+ self.unpack_events[(i - 1) % 2].synchronize()
+ torch.cuda.nvtx.range_pop()
+
+ # Ensure all NVSHMEM operations on send_stream complete (stream-ordered)
nvshmem.core.quiet(stream=self.send_stream)
- # Step 4: Global barrier
- torch.cuda.nvtx.range_push("Step 4: Barrier")
+ # Step 4b: Global barrier + CPU sync + record event
+ torch.cuda.nvtx.range_push("Step 4b: Barrier")
nvshmem.core.barrier_all(stream=self.send_stream)
- self.send_stream.sync()
+ # CPU-sync the send_stream to ensure barrier_all has actually
+ # completed (not just submitted). Without this, the barrier_event
+ # can fire before RDMA data from the remote PE is visible, because
+ # stream-ordered operations are only guaranteed to be submitted,
+ # not completed, when the event is recorded.
+ self.torch_send_stream_wrapper.synchronize()
+ self.barrier_events[slot].record(stream=self.torch_send_stream_wrapper)
torch.cuda.nvtx.range_pop()
- # Step 5: Wait for async pack/unpack to complete
- torch.cuda.nvtx.range_push("Step 5: Wait Async")
- if has_prior_recv:
- self.unpack_events[(i - 1) % 2].synchronize()
+ # Step 5: Wait for async pack to complete (double-buffer safety)
+ torch.cuda.nvtx.range_push("Step 5: Wait Pack")
if has_next_send:
self.pack_events[(i + 1) % 2].synchronize()
torch.cuda.nvtx.range_pop()
@@ -198,6 +231,10 @@ def execute_pipeline(
PELogger.debug(f"Final unpack: iteration {num_iterations-1}")
last_recv = iter_schedules[num_iterations - 1]["recv"]
assert last_recv is not None
+ # GPU-level event wait for NVSHMEM RDMA data visibility
+ self.torch_unpack_stream_wrapper.wait_event(
+ self.barrier_events[(num_iterations - 1) % 2]
+ )
self._launch_unpack(num_iterations - 1, last_recv)
self.unpack_events[(num_iterations - 1) % 2].synchronize()
torch.cuda.nvtx.range_pop()
@@ -212,7 +249,7 @@ def _launch_pack(self, iteration: int, batch: ScheduledBatch) -> None:
self.kernel_launcher.launch_pack(
batch.gpu_plan,
self.pack_stream,
- self.torch_pack_stream,
+ self.torch_pack_stream_wrapper,
self.pack_events[iteration % 2],
)
@@ -224,7 +261,7 @@ def _launch_unpack(self, iteration: int, batch: ScheduledBatch) -> None:
self.kernel_launcher.launch_unpack(
batch.gpu_plan,
self.unpack_stream,
- self.torch_unpack_stream,
+ self.torch_unpack_stream_wrapper,
self.unpack_events[iteration % 2],
)
@@ -248,7 +285,7 @@ def process_self_moves(
PELogger.debug(f"Processing {len(local_recvs)} self-moves")
num_processed = 0
- with torch.cuda.stream(self.torch_copy_stream):
+ with torch.cuda.stream(self.torch_copy_stream_wrapper):
for recv_req in local_recvs:
if recv_req.task_id in local_sends:
send_req = local_sends[recv_req.task_id]
@@ -269,7 +306,7 @@ def process_self_moves(
num_processed += 1
# Synchronize the PyTorch stream
- self.torch_copy_stream.synchronize()
+ self.torch_copy_stream_wrapper.synchronize()
if num_processed > 0:
PELogger.info("Self-moves complete: %d transfers", num_processed)
diff --git a/megatron/core/resharding/nvshmem_copy_service/kernels/chunked_kernel.cu b/megatron/core/resharding/nvshmem_copy_service/kernels/chunked_kernel.cu
index e5b8fcc9a85..8b46c4f4ac7 100644
--- a/megatron/core/resharding/nvshmem_copy_service/kernels/chunked_kernel.cu
+++ b/megatron/core/resharding/nvshmem_copy_service/kernels/chunked_kernel.cu
@@ -96,6 +96,13 @@ __global__ void chunked_batched_copy_kernel(
}
}
}
+
+ // System-wide memory fence: ensures all writes from this thread are visible
+ // to system agents (NIC DMA engines, other GPUs via PCIe/NVLink).
+ // Required so that NVSHMEM put() on another CUDA stream can read correct
+ // pack data from the send buffer. Without this, writes may remain in L2
+ // cache, invisible to the NIC's DMA engine.
+ __threadfence_system();
}
}
diff --git a/megatron/core/resharding/nvshmem_copy_service/memory/double_buffer_manager.py b/megatron/core/resharding/nvshmem_copy_service/memory/double_buffer_manager.py
index 079b2c17610..bfdaac7eb4b 100644
--- a/megatron/core/resharding/nvshmem_copy_service/memory/double_buffer_manager.py
+++ b/megatron/core/resharding/nvshmem_copy_service/memory/double_buffer_manager.py
@@ -6,6 +6,10 @@
Manages send and receive buffers with double-buffering for pipelined communication.
"""
+from ..compat import ensure_nvshmem_compat
+
+ensure_nvshmem_compat()
+
try:
import nvshmem.core.interop.torch
diff --git a/megatron/core/resharding/nvshmem_copy_service/planning/communication_scheduler.py b/megatron/core/resharding/nvshmem_copy_service/planning/communication_scheduler.py
index 0f299a84e40..629c9958db9 100644
--- a/megatron/core/resharding/nvshmem_copy_service/planning/communication_scheduler.py
+++ b/megatron/core/resharding/nvshmem_copy_service/planning/communication_scheduler.py
@@ -10,13 +10,14 @@ class CommunicationScheduler:
"""
Builds a conflict-free, iteration-based schedule for communication.
Ensures that in any given iteration, a PE is not overloaded.
+ Uses greedy first-fit scheduling algorithm.
"""
def __init__(self):
self.num_iterations = 0
def build_schedule(
- self, workloads: Dict[int, List[WorkloadGroup]], my_pe: int, n_pes: int
+ self, workloads: Dict[int, List[WorkloadGroup]], my_pe: int, n_pes: int, group=None
) -> Tuple[Dict[int, List[ScheduledBatch]], Dict[Tuple[int, int, int], WorkloadSummary]]:
"""
Main scheduling method.
@@ -25,24 +26,30 @@ def build_schedule(
3. Returns:
- local schedule (iteration -> list of batches)
- global workload summaries (key: (src, dest, batch_idx) -> summary)
+
+ Args:
+ workloads: Dict mapping destination PE to list of workload groups.
+ my_pe: This PE's rank.
+ n_pes: Total number of PEs.
+ group: Optional ProcessGroup for distributed operations.
"""
total_local_batches = sum(len(groups) for groups in workloads.values())
PELogger.info(f"Building schedule: {total_local_batches} local batches, {n_pes} PEs")
# Step 1: Collect all batches across all PE pairs
PELogger.debug("Collecting batches from all PEs...")
- all_batches = self._collect_all_batches(workloads, my_pe, n_pes)
+ all_batches = self._collect_all_batches(workloads, my_pe, n_pes, group=group)
PELogger.debug(f"Collected {len(all_batches)} total batches globally")
- # Step 2: Assign batches to iterations using conflict-free algorithm
- PELogger.debug("Assigning batches to iterations...")
+ # Step 2: Assign batches to iterations using greedy conflict-free algorithm
+ PELogger.debug("Assigning batches to iterations using greedy conflict-free algorithm...")
self._assign_iterations(all_batches)
PELogger.info(f"Schedule built: {self.num_iterations} iterations")
# Step 3: Exchange detailed workload summaries (Task IDs/Sizes)
# This is needed for receivers to know what tasks are in each batch
PELogger.debug("Exchanging workload summaries...")
- global_summaries = self._exchange_workload_summaries(workloads, my_pe, n_pes)
+ global_summaries = self._exchange_workload_summaries(workloads, my_pe, n_pes, group=group)
PELogger.debug(f"Exchanged {len(global_summaries)} workload summaries")
# Step 4: Build schedule map for this PE
@@ -56,7 +63,7 @@ def build_schedule(
return final_schedule, global_summaries
def _collect_all_batches(
- self, workloads: Dict[int, List[WorkloadGroup]], my_pe: int, n_pes: int
+ self, workloads: Dict[int, List[WorkloadGroup]], my_pe: int, n_pes: int, group=None
) -> List[ScheduledBatch]:
"""
Exchanges batch counts and details with all PEs to build a global view.
@@ -77,7 +84,7 @@ def _collect_all_batches(
# Gather all batches from all PEs using torch.distributed
all_batches_list: List[List[Tuple[int, int, int]] | None] = [None] * n_pes
- dist.all_gather_object(all_batches_list, local_batches)
+ dist.all_gather_object(all_batches_list, local_batches, group=group)
# Flatten into global batch list
global_batches: List[ScheduledBatch] = []
@@ -101,35 +108,92 @@ def _collect_all_batches(
return global_batches
def _assign_iterations(self, batches: List[ScheduledBatch]):
+ """
+ Greedy first-fit scheduling algorithm.
+
+ Assigns batches to iterations using simple greedy first-fit.
+ Processes batches in sorted order and assigns each to the first
+ available iteration with no conflicts.
+ """
self.num_iterations = 0
- batches.sort(key=lambda x: (x.src_pe, x.dest_pe, x.batch_index))
+
+ # Calculate degree (conflict count) for each batch
+ def calc_degree(batch: ScheduledBatch, all_batches: List[ScheduledBatch]) -> int:
+ """Count how many other batches conflict with this batch."""
+ conflicts = 0
+ batch_pes = {batch.src_pe, batch.dest_pe}
+ for other in all_batches:
+ if other is batch:
+ continue
+ other_pes = {other.src_pe, other.dest_pe}
+ # Conflict if they share any PE
+ if batch_pes & other_pes:
+ conflicts += 1
+ return conflicts
+
+ def has_conflict(batch: ScheduledBatch, iteration_state: Dict) -> bool:
+ """
+ Check if a batch conflicts with an iteration's current PE usage.
+
+ A batch conflicts if either its source or destination PE is already
+ being used (as sender or receiver) in the iteration.
+
+ Args:
+ batch: The batch to check
+ iteration_state: Dict with 'src_pes' and 'dst_pes' sets
+
+ Returns:
+ True if there's a conflict, False if the batch can be scheduled
+ """
+ return (
+ batch.src_pe in iteration_state['src_pes']
+ or batch.src_pe in iteration_state['dst_pes']
+ or batch.dest_pe in iteration_state['src_pes']
+ or batch.dest_pe in iteration_state['dst_pes']
+ )
+
+ # Sort batches: process batches with more potential conflicts first
+ # This heuristic (largest-degree-first) often produces better colorings
+ # Sort by degree (descending), then total_size (descending) for tie-breaking
+ batches.sort(key=lambda b: (-calc_degree(b, batches), -b.total_size))
+
+ # Track which PEs are busy (sending or receiving) in each iteration
+ # iteration -> {src_pes: set, dst_pes: set}
+ iteration_usage = []
for batch in batches:
- iteration = 0
+ # Find first iteration where this batch fits (no conflicts)
assigned = False
- while not assigned:
- if not self._has_conflict(batch, iteration, batches):
- batch.iteration = iteration
- self.num_iterations = max(self.num_iterations, iteration + 1)
+ for iter_idx in range(len(iteration_usage)):
+ if not has_conflict(batch, iteration_usage[iter_idx]):
+ # No conflict - assign to this iteration
+ batch.iteration = iter_idx
+ iteration_usage[iter_idx]['src_pes'].add(batch.src_pe)
+ iteration_usage[iter_idx]['dst_pes'].add(batch.dest_pe)
assigned = True
PELogger.debug(
f" Assigned batch ({batch.src_pe} → {batch.dest_pe}, "
- f"idx={batch.batch_index}) to iteration {iteration}"
+ f"idx={batch.batch_index}) to iteration {iter_idx}"
)
- else:
- iteration += 1
-
- def _has_conflict(
- self, batch: ScheduledBatch, iteration: int, all_batches: List[ScheduledBatch]
- ) -> bool:
- for other in all_batches:
- if other.iteration == iteration and other is not batch:
- if other.src_pe == batch.src_pe or other.dest_pe == batch.dest_pe:
- return True
- return False
+ break
+
+ if not assigned:
+ # Need a new iteration
+ new_iter = len(iteration_usage)
+ batch.iteration = new_iter
+ iteration_usage.append({'src_pes': {batch.src_pe}, 'dst_pes': {batch.dest_pe}})
+ PELogger.debug(
+ f" Assigned batch ({batch.src_pe} → {batch.dest_pe}, "
+ f"idx={batch.batch_index}) to NEW iteration {new_iter}"
+ )
+
+ self.num_iterations = len(iteration_usage)
+ PELogger.info(
+ f"Greedy scheduling: {len(batches)} batches → {self.num_iterations} iterations"
+ )
def _exchange_workload_summaries(
- self, workloads: Dict[int, List[WorkloadGroup]], my_pe: int, n_pes: int
+ self, workloads: Dict[int, List[WorkloadGroup]], my_pe: int, n_pes: int, group=None
) -> Dict[Tuple[int, int, int], WorkloadSummary]:
"""
Exchange detailed workload content using torch.distributed.
@@ -146,15 +210,15 @@ def _exchange_workload_summaries(
for dest_pe, groups in workloads.items():
if dest_pe == my_pe:
continue
- for batch_idx, group in enumerate(groups):
+ for batch_idx, wl_group in enumerate(groups):
key = (my_pe, dest_pe, batch_idx)
local_summaries[key] = {
- "total_size": group.total_size,
- "task_ids": [t.task_id for t in group.tasks],
- "task_sizes": [t.size for t in group.tasks],
+ "total_size": wl_group.total_size,
+ "task_ids": [t.task_id for t in wl_group.tasks],
+ "task_sizes": [t.size for t in wl_group.tasks],
}
batch_count += 1
- total_tasks += len(group.tasks)
+ total_tasks += len(wl_group.tasks)
PELogger.debug(f" Local summaries: {batch_count} batches, {total_tasks} tasks")
@@ -162,7 +226,7 @@ def _exchange_workload_summaries(
all_summaries_list: List[Dict[Tuple[int, int, int], Dict[str, object]] | None] = [
None
] * n_pes
- dist.all_gather_object(all_summaries_list, local_summaries)
+ dist.all_gather_object(all_summaries_list, local_summaries, group=group)
# Merge into global map
global_map: Dict[Tuple[int, int, int], WorkloadSummary] = {}
diff --git a/megatron/core/resharding/nvshmem_copy_service/service.py b/megatron/core/resharding/nvshmem_copy_service/service.py
index 631e63ae41b..f5665088bfd 100644
--- a/megatron/core/resharding/nvshmem_copy_service/service.py
+++ b/megatron/core/resharding/nvshmem_copy_service/service.py
@@ -10,6 +10,10 @@
from typing import Dict, List, Optional, Tuple
+from .compat import ensure_nvshmem_compat
+
+ensure_nvshmem_compat()
+
try:
import nvshmem.core
@@ -17,6 +21,7 @@
except ImportError:
HAVE_NVSHMEM = False
+import torch
import torch.cuda.nvtx as nvtx
from .core import GPUResourceManager, KernelLauncher, PipelineExecutor
@@ -34,7 +39,7 @@ class RemoteCopyService:
and executing pipelined communication with NVSHMEM.
"""
- def __init__(self):
+ def __init__(self, group=None):
# Core components
self.gpu_resources = GPUResourceManager()
self.buffer_manager = DoubleBufferManager()
@@ -47,6 +52,9 @@ def __init__(self):
self.comm_scheduler = CommunicationScheduler()
self.gpu_planner = GPUExecutionPlanner()
+ # Optional process group for distributed operations
+ self._group = group
+
# State
self.send_requests: List[SendRequest] = []
self.receive_requests: List[ReceiveRequest] = []
@@ -56,6 +64,7 @@ def __init__(self):
# Events for double-buffering
self.pack_events = []
self.unpack_events = []
+ self.barrier_events = []
@property
def my_pe(self) -> int:
@@ -93,14 +102,31 @@ def init(self, log_level: str = "INFO") -> None:
)
# Initialize GPU resources (NVSHMEM, device, streams)
- self.gpu_resources.init()
+ self.gpu_resources.init(group=self._group)
# Initialize logger after PE ID is known
PELogger.init(self.my_pe, level=log_level)
PELogger.info(f"Initializing RemoteCopyService on PE {self.my_pe}/{self.n_pes}")
+ # Barrier to ensure ALL PEs finish NVSHMEM init before ANY PE starts buffer allocation
+ # buffer_manager.allocate() calls bytetensor() which is a collective operation
+ # Without this barrier, early PEs call bytetensor() while late PEs
+ # are still in init() -> deadlock
+ nvshmem.core.barrier_all(stream=self.gpu_resources.send_stream)
+ self.gpu_resources.send_stream.sync() # Ensure barrier completes on CPU
+
# Allocate double-buffered send/recv slots
self.buffer_manager.allocate()
+ # The .zero_() calls inside allocate() go to the default CUDA stream.
+ # Sync it now so the zeros are fully committed before any NVShmem
+ # operations (which bypass CUDA streams via RDMA) touch the buffers.
+ # Without this, a still-running zero() can race with the first
+ # nvshmem.core.put() and overwrite received data.
+ torch.cuda.synchronize()
+
+ # Barrier to ensure all PEs complete buffer allocation before proceeding
+ nvshmem.core.barrier_all(stream=self.gpu_resources.send_stream)
+
PELogger.debug("Allocated double-buffered send/recv slots")
# Load CUDA kernels
@@ -124,10 +150,20 @@ def init(self, log_level: str = "INFO") -> None:
self.gpu_resources.unpack_stream,
self.gpu_resources.send_stream,
self.gpu_resources.copy_stream,
- self.gpu_resources.torch_pack_stream,
- self.gpu_resources.torch_unpack_stream,
- self.gpu_resources.torch_copy_stream,
+ self.gpu_resources.torch_pack_stream_wrapper,
+ self.gpu_resources.torch_unpack_stream_wrapper,
+ self.gpu_resources.torch_send_stream_wrapper,
+ self.gpu_resources.torch_copy_stream_wrapper,
)
+
+ # Synchronize all NVSHMEM streams before returning
+ # This ensures all barrier operations complete and streams are idle
+ # Without this, subsequent torch.cuda.synchronize() may hang waiting for pending work
+ self.gpu_resources.send_stream.sync()
+ self.gpu_resources.pack_stream.sync()
+ self.gpu_resources.unpack_stream.sync()
+ self.gpu_resources.copy_stream.sync()
+
PELogger.info("Initialization complete")
def register_send(
@@ -212,7 +248,7 @@ def schedule(self) -> None:
# Step 3: Schedule workloads to iterations
PELogger.debug("Step 3: Building communication schedule...")
schedule, global_summaries = self.comm_scheduler.build_schedule(
- workloads, self.my_pe, self.n_pes
+ workloads, self.my_pe, self.n_pes, group=self._group
)
self.num_iterations = self.comm_scheduler.num_iterations
@@ -235,8 +271,10 @@ def schedule(self) -> None:
# Step 6: Create double-buffered events
PELogger.debug("Step 6: Creating synchronization events...")
- self.pack_events, self.unpack_events = self.gpu_resources.create_events(num_events=2)
- self.pipeline_executor.set_events(self.pack_events, self.unpack_events)
+ self.pack_events, self.unpack_events, self.barrier_events = (
+ self.gpu_resources.create_events(num_events=2)
+ )
+ self.pipeline_executor.set_events(self.pack_events, self.unpack_events, self.barrier_events)
PELogger.info(f"Schedule complete: {self.num_iterations} iterations ready")
@@ -293,6 +331,7 @@ def clear_requests(self) -> None:
self.num_iterations = 0
self.pack_events = []
self.unpack_events = []
+ self.barrier_events = []
def finalize(self) -> None:
"""Cleanup resources."""
diff --git a/megatron/core/resharding/planner.py b/megatron/core/resharding/planner.py
index 31045fbfc01..a5fd1f877bf 100644
--- a/megatron/core/resharding/planner.py
+++ b/megatron/core/resharding/planner.py
@@ -90,6 +90,7 @@ def _plan_multi_dim_lcm(
d = descriptors[0]
if my_global_rank not in d.dst_dim_ranks:
return []
+
src_shape = tuple(src_metadata.shape)
dst_shape = tuple(dst_metadata.shape)
dim = d.dim
@@ -151,6 +152,126 @@ def dst_key(op):
return ops
+def _plan_block_interleaved(
+ param_name: str,
+ src_metadata: ParameterMetadata,
+ dst_metadata: ParameterMetadata,
+ descriptors: list[ShardingDescriptor],
+ my_global_rank: int,
+) -> list[tuple[int, tuple[slice, ...], tuple[slice, ...]]]:
+ """
+ Block-interleaved TP planner for parameters with ``partition_sizes``.
+
+ When a parameter packs multiple independently-sharded components of
+ *different* sizes (e.g. Mamba in_proj packs z, x, B, C, dt), a simple
+ contiguous concat produces the wrong layout. This function treats each
+ block independently: it gathers (or scatters) each block across TP ranks
+ before moving to the next block.
+
+ ``partition_sizes`` lists the per-TP-rank block sizes along the partition
+ dim. Block *i* occupies ``[sum(sizes[:i]), sum(sizes[:i+1]))`` in the
+ local tensor on every TP rank. In the *full* (TP-gathered) tensor, block
+ *i* occupies ``[sum(full_sizes[:i]), sum(full_sizes[:i+1]))`` where
+ ``full_sizes[i] = sizes[i] * src_tp_world``.
+ """
+ if not descriptors or descriptors[0].name != "tp":
+ return []
+ d = descriptors[0]
+ if my_global_rank not in d.dst_dim_ranks:
+ return []
+
+ dim = d.dim
+ src_shape = tuple(src_metadata.shape)
+ dst_shape = tuple(dst_metadata.shape)
+ src_world = len(d.src_dim_ranks)
+ dst_world = len(d.dst_dim_ranks)
+ dst_local_rank = _get_rank_in_group(my_global_rank, d.dst_dim_ranks)
+
+ # Use partition_sizes from whichever side has it (prefer src)
+ src_sizes = src_metadata.partition_sizes
+ dst_sizes = dst_metadata.partition_sizes
+
+ if src_sizes is None and dst_sizes is None:
+ raise RuntimeError(f"{param_name}: _plan_block_interleaved called without partition_sizes")
+
+ # Derive the full (un-sharded) block sizes
+ if src_sizes is not None:
+ num_blocks = len(src_sizes)
+ full_sizes = [s * src_world for s in src_sizes]
+ else:
+ num_blocks = len(dst_sizes)
+ full_sizes = [s * dst_world for s in dst_sizes]
+
+ # Compute per-rank block sizes for both sides
+ if src_sizes is None:
+ src_sizes = [f // src_world for f in full_sizes]
+ if dst_sizes is None:
+ dst_sizes = [f // dst_world for f in full_sizes]
+
+ # Validate conservation
+ for i in range(num_blocks):
+ if src_sizes[i] * src_world != dst_sizes[i] * dst_world:
+ raise RuntimeError(
+ f"{param_name}: block {i} size mismatch: "
+ f"src_sizes[{i}]={src_sizes[i]}*{src_world} != "
+ f"dst_sizes[{i}]={dst_sizes[i]}*{dst_world}"
+ )
+
+ ops: list[tuple[int, tuple[slice, ...], tuple[slice, ...]]] = []
+
+ # For each block, compute the transfer ops independently
+ src_block_offset = 0 # cumulative offset in source local tensor
+ dst_block_offset = 0 # cumulative offset in destination local tensor
+
+ for blk in range(num_blocks):
+ src_blk_sz = src_sizes[blk] # per-src-rank size of this block
+ dst_blk_sz = dst_sizes[blk] # per-dst-rank size of this block
+ full_blk_sz = full_sizes[blk]
+
+ # Within this block, use simple LCM tiling (stride=1)
+ Ns = src_world
+ Nd = dst_world
+ g = math.gcd(Ns, Nd)
+ L = (Ns // g) * Nd
+ if full_blk_sz % L != 0:
+ raise RuntimeError(
+ f"{param_name}: block {blk} full_size {full_blk_sz} not divisible by LCM {L}"
+ )
+ unit = full_blk_sz // L
+ cps = L // Ns
+ cpd = L // Nd
+
+ # This dst rank's segment within the block
+ g_dst_seg = dst_local_rank
+ for off in range(cpd):
+ g_micro = g_dst_seg * cpd + off
+ s_idx = g_micro // cps
+ in_seg = g_micro % cps
+ src_owner_in_dim = s_idx % src_world
+ src_global_rank = d.src_dim_ranks[src_owner_in_dim]
+ src_local_seg_idx = s_idx // src_world
+ src_start = src_block_offset + src_local_seg_idx * (cps * unit) + in_seg * unit
+ dst_start = dst_block_offset + off * unit
+
+ src_slice = [slice(None)] * len(src_shape)
+ dst_slice = [slice(None)] * len(dst_shape)
+ src_slice[dim] = slice(src_start, src_start + unit)
+ dst_slice[dim] = slice(dst_start, dst_start + unit)
+ ops.append((src_global_rank, tuple(src_slice), tuple(dst_slice)))
+
+ src_block_offset += src_blk_sz
+ dst_block_offset += dst_blk_sz
+
+ # Stable sort by destination offset
+ def dst_key(op):
+ _, _, dsl = op
+ s = dsl[dim]
+ return s.start if isinstance(s, slice) else 0
+
+ ops.sort(key=dst_key)
+ return ops
+
+
def _finalize_dp_transfers(
param_name: str,
src_metadata: ParameterMetadata,
@@ -174,8 +295,8 @@ def _finalize_dp_transfers(
dst_shape = dst_metadata.shape
- # Same DP layout - local copy
- if src_dp_ranks == dst_dp_ranks:
+ # Same DP layout - local copy (only if this rank has the source parameter)
+ if src_dp_ranks == dst_dp_ranks and my_global_rank in src_dp_ranks:
full_slice = tuple(slice(None) for _ in range(len(dst_shape)))
return [(my_global_rank, full_slice, full_slice)]
@@ -183,7 +304,17 @@ def _finalize_dp_transfers(
# better load balancing across source ranks. This ensures that destination
# ranks are distributed across source ranks even when they have the same
# position within their respective DP groups.
+ #
+ # In non-collocated mode, src_dp_ranks might include ranks that don't
+ # have the source model (e.g., idle ranks or destination ranks). Filter to only
+ # include the rank that provided this metadata (src_metadata.owner_rank).
+ # src_metadata was selected by select_src_metadata_balanced, so owner_rank is the
+ # actual source rank for this parameter.
+ actual_src_rank = src_metadata.owner_rank
src_global_rank = src_dp_ranks[my_global_rank % len(src_dp_ranks)]
+ # Override with the actual source rank if the selected rank doesn't have the parameter
+ if src_global_rank != actual_src_rank:
+ src_global_rank = actual_src_rank
full_slice = tuple(slice(None) for _ in range(len(dst_shape)))
return [(src_global_rank, full_slice, full_slice)]
@@ -199,6 +330,16 @@ def _determine_source_ranks_for_dst_param(
# Regular TP/DP planning with EP-resolved metadata
descriptors = _build_descriptors_for_param(src_metadata=src_metadata, dst_metadata=dst_metadata)
if descriptors:
+ # Use block-interleaved planner when partition_sizes is present
+ # (e.g. Mamba in_proj packs components of different sizes)
+ if src_metadata.partition_sizes is not None or dst_metadata.partition_sizes is not None:
+ return _plan_block_interleaved(
+ param_name=param_name,
+ src_metadata=src_metadata,
+ dst_metadata=dst_metadata,
+ descriptors=descriptors,
+ my_global_rank=my_global_rank,
+ )
return _plan_multi_dim_lcm(
param_name=param_name,
src_metadata=src_metadata,
@@ -211,71 +352,113 @@ def _determine_source_ranks_for_dst_param(
def build_centralized_reshard_plan(
- src_module: torch.nn.Module, dst_module: torch.nn.Module, num_experts: int = None
+ src_module: torch.nn.Module,
+ dst_module: torch.nn.Module,
+ num_experts: int = None,
+ group=None,
+ src_rank_offset: int = 0,
+ dst_rank_offset: int = 0,
) -> ReshardPlan:
"""
Centralized planning: Rank 0 builds complete plan for all ranks, then scatters.
+
+ Supports None for src_module and/or dst_module to enable non-collocated mode:
+ - src_module=None: Rank doesn't have source model (destination-only)
+ - dst_module=None: Rank doesn't have destination model (source-only)
+ - Both provided: Rank has both models (collocated mode)
+
+ Each rank provides metadata only for the models it owns, including parallel group
+ membership (tensor_parallel_group_ranks, expert_parallel_group_ranks, etc.).
+ This metadata is sufficient for rank 0 to build correct transfer plans without
+ requiring dummy models.
"""
- my_global_rank = dist.get_rank()
- world_size = dist.get_world_size()
-
- # Get process groups
- src_pg = getattr(src_module, "pg_collection", None)
- dst_pg = getattr(dst_module, "pg_collection", None)
- if src_pg is None or dst_pg is None:
- raise ValueError("Both modules must have pg_collection")
-
- # Gather param metadata from all ranks
- my_src_params = {name: p for name, p in src_module.named_parameters(recurse=True)}
- my_dst_params = {name: p for name, p in dst_module.named_parameters(recurse=True)}
-
- # Build PP layer prefix maps to be used for parameter name rewriting
- src_layer_prefix_map = _build_layer_module_prefix_map(src_module)
- dst_layer_prefix_map = _build_layer_module_prefix_map(dst_module)
-
- my_src_metadata = [
- extract_param_metadata(
- p,
- name,
- my_global_rank,
- src_pg,
- num_experts=num_experts,
- layer_module_prefix_map=src_layer_prefix_map,
- )
- for name, p in my_src_params.items()
- ]
- my_dst_metadata = [
- extract_param_metadata(
- p,
- name,
- my_global_rank,
- dst_pg,
- num_experts=num_experts,
- layer_module_prefix_map=dst_layer_prefix_map,
- )
- for name, p in my_dst_params.items()
- ]
+ # Use group.rank() instead of dist.get_rank(group) to support cross-cluster
+ # ProcessGroups where members have independent default PGs (same default rank).
+ my_global_rank = group.rank() if group is not None else dist.get_rank()
+ world_size = group.size() if group is not None else dist.get_world_size()
+
+ # Shared cache for deduplicating rank lists across all metadata on this
+ # rank. Params sharing the same TP/DP/EP/PP groups will reference one
+ # list object, making pickle ~75% smaller for the gather.
+ _rank_list_cache: dict = {}
+
+ # Extract metadata from source model if present
+ if src_module is not None:
+ src_pg = getattr(src_module, "pg_collection", None)
+ if src_pg is None:
+ raise ValueError("Source module must have pg_collection")
+ my_src_params = {name: p for name, p in src_module.named_parameters(recurse=True)}
+ src_layer_prefix_map = _build_layer_module_prefix_map(src_module)
+ my_src_metadata = [
+ extract_param_metadata(
+ p,
+ name,
+ my_global_rank,
+ src_pg,
+ num_experts=num_experts,
+ layer_module_prefix_map=src_layer_prefix_map,
+ rank_offset=src_rank_offset,
+ _rank_list_cache=_rank_list_cache,
+ )
+ for name, p in my_src_params.items()
+ ]
+ else:
+ # No source model on this rank - provide empty metadata
+ my_src_metadata = []
+
+ # Extract metadata from destination model if present
+ if dst_module is not None:
+ dst_pg = getattr(dst_module, "pg_collection", None)
+ if dst_pg is None:
+ raise ValueError("Destination module must have pg_collection")
+ my_dst_params = {name: p for name, p in dst_module.named_parameters(recurse=True)}
+ dst_layer_prefix_map = _build_layer_module_prefix_map(dst_module)
+ my_dst_metadata = [
+ extract_param_metadata(
+ p,
+ name,
+ my_global_rank,
+ dst_pg,
+ num_experts=num_experts,
+ layer_module_prefix_map=dst_layer_prefix_map,
+ rank_offset=dst_rank_offset,
+ _rank_list_cache=_rank_list_cache,
+ )
+ for name, p in my_dst_params.items()
+ ]
+ else:
+ # No destination model on this rank - provide empty metadata
+ my_dst_metadata = []
- all_src_metadata_by_rank = [None] * world_size
- all_dst_metadata_by_rank = [None] * world_size
- dist.all_gather_object(all_src_metadata_by_rank, my_src_metadata)
- dist.all_gather_object(all_dst_metadata_by_rank, my_dst_metadata)
+ # Gather metadata to rank 0 only (not all ranks) to save CPU memory.
+ # Other ranks don't need the full metadata — they only need their own plan.
+ all_src_metadata_by_rank = [None] * world_size if my_global_rank == 0 else None
+ all_dst_metadata_by_rank = [None] * world_size if my_global_rank == 0 else None
+ dist.gather_object(my_src_metadata, all_src_metadata_by_rank, group_dst=0, group=group)
+ dist.gather_object(my_dst_metadata, all_dst_metadata_by_rank, group_dst=0, group=group)
- # Parameter to metadata maps keyed by resolved_name
+ # Free local metadata — no longer needed after gather.
+ del my_src_metadata, my_dst_metadata
+
+ # Parameter to metadata maps keyed by resolved_name (only populated on rank 0)
src_param_metadata_by_rank = {}
dst_param_metadata_by_rank = {}
src_param_metadata: dict[str, list[ParameterMetadata]] = {}
- for rank_id, rank_metadata_list in enumerate(all_src_metadata_by_rank):
- src_param_metadata_by_rank[rank_id] = {m.resolved_name: m for m in rank_metadata_list}
- for rank_id, rank_metadata_list in enumerate(all_dst_metadata_by_rank):
- dst_param_metadata_by_rank[rank_id] = {m.resolved_name: m for m in rank_metadata_list}
- for rank_metadata_list in all_src_metadata_by_rank:
- for metadata in rank_metadata_list:
- key = metadata.resolved_name
- if key not in src_param_metadata:
- src_param_metadata[key] = []
- src_param_metadata[key].append(metadata)
+ if my_global_rank == 0:
+ for rank_id, rank_metadata_list in enumerate(all_src_metadata_by_rank):
+ src_param_metadata_by_rank[rank_id] = {m.resolved_name: m for m in rank_metadata_list}
+ for rank_id, rank_metadata_list in enumerate(all_dst_metadata_by_rank):
+ dst_param_metadata_by_rank[rank_id] = {m.resolved_name: m for m in rank_metadata_list}
+ for rank_metadata_list in all_src_metadata_by_rank:
+ for metadata in rank_metadata_list:
+ key = metadata.resolved_name
+ if key not in src_param_metadata:
+ src_param_metadata[key] = []
+ src_param_metadata[key].append(metadata)
+
+ # Free the raw gathered lists — data is now in the indexed dicts.
+ del all_src_metadata_by_rank, all_dst_metadata_by_rank
# Build the plan on global rank 0 and broadcast to all ranks
if my_global_rank == 0:
@@ -332,10 +515,18 @@ def build_centralized_reshard_plan(
)
)
plans_list = [plans_for_all_ranks[r] for r in range(world_size)]
+
+ # Free planning intermediates on rank 0 before the scatter.
+ del plans_for_all_ranks, src_param_metadata_by_rank
+ del dst_param_metadata_by_rank, src_param_metadata
else:
- plans_list = [None] * world_size
- torch.distributed.broadcast_object_list(plans_list, src=0)
- my_plan = plans_list[my_global_rank]
+ plans_list = None
+
+ # Scatter: each rank receives only its own plan (not all plans).
+ my_plan_list = [None]
+ torch.distributed.scatter_object_list(my_plan_list, plans_list, group_src=0, group=group)
+ my_plan = my_plan_list[0]
+ del plans_list # Free the full list on rank 0.
logger.info(
f"Rank {my_global_rank}: Received plan - {len(my_plan.recv_ops)} recvs, "
diff --git a/megatron/core/resharding/refit.py b/megatron/core/resharding/refit.py
index 5461b8d3900..b4d86b6a311 100644
--- a/megatron/core/resharding/refit.py
+++ b/megatron/core/resharding/refit.py
@@ -3,13 +3,21 @@
"""
High-level refit/reshard orchestration:
+- prepare_swap_model_weights: build and cache the reshard plan without any transfer.
- swap_model_weights: public API; accepts a backend name or CopyService and delegates.
- reshard_model_weights: transport-agnostic core; builds/caches plan and executes.
"""
-from typing import Any, Literal, Optional, Union
+from dataclasses import dataclass
+from typing import Any, Literal, Optional, Tuple, Union
+
+import torch
from megatron.core import parallel_state
+from megatron.core.inference.quantization.utils import (
+ _should_quantize_param,
+ quantize_params_to_mxfp8,
+)
from megatron.core.models.common.language_module.language_module import LanguageModule
from megatron.core.utils import unwrap_model
@@ -18,29 +26,99 @@
from .copy_services.gloo_copy_service import GlooCopyService
from .copy_services.nccl_copy_service import NCCLCopyService
from .copy_services.nvshmem_copy_service import NVSHMEMCopyService
+from .transforms import MXFP8ReshardTransform, ReshardTransform
# Supported refit backend names
RefitBackendName = Literal["nccl", "gloo", "nvshmem"]
+
+@dataclass(frozen=True)
+class _PlanCacheKey:
+ """
+ Cache key for reshard plans.
+ """
+
+ rank: int
+ # Parallelism configuration: (TP, PP, EP, DP, expt_tp) or None for non-collocated ranks
+ src_config: Optional[Tuple[int, int, int, int, int]]
+ dst_config: Optional[Tuple[int, int, int, int, int]]
+ num_experts: Optional[int]
+
+
+def _get_config_tuple(core) -> Optional[Tuple[int, int, int, int, int]]:
+ """Extract (TP, PP, EP, DP, expt_tp) sizes from a model core.
+
+ Returns:
+ Tuple of (TP, PP, EP, DP, expt_tp) sizes, or None if core is None.
+ - TP: Tensor parallelism
+ - PP: Pipeline parallelism
+ - EP: Expert parallelism
+ - DP: Data parallelism
+ - expt_tp: Expert tensor parallelism
+ """
+ if core is None:
+ return None
+ pg = core.pg_collection
+ return (
+ len(torch.distributed.get_process_group_ranks(pg.tp)) if pg.tp else 1,
+ len(torch.distributed.get_process_group_ranks(pg.pp)) if pg.pp else 1,
+ len(torch.distributed.get_process_group_ranks(pg.ep)) if pg.ep else 1,
+ len(torch.distributed.get_process_group_ranks(pg.dp)) if pg.dp else 1,
+ (
+ len(torch.distributed.get_process_group_ranks(pg.expt_tp))
+ if hasattr(pg, 'expt_tp') and pg.expt_tp
+ else 1
+ ),
+ )
+
+
+def _build_plan_cache_key(
+ src_core, tgt_core, num_experts: Optional[int], group=None
+) -> _PlanCacheKey:
+ """Build cache key for reshard plan.
+
+ Args:
+ src_core: Source model core (or None for non-collocated destination/idle ranks)
+ tgt_core: Target model core (or None for non-collocated source/idle ranks)
+ num_experts: Number of MoE experts (or None for non-MoE models)
+ group: Optional process group for rank query
+
+ Returns:
+ Cache key that uniquely identifies this reshard configuration for this rank
+ """
+ # Use group.rank() to support cross-cluster ProcessGroups
+ rank = group.rank() if group is not None else torch.distributed.get_rank()
+ src_config = _get_config_tuple(src_core)
+ dst_config = _get_config_tuple(tgt_core)
+ return _PlanCacheKey(
+ rank=rank, src_config=src_config, dst_config=dst_config, num_experts=num_experts
+ )
+
+
# Module-level cache for refit services to avoid repeated allocations
_service_cache: dict[str, CopyService] = {}
+_plan_cache: dict[_PlanCacheKey, Any] = {}
-def get_or_create_service(backend: RefitBackendName) -> CopyService:
+def get_or_create_service(backend: RefitBackendName, group=None) -> CopyService:
"""Get or create a cached CopyService instance for the given backend.
This avoids expensive repeated allocations (especially for NVSHMEM buffers)
when swap_model_weights is called multiple times with the same backend.
+
+ Args:
+ backend: Backend name ("nccl", "gloo", or "nvshmem").
+ group: Optional process group for NCCL backend.
"""
if backend in _service_cache:
return _service_cache[backend]
if backend == "nccl":
- service = NCCLCopyService()
+ service = NCCLCopyService(group=group)
elif backend == "gloo":
- service = GlooCopyService()
+ service = GlooCopyService(group=group)
elif backend == "nvshmem":
- service = NVSHMEMCopyService()
+ service = NVSHMEMCopyService(group=group)
else:
raise ValueError(f"Unknown backend '{backend}'")
@@ -53,62 +131,344 @@ def clear_service_cache():
Call this if you need to invalidate the cache, for example when
reinitializing distributed state.
+
+ This properly finalizes services to free GPU buffers
+ before clearing the cache.
"""
global _service_cache
+
+ # Finalize services to free resources for NVSHMEM backend
+ # NCCL/Gloo services have no cleanup needed
+ for backend_name, service in _service_cache.items():
+ if hasattr(service, '_remote') and hasattr(service._remote, 'finalize'):
+ service._remote.finalize()
+
_service_cache.clear()
+def clear_plan_cache():
+ """
+ Clear the cached refit plans.
+ """
+ global _plan_cache
+ _plan_cache.clear()
+
+
+def clear_all_caches():
+ """
+ Clear both service and plan caches.
+ """
+ clear_service_cache()
+ clear_plan_cache()
+
+
+def _unwrap_model_cores(src_model, target_model):
+ """Extract (src_core, tgt_core, num_experts) from model arguments.
+
+ Handles list-wrapped modules and None (non-collocated) models.
+ Fills in missing DP groups from Megatron's parallel state on the source.
+
+ Returns:
+ (src_core, tgt_core, num_experts)
+ """
+ src_core = None
+ tgt_core = None
+ num_experts = None
+
+ if src_model is not None:
+ src_lm = src_model[0] if isinstance(src_model, (list, tuple)) else src_model
+ num_experts = src_lm.config.num_moe_experts
+ src_core = unwrap_model(src_lm)
+ if not hasattr(src_core, "pg_collection") or src_core.pg_collection is None:
+ raise RuntimeError("Source model missing pg_collection required for reshard")
+ # Fill missing DP group on the source using Megatron's parallel state if not provided
+ if getattr(src_core.pg_collection, "dp", None) is None:
+ src_core.pg_collection.dp = parallel_state.get_data_parallel_group()
+
+ if target_model is not None:
+ tgt_lm = target_model[0] if isinstance(target_model, (list, tuple)) else target_model
+ if num_experts is None:
+ num_experts = tgt_lm.config.num_moe_experts
+ tgt_core = unwrap_model(tgt_lm)
+ if not hasattr(tgt_core, "pg_collection") or tgt_core.pg_collection is None:
+ raise RuntimeError("Target model missing pg_collection required for reshard")
+
+ return src_core, tgt_core, num_experts
+
+
+def _build_or_get_plan(src_core, tgt_core, num_experts, group, src_rank_offset, dst_rank_offset):
+ """Return the cached reshard plan, building it (collectively) if not yet cached.
+
+ All participating ranks must call this simultaneously when the plan is not
+ yet cached, because build_centralized_reshard_plan uses collective communication.
+ """
+ global _plan_cache
+ cache_key = _build_plan_cache_key(src_core, tgt_core, num_experts, group=group)
+ if cache_key not in _plan_cache:
+ _plan_cache[cache_key] = build_centralized_reshard_plan(
+ src_core,
+ tgt_core,
+ num_experts=num_experts,
+ group=group,
+ src_rank_offset=src_rank_offset,
+ dst_rank_offset=dst_rank_offset,
+ )
+ return _plan_cache[cache_key]
+
+
+def _needs_mxfp8_conversion(model) -> bool:
+ """Check if a model uses FlashInfer MXFP8 inference and needs weight conversion."""
+ if model is None:
+ return False
+ lm = model[0] if isinstance(model, (list, tuple)) else model
+ config = lm.config
+ return (
+ getattr(config, 'transformer_impl', None) == 'inference_optimized'
+ and getattr(config, 'fp8_recipe', None) == 'mxfp8'
+ )
+
+
+def _setup_mxfp8_transform_on_plan(plan, target_model) -> None:
+ """Detect MXFP8 needs and attach a transform to the plan if required.
+
+ If the *target_model* uses an inference-optimized layer spec with MXFP8,
+ this function:
+ 1. Computes which params are eligible for MXFP8 conversion.
+ 2. Quantizes the target model's decoder weights to FlashInfer MXFP8Tensor
+ (creating persistent buffers whose addresses are later captured by
+ CUDA graphs).
+ 3. Builds an ``MXFP8ReshardTransform`` and attaches it to the plan as
+ ``plan.transform``.
+
+ If the model doesn't need MXFP8, ``plan.transform`` is set to None.
+ Subsequent calls are no-ops if the plan already has a transform attribute.
+ """
+ if hasattr(plan, 'transform'):
+ return # Already set up
+
+ if not _needs_mxfp8_conversion(target_model):
+ plan.transform = None
+ return
+
+ lm = target_model[0] if isinstance(target_model, (list, tuple)) else target_model
+ core = unwrap_model(lm)
+ decoder = core.decoder if hasattr(core, 'decoder') else core
+
+ # 1. Compute which parameters are eligible for MXFP8 conversion.
+ # Must be done while params are still visible as nn.Parameter (BF16).
+ convertible: set[str] = set()
+ for name, param in decoder.named_parameters():
+ if _should_quantize_param(param):
+ convertible.add(f"decoder.{name}")
+
+ # 2. Quantize decoder weights → persistent MXFP8Tensor buffers.
+ persistent_buffers = quantize_params_to_mxfp8(decoder)
+
+ # 3. Build the transform and attach it to the plan.
+ plan.transform = MXFP8ReshardTransform(
+ convertible_params=convertible,
+ persistent_buffers=persistent_buffers,
+ buffer_key_prefix="decoder.",
+ )
+
+
+def prepare_swap_model_weights(
+ src_model: LanguageModule,
+ target_model: LanguageModule,
+ group=None,
+ src_rank_offset: int = 0,
+ dst_rank_offset: int = 0,
+):
+ """Pre-build and cache the reshard plan and any format-conversion transforms.
+
+ Call this during initialization while models are in their native (BF16) format,
+ before any weight format conversion (e.g., MXFP8). The plan is stored in the
+ same module-level cache as swap_model_weights, so subsequent calls reuse it
+ without needing to inspect named_parameters() again.
+
+ If the *target_model* uses an inference-optimized layer spec with MXFP8
+ (``config.transformer_impl == 'inference_optimized'`` and
+ ``config.fp8_recipe == 'mxfp8'``), this function also:
+ - computes which parameters are eligible for MXFP8 conversion,
+ - quantizes the target decoder weights to persistent FlashInfer
+ MXFP8Tensor buffers (whose addresses are later baked into CUDA graphs),
+ - creates an ``MXFP8ReshardTransform`` that subsequent
+ ``swap_model_weights`` calls use automatically.
+
+ Callers do **not** need to know about MXFP8 — the transform is created and
+ cached transparently.
+
+ All participating ranks must call this simultaneously — the plan builder uses
+ collective communication internally.
+
+ Args:
+ src_model: Source model, or None if this rank only receives weights.
+ target_model: Target model, or None if this rank only sends weights.
+ group: Optional process group for collective communication.
+ src_rank_offset: Rank offset for source (training) workers.
+ dst_rank_offset: Rank offset for destination (inference) workers.
+ """
+ src_core, tgt_core, num_experts = _unwrap_model_cores(src_model, target_model)
+ plan = _build_or_get_plan(
+ src_core, tgt_core, num_experts, group, src_rank_offset, dst_rank_offset
+ )
+
+ # Auto-detect and set up MXFP8 transform on the plan for the target model.
+ # This must happen after the plan is built (while BF16 params are still visible)
+ # and before any swap_model_weights call.
+ _setup_mxfp8_transform_on_plan(plan, target_model)
+
+
def swap_model_weights(
src_model: LanguageModule,
target_model: LanguageModule,
refit_method: Union[RefitBackendName, CopyService],
+ group=None,
+ src_rank_offset: int = 0,
+ dst_rank_offset: int = 0,
+ transform: Optional[ReshardTransform] = None,
):
"""
Orchestrate weight swap/refit.
- - refit_method can be:
- * a string backend name (one of the supported refit backends), or
- * a CopyService instance.
+
+ If *transform* is not explicitly provided, the function automatically uses
+ any ``MXFP8ReshardTransform`` that was created and cached by a prior
+ ``prepare_swap_model_weights`` call for the same model pair. This makes
+ MXFP8 handling transparent to callers.
+
+ Args:
+ refit_method: a string backend name (one of the supported refit
+ backends) or a CopyService instance.
+ group: Optional process group for communication.
+ src_rank_offset / dst_rank_offset: Offsets applied to local process
+ group ranks so that metadata contains globally unique rank IDs
+ across independent torch.distributed worlds.
+ transform: Optional ReshardTransform for custom format conversion.
+ If None, the cached transform (from prepare_swap_model_weights)
+ is used automatically when the receiver needs MXFP8 conversion.
"""
- if isinstance(refit_method, CopyService):
+ if isinstance(refit_method, str):
+ service = get_or_create_service(refit_method, group=group)
+ elif hasattr(refit_method, 'submit_send') and hasattr(refit_method, 'run'):
service = refit_method
- reshard_model_weights(src_model, target_model, service=service)
- elif isinstance(refit_method, str):
- service = get_or_create_service(refit_method)
- reshard_model_weights(src_model, target_model, service=service)
else:
- raise TypeError("refit_method must be a str backend name or a CopyService instance")
+ raise TypeError(
+ "refit_method must be a str backend name or a CopyService-compatible instance"
+ )
+
+ # Auto-resolve MXFP8 transform from the cached plan when no
+ # explicit transform was provided.
+ if transform is None:
+ src_core, tgt_core, num_experts = _unwrap_model_cores(src_model, target_model)
+ plan = _build_or_get_plan(
+ src_core, tgt_core, num_experts, group, src_rank_offset, dst_rank_offset
+ )
+ transform = getattr(plan, 'transform', None)
+
+ reshard_model_weights(
+ src_model,
+ target_model,
+ service=service,
+ group=group,
+ src_rank_offset=src_rank_offset,
+ dst_rank_offset=dst_rank_offset,
+ transform=transform,
+ )
def reshard_model_weights(
- src_model: LanguageModule, target_model: LanguageModule, service: CopyService
+ src_model: LanguageModule,
+ target_model: LanguageModule,
+ service: CopyService,
+ group=None,
+ src_rank_offset: int = 0,
+ dst_rank_offset: int = 0,
+ transform: Optional[ReshardTransform] = None,
):
- """Reshard and copy model weights from ``src_model`` to ``target_model`` using ``service``."""
- # Handle list-wrapped modules used throughout training utils
- src_lm = src_model[0] if isinstance(src_model, (list, tuple)) else src_model
- tgt_lm = target_model[0] if isinstance(target_model, (list, tuple)) else target_model
-
- num_experts = src_lm.config.num_moe_experts
-
- # Unwrap to get owning modules (with parameters and pg_collection)
- src_core = unwrap_model(src_lm)
- tgt_core = unwrap_model(tgt_lm)
-
- # Ensure pg_collection exists
- if not hasattr(src_core, "pg_collection") or src_core.pg_collection is None:
- raise RuntimeError("Source model missing pg_collection required for NCCL reshard")
- if not hasattr(tgt_core, "pg_collection") or tgt_core.pg_collection is None:
- raise RuntimeError("Target model missing pg_collection required for NCCL reshard")
-
- # Fill missing DP group on the source using Megatron's parallel state if not provided
- if getattr(src_core.pg_collection, "dp", None) is None:
- src_core.pg_collection.dp = parallel_state.get_data_parallel_group()
-
- # caching plan for reuse
- cached_plan: Optional[Any] = getattr(tgt_core, "_cached_reshard_plan", None)
- if cached_plan is None:
- plan = build_centralized_reshard_plan(src_core, tgt_core, num_experts=num_experts)
- setattr(tgt_core, "_cached_reshard_plan", plan)
+ """Reshard and copy model weights from ``src_model`` to ``target_model`` using ``service``.
+
+ Supports None for src_model and/or target_model to enable non-collocated mode:
+ - (src_model, target_model): Both models present (collocated mode)
+ - (src_model, None): Source rank - only sends data (non-collocated)
+ - (None, target_model): Destination rank - only receives data (non-collocated)
+ - (None, None): Idle rank - participates in collectives but has no transfers (non-collocated)
+
+ Args:
+ group: Optional process group for collective communication.
+ src_rank_offset / dst_rank_offset: Offsets for mapping local ranks to global ranks
+ in independent torch.distributed worlds.
+ transform: Optional ReshardTransform for custom format conversion.
+ """
+ global _plan_cache
+
+ # Handle idle ranks (both models None) - they participate in collectives but have no work
+ if src_model is None and target_model is None:
+ cache_key = _build_plan_cache_key(
+ src_core=None, tgt_core=None, num_experts=None, group=group
+ )
+
+ # Use cached plan if available, otherwise build (with collective participation)
+ if cache_key not in _plan_cache:
+ plan = build_centralized_reshard_plan(
+ None,
+ None,
+ num_experts=None,
+ group=group,
+ src_rank_offset=src_rank_offset,
+ dst_rank_offset=dst_rank_offset,
+ )
+ _plan_cache[cache_key] = plan
+ else:
+ plan = _plan_cache[cache_key]
+ execute_reshard_plan(plan, None, None, service=service, group=group, transform=transform)
+ return
+
+ # Handle None models - extract core modules only from non-None models
+ src_core = None
+ tgt_core = None
+ num_experts = None
+
+ if src_model is not None:
+ # Handle list-wrapped modules
+ src_lm = src_model[0] if isinstance(src_model, (list, tuple)) else src_model
+ num_experts = src_lm.config.num_moe_experts
+ # Unwrap to get owning modules (with parameters and pg_collection)
+ src_core = unwrap_model(src_lm)
+ # Ensure pg_collection exists
+ if not hasattr(src_core, "pg_collection") or src_core.pg_collection is None:
+ raise RuntimeError("Source model missing pg_collection required for reshard")
+ # Fill missing DP group on the source using Megatron's parallel state if not provided
+ if getattr(src_core.pg_collection, "dp", None) is None:
+ src_core.pg_collection.dp = parallel_state.get_data_parallel_group()
+
+ if target_model is not None:
+ # Handle list-wrapped modules
+ tgt_lm = target_model[0] if isinstance(target_model, (list, tuple)) else target_model
+ if num_experts is None:
+ num_experts = tgt_lm.config.num_moe_experts
+ # Unwrap to get owning modules (with parameters and pg_collection)
+ tgt_core = unwrap_model(tgt_lm)
+ # Ensure pg_collection exists
+ if not hasattr(tgt_core, "pg_collection") or tgt_core.pg_collection is None:
+ raise RuntimeError("Target model missing pg_collection required for reshard")
+
+ # Build or retrieve cached plan
+ cache_key = _build_plan_cache_key(src_core, tgt_core, num_experts, group=group)
+
+ if cache_key not in _plan_cache:
+ # All ranks must participate in planning (collective operations)
+ plan = build_centralized_reshard_plan(
+ src_core,
+ tgt_core,
+ num_experts=num_experts,
+ group=group,
+ src_rank_offset=src_rank_offset,
+ dst_rank_offset=dst_rank_offset,
+ )
+ _plan_cache[cache_key] = plan
else:
- plan = cached_plan
+ plan = _plan_cache[cache_key]
- execute_reshard_plan(plan, src_core, tgt_core, service=service)
+ execute_reshard_plan(
+ plan, src_core, tgt_core, service=service, group=group, transform=transform
+ )
diff --git a/megatron/core/resharding/transforms.py b/megatron/core/resharding/transforms.py
new file mode 100644
index 00000000000..45a250d3541
--- /dev/null
+++ b/megatron/core/resharding/transforms.py
@@ -0,0 +1,261 @@
+# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
+from __future__ import annotations
+
+"""
+Reshard transforms for custom send/recv/writeback during weight transfer.
+
+- ReshardTransform: base class for pluggable format conversion hooks.
+- MXFP8ReshardTransform: writes received BF16 data into persistent FlashInfer
+ MXFP8Tensor buffers so CUDA-graph device-pointer captures remain valid.
+"""
+
+import torch
+
+from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+
+
+class ReshardTransform:
+ """Hook for custom send/recv/writeback during reshard execution.
+
+ Implementations override the four methods below. When an instance is
+ passed to ``execute_reshard_plan``, each ``TransferOp`` is checked via
+ ``should_transform(param_name)``; if True the transform methods are used
+ instead of the default send/recv/writeback logic.
+
+ The transform may change the wire format (e.g. send MXFP8 data+scale
+ instead of BF16) **or** keep the same wire format and only post-process
+ on the receive side (e.g. receive BF16, convert to MXFP8 in
+ ``finalize_recv``). The only constraint is that ``prepare_send`` and
+ ``prepare_recv`` must return the same number of tensors for a given
+ parameter so that send/recv pairs match.
+ """
+
+ def should_transform(self, param_name: str) -> bool:
+ """Return True if *param_name* should use the transform path."""
+ return False
+
+ def prepare_send(
+ self, param_name: str, src_slice: tuple[slice, ...], src_param: torch.nn.Parameter
+ ) -> list[torch.Tensor]:
+ """Produce tensor(s) to send for *param_name*.
+
+ May return multiple tensors (e.g. data + scale when converting to
+ MXFP8 on the sender side). The default implementation sends the
+ BF16 slice unchanged (single tensor).
+ """
+ raise NotImplementedError
+
+ def prepare_recv(self, param_name: str, dst_slice: tuple[slice, ...]) -> list[torch.Tensor]:
+ """Allocate receive buffer(s). Count must match ``prepare_send`` output."""
+ raise NotImplementedError
+
+ def finalize_recv(
+ self, param_name: str, dst_slice: tuple[slice, ...], recv_buffers: list[torch.Tensor]
+ ) -> None:
+ """Write received data into final destination (e.g. persistent buffers).
+
+ This is where receiver-side format conversion can happen (e.g.
+ converting a BF16 recv buffer to MXFP8 before writing into
+ persistent storage).
+ """
+ raise NotImplementedError
+
+
+# ---------------------------------------------------------------------------
+# MXFP8 transform helpers
+# ---------------------------------------------------------------------------
+
+
+def _scale_slice_from_data_slice(
+ data_slice: tuple[slice, ...], block_size: int = 32
+) -> tuple[slice, ...]:
+ """Convert an MXFP8 data slice to the corresponding scale slice.
+
+ In MXFP8, each group of ``block_size`` elements along the last (K)
+ dimension shares a single scale value. All dimensions except the last
+ are passed through unchanged; the last ``slice`` has its start/stop
+ divided by ``block_size``. Integer index on the last dim is converted
+ to scale index as idx // block_size.
+ """
+ adjusted = list(data_slice)
+ last = adjusted[-1]
+ if isinstance(last, slice):
+ if last.start is not None and last.start % block_size != 0:
+ raise AssertionError(
+ f"MXFP8 data slice last dim ({last}) must be aligned to block_size={block_size}"
+ )
+ if last.stop is not None and last.stop % block_size != 0:
+ raise AssertionError(
+ f"MXFP8 data slice last dim ({last}) must be aligned to block_size={block_size}"
+ )
+ scale_start = (last.start // block_size) if last.start is not None else None
+ scale_stop = (last.stop // block_size) if last.stop is not None else None
+ # Scale has one value per block; do not use last.step (would index scale wrong).
+ adjusted[-1] = slice(scale_start, scale_stop)
+ elif isinstance(last, int):
+ adjusted[-1] = last // block_size
+ return tuple(adjusted)
+
+
+def _ensure_sendable(param: torch.Tensor) -> torch.Tensor:
+ """Return a standard-dtype tensor suitable for wire transmission.
+
+ Quantized parameter types (e.g., Transformer Engine MXFP8Tensor) are
+ dequantized to their original precision (usually BF16). Standard
+ parameters are returned via ``.data`` (unwrapped from autograd).
+ """
+ try:
+ from transformer_engine.pytorch.tensor.mxfp8_tensor import MXFP8Tensor as _TEMXFP8
+
+ if isinstance(param, _TEMXFP8):
+ return param.dequantize()
+ except ImportError:
+ pass
+ return param.data
+
+
+class MXFP8ReshardTransform(ReshardTransform):
+ """MXFP8 format-conversion transform for reshard.
+
+ Writes received weight data directly into persistent ``MXFP8Tensor``
+ buffers so that CUDA-graph device-pointer captures remain valid across
+ refits.
+
+ Two modes are supported, controlled by *convert_on_send*:
+
+ ``convert_on_send=False`` (default — **receiver-side conversion**):
+ The sender transmits plain BF16 (one tensor per op, identical to the
+ default reshard path). The receiver allocates a BF16 receive buffer,
+ then ``finalize_recv`` converts BF16 → MXFP8 and writes into the
+ persistent buffers. Because the wire format is unchanged the sender
+ does **not** need a transform — only the receiver creates one. This
+ is the simplest mode and avoids any sender/receiver coordination.
+
+ ``convert_on_send=True`` (**sender-side conversion**):
+ The sender converts each BF16 slice to MXFP8 and sends **two**
+ tensors (data + scale) per op. The receiver allocates matching
+ MXFP8 buffers and ``finalize_recv`` copies them directly. Both
+ sender and receiver must use the transform so that tensor counts
+ match. This mode halves wire bandwidth (~1 byte/elem vs 2).
+
+ **Caveat**: CopyService backends that match local (same-rank)
+ transfers by ``task_id`` (Gloo, NVSHMEM) will break if multiple
+ tensors share the same ``task_id``. This mode is therefore only
+ safe for non-colocated setups where sender and receiver are on
+ different ranks. A future fix could generate unique sub-IDs.
+
+ Args:
+ convertible_params: set of fully-qualified parameter names that
+ should use this transform.
+ persistent_buffers: dict mapping parameter names (without
+ *buffer_key_prefix*) to ``MXFP8Tensor`` objects that hold the
+ receiver's persistent data/scale storage. Empty on the sender
+ when using ``convert_on_send=True``.
+ buffer_key_prefix: prefix to strip from ``param_name`` when looking
+ up entries in *persistent_buffers* (e.g. ``"decoder."``).
+ convert_on_send: if True, convert BF16 → MXFP8 on the sender and
+ transmit two tensors (data + scale). If False (default),
+ transmit BF16 and convert on the receiver in ``finalize_recv``.
+ """
+
+ def __init__(
+ self,
+ convertible_params: set[str],
+ persistent_buffers: dict,
+ buffer_key_prefix: str = "",
+ convert_on_send: bool = False,
+ ):
+ self.convertible_params = convertible_params
+ self.persistent_buffers = persistent_buffers
+ self.buffer_key_prefix = buffer_key_prefix
+ self.convert_on_send = convert_on_send
+ # Accumulation buffers for 1D-scale params that arrive in partial slices.
+ # The 1D swizzled FlashInfer scale can't be updated partially; we collect
+ # all BF16 slices here and quantize the full weight once it's assembled.
+ # Maps buf_key -> (full-size BF16 accumulation tensor, elements written so far).
+ self._pending_1d: dict = {}
+
+ def should_transform(self, param_name: str) -> bool:
+ return param_name in self.convertible_params
+
+ # -- send ----------------------------------------------------------------
+
+ def prepare_send(self, param_name, src_slice, src_param):
+ src_data = _ensure_sendable(src_param)
+ if self.convert_on_send:
+
+ bf16_data = src_data[src_slice].contiguous().to(torch.bfloat16)
+ mxfp8 = MXFP8Tensor.from_bf16(bf16_data)
+ return [mxfp8.data.contiguous(), mxfp8.scale.contiguous()]
+ else:
+ # BF16 on the wire — same as the default reshard path.
+ return [src_data[src_slice].contiguous()]
+
+ # -- recv ----------------------------------------------------------------
+
+ def prepare_recv(self, param_name, dst_slice):
+ buf_key = param_name.removeprefix(self.buffer_key_prefix)
+ buf = self.persistent_buffers[buf_key]
+
+ if self.convert_on_send:
+ # Receive MXFP8 data + scale (2 buffers).
+ if buf.scale.ndim == 1:
+ # 1D swizzled scale can't be partially reconstructed from sender-quantized
+ # slices. Use convert_on_send=False for models with 1D-scale params.
+ raise NotImplementedError(
+ f"convert_on_send=True is not supported for parameters with 1D swizzled "
+ f"scale (param={param_name!r}). Use convert_on_send=False instead, which "
+ f"receives BF16 and quantizes the full weight on the receiver."
+ )
+ scale_slice = _scale_slice_from_data_slice(dst_slice)
+ return [
+ torch.empty_like(buf.data[dst_slice].contiguous()),
+ torch.empty_like(buf.scale[scale_slice].contiguous()),
+ ]
+ else:
+ # Receive BF16 (1 buffer, same shape as the MXFP8 data slice).
+ shape = buf.data[dst_slice].shape
+ return [torch.empty(shape, dtype=torch.bfloat16, device=buf.data.device)]
+
+ def finalize_recv(self, param_name, dst_slice, recv_buffers):
+ buf_key = param_name.removeprefix(self.buffer_key_prefix)
+ buf = self.persistent_buffers[buf_key]
+
+ if self.convert_on_send:
+ # Already MXFP8 on the wire — copy data and 2D scale slices directly.
+ # (1D scale is rejected at prepare_recv time, so only 2D reaches here.)
+ buf.data[dst_slice].copy_(recv_buffers[0])
+ scale_slice = _scale_slice_from_data_slice(dst_slice)
+ buf.scale[scale_slice].copy_(recv_buffers[1])
+ elif buf.scale.ndim == 1:
+ # 1D swizzled scale (FlashInfer format) encodes scale values across the
+ # full weight tensor; partial updates would corrupt the swizzle layout.
+ # Accumulate BF16 slices and quantize once all slices are assembled.
+ if buf_key not in self._pending_1d:
+ # Use zeros so that any un-filled slice produces zeros rather than garbage.
+ self._pending_1d[buf_key] = [
+ torch.zeros_like(buf.data, dtype=torch.bfloat16),
+ 0, # elements written so far
+ ]
+ accum, written = self._pending_1d[buf_key]
+ accum[dst_slice].copy_(recv_buffers[0])
+ written += recv_buffers[0].numel()
+ if written >= buf.data.numel():
+ if written != buf.data.numel():
+ raise AssertionError(
+ f"1D-scale param {param_name!r}: received {written} elements, "
+ f"expected {buf.data.numel()} (duplicate or missing slices?)"
+ )
+ mxfp8 = MXFP8Tensor.from_bf16(accum)
+ buf.data.copy_(mxfp8.data)
+ buf.scale.copy_(mxfp8.scale)
+ del self._pending_1d[buf_key]
+ else:
+ self._pending_1d[buf_key][1] = written
+ else:
+ # 2D scale: each scale row covers exactly one data row, so partial
+ # row-wise updates are independent and can be applied immediately.
+ mxfp8 = MXFP8Tensor.from_bf16(recv_buffers[0])
+ buf.data[dst_slice].copy_(mxfp8.data)
+ scale_slice = _scale_slice_from_data_slice(dst_slice)
+ buf.scale[scale_slice].copy_(mxfp8.scale)
diff --git a/megatron/core/resharding/utils.py b/megatron/core/resharding/utils.py
index 7fc9e9ad3a7..c41413103a8 100644
--- a/megatron/core/resharding/utils.py
+++ b/megatron/core/resharding/utils.py
@@ -44,6 +44,11 @@ class ParameterMetadata:
is_tp: bool = False
partition_dim: int = 0
partition_stride: int = 1
+ # For parameters that pack multiple independently-sharded components of
+ # different sizes (e.g. Mamba in_proj packs z, x, B, C, dt). When present,
+ # lists the per-TP-rank block sizes along partition_dim. The refit planner
+ # interleaves these blocks rather than doing a simple contiguous concat.
+ partition_sizes: list[int] | None = None
# EP sharding info (fused/grouped MoE)
is_ep: bool = False
@@ -251,50 +256,106 @@ def extract_param_metadata(
pg_collection,
num_experts: Optional[int] = None,
layer_module_prefix_map: Mapping[str, str] | None = None,
+ rank_offset: int = 0,
+ _rank_list_cache: dict | None = None,
) -> ParameterMetadata:
- """Extract metadata from a parameter for cross-rank communication."""
+ """Extract metadata from a parameter for cross-rank communication.
+
+ Args:
+ _rank_list_cache: Optional dict used to deduplicate rank lists so
+ that params sharing the same process group reuse one object.
+ This dramatically shrinks pickle size when metadata is gathered
+ across many ranks (pickle uses backreferences for same-``id()``
+ objects, avoiding re-serialization of identical group lists).
+ """
# TP flags from attributes (set by Megatron linear layers)
is_tp = bool(getattr(param, 'tensor_model_parallel', False))
partition_dim = int(getattr(param, 'partition_dim', 0))
partition_stride = int(getattr(param, 'partition_stride', 1))
-
- # SwiGLU/GLU compatibility: For gated linear units, fc1 stores interleaved [gate, up] portions
- # and requires partition_stride=2 for correct resharding. New models set this at construction
- # time (MLP sets partition_stride=2 on weight when gated_linear_unit=True). For legacy models
- # where stride=1 was left as default, we apply stride=2 as a fallback for fc1 parameters.
- # This is safe because: (1) gated models need it, and (2) non-gated models have smaller fc1
- # and stride doesn't affect single-block transfers.
- # if 'mlp.linear_fc1' in param_name and is_tp and partition_stride == 1:
- # partition_stride = 2
+ partition_sizes = getattr(param, 'partition_sizes', None)
+ if partition_sizes is not None:
+ partition_sizes = list(partition_sizes)
# EP detection: Megatron convention - expert params are not allreduced
is_ep = not bool(getattr(param, 'allreduce', True))
+ # Expert-param detection for TP inference. When explicit_expert_comm is
+ # active (is_expert and (tp_size>1 or ep)), TE clears parallel_mode so
+ # tensor_model_parallel is never stamped — yet the weight IS TP-sharded
+ # when tp_size > 1. We detect expert params via num_experts + the
+ # per-expert naming convention (weightK / biasK in TEGroupedLinear).
+ is_expert_param = (
+ num_experts is not None and _detect_expert_index_from_param_name(param_name) is not None
+ )
+
tensor_parallel_group_ranks: list[int] | None = None
expert_parallel_group_ranks: list[int] | None = None
data_parallel_group_ranks: list[int] | None = None
pipeline_parallel_group_ranks: list[int] | None = None
- if is_ep:
- expert_parallel_group_ranks = dist.get_process_group_ranks(pg_collection.ep)
- # For MoE params, prefer expert TP group when available, else regular TP
- if is_tp and hasattr(pg_collection, 'expt_tp') and pg_collection.expt_tp is not None:
- tensor_parallel_group_ranks = dist.get_process_group_ranks(pg_collection.expt_tp)
- elif is_tp and hasattr(pg_collection, 'tp') and pg_collection.tp is not None:
- tensor_parallel_group_ranks = dist.get_process_group_ranks(pg_collection.tp)
- data_parallel_group_ranks = dist.get_process_group_ranks(pg_collection.dp)
+ # Deduplicate rank lists: params sharing the same TP/DP/EP/PP group get
+ # one shared list object instead of separate copies. This shrinks pickle
+ # size ~75% when metadata is gathered across many ranks (pickle uses
+ # backreferences for same-id() objects).
+ if _rank_list_cache is None:
+ _rank_list_cache = {}
+
+ def _dedup_ranks(ranks: list[int]) -> list[int]:
+ key = tuple(ranks)
+ if key not in _rank_list_cache:
+ _rank_list_cache[key] = list(key)
+ return _rank_list_cache[key]
+
+ def _offset_ranks(ranks: list[int]) -> list[int]:
+ result = [r + rank_offset for r in ranks] if rank_offset else ranks
+ return _dedup_ranks(result)
+
+ if is_ep or is_expert_param:
+ if is_ep:
+ expert_parallel_group_ranks = _offset_ranks(
+ dist.get_process_group_ranks(pg_collection.ep)
+ )
+ # For expert params, always provide TP group ranks so the planner can
+ # handle TP size transitions (e.g., TP2→TP1). When explicit_expert_comm
+ # clears TE's parallel_mode, tensor_model_parallel may not be set even
+ # though the weight IS TP-sharded. Detect TP via group size instead.
+ expt_tp = getattr(pg_collection, 'expt_tp', None)
+ tp_grp = expt_tp if expt_tp is not None else getattr(pg_collection, 'tp', None)
+ if tp_grp is not None:
+ tp_ranks = _offset_ranks(dist.get_process_group_ranks(tp_grp))
+ tensor_parallel_group_ranks = tp_ranks
+ if not is_tp and len(tp_ranks) > 1:
+ is_tp = True
+ data_parallel_group_ranks = _offset_ranks(dist.get_process_group_ranks(pg_collection.dp))
elif is_tp:
# Non-EP: use regular TP group
if hasattr(pg_collection, 'tp') and pg_collection.tp is not None:
- tensor_parallel_group_ranks = dist.get_process_group_ranks(pg_collection.tp)
- data_parallel_group_ranks = dist.get_process_group_ranks(pg_collection.dp)
+ tensor_parallel_group_ranks = _offset_ranks(
+ dist.get_process_group_ranks(pg_collection.tp)
+ )
+ data_parallel_group_ranks = _offset_ranks(dist.get_process_group_ranks(pg_collection.dp))
else:
- data_parallel_group_ranks = dist.get_process_group_ranks(pg_collection.dp)
+ data_parallel_group_ranks = _offset_ranks(dist.get_process_group_ranks(pg_collection.dp))
+
+ # Always provide TP group ranks so the planner can handle TP size transitions
+ # (e.g., TP2→TP1). When is_tp=False the param is replicated across the TP group,
+ # but the planner still needs to know the TP topology to plan gather/scatter ops
+ # when the *other* side of the reshard IS TP-sharded.
+ if (
+ tensor_parallel_group_ranks is None
+ and hasattr(pg_collection, 'tp')
+ and pg_collection.tp is not None
+ ):
+ tensor_parallel_group_ranks = _offset_ranks(dist.get_process_group_ranks(pg_collection.tp))
if hasattr(pg_collection, 'pp') and pg_collection.pp is not None:
- pipeline_parallel_group_ranks = dist.get_process_group_ranks(pg_collection.pp)
+ pipeline_parallel_group_ranks = _offset_ranks(
+ dist.get_process_group_ranks(pg_collection.pp)
+ )
else:
- pipeline_parallel_group_ranks = list(range(dist.get_world_size()))
+ pipeline_parallel_group_ranks = _dedup_ranks(
+ list(range(rank_offset, rank_offset + dist.get_world_size()))
+ )
meta = ParameterMetadata(
name=param_name,
@@ -304,6 +365,7 @@ def extract_param_metadata(
is_tp=is_tp,
partition_dim=partition_dim,
partition_stride=partition_stride,
+ partition_sizes=partition_sizes,
is_ep=is_ep,
num_experts=num_experts,
owner_rank=owner_rank,
@@ -324,38 +386,132 @@ def select_src_metadata_balanced(
) -> ParameterMetadata:
"""Choose a representative source `ParameterMetadata` for a destination rank.
- Multiple source data-parallel (DP) groups may hold the same logical parameter.
- To avoid always reading from the same group, we:
- - bucket `src_meta_list` by their DP group (tuple of ranks)
- - if there is only one bucket, just return the first entry
- - otherwise, use the destination rank's global rank to select a source
- DP group in a round-robin fashion, ensuring even distribution of load
- across all source DP groups.
+ The selected metadata provides topology information (TP/EP/DP group ranks) that the
+ LCM transfer planner uses to compute actual source ranks and slices. This function
+ doesn't perform transfers itself - it just picks which source configuration to use
+ as reference for planning.
+
+ Two scenarios for EP-sharded parameters:
+ 1. Non-collocated mode (same EP size, different rank numbering):
+ - Filter by matching EP local rank to pair ranks with same expert position
+ - Example: src ranks [0-63] and dst ranks [64-127] both with EP=8
+ - Dst EP local 0 should use src EP local 0 as reference (same experts)
+
+ 2. Resharding mode (different EP sizes):
+ - Skip EP local rank filtering (sizes don't correspond)
+ - Example: EP=8→EP=16 means dst EP local 8 has no matching src EP local
+ - Expert matching handled by resolved_name; LCM handles TP dimension changes
+
+ Finally, balances across data-parallel (DP) groups to distribute load:
+ - Groups src_meta_list by DP group
+ - Selects source DP group via round-robin: dst_rank % num_src_dp_groups
+ - Ensures even distribution of transfer load across source DP replicas
"""
if not src_meta_list:
raise ValueError("src_meta_list must be non-empty")
- # Group source metadata by their DP group layout so we can balance across groups.
- # (dp_rank0, dp_rank1, ...) -> [ParameterMetadata for that DP group]
+ # ============================================================================
+ # EXPERT PARALLELISM (EP) LOCAL RANK FILTERING
+ # ============================================================================
+ # Purpose: In non-collocated mode with same EP size, ensure destination ranks
+ # use source metadata from ranks with the same EP local position (same experts).
+ #
+ # Why size check matters:
+ # - Same size (EP=8→EP=8): Local ranks 0-7 exist in both src and dst
+ # → Filter ensures dst EP local 0 uses src EP local 0 (same global experts)
+ # - Different size (EP=8→EP=16): Local ranks 0-15 in dst, only 0-7 in src
+ # → Dst EP local 8 has no corresponding src EP local rank
+ # → Skip filter; expert reassignment handled by resolved_name matching
+ #
+ # Expert routing: When EP size changes, each expert parameter is matched via
+ # resolved_name (which includes global expert index). The LCM/TP planner
+ # handles any TP dimension changes, and DP round-robin distributes load.
+ # ============================================================================
+ dst_ep_group = dst_metadata.expert_parallel_group_ranks
+ if dst_ep_group is not None:
+ dst_ep_local = dst_ep_group.index(dst_rank)
+ # Check if EP sizes match between source and destination
+ src_ep_size = (
+ len(src_meta_list[0].expert_parallel_group_ranks)
+ if src_meta_list[0].expert_parallel_group_ranks
+ else None
+ )
+ dst_ep_size = len(dst_ep_group)
+
+ # Only filter by EP local rank when sizes match (non-collocated, not resharding)
+ if src_ep_size == dst_ep_size:
+ matching_ep = [
+ m
+ for m in src_meta_list
+ if m.expert_parallel_group_ranks
+ and m.expert_parallel_group_ranks.index(m.owner_rank) == dst_ep_local
+ ]
+ if not matching_ep:
+ # This indicates a configuration bug: sizes match but no local rank match
+ def _ep_local(m):
+ return (
+ m.expert_parallel_group_ranks.index(m.owner_rank)
+ if m.expert_parallel_group_ranks
+ else None
+ )
+
+ available = [(m.owner_rank, _ep_local(m)) for m in src_meta_list]
+ raise ValueError(
+ f"No source metadata with EP local rank {dst_ep_local}"
+ f" found for dst rank {dst_rank}. Available: {available}"
+ )
+ src_meta_list = matching_ep
+ # else: EP resharding mode (sizes differ) - skip filter, keep all source candidates
+
+ # ============================================================================
+ # LOCAL COPY OPTIMIZATION (COLLOCATED MODE)
+ # ============================================================================
+ # In collocated mode, prefer local copies when available. If dst_rank appears
+ # in the source metadata list (after TP/EP filtering), use it directly to
+ # avoid unnecessary data transfers.
+ #
+ # A local copy is essentially free
+ # (tensor.copy_() on same GPU), while any remote transfer incurs significant
+ # overhead even within the same node.
+ # ============================================================================
+ local_meta = [m for m in src_meta_list if m.owner_rank == dst_rank]
+ if local_meta:
+ # Found local metadata - use it for a free local copy
+ return local_meta[0]
+
+ # ============================================================================
+ # DATA PARALLELISM (DP) LOAD BALANCING
+ # ============================================================================
+ # After TP/EP filtering (if applicable), balance transfer load across source
+ # data-parallel replicas. Each DP group holds a complete copy of the model,
+ # so we can read from any DP group - choosing via round-robin spreads load.
+ #
+ # Load distribution: dst_rank % num_src_dp_groups ensures even distribution
+ # even when destination has different DP configuration than source.
+ # ============================================================================
grouped_by_dp: dict[tuple[int, ...], list[ParameterMetadata]] = {}
for meta in src_meta_list:
dp_group = tuple(meta.data_parallel_group_ranks or [])
grouped_by_dp.setdefault(dp_group, []).append(meta)
- # Fast path: only one DP layout present; no balancing necessary.
+ # Fast path: only one DP group present; no balancing necessary
if len(grouped_by_dp) == 1:
return src_meta_list[0]
- # Use the destination rank's global rank to select a source DP group in a
- # round-robin fashion. This ensures that even when multiple destination ranks
- # have the same DP index (e.g., ranks 0,1,2,3 all being at position 0 in their
- # respective DP groups), they still get distributed across different source
- # DP groups based on their global rank.
+ # Round-robin selection across source DP groups based on destination global rank
+ # This ensures even distribution: if we have 4 src DP groups and 128 dst ranks,
+ # each src DP group will be selected by 32 dst ranks (128 / 4 = 32)
sorted_dp_groups = sorted(grouped_by_dp.keys())
chosen_group = sorted_dp_groups[dst_rank % len(sorted_dp_groups)]
- # Within the chosen group, any representative metadata works; use the first.
- return grouped_by_dp[chosen_group][0]
+ # Within the chosen DP group, distribute across available metadata entries
+ # to balance load across all TP groups in the DP replica.
+ # Example: With 4 TP groups in a DP group, dst_ranks will cycle through all 4
+ # instead of always using the first one, better distributing transfer load.
+ group_metadata = grouped_by_dp[chosen_group]
+ within_group_idx = (dst_rank // len(sorted_dp_groups)) % len(group_metadata)
+ selected = group_metadata[within_group_idx]
+ return selected
logger = logging.getLogger(__name__)
diff --git a/megatron/core/ssm/gated_delta_net.py b/megatron/core/ssm/gated_delta_net.py
index 70e749724dc..e23599689bb 100644
--- a/megatron/core/ssm/gated_delta_net.py
+++ b/megatron/core/ssm/gated_delta_net.py
@@ -174,8 +174,10 @@ def __init__(
dtype=config.params_dtype,
)
setattr(self.conv1d.weight, "tensor_model_parallel", True)
+ setattr(self.conv1d.weight, "partition_dim", 0)
if conv_bias:
setattr(self.conv1d.bias, "tensor_model_parallel", True)
+ setattr(self.conv1d.bias, "partition_dim", 0)
# Time step projection (discretization)
self.num_v_heads_local_tp = self.num_value_heads // self.tp_size
@@ -188,6 +190,7 @@ def __init__(
)
)
setattr(self.dt_bias, "tensor_model_parallel", True)
+ setattr(self.dt_bias, "partition_dim", 0)
# A_log parameter
self.A_log = nn.Parameter(
torch.empty(
@@ -197,6 +200,7 @@ def __init__(
)
)
setattr(self.A_log, "tensor_model_parallel", True)
+ setattr(self.A_log, "partition_dim", 0)
# Output layernorm before projection
self.out_norm = build_module(
@@ -508,6 +512,19 @@ def sharded_state_dict(self, prefix="", sharded_offsets=(), metadata=None, tp_gr
return sharded_state_dict
+ def backward_dw(self):
+ """Execute weight gradient computation for all linear layers."""
+ self._backward_in_proj()
+ self._backward_out_proj()
+
+ def _backward_in_proj(self):
+ """Computes weight gradients of input projection layer."""
+ self.in_proj.backward_dw()
+
+ def _backward_out_proj(self):
+ """Computes weight gradients of output projection layer."""
+ self.out_proj.backward_dw()
+
def _split_tensor_factory(
orig_sh_ten: ShardedTensor, split_sections: List[int], split_names: List[str], split_dim: int
diff --git a/megatron/core/ssm/mamba_block.py b/megatron/core/ssm/mamba_block.py
index ffb7b8f6fdb..f42f3542c3d 100644
--- a/megatron/core/ssm/mamba_block.py
+++ b/megatron/core/ssm/mamba_block.py
@@ -16,16 +16,16 @@
from megatron.core.dist_checkpointing.utils import replace_prefix_for_sharding
from megatron.core.enums import Fp8Recipe
from megatron.core.extensions.transformer_engine import TENorm
+from megatron.core.fp4_utils import get_fp4_context
from megatron.core.fp8_utils import get_fp8_context
from megatron.core.inference.contexts import BaseInferenceContext
from megatron.core.packed_seq_params import PackedSeqParams
from megatron.core.process_groups_config import ProcessGroupCollection
from megatron.core.ssm.mamba_hybrid_layer_allocation import Symbols as LayerSymbols
-from megatron.core.ssm.mamba_hybrid_layer_allocation import allocate_layers
from megatron.core.transformer import TransformerConfig
from megatron.core.transformer.enums import CudaGraphScope
from megatron.core.transformer.identity_op import IdentityOp
-from megatron.core.transformer.module import MegatronModule
+from megatron.core.transformer.module import GraphableMegatronModule, MegatronModule
from megatron.core.transformer.spec_utils import ModuleSpec, build_module
from megatron.core.transformer.transformer_layer import TransformerLayer
from megatron.core.transformer.utils import sharded_state_dict_default
@@ -39,29 +39,27 @@ class MambaStackSubmodules:
"""
mamba_layer: Union[ModuleSpec, type] = IdentityOp
+ gdn_layer: Union[ModuleSpec, type] = IdentityOp
attention_layer: Union[ModuleSpec, type] = IdentityOp
mlp_layer: Union[ModuleSpec, type] = IdentityOp
moe_layer: Union[ModuleSpec, type] = IdentityOp
mtp_block_spec: Optional[ModuleSpec] = None
-class MambaStack(MegatronModule):
+class MambaStack(GraphableMegatronModule, MegatronModule):
"""
Constructor for the MambaStack class.
Args:
config (TransformerConfig): the model configuration
submodules (MambaStackSubmodules): the submodules for the stack
- residual_in_fp32 (bool, optional): whether to do residual connections
- in fp32. Defaults to False.
pre_process (bool, optional): whether to include an embedding layer.
Defaults to True.
- hybrid_attention_ratio (float, optional): the target ratio of attention layers to
- total layers. Defaults to 0.0.
- hybrid_mlp_ratio (float, optional): the target ratio of mlp layers to total
- layers. Defaults to 0.0.
- hybrid_override_pattern (str, optional): the hybrid layer pattern to override
- with. Defaults to None.
+ layer_type_list (list, optional): pre-computed list of layer type symbols for
+ this pipeline segment. When provided (by MambaModel), pipeline stage
+ selection has already been done via '|' separators in the pattern.
+ pp_layer_offset (int, optional): the global layer offset for this pipeline
+ segment. Defaults to 0.
post_layer_norm (bool, optional): whether to include a final layer norm.
Defaults to True.
post_process (bool, optional): whether to include an output layer.
@@ -70,17 +68,16 @@ class MambaStack(MegatronModule):
dtype (optional): the data type to use. Defaults to None.
pg_collection (ProcessGroupCollection): the required model communication
process groups to use.
+ is_mtp_layer (bool, optional): whether this is an MTP layer. Defaults to False.
"""
def __init__(
self,
config: TransformerConfig,
submodules: MambaStackSubmodules,
- residual_in_fp32=False,
pre_process: bool = True,
- hybrid_attention_ratio: float = 0.0,
- hybrid_mlp_ratio: float = 0.0,
- hybrid_override_pattern: str = None,
+ layer_type_list: Optional[list[str]] = None,
+ pp_layer_offset: int = 0,
post_layer_norm: bool = True,
post_process: bool = True,
device=None,
@@ -89,7 +86,6 @@ def __init__(
is_mtp_layer: bool = False,
) -> None:
super().__init__(config=config)
- self.residual_in_fp32 = residual_in_fp32
self.pre_process = pre_process
self.post_layer_norm = post_layer_norm
self.post_process = post_process
@@ -102,73 +98,67 @@ def __init__(
# Required for pipeline parallel schedules
self.input_tensor = None
-
- self.hybrid_attention_ratio = hybrid_attention_ratio
- self.hybrid_mlp_ratio = hybrid_mlp_ratio
- self.hybrid_override_pattern = hybrid_override_pattern
self.pg_collection = pg_collection
- # For MTP layers, always use pattern length (config.num_layers is for main decoder)
- if self.is_mtp_layer:
- num_layers_for_allocation = len(self.hybrid_override_pattern)
- else:
- num_layers_for_allocation = (
- self.config.num_layers
- if self.config.num_layers is not None
- else len(self.hybrid_override_pattern)
- )
-
- self.layer_type_list = allocate_layers(
- num_layers_for_allocation,
- self.hybrid_attention_ratio,
- self.hybrid_mlp_ratio,
- self.hybrid_override_pattern,
- silent=self.is_mtp_layer,
+ assert layer_type_list is not None, (
+ "layer_type_list must be provided. It should be pre-computed from "
+ "--hybrid-layer-pattern by MambaModel."
)
+ self.layer_type_list = layer_type_list
- pp_layer_offset = 0
- if self.pp_group.size() > 1 and not self.is_mtp_layer:
- pp_layer_offset, self.layer_type_list = self._select_layers_for_pipeline_parallel(
- self.layer_type_list
- )
- # Build main decoder layers using shared layer builder
+ # Build layers from the pre-selected segment
self.layers = nn.ModuleList()
for i, layer_type in enumerate(self.layer_type_list):
- fp8_init_context = get_fp8_context(self.config, i + pp_layer_offset, is_init=True)
- with fp8_init_context:
+ layer_number = i + 1 + pp_layer_offset
+ if self.config.fp8:
+ quant_init_context = get_fp8_context(self.config, i + pp_layer_offset, is_init=True)
+ elif self.config.fp4:
+ quant_init_context = get_fp4_context(self.config, i + pp_layer_offset, is_init=True)
+ else:
+ quant_init_context = nullcontext()
+ with quant_init_context:
if layer_type == LayerSymbols.MAMBA:
layer = build_module(
submodules.mamba_layer,
config=self.config,
- residual_in_fp32=residual_in_fp32,
- layer_number=i + 1 + pp_layer_offset,
+ layer_number=layer_number,
pp_layer_offset=pp_layer_offset,
pg_collection=pg_collection,
)
elif layer_type == LayerSymbols.ATTENTION:
- # Transformer layers apply their own pp_layer_offset
layer = build_module(
submodules.attention_layer,
config=self.config,
- layer_number=i + 1,
+ layer_number=layer_number,
pg_collection=pg_collection,
is_mtp_layer=is_mtp_layer,
+ add_layer_offset=False,
+ pp_layer_offset=pp_layer_offset,
)
elif layer_type == LayerSymbols.MLP:
- # MLP layers apply their own pp_layer_offset
layer = build_module(
submodules.mlp_layer,
config=self.config,
- layer_number=i + 1,
+ layer_number=layer_number,
pg_collection=pg_collection,
+ add_layer_offset=False,
)
elif layer_type == LayerSymbols.MOE:
- # MoE layers apply their own pp_layer_offset
layer = build_module(
submodules.moe_layer,
config=self.config,
- layer_number=i + 1,
+ layer_number=layer_number,
pg_collection=pg_collection,
+ add_layer_offset=False,
+ )
+ elif layer_type == LayerSymbols.GDN:
+ layer = build_module(
+ submodules.gdn_layer,
+ config=self.config,
+ layer_number=layer_number,
+ pg_collection=pg_collection,
+ # Set to False as we do not want to change offset.
+ add_layer_offset=False,
)
else:
assert False, "unexpected layer_type"
@@ -185,19 +175,6 @@ def __init__(
eps=self.config.layernorm_epsilon,
)
- def _select_layers_for_pipeline_parallel(self, layer_type_list):
- num_layers_per_pipeline_rank = self.config.num_layers // self.pp_group.size()
-
- assert self.config.virtual_pipeline_model_parallel_size is None, (
- "The Mamba hybrid model does not currently support "
- "virtual/interleaved pipeline parallelism"
- )
-
- offset = self.pp_group.rank() * num_layers_per_pipeline_rank
- selected_list = layer_type_list[offset : offset + num_layers_per_pipeline_rank]
-
- return offset, selected_list
-
def set_input_tensor(self, input_tensor: Tensor):
"""Set input tensor to be used instead of forward()'s input.
@@ -218,6 +195,39 @@ def mamba_state_shapes_per_request(self) -> Optional[Tuple[Tuple[int], Tuple[int
return layer.mamba_state_shapes_per_request()
return None
+ def _should_call_local_cudagraph(self, *args, **kwargs):
+ """
+ Check if we should call the local cudagraph path.
+ """
+ if (
+ not self.training
+ and hasattr(self, 'cudagraph_manager')
+ and kwargs['attention_mask'] is None
+ and (
+ kwargs.get('inference_context') is not None
+ or kwargs.get('inference_params') is not None
+ )
+ and CudaGraphScope.full_iteration_inference in self.config.cuda_graph_scope
+ ):
+ if kwargs['inference_context'].is_static_batching():
+ using_cuda_graph = kwargs['inference_context'].is_decode_only()
+ else:
+ using_cuda_graph = kwargs['inference_context'].using_cuda_graph_this_step()
+
+ if using_cuda_graph:
+ return True
+ return False
+
+ def __call__(self, *args, **kwargs):
+ if self._should_call_local_cudagraph(*args, **kwargs):
+ kwargs['hidden_states'] = (
+ kwargs['hidden_states'].unwrap()
+ if isinstance(kwargs['hidden_states'], WrappedTensor)
+ else kwargs['hidden_states']
+ )
+ return super().__call__(*args, **kwargs)[0]
+ return super().__call__(*args, **kwargs)
+
def forward(
self,
hidden_states: Union[Tensor, WrappedTensor],
@@ -292,16 +302,29 @@ def forward(
# control which layer will be fp8 or bf16
use_outer_fp8_context = self.config.fp8 and self.config.fp8_recipe == Fp8Recipe.delayed
use_inner_fp8_context = self.config.fp8 and self.config.fp8_recipe != Fp8Recipe.delayed
+ use_fp4_context = self.config.fp4 is not None
outer_fp8_context = get_fp8_context(self.config) if use_outer_fp8_context else nullcontext()
+ if use_inner_fp8_context:
+
+ def get_inner_quant_context(config, layer_number):
+ return get_fp8_context(config, layer_number)
+
+ elif use_fp4_context:
+
+ def get_inner_quant_context(config, layer_number):
+ return get_fp4_context(config, layer_number)
+
+ else:
+
+ def get_inner_quant_context(config, layer_number):
+ return nullcontext()
+
with outer_fp8_context:
for layer in self.layers:
- inner_fp8_context = (
- get_fp8_context(self.config, layer.layer_number - 1)
- if use_inner_fp8_context
- else nullcontext()
- )
- with inner_fp8_context:
+ # Layers have 1-indexed layer numbers attribute.
+ inner_quant_context = get_inner_quant_context(self.config, layer.layer_number - 1)
+ with inner_quant_context:
if isinstance(layer, TransformerLayer):
hidden_states, _ = layer(
hidden_states=hidden_states,
@@ -312,7 +335,7 @@ def forward(
packed_seq_params=packed_seq_params,
padding_mask=padding_mask,
)
- else: # MambaLayer
+ else: # MambaLayer, Expert, or MLP
hidden_states = layer(
hidden_states=hidden_states,
attention_mask=attention_mask,
diff --git a/megatron/core/ssm/mamba_hybrid_layer_allocation.py b/megatron/core/ssm/mamba_hybrid_layer_allocation.py
index d7002b2915d..1cb1e4a31d1 100644
--- a/megatron/core/ssm/mamba_hybrid_layer_allocation.py
+++ b/megatron/core/ssm/mamba_hybrid_layer_allocation.py
@@ -4,34 +4,24 @@
from dataclasses import dataclass
from typing import Dict, List, Optional, Tuple
-if __name__ != "__main__":
- from megatron.core.utils import log_single_rank
-else:
- from typing import Any
-
- import torch
-
- def log_single_rank(logger: logging.Logger, *args: Any, rank: int = 0, **kwargs: Any):
- """Logs a message to the given rank."""
- if torch.distributed.is_initialized():
- if torch.distributed.get_rank() == rank:
- logger.log(*args, **kwargs)
- else:
- logger.log(*args, **kwargs)
+import torch
+from megatron.core.utils import log_on_each_pipeline_stage, log_single_rank
logger = logging.getLogger(__name__)
class Symbols:
- """Symbols for different layer types."""
+ """Symbols for different layer types and pattern separators."""
MAMBA = "M"
+ GDN = 'G'
ATTENTION = "*"
MLP = "-"
MOE = 'E'
+ PIPE = '|'
MTP_SEPARATOR = "/"
- VALID = {MAMBA, ATTENTION, MLP, MOE}
+ VALID_LAYERS = {MAMBA, GDN, ATTENTION, MLP, MOE}
@dataclass
@@ -39,7 +29,9 @@ class ParsedHybridPattern:
"""Result of parsing a unified hybrid pattern string.
A unified pattern encodes both the main decoder pattern and the MTP pattern
- in a single string using "/" as a separator.
+ in a single string using "/" as a separator. The main pattern may also
+ contain "|" pipe symbols to define pipeline stage boundaries for flexible
+ virtual pipeline parallelism (fVPP).
Format: "///..."
@@ -47,12 +39,15 @@ class ParsedHybridPattern:
- "M*M*" -> main="M*M*", mtp=None, depths=0 (no MTP)
- "M*M*/MM/MM" -> main="M*M*", mtp="MM", depths=2
- "MMMM/*M/*M/*M" -> main="MMMM", mtp="*M", depths=3
+ - "M-M-|M-M*-/MM/MM" -> main="M-M-|M-M*-" (2 PP stages), mtp="MM", depths=2
The "/" symbol introduces MTP patterns. Each repeated pattern after the main
decoder represents one MTP prediction depth.
+ The "|" symbol in the main pattern defines pipeline stage boundaries.
+
Attributes:
- main_pattern: The main decoder layer pattern (e.g., "M*M*")
+ main_pattern: The main decoder layer pattern (e.g., "M*M*" or "M-M-|M-M*-")
mtp_pattern: The MTP layer pattern per depth (e.g., "MM"), or None if no MTP
mtp_num_depths: Number of MTP prediction depths (0 if no MTP)
"""
@@ -62,12 +57,145 @@ class ParsedHybridPattern:
mtp_num_depths: int
+def pattern_from_ratios(
+ num_layers: int, attention_ratio: float = 0.0, mlp_ratio: float = 0.0
+) -> str:
+ """Convert deprecated ratio arguments to a layer pattern string.
+
+ Generates an evenly-spaced hybrid layer pattern from target attention and MLP
+ ratios. This exists for backward compatibility with code that uses the deprecated
+ hybrid_attention_ratio and hybrid_mlp_ratio parameters.
+
+ Args:
+ num_layers: Total number of layers.
+ attention_ratio: Target ratio of attention layers to total layers.
+ mlp_ratio: Target ratio of MLP layers to total layers.
+
+ Returns:
+ A layer pattern string (e.g., "MMM*MMM*MM").
+ """
+ assert num_layers > 0
+ assert 0.0 <= attention_ratio <= 1.0
+ assert 0.0 <= mlp_ratio <= 1.0
+ assert attention_ratio + mlp_ratio <= 1.0
+
+ # Allocate attention layers (evenly spaced, starting and ending with mamba)
+ attention_count = round(num_layers * attention_ratio)
+ mamba_count = num_layers - attention_count
+ sections = attention_count + 1
+ section_len = mamba_count / sections
+
+ layer_types = [Symbols.MAMBA] * num_layers
+ x = section_len
+ for i in range(num_layers):
+ if x < 0.5:
+ layer_types[i] = Symbols.ATTENTION
+ x += section_len
+ else:
+ x -= 1
+
+ # Allocate MLP layers (evenly distributed, not replacing attention)
+ mlp_count = round(num_layers * mlp_ratio)
+ if mlp_count > 0:
+ mamba_count -= mlp_count
+ ratio = mamba_count / mlp_count
+ x = ratio
+ for i in range(num_layers):
+ if layer_types[i] == Symbols.MAMBA:
+ if x < 0.5:
+ layer_types[i] = Symbols.MLP
+ x += ratio
+ else:
+ x -= 1
+
+ return ''.join(layer_types)
+
+
+def get_hybrid_total_layer_count(pattern: str) -> int:
+ """Returns the total number of main decoder layers in a hybrid layer pattern.
+
+ Extracts the main pattern (before the first MTP separator '/'), strips
+ pipeline stage separators '|', and returns the character count.
+
+ Args:
+ pattern: Full hybrid layer pattern, possibly including MTP and pipe separators.
+
+ Returns:
+ Total number of layers in the main decoder pattern.
+ """
+ main_pattern = pattern.split(Symbols.MTP_SEPARATOR)[0]
+ _validate_pattern(main_pattern, "main", allow_pipe=True)
+ return len(main_pattern.replace(Symbols.PIPE, ''))
+
+
+def get_hybrid_total_pipeline_segment_count(pattern: str) -> int:
+ """Returns the number of pipeline segments in a hybrid layer pattern.
+
+ Extracts the main pattern (before the first MTP separator '/') and counts
+ the number of segments delimited by '|'.
+
+ Args:
+ pattern: Full hybrid layer pattern, possibly including MTP and pipe separators.
+
+ Returns:
+ Number of pipeline segments (pipe count + 1).
+ """
+ main_pattern = pattern.split(Symbols.MTP_SEPARATOR)[0]
+ return main_pattern.count(Symbols.PIPE) + 1
+
+
+def get_hybrid_layer_counts(pattern: str) -> Dict[str, int]:
+ """Count layers by type across the full hybrid pattern (main + MTP).
+
+ Parses the pattern to extract main and MTP components, then counts
+ each layer type. Main pattern '|' separators are skipped. MTP layers
+ are counted once per MTP depth.
+
+ Args:
+ pattern: Full hybrid layer pattern string.
+
+ Returns:
+ Dictionary mapping layer symbol to count. Keys are Symbols.MAMBA,
+ Symbols.GDN, Symbols.ATTENTION, Symbols.MLP, and Symbols.MOE.
+
+ Examples:
+ >>> get_hybrid_layer_counts("M*M*")
+ {'M': 2, 'G': 0, '*': 2, '-': 0, 'E': 0}
+
+ >>> get_hybrid_layer_counts("M-M-|M-M*-/MM/MM")
+ {'M': 8, 'G': 0, '*': 1, '-': 4, 'E': 0}
+ """
+ parsed = parse_hybrid_pattern(pattern)
+ counts = {
+ Symbols.MAMBA: 0,
+ Symbols.GDN: 0,
+ Symbols.ATTENTION: 0,
+ Symbols.MLP: 0,
+ Symbols.MOE: 0,
+ }
+
+ # Count main decoder layers (skip '|' pipe separators)
+ if parsed.main_pattern:
+ for char in parsed.main_pattern:
+ if char in counts:
+ counts[char] += 1
+
+ # Count MTP layers (pattern repeated mtp_num_depths times)
+ if parsed.mtp_pattern and parsed.mtp_num_depths > 0:
+ for char in parsed.mtp_pattern:
+ if char in counts:
+ counts[char] += parsed.mtp_num_depths
+
+ return counts
+
+
def parse_hybrid_pattern(pattern: Optional[str]) -> ParsedHybridPattern:
"""Parse a unified hybrid pattern string into main and MTP components.
The pattern uses "/" as a separator between the main decoder pattern and
MTP patterns. Each MTP pattern after the separator represents one prediction
- depth.
+ depth. The main pattern may contain "|" pipe symbols for pipeline stage
+ boundaries.
Format: "///..."
@@ -90,6 +218,9 @@ def parse_hybrid_pattern(pattern: Optional[str]) -> ParsedHybridPattern:
>>> parse_hybrid_pattern("MMMM/*M/*M/*M")
ParsedHybridPattern(main_pattern="MMMM", mtp_pattern="*M", mtp_num_depths=3)
+
+ >>> parse_hybrid_pattern("M-M-|M-M*-/MM/MM")
+ ParsedHybridPattern(main_pattern="M-M-|M-M*-", mtp_pattern="MM", mtp_num_depths=2)
"""
if pattern is None:
return ParsedHybridPattern(main_pattern=None, mtp_pattern=None, mtp_num_depths=0)
@@ -99,13 +230,13 @@ def parse_hybrid_pattern(pattern: Optional[str]) -> ParsedHybridPattern:
if len(parts) == 1:
# No MTP separator found - pattern is main decoder only
main_pattern = parts[0]
- _validate_pattern(main_pattern, "main")
+ _validate_pattern(main_pattern, "main", allow_pipe=True)
return ParsedHybridPattern(main_pattern=main_pattern, mtp_pattern=None, mtp_num_depths=0)
# First part is main decoder pattern
main_pattern = parts[0]
if main_pattern:
- _validate_pattern(main_pattern, "main")
+ _validate_pattern(main_pattern, "main", allow_pipe=True)
# Remaining parts are MTP patterns (one per depth)
mtp_parts = parts[1:]
@@ -126,7 +257,7 @@ def parse_hybrid_pattern(pattern: Optional[str]) -> ParsedHybridPattern:
f"Full pattern: '{pattern}'"
)
- _validate_pattern(mtp_pattern, "MTP")
+ _validate_pattern(mtp_pattern, "MTP", allow_pipe=False)
return ParsedHybridPattern(
main_pattern=main_pattern if main_pattern else None,
@@ -135,210 +266,219 @@ def parse_hybrid_pattern(pattern: Optional[str]) -> ParsedHybridPattern:
)
-def _validate_pattern(pattern: str, pattern_name: str) -> None:
+def _validate_pattern(pattern: str, pattern_name: str, allow_pipe: bool = False) -> None:
"""Validate that a pattern contains only valid layer symbols.
Args:
pattern: Layer pattern string to validate
pattern_name: Name of pattern for error messages (e.g., "main" or "MTP")
+ allow_pipe: Whether to allow the pipe '|' separator (for main patterns)
Raises:
ValueError: If pattern contains invalid symbols
"""
+ valid_chars = Symbols.VALID_LAYERS | {Symbols.PIPE} if allow_pipe else Symbols.VALID_LAYERS
for char in pattern:
- if char not in Symbols.VALID:
+ if char not in valid_chars:
raise ValueError(
f"In {pattern_name} pattern, '{char}' is not a valid layer symbol. "
- f"Valid symbols are: {Symbols.VALID}"
+ f"Valid symbols are: {valid_chars}"
)
-def _allocate_auto(
- total_layers_count: int, target_attention_ratio: float, target_mlp_ratio: float
-) -> list:
- # First, allocate attention (evenly spaced, starting and ending with mamba)
- attention_layers_count: int = round(total_layers_count * target_attention_ratio)
- mamba_layers_count: int = total_layers_count - attention_layers_count
- mamba_sections_count: int = attention_layers_count + 1
- mamba_section_length: float = mamba_layers_count / mamba_sections_count
+def validate_segment_layers(segment: str) -> List[str]:
+ """Validate and convert a single pipeline segment pattern to a layer type list.
- layer_type_list = [Symbols.MAMBA] * total_layers_count
- x: float = mamba_section_length
- for l in range(total_layers_count):
- if x < 0.5:
- layer_type_list[l] = Symbols.ATTENTION
- x += mamba_section_length
- else:
- x -= 1
+ This is used after the main pattern has been split by '|' into segments.
+ Each segment should contain only valid layer symbols (no '|').
- # Next, allocate mlp
- # (evenly distributed, but right-justified, not replacing attention)
- mlp_layers_count: int = round(total_layers_count * target_mlp_ratio)
- if mlp_layers_count > 0:
- mamba_layers_count -= mlp_layers_count
- mamba_to_mlp_ratio: float = mamba_layers_count / mlp_layers_count
+ Args:
+ segment: A single pipeline segment pattern string (e.g., "M-M*-")
- x: float = mamba_to_mlp_ratio
- for l in range(total_layers_count):
- if layer_type_list[l] == Symbols.MAMBA:
- if x < 0.5:
- layer_type_list[l] = Symbols.MLP
- x += mamba_to_mlp_ratio
- else:
- x -= 1
+ Returns:
+ List of layer type characters.
+ Raises:
+ ValueError: If segment contains invalid layer symbols.
+ """
+ layer_type_list = list(segment)
+ for layer_char in layer_type_list:
+ if layer_char not in Symbols.VALID_LAYERS:
+ raise ValueError(
+ f"In hybrid layer pattern segment, '{layer_char}' is not "
+ f"one of {Symbols.VALID_LAYERS}"
+ )
return layer_type_list
-def _allocate_override(total_layers_count: int, override_pattern: str) -> list:
- layer_type_list = list(override_pattern)
- override_pattern_length = len(layer_type_list)
- if override_pattern_length != total_layers_count:
- raise ValueError(
- "The hybrid override pattern is the wrong "
- f"length: got {override_pattern_length}, expected "
- f"{total_layers_count}"
- )
- for l in layer_type_list:
- if l not in Symbols.VALID:
- raise ValueError(f"In hybrid override pattern, '{l}' is not one of {Symbols.VALID}")
+def select_pipeline_segment(
+ main_pattern: str,
+ pp_group: Optional[torch.distributed.ProcessGroup],
+ vp_stage: Optional[int],
+ first_stage_layers: Optional[int] = None,
+ last_stage_layers: Optional[int] = None,
+) -> Tuple[List[str], int]:
+ """Select and validate the pipeline segment for the given PP rank and VP stage.
- return layer_type_list
+ When the main pattern contains '|' pipe separators, splits by '|' into
+ pipeline segments and selects the segment for the current PP rank / VP stage.
+ When the pattern has no pipes but pp_size > 1, falls back to runtime layer
+ slicing (for backwards compatibility), supporting both even and uneven PP splits
+ via first_stage_layers / last_stage_layers.
-def _layer_counts_match(a: list, b: list) -> bool:
- for s in Symbols.VALID:
- if a.count(s) != b.count(s):
- return False
- return True
-
-
-def allocate_layers(
- total_layers_count: int,
- target_attention_ratio: float,
- target_mlp_ratio: float,
- override_pattern: str = None,
- silent: bool = False,
-) -> list:
- """Allocates layers according to the requested distribution of layer types."""
- assert total_layers_count > 0
- assert target_attention_ratio >= 0.0 and target_attention_ratio <= 1.0
- assert target_mlp_ratio >= 0.0 and target_mlp_ratio <= 1.0
- assert target_attention_ratio + target_mlp_ratio <= 1.0
- maybe_log_single_rank = (lambda *args, **kwargs: None) if silent else log_single_rank
- # Note: target_mamba_ratio = 1.0 - target_attention_ratio - target_mlp_ratio
-
- layer_type_list = _allocate_auto(total_layers_count, target_attention_ratio, target_mlp_ratio)
-
- if override_pattern is not None:
- layer_type_list_override = _allocate_override(total_layers_count, override_pattern)
- maybe_log_single_rank(logger, logging.INFO, "Using hybrid override pattern")
- if (target_attention_ratio > 0.0 or target_mlp_ratio > 0.0) and not _layer_counts_match(
- layer_type_list_override, layer_type_list
- ):
+ Args:
+ main_pattern: Main decoder pattern (may contain '|' separators).
+ Empty string is allowed (produces one empty segment).
+ pp_group: Pipeline parallel process group, or None if not using PP.
+ vp_stage: Virtual pipeline stage, or None if not using VPP.
+ first_stage_layers: Number of layers on the first pipeline stage for
+ uneven PP. Only valid when the pattern has no pipe separators.
+ last_stage_layers: Number of layers on the last pipeline stage for
+ uneven PP. Only valid when the pattern has no pipe separators.
+
+ Returns:
+ Tuple of (layer_type_list, layer_offset) where layer_type_list is
+ the list of layer type characters for this segment, and layer_offset
+ is the sum of layer counts from all preceding segments.
+
+ Raises:
+ ValueError: If the segment contains invalid layer symbols, if
+ first/last_stage_layers are used with pipe separators, if VPP is
+ requested without pipe separators, or if layer counts are not
+ evenly divisible across pipeline stages.
+ """
+ segments = main_pattern.split(Symbols.PIPE) if main_pattern else ['']
+
+ pp_rank = torch.distributed.get_rank(pp_group) if pp_group is not None else 0
+ pp_size = torch.distributed.get_world_size(pp_group) if pp_group is not None else 1
+
+ if len(segments) > 1 and (first_stage_layers is not None or last_stage_layers is not None):
+ raise ValueError(
+ "Cannot specify num_layers_in_first_pipeline_stage or "
+ "num_layers_in_last_pipeline_stage when hybrid_layer_pattern "
+ "contains pipe ('|') separators. The pipeline layout is already "
+ "explicitly defined by the pipe separators."
+ )
+
+ if len(segments) == 1 and pp_size > 1:
+ if vp_stage is not None:
raise ValueError(
- "The number of each type of layer in the override "
- "pattern must match the number in the overridden "
- "pattern."
- )
- if layer_type_list_override == layer_type_list:
- maybe_log_single_rank(
- logger, logging.INFO, "The override pattern matches the overridden pattern"
+ "Virtual pipeline parallelism (vp_stage != None) is not supported "
+ "when hybrid_layer_pattern has no pipe ('|') separators. "
+ "Add '|' separators to define explicit pipeline/virtual-pipeline "
+ "stage boundaries."
)
- else:
- maybe_log_single_rank(
- logger, logging.INFO, "Warning: overriding pattern A with pattern B"
- )
- maybe_log_single_rank(logger, logging.INFO, f"A: {''.join(layer_type_list)}")
- maybe_log_single_rank(logger, logging.INFO, f"B: {''.join(layer_type_list_override)}")
- layer_type_list = layer_type_list_override
-
- if target_attention_ratio > 0.0 or target_mlp_ratio > 0.0 or override_pattern is not None:
- actual_attention_layers_count = layer_type_list.count(Symbols.ATTENTION)
- actual_attention_ratio = actual_attention_layers_count / total_layers_count
- actual_mlp_layers_count = layer_type_list.count(Symbols.MLP)
- actual_mlp_ratio = actual_mlp_layers_count / total_layers_count
- allocation_string = "".join(layer_type_list)
- maybe_log_single_rank(
- logger,
- logging.INFO,
- f"Hybrid allocation ({Symbols.MAMBA} is mamba, "
- f"{Symbols.ATTENTION} is attention, "
- f"{Symbols.MLP} is mlp):",
- )
- maybe_log_single_rank(logger, logging.INFO, allocation_string)
- maybe_log_single_rank(
+ log_single_rank(
logger,
- logging.INFO,
- f"{actual_attention_layers_count} attention layers in "
- f"{total_layers_count} total layers.",
+ logging.WARNING,
+ "DEPRECATION: Using hybrid_layer_pattern without pipe ('|') separators "
+ "with pipeline_model_parallel_size > 1 is deprecated. Please add '|' "
+ "separators to explicitly define pipeline stage boundaries. "
+ "Example: 'M*M*M*M*' with pp_size=2 should become 'M*M*|M*M*'.",
)
- maybe_log_single_rank(
+ full_pattern = segments[0]
+ layer_type_list = validate_segment_layers(full_pattern)
+ num_layers = len(layer_type_list)
+
+ if first_stage_layers is not None or last_stage_layers is not None:
+ first = first_stage_layers or 0
+ last = last_stage_layers or 0
+ middle_num_layers = num_layers - first - last
+ middle_stages = pp_size - sum(
+ 1 for x in (first_stage_layers, last_stage_layers) if x is not None
+ )
+ if middle_stages > 0:
+ if middle_num_layers % middle_stages != 0:
+ raise ValueError(
+ f"Middle layers ({middle_num_layers}) must be evenly divisible "
+ f"by middle pipeline stages ({middle_stages})."
+ )
+ layers_per_middle = middle_num_layers // middle_stages
+ else:
+ layers_per_middle = 0
+
+ is_first = first_stage_layers is not None and pp_rank == 0
+ is_last = last_stage_layers is not None and pp_rank == pp_size - 1
+
+ if is_first:
+ offset = 0
+ count = first
+ elif is_last:
+ offset = num_layers - last
+ count = last
+ else:
+ middle_rank = pp_rank if first_stage_layers is None else pp_rank - 1
+ offset = middle_rank * layers_per_middle + first
+ count = layers_per_middle
+ else:
+ if num_layers % pp_size != 0:
+ raise ValueError(
+ f"Number of layers ({num_layers}) must be evenly divisible "
+ f"by pipeline-model-parallel-size ({pp_size}) when no pipe "
+ f"separators are specified in the pattern."
+ )
+ layers_per_rank = num_layers // pp_size
+ offset = pp_rank * layers_per_rank
+ count = layers_per_rank
+
+ selected = layer_type_list[offset : offset + count]
+ log_on_each_pipeline_stage(
logger,
logging.INFO,
- f"Target attention ratio: {target_attention_ratio:.2f}. "
- f"Actual attention ratio: {actual_attention_ratio:.2f}.",
+ f"MambaModel: pp_rank={pp_rank}/{pp_size}, vp_stage={vp_stage}, "
+ f"layers='{''.join(selected)}' ({len(selected)} layers), "
+ f"layer_offset={offset} (auto-split)",
)
- maybe_log_single_rank(
- logger,
- logging.INFO,
- f"{actual_mlp_layers_count} mlp layers in " f"{total_layers_count} total layers.",
+ return selected, offset
+
+ # Pipe-based segment selection
+ if len(segments) > 1 and len(segments) % pp_size != 0:
+ raise ValueError(
+ f"The number of pipe-delimited segments ({len(segments)}) in "
+ f"hybrid_layer_pattern must be evenly divisible by "
+ f"pipeline_model_parallel_size ({pp_size})."
)
- maybe_log_single_rank(
- logger,
- logging.INFO,
- f"Target mlp ratio: {target_mlp_ratio:.2f}. "
- f"Actual mlp ratio: {actual_mlp_ratio:.2f}.",
+
+ vp_rel = vp_stage if vp_stage is not None else 0
+ segment_index = vp_rel * pp_size + pp_rank
+
+ if segment_index >= len(segments):
+ raise ValueError(
+ f"Pipeline segment index {segment_index} (pp_rank={pp_rank}, "
+ f"vp_stage={vp_rel}) is out of range for {len(segments)} segments. "
+ f"The pattern does not define enough pipe-delimited segments for "
+ f"the current PP/VPP configuration."
)
- return layer_type_list
+
+ layer_offset = sum(len(segments[i]) for i in range(segment_index))
+ my_segment = segments[segment_index]
+
+ layer_type_list = validate_segment_layers(my_segment)
+
+ log_on_each_pipeline_stage(
+ logger,
+ logging.INFO,
+ f"MambaModel: pp_rank={pp_rank}/{pp_size}, vp_stage={vp_rel}, "
+ f"segment_index={segment_index}/{len(segments)}, "
+ f"layers='{my_segment}' ({len(layer_type_list)} layers), "
+ f"layer_offset={layer_offset}",
+ )
+
+ return layer_type_list, layer_offset
def get_layer_maps_from_layer_type_list(
layer_type_list: List[str],
-) -> Tuple[Dict[int, int], Dict[int, int], Dict[int, int]]:
+) -> Tuple[Dict[int, int], Dict[int, int], Dict[int, int], Dict[int, int], Dict[int, int]]:
"""
Returns maps from global layer index to the corresponding layer index
- for each layer type in [Attention, Mamba, MLP, MoE] given a layer type list.
+ for each layer type in [Mamba, GDN, Attention, MLP, MoE] given a layer type list.
"""
- layer_types = [Symbols.ATTENTION, Symbols.MAMBA, Symbols.MLP, Symbols.MOE]
+ layer_types = [Symbols.MAMBA, Symbols.GDN, Symbols.ATTENTION, Symbols.MLP, Symbols.MOE]
layer_maps = {layer_type: {} for layer_type in layer_types}
for global_layer_idx, layer_type in enumerate(layer_type_list):
layer_map = layer_maps[layer_type]
local_layer_idx = len(layer_map)
layer_map[global_layer_idx] = local_layer_idx
return [layer_maps[layer_type] for layer_type in layer_types]
-
-
-if __name__ == "__main__":
- test_cases = [
- # (10, 0.2, 0.0),
- # (48, 0.0, 0.0), # will not print anything
- # (48, 0.1, 0.0),
- # 48, 0.3, 0.0),
- # (48, 0.5, 0.0),
- # (48, 0.6, 0.0),
- # (48, 0.7, 0.0),
- # (10, 0.0, 0.1),
- # (10, 0.0, 0.3),
- # (10, 0.0, 0.5),
- # (10, 0.1, 0.1),
- # (10, 0.2, 0.2),
- # (10, 0.3, 0.3),
- # (10, 0.5, 0.5),
- # (48, 0.2, 0.3),
- # (48, 0.5, 0.2),
- # (48, 0.5, 0.2, "MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-"),
- # (48, 0.25, 0.25, "MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-"),
- # (48, 0.25, 0.25, "MM-*MM-*MM*-MM*-MM*-MM*-M*M-M*M-M*M-M*M-*MM-*MM-"),
- # (48, 0.0, 0.2, "MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-"),
- # (48, 0.2, 0.0, "MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-"),
- # (48, 0.0, 0.0, "MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-MM*-"),
- # (48, 0.5, 0.5),
- # (10, 0.3, 0.2, "MMM*-*M*M-"),
- # (10, 0.3, 0.2, "MM*M-*M*M-"),
- (9, 0.0, 0.0, "M*-M*-M*-"),
- (9, 0.0, 0.0, "MMMMMMMMM"),
- ]
- for t in test_cases:
- logging.info("")
- allocate_layers(*t)
diff --git a/megatron/core/ssm/mamba_layer.py b/megatron/core/ssm/mamba_layer.py
index ac6e8b5bf40..17903cebf3b 100644
--- a/megatron/core/ssm/mamba_layer.py
+++ b/megatron/core/ssm/mamba_layer.py
@@ -6,7 +6,7 @@
# LICENSE file in the root directory of this source tree.
from dataclasses import dataclass, field
-from typing import Dict, Optional, Tuple, Union
+from typing import Dict, Optional, Protocol, Tuple, Union
import torch
from torch import Tensor
@@ -20,10 +20,18 @@
from megatron.core.transformer.identity_op import IdentityOp
from megatron.core.transformer.module import GraphableMegatronModule
from megatron.core.transformer.spec_utils import ModuleSpec, build_module
+from megatron.core.transformer.torch_norm import LayerNormInterface
from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.typed_torch import apply_module
from megatron.core.utils import deprecate_inference_params
+class LayerNormBuilder(Protocol):
+ """A protocol showing how MambaLayer expects to construct its LayerNorm."""
+
+ def __call__(self, config: TransformerConfig, hidden_size: int, /) -> LayerNormInterface: ...
+
+
@dataclass
class MambaLayerSubmodules:
"""
@@ -40,7 +48,7 @@ class MambaLayerSubmodules:
after the mixer.
"""
- norm: Union[ModuleSpec, type] = IdentityOp
+ norm: LayerNormBuilder = IdentityOp
mixer: Union[ModuleSpec, type] = IdentityOp
mamba_bda: Union[ModuleSpec, type] = IdentityOp
@@ -61,7 +69,6 @@ def __init__(
config: TransformerConfig,
submodules: MambaLayerSubmodules,
layer_number: int = 1,
- residual_in_fp32=False,
pg_collection: ProcessGroupCollection = None,
pp_layer_offset: int = 0,
):
@@ -72,7 +79,6 @@ def __init__(
self.config = config
self.submodules_config = submodules
self.layer_number = layer_number
- self.residual_in_fp32 = residual_in_fp32
self.hidden_dropout = config.hidden_dropout
self.mixer = build_module(
submodules.mixer,
@@ -82,7 +88,7 @@ def __init__(
pg_collection=pg_collection,
pp_layer_offset=pp_layer_offset,
)
- self.norm = build_module(submodules.norm, self.config, self.config.hidden_size)
+ self.norm = submodules.norm(self.config, self.config.hidden_size)
self.mamba_bda = build_module(submodules.mamba_bda)
self.bias_dropout_add_exec_handler = torch.enable_grad
@@ -128,11 +134,11 @@ def forward(
inference_context = deprecate_inference_params(inference_context, inference_params)
residual = hidden_states
- if self.residual_in_fp32:
- residual = residual.to(torch.float32)
+ if self.config.fp32_residual_connection:
+ residual = residual.float()
hidden_states = hidden_states.to(dtype=self.config.params_dtype)
- hidden_states = self.norm(hidden_states)
+ hidden_states = apply_module(self.norm)(hidden_states)
mixer_out_with_bias = self.mixer(
hidden_states, inference_context=inference_context, packed_seq_params=packed_seq_params
@@ -185,14 +191,22 @@ def _should_call_local_cudagraph(self, *args, **kwargs):
"""
Check if we should call the local cudagraph path.
"""
- # Training and validation mode CUDA graphs
- if hasattr(self, 'cudagraph_manager') and kwargs.get('inference_context') is None:
+ # Training and validation mode CUDA graphs.
+ if (
+ hasattr(self, 'cudagraph_manager')
+ and kwargs.get('inference_context') is None
+ and not torch.is_inference_mode_enabled() # for inference eager dummy_forward
+ ):
return True
elif not self.training and (
hasattr(self, 'cudagraph_manager')
and kwargs.get('attention_mask') is None
and kwargs.get('inference_context') is not None
+ and not self.config.cuda_graph_scope # empty-list = per-layer CUDA graphs
):
- using_cuda_graph = kwargs['inference_context'].using_cuda_graph_this_step()
+ context = kwargs['inference_context']
+ using_cuda_graph = (context.is_static_batching() and context.is_decode_only()) or (
+ not context.is_static_batching() and context.using_cuda_graph_this_step()
+ )
return using_cuda_graph
return False
diff --git a/megatron/core/ssm/mamba_mixer.py b/megatron/core/ssm/mamba_mixer.py
index cc71cdc32f6..314790f6a5d 100644
--- a/megatron/core/ssm/mamba_mixer.py
+++ b/megatron/core/ssm/mamba_mixer.py
@@ -7,7 +7,6 @@
import logging
import math
-import warnings
from dataclasses import dataclass, replace
from typing import List, Optional, Tuple, Union
@@ -26,6 +25,8 @@
)
from megatron.core.packed_seq_params import PackedSeqParams
from megatron.core.process_groups_config import ProcessGroupCollection
+from megatron.core.ssm.ops.causal_conv1d_triton import causal_conv1d_update
+from megatron.core.ssm.ops.mamba_ssm import selective_state_update
from megatron.core.tensor_parallel import get_cuda_rng_tracker
from megatron.core.transformer import TransformerConfig
from megatron.core.transformer.module import MegatronModule
@@ -39,22 +40,18 @@
deprecate_inference_params,
is_causal_conv1d_min_version,
is_mamba_min_version,
+ is_using_quantization_scales,
log_single_rank,
)
from .mamba_context_parallel import MambaContextParallel
try:
- from mamba_ssm.ops.triton.selective_state_update import selective_state_update
-except ImportError:
- selective_state_update = None
-
-try:
- from causal_conv1d import causal_conv1d_fn, causal_conv1d_update
+ from causal_conv1d import causal_conv1d_fn
from causal_conv1d.causal_conv1d_varlen import causal_conv1d_varlen_states
+
except ImportError:
causal_conv1d_fn = None
- causal_conv1d_update = None
try:
from mamba_ssm.ops.triton.layernorm_gated import RMSNorm as RMSNormGated
@@ -65,6 +62,19 @@
HAVE_MAMBA_SSM = True
except ImportError:
+ mamba_chunk_scan_combined = None
+ mamba_split_conv1d_scan_combined = None
+ HAVE_MAMBA_SSM = False
+
+try:
+ from megatron.core.ssm.ops.ssd_combined import mamba_chunk_scan_combined_varlen
+
+ HAVE_SSM_OPS_VARLEN = True
+except ImportError:
+ mamba_chunk_scan_combined_varlen = None
+ HAVE_SSM_OPS_VARLEN = False
+
+if not HAVE_MAMBA_SSM:
from unittest.mock import MagicMock
RMSNormGated = MagicMock()
@@ -134,7 +144,7 @@ class MambaMixer(MegatronModule):
dt_init_floor: The minimum value of the dt parameter after initialization.
bias: Whether to use bias in the linear layers.
conv_bias: Whether to use bias in the causal convolution.
- chunk_size: The chunk size for the fused kernel.
+ chunk_size: The chunk size for the Mamba SSM fused kernel.
use_mem_eff_path: Whether to use the memory-efficient path for the Mamba model.
layer_number: The layer number of this Mamba layer.
pg_collection: The required process groups to use for tensor model parallel and context
@@ -163,10 +173,6 @@ def __init__(
# Fused kernel and sharding options
chunk_size=128,
layer_number=None,
- use_mem_eff_path=None,
- d_state=None,
- headdim=None,
- ngroups=None,
pg_collection: ProcessGroupCollection = None,
pp_layer_offset: int = 0,
):
@@ -194,33 +200,6 @@ def __init__(
self.cached_batch_size = None
assert pg_collection is not None, "pg_collection must be provided for MambaMixer"
self.pg_collection = pg_collection
-
- # Check for deprecated arguments and raise warnings
- if use_mem_eff_path is not None:
- warnings.warn(
- "The 'use_mem_eff_path' argument is deprecated and will be removed in the future. "
- "Please use the value from the TransformerConfig object instead.",
- DeprecationWarning,
- )
- if d_state is not None:
- warnings.warn(
- "The 'd_state' argument is deprecated and will be removed in the future. "
- "Please use the value from the TransformerConfig object instead.",
- DeprecationWarning,
- )
- if headdim is not None:
- warnings.warn(
- "The 'headdim' argument is deprecated and will be removed in the future. "
- "Please use the value from the TransformerConfig object instead.",
- DeprecationWarning,
- )
- if ngroups is not None:
- warnings.warn(
- "The 'ngroups' argument is deprecated and will be removed in the future. "
- "Please use the value from the TransformerConfig object instead.",
- DeprecationWarning,
- )
-
self.use_mem_eff_path = self.config.use_mamba_mem_eff_path
self.d_state = self.config.mamba_state_dim
self.headdim = self.config.mamba_head_dim
@@ -278,6 +257,19 @@ def __init__(
tp_comm_buffer_name="fc1",
tp_group=self.pg_collection.tp,
)
+ # in_proj packs [z, x, B, C, dt] into one ColumnParallelLinear. Each
+ # component is independently TP-sharded but with different sizes. When
+ # resharding across different TP sizes the planner must interleave
+ # per-component blocks rather than doing a contiguous concat.
+ # partition_sizes lists the per-TP-rank block sizes along partition_dim.
+ in_proj_partition_sizes = [
+ self.d_inner_local_tp, # z
+ self.d_inner_local_tp, # x
+ self.ngroups_local_tp * self.d_state, # B
+ self.ngroups_local_tp * self.d_state, # C
+ self.nheads_local_tp, # dt
+ ]
+ setattr(self.in_proj.weight, "partition_sizes", in_proj_partition_sizes)
if not self.use_mem_eff_path:
log_single_rank(
@@ -304,7 +296,20 @@ def __init__(
dtype=config.params_dtype,
)
setattr(self.conv1d.weight, "tensor_model_parallel", True)
+ setattr(self.conv1d.weight, "partition_dim", 0)
setattr(self.conv1d.bias, "tensor_model_parallel", True)
+ setattr(self.conv1d.bias, "partition_dim", 0)
+ # partition_sizes describes the per-TP-rank block sizes along the
+ # partition dim. conv1d packs [x, B, C] whose local sizes differ,
+ # so a plain contiguous concat would produce the wrong layout when
+ # resharding across different TP sizes.
+ conv_partition_sizes = [
+ self.d_inner_local_tp,
+ self.ngroups_local_tp * self.d_state,
+ self.ngroups_local_tp * self.d_state,
+ ]
+ setattr(self.conv1d.weight, "partition_sizes", conv_partition_sizes)
+ setattr(self.conv1d.bias, "partition_sizes", conv_partition_sizes)
if self.config.perform_initialization:
if self.conv_init is not None:
nn.init.uniform_(self.conv1d.weight, -self.conv_init, self.conv_init)
@@ -329,6 +334,7 @@ def __init__(
inv_dt = dt + torch.log(-torch.expm1(-dt))
self.dt_bias = nn.Parameter(inv_dt)
setattr(self.dt_bias, "tensor_model_parallel", True)
+ setattr(self.dt_bias, "partition_dim", 0)
# A parameter
assert A_init_range[0] > 0 and A_init_range[1] >= A_init_range[0]
@@ -340,7 +346,7 @@ def __init__(
A_log = torch.log(A) # Keep A_log in fp32
self.A_log = nn.Parameter(A_log)
setattr(self.A_log, "tensor_model_parallel", True)
-
+ setattr(self.A_log, "partition_dim", 0)
# D "skip" parameter
self.D = nn.Parameter(
torch.ones(
@@ -349,7 +355,7 @@ def __init__(
)
) # Keep in fp32
setattr(self.D, "tensor_model_parallel", True)
-
+ setattr(self.D, "partition_dim", 0)
if self.rmsnorm:
assert RMSNormGated is not None
self.norm = ExtendedRMSNorm(
@@ -361,7 +367,7 @@ def __init__(
dtype=config.params_dtype,
)
setattr(self.norm.weight, "tensor_model_parallel", True)
-
+ setattr(self.norm.weight, "partition_dim", 0)
# Assume sequence parallelism: input is partitioned along d_inner and
# output is partitioned along the sequence dimension
self.out_proj = build_module(
@@ -451,134 +457,80 @@ def forward(
def _dynamic_inference(self, hidden_states: torch.Tensor, context: DynamicInferenceContext):
"""
Executes dynamic inference by separating decode and prefill requests and
- running them independently. Also runs the chunked prefill request independently
- if it exists.
+ running them independently.
"""
sequence_packing_available, reason_for_no_sequence_packing = (
_check_mamba_sequence_packing_support(for_inference_not_training=True)
)
assert sequence_packing_available, reason_for_no_sequence_packing
+ # Grab standard states
conv_state, ssm_state = context.mamba_states_cache(self.layer_number - self.pp_layer_offset)
- padded_dims = context.padded_batch_dimensions
+ # Fetch intermediate states for speculative decoding
+ # (just buffers, existing data is overwritten)
+ int_conv_state = None
+ int_ssm_state = None
+ if context.num_speculative_tokens > 0:
+ int_conv_state, int_ssm_state = context.mamba_states_cache(
+ self.layer_number - self.pp_layer_offset, intermediate=True
+ )
+ padded_dims = context.padded_batch_dimensions
token_count = padded_dims.token_count
decode_req_count = padded_dims.decode_req_count
prefill_req_count = padded_dims.prefill_req_count
- has_explicit_chunked_prefill_req = padded_dims.has_explicit_chunked_prefill_req
# Input projection
zxBCdt, _ = self.in_proj(hidden_states)
- if decode_req_count > 0 and prefill_req_count == 0:
- # Decode-only
- y = self._ssm_decode(
- zxBCdt.transpose(0, 1),
- conv_state,
- ssm_state,
- context.mamba_metadata.batch_indices_decode,
- ).transpose(0, 1)
- elif decode_req_count == 0 and (prefill_req_count > 0 or has_explicit_chunked_prefill_req):
- if prefill_req_count > 0:
- # Prefill only (regular prefill requests)
- y_prefill = self._ssm_prefill(
- zxBCdt,
- conv_state=conv_state,
- ssm_state=ssm_state,
- seq_idx=context.mamba_metadata.seq_idx,
- cu_seqlens=context.mamba_metadata.cu_seqlens,
- return_varlen_states=True,
- batch_indices=context.mamba_metadata.batch_indices_prefill,
- )
- if has_explicit_chunked_prefill_req:
- # Prefill only (chunked prefill request)
- zxBCdt_chunked_prefill = torch.empty_like(zxBCdt)
- tensor_get_slice_after(
- zxBCdt,
- zxBCdt_chunked_prefill,
- context.mamba_metadata.device_chunked_prefill,
- check_bounds=False,
- )
- y_chunked_prefill = self._ssm_prefill(
- zxBCdt_chunked_prefill[: context.mamba_metadata.device_chunked_prefill[1]],
- conv_state=conv_state,
- ssm_state=ssm_state,
- batch_indices=context.mamba_metadata.batch_indices_chunked_prefill,
- is_chunked_prefill=True,
- )
- if prefill_req_count > 0 and has_explicit_chunked_prefill_req:
- # Merge regular prefill and chunked prefill parts
- tensor_merge(
- y_prefill, y_chunked_prefill, context.mamba_metadata.device_chunked_prefill
- )
- y = y_prefill
- elif prefill_req_count > 0:
- # Prefill-only without chunked prefill
- y = y_prefill
- else:
- # Prefill-only with only chunked prefill
- y = y_chunked_prefill
- else:
- # Mix of decode and prefill
- zxBCdt_prefill = torch.empty_like(zxBCdt)
- tensor_get_slice_after(
- zxBCdt,
- zxBCdt_prefill,
- context.mamba_metadata.device_decode_prefill,
- check_bounds=False,
- )
- # Decode requests
+ y_decode = None
+ y_prefill = None
+
+ # Decode
+ if decode_req_count > 0:
+ # For mixed batch, the decode tokens are at the start of zxBCdt
+ seq_len = 1 + context.num_speculative_tokens
+ decode_token_count = decode_req_count * seq_len
+
+ zxBCdt_decode = zxBCdt[:decode_token_count] if prefill_req_count > 0 else zxBCdt
+
+ # Reshape from [N*S, 1, d] to [N, S, d] for the 3D Triton kernels
+ zxBCdt_decode = zxBCdt_decode.squeeze(1).view(decode_req_count, seq_len, -1)
+
y_decode = self._ssm_decode(
- zxBCdt[:decode_req_count].transpose(0, 1),
+ zxBCdt_decode,
conv_state,
ssm_state,
- context.mamba_metadata.batch_indices_decode,
- ).transpose(0, 1)
- y_prefill, y_chunked_prefill = None, None
- if prefill_req_count > 0:
- # Regular prefill requests
- y_prefill = self._ssm_prefill(
- zxBCdt_prefill,
- conv_state=conv_state,
- ssm_state=ssm_state,
- seq_idx=context.mamba_metadata.seq_idx,
- cu_seqlens=context.mamba_metadata.cu_seqlens,
- return_varlen_states=True,
- batch_indices=context.mamba_metadata.batch_indices_prefill,
- )
- if has_explicit_chunked_prefill_req:
- # Chunked prefill request
- zxBCdt_chunked_prefill = torch.empty_like(zxBCdt_prefill)
+ batch_indices=context.mamba_metadata.batch_indices_decode,
+ intermediate_conv_state=int_conv_state,
+ intermediate_ssm_state=int_ssm_state,
+ )
+
+ # Flatten back to [N*S, 1, d] to match merge logic
+ y_decode = y_decode.view(decode_token_count, 1, -1)
+
+ # Prefill
+ if prefill_req_count > 0:
+ if decode_req_count > 0:
+ # If mixed, slice the prefill portion out of zxBCdt
+ zxBCdt_prefill = torch.empty_like(zxBCdt)
tensor_get_slice_after(
+ zxBCdt,
zxBCdt_prefill,
- zxBCdt_chunked_prefill,
- context.mamba_metadata.device_chunked_prefill,
+ context.mamba_metadata.device_decode_prefill,
check_bounds=False,
)
- y_chunked_prefill = self._ssm_prefill(
- zxBCdt_chunked_prefill[: context.mamba_metadata.device_chunked_prefill[1]],
- conv_state=conv_state,
- ssm_state=ssm_state,
- batch_indices=context.mamba_metadata.batch_indices_chunked_prefill,
- is_chunked_prefill=True,
- )
- if prefill_req_count > 0 and has_explicit_chunked_prefill_req:
- # Merge regular prefill and chunked prefill parts
- assert y_prefill is not None
- assert y_chunked_prefill is not None
- tensor_merge(
- y_prefill, y_chunked_prefill, context.mamba_metadata.device_chunked_prefill
- )
- elif has_explicit_chunked_prefill_req:
- # Chunked prefill only
- assert y_prefill is None
- assert y_chunked_prefill is not None
- y_prefill = y_chunked_prefill
else:
- # Regular prefill only; y_prefill is already set, nothing more to be done
- assert y_prefill is not None
- # Merge decode and prefill parts
+ zxBCdt_prefill = zxBCdt
+
+ mamba_layer_idx = context.layer_map[self.layer_number - self.pp_layer_offset - 1]
+ y_prefill = self._dynamic_inference_prefill(
+ zxBCdt_prefill, context, conv_state, ssm_state, mamba_layer_idx=mamba_layer_idx
+ )
+
+ # Merge decode and prefill results if necessary
+ if y_decode is not None and y_prefill is not None:
y = torch.empty(
[token_count, 1, y_prefill.shape[-1]],
dtype=y_prefill.dtype,
@@ -587,12 +539,91 @@ def _dynamic_inference(self, hidden_states: torch.Tensor, context: DynamicInfere
tensor_merge(
y_decode, y_prefill, context.mamba_metadata.device_decode_prefill, output_tensor=y
)
+ elif y_decode is not None:
+ y = y_decode
+ elif y_prefill is not None:
+ y = y_prefill
+ else:
+ raise RuntimeError("Dynamic inference called with 0 decode and 0 prefill requests")
+
+ # Clear the outputs for padding tokens when using quantization scales
+ # to avoid corrupting amax calculations
+ if is_using_quantization_scales(self.config):
+ y[context.padding_slice] = 0.0
# Output projection
out, out_bias = self.out_proj(y)
return out, out_bias
+ def _dynamic_inference_prefill(
+ self,
+ zxBCdt: torch.Tensor,
+ context: DynamicInferenceContext,
+ conv_state: torch.Tensor,
+ ssm_state: torch.Tensor,
+ mamba_layer_idx: Optional[int] = None,
+ ) -> torch.Tensor:
+ """Helper to run dynamic inference prefill.
+
+ All prefill requests (including chunked prefill) are processed together
+ through the unified varlen path. Uses precomputed metadata from
+ MambaMetadata.update() to avoid .item() calls and data-dependent
+ control flow, enabling CUDA graph compatibility.
+
+ Intermediate state extraction (for Mamba prefix caching) is performed
+ inside _ssm_prefill via pre-allocated output buffers, making it fully
+ CUDA graph compatible.
+ """
+ metadata = context.mamba_metadata
+ real_prefill_count = context.batch_dimensions.prefill_req_count
+ if real_prefill_count <= 0:
+ return None
+
+ # Use precomputed metadata (no .item() calls, no stripping).
+ cu_seqlens = metadata.cu_seqlens
+ batch_indices = metadata.batch_indices_prefill
+ real_token_count = metadata.real_prefill_token_count
+ seq_idx = metadata.seq_idx
+
+ # Pass full padded tensor — SSM kernel uses cu_chunk_seqlens for
+ # boundaries and never accesses tokens beyond the last boundary.
+ # Output y is initialized to zeros in _ssm_prefill so padding
+ # positions remain zero (safe for RMSNorm and downstream ops).
+
+ # Prepare intermediate extraction buffers (always passed, CUDA graph compat)
+ slot_allocator = context.mamba_slot_allocator
+ intermediate_chunk_indices = metadata.intermediate_chunk_indices
+ intermediate_abs_positions = metadata.intermediate_abs_positions
+ intermediate_ssm_out = None
+ intermediate_conv_out = None
+ if slot_allocator is not None and mamba_layer_idx is not None:
+ intermediate_ssm_out = slot_allocator.intermediate_ssm_out[mamba_layer_idx]
+ intermediate_conv_out = slot_allocator.intermediate_conv_out[mamba_layer_idx]
+
+ y_prefill = self._ssm_prefill(
+ zxBCdt,
+ conv_state=conv_state,
+ ssm_state=ssm_state,
+ seq_idx=seq_idx,
+ cu_seqlens=cu_seqlens,
+ batch_indices=batch_indices,
+ intermediate_chunk_indices=intermediate_chunk_indices,
+ intermediate_abs_positions=intermediate_abs_positions,
+ intermediate_ssm_out=intermediate_ssm_out,
+ intermediate_conv_out=intermediate_conv_out,
+ conv_gather_offsets=metadata.conv_gather_offsets,
+ cu_chunk_seqlens=metadata.cu_chunk_seqlens,
+ last_chunk_indices=metadata.last_chunk_indices,
+ seq_idx_for_varlen=metadata.seq_idx_for_varlen,
+ cu_seqlens_list=metadata.cu_seqlens_list,
+ real_token_count=real_token_count,
+ conv_seq_idx=metadata.conv_seq_idx,
+ conv_seq_start=metadata.conv_seq_start,
+ )
+
+ return y_prefill
+
def _decode(
self, hidden_states, conv_state, ssm_state, batch_indices: Optional[torch.Tensor] = None
) -> Tuple[torch.Tensor, torch.Tensor]:
@@ -654,7 +685,7 @@ def _ssm_training(
_check_mamba_sequence_packing_support(for_inference_not_training=False)
)
assert sequence_packing_available, reason_for_no_sequence_packing
- seq_idx = self._create_packed_seq_idx(packed_seq_params, zxBCdt.shape[1])
+ seq_idx = packed_seq_params.seq_idx
y = mamba_split_conv1d_scan_combined(
zxBCdt,
@@ -683,36 +714,6 @@ def _ssm_training(
return y
- def _create_packed_seq_idx(self, packed_seq_params: PackedSeqParams, total_tokens: int):
- """
- If total_tokens is 16 (for example), this method takes packed_seq_params.cu_seqlens_q_padded
- (or cu_seqlens_q) which is of the form [0, 5, 7, 11] and returns a tensor of the form
- [0, 0, 0, 0, 0, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3],
- which is [0]*(5-0) + [1]*(7-5) + [2]*(11-7) + [3]*(16-11)
- In the above example, there are three sequences in the pack.
- In general, the output has an additional sequence index (e.g. 0, 1, 2, 3) so that any tokens
- beyond the last padded input sequence are accounted for as an extra sequence. However, If
- cu_seqlens_q_padded[-1] == max_seqlen then this additional sequence index will not be
- included.
- """
- # Example: [0, 5, 7, 11] -> [0, 5, 7, 11, 16]
- if packed_seq_params.cu_seqlens_q_padded is not None:
- cu_seqlens = packed_seq_params.cu_seqlens_q_padded
- else:
- cu_seqlens = packed_seq_params.cu_seqlens_q
- total_tokens_tensor = torch.tensor(
- [total_tokens], dtype=cu_seqlens.dtype, device=cu_seqlens.device
- )
- cu_seqlens_with_max = torch.cat([cu_seqlens, total_tokens_tensor])
- # Example: [0, 5, 7, 11, 16] -> [5, 2, 4, 5]
- seq_lengths = cu_seqlens_with_max[1:] - cu_seqlens_with_max[:-1]
- # Example: [5, 2, 4, 5] -> [0, 0, 0, 0, 0, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3]
- seq_idx = torch.repeat_interleave(
- torch.arange(seq_lengths.numel(), device=cu_seqlens.device), seq_lengths
- )
- seq_idx = seq_idx.to(torch.int32).unsqueeze(0) # Add a batch dimension
- return seq_idx
-
def _ssm_prefill(
self,
zxBCdt: torch.Tensor,
@@ -720,9 +721,19 @@ def _ssm_prefill(
ssm_state: Optional[torch.Tensor],
seq_idx: Optional[torch.Tensor] = None,
cu_seqlens: Optional[torch.Tensor] = None,
- return_varlen_states: bool = False,
batch_indices: Optional[torch.Tensor] = None,
- is_chunked_prefill: bool = False,
+ intermediate_chunk_indices: Optional[torch.Tensor] = None,
+ intermediate_abs_positions: Optional[torch.Tensor] = None,
+ intermediate_ssm_out: Optional[torch.Tensor] = None,
+ intermediate_conv_out: Optional[torch.Tensor] = None,
+ conv_gather_offsets: Optional[torch.Tensor] = None,
+ cu_chunk_seqlens: Optional[torch.Tensor] = None,
+ last_chunk_indices: Optional[torch.Tensor] = None,
+ seq_idx_for_varlen: Optional[torch.Tensor] = None,
+ cu_seqlens_list: Optional[List[int]] = None,
+ real_token_count: Optional[int] = None,
+ conv_seq_idx: Optional[torch.Tensor] = None,
+ conv_seq_start: Optional[torch.Tensor] = None,
) -> torch.Tensor:
"""
Performs SSM computation for inference prefill step.
@@ -734,18 +745,31 @@ def _ssm_prefill(
ssm_state: The selective scan state tensor for inference.
seq_idx: A map from token index to request index for variable-length sequences.
cu_seqlens: Cumulative sequence lengths for variable-length sequences.
- return_varlen_states: Whether to return variable-length states from the SSM kernel.
batch_indices: A map from batch id to position in the Mamba state tensors for
dynamic inference.
- is_chunked_prefill: Whether the request is a chunked prefill request.
+ intermediate_chunk_indices: Pre-allocated tensor of chunk indices for
+ intermediate state extraction (fixed size, padded with 0).
+ intermediate_abs_positions: Pre-allocated tensor of absolute token
+ positions for conv state extraction (fixed size, padded with d_conv).
+ intermediate_ssm_out: Output buffer for extracted SSM states
+ [max_intermediate_count, *ssm_shape].
+ intermediate_conv_out: Output buffer for extracted conv states
+ [max_intermediate_count, *conv_shape].
+ conv_gather_offsets: Constant tensor [-d_conv, ..., -1] for gathering
+ conv states.
+ cu_chunk_seqlens: Precomputed chunk boundaries from MambaMetadata.
+ last_chunk_indices: Precomputed last chunk index per sequence.
+ seq_idx_for_varlen: Precomputed request ID per chunk.
+ cu_seqlens_list: Python list of cumulative sequence lengths (avoids .item()).
+ real_token_count: Number of real (non-padding) tokens.
+ conv_seq_idx: Precomputed per-token request ID for Triton conv1d.
+ conv_seq_start: Precomputed per-token request start for Triton conv1d.
Returns:
- The output tensor of shape (l, b, d).
+ Output tensor of shape (l, b, d). Intermediate states (if any) are
+ written directly to intermediate_ssm_out and intermediate_conv_out.
"""
is_dynamic_batching = seq_idx is not None
- assert not (
- is_dynamic_batching and is_chunked_prefill
- ), "Cannot use chunked prefill with dynamic batching"
# transpose: l b pd --> b l pd
zxBCdt = rearrange(zxBCdt, "l b d -> b l d").contiguous()
@@ -764,31 +788,52 @@ def _ssm_prefill(
)
# Compute short convolution
- initial_conv_state = None
+ xBC_pre_conv = None
if conv_state is not None and is_dynamic_batching:
- # xBC should have shape (b l d) for causal_conv1d_varlen_states
assert batch_indices is not None
+
+ # Extract initial conv states BEFORE saving new ones.
+ # causal_conv1d_varlen_states computes the final conv state from the
+ # input sequence and tensor_masked_update writes it into the conv_state
+ # buffer. If we read initial_conv_states after this write, restored
+ # requests see their own newly-computed states instead of the cached
+ # initial states from a previous request, corrupting the conv output.
+ initial_conv_states = conv_state[batch_indices, :, 1:]
+
+ # Save final conv states from the input sequence
conv_varlen_states = causal_conv1d_varlen_states(
xBC.squeeze(0), cu_seqlens, state_len=conv_state.shape[-1]
)
tensor_masked_update(conv_state, batch_indices, conv_varlen_states)
- # Maintain channels-last memory layout to use seq_idx for causal_conv1d_fn
- # See https://github.com/Dao-AILab/causal-conv1d/blob/69e6dadc28b169a4c49cb86b586f64ee90242c70/csrc/causal_conv1d.cpp#L174 # pylint: disable=line-too-long
- xBC = xBC.transpose(1, 2)
- elif is_chunked_prefill:
- # Maintain channels-last memory layout to use initial_states for causal_conv1d_fn
- # See https://github.com/Dao-AILab/causal-conv1d/blob/69e6dadc28b169a4c49cb86b586f64ee90242c70/csrc/causal_conv1d.cpp#L200 # pylint: disable=line-too-long
- assert batch_indices is not None
- initial_conv_state = (
- conv_state[batch_indices, :, 1:].permute(0, 2, 1).contiguous().transpose(1, 2)
+ # Conv state dtype might differ from params dtype, so cast xBC and weight / bias
+ # tensors to the conv state dtype for causal_conv1d_varlen_fn and then cast xBC
+ # back to the original dtype
+ xBC_dtype = xBC.dtype
+ conv_state_dtype = conv_state.dtype
+
+ xBC = xBC.to(conv_state_dtype)
+ conv_weight = rearrange(self.cp.get_conv1d_weight(), "d 1 w -> d w").to(
+ conv_state_dtype
)
- xBC = xBC.transpose(1, 2)
- tensor_masked_update(
- conv_state, batch_indices, F.pad(xBC, (self.d_conv - xBC.shape[-1], 0))
+ conv_bias = self.cp.get_conv1d_bias().to(conv_state_dtype)
+
+ xBC_pre_conv = xBC if intermediate_conv_out is not None else None
+ from megatron.core.ssm.ops.causal_conv1d_varlen import causal_conv1d_varlen_fn
+
+ xBC_out = causal_conv1d_varlen_fn(
+ x=xBC.squeeze(0).contiguous(),
+ weight=conv_weight,
+ bias=conv_bias,
+ cu_seqlens=cu_seqlens,
+ initial_states=initial_conv_states,
+ activation=self.activation,
+ precomputed_seq_idx=conv_seq_idx,
+ precomputed_seq_start=conv_seq_start,
)
+ xBC = xBC_out.to(xBC_dtype).unsqueeze(0)
else:
- # transpose: b l pd --> b pd l
+ # Non-dynamic-batching path (static batching / training fallback)
xBC = rearrange(xBC, "b l d -> b d l").contiguous()
if conv_state is not None:
# If we just take x[:, :, -self.d_conv :], it will error if seqlen < self.d_conv
@@ -797,22 +842,19 @@ def _ssm_prefill(
F.pad(xBC, (self.d_conv - xBC.shape[-1], 0))
) # Update state (B D W)
- seqlen = xBC.size(2)
- if causal_conv1d_fn is None:
- xBC = self.act(self.cp.conv1d(xBC)[..., :seqlen])
- else:
- assert self.activation in ["silu", "swish"]
- xBC = causal_conv1d_fn(
- x=xBC,
- weight=rearrange(self.cp.get_conv1d_weight(), "d 1 w -> d w"),
- bias=self.cp.get_conv1d_bias(),
- activation=self.activation,
- seq_idx=seq_idx,
- initial_states=initial_conv_state,
- )
-
- # transpose b pd l --> b l pd
- xBC = rearrange(xBC, "b d l -> b l d").contiguous()
+ seqlen = xBC.size(2)
+ if causal_conv1d_fn is None:
+ xBC = self.act(self.cp.conv1d(xBC)[..., :seqlen])
+ else:
+ assert self.activation in ["silu", "swish"]
+ xBC = causal_conv1d_fn(
+ x=xBC,
+ weight=rearrange(self.cp.get_conv1d_weight(), "d 1 w -> d w"),
+ bias=self.cp.get_conv1d_bias(),
+ activation=self.activation,
+ seq_idx=seq_idx,
+ )
+ xBC = rearrange(xBC, "b d l -> b l d").contiguous()
x, B, C = torch.split(
xBC,
@@ -839,51 +881,122 @@ def _ssm_prefill(
self.cp.cp_size == 1 or self.rmsnorm
), "Context parallel not supported for use_mem_eff_path==False and rmsnorm==False"
- if is_chunked_prefill:
+ if is_dynamic_batching:
+ # Unified varlen SSM path: all prefill requests through single kernel call
initial_ssm_state = ssm_state[batch_indices]
- else:
- initial_ssm_state = None
- # Note that both `seq_idx` and `cu_seqlens` must be passed in
- # for variable length generation.
- # See https://github.com/state-spaces/mamba/blob/e0761ece1db07e0949dd88b4f4cd440420a19fd9/tests/test_generation.py#L97 # pylint: disable=line-too-long
- y = mamba_chunk_scan_combined(
- x,
- dt,
- A,
- B,
- C,
- self.chunk_size,
- D=(
- rearrange(self.cp.get_D().float(), "(h p) -> h p", p=self.headdim)
- if self.D_has_hdim
- else self.cp.get_D()
- ),
- z=z if not self.rmsnorm else None,
- dt_bias=self.cp.get_dt_bias().float(),
- dt_softplus=True,
- return_final_states=ssm_state is not None,
- seq_idx=seq_idx,
- cu_seqlens=cu_seqlens,
- return_varlen_states=return_varlen_states,
- initial_states=initial_ssm_state,
- )
-
- if ssm_state is not None:
- if return_varlen_states:
- assert batch_indices is not None
-
- y, _, ssm_varlen_states = y
+ x = x.squeeze(0)
+ dt = dt.squeeze(0)
+ A = A.squeeze(0)
+ B = B.squeeze(0)
+ C = C.squeeze(0)
+ z = z.squeeze(0)
+ # Initialize with zeros so padding positions (beyond cu_chunk_seqlens
+ # boundaries) remain zero, which is safe for RMSNorm and downstream ops.
+ y = torch.zeros_like(x)
+
+ if cu_chunk_seqlens is not None:
+ # Use precomputed chunk metadata (CUDA graph compatible, no .item())
+ pass
+ else:
+ # Fallback: build chunk metadata from cu_seqlens (non-precomputed)
+ chunk_boundaries = [0]
+ last_chunk_indices_list = []
+ num_seqs = cu_seqlens.numel() - 1
+ for i in range(num_seqs):
+ start = cu_seqlens[i].item()
+ end = cu_seqlens[i + 1].item()
+ pos = start + self.chunk_size
+ while pos < end:
+ chunk_boundaries.append(pos)
+ pos += self.chunk_size
+ chunk_boundaries.append(end)
+ last_chunk_indices_list.append(len(chunk_boundaries) - 2)
+
+ cu_chunk_seqlens = cu_seqlens.new_tensor(chunk_boundaries)
+ last_chunk_indices = cu_seqlens.new_tensor(last_chunk_indices_list)
+
+ seq_idx_for_varlen = None
+ if seq_idx is not None:
+ chunk_starts = cu_chunk_seqlens[:-1]
+ seq_idx_for_varlen = seq_idx[0, chunk_starts].contiguous()
+
+ ssm_varlen_result = mamba_chunk_scan_combined_varlen(
+ x=x,
+ dt=dt,
+ A=A,
+ B=B,
+ C=C,
+ chunk_size=self.chunk_size,
+ cu_chunk_seqlens=cu_chunk_seqlens,
+ last_chunk_indices=last_chunk_indices,
+ seq_idx=seq_idx_for_varlen,
+ out=y,
+ D=(
+ rearrange(self.cp.get_D().float(), "(h p) -> h p", p=self.headdim)
+ if self.D_has_hdim
+ else self.cp.get_D()
+ ),
+ z=z if not self.rmsnorm else None,
+ dt_bias=self.cp.get_dt_bias().float(),
+ initial_states=initial_ssm_state,
+ return_intermediate_states=False,
+ intermediate_chunk_indices=intermediate_chunk_indices,
+ dt_softplus=True,
+ dt_limit=(0.0, float("inf")),
+ state_dtype=ssm_state.dtype,
+ )
- # This has to be varlen_states, NOT last_state
- # See reference implementation:
- # https://github.com/state-spaces/mamba/blob/e0761ece1db07e0949dd88b4f4cd440420a19fd9/mamba_ssm/modules/mamba2.py#L267 # pylint: disable=line-too-long
- tensor_masked_update(ssm_state, batch_indices, ssm_varlen_states)
- elif is_chunked_prefill:
- assert batch_indices is not None
- y, last_state = y
- tensor_masked_update(ssm_state, batch_indices, last_state)
+ if intermediate_chunk_indices is not None:
+ ssm_varlen_states, intermediate_ssm_states = ssm_varlen_result
else:
+ ssm_varlen_states = ssm_varlen_result
+ intermediate_ssm_states = None
+
+ y = y.unsqueeze(0)
+ z = z.unsqueeze(0)
+
+ tensor_masked_update(ssm_state, batch_indices, ssm_varlen_states)
+
+ # Write intermediate states to pre-allocated output buffers
+ # All tensor ops, no Python loops, fully CUDA graph compatible
+ if intermediate_chunk_indices is not None and intermediate_ssm_out is not None:
+ intermediate_ssm_out.copy_(intermediate_ssm_states)
+
+ # Vectorized conv state extraction
+ # intermediate_abs_positions: [max_intermediate_count]
+ # conv_gather_offsets: [d_conv] = [-d_conv, ..., -1]
+ gather_positions = (
+ intermediate_abs_positions.unsqueeze(1).long()
+ + conv_gather_offsets.unsqueeze(0).long()
+ ) # [max_intermediate_count, d_conv]
+ intermediate_conv = xBC_pre_conv[0, gather_positions, :]
+ # [max_intermediate_count, d_conv, conv_dim]
+ intermediate_conv_out.copy_(intermediate_conv.transpose(1, 2))
+ # [max_intermediate_count, conv_dim, d_conv]
+ else:
+ # Non-dynamic-batching path (static batching)
+ initial_ssm_state = None
+ y = mamba_chunk_scan_combined(
+ x,
+ dt,
+ A,
+ B,
+ C,
+ self.chunk_size,
+ D=(
+ rearrange(self.cp.get_D().float(), "(h p) -> h p", p=self.headdim)
+ if self.D_has_hdim
+ else self.cp.get_D()
+ ),
+ z=z if not self.rmsnorm else None,
+ dt_bias=self.cp.get_dt_bias().float(),
+ dt_softplus=True,
+ return_final_states=ssm_state is not None,
+ initial_states=initial_ssm_state,
+ )
+
+ if ssm_state is not None:
y, last_state = y
ssm_state.copy_(last_state)
@@ -903,27 +1016,29 @@ def _ssm_decode(
conv_state: torch.Tensor,
ssm_state: torch.Tensor,
batch_indices: Optional[torch.Tensor] = None,
+ intermediate_conv_state: Optional[torch.Tensor] = None,
+ intermediate_ssm_state: Optional[torch.Tensor] = None,
) -> torch.Tensor:
"""
Performs SSM computation for inference decode step.
Args:
- zxBCdt: The input tensor of shape (l, b, d), which is a concatenation of
- z, x, B, C, and dt projections. For decoding, l must be 1.
+ zxBCdt: The input tensor of shape (b, s, d), which is a concatenation of
+ z, x, B, C, and dt projections.
+ s is the sequence length (1 + num_speculative_tokens).
conv_state: The convolution state tensor for inference.
ssm_state: The selective scan state tensor for inference.
- batch_indices: A map from batch id to position in the Mamba state tensors for
- dynamic inference.
+ batch_indices: A map from batch id to position in the Mamba state tensors.
+ intermediate_conv_state: Optional buffer for storing conv state at each
+ sequence step (for speculative decoding rollback).
+ intermediate_ssm_state: Optional buffer for storing SSM state at each
+ sequence step (for speculative decoding rollback).
Returns:
- The output tensor of shape (l, b, d).
+ The output tensor of shape (b, s, d).
"""
- seq_len, batch_size, _ = zxBCdt.shape
+ batch_size, seq_len, _ = zxBCdt.shape
dtype = zxBCdt.dtype
- assert seq_len == 1, "Only support decoding with 1 token at a time for now"
-
- # Remove sequence dimension
- zxBCdt = zxBCdt.squeeze(0)
z, xBC, dt = torch.split(
zxBCdt,
@@ -937,23 +1052,32 @@ def _ssm_decode(
# Conv step
if causal_conv1d_update is None:
+ # TODO(ksanthanam): Consider deprecating this path
+ assert seq_len == 1, "Native PyTorch fallback only supports 1 token at a time"
+ xBC_squeeze = xBC.squeeze(1)
conv_state.copy_(torch.roll(conv_state, shifts=-1, dims=-1)) # Update state (B D W)
- conv_state[:, :, -1] = xBC
- xBC = torch.sum(
+ conv_state[:, :, -1] = xBC_squeeze
+ xBC_squeeze = torch.sum(
conv_state * rearrange(self.conv1d.weight, "d 1 w -> d w"), dim=-1
) # (B D)
if self.conv1d.bias is not None:
- xBC = xBC + self.conv1d.bias
- xBC = self.act(xBC).to(dtype=xBC.dtype)
+ xBC_squeeze = xBC_squeeze + self.conv1d.bias
+ xBC = self.act(xBC_squeeze).to(dtype=xBC.dtype).unsqueeze(1)
else:
+ # Conv state dtype might differ from params dtype, so cast xBC and weight / bias
+ # tensors to the conv state dtype for causal_conv1d_update and then cast xBC
+ # back to the original dtype
+ xBC_dtype = xBC.dtype
+ weight = rearrange(self.conv1d.weight, "d 1 w -> d w")
xBC = causal_conv1d_update(
- xBC,
+ xBC.to(conv_state.dtype),
conv_state,
- rearrange(self.conv1d.weight, "d 1 w -> d w"),
- self.conv1d.bias,
+ weight.to(conv_state.dtype),
+ self.conv1d.bias.to(conv_state.dtype),
self.activation,
conv_state_indices=batch_indices,
- )
+ intermediate_conv_states=intermediate_conv_state,
+ ).to(xBC_dtype)
x, B, C = torch.split(
xBC,
@@ -968,6 +1092,16 @@ def _ssm_decode(
# SSM step
if selective_state_update is None:
+ # TODO(ksanthanam): Consider deprecating this path
+ assert seq_len == 1, "Native PyTorch fallback only supports 1 token at a time"
+
+ x = x.squeeze(1)
+ B = B.squeeze(1)
+ C = C.squeeze(1)
+ dt = dt.squeeze(1)
+ if z is not None:
+ z = z.squeeze(1)
+
if self.ngroups_local_tp > 1:
B = rearrange(B, "b (g n) -> b g n", n=self.d_state)
C = rearrange(C, "b (g n) -> b g n", n=self.d_state)
@@ -1012,16 +1146,20 @@ def _ssm_decode(
y = rearrange(y, "b h p -> b (h p)")
if not self.rmsnorm:
y = y * self.act(z) # (B D)
+
+ y = y.unsqueeze(1) # Restore seq dimension
else:
A = repeat(A, "h -> h p n", p=self.headdim, n=self.d_state).to(dtype=torch.float32)
- dt = repeat(dt, "b h -> b h p", p=self.headdim)
+
+ # Incorporate sequence dimension in einops rearrengements
+ dt = repeat(dt, "b s h -> b s h p", p=self.headdim)
dt_bias = repeat(self.dt_bias, "h -> h p", p=self.headdim)
D = repeat(self.D, "h -> h p", p=self.headdim)
- B = rearrange(B, "b (g n) -> b g n", g=self.ngroups_local_tp)
- C = rearrange(C, "b (g n) -> b g n", g=self.ngroups_local_tp)
- x_reshaped = rearrange(x, "b (h p) -> b h p", p=self.headdim)
+ B = rearrange(B, "b s (g n) -> b s g n", g=self.ngroups_local_tp)
+ C = rearrange(C, "b s (g n) -> b s g n", g=self.ngroups_local_tp)
+ x_reshaped = rearrange(x, "b s (h p) -> b s h p", p=self.headdim)
if not self.rmsnorm:
- z = rearrange(z, "b (h p) -> b h p", p=self.headdim)
+ z = rearrange(z, "b s (h p) -> b s h p", p=self.headdim)
# Upcast the batch_indices to prevent integer overflow errors in the case of
# large max request counts.
@@ -1040,14 +1178,14 @@ def _ssm_decode(
dt_bias=dt_bias,
dt_softplus=True,
state_batch_indices=batch_indices,
+ intermediate_ssm_states=intermediate_ssm_state, # SSM only
)
- y = rearrange(y, "b h p -> b (h p)")
+ y = rearrange(y, "b s h p -> b s (h p)")
if self.rmsnorm:
y = self.norm(y, z)
- # Restore sequence dimension
- return y.unsqueeze(0)
+ return y
def mamba_state_shapes_per_request(self) -> Tuple[Tuple[int], Tuple[int]]:
"""Returns the Mamba conv and ssm states shapes per request."""
diff --git a/megatron/core/ssm/mlp_layer.py b/megatron/core/ssm/mlp_layer.py
index 19aec5878b2..e1668a01381 100644
--- a/megatron/core/ssm/mlp_layer.py
+++ b/megatron/core/ssm/mlp_layer.py
@@ -20,6 +20,7 @@ def __init__(
layer_number: int = 1,
hidden_dropout: float = None,
pg_collection: Optional[ProcessGroupCollection] = None,
+ add_layer_offset: bool = True,
):
super().__init__(
config=config,
@@ -27,4 +28,5 @@ def __init__(
layer_number=layer_number,
hidden_dropout=hidden_dropout,
pg_collection=pg_collection,
+ add_layer_offset=add_layer_offset,
)
diff --git a/megatron/core/ssm/ops/__init__.py b/megatron/core/ssm/ops/__init__.py
new file mode 100644
index 00000000000..c8a06391843
--- /dev/null
+++ b/megatron/core/ssm/ops/__init__.py
@@ -0,0 +1,14 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+# Triton kernels for Mamba SSM (adapted from vLLM / state-spaces/mamba).
+
+try:
+ from .ssd_combined import mamba_chunk_scan_combined_varlen
+except ImportError:
+ mamba_chunk_scan_combined_varlen = None
+
+try:
+ from .causal_conv1d_varlen import causal_conv1d_varlen_fn
+except ImportError:
+ causal_conv1d_varlen_fn = None
+
+__all__ = ["mamba_chunk_scan_combined_varlen", "causal_conv1d_varlen_fn"]
diff --git a/megatron/core/ssm/ops/causal_conv1d_triton.py b/megatron/core/ssm/ops/causal_conv1d_triton.py
new file mode 100644
index 00000000000..f57f5d94cea
--- /dev/null
+++ b/megatron/core/ssm/ops/causal_conv1d_triton.py
@@ -0,0 +1,287 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+# Some of this code was adopted from https://github.com/Dao-AILab/causal-conv1d/
+# This source code is licensed under the BSD license found in the
+# LICENSE file in the root directory of this source tree.
+
+import torch
+
+try:
+ import triton
+ import triton.language as tl
+
+ HAVE_TRITON = True
+except ImportError:
+ from unittest.mock import MagicMock
+
+ from megatron.core.utils import null_decorator
+
+ triton = MagicMock()
+ triton.jit = null_decorator
+ tl = MagicMock()
+ HAVE_TRITON = False
+
+
+@triton.jit
+def causal_conv1d_update_kernel(
+ x_ptr,
+ x_b_stride,
+ x_s_stride,
+ x_c_stride,
+ conv_state_ptr,
+ conv_state_b_stride,
+ conv_state_c_stride,
+ conv_state_l_stride,
+ int_state_ptr,
+ int_state_b_stride,
+ int_state_s_stride,
+ int_state_c_stride,
+ int_state_l_stride,
+ weight_ptr,
+ weight_c_stride,
+ weight_width_stride,
+ bias_ptr,
+ bias_stride,
+ out_ptr,
+ out_b_stride,
+ out_s_stride,
+ out_c_stride,
+ conv_state_indices_ptr,
+ batch,
+ seq_len,
+ dim,
+ state_len,
+ WIDTH: tl.constexpr,
+ BLOCK_DIM: tl.constexpr,
+ HAS_BIAS: tl.constexpr,
+ HAS_STATE_INDICES: tl.constexpr,
+ HAS_INT_STATE: tl.constexpr,
+ SILU_ACTIVATION: tl.constexpr,
+):
+ """Triton implementation of causal_conv1d_update (kernel)."""
+ batch_id = tl.program_id(0)
+ channel_block_id = tl.program_id(1)
+
+ channel_offsets = channel_block_id * BLOCK_DIM + tl.arange(0, BLOCK_DIM)
+ mask = channel_offsets < dim
+
+ # State batch coordinate mapping
+ if HAS_STATE_INDICES:
+ state_batch_coord = tl.load(conv_state_indices_ptr + batch_id)
+ else:
+ state_batch_coord = batch_id
+
+ # Base Pointers
+ conv_state_ptrs = (
+ conv_state_ptr
+ + state_batch_coord * conv_state_b_stride
+ + channel_offsets * conv_state_c_stride
+ )
+ weight_ptrs = weight_ptr + channel_offsets * weight_c_stride
+
+ # Skip padding tokens (block-level uniform condition)
+ if state_batch_coord < 0:
+ for s in range(seq_len):
+ out_ptrs = (
+ out_ptr
+ + batch_id * out_b_stride
+ + s * out_s_stride
+ + channel_offsets * out_c_stride
+ )
+ tl.store(out_ptrs, 0.0, mask=mask)
+ return
+
+ # Load Bias
+ if HAS_BIAS:
+ bias_val = tl.load(bias_ptr + channel_offsets * bias_stride, mask=mask).to(tl.float32)
+ else:
+ bias_val = tl.zeros([BLOCK_DIM], dtype=tl.float32)
+
+ # Load Weights
+ if WIDTH == 2:
+ w0 = tl.load(weight_ptrs + 0 * weight_width_stride, mask=mask).to(tl.float32)
+ w1 = tl.load(weight_ptrs + 1 * weight_width_stride, mask=mask).to(tl.float32)
+ elif WIDTH == 3:
+ w0 = tl.load(weight_ptrs + 0 * weight_width_stride, mask=mask).to(tl.float32)
+ w1 = tl.load(weight_ptrs + 1 * weight_width_stride, mask=mask).to(tl.float32)
+ w2 = tl.load(weight_ptrs + 2 * weight_width_stride, mask=mask).to(tl.float32)
+ elif WIDTH == 4:
+ w0 = tl.load(weight_ptrs + 0 * weight_width_stride, mask=mask).to(tl.float32)
+ w1 = tl.load(weight_ptrs + 1 * weight_width_stride, mask=mask).to(tl.float32)
+ w2 = tl.load(weight_ptrs + 2 * weight_width_stride, mask=mask).to(tl.float32)
+ w3 = tl.load(weight_ptrs + 3 * weight_width_stride, mask=mask).to(tl.float32)
+
+ # Initialize independent x_vals to match unrolled float array
+ x_val_0 = tl.zeros([BLOCK_DIM], dtype=tl.float32)
+ x_val_1 = tl.zeros([BLOCK_DIM], dtype=tl.float32)
+ x_val_2 = tl.zeros([BLOCK_DIM], dtype=tl.float32)
+ x_val_3 = tl.zeros([BLOCK_DIM], dtype=tl.float32)
+
+ # Loop over the sequence dimension (e.g., speculative tokens)
+ for s in range(seq_len):
+ x_ptrs = x_ptr + batch_id * x_b_stride + s * x_s_stride + channel_offsets * x_c_stride
+ out_ptrs = (
+ out_ptr + batch_id * out_b_stride + s * out_s_stride + channel_offsets * out_c_stride
+ )
+
+ # Load the last (WIDTH - 1) elements to use them BEFORE they are overwritten
+ # by the shift
+ if WIDTH >= 2:
+ x_val_0 = tl.load(
+ conv_state_ptrs + (state_len - WIDTH + 1) * conv_state_l_stride, mask=mask
+ ).to(tl.float32)
+ if WIDTH >= 3:
+ x_val_1 = tl.load(
+ conv_state_ptrs + (state_len - WIDTH + 2) * conv_state_l_stride, mask=mask
+ ).to(tl.float32)
+ if WIDTH >= 4:
+ x_val_2 = tl.load(
+ conv_state_ptrs + (state_len - WIDTH + 3) * conv_state_l_stride, mask=mask
+ ).to(tl.float32)
+
+ # Shift the linear state buffer left by 1
+ i = 0
+ while i < state_len - 1:
+ val = tl.load(conv_state_ptrs + (i + 1) * conv_state_l_stride, mask=mask)
+ tl.store(conv_state_ptrs + i * conv_state_l_stride, val, mask=mask)
+ i += 1
+
+ # Process the single token for the current sequence step
+ x_val = tl.load(x_ptrs, mask=mask)
+
+ # Store the new token at the end of the linear state buffer
+ tl.store(conv_state_ptrs + (state_len - 1) * conv_state_l_stride, x_val, mask=mask)
+
+ # Write out to the intermediate state buffer if requested
+ if HAS_INT_STATE:
+ i = 0
+ while i < state_len:
+ val = tl.load(conv_state_ptrs + i * conv_state_l_stride, mask=mask)
+ int_ptr = (
+ int_state_ptr
+ + state_batch_coord * int_state_b_stride
+ + s * int_state_s_stride
+ + channel_offsets * int_state_c_stride
+ + i * int_state_l_stride
+ )
+ tl.store(int_ptr, val, mask=mask)
+ i += 1
+
+ # Advance registers for calculation
+ x_val_f32 = x_val.to(tl.float32)
+ if WIDTH == 2:
+ x_val_1 = x_val_f32
+ elif WIDTH == 3:
+ x_val_2 = x_val_f32
+ elif WIDTH == 4:
+ x_val_3 = x_val_f32
+
+ # Compute output
+ out_val = bias_val
+ if WIDTH == 2:
+ out_val += w0 * x_val_0 + w1 * x_val_1
+ elif WIDTH == 3:
+ out_val += w0 * x_val_0 + w1 * x_val_1 + w2 * x_val_2
+ elif WIDTH == 4:
+ out_val += w0 * x_val_0 + w1 * x_val_1 + w2 * x_val_2 + w3 * x_val_3
+
+ if SILU_ACTIVATION:
+ out_val = out_val * tl.sigmoid(out_val)
+
+ tl.store(out_ptrs, out_val.to(out_ptrs.dtype.element_ty), mask=mask)
+
+
+def causal_conv1d_update(
+ x: torch.Tensor,
+ conv_state: torch.Tensor,
+ weight: torch.Tensor,
+ bias: torch.Tensor | None,
+ silu_activation: bool,
+ conv_state_indices: torch.Tensor | None,
+ intermediate_conv_states: torch.Tensor | None = None,
+) -> torch.Tensor:
+ """Triton implementation of causal_conv1d_update (entrypoint)."""
+
+ # Check if input is 2D, temporarily treat as 3D for uniform processing
+ is_2d = x.dim() == 2
+ if is_2d:
+ x = x.unsqueeze(1)
+
+ batch, seq_len, dim = x.shape
+ out = torch.empty_like(x)
+ state_len = conv_state.shape[-1]
+ width = weight.shape[-1]
+
+ if bias is not None:
+ bias_stride = bias.stride(0)
+ has_bias = True
+ else:
+ bias = x # Dummy pointer
+ bias_stride = 0
+ has_bias = False
+
+ if conv_state_indices is not None:
+ has_state_indices = True
+ else:
+ conv_state_indices = x # Dummy pointer
+ has_state_indices = False
+
+ # Extract intermediate state strides if provided
+ if intermediate_conv_states is not None:
+ has_int_state = True
+ int_state_ptr = intermediate_conv_states
+ int_state_b_stride = intermediate_conv_states.stride(0)
+ int_state_s_stride = intermediate_conv_states.stride(1)
+ int_state_c_stride = intermediate_conv_states.stride(2)
+ int_state_l_stride = intermediate_conv_states.stride(3)
+ else:
+ has_int_state = False
+ int_state_ptr = x # Dummy pointer
+ int_state_b_stride = 0
+ int_state_s_stride = 0
+ int_state_c_stride = 0
+ int_state_l_stride = 0
+
+ BLOCK_DIM = 64
+ grid = (batch, triton.cdiv(dim, BLOCK_DIM))
+
+ causal_conv1d_update_kernel[grid](
+ x,
+ x.stride(0),
+ x.stride(1),
+ x.stride(2),
+ conv_state,
+ conv_state.stride(0),
+ conv_state.stride(1),
+ conv_state.stride(2),
+ int_state_ptr,
+ int_state_b_stride,
+ int_state_s_stride,
+ int_state_c_stride,
+ int_state_l_stride,
+ weight,
+ weight.stride(0),
+ weight.stride(1),
+ bias,
+ bias_stride,
+ out,
+ out.stride(0),
+ out.stride(1),
+ out.stride(2),
+ conv_state_indices,
+ batch,
+ seq_len,
+ dim,
+ state_len,
+ WIDTH=width,
+ BLOCK_DIM=BLOCK_DIM,
+ HAS_BIAS=has_bias,
+ HAS_STATE_INDICES=has_state_indices,
+ HAS_INT_STATE=has_int_state,
+ SILU_ACTIVATION=silu_activation == "silu",
+ )
+
+ if is_2d:
+ out = out.squeeze(1)
+
+ return out
diff --git a/megatron/core/ssm/ops/causal_conv1d_varlen.py b/megatron/core/ssm/ops/causal_conv1d_varlen.py
new file mode 100644
index 00000000000..327ce1bfc48
--- /dev/null
+++ b/megatron/core/ssm/ops/causal_conv1d_varlen.py
@@ -0,0 +1,259 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""Triton varlen depthwise causal 1D convolution with per-sequence initial states and fused SiLU.
+
+Supports packed variable-length sequences where `causal_conv1d_fn` cannot accept
+both `seq_idx` and `initial_states` simultaneously.
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+from megatron.core.ssm.ops.determinism import autotune_configs
+
+
+@triton.autotune(
+ configs=autotune_configs(
+ [
+ triton.Config({"BLOCK_T": 128, "BLOCK_C": 64}, num_warps=4),
+ triton.Config({"BLOCK_T": 128, "BLOCK_C": 128}, num_warps=4),
+ triton.Config({"BLOCK_T": 256, "BLOCK_C": 64}, num_warps=4),
+ triton.Config({"BLOCK_T": 256, "BLOCK_C": 128}, num_warps=8),
+ ]
+ ),
+ key=["conv_dim"],
+)
+@triton.jit
+def _causal_conv1d_varlen_kernel(
+ x_ptr,
+ weight_ptr,
+ bias_ptr,
+ seq_idx_ptr,
+ seq_start_ptr,
+ initial_states_ptr,
+ out_ptr,
+ total_tokens,
+ conv_dim: tl.constexpr,
+ initial_states_stride_req,
+ initial_states_stride_dim,
+ WIDTH: tl.constexpr,
+ BLOCK_T: tl.constexpr,
+ BLOCK_C: tl.constexpr,
+ HAS_INITIAL_STATES: tl.constexpr,
+):
+ """Depthwise causal conv1d over packed varlen sequences with initial states and SiLU.
+
+ Fully vectorized over BLOCK_T tokens x BLOCK_C channels per thread block.
+ """
+ pid_c = tl.program_id(0)
+ pid_t = tl.program_id(1)
+
+ c_off = pid_c * BLOCK_C + tl.arange(0, BLOCK_C) # (BLOCK_C,)
+ c_mask = c_off < conv_dim
+ t_off = pid_t * BLOCK_T + tl.arange(0, BLOCK_T) # (BLOCK_T,)
+ t_mask = t_off < total_tokens
+
+ # Load bias: (BLOCK_C,) broadcast to (BLOCK_T, BLOCK_C)
+ bias = tl.load(bias_ptr + c_off, mask=c_mask, other=0.0).to(tl.float32)
+ acc = tl.zeros((BLOCK_T, BLOCK_C), dtype=tl.float32) + bias[None, :]
+
+ # Load per-token request ID and request start position
+ req_id = tl.load(seq_idx_ptr + t_off, mask=t_mask, other=0) # (BLOCK_T,)
+ req_start = tl.load(seq_start_ptr + t_off, mask=t_mask, other=0) # (BLOCK_T,)
+
+ # Unrolled convolution over WIDTH taps (typically 4)
+ for j in tl.static_range(WIDTH):
+ # Load weight column j: (BLOCK_C,)
+ w_j = tl.load(weight_ptr + c_off * WIDTH + j, mask=c_mask, other=0.0).to(tl.float32)
+
+ # Source position for this tap
+ src = t_off - (WIDTH - 1) + j # (BLOCK_T,)
+ in_seq = src >= req_start # (BLOCK_T,) — True if source is within the sequence
+
+ # Load from x for in-sequence positions (mask out out-of-bounds)
+ src_safe = tl.maximum(src, 0)
+ x_val = tl.load(
+ x_ptr + src_safe[:, None] * conv_dim + c_off[None, :],
+ mask=t_mask[:, None] & c_mask[None, :] & in_seq[:, None],
+ other=0.0,
+ ).to(
+ tl.float32
+ ) # (BLOCK_T, BLOCK_C)
+
+ if HAS_INITIAL_STATES:
+ # For tokens where src < req_start, load from initial_states
+ state_col = (WIDTH - 1) - (req_start - src) # (BLOCK_T,)
+ valid_state = (~in_seq) & (state_col >= 0) # (BLOCK_T,)
+ state_col_safe = tl.maximum(state_col, 0)
+
+ state_val = tl.load(
+ initial_states_ptr
+ + req_id[:, None] * initial_states_stride_req
+ + c_off[None, :] * initial_states_stride_dim
+ + state_col_safe[:, None],
+ mask=t_mask[:, None] & c_mask[None, :] & valid_state[:, None],
+ other=0.0,
+ ).to(
+ tl.float32
+ ) # (BLOCK_T, BLOCK_C)
+
+ tap = tl.where(in_seq[:, None], x_val, state_val)
+ else:
+ tap = x_val
+
+ acc += tap * w_j[None, :]
+
+ # SiLU activation: x * sigmoid(x)
+ sigmoid_acc = 1.0 / (1.0 + tl.exp(-acc))
+ result = acc * sigmoid_acc
+
+ # Store output (cast back to input dtype)
+ tl.store(
+ out_ptr + t_off[:, None] * conv_dim + c_off[None, :],
+ result,
+ mask=t_mask[:, None] & c_mask[None, :],
+ )
+
+
+def causal_conv1d_varlen_fn(
+ x: torch.Tensor,
+ weight: torch.Tensor,
+ bias: torch.Tensor,
+ cu_seqlens: torch.Tensor,
+ initial_states: torch.Tensor = None,
+ activation: str = "silu",
+ precomputed_seq_idx: torch.Tensor = None,
+ precomputed_seq_start: torch.Tensor = None,
+) -> torch.Tensor:
+ """Depthwise causal 1D convolution over packed variable-length sequences.
+
+ Supports both `cu_seqlens` (sequence boundaries) and `initial_states`
+ simultaneously, unlike `causal_conv1d_fn` which requires mutual exclusivity
+ between `seq_idx` and `initial_states`.
+
+ Args:
+ x: Input tensor of shape (total_tokens, conv_dim), channels-last packed.
+ weight: Convolution weights of shape (conv_dim, d_conv).
+ bias: Bias of shape (conv_dim,).
+ cu_seqlens: Cumulative sequence lengths of shape (num_requests + 1,), int32.
+ initial_states: Per-request initial conv states of shape
+ (num_requests, conv_dim, d_conv - 1). If None, uses zeros.
+ activation: Activation function, must be "silu".
+ precomputed_seq_idx: Precomputed per-token request ID of shape
+ (total_tokens,). If provided, skips repeat_interleave (CUDA
+ graph compatible). Padding tokens should use 0 as sentinel.
+ precomputed_seq_start: Precomputed per-token request start position
+ of shape (total_tokens,). Must be provided together with
+ precomputed_seq_idx.
+
+ Returns:
+ Output tensor of shape (total_tokens, conv_dim).
+ """
+ assert activation == "silu", f"Only silu activation is supported, got {activation}"
+ assert x.is_contiguous(), "x must be contiguous"
+ assert weight.is_contiguous(), "weight must be contiguous"
+
+ total_tokens, conv_dim = x.shape
+ d_conv = weight.shape[1]
+ num_requests = cu_seqlens.shape[0] - 1
+
+ out = torch.empty_like(x)
+
+ # Use precomputed per-token metadata if provided (CUDA graph compatible),
+ # otherwise compute from cu_seqlens via repeat_interleave.
+ if precomputed_seq_idx is not None:
+ assert precomputed_seq_start is not None
+ seq_idx = precomputed_seq_idx
+ seq_start = precomputed_seq_start
+ else:
+ seq_lengths = cu_seqlens[1:] - cu_seqlens[:-1]
+ seq_idx = torch.repeat_interleave(
+ torch.arange(num_requests, device=x.device, dtype=torch.int32), seq_lengths
+ )
+ seq_start = torch.repeat_interleave(cu_seqlens[:-1], seq_lengths).to(torch.int32)
+
+ has_initial_states = initial_states is not None
+ if not has_initial_states:
+ initial_states = torch.empty((1, 1, 1), dtype=x.dtype, device=x.device)
+ is_stride_req = 1
+ is_stride_dim = 1
+ else:
+ if precomputed_seq_idx is None:
+ assert initial_states.shape == (num_requests, conv_dim, d_conv - 1)
+ is_stride_req = initial_states.stride(0)
+ is_stride_dim = initial_states.stride(1)
+
+ grid = lambda meta: (
+ triton.cdiv(conv_dim, meta["BLOCK_C"]),
+ triton.cdiv(total_tokens, meta["BLOCK_T"]),
+ )
+
+ _causal_conv1d_varlen_kernel[grid](
+ x,
+ weight,
+ bias,
+ seq_idx,
+ seq_start,
+ initial_states,
+ out,
+ total_tokens,
+ conv_dim,
+ is_stride_req,
+ is_stride_dim,
+ WIDTH=d_conv,
+ HAS_INITIAL_STATES=has_initial_states,
+ )
+
+ return out
+
+
+def _causal_conv1d_varlen_simple(
+ x: torch.Tensor,
+ weight: torch.Tensor,
+ bias: torch.Tensor,
+ cu_seqlens: torch.Tensor,
+ initial_states: torch.Tensor,
+ out: torch.Tensor,
+) -> None:
+ """Simple PyTorch implementation of varlen causal conv1d with initial states and SiLU.
+
+ This is a reference implementation for testing. Processes each request and token
+ sequentially.
+ """
+ total_tokens, conv_dim = x.shape
+ d_conv = weight.shape[1]
+ num_requests = cu_seqlens.shape[0] - 1
+
+ for r in range(num_requests):
+ start = cu_seqlens[r].item()
+ end = cu_seqlens[r + 1].item()
+ seq_len = end - start
+
+ if seq_len == 0:
+ continue
+
+ if initial_states is not None:
+ init_state = initial_states[r] # (conv_dim, d_conv - 1)
+ else:
+ init_state = torch.zeros((conv_dim, d_conv - 1), dtype=x.dtype, device=x.device)
+
+ x_seq = x[start:end] # (seq_len, conv_dim)
+
+ for t in range(seq_len):
+ acc = bias.float() # (conv_dim,)
+ for j in range(d_conv):
+ src_pos = t - (d_conv - 1) + j
+ if src_pos < 0:
+ state_col = (d_conv - 1) + src_pos
+ if state_col >= 0 and state_col < d_conv - 1:
+ tap = init_state[:, state_col].float()
+ else:
+ tap = torch.zeros(conv_dim, dtype=torch.float32, device=x.device)
+ else:
+ tap = x_seq[src_pos].float()
+
+ acc = acc + tap * weight[:, j].float()
+
+ result = acc * torch.sigmoid(acc)
+ out[start + t] = result.to(out.dtype)
diff --git a/megatron/core/ssm/ops/determinism.py b/megatron/core/ssm/ops/determinism.py
new file mode 100644
index 00000000000..d642e8bba01
--- /dev/null
+++ b/megatron/core/ssm/ops/determinism.py
@@ -0,0 +1,123 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2024, Tri Dao, Albert Gu.
+
+# Some of this code was adopted from https://github.com/state-spaces/mamba/
+# This source code is licensed under the Apache license found in the
+# LICENSE file in the root directory of this source tree.
+
+import os
+import warnings
+
+import torch
+from packaging import version
+
+try:
+ import triton
+
+ TRITON_VERSION = version.parse(triton.__version__)
+except ImportError:
+ TRITON_VERSION = version.parse("0.0.0")
+
+TRITON_HAS_CACHE_RESULTS = TRITON_VERSION >= version.parse("3.4.0")
+_autotune_warning_issued = False
+
+_deterministic_override = None
+
+
+def use_deterministic_mode():
+ """Use torch deterministic mode."""
+ if _deterministic_override is not None:
+ return _deterministic_override
+ env = os.environ.get('MAMBA_DETERMINISTIC')
+ if env:
+ return env[0] == '1'
+ return torch.are_deterministic_algorithms_enabled()
+
+
+def set_deterministic_mode(value):
+ """Set torch deterministic mode."""
+ global _deterministic_override
+ _deterministic_override = value
+
+
+def _estimate_config_cost(cfg):
+ """Estimate shared memory cost of a config. Lower is cheaper.
+
+ Returns a tuple (block_cost, num_warps) so that ties in block cost
+ are broken deterministically by warp count (fewer warps = cheaper).
+ """
+ block_product = 1
+ for key, val in cfg.kwargs.items():
+ if key.startswith('BLOCK') and isinstance(val, int):
+ block_product *= val
+ stages = getattr(cfg, 'num_stages', 1) or 1
+ warps = getattr(cfg, 'num_warps', 1) or 1
+ return (block_product * stages, warps)
+
+
+def _filter_configs_by_block_sizes(configs):
+ """Filter configs by TRITON_AUTOTUNE_BLOCK_* env vars.
+
+ Scans environment for any variable matching TRITON_AUTOTUNE_BLOCK_*
+ (e.g. TRITON_AUTOTUNE_BLOCK_SIZE_M, TRITON_AUTOTUNE_BLOCK_SIZE_H,
+ TRITON_AUTOTUNE_BLOCK_T, TRITON_AUTOTUNE_BLOCK_C, TRITON_AUTOTUNE_BLOCK_SIZE)
+ and maps them to the corresponding kernel kwarg (BLOCK_SIZE_M, BLOCK_SIZE_H,
+ BLOCK_T, BLOCK_C, BLOCK_SIZE).
+ """
+ prefix = "TRITON_AUTOTUNE_"
+ env_filters = {}
+ for env_key, env_val in os.environ.items():
+ if env_key.startswith(prefix + "BLOCK") and env_val:
+ kwarg_name = env_key[len(prefix) :]
+ env_filters[kwarg_name] = int(env_val)
+ if not env_filters:
+ return None
+ matching = configs
+ for key, target in sorted(env_filters.items()):
+ matching = [c for c in matching if c.kwargs.get(key) == target]
+ return matching[:1] if matching else None
+
+
+def autotune_configs(configs):
+ """Select autotune configs for deterministic mode.
+
+ Uses cached autotuning (TRITON_CACHE_AUTOTUNING=1) if Triton >= 3.4.0,
+ otherwise auto-selects the cheapest config by block size * stages.
+ """
+ if not configs or not use_deterministic_mode():
+ return configs
+ if TRITON_HAS_CACHE_RESULTS and os.environ.get("TRITON_CACHE_AUTOTUNING") == "1":
+ return configs
+ global _autotune_warning_issued
+ if not _autotune_warning_issued:
+ _autotune_warning_issued = True
+ msg = (
+ "Deterministic mode: set TRITON_CACHE_AUTOTUNING=1 for cached autotuning."
+ if TRITON_HAS_CACHE_RESULTS
+ else "Deterministic mode: upgrade to Triton >= 3.4.0 for cached autotuning."
+ )
+ warnings.warn(msg)
+ filtered = _filter_configs_by_block_sizes(configs)
+ if filtered:
+ return filtered
+ return [min(configs, key=_estimate_config_cost)]
+
+
+def alloc_tile_workspace(base_shape, tile_dim, dtype, device, deterministic, *, zero_init=True):
+ """Allocate buffer for deterministic per-program reductions."""
+ if base_shape is None:
+ return None, 0
+ if deterministic:
+ factory = torch.zeros if zero_init else torch.empty
+ tensor = factory(*base_shape, tile_dim, device=device, dtype=dtype)
+ return tensor, tensor.stride(-1)
+ return torch.empty(*base_shape, device=device, dtype=dtype), 0
+
+
+def finalize_tile_workspace(tensor, deterministic):
+ """Finalize tile workspace."""
+ if tensor is None:
+ return None
+ if deterministic:
+ tensor = tensor.sum(dim=-1)
+ return tensor
diff --git a/megatron/core/ssm/ops/mamba_ssm.py b/megatron/core/ssm/ops/mamba_ssm.py
new file mode 100644
index 00000000000..4e079da8a31
--- /dev/null
+++ b/megatron/core/ssm/ops/mamba_ssm.py
@@ -0,0 +1,454 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2024, Tri Dao, Albert Gu.
+
+# Some of this code was adopted from https://github.com/state-spaces/mamba/
+# This source code is licensed under the Apache license found in the
+# LICENSE file in the root directory of this source tree.
+
+import torch
+from packaging import version
+
+try:
+ import triton
+ import triton.language as tl
+
+ TRITON3 = version.parse(triton.__version__) >= version.parse("3.0.0")
+
+ HAVE_TRITON = True
+except ImportError:
+ from unittest.mock import MagicMock
+
+ from megatron.core.utils import null_decorator
+
+ triton = MagicMock()
+ triton.jit = null_decorator
+ tl = MagicMock()
+ HAVE_TRITON = False
+ TRITON3 = False
+
+if TRITON3:
+
+ @triton.jit
+ def softplus(dt):
+ """Optimized softplus."""
+ return tl.math.log(tl.math.exp(dt) + 1)
+
+elif HAVE_TRITON:
+
+ @triton.jit
+ def softplus(dt):
+ """Optimized softplus."""
+ return tl.math.log1p(tl.exp(dt))
+
+
+@triton.heuristics({"HAS_DT_BIAS": lambda args: args["dt_bias_ptr"] is not None})
+@triton.heuristics({"HAS_D": lambda args: args["D_ptr"] is not None})
+@triton.heuristics({"HAS_Z": lambda args: args["z_ptr"] is not None})
+@triton.heuristics(
+ {"HAS_STATE_BATCH_INDICES": lambda args: args["state_batch_indices_ptr"] is not None}
+)
+@triton.heuristics({"HAS_INT_STATE": lambda args: args["int_state_ptr"] is not None})
+@triton.heuristics({"BLOCK_SIZE_DSTATE": lambda args: triton.next_power_of_2(args["dstate"])})
+@triton.jit
+def _selective_scan_update_kernel(
+ # Pointers to matrices
+ state_ptr,
+ x_ptr,
+ dt_ptr,
+ dt_bias_ptr,
+ A_ptr,
+ B_ptr,
+ C_ptr,
+ D_ptr,
+ z_ptr,
+ out_ptr,
+ state_batch_indices_ptr,
+ int_state_ptr,
+ # Matrix dimensions
+ batch,
+ seq_len,
+ nheads,
+ dim,
+ dstate,
+ nheads_ngroups_ratio,
+ # Strides
+ stride_state_batch,
+ stride_state_head,
+ stride_state_dim,
+ stride_state_dstate,
+ stride_x_batch,
+ stride_x_seq,
+ stride_x_head,
+ stride_x_dim,
+ stride_dt_batch,
+ stride_dt_seq,
+ stride_dt_head,
+ stride_dt_dim,
+ stride_dt_bias_head,
+ stride_dt_bias_dim,
+ stride_A_head,
+ stride_A_dim,
+ stride_A_dstate,
+ stride_B_batch,
+ stride_B_seq,
+ stride_B_group,
+ stride_B_dstate,
+ stride_C_batch,
+ stride_C_seq,
+ stride_C_group,
+ stride_C_dstate,
+ stride_D_head,
+ stride_D_dim,
+ stride_z_batch,
+ stride_z_seq,
+ stride_z_head,
+ stride_z_dim,
+ stride_out_batch,
+ stride_out_seq,
+ stride_out_head,
+ stride_out_dim,
+ stride_int_batch,
+ stride_int_seq,
+ stride_int_head,
+ stride_int_dim,
+ stride_int_dstate,
+ # Meta-parameters
+ DT_SOFTPLUS: tl.constexpr,
+ TIE_HDIM: tl.constexpr,
+ BLOCK_SIZE_M: tl.constexpr,
+ HAS_DT_BIAS: tl.constexpr,
+ HAS_D: tl.constexpr,
+ HAS_Z: tl.constexpr,
+ HAS_STATE_BATCH_INDICES: tl.constexpr,
+ HAS_INT_STATE: tl.constexpr,
+ BLOCK_SIZE_DSTATE: tl.constexpr,
+):
+ pid_m = tl.program_id(axis=0)
+ pid_b = tl.program_id(axis=1)
+ pid_h = tl.program_id(axis=2)
+
+ offs_m = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+ offs_n = tl.arange(0, BLOCK_SIZE_DSTATE)
+
+ out_ptr += pid_b * stride_out_batch + pid_h * stride_out_head
+ out_ptrs = out_ptr + offs_m * stride_out_dim
+
+ # 1. State Mapping (handles dynamic batching slot allocation)
+ if HAS_STATE_BATCH_INDICES:
+ state_batch_indices_ptr += pid_b
+ state_batch_idx = tl.load(state_batch_indices_ptr)
+ # Skip padding tokens (e.g. from graph capture or inactive slots)
+ if state_batch_idx < 0:
+ for s in range(seq_len):
+ out_s_ptrs = out_ptrs + s * stride_out_seq
+ tl.store(out_s_ptrs, 0.0, mask=offs_m < dim)
+ return
+ state_ptr += state_batch_idx * stride_state_batch + pid_h * stride_state_head
+ if HAS_INT_STATE:
+ int_state_ptr += state_batch_idx * stride_int_batch + pid_h * stride_int_head
+ else:
+ state_ptr += pid_b * stride_state_batch + pid_h * stride_state_head
+ if HAS_INT_STATE:
+ int_state_ptr += pid_b * stride_int_batch + pid_h * stride_int_head
+
+ # Base Pointers for Sequence iteration
+ x_ptr += pid_b * stride_x_batch + pid_h * stride_x_head
+ dt_ptr += pid_b * stride_dt_batch + pid_h * stride_dt_head
+ if HAS_DT_BIAS:
+ dt_bias_ptr += pid_h * stride_dt_bias_head
+
+ A_ptr += pid_h * stride_A_head
+ B_ptr += pid_b * stride_B_batch + (pid_h // nheads_ngroups_ratio) * stride_B_group
+ C_ptr += pid_b * stride_C_batch + (pid_h // nheads_ngroups_ratio) * stride_C_group
+ if HAS_Z:
+ z_ptr += pid_b * stride_z_batch + pid_h * stride_z_head
+
+ # Constant offsets (A, D, and bias do not have a sequence dimension)
+ state_ptrs = state_ptr + (
+ offs_m[:, None] * stride_state_dim + offs_n[None, :] * stride_state_dstate
+ )
+ if HAS_INT_STATE:
+ int_state_ptrs = int_state_ptr + (
+ offs_m[:, None] * stride_int_dim + offs_n[None, :] * stride_int_dstate
+ )
+
+ x_ptrs = x_ptr + offs_m * stride_x_dim
+ dt_ptrs = dt_ptr + offs_m * stride_dt_dim
+
+ if HAS_DT_BIAS:
+ dt_bias_ptrs = dt_bias_ptr + offs_m * stride_dt_bias_dim
+ if HAS_D:
+ D_ptr += pid_h * stride_D_head
+ D_ptrs = D_ptr + offs_m * stride_D_dim
+
+ A_ptrs = A_ptr + (offs_m[:, None] * stride_A_dim + offs_n[None, :] * stride_A_dstate)
+ B_ptrs = B_ptr + offs_n * stride_B_dstate
+ C_ptrs = C_ptr + offs_n * stride_C_dstate
+
+ if HAS_Z:
+ z_ptrs = z_ptr + offs_m * stride_z_dim
+
+ # Load initial historical state and constant parameters
+ state = tl.load(
+ state_ptrs, mask=(offs_m[:, None] < dim) & (offs_n[None, :] < dstate), other=0.0
+ ).to(tl.float32)
+
+ if not TIE_HDIM:
+ A = tl.load(
+ A_ptrs, mask=(offs_m[:, None] < dim) & (offs_n[None, :] < dstate), other=0.0
+ ).to(tl.float32)
+ else:
+ A = tl.load(A_ptr).to(tl.float32)
+
+ if HAS_D:
+ D = tl.load(D_ptrs, mask=offs_m < dim, other=0.0).to(tl.float32)
+
+ # ----------------------------------------------------
+ # Sequence Loop (Processes Main Token + Speculative Drafts)
+ # ----------------------------------------------------
+ for s in range(seq_len):
+ x_s_ptrs = x_ptrs + s * stride_x_seq
+ dt_s_ptrs = dt_ptrs + s * stride_dt_seq
+ B_s_ptrs = B_ptrs + s * stride_B_seq
+ C_s_ptrs = C_ptrs + s * stride_C_seq
+ if HAS_Z:
+ z_s_ptrs = z_ptrs + s * stride_z_seq
+
+ x = tl.load(x_s_ptrs, mask=offs_m < dim, other=0.0).to(tl.float32)
+
+ # Calculate dt and dA
+ if not TIE_HDIM:
+ dt = tl.load(dt_s_ptrs, mask=offs_m < dim, other=0.0).to(tl.float32)
+ if HAS_DT_BIAS:
+ dt += tl.load(dt_bias_ptrs, mask=offs_m < dim, other=0.0).to(tl.float32)
+ if DT_SOFTPLUS:
+ dt = tl.where(dt <= 20.0, softplus(dt), dt)
+ dA = tl.exp(A * dt[:, None])
+ else:
+ dt = tl.load(dt_ptr + s * stride_dt_seq).to(tl.float32)
+ if HAS_DT_BIAS:
+ dt += tl.load(dt_bias_ptr).to(tl.float32)
+ if DT_SOFTPLUS:
+ dt = tl.where(dt <= 20.0, softplus(dt), dt)
+ dA = tl.exp(A * dt)
+
+ # Load B and C
+ B = tl.load(B_s_ptrs, mask=offs_n < dstate, other=0.0).to(tl.float32)
+ C = tl.load(C_s_ptrs, mask=offs_n < dstate, other=0.0).to(tl.float32)
+ if HAS_Z:
+ z = tl.load(z_s_ptrs, mask=offs_m < dim, other=0.0).to(tl.float32)
+
+ if not TIE_HDIM:
+ dB = B[None, :] * dt[:, None]
+ else:
+ dB = B * dt
+
+ # ----------------------------------------------------
+ # The Core State Recurrence (h_t = dA * h_{t-1} + dB * x_t)
+ # ----------------------------------------------------
+ state = state * dA + dB * x[:, None]
+
+ # ----------------------------------------------------
+ # Dump Intermediate Speculative State Snapshot
+ # ----------------------------------------------------
+ if HAS_INT_STATE:
+ int_state_s_ptrs = int_state_ptrs + s * stride_int_seq
+ tl.store(
+ int_state_s_ptrs, state, mask=(offs_m[:, None] < dim) & (offs_n[None, :] < dstate)
+ )
+
+ # Calculate Output
+ out = tl.sum(state * C[None, :], axis=1)
+ if HAS_D:
+ out += x * D
+ if HAS_Z:
+ out *= z * tl.sigmoid(z)
+
+ out_s_ptrs = out_ptrs + s * stride_out_seq
+ tl.store(out_s_ptrs, out, mask=offs_m < dim)
+
+ # After processing all sequence steps, persist the final state back to HBM
+ tl.store(state_ptrs, state, mask=(offs_m[:, None] < dim) & (offs_n[None, :] < dstate))
+
+
+def selective_state_update(
+ state,
+ x,
+ dt,
+ A,
+ B,
+ C,
+ D=None,
+ z=None,
+ dt_bias=None,
+ dt_softplus=False,
+ state_batch_indices=None,
+ intermediate_ssm_states=None,
+):
+ """
+ Argument:
+ state: (batch, dim, dstate) or (batch, nheads, dim, dstate)
+ x: (batch, dim), (batch, seqlen, dim), (batch, nheads, dim) or (batch, seqlen, nheads, dim)
+ dt: Matches x
+ A: (dim, dstate) or (nheads, dim, dstate)
+ B: (batch, dstate), (batch, seqlen, dstate), (batch, ngroups, dstate) or
+ (batch, seqlen, ngroups, dstate)
+ C: Matches B
+ D: (dim,) or (nheads, dim)
+ z: Matches x
+ dt_bias: (dim,) or (nheads, dim)
+ intermediate_ssm_states: Optional buffer of shape (batch, seqlen, nheads, dim, dstate)
+ or (batch, seqlen, dim, dstate)
+ Return:
+ out: shape matches x
+ """
+ has_heads = state.dim() > 3
+ if not has_heads:
+ state = state.unsqueeze(1)
+
+ # Standardize inputs to explicit sequence and head dimensions: (batch, seq_len, nheads, dim)
+ is_seq_unsq = False
+ if has_heads:
+ if x.dim() == 3: # (batch, nheads, dim) -> (batch, 1, nheads, dim)
+ x = x.unsqueeze(1)
+ dt = dt.unsqueeze(1)
+ B = B.unsqueeze(1)
+ C = C.unsqueeze(1)
+ if z is not None:
+ z = z.unsqueeze(1)
+ is_seq_unsq = True
+ else:
+ if x.dim() == 2: # (batch, dim) -> (batch, 1, 1, dim)
+ x = x.unsqueeze(1).unsqueeze(2)
+ dt = dt.unsqueeze(1).unsqueeze(2)
+ B = B.unsqueeze(1).unsqueeze(2)
+ C = C.unsqueeze(1).unsqueeze(2)
+ if z is not None:
+ z = z.unsqueeze(1).unsqueeze(2)
+ is_seq_unsq = True
+ elif x.dim() == 3: # (batch, seqlen, dim) -> (batch, seqlen, 1, dim)
+ x = x.unsqueeze(2)
+ dt = dt.unsqueeze(2)
+ B = B.unsqueeze(2)
+ C = C.unsqueeze(2)
+ if z is not None:
+ z = z.unsqueeze(2)
+
+ if A.dim() == 2:
+ A = A.unsqueeze(0)
+ if D is not None and D.dim() == 1:
+ D = D.unsqueeze(0)
+ if dt_bias is not None and dt_bias.dim() == 1:
+ dt_bias = dt_bias.unsqueeze(0)
+
+ # Set up Intermediate State standardization
+ if intermediate_ssm_states is not None:
+ if not has_heads and intermediate_ssm_states.dim() == 4:
+ intermediate_ssm_states = intermediate_ssm_states.unsqueeze(
+ 2
+ ) # (batch, seqlen, 1, dim, dstate)
+ int_state_strides = (
+ intermediate_ssm_states.stride(0),
+ intermediate_ssm_states.stride(1),
+ intermediate_ssm_states.stride(2),
+ intermediate_ssm_states.stride(3),
+ intermediate_ssm_states.stride(4),
+ )
+ else:
+ intermediate_ssm_states = x # Dummy pointer
+ int_state_strides = (0, 0, 0, 0, 0)
+
+ batch, seq_len, nheads, dim = x.shape
+ dstate = state.shape[-1]
+ ngroups = B.shape[-2]
+
+ out = torch.empty_like(x)
+ grid = lambda META: (triton.cdiv(dim, META['BLOCK_SIZE_M']), batch, nheads)
+ z_strides = (
+ (z.stride(0), z.stride(1), z.stride(2), z.stride(3)) if z is not None else (0, 0, 0, 0)
+ )
+
+ BLOCK_SIZE_M, num_warps = (
+ (32, 4)
+ if dstate <= 16
+ else (
+ (16, 4)
+ if dstate <= 32
+ else ((8, 4) if dstate <= 64 else ((4, 4) if dstate <= 128 else ((4, 8))))
+ )
+ )
+
+ tie_hdim = (
+ A.stride(-1) == 0
+ and A.stride(-2) == 0
+ and dt.stride(-1) == 0
+ and (dt_bias is None or dt_bias.stride(-1) == 0)
+ )
+
+ with torch.cuda.device(x.device.index):
+ _selective_scan_update_kernel[grid](
+ state,
+ x,
+ dt,
+ dt_bias,
+ A,
+ B,
+ C,
+ D,
+ z,
+ out,
+ state_batch_indices,
+ intermediate_ssm_states,
+ batch,
+ seq_len,
+ nheads,
+ dim,
+ dstate,
+ nheads // ngroups,
+ state.stride(0),
+ state.stride(1),
+ state.stride(2),
+ state.stride(3),
+ x.stride(0),
+ x.stride(1),
+ x.stride(2),
+ x.stride(3),
+ dt.stride(0),
+ dt.stride(1),
+ dt.stride(2),
+ dt.stride(3),
+ *(dt_bias.stride(0), dt_bias.stride(1)) if dt_bias is not None else (0, 0),
+ A.stride(0),
+ A.stride(1),
+ A.stride(2),
+ B.stride(0),
+ B.stride(1),
+ B.stride(2),
+ B.stride(3),
+ C.stride(0),
+ C.stride(1),
+ C.stride(2),
+ C.stride(3),
+ *(D.stride(0), D.stride(1)) if D is not None else (0, 0),
+ z_strides[0],
+ z_strides[1],
+ z_strides[2],
+ z_strides[3],
+ out.stride(0),
+ out.stride(1),
+ out.stride(2),
+ out.stride(3),
+ *int_state_strides,
+ dt_softplus,
+ tie_hdim,
+ BLOCK_SIZE_M,
+ num_warps=num_warps,
+ )
+
+ # Revert dimensions back to match original x format
+ if not has_heads:
+ out = out.squeeze(2)
+ if is_seq_unsq:
+ out = out.squeeze(1)
+
+ return out
diff --git a/megatron/core/ssm/ops/ssd_bmm.py b/megatron/core/ssm/ops/ssd_bmm.py
new file mode 100644
index 00000000000..0cbb07fdbf5
--- /dev/null
+++ b/megatron/core/ssm/ops/ssd_bmm.py
@@ -0,0 +1,199 @@
+# Copyright (c) 2024, Tri Dao, Albert Gu.
+# Adapted from https://github.com/state-spaces/mamba/blob/v2.2.4/mamba_ssm/ops/triton/ssd_bmm.py
+# Adapted from vLLM project (Apache-2.0).
+
+import torch
+import triton
+import triton.language as tl
+
+from megatron.core.ssm.ops.determinism import autotune_configs
+
+
+@triton.autotune(
+ configs=autotune_configs(
+ [
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 256, "BLOCK_SIZE_K": 64},
+ num_stages=3,
+ num_warps=8,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 256, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 128, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 64, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 128, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 32, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 32, "BLOCK_SIZE_K": 32},
+ num_stages=5,
+ num_warps=2,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 32, "BLOCK_SIZE_N": 64, "BLOCK_SIZE_K": 32},
+ num_stages=5,
+ num_warps=2,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 64, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=2,
+ ),
+ ]
+ ),
+ key=["chunk_size", "K", "IS_CAUSAL"],
+)
+@triton.jit
+def _bmm_chunk_fwd_kernel(
+ # Pointers to matrices
+ a_ptr,
+ b_ptr,
+ out_ptr,
+ cu_chunk_seqlens_ptr,
+ # Matrix dimensions
+ seqlen,
+ chunk_size: tl.constexpr,
+ K: tl.constexpr,
+ ngroups: tl.constexpr,
+ stride_a_seqlen: tl.int64,
+ stride_a_head: tl.int64,
+ stride_ak: tl.constexpr,
+ stride_b_seqlen: tl.int64,
+ stride_b_head: tl.int64,
+ stride_bk: tl.constexpr,
+ stride_out_chunk: tl.int64,
+ stride_out_head: tl.int64,
+ stride_outm: tl.int64,
+ stride_outn: tl.constexpr,
+ # Meta-parameters
+ IS_CAUSAL: tl.constexpr,
+ dot_dtype: tl.constexpr,
+ BLOCK_SIZE_M: tl.constexpr,
+ BLOCK_SIZE_N: tl.constexpr,
+ BLOCK_SIZE_K: tl.constexpr,
+):
+ pid_ch = tl.program_id(axis=1).to(tl.int64)
+ pid_c = pid_ch // ngroups
+ pid_h = pid_ch - pid_c * ngroups
+ num_pid_n = tl.cdiv(chunk_size, BLOCK_SIZE_N)
+ pid_m = tl.program_id(axis=0) // num_pid_n
+ pid_n = tl.program_id(axis=0) % num_pid_n
+ if IS_CAUSAL:
+ if pid_n * BLOCK_SIZE_N >= (pid_m + 1) * BLOCK_SIZE_M:
+ return
+
+ chunk_seqlen_start = tl.load(cu_chunk_seqlens_ptr + pid_c)
+ chunk_seqlen_end = tl.load(cu_chunk_seqlens_ptr + pid_c + 1)
+
+ a_ptr += chunk_seqlen_start * stride_a_seqlen + pid_h * stride_a_head
+ b_ptr += chunk_seqlen_start * stride_b_seqlen + pid_h * stride_b_head
+
+ offs_m = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+ offs_n = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+ offs_k = tl.arange(0, BLOCK_SIZE_K)
+ a_ptrs = a_ptr + (offs_m[:, None] * stride_a_seqlen + offs_k[None, :] * stride_ak)
+ b_ptrs = b_ptr + (offs_k[:, None] * stride_bk + offs_n[None, :] * stride_b_seqlen)
+ chunk_size_limit = chunk_seqlen_end - chunk_seqlen_start
+
+ acc = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
+
+ # compute a * b.T
+ for k in range(0, tl.cdiv(K, BLOCK_SIZE_K)):
+ a = tl.load(
+ a_ptrs,
+ mask=(offs_m[:, None] < chunk_size_limit) & (offs_k[None, :] < K - k * BLOCK_SIZE_K),
+ other=0.0,
+ ).to(dot_dtype)
+ b = tl.load(
+ b_ptrs,
+ mask=(offs_k[:, None] < K - k * BLOCK_SIZE_K) & (offs_n[None, :] < chunk_size_limit),
+ other=0.0,
+ ).to(dot_dtype)
+ acc += tl.dot(a, b)
+ a_ptrs += BLOCK_SIZE_K * stride_ak
+ b_ptrs += BLOCK_SIZE_K * stride_bk
+
+ offs_m = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+ offs_n = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+
+ out = acc.to(out_ptr.dtype.element_ty)
+ out_ptr += pid_c * stride_out_chunk + pid_h * stride_out_head
+ out_ptrs = out_ptr + (stride_outm * offs_m[:, None] + offs_n[None, :] * stride_outn)
+ tl.store(out_ptrs, out, mask=(offs_m[:, None] < chunk_size) & (offs_n[None, :] < chunk_size))
+
+
+def _bmm_chunk_fwd(a, b, chunk_size, cu_chunk_seqlens, causal=False, output_dtype=None):
+ """
+ Argument:
+ a: (seqlen, ngroups, k)
+ b: (seqlen, ngroups, k)
+ chunk_size: int
+ cu_chunk_seq_lens: (nchunks+1,)
+ causal: if True, then out[i, j] for i > j will be arbitrary, only out[i, j] for i <= j are
+ guaranteed to be correct.
+ Return:
+ out: (nchunks, ngroups, chunk_size, chunk_size)
+ """
+ seqlen, ngroups, k = a.shape
+ assert b.shape == a.shape
+ if a.stride(-1) != 1 and a.stride(0) != 1:
+ a = a.contiguous()
+ if b.stride(-1) != 1 and b.stride(0) != 1:
+ b = b.contiguous()
+
+ nchunks = len(cu_chunk_seqlens) - 1
+ # Allocates output.
+ out_dtype = a.dtype if output_dtype is None else output_dtype
+ out = torch.empty((nchunks, ngroups, chunk_size, chunk_size), device=a.device, dtype=out_dtype)
+ dot_dtype = (
+ tl.bfloat16
+ if a.dtype == torch.bfloat16 or b.dtype == torch.bfloat16
+ else (tl.float16 if a.dtype == torch.float16 or b.dtype == torch.float16 else tl.float32)
+ )
+ grid = lambda META: (
+ triton.cdiv(chunk_size, META["BLOCK_SIZE_M"])
+ * triton.cdiv(chunk_size, META["BLOCK_SIZE_N"]),
+ nchunks * ngroups,
+ )
+ with torch.cuda.device(a.device.index):
+ _bmm_chunk_fwd_kernel[grid](
+ a_ptr=a,
+ b_ptr=b,
+ out_ptr=out,
+ cu_chunk_seqlens_ptr=cu_chunk_seqlens,
+ seqlen=seqlen,
+ chunk_size=chunk_size,
+ K=k,
+ ngroups=ngroups,
+ stride_a_seqlen=a.stride(0),
+ stride_a_head=a.stride(1),
+ stride_ak=a.stride(2),
+ stride_b_seqlen=b.stride(0),
+ stride_b_head=b.stride(1),
+ stride_bk=b.stride(2),
+ stride_out_chunk=out.stride(0),
+ stride_out_head=out.stride(1),
+ stride_outm=out.stride(-2),
+ stride_outn=out.stride(-1),
+ IS_CAUSAL=causal,
+ dot_dtype=dot_dtype,
+ )
+ return out
diff --git a/megatron/core/ssm/ops/ssd_chunk_scan.py b/megatron/core/ssm/ops/ssd_chunk_scan.py
new file mode 100644
index 00000000000..521a294db5d
--- /dev/null
+++ b/megatron/core/ssm/ops/ssd_chunk_scan.py
@@ -0,0 +1,424 @@
+# Copyright (c) 2024, Tri Dao, Albert Gu.
+# Adapted from:
+# https://github.com/state-spaces/mamba/blob/v2.2.4/mamba_ssm/ops/triton/ssd_chunk_scan.py
+# Adapted from vLLM project (Apache-2.0).
+
+import triton
+import triton.language as tl
+from packaging import version
+
+from megatron.core.ssm.ops.determinism import autotune_configs
+
+TRITON_22 = version.parse(triton.__version__) >= version.parse("2.2.0")
+
+
+@triton.autotune(
+ configs=autotune_configs(
+ [
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 256, "BLOCK_SIZE_K": 64},
+ num_stages=3,
+ num_warps=8,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 256, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 128, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 64, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 128, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 64, "BLOCK_SIZE_K": 64},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 128, "BLOCK_SIZE_K": 64},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 32, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 32, "BLOCK_SIZE_K": 32},
+ num_stages=5,
+ num_warps=2,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 32, "BLOCK_SIZE_N": 64, "BLOCK_SIZE_K": 32},
+ num_stages=5,
+ num_warps=2,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 64, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=2,
+ ),
+ ]
+ ),
+ key=["chunk_size", "hdim", "dstate", "IS_CAUSAL"],
+)
+@triton.jit
+def _chunk_scan_fwd_kernel(
+ # Pointers to matrices
+ cb_ptr,
+ x_ptr,
+ z_ptr,
+ out_ptr,
+ dt_ptr,
+ dA_cumsum_ptr,
+ seq_idx_ptr,
+ C_ptr,
+ states_ptr,
+ D_ptr,
+ initstates_ptr,
+ cu_chunk_seqlens_ptr,
+ # Matrix dimensions
+ chunk_size: tl.constexpr,
+ hdim: tl.constexpr,
+ dstate: tl.constexpr,
+ seqlen,
+ nheads_ngroups_ratio: tl.constexpr,
+ # Strides
+ stride_cb_chunk: tl.int64,
+ stride_cb_head: tl.int64,
+ stride_cb_csize_m: tl.int64,
+ stride_cb_csize_k: tl.constexpr,
+ stride_x_seqlen: tl.int64,
+ stride_x_head: tl.int64,
+ stride_x_hdim: tl.constexpr,
+ stride_z_seqlen: tl.int64,
+ stride_z_head: tl.int64,
+ stride_z_hdim: tl.constexpr,
+ stride_out_seqlen: tl.int64,
+ stride_out_head: tl.int64,
+ stride_out_hdim: tl.constexpr,
+ stride_dt_chunk: tl.int64,
+ stride_dt_head: tl.int64,
+ stride_dt_csize: tl.constexpr,
+ stride_dA_cs_chunk: tl.int64,
+ stride_dA_cs_head: tl.int64,
+ stride_dA_cs_csize: tl.constexpr,
+ stride_seq_idx_chunk: tl.constexpr,
+ stride_C_seqlen: tl.int64,
+ stride_C_head: tl.int64,
+ stride_C_dstate: tl.constexpr,
+ stride_states_chunk: tl.int64,
+ stride_states_head: tl.int64,
+ stride_states_hdim: tl.int64,
+ stride_states_dstate: tl.constexpr,
+ stride_init_states_batch: tl.int64,
+ stride_init_states_head: tl.int64,
+ stride_init_states_hdim: tl.int64,
+ stride_init_states_dstate: tl.constexpr,
+ stride_D_head: tl.constexpr,
+ # Meta-parameters
+ IS_CAUSAL: tl.constexpr,
+ HAS_D: tl.constexpr,
+ D_HAS_HDIM: tl.constexpr,
+ HAS_Z: tl.constexpr,
+ BLOCK_SIZE_M: tl.constexpr,
+ BLOCK_SIZE_N: tl.constexpr,
+ BLOCK_SIZE_K: tl.constexpr,
+ BLOCK_SIZE_DSTATE: tl.constexpr,
+ IS_TRITON_22: tl.constexpr,
+ HAS_INITSTATES: tl.constexpr,
+):
+ pid_c = tl.program_id(axis=1).to(tl.int64)
+ pid_h = tl.program_id(axis=2)
+ num_pid_n = tl.cdiv(hdim, BLOCK_SIZE_N)
+ pid_m = tl.program_id(axis=0) // num_pid_n
+ pid_n = tl.program_id(axis=0) % num_pid_n
+ cb_ptr += pid_c * stride_cb_chunk + (pid_h // nheads_ngroups_ratio) * stride_cb_head
+ chunk_seqlen_start = tl.load(cu_chunk_seqlens_ptr + pid_c)
+ chunk_seqlen_end = tl.load(cu_chunk_seqlens_ptr + pid_c + 1)
+ x_ptr += chunk_seqlen_start * stride_x_seqlen + pid_h * stride_x_head
+ dt_ptr += pid_c * stride_dt_chunk + pid_h * stride_dt_head
+ dA_cumsum_ptr += pid_c * stride_dA_cs_chunk + pid_h * stride_dA_cs_head
+ C_ptr += chunk_seqlen_start * stride_C_seqlen + (pid_h // nheads_ngroups_ratio) * stride_C_head
+
+ # M-block offsets and prev states
+ # - logic in next block may override these if there is an active offset
+ offs_m = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+
+ seq_idx_ptr += pid_c * stride_seq_idx_chunk
+ seq_idx = tl.load(seq_idx_ptr)
+ seq_idx_prev = tl.load(seq_idx_ptr - stride_seq_idx_chunk, mask=pid_c >= 1, other=-1)
+
+ if HAS_INITSTATES and (seq_idx != seq_idx_prev):
+ prev_states_ptr = (
+ initstates_ptr + seq_idx * stride_init_states_batch + pid_h * stride_init_states_head
+ )
+ prev_states_hdim = stride_init_states_hdim
+ prev_states_dstate = stride_init_states_dstate
+ else:
+ prev_states_ptr = (
+ states_ptr + (pid_c - 1) * stride_states_chunk + pid_h * stride_states_head
+ )
+ prev_states_hdim = stride_states_hdim
+ prev_states_dstate = stride_states_dstate
+
+ chunk_size_limit = chunk_seqlen_end - chunk_seqlen_start
+
+ offs_n = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+ dA_cs_m = tl.load(
+ dA_cumsum_ptr + offs_m * stride_dA_cs_csize, mask=offs_m < chunk_size, other=0.0
+ ).to(tl.float32)
+
+ acc = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
+
+ offs_out_m = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+ offs_out_n = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+
+ # Faster to just do 1 iteration with larger BLOCK_SIZE_K, up to block size 128
+ offs_k_dstate = tl.arange(0, BLOCK_SIZE_DSTATE if BLOCK_SIZE_DSTATE <= 128 else BLOCK_SIZE_K)
+ C_ptrs = C_ptr + (offs_m[:, None] * stride_C_seqlen + offs_k_dstate[None, :] * stride_C_dstate)
+
+ scale_m = tl.exp(dA_cs_m)
+ if BLOCK_SIZE_DSTATE <= 128:
+ C = tl.load(
+ C_ptrs,
+ mask=(offs_m[:, None] < chunk_size_limit) & (offs_k_dstate[None, :] < dstate),
+ other=0.0,
+ )
+
+ if not HAS_INITSTATES and (seq_idx != seq_idx_prev):
+ # if no init states AND starting a new sequence, we need zeros
+ prev_states = tl.zeros((BLOCK_SIZE_DSTATE, BLOCK_SIZE_N), dtype=C_ptr.dtype.element_ty)
+ else:
+ # otherwise read the previous state
+ prev_states_ptrs = (
+ prev_states_ptr
+ + offs_n[None, :] * prev_states_hdim
+ + offs_k_dstate[:, None] * prev_states_dstate
+ )
+ prev_states = tl.load(
+ prev_states_ptrs,
+ mask=(offs_k_dstate[:, None] < dstate) & (offs_n[None, :] < hdim),
+ other=0.0,
+ )
+ prev_states = prev_states.to(C_ptr.dtype.element_ty)
+
+ acc = tl.dot(C, prev_states) * scale_m[:, None]
+
+ else:
+ prev_states_ptrs = (
+ prev_states_ptr
+ + offs_n[None, :] * prev_states_hdim
+ + offs_k_dstate[:, None] * prev_states_dstate
+ )
+ for k in range(0, dstate, BLOCK_SIZE_K):
+ C = tl.load(
+ C_ptrs,
+ mask=(offs_m[:, None] < chunk_size_limit) & (offs_k_dstate[None, :] < dstate - k),
+ other=0.0,
+ )
+ if not HAS_INITSTATES and (seq_idx != seq_idx_prev):
+ prev_states = tl.zeros((BLOCK_SIZE_K, BLOCK_SIZE_N), dtype=C_ptr.dtype.element_ty)
+ else:
+ prev_states = tl.load(
+ prev_states_ptrs,
+ mask=(offs_k_dstate[:, None] < dstate - k) & (offs_n[None, :] < hdim),
+ other=0.0,
+ )
+ prev_states = prev_states.to(C_ptr.dtype.element_ty)
+ acc += tl.dot(C, prev_states)
+ C_ptrs += BLOCK_SIZE_K
+ prev_states_ptrs += BLOCK_SIZE_K
+ acc *= scale_m[:, None]
+
+ offs_k = tl.arange(0, BLOCK_SIZE_K)
+ cb_ptrs = cb_ptr + (offs_m[:, None] * stride_cb_csize_m + offs_k[None, :] * stride_cb_csize_k)
+ x_ptrs = x_ptr + (offs_k[:, None] * stride_x_seqlen + offs_n[None, :] * stride_x_hdim)
+ dt_ptrs = dt_ptr + offs_k * stride_dt_csize
+ dA_cumsum_ptrs = dA_cumsum_ptr + offs_k * stride_dA_cs_csize
+ K_MAX = chunk_size_limit if not IS_CAUSAL else min((pid_m + 1) * BLOCK_SIZE_M, chunk_size_limit)
+ for k in range(0, K_MAX, BLOCK_SIZE_K):
+ cb = tl.load(
+ cb_ptrs,
+ mask=(offs_m[:, None] < chunk_size) & (offs_k[None, :] < chunk_size - k),
+ other=0.0,
+ ).to(tl.float32)
+ dA_cs_k = tl.load(dA_cumsum_ptrs, mask=offs_k < chunk_size - k, other=0.0).to(tl.float32)
+ # If there's seq_idx, we already set cb[i, j] = 0 for seq_idx[i] != seq_idx[j].
+ # So we don't need masking wrt seq_idx here.
+ cb *= tl.exp(tl.minimum(dA_cs_m[:, None] - dA_cs_k[None, :], 0.0))
+ dt_k = tl.load(dt_ptrs, mask=offs_k < chunk_size - k, other=0.0).to(tl.float32)
+ cb *= dt_k
+ if IS_CAUSAL:
+ mask = offs_m[:, None] >= k + offs_k[None, :]
+ cb = tl.where(mask, cb, 0.0)
+ cb = cb.to(x_ptr.dtype.element_ty)
+ x = tl.load(
+ x_ptrs,
+ mask=(offs_k[:, None] < chunk_size_limit - k) & (offs_n[None, :] < hdim),
+ other=0.0,
+ )
+ acc += tl.dot(cb, x)
+ cb_ptrs += BLOCK_SIZE_K * stride_cb_csize_k
+ x_ptrs += BLOCK_SIZE_K * stride_x_seqlen
+ dt_ptrs += BLOCK_SIZE_K * stride_dt_csize
+ dA_cumsum_ptrs += BLOCK_SIZE_K * stride_dA_cs_csize
+
+ offs_out_m = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+ offs_out_n = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+
+ if HAS_D:
+ if D_HAS_HDIM:
+ D = tl.load(D_ptr + pid_h * stride_D_head + offs_n, mask=offs_n < hdim, other=0.0).to(
+ tl.float32
+ )
+ else:
+ D = tl.load(D_ptr + pid_h * stride_D_head).to(tl.float32)
+ x_residual = tl.load(
+ x_ptr + (offs_m[:, None] * stride_x_seqlen + offs_n[None, :] * stride_x_hdim),
+ mask=(offs_m[:, None] < chunk_size_limit) & (offs_n[None, :] < hdim),
+ other=0.0,
+ ).to(tl.float32)
+ acc += x_residual * D
+
+ if HAS_Z:
+ z_ptr += chunk_seqlen_start * stride_z_seqlen + pid_h * stride_z_head
+ z_ptrs = z_ptr + (
+ stride_z_seqlen * offs_out_m[:, None] + stride_z_hdim * offs_out_n[None, :]
+ )
+ z = tl.load(
+ z_ptrs,
+ mask=(offs_out_m[:, None] < chunk_size_limit) & (offs_out_n[None, :] < hdim),
+ other=0.0,
+ ).to(tl.float32)
+ acc *= z * tl.sigmoid(z)
+
+ out_ptr += chunk_seqlen_start * stride_out_seqlen + pid_h * stride_out_head
+ out_ptrs = out_ptr + (
+ stride_out_seqlen * offs_out_m[:, None] + offs_out_n[None, :] * stride_out_hdim
+ )
+ tl.store(
+ out_ptrs, acc, mask=(offs_out_m[:, None] < chunk_size_limit) & (offs_out_n[None, :] < hdim)
+ )
+
+
+def _chunk_scan_fwd(
+ cb,
+ x,
+ dt,
+ dA_cumsum,
+ C,
+ states,
+ cu_chunk_seqlens,
+ out,
+ seq_idx,
+ D=None,
+ z=None,
+ initial_states=None,
+):
+ assert seq_idx is not None, "this implementation requires seq_idx"
+
+ seqlen, nheads, headdim = x.shape
+ _, nchunks, chunk_size = dt.shape
+ _, ngroups, dstate = C.shape
+ assert nheads % ngroups == 0
+ assert C.shape == (seqlen, ngroups, dstate)
+ assert cb.shape == (nchunks, ngroups, chunk_size, chunk_size)
+ if D is not None:
+ assert D.shape == (nheads, headdim) or D.shape == (nheads,)
+ if z is not None:
+ assert z.shape == x.shape
+ assert dt.shape == (nheads, nchunks, chunk_size)
+ assert dA_cumsum.shape == (nheads, nchunks, chunk_size)
+ assert states.shape == (nchunks, nheads, headdim, dstate)
+ assert seq_idx.shape == (nchunks,)
+
+ grid = lambda META: (
+ triton.cdiv(chunk_size, META["BLOCK_SIZE_M"]) * triton.cdiv(headdim, META["BLOCK_SIZE_N"]),
+ nchunks,
+ nheads,
+ )
+
+ z_strides = (z.stride(0), z.stride(1), z.stride(2)) if z is not None else (0, 0, 0)
+ initial_states_strides = (
+ (
+ initial_states.stride(0),
+ initial_states.stride(1),
+ initial_states.stride(2),
+ initial_states.stride(3),
+ )
+ if initial_states is not None
+ else (0, 0, 0, 0)
+ )
+
+ _chunk_scan_fwd_kernel[grid](
+ cb_ptr=cb,
+ x_ptr=x,
+ z_ptr=z,
+ out_ptr=out,
+ dt_ptr=dt,
+ dA_cumsum_ptr=dA_cumsum,
+ seq_idx_ptr=seq_idx,
+ C_ptr=C,
+ states_ptr=states,
+ D_ptr=D,
+ initstates_ptr=initial_states,
+ cu_chunk_seqlens_ptr=cu_chunk_seqlens,
+ chunk_size=chunk_size,
+ hdim=headdim,
+ dstate=dstate,
+ seqlen=seqlen,
+ nheads_ngroups_ratio=nheads // ngroups,
+ stride_cb_chunk=cb.stride(0),
+ stride_cb_head=cb.stride(1),
+ stride_cb_csize_m=cb.stride(2),
+ stride_cb_csize_k=cb.stride(3),
+ stride_x_seqlen=x.stride(0),
+ stride_x_head=x.stride(1),
+ stride_x_hdim=x.stride(2),
+ stride_z_seqlen=z_strides[0],
+ stride_z_head=z_strides[1],
+ stride_z_hdim=z_strides[2],
+ stride_out_seqlen=out.stride(0),
+ stride_out_head=out.stride(1),
+ stride_out_hdim=out.stride(2),
+ stride_dt_chunk=dt.stride(1),
+ stride_dt_head=dt.stride(0),
+ stride_dt_csize=dt.stride(2),
+ stride_dA_cs_chunk=dA_cumsum.stride(1),
+ stride_dA_cs_head=dA_cumsum.stride(0),
+ stride_dA_cs_csize=dA_cumsum.stride(2),
+ stride_seq_idx_chunk=seq_idx.stride(0),
+ stride_C_seqlen=C.stride(0),
+ stride_C_head=C.stride(1),
+ stride_C_dstate=C.stride(2),
+ stride_states_chunk=states.stride(0),
+ stride_states_head=states.stride(1),
+ stride_states_hdim=states.stride(2),
+ stride_states_dstate=states.stride(3),
+ stride_init_states_batch=initial_states_strides[0],
+ stride_init_states_head=initial_states_strides[1],
+ stride_init_states_hdim=initial_states_strides[2],
+ stride_init_states_dstate=initial_states_strides[3],
+ stride_D_head=D.stride(0) if D is not None else 0,
+ IS_CAUSAL=True,
+ HAS_D=D is not None,
+ D_HAS_HDIM=D.dim() == 2 if D is not None else True,
+ HAS_Z=z is not None,
+ BLOCK_SIZE_DSTATE=max(triton.next_power_of_2(dstate), 16),
+ IS_TRITON_22=TRITON_22,
+ HAS_INITSTATES=initial_states is not None,
+ )
+ return
diff --git a/megatron/core/ssm/ops/ssd_chunk_state.py b/megatron/core/ssm/ops/ssd_chunk_state.py
new file mode 100644
index 00000000000..473af1491aa
--- /dev/null
+++ b/megatron/core/ssm/ops/ssd_chunk_state.py
@@ -0,0 +1,679 @@
+# Copyright (c) 2024, Tri Dao, Albert Gu.
+# Adapted from:
+# https://github.com/state-spaces/mamba/blob/v2.2.4/mamba_ssm/ops/triton/ssd_chunk_state.py
+# Adapted from vLLM project (Apache-2.0).
+
+import torch
+import triton
+import triton.language as tl
+from packaging import version
+
+from megatron.core.ssm.ops.determinism import autotune_configs
+
+try:
+ TRITON3 = version.parse(triton.__version__) >= version.parse("3.0.0")
+except:
+ raise ImportError("Triton version 3.0.0 or higher is required")
+
+if TRITON3:
+
+ @triton.jit
+ def softplus(dt): # pylint: disable=C0116
+ dt = tl.where(dt <= 20.0, tl.math.log(tl.math.exp(dt) + 1), dt)
+ return dt
+
+else:
+
+ @triton.jit
+ def softplus(dt): # pylint: disable=C0116
+ dt = tl.where(dt <= 20.0, tl.math.log1p(tl.exp(dt)), dt)
+ return dt
+
+
+@triton.autotune(
+ configs=autotune_configs(
+ [
+ triton.Config({"BLOCK_SIZE_H": 2}),
+ triton.Config({"BLOCK_SIZE_H": 4}),
+ triton.Config({"BLOCK_SIZE_H": 8}),
+ triton.Config({"BLOCK_SIZE_H": 16}),
+ triton.Config({"BLOCK_SIZE_H": 32}),
+ triton.Config({"BLOCK_SIZE_H": 64}),
+ ]
+ ),
+ key=["chunk_size", "nheads"],
+)
+@triton.jit
+def _chunk_cumsum_fwd_kernel(
+ # Pointers to matrices
+ dt_ptr,
+ A_ptr,
+ dt_bias_ptr,
+ dt_out_ptr,
+ dA_cumsum_ptr,
+ cu_chunk_seqlens_ptr,
+ # Matrix dimension
+ seqlen,
+ nheads: tl.constexpr,
+ chunk_size: tl.constexpr,
+ dt_min: tl.constexpr,
+ dt_max: tl.constexpr,
+ # Strides
+ stride_dt_seqlen: tl.int64,
+ stride_dt_head: tl.constexpr,
+ stride_A_head: tl.constexpr,
+ stride_dt_bias_head: tl.constexpr,
+ stride_dt_out_head: tl.int64,
+ stride_dt_out_chunk: tl.int64,
+ stride_dt_out_csize: tl.constexpr,
+ stride_dA_cs_head: tl.int64,
+ stride_dA_cs_chunk: tl.int64,
+ stride_dA_cs_csize: tl.constexpr,
+ # Meta-parameters
+ DT_SOFTPLUS: tl.constexpr,
+ HAS_DT_BIAS: tl.constexpr,
+ BLOCK_SIZE_H: tl.constexpr,
+ BLOCK_SIZE_CHUNK: tl.constexpr,
+):
+ # if dt is long, may cause problems, so use 64 bit
+ # https://github.com/triton-lang/triton/issues/1058
+ pid_c = tl.program_id(axis=0).to(tl.int64)
+ pid_h = tl.program_id(axis=1)
+
+ chunk_seqlen_start = tl.load(cu_chunk_seqlens_ptr + pid_c)
+ chunk_seqlen_end = tl.load(cu_chunk_seqlens_ptr + pid_c + 1)
+
+ dt_ptr += chunk_seqlen_start * stride_dt_seqlen
+ dt_out_ptr += pid_c * stride_dt_out_chunk
+ dA_cumsum_ptr += pid_c * stride_dA_cs_chunk
+
+ offs_h = pid_h * BLOCK_SIZE_H + tl.arange(0, BLOCK_SIZE_H)
+ offs_c = tl.arange(0, BLOCK_SIZE_CHUNK)
+ dt_ptrs = dt_ptr + (offs_h[:, None] * stride_dt_head + offs_c[None, :] * stride_dt_seqlen)
+ A_ptrs = A_ptr + offs_h * stride_A_head
+ dt_out_ptrs = dt_out_ptr + (
+ offs_h[:, None] * stride_dt_out_head + offs_c[None, :] * stride_dt_out_csize
+ )
+ dA_cs_ptrs = dA_cumsum_ptr + (
+ offs_h[:, None] * stride_dA_cs_head + offs_c[None, :] * stride_dA_cs_csize
+ )
+ chunk_size_limit = chunk_seqlen_end - chunk_seqlen_start
+
+ dt = tl.load(
+ dt_ptrs, mask=(offs_h[:, None] < nheads) & (offs_c[None, :] < chunk_size_limit), other=0.0
+ ).to(tl.float32)
+ if HAS_DT_BIAS:
+ dt_bias = tl.load(
+ dt_bias_ptr + offs_h * stride_dt_bias_head, mask=offs_h < nheads, other=0.0
+ ).to(tl.float32)
+ dt += dt_bias[:, None]
+ if DT_SOFTPLUS:
+ dt = tl.where(dt <= 20.0, softplus(dt), dt)
+
+ dt = tl.clamp(dt, dt_min, dt_max)
+ dt = tl.where((offs_h[:, None] < nheads) & (offs_c[None, :] < chunk_size_limit), dt, 0.0)
+ tl.store(dt_out_ptrs, dt, mask=(offs_h[:, None] < nheads) & (offs_c[None, :] < chunk_size))
+ A = tl.load(A_ptrs, mask=offs_h < nheads, other=0.0).to(tl.float32)
+ dA = dt * A[:, None]
+ dA_cs = tl.cumsum(dA, axis=1)
+ tl.store(dA_cs_ptrs, dA_cs, mask=(offs_h[:, None] < nheads) & (offs_c[None, :] < chunk_size))
+
+
+@triton.autotune(
+ configs=autotune_configs(
+ [
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 256, "BLOCK_SIZE_K": 64},
+ num_stages=3,
+ num_warps=8,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 256, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 128, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 64, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 128, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 32, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 32, "BLOCK_SIZE_K": 32},
+ num_stages=5,
+ num_warps=2,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 32, "BLOCK_SIZE_N": 64, "BLOCK_SIZE_K": 32},
+ num_stages=5,
+ num_warps=2,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 64, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=2,
+ ),
+ ]
+ ),
+ key=["hdim", "dstate", "chunk_size"],
+)
+@triton.jit
+def _chunk_state_fwd_kernel(
+ # Pointers to matrices
+ x_ptr,
+ b_ptr,
+ states_ptr,
+ dt_ptr,
+ dA_cumsum_ptr,
+ cu_chunk_seqlens_ptr,
+ # Matrix dimensions
+ hdim: tl.constexpr,
+ dstate: tl.constexpr,
+ chunk_size: tl.constexpr,
+ seqlen,
+ nheads_ngroups_ratio: tl.constexpr,
+ # Strides
+ stride_x_seqlen: tl.int64,
+ stride_x_head: tl.int64,
+ stride_x_hdim: tl.constexpr,
+ stride_b_seqlen: tl.int64,
+ stride_b_head: tl.int64,
+ stride_b_dstate: tl.constexpr,
+ stride_states_chunk: tl.int64,
+ stride_states_head: tl.int64,
+ stride_states_hdim: tl.int64,
+ stride_states_dstate: tl.constexpr,
+ stride_dt_head: tl.int64,
+ stride_dt_chunk: tl.int64,
+ stride_dt_csize: tl.constexpr,
+ stride_dA_cs_head: tl.int64,
+ stride_dA_cs_chunk: tl.int64,
+ stride_dA_cs_csize: tl.constexpr,
+ # Meta-parameters
+ BLOCK_SIZE_M: tl.constexpr,
+ BLOCK_SIZE_N: tl.constexpr,
+ BLOCK_SIZE_K: tl.constexpr,
+):
+ pid_c = tl.program_id(axis=1).to(tl.int64)
+ pid_h = tl.program_id(axis=2)
+ num_pid_n = tl.cdiv(dstate, BLOCK_SIZE_N)
+ pid_m = tl.program_id(axis=0) // num_pid_n
+ pid_n = tl.program_id(axis=0) % num_pid_n
+ chunk_seqlen_start = tl.load(cu_chunk_seqlens_ptr + pid_c)
+ chunk_seqlen_end = tl.load(cu_chunk_seqlens_ptr + pid_c + 1)
+ b_ptr += chunk_seqlen_start * stride_b_seqlen + (pid_h // nheads_ngroups_ratio) * stride_b_head
+ x_ptr += chunk_seqlen_start * stride_x_seqlen + pid_h * stride_x_head
+ dt_ptr += pid_c * stride_dt_chunk + pid_h * stride_dt_head
+ dA_cumsum_ptr += pid_c * stride_dA_cs_chunk + pid_h * stride_dA_cs_head
+
+ offs_m = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+ offs_n = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+ offs_k = tl.arange(0, BLOCK_SIZE_K)
+ x_ptrs = x_ptr + (offs_m[:, None] * stride_x_hdim + offs_k[None, :] * stride_x_seqlen)
+ b_ptrs = b_ptr + (offs_n[None, :] * stride_b_dstate + offs_k[:, None] * stride_b_seqlen)
+ dt_ptrs = dt_ptr + offs_k * stride_dt_csize
+ dA_cs_last = tl.load(dA_cumsum_ptr + (chunk_size - 1) * stride_dA_cs_csize).to(tl.float32)
+ dA_cumsum_ptrs = dA_cumsum_ptr + offs_k * stride_dA_cs_csize
+
+ chunk_size_limit = chunk_seqlen_end - chunk_seqlen_start
+
+ acc = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
+ for k in range(0, chunk_size_limit, BLOCK_SIZE_K):
+ x = tl.load(
+ x_ptrs,
+ mask=(offs_m[:, None] < hdim) & (offs_k[None, :] < chunk_size_limit - k),
+ other=0.0,
+ )
+ b = tl.load(
+ b_ptrs,
+ mask=(offs_k[:, None] < chunk_size_limit - k) & (offs_n[None, :] < dstate),
+ other=0.0,
+ ).to(tl.float32)
+ dA_cs_k = tl.load(dA_cumsum_ptrs, mask=offs_k < chunk_size_limit - k, other=0.0).to(
+ tl.float32
+ )
+ dt_k = tl.load(dt_ptrs, mask=offs_k < chunk_size_limit - k, other=0.0).to(tl.float32)
+ scale = tl.exp(tl.minimum(dA_cs_last - dA_cs_k, 0.0)) * dt_k
+ b *= scale[:, None]
+ b = b.to(x_ptr.dtype.element_ty)
+ acc += tl.dot(x, b)
+
+ x_ptrs += BLOCK_SIZE_K * stride_x_seqlen
+ b_ptrs += BLOCK_SIZE_K * stride_b_seqlen
+ dt_ptrs += BLOCK_SIZE_K * stride_dt_csize
+ dA_cumsum_ptrs += BLOCK_SIZE_K * stride_dA_cs_csize
+
+ states = acc.to(states_ptr.dtype.element_ty)
+
+ states_ptr += pid_c * stride_states_chunk + pid_h * stride_states_head
+ offs_m = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+ offs_n = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+ states_ptrs = states_ptr + (
+ offs_m[:, None] * stride_states_hdim + offs_n[None, :] * stride_states_dstate
+ )
+ c_mask = (offs_m[:, None] < hdim) & (offs_n[None, :] < dstate)
+ tl.store(states_ptrs, states, mask=c_mask)
+
+
+def _chunk_cumsum_fwd(
+ dt,
+ A,
+ chunk_size,
+ cu_chunk_seqlens,
+ dt_bias=None,
+ dt_softplus=False,
+ dt_limit=(0.0, float("inf")),
+):
+ seqlen, nheads = dt.shape
+ assert A.shape == (nheads,)
+ if dt_bias is not None:
+ assert dt_bias.shape == (nheads,)
+ nchunks = cu_chunk_seqlens.shape[0] - 1
+ dt_out = torch.empty(nheads, nchunks, chunk_size, device=dt.device, dtype=torch.float32)
+ dA_cumsum = torch.empty(nheads, nchunks, chunk_size, device=dt.device, dtype=torch.float32)
+ grid_chunk_cs = lambda META: (nchunks, triton.cdiv(nheads, META["BLOCK_SIZE_H"]))
+ with torch.cuda.device(dt.device.index):
+ _chunk_cumsum_fwd_kernel[grid_chunk_cs](
+ dt_ptr=dt,
+ A_ptr=A,
+ dt_bias_ptr=dt_bias,
+ dt_out_ptr=dt_out,
+ dA_cumsum_ptr=dA_cumsum,
+ cu_chunk_seqlens_ptr=cu_chunk_seqlens,
+ seqlen=seqlen,
+ nheads=nheads,
+ chunk_size=chunk_size,
+ dt_min=dt_limit[0],
+ dt_max=dt_limit[1],
+ stride_dt_seqlen=dt.stride(0),
+ stride_dt_head=dt.stride(1),
+ stride_A_head=A.stride(0),
+ stride_dt_bias_head=dt_bias.stride(0) if dt_bias is not None else 0,
+ stride_dt_out_head=dt_out.stride(0),
+ stride_dt_out_chunk=dt_out.stride(1),
+ stride_dt_out_csize=dt_out.stride(2),
+ stride_dA_cs_head=dA_cumsum.stride(0),
+ stride_dA_cs_chunk=dA_cumsum.stride(1),
+ stride_dA_cs_csize=dA_cumsum.stride(2),
+ DT_SOFTPLUS=dt_softplus,
+ HAS_DT_BIAS=dt_bias is not None,
+ BLOCK_SIZE_CHUNK=triton.next_power_of_2(chunk_size),
+ )
+ return dA_cumsum, dt_out
+
+
+def _chunk_state_fwd(B, x, dt, dA_cumsum, cu_chunk_seqlens, states=None, states_in_fp32=True):
+ seqlen, nheads, headdim = x.shape
+ _, nchunks, chunk_size = dt.shape
+ _, ngroups, dstate = B.shape
+ assert nheads % ngroups == 0
+ assert B.shape == (seqlen, ngroups, dstate)
+ assert dt.shape == (nheads, nchunks, chunk_size)
+ assert dA_cumsum.shape == dt.shape
+
+ if states is not None:
+ assert states.shape == (nchunks, nheads, headdim, dstate)
+ else:
+ states_dtype = torch.float32 if states_in_fp32 else B.dtype
+ states = torch.empty(
+ (nchunks, nheads, headdim, dstate), device=x.device, dtype=states_dtype
+ )
+
+ grid = lambda META: (
+ triton.cdiv(headdim, META["BLOCK_SIZE_M"]) * triton.cdiv(dstate, META["BLOCK_SIZE_N"]),
+ nchunks,
+ nheads,
+ )
+ with torch.cuda.device(x.device.index):
+ _chunk_state_fwd_kernel[grid](
+ x_ptr=x,
+ b_ptr=B,
+ states_ptr=states,
+ dt_ptr=dt,
+ dA_cumsum_ptr=dA_cumsum,
+ cu_chunk_seqlens_ptr=cu_chunk_seqlens,
+ hdim=headdim,
+ dstate=dstate,
+ chunk_size=chunk_size,
+ seqlen=seqlen,
+ nheads_ngroups_ratio=nheads // ngroups,
+ stride_x_seqlen=x.stride(0),
+ stride_x_head=x.stride(1),
+ stride_x_hdim=x.stride(2),
+ stride_b_seqlen=B.stride(0),
+ stride_b_head=B.stride(1),
+ stride_b_dstate=B.stride(2),
+ stride_states_chunk=states.stride(0),
+ stride_states_head=states.stride(1),
+ stride_states_hdim=states.stride(2),
+ stride_states_dstate=states.stride(3),
+ stride_dt_head=dt.stride(0),
+ stride_dt_chunk=dt.stride(1),
+ stride_dt_csize=dt.stride(2),
+ stride_dA_cs_head=dA_cumsum.stride(0),
+ stride_dA_cs_chunk=dA_cumsum.stride(1),
+ stride_dA_cs_csize=dA_cumsum.stride(2),
+ )
+ return states
+
+
+@triton.autotune(
+ configs=autotune_configs(
+ [
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 256, "BLOCK_SIZE_K": 64},
+ num_stages=3,
+ num_warps=8,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 256, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 128, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 64, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 128, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 128, "BLOCK_SIZE_N": 32, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=4,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 32, "BLOCK_SIZE_K": 32},
+ num_stages=5,
+ num_warps=2,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 32, "BLOCK_SIZE_N": 64, "BLOCK_SIZE_K": 32},
+ num_stages=5,
+ num_warps=2,
+ ),
+ triton.Config(
+ {"BLOCK_SIZE_M": 64, "BLOCK_SIZE_N": 64, "BLOCK_SIZE_K": 32},
+ num_stages=4,
+ num_warps=2,
+ ),
+ ]
+ ),
+ key=["hdim", "dstate", "chunk_size"],
+)
+@triton.jit
+def _chunk_state_varlen_kernel(
+ x_ptr,
+ b_ptr,
+ dt_ptr,
+ dA_cumsum_ptr,
+ chunk_states_ptr,
+ cu_seqlens_ptr,
+ last_chunk_indices_ptr,
+ cu_chunk_seqlens_ptr,
+ states_ptr,
+ initstates_ptr,
+ hdim: tl.constexpr,
+ dstate: tl.constexpr,
+ chunk_size: tl.constexpr,
+ nheads_ngroups_ratio: tl.constexpr,
+ stride_x_seqlen: tl.int64,
+ stride_x_head: tl.int64,
+ stride_x_hdim: tl.constexpr,
+ stride_b_seqlen: tl.int64,
+ stride_b_head: tl.int64,
+ stride_b_dstate: tl.constexpr,
+ stride_dt_head: tl.int64,
+ stride_dt_chunk: tl.int64,
+ stride_dt_csize: tl.constexpr,
+ stride_dA_cs_head: tl.int64,
+ stride_dA_cs_chunk: tl.int64,
+ stride_dA_cs_csize: tl.constexpr,
+ stride_chunk_states_chunk: tl.int64,
+ stride_chunk_states_head: tl.int64,
+ stride_chunk_states_hdim: tl.int64,
+ stride_chunk_states_dstate: tl.constexpr,
+ stride_states_batch: tl.int64,
+ stride_states_head: tl.int64,
+ stride_states_hdim: tl.int64,
+ stride_states_dstate: tl.constexpr,
+ stride_init_states_batch: tl.int64,
+ stride_init_states_head: tl.int64,
+ stride_init_states_hdim: tl.int64,
+ stride_init_states_dstate: tl.constexpr,
+ BLOCK_SIZE_M: tl.constexpr,
+ BLOCK_SIZE_N: tl.constexpr,
+ BLOCK_SIZE_K: tl.constexpr,
+ HAS_INITSTATES: tl.constexpr,
+ USE_LAST_CHUNK_INDICES: tl.constexpr,
+):
+ pid_b = tl.program_id(axis=1)
+ pid_h = tl.program_id(axis=2)
+ num_pid_n = tl.cdiv(dstate, BLOCK_SIZE_N)
+ pid_m = tl.program_id(axis=0) // num_pid_n
+ pid_n = tl.program_id(axis=0) % num_pid_n
+ end_idx = tl.load(cu_seqlens_ptr + pid_b + 1)
+ start_idx = tl.load(cu_seqlens_ptr + pid_b)
+ if USE_LAST_CHUNK_INDICES:
+ pid_c = tl.load(last_chunk_indices_ptr + pid_b).to(tl.int64)
+ chunk_start = tl.load(cu_chunk_seqlens_ptr + pid_c)
+ chunk_size_limit = tl.load(cu_chunk_seqlens_ptr + pid_c + 1) - chunk_start
+ else:
+ pid_c = (end_idx - 1) // chunk_size
+ chunk_start = pid_c * chunk_size
+ chunk_size_limit = end_idx - chunk_start
+ b_ptr += chunk_start * stride_b_seqlen + (pid_h // nheads_ngroups_ratio) * stride_b_head
+ x_ptr += chunk_start * stride_x_seqlen + pid_h * stride_x_head
+ dt_ptr += pid_c * stride_dt_chunk + pid_h * stride_dt_head
+ dA_cumsum_ptr += pid_c * stride_dA_cs_chunk + pid_h * stride_dA_cs_head
+ chunk_states_ptr += pid_c * stride_chunk_states_chunk + pid_h * stride_chunk_states_head
+
+ if HAS_INITSTATES:
+ initstates_ptr += pid_h * stride_init_states_head
+
+ offs_m = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+ offs_n = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+ offs_k = tl.arange(0, BLOCK_SIZE_K)
+ x_ptrs = x_ptr + (offs_m[:, None] * stride_x_hdim + offs_k[None, :] * stride_x_seqlen)
+ b_ptrs = b_ptr + (offs_n[None, :] * stride_b_dstate + offs_k[:, None] * stride_b_seqlen)
+ dt_ptrs = dt_ptr + offs_k * stride_dt_csize
+ dA_cs_last = tl.load(dA_cumsum_ptr + (end_idx - 1 - chunk_start) * stride_dA_cs_csize).to(
+ tl.float32
+ )
+ dA_cumsum_ptrs = dA_cumsum_ptr + offs_k * stride_dA_cs_csize
+
+ start_idx_cur = tl.maximum(start_idx - chunk_start, 0)
+
+ acc = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
+ for k in range(0, chunk_size_limit, BLOCK_SIZE_K):
+ x = tl.load(
+ x_ptrs,
+ mask=(offs_m[:, None] < hdim)
+ & (offs_k[None, :] < chunk_size_limit - k)
+ & (offs_k[None, :] >= start_idx_cur - k),
+ other=0.0,
+ )
+ b = tl.load(
+ b_ptrs,
+ mask=(offs_k[:, None] < chunk_size_limit - k)
+ & (offs_n[None, :] < dstate)
+ & (offs_k[:, None] >= start_idx_cur - k),
+ other=0.0,
+ ).to(tl.float32)
+ dA_cs_k = tl.load(dA_cumsum_ptrs, mask=offs_k < chunk_size_limit - k, other=0.0).to(
+ tl.float32
+ )
+ dt_k = tl.load(dt_ptrs, mask=offs_k < chunk_size_limit - k, other=0.0).to(tl.float32)
+ scale = tl.where(
+ (offs_k >= start_idx_cur - k) & (offs_k < chunk_size_limit - k),
+ tl.exp(tl.minimum(dA_cs_last - dA_cs_k, 0.0)) * dt_k,
+ 0.0,
+ )
+ b *= scale[:, None]
+ b = b.to(x_ptr.dtype.element_ty)
+ acc += tl.dot(x, b)
+ x_ptrs += BLOCK_SIZE_K * stride_x_seqlen
+ b_ptrs += BLOCK_SIZE_K * stride_b_seqlen
+ dt_ptrs += BLOCK_SIZE_K * stride_dt_csize
+ dA_cumsum_ptrs += BLOCK_SIZE_K * stride_dA_cs_csize
+
+ if (start_idx < chunk_start) or (HAS_INITSTATES):
+ dA_cs_boundary = 0.0
+ if not HAS_INITSTATES:
+ past_states_ptrs = chunk_states_ptr + (
+ offs_m[:, None] * stride_chunk_states_hdim
+ + offs_n[None, :] * stride_chunk_states_dstate
+ )
+ else:
+ if start_idx < chunk_start:
+ past_states_ptrs = chunk_states_ptr + (
+ offs_m[:, None] * stride_chunk_states_hdim
+ + offs_n[None, :] * stride_chunk_states_dstate
+ )
+ else:
+ past_states_ptrs = initstates_ptr + (
+ pid_b * stride_init_states_batch
+ + offs_m[:, None] * stride_init_states_hdim
+ + offs_n[None, :] * stride_init_states_dstate
+ )
+ if start_idx > chunk_start:
+ dA_cs_boundary = tl.load(
+ dA_cumsum_ptr + (start_idx - chunk_start - 1) * stride_dA_cs_csize
+ ).to(tl.float32)
+
+ past_states = tl.load(
+ past_states_ptrs, mask=(offs_m[:, None] < hdim) & (offs_n[None, :] < dstate), other=0.0
+ ).to(tl.float32)
+ scale = tl.exp(tl.minimum(dA_cs_last - dA_cs_boundary, 0.0))
+ acc += past_states * scale
+
+ states = acc.to(states_ptr.dtype.element_ty)
+ states_ptr += pid_b * stride_states_batch + pid_h * stride_states_head
+ offs_m = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+ offs_n = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+ states_ptrs = states_ptr + (
+ offs_m[:, None] * stride_states_hdim + offs_n[None, :] * stride_states_dstate
+ )
+ c_mask = (offs_m[:, None] < hdim) & (offs_n[None, :] < dstate)
+ tl.store(states_ptrs, states, mask=c_mask)
+
+
+def chunk_state_varlen(
+ B,
+ x,
+ dt,
+ dA_cumsum,
+ cu_seqlens,
+ chunk_states,
+ initial_states=None,
+ last_chunk_indices=None,
+ cu_chunk_seqlens=None,
+):
+ """Compute per-sequence final SSM state from chunk states.
+
+ Correct when sequences share chunks.
+ """
+ total_seqlen, nheads, headdim = x.shape
+ _, nchunks, chunk_size = dt.shape
+ _, ngroups, dstate = B.shape
+ batch = cu_seqlens.shape[0] - 1
+ cu_seqlens = cu_seqlens.contiguous()
+ assert nheads % ngroups == 0
+ assert B.shape == (total_seqlen, ngroups, dstate)
+ assert dt.shape == (nheads, nchunks, chunk_size)
+ assert dA_cumsum.shape == dt.shape
+ assert chunk_states.shape == (nchunks, nheads, headdim, dstate)
+ if initial_states is not None:
+ assert initial_states.shape == (batch, nheads, headdim, dstate)
+ use_last_chunk = last_chunk_indices is not None and cu_chunk_seqlens is not None
+ if use_last_chunk:
+ last_chunk_indices = last_chunk_indices.contiguous().to(x.device)
+ cu_chunk_seqlens = cu_chunk_seqlens.contiguous().to(x.device)
+ else:
+ last_chunk_indices = torch.zeros(1, dtype=torch.int64, device=x.device)
+ cu_chunk_seqlens = cu_seqlens
+
+ states = torch.empty(
+ batch, nheads, headdim, dstate, dtype=chunk_states.dtype, device=chunk_states.device
+ )
+ initial_states_strides = (
+ (
+ initial_states.stride(0),
+ initial_states.stride(1),
+ initial_states.stride(2),
+ initial_states.stride(3),
+ )
+ if initial_states is not None
+ else (0, 0, 0, 0)
+ )
+ grid = lambda META: (
+ triton.cdiv(headdim, META["BLOCK_SIZE_M"]) * triton.cdiv(dstate, META["BLOCK_SIZE_N"]),
+ batch,
+ nheads,
+ )
+ with torch.cuda.device(x.device.index):
+ _chunk_state_varlen_kernel[grid](
+ x_ptr=x,
+ b_ptr=B,
+ dt_ptr=dt,
+ dA_cumsum_ptr=dA_cumsum,
+ chunk_states_ptr=chunk_states,
+ cu_seqlens_ptr=cu_seqlens,
+ last_chunk_indices_ptr=last_chunk_indices,
+ cu_chunk_seqlens_ptr=cu_chunk_seqlens,
+ states_ptr=states,
+ initstates_ptr=initial_states,
+ hdim=headdim,
+ dstate=dstate,
+ chunk_size=chunk_size,
+ nheads_ngroups_ratio=nheads // ngroups,
+ stride_x_seqlen=x.stride(0),
+ stride_x_head=x.stride(1),
+ stride_x_hdim=x.stride(2),
+ stride_b_seqlen=B.stride(0),
+ stride_b_head=B.stride(1),
+ stride_b_dstate=B.stride(2),
+ stride_dt_head=dt.stride(0),
+ stride_dt_chunk=dt.stride(1),
+ stride_dt_csize=dt.stride(2),
+ stride_dA_cs_head=dA_cumsum.stride(0),
+ stride_dA_cs_chunk=dA_cumsum.stride(1),
+ stride_dA_cs_csize=dA_cumsum.stride(2),
+ stride_chunk_states_chunk=chunk_states.stride(0),
+ stride_chunk_states_head=chunk_states.stride(1),
+ stride_chunk_states_hdim=chunk_states.stride(2),
+ stride_chunk_states_dstate=chunk_states.stride(3),
+ stride_states_batch=states.stride(0),
+ stride_states_head=states.stride(1),
+ stride_states_hdim=states.stride(2),
+ stride_states_dstate=states.stride(3),
+ stride_init_states_batch=initial_states_strides[0],
+ stride_init_states_head=initial_states_strides[1],
+ stride_init_states_hdim=initial_states_strides[2],
+ stride_init_states_dstate=initial_states_strides[3],
+ HAS_INITSTATES=initial_states is not None,
+ USE_LAST_CHUNK_INDICES=use_last_chunk,
+ )
+ return states
diff --git a/megatron/core/ssm/ops/ssd_combined.py b/megatron/core/ssm/ops/ssd_combined.py
new file mode 100644
index 00000000000..4fcee98b13e
--- /dev/null
+++ b/megatron/core/ssm/ops/ssd_combined.py
@@ -0,0 +1,234 @@
+# Copyright (c) 2024, Tri Dao, Albert Gu.
+# Adapted from:
+# https://github.com/state-spaces/mamba/blob/v2.2.4/mamba_ssm/ops/triton/ssd_combined.py
+# Adapted from vLLM project (Apache-2.0).
+
+import torch
+import triton
+from packaging import version
+
+from .ssd_bmm import _bmm_chunk_fwd
+from .ssd_chunk_scan import _chunk_scan_fwd
+from .ssd_chunk_state import _chunk_cumsum_fwd, _chunk_state_fwd
+from .ssd_state_passing import _state_passing_fwd
+
+TRITON_22 = version.parse(triton.__version__) >= version.parse("2.2.0")
+
+
+def is_int_pow_2(n):
+ """Return True if n is a positive integer power of 2."""
+ return isinstance(n, int) and n > 0 and (n & (n - 1)) == 0
+
+
+def _mamba_chunk_scan_combined_fwd(
+ x,
+ dt,
+ A,
+ B,
+ C,
+ chunk_size,
+ out,
+ D=None,
+ z=None,
+ dt_bias=None,
+ initial_states=None,
+ return_intermediate_states=False,
+ seq_idx=None,
+ cu_chunk_seqlens=None,
+ last_chunk_indices=None,
+ intermediate_chunk_indices=None,
+ dt_softplus=False,
+ dt_limit=(0.0, float("inf")),
+ state_dtype=None,
+):
+ assert is_int_pow_2(chunk_size), "chunk_size must be integer power of 2"
+ seqlen, nheads, headdim = x.shape
+ _, ngroups, dstate = B.shape
+ assert nheads % ngroups == 0
+ assert B.shape == (
+ seqlen,
+ ngroups,
+ dstate,
+ ), f"B.shape={B.shape} != ({seqlen}, {ngroups}, {dstate})"
+ assert dt.shape == (seqlen, nheads)
+ assert A.shape == (nheads,)
+ assert C.shape == B.shape
+ if z is not None:
+ assert z.shape == x.shape
+ if D is not None:
+ assert D.shape == (nheads, headdim) or D.shape == (nheads,)
+ if seq_idx is not None:
+ assert seq_idx.shape == (cu_chunk_seqlens.shape[0] - 1,)
+ if B.stride(-1) != 1:
+ B = B.contiguous()
+ if C.stride(-1) != 1:
+ C = C.contiguous()
+ if x.stride(-1) != 1 and x.stride(0) != 1: # Either M or K dimension should be contiguous
+ x = x.contiguous()
+ if (
+ z is not None and z.stride(-1) != 1 and z.stride(0) != 1
+ ): # Either M or K dimension should be contiguous
+ z = z.contiguous()
+ if D is not None and D.stride(-1) != 1:
+ D = D.contiguous()
+ assert cu_chunk_seqlens is not None, "Assuming varlen input - must supply cu_chunk_seqlens"
+ assert last_chunk_indices is not None, "last_chunk_indices must be provided"
+
+ if initial_states is not None:
+ num_seqs = last_chunk_indices.shape[0]
+ assert initial_states.shape == (num_seqs, nheads, headdim, dstate)
+
+ # This function executes 5 sub-functions for computing mamba
+ # - a good resource is the blog https://goombalab.github.io/blog/2024/mamba2-part3-algorithm/
+ # which has a minimal implementation to understand the below operations
+ # - as explained by the blog, mamba is a special case of causal attention
+ # - the idea is to chunk the attention matrix and compute each
+ # submatrix separately using different optimizations.
+ # - see the blog and paper for a visualization of the submatrices
+ # which we refer to in the comments below
+
+ # 1. Compute chunked cumsum of A * dt
+ # - here dt may go through a softplus activation
+ dA_cumsum, dt = _chunk_cumsum_fwd(
+ dt,
+ A,
+ chunk_size,
+ cu_chunk_seqlens,
+ dt_bias=dt_bias,
+ dt_softplus=dt_softplus,
+ dt_limit=dt_limit,
+ )
+
+ # 2. Compute the state for each intra-chunk
+ # (right term of low-rank factorization of off-diagonal blocks; B terms)
+ states = _chunk_state_fwd(B, x, dt, dA_cumsum, cu_chunk_seqlens, states_in_fp32=True)
+
+ # 3. Compute the inter-chunk SSM recurrence; produces correct SSM states at chunk boundaries
+ # (middle term of factorization of off-diag blocks; A terms)
+ # - for handling chunked prefill, this requires i) initial_states and
+ # ii) seq_idx to be all specified.
+ # - When a new seq_idx is detected, we will stop passing the prev_state
+ # and switch accordingly to the init_state corresponding to the new seq_idx.
+ states = _state_passing_fwd(
+ states.flatten(-2), # ... p n -> ... (p n)
+ dA_cumsum, # (nheads, nchunks, chunk_size)
+ cu_chunk_seqlens,
+ initial_states=(
+ initial_states.flatten(-2) if initial_states is not None else None
+ ), # (batch, nheads, headdim*dstate)
+ seq_idx=seq_idx,
+ out_dtype=state_dtype if state_dtype is not None else C.dtype,
+ )
+ states = states.unflatten(-1, (-1, dstate))
+
+ # 4. Compute batched matrix multiply for C_j^T B_i terms
+ CB = _bmm_chunk_fwd(C, B, chunk_size, cu_chunk_seqlens, output_dtype=torch.float32)
+
+ # 5. Scan and compute the diagonal blocks, taking into
+ # account past causal states.
+ # - if initial states are provided, then states information will be
+ # augmented with initial_states.
+ # - to do this properly, we need to account for example changes in
+ # the continuous batch, therefore we introduce pseudo chunks, which is
+ # a chunk that is split up each time an example changes.
+ # - in each (pseudo) chunk, we detect if the previous (pseudo) chunk had
+ # a seq_idx change, in which case we take states information from
+ # init_states.
+ _chunk_scan_fwd(
+ CB,
+ x,
+ dt,
+ dA_cumsum,
+ C,
+ states,
+ cu_chunk_seqlens,
+ out, # in-place update
+ seq_idx,
+ D=D,
+ z=z,
+ initial_states=initial_states,
+ )
+
+ if return_intermediate_states:
+ return states
+
+ final_states = states[last_chunk_indices]
+ if intermediate_chunk_indices is not None:
+ intermediate_states = states[intermediate_chunk_indices]
+ return final_states, intermediate_states
+ else:
+ return final_states
+
+
+def mamba_chunk_scan_combined_varlen(
+ x,
+ dt,
+ A,
+ B,
+ C,
+ chunk_size,
+ cu_chunk_seqlens,
+ last_chunk_indices,
+ seq_idx,
+ out,
+ D=None,
+ z=None,
+ dt_bias=None,
+ initial_states=None,
+ dt_softplus=False,
+ dt_limit=(0.0, float("inf")),
+ return_intermediate_states=False,
+ intermediate_chunk_indices=None,
+ state_dtype=None,
+):
+ """
+ Argument:
+ x: (seqlen, nheads, headdim)
+ dt: (seqlen, nheads)
+ A: (nheads)
+ B: (seqlen, ngroups, dstate)
+ C: (seqlen, ngroups, dstate)
+ chunk_size: int
+ cu_chunk_seqlens: (nchunks + 1,)
+ last_chunk_indices: (batch,)
+ seq_idx: (nchunks,)
+ out: (seqlen, nheads, headdim) preallocated output tensor
+ D: (nheads, headdim) or (nheads,)
+ z: (seqlen, nheads, headdim)
+ dt_bias: (nheads,)
+ initial_states: (batch, nheads, headdim, dstate)
+ dt_softplus: Whether to apply softplus to dt
+ intermediate_chunk_indices: (N,) optional int64 tensor of chunk indices at which to
+ extract intermediate SSM states. When provided, returns (final_states,
+ intermediate_states) instead of just final_states.
+ state_dtype: The data type of the ssm state
+ Return:
+ varlen_states: (batch, nheads, headdim, dstate), or
+ (varlen_states, intermediate_states) if intermediate_chunk_indices is provided
+ """
+
+ assert seq_idx is not None
+
+ varlen_states = _mamba_chunk_scan_combined_fwd(
+ x,
+ dt,
+ A,
+ B,
+ C,
+ chunk_size,
+ out,
+ D=D,
+ z=z,
+ dt_bias=dt_bias,
+ initial_states=initial_states,
+ return_intermediate_states=return_intermediate_states,
+ seq_idx=seq_idx,
+ cu_chunk_seqlens=cu_chunk_seqlens,
+ last_chunk_indices=last_chunk_indices,
+ intermediate_chunk_indices=intermediate_chunk_indices,
+ dt_softplus=dt_softplus,
+ dt_limit=dt_limit,
+ state_dtype=state_dtype,
+ )
+
+ return varlen_states
diff --git a/megatron/core/ssm/ops/ssd_state_passing.py b/megatron/core/ssm/ops/ssd_state_passing.py
new file mode 100644
index 00000000000..65b81a0ec31
--- /dev/null
+++ b/megatron/core/ssm/ops/ssd_state_passing.py
@@ -0,0 +1,149 @@
+# Copyright (c) 2024, Tri Dao, Albert Gu.
+# Adapted from:
+# https://github.com/state-spaces/mamba/blob/v2.2.4/mamba_ssm/ops/triton/ssd_state_passing.py
+# Adapted from vLLM project (Apache-2.0).
+
+import torch
+import triton
+import triton.language as tl
+
+from megatron.core.ssm.ops.determinism import autotune_configs
+
+
+@triton.autotune(
+ configs=autotune_configs(
+ [
+ triton.Config({"BLOCK_SIZE": 64}),
+ triton.Config({"BLOCK_SIZE": 128}),
+ triton.Config({"BLOCK_SIZE": 256}),
+ triton.Config({"BLOCK_SIZE": 512}),
+ triton.Config({"BLOCK_SIZE": 1024}),
+ triton.Config({"BLOCK_SIZE": 2048}),
+ ]
+ ),
+ key=["dim"],
+)
+@triton.jit
+def _state_passing_fwd_kernel(
+ # Pointers to matrices
+ states_ptr,
+ out_ptr,
+ dA_cs_ptr,
+ initstates_ptr,
+ seq_idx_ptr,
+ cu_chunk_seqlens_ptr,
+ # Matrix dimensions
+ dim: tl.constexpr,
+ nchunks,
+ seqlen,
+ chunk_size: tl.constexpr,
+ # Strides
+ stride_states_chunk: tl.int64,
+ stride_states_head: tl.int64,
+ stride_states_dim: tl.constexpr,
+ stride_out_chunk: tl.int64,
+ stride_out_head: tl.int64,
+ stride_out_dim: tl.constexpr,
+ stride_dA_cs_head: tl.int64,
+ stride_dA_cs_chunk: tl.int64,
+ stride_dA_cs_csize: tl.constexpr,
+ stride_initstates_batch: tl.int64,
+ stride_initstates_head: tl.int64,
+ stride_initstates_dim: tl.constexpr,
+ stride_seq_idx_chunk: tl.constexpr,
+ # Meta-parameters
+ HAS_INITSTATES: tl.constexpr,
+ BLOCK_SIZE: tl.constexpr,
+):
+ pid_h = tl.program_id(axis=1)
+ pid_m = tl.program_id(axis=0)
+
+ states_ptr += pid_h * stride_states_head
+ dA_cs_ptr += pid_h * stride_dA_cs_head + (chunk_size - 1) * stride_dA_cs_csize
+ out_ptr += pid_h * stride_out_head
+
+ offs_m = pid_m * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE)
+ states_ptrs = states_ptr + offs_m * stride_states_dim
+ out_ptrs = out_ptr + offs_m * stride_out_dim
+
+ if HAS_INITSTATES:
+ initstates_ptrs = (
+ initstates_ptr + pid_h * stride_initstates_head + offs_m * stride_initstates_dim
+ )
+
+ states = tl.load(initstates_ptrs, mask=offs_m < dim, other=0.0).to(tl.float32)
+ else:
+ states = tl.zeros((BLOCK_SIZE,), dtype=tl.float32)
+
+ prev_seq_idx = 0
+ for c in range(nchunks):
+ new_states = tl.load(states_ptrs, mask=offs_m < dim, other=0.0).to(tl.float32)
+ dA_cs = tl.load(dA_cs_ptr).to(tl.float32)
+ seq_idx = tl.load(seq_idx_ptr + c * stride_seq_idx_chunk)
+ # we have started a new sequence
+ if prev_seq_idx != seq_idx:
+ if HAS_INITSTATES:
+ initstates_ptrs = (
+ initstates_ptr
+ + seq_idx * stride_initstates_batch
+ + pid_h * stride_initstates_head
+ + offs_m * stride_initstates_dim
+ )
+ states = tl.load(initstates_ptrs, mask=offs_m < dim, other=0.0).to(tl.float32)
+ else:
+ states = tl.zeros((BLOCK_SIZE,), dtype=tl.float32)
+
+ prev_seq_idx = seq_idx
+ states = tl.exp(dA_cs) * states + new_states
+ tl.store(out_ptrs, states, mask=offs_m < dim)
+
+ states_ptrs += stride_states_chunk
+ dA_cs_ptr += stride_dA_cs_chunk
+ out_ptrs += stride_out_chunk
+
+
+def _state_passing_fwd(
+ states, dA_cumsum, cu_chunk_seqlens, seq_idx, initial_states=None, out_dtype=None
+):
+ nchunks, nheads, dim = states.shape
+ chunk_size = dA_cumsum.shape[-1]
+ assert dA_cumsum.shape == (nheads, nchunks, chunk_size)
+ seqlen = seq_idx.shape[-1]
+ out_dtype = states.dtype if out_dtype is None else out_dtype
+ out = torch.empty((nchunks, nheads, dim), device=states.device, dtype=out_dtype)
+
+ initial_states_strides = (
+ (initial_states.stride(0), initial_states.stride(1), initial_states.stride(2))
+ if initial_states is not None
+ else (0, 0, 0)
+ )
+
+ grid = lambda META: (triton.cdiv(dim, META["BLOCK_SIZE"]), nheads)
+ with torch.cuda.device(states.device.index):
+ _state_passing_fwd_kernel[grid](
+ states_ptr=states,
+ out_ptr=out,
+ dA_cs_ptr=dA_cumsum,
+ initstates_ptr=initial_states,
+ seq_idx_ptr=seq_idx,
+ cu_chunk_seqlens_ptr=cu_chunk_seqlens,
+ dim=dim,
+ nchunks=nchunks,
+ seqlen=seqlen if seq_idx is not None else 0,
+ chunk_size=chunk_size if seq_idx is not None else 0,
+ stride_states_chunk=states.stride(0),
+ stride_states_head=states.stride(1),
+ stride_states_dim=states.stride(2),
+ stride_out_chunk=out.stride(0),
+ stride_out_head=out.stride(1),
+ stride_out_dim=out.stride(2),
+ stride_dA_cs_head=dA_cumsum.stride(0),
+ stride_dA_cs_chunk=dA_cumsum.stride(1),
+ stride_dA_cs_csize=dA_cumsum.stride(2),
+ stride_initstates_batch=initial_states_strides[0],
+ stride_initstates_head=initial_states_strides[1],
+ stride_initstates_dim=initial_states_strides[2],
+ stride_seq_idx_chunk=seq_idx.stride(0),
+ HAS_INITSTATES=initial_states is not None,
+ )
+ return out
diff --git a/megatron/core/tensor_parallel/__init__.py b/megatron/core/tensor_parallel/__init__.py
index 2140ee54b37..0852014a859 100644
--- a/megatron/core/tensor_parallel/__init__.py
+++ b/megatron/core/tensor_parallel/__init__.py
@@ -1,7 +1,11 @@
# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
from .cross_entropy import vocab_parallel_cross_entropy
from .data import broadcast_data
-from .inference_layers import InferenceLayerNormColumnParallelLinear, InferenceRowParallelLinear
+from .inference_layers import (
+ InferenceColumnParallelLinear,
+ InferenceLayerNormColumnParallelLinear,
+ InferenceRowParallelLinear,
+)
from .layers import (
ColumnParallelLinear,
RowParallelLinear,
diff --git a/megatron/core/tensor_parallel/inference_layers.py b/megatron/core/tensor_parallel/inference_layers.py
index 9c1adbc6717..80aa754dd50 100644
--- a/megatron/core/tensor_parallel/inference_layers.py
+++ b/megatron/core/tensor_parallel/inference_layers.py
@@ -1,20 +1,25 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-from typing import Callable, Optional, Tuple
+from typing import Callable, Optional, Tuple, Union
import torch
import torch.distributed as dist
from megatron.core.extensions.transformer_engine import (
+ TEColumnParallelLinear,
TELayerNormColumnParallelLinear,
+ TELinear,
TERowParallelLinear,
)
from megatron.core.inference.communication.torch_symm_triton import (
+ are_tensors_nvls_eligible,
fused_multimem_rs_add_norm_ag,
multimem_all_gather,
multimem_reduce_scatter,
)
+from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+from megatron.core.inference.quantization.utils import mm_mxfp8
+from megatron.core.inference.symmetric_memory import SymmetricMemoryManager
from megatron.core.model_parallel_config import ModelParallelConfig
-from megatron.core.parallel_state import get_global_symmetric_memory_buffer
from megatron.core.transformer.transformer_config import TransformerConfig
from megatron.core.utils import get_tensor_model_parallel_group_if_none
@@ -41,6 +46,65 @@ def _te_rms_norm_kernel(x: torch.Tensor, weight: torch.Tensor, eps: float):
return out.to(x.dtype)
+def _apply_linear(
+ x: torch.Tensor,
+ weight: Union[torch.Tensor, MXFP8Tensor],
+ config: TransformerConfig,
+ out: Optional[torch.Tensor] = None,
+) -> torch.Tensor:
+ """
+ Helper to apply either MXFP8 or standard GEMM based on the configuration.
+ """
+ kwargs = {"out": out} if out is not None else {}
+ if isinstance(weight, MXFP8Tensor):
+ return mm_mxfp8(x, weight, **kwargs)
+ return torch.matmul(x, weight.t(), **kwargs)
+
+
+class InferenceLinear(TELinear):
+ """Inference optimized version of TELinear."""
+
+ def __init__(
+ self,
+ input_size: int,
+ output_size: int,
+ *,
+ parallel_mode: Optional[str],
+ config: ModelParallelConfig,
+ init_method: Callable,
+ bias: bool,
+ skip_bias_add: bool,
+ skip_weight_param_allocation: bool,
+ tp_comm_buffer_name: Optional[str] = None,
+ is_expert: bool = False,
+ symmetric_ar_type: Optional[str] = None,
+ tp_group: Optional[torch.distributed.ProcessGroup] = None,
+ ):
+ assert HAVE_TE, "--transformer-impl=inference_optimized requires transformer engine"
+ super().__init__(
+ input_size,
+ output_size,
+ parallel_mode=parallel_mode,
+ config=config,
+ init_method=init_method,
+ bias=bias,
+ skip_bias_add=skip_bias_add,
+ skip_weight_param_allocation=skip_weight_param_allocation,
+ tp_comm_buffer_name=tp_comm_buffer_name,
+ is_expert=is_expert,
+ symmetric_ar_type=symmetric_ar_type,
+ tp_group=tp_group,
+ )
+
+ def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, None]:
+ """Forward pass."""
+ if self.training:
+ return super().forward(x)
+
+ x = _apply_linear(x, self.weight, self.config)
+ return x, None
+
+
class InferenceLayerNormColumnParallelLinear(TELayerNormColumnParallelLinear):
"""
Inference optimized version of TELayerNormColumnParallelLinear.
@@ -91,6 +155,8 @@ def __init__(
config.sequence_parallel
), "--transformer-impl=inference_optimized requires --sequence-parallel"
+ self.triton_nvls_kernels_allowed = not config.inference_disable_triton_nvls_kernels
+
# Boolean to be toggled externally for skipping norm and all-gather.
# This is used when enabling fused reduce-scatter + add + rms-norm + all-gather
# in tensor parallelism. In this case, the preceeding RowParallelLinear layer
@@ -103,9 +169,8 @@ def _maybe_allocate_symmetric_buffer(self, x: torch.Tensor):
"""
symm_mem_buffer_dims = list(x.size())
symm_mem_buffer_dims[0] *= self.tp_size
- symm_mem_buffer = get_global_symmetric_memory_buffer().maybe_get_tensor(
- symm_mem_buffer_dims, dtype=x.dtype
- )
+ buf = SymmetricMemoryManager.get_buffer("tp", process_group=self.tp_group)
+ symm_mem_buffer = buf.maybe_get_tensor(symm_mem_buffer_dims, dtype=x.dtype)
return symm_mem_buffer
def _all_gather(self, x: torch.Tensor, symm_mem_buffer: dict) -> None:
@@ -116,16 +181,14 @@ def _all_gather(self, x: torch.Tensor, symm_mem_buffer: dict) -> None:
if self.tp_size == 1:
return x
- # 1. check if bf16
- is_bf16 = x.dtype == torch.bfloat16
- # 2. check if hopper or newer
- is_hopper_or_newer = torch.cuda.get_device_properties(x.device).major >= 9
- # 3. check if symmetric memory buffer is available
- has_enough_symmetric_memory = symm_mem_buffer["handle"] is not None
- can_use_custom_nvls_collectives = (
- is_bf16 and is_hopper_or_newer and has_enough_symmetric_memory
+ # Check input only: if input is 16-byte divisible, the output
+ # (world_size * input) is too.
+ can_use_nvls = (
+ self.triton_nvls_kernels_allowed
+ and are_tensors_nvls_eligible(x)
+ and symm_mem_buffer["handle"] is not None
)
- if can_use_custom_nvls_collectives:
+ if can_use_nvls:
# do multimem all gather
multimem_all_gather(symm_mem_buffer["tensor"], x, symm_mem_buffer["handle"])
return symm_mem_buffer["tensor"]
@@ -134,7 +197,6 @@ def _all_gather(self, x: torch.Tensor, symm_mem_buffer: dict) -> None:
x, _ = gather_along_first_dim(x, process_group=self.tp_group)
return x
- @torch.no_grad()
def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, None]:
"""
Forward pass.
@@ -144,6 +206,15 @@ def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, None]:
# 1. skip_norm_and_all_gather is True
# 2. tp_size > 1
# 3. enough symmetric memory is available - if available it already has the output
+
+ if self.training:
+ return super().forward(x)
+
+ if self.tp_size == 1:
+ x = _te_rms_norm_kernel(x=x, weight=self.layer_norm_weight, eps=self.eps)
+ x = _apply_linear(x, self.weight, self.config)
+ return x, None
+
symm_mem_buffer = self._maybe_allocate_symmetric_buffer(x)
is_in_fused_mode = (
self.skip_norm_and_all_gather
@@ -156,7 +227,105 @@ def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, None]:
x = _te_rms_norm_kernel(x=x, weight=self.layer_norm_weight, eps=self.eps)
x = self._all_gather(x, symm_mem_buffer)
- x = torch.matmul(x, self.weight.t())
+ x = _apply_linear(x, self.weight, self.config)
+
+ return x, None
+
+
+class InferenceColumnParallelLinear(TEColumnParallelLinear):
+ """
+ Inference optimized version of TEColumnParallelLinear.
+ """
+
+ def __init__(
+ self,
+ input_size: int,
+ output_size: int,
+ *,
+ config: TransformerConfig,
+ init_method: Callable,
+ gather_output: bool,
+ bias: bool,
+ skip_bias_add: bool,
+ is_expert: bool,
+ stride: int = 1,
+ skip_weight_param_allocation: bool = False,
+ tp_comm_buffer_name: Optional[str] = None,
+ tp_group: Optional[torch.distributed.ProcessGroup] = None,
+ ):
+ assert HAVE_TE, "--transformer-impl=inference_optimized requires transformer engine"
+ super().__init__(
+ input_size,
+ output_size,
+ config=config,
+ init_method=init_method,
+ gather_output=gather_output,
+ bias=bias,
+ skip_bias_add=skip_bias_add,
+ is_expert=is_expert,
+ stride=stride,
+ skip_weight_param_allocation=skip_weight_param_allocation,
+ tp_comm_buffer_name=tp_comm_buffer_name,
+ tp_group=tp_group,
+ )
+ self.tp_group = get_tensor_model_parallel_group_if_none(tp_group, is_expert=is_expert)
+ self.tp_size = dist.get_world_size(self.tp_group)
+
+ assert (
+ output_size % self.tp_size == 0
+ ), f"output_size ({output_size}) must be divisible by tp_size ({self.tp_size})"
+
+ if self.tp_size > 1:
+ assert (
+ config.sequence_parallel
+ ), "--transformer-impl=inference_optimized requires --sequence-parallel"
+
+ self.triton_nvls_kernels_allowed = not config.inference_disable_triton_nvls_kernels
+
+ def _maybe_allocate_symmetric_buffer(self, x: torch.Tensor):
+ """
+ Attempt to allocate symmetric memory buffer for all-gather.
+ """
+ symm_mem_buffer_dims = list(x.size())
+ symm_mem_buffer_dims[0] *= self.tp_size
+ buf = SymmetricMemoryManager.get_buffer("tp", process_group=self.tp_group)
+ symm_mem_buffer = buf.maybe_get_tensor(symm_mem_buffer_dims, dtype=x.dtype)
+ return symm_mem_buffer
+
+ def _all_gather(self, x: torch.Tensor, symm_mem_buffer: dict) -> None:
+ """
+ Attempt an NVLS all-gather into symmetric memory. If not possible,
+ revert to torch dist (NCCL) all-gather.
+ """
+ if self.tp_size == 1:
+ return x
+
+ can_use_nvls = (
+ self.triton_nvls_kernels_allowed
+ and are_tensors_nvls_eligible(x)
+ and symm_mem_buffer["handle"] is not None
+ )
+ if can_use_nvls:
+ multimem_all_gather(symm_mem_buffer["tensor"], x, symm_mem_buffer["handle"])
+ return symm_mem_buffer["tensor"]
+ else:
+ x, _ = gather_along_first_dim(x, process_group=self.tp_group)
+ return x
+
+ def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, None]:
+ """
+ Forward pass.
+ """
+ if self.training:
+ return super().forward(x)
+
+ if self.tp_size == 1:
+ x = _apply_linear(x, self.weight, self.config)
+ return x, None
+
+ symm_mem_buffer = self._maybe_allocate_symmetric_buffer(x)
+ x = self._all_gather(x, symm_mem_buffer)
+ x = _apply_linear(x, self.weight, self.config)
return x, None
@@ -204,6 +373,10 @@ def __init__(
config.sequence_parallel
), "--transformer-impl=inference_optimized requires --sequence-parallel"
+ self.triton_nvls_kernels_allowed = not getattr(
+ config, 'inference_disable_triton_nvls_kernels', False
+ )
+
# Placeholder for next layer norm weights for fused
# reduce-scatter + add + rms-norm + all-gather
self.next_layer_norm_weights = None
@@ -216,24 +389,30 @@ def _matmul_reduce_scatter(self, x, residual=None):
and perform an NVLS multicast reduce-scatter. If that is not possible,
it will revert to torch.dist (NCCL) reduce-scatter.
"""
- # 1. check if bf16
- is_bf16 = x.dtype == torch.bfloat16
- # 2. check if hopper
- is_hopper_or_newer = torch.cuda.get_device_properties(x.device).major >= 9
- # 3. attempt to ask for symmetric memory
+ use_mxfp8 = isinstance(self.weight, MXFP8Tensor)
symm_mem_buffer_dims = list(x.size())
+ if use_mxfp8:
+ # Remove seq_len dimension for MXFP8 (mm_mxfp8 squeezes internally)
+ del symm_mem_buffer_dims[1]
symm_mem_buffer_dims[-1] = self.weight.size(0)
- symm_mem_buffer = get_global_symmetric_memory_buffer().maybe_get_tensor(
- symm_mem_buffer_dims, dtype=x.dtype
- )
- has_enough_symmetric_memory = symm_mem_buffer["handle"] is not None
- can_use_custom_nvls_collectives = (
- is_bf16 and is_hopper_or_newer and has_enough_symmetric_memory
+ buf = SymmetricMemoryManager.get_buffer("tp", process_group=self.tp_group)
+ symm_mem_buffer = buf.maybe_get_tensor(symm_mem_buffer_dims, dtype=x.dtype)
+
+ # RS requires bf16 (hardware multimem reduce is bf16-only).
+ # Check the matmul output shape: if it is NVLS-eligible, the RS output
+ # (world_size times smaller on dim 0) is too.
+ can_use_nvls = (
+ self.triton_nvls_kernels_allowed
+ and x.dtype == torch.bfloat16
+ and are_tensors_nvls_eligible(x)
+ and symm_mem_buffer["handle"] is not None
)
- if can_use_custom_nvls_collectives:
+
+ if can_use_nvls:
# Write output of matmul directly onto the symmetric memory buffer
- torch.matmul(x, self.weight.t(), out=symm_mem_buffer["tensor"])
- x = symm_mem_buffer["tensor"]
+
+ x = _apply_linear(x, self.weight, self.config, out=symm_mem_buffer["tensor"])
+
# perform nvls reduce-scatter
if self.next_layer_norm_weights is None:
output_dims = list(x.size())
@@ -263,7 +442,7 @@ def _matmul_reduce_scatter(self, x, residual=None):
return residual
else:
# revert to torch dist (NCCL) reduce-scatter
- x = torch.matmul(x, self.weight.t())
+ x = _apply_linear(x, self.weight, self.config)
x, _ = reduce_scatter_along_first_dim(x, tp_group=self.tp_group)
return x
@@ -279,13 +458,17 @@ def _set_residual(self, residual: torch.Tensor):
"""
self.residual = residual
- @torch.no_grad()
- def forward(self, x: torch.Tensor, residual: Optional[torch.Tensor] = None) -> torch.Tensor:
+ def forward(
+ self, x: torch.Tensor, residual: Optional[torch.Tensor] = None
+ ) -> tuple[torch.Tensor, None]:
"""
Forward pass.
"""
+ if self.training:
+ return super().forward(x)
+
if self.tp_size == 1:
- x = torch.matmul(x, self.weight.t())
+ x = _apply_linear(x, self.weight, self.config)
return x, None
else:
x = self._matmul_reduce_scatter(x)
diff --git a/megatron/core/tensor_parallel/layers.py b/megatron/core/tensor_parallel/layers.py
index 69f442eb2d4..666245a9f6f 100644
--- a/megatron/core/tensor_parallel/layers.py
+++ b/megatron/core/tensor_parallel/layers.py
@@ -2,6 +2,7 @@
# Parts of the code here are adapted from PyTorch
# repo: https://github.com/pytorch/pytorch
+from __future__ import annotations
import os
import warnings
@@ -11,6 +12,7 @@
import torch
import torch.nn.functional as F
from torch.nn.parameter import Parameter
+from typing_extensions import override
from megatron.core.model_parallel_config import ModelParallelConfig
from megatron.core.parallel_state import (
@@ -367,7 +369,6 @@ def linear_with_frozen_weight(
tp_group: Optional[torch.distributed.ProcessGroup],
grad_output_buffer: Optional[List[torch.Tensor]] = None,
wgrad_deferral_limit: None = None,
- async_grad_allreduce: Optional[bool] = None,
) -> torch.Tensor:
"""Linear layer execution with weight.requires_grad == False.
@@ -404,19 +405,8 @@ def linear_with_frozen_weight(
wgrad_deferral_limit (int optional): dummy argument, used to
keep the API unified between all forward implementation functions.
-
-
- async_grad_allreduce (bool optional): Will be removed with 0.11.0.
- Please use allreduce_dgrad instead.
-
"""
- if async_grad_allreduce is not None:
- warnings.warn(
- "async_grad_allreduce is deprecated, not in use anymore and will"
- " be fully removed with 0.11.0. Please use allreduce_dgrad instead."
- )
-
assert grad_output_buffer is None, (
"grad_output_buffer kwarg is only supported with "
"linear_with_grad_accumulation_and_async_allreduce"
@@ -635,7 +625,6 @@ def linear_with_grad_accumulation_and_async_allreduce(
sequence_parallel: bool,
grad_output_buffer: Optional[List[torch.Tensor]] = None,
wgrad_deferral_limit: Optional[int] = 0,
- async_grad_allreduce: Optional[bool] = None,
tp_group: Optional[torch.distributed.ProcessGroup] = None,
) -> torch.Tensor:
"""Linear layer execution with asynchronous communication and
@@ -699,17 +688,8 @@ def linear_with_grad_accumulation_and_async_allreduce(
wgrad_deferral_limit (int optional): Limit on the number of
micro-batches for which embedding weight gradient GEMM should be
deferred. Disable by setting this to 0. Defaults to 0.
-
- async_grad_allreduce (bool optional): Will be removed with 0.11.0.
- Please use allreduce_dgrad instead.
"""
- if async_grad_allreduce is not None:
- warnings.warn(
- "async_grad_allreduce is deprecated, not in use anymore and will"
- " be fully removed with 0.11.0. Please use allreduce_dgrad instead."
- )
-
tp_group = get_tensor_model_parallel_group_if_none(tp_group)
args = [
@@ -1076,12 +1056,16 @@ def get_extra_state(self) -> None:
"""Keep compatibility with TE state dict."""
return None
- def __repr__(self):
+ @override
+ def extra_repr(self) -> str:
+ """Extra context to add to the module's string representation."""
tp = self.output_size // self.output_size_per_partition
use_bias = self.bias is not None and self.bias is True
return (
- f"{type(self).__name__}(in_features={self.input_size}, "
- f"out_features={self.output_size}, bias={use_bias}, TP={tp})"
+ f"in_features={self.input_size}, "
+ f"out_features={self.output_size}, "
+ f"bias={use_bias}, "
+ f"TP={tp}"
)
@@ -1134,7 +1118,7 @@ def __init__(
stride: int = 1,
keep_master_weight_for_test: bool = False,
is_expert: bool = False,
- tp_comm_buffer_name: str = None, # Not used
+ tp_comm_buffer_name: str | None = None, # Not used
tp_group: Optional[torch.distributed.ProcessGroup] = None,
):
super(RowParallelLinear, self).__init__()
@@ -1242,7 +1226,7 @@ def _forward_impl(self, input, weight, *args, **kwargs):
else:
return linear_with_grad_accumulation_and_async_allreduce(input, weight, *args, **kwargs)
- def forward(self, input_):
+ def forward(self, input_: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
"""Forward of RowParallelLinear
Args:
@@ -1300,6 +1284,13 @@ def forward(self, input_):
output_bias = self.bias
return output, output_bias
+ def backward_dw(self) -> None:
+ """Compute weight gradients during the backward pass if delay_wgrad_compute is enabled.
+
+ Not supported - does nothing.
+ """
+ pass
+
def sharded_state_dict(self, prefix="", sharded_offsets=(), metadata=None):
"""Sharding along axis 1, bias not sharded"""
state_dict = self.state_dict(prefix="", keep_vars=True)
@@ -1319,10 +1310,14 @@ def get_extra_state(self) -> None:
"""Keep compatibility with TE state dict."""
return None
- def __repr__(self):
+ @override
+ def extra_repr(self) -> str:
+ """Extra context to add to the module's string representation."""
tp = self.input_size // self.input_size_per_partition
use_bias = self.bias is not None and self.bias is True
return (
- f"{type(self).__name__}(in_features={self.input_size}, "
- f"out_features={self.output_size}, bias={use_bias}, TP={tp})"
+ f"in_features={self.input_size}, "
+ f"out_features={self.output_size}, "
+ f"bias={use_bias}, "
+ f"TP={tp}"
)
diff --git a/megatron/core/tensor_parallel/random.py b/megatron/core/tensor_parallel/random.py
index bf00717ab6c..92d39ba92ef 100644
--- a/megatron/core/tensor_parallel/random.py
+++ b/megatron/core/tensor_parallel/random.py
@@ -2,16 +2,20 @@
# Parts of the code here are adapted from PyTorch
# repo: https://github.com/pytorch/pytorch
+from __future__ import annotations
import contextlib
import logging
-from typing import Optional, Union
+from collections.abc import Callable
+from typing import Any, Optional, TypeVar, Union
import torch
from torch import _C
from torch.cuda import _lazy_call, _lazy_init
from torch.cuda import device as device_ctx_manager
from torch.utils.checkpoint import detach_variable
+from torch.utils.cpp_extension import load_inline
+from typing_extensions import TypeVarTuple, Unpack
from megatron.core.parallel_state import (
get_expert_model_parallel_rank,
@@ -20,6 +24,57 @@
)
from megatron.core.utils import is_te_min_version, safely_set_viewless_tensor_data
+# ---------------------------------------------------------------------------
+# C++ extension: zero-copy storage sharing for CheckpointWithoutOutput
+# ---------------------------------------------------------------------------
+# Makes dst's UntypedStorage point to src's data WITHOUT copying bytes.
+# Holds a refcounted reference to src's StorageImpl so the memory stays alive.
+# Operates below the Tensor / autograd layer → no version-counter bump,
+# and ALL TensorImpls that reference dst's StorageImpl (including views
+# created by reshape / split / etc. inside TE GroupedLinear) see the data.
+# ---------------------------------------------------------------------------
+
+_SHARE_STORAGE_SRC = r"""
+#include
+
+void share_storage(at::Tensor dst, at::Tensor src) {
+ auto* dst_impl = dst.storage().unsafeGetStorageImpl();
+
+ // Copy src's c10::Storage (increments StorageImpl refcount).
+ auto* src_storage_ref = new c10::Storage(src.storage());
+
+ void* data = src_storage_ref->data_ptr().get();
+ size_t nbytes = src_storage_ref->nbytes();
+ c10::Device device = src_storage_ref->device();
+
+ // Build a DataPtr whose deleter releases our StorageImpl reference.
+ c10::DataPtr shared(
+ data,
+ static_cast(src_storage_ref),
+ [](void* ctx) { delete static_cast(ctx); },
+ device);
+
+ dst_impl->set_data_ptr(std::move(shared));
+ dst_impl->set_nbytes(nbytes);
+}
+"""
+
+_share_storage_ext = None
+
+
+def _get_share_storage():
+ """Lazily compile & cache the share_storage extension."""
+ global _share_storage_ext
+ if _share_storage_ext is None:
+ _share_storage_ext = load_inline(
+ name="share_storage_ext",
+ cpp_sources=_SHARE_STORAGE_SRC,
+ functions=["share_storage"],
+ verbose=False,
+ )
+ return _share_storage_ext.share_storage
+
+
from .utils import gather_split_1d_tensor, split_tensor_into_1d_equal_chunks
try:
@@ -493,6 +548,10 @@ def is_checkpointing():
return IS_CHECKPOINTING
+_R = TypeVar('_R')
+_Ts = TypeVarTuple('_Ts')
+
+
class CheckpointFunction(torch.autograd.Function):
"""Checkpoint Function
@@ -503,7 +562,12 @@ class CheckpointFunction(torch.autograd.Function):
# pylint: disable=missing-function-docstring
@staticmethod
- def forward(ctx, run_function, distribute_saved_activations, *args):
+ def forward(
+ ctx: Any,
+ run_function: Callable[[Unpack[_Ts]], _R],
+ distribute_saved_activations: bool,
+ *args: Unpack[_Ts],
+ ) -> _R:
"""Forward pass."""
_set_checkpointing()
@@ -570,7 +634,9 @@ def backward(ctx, *args):
return (None, None) + grads
-def checkpoint(function, distribute_saved_activations, *args):
+def checkpoint(
+ function: Callable[[Unpack[_Ts]], _R], distribute_saved_activations: bool, *args: Unpack[_Ts]
+) -> _R:
"""Checkpoint a model or part of the model.
This has been directly copied from torch.utils.checkpoint."""
return CheckpointFunction.apply(function, distribute_saved_activations, *args)
@@ -578,12 +644,17 @@ def checkpoint(function, distribute_saved_activations, *args):
class CheckpointWithoutOutputFunction(torch.autograd.Function):
"""
- Checkpoint Function Helper for CheckpointWithouOutput.
+ Checkpoint Function Helper for CheckpointWithoutOutput.
Save context for recompute.
"""
@staticmethod
- def forward(ctx, run_function, checkpoint_without_output_obj, *args):
+ def forward(
+ ctx: Any,
+ run_function: Callable[[Unpack[_Ts]], _R],
+ checkpoint_without_output_obj: CheckpointWithoutOutput,
+ *args: Unpack[_Ts],
+ ) -> _R:
"""Forward pass."""
if checkpoint_without_output_obj.fp8:
fp8 = FP8GlobalStateManager.is_fp8_enabled()
@@ -641,7 +712,7 @@ def __init__(self, fp8=False):
self.ctx = None
self.outputs = None
- def checkpoint(self, run_function, *args):
+ def checkpoint(self, run_function: Callable[[Unpack[_Ts]], _R], *args: Unpack[_Ts]) -> _R:
"""Checkpoint function."""
# If in cuda graph warmup, disable checkpointing, as 'discard_output_and_register_recompute'
@@ -709,12 +780,14 @@ def detach(t):
if isinstance(outputs, torch.Tensor):
outputs = (outputs,)
- # restore the recomputed memory without changing the metadata
- with torch.no_grad():
- for output, recomputation_output in zip(self.outputs, outputs):
- output_size = recomputation_output.untyped_storage().size()
- output.untyped_storage().resize_(output_size)
- output.untyped_storage().copy_(recomputation_output.untyped_storage())
+ # Zero-copy: make output's StorageImpl point to recomputation_output's data.
+ # This operates at the UntypedStorage level (below TensorImpl), so:
+ # - ALL views / reshapes that reference output's StorageImpl see the data
+ # (e.g. TE GroupedLinear's inp.reshape() + torch.split() saved for backward)
+ # - No tensor version-counter bump (no autograd complaint)
+ share_storage = _get_share_storage()
+ for output, recomputation_output in zip(self.outputs, outputs):
+ share_storage(output, recomputation_output)
self.ctx.outputs = outputs
self.ctx.inputs = inputs
diff --git a/megatron/core/tokenizers/chat_template_utils.py b/megatron/core/tokenizers/chat_template_utils.py
new file mode 100644
index 00000000000..5c5c6330308
--- /dev/null
+++ b/megatron/core/tokenizers/chat_template_utils.py
@@ -0,0 +1,46 @@
+# Copyright (c) 2026, Advanced Micro Devices, Inc. All rights reserved.
+
+"""Helpers for HuggingFace ``apply_chat_template`` tokenized outputs."""
+
+from collections.abc import Mapping
+
+import numpy as np
+
+
+def token_ids_from_chat_template_output(output):
+ """Normalize ``apply_chat_template(..., tokenize=True, ...)`` to 1-D int64 token ids.
+
+ Depending on ``transformers`` / ``tokenizers`` versions, tokenized output may be a
+ ``numpy`` array, ``transformers.BatchEncoding`` (``input_ids``), a ``tokenizers.Encoding``,
+ or a length-1 batch of encodings.
+ """
+ if isinstance(output, Mapping) and "input_ids" in output:
+ return token_ids_from_chat_template_output(output["input_ids"])
+
+ try:
+ import torch
+
+ if isinstance(output, torch.Tensor):
+ output = output.detach().cpu().numpy()
+ except ImportError:
+ pass
+
+ if isinstance(output, np.ndarray):
+ arr = np.asarray(output, dtype=np.int64)
+ if arr.ndim >= 2:
+ arr = arr[0]
+ return arr.reshape(-1)
+
+ # tokenizers.Encoding (not dict-like)
+ if hasattr(output, "ids") and not isinstance(output, (str, bytes, list, tuple, Mapping)):
+ return np.asarray(output.ids, dtype=np.int64).reshape(-1)
+
+ if isinstance(output, (list, tuple)):
+ if len(output) == 1 and hasattr(output[0], "ids") and not isinstance(output[0], Mapping):
+ return np.asarray(output[0].ids, dtype=np.int64).reshape(-1)
+ arr = np.asarray(output, dtype=np.int64)
+ if arr.ndim >= 2:
+ arr = arr[0]
+ return arr.reshape(-1)
+
+ raise TypeError(f"Unexpected apply_chat_template output type: {type(output)}")
diff --git a/megatron/core/tokenizers/megatron_tokenizer.py b/megatron/core/tokenizers/megatron_tokenizer.py
index 14b273e909b..31694d91af0 100644
--- a/megatron/core/tokenizers/megatron_tokenizer.py
+++ b/megatron/core/tokenizers/megatron_tokenizer.py
@@ -1,5 +1,6 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+import importlib
import json
import logging
import os
@@ -10,15 +11,25 @@
TOKENIZER_MAPPING_NAMES = OrderedDict(
[
- ("default", "DefaultTokenizerText"),
+ ("default-text", "DefaultTokenizerText"),
("gpt", "GPTTokenizer"),
("mamba", "MambaTokenizer"),
("bert", "BertTokenizer"),
("t5", "T5Tokenizer"),
+ ("default-vision", "DefaultTokenizerVision"),
]
)
-TOKENIZER_LIBRARIES = ["sentencepiece", "huggingface", "megatron", "tiktoken", "byte-level", "null"]
+TEXT_LIBRARIES = [
+ "sentencepiece",
+ "huggingface",
+ "megatron",
+ "tiktoken",
+ "byte-level",
+ "null-text",
+ "sft",
+]
+VISION_LIBRARIES = ["multimodal", "null-multimodal"]
logger = logging.getLogger(__name__)
@@ -69,19 +80,22 @@ def from_pretrained(
f"Expected metadata_path to be str or dict, but got {type(metadata_path)}."
)
- if metadata.get('library', None) not in ['byte-level', 'null']:
+ tokenizer_library = metadata.get('library', None)
+ if tokenizer_library not in ['byte-level', 'null-text', 'null-multimodal']:
assert tokenizer_path, "Tokenizer path must be specified."
+ if tokenizer_library in ['multimodal']:
+ assert 'prompt_format' in kwargs, "Prompt format (`prompt_format`) must be specified."
+ assert (
+ 'special_tokens' in kwargs
+ ), "Special tokens (`special_tokens`) must be specified."
+ assert (
+ 'image_tag_type' in kwargs
+ ), "Image tag type (`image_tag_type`) must be specified."
+
# Initialize tokenizer object
- if metadata.get('tokenizer_class', None):
- tokenizer_cls = getattr(
- metadata['tokenizer_class_path'], metadata['tokenizer_class_name']
- )
- else:
- import megatron.core.tokenizers.text.models as models
+ tokenizer_cls = _get_tokenizer_model_class(tokenizer_library, metadata)
- model_type = metadata.get('model_type', 'default')
- tokenizer_cls = getattr(models, TOKENIZER_MAPPING_NAMES[model_type])
metadata['metadata_path'] = metadata_path
tokenizer = tokenizer_cls(path=tokenizer_path, config=metadata, **kwargs)
@@ -122,12 +136,13 @@ def write_metadata(
assert os.path.exists(
tokenizer_path
), "Tokenizer path doesn't exist. Please, provide the correct path to the tokenizer."
- assert tokenizer_library in TOKENIZER_LIBRARIES, (
+ assert tokenizer_library in TEXT_LIBRARIES or tokenizer_library in VISION_LIBRARIES, (
"Tokenizer library is not supported. Please, see the list of available "
- f"tokenizer libraries: {TOKENIZER_LIBRARIES}."
+ f"tokenizer libraries: text: {TEXT_LIBRARIES}, vision: {VISION_LIBRARIES}."
)
+ tokenizer_type = 'text' if tokenizer_library in TEXT_LIBRARIES else 'vision'
if model_type is None and tokenizer_class is None:
- model_type = "default"
+ model_type = f"default-{tokenizer_type}"
# Write metadata
if not metadata_path:
@@ -168,3 +183,33 @@ def _get_metadata_path(tokenizer_path: str) -> str:
metadata_path = f'{dir_path}/tokenizer_metadata.json'
return metadata_path
+
+
+def _get_tokenizer_model_class(library: str, metadata: dict) -> MegatronTokenizerBase:
+ """
+ Returns a class which corresponds to choosen tokenizer model type.
+
+ Args:
+ library (str): tokenizer library.
+ metadata (dict): tokenizer metadata.
+
+ Returns:
+ MegatronTokenizerBase: class for choosen tokenizer model type.
+ """
+ # Return tokenizer class if it was specified in metadata.
+ if metadata.get('tokenizer_class', None):
+ return getattr(metadata['tokenizer_class_path'], metadata['tokenizer_class_name'])
+
+ # Define tokenizer type
+ tokenizer_type = 'text' if library in TEXT_LIBRARIES else 'vision'
+
+ module_name = f"megatron.core.tokenizers.{tokenizer_type}.models"
+ models = importlib.import_module(module_name)
+
+ model_type = metadata.get("model_type", None)
+ if model_type is None:
+ model_type = f"default-{tokenizer_type}"
+
+ tokenizer_cls = getattr(models, TOKENIZER_MAPPING_NAMES[model_type])
+
+ return tokenizer_cls
diff --git a/megatron/core/tokenizers/text/libraries/__init__.py b/megatron/core/tokenizers/text/libraries/__init__.py
index e9141b81b8d..92a97b31f9a 100644
--- a/megatron/core/tokenizers/text/libraries/__init__.py
+++ b/megatron/core/tokenizers/text/libraries/__init__.py
@@ -5,4 +5,5 @@
from megatron.core.tokenizers.text.libraries.megatron_hf_tokenizer import MegatronHFTokenizer
from megatron.core.tokenizers.text.libraries.null_tokenizer import NullTokenizer
from megatron.core.tokenizers.text.libraries.sentencepiece_tokenizer import SentencePieceTokenizer
+from megatron.core.tokenizers.text.libraries.sft_tokenizer import SFTTokenizer
from megatron.core.tokenizers.text.libraries.tiktoken_tokenizer import TikTokenTokenizer
diff --git a/megatron/core/tokenizers/text/libraries/huggingface_tokenizer.py b/megatron/core/tokenizers/text/libraries/huggingface_tokenizer.py
index 965f43733a6..4e3387b125c 100644
--- a/megatron/core/tokenizers/text/libraries/huggingface_tokenizer.py
+++ b/megatron/core/tokenizers/text/libraries/huggingface_tokenizer.py
@@ -36,9 +36,9 @@ def __init__(
cls_token: Optional[str] = None,
unk_token: Optional[str] = None,
additional_special_tokens: Optional[List] = [],
- use_fast: Optional[bool] = False,
+ use_fast: Optional[bool] = True,
trust_remote_code: Optional[bool] = False,
- include_special_tokens: bool = False,
+ include_special_tokens: bool = True,
chat_template: str = None,
):
"""
@@ -249,13 +249,22 @@ def text_to_ids(self, text: str) -> List[int]:
ids = self.tokens_to_ids(tokens)
return ids
- def ids_to_text(self, ids: List[int], remove_special_tokens: bool = True) -> str:
- """Converts list of ids to text."""
+ def ids_to_text(self, ids: List[int], remove_special_tokens: Optional[bool] = None) -> str:
+ """Converts list of ids to text.
+
+ When remove_special_tokens is None, uses not self.include_special_tokens so that
+ --tokenizer-hf-include-special-tokens keeps EOS (and other special tokens) in
+ detokenized output (e.g. for RL trajectory consistency).
+ """
+ if remove_special_tokens is None:
+ remove_special_tokens = not self.include_special_tokens
tokens = self.ids_to_tokens(ids)
if remove_special_tokens:
- tokens_clean = [t for t in tokens if t not in self.tokenizer.all_special_tokens]
+ tokens_clean = [
+ t for t in tokens if t is not None and t not in self.tokenizer.all_special_tokens
+ ]
else:
- tokens_clean = tokens
+ tokens_clean = [t for t in tokens if t is not None]
text = self.tokens_to_text(tokens_clean)
return text
diff --git a/megatron/core/tokenizers/text/libraries/sentencepiece_tokenizer.py b/megatron/core/tokenizers/text/libraries/sentencepiece_tokenizer.py
index c7a5fe0b7dc..feaf1c4e9a1 100644
--- a/megatron/core/tokenizers/text/libraries/sentencepiece_tokenizer.py
+++ b/megatron/core/tokenizers/text/libraries/sentencepiece_tokenizer.py
@@ -310,6 +310,10 @@ def add_special_tokens(self, special_tokens: Union[list, dict]) -> None:
"Expected special_tokens to be a list or a dict " + str(type(special_tokens))
)
+ def offsets(self, ids: list[int], text: str) -> list[int]:
+ """Calculate offsets."""
+ return [p.begin for p in self.tokenizer.decode_ids_as_immutable_proto(ids).pieces]
+
@property
def pad_id(self) -> int:
"""Returns id of padding token."""
diff --git a/megatron/training/tokenizer/sft_tokenizer.py b/megatron/core/tokenizers/text/libraries/sft_tokenizer.py
similarity index 60%
rename from megatron/training/tokenizer/sft_tokenizer.py
rename to megatron/core/tokenizers/text/libraries/sft_tokenizer.py
index 274c6f6c944..efb989829dd 100644
--- a/megatron/training/tokenizer/sft_tokenizer.py
+++ b/megatron/core/tokenizers/text/libraries/sft_tokenizer.py
@@ -1,24 +1,55 @@
-# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2026, Advanced Micro Devices, Inc. All rights reserved. Modified for portability.
-"""SFT tokenizer."""
+from dataclasses import dataclass
from typing import Dict, List, Union
+
import numpy as np
-nemotron_h_aligned_custom_template = """{% for message in messages %}{% if message['role'] == 'system' %}{{ 'System\n' + message['content'].strip() + '\n' }}{% elif message['role'] == 'user' %}{{ 'User\n' + message['content'].strip() + '\n' + 'Assistant\n' }}{% elif message['role'] == 'assistant' %}{{ message['content'].strip() + '\n' }}{% endif %}{% endfor %}"""
-nemotron_nano_v2_custom_template = """{% for message in messages %}{% set content = message['content'] %}{% if message['role'] == 'system' %}{{ 'System\n' + content.replace('/think', '').replace('/no_think', '').strip() + '\n' }}{% elif message['role'] == 'user' %}{{ 'User\n' + content.replace('/think', '').replace('/no_think', '').strip() + '\n' }}{% elif message['role'] == 'assistant' %}{{ 'Assistant\n' + content.strip() + '\n\n' }}{% endif %}{% endfor %}"""
+from megatron.core.tokenizers.chat_template_utils import token_ids_from_chat_template_output
+
+try:
+ import transformers
+
+ HAVE_TRANSFORMERS = True
+except ModuleNotFoundError:
+ HAVE_TRANSFORMERS = False
+
+
+# fmt: off
+nemotron_h_aligned_custom_template = """{% for message in messages %}{% if message['role'] == 'system' %}{{ 'System\n' + message['content'].strip() + '\n' }}{% elif message['role'] == 'user' %}{{ 'User\n' + message['content'].strip() + '\n' + 'Assistant\n' }}{% elif message['role'] == 'assistant' %}{{ message['content'].strip() + '\n' }}{% endif %}{% endfor %}""" # pylint: disable=line-too-long
+nemotron_nano_v2_custom_template = """{% for message in messages %}{% set content = message['content'] %}{% if message['role'] == 'system' %}{{ 'System\n' + content.replace('/think', '').replace('/no_think', '').strip() + '\n' }}{% elif message['role'] == 'user' %}{{ 'User\n' + content.replace('/think', '').replace('/no_think', '').strip() + '\n' }}{% elif message['role'] == 'assistant' %}{{ 'Assistant\n' + content.strip() + '\n\n' }}{% endif %}{% endfor %}""" # pylint: disable=line-too-long
+identity_template = """{% for message in messages %}{{ message['content'] }}{% endfor %}"""
+# fmt: on
+
+
+IGNORE_INDEX = -100
+
+
+@dataclass
+class PromptConfig:
+ """Config options for different prompt formats."""
-from megatron.core.datasets.megatron_tokenizer import MegatronLegacyTokenizer
-from megatron.training.datasets.sft_dataset import IGNORE_INDEX
-from megatron.training.tokenizer.multimodal_tokenizer import PromptConfig
+ # How many tokens are used for the assistant prefix, e.g. "<|im_start|>assistant\n".
+ # Used for masking the assistant prefix.
+ assistant_prefix_len: int
+ # Padding token ID.
+ pad_token_id: int
+ # For overriding the default chat format template.
+ custom_chat_template: str
+ # If the tokenizer inserts BOS token by default.
+ has_bos: bool
+ # If the tokenizer supports a separate role for system messages.
+ has_system_role: bool
+ # Wether to force a specific system message.
+ force_system_message: bool = False
+ system_default: dict = None
-class SFTTokenizer(MegatronLegacyTokenizer):
+
+class SFTTokenizer:
"""SFT Tokenizer."""
- def __init__(
- self,
- tokenizer_path: str,
- prompt_format: str,
- ):
+ def __init__(self, tokenizer_path: str, prompt_format: str):
"""
Note: Currently, only HuggingFaceTokenizer is supported as the underlying text tokenizer.
@@ -26,19 +57,16 @@ def __init__(
tokenizer_path (str): Underlying tokenizer path.
prompt_format (str): Prompt format for the tokenizer.
"""
- super().__init__(tokenizer_path, prompt_format=prompt_format)
- try:
- import transformers
- except ImportError:
+ if HAVE_TRANSFORMERS:
+ # Currently, only HuggingFace tokenizers are supported.
+ tokenizer = transformers.AutoTokenizer.from_pretrained(
+ pretrained_model_name_or_path=tokenizer_path
+ )
+ else:
raise ImportError(
"SFTTokenizer currently requires transformers library to be installed"
)
- # Currently, only HuggingFace tokenizers are supported.
- tokenizer = transformers.AutoTokenizer.from_pretrained(
- pretrained_model_name_or_path=tokenizer_path,
- )
-
self._vocab_size = len(tokenizer)
self._tokenizer = tokenizer
@@ -58,12 +86,31 @@ def __init__(
has_bos=False,
has_system_role=True,
)
+ elif prompt_format == "identity":
+ self._prompt_config = PromptConfig(
+ assistant_prefix_len=0,
+ pad_token_id=tokenizer.convert_tokens_to_ids(""),
+ custom_chat_template=identity_template,
+ has_bos=False,
+ has_system_role=True,
+ )
+ elif prompt_format == "default":
+ self._prompt_config = PromptConfig(
+ assistant_prefix_len=0,
+ pad_token_id=(
+ tokenizer.pad_token_id
+ if tokenizer.pad_token_id is not None
+ else tokenizer.eos_token_id
+ ),
+ custom_chat_template=tokenizer.chat_template,
+ has_bos=tokenizer.bos_token_id is not None,
+ has_system_role=True,
+ )
else:
raise NotImplementedError("unknown SFT prompt format", prompt_format)
self._prompt_format = prompt_format
-
def tokenize_conversation(
self, conversation: List[Dict], return_target: bool, add_generation_prompt: bool
):
@@ -84,32 +131,39 @@ def tokenize_conversation(
if not self._prompt_config.has_system_role and conversation[0]["role"] == "system":
conversation = conversation[1:]
- tokens = self._tokenizer.apply_chat_template(
+ raw = self._tokenizer.apply_chat_template(
conversation,
tokenize=True,
add_generation_prompt=add_generation_prompt,
return_assistant_token_mask=False,
return_tensors="np",
chat_template=self._prompt_config.custom_chat_template,
- )[0]
+ )
+ tokens = token_ids_from_chat_template_output(raw)
if not return_target:
return tokens
target = tokens.copy()
+ # When using the default prompt format, we do not replace any tokens with IGNORE_INDEX.
+ # Instead, all token losses will be used for simplicity.
+ if self._prompt_format == "default":
+ return tokens, target
+
# Mask system and user tokens in the target.
idx = 0
for turn_idx, turn in enumerate(conversation):
-
+
if turn["role"].lower() == "assistant" and len(turn["content"]) == 0:
raise ValueError(f"empty assistant turn in conversation: {conversation}.")
if turn["role"].lower() == "assistant":
- assert conversation[turn_idx-1]["role"].lower() == "user"
+ assert conversation[turn_idx - 1]["role"].lower() in ("user", "tool")
- turn_tokens = self._tokenizer.apply_chat_template(
+ turn_raw = self._tokenizer.apply_chat_template(
[turn], tokenize=True, chat_template=self._prompt_config.custom_chat_template
)
+ turn_tokens = token_ids_from_chat_template_output(turn_raw)
# There should be only one BOS at the very beginning.
# After the first turn, skip BOS token.
@@ -118,55 +172,69 @@ def tokenize_conversation(
turn_len = len(turn_tokens)
role = turn["role"].lower()
- if role in ("system", "user"):
+ if role in ("system", "user", "tool"):
target[idx : idx + turn_len] = IGNORE_INDEX
elif role == "assistant":
if self._prompt_config.assistant_prefix_len > 0:
target[idx : idx + self._prompt_config.assistant_prefix_len] = IGNORE_INDEX
else:
- raise ValueError(f"Wrong role value.")
+ raise ValueError("Wrong role value.")
assert np.allclose(
tokens[idx : idx + turn_len], turn_tokens
), f"expected turn tokens to match tokens in conversation {conversation}"
idx += turn_len
-
+
assert idx == len(tokens), f"mismatch in target masking the conversation {conversation}"
return tokens, target
- def tokenize(self, text: Union[str, List[Dict]]):
+ def text_to_ids(self, text: Union[str, List[Dict]]):
"""Tokenize conversation or string input."""
if isinstance(text, list):
# This code path is used by the inference code currently.
- return self.tokenize_conversation(text, return_target=False, add_generation_prompt=True).tolist()
-
- return self._encode(text)
+ return self.tokenize_conversation(
+ text, return_target=False, add_generation_prompt=True
+ ).tolist()
- def _encode(self, text: str):
- """Tokenize text input, w/o chat template"""
return self._tokenizer.encode(text)
- def convert_tokens_to_ids(self, tokens: List[str]):
+ def tokens_to_ids(self, tokens: List[str]):
"""Convert tokens to IDs."""
return self._tokenizer.convert_tokens_to_ids(tokens)
- def detokenize(self, tokens: List[int]):
+ def ids_to_text(self, tokens: List[int]):
"""Detokenize tokens."""
return self._tokenizer.decode(tokens)
+ def ids_to_tokens(self):
+ """Converts ids to tokens."""
+ raise NotImplementedError("This method is not supported for SFTTokenizer.")
+
+ def text_to_tokens(self):
+ """Converts text to tokens."""
+ raise NotImplementedError("This method is not supported for SFTTokenizer.")
+
+ def tokens_to_text(self):
+ """Converts tokens to text."""
+ raise NotImplementedError("This method is not supported for SFTTokenizer.")
+
def get_special_tokens(self):
"""Get special tokens."""
return self._tokenizer.get_added_vocab()
+ def add_special_tokens(self):
+ """Add special tokens."""
+ raise NotImplementedError("This method is not supported for SFTTokenizer.")
+
@property
- def pad(self):
+ def pad_id(self):
"""Pad token ID."""
return self._prompt_config.pad_token_id
@property
- def bos(self):
+ def bos_id(self):
"""Beginning of sequence token ID."""
return self._tokenizer.bos_token_id
diff --git a/megatron/core/tokenizers/text/libraries/tiktoken_tokenizer.py b/megatron/core/tokenizers/text/libraries/tiktoken_tokenizer.py
index 20e13206ceb..39228ad4afd 100644
--- a/megatron/core/tokenizers/text/libraries/tiktoken_tokenizer.py
+++ b/megatron/core/tokenizers/text/libraries/tiktoken_tokenizer.py
@@ -15,16 +15,17 @@
from .abstract_tokenizer import MegatronTokenizerTextAbstract
from .chat_template import MegatronTokenizerChatTemplate
-PATTERN_TIKTOKEN_V1 = (
+logger = logging.getLogger(__name__)
+
+_PATTERN_TIKTOKEN_V1 = (
r"[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]++[\r\n]*|\s*[\r\n]|\s+(?!\S)|\s+"
)
-PATTERN_TIKTOKEN_V2 = "[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]*[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]+|[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]+[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]*|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n/]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" # pylint: disable=line-too-long
+_PATTERN_TIKTOKEN_V2 = "[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]*[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]+|[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]+[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]*|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n/]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" # pylint: disable=line-too-long
DEFAULT_TIKTOKEN_MAX_VOCAB = 2**17 # 131072
SPECIAL_TOKENS = ["", "", " ", "", "", "", ""]
SPECIAL_TOKEN_TEMPLATE = ""
-
-logger = logging.getLogger(__name__)
+__all__ = ["TikTokenTokenizer", "reload_mergeable_ranks"]
def reload_mergeable_ranks(
@@ -101,9 +102,9 @@ def __init__(
special_tokens = SPECIAL_TOKENS.copy()
if pattern == "v1":
- pattern = PATTERN_TIKTOKEN_V1
+ pattern = _PATTERN_TIKTOKEN_V1
elif pattern == "v2":
- pattern = PATTERN_TIKTOKEN_V2
+ pattern = _PATTERN_TIKTOKEN_V2
else:
raise ValueError(f"Expected tiktoken pattern to be `v1` or `v2`, but got {pattern}.")
@@ -156,9 +157,7 @@ def __init__(
self.shifted_id2token = {i: tok for i, tok in enumerate(self.special_tokens)}
for key, value in self.id2token.items():
- self.shifted_id2token[key + self.num_special_tokens] = value.decode(
- 'utf-8', errors='replace'
- )
+ self.shifted_id2token[key + self.num_special_tokens] = value
special_tokens_dict = {t: i for i, t in enumerate(self.special_tokens)}
self.tokenizer = tiktoken.Encoding(
@@ -168,6 +167,8 @@ def __init__(
special_tokens=special_tokens_dict, # special tokens are handled manually
)
+ self._vocab = special_tokens_dict | self.token2id
+
def text_to_tokens(self, text: str) -> List[str]:
"""Converts text to tokens."""
token_ids = self.tokenizer.encode(text)
@@ -232,6 +233,24 @@ def add_special_tokens(self, special_tokens_dict: dict):
"""Adds special tokens to the tokenizer."""
raise NotImplementedError("This method is not supported for TikToken tokenizers.")
+ def offsets(self, ids: list[int], text: str) -> list[int]:
+ """Calculate offsets."""
+ try:
+ return self.tokenizer.decode_with_offsets(ids)[1]
+ except UnicodeDecodeError:
+ # Tiktoken has an unnecessary check that raises UnicodeDecodeError
+ # from `text = b"".join(token_bytes).decode("utf-8", errors="strict")`
+ # which is not needed for our use case. So we re-implement it, without
+ # the check.
+
+ token_bytes = self.tokenizer.decode_tokens_bytes(ids)
+ text_len = 0
+ offsets = []
+ for token in token_bytes:
+ offsets.append(max(0, text_len - (0x80 <= token[0] < 0xC0)))
+ text_len += sum(1 for c in token if not 0x80 <= c < 0xC0)
+ return offsets
+
@property
def additional_special_tokens_ids(self) -> list:
"""
@@ -291,7 +310,7 @@ def sep_id(self) -> int:
@property
def vocab(self):
"""Returns tokenizer vocab."""
- return self.token2id
+ return self._vocab
@property
def decoder(self):
@@ -301,7 +320,7 @@ def decoder(self):
@property
def encoder(self):
""" """
- return self.vocab
+ return self._vocab
@property
def vocab_size(self) -> int:
diff --git a/megatron/core/tokenizers/text/parsers/__init__.py b/megatron/core/tokenizers/text/parsers/__init__.py
new file mode 100644
index 00000000000..dc27763f905
--- /dev/null
+++ b/megatron/core/tokenizers/text/parsers/__init__.py
@@ -0,0 +1,12 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+from megatron.core.tokenizers.text.parsers.deepseek_r1_reasoning_parser import (
+ DeepSeekR1ReasoningParser,
+)
+from megatron.core.tokenizers.text.parsers.qwen3_coder_tool_parser import Qwen3CoderToolParser
+
+PARSER_MAPPING = {
+ "deepseek-r1-reasoning": DeepSeekR1ReasoningParser,
+ "qwen3-coder-tool": Qwen3CoderToolParser,
+}
+
+__all__ = ["PARSER_MAPPING"]
diff --git a/megatron/core/tokenizers/text/parsers/base_parser.py b/megatron/core/tokenizers/text/parsers/base_parser.py
new file mode 100644
index 00000000000..afd847d1bfc
--- /dev/null
+++ b/megatron/core/tokenizers/text/parsers/base_parser.py
@@ -0,0 +1,21 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+from typing import Any
+
+
+class BaseParser:
+ """Base class for text parsers."""
+
+ @staticmethod
+ def parse(text: str, **kwargs) -> tuple[str, dict[str, Any]]:
+ """
+ Parses the text into a tuple containing extracted content
+ and a dictionary of additional information.
+
+ Args:
+ text (str): The text to parse.
+
+ Returns:
+ tuple[str, dict[str, Any]]: A tuple containing the unprocessed text
+ and a dictionary with the extracted information.
+ """
+ return text, {}
diff --git a/megatron/core/tokenizers/text/parsers/deepseek_r1_reasoning_parser.py b/megatron/core/tokenizers/text/parsers/deepseek_r1_reasoning_parser.py
new file mode 100644
index 00000000000..17952c61daf
--- /dev/null
+++ b/megatron/core/tokenizers/text/parsers/deepseek_r1_reasoning_parser.py
@@ -0,0 +1,33 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+from megatron.core.tokenizers.text.parsers.base_parser import BaseParser
+
+
+class DeepSeekR1ReasoningParser(BaseParser):
+ """Parser for DeepSeek R1 style reasoning output."""
+
+ @staticmethod
+ def parse(text: str, **kwargs) -> tuple[str, dict[str, str]]:
+ """
+ Extracts the reasoning content from the text using ... tags.
+ Only extracts the first set of think tags.
+ If an initial tag is not present but a tag is,
+ it will infer a tag at the beginning of the text.
+
+ Args:
+ text (str): The text to parse.
+
+ Returns:
+ tuple[str, dict[str, str]]: A tuple containing the unprocessed text
+ and a dictionary with the extracted reasoning content.
+ """
+
+ if " " in text:
+ if "" in text:
+ # Strip the prefix (it might not be present if it was part of the prompt)
+ pre_text, text = text.split("", maxsplit=1)
+ else:
+ pre_text = ""
+ reasoning_content, remaining_text = text.split(" ", maxsplit=1)
+ return pre_text + remaining_text, {'reasoning': reasoning_content}
+ else:
+ return text, {}
diff --git a/megatron/core/tokenizers/text/parsers/qwen3_coder_tool_parser.py b/megatron/core/tokenizers/text/parsers/qwen3_coder_tool_parser.py
new file mode 100644
index 00000000000..1d1f20a3a5c
--- /dev/null
+++ b/megatron/core/tokenizers/text/parsers/qwen3_coder_tool_parser.py
@@ -0,0 +1,282 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+import ast
+import json
+import logging
+import re
+import uuid
+from types import SimpleNamespace
+from typing import Any
+
+from megatron.core.tokenizers.text.parsers.base_parser import BaseParser
+
+logger = logging.getLogger(__name__)
+
+# These map to vLLM types but we just use dictionaries for now
+ToolCall = dict[str, Any]
+FunctionCall = dict[str, Any]
+ChatCompletionToolsParam = dict[str, Any]
+ChatCompletionRequest = dict[str, Any]
+ExtractedToolCallInformation = dict
+
+
+class _Qwen3CoderToolParser:
+
+ # Sentinel tokens for streaming mode
+ tool_call_start_token: str = ""
+ tool_call_end_token: str = " "
+ tool_call_prefix: str = "(.*?)", re.DOTALL)
+ tool_call_regex = re.compile(r"(.*?) |(.*?)$", re.DOTALL)
+ tool_call_function_regex = re.compile(r"||(?=)|$)", re.DOTALL
+ )
+
+ def _generate_tool_call_id(self) -> str:
+ """Generate a unique tool call ID."""
+ return f"call_{uuid.uuid4().hex[:24]}"
+
+ def _get_arguments_config(
+ self, func_name: str, tools: list[ChatCompletionToolsParam] | None
+ ) -> dict:
+ """Extract argument configuration for a function."""
+ if tools is None:
+ return {}
+ for config in tools:
+ config = SimpleNamespace(**config) # Convert to SimpleNamespace for ease of access
+ if not hasattr(config, "type") or not (
+ hasattr(config, "function") and hasattr(config.function, "name")
+ ):
+ continue
+ if config.type == "function" and config.function.name == func_name:
+ if not hasattr(config.function, "parameters"):
+ return {}
+ params = config.function.parameters
+ if isinstance(params, dict) and "properties" in params:
+ return params["properties"]
+ elif isinstance(params, dict):
+ return params
+ else:
+ return {}
+ logger.debug("Tool '%s' is not defined in the tools list.", func_name)
+ return {}
+
+ def _convert_param_value(
+ self, param_value: str, param_name: str, param_config: dict, func_name: str
+ ) -> Any:
+ """Convert parameter value based on its type in the schema."""
+ # Handle null value for any type
+ if param_value.lower() == "null":
+ return None
+
+ if param_name not in param_config:
+ if param_config != {}:
+ logger.debug(
+ "Parsed parameter '%s' is not defined in the tool "
+ "parameters for tool '%s', directly returning the "
+ "string value.",
+ param_name,
+ func_name,
+ )
+ return param_value
+
+ if isinstance(param_config[param_name], dict) and "type" in param_config[param_name]:
+ param_type = str(param_config[param_name]["type"]).strip().lower()
+ else:
+ param_type = "string"
+ if param_type in ["string", "str", "text", "varchar", "char", "enum"]:
+ return param_value
+ elif (
+ param_type.startswith("int")
+ or param_type.startswith("uint")
+ or param_type.startswith("long")
+ or param_type.startswith("short")
+ or param_type.startswith("unsigned")
+ ):
+ try:
+ return int(param_value)
+ except (ValueError, TypeError):
+ logger.debug(
+ "Parsed value '%s' of parameter '%s' is not an "
+ "integer in tool '%s', degenerating to string.",
+ param_value,
+ param_name,
+ func_name,
+ )
+ return param_value
+ elif param_type.startswith("num") or param_type.startswith("float"):
+ try:
+ float_param_value = float(param_value)
+ return (
+ float_param_value
+ if float_param_value - int(float_param_value) != 0
+ else int(float_param_value)
+ )
+ except (ValueError, TypeError):
+ logger.debug(
+ "Parsed value '%s' of parameter '%s' is not a float "
+ "in tool '%s', degenerating to string.",
+ param_value,
+ param_name,
+ func_name,
+ )
+ return param_value
+ elif param_type in ["boolean", "bool", "binary"]:
+ param_value = param_value.lower()
+ if param_value not in ["true", "false"]:
+ logger.debug(
+ "Parsed value '%s' of parameter '%s' is not a boolean "
+ "(`true` or `false`) in tool '%s', degenerating to "
+ "false.",
+ param_value,
+ param_name,
+ func_name,
+ )
+ return param_value == "true"
+ else:
+ if (
+ param_type in ["object", "array", "arr"]
+ or param_type.startswith("dict")
+ or param_type.startswith("list")
+ ):
+ try:
+ param_value = json.loads(param_value)
+ return param_value
+ except (json.JSONDecodeError, TypeError, ValueError):
+ logger.debug(
+ "Parsed value '%s' of parameter '%s' cannot be "
+ "parsed with json.loads in tool '%s', will try "
+ "other methods to parse it.",
+ param_value,
+ param_name,
+ func_name,
+ )
+ try:
+ param_value = ast.literal_eval(param_value) # safer
+ except (ValueError, SyntaxError, TypeError):
+ logger.debug(
+ "Parsed value '%s' of parameter '%s' cannot be "
+ "converted via Python `ast.literal_eval()` in tool "
+ "'%s', degenerating to string.",
+ param_value,
+ param_name,
+ func_name,
+ )
+ return param_value
+
+ def _parse_xml_function_call(
+ self, function_call_str: str, tools: list[ChatCompletionToolsParam] | None
+ ) -> ToolCall | None:
+ # Extract function name
+ end_index = function_call_str.index(">")
+ function_name = function_call_str[:end_index]
+ param_config = self._get_arguments_config(function_name, tools)
+ parameters = function_call_str[end_index + 1 :]
+ param_dict = {}
+ for match_text in self.tool_call_parameter_regex.findall(parameters):
+ idx = match_text.index(">")
+ param_name = match_text[:idx]
+ param_value = str(match_text[idx + 1 :])
+ # Remove prefix and trailing \n
+ if param_value.startswith("\n"):
+ param_value = param_value[1:]
+ if param_value.endswith("\n"):
+ param_value = param_value[:-1]
+
+ param_dict[param_name] = self._convert_param_value(
+ param_value, param_name, param_config, function_name
+ )
+ return ToolCall(
+ type="function",
+ id=self._generate_tool_call_id(),
+ function=FunctionCall(
+ name=function_name, arguments=json.dumps(param_dict, ensure_ascii=False)
+ ),
+ )
+
+ def _get_function_calls(self, model_output: str) -> list[str]:
+ # Find all tool calls
+ matched_ranges = self.tool_call_regex.findall(model_output)
+ raw_tool_calls = [match[0] if match[0] else match[1] for match in matched_ranges]
+
+ # Back-off strategy if no tool_call tags found
+ if len(raw_tool_calls) == 0:
+ raw_tool_calls = [model_output]
+
+ raw_function_calls = []
+ for tool_call in raw_tool_calls:
+ raw_function_calls.extend(self.tool_call_function_regex.findall(tool_call))
+
+ function_calls = [match[0] if match[0] else match[1] for match in raw_function_calls]
+ return function_calls
+
+ def extract_tool_calls(
+ self, model_output: str, tools: list[ChatCompletionToolsParam] | None
+ ) -> ExtractedToolCallInformation:
+ """Extracts the tool calls from the text using ... tags."""
+ # Quick check to avoid unnecessary processing
+ if self.tool_call_prefix not in model_output:
+ return ExtractedToolCallInformation(
+ tools_called=False, tool_calls=[], content=model_output
+ )
+
+ try:
+ function_calls = self._get_function_calls(model_output)
+ if len(function_calls) == 0:
+ return ExtractedToolCallInformation(
+ tools_called=False, tool_calls=[], content=model_output
+ )
+
+ tool_calls = [
+ self._parse_xml_function_call(function_call_str, tools)
+ for function_call_str in function_calls
+ ]
+
+ # Extract content before tool calls
+ content_index = model_output.find(self.tool_call_start_token)
+ idx = model_output.find(self.tool_call_prefix)
+ content_index = content_index if content_index >= 0 else idx
+ content = model_output[:content_index] # .rstrip()
+
+ return ExtractedToolCallInformation(
+ tools_called=(len(tool_calls) > 0),
+ tool_calls=tool_calls,
+ content=content if content else None,
+ )
+
+ except Exception:
+ logger.exception("Error in extracting tool call from response.")
+ return ExtractedToolCallInformation(
+ tools_called=False, tool_calls=[], content=model_output
+ )
+
+
+class Qwen3CoderToolParser(BaseParser):
+ """Parser for Qwen3 Coder style tool calls."""
+
+ @staticmethod
+ def parse(text: str, **kwargs) -> tuple[str, dict[str, list[dict]]]:
+ """
+ Extracts the tool calls from the text using ... tags.
+ Uses the _Qwen3CoderToolParser class (copied from vLLM) to extract the tool calls.
+
+ Args:
+ text (str): The text to parse.
+
+ Returns:
+ tuple[str, dict[str, str]]: A tuple containing the unprocessed text
+ and a dictionary with the extracted tool calls.
+ """
+
+ information = _Qwen3CoderToolParser().extract_tool_calls(
+ text, tools=kwargs.get("tools", [])
+ )
+ if information.get("tools_called", False):
+ return information.get("content", ""), {"tool_calls": information.get("tool_calls", [])}
+ else:
+ return text, {}
diff --git a/megatron/core/tokenizers/text/text_tokenizer.py b/megatron/core/tokenizers/text/text_tokenizer.py
index 4e0c624e006..0145ae353ca 100644
--- a/megatron/core/tokenizers/text/text_tokenizer.py
+++ b/megatron/core/tokenizers/text/text_tokenizer.py
@@ -13,7 +13,8 @@
("megatron", "MegatronHFTokenizer"),
("tiktoken", "TikTokenTokenizer"),
("byte-level", "ByteLevelTokenizer"),
- ("null", "NullTokenizer"),
+ ("null-text", "NullTokenizer"),
+ ("sft", "SFTTokenizer"),
]
)
@@ -56,7 +57,7 @@ def _restore_model(self, **kwargs) -> MegatronTokenizerTextAbstract:
library_class = getattr(tokenizers, TOKENIZER_MAPPING_LIBRARIES[self.library])
- if self.library in ['byte-level', 'null']:
+ if self.library in ['byte-level', 'null-text']:
return library_class(**kwargs)
else:
return library_class(self.path, **kwargs)
@@ -82,7 +83,7 @@ def detokenize(self, ids: List[int]) -> str:
ids (list): text to be tokenized.
Returns:
- text: dettokenized text.
+ text: detokenized text.
"""
return self._tokenizer.ids_to_text(ids)
@@ -112,6 +113,32 @@ def apply_chat_template(
conversation=conversation, chat_template=chat_template, **kwargs
)
+ def tokenize_conversation(
+ self, conversation: List[Dict], return_target: bool, add_generation_prompt: bool
+ ):
+ """Convert a conversation to tokens. Needed for SFTTokenizer.
+
+ Args:
+ conversation (List[Dict]): Sequence of system/user/assistant messages.
+ Must be in the following format:
+ [
+ {"role": "system", "content": "something"},
+ {"role": "user", "content": "something1"},
+ {"role": "assistant", "content": "something2"},
+ ]
+ return_target (bool): Return target tokens with system and assistant masked.
+ add_generation_prompt (bool): Add assistant prefix to the end.
+ """
+
+ if self.library == 'sft':
+ return self._tokenizer.tokenize_conversation(
+ conversation=conversation,
+ return_target=return_target,
+ add_generation_prompt=add_generation_prompt,
+ )
+ else:
+ raise NotImplementedError("This method is supported only for SFTTokenizer.")
+
def save_pretrained(self, path: str) -> None:
"""
Saves HF tokenizer files.
@@ -141,6 +168,20 @@ def add_special_tokens(self, special_tokens: Union[list, dict]) -> None:
self._tokenizer.add_special_tokens(special_tokens)
+ def offsets(self, ids: list[int], text: str) -> list[int]:
+ """Calculate offsets."""
+ return self._tokenizer.offsets(ids=ids, text=text)
+
+ @property
+ def space_sensitive(self):
+ """Check if tokenizer is space sensetive."""
+ if self.library in ['sentencepiece', 'huggingface']:
+ return self._tokenizer.space_sensitive
+ else:
+ raise NotImplementedError(
+ f"This method is not supported for {self.library} tokenizers."
+ )
+
@property
def additional_special_tokens_ids(self) -> list:
"""Returns a list of the additional special tokens."""
diff --git a/megatron/core/tokenizers/text/utils/build_tokenizer.py b/megatron/core/tokenizers/utils/build_tokenizer.py
similarity index 54%
rename from megatron/core/tokenizers/text/utils/build_tokenizer.py
rename to megatron/core/tokenizers/utils/build_tokenizer.py
index 2017945204d..bf02451ae6c 100644
--- a/megatron/core/tokenizers/text/utils/build_tokenizer.py
+++ b/megatron/core/tokenizers/utils/build_tokenizer.py
@@ -1,14 +1,19 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+import logging
+import math
+
from megatron.core.tokenizers import MegatronTokenizer
MEGATRON_TOKENIZERS = ['BertWordPieceLowerCase', 'BertWordPieceCase', 'GPT2BPETokenizer']
SP_TOKENIZERS = ['SentencePieceTokenizer', 'GPTSentencePieceTokenizer', 'Llama2Tokenizer']
+logger = logging.getLogger(__name__)
+
-def build_tokenizer(args):
- """ """
+def build_tokenizer(args, **kwargs):
+ """Initialize tokenizer."""
kwargs = {}
tokenizer_library = None
tokenizer_path = None
@@ -24,9 +29,9 @@ def build_tokenizer(args):
kwargs = special_tokens
kwargs['vocab_file'] = args.vocab_file
kwargs['merges_file'] = args.merge_file
- kwargs['use_fast'] = args.tokenizer_hf_use_fast
+ kwargs['use_fast'] = not args.tokenizer_hf_no_use_fast
kwargs['trust_remote_code'] = args.trust_remote_code
- kwargs['include_special_tokens'] = args.tokenizer_hf_include_special_tokens
+ kwargs['include_special_tokens'] = not args.tokenizer_hf_no_include_special_tokens
elif args.tokenizer_type in SP_TOKENIZERS:
tokenizer_library = 'sentencepiece'
tokenizer_path = args.tokenizer_model
@@ -49,16 +54,31 @@ def build_tokenizer(args):
kwargs['additional_special_tokens'] = (
args.tokenizer_special_tokens if args.tokenizer_special_tokens else []
)
- kwargs['use_fast'] = args.tokenizer_hf_use_fast
+ kwargs['use_fast'] = not args.tokenizer_hf_no_use_fast
kwargs['trust_remote_code'] = args.trust_remote_code
- kwargs['include_special_tokens'] = args.tokenizer_hf_include_special_tokens
- elif args.tokenizer_type == 'NullTokenizer':
- tokenizer_library = 'null'
+ kwargs['include_special_tokens'] = not args.tokenizer_hf_no_include_special_tokens
+ elif args.tokenizer_type == 'MultimodalTokenizer':
+ tokenizer_library = 'multimodal'
+ kwargs['prompt_format'] = args.tokenizer_prompt_format
+ kwargs['special_tokens'] = args.special_tokens
+ kwargs['image_tag_type'] = args.image_tag_type
+ kwargs['force_system_message'] = args.force_system_message
+ elif args.tokenizer_type == 'SFTTokenizer':
+ tokenizer_library = 'sft'
+ tokenizer_path = args.tokenizer_model
+ kwargs['prompt_format'] = args.sft_tokenizer_prompt_format
+ elif args.tokenizer_type in ['NullTokenizer', 'NullMultimodalTokenizer']:
+ tokenizer_library = (
+ 'null-text' if args.tokenizer_type == 'NullTokenizer' else 'null-multimodal'
+ )
metadata = {'library': tokenizer_library}
if args.vocab_size:
kwargs['vocab_size'] = args.vocab_size
tokenizer = MegatronTokenizer.from_pretrained(metadata_path=metadata, **kwargs)
+ # Add vocab size (if not already set from a checkpoint).
+ _set_padded_vocab_size(args, tokenizer)
+
return tokenizer
if args.tokenizer_metadata:
@@ -69,4 +89,29 @@ def build_tokenizer(args):
tokenizer_path=tokenizer_path, metadata_path=metadata, **kwargs
)
+ # Add vocab size (if not already set from a checkpoint).
+ _set_padded_vocab_size(args, tokenizer)
+
return tokenizer
+
+
+def vocab_size_with_padding(orig_vocab_size, args, logging_enabled=True):
+ """Pad vocab size so it is divisible by model parallel size and
+ still having GPU friendly size."""
+
+ after = orig_vocab_size
+ multiple = args.make_vocab_size_divisible_by * args.tensor_model_parallel_size
+ after = int(math.ceil(after / multiple) * multiple)
+ if args.rank == 0 and logging_enabled:
+ logger.info(
+ f' > padded vocab (size: {orig_vocab_size}) with '
+ f'{after - orig_vocab_size} dummy tokens '
+ f'(new size: {after})'
+ )
+ return after
+
+
+def _set_padded_vocab_size(args, tokenizer):
+ """Sets padded vocab size if None."""
+ if getattr(args, "padded_vocab_size", None) is None:
+ args.padded_vocab_size = vocab_size_with_padding(tokenizer.vocab_size, args)
diff --git a/megatron/core/tokenizers/vision/__init__.py b/megatron/core/tokenizers/vision/__init__.py
new file mode 100644
index 00000000000..6684427279d
--- /dev/null
+++ b/megatron/core/tokenizers/vision/__init__.py
@@ -0,0 +1,3 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+from megatron.core.tokenizers.vision.vision_tokenizer import MegatronTokenizerVision
diff --git a/megatron/core/tokenizers/vision/libraries/__init__.py b/megatron/core/tokenizers/vision/libraries/__init__.py
new file mode 100644
index 00000000000..1dd57f4b7e5
--- /dev/null
+++ b/megatron/core/tokenizers/vision/libraries/__init__.py
@@ -0,0 +1,8 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+from megatron.core.tokenizers.vision.libraries.multimodal_tokenizer import (
+ MegatronMultimodalTokenizer,
+)
+from megatron.core.tokenizers.vision.libraries.null_multimodal_tokenizer import (
+ MegatronNullMultimodalTokenizer,
+)
diff --git a/megatron/training/tokenizer/multimodal_tokenizer.py b/megatron/core/tokenizers/vision/libraries/multimodal_tokenizer.py
similarity index 85%
rename from megatron/training/tokenizer/multimodal_tokenizer.py
rename to megatron/core/tokenizers/vision/libraries/multimodal_tokenizer.py
index 47aae9e43c1..e8548ac524a 100644
--- a/megatron/training/tokenizer/multimodal_tokenizer.py
+++ b/megatron/core/tokenizers/vision/libraries/multimodal_tokenizer.py
@@ -1,16 +1,21 @@
-# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2026, Advanced Micro Devices, Inc. All rights reserved. Modified for portability.
-"""Multimodal tokenizer."""
-from dataclasses import dataclass
-from typing import Dict, List, Union, Optional
+from typing import Dict, List, Union
import numpy as np
-from megatron.core.datasets.megatron_tokenizer import MegatronLegacyTokenizer
-
-# Mark tokens that will be ignored in the loss function with this value.
-# Same ignore_index in https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html
from megatron.core.models.multimodal.llava_model import IGNORE_INDEX, IMAGE_TOKEN
+from megatron.core.tokenizers.chat_template_utils import token_ids_from_chat_template_output
+from megatron.core.tokenizers.text.libraries.sft_tokenizer import PromptConfig
+
+try:
+ import transformers
+
+ HAVE_TRANSFORMERS = True
+except (ImportError, ModuleNotFoundError):
+ HAVE_TRANSFORMERS = False
+
IMAGE_TAGS = {
"nvlm": ("", " "),
@@ -33,62 +38,55 @@
"""
-nvlm_yi_34b_template = "{{- bos_token }}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
+nvlm_yi_34b_template = "{{- bos_token }}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}" # pylint: disable=line-too-long
-qwen2p0_custom_template = "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
+qwen2p0_custom_template = "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}" # pylint: disable=line-too-long
-# Note: this is the same template as https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/tokenizer_config.json#L2053
+# Note: this is the same template as
+# https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/tokenizer_config.json#L2053
# but we removed the forced system message.
-llama3p1_chat_template = """{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = none %}\n{%- endif %}\n\n{%- if system_message is not none %}{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{%-endif %}{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n"""
-
-nemotron_custom_template = "{{- bos_token }}{% for message in messages %}{{'' + message['role'] + '\n' + message['content'] + '' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ 'assistant\n' }}{% endif %}"
+llama3p1_chat_template = """{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = none %}\n{%- endif %}\n\n{%- if system_message is not none %}{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{%-endif %}{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n""" # pylint: disable=line-too-long
-nemotron_aligned_custom_template = "{{- bos_token}}{% for message in messages %}{{message['role'] + '\n' + message['content'] + '\n' + '[PREFIX]'}}{% endfor %}{% if add_generation_prompt %}{{ 'Assistant\n' }}{% endif %}"
+nemotron_custom_template = "{{- bos_token }}{% for message in messages %}{{'' + message['role'] + '\n' + message['content'] + '' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ 'assistant\n' }}{% endif %}" # pylint: disable=line-too-long
+nemotron_aligned_custom_template = "{{- bos_token}}{% for message in messages %}{{message['role'] + '\n' + message['content'] + '\n' + '[PREFIX]'}}{% endfor %}{% if add_generation_prompt %}{{ 'Assistant\n' }}{% endif %}" # pylint: disable=line-too-long
-@dataclass
-class PromptConfig:
- """Config options for different prompt formats."""
- # How many tokens are used for the assistant prefix, e.g. "<|im_start|>assistant\n".
- # Used for masking the assistant prefix.
- assistant_prefix_len: int
- # Padding token ID.
- pad_token_id: int
- # For overriding the default chat format template.
- custom_chat_template: str
- # If the tokenizer inserts BOS token by default.
- has_bos: bool
- # If the tokenizer supports a separate role for system messages.
- has_system_role: bool
- # Wether to force a specific system message.
- force_system_message: bool = False
- system_default: dict = None
-
-
-class MultimodalTokenizer(MegatronLegacyTokenizer):
+class MegatronMultimodalTokenizer:
"""Multimodal Tokenizer."""
def __init__(
self,
- tokenizer: MegatronLegacyTokenizer,
+ path: str,
prompt_format: str,
special_tokens: List[str],
image_tag_type: str,
force_system_message: bool = False,
+ **kwargs,
):
"""Tokenizer with a support for non-text inputs.
Note: Currently, only HuggingFaceTokenizer is supported as the underlying text tokenizer.
Args:
- tokenizer (MegatronLegacyTokenizer): Underlying tokenizer.
+ path (str): Path to the underlying tokenizer.
prompt_format (str): Prompt format for the tokenizer.
special_tokens (List[str]): Non-text tokens.
image_tag_type (str): Image tag to apply, if any. For example .
"""
+ if not HAVE_TRANSFORMERS:
+ raise ImportError(
+ "MegatronMultimodalTokenizer currently requires "
+ "transformers library to be installed."
+ )
+ if prompt_format == "nvlm-yi-34b":
+ kwargs.update({"from_slow": True, "legacy": False, "add_bos_token": True})
+ tokenizer = transformers.AutoTokenizer.from_pretrained(
+ pretrained_model_name_or_path=path, **kwargs
+ )
+
self._vocab_size = len(tokenizer)
num_added_tokens = tokenizer.add_tokens(special_tokens, special_tokens=True)
@@ -96,7 +94,7 @@ def __init__(
special_tokens
), f"failed to add {len(special_tokens)} special tokens; only added {num_added_tokens}"
- self._tokenizer = tokenizer
+ self.tokenizer = tokenizer
if prompt_format == "mistral":
# Mistral format doesn't have prefix for the assistant message.
@@ -170,8 +168,10 @@ def __init__(
has_bos=False,
has_system_role=True,
force_system_message=force_system_message,
- system_default={"role": "system",
- "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}
+ system_default={
+ "role": "system",
+ "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.", # pylint: disable=line-too-long
+ },
)
elif prompt_format == "llama3p1":
# "<|start_header_id|>assistant<|end_header|>\n\n" is the prefix for assistant messages.
@@ -215,7 +215,7 @@ def tokenize(self, text: Union[str, List[Dict]]):
def _encode(self, text: str):
"""Tokenize text input."""
text = self._apply_image_tag(text)
- return self._tokenizer.encode(text)
+ return self.tokenizer.encode(text)
def tokenize_conversation(
self, conversation: List[Dict], return_target: bool, add_generation_prompt: bool
@@ -237,7 +237,9 @@ def tokenize_conversation(
conversation = conversation[1:]
if self._prompt_config.force_system_message:
- assert self._prompt_config.system_default is not None, "Trying to force system message with empty system default"
+ assert (
+ self._prompt_config.system_default is not None
+ ), "Trying to force system message with empty system default"
if conversation[0]["role"] == "system":
conversation[0] = self._prompt_config.system_default
else:
@@ -251,14 +253,15 @@ def tokenize_conversation(
# Apply possible image tag.
conversation = self._apply_image_tag(conversation)
- tokens = self._tokenizer.apply_chat_template(
+ raw = self.tokenizer.apply_chat_template(
conversation,
tokenize=True,
add_generation_prompt=add_generation_prompt,
return_assistant_token_mask=False,
return_tensors="np",
chat_template=self._prompt_config.custom_chat_template,
- )[0]
+ )
+ tokens = token_ids_from_chat_template_output(raw)
if not return_target:
return tokens
@@ -271,9 +274,10 @@ def tokenize_conversation(
if len(turn["content"]) == 0:
raise ValueError(f"empty turn in conversation: {conversation}. Skipping.")
- turn_tokens = self._tokenizer.apply_chat_template(
+ turn_raw = self.tokenizer.apply_chat_template(
[turn], tokenize=True, chat_template=self._prompt_config.custom_chat_template
)
+ turn_tokens = token_ids_from_chat_template_output(turn_raw)
# There should be only one BOS at the very beginning.
# After the first turn, skip BOS token.
@@ -304,15 +308,19 @@ def tokenize_conversation(
def convert_tokens_to_ids(self, tokens: List[str]):
"""Convert tokens to IDs."""
- return self._tokenizer.convert_tokens_to_ids(tokens)
+ return self.tokenizer.convert_tokens_to_ids(tokens)
def detokenize(self, tokens: List[int]):
"""Detokenize tokens."""
- return self._tokenizer.decode(tokens)
+ return self.tokenizer.decode(tokens)
+
+ def add_special_tokens(self, special_tokens: List[str]):
+ """Add special tokens."""
+ self.tokenizer.add_tokens(special_tokens, special_tokens=True)
def get_special_tokens(self):
"""Get special tokens."""
- return self._tokenizer.get_added_vocab()
+ return self.tokenizer.get_added_vocab()
@property
def pad(self):
@@ -322,19 +330,14 @@ def pad(self):
@property
def eod(self):
"""End of sentence token ID."""
- return self._tokenizer.eos_token_id
-
- @property
- def vocab(self):
- """Vocab."""
- return NotImplementedError("not used")
-
- @property
- def inv_vocab(self):
- """Inverse vocab."""
- return NotImplementedError("not used")
+ return self.tokenizer.eos_token_id
@property
def vocab_size(self):
"""Vocabulary size."""
return self._vocab_size
+
+ @property
+ def vocab(self):
+ """Tokenizer vocab."""
+ return self.tokenizer.get_vocab()
diff --git a/megatron/core/tokenizers/vision/libraries/null_multimodal_tokenizer.py b/megatron/core/tokenizers/vision/libraries/null_multimodal_tokenizer.py
new file mode 100644
index 00000000000..9c64c409bb2
--- /dev/null
+++ b/megatron/core/tokenizers/vision/libraries/null_multimodal_tokenizer.py
@@ -0,0 +1,90 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+
+class MegatronNullMultimodalTokenizer:
+ """Megatron Null Multimodal Tokenizer"""
+
+ def __init__(self, vocab_size, image_token=None, image_token_id=None):
+ """ """
+ self._vocab_size_without_eod = int(vocab_size)
+ self._eod_id = self._vocab_size_without_eod
+
+ from megatron.core.models.multimodal.llava_model import (
+ DEFAULT_IMAGE_TOKEN_INDEX,
+ IMAGE_TOKEN,
+ )
+
+ self._image_token = image_token if image_token is not None else IMAGE_TOKEN
+ self._image_token_id = (
+ image_token_id if image_token_id is not None else DEFAULT_IMAGE_TOKEN_INDEX
+ )
+
+ def tokenize(self, text):
+ """
+ Text tokenization.
+
+ Args:
+ text (str | list): text to be tokenized.
+
+ Returns:
+ list: list of ids.
+ """
+ return [int(x) for x in text.split(' ')]
+
+ def detokenize(self, ids):
+ """
+ Text detokenization.
+
+ Args:
+ ids (list): text to be tokenized.
+
+ Returns:
+ text: detokenized text.
+ """
+ text = [str(x) for x in ids]
+ return ' '.join(text)
+
+ def offsets(self, ids: list[int], text: str) -> list[int]:
+ """Offsets calculation."""
+ offsets, start_idx = [], 0
+ for id_ in ids:
+ offsets.append(start_idx)
+ start_idx += 1 + len(str(id_))
+ return offsets
+
+ def convert_tokens_to_ids(self, tokens):
+ """Convert tokens to IDs."""
+ ids = [
+ (int(t) if t != self._image_token else self._image_token_id) for t in tokens.split(' ')
+ ]
+ return ids if len(ids) > 1 else ids[0]
+
+ @property
+ def vocab_size(self):
+ """Vocab size."""
+ return self._vocab_size_without_eod + 1
+
+ @property
+ def cls(self):
+ """CLS token id."""
+ return -1
+
+ @property
+ def sep(self):
+ """SEP token id."""
+ return -1
+
+ @property
+ def mask(self):
+ """MASK token id."""
+ return -1
+
+ @property
+ def eod(self):
+ """EOD token id."""
+ return self._eod_id
+
+ @property
+ def additional_special_tokens_ids(self):
+ """Returns IDs of additional special tokens."""
+ return None
diff --git a/megatron/core/tokenizers/vision/models/__init__.py b/megatron/core/tokenizers/vision/models/__init__.py
new file mode 100644
index 00000000000..b9387786024
--- /dev/null
+++ b/megatron/core/tokenizers/vision/models/__init__.py
@@ -0,0 +1,3 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+from megatron.core.tokenizers.vision.models.default_tokenizer import DefaultTokenizerVision
diff --git a/megatron/core/tokenizers/vision/models/default_tokenizer.py b/megatron/core/tokenizers/vision/models/default_tokenizer.py
new file mode 100644
index 00000000000..5366820727d
--- /dev/null
+++ b/megatron/core/tokenizers/vision/models/default_tokenizer.py
@@ -0,0 +1,12 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+
+from megatron.core.tokenizers.vision.vision_tokenizer import MegatronTokenizerVision
+
+
+class DefaultTokenizerVision(MegatronTokenizerVision):
+ """Base class for Megatron default vision tokenizer."""
+
+ def __init__(self, path: str = None, config: dict = None, **kwargs) -> None:
+ config['class_name'] = self.__class__.__name__
+ config['class_path'] = self.__class__.__module__
+ super().__init__(path, config, **kwargs)
diff --git a/megatron/core/tokenizers/vision/vision_tokenizer.py b/megatron/core/tokenizers/vision/vision_tokenizer.py
new file mode 100644
index 00000000000..5e1769116a6
--- /dev/null
+++ b/megatron/core/tokenizers/vision/vision_tokenizer.py
@@ -0,0 +1,142 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+from collections import OrderedDict
+from typing import Dict, List, Union
+
+from megatron.core.tokenizers.base_tokenizer import MegatronTokenizerBase
+
+TOKENIZER_MAPPING_LIBRARIES = OrderedDict(
+ [
+ ("multimodal", "MegatronMultimodalTokenizer"),
+ ("null-multimodal", "MegatronNullMultimodalTokenizer"),
+ ]
+)
+
+
+class MegatronTokenizerVision(MegatronTokenizerBase):
+ """Base class for Megatron vision tokenizers."""
+
+ def __init__(self, path: str, config: dict, **kwargs) -> None:
+ """
+ Args:
+ path (str): path to the tokenizer model.
+ config (dict): tokenizer parameters.
+ library (str): tokenizer library.
+ class_name (str): name of tokenizer class.
+ class_path (str): path to tokenizer class.
+ model_type (str): type of the model to be used with tokenizer.
+ """
+
+ super().__init__(path, config, **kwargs)
+ self._tokenizer = self._restore_model(**kwargs)
+ self.path = path
+
+ def _restore_model(self, **kwargs):
+ """Returns tokenizer library object."""
+
+ import megatron.core.tokenizers.vision.libraries as tokenizers
+
+ library_class = getattr(tokenizers, TOKENIZER_MAPPING_LIBRARIES[self.library])
+
+ if self.library in ['null-multimodal']:
+ return library_class(**kwargs)
+ else:
+ return library_class(self.path, **kwargs)
+
+ def tokenize(self, text: Union[str, List[Dict]]) -> List[int]:
+ """
+ Text tokenization.
+
+ Args:
+ text (str | list): text to be tokenized.
+
+ Returns:
+ list: list of ids.
+ """
+
+ return self._tokenizer.tokenize(text)
+
+ def detokenize(self, ids: List[int]) -> str:
+ """
+ Text detokenization.
+
+ Args:
+ ids (list): text to be tokenized.
+
+ Returns:
+ text: detokenized text.
+ """
+
+ return self._tokenizer.detokenize(ids)
+
+ def tokenize_conversation(
+ self, conversation: List[Dict], return_target: bool, add_generation_prompt: bool
+ ):
+ """Convert a conversation to tokens.
+
+ Args:
+ conversation (List[Dict]): Sequence of system/user/assistant messages.
+ Must be in the following format:
+ [
+ {"role": "user", "content": "something"},
+ {"role": "assistant", "content": "something2"},
+ ]
+ return_target (bool): Return target tokens with system and assistant masked.
+ add_generation_prompt (bool): Add assistant prefix to the end.
+ """
+
+ return self._tokenizer.tokenize_conversation(
+ conversation=conversation,
+ return_target=return_target,
+ add_generation_prompt=add_generation_prompt,
+ )
+
+ def add_special_tokens(self, special_tokens: Union[list, dict]) -> None:
+ """
+ Adds a dictionary of special tokens (eos, pad, cls...).
+ Tokens are only added if they are not already in the vocabulary.
+ Indexed starting from the last index of the current vocabulary.
+
+ Args:
+ special_tokens_dict: dict of string. Keys should be in the list of predefined
+ special attributes: [``bos_token``, ``eos_token``, ``unk_token``, ``sep_token``,
+ ``pad_token``, ``cls_token``, ``mask_token``, ``additional_special_tokens``].
+ """
+
+ self._tokenizer.add_special_tokens(special_tokens)
+
+ def convert_tokens_to_ids(self, tokens: List[str]):
+ """Convert tokens to IDs."""
+ return self._tokenizer.convert_tokens_to_ids(tokens)
+
+ def apply_chat_template(self):
+ """Applies tokenizer's chat template."""
+ raise NotImplementedError("This method is not supported for vision tokenizers.")
+
+ def get_special_tokens(self) -> list:
+ """Returns a list of the additional special tokens."""
+ return self._tokenizer.get_special_tokens()
+
+ def offsets(self, ids: list[int], text: str) -> list[int]:
+ """Calculate offsets."""
+ return self._tokenizer.offsets(ids=ids, text=text)
+
+ @property
+ def vocab(self):
+ """Tokenizer vocab."""
+ return self._tokenizer.vocab
+
+ @property
+ def vocab_size(self) -> int:
+ """Returns vocabulary size."""
+ return self._tokenizer.vocab_size
+
+ @property
+ def pad(self):
+ """Pad token ID."""
+ return self._tokenizer.pad
+
+ @property
+ def eod(self):
+ """End of sentence token ID."""
+ return self._tokenizer.eod
diff --git a/megatron/core/transformer/attention.py b/megatron/core/transformer/attention.py
index cf9b6e2c0ca..d7c04d001b4 100644
--- a/megatron/core/transformer/attention.py
+++ b/megatron/core/transformer/attention.py
@@ -24,6 +24,7 @@
from torch import Tensor
from megatron.core import tensor_parallel
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.inference.contexts import BaseInferenceContext
from megatron.core.jit import jit_fuser
from megatron.core.models.common.embeddings.rope_utils import (
@@ -47,6 +48,7 @@
from megatron.core.transformer.identity_op import IdentityOp
from megatron.core.transformer.module import MegatronModule
from megatron.core.transformer.spec_utils import ModuleSpec, build_module
+from megatron.core.transformer.torch_norm import LayerNormBuilder
from megatron.core.typed_torch import apply_module, not_none
from megatron.core.utils import (
deprecate_inference_params,
@@ -109,17 +111,13 @@
flash_attn_varlen_func = None
flash_attn_with_kvcache = None
-try:
- import transformer_engine # pylint: disable=unused-import
-
- HAVE_TE = True
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import (
SplitAlongDim,
TELinear,
set_save_original_input,
)
-except ImportError:
- HAVE_TE = False
+else:
SplitAlongDim, TELinear, set_save_original_input = None, None, None
try:
@@ -232,8 +230,8 @@ class SelfAttentionSubmodules:
linear_qkv: LinearQkvBuilder
core_attention: CoreAttentionBuilder
linear_proj: Union[ModuleSpec, type] = None
- q_layernorm: Union[ModuleSpec, type] = None
- k_layernorm: Union[ModuleSpec, type] = None
+ q_layernorm: LayerNormBuilder | None = None
+ k_layernorm: LayerNormBuilder | None = None
linear_q_proj: Union[ModuleSpec, type] = None
linear_q_down_proj: Union[ModuleSpec, type] = None
@@ -271,11 +269,13 @@ def __init__(
attention_type: str,
cp_comm_type: str | None = None,
pg_collection: ProcessGroupCollection | None = None,
+ pp_layer_offset: Optional[int] = None,
):
super().__init__(config=config)
self.config = config
self.layer_number = layer_number
+ self._pp_layer_offset = pp_layer_offset
self.attn_mask_type = attn_mask_type
self.attention_type = attention_type
@@ -451,7 +451,16 @@ def _allocate_memory(self, inference_max_sequence_length, batch_size, dim, dtype
)
def _get_pp_layer_offset_for_inference(self):
- """Return the pipeline parallel layer offset for inference."""
+ """Return the pipeline parallel layer offset for inference.
+
+ When pp_layer_offset was explicitly provided (e.g. by MambaBlock for
+ hybrid models using --hybrid-layer-pattern with fVPP), use that value
+ directly. Otherwise fall back to the standard computation which assumes
+ uniform layer distribution across pipeline stages.
+ """
+ if self._pp_layer_offset is not None:
+ return self._pp_layer_offset
+
assert (
self.config.virtual_pipeline_model_parallel_size is None
), "Virtual pipeline parallelism is not supported for inference"
@@ -807,7 +816,7 @@ def flash_decode_and_prefill(
assert block_table is not None
# Flash attn kernel.
- if not is_decode_only:
+ if max_seqlen_q > 1:
q = q.squeeze(1)
if getattr(self, "softmax_scale", None) is not None:
softmax_scale = self.softmax_scale
@@ -1270,6 +1279,7 @@ def __init__(
attn_mask_type: AttnMaskType = AttnMaskType.padding,
cp_comm_type: str | None = None,
pg_collection: ProcessGroupCollection | None = None,
+ pp_layer_offset: Optional[int] = None,
):
super().__init__(
config=config,
@@ -1279,6 +1289,7 @@ def __init__(
attention_type="self",
cp_comm_type=cp_comm_type,
pg_collection=pg_collection,
+ pp_layer_offset=pp_layer_offset,
)
self.linear_qkv_out_dim = self.query_projection_size + 2 * self.kv_projection_size
@@ -1298,8 +1309,7 @@ def __init__(
)
if submodules.q_layernorm is not None:
- self.q_layernorm = build_module(
- submodules.q_layernorm,
+ self.q_layernorm = submodules.q_layernorm(
hidden_size=self.hidden_size_per_attention_head,
config=self.config,
eps=self.config.layernorm_epsilon,
@@ -1308,8 +1318,7 @@ def __init__(
self.q_layernorm = None
if submodules.k_layernorm is not None:
- self.k_layernorm = build_module(
- submodules.k_layernorm,
+ self.k_layernorm = submodules.k_layernorm(
hidden_size=self.hidden_size_per_attention_head,
config=self.config,
eps=self.config.layernorm_epsilon,
@@ -1428,8 +1437,10 @@ def get_query_key_value_tensors(
# 4. Further index into query to get only the q_heads that this rank is
# responsible for (e.g., q1).
# The block of code below performs steps 1 and 2.
- mixed_qkv = all_gather_last_dim_from_tensor_parallel_region(mixed_qkv)
- idx = get_tensor_model_parallel_rank() // (
+ mixed_qkv = all_gather_last_dim_from_tensor_parallel_region(
+ mixed_qkv, group=self.pg_collection.tp
+ )
+ idx = get_pg_rank(self.pg_collection.tp) // (
self.world_size // self.config.num_query_groups
)
size = mixed_qkv.size()[-1] // self.config.num_query_groups
@@ -1486,7 +1497,7 @@ def get_query_key_value_tensors(
# query above corresponds to (num_q_heads / num_kv_heads) q_heads.
# Index appropriately into query to get (num_q_heads / tp_size) q_heads.
# This is step 4 in the list of steps above.
- idx = get_tensor_model_parallel_rank() % (
+ idx = get_pg_rank(self.pg_collection.tp) % (
self.world_size // self.config.num_query_groups
)
size = self.num_attention_heads_per_partition // (
@@ -1495,10 +1506,10 @@ def get_query_key_value_tensors(
query = query[:, :, idx * size : (idx + 1) * size, :]
if self.q_layernorm is not None:
- query = self.q_layernorm(query)
+ query = apply_module(self.q_layernorm)(query)
if self.k_layernorm is not None:
- key = self.k_layernorm(key)
+ key = apply_module(self.k_layernorm)(key)
if self.config.test_mode:
self.run_realtime_tests()
@@ -1506,6 +1517,14 @@ def get_query_key_value_tensors(
if output_gate:
# Gate [sq, b, ng, np/ng * hn] -> [sq, b, np, hn]
gate = gate.reshape(*gate.shape[:2], -1, self.hidden_size_per_attention_head)
+ if self.config.num_query_groups < self.world_size:
+ idx = get_tensor_model_parallel_rank() % (
+ self.world_size // self.config.num_query_groups
+ )
+ size = self.num_attention_heads_per_partition // (
+ self.world_size // self.config.num_query_groups
+ )
+ gate = gate[:, :, idx * size : (idx + 1) * size, :]
return query, key, value, gate
return query, key, value
diff --git a/megatron/core/transformer/cuda_graphs.py b/megatron/core/transformer/cuda_graphs.py
index 921da4305f5..f5f6bd4fdeb 100644
--- a/megatron/core/transformer/cuda_graphs.py
+++ b/megatron/core/transformer/cuda_graphs.py
@@ -117,6 +117,7 @@ def _set_warmup_start():
def _set_warmup_end():
"""Set graph warmup has ended."""
global _IS_GRAPH_WARMUP
+ _IS_GRAPH_WARMUP = False
@dataclass
@@ -247,7 +248,7 @@ def _check_supported_type(meta):
}
assert meta.type in _SUPPORTED_TYPES or is_dataclass(
meta.value
- ), f"Cudagraphs recieved an arg of type {meta.type} which is not supported."
+ ), f"Cudagraphs received an arg of type {meta.type} which is not supported."
def _determine_if_first_last_layer_of_this_vp_chunk(base_module):
@@ -589,7 +590,8 @@ def forward(ctx, runner, is_first_microbatch, *inputs):
need_copy_inputs.append(user_input)
assert user_input.data_ptr() == cudagraph_input.data_ptr()
else:
- cudagraph_input.copy_(user_input)
+ if user_input.data_ptr() != cudagraph_input.data_ptr():
+ cudagraph_input.copy_(user_input)
ctx.runner = runner
ctx.save_for_backward(*need_copy_inputs)
@@ -933,6 +935,7 @@ def clone_ten(ten):
only_inputs=True,
allow_unused=True,
)
+
_set_warmup_end()
with self.get_quantization_context():
@@ -1133,19 +1136,41 @@ def replace_with_weak_ref(arg):
if not torch.is_tensor(arg):
return arg
- ref = make_weak_ref(arg)
+ try:
+ ref = make_weak_ref(arg)
+ except RuntimeError:
+ # Fallback to keeping a strong reference. There is a known bug where some
+ # dtypes (e.g. torch.float64) are not mapped to a representation in
+ # transformer_engine/pytorch/utils.py.
+ if torch.distributed.get_rank() == 0:
+ logger.warning(
+ f"Could not create weak ref for tensor with dtype {arg.dtype}; "
+ f"keeping strong ref with a potential memory overhead."
+ )
+ return arg
ref.requires_grad = arg.requires_grad
if hasattr(arg, "can_skip_replay_copy"):
ref.can_skip_replay_copy = arg.can_skip_replay_copy
return ref
+ # Weak refs replace tensors with raw-pointer wrappers that do not hold a storage
+ # reference. This is safe for surfaces whose memory is managed by the CUDA graph pool
+ # (driver-pinned, stable addresses) but not safe for tensors allocated by the caching
+ # allocator, whose data_ptr() may be invalidated by block coalescing or empty_cache().
+ def replace_with_weak_ref_for_input_surface(arg):
+ if torch.is_tensor(arg) and _CudagraphGlobalRecord.tensor_reuse_pool.owns(arg):
+ return arg
+ return replace_with_weak_ref(arg)
+
self.fwd_graph_input_surface = tree_map(
- replace_with_weak_ref, self.fwd_graph_input_surface
+ replace_with_weak_ref_for_input_surface, self.fwd_graph_input_surface
)
+
self.fwd_graph_input_args = tree_map(replace_with_weak_ref, self.fwd_graph_input_args)
self.fwd_graph_input_kwargs = tree_map(
replace_with_weak_ref, self.fwd_graph_input_kwargs
)
+ # Outputs can be weakref'd as they are managed by the graph pool
self.fwd_graph_output_surface = tree_map(
replace_with_weak_ref, self.fwd_graph_output_surface
)
@@ -1272,17 +1297,19 @@ def check(val, ref, context):
_check_supported_type(ref)
if val.type != ref.type and not (is_dataclass(val.value) and is_dataclass(ref.value)):
- add_error(f"Type mismatch at {context}: {val.type} vs {ref.type}")
+ add_error(
+ f"Type mismatch at {context}: Received {val.type} but expected {ref.type}"
+ )
return False
if ref.type == torch.Tensor or issubclass(ref.type, torch.Tensor):
mismatches = []
if val.shape != ref.shape:
- mismatches.append(f"expected shape {val.shape} vs. {ref.shape}")
+ mismatches.append(f"Received shape {val.shape} but expected {ref.shape}")
if val.dtype != ref.dtype:
- mismatches.append(f"expected dtype {val.dtype} vs. {ref.dtype}")
+ mismatches.append(f"Received dtype {val.dtype} but expected {ref.dtype}")
if val.device != ref.device:
- mismatches.append(f"expected device {val.device} vs. {ref.device}")
+ mismatches.append(f"Received device {val.device} but expected {ref.device}")
if mismatches:
add_error(f"Tensor mismatch at {context}: {', '.join(mismatches)}")
@@ -1560,7 +1587,6 @@ def __call__(self, megatron_module, args, kwargs):
if is_inference_mode:
# Inference generation mode creates graphs immediately
runner = self.get_cudagraph_runner(megatron_module, args, kwargs, True)
- runner.eval()
if not runner.fwd_graph_recorded:
# Reuse graph input-output buffers for inference
@@ -1591,6 +1617,7 @@ def __call__(self, megatron_module, args, kwargs):
)
runner.fwd_graph_recorded = True
runner.cudagraph_created = True
+ runner = runner.eval()
# Record this to the global execution record
_CudagraphGlobalRecord.cudagraph_inference_record.append(
@@ -1706,14 +1733,24 @@ def __init__(self, model, config, seq_length, micro_batch_size, optimizers=[]):
# Number of microbatches to capture. The value will be set in _get_cuda_graph_input_data().
self.num_microbatches = None
- # Get callables with captureable layers.
+ self._discover_layers()
+
+ # Flags to track CUDA Graph state:
+ # - _capture_finished: Whether create_cudagraphs() has been called (used by training loop)
+ # - _graphs_created: Whether any graphs were actually created (may be False if no
+ # layers found)
+ self._capture_finished = False
+ self._graphs_created = False
+
+ def _discover_layers(self):
+ """Discover captureable layers from the model and populate internal data structures."""
self.chunks_with_decoder = []
self.num_layers_per_chunk = []
self.callables_per_chunk = []
self.callables_per_chunk_is_mtp = []
self.flattened_callables = []
self.flattened_callables_is_mtp = []
- for chunk_number, model_chunk in enumerate(model):
+ for chunk_number, model_chunk in enumerate(self.model):
try:
chunk_with_decoder = get_attr_wrapped_model(
model_chunk, 'decoder', allow_none=False, return_model_obj=True
@@ -1738,13 +1775,13 @@ def __init__(self, model, config, seq_length, micro_batch_size, optimizers=[]):
callables, callables_is_mtp = [], []
for layer_number in range(num_decoder_layers):
layer = chunk_with_decoder.decoder.layers[layer_number]
- if _layer_is_graphable(layer, config):
+ if _layer_is_graphable(layer, self.config):
num_graphable_layers += 1
callables.append(layer)
callables_is_mtp.append(False)
for layer_number in range(num_mtp_layers):
- layer = chunk_with_decoder.mtp.layers[layer_number].transformer_layer
- if _layer_is_graphable(layer, config):
+ layer = chunk_with_decoder.mtp.layers[layer_number].mtp_model_layer
+ if _layer_is_graphable(layer, self.config):
num_graphable_layers += 1
callables.append(layer)
callables_is_mtp.append(True)
@@ -1780,13 +1817,22 @@ def __init__(self, model, config, seq_length, micro_batch_size, optimizers=[]):
f'{len(self.flattened_callables)} graphable layers.',
)
- # One helper object can only capture CUDA Graphs once. Use this flag to check if the graphs
- # have been created.
- self._graphs_created = False
+ def capture_finished(self):
+ """
+ Returns whether create_cudagraphs() has been called.
+
+ This is used by the training loop to determine if the capture process has run.
+ Returns True after create_cudagraphs() completes, regardless of whether any
+ graphs were actually created.
+ """
+ return self._capture_finished
def graphs_created(self):
"""
- Returns whether the CUDA Graphs have been created.
+ Returns whether any CUDA Graphs were actually created.
+
+ This returns False if create_cudagraphs() was called but no graphable layers
+ were found, and True if at least one graph was successfully created.
"""
return self._graphs_created
@@ -1860,7 +1906,7 @@ def _get_layer_static_inputs(layer, chunk_of_the_layer):
Get the static inputs for a layer.
"""
assert layer in chunk_of_the_layer.decoder.layers or any(
- layer is mtp_layer.transformer_layer for mtp_layer in chunk_of_the_layer.mtp.layers
+ layer is mtp_layer.mtp_model_layer for mtp_layer in chunk_of_the_layer.mtp.layers
), "Layer is not in the chunk"
def get_rotary_pos_emb(transformer_module, transformer_input):
@@ -2188,9 +2234,9 @@ def _start_capturing(self):
"""
Start capturing CUDA Graphs.
"""
- assert not self._graphs_created, "CUDA Graphs have already been created."
+ assert not self._capture_finished, "CUDA Graph capture has already been finished."
- torch.distributed.barrier()
+ torch.cuda.synchronize()
gc.collect()
torch.cuda.empty_cache()
if FREEZE_GC:
@@ -2200,22 +2246,13 @@ def _start_capturing(self):
log_single_rank(logger, logging.INFO, f'Start CUDA Graphs capture...')
return time.time()
- def _finish_capturing(self, start_time):
+ def _reset_after_capture(self):
"""
- Finish capturing CUDA Graphs and clean up the related state.
+ Reset the model and optimizer state after capturing CUDA Graphs.
"""
- log_single_rank(
- logger,
- logging.INFO,
- f'Time spent in CUDA Graphs capture on rank {torch.distributed.get_rank()}: '
- f'{time.time() - start_time}s',
- )
- _set_capture_end()
-
from megatron.core.distributed.finalize_model_grads import reset_model_temporary_tensors
from megatron.core.transformer.moe.moe_utils import clear_aux_losses_tracker
- torch.distributed.barrier()
for model_chunk in self.model:
model_chunk.zero_grad_buffer()
for optimizer in self.optimizers:
@@ -2223,12 +2260,26 @@ def _finish_capturing(self, start_time):
clear_aux_losses_tracker()
reset_model_temporary_tensors(self.config, self.model)
+ def _finish_capturing(self, start_time):
+ """
+ Finish capturing CUDA Graphs and clean up the related state.
+ """
+ log_single_rank(
+ logger,
+ logging.INFO,
+ f'Time spent in CUDA Graphs capture on rank {torch.distributed.get_rank()}: '
+ f'{time.time() - start_time}s',
+ )
+ _set_capture_end()
+
+ torch.cuda.synchronize()
+ self._reset_after_capture()
if FREEZE_GC:
gc.unfreeze()
gc.collect()
torch.cuda.empty_cache()
- self._graphs_created = True
+ self._capture_finished = True
def create_cudagraphs(self):
"""
@@ -2236,33 +2287,44 @@ def create_cudagraphs(self):
"""
start_time = self._start_capturing()
- # Prepare CUDA Graph capturing input data and call `make_graphed_callables`.
- sample_args, kwargs = self._get_cuda_graph_input_data()
- if self.config.sequence_parallel:
- rng_context = get_cuda_rng_tracker().fork()
+ if not self.flattened_callables:
+ # Check if there are any graphable layers. If not, log a warning and skip capture,
+ # but still call _finish_capturing to ensure all ranks complete the capture phase.
+ logger.warning(
+ 'TECudaGraphHelper: No graphable layers found. Skipping CUDA graph capture.'
+ )
else:
- rng_context = nullcontext()
- with rng_context:
- graphs = make_graphed_callables(tuple(self.flattened_callables), sample_args, **kwargs)
+ # Prepare CUDA Graph capturing input data and call `make_graphed_callables`.
+ sample_args, kwargs = self._get_cuda_graph_input_data()
+ if self.config.sequence_parallel:
+ rng_context = get_cuda_rng_tracker().fork()
+ else:
+ rng_context = nullcontext()
+ with rng_context:
+ graphs = make_graphed_callables(
+ tuple(self.flattened_callables), sample_args, **kwargs
+ )
- # Push the captured graphs to the corresponding TransformerBlock.
- num_layers_accumulated = 0
- for layers in self.callables_per_chunk:
- for layer_number, layer in enumerate(layers):
- layer.cuda_graphs = []
- for batch_number in range(self.num_microbatches):
- if self.config.overlap_moe_expert_parallel_comm:
- graph_idx = (
- num_layers_accumulated + layer_number
- ) * self.num_microbatches + batch_number
- else:
- graph_idx = (
- num_layers_accumulated * self.num_microbatches
- + batch_number * len(layers)
- + layer_number
- )
- layer.cuda_graphs.append(graphs[graph_idx])
- num_layers_accumulated += len(layers)
+ # Push the captured graphs to the corresponding TransformerBlock.
+ num_layers_accumulated = 0
+ for layers in self.callables_per_chunk:
+ for layer_number, layer in enumerate(layers):
+ layer.cuda_graphs = []
+ for batch_number in range(self.num_microbatches):
+ if self.config.overlap_moe_expert_parallel_comm:
+ graph_idx = (
+ num_layers_accumulated + layer_number
+ ) * self.num_microbatches + batch_number
+ else:
+ graph_idx = (
+ num_layers_accumulated * self.num_microbatches
+ + batch_number * len(layers)
+ + layer_number
+ )
+ layer.cuda_graphs.append(graphs[graph_idx])
+ num_layers_accumulated += len(layers)
+
+ self._graphs_created = True
self._finish_capturing(start_time)
@@ -2280,7 +2342,7 @@ def delete_cuda_graphs(self):
"""
Delete all CUDA graphs.
"""
- assert self._graphs_created, "CUDA Graphs have not been created."
+ assert self._graphs_created, "No CUDA Graphs were created to delete."
graph_resettable = is_te_min_version("2.10.0")
graphs_reset, graphs_not_reset = 0, 0
@@ -2435,3 +2497,283 @@ def get_layer_range(c_id):
add_order(c_id, l_b, is_wgrad=True)
return new_order, chunk_id_list
+
+
+# ---------------------------------------------------------------------------
+# set_current_microbatch: sets per-layer microbatch index for TE graph replay
+# ---------------------------------------------------------------------------
+
+
+def set_current_microbatch(model, microbatch_id):
+ """Set the current microbatch on all layers that use TE CUDA graph replay.
+
+ current_microbatch is read by _te_cuda_graph_replay to select the
+ correct graph index. This helper is called from the pipeline-parallel
+ schedule before each forward step.
+ """
+ decoder_exists = True
+ model_with_decoder = None
+ try:
+ model_with_decoder = get_attr_wrapped_model(
+ model, "decoder", allow_none=False, return_model_obj=True
+ )
+ except RuntimeError:
+ decoder_exists = False
+ if decoder_exists and model_with_decoder is not None:
+ for layer in model_with_decoder.decoder.layers:
+ layer.current_microbatch = microbatch_id
+ if hasattr(model_with_decoder, 'mtp'):
+ for layer in model_with_decoder.mtp.layers:
+ assert hasattr(
+ layer, 'mtp_model_layer'
+ ), f"MTP layer {layer} must have 'mtp_model_layer' attribute"
+ layer.mtp_model_layer.current_microbatch = microbatch_id
+
+ # Also set current_microbatch on vision encoder layers so that
+ # _te_cuda_graph_replay selects the correct graph index. Without this,
+ # vision layers always use graph 0 (since current_microbatch defaults to 0),
+ # causing all microbatch forwards to overwrite the same static buffers.
+ # When backward runs for earlier microbatches, the buffers contain stale
+ # data from later forwards, producing NaN gradients.
+ try:
+ model_with_vision = get_attr_wrapped_model(
+ model, "vision_model", allow_none=True, return_model_obj=True
+ )
+ except RuntimeError:
+ model_with_vision = None
+ if model_with_vision is not None and hasattr(model_with_vision, 'vision_model'):
+ vision_model = model_with_vision.vision_model
+ if hasattr(vision_model, 'decoder') and hasattr(vision_model.decoder, 'layers'):
+ for layer in vision_model.decoder.layers:
+ layer.current_microbatch = microbatch_id
+
+
+# ---------------------------------------------------------------------------
+# Vision encoder CUDA graph helpers
+# ---------------------------------------------------------------------------
+
+
+def _wrap_graph_for_vision(graph_fn):
+ """Wrap a graphed callable to filter out None outputs.
+
+ During make_graphed_callables warmup, vision encoder layers go through their
+ normal forward() path which returns (output, context=None). _te_cuda_graph_replay
+ asserts len(output) == 1 but gets 2 elements. This wrapper filters out None
+ values so replay sees (output,) instead of (output, None).
+ """
+
+ def wrapped(*args, **kwargs):
+ result = graph_fn(*args, **kwargs)
+ if isinstance(result, tuple):
+ filtered = tuple(r for r in result if r is not None)
+ return filtered if filtered else result
+ return result
+
+ for attr in ('backward_dw', 'reset'):
+ if hasattr(graph_fn, attr):
+ setattr(wrapped, attr, getattr(graph_fn, attr))
+ return wrapped
+
+
+def get_vision_cuda_graph_seq_length(vision_config, default_seq_length: int = 4096) -> int:
+ """Calculate the sequence length for vision encoder CUDA graphs.
+
+ For vision encoders, the sequence length depends on:
+ - max_vision_cuda_graph_seq_length: explicit maximum (if set)
+ - num_position_embeddings: maximum number of patches
+ - spatial_merge_size: pooling factor that reduces sequence length
+
+ Args:
+ vision_config: The TransformerConfig for vision encoder
+ default_seq_length: Default sequence length if cannot be calculated
+
+ Returns:
+ The sequence length to use for CUDA graph capture
+ """
+ if (
+ hasattr(vision_config, 'max_vision_cuda_graph_seq_length')
+ and vision_config.max_vision_cuda_graph_seq_length
+ ):
+ return vision_config.max_vision_cuda_graph_seq_length
+
+ if hasattr(vision_config, 'num_position_embeddings'):
+ seq_length = vision_config.num_position_embeddings
+ if hasattr(vision_config, 'spatial_merge_size'):
+ merge_factor = vision_config.spatial_merge_size**2
+ seq_length = seq_length // merge_factor
+ return seq_length
+
+ return default_seq_length
+
+
+class VisionTECudaGraphHelper(TECudaGraphHelper):
+ """Helper to capture CUDA Graphs for vision encoder layers using TE.
+
+ Inherits from TECudaGraphHelper and overrides only the
+ vision-specific behaviour:
+
+ * Layer discovery finds vision_model.decoder.layers instead of the
+ language decoder layers.
+ * num_model_chunks is always 1 (vision has no virtual pipeline stages).
+ * Batch dimension is always 1 (images are concatenated along the sequence
+ dimension).
+ * Sample argument generation uses a simple loop (no rotary embeddings or
+ buffer-reuse optimization).
+ * _finish_capturing wraps captured graphs to filter None values that arise
+ from vision encoder layers returning (output, None), and skips cleanup
+ that is handled by the LM decoder helper.
+
+ Note:
+ With pipeline parallelism > 1, only the first pipeline stage typically
+ has vision layers. Ranks without vision layers can safely skip calling
+ create_cudagraphs() or will gracefully return with no graphs created.
+
+ Args:
+ model: The full model (list of model chunks) containing vision_model.
+ vision_config: TransformerConfig for the vision encoder.
+ vision_seq_length: Sequence length for vision (max vision tokens).
+ micro_batch_size: Micro-batch size (unused for sample-arg generation
+ since the vision encoder always uses batch-dim = 1).
+ num_microbatches: Number of microbatches per step.
+ """
+
+ def __init__(
+ self,
+ model,
+ vision_config,
+ vision_seq_length: int,
+ micro_batch_size: int,
+ num_microbatches: int = 1,
+ ):
+ super().__init__(model, vision_config, vision_seq_length, micro_batch_size)
+ # Vision encoder concatenates all images along the sequence dimension
+ # with a fixed batch dimension of 1, regardless of the training MBS.
+ self.micro_batch_size = 1
+ self.num_model_chunks = 1
+ self.num_microbatches = num_microbatches
+
+ def _discover_layers(self):
+ """Discover captureable layers from the vision encoder."""
+ self.vision_model = None
+ vision_layers = []
+
+ for model_chunk in self.model:
+ try:
+ unwrapped = get_attr_wrapped_model(
+ model_chunk, 'vision_model', allow_none=True, return_model_obj=True
+ )
+ if unwrapped is not None and hasattr(unwrapped, 'vision_model'):
+ self.vision_model = unwrapped.vision_model
+ break
+ except (RuntimeError, AttributeError):
+ continue
+
+ if self.vision_model is not None:
+ if hasattr(self.vision_model, 'decoder') and hasattr(
+ self.vision_model.decoder, 'layers'
+ ):
+ for layer in self.vision_model.decoder.layers:
+ if _layer_is_graphable(layer, self.config):
+ vision_layers.append(layer)
+
+ if vision_layers:
+ self.chunks_with_decoder = [self.vision_model]
+ self.num_layers_per_chunk = [len(vision_layers)]
+ self.callables_per_chunk = [vision_layers]
+ self.callables_per_chunk_is_mtp = [[False] * len(vision_layers)]
+ self.flattened_callables = list(vision_layers)
+ self.flattened_callables_is_mtp = [False] * len(vision_layers)
+ else:
+ if self.vision_model is None:
+ logger.warning(
+ 'VisionTECudaGraphHelper: No vision_model found in model. '
+ 'CUDA graphs will not be captured for vision encoder.'
+ )
+ self.chunks_with_decoder = [None]
+ self.num_layers_per_chunk = [0]
+ self.callables_per_chunk = [[]]
+ self.callables_per_chunk_is_mtp = [[]]
+ self.flattened_callables = []
+ self.flattened_callables_is_mtp = []
+
+ # backward-compat aliases used by callers / tests
+ self.callables = vision_layers
+ self.num_layers = len(vision_layers)
+
+ if vision_layers:
+ logger.info(
+ f'VisionTECudaGraphHelper: Found {self.num_layers} graphable vision encoder '
+ f'layers. seq_length={self.seq_length} (all images concatenated, batch_dim=1)'
+ )
+
+ def _reset_after_capture(self):
+ """
+ No-op: vision encoder layers do not require any reset:
+ - model_chunk.zero_grad_buffer() / optimizer.zero_grad() (handled
+ by the LM decoder helper's _finish_capturing which runs on all ranks).
+ - clear_aux_losses_tracker / reset_model_temporary_tensors
+ (LM-specific cleanup already handled by the LM helper).
+ """
+
+ def _finish_capturing(self, start_time):
+ """
+ Before calling super()._finish_capturing, wrap the captured graphs with
+ _wrap_graph_for_vision to filter None from (output, None) tuples so that
+ _te_cuda_graph_replay's len == 1 assertion passes.
+ """
+ # Wrap the captured graphs before finishing
+ for layer in self.flattened_callables:
+ if hasattr(layer, 'cuda_graphs'):
+ layer.cuda_graphs = [_wrap_graph_for_vision(g) for g in layer.cuda_graphs]
+
+ super()._finish_capturing(start_time)
+
+ def _get_sample_arguments(self, order, chunk_id_list=None):
+ """Generate sample arguments for vision encoder CUDA Graph capturing.
+
+ Vision uses a simple per-layer-per-microbatch loop with batch_dim=1
+ and no rotary embeddings (unlike the parent's buffer-reuse
+ optimization). The order and chunk_id_list arguments are
+ unused because vision has num_model_chunks=1 and does not need
+ the pipeline-schedule-aware buffer lifecycle tracking.
+
+ Returns:
+ Tuple of (sample_args, sample_kwargs) lists for each
+ (layer, microbatch) pair.
+ """
+ if not self.flattened_callables:
+ return [], []
+
+ sample_args = []
+ sample_kwargs_list = []
+ hidden_size = self.config.hidden_size
+
+ for _microbatch_idx in range(self.num_microbatches):
+ for layer in self.flattened_callables:
+ hidden_states = torch.zeros(
+ self.seq_length,
+ 1,
+ hidden_size,
+ dtype=torch.bfloat16,
+ device='cuda',
+ requires_grad=True,
+ )
+
+ if hasattr(layer, 'get_layer_static_inputs'):
+ static_inputs = layer.get_layer_static_inputs(self.seq_length, 1)
+ hidden_states = static_inputs.pop('hidden_states', hidden_states)
+ sample_args.append((hidden_states,))
+ sample_kwargs_list.append(static_inputs)
+ else:
+ sample_args.append((hidden_states,))
+ sample_kwargs_list.append({})
+
+ return sample_args, sample_kwargs_list
+
+ def cuda_graph_set_manual_hooks(self):
+ """No-op: vision encoder layers do not use DDP parameter-gather hooks.
+
+ The parent derives hooks from model_chunk._make_forward_pre_hook which
+ requires overlap_param_gather=True. Vision encoder parameters are not
+ distributed with the same overlap strategy, so we skip hook setup.
+ """
diff --git a/megatron/core/transformer/custom_layers/batch_invariant_kernels.py b/megatron/core/transformer/custom_layers/batch_invariant_kernels.py
index c44bbf4c8fb..6b4311fe540 100644
--- a/megatron/core/transformer/custom_layers/batch_invariant_kernels.py
+++ b/megatron/core/transformer/custom_layers/batch_invariant_kernels.py
@@ -12,8 +12,21 @@
from typing import Any, Dict, List, Optional
import torch
-import triton
-import triton.language as tl
+
+try:
+ import triton
+ import triton.language as tl
+
+ HAVE_TRITON = True
+except ImportError:
+ from unittest.mock import MagicMock
+
+ from megatron.core.utils import null_decorator
+
+ triton = MagicMock()
+ triton.jit = null_decorator
+ tl = MagicMock()
+ HAVE_TRITON = False
__all__ = [
"set_batch_invariant_mode",
diff --git a/megatron/core/transformer/custom_layers/transformer_engine.py b/megatron/core/transformer/custom_layers/transformer_engine.py
deleted file mode 100644
index 02ce9ad5a75..00000000000
--- a/megatron/core/transformer/custom_layers/transformer_engine.py
+++ /dev/null
@@ -1,12 +0,0 @@
-# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
-
-import warnings
-
-warnings.warn(
- """The 'megatron.core.transformer.custom_layers.transformer_engine'
- module is deprecated and will be removed in 0.10.0. Please use
- 'megatron.core.extensions.transformer_engine' instead.""",
- DeprecationWarning,
- stacklevel=2,
-)
-from megatron.core.extensions.transformer_engine import *
diff --git a/megatron/core/transformer/enums.py b/megatron/core/transformer/enums.py
index d06d58d65f2..1bf16095908 100644
--- a/megatron/core/transformer/enums.py
+++ b/megatron/core/transformer/enums.py
@@ -9,19 +9,10 @@ class ModelType(enum.Enum):
"""Model Type
encoder_or_decoder for bert, gpt etc
- encoder_and_decoder for multimodal , T5 etc
"""
encoder_or_decoder = 1
- @property
- def encoder_and_decoder(self):
- """Deprecated property - use encoder_or_decoder instead."""
- raise ValueError(
- "ModelType.encoder_and_decoder is deprecated. Please use ModelType.encoder_or_decoder "
- "instead."
- )
-
class LayerType(enum.Enum):
"""Layer type
@@ -70,10 +61,11 @@ class AttnBackend(enum.Enum):
class CudaGraphScope(enum.Enum):
"""Cuda Graph Scope - defines which parts of the model to capture."""
- full_iteration = 1 # Captures the entire training/inference iteration
+ full_iteration = 1 # Captures the entire training iteration
attn = 2 # Captures attention layers
mlp = 3 # Captures MLP layers (dense layers only)
moe = 4 # Captures MoE layers (drop-and-pad MoE layers only)
moe_router = 5 # Captures MoE router part
moe_preprocess = 6 # Captures MoE preprocessing part (requires moe_router)
mamba = 7 # Captures Mamba layers
+ full_iteration_inference = 8 # Captures the entire inference iteration
diff --git a/megatron/core/transformer/experimental_attention_variant/absorbed_mla.py b/megatron/core/transformer/experimental_attention_variant/absorbed_mla.py
new file mode 100644
index 00000000000..860118b17a3
--- /dev/null
+++ b/megatron/core/transformer/experimental_attention_variant/absorbed_mla.py
@@ -0,0 +1,955 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""
+Absorbed Multi-Latent Attention implementation.
+
+This module implements MLA with matrix absorption:
+- Absorbs K's up projection into Q: Q' = Q @ K_up_proj^T
+- Applies V's up projection after core attention
+- Core attention operates in MQA form with KV being single-head.
+
+The absorption is mathematically equivalent to standard MLA but enables MQA-style attention which
+can be more efficient for certain attention variants.
+"""
+
+import math
+from dataclasses import dataclass
+from typing import NoReturn, Optional, Union
+
+import torch
+
+from megatron.core.extensions.transformer_engine import HAVE_TE
+from megatron.core.models.common.embeddings import (
+ RotaryEmbedding,
+ YarnRotaryEmbedding,
+ _yarn_get_mscale,
+ apply_rotary_pos_emb,
+)
+from megatron.core.process_groups_config import ProcessGroupCollection
+from megatron.core.tensor_parallel.layers import ColumnParallelLinear
+from megatron.core.tensor_parallel.mappings import (
+ gather_from_sequence_parallel_region,
+ gather_from_tensor_model_parallel_region,
+ scatter_to_sequence_parallel_region,
+)
+from megatron.core.transformer.attention import Attention
+from megatron.core.transformer.enums import AttnMaskType
+from megatron.core.transformer.spec_utils import ModuleSpec, build_module
+from megatron.core.transformer.transformer_config import MLATransformerConfig
+from megatron.core.utils import deprecate_inference_params, get_pg_size
+
+try:
+ from megatron.core.fusions.fused_mla_yarn_rope_apply import (
+ fused_apply_mla_rope_for_kv,
+ fused_apply_mla_rope_for_q,
+ )
+except ImportError:
+ fused_apply_mla_rope_for_kv = None
+ fused_apply_mla_rope_for_q = None
+
+if HAVE_TE:
+ from megatron.core.extensions.transformer_engine import (
+ TEColumnParallelLinear,
+ TELinear,
+ set_save_original_input,
+ )
+ from megatron.core.post_training.modelopt.layers import Linear
+else:
+ TEColumnParallelLinear, TELinear, Linear, set_save_original_input = None, None, None, None
+
+
+@dataclass
+class AbsorbedMLASelfAttentionSubmodules:
+ """
+ Configuration class for specifying the submodules of absorbed multi-latent self-attention.
+ """
+
+ linear_q_proj: Union[ModuleSpec, type] = None
+ linear_q_down_proj: Union[ModuleSpec, type] = None
+ linear_q_up_proj: Union[ModuleSpec, type] = None
+ linear_kv_down_proj: Union[ModuleSpec, type] = None
+ linear_k_up_proj: Union[ModuleSpec, type] = None
+ linear_v_up_proj: Union[ModuleSpec, type] = None
+ core_attention: Union[ModuleSpec, type] = None
+ linear_proj: Union[ModuleSpec, type] = None
+ q_layernorm: Union[ModuleSpec, type] = None
+ kv_layernorm: Union[ModuleSpec, type] = None
+
+
+class AbsorbedMLASelfAttention(Attention):
+ """Multi-latent self-attention layer with matrix absorption.
+
+ This layer takes input with shape [s, b, h] and returns output of the same shape.
+
+ Compared to standard MLA, this class implements matrix absorption:
+ - K's up projection is applied to the query before core attention, not to the compressed KV.
+ - V's up projection is applied to the output of core attention, not to the compressed KV.
+ - Core attention operates in MQA form with KV being single-head.
+
+ The absorption is mathematically equivalent to standard MLA but enables MQA-style attention
+ computation which can be more efficient for certain attention variants.
+ """
+
+ def __init__(
+ self,
+ config: MLATransformerConfig,
+ submodules: AbsorbedMLASelfAttentionSubmodules,
+ layer_number: int,
+ attn_mask_type=AttnMaskType.padding,
+ cp_comm_type: Optional[str] = None,
+ pg_collection: ProcessGroupCollection = None,
+ ):
+ if pg_collection is None:
+ pg_collection = ProcessGroupCollection.use_mpu_process_groups()
+
+ super().__init__(
+ config=config,
+ submodules=submodules,
+ layer_number=layer_number,
+ attn_mask_type=attn_mask_type,
+ attention_type="self",
+ pg_collection=pg_collection,
+ )
+
+ assert not config.add_bias_linear, "add_bias_linear is not supported for AbsorbedMLA"
+
+ self.query_projection_size = self.config.v_head_dim * self.config.num_attention_heads
+ self.q_head_dim = self.config.qk_head_dim + self.config.qk_pos_emb_head_dim
+
+ # Inference is currently not supported.
+ self.key_hidden_size = None
+ self.val_hidden_size = None
+
+ self.recompute_up_proj = (
+ self.config.recompute_granularity == 'selective'
+ and "mla_up_proj" in self.config.recompute_modules
+ )
+ self.qkv_up_checkpoint = None
+
+ mscale = _yarn_get_mscale(self.config.rotary_scaling_factor, self.config.mscale_all_dim)
+ self.softmax_scale = mscale * mscale / math.sqrt(self.q_head_dim)
+ self.cache_mla_latents = self.config.cache_mla_latents
+ assert not self.cache_mla_latents, "cache_mla_latents is not supported for AbsorbedMLA"
+
+ if self.config.rope_type == "rope":
+ self.rotary_pos_emb = RotaryEmbedding(
+ self.config.qk_pos_emb_head_dim,
+ rotary_percent=self.config.rotary_percent,
+ rotary_base=self.config.rotary_base,
+ cp_group=self.pg_collection.cp,
+ )
+ elif self.config.rope_type == "yarn":
+ self.rotary_pos_emb = YarnRotaryEmbedding(
+ self.config.qk_pos_emb_head_dim,
+ rotary_base=self.config.rotary_base,
+ scaling_factor=self.config.rotary_scaling_factor,
+ original_max_position_embeddings=self.config.original_max_position_embeddings,
+ beta_fast=self.config.beta_fast,
+ beta_slow=self.config.beta_slow,
+ mscale=self.config.mscale,
+ mscale_all_dim=self.config.mscale_all_dim,
+ cp_group=self.pg_collection.cp,
+ )
+ else:
+ raise ValueError(
+ f"Unsupported RoPE type: {self.config.rope_type}, supported types are "
+ "'rope' and 'yarn'"
+ )
+
+ self.core_attention = build_module(
+ submodules.core_attention,
+ config=self.config,
+ layer_number=self.layer_number,
+ attn_mask_type=self.attn_mask_type,
+ attention_type="self",
+ softmax_scale=self.softmax_scale,
+ k_channels=self.config.kv_lora_rank + self.config.qk_pos_emb_head_dim,
+ v_channels=self.config.kv_lora_rank,
+ cp_comm_type=cp_comm_type,
+ pg_collection=self.pg_collection,
+ )
+
+ # Output.
+ self.linear_proj = build_module(
+ submodules.linear_proj,
+ self.query_projection_size,
+ self.config.hidden_size,
+ config=self.config,
+ init_method=self.config.output_layer_init_method,
+ bias=self.config.add_bias_linear,
+ input_is_parallel=True,
+ skip_bias_add=True,
+ is_expert=False,
+ tp_comm_buffer_name='proj',
+ tp_group=self.pg_collection.tp,
+ )
+
+ if (
+ HAVE_TE
+ and isinstance(self.linear_proj, TELinear)
+ and (
+ (
+ self.config.fp8
+ and self.config.fp8_recipe != 'delayed'
+ and is_te_min_version("2.6.0dev0")
+ )
+ or (self.config.fp4 and is_te_min_version("2.7.0.dev0"))
+ )
+ ):
+ # For fp8/fp4 training, the output of the fused core_attn is saved by itself, and
+ # linear_proj also saves the quantized tensor of this output. Here we set the
+ # linear_proj to save the original input tensors to avoid the extra memory usage of
+ # the quantized tensor.
+ set_save_original_input(self.linear_proj)
+
+ if self.config.q_lora_rank is None:
+ # Not projecting query
+ self.linear_q_proj = build_module(
+ submodules.linear_q_proj,
+ self.config.hidden_size,
+ self.config.num_attention_heads * self.q_head_dim,
+ config=self.config,
+ init_method=self.config.init_method,
+ gather_output=False,
+ bias=False,
+ skip_bias_add=False,
+ is_expert=False,
+ tp_comm_buffer_name='q_proj',
+ )
+ else:
+ q_down_proj_kwargs = {}
+ if submodules.linear_q_down_proj in [TELinear]:
+ q_down_proj_kwargs['parallel_mode'] = 'duplicated'
+ elif submodules.linear_q_down_proj in [
+ Linear,
+ TEColumnParallelLinear,
+ ColumnParallelLinear,
+ ]:
+ q_down_proj_kwargs['gather_output'] = False
+ else:
+ raise ValueError(f"Unsupported linear_q_down_proj: {submodules.linear_q_down_proj}")
+
+ self.linear_q_down_proj = build_module(
+ submodules.linear_q_down_proj,
+ self.config.hidden_size,
+ self.config.q_lora_rank,
+ config=self.config,
+ init_method=self.config.init_method,
+ bias=False,
+ skip_bias_add=False,
+ is_expert=False,
+ tp_comm_buffer_name='q_down_proj',
+ skip_weight_param_allocation=False,
+ tp_group=(
+ pg_collection.tp
+ if q_down_proj_kwargs.get('parallel_mode') != 'duplicated'
+ else None
+ ),
+ **q_down_proj_kwargs,
+ )
+
+ self.linear_q_up_proj = build_module(
+ submodules.linear_q_up_proj,
+ self.config.q_lora_rank,
+ self.config.num_attention_heads * self.q_head_dim,
+ config=self.config,
+ init_method=self.config.init_method,
+ gather_output=False,
+ bias=False,
+ skip_bias_add=False,
+ is_expert=False,
+ tp_comm_buffer_name='q_up_proj',
+ tp_group=pg_collection.tp,
+ )
+
+ kv_down_proj_kwargs = {}
+ if submodules.linear_kv_down_proj in [TELinear]:
+ kv_down_proj_kwargs['parallel_mode'] = 'duplicated'
+ elif submodules.linear_kv_down_proj in [
+ Linear,
+ TEColumnParallelLinear,
+ ColumnParallelLinear,
+ ]:
+ kv_down_proj_kwargs['gather_output'] = False
+ else:
+ raise ValueError(f"Unsupported linear_kv_down_proj: {submodules.linear_kv_down_proj}")
+
+ self.linear_kv_down_proj = build_module(
+ submodules.linear_kv_down_proj,
+ self.config.hidden_size,
+ self.config.kv_lora_rank + self.config.qk_pos_emb_head_dim,
+ config=self.config,
+ init_method=self.config.init_method,
+ bias=False,
+ skip_bias_add=False,
+ is_expert=False,
+ tp_comm_buffer_name='kv_down_proj',
+ skip_weight_param_allocation=False,
+ tp_group=(
+ pg_collection.tp
+ if kv_down_proj_kwargs.get('parallel_mode') != 'duplicated'
+ else None
+ ),
+ **kv_down_proj_kwargs,
+ )
+
+ # Build separate K and V up projections
+ self.linear_k_up_proj = build_module(
+ submodules.linear_k_up_proj,
+ self.config.kv_lora_rank,
+ self.config.num_attention_heads * self.config.qk_head_dim,
+ config=self.config,
+ init_method=self.config.init_method,
+ gather_output=False,
+ bias=False,
+ skip_bias_add=False,
+ is_expert=False,
+ tp_comm_buffer_name='k_up_proj',
+ tp_group=pg_collection.tp,
+ )
+ self.linear_v_up_proj = build_module(
+ submodules.linear_v_up_proj,
+ self.config.kv_lora_rank,
+ self.config.num_attention_heads * self.config.v_head_dim,
+ config=self.config,
+ init_method=self.config.init_method,
+ gather_output=False,
+ bias=False,
+ skip_bias_add=False,
+ is_expert=False,
+ tp_comm_buffer_name='v_up_proj',
+ tp_group=pg_collection.tp,
+ )
+
+ if self.config.q_lora_rank is not None:
+ self.q_layernorm = build_module(
+ submodules.q_layernorm,
+ hidden_size=self.config.q_lora_rank,
+ config=self.config,
+ eps=self.config.layernorm_epsilon,
+ )
+
+ self.kv_layernorm = build_module(
+ submodules.kv_layernorm,
+ hidden_size=self.config.kv_lora_rank,
+ config=self.config,
+ eps=self.config.layernorm_epsilon,
+ )
+
+ def get_query_key_value_tensors(
+ self,
+ hidden_states,
+ key_value_states=None,
+ packed_seq_params=None,
+ inference_context=None,
+ *,
+ inference_params=None,
+ ):
+ """
+ Derives absorbed q, compressed q, and compressed kv tensors from `hidden_states`.
+ """
+ # s = sequence length, b = batch size, h = hidden size
+ assert (
+ hidden_states.ndim == 3
+ ), f"hidden_states should be 3D, [s, b, h], got {hidden_states.ndim}D"
+ if packed_seq_params is not None:
+ assert (
+ packed_seq_params.local_cp_size is None
+ ), "dynamic context parallel is not supported with MLA yet and is planned for future. \
+ Please disable dynamic context parallel."
+
+ inference_context = deprecate_inference_params(inference_context, inference_params)
+
+ # =========================================
+ # Prepare RoPE and seqlen related params
+ # =========================================
+ rotary_seq_len = self.rotary_pos_emb.get_rotary_seq_len(
+ inference_context, None, hidden_states, self.config, packed_seq_params
+ )
+
+ mscale = 1.0
+ rotary_pos_cos = None
+ rotary_pos_sin = None
+ packed_seq = packed_seq_params is not None and packed_seq_params.qkv_format == 'thd'
+ if self.config.rope_type == "rope":
+ rotary_pos_emb = self.rotary_pos_emb(rotary_seq_len, packed_seq=packed_seq)
+ else:
+ if self.config.apply_rope_fusion:
+ rotary_pos_cos, rotary_pos_sin = self.rotary_pos_emb.get_cached_cos_sin(
+ rotary_seq_len, dtype=hidden_states.dtype, packed_seq=packed_seq
+ )
+ rotary_pos_emb = None
+ assert inference_context is None, "Inference with MLA RoPE fusion is not supported"
+ assert (
+ fused_apply_mla_rope_for_q is not None
+ and fused_apply_mla_rope_for_kv is not None
+ ), "Fused MLA RoPE apply is not imported successfully"
+ else:
+ rotary_pos_emb, mscale = self.rotary_pos_emb(rotary_seq_len, packed_seq=packed_seq)
+
+ if packed_seq_params is not None and packed_seq_params.qkv_format == 'thd':
+ if packed_seq_params.cu_seqlens_q_padded is not None:
+ cu_seqlens_q = packed_seq_params.cu_seqlens_q_padded
+ else:
+ cu_seqlens_q = packed_seq_params.cu_seqlens_q
+ if packed_seq_params.cu_seqlens_kv_padded is not None:
+ cu_seqlens_kv = packed_seq_params.cu_seqlens_kv_padded
+ else:
+ cu_seqlens_kv = packed_seq_params.cu_seqlens_kv
+ else:
+ cu_seqlens_q = cu_seqlens_kv = None
+
+ # =========================================
+ # Q down projection
+ # =========================================
+ if self.config.q_lora_rank is not None:
+ # if linear_q_down_proj is ColumnParallelLinear:
+ # q_compressed: [s, b, q_lora_rank / TP]
+ # elif linear_q_down_proj is Linear:
+ # q_compressed: [s / TP, b, q_lora_rank]
+ q_compressed, _ = self.linear_q_down_proj(hidden_states)
+
+ # When output is sharded (ColumnParallelLinear), two things are needed to be
+ # identical to a normal Linear.
+ # 1. Manually gather output to restore output dim q_lora_rank;
+ # 2. Scatter sequence back to s / TP if sequence-parallel since it was
+ # gathered by ColumnParallelLinear.
+ if q_compressed.size(-1) != self.config.q_lora_rank:
+ q_compressed = gather_from_tensor_model_parallel_region(q_compressed)
+ if self.config.sequence_parallel:
+ q_compressed = scatter_to_sequence_parallel_region(q_compressed)
+ else:
+ q_compressed = hidden_states
+
+ # =========================================
+ # KV down projection
+ # =========================================
+ # if linear_kv_down_proj is ColumnParallelLinear:
+ # kv_combined: [s, b, (kv_lora_rank + qk_pos_emb_head_dim) / TP]
+ # elif linear_kv_down_proj is Linear:
+ # kv_combined: [s / TP, b, (kv_lora_rank + qk_pos_emb_head_dim)]
+ kv_combined, _ = self.linear_kv_down_proj(hidden_states)
+ if kv_combined.size(-1) != self.config.kv_lora_rank + self.config.qk_pos_emb_head_dim:
+ # kv_combined: [s, b, (kv_lora_rank + qk_pos_emb_head_dim)]
+ kv_combined = gather_from_tensor_model_parallel_region(kv_combined)
+ # kv_compressed:[s, b, kv_lora_rank], k_pos_emb: [s, b, qk_pos_emb_head_dim]
+ kv_compressed, k_pos_emb = torch.split(
+ kv_combined, [self.config.kv_lora_rank, self.config.qk_pos_emb_head_dim], dim=-1
+ )
+ if self.config.sequence_parallel:
+ # kv_compressed:[s / TP, b, kv_lora_rank]
+ kv_compressed = scatter_to_sequence_parallel_region(kv_compressed)
+ else:
+ # kv_compressed:[s / TP, b, kv_lora_rank], k_pos_emb: [s / TP, b, qk_pos_emb_head_dim]
+ kv_compressed, k_pos_emb = torch.split(
+ kv_combined, [self.config.kv_lora_rank, self.config.qk_pos_emb_head_dim], dim=-1
+ )
+ if get_pg_size(self.tp_group) > 1 and self.config.sequence_parallel:
+ # k_pos_emb: [s, b, qk_pos_emb_head_dim]
+ k_pos_emb = gather_from_sequence_parallel_region(k_pos_emb, group=self.tp_group)
+
+ if packed_seq_params is not None:
+ assert q_compressed.ndim == 3 and q_compressed.size(1) == 1
+ assert kv_compressed.ndim == 3 and kv_compressed.size(1) == 1
+ assert k_pos_emb.ndim == 3 and k_pos_emb.size(1) == 1
+ # If sequence packing, TE expect [t, h, d] shaped qkv input.
+ # In Megatron-Core, the qkv shape is [t, 1, h, d].
+ # So we need to reshape qkv from [t, 1, h, d] to [t, h, d].
+ q_compressed = q_compressed.squeeze(1)
+ kv_compressed = kv_compressed.squeeze(1)
+ k_pos_emb = k_pos_emb.squeeze(1)
+
+ # =========================================
+ # Apply norm
+ # =========================================
+ if self.config.q_lora_rank is not None:
+ # q_compressed: [num_tokens, q_lora_rank]
+ q_compressed = self.q_layernorm(q_compressed)
+
+ kv_compressed = self.kv_layernorm(kv_compressed)
+ # Because we won't apply V up projection to the compressed KV, so we need to gather it
+ # manually.
+ if get_pg_size(self.tp_group) > 1 and self.config.sequence_parallel:
+ kv_compressed = gather_from_sequence_parallel_region(kv_compressed, group=self.tp_group)
+
+ # =========================================
+ # QKV up projection and RoPE apply
+ # =========================================
+
+ def qkv_up_proj_and_rope_apply(q_compressed, kv_compressed, k_pos_emb, rotary_pos_emb):
+ """
+ Apply the up projection and RoPE to the query and key.
+ When sequence packing enabled, the input tensors adopt a packed shape of [t, ...];
+ otherwise, they maintain the unpacked shape [s, b, ...]. In subsequent code comments,
+ we uniformly use [num_tokens, ...] to denote [s, b, ...] or [t, ...] for two cases.
+ """
+ if self.config.q_lora_rank is not None:
+ # q_compressed: [num_tokens, q_lora_rank]
+ # q: [num_tokens, n * (qk_head_dim + qk_pos_emb_head_dim)]
+ q, _ = self.linear_q_up_proj(q_compressed)
+ else:
+ # q_compressed: [num_tokens, hidden_size]
+ # q: [num_tokens, n * (qk_head_dim + qk_pos_emb_head_dim)]
+ q, _ = self.linear_q_proj(q_compressed)
+
+ # q: [num_tokens, n, q_head_dim]
+ q = q.view(*q.size()[:-1], self.num_attention_heads_per_partition, self.q_head_dim)
+
+ # [num_tokens, kv_lora_rank] -> [num_tokens, 1, kv_lora_rank]
+ kv_compressed = torch.unsqueeze(kv_compressed, -2)
+ # [num_tokens, qk_pos_emb_head_dim] -> [num_tokens, 1, qk_pos_emb_head_dim]
+ k_pos_emb = torch.unsqueeze(k_pos_emb, -2)
+
+ # Prepare k_up_weight for absorption
+ # k_up_weight: linear_k_up_proj.weight viewed as [n, qk_head_dim, kv_lora_rank]
+ assert self.linear_k_up_proj.weight.size(0) == (
+ self.num_attention_heads_per_partition * self.config.qk_head_dim
+ )
+ assert self.linear_k_up_proj.weight.size(1) == self.config.kv_lora_rank
+ k_up_weight = self.linear_k_up_proj.weight.view(
+ self.num_attention_heads_per_partition,
+ self.config.qk_head_dim,
+ self.config.kv_lora_rank,
+ )
+
+ if self.config.apply_rope_fusion:
+ # q_no_pe: [num_tokens, n, qk_head_dim]
+ # q_pos_emb: [num_tokens, n, qk_pos_emb_head_dim]
+ q_no_pe, q_pos_emb = torch.split(
+ q, [self.config.qk_head_dim, self.config.qk_pos_emb_head_dim], dim=-1
+ )
+
+ # Absorb k_up_weight into q_no_pe
+ # q_absorbed: [num_tokens, n, kv_lora_rank]
+ q_absorbed = torch.einsum("...nd,ndk->...nk", q_no_pe, k_up_weight)
+ q_absorbed = q_absorbed.contiguous()
+ assert q_absorbed.ndim == q.ndim
+ assert q_absorbed.shape[:-1] == q.shape[:-1]
+ assert q_absorbed.size(-1) == self.config.kv_lora_rank
+
+ # q_absorbed: [num_tokens, n, (kv_lora_rank + qk_pos_emb_head_dim)]
+ q_absorbed = torch.cat([q_absorbed, q_pos_emb], dim=-1)
+ # kv_compressed: [num_tokens, 1, (kv_lora_rank + qk_pos_emb_head_dim)]
+ kv_compressed = torch.cat([kv_compressed, k_pos_emb], dim=-1)
+
+ cp_rank = self.pg_collection.cp.rank()
+ cp_size = self.pg_collection.cp.size()
+ q_absorbed = fused_apply_mla_rope_for_q(
+ q_absorbed,
+ rotary_pos_cos,
+ rotary_pos_sin,
+ self.config.kv_lora_rank,
+ self.config.qk_pos_emb_head_dim,
+ cu_seqlens_q,
+ cp_rank,
+ cp_size,
+ )
+ kv_compressed = fused_apply_mla_rope_for_q(
+ kv_compressed,
+ rotary_pos_cos,
+ rotary_pos_sin,
+ self.config.kv_lora_rank,
+ self.config.qk_pos_emb_head_dim,
+ cu_seqlens_kv,
+ cp_rank,
+ cp_size,
+ )
+ else:
+ q_len = q.size()[0]
+ if inference_context is not None:
+ # add offset to the sequence start for inference
+ sequence_start = inference_context.sequence_len_offset
+ sequence_end = sequence_start + q_len
+ rotary_pos_emb = rotary_pos_emb[sequence_start:sequence_end]
+ elif packed_seq_params is None or self.config.context_parallel_size == 1:
+ # Shorten rotary_pos_emb to the sequence length when inference_params
+ # is not provided. This makes sure we can run forward directly with
+ # any sequence length. During training, the sequence length is always
+ # the full rotary_pos_emb length, except for sequence packing + CP.
+ # When sequence packing and context parallel are both enabled, the
+ # position embedding will not split rotary_pos_emb, so it may exceed
+ # the sequence length on this CP rank, but we need the full rotary_pos_emb
+ # to cover the full sequence, so we do not shorten it here.
+ rotary_pos_emb = rotary_pos_emb[0:q_len]
+
+ # q_no_pe: [num_tokens, n, qk_head_dim]
+ # q_pos_emb: [num_tokens, n, qk_pos_emb_head_dim]
+ q_no_pe, q_pos_emb = torch.split(
+ q, [self.config.qk_head_dim, self.config.qk_pos_emb_head_dim], dim=-1
+ )
+
+ # Absorb k_up_weight into q_no_pe
+ # q_absorbed: [num_tokens, n, kv_lora_rank]
+ q_absorbed = torch.einsum("...nd,ndk->...nk", q_no_pe, k_up_weight)
+ q_absorbed = q_absorbed.contiguous()
+ assert q_absorbed.ndim == q.ndim
+ assert q_absorbed.shape[:-1] == q.shape[:-1]
+ assert q_absorbed.size(-1) == self.config.kv_lora_rank
+
+ # Apply RoPE to q_pos_emb: [num_tokens, n, qk_pos_emb_head_dim]
+ q_pos_emb = apply_rotary_pos_emb(
+ q_pos_emb,
+ rotary_pos_emb,
+ config=self.config,
+ cu_seqlens=cu_seqlens_q,
+ mscale=mscale,
+ cp_group=self.pg_collection.cp,
+ )
+ # k_pos_emb:[num_tokens, 1, qk_pos_emb_head_dim]
+ k_pos_emb = apply_rotary_pos_emb(
+ k_pos_emb,
+ rotary_pos_emb,
+ config=self.config,
+ cu_seqlens=cu_seqlens_kv,
+ mscale=mscale,
+ cp_group=self.pg_collection.cp,
+ )
+
+ # query: [num_tokens, n, (kv_lora_rank + qk_pos_emb_head_dim)]
+ q_absorbed = torch.cat([q_absorbed, q_pos_emb], dim=-1)
+ # key: [num_tokens, 1, (kv_lora_rank + qk_pos_emb_head_dim)]
+ kv_compressed = torch.cat([kv_compressed, k_pos_emb], dim=-1)
+
+ assert q_absorbed.is_contiguous()
+ assert kv_compressed.is_contiguous()
+
+ return q_absorbed, kv_compressed
+
+ if self.recompute_up_proj:
+ quantization = self.config.fp8 or self.config.fp4
+ assert not quantization, "FP8/FP4 is not supported for AbsorbedMLA"
+ self.qkv_up_checkpoint = tensor_parallel.CheckpointWithoutOutput(fp8=quantization)
+ q_absorbed, kv_compressed = self.qkv_up_checkpoint.checkpoint(
+ qkv_up_proj_and_rope_apply, q_compressed, kv_compressed, k_pos_emb, rotary_pos_emb
+ )
+ else:
+ assert not self.cache_mla_latents, "cache_mla_latents is not supported for AbsorbedMLA"
+ q_absorbed, kv_compressed = qkv_up_proj_and_rope_apply(
+ q_compressed, kv_compressed, k_pos_emb, rotary_pos_emb
+ )
+
+ return q_absorbed, kv_compressed, q_compressed
+
+ def _checkpointed_attention_forward(
+ self,
+ q_absorbed,
+ k_compressed,
+ v_compressed,
+ hidden_states,
+ q_compressed,
+ attention_mask,
+ rotary_pos_emb=None,
+ attn_mask_type=None,
+ attention_bias=None,
+ packed_seq_params=None,
+ ):
+ """Forward method with selective activation checkpointing."""
+
+ def custom_forward(*inputs):
+ q_absorbed = inputs[0]
+ k_compressed = inputs[1]
+ v_compressed = inputs[2]
+ hidden_states = inputs[3]
+ q_compressed = inputs[4]
+ attention_mask = inputs[5]
+ attn_mask_type = inputs[7]
+ attention_bias = inputs[8]
+ packed_seq_params = inputs[9]
+ attn_mask_type = AttnMaskType(attn_mask_type.item())
+ output_ = self.core_attention(
+ q_absorbed,
+ k_compressed,
+ v_compressed,
+ hidden_states,
+ q_compressed,
+ attention_mask,
+ attn_mask_type=attn_mask_type,
+ attention_bias=attention_bias,
+ packed_seq_params=packed_seq_params,
+ )
+ return output_
+
+ if attn_mask_type is None:
+ attn_mask_type = self.attn_mask_type
+ attn_mask_type = torch.tensor([attn_mask_type.value], dtype=torch.int)
+ hidden_states = tensor_parallel.checkpoint(
+ custom_forward,
+ False,
+ q_absorbed,
+ k_compressed,
+ v_compressed,
+ hidden_states,
+ q_compressed,
+ attention_mask,
+ rotary_pos_emb,
+ attn_mask_type,
+ attention_bias,
+ packed_seq_params,
+ )
+
+ return hidden_states
+
+ def forward(
+ self,
+ hidden_states,
+ attention_mask,
+ key_value_states=None,
+ inference_context=None,
+ rotary_pos_emb=None,
+ rotary_pos_cos=None,
+ rotary_pos_sin=None,
+ rotary_pos_cos_sin=None,
+ attention_bias=None,
+ packed_seq_params=None,
+ sequence_len_offset=None,
+ *,
+ inference_params=None,
+ ):
+ """Forward pass for multi-latent attention with matrix absorption"""
+ assert rotary_pos_emb is None, "Rotary position embeddings should not be passed into MLA."
+ assert attention_bias is None, "Attention bias should not be passed into MLA."
+ assert (
+ rotary_pos_cos is None and rotary_pos_sin is None
+ ), "MLA does not support Flash Decoding"
+ assert not rotary_pos_cos_sin, "Flash-infer rope has not been tested with MLA."
+ assert not (
+ self.training and self.cache_mla_latents
+ ), "cache_mla_latents conflicts with training."
+ assert (
+ inference_context is None and inference_params is None
+ ), "Inference is not supported for AbsorbedMLA"
+
+ # =====================
+ # Query, Key, and Value
+ # =====================
+ q_absorbed, kv_compressed, q_compressed = self.get_query_key_value_tensors(
+ hidden_states, key_value_states, packed_seq_params, inference_context=inference_context
+ )
+
+ assert q_absorbed.is_contiguous()
+ assert q_compressed.is_contiguous()
+ assert kv_compressed.is_contiguous()
+
+ # ==================================
+ # Core attention computation
+ # ==================================
+ if self.checkpoint_core_attention and self.training:
+ core_attn_out = self._checkpointed_attention_forward(
+ q_absorbed,
+ kv_compressed,
+ None,
+ hidden_states,
+ q_compressed,
+ attention_mask,
+ packed_seq_params=packed_seq_params,
+ )
+ else:
+ core_attn_out = self.core_attention(
+ q_absorbed,
+ kv_compressed,
+ None,
+ hidden_states,
+ q_compressed,
+ attention_mask,
+ packed_seq_params=packed_seq_params,
+ attn_mask_type=self.attn_mask_type,
+ )
+
+ # ==================================
+ # Apply V up projection
+ # ==================================
+ assert self.linear_v_up_proj.weight.size(0) == (
+ self.num_attention_heads_per_partition * self.config.v_head_dim
+ )
+ assert self.linear_v_up_proj.weight.size(1) == self.config.kv_lora_rank
+ v_up_weight = self.linear_v_up_proj.weight.view(
+ self.num_attention_heads_per_partition, self.config.v_head_dim, self.config.kv_lora_rank
+ )
+ core_attn_out = core_attn_out.view(
+ *core_attn_out.shape[:-1],
+ self.num_attention_heads_per_partition,
+ self.config.kv_lora_rank,
+ )
+ core_attn_out = torch.einsum("...nc,ndc->...nd", core_attn_out, v_up_weight)
+ core_attn_out = core_attn_out.contiguous()
+ core_attn_out = core_attn_out.view(*core_attn_out.shape[:-2], -1)
+
+ if packed_seq_params is not None and packed_seq_params.qkv_format == 'thd':
+ core_attn_out = core_attn_out.unsqueeze(1)
+
+ assert core_attn_out.ndim == hidden_states.ndim
+ assert core_attn_out.shape[0] == (
+ hidden_states.shape[0] * self.config.tensor_model_parallel_size
+ ), (
+ f"{core_attn_out.shape[0]} != "
+ f"{hidden_states.shape[0]} * "
+ f"{self.config.tensor_model_parallel_size}"
+ )
+ assert core_attn_out.shape[1:-1] == hidden_states.shape[1:-1]
+ assert core_attn_out.size(-1) == (
+ self.config.v_head_dim * self.num_attention_heads_per_partition
+ )
+
+ if self.recompute_up_proj:
+ assert self.qkv_up_checkpoint is not None
+ self.qkv_up_checkpoint.discard_output_and_register_recompute(core_attn_out)
+ self.qkv_up_checkpoint = None
+
+ # =================
+ # Output. [sq, b, h]
+ # =================
+ output, bias = self.linear_proj(core_attn_out)
+
+ return output, bias
+
+ def backward_dw(self) -> NoReturn:
+ """Execute weight gradient computation."""
+ self._backward_kv_proj()
+ self._backward_q_proj()
+ self._backward_output_proj()
+
+ def _backward_kv_proj(self):
+ """Computes weight gradients of KV projection layers."""
+ self.linear_k_up_proj.backward_dw()
+ self.linear_v_up_proj.backward_dw()
+ self.linear_kv_down_proj.backward_dw()
+
+ def _backward_q_proj(self):
+ """Computes weight gradients of Q projection layers."""
+ if self.config.q_lora_rank is None:
+ self.linear_q_proj.backward_dw()
+ else:
+ self.linear_q_down_proj.backward_dw()
+ self.linear_q_up_proj.backward_dw()
+
+ def _backward_output_proj(self):
+ """Computes weight gradients of output projection layer."""
+ self.linear_proj.backward_dw()
+
+ def set_for_recompute_input_layernorm(self):
+ """Set the attention layer for recompute input_layernorm. Only needed for fp8/fp4."""
+ from megatron.core.extensions.transformer_engine import set_save_original_input
+
+ if self.config.q_lora_rank is not None:
+ set_save_original_input(self.linear_q_down_proj)
+ set_save_original_input(self.linear_kv_down_proj)
+
+ def clip_qk(self):
+ """
+ QK Clipping is a technique to clip the query and key attention logits to prevent the
+ attention logits from exploding. Per MuonClip usage, we update the weight by calling this
+ function after Muon optimizer step.
+ """
+ raise NotImplementedError("clip_qk is not implemented for AbsorbedMLA")
+
+ def _combine_kv_weights(self, k_weight, v_weight):
+ """Combine separate K and V weights into MLA's interleaved format.
+
+ MLA's linear_kv_up_proj weight layout (per head interleaved):
+ [head0_K, head0_V, head1_K, head1_V, ...]
+
+ AbsorbedMLA's separate weights layout:
+ K: [head0_K, head1_K, ...]
+ V: [head0_V, head1_V, ...]
+
+ This method interleaves K and V per head to match MLA's format.
+
+ Args:
+ k_weight: [num_heads_per_partition * qk_head_dim, kv_lora_rank]
+ v_weight: [num_heads_per_partition * v_head_dim, kv_lora_rank]
+
+ Returns:
+ combined: [num_heads_per_partition * (qk_head_dim + v_head_dim), kv_lora_rank]
+ """
+ n = self.num_attention_heads_per_partition
+ qk_dim = self.config.qk_head_dim
+ v_dim = self.config.v_head_dim
+ lora_rank = self.config.kv_lora_rank
+
+ # Reshape to per-head format
+ k_per_head = k_weight.view(n, qk_dim, lora_rank)
+ v_per_head = v_weight.view(n, v_dim, lora_rank)
+
+ # Concatenate K and V for each head along dim=1
+ # Result: [n, qk_dim + v_dim, lora_rank]
+ combined_per_head = torch.cat([k_per_head, v_per_head], dim=1)
+
+ # Reshape back to linear weight format
+ combined_weight = combined_per_head.view(n * (qk_dim + v_dim), lora_rank)
+
+ return combined_weight
+
+ def _split_kv_weights(self, combined_weight):
+ """Split MLA's interleaved KV weight into separate K and V weights.
+
+ MLA's linear_kv_up_proj weight layout (per head interleaved):
+ [head0_K, head0_V, head1_K, head1_V, ...]
+
+ This method extracts K and V into separate tensors:
+ K: [head0_K, head1_K, ...]
+ V: [head0_V, head1_V, ...]
+
+ Args:
+ combined_weight: [num_heads_per_partition * (qk_head_dim + v_head_dim), kv_lora_rank]
+
+ Returns:
+ k_weight: [num_heads_per_partition * qk_head_dim, kv_lora_rank]
+ v_weight: [num_heads_per_partition * v_head_dim, kv_lora_rank]
+ """
+ n = self.num_attention_heads_per_partition
+ qk_dim = self.config.qk_head_dim
+ v_dim = self.config.v_head_dim
+ lora_rank = self.config.kv_lora_rank
+
+ # Reshape to per-head format
+ combined_per_head = combined_weight.view(n, qk_dim + v_dim, lora_rank)
+
+ # Split K and V for each head (slicing creates non-contiguous views)
+ k_per_head = combined_per_head[:, :qk_dim, :] # [n, qk_dim, lora_rank]
+ v_per_head = combined_per_head[:, qk_dim:, :] # [n, v_dim, lora_rank]
+
+ # Make contiguous and reshape back to linear weight format
+ k_weight = k_per_head.contiguous().view(n * qk_dim, lora_rank)
+ v_weight = v_per_head.contiguous().view(n * v_dim, lora_rank)
+
+ return k_weight, v_weight
+
+ def _load_from_state_dict(self, state_dict, prefix, *args, **kwargs):
+ """Handle loading from checkpoints with combined KV up projection weights.
+
+ This method splits the combined 'linear_kv_up_proj.weight' (which has per-head
+ interleaved K and V) into separate 'linear_k_up_proj.weight' and 'linear_v_up_proj.weight'.
+ """
+ combined_key = f'{prefix}linear_kv_up_proj.weight'
+ k_up_key = f'{prefix}linear_k_up_proj.weight'
+ v_up_key = f'{prefix}linear_v_up_proj.weight'
+
+ # Split combined KV weights into separate K and V
+ if combined_key in state_dict:
+ combined_weight = state_dict[combined_key]
+
+ # Split with proper per-head de-interleaving
+ k_weight, v_weight = self._split_kv_weights(combined_weight)
+
+ state_dict[k_up_key] = k_weight
+ state_dict[v_up_key] = v_weight
+
+ del state_dict[combined_key]
+
+ combined_extra_state_key = f'{prefix}linear_kv_up_proj._extra_state'
+ k_up_extra_state_key = f'{prefix}linear_k_up_proj._extra_state'
+ v_up_extra_state_key = f'{prefix}linear_v_up_proj._extra_state'
+
+ if combined_extra_state_key in state_dict:
+ combined_extra_state = state_dict[combined_extra_state_key]
+
+ assert isinstance(combined_extra_state, torch.Tensor)
+ # Now we can only handle the case where the extra state is empty.
+ assert combined_extra_state.numel() == 0
+
+ state_dict[k_up_extra_state_key] = combined_extra_state.clone()
+ state_dict[v_up_extra_state_key] = combined_extra_state.clone()
+
+ del state_dict[combined_extra_state_key]
+
+ super()._load_from_state_dict(state_dict, prefix, *args, **kwargs)
diff --git a/megatron/core/transformer/experimental_attention_variant/dsa.py b/megatron/core/transformer/experimental_attention_variant/dsa.py
index 88b4713dc60..3734db7043f 100644
--- a/megatron/core/transformer/experimental_attention_variant/dsa.py
+++ b/megatron/core/transformer/experimental_attention_variant/dsa.py
@@ -252,6 +252,330 @@ def compute_dsa_indexer_loss(
return indexer_loss
+def _compute_index_scores(q: torch.Tensor, weights: torch.Tensor, k: torch.Tensor) -> torch.Tensor:
+ """
+ Perform index score using BF16 precision.
+
+ Reference:
+ https://github.com/deepseek-ai/DeepSeek-V3.2-Exp/blob/main/inference/kernel.py#L254-L274
+ This is a BF16 implementation of the `fp8_index` logic:
+ 1. Compute attention scores: q @ k^T;
+ 2. Apply ReLU activation;
+ 3. Weight by attention weights;
+ 4. Sum across attention heads.
+
+ Args:
+ q: BF16 [seqlen_q, batch, index_n_heads, index_head_dim], the query tensor.
+ weights: BF16 [seqlen_q, batch, index_n_heads], the attention weights.
+ k: BF16 [seqlen_k, batch, index_head_dim], the key tensor.
+
+ Returns:
+ index_scores: FP32 [batch, seqlen_q, seqlen_k], the index scores.
+ """
+ # Compute attention scores: q @ k^T
+ # [seqlen_q, batch, index_n_heads, index_head_dim] @ [seqlen_k, batch, index_head_dim]^T
+ # -> [seqlen_q, batch, index_n_heads, seqlen_k]
+ index_scores = torch.einsum('sbhd,tbd->sbht', q.float(), k.float())
+
+ # Apply ReLU activation.
+ index_scores = torch.relu(index_scores)
+
+ # Weight each head by attention weights.
+ # [seqlen_q, batch, index_n_heads, seqlen_k] * [seqlen_q, batch, index_n_heads, 1]
+ # -> [seqlen_q, batch, index_n_heads, seqlen_k]
+ index_scores = index_scores * weights.unsqueeze(-1)
+
+ # Sum across attention heads.
+ # [seqlen_q, batch, index_n_heads, seqlen_k] -> [seqlen_q, batch, seqlen_k]
+ index_scores = index_scores.sum(dim=2)
+
+ # Transpose to [batch, seqlen_q, seqlen_k].
+ index_scores = index_scores.transpose(0, 1)
+
+ return index_scores
+
+
+def fused_qk_topk_naive(
+ q: torch.Tensor,
+ k: torch.Tensor,
+ weights: torch.Tensor,
+ index_topk: int,
+ mask: Optional[torch.Tensor] = None,
+):
+ """Naive implementation of QK Topk."""
+ seqlen = q.size(0)
+ # =========================================
+ # Compute index scores
+ # =========================================
+ # [batch, seqlen, seqlen]
+ index_scores = _compute_index_scores(q, weights, k)
+ if mask is not None:
+ assert mask.dtype == index_scores.dtype, "Mask dtype must match index scores dtype"
+ index_scores = index_scores + mask
+
+ # =========================================
+ # Select top-k indices
+ # =========================================
+ topk_k = min(index_topk, seqlen)
+ # [batch, seqlen, index_topk]
+ topk_indices = index_scores.topk(topk_k, dim=-1)[1]
+
+ return index_scores, topk_indices
+
+
+def fwd_fused_indexer_loss_naive(
+ q, weights, k, query, key, topk, softmax_scale, loss_coeff, mask, sparse_loss, pg_collection
+):
+ """Naive implementation of forward pass for indexer loss."""
+ index_scores, topk_indices = fused_qk_topk_naive(q, k, weights, topk, mask)
+
+ indexer_loss = compute_dsa_indexer_loss(
+ index_scores,
+ topk_indices,
+ query,
+ key,
+ softmax_scale,
+ loss_coeff,
+ sparse_loss,
+ pg_collection,
+ )
+
+ return topk_indices, indexer_loss
+
+
+def bwd_fused_indexer_loss_naive(
+ q,
+ weights,
+ k,
+ query,
+ key,
+ topk_indices,
+ softmax_scale,
+ loss_coeff,
+ sparse_loss,
+ grad_loss,
+ pg_collection,
+):
+ """Naive implementation of backward pass for indexer loss."""
+ index_scores = _compute_index_scores(q, weights, k) # [B, Sq, Sk]
+
+ sq, b, np, hn = query.size()
+ sk = key.size(0)
+
+ # [sq, b, np, hn] -> [b, np, sq, hn] -> [b * np, sq, hn]
+ query_reshaped = query.permute(1, 2, 0, 3).reshape(b * np, sq, hn)
+ # [sk, b, np, hn] -> [b, np, hn, sk] -> [b * np, hn, sk]
+ key_reshaped = key.permute(1, 2, 3, 0).reshape(b * np, hn, sk)
+ # Compute attention scores [b * np, sq, sk]
+ attention_scores = torch.bmm(query_reshaped.float(), key_reshaped.float()) * softmax_scale
+ # Free reshaped tensors - no longer needed after bmm
+ del query_reshaped, key_reshaped
+
+ # Reshape to [b, np, sq, sk]
+ attention_scores = attention_scores.reshape(b, np, sq, sk)
+
+ # causal_mask [sq, sk]
+ causal_mask = torch.triu(
+ torch.full((sq, sk), float('-inf'), dtype=torch.float32, device=attention_scores.device),
+ diagonal=1,
+ )
+ # index_mask [b, sq, sk]
+ index_mask = torch.full(
+ (b, sq, sk), float("-inf"), dtype=torch.float32, device=causal_mask.device
+ ).scatter_(-1, topk_indices, 0)
+
+ # Apply causal mask to both attention and index scores
+ # [b, np, sq, skv] + [1, 1, sq, skv] -> [b, np, sq, skv]
+ attention_scores = attention_scores + causal_mask.view(1, 1, sq, sk)
+ # [b, sq, sk] + [1, sq, sk] -> [b, sq, sk]
+ index_scores = index_scores + causal_mask.unsqueeze(0)
+ # Free causal_mask - no longer needed
+ del causal_mask
+
+ if sparse_loss:
+ # [b, np, sq, sk] + [b, 1, sq, sk] -> [b, np, sq, sk]
+ attention_scores = attention_scores + index_mask.view(b, 1, sq, sk)
+ # [b, sq, sk] + [b, sq, sk] -> [b, sq, sk]
+ index_scores = index_scores + index_mask
+
+ # Compute softmax for both
+ attention_scores_softmax = torch.nn.functional.softmax(
+ attention_scores, dim=-1, dtype=torch.float32
+ )
+ # Free attention_scores immediately
+ del attention_scores
+
+ index_scores_softmax = torch.nn.functional.softmax(index_scores, dim=-1, dtype=torch.float32)
+ # Free index_scores - no longer needed after softmax
+ del index_scores
+
+ # Sum attention scores across heads: [b, np, sq, sk] -> [b, sq, sk]
+ attention_scores_sum = attention_scores_softmax.sum(dim=1)
+ # Free attention_scores_softmax
+ del attention_scores_softmax
+
+ if pg_collection.tp.size() > 1:
+ # attention scores are scattered to TP ranks in head dimension.
+ torch.distributed.all_reduce(attention_scores_sum.contiguous(), group=pg_collection.tp)
+
+ # L1 normalize
+ attention_scores_normalized = attention_scores_sum / attention_scores_sum.sum(
+ dim=-1, keepdim=True
+ )
+ # Free attention_scores_sum - no longer needed after normalization
+ del attention_scores_sum
+
+ # Backward through loss = kl_div * loss_coeff
+ # where kl_div = kl_per_element.sum(dim=-1).mean()
+ grad_kl_div = grad_loss * loss_coeff # scalar
+
+ # Backward through mean: distribute gradient equally
+ grad_kl_per_row = grad_kl_div / (b * sq) # scalar value for each row
+
+ # Backward through sum(dim=-1): broadcast back to [b, sq, sk]
+ # Each element in a row contributes to the sum, so gradient is same for all
+ grad_kl_per_element = grad_kl_per_row.view(1, 1, 1).expand(b, sq, sk)
+
+ # Backward through kl_per_element = target * (log(target) - log(index))
+ # ∂kl/∂index_softmax = -target / index_softmax
+ grad_index_scores_softmax = (
+ -attention_scores_normalized / (index_scores_softmax + 1e-10) * grad_kl_per_element
+ )
+ # Free attention_scores_normalized - no longer needed
+ del attention_scores_normalized
+
+ # Backward through softmax: ∂L/∂x = softmax * (∂L/∂softmax - sum(∂L/∂softmax * softmax))
+ sum_grad = (grad_index_scores_softmax * index_scores_softmax).sum(dim=-1, keepdim=True)
+ grad_index_scores_logits = index_scores_softmax * (grad_index_scores_softmax - sum_grad)
+ # Free intermediate tensors
+ del index_scores_softmax, grad_index_scores_softmax, sum_grad
+
+ # Zero out gradients for masked positions
+ # Create a mask for valid (non-masked) positions
+ # Causal mask: position (i, j) is valid if j <= i
+ causal_valid_mask = torch.tril(
+ torch.ones((sq, sk), device=q.device, dtype=torch.bool)
+ ) # [sq, sk]
+ if sparse_loss:
+ # Also apply index mask - only topk positions are valid
+ index_valid_mask = index_mask == 0 # [b, sq, sk]
+ del index_mask # Free index_mask immediately after use
+ valid_mask = causal_valid_mask.unsqueeze(0) & index_valid_mask # [b, sq, sk]
+ del index_valid_mask
+ else:
+ del index_mask # Free index_mask even if not used for sparse_loss
+ valid_mask = causal_valid_mask.unsqueeze(0).expand(b, sq, sk) # [b, sq, sk]
+ del causal_valid_mask
+
+ grad_index_scores_logits = grad_index_scores_logits * valid_mask.float()
+ del valid_mask
+
+ # Transpose from [b, sq, sk] to [sq, b, sk]
+ grad_index_scores = grad_index_scores_logits.transpose(0, 1) # [sq, b, sk]
+ del grad_index_scores_logits
+
+ # Backward through sum over heads: expand gradient
+ grad_weighted_scores = grad_index_scores.unsqueeze(2) # [sq, b, 1, sk]
+ del grad_index_scores
+
+ # Compute forward values needed for backward
+ scores = torch.einsum('sbhd,tbd->sbht', q.float(), k.float()) # [sq, b, h, sk]
+ # Compute relu_mask before relu (saves memory vs keeping both scores and relu output)
+ relu_mask = scores > 0
+ scores_after_relu = torch.relu(scores)
+ del scores
+
+ # Backward through multiplication by weights: index_scores_per_head * weights
+ # ∂L/∂weights = grad * relu_scores (sum over sk)
+ grad_weights = (grad_weighted_scores * scores_after_relu).sum(dim=-1) # [sq, b, h]
+
+ # ∂L/∂relu_scores = grad * weights
+ grad_scores_after_relu = grad_weighted_scores * weights.unsqueeze(-1) # [sq, b, h, sk]
+ del grad_weighted_scores, scores_after_relu
+
+ # Backward through ReLU
+ grad_scores = grad_scores_after_relu * relu_mask.float() # [sq, b, h, sk]
+ del grad_scores_after_relu, relu_mask
+
+ # Backward through einsum 'sbhd,tbd->sbht'
+ # ∂L/∂q = einsum('sbht,tbd->sbhd', grad_scores, k)
+ grad_q = torch.einsum('sbht,tbd->sbhd', grad_scores, k.float()) # [sq, b, h, d]
+ # ∂L/∂k = einsum('sbht,sbhd->tbd', grad_scores, q)
+ grad_k = torch.einsum('sbht,sbhd->tbd', grad_scores, q.float()) # [sk, b, d]
+ del grad_scores
+
+ return grad_q.to(q.dtype), grad_weights.to(weights.dtype), grad_k.to(k.dtype)
+
+
+class FusedDSAIndexerLoss(torch.autograd.Function):
+ """Fused implementation of DSA Indexer Loss."""
+
+ @staticmethod
+ def forward(
+ ctx,
+ q,
+ weights,
+ k,
+ query,
+ key,
+ softmax_scale,
+ topk,
+ loss_coeff,
+ mask,
+ sparse_loss,
+ pg_collection,
+ ):
+ """
+ Fused forward: index_scores never materialized in full.
+ """
+ topk_indices, loss = fwd_fused_indexer_loss_naive(
+ q,
+ weights,
+ k,
+ query,
+ key,
+ topk,
+ softmax_scale,
+ loss_coeff,
+ mask,
+ sparse_loss,
+ pg_collection,
+ )
+
+ # Save for backward (recomputation strategy)
+ ctx.save_for_backward(q, weights, k, query, key, topk_indices)
+ ctx.softmax_scale = softmax_scale
+ ctx.loss_coeff = loss_coeff
+ ctx.sparse_loss = sparse_loss
+ ctx.pg_collection = pg_collection
+
+ return topk_indices, loss
+
+ @staticmethod
+ def backward(ctx, grad_topk_indices, grad_loss):
+ """
+ Backward: Recompute what we need.
+ """
+ q, weights, k, query, key, topk_indices = ctx.saved_tensors
+
+ grad_q, grad_weights, grad_k = bwd_fused_indexer_loss_naive(
+ q,
+ weights,
+ k,
+ query,
+ key,
+ topk_indices,
+ ctx.softmax_scale,
+ ctx.loss_coeff,
+ ctx.sparse_loss,
+ grad_loss,
+ ctx.pg_collection,
+ )
+
+ # query and key are detached in forward, so return None for their gradients
+ return grad_q, grad_weights, grad_k, None, None, None, None, None, None, None, None
+
+
class DSAIndexerLossAutoScaler(torch.autograd.Function):
"""An AutoScaler that triggers the backward pass and scales the grad for indexer loss.
@@ -471,74 +795,10 @@ def _apply_rope(self, x: torch.Tensor, rotary_pos_emb: torch.Tensor, mscale: flo
x = torch.cat([x_nope, x_pe], dim=-1)
return x
- def _compute_index_scores(
- self, q: torch.Tensor, weights: torch.Tensor, k: torch.Tensor
- ) -> torch.Tensor:
- """
- Perform index score using BF16 precision.
-
- Reference:
- https://github.com/deepseek-ai/DeepSeek-V3.2-Exp/blob/main/inference/kernel.py#L254-L274
- This is a BF16 implementation of the `fp8_index` logic:
- 1. Compute attention scores: q @ k^T;
- 2. Apply ReLU activation;
- 3. Weight by attention weights;
- 4. Sum across attention heads.
-
- Args:
- q: BF16 [seqlen_q, batch, index_n_heads, index_head_dim], the query tensor.
- weights: BF16 [seqlen_q, batch, index_n_heads], the attention weights.
- k: BF16 [seqlen_k, batch, index_head_dim], the key tensor.
-
- Returns:
- index_scores: FP32 [batch, seqlen_q, seqlen_k], the index scores.
- """
- # Compute attention scores: q @ k^T
- # [seqlen_q, batch, index_n_heads, index_head_dim] @ [seqlen_k, batch, index_head_dim]^T
- # -> [seqlen_q, batch, index_n_heads, seqlen_k]
- index_scores = torch.einsum('sbhd,tbd->sbht', q.float(), k.float())
-
- # Apply ReLU activation.
- index_scores = torch.relu(index_scores)
-
- # Weight each head by attention weights.
- # [seqlen_q, batch, index_n_heads, seqlen_k] * [seqlen_q, batch, index_n_heads, 1]
- # -> [seqlen_q, batch, index_n_heads, seqlen_k]
- index_scores = index_scores * weights.unsqueeze(-1)
-
- # Sum across attention heads.
- # [seqlen_q, batch, index_n_heads, seqlen_k] -> [seqlen_q, batch, seqlen_k]
- index_scores = index_scores.sum(dim=2)
-
- # Transpose to [batch, seqlen_q, seqlen_k].
- index_scores = index_scores.transpose(0, 1)
-
- return index_scores
-
- def forward_with_scores(
- self,
- x: torch.Tensor,
- qr: torch.Tensor,
- mask: Optional[torch.Tensor] = None,
- packed_seq_params: Optional[PackedSeqParams] = None,
+ def forward_before_topk(
+ self, x: torch.Tensor, qr: torch.Tensor, packed_seq_params: Optional[PackedSeqParams] = None
) -> Tuple[torch.Tensor, torch.Tensor]:
- """
- Forward pass for DSA Indexer that returns both index scores and top-k indices.
-
- This is used when KL loss is enabled to compare indexer scores with true attention scores.
-
- Args:
- x: hidden states [seqlen, batch, hidden_size].
- qr: Low-rank query tensor [seqlen, batch, q_lora_rank].
- mask: Attention mask [batch, seqlen, seqlen].
- packed_seq_params: Packed sequence parameters for variable length sequences.
-
- Returns:
- index_scores: Index scores [batch, seqlen, seqlen].
- topk_indices: Top-k indices [batch, seqlen, index_topk].
- """
- assert packed_seq_params is None, "Packed sequence is not supported for DSAttention"
-
+ """All computations before topk."""
# =========================================
# Prepare RoPE params
# =========================================
@@ -592,23 +852,45 @@ def forward_with_scores(
k = rotate_activation(k)
# =========================================
- # Compute index scores
+ # Prepare weights for index scores
# =========================================
# [seqlen, batch, hidden_size] -> [seqlen, batch, index_n_heads]
weights, _ = self.linear_weights_proj(x)
weights = weights * (self.index_n_heads**-0.5) * self.softmax_scale
- # [batch, seqlen, seqlen]
- index_scores = self._compute_index_scores(q, weights, k)
- if mask is not None:
- assert mask.dtype == index_scores.dtype, "Mask dtype must match index scores dtype"
- index_scores = index_scores + mask
- # =========================================
- # Select top-k indices
- # =========================================
- topk_k = min(self.index_topk, seqlen)
- # [batch, seqlen, index_topk]
- topk_indices = index_scores.topk(topk_k, dim=-1)[1]
+ return q, k, weights
+
+ def forward_with_scores(
+ self,
+ x: torch.Tensor,
+ qr: torch.Tensor,
+ mask: Optional[torch.Tensor] = None,
+ packed_seq_params: Optional[PackedSeqParams] = None,
+ ) -> Tuple[torch.Tensor, torch.Tensor]:
+ """
+ Forward pass for DSA Indexer that returns both index scores and top-k indices.
+
+ This is used when KL loss is enabled to compare indexer scores with true attention scores.
+
+ Args:
+ x: hidden states [seqlen, batch, hidden_size].
+ qr: Low-rank query tensor [seqlen, batch, q_lora_rank].
+ mask: Attention mask [batch, seqlen, seqlen].
+ packed_seq_params: Packed sequence parameters for variable length sequences.
+
+ Returns:
+ index_scores: Index scores [batch, seqlen, seqlen].
+ topk_indices: Top-k indices [batch, seqlen, index_topk].
+ """
+ assert packed_seq_params is None, "Packed sequence is not supported for DSAttention"
+
+ # [seqlen, batch, index_n_heads * index_head_dim]
+ # [seqlen, batch, index_head_dim]
+ # [seqlen, batch, index_n_heads]
+ q, k, weights = self.forward_before_topk(x, qr, packed_seq_params)
+
+ # [batch, seqlen, seqlen], [batch, seqlen, index_topk]
+ index_scores, topk_indices = fused_qk_topk_naive(q, k, weights, self.index_topk, mask)
return index_scores, topk_indices
@@ -781,31 +1063,27 @@ def forward(
mask, float('-inf')
)
- # ===================================
- # Get index scores and top-k indices
- # ===================================
- index_scores, topk_indices = self.indexer.forward_with_scores(
- x, qr, mask=float_mask, packed_seq_params=packed_seq_params
- )
-
- # ===================================
- # Run sparse attention kernel
- # ===================================
- output = unfused_dsa_fn(query, key, value, topk_indices, self.softmax_scale)
-
- # ===================================
- # Attach indexer loss
- # ===================================
if self.training and torch.is_grad_enabled():
- # Compute KL divergence loss between indexer scores and true attention scores
+ # ===================================
+ # Prepare inputs for indexer loss
+ # ===================================
+ q, k, weights = self.indexer.forward_before_topk(x, qr, packed_seq_params)
indexer_loss_coeff = getattr(self.config, 'dsa_indexer_loss_coeff', 0.0)
- indexer_loss = compute_dsa_indexer_loss(
- index_scores,
- topk_indices,
+
+ # ===================================
+ # Attach indexer topk and loss
+ # ===================================
+ # Compute KL divergence loss between indexer scores and true attention scores
+ topk_indices, indexer_loss = FusedDSAIndexerLoss.apply(
+ q,
+ weights,
+ k,
query.detach(),
key.detach(),
self.softmax_scale,
+ self.indexer.index_topk,
indexer_loss_coeff,
+ float_mask,
getattr(self.config, "dsa_indexer_use_sparse_loss", False),
self.indexer.pg_collection,
)
@@ -816,7 +1094,26 @@ def forward(
layer_number=self.layer_number,
num_layers=self.config.num_layers,
)
+
+ # ===================================
+ # Run sparse attention kernel
+ # ===================================
+ output = unfused_dsa_fn(query, key, value, topk_indices, self.softmax_scale)
+
# Attach loss to output
output = DSAIndexerLossAutoScaler.apply(output, indexer_loss)
+ else:
+ # ===================================
+ # Get index scores and top-k indices
+ # ===================================
+ _, topk_indices = self.indexer.forward_with_scores(
+ x, qr, mask=float_mask, packed_seq_params=packed_seq_params
+ )
+
+ # ===================================
+ # Run sparse attention kernel
+ # ===================================
+ output = unfused_dsa_fn(query, key, value, topk_indices, self.softmax_scale)
+
return output
diff --git a/megatron/core/transformer/fsdp_dtensor_checkpoint.py b/megatron/core/transformer/fsdp_dtensor_checkpoint.py
index f7a938aff2a..e408209c778 100644
--- a/megatron/core/transformer/fsdp_dtensor_checkpoint.py
+++ b/megatron/core/transformer/fsdp_dtensor_checkpoint.py
@@ -45,6 +45,7 @@
from megatron.core import parallel_state
from megatron.core.tensor_parallel.layers import copy_tensor_model_parallel_attributes
from megatron.core.transformer.transformer_layer import TransformerLayer
+from megatron.core.utils import get_attr_wrapped_model
def get_ep_layer_offset(num_experts: int | None = None) -> int:
@@ -183,7 +184,10 @@ def handle_swiglu_in_state_dict(model, model_state_dict, optimizer_state_dict):
assert HAVE_MEGATRON_FSDP, "This function requires Megatron-FSDP to be installed."
# Extract num_experts from model config for expert parameter processing
- num_experts = model.config.num_moe_experts if hasattr(model, 'config') else None
+ model_config = get_attr_wrapped_model(model, "config", allow_none=True)
+ num_experts = (
+ getattr(model_config, 'num_moe_experts', None) if model_config is not None else None
+ )
def intersection(s1, s2):
# Only works for step=1
diff --git a/megatron/core/transformer/heterogeneous/linear_replacements.py b/megatron/core/transformer/heterogeneous/linear_replacements.py
index 78ac8d37df9..9cf56d62205 100644
--- a/megatron/core/transformer/heterogeneous/linear_replacements.py
+++ b/megatron/core/transformer/heterogeneous/linear_replacements.py
@@ -3,6 +3,7 @@
import torch.nn.functional as F
from torch import Tensor
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.parallel_state import (
get_tensor_model_parallel_rank,
get_tensor_model_parallel_world_size,
@@ -15,14 +16,10 @@
from megatron.core.transformer.transformer_config import TransformerConfig
from megatron.core.utils import divide
-try:
- import transformer_engine as te # pylint: disable=unused-import
-
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import TELayerNormColumnParallelLinear
-
- HAVE_TE = True
-except ImportError:
- HAVE_TE = False
+else:
+ TELayerNormColumnParallelLinear = None
def _gather_from_tensor_parallel_region(x: Tensor, config: TransformerConfig) -> Tensor:
diff --git a/megatron/core/transformer/mlp.py b/megatron/core/transformer/mlp.py
index 729c280a73a..1ece7f2b790 100644
--- a/megatron/core/transformer/mlp.py
+++ b/megatron/core/transformer/mlp.py
@@ -12,12 +12,14 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
+from __future__ import annotations
import gc
import logging
import warnings
+from collections.abc import Callable
from dataclasses import dataclass
-from typing import Optional, Union
+from typing import Optional, Protocol, cast
import numpy as np
import torch
@@ -37,8 +39,8 @@
from megatron.core.fusions.fused_bias_gelu import bias_gelu_impl
from megatron.core.fusions.fused_bias_swiglu import bias_swiglu_impl, weighted_bias_swiglu_impl
from megatron.core.transformer.module import MegatronModule
-from megatron.core.transformer.spec_utils import ModuleSpec, build_module
from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.typed_torch import apply_module, not_none
from megatron.core.utils import (
get_tensor_model_parallel_group_if_none,
nvtx_range_pop,
@@ -56,7 +58,91 @@
logger = logging.getLogger(__name__)
-# pylint: disable=missing-class-docstring
+class LinearFc1Interface(Protocol):
+ """Interface for linear_fc1 module in MLP."""
+
+ def forward(self, hidden_states: torch.Tensor, /) -> tuple[torch.Tensor, torch.Tensor | None]:
+ """Forward method for linear_fc1 module."""
+ ...
+
+ def backward_dw(self) -> None:
+ """Backward method for linear_fc1 module."""
+ ...
+
+
+class LinearFc1Builder(Protocol):
+ """Protocol describing how to build a linear_fc1 layer in MLP."""
+
+ def __call__(
+ self,
+ input_size: int,
+ output_size: int,
+ /,
+ *,
+ config: TransformerConfig,
+ init_method: Callable[[torch.Tensor], None],
+ gather_output: bool,
+ bias: bool,
+ skip_bias_add: bool,
+ is_expert: bool,
+ tp_comm_buffer_name: str | None,
+ tp_group: torch.distributed.ProcessGroup | None,
+ stride: int = 1,
+ ) -> LinearFc1Interface:
+ """Builds a linear_fc1 layer for MLP."""
+ ...
+
+
+class TEActivationFunctionInterface(Protocol):
+ """Interface for activation_function module in MLP."""
+
+ def forward(self, input_: torch.Tensor, /) -> torch.Tensor:
+ """Forward method for activation_function module."""
+ ...
+
+
+class TEActivationFunctionBuilder(Protocol):
+ """Protocol for activation_function module in MLP."""
+
+ def __call__(self, *, config: TransformerConfig) -> TEActivationFunctionInterface:
+ """Builds an activation function module for MLP."""
+ ...
+
+
+class LinearFc2Interface(Protocol):
+ """Interface for linear_fc2 module in MLP."""
+
+ def forward(self, hidden_states: torch.Tensor, /) -> tuple[torch.Tensor, torch.Tensor | None]:
+ """Forward method for linear_fc2 module."""
+ ...
+
+ def backward_dw(self) -> None:
+ """Backward method for linear_fc2 module."""
+ ...
+
+
+class LinearFc2Builder(Protocol):
+ """Protocol describing how to build a linear_fc2 layer in MLP."""
+
+ def __call__(
+ self,
+ input_size: int,
+ output_size: int,
+ /,
+ *,
+ config: TransformerConfig,
+ init_method: Callable[[torch.Tensor], None],
+ bias: bool,
+ input_is_parallel: bool,
+ skip_bias_add: bool,
+ is_expert: bool,
+ tp_comm_buffer_name: str | None,
+ tp_group: torch.distributed.ProcessGroup | None,
+ ) -> LinearFc2Interface:
+ """Builds a linear_fc2 layer for MLP."""
+ ...
+
+
@dataclass
class MLPSubmodules:
"""
@@ -64,9 +150,14 @@ class MLPSubmodules:
including linear fc1, activation function, linear fc2.
"""
- linear_fc1: Union[ModuleSpec, type] = None
- activation_func: Union[ModuleSpec, type] = None
- linear_fc2: Union[ModuleSpec, type] = None
+ linear_fc1: LinearFc1Builder
+
+ linear_fc2: LinearFc2Builder
+
+ activation_func: TEActivationFunctionBuilder | None = None
+ """
+ Builder for an activation function module; only used if config.use_te_activation_func is True.
+ """
class MLP(MegatronModule):
@@ -112,14 +203,7 @@ def __init__(
DeprecationWarning,
stacklevel=2,
)
- # If this is a gated linear unit we double the output width, see https://arxiv.org/pdf/2002.05202.pdf
- if is_expert:
- ffn_hidden_size = self.config.ffn_hidden_size if not self.config.moe_ffn_hidden_size else self.config.moe_ffn_hidden_size
- else:
- if not is_shared_expert:
- ffn_hidden_size = self.config.ffn_hidden_size
- else:
- ffn_hidden_size = self.config.moe_ffn_hidden_size * self.config.num_shared_experts
+ ffn_hidden_size = not_none(self.config.ffn_hidden_size)
# For GLU/SwiGLU, use stride=2 because each TP rank stores interleaved [gate, up] portions.
# This is critical for correct weight resharding across different TP sizes.
@@ -137,12 +221,11 @@ def __init__(
# shared_experts.
use_latent_size = (self.config.moe_latent_size is not None) and is_expert
- self.linear_fc1 = build_module(
- submodules.linear_fc1,
- self.input_size if not use_latent_size else self.config.moe_latent_size,
+ self.linear_fc1 = submodules.linear_fc1(
+ self.input_size if not use_latent_size else not_none(self.config.moe_latent_size),
ffn_hidden_size,
config=self.config,
- init_method=self.config.init_method,
+ init_method=not_none(self.config.init_method),
gather_output=False,
bias=self.config.add_bias_linear,
skip_bias_add=True,
@@ -153,16 +236,17 @@ def __init__(
)
if self.config.use_te_activation_func and not (submodules.activation_func is None):
- self.activation_func = build_module(submodules.activation_func, config=self.config)
+ self.activation_func = apply_module(submodules.activation_func(config=self.config))
else:
self.activation_func = self.config.activation_func
- self.linear_fc2 = build_module(
- submodules.linear_fc2,
- self.config.ffn_hidden_size,
- self.config.hidden_size if not use_latent_size else self.config.moe_latent_size,
+ self.linear_fc2 = submodules.linear_fc2(
+ not_none(self.config.ffn_hidden_size),
+ not_none(
+ self.config.hidden_size if not use_latent_size else self.config.moe_latent_size
+ ),
config=self.config,
- init_method=self.config.output_layer_init_method,
+ init_method=not_none(self.config.output_layer_init_method),
bias=self.config.add_bias_linear,
input_is_parallel=True,
skip_bias_add=True,
@@ -171,11 +255,13 @@ def __init__(
tp_group=tp_group,
)
- def forward(self, hidden_states, per_token_scale=None, **kwargs):
+ def forward(
+ self, hidden_states: torch.Tensor, per_token_scale: torch.Tensor | None = None, **kwargs
+ ):
"""Perform the forward pass through the MLP block."""
# [s, b, 4 * h/p]
nvtx_range_push(suffix="linear_fc1")
- intermediate_parallel, bias_parallel = self.linear_fc1(hidden_states)
+ intermediate_parallel, bias_parallel = apply_module(self.linear_fc1)(hidden_states)
nvtx_range_pop(suffix="linear_fc1")
nvtx_range_push(suffix="activation")
@@ -257,7 +343,9 @@ def glu(x):
# [s, b, h]
nvtx_range_push(suffix="linear_fc2")
- output, output_bias = self.linear_fc2(intermediate_parallel)
+ output, output_bias = apply_module(self.linear_fc2)(
+ cast(torch.Tensor, intermediate_parallel)
+ )
nvtx_range_pop(suffix="linear_fc2")
if per_token_scale is not None and output_bias is not None:
diff --git a/megatron/core/transformer/module.py b/megatron/core/transformer/module.py
index c30c107e791..6539ee36105 100644
--- a/megatron/core/transformer/module.py
+++ b/megatron/core/transformer/module.py
@@ -10,6 +10,7 @@
from megatron.core import parallel_state
from megatron.core.dist_checkpointing.mapping import ShardedStateDict
+from megatron.core.transformer.enums import CudaGraphScope
from megatron.core.transformer.transformer_config import TransformerConfig
from megatron.core.transformer.utils import (
ensure_metadata_has_dp_cp_group,
@@ -168,7 +169,10 @@ def __init__(self, config: TransformerConfig, vp_stage: Optional[int] = None):
assert isinstance(config, TransformerConfig), "config must be a TransformerConfig"
# Enable cuda graphs.
- if config.cuda_graph_impl == "local":
+ if (
+ config.cuda_graph_impl == "local"
+ and CudaGraphScope.full_iteration not in config.cuda_graph_scope
+ ):
if hasattr(self, "create_mcore_cudagraph_manager"):
self.create_mcore_cudagraph_manager(config)
else:
diff --git a/megatron/core/transformer/moe/README.md b/megatron/core/transformer/moe/README.md
index e5eff445cac..558e2ccd39a 100644
--- a/megatron/core/transformer/moe/README.md
+++ b/megatron/core/transformer/moe/README.md
@@ -687,7 +687,7 @@ torchrun ${DISTRIBUTED_ARGS[@]} pretrain_gpt.py \
## Contributing
-We welcome contributions! Please see [CONTRIBUTING.md](../../../../CONTRIBUTING.md) for guidelines.
+We welcome contributions! Please see [CONTRIBUTING.md](https://github.com/NVIDIA/Megatron-LM/blob/main/CONTRIBUTING.md) for guidelines.
## Support
diff --git a/megatron/core/transformer/moe/experts.py b/megatron/core/transformer/moe/experts.py
index 21efa6e7390..2d96fec4caf 100644
--- a/megatron/core/transformer/moe/experts.py
+++ b/megatron/core/transformer/moe/experts.py
@@ -1,512 +1,156 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+from __future__ import annotations
-import copy
import logging
+from collections.abc import Callable
from copy import deepcopy
-from functools import partial
+from dataclasses import dataclass
from math import ceil
-from typing import Optional, Tuple
+from typing import Optional, Protocol, Tuple
import torch
import torch.nn.functional as F
-from torch.nn.parameter import Parameter
from megatron.core import tensor_parallel
from megatron.core.activations import squared_relu
-from megatron.core.dist_checkpointing import ShardedTensor
-from megatron.core.dist_checkpointing.mapping import (
- LocalNonpersistentObject,
- ReplicaId,
- ShardedStateDict,
- ShardedTensorFactory,
-)
+from megatron.core.dist_checkpointing.mapping import ShardedStateDict
from megatron.core.dist_checkpointing.utils import replace_prefix_for_sharding
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.fusions.fused_bias_geglu import quick_gelu, weighted_bias_quick_geglu_impl
from megatron.core.fusions.fused_bias_swiglu import weighted_bias_swiglu_impl
from megatron.core.fusions.fused_weighted_squared_relu import weighted_squared_relu_impl
-from megatron.core.jit import jit_fuser
+from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
from megatron.core.pipeline_parallel.fine_grained_activation_offload import (
FineGrainedActivationOffloadingInterface as off_interface,
)
-from megatron.core.tensor_parallel.layers import (
- _initialize_affine_weight_cpu,
- _initialize_affine_weight_gpu,
+from megatron.core.transformer.mlp import (
+ MLP,
+ MLPSubmodules,
+ TEActivationFunctionBuilder,
+ apply_swiglu_sharded_factory,
)
-from megatron.core.tensor_parallel.utils import divide
-from megatron.core.transformer.mlp import MLP, MLPSubmodules, apply_swiglu_sharded_factory
from megatron.core.transformer.module import MegatronModule
-from megatron.core.transformer.moe import grouped_gemm_util as gg
from megatron.core.transformer.moe.moe_utils import (
ProcessGroupCollection,
get_align_size_for_quantization,
)
-from megatron.core.transformer.spec_utils import build_module
from megatron.core.transformer.transformer_config import TransformerConfig
from megatron.core.transformer.utils import (
ensure_metadata_has_dp_cp_group,
- make_sharded_object_for_checkpoint,
sharded_state_dict_default,
)
+from megatron.core.typed_torch import apply_module, not_none
-try:
- import transformer_engine as te # pylint: disable=unused-import
-
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import Fp8Padding, Fp8Unpadding
+else:
+ Fp8Padding, Fp8Unpadding = None, None
- HAVE_TE = True
+try:
+ import flashinfer.fused_moe as fused_moe
+ from flashinfer.fused_moe.core import ActivationType
+ HAVE_FLASHINFER = True
except ImportError:
+ HAVE_FLASHINFER = False
- HAVE_TE = False
+from megatron.core.inference.moe import ActivationType as McoreActivationType
+from megatron.core.inference.moe import (
+ InferenceGroupedGemmBackend,
+ mcore_fused_moe,
+ resolve_inference_grouped_gemm_backend,
+)
logger = logging.getLogger(__name__)
-class GroupedMLP(MegatronModule):
- """An efficient implementation of the Experts layer using GroupedGEMM.
-
- Executes multiple experts in parallel to maximize computational efficiency.
- """
+class GroupedLinearFc1Interface(Protocol):
+ """Interface for linear_fc1 module in TEGroupedMLP."""
- # TODO(M4): breaking api, switched from pass in tp_group to pass in pg_collection.
- def __init__(
- self,
- num_local_experts: int,
- config: TransformerConfig,
- pg_collection: Optional[ProcessGroupCollection] = None,
- ):
- super().__init__(config=config)
- self.config: TransformerConfig = config
- self.num_local_experts = num_local_experts
- gg.assert_grouped_gemm_is_available()
- assert (
- config.add_bias_linear == False
- ), "bias not supported in Grouped GEMM yet, please set '--disable-bias-linear' instead."
- assert (
- config.moe_latent_size is None
- ), "MoE latent projection not supported in GroupedMLP yet."
-
- self.expert_parallel = config.expert_model_parallel_size > 1
- if self.config.gated_linear_unit:
- if self.config.activation_func not in (F.silu, F.gelu):
- raise ValueError("Activation function must be silu or gelu when using GroupedMLP.")
-
- @jit_fuser
- def glu(x):
- x = torch.chunk(x, 2, dim=-1)
- return self.config.activation_func(x[0]) * x[1]
-
- self.activation_func = glu
- else:
- self.activation_func = self.config.activation_func
- self.activation_recompute = (
- self.config.recompute_granularity == 'selective'
- and "moe_act" in self.config.recompute_modules
- )
- if self.activation_recompute and (self.config.fp8 or self.config.fp4):
- raise ValueError(
- "moe_act recompute for fp8 or fp4 cannot work with the legacy GroupedMLP."
- )
-
- @jit_fuser
- def activation_func_with_probs(x, probs):
- dtype = x.dtype
- res = self.activation_func(x) * probs
- return res.to(dtype)
-
- self.activation_func_with_probs = activation_func_with_probs
+ def forward(
+ self, permuted_local_hidden_states: torch.Tensor, tokens_per_expert: list[int], /
+ ) -> tuple[torch.Tensor, torch.Tensor | None]:
+ """Forward method for linear_fc1 module."""
+ ...
- self.ep_group = pg_collection.ep
- # use pg_collection.expt_tp_group as tensor parallel group in this module.
- self.tp_group = pg_collection.expt_tp
- # use pg_collection.expt_dp_group as data parallel group in this module.
- self.dp_group = pg_collection.expt_dp
- # How many feature each rank holds for fc1 and fc2, respectively.
- tp_size = self.tp_group.size()
- tp_rank = self.tp_group.rank()
-
- fc1_output_size = self.config.moe_ffn_hidden_size * self.num_local_experts
- if config.gated_linear_unit:
- # Project to 4h. If using swiglu double the output width,
- # see https://arxiv.org/pdf/2002.05202.pdf
- fc1_output_size *= 2
- fc1_output_size_per_partition = divide(fc1_output_size, tp_size)
-
- fc2_input_size = self.config.moe_ffn_hidden_size * self.num_local_experts
- fc2_input_size_per_partition = divide(fc2_input_size, tp_size)
-
- # Note: The current kernel implementations of grouped_gemm
- # does not support transposition with CUTLASS grouped GEMM
- # (https://github.com/fanshiqing/grouped_gemm/blob/main/csrc/grouped_gemm.cu#L355-L358)
- # and as a result we avoid allocate the transpose of weights.
- # Initialize weight.
- if config.use_cpu_initialization:
- self.weight1 = Parameter(
- torch.empty(
- self.config.hidden_size,
- fc1_output_size_per_partition,
- dtype=config.params_dtype,
- )
- )
- self.weight2 = Parameter(
- torch.empty(
- fc2_input_size_per_partition, self.config.hidden_size, dtype=config.params_dtype
- )
- )
- if config.perform_initialization:
- _initialize_affine_weight_cpu(
- self.weight1,
- self.config.hidden_size,
- fc1_output_size,
- fc1_output_size_per_partition,
- partition_dim=1,
- init_method=config.init_method,
- params_dtype=config.params_dtype,
- rank=tp_rank,
- world_size=tp_size,
- )
- _initialize_affine_weight_cpu(
- self.weight2,
- fc2_input_size,
- self.config.hidden_size,
- fc2_input_size_per_partition,
- partition_dim=0,
- init_method=config.output_layer_init_method,
- params_dtype=config.params_dtype,
- rank=tp_rank,
- world_size=tp_size,
- )
- else:
- self.weight1 = Parameter(
- torch.empty(
- self.config.hidden_size,
- fc1_output_size_per_partition,
- device=torch.cuda.current_device(),
- dtype=config.params_dtype,
- )
- )
- self.weight2 = Parameter(
- torch.empty(
- fc2_input_size_per_partition,
- self.config.hidden_size,
- device=torch.cuda.current_device(),
- dtype=config.params_dtype,
- )
- )
- if config.perform_initialization:
- _initialize_affine_weight_gpu(
- self.weight1, config.init_method, partition_dim=1, is_expert=True
- )
- _initialize_affine_weight_gpu(
- self.weight2, config.output_layer_init_method, partition_dim=0, is_expert=True
- )
- setattr(self.weight1, 'allreduce', not self.expert_parallel)
- setattr(self.weight2, 'allreduce', not self.expert_parallel)
+ def backward_dw(self) -> None:
+ """Backward method for linear_fc1 module."""
+ ...
- def remove_extra_states_check(self, incompatible_keys):
- """
- Remove _extra_state from unexpected keys.
- These keys are for dist ckpt compatibility with SequentialMLP.
- """
- keys = deepcopy(incompatible_keys.unexpected_keys)
- for key in keys:
- if '_extra_state' in key:
- incompatible_keys.unexpected_keys.remove(key)
- self.register_load_state_dict_post_hook(remove_extra_states_check)
+class GroupedLinearFc1Builder(Protocol):
+ """Protocol describing how to build a linear_fc1 layer in TEGroupedMLP."""
- def forward(
+ def __call__(
self,
- permuted_local_hidden_states: torch.Tensor,
- tokens_per_expert: torch.Tensor,
- permuted_probs: torch.Tensor,
- ):
- """Forward step of the GroupedMLP."""
- assert self.config.bf16, "Currently GroupedMLP for MoE only supports bf16."
- if self.activation_recompute:
- self.activation_checkpoint = tensor_parallel.CheckpointWithoutOutput()
+ num_local_experts: int,
+ input_size: int,
+ output_size: int,
+ /,
+ *,
+ config: TransformerConfig,
+ init_method: Callable[[torch.Tensor], None],
+ bias: bool,
+ skip_bias_add: bool,
+ is_expert: bool,
+ tp_comm_buffer_name: str | None,
+ pg_collection: ProcessGroupCollection | None,
+ ) -> GroupedLinearFc1Interface:
+ """Builds a linear_fc1 layer for TEGroupedMLP."""
+ ...
- if self.config.moe_apply_probs_on_input:
- assert (
- self.config.moe_router_topk == 1
- ), "`moe_apply_probs_on_input` only works with `moe_router_topk`=1."
- original_dtype = permuted_local_hidden_states.dtype
- permuted_local_hidden_states = (
- permuted_probs.unsqueeze(-1) * permuted_local_hidden_states
- )
- permuted_local_hidden_states = permuted_local_hidden_states.to(original_dtype)
- # Probs already applied, so reset to 1.
- permuted_probs = torch.ones_like(permuted_probs)
- if permuted_local_hidden_states.nelement() != 0:
- # Reshape the weights for the grouped GEMMs.
- w1 = self.weight1.view(self.num_local_experts, self.config.hidden_size, -1)
- w2 = self.weight2.view(self.num_local_experts, -1, self.config.hidden_size)
+class GroupedLinearFc2Interface(Protocol):
+ """Protocol for linear_fc2 module in TEGroupedMLP."""
- fc1_output = gg.ops.gmm(
- permuted_local_hidden_states, w1, tokens_per_expert, trans_b=False
- )
- if self.activation_recompute:
- intermediate_parallel = self.activation_checkpoint.checkpoint(
- self.activation_func_with_probs, fc1_output, permuted_probs.unsqueeze(-1)
- )
- fc2_output = gg.ops.gmm(intermediate_parallel, w2, tokens_per_expert, trans_b=False)
- self.activation_checkpoint.discard_output_and_register_recompute(fc2_output)
- else:
- intermediate_parallel = self.activation_func_with_probs(
- fc1_output, permuted_probs.unsqueeze(-1)
- )
- fc2_output = gg.ops.gmm(intermediate_parallel, w2, tokens_per_expert, trans_b=False)
- else:
- # No token is allocated for local experts.
- assert torch.count_nonzero(tokens_per_expert) == 0
-
- # Make sure params of experts still have gradients even given zero tokens.
- w1 = self.weight1.view(self.config.hidden_size, -1)
- w2 = self.weight2.view(-1, self.config.hidden_size)
- h = torch.matmul(permuted_local_hidden_states, w1)
- if self.activation_recompute:
- h = self.activation_checkpoint.checkpoint(
- self.activation_func_with_probs, h, permuted_probs.unsqueeze(-1)
- )
- fc2_output = torch.matmul(h, w2)
- self.activation_checkpoint.discard_output_and_register_recompute(fc2_output)
- else:
- h = self.activation_func_with_probs(h, permuted_probs.unsqueeze(-1))
- fc2_output = torch.matmul(h, w2)
+ def forward(
+ self, intermediate_parallel: torch.Tensor, tokens_per_expert: list[int], /
+ ) -> tuple[torch.Tensor, torch.Tensor | None]:
+ """Forward method for linear_fc2 module."""
+ ...
- return fc2_output, None
+ def backward_dw(self) -> None:
+ """Backward method for linear_fc2 module."""
+ ...
- def sharded_state_dict(self, prefix='', sharded_offsets=(), metadata=None):
- """
- Maps local expert to global experts.
- The sharded_state_dict for the weight parts are compatible with the SequentialMLP,
- whereas the optimizer states are not due to the limitation from weight transposing.
- That is, for finetuning scenario, the checkpoint is compatible with the SequentialMLP.
- When `singleton_local_shards` metadata flag is True, experts are broken down into
- separate tensors and stored under separate global keys. Additionally, similarly to MLP,
- layers with GLU activations are broken down into separate `w` and `v` tensors.
- """
- singleton_local_shards = (metadata or {}).get('singleton_local_shards', False)
- sharded_state_dict = {}
- ep_size = self.ep_group.size()
- ep_rank = self.ep_group.rank()
- tp_size = self.tp_group.size()
- tp_rank = self.tp_group.rank()
- dp_rank = self.dp_group.rank()
- num_global_experts = ep_size * self.num_local_experts
- local_expert_indices_offset = ep_rank * self.num_local_experts
-
- prepend_axis_num = len(sharded_offsets)
- replica_id = (0, 0, dp_rank)
-
- local_ffn_dim_size = (
- self.weight2.numel() // self.num_local_experts // self.config.hidden_size
- )
+class GroupedLinearFc2Builder(Protocol):
+ """Protocol describing how to build a linear_fc2 layer in TEGroupedMLP."""
- def _break_into_individual_experts(
- experts_ten: torch.Tensor,
- key: str,
- tp_offset: Tuple[int, int, int],
- replica_id: ReplicaId,
- ):
- """Breaks experts into individual tensors and stores them under separate global keys"""
- experts_state = []
- assert len(experts_ten) == self.num_local_experts, (
- experts_ten.shape,
- self.num_local_experts,
- )
- for local_expert_idx, expert_ten in enumerate(experts_ten):
- global_expert_idx = local_expert_indices_offset + local_expert_idx
- expert_key = key.replace(
- f'{prefix}experts.', f'{prefix}experts.{global_expert_idx}.'
- )
- experts_state.append(
- ShardedTensor.from_rank_offsets(
- expert_key,
- expert_ten.contiguous(),
- *sharded_offsets,
- tp_offset,
- replica_id=replica_id,
- prepend_axis_num=prepend_axis_num,
- )
- )
- return experts_state
-
- @torch.no_grad()
- def sh_ten_build_fn(
- key: str,
- t: torch.Tensor,
- replica_id: ReplicaId,
- flattened_range: Optional[slice],
- tp_axis: int,
- with_glu: bool,
- ):
- # TODO: write a generic implementation to cover both cases with and without GLU
- if tp_axis == 1:
- # weight1
- if with_glu:
- last_dim_size = local_ffn_dim_size * 2
- else:
- last_dim_size = local_ffn_dim_size
- real_shape = (self.num_local_experts, self.config.hidden_size, last_dim_size)
- elif tp_axis == 0:
- # weight2
- real_shape = (self.num_local_experts, local_ffn_dim_size, self.config.hidden_size)
- assert with_glu == False
- else:
- raise ValueError("tp_axis should be 0 or 1.")
- if flattened_range is None:
- # weights
- t = t.view(real_shape).transpose(-1, -2)
- # change tp_axis due to the transposing
- tp_axis = 1 - tp_axis
- if with_glu:
- assert tp_axis == 0, tp_axis
- if singleton_local_shards:
- w_tensor, v_tensor = torch.chunk(t, 2, -2)
- w_key = f'{key}_w'
- v_key = f'{key}_v'
- sub_states = {
- 'singleton_local_shards': LocalNonpersistentObject(True),
- 'data': {
- 'w': _break_into_individual_experts(
- w_tensor,
- w_key,
- (prepend_axis_num, tp_rank, tp_size),
- replica_id,
- ),
- 'v': _break_into_individual_experts(
- v_tensor,
- v_key,
- (prepend_axis_num, tp_rank, tp_size),
- replica_id,
- ),
- },
- }
- else:
- local_tensors = torch.chunk(t, 2, -2)
- sub_states = [
- ShardedTensor.from_rank_offsets(
- key,
- local_tensors[0].contiguous(),
- *sharded_offsets,
- (prepend_axis_num, ep_rank, ep_size),
- (prepend_axis_num + 1, tp_rank, tp_size * 2),
- replica_id=replica_id,
- prepend_axis_num=prepend_axis_num,
- ),
- ShardedTensor.from_rank_offsets(
- key,
- local_tensors[1].contiguous(),
- *sharded_offsets,
- (prepend_axis_num, ep_rank, ep_size),
- (prepend_axis_num + 1, tp_size + tp_rank, tp_size * 2),
- replica_id=replica_id,
- prepend_axis_num=prepend_axis_num,
- ),
- ]
- else:
- if singleton_local_shards:
- sub_states = {
- 'singleton_local_shards': LocalNonpersistentObject(True),
- 'data': _break_into_individual_experts(
- t, key, (prepend_axis_num + tp_axis, tp_rank, tp_size), replica_id
- ),
- }
- else:
- sub_states = ShardedTensor.from_rank_offsets(
- key,
- t.contiguous(),
- *sharded_offsets,
- (prepend_axis_num, ep_rank, ep_size),
- (prepend_axis_num + 1 + tp_axis, tp_rank, tp_size),
- replica_id=replica_id,
- prepend_axis_num=prepend_axis_num,
- )
- return sub_states # pylint: disable=possibly-used-before-assignment
-
- @torch.no_grad()
- def sh_ten_merge_fn(sub_state_dict, tp_axis: int, with_glu: bool):
- if tp_axis == 1:
- # weight1
- weight_shape = (self.config.hidden_size, -1)
- elif tp_axis == 0:
- # weight2
- weight_shape = (-1, self.config.hidden_size)
- assert with_glu == False
- else:
- raise ValueError("tp_axis should be 0 or 1.")
- if isinstance(sub_state_dict, dict):
- assert sub_state_dict['singleton_local_shards']
- if with_glu:
- assert isinstance(sub_state_dict['data'], dict)
- sub_state_dict = torch.cat(
- (
- torch.stack(sub_state_dict['data']['w']),
- torch.stack(sub_state_dict['data']['v']),
- ),
- dim=-2,
- )
- else:
- assert isinstance(sub_state_dict['data'], list)
- sub_state_dict = torch.stack(sub_state_dict['data'])
- else:
- if with_glu:
- sub_state_dict = torch.cat(sub_state_dict, -2)
- return sub_state_dict.transpose(-1, -2).reshape(weight_shape)
-
- state_dict = self.state_dict(prefix='', keep_vars=True)
- for name, tensor in state_dict.items():
- if name == 'weight1':
- tp_axis = 1
- with_glu = self.config.gated_linear_unit
- wkey = f'{prefix}experts.linear_fc1.weight'
- else:
- tp_axis = 0
- with_glu = False
- wkey = f'{prefix}experts.linear_fc2.weight'
-
- this_replica_id = list(copy.deepcopy(replica_id))
-
- sharded_state_dict[f'{prefix}{name}'] = ShardedTensorFactory(
- wkey,
- tensor,
- partial(sh_ten_build_fn, tp_axis=tp_axis, with_glu=with_glu),
- partial(sh_ten_merge_fn, tp_axis=tp_axis, with_glu=with_glu),
- tuple(this_replica_id),
- )
+ def __call__(
+ self,
+ num_local_experts: int,
+ input_size: int,
+ output_size: int,
+ /,
+ *,
+ config: TransformerConfig,
+ init_method: Callable[[torch.Tensor], None],
+ bias: bool,
+ skip_bias_add: bool,
+ is_expert: bool,
+ tp_comm_buffer_name: str | None,
+ pg_collection: ProcessGroupCollection | None,
+ ) -> GroupedLinearFc2Interface:
+ """Builds a linear_fc2 layer for TEGroupedMLP."""
+ ...
+
+
+@dataclass
+class GroupedMLPSubmodules:
+ """
+ The dataclass for ModuleSpecs of TEGroupedMLP submodules
+ including linear fc1, activation function, linear fc2.
+ """
- replica_id = (0, tp_rank, dp_rank)
- # Add fake _extra_state to be compatible with SequentialMLP
- for expert_local_idx in range(self.num_local_experts):
- expert_global_idx = local_expert_indices_offset + expert_local_idx
- if singleton_local_shards:
- expert_sharded_offsets = sharded_offsets
- else:
- expert_sharded_offsets = (
- *sharded_offsets,
- (len(sharded_offsets), expert_global_idx, num_global_experts),
- )
- for mod in ['linear_fc1', 'linear_fc2']:
- if singleton_local_shards:
- expert_key = f'{prefix}experts.{expert_global_idx}.{mod}._extra_state'
- else:
- expert_key = f'{prefix}experts.{mod}._extra_state'
- sharded_state_dict[f'{prefix}expert{expert_global_idx}.{mod}._extra_state'] = (
- make_sharded_object_for_checkpoint(
- None, expert_key, expert_sharded_offsets, replica_id
- )
- )
+ linear_fc1: GroupedLinearFc1Builder
- return sharded_state_dict
+ linear_fc2: GroupedLinearFc2Builder
- def backward_dw(self):
- """Performs backward pass for weight gradients in Experts.
- Empty implementation for compatibility with SequentialMLP and TEGroupedMLP.
- """
- pass
+ activation_func: TEActivationFunctionBuilder | None = None
+ """
+ Builder for an activation function module; only used if config.use_te_activation_func is True.
+ """
class TEGroupedMLP(MegatronModule):
@@ -518,9 +162,9 @@ class TEGroupedMLP(MegatronModule):
# TODO(M4): breaking api, switched from pass in tp_group to pass in pg_collection.
def __init__(
self,
- num_local_experts,
+ num_local_experts: int,
config: TransformerConfig,
- submodules: MLPSubmodules,
+ submodules: GroupedMLPSubmodules,
pg_collection: Optional[ProcessGroupCollection] = None,
):
super().__init__(config=config)
@@ -534,17 +178,16 @@ def __init__(
self.tp_group = pg_collection.expt_tp
# Double the output width with gated linear unit, see https://arxiv.org/pdf/2002.05202.pdf
- ffn_hidden_size = self.config.moe_ffn_hidden_size
+ ffn_hidden_size = not_none(self.config.moe_ffn_hidden_size)
if self.config.gated_linear_unit:
ffn_hidden_size *= 2
- self.linear_fc1 = build_module(
- submodules.linear_fc1,
+ self.linear_fc1 = submodules.linear_fc1(
self.num_local_experts,
self.input_size if self.config.moe_latent_size is None else self.config.moe_latent_size,
ffn_hidden_size,
config=self.config,
- init_method=self.config.init_method,
+ init_method=not_none(self.config.init_method),
bias=self.config.add_bias_linear,
skip_bias_add=False,
is_expert=True,
@@ -553,21 +196,20 @@ def __init__(
)
if self.config.use_te_activation_func and not (submodules.activation_func is None):
- self.activation_func = build_module(submodules.activation_func, config=self.config)
+ self.activation_func = apply_module(submodules.activation_func(config=self.config))
else:
self.activation_func = self.config.activation_func
- self.linear_fc2 = build_module(
- submodules.linear_fc2,
+ self.linear_fc2 = submodules.linear_fc2(
self.num_local_experts,
- self.config.moe_ffn_hidden_size,
+ not_none(self.config.moe_ffn_hidden_size),
(
self.config.hidden_size
if self.config.moe_latent_size is None
else self.config.moe_latent_size
),
config=self.config,
- init_method=self.config.output_layer_init_method,
+ init_method=not_none(self.config.output_layer_init_method),
bias=self.config.add_bias_linear,
skip_bias_add=True,
is_expert=True,
@@ -625,6 +267,63 @@ def _apply_bias(intermediate_parallel, bias_parallel, tokens_per_expert, permute
.to(intermediate_parallel.dtype)
)
+ def bias_act_func(self, intermediate_parallel, bias_parallel, permuted_probs):
+ """
+ Applies bias and activation function to the output of linear_fc1.
+ """
+ if self.config.use_te_activation_func:
+ if bias_parallel is not None:
+ intermediate_parallel = intermediate_parallel + bias_parallel
+ intermediate_parallel = self.activation_func(intermediate_parallel)
+ if permuted_probs is not None:
+ original_dtype = intermediate_parallel.dtype
+ intermediate_parallel = intermediate_parallel * permuted_probs
+ intermediate_parallel = intermediate_parallel.to(original_dtype)
+ elif self.config.bias_activation_fusion:
+ if self.activation_func == F.silu and self.config.gated_linear_unit:
+ # dtype is handled inside the fused kernel
+ intermediate_parallel = weighted_bias_swiglu_impl(
+ intermediate_parallel,
+ bias_parallel,
+ permuted_probs,
+ self.config.activation_func_fp8_input_store,
+ )
+ elif self.activation_func == quick_gelu and self.config.gated_linear_unit:
+ intermediate_parallel = weighted_bias_quick_geglu_impl(
+ intermediate_parallel,
+ bias_parallel,
+ permuted_probs,
+ self.config.activation_func_fp8_input_store,
+ self.config.glu_linear_offset,
+ self.config.activation_func_clamp_value,
+ )
+ else:
+ raise ValueError("Only support fusion of swiglu and quick_gelu in TEGroupedMLP.")
+ elif self.activation_func == squared_relu and self.config.use_fused_weighted_squared_relu:
+ assert bias_parallel is None, "Bias is not supported with fused weighted squared relu."
+ intermediate_parallel = weighted_squared_relu_impl(
+ intermediate_parallel, permuted_probs
+ )
+ else:
+ if self.config.gated_linear_unit:
+
+ def glu(x):
+ x_glu, x_linear = torch.chunk(x, 2, dim=-1)
+ if (val := self.config.activation_func_clamp_value) is not None:
+ x_glu = x_glu.clamp(min=None, max=val)
+ x_linear = x_linear.clamp(min=-val, max=val)
+ return self.config.activation_func(x_glu) * (
+ x_linear + self.config.glu_linear_offset
+ )
+
+ intermediate_parallel = glu(intermediate_parallel)
+ else:
+ intermediate_parallel = self.activation_func(intermediate_parallel)
+ original_dtype = intermediate_parallel.dtype
+ intermediate_parallel = intermediate_parallel * permuted_probs
+ intermediate_parallel = intermediate_parallel.to(original_dtype)
+ return intermediate_parallel
+
def forward(
self,
permuted_local_hidden_states: torch.Tensor,
@@ -645,7 +344,7 @@ def forward(
tokens_per_expert_gpu = None
if isinstance(tokens_per_expert, tuple):
tokens_per_expert, tokens_per_expert_gpu = tokens_per_expert
- tokens_per_expert = tokens_per_expert.tolist()
+ tokens_per_expert: list[int] = tokens_per_expert.tolist()
if self.config.fp8 or self.config.fp4:
actual_tokens_per_expert = tokens_per_expert
permuted_local_hidden_states, tokens_per_expert = self.quantization_padding(
@@ -670,7 +369,7 @@ def forward(
with off_interface(
self.offload_expert_fc1, permuted_local_hidden_states, "expert_fc1"
) as permuted_local_hidden_states:
- fc1_output, bias_parallel = self.linear_fc1(
+ fc1_output, bias_parallel = apply_module(self.linear_fc1)(
permuted_local_hidden_states, tokens_per_expert, tokens_per_expert_gpu
)
if self.offload_expert_fc1:
@@ -680,75 +379,16 @@ def forward(
forced_released_tensors=[permuted_local_hidden_states],
)
- def bias_act_func(intermediate_parallel, bias_parallel, permuted_probs):
- if self.config.use_te_activation_func:
- if bias_parallel is not None:
- intermediate_parallel = intermediate_parallel + bias_parallel
- intermediate_parallel = self.activation_func(intermediate_parallel)
- if permuted_probs is not None:
- original_dtype = intermediate_parallel.dtype
- intermediate_parallel = intermediate_parallel * permuted_probs
- intermediate_parallel = intermediate_parallel.to(original_dtype)
- elif self.config.bias_activation_fusion:
- if self.activation_func == F.silu and self.config.gated_linear_unit:
- # dtype is handled inside the fused kernel
- intermediate_parallel = weighted_bias_swiglu_impl(
- intermediate_parallel,
- bias_parallel,
- permuted_probs,
- self.config.activation_func_fp8_input_store,
- )
- elif self.activation_func == quick_gelu and self.config.gated_linear_unit:
- intermediate_parallel = weighted_bias_quick_geglu_impl(
- intermediate_parallel,
- bias_parallel,
- permuted_probs,
- self.config.activation_func_fp8_input_store,
- self.config.glu_linear_offset,
- self.config.activation_func_clamp_value,
- )
- else:
- raise ValueError(
- "Only support fusion of swiglu and quick_gelu in TEGroupedMLP."
- )
- elif (
- self.activation_func == squared_relu and self.config.use_fused_weighted_squared_relu
- ):
- assert bias_parallel is None
- intermediate_parallel = weighted_squared_relu_impl(
- intermediate_parallel, permuted_probs
- )
- else:
- if self.config.gated_linear_unit:
-
- def glu(x):
- x_glu, x_linear = torch.chunk(x, 2, dim=-1)
- if (val := self.config.activation_func_clamp_value) is not None:
- x_glu = x_glu.clamp(min=None, max=val)
- x_linear = x_linear.clamp(min=-val, max=val)
- return self.config.activation_func(x_glu) * (
- x_linear + self.config.glu_linear_offset
- )
-
- intermediate_parallel = glu(intermediate_parallel)
- else:
- intermediate_parallel = self.activation_func(intermediate_parallel)
- original_dtype = intermediate_parallel.dtype
- intermediate_parallel = intermediate_parallel * permuted_probs
- intermediate_parallel = intermediate_parallel.to(original_dtype)
- return intermediate_parallel
-
if self.activation_recompute:
self.activation_checkpoint = tensor_parallel.CheckpointWithoutOutput()
with off_interface(self.offload_moe_act, fc1_output, "moe_act") as fc1_output:
bias_act_output = self.activation_checkpoint.checkpoint(
- bias_act_func, fc1_output, bias_parallel, permuted_probs
+ self.bias_act_func, fc1_output, bias_parallel, permuted_probs
)
else:
with off_interface(self.offload_moe_act, fc1_output, "moe_act") as fc1_output:
- bias_act_output = bias_act_func(fc1_output, bias_parallel, permuted_probs)
-
- output, output_bias = self.linear_fc2(bias_act_output, tokens_per_expert, tokens_per_expert_gpu)
+ bias_act_output = self.bias_act_func(fc1_output, bias_parallel, permuted_probs)
+ output, output_bias = apply_module(self.linear_fc2)(bias_act_output, tokens_per_expert, tokens_per_expert_gpu)
if self.activation_recompute:
self.activation_checkpoint.discard_output_and_register_recompute(output)
@@ -820,6 +460,273 @@ def backward_dw(self):
self.linear_fc1.backward_dw()
+class InferenceGroupedMLP(TEGroupedMLP):
+ """Inference-optimized GroupedMLP with GPU-resident offsets.
+
+ Inherits from TEGroupedMLP to reuse weight initialization and checkpoint compatibility.
+ Supports three forward paths:
+ - Training: delegates to parent TEGroupedMLP
+ - Inference + CUDA graphed: FlashInfer cutlass_fused_moe (fused permute + GEMM)
+ - Inference + eager: torch.nn.functional.grouped_mm with GPU-resident cumsum offsets
+ """
+
+ def __init__(
+ self,
+ num_local_experts: int,
+ config: TransformerConfig,
+ submodules: GroupedMLPSubmodules,
+ pg_collection: Optional[ProcessGroupCollection] = None,
+ ):
+ # Initialize parent TEGroupedMLP (creates linear_fc1, linear_fc2)
+ super().__init__(
+ num_local_experts=num_local_experts,
+ config=config,
+ submodules=submodules,
+ pg_collection=pg_collection,
+ )
+
+ # Concatenated weights are built lazily on first forward to ensure
+ # checkpoint loading has already populated the per-expert parameters.
+ self._concatenated_weights_built = False
+
+ self.is_inference_cuda_graphed_iteration = False
+
+ if HAVE_FLASHINFER:
+ self._flashinfer_activation_type = self._resolve_flashinfer_activation_type()
+
+ self._mcore_activation_type = self._resolve_mcore_activation_type()
+ self.inference_grouped_gemm_backend = config.inference_grouped_gemm_backend
+
+ def _resolve_flashinfer_activation_type(self):
+ """Map megatron activation config to FlashInfer ActivationType."""
+ assert (
+ HAVE_FLASHINFER
+ ), "flashinfer-python is required to resolve FlashInfer activation type."
+ func = self.config.activation_func
+ if func == F.silu:
+ return ActivationType.Silu
+ elif func == F.gelu:
+ return ActivationType.Gelu
+ elif func == F.relu:
+ return ActivationType.Relu
+ elif func == squared_relu:
+ return ActivationType.Relu2
+ raise ValueError(f"No FlashInfer ActivationType mapping for activation_func={func}")
+
+ def _resolve_mcore_activation_type(self):
+ """Map megatron activation config to mcore_fused_moe ActivationType."""
+ func = self.config.activation_func
+ if func == squared_relu:
+ return McoreActivationType.SQUARED_RELU
+ raise ValueError(f"No mcore_fused_moe ActivationType mapping for activation_func={func}")
+
+ def set_inference_cuda_graphed_iteration(self):
+ """Enable CUDA-graphed iteration mode."""
+ self.is_inference_cuda_graphed_iteration = True
+
+ def unset_inference_cuda_graphed_iteration(self):
+ """Disable CUDA-graphed iteration mode."""
+ self.is_inference_cuda_graphed_iteration = False
+
+ def _build_concatenated_mxfp8_weights(self):
+ """Build stacked MXFP8 weight tensors from per-expert MXFP8Tensor attributes.
+
+ After quantize_model_to_mxfp8, each per-expert weight (weight0, weight1, ...)
+ has been replaced with an MXFP8Tensor. This method stacks their data and
+ scales into _fc1_weight / _fc2_weight for scaled_grouped_mm.
+
+ Note: this creates a contiguous copy since per-expert MXFP8Tensor attributes
+ are not contiguous across experts. This is a one-time cost at first forward.
+
+ Unlike _build_concatenated_weights, this does not create nn.Parameter views
+ back into the buffer — MXFP8 weights are not nn.Parameters (they are plain
+ MXFP8Tensor attributes set by quantize_model_to_mxfp8). This path is only
+ intended for non-colocated inference.
+ """
+
+ for linear_name, buf_name in [('linear_fc1', '_fc1_weight'), ('linear_fc2', '_fc2_weight')]:
+ linear = getattr(self, linear_name)
+ q_list, s_list = [], []
+ for i in range(self.num_local_experts):
+ w = getattr(linear, f'weight{i}')
+ if isinstance(w, MXFP8Tensor):
+ mxfp8 = w
+ elif hasattr(w, 'data') and isinstance(w.data, MXFP8Tensor):
+ mxfp8 = w.data
+ else:
+ raise RuntimeError(
+ f"Expected MXFP8Tensor for {linear_name}.weight{i}, "
+ f"got {type(w).__name__}. Was quantize_model_to_mxfp8 called?"
+ )
+ q_list.append(mxfp8.data)
+ s_list.append(mxfp8.scale)
+
+ stacked_data = torch.stack(q_list, dim=0).contiguous()
+ stacked_scale = torch.stack(s_list, dim=0).contiguous()
+
+ setattr(self, buf_name, MXFP8Tensor(data=stacked_data, scale=stacked_scale))
+
+ # Redirect per-expert weight .data to views into the stacked buffer,
+ # mirroring _build_concatenated_weights. This frees the original
+ # allocations while keeping the Parameter objects intact.
+ for i in range(self.num_local_experts):
+ w = getattr(linear, f'weight{i}')
+ if isinstance(w, MXFP8Tensor):
+ w.data = stacked_data[i]
+ w.scale = stacked_scale[i]
+ elif hasattr(w, 'data') and isinstance(w.data, MXFP8Tensor):
+ w.data.data = stacked_data[i]
+ w.data.scale = stacked_scale[i]
+
+ @torch.inference_mode(False) # needed for non-colocated inference.
+ def _build_concatenated_weights(self):
+ """Create big contiguous weight tensors that share storage with TE's per-expert parameters.
+
+ Creates _fc1_weight and _fc2_weight as contiguous tensors of shape
+ [num_experts, out_features, in_features]. Instead of replacing TE's parameters
+ (which breaks TE's internal bookkeeping), we redirect each parameter's .data
+ to be a view into the contiguous buffer. The nn.Parameter objects themselves
+ remain untouched in TE's module, preserving FP8 scaling state, etc.
+
+ This allows:
+ - TE's forward to work correctly (same Parameter objects, same internal state)
+ - Training updates to flow through (param.data is a view into the big tensor)
+ - torch.nn.functional.grouped_mm / FlashInfer to use the big tensor directly
+ """
+ # Get device/dtype from existing TE weights
+ device = self.linear_fc1.weight0.device
+ dtype = self.linear_fc1.weight0.dtype
+
+ fc1_shape = self.linear_fc1.weight0.shape # [out_features, in_features]
+ fc2_shape = self.linear_fc2.weight0.shape
+
+ # Create big contiguous tensors
+ _fc1_weight = torch.empty(self.num_local_experts, *fc1_shape, device=device, dtype=dtype)
+ _fc2_weight = torch.empty(self.num_local_experts, *fc2_shape, device=device, dtype=dtype)
+
+ # Copy existing TE weights into big tensors, then point param.data to the views
+ for i in range(self.num_local_experts):
+ fc1_param = getattr(self.linear_fc1, f'weight{i}')
+ fc2_param = getattr(self.linear_fc2, f'weight{i}')
+
+ # Copy initialized data into contiguous buffer
+ _fc1_weight[i].copy_(fc1_param.data)
+ _fc2_weight[i].copy_(fc2_param.data)
+
+ # Redirect param.data to view into contiguous buffer.
+ # The nn.Parameter object stays the same — TE's internal state is preserved.
+ fc1_param.data = _fc1_weight[i]
+ fc2_param.data = _fc2_weight[i]
+
+ # Register big tensors as non-persistent buffers (for .to() device movement, not saved)
+ self.register_buffer('_fc1_weight', _fc1_weight, persistent=False)
+ self.register_buffer('_fc2_weight', _fc2_weight, persistent=False)
+
+ def _flashinfer_forward(self, hidden_states, routing_map, probs):
+ """FlashInfer fused MoE kernel for CUDA-graphed inference iterations."""
+ assert HAVE_FLASHINFER, "flashinfer-python is required for FlashInfer forward path."
+ assert probs.dtype == torch.float32, "FlashInfer forward path requires fp32 probabilities."
+ output = fused_moe.cutlass_fused_moe(
+ hidden_states,
+ routing_map.int(),
+ probs,
+ self._fc1_weight,
+ self._fc2_weight,
+ hidden_states.dtype,
+ quant_scales=None,
+ activation_type=self._flashinfer_activation_type,
+ ep_size=self.ep_group.size(),
+ ep_rank=self.ep_group.rank(),
+ )[0]
+ return output, None
+
+ def _mcore_fused_moe_forward(
+ self, hidden_states, probs, routing_map=None, tokens_per_expert=None, skip_permute=False
+ ):
+ """Torch grouped_mm fused MoE forward via mcore_fused_moe."""
+ local_expert_start = self.ep_group.rank() * self.num_local_experts
+ output = mcore_fused_moe(
+ hidden_states,
+ probs,
+ self._fc1_weight,
+ self._fc2_weight,
+ activation_type=self._mcore_activation_type,
+ num_local_experts=self.num_local_experts,
+ local_expert_start=local_expert_start,
+ routing_map=routing_map,
+ tokens_per_expert=tokens_per_expert,
+ skip_permute=skip_permute,
+ disable_fused_quant_kernels=self.config.inference_moe_disable_fused_quant_kernels,
+ )
+ return output, None
+
+ def forward(
+ self,
+ permuted_local_hidden_states: torch.Tensor,
+ tokens_per_expert: Optional[torch.Tensor],
+ permuted_probs: torch.Tensor,
+ routing_map: Optional[torch.Tensor] = None,
+ ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
+ """Forward pass with three modes:
+
+ - Training: delegates to parent TEGroupedMLP.
+ - Inference + CUDA graphed: FlashInfer cutlass_fused_moe. tokens_per_expert
+ is not used in this path; the FlashInfer kernel operates directly on
+ routing_map.
+ - Inference + eager: torch.nn.functional.grouped_mm with GPU-resident cumsum offsets.
+
+ Args:
+ permuted_local_hidden_states: [num_tokens, hidden_size] input hidden states.
+ tokens_per_expert: [num_experts] number of tokens routed to each expert.
+ None when using the CUDA-graphed FlashInfer path.
+ permuted_probs: [num_tokens, topk] routing probabilities.
+ routing_map: [num_tokens, topk] token-to-expert assignment indices.
+ Required for the FlashInfer CUDA-graphed path, None otherwise.
+ """
+
+ if self.training:
+ assert (
+ not self.config.fp8_recipe == "mxfp8"
+ ), "MXFP8 inference optimized is not compatible with training / colocated RL."
+ return super().forward(permuted_local_hidden_states, tokens_per_expert, permuted_probs)
+
+ # Lazily build concatenated weights on first forward (after checkpoint load)
+ if not self._concatenated_weights_built:
+ w = self.linear_fc1.weight0
+ if isinstance(w, MXFP8Tensor) or (
+ hasattr(w, 'data') and isinstance(w.data, MXFP8Tensor)
+ ):
+ self._build_concatenated_mxfp8_weights()
+ else:
+ self._build_concatenated_weights()
+ self._concatenated_weights_built = True
+
+ resolved_backend = resolve_inference_grouped_gemm_backend(
+ self.inference_grouped_gemm_backend,
+ self.is_inference_cuda_graphed_iteration,
+ is_mxfp8=self.config.fp8_recipe == "mxfp8",
+ )
+
+ if resolved_backend == InferenceGroupedGemmBackend.FLASHINFER:
+ assert routing_map is not None, "routing_map is required for FlashInfer forward pass."
+ assert (
+ self.is_inference_cuda_graphed_iteration
+ ), "FlashInfer forward path is only used in CUDA-graphed inference iterations."
+ return self._flashinfer_forward(
+ permuted_local_hidden_states, routing_map, permuted_probs
+ )
+ elif resolved_backend == InferenceGroupedGemmBackend.TORCH:
+ return self._mcore_fused_moe_forward(
+ permuted_local_hidden_states,
+ permuted_probs,
+ routing_map=routing_map,
+ tokens_per_expert=tokens_per_expert,
+ skip_permute=(not self.is_inference_cuda_graphed_iteration),
+ )
+ elif resolved_backend == InferenceGroupedGemmBackend.TE:
+ return super().forward(permuted_local_hidden_states, tokens_per_expert, permuted_probs)
+
+
class SequentialMLP(MegatronModule):
"""An implementation of the Experts layer using a sequence of MLP layers.
diff --git a/megatron/core/transformer/moe/grouped_gemm_util.py b/megatron/core/transformer/moe/grouped_gemm_util.py
deleted file mode 100644
index 5dd344816bd..00000000000
--- a/megatron/core/transformer/moe/grouped_gemm_util.py
+++ /dev/null
@@ -1,22 +0,0 @@
-# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
-
-try:
- import grouped_gemm
-except ImportError:
- grouped_gemm = None
-
-
-def grouped_gemm_is_available():
- """Check if grouped_gemm is available."""
- return grouped_gemm is not None
-
-
-def assert_grouped_gemm_is_available():
- """Assert that grouped_gemm is available."""
- assert grouped_gemm_is_available(), (
- "Grouped GEMM is not available. Please run "
- "`pip install git+https://github.com/fanshiqing/grouped_gemm@v1.1.4`."
- )
-
-
-ops = grouped_gemm.ops if grouped_gemm_is_available() else None
diff --git a/megatron/core/transformer/moe/moe_layer.py b/megatron/core/transformer/moe/moe_layer.py
index e5decdfb29b..ab71108c7fc 100644
--- a/megatron/core/transformer/moe/moe_layer.py
+++ b/megatron/core/transformer/moe/moe_layer.py
@@ -4,11 +4,12 @@
from abc import ABC, abstractmethod
from dataclasses import dataclass
-from typing import Optional, Protocol, Union
+from typing import Optional, Protocol
import torch
from megatron.core import parallel_state, tensor_parallel, utils
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.process_groups_config import ProcessGroupCollection
from megatron.core.transformer.module import MegatronModule
from megatron.core.transformer.moe.moe_utils import (
@@ -24,19 +25,84 @@
MoEFlexTokenDispatcher,
MoETokenDispatcher,
)
-from megatron.core.transformer.spec_utils import ModuleSpec, build_module
+from megatron.core.transformer.moe.token_dispatcher_inference import (
+ InferenceCUDAGraphTokenDispatcher,
+)
from megatron.core.transformer.transformer_config import TransformerConfig
-from megatron.core.typed_torch import apply_module
+from megatron.core.typed_torch import apply_module, not_none
from megatron.core.utils import internal_api
try:
- import transformer_engine as te # pylint: disable=unused-import
+ import flashinfer # pylint: disable=unused-import
+
+ HAVE_FLASHINFER = True
+except ImportError:
+ HAVE_FLASHINFER = False
+
+if HAVE_FLASHINFER:
+ try:
+ import flashinfer_cubin # pylint: disable=unused-import
+ import flashinfer_jit_cache # pylint: disable=unused-import
+ HAVE_FLASHINFER_CUBIN_AND_JIT_CACHE = True
+ except ImportError:
+ HAVE_FLASHINFER_CUBIN_AND_JIT_CACHE = False
+
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import TELinear, te_checkpoint
+else:
+ TELinear, te_checkpoint = None, None
- HAVE_TE = True
-except ImportError:
- HAVE_TE = False
+
+class ExpertsInterface(Protocol):
+ """Interface for the experts used in an MoELayer."""
+
+ def forward(
+ self,
+ dispatched_input: torch.Tensor,
+ tokens_per_expert: torch.Tensor,
+ permuted_probs: torch.Tensor,
+ /,
+ ) -> tuple[torch.Tensor, torch.Tensor | None]:
+ """Forward pass of the experts layer."""
+ ...
+
+ def backward_dw(self) -> None:
+ """Backward pass to compute weight gradients for the experts."""
+ ...
+
+
+class ExpertsBuilder(Protocol):
+ """Protocol for building the experts used in an MoELayer."""
+
+ def __call__(
+ self,
+ num_local_experts: int,
+ config: TransformerConfig,
+ /,
+ *,
+ pg_collection: ProcessGroupCollection | None,
+ ) -> ExpertsInterface: ...
+
+
+class SharedExpertsInterface(Protocol):
+ """Interface for the shared experts used in an MoELayer."""
+
+ def forward(self, hidden_states: torch.Tensor, /) -> torch.Tensor:
+ """Forward pass of the shared experts."""
+ ...
+
+ def backward_dw(self) -> None:
+ """Backward pass to compute weight gradients for the shared experts."""
+ ...
+
+
+class SharedExpertsBuilder(Protocol):
+ """Protocol for building the shared experts used in an MoELayer."""
+
+ def __call__(
+ self, *, config: TransformerConfig, pg_collection: ProcessGroupCollection | None, gate: bool
+ ) -> SharedExpertsInterface: ...
class RouterInterface(Protocol):
@@ -70,8 +136,8 @@ def __call__(
class MoESubmodules:
"""MoE Layer Submodule spec"""
- experts: Union[ModuleSpec, type] = None
- shared_experts: Union[ModuleSpec, type] = None
+ experts: ExpertsBuilder
+ shared_experts: SharedExpertsBuilder | None = None
router: RouterBuilder = TopKRouter
@@ -144,7 +210,7 @@ def __init__(
pg_collection: Optional[ProcessGroupCollection] = None,
is_mtp_layer: bool = False,
):
- self.submodules = submodules
+ self.submodules = not_none(submodules)
# TODO(Hepteract): delete the usage of the global parallel_state.
# Initialize process groups with the global parallel_state.
if pg_collection is None:
@@ -169,7 +235,7 @@ def __init__(
self.tp_group = pg_collection.tp
# Initialize router.
- self.router = submodules.router(
+ self.router = self.submodules.router(
config=self.config, pg_collection=pg_collection, is_mtp_layer=is_mtp_layer
)
self.tp_group = pg_collection.tp
@@ -177,7 +243,13 @@ def __init__(
# Initialize latent projections.
if self.config.moe_latent_size:
assert HAVE_TE, "TransformerEngine is required for MoE latent projections."
- self.fc1_latent_proj = TELinear(
+ if self.config.transformer_impl == "inference_optimized":
+ from megatron.core.tensor_parallel.inference_layers import InferenceLinear
+
+ linear_cls = InferenceLinear
+ else:
+ linear_cls = TELinear
+ self.fc1_latent_proj = linear_cls(
self.config.hidden_size,
self.config.moe_latent_size,
parallel_mode="duplicated",
@@ -188,7 +260,7 @@ def __init__(
skip_weight_param_allocation=False,
is_expert=False,
)
- self.fc2_latent_proj = TELinear(
+ self.fc2_latent_proj = linear_cls(
self.config.moe_latent_size,
self.config.hidden_size,
parallel_mode="duplicated",
@@ -228,17 +300,16 @@ def __init__(
)
# Initialize experts
- self.experts = build_module(
- self.submodules.experts,
- self.num_local_experts,
- self.config,
- pg_collection=pg_collection,
+ self.experts = self.submodules.experts(
+ self.num_local_experts, self.config, pg_collection=pg_collection
)
# Initialize shared experts
if self.use_shared_expert:
- self.shared_experts = build_module(
- self.submodules.shared_experts,
+ assert (
+ self.submodules.shared_experts is not None
+ ), "Shared experts builder is not provided in the module spec."
+ self.shared_experts = self.submodules.shared_experts(
config=self.config,
pg_collection=pg_collection,
gate=self.config.moe_shared_expert_gate,
@@ -246,10 +317,87 @@ def __init__(
if self.shared_expert_overlap:
self.token_dispatcher.set_shared_experts(self.shared_experts)
+ # Inference-optimized mode setup
+ if config.transformer_impl == "inference_optimized":
+ if config.inference_grouped_gemm_backend == 'auto':
+ assert HAVE_FLASHINFER, (
+ "inference_grouped_gemm_backend='auto'"
+ "requires flashinfer-python. "
+ "Install flashinfer-python or set "
+ "inference_grouped_gemm_backend to 'torch' or 'te'."
+ )
+
+ # Verify that pre-compiled FlashInfer CUTLASS kernels are available
+ # when using the FlashInfer backend. The flashinfer-jit-cache package
+ # must be installed ahead of time to avoid a multi-minute JIT
+ # compilation step at runtime.
+ from megatron.core.inference.utils import check_flashinfer_jit_cache_installed
+
+ check_flashinfer_jit_cache_installed()
+ elif config.inference_grouped_gemm_backend == 'torch':
+ assert hasattr(torch.nn.functional, 'grouped_mm'), (
+ "inference_grouped_gemm_backend='torch' requires "
+ "torch.nn.functional.grouped_mm (available since PyTorch 2.10)."
+ )
+ self._setup_inference_mode(pg_collection)
+
# Cudagraph tensor store for resuming the forward pass from the end of the cudagraph.
self.cudagraph_tensor_store = MoECudaGraphTensorStore()
self.fwd_execution_map = ["route", "expert_compute", "postprocess"]
+ def _setup_inference_mode(self, pg_collection):
+ """Set up inference-optimized token dispatcher and state.
+
+ Called from __init__ when config.transformer_impl == "inference_optimized".
+ Creates an InferenceCUDAGraphTokenDispatcher alongside the standard dispatcher,
+ which is swapped in during CUDA-graphed forward passes.
+ """
+
+ assert self.config.moe_token_dispatcher_type == "alltoall", (
+ f"Inference-optimized MoE requires 'alltoall' dispatcher, "
+ f"got '{self.config.moe_token_dispatcher_type}'"
+ )
+ self.is_inference_cuda_graphed_iteration = False
+ self._inference_token_dispatcher = InferenceCUDAGraphTokenDispatcher(
+ self.num_local_experts,
+ self.local_expert_indices,
+ config=self.config,
+ pg_collection=pg_collection,
+ )
+
+ def set_inference_cuda_graphed_iteration(self):
+ """Enable CUDA-graphed iteration mode on this layer, its router, and its experts.
+
+ Swaps in the inference-optimized token dispatcher and disables
+ shared expert overlap.
+ """
+ self.is_inference_cuda_graphed_iteration = True
+ if hasattr(self.router, "set_inference_cuda_graphed_iteration"):
+ self.router.set_inference_cuda_graphed_iteration()
+ if hasattr(self.experts, "set_inference_cuda_graphed_iteration"):
+ self.experts.set_inference_cuda_graphed_iteration()
+
+ if self._inference_token_dispatcher is not None:
+ self._saved_token_dispatcher = self.token_dispatcher
+ self.token_dispatcher = self._inference_token_dispatcher
+ self._saved_shared_expert_overlap = self.shared_expert_overlap
+ self.shared_expert_overlap = False
+
+ def unset_inference_cuda_graphed_iteration(self):
+ """Disable CUDA-graphed iteration mode on this layer, its router, and its experts.
+
+ Restores the standard token dispatcher and shared expert overlap setting.
+ """
+ self.is_inference_cuda_graphed_iteration = False
+ if hasattr(self.router, "unset_inference_cuda_graphed_iteration"):
+ self.router.unset_inference_cuda_graphed_iteration()
+ if hasattr(self.experts, "unset_inference_cuda_graphed_iteration"):
+ self.experts.unset_inference_cuda_graphed_iteration()
+
+ if hasattr(self, "_saved_token_dispatcher"):
+ self.token_dispatcher = self._saved_token_dispatcher
+ self.shared_expert_overlap = self._saved_shared_expert_overlap
+
@maybe_skip_or_early_return_by_cudagraph("route")
def route(self, hidden_states: torch.Tensor, padding_mask: Optional[torch.Tensor] = None):
"""Compute token routing for preprocessing.
@@ -302,7 +450,7 @@ def shared_experts_compute(self, hidden_states: torch.Tensor):
if self.shared_experts_recompute:
if self.config.fp8 or self.config.fp4:
shared_expert_output = te_checkpoint(
- self.shared_experts,
+ apply_module(self.shared_experts),
False,
tensor_parallel.random.get_cuda_rng_tracker,
parallel_state.get_tensor_model_parallel_group(),
@@ -310,10 +458,10 @@ def shared_experts_compute(self, hidden_states: torch.Tensor):
)
else:
shared_expert_output = tensor_parallel.checkpoint(
- self.shared_experts, False, hidden_states
+ apply_module(self.shared_experts), False, hidden_states
)
else:
- shared_expert_output = self.shared_experts(hidden_states)
+ shared_expert_output = apply_module(self.shared_experts)(hidden_states)
return shared_expert_output
@@ -328,7 +476,18 @@ def routed_experts_compute(self, hidden_states: torch.Tensor, probs: torch.Tenso
dispatched_input, tokens_per_expert, permuted_probs = (
self.token_dispatcher.dispatch_postprocess(hidden_states, probs)
)
- expert_output, mlp_bias = self.experts(dispatched_input, tokens_per_expert, permuted_probs)
+ if (
+ hasattr(self, "_inference_token_dispatcher")
+ and self.is_inference_cuda_graphed_iteration
+ ):
+ routing_map = self.token_dispatcher.routing_map
+ expert_output, mlp_bias = apply_module(self.experts)(
+ dispatched_input, tokens_per_expert, permuted_probs, routing_map=routing_map
+ )
+ else:
+ expert_output, mlp_bias = apply_module(self.experts)(
+ dispatched_input, tokens_per_expert, permuted_probs
+ )
assert mlp_bias is None, f"mlp_bias is not supported for {type(self.token_dispatcher)}"
output = self.token_dispatcher.combine_preprocess(expert_output)
@@ -437,7 +596,7 @@ def custom_forward(hidden_states, intermediate_tensors=None, padding_mask=None):
return output, mlp_bias
- if self.moe_layer_recompute:
+ if self.moe_layer_recompute and self.training:
if self.config.fp8 or self.config.fp4:
outputs = te_checkpoint(
custom_forward,
@@ -459,10 +618,25 @@ def custom_forward(hidden_states, intermediate_tensors=None, padding_mask=None):
def backward_dw(self, routed_experts: bool = True, shared_experts: bool = False):
"""Compute weight gradients for experts and shared experts."""
+ # TODO(Wohox): replace the "routed_experts" and "shared_experts" arguments with better
+ # naming to better explain that they are actually from different fine-grained callables,
+ # or use scanning to decide which backward_dw should be called.
if routed_experts:
self.experts.backward_dw()
- if shared_experts and self.use_shared_expert and not self.shared_expert_overlap:
- self.shared_experts.backward_dw()
+ if self.config.moe_latent_size:
+ # TODO(Wohox): fc2_latent_proj forward and backward are executed in comm stream,
+ # so we execute its backward_dw in the comm stream too. But this may harm the
+ # EP overlap performance. Better to check if there is a better way to handle this.
+ from megatron.core.pipeline_parallel.utils import get_comm_stream
+
+ comm_stream = get_comm_stream()
+ with torch.cuda.stream(comm_stream):
+ self.fc2_latent_proj.backward_dw()
+ if shared_experts:
+ if self.use_shared_expert and not self.shared_expert_overlap:
+ self.shared_experts.backward_dw()
+ if self.config.moe_latent_size:
+ self.fc1_latent_proj.backward_dw()
def set_for_recompute_pre_mlp_layernorm(self):
"""Set the MoE layer for recompute pre_mlp_layernorm. Only needed for fp8/fp4."""
diff --git a/megatron/core/transformer/moe/moe_utils.py b/megatron/core/transformer/moe/moe_utils.py
index 132316749fa..70209a5ba9f 100644
--- a/megatron/core/transformer/moe/moe_utils.py
+++ b/megatron/core/transformer/moe/moe_utils.py
@@ -8,24 +8,28 @@
import torch
from megatron.core import parallel_state
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.fp4_utils import get_fp4_align_size
from megatron.core.fp8_utils import get_fp8_align_size
from megatron.core.process_groups_config import ProcessGroupCollection
-from megatron.core.tensor_parallel import get_cuda_rng_tracker, get_expert_parallel_rng_tracker_name
+from megatron.core.tensor_parallel import (
+ get_cuda_rng_tracker,
+ get_data_parallel_rng_tracker_name,
+ get_expert_parallel_rng_tracker_name,
+)
from megatron.core.tensor_parallel.mappings import reduce_from_tensor_model_parallel_region
from megatron.core.transformer.cuda_graphs import is_graph_capturing
from megatron.core.transformer.enums import CudaGraphScope
from megatron.core.transformer.moe.router_replay import RouterReplay
from megatron.core.transformer.transformer_config import TransformerConfig
-from megatron.core.utils import internal_api
-
-try:
- import transformer_engine as te # pylint: disable=unused-import
+from megatron.core.utils import internal_api, is_te_min_version
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import (
fused_compute_score_for_moe_aux_loss,
fused_moe_aux_loss,
fused_permute,
+ fused_permute_and_pad_with_probs,
fused_permute_with_probs,
fused_sort_chunks_by_index,
fused_sort_chunks_by_index_with_probs,
@@ -33,10 +37,19 @@
fused_unpermute,
te_general_gemm,
)
-
- HAVE_TE = True
-except ImportError:
- HAVE_TE = False
+else:
+ (
+ fused_compute_score_for_moe_aux_loss,
+ fused_moe_aux_loss,
+ fused_permute,
+ fused_permute_and_pad_with_probs,
+ fused_permute_with_probs,
+ fused_sort_chunks_by_index,
+ fused_sort_chunks_by_index_with_probs,
+ fused_topk_with_score_function,
+ fused_unpermute,
+ te_general_gemm,
+ ) = (None, None, None, None, None, None, None, None, None, None)
# MOE logging
@@ -57,7 +70,7 @@ def switch_load_balancing_loss_func(
Refer to the Switch Transformer (https://arxiv.org/abs/2101.03961)
and Global Load Balancing Loss(https://arxiv.org/abs/2501.11873) for details.
- ### Detailed explanation of the auxiliary loss #######
+ Detailed explanation of the auxiliary loss:
The formula for the auxiliary loss is:
loss = E * Σ_{i=1}^{E} (f_i * P_i)
@@ -91,8 +104,6 @@ def switch_load_balancing_loss_func(
(either micro-batch or global batch)
- total_num_tokens: Should match the total token count at the same level as tokens_per_expert
- #########################################################
-
Args:
probs (torch.Tensor): Softmax probabilities output by the router for each token.
Shape in [num_tokens, num_experts].
@@ -345,7 +356,15 @@ def permute(
num_out_tokens: Optional[int] = None,
fused: bool = False,
drop_and_pad: bool = False,
-) -> Tuple[torch.Tensor, Optional[torch.Tensor], torch.Tensor]:
+ tokens_per_expert: Optional[torch.Tensor] = None,
+ align_size: int = -1,
+) -> Tuple[
+ torch.Tensor,
+ Optional[torch.Tensor],
+ torch.Tensor,
+ Optional[torch.Tensor],
+ Optional[torch.Tensor],
+]:
"""Permute the tokens and probs based on the mask.
Tokens with the same designated expert will be grouped together.
The shape of mask is [tokens, num_experts], it indicates which experts were selected
@@ -354,6 +373,9 @@ def permute(
When drop_and_pad=True, in routing_map, the number of non-zeros in each column equals to
expert capacity. This function exploits this feature to use ops that support cuda graph.
+ If the fused permute and pad kernel is available, it will pad the tokens to the align_size
+ and return the padded permuted tokens, pad_offsets and padded tokens per expert.
+
Args:
tokens (torch.Tensor): The input token tensor, [num_tokens, hidden].
routing_map (torch.Tensor): The sparse token to expert mapping, [num_tokens, num_experts].
@@ -365,10 +387,20 @@ def permute(
and pads the number of tokens to the expert capacity.
If set to true, routing_map has a fixed number of non-zeros
in each column.
+ tokens_per_expert (torch.Tensor, optional): Tensor of shape `[num_experts]` containing
+ actual token counts per expert.
+ align_size (int, optional): The alignment size for the input tensor for fp8 or fp4.
Returns:
- Tuple[torch.Tensor, Optional[torch.Tensor], torch.Tensor]:
- The permuted tokens, permuted probs, and sorted indices.
+ Tuple[
+ torch.Tensor,
+ Optional[torch.Tensor],
+ torch.Tensor,
+ Optional[torch.Tensor],
+ Optional[torch.Tensor],
+ ]:
+ The permuted tokens, (optional) permuted probs, sorted indices,
+ (optional) pad_offsets, (optional) padded_tokens_per_expert.
"""
if fused and probs is None:
if not HAVE_TE or fused_permute is None:
@@ -376,14 +408,26 @@ def permute(
permuted_input, sorted_indices = fused_permute(
tokens, routing_map, num_out_tokens=num_out_tokens
)
- return permuted_input, None, sorted_indices
+ return permuted_input, None, sorted_indices, None, tokens_per_expert
if fused and probs is not None:
- if not HAVE_TE or fused_permute_with_probs is None:
+ if not HAVE_TE or (
+ fused_permute_and_pad_with_probs is None and fused_permute_with_probs is None
+ ):
raise ValueError(
- "fused_permute_with_probs is not available. Please install TE >= 2.1.0."
+ "Transformer Engine (TE) fused kernel is not available. "
+ "fused_permute_with_probs typically requires TE >= 2.1.0, and "
+ "fused_permute_and_pad_with_probs` typically requires TE >= 2.12.0. "
+ )
+ if fused_permute_and_pad_with_probs is not None and tokens_per_expert is not None:
+ return fused_permute_and_pad_with_probs(
+ tokens, probs, routing_map, tokens_per_expert, align_size
+ )
+ else:
+ output, permuted_probs, row_id_map = fused_permute_with_probs(
+ tokens, probs, routing_map, num_out_tokens=num_out_tokens
)
- return fused_permute_with_probs(tokens, probs, routing_map, num_out_tokens=num_out_tokens)
+ return output, permuted_probs, row_id_map, None, tokens_per_expert
num_tokens, hidden = tokens.shape
num_experts = routing_map.shape[1]
@@ -411,22 +455,27 @@ def permute(
# get probs from indices
permuted_probs = probs_T_1D.index_select(0, indices_1D)
else:
+ assert (
+ num_out_tokens is not None
+ ), "num_out_tokens is required for the argsort-based permute"
+
# mask [num_tokens, num_experts] -> [num_experts, num_tokens]
routing_map = routing_map.bool().T.contiguous()
- # Create a dense expert-to-token mapping from the sparse token-to-expert mapping
- token_indices = (
- torch.arange(num_tokens, device=routing_map.device).unsqueeze(0).expand(num_experts, -1)
- )
- sorted_indices = token_indices.masked_select(routing_map)
+ # Use argsort to get indices of non-zero entries in row-major order.
+ # This is equivalent to masked_select but produces fixed-shape output,
+ # making it compatible with CUDA graph capture.
+ flat_sorted = routing_map.reshape(-1).argsort(descending=True, stable=True)
+ flat_sorted = flat_sorted[:num_out_tokens]
+ sorted_indices = flat_sorted % num_tokens
if probs is not None:
- permuted_probs = probs.T.contiguous().masked_select(routing_map)
+ permuted_probs = probs.T.contiguous().reshape(-1)[flat_sorted]
# use the mapping to permute the tokens
permuted_input = tokens.index_select(0, sorted_indices)
- return permuted_input, permuted_probs, sorted_indices
+ return permuted_input, permuted_probs, sorted_indices, None, tokens_per_expert
def unpermute(
@@ -437,6 +486,7 @@ def unpermute(
routing_map: Optional[torch.Tensor] = None,
fused: bool = False,
drop_and_pad: bool = False,
+ pad_offsets: Optional[torch.Tensor] = None,
) -> torch.Tensor:
"""
Restore the original order of tokens after permutation. If probs are provided, it
@@ -458,6 +508,10 @@ def unpermute(
fused (bool, optional): Whether use the fused unpermute function.
drop_and_pad (bool, optional): Whether or not the token dispatcher uses token-drop
and pads the number of tokens to the expert capacity.
+ pad_offsets (torch.Tensor, optional):
+ Tensor of per-expert cumulative padding offsets used to remove padding added
+ during permutation. This is the fourth output of `moe_permute_and_pad_with_probs`
+ and is required when unpermuting padded outputs. Defaults to None.
Returns:
torch.Tensor: The tokens restored to their original order.
@@ -465,8 +519,15 @@ def unpermute(
if fused:
if not HAVE_TE or fused_unpermute is None:
raise ValueError("fused_unpermute is not available. Please install TE >= 2.1.0.")
+ extra_kwargs = {}
+ if is_te_min_version("2.12.0"):
+ extra_kwargs["pad_offsets"] = pad_offsets
return fused_unpermute(
- permuted_tokens, sorted_indices, merging_probs=probs, restore_shape=restore_shape
+ permuted_tokens,
+ sorted_indices,
+ merging_probs=probs,
+ restore_shape=restore_shape,
+ **extra_kwargs,
)
_, hidden = restore_shape
@@ -668,6 +729,7 @@ def topk_routing_with_score_function(
expert_bias: Optional[torch.Tensor] = None,
fused: bool = False,
router_replay: Optional['RouterReplay'] = None,
+ dense_output: bool = False,
) -> Tuple[torch.Tensor, torch.Tensor]:
"""Compute the routing probabilities and map for top-k selection with score function.
@@ -690,14 +752,23 @@ def topk_routing_with_score_function(
recorded routing sequence.
Defaults to None.
+ dense_output (bool, optional): If True, return dense tensors [num_tokens, topk] instead of
+ sparse tensors [num_tokens, num_experts]. Defaults to False.
Returns:
Tuple[torch.Tensor, torch.Tensor]:
- - routing_probs (torch.Tensor): A tensor of shape [num_tokens, num_experts] containing
- the routing probabilities for each token to each expert.
- - routing_map (torch.Tensor): A mask tensor of shape [num_tokens, num_experts]
- indicating which experts were selected for each token. True values represent
- the selected experts.
+ When dense_output=False (default):
+ - routing_probs (torch.Tensor): Shape [num_tokens, num_experts]. Sparse tensor
+ containing the normalized routing probability for each token-expert pair. Non-zero
+ entries correspond to the top-k selected experts per token.
+ - routing_map (torch.Tensor): Shape [num_tokens, num_experts]. Boolean mask where
+ True indicates the token is routed to that expert (i.e. the expert was in the
+ token's top-k selection).
+ When dense_output=True:
+ - probs (torch.Tensor): Shape [num_tokens, topk]. The normalized routing
+ probabilities for each token's top-k selected experts.
+ - top_indices (torch.Tensor): Shape [num_tokens, topk]. The expert indices
+ selected for each token.
"""
assert logits.dim() == 2, f"Expected 2D logits [num_tokens, num_experts], got {logits.dim()}."
num_tokens, num_experts = logits.shape
@@ -780,6 +851,9 @@ def compute_topk(scores, topk, num_groups=None, group_topk=None):
if scaling_factor:
probs = probs * scaling_factor
+ if dense_output:
+ return probs, top_indices
+
if torch.are_deterministic_algorithms_enabled():
# build [num_tokens, num_experts] from [num_tokens, topk]
routing_probs = torch.zeros_like(logits)
@@ -832,7 +906,8 @@ def compute_routing_scores_for_aux_loss(
if score_function == "softmax":
scores = torch.softmax(logits, dim=-1, dtype=torch.float32)
elif score_function == "sigmoid":
- scores = torch.sigmoid(logits)
+ # Cast logits to float32 before sigmoid for stability
+ scores = torch.sigmoid(logits.to(torch.float32))
scores = scores / (scores.sum(dim=-1, keepdim=True) + 1e-20)
else:
raise ValueError(f"Invalid score_function: {score_function}")
@@ -1041,13 +1116,20 @@ def track_moe_metrics(
"""
# Aux loss logging
tracker = get_moe_layer_wise_logging_tracker()
- # Initialize the tracker if force_initialize is True
+ # Initialize the tracker if force_initialize is True.
+ # The values tensor size must match what the router creates in save_to_aux_losses_tracker,
+ # which uses (num_layers + mtp_num_layers). This is important for PP ranks that have no
+ # MoE layers (so the tracker is empty and force_initialize creates the entry); their tensor
+ # size must match ranks that do have MoE layers, otherwise all_reduce across PP will hang.
+ tracker_num_layers = num_layers
+ if mtp_num_layers is not None:
+ tracker_num_layers += mtp_num_layers
if force_initialize:
if track_names is not None:
for key in track_names:
if key not in tracker:
tracker[key] = {}
- tracker[key]["values"] = torch.zeros(num_layers, device="cuda")
+ tracker[key]["values"] = torch.zeros(tracker_num_layers, device="cuda")
tracker[key]["reduce_group"] = None
tracker[key]["avg_group"] = None
tracker[key]["reduce_group_has_dp"] = False
@@ -1209,6 +1291,55 @@ def apply_random_logits(logits: torch.Tensor) -> torch.Tensor:
return RandomSTE.apply(logits)
+@internal_api
+class RandomSTEShared(torch.autograd.Function):
+ """
+ Straight-Through Estimator(STE) function that returns random values
+ with a shared seed across all ranks.
+ When std < 0, caches and reuses values per layer.
+ """
+
+ _cache = {}
+
+ @staticmethod
+ def forward(ctx, logits, std, layer_number):
+ """Forward pass: apply random bias to logits."""
+ # Check cache if reuse mode (negative std)
+ if std < 0 and layer_number in RandomSTEShared._cache:
+ return logits + RandomSTEShared._cache[layer_number]
+
+ # Generate random bias with shared seed across all ranks
+ with get_cuda_rng_tracker().fork(get_data_parallel_rng_tracker_name()):
+ bias = torch.empty(logits.shape[-1], device=logits.device, dtype=logits.dtype).normal_(
+ std=abs(std)
+ )
+
+ # Cache if reuse mode
+ if std < 0 and layer_number is not None:
+ RandomSTEShared._cache[layer_number] = bias
+
+ return logits + bias
+
+ @staticmethod
+ def backward(ctx, grad_output):
+ """Backward pass: pass through gradients."""
+ return grad_output, None, None
+
+
+def apply_biased_logits(logits, std, layer_number=None):
+ """
+ Apply random bias to logits. All ranks get the same random values.
+
+ Args:
+ logits: Input logits tensor [num_tokens, num_experts]
+ std: Standard deviation for random bias. If negative, generate once
+ per layer and reuse (using abs(std) as actual std).
+ layer_number: Layer number for caching when std is negative.
+ """
+ logits = apply_random_logits(logits)
+ return RandomSTEShared.apply(logits, std, layer_number)
+
+
class RouterGatingLinearFunction(torch.autograd.Function):
"""
Autograd function for router gating linear.
diff --git a/megatron/core/transformer/moe/router.py b/megatron/core/transformer/moe/router.py
index c830732b17b..d4263f25e77 100644
--- a/megatron/core/transformer/moe/router.py
+++ b/megatron/core/transformer/moe/router.py
@@ -10,6 +10,7 @@
from megatron.core.transformer.moe.moe_utils import (
MoEAuxLossAutoScaler,
ProcessGroupCollection,
+ apply_biased_logits,
apply_random_logits,
apply_router_token_dropping,
compute_routing_scores_for_aux_loss,
@@ -698,6 +699,12 @@ def forward(self, input: torch.Tensor, padding_mask: Optional[torch.Tensor] = No
# Apply force load balancing with random logits for benchmark
logits = apply_random_logits(logits)
+ if self.config.moe_router_force_biased is not None:
+ # Apply biased logits with shared random bias across all ranks
+ logits = apply_biased_logits(
+ logits, self.config.moe_router_force_biased, self.layer_number
+ )
+
probs, routing_map = self.routing(logits, padding_mask=padding_mask)
return probs, routing_map
@@ -711,3 +718,115 @@ def _save_to_state_dict(self, *args, **kwargs):
"""Save the state dict of the router."""
self._maintain_float32_expert_bias() # switch to float32 before saving
return super()._save_to_state_dict(*args, **kwargs)
+
+
+class InferenceTopKRouter(TopKRouter):
+ """Inference-only top-k router that strips out training-specific overhead.
+
+ A stripped-down version of TopKRouter that skips z-loss, auxiliary load
+ balancing losses, token dropping, and expert bias updates. The _forward()
+ method is @torch.compile()'d and returns dense [num_tokens, topk] tensors
+ instead of sparse [num_tokens, num_experts] for compatibility with FlashInfer.
+
+ Falls back to the parent TopKRouter.forward() for training or
+ non-CUDA-graphed inference iterations.
+ """
+
+ def __init__(
+ self,
+ config: TransformerConfig,
+ pg_collection: Optional[ProcessGroupCollection] = None,
+ is_mtp_layer: bool = False,
+ ) -> None:
+ """Initialize the specialized inference top-k router.
+
+ Args:
+ config (TransformerConfig): The configuration for the transformer model.
+ pg_collection (ProcessGroupCollection, optional): Process groups for MoE operations.
+ """
+ # Enforce constraints before calling super().__init__
+ assert config.moe_router_num_groups is None, (
+ f"InferenceTopKRouter requires moe_router_num_groups=None, "
+ f"got {config.moe_router_num_groups}"
+ )
+ assert config.moe_router_score_function in ["sigmoid", "softmax"], (
+ f"InferenceTopKRouter requires moe_router_score_function in "
+ f"['sigmoid', 'softmax'], got '{config.moe_router_score_function}'"
+ )
+
+ super().__init__(config=config, pg_collection=pg_collection)
+
+ self.is_inference_cuda_graphed_iteration = False
+
+ def set_inference_cuda_graphed_iteration(self):
+ """Enable CUDA graph-compatible operations for the router."""
+ self.is_inference_cuda_graphed_iteration = True
+
+ def unset_inference_cuda_graphed_iteration(self):
+ """Disable CUDA graph-compatible operations for the router."""
+ self.is_inference_cuda_graphed_iteration = False
+
+ @staticmethod
+ @torch.compile
+ def _compiled_topk_routing(
+ logits,
+ topk,
+ use_pre_softmax,
+ num_groups,
+ group_topk,
+ scaling_factor,
+ score_function,
+ expert_bias,
+ fused,
+ router_replay,
+ dense_output,
+ ):
+ return topk_routing_with_score_function(
+ logits,
+ topk,
+ use_pre_softmax=use_pre_softmax,
+ num_groups=num_groups,
+ group_topk=group_topk,
+ scaling_factor=scaling_factor,
+ score_function=score_function,
+ expert_bias=expert_bias,
+ fused=fused,
+ router_replay=router_replay,
+ dense_output=dense_output,
+ )
+
+ def _forward(self, input: torch.Tensor, padding_mask: Optional[torch.Tensor] = None):
+ logits = self.gating(input).squeeze(1) # [num_tokens, num_experts]
+
+ probs, top_indices = self._compiled_topk_routing(
+ logits,
+ self.topk,
+ use_pre_softmax=self.config.moe_router_pre_softmax,
+ num_groups=self.config.moe_router_num_groups,
+ group_topk=self.config.moe_router_group_topk,
+ scaling_factor=self.config.moe_router_topk_scaling_factor,
+ score_function=self.score_function,
+ expert_bias=self.expert_bias,
+ fused=self.config.moe_router_fusion,
+ router_replay=self.router_replay,
+ dense_output=True,
+ )
+ return probs.squeeze(1), top_indices.squeeze(1)
+
+ def forward(self, input: torch.Tensor, padding_mask: Optional[torch.Tensor] = None):
+ """Simplified forward pass for inference - returns dense tensors only.
+
+ Args:
+ input (torch.Tensor): Input tensor of shape [seq_length, bsz, hidden_size].
+ padding_mask (torch.Tensor, optional): Not used in inference.
+
+ Returns:
+ Tuple[torch.Tensor, torch.Tensor]:
+ - probs: Normalized routing probabilities [num_tokens, topk]
+ - top_indices: Selected expert indices [num_tokens, topk]
+ """
+
+ if self.training or not self.is_inference_cuda_graphed_iteration:
+ return super().forward(input, padding_mask)
+
+ return self._forward(input, padding_mask)
diff --git a/megatron/core/transformer/moe/router_replay.py b/megatron/core/transformer/moe/router_replay.py
index b6b8e26a0a6..5430f75568f 100644
--- a/megatron/core/transformer/moe/router_replay.py
+++ b/megatron/core/transformer/moe/router_replay.py
@@ -74,6 +74,29 @@ def clear_global_router_replay_instances():
"""Clear the global list of router replay instances to prevent memory leaks."""
RouterReplay.global_router_replay_instances.clear()
+ @staticmethod
+ def set_global_static_buffers(static_buffer: torch.Tensor):
+ """Sets static buffers for all router instances from a combined buffer.
+
+ Args:
+ static_buffer: Tensor of shape [max_tokens, num_layers, topk].
+ Each layer's RouterReplay gets a slice [:, layer_idx, :].
+ """
+ num_layers = len(RouterReplay.global_router_replay_instances)
+ assert static_buffer.shape[1] == num_layers, (
+ f"Buffer has {static_buffer.shape[1]} layers but there are "
+ f"{num_layers} RouterReplay instances."
+ )
+ for layer_idx, router_instance in enumerate(RouterReplay.global_router_replay_instances):
+ # Each layer gets a view of shape [max_tokens, topk]
+ router_instance.set_static_buffer(static_buffer[:, layer_idx, :])
+
+ @staticmethod
+ def clear_global_static_buffers():
+ """Clears static buffers from all router instances."""
+ for router in RouterReplay.global_router_replay_instances:
+ router.clear_static_buffer()
+
def __init__(self):
"""Initializes a RouterReplay instance for a specific layer."""
self.target_topk_idx: Optional[torch.Tensor] = None # Target topk indices for replay
@@ -84,6 +107,7 @@ def __init__(self):
self.replay_backward_list: List[torch.Tensor] = (
[]
) # List of tensors for backward pass replay
+ self.static_buffer: Optional[torch.Tensor] = None # Static buffer for CUDA graph
RouterReplay.global_router_replay_instances.append(self)
def set_target_indices(self, topk_indices: torch.Tensor):
@@ -95,10 +119,6 @@ def get_recorded_indices(self) -> Optional[torch.Tensor]:
"""Returns the recorded topk indices."""
return self.recorded_topk_idx
- def record_indices(self, topk_indices: torch.Tensor):
- """Records the topk indices."""
- self.recorded_topk_idx = topk_indices
-
def clear_indices(self):
"""Clears the recorded and target topk indices."""
self.recorded_topk_idx = None
@@ -159,3 +179,29 @@ def get_replay_topk(
return probs, top_indices
else:
return default_compute_topk(scores, topk, num_groups, group_topk)
+
+ def set_static_buffer(self, buffer: torch.Tensor):
+ """Sets a static buffer for CUDA graph compatible recording.
+
+ Args:
+ buffer: Tensor of shape [max_tokens, topk] to copy routing indices into.
+ """
+ self.static_buffer = buffer
+
+ def clear_static_buffer(self):
+ """Clears the static buffer."""
+ self.static_buffer = None
+
+ def record_indices(self, topk_indices: torch.Tensor):
+ """Records the topk indices.
+
+ If a static buffer is set (for CUDA graph compatibility), copies into it.
+ Otherwise, just stores the tensor reference.
+ """
+ if self.static_buffer is not None:
+ # Copy into static buffer for CUDA graph compatibility.
+ num_tokens = topk_indices.shape[0]
+ self.static_buffer[:num_tokens].copy_(topk_indices)
+ self.recorded_topk_idx = self.static_buffer[:num_tokens]
+ else:
+ self.recorded_topk_idx = topk_indices
diff --git a/megatron/core/transformer/moe/shared_experts.py b/megatron/core/transformer/moe/shared_experts.py
index 3fa77c9ad25..d0bafba40db 100644
--- a/megatron/core/transformer/moe/shared_experts.py
+++ b/megatron/core/transformer/moe/shared_experts.py
@@ -9,6 +9,7 @@
from megatron.core.dist_checkpointing.mapping import ShardedStateDict
from megatron.core.enums import Fp8Recipe
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.fusions.fused_bias_geglu import bias_geglu_impl
from megatron.core.fusions.fused_bias_gelu import bias_gelu_impl
from megatron.core.fusions.fused_bias_swiglu import bias_swiglu_impl
@@ -21,12 +22,18 @@
from megatron.core.transformer.mlp import MLP, MLPSubmodules
from megatron.core.transformer.moe.moe_utils import ProcessGroupCollection
from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.typed_torch import apply_module
from megatron.core.utils import (
is_te_min_version,
is_torch_min_version,
make_sharded_tensor_for_checkpoint,
)
+if HAVE_TE:
+ from megatron.core.extensions.transformer_engine import TELinear, set_save_original_input
+else:
+ TELinear, set_save_original_input = None, None
+
class SharedExpertMLP(MLP):
"""
@@ -78,19 +85,8 @@ def __init__(
and "shared_experts" in config.recompute_modules
)
is_delayed_scaling = config.fp8_recipe == Fp8Recipe.delayed
- if not shared_experts_recompute and not is_delayed_scaling:
- try:
- HAVE_TE = True
- from megatron.core.extensions.transformer_engine import (
- TELinear,
- set_save_original_input,
- )
- except ImportError:
- HAVE_TE = False
- TELinear, set_save_original_input = None, None
-
- if HAVE_TE and isinstance(self.linear_fc1, TELinear):
- set_save_original_input(self.linear_fc1)
+ if not shared_experts_recompute and not is_delayed_scaling and HAVE_TE and isinstance(self.linear_fc1, TELinear):
+ set_save_original_input(self.linear_fc1)
if self.config.moe_shared_expert_overlap:
# disable TP related AG/RS communications in the linear module
@@ -187,7 +183,9 @@ def linear_fc1_forward_and_act(self, overlapped_comm_output=None):
set_tensor_grad_fn_sequence_sr(overlapped_comm_output, torch.iinfo(torch.int).max)
with torch.cuda.stream(self.stream):
# [s, b, 4 * h/p]
- intermediate_parallel, bias_parallel = self.linear_fc1(self.cached_fc1_input)
+ intermediate_parallel, bias_parallel = apply_module(self.linear_fc1)(
+ self.cached_fc1_input
+ )
self.cached_fc1_input = None
if self.config.use_te_activation_func:
@@ -238,7 +236,7 @@ def linear_fc2_forward(self, overlapped_comm_output=None):
set_tensor_grad_fn_sequence_sr(overlapped_comm_output, torch.iinfo(torch.int).max)
with torch.cuda.stream(self.stream):
# [s, b, h]
- self.cached_fc2_output, _ = self.linear_fc2(self.cached_fc2_input)
+ self.cached_fc2_output, _ = apply_module(self.linear_fc2)(self.cached_fc2_input)
self.cached_fc2_input = None
def post_forward_comm(self):
diff --git a/megatron/core/transformer/moe/token_dispatcher.py b/megatron/core/transformer/moe/token_dispatcher.py
index 6a327cfbfe4..987dd328753 100644
--- a/megatron/core/transformer/moe/token_dispatcher.py
+++ b/megatron/core/transformer/moe/token_dispatcher.py
@@ -295,11 +295,13 @@ def dispatch_postprocess(self, hidden_states, probs):
tokens_per_expert = self.local_map.sum(dim=0).long().cpu()
- (permuted_local_hidden_states, _, self.reversed_local_input_permutation_mapping) = permute(
- hidden_states,
- self.local_map,
- num_out_tokens=tokens_per_expert.sum(),
- fused=self.config.moe_permute_fusion,
+ permuted_local_hidden_states, _, self.reversed_local_input_permutation_mapping, _, _ = (
+ permute(
+ hidden_states,
+ self.local_map,
+ num_out_tokens=tokens_per_expert.sum().item(),
+ fused=self.config.moe_permute_fusion,
+ )
)
self.local_probs = self.local_probs.T.contiguous().masked_select(
@@ -636,6 +638,8 @@ def dispatch_preprocess(
permutated_local_input_tokens,
permuted_probs,
self.reversed_local_input_permutation_mapping,
+ _,
+ _,
) = permute(
hidden_states,
self.routing_map,
@@ -661,6 +665,7 @@ def token_dispatch(self, permutated_local_input_tokens, permuted_probs):
Returns:
A tuple of tokens and probabilities after All-to-All.
"""
+
# Perform expert parallel AlltoAll communication
self.tokens_per_expert = self._maybe_dtoh_and_synchronize(
"before_ep_alltoall", self.tokens_per_expert
@@ -1302,12 +1307,20 @@ def get_permuted_hidden_states_by_experts(self, hidden_states: torch.Tensor) ->
self.hidden_shape_before_permute = hidden_states.shape
assert self.dispatched_probs.dtype == torch.float32, "DeepEP only supports float32 probs"
- hidden_states, permuted_probs, self.reversed_mapping_for_combine = permute(
+ (
+ hidden_states,
+ permuted_probs,
+ self.reversed_mapping_for_combine,
+ self.pad_offsets,
+ self.tokens_per_expert,
+ ) = permute(
hidden_states,
self.dispatched_routing_map,
probs=self.dispatched_probs,
num_out_tokens=self.tokens_per_expert.sum().item(),
fused=self.permute_fusion,
+ tokens_per_expert=self.tokens_per_expert,
+ align_size=get_align_size_for_quantization(self.config),
)
if self.router_dtype == "fp64":
permuted_probs = permuted_probs.to(torch.float64)
@@ -1320,6 +1333,7 @@ def get_restored_hidden_states_by_experts(self, hidden_states: torch.Tensor) ->
restore_shape=self.hidden_shape_before_permute,
routing_map=self.dispatched_routing_map,
fused=self.permute_fusion,
+ pad_offsets=self.pad_offsets,
)
return hidden_states
diff --git a/megatron/core/transformer/moe/token_dispatcher_inference.py b/megatron/core/transformer/moe/token_dispatcher_inference.py
new file mode 100644
index 00000000000..b3b7b06b1b8
--- /dev/null
+++ b/megatron/core/transformer/moe/token_dispatcher_inference.py
@@ -0,0 +1,332 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""
+CUDA-graph-compatible token dispatcher for inference.
+
+This dispatcher is only used during CUDA-graphed inference iterations. It replaces
+AlltoAll with AllGather/ReduceScatter for token exchange, keeping all metadata
+GPU-resident to avoid host synchronizations that would break CUDA graph capture.
+
+Supports latency-optimized NVLS collectives (multimem all-gather/reduce-scatter)
+on Hopper+ GPUs with BF16, with automatic fallback to NCCL.
+"""
+
+from typing import List, Optional
+
+import torch
+
+from megatron.core.inference.communication.torch_symm_triton import (
+ are_tensors_nvls_eligible,
+ multimem_all_gather_fused,
+ multimem_reduce_scatter,
+)
+from megatron.core.inference.symmetric_memory import SymmetricMemoryManager
+from megatron.core.process_groups_config import ProcessGroupCollection
+from megatron.core.tensor_parallel import (
+ gather_from_sequence_parallel_region,
+ reduce_scatter_to_sequence_parallel_region,
+)
+from megatron.core.transformer.moe.token_dispatcher import MoEAllGatherTokenDispatcher
+from megatron.core.transformer.transformer_config import TransformerConfig
+
+
+class InferenceCUDAGraphTokenDispatcher(MoEAllGatherTokenDispatcher):
+ """
+ CUDA-graph-compatible AllGather token dispatcher for inference.
+
+ Only used during CUDA-graphed inference iterations. Swapped in by
+ MoELayer.set_inference_cuda_graphed_iteration() before graph capture
+ and swapped out by MoELayer.unset_inference_cuda_graphed_iteration() after.
+
+ Key features:
+ - AllGather/ReduceScatter instead of AlltoAll for CUDA graph compatibility
+ - GPU-resident metadata (no host synchronization)
+ - NVLS collectives on Hopper+ with automatic NCCL fallback
+ """
+
+ def __init__(
+ self,
+ num_local_experts: int,
+ local_expert_indices: List[int],
+ config: TransformerConfig,
+ pg_collection: Optional[ProcessGroupCollection] = None,
+ ) -> None:
+ """
+ Initialize the InferenceCUDAGraphTokenDispatcher.
+
+ Args:
+ num_local_experts: Number of experts on this rank.
+ local_expert_indices: Global indices of experts on this rank.
+ config: Transformer configuration.
+ pg_collection: Process group collection for distributed ops.
+ """
+ super().__init__(
+ num_local_experts=num_local_experts,
+ local_expert_indices=local_expert_indices,
+ config=config,
+ pg_collection=pg_collection,
+ )
+ self.topk = config.moe_router_topk
+
+ self.triton_nvls_kernels_allowed = not self.config.inference_disable_triton_nvls_kernels
+
+ def _maybe_allocate_ag_buffers(
+ self, routing_map: torch.Tensor, probs: torch.Tensor, hidden_states: torch.Tensor
+ ) -> dict:
+ """Allocate a single symmetric memory output buffer for fused all-gather.
+
+ Creates one contiguous symmetric memory buffer sized for the gathered
+ (global) routing_map, probs, and hidden_states, then returns sliced views
+ into it. This allows a single fused NVLS all-gather kernel to write all
+ three outputs in one launch.
+
+ Args:
+ routing_map (torch.Tensor): Local routing map, shape [local_tokens, topk].
+ Boolean or integer tensor mapping each token to its selected experts.
+ probs (torch.Tensor): Local routing probabilities, shape [local_tokens, topk].
+ Normalized weights for each token's selected experts.
+ hidden_states (torch.Tensor): Local hidden states, shape [local_tokens, hidden_dim].
+
+ Returns:
+ dict: A dictionary with the following keys:
+ - "handle": Symmetric memory handle for NVLS ops, or None if
+ symmetric memory is unavailable.
+ - "routing_map": Raw byte view for the gathered routing map output.
+ - "routing_map_offset": Byte offset of routing_map within the buffer.
+ - "probs": Raw byte view for the gathered probs output.
+ - "probs_offset": Byte offset of probs within the buffer.
+ - "hidden_states": Raw byte view for the gathered hidden states output.
+ - "hidden_states_offset": Byte offset of hidden_states within the buffer.
+ When allocation fails, all tensor views are None and offsets are 0.
+ """
+ _NONE = {
+ "handle": None,
+ "routing_map": None,
+ "routing_map_offset": 0,
+ "probs": None,
+ "probs_offset": 0,
+ "hidden_states": None,
+ "hidden_states_offset": 0,
+ }
+
+ local_tokens = probs.size(0)
+ global_tokens = local_tokens * self.ep_size
+ topk = probs.size(-1)
+ hidden_dim = hidden_states.size(-1)
+
+ result = SymmetricMemoryManager.get_buffer(
+ "ep", process_group=self.ep_group
+ ).maybe_get_tensors(
+ [
+ (global_tokens * topk, routing_map.dtype),
+ (global_tokens * topk, probs.dtype),
+ (global_tokens * hidden_dim, hidden_states.dtype),
+ ]
+ )
+
+ if result["handle"] is None:
+ return _NONE
+
+ (rm_buf, rm_off), (p_buf, p_off), (hs_buf, hs_off) = result["tensors"]
+ return {
+ "handle": result["handle"],
+ "routing_map": rm_buf,
+ "routing_map_offset": rm_off,
+ "probs": p_buf,
+ "probs_offset": p_off,
+ "hidden_states": hs_buf,
+ "hidden_states_offset": hs_off,
+ }
+
+ def _maybe_allocate_rs_buffer(self, x: torch.Tensor) -> dict:
+ """Allocate a symmetric memory buffer for reduce-scatter input.
+
+ The buffer has the same shape and dtype as x so that x can be copied
+ into it before the NVLS reduce-scatter kernel.
+
+ Args:
+ x (torch.Tensor): The global hidden states to be reduce-scattered,
+ shape [global_tokens, hidden_dim].
+
+ Returns:
+ dict: A dictionary with keys "handle" (symmetric memory handle, or
+ None if unavailable) and "tensor" (the allocated buffer, or None).
+ """
+ symm_mem_buffer = SymmetricMemoryManager.get_buffer(
+ "ep", process_group=self.ep_group
+ ).maybe_get_tensor(list(x.size()), dtype=x.dtype)
+ return symm_mem_buffer
+
+ def token_dispatch(self, hidden_states, probs):
+ """Gathers tokens from all EP ranks using AllGather.
+
+ Performs all-gather on routing_map (stored in self.routing_map), probs,
+ and hidden_states so that every rank holds the full global view.
+ Uses latency-optimized fused NVLS multimem_all_gather on Hopper+ GPUs
+ with BF16 when symmetric memory is available. Falls back to NCCL otherwise.
+
+ Args:
+ hidden_states (torch.Tensor): Local hidden states,
+ shape [local_tokens, hidden_dim].
+ probs (torch.Tensor): Local routing probabilities,
+ shape [local_tokens, topk]. Normalized weights for each token's
+ selected experts.
+
+ Returns:
+ tuple: (hidden_states, probs) gathered across all EP ranks.
+ - hidden_states (torch.Tensor): Shape [global_tokens, hidden_dim].
+ - probs (torch.Tensor): Shape [global_tokens, topk].
+ Also updates self.routing_map in-place to the gathered
+ shape [global_tokens, topk].
+ """
+ if self.ep_size == 1:
+ return hidden_states, probs
+
+ # 1. Check inputs only: if inputs are 16-byte divisible,
+ # outputs (world_size * input) are too.
+ nvls_eligible = self.triton_nvls_kernels_allowed and are_tensors_nvls_eligible(
+ hidden_states, probs, self.routing_map
+ )
+ ag_buffers = None
+
+ if nvls_eligible:
+ # 2. Now attempt to allocate symmetric memory buffers for
+ # all-gather outputs. If allocation fails, fallback to NCCL.
+ ag_buffers = self._maybe_allocate_ag_buffers(self.routing_map, probs, hidden_states)
+
+ # 3. Can use NVLS if eligible and buffers allocated successfully (handle is not None)
+ can_use_nvls = nvls_eligible and ag_buffers["handle"] is not None
+
+ if can_use_nvls:
+ # Capture shapes for reshaping after all-gather
+ # Output shape: [local_tokens * ep_size, dim]
+ local_tokens = probs.size(0)
+ global_tokens = local_tokens * self.ep_size
+ topk = probs.size(1)
+ hidden_dim = hidden_states.size(1)
+ routing_map_dtype = self.routing_map.dtype
+ probs_dtype = probs.dtype
+ hidden_dtype = hidden_states.dtype
+
+ # Fused NVLS all-gather: single kernel launch + single barrier for all 3 tensors
+ multimem_all_gather_fused(
+ ag_buffers["routing_map"].view(
+ torch.bfloat16
+ ), # .view does not change the underlying data
+ self.routing_map.view(torch.bfloat16),
+ ag_buffers["routing_map_offset"],
+ ag_buffers["probs"].view(torch.bfloat16),
+ probs.view(torch.bfloat16),
+ ag_buffers["probs_offset"],
+ ag_buffers["hidden_states"].view(torch.bfloat16),
+ hidden_states.view(torch.bfloat16),
+ ag_buffers["hidden_states_offset"],
+ ag_buffers["handle"],
+ )
+ self.routing_map = (
+ ag_buffers["routing_map"].view(routing_map_dtype).view(global_tokens, topk)
+ )
+ probs = ag_buffers["probs"].view(probs_dtype).view(global_tokens, topk)
+ hidden_states = (
+ ag_buffers["hidden_states"].view(hidden_dtype).view(global_tokens, hidden_dim)
+ )
+ else:
+ # Fallback to NCCL for all tensors
+ with torch.no_grad():
+ self.routing_map = gather_from_sequence_parallel_region(
+ self.routing_map, group=self.tp_ep_group
+ )
+ probs = gather_from_sequence_parallel_region(probs, group=self.tp_ep_group)
+ hidden_states = gather_from_sequence_parallel_region(
+ hidden_states, group=self.tp_ep_group
+ )
+
+ return hidden_states, probs
+
+ def dispatch_postprocess(self, hidden_states, probs):
+ """Pass-through: returns inputs directly without permutation.
+
+ Unlike the training dispatcher, this does not permute tokens or compute
+ tokens_per_expert. The downstream InferenceGroupedMLP (FlashInfer /
+ CUTLASS fused MoE kernel) operates directly on the routing map stored
+ in self.routing_map.
+
+ Args:
+ hidden_states (torch.Tensor): Gathered hidden states,
+ shape [global_tokens, hidden_dim].
+ probs (torch.Tensor): Gathered routing probabilities,
+ shape [global_tokens, topk].
+
+ Returns:
+ tuple: (hidden_states, tokens_per_expert, probs) where
+ tokens_per_expert is always None.
+ """
+ return hidden_states, None, probs
+
+ def combine_preprocess(self, expert_output):
+ """Pass-through: InferenceGroupedMLP already produces unpermuted output.
+
+ No unpermutation is needed because dispatch_postprocess did not permute
+ the tokens in the first place.
+
+ Args:
+ expert_output (torch.Tensor): Output from InferenceGroupedMLP,
+ shape [global_tokens, hidden_dim].
+
+ Returns:
+ torch.Tensor: The input tensor unchanged.
+ """
+ return expert_output
+
+ def token_combine(self, hidden_states):
+ """Combines expert outputs across EP ranks using Reduce-Scatter.
+
+ Reduces the global expert output (summing contributions from each rank)
+ and scatters the result so each rank receives its local token slice.
+ Uses latency-optimized NVLS multimem_reduce_scatter on Hopper+ GPUs
+ with BF16 when symmetric memory is available. Falls back to NCCL otherwise.
+
+ Args:
+ hidden_states (torch.Tensor): Combined expert output after routing
+ weights have been applied, shape [global_tokens, hidden_dim].
+
+ Returns:
+ torch.Tensor: Local slice of the reduced output,
+ shape [local_tokens, hidden_dim] where
+ local_tokens = global_tokens // ep_size.
+ """
+ if self.ep_size == 1:
+ return hidden_states
+
+ # Compute output shape first — check NVLS eligibility on the output,
+ # since if the smaller output is 16-byte divisible, the input is too.
+ output_shape = list(hidden_states.size())
+ output_shape[0] = hidden_states.size(0) // self.ep_size
+ output = torch.empty(output_shape, dtype=hidden_states.dtype, device=hidden_states.device)
+
+ # Check output only: if output is 16-byte divisible, input (world_size * output) is too.
+ nvls_eligible = (
+ self.triton_nvls_kernels_allowed
+ and output.dtype in (torch.bfloat16, torch.float32)
+ and are_tensors_nvls_eligible(output)
+ )
+ rs_buffer = None
+
+ if nvls_eligible:
+ rs_buffer = self._maybe_allocate_rs_buffer(hidden_states)
+
+ can_use_nvls = nvls_eligible and rs_buffer["handle"] is not None
+
+ if can_use_nvls:
+ # Copy input to symmetric memory for reduce-scatter
+ rs_buffer["tensor"].copy_(hidden_states)
+
+ # Use latency-optimized NVLS reduce-scatter
+ multimem_reduce_scatter(output, rs_buffer["tensor"], rs_buffer["handle"])
+ return output.to(torch.bfloat16)
+ else:
+ # Fallback to NCCL
+ hidden_states = reduce_scatter_to_sequence_parallel_region(
+ hidden_states, group=self.tp_ep_group
+ )
+ return hidden_states.to(torch.bfloat16)
diff --git a/megatron/core/transformer/multi_latent_attention.py b/megatron/core/transformer/multi_latent_attention.py
index bdbfd84c450..b070526c5fc 100644
--- a/megatron/core/transformer/multi_latent_attention.py
+++ b/megatron/core/transformer/multi_latent_attention.py
@@ -29,6 +29,8 @@
from megatron.core import tensor_parallel
+from megatron.core.dist_checkpointing.mapping import ShardedObject
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.enums import Fp8Recipe
from megatron.core.fp8_utils import get_fp8_align_size
from megatron.core.models.common.embeddings import (
@@ -48,13 +50,17 @@
scatter_to_sequence_parallel_region,
)
from megatron.core.transformer.attention import Attention
-from megatron.core.transformer.custom_layers.transformer_engine import (
- split_te_layernorm_column_parallel_linear,
-)
from megatron.core.transformer.enums import AttnMaskType
from megatron.core.transformer.spec_utils import ModuleSpec, build_module
+from megatron.core.transformer.torch_norm import LayerNormBuilder
from megatron.core.transformer.transformer_config import MLATransformerConfig
-from megatron.core.utils import deprecate_inference_params, get_pg_size, is_te_min_version
+from megatron.core.typed_torch import apply_module
+from megatron.core.utils import (
+ deprecate_inference_params,
+ get_pg_size,
+ is_te_min_version,
+ make_tp_sharded_tensor_for_checkpoint,
+)
try:
from megatron.core.fusions.fused_mla_yarn_rope_apply import (
@@ -66,33 +72,42 @@
fused_apply_mla_rope_for_q = None
-try:
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import (
TEColumnParallelLinear,
+ TELayerNormColumnParallelLinear,
TELinear,
set_save_original_input,
+ split_te_layernorm_column_parallel_linear,
)
from megatron.core.post_training.modelopt.layers import Linear
-
- HAVE_TE = True
-except ImportError:
- TEColumnParallelLinear, TELinear, Linear, set_save_original_input = None, None, None, None
- HAVE_TE = False
+else:
+ (
+ TEColumnParallelLinear,
+ TELayerNormColumnParallelLinear,
+ TELinear,
+ Linear,
+ set_save_original_input,
+ split_te_layernorm_column_parallel_linear,
+ ) = (None, None, None, None, None, None)
@dataclass
class MLASelfAttentionSubmodules:
"""Submodules for the MLA self-attention layer."""
+ # TODO(nschank): Move layernorms back to the bottom once all other layers have defaults removed.
+ q_layernorm: LayerNormBuilder
+ kv_layernorm: LayerNormBuilder
+
linear_q_proj: Union[ModuleSpec, type] = None
linear_q_down_proj: Union[ModuleSpec, type] = None
linear_q_up_proj: Union[ModuleSpec, type] = None
linear_kv_down_proj: Union[ModuleSpec, type] = None
linear_kv_up_proj: Union[ModuleSpec, type] = None
+ linear_qkv_down_proj: Union[ModuleSpec, type] = None
core_attention: Union[ModuleSpec, type] = None
linear_proj: Union[ModuleSpec, type] = None
- q_layernorm: Union[ModuleSpec, type] = None
- kv_layernorm: Union[ModuleSpec, type] = None
def get_attention_sink_bias(batch_size, num_heads, seq_len, window_size=None, sink_k=1, dtype=torch.bfloat16):
"""
@@ -582,20 +597,46 @@ def __init__(
)
if self.config.q_lora_rank is not None:
- self.q_layernorm = build_module(
- submodules.q_layernorm,
+ self.q_layernorm = submodules.q_layernorm(
hidden_size=self.config.q_lora_rank,
config=self.config,
eps=self.config.layernorm_epsilon,
)
- self.kv_layernorm = build_module(
- submodules.kv_layernorm,
+ self.kv_layernorm = submodules.kv_layernorm(
hidden_size=self.config.kv_lora_rank,
config=self.config,
eps=self.config.layernorm_epsilon,
)
+ def _qkv_down_projection(self, hidden_states):
+ """Unfused q/kv down projection path."""
+ if self.config.q_lora_rank is not None:
+ # if linear_q_down_proj is ColumnParallelLinear:
+ # q_compressed: [s, b, q_lora_rank / TP]
+ # elif linear_q_down_proj is Linear:
+ # q_compressed: [s / TP, b, q_lora_rank]
+ q_compressed, _ = self.linear_q_down_proj(hidden_states)
+
+ # When output is sharded (ColumnParallelLinear), two things are needed to be
+ # identical to a normal Linear.
+ # 1. Manually gather output to restore output dim q_lora_rank;
+ # 2. Scatter sequence back to s / TP if sequence-parallel since it was
+ # gathered by ColumnParallelLinear.
+ if q_compressed.size(-1) != self.config.q_lora_rank:
+ q_compressed = gather_from_tensor_model_parallel_region(q_compressed)
+ if self.config.sequence_parallel:
+ q_compressed = scatter_to_sequence_parallel_region(q_compressed)
+ else:
+ q_compressed = hidden_states
+
+ # if linear_kv_down_proj is ColumnParallelLinear:
+ # kv_combined: [s, b, (kv_lora_rank + qk_pos_emb_head_dim) / TP]
+ # elif linear_kv_down_proj is Linear:
+ # kv_combined: [s / TP, b, (kv_lora_rank + qk_pos_emb_head_dim)]
+ kv_combined, _ = self.linear_kv_down_proj(hidden_states)
+ return q_compressed, kv_combined
+
def get_query_key_value_tensors(
self,
hidden_states,
@@ -666,30 +707,7 @@ def get_query_key_value_tensors(
# =========================================
# QKV down projection and layernorm
# =========================================
- if self.config.q_lora_rank is not None:
- # if linear_q_down_proj is ColumnParallelLinear:
- # q_compressed: [s, b, q_lora_rank / TP]
- # elif linear_q_down_proj is Linear:
- # q_compressed: [s / TP, b, q_lora_rank]
- q_compressed, _ = self.linear_q_down_proj(hidden_states)
-
- # When output is sharded (ColumnParallelLinear), two things are needed to be
- # identical to a normal Linear.
- # 1. Manually gather output to restore output dim q_lora_rank;
- # 2. Scatter sequence back to s / TP if sequence-parallel since it was
- # gathered by ColumnParallelLinear.
- if q_compressed.size(-1) != self.config.q_lora_rank:
- q_compressed = gather_from_tensor_model_parallel_region(q_compressed)
- if self.config.sequence_parallel:
- q_compressed = scatter_to_sequence_parallel_region(q_compressed)
- else:
- q_compressed = hidden_states
-
- # if linear_kv_down_proj is ColumnParallelLinear:
- # kv_combined: [s, b, (kv_down_proj_out_size) / TP]
- # elif linear_kv_down_proj is Linear:
- # kv_combined: [s / TP, b, kv_down_proj_out_size]
- kv_combined, _ = self.linear_kv_down_proj(hidden_states)
+ q_compressed, kv_combined = self._qkv_down_projection(hidden_states)
kv_down_proj_out_size = (
self.config.kv_lora_rank
+ self.config.qk_pos_emb_head_dim
@@ -697,24 +715,31 @@ def get_query_key_value_tensors(
)
kv_is_tp_sharded = kv_combined.size(-1) != kv_down_proj_out_size
if kv_is_tp_sharded:
+ # kv_combined: [s, b, (kv_down_proj_out_size)]
kv_combined = gather_from_tensor_model_parallel_region(kv_combined)
- if self.kv_down_proj_mxfp8_padding > 0:
- kv_combined = kv_combined[
- ..., : self.config.kv_lora_rank + self.config.qk_pos_emb_head_dim
- ]
- # kv_compressed: [..., kv_lora_rank], k_pos_emb: [..., qk_pos_emb_head_dim]
- kv_compressed, k_pos_emb = torch.split(
- kv_combined, [self.config.kv_lora_rank, self.config.qk_pos_emb_head_dim], dim=-1
- )
- if kv_is_tp_sharded and self.config.sequence_parallel:
- kv_compressed = scatter_to_sequence_parallel_region(kv_compressed)
- elif (
- not kv_is_tp_sharded
- and get_pg_size(self.tp_group) > 1
- and self.config.sequence_parallel
- ):
- # k_pos_emb: [s, b, qk_pos_emb_head_dim]
- k_pos_emb = gather_from_sequence_parallel_region(k_pos_emb, group=self.tp_group)
+ if self.kv_down_proj_mxfp8_padding > 0:
+ kv_combined = kv_combined[
+ ..., : self.config.kv_lora_rank + self.config.qk_pos_emb_head_dim
+ ]
+ # kv_compressed:[s, b, kv_lora_rank], k_pos_emb: [s, b, qk_pos_emb_head_dim]
+ kv_compressed, k_pos_emb = torch.split(
+ kv_combined, [self.config.kv_lora_rank, self.config.qk_pos_emb_head_dim], dim=-1
+ )
+ if self.config.sequence_parallel:
+ # kv_compressed:[s / TP, b, kv_lora_rank]
+ kv_compressed = scatter_to_sequence_parallel_region(kv_compressed)
+ else:
+ if self.kv_down_proj_mxfp8_padding > 0:
+ kv_combined = kv_combined[
+ ..., : self.config.kv_lora_rank + self.config.qk_pos_emb_head_dim
+ ]
+ # kv_compressed:[s / TP, b, kv_lora_rank], k_pos_emb: [s / TP, b, qk_pos_emb_head_dim]
+ kv_compressed, k_pos_emb = torch.split(
+ kv_combined, [self.config.kv_lora_rank, self.config.qk_pos_emb_head_dim], dim=-1
+ )
+ if get_pg_size(self.tp_group) > 1 and self.config.sequence_parallel:
+ # k_pos_emb: [s, b, qk_pos_emb_head_dim]
+ k_pos_emb = gather_from_sequence_parallel_region(k_pos_emb, group=self.tp_group)
if packed_seq_params is not None:
# If sequence packing, TE expect [t, h, d] shaped qkv input.
@@ -730,9 +755,9 @@ def get_query_key_value_tensors(
if self.config.q_lora_rank is not None:
# q_compressed: [num_tokens, q_lora_rank]
- q_compressed = self.q_layernorm(q_compressed)
+ q_compressed = apply_module(self.q_layernorm)(q_compressed)
- kv_compressed = self.kv_layernorm(kv_compressed)
+ kv_compressed = apply_module(self.kv_layernorm)(kv_compressed)
# =========================================
# QKV up projection and RoPE apply
@@ -1063,8 +1088,6 @@ def _backward_output_proj(self):
def set_for_recompute_input_layernorm(self):
"""Set the attention layer for recompute input_layernorm. Only needed for fp8/fp4."""
- from megatron.core.extensions.transformer_engine import set_save_original_input
-
if self.config.q_lora_rank is not None:
set_save_original_input(self.linear_q_down_proj)
set_save_original_input(self.linear_kv_down_proj)
@@ -1188,3 +1211,246 @@ def _clip_kv_proj_weight(self, weight):
)
return weight_kv_updated
+
+
+class FusedMLASelfAttention(MLASelfAttention):
+ """MLA self-attention with fused q/kv down projection."""
+
+ def __init__(
+ self,
+ config: MLATransformerConfig,
+ submodules: MLASelfAttentionSubmodules,
+ layer_number: int,
+ attn_mask_type=AttnMaskType.padding,
+ cp_comm_type: Optional[str] = None,
+ pg_collection: Optional[ProcessGroupCollection] = None,
+ ):
+ if pg_collection is None:
+ pg_collection = ProcessGroupCollection.use_mpu_process_groups()
+
+ MultiLatentAttention.__init__(
+ self,
+ config=config,
+ submodules=submodules,
+ layer_number=layer_number,
+ attn_mask_type=attn_mask_type,
+ attention_type="self",
+ cp_comm_type=cp_comm_type,
+ pg_collection=pg_collection,
+ )
+
+ assert self.config.q_lora_rank is not None, (
+ "FusedMLASelfAttention requires q_lora_rank to be set; "
+ "fallback to MLASelfAttention for q_lora_rank=None."
+ )
+
+ qkv_down_proj_kwargs = {}
+ if submodules.linear_qkv_down_proj in [TELinear]:
+ qkv_down_proj_kwargs['parallel_mode'] = 'duplicated'
+ elif submodules.linear_qkv_down_proj in [
+ Linear,
+ TEColumnParallelLinear,
+ ColumnParallelLinear,
+ TELayerNormColumnParallelLinear,
+ ]:
+ qkv_down_proj_kwargs['gather_output'] = False
+ else:
+ raise ValueError(f"Unsupported linear_qkv_down_proj: {submodules.linear_qkv_down_proj}")
+
+ kv_down_proj_core_size = self.config.kv_lora_rank + self.config.qk_pos_emb_head_dim
+ self.kv_down_proj_mxfp8_padding = 0
+ qkv_down_out = self.config.q_lora_rank + kv_down_proj_core_size
+ if self.config.fp8 and self.config.fp8_recipe == Fp8Recipe.mxfp8:
+ align = get_fp8_align_size(Fp8Recipe.mxfp8)
+ remainder = kv_down_proj_core_size % align
+ if remainder != 0:
+ self.kv_down_proj_mxfp8_padding = align - remainder
+ qkv_down_out += self.kv_down_proj_mxfp8_padding
+
+ self.linear_qkv_down_proj = build_module(
+ submodules.linear_qkv_down_proj,
+ self.config.hidden_size,
+ qkv_down_out,
+ config=self.config,
+ init_method=self.config.init_method,
+ bias=False,
+ skip_bias_add=False,
+ is_expert=False,
+ tp_comm_buffer_name='qkv_down_proj',
+ skip_weight_param_allocation=False,
+ tp_group=(
+ pg_collection.tp
+ if qkv_down_proj_kwargs.get('parallel_mode') != 'duplicated'
+ else None
+ ),
+ **qkv_down_proj_kwargs,
+ )
+
+ self.linear_q_up_proj = build_module(
+ submodules.linear_q_up_proj,
+ self.config.q_lora_rank,
+ self.config.num_attention_heads * self.q_head_dim,
+ config=self.config,
+ init_method=self.config.init_method,
+ gather_output=False,
+ bias=False,
+ skip_bias_add=False,
+ is_expert=False,
+ tp_comm_buffer_name='q_up_proj',
+ tp_group=pg_collection.tp,
+ )
+
+ self.linear_kv_up_proj = build_module(
+ submodules.linear_kv_up_proj,
+ self.config.kv_lora_rank,
+ self.config.num_attention_heads * (self.config.qk_head_dim + self.config.v_head_dim),
+ config=self.config,
+ init_method=self.config.init_method,
+ gather_output=False,
+ bias=False,
+ skip_bias_add=False,
+ is_expert=False,
+ tp_comm_buffer_name='kv_up_proj',
+ tp_group=pg_collection.tp,
+ )
+
+ self.q_layernorm = submodules.q_layernorm(
+ hidden_size=self.config.q_lora_rank,
+ config=self.config,
+ eps=self.config.layernorm_epsilon,
+ )
+ self.kv_layernorm = submodules.kv_layernorm(
+ hidden_size=self.config.kv_lora_rank,
+ config=self.config,
+ eps=self.config.layernorm_epsilon,
+ )
+
+ def _qkv_down_projection(self, hidden_states):
+ """Fused q/kv down projection path."""
+ qkv, _ = self.linear_qkv_down_proj(hidden_states)
+ kv_dim = (
+ self.config.kv_lora_rank
+ + self.config.qk_pos_emb_head_dim
+ + self.kv_down_proj_mxfp8_padding
+ )
+ q_compressed, kv_combined = torch.split(
+ qkv,
+ [self.config.q_lora_rank, kv_dim],
+ dim=-1,
+ )
+ return q_compressed, kv_combined
+
+ def sharded_state_dict(self, prefix: str = "", sharded_offsets: tuple = (), metadata=None):
+ """Return a sharded state dict compatible with pre-fusion checkpoints."""
+ sharded_state_dict = super().sharded_state_dict(prefix, sharded_offsets, metadata)
+
+ def _clone_sharded_object_with_key(obj: ShardedObject, new_key: str) -> ShardedObject:
+ return ShardedObject(
+ key=new_key,
+ data=obj.data,
+ global_shape=obj.global_shape,
+ global_offset=obj.global_offset,
+ replica_id=obj.replica_id,
+ )
+
+ fused_prefix = f"{prefix}linear_qkv_down_proj."
+
+ fused_extra_keys = [
+ k
+ for k in sharded_state_dict.keys()
+ if k.startswith(fused_prefix) and "_extra_state" in k
+ ]
+ for fused_extra_key in fused_extra_keys:
+ suffix = fused_extra_key[len(fused_prefix) :]
+ q_extra_key = f"{prefix}linear_q_down_proj.{suffix}"
+ kv_extra_key = f"{prefix}linear_kv_down_proj.{suffix}"
+ fused_obj = sharded_state_dict.get(fused_extra_key)
+ if isinstance(fused_obj, ShardedObject):
+ sharded_state_dict[q_extra_key] = _clone_sharded_object_with_key(
+ fused_obj, q_extra_key
+ )
+ sharded_state_dict[kv_extra_key] = _clone_sharded_object_with_key(
+ fused_obj, kv_extra_key
+ )
+ elif fused_obj is not None:
+ sharded_state_dict[q_extra_key] = fused_obj
+ sharded_state_dict[kv_extra_key] = fused_obj
+
+ for key in list(sharded_state_dict.keys()):
+ if key.startswith(fused_prefix):
+ del sharded_state_dict[key]
+
+ fused_weight = self.linear_qkv_down_proj.weight
+ total_out = (
+ self.config.q_lora_rank
+ + self.config.kv_lora_rank
+ + self.config.qk_pos_emb_head_dim
+ + self.kv_down_proj_mxfp8_padding
+ )
+ tp_size = get_pg_size(self.tp_group)
+
+ if fused_weight.size(0) == total_out:
+ q_split = self.config.q_lora_rank
+ kv_split = (
+ self.config.kv_lora_rank
+ + self.config.qk_pos_emb_head_dim
+ + self.kv_down_proj_mxfp8_padding
+ )
+ else:
+ assert (
+ self.config.q_lora_rank % tp_size == 0
+ ), "q_lora_rank must be divisible by tensor-parallel size"
+ assert (
+ self.config.kv_lora_rank
+ + self.config.qk_pos_emb_head_dim
+ + self.kv_down_proj_mxfp8_padding
+ ) % tp_size == 0, (
+ "kv_lora_rank + qk_pos_emb_head_dim (+ mxfp8 padding) must be divisible by "
+ "tensor-parallel size"
+ )
+ q_split = self.config.q_lora_rank // tp_size
+ kv_split = (
+ self.config.kv_lora_rank
+ + self.config.qk_pos_emb_head_dim
+ + self.kv_down_proj_mxfp8_padding
+ ) // tp_size
+
+ if q_split + kv_split != fused_weight.size(0):
+ raise ValueError(
+ "Unexpected fused qkv-down weight shape: "
+ f"got {tuple(fused_weight.size())}, expected dim0 {q_split + kv_split}"
+ )
+
+ q_weight, kv_weight = torch.split(fused_weight, [q_split, kv_split], dim=0)
+
+ q_key = f"{prefix}linear_q_down_proj.weight"
+ kv_key = f"{prefix}linear_kv_down_proj.weight"
+
+ sharded_state_dict[q_key] = make_tp_sharded_tensor_for_checkpoint(
+ tensor=q_weight, key=q_key, tp_axis=0, prepend_offsets=sharded_offsets
+ )
+ sharded_state_dict[kv_key] = make_tp_sharded_tensor_for_checkpoint(
+ tensor=kv_weight, key=kv_key, tp_axis=0, prepend_offsets=sharded_offsets
+ )
+
+ return sharded_state_dict
+
+ def _load_from_state_dict(self, state_dict, prefix, *args, **kwargs):
+ """Load state dict with automatic unfused->fused conversion."""
+ q_key = f"{prefix}linear_q_down_proj.weight"
+ kv_key = f"{prefix}linear_kv_down_proj.weight"
+ fused_key = f"{prefix}linear_qkv_down_proj.weight"
+
+ def _as_tensor(x):
+ return x.data if hasattr(x, 'data') else x
+
+ if fused_key not in state_dict and q_key in state_dict and kv_key in state_dict:
+ q_weight = _as_tensor(state_dict[q_key])
+ kv_weight = _as_tensor(state_dict[kv_key])
+ state_dict[fused_key] = torch.cat([q_weight, kv_weight], dim=0)
+ del state_dict[q_key]
+ del state_dict[kv_key]
+ state_dict.pop(f"{prefix}linear_q_down_proj.bias", None)
+ state_dict.pop(f"{prefix}linear_kv_down_proj.bias", None)
+
+ return super()._load_from_state_dict(state_dict, prefix, *args, **kwargs)
diff --git a/megatron/core/transformer/multi_token_prediction.py b/megatron/core/transformer/multi_token_prediction.py
index 6432af36cde..8fe7a2636b0 100755
--- a/megatron/core/transformer/multi_token_prediction.py
+++ b/megatron/core/transformer/multi_token_prediction.py
@@ -1,16 +1,18 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+from __future__ import annotations
import warnings
from contextlib import nullcontext
from dataclasses import dataclass
-from typing import Callable, List, Optional, Union
+from typing import TYPE_CHECKING, Callable, List, Optional, Union
import torch
from torch import Tensor
from megatron.core import InferenceParams, parallel_state, tensor_parallel
from megatron.core.dist_checkpointing.mapping import ShardedStateDict
-from megatron.core.dist_checkpointing.utils import replace_prefix_for_sharding
+from megatron.core.dist_checkpointing.utils import apply_prefix_mapping, replace_prefix_for_sharding
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.fp8_utils import get_fp8_context
from megatron.core.models.backends import BackendSpecProvider, LocalSpecProvider
from megatron.core.packed_seq_params import PackedSeqParams
@@ -23,8 +25,10 @@
from megatron.core.transformer.enums import AttnMaskType, LayerType
from megatron.core.transformer.module import MegatronModule
from megatron.core.transformer.spec_utils import ModuleSpec, build_module
+from megatron.core.transformer.torch_norm import LayerNormBuilder
from megatron.core.transformer.transformer_block import TransformerBlockSubmodules
from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.typed_torch import apply_module
from megatron.core.utils import (
get_pg_rank,
is_torch_min_version,
@@ -32,6 +36,9 @@
make_viewless_tensor,
)
+if TYPE_CHECKING:
+ from megatron.core.ssm.mamba_block import MambaStackSubmodules
+
if is_torch_min_version("1.13.0"):
dist_all_gather_func = torch.distributed.all_gather_into_tensor
else:
@@ -44,14 +51,10 @@
AttnMaskType.padding_causal,
]
-try:
- import transformer_engine as te # pylint: disable=unused-import
-
+if HAVE_TE:
from megatron.core.extensions.transformer_engine_spec_provider import TESpecProvider
-
- HAVE_TE = True
-except ImportError:
- HAVE_TE = False
+else:
+ TESpecProvider = None
def tie_word_embeddings_state_dict(
@@ -265,6 +268,13 @@ def _roll_tensor_packed_seq(tensor, shifts, dims, packed_seq_params, cp_group=No
# the idx has been multiplied by cp_size, need to divide it by cp_size to get the local idx
local_start_idx = start_idx // cp_size
local_end_idx = end_idx // cp_size
+
+ # Skip empty sequences - this can happen when a sequence is very short and
+ # after dividing by cp_size, the local slice has zero length
+ local_seq_len = local_end_idx - local_start_idx
+ if local_seq_len == 0:
+ continue
+
tensor_slice = rolled_tensor[..., local_start_idx:local_end_idx].clone()
# The following code is very similar as the code in roll_tensor function
@@ -274,6 +284,15 @@ def _roll_tensor_packed_seq(tensor, shifts, dims, packed_seq_params, cp_group=No
tensor_send_list = []
tensor_recv_list = []
for chunk in rolled_chunks:
+ # Skip empty chunks that can occur when the sequence slice is very small
+ if chunk.size(dims) == 0:
+ tensor_send_list.append(
+ torch.empty(chunk.shape[:-1], dtype=chunk.dtype, device=chunk.device)
+ )
+ tensor_recv_list.append(
+ torch.empty(chunk.shape[:-1], dtype=chunk.dtype, device=chunk.device)
+ )
+ continue
boundary = chunk.select(dims, shifts).contiguous().clone()
tensor_send_list.append(boundary)
tensor_recv_list.append(torch.empty_like(boundary))
@@ -297,6 +316,9 @@ def _roll_tensor_packed_seq(tensor, shifts, dims, packed_seq_params, cp_group=No
index = [slice(None)] * rolled_chunks[0].dim()
index[dims] = shifts
for chunk, recv in zip(rolled_chunks, tensor_recv_list):
+ # Skip empty chunks
+ if chunk.size(dims) == 0:
+ continue
chunk[tuple(index)] = recv
seq_result = torch.cat(rolled_chunks, dim=dims)
@@ -390,21 +412,22 @@ class MultiTokenPredictionLayerSubmodules:
Dataclass for specifying the submodules of a MultiTokenPrediction module.
Args:
- hnorm (Union[ModuleSpec, type]): Specification or instance of the
- hidden states normalization to be applied.
- enorm (Union[ModuleSpec, type]): Specification or instance of the
- embedding normalization to be applied.
+ hnorm: Specification or instance of the hidden states normalization to be applied.
+ enorm: Specification or instance of the embedding normalization to be applied.
eh_proj (Union[ModuleSpec, type]): Specification or instance of the
linear projection to be applied.
mtp_model_layer (Union[ModuleSpec, type]): Specification
or instance of the transformer or mamba block to be applied.
"""
- enorm: Union[ModuleSpec, type] = None
- hnorm: Union[ModuleSpec, type] = None
+ enorm: LayerNormBuilder
+ hnorm: LayerNormBuilder
+ # TODO(nschank): Move this back below transformer_layer once eh_proj and transformer_layer have
+ # their defaults removed.
+ layer_norm: LayerNormBuilder
+
eh_proj: Union[ModuleSpec, type] = None
mtp_model_layer: Union[ModuleSpec, type] = None
- layer_norm: Union[ModuleSpec, type] = None
def get_mtp_layer_spec(
@@ -430,7 +453,7 @@ def get_mtp_layer_spec_for_backend(
ModuleSpec: Module specification with modules from the backend.
"""
column_parallel_linear_impl: type = backend.column_parallel_linear()
- layer_norm_impl: type = backend.layer_norm()
+ layer_norm_impl = backend.layer_norm()
mtp_layer_spec = ModuleSpec(
module=MultiTokenPredictionLayer,
submodules=MultiTokenPredictionLayerSubmodules(
@@ -598,6 +621,7 @@ def process_mtp_loss(
config: TransformerConfig,
cp_group: Optional[torch.distributed.ProcessGroup] = None,
packed_seq_params: Optional[PackedSeqParams] = None,
+ scale_logits_fn: Optional[Callable[[Tensor], Tensor]] = None,
) -> Tensor:
"""Process Multi-Token Prediction (MTP) loss computation.
@@ -616,23 +640,35 @@ def process_mtp_loss(
config (TransformerConfig): Model configuration containing mtp_num_layers etc.
cp_group (Optional[ProcessGroup]): Context parallelism process group.
packed_seq_params (Optional[PackedSeqParams]): Packed sequence parameters.
+ scale_logits_fn (Optional[Callable[[Tensor], Tensor]]): Optional function to
+ scale logits before loss computation (e.g., MuP output scaling).
Returns:
Tensor: Updated hidden states after MTP loss processing (first chunk only).
"""
- mtp_labels = labels.clone()
hidden_states_list = torch.chunk(hidden_states, 1 + config.mtp_num_layers, dim=0)
hidden_states = hidden_states_list[0]
+ if labels is None:
+ return hidden_states
+
+ mtp_labels = labels.clone()
if loss_mask is None:
loss_mask = torch.ones_like(mtp_labels)
+ # Store the original number of tokens before rolling for proper normalization
+ # when calculate_per_token_loss is enabled. This ensures MTP gradients are
+ # correctly scaled relative to the main loss gradients in finalize_model_grads.
+ original_num_tokens = loss_mask.sum()
+
for mtp_layer_number in range(config.mtp_num_layers):
mtp_logits, _ = output_layer(
hidden_states_list[mtp_layer_number + 1],
weight=output_weight,
runtime_gather_output=runtime_gather_output,
)
+ if scale_logits_fn is not None:
+ mtp_logits = scale_logits_fn(mtp_logits)
mtp_labels, _ = roll_tensor(
mtp_labels, shifts=-1, dims=-1, cp_group=cp_group, packed_seq_params=packed_seq_params
)
@@ -642,18 +678,32 @@ def process_mtp_loss(
mtp_loss = compute_language_model_loss(mtp_labels, mtp_logits)
mtp_loss = loss_mask * mtp_loss
if is_training:
+ mtp_loss_for_log = (
+ torch.sum(mtp_loss) / num_tokens if num_tokens > 0 else mtp_loss.new_tensor(0.0)
+ )
MTPLossLoggingHelper.save_loss_to_tracker(
- torch.sum(mtp_loss) / num_tokens,
+ mtp_loss_for_log,
mtp_layer_number,
config.mtp_num_layers,
avg_group=parallel_state.get_data_parallel_group(with_context_parallel=True),
)
mtp_loss_scale = config.mtp_loss_scaling_factor / config.mtp_num_layers
if config.calculate_per_token_loss:
- hidden_states = MTPLossAutoScaler.apply(hidden_states, mtp_loss_scale * mtp_loss)
+ # When calculate_per_token_loss is enabled, finalize_model_grads will
+ # divide all gradients by total_num_tokens (from main loss).
+ # However, MTP has fewer valid tokens due to rolling. To ensure correct
+ # per-token gradient weighting, we normalize by the rolled token count
+ # and re-scale by the original token count.
+ # Avoid division by zero
+ num_tokens_safe = torch.clamp(num_tokens, min=1)
+ mtp_loss_normalized = (
+ mtp_loss_scale * mtp_loss * (original_num_tokens / num_tokens_safe)
+ )
+ hidden_states = MTPLossAutoScaler.apply(hidden_states, mtp_loss_normalized)
else:
+ safe_num_tokens = num_tokens.clamp(min=1)
hidden_states = MTPLossAutoScaler.apply(
- hidden_states, mtp_loss_scale * mtp_loss / num_tokens
+ hidden_states, mtp_loss_scale * mtp_loss / safe_num_tokens
)
return hidden_states
@@ -675,8 +725,8 @@ class MultiTokenPredictionLayer(MegatronModule):
the linear projection. The combined serves as the input of the Transformer block at
the k-th depth to produce the output representation.
- for more information, please refer to DeepSeek-V3 Technical Report
- https://github.com/deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdf
+ For more information, refer to DeepSeek-V3 Technical Report
+ https://arxiv.org/pdf/2412.19437.pdf
"""
def __init__(
@@ -688,7 +738,7 @@ def __init__(
pg_collection: Optional[ProcessGroupCollection] = None,
# For Mamba path - pattern and submodules to build inner layers directly
mtp_layer_pattern: Optional[str] = None,
- mamba_submodules: Optional["MambaStackSubmodules"] = None,
+ mamba_submodules: Optional[MambaStackSubmodules] = None,
):
super().__init__(config=config)
self.sequence_parallel = config.sequence_parallel
@@ -696,41 +746,44 @@ def __init__(
self.layer_number = layer_number + get_mtp_layer_offset(self.config, vp_stage)
self.vp_stage = vp_stage
self.cp_group = pg_collection.cp
+ self.tp_group = pg_collection.tp if pg_collection is not None else None
self.mtp_layer_pattern = mtp_layer_pattern
# Validate attention mask type if using transformer-based inner layers
if self.submodules.mtp_model_layer is not None and hasattr(
self.submodules.mtp_model_layer, 'submodules'
):
- if hasattr(self.submodules.mtp_model_layer.submodules, 'attention_layer'):
- self_attention_spec = self.submodules.mtp_model_layer.submodules.attention_layer
- if self_attention_spec.submodules.self_attention is not None:
- self_attention_spec = self_attention_spec.submodules.self_attention
- attn_mask_type = self_attention_spec.params.get('attn_mask_type', '')
- assert attn_mask_type in SUPPORTED_ATTN_MASK, (
- f"Multi-Token Prediction (MTP) is not yet supported with "
- f"{attn_mask_type} attention mask type. "
- f"The supported attention mask types are {SUPPORTED_ATTN_MASK}."
- )
- elif hasattr(self.submodules.mtp_model_layer.submodules, 'self_attention'):
- self_attention_spec = self.submodules.mtp_model_layer.submodules.self_attention
- if self_attention_spec is not None:
- attn_mask_type = self_attention_spec.params.get('attn_mask_type', '')
- assert attn_mask_type in SUPPORTED_ATTN_MASK, (
- f"Multi-Token Prediction (MTP) is not yet supported with "
- f"{attn_mask_type} attention mask type. "
- f"The supported attention mask types are {SUPPORTED_ATTN_MASK}."
- )
+ from megatron.core.ssm.mamba_block import MambaStackSubmodules
+ from megatron.core.transformer.transformer_layer import TransformerLayerSubmodules
+
+ layer_submodules = None
+ if isinstance(self.submodules.mtp_model_layer.submodules, MambaStackSubmodules):
+ attention_layer_spec = self.submodules.mtp_model_layer.submodules.attention_layer
+ if hasattr(attention_layer_spec, 'submodules'):
+ assert isinstance(attention_layer_spec.submodules, TransformerLayerSubmodules)
+ layer_submodules = attention_layer_spec.submodules
+ elif isinstance(self.submodules.mtp_model_layer.submodules, TransformerLayerSubmodules):
+ layer_submodules = self.submodules.mtp_model_layer.submodules
+ else:
+ raise ValueError(
+ "Unsupported mtp_model_layer submodules type for attention mask validation."
+ )
+ if layer_submodules:
+ self_attention_spec = layer_submodules.self_attention
+ attn_mask_type = self_attention_spec.params.get('attn_mask_type', '')
+ assert attn_mask_type in SUPPORTED_ATTN_MASK, (
+ f"Multi-Token Prediction (MTP) is not yet supported with "
+ f"{attn_mask_type} attention mask type. "
+ f"The supported attention mask types are {SUPPORTED_ATTN_MASK}."
+ )
- self.enorm = build_module(
- self.submodules.enorm,
+ self.enorm = self.submodules.enorm(
config=self.config,
hidden_size=self.config.hidden_size,
eps=self.config.layernorm_epsilon,
)
- self.hnorm = build_module(
- self.submodules.hnorm,
+ self.hnorm = self.submodules.hnorm(
config=self.config,
hidden_size=self.config.hidden_size,
eps=self.config.layernorm_epsilon,
@@ -752,6 +805,7 @@ def __init__(
skip_bias_add=False,
is_expert=False,
tp_comm_buffer_name="mtp_eh_proj",
+ tp_group=pg_collection.tp if pg_collection is not None else None,
)
# Build inner layers: two possible paths
@@ -759,11 +813,13 @@ def __init__(
# 2. GPT path: single TransformerLayer
if mtp_layer_pattern is not None and mamba_submodules is not None:
from megatron.core.ssm.mamba_block import MambaStack
+ from megatron.core.ssm.mamba_hybrid_layer_allocation import validate_segment_layers
self.mtp_model_layer = MambaStack(
config=self.config,
submodules=mamba_submodules,
- hybrid_override_pattern=mtp_layer_pattern,
+ layer_type_list=validate_segment_layers(mtp_layer_pattern),
+ pp_layer_offset=0,
pre_process=True, # Always receives input from eh_proj
post_layer_norm=False, # MTP has its own final_layernorm
post_process=True, # MTP layer is self-contained
@@ -781,10 +837,10 @@ def __init__(
vp_stage=self.vp_stage,
layer_number=self.layer_number,
is_mtp_layer=True,
+ pg_collection=pg_collection,
)
- self.final_layernorm = build_module(
- self.submodules.layer_norm,
+ self.final_layernorm = self.submodules.layer_norm(
config=self.config,
hidden_size=self.config.hidden_size,
eps=self.config.layernorm_epsilon,
@@ -840,9 +896,9 @@ def _concat_embeddings(self, hidden_states: torch.Tensor, decoder_input: torch.T
"""
Concatenate the tokens before sending to transformer layer.
"""
- decoder_input = self.enorm(decoder_input)
+ decoder_input = apply_module(self.enorm)(decoder_input)
decoder_input = make_viewless_tensor(inp=decoder_input, requires_grad=True, keep_graph=True)
- hidden_states = self.hnorm(hidden_states)
+ hidden_states = apply_module(self.hnorm)(hidden_states)
hidden_states = make_viewless_tensor(inp=hidden_states, requires_grad=True, keep_graph=True)
# At the (k - 1)-th MTP module, concatenates the i-th token's hidden_states
# and the (i + K)-th token's embedding, and combine them with linear projection.
@@ -853,10 +909,10 @@ def _concat_embeddings(self, hidden_states: torch.Tensor, decoder_input: torch.T
# `all_gather_last_dim_from_tensor_parallel_region`, but that utility reduces
# the gradient in backward pass and was therefore incorrect in this context.
# It has been replaced with the correct `gather_from_tensor_model_parallel_region`.
- hidden_states = gather_from_tensor_model_parallel_region(hidden_states)
+ hidden_states = gather_from_tensor_model_parallel_region(hidden_states, group=self.tp_group)
# For sequence parallel, scatter after linear_fc and before transformer layer.
if self.sequence_parallel:
- hidden_states = scatter_to_sequence_parallel_region(hidden_states)
+ hidden_states = scatter_to_sequence_parallel_region(hidden_states, group=self.tp_group)
return hidden_states
def _proj_and_transformer_layer(
@@ -934,7 +990,7 @@ def _postprocess(self, hidden_states: torch.Tensor):
"""
# Layer norm before shared head layer.
- hidden_states = self.final_layernorm(hidden_states)
+ hidden_states = apply_module(self.final_layernorm)(hidden_states)
# TENorm produces a "viewed" tensor. This will result in schedule.py's
# deallocate_output_tensor() throwing an error, so a viewless tensor is
# created to prevent this.
@@ -942,6 +998,53 @@ def _postprocess(self, hidden_states: torch.Tensor):
return hidden_states
+ def forward_single_position(
+ self,
+ hidden_states: Tensor,
+ next_token_ids: Tensor,
+ position_ids: Tensor,
+ embedding: Callable,
+ attention_mask: Optional[Tensor] = None,
+ rotary_pos_emb: Optional[Tensor] = None,
+ rotary_pos_cos: Optional[Tensor] = None,
+ rotary_pos_sin: Optional[Tensor] = None,
+ inference_params=None,
+ packed_seq_params: Optional[PackedSeqParams] = None,
+ sequence_len_offset: Optional[Tensor] = None,
+ ) -> Tensor:
+ """Forward for single positions without roll_tensor (speculative decoding).
+
+ Unlike the regular forward which rolls input_ids to get the next token's
+ embedding, this method directly takes the correct next_token_ids. This is
+ used in speculative decoding where the correct next token is known after
+ verification.
+
+ Args:
+ hidden_states (Tensor): Hidden states at positions of interest [N, B, H].
+ next_token_ids (Tensor): The correct next token IDs [B, N].
+ position_ids (Tensor): Position IDs for the next tokens [B, N].
+ embedding (Callable): The embedding module.
+
+ Returns:
+ Tensor: MTP hidden states [N, B, H].
+ """
+ decoder_input = embedding(input_ids=next_token_ids, position_ids=position_ids)
+ hidden_states = make_viewless_tensor(
+ inp=hidden_states, requires_grad=False, keep_graph=False
+ )
+ hidden_states = self._proj_and_transformer_layer(
+ hidden_states=hidden_states,
+ decoder_input=decoder_input,
+ attention_mask=attention_mask,
+ rotary_pos_emb=rotary_pos_emb,
+ rotary_pos_cos=rotary_pos_cos,
+ rotary_pos_sin=rotary_pos_sin,
+ inference_params=inference_params,
+ packed_seq_params=packed_seq_params,
+ sequence_len_offset=sequence_len_offset,
+ )
+ return hidden_states
+
def _checkpointed_forward(self, forward_func, *args, **kwargs):
def checkpoint_handler():
"""Determines whether to use the `te_checkpoint` or `tensor_parallel.checkpoint`"""
@@ -1028,15 +1131,6 @@ def forward(
packed_seq_params=packed_seq_params,
)
- # Roll RoPE to match rolled positions (position_ids were rolled in _get_embeddings)
- # After rolling, index i should use RoPE for position i+1
- if rotary_pos_emb is not None:
- rotary_pos_emb = torch.roll(rotary_pos_emb, shifts=-1, dims=0)
- if rotary_pos_cos is not None:
- rotary_pos_cos = torch.roll(rotary_pos_cos, shifts=-1, dims=0)
- if rotary_pos_sin is not None:
- rotary_pos_sin = torch.roll(rotary_pos_sin, shifts=-1, dims=0)
-
if self.config.recompute_granularity == 'full' and self.training:
hidden_states = self._checkpointed_forward(
self._proj_and_transformer_layer,
@@ -1087,6 +1181,16 @@ def sharded_state_dict(
token prediction layer.
"""
sharded_state_dict = super().sharded_state_dict(prefix, sharded_offsets, metadata)
+
+ # Backward compatibility: GPT MTP checkpoints were saved with the submodule
+ # named 'transformer_layer'. Remap checkpoint keys so old checkpoints load
+ # correctly. Mamba MTP models keep 'mtp_model_layer' as their native format
+ # since no older checkpoints exist for them.
+ if self.mtp_layer_pattern is None:
+ apply_prefix_mapping(
+ sharded_state_dict, {f'{prefix}mtp_model_layer.': f'{prefix}transformer_layer.'}
+ )
+
return sharded_state_dict
@@ -1155,8 +1259,8 @@ class MultiTokenPredictionBlock(MegatronModule):
When `mtp_use_repeated_layer=True` in config, instead of creating N separate MTP layers,
only 1 layer is created and applied mtp_num_layers times.
- for more information, please refer to DeepSeek-V3 Technical Report
- https://github.com/deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdf
+ For more information, please refer to DeepSeek-V3 Technical Report
+ https://arxiv.org/pdf/2412.19437.pdf
"""
def __init__(
@@ -1191,7 +1295,7 @@ def __init__(
# to the roll_tensor function for proper boundary communication
if pg_collection is None:
# Use default MPU process groups if not provided
- pg_collection = ProcessGroupCollection.use_mpu_process_groups(required_pgs=['cp'])
+ pg_collection = ProcessGroupCollection.use_mpu_process_groups(required_pgs=['cp', 'tp'])
else:
# Ensure the provided process groups include CP
assert hasattr(
@@ -1359,7 +1463,7 @@ def sharded_state_dict(
ShardedStateDict: A dictionary containing the sharded state of the multi
token prediction module.
"""
- sharded_state_dict = super().sharded_state_dict(prefix, sharded_offsets, metadata)
+ sharded_state_dict = {}
layer_prefix = f'{prefix}layers.'
for layer in self.layers:
offset = get_mtp_layer_offset(self.config, self.vp_stage)
diff --git a/megatron/core/transformer/spec_utils.py b/megatron/core/transformer/spec_utils.py
index 09058084181..ba9c22f01b6 100644
--- a/megatron/core/transformer/spec_utils.py
+++ b/megatron/core/transformer/spec_utils.py
@@ -1,7 +1,8 @@
# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
-
+import functools
import logging
import types
+from collections.abc import Callable
from dataclasses import dataclass, field
from typing import Any, Tuple, Union
@@ -126,3 +127,16 @@ def build_module(spec_or_module: Union[ModuleSpec, type], *args, **kwargs):
raise type(e)(f"{str(e)} when instantiating {module.__name__}").with_traceback(
sys.exc_info()[2]
)
+
+
+def get_submodules(spec: Callable[..., Any]) -> object:
+ """Gets the `submodules` field from the provided spec.
+
+ Supports `partial` objects, as well as ModuleSpec or any other object with a `submodules` attr.
+ Raises a ValueError if the provided spec is not supported.
+ """
+ if isinstance(spec, functools.partial):
+ return spec.keywords['submodules']
+ if hasattr(spec, "submodules"):
+ return spec.submodules # type: ignore
+ raise ValueError(f"Could not find `submodules` in the provided spec: {spec!r}")
diff --git a/megatron/core/transformer/torch_norm.py b/megatron/core/transformer/torch_norm.py
index d0ceca7af41..5948ae600f9 100644
--- a/megatron/core/transformer/torch_norm.py
+++ b/megatron/core/transformer/torch_norm.py
@@ -1,4 +1,6 @@
# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
+from typing import Protocol
+
import torch
from megatron.core.jit import jit_fuser
@@ -6,6 +8,22 @@
from megatron.core.utils import is_torch_min_version
+class LayerNormInterface(Protocol):
+ """Interface that all LayerNorm implementations should follow."""
+
+ def forward(self, x: torch.Tensor, /) -> torch.Tensor:
+ """Forward method for a LayerNorm implementation."""
+ ...
+
+
+class LayerNormBuilder(Protocol):
+ """A protocol showing how Modules are expected to construct LayerNorms."""
+
+ def __call__(
+ self, *, config: TransformerConfig, hidden_size: int, eps: float
+ ) -> LayerNormInterface: ...
+
+
class WrappedTorchNorm:
"""
A conditional wrapper to initialize an instance of PyTorch's
@@ -22,7 +40,7 @@ def __new__(
persist_layer_norm: bool = False,
zero_centered_gamma: bool = False,
normalization: str = "LayerNorm",
- ):
+ ) -> LayerNormInterface:
assert (
not config.layernorm_zero_centered_gamma
), f"zero_centered_gamma not supported by torch LayerNorm"
@@ -51,7 +69,7 @@ def __new__(
return norm_cls(normalized_shape=hidden_size, eps=eps)
-class L2Norm(torch.nn.Module):
+class L2Norm(torch.nn.Module, LayerNormInterface):
"""
Applies L2 normalization to the input tensor along the last dimension.
@@ -70,7 +88,7 @@ def __init__(self, hidden_size: int, eps: float = 1e-6, **kwargs):
self.eps = eps
@jit_fuser
- def _norm(self, x):
+ def _norm(self, x: torch.Tensor) -> torch.Tensor:
"""
Performs the actual L2 normalization.
@@ -83,7 +101,7 @@ def _norm(self, x):
x_float = x.float()
return (x_float * torch.rsqrt(x_float.pow(2).mean(-1, keepdim=True) + self.eps)).type_as(x)
- def forward(self, x):
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
"""
Forward pass of the L2Norm module.
diff --git a/megatron/core/transformer/transformer_block.py b/megatron/core/transformer/transformer_block.py
index 2f4d855fbfd..edf13b45dc4 100755
--- a/megatron/core/transformer/transformer_block.py
+++ b/megatron/core/transformer/transformer_block.py
@@ -3,7 +3,7 @@
import logging
from contextlib import nullcontext
from dataclasses import dataclass
-from typing import List, Optional, Union
+from typing import List, Optional, Set, Union, cast
import torch
from torch import Tensor
@@ -12,6 +12,7 @@
from megatron.core.dist_checkpointing.mapping import ShardedStateDict
from megatron.core.dist_checkpointing.utils import replace_prefix_for_sharding
from megatron.core.enums import Fp8Recipe
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.fp4_utils import get_fp4_context
from megatron.core.fp8_utils import get_fp8_context
from megatron.core.fusions.fused_layer_norm import FusedLayerNorm
@@ -22,12 +23,14 @@
from megatron.core.transformer.enums import CudaGraphScope, LayerType
from megatron.core.transformer.module import GraphableMegatronModule, MegatronModule
from megatron.core.transformer.spec_utils import ModuleSpec, build_module
+from megatron.core.transformer.torch_norm import LayerNormBuilder
from megatron.core.transformer.transformer_config import TransformerConfig
from megatron.core.transformer.transformer_layer import (
BaseTransformerLayer,
get_transformer_layer_offset,
)
from megatron.core.transformer.utils import sharded_state_dict_default
+from megatron.core.typed_torch import apply_module, not_none
from megatron.core.utils import (
WrappedTensor,
deprecate_inference_params,
@@ -35,13 +38,6 @@
make_viewless_tensor,
)
-try:
- import transformer_engine.pytorch as te # pylint: disable=unused-import
-
- HAVE_TE = True
-except ImportError:
- HAVE_TE = False
-
try:
import apex # pylint: disable=unused-import
@@ -155,7 +151,7 @@ def get_num_layers_to_build(
assert (
num_layers % config.pipeline_model_parallel_size == 0
- ), "num_layers should be divisible by pipeline_model_parallel_size"
+ ), f"{num_layers=} should be divisible by {config.pipeline_model_parallel_size=}"
num_layers_per_pipeline_rank = num_layers // config.pipeline_model_parallel_size
vp_size = config.virtual_pipeline_model_parallel_size
@@ -220,7 +216,7 @@ class TransformerBlockSubmodules:
"""
layer_specs: Optional[List[ModuleSpec]] = None
- layer_norm: Optional[Union[ModuleSpec, torch.nn.Module]] = None
+ layer_norm: LayerNormBuilder | None = None
def _get_block_submodules(
@@ -310,6 +306,7 @@ def __init__(
self.config.cpu_offloading_activations,
self.config.cpu_offloading_weights,
self.config.cpu_offloading_double_buffering,
+ self.config.cpu_offloading_retain_pinned_cpu_buffers,
)
)
self.config._cpu_offloading_context = (
@@ -376,8 +373,7 @@ def build_layer(layer_spec, layer_number):
# In pipeline parallelism, we want to add this LN only to the last stage of the pipeline
# self.post_process and self.post_layer_norm guide this behavior
if self.has_final_layernorm_in_this_stage():
- self.final_layernorm = build_module(
- self.submodules.layer_norm,
+ self.final_layernorm = not_none(self.submodules.layer_norm)(
config=self.config,
hidden_size=self.config.hidden_size,
eps=self.config.layernorm_epsilon,
@@ -455,8 +451,26 @@ def _checkpointed_forward(
packed_seq_params: PackedSeqParams,
use_inner_quantization_context: bool,
padding_mask: Optional[Tensor] = None,
+ extract_layer_indices: Optional[Set[int]] = None,
+ layer_offset: int = 0,
):
- """Forward method with activation checkpointing."""
+ """Forward method with activation checkpointing.
+
+ Args:
+ extract_layer_indices (Set[int], optional): Global layer
+ indices (across all pipeline stages) from which to
+ extract features.
+ layer_offset (int): The global layer offset for the current
+ pipeline stage. Used to convert local layer indices to
+ global indices when checking extract_layer_indices.
+
+ Returns:
+ If extract_layer_indices is empty: hidden_states tensor
+ If extract_layer_indices is non-empty: (hidden_states, intermediate_hidden_states) tuple
+ """
+ if extract_layer_indices is None:
+ extract_layer_indices = set()
+ intermediate_hidden_states: List[Tensor] = []
def custom(start: int, end: int):
def custom_forward(
@@ -536,9 +550,19 @@ def checkpoint_handler(forward_func):
# A method to further reduce memory usage reducing checkpoints.
layer_idx = 0
while layer_idx < self.num_layers_per_pipeline_rank:
- hidden_states, context = checkpoint_handler(
- custom(layer_idx, layer_idx + self.config.recompute_num_layers)
+ chunk_end = min(
+ layer_idx + self.config.recompute_num_layers, self.num_layers_per_pipeline_rank
)
+ hidden_states, context = checkpoint_handler(custom(layer_idx, chunk_end))
+
+ # Feature extraction for uniform recompute: collect at end of each chunk
+ # Note: Only the last layer of each chunk can have features collected
+ for idx in range(layer_idx, chunk_end):
+ if (idx + layer_offset) in extract_layer_indices:
+ # For uniform recompute, we can only get features at chunk boundaries
+ # Limitation: for fine-grained extraction, use 'block'
+ if idx == chunk_end - 1:
+ intermediate_hidden_states.append(hidden_states)
layer_idx += self.config.recompute_num_layers
@@ -563,9 +587,17 @@ def checkpoint_handler(forward_func):
hidden_states, context = custom(layer_idx, layer_idx + 1)(
hidden_states, attention_mask, context, context_mask, rotary_pos_emb
)
+
+ # Feature extraction: collect hidden states at specified global layer indices
+ if (layer_idx + layer_offset) in extract_layer_indices:
+ intermediate_hidden_states.append(hidden_states)
else:
raise ValueError("Invalid activation recompute method.")
+ # Return intermediate hidden states if feature extraction was requested
+ if len(extract_layer_indices) > 0:
+ return hidden_states, intermediate_hidden_states
+
return hidden_states
def set_input_tensor(self, input_tensor: Tensor):
@@ -582,14 +614,15 @@ def _should_call_local_cudagraph(self, *args, **kwargs):
"""
Check if we should call the local cudagraph path.
"""
- if not self.training and (
- hasattr(self, 'cudagraph_manager')
+ if (
+ not self.training
+ and hasattr(self, 'cudagraph_manager')
and kwargs['attention_mask'] is None
and (
kwargs.get('inference_context') is not None
or kwargs.get('inference_params') is not None
)
- and CudaGraphScope.full_iteration in self.config.cuda_graph_scope
+ and CudaGraphScope.full_iteration_inference in self.config.cuda_graph_scope
):
if kwargs['inference_context'].is_static_batching():
using_cuda_graph = kwargs['inference_context'].is_decode_only()
@@ -607,16 +640,7 @@ def __call__(self, *args, **kwargs):
if isinstance(kwargs['hidden_states'], WrappedTensor)
else kwargs['hidden_states']
)
- # dynamic_inference_decode_only is not a real argument to forward, it is only used
- # to differentiate the cuda graph used for decode from the one used for non-decode
- # inference.
- dynamic_inference_decode_only = kwargs['inference_context'].is_decode_only()
- # cudagraphmanager returns a singleton tuple, whereas the
- # normal forward returns a tensor, therefore we need
- # to extract the tensor from the tuple
- return super().__call__(
- *args, dynamic_inference_decode_only=dynamic_inference_decode_only, **kwargs
- )[0]
+ return super().__call__(*args, **kwargs)[0]
return super().__call__(*args, **kwargs)
def forward(
@@ -634,6 +658,7 @@ def forward(
packed_seq_params: Optional[PackedSeqParams] = None,
sequence_len_offset: Optional[Tensor] = None,
padding_mask: Optional[Tensor] = None,
+ extract_layer_indices: Optional[Set[int]] = None,
*,
inference_params: Optional[BaseInferenceContext] = None,
dynamic_inference_decode_only: Optional[bool] = None,
@@ -665,13 +690,24 @@ def forward(
optimizations.
packed_seq_params (PackedSeqParams, optional): Parameters for packed sequence
processing.
+ extract_layer_indices (Set[int], optional): A set of global
+ layer indices (0-based across all pipeline stages) from
+ which to extract intermediate hidden states. If
+ non-empty, the forward pass will collect hidden_states
+ after each specified layer.
dynamic_inference_decode_only: Optional[bool]: If true, indicates that the current
inference context is for decode-only. This args is only used to uniquely
identify decode and non-decode cuda graph runners in the cuda graph manager.
Returns:
- Union[Tensor, Tuple[Tensor, Tensor]]: The output hidden states tensor of shape
- [s, b, h], and optionally the updated context tensor if cross-attention is used.
+ Union[Tensor, Tuple[Tensor, List[Tensor]]]:
+ - If extract_layer_indices is None or empty: Returns the output hidden states tensor
+ of shape [s, b, h].
+ - If extract_layer_indices is non-empty: Returns a tuple
+ of (hidden_states, intermediate_hidden_states) where
+ intermediate_hidden_states is a list of tensors
+ corresponding to hidden states after each layer in
+ extract_layer_indices.
"""
inference_context = deprecate_inference_params(inference_context, inference_params)
@@ -679,6 +715,18 @@ def forward(
# this is only used to uniquely identify decode and non-decode cuda graph
# runners in the cuda graph manager
+ # Initialize feature collection (consistent with FastGen's Wan implementation)
+ if extract_layer_indices is None:
+ extract_layer_indices = set()
+ intermediate_hidden_states: List[Tensor] = []
+
+ # Calculate the global layer offset for this pipeline stage
+ # This is needed to convert local layer indices to global indices for feature extraction
+ pp_group = self.pg_collection.pp if hasattr(self.pg_collection, 'pp') else None
+ layer_offset = get_transformer_layer_offset(
+ self.config, self.vp_stage, get_pg_rank(pp_group)
+ )
+
# Delete the obsolete reference to the initial input tensor if necessary
if isinstance(hidden_states, WrappedTensor):
hidden_states = hidden_states.unwrap()
@@ -734,7 +782,7 @@ def forward(
with rng_context, outer_quantization_context:
# Forward pass.
if self.config.recompute_granularity == 'full' and self.training:
- hidden_states = self._checkpointed_forward(
+ checkpointed_result = self._checkpointed_forward(
hidden_states=hidden_states,
attention_mask=attention_mask,
context=context,
@@ -744,7 +792,16 @@ def forward(
packed_seq_params=packed_seq_params,
use_inner_quantization_context=use_inner_quantization_context,
padding_mask=padding_mask,
+ extract_layer_indices=extract_layer_indices,
+ layer_offset=layer_offset,
)
+ # Handle return value from _checkpointed_forward
+ if len(extract_layer_indices) > 0:
+ # (hidden_states, intermediate_hidden_states) tuple
+ hidden_states, intermediate_hidden_states = checkpointed_result
+ else:
+ # No intermediate_hidden_states requested: just hidden_states
+ hidden_states = checkpointed_result
else:
for l_no, layer in enumerate(self.layers):
# Get appropriate inner quantization context
@@ -786,9 +843,13 @@ def forward(
):
hidden_states = self.group_prefetch_offload_commit_async(hidden_states)
+ # Extract intermediate embeddings using global layer index
+ if (l_no + layer_offset) in extract_layer_indices:
+ intermediate_hidden_states.append(hidden_states)
+
# Final layer norm.
if self.final_layernorm is not None:
- hidden_states = self.final_layernorm(hidden_states)
+ hidden_states = apply_module(self.final_layernorm)(cast(Tensor, hidden_states))
# TENorm produces a "viewed" tensor. This will result in schedule.py's
# deallocate_output_tensor() throwing an error, so a viewless tensor is
# created to prevent this.
@@ -801,6 +862,9 @@ def forward(
if not self.pre_process and len(self.layers) == 0 and not self.final_layernorm:
hidden_states = hidden_states.clone()
+ if len(extract_layer_indices) > 0:
+ return hidden_states, intermediate_hidden_states
+
return hidden_states
def sharded_state_dict(
diff --git a/megatron/core/transformer/transformer_config.py b/megatron/core/transformer/transformer_config.py
index 9986bb5142a..3ffc27ec5db 100644
--- a/megatron/core/transformer/transformer_config.py
+++ b/megatron/core/transformer/transformer_config.py
@@ -1,6 +1,8 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# Copyright (c) 2024, Advanced Micro Devices, Inc. All rights reserved.
+import logging
+import math
import warnings
from dataclasses import dataclass, field
from typing import Callable, List, Literal, Optional, Tuple, Union
@@ -13,6 +15,7 @@
from megatron.core.transformer.enums import AttnBackend, CudaGraphScope
from megatron.core.transformer.pipeline_parallel_layer_layout import PipelineParallelLayerLayout
+from .._rank_utils import log_single_rank
from ..fusions.fused_bias_geglu import quick_gelu
from ..model_parallel_config import ModelParallelConfig
from ..utils import (
@@ -20,9 +23,12 @@
init_method_normal,
is_te_min_version,
is_torch_min_version,
+ mup_scaled_init_method_normal,
scaled_init_method_normal,
)
+logger = logging.getLogger(__name__)
+
try:
from packaging.version import Version as PkgVersion
@@ -66,7 +72,7 @@ class TransformerConfig(ModelParallelConfig):
"""Use a single MTP layer repeatedly instead of multiple separate layers."""
mtp_hybrid_override_pattern: Optional[str] = None
- """DEPRECATED: Use unified hybrid_override_pattern instead.
+ """DEPRECATED: Use unified hybrid_layer_pattern instead.
Legacy argument for loading old checkpoints.
Force a specific hybrid layer pattern for MTP layers.
"""
@@ -182,7 +188,7 @@ class TransformerConfig(ModelParallelConfig):
gated_linear_unit: bool = False
"""Use a gated linear unit for the first linear layer in the MLP."""
- activation_func: Callable = F.gelu
+ activation_func: Callable[[torch.Tensor], torch.Tensor] = F.gelu
"""Activation function to use for the non-linearity in the MLP."""
activation_func_fp8_input_store: bool = False
@@ -318,7 +324,8 @@ class TransformerConfig(ModelParallelConfig):
output_layer_init_method: Optional[Callable] = None
"""Method to initialize weights of the output layer of both attention and MLP blocks. If None,
will be set to megatron.core.utils.scaled_init_method_normal(init_method_std) which is torch nn
- init normal with mean=0.0 and std=init_method_std / math.sqrt(2.0 * num_layers)."""
+ init normal with mean=0.0 and std=init_method_std / math.sqrt(2.0 * num_layers).
+ Note: this does not control vocab readout/unembedding initialization."""
init_method_std: float = 0.02
"""Standard deviation of the zero mean normal for the default initialization method, not used if
@@ -345,6 +352,59 @@ class TransformerConfig(ModelParallelConfig):
training of very large models. This feature is only works when megatron fsdp is turned on.
"""
+ ####################
+ # MuP (Maximal Update Parameterization)
+ ####################
+ use_mup: bool = False
+ """
+ Enable Maximal Update Parameterization (MuP) for hyperparameter transfer across
+ model widths. When enabled, learning rates and initialization are scaled according
+ to the width multiplier to ensure consistent training dynamics.
+ """
+
+ mup_width_mult: float = 1.0
+ """
+ Width multiplier for MuP scaling, computed as hidden_size / mup_base_hidden_size.
+ This value is automatically computed in __post_init__ when use_mup is enabled.
+ """
+
+ mup_base_hidden_size: Optional[int] = None
+ """
+ Base hidden size for MuP width scaling. This is the reference width from which
+ scaling factors are computed. Defaults to hidden_size if not specified (base model
+ case where width_mult=1.0). Set this to your base/proxy model's hidden size when
+ scaling up.
+ """
+
+ mup_embedding_mult: float = 1.0
+ """
+ Multiplier for embedding layer output. Applied after the embedding lookup.
+ Default: 1.0 (no scaling).
+ """
+
+ mup_output_mult: float = 1.0
+ """
+ Multiplier for output logits before softmax. When MuP is enabled and this is left
+ at 1.0, it is auto-set to 1/mup_width_mult to keep output variance stable across
+ widths. Override to customize output scaling.
+ Default: 1.0.
+ """
+
+ mup_base_head_dim: Optional[float] = None
+ """
+ Base head dimension for MuP attention scaling. When set,
+ softmax_scale = sqrt(mup_base_head_dim) / (kv_channels ** mup_attn_scale_power).
+ Set to base model's d_head (e.g., 64) to match standard 1/sqrt(d_head) scaling
+ at the base width, ensuring non-MuP compatibility for that specific value.
+ """
+
+ mup_attn_scale_power: float = 1.0
+ """
+ Power for attention scaling: softmax_scale = 1 / (kv_channels ** mup_attn_scale_power).
+ 0.5 = standard attention (1/sqrt(d_head)), 1.0 = MuP attention (1/d_head).
+ Default: 1.0 (MuP scaling when use_mup is True). Set to 0.5 for standard scaling.
+ """
+
####################
# mixed-precision
####################
@@ -389,6 +449,9 @@ class TransformerConfig(ModelParallelConfig):
fused_single_qkv_rope: bool = False
"""If set, avoid splitting QKV before ROPE forward and avoid concatenating ROPE dgrads."""
+ fused_residual_rmsnorm: bool = False
+ """If True, fuses residual connection and RMSNorm backward pass when TE is used."""
+
####################
# activation recomputation
####################
@@ -657,16 +720,19 @@ class TransformerConfig(ModelParallelConfig):
"""[Experimental] Force load balancing with random logits for MoE router, supports naive topk
and group-limited topk. This is an experimental feature and only for benchmark."""
+ moe_router_force_biased: Optional[float] = None
+ """Apply random expert bias in normal distribution with specified std
+ to router logits. Shared seed across all ranks ensures identical bias.
+ If positive, generates new random bias each forward pass.
+ If negative, generates bias once per layer and reuses it (abs value is std).
+ This is an experimental feature for benchmarking purposes."""
+
moe_grouped_gemm: bool = False
"""When there are multiple experts per rank, compress multiple local (potentially small) gemms
in a single kernel launch to improve the utilization and performance by leveraging the Grouped
GEMM feature introduced since CUTLASS 2.8 (https://github.com/fanshiqing/grouped_gemm).
"""
- moe_use_legacy_grouped_gemm: bool = False
- """Use legacy GroupedMLP rather than TEGroupedMLP.
- Note: The legacy one will be deprecated soon."""
-
moe_aux_loss_coeff: Union[float, List[float]] = 0.0
"""Scaling coefficient for the aux loss. A starting value of 1e-2 is recommended.
If a list of load balancing types is provided for `moe_router_load_balancing_type`,
@@ -707,6 +773,12 @@ class TransformerConfig(ModelParallelConfig):
the expert capacity length, effective only after the moe_expert_capacity_factor is set. The
default setting is False."""
+ moe_pad_experts_for_cuda_graph_inference: bool = False
+ """moe_pad_experts_for_cuda_graph_inference (bool): If True, the router will switch to dropping
+ and padding during decode time which does not have a D2H sync. The capacity factor is set to the
+ max that an expert could see during inference so no tokens are actually dropped. The default
+ setting is False."""
+
moe_token_drop_policy: Literal['probs', 'position'] = "probs"
"""The policy to drop tokens. Can be either "probs" or "position". If "probs", the tokens with
the lowest probabilities will be dropped. If "position", tokens at the end of each batch will
@@ -847,12 +919,36 @@ class TransformerConfig(ModelParallelConfig):
which is no use of symmetric memory.
"""
+ nccl_all_reduce_for_prefill: bool = False
+ """If True, use NCCL all-reduce kernels when symmetric all-reduce is enabled."""
+
use_inference_optimized_layers: bool = False
"""If True, use inference optimized transformer layers during inference."""
inference_fuse_tp_communication: bool = False
""" If true, uses a fused reduce-scatter-residual-norm-allgather kernel during inference. """
+ inference_disable_triton_nvls_kernels: bool = False
+ """ If true, disables the use of Triton NVLS kernels during inference. """
+
+ inference_grouped_gemm_backend: Literal['auto', 'torch', 'te'] = "auto"
+ """Specifies the backend to use for grouped GEMM operations during inference.
+ Options:
+ - 'auto': Uses FlashInfer for CUDA-graphed iterations (requires flashinfer-python),
+ and torch.nn.functional.grouped_mm for non-CUDA-graphed iterations (falls back to TE
+ if unavailable). Note: the heuristic for choosing backends in 'auto' mode may change
+ in future releases.
+ - 'torch': Uses torch.nn.functional.grouped_mm. For CUDA-graphed iterations, uses
+ mcore_fused_moe (permute/unpermute + grouped_mm with Triton kernels).
+ - 'te': Uses TE GroupedGEMM only. Not supported with CUDA graphs.
+ """
+
+ inference_moe_disable_fused_quant_kernels: bool = False
+ """When False (default), use fused kernels that combine permute/activation with
+ MXFP8 quantization + swizzle into a single kernel launch. Only applies when
+ fp8_recipe='mxfp8'. Set to True to disable fusion and use separate kernel
+ launches (useful for debugging)."""
+
mrope_section: Optional[List[int]] = None
""" Multimodal rope section is for channel dimension of temporal, height and width
in rope calculation. """
@@ -929,6 +1025,20 @@ def __post_init__(self):
details.
"""
super().__post_init__()
+
+ # When fp32 residual connections are enabled, pipeline parallel communication must
+ # use fp32 to match the dtype of the residual stream between pipeline stages.
+ if self.fp32_residual_connection and self.pipeline_dtype is not None:
+ if self.pipeline_dtype != torch.float:
+ log_single_rank(
+ logger,
+ logging.WARNING,
+ f"fp32_residual_connection is enabled, overriding pipeline_dtype "
+ f"from {self.pipeline_dtype} to torch.float to match the "
+ f"residual stream dtype between pipeline stages.",
+ )
+ self.pipeline_dtype = torch.float
+
if self.fp16 and self.bf16:
raise ValueError(
f"Only one of self.fp16: {self.fp16} and self.bf16 {self.bf16} should be True."
@@ -1065,6 +1175,54 @@ def __post_init__(self):
if self.expert_model_parallel_size > 1 and self.num_moe_experts is None:
raise ValueError("num_moe_experts must be non None to use expert-parallel.")
+ if self.transformer_impl == "inference_optimized" and self.num_moe_experts is not None:
+ if self.expert_tensor_parallel_size > 1:
+ raise ValueError(
+ "Inference-optimized MoE layers does not support expert tensor parallelism."
+ )
+ if self.moe_expert_capacity_factor is not None:
+ raise ValueError("Inference-optimized MoE layers only support dropless MoE ")
+ if self.moe_router_padding_for_quantization:
+ raise ValueError(
+ "Inference-optimized MoE layers do not support padded "
+ "routing map for quantization."
+ )
+ if self.moe_router_dtype != "fp32":
+ raise ValueError(
+ "--transformer-impl='inference_optimized' requires --moe-router-dtype=fp32 "
+ "to avoid costly dtype conversions during decode."
+ )
+
+ if self.gated_linear_unit and self.cuda_graph_impl == "local":
+ raise ValueError(
+ "--transformer-impl='inference_optimized' does not yet support CUDA graphs "
+ "with gated linear units (SwiGLU/GeGLU) due to differences in weight "
+ "layouts between the FlashInfer kernel and mcore. Either disable CUDA "
+ "graphs (--cuda-graph-impl=none) or use a non-gated activation "
+ "(e.g. squared_relu)."
+ )
+
+ if self.fp8 == "mxfp8":
+ if not self.fp8_param:
+ raise ValueError(
+ "fp8_param must be enabled when using "
+ "--transformer-impl='inference_optimized' with --fp8-recipe='mxfp8'. "
+ "Please set --fp8-param-gather."
+ )
+
+ assert self.inference_grouped_gemm_backend in ('auto', 'torch', 'te'), (
+ f"inference_grouped_gemm_backend must be 'auto', 'torch', or 'te', "
+ f"got '{self.inference_grouped_gemm_backend}'"
+ )
+
+ if self.cuda_graph_impl == "local":
+ if self.inference_grouped_gemm_backend == "te":
+ raise ValueError(
+ "TE GroupedGEMM is not supported with CUDA graphs. Please set "
+ "inference_grouped_gemm_backend to 'auto' or 'torch', or disable "
+ "CUDA graphs (--cuda-graph-impl=none)."
+ )
+
if self.num_moe_experts is not None and self.num_moe_experts <= 0:
raise ValueError("num_moe_experts must be non-negative.")
@@ -1532,6 +1690,12 @@ def __post_init__(self):
"to True and use_te_activation_func to False."
)
+ if self.fused_residual_rmsnorm:
+ if self.normalization != "RMSNorm":
+ raise ValueError(
+ "fused_residual_rmsnorm is only supported when normalization is RMSNorm."
+ )
+
if self.use_te_activation_func:
if self.activation_func not in (F.gelu, F.silu, F.relu):
raise ValueError(
@@ -1575,7 +1739,48 @@ def __post_init__(self):
if self.multi_latent_attention and self.rotary_interleaved:
raise ValueError("rotary_interleaved does not work with multi_latent_attention.")
- # Set the embedding init method
+ # MuP (Maximal Update Parameterization) configuration
+ if self.use_mup:
+ # Default base_hidden_size to hidden_size (base model case, width_mult=1.0)
+ if self.mup_base_hidden_size is None:
+ self.mup_base_hidden_size = self.hidden_size
+ assert self.mup_base_hidden_size > 0, "--mup-base-hidden-size must be positive."
+ # Compute width multiplier
+ self.mup_width_mult = self.hidden_size / self.mup_base_hidden_size
+
+ # MuP attention scaling: 1/d_head instead of 1/sqrt(d_head).
+ if self.softmax_scale is None:
+ base_head_scale = (
+ 1.0 if self.mup_base_head_dim is None else self.mup_base_head_dim**0.5
+ )
+ self.softmax_scale = base_head_scale / (self.kv_channels**self.mup_attn_scale_power)
+
+ # MuP output scaling: scale logits by 1/width_mult to keep outputs O(1).
+ # Only auto-set if user hasn't explicitly configured it.
+ if self.mup_output_mult == 1.0 and self.mup_width_mult != 1.0:
+ self.mup_output_mult = 1.0 / self.mup_width_mult
+
+ overridden_init_methods = []
+ if self.init_method is not None:
+ overridden_init_methods.append("init_method")
+ if self.output_layer_init_method is not None:
+ overridden_init_methods.append("output_layer_init_method")
+ if overridden_init_methods:
+ overridden_init_methods_text = " and ".join(overridden_init_methods)
+ verb = "is" if len(overridden_init_methods) == 1 else "are"
+ warnings.warn(
+ "use_mup is enabled, but custom "
+ + overridden_init_methods_text
+ + f" {verb} set. This may break MuP initialization assumptions.",
+ UserWarning,
+ )
+
+ # Set the embedding init method.
+ # NOTE: This block must run AFTER the MuP block above but BEFORE the init_method
+ # block below. When MuP is enabled and init_method is None (the common case),
+ # embedding_init_method is set here using the unscaled init_method_std, while
+ # init_method (set below) gets MuP width-scaling. This ordering ensures embeddings
+ # use the base (unscaled) initialization as required by MuP.
if self.embedding_init_method_std is None:
# By default, use the same init std as you use for every other non-output layer.
self.embedding_init_method_std = self.init_method_std
@@ -1593,14 +1798,29 @@ def __post_init__(self):
self.embedding_init_method = self.init_method
if self.init_method is None:
- self.init_method = init_method_normal(self.init_method_std)
+ if self.use_mup:
+ # MuP: scale std by 1/sqrt(width_mult).
+ self.init_method = init_method_normal(
+ self.init_method_std / math.sqrt(self.mup_width_mult)
+ )
+ else:
+ self.init_method = init_method_normal(self.init_method_std)
if self.output_layer_init_method is None:
- self.output_layer_init_method = scaled_init_method_normal(
- self.init_method_std,
- self.num_layers,
- multiplier=2.0 if not self.is_hybrid_model else 1.0,
- )
+ if self.use_mup:
+ # MuP: depth and width scaling for output layers.
+ self.output_layer_init_method = mup_scaled_init_method_normal(
+ self.init_method_std,
+ self.num_layers,
+ self.mup_width_mult,
+ multiplier=2.0 if not self.is_hybrid_model else 1.0,
+ )
+ else:
+ self.output_layer_init_method = scaled_init_method_normal(
+ self.init_method_std,
+ self.num_layers,
+ multiplier=2.0 if not self.is_hybrid_model else 1.0,
+ )
if self.num_moe_experts is not None and self.add_bias_linear:
assert (
@@ -1894,6 +2114,9 @@ def __post_init__(self):
assert (
self.recompute_num_layers is None
), 'recompute_num_layers must be None when enabling overlap_moe_expert_parallel_comm'
+ assert (
+ "moe" not in self.recompute_modules
+ ), 'disable moe in recompute_modules when enabling overlap_moe_expert_parallel_comm'
# Check if bf16 or fp16 is used
assert (
@@ -1927,9 +2150,6 @@ def __post_init__(self):
assert (
self.overlap_moe_expert_parallel_comm
), 'overlap_moe_expert_parallel_comm must be enabled when enabling delay_wgrad_compute'
- assert (
- not self.moe_use_legacy_grouped_gemm
- ), 'delay_wgrad_compute is not supported with legacy groupedgemm implementation'
if self.cuda_graph_impl == "transformer_engine":
assert is_te_min_version("2.10.0"), (
'TE version >= 2.10.0 is required for delay_wgrad_compute with '
@@ -2012,6 +2232,15 @@ def __post_init__(self):
"inference_fuse_tp_communication is only supported "
"for inference_optimized transformer implementation."
)
+ assert (
+ self.num_moe_experts is None
+ ), "--inference-fuse-tp-communication is not supported for MoE models."
+
+ if self.inference_disable_triton_nvls_kernels:
+ assert self.transformer_impl == "inference_optimized", (
+ "inference_disable_triton_nvls_kernels is only supported "
+ "for inference_optimized transformer implementation."
+ )
if self.batch_invariant_mode:
assert (
@@ -2083,6 +2312,11 @@ class MLATransformerConfig(TransformerConfig):
This is only for the dynamic inference backend and requires that
Flash MLA is installed."""
+ mla_down_proj_fusion: bool = False
+ """Enable fused q/kv down-projection and fused input layernorm when backend supports.
+ Otherwise fall back to the unfused MLA.
+ """
+
def __post_init__(self):
super().__post_init__()
if self.multi_latent_attention and self.apply_rope_fusion and self.rope_type != "yarn":
diff --git a/megatron/core/transformer/transformer_layer.py b/megatron/core/transformer/transformer_layer.py
index 30c6af59fea..6e7e9172bfb 100644
--- a/megatron/core/transformer/transformer_layer.py
+++ b/megatron/core/transformer/transformer_layer.py
@@ -1,12 +1,14 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# Copyright (c) 2024, Advanced Micro Devices, Inc. All rights reserved.
+from __future__ import annotations
+
import functools
import logging
import warnings
from abc import ABC
from dataclasses import dataclass, field
-from typing import Any, Dict, Optional, Union
+from typing import TYPE_CHECKING, Any, Dict, Optional, Union
import torch
import torch.distributed
@@ -23,7 +25,9 @@
from megatron.core.transformer.mlp import MLP
from megatron.core.transformer.module import GraphableMegatronModule
from megatron.core.transformer.spec_utils import ModuleSpec, build_module
+from megatron.core.transformer.torch_norm import LayerNormBuilder
from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.typed_torch import apply_module, copy_signature
from megatron.core.utils import (
deprecate_inference_params,
get_pg_rank,
@@ -34,6 +38,9 @@
nvtx_range_push,
)
+if TYPE_CHECKING:
+ from megatron.core.inference.contexts import BaseInferenceContext
+
logger = logging.getLogger(__name__)
@@ -204,16 +211,16 @@ class TransformerLayerSubmodules:
of the layer's architecture.
Args:
- input_layernorm (Union[ModuleSpec, type]): Specification for the input layer normalization.
+ input_layernorm: Specification for the input layer normalization.
self_attention (Union[ModuleSpec, type]): Specification for the self-attention mechanism.
self_attn_bda (Union[ModuleSpec, type]): Specification for the bias-dropout-add operation
after self-attention.
- pre_cross_attn_layernorm (Union[ModuleSpec, type]): Specification for the layer
+ pre_cross_attn_layernorm: Specification for the layer
normalization before cross-attention.
cross_attention (Union[ModuleSpec, type]): Specification for the cross-attention mechanism.
cross_attn_bda (Union[ModuleSpec, type]): Specification for the bias-dropout-add operation
after cross-attention.
- pre_mlp_layernorm (Union[ModuleSpec, type]): Specification for the layer normalization
+ pre_mlp_layernorm: Specification for the layer normalization
before the MLP.
mlp (Union[ModuleSpec, type]): Specification for the MLP in Dense layer.
mlp_bda (Union[ModuleSpec, type]): Specification for the bias-dropout-add operation
@@ -222,15 +229,15 @@ class TransformerLayerSubmodules:
in the `sharded_state_dict` method.
"""
- input_layernorm: Union[ModuleSpec, type] = IdentityOp
+ input_layernorm: LayerNormBuilder = IdentityOp
self_attention: Union[ModuleSpec, type] = IdentityOp
self_attn_bda: Union[ModuleSpec, type] = IdentityFuncOp
- pre_cross_attn_layernorm: Union[ModuleSpec, type] = IdentityOp
+ pre_cross_attn_layernorm: LayerNormBuilder = IdentityOp
cross_attention: Union[ModuleSpec, type] = IdentityOp
cross_attn_bda: Union[ModuleSpec, type] = IdentityFuncOp
- pre_mlp_layernorm: Union[ModuleSpec, type] = IdentityOp
+ pre_mlp_layernorm: LayerNormBuilder = IdentityOp
mlp: Union[ModuleSpec, type] = IdentityOp
mlp_dense: Union[ModuleSpec, type] = IdentityOp
mlp_bda: Union[ModuleSpec, type] = IdentityFuncOp
@@ -271,6 +278,8 @@ def __init__(
pg_collection: Optional[ProcessGroupCollection] = None,
vp_stage: Optional[int] = None,
is_mtp_layer: bool = False,
+ add_layer_offset: bool = True,
+ pp_layer_offset: Optional[int] = None,
):
self.submodules_config = submodules
super().__init__(config=config, vp_stage=vp_stage)
@@ -284,7 +293,10 @@ def __init__(
# so they should NOT add the decoder layer offset. The router.py handles MTP layer
# numbering separately by adding config.num_layers to distinguish MTP layers from decoder
# layers in the aux loss tracker.
- if is_mtp_layer:
+ #
+ # When add_layer_offset is False, the caller has already included the correct offset
+ # in layer_number (e.g. when using --hybrid-layer-pattern with fVPP).
+ if is_mtp_layer or not add_layer_offset:
self.layer_number = layer_number
else:
self.layer_number = layer_number + get_transformer_layer_offset(
@@ -295,8 +307,7 @@ def __init__(
# [Module 1: Input Layernorm] Optional Layernorm on the input data
# TODO: add pytorch only layernorm
- self.input_layernorm = build_module(
- submodules.input_layernorm,
+ self.input_layernorm = submodules.input_layernorm(
config=self.config,
hidden_size=self.config.hidden_size,
eps=self.config.layernorm_epsilon,
@@ -305,11 +316,16 @@ def __init__(
attention_optional_kwargs = {}
if config.context_parallel_size > 1 and config.cp_comm_type is not None:
if isinstance(config.cp_comm_type, list):
- attention_optional_kwargs["cp_comm_type"] = config.cp_comm_type[self.layer_number]
+ # layer_number is 1-indexed, so we need to subtract 1 to get the correct index
+ attention_optional_kwargs["cp_comm_type"] = config.cp_comm_type[
+ self.layer_number - 1
+ ]
else:
attention_optional_kwargs["cp_comm_type"] = config.cp_comm_type
attention_optional_kwargs["pg_collection"] = pg_collection
+ if pp_layer_offset is not None:
+ attention_optional_kwargs["pp_layer_offset"] = pp_layer_offset
# [Module 2: SelfAttention]
self.self_attention = build_module(
@@ -323,8 +339,7 @@ def __init__(
self.self_attn_bda = build_module(submodules.self_attn_bda)
# [Module 4: Post SelfAttention] Optional Layernorm after self-attn
- self.pre_cross_attn_layernorm = build_module(
- submodules.pre_cross_attn_layernorm,
+ self.pre_cross_attn_layernorm = submodules.pre_cross_attn_layernorm(
config=self.config,
hidden_size=self.config.hidden_size,
eps=self.config.layernorm_epsilon,
@@ -342,8 +357,7 @@ def __init__(
self.cross_attn_bda = build_module(submodules.cross_attn_bda, config=self.config)
# [Module 7: Pre MLP] Optional Layernorm before MLP
- self.pre_mlp_layernorm = build_module(
- submodules.pre_mlp_layernorm,
+ self.pre_mlp_layernorm = submodules.pre_mlp_layernorm(
config=self.config,
hidden_size=self.config.hidden_size,
eps=self.config.layernorm_epsilon,
@@ -352,7 +366,7 @@ def __init__(
additional_mlp_kwargs = {}
# import here to avoid circular import
from megatron.core.extensions.transformer_engine import TEFusedMLP
- from megatron.core.transformer.moe.experts import GroupedMLP, SequentialMLP, TEGroupedMLP
+ from megatron.core.transformer.moe.experts import SequentialMLP, TEGroupedMLP
from megatron.core.transformer.moe.moe_layer import MoELayer
# MLP expects tp_group but MoELayer expects pg_collection to be passed in.
@@ -360,7 +374,7 @@ def __init__(
# The conditional below is to make the logic explicit
# if submodules.mlp is not a ModuleSpec,we dont have to handle passing additional kwargs
if isinstance(submodules.mlp, ModuleSpec):
- if submodules.mlp.module in (MoELayer, GroupedMLP, TEGroupedMLP, SequentialMLP):
+ if submodules.mlp.module in (MoELayer, TEGroupedMLP, SequentialMLP):
additional_mlp_kwargs["pg_collection"] = pg_collection
# Pass is_mtp_layer flag to MoELayer to distinguish MTP MoE layers.
if submodules.mlp.module == MoELayer:
@@ -394,6 +408,7 @@ def __init__(
self.recompute_pre_mlp_layernorm = False
self.recompute_mlp = False
if self.config.recompute_granularity == 'selective':
+ assert self.config.recompute_modules is not None
if "layernorm" in self.config.recompute_modules:
if not isinstance(self.input_layernorm, IdentityOp):
self.recompute_input_layernorm = True
@@ -513,25 +528,6 @@ def _get_layer_offset(config: TransformerConfig):
)
return get_transformer_layer_offset(config)
- def forward(self, *args, **kwargs):
- """
- Perform a forward pass through the transformer layer.
-
- This method calls the core computation of a transformer layer, including
- self-attention, cross-attention (if applicable), and feed-forward operations.
- """
- # Remove 'dynamic_inference_decode_only' from kwargs if present
- # this is only used to uniquely identify decode and non-decode cuda graph
- # runners in the cuda graph manager
- kwargs.pop("dynamic_inference_decode_only", None)
- hidden_states, context = self._forward_attention(*args, **kwargs)
- output = self._forward_mlp(
- hidden_states,
- kwargs.get("inference_context", None),
- padding_mask=kwargs.get("padding_mask", None),
- )
- return output, context
-
def _forward_attention(
self,
hidden_states: Tensor,
@@ -543,7 +539,7 @@ def _forward_attention(
rotary_pos_sin: Optional[Tensor] = None,
rotary_pos_cos_sin: Optional[Tensor] = None,
attention_bias: Optional[Tensor] = None,
- inference_context: Optional[Any] = None,
+ inference_context: Optional[BaseInferenceContext] = None,
packed_seq_params: Optional[PackedSeqParams] = None,
sequence_len_offset: Optional[Tensor] = None,
padding_mask: Optional[Tensor] = None,
@@ -584,19 +580,30 @@ def _forward_attention(
inference_context = deprecate_inference_params(inference_context, inference_params)
- # Residual connection.
- residual = hidden_states
-
# Optional Input Layer norm
if self.recompute_input_layernorm:
self.input_layernorm_checkpoint = tensor_parallel.CheckpointWithoutOutput()
with off_interface(self.offload_attn_norm, hidden_states, "attn_norm") as hidden_states:
input_layernorm_output = self.input_layernorm_checkpoint.checkpoint(
- self.input_layernorm, hidden_states
+ apply_module(self.input_layernorm), hidden_states
)
else:
with off_interface(self.offload_attn_norm, hidden_states, "attn_norm") as hidden_states:
- input_layernorm_output = self.input_layernorm(hidden_states)
+ input_layernorm_output = apply_module(self.input_layernorm)(hidden_states)
+
+ if isinstance(input_layernorm_output, tuple):
+ if len(input_layernorm_output) != 2:
+ raise ValueError(
+ f"When the output of input_layernorm is a tuple, it is "
+ f"expected to have 2 elements (output, residual), but "
+ f"got {len(input_layernorm_output)}"
+ )
+ input_layernorm_output, residual = input_layernorm_output
+ else:
+ residual = hidden_states
+
+ if self.config.fp32_residual_connection:
+ residual = residual.float()
using_fused_tp_inference_kernel = (not self.training) and (
self.config.inference_fuse_tp_communication
@@ -653,12 +660,23 @@ def _forward_attention(
hidden_states, name="attn_norm", forced_released_tensors=[residual]
)
- # Residual connection.
- residual = hidden_states
-
# Optional Layer norm after self-attention
- pre_cross_attn_layernorm_output = self.pre_cross_attn_layernorm(hidden_states)
+ pre_cross_attn_layernorm_output = apply_module(self.pre_cross_attn_layernorm)(hidden_states)
+
+ if isinstance(pre_cross_attn_layernorm_output, tuple):
+ if len(pre_cross_attn_layernorm_output) != 2:
+ raise ValueError(
+ f"When the output of pre_cross_attn_layernorm_output "
+ f"is a tuple, it is expected to have 2 elements "
+ f"(output, residual), but "
+ f"got {len(pre_cross_attn_layernorm_output)}"
+ )
+ pre_cross_attn_layernorm_output, residual = pre_cross_attn_layernorm_output
+ else:
+ residual = hidden_states
+ if self.config.fp32_residual_connection:
+ residual = residual.float()
# Cross attention.
attention_output_with_bias = self.cross_attention(
pre_cross_attn_layernorm_output,
@@ -679,7 +697,23 @@ def _forward_attention(
return hidden_states, context
- def _forward_pre_mlp_layernorm(self, hidden_states):
+ @copy_signature(_forward_attention)
+ def forward(self, *args, **kwargs):
+ """
+ Perform a forward pass through the transformer layer.
+
+ This method calls the core computation of a transformer layer, including
+ self-attention, cross-attention (if applicable), and feed-forward operations.
+ """
+ hidden_states, context = self._forward_attention(*args, **kwargs)
+ output = self._forward_mlp(
+ hidden_states,
+ kwargs.get("inference_context", None),
+ padding_mask=kwargs.get("padding_mask", None),
+ )
+ return output, context
+
+ def _forward_pre_mlp_layernorm(self, hidden_states: Tensor):
from megatron.core.pipeline_parallel.fine_grained_activation_offload import (
FineGrainedActivationOffloadingInterface as off_interface,
)
@@ -688,15 +722,20 @@ def _forward_pre_mlp_layernorm(self, hidden_states):
self.pre_mlp_norm_checkpoint = tensor_parallel.CheckpointWithoutOutput()
with off_interface(self.offload_mlp_norm, hidden_states, "mlp_norm") as hidden_states:
pre_mlp_layernorm_output = self.pre_mlp_norm_checkpoint.checkpoint(
- self.pre_mlp_layernorm, hidden_states
+ apply_module(self.pre_mlp_layernorm), hidden_states
)
else:
with off_interface(self.offload_mlp_norm, hidden_states, "mlp_norm") as hidden_states:
- pre_mlp_layernorm_output = self.pre_mlp_layernorm(hidden_states)
+ pre_mlp_layernorm_output = apply_module(self.pre_mlp_layernorm)(hidden_states)
return pre_mlp_layernorm_output
- def _forward_mlp(self, hidden_states, inference_context=None, padding_mask=None):
+ def _forward_mlp(
+ self,
+ hidden_states: Tensor,
+ inference_context: BaseInferenceContext | None = None,
+ padding_mask: Tensor | None = None,
+ ) -> Tensor | list[Tensor | None]:
"""
Perform a forward pass through the feed-forward layer.
@@ -712,12 +751,24 @@ def _forward_mlp(self, hidden_states, inference_context=None, padding_mask=None)
output (Tensor): Transformed hidden states of shape [s, b, h].
"""
- # Residual connection.
- residual = hidden_states
-
# Optional Layer norm post the cross-attention.
pre_mlp_layernorm_output = self._forward_pre_mlp_layernorm(hidden_states)
+ if isinstance(pre_mlp_layernorm_output, tuple):
+ if len(pre_mlp_layernorm_output) != 2:
+ raise ValueError(
+ f"When the output of pre_mlp_layernorm is a tuple, it is "
+ f"expected to have 2 elements (output, residual), but "
+ f"got {len(pre_mlp_layernorm_output)}"
+ )
+ pre_mlp_layernorm_output, residual = pre_mlp_layernorm_output
+ else:
+ # Residual connection.
+ residual = hidden_states
+
+ if self.config.fp32_residual_connection:
+ residual = residual.float()
+
nvtx_range_push(suffix="mlp")
# Potentially chunk the MLP computation during prefill to minimize the peak activation size
should_chunk_mlp_for_prefill = (
@@ -791,7 +842,9 @@ def _forward_mlp(self, hidden_states, inference_context=None, padding_mask=None)
else:
return self._forward_post_mlp(mlp_output_with_bias, residual)
- def _forward_post_mlp(self, mlp_output_with_bias, residual):
+ def _forward_post_mlp(
+ self, mlp_output_with_bias: tuple[Tensor, Tensor | None], residual: Tensor
+ ) -> Tensor:
"""
Perform operations after the MLP computation.
@@ -1126,7 +1179,16 @@ def _te_cuda_graph_replay(self, *args, **kwargs):
residual = cuda_graph_output.pop()
if not self.is_moe_layer:
return residual, None, None, None
- hidden_states = self.pre_mlp_layernorm(residual)
+ hidden_states = apply_module(self.pre_mlp_layernorm)(residual)
+ if isinstance(hidden_states, tuple):
+ if len(hidden_states) != 2:
+ raise ValueError(
+ f"When the output of pre_mlp_layernorm is a tuple, it is "
+ f"expected to have 2 elements (output, residual), but "
+ f"got {len(hidden_states)}"
+ )
+ hidden_states, residual = hidden_states
+
shared_expert_output = self.mlp.shared_experts_compute(hidden_states)
probs, routing_map = self.mlp.route(hidden_states)
hidden_states, probs = self.mlp.preprocess(hidden_states, probs, routing_map)
@@ -1195,8 +1257,12 @@ def _should_call_local_cudagraph(self, *args, **kwargs):
"""
Check if we should call the local cudagraph path.
"""
- # Training and validation mode CUDA graphs
- if hasattr(self, 'cudagraph_manager') and kwargs.get('inference_context') is None:
+ # Training and validation mode CUDA graphs.
+ if (
+ hasattr(self, 'cudagraph_manager')
+ and kwargs.get('inference_context') is None
+ and not torch.is_inference_mode_enabled() # for inference eager dummy_forward
+ ):
return True
# Inference mode. CUDA graphs are used in the decode phase only, when attn mask is None
elif not self.training and (
@@ -1206,7 +1272,7 @@ def _should_call_local_cudagraph(self, *args, **kwargs):
(kwargs.get('inference_context') is not None)
or (kwargs.get('inference_params') is not None)
)
- and CudaGraphScope.full_iteration not in self.config.cuda_graph_scope
+ and not self.config.cuda_graph_scope # empty-list = per-layer CUDA graphs
):
if kwargs['inference_context'].is_static_batching():
using_cuda_graph = kwargs['inference_context'].is_decode_only()
@@ -1219,19 +1285,6 @@ def _should_call_local_cudagraph(self, *args, **kwargs):
return True
return False
- def __call__(self, *args, **kwargs):
- if self._should_call_local_cudagraph(*args, **kwargs):
- # Inference mode.
- if kwargs.get('inference_context') is not None:
- # dynamic_inference_decode_only is not a real argument to forward, it is only used
- # to differentiate the cuda graph used for decode from the one used for non-decode
- # inference.
- kwargs["dynamic_inference_decode_only"] = kwargs[
- 'inference_context'
- ].is_decode_only()
-
- return super().__call__(*args, **kwargs)
-
def get_layer_norm_weights(self):
"""
Get the weights of all layernorms (attention and MLP) in the transformer layer.
@@ -1259,6 +1312,62 @@ def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
+ def _should_call_local_cudagraph(self, *args, **kwargs):
+ """
+ Controls whether the full-layer cudagraph_manager captures the entire forward call
+ as a single graph. Returns False to skip full-layer capture and route through _forward_mlp.
+
+ MoE layers have two cudagraph modes:
+ - Full-layer (use_partial_cudagraphs=False): the full-layer cudagraph_manager captures
+ the forward pass as one graph. This is used during inference.
+ - Partial (use_partial_cudagraphs=True): the full-layer manager is bypassed (returns
+ False), and _forward_mlp routes through cudagraph_manager_router and
+ cudagraph_manager_postprocess, which are monkey-patched onto _forward_mlp_router
+ and _forward_mlp_postprocess by CudaGraphManager.__init__. The expert dispatch
+ in between runs eagerly. This is used during training.
+ """
+ if self.use_partial_cudagraphs:
+ return False
+ if self.config.cuda_graph_impl != "local":
+ return False
+ return super()._should_call_local_cudagraph(*args, **kwargs)
+
+ def transition_cudagraph_scope(self, mode):
+ """Transition between full-layer and partial CUDA graph capture.
+
+ Args:
+ mode: 'full' for inference (full-layer capture) or 'partial' for training
+ (router + postprocess captured, expert dispatch runs eagerly).
+ """
+ from megatron.core.transformer.cuda_graphs import CudaGraphManager
+
+ if mode == 'partial':
+ self.use_partial_cudagraphs = True
+ self.moe_layer_recompute = (
+ self.config.recompute_granularity == 'selective'
+ and "moe" in self.config.recompute_modules
+ and self.config.cuda_graph_impl == "local"
+ )
+ if not hasattr(self, '_router_dtoh_event'):
+ self._router_dtoh_event = torch.cuda.Event()
+ if not hasattr(self, 'cudagraph_manager_router'):
+ self.cudagraph_manager_router = CudaGraphManager(
+ self.config, self, function_name="_forward_mlp_router"
+ )
+ if not hasattr(self, 'cudagraph_manager_postprocess'):
+ self.cudagraph_manager_postprocess = CudaGraphManager(
+ self.config, self, function_name="_forward_mlp_postprocess"
+ )
+ elif mode == 'full':
+ self.use_partial_cudagraphs = False
+ self.mlp.fwd_execution_map = ["route", "expert_compute", "postprocess"]
+ assert hasattr(self, 'cudagraph_manager'), (
+ "MoETransformerLayer missing full cudagraph_manager; "
+ "expected it to be created at __init__ with scope = [] "
+ )
+ else:
+ raise ValueError(f"Unknown MoE cudagraph mode: {mode}, expected 'full' or 'partial'")
+
def create_mcore_cudagraph_manager(self, config):
"""
Initializes the CUDA graph manager(s) for the MoE layer.
@@ -1277,21 +1386,7 @@ def create_mcore_cudagraph_manager(self, config):
CudaGraphScope.moe_router in self.config.cuda_graph_scope
or CudaGraphScope.moe_preprocess in self.config.cuda_graph_scope
):
- # full MoE layer recompute with partial_cudagraphs. If not partial cudagraphs, MoE
- # layer recompute is handled by the moe_layer.MoELayer class
- self.moe_layer_recompute = (
- self.config.recompute_granularity == 'selective'
- and "moe" in self.config.recompute_modules
- and self.config.cuda_graph_impl == "local"
- )
-
- self.use_partial_cudagraphs = True
- self.cudagraph_manager_router = CudaGraphManager(
- self.config, self, function_name="_forward_mlp_router"
- )
- self.cudagraph_manager_postprocess = CudaGraphManager(
- self.config, self, function_name="_forward_mlp_postprocess"
- )
+ self.transition_cudagraph_scope('partial')
def _forward_mlp_router(self, hidden_states, padding_mask=None):
"""
@@ -1301,15 +1396,31 @@ def _forward_mlp_router(self, hidden_states, padding_mask=None):
This method is isolated so it can be captured by `cudagraph_manager_router`.
"""
- residual = hidden_states
self.mlp.fwd_execution_map = "route"
pre_mlp_layernorm_output = self._forward_pre_mlp_layernorm(hidden_states)
+ if isinstance(pre_mlp_layernorm_output, tuple):
+ if len(pre_mlp_layernorm_output) != 2:
+ raise ValueError(
+ f"When the output of pre_mlp_layernorm is a tuple, it is "
+ f"expected to have 2 elements (output, residual), but "
+ f"got {len(pre_mlp_layernorm_output)}"
+ )
+ pre_mlp_layernorm_output, residual = pre_mlp_layernorm_output
+ else:
+ residual = hidden_states
+
+ if self.config.fp32_residual_connection:
+ residual = residual.float()
+
router_outputs = self.mlp(
pre_mlp_layernorm_output, intermediate_tensors=(), padding_mask=padding_mask
)
for attr_name in self.mlp.token_dispatcher.cudagraph_attrs:
- attr = getattr(self.mlp.token_dispatcher, attr_name)
+ hier_attr_name = attr_name.split('.')
+ attr = self.mlp.token_dispatcher
+ for name in hier_attr_name:
+ attr = getattr(attr, name)
if torch.is_tensor(attr):
if attr_name in self.token_dispatcher_attrs:
self.token_dispatcher_attrs[attr_name].copy_(attr)
@@ -1327,8 +1438,12 @@ def _forward_mlp_expert_compute(self, hidden_states, probs):
step runs eagerly between the router and postprocess graph replays.
"""
- for name, attr in self.token_dispatcher_attrs.items():
- setattr(self.mlp.token_dispatcher, name, attr)
+ for attr_name, attr in self.token_dispatcher_attrs.items():
+ hier_attr_name = attr_name.split('.')
+ obj = self.mlp.token_dispatcher
+ for name in hier_attr_name[:-1]:
+ obj = getattr(obj, name)
+ setattr(obj, hier_attr_name[-1], attr)
self.mlp.fwd_execution_map = "expert_compute"
return self.mlp(None, intermediate_tensors=(hidden_states, probs))
@@ -1343,6 +1458,12 @@ def _forward_mlp_postprocess(self, residual, output, shared_expert_output, mlp_b
"""
+ # Restore token dispatcher attributes. During graph warmup, the router capture leaves these
+ # attrs pointing into cudagraph pool memory; restoring them here ensures the postprocess
+ # graph captures with valid pointers.
+ for name, attr in self.token_dispatcher_attrs.items():
+ setattr(self.mlp.token_dispatcher, name, attr)
+
self.mlp.fwd_execution_map = "postprocess"
output = self.mlp(None, intermediate_tensors=(output, shared_expert_output))
return self._forward_post_mlp((output, mlp_bias), residual)
@@ -1368,6 +1489,16 @@ def _forward_mlp_partial_cudagraphs(
residual, hidden_states, probs, shared_expert_output = self._forward_mlp_router(
hidden_states, padding_mask=padding_mask
)
+
+ # After the router graph replays, the captured .copy_() operations that update
+ # self.token_dispatcher_attrs via `_maybe_dtoh_and_synchronize` are queued on the
+ # current stream but may not have completed. Record an event after the router
+ # graph and wait on it, so we block only until the router's D2H copies complete.
+ self._router_dtoh_event.record()
+ self._router_dtoh_event.synchronize()
+ for name, attr in self.token_dispatcher_attrs.items():
+ setattr(self.mlp.token_dispatcher, name, attr)
+
expert_output, mlp_bias = self._forward_mlp_expert_compute(hidden_states, probs)
return self._forward_mlp_postprocess(
residual, expert_output, shared_expert_output, mlp_bias
diff --git a/megatron/core/transformer/utils.py b/megatron/core/transformer/utils.py
index 880c5309933..d1df4898d18 100644
--- a/megatron/core/transformer/utils.py
+++ b/megatron/core/transformer/utils.py
@@ -391,14 +391,12 @@ def find_modules_with_attrs(module):
find_modules_with_attrs(model_modules)
-def toggle_cuda_graphs(model, set_to="none", reset_cuda_graphs=True):
+def toggle_cuda_graphs(model, set_to="none"):
"""
Toggle CUDA graph-related attributes for the model and its modules.
Args:
set_to (str): Value to set for CUDA graph-related attributes.
- reset_cuda_graphs (bool): If True, remake the CUDA graph;
- if False, use cached CUDA graph managers.
"""
global cuda_graph_attr_cache
model_id = id(model)
@@ -427,25 +425,29 @@ def toggle_cuda_graphs(model, set_to="none", reset_cuda_graphs=True):
elif attribute == "cudagraph_manager":
for module in modules:
if set_to == "local":
- if reset_cuda_graphs:
- from megatron.core.transformer.cuda_graphs import CudaGraphManager
-
- # If we are resetting cuda graphs we create a new cuda graph manager
- setattr(module[0], attribute, CudaGraphManager(model.config))
- else:
- # If we are not resetting cuda graphs we set it to its cached cuda graph
- setattr(module[0], attribute, module[1])
+ # If we are not resetting cuda graphs we set it to its cached cuda graph
+ setattr(module[0], attribute, module[1])
else:
for module in modules:
# If we are deleting the cuda graph, we delete its attribute
if hasattr(module[0], "cudagraph_manager"):
delattr(module[0], "cudagraph_manager")
- from megatron.core.transformer.cuda_graphs import delete_cuda_graphs
- # if we are resetting cuda graphs we need to reset all the state
- if reset_cuda_graphs and set_to == "none":
- delete_cuda_graphs()
+def transition_moe_cudagraphs(model, scope: str):
+ """
+ Switch MoE layers to the given cudagraph scope. Flips between 'partial' and 'full'.
+
+ Args:
+ model: The model with MoE layers which will be transitioned.
+ scope: 'partial' for training (router + postprocess captured, expert dispatch eager)
+ or 'full' for inference (full-layer graph capture).
+ """
+ from megatron.core.transformer.transformer_layer import MoETransformerLayer
+
+ for module in model.modules():
+ if isinstance(module, MoETransformerLayer):
+ module.transition_cudagraph_scope(scope)
def is_layer_window_attention(
diff --git a/megatron/core/typed_torch.py b/megatron/core/typed_torch.py
index bcbf388facc..57d31fc2867 100644
--- a/megatron/core/typed_torch.py
+++ b/megatron/core/typed_torch.py
@@ -2,8 +2,9 @@
"""Utilities for improved type hinting with torch interfaces."""
from __future__ import annotations
+import inspect
from collections.abc import Callable
-from typing import Generic, ParamSpec, Protocol, TypeVar
+from typing import Any, Concatenate, Generic, Literal, ParamSpec, Protocol, TypeVar, overload
import torch
@@ -48,3 +49,174 @@ def not_none(value: T | None) -> T:
if value is None:
raise ValueError('Expected value to be not None')
return value
+
+
+R_src = TypeVar('R_src')
+R_dst = TypeVar('R_dst')
+P_src = ParamSpec('P_src')
+P_dst = ParamSpec('P_dst')
+First_dst = TypeVar('First_dst')
+
+
+@overload
+def copy_signature(
+ source: Callable[P_src, Any],
+ /,
+ *,
+ handle_return_type: Literal['preserve'] = 'preserve',
+ handle_first_src_param: Literal['copy'] = 'copy',
+ handle_first_dst_param: Literal['drop'] = 'drop',
+) -> Callable[[Callable[..., R_dst]], Callable[P_src, R_dst]]: ...
+
+
+@overload
+def copy_signature(
+ source: Callable[P_src, R_src],
+ /,
+ *,
+ handle_return_type: Literal['overwrite'],
+ handle_first_src_param: Literal['copy'] = 'copy',
+ handle_first_dst_param: Literal['drop'] = 'drop',
+) -> Callable[[Callable[..., Any]], Callable[P_src, R_src]]: ...
+
+
+@overload
+def copy_signature(
+ source: Callable[Concatenate[Any, P_src], Any],
+ /,
+ *,
+ handle_return_type: Literal['preserve'] = 'preserve',
+ handle_first_src_param: Literal['skip'],
+ handle_first_dst_param: Literal['drop'] = 'drop',
+) -> Callable[[Callable[..., R_dst]], Callable[P_src, R_dst]]: ...
+
+
+@overload
+def copy_signature(
+ source: Callable[Concatenate[Any, P_src], R_src],
+ /,
+ *,
+ handle_return_type: Literal['overwrite'],
+ handle_first_src_param: Literal['skip'],
+ handle_first_dst_param: Literal['drop'] = 'drop',
+) -> Callable[[Callable[..., Any]], Callable[P_src, R_src]]: ...
+
+
+@overload
+def copy_signature(
+ source: Callable[P_src, Any],
+ /,
+ *,
+ handle_return_type: Literal['preserve'] = 'preserve',
+ handle_first_src_param: Literal['copy'] = 'copy',
+ handle_first_dst_param: Literal['preserve'],
+) -> Callable[
+ [Callable[Concatenate[First_dst, ...], R_dst]], Callable[Concatenate[First_dst, P_src], R_dst]
+]: ...
+
+
+@overload
+def copy_signature(
+ source: Callable[P_src, R_src],
+ /,
+ *,
+ handle_return_type: Literal['overwrite'],
+ handle_first_src_param: Literal['copy'] = 'copy',
+ handle_first_dst_param: Literal['preserve'],
+) -> Callable[
+ [Callable[Concatenate[First_dst, ...], Any]], Callable[Concatenate[First_dst, P_src], R_src]
+]: ...
+
+
+@overload
+def copy_signature(
+ source: Callable[Concatenate[Any, P_src], Any],
+ /,
+ *,
+ handle_return_type: Literal['preserve'] = 'preserve',
+ handle_first_src_param: Literal['skip'],
+ handle_first_dst_param: Literal['preserve'],
+) -> Callable[
+ [Callable[Concatenate[First_dst, ...], R_dst]], Callable[Concatenate[First_dst, P_src], R_dst]
+]: ...
+
+
+@overload
+def copy_signature(
+ source: Callable[Concatenate[Any, P_src], R_src],
+ /,
+ *,
+ handle_return_type: Literal['overwrite'],
+ handle_first_src_param: Literal['skip'],
+ handle_first_dst_param: Literal['preserve'],
+) -> Callable[
+ [Callable[Concatenate[First_dst, ...], Any]], Callable[Concatenate[First_dst, P_src], R_src]
+]: ...
+
+
+def copy_signature(
+ source: Callable[..., Any],
+ /,
+ *,
+ handle_return_type: Literal['preserve', 'overwrite'] = 'preserve',
+ handle_first_src_param: Literal['copy', 'skip'] = 'copy',
+ handle_first_dst_param: Literal['preserve', 'drop'] = 'drop',
+):
+ """Decorator to copy the signature from one function to another.
+
+ Similar to `functools.wraps`, but preserves the signature instead of the
+ metadata. Useful when writing adapter/wrapper functions that forward arguments
+ to another function, as in:
+
+ def function_with_lots_of_args(
+ a: int,
+ b: str,
+ c: float,
+ ...
+ ) -> BigObject:
+ ...
+
+ @copy_signature(function_with_lots_of_args)
+ def convenient_wrapper(*args: Any, **kwargs: Any) -> str:
+ return function_with_lots_of_args(*args, **kwargs).to_string()
+
+ Args:
+ source: The function or callable from which to copy the signature.
+ handle_return_type: How to handle the return type of the decorated
+ function. 'preserve' to keep the decorated function's return type
+ (the default, since many wrappers are specifically written to return a
+ different type), or 'overwrite' to copy the source function's return
+ type as well.
+ handle_first_src_param: Whether to include the first parameter of the
+ source function. 'copy' to include it in the decorated function's
+ signature (the default), 'skip' to exclude it (useful for removing
+ 'self' or 'cls').
+ handle_first_dst_param: Whether to keep the first parameter of the
+ decorated function. 'drop' to overwrite it just like any other parameter
+ (the default), or 'preserve' to keep it in the decorated function's
+ signature (useful for preserving 'self' or 'cls').
+
+ Returns:
+ A decorator that copies the signature from `source` to the decorated function.
+ """
+ source_signature = inspect.signature(source)
+
+ def decorator(decorated: Callable[..., Any], /) -> Callable[..., Any]:
+ dest_signature = inspect.signature(decorated)
+ new_params = []
+ if handle_first_dst_param == 'preserve':
+ new_params.append(next(iter(dest_signature.parameters.values())))
+ src_params_iter = iter(source_signature.parameters.values())
+ if handle_first_src_param == 'skip':
+ next(src_params_iter)
+ new_params.extend(src_params_iter)
+ new_signature = dest_signature.replace(parameters=new_params)
+ if handle_return_type == 'overwrite':
+ new_signature = new_signature.replace(
+ return_annotation=source_signature.return_annotation
+ )
+
+ decorated.__signature__ = new_signature # type: ignore
+ return decorated
+
+ return decorator
diff --git a/megatron/core/utils.py b/megatron/core/utils.py
index b86ea66a18b..546b74a23c4 100644
--- a/megatron/core/utils.py
+++ b/megatron/core/utils.py
@@ -29,23 +29,10 @@
import numpy
import torch
-try:
- import torch.distributed._symmetric_memory as symm_mem
-
- HAVE_TORCH_SYMM_MEM = True
-except ImportError:
- HAVE_TORCH_SYMM_MEM = False
-
-try:
- import triton # pylint: disable=unused-import
-
- HAVE_TRITON = True
-except ImportError:
- HAVE_TRITON = False
-
from megatron.core import config
from megatron.core._rank_utils import log_single_rank
from megatron.core.package_info import __version__ as mcore_version
+from megatron.core.packed_seq_params import PackedSeqParams
try:
from torch.distributed._tensor import DTensor
@@ -91,6 +78,7 @@
_torch_version = PkgVersion("0.0.0") if HAVE_PACKAGING else "0.0.0"
_te_version = None
_fa_version = None
+_flashinfer_version = None
_mamba_ssm_version = None
_causal_conv1d_version = None
@@ -308,28 +296,6 @@ def __getattribute__(self, attr):
return validator
-def get_torch_version():
- """Get pytorch version from __version__; if not available use pip's. Use caching."""
-
- if not HAVE_PACKAGING:
- raise ImportError(
- "packaging is not installed. Please install it with `pip install packaging`."
- )
-
- def get_torch_version_str():
- import torch
-
- if hasattr(torch, "__version__"):
- return str(torch.__version__)
- else:
- return version("torch")
-
- global _torch_version
- if _torch_version is None:
- _torch_version = PkgVersion(get_torch_version_str())
- return _torch_version
-
-
def get_te_version():
"""Get TE version from __version__; if not available use pip's. Use caching."""
if not HAVE_PACKAGING:
@@ -353,8 +319,11 @@ def get_te_version_str():
return version("transformer-engine")
global _te_version
- if _te_version is None and HAVE_TE:
- _te_version = PkgVersion(get_te_version_str())
+ if _te_version is None:
+ if HAVE_TE:
+ _te_version = PkgVersion(get_te_version_str())
+ else:
+ _te_version = PkgVersion("0.0.0")
return _te_version
@@ -484,6 +453,44 @@ def is_causal_conv1d_min_version(version, check_equality=True):
return get_causal_conv1d_version() > PkgVersion(version)
+def get_flashinfer_version():
+ """Get flashinfer version from __version__; if not available use pip's. Use caching."""
+ if not HAVE_PACKAGING:
+ raise ImportError(
+ "packaging is not installed. Please install it with `pip install packaging`."
+ )
+
+ def get_flashinfer_version_str():
+ try:
+ import flashinfer
+ except ImportError:
+ return None
+
+ if hasattr(flashinfer, "__version__"):
+ return str(flashinfer.__version__)
+ else:
+ return version("flashinfer")
+
+ global _flashinfer_version
+ if _flashinfer_version is None:
+ if (flashinfer_version_str := get_flashinfer_version_str()) is not None:
+ _flashinfer_version = PkgVersion(flashinfer_version_str)
+ return _flashinfer_version
+
+
+def is_flashinfer_min_version(version, check_equality=True):
+ """Check if minimum version of `flashinfer` is installed."""
+ if not HAVE_PACKAGING:
+ raise ImportError(
+ "packaging is not installed. Please install it with `pip install packaging`."
+ )
+ if (flashinfer_version := get_flashinfer_version()) is None:
+ return False
+ if check_equality:
+ return flashinfer_version >= PkgVersion(version)
+ return flashinver_version > PkgVersion(version)
+
+
def ensure_divisibility(numerator, denominator):
"""Ensure that numerator is divisible by the denominator."""
assert numerator % denominator == 0, "{} is not divisible by {}".format(numerator, denominator)
@@ -496,17 +503,6 @@ def divide(numerator, denominator):
return numerator // denominator
-def deprecate_inference_params(inference_context, inference_params):
- """Print warning for deprecated `inference_params`."""
- if inference_context is None and inference_params is not None:
- warnings.warn(
- "`inference_params` renamed to `inference_context`, and will be "
- "removed in `megatron-core` 0.13."
- )
- return inference_params
- return inference_context
-
-
def get_tensor_model_parallel_group_if_none(tp_group, is_expert=False, check_initialized=True):
"""Issue a deprecation warning if tp_group is None and return the default tp group."""
# TODO(zijiey): remove this function later.
@@ -656,65 +652,6 @@ def get_tensor(self, tensor_shape, dtype, name, mem_alloc_context: Optional[Call
return self.buffer[(name, dtype)][0:required_len].view(*tensor_shape)
-class GlobalSymmetricMemoryBuffer:
- """
- Global symmetric memory buffer used in inference.
- This buffer is used by mcore-inference's low-latency
- NVLS all-gather and reduce-scatter collectives.
- """
-
- def __init__(self, size_in_mb, process_group):
- if not HAVE_TORCH_SYMM_MEM or not HAVE_TRITON:
- # This should be hit if the user is running an older
- # version of torch, or if they do not have triton
- # installed.
- self.symm_buffer = None
- self.symm_mem_hdl = None
- else:
- numel = int(size_in_mb * 1024 * 1024) # size in bytes
- try:
- symm_mem.enable_symm_mem_for_group(process_group.group_name)
- self.symm_buffer = symm_mem.empty(numel, dtype=torch.uint8, device='cuda')
- self.symm_mem_hdl = symm_mem.rendezvous(self.symm_buffer, process_group)
- except RuntimeError as e:
- # If symmetric memory initialization fails, set buffer and handle to None
- # This should happen if the process group is not contained within NVlink
- self.symm_buffer = None
- self.symm_mem_hdl = None
-
- def _can_allocate(self, numel, dtype) -> bool:
- """
- Returns whether enough symmetric memory is available
- for the given tensor shape and dtype.
- """
- if self.symm_mem_hdl is None:
- return False
- size_of_dtype = torch.tensor([], dtype=dtype).element_size()
- required_len = numel * size_of_dtype
- return required_len <= self.symm_buffer.numel()
-
- def _allocate(self, numel, dtype) -> torch.Tensor:
- """
- Allocates a sub-tensor from the self.symm_buffer for the given numel and dtype"""
- required_bytes = numel * torch.tensor([], dtype=dtype).element_size()
- return self.symm_buffer[0:required_bytes].view(dtype).view(numel)
-
- def maybe_get_tensor(self, tensor_shape, dtype):
- """
- Returns (potentially) a sub-tensor from the self.symm_buffer for the given shape.
- If enough symmetric memory is not available, returns None.
- """
- if self.symm_mem_hdl is None:
- return {"tensor": None, "handle": None}
- numel = reduce(operator.mul, tensor_shape, 1)
- if not self._can_allocate(numel, dtype):
- return {"tensor": None, "handle": None}
- return {
- "tensor": self._allocate(numel, dtype).view(*tensor_shape),
- "handle": self.symm_mem_hdl,
- }
-
-
def _kernel_make_viewless_tensor(inp, requires_grad):
"""Make a viewless tensor.
@@ -833,6 +770,26 @@ def scaled_init_method_normal(sigma, num_layers, multiplier=2.0):
return functools.partial(torch.nn.init.normal_, mean=0.0, std=std)
+def mup_scaled_init_method_normal(sigma, num_layers, width_mult, multiplier=2.0):
+ """MuP scaled init method for output layers: N(0, sigma / (sqrt(2*L) * sqrt(m))).
+
+ Combines the standard scaled initialization (for output projection layers)
+ with MuP width scaling. This ensures that both depth and width scaling
+ are accounted for in the initialization.
+
+ Args:
+ sigma (float): Base standard deviation for initialization.
+ num_layers (int): Number of transformer layers.
+ width_mult (float): Width multiplier (hidden_size / base_hidden_size).
+ multiplier (float): Multiplier for depth scaling (default: 2.0).
+
+ Returns:
+ Callable: Initialization function for torch.nn.init.
+ """
+ std = sigma / (math.sqrt(multiplier * num_layers) * math.sqrt(width_mult))
+ return functools.partial(torch.nn.init.normal_, mean=0.0, std=std)
+
+
def log_on_each_pipeline_stage(
logger: logging.Logger,
*args: Any,
@@ -2081,8 +2038,8 @@ def get_thd_batch_on_this_cp_rank(
max_seqlen_kv=int(max_seqlen[0].item()),
)
- cp_size = get_context_parallel_world_size() if cp_size is None else cp_size
- cp_rank = get_context_parallel_rank() if cp_rank is None else cp_rank
+ cp_size = parallel_state.get_context_parallel_world_size() if cp_size is None else cp_size
+ cp_rank = parallel_state.get_context_parallel_rank() if cp_rank is None else cp_rank
if cp_size > 1: # slice batch along sequence dimension for context parallelism
assert tex is not None and is_te_min_version("1.10.0"), (
"Please update Transformer Engine to >= 1.10 to use "
@@ -2405,25 +2362,6 @@ async def wrapper(*args, **kwargs):
return _decorate if func is None else _decorate(func)
-def get_mamba_inference_state_config_from_model(model) -> Optional["MambaInferenceStateConfig"]:
- """Returns Mamba inference state config from the model if it is a hybrid model."""
- from megatron.core.inference.contexts.attention_context.mamba_metadata import (
- MambaInferenceStateConfig,
- )
- from megatron.core.ssm.mamba_hybrid_layer_allocation import Symbols
-
- decoder = get_attr_wrapped_model(model, "decoder")
- layer_type_list = getattr(decoder, "layer_type_list", None)
- if layer_type_list is not None and Symbols.MAMBA in layer_type_list:
- (mamba_conv_states_shape, mamba_ssm_states_shape) = decoder.mamba_state_shapes_per_request()
- return MambaInferenceStateConfig(
- layer_type_list=layer_type_list,
- mamba_conv_states_shape=mamba_conv_states_shape,
- mamba_ssm_states_shape=mamba_ssm_states_shape,
- )
- return None
-
-
# ============================================================================
# Backward Compatibility Decorators
# ============================================================================
@@ -2558,3 +2496,43 @@ class ExperimentalModel:
"""
func._experimental_api = True
return func
+
+
+def deprecate_args(
+ *deprecated_keys, message="Argument '{name}' has been deprecated and should not be used."
+):
+ """
+ Intercepts specific keyword arguments to raise a custom TypeError.
+
+ Args:
+ *deprecated_keys: Strings representing the argument names to block.
+ message: Custom error message string. Use {name} as a placeholder.
+ """
+
+ def decorator(func):
+ @functools.wraps(func)
+ def wrapper(*args, **kwargs):
+ # Check if any deprecated key is present in kwargs
+ found_deprecated = set(deprecated_keys) & set(kwargs.keys())
+
+ if found_deprecated:
+ bad_key = list(found_deprecated)[0]
+ raise TypeError(message.format(name=bad_key))
+
+ # Send args to the real function
+ return func(*args, **kwargs)
+
+ return wrapper
+
+ return decorator
+
+
+def deprecate_inference_params(inference_context, inference_params):
+ """Print warning for deprecated `inference_params`."""
+ if inference_context is None and inference_params is not None:
+ warnings.warn(
+ "`inference_params` renamed to `inference_context`, and will be "
+ "removed in `megatron-core` 0.13."
+ )
+ return inference_params
+ return inference_context
diff --git a/megatron/inference/__init__.py b/megatron/inference/__init__.py
new file mode 100644
index 00000000000..26496bfed70
--- /dev/null
+++ b/megatron/inference/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
diff --git a/megatron/inference/utils.py b/megatron/inference/utils.py
new file mode 100644
index 00000000000..b59e2b77247
--- /dev/null
+++ b/megatron/inference/utils.py
@@ -0,0 +1,379 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+import logging
+from argparse import ArgumentParser
+from functools import partial
+from typing import Optional
+import torch
+
+from gpt_builders import gpt_builder
+from mamba_builders import mamba_builder
+from megatron.core.inference.config import (
+ InferenceConfig,
+ KVCacheManagementMode,
+ MambaInferenceStateConfig,
+ PrefixCachingCoordinatorPolicy,
+ PrefixCachingEvictionPolicy,
+)
+from megatron.core.inference.contexts import DynamicInferenceContext
+from megatron.core.inference.engines import DynamicInferenceEngine
+from megatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapper import (
+ GPTInferenceWrapper,
+)
+from megatron.core.inference.quantization.utils import quantize_model_to_mxfp8
+from megatron.core.inference.text_generation_controllers.text_generation_controller import (
+ TextGenerationController,
+)
+from megatron.core.tokenizers.utils.build_tokenizer import build_tokenizer
+from megatron.core.transformer.module import MegatronModule
+from megatron.core.utils import get_attr_wrapped_model, log_single_rank, unwrap_model
+from megatron.training import get_args
+from megatron.training import get_model as _get_model
+from megatron.training import get_tokenizer, get_wandb_writer
+from megatron.training.checkpointing import load_checkpoint
+from model_provider import model_provider
+
+logger = logging.getLogger(__name__)
+
+
+def get_model_for_inference() -> MegatronModule:
+ """Initialize model and load checkpoint for inference."""
+
+ args = get_args()
+
+ if args.model_provider == "gpt":
+ model_builder = gpt_builder
+ elif args.model_provider == "mamba":
+ model_builder = mamba_builder
+ else:
+ raise ValueError(f"Invalid model provider {args.model_provider}")
+
+ # Build model.
+ model = _get_model(partial(model_provider, model_builder), wrap_with_ddp=False)
+
+ # Load checkpoint.
+ assert args.load is not None
+ args.exit_on_missing_checkpoint = True
+ load_checkpoint(
+ ddp_model=model,
+ optimizer=None,
+ opt_param_scheduler=None,
+ strict=not args.inference_ckpt_non_strict,
+ )
+
+ # No virtual PP.
+ assert len(model) == 1, "Above condition should have caught this"
+ model = model[0]
+
+ # Eval mode.
+ model.eval()
+
+ if args.transformer_impl == "inference_optimized" and args.fp8_recipe == "mxfp8":
+ backend = args.inference_grouped_gemm_backend
+ if backend == "auto" or backend == "torch":
+ quant_backend = "triton"
+ elif backend == "te":
+ raise ValueError(
+ "MXFP8 quantization is not supported with "
+ "inference_grouped_gemm_backend='te'."
+ )
+ quantize_model_to_mxfp8(unwrap_model(model), backend=quant_backend)
+ return model
+
+
+def add_inference_args(parser: ArgumentParser) -> ArgumentParser:
+ """Add inference command line arguments to the parser."""
+
+ group = parser.add_argument_group(title='Inference')
+
+ group.add_argument("--temperature", type=float, default=1.0, help='Sampling temperature.')
+ group.add_argument("--top_k", type=int, default=1, help='Top k sampling.')
+ group.add_argument("--top_p", type=float, default=0.0, help='Top p sampling.')
+ group.add_argument(
+ "--return-log-probs",
+ action='store_true',
+ default=False,
+ help='Return the log probabilities of the final output tokens',
+ )
+ group.add_argument(
+ "--prompts",
+ metavar='N',
+ type=str,
+ nargs='+',
+ help='Input prompts with each prompt within quotes and separated by space',
+ )
+ group.add_argument(
+ "--num-tokens-to-prompt",
+ type=int,
+ nargs="+",
+ default=[64, 1024],
+ help='Number of tokens to use for simulated prompts. This should be a '
+ 'space-separated pair of integers, and the generated prompt lengths will '
+ 'be uniformly sampled within this range.',
+ )
+ group.add_argument(
+ "--num-tokens-to-generate",
+ type=int,
+ default=30,
+ help='Number of tokens to generate for each prompt',
+ )
+ group.add_argument(
+ "--num-tokens-from-file",
+ action='store_true',
+ default=False,
+ help='Use per-prompt num_tokens_to_generate from prompt file',
+ )
+ group.add_argument(
+ "--top-n-logprobs",
+ type=int,
+ default=0,
+ help=(
+ "Return the top n logprobs for the generated tokens and their "
+ "corresponding token as a dictionary"
+ ),
+ )
+ group.add_argument(
+ "--incoming-requests-per-step",
+ type=int,
+ default=None,
+ help="Add a deterministic number of requests per step. This arg is "
+ "prioritized over `--incoming-requests-per-sec` below (which is non-"
+ "deterministic). Note that the number of requests added per step is "
+ "additionally limited by the inference context's `max_requests`, "
+ "`max_tokens`, and KV buffer size.",
+ )
+ group.add_argument(
+ "--incoming-requests-per-sec",
+ type=float,
+ default=100.0,
+ help="Simulated number of requests per second. Set to -1 to add all requests together.",
+ )
+ group.add_argument(
+ "--incoming-requests-duration",
+ type=float,
+ default=10.0,
+ help="Total amount of time to simulate that requests are "
+ "arriving. Multiply this value with "
+ "`--incoming-requests-per-sec` to get the approximate "
+ "total number of requests. Set to -1 to add all requests together.",
+ )
+ group.add_argument(
+ "--model-provider", choices=["mamba", "gpt"], default="gpt", help="Model provider"
+ )
+ group.add_argument(
+ "--skip-prompt-log-probs", action='store_true', default=False, help='Skip prompt log probs.'
+ )
+ group.add_argument(
+ "--stop-words",
+ metavar='WORD',
+ type=str,
+ nargs='+',
+ default=None,
+ help='Stop words to terminate generation. Each word should be quoted and '
+ 'separated by space. Example: --stop-words "\\n\\n" "END" "###"',
+ )
+ group.add_argument(
+ "--output-path", type=str, default=None, help="Path to save generations as JSON"
+ )
+ group.add_argument(
+ "--output-every-n-results",
+ type=int,
+ default=1,
+ help="To minimize the output file size of larger runs, only write the "
+ "results of every `n` requests.",
+ )
+ group.add_argument(
+ "--output-request-events",
+ action='store_true',
+ default=False,
+ help="Include request events (lifecycle + per-token block allocator metrics) "
+ "in the JSON output.",
+ )
+ group.add_argument(
+ "--prompt-file",
+ help='Jsonl file containing input prompts, where each item (i.e., line) '
+ 'contains the field \'text\' where the value is the prompt. All other '
+ 'fields within each item are ignored, and may be customized for each '
+ 'application.',
+ )
+ group.add_argument(
+ "--prompt-file-num-truncate",
+ type=int,
+ help='Number of samples to use from the loaded prompt file (see '
+ '`--prompt-file` above). The first `--prompt-file-num-truncate` samples '
+ 'will be used, in order.',
+ )
+ group.add_argument(
+ "--use-flashinfer-fused-rope",
+ action='store_true',
+ default=False,
+ help='Use flashinfer fused rope implementation.',
+ )
+ group.add_argument(
+ "--no-record-throughput",
+ action='store_false',
+ dest="record_throughput",
+ help="Disable throughput recording in --output-file",
+ )
+ group.add_argument(
+ "--inference-ckpt-non-strict",
+ action="store_true",
+ help="Load checkpoint with `strict=False`.",
+ )
+ group.add_argument(
+ "--termination-id",
+ type=int,
+ default=None,
+ help="Termination ID that overrides `tokenizer.eod`.",
+ )
+ group.add_argument(
+ "--suspend-resume-interval",
+ type=int,
+ default=None,
+ help="Suspend and resume the dynamic engine every "
+ "`suspend_resume_interval` requests. This is used to test the suspend/resume "
+ "system.",
+ )
+ group.add_argument(
+ "--suspend-timeout",
+ type=float,
+ default=0.0,
+ help="Seconds to sleep while the engine is suspended (simulates a training step).",
+ )
+ group.add_argument(
+ "--inference-repeat-n",
+ type=int,
+ default=1,
+ help="Repeat inference iterations N times for benchmarking.",
+ )
+ group.add_argument(
+ "--throughput-check-only",
+ action='store_true',
+ default=False,
+ help="If true, only run throughput check without verifying outputs.",
+ )
+ group.add_argument(
+ "--drain-between-batches",
+ action='store_true',
+ default=False,
+ help="Process requests in batches, draining all active requests between batches.",
+ )
+ group.add_argument(
+ "--batch-boundaries",
+ type=str,
+ default=None,
+ help="Comma-separated list of request indices where each batch starts. "
+ "Used with --drain-between-batches.",
+ )
+ group.add_argument(
+ "--coordinator-schedule-output-path",
+ type=str,
+ default=None,
+ help="Path to write coordinator request scheduling decisions as JSON",
+ )
+ return parser
+
+
+def get_inference_config_from_model_and_args(model: MegatronModule, args):
+ """Returns a `InferenceConfig` constructed from the model and command line arguments."""
+
+ # Max sequence length.
+ position_embedding_type = get_attr_wrapped_model(model, "position_embedding_type")
+ model_max_seq_len = get_attr_wrapped_model(model, "max_sequence_length")
+ inf_max_seq_len = args.inference_max_seq_length
+ max_batch_size = args.inference_dynamic_batching_max_requests
+
+ if position_embedding_type == "learned_absolute":
+ # When using absolute position embeddings, it is critical that the
+ # context's `max_sequence_length` is less than or equal to the model's
+ # `max_sequence_length`. Otherwise, the context's `position_ids` will
+ # contain ids greater than the dimension of the position embedding
+ # tensor, which will result in an index error.
+ if inf_max_seq_len:
+ max_sequence_length = min(model_max_seq_len, inf_max_seq_len)
+ else:
+ max_sequence_length = model_max_seq_len
+ assert max_batch_size is None or max_batch_size <= model_max_seq_len
+ else:
+ max_sequence_length = inf_max_seq_len
+ if args.inference_dynamic_batching_max_requests is not None:
+ max_sequence_length = max(max_sequence_length, max_batch_size)
+
+ mamba_inference_state_config = MambaInferenceStateConfig.from_model(
+ model,
+ conv_states_dtype=args.mamba_inference_conv_states_dtype,
+ ssm_states_dtype=args.mamba_inference_ssm_states_dtype,
+ )
+ pg_collection = get_attr_wrapped_model(model, "pg_collection")
+
+ # Get inference logging configuration from args
+ log_inference_wandb = args.inference_wandb_logging
+ inference_logging_step_interval = args.inference_logging_step_interval
+
+ # Get metrics writer if logging is enabled and on the logging rank
+ # Use the same rank convention as training (last rank logs)
+ metrics_writer = None
+ if (
+ inference_logging_step_interval > 0
+ and log_inference_wandb
+ and args.rank == (args.world_size - 1)
+ ):
+ metrics_writer = get_wandb_writer()
+ if metrics_writer is None:
+ log_single_rank(
+ logger,
+ logging.WARNING,
+ "WARNING: --rl-inference-logging-step-interval is set but no metrics writer "
+ "wandb module is available. Inference logging will be disabled.",
+ )
+
+ return InferenceConfig(
+ block_size_tokens=args.inference_dynamic_batching_block_size,
+ buffer_size_gb=args.inference_dynamic_batching_buffer_size_gb,
+ paused_buffer_size_gb=args.inference_dynamic_batching_paused_buffer_size_gb,
+ mamba_memory_ratio=args.inference_dynamic_batching_mamba_memory_ratio,
+ num_cuda_graphs=(
+ args.inference_dynamic_batching_num_cuda_graphs
+ if args.cuda_graph_impl == "local"
+ else None
+ ),
+ max_requests=args.inference_dynamic_batching_max_requests,
+ max_tokens=args.inference_dynamic_batching_max_tokens,
+ unified_memory_level=args.inference_dynamic_batching_unified_memory_level,
+ kv_cache_management_mode=KVCacheManagementMode(args.rl_kv_cache_management_mode),
+ cuda_graph_mixed_prefill_count=args.inference_dynamic_batching_cuda_graph_mixed_prefill_count, # pylint: disable=line-too-long
+ use_cuda_graphs_for_non_decode_steps=not args.decode_only_cuda_graphs,
+ static_kv_memory_pointers=args.rl_persist_cuda_graphs,
+ max_sequence_length=max_sequence_length,
+ mamba_inference_state_config=mamba_inference_state_config,
+ pg_collection=pg_collection,
+ use_flashinfer_fused_rope=args.use_flashinfer_fused_rope,
+ materialize_only_last_token_logits=(not args.return_log_probs and args.num_speculative_tokens == 0),
+ track_generated_token_events=args.inference_dynamic_batching_track_generated_token_events,
+ track_paused_request_events=args.inference_dynamic_batching_track_paused_request_events,
+ enable_chunked_prefill=args.enable_chunked_prefill,
+ enable_prefix_caching=args.inference_dynamic_batching_enable_prefix_caching,
+ prefix_caching_eviction_policy=PrefixCachingEvictionPolicy(args.inference_dynamic_batching_prefix_caching_eviction_policy),
+ prefix_caching_coordinator_policy=PrefixCachingCoordinatorPolicy(args.inference_dynamic_batching_prefix_caching_coordinator_policy),
+ prefix_caching_routing_alpha=getattr(args, 'inference_dynamic_batching_prefix_caching_routing_alpha', 0.5),
+ prefix_caching_mamba_gb=getattr(args, 'inference_dynamic_batching_prefix_caching_mamba_gb', None),
+ metrics_writer=metrics_writer,
+ logging_step_interval=args.inference_logging_step_interval,
+ num_speculative_tokens=args.num_speculative_tokens,
+ use_synchronous_zmq_collectives=args.inference_use_synchronous_zmq_collectives,
+ )
+
+
+def get_dynamic_inference_engine(model: Optional[MegatronModule] = None) -> DynamicInferenceEngine:
+ """Builds a `DynamicInferenceEngine`."""
+ args = get_args()
+ if model is None:
+ model = get_model_for_inference()
+ tokenizer = build_tokenizer(args)
+
+ inference_config = get_inference_config_from_model_and_args(model, args)
+ context = DynamicInferenceContext(model.config, inference_config)
+ inference_wrapped_model = GPTInferenceWrapper(model, context)
+ controller = TextGenerationController(inference_wrapped_model, tokenizer)
+ engine = DynamicInferenceEngine(controller, context)
+ return engine
diff --git a/megatron/legacy/data/__init__.py b/megatron/legacy/data/__init__.py
deleted file mode 100644
index f8011007a50..00000000000
--- a/megatron/legacy/data/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
diff --git a/megatron/legacy/data/autoaugment.py b/megatron/legacy/data/autoaugment.py
deleted file mode 100644
index d86127a60b2..00000000000
--- a/megatron/legacy/data/autoaugment.py
+++ /dev/null
@@ -1,321 +0,0 @@
-# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
-"""AutoAugment data augmentation policy for ImageNet.
-
--- Begin license text.
-
-MIT License
-
-Copyright (c) 2018 Philip Popien
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
-
--- End license text.
-
-Code adapted from https://github.com/DeepVoltaire/AutoAugment.
-
-This module implements the fixed AutoAugment data augmentation policy for ImageNet provided in
-Appendix A, Table 9 of reference [1]. It does not include any of the search code for augmentation
-policies.
-
-Reference:
-[1] https://arxiv.org/abs/1805.09501
-"""
-
-import random
-
-import numpy as np
-from PIL import Image
-from PIL import ImageEnhance
-from PIL import ImageOps
-
-_MAX_LEVEL = 10 # Maximum integer strength of an augmentation, if applicable.
-
-
-class ImageNetPolicy:
- """Definition of an ImageNetPolicy.
-
- Implements a fixed AutoAugment data augmentation policy targeted at
- ImageNet training by randomly applying at runtime one of the 25 pre-defined
- data augmentation sub-policies provided in Reference [1].
-
- Usage example as a Pytorch Transform:
- >>> transform=transforms.Compose([transforms.Resize(256),
- >>> ImageNetPolicy(),
- >>> transforms.ToTensor()])
- """
-
- def __init__(self, fillcolor=(128, 128, 128)):
- """Initialize an ImageNetPolicy.
-
- Args:
- fillcolor (tuple): RGB color components of the color to be used for
- filling when needed (default: (128, 128, 128), which
- corresponds to gray).
- """
- # Instantiate a list of sub-policies.
- # Each entry of the list is a SubPolicy which consists of
- # two augmentation operations,
- # each of those parametrized as operation, probability, magnitude.
- # Those two operations are applied sequentially on the image upon call.
- self.policies = [
- SubPolicy("posterize", 0.4, 8, "rotate", 0.6, 9, fillcolor),
- SubPolicy("solarize", 0.6, 5, "autocontrast", 0.6, 5, fillcolor),
- SubPolicy("equalize", 0.8, 8, "equalize", 0.6, 3, fillcolor),
- SubPolicy("posterize", 0.6, 7, "posterize", 0.6, 6, fillcolor),
- SubPolicy("equalize", 0.4, 7, "solarize", 0.2, 4, fillcolor),
- SubPolicy("equalize", 0.4, 4, "rotate", 0.8, 8, fillcolor),
- SubPolicy("solarize", 0.6, 3, "equalize", 0.6, 7, fillcolor),
- SubPolicy("posterize", 0.8, 5, "equalize", 1.0, 2, fillcolor),
- SubPolicy("rotate", 0.2, 3, "solarize", 0.6, 8, fillcolor),
- SubPolicy("equalize", 0.6, 8, "posterize", 0.4, 6, fillcolor),
- SubPolicy("rotate", 0.8, 8, "color", 0.4, 0, fillcolor),
- SubPolicy("rotate", 0.4, 9, "equalize", 0.6, 2, fillcolor),
- SubPolicy("equalize", 0.0, 7, "equalize", 0.8, 8, fillcolor),
- SubPolicy("invert", 0.6, 4, "equalize", 1.0, 8, fillcolor),
- SubPolicy("color", 0.6, 4, "contrast", 1.0, 8, fillcolor),
- SubPolicy("rotate", 0.8, 8, "color", 1.0, 2, fillcolor),
- SubPolicy("color", 0.8, 8, "solarize", 0.8, 7, fillcolor),
- SubPolicy("sharpness", 0.4, 7, "invert", 0.6, 8, fillcolor),
- SubPolicy("shearX", 0.6, 5, "equalize", 1.0, 9, fillcolor),
- SubPolicy("color", 0.4, 0, "equalize", 0.6, 3, fillcolor),
- SubPolicy("equalize", 0.4, 7, "solarize", 0.2, 4, fillcolor),
- SubPolicy("solarize", 0.6, 5, "autocontrast", 0.6, 5, fillcolor),
- SubPolicy("invert", 0.6, 4, "equalize", 1.0, 8, fillcolor),
- SubPolicy("color", 0.6, 4, "contrast", 1.0, 8, fillcolor),
- SubPolicy("equalize", 0.8, 8, "equalize", 0.6, 3, fillcolor),
- ]
-
- def __call__(self, img):
- """Define call method for ImageNetPolicy class."""
- policy_idx = random.randint(0, len(self.policies) - 1)
- return self.policies[policy_idx](img)
-
- def __repr__(self):
- """Define repr method for ImageNetPolicy class."""
- return "ImageNetPolicy"
-
-
-class SubPolicy:
- """Definition of a SubPolicy.
-
- A SubPolicy consists of two augmentation operations,
- each of those parametrized as operation, probability, magnitude.
- The two operations are applied sequentially on the image upon call.
- """
-
- def __init__(
- self,
- operation1,
- probability1,
- magnitude_idx1,
- operation2,
- probability2,
- magnitude_idx2,
- fillcolor,
- ):
- """Initialize a SubPolicy.
-
- Args:
- operation1 (str): Key specifying the first augmentation operation.
- There are fourteen key values altogether (see supported_ops below
- listing supported operations). probability1 (float): Probability
- within [0., 1.] of applying the first augmentation operation.
- magnitude_idx1 (int): Integer specifiying the strength of the first
- operation as an index further used to derive the magnitude from a
- range of possible values.
- operation2 (str): Key specifying the second augmentation operation.
- probability2 (float): Probability within [0., 1.] of applying the
- second augmentation operation.
- magnitude_idx2 (int): Integer specifiying the strength of the
- second operation as an index further used to derive the magnitude
- from a range of possible values.
- fillcolor (tuple): RGB color components of the color to be used for
- filling.
- Returns:
- """
- # List of supported operations for operation1 and operation2.
- supported_ops = [
- "shearX",
- "shearY",
- "translateX",
- "translateY",
- "rotate",
- "color",
- "posterize",
- "solarize",
- "contrast",
- "sharpness",
- "brightness",
- "autocontrast",
- "equalize",
- "invert",
- ]
- assert (operation1 in supported_ops) and (
- operation2 in supported_ops
- ), "SubPolicy:one of oper1 or oper2 refers to an unsupported operation."
-
- assert (
- 0.0 <= probability1 <= 1.0 and 0.0 <= probability2 <= 1.0
- ), "SubPolicy: prob1 and prob2 should be within [0., 1.]."
-
- assert (
- isinstance(magnitude_idx1, int) and 0 <= magnitude_idx1 <= 10
- ), "SubPolicy: idx1 should be specified as an integer within [0, 10]."
-
- assert (
- isinstance(magnitude_idx2, int) and 0 <= magnitude_idx2 <= 10
- ), "SubPolicy: idx2 should be specified as an integer within [0, 10]."
-
- # Define a dictionary where each key refers to a specific type of
- # augmentation and the corresponding value is a range of ten possible
- # magnitude values for that augmentation.
- num_levels = _MAX_LEVEL + 1
- ranges = {
- "shearX": np.linspace(0, 0.3, num_levels),
- "shearY": np.linspace(0, 0.3, num_levels),
- "translateX": np.linspace(0, 150 / 331, num_levels),
- "translateY": np.linspace(0, 150 / 331, num_levels),
- "rotate": np.linspace(0, 30, num_levels),
- "color": np.linspace(0.0, 0.9, num_levels),
- "posterize": np.round(np.linspace(8, 4, num_levels), 0).astype(
- np.int32
- ),
- "solarize": np.linspace(256, 0, num_levels), # range [0, 256]
- "contrast": np.linspace(0.0, 0.9, num_levels),
- "sharpness": np.linspace(0.0, 0.9, num_levels),
- "brightness": np.linspace(0.0, 0.9, num_levels),
- "autocontrast": [0]
- * num_levels, # This augmentation doesn't use magnitude parameter.
- "equalize": [0]
- * num_levels, # This augmentation doesn't use magnitude parameter.
- "invert": [0]
- * num_levels, # This augmentation doesn't use magnitude parameter.
- }
-
- def rotate_with_fill(img, magnitude):
- """Define rotation transformation with fill.
-
- The input image is first rotated, then it is blended together with
- a gray mask of the same size. Note that fillcolor as defined
- elsewhere in this module doesn't apply here.
-
- Args:
- magnitude (float): rotation angle in degrees.
- Returns:
- rotated_filled (PIL Image): rotated image with gray filling for
- disoccluded areas unveiled by the rotation.
- """
- rotated = img.convert("RGBA").rotate(magnitude)
- rotated_filled = Image.composite(
- rotated, Image.new("RGBA", rotated.size, (128,) * 4), rotated
- )
- return rotated_filled.convert(img.mode)
-
- # Define a dictionary of augmentation functions where each key refers
- # to a specific type of augmentation and the corresponding value defines
- # the augmentation itself using a lambda function.
- # pylint: disable=unnecessary-lambda
- func_dict = {
- "shearX": lambda img, magnitude: img.transform(
- img.size,
- Image.AFFINE,
- (1, magnitude * random.choice([-1, 1]), 0, 0, 1, 0),
- Image.BICUBIC,
- fillcolor=fillcolor,
- ),
- "shearY": lambda img, magnitude: img.transform(
- img.size,
- Image.AFFINE,
- (1, 0, 0, magnitude * random.choice([-1, 1]), 1, 0),
- Image.BICUBIC,
- fillcolor=fillcolor,
- ),
- "translateX": lambda img, magnitude: img.transform(
- img.size,
- Image.AFFINE,
- (
- 1,
- 0,
- magnitude * img.size[0] * random.choice([-1, 1]),
- 0,
- 1,
- 0,
- ),
- fillcolor=fillcolor,
- ),
- "translateY": lambda img, magnitude: img.transform(
- img.size,
- Image.AFFINE,
- (
- 1,
- 0,
- 0,
- 0,
- 1,
- magnitude * img.size[1] * random.choice([-1, 1]),
- ),
- fillcolor=fillcolor,
- ),
- "rotate": lambda img, magnitude: rotate_with_fill(img, magnitude),
- "color": lambda img, magnitude: ImageEnhance.Color(img).enhance(
- 1 + magnitude * random.choice([-1, 1])
- ),
- "posterize": lambda img, magnitude: ImageOps.posterize(
- img, magnitude
- ),
- "solarize": lambda img, magnitude: ImageOps.solarize(
- img, magnitude
- ),
- "contrast": lambda img, magnitude: ImageEnhance.Contrast(
- img
- ).enhance(1 + magnitude * random.choice([-1, 1])),
- "sharpness": lambda img, magnitude: ImageEnhance.Sharpness(
- img
- ).enhance(1 + magnitude * random.choice([-1, 1])),
- "brightness": lambda img, magnitude: ImageEnhance.Brightness(
- img
- ).enhance(1 + magnitude * random.choice([-1, 1])),
- "autocontrast": lambda img, magnitude: ImageOps.autocontrast(img),
- "equalize": lambda img, magnitude: ImageOps.equalize(img),
- "invert": lambda img, magnitude: ImageOps.invert(img),
- }
-
- # Store probability, function and magnitude of the first augmentation
- # for the sub-policy.
- self.probability1 = probability1
- self.operation1 = func_dict[operation1]
- self.magnitude1 = ranges[operation1][magnitude_idx1]
-
- # Store probability, function and magnitude of the second augmentation
- # for the sub-policy.
- self.probability2 = probability2
- self.operation2 = func_dict[operation2]
- self.magnitude2 = ranges[operation2][magnitude_idx2]
-
- def __call__(self, img):
- """Define call method for SubPolicy class."""
- # Randomly apply operation 1.
- if random.random() < self.probability1:
- img = self.operation1(img, self.magnitude1)
-
- # Randomly apply operation 2.
- if random.random() < self.probability2:
- img = self.operation2(img, self.magnitude2)
-
- return img
diff --git a/megatron/legacy/data/biencoder_dataset_utils.py b/megatron/legacy/data/biencoder_dataset_utils.py
deleted file mode 100644
index 6d69fabbe48..00000000000
--- a/megatron/legacy/data/biencoder_dataset_utils.py
+++ /dev/null
@@ -1,212 +0,0 @@
-# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
-import os
-import time
-
-import numpy as np
-import torch
-
-from megatron.core import mpu, tensor_parallel
-from megatron.legacy.data.dataset_utils import (
- create_masked_lm_predictions,
- pad_and_convert_to_numpy,
-)
-from megatron.training import get_args, get_tokenizer, print_rank_0
-from megatron.training.datasets.data_samplers import MegatronPretrainingSampler
-
-
-def make_attention_mask(source_block, target_block):
- """
- Returns a 2-dimensional (2-D) attention mask
- :param source_block: 1-D array
- :param target_block: 1-D array
- """
- mask = (target_block[None, :] >= 1) * (source_block[:, None] >= 1)
- mask = mask.astype(np.int64)
- # (source_length, target_length)
- return mask
-
-def get_one_epoch_dataloader(dataset, micro_batch_size=None):
- """Specifically one epoch to be used in an indexing job."""
- args = get_args()
-
- if micro_batch_size is None:
- micro_batch_size = args.micro_batch_size
- num_workers = args.num_workers
-
- # Use megatron's sampler with consumed samples set to 0 as
- # this is only for evaluation and don't intend to resume half way.
- # Also, set the drop last to false as don't intend to remove
- # the last batch
- batch_sampler = MegatronPretrainingSampler(
- total_samples=len(dataset),
- consumed_samples=0,
- micro_batch_size=args.micro_batch_size,
- data_parallel_rank=mpu.get_data_parallel_rank(),
- data_parallel_size=mpu.get_data_parallel_world_size(),
- drop_last=False)
-
- return torch.utils.data.DataLoader(dataset,
- batch_sampler=batch_sampler,
- num_workers=num_workers,
- pin_memory=True)
-
-
-def get_ict_batch(data_iterator):
- # Items and their type.
- keys = ['query_tokens', 'query_mask',
- 'context_tokens', 'context_mask', 'block_data']
- datatype = torch.int64
-
- # Broadcast data.
- if data_iterator is None:
- data = None
- else:
- data = next(data_iterator)
- data_b = tensor_parallel.broadcast_data(keys, data, datatype)
-
- # Unpack.
- query_tokens = data_b['query_tokens'].long()
- query_mask = data_b['query_mask'] < 0.5
- context_tokens = data_b['context_tokens'].long()
- context_mask = data_b['context_mask'] < 0.5
- block_indices = data_b['block_data'].long()
-
- return query_tokens, query_mask,\
- context_tokens, context_mask, block_indices
-
-
-def join_str_list(str_list):
- """Join a list of strings, handling spaces appropriately"""
- result = ""
- for s in str_list:
- if s.startswith("##"):
- result += s[2:]
- else:
- result += " " + s
- return result
-
-
-class BlockSampleData(object):
- """A struct for fully describing a fixed-size block of data as used in REALM
-
- :param start_idx: for first sentence of the block
- :param end_idx: for last sentence of the block (may be partially truncated in sample construction)
- :param doc_idx: the index of the document from which the block comes in the original indexed dataset
- :param block_idx: a unique integer identifier given to every block.
- """
- def __init__(self, start_idx, end_idx, doc_idx, block_idx):
- self.start_idx = start_idx
- self.end_idx = end_idx
- self.doc_idx = doc_idx
- self.block_idx = block_idx
-
- def as_array(self):
- return np.array([self.start_idx, self.end_idx, self.doc_idx, self.block_idx]).astype(np.int64)
-
- def as_tuple(self):
- return self.start_idx, self.end_idx, self.doc_idx, self.block_idx
-
-
-class BlockSamplesMapping(object):
- def __init__(self, mapping_array):
- # make sure that the array is compatible with BlockSampleData
- assert mapping_array.shape[1] == 4
- self.mapping_array = mapping_array
-
- def __len__(self):
- return self.mapping_array.shape[0]
-
- def __getitem__(self, idx):
- """Get the data associated with an indexed sample."""
- sample_data = BlockSampleData(*self.mapping_array[idx])
- return sample_data
-
-
-def get_block_samples_mapping(block_dataset, title_dataset, data_prefix, num_epochs,
- max_num_samples, max_seq_length, seed, name, use_one_sent_docs=False):
- """Get samples mapping for a dataset over fixed size blocks. This function also requires
- a dataset of the titles for the source documents since their lengths must be taken into account.
-
- :return: samples_mapping (BlockSamplesMapping)
- """
-
- if not num_epochs:
- if not max_num_samples:
- raise ValueError("Need to specify either max_num_samples "
- "or num_epochs")
- num_epochs = np.iinfo(np.int32).max - 1
- if not max_num_samples:
- max_num_samples = np.iinfo(np.int64).max - 1
-
- # Filename of the index mapping
- indexmap_filename = data_prefix
- indexmap_filename += '_{}_indexmap'.format(name)
- if num_epochs != (np.iinfo(np.int32).max - 1):
- indexmap_filename += '_{}ep'.format(num_epochs)
- if max_num_samples != (np.iinfo(np.int64).max - 1):
- indexmap_filename += '_{}mns'.format(max_num_samples)
- indexmap_filename += '_{}msl'.format(max_seq_length)
- indexmap_filename += '_{}s'.format(seed)
- if use_one_sent_docs:
- indexmap_filename += '_1sentok'
- indexmap_filename += '.npy'
-
- # Build the indexed mapping if not exist.
- if mpu.get_data_parallel_rank() == 0 and \
- not os.path.isfile(indexmap_filename):
- print(' > WARNING: could not find index map file {}, building '
- 'the indices on rank 0 ...'.format(indexmap_filename))
-
- # Make sure the types match the helpers input types.
- assert block_dataset.document_indices.dtype == np.int64
- assert block_dataset.sequence_lengths.dtype == np.int32
-
- # Build samples mapping
- verbose = torch.distributed.get_rank() == 0
- start_time = time.time()
- print_rank_0(' > building samples index mapping for {} ...'.format(
- name))
-
- from megatron.core.datasets import helpers
- mapping_array = helpers.build_blocks_mapping(
- block_dataset.document_indices,
- block_dataset.sequence_lengths,
- title_dataset.sequence_lengths,
- num_epochs,
- max_num_samples,
- max_seq_length - 3, # account for added tokens
- seed,
- verbose,
- use_one_sent_docs)
-
-
- print_rank_0(' > done building samples index mapping')
- np.save(indexmap_filename, mapping_array, allow_pickle=True)
- print_rank_0(' > saved the index mapping in {}'.format(
- indexmap_filename))
- # Make sure all the ranks have built the mapping
- print_rank_0(' > elapsed time to build and save samples mapping '
- '(seconds): {:4f}'.format(
- time.time() - start_time))
-
- # This should be a barrier but nccl barrier assumes
- # device_index=rank which is not the case for model
- # parallel case
- counts = torch.tensor([1], dtype=torch.long, device='cuda')
- torch.distributed.all_reduce(counts, group=mpu.get_data_parallel_group())
- assert counts[0].item() == mpu.get_data_parallel_group().size()
-
- # Load indexed dataset.
- print_rank_0(' > loading indexed mapping from {}'.format(
- indexmap_filename))
- start_time = time.time()
-
- mapping_array = np.load(indexmap_filename, allow_pickle=True, mmap_mode='r')
- samples_mapping = BlockSamplesMapping(mapping_array)
-
- print_rank_0(' loaded indexed file in {:3.3f} seconds'.format(
- time.time() - start_time))
- print_rank_0(' total number of samples: {}'.format(
- mapping_array.shape[0]))
-
- return samples_mapping
diff --git a/megatron/legacy/data/dataset_utils.py b/megatron/legacy/data/dataset_utils.py
deleted file mode 100644
index 1f372147ebb..00000000000
--- a/megatron/legacy/data/dataset_utils.py
+++ /dev/null
@@ -1,727 +0,0 @@
-# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
-# coding=utf-8
-# Copyright 2018 The Google AI Language Team Authors, and NVIDIA.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-# Most of the code here has been copied from:
-# https://github.com/google-research/albert/blob/master/create_pretraining_data.py
-# with some modifications.
-
-import math
-import os
-import time
-import collections
-
-import numpy as np
-import torch
-
-from megatron.training import (
- get_args,
- print_rank_0
-)
-from megatron.core import mpu
-from megatron.core.datasets.indexed_dataset import IndexedDataset
-
-
-DSET_TYPE_BERT = 'standard_bert'
-DSET_TYPE_ICT = 'ict'
-DSET_TYPE_T5 = 't5'
-DSET_TYPE_MULTIMODAL = 'multimodal'
-
-DSET_TYPES = [DSET_TYPE_BERT, DSET_TYPE_ICT, DSET_TYPE_T5, DSET_TYPE_MULTIMODAL]
-
-
-def get_datasets_weights_and_num_samples(data_prefix,
- train_valid_test_num_samples):
-
- # The data prefix should be in the format of:
- # weight-1, data-prefix-1, weight-2, data-prefix-2, ..
- assert len(data_prefix) % 2 == 0
- num_datasets = len(data_prefix) // 2
- weights = [0]*num_datasets
- prefixes = [0]*num_datasets
- for i in range(num_datasets):
- weights[i] = float(data_prefix[2*i])
- prefixes[i] = (data_prefix[2*i+1]).strip()
- # Normalize weights
- weight_sum = 0.0
- for weight in weights:
- weight_sum += weight
- assert weight_sum > 0.0
- weights = [weight / weight_sum for weight in weights]
-
- # Add 0.5% (the 1.005 factor) so in case the bleding dataset does
- # not uniformly distribute the number of samples, we still have
- # samples left to feed to the network.
- if isinstance(train_valid_test_num_samples, list):
- datasets_train_valid_test_num_samples = []
- for weight in weights:
- datasets_train_valid_test_num_samples.append(
- [int(math.ceil(val * weight * 1.005))
- for val in train_valid_test_num_samples])
- else:
- # Used when separate dataset files are provided for train,
- # valid and test
- datasets_train_valid_test_num_samples = [
- int(math.ceil(train_valid_test_num_samples * weight * 1.005))
- for weight in weights]
-
- return prefixes, weights, datasets_train_valid_test_num_samples
-
-
-def get_a_and_b_segments(sample, np_rng):
- """Divide sample into a and b segments."""
-
- # Number of sentences in the sample.
- n_sentences = len(sample)
- # Make sure we always have two sentences.
- assert n_sentences > 1, 'make sure each sample has at least two sentences.'
-
- # First part:
- # `a_end` is how many sentences go into the `A`.
- a_end = 1
- if n_sentences >= 3:
- # Note that randin in numpy is exclusive.
- a_end = np_rng.randint(1, n_sentences)
- tokens_a = []
- for j in range(a_end):
- tokens_a.extend(sample[j])
-
- # Second part:
- tokens_b = []
- for j in range(a_end, n_sentences):
- tokens_b.extend(sample[j])
-
- # Random next:
- is_next_random = False
- if np_rng.random() < 0.5:
- is_next_random = True
- tokens_a, tokens_b = tokens_b, tokens_a
-
- return tokens_a, tokens_b, is_next_random
-
-
-def truncate_segments(tokens_a, tokens_b, len_a, len_b, max_num_tokens, np_rng):
- """Truncates a pair of sequences to a maximum sequence length."""
- #print(len_a, len_b, max_num_tokens)
- assert len_a > 0
- if len_a + len_b <= max_num_tokens:
- return False
- while len_a + len_b > max_num_tokens:
- if len_a > len_b:
- len_a -= 1
- tokens = tokens_a
- else:
- len_b -= 1
- tokens = tokens_b
- if np_rng.random() < 0.5:
- del tokens[0]
- else:
- tokens.pop()
- return True
-
-
-def create_tokens_and_tokentypes(tokens_a, tokens_b, cls_id, sep_id):
- """Merge segments A and B, add [CLS] and [SEP] and build tokentypes."""
-
- tokens = []
- tokentypes = []
- # [CLS].
- tokens.append(cls_id)
- tokentypes.append(0)
- # Segment A.
- for token in tokens_a:
- tokens.append(token)
- tokentypes.append(0)
- # [SEP].
- tokens.append(sep_id)
- tokentypes.append(0)
- # Segment B.
- for token in tokens_b:
- tokens.append(token)
- tokentypes.append(1)
- if tokens_b:
- # [SEP].
- tokens.append(sep_id)
- tokentypes.append(1)
-
- return tokens, tokentypes
-
-
-MaskedLmInstance = collections.namedtuple("MaskedLmInstance",
- ["index", "label"])
-
-
-def is_start_piece(piece):
- """Check if the current word piece is the starting piece (BERT)."""
- # When a word has been split into
- # WordPieces, the first token does not have any marker and any subsequence
- # tokens are prefixed with ##. So whenever we see the ## token, we
- # append it to the previous set of word indexes.
- return not piece.startswith("##")
-
-
-def create_masked_lm_predictions(tokens,
- vocab_id_list, vocab_id_to_token_dict,
- masked_lm_prob,
- cls_id, sep_id, mask_id,
- max_predictions_per_seq,
- np_rng,
- max_ngrams=3,
- do_whole_word_mask=True,
- favor_longer_ngram=False,
- do_permutation=False,
- geometric_dist=False,
- masking_style="bert"):
- """Creates the predictions for the masked LM objective.
- Note: Tokens here are vocab ids and not text tokens."""
-
- cand_indexes = []
- # Note(mingdachen): We create a list for recording if the piece is
- # the starting piece of current token, where 1 means true, so that
- # on-the-fly whole word masking is possible.
- token_boundary = [0] * len(tokens)
-
- for (i, token) in enumerate(tokens):
- if token == cls_id or token == sep_id:
- token_boundary[i] = 1
- continue
- # Whole Word Masking means that if we mask all of the wordpieces
- # corresponding to an original word.
- #
- # Note that Whole Word Masking does *not* change the training code
- # at all -- we still predict each WordPiece independently, softmaxed
- # over the entire vocabulary.
- if (do_whole_word_mask and len(cand_indexes) >= 1 and
- not is_start_piece(vocab_id_to_token_dict[token])):
- cand_indexes[-1].append(i)
- else:
- cand_indexes.append([i])
- if is_start_piece(vocab_id_to_token_dict[token]):
- token_boundary[i] = 1
-
- output_tokens = list(tokens)
-
- masked_lm_positions = []
- masked_lm_labels = []
-
- if masked_lm_prob == 0:
- return (output_tokens, masked_lm_positions,
- masked_lm_labels, token_boundary)
-
- num_to_predict = min(max_predictions_per_seq,
- max(1, int(round(len(tokens) * masked_lm_prob))))
-
- ngrams = np.arange(1, max_ngrams + 1, dtype=np.int64)
- if not geometric_dist:
- # Note(mingdachen):
- # By default, we set the probilities to favor shorter ngram sequences.
- pvals = 1. / np.arange(1, max_ngrams + 1)
- pvals /= pvals.sum(keepdims=True)
- if favor_longer_ngram:
- pvals = pvals[::-1]
-
- ngram_indexes = []
- for idx in range(len(cand_indexes)):
- ngram_index = []
- for n in ngrams:
- ngram_index.append(cand_indexes[idx:idx + n])
- ngram_indexes.append(ngram_index)
-
- np_rng.shuffle(ngram_indexes)
-
- (masked_lms, masked_spans) = ([], [])
- covered_indexes = set()
- for cand_index_set in ngram_indexes:
- if len(masked_lms) >= num_to_predict:
- break
- if not cand_index_set:
- continue
- # Note(mingdachen):
- # Skip current piece if they are covered in lm masking or previous ngrams.
- for index_set in cand_index_set[0]:
- for index in index_set:
- if index in covered_indexes:
- continue
-
- if not geometric_dist:
- n = np_rng.choice(ngrams[:len(cand_index_set)],
- p=pvals[:len(cand_index_set)] /
- pvals[:len(cand_index_set)].sum(keepdims=True))
- else:
- # Sampling "n" from the geometric distribution and clipping it to
- # the max_ngrams. Using p=0.2 default from the SpanBERT paper
- # https://arxiv.org/pdf/1907.10529.pdf (Sec 3.1)
- n = min(np_rng.geometric(0.2), max_ngrams)
-
- index_set = sum(cand_index_set[n - 1], [])
- n -= 1
- # Note(mingdachen):
- # Repeatedly looking for a candidate that does not exceed the
- # maximum number of predictions by trying shorter ngrams.
- while len(masked_lms) + len(index_set) > num_to_predict:
- if n == 0:
- break
- index_set = sum(cand_index_set[n - 1], [])
- n -= 1
- # If adding a whole-word mask would exceed the maximum number of
- # predictions, then just skip this candidate.
- if len(masked_lms) + len(index_set) > num_to_predict:
- continue
- is_any_index_covered = False
- for index in index_set:
- if index in covered_indexes:
- is_any_index_covered = True
- break
- if is_any_index_covered:
- continue
- for index in index_set:
- covered_indexes.add(index)
- masked_token = None
- if masking_style == "bert":
- # 80% of the time, replace with [MASK]
- if np_rng.random() < 0.8:
- masked_token = mask_id
- else:
- # 10% of the time, keep original
- if np_rng.random() < 0.5:
- masked_token = tokens[index]
- # 10% of the time, replace with random word
- else:
- masked_token = vocab_id_list[np_rng.randint(0, len(vocab_id_list))]
- elif masking_style == "t5":
- masked_token = mask_id
- else:
- raise ValueError("invalid value of masking style")
-
- output_tokens[index] = masked_token
- masked_lms.append(MaskedLmInstance(index=index, label=tokens[index]))
-
- masked_spans.append(MaskedLmInstance(
- index=index_set,
- label=[tokens[index] for index in index_set]))
-
- assert len(masked_lms) <= num_to_predict
- np_rng.shuffle(ngram_indexes)
-
- select_indexes = set()
- if do_permutation:
- for cand_index_set in ngram_indexes:
- if len(select_indexes) >= num_to_predict:
- break
- if not cand_index_set:
- continue
- # Note(mingdachen):
- # Skip current piece if they are covered in lm masking or previous ngrams.
- for index_set in cand_index_set[0]:
- for index in index_set:
- if index in covered_indexes or index in select_indexes:
- continue
-
- n = np.random.choice(ngrams[:len(cand_index_set)],
- p=pvals[:len(cand_index_set)] /
- pvals[:len(cand_index_set)].sum(keepdims=True))
- index_set = sum(cand_index_set[n - 1], [])
- n -= 1
-
- while len(select_indexes) + len(index_set) > num_to_predict:
- if n == 0:
- break
- index_set = sum(cand_index_set[n - 1], [])
- n -= 1
- # If adding a whole-word mask would exceed the maximum number of
- # predictions, then just skip this candidate.
- if len(select_indexes) + len(index_set) > num_to_predict:
- continue
- is_any_index_covered = False
- for index in index_set:
- if index in covered_indexes or index in select_indexes:
- is_any_index_covered = True
- break
- if is_any_index_covered:
- continue
- for index in index_set:
- select_indexes.add(index)
- assert len(select_indexes) <= num_to_predict
-
- select_indexes = sorted(select_indexes)
- permute_indexes = list(select_indexes)
- np_rng.shuffle(permute_indexes)
- orig_token = list(output_tokens)
-
- for src_i, tgt_i in zip(select_indexes, permute_indexes):
- output_tokens[src_i] = orig_token[tgt_i]
- masked_lms.append(MaskedLmInstance(index=src_i, label=orig_token[src_i]))
-
- masked_lms = sorted(masked_lms, key=lambda x: x.index)
- # Sort the spans by the index of the first span
- masked_spans = sorted(masked_spans, key=lambda x: x.index[0])
-
- for p in masked_lms:
- masked_lm_positions.append(p.index)
- masked_lm_labels.append(p.label)
- return (output_tokens, masked_lm_positions, masked_lm_labels, token_boundary, masked_spans)
-
-
-def pad_and_convert_to_numpy(tokens, tokentypes, masked_positions,
- masked_labels, pad_id, max_seq_length):
- """Pad sequences and convert them to numpy."""
-
- # Some checks.
- num_tokens = len(tokens)
- padding_length = max_seq_length - num_tokens
- assert padding_length >= 0
- assert len(tokentypes) == num_tokens
- assert len(masked_positions) == len(masked_labels)
-
- # Tokens and token types.
- filler = [pad_id] * padding_length
- tokens_np = np.array(tokens + filler, dtype=np.int64)
- tokentypes_np = np.array(tokentypes + filler, dtype=np.int64)
-
- # Padding mask.
- padding_mask_np = np.array([1] * num_tokens + [0] * padding_length,
- dtype=np.int64)
-
- # Lables and loss mask.
- labels = [-1] * max_seq_length
- loss_mask = [0] * max_seq_length
- for i in range(len(masked_positions)):
- assert masked_positions[i] < num_tokens
- labels[masked_positions[i]] = masked_labels[i]
- loss_mask[masked_positions[i]] = 1
- labels_np = np.array(labels, dtype=np.int64)
- loss_mask_np = np.array(loss_mask, dtype=np.int64)
-
- return tokens_np, tokentypes_np, labels_np, padding_mask_np, loss_mask_np
-
-
-def build_train_valid_test_datasets_with_prefixes(train_valid_test_num_samples,
- max_seq_length,
- seed,
- train_data_prefix=None,
- valid_data_prefix=None,
- test_data_prefix=None,
- binary_head=False,
- max_seq_length_dec=None,
- dataset_type='standard_bert'):
- print_rank_0("Separate data paths provided for train, valid & test.")
-
- train_dataset, valid_dataset, test_dataset = None, None, None
- # Single dataset.
- if train_data_prefix is not None:
- train_dataset = build_dataset("train", train_data_prefix,
- train_valid_test_num_samples[0],
- max_seq_length, seed,
- binary_head, max_seq_length_dec,
- dataset_type=dataset_type)
-
- if valid_data_prefix is not None:
- valid_dataset = build_dataset("valid", valid_data_prefix,
- train_valid_test_num_samples[1],
- max_seq_length, seed, False,
- binary_head, max_seq_length_dec,
- dataset_type=dataset_type)
-
- if test_data_prefix is not None:
- test_dataset = build_dataset("test", test_data_prefix,
- train_valid_test_num_samples[2],
- max_seq_length, seed, False,
- binary_head, max_seq_length_dec,
- dataset_type=dataset_type)
-
- return (train_dataset, valid_dataset, test_dataset)
-
-
-def build_train_valid_test_datasets(data_prefix, splits_string,
- train_valid_test_num_samples,
- max_seq_length, seed,
- binary_head=False,
- max_seq_length_dec=None,
- dataset_type='standard_bert'):
-
- if len(data_prefix) == 1:
- return _build_train_valid_test_datasets(data_prefix[0],
- splits_string,
- train_valid_test_num_samples,
- max_seq_length, seed,
- binary_head,
- max_seq_length_dec,
- dataset_type=dataset_type)
-
- raise NotImplementedError("Blending currently unsupported for non-GPT dataset instances")
-
-
-def _build_train_valid_test_datasets(data_prefix, splits_string,
- train_valid_test_num_samples,
- max_seq_length, seed,
- binary_head,
- max_seq_length_dec,
- dataset_type='standard_bert'):
-
- # Indexed dataset.
- indexed_dataset = get_indexed_dataset_(data_prefix,
- dataset_type)
-
- # Get start and end indices of train/valid/train into doc-idx
- # Note that doc-idx is desinged to be num-docs + 1 so we can
- # easily iterate over it.
- total_num_of_documents = indexed_dataset.document_indices.shape[0] - 1
- splits = get_train_valid_test_split_(splits_string, total_num_of_documents)
-
- # Print stats about the splits.
- print_rank_0(' > dataset split:')
-
- def print_split_stats(name, index):
- print_rank_0(' {}:'.format(name))
- print_rank_0(' document indices in [{}, {}) total of {} '
- 'documents'.format(splits[index], splits[index + 1],
- splits[index + 1] - splits[index]))
- start_index = indexed_dataset.document_indices[splits[index]]
- end_index = indexed_dataset.document_indices[splits[index + 1]]
- print_rank_0(' sentence indices in [{}, {}) total of {} '
- 'sentences'.format(start_index, end_index,
- end_index - start_index))
- print_split_stats('train', 0)
- print_split_stats('validation', 1)
- print_split_stats('test', 2)
-
- def build_split_dataset(index, name):
- dataset = None
- if splits[index + 1] > splits[index]:
- # Get the pointer to the original doc-idx so we can set it later.
- doc_idx_ptr = indexed_dataset.get_document_indices()
- # Slice the doc-idx
- start_index = splits[index]
- # Add +1 so we can index into the dataset to get the upper bound.
- end_index = splits[index + 1] + 1
- # New doc_idx view.
- indexed_dataset.set_document_indices(doc_idx_ptr[start_index:end_index])
-
- dataset = build_dataset(
- name, data_prefix,
- train_valid_test_num_samples[index], max_seq_length,
- seed, binary_head, max_seq_length_dec,
- dataset_type, indexed_dataset)
-
- # Set the original pointer so dataset remains the main dataset.
- indexed_dataset.set_document_indices(doc_idx_ptr)
- # Checks.
- assert indexed_dataset.document_indices[0] == 0
- assert indexed_dataset.document_indices.shape[0] == \
- (total_num_of_documents + 1)
- return dataset
-
- train_dataset = build_split_dataset(0, 'train')
- valid_dataset = build_split_dataset(1, 'valid')
- test_dataset = build_split_dataset(2, 'test')
-
- return (train_dataset, valid_dataset, test_dataset)
-
-
-def build_dataset(name, data_prefix, max_num_samples,
- max_seq_length, seed, binary_head,
- max_seq_length_dec, dataset_type='standard_bert',
- indexed_dataset=None):
-
- from megatron.legacy.data.ict_dataset import ICTDataset
- from megatron.legacy.data.multimodal_dataset import MultiModalDataset
-
- if dataset_type == DSET_TYPE_BERT or dataset_type == DSET_TYPE_T5:
- raise ValueError("The Megatron-LM BERT and T5 datasets are deprecated.")
-
- if dataset_type not in DSET_TYPES:
- raise ValueError("Invalid dataset_type: ", dataset_type)
-
- if indexed_dataset is None:
- indexed_dataset = get_indexed_dataset_(data_prefix,
- dataset_type)
-
- kwargs = dict(
- name=name,
- data_prefix=data_prefix,
- num_epochs=None,
- max_num_samples=max_num_samples,
- max_seq_length=max_seq_length,
- seed=seed,
- )
-
- if dataset_type == DSET_TYPE_ICT:
- args = get_args()
-
- title_dataset = get_indexed_dataset_(
- args.titles_data_path,
- dataset_type)
-
- dataset = ICTDataset(
- block_dataset=indexed_dataset,
- title_dataset=title_dataset,
- query_in_block_prob=args.query_in_block_prob,
- use_one_sent_docs=args.use_one_sent_docs,
- binary_head=binary_head,
- **kwargs
- )
- elif dataset_type == DSET_TYPE_MULTIMODAL:
- args = get_args()
- dataset = MultiModalDataset(
- name=name,
- data_prefix=data_prefix,
- indexed_dataset=indexed_dataset,
- num_samples=max_num_samples,
- seq_length=max_seq_length,
- seed=seed,
- img_h=args.img_h,
- img_w=args.img_w,
- )
- else:
- raise NotImplementedError("Dataset type not fully implemented.")
-
- return dataset
-
-
-def get_indexed_dataset_(data_prefix, dataset_type):
-
- print_rank_0(' > building dataset index ...')
-
- start_time = time.time()
- multimodal = dataset_type == DSET_TYPE_MULTIMODAL
- indexed_dataset = IndexedDataset(data_prefix, multimodal)
- assert indexed_dataset.sequence_lengths.shape[0] == indexed_dataset.document_indices[-1]
- print_rank_0(' > finished creating indexed dataset in {:4f} '
- 'seconds'.format(time.time() - start_time))
-
- print_rank_0(' > indexed dataset stats:')
- print_rank_0(' number of documents: {}'.format(
- indexed_dataset.document_indices.shape[0] - 1))
- print_rank_0(' number of sentences: {}'.format(
- indexed_dataset.sequence_lengths.shape[0]))
-
- return indexed_dataset
-
-
-def get_train_valid_test_split_(splits_string, size):
- """ Get dataset splits from comma or '/' separated string list."""
-
- splits = []
- if splits_string.find(',') != -1:
- splits = [float(s) for s in splits_string.split(',')]
- elif splits_string.find('/') != -1:
- splits = [float(s) for s in splits_string.split('/')]
- else:
- splits = [float(splits_string)]
- while len(splits) < 3:
- splits.append(0.)
- splits = splits[:3]
- splits_sum = sum(splits)
- assert splits_sum > 0.0
- splits = [split / splits_sum for split in splits]
- splits_index = [0]
- for index, split in enumerate(splits):
- splits_index.append(splits_index[index] +
- int(round(split * float(size))))
- diff = splits_index[-1] - size
- for index in range(1, len(splits_index)):
- splits_index[index] -= diff
- assert len(splits_index) == 4
- assert splits_index[-1] == size
- return splits_index
-
-def get_samples_mapping(indexed_dataset,
- data_prefix,
- num_epochs,
- max_num_samples,
- max_seq_length,
- short_seq_prob,
- seed,
- name,
- binary_head):
- """Get a list that maps a sample index to a starting sentence index, end sentence index, and length"""
-
- if not num_epochs:
- if not max_num_samples:
- raise ValueError("Need to specify either max_num_samples "
- "or num_epochs")
- num_epochs = np.iinfo(np.int32).max - 1
- if not max_num_samples:
- max_num_samples = np.iinfo(np.int64).max - 1
-
- # Filename of the index mapping
- indexmap_filename = data_prefix
- indexmap_filename += '_{}_indexmap'.format(name)
- if num_epochs != (np.iinfo(np.int32).max - 1):
- indexmap_filename += '_{}ep'.format(num_epochs)
- if max_num_samples != (np.iinfo(np.int64).max - 1):
- indexmap_filename += '_{}mns'.format(max_num_samples)
- indexmap_filename += '_{}msl'.format(max_seq_length)
- indexmap_filename += '_{:0.2f}ssp'.format(short_seq_prob)
- indexmap_filename += '_{}s'.format(seed)
- indexmap_filename += '.npy'
-
- # Build the indexed mapping if not exist.
- if torch.distributed.get_rank() == 0 and \
- not os.path.isfile(indexmap_filename):
- print(' > WARNING: could not find index map file {}, building '
- 'the indices on rank 0 ...'.format(indexmap_filename))
-
- # Make sure the types match the helpers input types.
- assert indexed_dataset.document_indices.dtype == np.int64
- assert indexed_dataset.sequence_lengths.dtype == np.int32
-
- # Build samples mapping
- verbose = torch.distributed.get_rank() == 0
- start_time = time.time()
- print_rank_0(' > building samples index mapping for {} ...'.format(
- name))
- # First compile and then import.
- from megatron.core.datasets import helpers
- samples_mapping = helpers.build_mapping(
- indexed_dataset.document_indices,
- indexed_dataset.sequence_lengths,
- num_epochs,
- max_num_samples,
- max_seq_length,
- short_seq_prob,
- seed,
- verbose,
- 2 if binary_head else 1)
- print_rank_0(' > done building samples index maping')
- np.save(indexmap_filename, samples_mapping, allow_pickle=True)
- print_rank_0(' > saved the index mapping in {}'.format(
- indexmap_filename))
- # Make sure all the ranks have built the mapping
- print_rank_0(' > elasped time to build and save samples mapping '
- '(seconds): {:4f}'.format(
- time.time() - start_time))
- # This should be a barrier but nccl barrier assumes
- # device_index=rank which is not the case for model
- # parallel case
- counts = torch.tensor([1], dtype=torch.long, device='cuda')
- torch.distributed.all_reduce(counts, group=mpu.get_data_parallel_group())
- torch.distributed.all_reduce(counts, group=mpu.get_pipeline_model_parallel_group())
- assert counts[0].item() == (
- torch.distributed.get_world_size() //
- mpu.get_tensor_model_parallel_group().size())
-
- # Load indexed dataset.
- print_rank_0(' > loading indexed mapping from {}'.format(
- indexmap_filename))
- start_time = time.time()
- samples_mapping = np.load(indexmap_filename, allow_pickle=True, mmap_mode='r')
- print_rank_0(' loaded indexed file in {:3.3f} seconds'.format(
- time.time() - start_time))
- print_rank_0(' total number of samples: {}'.format(
- samples_mapping.shape[0]))
-
- return samples_mapping
diff --git a/megatron/legacy/data/ict_dataset.py b/megatron/legacy/data/ict_dataset.py
deleted file mode 100644
index 9af552d6367..00000000000
--- a/megatron/legacy/data/ict_dataset.py
+++ /dev/null
@@ -1,157 +0,0 @@
-# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
-import itertools
-import random
-
-import numpy as np
-from torch.utils.data import Dataset
-
-from megatron.training import get_tokenizer
-from megatron.training import get_args
-from megatron.legacy.data.dataset_utils import get_indexed_dataset_
-from megatron.legacy.data.realm_dataset_utils import get_block_samples_mapping
-
-def make_attention_mask(source_block, target_block):
- """
- Returns a 2-dimensional (2-D) attention mask
- :param source_block: 1-D array
- :param target_block: 1-D array
- """
- mask = (target_block[None, :] >= 1) * (source_block[:, None] >= 1)
- mask = mask.astype(np.int64)
- # (source_length, target_length)
- return mask
-
-def get_ict_dataset(use_titles=True, query_in_block_prob=1):
- """Get a dataset which uses block samples mappings to get ICT/block indexing data (via get_block())
- rather than for training, since it is only built with a single epoch sample mapping.
- """
- args = get_args()
- block_dataset = get_indexed_dataset_(args.data_path, 'mmap', True)
- titles_dataset = get_indexed_dataset_(args.titles_data_path, 'mmap', True)
-
- kwargs = dict(
- name='full',
- block_dataset=block_dataset,
- title_dataset=titles_dataset,
- data_prefix=args.data_path,
- num_epochs=1,
- max_num_samples=None,
- max_seq_length=args.seq_length,
- seed=1,
- query_in_block_prob=query_in_block_prob,
- use_titles=use_titles,
- use_one_sent_docs=args.use_one_sent_docs
- )
- dataset = ICTDataset(**kwargs)
- return dataset
-
-
-class ICTDataset(Dataset):
- """Dataset containing sentences and their blocks for an inverse cloze task."""
- def __init__(self, name, block_dataset, title_dataset, data_prefix,
- num_epochs, max_num_samples, max_seq_length, query_in_block_prob,
- seed, use_titles=True, use_one_sent_docs=False, binary_head=False):
- self.name = name
- self.seed = seed
- self.max_seq_length = max_seq_length
- self.query_in_block_prob = query_in_block_prob
- self.block_dataset = block_dataset
- self.title_dataset = title_dataset
- self.rng = random.Random(self.seed)
- self.use_titles = use_titles
- self.use_one_sent_docs = use_one_sent_docs
-
- self.samples_mapping = get_block_samples_mapping(
- block_dataset, title_dataset, data_prefix, num_epochs,
- max_num_samples, max_seq_length, seed, name, use_one_sent_docs)
- self.tokenizer = get_tokenizer()
- self.vocab_id_list = list(self.tokenizer.inv_vocab.keys())
- self.vocab_id_to_token_list = self.tokenizer.inv_vocab
- self.cls_id = self.tokenizer.cls
- self.sep_id = self.tokenizer.sep
- self.mask_id = self.tokenizer.mask
- self.pad_id = self.tokenizer.pad
-
- def __len__(self):
- return len(self.samples_mapping)
-
- def __getitem__(self, idx):
- """Get an ICT example of a pseudo-query and the block of text from which it was extracted"""
- sample_data = self.samples_mapping[idx]
- start_idx, end_idx, doc_idx, block_idx = sample_data.as_tuple()
-
- if self.use_titles:
- title = self.title_dataset[int(doc_idx)]
- title_pad_offset = 3 + len(title)
- else:
- title = None
- title_pad_offset = 2
- block = [self.block_dataset[i] for i in range(start_idx, end_idx)]
- assert len(block) > 1 or self.use_one_sent_docs or self.query_in_block_prob == 1
-
- # randint() is inclusive for Python rng
- rand_sent_idx = self.rng.randint(0, len(block) - 1)
-
- # keep the query in the context query_in_block_prob fraction of the time.
- if self.rng.random() < self.query_in_block_prob:
- query = block[rand_sent_idx].copy()
- else:
- query = block.pop(rand_sent_idx)
-
- # still need to truncate because blocks are concluded when
- # the sentence lengths have exceeded max_seq_length.
- query = query[:self.max_seq_length - 2]
- block = list(itertools.chain(*block))[:self.max_seq_length - title_pad_offset]
-
- query_tokens, query_pad_mask = self.concat_and_pad_tokens(query)
- context_tokens, context_pad_mask = self.concat_and_pad_tokens(block, title)
-
- query_mask = make_attention_mask(query_tokens, query_tokens)
- context_mask = make_attention_mask(context_tokens, context_tokens)
-
- block_data = sample_data.as_array()
-
- sample = {
- 'query_tokens': query_tokens,
- 'query_mask': query_mask,
- 'query_pad_mask': query_pad_mask,
- 'context_tokens': context_tokens,
- 'context_mask': context_mask,
- 'context_pad_mask': context_pad_mask,
- 'block_data': block_data,
- }
-
- return sample
-
- def get_block(self, start_idx, end_idx, doc_idx):
- """Get the IDs for an evidence block plus the title of the corresponding document"""
- block = [self.block_dataset[i] for i in range(start_idx, end_idx)]
- title = self.title_dataset[int(doc_idx)]
-
- block = list(itertools.chain(*block))[:self.max_seq_length - (3 + len(title))]
- block_tokens, block_pad_mask = self.concat_and_pad_tokens(block, title)
-
- return block_tokens, block_pad_mask
-
- def get_null_block(self):
- """Get empty block and title - used in REALM pretraining"""
- block, title = [], []
- block_tokens, block_pad_mask = self.concat_and_pad_tokens(block, title)
-
- return block_tokens, block_pad_mask
-
- def concat_and_pad_tokens(self, tokens, title=None):
- """Concat with special tokens and pad sequence to self.max_seq_length"""
- tokens = list(tokens)
- if title is None:
- tokens = [self.cls_id] + tokens + [self.sep_id]
- else:
- title = list(title)
- tokens = [self.cls_id] + title + [self.sep_id] + tokens + [self.sep_id]
- assert len(tokens) <= self.max_seq_length
-
- num_pad = self.max_seq_length - len(tokens)
- pad_mask = [1] * len(tokens) + [0] * num_pad
- tokens += [self.pad_id] * num_pad
-
- return np.array(tokens), np.array(pad_mask)
diff --git a/megatron/legacy/data/image_folder.py b/megatron/legacy/data/image_folder.py
deleted file mode 100644
index de15b29bf06..00000000000
--- a/megatron/legacy/data/image_folder.py
+++ /dev/null
@@ -1,302 +0,0 @@
-# BSD 3-Clause License
-#
-# Copyright (c) Soumith Chintala 2016,
-# All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are met:
-#
-# * Redistributions of source code must retain the above copyright notice, this
-# list of conditions and the following disclaimer.
-#
-# * Redistributions in binary form must reproduce the above copyright notice,
-# this list of conditions and the following disclaimer in the documentation
-# and/or other materials provided with the distribution.
-#
-# * Neither the name of the copyright holder nor the names of its
-# contributors may be used to endorse or promote products derived from
-# this software without specific prior written permission.
-
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
-# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
-# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
-# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
-# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-# code taken from
-# https://github.com/pytorch/vision/blob/main/torchvision/datasets/folder.py
-# added support for classes_fraction and data_per_class_fraction
-
-from torchvision.datasets import VisionDataset
-from PIL import Image
-
-import os
-import os.path
-from typing import Any, Callable, cast, Dict, List, Optional, Tuple
-import numpy as np
-
-def has_file_allowed_extension(filename: str, extensions: Tuple[str, ...]) -> bool:
- """Checks if a file is an allowed extension.
- Args:
- filename (string): path to a file
- extensions (tuple of strings): extensions to consider (lowercase)
- Returns:
- bool: True if the filename ends with one of given extensions
- """
- return filename.lower().endswith(extensions)
-
-
-def is_image_file(filename: str) -> bool:
- """Checks if a file is an allowed image extension.
- Args:
- filename (string): path to a file
- Returns:
- bool: True if the filename ends with a known image extension
- """
- return has_file_allowed_extension(filename, IMG_EXTENSIONS)
-
-
-def make_dataset(
- directory: str,
- class_to_idx: Dict[str, int],
- data_per_class_fraction: float,
- extensions: Optional[Tuple[str, ...]] = None,
- is_valid_file: Optional[Callable[[str], bool]] = None,
-) -> List[Tuple[str, int]]:
- """Generates a list of samples of a form (path_to_sample, class).
- Args:
- directory (str): root dataset directory
- class_to_idx (Dict[str, int]): dictionary mapping class name to class index
- extensions (optional): A list of allowed extensions.
- Either extensions or is_valid_file should be passed. Defaults to None.
- is_valid_file (optional): A function that takes path of a file
- and checks if the file is a valid file
- (used to check of corrupt files) both extensions and
- is_valid_file should not be passed. Defaults to None.
- Raises:
- ValueError: In case ``extensions`` and ``is_valid_file`` are None or both are not None.
- Returns:
- List[Tuple[str, int]]: samples of a form (path_to_sample, class)
- """
- instances = []
- directory = os.path.expanduser(directory)
- both_none = extensions is None and is_valid_file is None
- both_something = extensions is not None and is_valid_file is not None
- if both_none or both_something:
- raise ValueError("Both extensions and is_valid_file cannot be None or not None at the same time")
- if extensions is not None:
- def is_valid_file(x: str) -> bool:
- return has_file_allowed_extension(x, cast(Tuple[str, ...], extensions))
- is_valid_file = cast(Callable[[str], bool], is_valid_file)
- for target_class in sorted(class_to_idx.keys()):
- class_index = class_to_idx[target_class]
- target_dir = os.path.join(directory, target_class)
- if not os.path.isdir(target_dir):
- continue
- local_instances = []
- for root, _, fnames in sorted(os.walk(target_dir, followlinks=True)):
- for fname in sorted(fnames):
- path = os.path.join(root, fname)
- if is_valid_file(path):
- item = path, class_index
- local_instances.append(item)
-
- instances.extend(local_instances[0:int(len(local_instances) * data_per_class_fraction)])
-
- return instances
-
-
-class DatasetFolder(VisionDataset):
- """A generic data loader where the samples are arranged in this way: ::
- root/class_x/xxx.ext
- root/class_x/xxy.ext
- root/class_x/[...]/xxz.ext
- root/class_y/123.ext
- root/class_y/nsdf3.ext
- root/class_y/[...]/asd932_.ext
- Args:
- root (string): Root directory path.
- loader (callable): A function to load a sample given its path.
- extensions (tuple[string]): A list of allowed extensions.
- both extensions and is_valid_file should not be passed.
- transform (callable, optional): A function/transform that takes in
- a sample and returns a transformed version.
- E.g, ``transforms.RandomCrop`` for images.
- target_transform (callable, optional): A function/transform that takes
- in the target and transforms it.
- is_valid_file (callable, optional): A function that takes path of a file
- and check if the file is a valid file (used to check of corrupt files)
- both extensions and is_valid_file should not be passed.
- Attributes:
- classes (list): List of the class names sorted alphabetically.
- class_to_idx (dict): Dict with items (class_name, class_index).
- samples (list): List of (sample path, class_index) tuples
- targets (list): The class_index value for each image in the dataset
- """
-
- def __init__(
- self,
- root: str,
- loader: Callable[[str], Any],
- extensions: Optional[Tuple[str, ...]] = None,
- transform: Optional[Callable] = None,
- target_transform: Optional[Callable] = None,
- classes_fraction=1.0,
- data_per_class_fraction=1.0,
- is_valid_file: Optional[Callable[[str], bool]] = None,
- ) -> None:
- super(DatasetFolder, self).__init__(root, transform=transform,
- target_transform=target_transform)
- self.classes_fraction = classes_fraction
- self.data_per_class_fraction = data_per_class_fraction
- classes, class_to_idx = self._find_classes(self.root)
- samples = self.make_dataset(self.root,
- class_to_idx,
- self.data_per_class_fraction,
- extensions,
- is_valid_file)
- if len(samples) == 0:
- msg = "Found 0 files in subfolders of: {}\n".format(self.root)
- if extensions is not None:
- msg += "Supported extensions are: {}".format(",".join(extensions))
- raise RuntimeError(msg)
-
- self.loader = loader
- self.extensions = extensions
- self.total = len(samples)
- self.classes = classes
- self.class_to_idx = class_to_idx
- self.samples = samples
- self.targets = [s[1] for s in samples]
-
- @staticmethod
- def make_dataset(
- directory: str,
- class_to_idx: Dict[str, int],
- data_per_class_fraction: float,
- extensions: Optional[Tuple[str, ...]] = None,
- is_valid_file: Optional[Callable[[str], bool]] = None,
- ) -> List[Tuple[str, int]]:
- return make_dataset(directory,
- class_to_idx,
- data_per_class_fraction,
- extensions=extensions,
- is_valid_file=is_valid_file)
-
- def _find_classes(self, dir: str) -> Tuple[List[str], Dict[str, int]]:
- """
- Finds the class folders in a dataset.
- Args:
- dir (string): Root directory path.
- Returns:
- tuple: (classes, class_to_idx) where classes are relative to (dir), and class_to_idx is a dictionary.
- Ensures:
- No class is a subdirectory of another.
- """
- all_classes = [d.name for d in os.scandir(dir) if d.is_dir()]
- classes = all_classes[0:int(len(all_classes) * self.classes_fraction)]
- classes.sort()
- class_to_idx = {cls_name: i for i, cls_name in enumerate(classes)}
- return classes, class_to_idx
-
- def __getitem__(self, index: int) -> Tuple[Any, Any]:
- """
- Args:
- index (int): Index
- Returns:
- tuple: (sample, target) where target is class_index of the target class.
- """
- curr_index = index
- for x in range(self.total):
- try:
- path, target = self.samples[curr_index]
- sample = self.loader(path)
- break
- except Exception as e:
- curr_index = np.random.randint(0, self.total)
-
- if self.transform is not None:
- sample = self.transform(sample)
- if self.target_transform is not None:
- target = self.target_transform(target)
-
- return sample, target
-
- def __len__(self) -> int:
- return len(self.samples)
-
-
-IMG_EXTENSIONS = ('.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm', '.tif', '.tiff', '.webp')
-
-
-def pil_loader(path: str) -> Image.Image:
- # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
- with open(path, 'rb') as f:
- img = Image.open(f)
- return img.convert('RGB')
-
-
-# TODO: specify the return type
-def accimage_loader(path: str) -> Any:
- import accimage
- try:
- return accimage.Image(path)
- except IOError:
- # Potentially a decoding problem, fall back to PIL.Image
- return pil_loader(path)
-
-
-def default_loader(path: str) -> Any:
- from torchvision import get_image_backend
- if get_image_backend() == 'accimage':
- return accimage_loader(path)
- else:
- return pil_loader(path)
-
-
-class ImageFolder(DatasetFolder):
- """A generic data loader where the images are arranged in this way: ::
- root/dog/xxx.png
- root/dog/xxy.png
- root/dog/[...]/xxz.png
- root/cat/123.png
- root/cat/nsdf3.png
- root/cat/[...]/asd932_.png
- Args:
- root (string): Root directory path.
- transform (callable, optional): A function/transform that takes in an PIL image
- and returns a transformed version. E.g, ``transforms.RandomCrop``
- target_transform (callable, optional): A function/transform that takes in the
- target and transforms it.
- loader (callable, optional): A function to load an image given its path.
- is_valid_file (callable, optional): A function that takes path of an Image file
- and check if the file is a valid file (used to check of corrupt files)
- Attributes:
- classes (list): List of the class names sorted alphabetically.
- class_to_idx (dict): Dict with items (class_name, class_index).
- imgs (list): List of (image path, class_index) tuples
- """
-
- def __init__(
- self,
- root: str,
- transform: Optional[Callable] = None,
- target_transform: Optional[Callable] = None,
- classes_fraction=1.0,
- data_per_class_fraction=1.0,
- loader: Callable[[str], Any] = default_loader,
- is_valid_file: Optional[Callable[[str], bool]] = None,
- ):
- super(ImageFolder, self).__init__(root, loader, IMG_EXTENSIONS if is_valid_file is None else None,
- transform=transform,
- target_transform=target_transform,
- classes_fraction=classes_fraction,
- data_per_class_fraction=data_per_class_fraction,
- is_valid_file=is_valid_file)
- self.imgs = self.samples
-
diff --git a/megatron/legacy/data/multimodal_dataset.py b/megatron/legacy/data/multimodal_dataset.py
deleted file mode 100644
index 93ea790329c..00000000000
--- a/megatron/legacy/data/multimodal_dataset.py
+++ /dev/null
@@ -1,54 +0,0 @@
-# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
-
-from PIL import Image, UnidentifiedImageError
-import numpy as np
-import io
-import torch
-
-try:
- from torchvision.transforms import InterpolationMode
- BICUBIC = InterpolationMode.BICUBIC
-except ImportError:
- BICUBIC = Image.BICUBIC
-
-from torchvision.transforms import Compose, ToTensor, Normalize, ToPILImage, RandomResizedCrop, Resize
-
-def _convert_image_to_rgb(image):
- return image.convert("RGB")
-
-def _transform(img_h, img_w):
- return Compose([
- ToPILImage(),
- RandomResizedCrop((img_h, img_w), scale=(0.5, 1.0), interpolation=BICUBIC),
- _convert_image_to_rgb,
- ToTensor(),
- Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711)),
- ])
-
-class MultiModalDataset(torch.utils.data.Dataset):
-
- def __init__(self, name, data_prefix, indexed_dataset,
- num_samples, seq_length, seed, img_h, img_w):
-
- self.name = name
- self.indexed_dataset = indexed_dataset
- self.doc_idx = indexed_dataset.get_document_indices()
- self.visual_transform = _transform(img_h, img_w)
-
- def __len__(self):
- return self.indexed_dataset.sequence_lengths.shape[0]
-
- def __getitem__(self, idx):
- text_sample, mode = self.indexed_dataset.get(self.doc_idx[idx])
- assert mode == 0
- img_sample, mode = self.indexed_dataset.get(self.doc_idx[idx]+1)
- assert mode == 1
- img_pad = img_sample[0].item()
- xs = img_sample[1:].tobytes(order='C')
- xs = xs[:len(xs)-img_pad]
-
- img_sample = np.array(Image.open(io.BytesIO(xs)))
- img_sample = self.visual_transform(img_sample).reshape(-1)
-
- return {'text': np.array(text_sample, dtype=np.int64),
- 'img': np.array(img_sample, dtype=np.float32)}
diff --git a/megatron/legacy/data/orqa_wiki_dataset.py b/megatron/legacy/data/orqa_wiki_dataset.py
deleted file mode 100644
index 033b2351cee..00000000000
--- a/megatron/legacy/data/orqa_wiki_dataset.py
+++ /dev/null
@@ -1,192 +0,0 @@
-# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
-
-"""Wikipedia dataset from DPR code for ORQA."""
-
-from abc import ABC
-import csv
-import numpy as np
-import random
-import torch
-from torch.utils.data import Dataset
-
-from megatron.training import print_rank_0, get_args, get_tokenizer
-from megatron.core import tensor_parallel
-from megatron.legacy.data.biencoder_dataset_utils import make_attention_mask
-
-def get_open_retrieval_wiki_dataset():
- args = get_args()
- tokenizer = get_tokenizer()
-
- dataset = OpenRetrievalEvidenceDataset('2018 Wikipedia from DPR codebase',
- 'evidence',
- args.evidence_data_path,
- tokenizer)
- return dataset
-
-
-def get_open_retrieval_batch(data_iterator):
- # Items and their type.
- keys = ['row_id', 'context', 'context_mask', 'context_types',
- 'context_pad_mask']
- datatype = torch.int64
-
- # Broadcast data.
- data = None if data_iterator is None else next(data_iterator)
- data_b = tensor_parallel.broadcast_data(keys, data, datatype)
-
- # Unpack.
- row_id = data_b['row_id'].long()
- context = data_b['context'].long()
-
- # TODO: make the context mask a binary one
- context_mask = (data_b['context_mask'] < 0.5)
-
- context_types = data_b['context_types'].long()
- context_pad_mask = data_b['context_pad_mask'].long()
-
- return row_id, context, context_mask, context_types, context_pad_mask
-
-
-def build_tokens_types_paddings_from_text(row, tokenizer, max_seq_length):
- """Build token types and paddings, trim if needed, and pad if needed."""
-
- title_ids = tokenizer.tokenize(row['title'])
- context_ids = tokenizer.tokenize(row['text'])
-
- # Appending the title of the context at front
- extended_context_ids = title_ids + [tokenizer.sep] + context_ids
-
- context_ids, context_types, context_pad_mask = \
- build_tokens_types_paddings_from_ids(extended_context_ids,
- max_seq_length, tokenizer.cls, tokenizer.sep, tokenizer.pad)
-
- return context_ids, context_types, context_pad_mask
-
-
-# noinspection DuplicatedCode
-def build_tokens_types_paddings_from_ids(text_ids, max_seq_length,
- cls_id, sep_id, pad_id):
- """Build token types and paddings, trim if needed, and pad if needed."""
- enc_ids = []
- tokentypes_enc = []
-
- # [CLS].
- enc_ids.append(cls_id)
- tokentypes_enc.append(0)
-
- # A.
- len_src = len(text_ids)
- enc_ids.extend(text_ids)
- tokentypes_enc.extend([0] * len_src)
-
- # Cap the size.
- if len(enc_ids) > max_seq_length - 1:
- enc_ids = enc_ids[0: max_seq_length - 1]
- tokentypes_enc = tokentypes_enc[0: max_seq_length - 1]
-
- # [SEP].
- enc_ids.append(sep_id)
- tokentypes_enc.append(0)
-
- num_tokens_enc = len(enc_ids)
- # Padding.
- padding_length = max_seq_length - len(enc_ids)
- if padding_length > 0:
- enc_ids.extend([pad_id] * padding_length)
- tokentypes_enc.extend([pad_id] * padding_length)
-
- pad_mask = ([1] * num_tokens_enc) + ([0] * padding_length)
- pad_mask = np.array(pad_mask, dtype=np.int64)
-
- return enc_ids, tokentypes_enc, pad_mask
-
-
-def build_sample(row_id, context_ids, context_types, context_pad_mask):
- """Convert to numpy and return a sample consumed by the batch producer."""
-
- context_ids = np.array(context_ids, dtype=np.int64)
- context_types = np.array(context_types, dtype=np.int64)
- context_mask = make_attention_mask(context_ids, context_ids)
-
- sample = ({
- 'row_id': row_id,
- 'context': context_ids,
- 'context_mask': context_mask,
- 'context_types': context_types,
- 'context_pad_mask': context_pad_mask
- })
- return sample
-
-
-class OpenRetrievalEvidenceDataset(ABC, Dataset):
- """Open Retrieval Evidence dataset class."""
-
- def __init__(self, task_name, dataset_name, datapath, tokenizer,
- max_seq_length):
- # Store inputs.
- self.task_name = task_name
- self.dataset_name = dataset_name
- self.tokenizer = tokenizer
- self.max_seq_length = max_seq_length
- print_rank_0(' > building {} dataset for {}:'.format(self.task_name,
- self.dataset_name))
- # Process the files.
- print_rank_0(datapath)
- self.samples, self.id2text = self.process_samples_from_single_path(
- datapath)
-
- args = get_args()
- if args.sample_rate < 1: # subsample
- k = int(len(self.samples) * args.sample_rate)
- self.samples = random.sample(self.samples, k)
-
- print_rank_0(' >> total number of samples: {}'.format(
- len(self.samples)))
-
- def __len__(self):
- return len(self.samples)
-
- def __getitem__(self, idx):
- row = self.samples[idx]
-
- context_ids, context_types, context_pad_mask = \
- build_tokens_types_paddings_from_text(row, self.tokenizer,
- self.max_seq_length)
-
- sample = build_sample(row['doc_id'],
- context_ids,
- context_types,
- context_pad_mask)
- return sample
-
- @staticmethod
- def process_samples_from_single_path(filename):
- print_rank_0(' > Processing {} ...'.format(filename))
- total = 0
-
- rows = []
- id2text = {}
-
- with open(filename) as tsvfile:
- reader = csv.reader(tsvfile, delimiter='\t')
- next(reader, None) # skip the headers
- for row in reader:
- # file format: doc_id, doc_text, title
- doc_id = int(row[0])
- text = row[1]
- title = row[2]
-
- rows.append({'doc_id': doc_id,
- 'text': text,
- 'title': title})
-
- assert doc_id not in id2text
- id2text[doc_id] = (text, title)
-
- total += 1
- if total % 100000 == 0:
- print_rank_0(' > processed {} rows so far ...'.format(
- total))
-
- print_rank_0(' >> processed {} samples.'.format(len(rows)))
- return rows, id2text
diff --git a/megatron/legacy/data/realm_dataset_utils.py b/megatron/legacy/data/realm_dataset_utils.py
deleted file mode 100644
index 70fa11a20c3..00000000000
--- a/megatron/legacy/data/realm_dataset_utils.py
+++ /dev/null
@@ -1,199 +0,0 @@
-# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
-import os
-import time
-
-import numpy as np
-import torch
-
-from megatron.training import print_rank_0
-from megatron.core import mpu, tensor_parallel
-from megatron.legacy.data.dataset_utils import create_masked_lm_predictions, pad_and_convert_to_numpy
-from megatron.training import get_args, get_tokenizer, print_rank_0
-
-
-def get_one_epoch_dataloader(dataset, micro_batch_size=None):
- """Specifically one epoch to be used in an indexing job."""
- args = get_args()
-
- world_size = mpu.get_data_parallel_world_size()
- rank = mpu.get_data_parallel_rank()
- if micro_batch_size is None:
- micro_batch_size = args.micro_batch_size
- global_batch_size = micro_batch_size * world_size
- num_workers = args.num_workers
-
- sampler = torch.utils.data.SequentialSampler(dataset)
- # importantly, drop_last must be False to get all the data.
- assert False, 'DistributedBatchSampler deprecated, change the implementation'
- from megatron.legacy.data.samplers import DistributedBatchSampler
- batch_sampler = DistributedBatchSampler(sampler,
- batch_size=global_batch_size,
- drop_last=False,
- rank=rank,
- world_size=world_size)
-
- return torch.utils.data.DataLoader(dataset,
- batch_sampler=batch_sampler,
- num_workers=num_workers,
- pin_memory=True)
-
-
-def get_ict_batch(data_iterator):
- # Items and their type.
- keys = ['query_tokens', 'query_pad_mask',
- 'block_tokens', 'block_pad_mask', 'block_data']
- datatype = torch.int64
-
- # Broadcast data.
- if data_iterator is None:
- data = None
- else:
- data = next(data_iterator)
- data_b = tensor_parallel.broadcast_data(keys, data, datatype)
-
- # Unpack.
- query_tokens = data_b['query_tokens'].long()
- query_pad_mask = data_b['query_pad_mask'].long()
- block_tokens = data_b['block_tokens'].long()
- block_pad_mask = data_b['block_pad_mask'].long()
- block_indices = data_b['block_data'].long()
-
- return query_tokens, query_pad_mask,\
- block_tokens, block_pad_mask, block_indices
-
-
-def join_str_list(str_list):
- """Join a list of strings, handling spaces appropriately"""
- result = ""
- for s in str_list:
- if s.startswith("##"):
- result += s[2:]
- else:
- result += " " + s
- return result
-
-
-class BlockSampleData(object):
- """A struct for fully describing a fixed-size block of data as used in REALM
-
- :param start_idx: for first sentence of the block
- :param end_idx: for last sentence of the block (may be partially truncated in sample construction)
- :param doc_idx: the index of the document from which the block comes in the original indexed dataset
- :param block_idx: a unique integer identifier given to every block.
- """
- def __init__(self, start_idx, end_idx, doc_idx, block_idx):
- self.start_idx = start_idx
- self.end_idx = end_idx
- self.doc_idx = doc_idx
- self.block_idx = block_idx
-
- def as_array(self):
- return np.array([self.start_idx, self.end_idx, self.doc_idx, self.block_idx]).astype(np.int64)
-
- def as_tuple(self):
- return self.start_idx, self.end_idx, self.doc_idx, self.block_idx
-
-
-class BlockSamplesMapping(object):
- def __init__(self, mapping_array):
- # make sure that the array is compatible with BlockSampleData
- assert mapping_array.shape[1] == 4
- self.mapping_array = mapping_array
-
- def __len__(self):
- return self.mapping_array.shape[0]
-
- def __getitem__(self, idx):
- """Get the data associated with an indexed sample."""
- sample_data = BlockSampleData(*self.mapping_array[idx])
- return sample_data
-
-
-def get_block_samples_mapping(block_dataset, title_dataset, data_prefix, num_epochs,
- max_num_samples, max_seq_length, seed, name, use_one_sent_docs=False):
- """Get samples mapping for a dataset over fixed size blocks. This function also requires
- a dataset of the titles for the source documents since their lengths must be taken into account.
-
- :return: samples_mapping (BlockSamplesMapping)
- """
-
- if not num_epochs:
- if not max_num_samples:
- raise ValueError("Need to specify either max_num_samples "
- "or num_epochs")
- num_epochs = np.iinfo(np.int32).max - 1
- if not max_num_samples:
- max_num_samples = np.iinfo(np.int64).max - 1
-
- # Filename of the index mapping
- indexmap_filename = data_prefix
- indexmap_filename += '_{}_indexmap'.format(name)
- if num_epochs != (np.iinfo(np.int32).max - 1):
- indexmap_filename += '_{}ep'.format(num_epochs)
- if max_num_samples != (np.iinfo(np.int64).max - 1):
- indexmap_filename += '_{}mns'.format(max_num_samples)
- indexmap_filename += '_{}msl'.format(max_seq_length)
- indexmap_filename += '_{}s'.format(seed)
- if use_one_sent_docs:
- indexmap_filename += '_1sentok'
- indexmap_filename += '.npy'
-
- # Build the indexed mapping if not exist.
- if mpu.get_data_parallel_rank() == 0 and \
- not os.path.isfile(indexmap_filename):
- print(' > WARNING: could not find index map file {}, building '
- 'the indices on rank 0 ...'.format(indexmap_filename))
-
- # Make sure the types match the helpers input types.
- assert block_dataset.document_indices.dtype == np.int64
- assert block_dataset.sequence_lengths.dtype == np.int32
-
- # Build samples mapping
- verbose = torch.distributed.get_rank() == 0
- start_time = time.time()
- print_rank_0(' > building samples index mapping for {} ...'.format(
- name))
-
- from megatron.core.datasets import helpers
- mapping_array = helpers.build_blocks_mapping(
- block_dataset.document_indices,
- block_dataset.sequence_lengths,
- title_dataset.sequence_lengths,
- num_epochs,
- max_num_samples,
- max_seq_length - 3, # account for added tokens
- seed,
- verbose,
- use_one_sent_docs)
-
-
- print_rank_0(' > done building samples index mapping')
- np.save(indexmap_filename, mapping_array, allow_pickle=True)
- print_rank_0(' > saved the index mapping in {}'.format(
- indexmap_filename))
- # Make sure all the ranks have built the mapping
- print_rank_0(' > elapsed time to build and save samples mapping '
- '(seconds): {:4f}'.format(
- time.time() - start_time))
-
- # This should be a barrier but nccl barrier assumes
- # device_index=rank which is not the case for model
- # parallel case
- counts = torch.tensor([1], dtype=torch.long, device='cuda')
- torch.distributed.all_reduce(counts, group=mpu.get_data_parallel_group())
- assert counts[0].item() == mpu.get_data_parallel_group().size()
-
- # Load indexed dataset.
- print_rank_0(' > loading indexed mapping from {}'.format(
- indexmap_filename))
- start_time = time.time()
-
- mapping_array = np.load(indexmap_filename, allow_pickle=True, mmap_mode='r')
- samples_mapping = BlockSamplesMapping(mapping_array)
-
- print_rank_0(' loaded indexed file in {:3.3f} seconds'.format(
- time.time() - start_time))
- print_rank_0(' total number of samples: {}'.format(
- mapping_array.shape[0]))
-
- return samples_mapping
diff --git a/megatron/legacy/data/realm_index.py b/megatron/legacy/data/realm_index.py
deleted file mode 100644
index dbe924a52ae..00000000000
--- a/megatron/legacy/data/realm_index.py
+++ /dev/null
@@ -1,225 +0,0 @@
-# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
-import itertools
-import os
-import pickle
-import shutil
-
-import numpy as np
-import torch
-
-from megatron.training import get_args
-from megatron.core import mpu
-
-
-def detach(tensor):
- return tensor.detach().cpu().numpy()
-
-
-class OpenRetreivalDataStore(object):
- """
- Serializable data structure for holding data for blocks --
- embeddings and necessary metadata for Retriever
- """
- def __init__(self, embedding_path=None, load_from_path=True, rank=None):
- self.embed_data = dict()
- if embedding_path is None:
- args = get_args()
- embedding_path = args.embedding_path
- rank = args.rank
- self.embedding_path = embedding_path
- self.rank = rank
-
- if load_from_path:
- self.load_from_file()
-
- block_data_name = os.path.splitext(self.embedding_path)[0]
- self.temp_dir_name = block_data_name + '_tmp'
-
- def state(self):
- return {
- 'embed_data': self.embed_data,
- }
-
- def clear(self):
- """
- Clear the embedding data structures to save memory.
- The metadata ends up getting used, and is also much smaller in
- dimensionality so it isn't really worth clearing.
- """
- self.embed_data = dict()
-
- def load_from_file(self):
- """Populate members from instance saved to file"""
-
- if not mpu.model_parallel_is_initialized() or mpu.get_data_parallel_rank() == 0:
- print("\n> Unpickling BlockData", flush=True)
- state_dict = pickle.load(open(self.embedding_path, 'rb'))
- if not mpu.model_parallel_is_initialized() or mpu.get_data_parallel_rank() == 0:
- print(">> Finished unpickling BlockData\n", flush=True)
-
- self.embed_data = state_dict['embed_data']
-
- def add_block_data(self, row_id, block_embeds, allow_overwrite=False):
- """
- Add data for set of blocks
- :param row_id: 1D array of unique int ids for the blocks
- :param block_embeds: 2D array of embeddings of the blocks
- In the case of retriever this will be [start_idx, end_idx, doc_idx]
- """
- for idx, embed in zip(row_id, block_embeds):
- if not allow_overwrite and idx in self.embed_data:
- raise ValueError("Unexpectedly tried to overwrite block data")
-
- self.embed_data[idx] = np.float16(embed)
-
- def save_shard(self):
- """
- Save the block data that was created this in this process
- """
- if not os.path.isdir(self.temp_dir_name):
- os.makedirs(self.temp_dir_name, exist_ok=True)
-
- # save the data for each shard
- with open('{}/{}.pkl'.format(self.temp_dir_name, self.rank), 'wb') \
- as writer:
- pickle.dump(self.state(), writer)
-
- def merge_shards_and_save(self):
- #Combine all the shards made using save_shard
- shard_names = os.listdir(self.temp_dir_name)
- seen_own_shard = False
-
- for fname in os.listdir(self.temp_dir_name):
- shard_rank = int(os.path.splitext(fname)[0])
- if shard_rank == self.rank:
- seen_own_shard = True
- continue
-
- with open('{}/{}'.format(self.temp_dir_name, fname), 'rb') as f:
- data = pickle.load(f)
- old_size = len(self.embed_data)
- shard_size = len(data['embed_data'])
-
- # add the shard's data and check to make sure there
- # is no overlap
- self.embed_data.update(data['embed_data'])
- assert len(self.embed_data) == old_size + shard_size
-
- assert seen_own_shard
-
- # save the consolidated shards and remove temporary directory
- with open(self.embedding_path, 'wb') as final_file:
- pickle.dump(self.state(), final_file)
- shutil.rmtree(self.temp_dir_name, ignore_errors=True)
-
- print("Finished merging {} shards for a total of {} embeds".format(
- len(shard_names), len(self.embed_data)), flush=True)
-
-
-class FaissMIPSIndex(object):
- """
- Wrapper object for a BlockData which similarity search via FAISS under the hood
- """
- def __init__(self, embed_size, embed_data=None, use_gpu=False):
- self.embed_size = embed_size
- self.embed_data = embed_data
- self.use_gpu = use_gpu
-
- self.mips_index = None
- self._set_mips_index()
-
- def _set_mips_index(self):
- """
- Create a Faiss Flat index with inner product as the metric
- to search against
- """
- try:
- import faiss
- except ImportError:
- raise Exception("Error: Please install faiss to use FaissMIPSIndex")
-
- if not mpu.model_parallel_is_initialized() or mpu.get_data_parallel_rank() == 0:
- print("\n> Building index", flush=True)
-
- cpu_index = faiss.IndexFlatIP(self.embed_size)
-
- if self.use_gpu:
- # create resources and config for GpuIndex
- config = faiss.GpuMultipleClonerOptions()
- config.shard = True
- config.useFloat16 = True
- gpu_index = faiss.index_cpu_to_all_gpus(cpu_index, co=config)
- self.mips_index = faiss.IndexIDMap(gpu_index)
- if not mpu.model_parallel_is_initialized() or mpu.get_data_parallel_rank() == 0:
- print(">> Initialized index on GPU", flush=True)
- else:
- # CPU index supports IDs so wrap with IDMap
- self.mips_index = faiss.IndexIDMap(cpu_index)
- if not mpu.model_parallel_is_initialized() or mpu.get_data_parallel_rank() == 0:
- print(">> Initialized index on CPU", flush=True)
-
- # if we were constructed with a BlockData, then automatically load it
- # when the FAISS structure is built
- if self.embed_data is not None:
- self.add_embed_data(self.embed_data)
-
- def reset_index(self):
- """Delete existing index and create a new"""
- del self.mips_index
-
- # reset the block data so that _set_block_index will reload it as well
- if self.embed_data is not None:
- embed_data_path = self.embed_data.embedding_path
- del self.embed_data
- self.embed_data = OpenRetreivalDataStore(embed_data_path)
-
- self._set_mips_index()
-
- def update_index(self):
- """Delete existing index and create a new"""
- del self.mips_index
-
- # reset the block data so that _set_mips_index will reload it as well
- if self.embed_data is not None:
- self.embed_data.load_from_file()
- self._set_mips_index()
-
- def add_embed_data(self, all_embed_data):
- """Add the embedding of each block to the underlying FAISS index"""
-
- # this assumes the embed_data is a dict : {int: np.array}
- block_indices, block_embeds = zip(*all_embed_data.embed_data.items())
-
- # the embeddings have to be entered in as float32 even though the math
- # internally is done with float16.
- embeds_arr = np.float32(np.array(block_embeds))
- indices_arr = np.array(block_indices)
-
- # we no longer need the embedding data since it's in the index now
- all_embed_data.clear()
-
- self.mips_index.add_with_ids(embeds_arr, indices_arr)
-
- if not mpu.model_parallel_is_initialized() or mpu.get_data_parallel_rank() == 0:
- print(">>> Finished adding block data to index", flush=True)
-
- def search_mips_index(self, query_embeds, top_k, reconstruct=True):
- """
- Get the top-k blocks by the index distance metric.
-
- :param reconstruct: if True: return a [num_queries x k x embed_dim]
- array of blocks
- if False: return [num_queries x k] array of
- distances, and another for indices
- """
- query_embeds = np.float32(detach(query_embeds))
-
- if reconstruct:
- # get the vectors themselves
- top_k_block_embeds = self.mips_index.search_and_reconstruct(\
- query_embeds, top_k)
- return top_k_block_embeds
- else:
- # get distances and indices of closest vectors
- distances, block_indices = self.mips_index.search(query_embeds, top_k)
- return distances, block_indices
diff --git a/megatron/legacy/data/vit_dataset.py b/megatron/legacy/data/vit_dataset.py
deleted file mode 100644
index 504075a5506..00000000000
--- a/megatron/legacy/data/vit_dataset.py
+++ /dev/null
@@ -1,251 +0,0 @@
-# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
-import os
-import random
-
-import numpy as np
-import torch
-import torchvision.transforms as T
-from PIL import Image, ImageFilter, ImageOps
-from torchvision import datasets
-
-from megatron.legacy.data.autoaugment import ImageNetPolicy
-from megatron.legacy.data.image_folder import ImageFolder
-from megatron.training import get_args
-from megatron.training.datasets.data_samplers import RandomSeedDataset
-
-
-class GaussianBlur(object):
- """
- Apply Gaussian Blur to the PIL image.
- """
- def __init__(self, p=0.5, radius_min=0.1, radius_max=2.):
- self.prob = p
- self.radius_min = radius_min
- self.radius_max = radius_max
-
- def __call__(self, img):
- do_it = random.random() <= self.prob
- if not do_it:
- return img
-
- return img.filter(
- ImageFilter.GaussianBlur(
- radius=random.uniform(self.radius_min, self.radius_max)
- )
- )
-
-
-class Solarization(object):
- """
- Apply Solarization to the PIL image.
- """
- def __init__(self, p):
- self.p = p
-
- def __call__(self, img):
- if random.random() < self.p:
- return ImageOps.solarize(img)
- else:
- return img
-
-
-class ClassificationTransform():
- def __init__(self, image_size, train=True):
- args = get_args()
- assert args.fp16 or args.bf16
- self.data_type = torch.half if args.fp16 else torch.bfloat16
- if train:
- self.transform = T.Compose([
- T.RandomResizedCrop(image_size),
- T.RandomHorizontalFlip(),
- T.ColorJitter(0.4, 0.4, 0.4, 0.1),
- ImageNetPolicy(),
- T.ToTensor(),
- T.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
- T.ConvertImageDtype(self.data_type)
- ])
- else:
- self.transform = T.Compose([
- T.Resize(image_size),
- T.CenterCrop(image_size),
- T.ToTensor(),
- T.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
- T.ConvertImageDtype(self.data_type)
- ])
-
- def __call__(self, input):
- output = self.transform(input)
- return output
-
-
-class InpaintingTransform():
- def __init__(self, image_size, train=True):
-
- args = get_args()
- self.mask_factor = args.mask_factor
- self.mask_type = args.mask_type
- self.image_size = image_size
- self.patch_size = args.patch_dim
- self.mask_size = int(self.mask_factor*(image_size[0]/self.patch_size)*(image_size[1]/self.patch_size))
- self.train = train
- assert args.fp16 or args.bf16
- self.data_type = torch.half if args.fp16 else torch.bfloat16
-
- if self.train:
- self.transform = T.Compose([
- T.RandomResizedCrop(self.image_size),
- T.RandomHorizontalFlip(),
- T.ColorJitter(0.4, 0.4, 0.4, 0.1),
- ImageNetPolicy(),
- T.ToTensor(),
- T.ConvertImageDtype(self.data_type)
- ])
- else:
- self.transform = T.Compose([
- T.Resize(self.image_size, interpolation=2),
- T.CenterCrop(self.image_size),
- T.ToTensor(),
- T.ConvertImageDtype(self.data_type)
- ])
-
- def gen_mask(self, image_size, mask_size, mask_type, patch_size):
- # output: mask as a list with indices for missing patches
- action_list = [[0, 1], [0, -1], [1, 0], [-1, 0]]
- assert image_size[0] == image_size[1]
- img_size_patch = image_size[0] // patch_size
-
- # drop masked patches
- mask = torch.zeros((image_size[0], image_size[1]), dtype=torch.float)
-
- if mask_type == 'random':
- x = torch.randint(0, img_size_patch, ())
- y = torch.randint(0, img_size_patch, ())
- for i in range(mask_size):
- r = torch.randint(0, len(action_list), ())
- x = torch.clamp(x + action_list[r][0], min=0, max=img_size_patch - 1)
- y = torch.clamp(y + action_list[r][1], min=0, max=img_size_patch - 1)
- x_offset = x * patch_size
- y_offset = y * patch_size
- mask[x_offset:x_offset+patch_size, y_offset:y_offset+patch_size] = 1
- else:
- assert mask_type == 'row'
- count = 0
- for x in reversed(range(img_size_patch)):
- for y in reversed(range(img_size_patch)):
- if (count < mask_size):
- count += 1
- x_offset = x * patch_size
- y_offset = y * patch_size
- mask[x_offset:x_offset+patch_size, y_offset:y_offset+patch_size] = 1
- return mask
-
- def __call__(self, input):
- trans_input = self.transform(input)
- mask = self.gen_mask(self.image_size, self.mask_size,
- self.mask_type, self.patch_size)
- mask = mask.unsqueeze(dim=0)
- return trans_input, mask
-
-
-class DinoTransform(object):
- def __init__(self, image_size, train=True):
- args = get_args()
- self.data_type = torch.half if args.fp16 else torch.bfloat16
-
- flip_and_color_jitter = T.Compose([
- T.RandomHorizontalFlip(p=0.5),
- T.RandomApply(
- [T.ColorJitter(brightness=0.4, contrast=0.4,
- saturation=0.2, hue=0.1)],
- p=0.8
- ),
- T.RandomGrayscale(p=0.2),
- ])
-
- if args.fp16 or args.bf16:
- normalize = T.Compose([
- T.ToTensor(),
- T.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
- T.ConvertImageDtype(self.data_type)
- ])
- else:
- normalize = T.Compose([
- T.ToTensor(),
- T.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
- ])
-
- # first global crop
- scale_const = 0.4
- self.global_transform1 = T.Compose([
- T.RandomResizedCrop(image_size,
- scale=(scale_const, 1),
- interpolation=Image.BICUBIC),
- flip_and_color_jitter,
- GaussianBlur(1.0),
- normalize
- ])
- # second global crop
- self.global_transform2 = T.Compose([
- T.RandomResizedCrop(image_size,
- scale=(scale_const, 1),
- interpolation=Image.BICUBIC),
- flip_and_color_jitter,
- GaussianBlur(0.1),
- Solarization(0.2),
- normalize
- ])
- # transformation for the local small crops
- self.local_crops_number = args.dino_local_crops_number
- self.local_transform = T.Compose([
- T.RandomResizedCrop(args.dino_local_img_size,
- scale=(0.05, scale_const),
- interpolation=Image.BICUBIC),
- flip_and_color_jitter,
- GaussianBlur(p=0.5),
- normalize
- ])
-
- def __call__(self, image):
- crops = []
- crops.append(self.global_transform1(image))
- crops.append(self.global_transform2(image))
- for _ in range(self.local_crops_number):
- crops.append(self.local_transform(image))
- return crops
-
-
-def build_train_valid_datasets(data_path, image_size=224):
- args = get_args()
-
- if args.vision_pretraining_type == 'classify':
- train_transform = ClassificationTransform(image_size)
- val_transform = ClassificationTransform(image_size, train=False)
- elif args.vision_pretraining_type == 'inpaint':
- train_transform = InpaintingTransform(image_size, train=False)
- val_transform = InpaintingTransform(image_size, train=False)
- elif args.vision_pretraining_type == 'dino':
- train_transform = DinoTransform(image_size, train=True)
- val_transform = ClassificationTransform(image_size, train=False)
- else:
- raise Exception('{} vit pretraining type is not supported.'.format(
- args.vit_pretraining_type))
-
- # training dataset
- train_data_path = data_path[0] if len(data_path) <= 2 else data_path[2]
- train_data = ImageFolder(
- root=train_data_path,
- transform=train_transform,
- classes_fraction=args.classes_fraction,
- data_per_class_fraction=args.data_per_class_fraction
- )
- train_data = RandomSeedDataset(train_data, args.seed)
-
- # validation dataset
- val_data_path = data_path[1]
- val_data = ImageFolder(
- root=val_data_path,
- transform=val_transform
- )
- val_data = RandomSeedDataset(val_data, args.seed)
-
- return train_data, val_data
diff --git a/megatron/legacy/fused_kernels/__init__.py b/megatron/legacy/fused_kernels/__init__.py
deleted file mode 100755
index f01088fd5a5..00000000000
--- a/megatron/legacy/fused_kernels/__init__.py
+++ /dev/null
@@ -1,77 +0,0 @@
-# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
-
-import os
-import pathlib
-import subprocess
-import torch
-from torch.utils import cpp_extension
-
-
-# Setting this param to a list has a problem of generating different
-# compilation commands (with diferent order of architectures) and
-# leading to recompilation of fused kernels. Set it to empty string
-# to avoid recompilation and assign arch flags explicity in
-# extra_cuda_cflags below
-os.environ["TORCH_CUDA_ARCH_LIST"] = ""
-
-
-def load(args):
-
- if torch.cuda.is_available() and torch.version.cuda:
- # Check if cuda 11 is installed for compute capability 8.0
- cc_flag = []
- _, bare_metal_major, bare_metal_minor = _get_cuda_bare_metal_version(
- cpp_extension.CUDA_HOME
- )
- if int(bare_metal_major) >= 11:
- cc_flag.append('-gencode')
- cc_flag.append('arch=compute_80,code=sm_80')
- if int(bare_metal_minor) >= 8:
- cc_flag.append('-gencode')
- cc_flag.append('arch=compute_90,code=sm_90')
-
- # Build path
- srcpath = pathlib.Path(__file__).parent.absolute()
- buildpath = srcpath / "build"
- _create_build_dir(buildpath)
-
- # Helper function to build the kernels.
- def _cpp_extention_load_helper(name, sources, extra_cuda_flags):
- return cpp_extension.load(
- name=name,
- sources=sources,
- build_directory=buildpath,
- extra_cflags=[
- "-O3",
- ],
- extra_cuda_cflags=[
- "-O3",
- "-gencode",
- "arch=compute_70,code=sm_70",
- "--use_fast_math",
- ]
- + extra_cuda_flags
- + cc_flag,
- verbose=(args.rank == 0),
- )
-
-
-def _get_cuda_bare_metal_version(cuda_dir):
- raw_output = subprocess.check_output(
- [cuda_dir + "/bin/nvcc", "-V"], universal_newlines=True
- )
- output = raw_output.split()
- release_idx = output.index("release") + 1
- release = output[release_idx].split(".")
- bare_metal_major = release[0]
- bare_metal_minor = release[1][0]
-
- return raw_output, bare_metal_major, bare_metal_minor
-
-
-def _create_build_dir(buildpath):
- try:
- os.mkdir(buildpath)
- except OSError:
- if not os.path.isdir(buildpath):
- print(f"Creation of the build directory {buildpath} failed")
diff --git a/megatron/legacy/fused_kernels/compat.h b/megatron/legacy/fused_kernels/compat.h
deleted file mode 100644
index 5495d780776..00000000000
--- a/megatron/legacy/fused_kernels/compat.h
+++ /dev/null
@@ -1,17 +0,0 @@
-/* Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved. */
-
-/*This code is copied fron NVIDIA apex:
- * https://github.com/NVIDIA/apex
- * with minor changes. */
-
-
-
-#ifndef TORCH_CHECK
-#define TORCH_CHECK AT_CHECK
-#endif
-
-#ifdef VERSION_GE_1_3
-#define DATA_PTR data_ptr
-#else
-#define DATA_PTR data
-#endif
diff --git a/megatron/legacy/fused_kernels/tests/__init__.py b/megatron/legacy/fused_kernels/tests/__init__.py
deleted file mode 100644
index e69de29bb2d..00000000000
diff --git a/megatron/legacy/fused_kernels/tests/test_fused_kernels.py b/megatron/legacy/fused_kernels/tests/test_fused_kernels.py
deleted file mode 100644
index f5b2b78a3f7..00000000000
--- a/megatron/legacy/fused_kernels/tests/test_fused_kernels.py
+++ /dev/null
@@ -1,389 +0,0 @@
-# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
-import math
-
-import torch
-from torch.nn import LayerNorm
-
-from megatron.legacy.model.enums import AttnMaskType
-from megatron.legacy.model.fused_layer_norm import MixedFusedLayerNorm
-from megatron.legacy.model.fused_softmax import FusedScaleMaskSoftmax
-from megatron.legacy.model.utils import attention_mask_func
-from megatron.legacy.fused_kernels import load
-
-def test_load_fused_kernels():
- try:
- import fused_layer_norm_cuda
- import scaled_masked_softmax_cuda
- import scaled_upper_triang_masked_softmax_cuda
- import torch
-
- print("[Success] load_fused_kernels")
- except ImportError as e:
- print("[Fail] load_fused_kernels")
- raise e
-
-def test_fused_softmax():
- bert = BertModel.from_pretrained("bert-base-cased").cuda().half()
- tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
- test_text = (
- "Hello. How are you? I am fine thank you and you? yes Good. "
- "hi hi hi hi hi hi hi hi hi hi hi hi hi" # 32
- )
-
- tokens = tokenizer(
- [test_text] * 4,
- return_tensors="pt",
- )
-
- embedding_output = bert.embeddings(
- input_ids=tokens["input_ids"].cuda(),
- position_ids=None,
- token_type_ids=tokens["token_type_ids"].cuda(),
- inputs_embeds=None,
- past_key_values_length=0,
- )
-
- # (bsz, 1, 1, seq_len)
- mask = bert.get_extended_attention_mask(
- attention_mask=tokens["attention_mask"].cuda(),
- input_shape=tokens["input_ids"].shape,
- device=bert.device,
- )
- # (bsz, 1, seq_len, seq_len)
- mask = mask.repeat(1, 1, mask.size()[-1], 1)
-
- attention = bert.encoder.layer[0].attention.self
- key_layer = attention.transpose_for_scores(attention.key(embedding_output))
- query_layer = attention.transpose_for_scores(attention.query(embedding_output))
-
- attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))
- attention_scores /= math.sqrt(key_layer.size()[-1])
-
- fused_softmax = (
- FusedScaleMaskSoftmax(
- input_in_fp16=True,
- input_in_bf16=False,
- mask_func=attention_mask_func,
- scale=None,
- softmax_in_fp32=False,
- attn_mask_type=AttnMaskType.padding,
- scaled_masked_softmax_fusion=True,
- )
- .cuda()
- .half()
- )
-
- fused_softmax_output = fused_softmax(
- attention_scores,
- (mask != 0),
- )
-
- torch_softmax = (
- FusedScaleMaskSoftmax(
- input_in_fp16=True,
- input_in_bf16=False,
- mask_func=attention_mask_func,
- scale=None,
- softmax_in_fp32=False,
- attn_mask_type=AttnMaskType.padding,
- scaled_masked_softmax_fusion=False,
- )
- .cuda()
- .half()
- )
-
- torch_softmax_output = torch_softmax(
- attention_scores,
- (mask != 0),
- )
-
- test_result = (fused_softmax_output - torch_softmax_output).abs()
-
- while test_result.dim() != 1:
- test_result = test_result.mean(dim=-1)
-
- diff = test_result.mean(dim=-1)
-
- if diff <= 1e-3:
- print(
- f"\n[Success] test_fused_softmax"
- f"\n > mean_difference={diff}"
- f"\n > fused_values={fused_softmax_output[-1][-1][-1][:5].tolist()}"
- f"\n > torch_values={torch_softmax_output[-1][-1][-1][:5].tolist()}"
- )
- else:
- print(
- f"\n[Fail] test_fused_softmax"
- f"\n > mean_difference={diff}, "
- f"\n > fused_values={fused_softmax_output[-1][-1][-1][:5].tolist()}, "
- f"\n > torch_values={torch_softmax_output[-1][-1][-1][:5].tolist()}"
- )
-
-
-def test_fused_upper_triangle_mask_softmax():
- gpt = GPT2Model.from_pretrained("gpt2").cuda().half()
- tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
- test_text = (
- "Hello. How are you? I am fine thank you and you? yes Good. "
- "hi hi hi hi hi hi hi" # 24
- )
-
- tokens = tokenizer(
- [test_text] * 4,
- return_tensors="pt",
- )
-
- attention_mask = tokens["attention_mask"].cuda()
- attention_mask = attention_mask.view(attention_mask.size(0), -1)
- attention_mask = attention_mask[:, None, None, :]
- attention_mask = (1.0 - attention_mask) * -10000.0
- attention_mask = attention_mask.repeat(1, 1, attention_mask.size()[-1], 1)
- attn = gpt.h[0]
-
- hidden_states = gpt.wte(tokens["input_ids"].cuda())
- q, k, v = attn.attn.c_attn(hidden_states).split(768, dim=-1)
- q = attn.attn._split_heads(q, attn.attn.num_heads, attn.attn.head_dim)
- k = attn.attn._split_heads(k, attn.attn.num_heads, attn.attn.head_dim)
- attn_weights = torch.matmul(q, k.transpose(-1, -2))
-
- sq, sk = q.size(-2), k.size(-2)
- causal_mask = attn.attn.bias[:, :, sk - sq : sk, :sk].bool()
- total_mask = ~(causal_mask & (attention_mask == 0))
- """
- tensor([[[[False, True, True, ..., True, True, True],
- [False, False, True, ..., True, True, True],
- [False, False, False, ..., True, True, True],
- ...,
- [False, False, False, ..., False, True, True],
- [False, False, False, ..., False, False, True],
- [False, False, False, ..., False, False, False]]]
- """
-
- fused_softmax = (
- FusedScaleMaskSoftmax(
- input_in_fp16=True,
- input_in_bf16=False,
- mask_func=attention_mask_func,
- scale=None,
- softmax_in_fp32=False,
- attn_mask_type=AttnMaskType.causal,
- scaled_masked_softmax_fusion=True,
- )
- .cuda()
- .half()
- )
-
- fused_softmax_output = fused_softmax(
- attn_weights,
- total_mask,
- )
-
- torch_softmax = (
- FusedScaleMaskSoftmax(
- input_in_fp16=True,
- input_in_bf16=False,
- mask_func=attention_mask_func,
- scale=None,
- softmax_in_fp32=False,
- attn_mask_type=AttnMaskType.causal,
- scaled_masked_softmax_fusion=False,
- )
- .cuda()
- .half()
- )
-
- torch_softmax_output = torch_softmax(
- attn_weights,
- total_mask,
- )
-
- test_result = (fused_softmax_output - torch_softmax_output).abs()
-
- while test_result.dim() != 1:
- test_result = test_result.mean(dim=-1)
-
- diff = test_result.mean(dim=-1)
-
- if diff <= 1e-3:
- print(
- f"\n[Success] test_fused_upper_triangle_mask_softmax"
- f"\n > mean_difference={diff}"
- f"\n > fused_values={fused_softmax_output[-1][-1][-1][:5].tolist()}"
- f"\n > torch_values={torch_softmax_output[-1][-1][-1][:5].tolist()}"
- )
- else:
- print(
- f"\n[Fail] test_fused_upper_triangle_mask_softmax"
- f"\n > mean_difference={diff}, "
- f"\n > fused_values={fused_softmax_output[-1][-1][-1][:5].tolist()}, "
- f"\n > torch_values={torch_softmax_output[-1][-1][-1][:5].tolist()}"
- )
-
-
-def test_layer_norm():
- bert = BertModel.from_pretrained("bert-base-cased").cuda().half()
- tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
- test_text = (
- "Hello. How are you? I am fine thank you and you? yes Good. "
- "hi hi hi hi hi hi hi hi hi hi hi hi hi" # 32
- )
-
- tokens = tokenizer(
- [test_text] * 4,
- return_tensors="pt",
- )
-
- # [bsz, seq_len, d_model]
- embedding_output = (
- bert.embeddings(
- input_ids=tokens["input_ids"].cuda(),
- position_ids=None,
- token_type_ids=tokens["token_type_ids"].cuda(),
- inputs_embeds=None,
- past_key_values_length=0,
- )
- .cuda()
- .half()
- )
-
- fused_layernorm_layer = (
- MixedFusedLayerNorm(normalized_shape=embedding_output.size(-1)).cuda().half()
- )
-
- torch_layernorm_layer = (
- LayerNorm(normalized_shape=embedding_output.size(-1)).cuda().half()
- )
-
- fused_output = fused_layernorm_layer(embedding_output)
- torch_output = torch_layernorm_layer(embedding_output)
- test_result = (fused_output - torch_output).abs()
-
- while test_result.dim() != 1:
- test_result = test_result.mean(dim=-1)
-
- diff = test_result.mean(dim=-1)
-
- if diff <= 1e-3:
- print(
- f"\n[Success] test_layer_norm"
- f"\n > mean_difference={diff}"
- f"\n > fused_values={fused_output[-1][-1][:5].tolist()}"
- f"\n > torch_values={torch_output[-1][-1][:5].tolist()}"
- )
- else:
- print(
- f"\n[Fail] test_layer_norm"
- f"\n > mean_difference={diff}, "
- f"\n > fused_values={fused_output[-1][-1][:5].tolist()}, "
- f"\n > torch_values={torch_output[-1][-1][:5].tolist()}"
- )
-
-
-def attention_mask_func(attention_scores, attention_mask):
- attention_scores.masked_fill_(attention_mask, -10000.0)
- return attention_scores
-
-
-def forward_torch_softmax(input, mask, scale):
- input = input * scale
- mask_output = attention_mask_func(input, mask) if mask is not None else input
- probs = torch.nn.Softmax(dim=-1)(mask_output)
- return probs
-
-
-def test_masked_softmax_forward():
- import scaled_masked_softmax_cuda
-
- batch = 2
- attn = 16
- scale_t = torch.tensor([1.0])
- for qlen in [128, 256, 1024, 2048, 4096]:
- for klen in [128, 256, 1024, 2048]:
- inputs = torch.normal(0, 2, (batch, attn, qlen, klen), dtype=torch.float16, device='cuda:0')
- masks = torch.randint(0, 2, (batch, 1, qlen, klen), dtype=torch.bool, device='cuda:0')
- softmax_results = scaled_masked_softmax_cuda.forward(inputs, masks, scale_t[0].item())
- softmax_results_torch = forward_torch_softmax(inputs, masks, scale_t[0].item())
- error = (softmax_results_torch - softmax_results).abs().max()
- assert error < 1e-3
-
-def test_masked_softmax_backward():
- import scaled_masked_softmax_cuda
-
- batch = 2
- attn = 16
- scale_t = torch.tensor([1.0])
- for qlen in [128, 256, 1024, 2048, 4096]:
- for klen in [128, 256, 1024, 2048]:
- inputs = torch.normal(0, 2, (batch, attn, qlen, klen), dtype=torch.float16, device='cuda:0')
- backward = torch.rand_like(inputs, dtype=torch.float16, device='cuda:0')
- masks = torch.randint(0, 2, (batch, 1, qlen, klen), dtype=torch.bool, device='cuda:0')
- softmax_results = scaled_masked_softmax_cuda.forward(inputs, masks, scale_t[0].item())
- back_grad = scaled_masked_softmax_cuda.backward(backward, softmax_results, scale_t[0].item())
-
- inputs.requires_grad = True
- softmax_results_torch = forward_torch_softmax(inputs, masks, scale_t[0].item())
- softmax_results_torch.backward(backward)
- error = (back_grad - inputs.grad).abs().max()
- assert error < 1e-3
-
-
-def test_allmasked_softmax_forward():
- import scaled_masked_softmax_cuda
-
- batch = 2
- attn = 16
- scale_t = torch.tensor([1.0])
- for qlen in [128, 256, 1024, 2048, 4096]:
- for klen in [128, 256, 1024, 2048]:
- inputs = torch.normal(0, 2, (batch, attn, qlen, klen), dtype=torch.float16, device='cuda:0')
- masks = torch.ones((batch, 1, qlen, klen), dtype=torch.bool, device='cuda:0')
- softmax_results = scaled_masked_softmax_cuda.forward(inputs, masks, scale_t[0].item())
- softmax_results_torch = torch.zeros_like(inputs)
- error = (softmax_results_torch - softmax_results).abs().max()
- assert error == 0.0
-
-
-def test_allmasked_softmax_backward():
- import scaled_masked_softmax_cuda
-
- batch = 2
- attn = 16
- scale_t = torch.tensor([1.0])
- for qlen in [128, 256, 1024, 2048, 4096]:
- for klen in [128, 256, 1024, 2048]:
- inputs = torch.normal(0, 2, (batch, attn, qlen, klen), dtype=torch.float16, device='cuda:0')
- backward = torch.rand_like(inputs, dtype=torch.float16, device='cuda:0')
- masks = torch.ones((batch, 1, qlen, klen), dtype=torch.bool, device='cuda:0')
- softmax_results = scaled_masked_softmax_cuda.forward(inputs, masks, scale_t[0].item())
- back_grad = scaled_masked_softmax_cuda.backward(backward, softmax_results, scale_t[0].item())
- inputs.requires_grad = True
- softmax_results_torch = forward_torch_softmax(inputs, masks, scale_t[0].item())
- softmax_results_torch.backward(backward)
- error = (back_grad - inputs.grad).abs().max()
- assert error < 1e-3
-
-
-if __name__ == "__main__":
- try:
- from transformers import BertTokenizer, GPT2Tokenizer
- from transformers.models.bert.modeling_bert import BertModel
- from transformers.models.gpt2.modeling_gpt2 import GPT2Model
- import transformers
-
- transformers.logging.set_verbosity(
- transformers.logging.FATAL,
- )
-
- except ImportError:
- print("\n[Fail] Please install `transformers` package to test fused kernels\n")
- exit(-1)
-
- load()
- test_masked_softmax_forward()
- test_masked_softmax_backward()
- test_allmasked_softmax_forward()
- test_allmasked_softmax_backward()
- test_load_fused_kernels()
- test_fused_softmax()
- test_fused_upper_triangle_mask_softmax()
- test_layer_norm()
diff --git a/megatron/legacy/fused_kernels/type_shim.h b/megatron/legacy/fused_kernels/type_shim.h
deleted file mode 100644
index d60a6f8c6fb..00000000000
--- a/megatron/legacy/fused_kernels/type_shim.h
+++ /dev/null
@@ -1,103 +0,0 @@
-/* Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved. */
-
-
-#include
-#include "compat.h"
-
-
-#define DISPATCH_HALF_AND_BFLOAT(TYPE, NAME, ...) \
- switch(TYPE) \
- { \
- case at::ScalarType::Half: \
- { \
- using scalar_t = at::Half; \
- __VA_ARGS__; \
- break; \
- } \
- case at::ScalarType::BFloat16: \
- { \
- using scalar_t = at::BFloat16; \
- __VA_ARGS__; \
- break; \
- } \
- default: \
- AT_ERROR(#NAME, " not implemented for '", toString(TYPE), "'"); \
- }
-
-
-#define DISPATCH_HALF_BFLOAT_AND_FLOAT(TYPE, NAME, ...) \
- switch(TYPE) \
- { \
- case at::ScalarType::Half: \
- { \
- using scalar_t = at::Half; \
- __VA_ARGS__; \
- break; \
- } \
- case at::ScalarType::BFloat16: \
- { \
- using scalar_t = at::BFloat16; \
- __VA_ARGS__; \
- break; \
- } \
- case at::ScalarType::Float: \
- { \
- using scalar_t = float; \
- __VA_ARGS__; \
- break; \
- } \
- default: \
- AT_ERROR(#NAME, " not implemented for '", toString(TYPE), "'"); \
- }
-
-
-
-#define DISPATCH_FLOAT_HALF_AND_BFLOAT_INOUT_TYPES(TYPEIN, TYPEOUT, NAME, ...) \
- switch(TYPEIN) \
- { \
- case at::ScalarType::Float: \
- { \
- using scalar_t_in = float; \
- switch(TYPEOUT) \
- { \
- case at::ScalarType::Float: \
- { \
- using scalar_t_out = float; \
- __VA_ARGS__; \
- break; \
- } \
- case at::ScalarType::Half: \
- { \
- using scalar_t_out = at::Half; \
- __VA_ARGS__; \
- break; \
- } \
- case at::ScalarType::BFloat16: \
- { \
- using scalar_t_out = at::BFloat16; \
- __VA_ARGS__; \
- break; \
- } \
- default: \
- AT_ERROR(#NAME, " not implemented for '", toString(TYPEOUT), "'"); \
- } \
- break; \
- } \
- case at::ScalarType::Half: \
- { \
- using scalar_t_in = at::Half; \
- using scalar_t_out = at::Half; \
- __VA_ARGS__; \
- break; \
- } \
- case at::ScalarType::BFloat16: \
- { \
- using scalar_t_in = at::BFloat16; \
- using scalar_t_out = at::BFloat16; \
- __VA_ARGS__; \
- break; \
- } \
- default: \
- AT_ERROR(#NAME, " not implemented for '", toString(TYPEIN), "'"); \
- }
-
diff --git a/megatron/legacy/indexer.py b/megatron/legacy/indexer.py
deleted file mode 100644
index 179e00e6cdf..00000000000
--- a/megatron/legacy/indexer.py
+++ /dev/null
@@ -1,130 +0,0 @@
-# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
-import sys
-import time
-import torch
-import torch.distributed as dist
-
-from megatron.training import get_args, print_rank_0
-from megatron.core import mpu
-from megatron.training.checkpointing import load_biencoder_checkpoint
-from megatron.legacy.data.orqa_wiki_dataset import get_open_retrieval_wiki_dataset
-from megatron.legacy.data.orqa_wiki_dataset import get_open_retrieval_batch
-from megatron.legacy.data.biencoder_dataset_utils import get_one_epoch_dataloader
-from megatron.legacy.data.realm_index import detach, OpenRetreivalDataStore
-from megatron.legacy.model.biencoder_model import get_model_provider
-from megatron.training import get_model
-
-
-class IndexBuilder(object):
- """
- Object for taking one pass over a dataset and creating a BlockData of its
- embeddings
- """
- def __init__(self):
- args = get_args()
- self.model = None
- self.dataloader = None
- self.evidence_embedder_obj = None
- self.biencoder_shared_query_context_model = \
- args.biencoder_shared_query_context_model
-
- # need to know whether we're using a REALM checkpoint (args.load)
- # or ICT checkpoint
- assert not (args.load and args.ict_load)
-
- self.log_interval = args.indexer_log_interval
- self.batch_size = args.indexer_batch_size
-
- self.load_attributes()
- self.is_main_builder = mpu.get_data_parallel_rank() == 0
- self.num_total_builders = mpu.get_data_parallel_world_size()
- self.iteration = self.total_processed = 0
-
- def load_attributes(self):
- """
- Load the necessary attributes: model, dataloader and empty BlockData
- """
- only_context_model = True
- if self.biencoder_shared_query_context_model:
- only_context_model = False
-
- model = get_model(get_model_provider(only_context_model=\
- only_context_model, biencoder_shared_query_context_model=\
- self.biencoder_shared_query_context_model))
-
- self.model = load_biencoder_checkpoint(model,
- only_context_model=only_context_model)
-
- assert len(self.model) == 1
- self.model[0].eval()
-
- self.dataset = get_open_retrieval_wiki_dataset()
- self.dataloader = iter(get_one_epoch_dataloader(self.dataset, \
- self.batch_size))
-
- self.evidence_embedder_obj = OpenRetreivalDataStore( \
- load_from_path=False)
-
- def track_and_report_progress(self, batch_size):
- """
- Utility function for tracking progress
- """
- self.iteration += 1
- self.total_processed += batch_size * self.num_total_builders
- if self.is_main_builder and self.iteration % self.log_interval == 0:
- print('Batch {:10d} | Total {:10d}'.format(self.iteration,
- self.total_processed), flush=True)
-
- def build_and_save_index(self):
- """
- Goes through one epoch of the dataloader and adds all data to this
- instance's BlockData.
-
- The copy of BlockData is saved as a shard, which when run in a
- distributed setting will be consolidated by the rank 0 process
- and saved as a final pickled BlockData.
- """
- assert len(self.model) == 1
- unwrapped_model = self.model[0]
-
- while not hasattr(unwrapped_model, 'embed_text'):
- unwrapped_model = unwrapped_model.module
-
- while True:
- try:
- # batch also has query_tokens and query_pad_data
- row_id, context_tokens, context_mask, context_types, \
- context_pad_mask = get_open_retrieval_batch( \
- self.dataloader)
- except (StopIteration, IndexError):
- break
-
- # TODO: can we add with torch.no_grad() to reduce memory usage
- # detach, separate fields and add to BlockData
- assert context_mask.dtype == torch.bool
- context_logits = unwrapped_model.embed_text(
- unwrapped_model.context_model, context_tokens, context_mask,
- context_types)
-
- context_logits = detach(context_logits)
- row_id = detach(row_id)
-
- self.evidence_embedder_obj.add_block_data(row_id, context_logits)
- self.track_and_report_progress(batch_size=len(row_id))
-
- # This process signals to finalize its shard and then synchronize with
- # the other processes
- self.evidence_embedder_obj.save_shard()
- torch.distributed.barrier()
- del self.model
-
- # rank 0 process builds the final copy
- if self.is_main_builder:
- self.evidence_embedder_obj.merge_shards_and_save()
- # make sure that every single piece of data was embedded
- assert len(self.evidence_embedder_obj.embed_data) == \
- len(self.dataset)
- self.evidence_embedder_obj.clear()
-
- # complete building the final copy
- torch.distributed.barrier()
diff --git a/megatron/legacy/model/module.py b/megatron/legacy/model/module.py
index 1659238494f..3a9651df99e 100644
--- a/megatron/legacy/model/module.py
+++ b/megatron/legacy/model/module.py
@@ -15,10 +15,6 @@
_BF16_TYPES = (torch.BFloat16Tensor, torch.cuda.BFloat16Tensor)
-def param_is_not_shared(param):
- return not hasattr(param, 'shared') or not param.shared
-
-
class MegatronModule(torch.nn.Module):
"""Megatron specific extensions of torch Module with support
for pipelining."""
diff --git a/megatron/legacy/model/utils.py b/megatron/legacy/model/utils.py
index 9d6789fd7e0..f268ac40d06 100644
--- a/megatron/legacy/model/utils.py
+++ b/megatron/legacy/model/utils.py
@@ -69,7 +69,7 @@ def get_norm(config):
sequence_parallel=config.sequence_parallel,
apply_layernorm_1p=args.apply_layernorm_1p)
elif args.normalization == "RMSNorm":
- if args.apply_layernorm_1p:
+ if "apply_layernorm_1p" in args and args.apply_layernorm_1p:
raise NotImplementedError('RMSNorm does not currently support the layernorm_1p formulation.')
return RMSNorm(hidden_size=config.hidden_size,
diff --git a/megatron/legacy/model/vision/esvit_swin_backbone.py b/megatron/legacy/model/vision/esvit_swin_backbone.py
index 2ee6b550cf3..8e7a7314280 100644
--- a/megatron/legacy/model/vision/esvit_swin_backbone.py
+++ b/megatron/legacy/model/vision/esvit_swin_backbone.py
@@ -726,7 +726,7 @@ def init_weights(self, pretrained='', pretrained_layers=[], verbose=True):
for k, v in pretrained_dict.items():
need_init = (
k.split('.')[0] in pretrained_layers
- or pretrained_layers[0] is '*'
+ or pretrained_layers[0] == '*'
or 'relative_position_index' not in k
or 'attn_mask' not in k
)
@@ -785,7 +785,7 @@ def freeze_pretrained_layers(self, frozen_layers=[]):
if (
name.split('.')[0] in frozen_layers
or '.'.join(name.split('.')[0:2]) in frozen_layers
- or (len(frozen_layers) > 0 and frozen_layers[0] is '*')
+ or (len(frozen_layers) > 0 and frozen_layers[0] == '*')
):
for _name, param in module.named_parameters():
param.requires_grad = False
@@ -796,7 +796,7 @@ def freeze_pretrained_layers(self, frozen_layers=[]):
for name, param in self.named_parameters():
if (
name.split('.')[0] in frozen_layers
- or (len(frozen_layers) > 0 and frozen_layers[0] is '*')
+ or (len(frozen_layers) > 0 and frozen_layers[0] == '*')
and param.requires_grad is True
):
param.requires_grad = False
diff --git a/megatron/legacy/mpu/tests/__init__.py b/megatron/legacy/mpu/tests/__init__.py
deleted file mode 100644
index e69de29bb2d..00000000000
diff --git a/megatron/legacy/mpu/tests/commons.py b/megatron/legacy/mpu/tests/commons.py
deleted file mode 100644
index 611daf0f666..00000000000
--- a/megatron/legacy/mpu/tests/commons.py
+++ /dev/null
@@ -1,70 +0,0 @@
-# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
-
-import argparse
-import os
-import random
-import numpy
-import torch
-
-import mpu
-
-
-class IdentityLayer(torch.nn.Module):
- def __init__(self, size, scale=1.0):
- super(IdentityLayer, self).__init__()
- self.weight = torch.nn.Parameter(scale * torch.randn(size))
-
- def forward(self):
- return self.weight
-
-
-def set_random_seed(seed):
- """Set random seed for reproducability."""
- random.seed(seed)
- numpy.random.seed(seed)
- torch.manual_seed(seed)
- mpu.model_parallel_cuda_manual_seed(seed)
-
-
-def initialize_distributed(backend='nccl'):
- """Initialize torch.distributed."""
- # Get local rank in case it is provided.
- parser = argparse.ArgumentParser()
- parser.add_argument('--local_rank', type=int, default=None,
- help='local rank passed from distributed launcher')
- args = parser.parse_args()
- local_rank = args.local_rank
-
- # Get rank and world size.
- rank = int(os.getenv('RANK', '0'))
- world_size = int(os.getenv("WORLD_SIZE", '1'))
-
- print('> initializing torch.distributed with local rank: {}, '
- 'rank: {}, world size: {}'.format(local_rank, rank, world_size))
-
- # Set the device id.
- device = rank % torch.cuda.device_count()
- if local_rank is not None:
- device = local_rank
- torch.cuda.set_device(device)
-
- # Call the init process.
- init_method = 'tcp://'
- master_ip = os.getenv('MASTER_ADDR', 'localhost')
- master_port = os.getenv('MASTER_PORT', '6000')
- init_method += master_ip + ':' + master_port
- torch.distributed.init_process_group(
- backend=backend,
- world_size=world_size,
- rank=rank,
- init_method=init_method)
-
-
-def print_separator(message):
- torch.distributed.barrier()
- filler_len = (78 - len(message)) // 2
- filler = '-' * filler_len
- string = '\n' + filler + ' {} '.format(message) + filler
- if torch.distributed.get_rank() == 0:
- print(string, flush=True)
- torch.distributed.barrier()
diff --git a/megatron/legacy/mpu/tests/test_cross_entropy.py b/megatron/legacy/mpu/tests/test_cross_entropy.py
deleted file mode 100644
index 00ae42228a9..00000000000
--- a/megatron/legacy/mpu/tests/test_cross_entropy.py
+++ /dev/null
@@ -1,95 +0,0 @@
-# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
-
-from commons import set_random_seed
-from commons import IdentityLayer
-from commons import print_separator
-from commons import initialize_distributed
-from mpu.cross_entropy import vocab_parallel_cross_entropy
-import mpu
-import torch.nn.functional as F
-import torch
-import random
-import sys
-sys.path.append("../..")
-
-
-def torch_cross_entropy(batch_size, seq_length, vocab_size,
- logits_scale, seed):
- set_random_seed(seed)
- identity = IdentityLayer((batch_size, seq_length, vocab_size),
- scale=logits_scale).cuda()
- logits = identity()
- target = torch.cuda.LongTensor(
- size=(batch_size, seq_length)).random_(0, vocab_size)
- loss = F.cross_entropy(logits.view(-1, logits.size()[-1]),
- target.view(-1),
- reduction='none').view_as(target).mean()
- loss.backward()
- return loss, identity.weight.grad
-
-
-def mpu_cross_entropy(batch_size, seq_length, vocab_size,
- logits_scale, seed):
- set_random_seed(seed)
- identity = IdentityLayer((batch_size, seq_length, vocab_size),
- scale=logits_scale).cuda()
- logits = identity()
- logits_parallel = mpu.scatter_to_tensor_model_parallel_region(logits)
- target = torch.cuda.LongTensor(
- size=(batch_size, seq_length)).random_(0, vocab_size)
- loss = vocab_parallel_cross_entropy(logits_parallel, target).mean()
- loss.backward()
- return loss, identity.weight.grad
-
-
-def test_cross_entropy(tensor_model_parallel_size):
-
- if torch.distributed.get_rank() == 0:
- print('> testing cross entropy with model parallel size {} ...'.
- format(tensor_model_parallel_size))
-
- mpu.initialize_model_parallel(tensor_model_parallel_size)
- tensor_model_parallel_size = mpu.get_tensor_model_parallel_world_size()
-
- batch_size = 13
- seq_length = 17
- vocab_size_per_partition = 11
- logits_scale = 1000.0
- vocab_size = vocab_size_per_partition * tensor_model_parallel_size
- seed = 1234
-
- loss_torch, grad_torch = torch_cross_entropy(batch_size, seq_length,
- vocab_size, logits_scale,
- seed)
- loss_mpu, grad_mpu = mpu_cross_entropy(batch_size, seq_length,
- vocab_size, logits_scale,
- seed)
-
- error = loss_torch.sub_(loss_mpu).abs().max()
- print(' max error in loss on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 1.0e-6
-
- error = grad_torch.sub_(grad_mpu).abs().max()
- print(' max error in grad on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 1.0e-6
-
- # Reset groups
- mpu.destroy_tensor_model_parallel()
-
- torch.distributed.barrier()
- if torch.distributed.get_rank() == 0:
- print('>> passed the test :-)')
-
-
-if __name__ == '__main__':
-
- initialize_distributed()
- world_size = torch.distributed.get_world_size()
-
- tensor_model_parallel_size = 1
- while tensor_model_parallel_size <= world_size:
- print_separator('test cross entropy')
- test_cross_entropy(tensor_model_parallel_size)
- tensor_model_parallel_size *= 2
diff --git a/megatron/legacy/mpu/tests/test_data.py b/megatron/legacy/mpu/tests/test_data.py
deleted file mode 100644
index c30bf4bb8d4..00000000000
--- a/megatron/legacy/mpu/tests/test_data.py
+++ /dev/null
@@ -1,75 +0,0 @@
-# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
-
-from commons import print_separator
-from commons import initialize_distributed
-from mpu import data as data_utils
-import mpu
-import torch
-import functools
-import operator
-import sys
-sys.path.append("../..")
-
-
-def test_broadcast_data(tensor_model_parallel_size):
-
- if torch.distributed.get_rank() == 0:
- print('> testing broadcast_data with model parallel size {} ...'.
- format(tensor_model_parallel_size))
-
- mpu.initialize_model_parallel(tensor_model_parallel_size)
- torch.manual_seed(1234 + mpu.get_data_parallel_rank())
- tensor_model_parallel_size = mpu.get_tensor_model_parallel_world_size()
-
- key_size_t = {'key1': [7, 11],
- 'key2': [8, 2, 1],
- 'key3': [13],
- 'key4': [5, 1, 2],
- 'key5': [5, 12]}
- keys = list(key_size_t.keys())
-
- data = {}
- data_t = {}
- for key in key_size_t:
- data[key] = torch.LongTensor(size=key_size_t[key]).random_(0, 1000)
- data_t[key] = data[key].clone()
- data['keyX'] = torch.FloatTensor(size=(5, )).random_(0, 1000)
- data_t['keyX'] = data['keyX'].clone()
- if mpu.get_tensor_model_parallel_rank() != 0:
- data = None
-
- data_utils._check_data_types(keys, data_t, torch.int64)
- key_size, key_numel, \
- total_numel = data_utils._build_key_size_numel_dictionaries(keys, data)
- for key in keys:
- assert key_size[key] == key_size_t[key]
- total_numel_t = 0
- for key in keys:
- target_size = functools.reduce(operator.mul, key_size_t[key], 1)
- assert key_numel[key] == target_size
- total_numel_t += target_size
- assert total_numel == total_numel_t
-
- data_b = data_utils.broadcast_data(keys, data, torch.int64)
- for key in keys:
- tensor = data_t[key].cuda()
- assert data_b[key].sub(tensor).abs().max() == 0
-
- # Reset groups
- mpu.destroy_tensor_model_parallel()
-
- torch.distributed.barrier()
- if torch.distributed.get_rank() == 0:
- print('>> passed the test :-)')
-
-
-if __name__ == '__main__':
-
- initialize_distributed()
- world_size = torch.distributed.get_world_size()
-
- tensor_model_parallel_size = 1
- while tensor_model_parallel_size <= world_size:
- print_separator('test test broadcast data')
- test_broadcast_data(tensor_model_parallel_size)
- tensor_model_parallel_size *= 2
diff --git a/megatron/legacy/mpu/tests/test_initialize.py b/megatron/legacy/mpu/tests/test_initialize.py
deleted file mode 100644
index 48652080a52..00000000000
--- a/megatron/legacy/mpu/tests/test_initialize.py
+++ /dev/null
@@ -1,82 +0,0 @@
-# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
-
-from commons import print_separator
-from commons import initialize_distributed
-import mpu
-import torch
-import sys
-sys.path.append("../..")
-
-
-def test_initialize_model_parallel(tensor_model_parallel_size):
-
- if torch.distributed.get_rank() == 0:
- print('> testing initialize_model_parallel with size {} ...'.format(
- tensor_model_parallel_size))
- tensor_model_parallel_size_ = min(tensor_model_parallel_size,
- torch.distributed.get_world_size())
- assert not mpu.model_parallel_is_initialized()
- mpu.initialize_model_parallel(tensor_model_parallel_size_)
- assert mpu.model_parallel_is_initialized()
-
- # Checks.
- def check(group, world_size, rank):
- assert world_size == group.size()
- assert rank == group.rank()
-
- # Model parallel.
- world_size = tensor_model_parallel_size_
- rank = torch.distributed.get_rank() % tensor_model_parallel_size_
- assert world_size == mpu.get_tensor_model_parallel_world_size()
- assert rank == mpu.get_tensor_model_parallel_rank()
- check(mpu.get_tensor_model_parallel_group(), world_size, rank)
-
- # Data parallel.
- world_size = torch.distributed.get_world_size() // tensor_model_parallel_size_
- rank = torch.distributed.get_rank() // tensor_model_parallel_size
- assert world_size == mpu.get_data_parallel_world_size()
- assert rank == mpu.get_data_parallel_rank()
- check(mpu.get_data_parallel_group(), world_size, rank)
-
- # Reset groups
- mpu.destroy_model_parallel()
-
- torch.distributed.barrier()
- if torch.distributed.get_rank() == 0:
- print('>> passed the test :-)')
-
-
-def test_get_tensor_model_parallel_src_rank(tensor_model_parallel_size_):
-
- if torch.distributed.get_rank() == 0:
- print('> testing get_tensor_model_parallel_src_rank with size {} ...'.format(
- tensor_model_parallel_size_))
- tensor_model_parallel_size = min(tensor_model_parallel_size_,
- torch.distributed.get_world_size())
- assert not mpu.model_parallel_is_initialized()
- mpu.initialize_model_parallel(tensor_model_parallel_size)
- assert mpu.model_parallel_is_initialized()
-
- # Checks
- src_rank = torch.distributed.get_rank() - mpu.get_tensor_model_parallel_rank()
- assert mpu.get_tensor_model_parallel_src_rank() == src_rank
-
- # Reset groups
- mpu.destroy_model_parallel()
-
- torch.distributed.barrier()
- if torch.distributed.get_rank() == 0:
- print('>> passed the test :-)')
-
-
-if __name__ == '__main__':
-
- initialize_distributed()
- world_size = torch.distributed.get_world_size()
- tensor_model_parallel_size = 1
- while tensor_model_parallel_size <= world_size:
- print_separator('test initialize model parallel')
- test_initialize_model_parallel(tensor_model_parallel_size)
- print_separator('test model parallel source rank')
- test_get_tensor_model_parallel_src_rank(tensor_model_parallel_size)
- tensor_model_parallel_size *= 2
diff --git a/megatron/legacy/mpu/tests/test_layers.py b/megatron/legacy/mpu/tests/test_layers.py
deleted file mode 100644
index 73ad4b94595..00000000000
--- a/megatron/legacy/mpu/tests/test_layers.py
+++ /dev/null
@@ -1,517 +0,0 @@
-# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
-
-from mpu import layers
-from commons import set_random_seed
-from commons import print_separator
-from commons import initialize_distributed
-import mpu
-from torch.nn.parameter import Parameter
-import torch.nn.init as init
-import torch
-import random
-import sys
-sys.path.append("../..")
-
-
-def test_parallel_embedding(tensor_model_parallel_size):
-
- if torch.distributed.get_rank() == 0:
- print('> testing parallel embedding with model parallel size {} ...'.
- format(tensor_model_parallel_size))
-
- mpu.initialize_model_parallel(tensor_model_parallel_size)
- tensor_model_parallel_size = mpu.get_tensor_model_parallel_world_size()
-
- batch_size = 17
- seq_length = 23
- vocab_size = 48
- hidden_size = 16
- seed = 1236
-
- set_random_seed(123)
- input_data = torch.LongTensor(
- size=(batch_size, seq_length)).random_(0, vocab_size).cuda()
- loss_weight = torch.randn([batch_size, seq_length, hidden_size]).cuda()
-
- set_random_seed(seed)
- embedding_original = torch.nn.Embedding(vocab_size, hidden_size).cuda()
-
- output = embedding_original(input_data)
- loss_original = torch.mul(output, loss_weight).sum()
- loss_original.backward()
-
- set_random_seed(seed)
- embedding_parallel = layers.ParallelEmbedding(
- vocab_size, hidden_size, init_method=init.normal_).cuda()
- output = embedding_parallel(input_data)
- loss_parallel = torch.mul(output, loss_weight).sum()
- loss_parallel.backward()
-
- set_random_seed(seed)
- embedding_vocab_parallel = layers.VocabParallelEmbedding(
- vocab_size, hidden_size, init_method=init.normal_).cuda()
- output = embedding_vocab_parallel(input_data)
- loss_vocab_parallel = torch.mul(output, loss_weight).sum()
- loss_vocab_parallel.backward()
-
- torch.distributed.barrier()
- error = loss_parallel.sub(loss_original).abs()
- print(' error in loss (parallel) on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 1.0e-12, 'error: {}'.format(error)
-
- torch.distributed.barrier()
- error = loss_vocab_parallel.sub(loss_original).abs()
- print(' error in loss (vocab parallel) on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 1.0e-12, 'error: {}'.format(error)
-
- weight_grad_orig = torch.split(embedding_original.weight.grad,
- hidden_size // tensor_model_parallel_size,
- 1)[mpu.get_tensor_model_parallel_rank()]
- error = embedding_parallel.weight.grad.sub(weight_grad_orig).abs().max()
- print(' error in grad (parallel) on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 1.0e-12, 'error: {}'.format(error)
-
- weight_grad_orig = torch.split(embedding_original.weight.grad,
- vocab_size // tensor_model_parallel_size,
- 0)[mpu.get_tensor_model_parallel_rank()]
- error = embedding_vocab_parallel.weight.grad.sub(
- weight_grad_orig).abs().max()
- print(' error in grad (vocab parallel) on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 1.0e-12, 'error: {}'.format(error)
-
- # Reset groups
- mpu.destroy_model_parallel()
-
- torch.distributed.barrier()
- if torch.distributed.get_rank() == 0:
- print('>> passed the test :-)')
-
-
-def test_initialize_affine_weight(tensor_model_parallel_size):
-
- mpu.initialize_model_parallel(tensor_model_parallel_size)
- if torch.distributed.get_rank() == 0:
- print('> testing initialize_affine_weight with model parallel '
- 'size: {}'.format(tensor_model_parallel_size))
- tensor_model_parallel_size = mpu.get_tensor_model_parallel_world_size()
-
- seed = 12345
- input_size_coeff = 13
- input_size = input_size_coeff * tensor_model_parallel_size
- output_size_coeff = 17
- output_size = output_size_coeff * tensor_model_parallel_size
-
- # ---------------
- # Column parallel
- # ---------------
- weight = torch.empty(output_size_coeff, input_size)
- set_random_seed(seed)
- layers._initialize_affine_weight(weight, output_size, input_size,
-
- output_size_coeff, 0,
- torch.nn.init.normal_)
- # Target.
- set_random_seed(seed)
- master_weight = torch.empty(output_size, input_size)
- torch.nn.init.normal_(master_weight)
- rank = mpu.get_tensor_model_parallel_rank()
- my_weight = torch.split(master_weight, output_size_coeff,
- dim=0)[rank].contiguous().clone()
-
- # Compare.
- error = weight.sub(my_weight).abs().max()
- torch.distributed.barrier()
- print(' column parallel max error (should be zero) on global rank '
- '{}: {}'.format(torch.distributed.get_rank(), error))
- assert error < 1.0e-6
-
- # ------------
- # Row parallel
- # ------------
- weight = torch.empty(output_size, input_size_coeff)
- set_random_seed(seed)
- mpu.layers._initialize_affine_weight(weight, output_size, input_size,
- input_size_coeff, 1,
- torch.nn.init.normal_)
- # Target.
- set_random_seed(seed)
- master_weight = torch.empty(output_size, input_size)
- torch.nn.init.normal_(master_weight)
- rank = mpu.get_tensor_model_parallel_rank()
- my_weight = torch.split(master_weight, input_size_coeff,
- dim=1)[rank].contiguous().clone()
-
- # Compare.
- error = weight.sub(my_weight).abs().max()
- torch.distributed.barrier()
- print(' row parallel max error (should be zero) on global rank '
- '{}: {}'.format(torch.distributed.get_rank(), error))
- assert error < 1.0e-6
-
- # Reset groups
- mpu.destroy_model_parallel()
-
- torch.distributed.barrier()
- if torch.distributed.get_rank() == 0:
- print(' >> passed the test :-)')
-
-
-class IdentityLayer2D(torch.nn.Module):
- def __init__(self, m, n):
- super(IdentityLayer2D, self).__init__()
- self.weight = Parameter(torch.Tensor(m, n))
- torch.nn.init.xavier_normal_(self.weight)
-
- def forward(self):
- return self.weight
-
-
-def test_column_parallel_linear(tensor_model_parallel_size):
-
- mpu.initialize_model_parallel(tensor_model_parallel_size)
- if torch.distributed.get_rank() == 0:
- print('> testing ColumnParallelLinear with model parallel '
- 'size: {}'.format(tensor_model_parallel_size))
- tensor_model_parallel_size = mpu.get_tensor_model_parallel_world_size()
-
- seed = 12345
- set_random_seed(seed)
- input_size_coeff = 13
- input_size = input_size_coeff * tensor_model_parallel_size
- output_size_coeff = 17
- output_size = output_size_coeff * tensor_model_parallel_size
- batch_size = 7
-
- # Network
- identity_layer = IdentityLayer2D(batch_size, input_size).cuda()
- linear_layer = mpu.ColumnParallelLinear(
- input_size, output_size, keep_master_weight_for_test=True).cuda()
- loss_weight = torch.randn([batch_size, output_size]).cuda()
- # Forward
- input_ = identity_layer()
- output = linear_layer(input_)
- loss = torch.mul(output, loss_weight).sum()
- # Backward
- loss.backward()
-
- # Values.
- dLdY = loss_weight
- X = identity_layer.weight
- A = linear_layer.master_weight.cuda()
- dLdA = torch.matmul(dLdY.t(), X)
- dLdb = torch.matmul(torch.ones(batch_size, 1).cuda().t(), dLdY).view(-1)
- dLdX = torch.matmul(dLdY, A)
-
- rank = mpu.get_tensor_model_parallel_rank()
- my_dLdA = torch.split(dLdA, output_size_coeff,
- dim=0)[rank].contiguous().clone()
- error = my_dLdA.sub(linear_layer.weight.grad).abs().max()
- torch.distributed.barrier()
- print(' error in dLdA on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 1.0e-6
-
- my_dLdb = torch.split(dLdb, output_size_coeff,
- dim=0)[rank].contiguous().clone()
- error = my_dLdb.sub(linear_layer.bias.grad).abs().max()
- torch.distributed.barrier()
- print(' error in dLdb on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 1.0e-6
-
- error = dLdX.sub(identity_layer.weight.grad).abs().max()
- torch.distributed.barrier()
- print(' error in dLdX on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 1.0e-6
-
- # Reset groups
- mpu.destroy_model_parallel()
-
- torch.distributed.barrier()
- if torch.distributed.get_rank() == 0:
- print(' >> passed the test :-)')
-
-
-def test_row_parallel_linear(tensor_model_parallel_size):
-
- mpu.initialize_model_parallel(tensor_model_parallel_size)
- if torch.distributed.get_rank() == 0:
- print('> testing RowParallelLinear with model parallel '
- 'size: {}'.format(tensor_model_parallel_size))
- tensor_model_parallel_size = mpu.get_tensor_model_parallel_world_size()
-
- seed = 12345
- set_random_seed(seed)
- input_size_coeff = 13
- input_size = input_size_coeff * tensor_model_parallel_size
- output_size_coeff = 17
- output_size = output_size_coeff * tensor_model_parallel_size
- batch_size = 7
-
- # Network
- identity_layer = IdentityLayer2D(batch_size, input_size).cuda()
- linear_layer = mpu.RowParallelLinear(
- input_size, output_size, keep_master_weight_for_test=True).cuda()
- loss_weight = torch.randn([batch_size, output_size]).cuda()
- # Forward
- input_ = identity_layer()
- output = linear_layer(input_)
- loss = torch.mul(output, loss_weight).sum()
- # Backward
- loss.backward()
-
- # Values.
- dLdY = loss_weight
- X = identity_layer.weight
- A = linear_layer.master_weight.cuda()
- dLdA = torch.matmul(dLdY.t(), X)
- dLdb = torch.matmul(torch.ones(batch_size, 1).cuda().t(), dLdY).view(-1)
- dLdX = torch.matmul(dLdY, A)
-
- rank = mpu.get_tensor_model_parallel_rank()
- my_dLdA = torch.split(dLdA, input_size_coeff,
- dim=1)[rank].contiguous().clone()
- error = my_dLdA.sub(linear_layer.weight.grad).abs().max()
- torch.distributed.barrier()
- print(' error in dLdA on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 1.0e-6
-
- error = dLdb.sub(linear_layer.bias.grad).abs().max()
- torch.distributed.barrier()
- print(' error in dLdb on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 1.0e-6
-
- error = dLdX.sub(identity_layer.weight.grad).abs().max()
- torch.distributed.barrier()
- print(' error in dLdX on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 1.0e-6
-
- # Reset groups
- mpu.destroy_model_parallel()
-
- torch.distributed.barrier()
- if torch.distributed.get_rank() == 0:
- print(' >> passed the test :-)')
-
-
-class IdentityLayer3D(torch.nn.Module):
- def __init__(self, m, n, k):
- super(IdentityLayer3D, self).__init__()
- self.weight = Parameter(torch.Tensor(m, n, k))
- torch.nn.init.xavier_normal_(self.weight)
-
- def forward(self):
- return self.weight
-
-
-def parallel_self_attention(tensor_model_parallel_size, num_att_heads_per_partition,
- hidden_size_per_att_head, dropout_prob, batch_size,
- sequence_length):
- mpu.initialize_model_parallel(tensor_model_parallel_size)
- tensor_model_parallel_size = mpu.get_tensor_model_parallel_world_size()
-
- seed = 12345
- set_random_seed(seed)
-
- num_att_heads = num_att_heads_per_partition * \
- torch.distributed.get_world_size()
- hidden_size = hidden_size_per_att_head * num_att_heads
-
- # Network
- identity_layer = IdentityLayer3D(batch_size, sequence_length,
- hidden_size).cuda()
- attention_layer = mpu.BertParallelSelfAttention(hidden_size, num_att_heads,
- dropout_prob).cuda()
- loss_weight = torch.randn([batch_size, sequence_length, hidden_size]).cuda()
- attention_mask = torch.randn([batch_size, 1, 1, sequence_length]).cuda()
- # Forward
- input_ = identity_layer()
- output = attention_layer(input_, attention_mask)
- loss = torch.mul(output, loss_weight).sum()
- # Backward
- loss.backward()
-
- rank = mpu.get_tensor_model_parallel_rank()
- mpu.destroy_model_parallel()
- return rank, hidden_size, tensor_model_parallel_size, loss, \
- attention_layer, identity_layer
-
-
-def test_parallel_self_attention(tensor_model_parallel_size):
-
- if torch.distributed.get_rank() == 0:
- print('> testing ParallelSelfAttention with model parallel '
- 'size: {}'.format(tensor_model_parallel_size))
-
- num_att_heads_per_partition = 3
- hidden_size_per_att_head = 7
- dropout_prob = 0.0 # has to be zero
- batch_size = 5
- sequence_length = 13
-
- rank_1, hideen_size_1, tensor_model_parallel_size_1, loss_1, \
- attention_layer_1, identity_layer_1 = parallel_self_attention(
- 1, num_att_heads_per_partition,
- hidden_size_per_att_head, dropout_prob, batch_size, sequence_length)
-
- rank, hidden_size, tensor_model_parallel_size, loss, \
- attention_layer, identity_layer = parallel_self_attention(
- tensor_model_parallel_size, num_att_heads_per_partition,
- hidden_size_per_att_head, dropout_prob, batch_size, sequence_length)
- assert hideen_size_1 == hidden_size
-
- error = loss_1.sub(loss).abs().max()
- torch.distributed.barrier()
- print(' loss error on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 5.0e-6
-
- my_lin_grad_list = torch.split(
- attention_layer_1.query_key_value.weight.grad,
- hidden_size // tensor_model_parallel_size, 0)[rank::tensor_model_parallel_size]
- my_lin_grad = torch.cat(my_lin_grad_list, dim=0)
- error = my_lin_grad.sub(
- attention_layer.query_key_value.weight.grad).abs().max()
- torch.distributed.barrier()
- print(' weight gradient error on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 5.0e-6
-
- error = identity_layer_1.weight.grad.sub(
- identity_layer.weight.grad).abs().max()
- torch.distributed.barrier()
- print(' input gradient error on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 5.0e-6
-
- torch.distributed.barrier()
- if torch.distributed.get_rank() == 0:
- print(' >> passed the test :-)')
-
-
-def parallel_transformer(tensor_model_parallel_size, num_att_heads_per_partition,
- hidden_size_per_att_head, batch_size, sequence_length):
-
- mpu.initialize_model_parallel(tensor_model_parallel_size)
- tensor_model_parallel_size = mpu.get_tensor_model_parallel_world_size()
-
- seed = 12345
- set_random_seed(seed)
-
- num_att_heads = num_att_heads_per_partition * \
- torch.distributed.get_world_size()
- hidden_size = hidden_size_per_att_head * num_att_heads
- intermediate_size = 4 * hidden_size
-
- # Network
- identity_layer = IdentityLayer3D(batch_size, sequence_length,
- hidden_size).cuda()
- transformer_layer = mpu.BertParallelTransformerLayer(
- hidden_size, intermediate_size, num_att_heads, 0.0, 0.0,
- torch.nn.functional.relu, 1.0e-5).cuda()
-
- loss_weight = torch.randn([batch_size, sequence_length, hidden_size]).cuda()
- attention_mask = torch.randn([batch_size, 1, 1, sequence_length]).cuda()
- # Forward
- input_ = identity_layer()
- output = transformer_layer(input_, attention_mask)
- loss = torch.mul(output, loss_weight).sum()
- # Backward
- loss.backward()
-
- rank = mpu.get_tensor_model_parallel_rank()
- mpu.destroy_model_parallel()
- return rank, hidden_size, tensor_model_parallel_size, loss, \
- transformer_layer, identity_layer
-
-
-def test_parallel_transformer_layer(tensor_model_parallel_size):
-
- if torch.distributed.get_rank() == 0:
- print('> testing ParallelTransformerLayer with model parallel '
- 'size: {}'.format(tensor_model_parallel_size))
-
- num_att_heads_per_partition = 3
- hidden_size_per_att_head = 7
- batch_size = 5
- sequence_length = 13
-
- rank_1, hidden_size_1, tensor_model_parallel_size_1, loss_1, \
- transformer_layer_1, identity_layer_1 = parallel_transformer(
- 1, num_att_heads_per_partition,
- hidden_size_per_att_head, batch_size, sequence_length)
-
- rank, hidden_size, tensor_model_parallel_size, loss, \
- transformer_layer, identity_layer = parallel_transformer(
- tensor_model_parallel_size, num_att_heads_per_partition,
- hidden_size_per_att_head, batch_size, sequence_length)
-
- error = loss_1.sub(loss).abs().max()
- torch.distributed.barrier()
- print(' loss error on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 5.0e-5, 'error: {}'.format(error)
-
- error = identity_layer_1.weight.grad.sub(
- identity_layer.weight.grad).abs().max()
- torch.distributed.barrier()
- print(' input gradient error on global rank {}: {}'.format(
- torch.distributed.get_rank(), error))
- assert error < 5.0e-5, 'error: {}'.format(error)
-
- torch.distributed.barrier()
- if torch.distributed.get_rank() == 0:
- print(' >> passed the test :-)')
-
-
-if __name__ == '__main__':
-
- torch.backends.cudnn.deterministic = True
- torch.backends.cudnn.benchmark = False
-
- initialize_distributed()
- world_size = torch.distributed.get_world_size()
-
- print_separator('test initialize affine weight')
- tensor_model_parallel_size = 1
- while tensor_model_parallel_size <= world_size:
- test_initialize_affine_weight(tensor_model_parallel_size)
- tensor_model_parallel_size *= 2
-
- tensor_model_parallel_size = 1
- while tensor_model_parallel_size <= world_size:
- print_separator('test parallel embedding')
- test_parallel_embedding(tensor_model_parallel_size)
- tensor_model_parallel_size *= 2
-
- print_separator('test column-parallel linear')
- tensor_model_parallel_size = 1
- while tensor_model_parallel_size <= world_size:
- test_column_parallel_linear(tensor_model_parallel_size)
- tensor_model_parallel_size *= 2
-
- print_separator('test row-parallel linear')
- tensor_model_parallel_size = 1
- while tensor_model_parallel_size <= world_size:
- test_row_parallel_linear(tensor_model_parallel_size)
- tensor_model_parallel_size *= 2
-
- print_separator('test parallel self-attention')
- tensor_model_parallel_size = 1
- while tensor_model_parallel_size <= world_size:
- test_parallel_self_attention(tensor_model_parallel_size)
- tensor_model_parallel_size *= 2
-
- print_separator('test parallel transformer')
- tensor_model_parallel_size = 1
- while tensor_model_parallel_size <= world_size:
- test_parallel_transformer_layer(tensor_model_parallel_size)
- tensor_model_parallel_size *= 2
diff --git a/megatron/legacy/mpu/tests/test_random.py b/megatron/legacy/mpu/tests/test_random.py
deleted file mode 100644
index 26092772cff..00000000000
--- a/megatron/legacy/mpu/tests/test_random.py
+++ /dev/null
@@ -1,191 +0,0 @@
-# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
-
-from commons import print_separator
-from commons import initialize_distributed
-import mpu
-import torch
-import sys
-sys.path.append("../..")
-
-
-def test_set_cuda_rng_state(tensor_model_parallel_size):
-
- if torch.distributed.get_rank() == 0:
- print('> testing set_rng_state with size {} ...'.
- format(tensor_model_parallel_size))
-
- mpu.initialize_model_parallel(tensor_model_parallel_size)
- tensor_model_parallel_size = mpu.get_tensor_model_parallel_world_size()
-
- size = 123
- seed = 1234
- torch.cuda.manual_seed(1234)
- tensor = torch.tensor(size, dtype=torch.float, device='cuda')
-
- # Get the state
- rng_state = torch.cuda.get_rng_state()
- rng_state_copy = rng_state.clone()
-
- # Do some stuff.
- for _ in range(5):
- torch.randn(size, out=tensor)
- result_1 = tensor.clone()
-
- assert rng_state.sub(rng_state_copy).max() == 0
- assert torch.cuda.get_rng_state().sub(rng_state_copy).max() > 0
-
- # State should be different.
- new_rng_state = torch.cuda.get_rng_state()
- max_diff = new_rng_state.sub(rng_state).max()
- print(' max diff in rng state (should be non-zero) on global rank {}: {}'.
- format(torch.distributed.get_rank(), max_diff))
- assert max_diff > 0
-
- # Reset the rng state and do the same stuff.
- mpu.random._set_cuda_rng_state(rng_state)
- for _ in range(5):
- torch.randn(size, out=tensor)
- mpu.random._set_cuda_rng_state(rng_state)
- for _ in range(5):
- torch.randn(size, out=tensor)
- result_2 = tensor.clone()
-
- # Results should be the same
- error = result_2.sub(result_1).abs().max()
- print(' max error in generated tensors (should be zero) on '
- 'global rank {}: {}'.format(torch.distributed.get_rank(), error))
- assert error < 1.0e-6
-
- # Input state should have remained intact.
- error = rng_state.sub(rng_state_copy).max()
- print(' max error in rng state (should be zero) on global rank {}: {}'.
- format(torch.distributed.get_rank(), error))
- assert error == 0
-
- # Reset groups
- mpu.destroy_model_parallel()
-
- torch.distributed.barrier()
- if torch.distributed.get_rank() == 0:
- print('>> passed the test :-)')
-
-
-def test_cuda_rng_tracker(tensor_model_parallel_size):
-
- if torch.distributed.get_rank() == 0:
- print('> testing cuda rng tracker with size {} ...'.
- format(tensor_model_parallel_size))
-
- mpu.initialize_model_parallel(tensor_model_parallel_size)
- tensor_model_parallel_size = mpu.get_tensor_model_parallel_world_size()
-
- seed_1 = 1234
- seed_2 = 4321
- size = [12, 21]
- tensor = torch.tensor(size, dtype=torch.float, device='cuda')
-
- # Set to seed_1 and generate two tensors.
- torch.cuda.manual_seed(seed_1)
- torch.randn(size, out=tensor)
- target_11 = tensor.clone()
- torch.randn(size, out=tensor)
- target_12 = tensor.clone()
-
- # Set to seed_2 and generate two tensors.
- torch.cuda.manual_seed(seed_2)
- torch.randn(size, out=tensor)
- target_21 = tensor.clone()
- torch.randn(size, out=tensor)
- target_22 = tensor.clone()
-
- # Now if we interleave seed_1 and seed_2,
- # we should still get the same tensors
- torch.cuda.manual_seed(seed_1)
- mpu.get_cuda_rng_tracker().add('test', seed_2)
-
- torch.randn(size, out=tensor)
- result_11 = tensor.clone()
-
- with mpu.get_cuda_rng_tracker().fork('test'):
- torch.randn(size, out=tensor)
- result_21 = tensor.clone()
-
- torch.randn(size, out=tensor)
- result_12 = tensor.clone()
-
- with mpu.get_cuda_rng_tracker().fork('test'):
- torch.randn(size, out=tensor)
- result_22 = tensor.clone()
-
- diff = result_11.sub(result_21).abs().max()
- diff = min(diff, result_12.sub(result_22).abs().max())
- print(' max diff in generated tensors (should be non-zero) on '
- 'global rank {}: {}'.format(torch.distributed.get_rank(), diff))
- assert diff > 1.0e-6
- error = max(result_11.sub(target_11).abs().max(),
- result_12.sub(target_12).abs().max())
- error = max(error, result_21.sub(target_21).abs().max())
- error = max(error, result_22.sub(target_22).abs().max())
- print(' max error in generated tensors (should be zero) on '
- 'global rank {}: {}'.format(torch.distributed.get_rank(), error))
- assert error < 1.0e-6
-
- # Reset the tracker
- mpu.get_cuda_rng_tracker().reset()
-
- # Reset groups
- mpu.destroy_model_parallel()
-
- torch.distributed.barrier()
- if torch.distributed.get_rank() == 0:
- print('>> passed the test :-)')
-
-
-def test_model_parallel_cuda_manual_seed(tensor_model_parallel_size):
-
- if torch.distributed.get_rank() == 0:
- print('> testing model parallel cuda manual seed with size {} ...'.
- format(tensor_model_parallel_size))
-
- mpu.initialize_model_parallel(tensor_model_parallel_size)
- tensor_model_parallel_size = mpu.get_tensor_model_parallel_world_size()
-
- mpu.model_parallel_cuda_manual_seed(12345)
- assert torch.cuda.initial_seed() == 12345
- with mpu.get_cuda_rng_tracker().fork():
- assert torch.cuda.initial_seed() == (12345 + 2718 +
- mpu.get_tensor_model_parallel_rank())
-
- # Reset the tracker
- mpu.get_cuda_rng_tracker().reset()
-
- # Reset groups
- mpu.destroy_model_parallel()
-
- torch.distributed.barrier()
- if torch.distributed.get_rank() == 0:
- print('>> passed the test :-)')
-
-
-if __name__ == '__main__':
-
- initialize_distributed()
- world_size = torch.distributed.get_world_size()
-
- tensor_model_parallel_size = 1
- while tensor_model_parallel_size <= world_size:
- print_separator('test set rng state')
- test_set_cuda_rng_state(tensor_model_parallel_size)
- tensor_model_parallel_size *= 2
-
- tensor_model_parallel_size = 1
- while tensor_model_parallel_size <= world_size:
- print_separator('test cuda rng tracker')
- test_cuda_rng_tracker(tensor_model_parallel_size)
- tensor_model_parallel_size *= 2
-
- tensor_model_parallel_size = 1
- while tensor_model_parallel_size <= world_size:
- print_separator('test model parallel cuda manual seed')
- test_model_parallel_cuda_manual_seed(tensor_model_parallel_size)
- tensor_model_parallel_size *= 2
diff --git a/megatron/post_training/arguments.py b/megatron/post_training/arguments.py
index 1e988680142..dc98c6d28e4 100644
--- a/megatron/post_training/arguments.py
+++ b/megatron/post_training/arguments.py
@@ -21,7 +21,17 @@ def add_modelopt_args(parser):
group.add_argument(
"--export-te-mcore-model",
action="store_true",
- help="Export a megatron-core transformer-engine checkpoint.",
+ help="Indicate the source checkpoint uses the fused Transformer-Engine mcore layer spec "
+ "(where layernorms are fused into linear layers). Enables state_dict key remapping so the "
+ "TE checkpoint can be loaded into the local ModelOpt spec for PTQ/export, and saved back "
+ "in TE-compatible format. Mutually exclusive with --export-default-te-spec.",
+ )
+ group.add_argument(
+ "--export-default-te-spec",
+ action="store_true",
+ help="Use the full Transformer-Engine layer spec for model building. "
+ "This builds the model with TELayerNormColumnParallelLinear, TERowParallelLinear, "
+ "TEGroupedMLP, TEDotProductAttention, etc., matching the canonical TE specs.",
)
group.add_argument(
"--export-force-local-attention",
diff --git a/megatron/post_training/generate.py b/megatron/post_training/generate.py
index 2a124734a30..c4bb07c353f 100644
--- a/megatron/post_training/generate.py
+++ b/megatron/post_training/generate.py
@@ -19,6 +19,7 @@ def simple_generate(
osl: int = 32,
eos_token_id: List[int] = [],
disable_tqdm: bool = False,
+ calibration_mode: bool = False,
):
"""A simple generate function without using KV-cache."""
model.eval()
@@ -64,20 +65,23 @@ def _forward_step_func(data, model):
else:
tokens = input_ids
- list_of_logits = get_forward_backward_func()(
+ logits_and_extras = get_forward_backward_func()(
forward_step_func=_forward_step_func,
data_iterator=[{"tokens": tokens}],
model=model,
num_microbatches=1,
seq_length=tokens.shape[-1],
- micro_batch_size=1,
+ micro_batch_size=tokens.shape[0],
decoder_seq_length=tokens.shape[-1],
forward_only=True,
collect_non_loss_data=True,
)
+ if calibration_mode:
+ continue # avoid unnecessary computation
+
if mpu.is_pipeline_last_stage():
- logits = gather_from_tensor_model_parallel_region(list_of_logits[0])
+ logits = gather_from_tensor_model_parallel_region(logits_and_extras[0])
eager_ids = logits[:, input_ids.shape[-1] - 1, :].argmax(dim=-1, keepdim=True).detach()
else:
eager_ids = None
diff --git a/megatron/post_training/model_builder.py b/megatron/post_training/model_builder.py
index fea837c96c3..085d188e811 100644
--- a/megatron/post_training/model_builder.py
+++ b/megatron/post_training/model_builder.py
@@ -2,6 +2,7 @@
"""ModelOpt GPT model provider."""
+import logging
import os
from argparse import Namespace
from typing import Any, Dict
@@ -105,6 +106,11 @@ def _load_teacher_model_config(checkpoint_path: str) -> Namespace:
del args_dict["kv_channels"] # not recalculated if present
args_dict.update(config)
+ # Backward compat: old checkpoints have hybrid_override_pattern but not hybrid_layer_pattern
+ if (args_dict.get('hybrid_override_pattern') is not None
+ and args_dict.get('hybrid_layer_pattern') is None):
+ args_dict['hybrid_layer_pattern'] = args_dict['hybrid_override_pattern']
+
return Namespace(**args_dict)
@@ -113,13 +119,10 @@ def _load_teacher_model(config, config_raw: Namespace, model_kwargs: Dict[str, A
args = get_args()
if config.is_hybrid_model:
- # These parameters are not part of the TransformerConfig and need to be passed separately.
- if "hybrid_override_pattern" in config_raw:
- model_kwargs["hybrid_override_pattern"] = config_raw.hybrid_override_pattern
- if "hybrid_attention_ratio" in config_raw:
- model_kwargs["hybrid_attention_ratio"] = config_raw.hybrid_attention_ratio
- if "hybrid_mlp_ratio" in config_raw:
- model_kwargs["hybrid_mlp_ratio"] = config_raw.hybrid_mlp_ratio
+ # This parameter is not part of the TransformerConfig and needs to be passed separately.
+ # Note: hybrid_override_pattern is remapped to hybrid_layer_pattern in
+ # _load_teacher_model_config, so config_raw.hybrid_layer_pattern is always set here.
+ model_kwargs["hybrid_layer_pattern"] = config_raw.hybrid_layer_pattern
teacher = MCoreMambaModel(config=config, **model_kwargs)
else:
@@ -135,7 +138,7 @@ def _load_teacher_model(config, config_raw: Namespace, model_kwargs: Dict[str, A
local_core_attention=False if config.context_parallel_size > 1 else args.export_force_local_attention,
remap_te_layernorm=args.export_te_mcore_model,
real_quant_cfg=args.export_real_quant_cfg,
- use_arbitrary_attention_mask=False if config.context_parallel_size > 1 else True,
+ use_arbitrary_attention_mask=False,
)
teacher = MCoreGPTModel(config=config, **model_kwargs)
_add_load_convert_hooks(teacher)
@@ -226,21 +229,18 @@ def modelopt_gpt_mamba_builder(
use_te=args.transformer_impl == "transformer_engine",
)
else:
- local_core_attention=args.export_force_local_attention
if config.context_parallel_size > 1:
print_rank_0("context_parallel_size > 1! Force using TEDotProductAttention!")
local_core_attention=False
- print_rank_0("context_parallel_size > 1! Force attention_mask_type to Causal. This can be wrong for EAGLE training!")
- use_arbitrary_attention_mask = False
else:
- use_arbitrary_attention_mask = True
+ local_core_attention=args.export_force_local_attention
transformer_layer_spec = get_gpt_modelopt_spec(
config=config,
local_core_attention=local_core_attention,
remap_te_layernorm=args.export_te_mcore_model,
real_quant_cfg=args.export_real_quant_cfg,
- use_arbitrary_attention_mask=use_arbitrary_attention_mask,
+ use_arbitrary_attention_mask=False,
)
model_kwargs = {
@@ -259,20 +259,26 @@ def modelopt_gpt_mamba_builder(
"pg_collection": pg_collection,
}
model = MCoreGPTModel(config=config, **model_kwargs)
- elif args.export_model_type == "MambaModel" or args.is_hybrid_model:
+ elif args.export_model_type == "MambaModel" or getattr(args, 'hybrid_layer_pattern', None) is not None:
from megatron.core.post_training.modelopt.mamba.model_specs import get_mamba_stack_modelopt_spec
+ if args.export_default_te_spec and args.export_te_mcore_model:
+ logging.getLogger(__name__).warning(
+ "--export-default-te-spec and --export-te-mcore-model are mutually exclusive. "
+ "Since --export-default-te-spec is given, --export-te-mcore-model will be disabled."
+ )
+ args.export_te_mcore_model = False
+
mamba_stack_spec = get_mamba_stack_modelopt_spec(
- remap_te_layernorm=args.export_te_mcore_model
+ remap_te_layernorm=args.export_te_mcore_model,
+ use_default_te_spec=args.export_default_te_spec,
)
model_kwargs = {
"mamba_stack_spec": mamba_stack_spec,
"vocab_size": args.padded_vocab_size,
"max_sequence_length": args.max_position_embeddings,
+ "hybrid_layer_pattern": args.hybrid_layer_pattern,
"pre_process": pre_process,
- "hybrid_attention_ratio": args.hybrid_attention_ratio,
- "hybrid_mlp_ratio": args.hybrid_mlp_ratio,
- "hybrid_override_pattern": args.hybrid_override_pattern,
"post_process": post_process,
"fp16_lm_cross_entropy": args.fp16_lm_cross_entropy,
"parallel_output": True,
@@ -317,6 +323,9 @@ def modelopt_gpt_mamba_builder(
assert (
not args.tp_comm_overlap
), "ModelOpt Distillation currently incompatible with `--tp-comm-overlap` option."
+ assert (
+ args.cross_entropy_fusion_impl != "te"
+ ), "ModelOpt Distillation currently incompatible with TransformerEngine Cross-Entropy implementation."
if args.pipeline_model_parallel_size > 1:
assert (
args.virtual_pipeline_model_parallel_size is None
diff --git a/megatron/rl/agent/api.py b/megatron/rl/agent/api.py
index 9568db3a54d..ed50427d00a 100644
--- a/megatron/rl/agent/api.py
+++ b/megatron/rl/agent/api.py
@@ -1,6 +1,7 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
import asyncio
+import logging
from abc import ABC, abstractmethod
from collections.abc import AsyncIterable
from typing import Generic, TypeVar
@@ -8,14 +9,12 @@
import numpy as np
from pydantic import BaseModel
+from megatron.core.inference.utils import asyncio_Queue, asyncio_QueueShutDown
from megatron.core.utils import trace_async_exceptions
from ..__init__ import Request, TypeLookupable
from ..inference import (
- ChatInferenceInterface,
- ChatInferenceRequest,
InferenceInterface,
- InferenceRequest,
LLMChatMessage,
ReturnsRaw,
)
@@ -41,6 +40,8 @@ class GroupedRolloutRequest(Request):
inference_interface: InferenceInterface
validation: bool = False
filter_groups_with_same_reward: bool = False
+ streaming: bool = False
+ enforce_order: bool = False
class Rollout(AgentBaseModel):
@@ -49,8 +50,11 @@ class Rollout(AgentBaseModel):
trajectory: list[str]
prompt_length: list[int] | None = None
reward: float = None
- env_id: str | None = None
+ env_id: str = ''
problem_id: str | None = None
+ policy_epoch: list[list[tuple[int, int]]]
+ kv_cache_epoch: list[list[tuple[int, int]]]
+ num_evictions: list[int]
class TokenRollout(AgentBaseModel):
@@ -60,8 +64,34 @@ class TokenRollout(AgentBaseModel):
reward: list[float] | float
generation_mask: list[list[bool]] | None = None
logprobs: list[list[float]] | None = None
- env_id: str | None = None
+ env_id: str = ''
problem_id: str | None = None
+ policy_epoch: list[list[tuple[int, int]]]
+ kv_cache_epoch: list[list[tuple[int, int]]]
+ num_evictions: list[int]
+
+
+Rollouts = list[TokenRollout | Rollout]
+
+
+class RolloutGroup(AgentBaseModel):
+ """A group of rollouts (e.g. multiple completions for one prompt) with batch metadata."""
+
+ rollouts: Rollouts
+ batch_id: int = 0
+ index_in_batch: int = 0
+
+ def __iter__(self):
+ return iter(self.rollouts)
+
+ def __len__(self):
+ return len(self.rollouts)
+
+ def __getitem__(self, idx):
+ return self.rollouts[idx]
+
+
+GroupedRollouts = list[RolloutGroup]
class ContrastiveRollout(AgentBaseModel):
@@ -102,7 +132,7 @@ class RewardEvaluationResult(EvaluationResult):
class EvaluationResponse(AgentBaseModel, TypeLookupable, Generic[T]):
- env_id: str | None = None
+ env_id: str
results: list[T]
def metrics(self):
@@ -124,11 +154,6 @@ async def get_reward_rollouts(self, request: RolloutRequest) -> list[Rollout]:
request.inference_interface, ReturnsRaw
), "InferenceInterface must support raw_text return to provide rollouts."
- if isinstance(request.inference_interface, ChatInferenceInterface):
- self.chat_mode = True
- else:
- self.chat_mode = False
-
return await asyncio.gather(
*[self.rollout(request=request) for _ in range(request.num_rollouts)]
)
@@ -158,11 +183,6 @@ async def get_reward_rollouts(self, request: RolloutRequest) -> list[TokenRollou
request.inference_interface, ReturnsRaw
), "InferenceInterface must support raw_text return to provide rollouts."
- if isinstance(request.inference_interface, ChatInferenceInterface):
- self.chat_mode = True
- else:
- self.chat_mode = False
-
return await asyncio.gather(
*[self.rollout(request=request) for _ in range(request.num_rollouts)]
)
@@ -187,38 +207,94 @@ async def get_grouped_rollouts(self, request: GroupedRolloutRequest):
request.inference_interface, ReturnsRaw
), "InferenceInterface must support raw_text return to provide rollouts."
- if isinstance(request.inference_interface, ChatInferenceInterface):
- self.chat_mode = True
- else:
- self.chat_mode = False
-
- # If num_groups is -1, we generate a stream of groups.
- # The buffer size is used to create backpressure for each agent in order to balance group generation in a multi-task setting.
- grouped_rollouts: asyncio.Queue[list[Rollout]] = asyncio.Queue(
- maxsize=self.buffer_size if request.num_groups < 0 else 0
+ # When streaming, use buffer_size to create backpressure
+ # for balanced generation in a multi-task setting.
+ grouped_rollouts: asyncio_Queue[RolloutGroup] = asyncio_Queue(
+ maxsize=self.buffer_size if request.streaming else 0
)
submitted_groups = 0
+ # num_groups controls how many groups each worker generates and yields together.
+ # When it's 1, the semaphore is a no-op.
+ groups_per_worker = request.num_groups
+ if groups_per_worker > 1:
+ assert not request.filter_groups_with_same_reward, \
+ "Cannot use filter_groups_with_same_reward with num_groups > 1."
+ assert self.parallel_generation_tasks >= groups_per_worker, \
+ f"{self.parallel_generation_tasks=} must be >= {groups_per_worker=}"
+ num_workers = self.parallel_generation_tasks // groups_per_worker
+ unused = self.parallel_generation_tasks % groups_per_worker
+ if unused:
+ logging.warning(
+ f"parallel_generation_tasks ({self.parallel_generation_tasks}) is not "
+ f"divisible by num_groups ({groups_per_worker}); "
+ f"{unused} generation task(s) will be unused."
+ )
+ submission_gate = asyncio.Semaphore(num_workers)
+
+ async def generate_and_enqueue(batch_id, index_in_batch):
+ group = await self.group_rollout(request=request)
+ if (
+ not request.filter_groups_with_same_reward
+ or np.std([r.reward for r in group]) > 1e-6
+ ):
+ await grouped_rollouts.put(
+ RolloutGroup(rollouts=group, batch_id=batch_id, index_in_batch=index_in_batch)
+ )
+ return True
+ return False
+
@trace_async_exceptions(verbose=True)
- async def group_task():
+ async def generate_task():
nonlocal submitted_groups
- while request.num_groups == -1 or submitted_groups < request.num_groups:
- submitted_groups += 1
- group = await self.group_rollout(request=request)
- if (
- not request.filter_groups_with_same_reward
- or np.std([r.reward for r in group]) > 1e-6
- ):
- await grouped_rollouts.put(group)
+ while request.streaming or submitted_groups < self.parallel_generation_tasks:
+ await submission_gate.acquire()
+ batch_id = submitted_groups // groups_per_worker
+ submitted_groups += groups_per_worker
+ if groups_per_worker > 1:
+ await asyncio.gather(*[
+ generate_and_enqueue(batch_id, i)
+ for i in range(groups_per_worker)
+ ])
else:
- submitted_groups -= 1
+ if not await generate_and_enqueue(batch_id, 0):
+ submitted_groups -= groups_per_worker
+ submission_gate.release()
+
+ tasks = [asyncio.create_task(generate_task()) for _ in range(num_workers)]
- tasks = [asyncio.create_task(group_task()) for _ in range(self.parallel_generation_tasks)]
+ async def shutdown_queue_when_done():
+ """Wait for all workers to finish, then shut down the queue."""
+ await asyncio.gather(*tasks)
+ grouped_rollouts.shutdown()
+
+ shutdown_task = asyncio.create_task(shutdown_queue_when_done())
try:
- while grouped_rollouts.qsize() > 0 or not all(task.done() for task in tasks):
- yield await grouped_rollouts.get()
+ next_batch_id = 0
+ pending: dict[int, GroupedRollouts] = {}
+ while True:
+ try:
+ group = await grouped_rollouts.get()
+ except asyncio_QueueShutDown:
+ break
+ if request.enforce_order:
+ # Accumulate groups and enforce submission order across batches.
+ pending.setdefault(group.batch_id, []).append(group)
+ while (l := len(pending.get(next_batch_id, []))) >= groups_per_worker:
+ assert l == groups_per_worker
+ batch = pending.pop(next_batch_id)
+ batch.sort(key=lambda g: g.index_in_batch)
+ next_batch_id += 1
+ for g in batch:
+ yield g
+ submission_gate.release()
+ else:
+ # Yield groups as soon as they're completed.
+ yield group
+ submission_gate.release()
finally:
+ shutdown_task.cancel()
for task in tasks:
task.cancel()
diff --git a/megatron/rl/agent/pass_at_evaluation_agent.py b/megatron/rl/agent/pass_at_evaluation_agent.py
index b10e3b897c8..c04e1c2772f 100644
--- a/megatron/rl/agent/pass_at_evaluation_agent.py
+++ b/megatron/rl/agent/pass_at_evaluation_agent.py
@@ -7,7 +7,7 @@
import numpy as np
from ..__init__ import GenericGenerationArgs
-from ..inference import ChatInferenceResponse, LLMChatMessage
+from ..inference import LLMChatMessage
from .api import EvaluationAgent, EvaluationRequest, EvaluationResponse, RewardEvaluationResult
diff --git a/megatron/rl/agent/reward_only_agent.py b/megatron/rl/agent/reward_only_agent.py
index 53b1f7407b2..42e3b2b08f4 100644
--- a/megatron/rl/agent/reward_only_agent.py
+++ b/megatron/rl/agent/reward_only_agent.py
@@ -7,8 +7,6 @@
from tqdm.asyncio import tqdm
from ..inference import (
- ChatInferenceInterface,
- ChatInferenceResponse,
InferenceResponse,
LLMChatMessage,
ReturnsRaw,
@@ -91,11 +89,7 @@ async def rollout_from_response(
), "InferenceInterface must support raw_text return to provide rollouts."
raw_text = response.raw_text
- response_text = (
- response.response.content
- if isinstance(response, ChatInferenceResponse)
- else response.response
- )
+ response_text = response.response.content
if isinstance(request.inference_interface, ReturnsTokens):
logprobs = response.logprobs
@@ -110,6 +104,9 @@ async def rollout_from_response(
generation_mask=[generation_mask],
env_id=self.env_id,
problem_id=golden['problem_id'] if 'problem_id' in golden else None,
+ policy_epoch=[response.policy_epoch],
+ kv_cache_epoch=[response.kv_cache_epoch],
+ num_evictions=[response.num_evictions],
)
else:
rollout = Rollout(
@@ -117,6 +114,9 @@ async def rollout_from_response(
reward=await self.get_reward(response_text, golden),
env_id=self.env_id,
problem_id=golden['problem_id'] if 'problem_id' in golden else None,
+ policy_epoch=[response.policy_epoch],
+ kv_cache_epoch=[response.kv_cache_epoch],
+ num_evictions=[response.num_evictions],
)
return rollout
@@ -126,14 +126,10 @@ async def rollout(self, request: RolloutRequest) -> Rollout:
prompt, golden = await self.get_prompt(validation=request.validation)
inference_request = request.inference_interface.prepare_request(
- [prompt], request.generation_args
+ prompt, request.generation_args
)
- responses = await request.inference_interface.agenerate(inference_request)
- assert (
- len(responses) == 1
- ), "get_reward_rollouts only requested a single response but got multiple responses"
- response = responses[0]
+ response = await request.inference_interface.agenerate(inference_request)
return await self.rollout_from_response(request, response, golden)
@@ -142,41 +138,22 @@ async def group_rollout(self, request: GroupedRolloutRequest) -> list[Rollout]:
prompt, golden = await self.get_prompt(validation=request.validation)
inference_request = request.inference_interface.prepare_request(
- [prompt], request.generation_args
- )
- inference_request.n = request.rollouts_per_group
-
- groups = await request.inference_interface.agenerate(inference_request)
- assert (
- len(groups) == 1
- ), "get_grouped_rollouts only requested a single group but got multiple groups"
- responses = groups[0].responses
-
- rollouts = await asyncio.gather(
- *[self.rollout_from_response(request, response, golden) for response in responses]
+ prompt, request.generation_args
)
- return rollouts
+ responses = await asyncio.gather(*[request.inference_interface.agenerate(inference_request) for _ in range(request.rollouts_per_group)])
+ return [await self.rollout_from_response(request, response, golden) for response in responses]
async def _evaluation(
self, prompt: str, golden: Any, request: EvaluationRequest
) -> RewardOnlyEvaluationResponse:
inference_request = request.inference_interface.prepare_request(
- [prompt], request.generation_args
+ prompt, request.generation_args
)
- responses = await request.inference_interface.agenerate(inference_request)
- assert (
- len(responses) == 1
- ), "evaluation only requested a single response but got multiple responses"
- response = responses[0]
-
- response_text = (
- response.response.content
- if isinstance(response, ChatInferenceResponse)
- else response.response
- )
+ response = await request.inference_interface.agenerate(inference_request)
+ response_text = response.response.content
result = RewardEvaluationResult(
env_id=self.env_id,
@@ -190,11 +167,6 @@ async def _evaluation(
async def run_evaluation(self, request: EvaluationRequest):
- if isinstance(request.inference_interface, ChatInferenceInterface):
- self.chat_mode = True
- else:
- self.chat_mode = False
-
# Get all prompts first
all_prompts = list(
await self.evaluation_prompts(
diff --git a/megatron/rl/agent/weighted_multi_task.py b/megatron/rl/agent/weighted_multi_task.py
index 4690d9f1600..63d42b12ee1 100644
--- a/megatron/rl/agent/weighted_multi_task.py
+++ b/megatron/rl/agent/weighted_multi_task.py
@@ -1,7 +1,7 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
import asyncio
-from typing import Any, AsyncGenerator, Optional, Type
+from typing import Any, Optional, Type
import numpy as np
@@ -98,7 +98,10 @@ def from_config(
)
)
- return cls(agent_configs)
+ instance = cls(agent_configs)
+ if parallel_generation_tasks is not None:
+ instance.parallel_generation_tasks = parallel_generation_tasks
+ return instance
def _distribute_counts(self, total_count: int, distribute_remainder: bool = True) -> list[int]:
"""Helper method to distribute counts according to weights.
@@ -182,33 +185,31 @@ async def get_reward_rollouts(self, request: RolloutRequest) -> list[Rollout]:
async def get_grouped_rollouts(self, request: GroupedRolloutRequest):
"""Distribute grouped rollouts across sub-agents according to weights."""
- if request.num_groups > 0:
- agent_groups = self._distribute_counts(request.num_groups)
- else:
- agent_groups = [-1 if not agent.evaluation_only else 0 for agent in self.agent_configs]
- parallel_generation_tasks = request.num_groups if request.num_groups > 0 else 10
- agent_slots = self._distribute_counts(parallel_generation_tasks, distribute_remainder=False)
+ agent_groups = self._distribute_counts(request.num_groups)
+ agent_pgts = self._distribute_counts(self.parallel_generation_tasks)
+ agent_slots = self._distribute_counts(request.num_groups, distribute_remainder=False)
agent_slots = np.array(agent_slots) / np.gcd.reduce(agent_slots)
# Create tasks for each agent with non-zero groups
- generators: list[AsyncGenerator[list[Rollout], None]] = []
- for agent, num_groups in zip(self.agents, agent_groups):
- if num_groups != 0:
+ generators = []
+ for agent, num_groups, pgt in zip(self.agents, agent_groups, agent_pgts, strict=True):
+ if num_groups > 0:
if not isinstance(agent, GroupedRolloutGenerator):
raise TypeError(
f"Agent of type {type(agent)} does not support grouped rollouts"
)
-
+ agent.parallel_generation_tasks = pgt
agent_request = GroupedRolloutRequest(
num_groups=num_groups,
+ streaming=request.streaming,
+ enforce_order=request.enforce_order,
rollouts_per_group=request.rollouts_per_group,
inference_interface=request.inference_interface,
validation=request.validation,
generation_args=request.generation_args,
filter_groups_with_same_reward=request.filter_groups_with_same_reward,
)
- group_generator = agent.get_grouped_rollouts(agent_request)
- generators.append(group_generator)
+ generators.append(agent.get_grouped_rollouts(agent_request))
else:
generators.append(None)
diff --git a/megatron/rl/inference/api.py b/megatron/rl/inference/api.py
index ae19380842e..48c5ec0a82d 100644
--- a/megatron/rl/inference/api.py
+++ b/megatron/rl/inference/api.py
@@ -11,42 +11,18 @@ class LLMChatMessage(BaseModel):
class InferenceRequest(Request):
- prompt: list[str]
- n: int | None = None
-
-
-class ChatInferenceRequest(InferenceRequest):
- prompt: list[list[LLMChatMessage]]
+ prompt: list[LLMChatMessage]
tools: list[dict] | None = None
-class GroupedInferenceRequest(InferenceRequest):
- group_size: int = 1
-
-
class InferenceResponse(BaseModel):
"""The minimum required response for an inference interface."""
- response: str
+ response: LLMChatMessage
raw_text: str | None = None
token_ids: list[int] | None = None
prompt_length: int | None = None
logprobs: list[float] | None = None
-
-
-class GroupedInferenceResponse(BaseModel):
- """An inference response which includes a list of responses."""
-
- responses: list[InferenceResponse]
-
-
-class ChatInferenceResponse(InferenceResponse):
- """The minimum required response for a chat inference interface."""
-
- response: LLMChatMessage
-
-
-class GroupedChatInferenceResponse(GroupedInferenceResponse):
- """A chat inference response which includes a list of responses."""
-
- responses: list[ChatInferenceResponse]
+ policy_epoch: list[tuple[int, int]]
+ kv_cache_epoch: list[tuple[int, int]]
+ num_evictions: int
diff --git a/megatron/rl/inference/chat_templates.py b/megatron/rl/inference/chat_templates.py
deleted file mode 100644
index 3e464842859..00000000000
--- a/megatron/rl/inference/chat_templates.py
+++ /dev/null
@@ -1,51 +0,0 @@
-# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-
-import warnings
-
-from pydantic import BaseModel, ConfigDict, Field
-from transformers import AutoTokenizer
-from transformers.tokenization_utils import PreTrainedTokenizer
-from transformers.tokenization_utils_fast import PreTrainedTokenizerFast
-
-from .api import InferenceResponse, LLMChatMessage
-
-
-class ConversationTemplate(BaseModel):
- """Transformers tokenizer based template."""
-
- model_config = ConfigDict(arbitrary_types_allowed=True)
-
- tokenizer: PreTrainedTokenizer | PreTrainedTokenizerFast = Field(repr=False)
- stop_words: list[str] = []
-
- def format(self, messages: list[LLMChatMessage], tools: list[dict] | None = None) -> str:
- return self.tokenizer.apply_chat_template(
- messages, add_generation_prompt=True, tokenize=False, tools=tools
- )
-
- def parse_response(self, responses: list[InferenceResponse]) -> list[LLMChatMessage]:
- return [
- LLMChatMessage(role="assistant", content=response.response) for response in responses
- ]
-
- @classmethod
- def from_string(cls, tokenizer_name: str) -> 'ConversationTemplate':
- if tokenizer_name == "null":
- warnings.warn(
- "Using NullConversationTemplate. This provides no chat templating to Chat requests."
- )
- return NullConversationTemplate()
- return cls(tokenizer=AutoTokenizer.from_pretrained(tokenizer_name))
-
-
-class NullConversationTemplate(ConversationTemplate):
-
- tokenizer: None = None
-
- def format(self, messages: list[LLMChatMessage], tools: list[dict] | None = None) -> str:
- return "\n".join([f"{message.content}" for message in messages]) + "\n"
-
- def parse_response(self, responses: list[InferenceResponse]) -> list[LLMChatMessage]:
- return [
- LLMChatMessage(role="assistant", content=response.response) for response in responses
- ]
diff --git a/megatron/rl/inference/inference_interface.py b/megatron/rl/inference/inference_interface.py
index e950792e72b..715f5ada9d4 100644
--- a/megatron/rl/inference/inference_interface.py
+++ b/megatron/rl/inference/inference_interface.py
@@ -1,83 +1,40 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
import asyncio
-from abc import abstractmethod
-from itertools import zip_longest
-from typing import Annotated, Any, ClassVar
-from pydantic import BaseModel, BeforeValidator, ValidationError
+from pydantic import BaseModel
from ..__init__ import GenericGenerationArgs
from ..inference.api import (
- ChatInferenceRequest,
- ChatInferenceResponse,
- GroupedChatInferenceResponse,
- GroupedInferenceResponse,
InferenceRequest,
InferenceResponse,
LLMChatMessage,
)
-from ..inference.chat_templates import ConversationTemplate
-
-
-# Used when generating n resposnes for a single prompt
-def grouper(iterable, n, fillvalue=None):
- """Fold an iterable into a list of lists of size n."""
- args = [iter(iterable)] * n
- return zip_longest(*args, fillvalue=fillvalue)
class InferenceInterface(BaseModel):
- """Inference interface that for base language models."""
+ """Inference interface for chat models."""
class Config:
arbitrary_types_allowed = True
- supports_n: ClassVar[bool] = False
-
def prepare_request(
- self, prompts: list[str], generation_args: GenericGenerationArgs
+ self, prompt: str | list[LLMChatMessage], generation_args: GenericGenerationArgs
) -> InferenceRequest:
- assert all(isinstance(p, str) for p in prompts), "Prompt must be a list of strings"
- return InferenceRequest(prompt=prompts, generation_args=generation_args)
-
- async def base_generate(self, request: InferenceRequest) -> list[InferenceResponse]:
- raise NotImplementedError(
- "Direct Inference Classes must implement the base_generate method."
- )
-
- def duplicate_requests(self, request: InferenceRequest, n: int) -> list[InferenceRequest]:
- return request.model_copy(update={'prompt': request.prompt * n})
+ prompt = [LLMChatMessage(role='user', content=prompt)] if isinstance(prompt, str) else prompt
+ return InferenceRequest(prompt=prompt, generation_args=generation_args)
- def fold_responses(
- self, responses: list[InferenceResponse], n: int
- ) -> list[GroupedInferenceResponse]:
- return [GroupedInferenceResponse(responses=x) for x in list(grouper(responses, n))]
+ async def base_generate(self, request: InferenceRequest) -> InferenceResponse:
+ assert NotImplementedError("Direct Inference Classes must implement the base_generate method.")
async def agenerate(
self, request: InferenceRequest
- ) -> list[InferenceResponse] | list[GroupedInferenceResponse]:
- if not self.supports_n and request.n is not None:
- request = self.duplicate_requests(request, request.n)
-
- generations = await self.base_generate(request)
-
- if request.n is not None:
- if self.supports_n:
- assert (
- len(generations) == len(request.prompt) * request.n
- ), f"Number of generations ({len(generations)}) does not match number of prompts ({len(request.prompt)} * {request.n})."
- else:
- assert len(generations) == len(
- request.prompt
- ), f"Number of generations ({len(generations)}) does not match number of prompts ({len(request.prompt)})."
- generations = self.fold_responses(generations, request.n)
-
- return generations
+ ) -> InferenceResponse:
+ return await self.base_generate(request)
def generate(
self, request: InferenceRequest
- ) -> list[InferenceResponse] | list[GroupedInferenceResponse]:
+ ) -> InferenceResponse:
try:
loop = asyncio.get_running_loop()
except RuntimeError:
@@ -85,58 +42,6 @@ def generate(
else:
return loop.run_until_complete(self.agenerate(request))
-
-def ensure_template(value: Any) -> ConversationTemplate:
- if isinstance(value, ConversationTemplate):
- return value
- elif isinstance(value, str):
- return ConversationTemplate.from_string(value)
- else:
- raise ValueError(f"Invalid conversation template: {value}")
-
-
-class ChatInferenceInterface(InferenceInterface):
- """Inference interface for chat models."""
-
- conversation_template: Annotated[ConversationTemplate, BeforeValidator(ensure_template)]
-
- def prepare_request(
- self, prompts: list[str | list[LLMChatMessage]], generation_args: GenericGenerationArgs
- ) -> ChatInferenceRequest:
- prompt = [
- [LLMChatMessage(role='user', content=p)] if isinstance(p, str) else p for p in prompts
- ]
- return ChatInferenceRequest(prompt=prompt, generation_args=generation_args)
-
- async def base_generate(self, request: ChatInferenceRequest) -> list[ChatInferenceResponse]:
- base_generate_results = await super().base_generate(
- InferenceRequest(
- prompt=[
- self.conversation_template.format(messages, request.tools)
- for messages in request.prompt
- ],
- generation_args=request.generation_args,
- )
- )
- chat_message_results = self.conversation_template.parse_response(base_generate_results)
- return [
- ChatInferenceResponse(
- response=chat_message, **response.model_dump(exclude={'response'})
- )
- for chat_message, response in zip(chat_message_results, base_generate_results)
- ]
-
- def generate(
- self, request: ChatInferenceRequest
- ) -> list[ChatInferenceResponse] | list[GroupedChatInferenceResponse]:
- return super().generate(request)
-
- async def agenerate(
- self, request: ChatInferenceRequest
- ) -> list[ChatInferenceResponse] | list[GroupedChatInferenceResponse]:
- return await super().agenerate(request)
-
-
class ReturnsRaw(InferenceInterface):
"""Mix-In for interface that supports returning complete string fed to the LLM."""
diff --git a/megatron/rl/inference/megatron.py b/megatron/rl/inference/megatron.py
index 4e9364b3ae9..005acfc4ffd 100644
--- a/megatron/rl/inference/megatron.py
+++ b/megatron/rl/inference/megatron.py
@@ -2,42 +2,26 @@
import asyncio
import logging
-from argparse import Namespace
+import httpx
import torch.distributed as dist
+from openai import AsyncOpenAI, DefaultAioHttpClient
from pydantic import PrivateAttr
-from megatron.core import parallel_state
-from megatron.core.inference.contexts.dynamic_context import DynamicInferenceContext
-from megatron.core.inference.engines.abstract_engine import AbstractEngine
-from megatron.core.inference.engines.dynamic_engine import DynamicInferenceEngine
-from megatron.core.inference.engines.mcore_engine import MCoreEngine
+try:
+ import h2 # noqa: F401
+ use_http2 = True
+except ImportError:
+ use_http2 = False
+
+from megatron.core.inference.config import KVCacheManagementMode
+from megatron.core.inference.engines.dynamic_engine import DynamicInferenceEngine, EngineState
from megatron.core.inference.inference_client import InferenceClient
-from megatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapper import (
- GPTInferenceWrapper,
-)
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
-from megatron.core.inference.sampling_params import SamplingParams
-from megatron.core.inference.text_generation_controllers.simple_text_generation_controller import (
- SimpleTextGenerationController,
-)
from megatron.core.models.gpt.gpt_model import GPTModel
-from megatron.core.pipeline_parallel.utils import is_pp_first_stage, is_pp_last_stage
-from megatron.core.ssm.mamba_hybrid_layer_allocation import Symbols
-from megatron.core.transformer.module import MegatronModule
-from megatron.core.utils import (
- get_attr_wrapped_model,
- get_mamba_inference_state_config_from_model,
- get_pg_size,
- log_single_rank,
-)
-from megatron.training import get_wandb_writer
+from megatron.core.utils import log_single_rank
from megatron.training.global_vars import get_args, get_tokenizer
from ..inference.inference_interface import (
- ChatInferenceInterface,
InferenceRequest,
InferenceResponse,
LLMChatMessage,
@@ -47,205 +31,61 @@
from ..server.api import InferenceServer
logger = logging.getLogger(__name__)
-
-
-## This code is copied from tools/run_text_generation_server.py
-def get_static_inference_engine(args: Namespace, model: MegatronModule) -> AbstractEngine:
- """Get the relevant backend for running inference.
-
- This function will automatically choose the TRTLLMBackend when possible,
- and default to Mcore backend if the user does not specify any backends.
- TRTLLMBackend is not implmented yet.
-
- Args:
- args (Namespace): The user arguments parsed from command line
- model (MegatronModule): The megatron model.
-
- Returns:
- AbstractBackend: The chosen backend
- """
- tokenizer = get_tokenizer()
-
- inference_wrapper_config = InferenceWrapperConfig(
- hidden_size=args.hidden_size,
- inference_batch_times_seqlen_threshold=args.inference_batch_times_seqlen_threshold,
- fp32_residual_connection=args.fp32_residual_connection,
- params_dtype=args.params_dtype,
- padded_vocab_size=args.padded_vocab_size,
- inference_max_seq_length=args.inference_max_seq_length,
- inference_max_requests=(
- args.inference_max_batch_size if args.inference_max_batch_size is not None else 1
- ),
- nccl_all_reduce_for_prefill=args.nccl_all_reduce_for_prefill,
- )
-
- inference_wrapped_model = GPTInferenceWrapper(model, inference_wrapper_config)
- pg_collection = get_attr_wrapped_model(model, "pg_collection")
- pp_group = pg_collection.pp
- text_generation_controller = SimpleTextGenerationController(
- inference_wrapped_model=inference_wrapped_model,
- tokenizer=tokenizer,
- pp_group=pp_group,
- )
- return MCoreEngine(
- text_generation_controller=text_generation_controller,
- max_batch_size=(
- args.inference_max_batch_size if args.inference_max_batch_size is not None else 1
- ),
- )
-
-
-## This code is copied from tools/run_text_generation_server.py
-def get_dynamic_inference_engine(
- args: Namespace,
- model: MegatronModule,
- inference_logging_step_interval: int = 0,
- metrics_writer = None
-) -> AbstractEngine:
- """Get the relevant backend for running inference.
-
- This function will automatically choose the TRTLLMBackend when possible,
- and default to Mcore backend if the user does not specify any backends.
- TRTLLMBackend is not implmented yet.
-
- Args:
- args (Namespace): The user arguments parsed from command line
- model (MegatronModule): The megatron model.
- inference_logging_step_interval (int): Step interval for logging inference metrics.
- metrics_writer: Metrics writer (wandb module) for logging.
-
- Returns:
- AbstractBackend: The chosen backend
- """
- tokenizer = get_tokenizer()
-
- enable_cuda_graph = args.cuda_graph_impl == "local"
-
- mamba_inference_state_config = get_mamba_inference_state_config_from_model(model)
-
- # DynamicInferenceContext must use the inference model's TP / PP size, not the
- # training TP / PP size from global args. The inference model may have a custom
- # ProcessGroupCollection with a different TP / PP size.
- pg_collection = get_attr_wrapped_model(model, "pg_collection")
- tp_group = getattr(pg_collection, 'tp', None) if pg_collection is not None else None
- if tp_group is not None:
- inference_tp_size = get_pg_size(tp_group)
- else:
- inference_tp_size = args.tensor_model_parallel_size
- pp_group = getattr(pg_collection, 'pp', None) if pg_collection is not None else None
- if pp_group is not None:
- inference_pp_size = get_pg_size(pp_group)
- else:
- inference_pp_size = args.pipeline_model_parallel_size
-
- # Inference context.
- inference_context = DynamicInferenceContext(
- params_dtype=args.params_dtype,
- num_layers=args.num_layers // inference_pp_size,
- kv_channels=args.kv_channels,
- num_attention_heads=(
- args.num_query_groups if args.group_query_attention else args.num_attention_heads
- ),
- max_sequence_length=args.inference_max_seq_length,
- num_cuda_graphs=(
- args.inference_dynamic_batching_num_cuda_graphs if enable_cuda_graph else None
- ),
- block_size_tokens=args.inference_dynamic_batching_block_size,
- buffer_size_gb=args.inference_dynamic_batching_buffer_size_gb,
- max_requests=args.inference_dynamic_batching_max_requests,
- max_tokens=args.inference_dynamic_batching_max_tokens,
- pg_collection=pg_collection, # TP/PP sizes are derived from the model's pg_collection.
- materialize_only_last_token_logits=True,
- mamba_inference_state_config=mamba_inference_state_config,
- cache_mla_latent=args.multi_latent_attention and args.cache_mla_latents,
- kv_lora_rank=args.kv_lora_rank if args.multi_latent_attention else None,
- qk_pos_emb_head_dim=args.qk_pos_emb_head_dim,
- use_cuda_graphs_for_non_decode_steps=not args.decode_only_cuda_graphs,
- use_flashinfer_fused_rope=None,
- unified_memory_level=args.inference_dynamic_batching_unified_memory_level,
- cuda_graph_max_tokens=args.inference_dynamic_batching_cuda_graph_max_tokens,
- cuda_graph_mixed_prefill_count=args.inference_dynamic_batching_cuda_graph_mixed_prefill_count,
- metrics_writer=metrics_writer,
- persist_cuda_graphs=args.rl_training_cuda_graphs,
- offload_kv_cache=args.rl_offload_kv_cache_during_training
- )
-
- inference_wrapped_model = GPTInferenceWrapper(model, args, inference_context, pg_collection=pg_collection)
-
- inference_wrapped_model.model_is_pipeline_parallel = not (
- is_pp_first_stage(pg_collection.pp) and is_pp_last_stage(pg_collection.pp)
- )
-
- pp_group = getattr(pg_collection, "pp", None)
- text_generation_controller = SimpleTextGenerationController(
- inference_wrapped_model=inference_wrapped_model,
- tokenizer=tokenizer,
- pp_group=pp_group,
- )
-
- return DynamicInferenceEngine(
- controller=text_generation_controller,
- context=inference_context,
- random_seed=args.seed,
- track_paused_request_events=args.inference_dynamic_batching_track_paused_request_events,
- enable_chunked_prefill=not args.disable_chunked_prefill,
- inference_logging_step_interval=inference_logging_step_interval,
- pg_collection=pg_collection,
- )
-
+logging.getLogger("httpx").setLevel(logging.WARNING)
class MegatronLocal(InferenceServer, ReturnsTokens, ReturnsRaw):
"""Interface to use MCoreEngine directly as an inference engine."""
+ host: str
+ port: int
+
_client: InferenceClient = PrivateAttr(None)
_inference_engine: DynamicInferenceEngine = PrivateAttr(None)
+ _rl_kv_cache_management_mode: KVCacheManagementMode = PrivateAttr(None)
+ _openai_client: AsyncOpenAI = PrivateAttr(None)
- async def base_generate(self, request: InferenceRequest):
-
- if any(isinstance(p, LLMChatMessage) for p in request.prompt):
- raise ValueError(
- "MegatronLocal does not support chat requests."
- "Use MegatronChatLocal to apply chat templating."
- )
- assert all(
- isinstance(p, str) for p in request.prompt
- ), "MegatronLocal only supports string prompts."
-
- assert self._client is not None, "Client is not initialized"
-
+ async def base_generate(self, request: InferenceRequest) -> InferenceResponse:
tokenizer = get_tokenizer()
args = get_args()
- sampling_params = SamplingParams(
- num_tokens_to_generate=None,
- num_tokens_total=request.generation_args.max_tokens,
+ # Use the shared, optimized client instead of spinning up a new one
+ client = self._openai_client
+
+ # Things that may be problematic when doing this switch
+ # - Add BOS token
+ # - Skip prompt logprobs
+ response = await client.chat.completions.create(
+ model="",
+ messages=[message.model_dump() for message in request.prompt],
temperature=request.generation_args.temperature or 1.0,
- top_k=request.generation_args.top_k or 0,
top_p=request.generation_args.top_p or 0.0,
- termination_id=self._inference_engine.controller.tokenizer.eod,
- return_log_probs=True,
- skip_prompt_log_probs=True,
- add_BOS=(not args.rl_skip_bos_token and tokenizer.bos is not None),
+ n=1,
+ logprobs=True,
+ extra_body={
+ "skip_prompt_log_probs": True,
+ "add_BOS": (not args.rl_skip_bos_token and tokenizer.bos is not None),
+ },
+ )
+
+ choice = response.choices[0]
+
+ return InferenceResponse(
+ # TODO: Handle tool calls and reasoning in LLMChatMessage
+ response=LLMChatMessage(**choice.message.model_dump(include={'role', 'content'})),
+ raw_text=choice.raw_text,
+ token_ids=choice.prompt_token_ids + choice.generation_token_ids,
+ logprobs=choice.generation_log_probs,
+ prompt_length=len(choice.prompt_token_ids),
+ policy_epoch=choice.policy_epoch,
+ kv_cache_epoch=choice.kv_cache_epoch,
+ num_evictions=getattr(choice, 'num_evictions', 0),
)
- requests = [
- self._client.add_request(prompt=prompt, sampling_params=sampling_params)
- for prompt in request.prompt
- ]
- records = await asyncio.gather(*requests)
- responses = [record[-1] for record in records]
- return [
- InferenceResponse(
- response=r.generated_text,
- raw_text=p + r.generated_text,
- token_ids=r.prompt_tokens.tolist() + r.generated_tokens,
- logprobs=r.generated_log_probs,
- prompt_length=len(r.prompt_tokens),
- )
- for p, r in zip(request.prompt, responses)
- ]
@classmethod
async def launch(cls, model: GPTModel, **kwargs):
+ # Import here to avoid circular imports
+ from megatron.inference.utils import get_dynamic_inference_engine
+
args = get_args()
tokenizer = get_tokenizer()
@@ -256,59 +96,96 @@ async def launch(cls, model: GPTModel, **kwargs):
"WARNING: Tokenizer has no BOS token so prompt will not have BOS token",
)
- # Get inference logging configuration from args
- log_inference_wandb = args.inference_wandb_logging
- inference_logging_step_interval = args.inference_logging_step_interval
-
- # Get metrics writer if logging is enabled and on the logging rank
- # Use the same rank convention as training (last rank logs)
- metrics_writer = None
- if (
- inference_logging_step_interval > 0
- and log_inference_wandb
- and args.rank == (args.world_size - 1)
- ):
- metrics_writer = get_wandb_writer()
- if metrics_writer is None:
- log_single_rank(
- logger,
- logging.WARNING,
- "WARNING: --rl-inference-logging-step-interval is set but no metrics writer "
- "wandb module is available. Inference logging will be disabled.",
- )
-
- inference_engine: DynamicInferenceEngine = get_dynamic_inference_engine(
- args, model, inference_logging_step_interval, metrics_writer
- )
+ inference_engine: DynamicInferenceEngine = get_dynamic_inference_engine(model=model)
dp_addr = await inference_engine.start_listening_to_data_parallel_coordinator(
inference_coordinator_port=41521, launch_inference_coordinator=True,
)
+
if dist.get_rank() == 0:
- # TODO: We have to do this only on the rank 0 process, should be fixed in the future when we have support for multiple inference clients. !2278
+ from megatron.core.inference.text_generation_server.dynamic_text_gen_server import start_text_gen_server
+
client = InferenceClient(inference_coordinator_address=dp_addr)
- await client.start()
+ client.start()
+
+ start_text_gen_server(
+ coordinator_addr=dp_addr,
+ tokenizer=inference_engine.controller.tokenizer,
+ rank=dist.get_rank(),
+ server_port=kwargs.get('port', 8294),
+ parsers=[],
+ verbose=kwargs.get('verbose', False),
+ )
else:
client = None
+
launched_server = cls(**kwargs)
launched_server._client = client
launched_server._inference_engine = inference_engine
+ launched_server._rl_kv_cache_management_mode = KVCacheManagementMode(
+ args.rl_kv_cache_management_mode
+ )
+
+ concurrency_limit = args.grpo_prompts_per_step * args.grpo_group_size * args.rl_parallel_generation_tasks
+ custom_limits = httpx.Limits(
+ max_connections=concurrency_limit,
+ max_keepalive_connections=concurrency_limit,
+ )
+ http_client = DefaultAioHttpClient(
+ timeout=None,
+ limits=custom_limits,
+ http2=use_http2
+ )
+
+ launched_server._openai_client = AsyncOpenAI(
+ base_url=f"http://{launched_server.host}:{launched_server.port}",
+ api_key="NONE",
+ http_client=http_client
+ )
return launched_server
async def kill(self):
+ # Gracefully close the shared OpenAI client connections
+ if self._openai_client is not None:
+ await self._openai_client.close()
+
+ if dist.get_rank() == 0:
+ self._client.pause_engines()
+ await self._inference_engine.wait_until(EngineState.PAUSED)
+
+ if dist.get_rank() == 0:
+ self._client.stop_engines()
+ await self._inference_engine.wait_until(EngineState.STOPPED)
+
if dist.get_rank() == 0:
- await self._client.stop_engines()
- await self._inference_engine.stopped.wait()
+ self._client.shutdown_coordinator()
+ self._client.stop()
+
+ if dist.get_rank() == 0:
+ from megatron.core.inference.text_generation_server.dynamic_text_gen_server import stop_text_gen_server
+ stop_text_gen_server()
+
+ def set_generation_epoch(self, generation_epoch: int):
+ if dist.get_rank() == 0:
+ self._client.set_generation_epoch(generation_epoch)
async def suspend(self):
if dist.get_rank() == 0:
- await self._client.pause_engines()
- await self._inference_engine.paused.wait()
+ self._client.pause_engines()
+ await self._inference_engine.wait_until(EngineState.PAUSED)
- async def resume(self):
if dist.get_rank() == 0:
- self._client.unpause_engines()
- await self._inference_engine.running.wait()
+ self._client.suspend_engines()
+ await self._inference_engine.wait_until(EngineState.SUSPENDED)
+ async def resume(self):
+ if self._inference_engine._state_events[EngineState.RUNNING].is_set():
+ return
+
+ if dist.get_rank() == 0:
+ self._client.resume_engines()
+ await self._inference_engine.wait_until(EngineState.RESUMED)
-class MegatronChatLocal(ChatInferenceInterface, MegatronLocal): ...
+ if dist.get_rank() == 0:
+ self._client.unpause_engines()
+ await self._inference_engine.wait_until(EngineState.RUNNING)
diff --git a/megatron/rl/parallel_utils.py b/megatron/rl/parallel_utils.py
index 9cab73daba9..da4ee8aa4cf 100644
--- a/megatron/rl/parallel_utils.py
+++ b/megatron/rl/parallel_utils.py
@@ -21,6 +21,7 @@ def build_inference_pg_collection(
ep_size: Optional[int] = None,
expt_tp_size: Optional[int] = None,
use_tp_pp_dp_mapping: bool = False,
+ rank_offset: int = 0,
) -> ProcessGroupCollection:
"""
Build a ProcessGroupCollection for an RL inference model with custom parallelism.
@@ -37,6 +38,8 @@ def build_inference_pg_collection(
ep_size: Expert parallel size. Defaults to training's EP size.
expt_tp_size: Expert tensor parallel size. Defaults to training's expert TP size.
use_tp_pp_dp_mapping: If True, use 'tp-pp-dp' order; otherwise 'tp-dp-pp'.
+ rank_offset: Starting rank when the grid doesn't span the entire communication world.
+ Used in non-collocated mode where model ranks don't start from 0.
Returns:
ProcessGroupCollection configured for the inference model.
@@ -78,13 +81,15 @@ def build_inference_pg_collection(
# Order: tp-cp-pp-dp
decoder_grid = HyperCommGrid(
[tp_size, cp_size, pp_size, dp_size],
- ["tp", "cp", "pp", "dp"]
+ ["tp", "cp", "pp", "dp"],
+ rank_offset=rank_offset
)
else:
# Order: tp-cp-dp-pp (default)
decoder_grid = HyperCommGrid(
[tp_size, cp_size, dp_size, pp_size],
- ["tp", "cp", "dp", "pp"]
+ ["tp", "cp", "dp", "pp"],
+ rank_offset=rank_offset
)
# Create dense layer groups from decoder_grid
@@ -105,13 +110,15 @@ def build_inference_pg_collection(
# Order: tp-ep-pp-dp
expert_grid = HyperCommGrid(
[expt_tp_size, ep_size, pp_size, expt_dp_size],
- ["tp", "ep", "pp", "dp"]
+ ["tp", "ep", "pp", "dp"],
+ rank_offset=rank_offset
)
else:
# Order: tp-ep-dp-pp (default)
expert_grid = HyperCommGrid(
[expt_tp_size, ep_size, expt_dp_size, pp_size],
- ["tp", "ep", "dp", "pp"]
+ ["tp", "ep", "dp", "pp"],
+ rank_offset=rank_offset
)
# Verify PP groups match between decoder and expert grids (required by mpu)
diff --git a/megatron/rl/rl_utils.py b/megatron/rl/rl_utils.py
index 364a80db81e..30c941d61dc 100644
--- a/megatron/rl/rl_utils.py
+++ b/megatron/rl/rl_utils.py
@@ -8,7 +8,8 @@
import itertools
import math
import logging
-import pickle
+import json
+import os
from collections import Counter, defaultdict
from contextlib import contextmanager, nullcontext
from dataclasses import dataclass
@@ -23,7 +24,7 @@
from torch.utils.tensorboard import SummaryWriter
from megatron.core import mpu
-from megatron.core.datasets.megatron_tokenizer import MegatronLegacyTokenizer
+from megatron.core.packed_seq_params import PackedSeqParams
from megatron.core.full_cuda_graph import FullCudaGraphWrapper
from megatron.core.models.common.language_module.language_module import LanguageModule
from megatron.core.num_microbatches_calculator import reconfigure_num_microbatches_calculator
@@ -31,14 +32,21 @@
from megatron.core.pipeline_parallel import get_forward_backward_func
from megatron.core.pipeline_parallel.utils import is_pp_last_stage, get_pp_last_rank
from megatron.core.rerun_state_machine import RerunDataIterator
+from megatron.core.tokenizers import MegatronTokenizer
+from megatron.core.tokenizers.text.libraries.huggingface_tokenizer import HuggingFaceTokenizer
from megatron.core.transformer.cuda_graphs import _CudagraphGlobalRecord
from megatron.core.transformer.enums import CudaGraphScope
-from megatron.core.transformer.utils import toggle_cuda_graphs
+from megatron.core.transformer.utils import (
+ toggle_cuda_graphs,
+ transition_moe_cudagraphs,
+)
+from megatron.core.inference.utils import set_decode_expert_padding
from megatron.core.resharding.refit import swap_model_weights
from megatron.core.inference.unified_memory import (
advise_managed_module_parameters_preferred_location,
prefetch_managed_module_parameters,
)
+from megatron.core.inference.utils import device_memory_summary
from megatron.core.utils import get_asyncio_loop, log_single_rank
from megatron.rl.sequence_packing_utils import (
get_microbatch_dataloader,
@@ -55,12 +63,15 @@
EvaluationRequest,
EvaluationResponse,
GroupedRolloutRequest,
+ GroupedRollouts,
RewardEvaluationResult,
Rollout,
+ RolloutGroup,
+ Rollouts,
TokenRollout,
)
from megatron.rl.agent.weighted_multi_task import WeightedMultiTask
-from megatron.rl.inference.megatron import MegatronChatLocal, MegatronLocal
+from megatron.rl.inference.megatron import MegatronLocal
from megatron.rl.logging import LOG_DIR as lang_rl_log_dir
from megatron.rl.logging import log as lang_rl_log
from megatron.rl.server.inference.inference_interface_server import InferenceInterfaceServer
@@ -70,7 +81,6 @@
get_tokenizer,
get_wandb_writer,
)
-from megatron.training.tokenizer.tokenizer import CustomTikTokenizer, _HuggingFaceTokenizer
from megatron.training.utils import (
get_ltor_masks_and_position_ids,
get_nvtx_range,
@@ -94,16 +104,64 @@
_GLOBAL_PACKING_CONTEXT = None
+# Track whether the inference model is currently paused (offloaded to CPU).
+# Model starts on GPU after creation and is used immediately, so starts as False.
+_INFERENCE_MODEL_IS_PAUSED = False
+
+
+def _torch_saver_swap_inference_model(*, to_cpu: bool) -> None:
+ """Swap RL inference model weights between CPU and GPU using torch_memory_saver.
+
+ Uses torch_memory_saver.pause()/resume() to transfer inference model weights
+ that were allocated within a torch_memory_saver.region() context.
+
+ Args:
+ to_cpu: If True, move weights to CPU (pause). If False, restore weights to GPU (resume).
+ """
+ global _INFERENCE_MODEL_IS_PAUSED
+
+ if not HAVE_TORCH_MEMORY_SAVER:
+ raise RuntimeError(
+ "torch_memory_saver is required for inference model offloading when not using UVM. "
+ "Please install it: pip install torch_memory_saver "
+ "(see https://github.com/fzyzcjy/torch_memory_saver)"
+ )
+
+ tag = "rl_inference_model"
+ if to_cpu:
+ if not _INFERENCE_MODEL_IS_PAUSED:
+ print_rank_0(f"torch_memory_saver: pausing {tag}, before: {device_memory_summary()}")
+ torch_memory_saver.pause(tag)
+ _INFERENCE_MODEL_IS_PAUSED = True
+ print_rank_0(f"torch_memory_saver: paused {tag}, after: {device_memory_summary()}")
+ else:
+ if _INFERENCE_MODEL_IS_PAUSED:
+ print_rank_0(f"torch_memory_saver: resuming {tag}, before: {device_memory_summary()}")
+ torch_memory_saver.resume(tag)
+ _INFERENCE_MODEL_IS_PAUSED = False
+ print_rank_0(f"torch_memory_saver: resumed {tag}, after: {device_memory_summary()}")
+
+
def _maybe_prefetch_separate_inference_model_weights(model_core, *, to_cpu: bool) -> None:
- """Prefetch RL *separate inference model* weights to CPU/GPU (UVM-only path).
+ """Prefetch RL *separate inference model* weights to CPU/GPU.
- Gated only by user args; this assumes the separate inference model was allocated with UVM when enabled.
+ Supports two modes:
+ 1. UVM-based offloading (when --rl-inference-model-unified-memory-level=1)
+ 2. torch_memory_saver-based offloading (when offloading is enabled but UVM is not)
+
+ Gated by user args; this assumes the separate inference model was allocated
+ with UVM or torch_memory_saver when enabled.
"""
args = get_args()
if not args.rl_offload_inference_model_weights_when_idle:
return
+
+ # Check for torch_memory_saver path (when offloading is enabled but UVM is not)
if args.rl_inference_model_unified_memory_level != 1:
+ _torch_saver_swap_inference_model(to_cpu=to_cpu)
return
+
+ # UVM-based path (when UVM level is 1)
device = -1 if to_cpu else int(torch.cuda.current_device())
# Note: include_buffers=False because buffers created with explicit device= in register_buffer()
# are not allocated via the UVM mempool and will fail UVM operations. Only parameters are UVM-allocated.
@@ -209,31 +267,29 @@ def verify_model_weights_swap(
if inf_was_training:
inf_core.train()
-Rollouts = list[TokenRollout | Rollout]
-GroupedRollouts = list[Rollouts]
@dataclass(slots=True)
class RolloutStats:
- mean_reward: float
- mean_length: float
- mean_length_std: float
- max_length: float
- min_length: float
- reward_means: list[float]
- reward_stds: list[float]
- rewards: list[float]
+ rewards: list[list[float]] # inner list is for a group
+ env_ids: list[str] # same length as len(rewards)
+ turn_lens: list[list[int]] # token lengths of turns, grouped.
+ traj_lens: list[list[int]] # all turns comprise one trajectory.
+ num_turns: None | list[list[int]] # num_turns per traj
+ advantages: None | list[list[float]]
min_piold_to_inf_prob: None | float
max_piold_to_inf_prob: None | float
mean_piold_to_inf_prob: None | float
min_inf_train_prob_abs_diff: None | float
max_inf_train_prob_abs_diff: None | float
mean_inf_train_prob_abs_diff: None | float
- advantages: None | list[list[float]]
min_inf_prob: None | float
max_inf_prob: None | float
mean_inf_prob: None | float
- num_turns: list[int] # num_turns per traj
+ policy_epoch: list[list[int]]
+ kv_cache_epoch: list[list[int]]
+ completed_epochs: list[list[int]]
+ num_evictions: list[list[int]]
# Runtime state container for RL-specific data that shouldn't be checkpointed
@@ -386,36 +442,13 @@ def get_agent(args, parallel_generation_tasks: int | None = None):
def get_inference_interface(args, loop, model):
global _INFERENCE_INTERFACE
if _INFERENCE_INTERFACE is None:
- rank = torch.distributed.get_rank()
- if rank == 0 and args.langrl_external_server:
- if args.langrl_inference_server_type == 'inplace_megatron':
- _INFERENCE_INTERFACE = loop.run_until_complete(
- InferenceInterfaceServer.launch(MegatronLocal, model=model[0])
- )
- elif args.langrl_inference_server_type == 'inplace_megatron_chat':
- _INFERENCE_INTERFACE = loop.run_until_complete(
- InferenceInterfaceServer.launch(
- MegatronChatLocal,
- model=model[0],
- conversation_template=args.langrl_inference_server_conversation_template,
- )
- )
- else:
- raise ValueError(f"Unknown inference_server_type {args.inference_server_type}")
- else:
- if args.langrl_inference_server_type == 'inplace_megatron':
- _INFERENCE_INTERFACE = loop.run_until_complete(MegatronLocal.launch(model[0]))
- elif args.langrl_inference_server_type == 'inplace_megatron_chat':
- _INFERENCE_INTERFACE = loop.run_until_complete(
- MegatronChatLocal.launch(
- model[0],
- conversation_template=args.langrl_inference_server_conversation_template,
- )
- )
- else:
- raise ValueError(
- f"Unknown inference_server_type {args.langrl_inference_server_type}"
- )
+ _INFERENCE_INTERFACE = loop.run_until_complete(
+ MegatronLocal.launch(
+ model[0],
+ host='0.0.0.0',
+ port=8294,
+ verbose=args.inference_text_gen_server_logging)
+ )
return _INFERENCE_INTERFACE
@@ -424,11 +457,11 @@ def get_inference_interface(args, loop, model):
def get_rollout_generator(args, inference_interface, n_prompts, samples_per_group):
global _ROLLOUT_GENERATOR
- if not args.rl_partial_rollouts or _ROLLOUT_GENERATOR is None:
+ if not (streaming := args.rl_partial_rollouts) or _ROLLOUT_GENERATOR is None:
agent = get_agent(args, parallel_generation_tasks=args.rl_parallel_generation_tasks)
- # Collect Rollouts
request = GroupedRolloutRequest(
- num_groups=-1 if args.rl_partial_rollouts else n_prompts,
+ num_groups=args.rl_generation_batch_size,
+ streaming=streaming,
rollouts_per_group=samples_per_group,
inference_interface=inference_interface,
generation_args={
@@ -438,6 +471,7 @@ def get_rollout_generator(args, inference_interface, n_prompts, samples_per_grou
'top_k': args.rl_default_top_k,
},
filter_groups_with_same_reward=args.grpo_filter_groups_with_same_reward,
+ enforce_order=args.rl_enforce_generation_order,
)
_ROLLOUT_GENERATOR = agent.get_grouped_rollouts(request)
return _ROLLOUT_GENERATOR
@@ -461,15 +495,22 @@ def get_environment_rollouts(
nvtx_range = get_nvtx_range()
if args.rl_offload_optimizer_during_inference:
- with nvtx_range("offload-optimizer-state-and-grad-buffers-during-inference"):
- model[0].offload_grad_buffers()
- optimizer.offload_to_cpu()
-
- # If we have seperate training and inference models we to refit weights from the training model to the inference model.
- if inference_model is not None:
+ with nvtx_range("rl/offload-optimizer-before-inference", time=True):
+ if not args.rl_training_cuda_graphs:
+ with nvtx_range("rl/offload/grad-buffers", time=True):
+ model[0].offload_grad_buffers()
+ else:
+ logger.warning(
+ "Gradient buffers will not be offloaded when training cudagraphs are enabled!")
+ with nvtx_range("rl/offload/optimizer-state", time=True):
+ optimizer.offload_to_cpu()
+
+ # If we have separate training and inference models we to refit weights from the training model to the inference model.
+ has_separate_inference_model = inference_model is not None
+ if has_separate_inference_model:
# If the separate inference model weights were prefetched to CPU while idle, bring them
# back to GPU before refit/copy and before any CUDA-graph'd inference.
- with nvtx_range("prefetch-inference-model-weights-to-gpu"):
+ with nvtx_range("rl/prefetch-weights-to-gpu", time=True):
inf_core = unwrap_model(inference_model[0])
_maybe_prefetch_separate_inference_model_weights(inf_core, to_cpu=False)
swap_model_weights(model, inference_model, args.refit_method)
@@ -484,23 +525,20 @@ def get_environment_rollouts(
inference_model = model
inference_pg_collection = get_attr_wrapped_model(inference_model[0], "pg_collection")
- assert (
- n_prompts % get_pg_size(inference_pg_collection.ep) == 0
- ), "n_prompts must be divisible by data_parallel_world_size"
+ pg_size = get_pg_size(inference_pg_collection.ep)
+ assert (n_prompts % pg_size == 0), f"{n_prompts=} must be divisible by {pg_size=}"
- with nvtx_range("rollout-collection"):
+ with nvtx_range("rl/rollout-collection", time=True):
loop = get_asyncio_loop()
with megatron_rl_inference_mode(
inference_model,
optimizer,
args.cuda_graph_impl,
- args.rl_reset_cuda_graphs,
False, # offload optimizer during rollout collection is handled above
- args.rl_offload_kv_cache_during_training,
- args.rl_remove_kv_cache_during_training,
+ training_model=model if has_separate_inference_model else None,
) as inference_interface:
- with nvtx_range("inference-setup"):
+ with nvtx_range("rl/inference-setup", time=True):
# Asyncronously run inference and rollout collection
rollout_generator = get_rollout_generator(
args, inference_interface, n_prompts, samples_per_group
@@ -508,7 +546,7 @@ def get_environment_rollouts(
# NOTE(jbarker): we need to double check this when using PP>1
rank = torch.distributed.get_rank()
- with nvtx_range("collect-rollouts"):
+ with nvtx_range("rl/collect-rollouts", time=True):
if rank == 0:
log_single_rank(
logger,
@@ -518,6 +556,10 @@ def get_environment_rollouts(
rollouts = [
loop.run_until_complete(anext(rollout_generator)) for _ in range(n_prompts)
]
+ # In deterministic mode, sort rollouts by problem_id for consistent ordering
+ # regardless of completion order due to system timing jitter.
+ if torch.are_deterministic_algorithms_enabled():
+ rollouts.sort(key=lambda group: group[0].problem_id if group and group[0].problem_id else "")
if not args.rl_partial_rollouts:
while True:
try:
@@ -529,25 +571,27 @@ def get_environment_rollouts(
# Just set up space to collect the rollouts
rollouts = [[None for _ in range(samples_per_group)] for _ in range(n_prompts)]
- with nvtx_range("sync-rollouts"):
+ with nvtx_range("rl/sync-rollouts", time=True):
# Wait for Rollouts to be collected
# TODO(jbarker): double check why this isn't causing rank 0 memory allocations
torch.distributed.broadcast_object_list(rollouts, src=0)
logger.debug(f"Got rollouts on rank {rank}")
if args.rl_offload_optimizer_during_inference:
- with nvtx_range("restore-optimizer-state-and-grad-buffers-after-inference"):
- model[0].restore_grad_buffers()
- optimizer.restore_from_cpu()
+ with nvtx_range("rl/restore-optimizer-after-inference", time=True):
+ with nvtx_range("rl/restore/grad-buffers", time=True):
+ model[0].restore_grad_buffers()
+ with nvtx_range("rl/restore/optimizer-state", time=True):
+ optimizer.restore_from_cpu()
if lang_rl_log_dir and rank == get_pg_rank(inference_pg_collection.tp):
with open(
lang_rl_log_dir
+ f'/rollouts_rank{rank}_iteration{args.curr_iteration}_'
- + f'{Path(args.langrl_env_config).stem}.pkl',
- 'wb',
+ + f'{Path(args.langrl_env_config).stem}.json',
+ 'w',
) as f:
- pickle.dump(rollouts, f)
+ json.dump([[r.model_dump() for r in group] for group in rollouts], f)
return rollouts
@@ -614,18 +658,33 @@ def get_logprobs(model, tokens, position_ids, no_grad=False, sequence_packing=Fa
"""
args = get_args()
- # Ensure packed_seq_params is always provided for CUDA graph signature consistency
- if packed_seq_params is None and sequence_packing:
- packed_seq_params = get_default_packed_seq_params(
- seq_length=tokens.shape[1],
- max_sequences_per_bin=args.rl_sequence_packing_max_sequences_per_bin,
- device=tokens.device,
- )
+ # Ensure packed_seq_params is always provided for CUDA graph signature consistency.
+ # When sequence_packing is enabled, construct from packing config (max_sequences_per_bin).
+ # When sequence_packing is disabled, construct a single-sequence default so the CUDA
+ # graph signature matches the training forward_step in train_rl.py.
+ # This is necessary because reference logprobs steps will reuse the training forward graph.
+ if packed_seq_params is None:
+ if sequence_packing:
+ packed_seq_params = get_default_packed_seq_params(
+ seq_length=tokens.shape[1],
+ max_sequences_per_bin=args.rl_sequence_packing_max_sequences_per_bin,
+ device=tokens.device,
+ )
+ else:
+ cu_seqlens = torch.tensor([0, tokens.shape[1]], dtype=torch.int32, device=tokens.device)
+ packed_seq_params = PackedSeqParams(
+ qkv_format='thd',
+ cu_seqlens_q=cu_seqlens,
+ cu_seqlens_kv=cu_seqlens,
+ max_seqlen_q=tokens.shape[1],
+ max_seqlen_kv=tokens.shape[1],
+ total_tokens=tokens.shape[1],
+ )
nvtx_range = get_nvtx_range()
- with nvtx_range("get-logprobs", time=False):
- with nvtx_range("forward-pass", time=False):
+ with nvtx_range("rl/get-logprobs", time=True):
+ with nvtx_range("rl/forward-pass", time=True):
# TODO(vitalyk): use fp16/bf16 as a function argument. Do not use args.
attention_mask_for_forward = None
@@ -652,7 +711,7 @@ def get_logprobs(model, tokens, position_ids, no_grad=False, sequence_packing=Fa
return logits_or_hidden_states
else:
logits = logits_or_hidden_states
- with nvtx_range("log-softmax", time=False):
+ with nvtx_range("rl/log-softmax", time=True):
# We do not need logprobs for the n+1 token.
logprobs = selective_log_softmax(logits[:, :-1, :], tokens[:, 1:])
return logprobs
@@ -686,7 +745,7 @@ def calculate_grpo_advantages(rewards: list[list[float]], num_turns: list[list[i
def compute_group_stats(
- rollouts: GroupedRollouts, tokenizer: MegatronLegacyTokenizer, seq_len: int,
+ rollouts: GroupedRollouts, tokenizer: MegatronTokenizer, seq_len: int,
) -> RolloutStats:
"""Add group-based rollout stats for logging.
@@ -701,18 +760,26 @@ def compute_group_stats(
# TODO (rkirby) Maybe do some of this after the tensor building
group_reward_means = []
group_reward_stds = []
- group_length_means = []
- group_length_stds = []
- group_length_maxs = []
- group_length_mins = []
+ turn_lens = []
+ traj_lens = []
rewards = []
+ env_ids = []
+ group_reward_ids = []
num_turns = [] # num_turns per traj
+ all_policy_epoch = []
+ all_kv_cache_epoch = []
+ all_completed_epochs = []
+ all_num_evictions = []
for group in rollouts:
group_rewards = []
- group_lengths = []
+ group_traj_lengths = []
+ group_turn_lengths = []
group_num_turns = []
+ group_policy_epoch = []
+ group_kv_epoch = []
+ group_completed_epochs = []
+ group_num_evictions = []
for rollout in group:
- group_num_turns.append(len(rollout.trajectory))
if isinstance(rollout, TokenRollout):
for turn_traj in rollout.trajectory:
detokenized_traj = tokenizer.detokenize(turn_traj)
@@ -727,29 +794,39 @@ def compute_group_stats(
lang_rl_log(
f"Rollout: [{rollout.env_id}] [{rollout.reward} : {len(rollout.trajectory)} chars] {rollout.trajectory}"
)
+ group_num_turns.append(len(rollout.trajectory))
group_rewards.append(rollout.reward)
- #TODO(vitalyk): What is the semantics behind traj length in multiturn? Should we take the last only? Average them instead of extending?
- group_lengths.extend(len(t) for t in rollout.trajectory)
-
- group_length_maxs.append(max(group_lengths))
- group_length_mins.append(min(group_lengths))
- group_reward_means.append(np.mean(group_rewards))
- group_reward_stds.append(np.std(group_rewards))
+ roll_turn_lens = [len(t) for t in rollout.trajectory]
+ group_turn_lengths.extend(roll_turn_lens)
+ group_traj_lengths.append(sum(roll_turn_lens))
+ assert rollout.policy_epoch, "Rollout has no policy_epoch data"
+ assert rollout.kv_cache_epoch, "Rollout has no kv_cache_epoch data"
+ group_policy_epoch.append(min(turn[0][1] for turn in rollout.policy_epoch))
+ group_kv_epoch.append(min(turn[0][1] for turn in rollout.kv_cache_epoch))
+ group_completed_epochs.extend(turn[-1][1] for turn in rollout.policy_epoch)
+ group_num_evictions.append(sum(rollout.num_evictions))
+ all_policy_epoch.append(group_policy_epoch)
+ all_kv_cache_epoch.append(group_kv_epoch)
+ all_completed_epochs.append(group_completed_epochs)
+ all_num_evictions.append(group_num_evictions)
+ traj_lens.append(group_traj_lengths)
+ turn_lens.append(group_turn_lengths)
+ env_ids.append(group[0].env_id) # All rollouts in a group share the env_id by design.
rewards.append(group_rewards)
- group_length_means.append(np.mean(group_lengths))
- # https://arxiv.org/abs/2504.21233 reports that lens variants hurts.
+ # https://arxiv.org/abs/2504.21233 reports that lens variance hurts.
# Let's track this.
- group_length_stds.append(np.std(group_lengths))
num_turns.append(group_num_turns)
stats = RolloutStats(
- mean_reward=np.mean(group_reward_means),
- mean_length=np.mean(group_length_means),
- mean_length_std=np.mean(group_length_stds),
- max_length=np.max(group_length_maxs),
- min_length=np.min(group_length_mins),
- reward_means=group_reward_means,
- reward_stds=group_reward_stds,
+ traj_lens=traj_lens,
+ turn_lens=turn_lens,
+ rewards=rewards,
+ # --------
+ # Everything above is per-group, i.e. it is a list of lists,
+ # with the inner list being the group data.
+ env_ids=env_ids,
+ num_turns=num_turns,
+ advantages=calculate_grpo_advantages(rewards, num_turns),
min_piold_to_inf_prob=None,
max_piold_to_inf_prob=None,
mean_piold_to_inf_prob=None,
@@ -759,20 +836,131 @@ def compute_group_stats(
min_inf_prob=None,
max_inf_prob=None,
mean_inf_prob=None,
- rewards=[r for group in rewards for r in group],
- advantages=calculate_grpo_advantages(rewards, num_turns),
- num_turns=[nt for group in num_turns for nt in group],
+ policy_epoch=all_policy_epoch,
+ kv_cache_epoch=all_kv_cache_epoch,
+ completed_epochs=all_completed_epochs,
+ num_evictions=all_num_evictions,
)
return stats
+
+def prep_wandb_metrics(
+ wandb_writer: wandb_run.Run,
+ traj_lens: List[List[int]],
+ turn_lens: List[List[int]],
+ rewards: List[List[float]],
+ num_turns: List[List[int]],
+ advantages: List[float],
+ policy_epoch: List[List[int]],
+ kv_cache_epoch: List[List[int]],
+ completed_epochs: List[List[int]],
+ num_evictions: List[List[int]],
+ current_iteration: int,
+ example_group: list[TokenRollout | Rollout] | None = None,
+ tokenizer: MegatronTokenizer | None = None,
+ ):
+
+ """Make a wandb-parseable dictionary of metrics for logging.
+
+ Args:
+ wandb_writer: Wandb run to log to.
+ traj_lens: Grouped list of trajectory lengths.
+ turn_lens: Grouped list of turn lengths.
+ rewards: Grouped list of rewards.
+ num_turns: Grouped list of number of turns in the trajectories.
+ advantages: Flattened list of advantages.
+ policy_epoch: Grouped list of per-rollout min policy epoch stamps.
+ kv_cache_epoch: Grouped list of per-rollout min KV cache epoch stamps.
+ completed_epochs: Grouped list of per-turn max policy epoch stamps.
+ num_evictions: Grouped list of per-rollout number of evictions.
+ current_iteration: Current training iteration.
+ example_group: A list of rollouts of one group to log examples of trajectories.
+ tokenizer: Tokenizer to untokenize trajectories for logging.
+ """
+
+ group_table = wandb_writer.Table(
+ columns=['group_means', 'group_stds'],
+ data=[[np.mean(g), np.std(g)] for g in rewards],
+ )
+
+ true_policy_staleness = [current_iteration - s for g in policy_epoch for s in g]
+ true_kv_staleness = [current_iteration - s for g in kv_cache_epoch for s in g]
+
+ metrics = {
+ 'group_means_hist': wandb_writer.plot.histogram(
+ group_table, 'group_means', 'Group Means'
+ ),
+ 'group_stds_hist': wandb_writer.plot.histogram(
+ group_table, 'group_stds', 'Group STDs'
+ ),
+ 'rewards_hist': wandb_writer.plot.histogram(
+ wandb_writer.Table(
+ columns=['reward'], data=[[r] for g in rewards for r in g]
+ ),
+ 'reward', 'All Rewards'
+ ),
+ 'advantages_hist': wandb_writer.plot.histogram(
+ wandb_writer.Table(
+ columns=['advantages'], data=[[x] for x in advantages]
+ ),
+ 'advantages', 'Advantages'
+ ),
+ 'rollout_table': wandb_writer.Table(
+ columns=['reward', 'traj_length', 'num_evictions'],
+ data=list(zip(
+ [r for g in rewards for r in g],
+ [l for g in traj_lens for l in g],
+ [e for g in num_evictions for e in g],
+ )),
+ ),
+ 'mean_turn_length': np.mean([np.mean(g) for g in turn_lens]),
+ 'mean_turn_length_std': np.mean([np.std(g) for g in turn_lens]),
+ 'max_turn_length': max([max(g) for g in turn_lens]),
+ 'min_turn_length': min([min(g) for g in turn_lens]),
+ 'mean_traj_length': np.mean([np.mean(g) for g in traj_lens]),
+ 'mean_traj_length_std': np.mean([np.std(g) for g in traj_lens]),
+ 'max_traj_length': max([max(g) for g in traj_lens]),
+ 'min_traj_length': min([min(g) for g in traj_lens]),
+ 'mean_num_turns': np.mean([np.mean(g) for g in num_turns]),
+ 'max_num_turns': max([max(g) for g in num_turns]),
+ 'min_num_turns': min([min(g) for g in num_turns]),
+ 'mean_reward': np.mean([np.mean(g) for g in rewards]),
+ 'mean_advantage': np.mean(advantages),
+ 'nonzero_groups_ratio': np.count_nonzero(advantages)
+ / len(advantages),
+ 'mean_policy_staleness': np.mean(true_policy_staleness),
+ 'max_policy_staleness': max(true_policy_staleness),
+ 'min_policy_staleness': min(true_policy_staleness),
+ 'mean_kv_cache_staleness': np.mean(true_kv_staleness),
+ 'max_kv_cache_staleness': max(true_kv_staleness),
+ 'min_kv_cache_staleness': min(true_kv_staleness),
+ 'total_eviction_count': sum([sum(g) for g in num_evictions]),
+ 'max_num_evictions': max([max(g) for g in num_evictions]),
+ 'mean_completion_gap': np.mean([current_iteration - s for g in completed_epochs for s in g]),
+ }
+ if example_group:
+ if tokenizer is None:
+ raise ValueError("If you provide an example group to log, you need to provide a tokenizer too.")
+ metrics['rollouts'] = wandb_writer.Table(
+ columns=['Trajectories', 'Tokens', 'Rewards'],
+ rows=[
+ [
+ tokenizer.detokenize(turn) if isinstance(r, TokenRollout) else turn,
+ r.trajectory,
+ r.reward,
+ ]
+ for r in example_group for turn in r.trajectory
+ ],
+ )
+ return metrics
+
+
def maybe_log_training_metrics(
group_stats: RolloutStats,
current_iteration: int,
- tokenizer: MegatronLegacyTokenizer,
- example_group: list[TokenRollout | Rollout],
- wandb_writer: wandb_run.Run | None = None,
- tb_writer: SummaryWriter | None = None,
+ tokenizer: MegatronTokenizer,
+ example_groups: dict[str, list[TokenRollout | Rollout]],
):
"""Log training metrics if writers are available.
@@ -780,78 +968,77 @@ def maybe_log_training_metrics(
group_stats: RolloutStats object to pass to writers.
current_iteration: Current training iteration.
tokenizer: Tokenizer to untokenize trajectories for logging.
- example_group: A list of rollouts of one group to log examples of trajectories.
- wandb_writer: W&B writer object.
- tb_writer: Tensorboard writer object.
+ example_groups: A dict with values as list of rollouts of one group to log examples of trajectories. Keys are env names.
"""
- if wandb_writer:
- group_table = wandb_writer.Table(
- columns=['group_means', 'group_stds'],
- data=list(zip(group_stats.reward_means, group_stats.reward_stds)),
- )
- rollout_table = wandb_writer.Table(
- columns=['reward'], data=[[r] for r in group_stats.rewards]
- )
- advantages = wandb_writer.Table(
- columns=['advantages'], data=[[x] for x in group_stats.advantages]
- )
- wandb_writer.log(
- {
- **{
- 'group_means_hist': wandb_writer.plot.histogram(
- group_table, 'group_means', 'Group Means'
- ),
- 'group_stds_hist': wandb_writer.plot.histogram(
- group_table, 'group_stds', 'Group STDs'
- ),
- 'rewards_hist': wandb_writer.plot.histogram(
- rollout_table, 'reward', 'All Rewards'
- ),
- 'mean_length': group_stats.mean_length,
- 'mean_length_std': group_stats.mean_length_std,
- 'max_length': group_stats.max_length,
- 'min_length': group_stats.min_length,
- 'mean_reward': group_stats.mean_reward,
- 'mean_advantage': np.mean(group_stats.advantages),
- 'advantages_hist': wandb_writer.plot.histogram(
- advantages, 'advantages', 'Advantages'
- ),
- 'nonzero_groups_ratio': np.count_nonzero(group_stats.advantages)
- / len(group_stats.advantages),
- 'min_piold_to_inf_prob': group_stats.min_piold_to_inf_prob,
- 'max_piold_to_inf_prob': group_stats.max_piold_to_inf_prob,
- 'mean_piold_to_inf_prob': group_stats.mean_piold_to_inf_prob,
- 'min_inf_train_prob_abs_diff': group_stats.min_inf_train_prob_abs_diff,
- 'max_inf_train_prob_abs_diff': group_stats.max_inf_train_prob_abs_diff,
- 'mean_inf_train_prob_abs_diff': group_stats.mean_inf_train_prob_abs_diff,
- 'min_inf_prob': group_stats.min_inf_prob,
- 'max_inf_prob': group_stats.max_inf_prob,
- 'mean_inf_prob': group_stats.mean_inf_prob,
- # For now only log the first group
- 'rollouts': wandb_writer.Table(
- columns=['Trajectories', 'Tokens', 'Rewards'],
- rows=[
- [
- [(tokenizer.detokenize(turn)
- if isinstance(r, TokenRollout)
- else turn) for turn in r.trajectory
- ],
- r.trajectory,
- r.reward,
- ]
- for r in example_group
- ],
- ),
- },
- },
- step=current_iteration,
- )
+
+ wandb_writer = get_wandb_writer()
+ tb_writer = get_tensorboard_writer()
if tb_writer:
- tb_writer.add_scalar('mean_reward', group_stats.mean_reward, current_iteration)
+ tb_writer.add_scalar('mean_reward', np.mean([np.mean(g) for g in group_stats.rewards]), current_iteration)
+ if not wandb_writer:
+ return
+
+ # We log these metrics for the aggregated data, no split per env.
+ metrics = {
+ 'min_piold_to_inf_prob': group_stats.min_piold_to_inf_prob,
+ 'max_piold_to_inf_prob': group_stats.max_piold_to_inf_prob,
+ 'mean_piold_to_inf_prob': group_stats.mean_piold_to_inf_prob,
+ 'min_inf_train_prob_abs_diff': group_stats.min_inf_train_prob_abs_diff,
+ 'max_inf_train_prob_abs_diff': group_stats.max_inf_train_prob_abs_diff,
+ 'mean_inf_train_prob_abs_diff': group_stats.mean_inf_train_prob_abs_diff,
+ 'min_inf_prob': group_stats.min_inf_prob,
+ 'max_inf_prob': group_stats.max_inf_prob,
+ 'mean_inf_prob': group_stats.mean_inf_prob,
+ }
+
+ traj_lens = group_stats.traj_lens
+ turn_lens = group_stats.turn_lens
+ rewards = group_stats.rewards
+ num_turns = group_stats.num_turns
+ advantages = group_stats.advantages
+ policy_epoch = group_stats.policy_epoch
+ kv_cache_epoch = group_stats.kv_cache_epoch
+ completed_epochs = group_stats.completed_epochs
+ num_evictions = group_stats.num_evictions
+
+ metrics = metrics | prep_wandb_metrics(wandb_writer=wandb_writer,
+ traj_lens=traj_lens, turn_lens=turn_lens, rewards=rewards, num_turns=num_turns, advantages=advantages,
+ policy_epoch=policy_epoch, kv_cache_epoch=kv_cache_epoch, completed_epochs=completed_epochs,
+ num_evictions=num_evictions, current_iteration=current_iteration)
+ env_stats = lambda cont, idx: [cont[i] for i in idx]
+ group_turn_counts = [sum(nt) for nt in num_turns]
+
+ for env_id in set(group_stats.env_ids):
+ env_idx = [i for i, eidx in enumerate(group_stats.env_ids) if eidx == env_id]
+
+ # Advantages are flattened, we need to be more careful with those.
+ env_advantages = []
+ for i in env_idx:
+ st = sum(group_turn_counts[:i])
+ end = st + group_turn_counts[i]
+ env_advantages.extend(advantages[st:end])
+
+ env_metrics = prep_wandb_metrics(wandb_writer=wandb_writer, traj_lens=env_stats(traj_lens, env_idx),
+ turn_lens=env_stats(turn_lens, env_idx),
+ rewards=env_stats(rewards, env_idx),
+ num_turns=env_stats(num_turns, env_idx),
+ advantages=env_advantages,
+ policy_epoch=env_stats(policy_epoch, env_idx),
+ kv_cache_epoch=env_stats(kv_cache_epoch, env_idx),
+ completed_epochs=env_stats(completed_epochs, env_idx),
+ num_evictions=env_stats(num_evictions, env_idx),
+ current_iteration=current_iteration,
+ example_group=example_groups[env_id],
+ tokenizer=tokenizer,
+ )
+ for k, v in env_metrics.items():
+ metrics[f"{env_id}_{k}"] = v
+
+ wandb_writer.log(metrics, step=current_iteration)
def prepare_trajectories(
- rollouts: Rollouts, tokenizer: MegatronLegacyTokenizer, seq_length: int, sequence_packing: bool, skip_bos_token: bool
+ rollouts: Rollouts, tokenizer: MegatronTokenizer, seq_length: int, sequence_packing: bool, skip_bos_token: bool
):
"""Pad trajectories and extract the generation masks.
Args:
@@ -868,21 +1055,21 @@ def prepare_trajectories(
# Track counts for each environment ID
env_id_counts = Counter()
- DEFAULT_PAD_TOKENS = ['<|finetune_right_pad_id|>']
-
+ DEFAULT_PAD_TOKENS = ['<|finetune_right_pad_id|>', '']
- if isinstance(tokenizer, _HuggingFaceTokenizer):
+ if tokenizer.library == "huggingface":
+ tokenizer : HuggingFaceTokenizer
if not tokenizer.pad:
for pad_token in DEFAULT_PAD_TOKENS:
- if pad_token in tokenizer.vocab:
+ if pad_token in tokenizer._tokenizer.tokenizer.get_vocab():
log_single_rank(
logger, logging.INFO, f"Updating tokenizer pad token to {pad_token}"
)
- tokenizer._tokenizer.pad_token_id = tokenizer.vocab[pad_token]
+ tokenizer._tokenizer.pad_token = pad_token
break
else:
raise ValueError("No pad token found in tokenizer vocabulary")
- elif isinstance(tokenizer, CustomTikTokenizer):
+ elif tokenizer.library == "tiktoken":
assert "" in tokenizer.vocab, "Pad token is NOT in the tokenizer"
tokenizer._pad_id = tokenizer.vocab[""]
@@ -999,7 +1186,7 @@ def logprobs_forward_step(data_iterator, model, is_correction, packing_context=N
b_trajs.cuda(),
b_posids.cuda(),
no_grad=True,
- sequence_packing=b_packed_seq_params is not None,
+ sequence_packing=packing_context is not None,
packed_seq_params=b_packed_seq_params,
),
None,
@@ -1008,7 +1195,7 @@ def logprobs_forward_step(data_iterator, model, is_correction, packing_context=N
return logprobs
-def _compute_logprobs_batch(
+def compute_logprobs_batch(
model,
data_loader,
forward_backward_func,
@@ -1020,6 +1207,7 @@ def _compute_logprobs_batch(
dtype,
pp_group,
is_correction,
+ collect_non_loss_data=False,
):
"""Compute logprobs for all batches in the data loader."""
logprobs_list = []
@@ -1035,6 +1223,7 @@ def _compute_logprobs_batch(
decoder_seq_length=decoder_seq_length,
forward_only=True,
adjust_tensor_shapes_fn=None,
+ collect_non_loss_data=collect_non_loss_data,
)
if is_pp_last_stage(pp_group):
logprobs_list.append(output_tensor[0].detach())
@@ -1060,10 +1249,10 @@ def prepare_data_for_update(
model: list[LanguageModule],
ref_state_dict: Dict[str, Any],
rollouts: GroupedRollouts,
- tokenizer: MegatronLegacyTokenizer,
+ tokenizer: MegatronTokenizer,
sequence_packing: bool,
is_correction: bool,
-) -> RerunDataIterator:
+) -> tuple[RerunDataIterator, RolloutStats, dict]:
"""Extract data for the update from raw rollouts.
Args:
@@ -1075,11 +1264,9 @@ def prepare_data_for_update(
is_correction: Prepare data for IS correction if True.
Returns:
- Cycled iterator over dataset batches. In GRPO we might want to go over the same data multiple times.
+ Tuple of (cycled iterator over dataset batches, group stats, example groups per env).
"""
args = get_args()
- wandb_writer = get_wandb_writer()
- tb_writer = get_tensorboard_writer()
nvtx_range = get_nvtx_range()
runtime_state = get_rl_runtime_state()
@@ -1087,13 +1274,13 @@ def prepare_data_for_update(
lang_module = (
model[0].module.module if hasattr(model[0].module, "module") else model[0].module
)
- toggle_cuda_graphs(lang_module, "none", reset_cuda_graphs=False)
+ toggle_cuda_graphs(lang_module, "none")
model = model[0]
dtype = torch.bfloat16 if args.bf16 else (torch.float16 if args.fp16 else torch.float32)
- with nvtx_range("prepare-data-for-update"):
- with nvtx_range("compute-group-stats"):
+ with nvtx_range("rl/prepare-data-for-update", time=True):
+ with nvtx_range("rl/compute-group-stats", time=True):
group_stats = compute_group_stats(rollouts, tokenizer, args.seq_length)
# TODO(vitalyk): why do we need global_advantages here? go inside packing
advantages = global_advantages = torch.tensor(group_stats.advantages, dtype=dtype).cuda()
@@ -1103,13 +1290,17 @@ def prepare_data_for_update(
# Note :- For EP, do not use the expert data parallel group here. Always
# use the regular data parallel group.
- # Use one group as an exampling for logging later.
- example_group = rollouts[0]
+ # Get example group per environment to log their rollouts.
+ example_groups = {}
+ for g in rollouts:
+ if g[0].env_id not in example_groups:
+ example_groups[g[0].env_id] = g
# Let's expand rollouts getting rid of the groups.
# We need this to correctly split the rollouts across dp groups.
# And we do not actually need them grouped in anything below anyways.
rollouts = [r for g in rollouts for r in g]
+ num_turns = [nt for g in group_stats.num_turns for nt in g]
total_turns_sampled = len(rollouts)
# We might sample more than we consume in one step.
@@ -1122,15 +1313,14 @@ def prepare_data_for_update(
mpu.get_data_parallel_rank() * data_split_size,
(mpu.get_data_parallel_rank() + 1) * data_split_size,
)
- # TODO(vitalyk): This has to be rewritten assuming we are multiturn now.
rollouts = rollouts[data_split_range[0] : data_split_range[1]]
- local_num_turns = sum(group_stats.num_turns[data_split_range[0] : data_split_range[1]])
- steps_before = sum(group_stats.num_turns[:data_split_range[0]])
+ local_num_turns = sum(num_turns[data_split_range[0] : data_split_range[1]])
+ steps_before = sum(num_turns[:data_split_range[0]])
advantages = advantages[steps_before:steps_before+local_num_turns]
# First we calculate them on a global level and then we split and recalculate on a local level.
# Sequence packing and reporting needs it global but non-packing wants it local.
- with nvtx_range("prepare_trajectories"):
+ with nvtx_range("rl/prepare-trajectories", time=True):
trajs, generation_masks, inference_logprobs = prepare_trajectories(
rollouts, tokenizer, args.seq_length, sequence_packing, args.rl_skip_bos_token
)
@@ -1138,7 +1328,7 @@ def prepare_data_for_update(
packing_context = None
# Build trajectories based on sequence packing or standard processing
if sequence_packing:
- with nvtx_range("sequence_packing", time=True):
+ with nvtx_range("rl/sequence-packing", time=True):
runtime_state.packing_context = packing_context = pack_all_trajectories(
trajs,
generation_masks,
@@ -1158,7 +1348,7 @@ def prepare_data_for_update(
logprobs_batch_size = 1
else:
# Always compute standard masks for the original data (we'll need them later)
- with nvtx_range("get_ltor_masks_and_position_ids"):
+ with nvtx_range("rl/get-ltor-masks", time=True):
_, original_loss_mask, original_position_ids = get_ltor_masks_and_position_ids(
trajs,
tokenizer.eod,
@@ -1177,17 +1367,16 @@ def prepare_data_for_update(
)
logprobs_batch_size = args.micro_batch_size
- with torch.no_grad(), nvtx_range("compute_logprobs", time=True):
+
+ with torch.no_grad(), nvtx_range("rl/compute-logprobs", time=True):
# Before we can update the model, we need to get the logprobs for the \pi_{old} model.
- # Wrap forward_backward_func for Full iteration CUDA graph
forward_backward_func = get_forward_backward_func()
if args.cuda_graph_impl == "local" and CudaGraphScope.full_iteration in args.cuda_graph_scope:
forward_backward_func = FullCudaGraphWrapper(
forward_backward_func, cuda_graph_warmup_steps=args.cuda_graph_warmup_steps
)
-
dtype = (
torch.bfloat16 if args.bf16 else (torch.float16 if args.fp16 else torch.float32)
)
@@ -1195,8 +1384,8 @@ def prepare_data_for_update(
pg_collection = get_attr_wrapped_model(model, "pg_collection")
pp_group = pg_collection.pp
- with torch.no_grad(), nvtx_range("compute_old_logprobs", time=True):
- old_logprobs = _compute_logprobs_batch(
+ with torch.no_grad(), nvtx_range("rl/compute-old-logprobs", time=True):
+ old_logprobs = compute_logprobs_batch(
model=model,
data_loader=data_loader,
forward_backward_func=forward_backward_func,
@@ -1210,13 +1399,13 @@ def prepare_data_for_update(
is_correction=args.rl_inference_logprobs_is_correction,
)
- with torch.no_grad(), nvtx_range("compute_ref_logprobs", time=True):
+ with torch.no_grad(), nvtx_range("rl/compute-ref-logprobs", time=True):
# We need to load the ref model state dict and compute the logprobs for the ref model
cur_st_dict = {
k: (v.cpu() if v is not None else v) for k, v in model.state_dict().items()
}
model.load_state_dict(ref_state_dict)
- ref_logprobs = _compute_logprobs_batch(
+ ref_logprobs = compute_logprobs_batch(
model=model,
data_loader=data_loader,
forward_backward_func=forward_backward_func,
@@ -1233,13 +1422,14 @@ def prepare_data_for_update(
# logprobs are [b, seq, h] now.
model.load_state_dict(cur_st_dict)
- torch.cuda.synchronize()
- gc.collect()
- torch.cuda.empty_cache()
+ with nvtx_range("rl/synchronize-cuda-and-collect-garbage", time=True):
+ torch.cuda.synchronize()
+ gc.collect()
+ torch.cuda.empty_cache()
if sequence_packing:
- with nvtx_range("pack_logprobs", time=True):
+ with nvtx_range("rl/pack-logprobs", time=True):
# Store logprobs on gpu in packing context
# Since PackingContext is a dataclass, we add these as new attributes
packing_context.old_logprobs = old_logprobs.cuda()
@@ -1267,7 +1457,7 @@ def prepare_data_for_update(
packing_context.packed_inference_logprobs = packed_inference_logprobs.cuda()
# Only mark as having inference logprobs for IS correction if enabled
packing_context.has_inference_logprobs = args.rl_inference_logprobs_is_correction
- with nvtx_range("create_dataloader"):
+ with nvtx_range("rl/create-dataloader", time=True):
# @vitalyk: This function also reconfigures the data loader to count the
# global_batch_size in the bins frame of reference.
# I think it will be a better design if we split the data loader creating and logic
@@ -1284,7 +1474,7 @@ def prepare_data_for_update(
)
loader = get_microbatch_dataloader(len(packing_context.packed_trajs), args.micro_batch_size)
else:
- with nvtx_range("align_inference_logprobs", time=True):
+ with nvtx_range("rl/align-inference-logprobs", time=True):
if inference_logprobs is not None:
inference_logprobs = align_unpacked_inference_logprobs(
inference_logprobs=inference_logprobs,
@@ -1297,7 +1487,7 @@ def prepare_data_for_update(
# Nullify logprobs if not used in IS correction,
if not args.rl_inference_logprobs_is_correction:
inference_logprobs = None
- with nvtx_range("create_dataloader"):
+ with nvtx_range("rl/create-dataloader", time=True):
# Because of multiturn, our batch sizes for non-sequence packed trajectories are not fixed anymore.
# As in sequence packing above, we need to reconfigure it too.
runtime_state.packing_context = None
@@ -1326,18 +1516,15 @@ def prepare_data_for_update(
data = TensorDataset(*dataset_tensors)
loader = DataLoader(data, batch_size=args.micro_batch_size)
-
- with nvtx_range("log-wandb-tb"):
+ with nvtx_range("rl/log-wandb-tb", time=True):
maybe_log_training_metrics(
group_stats=group_stats,
current_iteration=args.curr_iteration,
tokenizer=tokenizer,
- example_group=example_group,
- wandb_writer=wandb_writer,
- tb_writer=tb_writer,
+ example_groups=example_groups,
)
- return RerunDataIterator(itertools.cycle(loader))
+ return RerunDataIterator(itertools.cycle(loader)), group_stats, example_groups
def get_grpo_data_iterator(
@@ -1377,27 +1564,38 @@ def get_grpo_data_iterator(
RerunDataIterator for the current training step
"""
runtime_state = get_rl_runtime_state()
+ tokenizer = get_tokenizer()
# We collect new rollouts when we've gone over the collected data 'grpo_iterations' times.
- global_batches_per_collection = (grpo_prompts_per_step * grpo_group_size) // global_batch_size
+ global_batches_per_collection = (grpo_prompts_per_step * grpo_group_size) // global_batch_size
if (
buffered_rollouts is None or
- iteration == runtime_state.last_collection_iteration +
+ iteration == runtime_state.last_collection_iteration +
(grpo_iterations * global_batches_per_collection)
):
- buffered_rollouts = get_environment_rollouts(
+ rollouts = get_environment_rollouts(
model, inference_model, optimizer, grpo_prompts_per_step, grpo_group_size
)
- buffered_rollouts = prepare_data_for_update(model=model,
- ref_state_dict=ref_state_dict,
- rollouts=buffered_rollouts,
- tokenizer=get_tokenizer(),
+ buffered_rollouts, group_stats, example_groups = prepare_data_for_update(
+ model=model,
+ ref_state_dict=ref_state_dict,
+ rollouts=rollouts,
+ tokenizer=tokenizer,
sequence_packing=sequence_packing,
is_correction=is_correction,
- )
+ )
+ runtime_state.group_stats = group_stats
+ runtime_state.example_groups = example_groups
runtime_state.reset_iteration_counters(iteration)
+ maybe_log_training_metrics(
+ group_stats=runtime_state.group_stats,
+ current_iteration=iteration,
+ tokenizer=tokenizer,
+ example_groups=runtime_state.example_groups,
+ )
+
return buffered_rollouts
@@ -1407,14 +1605,17 @@ def evaluate_and_print_results_rl(
optimizer: MegatronOptimizer,
iteration: int,
write_to_tensorboard: bool = True,
+ training_model: Optional[list[LanguageModule]] = None,
):
"""Helper function to evaluate and dump results on screen.
Args:
data_iterator: Iterator over batches of evaluation dataset.
- model: Model to evaluate with.
+ model: Model to evaluate with (may be separate inference model).
iteration: Current training iteration.
write_to_tensorboard: Dumpt stuff to tensorboard or not.
+ training_model: Training model (if separate from inference model). Used to offload
+ grad buffers and restore to train mode. If None, uses model parameter.
"""
args = get_args()
@@ -1428,10 +1629,8 @@ def evaluate_and_print_results_rl(
model,
optimizer,
args.cuda_graph_impl,
- args.rl_reset_cuda_graphs,
args.rl_offload_optimizer_during_inference,
- args.rl_offload_kv_cache_during_training,
- args.rl_remove_kv_cache_during_training,
+ training_model,
) as inference_interface:
loop = get_asyncio_loop()
@@ -1452,9 +1651,10 @@ def evaluate_and_print_results_rl(
'top_k': args.rl_default_top_k,
},
)
- evaluation_responses = loop.run_until_complete(agent.run_evaluation(request))
- if not isinstance(evaluation_responses, list):
- evaluation_responses = [evaluation_responses]
+ with get_nvtx_range()("rl/run-evaluation", time=True):
+ evaluation_responses = loop.run_until_complete(agent.run_evaluation(request))
+ if not isinstance(evaluation_responses, list):
+ evaluation_responses = [evaluation_responses]
else:
evaluation_responses = None
@@ -1507,10 +1707,10 @@ def evaluate_and_print_results_rl(
with open(
lang_rl_log_dir
+ f'/eval_rank{rank}_iteration{args.curr_iteration}_'
- + f'{Path(args.langrl_env_config).stem}.pkl',
- 'wb',
+ + f'{Path(args.langrl_env_config).stem}.json',
+ 'w',
) as f:
- pickle.dump(dp_eval_results, f)
+ json.dump([[r.model_dump() for r in group] for group in dp_eval_results], f)
def calculate_grpo_loss(
@@ -1612,21 +1812,18 @@ def megatron_rl_inference_mode(
model: list[LanguageModule],
optimizer: MegatronOptimizer,
cuda_graph_impl: str,
- reset_cuda_graphs: bool,
offload_optimizer_during_inference: bool,
- offload_kv_cache_during_training: bool,
- remove_kv_cache_during_training: bool,
+ training_model: Optional[list[LanguageModule]] = None,
):
"""Manage the model inference context when collecting rollouts.
Args:
- model: model to prepare.
+ model: model to prepare for inference (may be separate inference model).
optimizer: optimizer used to train the model.
cuda_graph_impl: which cuda graph implementation to use.
- reset_cuda_graphs: rebuild cuda graphs for each inference stage or not.
offload_optimizer_during_inference: move optimizer to cpu during inference or not.
- offload_kv_cache_during_training: manually offload kv cache to host before training or not.
- remove_kv_cache_during_training: manually remove kv cache before training or not.
+ training_model: training model (if separate from inference model). Used to offload
+ grad buffers and restore to train mode. If None, uses model parameter.
Yields:
None: this context manager does not return a value.
@@ -1638,14 +1835,22 @@ def megatron_rl_inference_mode(
logger.debug(f"[{dist.get_rank()}] Entering inference mode")
+ # Change cudagraph scope for inference (empty list = full-layer capture)
+ model[0].config.cuda_graph_scope = []
+ model[0].config.cuda_graph_impl = "local"
+
# If we get a lower precision wrapper, we go one object deeper.
lang_module = model[0].module.module if hasattr(model[0].module, "module") else model[0].module
+ # Switch MoE layers to full CUDA graph capture for inference
+ if args.rl_training_cuda_graphs and args.num_experts is not None:
+ transition_moe_cudagraphs(lang_module, 'full')
+
lang_module.eval()
- # If this is a separate RL inference model allocated with UVM, ensure weights are resident on GPU
- # before any CUDA-graph capture/replay or inference.
- with nvtx_range("prefetch-inference-model-weights-to-gpu"):
- model_core = unwrap_model(model[0])
+ # If this is a separate RL inference model with offloading enabled, ensure weights are on GPU
+ # before any CUDA-graph capture/replay or inference. This is a no-op if already on GPU.
+ model_core = unwrap_model(model[0])
+ with nvtx_range("rl/prefetch-weights-to-gpu", time=True):
_maybe_prefetch_separate_inference_model_weights(model_core, to_cpu=False)
rotary_module = getattr(lang_module, "rotary_pos_emb", None)
@@ -1658,75 +1863,65 @@ def megatron_rl_inference_mode(
with torch.no_grad():
if offload_optimizer_during_inference:
- with nvtx_range("offload-optimizer-state-and-grad-buffers-before-inference"):
- model[0].offload_grad_buffers()
- optimizer.offload_to_cpu()
+ with nvtx_range("rl/offload-optimizer-before-inference", time=True):
+ if not args.rl_training_cuda_graphs:
+ with nvtx_range("rl/offload/grad-buffers", time=True):
+ model_for_grad_offload = training_model if training_model is not None else model
+ model_for_grad_offload[0].offload_grad_buffers()
+ else:
+ logger.warning(
+ "Gradient buffers will not be offloaded when training cudagraphs are enabled!")
+ with nvtx_range("rl/offload/optimizer-state", time=True):
+ optimizer.offload_to_cpu()
- # TODO: Remove this if statement once a change to `toggle_cuda_graphs` makes it safe to.
if cuda_graph_impl != "none" and not args.rl_training_cuda_graphs:
- toggle_cuda_graphs(lang_module, cuda_graph_impl, reset_cuda_graphs=reset_cuda_graphs)
+ toggle_cuda_graphs(lang_module, cuda_graph_impl)
inference_interface = get_inference_interface(args, loop, model)
-
- with nvtx_range("onload-kv-cache-before-inference"):
- if offload_kv_cache_during_training:
- # Restore the KV cache by re-binding physical pages to a consistent virtual address
- torch_memory_saver.resume("kv_cache")
-
- logger.debug(
- f"[{dist.get_rank()}] Restoring kv cache ({inference_interface._inference_engine.context.memory_buffer.numel() / 1024**3:.2f} GB) to GPU"
- )
- kv_cache = inference_interface._inference_engine.context.memory_buffer
- inference_interface._inference_engine.context.memory_buffer = kv_cache.cuda()
- elif remove_kv_cache_during_training:
- if inference_interface._inference_engine.context.memory_buffer is None:
- inference_interface._inference_engine.context.build_memory_buffer()
-
- # TODO: Improve this if statement once a change is made to CUDA graph handling.
- cuda_graph_exists = len(_CudagraphGlobalRecord.cudagraph_inference_record) != 0
- if cuda_graph_impl != "none" and not cuda_graph_exists:
- with nvtx_range("wait-for-decode-only"):
- while not inference_interface._inference_engine.context.is_decode_only():
- active_requests, finished_requests, step_time = loop.run_until_complete(
- inference_interface._inference_engine.async_step()
- )
- with nvtx_range("build-cuda-graphs"):
- inference_interface._inference_engine.create_cuda_graphs(reset_context=True)
-
+ inference_interface.set_generation_epoch(get_args().curr_iteration)
loop.run_until_complete(inference_interface.resume())
logger.debug(f"[{dist.get_rank()}] Entered inference mode")
yield inference_interface
- with nvtx_range("suspend-engine"):
+ with nvtx_range("rl/suspend-engine", time=True):
loop.run_until_complete(inference_interface.suspend())
- with nvtx_range("offload-kv-cache-after-inference"):
- if offload_kv_cache_during_training:
- kv_cache = inference_interface._inference_engine.context.memory_buffer
- logger.debug(
- f"[{dist.get_rank()}] Offloading kv cache ({kv_cache.numel() * kv_cache.element_size() / 1024**3:.2f} GB) to CPU"
- )
- torch_memory_saver.pause("kv_cache")
-
- elif remove_kv_cache_during_training:
- inference_interface._inference_engine.context.memory_buffer = None
-
- # TODO: Remove this if statement once a change to `toggle_cuda_graphs` makes it safe to.
if cuda_graph_impl != "none" and not args.rl_training_cuda_graphs:
- toggle_cuda_graphs(lang_module, 'none', reset_cuda_graphs=reset_cuda_graphs)
-
- # If this is a separate RL inference model, prefetch weights back to CPU so they don't consume
- # GPU memory during training.
+ toggle_cuda_graphs(lang_module, 'none')
+
+ # Reset drop_and_pad leaked from inference decode
+ set_decode_expert_padding(unwrap_model(model[0]), set_to=False)
+
+ # Restore partial capture cudagraph scope for training if this is MoE
+ if args.num_experts is not None:
+ model[0].config.cuda_graph_scope = [
+ CudaGraphScope.mamba,
+ CudaGraphScope.attn,
+ CudaGraphScope.moe_router,
+ CudaGraphScope.moe_preprocess,
+ ]
+
+ # Switch MoE layers to partial CUDA graph capture for training
+ if args.rl_training_cuda_graphs and args.num_experts is not None:
+ transition_moe_cudagraphs(lang_module, 'partial')
+
+ # If this is a separate RL inference model, prefetch weights back to CPU so they
+ # don't consume GPU memory during training.
with nvtx_range("prefetch-inference-model-weights-to-cpu"):
_maybe_prefetch_separate_inference_model_weights(model_core, to_cpu=True)
if offload_optimizer_during_inference:
- with nvtx_range("onload-optimizer-state-and-grad-buffers-after-inference"):
- model[0].restore_grad_buffers()
- optimizer.restore_from_cpu()
+ with nvtx_range("rl/restore-optimizer-after-inference", time=True):
+ with nvtx_range("rl/restore/grad-buffers", time=True):
+ model_for_grad_offload = training_model if training_model is not None else model
+ model_for_grad_offload[0].restore_grad_buffers()
+ with nvtx_range("rl/restore/optimizer-state", time=True):
+ optimizer.restore_from_cpu()
- lang_module.train()
+ # Set training model back to train mode (not inference model if they're separate)
+ training_lang_module = unwrap_model(training_model[0]) if training_model is not None else lang_module
+ training_lang_module.train()
if has_lru_cache:
rotary_module.forward.cache_clear()
@@ -1736,6 +1931,13 @@ def megatron_rl_inference_mode(
def rl_inference_interface_shutdown():
global _INFERENCE_INTERFACE
+ global _ROLLOUT_GENERATOR
+
+ if _ROLLOUT_GENERATOR is not None:
+ loop = get_asyncio_loop()
+ loop.run_until_complete(_ROLLOUT_GENERATOR.aclose())
+ _ROLLOUT_GENERATOR = None
+
if _INFERENCE_INTERFACE is not None:
loop = get_asyncio_loop()
loop.run_until_complete(_INFERENCE_INTERFACE.kill())
@@ -1743,6 +1945,12 @@ def rl_inference_interface_shutdown():
else:
logger.warning("No inference interface to shutdown. This should not happen.")
+ # TODO(rkirby): This is a hack to hard exit. There is a bug that is preventing us from using sys.exit(0).
+ # It seem the Flask server has non-daemon threads that are preventing the program from exiting.
+ # We need to find a way to gracefully complete all in progress requests and shutdown the Flask server.
+ import os
+ os._exit(0)
+
def get_iteration_sequence_count(args):
"""Get the total number of sequences processed in this iteration across all ranks."""
@@ -1781,4 +1989,3 @@ def _pad_nonnull_with_zeros(data: list[Optional[torch.Tensor]], max_len: int) ->
# Create zero tensor for None logprobs
padded_data.append(torch.zeros(max_len))
return torch.stack(padded_data)
-
diff --git a/megatron/rl/sequence_packing_utils.py b/megatron/rl/sequence_packing_utils.py
index 4d983764f77..6f3e4711ed7 100644
--- a/megatron/rl/sequence_packing_utils.py
+++ b/megatron/rl/sequence_packing_utils.py
@@ -24,7 +24,7 @@
@dataclass
class PackingInfo:
"""Information about how sequences are packed into bins.
-
+
Attributes:
bin_seq_indices: List where each element contains the global sequence indices in that bin
seq_starts: Dict mapping bin index to list of start positions for each sequence in that bin
@@ -42,7 +42,7 @@ class PackingInfo:
@dataclass
class PackingContext:
"""Context containing all information needed for sequence packing during training.
-
+
Attributes:
bin_size: Maximum size of each bin (in tokens)
packer: 'SequencePacker' instance used for packing
@@ -93,11 +93,11 @@ def load_packed_data_by_index(bin_idx: int, packing_context: PackingContext, log
old_logprobs = getattr(packing_context, 'old_logprobs', None)
if old_logprobs is not None:
old_logprobs = old_logprobs[idx]
-
+
ref_logprobs = getattr(packing_context, 'ref_logprobs', None)
if ref_logprobs is not None:
ref_logprobs = ref_logprobs[idx]
-
+
# Slice from position 1 because logprobs predict the next token, so they are
# shifted by 1 relative to the input tokens (logprobs has shape [batch, seq_len-1])
loss_mask = packing_context.packed_loss_mask[idx, 1:]
@@ -403,7 +403,7 @@ def get_default_packed_seq_params(seq_length: int, max_sequences_per_bin: int, d
means no actual packing boundaries
Args:
- seq_length: The sequence length
+ seq_length: The sequence length
max_sequences_per_bin: Max sequences to pack in a bin.
device: Device to create tensors on.
@@ -414,7 +414,10 @@ def get_default_packed_seq_params(seq_length: int, max_sequences_per_bin: int, d
args = get_args()
# Pad to the maximum number of sequences in the bin for the attention kernel.
- cu_seqlens = torch.full((max_sequences_per_bin,), seq_length, dtype=torch.int32, device=device)
+ # We add 2 to account for the initial 0 and the final bin_size.
+ cu_seqlens = torch.full(
+ (max_sequences_per_bin + 2,), seq_length, dtype=torch.int32, device=device,
+ )
cu_seqlens[0] = 0
return PackedSeqParams(
@@ -425,6 +428,7 @@ def get_default_packed_seq_params(seq_length: int, max_sequences_per_bin: int, d
cu_seqlens_kv_padded=None,
max_seqlen_q=seq_length,
max_seqlen_kv=seq_length,
+ total_tokens=seq_length,
)
def create_packed_seq_params(packing_context: PackingContext):
@@ -484,8 +488,9 @@ def create_packed_seq_params_for_bin(
# Pad cu_seqlens to bin_size by repeating the last value (creates zero-length ghost sequences)
# This ensures a fixed tensor size for CUDA graph compatibility
- if len(cu_seqlens) < max_sequences_per_bin:
- out = cu_seqlens.new_full((max_sequences_per_bin,), bin_size)
+ # We add 2 to account for the initial 0 and the final bin_size.
+ if len(cu_seqlens) < max_sequences_per_bin + 2:
+ out = cu_seqlens.new_full((max_sequences_per_bin + 2,), bin_size)
out[:len(cu_seqlens)] = cu_seqlens
cu_seqlens = out
@@ -499,6 +504,7 @@ def create_packed_seq_params_for_bin(
cu_seqlens_kv_padded=None,
max_seqlen_q=max_seqlen,
max_seqlen_kv=max_seqlen,
+ total_tokens=bin_size,
)
@@ -970,19 +976,19 @@ def pack_all_trajectories(trajs, generation_masks, inference_logprobs, global_ad
data_parallel_group = mpu.get_data_parallel_group()
nvtx_range = get_nvtx_range()
- with nvtx_range("regather_trajectories", time=True):
+ with nvtx_range("rl/regather-trajectories", time=True):
def _gather(data):
data = data.cuda()
data_list = [torch.empty_like(data) for _ in range(data_parallel_world_size)]
torch.distributed.all_gather(data_list, data, group=data_parallel_group)
return torch.cat(data_list, dim=0)
- trajs = _gather(trajs)
- generation_masks = _gather(generation_masks)
+ trajs = _gather(trajs)
+ generation_masks = _gather(generation_masks)
if inference_logprobs is not None:
inference_logprobs = _gather(inference_logprobs)
- with nvtx_range("pack_sequences", time=True):
+ with nvtx_range("rl/pack-sequences", time=True):
# Create packer with max sequences per bin limit to prevent extreme imbalance
packer = SequencePacker(
bin_size=bin_size,
@@ -1062,9 +1068,9 @@ def update_microbatch_calculator(
samples_ratio_per_step: float,
num_bins_this_rank: int,
bin_seq_indices: List[List[int]],
- global_batch_size: int,
- rampup_batch_size: int,
- micro_batch_size: int,
+ global_batch_size: int,
+ rampup_batch_size: int,
+ micro_batch_size: int,
decrease_batch_size_if_needed: bool,
):
"""Return a data loader with seqpacked indices with microbatches in bins frame of reference.
diff --git a/megatron/rl/server/agent/fastapi_env_server.py b/megatron/rl/server/agent/fastapi_env_server.py
index c378c5625ee..361642a422e 100644
--- a/megatron/rl/server/agent/fastapi_env_server.py
+++ b/megatron/rl/server/agent/fastapi_env_server.py
@@ -127,7 +127,7 @@ async def get_grouped_rollouts(
assert isinstance(
request.inference_interface, InferenceServer
), "Rollout requests to remote server must contain an InferenceServer object"
- assert request.num_groups != -1, "FastAPIEnvServer does not support group rollout streaming"
+ assert not request.streaming, "FastAPIEnvServer does not support group rollout streaming"
payload = request.model_dump()
payload["inference_interface"] = request.inference_interface.model_dump()
async with httpx.AsyncClient() as client:
diff --git a/megatron/rl/server/api.py b/megatron/rl/server/api.py
index 528e6e880dc..6635b3d48dc 100644
--- a/megatron/rl/server/api.py
+++ b/megatron/rl/server/api.py
@@ -18,7 +18,7 @@ async def launch(cls) -> Self:
async def suspend(self):
pass
- def resume(self):
+ async def resume(self):
pass
async def kill(self):
diff --git a/megatron/rl/server/inference/inference_interface_server.py b/megatron/rl/server/inference/inference_interface_server.py
index ba595c3ca0e..ceac4a6cab7 100644
--- a/megatron/rl/server/inference/inference_interface_server.py
+++ b/megatron/rl/server/inference/inference_interface_server.py
@@ -11,10 +11,10 @@
from typing_extensions import Self
from uvicorn import Config, Server
-from ...inference.api import ChatInferenceRequest, ChatInferenceResponse
from ...inference.inference_interface import (
- ChatInferenceInterface,
InferenceInterface,
+ InferenceRequest,
+ InferenceResponse,
ReturnsRaw,
ReturnsTokens,
)
@@ -22,20 +22,17 @@
@InferenceServer.register_subclass
-class InferenceInterfaceClient(ChatInferenceInterface, InferenceServer):
+class InferenceInterfaceClient(InferenceServer):
type_name: str = Field(default='InferenceInterfaceClient', frozen=True)
env_server_host_port: str
conversation_template: None = None
- async def base_generate(self, request: ChatInferenceRequest) -> list[ChatInferenceResponse]:
+ async def base_generate(self, request: InferenceRequest) -> InferenceResponse:
async with httpx.AsyncClient(timeout=None) as client:
response = await client.post(
f"http://{self.env_server_host_port}/base_generate/", json=request.model_dump()
)
- return [
- ChatInferenceResponse.model_validate(inference_response)
- for inference_response in response.json()
- ]
+ return InferenceResponse.model_validate(response.json())
@InferenceServer.register_subclass
@@ -69,7 +66,7 @@ async def launch(cls, interface_cls: type[InferenceInterface], **kwargs) -> Self
server_ref = weakref.ref(launched_server)
@app.post("/base_generate/")
- async def base_generate(request: ChatInferenceRequest):
+ async def base_generate(request: InferenceRequest) -> InferenceResponse:
server = server_ref()
if server is None:
raise RuntimeError("Server has been garbage collected")
diff --git a/megatron/training/arguments.py b/megatron/training/arguments.py
index b6f4fd2bead..89e85865f07 100644
--- a/megatron/training/arguments.py
+++ b/megatron/training/arguments.py
@@ -38,6 +38,7 @@
)
from megatron.core.utils import (
get_torch_version,
+ is_flashinfer_min_version,
is_te_min_version,
is_torch_min_version,
)
@@ -350,62 +351,129 @@ def validate_args(args, defaults={}):
total_model_size = args.tensor_model_parallel_size * args.pipeline_model_parallel_size * args.context_parallel_size
args.data_parallel_size = args.world_size // total_model_size
- # Assert that `torch_memory_saver` is installed if offloading KV cache during RL.
- if args.rl_offload_kv_cache_during_training:
- try:
- from torch_memory_saver import torch_memory_saver
- except ImportError:
- raise AssertionError("To use offload-kv-cache-during-training, `torch_memory_saver` must be installed. See https://github.com/fzyzcjy/torch_memory_saver.")
- assert not args.inference_dynamic_batching_unified_memory_level, "The KV cache should not be instantiated in unified memory when it is offloaded during training."
-
- # Batch size checks if running RL.
if args.perform_rl_step:
- assert not (args.rl_remove_kv_cache_during_training and args.rl_offload_kv_cache_during_training), \
- "Cannot use both remove-kv-cache-during-training and offload-kv-cache-during-training"
+ # ----------------------------------------------------------------
+ # CUDA graphs
+ #
+ # --cuda-graph-impl controls whether CUDA graphs are built.
+ # The sweep of various inference CUDA graphs is built inside inference, not the RL loop.
+ # Both training and inference CUDA graphs are gated by this flag.
+ #
+ # --rl-training-cuda-graphs controls whether CUDA graphs are used during training.
+ # Toggling CUDA graphs on and off is done inside the RL loop.
+ #
+ # --rl-persist-cuda-graphs controls whether CUDA graphs are built once, or repeatedly.
+ # When this flag is True, inference requires static memory pointers for the KV cache.
+ # When this flag is False, inference is in charge of deleting/rebuilding CUDA graphs.
+ #
+ # KV cache management (--rl-kv-cache-management-mode)
+ #
+ # Inference initializes the KV cache, inside either a normal memory pool, UVM, or TMS.
+ #
+ # On suspend (inference -> training):
+ # "persist" — no-op; KV cache stays on GPU.
+ # "offload" — KV cache is offloaded to CPU.
+ # "recompute" — KV cache is deleted entirely.
+ #
+ # On resume (training → inference):
+ # "persist" — no-op; KV cache is already on GPU.
+ # "offload" — KV cache is restored from CPU.
+ # "recompute" — KV cache is recomputed from scratch.
+ # ----------------------------------------------------------------
+
+ # Persisting CGs only makes sense if we build any CGs.
+ assert not args.rl_persist_cuda_graphs or args.cuda_graph_impl != "none", (
+ "--rl-persist-cuda-graphs is set but no CUDA graphs are being built."
+ )
+ # Training CGs only makes sense if we build any CGs.
+ assert not args.rl_training_cuda_graphs or args.cuda_graph_impl != "none", (
+ "--rl-training-cuda-graphs is set but no CUDA graphs are being built."
+ )
+ # If CUDA graphs persist and KV cache memory address is not static, we need
+ # either UVM or torch_memory_saver to maintain memory address stability for CGs.
+ if args.rl_persist_cuda_graphs and args.rl_kv_cache_management_mode != "persist":
+ try:
+ from torch_memory_saver import torch_memory_saver
+ except ImportError:
+ assert args.inference_dynamic_batching_unified_memory_level > 0, (
+ "Persisting CUDA graphs requires static KV cache memory. Use "
+ "--rl-kv-cache-management-mode=persist, UVM, or install torch_memory_saver."
+ )
- assert not (args.rl_partial_rollouts and args.rl_remove_kv_cache_during_training), \
- "Cannot use both partial-rollouts and remove-kv-cache-during-training"
+ # Offload mode requires CG persistence: CG recapture runs dummy forward
+ # passes that corrupt the preserved KV data.
+ assert (
+ (not args.rl_kv_cache_management_mode == "offload") or (args.rl_persist_cuda_graphs)
+ ), "--rl-kv-cache-management-mode=offload requires --rl-persist-cuda-graphs"
+ # There's no need to manually offload the KV cache with UVM.
assert not (
- args.rl_offload_inference_model_weights_when_idle
- and args.rl_inference_model_unified_memory_level != 1
- ), (
- "--rl-offload-inference-model-weights-when-idle requires "
- "--rl-inference-model-unified-memory-level=1."
- )
+ args.inference_dynamic_batching_unified_memory_level > 0
+ and args.rl_kv_cache_management_mode == "offload"
+ ), "--rl-kv-cache-management-mode=offload is incompatible with UVM"
+ # We currently cannot recapture CGs in offload mode.
+ assert not(
+ not args.rl_persist_cuda_graphs and args.rl_kv_cache_management_mode == "offload"
+ ), "Cannot recapture CUDA graphs while offloading KV cache."
+
+ # Validate inference model offloading - requires either UVM or torch_memory_saver
+ if args.rl_offload_inference_model_weights_when_idle:
+ if args.rl_inference_model_unified_memory_level != 1:
+ # Not using UVM, so we need torch_memory_saver
+ try:
+ from torch_memory_saver import torch_memory_saver
+ except ImportError:
+ raise AssertionError(
+ "To use --rl-offload-inference-model-weights-when-idle without UVM "
+ "(--rl-inference-model-unified-memory-level=1), `torch_memory_saver` must be "
+ "installed. See https://github.com/fzyzcjy/torch_memory_saver."
+ )
+
+ # Resolve deprecated --rl-parallel-generation-tasks -> --rl-num-parallel-generations.
+ assert args.rl_num_parallel_generations is None \
+ or args.rl_parallel_generation_tasks is None, \
+ "Cannot specify both --rl-num-parallel-generations and " \
+ "--rl-parallel-generation-tasks. Use --rl-num-parallel-generations " \
+ "(--rl-parallel-generation-tasks is deprecated)."
+ if args.rl_parallel_generation_tasks is not None:
+ print_rank_0(
+ "WARNING: --rl-parallel-generation-tasks is deprecated, "
+ "use --rl-num-parallel-generations instead.")
+ args.rl_num_parallel_generations = (
+ args.rl_parallel_generation_tasks * args.grpo_group_size)
+
+ # Resolve --rl-num-parallel-generations / --rl-num-parallel-generation-batches.
+ assert args.rl_num_parallel_generations is None \
+ or args.rl_num_parallel_generation_batches is None, \
+ "--rl-num-parallel-generations and --rl-num-parallel-generation-batches " \
+ "are mutually exclusive."
+ if args.rl_num_parallel_generations is not None:
+ assert args.rl_partial_rollouts, \
+ "--rl-num-parallel-generations requires --rl-partial-rollouts."
+ assert args.rl_num_parallel_generations % args.grpo_group_size == 0, \
+ f"--rl-num-parallel-generations ({args.rl_num_parallel_generations}) " \
+ f"must be divisible by --grpo-group-size ({args.grpo_group_size})."
+ args.rl_parallel_generation_tasks = (
+ args.rl_num_parallel_generations // args.grpo_group_size)
+ if args.rl_generation_batch_size is None:
+ args.rl_generation_batch_size = 1
+ elif args.rl_num_parallel_generation_batches is not None:
+ assert args.rl_partial_rollouts, \
+ "--rl-num-parallel-generation-batches requires --rl-partial-rollouts."
+ if args.rl_generation_batch_size is None:
+ args.rl_generation_batch_size = args.grpo_prompts_per_step
+ args.rl_parallel_generation_tasks = (
+ args.rl_num_parallel_generation_batches * args.rl_generation_batch_size)
+ else:
+ if args.rl_generation_batch_size is None:
+ args.rl_generation_batch_size = 1
+ args.rl_parallel_generation_tasks = 512
- # When using different EP sizes for inference and training (EP refit), the legacy
- # GroupedMLP is not supported. Only SequentialMLP or TEGroupedMLP can be used.
- if (
- args.rl_inference_expert_model_parallel_size is not None
- and args.rl_inference_expert_model_parallel_size != args.expert_model_parallel_size
- ):
- assert not args.moe_use_legacy_grouped_gemm, (
- "Legacy GroupedMLP (--moe-use-legacy-grouped-gemm) is not supported when using "
- "different expert parallelism sizes for inference and training. "
- "Use SequentialMLP (default when --moe-grouped-gemm is not set) or "
- "TEGroupedMLP (--moe-grouped-gemm without --moe-use-legacy-grouped-gemm)."
- )
+ # Derive enforce_order after all resolution is complete.
+ args.rl_enforce_generation_order = (args.rl_generation_batch_size > 1)
args.grpo_samples_per_iteration = args.grpo_prompts_per_step * args.grpo_group_size
- num_generated_samples_per_inference_iteration = (
- args.grpo_samples_per_iteration * args.grpo_iterations)
- # Ensure that the number of prompts we collect is a multiple of the global batch size.
- # TODO: Make this account for batch size rampup?
- assert num_generated_samples_per_inference_iteration % args.global_batch_size == 0, \
- f"grpo_group_size * grpo_prompts_per_step * grpo_iterations should be divisible by global_batch_size"
-
- # For now only exit/checkpoint on iterations where we generate data. We don't currently
- # have a way to checkpoint the generated data.
- num_training_iterations_per_inference_iteration = (
- num_generated_samples_per_inference_iteration // args.global_batch_size)
- if args.exit_interval is not None:
- assert args.exit_interval % num_training_iterations_per_inference_iteration == 0, \
- f"exit_interval should be divisible by number of global batches per inference iteration."
- if args.save_interval is not None:
- assert args.save_interval % num_training_iterations_per_inference_iteration == 0, \
- f"save_interval should be divisible by number of global batches per inference iteration."
if args.rl_use_sequence_packing:
assert args.micro_batch_size == 1, \
"micro_batch_size must be 1 when using sequence packing. To increase compute per micro batch increase the sequence length."
@@ -522,35 +590,150 @@ def validate_args(args, defaults={}):
print_rank_0('setting global batch size to {}'.format(args.global_batch_size))
assert args.global_batch_size > 0
- # === MTP validation ===
- # Deprecation warnings for legacy MTP arguments
+ if args.perform_rl_step:
+ num_generated_samples_per_inference_iteration = (
+ args.grpo_samples_per_iteration * args.grpo_iterations)
+
+ # Ensure that the number of prompts we collect is a multiple of the global batch size.
+ # TODO: Make this account for batch size rampup?
+ assert num_generated_samples_per_inference_iteration % args.global_batch_size == 0, \
+ f"grpo_group_size * grpo_prompts_per_step * grpo_iterations should be divisible by global_batch_size"
+
+ # For now only exit/checkpoint on iterations where we generate data. We don't currently
+ # have a way to checkpoint the generated data.
+ num_training_iterations_per_inference_iteration = (
+ num_generated_samples_per_inference_iteration // args.global_batch_size)
+ if args.exit_interval is not None:
+ assert args.exit_interval % num_training_iterations_per_inference_iteration == 0, \
+ f"exit_interval should be divisible by number of global batches per inference iteration."
+ if args.save_interval is not None:
+ assert args.save_interval % num_training_iterations_per_inference_iteration == 0, \
+ f"save_interval should be divisible by number of global batches per inference iteration."
+
+ # === Hybrid layer pattern: deprecation handling and validation ===
+
+ # Backward compat: --hybrid-override-pattern is deprecated in favor of --hybrid-layer-pattern
+ used_hybrid_override_pattern = False
+ if args.hybrid_override_pattern is not None:
+ assert args.hybrid_layer_pattern is None, (
+ '--hybrid-override-pattern and --hybrid-layer-pattern cannot both be specified. '
+ '--hybrid-override-pattern is deprecated; use --hybrid-layer-pattern instead.'
+ )
+ warn_rank_0(
+ "--hybrid-override-pattern is deprecated. Use --hybrid-layer-pattern instead.",
+ args.rank,
+ )
+ args.hybrid_layer_pattern = args.hybrid_override_pattern
+ used_hybrid_override_pattern = True
+
if args.mtp_hybrid_override_pattern is not None:
warn_rank_0(
"--mtp-hybrid-override-pattern is deprecated. "
- "For new hybrid models with MTP models, use unified --hybrid-override-pattern instead. "
+ "For new hybrid models with MTP, use unified --hybrid-layer-pattern instead. "
"Example: 'M*M*/MM/MM' means main='M*M*', MTP pattern='MM' with 2 depths. "
"This argument is kept only for loading old checkpoints.",
args.rank,
)
- # Backward compatibility: convert legacy mtp_hybrid_override_pattern to unified format
- from megatron.core.ssm.mamba_hybrid_layer_allocation import Symbols, parse_hybrid_pattern
+ from megatron.core.ssm.mamba_hybrid_layer_allocation import (
+ Symbols, parse_hybrid_pattern, get_hybrid_total_layer_count,
+ get_hybrid_total_pipeline_segment_count,
+ )
sep = Symbols.MTP_SEPARATOR
+
+ # Backward compat: convert legacy mtp_hybrid_override_pattern to unified format
if (
- getattr(args, 'mtp_hybrid_override_pattern', None) is not None
+ args.mtp_hybrid_override_pattern is not None
and args.mtp_num_layers is not None
and args.mtp_num_layers > 0
- and (args.hybrid_override_pattern is None or sep not in args.hybrid_override_pattern)
+ and (args.hybrid_layer_pattern is None or sep not in args.hybrid_layer_pattern)
):
- main_pattern = args.hybrid_override_pattern or ''
+ main_pattern = args.hybrid_layer_pattern or ''
mtp_pattern = args.mtp_hybrid_override_pattern
- args.hybrid_override_pattern = main_pattern + sep + sep.join([mtp_pattern] * args.mtp_num_layers)
+ args.hybrid_layer_pattern = main_pattern + sep + sep.join([mtp_pattern] * args.mtp_num_layers)
args.mtp_hybrid_override_pattern = None
- print_rank_0(f"Converted legacy MTP pattern to unified: {args.hybrid_override_pattern}")
+ print_rank_0(f"Converted legacy MTP pattern to unified: {args.hybrid_layer_pattern}")
+
+ if args.hybrid_layer_pattern is not None:
+ # Derive num_layers from pattern; hybrid_layer_pattern always overrides --num-layers when
+ # both are present (e.g. when loading from checkpoint with --use-checkpoint-args).
+ num_layers_in_pattern = get_hybrid_total_layer_count(args.hybrid_layer_pattern)
+ if args.num_layers is not None and args.num_layers != num_layers_in_pattern:
+ warn_rank_0(
+ f'--hybrid-layer-pattern is set; ignoring --num-layers ({args.num_layers}) and '
+ f'using the layer count derived from the pattern ({num_layers_in_pattern}).',
+ args.rank,
+ )
+ args.num_layers = num_layers_in_pattern
+
+ # first/last pipeline num layers are incompatible with pipe-separated patterns
+ # (the pipe separators already define the pipeline layout explicitly), but are
+ # allowed for pipe-free patterns where they control uneven PP splitting.
+ has_pipes = Symbols.PIPE in args.hybrid_layer_pattern.split(sep)[0]
+ if has_pipes:
+ assert args.decoder_first_pipeline_num_layers is None, (
+ 'If --hybrid-layer-pattern contains pipe separators, '
+ '--decoder-first-pipeline-num-layers should not be specified '
+ 'as the pipeline layout is explicitly defined.'
+ )
+ assert args.decoder_last_pipeline_num_layers is None, (
+ 'If --hybrid-layer-pattern contains pipe separators, '
+ '--decoder-last-pipeline-num-layers should not be specified '
+ 'as the pipeline layout is explicitly defined.'
+ )
+ assert args.num_layers_per_virtual_pipeline_stage is None, (
+ '--num-layers-per-virtual-pipeline-stage should not be used with '
+ '--hybrid-layer-pattern. To specify virtual pipelining, describe a number of '
+ 'pipeline segments in --hybrid-layer-pattern that is a multiple of '
+ '--pipeline-model-parallel-size greater than 1'
+ )
+ assert args.num_virtual_stages_per_pipeline_rank is None, (
+ '--num-virtual-stages-per-pipeline-rank should not be used with '
+ '--hybrid-layer-pattern. Virtual pipeline stages are derived from the '
+ 'number of | segments in the pattern.'
+ )
+ assert args.pipeline_model_parallel_layout is None, (
+ '--pipeline-model-parallel-layout should not be used with --hybrid-layer-pattern. '
+ 'Pipeline stage layout is defined by | separators in the pattern.'
+ )
+ assert not args.account_for_embedding_in_pipeline_split, (
+ '--account-for-embedding-in-pipeline-split should not be used with '
+ '--hybrid-layer-pattern. Pipeline stage layout is defined by | separators '
+ 'in the pattern.'
+ )
+ assert not args.account_for_loss_in_pipeline_split, (
+ '--account-for-loss-in-pipeline-split should not be used with '
+ '--hybrid-layer-pattern. Pipeline stage layout is defined by | separators '
+ 'in the pattern.'
+ )
+
+ # Derive VPP from pipe segments in the pattern
+ hybrid_pipeline_segments = get_hybrid_total_pipeline_segment_count(
+ args.hybrid_layer_pattern
+ )
+ if hybrid_pipeline_segments == 1 and args.transformer_pipeline_model_parallel_size > 1:
+ # No pipes in pattern -- PP will be handled by select_pipeline_segment
+ # at model init time (for backwards compatibility).
+ args.virtual_pipeline_model_parallel_size = None
+ else:
+ assert hybrid_pipeline_segments % args.transformer_pipeline_model_parallel_size == 0, (
+ 'The number of hybrid pipeline segments described by --hybrid-layer-pattern must '
+ 'be evenly divisible by --pipeline-model-parallel-size. '
+ f'Got {hybrid_pipeline_segments} segments and '
+ f'{args.transformer_pipeline_model_parallel_size} pipeline parallel size.'
+ )
+ if hybrid_pipeline_segments > args.transformer_pipeline_model_parallel_size:
+ # Must be set here in order to assign virtual parallel ranks in
+ # training.py/get_model
+ args.virtual_pipeline_model_parallel_size = (
+ hybrid_pipeline_segments // args.transformer_pipeline_model_parallel_size
+ )
+ else:
+ args.virtual_pipeline_model_parallel_size = None
# Infer mtp_num_layers from unified pattern
- if args.hybrid_override_pattern and sep in args.hybrid_override_pattern:
- parsed = parse_hybrid_pattern(args.hybrid_override_pattern)
+ if args.hybrid_layer_pattern and sep in args.hybrid_layer_pattern:
+ parsed = parse_hybrid_pattern(args.hybrid_layer_pattern)
if parsed.mtp_pattern and parsed.mtp_num_depths > 0:
inferred_mtp_num_layers = parsed.mtp_num_depths
if args.mtp_num_layers is None:
@@ -558,7 +741,8 @@ def validate_args(args, defaults={}):
elif args.mtp_num_layers != inferred_mtp_num_layers:
warn_rank_0(
f"--mtp-num-layers ({args.mtp_num_layers}) conflicts with "
- f"MTP depth count ({inferred_mtp_num_layers}) in pattern '{args.hybrid_override_pattern}'. "
+ f"MTP depth count ({inferred_mtp_num_layers}) in pattern "
+ f"'{args.hybrid_layer_pattern}'. "
f"Using the inferred value ({inferred_mtp_num_layers}).",
args.rank
)
@@ -573,14 +757,14 @@ def validate_args(args, defaults={}):
)
# Validate MTP args for hybrid vs non-hybrid models
- if args.is_hybrid_model:
+ if args.hybrid_layer_pattern is not None:
# Mamba/hybrid model MTP validation
- if args.mtp_num_layers and not (args.hybrid_override_pattern and sep in args.hybrid_override_pattern):
+ if args.mtp_num_layers and not (args.hybrid_layer_pattern and sep in args.hybrid_layer_pattern):
# Hybrid model wants MTP but no unified pattern - check for legacy args
if args.mtp_hybrid_override_pattern is None:
warn_rank_0(
"Hybrid model with --mtp-num-layers but no MTP pattern. "
- "Use unified --hybrid-override-pattern with '/' separator (e.g., 'M*M*/MM/MM') "
+ "Use unified --hybrid-layer-pattern with '/' separator (e.g., 'M*M*/MM/MM') "
"or legacy --mtp-hybrid-override-pattern for old checkpoints.",
args.rank
)
@@ -593,8 +777,8 @@ def validate_args(args, defaults={}):
"This argument will be ignored.",
args.rank
)
- # === End of MTP validation ===
-
+ # === End of hybrid layer pattern: deprecation handling and validation ===
+
# Uneven virtual pipeline parallelism
assert (
int(args.num_layers_per_virtual_pipeline_stage is not None)
@@ -649,12 +833,14 @@ def validate_args(args, defaults={}):
if args.virtual_pipeline_model_parallel_size == 1:
args.virtual_pipeline_model_parallel_size = None
else:
- args.virtual_pipeline_model_parallel_size = None
+ # Only set VPP to None if it wasn't already derived from --hybrid-layer-pattern
+ if args.hybrid_layer_pattern is None:
+ args.virtual_pipeline_model_parallel_size = None
if args.decoder_first_pipeline_num_layers is None and args.decoder_last_pipeline_num_layers is None:
# Divisibility check not applicable for T5 models which specify encoder_num_layers
- # and decoder_num_layers.
- if args.num_layers is not None:
+ # and decoder_num_layers, or for hybrid models using --hybrid-layer-pattern.
+ if args.num_layers is not None and args.hybrid_layer_pattern is None:
num_layers = args.num_layers
if args.account_for_embedding_in_pipeline_split:
@@ -691,8 +877,9 @@ def validate_args(args, defaults={}):
)
if args.overlap_param_gather:
- assert args.use_distributed_optimizer or args.use_megatron_fsdp, \
- '--overlap-param-gather only supported with distributed optimizer or megatron fsdp'
+ assert args.use_distributed_optimizer or args.use_megatron_fsdp \
+ or args.optimizer == 'dist_muon', \
+ '--overlap-param-gather only supported with distributed optimizer, megatron fsdp, or dist_muon'
assert args.overlap_grad_reduce, \
'Must use --overlap-param-gather with --overlap-grad-reduce'
assert not args.use_legacy_models, \
@@ -738,8 +925,9 @@ def validate_args(args, defaults={}):
assert not args.use_dist_ckpt, \
'--overlap-param-gather-with-optimizer-step not supported with distributed checkpointing yet'
+ # Map string data-type to torch.dtype.
dtype_map = {
- 'fp32': torch.float32, 'bf16': torch.bfloat16, 'fp16': torch.float16, 'fp8': torch.uint8,
+ 'fp32': torch.float32, 'bf16': torch.bfloat16, 'fp16': torch.float16, 'fp8': torch.uint8, 'auto': None, None: None,
}
map_dtype = lambda d: d if isinstance(d, torch.dtype) else dtype_map[d]
@@ -747,6 +935,14 @@ def validate_args(args, defaults={}):
args.main_params_dtype = map_dtype(args.main_params_dtype)
args.exp_avg_dtype = map_dtype(args.exp_avg_dtype)
args.exp_avg_sq_dtype = map_dtype(args.exp_avg_sq_dtype)
+ args.mamba_inference_conv_states_dtype = map_dtype(args.mamba_inference_conv_states_dtype)
+ args.mamba_inference_ssm_states_dtype = map_dtype(args.mamba_inference_ssm_states_dtype)
+
+ args.megatron_fsdp_main_params_dtype = map_dtype(args.megatron_fsdp_main_params_dtype)
+ args.megatron_fsdp_main_grads_dtype = map_dtype(args.megatron_fsdp_main_grads_dtype)
+ args.megatron_fsdp_grad_comm_dtype = map_dtype(args.megatron_fsdp_grad_comm_dtype)
+ if args.grad_reduce_in_bf16:
+ args.megatron_fsdp_grad_comm_dtype = torch.bfloat16
if args.fp8_param_gather:
assert args.use_distributed_optimizer or args.use_torch_fsdp2 or args.use_megatron_fsdp or not torch.is_grad_enabled(), \
@@ -767,6 +963,13 @@ def validate_args(args, defaults={}):
if args.fp4 and getattr(args, "fp4_recipe", "nvfp4") == "mxfp4" and not is_te_min_version("2.12.0.dev0"):
raise ValueError("--fp4-recipe=mxfp4 requires Transformer Engine >= 2.12.0.dev0 for MXFP4BlockScaling support.")
+ if (
+ args.fp8_recipe == 'mxfp8'
+ and args.transformer_impl == 'inference_optimized'
+ and not is_flashinfer_min_version("0.6.4")
+ ):
+ raise ValueError("MXFP8 with inference optimized layers requires FlashInfer >= 0.6.4")
+
if args.use_megatron_fsdp:
# NOTE: The flag `use_custom_fsdp` is deprecated and will be removed in future versions.
# Please use `use_megatron_fsdp` instead, as all functionality will be migrated there.
@@ -822,12 +1025,21 @@ def validate_args(args, defaults={}):
args.accumulate_allreduce_grads_in_fp32 = True
print_rank_0('accumulate and all-reduce gradients in fp32 for bfloat16 data type.')
if args.cuda_graph_impl == "local" and CudaGraphScope.full_iteration in args.cuda_graph_scope:
- if not args.inference_dynamic_batching:
- assert not args.check_for_nan_in_loss_and_grad, \
- "--no-check-for-nan-in-loss-and-grad should be set with full_iteration CUDA graph"
- else:
- assert args.fp8 is None, \
- "fp8 is not supported with inference dynamic batching and full_iteration CUDA graph"
+ assert not args.check_for_nan_in_loss_and_grad, \
+ "--no-check-for-nan-in-loss-and-grad should be set with --cuda-graph-scope=full_iteration for training. Note: If you are trying to use full_iteration CUDA graphs for inference, please use --cuda-graph-scope full_iteration_inference instead"
+
+ if args.cuda_graph_impl == "local" and CudaGraphScope.full_iteration_inference in args.cuda_graph_scope:
+ if args.fp8 is not None:
+ assert args.transformer_impl == "inference_optimized", \
+ "fp8 with full_iteration_inference CUDA graphs is only supported with " \
+ "--transformer-impl=inference_optimized"
+ assert args.fp8_recipe == "mxfp8", \
+ "Only --fp8-recipe=mxfp8 is supported with full_iteration_inference CUDA graphs"
+
+ if args.cuda_graph_impl == 'local':
+ assert args.inference_dynamic_batching_num_cuda_graphs > 0 or args.inference_dynamic_batching_num_cuda_graphs == -1, \
+ 'inference_dynamic_batching_num_cuda_graphs should be a positive integer or -1' \
+ '-1 means that we will automatically determine the number of CUDA graphs to capture based on the `max_requests` value.'
print_rank_0('using {} for parameters ...'.format(args.params_dtype))
@@ -852,7 +1064,8 @@ def validate_args(args, defaults={}):
args.variable_seq_lengths = False
# Iteration-based training.
- if args.train_iters:
+ # Skip these checks when skip_train is set: LR config is irrelevant.
+ if args.train_iters and not args.skip_train:
# If we use iteration-based training, make sure the
# sample-based options are off.
assert args.train_samples is None, \
@@ -868,7 +1081,7 @@ def validate_args(args, defaults={}):
'can only specify one of lr-warmup-fraction and lr-warmup-iters'
# Sample-based training.
- if args.train_samples:
+ if args.train_samples and not args.skip_train:
# If we use sample-based training, make sure the
# iteration-based options are off.
assert args.train_iters is None, \
@@ -1173,7 +1386,6 @@ def validate_args(args, defaults={}):
args.fim_eod_token,
]
assert not args.mock_data, "Mock dataset is not supported with FIM dataset."
- assert not args.legacy_tokenizer, "FIM dataset is not supported with legacy tokenizers."
assert args.fim_rate, "--fim-rate should be specified."
assert args.fim_spm_rate, "--fim-spm-rate should be specified."
assert all(token is not None for token in extra_tokens), "FIM extra tokens should be specified."
@@ -1201,7 +1413,7 @@ def validate_args(args, defaults={}):
args.iterations_to_skip.extend(iterations_to_skip_from_file)
# Make sure all functionality that requires Gloo process groups is disabled.
- if not args.enable_gloo_process_groups:
+ if not args.use_gloo_process_groups:
if args.use_distributed_optimizer:
# If using distributed optimizer, must use distributed checkpointing.
# Legacy checkpointing uses Gloo process groups to collect full distributed
@@ -1218,8 +1430,8 @@ def validate_args(args, defaults={}):
args.check_for_nan_in_loss_and_grad = False
warn_rank_0('check_for_nan_in_loss_and_grad is set to False for fake process group.')
# Disable gloo process groups for fake process group
- args.enable_gloo_process_groups = False
- warn_rank_0('enable_gloo_process_groups is set to False for fake process group.')
+ args.use_gloo_process_groups = False
+ warn_rank_0('use_gloo_process_groups is set to False for fake process group.')
# Checkpointing
if args.ckpt_fully_parallel_save_deprecated and args.rank == 0:
@@ -1241,6 +1453,13 @@ def validate_args(args, defaults={}):
print('--dist-ckpt-format is deprecated and has no effect.'
' Use --ckpt-format to select the checkpoint format.')
+ if args.use_dist_ckpt and args.ckpt_fully_parallel_load:
+ if args.ckpt_fully_parallel_load_exchange_algo != "broadcast":
+ warn_rank_0(
+ "Currently only the 'broadcast' exchange algorithm is supported for fully parallel load. "
+ "Other algorithms cannot guarantee numerical stability yet."
+ )
+
if args.load_main_params_from_ckpt:
assert args.no_load_optim, '--load-main-params-from-ckpt must be used with --no-load-optim.'
@@ -1273,7 +1492,7 @@ def validate_args(args, defaults={}):
assert args.inference_dynamic_batching_buffer_size_gb is not None
assert args.inference_dynamic_batching_block_size % 256 == 0, "block size should be a multiple of 256"
- if args.cuda_graph_impl == "local" and args.expert_model_parallel_size > 1:
+ if args.cuda_graph_impl == "local" and args.expert_model_parallel_size > 1 and args.transformer_impl != "inference_optimized":
assert args.moe_pad_experts_for_cuda_graph_inference, \
"--moe-pad-experts-for-cuda-graph-inference must be set when using CUDA graphs with expert parallelism"
@@ -1288,16 +1507,22 @@ def validate_args(args, defaults={}):
warn_rank_0('enabling --no-load-rng for upcycling.')
# --skip-train checks.
- if args.skip_train and not args.no_load_optim:
+ # In RL inference-only mode, --no-load-optim is user-controlled: it determines whether the
+ # optimizer is created (needed for --rl-offload-optimizer-during-inference) or skipped entirely.
+ if args.skip_train and not args.perform_rl_step and not args.no_load_optim:
args.no_load_optim = True
warn_rank_0('enabling --no-load-optim when skipping training.')
+ if args.skip_train and args.perform_rl_step and args.no_load_optim and args.rl_offload_optimizer_during_inference:
+ assert False, \
+ '--no-load-optim with --skip-train --perform-rl-step skips the optimizer; ' \
+ '--rl-offload-optimizer-during-inference is incompatible (no optimizer to offload).'
# Muon optimizer check
if 'muon' in args.optimizer:
- # TODO: remove these checks once we support them
- assert not args.overlap_grad_reduce, "Muon optimizer does not support overlap grad reduce for now."
- assert not args.overlap_param_gather, "Muon optimizer does not support overlap param gather for now."
+ if args.optimizer == 'muon':
+ assert not args.overlap_grad_reduce, "Muon optimizer does not support overlap grad reduce. Use dist_muon instead."
+ assert not args.overlap_param_gather, "Muon optimizer does not support overlap param gather. Use dist_muon instead."
assert not args.use_distributed_optimizer, "Muon optimizer does not support distributed optimizer for now."
assert not args.use_torch_fsdp2, "Muon optimizer does not support Torch-FSDP2 for now."
@@ -1331,6 +1556,25 @@ def validate_args(args, defaults={}):
assert is_te_min_version("2.8.0"), (
"overlap_grad_reduce is only supported with TE >= 2.8.0 when enabling delay_wgrad_compute"
)
+ wgrad_in_graph_scope = CudaGraphScope.attn in args.cuda_graph_scope or (
+ CudaGraphScope.moe_router in args.cuda_graph_scope
+ and args.moe_shared_expert_intermediate_size is not None
+ and not args.moe_shared_expert_overlap
+ )
+ if wgrad_in_graph_scope:
+ assert is_te_min_version(
+ "2.12.0"
+ ), "CUDA graph with delay_wgrad_compute requires TE version >= 2.12.0."
+ assert args.gradient_accumulation_fusion, (
+ 'CUDA graph with delay_wgrad_compute requires gradient_accumulation_fusion '
+ 'to be enabled. This is because the default gradient accumulation does not '
+ 'use static memory addresses, which breaks CUDA graph requirements.'
+ )
+ if CudaGraphScope.attn in args.cuda_graph_scope:
+ assert (
+ not args.add_bias_linear and not args.add_qkv_bias
+ ), "CUDA graph with delay_wgrad_compute doesn't support attn bias for now."
+
if not args.gradient_accumulation_fusion:
assert is_te_min_version("2.7.0"), (
"disabling gradient_accumulation_fusion is only supported with TE >= 2.7.0 "
@@ -1346,9 +1590,11 @@ def validate_args(args, defaults={}):
if args.mtp_num_layers:
assert not args.use_legacy_models, "The legacy Megatron models does not support Multi-Token Prediction (MTP)."
- assert args.position_embedding_type == "rope" or args.position_embedding_type == "none", (
- f"Multi-Token Prediction (MTP) is not supported with {args.position_embedding_type} position embedding type."
- + f"The supported position embedding types are rope and none."
+ # MTP is compatible with position embedding types that use position_ids.
+ supported_position_types = ["learned_absolute", "rope", "mrope", "none"]
+ assert args.position_embedding_type in supported_position_types, (
+ f"Multi-Token Prediction (MTP) is not supported with '{args.position_embedding_type}' position embedding type. "
+ f"The supported position embedding types are: {', '.join(supported_position_types)}."
)
if args.cpu_offloading_num_layers > 0:
@@ -1382,13 +1628,15 @@ def validate_args(args, defaults={}):
if args.multi_latent_attention:
assert not args.group_query_attention, "Group query attention is mutually exclusive with multi latent attention."
+
+ if args.mla_down_proj_fusion:
+ assert args.multi_latent_attention, "--mla-down-proj-fusion requires --multi-latent-attention"
# MoE latent projections
if args.moe_latent_size is not None:
assert args.moe_latent_size > 0, "MoE latent projection dimension has to be greater than zero."
assert args.num_experts is not None, "MoE latent projections are applicable only for MoE models."
assert not args.use_legacy_models, "MoE latent projections are only supported for mcore models."
- assert not args.moe_use_legacy_grouped_gemm, "MoE latent projection is not supported yet with legacy grouped GEMM."
if args.tiktoken_special_tokens and not args.tokenizer_special_tokens:
warn_rank_0(
@@ -1396,6 +1644,20 @@ def validate_args(args, defaults={}):
"Use --tokenizer-special-tokens instead."
)
args.tokenizer_special_tokens = args.tiktoken_special_tokens
+
+ if args.tokenizer_hf_use_fast:
+ warn_rank_0(
+ "--tokenizer-hf-use-fast argument is deprecated and will be removed soon. "
+ "`use_fast` is set to True by default for HF tokenizers."
+ "Use --tokenizer-hf-no-use-fast if you want to disable `use_fast`."
+ )
+
+ if args.tokenizer_hf_include_special_tokens:
+ warn_rank_0(
+ "--tokenizer-hf-include-special-tokens argument is deprecated and will be removed soon. "
+ "`include_special_tokens` is set to True by default for HF tokenizers."
+ "Use --tokenizer-hf-no-include-special-tokens if you want to disable `include_special_tokens`."
+ )
# Print arguments.
_print_args("arguments", args)
@@ -1480,8 +1742,8 @@ def core_transformer_config_from_args(args, config_class=None):
if len(args.cp_comm_type) == 1:
kw_args['cp_comm_type'] = args.cp_comm_type[0]
- if args.is_hybrid_model:
- kw_args['is_hybrid_model'] = args.is_hybrid_model
+ if args.hybrid_layer_pattern is not None:
+ kw_args['is_hybrid_model'] = True
kw_args['inference_sampling_seed'] = args.seed
@@ -1567,8 +1829,9 @@ def _add_inference_args(parser):
'including the shared experts if they are not overlapped with EP comm. '
'"moe_preprocess": captures operations in MoELayer.preprocess(). Must be used together with "moe_router". '
'"mamba": captures the mamba layer. '
- '"full_iteration": captures a whole iteration. '
- 'full_iteration scope is only supported with --cuda-graph-impl=local, other scopes are only supported with --cuda-graph-impl=transformer_engine. '
+ '"full_iteration": captures a whole training iteration. '
+ '"full_iteration_inference": captures a whole inference iteration. '
+ 'full_iteration and full_iteration_inference scopes are only supported with --cuda-graph-impl=local, other scopes are only supported with --cuda-graph-impl=transformer_engine. '
'If not specified, the default scope is to capture the whole Transformer layer. '
'For backward compatibility, we still allow passing "full" to specify capturing the whole layer, and convert it to an empty list.')
group.add_argument('--use-legacy-static-engine', action='store_true', default=False,
@@ -1576,13 +1839,10 @@ def _add_inference_args(parser):
dest='use_legacy_static_engine')
group.add_argument('--inference-max-requests', type=int, default=8,
help='Maximum number of requests for inference.',
- dest='inference_max_batch_size')
+ dest='inference_max_requests')
group.add_argument('--inference-max-seq-length', type=int, default=2560,
help='Maximum sequence length expected for inference (prefill + decode).',
dest='inference_max_seq_length')
- group.add_argument('--inference-max-batch-size', type=int, default=None,
- help='Maximum batch size for inference.',
- dest='inference_max_batch_size')
group.add_argument('--inference-dynamic-batching',
action='store_true', default=False,
help='Enable dynamic batching mode.')
@@ -1604,6 +1864,10 @@ def _add_inference_args(parser):
'the dynamic inference context. Active requests are '
'paused when there are not enough active blocks available '
'to continue generating a request.')
+ group.add_argument('--inference-dynamic-batching-mamba-memory-ratio', type=float, default=None,
+ help='Percentage of memory buffer to allocate for Mamba states. '
+ 'If not specified, allocates Mamba state tensors for each KV cache block. '
+ 'Only used for hybrid models.')
group.add_argument('--inference-dynamic-batching-block-size',
type=int, default=256,
help='KV cache block size. '
@@ -1622,12 +1886,19 @@ def _add_inference_args(parser):
'cuda graph batch sizes range from 1 to `max_requests`. '
'(See `dynamic_context.py` for details on how '
'`max_requests` is computed). Due to rounding, the actual '
- 'number of cuda graphs may not equal this argument.')
+ 'number of cuda graphs may not equal this argument.'
+ 'The user can also pass -1, in which case we automatically determine the number of graphs ' \
+ 'to capture based on the `max_requests`.')
group.add_argument('--inference-dynamic-batching-track-paused-request-events',
action='store_true',
help='Track paused request ids by adding \'paused\' events '
'to each request\'s event history. This has a very minor '
'impact on latency.')
+ group.add_argument('--inference-dynamic-batching-track-generated-token-events',
+ action='store_true',
+ help='Track per-token events with timestamps for each generated token. '
+ 'When enabled, each generated token creates a GENERATED_TOKEN event '
+ 'with a timestamp, useful for per-token latency analysis.')
group.add_argument('--decode-only-cuda-graphs',
action='store_true', default=False,
help='Only use cuda graphs for decode-only steps, not prefill and mixed steps.')
@@ -1638,15 +1909,48 @@ def _add_inference_args(parser):
'1) allocate `memory_buffer` in unified memory. '
'Eventually, additional levels will be included to '
'control other tensors within the context.')
- group.add_argument('--nccl-all-reduce-for-prefill',
+ group.add_argument('--enable-chunked-prefill', dest='enable_chunked_prefill',
action='store_true', default=False,
- help='When using symmeric all reduce kernels this will use regular nccl kernels for prefill. This can be more effecient when prefill is large as the nccl kernels can be more bandwith optimized')
- # TODO(ksanthanam): Clean this up in future PR
- group.add_argument('--enable-chunked-prefill', dest='disable_chunked_prefill',
- action='store_false', default=True,
help="Enable chunked prefill (disabled by default)")
- group.add_argument('--disable-chunked-prefill', dest='disable_chunked_prefill',
- action='store_true', help=argparse.SUPPRESS)
+ group.add_argument('--num-speculative-tokens', type=int, default=0,
+ help='Number of speculative tokens generated during decode')
+ group.add_argument('--inference-dynamic-batching-prefix-caching',
+ dest='inference_dynamic_batching_enable_prefix_caching',
+ action=argparse.BooleanOptionalAction,
+ default=False,
+ help='Enable/disable prefix caching for dynamic batching inference. '
+ 'When disabled, KV cache blocks cannot be shared between '
+ 'requests with identical prompt prefixes.')
+ group.add_argument('--inference-dynamic-batching-prefix-caching-eviction-policy',
+ type=str, default='ref_zero',
+ choices=['ref_zero', 'lru'],
+ dest='inference_dynamic_batching_prefix_caching_eviction_policy',
+ help='Eviction policy for prefix caching blocks. '
+ '"ref_zero" (default) immediately returns blocks to the '
+ 'free pool when ref_count hits 0. "lru" keeps blocks '
+ 'cached and evicts via LRU only when space is needed.')
+ group.add_argument('--inference-dynamic-batching-prefix-caching-coordinator-policy',
+ type=str, default='first_prefix_block',
+ choices=['longest_prefix', 'first_prefix_block', 'round_robin'],
+ dest='inference_dynamic_batching_prefix_caching_coordinator_policy',
+ help='Coordinator routing policy for prefix caching. '
+ '"first_prefix_block" (default) routes based on the first '
+ 'block hash only. "longest_prefix" routes to the rank with '
+ 'the longest matching prefix. "round_robin" ignores prefix '
+ 'affinity and cycles through ranks.')
+ group.add_argument('--inference-dynamic-batching-prefix-caching-routing-alpha',
+ type=float, default=0.5,
+ dest='inference_dynamic_batching_prefix_caching_routing_alpha',
+ help='Weight for prefix-aware routing score: '
+ 'score = alpha * match + (1 - alpha) * normalized_load. '
+ 'Higher alpha favors prefix cache hits; lower alpha '
+ 'favors load balance. Default: 0.5.')
+ group.add_argument('--inference-dynamic-batching-prefix-caching-mamba-gb',
+ type=float, default=None,
+ dest='inference_dynamic_batching_prefix_caching_mamba_gb',
+ help='GPU memory budget (in GB) for the Mamba state cache '
+ 'used by prefix caching on hybrid models. When set, Mamba '
+ 'states at block boundaries are cached for reuse.')
group.add_argument('--inference-dynamic-batching-cuda-graph-max-tokens',
type=int, default=16384,
help='Maximum number of tokens to capture in a cuda graph.')
@@ -1656,10 +1960,21 @@ def _add_inference_args(parser):
group.add_argument('--inference-logging-step-interval', type=int, default=0,
help='Step interval for logging inference metrics. '
'Default to 0 to disable inference logging.')
+ group.add_argument('--inference-text-gen-server-logging', action=argparse.BooleanOptionalAction,
+ required=False, default=False,
+ help='Enable per-request logging in the inference text generation server.')
group.add_argument('--inference-wandb-logging', action=argparse.BooleanOptionalAction,
required=False, default=False, help='Enable inference wandb logging.')
- group.add_argument("--inference-coordinator-port", type=int, default=12346,
+ group.add_argument("--inference-coordinator-port", type=int,
help="This port will be used to setup the inference coordinator on node-0")
+ group.add_argument('--mamba-inference-conv-states-dtype', type=str,
+ choices=['bf16', 'fp16', 'fp32'], default='bf16',
+ help='Dtype for the Mamba inference conv states tensor')
+ group.add_argument('--mamba-inference-ssm-states-dtype', type=str,
+ choices=['bf16', 'fp16', 'fp32'], default='bf16',
+ help='Dtype for the Mamba inference SSM states tensor')
+ group.add_argument('--inference-use-synchronous-zmq-collectives', action=argparse.BooleanOptionalAction,
+ required=False, default=False, help='Use synchronous ZMQ collectives for inference. Helps in reducing performance variability for MoEs.')
return parser
@@ -1837,7 +2152,7 @@ def _add_network_size_args(parser):
return parser
def _add_straggler_detector_args(parser):
- from megatron.training.resilience_config import StragglerDetectionConfig
+ from megatron.training.config import StragglerDetectionConfig
straggler_factory = ArgumentGroupFactory(StragglerDetectionConfig)
group = straggler_factory.build_group(parser, "straggler")
@@ -1945,7 +2260,7 @@ def _add_ft_package_args(parser):
def _add_logging_args(parser):
- from megatron.training.training_config import LoggerConfig
+ from megatron.training.config import LoggerConfig
log_factory = ArgumentGroupFactory(LoggerConfig, exclude = ["log_throughput_to_tensorboard", "throughput_window_size", "memory_keys", "log_l2_norm_grad_to_tensorboard", "log_runtime_to_tensorboard", "runtime_time_unit", "filter_warnings", "modules_to_filter", "set_level_for_all_loggers", "save_config_filepath"])
group = log_factory.build_group(parser, title="logging")
@@ -1982,10 +2297,17 @@ def _add_regularization_args(parser):
help='Whether to use Nesterov-style momentum in the internal SGD')
group.add_argument('--muon-scale-mode', type=str, default='spectral',
choices=['spectral', 'unit_rms_norm', 'shape_scaling'],
- help='Scale mode for Muon optimizer')
+ help='Scale mode for Muon optimizer. With MuP, set '
+ '--muon-scale-mode unit_rms_norm to use unit_rms_norm scaling, '
+ 'or set --muon-scale-mode spectral to keep spectral scaling.')
group.add_argument('--muon-fp32-matmul-prec', type=str, default='medium',
choices=['low', 'medium', 'high'],
help='FP32 matmul precision for Newton-Schulz iteration')
+ group.add_argument('--muon-coefficient-type', type=str, default='quintic',
+ help='Newton-Schulz coefficient type for the Muon optimizer. '
+ 'Valid types are discovered from the installed emerging_optimizers '
+ 'package (e.g. simple, quintic, polar_express, aol). '
+ 'Validated at optimizer creation time.')
group.add_argument('--muon-num-ns-steps', type=int, default=5,
help='Number of Newton-Schulz steps for Muon optimizer')
group.add_argument('--muon-tp-mode', type=str, default='blockwise',
@@ -1993,6 +2315,16 @@ def _add_regularization_args(parser):
help='How to perform NS calculation for tensor model parallel weights')
group.add_argument('--muon-extra-scale-factor', type=float, default=1.0,
help='Additional scale factor for the muon update')
+ group.add_argument('--muon-scalar-optimizer', type=str, default='adam',
+ choices=['adam', 'lion'],
+ help='Optimizer for scalar parameters (embeddings, biases, norms) '
+ 'when using muon. Defaults to adam.')
+ group.add_argument('--lion-beta1', type=float, default=0.95,
+ help='First beta coefficient for Lion optimizer '
+ '(used in sign update). Default: 0.95.')
+ group.add_argument('--lion-beta2', type=float, default=0.98,
+ help='Second beta coefficient for Lion optimizer '
+ '(used in momentum EMA update). Default: 0.98.')
group.add_argument('--no-weight-decay-cond-type', type=str, choices=['apply_wd_to_qk_layernorm'],
help='Type of no weight decay condition. Choices: '
@@ -2016,6 +2348,21 @@ def _add_rl_args(parser):
help="Number of GRPO groups (G in the paper).")
group.add_argument('--grpo-group-size', type=int, default=2,
help="Number of samples per a GRPO group.")
+ group.add_argument('--rl-num-parallel-generations', type=int, default=None,
+ help='Number of rollouts being generated by the inference engine simultaneously. '
+ 'Internally divided by grpo_group_size. '
+ 'Requires --rl-partial-rollouts. '
+ 'Mutually exclusive with --rl-num-parallel-generation-batches.')
+ group.add_argument('--rl-num-parallel-generation-batches', type=int, default=None,
+ help='Number of generation batches in flight. '
+ 'Set to L+1 to allow for L steps of staleness between the inference and training policies. '
+ 'Each batch contains grpo_prompts_per_step groups by default. '
+ 'Requires --rl-partial-rollouts. '
+ 'Mutually exclusive with --rl-num-parallel-generations.')
+ group.add_argument('--rl-generation-batch-size', type=int, default=None,
+ help='Override the number of groups per generation batch. '
+ 'Defaults to grpo_prompts_per_step when '
+ '--rl-num-parallel-generation-batches is set.')
group.add_argument('--grpo-iterations', type=int, default=2,
help="Number of iterations per a GRPO implementation.")
# As in DAPO, we keep upper/lower eps different.
@@ -2030,12 +2377,6 @@ def _add_rl_args(parser):
help="Entropy term weight in GRPO loss.")
group.add_argument('--grpo-filter-groups-with-same-reward', action='store_true',
help="Filter groups with same reward.")
- group.add_argument('--langrl-inference-server-type', type=str,
- choices=['inplace_megatron', 'inplace_megatron_chat'], default='inplace_megatron',
- help="Type of inference server to use.")
- group.add_argument('--langrl-inference-server-conversation-template', type=str, default=None,
- help="Conversation template, if using a chat server.")
- group.add_argument('--langrl-external-server', action=argparse.BooleanOptionalAction, required=False, default=False)
group.add_argument('--langrl-env-config', type=str, default=None,
help="Path to YAML config file for RL environment configuration.")
group.add_argument('--rl-default-temperature', type=float, default=1.0,
@@ -2046,14 +2387,21 @@ def _add_rl_args(parser):
help="Default top-k for model inference.")
group.add_argument('--rl-offload-optimizer-during-inference', action='store_true',
help='Offload optimizer state to CPU during inference/rollout to save GPU memory')
- group.add_argument('--rl-offload-kv-cache-during-training', action=argparse.BooleanOptionalAction, default=False,
- help='Offload KV cache to CPU during training to save GPU memory')
- group.add_argument('--rl-remove-kv-cache-during-training', action=argparse.BooleanOptionalAction, default=False,
- help='Remove KV cache during training to save GPU memory')
- group.add_argument('--rl-reset-cuda-graphs', action=argparse.BooleanOptionalAction, type=bool, default=False,
- help='Reset CUDA graphs between inference/training to save GPU memory')
+ group.add_argument('--rl-kv-cache-management-mode', type=str, default='persist',
+ choices=['persist', 'offload', 'recompute'],
+ help='KV cache management mode during RL training: '
+ 'persist: leave KV cache in GPU memory (default), '
+ 'offload: offload KV cache to CPU during training, '
+ 'recompute: deallocate KV cache and recompute from scratch each cycle')
+ group.add_argument('--rl-persist-cuda-graphs', action=argparse.BooleanOptionalAction, type=bool, default=False,
+ help='Persist CUDA graphs when the inference engine is suspended. '
+ 'If False, CUDA graphs are deleted on suspend and re-captured on resume.')
group.add_argument('--rl-partial-rollouts', action=argparse.BooleanOptionalAction, default=False,
- help='If set, use partial rollouts.')
+ help='Allow inference to continue generating rollouts while training updates '
+ 'the policy weights. This enables off-policy training where rollouts may '
+ 'be generated with a stale version of the policy. Use '
+ '--rl-num-parallel-generations or --rl-num-parallel-generation-batches '
+ 'to control the degree of staleness.')
group.add_argument('--rl-inference-logprobs-is-correction', action=argparse.BooleanOptionalAction, type=bool, default=False,
help='If set, use inference logprobs in importance sampling correction of the loss.')
group.add_argument('--rl-importance-sampling-truncation-coef', type=float, default=None,
@@ -2069,7 +2417,7 @@ def _add_rl_args(parser):
'round-robin: distribute bins cyclically across ranks for better load balancing')
group.add_argument('--rl-training-cuda-graphs', action=argparse.BooleanOptionalAction, type=bool,
default=False,
- help='If set, do not call `delete_cuda_graphs` or `toggle_cuda_graphs` when the inference engine is suspended.')
+ help='If set, do not toggle CUDA graphs on/off between inference and training phases.')
group.add_argument('--rl-inference-tensor-model-parallel-size', type=int, default=None,
help='Degree of tensor model parallelism for inference for RL.')
group.add_argument(
@@ -2110,9 +2458,10 @@ def _add_rl_args(parser):
required=False,
default=False,
help=(
- 'When using a separate RL inference model with UVM-enabled parameters, prefetch its weights '
- 'to CPU when not doing rollout inference, and prefetch back to GPU right before inference. '
- 'Requires --rl-inference-model-unified-memory-level=1.'
+ 'When using a separate RL inference model, offload its weights to CPU when not doing rollout '
+ 'inference, and restore to GPU right before inference. Works with two backends: '
+ '1) UVM (when --rl-inference-model-unified-memory-level=1), or '
+ '2) torch_memory_saver (when UVM is not enabled; requires torch_memory_saver to be installed).'
),
)
group.add_argument('--refit-method', type=str, default='gloo',
@@ -2125,15 +2474,18 @@ def _add_rl_args(parser):
help='If set, verify that the model weights were correctly transferred by comparing forward pass outputs on'
'the first swap of model weights.')
- group.add_argument('--rl-parallel-generation-tasks', type=int, default=512,
- help='Number of parallel generation tasks for RL inference.')
+ group.add_argument('--rl-parallel-generation-tasks', type=int, default=None,
+ help='Deprecated: use --rl-num-parallel-generations instead.')
group.add_argument('--rl-skip-bos-token', action=argparse.BooleanOptionalAction, type=bool, default=False,
help='Skip BOS token at the beginning of the sequences. Default is False.')
+ group.add_argument('--rl-inference-parsers', nargs='*', default=[],
+ help='List of response parsers to enable for RL inference '
+ '(e.g. --rl-inference-parsers deepseek-r1-reasoning qwen3-coder-tool).')
return parser
def _add_training_args(parser):
- from megatron.training.training_config import TrainingConfig
- from megatron.training.common_config import ProfilingConfig
+ from megatron.training.config import TrainingConfig
+ from megatron.training.config import ProfilingConfig
prof_factory = ArgumentGroupFactory(ProfilingConfig, exclude=["record_shapes", "nvtx_ranges"])
prof_group = prof_factory.build_group(parser, "profiling")
@@ -2155,9 +2507,6 @@ def _add_training_args(parser):
dest='check_for_large_grads')
group.add_argument('--result-rejected-tracker-filename', type=str, default=None,
help='Optional name of file tracking `result_rejected` events.')
- group.add_argument('--disable-gloo-process-groups', action='store_false',
- dest='enable_gloo_process_groups',
- help='Disables creation and usage of Gloo process groups.')
group.add_argument('--tp-comm-overlap-cfg', type=str, default=None,
help='Config file when tp_comm_overlap is enabled.')
@@ -2192,7 +2541,7 @@ def _add_training_args(parser):
help='use FlashAttention implementation of attention. '
'https://arxiv.org/abs/2205.14135')
group.add_argument('--optimizer', type=str, default='adam',
- choices=['adam', 'sgd', 'muon', 'dist_muon'],
+ choices=['adam', 'sgd', 'muon', 'dist_muon', 'lion'],
help='Optimizer function')
group.add_argument('--optimizer-cpu-offload', action='store_true',
help='Offload optimizer state to CPU')
@@ -2232,16 +2581,12 @@ def _add_training_args(parser):
group.add_argument('--use-legacy-models', action='store_true',
help='Use the legacy Megatron models, not Megatron-Core models.')
- group.add_argument('--high-priority-stream-groups', nargs='*', type=str, default=[],
- help='The communicator group names to use high priority streams.')
- group.add_argument('--disable-jit-fuser', action='store_true',
- help='Disable the JIT fuser.')
return parser
def _add_rerun_machine_args(parser):
- from megatron.training.resilience_config import RerunStateMachineConfig
+ from megatron.training.config import RerunStateMachineConfig
rerun_factory = ArgumentGroupFactory(RerunStateMachineConfig, exclude=["check_for_nan_in_loss"])
group = rerun_factory.build_group(parser, "rerun engine")
@@ -2250,7 +2595,7 @@ def _add_rerun_machine_args(parser):
def _add_initialization_args(parser):
- from megatron.training.common_config import RNGConfig
+ from megatron.training.config import RNGConfig
rng_factory = ArgumentGroupFactory(RNGConfig)
group = rng_factory.build_group(parser, "RNG and initialization")
@@ -2262,7 +2607,7 @@ def _add_initialization_args(parser):
def _add_learning_rate_args(parser):
- from megatron.training.training_config import SchedulerConfig
+ from megatron.training.config import SchedulerConfig
sched_factory = ArgumentGroupFactory(SchedulerConfig, exclude=["no_weight_decay_cond_type"])
group = sched_factory.build_group(parser, title="learning rate and weight decay")
@@ -2287,7 +2632,7 @@ def _add_learning_rate_args(parser):
def _add_checkpointing_args(parser):
- from megatron.training.training_config import CheckpointConfig
+ from megatron.training.config import CheckpointConfig
ckpt_factory = ArgumentGroupFactory(CheckpointConfig, exclude=["most_recent_k", "save_tokenizer_assets", "save_optim", "save_rng", "load_optim", "load_rng"])
group = ckpt_factory.build_group(parser, "checkpointing")
@@ -2306,6 +2651,10 @@ def _add_checkpointing_args(parser):
group.add_argument('--dist-ckpt-format',
dest='dist_ckpt_format_deprecated',
help='Deprecated: see --ckpt-format.')
+ group.add_argument('--dist-ckpt-workers', type=int, default=1,
+ help='Number of workers for distributed checkpointing. '
+ 'Only used for async save. '
+ 'If set to 1, the checkpointing is performed in a single process.')
group.add_argument('--ckpt-fully-parallel-save', action='store_true',
dest='ckpt_fully_parallel_save_deprecated',
help='Deprecated: see --no-ckpt-fully-parallel-save.')
@@ -2344,7 +2693,10 @@ def _add_mixed_precision_args(parser):
def _add_distributed_args(parser):
- group = parser.add_argument_group(title='distributed')
+ from megatron.training.config import DistributedInitConfig
+
+ dist_init_factory = ArgumentGroupFactory(DistributedInitConfig)
+ group = dist_init_factory.build_group(parser, "distributed init")
group.add_argument('--decoder-first-pipeline-num-layers',
type=int, default=None,
@@ -2372,20 +2724,8 @@ def _add_distributed_args(parser):
group.add_argument('--no-overlap-p2p-communication', action='store_false',
help='overlap pipeline parallel communication with forward and backward chunks in 1F1B',
dest='overlap_p2p_comm')
- group.add_argument('--distributed-backend', default='nccl',
- choices=['nccl', 'gloo'],
- help='Which backend to use for distributed training.')
- group.add_argument('--distributed-timeout-minutes', type=int, default=10,
- help='Default timeout minutes for torch.distributed.')
- group.add_argument('--distributed-timeout-seconds-after-init', type=int, default=None,
- help='Timeout seconds for process groups after initialization.'
- 'This timeout is applied to all process groups after initialization.')
group.add_argument('--overlap-grad-reduce', action='store_true',
default=False, help='If set, overlap DDP grad reduce.')
- group.add_argument('--no-align-grad-reduce', action='store_false',
- help='If not set, all PP stages will launch gradient reduces simultaneously. '
- 'Otherwise, each PP stage will independently launch as needed.',
- dest='align_grad_reduce')
group.add_argument('--ddp-num-buckets', type=int, default=None,
help='Number of buckets for data-parallel communication')
group.add_argument('--ddp-bucket-size', type=int, default=None,
@@ -2399,6 +2739,10 @@ def _add_distributed_args(parser):
default=False, help='If set, use a reduce-scatter implementation which sends lower-precision '
'values over the wire (using an all-to-all to keep total communication overhead in line '
'with the standard ring implementation) but performs accumulation locally in FP32.')
+ group.add_argument('--ddp-param-name-patterns-for-fp32-local-accumulation', nargs='+', default=[],
+ help='List of param_name patterns (in Python\'s fnmatch format) to match against '
+ 'to do local gradient accumulation in FP32. The special pattern \'all\' matches '
+ 'every parameter.')
group.add_argument('--ddp-average-in-collective', action='store_true',
default=False, help='If set, average directly in data-parallel communication collective.')
group.add_argument('--overlap-param-gather', action='store_true',
@@ -2412,14 +2756,6 @@ def _add_distributed_args(parser):
group.add_argument('--no-scatter-gather-tensors-in-pipeline', action='store_false',
help='If not set, use scatter/gather to optimize communication of tensors in pipeline.',
dest='scatter_gather_tensors_in_pipeline')
- group.add_argument('--local-rank', type=int, default=int(os.getenv('LOCAL_RANK', '0')),
- help='local rank passed from distributed launcher.')
- group.add_argument('--lazy-mpu-init', type=bool, required=False,
- help='If set to True, initialize_megatron() '
- 'skips DDP initialization and returns function to '
- 'complete it instead. Also turns on '
- '--use-cpu-initialization flag. This is for '
- 'external DDP manager.' )
group.add_argument('--use-distributed-optimizer', action='store_true',
help='Use distributed optimizer.')
group.add_argument('--use-nccl-ub', action='store_true', dest='nccl_ub',
@@ -2432,18 +2768,9 @@ def _add_distributed_args(parser):
group.add_argument('--fsdp-manual-registration', action='store_true', dest='fsdp_manual_registration',
default=False, help='Manually register the FSDP communication buffers to NCCL user buffer.'
'This option is only effective when use-megatron-fsdp and use-nccl-ub is set.')
- group.add_argument('--use-sharp', action='store_true',
- help='Required to enable SHARP communication.')
- group.add_argument('--sharp-enabled-group', type=str, default=None,
- choices=['dp', 'dp_replica'],
- help='IB SHARP can be enabled from only one communication group. '
- 'By default, it is enabled from dp group. '
- 'Available options: [dp, dp_replica]')
group.add_argument('--create-all-gather-group', action='store_true',
help='Create a separate process group for all-gather operations '
'to overlap reduce-scatter and all-gather operations.')
- group.add_argument('--use-megatron-fsdp', action='store_true',
- help='Use the Megatron FSDP code path in DDP.')
group.add_argument('--data-parallel-sharding-strategy', type=str, default='no_shard',
choices=['no_shard', 'optim', 'optim_grads', 'optim_grads_params'],
help='Sharding strategy of data parallelism.')
@@ -2472,9 +2799,6 @@ def _add_distributed_args(parser):
help='If set, enable full sharding in megatron-fsdp Hybrid Sharded Data Parallel (HSDP) mode.')
group.add_argument('--num-distributed-optimizer-instances', type=int, default=1,
help='Number of Distributed Optimizer copies across Data Parallel domain.')
- group.add_argument('--use-torch-fsdp2', action='store_true',
- help='Use the torch FSDP2 implementation. FSDP2 has not been tested with pipeline parallelism, '
- 'and may contain bugs.')
group.add_argument('--torch-fsdp2-no-reshard-after-forward', action='store_false', dest='torch_fsdp2_reshard_after_forward',
help='Whether to reshard weights after forward pass when using PyTorch FSDP2. '
'Set to enable FSDP ZeRO-2.')
@@ -2484,14 +2808,6 @@ def _add_distributed_args(parser):
'all layers will share the same communication type. Users can also '
'specify separated types for each layer like '
'--cp-comm-type p2p p2p a2a a2a a2a+p2p a2a+p2p')
- group.add_argument('--nccl-communicator-config-path', type=str, default=None,
- help='Path to the yaml file with NCCL communicator '
- 'configurations. The number of min/max thread groups and thread '
- 'group cluster size of each communicator can be configured by '
- 'setting `min_ctas`, `max_ctas`, and `cga_cluster_size`.')
- group.add_argument('--use-tp-pp-dp-mapping', action='store_true', default=False,
- help='If set, distributed ranks initialize order is changed '
- 'from tp-cp-ep-dp-pp to tp-cp-ep-pp-dp.')
group.add_argument('--fake-process-group', action='store_true', default=False,
help='If set, initialize with fake distributed process group and all distributed communication operations will be skipped. \
This is quite useful for profiling memory usage of distributed training with just one GPU. \
@@ -2500,7 +2816,7 @@ def _add_distributed_args(parser):
def _add_validation_args(parser):
- from megatron.training.training_config import ValidationConfig
+ from megatron.training.config import ValidationConfig
val_factory = ArgumentGroupFactory(ValidationConfig)
group = val_factory.build_group(parser, "validation")
@@ -2547,8 +2863,6 @@ def _add_tokenizer_args(parser):
group.add_argument('--tokenizer-special-tokens', type=str, nargs='+', default=None,
help='List of special tokens. For TikTokenizer needs to have '
'["", "", " ", "", "", "", ""]')
- group.add_argument('--legacy-tokenizer', action='store_true', default=False,
- help='To use Megatron-LM legacy tokenizer system.')
group.add_argument('--tiktoken-pattern', type=str, default=None,
help='Which tiktoken pattern to use. Options: [v1, v2]')
group.add_argument('--tiktoken-num-special-tokens', type=int, default=1000,
@@ -2558,10 +2872,14 @@ def _add_tokenizer_args(parser):
'["", "", " ", "", "", "", ""]')
group.add_argument('--tokenizer-sentencepiece-legacy', action='store_true', default=False,
help='SentencePiece tokenizer wrapper legacy behavior. Allows special tokens usage.')
- group.add_argument('--tokenizer-hf-use-fast', action='store_true', default=False,
+ group.add_argument('--tokenizer-hf-use-fast', action='store_true', default=True,
help='Whether to use fast HuggingFace tokenizer.')
- group.add_argument('--tokenizer-hf-include-special-tokens', action='store_true', default=False,
+ group.add_argument('--tokenizer-hf-include-special-tokens', action='store_true', default=True,
help='Converting text to ids will include special for HuggingFace tokenizer.')
+ group.add_argument('--tokenizer-hf-no-use-fast', action='store_true', default=False,
+ help='Whether to use fast HuggingFace tokenizer.')
+ group.add_argument('--tokenizer-hf-no-include-special-tokens', action='store_true', default=False,
+ help='Converting text to ids will not include special for HuggingFace tokenizer.')
group.add_argument("--trust-remote-code", action="store_true", default=False,
help='Whether or not to allow PreTrainedTokenizer to execute remote code')
return parser
@@ -3265,10 +3583,6 @@ def _add_moe_args(parser):
group.add_argument('--moe-upcycling-granularity', type=int, default=1,
help='This param sepecifics how many times smaller is the expert hidden size compared with the original dense FFN hidden size. '
'For using granular upcycling strategy, please set this param as a positive integer. If this param is set to 1, it means using the default upcycling strategy.')
- group.add_argument('--moe-pad-experts-for-cuda-graph-inference', action='store_true',
- help="some MoE routers have a D2H sync that will break cuda graphs. If this flag is set the router will switch" \
- " to dropping and padding during decode time which does not have a D2H sync. The capacity factor is set to the" \
- " max that an expert could see during inference so no tokens are actually dropped.")
return parser
def _add_mla_args(parser):
@@ -3291,6 +3605,13 @@ def _add_mla_args(parser):
help="Mscale all dimensions for YaRN RoPE in multi-latent attention.")
group.add_argument('--cache-mla-latents', action='store_true', default=False,
help="If set caches the mla down projected latents with mla flash decode.")
+ group.add_argument(
+ '--mla-down-proj-fusion',
+ action='store_true',
+ default=False,
+ help="Enable fused q/kv down-projection and fused input layernorm when backend supports. "
+ "Otherwise fall back to the unfused MLA.",
+ )
return parser
@@ -3370,32 +3691,28 @@ def _add_experimental_args(parser):
'To use local spec specify local as the argument.'
'For more details, see the model class, '
'`transformer_block.py`, or `transformer_layer.py`')
- group.add_argument('--hybrid-attention-ratio', type=float, default=0.0,
- help='Ratio of attention layers to total layers, in the '
- 'range [0.0, 1.0].')
- group.add_argument('--hybrid-mlp-ratio', type=float, default=0.0,
- help='Ratio of mlp layers to total layers, in the '
- 'range [0.0, 1.0].')
+ group.add_argument('--hybrid-layer-pattern', type=str, default=None,
+ help='Specify a hybrid layer pattern using M (mamba), G (gdn), '
+ '* (attention), - (mlp), E (moe). Use | to define pipeline stage '
+ 'boundaries for flexible virtual pipeline parallel (fVPP). Use / to '
+ 'separate MTP patterns. Example: "M-M-|M-M*-|M-M-|M-M*-" or "M-M-|M-M*-/MM/MM". '
+ 'When this flag is used, it is the sole indicator that a hybrid model '
+ 'is being run.')
group.add_argument('--hybrid-override-pattern', type=str, default=None,
- help='Force a specific hybrid layer pattern. The value'
- 'should be a string of characters chosen from'
- 'core.ssm.mamba_hybrid_layer_allocation.Symbols.'
- 'If a value greater than 0.0 is supplied to any of the '
- 'hybrid ratio arguments, then the number of each type'
- 'of layer in the override pattern must match number in'
- 'the overidden pattern')
+ help='Deprecated. Use --hybrid-layer-pattern instead. '
+ 'If specified, its value will be forwarded to --hybrid-layer-pattern.')
group.add_argument('--yaml-cfg', type=str, default=None,
help = 'Config file to add additional arguments')
- # Args of precision-aware optimizer
+ # Args of precision-aware optimizer.
group.add_argument('--use-precision-aware-optimizer', action='store_true',
help='Use the precision-aware optimizer in TransformerEngine, which allows '
'setting the main params and optimizer states to lower precision, such as '
'fp16, bf16 and fp8.')
group.add_argument('--main-grads-dtype', default='fp32', choices=['fp32', 'bf16'],
- help='Dtype of main grads when enabling precision-aware-optimizer')
+ help='Dtype of main grads when enabling precision-aware-optimizer.')
group.add_argument('--main-params-dtype', default='fp32', choices=['fp32', 'fp16'],
- help='Dtype of main params when enabling precision-aware-optimizer')
+ help='Dtype of main params when enabling precision-aware-optimizer.')
group.add_argument('--exp-avg-dtype', default='fp32', choices=['fp32', 'fp16', 'bf16', 'fp8'],
help='Dtype of exp_avg (1st moment in adam optimizer) when enabling '
'precision-aware-optimizer. This dtype is used for storing the '
@@ -3406,6 +3723,22 @@ def _add_experimental_args(parser):
'precision-aware-optimizer. This dtype is used for storing the '
'optimizer state in memory during training but does not affect '
'the precision in the kernel computation.')
+
+ # Megatron-FSDP Arguments
+ group.add_argument('--megatron-fsdp-main-params-dtype', default='fp32', choices=['fp32', 'bf16', 'fp16', 'auto'],
+ help="Data type for the main weight buffer utilized for distributed optimization "
+ "and quantization with Megatron-FSDP. If 'auto', then the native model parameter "
+ "data-type will be used for the main weight data-type.")
+ group.add_argument('--megatron-fsdp-main-grads-dtype', default='auto', choices=['fp32', 'bf16', 'fp16', 'auto'],
+ help="Data type for the main gradient buffer utilized for distributed optimization "
+ "with Megatron-FSDP. If 'auto', then the native model gradient data-type will "
+ "be used for the main gradient / accumulation data-type.")
+ group.add_argument("--megatron-fsdp-grad-comm-dtype", default='auto', choices=['fp32', 'fp16', 'bf16', 'auto'],
+ help="When using Megatron-FSDP, this controls the data-type used when communicating "
+ "model gradients during FSDP. If 'auto', then the main gradient data-type will "
+ "be used for the gradient communication / reduction data-type. When using NCCL "
+ "v2.27+, reduction is always computed in FP32 if using NCCL Symmetric kernels.")
+
return parser
diff --git a/megatron/training/async_utils.py b/megatron/training/async_utils.py
index 6cf0c1e5586..4e050ab6b17 100644
--- a/megatron/training/async_utils.py
+++ b/megatron/training/async_utils.py
@@ -5,37 +5,84 @@
the async checkpoint save calls.
"""
import logging
+import time
-from megatron.core.dist_checkpointing.strategies.async_utils import AsyncCallsQueue, AsyncRequest
-from megatron.core.dist_checkpointing.strategies.filesystem_async import _results_queue
+from abc import ABC
+
+from megatron.core.dist_checkpointing.strategies.async_utils import AsyncRequest
+from megatron.core.dist_checkpointing.strategies.torch import get_async_strategy
from megatron.training import get_args
from megatron.training.utils import print_rank_0
+try:
+ from nvidia_resiliency_ext.checkpointing.async_ckpt.core import AsyncRequest as NVRxAsyncRequest
+ from nvidia_resiliency_ext.checkpointing.async_ckpt.filesystem_async import _results_queue
+except (ImportError, ModuleNotFoundError):
+ from megatron.core.dist_checkpointing.strategies.filesystem_async import _results_queue
+
+ NVRxAsyncRequest = ABC
+
logger = logging.getLogger(__name__)
# Singleton manager of async calls
-# The default is `TemporalAsyncCaller`
-_async_calls_queue = AsyncCallsQueue()
+_async_calls_queue = None
+
+
+def _get_async_calls_queue():
+ """Get or lazily initialize the async calls queue."""
+ global _async_calls_queue
+ if _async_calls_queue is None:
+ args = get_args()
+ _, async_modules = get_async_strategy(getattr(args, "async_strategy", "nvrx"))
+ AsyncCallsQueue = async_modules["AsyncCallsQueue"]
+ _async_calls_queue = AsyncCallsQueue(
+ persistent=getattr(args, "use_persistent_ckpt_worker", False)
+ )
-def init_persistent_async_worker():
+ return _async_calls_queue
+
+
+def init_persistent_async_worker(rank: int, mp_mode: str = 'spawn'):
global _async_calls_queue
+ args = get_args()
+ async_strategy, async_modules = get_async_strategy(getattr(args, "async_strategy", "nvrx"))
+ AsyncCallsQueue = async_modules["AsyncCallsQueue"]
+ get_write_results_queue = async_modules["get_write_results_queue"]
# Recreate the async_calls_queue for persistent worker
# This duplicate step is for backward compatiblity
+ time_start = time.time()
+ if rank == 0:
+ print(f"init_persistent_async_worker: {rank}, Starting Async Caller", flush=True)
_async_calls_queue = AsyncCallsQueue(persistent=True)
-
-
-def schedule_async_save(async_request: AsyncRequest):
+ # initialize the persistent caller with QoS priorities from args
+ kwargs = {}
+ if async_strategy == "mcore":
+ # Note: nvidia-resiliency-ext uses is_daemon instead of mp_mode (always spawns)
+ kwargs["mp_mode"] = mp_mode
+ AsyncCallsQueue.warmup_persistent_caller(
+ rank,
+ cpu_priority=args.async_ckpt_cpu_priority,
+ io_priority=args.async_ckpt_io_priority,
+ **kwargs,
+ )
+ # initialize ckpt write results queue
+ get_write_results_queue('fork')
+ if rank == 0:
+ print(f"init_persistent_async_worker: rank {rank}, Async Caller Started in {time.time() - time_start} seconds", flush=True)
+
+
+def schedule_async_save(async_request: AsyncRequest | NVRxAsyncRequest):
"""Schedule the async save request.
Args:
- async_request (AsyncRequest): the async save request.
+ async_request (AsyncRequest | NVRxAsyncRequest): the async save request.
"""
- _async_calls_queue.schedule_async_request(async_request)
+ _get_async_calls_queue().schedule_async_request(async_request)
def maybe_finalize_async_save(blocking: bool = False, terminate=False):
- """Finalizes active async save calls.
+ """Finalizes active async save calls and cleans up deletion processes.
Args:
blocking (bool, optional): if True, will wait until all active requests
@@ -51,10 +98,17 @@ def maybe_finalize_async_save(blocking: bool = False, terminate=False):
if blocking and not is_empty_async_queue():
print_rank_0('Unfinalized async checkpoint saves. Finalizing them synchronously now.')
- _async_calls_queue.maybe_finalize_async_calls(blocking, no_dist=False)
+ async_calls_queue = _async_calls_queue
+ if async_calls_queue is not None:
+ async_calls_queue.maybe_finalize_async_calls(blocking, no_dist=False)
+
+ # Clean up finished deletion processes to prevent zombies
+ # Import here to avoid circular dependency
+ from .checkpointing import finalize_deletion_processes
+ finalize_deletion_processes(blocking=blocking or terminate)
- if terminate:
- _async_calls_queue.close()
+ if terminate and async_calls_queue is not None:
+ async_calls_queue.close()
def is_empty_async_queue() -> bool:
@@ -63,11 +117,12 @@ def is_empty_async_queue() -> bool:
Returns:
bool: True if there is any ongoing async call.
"""
- return _async_calls_queue.get_num_unfinalized_calls() == 0
+ return _async_calls_queue is None or _async_calls_queue.get_num_unfinalized_calls() == 0
-def reset_persistent_async_worker():
+def reset_persistent_async_worker(async_strategy):
global _async_calls_queue, _results_queue
+
if _async_calls_queue is not None:
_async_calls_queue.close(abort=True)
del _async_calls_queue
@@ -76,3 +131,5 @@ def reset_persistent_async_worker():
del _results_queue
_results_queue = None
_async_calls_queue = None
+ _, module = get_async_strategy(async_strategy, "CachedMetadataFileSystemReader")
+ module.clear_metadata_cache()
diff --git a/megatron/training/checkpointing.py b/megatron/training/checkpointing.py
index d8abf44fe3d..344a3a83759 100644
--- a/megatron/training/checkpointing.py
+++ b/megatron/training/checkpointing.py
@@ -3,6 +3,7 @@
"""Input/output checkpointing."""
import contextlib
+import multiprocessing
import os
import random
import shutil
@@ -23,7 +24,7 @@
from megatron.core import dist_checkpointing, mpu, tensor_parallel
from megatron.core.dist_checkpointing.mapping import ShardedObject
-from megatron.core.dist_checkpointing.serialization import get_default_load_sharded_strategy
+from megatron.core.dist_checkpointing.strategies.torch import TorchDistLoadShardedStrategy, TorchDistSaveShardedStrategy
from megatron.core.dist_checkpointing.strategies.fully_parallel import (
FullyParallelLoadStrategyWrapper,
FullyParallelSaveStrategyWrapper,
@@ -35,10 +36,10 @@
from megatron.core.rerun_state_machine import get_rerun_state_machine
from megatron.core.utils import get_torch_version, is_torch_min_version
-from ..core.dist_checkpointing.serialization import get_default_save_sharded_strategy
from ..core.dist_checkpointing.utils import _clean_metadata_for_serialization
from . import ft_integration, wandb_utils
from .async_utils import is_empty_async_queue, schedule_async_save
+from megatron.core.dist_checkpointing.strategies.async_utils import AsyncRequest, _disable_gc
from .global_vars import get_args
from .one_logger_utils import on_save_checkpoint_start, on_save_checkpoint_success
from .utils import append_to_progress_log, is_last_rank, print_rank_0, unwrap_model
@@ -70,6 +71,32 @@
logger = getLogger(__name__)
_NON_PERSISTENT_CKPT_SUBDIR = 'non_persistent'
+# Track deletion processes to prevent zombies
+_deletion_processes = []
+
+def finalize_deletion_processes(blocking=False):
+ """Clean up deletion processes to prevent zombie processes.
+
+ Args:
+ blocking (bool): If True, waits for all deletion processes to complete.
+ If False, only joins processes that have already finished.
+
+ Note: Deletion processes are daemon processes (auto-terminate if parent dies),
+ but we still need to join() them to reap zombie processes when they complete normally.
+ The daemon flag and join() serve different purposes:
+ - daemon=True: Auto-terminate if parent process dies abruptly
+ - join(): Reap zombie processes after normal completion
+ """
+ global _deletion_processes
+ finished = []
+ for proc in _deletion_processes:
+ if not proc.is_alive() or blocking:
+ logger.debug(f"Joining deletion process {proc.pid} (blocking={blocking}, is_alive={proc.is_alive()})")
+ proc.join()
+ finished.append(proc)
+ for proc in finished:
+ _deletion_processes.remove(proc)
+
def set_checkpoint_version(value):
global _CHECKPOINT_VERSION
if _CHECKPOINT_VERSION is not None:
@@ -360,27 +387,8 @@ def get_rng_state(ckpt_format: str, tp_group: torch.distributed.ProcessGroup, pp
pp_size = get_pg_size(pp_group)
tp_rank = get_pg_rank(tp_group)
tp_size = get_pg_size(tp_group)
- ep_size = mpu.get_expert_model_parallel_world_size()
-
- if ep_size > 1:
- # Shard RNG by PP, TP, DP when using expert parallelism.
- dp_rank = mpu.get_data_parallel_rank(with_context_parallel=True)
- dp_size = mpu.get_data_parallel_world_size(with_context_parallel=True)
- rng_state_list = ShardedObject(
- 'rng_state',
- rng_state_list,
- (pp_size, tp_size, dp_size),
- (pp_rank, tp_rank, dp_rank),
- replica_id=0,
- )
- else:
- rng_state_list = ShardedObject(
- 'rng_state',
- rng_state_list,
- (pp_size, tp_size),
- (pp_rank, tp_rank),
- replica_id=mpu.get_data_parallel_rank(with_context_parallel=True),
- )
+ rng_state_list = ShardedObject('rng_state', rng_state_list, (pp_size, tp_size), (pp_rank, tp_rank),
+ replica_id=mpu.get_data_parallel_rank(with_context_parallel=True))
elif ckpt_format == "fsdp_dtensor":
pp_rank = mpu.get_pipeline_model_parallel_rank()
tp_rank = mpu.get_tensor_model_parallel_rank()
@@ -614,9 +622,16 @@ def save_checkpoint(iteration, model, optimizer, opt_param_scheduler, num_floati
validate_sharding_integrity = not args.ckpt_assume_constant_structure
else:
validate_sharding_integrity = True
- save_strategy = get_default_save_sharded_strategy(args.ckpt_format)
+ save_strategy = TorchDistSaveShardedStrategy()
if args.ckpt_assume_constant_structure and args.ckpt_format == 'torch_dist':
save_strategy.use_cached_ckpt_structure = args.ckpt_assume_constant_structure
+ if args.async_save:
+ save_strategy.thread_count = args.dist_ckpt_workers
+ else:
+ # We don't allow per-rank parallel save for sync save
+ logger.warning('Per-rank parallel save is not supported for sync save. '
+ 'Setting args.dist_ckpt_workers to 1')
+ save_strategy.thread_count = 1
if checkpointing_context is not None and 'load_strategy' in checkpointing_context:
cached_global_metadata = getattr(checkpointing_context['load_strategy'], 'cached_global_metadata', None)
if cached_global_metadata is not None:
@@ -626,7 +641,11 @@ def save_checkpoint(iteration, model, optimizer, opt_param_scheduler, num_floati
logger.debug("Failed to plug in the read metadata from the load strategy...")
if args.ckpt_fully_parallel_save:
- save_strategy = FullyParallelSaveStrategyWrapper(save_strategy, mpu.get_data_parallel_group(with_context_parallel=True),
+ if args.ckpt_fully_parallel_save_process_group == 'dp':
+ process_group = mpu.get_data_parallel_group(with_context_parallel=True)
+ elif args.ckpt_fully_parallel_save_process_group == 'ep_dp':
+ process_group = mpu.get_expert_data_parallel_group()
+ save_strategy = FullyParallelSaveStrategyWrapper(save_strategy, process_group,
args.ckpt_assume_constant_structure)
# Store save strategy for future checkpoint saves
if checkpointing_context is not None:
@@ -637,7 +656,8 @@ def save_checkpoint(iteration, model, optimizer, opt_param_scheduler, num_floati
async_sharded_save=args.async_save,
validate_access_integrity=validate_sharding_integrity,
preprocess_common_before_consistancy_check=preprocess_common_state_dict_fn,
- content_metadata=_clean_metadata_for_serialization(sharded_sd_metadata))
+ content_metadata=_clean_metadata_for_serialization(sharded_sd_metadata),
+ async_strategy=args.async_strategy)
# [ModelOpt]: save sharded modelopt_state
if has_nvidia_modelopt:
save_sharded_modelopt_state(model, checkpoint_name, (args.ckpt_format, 1))
@@ -733,22 +753,6 @@ def iter_finalize_fn():
append_to_progress_log(f'Saved async checkpoint\tIteration: {iteration}',
barrier=False)
- def delete_checkpoint(args, iteration_to_delete):
- checkpoint_name = get_checkpoint_name(args.save, iteration=iteration_to_delete,
- return_base_dir=True)
- try:
- shutil.rmtree(checkpoint_name) # TODO: Make this work with MSC remote paths?
- print_rank_0(f" [{datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')}] successfully "
- f"deleted checkpoint from iteration {iteration_to_delete:7d} "
- f"at {args.save}")
- if args.log_progress:
- append_to_progress_log(f'Deleted checkpoint\tIteration: {iteration_to_delete}', barrier=False)
- except Exception as e:
- print_rank_0(f' encountered exception "{e}" when trying to delete checkpoint from '
- f'iteration {iteration_to_delete:7d} at {args.save}')
- # Any exception encountered in checkpoint deletion can be ignored and is not fatal.
- pass
-
if save_retain_interval is not None:
if prev_iteration > 0 and prev_iteration != iteration and prev_iteration % save_retain_interval != 0:
checkpoint_name = get_checkpoint_name(args.save, iteration=prev_iteration,
@@ -759,7 +763,23 @@ def delete_checkpoint(args, iteration_to_delete):
f'at {args.save} since it is a symbolic link')
else:
# Asynchronous version of delete_checkpoint(args, iteration_to_delete=prev_iteration).
- threading.Thread(target=delete_checkpoint, args=(args, prev_iteration,)).start()
+ # Use multiprocessing to delete checkpoint in background
+ if args.async_save:
+ # Clean up any finished deletion processes before starting a new one
+ finalize_deletion_processes(blocking=False)
+ ctx = multiprocessing.get_context('fork')
+ delete_process = ctx.Process(
+ target=_async_delete_checkpoint_impl,
+ args=(args.save, prev_iteration, args.log_progress, True,
+ args.async_ckpt_cpu_priority, args.async_ckpt_io_priority),
+ daemon=True
+ )
+ delete_process.start()
+ # Track the process so we can join it later to prevent zombies
+ _deletion_processes.append(delete_process)
+ else:
+ th = threading.Thread(target=_async_delete_checkpoint_impl, args=(args.save, prev_iteration, args.log_progress))
+ th.start()
if args.async_save:
assert async_save_request is not None
@@ -803,6 +823,42 @@ def wandb_finalize_fn():
ft_integration.on_checkpointing_end(is_async_finalization=False)
+@_disable_gc()
+def _async_delete_checkpoint_impl(save_path, iteration_to_delete, log_progress=False, lower_priority=False,
+ cpu_priority=None, io_priority=None):
+ """Module-level function for async checkpoint deletion.
+
+ This function can be pickled and executed by the async worker process.
+ Note: This is only called from rank 0, so we use regular print() instead of print_rank_0()
+ since torch.distributed won't be initialized in the async worker process.
+
+ Args:
+ save_path (str): Path to the checkpoints directory
+ iteration_to_delete (int): Iteration number of checkpoint to delete
+ log_progress (bool): Whether to log progress
+ lower_priority (bool): If True, set process QoS (e.g. nice, ionice) so deletion doesn't contend with training.
+ cpu_priority (int): Nice value for CPU when lower_priority is True (from args.async_ckpt_cpu_priority).
+ io_priority (int): I/O class when lower_priority is True (from args.async_ckpt_io_priority).
+ """
+ if lower_priority:
+ from megatron.core.dist_checkpointing.strategies.async_utils import _set_process_qos
+ _set_process_qos(cpu_priority=cpu_priority, io_priority=io_priority)
+
+ checkpoint_name = get_checkpoint_name(save_path, iteration=iteration_to_delete,
+ return_base_dir=True)
+ try:
+ shutil.rmtree(checkpoint_name) # TODO: Make this work with MSC remote paths?
+ print(f' successfully deleted checkpoint from iteration {iteration_to_delete:7d} '
+ f'at {save_path}', flush=True)
+ if log_progress:
+ append_to_progress_log(f'Deleted checkpoint\tIteration: {iteration_to_delete}', barrier=False)
+ except Exception as e:
+ print(f' encountered exception "{e}" when trying to delete checkpoint from '
+ f'iteration {iteration_to_delete:7d} at {save_path}', flush=True)
+ # Any exception encountered in checkpoint deletion can be ignored and is not fatal.
+ pass
+
+
def cleanup_old_non_persistent_checkpoint(save_dir, leave_ckpt_num=1, do_async=False):
if torch.distributed.is_initialized() and torch.distributed.get_rank() != 0:
return
@@ -1120,15 +1176,27 @@ def _load_global_dist_base_checkpoint(
)
checkpoint_name = get_checkpoint_name(load_dir, iteration, release, return_base_dir=True)
- load_strategy = get_default_load_sharded_strategy(checkpoint_name)
+ load_strategy = TorchDistLoadShardedStrategy(cache_metadata=args.ckpt_assume_constant_structure)
# NOTE: `args.ckpt_fully_parallel_load` applies to both persistent and non-persistent checkpoints.
if args.ckpt_fully_parallel_load:
+ if args.ckpt_fully_parallel_load_process_group == 'dp':
+ process_group = mpu.get_data_parallel_group(with_context_parallel=True)
+ elif args.ckpt_fully_parallel_load_process_group == 'ep_dp':
+ process_group = mpu.get_expert_data_parallel_group()
+ else:
+ raise ValueError(f"Invalid load process group: {args.ckpt_fully_parallel_load_process_group}")
+
load_strategy = FullyParallelLoadStrategyWrapper(
- load_strategy, mpu.get_data_parallel_group(with_context_parallel=True)
+ load_strategy, process_group, exchange_algo=args.ckpt_fully_parallel_load_exchange_algo
)
if checkpointing_context is not None:
checkpointing_context["load_strategy"] = load_strategy
- state_dict = dist_checkpointing.load(sharded_state_dict, checkpoint_name, load_strategy, strict=args.dist_ckpt_strictness)
+ state_dict = dist_checkpointing.load(
+ sharded_state_dict,
+ checkpoint_name,
+ load_strategy,
+ strict=args.dist_ckpt_strictness,
+ )
return state_dict, checkpoint_name, release, CheckpointType.GLOBAL
@@ -1376,6 +1444,18 @@ def load_args_from_checkpoint(
checkpoint_args, 'add_bias_linear', not getattr(checkpoint_args, 'disable_bias_linear')
)
+ # Backward compat: old checkpoints have hybrid_override_pattern but not hybrid_layer_pattern
+ if (getattr(checkpoint_args, 'hybrid_override_pattern', None) is not None
+ and getattr(checkpoint_args, 'hybrid_layer_pattern', None) is None):
+ setattr(
+ checkpoint_args, 'hybrid_layer_pattern',
+ getattr(checkpoint_args, 'hybrid_override_pattern'),
+ )
+ # num_layers is now derived from hybrid_layer_pattern in validate_args, and should not be
+ # set at the same time as hybrid_layer_pattern.
+ if hasattr(checkpoint_args, 'num_layers'):
+ setattr(checkpoint_args, 'num_layers', None)
+
def _set_arg(arg_name, old_arg_name=None, force=False):
if not force and getattr(args, arg_name, None) is not None:
return
@@ -1418,18 +1498,15 @@ def _set_arg(arg_name, old_arg_name=None, force=False):
_set_arg('attention_dropout', force=True)
_set_arg('hidden_dropout', force=True)
- _set_arg('hybrid_override_pattern', force=True)
-
# Legacy MTP pattern for old checkpoints
_set_arg('mtp_hybrid_override_pattern', force=True)
_set_arg('mtp_num_layers', force=True)
_set_arg('mtp_use_repeated_layer', force=True)
_set_arg('spec', force=True)
- _set_arg('hybrid_attention_ratio', force=True)
- _set_arg('hybrid_mlp_ratio', force=True)
_set_arg('num_experts', force=True)
+ _set_arg('mtp_num_layers', force=True)
_set_arg('moe_layer_freq', force=True)
if getattr(checkpoint_args, 'num_experts', None) is not None:
_set_arg('moe_ffn_hidden_size', force=True)
@@ -1449,7 +1526,9 @@ def _set_arg(arg_name, old_arg_name=None, force=False):
_set_arg('mamba_head_dim', force=True)
_set_arg('mamba_num_groups', force=True)
_set_arg('mamba_num_heads', force=True)
- _set_arg('is_hybrid_model', force=True)
+ # We need to be able to override hybrid_layer_pattern from the command-line so that different
+ # pipelining can be specified when re-loading a model (e.g. for inference or post-training).
+ _set_arg('hybrid_layer_pattern')
# Heterogeneous args.
_set_arg('heterogeneous_layers_config_path', force=True)
@@ -1459,8 +1538,8 @@ def _set_arg(arg_name, old_arg_name=None, force=False):
_set_arg('moe_latent_size', force=True)
# Tokenizer args.
- # Using checkpoint version might not always be safe (e.g., if running on different cluster).
if args.use_tokenizer_model_from_checkpoint_args:
+ # Using checkpoint version might not always be safe (e.g., if running on different cluster).
_set_arg('tokenizer_model', force=True)
_set_arg('tokenizer_type', force=True)
_set_arg('tiktoken_pattern', force=True)
diff --git a/megatron/training/common_config.py b/megatron/training/common_config.py
deleted file mode 100644
index d1096e91154..00000000000
--- a/megatron/training/common_config.py
+++ /dev/null
@@ -1,56 +0,0 @@
-# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
-from dataclasses import dataclass, field
-
-@dataclass(kw_only=True)
-class RNGConfig:
- """Configuration settings for random number generation."""
-
- seed: int = 1234
- """Random seed used for python, numpy, pytorch, and cuda."""
-
- te_rng_tracker: bool = False
- """Use the Transformer Engine version of the random number generator.
- Required for CUDA graphs support."""
-
- inference_rng_tracker: bool = False
- """Use a random number generator configured for inference."""
-
- data_parallel_random_init: bool = False
- """Enable random initialization of params across data parallel ranks"""
-
-
-@dataclass(kw_only=True)
-class ProfilingConfig:
- """Configuration settings for profiling the training process."""
-
- use_nsys_profiler: bool = field(default=False, metadata={"argparse_meta": {"arg_names": ["--profile"], "dest": "profile"}})
- """Enable nsys profiling. When using this option, nsys options should be specified in
- commandline. An example nsys commandline is
- `nsys profile -s none -t nvtx,cuda -o --force-overwrite true
- --capture-range=cudaProfilerApi --capture-range-end=stop`.
- """
-
- profile_step_start: int = 10
- """Global step to start profiling."""
-
- profile_step_end: int = 12
- """Global step to stop profiling."""
-
- use_pytorch_profiler: bool = False
- """Use the built-in pytorch profiler. Useful if you wish to view profiles in tensorboard."""
-
- profile_ranks: list[int] = field(default_factory=lambda: [0])
- """Global ranks to profile."""
-
- record_memory_history: bool = False
- """Record memory history in last rank."""
-
- memory_snapshot_path: str = "snapshot.pickle"
- """Specifies where to dump the memory history pickle."""
-
- record_shapes: bool = False
- """Record shapes of tensors."""
-
- nvtx_ranges: bool = False
- """Enable NVTX range annotations for profiling. When enabled, inserts NVTX markers
- to categorize execution in profiler output."""
diff --git a/megatron/training/config/__init__.py b/megatron/training/config/__init__.py
new file mode 100644
index 00000000000..3d346ddd8fe
--- /dev/null
+++ b/megatron/training/config/__init__.py
@@ -0,0 +1,18 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+
+from megatron.training.config.common_config import (
+ RNGConfig,
+ ProfilingConfig,
+ DistributedInitConfig,
+)
+from megatron.training.config.training_config import (
+ TrainingConfig,
+ ValidationConfig,
+ SchedulerConfig,
+ LoggerConfig,
+ CheckpointConfig,
+)
+from megatron.training.config.resilience_config import (
+ RerunStateMachineConfig,
+ StragglerDetectionConfig,
+)
diff --git a/megatron/training/config/common_config.py b/megatron/training/config/common_config.py
new file mode 100644
index 00000000000..cc629424a90
--- /dev/null
+++ b/megatron/training/config/common_config.py
@@ -0,0 +1,153 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+from dataclasses import dataclass, field
+from typing import Literal
+import os
+
+@dataclass(kw_only=True)
+class RNGConfig:
+ """Configuration settings for random number generation."""
+
+ seed: int = 1234
+ """Random seed used for python, numpy, pytorch, and cuda."""
+
+ te_rng_tracker: bool = False
+ """Use the Transformer Engine version of the random number generator.
+ Required for CUDA graphs support."""
+
+ inference_rng_tracker: bool = False
+ """Use a random number generator configured for inference."""
+
+ data_parallel_random_init: bool = False
+ """Enable random initialization of params across data parallel ranks"""
+
+
+@dataclass(kw_only=True)
+class ProfilingConfig:
+ """Configuration settings for profiling the training process."""
+
+ use_nsys_profiler: bool = field(default=False, metadata={"argparse_meta": {"arg_names": ["--profile"], "dest": "profile"}})
+ """Enable nsys profiling. When using this option, nsys options should be specified in
+ commandline. An example nsys commandline is
+ `nsys profile -s none -t nvtx,cuda -o --force-overwrite true
+ --capture-range=cudaProfilerApi --capture-range-end=stop`.
+ """
+
+ profile_step_start: int = 10
+ """Global step to start profiling."""
+
+ profile_step_end: int = 12
+ """Global step to stop profiling."""
+
+ use_pytorch_profiler: bool = False
+ """Use the built-in pytorch profiler. Useful if you wish to view profiles in tensorboard."""
+
+ pytorch_profiler_collect_shapes: bool = False
+ """Collect tensor shape in pytorch profiler."""
+
+ pytorch_profiler_collect_callstack: bool = False
+ """Collect callstack in pytorch profiler."""
+
+ pytorch_profiler_collect_chakra: bool = False
+ """Collect chakra trace in pytorch profiler."""
+
+ profile_ranks: list[int] = field(default_factory=lambda: [])
+ """Global ranks to profile."""
+
+ record_memory_history: bool = False
+ """Record memory history in last rank."""
+
+ memory_snapshot_path: str = "snapshot.pickle"
+ """Specifies where to dump the memory history pickle."""
+
+ record_shapes: bool = False
+ """Record shapes of tensors."""
+
+ nvtx_ranges: bool = False
+ """Enable NVTX range annotations for profiling. When enabled, inserts NVTX markers
+ to categorize execution in profiler output."""
+
+
+@dataclass(kw_only=True)
+class DistributedInitConfig:
+ """Configuration settings for distributed training initialization."""
+
+ distributed_backend: Literal["nccl", "gloo"] = "nccl"
+ """Which backend to use for distributed training."""
+
+ distributed_timeout_minutes: int = 10
+ """Timeout minutes for torch.distributed."""
+
+ align_grad_reduce: bool = True
+ """If not set, all PP stages will launch gradient reduces simultaneously.
+ Otherwise, each PP stage will independently launch as needed.
+ """
+
+ local_rank: int = field(default_factory=lambda: int(os.getenv("LOCAL_RANK", "0")))
+ """local rank passed from distributed launcher."""
+
+ lazy_mpu_init: bool = False
+ """If set to True, initialize_megatron() skips DDP initialization and returns function to complete it instead.
+ Also turns on --use-cpu-initialization flag. This is for external DDP manager."""
+
+ use_megatron_fsdp: bool = False
+ """Use Megatron's Fully Sharded Data Parallel. Cannot be used together with use_torch_fsdp2."""
+
+ use_torch_fsdp2: bool = False
+ """Use the torch FSDP2 implementation. FSDP2 is not currently working with Pipeline Parallel.
+ It is still not in a stable release stage, and may therefore contain bugs or other
+ potential issues."""
+
+ nccl_communicator_config_path: str | None = None
+ """Path to the yaml file with NCCL communicator configurations. The number of min/max thread
+ groups and thread group cluster size of each communicator can be configured by setting
+ `min_ctas`, `max_ctas`, and `cga_cluster_size`."""
+
+ use_tp_pp_dp_mapping: bool = False
+ """If set, distributed ranks initialize order is changed from tp-cp-ep-dp-pp to tp-cp-ep-pp-dp.
+ """
+
+ use_gloo_process_groups: bool = field(default=True, metadata={"argparse_meta": {"arg_names": ["--disable-gloo-process-groups"]}})
+ """If enabled, create Gloo process groups for communications."""
+
+ use_sharp: bool = False
+ """Set the use of SHARP for the collective communications of data-parallel process groups.
+ When `True`, run barrier within each data-parallel process group,
+ which specifies the SHARP application target groups.
+ """
+
+ sharp_enabled_group: Literal["dp", "dp_replica"] | None = None
+ """IB SHARP can be enabled from only one communication group.
+ By default, it is enabled from dp group if not specified and use_sharp=True.
+ Available options: [dp, dp_replica]
+ """
+
+ high_priority_stream_groups: list[str] | None = field(default_factory=list)
+ """Specify which communicator groups should use high priority streams during creation.
+ Assigning high priority to communication streams ensures that communication kernels
+ are scheduled with higher priority, minimizing the exposed communication when it is
+ overlapped with other computation kernels.
+ """
+
+ distributed_timeout_seconds_after_init: int | None = None
+ """Timeout in seconds for process groups after initialization. This timeout is applied to all process groups after initialization and the first iteration completes."""
+
+ flight_recorder_dump_path: str | None = None
+ """Path for NCCL flight recorder trace dumps. Sets TORCH_FR_DUMP_TEMP_FILE and TORCH_NCCL_DEBUG_INFO_TEMP_FILE env variables before distributed init."""
+
+ flight_recorder_trace_buffer_size: int = 2000
+ """Size of the NCCL flight recorder trace buffer (TORCH_NCCL_TRACE_BUFFER_SIZE)."""
+
+ flight_recorder_dump_on_timeout: bool = True
+ """Dump flight recorder traces on NCCL timeout (TORCH_NCCL_DUMP_ON_TIMEOUT)."""
+
+ flight_recorder_include_stack_trace: bool = False
+ """Include stack traces in flight recorder dumps (TORCH_INCLUDE_STACK_TRACE)."""
+
+ flight_recorder_include_only_active: bool = True
+ """Include only active operations in flight recorder dumps (TORCH_INCLUDE_ONLY_ACTIVE)."""
+
+ flight_recorder_extra_dump_on_exec: bool = True
+ """Enable extra flight recorder dump on execution (TORCH_NCCL_EXTRA_DUMP_ON_EXEC)."""
+
+ disable_jit_fuser: bool = False
+ """Disable the JIT fuser."""
diff --git a/megatron/training/resilience_config.py b/megatron/training/config/resilience_config.py
similarity index 100%
rename from megatron/training/resilience_config.py
rename to megatron/training/config/resilience_config.py
diff --git a/megatron/training/training_config.py b/megatron/training/config/training_config.py
similarity index 90%
rename from megatron/training/training_config.py
rename to megatron/training/config/training_config.py
index 4985b91162c..d66343c82bd 100644
--- a/megatron/training/training_config.py
+++ b/megatron/training/config/training_config.py
@@ -1,7 +1,7 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
from dataclasses import dataclass, field
import signal
-from typing import Literal
+from typing import Literal, Optional
@dataclass(kw_only=True)
class TrainingConfig:
@@ -67,6 +67,9 @@ class TrainingConfig:
exit_signal_handler_for_dataloader: bool = False
"""Use signal handler for dataloader workers"""
+ exit_signal_handler_for_training: bool = False
+ """Shutdown the training when SIGINT or SIGTERM received to avoid unclear traceback"""
+
manual_gc: bool = False
"""Disable the threshold-based default garbage collector and trigger the garbage collection
manually. Manual garbage collection helps to align the timing of the collection across ranks
@@ -84,7 +87,7 @@ class TrainingConfig:
"""
iterations_to_skip: list[int] = field(default_factory=list)
- """List of iterations to skip during training, empty by default."""
+ """List of 1-indexed iterations to skip during training, empty by default."""
@dataclass(kw_only=True)
@@ -436,7 +439,7 @@ class CheckpointConfig:
The legacy format was deprecated on Feb 13, 2024.
"""
- ckpt_fully_parallel_save: bool = True
+ fully_parallel_save: bool = field(default=True, metadata={"argparse_meta": {"arg_names": ["--no-ckpt-fully-parallel-save"], "dest": "ckpt_fully_parallel_save"}})
"""Disable applying full save parallelization across DP for distributed checkpoints.
Depending on ckpt format might decrease the number of files in the checkpoint.
Makes DistributedOptimizer checkpoint non-reshardable."""
@@ -444,14 +447,44 @@ class CheckpointConfig:
async_save: bool = False
"""Apply async checkpointing save. Currently works only with `torch_dist` distributed checkpoint format."""
+ async_strategy: Literal["nvrx", "mcore"] = "nvrx"
+ """Which async save strategy to use. Available strategies: nvrx, mcore."""
+
use_persistent_ckpt_worker: bool = False
"""Use a persistent background worker for async checkpoint saves. When enabled, creates a dedicated
worker thread/process for handling async saves. When disabled, uses temporal workers that are
created and destroyed for each save operation."""
- ckpt_fully_parallel_load: bool = False
+ async_ckpt_cpu_priority: int = 10
+ """CPU nice value target (0-19, higher = lower priority) for the async checkpoint writer process.
+ If it exceeds 19, it will be set to 19. If the current nice value is greater than the target, it will be left unchanged.
+ Only applies when using persistent ckpt worker."""
+
+ async_ckpt_io_priority: Optional[int] = 3
+ """I/O scheduling class (0-3, 3=idle) for the async checkpoint writer process."""
+
+ fully_parallel_load: bool = field(default=False, metadata={"argparse_meta": {"arg_names": ["--ckpt-fully-parallel-load"], "dest": "ckpt_fully_parallel_load"}})
"""Apply full load parallelization across DP for distributed checkpoints."""
+ ckpt_fully_parallel_load_exchange_algo: Literal["broadcast", "gather_rounds", "gather_object"] = "broadcast"
+ """Algorithm for fully parallel load of distributed checkpoints.
+ "broadcast"(default): Broadcast the checkpoint from rank 0 to all other ranks.
+ "gather_rounds": Gather the checkpoint from all ranks in rounds.
+ "gather_object": Gather the checkpoint from all ranks in a single operation.
+ """
+
+ ckpt_fully_parallel_save_process_group: Literal["dp", "ep_dp"] = "dp"
+ """Process group for fully parallel save of distributed checkpoints.
+ "dp"(default): Data parallel process group.
+ "ep_dp": Expert data parallel process group.
+ """
+
+ ckpt_fully_parallel_load_process_group: Literal["dp", "ep_dp"] = "dp"
+ """Process group for fully parallel load of distributed checkpoints.
+ "dp"(default): Data parallel process group.
+ "ep_dp": Expert data parallel process group.
+ """
+
ckpt_assume_constant_structure: bool = False
"""Assume the checkpoint structure is constant across saves to enable optimizations."""
@@ -499,3 +532,7 @@ class CheckpointConfig:
replication_factor: int = 2
"""Number of machines storing the replica of a given rank's data."""
+
+ def __post_init__(self):
+ assert self.async_strategy in ["nvrx", "mcore"], \
+ f"async_strategy {self.async_strategy} is not supported. Available strategies: nvrx, mcore."
diff --git a/megatron/training/datasets/data_samplers.py b/megatron/training/datasets/data_samplers.py
index ca4cc1b36a3..0e48b3f0865 100644
--- a/megatron/training/datasets/data_samplers.py
+++ b/megatron/training/datasets/data_samplers.py
@@ -73,10 +73,26 @@ def build_pretraining_data_loader(dataset, consumed_samples):
raise Exception('{} dataloader type is not supported.'.format(args.dataloader_type))
def worker_init_fn(_):
- DistributedSignalHandler(args.exit_signal).__enter__()
+ import os
+
+ # Defensively close GPU device FDs in worker processes so workers do not
+ # keep references into NVIDIA memory space. This helps ensure GPU memory
+ # can be reclaimed even if a dataloader worker is delayed or fails to exit.
+ def close_nvidia_fds():
+ for fd in os.listdir("/proc/self/fd"):
+ try:
+ path = os.readlink(f"/proc/self/fd/{fd}")
+ if path.startswith("/dev/nvidia"):
+ os.close(int(fd))
+ except OSError:
+ pass
+
+ close_nvidia_fds()
+ if args.exit_signal_handler:
+ DistributedSignalHandler(args.exit_signal).__enter__()
maybe_worker_init_fn = (
- worker_init_fn if args.exit_signal_handler and args.num_workers > 0 else None
+ worker_init_fn if args.num_workers > 0 else None
)
# Torch dataloader.
if args.hybrid_context_parallel:
diff --git a/megatron/training/datasets/fim_dataset.py b/megatron/training/datasets/fim_dataset.py
index 730b7e033a1..875f979c91b 100644
--- a/megatron/training/datasets/fim_dataset.py
+++ b/megatron/training/datasets/fim_dataset.py
@@ -249,7 +249,10 @@ def _permute(
"""
if self.np_rng.binomial(1, fim_rate): # sample bernoulli dist
- contents = tokenizer._tokenizer.ids_to_text(sample)
+ # Use remove_special_tokens=True so character-level boundaries and re-tokenization
+ # are consistent; otherwise ids_to_text(..., None) keeps special tokens when
+ # include_special_tokens=True, changing contents and breaking e.g. split_sample.
+ contents = tokenizer._tokenizer.ids_to_text(sample, remove_special_tokens=True)
# Do not apply FIM if the sample starts with no_fim_prefix
if no_fim_prefix is not None and contents.startswith(no_fim_prefix):
diff --git a/megatron/training/datasets/sft_dataset.py b/megatron/training/datasets/sft_dataset.py
index b313dafb0ec..9de5d2a52fe 100644
--- a/megatron/training/datasets/sft_dataset.py
+++ b/megatron/training/datasets/sft_dataset.py
@@ -117,10 +117,6 @@ def extend_with_padding(tokens, targets, positions, pad_len):
tokens_list = tokens.tolist()
targets_list = targets.tolist()
- # Add EOD, unless it's already present
- if tokens_list[-1] != eod:
- tokens_list.append(eod)
- targets_list.append(eod)
pack_tokens.extend(tokens_list)
pack_targets.extend(targets_list)
@@ -146,8 +142,8 @@ def extend_with_padding(tokens, targets, positions, pad_len):
max_body = pack_length
pack_tokens = pack_tokens[:max_body]
pack_targets = pack_targets[:max_body]
- pack_tokens.extend(pad)
- pack_targets.extend(pad)
+ pack_tokens.append(pad)
+ pack_targets.append(pad)
pack_positions = pack_positions[:pack_length+1]
# Note len({pack_tokens, pack_targets, pack_positions}) should be pack_length + 1
cu_seqlens[-1] = len(pack_tokens) - 1
diff --git a/megatron/training/global_vars.py b/megatron/training/global_vars.py
index 5e3d1b9af24..d30e550c752 100644
--- a/megatron/training/global_vars.py
+++ b/megatron/training/global_vars.py
@@ -3,16 +3,19 @@
"""Megatron global variables."""
import os
+import signal
import sys
import torch
+from datetime import timedelta
+
from megatron.core import Timers
from megatron.core.config import set_experimental_flag
from megatron.core.energy_monitor import EnergyMonitor
from megatron.core.jit import disable_jit_fuser
from megatron.core.num_microbatches_calculator import init_num_microbatches_calculator, unset_num_microbatches_calculator
+from megatron.core.tokenizers.utils.build_tokenizer import build_tokenizer
from megatron.training.dist_signal_handler import DistributedSignalHandler
-from megatron.training.tokenizer import build_tokenizer
_GLOBAL_ARGS = None
_GLOBAL_NUM_MICROBATCHES_CALCULATOR = None
@@ -87,6 +90,34 @@ def _set_signal_handler(exit_signal):
_GLOBAL_SIGNAL_HANDLER = DistributedSignalHandler(exit_signal).__enter__()
+def _graceful_shutdown(signum, frame):
+ """
+ Signal handler for user-initiated termination (SIGINT / SIGTERM).
+
+ This handler attempts a best-effort graceful shutdown:
+ - Logs a single termination message from rank 0
+ - Synchronizes all ranks (barrier)
+ - Destroys the distributed process group
+ - Exits the process cleanly
+ """
+ from megatron.training.utils import print_rank_0
+ print_rank_0("\nTermination requested. Performing orderly shutdown.")
+
+ try:
+ if torch.distributed.is_available() and torch.distributed.is_initialized():
+ # synchronize all ranks before exiting
+ try:
+ # avoid deadlock if ranks don't all reach here
+ torch.distributed.barrier(timeout=timedelta(seconds=5))
+ except Exception:
+ pass
+
+ torch.distributed.destroy_process_group()
+ except Exception:
+ pass
+
+ sys.exit(0)
+
def set_global_variables(args, build_tokenizer=True):
"""Set args, tokenizer, tensorboard-writer, adlr-autoresume, and timers."""
@@ -119,6 +150,10 @@ def set_global_variables(args, build_tokenizer=True):
if args.exit_signal_handler:
_set_signal_handler(args.exit_signal)
+ if args.exit_signal_handler_for_training:
+ signal.signal(signal.SIGINT, _graceful_shutdown)
+ signal.signal(signal.SIGTERM, _graceful_shutdown)
+
if args.disable_jit_fuser:
disable_jit_fuser()
diff --git a/megatron/training/initialize.py b/megatron/training/initialize.py
index c150ac3d5ca..22acd6b6405 100644
--- a/megatron/training/initialize.py
+++ b/megatron/training/initialize.py
@@ -24,7 +24,6 @@
)
from megatron.core.transformer.custom_layers.batch_invariant_kernels import enable_batch_invariant_mode
from megatron.core.utils import get_te_version, is_te_min_version, is_torch_min_version
-from megatron.legacy import fused_kernels
from megatron.training import get_adlr_autoresume, get_args, get_tensorboard_writer
from megatron.training.utils import print_rank_0, warn_rank_0
from megatron.training import inprocess_restart
@@ -83,9 +82,6 @@ def initialize_megatron(
load_args_from_checkpoint(args, load_arg='pretrained_checkpoint')
load_args_from_checkpoint(args)
- if args.async_save and args.use_persistent_ckpt_worker:
- init_persistent_async_worker()
-
if args.yaml_cfg is not None:
args = validate_yaml(args, args_defaults)
else:
@@ -98,6 +94,9 @@ def initialize_megatron(
# set logging level
setup_logging()
+ if args.async_save and args.use_persistent_ckpt_worker:
+ init_persistent_async_worker(args.rank, 'forkserver')
+
# init rerun state
def state_save_func():
return {'rng_tracker_states': tensor_parallel.get_cuda_rng_tracker().get_states()}
@@ -196,48 +195,7 @@ def _compile_dependencies():
flush=True,
)
- # ==================
- # Load fused kernels
- # ==================
-
- # Custom kernel constraints check.
- seq_len = args.seq_length
- attn_batch_size = (
- args.num_attention_heads / args.tensor_model_parallel_size
- ) * args.micro_batch_size
- # Constraints on sequence length and attn_batch_size to enable warp based
- # optimization and upper triangular optimization (for causal mask)
- custom_kernel_constraint = (
- seq_len > 16 and seq_len <= 16384 and seq_len % 4 == 0 and attn_batch_size % 4 == 0
- )
- # Print a warning.
- if not ((args.fp16 or args.bf16) and custom_kernel_constraint and args.masked_softmax_fusion):
- warn_rank_0(
- "Constraints for invoking optimized fused softmax kernel are not met. "
- "We default back to unfused kernel invocations."
- )
-
- # Always build on rank zero first.
- if torch.distributed.get_rank() == 0:
- start_time = time.time()
- print("> compiling and loading fused kernels ...", flush=True)
- fused_kernels.load(args)
- torch.distributed.barrier()
- else:
- torch.distributed.barrier()
- fused_kernels.load(args)
- # Simple barrier to make sure all ranks have passed the
- # compilation phase successfully before moving on to the
- # rest of the program. We think this might ensure that
- # the lock is released.
torch.distributed.barrier()
- if torch.distributed.get_rank() == 0:
- print(
- ">>> done with compiling and loading fused kernels. "
- "Compilation time: {:.3f} seconds".format(time.time() - start_time),
- flush=True,
- )
-
def _initialize_tp_communicators():
"""initializing the communicators with user buffers for high-performance tensor-model-parallel
@@ -337,6 +295,49 @@ def _initialize_distributed(get_embedding_ranks, get_position_embedding_ranks, s
if args.cuda_graph_impl == "transformer_engine":
torch.cuda.set_stream(torch.cuda.Stream())
+ # Set flight recorder env vars if specified.
+ # Priority: pre-existing environment variable > MLM argument.
+ # All vars follow the same setdefault semantics: if already set in the
+ # environment we warn and keep the user's value; otherwise we apply the
+ # value derived from the MLM argument / flag.
+ # The block is also triggered when either path env var is already set
+ # so that the remaining defaults are applied consistently.
+ _fr_path = (
+ args.flight_recorder_dump_path
+ or os.environ.get('TORCH_FR_DUMP_TEMP_FILE')
+ or os.environ.get('TORCH_NCCL_DEBUG_INFO_TEMP_FILE')
+ )
+ if _fr_path is not None:
+ _fr_dump_prefix = _fr_path
+ if os.path.isdir(_fr_path):
+ _fr_dump_prefix = os.path.join(_fr_path, '_dump_')
+ warn_rank_0(
+ "Flight recorder: using directory "
+ f"'{_fr_path}' for dump path, appending per-rank prefix "
+ f"'{_fr_dump_prefix}'."
+ )
+ _fr_env_defaults = {
+ 'TORCH_FR_DUMP_TEMP_FILE': _fr_dump_prefix,
+ 'TORCH_NCCL_DEBUG_INFO_TEMP_FILE': _fr_dump_prefix,
+ 'TORCH_NCCL_TRACE_BUFFER_SIZE': str(args.flight_recorder_trace_buffer_size),
+ 'TORCH_NCCL_DUMP_ON_TIMEOUT': str(int(args.flight_recorder_dump_on_timeout)),
+ 'TORCH_INCLUDE_STACK_TRACE': str(int(args.flight_recorder_include_stack_trace)),
+ 'TORCH_INCLUDE_ONLY_ACTIVE': str(int(args.flight_recorder_include_only_active)),
+ 'TORCH_NCCL_EXTRA_DUMP_ON_EXEC': str(int(args.flight_recorder_extra_dump_on_exec)),
+ }
+ for _var, _default in _fr_env_defaults.items():
+ if _var in os.environ:
+ warn_rank_0(
+ f"Flight recorder: environment variable {_var} is already set to "
+ f"'{os.environ[_var]}'; ignoring config value '{_default}'."
+ )
+ else:
+ os.environ[_var] = _default
+ print_rank_0(
+ "Flight recorder env vars:\n"
+ + "\n".join(f" {k}={os.environ[k]}" for k in _fr_env_defaults)
+ )
+
# Call the init process
init_process_group_kwargs = {
'backend': args.distributed_backend,
@@ -378,7 +379,7 @@ def _initialize_distributed(get_embedding_ranks, get_position_embedding_ranks, s
order='tp-cp-ep-dp-pp' if not args.use_tp_pp_dp_mapping else 'tp-cp-ep-pp-dp',
get_embedding_ranks=get_embedding_ranks,
get_position_embedding_ranks=get_position_embedding_ranks,
- create_gloo_process_groups=args.enable_gloo_process_groups,
+ create_gloo_process_groups=args.use_gloo_process_groups,
high_priority_stream_groups=args.high_priority_stream_groups,
sharp_enabled_group=args.sharp_enabled_group,
create_all_gather_group=args.create_all_gather_group,
diff --git a/megatron/training/inprocess_restart.py b/megatron/training/inprocess_restart.py
index 04d05038b20..ac377de4ac0 100644
--- a/megatron/training/inprocess_restart.py
+++ b/megatron/training/inprocess_restart.py
@@ -80,16 +80,18 @@ def inprocess_restart(train, args):
)
class AbortCheckpoint(inprocess.abort.Abort):
+ def __init__(self, async_strategy):
+ self.async_strategy = async_strategy
def __call__(
self, state: inprocess.state.FrozenState
) -> inprocess.state.FrozenState:
- reset_persistent_async_worker()
+ reset_persistent_async_worker(self.async_strategy)
return state
abort = inprocess.Compose(
inprocess.abort.AbortTransformerEngine(),
inprocess.abort.AbortTorchDistributed(),
- AbortCheckpoint(),
+ AbortCheckpoint(args.async_strategy),
inprocess.nested_restarter.NestedRestarterHandlingStarting(),
)
completion = inprocess.nested_restarter.NestedRestarterFinalized()
diff --git a/megatron/training/theoretical_memory_usage.py b/megatron/training/theoretical_memory_usage.py
index 7d4043b62d7..bdc5b9f5774 100644
--- a/megatron/training/theoretical_memory_usage.py
+++ b/megatron/training/theoretical_memory_usage.py
@@ -4,7 +4,7 @@
import math
-from .utils import print_rank_0
+from .utils import is_hybrid_model, print_rank_0
NUM_BYTES_IN_MEGABYTE = 1024 * 1024
@@ -116,6 +116,19 @@ def compute_weight_and_optimizer_memory(args, verbose=False):
)
+ self_attn_term
)
+ num_active_parameters_in_transformer_layer_moe = (
+ 2
+ * args.hidden_size
+ * (
+ # MoE MLP.
+ + (moe_ffn_hidden_size * args.moe_router_topk * gated_linear_multiplier)
+ # Shared MoE MLP.
+ + (shared_expert_ffn_hidden_size * gated_linear_multiplier)
+ # Transformer layernorms.
+ + (2)
+ )
+ + self_attn_term
+ )
embedding_size = args.hidden_size * args.padded_vocab_size
final_layernorm = norm_size * args.hidden_size
if args.untie_embeddings_and_output_weights:
@@ -127,6 +140,11 @@ def compute_weight_and_optimizer_memory(args, verbose=False):
+ num_parameters_in_transformer_layer_moe * num_moe_layers
+ final_layernorm
)
+ num_active_parameters_in_transformer_block = (
+ num_parameters_in_transformer_layer_dense * num_dense_layers
+ + num_active_parameters_in_transformer_layer_moe * num_moe_layers
+ + final_layernorm
+ )
num_parameters_in_mtp_block = (
num_parameters_in_transformer_layer_dense * mtp_num_dense_layers
+ num_parameters_in_transformer_layer_moe * mtp_num_moe_layers
@@ -136,11 +154,20 @@ def compute_weight_and_optimizer_memory(args, verbose=False):
+ num_parameters_in_mtp_block
+ num_parameters_in_embedding_layers
)
+ num_active_parameters = (
+ num_active_parameters_in_transformer_block
+ + num_parameters_in_mtp_block
+ + num_parameters_in_embedding_layers
+ )
if verbose:
print(
f"Number of parameters in transformer block in billions: "
f"{num_parameters_in_transformer_block / 10**9: .2f}"
)
+ print(
+ f"Number of active parameters in transformer block in billions: "
+ f"{num_active_parameters_in_transformer_block / 10**9: .2f}"
+ )
if args.mtp_num_layers is not None:
print(
f"Number of parameters in mtp block in billions: "
@@ -151,6 +178,7 @@ def compute_weight_and_optimizer_memory(args, verbose=False):
f"{num_parameters_in_embedding_layers / 10**9:.2f}"
)
print(f"Total number of parameters in billions: {num_total_parameters / 10**9:.2f}")
+ print(f"Total number of active parameters in billions: {num_active_parameters / 10**9:.2f}")
# Most loaded model shard has (1/pp_size transformer layers + 1 mtp block + 1 embedding layer) / tp_size.
num_parameters_on_most_loaded_model_shard = (
@@ -338,7 +366,7 @@ def compute_activation_memory_without_sp(args, num_microbatches, verbose=False):
def report_theoretical_memory(args, num_microbatches=None, verbose=False):
- if args.is_hybrid_model:
+ if is_hybrid_model(args):
print("Theoretical memory footprints not yet supported for hybrid Mamba-Transformer models.")
return
diff --git a/megatron/training/tokenizer/bert_tokenization.py b/megatron/training/tokenizer/bert_tokenization.py
deleted file mode 100644
index 642041e778e..00000000000
--- a/megatron/training/tokenizer/bert_tokenization.py
+++ /dev/null
@@ -1,431 +0,0 @@
-# coding=utf-8
-# Copyright 2018 The Google AI Language Team Authors.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Tokenization classes."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import collections
-import re
-import unicodedata
-import six
-
-
-def validate_case_matches_checkpoint(do_lower_case, init_checkpoint):
- """Checks whether the casing config is consistent with the checkpoint name."""
-
- # The casing has to be passed in by the user and there is no explicit check
- # as to whether it matches the checkpoint. The casing information probably
- # should have been stored in the bert_config.json file, but it's not, so
- # we have to heuristically detect it to validate.
-
- if not init_checkpoint:
- return
-
- m = re.match("^.*?([A-Za-z0-9_-]+)/bert_model.ckpt", init_checkpoint)
- if m is None:
- return
-
- model_name = m.group(1)
-
- lower_models = [
- "uncased_L-24_H-1024_A-16", "uncased_L-12_H-768_A-12",
- "multilingual_L-12_H-768_A-12", "chinese_L-12_H-768_A-12"
- ]
-
- cased_models = [
- "cased_L-12_H-768_A-12", "cased_L-24_H-1024_A-16",
- "multi_cased_L-12_H-768_A-12"
- ]
-
- is_bad_config = False
- if model_name in lower_models and not do_lower_case:
- is_bad_config = True
- actual_flag = "False"
- case_name = "lowercased"
- opposite_flag = "True"
-
- if model_name in cased_models and do_lower_case:
- is_bad_config = True
- actual_flag = "True"
- case_name = "cased"
- opposite_flag = "False"
-
- if is_bad_config:
- raise ValueError(
- "You passed in `--do_lower_case=%s` with `--init_checkpoint=%s`. "
- "However, `%s` seems to be a %s model, so you "
- "should pass in `--do_lower_case=%s` so that the fine-tuning matches "
- "how the model was pre-training. If this error is wrong, please "
- "just comment out this check." % (actual_flag, init_checkpoint,
- model_name, case_name, opposite_flag))
-
-
-def convert_to_unicode(text):
- """Converts `text` to Unicode (if it's not already), assuming utf-8 input."""
- if six.PY3:
- if isinstance(text, str):
- return text
- elif isinstance(text, bytes):
- return text.decode("utf-8", "ignore")
- else:
- raise ValueError("Unsupported string type: %s" % (type(text)))
- elif six.PY2:
- if isinstance(text, str):
- return text.decode("utf-8", "ignore")
- elif isinstance(text, unicode):
- return text
- else:
- raise ValueError("Unsupported string type: %s" % (type(text)))
- else:
- raise ValueError("Not running on Python2 or Python 3?")
-
-
-def printable_text(text):
- """Returns text encoded in a way suitable for print or `tf.logging`."""
-
- # These functions want `str` for both Python2 and Python3, but in one case
- # it's a Unicode string and in the other it's a byte string.
- if six.PY3:
- if isinstance(text, str):
- return text
- elif isinstance(text, bytes):
- return text.decode("utf-8", "ignore")
- else:
- raise ValueError("Unsupported string type: %s" % (type(text)))
- elif six.PY2:
- if isinstance(text, str):
- return text
- elif isinstance(text, unicode):
- return text.encode("utf-8")
- else:
- raise ValueError("Unsupported string type: %s" % (type(text)))
- else:
- raise ValueError("Not running on Python2 or Python 3?")
-
-
-def load_vocab(vocab_file):
- """Loads a vocabulary file into a dictionary."""
- vocab = collections.OrderedDict()
- index = 0
- with open(vocab_file, "r", encoding = "utf-8") as reader:
- while True:
- token = convert_to_unicode(reader.readline())
- if not token:
- break
- token = token.strip()
- vocab[token] = index
- index += 1
- return vocab
-
-
-def convert_by_vocab(vocab, items):
- """Converts a sequence of [tokens|ids] using the vocab."""
- output = []
- for item in items:
- output.append(vocab[item])
- return output
-
-
-def convert_tokens_to_ids(vocab, tokens):
- return convert_by_vocab(vocab, tokens)
-
-
-def convert_ids_to_tokens(inv_vocab, ids):
- return convert_by_vocab(inv_vocab, ids)
-
-
-def whitespace_tokenize(text):
- """Runs basic whitespace cleaning and splitting on a piece of text."""
- text = text.strip()
- if not text:
- return []
- tokens = text.split()
- return tokens
-
-
-class FullTokenizer(object):
- """Runs end-to-end tokenziation."""
-
- def __init__(self, vocab_file, do_lower_case=True):
- self.vocab = load_vocab(vocab_file)
- self.inv_vocab = {v: k for k, v in self.vocab.items()}
- self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case)
- self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)
-
- def tokenize(self, text):
- split_tokens = []
- for token in self.basic_tokenizer.tokenize(text):
- for sub_token in self.wordpiece_tokenizer.tokenize(token):
- split_tokens.append(sub_token)
-
- return split_tokens
-
- def convert_tokens_to_ids(self, tokens):
- return convert_by_vocab(self.vocab, tokens)
-
- def convert_ids_to_tokens(self, ids):
- return convert_by_vocab(self.inv_vocab, ids)
-
- @staticmethod
- def convert_tokens_to_string(tokens, clean_up_tokenization_spaces=True):
- """ Converts a sequence of tokens (string) in a single string. """
-
- def clean_up_tokenization(out_string):
- """ Clean up a list of simple English tokenization artifacts
- like spaces before punctuations and abreviated forms.
- """
- out_string = (
- out_string.replace(" .", ".")
- .replace(" ?", "?")
- .replace(" !", "!")
- .replace(" ,", ",")
- .replace(" ' ", "'")
- .replace(" n't", "n't")
- .replace(" 'm", "'m")
- .replace(" 's", "'s")
- .replace(" 've", "'ve")
- .replace(" 're", "'re")
- )
- return out_string
-
- text = ' '.join(tokens).replace(' ##', '').strip()
- if clean_up_tokenization_spaces:
- clean_text = clean_up_tokenization(text)
- return clean_text
- else:
- return text
-
- def vocab_size(self):
- return len(self.vocab)
-
-
-class BasicTokenizer(object):
- """Runs basic tokenization (punctuation splitting, lower casing, etc.)."""
-
- def __init__(self, do_lower_case=True):
- """Constructs a BasicTokenizer.
-
- Args:
- do_lower_case: Whether to lower case the input.
- """
- self.do_lower_case = do_lower_case
-
- def tokenize(self, text):
- """Tokenizes a piece of text."""
- text = convert_to_unicode(text)
- text = self._clean_text(text)
-
- # This was added on November 1st, 2018 for the multilingual and Chinese
- # models. This is also applied to the English models now, but it doesn't
- # matter since the English models were not trained on any Chinese data
- # and generally don't have any Chinese data in them (there are Chinese
- # characters in the vocabulary because Wikipedia does have some Chinese
- # words in the English Wikipedia.).
- text = self._tokenize_chinese_chars(text)
-
- orig_tokens = whitespace_tokenize(text)
- split_tokens = []
- for token in orig_tokens:
- if self.do_lower_case:
- token = token.lower()
- token = self._run_strip_accents(token)
- split_tokens.extend(self._run_split_on_punc(token))
-
- output_tokens = whitespace_tokenize(" ".join(split_tokens))
- return output_tokens
-
- def _run_strip_accents(self, text):
- """Strips accents from a piece of text."""
- text = unicodedata.normalize("NFD", text)
- output = []
- for char in text:
- cat = unicodedata.category(char)
- if cat == "Mn":
- continue
- output.append(char)
- return "".join(output)
-
- def _run_split_on_punc(self, text):
- """Splits punctuation on a piece of text."""
- chars = list(text)
- i = 0
- start_new_word = True
- output = []
- while i < len(chars):
- char = chars[i]
- if _is_punctuation(char):
- output.append([char])
- start_new_word = True
- else:
- if start_new_word:
- output.append([])
- start_new_word = False
- output[-1].append(char)
- i += 1
-
- return ["".join(x) for x in output]
-
- def _tokenize_chinese_chars(self, text):
- """Adds whitespace around any CJK character."""
- output = []
- for char in text:
- cp = ord(char)
- if self._is_chinese_char(cp):
- output.append(" ")
- output.append(char)
- output.append(" ")
- else:
- output.append(char)
- return "".join(output)
-
- def _is_chinese_char(self, cp):
- """Checks whether CP is the codepoint of a CJK character."""
- # This defines a "chinese character" as anything in the CJK Unicode block:
- # https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)
- #
- # Note that the CJK Unicode block is NOT all Japanese and Korean characters,
- # despite its name. The modern Korean Hangul alphabet is a different block,
- # as is Japanese Hiragana and Katakana. Those alphabets are used to write
- # space-separated words, so they are not treated specially and handled
- # like the all of the other languages.
- if ((cp >= 0x4E00 and cp <= 0x9FFF) or #
- (cp >= 0x3400 and cp <= 0x4DBF) or #
- (cp >= 0x20000 and cp <= 0x2A6DF) or #
- (cp >= 0x2A700 and cp <= 0x2B73F) or #
- (cp >= 0x2B740 and cp <= 0x2B81F) or #
- (cp >= 0x2B820 and cp <= 0x2CEAF) or
- (cp >= 0xF900 and cp <= 0xFAFF) or #
- (cp >= 0x2F800 and cp <= 0x2FA1F)): #
- return True
-
- return False
-
- def _clean_text(self, text):
- """Performs invalid character removal and whitespace cleanup on text."""
- output = []
- for char in text:
- cp = ord(char)
- if cp == 0 or cp == 0xfffd or _is_control(char):
- continue
- if _is_whitespace(char):
- output.append(" ")
- else:
- output.append(char)
- return "".join(output)
-
-
-class WordpieceTokenizer(object):
- """Runs WordPiece tokenziation."""
-
- def __init__(self, vocab, unk_token="[UNK]", max_input_chars_per_word=200):
- self.vocab = vocab
- self.unk_token = unk_token
- self.max_input_chars_per_word = max_input_chars_per_word
-
- def tokenize(self, text):
- """Tokenizes a piece of text into its word pieces.
-
- This uses a greedy longest-match-first algorithm to perform tokenization
- using the given vocabulary.
-
- For example:
- input = "unaffable"
- output = ["un", "##aff", "##able"]
-
- Args:
- text: A single token or whitespace separated tokens. This should have
- already been passed through `BasicTokenizer.
-
- Returns:
- A list of wordpiece tokens.
- """
-
- text = convert_to_unicode(text)
-
- output_tokens = []
- for token in whitespace_tokenize(text):
- chars = list(token)
- if len(chars) > self.max_input_chars_per_word:
- output_tokens.append(self.unk_token)
- continue
-
- is_bad = False
- start = 0
- sub_tokens = []
- while start < len(chars):
- end = len(chars)
- cur_substr = None
- while start < end:
- substr = "".join(chars[start:end])
- if start > 0:
- substr = "##" + substr
- if substr in self.vocab:
- cur_substr = substr
- break
- end -= 1
- if cur_substr is None:
- is_bad = True
- break
- sub_tokens.append(cur_substr)
- start = end
-
- if is_bad:
- output_tokens.append(self.unk_token)
- else:
- output_tokens.extend(sub_tokens)
- return output_tokens
-
-
-def _is_whitespace(char):
- """Checks whether `chars` is a whitespace character."""
- # \t, \n, and \r are technically contorl characters but we treat them
- # as whitespace since they are generally considered as such.
- if char == " " or char == "\t" or char == "\n" or char == "\r":
- return True
- cat = unicodedata.category(char)
- if cat == "Zs":
- return True
- return False
-
-
-def _is_control(char):
- """Checks whether `chars` is a control character."""
- # These are technically control characters but we count them as whitespace
- # characters.
- if char == "\t" or char == "\n" or char == "\r":
- return False
- cat = unicodedata.category(char)
- if cat in ("Cc", "Cf"):
- return True
- return False
-
-
-def _is_punctuation(char):
- """Checks whether `chars` is a punctuation character."""
- cp = ord(char)
- # We treat all non-letter/number ASCII as punctuation.
- # Characters such as "^", "$", and "`" are not in the Unicode
- # Punctuation class but we treat them as punctuation anyways, for
- # consistency.
- if ((cp >= 33 and cp <= 47) or (cp >= 58 and cp <= 64) or
- (cp >= 91 and cp <= 96) or (cp >= 123 and cp <= 126)):
- return True
- cat = unicodedata.category(char)
- if cat.startswith("P"):
- return True
- return False
diff --git a/megatron/training/tokenizer/gpt2_tokenization.py b/megatron/training/tokenizer/gpt2_tokenization.py
deleted file mode 100644
index 55b95b8ed9a..00000000000
--- a/megatron/training/tokenizer/gpt2_tokenization.py
+++ /dev/null
@@ -1,324 +0,0 @@
-# coding=utf-8
-# Copyright 2018 The Open AI Team Authors and The HuggingFace Inc. team.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Tokenization classes for OpenAI GPT."""
-
-from __future__ import (absolute_import, division, print_function,
- unicode_literals)
-
-import sys
-import json
-import logging
-import os
-import regex as re
-from io import open
-
-try:
- from functools import lru_cache
-except ImportError:
- # Just a dummy decorator to get the checks to run on python2
- # because honestly I don't want to support a byte-level unicode BPE
- # tokenizer on python 2 right now.
- def lru_cache():
- return lambda func: func
-
-
-logger = logging.getLogger(__name__)
-
-PRETRAINED_VOCAB_ARCHIVE_MAP = {
- 'gpt2': "https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json",
-}
-PRETRAINED_MERGES_ARCHIVE_MAP = {
- 'gpt2': "https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt",
-}
-PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP = {
- 'gpt2': 1024,
-}
-VOCAB_NAME = 'vocab.json'
-MERGES_NAME = 'merges.txt'
-SPECIAL_TOKENS_NAME = 'special_tokens.txt'
-
-
-@lru_cache()
-def bytes_to_unicode():
- """
- Returns list of utf-8 byte and a corresponding list of unicode strings.
- The reversible bpe codes work on unicode strings.
- This means you need a large # of unicode characters in your vocab if you want to avoid UNKs.
- When you're at something like a 10B token dataset you end up needing around 5K for decent coverage.
- This is a signficant percentage of your normal, say, 32K bpe vocab.
- To avoid that, we want lookup tables between utf-8 bytes and unicode strings.
- And avoids mapping to whitespace/control characters the bpe code barfs on.
- """
- _chr = unichr if sys.version_info[0] == 2 else chr
- bs = list(range(ord("!"), ord("~") + 1)) + list(range(ord("¡"), ord("¬") + 1)) + \
- list(range(ord("®"), ord("ÿ") + 1))
- cs = bs[:]
- n = 0
- for b in range(2**8):
- if b not in bs:
- bs.append(b)
- cs.append(2**8 + n)
- n += 1
- cs = [_chr(n) for n in cs]
- return dict(zip(bs, cs))
-
-
-def get_pairs(word):
- """Return set of symbol pairs in a word.
-
- Word is represented as tuple of symbols (symbols being variable-length strings).
- """
- pairs = set()
- prev_char = word[0]
- for char in word[1:]:
- pairs.add((prev_char, char))
- prev_char = char
- return pairs
-
-
-class GPT2Tokenizer(object):
- """
- GPT-2 BPE tokenizer. Peculiarities:
- - Byte-level BPE
- """
- @classmethod
- def from_pretrained(cls, pretrained_model_name_or_path, cache_dir=None, *inputs, **kwargs):
- """
- Instantiate a PreTrainedBertModel from a pre-trained model file.
- Download and cache the pre-trained model file if needed.
- """
- if pretrained_model_name_or_path in PRETRAINED_VOCAB_ARCHIVE_MAP:
- vocab_file = PRETRAINED_VOCAB_ARCHIVE_MAP[pretrained_model_name_or_path]
- merges_file = PRETRAINED_MERGES_ARCHIVE_MAP[pretrained_model_name_or_path]
- special_tokens_file = None
- else:
- vocab_file = os.path.join(pretrained_model_name_or_path, VOCAB_NAME)
- merges_file = os.path.join(pretrained_model_name_or_path, MERGES_NAME)
- special_tokens_file = os.path.join(pretrained_model_name_or_path, SPECIAL_TOKENS_NAME)
- if not os.path.exists(special_tokens_file):
- special_tokens_file = None
- else:
- logger.info("loading special tokens file {}".format(special_tokens_file))
- # redirect to the cache, if necessary
- try:
- from .file_utils import cached_path
- resolved_vocab_file = cached_path(vocab_file, cache_dir=cache_dir)
- resolved_merges_file = cached_path(merges_file, cache_dir=cache_dir)
- except EnvironmentError:
- logger.error(
- "Model name '{}' was not found in model name list ({}). "
- "We assumed '{}' was a path or url but couldn't find files {} and {} "
- "at this path or url.".format(
- pretrained_model_name_or_path,
- ', '.join(PRETRAINED_VOCAB_ARCHIVE_MAP.keys()),
- pretrained_model_name_or_path,
- vocab_file, merges_file))
- return None
- if resolved_vocab_file == vocab_file and resolved_merges_file == merges_file:
- logger.info("loading vocabulary file {}".format(vocab_file))
- logger.info("loading merges file {}".format(merges_file))
- else:
- logger.info("loading vocabulary file {} from cache at {}".format(
- vocab_file, resolved_vocab_file))
- logger.info("loading merges file {} from cache at {}".format(
- merges_file, resolved_merges_file))
- if pretrained_model_name_or_path in PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP:
- # if we're using a pretrained model, ensure the tokenizer wont index sequences longer
- # than the number of positional embeddings
- max_len = PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP[pretrained_model_name_or_path]
- kwargs['max_len'] = min(kwargs.get('max_len', int(1e12)), max_len)
- # Instantiate tokenizer.
- if special_tokens_file and 'special_tokens' not in kwargs:
- with open(special_tokens_file, encoding='utf-8') as f:
- special_tokens = f.read().split('\n')[:-1]
- else:
- special_tokens = kwargs.pop('special_tokens', [])
- tokenizer = cls(
- resolved_vocab_file,
- resolved_merges_file,
- special_tokens=special_tokens,
- *inputs,
- **kwargs)
- return tokenizer
-
- def __init__(self, vocab_file, merges_file, errors='replace',
- special_tokens=None, max_len=None):
- self.max_len = max_len if max_len is not None else int(1e12)
- with open(vocab_file) as f:
- self.encoder = json.load(f)
- self.decoder = {v: k for k, v in self.encoder.items()}
- self.errors = errors # how to handle errors in decoding
- self.byte_encoder = bytes_to_unicode()
- self.byte_decoder = {v: k for k, v in self.byte_encoder.items()}
- with open(merges_file, encoding='utf-8') as f:
- bpe_data = f.read().split('\n')[1:-1]
- bpe_merges = [tuple(merge.split()) for merge in bpe_data]
- self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges))))
- self.cache = {}
-
- # Should haved added re.IGNORECASE so BPE merges can happen for
- # capitalized versions of contractions
- self.pat = re.compile(
- r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""")
-
- self.special_tokens = {}
- self.special_tokens_decoder = {}
- self.set_special_tokens(special_tokens)
-
- def __len__(self):
- return len(self.encoder) + len(self.special_tokens)
-
- def set_special_tokens(self, special_tokens):
- """ Add a list of additional tokens to the encoder.
- The additional tokens are indexed starting from the last index of the
- current vocabulary in the order of the `special_tokens` list.
- """
- if not special_tokens:
- self.special_tokens = {}
- self.special_tokens_decoder = {}
- return
- self.special_tokens = dict((tok, len(self.encoder) + i)
- for i, tok in enumerate(special_tokens))
- self.special_tokens_decoder = {v: k for k, v in self.special_tokens.items()}
- logger.info("Special tokens {}".format(self.special_tokens))
-
- def bpe(self, token):
- if token in self.cache:
- return self.cache[token]
- word = tuple(token)
- pairs = get_pairs(word)
-
- if not pairs:
- return token
-
- while True:
- bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float('inf')))
- if bigram not in self.bpe_ranks:
- break
- first, second = bigram
- new_word = []
- i = 0
- while i < len(word):
- try:
- j = word.index(first, i)
- new_word.extend(word[i:j])
- i = j
- except Exception:
- new_word.extend(word[i:])
- break
-
- if word[i] == first and i < len(word) - 1 and word[i + 1] == second:
- new_word.append(first + second)
- i += 2
- else:
- new_word.append(word[i])
- i += 1
- new_word = tuple(new_word)
- word = new_word
- if len(word) == 1:
- break
- else:
- pairs = get_pairs(word)
- word = ' '.join(word)
- self.cache[token] = word
- return word
-
- def tokenize(self, text):
- """ Tokenize a string. """
- bpe_tokens = []
- for token in re.findall(self.pat, text):
- if sys.version_info[0] == 2:
- token = ''.join(self.byte_encoder[ord(b)] for b in token)
- else:
- token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8'))
- bpe_tokens.extend(bpe_token for bpe_token in self.bpe(token).split(' '))
- return bpe_tokens
-
- def convert_tokens_to_ids(self, tokens):
- """ Converts a sequence of tokens into ids using the vocab. """
- ids = []
- if isinstance(tokens, str) or (sys.version_info[0] == 2 and isinstance(tokens, unicode)):
- if tokens in self.special_tokens:
- return self.special_tokens[tokens]
- else:
- return self.encoder.get(tokens, 0)
- for token in tokens:
- if token in self.special_tokens:
- ids.append(self.special_tokens[token])
- else:
- ids.append(self.encoder.get(token, 0))
- if len(ids) > self.max_len:
- logger.warning(
- "Token indices sequence length is longer than the specified maximum "
- " sequence length for this OpenAI GPT model ({} > {}). Running this"
- " sequence through the model will result in indexing errors".format(
- len(ids), self.max_len)
- )
- return ids
-
- def convert_ids_to_tokens(self, ids, skip_special_tokens=False):
- """Converts a sequence of ids in BPE tokens using the vocab."""
- tokens = []
- for i in ids:
- if i in self.special_tokens_decoder:
- if not skip_special_tokens:
- tokens.append(self.special_tokens_decoder[i])
- else:
- tokens.append(self.decoder[i])
- return tokens
-
- def encode(self, text):
- return self.convert_tokens_to_ids(self.tokenize(text))
-
- def decode(self, tokens):
- text = ''.join([self.decoder[token] for token in tokens])
- text = bytearray([self.byte_decoder[c] for c in text]).decode('utf-8', errors=self.errors)
- return text
-
- def save_vocabulary(self, vocab_path):
- """Save the tokenizer vocabulary and merge files to a directory."""
- if not os.path.isdir(vocab_path):
- logger.error("Vocabulary path ({}) should be a directory".format(vocab_path))
- return
- vocab_file = os.path.join(vocab_path, VOCAB_NAME)
- merge_file = os.path.join(vocab_path, MERGES_NAME)
- special_tokens_file = os.path.join(vocab_path, SPECIAL_TOKENS_NAME)
-
- with open(vocab_file, 'w', encoding='utf-8') as f:
- f.write(json.dumps(self.encoder, ensure_ascii=False))
-
- index = 0
- with open(merge_file, "w", encoding="utf-8") as writer:
- writer.write(u'#version: 0.2\n')
- for bpe_tokens, token_index in sorted(self.bpe_ranks.items(), key=lambda kv: kv[1]):
- if index != token_index:
- logger.warning("Saving vocabulary to {}: BPE merge indices are not consecutive."
- " Please check that the tokenizer is not corrupted!".format(merge_file))
- index = token_index
- writer.write(' '.join(bpe_tokens) + u'\n')
- index += 1
-
- index = len(self.encoder)
- with open(special_tokens_file, 'w', encoding='utf-8') as writer:
- for token, token_index in sorted(self.special_tokens.items(), key=lambda kv: kv[1]):
- if index != token_index:
- logger.warning("Saving special tokens vocabulary to {}: BPE indices are not consecutive."
- " Please check that the tokenizer is not corrupted!".format(special_tokens_file))
- index = token_index
- writer.write(token + u'\n')
- index += 1
-
- return vocab_file, merge_file, special_tokens_file
diff --git a/megatron/training/tokenizer/tokenizer.py b/megatron/training/tokenizer/tokenizer.py
deleted file mode 100644
index ffca2f92409..00000000000
--- a/megatron/training/tokenizer/tokenizer.py
+++ /dev/null
@@ -1,1054 +0,0 @@
-# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
-# Copyright (c) 2023 Alibaba PAI Team.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Megatron tokenizers."""
-
-import base64
-import json
-import math
-import types
-from abc import ABC, abstractmethod
-from pathlib import Path
-from typing import Dict, List, Optional
-
-from megatron.core.datasets.megatron_tokenizer import MegatronLegacyTokenizer
-
-from .bert_tokenization import FullTokenizer as FullBertTokenizer
-from .gpt2_tokenization import GPT2Tokenizer
-from megatron.training.tokenizer.multimodal_tokenizer import MultimodalTokenizer
-from megatron.training.tokenizer.sft_tokenizer import SFTTokenizer
-
-def build_tokenizer(args, **kwargs):
- """Initialize tokenizer."""
- from megatron.training.utils import print_rank_0
- print_rank_0('> building {} tokenizer ...'.format(args.tokenizer_type))
-
- # Select and instantiate the tokenizer.
- if args.tokenizer_type == 'BertWordPieceLowerCase':
- assert args.vocab_file is not None
- tokenizer = _BertWordPieceTokenizer(
- vocab_file=args.vocab_file, lower_case=True, vocab_extra_ids=args.vocab_extra_ids
- )
- elif args.tokenizer_type == 'BertWordPieceCase':
- assert args.vocab_file is not None
- tokenizer = _BertWordPieceTokenizer(
- vocab_file=args.vocab_file, lower_case=False, vocab_extra_ids=args.vocab_extra_ids
- )
- elif args.tokenizer_type == 'GPT2BPETokenizer':
- assert args.vocab_file is not None
- assert args.merge_file is not None
- tokenizer = _GPT2BPETokenizer(args.vocab_file, args.merge_file)
- elif args.tokenizer_type == 'SentencePieceTokenizer':
- assert args.tokenizer_model is not None
- tokenizer = _SentencePieceTokenizer(
- args.tokenizer_model, vocab_extra_ids=args.vocab_extra_ids
- )
- elif args.tokenizer_type == 'GPTSentencePieceTokenizer':
- assert args.tokenizer_model is not None
- tokenizer = _GPTSentencePieceTokenizer(args.tokenizer_model)
- elif args.tokenizer_type == 'HuggingFaceTokenizer':
- tokenizer = _HuggingFaceTokenizer(
- args.tokenizer_model, trust_remote_code = args.trust_remote_code, **kwargs,
- )
- elif args.tokenizer_type == 'Llama2Tokenizer':
- assert args.tokenizer_model is not None
- tokenizer = _Llama2Tokenizer(args.tokenizer_model)
- elif args.tokenizer_type == 'TikTokenizer':
- assert args.tokenizer_model is not None
- assert args.tiktoken_pattern is not None
- assert args.tiktoken_pattern in {"v1", "v2"}
- pattern = PATTERN_TIKTOKEN if args.tiktoken_pattern == "v1" else PATTERN_TIKTOKEN_V2
- tokenizer = CustomTikTokenizer(
- path=args.tokenizer_model,
- pattern=pattern,
- vocab_size=args.vocab_size,
- num_special_tokens=args.tiktoken_num_special_tokens,
- special_tokens=args.tokenizer_special_tokens,
- )
- elif args.tokenizer_type == 'NullTokenizer':
- assert args.vocab_size is not None
- tokenizer = _NullTokenizer(args.vocab_size)
- elif args.tokenizer_type == "MultimodalTokenizer":
- try:
- import transformers
- except ImportError:
- raise ImportError(
- "MultimodalTokenizer currently requires transformers library to be installed"
- )
-
- kwargs = dict()
- if args.tokenizer_prompt_format == "nvlm-yi-34b":
- kwargs = {
- "from_slow": True,
- "legacy": False,
- "add_bos_token": True,
- }
-
- # Currently, only HuggingFace tokenizers are supported.
- underlying_tokenizer = transformers.AutoTokenizer.from_pretrained(
- pretrained_model_name_or_path=args.tokenizer_model, **kwargs
- )
-
- tokenizer = MultimodalTokenizer(
- underlying_tokenizer,
- args.tokenizer_prompt_format,
- args.special_tokens,
- args.image_tag_type,
- args.force_system_message,
- )
- elif args.tokenizer_type == "SFTTokenizer":
- tokenizer = SFTTokenizer(
- args.tokenizer_model,
- args.sft_tokenizer_prompt_format,
- )
- elif args.tokenizer_type == 'NullMultimodalTokenizer':
- assert args.vocab_size is not None
- tokenizer = _NullMultimodalTokenizer(args.vocab_size)
-
- elif args.tokenizer_type == 'DeepSeekV2Tokenizer':
- tokenizer = _DeepSeekV2Tokenizer(args.tokenizer_model,args.extra_vocab_size)
- args.padded_vocab_size = tokenizer.vocab_size
-
- elif args.tokenizer_type == 'DeepSeekV3Tokenizer':
- tokenizer = _DeepSeekV3Tokenizer(args.tokenizer_model,args.extra_vocab_size)
- args.padded_vocab_size = tokenizer.vocab_size
-
- else:
- raise NotImplementedError('{} tokenizer is not ' 'implemented.'.format(args.tokenizer_type))
-
- # Add vocab size (if not already set from a checkpoint).
- if getattr(args, "padded_vocab_size", None) is None:
- args.padded_vocab_size = _vocab_size_with_padding(tokenizer.vocab_size, args)
-
- return tokenizer
-
-
-def _vocab_size_with_padding(orig_vocab_size, args, logging_enabled=True):
- """Pad vocab size so it is divisible by model parallel size and
- still having GPU friendly size."""
-
- after = orig_vocab_size
- multiple = args.make_vocab_size_divisible_by * args.tensor_model_parallel_size
- after = int(math.ceil(after / multiple) * multiple)
- if args.rank == 0 and logging_enabled:
- print(
- ' > padded vocab (size: {}) with {} dummy tokens '
- '(new size: {})'.format(orig_vocab_size, after - orig_vocab_size, after),
- flush=True,
- )
- return after
-
-
-class _HuggingFaceTokenizer(MegatronLegacyTokenizer):
- def __init__(self, pretrained_model_name_or_path, trust_remote_code=False, **kwargs):
- super().__init__(pretrained_model_name_or_path, **kwargs)
- try:
- import transformers
- except ImportError:
- raise EnvironmentError(
- f"The transformers library must be installed to use huggingface_tokenizer_provider"
- )
-
- # TODO(bnorick): download tokenizer once to lustre and use force offline to make sure all tasks read it from there
- self._tokenizer = transformers.AutoTokenizer.from_pretrained(
- pretrained_model_name_or_path=pretrained_model_name_or_path,
- trust_remote_code=trust_remote_code,
- **kwargs
- )
- self._vocab = self._tokenizer.get_vocab()
- self._inv_vocab = {token_id: token for token, token_id in self._vocab.items()}
-
- @property
- def vocab_size(self):
- return len(self._tokenizer)
-
- @property
- def vocab(self):
- """Dictionary from vocab text token to id token."""
- return self._vocab
-
- @property
- def inv_vocab(self):
- """Dictionary from vocab id token to text token."""
- return self._inv_vocab
-
- @property
- def decoder(self):
- return self._inv_vocab
-
- def tokenize(self, text, **kwargs):
- return self._tokenizer(text, **kwargs).input_ids
-
- def detokenize(self, token_ids, **kwargs):
- return self._tokenizer.decode(token_ids, **kwargs)
-
- def offsets(self, ids: list[int], text: str) -> list[int]:
- retok_ids: "transformers.BatchEncoding" = self._tokenizer(text)
- offsets, next_start_idx = [], 0
- for i in range(len(ids)):
- span = retok_ids.token_to_chars(i)
- if span is not None:
- offsets.append(span.start)
- next_start_idx = span.end
- else:
- offsets.append(next_start_idx)
- return offsets
-
- @property
- def eod(self):
- return self._tokenizer.eos_token_id
-
- @property
- def bos(self):
- return self._tokenizer.bos_token_id
-
- @property
- def pad(self):
- return self._tokenizer.pad_token_id
-
-
-class _BertWordPieceTokenizer(MegatronLegacyTokenizer):
- """Original BERT wordpiece tokenizer."""
-
- def __init__(self, vocab_file, lower_case=True, vocab_extra_ids=0):
- super().__init__(vocab_file, lower_case=lower_case, vocab_extra_ids=vocab_extra_ids)
- self.tokenizer = FullBertTokenizer(vocab_file, do_lower_case=lower_case)
- self.cls_id = self.tokenizer.vocab['[CLS]']
- self.sep_id = self.tokenizer.vocab['[SEP]']
- self.pad_id = self.tokenizer.vocab['[PAD]']
- self.mask_id = self.tokenizer.vocab['[MASK]']
- self._additional_special_tokens = []
-
- # (dsachan) Add BOS and EOS tokens
- SPECIAL_TOKENS = {'eos_token': '[EOS]', 'bos_token': '[BOS]'}
- self._bos_token = '[BOS]'
- self.add_token(self._bos_token)
- self._bos_token_id = self.vocab.get(self._bos_token)
-
- self._eos_token = '[EOS]'
- self.add_token(self._eos_token)
- self._eos_token_id = self.vocab.get(self._eos_token)
-
- # (dsachan) Add additional special tokens
- # These can be used as sentinel tokens in T5 model inputs
- additional_special_tokens = []
- additional_special_tokens.extend(
- ["".format(i) for i in range(vocab_extra_ids)]
- )
- self.add_additional_special_tokens(additional_special_tokens)
-
- def add_token(self, token):
- if token not in self.vocab:
- self.inv_vocab[self.vocab_size] = token
- # self.vocab_size comes from len(vocab)
- # and it will increase as we add elements
- self.vocab[token] = self.vocab_size
-
- def add_additional_special_tokens(self, tokens_list):
- setattr(self, "additional_special_tokens", tokens_list)
- for value in tokens_list:
- self.add_token(value)
-
- @property
- def vocab_size(self):
- return self.tokenizer.vocab_size()
-
- @property
- def vocab(self):
- return self.tokenizer.vocab
-
- @property
- def inv_vocab(self):
- return self.tokenizer.inv_vocab
-
- def tokenize(self, text):
- text_tokens = self.tokenizer.tokenize(text)
- return self.tokenizer.convert_tokens_to_ids(text_tokens)
-
- def decode(self, ids):
- tokens = self.tokenizer.convert_ids_to_tokens(ids)
- return self.tokenizer.convert_tokens_to_string(tokens)
-
- def detokenize(self, token_ids):
- """Copy of decode() method for inference pipeline compatibility"""
- return self.decode(token_ids)
-
- def decode_token_ids(self, token_ids):
- tokens = self.tokenizer.convert_ids_to_tokens(token_ids)
- exclude_list = ['[PAD]', '[CLS]']
- non_pads = [t for t in tokens if t not in exclude_list]
-
- result = ""
- for s in non_pads:
- if s.startswith("##"):
- result += s[2:]
- else:
- result += " " + s
-
- return result
-
- @property
- def cls(self):
- return self.cls_id
-
- @property
- def sep(self):
- return self.sep_id
-
- @property
- def pad(self):
- return self.pad_id
-
- @property
- def mask(self):
- return self.mask_id
-
- @property
- def bos(self):
- """Id of the beginning of sentence token in the vocabulary."""
- return self._bos_token_id
-
- @property
- def eos(self):
- """Id of the end of sentence token in the vocabulary."""
- return self._eos_token_id
-
- @property
- def eod(self):
- """Copy of eod property for inference pipeline compatibility"""
- return self.eos
-
- @property
- def bos_token(self):
- """Beginning of sentence token id"""
- return self._bos_token
-
- @property
- def eos_token(self):
- """End of sentence token id"""
- return self._eos_token
-
- @property
- def additional_special_tokens(self):
- """All the additional special tokens you may want to use (list of strings)."""
- return self._additional_special_tokens
-
- @property
- def additional_special_tokens_ids(self):
- """Ids of all the additional special tokens in the vocabulary (list of integers)."""
- return [self.vocab.get(token) for token in self._additional_special_tokens]
-
- @additional_special_tokens.setter
- def additional_special_tokens(self, value):
- self._additional_special_tokens = value
-
-
-class _GPT2BPETokenizer(MegatronLegacyTokenizer):
- """Original GPT2 BPE tokenizer."""
-
- def __init__(self, vocab_file, merge_file):
- super().__init__(vocab_file, merge_file)
-
- self.tokenizer = GPT2Tokenizer(
- vocab_file, merge_file, errors='replace', special_tokens=[], max_len=None
- )
- self.eod_id = self.tokenizer.encoder['<|endoftext|>']
-
- @property
- def vocab_size(self):
- return len(self.tokenizer.encoder)
-
- @property
- def vocab(self):
- return self.tokenizer.encoder
-
- @property
- def inv_vocab(self):
- return self.tokenizer.decoder
-
- def tokenize(self, text):
- return self.tokenizer.encode(text)
-
- def detokenize(self, token_ids):
- return self.tokenizer.decode(token_ids)
-
- @property
- def eod(self):
- return self.eod_id
-
-
-class _SentencePieceTokenizer(MegatronLegacyTokenizer):
- """SentencePieceTokenizer-Megatron wrapper"""
-
- def __init__(self, model_file, vocab_extra_ids=0):
- super().__init__(model_file, vocab_extra_ids=vocab_extra_ids)
-
- import sentencepiece
-
- self.tokenizer = sentencepiece.SentencePieceProcessor(model_file=model_file)
- self._initalize(vocab_extra_ids)
-
- def _populate_vocab(self):
- self._vocab = {}
- self._inv_vocab = {}
-
- for i in range(len(self.tokenizer)):
- t = self.tokenizer.id_to_piece(i)
- self._inv_vocab[i] = t
- self._vocab[t] = i
-
- def _initalize(self, vocab_extra_ids):
- self._populate_vocab()
- self._special_tokens = {}
- self._inv_special_tokens = {}
-
- self._t5_tokens = []
-
- def _add_special_token(t):
- if t not in self._vocab:
- next_id = len(self._vocab)
- self._vocab[t] = next_id
- self._inv_vocab[next_id] = t
- self._special_tokens[t] = self._vocab[t]
- self._inv_special_tokens[self._vocab[t]] = t
-
- _add_special_token('')
- self._cls_id = self._vocab['']
- _add_special_token('')
- self._sep_id = self._vocab['']
- _add_special_token('')
- self._eod_id = self._vocab['']
- _add_special_token('')
- self._mask_id = self._vocab['']
-
- pad_id = self.tokenizer.pad_id()
- try:
- pad_token = self.tokenizer.id_to_piece(pad_id)
- except IndexError:
- pad_token = ''
- _add_special_token(pad_token)
- self._pad_id = self._vocab[pad_token]
-
- bos_id = self.tokenizer.bos_id()
- try:
- bos_token = self.tokenizer.id_to_piece(bos_id)
- except IndexError:
- bos_token = ''
- _add_special_token(bos_token)
- self._bos_id = self._vocab[bos_token]
-
- eos_id = self.tokenizer.eos_id()
- try:
- eos_token = self.tokenizer.id_to_piece(eos_id)
- except IndexError:
- eos_token = ''
- _add_special_token(eos_token)
- self._eos_id = self._vocab[eos_token]
-
- for i in range(vocab_extra_ids):
- t = "".format(i)
- _add_special_token(t)
- self._t5_tokens += [t]
-
- @property
- def vocab_size(self):
- return len(self._vocab)
-
- @property
- def vocab(self):
- return self._vocab
-
- @property
- def inv_vocab(self):
- return self._inv_vocab
-
- @property
- def decoder(self):
- return self._inv_vocab
-
- @property
- def encoder(self):
- return self._vocab
-
- # From:
- # https://github.com/NVIDIA/NeMo/blob/c8fa217e811d60d11d014827c7f3845ff6c99ae7/nemo/collections/common/tokenizers/sentencepiece_tokenizer.py#L89
- def tokenize(self, text):
- ids = []
- idx = 0
-
- while 1:
- indices = {}
- for token in self._special_tokens:
- try:
- indices[token] = text[idx:].index(token)
- except ValueError:
- continue
- if len(indices) == 0:
- break
-
- next_token = min(indices, key=indices.get)
- next_idx = idx + indices[next_token]
-
- ids.extend(self.tokenizer.encode_as_ids(text[idx:next_idx]))
- ids.append(self._special_tokens[next_token])
- idx = next_idx + len(next_token)
-
- ids.extend(self.tokenizer.encode_as_ids(text[idx:]))
- return ids
-
- # From:
- # https://github.com/NVIDIA/NeMo/blob/c8fa217e811d60d11d014827c7f3845ff6c99ae7/nemo/collections/common/tokenizers/sentencepiece_tokenizer.py#L125
- def detokenize(self, ids):
- text = ""
- last_i = 0
-
- for i, id in enumerate(ids):
- if id in self._inv_special_tokens:
- text += self.tokenizer.decode_ids(ids[last_i:i]) + " "
- text += self._inv_special_tokens[id] + " "
- last_i = i + 1
-
- text += self.tokenizer.decode_ids(ids[last_i:])
- return text
-
- def offsets(self, ids: list[int], text: str) -> list[int]:
- return [p.begin for p in self.tokenizer.decode_ids_as_immutable_proto(ids).pieces]
-
- @property
- def cls(self):
- return self._cls_id
-
- @property
- def sep(self):
- return self._sep_id
-
- @property
- def pad(self):
- return self._pad_id
-
- @property
- def bos(self):
- return self._bos_id
-
- @property
- def eod(self):
- return self._eod_id
-
- @property
- def eos(self):
- return self._eos_id
-
- @property
- def mask(self):
- return self._mask_id
-
- @property
- def additional_special_tokens_ids(self):
- return [self.vocab[k] for k in self._t5_tokens]
-
-
-class _GPTSentencePieceTokenizer(_SentencePieceTokenizer):
- """SentencePieceTokenizer-Megatron wrapper"""
-
- def __init__(self, model_file):
- super().__init__(model_file, vocab_extra_ids=0)
-
- def _initalize(self, vocab_extra_ids):
- self._populate_vocab()
-
- self._pad_id = self.tokenizer.pad_id()
- self._bos_id = self.tokenizer.bos_id()
- self._eos_id = self.tokenizer.eos_id()
-
- def tokenize(self, text):
- return self.tokenizer.encode_as_ids(text)
-
- def detokenize(self, ids):
- return self.tokenizer.decode_ids(ids)
-
- @property
- def cls(self):
- return -1
-
- @property
- def sep(self):
- return -1
-
- @property
- def mask(self):
- return -1
-
- @property
- def eod(self):
- return self._eos_id
-
- @property
- def additional_special_tokens_ids(self):
- return None
-
-
-class _Llama2Tokenizer(_SentencePieceTokenizer):
- """SentencePieceTokenizer-Megatron wrapper"""
-
- def __init__(self, model_file):
- super().__init__(model_file, vocab_extra_ids=0)
-
- def _initalize(self, vocab_extra_ids):
- self._populate_vocab()
-
- # BOS / EOS token IDs
- self.n_words: int = self.tokenizer.vocab_size()
- self.bos_id: int = self.tokenizer.bos_id()
- self.eos_id: int = self.tokenizer.eos_id()
- self.pad_id: int = self.tokenizer.pad_id()
- assert self.tokenizer.vocab_size() == self.tokenizer.get_piece_size()
-
- def tokenize(self, s: str, bos=True, eos=False):
- '''Default args for text completion, not chat/dialog.'''
- assert type(s) is str
- t = self.tokenizer.encode(s)
- if bos:
- t = [self.bos_id] + t
- if eos:
- t = t + [self.eos_id]
- return t
-
- def detokenize(self, ids):
- return self.tokenizer.decode_ids(ids)
-
- @property
- def cls(self):
- return -1
-
- @property
- def sep(self):
- return -1
-
- @property
- def mask(self):
- return -1
-
- @property
- def eod(self):
- return self.eos_id
-
- @property
- def additional_special_tokens_ids(self):
- return None
-
-
-def reload_mergeable_ranks(path: str, max_vocab: Optional[int] = None) -> Dict[bytes, int]:
- """
- Reload our tokenizer JSON file and convert it to Tiktoken format.
- """
- from ..utils import print_rank_0 # To prevent circular import.
-
- assert path.endswith(".json")
-
- # reload vocab
- with open(path, "r") as f:
- vocab = json.load(f)
- assert isinstance(vocab, list)
- print_rank_0(f"Vocab size: {len(vocab)}")
- if max_vocab is not None:
- vocab = vocab[:max_vocab]
- print_rank_0(f"Cutting vocab to first {len(vocab)} tokens.")
-
- # build ranks
- ranks: Dict[bytes, int] = {}
- for i, x in enumerate(vocab):
- assert x.keys() == {"rank", "token_bytes", "token_str"}
- assert x["rank"] == i
- merge = base64.b64decode(x["token_bytes"])
- assert i >= 256 or merge == bytes([i])
- ranks[merge] = x["rank"]
-
- # sanity check
- assert len(ranks) == len(vocab)
- assert set(ranks.values()) == set(range(len(ranks)))
-
- return ranks
-
-
-PATTERN_TIKTOKEN = (
- r"[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]++[\r\n]*|\s*[\r\n]|\s+(?!\S)|\s+"
-)
-PATTERN_TIKTOKEN_V2 = "[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]*[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]+|[^\\r\\n\\p{L}\\p{N}]?[\\p{Lu}\\p{Lt}\\p{Lm}\\p{Lo}\\p{M}]+[\\p{Ll}\\p{Lm}\\p{Lo}\\p{M}]*|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n/]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
-
-
-class CustomTikTokenizer(MegatronLegacyTokenizer):
- def __init__(
- self,
- path: str,
- pattern: str,
- vocab_size: Optional[int],
- num_special_tokens: int,
- special_tokens: Optional[List[str]],
- ):
- super().__init__(
- path,
- pattern=pattern,
- vocab_size=vocab_size,
- num_special_tokens=num_special_tokens,
- special_tokens=special_tokens,
- )
- import tiktoken
-
- from .. import print_rank_0 # To prevent circular import.
-
- if vocab_size is None:
- vocab_size = 2**17 # Fallback vocab size is 131072.
- self._vocab_size = vocab_size
-
- SPECIAL_TOKENS = ["", "", " "]
- if special_tokens is None:
- special_tokens = SPECIAL_TOKENS.copy()
- assert len(special_tokens) == len(
- set(special_tokens)
- ), f"Special tokens should be unique: {special_tokens}"
- assert len(special_tokens) <= num_special_tokens < self._vocab_size
- assert set(SPECIAL_TOKENS) <= set(
- special_tokens
- ), f"Custom special tokens should include {SPECIAL_TOKENS}"
-
- special_filler = [
- "".format(id=i) for i in range(len(special_tokens), num_special_tokens)
- ]
- if special_filler:
- print_rank_0(f"Adding special tokens {special_filler[0]}, ..., {special_filler[-1]}")
- special_tokens = special_tokens + special_filler
- assert len(set(special_tokens)) == len(special_tokens) == num_special_tokens, special_tokens
- inner_vocab_size = self._vocab_size - num_special_tokens
-
- token_to_id_without_special_tokens = reload_mergeable_ranks(
- path, max_vocab=inner_vocab_size
- )
- # Create space for special tokens.
- token_to_id_without_special_tokens = {
- t: i + num_special_tokens for t, i in token_to_id_without_special_tokens.items()
- }
-
- special_tokens = {t: i for i, t in enumerate(special_tokens)}
- self._unk_id = special_tokens[""]
- self._bos_id = special_tokens[""]
- self._eos_id = special_tokens[" "]
-
- # Create tiktoken model.
- self._model = tiktoken.Encoding(
- name=Path(path).parent.name,
- pat_str=pattern,
- mergeable_ranks=token_to_id_without_special_tokens,
- special_tokens=special_tokens,
- )
-
- # Create final _id_to_token and _token_to_id data structures with special tokens inserted
- # into appropriate locations.
- assert set(token_to_id_without_special_tokens.keys()).isdisjoint(set(special_tokens.keys()))
- self._token_to_id = token_to_id_without_special_tokens.copy()
- self._token_to_id.update(special_tokens)
- self._id_to_token = {v: k for k, v in self._token_to_id.items()}
- assert set(range(self._vocab_size)) == set(self._id_to_token.keys())
-
- @property
- def bos(self) -> int:
- return self._bos_id
-
- @property
- def eos(self) -> int:
- return self._eos_id
-
- @property
- def pad(self) -> int:
- return self._pad_id
-
- @property
- def unk(self) -> int:
- return self._unk_id
-
- @property
- def eod(self) -> int:
- return self._eos_id
-
- @property
- def vocab(self):
- return self._token_to_id
-
- @property
- def inv_vocab(self):
- return self._id_to_token
-
- def tokenize(self, s: str, bos: bool = False, eos: bool = False) -> List[int]:
- tokens = self._model.encode_ordinary(s)
- if bos:
- tokens = [self.bos, *tokens]
- if eos:
- tokens = [*tokens, self.eos]
-
- return tokens
-
- def detokenize(self, tokens: List[int]) -> str:
- return self._model.decode(tokens)
-
- def offsets(self, ids: list[int], text: str) -> list[int]:
- try:
- return self._model.decode_with_offsets(ids)[1]
- except UnicodeDecodeError:
- # Tiktoken has an unnecessary check that raises UnicodeDecodeError
- # from `text = b"".join(token_bytes).decode("utf-8", errors="strict")`
- # which is not needed for our use case. So we re-implement it, without
- # the check.
-
- token_bytes = self._model.decode_tokens_bytes(ids)
- text_len = 0
- offsets = []
- for token in token_bytes:
- offsets.append(max(0, text_len - (0x80 <= token[0] < 0xC0)))
- text_len += sum(1 for c in token if not 0x80 <= c < 0xC0)
- return offsets
-
- @property
- def vocab_size(self) -> int:
- return self._vocab_size
-
- @property
- def encoder(self):
- return self._token_to_id
-
- @property
- def decoder(self):
- return self._id_to_token
-
-
-class _NullTokenizer(MegatronLegacyTokenizer):
- def __init__(self, vocab_size):
- super().__init__(None, vocab_size=vocab_size)
- self._vocab_size_without_eod = int(vocab_size)
- self._eod_id = self._vocab_size_without_eod
-
- def tokenize(self, text):
- return [int(x) for x in text.split(' ')]
-
- def detokenize(self, ids):
- text = [str(x) for x in ids]
- return ' '.join(text)
-
- def offsets(self, ids: list[int], text: str) -> list[int]:
- offsets, start_idx = [], 0
- for id_ in ids:
- offsets.append(start_idx)
- start_idx += 1 + len(str(id_))
- return offsets
-
- @property
- def vocab_size(self):
- return self._vocab_size_without_eod + 1
-
- @property
- def vocab(self):
- raise NotImplementedError
-
- @property
- def inv_vocab(self):
- raise NotImplementedError
-
- @property
- def cls(self):
- return -1
-
- @property
- def sep(self):
- return -1
-
- @property
- def mask(self):
- return -1
-
- @property
- def eod(self):
- return self._eod_id
-
- @property
- def additional_special_tokens_ids(self):
- return None
-
-class _NullMultimodalTokenizer(MegatronLegacyTokenizer):
- def __init__(self, vocab_size, image_token=None, image_token_id=None):
- super().__init__(None, vocab_size=vocab_size)
- self._vocab_size_without_eod = int(vocab_size)
- self._eod_id = self._vocab_size_without_eod
-
- from megatron.core.models.multimodal.llava_model import DEFAULT_IMAGE_TOKEN_INDEX, IMAGE_TOKEN
- self._image_token = image_token if image_token is not None else IMAGE_TOKEN
- self._image_token_id = image_token_id if image_token_id is not None else DEFAULT_IMAGE_TOKEN_INDEX
-
- def tokenize(self, text):
- return [int(x) for x in text.split(' ')]
-
- def detokenize(self, ids):
- text = [str(x) for x in ids]
- return ' '.join(text)
-
- def offsets(self, ids: list[int], text: str) -> list[int]:
- offsets, start_idx = [], 0
- for id_ in ids:
- offsets.append(start_idx)
- start_idx += 1 + len(str(id_))
- return offsets
-
- def convert_tokens_to_ids(self, tokens):
- ids = [(int(t) if t != self._image_token else self._image_token_id) for t in tokens.split(' ')]
- return ids if len(ids) > 1 else ids[0]
-
- @property
- def vocab_size(self):
- return self._vocab_size_without_eod + 1
-
- @property
- def vocab(self):
- raise NotImplementedError
-
- @property
- def inv_vocab(self):
- raise NotImplementedError
-
- @property
- def cls(self):
- return -1
-
- @property
- def sep(self):
- return -1
-
- @property
- def mask(self):
- return -1
-
- @property
- def eod(self):
- return self._eod_id
-
- @property
- def additional_special_tokens_ids(self):
- return None
-
-
-class _DeepSeekV2Tokenizer(MegatronLegacyTokenizer):
- def __init__(self, tokenizer_path, extra_vocab_size=0):
- super().__init__(tokenizer_path, extra_vocab_size)
- try:
- import transformers
- except ImportError:
- raise EnvironmentError(f"The transformers library must be installed to use DeepSeekV2Tokenizer")
-
- self.tokenizer = transformers.AutoTokenizer.from_pretrained(
- tokenizer_path,
- trust_remote_code=True
- )
- self.extra_vocab_size = extra_vocab_size
-
- def __call__(self, text, return_tensors=None,
- padding=None, max_length=None, truncation=None, add_special_tokens=None):
-
- return self.tokenizer(text, return_tensors=return_tensors, padding=padding,
- max_length=max_length, truncation=truncation, add_special_tokens=add_special_tokens)
-
- @property
- def vocab_size(self):
- return self.tokenizer.vocab_size + self.extra_vocab_size
-
- @property
- def vocab(self):
- return self.tokenizer.encoder
-
- @property
- def inv_vocab(self):
- return self.tokenizer.decoder
-
- def tokenize(self, text):
- return self.tokenizer.encode(text)
-
- def detokenize(self, token_ids):
- return self.tokenizer.decode(token_ids)
-
- @property
- def eod(self):
- return self.tokenizer.eos_token_id
-
- @property
- def eos_token(self):
- return self.tokenizer.eos_token
-
- @property
- def pad_token_id(self):
- return self.tokenizer.pad_token_id
-
- @property
- def eos_token_id(self):
- return self.tokenizer.eos_token_id
-
-class _DeepSeekV3Tokenizer(MegatronLegacyTokenizer):
- def __init__(self, tokenizer_path, extra_vocab_size=0):
- super().__init__(tokenizer_path, extra_vocab_size)
- try:
- import transformers
- except ImportError:
- raise EnvironmentError(f"The transformers library must be installed to use DeepSeekV3Tokenizer")
-
- self.tokenizer = transformers.AutoTokenizer.from_pretrained(
- tokenizer_path,
- trust_remote_code=True
- )
- self.extra_vocab_size = extra_vocab_size
-
- def __call__(self, text, return_tensors=None,
- padding=None, max_length=None, truncation=None, add_special_tokens=None):
-
- return self.tokenizer(text, return_tensors=return_tensors, padding=padding,
- max_length=max_length, truncation=truncation, add_special_tokens=add_special_tokens)
-
- @property
- def vocab_size(self):
- return self.tokenizer.vocab_size + self.extra_vocab_size
-
- @property
- def vocab(self):
- return self.tokenizer.encoder
-
- @property
- def inv_vocab(self):
- return self.tokenizer.decoder
-
- def tokenize(self, text):
- return self.tokenizer.encode(text)
-
- def detokenize(self, token_ids):
- return self.tokenizer.decode(token_ids)
-
- @property
- def eod(self):
- return self.tokenizer.eos_token_id
-
- @property
- def eos_token(self):
- return self.tokenizer.eos_token
-
- @property
- def pad_token_id(self):
- return self.tokenizer.pad_token_id
-
- @property
- def eos_token_id(self):
- return self.tokenizer.eos_token_id
-
diff --git a/megatron/training/training.py b/megatron/training/training.py
index 162acbe5f60..2da84ffd91b 100644
--- a/megatron/training/training.py
+++ b/megatron/training/training.py
@@ -56,11 +56,13 @@ def set_startup_timestamps(program_start=None, main_entry=None):
import os
import sys
from contextlib import nullcontext
+from pathlib import Path
from typing import Any, Optional, Dict
import torch.distributed
from megatron.core.optimizer.distrib_optimizer import DistributedOptimizer
+from megatron.core.optimizer_param_scheduler import get_canonical_lr_for_logging
from .log_handler import CustomHandler
# Make default logging level INFO, but filter out all log messages not from MCore.
@@ -76,7 +78,56 @@ def set_startup_timestamps(program_start=None, main_entry=None):
has_rl_utils = True
except ImportError:
has_rl_utils = False
-from megatron.rl.parallel_utils import build_inference_pg_collection
+
+# Canonical list of RL timer names to include in timers_to_log.
+# When the profiling branch is merged, this will be imported from rl_profiling
+# as RL_LOGGABLE_TIMER_NAMES instead of being defined here.
+RL_LOGGABLE_TIMER_NAMES = [
+ # Top-level RL phases
+ 'rl/rollout-collection',
+ 'rl/prepare-data-for-update',
+ # Rollout collection breakdown
+ 'rl/inference-setup',
+ 'rl/collect-rollouts',
+ 'rl/sync-rollouts',
+ 'rl/suspend-engine',
+ # Optimizer offload/restore
+ 'rl/offload-optimizer-before-inference',
+ 'rl/restore-optimizer-after-inference',
+ 'rl/offload-kv-cache-after-inference',
+ 'rl/restore-kv-cache-before-inference',
+ # Fine-grained offload/restore breakdown
+ 'rl/restore/grad-buffers',
+ 'rl/restore/optimizer-state',
+ 'rl/restore/wait-for-transfers',
+ 'rl/offload/grad-buffers',
+ 'rl/offload/optimizer-state',
+ # Weight prefetching
+ 'rl/prefetch-weights-to-gpu',
+ 'rl/prefetch-weights-to-cpu',
+ # Data preparation
+ 'rl/compute-group-stats',
+ 'rl/prepare-advantages',
+ 'rl/prepare-trajectories',
+ 'rl/get-ltor-masks',
+ 'rl/create-dataloader',
+ 'rl/sequence-packing',
+ 'rl/align-inference-logprobs',
+ 'rl/log-wandb-tb',
+ 'rl/pack-sequences',
+ 'rl/regather-trajectories',
+ # Logprobs computation
+ 'rl/compute-logprobs',
+ 'rl/compute-old-logprobs',
+ 'rl/compute-ref-logprobs',
+ 'rl/get-logprobs',
+ 'rl/forward-pass',
+ 'rl/log-softmax',
+ # Inference / cuda graphs
+ 'rl/build-cuda-graphs',
+ 'rl/wait-for-decode-only',
+]
+
try:
from modelopt.torch.distill.plugins.megatron import (
get_tensor_shapes_adjust_fn_for_distillation,
@@ -112,7 +163,7 @@ def set_startup_timestamps(program_start=None, main_entry=None):
is_vp_first_stage,
is_vp_last_stage,
)
-from megatron.core.optimizer import get_standard_config_overrides
+from megatron.core.optimizer import get_mup_config_overrides, get_standard_config_overrides
from megatron.training.checkpointing import load_checkpoint
from megatron.training.checkpointing import save_checkpoint, save_grads
from megatron.training.checkpointing import checkpoint_exists
@@ -148,7 +199,7 @@ def set_startup_timestamps(program_start=None, main_entry=None):
from megatron.training.initialize import initialize_megatron
from megatron.training.initialize import write_args_to_tensorboard
from megatron.training.initialize import set_jit_fusion_options
-from megatron.training.utils import get_batch_on_this_cp_rank, get_batch_on_this_tp_rank
+from megatron.training.utils import get_batch_on_this_cp_rank, get_batch_on_this_tp_rank, is_hybrid_model
from megatron.training.datasets.data_samplers import build_pretraining_data_loader
from megatron.core.datasets.data_schedule import HybridCPDataLoaderWrapper
from megatron.core.optimizer_param_scheduler import OptimizerParamScheduler
@@ -159,13 +210,20 @@ def set_startup_timestamps(program_start=None, main_entry=None):
from megatron.core.transformer.multi_token_prediction import MTPLossLoggingHelper
from megatron.core.parallel_state import (
destroy_global_memory_buffer,
- destroy_global_symmetric_memory_buffer,
destroy_model_parallel,
update_pg_timeout
)
+from megatron.core.inference.symmetric_memory import SymmetricMemoryManager
from megatron.core.inference.unified_memory import create_unified_mempool
from megatron.core.resharding.refit import swap_model_weights
+try:
+ from torch_memory_saver import torch_memory_saver
+ torch_memory_saver.hook_mode = "torch"
+ HAVE_TORCH_MEMORY_SAVER = True
+except ImportError:
+ HAVE_TORCH_MEMORY_SAVER = False
+
from megatron.core.pipeline_parallel import get_forward_backward_func
from megatron.core.num_microbatches_calculator import (
destroy_num_microbatches_calculator,
@@ -215,7 +273,7 @@ def destroy_global_state():
destroy_global_vars()
destroy_num_microbatches_calculator()
destroy_global_memory_buffer()
- destroy_global_symmetric_memory_buffer()
+ SymmetricMemoryManager.destroy()
destroy_model_parallel()
destroy_rerun_state_machine()
@@ -231,31 +289,6 @@ def print_datetime(string, override_timestamp=None):
print_rank_0(f'[{string}] datetime: {time_str} ')
def num_floating_point_operations(args, batch_size):
- def calculate_layer_counts():
- """Calculate the number of attention, Mamba, and MLP layers."""
- if args.hybrid_override_pattern:
- from megatron.core.ssm.mamba_hybrid_layer_allocation import parse_hybrid_pattern
- # Parse unified pattern to separate main and MTP components
- parsed = parse_hybrid_pattern(args.hybrid_override_pattern)
- counts = {'M': 0, '*': 0, '-': 0, 'E': 0}
- # Count main decoder layers
- if parsed.main_pattern:
- for layer_type in parsed.main_pattern:
- if layer_type in counts:
- counts[layer_type] += 1
- # Count MTP layers (pattern repeated mtp_num_depths times)
- if parsed.mtp_pattern and parsed.mtp_num_depths > 0:
- for layer_type in parsed.mtp_pattern:
- if layer_type in counts:
- counts[layer_type] += parsed.mtp_num_depths
- return counts['*'], counts['M'], counts['-'], counts['E']
- else:
- num_attn_layers = round(args.num_layers * args.hybrid_attention_ratio)
- num_mlp_layers = round(args.num_layers * args.hybrid_mlp_ratio)
- num_mamba_layers = args.num_layers - num_attn_layers - num_mlp_layers
- num_moe_layers = 0
- return num_attn_layers, num_mamba_layers, num_mlp_layers, num_moe_layers
-
def mlp_layer_flops(batch_size, seq_len, hidden_size, expansion=4.0, swiglu=False):
"""Calculate FLOPs for an MLP layer."""
scale_factor = 3.0 / 2.0 if swiglu else 1.0
@@ -315,8 +348,29 @@ def mamba_layer_flops(batch_size, seq_len, hidden_size, state_dim=16,
+ (2 * batch_size * seq_len * d_in * hidden_size) # out_proj
)
+ def gdn_layer_flops(batch_size, seq_len, hidden_size,
+ qk_head_dim=128, v_head_dim=128,
+ num_qk_heads=16, num_v_heads=32,
+ conv_kernel_dim=4):
+ """Calculate FLOPs for a Gated Delta Net (GDN) layer."""
+ qk_dim = qk_head_dim * num_qk_heads
+ v_dim = v_head_dim * num_v_heads
+ return (
+ 2 * batch_size * seq_len * (
+ # in_proj: hidden_size -> (2*qk_dim + 2*v_dim + 2*num_v_heads)
+ hidden_size * (2 * qk_dim + 2 * v_dim + 2 * num_v_heads)
+ # conv1d
+ + conv_kernel_dim * (2 * qk_dim + v_dim)
+ # gated delta rule: KK^T, VK^T, S(a(I-bKK^T)), and SQ
+ + num_v_heads * (v_head_dim ** 2) * 4
+ # out_proj: v_dim -> hidden_size
+ + hidden_size * v_dim
+ )
+ )
+
def hybrid_flops(batch_size, seq_len, hidden_size,
num_attn_layers, num_mamba_layers, num_mlp_layers, num_moe_layers,
+ num_gdn_layers=0,
mamba_state_dim=128, mamba_head_dim=64,
mamba_num_groups=8, mamba_num_heads=128,
num_attn_heads=32, gqa=True,
@@ -324,6 +378,9 @@ def hybrid_flops(batch_size, seq_len, hidden_size,
mlp_expansion=4.0, swiglu=False,
moe_latent_size=None,
moe_ffn_hidden_size=2048, shared_expert_ffn_hidden_size=2048, num_experts_routed_to=1,
+ gdn_qk_head_dim=128, gdn_v_head_dim=128,
+ gdn_num_qk_heads=16, gdn_num_v_heads=32,
+ gdn_conv_kernel_dim=4,
vocab_size=256000, mtp_num_layers=0):
"""Calculate total FLOPs for the hybrid model."""
flops_fwd = (
@@ -337,6 +394,10 @@ def hybrid_flops(batch_size, seq_len, hidden_size,
num_moe_layers * moe_layer_flops(batch_size, seq_len, hidden_size, moe_ffn_hidden_size,
shared_expert_ffn_hidden_size, num_experts_routed_to,
moe_latent_size, swiglu) +
+ num_gdn_layers * gdn_layer_flops(batch_size, seq_len, hidden_size,
+ gdn_qk_head_dim, gdn_v_head_dim,
+ gdn_num_qk_heads, gdn_num_v_heads,
+ gdn_conv_kernel_dim) +
(2 * batch_size * seq_len * hidden_size * vocab_size * (1 + mtp_num_layers)) # logits computation
)
return flops_fwd * 3
@@ -388,6 +449,7 @@ def transformer_flops():
if args.moe_ffn_hidden_size is not None
else args.ffn_hidden_size
)
+ moe_latent_size = args.moe_latent_size
shared_expert_ffn_hidden_size = (
0
if args.moe_shared_expert_intermediate_size is None
@@ -569,7 +631,20 @@ def transformer_flops():
(args.ffn_hidden_size * ffn_expansion_factor)
* num_dense_layers
# routed experts
- + (moe_ffn_hidden_size * num_experts_routed_to * ffn_expansion_factor)
+ + (
+ (moe_ffn_hidden_size * num_experts_routed_to * ffn_expansion_factor)
+ if moe_latent_size is None
+ else (
+ (
+ moe_ffn_hidden_size
+ * num_experts_routed_to
+ * ffn_expansion_factor
+ * moe_latent_size
+ / args.hidden_size
+ ) # Routed experts run on moe_latent_size.
+ + 2 * moe_latent_size # Up proj and down proj.
+ )
+ )
* num_moe_layers
# Shared Experts.
+ (shared_expert_ffn_hidden_size * ffn_expansion_factor)
@@ -598,10 +673,20 @@ def transformer_flops():
return total_floating_point_operations
# Main entrypoint for FLOPs calculation.
- if args.is_hybrid_model:
+ if is_hybrid_model(args):
# Calculate the number of each type of layer.
- num_attn_layers, num_mamba_layers, num_mlp_layers, num_moe_layers = calculate_layer_counts()
+ from operator import itemgetter
+
+ from megatron.core.ssm.mamba_hybrid_layer_allocation import Symbols, get_hybrid_layer_counts
+ num_mamba_layers, num_gdn_layers, num_attn_layers, num_mlp_layers, num_moe_layers = (
+ itemgetter(Symbols.MAMBA, Symbols.GDN, Symbols.ATTENTION, Symbols.MLP, Symbols.MOE)(
+ get_hybrid_layer_counts(args.hybrid_layer_pattern)
+ )
+ )
+ mtp_num_layers = args.mtp_num_layers
+ if mtp_num_layers is None:
+ mtp_num_layers = 0
# Compute hybrid model FLOPs.
return hybrid_flops(
batch_size=batch_size,
@@ -611,6 +696,7 @@ def transformer_flops():
num_mamba_layers=num_mamba_layers,
num_mlp_layers=num_mlp_layers,
num_moe_layers=num_moe_layers,
+ num_gdn_layers=num_gdn_layers,
mamba_state_dim=args.mamba_state_dim,
mamba_head_dim=args.mamba_head_dim,
mamba_num_groups=args.mamba_num_groups,
@@ -627,8 +713,13 @@ def transformer_flops():
shared_expert_ffn_hidden_size=(0 if args.moe_shared_expert_intermediate_size is None
else args.moe_shared_expert_intermediate_size),
num_experts_routed_to=args.moe_router_topk,
+ gdn_qk_head_dim=args.linear_key_head_dim or 128,
+ gdn_v_head_dim=args.linear_value_head_dim or 128,
+ gdn_num_qk_heads=args.linear_num_key_heads or 16,
+ gdn_num_v_heads=args.linear_num_value_heads or 32,
+ gdn_conv_kernel_dim=args.linear_conv_kernel_dim or 4,
vocab_size=args.padded_vocab_size,
- mtp_num_layers=args.mtp_num_layers,
+ mtp_num_layers=mtp_num_layers,
)
else:
# Compute standard Transformer model FLOPs.
@@ -651,6 +742,7 @@ def get_start_time_from_progress_log():
start_time = None
start_num_floating_point_operations = None
latest_num_floating_point_operations = 0
+ latest_num_floating_point_operations_uncommitted = None
def _get_field(string, type):
return type(string.split(': ')[1])
@@ -662,6 +754,18 @@ def _get_field(string, type):
world_size_in_line = _get_field(line_tokens[2], int)
if line_tokens[3] == "Saved checkpoint":
latest_num_floating_point_operations = _get_field(line_tokens[7], float)
+ elif line_tokens[3] == "Saving async checkpoint":
+ # Checkpoint hasn't committed yet, so store for now in a different variable.
+ latest_num_floating_point_operations_uncommitted = _get_field(line_tokens[7], float)
+ elif line_tokens[3] == "Saved async checkpoint":
+ # Checkpoint has committed, so can update latest_num_floating_point_operations to
+ # value from latest 'Saving async checkpoint' message.
+ # Note: latest_num_floating_point_operations_uncommitted can be None if
+ # save_checkpoint was called directly (without save_checkpoint_and_time),
+ # which writes "Saved async checkpoint" but not "Saving async checkpoint".
+ if latest_num_floating_point_operations_uncommitted is not None:
+ latest_num_floating_point_operations = latest_num_floating_point_operations_uncommitted
+ latest_num_floating_point_operations_uncommitted = None
if world_size_in_line != args.world_size:
# Re-start search if we see a different world size.
start_time = None
@@ -674,6 +778,8 @@ def _get_field(string, type):
assert (
start_time is not None and start_num_floating_point_operations is not None
), "Should have seen at least one 'Starting job' entry with same world_size"
+ print_rank_0(f"megatron.training.get_start_time_from_progress_log: "
+ f"{start_time=}, {start_num_floating_point_operations=}")
return datetime.strptime(start_time, '%Y-%m-%d %H:%M:%S'), start_num_floating_point_operations
@@ -763,8 +869,14 @@ def pretrain(
to it. It is used for programs to add their own arguments.
args_defaults: a dictionary from argument-name to argument-value. It
to set already parse arguments.
- get_embedding_ranks (TODO):
- get_position_embedding_ranks (TODO):
+ get_embedding_ranks: a function that takes a list of ranks for a pipeline
+ group and returns those ranks that should have word embeddings.
+ For most models, these are the first and last pipeline stages.
+ If None, defaults to returning the first and last pipeline stages.
+ get_position_embedding_ranks: a function that takes a list of ranks for
+ a pipeline group and returns those ranks that should have position
+ embeddings. For most models, this is only the first pipeline stage.
+ If None, defaults to returning only the first pipeline stage.
non_loss_data_func (callable): A custom function to call during evaluation.
It can run e.g. benchmarks.
store: an optional instance of torch.distributed.Store, to be used by
@@ -940,6 +1052,8 @@ def pretrain(
or args.rl_inference_expert_model_parallel_size is not None
or args.rl_inference_expert_tensor_model_parallel_size is not None
):
+ from megatron.rl.parallel_utils import build_inference_pg_collection
+
print_rank_0(
"Building separate RL inference model with custom parallelism: "
f"TP={args.rl_inference_tensor_model_parallel_size}, "
@@ -975,15 +1089,30 @@ def pretrain(
# Optionally allocate the RL inference model weights from a unified virtual memory (UVM)
# mempool so we can prefetch weights to CPU when idle while keeping CUDA-graph-safe pointers.
+ # Alternatively, use torch_memory_saver to offload the weights to CPU when idle.
uvm_mempool = None
uvm_level = args.rl_inference_model_unified_memory_level
if uvm_level and uvm_level > 0:
uvm_mempool = create_unified_mempool()
- mempool_ctx = (
- torch.cuda.use_mem_pool(uvm_mempool) if uvm_mempool is not None else nullcontext()
+ # Determine which context manager to use for model allocation
+ # Use torch_memory_saver if offloading is requested but UVM is not enabled
+ use_torch_saver_for_inference_model = (
+ args.rl_offload_inference_model_weights_when_idle
+ and uvm_level == 0
+ and HAVE_TORCH_MEMORY_SAVER
)
- with mempool_ctx:
+ if use_torch_saver_for_inference_model:
+ # Use torch_memory_saver for offloading - allocate within a tagged region
+ model_alloc_ctx = torch_memory_saver.region(
+ tag="rl_inference_model", enable_cpu_backup=True
+ )
+ elif uvm_mempool is not None:
+ model_alloc_ctx = torch.cuda.use_mem_pool(uvm_mempool)
+ else:
+ model_alloc_ctx = nullcontext()
+
+ with model_alloc_ctx:
inference_model = get_model(
model_provider,
model_type,
@@ -993,7 +1122,12 @@ def pretrain(
)
inference_model[0].eval()
-
+ # Validate: offloading flag requires a separate inference model
+ if args.rl_offload_inference_model_weights_when_idle and inference_model is None:
+ raise ValueError(
+ "--rl-offload-inference-model-weights-when-idle requires a separate inference model. "
+ "This flag is only useful when doing refit since the weights are shared with the training model."
+ )
# Data stuff.
app_metrics['app_build_dataiters_start_time'] = one_logger_utils.get_timestamp_in_ms()
@@ -1046,11 +1180,15 @@ def pretrain(
# Add job name to the wandb config to make it easier to run more singleton dependency jobs.
wandb_writer.config.update({'slurm_job_name': os.getenv("SLURM_JOB_NAME", "N/A")})
- if not args.skip_train:
- print_rank_0('training ...')
+ if not args.skip_train or args.perform_rl_step:
+ if args.skip_train:
+ print_rank_0('RL inference-only mode (--skip-train --perform-rl-step) ...')
+ else:
+ print_rank_0('training ...')
iteration = 0
- if args.do_train and args.train_iters > 0:
+ args.curr_iteration = iteration
+ if args.do_train and (args.train_iters or 0) > 0:
iteration, num_floating_point_operations_so_far = train(
forward_step_func,
model,
@@ -1067,7 +1205,7 @@ def pretrain(
print_datetime('after training is done')
- if args.save and iteration != 0 and iteration % args.save_interval != 0:
+ if not args.skip_train and args.save and iteration != 0 and iteration % args.save_interval != 0:
save_checkpoint(
iteration,
model,
@@ -1090,8 +1228,9 @@ def pretrain(
if args.do_valid:
prefix = f'iteration {iteration} on validation set'
- if getattr(args, 'perform_rl_step', False):
+ if args.perform_rl_step:
rl_eval_model = model
+ rl_training_model = None
if inference_model is not None:
inf_core = unwrap_model(inference_model[0])
# If separate inference and training models, swap training weights
@@ -1099,11 +1238,14 @@ def pretrain(
rl_utils._maybe_prefetch_separate_inference_model_weights(inf_core, to_cpu=False)
swap_model_weights(model, inference_model, args.refit_method)
rl_eval_model = inference_model
+ rl_training_model = model
rl_utils.evaluate_and_print_results_rl(
valid_data_iterator,
rl_eval_model,
optimizer,
- iteration, write_to_tensorboard=not args.skip_train
+ iteration,
+ write_to_tensorboard=not args.skip_train,
+ training_model=rl_training_model,
)
else:
evaluate_and_print_results(
@@ -1141,7 +1283,7 @@ def pretrain(
{'app_finish_time': one_logger_utils.get_timestamp_in_ms()}
)
- if getattr(args, 'perform_rl_step', False):
+ if args.perform_rl_step:
rl_utils.rl_inference_interface_shutdown()
ft_integration.shutdown()
@@ -1328,7 +1470,15 @@ def build_model():
kwargs['bucket_size'] = args.ddp_bucket_size
kwargs['pad_buckets_for_high_nccl_busbw'] = args.ddp_pad_buckets_for_high_nccl_busbw
kwargs['reduce_scatter_with_fp32_accumulation'] = args.ddp_reduce_scatter_with_fp32_accumulation
+ kwargs['param_name_patterns_for_fp32_local_accumulation'] = \
+ tuple(args.ddp_param_name_patterns_for_fp32_local_accumulation)
kwargs['average_in_collective'] = args.ddp_average_in_collective
+ # Megatron-FSDP arguments.
+ kwargs['megatron_fsdp_main_params_dtype'] = args.megatron_fsdp_main_params_dtype
+ kwargs['megatron_fsdp_main_grads_dtype'] = args.megatron_fsdp_main_grads_dtype
+ kwargs['megatron_fsdp_grad_comm_dtype'] = args.megatron_fsdp_grad_comm_dtype
+
+ # Initialize DDPConfig.
ddp_config = DistributedDataParallelConfig(**kwargs)
# In the Megatron FSDP and DDP use path, we need to initialize the bucket size.
@@ -1343,6 +1493,7 @@ def build_model():
# Set bucket_size to infinity if overlap_grad_reduce is False.
if not ddp_config.overlap_grad_reduce:
ddp_config.bucket_size = None
+
# Setup stream for ddp initialization. The side-stream may be necessary for cuda graph
# capture support with DDP, but we sync it with the current stream to avoid races.
ddp_stream = torch.cuda.Stream()
@@ -1355,10 +1506,9 @@ def build_model():
config=config,
ddp_config=ddp_config,
module=model_chunk,
- # Turn off bucketing for model_chunk 2 onwards, since communication for these
- # model chunks is overlapped with compute anyway.
- disable_bucketing=(model_chunk_idx > 0)
- or args.overlap_param_gather_with_optimizer_step,
+ # Turn off bucketing for model_chunk 2 onwards, since communication
+ # for these model chunks is overlapped with compute anyway.
+ disable_bucketing=(model_chunk_idx > 0) or args.overlap_param_gather_with_optimizer_step,
)
for (model_chunk_idx, model_chunk) in enumerate(model)
]
@@ -1468,16 +1618,35 @@ def setup_model_and_optimizer(
timers = get_timers()
one_logger = get_one_logger()
- wrap_with_ddp = not args.skip_train
+ # Skip optimizer when not training. In RL inference-only mode (skip_train + perform_rl_step),
+ # --no-load-optim controls whether the optimizer is skipped (saving memory) or created
+ # (required for --rl-offload-optimizer-during-inference).
+ skip_optimizer = args.skip_train and (not args.perform_rl_step or args.no_load_optim)
+ wrap_with_ddp = not skip_optimizer
model = get_model(model_provider_func, model_type, wrap_with_ddp=wrap_with_ddp)
unwrapped_model = unwrap_model(model)
one_logger and one_logger.log_metrics({"app_build_optimzer_start_time": one_logger_utils.get_timestamp_in_ms()})
- if args.skip_train:
+ if skip_optimizer:
optimizer, opt_param_scheduler = None, None
+ # In RL inference-only mode, train_iters must still be set despite having no optimizer.
+ if args.perform_rl_step:
+ update_train_iters(args)
else:
config, config_overrides = get_megatron_optimizer_config(args)
config.timers = timers
+ if getattr(args, "use_mup", False):
+ model_config_source = (
+ unwrapped_model[0] if isinstance(unwrapped_model, list) else unwrapped_model
+ )
+ model_config = get_model_config(model_config_source)
+ mup_overrides = get_mup_config_overrides(
+ config=config,
+ mup_width_mult=model_config.mup_width_mult,
+ optimizer_type=config.optimizer,
+ )
+ if mup_overrides:
+ config_overrides = {**(config_overrides or {}), **mup_overrides}
if 'muon' not in config.optimizer:
# If the user is asking for a non-zero embedding init std, skip weight decay for embeddings
@@ -1487,7 +1656,7 @@ def setup_model_and_optimizer(
config,
model,
config_overrides=config_overrides,
- use_gloo_process_groups=args.enable_gloo_process_groups,
+ use_gloo_process_groups=args.use_gloo_process_groups,
dump_param_to_param_group_map=args.dump_param_to_param_group_map,
)
else:
@@ -1495,7 +1664,7 @@ def setup_model_and_optimizer(
config,
model,
config_overrides=config_overrides,
- use_gloo_process_groups=args.enable_gloo_process_groups,
+ use_gloo_process_groups=args.use_gloo_process_groups,
layer_wise_distributed_optimizer='dist' in config.optimizer,
)
opt_param_scheduler = get_optimizer_param_scheduler(optimizer)
@@ -1796,7 +1965,7 @@ def train_step(forward_step_func, data_iterator, model, optimizer, opt_param_sch
def training_log(
loss_dict,
total_loss_dict,
- learning_rate,
+ learning_rate: float | None,
iteration,
loss_scale,
report_memory_flag,
@@ -1879,14 +2048,7 @@ def training_log(
])
# Add timers from RL loop if needed.
if getattr(args, 'perform_rl_step', False):
- timers_to_log.extend(['rollout-collection', 'inference-setup', 'collect-rollouts', 'postrollout-gc-collect',
- 'sync-rollouts', 'prepare-data-for-update', 'compute-group-stats',
- 'prepare-trajectories', 'get-ltor-masks-and-position-ids', 'create-logprobs-dataloader',
- 'compute-logprobs', 'compute-ref-logprobs', 'compute-prob-stats',
- 'prepare-advantages', 'create-dataloader', 'log-wandb-tb',
- 'offload-optimizer-before-inference', 'onload-kv-cache-before-inference',
- 'wait-for-decode-only', 'build-cuda-graphs', 'suspend-engine',
- 'offload-kv-cache-after-inference', 'onload-optimizer-after-inference'])
+ timers_to_log.extend(RL_LOGGABLE_TIMER_NAMES)
# Calculate batch size.
batch_size = args.micro_batch_size * args.data_parallel_size * get_num_microbatches()
@@ -1897,15 +2059,16 @@ def training_log(
total_iterations = total_loss_dict[advanced_iters_key] + total_loss_dict[skipped_iters_key]
# learning rate will be None on ranks without trainable params, so we must gather across mp ranks
- learning_rate = reduce_max_stat_across_model_parallel_group(learning_rate)
+ learning_rate: float | None = reduce_max_stat_across_model_parallel_group(learning_rate)
# Tensorboard values.
if writer and (iteration % args.tensorboard_log_interval == 0):
if wandb_writer:
wandb_writer.log({'samples vs steps': args.consumed_train_samples}, iteration)
- writer.add_scalar('learning-rate', learning_rate, iteration)
- writer.add_scalar('learning-rate vs samples', learning_rate, args.consumed_train_samples)
- if wandb_writer:
- wandb_writer.log({'learning-rate': learning_rate}, iteration)
+ if learning_rate is not None:
+ writer.add_scalar('learning-rate', learning_rate, iteration)
+ writer.add_scalar('learning-rate vs samples', learning_rate, args.consumed_train_samples)
+ if wandb_writer:
+ wandb_writer.log({'learning-rate': learning_rate}, iteration)
if args.skipped_train_samples > 0:
writer.add_scalar('skipped-train-samples', args.skipped_train_samples, iteration)
if wandb_writer:
@@ -1953,7 +2116,7 @@ def training_log(
writer.add_scalar('params-norm vs samples', params_norm, args.consumed_train_samples)
if wandb_writer:
wandb_writer.log({'params-norm': params_norm}, iteration)
- if getattr(args, 'perform_rl_step', False):
+ if args.perform_rl_step:
grpo_collection_iteration = iteration // (args.grpo_iterations * ( ( args.grpo_samples_per_iteration )// args.global_batch_size ))
writer.add_scalar('grpo_collection_iteration', grpo_collection_iteration, iteration)
if wandb_writer:
@@ -1988,8 +2151,13 @@ def training_log(
if args.moe_z_loss_coeff is not None:
track_names.append("z_loss")
- if args.is_hybrid_model:
- layers = args.hybrid_override_pattern.count('E')
+ if is_hybrid_model(args):
+ from operator import itemgetter
+
+ from megatron.core.ssm.mamba_hybrid_layer_allocation import (
+ Symbols, get_hybrid_layer_counts,
+ )
+ layers = itemgetter(Symbols.MOE)(get_hybrid_layer_counts(args.hybrid_layer_pattern))
else:
layers = args.num_layers
@@ -2084,14 +2252,15 @@ def training_log(
wandb_writer.log({'iter-energy/gpu': energy}, iteration)
wandb_writer.log({'power/gpu': power}, iteration)
# Decoupled_learning_rate should be not None only on first and last pipeline stage.
- log_string += f' learning rate: {learning_rate:.6E} |'
+ if learning_rate is not None:
+ log_string += f' learning rate: {learning_rate:.6E} |'
log_string += f' global batch size: {batch_size:5d} |'
for key in total_loss_dict:
if key not in [advanced_iters_key, skipped_iters_key, nan_iters_key]:
avg = total_loss_dict[key].item() / float(
max(1, total_loss_dict[advanced_iters_key])
)
- if avg > 0.0:
+ if avg >= 0.0:
log_string += ' {}: {:.6E} |'.format(key, avg)
if should_reset:
total_loss_dict[key] = torch.tensor([0.0], dtype=torch.float, device='cuda')
@@ -2204,6 +2373,10 @@ def save_checkpoint_and_time(
timers = get_timers()
energy_monitor = get_energy_monitor()
+ # Synchronize forward pre-hook state before checkpoint save to avoid race conditions
+ if should_disable_forward_pre_hook(args):
+ force_param_sync(model)
+
# Stop timer to get accurate train interval time and exclude checkpointing duration
timers('interval-time').stop()
energy_monitor.pause()
@@ -2214,8 +2387,13 @@ def save_checkpoint_and_time(
# Log E2E metrics before save-checkpoint
one_logger_utils.track_e2e_metrics()
- if should_disable_forward_pre_hook(args):
- force_param_sync(model)
+ # Free overlap param-gather buffers and release cached GPU memory so
+ # that the async checkpoint worker process has enough GPU headroom for
+ # D2H tensor transfers.
+ for model_chunk in model:
+ if hasattr(model_chunk, 'free_overlap_buffers'):
+ model_chunk.free_overlap_buffers()
+ torch.cuda.empty_cache()
global num_checkpoints_memory_reported, MAX_NUM_CHECKPOINTS_MEMORY_REPORTED
should_report_memory = num_checkpoints_memory_reported < MAX_NUM_CHECKPOINTS_MEMORY_REPORTED
@@ -2308,11 +2486,14 @@ def post_training_step_callbacks(
if (
args.profile
and iteration == args.profile_step_end
- and torch.distributed.get_rank() in args.profile_ranks
+ and (len(args.profile_ranks) == 0 or
+ torch.distributed.get_rank() in args.profile_ranks)
):
if args.use_pytorch_profiler:
assert prof is not None
prof.stop()
+ if prof.execution_trace_observer is not None:
+ prof.execution_trace_observer.unregister_callback()
else:
torch.cuda.check_error(torch.cuda.cudart().cudaProfilerStop())
if nsys_nvtx_context is not None:
@@ -2456,47 +2637,54 @@ def train(
args = get_args()
timers = get_timers()
- if getattr(args, 'perform_rl_step', False):
+ if args.perform_rl_step:
assert has_rl_utils, "RL cannot run without the megatron.rl package"
# Additional variable initialization for RL training
- if getattr(args, 'perform_rl_step', False):
- print_rank_0("> Loading pretrained checkpoint for reference weights in RL training...")
- load, finetune, no_load_optim = args.load, args.finetune, args.no_load_optim
- args.no_load_optim = True
-
- # Load pretrained checkpoint
- args.load = None
- args.finetune = True
- load_checkpoint(
- model,
- None, # Don't load optimizer state
- None, # Don't load scheduler state
- checkpointing_context=checkpointing_context,
- skip_load_to_model_and_opt=HAVE_FSDP2
- and getattr(args, "use_torch_fsdp2", False)
- and args.ckpt_format == "torch_dist",
- )
- ref_state_dict = {k: (v.cpu() if v is not None else v) for k, v in model[0].state_dict().items()}
+ if args.perform_rl_step:
+ if args.skip_train:
+ # In inference-only mode, use current weights as reference.
+ print_rank_0("> RL inference-only: using current weights as reference.")
+ ref_state_dict = {
+ k: (v.cpu() if v is not None else v) for k, v in model[0].state_dict().items()
+ }
+ else:
+ print_rank_0("> Loading pretrained checkpoint for reference weights in RL training...")
+ load, finetune, no_load_optim = args.load, args.finetune, args.no_load_optim
+ args.no_load_optim = True
+
+ # Load pretrained checkpoint
+ args.load = None
+ args.finetune = True
+ load_checkpoint(
+ model,
+ None, # Don't load optimizer state
+ None, # Don't load scheduler state
+ checkpointing_context=checkpointing_context,
+ skip_load_to_model_and_opt=HAVE_FSDP2
+ and getattr(args, "use_torch_fsdp2", False)
+ and args.ckpt_format == "torch_dist",
+ )
+ ref_state_dict = {k: (v.cpu() if v is not None else v) for k, v in model[0].state_dict().items()}
- # Reload RL training checkpoint weights
- args.load = load
- args.finetune = finetune
- print_rank_0("> Reloading RL training checkpoint...")
- load_checkpoint(
- model,
- None,
- None,
- checkpointing_context=checkpointing_context,
- skip_load_to_model_and_opt=HAVE_FSDP2
- and getattr(args, "use_torch_fsdp2", False)
- and args.ckpt_format == "torch_dist",
- )
+ # Reload RL training checkpoint weights
+ args.load = load
+ args.finetune = finetune
+ print_rank_0("> Reloading RL training checkpoint...")
+ load_checkpoint(
+ model,
+ None,
+ None,
+ checkpointing_context=checkpointing_context,
+ skip_load_to_model_and_opt=HAVE_FSDP2
+ and getattr(args, "use_torch_fsdp2", False)
+ and args.ckpt_format == "torch_dist",
+ )
- args.no_load_optim = no_load_optim
+ args.no_load_optim = no_load_optim
# IMPORTANT FIX: For RL training, reinitialize the microbatch calculator with the correct configuration
- if getattr(args, 'perform_rl_step', False):
+ if args.perform_rl_step:
print_rank_0("> Reinitializing microbatch calculator for GRPO training...")
from megatron.core.num_microbatches_calculator import (
destroy_num_microbatches_calculator,
@@ -2569,7 +2757,7 @@ def train(
num_floating_point_operations_so_far = args.num_floating_point_operations_so_far
# Setup some training config params.
- config.grad_scale_func = optimizer.scale_loss
+ config.grad_scale_func = optimizer.scale_loss if optimizer is not None else None
config.timers = timers
if isinstance(model[0], (megatron_FSDP, DDP)) and args.overlap_grad_reduce:
assert config.no_sync_func is None, (
@@ -2664,9 +2852,20 @@ def get_e2e_base_metrics():
nsys_nvtx_context = None # reference to context for nsys profiling, so it can be cleaned up
if (
args.profile
- and torch.distributed.get_rank() in args.profile_ranks
+ and (len(args.profile_ranks) == 0 or
+ torch.distributed.get_rank() in args.profile_ranks)
and args.use_pytorch_profiler
):
+ if args.pytorch_profiler_collect_chakra:
+ et_dir = Path(f"{args.tensorboard_dir}/../chakra")
+ et_dir.mkdir(parents=True, exist_ok=True)
+ et = torch.profiler.ExecutionTraceObserver().register_callback(f"{et_dir}/rank-{torch.distributed.get_rank()}.json.gz")
+ else:
+ et = None
+ def trace_handler(p):
+ profile_dir = Path(f"{args.tensorboard_dir}/../torch_profile")
+ profile_dir.mkdir(parents=True, exist_ok=True)
+ p.export_chrome_trace(f"{profile_dir}/rank-{torch.distributed.get_rank()}.json.gz")
prof = torch.profiler.profile(
activities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA],
schedule=torch.profiler.schedule(
@@ -2675,10 +2874,11 @@ def get_e2e_base_metrics():
active=args.profile_step_end - args.profile_step_start,
repeat=1,
),
- on_trace_ready=torch.profiler.tensorboard_trace_handler(args.tensorboard_dir),
- record_shapes=True,
+ on_trace_ready=trace_handler,
+ record_shapes=args.pytorch_profiler_collect_shapes,
profile_memory=True,
- with_stack=True,
+ with_stack=args.pytorch_profiler_collect_callstack,
+ execution_trace_observer=et,
)
prof.start()
@@ -2714,7 +2914,9 @@ def get_e2e_base_metrics():
# Run training iterations till done.
buffered_rollouts = None
while iteration < args.train_iters:
- if args.profile and torch.distributed.get_rank() in args.profile_ranks:
+ if (args.profile
+ and (len(args.profile_ranks) == 0 or
+ torch.distributed.get_rank() in args.profile_ranks)):
if args.use_pytorch_profiler:
prof.step()
elif iteration == args.profile_step_start:
@@ -2770,7 +2972,7 @@ def get_e2e_base_metrics():
# Capture CUDA Graphs.
if (
args.cuda_graph_impl == "transformer_engine"
- and not cuda_graph_helper.graphs_created()
+ and not cuda_graph_helper.capture_finished()
and iteration - start_iteration == args.cuda_graph_warmup_steps
):
if args.cuda_graph_warmup_steps > 0 and should_disable_forward_pre_hook(args):
@@ -2781,7 +2983,7 @@ def get_e2e_base_metrics():
cuda_graph_helper.cuda_graph_set_manual_hooks()
# Completely skip iteration if needed.
- if iteration in args.iterations_to_skip:
+ if (iteration + 1) in args.iterations_to_skip:
# Dummy train_step to fast forward train_data_iterator.
dummy_train_step(train_data_iterator)
if iteration == start_iteration:
@@ -2798,7 +3000,10 @@ def get_e2e_base_metrics():
# For GRPO, we keep the data for a few epochs. DeepSeekMath paper calls this number $\mu$.
# It is similar to a PPO epoch.
- if getattr(args, 'perform_rl_step', False):
+ if args.perform_rl_step:
+ if optimizer is None:
+ # Release stale CUDA cached memory before inference.
+ torch.cuda.empty_cache()
with torch.no_grad():
train_data_iterator = rl_utils.get_grpo_data_iterator(
model, inference_model, optimizer, iteration, ref_state_dict,
@@ -2814,20 +3019,31 @@ def get_e2e_base_metrics():
# we use previously-generated data for an update.
buffered_rollouts = train_data_iterator
- ft_integration.on_training_step_start()
- (
- loss_dict,
- skipped_iter,
- should_checkpoint,
- should_exit,
- exit_code,
- grad_norm,
- num_zeros_in_grad,
- max_attention_logit,
- ) = train_step(
- forward_step_func, train_data_iterator, model, optimizer, opt_param_scheduler, config, forward_backward_func, memory_stats_collector, iteration=iteration
- )
- ft_integration.on_training_step_end()
+ if args.skip_train:
+ # RL inference-only mode: skip gradient updates, just collect rollouts.
+ loss_dict = {}
+ skipped_iter = 0
+ should_checkpoint = False
+ should_exit = False
+ exit_code = 0
+ grad_norm = 0.0
+ num_zeros_in_grad = 0
+ max_attention_logit = None
+ else:
+ ft_integration.on_training_step_start()
+ (
+ loss_dict,
+ skipped_iter,
+ should_checkpoint,
+ should_exit,
+ exit_code,
+ grad_norm,
+ num_zeros_in_grad,
+ max_attention_logit,
+ ) = train_step(
+ forward_step_func, train_data_iterator, model, optimizer, opt_param_scheduler, config, forward_backward_func, memory_stats_collector,iteration=iteration
+ )
+ ft_integration.on_training_step_end()
if should_checkpoint:
save_checkpoint_and_time(
iteration,
@@ -2864,8 +3080,8 @@ def get_e2e_base_metrics():
and args.cuda_graph_warmup_steps == 0
):
assert (
- cuda_graph_helper.graphs_created()
- ), "CUDA Graphs should have been created."
+ cuda_graph_helper.capture_finished()
+ ), "CUDA Graph capture should have been finished."
cuda_graph_helper.cuda_graph_set_manual_hooks()
iteration += 1
@@ -2884,7 +3100,7 @@ def get_e2e_base_metrics():
if pad_buf is not None:
pad_buf.manual_buffer_registration()
- if getattr(args, 'perform_rl_step', False) and args.rl_use_sequence_packing:
+ if args.perform_rl_step and args.rl_use_sequence_packing:
iteration_sequences = rl_utils.get_iteration_sequence_count(args)
# Track bins separately for packed mode
bin_count = (
@@ -2916,7 +3132,7 @@ def get_e2e_base_metrics():
num_floating_point_operations_since_last_log_event += num_floating_point_operations_in_batch
# Logging.
- if not optimizer.is_stub_optimizer:
+ if optimizer is not None and not optimizer.is_stub_optimizer:
loss_scale = optimizer.get_loss_scale().item()
else:
loss_scale = 1.0
@@ -2924,12 +3140,10 @@ def get_e2e_base_metrics():
if args.log_params_norm:
params_norm = calc_params_l2_norm(model)
- learning_rate = None
- for param_group in optimizer.param_groups:
- if len(param_group['params']) == 0:
- continue
- if param_group['default_config']:
- learning_rate = param_group['lr']
+ if optimizer is not None:
+ learning_rate = get_canonical_lr_for_logging(optimizer.param_groups)
+ else:
+ learning_rate = None
report_memory_flag = training_log(
loss_dict,
total_loss_dict,
@@ -2960,8 +3174,9 @@ def get_e2e_base_metrics():
gc.collect()
prefix = f'iteration {iteration}'
timers('eval-time', log_level=0).start(barrier=True)
- if getattr(args, 'perform_rl_step', False):
+ if args.perform_rl_step:
rl_eval_model = model
+ rl_training_model = None
# If separate inference and training models, swap training weights
# back to the inference model for RL evaluation.
if inference_model is not None:
@@ -2971,12 +3186,14 @@ def get_e2e_base_metrics():
)
swap_model_weights(model, inference_model, args.refit_method)
rl_eval_model = inference_model
+ rl_training_model = model
rl_utils.evaluate_and_print_results_rl(
valid_data_iterator,
rl_eval_model,
optimizer,
iteration,
write_to_tensorboard=True,
+ training_model=rl_training_model,
)
else:
evaluate_and_print_results(prefix, forward_step_func,
@@ -3062,7 +3279,7 @@ def get_e2e_base_metrics():
wandb_writer.finish()
ft_integration.shutdown()
one_logger_utils.finish()
- if getattr(args, 'perform_rl_step', False):
+ if args.perform_rl_step:
rl_utils.rl_inference_interface_shutdown()
sys.exit(exit_code)
@@ -3108,6 +3325,17 @@ def evaluate(
if args.cuda_graph_impl == "local" and CudaGraphScope.full_iteration in args.cuda_graph_scope:
forward_backward_func = FullCudaGraphWrapper(forward_backward_func, cuda_graph_warmup_steps=args.cuda_graph_warmup_steps)
+ if has_nvidia_modelopt:
+ # [ModelOpt]: Pipeline-parallel Distillation stacks student and teacher tensors
+ adjust_tensor_shapes_fn = get_tensor_shapes_adjust_fn_for_distillation(
+ model,
+ seq_length=args.seq_length,
+ micro_batch_size=args.micro_batch_size,
+ decoder_seq_length=args.decoder_seq_length,
+ )
+ else:
+ adjust_tensor_shapes_fn = None
+
if eval_iters is None:
eval_iters = args.eval_iters
@@ -3132,6 +3360,7 @@ def evaluate(
micro_batch_size=args.micro_batch_size,
decoder_seq_length=args.decoder_seq_length,
forward_only=True,
+ adjust_tensor_shapes_fn=adjust_tensor_shapes_fn,
)
ft_integration.on_eval_step_end()
config.timers = get_timers()
@@ -3144,16 +3373,14 @@ def evaluate(
# Reduce across processes.
for key in loss_dicts[0].keys():
if key not in total_loss_dict:
- total_loss_dict[key] = torch.tensor(
- [0.0, 0.0], dtype=torch.float
- ).cuda()
+ total_loss_dict[key] = torch.tensor([0.0, 0.0], dtype=torch.float, device='cuda')
val = [x[key].view(-1) for x in loss_dicts]
if val[0].numel() == 2:
if args.sft:
# normalize over micro batch instead of global
val = torch.vstack(val)
- val = val[:, 0] / val[:, 1]
+ val = val[:, 0] / val[:, 1].clamp(min=1)
val = val.mean()
torch.distributed.all_reduce(
val,
@@ -3221,8 +3448,6 @@ def evaluate(
rerun_state_machine.set_mode(rerun_mode)
- rerun_state_machine.set_mode(rerun_mode)
-
return total_loss_dict, collected_non_loss_data, False
@@ -3405,12 +3630,12 @@ def build_train_valid_test_data_loaders(build_train_valid_test_datasets_provider
if is_distributed or mpu.get_tensor_model_parallel_rank() == 0:
# Build datasets and dataloders.
- if getattr(args, 'perform_rl_step', True):
+ if args.perform_rl_step:
# we don't need to build any dataloaders for RL training
train_dataloader = None
valid_dataloaders = None
test_dataloader = None
- do_train = args.train_iters > 0
+ do_train = (args.train_iters or 0) > 0
do_valid = (args.full_validation or args.eval_iters > 0)
do_test = (args.full_validation or args.eval_iters > 0)
@@ -3450,9 +3675,6 @@ def build_train_valid_test_data_loaders(build_train_valid_test_datasets_provider
args.do_train = getattr(args, "do_train", False) or flags[0].item()
args.do_valid = getattr(args, "do_valid", False) or flags[1].item()
args.do_test = getattr(args, "do_test", False) or flags[2].item()
- if getattr(args, 'perform_rl_step', False):
- args.to_test = False
-
return train_dataloader, valid_dataloaders, test_dataloader
@@ -3532,4 +3754,8 @@ def _get_iterator(dataloader_type, dataloader):
def should_disable_forward_pre_hook(args):
"""Block forward pre-hook for certain configurations."""
- return not args.use_megatron_fsdp and args.use_distributed_optimizer and args.overlap_param_gather
+ return (
+ not args.use_megatron_fsdp
+ and (args.use_distributed_optimizer or 'dist' in args.optimizer)
+ and args.overlap_param_gather
+ )
diff --git a/megatron/training/utils.py b/megatron/training/utils.py
index 9d5e08e8fba..c4ca52d6187 100644
--- a/megatron/training/utils.py
+++ b/megatron/training/utils.py
@@ -55,7 +55,8 @@
to_local_if_dtensor,
unwrap_model,
)
-from megatron.legacy.model.module import param_is_not_shared
+
+from megatron.core.transformer.module import param_is_not_shared
from megatron.training.global_vars import get_tokenizer
@@ -254,7 +255,7 @@ def average_losses_across_data_parallel_group(losses):
return averaged_losses
-def reduce_max_stat_across_model_parallel_group(stat: float) -> float:
+def reduce_max_stat_across_model_parallel_group(stat: float) -> float | None:
"""
Ranks without an optimizer will have no grad_norm or num_zeros_in_grad stats.
We need to ensure the logging and writer rank has those values.
@@ -269,6 +270,7 @@ def reduce_max_stat_across_model_parallel_group(stat: float) -> float:
stat, op=torch.distributed.ReduceOp.MAX, group=mpu.get_model_parallel_group()
)
if stat.item() == -1.0:
+ # No rank has a valid stat, so return None to indicate that it is None across all ranks.
return None
else:
return stat.item()
@@ -476,6 +478,11 @@ def print_rank_last(message):
print(message, flush=True)
+def is_hybrid_model(args):
+ """Returns True if the model is a hybrid Mamba-Transformer model."""
+ return args.hybrid_layer_pattern is not None
+
+
def is_first_or_last_pipeline_stage(vp_stage):
"""Return True if on first or last pipeline stage, taking into account virtual
pipeline parallelism."""
@@ -646,25 +653,6 @@ def _broadcast_cu_seqlens(cu_seqlens):
_broadcast(batch['loss_mask'])
_broadcast(batch['attention_mask'])
- def _broadcast_cu_seqlens(cu_seqlens):
- dev = torch.cuda.current_device()
-
- n = 0 if cu_seqlens is None else int(cu_seqlens.numel())
- n_tensor = torch.tensor(n, dtype=torch.int64, device=dev)
- _broadcast(n_tensor)
-
- if n == 0:
- buf = torch.empty(0, dtype=torch.int32, device=dev)
- else:
- assert isinstance(cu_seqlens, torch.Tensor)
- assert cu_seqlens.dtype == torch.int32
- assert cu_seqlens.shape[0] == 1, "micro-batch-size must be 1 for packing"
- buf = cu_seqlens.to(device=dev, non_blocking=True).contiguous()
- _broadcast(buf)
-
- _broadcast_cu_seqlens(batch['cu_seqlens'])
- _broadcast(batch['max_seqlen'])
-
else:
if args.hybrid_context_parallel:
seq_len = torch.tensor(0, dtype=torch.int32, device=torch.cuda.current_device())
@@ -703,7 +691,7 @@ def _broadcast_cu_seqlens(cu_seqlens):
device=torch.cuda.current_device(),
)
cu_seqlens = None
- if args.sft:
+ if args.hybrid_context_parallel or args.sft:
max_seqlen = torch.empty(
1,
dtype=torch.int32,
@@ -711,13 +699,7 @@ def _broadcast_cu_seqlens(cu_seqlens):
)
else:
max_seqlen = None
-
- cu_seqlens = None
- max_seqlen = torch.empty(
- 1,
- dtype=torch.int32,
- device=torch.cuda.current_device(),
- ) if args.hybrid_context_parallel else None
+
local_cp_size = torch.empty(
1,
dtype=torch.int32,
@@ -772,24 +754,6 @@ def _broadcast_cu_seqlens():
_broadcast(loss_mask)
_broadcast(attention_mask)
- def _broadcast_cu_seqlens():
- dev = torch.cuda.current_device()
-
- n = torch.empty((), dtype=torch.int64, device=dev)
- _broadcast(n)
- n = int(n.item())
-
- if n == 0:
- cu_seqlens = torch.empty(0, dtype=torch.int32, device=dev)
- else:
- cu_seqlens = torch.empty((args.micro_batch_size, n), dtype=torch.int32, device=dev)
- _broadcast(cu_seqlens)
-
- return cu_seqlens if n > 0 else None
-
- cu_seqlens = _broadcast_cu_seqlens()
- _broadcast(max_seqlen)
-
batch = {
'tokens': tokens,
'labels': labels,
@@ -837,27 +801,37 @@ def _empty_like_if_meta(tensor: torch.Tensor, *, device: torch.device):
def get_nvtx_range():
- """Create an NVTX range context manager."""
+ """Create an NVTX range context manager.
+
+ Returns a context manager that:
+ - Creates an NVTX range for profiling (nsight-systems compatible)
+ - Optionally tracks time via Megatron timers when time=True
+
+ Args (for returned context manager):
+ msg: Name of the range/timer
+ time: If True, also track with Megatron timers (default: False)
+ log_level: Timer log level (0=always, 1=default, 2=verbose). Default: 1
+ """
try:
from torch.cuda import nvtx
@contextmanager
- def nvtx_range(msg, time=False):
+ def nvtx_range(msg, time=False, log_level=1):
if time:
timers = get_timers()
- timers(msg, log_level=0).start()
+ timers(msg, log_level=log_level).start()
try:
nvtx.range_push(msg)
yield
finally:
nvtx.range_pop()
if time:
- timers(msg, log_level=0).stop()
+ timers(msg, log_level=log_level).stop()
return nvtx_range
except:
@contextmanager
- def dummy_range(msg):
+ def dummy_range(msg, time=False, log_level=1):
yield
return dummy_range
diff --git a/megatron/training/yaml_arguments.py b/megatron/training/yaml_arguments.py
index 70ccac4402c..db90949082d 100644
--- a/megatron/training/yaml_arguments.py
+++ b/megatron/training/yaml_arguments.py
@@ -304,20 +304,11 @@ def validate_yaml(args, defaults={}):
if args.model_parallel.tensor_model_parallel_size == 1:
args.model_parallel.sequence_parallel = False
- # disable async_tensor_model_parallel_allreduce when
- # model parallel memory optimization is enabled
- if args.model_parallel.sequence_parallel:
- args.model_parallel.async_tensor_model_parallel_allreduce = False
-
if os.environ.get('CUDA_DEVICE_MAX_CONNECTIONS') != "1":
if args.model_parallel.sequence_parallel:
raise RuntimeError(
"Using sequence parallelism requires setting the environment variable "
"CUDA_DEVICE_MAX_CONNECTIONS to 1")
- if args.model_parallel.async_tensor_model_parallel_allreduce:
- raise RuntimeError(
- "Using async gradient all reduce requires setting the environment "
- "variable CUDA_DEVICE_MAX_CONNECTIONS to 1")
# MoE Spec check
if args.language_model.num_moe_experts is not None:
diff --git a/model_provider.py b/model_provider.py
index f8f6ccae01c..0c80c54dfdb 100644
--- a/model_provider.py
+++ b/model_provider.py
@@ -50,13 +50,9 @@ def model_provider(
def oom_observer(device, alloc, device_alloc, device_free):
# snapshot right after an OOM happened
print('saving allocated state during OOM')
- snapshot = torch.cuda.memory._snapshot()
- from pickle import dump
- dump(
- snapshot,
- open(f"oom_rank-{torch.distributed.get_rank()}_{args.memory_snapshot_path}", 'wb'),
- )
+ filename = f"oom_rank-{torch.distributed.get_rank()}_{args.memory_snapshot_path}"
+ torch.cuda.memory._dump_snapshot(filename)
torch._C._cuda_attach_out_of_memory_observer(oom_observer)
diff --git a/pretrain_bert.py b/pretrain_bert.py
index 7cda429a849..9b11908811b 100644
--- a/pretrain_bert.py
+++ b/pretrain_bert.py
@@ -8,7 +8,6 @@
import torch.nn.functional as F
from megatron.training import get_args
-from megatron.training import get_tokenizer
from megatron.training import print_rank_0
from megatron.training import get_timers
from megatron.core import tensor_parallel
@@ -20,12 +19,11 @@
from megatron.training.arguments import core_transformer_config_from_args
from megatron.core.transformer.spec_utils import import_module
from megatron.core.models.bert.bert_layer_specs import bert_layer_with_transformer_engine_spec, bert_layer_local_spec
-from megatron.core.tokenizers.text.utils.build_tokenizer import build_tokenizer
+from megatron.core.tokenizers.utils.build_tokenizer import build_tokenizer
from megatron.core.datasets.blended_megatron_dataset_builder import BlendedMegatronDatasetBuilder
from megatron.core.datasets.bert_dataset import BERTMaskedWordPieceDataset, BERTMaskedWordPieceDatasetConfig
from megatron.core.datasets.utils import get_blend_from_list
from megatron.core import mpu, tensor_parallel
-from megatron.core.tokenizers import MegatronTokenizer
def model_provider(pre_process=True, post_process=True, vp_stage=None, config=None, pg_collection=None):
@@ -147,10 +145,7 @@ def train_valid_test_datasets_provider(train_val_test_num_samples, vp_stage=None
"""Build train, valid, and test datasets."""
args = get_args()
- if args.legacy_tokenizer:
- tokenizer = get_tokenizer()
- else:
- tokenizer = build_tokenizer(args)
+ tokenizer = build_tokenizer(args)
config = BERTMaskedWordPieceDatasetConfig(
random_seed=args.seed,
diff --git a/pretrain_gpt.py b/pretrain_gpt.py
index b8c97518bb6..048b27e2a5d 100644
--- a/pretrain_gpt.py
+++ b/pretrain_gpt.py
@@ -27,16 +27,16 @@
from megatron.core.datasets.blended_megatron_dataset_builder import BlendedMegatronDatasetBuilder
from megatron.core.datasets.gpt_dataset import GPTDataset, GPTDatasetConfig, MockGPTDataset
from megatron.core.enums import ModelType
+from megatron.core.packed_seq_params import PackedSeqParams
from megatron.core.models.gpt import GPTModel
from megatron.training.arguments import get_patch_args
from megatron.core.rerun_state_machine import get_rerun_state_machine
+from megatron.core.tokenizers.utils.build_tokenizer import build_tokenizer
from megatron.core.utils import get_attr_wrapped_model, get_thd_batch_on_this_cp_rank, get_batch_on_this_hybrid_cp_rank, StragglerDetector
-from megatron.core.tokenizers.text.utils.build_tokenizer import build_tokenizer
from megatron.training import (
get_args,
get_timers,
- get_tokenizer,
inprocess_restart,
pretrain,
print_rank_0,
@@ -66,11 +66,60 @@
def get_batch(data_iterator, vp_stage: Optional[int] = None):
- """Generate a batch."""
+ """Generate a batch.
+
+ Packed sequence support (SFT / ``--sft`` flag):
+ When ``args.sft`` is True, the dataset emits THD-format batches where
+ multiple sequences are concatenated into a single flat token tensor.
+ The batch includes ``cu_seqlens`` (cumulative sequence lengths, shape
+ ``[1, S+1]``) and ``max_seqlen`` (shape ``[1]``) that describe the
+ individual sequence boundaries.
+
+ This function validates and squeezes those fields:
+ - ``cu_seqlens``: asserted to have shape ``[1, S+1]`` (micro-batch
+ size must be 1 for packing), then squeezed to ``[S+1]``.
+ - ``max_seqlen``: asserted to be 1-D; kept as a tensor and passed
+ to ``get_thd_batch_on_this_cp_rank`` which performs the final
+ scalar conversion internally.
+
+ Pipeline stage handling:
+ - First/last PP stages: fetch the full batch (tokens + labels) and
+ route through ``get_thd_batch_on_this_cp_rank`` to produce a
+ ``PackedSeqParams`` object that carries ``cu_seqlens`` and
+ ``max_seqlen`` to the attention kernel.
+ - Middle PP stages: only ``cu_seqlens`` and ``max_seqlen`` are
+ needed for attention masking; all other fields are returned as
+ ``None`` with a ``PackedSeqParams`` built directly here.
+ - MTP ranks (``mtp_on_this_rank``) also receive the full batch,
+ regardless of pipeline stage.
+
+ Difference from ``pretrain_mamba.py``:
+ - Return format: GPT returns a 6-tuple
+ ``(tokens, labels, loss_mask, attention_mask, position_ids,
+ packed_seq_params)`` where ``packed_seq_params`` is a
+ ``PackedSeqParams`` dataclass. Mamba returns 7 values via
+ ``batch.values()`` with ``cu_seqlens`` and ``max_seqlen`` as
+ separate dict entries (no ``PackedSeqParams`` wrapper).
+ - Middle-stage return: GPT returns ``(None×5, PackedSeqParams)``;
+ Mamba returns an ``empty_batch`` dict with ``cu_seqlens`` and
+ ``max_seqlen`` set.
+ - CP with packed sequences: GPT delegates to
+ ``get_thd_batch_on_this_cp_rank`` (MCore utility); Mamba
+ implements the ``tex.thd_get_partitioned_indices`` CP slicing
+ inline and does not call that helper.
+ - MTP: GPT passes ``mtp_on_this_rank`` to ``get_batch_on_this_tp_rank``
+ and uses it to gate the early-return; Mamba has no MTP support.
+ - ``max_seqlen`` conversion: Mamba converts to a Python int scalar
+ before returning (``int(max_seqlen[0].item())``); GPT keeps it as
+ a tensor and lets ``get_thd_batch_on_this_cp_rank`` convert it,
+ except for the middle-stage ``PackedSeqParams`` where conversion
+ is done inline.
+ """
args = get_args()
config = core_transformer_config_from_args(args)
# TODO: this is pretty hacky, find a better way
- if not is_first_or_last_pipeline_stage(vp_stage) and (
+ is_packed_sequence = get_args().sft # SFT always uses packed sequence
+ if not is_first_or_last_pipeline_stage(vp_stage) and not is_packed_sequence and (
(not mtp_on_this_rank(config, ignore_virtual=False, vp_stage=vp_stage))):
return None, None, None, None, None, None
@@ -87,16 +136,33 @@ def get_batch(data_iterator, vp_stage: Optional[int] = None):
if local_cp_size is not None:
local_cp_size = int(local_cp_size.item())
+ if cu_seqlens is not None:
+ assert (
+ cu_seqlens.dim() == 2 and cu_seqlens.shape[0] == 1
+ ), "micro-batch-size must be 1 for packing"
+ cu_seqlens = cu_seqlens[0]
+ assert max_seqlen.dim() == 1
+
+ # For middle pipeline stages with packed sequences, only cu_seqlens and
+ # max_seqlen are needed (for attention masking); skip the full batch.
+ if not is_first_or_last_pipeline_stage(vp_stage) and is_packed_sequence:
+ return None, None, None, None, None, PackedSeqParams(
+ cu_seqlens_q=cu_seqlens,
+ cu_seqlens_kv=cu_seqlens,
+ max_seqlen_q=int(max_seqlen[0].item()),
+ max_seqlen_kv=int(max_seqlen[0].item()),
+ qkv_format='thd',
+ )
+
if cu_seqlens is None and local_cp_size is None:
# slice batch along sequence dimension for context parallelism
batch = get_batch_on_this_cp_rank(batch) # The implementation of this function is in MCore
packed_seq_params = None
elif local_cp_size is None: # Packed THD format
- assert max_seqlen.dim() == 1
batch, packed_seq_params = get_thd_batch_on_this_cp_rank(batch, cu_seqlens, cu_seqlens_padded, max_seqlen)
else: # Hybrid CP format
batch, packed_seq_params = get_batch_on_this_hybrid_cp_rank(batch, local_cp_size)
-
+
return (*batch.values(), packed_seq_params)
@@ -205,20 +271,21 @@ def forward_step(data_iterator, model: GPTModel, return_schedule_plan: bool = Fa
return output_tensor, partial(loss_func, loss_mask, model=model)
-def is_dataset_built_on_rank(vp_stage=None):
+def is_dataset_built_on_rank(vp_stage=None, is_packed_sequence=False):
args = get_args()
config = core_transformer_config_from_args(args)
+ if parallel_state.get_tensor_model_parallel_rank() != 0:
+ return False
+ elif is_packed_sequence:
+ return True
return (
is_first_or_last_pipeline_stage(vp_stage)
or mtp_on_this_rank(config, ignore_virtual=False, vp_stage=vp_stage)
- ) and parallel_state.get_tensor_model_parallel_rank() == 0
+ )
def core_gpt_dataset_config_from_args(args):
- if args.legacy_tokenizer:
- tokenizer = get_tokenizer()
- else:
- tokenizer = build_tokenizer(args)
+ tokenizer = build_tokenizer(args)
# Sometimes --data-path is too long, instead we parse it from a file.
blend: Optional[Tuple[List[str], Optional[List[float]]]]
@@ -292,8 +359,11 @@ def train_valid_test_datasets_provider(train_val_test_num_samples, vp_stage=None
config = core_gpt_dataset_config_from_args(args)
+
+ is_packed_sequence = False
if args.sft:
dataset_type = SFTDataset
+ is_packed_sequence = True # SFT always uses packed sequence
else:
if args.mock_data:
dataset_type = MockGPTDataset
@@ -304,9 +374,9 @@ def train_valid_test_datasets_provider(train_val_test_num_samples, vp_stage=None
print_rank_0("> building train, validation, and test datasets for GPT ...")
- is_dataset_built = partial(is_dataset_built_on_rank, vp_stage=vp_stage)
+ is_dataset_built = partial(is_dataset_built_on_rank, vp_stage=vp_stage, is_packed_sequence=is_packed_sequence)
train_ds, valid_ds, test_ds = BlendedMegatronDatasetBuilder(
- dataset_type, train_val_test_num_samples, partial(is_dataset_built_on_rank, vp_stage=vp_stage), config
+ dataset_type, train_val_test_num_samples, is_dataset_built, config
).build()
print_rank_0("> finished creating GPT datasets ...")
diff --git a/pretrain_ict.py b/pretrain_ict.py
deleted file mode 100644
index bf70256bfa1..00000000000
--- a/pretrain_ict.py
+++ /dev/null
@@ -1,166 +0,0 @@
-# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
-
-"""Pretrain BERT for Inverse Cloze Task"""
-
-from functools import partial
-import math
-
-import torch
-import torch.distributed as dist
-import torch.nn.functional as F
-
-from megatron.training import get_args
-from megatron.training import print_rank_0
-from megatron.training import get_timers
-from megatron.core import mpu
-from megatron.core.enums import ModelType
-from megatron.legacy.data.biencoder_dataset_utils import get_ict_batch
-from megatron.legacy.data.dataset_utils import build_train_valid_test_datasets
-from megatron.legacy.model.biencoder_model import biencoder_model_provider
-from megatron.training import pretrain
-from megatron.training.utils import average_losses_across_data_parallel_group
-
-
-def pretrain_ict_model_provider(pre_process=True, post_process=True):
- args = get_args()
-
- model = biencoder_model_provider(
- only_context_model=False,
- only_query_model=False,
- biencoder_shared_query_context_model=\
- args.biencoder_shared_query_context_model,
- pre_process=pre_process, post_process=post_process)
-
- return model
-
-def get_group_world_size_rank():
-
- group = mpu.get_data_parallel_group()
- rank = group.rank()
- world_size = group.size()
-
- return group, rank, world_size
-
-
-class AllgatherFromDataParallelRegion(torch.autograd.Function):
-
- @staticmethod
- def forward(ctx, input_):
- assert input_.dim() == 2
- group, rank, world_size = get_group_world_size_rank()
-
- tensor_list = [torch.empty_like(input_) for _ in range(world_size)]
- tensor_list[rank] = input_
- torch.distributed.all_gather(tensor_list, input_, group=group)
-
- output = torch.cat(tensor_list, dim=0).contiguous()
-
- return output
-
-
- @staticmethod
- def backward(ctx, grad_output):
- group, rank, world_size = get_group_world_size_rank()
-
- assert grad_output.shape[0] % world_size == 0
- dim_size = grad_output.shape[0] // world_size
- output_list = torch.split(grad_output, dim_size, dim=0)
-
- # get chunk from this rank
- output = output_list[rank].contiguous()
- return output
-
-def loss_func(output_tensor):
- args = get_args()
- query_logits, context_logits = output_tensor
-
- micro_batch_size = query_logits.shape[0]
- # recall we assert that tensor_model_parallel_size == 1
- assert mpu.get_tensor_model_parallel_world_size() == 1, \
- "Model parallel size > 1 not supported for ICT"
-
- global_batch_size = dist.get_world_size() * micro_batch_size
- all_query_logits = AllgatherFromDataParallelRegion.apply(query_logits)
- all_context_logits = AllgatherFromDataParallelRegion.apply(context_logits)
-
- # scores are inner products between query and context embeddings
- retrieval_scores = torch.matmul(all_query_logits,
- torch.transpose(all_context_logits, 0, 1))
- # scaling the retriever scores
- if args.retriever_score_scaling:
- retrieval_scores = retrieval_scores / math.sqrt(args.hidden_size)
-
- softmax_scores = F.log_softmax(retrieval_scores, dim=1)
- sorted_vals, sorted_indices = torch.topk(softmax_scores,
- k=softmax_scores.shape[1], sorted=True)
-
- def topk_accuracy(k):
- return torch.cuda.FloatTensor([sum([int(i in sorted_indices[i, :k]) \
- for i in range(global_batch_size)]) / global_batch_size])
-
- topk_accs = [topk_accuracy(int(k)) for k in args.retriever_report_topk_accuracies]
-
- labels = torch.arange(global_batch_size).long().cuda()
- loss = F.nll_loss(softmax_scores, labels, reduction='mean')
- reduced_losses = average_losses_across_data_parallel_group([loss, *topk_accs])
-
- # Scale the retrieval loss
- loss = loss * mpu.get_data_parallel_world_size()
-
- # create stats_dict with retrieval loss and all specified top-k accuracies
- topk_acc_dict = {'top{}_acc'.format(k): v * 100 for k, v in \
- zip(args.retriever_report_topk_accuracies, reduced_losses[1:])}
- stats_dict = dict(loss=reduced_losses[0], **topk_acc_dict)
- return loss, stats_dict
-
-
-
-def forward_step(data_iterator, model):
- """Forward step."""
- args = get_args()
- timers = get_timers()
-
- # Get the batch.
- timers('batch-generator', log_level=2).start()
- query_tokens, query_mask, \
- context_tokens, context_mask, context_indices = get_ict_batch(data_iterator)
- timers('batch-generator').stop()
-
- # Query and Context Types
- query_types = torch.cuda.LongTensor(*query_tokens.shape).fill_(0)
- context_types = torch.cuda.LongTensor(*context_tokens.shape).fill_(0)
-
- # Forward model.
- output_tensor = model(query_tokens, query_mask, query_types, context_tokens,
- context_mask, context_types)
-
- return output_tensor, partial(loss_func)
-
-def train_valid_test_datasets_provider(train_val_test_num_samples):
- """Build train, valid and test datasets."""
- args = get_args()
- print_rank_0('> building train, validation, and test datasets '
- 'for BERT ICT...')
-
- train_ds, valid_ds, test_ds = build_train_valid_test_datasets(
- data_prefix=args.data_path,
- splits_string=args.split,
- train_valid_test_num_samples=train_val_test_num_samples,
- max_seq_length=args.seq_length,
- masked_lm_prob=args.mask_prob,
- short_seq_prob=args.short_seq_prob,
- seed=args.seed,
- binary_head=False,
- dataset_type='ict')
- print_rank_0("> finished creating BERT ICT datasets ...")
-
- return train_ds, valid_ds, test_ds
-
-
-if __name__ == "__main__":
- print_rank_0("WARNING : This script is DEPRECATED. Will be removed in mcore release 0.9")
- pretrain(train_valid_test_datasets_provider,
- pretrain_ict_model_provider,
- ModelType.encoder_or_decoder,
- forward_step,
- args_defaults={'tokenizer_type': 'BertWordPieceLowerCase'})
diff --git a/pretrain_mamba.py b/pretrain_mamba.py
index c41c485c866..a30979af714 100644
--- a/pretrain_mamba.py
+++ b/pretrain_mamba.py
@@ -32,12 +32,11 @@
)
from megatron.core.models.mamba import MambaModel
from megatron.core.rerun_state_machine import get_rerun_state_machine
-from megatron.core.tokenizers.text.utils.build_tokenizer import build_tokenizer
+from megatron.core.tokenizers.utils.build_tokenizer import build_tokenizer
from megatron.core.utils import get_attr_wrapped_model, is_te_min_version, StragglerDetector
from megatron.training import (
get_args,
get_timers,
- get_tokenizer,
inprocess_restart,
pretrain,
print_rank_0,
@@ -237,7 +236,7 @@ def forward_step(data_iterator, model: MambaModel):
if cu_seqlens is None:
packed_seq_params = None
else:
- # TODO(duncan): This class seems overly complex for what needs to be conveyed
+ total_tokens = tokens.size(1) if tokens is not None else labels.size(1)
packed_seq_params = PackedSeqParams(
qkv_format="thd",
cu_seqlens_q=cu_seqlens,
@@ -246,6 +245,7 @@ def forward_step(data_iterator, model: MambaModel):
cu_seqlens_kv_padded=None,
max_seqlen_q=max_seqlen,
max_seqlen_kv=max_seqlen,
+ total_tokens=total_tokens,
)
timers('batch-generator').stop()
@@ -274,10 +274,7 @@ def is_dataset_built_on_rank(vp_stage=None, is_packed_sequence=False):
def core_gpt_dataset_config_from_args(args):
- if args.legacy_tokenizer:
- tokenizer = get_tokenizer()
- else:
- tokenizer = build_tokenizer(args)
+ tokenizer = build_tokenizer(args)
# Sometimes --data-path is too long, instead we parse it from a file.
blend: Optional[Tuple[List[str], Optional[List[float]]]]
diff --git a/pretrain_t5.py b/pretrain_t5.py
index 2badae9d5ee..f849a5098c3 100644
--- a/pretrain_t5.py
+++ b/pretrain_t5.py
@@ -10,7 +10,7 @@
import megatron
from megatron.core import mpu, tensor_parallel
-from megatron.core.tokenizers.text.utils.build_tokenizer import build_tokenizer
+from megatron.core.tokenizers.utils.build_tokenizer import build_tokenizer
from megatron.core.datasets.blended_megatron_dataset_builder import BlendedMegatronDatasetBuilder
from megatron.core.datasets.t5_dataset import (
T5MaskedWordPieceDataset,
@@ -25,9 +25,8 @@
get_t5_encoder_with_local_block_spec,
get_t5_encoder_with_transformer_engine_block_spec,
)
-from megatron.training import get_args, get_tokenizer, get_timers, pretrain, print_rank_0
+from megatron.training import get_args, get_timers, pretrain, print_rank_0
from megatron.training.arguments import core_transformer_config_from_args
-from megatron.core.tokenizers import MegatronTokenizer
from pretrain_gpt import loss_func
"""
@@ -213,10 +212,7 @@ def train_valid_test_datasets_provider(train_val_test_num_samples: int):
"""
args = get_args()
- if args.legacy_tokenizer:
- tokenizer = get_tokenizer()
- else:
- tokenizer = build_tokenizer(args)
+ tokenizer = build_tokenizer(args)
config = T5MaskedWordPieceDatasetConfig(
random_seed=args.seed,
diff --git a/pretrain_vision_classify.py b/pretrain_vision_classify.py
deleted file mode 100644
index 8d9b28baeb9..00000000000
--- a/pretrain_vision_classify.py
+++ /dev/null
@@ -1,105 +0,0 @@
-# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
-
-"""Pretrain VIT"""
-
-import torch
-import torch.nn.functional as F
-from functools import partial
-from megatron.training import get_args, get_timers, print_rank_0
-from megatron.core.enums import ModelType
-from megatron.legacy.data.vit_dataset import build_train_valid_datasets
-from megatron.legacy.model.vision.classification import VitClassificationModel
-from megatron.legacy.model.vision.classification import MitClassificationModel
-from megatron.training import pretrain
-from megatron.training.utils import average_losses_across_data_parallel_group
-from megatron.training.arguments import core_transformer_config_from_args
-
-
-def model_provider(pre_process=True, post_process=True):
- """Build the model."""
-
- args = get_args()
- config = core_transformer_config_from_args(args)
- if args.vision_backbone_type == 'vit':
- print_rank_0("building VIT model ...")
- model = VitClassificationModel(config=config,
- num_classes=args.num_classes,
- pre_process=pre_process,
- post_process=post_process)
- elif args.vision_backbone_type == 'mit':
- print_rank_0("building MIT model ...")
- model = MitClassificationModel(num_classes=args.num_classes,
- pre_process=pre_process,
- post_process=post_process)
- else:
- raise Exception('{} vision backbone is not supported.'.format(
- args.vision_backbone_type))
- return model
-
-
-def get_batch(data_iterator):
- """Build the batch."""
- data = next(data_iterator)
-
- # only data parallelism; no need for broadcast
- images = data[0].cuda()
- labels = data[1].cuda()
-
- return images, labels
-
-
-def loss_func(labels, output_tensor):
- logits = output_tensor.contiguous().float()
- loss = F.cross_entropy(logits, labels)
-
- outputs = torch.argmax(logits, -1)
- correct = (outputs == labels).float()
- accuracy = torch.mean(correct)
-
- averaged_loss = average_losses_across_data_parallel_group([loss, accuracy])
-
- return loss, {"loss": averaged_loss[0], "accuracy": averaged_loss[1]}
-
-
-def forward_step(data_iterator, model):
- """Forward step."""
- timers = get_timers()
-
- # Get the batch.
- timers("batch-generator", log_level=2).start()
- (
- images,
- labels,
- ) = get_batch(data_iterator)
- timers("batch-generator").stop()
-
- # Forward model. lm_labels
- output_tensor = model(images)
-
- return output_tensor, partial(loss_func, labels)
-
-def train_valid_test_datasets_provider(train_val_test_num_samples):
- """Build train, valid, and test datasets."""
- args = get_args()
-
- print_rank_0(
- "> building train, validation, and test datasets " "for VIT ..."
- )
- train_ds, valid_ds = build_train_valid_datasets(
- data_path=args.data_path,
- image_size=(args.img_h, args.img_w)
- )
- print_rank_0("> finished creating VIT datasets ...")
-
- return train_ds, valid_ds, None
-
-
-if __name__ == "__main__":
-
- pretrain(
- train_valid_test_datasets_provider,
- model_provider,
- ModelType.encoder_or_decoder,
- forward_step,
- args_defaults={'dataloader_type': 'cyclic', 'vision_pretraining': True}
- )
diff --git a/pretrain_vision_dino.py b/pretrain_vision_dino.py
deleted file mode 100644
index f75280c42d7..00000000000
--- a/pretrain_vision_dino.py
+++ /dev/null
@@ -1,105 +0,0 @@
-# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
-
-import torch
-import torch.nn.functional as F
-import torch.nn as nn
-import numpy as np
-import torch.distributed as dist
-from functools import partial
-from megatron.training import get_args, get_timers, print_rank_0
-from megatron.core.enums import ModelType
-from megatron.legacy.data.vit_dataset import build_train_valid_datasets
-from megatron.legacy.model.vision.dino import DINOPretrainModel
-from megatron.legacy.model.vision.knn_monitor import knn_predict, get_feature_bank
-from megatron.training import pretrain
-from megatron.training.utils import average_losses_across_data_parallel_group, unwrap_model
-from megatron.training.arguments import core_transformer_config_from_args
-
-def model_provider(pre_process=True, post_process=True):
- """Build the model."""
- config = core_transformer_config_from_args(get_args())
- return DINOPretrainModel(config, pre_process=pre_process, post_process=post_process)
-
-def get_batch(data_iterator):
- """Build the batch."""
- data = next(data_iterator)
-
- # only data parallelism; no need for broadcast
- if isinstance(data[0], list):
- images = [aug.cuda() for aug in data[0]]
- else:
- images = data[0].cuda()
- labels = data[1].cuda()
-
- return images, labels
-
-
-def loss_func(model, labels, output_tensor, collect_data=False):
- args = get_args()
-
- model = unwrap_model(model)
- if model.training:
- student_output, teacher_output = output_tensor
- loss = model.dino_loss(student_output, teacher_output, args.curr_iteration)
- averaged_loss = average_losses_across_data_parallel_group([loss])
- return loss, {"loss": averaged_loss[0]}
- else:
- _, teacher_feature = output_tensor
- feature_bank, feature_labels, classes = get_feature_bank()
- feature = F.normalize(teacher_feature.float(), dim=1)
-
- knn_accs = []
- for k in [10, 20, 100, 200]:
- pred_labels = knn_predict(feature, feature_bank,
- feature_labels, classes, k, 0.07)
- knn_acc = (pred_labels[:, 0] == labels).float().mean()
- knn_accs.append(knn_acc)
-
- averaged_loss = average_losses_across_data_parallel_group(knn_accs)
- return 0, {"knn_acc_10": averaged_loss[0],
- "knn_acc_20": averaged_loss[1],
- "knn_acc_100": averaged_loss[2],
- "knn_acc_200": averaged_loss[3]}
-
-
-def forward_step(data_iterator, model):
- """Forward step."""
- timers = get_timers()
-
- # Get the batch.
- timers("batch-generator", log_level=2).start()
- (
- images,
- labels,
- ) = get_batch(data_iterator)
- timers("batch-generator").stop()
-
- return model(images), partial(loss_func, model, labels)
-
-
-def train_valid_test_datasets_provider(train_val_test_num_samples):
- """Build train, valid, and test datasets."""
- args = get_args()
-
- print_rank_0(
- "> building train, validation, and test datasets " "for VIT ..."
- )
- train_ds, valid_ds = build_train_valid_datasets(
- data_path=args.data_path,
- image_size=(args.img_h, args.img_w)
- )
- print_rank_0("> finished creating VIT datasets ...")
-
- return train_ds, valid_ds, None
-
-
-if __name__ == "__main__":
-
- pretrain(
- train_valid_test_datasets_provider,
- model_provider,
- ModelType.encoder_or_decoder,
- forward_step,
- args_defaults={'dataloader_type': 'cyclic', 'vision_pretraining': True}
- )
-
diff --git a/pretrain_vision_inpaint.py b/pretrain_vision_inpaint.py
deleted file mode 100644
index 8570baab5b4..00000000000
--- a/pretrain_vision_inpaint.py
+++ /dev/null
@@ -1,141 +0,0 @@
-# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
-
-"""Pretrain VIT"""
-
-import torch
-import torch.nn.functional as F
-from functools import partial
-from megatron.training import get_args, get_timers, print_rank_0, print_rank_last
-from megatron.core.enums import ModelType
-from megatron.legacy.data.vit_dataset import build_train_valid_datasets
-from megatron.legacy.model.vision.inpainting import VitInpaintingModel
-from megatron.legacy.model.vision.inpainting import MitInpaintingModel
-from megatron.training import pretrain
-from megatron.training.utils import average_losses_across_data_parallel_group
-from tasks.vision.segmentation.metrics import SSIM, PSNR
-from megatron.training.arguments import core_transformer_config_from_args
-
-def model_provider(pre_process=True, post_process=True):
- """Build the model."""
- args = get_args()
- config = core_transformer_config_from_args(args)
- if args.vision_backbone_type == 'vit':
- model = VitInpaintingModel(config=config,
- pre_process=pre_process,
- post_process=post_process)
- elif args.vision_backbone_type == 'mit':
- model = MitInpaintingModel(config=config,
- pre_process=pre_process,
- post_process=post_process)
- else:
- raise Exception('{} vision backbone is not supported.'.format(
- args.vision_backbone_type))
- return model
-
-
-def get_batch(data_iterator):
- """Build the batch."""
- data = next(data_iterator)
-
- # only data parallelism; no need for broadcast
- images = data[0][0].cuda()
- masks = data[0][1].cuda()
- return images, masks
-
-
-def loss_func(images, masks, masked_images, outputs, non_loss_data=False):
- outputs = outputs.contiguous().float()
- masks_flip = 1-masks
- flip_masked_outputs = outputs.masked_fill(masks_flip.bool(), 0)
- flip_masked_images = images.masked_fill(masks_flip.bool(), 0)
-
- ssim_fun = SSIM()
- psnr_fun = PSNR()
-
- if not non_loss_data:
- mask_count = torch.count_nonzero(masks)
- loss = F.mse_loss(
- flip_masked_outputs,
- flip_masked_images.float(),
- reduction="sum"
- )
- loss = loss/mask_count
- ssim = ssim_fun(flip_masked_outputs, flip_masked_images.float())
- psnr = psnr_fun(flip_masked_outputs, flip_masked_images.float())
-
- averaged_loss = average_losses_across_data_parallel_group(
- [loss, psnr, ssim]
- )
-
- return loss, {"loss": averaged_loss[0],
- "psnr": averaged_loss[1],
- 'ssim': averaged_loss[2]}
- else:
- synth_images = masked_images.float() + flip_masked_outputs
- ssim = ssim_fun(synth_images, images.float())
- psnr = psnr_fun(synth_images, images.float())
- return torch.cat((images, masked_images, synth_images), dim=2), ssim, psnr
-
-
-def forward_step(data_iterator, model):
- """Forward step."""
- timers = get_timers()
-
- # Get the batch.
- timers("batch-generator", log_level=2).start()
- (
- images,
- masks,
- ) = get_batch(data_iterator)
- timers("batch-generator").stop()
-
- masked_images = images.masked_fill(masks.bool(), 0)
- outputs = model(masked_images)
-
- # Forward mode
- return outputs, partial(loss_func, images, masks, masked_images)
-
-
-def process_non_loss_data(data, iteration, writer):
- psnr_sum = 0
- ssim_sum = 0
- for (output_tb, ssim, psnr) in data:
- output_tb[output_tb < 0] = 0
- output_tb[output_tb > 1] = 1
- writer.add_images("gt-input-output-vald", output_tb,
- global_step=iteration, walltime=None,
- dataformats='NCHW')
- psnr_sum = psnr_sum + psnr.item()
- ssim_sum = ssim_sum + ssim.item()
- psnr = psnr_sum/len(data)
- ssim = ssim_sum/len(data)
- writer.add_scalar('PSNR generate value-validation', psnr, iteration)
- writer.add_scalar('SSIM generate value-validation', ssim, iteration)
-
-
-def train_valid_test_datasets_provider(train_val_test_num_samples):
- """Build train, valid, and test datasets."""
- args = get_args()
-
- print_rank_0(
- "> building train, validation, and test datasets " "for VIT ..."
- )
- train_ds, valid_ds = build_train_valid_datasets(
- data_path=args.data_path,
- image_size=(args.img_h, args.img_w)
- )
- print_rank_0("> finished creating VIT datasets ...")
-
- return train_ds, valid_ds, None
-
-
-if __name__ == "__main__":
-
- pretrain(
- train_valid_test_datasets_provider,
- model_provider,
- ModelType.encoder_or_decoder,
- forward_step,
- process_non_loss_data,
- args_defaults={'dataloader_type': 'cyclic', 'vision_pretraining': True}
- )
diff --git a/pyproject.toml b/pyproject.toml
index 7ffa8a83ebf..d3c11e58a92 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -8,7 +8,12 @@ build-backend = "setuptools.build_meta"
include-package-data = true
[tool.setuptools.packages.find]
-include = ["megatron.core", "megatron.core.*"]
+include = [
+ "megatron.core",
+ "megatron.core.*",
+ "megatron.training",
+ "megatron.training.*",
+]
[tool.setuptools.dynamic]
version = { attr = "megatron.core.package_info.__version__" }
@@ -18,7 +23,7 @@ readme = { file = "README.md", content-type = "text/markdown" }
name = "megatron-core"
dynamic = ["version", "readme"]
description = "Megatron Core - a library for efficient and scalable training of transformer based models"
-requires-python = ">=3.10"
+requires-python = ">=3.12"
license = { text = "Apache 2.0" }
dependencies = ["torch>=2.6.0", "numpy", "packaging>=24.2", "regex"]
authors = [{ name = "NVIDIA", email = "nemo-toolkit@nvidia.com" }]
@@ -30,7 +35,6 @@ keywords = [
"gpu",
"language",
"learning",
- "learning",
"machine",
"nvidia",
"pytorch",
@@ -61,14 +65,31 @@ classifiers = [
[project.urls]
Download = "https://github.com/NVIDIA/Megatron-LM/releases"
-Homepage = "https://github.com/NVIDIA/Megatron-LM/megatron/core"
+Homepage = "https://github.com/NVIDIA/Megatron-LM"
[project.optional-dependencies]
-mlm = ["flask-restful", "sentencepiece", "tiktoken", "wandb", "transformers"]
+training = [
+ "flask-restful",
+ "sentencepiece",
+ "tiktoken",
+ "wandb",
+ "transformers",
+ "accelerate",
+]
+
+### 'mlm' group is deprecated. please use 'training' instead ###
+mlm = [
+ "flask-restful",
+ "sentencepiece",
+ "tiktoken",
+ "wandb",
+ "transformers",
+ "accelerate",
+]
dev = [
"nvidia-modelopt[torch]; sys_platform != 'darwin'",
- "transformer-engine[pytorch,core_cu13]>=2.9.0a0,<2.12.0",
+ "transformer-engine[pytorch,core_cu13]",
"nvidia-resiliency-ext",
"tqdm",
"einops~=0.8",
@@ -79,15 +100,18 @@ dev = [
"mamba-ssm~=2.2",
"causal-conv1d~=1.5",
"flash-linear-attention~=0.4.0",
- "nv-grouped-gemm~=1.1",
"megatron-energon[av_decode]~=6.0",
"av",
"flashinfer-python~=0.5.0",
"wget",
"onnxscript",
- "fastapi~=0.50", # Forcing a little bit more recent version of fastapi to be compatible with pydantic 2.0
+ "fastapi~=0.50", # Forcing a little bit more recent version of fastapi to be compatible with pydantic 2.0
"datasets",
"emerging_optimizers",
+ "hypercorn",
+ "quart",
+ "openai[aiohttp]",
+ "orjson",
]
lts = [
@@ -99,7 +123,6 @@ lts = [
"opentelemetry-api~=1.33.1",
"mamba-ssm~=2.2",
"causal-conv1d~=1.5",
- "nv-grouped-gemm~=1.1",
"megatron-energon[av_decode]~=6.0",
"av",
"flashinfer-python~=0.5.0",
@@ -117,6 +140,7 @@ test = [
"wrapt",
"pytest==8.3.5",
"pytest-mock",
+ "mock",
"pytest-cov",
"pytest-random-order",
"pytest-asyncio",
@@ -157,7 +181,6 @@ no_pypi_wheels = ["flash_mla", "emerging_optimizers"]
default-groups = ["linting", "build", "test"]
no-build-isolation-package = [
"causal-conv1d",
- "nv-grouped-gemm",
"flash_mla",
"mamba-ssm",
"transformer-engine",
@@ -179,14 +202,15 @@ override-dependencies = [
flash_mla = [
{ git = "https://github.com/deepseek-ai/FlashMLA", rev = "9edee0c022cd0938148a18e334203b0aab43aa19" },
]
-transformer-engine = { git = "https://github.com/NVIDIA/TransformerEngine.git", rev = "d9b7fc5770a88af06e2e9c2bd97b550614c3a69f" }
+transformer-engine = { git = "https://github.com/NVIDIA/TransformerEngine.git", rev = "71bbefbf153418f943640df0f7373625dc93fa46" }
nemo-run = { git = "https://github.com/NVIDIA-NeMo/Run.git", rev = "01a9a8ba360f7b2908728ad0516e0ad9d936966d" }
emerging_optimizers = { git = "https://github.com/NVIDIA-NeMo/Emerging-Optimizers.git", rev = "v0.1.0" }
+nvidia-resiliency-ext = { git = "https://github.com/NVIDIA/nvidia-resiliency-ext.git", rev = "63154570cea17f8805a7fd15cc3b8cc2919ba575" }
[tool.isort]
profile = "black" # black-compatible
line_length = 100 # should match black parameters
-py_version = 310 # python 3.8 as a target version
+py_version = 310 # python 3.10 as a target version
known_first_party = ["megatron"] # FIRSTPARTY section
known_third_party = ["transformer_engine"] # THIRDPARTY section
sections = ["FUTURE", "STDLIB", "THIRDPARTY", "FIRSTPARTY", "LOCALFOLDER"]
@@ -196,7 +220,7 @@ extend_skip = ["setup.py"]
[tool.black]
line_length = 100
skip_string_normalization = true
-# recongized by future versions, disallows to reformat code with incompatible versions
+# recognized by future versions, disallows to reformat code with incompatible versions
# Matches NeMO version so people working on both codebases don't need two different version of black installed
required_version = "24"
skip_magic_trailing_comma = true
@@ -228,8 +252,10 @@ markers = [
concurrency = ["thread", "multiprocessing"]
omit = [
"/tmp/*",
- "/workspace/tests/*",
+ "/opt/megatron-lm/tests/*",
+ "/opt/megatron-lm/tools/*",
"/usr/local/lib/python3.12/dist-packages/*",
+ "/opt/megatron-lm/_remote_module_non_scriptable",
]
parallel = true
sigterm = false
diff --git a/run_unit_tests.sh b/run_unit_tests.sh
index ecd94d0e9bc..57a201844fe 100755
--- a/run_unit_tests.sh
+++ b/run_unit_tests.sh
@@ -3,9 +3,6 @@
set -u -o pipefail
set -x
-# Install mock for unit tests
-pip install mock
-
NUM_GPUS=$(python -c "import torch; print(torch.cuda.device_count())")
export HIP_VISIBLE_DEVICES=$(seq -s, 0 $((NUM_GPUS-1)))
echo "Number of GPUs: $NUM_GPUS"
diff --git a/setup.py b/setup.py
index df857cadc41..4ba661321c3 100644
--- a/setup.py
+++ b/setup.py
@@ -1,22 +1,16 @@
-import subprocess
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-from setuptools import Extension, setup
+from pybind11.setup_helpers import Pybind11Extension
+from setuptools import setup
-setup_args = dict(
+setup(
ext_modules=[
- Extension(
+ Pybind11Extension(
"megatron.core.datasets.helpers_cpp",
sources=["megatron/core/datasets/helpers.cpp"],
language="c++",
- extra_compile_args=(
- subprocess.check_output(["python3", "-m", "pybind11", "--includes"])
- .decode("utf-8")
- .strip()
- .split()
- )
- + ["-O3", "-Wall", "-std=c++17"],
+ extra_compile_args=["-O3", "-Wall", "-std=c++17"],
optional=True,
)
]
)
-setup(**setup_args)
diff --git a/tests/README.md b/tests/README.md
new file mode 100644
index 00000000000..b412eb38934
--- /dev/null
+++ b/tests/README.md
@@ -0,0 +1,18 @@
+# Megatron-LM Tests
+
+## Updating Functional Test Golden Values
+
+When adding new functional tests, it may be necessary to update the golden values used to verify if the test is
+passing as expected.
+
+1. Add the new functional test case with the scope set to `mr-github`
+2. Open a PR with the new test. Ensure the label `Run functional tests` is added
+3. Run the PR CI tests
+4. Run the script to download golden values from a Github CI run
+ a. Ensure click, requests, and python-gitlab are installed in your environment
+ b. Ensure a Github access token is set as an environment variable `GITHUB_TOKEN`
+ c. Run the script `python tests/test_utils/python_scripts/download_golden_values.py --source github --pipeline-id `
+ d. Optionally pass in `--only-failing` to only download golden values for failing tests only
+ e. Ensure you are only checking-in golden values for tests are you updating
+
+The Github CI infra may not be appropriate for Perf tests. Perf tests may be more appropriate for nightly jobs on other infra.
diff --git a/tests/functional_tests/python_test_utils/test_grpo_training_loop.py b/tests/functional_tests/python_test_utils/test_grpo_training_loop.py
index 6faca9b11b3..b4447f5f761 100644
--- a/tests/functional_tests/python_test_utils/test_grpo_training_loop.py
+++ b/tests/functional_tests/python_test_utils/test_grpo_training_loop.py
@@ -84,9 +84,8 @@ def test_grpo_training_loop(
with open(model_config_path, 'r') as f:
model_config = yaml.safe_load(f)
metrics = model_config["METRICS"]
- if "THROUGHPUT_TEST_PARAMS" in model_config:
- throughput_test_params = model_config["THROUGHPUT_TEST_PARAMS"]
- start_step = throughput_test_params["--start_step"]
+ if "ENV_VARS" in model_config and "THROUGHPUT_START_STEP" in model_config["ENV_VARS"]:
+ start_step = model_config["ENV_VARS"]["THROUGHPUT_START_STEP"]
else:
start_step = 1
diff --git a/tests/functional_tests/python_test_utils/test_inference_regular_pipeline.py b/tests/functional_tests/python_test_utils/test_inference_regular_pipeline.py
index 346b464b79d..165e17b102b 100644
--- a/tests/functional_tests/python_test_utils/test_inference_regular_pipeline.py
+++ b/tests/functional_tests/python_test_utils/test_inference_regular_pipeline.py
@@ -3,14 +3,19 @@
import json
import logging
import math
+import os
from statistics import median
+import pytest
+import yaml
+
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
_NON_REQUEST_TOP_LEVEL_KEYS = {
# System-level metrics
"throughput",
+ "lifetime_prefill_token_count",
# Peak memory metrics (added by inference scripts; optionally checked if present in golden values)
"mem-max-allocated-bytes",
}
@@ -35,11 +40,25 @@ def _bytes_to_gib(num_bytes: float) -> float:
return float(num_bytes) / (1024.0**3)
-def test_inference_pipeline(golden_values_path: str, test_values_path: str) -> None:
+def test_inference_pipeline(
+ golden_values_path: str, test_values_path: str, model_config_path: str
+) -> None:
+ if os.getenv("ENABLE_LIGHTWEIGHT_MODE") == "true":
+ pytest.skip("Lightweight mode enabled. Skipping test.")
- with open(golden_values_path, 'r') as f1, open(test_values_path, 'r') as f2:
+ with (
+ open(golden_values_path, 'r') as f1,
+ open(test_values_path, 'r') as f2,
+ open(model_config_path, 'r') as f3,
+ ):
golden_values_content = f1.read()
tensorboard_content = f2.read()
+ model_config_content = f3.read()
+
+ metrics = yaml.safe_load(model_config_content)["METRICS"]
+ if not metrics:
+ print("No metrics defined in model_config.yaml, skipping validation.")
+ return
output_groundtruth = json.loads(golden_values_content)
@@ -68,57 +87,69 @@ def test_inference_pipeline(golden_values_path: str, test_values_path: str) -> N
# Throughput assertions.
if "throughput" in output_groundtruth.keys():
+ if "throughput" in metrics:
+ # First warmup iteration is excluded from throughput statistics.
+ throughput_sampled = median(output_current["throughput"][1:])
+ throughput_golden = median(output_groundtruth["throughput"][1:])
- # First warmup iteration is excluded from throughput statistics.
- throughput_sampled = median(output_current["throughput"][1:])
- throughput_golden = median(output_groundtruth["throughput"][1:])
-
- # 10% is empirically observed to be within hardware variance.
- assert (
- throughput_sampled >= 0.9 * throughput_golden
- ), f"Throughput is slower than expected! Expected to be within 10% of ~{throughput_golden} tok/s but benchmarked {output_current['throughput']} tok/s"
+ # 10% is empirically observed to be within hardware variance.
+ assert (
+ throughput_sampled >= 0.9 * throughput_golden
+ ), f"Throughput is slower than expected! Expected to be within 10% of ~{throughput_golden} tok/s but benchmarked {output_current['throughput']} tok/s"
- # If throughput is significantly improved (> 20%), update golden values accordingly.
- assert (
- throughput_sampled < throughput_golden * 1.2
- ), f"Throughput has been improved from expected ~{throughput_golden} tok/s to {output_current['throughput']} tok/s. Please update golden values in the functional tests."
+ # If throughput is significantly improved (> 20%), update golden values accordingly.
+ assert (
+ throughput_sampled < throughput_golden * 1.2
+ ), f"Throughput has been improved from expected ~{throughput_golden} tok/s to {output_current['throughput']} tok/s. Please update golden values in the functional tests."
output_groundtruth.pop('throughput')
# Peak memory regression checks (optional: only if present in golden values).
if "mem-max-allocated-bytes" in output_groundtruth:
- assert "mem-max-allocated-bytes" in output_current, (
- f"Golden values include mem-max-allocated-bytes but current output does not. "
- "Ensure the inference script records memory metrics to the output JSON."
- )
- sampled = _median_as_float(output_current["mem-max-allocated-bytes"])
- golden = _median_as_float(output_groundtruth["mem-max-allocated-bytes"])
- assert golden > 0, f"Golden mem_max_allocated_bytes must be > 0, got {golden}."
-
- low = 0.95 * golden
- high = 1.05 * golden
-
- if sampled < low:
- raise AssertionError(
- f"Memory is too low for mem-max-allocated-bytes: "
- f"expected within 5% of {golden:.0f} bytes ({_bytes_to_gib(golden):.3f} GiB) "
- f"but got {sampled:.0f} bytes ({_bytes_to_gib(sampled):.3f} GiB). "
- "This is >5% lower than expected; please update golden values in the functional tests."
- )
- if sampled > high:
- raise AssertionError(
- f"Memory is too high for mem-max-allocated-bytes: "
- f"expected within ±5% of {golden:.0f} bytes ({_bytes_to_gib(golden):.3f} GiB) "
- f"but got {sampled:.0f} bytes ({_bytes_to_gib(sampled):.3f} GiB). "
- "This is >5% higher than expected; this is likely a regression."
+ if "mem-max-allocated-bytes" in metrics:
+ assert "mem-max-allocated-bytes" in output_current, (
+ f"Golden values include mem-max-allocated-bytes but current output does not. "
+ "Ensure the inference script records memory metrics to the output JSON."
)
+ sampled = _median_as_float(output_current["mem-max-allocated-bytes"])
+ golden = _median_as_float(output_groundtruth["mem-max-allocated-bytes"])
+ assert golden > 0, f"Golden mem_max_allocated_bytes must be > 0, got {golden}."
+
+ low = 0.95 * golden
+ high = 1.05 * golden
+
+ if sampled < low:
+ raise AssertionError(
+ f"Memory is too low for mem-max-allocated-bytes: "
+ f"expected within 5% of {golden:.0f} bytes ({_bytes_to_gib(golden):.3f} GiB) "
+ f"but got {sampled:.0f} bytes ({_bytes_to_gib(sampled):.3f} GiB). "
+ "This is >5% lower than expected; please update golden values in the functional tests."
+ )
+ if sampled > high:
+ raise AssertionError(
+ f"Memory is too high for mem-max-allocated-bytes: "
+ f"expected within ±5% of {golden:.0f} bytes ({_bytes_to_gib(golden):.3f} GiB) "
+ f"but got {sampled:.0f} bytes ({_bytes_to_gib(sampled):.3f} GiB). "
+ "This is >5% higher than expected; this is likely a regression."
+ )
output_groundtruth.pop("mem-max-allocated-bytes")
+ lptc_key = "lifetime_prefill_token_count"
+ if lptc_key in output_groundtruth and lptc_key not in metrics:
+ # metrics does not have lifetime_prefill_token_count, so ignore it
+ output_groundtruth.pop(lptc_key)
+ elif lptc_key in metrics:
+ # Ground truth does not have lifetime_prefill_token_count, so ignore it
+ metrics.pop(lptc_key)
+ elif lptc_key in output_groundtruth and lptc_key in metrics:
+ # TODO: Compare liftime_prefill_token_count to groundtruth
+ pass
+
for request_id, groundtruth_results in output_groundtruth.items():
current_results = output_current[request_id]
at_least_one_test_loop = False
- if "generated_tokens" in groundtruth_results:
+ if "generated_tokens" in groundtruth_results and "generated_tokens" in metrics:
at_least_one_test_loop = True
tokens_groundtruth = groundtruth_results["generated_tokens"]
tokens_current = current_results["generated_tokens"]
@@ -127,7 +158,7 @@ def test_inference_pipeline(golden_values_path: str, test_values_path: str) -> N
tokens_groundtruth == tokens_current
), f"Token mismatch:\nGround truth: {tokens_groundtruth}\nCurrent: {tokens_current}"
- if "logprobs" in groundtruth_results:
+ if "logprobs" in groundtruth_results and "logprobs" in metrics:
at_least_one_test_loop = True
logprobs_groundtruth = groundtruth_results["logprobs"]
logprobs_current = current_results["logprobs"]
@@ -141,7 +172,7 @@ def test_inference_pipeline(golden_values_path: str, test_values_path: str) -> N
lp1, lp2, abs_tol=0.001
), f"Logprobs differ at index {i}: {lp1:.5f} vs {lp2:.5f}"
- if "generated_text" in groundtruth_results:
+ if "generated_text" in groundtruth_results and "generated_text" in metrics:
at_least_one_test_loop = True
generated_text_groundtruth = groundtruth_results["generated_text"]
generated_text_current = current_results["generated_text"]
@@ -156,5 +187,13 @@ def test_inference_pipeline(golden_values_path: str, test_values_path: str) -> N
f"\nCurrent (truncated to {min_len} chars): {generated_text_current[:min_len]}"
)
+ if "routing_indices" in groundtruth_results and "routing_indices" in metrics:
+ at_least_one_test_loop = True
+ routing_indices_groundtruth = groundtruth_results["routing_indices"]
+ routing_indices_current = current_results["routing_indices"]
+ assert (
+ routing_indices_groundtruth == routing_indices_current
+ ), f"Routing indices mismatch:\nGround truth: {routing_indices_groundtruth}\nCurrent: {routing_indices_current}"
+
if not at_least_one_test_loop:
raise AssertionError(f"No test performed for output {groundtruth_results}")
diff --git a/tests/functional_tests/shell_test_utils/_run_training.sh b/tests/functional_tests/shell_test_utils/_run_training.sh
index 72fd187d19d..8f848a24add 100644
--- a/tests/functional_tests/shell_test_utils/_run_training.sh
+++ b/tests/functional_tests/shell_test_utils/_run_training.sh
@@ -124,8 +124,8 @@ else
value=$(echo "$value" | sed 's/^\[//;s/\]$//')
TRAINING_PARAMS_FROM_CONFIG+="$key $value "
- # Case: contains spaces
- elif [[ "$value" == *" "* ]]; then
+ # Case: contains spaces or shell metacharacters
+ elif [[ "$value" == *" "* || "$value" == *"|"* || "$value" == *"("* || "$value" == *")"* ]]; then
TRAINING_PARAMS_FROM_CONFIG+="$key \"$value\" "
# Case: default
else
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp2/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp2/golden_values_dev_dgx_h100.json
index b9b1236875c..d2bad17b9d0 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp2/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp2/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.47723,
- "2": 10.47576,
- "3": 10.46809,
- "4": 10.47326,
- "5": 10.47148,
- "6": 10.46049,
- "7": 10.46357,
- "8": 10.47334,
- "9": 10.48063,
- "10": 10.46319,
- "11": 10.47102,
- "12": 10.45502,
- "13": 10.44665,
- "14": 10.451,
- "15": 10.48846,
- "16": 10.4509,
- "17": 10.44648,
- "18": 10.44272,
- "19": 10.43057,
- "20": 10.44534,
- "21": 10.41771,
- "22": 10.38656,
- "23": 10.39328,
- "24": 10.37849,
- "25": 10.35466,
- "26": 10.35965,
- "27": 10.34523,
- "28": 10.33556,
- "29": 10.25418,
- "30": 10.23008,
- "31": 10.14093,
- "32": 10.13603,
- "33": 10.13936,
- "34": 10.11381,
- "35": 10.08888,
- "36": 10.09238,
- "37": 10.06851,
- "38": 10.0466,
- "39": 9.97582,
- "40": 9.93764,
- "41": 9.90872,
- "42": 9.84882,
- "43": 9.85772,
- "44": 9.7925,
- "45": 9.80329,
- "46": 9.70285,
- "47": 9.73423,
- "48": 9.70106,
- "49": 9.69966,
- "50": 9.70252
+ "1": 10.60669,
+ "2": 10.59353,
+ "3": 10.60584,
+ "4": 10.59649,
+ "5": 10.60864,
+ "6": 10.59356,
+ "7": 10.59007,
+ "8": 10.59863,
+ "9": 10.59751,
+ "10": 10.58406,
+ "11": 10.5871,
+ "12": 10.5868,
+ "13": 10.58798,
+ "14": 10.584,
+ "15": 10.59476,
+ "16": 10.57583,
+ "17": 10.56453,
+ "18": 10.57247,
+ "19": 10.57124,
+ "20": 10.5692,
+ "21": 10.55506,
+ "22": 10.50706,
+ "23": 10.49008,
+ "24": 10.48057,
+ "25": 10.46669,
+ "26": 10.47293,
+ "27": 10.46421,
+ "28": 10.45935,
+ "29": 10.41081,
+ "30": 10.31751,
+ "31": 10.27443,
+ "32": 10.23404,
+ "33": 10.24168,
+ "34": 10.18728,
+ "35": 10.21275,
+ "36": 10.17155,
+ "37": 10.15185,
+ "38": 10.12989,
+ "39": 10.09684,
+ "40": 10.05497,
+ "41": 9.99663,
+ "42": 9.94055,
+ "43": 9.92416,
+ "44": 9.8772,
+ "45": 9.85131,
+ "46": 9.79438,
+ "47": 9.77933,
+ "48": 9.7565,
+ "49": 9.79733,
+ "50": 9.7547
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 2137.0,
- "2": 1618.0,
- "3": 1561.0,
- "4": 1871.0,
- "5": 1983.0,
- "6": 1565.0,
- "7": 2779.0,
- "8": 2108.0,
- "9": 2008.0,
- "10": 2086.0,
- "11": 2534.0,
- "12": 1686.0,
- "13": 2120.0,
- "14": 2814.0,
- "15": 1735.0,
- "16": 2535.0,
- "17": 2409.0,
- "18": 2345.0,
- "19": 2374.0,
- "20": 2743.0,
- "21": 2039.0,
- "22": 2925.0,
- "23": 2630.0,
- "24": 2821.0,
- "25": 2366.0,
- "26": 2633.0,
- "27": 2921.0,
- "28": 2760.0,
- "29": 2635.0,
- "30": 2614.0,
- "31": 2073.0,
- "32": 2275.0,
- "33": 2130.0,
- "34": 2185.0,
- "35": 2312.0,
- "36": 2789.0,
- "37": 2937.0,
- "38": 2652.0,
- "39": 2929.0,
- "40": 3348.0,
- "41": 1812.0,
- "42": 1441.0,
- "43": 1726.0,
- "44": 2437.0,
- "45": 3263.0,
- "46": 2813.0,
- "47": 2668.0,
- "48": 3411.0,
- "49": 3174.0,
- "50": 2441.0
+ "1": 2234.0,
+ "2": 2602.0,
+ "3": 2388.0,
+ "4": 2437.0,
+ "5": 2787.0,
+ "6": 2606.0,
+ "7": 1584.0,
+ "8": 2304.0,
+ "9": 2266.0,
+ "10": 2268.0,
+ "11": 2359.0,
+ "12": 2624.0,
+ "13": 2299.0,
+ "14": 2110.0,
+ "15": 2760.0,
+ "16": 2401.0,
+ "17": 2626.0,
+ "18": 2654.0,
+ "19": 2723.0,
+ "20": 2332.0,
+ "21": 2381.0,
+ "22": 2719.0,
+ "23": 2332.0,
+ "24": 2495.0,
+ "25": 2429.0,
+ "26": 2493.0,
+ "27": 2682.0,
+ "28": 2441.0,
+ "29": 2649.0,
+ "30": 2567.0,
+ "31": 3215.0,
+ "32": 3027.0,
+ "33": 2690.0,
+ "34": 2936.0,
+ "35": 3006.0,
+ "36": 3090.0,
+ "37": 3011.0,
+ "38": 2635.0,
+ "39": 2474.0,
+ "40": 2176.0,
+ "41": 3476.0,
+ "42": 3588.0,
+ "43": 3497.0,
+ "44": 3612.0,
+ "45": 3666.0,
+ "46": 2814.0,
+ "47": 2046.0,
+ "48": 3135.0,
+ "49": 3297.0,
+ "50": 3666.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 3405920768.0,
- "2": 3405920768.0,
- "3": 3405920768.0,
- "4": 3405920768.0,
- "5": 3405920768.0,
- "6": 3405920768.0,
- "7": 3405920768.0,
- "8": 3405920768.0,
- "9": 3405920768.0,
- "10": 3405920768.0,
- "11": 3405920768.0,
- "12": 3405920768.0,
- "13": 3405920768.0,
- "14": 3405920768.0,
- "15": 3405920768.0,
- "16": 3405920768.0,
- "17": 3405920768.0,
- "18": 3405920768.0,
- "19": 3405920768.0,
- "20": 3405920768.0,
- "21": 3405920768.0,
- "22": 3405920768.0,
- "23": 3405920768.0,
- "24": 3405920768.0,
- "25": 3405920768.0,
- "26": 3405920768.0,
- "27": 3405920768.0,
- "28": 3405920768.0,
- "29": 3405920768.0,
- "30": 3405920768.0,
- "31": 3405920768.0,
- "32": 3405920768.0,
- "33": 3405920768.0,
- "34": 3405920768.0,
- "35": 3405920768.0,
- "36": 3405920768.0,
- "37": 3405920768.0,
- "38": 3405920768.0,
- "39": 3405920768.0,
- "40": 3405920768.0,
- "41": 3405920768.0,
- "42": 3405920768.0,
- "43": 3405920768.0,
- "44": 3405920768.0,
- "45": 3405920768.0,
- "46": 3405920768.0,
- "47": 3405920768.0,
- "48": 3405920768.0,
- "49": 3405920768.0,
- "50": 3405920768.0
+ "1": 3434522112.0,
+ "2": 3434522112.0,
+ "3": 3434522112.0,
+ "4": 3434522112.0,
+ "5": 3434522112.0,
+ "6": 3435308544.0,
+ "7": 3435308544.0,
+ "8": 3435308544.0,
+ "9": 3434522112.0,
+ "10": 3435308544.0,
+ "11": 3435308544.0,
+ "12": 3435308544.0,
+ "13": 3435308544.0,
+ "14": 3435308544.0,
+ "15": 3435308544.0,
+ "16": 3435308544.0,
+ "17": 3435308544.0,
+ "18": 3434522112.0,
+ "19": 3435308544.0,
+ "20": 3435308544.0,
+ "21": 3434522112.0,
+ "22": 3434522112.0,
+ "23": 3435308544.0,
+ "24": 3435308544.0,
+ "25": 3435308544.0,
+ "26": 3435308544.0,
+ "27": 3435308544.0,
+ "28": 3435308544.0,
+ "29": 3434522112.0,
+ "30": 3434522112.0,
+ "31": 3435308544.0,
+ "32": 3435308544.0,
+ "33": 3434522112.0,
+ "34": 3435308544.0,
+ "35": 3434522112.0,
+ "36": 3435308544.0,
+ "37": 3435308544.0,
+ "38": 3435308544.0,
+ "39": 3435308544.0,
+ "40": 3434522112.0,
+ "41": 3435308544.0,
+ "42": 3434522112.0,
+ "43": 3434522112.0,
+ "44": 3435308544.0,
+ "45": 3435308544.0,
+ "46": 3435308544.0,
+ "47": 3435308544.0,
+ "48": 3434522112.0,
+ "49": 3435308544.0,
+ "50": 3434522112.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 4195575808.0,
- "2": 5662015488.0,
- "3": 5662015488.0,
- "4": 5662015488.0,
- "5": 5662015488.0,
- "6": 5662015488.0,
- "7": 5662015488.0,
- "8": 5662015488.0,
- "9": 5662015488.0,
- "10": 5662015488.0,
- "11": 5662015488.0,
- "12": 5662015488.0,
- "13": 5662015488.0,
- "14": 5662015488.0,
- "15": 5662015488.0,
- "16": 5662015488.0,
- "17": 5662015488.0,
- "18": 5662015488.0,
- "19": 5662015488.0,
- "20": 5662015488.0,
- "21": 5662015488.0,
- "22": 5662015488.0,
- "23": 5662015488.0,
- "24": 5662015488.0,
- "25": 5662015488.0,
- "26": 5662015488.0,
- "27": 5662015488.0,
- "28": 5662015488.0,
- "29": 5662015488.0,
- "30": 5662015488.0,
- "31": 5662015488.0,
- "32": 5662015488.0,
- "33": 5662015488.0,
- "34": 5662015488.0,
- "35": 5662015488.0,
- "36": 5662015488.0,
- "37": 5662015488.0,
- "38": 5662015488.0,
- "39": 5662015488.0,
- "40": 5662015488.0,
- "41": 5662015488.0,
- "42": 5662015488.0,
- "43": 5662015488.0,
- "44": 5662015488.0,
- "45": 5662015488.0,
- "46": 5662015488.0,
- "47": 5662015488.0,
- "48": 5662015488.0,
- "49": 5662015488.0,
- "50": 5662015488.0
+ "1": 4230456320.0,
+ "2": 5709228032.0,
+ "3": 5709228032.0,
+ "4": 5709228032.0,
+ "5": 5709229056.0,
+ "6": 5709229056.0,
+ "7": 5709229056.0,
+ "8": 5709229056.0,
+ "9": 5709229056.0,
+ "10": 5709229056.0,
+ "11": 5709229056.0,
+ "12": 5709229056.0,
+ "13": 5709229056.0,
+ "14": 5709229056.0,
+ "15": 5709229056.0,
+ "16": 5709229056.0,
+ "17": 5709229056.0,
+ "18": 5709229056.0,
+ "19": 5709229056.0,
+ "20": 5709229056.0,
+ "21": 5709229056.0,
+ "22": 5709229056.0,
+ "23": 5709229056.0,
+ "24": 5709229056.0,
+ "25": 5709229056.0,
+ "26": 5709229056.0,
+ "27": 5709229056.0,
+ "28": 5709229056.0,
+ "29": 5709229056.0,
+ "30": 5709229056.0,
+ "31": 5709229056.0,
+ "32": 5709229056.0,
+ "33": 5709229056.0,
+ "34": 5709229056.0,
+ "35": 5709229056.0,
+ "36": 5709229056.0,
+ "37": 5709229056.0,
+ "38": 5709229056.0,
+ "39": 5709229056.0,
+ "40": 5709229056.0,
+ "41": 5709229056.0,
+ "42": 5709229056.0,
+ "43": 5709229056.0,
+ "44": 5709229056.0,
+ "45": 5709229056.0,
+ "46": 5709229056.0,
+ "47": 5709229056.0,
+ "48": 5709229056.0,
+ "49": 5709229056.0,
+ "50": 5709229056.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 9.33953,
- "2": 0.53319,
- "3": 0.47492,
- "4": 0.43971,
- "5": 0.43812,
- "6": 0.43852,
- "7": 0.4386,
- "8": 0.43696,
- "9": 0.4374,
- "10": 0.43581,
- "11": 0.71474,
- "12": 0.44321,
- "13": 0.73975,
- "14": 0.44195,
- "15": 0.43796,
- "16": 0.43687,
- "17": 0.43648,
- "18": 0.43733,
- "19": 0.43826,
- "20": 0.44179,
- "21": 1.02916,
- "22": 0.7107,
- "23": 0.70393,
- "24": 0.904,
- "25": 0.43822,
- "26": 0.43864,
- "27": 0.46131,
- "28": 0.44753,
- "29": 0.43372,
- "30": 0.43644,
- "31": 0.45145,
- "32": 0.44608,
- "33": 0.43714,
- "34": 0.43395,
- "35": 0.43358,
- "36": 0.43471,
- "37": 0.43343,
- "38": 0.43378,
- "39": 0.43774,
- "40": 0.43399,
- "41": 0.43662,
- "42": 0.43501,
- "43": 0.43703,
- "44": 0.44084,
- "45": 0.43443,
- "46": 0.43652,
- "47": 0.84278,
- "48": 0.44024,
- "49": 0.4409,
- "50": 0.43833
+ "1": "nan",
+ "2": 6.3648,
+ "3": 0.50773,
+ "4": 0.48828,
+ "5": 0.46858,
+ "6": 0.46849,
+ "7": 0.46721,
+ "8": 0.47128,
+ "9": 0.46839,
+ "10": 0.46898,
+ "11": 0.46864,
+ "12": 0.47069,
+ "13": 0.46803,
+ "14": 0.47103,
+ "15": 0.46722,
+ "16": 0.46802,
+ "17": 0.4671,
+ "18": 0.44126,
+ "19": 0.44637,
+ "20": 0.44305,
+ "21": 0.44231,
+ "22": 0.45492,
+ "23": 0.45474,
+ "24": 0.46736,
+ "25": 0.45949,
+ "26": 0.46093,
+ "27": 0.47387,
+ "28": 0.45804,
+ "29": 0.46041,
+ "30": 0.45964,
+ "31": 0.45448,
+ "32": 0.4596,
+ "33": 0.46132,
+ "34": 0.4511,
+ "35": 0.44276,
+ "36": 0.44321,
+ "37": 0.44297,
+ "38": 0.44333,
+ "39": 0.443,
+ "40": 0.78942,
+ "41": 0.44803,
+ "42": 0.82322,
+ "43": 0.72677,
+ "44": 0.9334,
+ "45": 1.01619,
+ "46": 0.44666,
+ "47": 0.88907,
+ "48": 0.44404,
+ "49": 0.77042,
+ "50": 0.88736
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp2/model_config.yaml b/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp2/model_config.yaml
index 60537ce8776..0dc97066835 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp2/model_config.yaml
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp2/model_config.yaml
@@ -22,8 +22,8 @@ MODEL_ARGS:
--lr-decay-iters: 990000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/bert_shard00/my-bert_00_text_sentence
- --vocab-file: ${DATA_PATH}/text/the_pile/bert_shard00/vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-bert_00_text_sentence
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/vocab.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.0001
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp4_vp2/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp4_vp2/golden_values_dev_dgx_h100.json
index 30fa7e80d5a..0f24dce1ddb 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp4_vp2/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp4_vp2/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.55236,
- "2": 10.52891,
- "3": 10.55085,
- "4": 10.55035,
- "5": 10.52311,
- "6": 10.53328,
- "7": 10.53097,
- "8": 10.54323,
- "9": 10.54514,
- "10": 10.53676,
- "11": 10.53791,
- "12": 10.54319,
- "13": 10.5263,
- "14": 10.5316,
- "15": 10.52714,
- "16": 10.50594,
- "17": 10.5009,
- "18": 10.51024,
- "19": 10.49283,
- "20": 10.48852,
- "21": 10.47463,
- "22": 10.42802,
- "23": 10.42674,
- "24": 10.40359,
- "25": 10.39998,
- "26": 10.38464,
- "27": 10.38236,
- "28": 10.36891,
- "29": 10.32202,
- "30": 10.22049,
- "31": 10.17103,
- "32": 10.12583,
- "33": 10.10622,
- "34": 10.09458,
- "35": 10.07043,
- "36": 10.07484,
- "37": 10.03646,
- "38": 10.0182,
- "39": 9.9686,
- "40": 9.93086,
- "41": 9.87312,
- "42": 9.8185,
- "43": 9.81546,
- "44": 9.73852,
- "45": 9.76279,
- "46": 9.67679,
- "47": 9.68692,
- "48": 9.66292,
- "49": 9.67587,
- "50": 9.67447
+ "1": 10.50372,
+ "2": 10.50518,
+ "3": 10.50375,
+ "4": 10.5122,
+ "5": 10.49605,
+ "6": 10.50635,
+ "7": 10.50842,
+ "8": 10.49712,
+ "9": 10.50322,
+ "10": 10.5124,
+ "11": 10.51046,
+ "12": 10.50236,
+ "13": 10.50045,
+ "14": 10.49454,
+ "15": 10.49733,
+ "16": 10.46631,
+ "17": 10.47165,
+ "18": 10.48801,
+ "19": 10.47165,
+ "20": 10.47134,
+ "21": 10.45835,
+ "22": 10.42035,
+ "23": 10.41806,
+ "24": 10.41512,
+ "25": 10.37515,
+ "26": 10.38894,
+ "27": 10.35288,
+ "28": 10.37449,
+ "29": 10.32003,
+ "30": 10.22366,
+ "31": 10.17914,
+ "32": 10.1594,
+ "33": 10.16322,
+ "34": 10.12997,
+ "35": 10.10282,
+ "36": 10.08198,
+ "37": 10.07429,
+ "38": 10.08263,
+ "39": 10.02137,
+ "40": 9.97876,
+ "41": 9.93118,
+ "42": 9.87334,
+ "43": 9.87304,
+ "44": 9.83579,
+ "45": 9.81241,
+ "46": 9.74637,
+ "47": 9.73201,
+ "48": 9.71622,
+ "49": 9.76511,
+ "50": 9.72575
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 2320.0,
- "2": 2645.0,
- "3": 2441.0,
- "4": 2417.0,
- "5": 2730.0,
- "6": 2332.0,
- "7": 1661.0,
- "8": 2386.0,
- "9": 2256.0,
- "10": 2428.0,
- "11": 2152.0,
- "12": 2337.0,
- "13": 2643.0,
- "14": 2209.0,
- "15": 2607.0,
- "16": 2411.0,
- "17": 2529.0,
- "18": 2392.0,
- "19": 2417.0,
- "20": 2269.0,
- "21": 2382.0,
- "22": 2652.0,
- "23": 2420.0,
- "24": 2251.0,
- "25": 2616.0,
- "26": 2433.0,
- "27": 2470.0,
- "28": 2335.0,
- "29": 2270.0,
- "30": 2689.0,
- "31": 2960.0,
- "32": 2808.0,
- "33": 2659.0,
- "34": 2932.0,
- "35": 2926.0,
- "36": 3103.0,
- "37": 3227.0,
- "38": 2634.0,
- "39": 2132.0,
- "40": 2236.0,
- "41": 3589.0,
- "42": 3470.0,
- "43": 3467.0,
- "44": 4038.0,
- "45": 4173.0,
- "46": 2993.0,
- "47": 1996.0,
- "48": 3318.0,
- "49": 3662.0,
- "50": 3572.0
+ "1": 2517.0,
+ "2": 1916.0,
+ "3": 1789.0,
+ "4": 2431.0,
+ "5": 2093.0,
+ "6": 1896.0,
+ "7": 2326.0,
+ "8": 2342.0,
+ "9": 2220.0,
+ "10": 2183.0,
+ "11": 2757.0,
+ "12": 2022.0,
+ "13": 2145.0,
+ "14": 2737.0,
+ "15": 2215.0,
+ "16": 2683.0,
+ "17": 2675.0,
+ "18": 2661.0,
+ "19": 2489.0,
+ "20": 2625.0,
+ "21": 2081.0,
+ "22": 2656.0,
+ "23": 2660.0,
+ "24": 2796.0,
+ "25": 2638.0,
+ "26": 2650.0,
+ "27": 2770.0,
+ "28": 2834.0,
+ "29": 2491.0,
+ "30": 2958.0,
+ "31": 2507.0,
+ "32": 2724.0,
+ "33": 2451.0,
+ "34": 2602.0,
+ "35": 2690.0,
+ "36": 3159.0,
+ "37": 3123.0,
+ "38": 2775.0,
+ "39": 2712.0,
+ "40": 3571.0,
+ "41": 1909.0,
+ "42": 1523.0,
+ "43": 1765.0,
+ "44": 3233.0,
+ "45": 3818.0,
+ "46": 3184.0,
+ "47": 2878.0,
+ "48": 3078.0,
+ "49": 2752.0,
+ "50": 2196.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 2062574080.0,
- "2": 2062574080.0,
- "3": 2062574080.0,
- "4": 2062574080.0,
- "5": 2062574080.0,
- "6": 2062574080.0,
- "7": 2062574080.0,
- "8": 2062574080.0,
- "9": 2062574080.0,
- "10": 2062574080.0,
- "11": 2062574080.0,
- "12": 2062574080.0,
- "13": 2062574080.0,
- "14": 2062574080.0,
- "15": 2062574080.0,
- "16": 2062574080.0,
- "17": 2062574080.0,
- "18": 2062574080.0,
- "19": 2062574080.0,
- "20": 2062574080.0,
- "21": 2062574080.0,
- "22": 2062574080.0,
- "23": 2062574080.0,
- "24": 2062574080.0,
- "25": 2062574080.0,
- "26": 2062574080.0,
- "27": 2062574080.0,
- "28": 2062574080.0,
- "29": 2062574080.0,
- "30": 2062574080.0,
- "31": 2062574080.0,
- "32": 2062574080.0,
- "33": 2062574080.0,
- "34": 2062574080.0,
- "35": 2062574080.0,
- "36": 2062574080.0,
- "37": 2062574080.0,
- "38": 2062574080.0,
- "39": 2062574080.0,
- "40": 2062574080.0,
- "41": 2062574080.0,
- "42": 2062574080.0,
- "43": 2062574080.0,
- "44": 2062574080.0,
- "45": 2062574080.0,
- "46": 2062574080.0,
- "47": 2062574080.0,
- "48": 2062574080.0,
- "49": 2062574080.0,
- "50": 2062574080.0
+ "1": 2091169280.0,
+ "2": 2091169280.0,
+ "3": 2091169280.0,
+ "4": 2091169280.0,
+ "5": 2091169280.0,
+ "6": 2091169280.0,
+ "7": 2091169280.0,
+ "8": 2091169280.0,
+ "9": 2091169280.0,
+ "10": 2091169280.0,
+ "11": 2091169280.0,
+ "12": 2091169280.0,
+ "13": 2091169280.0,
+ "14": 2091169280.0,
+ "15": 2091169280.0,
+ "16": 2091169280.0,
+ "17": 2091169280.0,
+ "18": 2091169280.0,
+ "19": 2091169280.0,
+ "20": 2091169280.0,
+ "21": 2091169280.0,
+ "22": 2091169280.0,
+ "23": 2091169280.0,
+ "24": 2091169280.0,
+ "25": 2091169280.0,
+ "26": 2091169280.0,
+ "27": 2091169280.0,
+ "28": 2091169280.0,
+ "29": 2091169280.0,
+ "30": 2091169280.0,
+ "31": 2091169280.0,
+ "32": 2091169280.0,
+ "33": 2091169280.0,
+ "34": 2091169280.0,
+ "35": 2091169280.0,
+ "36": 2091169280.0,
+ "37": 2091169280.0,
+ "38": 2091169280.0,
+ "39": 2091169280.0,
+ "40": 2091169280.0,
+ "41": 2091169280.0,
+ "42": 2091169280.0,
+ "43": 2091169280.0,
+ "44": 2091169280.0,
+ "45": 2091169280.0,
+ "46": 2091169280.0,
+ "47": 2091169280.0,
+ "48": 2091169280.0,
+ "49": 2091169280.0,
+ "50": 2091169280.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 4386474496.0,
- "2": 5246722560.0,
- "3": 5246722560.0,
- "4": 5246722560.0,
- "5": 5246722560.0,
- "6": 5246722560.0,
- "7": 5246722560.0,
- "8": 5246722560.0,
- "9": 5246722560.0,
- "10": 5246722560.0,
- "11": 5246722560.0,
- "12": 5246722560.0,
- "13": 5246722560.0,
- "14": 5246722560.0,
- "15": 5246722560.0,
- "16": 5246722560.0,
- "17": 5246722560.0,
- "18": 5246722560.0,
- "19": 5246722560.0,
- "20": 5246722560.0,
- "21": 5246722560.0,
- "22": 5246722560.0,
- "23": 5246722560.0,
- "24": 5246722560.0,
- "25": 5246722560.0,
- "26": 5246722560.0,
- "27": 5246722560.0,
- "28": 5246722560.0,
- "29": 5246722560.0,
- "30": 5246722560.0,
- "31": 5246722560.0,
- "32": 5246722560.0,
- "33": 5246722560.0,
- "34": 5246722560.0,
- "35": 5246722560.0,
- "36": 5246722560.0,
- "37": 5246722560.0,
- "38": 5246722560.0,
- "39": 5246722560.0,
- "40": 5246722560.0,
- "41": 5246722560.0,
- "42": 5246722560.0,
- "43": 5246722560.0,
- "44": 5246722560.0,
- "45": 5246722560.0,
- "46": 5246722560.0,
- "47": 5246722560.0,
- "48": 5246722560.0,
- "49": 5246722560.0,
- "50": 5246722560.0
+ "1": 4421607424.0,
+ "2": 5294192128.0,
+ "3": 5294192128.0,
+ "4": 5294192128.0,
+ "5": 5294192128.0,
+ "6": 5294192128.0,
+ "7": 5294192128.0,
+ "8": 5294192128.0,
+ "9": 5294192128.0,
+ "10": 5294192128.0,
+ "11": 5294192128.0,
+ "12": 5294192128.0,
+ "13": 5294977024.0,
+ "14": 5294977024.0,
+ "15": 5294977024.0,
+ "16": 5294977024.0,
+ "17": 5294977024.0,
+ "18": 5294977024.0,
+ "19": 5294977024.0,
+ "20": 5294977024.0,
+ "21": 5294977024.0,
+ "22": 5294977024.0,
+ "23": 5294977024.0,
+ "24": 5294977024.0,
+ "25": 5294977024.0,
+ "26": 5294977024.0,
+ "27": 5294977024.0,
+ "28": 5294977024.0,
+ "29": 5294977024.0,
+ "30": 5294977024.0,
+ "31": 5294977024.0,
+ "32": 5294977024.0,
+ "33": 5294977024.0,
+ "34": 5294977024.0,
+ "35": 5294977024.0,
+ "36": 5294977024.0,
+ "37": 5294977024.0,
+ "38": 5294977024.0,
+ "39": 5294977024.0,
+ "40": 5294977024.0,
+ "41": 5294977024.0,
+ "42": 5294977024.0,
+ "43": 5294977024.0,
+ "44": 5294977024.0,
+ "45": 5294977024.0,
+ "46": 5294977024.0,
+ "47": 5294977024.0,
+ "48": 5294977024.0,
+ "49": 5294977024.0,
+ "50": 5294977024.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 12.53778,
- "2": 0.64042,
- "3": 0.57704,
- "4": 0.56942,
- "5": 0.55857,
- "6": 1.5214,
- "7": 0.8799,
- "8": 0.58802,
- "9": 0.58845,
- "10": 0.91566,
- "11": 1.66597,
- "12": 1.31669,
- "13": 0.9054,
- "14": 0.55959,
- "15": 0.55349,
- "16": 0.56731,
- "17": 0.54994,
- "18": 0.56124,
- "19": 0.54032,
- "20": 0.54467,
- "21": 0.56577,
- "22": 0.59073,
- "23": 0.55848,
- "24": 0.5515,
- "25": 0.56783,
- "26": 0.58223,
- "27": 0.56278,
- "28": 0.55385,
- "29": 0.54473,
- "30": 0.54779,
- "31": 0.54239,
- "32": 0.53324,
- "33": 0.54812,
- "34": 0.57008,
- "35": 0.56814,
- "36": 0.55146,
- "37": 0.56138,
- "38": 0.80574,
- "39": 0.5919,
- "40": 0.83084,
- "41": 0.9006,
- "42": 0.82734,
- "43": 0.98233,
- "44": 1.08635,
- "45": 1.33415,
- "46": 1.29362,
- "47": 1.03481,
- "48": 1.02838,
- "49": 0.56104,
- "50": 0.57748
+ "1": "nan",
+ "2": 8.80363,
+ "3": 0.56371,
+ "4": 0.55956,
+ "5": 0.57722,
+ "6": 0.5559,
+ "7": 0.55109,
+ "8": 0.56338,
+ "9": 0.55675,
+ "10": 0.54615,
+ "11": 0.56947,
+ "12": 0.54313,
+ "13": 0.53579,
+ "14": 0.5348,
+ "15": 0.55059,
+ "16": 0.58449,
+ "17": 1.72193,
+ "18": 1.00286,
+ "19": 1.00508,
+ "20": 1.37568,
+ "21": 1.0004,
+ "22": 1.61847,
+ "23": 1.37176,
+ "24": 1.86248,
+ "25": 1.3993,
+ "26": 0.56478,
+ "27": 0.54444,
+ "28": 0.55376,
+ "29": 0.5687,
+ "30": 0.58734,
+ "31": 0.57557,
+ "32": 0.58734,
+ "33": 0.58598,
+ "34": 0.57746,
+ "35": 0.57728,
+ "36": 0.55998,
+ "37": 0.54419,
+ "38": 0.54307,
+ "39": 0.54541,
+ "40": 0.56237,
+ "41": 0.55314,
+ "42": 0.55352,
+ "43": 0.57847,
+ "44": 0.5425,
+ "45": 0.83687,
+ "46": 0.56484,
+ "47": 1.43437,
+ "48": 1.15295,
+ "49": 1.09052,
+ "50": 0.85934
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp4_vp2/model_config.yaml b/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp4_vp2/model_config.yaml
index 0e908381456..05117c8f4a0 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp4_vp2/model_config.yaml
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp1_pp4_vp2/model_config.yaml
@@ -22,8 +22,8 @@ MODEL_ARGS:
--lr-decay-iters: 990000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/bert_shard00/my-bert_00_text_sentence
- --vocab-file: ${DATA_PATH}/text/the_pile/bert_shard00/vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-bert_00_text_sentence
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/vocab.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.0001
@@ -42,5 +42,4 @@ MODEL_ARGS:
--bf16: true
--ckpt-format: torch
--attention-backend: unfused
- --legacy-tokenizer: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2/golden_values_dev_dgx_h100.json
index 7a21f7ae2f9..d8512022d7b 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.48367,
- "2": 10.48426,
- "3": 10.48254,
- "4": 10.48311,
- "5": 10.4764,
- "6": 10.4844,
- "7": 10.48458,
- "8": 10.48829,
- "9": 10.49008,
- "10": 10.47268,
- "11": 10.47256,
- "12": 10.48259,
- "13": 10.47857,
- "14": 10.45154,
- "15": 10.47925,
- "16": 10.45346,
- "17": 10.45145,
- "18": 10.46238,
- "19": 10.44113,
- "20": 10.45448,
- "21": 10.43454,
- "22": 10.40591,
- "23": 10.39975,
- "24": 10.37583,
- "25": 10.38168,
- "26": 10.3515,
- "27": 10.35388,
- "28": 10.34965,
- "29": 10.28701,
- "30": 10.21143,
- "31": 10.17272,
- "32": 10.13416,
- "33": 10.14725,
- "34": 10.10738,
- "35": 10.10592,
- "36": 10.08739,
- "37": 10.08157,
- "38": 10.07245,
- "39": 10.00093,
- "40": 9.98138,
- "41": 9.92543,
- "42": 9.87534,
- "43": 9.88716,
- "44": 9.80646,
- "45": 9.82342,
- "46": 9.73786,
- "47": 9.74811,
- "48": 9.71614,
- "49": 9.74493,
- "50": 9.73
+ "1": 10.52582,
+ "2": 10.54385,
+ "3": 10.53319,
+ "4": 10.53921,
+ "5": 10.53684,
+ "6": 10.54381,
+ "7": 10.54794,
+ "8": 10.54122,
+ "9": 10.53291,
+ "10": 10.52971,
+ "11": 10.5324,
+ "12": 10.5384,
+ "13": 10.5393,
+ "14": 10.52549,
+ "15": 10.52273,
+ "16": 10.50512,
+ "17": 10.51237,
+ "18": 10.50762,
+ "19": 10.50413,
+ "20": 10.51125,
+ "21": 10.51037,
+ "22": 10.44522,
+ "23": 10.43219,
+ "24": 10.44115,
+ "25": 10.40728,
+ "26": 10.41482,
+ "27": 10.39808,
+ "28": 10.41186,
+ "29": 10.36611,
+ "30": 10.28309,
+ "31": 10.2298,
+ "32": 10.20571,
+ "33": 10.21465,
+ "34": 10.17029,
+ "35": 10.14644,
+ "36": 10.12745,
+ "37": 10.11693,
+ "38": 10.11809,
+ "39": 10.07868,
+ "40": 10.01082,
+ "41": 9.96752,
+ "42": 9.92742,
+ "43": 9.91995,
+ "44": 9.86345,
+ "45": 9.83762,
+ "46": 9.7791,
+ "47": 9.77306,
+ "48": 9.74906,
+ "49": 9.77865,
+ "50": 9.7532
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 2570.0,
- "2": 1923.0,
- "3": 1512.0,
- "4": 2322.0,
- "5": 2033.0,
- "6": 1774.0,
- "7": 2781.0,
- "8": 2460.0,
- "9": 2308.0,
- "10": 2635.0,
- "11": 2397.0,
- "12": 1817.0,
- "13": 2348.0,
- "14": 2749.0,
- "15": 2027.0,
- "16": 2719.0,
- "17": 2487.0,
- "18": 2533.0,
- "19": 2547.0,
- "20": 2850.0,
- "21": 1990.0,
- "22": 2964.0,
- "23": 2695.0,
- "24": 2772.0,
- "25": 2524.0,
- "26": 2977.0,
- "27": 2627.0,
- "28": 2776.0,
- "29": 2514.0,
- "30": 2843.0,
- "31": 2070.0,
- "32": 2362.0,
- "33": 2211.0,
- "34": 2574.0,
- "35": 2499.0,
- "36": 2943.0,
- "37": 3347.0,
- "38": 2628.0,
- "39": 2781.0,
- "40": 3335.0,
- "41": 1800.0,
- "42": 1598.0,
- "43": 1719.0,
- "44": 2631.0,
- "45": 3492.0,
- "46": 2988.0,
- "47": 2784.0,
- "48": 2951.0,
- "49": 2907.0,
- "50": 2113.0
+ "1": 2713.0,
+ "2": 2105.0,
+ "3": 1711.0,
+ "4": 2467.0,
+ "5": 2378.0,
+ "6": 2049.0,
+ "7": 2371.0,
+ "8": 2546.0,
+ "9": 2514.0,
+ "10": 2558.0,
+ "11": 2696.0,
+ "12": 2058.0,
+ "13": 2341.0,
+ "14": 2864.0,
+ "15": 2242.0,
+ "16": 2780.0,
+ "17": 2769.0,
+ "18": 2705.0,
+ "19": 2663.0,
+ "20": 2799.0,
+ "21": 2296.0,
+ "22": 2828.0,
+ "23": 2685.0,
+ "24": 2823.0,
+ "25": 2678.0,
+ "26": 2723.0,
+ "27": 2650.0,
+ "28": 2920.0,
+ "29": 2640.0,
+ "30": 2945.0,
+ "31": 2442.0,
+ "32": 2849.0,
+ "33": 2386.0,
+ "34": 2613.0,
+ "35": 2676.0,
+ "36": 3018.0,
+ "37": 3312.0,
+ "38": 2732.0,
+ "39": 2933.0,
+ "40": 3341.0,
+ "41": 1746.0,
+ "42": 1496.0,
+ "43": 1694.0,
+ "44": 2861.0,
+ "45": 3761.0,
+ "46": 3341.0,
+ "47": 3180.0,
+ "48": 2692.0,
+ "49": 2538.0,
+ "50": 2128.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1785063936.0,
- "2": 1785063936.0,
- "3": 1785063936.0,
- "4": 1785063936.0,
- "5": 1785063936.0,
- "6": 1785063936.0,
- "7": 1785063936.0,
- "8": 1785063936.0,
- "9": 1785063936.0,
- "10": 1785063936.0,
- "11": 1785063936.0,
- "12": 1785063936.0,
- "13": 1785063936.0,
- "14": 1785063936.0,
- "15": 1785063936.0,
- "16": 1785063936.0,
- "17": 1785063936.0,
- "18": 1785063936.0,
- "19": 1785063936.0,
- "20": 1785063936.0,
- "21": 1785063936.0,
- "22": 1785063936.0,
- "23": 1785063936.0,
- "24": 1785063936.0,
- "25": 1785063936.0,
- "26": 1785063936.0,
- "27": 1785063936.0,
- "28": 1785063936.0,
- "29": 1785063936.0,
- "30": 1785063936.0,
- "31": 1785063936.0,
- "32": 1785063936.0,
- "33": 1785063936.0,
- "34": 1785063936.0,
- "35": 1785063936.0,
- "36": 1785063936.0,
- "37": 1785063936.0,
- "38": 1785063936.0,
- "39": 1785063936.0,
- "40": 1785063936.0,
- "41": 1785063936.0,
- "42": 1785063936.0,
- "43": 1785063936.0,
- "44": 1785063936.0,
- "45": 1785063936.0,
- "46": 1785063936.0,
- "47": 1785063936.0,
- "48": 1785063936.0,
- "49": 1785063936.0,
- "50": 1785063936.0
+ "1": 1796646400.0,
+ "2": 1796646400.0,
+ "3": 1796646400.0,
+ "4": 1796646400.0,
+ "5": 1796646400.0,
+ "6": 1796646400.0,
+ "7": 1796646400.0,
+ "8": 1796646400.0,
+ "9": 1796646400.0,
+ "10": 1796646400.0,
+ "11": 1796646400.0,
+ "12": 1796646400.0,
+ "13": 1796646400.0,
+ "14": 1796646400.0,
+ "15": 1796646400.0,
+ "16": 1796646400.0,
+ "17": 1796646400.0,
+ "18": 1797694976.0,
+ "19": 1796646400.0,
+ "20": 1796646400.0,
+ "21": 1796646400.0,
+ "22": 1796646400.0,
+ "23": 1796646400.0,
+ "24": 1796646400.0,
+ "25": 1796646400.0,
+ "26": 1796646400.0,
+ "27": 1796646400.0,
+ "28": 1796646400.0,
+ "29": 1796646400.0,
+ "30": 1796646400.0,
+ "31": 1796646400.0,
+ "32": 1796646400.0,
+ "33": 1796646400.0,
+ "34": 1796646400.0,
+ "35": 1796646400.0,
+ "36": 1796646400.0,
+ "37": 1796646400.0,
+ "38": 1796646400.0,
+ "39": 1796646400.0,
+ "40": 1796646400.0,
+ "41": 1796646400.0,
+ "42": 1796646400.0,
+ "43": 1796646400.0,
+ "44": 1796646400.0,
+ "45": 1796646400.0,
+ "46": 1796646400.0,
+ "47": 1796646400.0,
+ "48": 1796646400.0,
+ "49": 1796646400.0,
+ "50": 1796646400.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 2366910464.0,
- "2": 3109894144.0,
- "3": 3109894144.0,
- "4": 3109894144.0,
- "5": 3109894144.0,
- "6": 3109894144.0,
- "7": 3109894144.0,
- "8": 3109894144.0,
- "9": 3109894144.0,
- "10": 3109894144.0,
- "11": 3109894144.0,
- "12": 3109894144.0,
- "13": 3109894144.0,
- "14": 3109894144.0,
- "15": 3109897216.0,
- "16": 3109897216.0,
- "17": 3109897216.0,
- "18": 3109897216.0,
- "19": 3109897216.0,
- "20": 3109897216.0,
- "21": 3109897216.0,
- "22": 3109897216.0,
- "23": 3109897216.0,
- "24": 3109897216.0,
- "25": 3109897216.0,
- "26": 3109897216.0,
- "27": 3109897216.0,
- "28": 3109897216.0,
- "29": 3109897216.0,
- "30": 3109897216.0,
- "31": 3109897216.0,
- "32": 3109897216.0,
- "33": 3109897216.0,
- "34": 3109897216.0,
- "35": 3109897216.0,
- "36": 3109897216.0,
- "37": 3109897216.0,
- "38": 3109897216.0,
- "39": 3109897216.0,
- "40": 3109897216.0,
- "41": 3109897216.0,
- "42": 3109897216.0,
- "43": 3109897216.0,
- "44": 3109897216.0,
- "45": 3109897216.0,
- "46": 3109897216.0,
- "47": 3109897216.0,
- "48": 3109897216.0,
- "49": 3109897216.0,
- "50": 3109897216.0
+ "1": 2376915456.0,
+ "2": 3124626944.0,
+ "3": 3124626944.0,
+ "4": 3126723584.0,
+ "5": 3126723584.0,
+ "6": 3126723584.0,
+ "7": 3126723584.0,
+ "8": 3126723584.0,
+ "9": 3126723584.0,
+ "10": 3126723584.0,
+ "11": 3126723584.0,
+ "12": 3126723584.0,
+ "13": 3126723584.0,
+ "14": 3126723584.0,
+ "15": 3126723584.0,
+ "16": 3126723584.0,
+ "17": 3126723584.0,
+ "18": 3126724096.0,
+ "19": 3126724096.0,
+ "20": 3126724096.0,
+ "21": 3126724096.0,
+ "22": 3126724096.0,
+ "23": 3126724096.0,
+ "24": 3126724096.0,
+ "25": 3126724096.0,
+ "26": 3126724096.0,
+ "27": 3126724096.0,
+ "28": 3126724096.0,
+ "29": 3126724096.0,
+ "30": 3126724096.0,
+ "31": 3128818688.0,
+ "32": 3128818688.0,
+ "33": 3128818688.0,
+ "34": 3128818688.0,
+ "35": 3128818688.0,
+ "36": 3128818688.0,
+ "37": 3128818688.0,
+ "38": 3128818688.0,
+ "39": 3128818688.0,
+ "40": 3128818688.0,
+ "41": 3128818688.0,
+ "42": 3128818688.0,
+ "43": 3128818688.0,
+ "44": 3128818688.0,
+ "45": 3128818688.0,
+ "46": 3128818688.0,
+ "47": 3128818688.0,
+ "48": 3128818688.0,
+ "49": 3128818688.0,
+ "50": 3128818688.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.5121,
- "2": 1.00958,
- "3": 0.92732,
- "4": 0.90421,
- "5": 0.90504,
- "6": 0.89943,
- "7": 0.90319,
- "8": 1.1748,
- "9": 1.95208,
- "10": 0.92148,
- "11": 0.91859,
- "12": 0.92137,
- "13": 0.92531,
- "14": 1.25591,
- "15": 0.92418,
- "16": 0.91961,
- "17": 0.90838,
- "18": 0.90766,
- "19": 0.90747,
- "20": 0.9061,
- "21": 0.93723,
- "22": 0.90644,
- "23": 0.91067,
- "24": 1.66749,
- "25": 0.91188,
- "26": 0.91194,
- "27": 0.988,
- "28": 0.92516,
- "29": 0.91117,
- "30": 1.435,
- "31": 0.89868,
- "32": 0.90735,
- "33": 1.29737,
- "34": 1.32235,
- "35": 0.91506,
- "36": 0.91851,
- "37": 0.92715,
- "38": 0.92769,
- "39": 0.92632,
- "40": 1.26827,
- "41": 1.07193,
- "42": 1.07217,
- "43": 0.98674,
- "44": 1.07179,
- "45": 1.09756,
- "46": 1.10568,
- "47": 0.92215,
- "48": 0.92051,
- "49": 0.92335,
- "50": 0.92251
+ "1": "nan",
+ "2": 7.37428,
+ "3": 0.92991,
+ "4": 0.9078,
+ "5": 0.90709,
+ "6": 0.90742,
+ "7": 0.90431,
+ "8": 0.90471,
+ "9": 0.90657,
+ "10": 0.90486,
+ "11": 0.90481,
+ "12": 0.90401,
+ "13": 0.90933,
+ "14": 0.90583,
+ "15": 0.90584,
+ "16": 0.90548,
+ "17": 0.90677,
+ "18": 0.90774,
+ "19": 1.62221,
+ "20": 0.90756,
+ "21": 0.90392,
+ "22": 1.30376,
+ "23": 1.19849,
+ "24": 1.18039,
+ "25": 1.24273,
+ "26": 0.90511,
+ "27": 0.90514,
+ "28": 0.90501,
+ "29": 0.90556,
+ "30": 0.90459,
+ "31": 0.90475,
+ "32": 0.90442,
+ "33": 0.90462,
+ "34": 0.90382,
+ "35": 0.90348,
+ "36": 0.90487,
+ "37": 0.90596,
+ "38": 0.90271,
+ "39": 0.90357,
+ "40": 0.90215,
+ "41": 0.9009,
+ "42": 0.9012,
+ "43": 0.90294,
+ "44": 0.90111,
+ "45": 0.90371,
+ "46": 0.9033,
+ "47": 1.10365,
+ "48": 0.90243,
+ "49": 1.04847,
+ "50": 0.90108
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2/model_config.yaml b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2/model_config.yaml
index f965ee1d9ef..01ba8adeccf 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2/model_config.yaml
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2/model_config.yaml
@@ -22,8 +22,8 @@ MODEL_ARGS:
--lr-decay-iters: 990000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/bert_shard00/my-bert_00_text_sentence
- --vocab-file: ${DATA_PATH}/text/the_pile/bert_shard00/vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-bert_00_text_sentence
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/vocab.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.0001
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_frozen_resume_torch_dist/model_config.yaml b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_frozen_resume_torch_dist/model_config.yaml
index fc4c836c98a..680c3c69ea7 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_frozen_resume_torch_dist/model_config.yaml
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_frozen_resume_torch_dist/model_config.yaml
@@ -22,8 +22,8 @@ MODEL_ARGS:
--lr-decay-iters: 990000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/bert_shard00/my-bert_00_text_sentence
- --vocab-file: ${DATA_PATH}/text/the_pile/bert_shard00/vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-bert_00_text_sentence
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/vocab.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.0001
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_local_spec/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_local_spec/golden_values_dev_dgx_h100.json
index d034c6bf7d8..cd5ae5291ff 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_local_spec/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_local_spec/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.4837,
- "2": 10.48435,
- "3": 10.48251,
- "4": 10.48303,
- "5": 10.47647,
- "6": 10.48423,
- "7": 10.48457,
- "8": 10.48837,
- "9": 10.49003,
- "10": 10.47255,
- "11": 10.47245,
- "12": 10.4828,
- "13": 10.47855,
- "14": 10.45162,
- "15": 10.47936,
- "16": 10.45364,
- "17": 10.45143,
- "18": 10.46239,
- "19": 10.44136,
- "20": 10.45438,
- "21": 10.43469,
- "22": 10.40587,
- "23": 10.39982,
- "24": 10.37585,
- "25": 10.38173,
- "26": 10.35154,
- "27": 10.35401,
- "28": 10.3497,
- "29": 10.28714,
- "30": 10.21194,
- "31": 10.17274,
- "32": 10.13439,
- "33": 10.14753,
- "34": 10.10759,
- "35": 10.10592,
- "36": 10.08756,
- "37": 10.08177,
- "38": 10.07257,
- "39": 10.0013,
- "40": 9.9816,
- "41": 9.92551,
- "42": 9.87537,
- "43": 9.88725,
- "44": 9.80659,
- "45": 9.82349,
- "46": 9.73821,
- "47": 9.74829,
- "48": 9.71628,
- "49": 9.74489,
- "50": 9.73004
+ "1": 10.52551,
+ "2": 10.54472,
+ "3": 10.53112,
+ "4": 10.53996,
+ "5": 10.5371,
+ "6": 10.54388,
+ "7": 10.54852,
+ "8": 10.5417,
+ "9": 10.5317,
+ "10": 10.52961,
+ "11": 10.52976,
+ "12": 10.53761,
+ "13": 10.53871,
+ "14": 10.52447,
+ "15": 10.52205,
+ "16": 10.50381,
+ "17": 10.51177,
+ "18": 10.50815,
+ "19": 10.50379,
+ "20": 10.51035,
+ "21": 10.5095,
+ "22": 10.44506,
+ "23": 10.43405,
+ "24": 10.44154,
+ "25": 10.40759,
+ "26": 10.41686,
+ "27": 10.39998,
+ "28": 10.41047,
+ "29": 10.36664,
+ "30": 10.28289,
+ "31": 10.23013,
+ "32": 10.20411,
+ "33": 10.21551,
+ "34": 10.17208,
+ "35": 10.14779,
+ "36": 10.12817,
+ "37": 10.11674,
+ "38": 10.11981,
+ "39": 10.08045,
+ "40": 10.01189,
+ "41": 9.96762,
+ "42": 9.92708,
+ "43": 9.92037,
+ "44": 9.8642,
+ "45": 9.83846,
+ "46": 9.7798,
+ "47": 9.77341,
+ "48": 9.74902,
+ "49": 9.77849,
+ "50": 9.75354
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 2554.0,
- "2": 1919.0,
- "3": 1521.0,
- "4": 2330.0,
- "5": 2010.0,
- "6": 1725.0,
- "7": 2803.0,
- "8": 2435.0,
- "9": 2286.0,
- "10": 2570.0,
- "11": 2438.0,
- "12": 1829.0,
- "13": 2332.0,
- "14": 2832.0,
- "15": 2008.0,
- "16": 2659.0,
- "17": 2454.0,
- "18": 2500.0,
- "19": 2588.0,
- "20": 2834.0,
- "21": 2042.0,
- "22": 3037.0,
- "23": 2702.0,
- "24": 2700.0,
- "25": 2568.0,
- "26": 2896.0,
- "27": 2735.0,
- "28": 2699.0,
- "29": 2548.0,
- "30": 2843.0,
- "31": 2160.0,
- "32": 2458.0,
- "33": 2130.0,
- "34": 2517.0,
- "35": 2597.0,
- "36": 3001.0,
- "37": 3305.0,
- "38": 2682.0,
- "39": 2805.0,
- "40": 3430.0,
- "41": 1767.0,
- "42": 1516.0,
- "43": 1798.0,
- "44": 2790.0,
- "45": 3578.0,
- "46": 3016.0,
- "47": 2890.0,
- "48": 3065.0,
- "49": 2914.0,
- "50": 2208.0
+ "1": 2667.0,
+ "2": 2111.0,
+ "3": 1710.0,
+ "4": 2527.0,
+ "5": 2170.0,
+ "6": 2007.0,
+ "7": 2321.0,
+ "8": 2548.0,
+ "9": 2544.0,
+ "10": 2545.0,
+ "11": 2696.0,
+ "12": 1990.0,
+ "13": 2361.0,
+ "14": 2930.0,
+ "15": 2273.0,
+ "16": 2839.0,
+ "17": 2683.0,
+ "18": 2587.0,
+ "19": 2795.0,
+ "20": 2732.0,
+ "21": 2188.0,
+ "22": 2794.0,
+ "23": 2638.0,
+ "24": 2839.0,
+ "25": 2636.0,
+ "26": 2794.0,
+ "27": 2727.0,
+ "28": 2852.0,
+ "29": 2479.0,
+ "30": 2902.0,
+ "31": 2482.0,
+ "32": 2871.0,
+ "33": 2457.0,
+ "34": 2512.0,
+ "35": 2744.0,
+ "36": 2997.0,
+ "37": 3284.0,
+ "38": 2786.0,
+ "39": 2916.0,
+ "40": 3534.0,
+ "41": 1715.0,
+ "42": 1502.0,
+ "43": 1776.0,
+ "44": 2874.0,
+ "45": 3603.0,
+ "46": 3510.0,
+ "47": 3195.0,
+ "48": 2695.0,
+ "49": 2483.0,
+ "50": 2041.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1768285696.0,
- "2": 1768285696.0,
- "3": 1768285696.0,
- "4": 1768285696.0,
- "5": 1768285696.0,
- "6": 1768285696.0,
- "7": 1768285696.0,
- "8": 1768285696.0,
- "9": 1768285696.0,
- "10": 1768285696.0,
- "11": 1768285696.0,
- "12": 1768285696.0,
- "13": 1768285696.0,
- "14": 1768285696.0,
- "15": 1768285696.0,
- "16": 1768285696.0,
- "17": 1768285696.0,
- "18": 1768285696.0,
- "19": 1768285696.0,
- "20": 1768285696.0,
- "21": 1768285696.0,
- "22": 1768285696.0,
- "23": 1768285696.0,
- "24": 1768285696.0,
- "25": 1768285696.0,
- "26": 1768285696.0,
- "27": 1768285696.0,
- "28": 1768285696.0,
- "29": 1768285696.0,
- "30": 1768285696.0,
- "31": 1768285696.0,
- "32": 1768285696.0,
- "33": 1768285696.0,
- "34": 1768285696.0,
- "35": 1768285696.0,
- "36": 1768285696.0,
- "37": 1768285696.0,
- "38": 1768285696.0,
- "39": 1768285696.0,
- "40": 1768285696.0,
- "41": 1768285696.0,
- "42": 1768285696.0,
- "43": 1768285696.0,
- "44": 1768285696.0,
- "45": 1768285696.0,
- "46": 1768285696.0,
- "47": 1768285696.0,
- "48": 1768285696.0,
- "49": 1768285696.0,
- "50": 1768285696.0
+ "1": 1779868160.0,
+ "2": 1779868160.0,
+ "3": 1779868160.0,
+ "4": 1779868160.0,
+ "5": 1779868160.0,
+ "6": 1779868160.0,
+ "7": 1779868160.0,
+ "8": 1779868160.0,
+ "9": 1779868160.0,
+ "10": 1779868160.0,
+ "11": 1779868160.0,
+ "12": 1779868160.0,
+ "13": 1779868160.0,
+ "14": 1779868160.0,
+ "15": 1779868160.0,
+ "16": 1779868160.0,
+ "17": 1779868160.0,
+ "18": 1779868160.0,
+ "19": 1779868160.0,
+ "20": 1779868160.0,
+ "21": 1779868160.0,
+ "22": 1779868160.0,
+ "23": 1779868160.0,
+ "24": 1779868160.0,
+ "25": 1779868160.0,
+ "26": 1779868160.0,
+ "27": 1779868160.0,
+ "28": 1779868160.0,
+ "29": 1779868160.0,
+ "30": 1779868160.0,
+ "31": 1779868160.0,
+ "32": 1779868160.0,
+ "33": 1779868160.0,
+ "34": 1779868160.0,
+ "35": 1779868160.0,
+ "36": 1779868160.0,
+ "37": 1779868160.0,
+ "38": 1779868160.0,
+ "39": 1779868160.0,
+ "40": 1779868160.0,
+ "41": 1779868160.0,
+ "42": 1779868160.0,
+ "43": 1779868160.0,
+ "44": 1779868160.0,
+ "45": 1779868160.0,
+ "46": 1779868160.0,
+ "47": 1779868160.0,
+ "48": 1779868160.0,
+ "49": 1779868160.0,
+ "50": 1779868160.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 2337549312.0,
- "2": 3080536064.0,
- "3": 3082107392.0,
- "4": 3082107392.0,
- "5": 3082107392.0,
- "6": 3082107392.0,
- "7": 3082107392.0,
- "8": 3082107392.0,
- "9": 3082107392.0,
- "10": 3082107392.0,
- "11": 3082107392.0,
- "12": 3082107392.0,
- "13": 3082107392.0,
- "14": 3082107392.0,
- "15": 3082107392.0,
- "16": 3082108928.0,
- "17": 3082108928.0,
- "18": 3082108928.0,
- "19": 3082108928.0,
- "20": 3082108928.0,
- "21": 3082108928.0,
- "22": 3082108928.0,
- "23": 3082108928.0,
- "24": 3082108928.0,
- "25": 3082108928.0,
- "26": 3082108928.0,
- "27": 3082108928.0,
- "28": 3082108928.0,
- "29": 3082108928.0,
- "30": 3082108928.0,
- "31": 3082108928.0,
- "32": 3082108928.0,
- "33": 3082108928.0,
- "34": 3082108928.0,
- "35": 3082108928.0,
- "36": 3082108928.0,
- "37": 3082108928.0,
- "38": 3082108928.0,
- "39": 3082108928.0,
- "40": 3082108928.0,
- "41": 3082108928.0,
- "42": 3082108928.0,
- "43": 3082108928.0,
- "44": 3082108928.0,
- "45": 3082108928.0,
- "46": 3082108928.0,
- "47": 3082108928.0,
- "48": 3082108928.0,
- "49": 3082108928.0,
- "50": 3082108928.0
+ "1": 2347554304.0,
+ "2": 3095265792.0,
+ "3": 3097362944.0,
+ "4": 3097362944.0,
+ "5": 3097362944.0,
+ "6": 3097362944.0,
+ "7": 3097362944.0,
+ "8": 3097362944.0,
+ "9": 3097362944.0,
+ "10": 3097362944.0,
+ "11": 3097362944.0,
+ "12": 3097362944.0,
+ "13": 3097362944.0,
+ "14": 3097362944.0,
+ "15": 3097362944.0,
+ "16": 3097362944.0,
+ "17": 3097362944.0,
+ "18": 3097362944.0,
+ "19": 3097362944.0,
+ "20": 3097362944.0,
+ "21": 3097362944.0,
+ "22": 3097362944.0,
+ "23": 3097362944.0,
+ "24": 3097362944.0,
+ "25": 3097362944.0,
+ "26": 3097362944.0,
+ "27": 3097362944.0,
+ "28": 3097362944.0,
+ "29": 3097362944.0,
+ "30": 3097362944.0,
+ "31": 3097362944.0,
+ "32": 3097362944.0,
+ "33": 3097362944.0,
+ "34": 3097362944.0,
+ "35": 3097362944.0,
+ "36": 3097362944.0,
+ "37": 3097362944.0,
+ "38": 3097362944.0,
+ "39": 3097362944.0,
+ "40": 3097362944.0,
+ "41": 3097362944.0,
+ "42": 3097362944.0,
+ "43": 3097362944.0,
+ "44": 3097362944.0,
+ "45": 3097362944.0,
+ "46": 3097362944.0,
+ "47": 3097362944.0,
+ "48": 3097362944.0,
+ "49": 3097362944.0,
+ "50": 3097362944.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.51798,
- "2": 0.89864,
- "3": 0.7978,
- "4": 0.74774,
- "5": 0.73987,
- "6": 0.74277,
- "7": 0.76779,
- "8": 0.74313,
- "9": 1.58315,
- "10": 0.73453,
- "11": 0.73215,
- "12": 0.72957,
- "13": 0.72967,
- "14": 0.73868,
- "15": 0.73216,
- "16": 1.10392,
- "17": 0.73363,
- "18": 0.73647,
- "19": 0.76464,
- "20": 0.73565,
- "21": 0.72858,
- "22": 0.72652,
- "23": 0.72858,
- "24": 0.74508,
- "25": 0.74166,
- "26": 0.7704,
- "27": 1.15428,
- "28": 1.146,
- "29": 0.73283,
- "30": 0.73304,
- "31": 0.73237,
- "32": 0.7343,
- "33": 0.73304,
- "34": 0.72879,
- "35": 0.73286,
- "36": 1.74169,
- "37": 1.10377,
- "38": 0.73148,
- "39": 0.73227,
- "40": 0.73028,
- "41": 0.73026,
- "42": 1.15127,
- "43": 1.11655,
- "44": 0.73185,
- "45": 1.17599,
- "46": 1.07292,
- "47": 0.72983,
- "48": 0.72804,
- "49": 0.73205,
- "50": 0.72929
+ "1": "nan",
+ "2": 6.8133,
+ "3": 0.78889,
+ "4": 0.76348,
+ "5": 0.76062,
+ "6": 0.7603,
+ "7": 0.75938,
+ "8": 0.75976,
+ "9": 0.75832,
+ "10": 0.7602,
+ "11": 0.75916,
+ "12": 0.75773,
+ "13": 0.76091,
+ "14": 0.75933,
+ "15": 0.76144,
+ "16": 0.75858,
+ "17": 0.76001,
+ "18": 0.75869,
+ "19": 0.75878,
+ "20": 0.75808,
+ "21": 1.13983,
+ "22": 1.05768,
+ "23": 1.04222,
+ "24": 1.20576,
+ "25": 1.08454,
+ "26": 1.32941,
+ "27": 1.04313,
+ "28": 0.75803,
+ "29": 0.75938,
+ "30": 0.75902,
+ "31": 0.75689,
+ "32": 0.75871,
+ "33": 0.75866,
+ "34": 0.75745,
+ "35": 0.76139,
+ "36": 0.75892,
+ "37": 0.79114,
+ "38": 0.76162,
+ "39": 0.7674,
+ "40": 0.75973,
+ "41": 0.77146,
+ "42": 0.76048,
+ "43": 0.75813,
+ "44": 0.76005,
+ "45": 0.75842,
+ "46": 0.76133,
+ "47": 0.75955,
+ "48": 0.75994,
+ "49": 1.01962,
+ "50": 0.76007
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_local_spec/model_config.yaml b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_local_spec/model_config.yaml
index 8974bc1ea24..c372de7180a 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_local_spec/model_config.yaml
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_local_spec/model_config.yaml
@@ -22,8 +22,8 @@ MODEL_ARGS:
--lr-decay-iters: 990000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/bert_shard00/my-bert_00_text_sentence
- --vocab-file: ${DATA_PATH}/text/the_pile/bert_shard00/vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-bert_00_text_sentence
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/vocab.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.0001
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_h100.json
index 4302b8e40ca..73f9ef61d1b 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.48367,
- "2": 10.48426,
- "3": 10.48254,
- "4": 10.48311,
- "5": 10.4764,
- "6": 10.4844,
- "7": 10.48458,
- "8": 10.48829,
- "9": 10.49008,
- "10": 10.47268,
- "11": 10.47256,
- "12": 10.48259,
- "13": 10.47857,
- "14": 10.45154,
- "15": 10.47925,
- "16": 10.45346,
- "17": 10.45145,
- "18": 10.46238,
- "19": 10.44113,
- "20": 10.45448,
- "21": 10.43454,
- "22": 10.40591,
- "23": 10.39975,
- "24": 10.37583,
- "25": 10.38168,
- "26": 10.3515,
- "27": 10.35388,
- "28": 10.34965,
- "29": 10.28701,
- "30": 10.21143,
- "31": 10.17272,
- "32": 10.13416,
- "33": 10.14725,
- "34": 10.10738,
- "35": 10.10592,
- "36": 10.08739,
- "37": 10.08157,
- "38": 10.07245,
- "39": 10.00093,
- "40": 9.98138,
- "41": 9.92543,
- "42": 9.87534,
- "43": 9.88716,
- "44": 9.80646,
- "45": 9.82342,
- "46": 9.73786,
- "47": 9.74811,
- "48": 9.71614,
- "49": 9.74493,
- "50": 9.73,
- "51": 9.71492,
- "52": 9.66464,
- "53": 9.60912,
- "54": 9.62726,
- "55": 9.6101,
- "56": 9.61721,
- "57": 9.56794,
- "58": 9.52741,
- "59": 9.51674,
- "60": 9.51863,
- "61": 9.53132,
- "62": 9.45018,
- "63": 9.4572,
- "64": 9.43437,
- "65": 9.45816,
- "66": 9.43669,
- "67": 9.39678,
- "68": 9.36478,
- "69": 9.40956,
- "70": 9.37595,
- "71": 9.41738,
- "72": 9.42564,
- "73": 9.37611,
- "74": 9.41543,
- "75": 9.3788,
- "76": 9.28012,
- "77": 9.32212,
- "78": 9.35744,
- "79": 9.3215,
- "80": 9.31497,
- "81": 9.26785,
- "82": 9.34183,
- "83": 9.32151,
- "84": 9.24796,
- "85": 9.35033,
- "86": 9.224,
- "87": 9.30611,
- "88": 9.29894,
- "89": 9.22704,
- "90": 9.28479,
- "91": 9.2311,
- "92": 9.27474,
- "93": 9.19219,
- "94": 9.23969,
- "95": 9.28,
- "96": 9.17525,
- "97": 9.21888,
- "98": 9.1721,
- "99": 9.16455,
- "100": 9.1482
+ "1": 10.5255,
+ "2": 10.54543,
+ "3": 10.53296,
+ "4": 10.54172,
+ "5": 10.53695,
+ "6": 10.54202,
+ "7": 10.54756,
+ "8": 10.54057,
+ "9": 10.5332,
+ "10": 10.52931,
+ "11": 10.53027,
+ "12": 10.53716,
+ "13": 10.53856,
+ "14": 10.52431,
+ "15": 10.52246,
+ "16": 10.50461,
+ "17": 10.51158,
+ "18": 10.50818,
+ "19": 10.50562,
+ "20": 10.51022,
+ "21": 10.5106,
+ "22": 10.44632,
+ "23": 10.43227,
+ "24": 10.44226,
+ "25": 10.40713,
+ "26": 10.41644,
+ "27": 10.39817,
+ "28": 10.41053,
+ "29": 10.36603,
+ "30": 10.28268,
+ "31": 10.23034,
+ "32": 10.20526,
+ "33": 10.21539,
+ "34": 10.17031,
+ "35": 10.14776,
+ "36": 10.12828,
+ "37": 10.1162,
+ "38": 10.11801,
+ "39": 10.07925,
+ "40": 10.01108,
+ "41": 9.96785,
+ "42": 9.92765,
+ "43": 9.92013,
+ "44": 9.86313,
+ "45": 9.8384,
+ "46": 9.77939,
+ "47": 9.77261,
+ "48": 9.74972,
+ "49": 9.77876,
+ "50": 9.75325,
+ "51": 9.75982,
+ "52": 9.70938,
+ "53": 9.67157,
+ "54": 9.69254,
+ "55": 9.68023,
+ "56": 9.6724,
+ "57": 9.60149,
+ "58": 9.61869,
+ "59": 9.54613,
+ "60": 9.61183,
+ "61": 9.5485,
+ "62": 9.53684,
+ "63": 9.5229,
+ "64": 9.51064,
+ "65": 9.52236,
+ "66": 9.49106,
+ "67": 9.458,
+ "68": 9.44145,
+ "69": 9.44261,
+ "70": 9.44,
+ "71": 9.47046,
+ "72": 9.45683,
+ "73": 9.40592,
+ "74": 9.45691,
+ "75": 9.40451,
+ "76": 9.37381,
+ "77": 9.34376,
+ "78": 9.37822,
+ "79": 9.41047,
+ "80": 9.34538,
+ "81": 9.33084,
+ "82": 9.34567,
+ "83": 9.31601,
+ "84": 9.29953,
+ "85": 9.33611,
+ "86": 9.26941,
+ "87": 9.3142,
+ "88": 9.29905,
+ "89": 9.26883,
+ "90": 9.34101,
+ "91": 9.25606,
+ "92": 9.29678,
+ "93": 9.2997,
+ "94": 9.27544,
+ "95": 9.28063,
+ "96": 9.1827,
+ "97": 9.26469,
+ "98": 9.19697,
+ "99": 9.21958,
+ "100": 9.22941
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 2570.0,
- "2": 1923.0,
- "3": 1512.0,
- "4": 2322.0,
- "5": 2033.0,
- "6": 1774.0,
- "7": 2781.0,
- "8": 2460.0,
- "9": 2308.0,
- "10": 2635.0,
- "11": 2397.0,
- "12": 1817.0,
- "13": 2348.0,
- "14": 2749.0,
- "15": 2027.0,
- "16": 2719.0,
- "17": 2487.0,
- "18": 2533.0,
- "19": 2547.0,
- "20": 2850.0,
- "21": 1990.0,
- "22": 2964.0,
- "23": 2695.0,
- "24": 2772.0,
- "25": 2524.0,
- "26": 2977.0,
- "27": 2627.0,
- "28": 2776.0,
- "29": 2514.0,
- "30": 2843.0,
- "31": 2070.0,
- "32": 2362.0,
- "33": 2211.0,
- "34": 2574.0,
- "35": 2499.0,
- "36": 2943.0,
- "37": 3347.0,
- "38": 2628.0,
- "39": 2781.0,
- "40": 3335.0,
- "41": 1800.0,
- "42": 1598.0,
- "43": 1719.0,
- "44": 2631.0,
- "45": 3492.0,
- "46": 2988.0,
- "47": 2784.0,
- "48": 2951.0,
- "49": 2907.0,
- "50": 2113.0,
- "51": 1961.0,
- "52": 2445.0,
- "53": 3654.0,
- "54": 3489.0,
- "55": 3419.0,
- "56": 4364.0,
- "57": 4145.0,
- "58": 4155.0,
- "59": 1699.0,
- "60": 2358.0,
- "61": 2070.0,
- "62": 4094.0,
- "63": 3516.0,
- "64": 4287.0,
- "65": 2891.0,
- "66": 1733.0,
- "67": 1914.0,
- "68": 4420.0,
- "69": 4479.0,
- "70": 4656.0,
- "71": 2135.0,
- "72": 4476.0,
- "73": 4048.0,
- "74": 3199.0,
- "75": 4735.0,
- "76": 2218.0,
- "77": 4952.0,
- "78": 4158.0,
- "79": 2657.0,
- "80": 3846.0,
- "81": 3472.0,
- "82": 2979.0,
- "83": 5364.0,
- "84": 4430.0,
- "85": 4249.0,
- "86": 3509.0,
- "87": 4817.0,
- "88": 3434.0,
- "89": 4711.0,
- "90": 4448.0,
- "91": 4374.0,
- "92": 3507.0,
- "93": 5549.0,
- "94": 3635.0,
- "95": 4540.0,
- "96": 3659.0,
- "97": 3756.0,
- "98": 4513.0,
- "99": 4491.0,
- "100": 3445.0
+ "1": 2627.0,
+ "2": 2077.0,
+ "3": 1787.0,
+ "4": 2543.0,
+ "5": 2299.0,
+ "6": 2005.0,
+ "7": 2340.0,
+ "8": 2609.0,
+ "9": 2445.0,
+ "10": 2549.0,
+ "11": 2729.0,
+ "12": 1973.0,
+ "13": 2344.0,
+ "14": 2815.0,
+ "15": 2138.0,
+ "16": 2695.0,
+ "17": 2734.0,
+ "18": 2618.0,
+ "19": 2684.0,
+ "20": 2774.0,
+ "21": 2232.0,
+ "22": 2741.0,
+ "23": 2766.0,
+ "24": 2765.0,
+ "25": 2603.0,
+ "26": 2752.0,
+ "27": 2710.0,
+ "28": 2850.0,
+ "29": 2548.0,
+ "30": 2860.0,
+ "31": 2455.0,
+ "32": 2905.0,
+ "33": 2370.0,
+ "34": 2633.0,
+ "35": 2616.0,
+ "36": 2951.0,
+ "37": 3244.0,
+ "38": 2689.0,
+ "39": 2961.0,
+ "40": 3435.0,
+ "41": 1681.0,
+ "42": 1506.0,
+ "43": 1708.0,
+ "44": 2844.0,
+ "45": 3578.0,
+ "46": 3385.0,
+ "47": 3166.0,
+ "48": 2665.0,
+ "49": 2581.0,
+ "50": 2070.0,
+ "51": 1804.0,
+ "52": 2468.0,
+ "53": 3818.0,
+ "54": 3687.0,
+ "55": 3358.0,
+ "56": 4355.0,
+ "57": 3881.0,
+ "58": 4252.0,
+ "59": 1801.0,
+ "60": 2582.0,
+ "61": 2222.0,
+ "62": 4043.0,
+ "63": 4021.0,
+ "64": 4455.0,
+ "65": 3146.0,
+ "66": 1989.0,
+ "67": 2145.0,
+ "68": 4101.0,
+ "69": 4371.0,
+ "70": 4063.0,
+ "71": 2082.0,
+ "72": 4105.0,
+ "73": 3483.0,
+ "74": 2475.0,
+ "75": 5400.0,
+ "76": 2650.0,
+ "77": 4081.0,
+ "78": 4434.0,
+ "79": 2230.0,
+ "80": 3435.0,
+ "81": 3996.0,
+ "82": 3714.0,
+ "83": 4848.0,
+ "84": 5477.0,
+ "85": 4379.0,
+ "86": 3982.0,
+ "87": 3568.0,
+ "88": 4496.0,
+ "89": 3904.0,
+ "90": 4523.0,
+ "91": 4699.0,
+ "92": 3994.0,
+ "93": 3782.0,
+ "94": 3063.0,
+ "95": 4124.0,
+ "96": 3709.0,
+ "97": 3391.0,
+ "98": 4634.0,
+ "99": 3766.0,
+ "100": 3345.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1785063936.0,
- "2": 1785063936.0,
- "3": 1785063936.0,
- "4": 1785063936.0,
- "5": 1785063936.0,
- "6": 1785063936.0,
- "7": 1785063936.0,
- "8": 1785063936.0,
- "9": 1785063936.0,
- "10": 1785063936.0,
- "11": 1785063936.0,
- "12": 1785063936.0,
- "13": 1785063936.0,
- "14": 1785063936.0,
- "15": 1785063936.0,
- "16": 1785063936.0,
- "17": 1785063936.0,
- "18": 1785063936.0,
- "19": 1785063936.0,
- "20": 1785063936.0,
- "21": 1785063936.0,
- "22": 1785063936.0,
- "23": 1785063936.0,
- "24": 1785063936.0,
- "25": 1785063936.0,
- "26": 1785063936.0,
- "27": 1785063936.0,
- "28": 1785588224.0,
- "29": 1785063936.0,
- "30": 1785063936.0,
- "31": 1785063936.0,
- "32": 1785063936.0,
- "33": 1785063936.0,
- "34": 1785063936.0,
- "35": 1785063936.0,
- "36": 1785063936.0,
- "37": 1785063936.0,
- "38": 1785063936.0,
- "39": 1785063936.0,
- "40": 1785063936.0,
- "41": 1785063936.0,
- "42": 1785063936.0,
- "43": 1785063936.0,
- "44": 1785063936.0,
- "45": 1785063936.0,
- "46": 1785063936.0,
- "47": 1785063936.0,
- "48": 1785063936.0,
- "49": 1785063936.0,
- "50": 1785063936.0,
- "51": 1785063936.0,
- "52": 1785063936.0,
- "53": 1785063936.0,
- "54": 1785063936.0,
- "55": 1785063936.0,
- "56": 1785063936.0,
- "57": 1785063936.0,
- "58": 1785063936.0,
- "59": 1785063936.0,
- "60": 1785063936.0,
- "61": 1785063936.0,
- "62": 1785063936.0,
- "63": 1785063936.0,
- "64": 1785063936.0,
- "65": 1785063936.0,
- "66": 1785063936.0,
- "67": 1785063936.0,
- "68": 1785063936.0,
- "69": 1785063936.0,
- "70": 1785063936.0,
- "71": 1785063936.0,
- "72": 1785063936.0,
- "73": 1785063936.0,
- "74": 1785063936.0,
- "75": 1785063936.0,
- "76": 1785063936.0,
- "77": 1785063936.0,
- "78": 1785063936.0,
- "79": 1785063936.0,
- "80": 1785063936.0,
- "81": 1785063936.0,
- "82": 1785063936.0,
- "83": 1785063936.0,
- "84": 1785063936.0,
- "85": 1785063936.0,
- "86": 1785063936.0,
- "87": 1785063936.0,
- "88": 1785063936.0,
- "89": 1785063936.0,
- "90": 1785063936.0,
- "91": 1785063936.0,
- "92": 1785063936.0,
- "93": 1785063936.0,
- "94": 1785063936.0,
- "95": 1785063936.0,
- "96": 1785063936.0,
- "97": 1785063936.0,
- "98": 1785063936.0,
- "99": 1785063936.0,
- "100": 1785063936.0
+ "1": 1796646400.0,
+ "2": 1796646400.0,
+ "3": 1796646400.0,
+ "4": 1796646400.0,
+ "5": 1796646400.0,
+ "6": 1796646400.0,
+ "7": 1796646400.0,
+ "8": 1796646400.0,
+ "9": 1796646400.0,
+ "10": 1796646400.0,
+ "11": 1796646400.0,
+ "12": 1796646400.0,
+ "13": 1796646400.0,
+ "14": 1796646400.0,
+ "15": 1796646400.0,
+ "16": 1796646400.0,
+ "17": 1796646400.0,
+ "18": 1797694976.0,
+ "19": 1796646400.0,
+ "20": 1796646400.0,
+ "21": 1796646400.0,
+ "22": 1796646400.0,
+ "23": 1796646400.0,
+ "24": 1796646400.0,
+ "25": 1796646400.0,
+ "26": 1796646400.0,
+ "27": 1796646400.0,
+ "28": 1796646400.0,
+ "29": 1796646400.0,
+ "30": 1796646400.0,
+ "31": 1796646400.0,
+ "32": 1796646400.0,
+ "33": 1796646400.0,
+ "34": 1796646400.0,
+ "35": 1796646400.0,
+ "36": 1796646400.0,
+ "37": 1796646400.0,
+ "38": 1796646400.0,
+ "39": 1796646400.0,
+ "40": 1796646400.0,
+ "41": 1796646400.0,
+ "42": 1796646400.0,
+ "43": 1796646400.0,
+ "44": 1796646400.0,
+ "45": 1796646400.0,
+ "46": 1796646400.0,
+ "47": 1796646400.0,
+ "48": 1796646400.0,
+ "49": 1796646400.0,
+ "50": 1796646400.0,
+ "51": 1796646400.0,
+ "52": 1796646400.0,
+ "53": 1796646400.0,
+ "54": 1796646400.0,
+ "55": 1796646400.0,
+ "56": 1796646400.0,
+ "57": 1796646400.0,
+ "58": 1796646400.0,
+ "59": 1796646400.0,
+ "60": 1796646400.0,
+ "61": 1796646400.0,
+ "62": 1796646400.0,
+ "63": 1796646400.0,
+ "64": 1796646400.0,
+ "65": 1796646400.0,
+ "66": 1796646400.0,
+ "67": 1796646400.0,
+ "68": 1796646400.0,
+ "69": 1796646400.0,
+ "70": 1796646400.0,
+ "71": 1796646400.0,
+ "72": 1796646400.0,
+ "73": 1796646400.0,
+ "74": 1796646400.0,
+ "75": 1796646400.0,
+ "76": 1796646400.0,
+ "77": 1796646400.0,
+ "78": 1796646400.0,
+ "79": 1796646400.0,
+ "80": 1796646400.0,
+ "81": 1796646400.0,
+ "82": 1796646400.0,
+ "83": 1796646400.0,
+ "84": 1796646400.0,
+ "85": 1796646400.0,
+ "86": 1796646400.0,
+ "87": 1796646400.0,
+ "88": 1796646400.0,
+ "89": 1796646400.0,
+ "90": 1796646400.0,
+ "91": 1796646400.0,
+ "92": 1796646400.0,
+ "93": 1796646400.0,
+ "94": 1796646400.0,
+ "95": 1796646400.0,
+ "96": 1796646400.0,
+ "97": 1796646400.0,
+ "98": 1796646400.0,
+ "99": 1796646400.0,
+ "100": 1796646400.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 2366910464.0,
- "2": 3109372928.0,
- "3": 3109372928.0,
- "4": 3109372928.0,
- "5": 3109372928.0,
- "6": 3109892608.0,
- "7": 3109892608.0,
- "8": 3111465472.0,
- "9": 3111465472.0,
- "10": 3111465472.0,
- "11": 3111465472.0,
- "12": 3111470080.0,
- "13": 3111470080.0,
- "14": 3111470080.0,
- "15": 3111470080.0,
- "16": 3111470080.0,
- "17": 3111470080.0,
- "18": 3111470080.0,
- "19": 3111470080.0,
- "20": 3111470080.0,
- "21": 3111470080.0,
- "22": 3111470080.0,
- "23": 3111470080.0,
- "24": 3111470080.0,
- "25": 3111470080.0,
- "26": 3111470080.0,
- "27": 3111470080.0,
- "28": 3111470080.0,
- "29": 3111470080.0,
- "30": 3111470080.0,
- "31": 3111470080.0,
- "32": 3111470080.0,
- "33": 3111470080.0,
- "34": 3111470080.0,
- "35": 3111470080.0,
- "36": 3111988224.0,
- "37": 3111988224.0,
- "38": 3111988224.0,
- "39": 3111988224.0,
- "40": 3111988224.0,
- "41": 3111988224.0,
- "42": 3111988224.0,
- "43": 3111988224.0,
- "44": 3111988224.0,
- "45": 3111988224.0,
- "46": 3111988224.0,
- "47": 3111988224.0,
- "48": 3111988224.0,
- "49": 3111988224.0,
- "50": 3111988224.0,
- "51": 3111988224.0,
- "52": 3111988224.0,
- "53": 3111988224.0,
- "54": 3111988224.0,
- "55": 3111988224.0,
- "56": 3111988224.0,
- "57": 3111988224.0,
- "58": 3111988224.0,
- "59": 3111988224.0,
- "60": 3111988224.0,
- "61": 3111988224.0,
- "62": 3111988224.0,
- "63": 3111988224.0,
- "64": 3111988224.0,
- "65": 3111988224.0,
- "66": 3111988224.0,
- "67": 3111988224.0,
- "68": 3111988224.0,
- "69": 3111988224.0,
- "70": 3111988224.0,
- "71": 3111988224.0,
- "72": 3111988224.0,
- "73": 3111988224.0,
- "74": 3111988224.0,
- "75": 3111988224.0,
- "76": 3111988224.0,
- "77": 3111988224.0,
- "78": 3111988224.0,
- "79": 3111988224.0,
- "80": 3111988224.0,
- "81": 3111988224.0,
- "82": 3111988224.0,
- "83": 3111988224.0,
- "84": 3111988224.0,
- "85": 3111988224.0,
- "86": 3111988224.0,
- "87": 3111988224.0,
- "88": 3111988224.0,
- "89": 3111988224.0,
- "90": 3111988224.0,
- "91": 3111988224.0,
- "92": 3111988224.0,
- "93": 3111988224.0,
- "94": 3111988224.0,
- "95": 3111988224.0,
- "96": 3111988224.0,
- "97": 3111988224.0,
- "98": 3111988224.0,
- "99": 3111988224.0,
- "100": 3111988224.0
+ "1": 2376915456.0,
+ "2": 3126718464.0,
+ "3": 3126723584.0,
+ "4": 3126723584.0,
+ "5": 3126723584.0,
+ "6": 3126723584.0,
+ "7": 3126723584.0,
+ "8": 3126723584.0,
+ "9": 3126723584.0,
+ "10": 3126723584.0,
+ "11": 3126723584.0,
+ "12": 3126723584.0,
+ "13": 3126724096.0,
+ "14": 3126724096.0,
+ "15": 3126724096.0,
+ "16": 3126724096.0,
+ "17": 3126724096.0,
+ "18": 3126724096.0,
+ "19": 3126724096.0,
+ "20": 3126724096.0,
+ "21": 3126724096.0,
+ "22": 3126724096.0,
+ "23": 3126724096.0,
+ "24": 3126724096.0,
+ "25": 3126724096.0,
+ "26": 3126724096.0,
+ "27": 3126724096.0,
+ "28": 3126724096.0,
+ "29": 3126724096.0,
+ "30": 3126724096.0,
+ "31": 3126724096.0,
+ "32": 3126724096.0,
+ "33": 3126724096.0,
+ "34": 3126724096.0,
+ "35": 3128817152.0,
+ "36": 3128817152.0,
+ "37": 3128817152.0,
+ "38": 3128817152.0,
+ "39": 3128817152.0,
+ "40": 3128817152.0,
+ "41": 3128817152.0,
+ "42": 3128817152.0,
+ "43": 3128817152.0,
+ "44": 3128817152.0,
+ "45": 3128817152.0,
+ "46": 3128817152.0,
+ "47": 3128817152.0,
+ "48": 3128817152.0,
+ "49": 3128817152.0,
+ "50": 3128817152.0,
+ "51": 3128817152.0,
+ "52": 3128817152.0,
+ "53": 3128817152.0,
+ "54": 3128817152.0,
+ "55": 3128817152.0,
+ "56": 3128817152.0,
+ "57": 3128817152.0,
+ "58": 3128817152.0,
+ "59": 3128817152.0,
+ "60": 3128817152.0,
+ "61": 3128817152.0,
+ "62": 3128817152.0,
+ "63": 3128817152.0,
+ "64": 3128817152.0,
+ "65": 3128817152.0,
+ "66": 3128817152.0,
+ "67": 3128817152.0,
+ "68": 3128817152.0,
+ "69": 3128817152.0,
+ "70": 3128817152.0,
+ "71": 3128817152.0,
+ "72": 3128817152.0,
+ "73": 3128817152.0,
+ "74": 3128817152.0,
+ "75": 3128817152.0,
+ "76": 3128817152.0,
+ "77": 3128817152.0,
+ "78": 3128817152.0,
+ "79": 3128817152.0,
+ "80": 3128817152.0,
+ "81": 3128817152.0,
+ "82": 3128817152.0,
+ "83": 3128817152.0,
+ "84": 3128817152.0,
+ "85": 3128817152.0,
+ "86": 3128817152.0,
+ "87": 3128817152.0,
+ "88": 3128817152.0,
+ "89": 3128817152.0,
+ "90": 3128817152.0,
+ "91": 3128817152.0,
+ "92": 3128817152.0,
+ "93": 3128817152.0,
+ "94": 3128817152.0,
+ "95": 3128817152.0,
+ "96": 3128817152.0,
+ "97": 3128817152.0,
+ "98": 3128817152.0,
+ "99": 3128817152.0,
+ "100": 3128817152.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 11.18542,
- "2": 0.99156,
- "3": 0.93327,
- "4": 0.90681,
- "5": 0.90504,
- "6": 0.90415,
- "7": 0.90281,
- "8": 1.14692,
- "9": 1.44306,
- "10": 0.89873,
- "11": 0.90113,
- "12": 0.89984,
- "13": 1.24688,
- "14": 0.90399,
- "15": 0.90327,
- "16": 0.89945,
- "17": 0.90194,
- "18": 0.89984,
- "19": 0.89878,
- "20": 0.89865,
- "21": 0.90167,
- "22": 0.90176,
- "23": 0.90423,
- "24": 2.02738,
- "25": 0.90411,
- "26": 0.90354,
- "27": 0.90203,
- "28": 1.26668,
- "29": 0.89854,
- "30": 1.45828,
- "31": 0.90574,
- "32": 0.90137,
- "33": 1.70784,
- "34": 0.89924,
- "35": 0.90059,
- "36": 0.90525,
- "37": 0.90801,
- "38": 0.90691,
- "39": 0.9048,
- "40": 1.47233,
- "41": 0.91116,
- "42": 1.22468,
- "43": 1.0011,
- "44": 1.22804,
- "45": 1.12037,
- "46": 1.00115,
- "47": 0.91003,
- "48": 0.91208,
- "49": 0.91545,
- "50": 0.91,
- "51": 0.91471,
- "52": 0.91238,
- "53": 0.90865,
- "54": 0.91588,
- "55": 0.91889,
- "56": 0.91882,
- "57": 0.92072,
- "58": 0.9202,
- "59": 0.92355,
- "60": 0.92097,
- "61": 0.91924,
- "62": 0.91496,
- "63": 0.91648,
- "64": 0.91615,
- "65": 0.91333,
- "66": 0.91743,
- "67": 0.9094,
- "68": 0.91122,
- "69": 0.90894,
- "70": 0.91968,
- "71": 0.92199,
- "72": 0.91976,
- "73": 0.92156,
- "74": 0.91995,
- "75": 0.90852,
- "76": 0.90983,
- "77": 1.19595,
- "78": 0.9092,
- "79": 1.16564,
- "80": 1.06882,
- "81": 0.90637,
- "82": 0.90812,
- "83": 0.91,
- "84": 0.90847,
- "85": 0.88526,
- "86": 0.87691,
- "87": 0.88881,
- "88": 0.87995,
- "89": 0.9042,
- "90": 0.90269,
- "91": 0.90587,
- "92": 0.90035,
- "93": 0.89985,
- "94": 0.90093,
- "95": 0.90088,
- "96": 0.89612,
- "97": 0.89401,
- "98": 0.89773,
- "99": 0.90081,
- "100": 0.8988
+ "1": "nan",
+ "2": 6.76631,
+ "3": 0.95984,
+ "4": 0.94975,
+ "5": 0.93831,
+ "6": 0.945,
+ "7": 0.93711,
+ "8": 0.94821,
+ "9": 0.94811,
+ "10": 0.90992,
+ "11": 0.9045,
+ "12": 0.95433,
+ "13": 0.91015,
+ "14": 0.9083,
+ "15": 0.90314,
+ "16": 0.90971,
+ "17": 0.8884,
+ "18": 0.88846,
+ "19": 0.89125,
+ "20": 0.88742,
+ "21": 1.19785,
+ "22": 1.20611,
+ "23": 1.55135,
+ "24": 1.48332,
+ "25": 0.90669,
+ "26": 0.90463,
+ "27": 0.89318,
+ "28": 0.89164,
+ "29": 0.88961,
+ "30": 0.90018,
+ "31": 0.90462,
+ "32": 0.90641,
+ "33": 0.90556,
+ "34": 0.98589,
+ "35": 1.1539,
+ "36": 0.88875,
+ "37": 0.89282,
+ "38": 0.89295,
+ "39": 0.8889,
+ "40": 0.88835,
+ "41": 0.95698,
+ "42": 0.89214,
+ "43": 0.89368,
+ "44": 0.89819,
+ "45": 0.89685,
+ "46": 0.89476,
+ "47": 0.89421,
+ "48": 0.89614,
+ "49": 0.89461,
+ "50": 0.89638,
+ "51": 1.25608,
+ "52": 1.33601,
+ "53": 1.33532,
+ "54": 1.10995,
+ "55": 1.16395,
+ "56": 0.89218,
+ "57": 0.89161,
+ "58": 0.89187,
+ "59": 0.89198,
+ "60": 0.88939,
+ "61": 0.8924,
+ "62": 0.89611,
+ "63": 0.90486,
+ "64": 0.89408,
+ "65": 0.88917,
+ "66": 0.89096,
+ "67": 0.93479,
+ "68": 0.88888,
+ "69": 0.89018,
+ "70": 1.29881,
+ "71": 1.34537,
+ "72": 0.89411,
+ "73": 0.89067,
+ "74": 1.35116,
+ "75": 0.90202,
+ "76": 0.92355,
+ "77": 0.9139,
+ "78": 0.90763,
+ "79": 1.13089,
+ "80": 0.91411,
+ "81": 0.91094,
+ "82": 1.55434,
+ "83": 1.12496,
+ "84": 1.09892,
+ "85": 1.10172,
+ "86": 0.89499,
+ "87": 0.89706,
+ "88": 0.89478,
+ "89": 1.30503,
+ "90": 0.89391,
+ "91": 0.89533,
+ "92": 0.89547,
+ "93": 0.89634,
+ "94": 0.89647,
+ "95": 0.89626,
+ "96": 0.8957,
+ "97": 1.35845,
+ "98": 0.89865,
+ "99": 0.89531,
+ "100": 0.89447
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_h100_2nd.json b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_h100_2nd.json
index a5b9c2f1ab2..7990f924ec5 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_h100_2nd.json
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_h100_2nd.json
@@ -54,56 +54,56 @@
"48": "nan",
"49": "nan",
"50": "nan",
- "51": 9.71492,
- "52": 9.66464,
- "53": 9.60912,
- "54": 9.62726,
- "55": 9.6101,
- "56": 9.61721,
- "57": 9.56794,
- "58": 9.52741,
- "59": 9.51674,
- "60": 9.51863,
- "61": 9.53132,
- "62": 9.45018,
- "63": 9.4572,
- "64": 9.43437,
- "65": 9.45816,
- "66": 9.43669,
- "67": 9.39678,
- "68": 9.36478,
- "69": 9.40956,
- "70": 9.37595,
- "71": 9.41738,
- "72": 9.42564,
- "73": 9.37611,
- "74": 9.41543,
- "75": 9.3788,
- "76": 9.28012,
- "77": 9.32212,
- "78": 9.35744,
- "79": 9.3215,
- "80": 9.31497,
- "81": 9.26785,
- "82": 9.34183,
- "83": 9.32151,
- "84": 9.24796,
- "85": 9.35033,
- "86": 9.224,
- "87": 9.30611,
- "88": 9.29894,
- "89": 9.22704,
- "90": 9.28479,
- "91": 9.2311,
- "92": 9.27474,
- "93": 9.19219,
- "94": 9.23969,
- "95": 9.28,
- "96": 9.17525,
- "97": 9.21888,
- "98": 9.1721,
- "99": 9.16455,
- "100": 9.1482
+ "51": 9.76001,
+ "52": 9.70981,
+ "53": 9.67192,
+ "54": 9.69327,
+ "55": 9.67994,
+ "56": 9.67301,
+ "57": 9.60209,
+ "58": 9.61874,
+ "59": 9.54659,
+ "60": 9.61259,
+ "61": 9.54836,
+ "62": 9.53716,
+ "63": 9.52333,
+ "64": 9.51044,
+ "65": 9.52202,
+ "66": 9.4896,
+ "67": 9.4575,
+ "68": 9.44091,
+ "69": 9.44314,
+ "70": 9.43974,
+ "71": 9.4698,
+ "72": 9.45651,
+ "73": 9.40468,
+ "74": 9.45623,
+ "75": 9.40499,
+ "76": 9.37331,
+ "77": 9.34347,
+ "78": 9.37859,
+ "79": 9.41089,
+ "80": 9.34502,
+ "81": 9.33074,
+ "82": 9.34623,
+ "83": 9.31635,
+ "84": 9.29926,
+ "85": 9.33611,
+ "86": 9.26905,
+ "87": 9.31456,
+ "88": 9.29844,
+ "89": 9.26888,
+ "90": 9.34061,
+ "91": 9.25718,
+ "92": 9.29635,
+ "93": 9.29969,
+ "94": 9.2754,
+ "95": 9.28117,
+ "96": 9.18254,
+ "97": 9.26445,
+ "98": 9.1966,
+ "99": 9.21992,
+ "100": 9.22929
}
},
"num-zeros": {
@@ -161,56 +161,56 @@
"48": "nan",
"49": "nan",
"50": "nan",
- "51": 1961.0,
- "52": 2445.0,
- "53": 3654.0,
- "54": 3489.0,
- "55": 3419.0,
- "56": 4364.0,
- "57": 4145.0,
- "58": 4155.0,
- "59": 1699.0,
- "60": 2358.0,
- "61": 2070.0,
- "62": 4094.0,
- "63": 3516.0,
- "64": 4287.0,
- "65": 2891.0,
- "66": 1733.0,
- "67": 1914.0,
- "68": 4420.0,
- "69": 4479.0,
- "70": 4656.0,
- "71": 2135.0,
- "72": 4476.0,
- "73": 4048.0,
- "74": 3199.0,
- "75": 4735.0,
- "76": 2218.0,
- "77": 4952.0,
- "78": 4158.0,
- "79": 2657.0,
- "80": 3846.0,
- "81": 3472.0,
- "82": 2979.0,
- "83": 5364.0,
- "84": 4430.0,
- "85": 4249.0,
- "86": 3509.0,
- "87": 4817.0,
- "88": 3434.0,
- "89": 4711.0,
- "90": 4448.0,
- "91": 4374.0,
- "92": 3507.0,
- "93": 5549.0,
- "94": 3635.0,
- "95": 4540.0,
- "96": 3659.0,
- "97": 3756.0,
- "98": 4513.0,
- "99": 4491.0,
- "100": 3445.0
+ "51": 1818.0,
+ "52": 2620.0,
+ "53": 3744.0,
+ "54": 3604.0,
+ "55": 3334.0,
+ "56": 4304.0,
+ "57": 4040.0,
+ "58": 4220.0,
+ "59": 1787.0,
+ "60": 2678.0,
+ "61": 2295.0,
+ "62": 3929.0,
+ "63": 3820.0,
+ "64": 4560.0,
+ "65": 3100.0,
+ "66": 2039.0,
+ "67": 2163.0,
+ "68": 4135.0,
+ "69": 4393.0,
+ "70": 4091.0,
+ "71": 2120.0,
+ "72": 4062.0,
+ "73": 3510.0,
+ "74": 2614.0,
+ "75": 5305.0,
+ "76": 2601.0,
+ "77": 4058.0,
+ "78": 4315.0,
+ "79": 2234.0,
+ "80": 3448.0,
+ "81": 4090.0,
+ "82": 3752.0,
+ "83": 4925.0,
+ "84": 5349.0,
+ "85": 4450.0,
+ "86": 4011.0,
+ "87": 3738.0,
+ "88": 4415.0,
+ "89": 3811.0,
+ "90": 4620.0,
+ "91": 4703.0,
+ "92": 4036.0,
+ "93": 3711.0,
+ "94": 3059.0,
+ "95": 4017.0,
+ "96": 3793.0,
+ "97": 3300.0,
+ "98": 4562.0,
+ "99": 3832.0,
+ "100": 3458.0
}
},
"mem-allocated-bytes": {
@@ -268,56 +268,56 @@
"48": "nan",
"49": "nan",
"50": "nan",
- "51": 1786112512.0,
- "52": 1786112512.0,
- "53": 1786112512.0,
- "54": 1786112512.0,
- "55": 1786112512.0,
- "56": 1786112512.0,
- "57": 1786112512.0,
- "58": 1786112512.0,
- "59": 1786112512.0,
- "60": 1786112512.0,
- "61": 1786112512.0,
- "62": 1786112512.0,
- "63": 1786112512.0,
- "64": 1786112512.0,
- "65": 1786112512.0,
- "66": 1786112512.0,
- "67": 1786112512.0,
- "68": 1786112512.0,
- "69": 1786112512.0,
- "70": 1786112512.0,
- "71": 1786112512.0,
- "72": 1786112512.0,
- "73": 1786112512.0,
- "74": 1786112512.0,
- "75": 1786112512.0,
- "76": 1786112512.0,
- "77": 1786112512.0,
- "78": 1786112512.0,
- "79": 1786112512.0,
- "80": 1786112512.0,
- "81": 1786112512.0,
- "82": 1786112512.0,
- "83": 1786112512.0,
- "84": 1786112512.0,
- "85": 1786112512.0,
- "86": 1786112512.0,
- "87": 1786112512.0,
- "88": 1786112512.0,
- "89": 1786112512.0,
- "90": 1786112512.0,
- "91": 1786112512.0,
- "92": 1786112512.0,
- "93": 1786112512.0,
- "94": 1786112512.0,
- "95": 1786112512.0,
- "96": 1786112512.0,
- "97": 1786112512.0,
- "98": 1786112512.0,
- "99": 1786112512.0,
- "100": 1786112512.0
+ "51": 1796646400.0,
+ "52": 1796646400.0,
+ "53": 1796646400.0,
+ "54": 1796646400.0,
+ "55": 1796646400.0,
+ "56": 1796646400.0,
+ "57": 1796646400.0,
+ "58": 1796646400.0,
+ "59": 1796646400.0,
+ "60": 1796646400.0,
+ "61": 1796646400.0,
+ "62": 1796646400.0,
+ "63": 1796646400.0,
+ "64": 1796646400.0,
+ "65": 1796646400.0,
+ "66": 1796646400.0,
+ "67": 1796646400.0,
+ "68": 1796646400.0,
+ "69": 1796646400.0,
+ "70": 1796646400.0,
+ "71": 1796646400.0,
+ "72": 1796646400.0,
+ "73": 1796646400.0,
+ "74": 1796646400.0,
+ "75": 1796646400.0,
+ "76": 1796646400.0,
+ "77": 1796646400.0,
+ "78": 1796646400.0,
+ "79": 1796646400.0,
+ "80": 1796646400.0,
+ "81": 1796646400.0,
+ "82": 1796646400.0,
+ "83": 1796646400.0,
+ "84": 1796646400.0,
+ "85": 1796646400.0,
+ "86": 1796646400.0,
+ "87": 1796646400.0,
+ "88": 1796646400.0,
+ "89": 1796646400.0,
+ "90": 1796646400.0,
+ "91": 1796646400.0,
+ "92": 1796646400.0,
+ "93": 1796646400.0,
+ "94": 1796646400.0,
+ "95": 1796646400.0,
+ "96": 1796646400.0,
+ "97": 1796646400.0,
+ "98": 1796646400.0,
+ "99": 1796646400.0,
+ "100": 1796646400.0
}
},
"mem-max-allocated-bytes": {
@@ -375,56 +375,56 @@
"48": "nan",
"49": "nan",
"50": "nan",
- "51": 3110419456.0,
- "52": 3110421504.0,
- "53": 3110421504.0,
- "54": 3110421504.0,
- "55": 3110421504.0,
- "56": 3110421504.0,
- "57": 3110421504.0,
- "58": 3110421504.0,
- "59": 3110421504.0,
- "60": 3110421504.0,
- "61": 3110421504.0,
- "62": 3110421504.0,
- "63": 3110421504.0,
- "64": 3110421504.0,
- "65": 3110421504.0,
- "66": 3110421504.0,
- "67": 3110421504.0,
- "68": 3110421504.0,
- "69": 3110421504.0,
- "70": 3110421504.0,
- "71": 3110421504.0,
- "72": 3110421504.0,
- "73": 3110421504.0,
- "74": 3110421504.0,
- "75": 3110421504.0,
- "76": 3110421504.0,
- "77": 3110421504.0,
- "78": 3110421504.0,
- "79": 3110421504.0,
- "80": 3110421504.0,
- "81": 3110421504.0,
- "82": 3110421504.0,
- "83": 3110421504.0,
- "84": 3110421504.0,
- "85": 3110421504.0,
- "86": 3110421504.0,
- "87": 3110421504.0,
- "88": 3110421504.0,
- "89": 3110421504.0,
- "90": 3110421504.0,
- "91": 3110421504.0,
- "92": 3110421504.0,
- "93": 3110421504.0,
- "94": 3110421504.0,
- "95": 3110421504.0,
- "96": 3110421504.0,
- "97": 3110421504.0,
- "98": 3110421504.0,
- "99": 3110421504.0,
- "100": 3110421504.0
+ "51": 3124624896.0,
+ "52": 3124626944.0,
+ "53": 3124626944.0,
+ "54": 3124626944.0,
+ "55": 3124626944.0,
+ "56": 3124626944.0,
+ "57": 3124626944.0,
+ "58": 3124626944.0,
+ "59": 3124626944.0,
+ "60": 3124626944.0,
+ "61": 3124626944.0,
+ "62": 3124626944.0,
+ "63": 3124626944.0,
+ "64": 3124626944.0,
+ "65": 3124626944.0,
+ "66": 3124626944.0,
+ "67": 3124626944.0,
+ "68": 3124626944.0,
+ "69": 3124626944.0,
+ "70": 3124626944.0,
+ "71": 3124626944.0,
+ "72": 3124626944.0,
+ "73": 3124626944.0,
+ "74": 3124626944.0,
+ "75": 3124626944.0,
+ "76": 3124626944.0,
+ "77": 3124626944.0,
+ "78": 3124626944.0,
+ "79": 3124626944.0,
+ "80": 3124626944.0,
+ "81": 3124626944.0,
+ "82": 3124626944.0,
+ "83": 3124626944.0,
+ "84": 3124626944.0,
+ "85": 3124626944.0,
+ "86": 3124626944.0,
+ "87": 3124626944.0,
+ "88": 3124626944.0,
+ "89": 3124626944.0,
+ "90": 3124626944.0,
+ "91": 3124626944.0,
+ "92": 3124626944.0,
+ "93": 3124626944.0,
+ "94": 3124626944.0,
+ "95": 3126723584.0,
+ "96": 3126723584.0,
+ "97": 3126723584.0,
+ "98": 3126723584.0,
+ "99": 3126723584.0,
+ "100": 3126723584.0
}
},
"iteration-time": {
@@ -482,56 +482,56 @@
"48": "nan",
"49": "nan",
"50": "nan",
- "51": 10.75043,
- "52": 1.0039,
- "53": 0.95516,
- "54": 0.91159,
- "55": 0.90836,
- "56": 0.94785,
- "57": 1.16936,
- "58": 1.19663,
- "59": 1.28755,
- "60": 0.88429,
- "61": 0.8835,
- "62": 0.91894,
- "63": 0.88317,
- "64": 0.89119,
- "65": 0.88844,
- "66": 1.26569,
- "67": 0.88764,
- "68": 0.88401,
- "69": 0.89243,
- "70": 0.8883,
- "71": 0.89113,
- "72": 0.91101,
- "73": 0.89072,
- "74": 2.04797,
- "75": 0.90184,
- "76": 0.93408,
- "77": 1.2869,
- "78": 0.95072,
- "79": 0.96458,
- "80": 0.90559,
- "81": 0.95787,
- "82": 0.90855,
- "83": 1.71942,
- "84": 0.94521,
- "85": 0.88307,
- "86": 0.88152,
- "87": 0.89039,
- "88": 0.88803,
- "89": 0.90894,
- "90": 0.89894,
- "91": 1.05886,
- "92": 1.19588,
- "93": 1.37335,
- "94": 0.8898,
- "95": 1.07004,
- "96": 0.88806,
- "97": 0.89083,
- "98": 0.90547,
- "99": 0.94317,
- "100": 0.90081
+ "51": "nan",
+ "52": 6.03911,
+ "53": 0.97416,
+ "54": 0.94779,
+ "55": 0.9496,
+ "56": 0.94003,
+ "57": 0.95104,
+ "58": 0.9583,
+ "59": 0.94134,
+ "60": 0.94902,
+ "61": 0.94593,
+ "62": 0.93544,
+ "63": 0.94434,
+ "64": 0.94553,
+ "65": 0.92679,
+ "66": 0.90378,
+ "67": 0.90184,
+ "68": 0.91076,
+ "69": 0.90954,
+ "70": 0.91372,
+ "71": 0.91281,
+ "72": 0.91175,
+ "73": 0.91046,
+ "74": 0.91357,
+ "75": 0.90873,
+ "76": 0.91033,
+ "77": 0.91092,
+ "78": 1.19718,
+ "79": 1.185,
+ "80": 1.2732,
+ "81": 0.90464,
+ "82": 0.90482,
+ "83": 0.90412,
+ "84": 0.90648,
+ "85": 0.9074,
+ "86": 0.91479,
+ "87": 0.91427,
+ "88": 0.91177,
+ "89": 0.91209,
+ "90": 0.913,
+ "91": 0.9133,
+ "92": 0.98243,
+ "93": 0.91047,
+ "94": 0.91069,
+ "95": 0.91618,
+ "96": 0.91277,
+ "97": 0.90968,
+ "98": 0.91034,
+ "99": 0.9131,
+ "100": 0.91106
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist/model_config.yaml b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist/model_config.yaml
index 49135684124..4afcb0c9d47 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist/model_config.yaml
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist/model_config.yaml
@@ -22,8 +22,8 @@ MODEL_ARGS:
--lr-decay-iters: 990000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/bert_shard00/my-bert_00_text_sentence
- --vocab-file: ${DATA_PATH}/text/the_pile/bert_shard00/vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-bert_00_text_sentence
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/vocab.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.0001
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist_local_spec/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist_local_spec/golden_values_dev_dgx_h100.json
index fe766022589..87a9a3837b2 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist_local_spec/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist_local_spec/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.4837,
- "2": 10.48435,
- "3": 10.48251,
- "4": 10.48303,
- "5": 10.47647,
- "6": 10.48423,
- "7": 10.48457,
- "8": 10.48837,
- "9": 10.49003,
- "10": 10.47255,
- "11": 10.47245,
- "12": 10.4828,
- "13": 10.47855,
- "14": 10.45162,
- "15": 10.47936,
- "16": 10.45364,
- "17": 10.45143,
- "18": 10.46239,
- "19": 10.44136,
- "20": 10.45438,
- "21": 10.43469,
- "22": 10.40587,
- "23": 10.39982,
- "24": 10.37585,
- "25": 10.38173,
- "26": 10.35154,
- "27": 10.35401,
- "28": 10.3497,
- "29": 10.28714,
- "30": 10.21194,
- "31": 10.17274,
- "32": 10.13439,
- "33": 10.14753,
- "34": 10.10759,
- "35": 10.10592,
- "36": 10.08756,
- "37": 10.08177,
- "38": 10.07257,
- "39": 10.0013,
- "40": 9.9816,
- "41": 9.92551,
- "42": 9.87537,
- "43": 9.88725,
- "44": 9.80659,
- "45": 9.82349,
- "46": 9.73821,
- "47": 9.74829,
- "48": 9.71628,
- "49": 9.74489,
- "50": 9.73004,
- "51": 9.71501,
- "52": 9.66488,
- "53": 9.60917,
- "54": 9.62733,
- "55": 9.61022,
- "56": 9.61723,
- "57": 9.56794,
- "58": 9.52733,
- "59": 9.51677,
- "60": 9.5188,
- "61": 9.53149,
- "62": 9.45031,
- "63": 9.45717,
- "64": 9.43441,
- "65": 9.45812,
- "66": 9.43672,
- "67": 9.39687,
- "68": 9.36469,
- "69": 9.40964,
- "70": 9.37606,
- "71": 9.41737,
- "72": 9.42585,
- "73": 9.37601,
- "74": 9.4154,
- "75": 9.37896,
- "76": 9.28004,
- "77": 9.32212,
- "78": 9.35755,
- "79": 9.3216,
- "80": 9.31491,
- "81": 9.26783,
- "82": 9.342,
- "83": 9.32159,
- "84": 9.24786,
- "85": 9.35018,
- "86": 9.22384,
- "87": 9.30618,
- "88": 9.29905,
- "89": 9.22708,
- "90": 9.28498,
- "91": 9.23123,
- "92": 9.27487,
- "93": 9.19233,
- "94": 9.23985,
- "95": 9.28002,
- "96": 9.17532,
- "97": 9.21898,
- "98": 9.17203,
- "99": 9.16444,
- "100": 9.14821
+ "1": 10.52538,
+ "2": 10.54405,
+ "3": 10.53279,
+ "4": 10.53976,
+ "5": 10.53691,
+ "6": 10.543,
+ "7": 10.54588,
+ "8": 10.5404,
+ "9": 10.5336,
+ "10": 10.52908,
+ "11": 10.5317,
+ "12": 10.53731,
+ "13": 10.53878,
+ "14": 10.52481,
+ "15": 10.52268,
+ "16": 10.50433,
+ "17": 10.513,
+ "18": 10.50762,
+ "19": 10.50404,
+ "20": 10.51124,
+ "21": 10.50937,
+ "22": 10.44494,
+ "23": 10.43154,
+ "24": 10.44097,
+ "25": 10.40599,
+ "26": 10.41479,
+ "27": 10.40001,
+ "28": 10.41072,
+ "29": 10.36647,
+ "30": 10.28229,
+ "31": 10.22963,
+ "32": 10.20468,
+ "33": 10.21549,
+ "34": 10.1706,
+ "35": 10.14875,
+ "36": 10.12819,
+ "37": 10.11663,
+ "38": 10.11882,
+ "39": 10.08011,
+ "40": 10.01123,
+ "41": 9.96741,
+ "42": 9.92753,
+ "43": 9.9211,
+ "44": 9.86398,
+ "45": 9.8389,
+ "46": 9.77921,
+ "47": 9.77269,
+ "48": 9.74952,
+ "49": 9.77861,
+ "50": 9.75389,
+ "51": 9.76089,
+ "52": 9.71033,
+ "53": 9.67207,
+ "54": 9.69277,
+ "55": 9.68065,
+ "56": 9.67265,
+ "57": 9.60244,
+ "58": 9.61924,
+ "59": 9.54677,
+ "60": 9.61248,
+ "61": 9.54764,
+ "62": 9.53711,
+ "63": 9.52351,
+ "64": 9.51103,
+ "65": 9.52359,
+ "66": 9.49117,
+ "67": 9.45813,
+ "68": 9.44041,
+ "69": 9.44341,
+ "70": 9.43985,
+ "71": 9.46988,
+ "72": 9.45668,
+ "73": 9.40509,
+ "74": 9.45601,
+ "75": 9.40461,
+ "76": 9.37348,
+ "77": 9.34344,
+ "78": 9.37804,
+ "79": 9.41146,
+ "80": 9.34475,
+ "81": 9.3307,
+ "82": 9.34695,
+ "83": 9.31574,
+ "84": 9.29963,
+ "85": 9.3365,
+ "86": 9.26935,
+ "87": 9.31414,
+ "88": 9.29826,
+ "89": 9.26944,
+ "90": 9.34131,
+ "91": 9.25633,
+ "92": 9.29676,
+ "93": 9.29998,
+ "94": 9.27507,
+ "95": 9.28001,
+ "96": 9.18223,
+ "97": 9.26464,
+ "98": 9.19677,
+ "99": 9.21997,
+ "100": 9.22913
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 2554.0,
- "2": 1919.0,
- "3": 1521.0,
- "4": 2330.0,
- "5": 2010.0,
- "6": 1725.0,
- "7": 2803.0,
- "8": 2435.0,
- "9": 2286.0,
- "10": 2570.0,
- "11": 2438.0,
- "12": 1829.0,
- "13": 2332.0,
- "14": 2832.0,
- "15": 2008.0,
- "16": 2659.0,
- "17": 2454.0,
- "18": 2500.0,
- "19": 2588.0,
- "20": 2834.0,
- "21": 2042.0,
- "22": 3037.0,
- "23": 2702.0,
- "24": 2700.0,
- "25": 2568.0,
- "26": 2896.0,
- "27": 2735.0,
- "28": 2699.0,
- "29": 2548.0,
- "30": 2843.0,
- "31": 2160.0,
- "32": 2458.0,
- "33": 2130.0,
- "34": 2517.0,
- "35": 2597.0,
- "36": 3001.0,
- "37": 3305.0,
- "38": 2682.0,
- "39": 2805.0,
- "40": 3430.0,
- "41": 1767.0,
- "42": 1516.0,
- "43": 1798.0,
- "44": 2790.0,
- "45": 3578.0,
- "46": 3016.0,
- "47": 2890.0,
- "48": 3065.0,
- "49": 2914.0,
- "50": 2208.0,
- "51": 1900.0,
- "52": 2483.0,
- "53": 3763.0,
- "54": 3478.0,
- "55": 3412.0,
- "56": 4400.0,
- "57": 4019.0,
- "58": 4253.0,
- "59": 1805.0,
- "60": 2457.0,
- "61": 2045.0,
- "62": 3994.0,
- "63": 3650.0,
- "64": 4466.0,
- "65": 2968.0,
- "66": 1837.0,
- "67": 1961.0,
- "68": 4347.0,
- "69": 4441.0,
- "70": 4452.0,
- "71": 2131.0,
- "72": 4523.0,
- "73": 4105.0,
- "74": 3300.0,
- "75": 4651.0,
- "76": 2216.0,
- "77": 4932.0,
- "78": 4218.0,
- "79": 2784.0,
- "80": 3824.0,
- "81": 3472.0,
- "82": 2976.0,
- "83": 5282.0,
- "84": 4464.0,
- "85": 4344.0,
- "86": 3460.0,
- "87": 4774.0,
- "88": 3426.0,
- "89": 4600.0,
- "90": 4360.0,
- "91": 4283.0,
- "92": 3362.0,
- "93": 5633.0,
- "94": 3676.0,
- "95": 4610.0,
- "96": 3449.0,
- "97": 3751.0,
- "98": 4524.0,
- "99": 4399.0,
- "100": 3295.0
+ "1": 2706.0,
+ "2": 2115.0,
+ "3": 1731.0,
+ "4": 2535.0,
+ "5": 2330.0,
+ "6": 1911.0,
+ "7": 2378.0,
+ "8": 2557.0,
+ "9": 2479.0,
+ "10": 2466.0,
+ "11": 2681.0,
+ "12": 1962.0,
+ "13": 2320.0,
+ "14": 2872.0,
+ "15": 2205.0,
+ "16": 2769.0,
+ "17": 2689.0,
+ "18": 2529.0,
+ "19": 2708.0,
+ "20": 2798.0,
+ "21": 2344.0,
+ "22": 2776.0,
+ "23": 2692.0,
+ "24": 2751.0,
+ "25": 2705.0,
+ "26": 2818.0,
+ "27": 2692.0,
+ "28": 2763.0,
+ "29": 2618.0,
+ "30": 2963.0,
+ "31": 2474.0,
+ "32": 2882.0,
+ "33": 2355.0,
+ "34": 2505.0,
+ "35": 2616.0,
+ "36": 2980.0,
+ "37": 3228.0,
+ "38": 2763.0,
+ "39": 2986.0,
+ "40": 3365.0,
+ "41": 1718.0,
+ "42": 1468.0,
+ "43": 1770.0,
+ "44": 2838.0,
+ "45": 3671.0,
+ "46": 3406.0,
+ "47": 3197.0,
+ "48": 2714.0,
+ "49": 2486.0,
+ "50": 2073.0,
+ "51": 1798.0,
+ "52": 2521.0,
+ "53": 3896.0,
+ "54": 3642.0,
+ "55": 3344.0,
+ "56": 4315.0,
+ "57": 4027.0,
+ "58": 4215.0,
+ "59": 1770.0,
+ "60": 2626.0,
+ "61": 2300.0,
+ "62": 3964.0,
+ "63": 4018.0,
+ "64": 4477.0,
+ "65": 3066.0,
+ "66": 2008.0,
+ "67": 2066.0,
+ "68": 4065.0,
+ "69": 4448.0,
+ "70": 4129.0,
+ "71": 2066.0,
+ "72": 4050.0,
+ "73": 3466.0,
+ "74": 2479.0,
+ "75": 5317.0,
+ "76": 2644.0,
+ "77": 4044.0,
+ "78": 4339.0,
+ "79": 2270.0,
+ "80": 3473.0,
+ "81": 3809.0,
+ "82": 3643.0,
+ "83": 4759.0,
+ "84": 5407.0,
+ "85": 4356.0,
+ "86": 4011.0,
+ "87": 3620.0,
+ "88": 4507.0,
+ "89": 3831.0,
+ "90": 4585.0,
+ "91": 4698.0,
+ "92": 3942.0,
+ "93": 3759.0,
+ "94": 3238.0,
+ "95": 3976.0,
+ "96": 3651.0,
+ "97": 3461.0,
+ "98": 4613.0,
+ "99": 3821.0,
+ "100": 3238.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1768285696.0,
- "2": 1768285696.0,
- "3": 1768285696.0,
- "4": 1768285696.0,
- "5": 1768285696.0,
- "6": 1768285696.0,
- "7": 1768285696.0,
- "8": 1768285696.0,
- "9": 1768285696.0,
- "10": 1768285696.0,
- "11": 1768285696.0,
- "12": 1768285696.0,
- "13": 1768285696.0,
- "14": 1768285696.0,
- "15": 1768285696.0,
- "16": 1768285696.0,
- "17": 1768285696.0,
- "18": 1768285696.0,
- "19": 1768285696.0,
- "20": 1768285696.0,
- "21": 1768285696.0,
- "22": 1768285696.0,
- "23": 1768285696.0,
- "24": 1768285696.0,
- "25": 1768285696.0,
- "26": 1768285696.0,
- "27": 1768285696.0,
- "28": 1768285696.0,
- "29": 1768285696.0,
- "30": 1768285696.0,
- "31": 1768285696.0,
- "32": 1768285696.0,
- "33": 1768285696.0,
- "34": 1768285696.0,
- "35": 1768285696.0,
- "36": 1768285696.0,
- "37": 1768285696.0,
- "38": 1768285696.0,
- "39": 1768285696.0,
- "40": 1768285696.0,
- "41": 1768285696.0,
- "42": 1768285696.0,
- "43": 1768285696.0,
- "44": 1768285696.0,
- "45": 1768285696.0,
- "46": 1768285696.0,
- "47": 1768285696.0,
- "48": 1768285696.0,
- "49": 1768285696.0,
- "50": 1768285696.0,
- "51": 1768285696.0,
- "52": 1768285696.0,
- "53": 1768285696.0,
- "54": 1768285696.0,
- "55": 1768285696.0,
- "56": 1768285696.0,
- "57": 1768285696.0,
- "58": 1768285696.0,
- "59": 1768285696.0,
- "60": 1768285696.0,
- "61": 1768285696.0,
- "62": 1768285696.0,
- "63": 1768285696.0,
- "64": 1768285696.0,
- "65": 1768285696.0,
- "66": 1768285696.0,
- "67": 1768285696.0,
- "68": 1768285696.0,
- "69": 1768285696.0,
- "70": 1768285696.0,
- "71": 1768285696.0,
- "72": 1768285696.0,
- "73": 1768285696.0,
- "74": 1769334272.0,
- "75": 1768285696.0,
- "76": 1768285696.0,
- "77": 1768285696.0,
- "78": 1768285696.0,
- "79": 1768285696.0,
- "80": 1768285696.0,
- "81": 1768285696.0,
- "82": 1768285696.0,
- "83": 1768285696.0,
- "84": 1768285696.0,
- "85": 1768285696.0,
- "86": 1768285696.0,
- "87": 1768285696.0,
- "88": 1768285696.0,
- "89": 1768285696.0,
- "90": 1768285696.0,
- "91": 1768285696.0,
- "92": 1768285696.0,
- "93": 1768285696.0,
- "94": 1768285696.0,
- "95": 1768285696.0,
- "96": 1768285696.0,
- "97": 1768285696.0,
- "98": 1768285696.0,
- "99": 1768285696.0,
- "100": 1768285696.0
+ "1": 1779868160.0,
+ "2": 1779868160.0,
+ "3": 1779868160.0,
+ "4": 1779868160.0,
+ "5": 1779868160.0,
+ "6": 1779868160.0,
+ "7": 1779868160.0,
+ "8": 1779868160.0,
+ "9": 1779868160.0,
+ "10": 1779868160.0,
+ "11": 1779868160.0,
+ "12": 1779868160.0,
+ "13": 1779868160.0,
+ "14": 1779868160.0,
+ "15": 1779868160.0,
+ "16": 1779868160.0,
+ "17": 1779868160.0,
+ "18": 1779868160.0,
+ "19": 1779868160.0,
+ "20": 1779868160.0,
+ "21": 1779868160.0,
+ "22": 1779868160.0,
+ "23": 1779868160.0,
+ "24": 1779868160.0,
+ "25": 1779868160.0,
+ "26": 1779868160.0,
+ "27": 1779868160.0,
+ "28": 1779868160.0,
+ "29": 1779868160.0,
+ "30": 1779868160.0,
+ "31": 1779868160.0,
+ "32": 1779868160.0,
+ "33": 1779868160.0,
+ "34": 1779868160.0,
+ "35": 1779868160.0,
+ "36": 1779868160.0,
+ "37": 1779868160.0,
+ "38": 1779868160.0,
+ "39": 1779868160.0,
+ "40": 1779868160.0,
+ "41": 1779868160.0,
+ "42": 1779868160.0,
+ "43": 1779868160.0,
+ "44": 1779868160.0,
+ "45": 1779868160.0,
+ "46": 1779868160.0,
+ "47": 1779868160.0,
+ "48": 1779868160.0,
+ "49": 1779868160.0,
+ "50": 1779868160.0,
+ "51": 1779868160.0,
+ "52": 1779868160.0,
+ "53": 1779868160.0,
+ "54": 1779868160.0,
+ "55": 1779868160.0,
+ "56": 1779868160.0,
+ "57": 1779868160.0,
+ "58": 1779868160.0,
+ "59": 1779868160.0,
+ "60": 1779868160.0,
+ "61": 1779868160.0,
+ "62": 1779868160.0,
+ "63": 1779868160.0,
+ "64": 1779868160.0,
+ "65": 1779868160.0,
+ "66": 1779868160.0,
+ "67": 1779868160.0,
+ "68": 1779868160.0,
+ "69": 1779868160.0,
+ "70": 1779868160.0,
+ "71": 1779868160.0,
+ "72": 1779868160.0,
+ "73": 1779868160.0,
+ "74": 1779868160.0,
+ "75": 1779868160.0,
+ "76": 1779868160.0,
+ "77": 1779868160.0,
+ "78": 1779868160.0,
+ "79": 1779868160.0,
+ "80": 1779868160.0,
+ "81": 1779868160.0,
+ "82": 1779868160.0,
+ "83": 1779868160.0,
+ "84": 1779868160.0,
+ "85": 1779868160.0,
+ "86": 1779868160.0,
+ "87": 1779868160.0,
+ "88": 1779868160.0,
+ "89": 1779868160.0,
+ "90": 1779868160.0,
+ "91": 1779868160.0,
+ "92": 1779868160.0,
+ "93": 1779868160.0,
+ "94": 1779868160.0,
+ "95": 1779868160.0,
+ "96": 1779868160.0,
+ "97": 1779868160.0,
+ "98": 1779868160.0,
+ "99": 1779868160.0,
+ "100": 1779868160.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 2337549312.0,
- "2": 3080536064.0,
- "3": 3080536064.0,
- "4": 3080536064.0,
- "5": 3080536064.0,
- "6": 3080536064.0,
- "7": 3080536064.0,
- "8": 3080536064.0,
- "9": 3080536064.0,
- "10": 3080536064.0,
- "11": 3080536064.0,
- "12": 3080536064.0,
- "13": 3080536064.0,
- "14": 3080536064.0,
- "15": 3080536064.0,
- "16": 3080536064.0,
- "17": 3080536064.0,
- "18": 3080536064.0,
- "19": 3080536064.0,
- "20": 3080536064.0,
- "21": 3080536064.0,
- "22": 3080536064.0,
- "23": 3082107392.0,
- "24": 3082107392.0,
- "25": 3082107392.0,
- "26": 3082107392.0,
- "27": 3082107392.0,
- "28": 3082107392.0,
- "29": 3082107392.0,
- "30": 3082107392.0,
- "31": 3082107392.0,
- "32": 3082107392.0,
- "33": 3082107392.0,
- "34": 3082107392.0,
- "35": 3082107392.0,
- "36": 3082107392.0,
- "37": 3082107392.0,
- "38": 3082107392.0,
- "39": 3082107392.0,
- "40": 3082107392.0,
- "41": 3082107392.0,
- "42": 3082107392.0,
- "43": 3082107392.0,
- "44": 3082107392.0,
- "45": 3082107392.0,
- "46": 3082107392.0,
- "47": 3082107392.0,
- "48": 3082107392.0,
- "49": 3082107392.0,
- "50": 3082107392.0,
- "51": 3082107392.0,
- "52": 3082107392.0,
- "53": 3082107392.0,
- "54": 3082107392.0,
- "55": 3082107392.0,
- "56": 3082107392.0,
- "57": 3082107392.0,
- "58": 3082107392.0,
- "59": 3082107392.0,
- "60": 3082107392.0,
- "61": 3082107392.0,
- "62": 3082107392.0,
- "63": 3082107392.0,
- "64": 3082107392.0,
- "65": 3082107392.0,
- "66": 3082107392.0,
- "67": 3082107392.0,
- "68": 3082107392.0,
- "69": 3082107392.0,
- "70": 3082107392.0,
- "71": 3082107392.0,
- "72": 3082107392.0,
- "73": 3082107392.0,
- "74": 3082108928.0,
- "75": 3082108928.0,
- "76": 3082108928.0,
- "77": 3082108928.0,
- "78": 3082108928.0,
- "79": 3082108928.0,
- "80": 3082108928.0,
- "81": 3082108928.0,
- "82": 3082108928.0,
- "83": 3082108928.0,
- "84": 3082108928.0,
- "85": 3082108928.0,
- "86": 3082108928.0,
- "87": 3082108928.0,
- "88": 3082108928.0,
- "89": 3082108928.0,
- "90": 3082108928.0,
- "91": 3082108928.0,
- "92": 3082108928.0,
- "93": 3082108928.0,
- "94": 3082108928.0,
- "95": 3082108928.0,
- "96": 3082108928.0,
- "97": 3082108928.0,
- "98": 3082108928.0,
- "99": 3082108928.0,
- "100": 3082108928.0
+ "1": 2347554304.0,
+ "2": 3095265792.0,
+ "3": 3095265792.0,
+ "4": 3095265792.0,
+ "5": 3097362432.0,
+ "6": 3097362432.0,
+ "7": 3097362432.0,
+ "8": 3097362432.0,
+ "9": 3097362432.0,
+ "10": 3097362432.0,
+ "11": 3097362432.0,
+ "12": 3097362432.0,
+ "13": 3097362432.0,
+ "14": 3097362432.0,
+ "15": 3097362432.0,
+ "16": 3097362432.0,
+ "17": 3097362432.0,
+ "18": 3097362432.0,
+ "19": 3097362432.0,
+ "20": 3097362432.0,
+ "21": 3097362432.0,
+ "22": 3097362432.0,
+ "23": 3097362432.0,
+ "24": 3097362432.0,
+ "25": 3097362432.0,
+ "26": 3097362432.0,
+ "27": 3097362432.0,
+ "28": 3097362432.0,
+ "29": 3097362432.0,
+ "30": 3097362432.0,
+ "31": 3097362432.0,
+ "32": 3097362432.0,
+ "33": 3097362432.0,
+ "34": 3097362432.0,
+ "35": 3097362432.0,
+ "36": 3097362432.0,
+ "37": 3097362432.0,
+ "38": 3097362432.0,
+ "39": 3097362432.0,
+ "40": 3097362432.0,
+ "41": 3097362432.0,
+ "42": 3097362432.0,
+ "43": 3097362432.0,
+ "44": 3097362432.0,
+ "45": 3097362432.0,
+ "46": 3097362432.0,
+ "47": 3097362432.0,
+ "48": 3097362432.0,
+ "49": 3097362432.0,
+ "50": 3097362432.0,
+ "51": 3097362432.0,
+ "52": 3097362432.0,
+ "53": 3097362432.0,
+ "54": 3097362432.0,
+ "55": 3097362432.0,
+ "56": 3097362432.0,
+ "57": 3097362432.0,
+ "58": 3097362432.0,
+ "59": 3097362432.0,
+ "60": 3097362432.0,
+ "61": 3097362432.0,
+ "62": 3097362432.0,
+ "63": 3097362432.0,
+ "64": 3097362432.0,
+ "65": 3097362432.0,
+ "66": 3097362432.0,
+ "67": 3097362432.0,
+ "68": 3097362432.0,
+ "69": 3097362432.0,
+ "70": 3097362432.0,
+ "71": 3097362432.0,
+ "72": 3097362432.0,
+ "73": 3097362432.0,
+ "74": 3097362432.0,
+ "75": 3097362432.0,
+ "76": 3097362432.0,
+ "77": 3097362432.0,
+ "78": 3097362432.0,
+ "79": 3097362432.0,
+ "80": 3097362944.0,
+ "81": 3097362944.0,
+ "82": 3097362944.0,
+ "83": 3097362944.0,
+ "84": 3097362944.0,
+ "85": 3097362944.0,
+ "86": 3097362944.0,
+ "87": 3097362944.0,
+ "88": 3097362944.0,
+ "89": 3097362944.0,
+ "90": 3097362944.0,
+ "91": 3097362944.0,
+ "92": 3097362944.0,
+ "93": 3097362944.0,
+ "94": 3097362944.0,
+ "95": 3097362944.0,
+ "96": 3097362944.0,
+ "97": 3097362944.0,
+ "98": 3097362944.0,
+ "99": 3097362944.0,
+ "100": 3097362944.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.24286,
- "2": 0.82679,
- "3": 0.79409,
- "4": 0.76435,
- "5": 0.77118,
- "6": 0.74558,
- "7": 0.74667,
- "8": 0.77701,
- "9": 1.97605,
- "10": 0.75455,
- "11": 0.74398,
- "12": 0.74114,
- "13": 0.7501,
- "14": 0.74704,
- "15": 0.74029,
- "16": 1.1307,
- "17": 0.73862,
- "18": 0.73445,
- "19": 0.73384,
- "20": 0.73927,
- "21": 0.74153,
- "22": 0.73755,
- "23": 0.76958,
- "24": 0.7377,
- "25": 0.73987,
- "26": 0.77483,
- "27": 1.30185,
- "28": 0.76,
- "29": 0.75644,
- "30": 0.77716,
- "31": 0.83125,
- "32": 0.80226,
- "33": 0.74041,
- "34": 0.74334,
- "35": 1.17386,
- "36": 1.53868,
- "37": 0.77003,
- "38": 0.76358,
- "39": 0.77015,
- "40": 0.77216,
- "41": 0.76865,
- "42": 1.214,
- "43": 1.04802,
- "44": 0.758,
- "45": 1.27424,
- "46": 1.12734,
- "47": 0.7573,
- "48": 0.74875,
- "49": 0.74989,
- "50": 0.75416,
- "51": 0.75904,
- "52": 0.75338,
- "53": 0.75124,
- "54": 0.73937,
- "55": 0.74096,
- "56": 0.75129,
- "57": 0.75097,
- "58": 0.74724,
- "59": 0.74661,
- "60": 0.74245,
- "61": 0.74378,
- "62": 0.74491,
- "63": 0.74147,
- "64": 0.74756,
- "65": 0.74511,
- "66": 0.74967,
- "67": 0.7462,
- "68": 0.74176,
- "69": 0.74258,
- "70": 0.74323,
- "71": 0.74412,
- "72": 0.74522,
- "73": 0.74053,
- "74": 0.74312,
- "75": 0.74157,
- "76": 1.12862,
- "77": 0.74522,
- "78": 1.08987,
- "79": 0.94746,
- "80": 0.877,
- "81": 0.74472,
- "82": 0.74142,
- "83": 0.74342,
- "84": 0.7418,
- "85": 0.74017,
- "86": 0.7399,
- "87": 0.73594,
- "88": 0.73916,
- "89": 0.73537,
- "90": 0.75037,
- "91": 0.7341,
- "92": 0.73469,
- "93": 0.7333,
- "94": 0.73221,
- "95": 0.73055,
- "96": 0.73133,
- "97": 0.73591,
- "98": 0.74108,
- "99": 0.74467,
- "100": 0.73711
+ "1": "nan",
+ "2": 6.58997,
+ "3": 0.78114,
+ "4": 0.7669,
+ "5": 0.76339,
+ "6": 0.76181,
+ "7": 0.76359,
+ "8": 0.76853,
+ "9": 0.76406,
+ "10": 0.76522,
+ "11": 0.76932,
+ "12": 0.77188,
+ "13": 1.03387,
+ "14": 0.77087,
+ "15": 0.76243,
+ "16": 0.77006,
+ "17": 0.78137,
+ "18": 0.78189,
+ "19": 0.78203,
+ "20": 1.36049,
+ "21": 0.77951,
+ "22": 1.30811,
+ "23": 1.06829,
+ "24": 1.08426,
+ "25": 0.77253,
+ "26": 1.09007,
+ "27": 0.76186,
+ "28": 0.76255,
+ "29": 0.76136,
+ "30": 0.76279,
+ "31": 0.7638,
+ "32": 0.76071,
+ "33": 0.77361,
+ "34": 0.76883,
+ "35": 0.76901,
+ "36": 0.76148,
+ "37": 0.76069,
+ "38": 0.7629,
+ "39": 0.76414,
+ "40": 0.75813,
+ "41": 0.75661,
+ "42": 0.75626,
+ "43": 0.75805,
+ "44": 0.75655,
+ "45": 0.757,
+ "46": 0.76864,
+ "47": 0.78359,
+ "48": 0.95931,
+ "49": 1.04485,
+ "50": 1.23733,
+ "51": 0.89192,
+ "52": 1.03607,
+ "53": 0.76537,
+ "54": 1.19702,
+ "55": 0.76311,
+ "56": 0.75511,
+ "57": 0.75574,
+ "58": 0.79074,
+ "59": 0.94641,
+ "60": 0.7597,
+ "61": 0.75771,
+ "62": 0.75846,
+ "63": 0.76455,
+ "64": 0.76221,
+ "65": 0.76078,
+ "66": 0.76248,
+ "67": 0.76124,
+ "68": 0.75924,
+ "69": 0.76384,
+ "70": 0.76354,
+ "71": 0.76298,
+ "72": 0.76259,
+ "73": 0.76264,
+ "74": 0.75954,
+ "75": 1.15842,
+ "76": 0.96979,
+ "77": 0.75738,
+ "78": 0.95879,
+ "79": 0.98573,
+ "80": 0.98464,
+ "81": 0.98381,
+ "82": 0.75507,
+ "83": 1.39042,
+ "84": 0.75587,
+ "85": 0.75366,
+ "86": 0.75456,
+ "87": 0.75427,
+ "88": 1.47865,
+ "89": 0.76061,
+ "90": 0.75856,
+ "91": 0.75496,
+ "92": 0.75483,
+ "93": 1.3078,
+ "94": 1.22248,
+ "95": 0.75751,
+ "96": 0.75614,
+ "97": 0.75908,
+ "98": 0.75746,
+ "99": 0.75722,
+ "100": 0.76347
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist_local_spec/golden_values_dev_dgx_h100_2nd.json b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist_local_spec/golden_values_dev_dgx_h100_2nd.json
index de97d194787..5369a1b5598 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist_local_spec/golden_values_dev_dgx_h100_2nd.json
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist_local_spec/golden_values_dev_dgx_h100_2nd.json
@@ -54,56 +54,56 @@
"48": "nan",
"49": "nan",
"50": "nan",
- "51": 9.71501,
- "52": 9.66488,
- "53": 9.60917,
- "54": 9.62733,
- "55": 9.61022,
- "56": 9.61723,
- "57": 9.56794,
- "58": 9.52733,
- "59": 9.51677,
- "60": 9.5188,
- "61": 9.53149,
- "62": 9.45031,
- "63": 9.45717,
- "64": 9.43441,
- "65": 9.45812,
- "66": 9.43672,
- "67": 9.39687,
- "68": 9.36469,
- "69": 9.40964,
- "70": 9.37606,
- "71": 9.41737,
- "72": 9.42585,
- "73": 9.37601,
- "74": 9.4154,
- "75": 9.37896,
- "76": 9.28004,
- "77": 9.32212,
- "78": 9.35755,
- "79": 9.3216,
- "80": 9.31491,
- "81": 9.26783,
- "82": 9.342,
- "83": 9.32159,
- "84": 9.24786,
- "85": 9.35018,
- "86": 9.22384,
- "87": 9.30618,
- "88": 9.29905,
- "89": 9.22708,
- "90": 9.28498,
- "91": 9.23123,
- "92": 9.27487,
- "93": 9.19233,
- "94": 9.23985,
- "95": 9.28002,
- "96": 9.17532,
- "97": 9.21898,
- "98": 9.17203,
- "99": 9.16444,
- "100": 9.14821
+ "51": 9.76077,
+ "52": 9.71037,
+ "53": 9.67209,
+ "54": 9.69289,
+ "55": 9.67976,
+ "56": 9.67195,
+ "57": 9.60235,
+ "58": 9.62054,
+ "59": 9.54681,
+ "60": 9.61233,
+ "61": 9.54777,
+ "62": 9.53717,
+ "63": 9.523,
+ "64": 9.51039,
+ "65": 9.52262,
+ "66": 9.49082,
+ "67": 9.45855,
+ "68": 9.44092,
+ "69": 9.44399,
+ "70": 9.43963,
+ "71": 9.46958,
+ "72": 9.45728,
+ "73": 9.40619,
+ "74": 9.45672,
+ "75": 9.40517,
+ "76": 9.37395,
+ "77": 9.34295,
+ "78": 9.37775,
+ "79": 9.41154,
+ "80": 9.34542,
+ "81": 9.33056,
+ "82": 9.34678,
+ "83": 9.31539,
+ "84": 9.2998,
+ "85": 9.33665,
+ "86": 9.26983,
+ "87": 9.31445,
+ "88": 9.29902,
+ "89": 9.27005,
+ "90": 9.34069,
+ "91": 9.25634,
+ "92": 9.29607,
+ "93": 9.29997,
+ "94": 9.27642,
+ "95": 9.28119,
+ "96": 9.18249,
+ "97": 9.26425,
+ "98": 9.19693,
+ "99": 9.22054,
+ "100": 9.23002
}
},
"num-zeros": {
@@ -161,56 +161,56 @@
"48": "nan",
"49": "nan",
"50": "nan",
- "51": 1900.0,
- "52": 2483.0,
- "53": 3763.0,
- "54": 3478.0,
- "55": 3412.0,
- "56": 4400.0,
- "57": 4019.0,
- "58": 4253.0,
- "59": 1805.0,
- "60": 2457.0,
- "61": 2045.0,
- "62": 3994.0,
- "63": 3650.0,
- "64": 4466.0,
- "65": 2968.0,
- "66": 1837.0,
- "67": 1961.0,
- "68": 4347.0,
- "69": 4441.0,
- "70": 4452.0,
- "71": 2131.0,
- "72": 4523.0,
- "73": 4105.0,
- "74": 3300.0,
- "75": 4651.0,
- "76": 2216.0,
- "77": 4932.0,
- "78": 4218.0,
- "79": 2784.0,
- "80": 3824.0,
- "81": 3472.0,
- "82": 2976.0,
- "83": 5282.0,
- "84": 4464.0,
- "85": 4344.0,
- "86": 3460.0,
- "87": 4774.0,
- "88": 3426.0,
- "89": 4600.0,
- "90": 4360.0,
- "91": 4283.0,
- "92": 3362.0,
- "93": 5633.0,
- "94": 3676.0,
- "95": 4610.0,
- "96": 3449.0,
- "97": 3751.0,
- "98": 4524.0,
- "99": 4399.0,
- "100": 3295.0
+ "51": 1745.0,
+ "52": 2480.0,
+ "53": 3940.0,
+ "54": 3685.0,
+ "55": 3360.0,
+ "56": 4255.0,
+ "57": 3929.0,
+ "58": 4275.0,
+ "59": 1739.0,
+ "60": 2646.0,
+ "61": 2218.0,
+ "62": 3977.0,
+ "63": 3950.0,
+ "64": 4525.0,
+ "65": 3019.0,
+ "66": 1895.0,
+ "67": 2133.0,
+ "68": 4062.0,
+ "69": 4413.0,
+ "70": 4046.0,
+ "71": 2204.0,
+ "72": 4079.0,
+ "73": 3476.0,
+ "74": 2504.0,
+ "75": 5344.0,
+ "76": 2574.0,
+ "77": 3970.0,
+ "78": 4325.0,
+ "79": 2280.0,
+ "80": 3423.0,
+ "81": 3927.0,
+ "82": 3691.0,
+ "83": 4786.0,
+ "84": 5548.0,
+ "85": 4400.0,
+ "86": 3970.0,
+ "87": 3605.0,
+ "88": 4496.0,
+ "89": 3951.0,
+ "90": 4531.0,
+ "91": 4630.0,
+ "92": 4008.0,
+ "93": 3810.0,
+ "94": 3128.0,
+ "95": 4023.0,
+ "96": 3596.0,
+ "97": 3401.0,
+ "98": 4733.0,
+ "99": 3799.0,
+ "100": 3329.0
}
},
"mem-allocated-bytes": {
@@ -268,56 +268,56 @@
"48": "nan",
"49": "nan",
"50": "nan",
- "51": 1769334272.0,
- "52": 1769334272.0,
- "53": 1769334272.0,
- "54": 1769334272.0,
- "55": 1769334272.0,
- "56": 1769334272.0,
- "57": 1769334272.0,
- "58": 1769334272.0,
- "59": 1769334272.0,
- "60": 1769334272.0,
- "61": 1769334272.0,
- "62": 1769334272.0,
- "63": 1769334272.0,
- "64": 1769334272.0,
- "65": 1769334272.0,
- "66": 1769334272.0,
- "67": 1769334272.0,
- "68": 1769334272.0,
- "69": 1769334272.0,
- "70": 1769334272.0,
- "71": 1769334272.0,
- "72": 1769334272.0,
- "73": 1769334272.0,
- "74": 1769334272.0,
- "75": 1769334272.0,
- "76": 1769334272.0,
- "77": 1769334272.0,
- "78": 1769334272.0,
- "79": 1769334272.0,
- "80": 1769334272.0,
- "81": 1769334272.0,
- "82": 1769334272.0,
- "83": 1769334272.0,
- "84": 1769334272.0,
- "85": 1769334272.0,
- "86": 1769334272.0,
- "87": 1769334272.0,
- "88": 1769334272.0,
- "89": 1769334272.0,
- "90": 1769334272.0,
- "91": 1769334272.0,
- "92": 1769334272.0,
- "93": 1769334272.0,
- "94": 1769334272.0,
- "95": 1769334272.0,
- "96": 1769334272.0,
- "97": 1769334272.0,
- "98": 1769334272.0,
- "99": 1769334272.0,
- "100": 1769334272.0
+ "51": 1779868160.0,
+ "52": 1779868160.0,
+ "53": 1779868160.0,
+ "54": 1779868160.0,
+ "55": 1779868160.0,
+ "56": 1779868160.0,
+ "57": 1779868160.0,
+ "58": 1779868160.0,
+ "59": 1779868160.0,
+ "60": 1779868160.0,
+ "61": 1779868160.0,
+ "62": 1779868160.0,
+ "63": 1779868160.0,
+ "64": 1779868160.0,
+ "65": 1779868160.0,
+ "66": 1779868160.0,
+ "67": 1779868160.0,
+ "68": 1779868160.0,
+ "69": 1779868160.0,
+ "70": 1779868160.0,
+ "71": 1779868160.0,
+ "72": 1779868160.0,
+ "73": 1779868160.0,
+ "74": 1779868160.0,
+ "75": 1779868160.0,
+ "76": 1779868160.0,
+ "77": 1779868160.0,
+ "78": 1779868160.0,
+ "79": 1779868160.0,
+ "80": 1779868160.0,
+ "81": 1779868160.0,
+ "82": 1779868160.0,
+ "83": 1779868160.0,
+ "84": 1779868160.0,
+ "85": 1779868160.0,
+ "86": 1779868160.0,
+ "87": 1779868160.0,
+ "88": 1779868160.0,
+ "89": 1779868160.0,
+ "90": 1779868160.0,
+ "91": 1779868160.0,
+ "92": 1779868160.0,
+ "93": 1779868160.0,
+ "94": 1779868160.0,
+ "95": 1779868160.0,
+ "96": 1779868160.0,
+ "97": 1779868160.0,
+ "98": 1779868160.0,
+ "99": 1779868160.0,
+ "100": 1779868160.0
}
},
"mem-max-allocated-bytes": {
@@ -375,56 +375,56 @@
"48": "nan",
"49": "nan",
"50": "nan",
- "51": 3081058304.0,
- "52": 3081060352.0,
- "53": 3081060352.0,
- "54": 3081060352.0,
- "55": 3081060352.0,
- "56": 3081060352.0,
- "57": 3081060352.0,
- "58": 3081060352.0,
- "59": 3081060352.0,
- "60": 3081060352.0,
- "61": 3081060352.0,
- "62": 3081060352.0,
- "63": 3081060352.0,
- "64": 3081060352.0,
- "65": 3081060352.0,
- "66": 3081060352.0,
- "67": 3081060352.0,
- "68": 3081060352.0,
- "69": 3081060352.0,
- "70": 3081060352.0,
- "71": 3081060352.0,
- "72": 3081060352.0,
- "73": 3081060352.0,
- "74": 3081060352.0,
- "75": 3081060352.0,
- "76": 3081060352.0,
- "77": 3081060352.0,
- "78": 3081060352.0,
- "79": 3081060352.0,
- "80": 3081060352.0,
- "81": 3081060352.0,
- "82": 3081060352.0,
- "83": 3081060352.0,
- "84": 3081060352.0,
- "85": 3081060352.0,
- "86": 3081060352.0,
- "87": 3081060352.0,
- "88": 3081060352.0,
- "89": 3081060352.0,
- "90": 3081060352.0,
- "91": 3081060352.0,
- "92": 3081060352.0,
- "93": 3081060352.0,
- "94": 3081060352.0,
- "95": 3081060352.0,
- "96": 3081060352.0,
- "97": 3081060352.0,
- "98": 3081060352.0,
- "99": 3081060352.0,
- "100": 3081060352.0
+ "51": 3095263744.0,
+ "52": 3095265792.0,
+ "53": 3095265792.0,
+ "54": 3095265792.0,
+ "55": 3095265792.0,
+ "56": 3095265792.0,
+ "57": 3095265792.0,
+ "58": 3095265792.0,
+ "59": 3095265792.0,
+ "60": 3095265792.0,
+ "61": 3095265792.0,
+ "62": 3095265792.0,
+ "63": 3095265792.0,
+ "64": 3095265792.0,
+ "65": 3095265792.0,
+ "66": 3095265792.0,
+ "67": 3095265792.0,
+ "68": 3095265792.0,
+ "69": 3095265792.0,
+ "70": 3095265792.0,
+ "71": 3095265792.0,
+ "72": 3095265792.0,
+ "73": 3095265792.0,
+ "74": 3095265792.0,
+ "75": 3095265792.0,
+ "76": 3095265792.0,
+ "77": 3095265792.0,
+ "78": 3095265792.0,
+ "79": 3095265792.0,
+ "80": 3095265792.0,
+ "81": 3095265792.0,
+ "82": 3095265792.0,
+ "83": 3095265792.0,
+ "84": 3095265792.0,
+ "85": 3095265792.0,
+ "86": 3095265792.0,
+ "87": 3095265792.0,
+ "88": 3095265792.0,
+ "89": 3095265792.0,
+ "90": 3095265792.0,
+ "91": 3095265792.0,
+ "92": 3095265792.0,
+ "93": 3095265792.0,
+ "94": 3095265792.0,
+ "95": 3095265792.0,
+ "96": 3095265792.0,
+ "97": 3095265792.0,
+ "98": 3095265792.0,
+ "99": 3095265792.0,
+ "100": 3095265792.0
}
},
"iteration-time": {
@@ -482,56 +482,56 @@
"48": "nan",
"49": "nan",
"50": "nan",
- "51": 10.15551,
- "52": 0.8598,
- "53": 0.74904,
- "54": 0.7512,
- "55": 0.75011,
- "56": 0.7593,
- "57": 1.36317,
- "58": 1.3678,
- "59": 0.75114,
- "60": 0.74624,
- "61": 0.74824,
- "62": 0.75285,
- "63": 0.75097,
- "64": 0.7539,
- "65": 1.11179,
- "66": 0.7482,
- "67": 0.75224,
- "68": 0.75225,
- "69": 0.73791,
- "70": 0.74141,
- "71": 0.74372,
- "72": 0.74097,
- "73": 1.17879,
- "74": 1.13369,
- "75": 0.75135,
- "76": 0.74737,
- "77": 0.7455,
- "78": 0.74472,
- "79": 1.10005,
- "80": 0.74804,
- "81": 0.75235,
- "82": 2.07286,
- "83": 0.74595,
- "84": 0.75659,
- "85": 0.74796,
- "86": 0.73902,
- "87": 0.73952,
- "88": 0.73743,
- "89": 0.74161,
- "90": 0.94861,
- "91": 0.94405,
- "92": 1.05613,
- "93": 1.27634,
- "94": 0.80928,
- "95": 0.77886,
- "96": 1.11223,
- "97": 0.73925,
- "98": 0.773,
- "99": 0.74424,
- "100": 0.78256
+ "51": "nan",
+ "52": 5.73605,
+ "53": 0.77824,
+ "54": 0.75725,
+ "55": 0.7532,
+ "56": 0.7616,
+ "57": 0.75553,
+ "58": 0.75296,
+ "59": 0.75491,
+ "60": 0.75523,
+ "61": 0.75657,
+ "62": 0.75208,
+ "63": 0.75617,
+ "64": 0.75276,
+ "65": 0.75464,
+ "66": 0.75206,
+ "67": 0.75496,
+ "68": 0.74988,
+ "69": 0.75191,
+ "70": 0.75373,
+ "71": 0.75115,
+ "72": 0.75283,
+ "73": 0.75392,
+ "74": 0.81749,
+ "75": 1.57609,
+ "76": 1.0807,
+ "77": 1.03733,
+ "78": 0.75769,
+ "79": 0.76457,
+ "80": 0.76184,
+ "81": 0.75954,
+ "82": 0.76037,
+ "83": 0.74979,
+ "84": 0.74721,
+ "85": 0.75789,
+ "86": 0.75931,
+ "87": 0.75373,
+ "88": 0.75519,
+ "89": 0.77826,
+ "90": 0.77393,
+ "91": 0.7716,
+ "92": 0.77446,
+ "93": 0.77897,
+ "94": 0.77838,
+ "95": 0.7783,
+ "96": 0.77066,
+ "97": 0.77269,
+ "98": 0.78271,
+ "99": 0.76978,
+ "100": 0.77044
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist_local_spec/model_config.yaml b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist_local_spec/model_config.yaml
index 6c0dc550515..8a776e6bfe5 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist_local_spec/model_config.yaml
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp2_pp2_resume_torch_dist_local_spec/model_config.yaml
@@ -22,8 +22,8 @@ MODEL_ARGS:
--lr-decay-iters: 990000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/bert_shard00/my-bert_00_text_sentence
- --vocab-file: ${DATA_PATH}/text/the_pile/bert_shard00/vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-bert_00_text_sentence
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/vocab.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.0001
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp4_pp1/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/bert/bert_mcore_tp4_pp1/golden_values_dev_dgx_h100.json
index bc0ee3bcb1e..2af04002e5b 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp4_pp1/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp4_pp1/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.42626,
- "2": 10.41171,
- "3": 10.41885,
- "4": 10.42153,
- "5": 10.42192,
- "6": 10.41563,
- "7": 10.42859,
- "8": 10.42079,
- "9": 10.43013,
- "10": 10.4087,
- "11": 10.43493,
- "12": 10.40244,
- "13": 10.42282,
- "14": 10.41239,
- "15": 10.40952,
- "16": 10.40789,
- "17": 10.38944,
- "18": 10.38859,
- "19": 10.37154,
- "20": 10.40445,
- "21": 10.36609,
- "22": 10.34962,
- "23": 10.354,
- "24": 10.30131,
- "25": 10.3111,
- "26": 10.30252,
- "27": 10.28202,
- "28": 10.27924,
- "29": 10.23941,
- "30": 10.14739,
- "31": 10.10547,
- "32": 10.09424,
- "33": 10.09034,
- "34": 10.0645,
- "35": 10.04644,
- "36": 10.03308,
- "37": 10.00522,
- "38": 10.00297,
- "39": 9.91428,
- "40": 9.91112,
- "41": 9.86566,
- "42": 9.78083,
- "43": 9.79476,
- "44": 9.73084,
- "45": 9.74269,
- "46": 9.63796,
- "47": 9.68694,
- "48": 9.63705,
- "49": 9.65524,
- "50": 9.65788
+ "1": 10.51871,
+ "2": 10.4956,
+ "3": 10.51255,
+ "4": 10.50998,
+ "5": 10.49587,
+ "6": 10.48692,
+ "7": 10.49277,
+ "8": 10.50311,
+ "9": 10.50245,
+ "10": 10.49512,
+ "11": 10.49932,
+ "12": 10.51527,
+ "13": 10.49255,
+ "14": 10.48631,
+ "15": 10.50134,
+ "16": 10.4797,
+ "17": 10.47762,
+ "18": 10.48507,
+ "19": 10.47929,
+ "20": 10.47261,
+ "21": 10.47354,
+ "22": 10.42928,
+ "23": 10.41817,
+ "24": 10.41344,
+ "25": 10.40719,
+ "26": 10.3839,
+ "27": 10.37341,
+ "28": 10.37565,
+ "29": 10.32619,
+ "30": 10.24178,
+ "31": 10.23444,
+ "32": 10.19319,
+ "33": 10.20664,
+ "34": 10.16671,
+ "35": 10.15801,
+ "36": 10.12188,
+ "37": 10.11365,
+ "38": 10.10678,
+ "39": 10.06551,
+ "40": 10.02612,
+ "41": 9.98665,
+ "42": 9.92401,
+ "43": 9.90815,
+ "44": 9.88451,
+ "45": 9.84976,
+ "46": 9.81212,
+ "47": 9.79634,
+ "48": 9.76934,
+ "49": 9.82731,
+ "50": 9.78659
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 3452.0,
- "2": 2890.0,
- "3": 1856.0,
- "4": 3256.0,
- "5": 3333.0,
- "6": 2985.0,
- "7": 3208.0,
- "8": 3314.0,
- "9": 3210.0,
- "10": 3297.0,
- "11": 2833.0,
- "12": 2982.0,
- "13": 3178.0,
- "14": 3705.0,
- "15": 3252.0,
- "16": 3615.0,
- "17": 3789.0,
- "18": 3620.0,
- "19": 3327.0,
- "20": 3539.0,
- "21": 3129.0,
- "22": 3597.0,
- "23": 3595.0,
- "24": 2781.0,
- "25": 3585.0,
- "26": 3607.0,
- "27": 4015.0,
- "28": 3836.0,
- "29": 3716.0,
- "30": 4150.0,
- "31": 3472.0,
- "32": 3024.0,
- "33": 3553.0,
- "34": 3793.0,
- "35": 3757.0,
- "36": 4205.0,
- "37": 4221.0,
- "38": 3819.0,
- "39": 3866.0,
- "40": 3554.0,
- "41": 2883.0,
- "42": 2592.0,
- "43": 2856.0,
- "44": 3173.0,
- "45": 4948.0,
- "46": 4572.0,
- "47": 4077.0,
- "48": 4355.0,
- "49": 3885.0,
- "50": 3266.0
+ "1": 3597.0,
+ "2": 3019.0,
+ "3": 2834.0,
+ "4": 3501.0,
+ "5": 3175.0,
+ "6": 2954.0,
+ "7": 3807.0,
+ "8": 3593.0,
+ "9": 3344.0,
+ "10": 3244.0,
+ "11": 3905.0,
+ "12": 1810.0,
+ "13": 3393.0,
+ "14": 4129.0,
+ "15": 3308.0,
+ "16": 3989.0,
+ "17": 3605.0,
+ "18": 3496.0,
+ "19": 3861.0,
+ "20": 3809.0,
+ "21": 2265.0,
+ "22": 2920.0,
+ "23": 3855.0,
+ "24": 3841.0,
+ "25": 3679.0,
+ "26": 3938.0,
+ "27": 2749.0,
+ "28": 4080.0,
+ "29": 3641.0,
+ "30": 3920.0,
+ "31": 3465.0,
+ "32": 3678.0,
+ "33": 3345.0,
+ "34": 3549.0,
+ "35": 3593.0,
+ "36": 4149.0,
+ "37": 4239.0,
+ "38": 4040.0,
+ "39": 4130.0,
+ "40": 4490.0,
+ "41": 2672.0,
+ "42": 2521.0,
+ "43": 2574.0,
+ "44": 3379.0,
+ "45": 3999.0,
+ "46": 3953.0,
+ "47": 2708.0,
+ "48": 4246.0,
+ "49": 3920.0,
+ "50": 3314.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1662815232.0,
- "2": 1662815232.0,
- "3": 1662815232.0,
- "4": 1662815232.0,
- "5": 1662815232.0,
- "6": 1662815232.0,
- "7": 1662815232.0,
- "8": 1662815232.0,
- "9": 1662815232.0,
- "10": 1662815232.0,
- "11": 1662815232.0,
- "12": 1662815232.0,
- "13": 1662815232.0,
- "14": 1662815232.0,
- "15": 1662815232.0,
- "16": 1662815232.0,
- "17": 1662815232.0,
- "18": 1662815232.0,
- "19": 1662815232.0,
- "20": 1662815232.0,
- "21": 1662815232.0,
- "22": 1662815232.0,
- "23": 1662815232.0,
- "24": 1662815232.0,
- "25": 1662815232.0,
- "26": 1662815232.0,
- "27": 1662815232.0,
- "28": 1662815232.0,
- "29": 1662815232.0,
- "30": 1662815232.0,
- "31": 1662815232.0,
- "32": 1662815232.0,
- "33": 1662815232.0,
- "34": 1662815232.0,
- "35": 1662815232.0,
- "36": 1662815232.0,
- "37": 1662815232.0,
- "38": 1662815232.0,
- "39": 1662815232.0,
- "40": 1662815232.0,
- "41": 1662815232.0,
- "42": 1662815232.0,
- "43": 1662815232.0,
- "44": 1662815232.0,
- "45": 1662815232.0,
- "46": 1662815232.0,
- "47": 1662815232.0,
- "48": 1662815232.0,
- "49": 1662815232.0,
- "50": 1662815232.0
+ "1": 1670994432.0,
+ "2": 1670994432.0,
+ "3": 1670994432.0,
+ "4": 1670994432.0,
+ "5": 1670994432.0,
+ "6": 1670994432.0,
+ "7": 1670994432.0,
+ "8": 1670994432.0,
+ "9": 1670994432.0,
+ "10": 1670994432.0,
+ "11": 1670994432.0,
+ "12": 1670994432.0,
+ "13": 1670994432.0,
+ "14": 1670994432.0,
+ "15": 1670994432.0,
+ "16": 1670994432.0,
+ "17": 1670994432.0,
+ "18": 1670994432.0,
+ "19": 1670994432.0,
+ "20": 1670994432.0,
+ "21": 1670994432.0,
+ "22": 1670994432.0,
+ "23": 1670994432.0,
+ "24": 1670994432.0,
+ "25": 1670994432.0,
+ "26": 1670994432.0,
+ "27": 1670994432.0,
+ "28": 1670994432.0,
+ "29": 1670994432.0,
+ "30": 1670994432.0,
+ "31": 1670994432.0,
+ "32": 1670994432.0,
+ "33": 1670994432.0,
+ "34": 1670994432.0,
+ "35": 1670994432.0,
+ "36": 1670994432.0,
+ "37": 1670994432.0,
+ "38": 1670994432.0,
+ "39": 1670994432.0,
+ "40": 1670994432.0,
+ "41": 1670994432.0,
+ "42": 1670994432.0,
+ "43": 1670994432.0,
+ "44": 1670994432.0,
+ "45": 1670994432.0,
+ "46": 1670994432.0,
+ "47": 1670994432.0,
+ "48": 1670994432.0,
+ "49": 1670994432.0,
+ "50": 1670994432.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 2507528704.0,
- "2": 3206498816.0,
- "3": 3206498816.0,
- "4": 3206498816.0,
- "5": 3206498816.0,
- "6": 3206498816.0,
- "7": 3206498816.0,
- "8": 3206498816.0,
- "9": 3206498816.0,
- "10": 3206498816.0,
- "11": 3206498816.0,
- "12": 3206498816.0,
- "13": 3206498816.0,
- "14": 3206498816.0,
- "15": 3206498816.0,
- "16": 3206498816.0,
- "17": 3206498816.0,
- "18": 3206498816.0,
- "19": 3206498816.0,
- "20": 3206498816.0,
- "21": 3206498816.0,
- "22": 3206498816.0,
- "23": 3206498816.0,
- "24": 3206498816.0,
- "25": 3206498816.0,
- "26": 3206498816.0,
- "27": 3206498816.0,
- "28": 3206498816.0,
- "29": 3206498816.0,
- "30": 3206498816.0,
- "31": 3206498816.0,
- "32": 3206498816.0,
- "33": 3206498816.0,
- "34": 3206498816.0,
- "35": 3206498816.0,
- "36": 3206498816.0,
- "37": 3206498816.0,
- "38": 3206498816.0,
- "39": 3206498816.0,
- "40": 3206498816.0,
- "41": 3206498816.0,
- "42": 3206498816.0,
- "43": 3206498816.0,
- "44": 3206498816.0,
- "45": 3206498816.0,
- "46": 3206498816.0,
- "47": 3206498816.0,
- "48": 3206498816.0,
- "49": 3206498816.0,
- "50": 3206498816.0
+ "1": 2521210368.0,
+ "2": 3221232128.0,
+ "3": 3221232128.0,
+ "4": 3221232128.0,
+ "5": 3221232128.0,
+ "6": 3221232128.0,
+ "7": 3221232128.0,
+ "8": 3221232128.0,
+ "9": 3221232128.0,
+ "10": 3221232128.0,
+ "11": 3221232128.0,
+ "12": 3221232128.0,
+ "13": 3221232128.0,
+ "14": 3221232128.0,
+ "15": 3221232128.0,
+ "16": 3221232128.0,
+ "17": 3221232128.0,
+ "18": 3221232128.0,
+ "19": 3221232128.0,
+ "20": 3221232128.0,
+ "21": 3221232128.0,
+ "22": 3221232128.0,
+ "23": 3221232128.0,
+ "24": 3221232128.0,
+ "25": 3221232128.0,
+ "26": 3221232128.0,
+ "27": 3221232128.0,
+ "28": 3221232128.0,
+ "29": 3221232128.0,
+ "30": 3221232128.0,
+ "31": 3221232128.0,
+ "32": 3221232128.0,
+ "33": 3221232128.0,
+ "34": 3221232128.0,
+ "35": 3221232128.0,
+ "36": 3221232128.0,
+ "37": 3221232128.0,
+ "38": 3221232128.0,
+ "39": 3221232128.0,
+ "40": 3221232128.0,
+ "41": 3221232128.0,
+ "42": 3221232128.0,
+ "43": 3221232128.0,
+ "44": 3221232128.0,
+ "45": 3221232128.0,
+ "46": 3221232128.0,
+ "47": 3221232128.0,
+ "48": 3221232128.0,
+ "49": 3221232128.0,
+ "50": 3221232128.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.8403,
- "2": 1.75656,
- "3": 1.70317,
- "4": 1.66346,
- "5": 1.6703,
- "6": 1.66753,
- "7": 2.21547,
- "8": 1.68918,
- "9": 1.77005,
- "10": 1.75261,
- "11": 1.77153,
- "12": 1.65933,
- "13": 1.65337,
- "14": 2.37845,
- "15": 2.04839,
- "16": 2.07092,
- "17": 1.67053,
- "18": 1.6729,
- "19": 1.65463,
- "20": 1.67298,
- "21": 1.66273,
- "22": 1.64743,
- "23": 1.64351,
- "24": 1.63695,
- "25": 1.66076,
- "26": 1.66885,
- "27": 1.64423,
- "28": 1.64773,
- "29": 1.64565,
- "30": 1.64171,
- "31": 1.63705,
- "32": 1.64216,
- "33": 1.64504,
- "34": 1.64255,
- "35": 1.64762,
- "36": 1.64913,
- "37": 1.63831,
- "38": 1.65213,
- "39": 1.66065,
- "40": 1.63954,
- "41": 1.63964,
- "42": 1.64408,
- "43": 1.64113,
- "44": 1.65016,
- "45": 1.63618,
- "46": 1.65229,
- "47": 1.64761,
- "48": 1.76963,
- "49": 1.62535,
- "50": 1.63142
+ "1": "nan",
+ "2": 6.15099,
+ "3": 1.65293,
+ "4": 1.63949,
+ "5": 1.64244,
+ "6": 1.6438,
+ "7": 1.67622,
+ "8": 1.6456,
+ "9": 1.654,
+ "10": 1.6418,
+ "11": 1.6412,
+ "12": 1.63799,
+ "13": 1.63405,
+ "14": 1.63852,
+ "15": 1.64249,
+ "16": 1.63757,
+ "17": 1.63078,
+ "18": 1.62993,
+ "19": 1.72481,
+ "20": 1.63412,
+ "21": 1.74828,
+ "22": 1.7055,
+ "23": 2.19945,
+ "24": 1.63616,
+ "25": 1.63702,
+ "26": 1.64273,
+ "27": 1.64261,
+ "28": 1.6368,
+ "29": 2.16015,
+ "30": 2.43465,
+ "31": 2.08006,
+ "32": 1.64052,
+ "33": 2.41778,
+ "34": 2.04121,
+ "35": 2.02716,
+ "36": 1.64041,
+ "37": 1.64053,
+ "38": 1.64108,
+ "39": 1.64424,
+ "40": 1.63863,
+ "41": 1.63966,
+ "42": 1.63523,
+ "43": 1.64084,
+ "44": 1.64004,
+ "45": 1.64161,
+ "46": 1.65273,
+ "47": 1.6401,
+ "48": 1.64477,
+ "49": 1.65398,
+ "50": 1.64766
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/bert/bert_mcore_tp4_pp1/model_config.yaml b/tests/functional_tests/test_cases/bert/bert_mcore_tp4_pp1/model_config.yaml
index e001ea4dc08..15ec6afdebe 100644
--- a/tests/functional_tests/test_cases/bert/bert_mcore_tp4_pp1/model_config.yaml
+++ b/tests/functional_tests/test_cases/bert/bert_mcore_tp4_pp1/model_config.yaml
@@ -22,8 +22,8 @@ MODEL_ARGS:
--lr-decay-iters: 990000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/bert_shard00/my-bert_00_text_sentence
- --vocab-file: ${DATA_PATH}/text/the_pile/bert_shard00/vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-bert_00_text_sentence
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/vocab.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.0001
diff --git a/tests/functional_tests/test_cases/bert/bert_release/model_config.yaml b/tests/functional_tests/test_cases/bert/bert_release/model_config.yaml
index af341b0f670..860215bd473 100644
--- a/tests/functional_tests/test_cases/bert/bert_release/model_config.yaml
+++ b/tests/functional_tests/test_cases/bert/bert_release/model_config.yaml
@@ -27,7 +27,7 @@ MODEL_ARGS:
--pipeline-model-parallel-size: 8
# Data args
--data-path: ${DATA_BLEND}
- --vocab-file: ${DATA_PATH}/text/the_pile/bert_shard00/vocab.txt
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/vocab.txt
--split: 949,50,1
--data-cache-path: ${DATA_CACHE_PATH}
# EVAL_AND_LOGGING_ARGS
diff --git a/tests/functional_tests/test_cases/bert/bert_release_sm/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/bert/bert_release_sm/golden_values_dev_dgx_gb200.json
new file mode 100644
index 00000000000..812804fe033
--- /dev/null
+++ b/tests/functional_tests/test_cases/bert/bert_release_sm/golden_values_dev_dgx_gb200.json
@@ -0,0 +1,2042 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 5,
+ "values": {
+ "1": 10.57016,
+ "5": 10.57457,
+ "10": 10.5895,
+ "15": 10.5701,
+ "20": 10.13778,
+ "25": 9.49357,
+ "30": 9.4056,
+ "35": 9.1618,
+ "40": 9.08637,
+ "45": 8.96435,
+ "50": 8.67005,
+ "55": 8.77614,
+ "60": 8.51912,
+ "65": 8.52143,
+ "70": 8.20591,
+ "75": 8.31295,
+ "80": 8.00603,
+ "85": 7.79543,
+ "90": 7.70836,
+ "95": 7.57266,
+ "100": 7.53591,
+ "105": 7.35517,
+ "110": 7.44978,
+ "115": 7.20792,
+ "120": 6.95942,
+ "125": 7.27887,
+ "130": 7.01311,
+ "135": 6.80831,
+ "140": 7.21623,
+ "145": 7.00076,
+ "150": 7.08728,
+ "155": 6.98439,
+ "160": 6.99868,
+ "165": 7.03626,
+ "170": 6.92491,
+ "175": 6.87684,
+ "180": 6.7349,
+ "185": 6.92182,
+ "190": 7.09627,
+ "195": 6.95652,
+ "200": 6.88737,
+ "205": 6.88212,
+ "210": 6.75061,
+ "215": 6.9828,
+ "220": 6.66761,
+ "225": 6.77153,
+ "230": 6.73327,
+ "235": 6.84111,
+ "240": 6.58632,
+ "245": 7.08674,
+ "250": 6.60849,
+ "255": 6.65796,
+ "260": 6.74518,
+ "265": 6.62708,
+ "270": 6.59842,
+ "275": 6.82877,
+ "280": 6.53927,
+ "285": 6.5668,
+ "290": 6.74581,
+ "295": 6.73208,
+ "300": 6.43994,
+ "305": 6.34549,
+ "310": 6.95999,
+ "315": 6.93016,
+ "320": 6.86096,
+ "325": 6.66038,
+ "330": 6.63878,
+ "335": 6.64048,
+ "340": 6.58574,
+ "345": 6.53944,
+ "350": 6.396,
+ "355": 6.51173,
+ "360": 6.56525,
+ "365": 6.61797,
+ "370": 6.52094,
+ "375": 6.64693,
+ "380": 6.45276,
+ "385": 6.40779,
+ "390": 6.46608,
+ "395": 6.25858,
+ "400": 6.64783,
+ "405": 6.47696,
+ "410": 6.36974,
+ "415": 6.56753,
+ "420": 6.58633,
+ "425": 6.53258,
+ "430": 6.56715,
+ "435": 6.51055,
+ "440": 6.54104,
+ "445": 6.57032,
+ "450": 6.60459,
+ "455": 6.64806,
+ "460": 6.74269,
+ "465": 6.71717,
+ "470": 6.49106,
+ "475": 6.21387,
+ "480": 6.46245,
+ "485": 6.5334,
+ "490": 6.18455,
+ "495": 6.21143,
+ "500": 6.61963,
+ "505": 6.60945,
+ "510": 6.51889,
+ "515": 6.20099,
+ "520": 6.42318,
+ "525": 6.43891,
+ "530": 6.41375,
+ "535": 6.0017,
+ "540": 6.46755,
+ "545": 6.34931,
+ "550": 6.42553,
+ "555": 6.51621,
+ "560": 6.39803,
+ "565": 6.19623,
+ "570": 6.28741,
+ "575": 6.63905,
+ "580": 6.42899,
+ "585": 6.42321,
+ "590": 6.18413,
+ "595": 6.26483,
+ "600": 6.48357,
+ "605": 6.49681,
+ "610": 6.29344,
+ "615": 6.42897,
+ "620": 6.50728,
+ "625": 6.42103,
+ "630": 6.45875,
+ "635": 6.48144,
+ "640": 6.34499,
+ "645": 6.27084,
+ "650": 6.24999,
+ "655": 6.55351,
+ "660": 6.34522,
+ "665": 6.27813,
+ "670": 6.35548,
+ "675": 6.2648,
+ "680": 6.36674,
+ "685": 6.53052,
+ "690": 6.54112,
+ "695": 6.21604,
+ "700": 5.99887,
+ "705": 6.28967,
+ "710": 6.17574,
+ "715": 6.38195,
+ "720": 6.25111,
+ "725": 6.43415,
+ "730": 6.53264,
+ "735": 6.61096,
+ "740": 6.5891,
+ "745": 6.29544,
+ "750": 6.26264,
+ "755": 6.00057,
+ "760": 6.57044,
+ "765": 6.37664,
+ "770": 6.44316,
+ "775": 6.14596,
+ "780": 6.24057,
+ "785": 6.29758,
+ "790": 6.48672,
+ "795": 6.375,
+ "800": 6.34408,
+ "805": 6.4855,
+ "810": 6.17661,
+ "815": 6.24969,
+ "820": 6.46937,
+ "825": 6.49875,
+ "830": 6.41187,
+ "835": 6.38815,
+ "840": 6.33676,
+ "845": 6.19728,
+ "850": 6.37738,
+ "855": 6.40839,
+ "860": 6.27525,
+ "865": 6.28608,
+ "870": 6.36816,
+ "875": 6.40001,
+ "880": 6.29208,
+ "885": 6.49235,
+ "890": 6.2164,
+ "895": 6.25901,
+ "900": 6.25059,
+ "905": 6.36206,
+ "910": 6.07771,
+ "915": 6.50532,
+ "920": 6.34417,
+ "925": 6.2606,
+ "930": 6.21949,
+ "935": 6.22412,
+ "940": 6.36739,
+ "945": 6.03716,
+ "950": 6.16067,
+ "955": 6.16405,
+ "960": 6.2354,
+ "965": 6.26383,
+ "970": 6.28899,
+ "975": 6.18494,
+ "980": 6.07856,
+ "985": 6.26264,
+ "990": 6.16384,
+ "995": 6.40483,
+ "1000": 6.21386,
+ "1005": 6.42444,
+ "1010": 6.10073,
+ "1015": 6.18252,
+ "1020": 6.09469,
+ "1025": 6.37207,
+ "1030": 6.16964,
+ "1035": 5.92432,
+ "1040": 6.02639,
+ "1045": 6.33766,
+ "1050": 6.45656,
+ "1055": 6.09236,
+ "1060": 6.09897,
+ "1065": 5.97221,
+ "1070": 6.1916,
+ "1075": 5.97914,
+ "1080": 6.14591,
+ "1085": 6.01944,
+ "1090": 6.1107,
+ "1095": 6.41051,
+ "1100": 6.07832,
+ "1105": 6.28098,
+ "1110": 6.17453,
+ "1115": 6.06118,
+ "1120": 6.04548,
+ "1125": 6.14937,
+ "1130": 5.92201,
+ "1135": 6.33183,
+ "1140": 6.4784,
+ "1145": 6.2523,
+ "1150": 6.1203,
+ "1155": 5.99932,
+ "1160": 6.08841,
+ "1165": 6.13199,
+ "1170": 6.32067,
+ "1175": 6.09731,
+ "1180": 6.2554,
+ "1185": 6.28789,
+ "1190": 6.33396,
+ "1195": 6.22166,
+ "1200": 6.18768,
+ "1205": 6.32714,
+ "1210": 6.14663,
+ "1215": 6.06385,
+ "1220": 6.20885,
+ "1225": 6.33596,
+ "1230": 6.29505,
+ "1235": 6.07554,
+ "1240": 6.4444,
+ "1245": 6.12619,
+ "1250": 5.92236,
+ "1255": 6.24939,
+ "1260": 6.21401,
+ "1265": 5.99277,
+ "1270": 6.05496,
+ "1275": 6.11892,
+ "1280": 5.80143,
+ "1285": 5.97887,
+ "1290": 6.03117,
+ "1295": 6.1527,
+ "1300": 6.3507,
+ "1305": 5.92647,
+ "1310": 6.00999,
+ "1315": 6.0662,
+ "1320": 6.10065,
+ "1325": 6.1658,
+ "1330": 6.11215,
+ "1335": 6.06255,
+ "1340": 6.1044,
+ "1345": 6.19047,
+ "1350": 5.94001,
+ "1355": 6.08141,
+ "1360": 6.43968,
+ "1365": 5.9775,
+ "1370": 6.06872,
+ "1375": 6.26164,
+ "1380": 6.0897,
+ "1385": 5.99699,
+ "1390": 5.86771,
+ "1395": 5.84369,
+ "1400": 6.14356,
+ "1405": 6.1545,
+ "1410": 6.05662,
+ "1415": 6.24539,
+ "1420": 6.19344,
+ "1425": 6.19564,
+ "1430": 6.01059,
+ "1435": 6.4846,
+ "1440": 6.33193,
+ "1445": 6.31451,
+ "1450": 6.0827,
+ "1455": 5.78442,
+ "1460": 6.19138,
+ "1465": 6.18086,
+ "1470": 6.25935,
+ "1475": 6.23589,
+ "1480": 6.33248,
+ "1485": 6.05732,
+ "1490": 6.23182,
+ "1495": 6.26111,
+ "1500": 6.18782,
+ "1505": 6.2482,
+ "1510": 6.39892,
+ "1515": 6.05634,
+ "1520": 5.82788,
+ "1525": 5.91483,
+ "1530": 5.88868,
+ "1535": 6.46938,
+ "1540": 6.40193,
+ "1545": 6.33528,
+ "1550": 6.12725,
+ "1555": 6.06753,
+ "1560": 6.1525,
+ "1565": 6.06122,
+ "1570": 6.01284,
+ "1575": 6.12295,
+ "1580": 6.24347,
+ "1585": 6.0764,
+ "1590": 6.24692,
+ "1595": 6.19689,
+ "1600": 6.10773,
+ "1605": 6.05819,
+ "1610": 6.13265,
+ "1615": 6.14311,
+ "1620": 6.04295,
+ "1625": 6.03832,
+ "1630": 6.09542,
+ "1635": 6.31795,
+ "1640": 6.14846,
+ "1645": 6.19578,
+ "1650": 6.29627,
+ "1655": 6.35313,
+ "1660": 6.22447,
+ "1665": 6.05814,
+ "1670": 6.181,
+ "1675": 5.87418,
+ "1680": 6.10235,
+ "1685": 6.37124,
+ "1690": 6.17879,
+ "1695": 6.0453,
+ "1700": 6.23077,
+ "1705": 6.07609,
+ "1710": 6.03402,
+ "1715": 6.05391,
+ "1720": 6.18997,
+ "1725": 6.39643,
+ "1730": 6.19031,
+ "1735": 6.02531,
+ "1740": 6.10795,
+ "1745": 6.09931,
+ "1750": 6.32244,
+ "1755": 6.27191,
+ "1760": 5.98249,
+ "1765": 5.99969,
+ "1770": 6.1287,
+ "1775": 5.99984,
+ "1780": 6.02211,
+ "1785": 5.8221,
+ "1790": 5.96942,
+ "1795": 6.30099,
+ "1800": 6.17733,
+ "1805": 5.81159,
+ "1810": 6.48243,
+ "1815": 6.16965,
+ "1820": 6.48046,
+ "1825": 6.13125,
+ "1830": 6.05526,
+ "1835": 5.95849,
+ "1840": 5.90028,
+ "1845": 6.29489,
+ "1850": 6.23845,
+ "1855": 6.13384,
+ "1860": 6.0453,
+ "1865": 6.00502,
+ "1870": 6.32861,
+ "1875": 6.09238,
+ "1880": 5.91293,
+ "1885": 6.30309,
+ "1890": 6.11072,
+ "1895": 5.96072,
+ "1900": 6.09184,
+ "1905": 6.28475,
+ "1910": 6.29417,
+ "1915": 6.33553,
+ "1920": 6.16605,
+ "1925": 5.96987,
+ "1930": 6.10131,
+ "1935": 6.00775,
+ "1940": 6.13005,
+ "1945": 6.08146,
+ "1950": 6.03616,
+ "1955": 6.03111,
+ "1960": 6.01663,
+ "1965": 5.98703,
+ "1970": 6.07541,
+ "1975": 6.25125,
+ "1980": 6.39044,
+ "1985": 6.12075,
+ "1990": 5.97305,
+ "1995": 5.92903,
+ "2000": 6.09087
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 5,
+ "values": {
+ "1": 192387584.0,
+ "5": 192387584.0,
+ "10": 191994368.0,
+ "15": 192387584.0,
+ "20": 270223360.0,
+ "25": 270223360.0,
+ "30": 270223360.0,
+ "35": 270223360.0,
+ "40": 270223360.0,
+ "45": 270223360.0,
+ "50": 270223360.0,
+ "55": 270223360.0,
+ "60": 270223360.0,
+ "65": 270223360.0,
+ "70": 270223360.0,
+ "75": 270223360.0,
+ "80": 270223360.0,
+ "85": 270223360.0,
+ "90": 270223360.0,
+ "95": 270223360.0,
+ "100": 270223360.0,
+ "105": 270223360.0,
+ "110": 270223360.0,
+ "115": 270223360.0,
+ "120": 270223360.0,
+ "125": 270223360.0,
+ "130": 270223360.0,
+ "135": 270223360.0,
+ "140": 270223360.0,
+ "145": 270223360.0,
+ "150": 270223360.0,
+ "155": 270223360.0,
+ "160": 270223360.0,
+ "165": 270223360.0,
+ "170": 270223360.0,
+ "175": 270223360.0,
+ "180": 270223360.0,
+ "185": 270223360.0,
+ "190": 270223360.0,
+ "195": 270223360.0,
+ "200": 270223360.0,
+ "205": 270223360.0,
+ "210": 270223360.0,
+ "215": 270223360.0,
+ "220": 270223360.0,
+ "225": 270223360.0,
+ "230": 270223360.0,
+ "235": 270223360.0,
+ "240": 270223360.0,
+ "245": 270223360.0,
+ "250": 270223360.0,
+ "255": 270223360.0,
+ "260": 270223360.0,
+ "265": 270223360.0,
+ "270": 270223360.0,
+ "275": 270223360.0,
+ "280": 270223360.0,
+ "285": 270223360.0,
+ "290": 270223360.0,
+ "295": 270223360.0,
+ "300": 270223360.0,
+ "305": 270223360.0,
+ "310": 270223360.0,
+ "315": 270223360.0,
+ "320": 270223360.0,
+ "325": 270223360.0,
+ "330": 270223360.0,
+ "335": 270223360.0,
+ "340": 270223360.0,
+ "345": 270223360.0,
+ "350": 270223360.0,
+ "355": 270223360.0,
+ "360": 270223360.0,
+ "365": 270223360.0,
+ "370": 270223360.0,
+ "375": 270223360.0,
+ "380": 270223360.0,
+ "385": 270223360.0,
+ "390": 270223360.0,
+ "395": 270223360.0,
+ "400": 270223360.0,
+ "405": 270223360.0,
+ "410": 270223360.0,
+ "415": 270223360.0,
+ "420": 270223360.0,
+ "425": 270223360.0,
+ "430": 270223360.0,
+ "435": 270223360.0,
+ "440": 270223360.0,
+ "445": 270223360.0,
+ "450": 270223360.0,
+ "455": 270223360.0,
+ "460": 270223360.0,
+ "465": 270223360.0,
+ "470": 270223360.0,
+ "475": 270223360.0,
+ "480": 270223360.0,
+ "485": 270223360.0,
+ "490": 270223360.0,
+ "495": 270223360.0,
+ "500": 270223360.0,
+ "505": 270223360.0,
+ "510": 270223360.0,
+ "515": 270223360.0,
+ "520": 270223360.0,
+ "525": 270223360.0,
+ "530": 270223360.0,
+ "535": 270223360.0,
+ "540": 270223360.0,
+ "545": 270223360.0,
+ "550": 270223360.0,
+ "555": 270223360.0,
+ "560": 270223360.0,
+ "565": 270223360.0,
+ "570": 270223360.0,
+ "575": 270223360.0,
+ "580": 270223360.0,
+ "585": 270223360.0,
+ "590": 270223360.0,
+ "595": 270223360.0,
+ "600": 270223360.0,
+ "605": 270223360.0,
+ "610": 270223360.0,
+ "615": 270223360.0,
+ "620": 270223360.0,
+ "625": 270223360.0,
+ "630": 270223360.0,
+ "635": 270223360.0,
+ "640": 270223360.0,
+ "645": 270223360.0,
+ "650": 270223360.0,
+ "655": 270223360.0,
+ "660": 270223360.0,
+ "665": 270223360.0,
+ "670": 270223360.0,
+ "675": 270223360.0,
+ "680": 270223360.0,
+ "685": 270223360.0,
+ "690": 270223360.0,
+ "695": 270223360.0,
+ "700": 270223360.0,
+ "705": 270223360.0,
+ "710": 270223360.0,
+ "715": 270223360.0,
+ "720": 270223360.0,
+ "725": 270223360.0,
+ "730": 270223360.0,
+ "735": 270223360.0,
+ "740": 270223360.0,
+ "745": 270223360.0,
+ "750": 270223360.0,
+ "755": 270223360.0,
+ "760": 270223360.0,
+ "765": 270223360.0,
+ "770": 270223360.0,
+ "775": 270223360.0,
+ "780": 270223360.0,
+ "785": 270223360.0,
+ "790": 270223360.0,
+ "795": 270223360.0,
+ "800": 270223360.0,
+ "805": 270223360.0,
+ "810": 270223360.0,
+ "815": 270223360.0,
+ "820": 270223360.0,
+ "825": 270223360.0,
+ "830": 270223360.0,
+ "835": 270223360.0,
+ "840": 270223360.0,
+ "845": 270223360.0,
+ "850": 270223360.0,
+ "855": 270223360.0,
+ "860": 270223360.0,
+ "865": 270223360.0,
+ "870": 270223360.0,
+ "875": 270223360.0,
+ "880": 270223360.0,
+ "885": 270223360.0,
+ "890": 270223360.0,
+ "895": 270223360.0,
+ "900": 270223360.0,
+ "905": 270223360.0,
+ "910": 270223360.0,
+ "915": 270223360.0,
+ "920": 270223360.0,
+ "925": 270223360.0,
+ "930": 270223360.0,
+ "935": 270223360.0,
+ "940": 270223360.0,
+ "945": 270223360.0,
+ "950": 270223360.0,
+ "955": 270223360.0,
+ "960": 270223360.0,
+ "965": 270223360.0,
+ "970": 270223360.0,
+ "975": 270223360.0,
+ "980": 270223360.0,
+ "985": 270223360.0,
+ "990": 270223360.0,
+ "995": 270223360.0,
+ "1000": 270223360.0,
+ "1005": 270223360.0,
+ "1010": 270223360.0,
+ "1015": 270223360.0,
+ "1020": 270223360.0,
+ "1025": 270223360.0,
+ "1030": 270223360.0,
+ "1035": 270223360.0,
+ "1040": 270223360.0,
+ "1045": 270223360.0,
+ "1050": 270223360.0,
+ "1055": 270223360.0,
+ "1060": 270223360.0,
+ "1065": 270223360.0,
+ "1070": 270223360.0,
+ "1075": 270223360.0,
+ "1080": 270223360.0,
+ "1085": 270223360.0,
+ "1090": 270223360.0,
+ "1095": 270223360.0,
+ "1100": 270223360.0,
+ "1105": 270223360.0,
+ "1110": 270223360.0,
+ "1115": 270223360.0,
+ "1120": 270223360.0,
+ "1125": 270223360.0,
+ "1130": 270223360.0,
+ "1135": 270223360.0,
+ "1140": 270223360.0,
+ "1145": 270223360.0,
+ "1150": 270223360.0,
+ "1155": 270223360.0,
+ "1160": 270223360.0,
+ "1165": 270223360.0,
+ "1170": 270223360.0,
+ "1175": 270223360.0,
+ "1180": 270223360.0,
+ "1185": 270223360.0,
+ "1190": 270223360.0,
+ "1195": 270223360.0,
+ "1200": 270223360.0,
+ "1205": 270223360.0,
+ "1210": 270223360.0,
+ "1215": 270223360.0,
+ "1220": 270223360.0,
+ "1225": 270223360.0,
+ "1230": 270223360.0,
+ "1235": 270223360.0,
+ "1240": 270223360.0,
+ "1245": 270223360.0,
+ "1250": 270223360.0,
+ "1255": 270223360.0,
+ "1260": 270223360.0,
+ "1265": 270223360.0,
+ "1270": 270223360.0,
+ "1275": 270223360.0,
+ "1280": 270223360.0,
+ "1285": 270223360.0,
+ "1290": 270223360.0,
+ "1295": 270223360.0,
+ "1300": 270223360.0,
+ "1305": 270223360.0,
+ "1310": 270223360.0,
+ "1315": 270223360.0,
+ "1320": 270223360.0,
+ "1325": 270223360.0,
+ "1330": 270223360.0,
+ "1335": 270223360.0,
+ "1340": 270223360.0,
+ "1345": 270223360.0,
+ "1350": 270223360.0,
+ "1355": 270223360.0,
+ "1360": 270223360.0,
+ "1365": 270223360.0,
+ "1370": 270223360.0,
+ "1375": 270223360.0,
+ "1380": 270223360.0,
+ "1385": 270223360.0,
+ "1390": 270223360.0,
+ "1395": 270223360.0,
+ "1400": 270223360.0,
+ "1405": 270223360.0,
+ "1410": 270223360.0,
+ "1415": 270223360.0,
+ "1420": 270223360.0,
+ "1425": 270223360.0,
+ "1430": 270223360.0,
+ "1435": 270223360.0,
+ "1440": 270223360.0,
+ "1445": 270223360.0,
+ "1450": 270223360.0,
+ "1455": 270223360.0,
+ "1460": 270223360.0,
+ "1465": 270223360.0,
+ "1470": 270223360.0,
+ "1475": 270223360.0,
+ "1480": 270223360.0,
+ "1485": 270223360.0,
+ "1490": 270223360.0,
+ "1495": 270223360.0,
+ "1500": 270223360.0,
+ "1505": 270223360.0,
+ "1510": 270223360.0,
+ "1515": 270223360.0,
+ "1520": 270223360.0,
+ "1525": 270223360.0,
+ "1530": 270223360.0,
+ "1535": 270223360.0,
+ "1540": 270223360.0,
+ "1545": 270223360.0,
+ "1550": 270223360.0,
+ "1555": 270223360.0,
+ "1560": 270223360.0,
+ "1565": 270223360.0,
+ "1570": 270223360.0,
+ "1575": 270223360.0,
+ "1580": 270223360.0,
+ "1585": 270223360.0,
+ "1590": 270223360.0,
+ "1595": 270223360.0,
+ "1600": 270223360.0,
+ "1605": 270223360.0,
+ "1610": 270223360.0,
+ "1615": 270223360.0,
+ "1620": 270223360.0,
+ "1625": 270223360.0,
+ "1630": 270223360.0,
+ "1635": 270223360.0,
+ "1640": 270223360.0,
+ "1645": 270223360.0,
+ "1650": 270223360.0,
+ "1655": 270223360.0,
+ "1660": 270223360.0,
+ "1665": 270223360.0,
+ "1670": 270223360.0,
+ "1675": 270223360.0,
+ "1680": 270223360.0,
+ "1685": 270223360.0,
+ "1690": 270223360.0,
+ "1695": 270223360.0,
+ "1700": 270223360.0,
+ "1705": 270223360.0,
+ "1710": 270223360.0,
+ "1715": 270223360.0,
+ "1720": 270223360.0,
+ "1725": 270223360.0,
+ "1730": 270223360.0,
+ "1735": 270223360.0,
+ "1740": 270223360.0,
+ "1745": 270223360.0,
+ "1750": 270223360.0,
+ "1755": 270223360.0,
+ "1760": 270223360.0,
+ "1765": 270223360.0,
+ "1770": 270223360.0,
+ "1775": 270223360.0,
+ "1780": 270223360.0,
+ "1785": 270223360.0,
+ "1790": 270223360.0,
+ "1795": 270223360.0,
+ "1800": 270223360.0,
+ "1805": 270223360.0,
+ "1810": 270223360.0,
+ "1815": 270223360.0,
+ "1820": 270223360.0,
+ "1825": 270223360.0,
+ "1830": 270223360.0,
+ "1835": 270223360.0,
+ "1840": 270223360.0,
+ "1845": 270223360.0,
+ "1850": 270223360.0,
+ "1855": 270223360.0,
+ "1860": 270223360.0,
+ "1865": 270223360.0,
+ "1870": 270223360.0,
+ "1875": 270223360.0,
+ "1880": 270223360.0,
+ "1885": 270223360.0,
+ "1890": 270223360.0,
+ "1895": 270223360.0,
+ "1900": 270223360.0,
+ "1905": 270223360.0,
+ "1910": 270223360.0,
+ "1915": 270223360.0,
+ "1920": 270223360.0,
+ "1925": 270223360.0,
+ "1930": 270223360.0,
+ "1935": 270223360.0,
+ "1940": 270223360.0,
+ "1945": 270223360.0,
+ "1950": 270223360.0,
+ "1955": 270223360.0,
+ "1960": 270223360.0,
+ "1965": 270223360.0,
+ "1970": 270223360.0,
+ "1975": 270223360.0,
+ "1980": 270223360.0,
+ "1985": 270223360.0,
+ "1990": 270223360.0,
+ "1995": 270223360.0,
+ "2000": 270223360.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 5,
+ "values": {
+ "1": 402606592.0,
+ "5": 402606592.0,
+ "10": 402606592.0,
+ "15": 402606592.0,
+ "20": 450471936.0,
+ "25": 450471936.0,
+ "30": 450471936.0,
+ "35": 450471936.0,
+ "40": 450471936.0,
+ "45": 450471936.0,
+ "50": 450471936.0,
+ "55": 450471936.0,
+ "60": 450471936.0,
+ "65": 450471936.0,
+ "70": 450471936.0,
+ "75": 450471936.0,
+ "80": 450471936.0,
+ "85": 450471936.0,
+ "90": 450471936.0,
+ "95": 450471936.0,
+ "100": 450471936.0,
+ "105": 450471936.0,
+ "110": 450471936.0,
+ "115": 450471936.0,
+ "120": 450471936.0,
+ "125": 450471936.0,
+ "130": 450471936.0,
+ "135": 450471936.0,
+ "140": 450471936.0,
+ "145": 450471936.0,
+ "150": 450471936.0,
+ "155": 450471936.0,
+ "160": 450471936.0,
+ "165": 450471936.0,
+ "170": 450471936.0,
+ "175": 450471936.0,
+ "180": 450471936.0,
+ "185": 450471936.0,
+ "190": 450471936.0,
+ "195": 450471936.0,
+ "200": 450471936.0,
+ "205": 450471936.0,
+ "210": 450471936.0,
+ "215": 450471936.0,
+ "220": 450471936.0,
+ "225": 450471936.0,
+ "230": 450471936.0,
+ "235": 450471936.0,
+ "240": 450471936.0,
+ "245": 450471936.0,
+ "250": 450471936.0,
+ "255": 450471936.0,
+ "260": 450471936.0,
+ "265": 450471936.0,
+ "270": 450471936.0,
+ "275": 450471936.0,
+ "280": 450471936.0,
+ "285": 450471936.0,
+ "290": 450471936.0,
+ "295": 450471936.0,
+ "300": 450471936.0,
+ "305": 450471936.0,
+ "310": 450471936.0,
+ "315": 450471936.0,
+ "320": 450471936.0,
+ "325": 450471936.0,
+ "330": 450471936.0,
+ "335": 450471936.0,
+ "340": 450471936.0,
+ "345": 450471936.0,
+ "350": 450471936.0,
+ "355": 450471936.0,
+ "360": 450471936.0,
+ "365": 450471936.0,
+ "370": 450471936.0,
+ "375": 450471936.0,
+ "380": 450471936.0,
+ "385": 450471936.0,
+ "390": 450471936.0,
+ "395": 450471936.0,
+ "400": 450471936.0,
+ "405": 450471936.0,
+ "410": 450471936.0,
+ "415": 450471936.0,
+ "420": 450471936.0,
+ "425": 450471936.0,
+ "430": 450471936.0,
+ "435": 450471936.0,
+ "440": 450471936.0,
+ "445": 450471936.0,
+ "450": 450471936.0,
+ "455": 450471936.0,
+ "460": 450471936.0,
+ "465": 450471936.0,
+ "470": 450471936.0,
+ "475": 450471936.0,
+ "480": 450471936.0,
+ "485": 450471936.0,
+ "490": 450471936.0,
+ "495": 450471936.0,
+ "500": 450471936.0,
+ "505": 450471936.0,
+ "510": 450471936.0,
+ "515": 450471936.0,
+ "520": 450471936.0,
+ "525": 450471936.0,
+ "530": 450471936.0,
+ "535": 450471936.0,
+ "540": 450471936.0,
+ "545": 450471936.0,
+ "550": 450471936.0,
+ "555": 450471936.0,
+ "560": 450471936.0,
+ "565": 450471936.0,
+ "570": 450471936.0,
+ "575": 450471936.0,
+ "580": 450471936.0,
+ "585": 450471936.0,
+ "590": 450471936.0,
+ "595": 450471936.0,
+ "600": 450471936.0,
+ "605": 450471936.0,
+ "610": 450471936.0,
+ "615": 450471936.0,
+ "620": 450471936.0,
+ "625": 450471936.0,
+ "630": 450471936.0,
+ "635": 450471936.0,
+ "640": 450471936.0,
+ "645": 450471936.0,
+ "650": 450471936.0,
+ "655": 450471936.0,
+ "660": 450471936.0,
+ "665": 450471936.0,
+ "670": 450471936.0,
+ "675": 450471936.0,
+ "680": 450471936.0,
+ "685": 450471936.0,
+ "690": 450471936.0,
+ "695": 450471936.0,
+ "700": 450471936.0,
+ "705": 450471936.0,
+ "710": 450471936.0,
+ "715": 450471936.0,
+ "720": 450471936.0,
+ "725": 450471936.0,
+ "730": 450471936.0,
+ "735": 450471936.0,
+ "740": 450471936.0,
+ "745": 450471936.0,
+ "750": 450471936.0,
+ "755": 450471936.0,
+ "760": 450471936.0,
+ "765": 450471936.0,
+ "770": 450471936.0,
+ "775": 450471936.0,
+ "780": 450471936.0,
+ "785": 450471936.0,
+ "790": 450471936.0,
+ "795": 450471936.0,
+ "800": 450471936.0,
+ "805": 450471936.0,
+ "810": 450471936.0,
+ "815": 450471936.0,
+ "820": 450471936.0,
+ "825": 450471936.0,
+ "830": 450471936.0,
+ "835": 450471936.0,
+ "840": 450471936.0,
+ "845": 450471936.0,
+ "850": 450471936.0,
+ "855": 450471936.0,
+ "860": 450471936.0,
+ "865": 450471936.0,
+ "870": 450471936.0,
+ "875": 450471936.0,
+ "880": 450471936.0,
+ "885": 450471936.0,
+ "890": 450471936.0,
+ "895": 450471936.0,
+ "900": 450471936.0,
+ "905": 450471936.0,
+ "910": 450471936.0,
+ "915": 450471936.0,
+ "920": 450471936.0,
+ "925": 450471936.0,
+ "930": 450471936.0,
+ "935": 450471936.0,
+ "940": 450471936.0,
+ "945": 450471936.0,
+ "950": 450471936.0,
+ "955": 450471936.0,
+ "960": 450471936.0,
+ "965": 450471936.0,
+ "970": 450471936.0,
+ "975": 450471936.0,
+ "980": 450471936.0,
+ "985": 450471936.0,
+ "990": 450471936.0,
+ "995": 450471936.0,
+ "1000": 450471936.0,
+ "1005": 450471936.0,
+ "1010": 450471936.0,
+ "1015": 450471936.0,
+ "1020": 450471936.0,
+ "1025": 450471936.0,
+ "1030": 450471936.0,
+ "1035": 450471936.0,
+ "1040": 450471936.0,
+ "1045": 450471936.0,
+ "1050": 450471936.0,
+ "1055": 450471936.0,
+ "1060": 450471936.0,
+ "1065": 450471936.0,
+ "1070": 450471936.0,
+ "1075": 450471936.0,
+ "1080": 450471936.0,
+ "1085": 450471936.0,
+ "1090": 450471936.0,
+ "1095": 450471936.0,
+ "1100": 450471936.0,
+ "1105": 450471936.0,
+ "1110": 450471936.0,
+ "1115": 450471936.0,
+ "1120": 450471936.0,
+ "1125": 450471936.0,
+ "1130": 450471936.0,
+ "1135": 450471936.0,
+ "1140": 450471936.0,
+ "1145": 450471936.0,
+ "1150": 450471936.0,
+ "1155": 450471936.0,
+ "1160": 450471936.0,
+ "1165": 450471936.0,
+ "1170": 450471936.0,
+ "1175": 450471936.0,
+ "1180": 450471936.0,
+ "1185": 450471936.0,
+ "1190": 450471936.0,
+ "1195": 450471936.0,
+ "1200": 450471936.0,
+ "1205": 450471936.0,
+ "1210": 450471936.0,
+ "1215": 450471936.0,
+ "1220": 450471936.0,
+ "1225": 450471936.0,
+ "1230": 450471936.0,
+ "1235": 450471936.0,
+ "1240": 450471936.0,
+ "1245": 450471936.0,
+ "1250": 450471936.0,
+ "1255": 450471936.0,
+ "1260": 450471936.0,
+ "1265": 450471936.0,
+ "1270": 450471936.0,
+ "1275": 450471936.0,
+ "1280": 450471936.0,
+ "1285": 450471936.0,
+ "1290": 450471936.0,
+ "1295": 450471936.0,
+ "1300": 450471936.0,
+ "1305": 450471936.0,
+ "1310": 450471936.0,
+ "1315": 450471936.0,
+ "1320": 450471936.0,
+ "1325": 450471936.0,
+ "1330": 450471936.0,
+ "1335": 450471936.0,
+ "1340": 450471936.0,
+ "1345": 450471936.0,
+ "1350": 450471936.0,
+ "1355": 450471936.0,
+ "1360": 450471936.0,
+ "1365": 450471936.0,
+ "1370": 450471936.0,
+ "1375": 450471936.0,
+ "1380": 450471936.0,
+ "1385": 450471936.0,
+ "1390": 450471936.0,
+ "1395": 450471936.0,
+ "1400": 450471936.0,
+ "1405": 450471936.0,
+ "1410": 450471936.0,
+ "1415": 450471936.0,
+ "1420": 450471936.0,
+ "1425": 450471936.0,
+ "1430": 450471936.0,
+ "1435": 450471936.0,
+ "1440": 450471936.0,
+ "1445": 450471936.0,
+ "1450": 450471936.0,
+ "1455": 450471936.0,
+ "1460": 450471936.0,
+ "1465": 450471936.0,
+ "1470": 450471936.0,
+ "1475": 450471936.0,
+ "1480": 450471936.0,
+ "1485": 450471936.0,
+ "1490": 450471936.0,
+ "1495": 450471936.0,
+ "1500": 450471936.0,
+ "1505": 450471936.0,
+ "1510": 450471936.0,
+ "1515": 450471936.0,
+ "1520": 450471936.0,
+ "1525": 450471936.0,
+ "1530": 450471936.0,
+ "1535": 450471936.0,
+ "1540": 450471936.0,
+ "1545": 450471936.0,
+ "1550": 450471936.0,
+ "1555": 450471936.0,
+ "1560": 450471936.0,
+ "1565": 450471936.0,
+ "1570": 450471936.0,
+ "1575": 450471936.0,
+ "1580": 450471936.0,
+ "1585": 450471936.0,
+ "1590": 450471936.0,
+ "1595": 450471936.0,
+ "1600": 450471936.0,
+ "1605": 450471936.0,
+ "1610": 450471936.0,
+ "1615": 450471936.0,
+ "1620": 450471936.0,
+ "1625": 450471936.0,
+ "1630": 450471936.0,
+ "1635": 450471936.0,
+ "1640": 450471936.0,
+ "1645": 450471936.0,
+ "1650": 450471936.0,
+ "1655": 450471936.0,
+ "1660": 450471936.0,
+ "1665": 450471936.0,
+ "1670": 450471936.0,
+ "1675": 450471936.0,
+ "1680": 450471936.0,
+ "1685": 450471936.0,
+ "1690": 450471936.0,
+ "1695": 450471936.0,
+ "1700": 450471936.0,
+ "1705": 450471936.0,
+ "1710": 450471936.0,
+ "1715": 450471936.0,
+ "1720": 450471936.0,
+ "1725": 450471936.0,
+ "1730": 450471936.0,
+ "1735": 450471936.0,
+ "1740": 450471936.0,
+ "1745": 450471936.0,
+ "1750": 450471936.0,
+ "1755": 450471936.0,
+ "1760": 450471936.0,
+ "1765": 450471936.0,
+ "1770": 450471936.0,
+ "1775": 450471936.0,
+ "1780": 450471936.0,
+ "1785": 450471936.0,
+ "1790": 450471936.0,
+ "1795": 450471936.0,
+ "1800": 450471936.0,
+ "1805": 450471936.0,
+ "1810": 450471936.0,
+ "1815": 450471936.0,
+ "1820": 450471936.0,
+ "1825": 450471936.0,
+ "1830": 450471936.0,
+ "1835": 450471936.0,
+ "1840": 450471936.0,
+ "1845": 450471936.0,
+ "1850": 450471936.0,
+ "1855": 450471936.0,
+ "1860": 450471936.0,
+ "1865": 450471936.0,
+ "1870": 450471936.0,
+ "1875": 450471936.0,
+ "1880": 450471936.0,
+ "1885": 450471936.0,
+ "1890": 450471936.0,
+ "1895": 450471936.0,
+ "1900": 450471936.0,
+ "1905": 450471936.0,
+ "1910": 450471936.0,
+ "1915": 450471936.0,
+ "1920": 450471936.0,
+ "1925": 450471936.0,
+ "1930": 450471936.0,
+ "1935": 450471936.0,
+ "1940": 450471936.0,
+ "1945": 450471936.0,
+ "1950": 450471936.0,
+ "1955": 450471936.0,
+ "1960": 450471936.0,
+ "1965": 450471936.0,
+ "1970": 450471936.0,
+ "1975": 450471936.0,
+ "1980": 450471936.0,
+ "1985": 450471936.0,
+ "1990": 450471936.0,
+ "1995": 450471936.0,
+ "2000": 450471936.0
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 5,
+ "values": {
+ "1": "nan",
+ "5": "nan",
+ "10": "nan",
+ "15": "nan",
+ "20": 11992.0,
+ "25": 12791.0,
+ "30": 17318.0,
+ "35": 13313.0,
+ "40": 13889.0,
+ "45": 15256.0,
+ "50": 11370.0,
+ "55": 14792.0,
+ "60": 13572.0,
+ "65": 14618.0,
+ "70": 10636.0,
+ "75": 12479.0,
+ "80": 11546.0,
+ "85": 17330.0,
+ "90": 10222.0,
+ "95": 12866.0,
+ "100": 10240.0,
+ "105": 12091.0,
+ "110": 15229.0,
+ "115": 13316.0,
+ "120": 15329.0,
+ "125": 10543.0,
+ "130": 13341.0,
+ "135": 11373.0,
+ "140": 12167.0,
+ "145": 16571.0,
+ "150": 13896.0,
+ "155": 16085.0,
+ "160": 12143.0,
+ "165": 16313.0,
+ "170": 10658.0,
+ "175": 13633.0,
+ "180": 10238.0,
+ "185": 12445.0,
+ "190": 11320.0,
+ "195": 15702.0,
+ "200": 14683.0,
+ "205": 15921.0,
+ "210": 13240.0,
+ "215": 13450.0,
+ "220": 18456.0,
+ "225": 12420.0,
+ "230": 13176.0,
+ "235": 13557.0,
+ "240": 14613.0,
+ "245": 13040.0,
+ "250": 14162.0,
+ "255": 14512.0,
+ "260": 13823.0,
+ "265": 18504.0,
+ "270": 15208.0,
+ "275": 13893.0,
+ "280": 11858.0,
+ "285": 14474.0,
+ "290": 15059.0,
+ "295": 15372.0,
+ "300": 12492.0,
+ "305": 13020.0,
+ "310": 16763.0,
+ "315": 16313.0,
+ "320": 14840.0,
+ "325": 14424.0,
+ "330": 14796.0,
+ "335": 13147.0,
+ "340": 13996.0,
+ "345": 16086.0,
+ "350": 16038.0,
+ "355": 11910.0,
+ "360": 13482.0,
+ "365": 15937.0,
+ "370": 15123.0,
+ "375": 11569.0,
+ "380": 18286.0,
+ "385": 15464.0,
+ "390": 13894.0,
+ "395": 14113.0,
+ "400": 16376.0,
+ "405": 17310.0,
+ "410": 13934.0,
+ "415": 19325.0,
+ "420": 14179.0,
+ "425": 20879.0,
+ "430": 12685.0,
+ "435": 14301.0,
+ "440": 16235.0,
+ "445": 17108.0,
+ "450": 13875.0,
+ "455": 17519.0,
+ "460": 15208.0,
+ "465": 15818.0,
+ "470": 14442.0,
+ "475": 16949.0,
+ "480": 15538.0,
+ "485": 16196.0,
+ "490": 15341.0,
+ "495": 16051.0,
+ "500": 13419.0,
+ "505": 15552.0,
+ "510": 16935.0,
+ "515": 16824.0,
+ "520": 12877.0,
+ "525": 14026.0,
+ "530": 13620.0,
+ "535": 13850.0,
+ "540": 14720.0,
+ "545": 12288.0,
+ "550": 15016.0,
+ "555": 16905.0,
+ "560": 15507.0,
+ "565": 17081.0,
+ "570": 16282.0,
+ "575": 15907.0,
+ "580": 14680.0,
+ "585": 17563.0,
+ "590": 14132.0,
+ "595": 17477.0,
+ "600": 17725.0,
+ "605": 14044.0,
+ "610": 15590.0,
+ "615": 16606.0,
+ "620": 17061.0,
+ "625": 14453.0,
+ "630": 17025.0,
+ "635": 17716.0,
+ "640": 15419.0,
+ "645": 13890.0,
+ "650": 16352.0,
+ "655": 15647.0,
+ "660": 15850.0,
+ "665": 15369.0,
+ "670": 16123.0,
+ "675": 17523.0,
+ "680": 14561.0,
+ "685": 13141.0,
+ "690": 14711.0,
+ "695": 15253.0,
+ "700": 14874.0,
+ "705": 15332.0,
+ "710": 14777.0,
+ "715": 16931.0,
+ "720": 15833.0,
+ "725": 19244.0,
+ "730": 14076.0,
+ "735": 16320.0,
+ "740": 16859.0,
+ "745": 13481.0,
+ "750": 14972.0,
+ "755": 15488.0,
+ "760": 15341.0,
+ "765": 16116.0,
+ "770": 15517.0,
+ "775": 14491.0,
+ "780": 15819.0,
+ "785": 14058.0,
+ "790": 17767.0,
+ "795": 13865.0,
+ "800": 15919.0,
+ "805": 15144.0,
+ "810": 16693.0,
+ "815": 18203.0,
+ "820": 16243.0,
+ "825": 13355.0,
+ "830": 17683.0,
+ "835": 14729.0,
+ "840": 18498.0,
+ "845": 16504.0,
+ "850": 16146.0,
+ "855": 15142.0,
+ "860": 15373.0,
+ "865": 15746.0,
+ "870": 16742.0,
+ "875": 15626.0,
+ "880": 13812.0,
+ "885": 14653.0,
+ "890": 16362.0,
+ "895": 15496.0,
+ "900": 16213.0,
+ "905": 16081.0,
+ "910": 17628.0,
+ "915": 15995.0,
+ "920": 15730.0,
+ "925": 15026.0,
+ "930": 15889.0,
+ "935": 15551.0,
+ "940": 17077.0,
+ "945": 15037.0,
+ "950": 17784.0,
+ "955": 16426.0,
+ "960": 18232.0,
+ "965": 13735.0,
+ "970": 12534.0,
+ "975": 14917.0,
+ "980": 15858.0,
+ "985": 16929.0,
+ "990": 16031.0,
+ "995": 14929.0,
+ "1000": 18652.0,
+ "1005": 14100.0,
+ "1010": 14165.0,
+ "1015": 16922.0,
+ "1020": 15414.0,
+ "1025": 17850.0,
+ "1030": 15709.0,
+ "1035": 14603.0,
+ "1040": 14928.0,
+ "1045": 18600.0,
+ "1050": 15802.0,
+ "1055": 19174.0,
+ "1060": 15022.0,
+ "1065": 14666.0,
+ "1070": 15402.0,
+ "1075": 14816.0,
+ "1080": 15724.0,
+ "1085": 14881.0,
+ "1090": 18898.0,
+ "1095": 16011.0,
+ "1100": 16835.0,
+ "1105": 18298.0,
+ "1110": 13161.0,
+ "1115": 19553.0,
+ "1120": 15198.0,
+ "1125": 15939.0,
+ "1130": 16832.0,
+ "1135": 16160.0,
+ "1140": 17129.0,
+ "1145": 18492.0,
+ "1150": 13452.0,
+ "1155": 15636.0,
+ "1160": 15013.0,
+ "1165": 17731.0,
+ "1170": 21903.0,
+ "1175": 16121.0,
+ "1180": 15737.0,
+ "1185": 19161.0,
+ "1190": 18360.0,
+ "1195": 15405.0,
+ "1200": 17015.0,
+ "1205": 12343.0,
+ "1210": 14123.0,
+ "1215": 15810.0,
+ "1220": 13957.0,
+ "1225": 14139.0,
+ "1230": 17362.0,
+ "1235": 15869.0,
+ "1240": 15904.0,
+ "1245": 18478.0,
+ "1250": 16019.0,
+ "1255": 14828.0,
+ "1260": 14875.0,
+ "1265": 14493.0,
+ "1270": 14007.0,
+ "1275": 13660.0,
+ "1280": 14056.0,
+ "1285": 17708.0,
+ "1290": 15145.0,
+ "1295": 18088.0,
+ "1300": 17203.0,
+ "1305": 16560.0,
+ "1310": 15669.0,
+ "1315": 17341.0,
+ "1320": 16307.0,
+ "1325": 17612.0,
+ "1330": 13539.0,
+ "1335": 13802.0,
+ "1340": 16415.0,
+ "1345": 17711.0,
+ "1350": 17117.0,
+ "1355": 14693.0,
+ "1360": 17885.0,
+ "1365": 17267.0,
+ "1370": 16646.0,
+ "1375": 16270.0,
+ "1380": 17787.0,
+ "1385": 20402.0,
+ "1390": 22226.0,
+ "1395": 16008.0,
+ "1400": 13993.0,
+ "1405": 16304.0,
+ "1410": 16616.0,
+ "1415": 14189.0,
+ "1420": 20514.0,
+ "1425": 19277.0,
+ "1430": 20527.0,
+ "1435": 20429.0,
+ "1440": 15174.0,
+ "1445": 18024.0,
+ "1450": 15190.0,
+ "1455": 15578.0,
+ "1460": 15951.0,
+ "1465": 15106.0,
+ "1470": 17513.0,
+ "1475": 15760.0,
+ "1480": 16046.0,
+ "1485": 19743.0,
+ "1490": 14777.0,
+ "1495": 17049.0,
+ "1500": 14367.0,
+ "1505": 15647.0,
+ "1510": 21215.0,
+ "1515": 16621.0,
+ "1520": 15834.0,
+ "1525": 16291.0,
+ "1530": 17942.0,
+ "1535": 14546.0,
+ "1540": 15021.0,
+ "1545": 16479.0,
+ "1550": 13428.0,
+ "1555": 17283.0,
+ "1560": 14157.0,
+ "1565": 22260.0,
+ "1570": 17688.0,
+ "1575": 14820.0,
+ "1580": 15551.0,
+ "1585": 17850.0,
+ "1590": 14709.0,
+ "1595": 13862.0,
+ "1600": 18194.0,
+ "1605": 14203.0,
+ "1610": 15675.0,
+ "1615": 19634.0,
+ "1620": 18751.0,
+ "1625": 15551.0,
+ "1630": 16515.0,
+ "1635": 15822.0,
+ "1640": 15486.0,
+ "1645": 19133.0,
+ "1650": 15387.0,
+ "1655": 15879.0,
+ "1660": 17098.0,
+ "1665": 20649.0,
+ "1670": 15996.0,
+ "1675": 17422.0,
+ "1680": 16103.0,
+ "1685": 15754.0,
+ "1690": 15361.0,
+ "1695": 14877.0,
+ "1700": 16444.0,
+ "1705": 15040.0,
+ "1710": 22005.0,
+ "1715": 16108.0,
+ "1720": 17863.0,
+ "1725": 17126.0,
+ "1730": 15137.0,
+ "1735": 16200.0,
+ "1740": 16536.0,
+ "1745": 17812.0,
+ "1750": 12662.0,
+ "1755": 17016.0,
+ "1760": 17337.0,
+ "1765": 16694.0,
+ "1770": 15580.0,
+ "1775": 20158.0,
+ "1780": 15690.0,
+ "1785": 17227.0,
+ "1790": 16492.0,
+ "1795": 16192.0,
+ "1800": 15733.0,
+ "1805": 15477.0,
+ "1810": 15899.0,
+ "1815": 19125.0,
+ "1820": 16134.0,
+ "1825": 14656.0,
+ "1830": 17925.0,
+ "1835": 15461.0,
+ "1840": 15620.0,
+ "1845": 17909.0,
+ "1850": 16805.0,
+ "1855": 16469.0,
+ "1860": 16604.0,
+ "1865": 18986.0,
+ "1870": 14919.0,
+ "1875": 16415.0,
+ "1880": 18604.0,
+ "1885": 15624.0,
+ "1890": 18572.0,
+ "1895": 16082.0,
+ "1900": 13936.0,
+ "1905": 15561.0,
+ "1910": 18143.0,
+ "1915": 15272.0,
+ "1920": 14559.0,
+ "1925": 16145.0,
+ "1930": 14397.0,
+ "1935": 18377.0,
+ "1940": 16762.0,
+ "1945": 16837.0,
+ "1950": 18209.0,
+ "1955": 17811.0,
+ "1960": 18479.0,
+ "1965": 16120.0,
+ "1970": 18199.0,
+ "1975": 16385.0,
+ "1980": 15824.0,
+ "1985": 18034.0,
+ "1990": 16429.0,
+ "1995": 15744.0,
+ "2000": 16481.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 5,
+ "values": {
+ "1": "nan",
+ "5": "nan",
+ "10": "nan",
+ "15": "nan",
+ "20": "nan",
+ "25": "nan",
+ "30": "nan",
+ "35": "nan",
+ "40": "nan",
+ "45": "nan",
+ "50": "nan",
+ "55": "nan",
+ "60": "nan",
+ "65": "nan",
+ "70": "nan",
+ "75": "nan",
+ "80": "nan",
+ "85": "nan",
+ "90": "nan",
+ "95": "nan",
+ "100": 0.7866,
+ "105": "nan",
+ "110": "nan",
+ "115": "nan",
+ "120": "nan",
+ "125": "nan",
+ "130": "nan",
+ "135": "nan",
+ "140": "nan",
+ "145": "nan",
+ "150": "nan",
+ "155": "nan",
+ "160": "nan",
+ "165": "nan",
+ "170": "nan",
+ "175": "nan",
+ "180": "nan",
+ "185": "nan",
+ "190": "nan",
+ "195": "nan",
+ "200": 0.46824,
+ "205": "nan",
+ "210": "nan",
+ "215": "nan",
+ "220": "nan",
+ "225": "nan",
+ "230": "nan",
+ "235": "nan",
+ "240": "nan",
+ "245": "nan",
+ "250": "nan",
+ "255": "nan",
+ "260": "nan",
+ "265": "nan",
+ "270": "nan",
+ "275": "nan",
+ "280": "nan",
+ "285": "nan",
+ "290": "nan",
+ "295": "nan",
+ "300": 0.39004,
+ "305": "nan",
+ "310": "nan",
+ "315": "nan",
+ "320": "nan",
+ "325": "nan",
+ "330": "nan",
+ "335": "nan",
+ "340": "nan",
+ "345": "nan",
+ "350": "nan",
+ "355": "nan",
+ "360": "nan",
+ "365": "nan",
+ "370": "nan",
+ "375": "nan",
+ "380": "nan",
+ "385": "nan",
+ "390": "nan",
+ "395": "nan",
+ "400": 0.38578,
+ "405": "nan",
+ "410": "nan",
+ "415": "nan",
+ "420": "nan",
+ "425": "nan",
+ "430": "nan",
+ "435": "nan",
+ "440": "nan",
+ "445": "nan",
+ "450": "nan",
+ "455": "nan",
+ "460": "nan",
+ "465": "nan",
+ "470": "nan",
+ "475": "nan",
+ "480": "nan",
+ "485": "nan",
+ "490": "nan",
+ "495": "nan",
+ "500": 0.38197,
+ "505": "nan",
+ "510": "nan",
+ "515": "nan",
+ "520": "nan",
+ "525": "nan",
+ "530": "nan",
+ "535": "nan",
+ "540": "nan",
+ "545": "nan",
+ "550": "nan",
+ "555": "nan",
+ "560": "nan",
+ "565": "nan",
+ "570": "nan",
+ "575": "nan",
+ "580": "nan",
+ "585": "nan",
+ "590": "nan",
+ "595": "nan",
+ "600": 0.40009,
+ "605": "nan",
+ "610": "nan",
+ "615": "nan",
+ "620": "nan",
+ "625": "nan",
+ "630": "nan",
+ "635": "nan",
+ "640": "nan",
+ "645": "nan",
+ "650": "nan",
+ "655": "nan",
+ "660": "nan",
+ "665": "nan",
+ "670": "nan",
+ "675": "nan",
+ "680": "nan",
+ "685": "nan",
+ "690": "nan",
+ "695": "nan",
+ "700": 0.4229,
+ "705": "nan",
+ "710": "nan",
+ "715": "nan",
+ "720": "nan",
+ "725": "nan",
+ "730": "nan",
+ "735": "nan",
+ "740": "nan",
+ "745": "nan",
+ "750": "nan",
+ "755": "nan",
+ "760": "nan",
+ "765": "nan",
+ "770": "nan",
+ "775": "nan",
+ "780": "nan",
+ "785": "nan",
+ "790": "nan",
+ "795": "nan",
+ "800": 0.41403,
+ "805": "nan",
+ "810": "nan",
+ "815": "nan",
+ "820": "nan",
+ "825": "nan",
+ "830": "nan",
+ "835": "nan",
+ "840": "nan",
+ "845": "nan",
+ "850": "nan",
+ "855": "nan",
+ "860": "nan",
+ "865": "nan",
+ "870": "nan",
+ "875": "nan",
+ "880": "nan",
+ "885": "nan",
+ "890": "nan",
+ "895": "nan",
+ "900": 0.43862,
+ "905": "nan",
+ "910": "nan",
+ "915": "nan",
+ "920": "nan",
+ "925": "nan",
+ "930": "nan",
+ "935": "nan",
+ "940": "nan",
+ "945": "nan",
+ "950": "nan",
+ "955": "nan",
+ "960": "nan",
+ "965": "nan",
+ "970": "nan",
+ "975": "nan",
+ "980": "nan",
+ "985": "nan",
+ "990": "nan",
+ "995": "nan",
+ "1000": 0.40449,
+ "1005": "nan",
+ "1010": "nan",
+ "1015": "nan",
+ "1020": "nan",
+ "1025": "nan",
+ "1030": "nan",
+ "1035": "nan",
+ "1040": "nan",
+ "1045": "nan",
+ "1050": "nan",
+ "1055": "nan",
+ "1060": "nan",
+ "1065": "nan",
+ "1070": "nan",
+ "1075": "nan",
+ "1080": "nan",
+ "1085": "nan",
+ "1090": "nan",
+ "1095": "nan",
+ "1100": 0.37487,
+ "1105": "nan",
+ "1110": "nan",
+ "1115": "nan",
+ "1120": "nan",
+ "1125": "nan",
+ "1130": "nan",
+ "1135": "nan",
+ "1140": "nan",
+ "1145": "nan",
+ "1150": "nan",
+ "1155": "nan",
+ "1160": "nan",
+ "1165": "nan",
+ "1170": "nan",
+ "1175": "nan",
+ "1180": "nan",
+ "1185": "nan",
+ "1190": "nan",
+ "1195": "nan",
+ "1200": 0.38459,
+ "1205": "nan",
+ "1210": "nan",
+ "1215": "nan",
+ "1220": "nan",
+ "1225": "nan",
+ "1230": "nan",
+ "1235": "nan",
+ "1240": "nan",
+ "1245": "nan",
+ "1250": "nan",
+ "1255": "nan",
+ "1260": "nan",
+ "1265": "nan",
+ "1270": "nan",
+ "1275": "nan",
+ "1280": "nan",
+ "1285": "nan",
+ "1290": "nan",
+ "1295": "nan",
+ "1300": 0.40402,
+ "1305": "nan",
+ "1310": "nan",
+ "1315": "nan",
+ "1320": "nan",
+ "1325": "nan",
+ "1330": "nan",
+ "1335": "nan",
+ "1340": "nan",
+ "1345": "nan",
+ "1350": "nan",
+ "1355": "nan",
+ "1360": "nan",
+ "1365": "nan",
+ "1370": "nan",
+ "1375": "nan",
+ "1380": "nan",
+ "1385": "nan",
+ "1390": "nan",
+ "1395": "nan",
+ "1400": 0.37795,
+ "1405": "nan",
+ "1410": "nan",
+ "1415": "nan",
+ "1420": "nan",
+ "1425": "nan",
+ "1430": "nan",
+ "1435": "nan",
+ "1440": "nan",
+ "1445": "nan",
+ "1450": "nan",
+ "1455": "nan",
+ "1460": "nan",
+ "1465": "nan",
+ "1470": "nan",
+ "1475": "nan",
+ "1480": "nan",
+ "1485": "nan",
+ "1490": "nan",
+ "1495": "nan",
+ "1500": 0.38043,
+ "1505": "nan",
+ "1510": "nan",
+ "1515": "nan",
+ "1520": "nan",
+ "1525": "nan",
+ "1530": "nan",
+ "1535": "nan",
+ "1540": "nan",
+ "1545": "nan",
+ "1550": "nan",
+ "1555": "nan",
+ "1560": "nan",
+ "1565": "nan",
+ "1570": "nan",
+ "1575": "nan",
+ "1580": "nan",
+ "1585": "nan",
+ "1590": "nan",
+ "1595": "nan",
+ "1600": 0.38498,
+ "1605": "nan",
+ "1610": "nan",
+ "1615": "nan",
+ "1620": "nan",
+ "1625": "nan",
+ "1630": "nan",
+ "1635": "nan",
+ "1640": "nan",
+ "1645": "nan",
+ "1650": "nan",
+ "1655": "nan",
+ "1660": "nan",
+ "1665": "nan",
+ "1670": "nan",
+ "1675": "nan",
+ "1680": "nan",
+ "1685": "nan",
+ "1690": "nan",
+ "1695": "nan",
+ "1700": 0.38993,
+ "1705": "nan",
+ "1710": "nan",
+ "1715": "nan",
+ "1720": "nan",
+ "1725": "nan",
+ "1730": "nan",
+ "1735": "nan",
+ "1740": "nan",
+ "1745": "nan",
+ "1750": "nan",
+ "1755": "nan",
+ "1760": "nan",
+ "1765": "nan",
+ "1770": "nan",
+ "1775": "nan",
+ "1780": "nan",
+ "1785": "nan",
+ "1790": "nan",
+ "1795": "nan",
+ "1800": 0.37943,
+ "1805": "nan",
+ "1810": "nan",
+ "1815": "nan",
+ "1820": "nan",
+ "1825": "nan",
+ "1830": "nan",
+ "1835": "nan",
+ "1840": "nan",
+ "1845": "nan",
+ "1850": "nan",
+ "1855": "nan",
+ "1860": "nan",
+ "1865": "nan",
+ "1870": "nan",
+ "1875": "nan",
+ "1880": "nan",
+ "1885": "nan",
+ "1890": "nan",
+ "1895": "nan",
+ "1900": 0.38578,
+ "1905": "nan",
+ "1910": "nan",
+ "1915": "nan",
+ "1920": "nan",
+ "1925": "nan",
+ "1930": "nan",
+ "1935": "nan",
+ "1940": "nan",
+ "1945": "nan",
+ "1950": "nan",
+ "1955": "nan",
+ "1960": "nan",
+ "1965": "nan",
+ "1970": "nan",
+ "1975": "nan",
+ "1980": "nan",
+ "1985": "nan",
+ "1990": "nan",
+ "1995": "nan",
+ "2000": 0.44172
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/bert/bert_release_sm/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/bert/bert_release_sm/golden_values_dev_dgx_h100.json
new file mode 100644
index 00000000000..d1ae85719d6
--- /dev/null
+++ b/tests/functional_tests/test_cases/bert/bert_release_sm/golden_values_dev_dgx_h100.json
@@ -0,0 +1,2042 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 5,
+ "values": {
+ "1": 10.52518,
+ "5": 10.54711,
+ "10": 10.57625,
+ "15": 10.53228,
+ "20": 10.1977,
+ "25": 9.60859,
+ "30": 9.42961,
+ "35": 9.13438,
+ "40": 9.11757,
+ "45": 8.9598,
+ "50": 8.69439,
+ "55": 8.74614,
+ "60": 8.5262,
+ "65": 8.48894,
+ "70": 8.18786,
+ "75": 8.30128,
+ "80": 7.98015,
+ "85": 7.75995,
+ "90": 7.67797,
+ "95": 7.58098,
+ "100": 7.53685,
+ "105": 7.37115,
+ "110": 7.47741,
+ "115": 7.16871,
+ "120": 6.95404,
+ "125": 7.22744,
+ "130": 6.98847,
+ "135": 6.78308,
+ "140": 7.19909,
+ "145": 6.9713,
+ "150": 7.073,
+ "155": 6.97305,
+ "160": 7.00146,
+ "165": 7.0568,
+ "170": 6.9259,
+ "175": 6.8686,
+ "180": 6.70845,
+ "185": 6.92037,
+ "190": 7.10711,
+ "195": 6.94684,
+ "200": 6.84551,
+ "205": 6.89554,
+ "210": 6.74732,
+ "215": 6.98353,
+ "220": 6.66662,
+ "225": 6.73845,
+ "230": 6.68939,
+ "235": 6.80476,
+ "240": 6.57309,
+ "245": 7.09103,
+ "250": 6.60586,
+ "255": 6.6257,
+ "260": 6.71981,
+ "265": 6.60514,
+ "270": 6.58499,
+ "275": 6.84108,
+ "280": 6.51956,
+ "285": 6.5881,
+ "290": 6.77888,
+ "295": 6.73633,
+ "300": 6.45158,
+ "305": 6.35295,
+ "310": 6.95462,
+ "315": 6.92967,
+ "320": 6.85163,
+ "325": 6.65274,
+ "330": 6.63877,
+ "335": 6.6328,
+ "340": 6.58326,
+ "345": 6.55319,
+ "350": 6.39654,
+ "355": 6.48845,
+ "360": 6.55577,
+ "365": 6.60764,
+ "370": 6.50698,
+ "375": 6.60894,
+ "380": 6.43397,
+ "385": 6.40337,
+ "390": 6.45377,
+ "395": 6.25318,
+ "400": 6.64296,
+ "405": 6.48078,
+ "410": 6.36101,
+ "415": 6.56149,
+ "420": 6.58523,
+ "425": 6.51312,
+ "430": 6.53315,
+ "435": 6.51013,
+ "440": 6.53088,
+ "445": 6.54469,
+ "450": 6.58192,
+ "455": 6.61907,
+ "460": 6.7284,
+ "465": 6.7023,
+ "470": 6.4933,
+ "475": 6.22155,
+ "480": 6.44352,
+ "485": 6.51276,
+ "490": 6.16963,
+ "495": 6.18146,
+ "500": 6.58793,
+ "505": 6.61849,
+ "510": 6.51771,
+ "515": 6.19808,
+ "520": 6.38782,
+ "525": 6.4449,
+ "530": 6.38794,
+ "535": 5.9946,
+ "540": 6.43932,
+ "545": 6.34319,
+ "550": 6.42034,
+ "555": 6.48463,
+ "560": 6.41058,
+ "565": 6.18975,
+ "570": 6.26475,
+ "575": 6.60776,
+ "580": 6.40861,
+ "585": 6.44091,
+ "590": 6.17827,
+ "595": 6.26133,
+ "600": 6.48062,
+ "605": 6.48987,
+ "610": 6.30567,
+ "615": 6.41793,
+ "620": 6.50242,
+ "625": 6.41944,
+ "630": 6.42269,
+ "635": 6.48993,
+ "640": 6.3142,
+ "645": 6.26425,
+ "650": 6.24968,
+ "655": 6.54216,
+ "660": 6.32704,
+ "665": 6.28362,
+ "670": 6.34635,
+ "675": 6.25007,
+ "680": 6.36366,
+ "685": 6.51787,
+ "690": 6.52306,
+ "695": 6.19164,
+ "700": 6.02684,
+ "705": 6.27937,
+ "710": 6.15186,
+ "715": 6.37122,
+ "720": 6.24474,
+ "725": 6.4324,
+ "730": 6.54322,
+ "735": 6.59039,
+ "740": 6.56551,
+ "745": 6.25838,
+ "750": 6.26025,
+ "755": 5.99142,
+ "760": 6.56569,
+ "765": 6.36209,
+ "770": 6.41801,
+ "775": 6.14492,
+ "780": 6.22431,
+ "785": 6.2837,
+ "790": 6.49366,
+ "795": 6.36524,
+ "800": 6.31734,
+ "805": 6.50546,
+ "810": 6.18094,
+ "815": 6.23859,
+ "820": 6.44193,
+ "825": 6.50405,
+ "830": 6.39936,
+ "835": 6.3749,
+ "840": 6.32574,
+ "845": 6.18599,
+ "850": 6.37284,
+ "855": 6.39409,
+ "860": 6.25521,
+ "865": 6.28276,
+ "870": 6.36092,
+ "875": 6.40236,
+ "880": 6.28502,
+ "885": 6.48567,
+ "890": 6.18932,
+ "895": 6.27216,
+ "900": 6.26478,
+ "905": 6.36788,
+ "910": 6.06337,
+ "915": 6.49909,
+ "920": 6.33728,
+ "925": 6.27291,
+ "930": 6.20875,
+ "935": 6.2218,
+ "940": 6.36607,
+ "945": 6.03875,
+ "950": 6.16958,
+ "955": 6.1498,
+ "960": 6.23218,
+ "965": 6.27167,
+ "970": 6.27182,
+ "975": 6.18992,
+ "980": 6.0871,
+ "985": 6.23515,
+ "990": 6.15401,
+ "995": 6.39715,
+ "1000": 6.199,
+ "1005": 6.41564,
+ "1010": 6.10307,
+ "1015": 6.18253,
+ "1020": 6.0715,
+ "1025": 6.34874,
+ "1030": 6.15345,
+ "1035": 5.91993,
+ "1040": 6.00182,
+ "1045": 6.32773,
+ "1050": 6.44289,
+ "1055": 6.09205,
+ "1060": 6.11626,
+ "1065": 5.94789,
+ "1070": 6.18822,
+ "1075": 5.96807,
+ "1080": 6.14028,
+ "1085": 6.00403,
+ "1090": 6.07878,
+ "1095": 6.39724,
+ "1100": 6.07371,
+ "1105": 6.27733,
+ "1110": 6.17767,
+ "1115": 6.05213,
+ "1120": 6.03505,
+ "1125": 6.14642,
+ "1130": 5.91905,
+ "1135": 6.33701,
+ "1140": 6.4673,
+ "1145": 6.24062,
+ "1150": 6.11148,
+ "1155": 6.00366,
+ "1160": 6.07106,
+ "1165": 6.12458,
+ "1170": 6.31238,
+ "1175": 6.06133,
+ "1180": 6.23575,
+ "1185": 6.28767,
+ "1190": 6.32616,
+ "1195": 6.20409,
+ "1200": 6.1787,
+ "1205": 6.27853,
+ "1210": 6.124,
+ "1215": 6.02269,
+ "1220": 6.21049,
+ "1225": 6.31462,
+ "1230": 6.27525,
+ "1235": 6.07179,
+ "1240": 6.44848,
+ "1245": 6.08257,
+ "1250": 5.89585,
+ "1255": 6.22473,
+ "1260": 6.21804,
+ "1265": 5.98499,
+ "1270": 6.05378,
+ "1275": 6.11328,
+ "1280": 5.79212,
+ "1285": 5.96076,
+ "1290": 6.02109,
+ "1295": 6.15922,
+ "1300": 6.33887,
+ "1305": 5.93778,
+ "1310": 5.98927,
+ "1315": 6.04956,
+ "1320": 6.11263,
+ "1325": 6.15042,
+ "1330": 6.10207,
+ "1335": 6.03588,
+ "1340": 6.09098,
+ "1345": 6.17077,
+ "1350": 5.94377,
+ "1355": 6.10134,
+ "1360": 6.44012,
+ "1365": 5.97453,
+ "1370": 6.08617,
+ "1375": 6.26098,
+ "1380": 6.08712,
+ "1385": 5.99329,
+ "1390": 5.83651,
+ "1395": 5.87328,
+ "1400": 6.13648,
+ "1405": 6.12933,
+ "1410": 6.0632,
+ "1415": 6.25026,
+ "1420": 6.196,
+ "1425": 6.18042,
+ "1430": 6.01123,
+ "1435": 6.51218,
+ "1440": 6.34508,
+ "1445": 6.3179,
+ "1450": 6.0905,
+ "1455": 5.80082,
+ "1460": 6.19257,
+ "1465": 6.16862,
+ "1470": 6.26836,
+ "1475": 6.25295,
+ "1480": 6.30886,
+ "1485": 6.05106,
+ "1490": 6.25044,
+ "1495": 6.25631,
+ "1500": 6.17856,
+ "1505": 6.24903,
+ "1510": 6.38972,
+ "1515": 6.04357,
+ "1520": 5.83213,
+ "1525": 5.90258,
+ "1530": 5.87533,
+ "1535": 6.49284,
+ "1540": 6.40956,
+ "1545": 6.36558,
+ "1550": 6.13092,
+ "1555": 6.06429,
+ "1560": 6.15867,
+ "1565": 6.04309,
+ "1570": 5.99586,
+ "1575": 6.10514,
+ "1580": 6.22456,
+ "1585": 6.04057,
+ "1590": 6.25663,
+ "1595": 6.16142,
+ "1600": 6.07455,
+ "1605": 6.05807,
+ "1610": 6.14498,
+ "1615": 6.12337,
+ "1620": 6.03388,
+ "1625": 6.07928,
+ "1630": 6.09847,
+ "1635": 6.32276,
+ "1640": 6.14014,
+ "1645": 6.19561,
+ "1650": 6.29657,
+ "1655": 6.36122,
+ "1660": 6.224,
+ "1665": 6.06213,
+ "1670": 6.16801,
+ "1675": 5.86985,
+ "1680": 6.09215,
+ "1685": 6.38653,
+ "1690": 6.14996,
+ "1695": 5.99868,
+ "1700": 6.23211,
+ "1705": 6.07235,
+ "1710": 6.00732,
+ "1715": 6.02437,
+ "1720": 6.17613,
+ "1725": 6.38947,
+ "1730": 6.18961,
+ "1735": 6.02104,
+ "1740": 6.11735,
+ "1745": 6.1043,
+ "1750": 6.32512,
+ "1755": 6.27834,
+ "1760": 5.98423,
+ "1765": 6.01155,
+ "1770": 6.1114,
+ "1775": 5.9844,
+ "1780": 6.04965,
+ "1785": 5.822,
+ "1790": 5.96921,
+ "1795": 6.31597,
+ "1800": 6.16889,
+ "1805": 5.81951,
+ "1810": 6.48012,
+ "1815": 6.15535,
+ "1820": 6.46764,
+ "1825": 6.1539,
+ "1830": 6.08742,
+ "1835": 5.96482,
+ "1840": 5.88116,
+ "1845": 6.30056,
+ "1850": 6.25113,
+ "1855": 6.11407,
+ "1860": 6.04051,
+ "1865": 6.0148,
+ "1870": 6.34421,
+ "1875": 6.09027,
+ "1880": 5.9285,
+ "1885": 6.28984,
+ "1890": 6.10867,
+ "1895": 5.95919,
+ "1900": 6.07489,
+ "1905": 6.31557,
+ "1910": 6.30316,
+ "1915": 6.31794,
+ "1920": 6.18453,
+ "1925": 5.99466,
+ "1930": 6.09093,
+ "1935": 5.9949,
+ "1940": 6.13025,
+ "1945": 6.08214,
+ "1950": 6.02806,
+ "1955": 6.03901,
+ "1960": 6.00841,
+ "1965": 6.00114,
+ "1970": 6.05624,
+ "1975": 6.24372,
+ "1980": 6.3997,
+ "1985": 6.11374,
+ "1990": 5.97077,
+ "1995": 5.92911,
+ "2000": 6.07324
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 5,
+ "values": {
+ "1": 244161024.0,
+ "5": 244161024.0,
+ "10": 244161024.0,
+ "15": 244161024.0,
+ "20": 322521088.0,
+ "25": 320948224.0,
+ "30": 320948224.0,
+ "35": 320948224.0,
+ "40": 322521088.0,
+ "45": 320948224.0,
+ "50": 320948224.0,
+ "55": 320948224.0,
+ "60": 320948224.0,
+ "65": 320948224.0,
+ "70": 320948224.0,
+ "75": 320948224.0,
+ "80": 322521088.0,
+ "85": 320948224.0,
+ "90": 322521088.0,
+ "95": 322521088.0,
+ "100": 322521088.0,
+ "105": 322521088.0,
+ "110": 322521088.0,
+ "115": 322521088.0,
+ "120": 322521088.0,
+ "125": 322521088.0,
+ "130": 322521088.0,
+ "135": 320948224.0,
+ "140": 320948224.0,
+ "145": 322521088.0,
+ "150": 322521088.0,
+ "155": 322521088.0,
+ "160": 322521088.0,
+ "165": 322521088.0,
+ "170": 322521088.0,
+ "175": 320948224.0,
+ "180": 320948224.0,
+ "185": 320948224.0,
+ "190": 322521088.0,
+ "195": 322521088.0,
+ "200": 322521088.0,
+ "205": 320948224.0,
+ "210": 320948224.0,
+ "215": 322521088.0,
+ "220": 322521088.0,
+ "225": 320948224.0,
+ "230": 320948224.0,
+ "235": 322521088.0,
+ "240": 322521088.0,
+ "245": 320948224.0,
+ "250": 320948224.0,
+ "255": 320948224.0,
+ "260": 322521088.0,
+ "265": 322521088.0,
+ "270": 322521088.0,
+ "275": 322521088.0,
+ "280": 322521088.0,
+ "285": 322521088.0,
+ "290": 320948224.0,
+ "295": 320948224.0,
+ "300": 320948224.0,
+ "305": 320948224.0,
+ "310": 320948224.0,
+ "315": 320948224.0,
+ "320": 320948224.0,
+ "325": 320948224.0,
+ "330": 320948224.0,
+ "335": 320948224.0,
+ "340": 320948224.0,
+ "345": 320948224.0,
+ "350": 320948224.0,
+ "355": 320948224.0,
+ "360": 320948224.0,
+ "365": 320948224.0,
+ "370": 320948224.0,
+ "375": 320948224.0,
+ "380": 320948224.0,
+ "385": 320948224.0,
+ "390": 320948224.0,
+ "395": 320948224.0,
+ "400": 320948224.0,
+ "405": 320948224.0,
+ "410": 322521088.0,
+ "415": 322521088.0,
+ "420": 322521088.0,
+ "425": 322521088.0,
+ "430": 322521088.0,
+ "435": 320948224.0,
+ "440": 320948224.0,
+ "445": 320948224.0,
+ "450": 320948224.0,
+ "455": 322521088.0,
+ "460": 320948224.0,
+ "465": 322521088.0,
+ "470": 322521088.0,
+ "475": 322521088.0,
+ "480": 322521088.0,
+ "485": 320948224.0,
+ "490": 322521088.0,
+ "495": 322521088.0,
+ "500": 322521088.0,
+ "505": 322521088.0,
+ "510": 320948224.0,
+ "515": 322521088.0,
+ "520": 322521088.0,
+ "525": 322521088.0,
+ "530": 320948224.0,
+ "535": 320948224.0,
+ "540": 320948224.0,
+ "545": 320948224.0,
+ "550": 320948224.0,
+ "555": 320948224.0,
+ "560": 320948224.0,
+ "565": 320948224.0,
+ "570": 320948224.0,
+ "575": 320948224.0,
+ "580": 320948224.0,
+ "585": 320948224.0,
+ "590": 320948224.0,
+ "595": 320948224.0,
+ "600": 320948224.0,
+ "605": 320948224.0,
+ "610": 320948224.0,
+ "615": 320948224.0,
+ "620": 320948224.0,
+ "625": 320948224.0,
+ "630": 320948224.0,
+ "635": 320948224.0,
+ "640": 320948224.0,
+ "645": 320948224.0,
+ "650": 320948224.0,
+ "655": 320948224.0,
+ "660": 322521088.0,
+ "665": 320948224.0,
+ "670": 320948224.0,
+ "675": 320948224.0,
+ "680": 322521088.0,
+ "685": 322521088.0,
+ "690": 322521088.0,
+ "695": 322521088.0,
+ "700": 322521088.0,
+ "705": 322521088.0,
+ "710": 322521088.0,
+ "715": 322521088.0,
+ "720": 322521088.0,
+ "725": 322521088.0,
+ "730": 320948224.0,
+ "735": 320948224.0,
+ "740": 320948224.0,
+ "745": 320948224.0,
+ "750": 320948224.0,
+ "755": 320948224.0,
+ "760": 320948224.0,
+ "765": 320948224.0,
+ "770": 320948224.0,
+ "775": 320948224.0,
+ "780": 320948224.0,
+ "785": 322521088.0,
+ "790": 320948224.0,
+ "795": 320948224.0,
+ "800": 320948224.0,
+ "805": 320948224.0,
+ "810": 320948224.0,
+ "815": 322521088.0,
+ "820": 322521088.0,
+ "825": 322521088.0,
+ "830": 322521088.0,
+ "835": 322521088.0,
+ "840": 322521088.0,
+ "845": 322521088.0,
+ "850": 320948224.0,
+ "855": 320948224.0,
+ "860": 320948224.0,
+ "865": 320948224.0,
+ "870": 320948224.0,
+ "875": 320948224.0,
+ "880": 322521088.0,
+ "885": 320948224.0,
+ "890": 320948224.0,
+ "895": 322521088.0,
+ "900": 322521088.0,
+ "905": 320948224.0,
+ "910": 320948224.0,
+ "915": 320948224.0,
+ "920": 320948224.0,
+ "925": 320948224.0,
+ "930": 320948224.0,
+ "935": 320948224.0,
+ "940": 320948224.0,
+ "945": 320948224.0,
+ "950": 322521088.0,
+ "955": 320948224.0,
+ "960": 320948224.0,
+ "965": 320948224.0,
+ "970": 320948224.0,
+ "975": 320948224.0,
+ "980": 320948224.0,
+ "985": 320948224.0,
+ "990": 322521088.0,
+ "995": 322521088.0,
+ "1000": 322521088.0,
+ "1005": 322521088.0,
+ "1010": 322521088.0,
+ "1015": 322521088.0,
+ "1020": 322521088.0,
+ "1025": 322521088.0,
+ "1030": 322521088.0,
+ "1035": 322521088.0,
+ "1040": 322521088.0,
+ "1045": 322521088.0,
+ "1050": 322521088.0,
+ "1055": 322521088.0,
+ "1060": 320948224.0,
+ "1065": 320948224.0,
+ "1070": 320948224.0,
+ "1075": 320948224.0,
+ "1080": 320948224.0,
+ "1085": 320948224.0,
+ "1090": 320948224.0,
+ "1095": 320948224.0,
+ "1100": 320948224.0,
+ "1105": 320948224.0,
+ "1110": 320948224.0,
+ "1115": 320948224.0,
+ "1120": 320948224.0,
+ "1125": 320948224.0,
+ "1130": 320948224.0,
+ "1135": 320948224.0,
+ "1140": 320948224.0,
+ "1145": 320948224.0,
+ "1150": 320948224.0,
+ "1155": 320948224.0,
+ "1160": 320948224.0,
+ "1165": 320948224.0,
+ "1170": 320948224.0,
+ "1175": 322521088.0,
+ "1180": 322521088.0,
+ "1185": 322521088.0,
+ "1190": 322521088.0,
+ "1195": 322521088.0,
+ "1200": 322521088.0,
+ "1205": 322521088.0,
+ "1210": 322521088.0,
+ "1215": 322521088.0,
+ "1220": 322521088.0,
+ "1225": 322521088.0,
+ "1230": 322521088.0,
+ "1235": 320948224.0,
+ "1240": 322521088.0,
+ "1245": 320948224.0,
+ "1250": 322521088.0,
+ "1255": 322521088.0,
+ "1260": 320948224.0,
+ "1265": 320948224.0,
+ "1270": 320948224.0,
+ "1275": 320948224.0,
+ "1280": 322521088.0,
+ "1285": 320948224.0,
+ "1290": 320948224.0,
+ "1295": 320948224.0,
+ "1300": 322521088.0,
+ "1305": 322521088.0,
+ "1310": 322521088.0,
+ "1315": 322521088.0,
+ "1320": 322521088.0,
+ "1325": 322521088.0,
+ "1330": 322521088.0,
+ "1335": 322521088.0,
+ "1340": 322521088.0,
+ "1345": 322521088.0,
+ "1350": 322521088.0,
+ "1355": 322521088.0,
+ "1360": 322521088.0,
+ "1365": 320948224.0,
+ "1370": 320948224.0,
+ "1375": 320948224.0,
+ "1380": 320948224.0,
+ "1385": 320948224.0,
+ "1390": 320948224.0,
+ "1395": 320948224.0,
+ "1400": 320948224.0,
+ "1405": 320948224.0,
+ "1410": 322521088.0,
+ "1415": 322521088.0,
+ "1420": 320948224.0,
+ "1425": 322521088.0,
+ "1430": 322521088.0,
+ "1435": 320948224.0,
+ "1440": 322521088.0,
+ "1445": 322521088.0,
+ "1450": 322521088.0,
+ "1455": 322521088.0,
+ "1460": 322521088.0,
+ "1465": 320948224.0,
+ "1470": 320948224.0,
+ "1475": 320948224.0,
+ "1480": 322521088.0,
+ "1485": 322521088.0,
+ "1490": 322521088.0,
+ "1495": 322521088.0,
+ "1500": 322521088.0,
+ "1505": 320948224.0,
+ "1510": 320948224.0,
+ "1515": 320948224.0,
+ "1520": 320948224.0,
+ "1525": 320948224.0,
+ "1530": 320948224.0,
+ "1535": 320948224.0,
+ "1540": 320948224.0,
+ "1545": 322521088.0,
+ "1550": 320948224.0,
+ "1555": 320948224.0,
+ "1560": 320948224.0,
+ "1565": 320948224.0,
+ "1570": 320948224.0,
+ "1575": 322521088.0,
+ "1580": 322521088.0,
+ "1585": 320948224.0,
+ "1590": 320948224.0,
+ "1595": 320948224.0,
+ "1600": 322521088.0,
+ "1605": 320948224.0,
+ "1610": 320948224.0,
+ "1615": 320948224.0,
+ "1620": 320948224.0,
+ "1625": 320948224.0,
+ "1630": 320948224.0,
+ "1635": 320948224.0,
+ "1640": 320948224.0,
+ "1645": 320948224.0,
+ "1650": 320948224.0,
+ "1655": 320948224.0,
+ "1660": 322521088.0,
+ "1665": 322521088.0,
+ "1670": 322521088.0,
+ "1675": 322521088.0,
+ "1680": 322521088.0,
+ "1685": 320948224.0,
+ "1690": 322521088.0,
+ "1695": 322521088.0,
+ "1700": 322521088.0,
+ "1705": 322521088.0,
+ "1710": 322521088.0,
+ "1715": 322521088.0,
+ "1720": 322521088.0,
+ "1725": 322521088.0,
+ "1730": 322521088.0,
+ "1735": 322521088.0,
+ "1740": 322521088.0,
+ "1745": 322521088.0,
+ "1750": 322521088.0,
+ "1755": 322521088.0,
+ "1760": 322521088.0,
+ "1765": 322521088.0,
+ "1770": 322521088.0,
+ "1775": 320948224.0,
+ "1780": 322521088.0,
+ "1785": 322521088.0,
+ "1790": 320948224.0,
+ "1795": 320948224.0,
+ "1800": 320948224.0,
+ "1805": 320948224.0,
+ "1810": 320948224.0,
+ "1815": 320948224.0,
+ "1820": 322521088.0,
+ "1825": 322521088.0,
+ "1830": 322521088.0,
+ "1835": 322521088.0,
+ "1840": 320948224.0,
+ "1845": 320948224.0,
+ "1850": 322521088.0,
+ "1855": 322521088.0,
+ "1860": 320948224.0,
+ "1865": 320948224.0,
+ "1870": 320948224.0,
+ "1875": 320948224.0,
+ "1880": 320948224.0,
+ "1885": 320948224.0,
+ "1890": 322521088.0,
+ "1895": 320948224.0,
+ "1900": 320948224.0,
+ "1905": 320948224.0,
+ "1910": 322521088.0,
+ "1915": 320948224.0,
+ "1920": 322521088.0,
+ "1925": 322521088.0,
+ "1930": 320948224.0,
+ "1935": 320948224.0,
+ "1940": 322521088.0,
+ "1945": 320948224.0,
+ "1950": 320948224.0,
+ "1955": 320948224.0,
+ "1960": 322521088.0,
+ "1965": 322521088.0,
+ "1970": 322521088.0,
+ "1975": 322521088.0,
+ "1980": 322521088.0,
+ "1985": 322521088.0,
+ "1990": 322521088.0,
+ "1995": 322521088.0,
+ "2000": 322521088.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 5,
+ "values": {
+ "1": 420345856.0,
+ "5": 420346368.0,
+ "10": 420346368.0,
+ "15": 420346368.0,
+ "20": 501327360.0,
+ "25": 501327872.0,
+ "30": 501327872.0,
+ "35": 501327872.0,
+ "40": 501327872.0,
+ "45": 501327872.0,
+ "50": 501327872.0,
+ "55": 501327872.0,
+ "60": 501327872.0,
+ "65": 501327872.0,
+ "70": 501327872.0,
+ "75": 501327872.0,
+ "80": 501327872.0,
+ "85": 501327872.0,
+ "90": 501327872.0,
+ "95": 501327872.0,
+ "100": 501327872.0,
+ "105": 501327872.0,
+ "110": 501327872.0,
+ "115": 501327872.0,
+ "120": 501327872.0,
+ "125": 501327872.0,
+ "130": 501327872.0,
+ "135": 501327872.0,
+ "140": 501327872.0,
+ "145": 501327872.0,
+ "150": 501327872.0,
+ "155": 501327872.0,
+ "160": 501327872.0,
+ "165": 501327872.0,
+ "170": 501327872.0,
+ "175": 501327872.0,
+ "180": 501327872.0,
+ "185": 501327872.0,
+ "190": 501327872.0,
+ "195": 501327872.0,
+ "200": 501327872.0,
+ "205": 501327872.0,
+ "210": 501327872.0,
+ "215": 501327872.0,
+ "220": 501327872.0,
+ "225": 501327872.0,
+ "230": 501327872.0,
+ "235": 501327872.0,
+ "240": 501327872.0,
+ "245": 501327872.0,
+ "250": 501327872.0,
+ "255": 501327872.0,
+ "260": 501327872.0,
+ "265": 501327872.0,
+ "270": 501327872.0,
+ "275": 501327872.0,
+ "280": 501327872.0,
+ "285": 501327872.0,
+ "290": 501327872.0,
+ "295": 501327872.0,
+ "300": 501327872.0,
+ "305": 501327872.0,
+ "310": 501327872.0,
+ "315": 501327872.0,
+ "320": 501327872.0,
+ "325": 501327872.0,
+ "330": 501327872.0,
+ "335": 501327872.0,
+ "340": 501327872.0,
+ "345": 501327872.0,
+ "350": 501327872.0,
+ "355": 501327872.0,
+ "360": 501327872.0,
+ "365": 501327872.0,
+ "370": 501327872.0,
+ "375": 501327872.0,
+ "380": 501327872.0,
+ "385": 501327872.0,
+ "390": 501327872.0,
+ "395": 501327872.0,
+ "400": 501327872.0,
+ "405": 501327872.0,
+ "410": 501327872.0,
+ "415": 501327872.0,
+ "420": 501327872.0,
+ "425": 501327872.0,
+ "430": 501327872.0,
+ "435": 501327872.0,
+ "440": 501327872.0,
+ "445": 501327872.0,
+ "450": 501327872.0,
+ "455": 501327872.0,
+ "460": 501327872.0,
+ "465": 501327872.0,
+ "470": 501327872.0,
+ "475": 501327872.0,
+ "480": 501327872.0,
+ "485": 501327872.0,
+ "490": 501327872.0,
+ "495": 501327872.0,
+ "500": 501327872.0,
+ "505": 501327872.0,
+ "510": 501327872.0,
+ "515": 501327872.0,
+ "520": 501327872.0,
+ "525": 501327872.0,
+ "530": 501327872.0,
+ "535": 501327872.0,
+ "540": 501327872.0,
+ "545": 501327872.0,
+ "550": 501327872.0,
+ "555": 501327872.0,
+ "560": 501327872.0,
+ "565": 501327872.0,
+ "570": 501327872.0,
+ "575": 501327872.0,
+ "580": 501327872.0,
+ "585": 501327872.0,
+ "590": 501327872.0,
+ "595": 501327872.0,
+ "600": 501327872.0,
+ "605": 501327872.0,
+ "610": 501327872.0,
+ "615": 501327872.0,
+ "620": 501327872.0,
+ "625": 501327872.0,
+ "630": 501327872.0,
+ "635": 501327872.0,
+ "640": 501327872.0,
+ "645": 501327872.0,
+ "650": 501327872.0,
+ "655": 501327872.0,
+ "660": 501327872.0,
+ "665": 501327872.0,
+ "670": 501327872.0,
+ "675": 501327872.0,
+ "680": 501327872.0,
+ "685": 501327872.0,
+ "690": 501327872.0,
+ "695": 501327872.0,
+ "700": 501327872.0,
+ "705": 501327872.0,
+ "710": 501327872.0,
+ "715": 501327872.0,
+ "720": 501327872.0,
+ "725": 501327872.0,
+ "730": 501327872.0,
+ "735": 501327872.0,
+ "740": 501327872.0,
+ "745": 501327872.0,
+ "750": 501327872.0,
+ "755": 501327872.0,
+ "760": 501327872.0,
+ "765": 501327872.0,
+ "770": 501327872.0,
+ "775": 501327872.0,
+ "780": 501327872.0,
+ "785": 501327872.0,
+ "790": 501327872.0,
+ "795": 501327872.0,
+ "800": 501327872.0,
+ "805": 501327872.0,
+ "810": 501327872.0,
+ "815": 501327872.0,
+ "820": 501327872.0,
+ "825": 501327872.0,
+ "830": 501327872.0,
+ "835": 501327872.0,
+ "840": 501327872.0,
+ "845": 501327872.0,
+ "850": 501327872.0,
+ "855": 501327872.0,
+ "860": 501327872.0,
+ "865": 501327872.0,
+ "870": 501327872.0,
+ "875": 501327872.0,
+ "880": 501327872.0,
+ "885": 501327872.0,
+ "890": 501327872.0,
+ "895": 501327872.0,
+ "900": 501327872.0,
+ "905": 501327872.0,
+ "910": 501327872.0,
+ "915": 501327872.0,
+ "920": 501327872.0,
+ "925": 501327872.0,
+ "930": 501327872.0,
+ "935": 501327872.0,
+ "940": 501327872.0,
+ "945": 501327872.0,
+ "950": 501327872.0,
+ "955": 501327872.0,
+ "960": 501327872.0,
+ "965": 501327872.0,
+ "970": 501327872.0,
+ "975": 501327872.0,
+ "980": 501327872.0,
+ "985": 501327872.0,
+ "990": 501327872.0,
+ "995": 501327872.0,
+ "1000": 501327872.0,
+ "1005": 501327872.0,
+ "1010": 501327872.0,
+ "1015": 501327872.0,
+ "1020": 501327872.0,
+ "1025": 501327872.0,
+ "1030": 501327872.0,
+ "1035": 501327872.0,
+ "1040": 501327872.0,
+ "1045": 501327872.0,
+ "1050": 501327872.0,
+ "1055": 501327872.0,
+ "1060": 501327872.0,
+ "1065": 501327872.0,
+ "1070": 501327872.0,
+ "1075": 501327872.0,
+ "1080": 501327872.0,
+ "1085": 501327872.0,
+ "1090": 501327872.0,
+ "1095": 501327872.0,
+ "1100": 501327872.0,
+ "1105": 501327872.0,
+ "1110": 501327872.0,
+ "1115": 501327872.0,
+ "1120": 501327872.0,
+ "1125": 501327872.0,
+ "1130": 501327872.0,
+ "1135": 501327872.0,
+ "1140": 501327872.0,
+ "1145": 501327872.0,
+ "1150": 501327872.0,
+ "1155": 501327872.0,
+ "1160": 501327872.0,
+ "1165": 501327872.0,
+ "1170": 501327872.0,
+ "1175": 501327872.0,
+ "1180": 501327872.0,
+ "1185": 501327872.0,
+ "1190": 501327872.0,
+ "1195": 501327872.0,
+ "1200": 501327872.0,
+ "1205": 501327872.0,
+ "1210": 501327872.0,
+ "1215": 501327872.0,
+ "1220": 501327872.0,
+ "1225": 501327872.0,
+ "1230": 501327872.0,
+ "1235": 501327872.0,
+ "1240": 501327872.0,
+ "1245": 501327872.0,
+ "1250": 501327872.0,
+ "1255": 501327872.0,
+ "1260": 501327872.0,
+ "1265": 501327872.0,
+ "1270": 501327872.0,
+ "1275": 501327872.0,
+ "1280": 501327872.0,
+ "1285": 501327872.0,
+ "1290": 501327872.0,
+ "1295": 501327872.0,
+ "1300": 501327872.0,
+ "1305": 501327872.0,
+ "1310": 501327872.0,
+ "1315": 501327872.0,
+ "1320": 501327872.0,
+ "1325": 501327872.0,
+ "1330": 501327872.0,
+ "1335": 501327872.0,
+ "1340": 501327872.0,
+ "1345": 501327872.0,
+ "1350": 501327872.0,
+ "1355": 501327872.0,
+ "1360": 501327872.0,
+ "1365": 501327872.0,
+ "1370": 501327872.0,
+ "1375": 501327872.0,
+ "1380": 501327872.0,
+ "1385": 501327872.0,
+ "1390": 501327872.0,
+ "1395": 501327872.0,
+ "1400": 501327872.0,
+ "1405": 501327872.0,
+ "1410": 501327872.0,
+ "1415": 501327872.0,
+ "1420": 501327872.0,
+ "1425": 501327872.0,
+ "1430": 501327872.0,
+ "1435": 501327872.0,
+ "1440": 501327872.0,
+ "1445": 501327872.0,
+ "1450": 501327872.0,
+ "1455": 501327872.0,
+ "1460": 501327872.0,
+ "1465": 501327872.0,
+ "1470": 501327872.0,
+ "1475": 501327872.0,
+ "1480": 501327872.0,
+ "1485": 501327872.0,
+ "1490": 501327872.0,
+ "1495": 501327872.0,
+ "1500": 501327872.0,
+ "1505": 501327872.0,
+ "1510": 501327872.0,
+ "1515": 501327872.0,
+ "1520": 501327872.0,
+ "1525": 501327872.0,
+ "1530": 501327872.0,
+ "1535": 501327872.0,
+ "1540": 501327872.0,
+ "1545": 501327872.0,
+ "1550": 501327872.0,
+ "1555": 501327872.0,
+ "1560": 501327872.0,
+ "1565": 501327872.0,
+ "1570": 501327872.0,
+ "1575": 501327872.0,
+ "1580": 501327872.0,
+ "1585": 501327872.0,
+ "1590": 501327872.0,
+ "1595": 501327872.0,
+ "1600": 501327872.0,
+ "1605": 501327872.0,
+ "1610": 501327872.0,
+ "1615": 501327872.0,
+ "1620": 501327872.0,
+ "1625": 501327872.0,
+ "1630": 501327872.0,
+ "1635": 501327872.0,
+ "1640": 501327872.0,
+ "1645": 501327872.0,
+ "1650": 501327872.0,
+ "1655": 501327872.0,
+ "1660": 501327872.0,
+ "1665": 501327872.0,
+ "1670": 501327872.0,
+ "1675": 501327872.0,
+ "1680": 501327872.0,
+ "1685": 501327872.0,
+ "1690": 501327872.0,
+ "1695": 501327872.0,
+ "1700": 501327872.0,
+ "1705": 501327872.0,
+ "1710": 501327872.0,
+ "1715": 501327872.0,
+ "1720": 501327872.0,
+ "1725": 501327872.0,
+ "1730": 501327872.0,
+ "1735": 501327872.0,
+ "1740": 501327872.0,
+ "1745": 501327872.0,
+ "1750": 501327872.0,
+ "1755": 501327872.0,
+ "1760": 501327872.0,
+ "1765": 501327872.0,
+ "1770": 501327872.0,
+ "1775": 501327872.0,
+ "1780": 501327872.0,
+ "1785": 501327872.0,
+ "1790": 501327872.0,
+ "1795": 501327872.0,
+ "1800": 501327872.0,
+ "1805": 501327872.0,
+ "1810": 501327872.0,
+ "1815": 501327872.0,
+ "1820": 501327872.0,
+ "1825": 501327872.0,
+ "1830": 501327872.0,
+ "1835": 501327872.0,
+ "1840": 501327872.0,
+ "1845": 501327872.0,
+ "1850": 501327872.0,
+ "1855": 501327872.0,
+ "1860": 501327872.0,
+ "1865": 501327872.0,
+ "1870": 501327872.0,
+ "1875": 501327872.0,
+ "1880": 501327872.0,
+ "1885": 501327872.0,
+ "1890": 501327872.0,
+ "1895": 501327872.0,
+ "1900": 501327872.0,
+ "1905": 501327872.0,
+ "1910": 501327872.0,
+ "1915": 501327872.0,
+ "1920": 501327872.0,
+ "1925": 501327872.0,
+ "1930": 501327872.0,
+ "1935": 501327872.0,
+ "1940": 501327872.0,
+ "1945": 501327872.0,
+ "1950": 501327872.0,
+ "1955": 501327872.0,
+ "1960": 501327872.0,
+ "1965": 501327872.0,
+ "1970": 501327872.0,
+ "1975": 501327872.0,
+ "1980": 501327872.0,
+ "1985": 501327872.0,
+ "1990": 501327872.0,
+ "1995": 501327872.0,
+ "2000": 501327872.0
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 5,
+ "values": {
+ "1": "nan",
+ "5": "nan",
+ "10": "nan",
+ "15": "nan",
+ "20": 12399.0,
+ "25": 12768.0,
+ "30": 16641.0,
+ "35": 12861.0,
+ "40": 13048.0,
+ "45": 13826.0,
+ "50": 10792.0,
+ "55": 12500.0,
+ "60": 13751.0,
+ "65": 15066.0,
+ "70": 10815.0,
+ "75": 11843.0,
+ "80": 10739.0,
+ "85": 17210.0,
+ "90": 12167.0,
+ "95": 12456.0,
+ "100": 11589.0,
+ "105": 13445.0,
+ "110": 14807.0,
+ "115": 11882.0,
+ "120": 12425.0,
+ "125": 11696.0,
+ "130": 12874.0,
+ "135": 11719.0,
+ "140": 12660.0,
+ "145": 17034.0,
+ "150": 12668.0,
+ "155": 14247.0,
+ "160": 10069.0,
+ "165": 16144.0,
+ "170": 9903.0,
+ "175": 14183.0,
+ "180": 12261.0,
+ "185": 12523.0,
+ "190": 10611.0,
+ "195": 15090.0,
+ "200": 13729.0,
+ "205": 13604.0,
+ "210": 12246.0,
+ "215": 11912.0,
+ "220": 16876.0,
+ "225": 14473.0,
+ "230": 14743.0,
+ "235": 14900.0,
+ "240": 14588.0,
+ "245": 13489.0,
+ "250": 15237.0,
+ "255": 15391.0,
+ "260": 13542.0,
+ "265": 17355.0,
+ "270": 14088.0,
+ "275": 13259.0,
+ "280": 14138.0,
+ "285": 13254.0,
+ "290": 14182.0,
+ "295": 15382.0,
+ "300": 13137.0,
+ "305": 12597.0,
+ "310": 16148.0,
+ "315": 16526.0,
+ "320": 15265.0,
+ "325": 15245.0,
+ "330": 14420.0,
+ "335": 12278.0,
+ "340": 15325.0,
+ "345": 14203.0,
+ "350": 15636.0,
+ "355": 13620.0,
+ "360": 12816.0,
+ "365": 15947.0,
+ "370": 16936.0,
+ "375": 13675.0,
+ "380": 17271.0,
+ "385": 14395.0,
+ "390": 13666.0,
+ "395": 14357.0,
+ "400": 18807.0,
+ "405": 18024.0,
+ "410": 15076.0,
+ "415": 18044.0,
+ "420": 13713.0,
+ "425": 19144.0,
+ "430": 13480.0,
+ "435": 13479.0,
+ "440": 16351.0,
+ "445": 16067.0,
+ "450": 15695.0,
+ "455": 16439.0,
+ "460": 15762.0,
+ "465": 16117.0,
+ "470": 14570.0,
+ "475": 16073.0,
+ "480": 16836.0,
+ "485": 15823.0,
+ "490": 17428.0,
+ "495": 15714.0,
+ "500": 15473.0,
+ "505": 14010.0,
+ "510": 17357.0,
+ "515": 16311.0,
+ "520": 15036.0,
+ "525": 13085.0,
+ "530": 12049.0,
+ "535": 15795.0,
+ "540": 15218.0,
+ "545": 13095.0,
+ "550": 14667.0,
+ "555": 16625.0,
+ "560": 14978.0,
+ "565": 16884.0,
+ "570": 17396.0,
+ "575": 15771.0,
+ "580": 14244.0,
+ "585": 17594.0,
+ "590": 14149.0,
+ "595": 16441.0,
+ "600": 16689.0,
+ "605": 14090.0,
+ "610": 13665.0,
+ "615": 15655.0,
+ "620": 15122.0,
+ "625": 14692.0,
+ "630": 18197.0,
+ "635": 16871.0,
+ "640": 16280.0,
+ "645": 15121.0,
+ "650": 17339.0,
+ "655": 15328.0,
+ "660": 15865.0,
+ "665": 11765.0,
+ "670": 14995.0,
+ "675": 17923.0,
+ "680": 14080.0,
+ "685": 14368.0,
+ "690": 13369.0,
+ "695": 15366.0,
+ "700": 13216.0,
+ "705": 15326.0,
+ "710": 15574.0,
+ "715": 15741.0,
+ "720": 16788.0,
+ "725": 18627.0,
+ "730": 15154.0,
+ "735": 15522.0,
+ "740": 16941.0,
+ "745": 15349.0,
+ "750": 16371.0,
+ "755": 14916.0,
+ "760": 16842.0,
+ "765": 15406.0,
+ "770": 16940.0,
+ "775": 15582.0,
+ "780": 16231.0,
+ "785": 15112.0,
+ "790": 17318.0,
+ "795": 14480.0,
+ "800": 14988.0,
+ "805": 14551.0,
+ "810": 14601.0,
+ "815": 17625.0,
+ "820": 16695.0,
+ "825": 12130.0,
+ "830": 16119.0,
+ "835": 15071.0,
+ "840": 18768.0,
+ "845": 14302.0,
+ "850": 15455.0,
+ "855": 15110.0,
+ "860": 15268.0,
+ "865": 15904.0,
+ "870": 15028.0,
+ "875": 16093.0,
+ "880": 12820.0,
+ "885": 14034.0,
+ "890": 15560.0,
+ "895": 16321.0,
+ "900": 15868.0,
+ "905": 15995.0,
+ "910": 15372.0,
+ "915": 15635.0,
+ "920": 15878.0,
+ "925": 17142.0,
+ "930": 16179.0,
+ "935": 14544.0,
+ "940": 16970.0,
+ "945": 15322.0,
+ "950": 16045.0,
+ "955": 15838.0,
+ "960": 19384.0,
+ "965": 14166.0,
+ "970": 13564.0,
+ "975": 13604.0,
+ "980": 15645.0,
+ "985": 16705.0,
+ "990": 16844.0,
+ "995": 15594.0,
+ "1000": 18440.0,
+ "1005": 13331.0,
+ "1010": 15473.0,
+ "1015": 17161.0,
+ "1020": 15060.0,
+ "1025": 16612.0,
+ "1030": 14790.0,
+ "1035": 14648.0,
+ "1040": 15360.0,
+ "1045": 19850.0,
+ "1050": 14983.0,
+ "1055": 18180.0,
+ "1060": 15044.0,
+ "1065": 15459.0,
+ "1070": 16779.0,
+ "1075": 14457.0,
+ "1080": 14730.0,
+ "1085": 14956.0,
+ "1090": 18878.0,
+ "1095": 17996.0,
+ "1100": 15911.0,
+ "1105": 18119.0,
+ "1110": 14685.0,
+ "1115": 18203.0,
+ "1120": 14214.0,
+ "1125": 16461.0,
+ "1130": 17344.0,
+ "1135": 16215.0,
+ "1140": 17530.0,
+ "1145": 17567.0,
+ "1150": 14462.0,
+ "1155": 15942.0,
+ "1160": 14909.0,
+ "1165": 18851.0,
+ "1170": 22578.0,
+ "1175": 14920.0,
+ "1180": 16032.0,
+ "1185": 18810.0,
+ "1190": 16783.0,
+ "1195": 14758.0,
+ "1200": 18966.0,
+ "1205": 12641.0,
+ "1210": 14496.0,
+ "1215": 16522.0,
+ "1220": 14614.0,
+ "1225": 14367.0,
+ "1230": 17478.0,
+ "1235": 15641.0,
+ "1240": 15478.0,
+ "1245": 18552.0,
+ "1250": 16757.0,
+ "1255": 15297.0,
+ "1260": 15694.0,
+ "1265": 14704.0,
+ "1270": 14225.0,
+ "1275": 14655.0,
+ "1280": 14686.0,
+ "1285": 17248.0,
+ "1290": 16072.0,
+ "1295": 18917.0,
+ "1300": 17183.0,
+ "1305": 17118.0,
+ "1310": 15283.0,
+ "1315": 17086.0,
+ "1320": 16520.0,
+ "1325": 18610.0,
+ "1330": 14080.0,
+ "1335": 14118.0,
+ "1340": 17189.0,
+ "1345": 16845.0,
+ "1350": 17053.0,
+ "1355": 14369.0,
+ "1360": 16477.0,
+ "1365": 17514.0,
+ "1370": 15038.0,
+ "1375": 17672.0,
+ "1380": 18278.0,
+ "1385": 16726.0,
+ "1390": 22921.0,
+ "1395": 15140.0,
+ "1400": 15183.0,
+ "1405": 14967.0,
+ "1410": 16606.0,
+ "1415": 15109.0,
+ "1420": 19431.0,
+ "1425": 17505.0,
+ "1430": 18660.0,
+ "1435": 20507.0,
+ "1440": 15585.0,
+ "1445": 18728.0,
+ "1450": 14608.0,
+ "1455": 16242.0,
+ "1460": 15575.0,
+ "1465": 15658.0,
+ "1470": 17025.0,
+ "1475": 16266.0,
+ "1480": 15608.0,
+ "1485": 17129.0,
+ "1490": 15290.0,
+ "1495": 16548.0,
+ "1500": 15479.0,
+ "1505": 15923.0,
+ "1510": 19409.0,
+ "1515": 13514.0,
+ "1520": 16060.0,
+ "1525": 15836.0,
+ "1530": 18468.0,
+ "1535": 13807.0,
+ "1540": 14881.0,
+ "1545": 15470.0,
+ "1550": 12872.0,
+ "1555": 17358.0,
+ "1560": 16032.0,
+ "1565": 23723.0,
+ "1570": 19253.0,
+ "1575": 14498.0,
+ "1580": 16635.0,
+ "1585": 16477.0,
+ "1590": 14446.0,
+ "1595": 13727.0,
+ "1600": 19013.0,
+ "1605": 16523.0,
+ "1610": 15293.0,
+ "1615": 19549.0,
+ "1620": 17366.0,
+ "1625": 14557.0,
+ "1630": 16260.0,
+ "1635": 17512.0,
+ "1640": 15433.0,
+ "1645": 19414.0,
+ "1650": 17363.0,
+ "1655": 15694.0,
+ "1660": 16322.0,
+ "1665": 19927.0,
+ "1670": 15723.0,
+ "1675": 17732.0,
+ "1680": 16517.0,
+ "1685": 16166.0,
+ "1690": 14234.0,
+ "1695": 15753.0,
+ "1700": 15548.0,
+ "1705": 15450.0,
+ "1710": 20463.0,
+ "1715": 15973.0,
+ "1720": 17542.0,
+ "1725": 18316.0,
+ "1730": 15131.0,
+ "1735": 15759.0,
+ "1740": 16241.0,
+ "1745": 15234.0,
+ "1750": 14370.0,
+ "1755": 16531.0,
+ "1760": 16839.0,
+ "1765": 17326.0,
+ "1770": 15857.0,
+ "1775": 21195.0,
+ "1780": 14469.0,
+ "1785": 16678.0,
+ "1790": 16776.0,
+ "1795": 17759.0,
+ "1800": 15019.0,
+ "1805": 14821.0,
+ "1810": 18102.0,
+ "1815": 20126.0,
+ "1820": 14974.0,
+ "1825": 14647.0,
+ "1830": 15838.0,
+ "1835": 14329.0,
+ "1840": 14177.0,
+ "1845": 18461.0,
+ "1850": 15969.0,
+ "1855": 16030.0,
+ "1860": 17732.0,
+ "1865": 20004.0,
+ "1870": 14896.0,
+ "1875": 16398.0,
+ "1880": 17982.0,
+ "1885": 15943.0,
+ "1890": 17442.0,
+ "1895": 15338.0,
+ "1900": 12685.0,
+ "1905": 14507.0,
+ "1910": 17727.0,
+ "1915": 16421.0,
+ "1920": 15923.0,
+ "1925": 15230.0,
+ "1930": 15692.0,
+ "1935": 18265.0,
+ "1940": 16992.0,
+ "1945": 16655.0,
+ "1950": 16376.0,
+ "1955": 17354.0,
+ "1960": 19155.0,
+ "1965": 15851.0,
+ "1970": 16382.0,
+ "1975": 17019.0,
+ "1980": 17139.0,
+ "1985": 17579.0,
+ "1990": 17860.0,
+ "1995": 16695.0,
+ "2000": 15874.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 5,
+ "values": {
+ "1": "nan",
+ "5": "nan",
+ "10": "nan",
+ "15": "nan",
+ "20": "nan",
+ "25": "nan",
+ "30": "nan",
+ "35": "nan",
+ "40": "nan",
+ "45": "nan",
+ "50": "nan",
+ "55": "nan",
+ "60": "nan",
+ "65": "nan",
+ "70": "nan",
+ "75": "nan",
+ "80": "nan",
+ "85": "nan",
+ "90": "nan",
+ "95": "nan",
+ "100": 0.58989,
+ "105": "nan",
+ "110": "nan",
+ "115": "nan",
+ "120": "nan",
+ "125": "nan",
+ "130": "nan",
+ "135": "nan",
+ "140": "nan",
+ "145": "nan",
+ "150": "nan",
+ "155": "nan",
+ "160": "nan",
+ "165": "nan",
+ "170": "nan",
+ "175": "nan",
+ "180": "nan",
+ "185": "nan",
+ "190": "nan",
+ "195": "nan",
+ "200": 0.26173,
+ "205": "nan",
+ "210": "nan",
+ "215": "nan",
+ "220": "nan",
+ "225": "nan",
+ "230": "nan",
+ "235": "nan",
+ "240": "nan",
+ "245": "nan",
+ "250": "nan",
+ "255": "nan",
+ "260": "nan",
+ "265": "nan",
+ "270": "nan",
+ "275": "nan",
+ "280": "nan",
+ "285": "nan",
+ "290": "nan",
+ "295": "nan",
+ "300": 0.26925,
+ "305": "nan",
+ "310": "nan",
+ "315": "nan",
+ "320": "nan",
+ "325": "nan",
+ "330": "nan",
+ "335": "nan",
+ "340": "nan",
+ "345": "nan",
+ "350": "nan",
+ "355": "nan",
+ "360": "nan",
+ "365": "nan",
+ "370": "nan",
+ "375": "nan",
+ "380": "nan",
+ "385": "nan",
+ "390": "nan",
+ "395": "nan",
+ "400": 0.2606,
+ "405": "nan",
+ "410": "nan",
+ "415": "nan",
+ "420": "nan",
+ "425": "nan",
+ "430": "nan",
+ "435": "nan",
+ "440": "nan",
+ "445": "nan",
+ "450": "nan",
+ "455": "nan",
+ "460": "nan",
+ "465": "nan",
+ "470": "nan",
+ "475": "nan",
+ "480": "nan",
+ "485": "nan",
+ "490": "nan",
+ "495": "nan",
+ "500": 0.25904,
+ "505": "nan",
+ "510": "nan",
+ "515": "nan",
+ "520": "nan",
+ "525": "nan",
+ "530": "nan",
+ "535": "nan",
+ "540": "nan",
+ "545": "nan",
+ "550": "nan",
+ "555": "nan",
+ "560": "nan",
+ "565": "nan",
+ "570": "nan",
+ "575": "nan",
+ "580": "nan",
+ "585": "nan",
+ "590": "nan",
+ "595": "nan",
+ "600": 0.27358,
+ "605": "nan",
+ "610": "nan",
+ "615": "nan",
+ "620": "nan",
+ "625": "nan",
+ "630": "nan",
+ "635": "nan",
+ "640": "nan",
+ "645": "nan",
+ "650": "nan",
+ "655": "nan",
+ "660": "nan",
+ "665": "nan",
+ "670": "nan",
+ "675": "nan",
+ "680": "nan",
+ "685": "nan",
+ "690": "nan",
+ "695": "nan",
+ "700": 0.30298,
+ "705": "nan",
+ "710": "nan",
+ "715": "nan",
+ "720": "nan",
+ "725": "nan",
+ "730": "nan",
+ "735": "nan",
+ "740": "nan",
+ "745": "nan",
+ "750": "nan",
+ "755": "nan",
+ "760": "nan",
+ "765": "nan",
+ "770": "nan",
+ "775": "nan",
+ "780": "nan",
+ "785": "nan",
+ "790": "nan",
+ "795": "nan",
+ "800": 0.29186,
+ "805": "nan",
+ "810": "nan",
+ "815": "nan",
+ "820": "nan",
+ "825": "nan",
+ "830": "nan",
+ "835": "nan",
+ "840": "nan",
+ "845": "nan",
+ "850": "nan",
+ "855": "nan",
+ "860": "nan",
+ "865": "nan",
+ "870": "nan",
+ "875": "nan",
+ "880": "nan",
+ "885": "nan",
+ "890": "nan",
+ "895": "nan",
+ "900": 0.29862,
+ "905": "nan",
+ "910": "nan",
+ "915": "nan",
+ "920": "nan",
+ "925": "nan",
+ "930": "nan",
+ "935": "nan",
+ "940": "nan",
+ "945": "nan",
+ "950": "nan",
+ "955": "nan",
+ "960": "nan",
+ "965": "nan",
+ "970": "nan",
+ "975": "nan",
+ "980": "nan",
+ "985": "nan",
+ "990": "nan",
+ "995": "nan",
+ "1000": 0.27405,
+ "1005": "nan",
+ "1010": "nan",
+ "1015": "nan",
+ "1020": "nan",
+ "1025": "nan",
+ "1030": "nan",
+ "1035": "nan",
+ "1040": "nan",
+ "1045": "nan",
+ "1050": "nan",
+ "1055": "nan",
+ "1060": "nan",
+ "1065": "nan",
+ "1070": "nan",
+ "1075": "nan",
+ "1080": "nan",
+ "1085": "nan",
+ "1090": "nan",
+ "1095": "nan",
+ "1100": 0.27684,
+ "1105": "nan",
+ "1110": "nan",
+ "1115": "nan",
+ "1120": "nan",
+ "1125": "nan",
+ "1130": "nan",
+ "1135": "nan",
+ "1140": "nan",
+ "1145": "nan",
+ "1150": "nan",
+ "1155": "nan",
+ "1160": "nan",
+ "1165": "nan",
+ "1170": "nan",
+ "1175": "nan",
+ "1180": "nan",
+ "1185": "nan",
+ "1190": "nan",
+ "1195": "nan",
+ "1200": 0.27559,
+ "1205": "nan",
+ "1210": "nan",
+ "1215": "nan",
+ "1220": "nan",
+ "1225": "nan",
+ "1230": "nan",
+ "1235": "nan",
+ "1240": "nan",
+ "1245": "nan",
+ "1250": "nan",
+ "1255": "nan",
+ "1260": "nan",
+ "1265": "nan",
+ "1270": "nan",
+ "1275": "nan",
+ "1280": "nan",
+ "1285": "nan",
+ "1290": "nan",
+ "1295": "nan",
+ "1300": 0.27282,
+ "1305": "nan",
+ "1310": "nan",
+ "1315": "nan",
+ "1320": "nan",
+ "1325": "nan",
+ "1330": "nan",
+ "1335": "nan",
+ "1340": "nan",
+ "1345": "nan",
+ "1350": "nan",
+ "1355": "nan",
+ "1360": "nan",
+ "1365": "nan",
+ "1370": "nan",
+ "1375": "nan",
+ "1380": "nan",
+ "1385": "nan",
+ "1390": "nan",
+ "1395": "nan",
+ "1400": 0.25948,
+ "1405": "nan",
+ "1410": "nan",
+ "1415": "nan",
+ "1420": "nan",
+ "1425": "nan",
+ "1430": "nan",
+ "1435": "nan",
+ "1440": "nan",
+ "1445": "nan",
+ "1450": "nan",
+ "1455": "nan",
+ "1460": "nan",
+ "1465": "nan",
+ "1470": "nan",
+ "1475": "nan",
+ "1480": "nan",
+ "1485": "nan",
+ "1490": "nan",
+ "1495": "nan",
+ "1500": 0.27252,
+ "1505": "nan",
+ "1510": "nan",
+ "1515": "nan",
+ "1520": "nan",
+ "1525": "nan",
+ "1530": "nan",
+ "1535": "nan",
+ "1540": "nan",
+ "1545": "nan",
+ "1550": "nan",
+ "1555": "nan",
+ "1560": "nan",
+ "1565": "nan",
+ "1570": "nan",
+ "1575": "nan",
+ "1580": "nan",
+ "1585": "nan",
+ "1590": "nan",
+ "1595": "nan",
+ "1600": 0.27844,
+ "1605": "nan",
+ "1610": "nan",
+ "1615": "nan",
+ "1620": "nan",
+ "1625": "nan",
+ "1630": "nan",
+ "1635": "nan",
+ "1640": "nan",
+ "1645": "nan",
+ "1650": "nan",
+ "1655": "nan",
+ "1660": "nan",
+ "1665": "nan",
+ "1670": "nan",
+ "1675": "nan",
+ "1680": "nan",
+ "1685": "nan",
+ "1690": "nan",
+ "1695": "nan",
+ "1700": 0.27607,
+ "1705": "nan",
+ "1710": "nan",
+ "1715": "nan",
+ "1720": "nan",
+ "1725": "nan",
+ "1730": "nan",
+ "1735": "nan",
+ "1740": "nan",
+ "1745": "nan",
+ "1750": "nan",
+ "1755": "nan",
+ "1760": "nan",
+ "1765": "nan",
+ "1770": "nan",
+ "1775": "nan",
+ "1780": "nan",
+ "1785": "nan",
+ "1790": "nan",
+ "1795": "nan",
+ "1800": 0.27527,
+ "1805": "nan",
+ "1810": "nan",
+ "1815": "nan",
+ "1820": "nan",
+ "1825": "nan",
+ "1830": "nan",
+ "1835": "nan",
+ "1840": "nan",
+ "1845": "nan",
+ "1850": "nan",
+ "1855": "nan",
+ "1860": "nan",
+ "1865": "nan",
+ "1870": "nan",
+ "1875": "nan",
+ "1880": "nan",
+ "1885": "nan",
+ "1890": "nan",
+ "1895": "nan",
+ "1900": 0.27903,
+ "1905": "nan",
+ "1910": "nan",
+ "1915": "nan",
+ "1920": "nan",
+ "1925": "nan",
+ "1930": "nan",
+ "1935": "nan",
+ "1940": "nan",
+ "1945": "nan",
+ "1950": "nan",
+ "1955": "nan",
+ "1960": "nan",
+ "1965": "nan",
+ "1970": "nan",
+ "1975": "nan",
+ "1980": "nan",
+ "1985": "nan",
+ "1990": "nan",
+ "1995": "nan",
+ "2000": 0.31051
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/bert/bert_release_sm/model_config.yaml b/tests/functional_tests/test_cases/bert/bert_release_sm/model_config.yaml
new file mode 100644
index 00000000000..1d982960246
--- /dev/null
+++ b/tests/functional_tests/test_cases/bert/bert_release_sm/model_config.yaml
@@ -0,0 +1,57 @@
+ENV_VARS:
+ CUDA_DEVICE_MAX_CONNECTIONS: "1"
+ NVTE_ALLOW_NONDETERMINISTIC_ALGO: "1"
+ NON_DETERMINSTIC_RESULTS: 1
+TEST_TYPE: "release"
+MODEL_ARGS:
+ # Bert model args
+ --num-layers: 24
+ --hidden-size: 1024
+ --num-attention-heads: 16
+ --seq-length: 512
+ --max-position-embeddings: 512
+ # Training args
+ --micro-batch-size: 4
+ --global-batch-size: 32
+ --train-iters: 20000
+ --weight-decay: 1e-2
+ --clip-grad: 1.0
+ --fp16: true
+ --lr: 0.0001
+ --lr-decay-style: linear
+ --min-lr: 1.0e-5
+ --lr-warmup-fraction: .01
+ --bert-no-binary-head: true
+ # Model parallel
+ --tensor-model-parallel-size: 8
+ --pipeline-model-parallel-size: 8
+ # Data args
+ --data-path: ${DATA_BLEND}
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/vocab.txt
+ --split: 949,50,1
+ --data-cache-path: ${DATA_CACHE_PATH}
+ # EVAL_AND_LOGGING_ARGS
+ --log-interval: 100
+ --save-interval: 2000
+ --save-retain-interval: 10000
+ --eval-interval: 1000
+ --save: ${CHECKPOINT_SAVE_PATH}
+ --load: ${CHECKPOINT_LOAD_PATH}
+ --eval-iters: 10
+ --tensorboard-dir: ${TENSORBOARD_PATH}
+ --log-timers-to-tensorboard: true
+ --log-memory-to-tensorboard: true
+ --log-num-zeros-in-grad: true
+ --log-params-norm: true
+ --log-validation-ppl-to-tensorboard: true
+ --wandb-project: megatron-core-release-runs
+ --wandb-entity: adlr
+ --wandb-exp-name: ${WANDB_EXPERIMENT}
+ --attention-backend: unfused
+ --exit-interval: 2000
+ --wandb-save-dir: ${WANDB_SAVE_PATH}
+METRICS:
+ - "iteration-time"
+ - "lm loss"
+ - "mem-allocated-bytes"
+ - "mem-max-allocated-bytes"
diff --git a/tests/functional_tests/test_cases/common/ckpt_converter/__main__.py b/tests/functional_tests/test_cases/common/ckpt_converter/__main__.py
index 62d86a29358..543ddd36a6d 100644
--- a/tests/functional_tests/test_cases/common/ckpt_converter/__main__.py
+++ b/tests/functional_tests/test_cases/common/ckpt_converter/__main__.py
@@ -8,7 +8,6 @@
import types
import typing as T
from collections import namedtuple
-from copy import deepcopy
from functools import partial
import numpy as np
@@ -19,14 +18,10 @@
from megatron.core import parallel_state
from megatron.core.datasets.gpt_dataset import _get_ltor_masks_and_position_ids
from megatron.core.enums import ModelType
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
-from megatron.core.models.multimodal.llava_model import DEFAULT_IMAGE_TOKEN_INDEX, LLaVAModel
-from megatron.core.models.vision.vit_layer_specs import get_vit_layer_with_transformer_engine_spec
+from megatron.core.models.multimodal.llava_model import DEFAULT_IMAGE_TOKEN_INDEX
from megatron.core.pipeline_parallel import get_forward_backward_func
from megatron.core.tensor_parallel.mappings import gather_from_tensor_model_parallel_region
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
-from megatron.core.transformer.transformer_config import TransformerConfig
-from megatron.core.utils import get_attr_wrapped_model
from megatron.training import get_args, get_tokenizer
from megatron.training.arguments import parse_args, validate_args
from megatron.training.checkpointing import load_checkpoint as _load_checkpoint
diff --git a/tests/functional_tests/test_cases/common/moe_perf/__main__.py b/tests/functional_tests/test_cases/common/moe_perf/__main__.py
index ace44c7ca4f..f1dea5f93c1 100644
--- a/tests/functional_tests/test_cases/common/moe_perf/__main__.py
+++ b/tests/functional_tests/test_cases/common/moe_perf/__main__.py
@@ -8,9 +8,8 @@
import os
import statistics
from contextlib import nullcontext
-from dataclasses import dataclass
from pathlib import Path
-from typing import Any, Dict, Iterable, Mapping, Optional, Sequence, cast
+from typing import Any, Dict, Mapping, Sequence, cast
import pytest # type: ignore[import]
import torch
@@ -18,8 +17,7 @@
from megatron.core.config import set_experimental_flag
from megatron.core.fp8_utils import get_fp8_context
from megatron.core.models.gpt.gpt_layer_specs import (
- get_gpt_layer_local_spec,
- get_gpt_layer_with_transformer_engine_spec,
+ get_gpt_layer_with_transformer_engine_submodules,
)
from megatron.core.transformer.moe.fused_a2a import HAVE_DEEP_EP, HAVE_HYBRIDEP
from megatron.core.transformer.moe.moe_layer import MoELayer
@@ -89,10 +87,9 @@ def _build_transformer_config(case: MoEPerformanceCase) -> TransformerConfig:
# NOTE: Only TE backend is covered in this test.
def _resolve_moe_submodules(case: MoEPerformanceCase):
- layer_spec = get_gpt_layer_with_transformer_engine_spec(
+ return get_gpt_layer_with_transformer_engine_submodules(
num_experts=case.model.num_experts, moe_grouped_gemm=True
- )
- return layer_spec.submodules.mlp.submodules
+ ).mlp.submodules
def _load_baselines() -> Dict[str, Dict[str, float]]:
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release/model_config.yaml
index 692e3882e02..91c8cf977a7 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release/model_config.yaml
@@ -70,8 +70,8 @@ MODEL_ARGS:
# Add checkpointing args
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --save-interval: 5000
- --save-retain-interval: 10000
+ --save-interval: 1000
+ --save-retain-interval: 5000
# Add initialization args
--init-method-std: 0.0134
# Add logging args
@@ -90,6 +90,8 @@ MODEL_ARGS:
--bf16: true
--exit-interval: 51000
--wandb-save-dir: ${WANDB_SAVE_PATH}
+ --async-save: true
+ --use-persistent-ckpt-worker: true
METRICS:
- "iteration-time"
- "lm loss"
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_gb200/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_gb200/model_config.yaml
new file mode 100644
index 00000000000..3227158130d
--- /dev/null
+++ b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_gb200/model_config.yaml
@@ -0,0 +1,102 @@
+ENV_VARS:
+ CUDA_DEVICE_MAX_CONNECTIONS: 1
+ NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1
+ NON_DETERMINSTIC_RESULTS: 1
+ NVTE_NORM_FWD_USE_CUDNN: 1
+ NVTE_NORM_BWD_USE_CUDNN: 1
+ NVTE_FUSED_ATTN: 1
+ PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
+ USE_MNNVL: 1
+TEST_TYPE: "release"
+MODEL_ARGS:
+ # Distributed args
+ --distributed-timeout-minutes: 60
+ --tensor-model-parallel-size: 4
+ --pipeline-model-parallel-size: 2
+ --num-layers-per-virtual-pipeline-stage: 8
+ --use-distributed-optimizer: true
+ --overlap-grad-reduce: true
+ --overlap-param-gather: true
+ --tp-comm-overlap: true
+ # Training args
+ --use-mcore-models: true
+ --sequence-parallel: true
+ --disable-bias-linear: true
+ --micro-batch-size: 4
+ --rampup-batch-size: "[384 384 97656250]"
+ --global-batch-size: 1152
+ --train-samples: 19531250
+ --manual-gc: true
+ --cross-entropy-loss-fusion: true
+ --cross-entropy-fusion-impl: te
+ # Transformer Engine args
+ --transformer-impl: transformer_engine
+ # Data args
+ --data-cache-path: ${DATA_CACHE_PATH}
+ --tokenizer-type: GPTSentencePieceTokenizer
+ --tokenizer-model: ${DATA_PATH}/utils/nemotron_2_256k.model
+ --data-path: $DATA_BLEND
+ --split: 99,1,0
+ --no-mmap-bin-files: true
+ --num-workers: 6
+ # Add network size args
+ --apply-layernorm-1p: true
+ --untie-embeddings-and-output-weights: true
+ --position-embedding-type: rope
+ --rotary-percent: 0.5
+ --squared-relu: true
+ --num-layers: 32
+ --hidden-size: 6144
+ --num-attention-heads: 48
+ --group-query-attention: true
+ --num-query-groups: 8
+ --seq-length: 4096
+ --max-position-embeddings: 4096
+ # Add regularization args
+ --attention-dropout: 0.0
+ --hidden-dropout: 0.0
+ --clip-grad: 1.0
+ --weight-decay: 0.1
+ # Add learning rate args
+ --lr-decay-samples: 1949218748
+ --lr-warmup-samples: 3906252
+ --lr: 4.5e-4
+ --min-lr: 4.5e-5
+ --decoupled-lr: 5.0e-4
+ --decoupled-min-lr: 4.5e-5
+ --lr-decay-style: cosine
+ --adam-beta1: 0.9
+ --adam-beta2: 0.95
+ # Add validation args
+ --eval-iters: 32
+ --eval-interval: 2000
+ # Add checkpointing args
+ --save: ${CHECKPOINT_SAVE_PATH}
+ --load: ${CHECKPOINT_LOAD_PATH}
+ --save-interval: 1000
+ --save-retain-interval: 5000
+ # Add initialization args
+ --init-method-std: 0.0134
+ # Add logging args
+ --log-timers-to-tensorboard: true
+ --log-memory-to-tensorboard: true
+ --log-num-zeros-in-grad: true
+ --log-params-norm: true
+ --log-validation-ppl-to-tensorboard: true
+ --log-throughput: true
+ --log-interval: 100
+ --tensorboard-dir: ${TENSORBOARD_PATH}
+ --wandb-project: megatron-core-release-runs
+ --wandb-entity: adlr
+ --wandb-exp-name: ${WANDB_EXPERIMENT}
+ # Add mixed precision args
+ --bf16: true
+ --exit-interval: 51000
+ --wandb-save-dir: ${WANDB_SAVE_PATH}
+ --async-save: true
+ --use-persistent-ckpt-worker: true
+METRICS:
+ - "iteration-time"
+ - "lm loss"
+ - "mem-allocated-bytes"
+ - "mem-max-allocated-bytes"
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm/golden_values_dev_dgx_gb200.json
new file mode 100644
index 00000000000..11918b97283
--- /dev/null
+++ b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm/golden_values_dev_dgx_gb200.json
@@ -0,0 +1,13042 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": 13.05008,
+ "5": 13.01204,
+ "10": 12.1655,
+ "15": 11.32234,
+ "20": 10.41894,
+ "25": 10.0578,
+ "30": 9.71991,
+ "35": 9.42902,
+ "40": 9.23153,
+ "45": 9.0416,
+ "50": 8.84482,
+ "55": 8.66178,
+ "60": 8.60539,
+ "65": 8.47775,
+ "70": 8.43752,
+ "75": 8.34325,
+ "80": 8.18407,
+ "85": 8.12181,
+ "90": 7.96741,
+ "95": 7.95709,
+ "100": 7.84521,
+ "105": 7.70676,
+ "110": 7.56897,
+ "115": 7.458,
+ "120": 7.42069,
+ "125": 7.44964,
+ "130": 7.26452,
+ "135": 7.24116,
+ "140": 7.20033,
+ "145": 7.00832,
+ "150": 7.14295,
+ "155": 6.99244,
+ "160": 6.92185,
+ "165": 6.89548,
+ "170": 6.80023,
+ "175": 6.82899,
+ "180": 6.83583,
+ "185": 6.74551,
+ "190": 6.71201,
+ "195": 6.64915,
+ "200": 6.63261,
+ "205": 6.6249,
+ "210": 6.52711,
+ "215": 6.50221,
+ "220": 6.50404,
+ "225": 6.50149,
+ "230": 6.54783,
+ "235": 6.46088,
+ "240": 6.37609,
+ "245": 6.3747,
+ "250": 6.30165,
+ "255": 6.43597,
+ "260": 6.33348,
+ "265": 6.28434,
+ "270": 6.21703,
+ "275": 6.2455,
+ "280": 6.17513,
+ "285": 6.20833,
+ "290": 6.16109,
+ "295": 6.1242,
+ "300": 6.12642,
+ "305": 6.01347,
+ "310": 6.07181,
+ "315": 6.07118,
+ "320": 5.95351,
+ "325": 5.90429,
+ "330": 5.98027,
+ "335": 6.01029,
+ "340": 5.92597,
+ "345": 5.90736,
+ "350": 5.91072,
+ "355": 5.82381,
+ "360": 5.85127,
+ "365": 5.83524,
+ "370": 5.78767,
+ "375": 5.82558,
+ "380": 5.84099,
+ "385": 5.77487,
+ "390": 5.75924,
+ "395": 5.68715,
+ "400": 5.63262,
+ "405": 5.64579,
+ "410": 5.6491,
+ "415": 5.7162,
+ "420": 5.6389,
+ "425": 5.64386,
+ "430": 5.63857,
+ "435": 5.56311,
+ "440": 5.6115,
+ "445": 5.50046,
+ "450": 5.5841,
+ "455": 5.49511,
+ "460": 5.5081,
+ "465": 5.58817,
+ "470": 5.54722,
+ "475": 5.49015,
+ "480": 5.46104,
+ "485": 5.51932,
+ "490": 5.50384,
+ "495": 5.46751,
+ "500": 5.40583,
+ "505": 5.37976,
+ "510": 5.41077,
+ "515": 5.41349,
+ "520": 5.46389,
+ "525": 5.30176,
+ "530": 5.30865,
+ "535": 5.31487,
+ "540": 5.30664,
+ "545": 5.39279,
+ "550": 5.34384,
+ "555": 5.23202,
+ "560": 5.34449,
+ "565": 5.29541,
+ "570": 5.23678,
+ "575": 5.29742,
+ "580": 5.21604,
+ "585": 5.20684,
+ "590": 5.2311,
+ "595": 5.21895,
+ "600": 5.27384,
+ "605": 5.22693,
+ "610": 5.2111,
+ "615": 5.20312,
+ "620": 5.20466,
+ "625": 5.18657,
+ "630": 5.15555,
+ "635": 5.12189,
+ "640": 5.09743,
+ "645": 5.12261,
+ "650": 5.13982,
+ "655": 5.1289,
+ "660": 5.04202,
+ "665": 5.09654,
+ "670": 5.03729,
+ "675": 5.0268,
+ "680": 5.01185,
+ "685": 4.98136,
+ "690": 5.01416,
+ "695": 4.95502,
+ "700": 4.96149,
+ "705": 4.9334,
+ "710": 4.98004,
+ "715": 4.90045,
+ "720": 4.84477,
+ "725": 4.79493,
+ "730": 4.82978,
+ "735": 4.80907,
+ "740": 4.82743,
+ "745": 4.70279,
+ "750": 4.70949,
+ "755": 4.78826,
+ "760": 4.77357,
+ "765": 4.72919,
+ "770": 4.67543,
+ "775": 4.66386,
+ "780": 4.68062,
+ "785": 4.75851,
+ "790": 4.65081,
+ "795": 4.59989,
+ "800": 4.58234,
+ "805": 4.60231,
+ "810": 4.64497,
+ "815": 4.57763,
+ "820": 4.6061,
+ "825": 4.57606,
+ "830": 4.54268,
+ "835": 4.53145,
+ "840": 4.46479,
+ "845": 4.46978,
+ "850": 4.42697,
+ "855": 4.48821,
+ "860": 4.43561,
+ "865": 4.48227,
+ "870": 4.45584,
+ "875": 4.3794,
+ "880": 4.40382,
+ "885": 4.35898,
+ "890": 4.4257,
+ "895": 4.38718,
+ "900": 4.37761,
+ "905": 4.32099,
+ "910": 4.34998,
+ "915": 4.33433,
+ "920": 4.36562,
+ "925": 4.35701,
+ "930": 4.30214,
+ "935": 4.28572,
+ "940": 4.32955,
+ "945": 4.28423,
+ "950": 4.31902,
+ "955": 4.25429,
+ "960": 4.18985,
+ "965": 4.27248,
+ "970": 4.26224,
+ "975": 4.22351,
+ "980": 4.21989,
+ "985": 4.15887,
+ "990": 4.13424,
+ "995": 4.16609,
+ "1000": 4.21284,
+ "1005": 4.1657,
+ "1010": 4.17265,
+ "1015": 4.09445,
+ "1020": 4.14749,
+ "1025": 4.2074,
+ "1030": 4.11261,
+ "1035": 4.08674,
+ "1040": 4.08595,
+ "1045": 4.09037,
+ "1050": 4.13206,
+ "1055": 4.08609,
+ "1060": 4.10286,
+ "1065": 4.05057,
+ "1070": 4.03731,
+ "1075": 4.06262,
+ "1080": 4.05661,
+ "1085": 4.08637,
+ "1090": 4.02242,
+ "1095": 4.08239,
+ "1100": 4.04944,
+ "1105": 4.05282,
+ "1110": 4.02139,
+ "1115": 4.01265,
+ "1120": 3.99,
+ "1125": 3.98572,
+ "1130": 4.03287,
+ "1135": 3.99997,
+ "1140": 3.98314,
+ "1145": 3.93997,
+ "1150": 4.048,
+ "1155": 3.99054,
+ "1160": 3.9513,
+ "1165": 3.86271,
+ "1170": 3.92237,
+ "1175": 3.93401,
+ "1180": 3.95735,
+ "1185": 3.97312,
+ "1190": 3.92353,
+ "1195": 3.93659,
+ "1200": 3.88343,
+ "1205": 3.87122,
+ "1210": 3.98003,
+ "1215": 3.81935,
+ "1220": 3.86489,
+ "1225": 3.80317,
+ "1230": 3.90989,
+ "1235": 3.88555,
+ "1240": 3.87765,
+ "1245": 3.79262,
+ "1250": 3.8313,
+ "1255": 3.86382,
+ "1260": 3.88293,
+ "1265": 3.78626,
+ "1270": 3.87347,
+ "1275": 3.8262,
+ "1280": 3.81649,
+ "1285": 3.8459,
+ "1290": 3.87404,
+ "1295": 3.84352,
+ "1300": 3.80631,
+ "1305": 3.82946,
+ "1310": 3.80451,
+ "1315": 3.81158,
+ "1320": 3.81004,
+ "1325": 3.7133,
+ "1330": 3.78036,
+ "1335": 3.75635,
+ "1340": 3.74424,
+ "1345": 3.75023,
+ "1350": 3.74904,
+ "1355": 3.7756,
+ "1360": 3.73427,
+ "1365": 3.73041,
+ "1370": 3.73521,
+ "1375": 3.73462,
+ "1380": 3.73031,
+ "1385": 3.73409,
+ "1390": 3.65424,
+ "1395": 3.74026,
+ "1400": 3.72338,
+ "1405": 3.66567,
+ "1410": 3.66913,
+ "1415": 3.64555,
+ "1420": 3.69661,
+ "1425": 3.70429,
+ "1430": 3.66615,
+ "1435": 3.66413,
+ "1440": 3.6299,
+ "1445": 3.6837,
+ "1450": 3.67379,
+ "1455": 3.62801,
+ "1460": 3.64174,
+ "1465": 3.66948,
+ "1470": 3.62106,
+ "1475": 3.68562,
+ "1480": 3.65,
+ "1485": 3.66033,
+ "1490": 3.62097,
+ "1495": 3.59333,
+ "1500": 3.63003,
+ "1505": 3.68557,
+ "1510": 3.5576,
+ "1515": 3.60595,
+ "1520": 3.63257,
+ "1525": 3.59376,
+ "1530": 3.58343,
+ "1535": 3.59111,
+ "1540": 3.61262,
+ "1545": 3.61139,
+ "1550": 3.55891,
+ "1555": 3.56505,
+ "1560": 3.61415,
+ "1565": 3.60344,
+ "1570": 3.5834,
+ "1575": 3.54504,
+ "1580": 3.58566,
+ "1585": 3.55795,
+ "1590": 3.46436,
+ "1595": 3.49688,
+ "1600": 3.5123,
+ "1605": 3.58015,
+ "1610": 3.57647,
+ "1615": 3.48342,
+ "1620": 3.51893,
+ "1625": 3.45637,
+ "1630": 3.49954,
+ "1635": 3.541,
+ "1640": 3.52002,
+ "1645": 3.54913,
+ "1650": 3.49346,
+ "1655": 3.4684,
+ "1660": 3.51337,
+ "1665": 3.44456,
+ "1670": 3.50579,
+ "1675": 3.49404,
+ "1680": 3.46873,
+ "1685": 3.47674,
+ "1690": 3.47471,
+ "1695": 3.48546,
+ "1700": 3.45526,
+ "1705": 3.38706,
+ "1710": 3.48407,
+ "1715": 3.49204,
+ "1720": 3.44268,
+ "1725": 3.42354,
+ "1730": 3.42642,
+ "1735": 3.46923,
+ "1740": 3.457,
+ "1745": 3.43155,
+ "1750": 3.40389,
+ "1755": 3.40607,
+ "1760": 3.37866,
+ "1765": 3.41549,
+ "1770": 3.4484,
+ "1775": 3.38138,
+ "1780": 3.42122,
+ "1785": 3.40965,
+ "1790": 3.38547,
+ "1795": 3.40935,
+ "1800": 3.34031,
+ "1805": 3.3907,
+ "1810": 3.33733,
+ "1815": 3.41598,
+ "1820": 3.4039,
+ "1825": 3.37123,
+ "1830": 3.32433,
+ "1835": 3.4153,
+ "1840": 3.40482,
+ "1845": 3.43338,
+ "1850": 3.38928,
+ "1855": 3.39458,
+ "1860": 3.36025,
+ "1865": 3.38634,
+ "1870": 3.30473,
+ "1875": 3.45012,
+ "1880": 3.34557,
+ "1885": 3.34827,
+ "1890": 3.33046,
+ "1895": 3.38303,
+ "1900": 3.37378,
+ "1905": 3.2966,
+ "1910": 3.32688,
+ "1915": 3.3148,
+ "1920": 3.3538,
+ "1925": 3.33785,
+ "1930": 3.30674,
+ "1935": 3.31319,
+ "1940": 3.36196,
+ "1945": 3.25945,
+ "1950": 3.40875,
+ "1955": 3.30383,
+ "1960": 3.28738,
+ "1965": 3.27524,
+ "1970": 3.28884,
+ "1975": 3.339,
+ "1980": 3.33871,
+ "1985": 3.23068,
+ "1990": 3.29676,
+ "1995": 3.27317,
+ "2000": 3.27611,
+ "2005": 3.25628,
+ "2010": 3.25614,
+ "2015": 3.23395,
+ "2020": 3.27251,
+ "2025": 3.27212,
+ "2030": 3.26956,
+ "2035": 3.29321,
+ "2040": 3.24341,
+ "2045": 3.24512,
+ "2050": 3.27219,
+ "2055": 3.31028,
+ "2060": 3.27911,
+ "2065": 3.22105,
+ "2070": 3.30262,
+ "2075": 3.24308,
+ "2080": 3.23722,
+ "2085": 3.28743,
+ "2090": 3.14147,
+ "2095": 3.29285,
+ "2100": 3.23507,
+ "2105": 3.18698,
+ "2110": 3.19488,
+ "2115": 3.22507,
+ "2120": 3.17281,
+ "2125": 3.20994,
+ "2130": 3.21504,
+ "2135": 3.2795,
+ "2140": 3.20055,
+ "2145": 3.21514,
+ "2150": 3.20567,
+ "2155": 3.22994,
+ "2160": 3.21817,
+ "2165": 3.28593,
+ "2170": 3.22314,
+ "2175": 3.17419,
+ "2180": 3.22019,
+ "2185": 3.24188,
+ "2190": 3.22977,
+ "2195": 3.15677,
+ "2200": 3.18661,
+ "2205": 3.16602,
+ "2210": 3.12392,
+ "2215": 3.20063,
+ "2220": 3.19672,
+ "2225": 3.17772,
+ "2230": 3.12195,
+ "2235": 3.17896,
+ "2240": 3.21655,
+ "2245": 3.2019,
+ "2250": 3.21383,
+ "2255": 3.15541,
+ "2260": 3.16941,
+ "2265": 3.21191,
+ "2270": 3.17741,
+ "2275": 3.13051,
+ "2280": 3.17617,
+ "2285": 3.15025,
+ "2290": 3.15609,
+ "2295": 3.20156,
+ "2300": 3.14421,
+ "2305": 3.15813,
+ "2310": 3.12811,
+ "2315": 3.05602,
+ "2320": 3.12288,
+ "2325": 3.17284,
+ "2330": 3.13107,
+ "2335": 3.10638,
+ "2340": 3.15454,
+ "2345": 3.11338,
+ "2350": 3.12349,
+ "2355": 3.11221,
+ "2360": 3.1587,
+ "2365": 3.13477,
+ "2370": 3.15842,
+ "2375": 3.14138,
+ "2380": 3.11083,
+ "2385": 3.07931,
+ "2390": 3.09995,
+ "2395": 3.08487,
+ "2400": 3.08246,
+ "2405": 3.09178,
+ "2410": 3.08889,
+ "2415": 3.08364,
+ "2420": 3.0794,
+ "2425": 3.07602,
+ "2430": 3.07131,
+ "2435": 3.0765,
+ "2440": 3.08722,
+ "2445": 3.06039,
+ "2450": 3.12452,
+ "2455": 3.14884,
+ "2460": 3.08647,
+ "2465": 3.06823,
+ "2470": 3.02841,
+ "2475": 3.06105,
+ "2480": 3.08627,
+ "2485": 3.05418,
+ "2490": 3.0659,
+ "2495": 3.10432,
+ "2500": 3.08628,
+ "2505": 3.10056,
+ "2510": 3.11958,
+ "2515": 3.04463,
+ "2520": 3.06461,
+ "2525": 3.03564,
+ "2530": 3.04561,
+ "2535": 3.08874,
+ "2540": 3.07717,
+ "2545": 3.04594,
+ "2550": 2.98846,
+ "2555": 3.06822,
+ "2560": 3.04282,
+ "2565": 3.1096,
+ "2570": 2.99987,
+ "2575": 3.05028,
+ "2580": 3.07924,
+ "2585": 3.01705,
+ "2590": 3.06415,
+ "2595": 2.99331,
+ "2600": 3.06187,
+ "2605": 3.0334,
+ "2610": 3.04104,
+ "2615": 3.06401,
+ "2620": 2.98076,
+ "2625": 3.00513,
+ "2630": 3.03888,
+ "2635": 3.05046,
+ "2640": 3.00752,
+ "2645": 3.05044,
+ "2650": 3.02104,
+ "2655": 2.99029,
+ "2660": 3.01279,
+ "2665": 3.03456,
+ "2670": 2.97516,
+ "2675": 2.9664,
+ "2680": 2.99127,
+ "2685": 3.00057,
+ "2690": 3.00136,
+ "2695": 2.9829,
+ "2700": 3.02461,
+ "2705": 2.98686,
+ "2710": 2.97101,
+ "2715": 2.96259,
+ "2720": 3.02765,
+ "2725": 2.9843,
+ "2730": 3.07085,
+ "2735": 3.00313,
+ "2740": 2.99675,
+ "2745": 3.01003,
+ "2750": 3.00601,
+ "2755": 2.9679,
+ "2760": 2.99515,
+ "2765": 3.0052,
+ "2770": 2.97319,
+ "2775": 2.98775,
+ "2780": 3.01171,
+ "2785": 2.96171,
+ "2790": 2.9614,
+ "2795": 2.95103,
+ "2800": 2.95137,
+ "2805": 2.92932,
+ "2810": 2.97584,
+ "2815": 2.95426,
+ "2820": 3.02824,
+ "2825": 3.03446,
+ "2830": 2.99241,
+ "2835": 2.92405,
+ "2840": 2.924,
+ "2845": 3.01764,
+ "2850": 3.02928,
+ "2855": 3.00953,
+ "2860": 2.96462,
+ "2865": 2.91418,
+ "2870": 2.98414,
+ "2875": 2.91364,
+ "2880": 2.94595,
+ "2885": 2.91375,
+ "2890": 2.97568,
+ "2895": 2.92367,
+ "2900": 2.95047,
+ "2905": 2.99681,
+ "2910": 2.96428,
+ "2915": 2.97992,
+ "2920": 2.96376,
+ "2925": 2.94181,
+ "2930": 2.95351,
+ "2935": 2.94268,
+ "2940": 2.95302,
+ "2945": 2.92122,
+ "2950": 2.98363,
+ "2955": 2.90414,
+ "2960": 2.96628,
+ "2965": 2.86677,
+ "2970": 2.96148,
+ "2975": 2.98973,
+ "2980": 2.93798,
+ "2985": 3.03145,
+ "2990": 2.93902,
+ "2995": 2.87289,
+ "3000": 2.93094,
+ "3005": 2.88807,
+ "3010": 2.93649,
+ "3015": 2.93728,
+ "3020": 2.9765,
+ "3025": 2.94765,
+ "3030": 2.94829,
+ "3035": 2.96692,
+ "3040": 2.94429,
+ "3045": 2.8457,
+ "3050": 2.9005,
+ "3055": 2.90093,
+ "3060": 2.93402,
+ "3065": 2.92043,
+ "3070": 2.88416,
+ "3075": 2.89414,
+ "3080": 2.92836,
+ "3085": 2.90419,
+ "3090": 2.92118,
+ "3095": 2.92732,
+ "3100": 2.86811,
+ "3105": 2.93241,
+ "3110": 2.92174,
+ "3115": 2.95416,
+ "3120": 2.9617,
+ "3125": 2.86072,
+ "3130": 2.93315,
+ "3135": 2.92603,
+ "3140": 2.8763,
+ "3145": 2.91992,
+ "3150": 2.85429,
+ "3155": 2.85441,
+ "3160": 2.84409,
+ "3165": 2.8445,
+ "3170": 2.90103,
+ "3175": 2.91615,
+ "3180": 2.87162,
+ "3185": 2.93749,
+ "3190": 2.94414,
+ "3195": 2.95906,
+ "3200": 2.9346,
+ "3205": 2.86154,
+ "3210": 2.86991,
+ "3215": 2.91437,
+ "3220": 2.86962,
+ "3225": 2.86687,
+ "3230": 2.81718,
+ "3235": 2.87781,
+ "3240": 2.87293,
+ "3245": 2.90545,
+ "3250": 2.86599,
+ "3255": 2.85075,
+ "3260": 2.86584,
+ "3265": 2.87425,
+ "3270": 2.84613,
+ "3275": 2.86845,
+ "3280": 2.80107,
+ "3285": 2.81792,
+ "3290": 2.87182,
+ "3295": 2.9051,
+ "3300": 2.87482,
+ "3305": 2.86457,
+ "3310": 2.8617,
+ "3315": 2.82258,
+ "3320": 2.84626,
+ "3325": 2.83293,
+ "3330": 2.83744,
+ "3335": 2.85803,
+ "3340": 2.83014,
+ "3345": 2.84335,
+ "3350": 2.92003,
+ "3355": 2.88331,
+ "3360": 2.81284,
+ "3365": 2.85791,
+ "3370": 2.84692,
+ "3375": 2.83772,
+ "3380": 2.85031,
+ "3385": 2.8801,
+ "3390": 2.87178,
+ "3395": 2.81384,
+ "3400": 2.78856,
+ "3405": 2.8419,
+ "3410": 2.85225,
+ "3415": 2.86352,
+ "3420": 2.82864,
+ "3425": 2.81295,
+ "3430": 2.83167,
+ "3435": 2.89686,
+ "3440": 2.8282,
+ "3445": 2.88078,
+ "3450": 2.8261,
+ "3455": 2.79448,
+ "3460": 2.82071,
+ "3465": 2.8541,
+ "3470": 2.84374,
+ "3475": 2.77799,
+ "3480": 2.8464,
+ "3485": 2.82783,
+ "3490": 2.89586,
+ "3495": 2.86003,
+ "3500": 2.83569,
+ "3505": 2.82809,
+ "3510": 2.82659,
+ "3515": 2.83789,
+ "3520": 2.78684,
+ "3525": 2.8091,
+ "3530": 2.85704,
+ "3535": 2.78933,
+ "3540": 2.84286,
+ "3545": 2.81678,
+ "3550": 2.80459,
+ "3555": 2.82485,
+ "3560": 2.82756,
+ "3565": 2.83286,
+ "3570": 2.80757,
+ "3575": 2.81456,
+ "3580": 2.82206,
+ "3585": 2.83994,
+ "3590": 2.83354,
+ "3595": 2.78616,
+ "3600": 2.75448,
+ "3605": 2.79347,
+ "3610": 2.8501,
+ "3615": 2.75759,
+ "3620": 2.81049,
+ "3625": 2.89049,
+ "3630": 2.78648,
+ "3635": 2.79541,
+ "3640": 2.78992,
+ "3645": 2.77597,
+ "3650": 2.81076,
+ "3655": 2.82175,
+ "3660": 2.77324,
+ "3665": 2.78879,
+ "3670": 2.77345,
+ "3675": 2.77485,
+ "3680": 2.82264,
+ "3685": 2.81255,
+ "3690": 2.80639,
+ "3695": 2.81455,
+ "3700": 2.79382,
+ "3705": 2.79333,
+ "3710": 2.75729,
+ "3715": 2.80607,
+ "3720": 2.80223,
+ "3725": 2.79579,
+ "3730": 2.84735,
+ "3735": 2.80322,
+ "3740": 2.75398,
+ "3745": 2.8006,
+ "3750": 2.80967,
+ "3755": 2.80087,
+ "3760": 2.76493,
+ "3765": 2.75981,
+ "3770": 2.76212,
+ "3775": 2.77797,
+ "3780": 2.76132,
+ "3785": 2.78468,
+ "3790": 2.75638,
+ "3795": 2.79596,
+ "3800": 2.8073,
+ "3805": 2.75971,
+ "3810": 2.8053,
+ "3815": 2.76711,
+ "3820": 2.79005,
+ "3825": 2.73907,
+ "3830": 2.75002,
+ "3835": 2.81788,
+ "3840": 2.72938,
+ "3845": 2.71849,
+ "3850": 2.78452,
+ "3855": 2.72543,
+ "3860": 2.80742,
+ "3865": 2.75975,
+ "3870": 2.77909,
+ "3875": 2.75698,
+ "3880": 2.79154,
+ "3885": 2.79772,
+ "3890": 2.74732,
+ "3895": 2.81224,
+ "3900": 2.7667,
+ "3905": 2.72768,
+ "3910": 2.75146,
+ "3915": 2.75568,
+ "3920": 2.7965,
+ "3925": 2.7834,
+ "3930": 2.72104,
+ "3935": 2.7481,
+ "3940": 2.75988,
+ "3945": 2.74416,
+ "3950": 2.73118,
+ "3955": 2.78157,
+ "3960": 2.76135,
+ "3965": 2.74076,
+ "3970": 2.75926,
+ "3975": 2.73309,
+ "3980": 2.744,
+ "3985": 2.75444,
+ "3990": 2.69665,
+ "3995": 2.78631,
+ "4000": 2.73901,
+ "4005": 2.77258,
+ "4010": 2.71546,
+ "4015": 2.73038,
+ "4020": 2.75439,
+ "4025": 2.74097,
+ "4030": 2.66413,
+ "4035": 2.70287,
+ "4040": 2.75553,
+ "4045": 2.75354,
+ "4050": 2.79756,
+ "4055": 2.72714,
+ "4060": 2.72047,
+ "4065": 2.65865,
+ "4070": 2.81308,
+ "4075": 2.76334,
+ "4080": 2.72285,
+ "4085": 2.75607,
+ "4090": 2.68226,
+ "4095": 2.69234,
+ "4100": 2.71963,
+ "4105": 2.74397,
+ "4110": 2.73704,
+ "4115": 2.70976,
+ "4120": 2.73378,
+ "4125": 2.70953,
+ "4130": 2.70207,
+ "4135": 2.69241,
+ "4140": 2.68559,
+ "4145": 2.78455,
+ "4150": 2.71388,
+ "4155": 2.7486,
+ "4160": 2.76565,
+ "4165": 2.72377,
+ "4170": 2.67692,
+ "4175": 2.72423,
+ "4180": 2.73214,
+ "4185": 2.73112,
+ "4190": 2.74361,
+ "4195": 2.70199,
+ "4200": 2.71084,
+ "4205": 2.74634,
+ "4210": 2.68089,
+ "4215": 2.6701,
+ "4220": 2.66543,
+ "4225": 2.71006,
+ "4230": 2.72178,
+ "4235": 2.73702,
+ "4240": 2.70895,
+ "4245": 2.69911,
+ "4250": 2.72113,
+ "4255": 2.65314,
+ "4260": 2.72897,
+ "4265": 2.74195,
+ "4270": 2.72821,
+ "4275": 2.69356,
+ "4280": 2.71212,
+ "4285": 2.74058,
+ "4290": 2.69094,
+ "4295": 2.6967,
+ "4300": 2.70408,
+ "4305": 2.70382,
+ "4310": 2.73923,
+ "4315": 2.71253,
+ "4320": 2.7063,
+ "4325": 2.70787,
+ "4330": 2.7135,
+ "4335": 2.69438,
+ "4340": 2.7039,
+ "4345": 2.72843,
+ "4350": 2.68147,
+ "4355": 2.70469,
+ "4360": 2.7148,
+ "4365": 2.7912,
+ "4370": 2.74169,
+ "4375": 2.74601,
+ "4380": 2.70617,
+ "4385": 2.7023,
+ "4390": 2.70399,
+ "4395": 2.75676,
+ "4400": 2.66802,
+ "4405": 2.67255,
+ "4410": 2.68624,
+ "4415": 2.70971,
+ "4420": 2.71023,
+ "4425": 2.72318,
+ "4430": 2.696,
+ "4435": 2.68512,
+ "4440": 2.69942,
+ "4445": 2.68608,
+ "4450": 2.66467,
+ "4455": 2.67,
+ "4460": 2.6879,
+ "4465": 2.70646,
+ "4470": 2.67224,
+ "4475": 2.69263,
+ "4480": 2.66016,
+ "4485": 2.70684,
+ "4490": 2.65797,
+ "4495": 2.71483,
+ "4500": 2.70827,
+ "4505": 2.70383,
+ "4510": 2.65407,
+ "4515": 2.70511,
+ "4520": 2.67563,
+ "4525": 2.67431,
+ "4530": 2.67629,
+ "4535": 2.67349,
+ "4540": 2.71319,
+ "4545": 2.66393,
+ "4550": 2.70558,
+ "4555": 2.68533,
+ "4560": 2.65973,
+ "4565": 2.64332,
+ "4570": 2.6436,
+ "4575": 2.67031,
+ "4580": 2.68851,
+ "4585": 2.68755,
+ "4590": 2.62319,
+ "4595": 2.66776,
+ "4600": 2.68266,
+ "4605": 2.6814,
+ "4610": 2.66475,
+ "4615": 2.66996,
+ "4620": 2.67313,
+ "4625": 2.69613,
+ "4630": 2.67469,
+ "4635": 2.64781,
+ "4640": 2.69472,
+ "4645": 2.64862,
+ "4650": 2.70415,
+ "4655": 2.70835,
+ "4660": 2.67805,
+ "4665": 2.68807,
+ "4670": 2.67892,
+ "4675": 2.68891,
+ "4680": 2.66775,
+ "4685": 2.65991,
+ "4690": 2.705,
+ "4695": 2.65936,
+ "4700": 2.67667,
+ "4705": 2.65892,
+ "4710": 2.67964,
+ "4715": 2.65107,
+ "4720": 2.72776,
+ "4725": 2.63773,
+ "4730": 2.6601,
+ "4735": 2.69113,
+ "4740": 2.64705,
+ "4745": 2.65631,
+ "4750": 2.64529,
+ "4755": 2.65759,
+ "4760": 2.66598,
+ "4765": 2.64721,
+ "4770": 2.62567,
+ "4775": 2.66265,
+ "4780": 2.66479,
+ "4785": 2.69336,
+ "4790": 2.65185,
+ "4795": 2.67703,
+ "4800": 2.63477,
+ "4805": 2.64622,
+ "4810": 2.66639,
+ "4815": 2.64958,
+ "4820": 2.67431,
+ "4825": 2.65801,
+ "4830": 2.62203,
+ "4835": 2.65316,
+ "4840": 2.66379,
+ "4845": 2.66014,
+ "4850": 2.63136,
+ "4855": 2.60636,
+ "4860": 2.65637,
+ "4865": 2.63154,
+ "4870": 2.64347,
+ "4875": 2.62372,
+ "4880": 2.63111,
+ "4885": 2.63425,
+ "4890": 2.6904,
+ "4895": 2.66758,
+ "4900": 2.62197,
+ "4905": 2.62442,
+ "4910": 2.64166,
+ "4915": 2.61951,
+ "4920": 2.65575,
+ "4925": 2.65667,
+ "4930": 2.57607,
+ "4935": 2.65457,
+ "4940": 2.63403,
+ "4945": 2.64168,
+ "4950": 2.63147,
+ "4955": 2.62132,
+ "4960": 2.6207,
+ "4965": 2.66066,
+ "4970": 2.60408,
+ "4975": 2.65875,
+ "4980": 2.62384,
+ "4985": 2.63572,
+ "4990": 2.66221,
+ "4995": 2.58438,
+ "5000": 2.66598,
+ "5005": 2.66961,
+ "5010": 2.68942,
+ "5015": 2.63994,
+ "5020": 2.64342,
+ "5025": 2.68953,
+ "5030": 2.64884,
+ "5035": 2.62109,
+ "5040": 2.62556,
+ "5045": 2.61018,
+ "5050": 2.6284,
+ "5055": 2.65151,
+ "5060": 2.64873,
+ "5065": 2.69265,
+ "5070": 2.61131,
+ "5075": 2.61722,
+ "5080": 2.61434,
+ "5085": 2.61211,
+ "5090": 2.59602,
+ "5095": 2.65863,
+ "5100": 2.65368,
+ "5105": 2.61404,
+ "5110": 2.66578,
+ "5115": 2.62643,
+ "5120": 2.67473,
+ "5125": 2.63416,
+ "5130": 2.62146,
+ "5135": 2.64001,
+ "5140": 2.59109,
+ "5145": 2.636,
+ "5150": 2.65103,
+ "5155": 2.64678,
+ "5160": 2.67741,
+ "5165": 2.59135,
+ "5170": 2.59663,
+ "5175": 2.62169,
+ "5180": 2.60891,
+ "5185": 2.62655,
+ "5190": 2.6275,
+ "5195": 2.67374,
+ "5200": 2.60143,
+ "5205": 2.60779,
+ "5210": 2.60797,
+ "5215": 2.64883,
+ "5220": 2.58969,
+ "5225": 2.55564,
+ "5230": 2.63704,
+ "5235": 2.61808,
+ "5240": 2.61892,
+ "5245": 2.6334,
+ "5250": 2.59759,
+ "5255": 2.61999,
+ "5260": 2.5638,
+ "5265": 2.6022,
+ "5270": 2.5957,
+ "5275": 2.62453,
+ "5280": 2.61679,
+ "5285": 2.609,
+ "5290": 2.63809,
+ "5295": 2.62998,
+ "5300": 2.58882,
+ "5305": 2.63394,
+ "5310": 2.62559,
+ "5315": 2.59505,
+ "5320": 2.61946,
+ "5325": 2.64889,
+ "5330": 2.60433,
+ "5335": 2.58462,
+ "5340": 2.56677,
+ "5345": 2.66119,
+ "5350": 2.62072,
+ "5355": 2.58224,
+ "5360": 2.59753,
+ "5365": 2.62368,
+ "5370": 2.61875,
+ "5375": 2.63305,
+ "5380": 2.58162,
+ "5385": 2.56549,
+ "5390": 2.58914,
+ "5395": 2.62188,
+ "5400": 2.61137,
+ "5405": 2.54782,
+ "5410": 2.61765,
+ "5415": 2.60072,
+ "5420": 2.6175,
+ "5425": 2.62744,
+ "5430": 2.63046,
+ "5435": 2.5818,
+ "5440": 2.59048,
+ "5445": 2.63239,
+ "5450": 2.65266,
+ "5455": 2.6172,
+ "5460": 2.59535,
+ "5465": 2.60935,
+ "5470": 2.60473,
+ "5475": 2.62847,
+ "5480": 2.59117,
+ "5485": 2.59387,
+ "5490": 2.58,
+ "5495": 2.57536,
+ "5500": 2.57456,
+ "5505": 2.62145,
+ "5510": 2.62833,
+ "5515": 2.58533,
+ "5520": 2.56247,
+ "5525": 2.59009,
+ "5530": 2.66987,
+ "5535": 2.62544,
+ "5540": 2.57594,
+ "5545": 2.60009,
+ "5550": 2.55588,
+ "5555": 2.57975,
+ "5560": 2.56808,
+ "5565": 2.61179,
+ "5570": 2.66123,
+ "5575": 2.63601,
+ "5580": 2.57869,
+ "5585": 2.59855,
+ "5590": 2.56836,
+ "5595": 2.59581,
+ "5600": 2.55933,
+ "5605": 2.60378,
+ "5610": 2.58997,
+ "5615": 2.58968,
+ "5620": 2.58638,
+ "5625": 2.55629,
+ "5630": 2.5763,
+ "5635": 2.63778,
+ "5640": 2.59795,
+ "5645": 2.57553,
+ "5650": 2.58201,
+ "5655": 2.55416,
+ "5660": 2.56329,
+ "5665": 2.59005,
+ "5670": 2.56952,
+ "5675": 2.61052,
+ "5680": 2.53178,
+ "5685": 2.57099,
+ "5690": 2.60544,
+ "5695": 2.56507,
+ "5700": 2.60075,
+ "5705": 2.60105,
+ "5710": 2.58147,
+ "5715": 2.58925,
+ "5720": 2.53967,
+ "5725": 2.60663,
+ "5730": 2.57648,
+ "5735": 2.61186,
+ "5740": 2.5967,
+ "5745": 2.56064,
+ "5750": 2.54451,
+ "5755": 2.56468,
+ "5760": 2.62209,
+ "5765": 2.56228,
+ "5770": 2.54488,
+ "5775": 2.58976,
+ "5780": 2.58017,
+ "5785": 2.54338,
+ "5790": 2.56917,
+ "5795": 2.60813,
+ "5800": 2.54915,
+ "5805": 2.53928,
+ "5810": 2.56066,
+ "5815": 2.52773,
+ "5820": 2.60045,
+ "5825": 2.50845,
+ "5830": 2.50043,
+ "5835": 2.6004,
+ "5840": 2.54297,
+ "5845": 2.55645,
+ "5850": 2.6166,
+ "5855": 2.51288,
+ "5860": 2.56548,
+ "5865": 2.52098,
+ "5870": 2.57866,
+ "5875": 2.61158,
+ "5880": 2.58684,
+ "5885": 2.5689,
+ "5890": 2.58709,
+ "5895": 2.55628,
+ "5900": 2.61888,
+ "5905": 2.56102,
+ "5910": 2.60329,
+ "5915": 2.6141,
+ "5920": 2.59205,
+ "5925": 2.53533,
+ "5930": 2.57933,
+ "5935": 2.55532,
+ "5940": 2.57406,
+ "5945": 2.52396,
+ "5950": 2.56069,
+ "5955": 2.59134,
+ "5960": 2.56674,
+ "5965": 2.62254,
+ "5970": 2.55477,
+ "5975": 2.58563,
+ "5980": 2.56214,
+ "5985": 2.56218,
+ "5990": 2.55816,
+ "5995": 2.55939,
+ "6000": 2.55916,
+ "6005": 2.52582,
+ "6010": 2.56294,
+ "6015": 2.5274,
+ "6020": 2.53613,
+ "6025": 2.55915,
+ "6030": 2.60835,
+ "6035": 2.54559,
+ "6040": 2.551,
+ "6045": 2.49287,
+ "6050": 2.59671,
+ "6055": 2.52113,
+ "6060": 2.54903,
+ "6065": 2.52668,
+ "6070": 2.53063,
+ "6075": 2.54125,
+ "6080": 2.53828,
+ "6085": 2.6014,
+ "6090": 2.57326,
+ "6095": 2.53938,
+ "6100": 2.54651,
+ "6105": 2.52417,
+ "6110": 2.55763,
+ "6115": 2.58953,
+ "6120": 2.56015,
+ "6125": 2.54649,
+ "6130": 2.47548,
+ "6135": 2.56256,
+ "6140": 2.55995,
+ "6145": 2.56011,
+ "6150": 2.52806,
+ "6155": 2.51155,
+ "6160": 2.54318,
+ "6165": 2.57878,
+ "6170": 2.55001,
+ "6175": 2.60347,
+ "6180": 2.51199,
+ "6185": 2.55297,
+ "6190": 2.4939,
+ "6195": 2.58162,
+ "6200": 2.55703,
+ "6205": 2.54407,
+ "6210": 2.52438,
+ "6215": 2.51685,
+ "6220": 2.56896,
+ "6225": 2.51605,
+ "6230": 2.51271,
+ "6235": 2.56294,
+ "6240": 2.55265,
+ "6245": 2.5273,
+ "6250": 2.53499,
+ "6255": 2.57521,
+ "6260": 2.52745,
+ "6265": 2.57783,
+ "6270": 2.52798,
+ "6275": 2.56887,
+ "6280": 2.52652,
+ "6285": 2.52461,
+ "6290": 2.52333,
+ "6295": 2.50807,
+ "6300": 2.55739,
+ "6305": 2.52703,
+ "6310": 2.51458,
+ "6315": 2.53888,
+ "6320": 2.49247,
+ "6325": 2.60019,
+ "6330": 2.55901,
+ "6335": 2.51425,
+ "6340": 2.51261,
+ "6345": 2.55869,
+ "6350": 2.56028,
+ "6355": 2.52664,
+ "6360": 2.52402,
+ "6365": 2.48552,
+ "6370": 2.53722,
+ "6375": 2.49389,
+ "6380": 2.56229,
+ "6385": 2.57823,
+ "6390": 2.50771,
+ "6395": 2.55326,
+ "6400": 2.50997,
+ "6405": 2.53036,
+ "6410": 2.52054,
+ "6415": 2.52596,
+ "6420": 2.54097,
+ "6425": 2.53566,
+ "6430": 2.5808,
+ "6435": 2.54817,
+ "6440": 2.53987,
+ "6445": 2.53241,
+ "6450": 2.53514,
+ "6455": 2.52675,
+ "6460": 2.51878,
+ "6465": 2.56707,
+ "6470": 2.52212,
+ "6475": 2.5281,
+ "6480": 2.4893,
+ "6485": 2.52991,
+ "6490": 2.51048,
+ "6495": 2.50057,
+ "6500": 2.52728,
+ "6505": 2.49759,
+ "6510": 2.54453,
+ "6515": 2.51139,
+ "6520": 2.51311,
+ "6525": 2.49504,
+ "6530": 2.54469,
+ "6535": 2.5335,
+ "6540": 2.53179,
+ "6545": 2.56322,
+ "6550": 2.50205,
+ "6555": 2.55879,
+ "6560": 2.51272,
+ "6565": 2.52344,
+ "6570": 2.58812,
+ "6575": 2.52412,
+ "6580": 2.50017,
+ "6585": 2.50818,
+ "6590": 2.51163,
+ "6595": 2.49887,
+ "6600": 2.4898,
+ "6605": 2.5406,
+ "6610": 2.47882,
+ "6615": 2.57,
+ "6620": 2.53557,
+ "6625": 2.51312,
+ "6630": 2.51294,
+ "6635": 2.47504,
+ "6640": 2.54233,
+ "6645": 2.59784,
+ "6650": 2.5111,
+ "6655": 2.50078,
+ "6660": 2.57583,
+ "6665": 2.52098,
+ "6670": 2.57063,
+ "6675": 2.46875,
+ "6680": 2.54988,
+ "6685": 2.53786,
+ "6690": 2.51562,
+ "6695": 2.4897,
+ "6700": 2.52535,
+ "6705": 2.5205,
+ "6710": 2.49369,
+ "6715": 2.51757,
+ "6720": 2.5118,
+ "6725": 2.52401,
+ "6730": 2.52057,
+ "6735": 2.48485,
+ "6740": 2.5167,
+ "6745": 2.49467,
+ "6750": 2.55883,
+ "6755": 2.47556,
+ "6760": 2.54412,
+ "6765": 2.48839,
+ "6770": 2.51956,
+ "6775": 2.51224,
+ "6780": 2.53968,
+ "6785": 2.47432,
+ "6790": 2.54663,
+ "6795": 2.50016,
+ "6800": 2.5269,
+ "6805": 2.51219,
+ "6810": 2.50666,
+ "6815": 2.52359,
+ "6820": 2.48606,
+ "6825": 2.50599,
+ "6830": 2.54009,
+ "6835": 2.50773,
+ "6840": 2.50931,
+ "6845": 2.5268,
+ "6850": 2.47552,
+ "6855": 2.51576,
+ "6860": 2.50426,
+ "6865": 2.48961,
+ "6870": 2.55456,
+ "6875": 2.47601,
+ "6880": 2.5516,
+ "6885": 2.47979,
+ "6890": 2.54624,
+ "6895": 2.50264,
+ "6900": 2.48907,
+ "6905": 2.49713,
+ "6910": 2.51911,
+ "6915": 2.5182,
+ "6920": 2.53495,
+ "6925": 2.54115,
+ "6930": 2.49071,
+ "6935": 2.5187,
+ "6940": 2.49868,
+ "6945": 2.46109,
+ "6950": 2.48692,
+ "6955": 2.52809,
+ "6960": 2.52197,
+ "6965": 2.49412,
+ "6970": 2.47206,
+ "6975": 2.52296,
+ "6980": 2.45471,
+ "6985": 2.5177,
+ "6990": 2.53059,
+ "6995": 2.46347,
+ "7000": 2.48934,
+ "7005": 2.46926,
+ "7010": 2.47439,
+ "7015": 2.52132,
+ "7020": 2.46643,
+ "7025": 2.45331,
+ "7030": 2.4875,
+ "7035": 2.47931,
+ "7040": 2.50617,
+ "7045": 2.5207,
+ "7050": 2.52899,
+ "7055": 2.44321,
+ "7060": 2.47483,
+ "7065": 2.48776,
+ "7070": 2.49125,
+ "7075": 2.49322,
+ "7080": 2.53446,
+ "7085": 2.48571,
+ "7090": 2.47712,
+ "7095": 2.50173,
+ "7100": 2.51547,
+ "7105": 2.4879,
+ "7110": 2.48725,
+ "7115": 2.50475,
+ "7120": 2.47235,
+ "7125": 2.46446,
+ "7130": 2.48765,
+ "7135": 2.51409,
+ "7140": 2.50062,
+ "7145": 2.49946,
+ "7150": 2.51067,
+ "7155": 2.50512,
+ "7160": 2.47574,
+ "7165": 2.4578,
+ "7170": 2.50841,
+ "7175": 2.50613,
+ "7180": 2.50375,
+ "7185": 2.48163,
+ "7190": 2.46209,
+ "7195": 2.47862,
+ "7200": 2.51196,
+ "7205": 2.49121,
+ "7210": 2.44589,
+ "7215": 2.48125,
+ "7220": 2.44456,
+ "7225": 2.5132,
+ "7230": 2.50961,
+ "7235": 2.48266,
+ "7240": 2.47731,
+ "7245": 2.50007,
+ "7250": 2.50923,
+ "7255": 2.49701,
+ "7260": 2.46243,
+ "7265": 2.45514,
+ "7270": 2.47182,
+ "7275": 2.50104,
+ "7280": 2.49421,
+ "7285": 2.42317,
+ "7290": 2.48017,
+ "7295": 2.48888,
+ "7300": 2.41969,
+ "7305": 2.44782,
+ "7310": 2.45036,
+ "7315": 2.49163,
+ "7320": 2.48463,
+ "7325": 2.45999,
+ "7330": 2.49158,
+ "7335": 2.47761,
+ "7340": 2.4651,
+ "7345": 2.49594,
+ "7350": 2.50993,
+ "7355": 2.49634,
+ "7360": 2.48177,
+ "7365": 2.47188,
+ "7370": 2.47367,
+ "7375": 2.45059,
+ "7380": 2.49507,
+ "7385": 2.48552,
+ "7390": 2.47293,
+ "7395": 2.47305,
+ "7400": 2.48041,
+ "7405": 2.43937,
+ "7410": 2.48234,
+ "7415": 2.47393,
+ "7420": 2.49325,
+ "7425": 2.45705,
+ "7430": 2.5244,
+ "7435": 2.49247,
+ "7440": 2.52053,
+ "7445": 2.50973,
+ "7450": 2.47691,
+ "7455": 2.45855,
+ "7460": 2.46721,
+ "7465": 2.47716,
+ "7470": 2.45147,
+ "7475": 2.45825,
+ "7480": 2.51369,
+ "7485": 2.44885,
+ "7490": 2.47507,
+ "7495": 2.48093,
+ "7500": 2.49604,
+ "7505": 2.44117,
+ "7510": 2.43666,
+ "7515": 2.42007,
+ "7520": 2.49219,
+ "7525": 2.49742,
+ "7530": 2.47703,
+ "7535": 2.46038,
+ "7540": 2.4741,
+ "7545": 2.47511,
+ "7550": 2.49065,
+ "7555": 2.45383,
+ "7560": 2.43052,
+ "7565": 2.50969,
+ "7570": 2.48512,
+ "7575": 2.44278,
+ "7580": 2.45861,
+ "7585": 2.48267,
+ "7590": 2.48145,
+ "7595": 2.46474,
+ "7600": 2.46272,
+ "7605": 2.44838,
+ "7610": 2.45322,
+ "7615": 2.42741,
+ "7620": 2.5483,
+ "7625": 2.48566,
+ "7630": 2.42481,
+ "7635": 2.42734,
+ "7640": 2.45557,
+ "7645": 2.47446,
+ "7650": 2.46405,
+ "7655": 2.48523,
+ "7660": 2.45448,
+ "7665": 2.43415,
+ "7670": 2.44222,
+ "7675": 2.45634,
+ "7680": 2.48669,
+ "7685": 2.43365,
+ "7690": 2.48235,
+ "7695": 2.45562,
+ "7700": 2.48221,
+ "7705": 2.50089,
+ "7710": 2.49559,
+ "7715": 2.44301,
+ "7720": 2.47101,
+ "7725": 2.48149,
+ "7730": 2.45764,
+ "7735": 2.47245,
+ "7740": 2.43789,
+ "7745": 2.45085,
+ "7750": 2.43894,
+ "7755": 2.46672,
+ "7760": 2.45099,
+ "7765": 2.45461,
+ "7770": 2.46974,
+ "7775": 2.45582,
+ "7780": 2.41891,
+ "7785": 2.44452,
+ "7790": 2.48343,
+ "7795": 2.44094,
+ "7800": 2.46248,
+ "7805": 2.48205,
+ "7810": 2.50456,
+ "7815": 2.48933,
+ "7820": 2.44792,
+ "7825": 2.51403,
+ "7830": 2.45315,
+ "7835": 2.46857,
+ "7840": 2.47863,
+ "7845": 2.45925,
+ "7850": 2.41753,
+ "7855": 2.47259,
+ "7860": 2.49941,
+ "7865": 2.42445,
+ "7870": 2.46877,
+ "7875": 2.44758,
+ "7880": 2.45498,
+ "7885": 2.46271,
+ "7890": 2.4704,
+ "7895": 2.44746,
+ "7900": 2.43849,
+ "7905": 2.43611,
+ "7910": 2.42508,
+ "7915": 2.48223,
+ "7920": 2.47586,
+ "7925": 2.42204,
+ "7930": 2.47227,
+ "7935": 2.45144,
+ "7940": 2.42164,
+ "7945": 2.46858,
+ "7950": 2.44519,
+ "7955": 2.41938,
+ "7960": 2.48925,
+ "7965": 2.51812,
+ "7970": 2.52365,
+ "7975": 2.45089,
+ "7980": 2.44183,
+ "7985": 2.46779,
+ "7990": 2.43205,
+ "7995": 2.47124,
+ "8000": 2.43695,
+ "8005": 2.41847,
+ "8010": 2.45827,
+ "8015": 2.47003,
+ "8020": 2.48312,
+ "8025": 2.4743,
+ "8030": 2.45211,
+ "8035": 2.47209,
+ "8040": 2.42113,
+ "8045": 2.47922,
+ "8050": 2.45648,
+ "8055": 2.42799,
+ "8060": 2.44378,
+ "8065": 2.46133,
+ "8070": 2.45509,
+ "8075": 2.45782,
+ "8080": 2.44539,
+ "8085": 2.44103,
+ "8090": 2.42385,
+ "8095": 2.42411,
+ "8100": 2.4383,
+ "8105": 2.49409,
+ "8110": 2.43711,
+ "8115": 2.44483,
+ "8120": 2.46723,
+ "8125": 2.46835,
+ "8130": 2.45534,
+ "8135": 2.453,
+ "8140": 2.43964,
+ "8145": 2.42681,
+ "8150": 2.42282,
+ "8155": 2.48725,
+ "8160": 2.45564,
+ "8165": 2.44353,
+ "8170": 2.43601,
+ "8175": 2.42347,
+ "8180": 2.49602,
+ "8185": 2.4266,
+ "8190": 2.4682,
+ "8195": 2.45714,
+ "8200": 2.44755,
+ "8205": 2.44698,
+ "8210": 2.43158,
+ "8215": 2.44097,
+ "8220": 2.43711,
+ "8225": 2.41111,
+ "8230": 2.43991,
+ "8235": 2.46523,
+ "8240": 2.42792,
+ "8245": 2.44877,
+ "8250": 2.4449,
+ "8255": 2.43746,
+ "8260": 2.4329,
+ "8265": 2.42841,
+ "8270": 2.43338,
+ "8275": 2.44139,
+ "8280": 2.39763,
+ "8285": 2.43908,
+ "8290": 2.48053,
+ "8295": 2.44765,
+ "8300": 2.45824,
+ "8305": 2.40708,
+ "8310": 2.43736,
+ "8315": 2.45487,
+ "8320": 2.39821,
+ "8325": 2.39302,
+ "8330": 2.43427,
+ "8335": 2.44307,
+ "8340": 2.48835,
+ "8345": 2.44982,
+ "8350": 2.4483,
+ "8355": 2.40703,
+ "8360": 2.39899,
+ "8365": 2.45388,
+ "8370": 2.4522,
+ "8375": 2.42571,
+ "8380": 2.41956,
+ "8385": 2.42303,
+ "8390": 2.43482,
+ "8395": 2.43963,
+ "8400": 2.4396,
+ "8405": 2.48867,
+ "8410": 2.43761,
+ "8415": 2.43376,
+ "8420": 2.41526,
+ "8425": 2.43843,
+ "8430": 2.46211,
+ "8435": 2.40284,
+ "8440": 2.45067,
+ "8445": 2.45871,
+ "8450": 2.40746,
+ "8455": 2.45971,
+ "8460": 2.4544,
+ "8465": 2.4362,
+ "8470": 2.40884,
+ "8475": 2.47657,
+ "8480": 2.40164,
+ "8485": 2.41272,
+ "8490": 2.46525,
+ "8495": 2.43663,
+ "8500": 2.44487,
+ "8505": 2.40411,
+ "8510": 2.40402,
+ "8515": 2.42904,
+ "8520": 2.42561,
+ "8525": 2.49154,
+ "8530": 2.37412,
+ "8535": 2.4007,
+ "8540": 2.48529,
+ "8545": 2.38045,
+ "8550": 2.43845,
+ "8555": 2.45138,
+ "8560": 2.46969,
+ "8565": 2.42022,
+ "8570": 2.43069,
+ "8575": 2.44782,
+ "8580": 2.44057,
+ "8585": 2.4206,
+ "8590": 2.40434,
+ "8595": 2.42633,
+ "8600": 2.41208,
+ "8605": 2.49153,
+ "8610": 2.4194,
+ "8615": 2.38712,
+ "8620": 2.44807,
+ "8625": 2.42543,
+ "8630": 2.45639,
+ "8635": 2.44968,
+ "8640": 2.43446,
+ "8645": 2.47511,
+ "8650": 2.42064,
+ "8655": 2.45306,
+ "8660": 2.45562,
+ "8665": 2.38502,
+ "8670": 2.40915,
+ "8675": 2.42743,
+ "8680": 2.44652,
+ "8685": 2.42977,
+ "8690": 2.40998,
+ "8695": 2.44244,
+ "8700": 2.43397,
+ "8705": 2.41831,
+ "8710": 2.42796,
+ "8715": 2.44873,
+ "8720": 2.4762,
+ "8725": 2.41193,
+ "8730": 2.3907,
+ "8735": 2.43509,
+ "8740": 2.43059,
+ "8745": 2.3966,
+ "8750": 2.43639,
+ "8755": 2.42428,
+ "8760": 2.4016,
+ "8765": 2.43375,
+ "8770": 2.40409,
+ "8775": 2.43681,
+ "8780": 2.41926,
+ "8785": 2.47235,
+ "8790": 2.42096,
+ "8795": 2.41752,
+ "8800": 2.41464,
+ "8805": 2.4061,
+ "8810": 2.4098,
+ "8815": 2.47301,
+ "8820": 2.45432,
+ "8825": 2.42408,
+ "8830": 2.38686,
+ "8835": 2.42272,
+ "8840": 2.39428,
+ "8845": 2.42523,
+ "8850": 2.43381,
+ "8855": 2.40267,
+ "8860": 2.42873,
+ "8865": 2.42868,
+ "8870": 2.43634,
+ "8875": 2.43759,
+ "8880": 2.41231,
+ "8885": 2.3937,
+ "8890": 2.4451,
+ "8895": 2.42855,
+ "8900": 2.41221,
+ "8905": 2.40437,
+ "8910": 2.40297,
+ "8915": 2.42005,
+ "8920": 2.43194,
+ "8925": 2.46564,
+ "8930": 2.41557,
+ "8935": 2.40817,
+ "8940": 2.38863,
+ "8945": 2.39302,
+ "8950": 2.41651,
+ "8955": 2.39575,
+ "8960": 2.43244,
+ "8965": 2.41533,
+ "8970": 2.4034,
+ "8975": 2.4766,
+ "8980": 2.43987,
+ "8985": 2.37327,
+ "8990": 2.4078,
+ "8995": 2.41701,
+ "9000": 2.45566,
+ "9005": 2.41239,
+ "9010": 2.37445,
+ "9015": 2.40781,
+ "9020": 2.39772,
+ "9025": 2.36899,
+ "9030": 2.40082,
+ "9035": 2.42402,
+ "9040": 2.4202,
+ "9045": 2.41956,
+ "9050": 2.39708,
+ "9055": 2.41909,
+ "9060": 2.41955,
+ "9065": 2.40543,
+ "9070": 2.44567,
+ "9075": 2.39519,
+ "9080": 2.43479,
+ "9085": 2.42815,
+ "9090": 2.41277,
+ "9095": 2.39677,
+ "9100": 2.40146,
+ "9105": 2.35866,
+ "9110": 2.466,
+ "9115": 2.41729,
+ "9120": 2.40494,
+ "9125": 2.45833,
+ "9130": 2.3942,
+ "9135": 2.44764,
+ "9140": 2.43538,
+ "9145": 2.42864,
+ "9150": 2.42436,
+ "9155": 2.37413,
+ "9160": 2.41723,
+ "9165": 2.4237,
+ "9170": 2.37298,
+ "9175": 2.41773,
+ "9180": 2.37794,
+ "9185": 2.43884,
+ "9190": 2.41245,
+ "9195": 2.40368,
+ "9200": 2.39457,
+ "9205": 2.45154,
+ "9210": 2.36217,
+ "9215": 2.46422,
+ "9220": 2.44676,
+ "9225": 2.3842,
+ "9230": 2.44585,
+ "9235": 2.39756,
+ "9240": 2.40256,
+ "9245": 2.43616,
+ "9250": 2.44676,
+ "9255": 2.43332,
+ "9260": 2.38684,
+ "9265": 2.4388,
+ "9270": 2.43616,
+ "9275": 2.39245,
+ "9280": 2.38882,
+ "9285": 2.42149,
+ "9290": 2.40403,
+ "9295": 2.38344,
+ "9300": 2.42258,
+ "9305": 2.40398,
+ "9310": 2.41555,
+ "9315": 2.40892,
+ "9320": 2.44315,
+ "9325": 2.36939,
+ "9330": 2.40243,
+ "9335": 2.36125,
+ "9340": 2.40817,
+ "9345": 2.41574,
+ "9350": 2.44011,
+ "9355": 2.47744,
+ "9360": 2.43769,
+ "9365": 2.38878,
+ "9370": 2.43788,
+ "9375": 2.4353,
+ "9380": 2.35334,
+ "9385": 2.40158,
+ "9390": 2.38263,
+ "9395": 2.38979,
+ "9400": 2.44357,
+ "9405": 2.415,
+ "9410": 2.39793,
+ "9415": 2.4369,
+ "9420": 2.44583,
+ "9425": 2.43136,
+ "9430": 2.4473,
+ "9435": 2.41443,
+ "9440": 2.47941,
+ "9445": 2.37575,
+ "9450": 2.39465,
+ "9455": 2.40401,
+ "9460": 2.38595,
+ "9465": 2.37979,
+ "9470": 2.38285,
+ "9475": 2.36564,
+ "9480": 2.43384,
+ "9485": 2.38592,
+ "9490": 2.42196,
+ "9495": 2.38314,
+ "9500": 2.36654,
+ "9505": 2.43133,
+ "9510": 2.4008,
+ "9515": 2.43151,
+ "9520": 2.41924,
+ "9525": 2.38934,
+ "9530": 2.45494,
+ "9535": 2.40096,
+ "9540": 2.41854,
+ "9545": 2.37751,
+ "9550": 2.42252,
+ "9555": 2.39166,
+ "9560": 2.42327,
+ "9565": 2.40878,
+ "9570": 2.37091,
+ "9575": 2.4107,
+ "9580": 2.39618,
+ "9585": 2.42387,
+ "9590": 2.42987,
+ "9595": 2.44822,
+ "9600": 2.39109,
+ "9605": 2.38373,
+ "9610": 2.42148,
+ "9615": 2.41483,
+ "9620": 2.41408,
+ "9625": 2.4471,
+ "9630": 2.39887,
+ "9635": 2.40319,
+ "9640": 2.44771,
+ "9645": 2.40968,
+ "9650": 2.39895,
+ "9655": 2.37493,
+ "9660": 2.42625,
+ "9665": 2.39151,
+ "9670": 2.38394,
+ "9675": 2.3568,
+ "9680": 2.39733,
+ "9685": 2.39587,
+ "9690": 2.46251,
+ "9695": 2.38057,
+ "9700": 2.37624,
+ "9705": 2.38424,
+ "9710": 2.36525,
+ "9715": 2.38907,
+ "9720": 2.43577,
+ "9725": 2.44198,
+ "9730": 2.42952,
+ "9735": 2.38556,
+ "9740": 2.38107,
+ "9745": 2.42781,
+ "9750": 2.39866,
+ "9755": 2.40833,
+ "9760": 2.41151,
+ "9765": 2.36766,
+ "9770": 2.45373,
+ "9775": 2.40095,
+ "9780": 2.36241,
+ "9785": 2.40262,
+ "9790": 2.40674,
+ "9795": 2.36207,
+ "9800": 2.39716,
+ "9805": 2.40678,
+ "9810": 2.40944,
+ "9815": 2.37834,
+ "9820": 2.37627,
+ "9825": 2.40529,
+ "9830": 2.4226,
+ "9835": 2.38429,
+ "9840": 2.41358,
+ "9845": 2.36216,
+ "9850": 2.39986,
+ "9855": 2.39508,
+ "9860": 2.38986,
+ "9865": 2.3818,
+ "9870": 2.38527,
+ "9875": 2.38223,
+ "9880": 2.45096,
+ "9885": 2.39236,
+ "9890": 2.3539,
+ "9895": 2.31957,
+ "9900": 2.39578,
+ "9905": 2.42362,
+ "9910": 2.35552,
+ "9915": 2.36371,
+ "9920": 2.41083,
+ "9925": 2.39762,
+ "9930": 2.38206,
+ "9935": 2.349,
+ "9940": 2.3851,
+ "9945": 2.37954,
+ "9950": 2.4033,
+ "9955": 2.4499,
+ "9960": 2.43018,
+ "9965": 2.35621,
+ "9970": 2.4091,
+ "9975": 2.38371,
+ "9980": 2.33142,
+ "9985": 2.405,
+ "9990": 2.39633,
+ "9995": 2.39584,
+ "10000": 2.36742,
+ "10005": 2.37207,
+ "10010": 2.38237,
+ "10015": 2.44528,
+ "10020": 2.36262,
+ "10025": 2.38754,
+ "10030": 2.38683,
+ "10035": 2.40902,
+ "10040": 2.40526,
+ "10045": 2.3827,
+ "10050": 2.35168,
+ "10055": 2.3679,
+ "10060": 2.41766,
+ "10065": 2.37238,
+ "10070": 2.4226,
+ "10075": 2.37237,
+ "10080": 2.36326,
+ "10085": 2.36945,
+ "10090": 2.34646,
+ "10095": 2.40112,
+ "10100": 2.31438,
+ "10105": 2.38271,
+ "10110": 2.41097,
+ "10115": 2.38644,
+ "10120": 2.35762,
+ "10125": 2.37234,
+ "10130": 2.35976,
+ "10135": 2.38285,
+ "10140": 2.41246,
+ "10145": 2.407,
+ "10150": 2.37571,
+ "10155": 2.39507,
+ "10160": 2.36146,
+ "10165": 2.38316,
+ "10170": 2.42297,
+ "10175": 2.32401,
+ "10180": 2.39308,
+ "10185": 2.38368,
+ "10190": 2.44327,
+ "10195": 2.40286,
+ "10200": 2.39067,
+ "10205": 2.38654,
+ "10210": 2.36842,
+ "10215": 2.34516,
+ "10220": 2.41929,
+ "10225": 2.43133,
+ "10230": 2.3568,
+ "10235": 2.38752,
+ "10240": 2.3752,
+ "10245": 2.39113,
+ "10250": 2.38864,
+ "10255": 2.41216,
+ "10260": 2.33416,
+ "10265": 2.34887,
+ "10270": 2.35033,
+ "10275": 2.37285,
+ "10280": 2.44994,
+ "10285": 2.35789,
+ "10290": 2.38502,
+ "10295": 2.37373,
+ "10300": 2.36727,
+ "10305": 2.41438,
+ "10310": 2.38817,
+ "10315": 2.36009,
+ "10320": 2.36681,
+ "10325": 2.36022,
+ "10330": 2.41314,
+ "10335": 2.36278,
+ "10340": 2.41978,
+ "10345": 2.36984,
+ "10350": 2.35828,
+ "10355": 2.39688,
+ "10360": 2.37367,
+ "10365": 2.36112,
+ "10370": 2.34104,
+ "10375": 2.35717,
+ "10380": 2.41878,
+ "10385": 2.4051,
+ "10390": 2.38062,
+ "10395": 2.35893,
+ "10400": 2.3779,
+ "10405": 2.35068,
+ "10410": 2.34134,
+ "10415": 2.41698,
+ "10420": 2.37934,
+ "10425": 2.32763,
+ "10430": 2.35959,
+ "10435": 2.37178,
+ "10440": 2.37103,
+ "10445": 2.36068,
+ "10450": 2.36055,
+ "10455": 2.37973,
+ "10460": 2.3816,
+ "10465": 2.30311,
+ "10470": 2.35728,
+ "10475": 2.38121,
+ "10480": 2.36377,
+ "10485": 2.36123,
+ "10490": 2.41286,
+ "10495": 2.36732,
+ "10500": 2.36292,
+ "10505": 2.36988,
+ "10510": 2.38283,
+ "10515": 2.37416,
+ "10520": 2.40348,
+ "10525": 2.39031,
+ "10530": 2.39169,
+ "10535": 2.3553,
+ "10540": 2.40457,
+ "10545": 2.35849,
+ "10550": 2.37876,
+ "10555": 2.35964,
+ "10560": 2.34048,
+ "10565": 2.37134,
+ "10570": 2.37385,
+ "10575": 2.35597,
+ "10580": 2.37862,
+ "10585": 2.36882,
+ "10590": 2.38027,
+ "10595": 2.37758,
+ "10600": 2.33268,
+ "10605": 2.37212,
+ "10610": 2.36594,
+ "10615": 2.36345,
+ "10620": 2.34684,
+ "10625": 2.41919,
+ "10630": 2.37018,
+ "10635": 2.32323,
+ "10640": 2.3649,
+ "10645": 2.42247,
+ "10650": 2.36086,
+ "10655": 2.30764,
+ "10660": 2.348,
+ "10665": 2.39867,
+ "10670": 2.31573,
+ "10675": 2.41654,
+ "10680": 2.35743,
+ "10685": 2.28909,
+ "10690": 2.38576,
+ "10695": 2.33077,
+ "10700": 2.38759,
+ "10705": 2.3863,
+ "10710": 2.34409,
+ "10715": 2.38283,
+ "10720": 2.32748,
+ "10725": 2.35498,
+ "10730": 2.35039,
+ "10735": 2.35471,
+ "10740": 2.31827,
+ "10745": 2.34059,
+ "10750": 2.33574,
+ "10755": 2.40551,
+ "10760": 2.36631,
+ "10765": 2.33806,
+ "10770": 2.37451,
+ "10775": 2.38596,
+ "10780": 2.37059,
+ "10785": 2.39212,
+ "10790": 2.35117,
+ "10795": 2.3904,
+ "10800": 2.32374,
+ "10805": 2.39726,
+ "10810": 2.37614,
+ "10815": 2.3553,
+ "10820": 2.34478,
+ "10825": 2.37237,
+ "10830": 2.33964,
+ "10835": 2.3485,
+ "10840": 2.33094,
+ "10845": 2.38895,
+ "10850": 2.33302,
+ "10855": 2.36518,
+ "10860": 2.33262,
+ "10865": 2.32199,
+ "10870": 2.32482,
+ "10875": 2.30493,
+ "10880": 2.39438,
+ "10885": 2.40446,
+ "10890": 2.36207,
+ "10895": 2.37301,
+ "10900": 2.33379,
+ "10905": 2.31397,
+ "10910": 2.40899,
+ "10915": 2.37243,
+ "10920": 2.3758,
+ "10925": 2.36623,
+ "10930": 2.32019,
+ "10935": 2.36161,
+ "10940": 2.35709,
+ "10945": 2.34878,
+ "10950": 2.36962,
+ "10955": 2.36398,
+ "10960": 2.31397,
+ "10965": 2.36562,
+ "10970": 2.3583,
+ "10975": 2.40985,
+ "10980": 2.3781,
+ "10985": 2.34498,
+ "10990": 2.40001,
+ "10995": 2.3672,
+ "11000": 2.33893,
+ "11005": 2.36245,
+ "11010": 2.34836,
+ "11015": 2.32906,
+ "11020": 2.33621,
+ "11025": 2.36933,
+ "11030": 2.34186,
+ "11035": 2.31576,
+ "11040": 2.31933,
+ "11045": 2.31919,
+ "11050": 2.31897,
+ "11055": 2.29051,
+ "11060": 2.33978,
+ "11065": 2.3119,
+ "11070": 2.39842,
+ "11075": 2.32101,
+ "11080": 2.35606,
+ "11085": 2.34028,
+ "11090": 2.34747,
+ "11095": 2.37211,
+ "11100": 2.32957,
+ "11105": 2.31737,
+ "11110": 2.36384,
+ "11115": 2.37329,
+ "11120": 2.38274,
+ "11125": 2.31626,
+ "11130": 2.35197,
+ "11135": 2.33464,
+ "11140": 2.37551,
+ "11145": 2.35157,
+ "11150": 2.39785,
+ "11155": 2.34206,
+ "11160": 2.36772,
+ "11165": 2.36464,
+ "11170": 2.34238,
+ "11175": 2.33468,
+ "11180": 2.37671,
+ "11185": 2.31336,
+ "11190": 2.28079,
+ "11195": 2.33083,
+ "11200": 2.34705,
+ "11205": 2.36331,
+ "11210": 2.33356,
+ "11215": 2.31979,
+ "11220": 2.3448,
+ "11225": 2.37516,
+ "11230": 2.36812,
+ "11235": 2.32151,
+ "11240": 2.34463,
+ "11245": 2.35847,
+ "11250": 2.33422,
+ "11255": 2.33794,
+ "11260": 2.35873,
+ "11265": 2.39072,
+ "11270": 2.2892,
+ "11275": 2.3165,
+ "11280": 2.3704,
+ "11285": 2.32406,
+ "11290": 2.34783,
+ "11295": 2.36604,
+ "11300": 2.38247,
+ "11305": 2.33566,
+ "11310": 2.33149,
+ "11315": 2.30059,
+ "11320": 2.30617,
+ "11325": 2.31599,
+ "11330": 2.35495,
+ "11335": 2.34008,
+ "11340": 2.30955,
+ "11345": 2.31469,
+ "11350": 2.29656,
+ "11355": 2.32534,
+ "11360": 2.35152,
+ "11365": 2.29442,
+ "11370": 2.35416,
+ "11375": 2.32906,
+ "11380": 2.34243,
+ "11385": 2.34892,
+ "11390": 2.33603,
+ "11395": 2.28851,
+ "11400": 2.3102,
+ "11405": 2.35523,
+ "11410": 2.35698,
+ "11415": 2.38675,
+ "11420": 2.35342,
+ "11425": 2.31015,
+ "11430": 2.37109,
+ "11435": 2.36371,
+ "11440": 2.35039,
+ "11445": 2.36451,
+ "11450": 2.32322,
+ "11455": 2.30546,
+ "11460": 2.35207,
+ "11465": 2.3442,
+ "11470": 2.37537,
+ "11475": 2.31381,
+ "11480": 2.32586,
+ "11485": 2.31116,
+ "11490": 2.34703,
+ "11495": 2.40854,
+ "11500": 2.34205,
+ "11505": 2.3525,
+ "11510": 2.3652,
+ "11515": 2.32359,
+ "11520": 2.30633,
+ "11525": 2.36165,
+ "11530": 2.31593,
+ "11535": 2.32316,
+ "11540": 2.34794,
+ "11545": 2.34435,
+ "11550": 2.36612,
+ "11555": 2.32736,
+ "11560": 2.3511,
+ "11565": 2.34131,
+ "11570": 2.35152,
+ "11575": 2.29761,
+ "11580": 2.3296,
+ "11585": 2.35345,
+ "11590": 2.36481,
+ "11595": 2.33643,
+ "11600": 2.35888,
+ "11605": 2.3235,
+ "11610": 2.3627,
+ "11615": 2.3609,
+ "11620": 2.29636,
+ "11625": 2.2769,
+ "11630": 2.32824,
+ "11635": 2.34398,
+ "11640": 2.30587,
+ "11645": 2.30877,
+ "11650": 2.32859,
+ "11655": 2.35253,
+ "11660": 2.33697,
+ "11665": 2.33118,
+ "11670": 2.30107,
+ "11675": 2.29905,
+ "11680": 2.32579,
+ "11685": 2.3365,
+ "11690": 2.34545,
+ "11695": 2.3185,
+ "11700": 2.32594,
+ "11705": 2.30312,
+ "11710": 2.34508,
+ "11715": 2.3159,
+ "11720": 2.30013,
+ "11725": 2.34135,
+ "11730": 2.30673,
+ "11735": 2.32959,
+ "11740": 2.27327,
+ "11745": 2.31924,
+ "11750": 2.32847,
+ "11755": 2.35343,
+ "11760": 2.31323,
+ "11765": 2.33591,
+ "11770": 2.27588,
+ "11775": 2.32611,
+ "11780": 2.2558,
+ "11785": 2.29841,
+ "11790": 2.31362,
+ "11795": 2.32061,
+ "11800": 2.33531,
+ "11805": 2.30389,
+ "11810": 2.30648,
+ "11815": 2.33051,
+ "11820": 2.32042,
+ "11825": 2.36169,
+ "11830": 2.31796,
+ "11835": 2.33908,
+ "11840": 2.34234,
+ "11845": 2.31871,
+ "11850": 2.30518,
+ "11855": 2.31532,
+ "11860": 2.34365,
+ "11865": 2.35918,
+ "11870": 2.37963,
+ "11875": 2.28094,
+ "11880": 2.29379,
+ "11885": 2.33757,
+ "11890": 2.29331,
+ "11895": 2.29026,
+ "11900": 2.33451,
+ "11905": 2.31957,
+ "11910": 2.27792,
+ "11915": 2.31251,
+ "11920": 2.33527,
+ "11925": 2.30383,
+ "11930": 2.30797,
+ "11935": 2.31763,
+ "11940": 2.31918,
+ "11945": 2.34259,
+ "11950": 2.3003,
+ "11955": 2.31499,
+ "11960": 2.339,
+ "11965": 2.29617,
+ "11970": 2.28244,
+ "11975": 2.33742,
+ "11980": 2.307,
+ "11985": 2.27878,
+ "11990": 2.3052,
+ "11995": 2.3307,
+ "12000": 2.3261,
+ "12005": 2.32673,
+ "12010": 2.28887,
+ "12015": 2.31036,
+ "12020": 2.32835,
+ "12025": 2.3354,
+ "12030": 2.31192,
+ "12035": 2.336,
+ "12040": 2.31487,
+ "12045": 2.31196,
+ "12050": 2.30879,
+ "12055": 2.33271,
+ "12060": 2.29798,
+ "12065": 2.32949,
+ "12070": 2.30235,
+ "12075": 2.27654,
+ "12080": 2.3519,
+ "12085": 2.33956,
+ "12090": 2.33249,
+ "12095": 2.28215,
+ "12100": 2.31668,
+ "12105": 2.31,
+ "12110": 2.32992,
+ "12115": 2.30644,
+ "12120": 2.30659,
+ "12125": 2.2963,
+ "12130": 2.30302,
+ "12135": 2.32825,
+ "12140": 2.29602,
+ "12145": 2.25652,
+ "12150": 2.2603,
+ "12155": 2.34303,
+ "12160": 2.35823,
+ "12165": 2.31986,
+ "12170": 2.33446,
+ "12175": 2.34322,
+ "12180": 2.33054,
+ "12185": 2.34225,
+ "12190": 2.3349,
+ "12195": 2.29838,
+ "12200": 2.30196,
+ "12205": 2.32374,
+ "12210": 2.35625,
+ "12215": 2.3039,
+ "12220": 2.3005,
+ "12225": 2.24425,
+ "12230": 2.33456,
+ "12235": 2.33882,
+ "12240": 2.32397,
+ "12245": 2.28614,
+ "12250": 2.27273,
+ "12255": 2.33512,
+ "12260": 2.31251,
+ "12265": 2.34116,
+ "12270": 2.31202,
+ "12275": 2.3128,
+ "12280": 2.31892,
+ "12285": 2.2861,
+ "12290": 2.31089,
+ "12295": 2.26519,
+ "12300": 2.32882,
+ "12305": 2.268,
+ "12310": 2.28705,
+ "12315": 2.3539,
+ "12320": 2.29568,
+ "12325": 2.32004,
+ "12330": 2.2996,
+ "12335": 2.31922,
+ "12340": 2.33993,
+ "12345": 2.36806,
+ "12350": 2.34457,
+ "12355": 2.30697,
+ "12360": 2.31406,
+ "12365": 2.33109,
+ "12370": 2.29205,
+ "12375": 2.30035,
+ "12380": 2.29175,
+ "12385": 2.29129,
+ "12390": 2.25008,
+ "12395": 2.30515,
+ "12400": 2.29975,
+ "12405": 2.31139,
+ "12410": 2.30433,
+ "12415": 2.28207,
+ "12420": 2.31827,
+ "12425": 2.30059,
+ "12430": 2.31595,
+ "12435": 2.30009,
+ "12440": 2.33225,
+ "12445": 2.32061,
+ "12450": 2.30677,
+ "12455": 2.24051,
+ "12460": 2.33765,
+ "12465": 2.36395,
+ "12470": 2.27666,
+ "12475": 2.27404,
+ "12480": 2.2927,
+ "12485": 2.30694,
+ "12490": 2.33069,
+ "12495": 2.26943,
+ "12500": 2.32076,
+ "12505": 2.3358,
+ "12510": 2.35633,
+ "12515": 2.26977,
+ "12520": 2.31981,
+ "12525": 2.28667,
+ "12530": 2.3207,
+ "12535": 2.27236,
+ "12540": 2.28479,
+ "12545": 2.29083,
+ "12550": 2.31729,
+ "12555": 2.32436,
+ "12560": 2.30126,
+ "12565": 2.33569,
+ "12570": 2.28016,
+ "12575": 2.30108,
+ "12580": 2.31066,
+ "12585": 2.29232,
+ "12590": 2.33444,
+ "12595": 2.3246,
+ "12600": 2.28237,
+ "12605": 2.31981,
+ "12610": 2.36453,
+ "12615": 2.30709,
+ "12620": 2.33321,
+ "12625": 2.33082,
+ "12630": 2.29784,
+ "12635": 2.33603,
+ "12640": 2.29527,
+ "12645": 2.28038,
+ "12650": 2.32632,
+ "12655": 2.26534,
+ "12660": 2.34185,
+ "12665": 2.3183,
+ "12670": 2.31281,
+ "12675": 2.3199,
+ "12680": 2.275,
+ "12685": 2.3665,
+ "12690": 2.30372,
+ "12695": 2.33211,
+ "12700": 2.29318,
+ "12705": 2.30709,
+ "12710": 2.30722,
+ "12715": 2.28764,
+ "12720": "nan",
+ "12725": "nan",
+ "12730": "nan",
+ "12735": "nan",
+ "12740": "nan",
+ "12745": "nan",
+ "12750": "nan",
+ "12755": "nan",
+ "12760": "nan",
+ "12765": "nan",
+ "12770": "nan",
+ "12775": "nan",
+ "12780": "nan",
+ "12785": "nan",
+ "12790": "nan",
+ "12795": "nan",
+ "12800": "nan",
+ "12805": "nan",
+ "12810": "nan",
+ "12815": "nan",
+ "12820": "nan",
+ "12825": "nan",
+ "12830": "nan",
+ "12835": "nan",
+ "12840": "nan",
+ "12845": "nan",
+ "12850": "nan",
+ "12855": "nan",
+ "12860": "nan",
+ "12865": "nan",
+ "12870": "nan",
+ "12875": "nan",
+ "12880": "nan",
+ "12885": "nan",
+ "12890": "nan",
+ "12895": "nan",
+ "12900": "nan",
+ "12905": "nan",
+ "12910": "nan",
+ "12915": "nan",
+ "12920": "nan",
+ "12925": "nan",
+ "12930": "nan",
+ "12935": "nan",
+ "12940": "nan",
+ "12945": "nan",
+ "12950": "nan",
+ "12955": "nan",
+ "12960": "nan",
+ "12965": "nan",
+ "12970": "nan",
+ "12975": "nan",
+ "12980": "nan",
+ "12985": "nan",
+ "12990": "nan",
+ "12995": "nan",
+ "13000": "nan"
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": 956237760.0,
+ "5": 967339648.0,
+ "10": 971387456.0,
+ "15": 946482432.0,
+ "20": 960006656.0,
+ "25": 1076542208.0,
+ "30": 1190547584.0,
+ "35": 1270033664.0,
+ "40": 1234848000.0,
+ "45": 1146008832.0,
+ "50": 1105336704.0,
+ "55": 1058935168.0,
+ "60": 1024844352.0,
+ "65": 1009768384.0,
+ "70": 987690240.0,
+ "75": 988810304.0,
+ "80": 1012225024.0,
+ "85": 1010933120.0,
+ "90": 989566016.0,
+ "95": 961647488.0,
+ "100": 975055936.0,
+ "105": 981783040.0,
+ "110": 979816448.0,
+ "115": 977699648.0,
+ "120": 961670144.0,
+ "125": 943075072.0,
+ "130": 975285440.0,
+ "135": 964823040.0,
+ "140": 963104128.0,
+ "145": 976365952.0,
+ "150": 921578432.0,
+ "155": 968343744.0,
+ "160": 956751936.0,
+ "165": 959846848.0,
+ "170": 974339456.0,
+ "175": 949012608.0,
+ "180": 946737088.0,
+ "185": 972001408.0,
+ "190": 969072000.0,
+ "195": 985125248.0,
+ "200": 945786816.0,
+ "205": 958356736.0,
+ "210": 979441280.0,
+ "215": 967494208.0,
+ "220": 956434816.0,
+ "225": 962397632.0,
+ "230": 948169600.0,
+ "235": 965207168.0,
+ "240": 966058176.0,
+ "245": 969162752.0,
+ "250": 974444672.0,
+ "255": 925064192.0,
+ "260": 965640832.0,
+ "265": 970673600.0,
+ "270": 959140224.0,
+ "275": 954000256.0,
+ "280": 963433792.0,
+ "285": 945782912.0,
+ "290": 974115392.0,
+ "295": 966706496.0,
+ "300": 967148032.0,
+ "305": 964512000.0,
+ "310": 940358208.0,
+ "315": 967391168.0,
+ "320": 969008512.0,
+ "325": 980559872.0,
+ "330": 972096384.0,
+ "335": 946862720.0,
+ "340": 966597184.0,
+ "345": 973025408.0,
+ "350": 973919488.0,
+ "355": 963264512.0,
+ "360": 948352576.0,
+ "365": 964823104.0,
+ "370": 962953728.0,
+ "375": 958455040.0,
+ "380": 947150976.0,
+ "385": 955996928.0,
+ "390": 945403264.0,
+ "395": 970420608.0,
+ "400": 979779712.0,
+ "405": 968353664.0,
+ "410": 970074112.0,
+ "415": 953158144.0,
+ "420": 943571904.0,
+ "425": 954780736.0,
+ "430": 962666816.0,
+ "435": 977082496.0,
+ "440": 954811328.0,
+ "445": 971893696.0,
+ "450": 963510208.0,
+ "455": 973132288.0,
+ "460": 983712448.0,
+ "465": 945280448.0,
+ "470": 942057216.0,
+ "475": 967006592.0,
+ "480": 966105088.0,
+ "485": 976415616.0,
+ "490": 962534848.0,
+ "495": 945465216.0,
+ "500": 964458880.0,
+ "505": 986012672.0,
+ "510": 965685888.0,
+ "515": 943412224.0,
+ "520": 945015360.0,
+ "525": 971263040.0,
+ "530": 971893056.0,
+ "535": 979142848.0,
+ "540": 969537664.0,
+ "545": 954128640.0,
+ "550": 951271424.0,
+ "555": 987225408.0,
+ "560": 960430848.0,
+ "565": 966615360.0,
+ "570": 975733120.0,
+ "575": 927225472.0,
+ "580": 970700032.0,
+ "585": 961176896.0,
+ "590": 972961408.0,
+ "595": 963684608.0,
+ "600": 937080000.0,
+ "605": 951480064.0,
+ "610": 963364224.0,
+ "615": 970012416.0,
+ "620": 976467264.0,
+ "625": 949584192.0,
+ "630": 954445952.0,
+ "635": 986042496.0,
+ "640": 980976832.0,
+ "645": 955012608.0,
+ "650": 958549760.0,
+ "655": 951650048.0,
+ "660": 961041984.0,
+ "665": 967554368.0,
+ "670": 962514432.0,
+ "675": 968335808.0,
+ "680": 965619904.0,
+ "685": 962866496.0,
+ "690": 961917952.0,
+ "695": 954772480.0,
+ "700": 970340928.0,
+ "705": 945515968.0,
+ "710": 943884416.0,
+ "715": 973357632.0,
+ "720": 968367232.0,
+ "725": 978492544.0,
+ "730": 952195904.0,
+ "735": 948814720.0,
+ "740": 955634432.0,
+ "745": 975869376.0,
+ "750": 981239232.0,
+ "755": 962157824.0,
+ "760": 951961152.0,
+ "765": 967343296.0,
+ "770": 976150784.0,
+ "775": 970550080.0,
+ "780": 977540480.0,
+ "785": 931531136.0,
+ "790": 960443264.0,
+ "795": 964584768.0,
+ "800": 967026688.0,
+ "805": 962321216.0,
+ "810": 940975872.0,
+ "815": 949036416.0,
+ "820": 953185792.0,
+ "825": 954505664.0,
+ "830": 976445888.0,
+ "835": 956076032.0,
+ "840": 948407360.0,
+ "845": 965153408.0,
+ "850": 966027904.0,
+ "855": 960903808.0,
+ "860": 976024384.0,
+ "865": 938161280.0,
+ "870": 966416896.0,
+ "875": 972313728.0,
+ "880": 963123712.0,
+ "885": 967747712.0,
+ "890": 949970752.0,
+ "895": 960021504.0,
+ "900": 974229184.0,
+ "905": 963966528.0,
+ "910": 958435776.0,
+ "915": 956355328.0,
+ "920": 943975104.0,
+ "925": 960835136.0,
+ "930": 978849408.0,
+ "935": 971071936.0,
+ "940": 960909504.0,
+ "945": 945066880.0,
+ "950": 957427712.0,
+ "955": 979038848.0,
+ "960": 983588864.0,
+ "965": 966168320.0,
+ "970": 951230336.0,
+ "975": 961575680.0,
+ "980": 968069120.0,
+ "985": 968990784.0,
+ "990": 984392960.0,
+ "995": 953292288.0,
+ "1000": 934782656.0,
+ "1005": 960147584.0,
+ "1010": 971538304.0,
+ "1015": 985188224.0,
+ "1020": 962782144.0,
+ "1025": 935007616.0,
+ "1030": 974681024.0,
+ "1035": 964994304.0,
+ "1040": 980466944.0,
+ "1045": 960830592.0,
+ "1050": 955201664.0,
+ "1055": 957781376.0,
+ "1060": 967747584.0,
+ "1065": 967118080.0,
+ "1070": 966603520.0,
+ "1075": 950061440.0,
+ "1080": 954509312.0,
+ "1085": 967249408.0,
+ "1090": 977134912.0,
+ "1095": 961238080.0,
+ "1100": 979613184.0,
+ "1105": 953366336.0,
+ "1110": 965955456.0,
+ "1115": 966984256.0,
+ "1120": 970352576.0,
+ "1125": 965709056.0,
+ "1130": 954945792.0,
+ "1135": 965843264.0,
+ "1140": 965177344.0,
+ "1145": 970987328.0,
+ "1150": 955554752.0,
+ "1155": 930579584.0,
+ "1160": 957776000.0,
+ "1165": 978125120.0,
+ "1170": 974299968.0,
+ "1175": 973059200.0,
+ "1180": 973082688.0,
+ "1185": 947341888.0,
+ "1190": 964790912.0,
+ "1195": 953137024.0,
+ "1200": 972845248.0,
+ "1205": 988480128.0,
+ "1210": 931125888.0,
+ "1215": 968647168.0,
+ "1220": 969161216.0,
+ "1225": 975952320.0,
+ "1230": 967334016.0,
+ "1235": 943447296.0,
+ "1240": 955856384.0,
+ "1245": 981504448.0,
+ "1250": 966112256.0,
+ "1255": 973670208.0,
+ "1260": 946496640.0,
+ "1265": 963999616.0,
+ "1270": 960487744.0,
+ "1275": 973616640.0,
+ "1280": 961113664.0,
+ "1285": 957581760.0,
+ "1290": 952528704.0,
+ "1295": 971614912.0,
+ "1300": 968864896.0,
+ "1305": 963739712.0,
+ "1310": 963335552.0,
+ "1315": 943555456.0,
+ "1320": 966308096.0,
+ "1325": 989786880.0,
+ "1330": 969508480.0,
+ "1335": 972302080.0,
+ "1340": 972270272.0,
+ "1345": 960656896.0,
+ "1350": 968636672.0,
+ "1355": 955852544.0,
+ "1360": 971822400.0,
+ "1365": 960385920.0,
+ "1370": 948788736.0,
+ "1375": 973532544.0,
+ "1380": 953470208.0,
+ "1385": 969149184.0,
+ "1390": 975719552.0,
+ "1395": 931676800.0,
+ "1400": 945855296.0,
+ "1405": 976752896.0,
+ "1410": 974510528.0,
+ "1415": 967572608.0,
+ "1420": 966746304.0,
+ "1425": 937378048.0,
+ "1430": 973914368.0,
+ "1435": 978334336.0,
+ "1440": 964180992.0,
+ "1445": 958057600.0,
+ "1450": 946145984.0,
+ "1455": 983924736.0,
+ "1460": 968651840.0,
+ "1465": 948746880.0,
+ "1470": 984245632.0,
+ "1475": 943905664.0,
+ "1480": 963976192.0,
+ "1485": 957350016.0,
+ "1490": 961261824.0,
+ "1495": 980541184.0,
+ "1500": 958332224.0,
+ "1505": 942866816.0,
+ "1510": 984179456.0,
+ "1515": 959092992.0,
+ "1520": 959106304.0,
+ "1525": 952786752.0,
+ "1530": 957745472.0,
+ "1535": 949430784.0,
+ "1540": 971086464.0,
+ "1545": 963132736.0,
+ "1550": 978667392.0,
+ "1555": 952319680.0,
+ "1560": 980089024.0,
+ "1565": 967317120.0,
+ "1570": 973840704.0,
+ "1575": 975495680.0,
+ "1580": 941863424.0,
+ "1585": 970031104.0,
+ "1590": 983821568.0,
+ "1595": 948632576.0,
+ "1600": 967443968.0,
+ "1605": 952444544.0,
+ "1610": 969617536.0,
+ "1615": 983148800.0,
+ "1620": 968018688.0,
+ "1625": 970716288.0,
+ "1630": 962888192.0,
+ "1635": 942311616.0,
+ "1640": 981611968.0,
+ "1645": 973976960.0,
+ "1650": 974184960.0,
+ "1655": 967265728.0,
+ "1660": 940687936.0,
+ "1665": 961705216.0,
+ "1670": 962902528.0,
+ "1675": 971279360.0,
+ "1680": 980877696.0,
+ "1685": 944417856.0,
+ "1690": 964687488.0,
+ "1695": 965642752.0,
+ "1700": 966344896.0,
+ "1705": 985201280.0,
+ "1710": 978355328.0,
+ "1715": 943212288.0,
+ "1720": 977087360.0,
+ "1725": 965872832.0,
+ "1730": 968971328.0,
+ "1735": 965087168.0,
+ "1740": 949714688.0,
+ "1745": 970011392.0,
+ "1750": 959684352.0,
+ "1755": 960088896.0,
+ "1760": 966381184.0,
+ "1765": 951817856.0,
+ "1770": 954665024.0,
+ "1775": 973751808.0,
+ "1780": 970535104.0,
+ "1785": 968826944.0,
+ "1790": 950233984.0,
+ "1795": 945131712.0,
+ "1800": 984665088.0,
+ "1805": 987162880.0,
+ "1810": 977768192.0,
+ "1815": 948005760.0,
+ "1820": 949209088.0,
+ "1825": 978853760.0,
+ "1830": 966363200.0,
+ "1835": 964133760.0,
+ "1840": 972319232.0,
+ "1845": 935413504.0,
+ "1850": 952499072.0,
+ "1855": 980048576.0,
+ "1860": 975868864.0,
+ "1865": 958965376.0,
+ "1870": 958948224.0,
+ "1875": 932594304.0,
+ "1880": 973574208.0,
+ "1885": 978846784.0,
+ "1890": 971359808.0,
+ "1895": 959212224.0,
+ "1900": 947392256.0,
+ "1905": 981830016.0,
+ "1910": 969123136.0,
+ "1915": 970039744.0,
+ "1920": 975599104.0,
+ "1925": 960493312.0,
+ "1930": 977924736.0,
+ "1935": 963249792.0,
+ "1940": 952459776.0,
+ "1945": 981338048.0,
+ "1950": 939170368.0,
+ "1955": 960602880.0,
+ "1960": 970031808.0,
+ "1965": 981175168.0,
+ "1970": 962046208.0,
+ "1975": 952821760.0,
+ "1980": 936848960.0,
+ "1985": 975939456.0,
+ "1990": 965967104.0,
+ "1995": 962611712.0,
+ "2000": 960555584.0,
+ "2005": 954499392.0,
+ "2010": 975580672.0,
+ "2015": 991801600.0,
+ "2020": 975435264.0,
+ "2025": 974304768.0,
+ "2030": 952085696.0,
+ "2035": 967849344.0,
+ "2040": 987458752.0,
+ "2045": 976478912.0,
+ "2050": 984702464.0,
+ "2055": 942838592.0,
+ "2060": 942596352.0,
+ "2065": 966210752.0,
+ "2070": 969622464.0,
+ "2075": 980553600.0,
+ "2080": 977598720.0,
+ "2085": 939639168.0,
+ "2090": 969874176.0,
+ "2095": 961276288.0,
+ "2100": 976718208.0,
+ "2105": 972536576.0,
+ "2110": 959902720.0,
+ "2115": 956877568.0,
+ "2120": 977482432.0,
+ "2125": 962568064.0,
+ "2130": 979619456.0,
+ "2135": 950537152.0,
+ "2140": 946995328.0,
+ "2145": 962277376.0,
+ "2150": 973403072.0,
+ "2155": 972689664.0,
+ "2160": 970312512.0,
+ "2165": 948642560.0,
+ "2170": 961545216.0,
+ "2175": 969376384.0,
+ "2180": 969329728.0,
+ "2185": 947448064.0,
+ "2190": 940481664.0,
+ "2195": 986087808.0,
+ "2200": 961862144.0,
+ "2205": 978924864.0,
+ "2210": 964102144.0,
+ "2215": 963500416.0,
+ "2220": 951312128.0,
+ "2225": 969315520.0,
+ "2230": 976330816.0,
+ "2235": 974025728.0,
+ "2240": 975493376.0,
+ "2245": 960229056.0,
+ "2250": 967640704.0,
+ "2255": 969131584.0,
+ "2260": 975063552.0,
+ "2265": 968259840.0,
+ "2270": 951744448.0,
+ "2275": 962767488.0,
+ "2280": 969640000.0,
+ "2285": 971694016.0,
+ "2290": 962890944.0,
+ "2295": 931408448.0,
+ "2300": 959905152.0,
+ "2305": 970426368.0,
+ "2310": 967444352.0,
+ "2315": 970905152.0,
+ "2320": 975589824.0,
+ "2325": 938587264.0,
+ "2330": 988437184.0,
+ "2335": 977489856.0,
+ "2340": 964595584.0,
+ "2345": 964166528.0,
+ "2350": 947556096.0,
+ "2355": 977029056.0,
+ "2360": 966899136.0,
+ "2365": 977294400.0,
+ "2370": 965070848.0,
+ "2375": 953964736.0,
+ "2380": 962918016.0,
+ "2385": 967194880.0,
+ "2390": 963073664.0,
+ "2395": 974466432.0,
+ "2400": 958409984.0,
+ "2405": 968120704.0,
+ "2410": 951585152.0,
+ "2415": 965904832.0,
+ "2420": 966516864.0,
+ "2425": 959046336.0,
+ "2430": 956685696.0,
+ "2435": 961387264.0,
+ "2440": 959755008.0,
+ "2445": 970889216.0,
+ "2450": 961997248.0,
+ "2455": 922721536.0,
+ "2460": 951951360.0,
+ "2465": 955731008.0,
+ "2470": 972571648.0,
+ "2475": 973811776.0,
+ "2480": 943895936.0,
+ "2485": 944186624.0,
+ "2490": 972411328.0,
+ "2495": 974450496.0,
+ "2500": 973908096.0,
+ "2505": 958491648.0,
+ "2510": 939508544.0,
+ "2515": 979552832.0,
+ "2520": 970472960.0,
+ "2525": 964387776.0,
+ "2530": 955798144.0,
+ "2535": 936597568.0,
+ "2540": 969027328.0,
+ "2545": 970384320.0,
+ "2550": 969462912.0,
+ "2555": 969439936.0,
+ "2560": 964979072.0,
+ "2565": 959765312.0,
+ "2570": 985176576.0,
+ "2575": 957425792.0,
+ "2580": 967424000.0,
+ "2585": 966024064.0,
+ "2590": 956356288.0,
+ "2595": 981830144.0,
+ "2600": 959531712.0,
+ "2605": 963001600.0,
+ "2610": 965973504.0,
+ "2615": 951925696.0,
+ "2620": 971242432.0,
+ "2625": 976456064.0,
+ "2630": 974410688.0,
+ "2635": 948071424.0,
+ "2640": 948138368.0,
+ "2645": 963037376.0,
+ "2650": 953982592.0,
+ "2655": 977112768.0,
+ "2660": 949623296.0,
+ "2665": 953930048.0,
+ "2670": 959063680.0,
+ "2675": 979276032.0,
+ "2680": 961397824.0,
+ "2685": 970702272.0,
+ "2690": 965222464.0,
+ "2695": 943554880.0,
+ "2700": 969425152.0,
+ "2705": 978962240.0,
+ "2710": 971811328.0,
+ "2715": 990814528.0,
+ "2720": 942649344.0,
+ "2725": 967956032.0,
+ "2730": 955465088.0,
+ "2735": 970673152.0,
+ "2740": 977920768.0,
+ "2745": 932279232.0,
+ "2750": 947856640.0,
+ "2755": 956317824.0,
+ "2760": 981699840.0,
+ "2765": 966112448.0,
+ "2770": 948914688.0,
+ "2775": 935832064.0,
+ "2780": 964779712.0,
+ "2785": 969568704.0,
+ "2790": 974273408.0,
+ "2795": 966887168.0,
+ "2800": 944389952.0,
+ "2805": 964353920.0,
+ "2810": 969609792.0,
+ "2815": 975845952.0,
+ "2820": 963085568.0,
+ "2825": 937630144.0,
+ "2830": 956741440.0,
+ "2835": 986321792.0,
+ "2840": 961759296.0,
+ "2845": 967500032.0,
+ "2850": 951712064.0,
+ "2855": 962088704.0,
+ "2860": 954242816.0,
+ "2865": 955881728.0,
+ "2870": 944663168.0,
+ "2875": 974664512.0,
+ "2880": 968201792.0,
+ "2885": 981081344.0,
+ "2890": 953455232.0,
+ "2895": 957179904.0,
+ "2900": 964991552.0,
+ "2905": 931708288.0,
+ "2910": 955730816.0,
+ "2915": 979475584.0,
+ "2920": 970492672.0,
+ "2925": 964976576.0,
+ "2930": 964047872.0,
+ "2935": 940141568.0,
+ "2940": 964910976.0,
+ "2945": 989149696.0,
+ "2950": 965208448.0,
+ "2955": 965105920.0,
+ "2960": 933161408.0,
+ "2965": 968794496.0,
+ "2970": 973034880.0,
+ "2975": 958093184.0,
+ "2980": 964497856.0,
+ "2985": 937269888.0,
+ "2990": 951254144.0,
+ "2995": 978316736.0,
+ "3000": 969276160.0,
+ "3005": 974687680.0,
+ "3010": 950236416.0,
+ "3015": 943841920.0,
+ "3020": 958436160.0,
+ "3025": 975185280.0,
+ "3030": 965017088.0,
+ "3035": 963452544.0,
+ "3040": 952132352.0,
+ "3045": 989793344.0,
+ "3050": 965544896.0,
+ "3055": 982520768.0,
+ "3060": 971226112.0,
+ "3065": 943916032.0,
+ "3070": 978408000.0,
+ "3075": 975203840.0,
+ "3080": 960991552.0,
+ "3085": 962352192.0,
+ "3090": 945951680.0,
+ "3095": 938116416.0,
+ "3100": 972927936.0,
+ "3105": 962001024.0,
+ "3110": 970668800.0,
+ "3115": 963402048.0,
+ "3120": 947128832.0,
+ "3125": 972719488.0,
+ "3130": 952985856.0,
+ "3135": 966040384.0,
+ "3140": 968500800.0,
+ "3145": 937851968.0,
+ "3150": 975022144.0,
+ "3155": 976826368.0,
+ "3160": 969641856.0,
+ "3165": 982207040.0,
+ "3170": 937973440.0,
+ "3175": 953838080.0,
+ "3180": 983821312.0,
+ "3185": 965180288.0,
+ "3190": 968481408.0,
+ "3195": 950943936.0,
+ "3200": 945101696.0,
+ "3205": 959862912.0,
+ "3210": 957499456.0,
+ "3215": 958032768.0,
+ "3220": 968142016.0,
+ "3225": 935626752.0,
+ "3230": 962588608.0,
+ "3235": 975788416.0,
+ "3240": 962634816.0,
+ "3245": 981286080.0,
+ "3250": 943272704.0,
+ "3255": 954612224.0,
+ "3260": 980375424.0,
+ "3265": 963632192.0,
+ "3270": 965176512.0,
+ "3275": 959744000.0,
+ "3280": 967058432.0,
+ "3285": 982489984.0,
+ "3290": 947702528.0,
+ "3295": 966434112.0,
+ "3300": 959179136.0,
+ "3305": 949142784.0,
+ "3310": 979523008.0,
+ "3315": 964295552.0,
+ "3320": 969206848.0,
+ "3325": 956206080.0,
+ "3330": 941178880.0,
+ "3335": 964986816.0,
+ "3340": 956914432.0,
+ "3345": 972501568.0,
+ "3350": 964584448.0,
+ "3355": 943359616.0,
+ "3360": 970048192.0,
+ "3365": 969467008.0,
+ "3370": 954775680.0,
+ "3375": 958689792.0,
+ "3380": 971476160.0,
+ "3385": 947985792.0,
+ "3390": 965793152.0,
+ "3395": 978403584.0,
+ "3400": 978139840.0,
+ "3405": 976735808.0,
+ "3410": 924207040.0,
+ "3415": 955437376.0,
+ "3420": 971832192.0,
+ "3425": 977167040.0,
+ "3430": 973838592.0,
+ "3435": 936083776.0,
+ "3440": 970515200.0,
+ "3445": 957315776.0,
+ "3450": 959852224.0,
+ "3455": 963865664.0,
+ "3460": 967881856.0,
+ "3465": 931330944.0,
+ "3470": 952347136.0,
+ "3475": 973717632.0,
+ "3480": 959751680.0,
+ "3485": 979947840.0,
+ "3490": 944683264.0,
+ "3495": 953917632.0,
+ "3500": 969331264.0,
+ "3505": 964360704.0,
+ "3510": 971234112.0,
+ "3515": 955945984.0,
+ "3520": 958734336.0,
+ "3525": 971927296.0,
+ "3530": 964137664.0,
+ "3535": 983204352.0,
+ "3540": 937502016.0,
+ "3545": 944743040.0,
+ "3550": 984474624.0,
+ "3555": 978070528.0,
+ "3560": 974387072.0,
+ "3565": 968812544.0,
+ "3570": 946707328.0,
+ "3575": 976117248.0,
+ "3580": 977505536.0,
+ "3585": 954581376.0,
+ "3590": 956437952.0,
+ "3595": 951456640.0,
+ "3600": 989010176.0,
+ "3605": 962020480.0,
+ "3610": 965066240.0,
+ "3615": 974647360.0,
+ "3620": 954887488.0,
+ "3625": 939528768.0,
+ "3630": 990161664.0,
+ "3635": 971447744.0,
+ "3640": 976038464.0,
+ "3645": 961501440.0,
+ "3650": 945815680.0,
+ "3655": 965794816.0,
+ "3660": 976207360.0,
+ "3665": 964019648.0,
+ "3670": 977444032.0,
+ "3675": 943500672.0,
+ "3680": 958209408.0,
+ "3685": 964297920.0,
+ "3690": 982118912.0,
+ "3695": 963137856.0,
+ "3700": 950582528.0,
+ "3705": 947345088.0,
+ "3710": 982367872.0,
+ "3715": 972706624.0,
+ "3720": 976137664.0,
+ "3725": 964038080.0,
+ "3730": 948861504.0,
+ "3735": 967078400.0,
+ "3740": 960973440.0,
+ "3745": 969320832.0,
+ "3750": 963948800.0,
+ "3755": 953432640.0,
+ "3760": 976653952.0,
+ "3765": 979841920.0,
+ "3770": 972371840.0,
+ "3775": 972384576.0,
+ "3780": 952586048.0,
+ "3785": 960237888.0,
+ "3790": 985598208.0,
+ "3795": 969183872.0,
+ "3800": 957876992.0,
+ "3805": 972407552.0,
+ "3810": 954517952.0,
+ "3815": 974567360.0,
+ "3820": 963019264.0,
+ "3825": 962071104.0,
+ "3830": 969396224.0,
+ "3835": 934733056.0,
+ "3840": 971256576.0,
+ "3845": 986850048.0,
+ "3850": 968848192.0,
+ "3855": 965247744.0,
+ "3860": 948069952.0,
+ "3865": 975049344.0,
+ "3870": 985111424.0,
+ "3875": 983042496.0,
+ "3880": 963615296.0,
+ "3885": 953023360.0,
+ "3890": 960284608.0,
+ "3895": 960602624.0,
+ "3900": 984925312.0,
+ "3905": 976210432.0,
+ "3910": 987334656.0,
+ "3915": 946017792.0,
+ "3920": 974866176.0,
+ "3925": 961235584.0,
+ "3930": 976763392.0,
+ "3935": 978900480.0,
+ "3940": 950267456.0,
+ "3945": 960260992.0,
+ "3950": 974185344.0,
+ "3955": 972955136.0,
+ "3960": 974053120.0,
+ "3965": 950861824.0,
+ "3970": 980668096.0,
+ "3975": 960724992.0,
+ "3980": 977530496.0,
+ "3985": 962936320.0,
+ "3990": 972730048.0,
+ "3995": 953687936.0,
+ "4000": 974987904.0,
+ "4005": 971633792.0,
+ "4010": 978383488.0,
+ "4015": 971478976.0,
+ "4020": 950323136.0,
+ "4025": 968418944.0,
+ "4030": 997923008.0,
+ "4035": 978535744.0,
+ "4040": 959811840.0,
+ "4045": 939638656.0,
+ "4050": 944717824.0,
+ "4055": 980959616.0,
+ "4060": 977683456.0,
+ "4065": 975729984.0,
+ "4070": 942139968.0,
+ "4075": 945740416.0,
+ "4080": 988715584.0,
+ "4085": 962066688.0,
+ "4090": 983344448.0,
+ "4095": 986946816.0,
+ "4100": 957186624.0,
+ "4105": 954085888.0,
+ "4110": 966462976.0,
+ "4115": 976029056.0,
+ "4120": 983518144.0,
+ "4125": 960016640.0,
+ "4130": 967253312.0,
+ "4135": 971431232.0,
+ "4140": 963157504.0,
+ "4145": 956188416.0,
+ "4150": 960281856.0,
+ "4155": 946191936.0,
+ "4160": 968439616.0,
+ "4165": 970303104.0,
+ "4170": 971971200.0,
+ "4175": 955861120.0,
+ "4180": 940972800.0,
+ "4185": 968293568.0,
+ "4190": 967979392.0,
+ "4195": 989198144.0,
+ "4200": 962629056.0,
+ "4205": 960582656.0,
+ "4210": 971851008.0,
+ "4215": 974120256.0,
+ "4220": 981144192.0,
+ "4225": 975207616.0,
+ "4230": 952653184.0,
+ "4235": 958457472.0,
+ "4240": 966762368.0,
+ "4245": 961708096.0,
+ "4250": 965808640.0,
+ "4255": 958217536.0,
+ "4260": 949521216.0,
+ "4265": 964125888.0,
+ "4270": 978346752.0,
+ "4275": 975384960.0,
+ "4280": 962647936.0,
+ "4285": 951425856.0,
+ "4290": 980017024.0,
+ "4295": 968791680.0,
+ "4300": 958273792.0,
+ "4305": 966766720.0,
+ "4310": 939564096.0,
+ "4315": 949379008.0,
+ "4320": 984485696.0,
+ "4325": 982522048.0,
+ "4330": 974697600.0,
+ "4335": 949396736.0,
+ "4340": 959441600.0,
+ "4345": 956595456.0,
+ "4350": 979898368.0,
+ "4355": 968820928.0,
+ "4360": 966300672.0,
+ "4365": 941195776.0,
+ "4370": 969445376.0,
+ "4375": 972957440.0,
+ "4380": 966111296.0,
+ "4385": 971986624.0,
+ "4390": 954114560.0,
+ "4395": 951807488.0,
+ "4400": 973592000.0,
+ "4405": 972416640.0,
+ "4410": 967883456.0,
+ "4415": 958912768.0,
+ "4420": 960729856.0,
+ "4425": 976366400.0,
+ "4430": 965893824.0,
+ "4435": 975864320.0,
+ "4440": 962282176.0,
+ "4445": 954800192.0,
+ "4450": 978281664.0,
+ "4455": 960169216.0,
+ "4460": 968545280.0,
+ "4465": 968756224.0,
+ "4470": 944114816.0,
+ "4475": 951960640.0,
+ "4480": 978776320.0,
+ "4485": 968266624.0,
+ "4490": 956987776.0,
+ "4495": 938865152.0,
+ "4500": 953203968.0,
+ "4505": 977130368.0,
+ "4510": 978652160.0,
+ "4515": 962467840.0,
+ "4520": 958787584.0,
+ "4525": 958077952.0,
+ "4530": 964403968.0,
+ "4535": 976543488.0,
+ "4540": 976798272.0,
+ "4545": 970083840.0,
+ "4550": 953137344.0,
+ "4555": 959504640.0,
+ "4560": 972460992.0,
+ "4565": 973417600.0,
+ "4570": 978848640.0,
+ "4575": 957772992.0,
+ "4580": 963142528.0,
+ "4585": 957345024.0,
+ "4590": 986468992.0,
+ "4595": 960139520.0,
+ "4600": 952193152.0,
+ "4605": 959225216.0,
+ "4610": 963698432.0,
+ "4615": 957937536.0,
+ "4620": 960168960.0,
+ "4625": 973774208.0,
+ "4630": 944470464.0,
+ "4635": 976988288.0,
+ "4640": 960261824.0,
+ "4645": 981943488.0,
+ "4650": 962325632.0,
+ "4655": 939396864.0,
+ "4660": 963974784.0,
+ "4665": 962479040.0,
+ "4670": 976585664.0,
+ "4675": 963416384.0,
+ "4680": 957403008.0,
+ "4685": 949551936.0,
+ "4690": 956901952.0,
+ "4695": 969876992.0,
+ "4700": 961140800.0,
+ "4705": 970707136.0,
+ "4710": 934407872.0,
+ "4715": 970388608.0,
+ "4720": 966252096.0,
+ "4725": 980180160.0,
+ "4730": 965752192.0,
+ "4735": 937796864.0,
+ "4740": 960051136.0,
+ "4745": 975947392.0,
+ "4750": 967910656.0,
+ "4755": 984873536.0,
+ "4760": 959099008.0,
+ "4765": 955138112.0,
+ "4770": 958547200.0,
+ "4775": 990993920.0,
+ "4780": 976744512.0,
+ "4785": 967471104.0,
+ "4790": 943658112.0,
+ "4795": 955775680.0,
+ "4800": 967660672.0,
+ "4805": 976444544.0,
+ "4810": 965075072.0,
+ "4815": 957944832.0,
+ "4820": 973918976.0,
+ "4825": 961444160.0,
+ "4830": 962527616.0,
+ "4835": 972470272.0,
+ "4840": 948840256.0,
+ "4845": 965641344.0,
+ "4850": 960193984.0,
+ "4855": 964069248.0,
+ "4860": 962964928.0,
+ "4865": 967457920.0,
+ "4870": 957132800.0,
+ "4875": 983500672.0,
+ "4880": 956973824.0,
+ "4885": 976940608.0,
+ "4890": 959727744.0,
+ "4895": 942127296.0,
+ "4900": 973552320.0,
+ "4905": 975127424.0,
+ "4910": 969170944.0,
+ "4915": 970053056.0,
+ "4920": 941108480.0,
+ "4925": 954698816.0,
+ "4930": 976972928.0,
+ "4935": 963676928.0,
+ "4940": 972549504.0,
+ "4945": 959994368.0,
+ "4950": 940720256.0,
+ "4955": 968012160.0,
+ "4960": 976789184.0,
+ "4965": 960947328.0,
+ "4970": 958579328.0,
+ "4975": 933677440.0,
+ "4980": 960725888.0,
+ "4985": 962937792.0,
+ "4990": 963466944.0,
+ "4995": 986209664.0,
+ "5000": 940640704.0,
+ "5005": 968874752.0,
+ "5010": 970197824.0,
+ "5015": 965159680.0,
+ "5020": 966679232.0,
+ "5025": 949349824.0,
+ "5030": 953478208.0,
+ "5035": 967279104.0,
+ "5040": 955578368.0,
+ "5045": 969055232.0,
+ "5050": 953342976.0,
+ "5055": 954799232.0,
+ "5060": 962992704.0,
+ "5065": 952133248.0,
+ "5070": 973547712.0,
+ "5075": 978507392.0,
+ "5080": 942804736.0,
+ "5085": 965765184.0,
+ "5090": 972832256.0,
+ "5095": 964348416.0,
+ "5100": 958305664.0,
+ "5105": 965256064.0,
+ "5110": 950316352.0,
+ "5115": 972296256.0,
+ "5120": 960314560.0,
+ "5125": 969694272.0,
+ "5130": 938721536.0,
+ "5135": 943668992.0,
+ "5140": 969825408.0,
+ "5145": 968528896.0,
+ "5150": 970513024.0,
+ "5155": 972533568.0,
+ "5160": 926488576.0,
+ "5165": 961518208.0,
+ "5170": 966811776.0,
+ "5175": 966011776.0,
+ "5180": 963618304.0,
+ "5185": 930814464.0,
+ "5190": 949718016.0,
+ "5195": 972335808.0,
+ "5200": 973709440.0,
+ "5205": 968151680.0,
+ "5210": 960453248.0,
+ "5215": 928729984.0,
+ "5220": 979075840.0,
+ "5225": 984717760.0,
+ "5230": 974973440.0,
+ "5235": 975006144.0,
+ "5240": 944288384.0,
+ "5245": 970786240.0,
+ "5250": 972391040.0,
+ "5255": 966871168.0,
+ "5260": 976692544.0,
+ "5265": 942199104.0,
+ "5270": 969176896.0,
+ "5275": 970048640.0,
+ "5280": 962752640.0,
+ "5285": 964022784.0,
+ "5290": 932477120.0,
+ "5295": 951640192.0,
+ "5300": 975575680.0,
+ "5305": 951864064.0,
+ "5310": 967966144.0,
+ "5315": 955743360.0,
+ "5320": 950921664.0,
+ "5325": 972942144.0,
+ "5330": 967696128.0,
+ "5335": 967535616.0,
+ "5340": 966417280.0,
+ "5345": 962874624.0,
+ "5350": 978862976.0,
+ "5355": 972203328.0,
+ "5360": 963819072.0,
+ "5365": 965210368.0,
+ "5370": 947812992.0,
+ "5375": 948759872.0,
+ "5380": 967162432.0,
+ "5385": 980467008.0,
+ "5390": 965227392.0,
+ "5395": 955172224.0,
+ "5400": 948212288.0,
+ "5405": 974243520.0,
+ "5410": 967795200.0,
+ "5415": 975977728.0,
+ "5420": 967321728.0,
+ "5425": 937376192.0,
+ "5430": 963821760.0,
+ "5435": 971825536.0,
+ "5440": 969029120.0,
+ "5445": 957378176.0,
+ "5450": 919443456.0,
+ "5455": 952003968.0,
+ "5460": 962289664.0,
+ "5465": 978917504.0,
+ "5470": 980871040.0,
+ "5475": 941549312.0,
+ "5480": 955693632.0,
+ "5485": 964840000.0,
+ "5490": 975924608.0,
+ "5495": 962731904.0,
+ "5500": 971132416.0,
+ "5505": 957026112.0,
+ "5510": 968525056.0,
+ "5515": 945503808.0,
+ "5520": 963158144.0,
+ "5525": 975845632.0,
+ "5530": 936597824.0,
+ "5535": 970633088.0,
+ "5540": 960331776.0,
+ "5545": 971949952.0,
+ "5550": 967847296.0,
+ "5555": 955832704.0,
+ "5560": 954630592.0,
+ "5565": 968800768.0,
+ "5570": 945161664.0,
+ "5575": 960575744.0,
+ "5580": 960610816.0,
+ "5585": 959494656.0,
+ "5590": 977618816.0,
+ "5595": 975182400.0,
+ "5600": 963009472.0,
+ "5605": 964090752.0,
+ "5610": 943130880.0,
+ "5615": 966527680.0,
+ "5620": 963273472.0,
+ "5625": 982324096.0,
+ "5630": 975962688.0,
+ "5635": 957432256.0,
+ "5640": 951431040.0,
+ "5645": 967756224.0,
+ "5650": 979095360.0,
+ "5655": 983558976.0,
+ "5660": 956530112.0,
+ "5665": 953585984.0,
+ "5670": 966085888.0,
+ "5675": 967701504.0,
+ "5680": 978635136.0,
+ "5685": 961971200.0,
+ "5690": 935896576.0,
+ "5695": 963738688.0,
+ "5700": 952586560.0,
+ "5705": 974499776.0,
+ "5710": 971280128.0,
+ "5715": 946048000.0,
+ "5720": 975003520.0,
+ "5725": 967399872.0,
+ "5730": 978524800.0,
+ "5735": 964881152.0,
+ "5740": 943709760.0,
+ "5745": 971179264.0,
+ "5750": 982024640.0,
+ "5755": 956767360.0,
+ "5760": 963708224.0,
+ "5765": 957645376.0,
+ "5770": 955673664.0,
+ "5775": 970860288.0,
+ "5780": 962959104.0,
+ "5785": 970821760.0,
+ "5790": 974947840.0,
+ "5795": 949912192.0,
+ "5800": 965951296.0,
+ "5805": 968749504.0,
+ "5810": 976207616.0,
+ "5815": 970065984.0,
+ "5820": 936474496.0,
+ "5825": 969549184.0,
+ "5830": 977626880.0,
+ "5835": 975078784.0,
+ "5840": 963147584.0,
+ "5845": 968825088.0,
+ "5850": 943106496.0,
+ "5855": 976036352.0,
+ "5860": 979622976.0,
+ "5865": 978382080.0,
+ "5870": 968998016.0,
+ "5875": 942376832.0,
+ "5880": 964595200.0,
+ "5885": 974769664.0,
+ "5890": 972745728.0,
+ "5895": 965767680.0,
+ "5900": 941346816.0,
+ "5905": 961840832.0,
+ "5910": 958716672.0,
+ "5915": 968075840.0,
+ "5920": 977555200.0,
+ "5925": 959702336.0,
+ "5930": 946848384.0,
+ "5935": 952396160.0,
+ "5940": 977736000.0,
+ "5945": 984818176.0,
+ "5950": 980776512.0,
+ "5955": 935216768.0,
+ "5960": 961736256.0,
+ "5965": 966035648.0,
+ "5970": 970636544.0,
+ "5975": 962117248.0,
+ "5980": 958208576.0,
+ "5985": 964636928.0,
+ "5990": 973554176.0,
+ "5995": 956128704.0,
+ "6000": 955744128.0,
+ "6005": 961380352.0,
+ "6010": 953014016.0,
+ "6015": 974649856.0,
+ "6020": 978278656.0,
+ "6025": 972129536.0,
+ "6030": 955373376.0,
+ "6035": 946856064.0,
+ "6040": 962581056.0,
+ "6045": 983964992.0,
+ "6050": 956601664.0,
+ "6055": 963618048.0,
+ "6060": 945903232.0,
+ "6065": 958737792.0,
+ "6070": 978427904.0,
+ "6075": 978185408.0,
+ "6080": 957702656.0,
+ "6085": 947826368.0,
+ "6090": 953768768.0,
+ "6095": 964843776.0,
+ "6100": 979825280.0,
+ "6105": 971208064.0,
+ "6110": 961876992.0,
+ "6115": 943727424.0,
+ "6120": 968622720.0,
+ "6125": 960862528.0,
+ "6130": 984113856.0,
+ "6135": 961117184.0,
+ "6140": 958732672.0,
+ "6145": 971396352.0,
+ "6150": 968554112.0,
+ "6155": 975040384.0,
+ "6160": 977190848.0,
+ "6165": 953025344.0,
+ "6170": 951252608.0,
+ "6175": 963543552.0,
+ "6180": 969390144.0,
+ "6185": 966295168.0,
+ "6190": 963964672.0,
+ "6195": 947167616.0,
+ "6200": 969488064.0,
+ "6205": 967247424.0,
+ "6210": 959217344.0,
+ "6215": 973036224.0,
+ "6220": 936472832.0,
+ "6225": 978690944.0,
+ "6230": 976060672.0,
+ "6235": 971581824.0,
+ "6240": 965871552.0,
+ "6245": 956004864.0,
+ "6250": 956719616.0,
+ "6255": 973240704.0,
+ "6260": 978976192.0,
+ "6265": 974930880.0,
+ "6270": 958670912.0,
+ "6275": 963688832.0,
+ "6280": 973129344.0,
+ "6285": 966123392.0,
+ "6290": 970966464.0,
+ "6295": 987367680.0,
+ "6300": 947468160.0,
+ "6305": 964820928.0,
+ "6310": 979103360.0,
+ "6315": 978343680.0,
+ "6320": 971773376.0,
+ "6325": 922855744.0,
+ "6330": 959107840.0,
+ "6335": 974816768.0,
+ "6340": 984714752.0,
+ "6345": 967007936.0,
+ "6350": 944575168.0,
+ "6355": 957977920.0,
+ "6360": 972760000.0,
+ "6365": 972380416.0,
+ "6370": 959361152.0,
+ "6375": 967129344.0,
+ "6380": 951326848.0,
+ "6385": 973561856.0,
+ "6390": 965380288.0,
+ "6395": 975032448.0,
+ "6400": 984241024.0,
+ "6405": 944249728.0,
+ "6410": 977450112.0,
+ "6415": 971759936.0,
+ "6420": 956665088.0,
+ "6425": 961024896.0,
+ "6430": 957475648.0,
+ "6435": 960222720.0,
+ "6440": 969046080.0,
+ "6445": 973418112.0,
+ "6450": 974342784.0,
+ "6455": 962497664.0,
+ "6460": 941166912.0,
+ "6465": 974334976.0,
+ "6470": 979985920.0,
+ "6475": 960811968.0,
+ "6480": 967643392.0,
+ "6485": 948782528.0,
+ "6490": 971032000.0,
+ "6495": 988664448.0,
+ "6500": 980469504.0,
+ "6505": 972048256.0,
+ "6510": 951587776.0,
+ "6515": 957811520.0,
+ "6520": 978840640.0,
+ "6525": 979014912.0,
+ "6530": 973383168.0,
+ "6535": 967829248.0,
+ "6540": 950430336.0,
+ "6545": 966515392.0,
+ "6550": 979372032.0,
+ "6555": 967131776.0,
+ "6560": 975416256.0,
+ "6565": 949474816.0,
+ "6570": 952104768.0,
+ "6575": 962492352.0,
+ "6580": 975657472.0,
+ "6585": 979594176.0,
+ "6590": 949188672.0,
+ "6595": 961521152.0,
+ "6600": 961195520.0,
+ "6605": 961424320.0,
+ "6610": 985028736.0,
+ "6615": 959654912.0,
+ "6620": 944341824.0,
+ "6625": 970881536.0,
+ "6630": 971369024.0,
+ "6635": 963908032.0,
+ "6640": 959870080.0,
+ "6645": 950833152.0,
+ "6650": 978909248.0,
+ "6655": 965790528.0,
+ "6660": 968642432.0,
+ "6665": 968965312.0,
+ "6670": 932863360.0,
+ "6675": 970957696.0,
+ "6680": 968819392.0,
+ "6685": 958796800.0,
+ "6690": 956030912.0,
+ "6695": 955610944.0,
+ "6700": 962074688.0,
+ "6705": 979070272.0,
+ "6710": 971393472.0,
+ "6715": 966740672.0,
+ "6720": 974002944.0,
+ "6725": 941977984.0,
+ "6730": 979547776.0,
+ "6735": 994587008.0,
+ "6740": 976441984.0,
+ "6745": 974481344.0,
+ "6750": 939296384.0,
+ "6755": 977465216.0,
+ "6760": 969855936.0,
+ "6765": 978564992.0,
+ "6770": 975255488.0,
+ "6775": 943450112.0,
+ "6780": 947295872.0,
+ "6785": 975604800.0,
+ "6790": 960471296.0,
+ "6795": 976304832.0,
+ "6800": 973159872.0,
+ "6805": 946486464.0,
+ "6810": 958437888.0,
+ "6815": 970647104.0,
+ "6820": 978037248.0,
+ "6825": 969144448.0,
+ "6830": 950595392.0,
+ "6835": 981587392.0,
+ "6840": 983195008.0,
+ "6845": 948960960.0,
+ "6850": 965662592.0,
+ "6855": 953732736.0,
+ "6860": 979050304.0,
+ "6865": 983914432.0,
+ "6870": 965064128.0,
+ "6875": 978995328.0,
+ "6880": 950605696.0,
+ "6885": 958238016.0,
+ "6890": 960190016.0,
+ "6895": 965887232.0,
+ "6900": 985198720.0,
+ "6905": 968454016.0,
+ "6910": 949851520.0,
+ "6915": 970890688.0,
+ "6920": 967265216.0,
+ "6925": 965108480.0,
+ "6930": 964729792.0,
+ "6935": 952209152.0,
+ "6940": 963044736.0,
+ "6945": 986545280.0,
+ "6950": 973486592.0,
+ "6955": 964912768.0,
+ "6960": 940139392.0,
+ "6965": 974504448.0,
+ "6970": 978284928.0,
+ "6975": 985103936.0,
+ "6980": 982446912.0,
+ "6985": 959605952.0,
+ "6990": 945558016.0,
+ "6995": 987421568.0,
+ "7000": 963206400.0,
+ "7005": 962628800.0,
+ "7010": 985217408.0,
+ "7015": 945595392.0,
+ "7020": 983020288.0,
+ "7025": 968805184.0,
+ "7030": 953673536.0,
+ "7035": 982738304.0,
+ "7040": 950484416.0,
+ "7045": 955734592.0,
+ "7050": 959912128.0,
+ "7055": 963461824.0,
+ "7060": 976974208.0,
+ "7065": 967937344.0,
+ "7070": 953363520.0,
+ "7075": 956584256.0,
+ "7080": 969236032.0,
+ "7085": 966041472.0,
+ "7090": 969411904.0,
+ "7095": 959820608.0,
+ "7100": 973737984.0,
+ "7105": 973259264.0,
+ "7110": 970316864.0,
+ "7115": 958425088.0,
+ "7120": 948966656.0,
+ "7125": 963190016.0,
+ "7130": 971607488.0,
+ "7135": 964616128.0,
+ "7140": 961313152.0,
+ "7145": 930553024.0,
+ "7150": 946378880.0,
+ "7155": 990761152.0,
+ "7160": 968173568.0,
+ "7165": 956889344.0,
+ "7170": 968251776.0,
+ "7175": 955767296.0,
+ "7180": 958036608.0,
+ "7185": 984732992.0,
+ "7190": 978833856.0,
+ "7195": 973712128.0,
+ "7200": 935857728.0,
+ "7205": 957365120.0,
+ "7210": 967041152.0,
+ "7215": 969412864.0,
+ "7220": 982109888.0,
+ "7225": 928727552.0,
+ "7230": 949316352.0,
+ "7235": 967054720.0,
+ "7240": 966895360.0,
+ "7245": 967651904.0,
+ "7250": 949626496.0,
+ "7255": 957388288.0,
+ "7260": 970176448.0,
+ "7265": 975035776.0,
+ "7270": 959863168.0,
+ "7275": 959348544.0,
+ "7280": 957000192.0,
+ "7285": 977982528.0,
+ "7290": 977201792.0,
+ "7295": 963098624.0,
+ "7300": 975327616.0,
+ "7305": 964145344.0,
+ "7310": 977565952.0,
+ "7315": 966761856.0,
+ "7320": 974417920.0,
+ "7325": 966895104.0,
+ "7330": 959593152.0,
+ "7335": 963913984.0,
+ "7340": 977791360.0,
+ "7345": 967878336.0,
+ "7350": 984658752.0,
+ "7355": 959651584.0,
+ "7360": 948746624.0,
+ "7365": 972749248.0,
+ "7370": 982520512.0,
+ "7375": 963272640.0,
+ "7380": 964382592.0,
+ "7385": 948591936.0,
+ "7390": 963947200.0,
+ "7395": 958553984.0,
+ "7400": 969678464.0,
+ "7405": 987651072.0,
+ "7410": 952276480.0,
+ "7415": 950530880.0,
+ "7420": 966746944.0,
+ "7425": 982829184.0,
+ "7430": 965909824.0,
+ "7435": 972889664.0,
+ "7440": 937252992.0,
+ "7445": 968829376.0,
+ "7450": 980216320.0,
+ "7455": 971489344.0,
+ "7460": 972282432.0,
+ "7465": 939402112.0,
+ "7470": 971692608.0,
+ "7475": 957750656.0,
+ "7480": 969468224.0,
+ "7485": 961552064.0,
+ "7490": 934133440.0,
+ "7495": 957481664.0,
+ "7500": 969315456.0,
+ "7505": 970271936.0,
+ "7510": 972148224.0,
+ "7515": 979570752.0,
+ "7520": 951797248.0,
+ "7525": 970358464.0,
+ "7530": 954349120.0,
+ "7535": 971249024.0,
+ "7540": 979388480.0,
+ "7545": 959124288.0,
+ "7550": 960365824.0,
+ "7555": 959826496.0,
+ "7560": 970185408.0,
+ "7565": 954961536.0,
+ "7570": 942754432.0,
+ "7575": 965398080.0,
+ "7580": 982261824.0,
+ "7585": 978833664.0,
+ "7590": 970328192.0,
+ "7595": 950078208.0,
+ "7600": 946067392.0,
+ "7605": 982220672.0,
+ "7610": 969327872.0,
+ "7615": 988638464.0,
+ "7620": 957197312.0,
+ "7625": 941357056.0,
+ "7630": 971415232.0,
+ "7635": 984228480.0,
+ "7640": 983531520.0,
+ "7645": 968049152.0,
+ "7650": 959283328.0,
+ "7655": 962240256.0,
+ "7660": 968743872.0,
+ "7665": 978220032.0,
+ "7670": 975769792.0,
+ "7675": 975312960.0,
+ "7680": 942613504.0,
+ "7685": 959860928.0,
+ "7690": 975230208.0,
+ "7695": 982332096.0,
+ "7700": 979320960.0,
+ "7705": 940520832.0,
+ "7710": 974873024.0,
+ "7715": 979676800.0,
+ "7720": 968007552.0,
+ "7725": 960515968.0,
+ "7730": 943202432.0,
+ "7735": 968229568.0,
+ "7740": 980156864.0,
+ "7745": 964188416.0,
+ "7750": 963657920.0,
+ "7755": 959910912.0,
+ "7760": 970408448.0,
+ "7765": 971294848.0,
+ "7770": 962203392.0,
+ "7775": 981663808.0,
+ "7780": 964605376.0,
+ "7785": 959948992.0,
+ "7790": 968295808.0,
+ "7795": 968590016.0,
+ "7800": 971622656.0,
+ "7805": 968209216.0,
+ "7810": 945873792.0,
+ "7815": 963823360.0,
+ "7820": 974418688.0,
+ "7825": 963931136.0,
+ "7830": 956929024.0,
+ "7835": 949759104.0,
+ "7840": 957601664.0,
+ "7845": 954126848.0,
+ "7850": 979673472.0,
+ "7855": 987113216.0,
+ "7860": 947068928.0,
+ "7865": 949487168.0,
+ "7870": 965175808.0,
+ "7875": 975789312.0,
+ "7880": 968761472.0,
+ "7885": 968941312.0,
+ "7890": 951623680.0,
+ "7895": 974831232.0,
+ "7900": 963584064.0,
+ "7905": 964137344.0,
+ "7910": 965500672.0,
+ "7915": 943576256.0,
+ "7920": 950969088.0,
+ "7925": 969635456.0,
+ "7930": 964956608.0,
+ "7935": 984559872.0,
+ "7940": 964905344.0,
+ "7945": 950760448.0,
+ "7950": 962209216.0,
+ "7955": 979885312.0,
+ "7960": 963566784.0,
+ "7965": 953230464.0,
+ "7970": 951716608.0,
+ "7975": 969856896.0,
+ "7980": 965181696.0,
+ "7985": 959437312.0,
+ "7990": 968092800.0,
+ "7995": 946804480.0,
+ "8000": 962951168.0,
+ "8005": 980789120.0,
+ "8010": 965420992.0,
+ "8015": 982784064.0,
+ "8020": 961005248.0,
+ "8025": 965458624.0,
+ "8030": 958618560.0,
+ "8035": 975777024.0,
+ "8040": 960459776.0,
+ "8045": 948136064.0,
+ "8050": 959229696.0,
+ "8055": 978929664.0,
+ "8060": 969198848.0,
+ "8065": 957854656.0,
+ "8070": 963932352.0,
+ "8075": 942315968.0,
+ "8080": 966157824.0,
+ "8085": 966925696.0,
+ "8090": 983454528.0,
+ "8095": 988527104.0,
+ "8100": 966421504.0,
+ "8105": 944303744.0,
+ "8110": 968909504.0,
+ "8115": 985647488.0,
+ "8120": 974936064.0,
+ "8125": 964178432.0,
+ "8130": 966359552.0,
+ "8135": 967737408.0,
+ "8140": 963995520.0,
+ "8145": 995087872.0,
+ "8150": 973252352.0,
+ "8155": 938501696.0,
+ "8160": 964363584.0,
+ "8165": 973418048.0,
+ "8170": 968656768.0,
+ "8175": 961648896.0,
+ "8180": 936115904.0,
+ "8185": 962846464.0,
+ "8190": 968107072.0,
+ "8195": 977152448.0,
+ "8200": 956773632.0,
+ "8205": 961033152.0,
+ "8210": 946889344.0,
+ "8215": 982632448.0,
+ "8220": 988185600.0,
+ "8225": 966648832.0,
+ "8230": 962897536.0,
+ "8235": 933948096.0,
+ "8240": 980489664.0,
+ "8245": 976301056.0,
+ "8250": 963861056.0,
+ "8255": 977137152.0,
+ "8260": 956588608.0,
+ "8265": 982784640.0,
+ "8270": 952701696.0,
+ "8275": 974062528.0,
+ "8280": 974439680.0,
+ "8285": 953395264.0,
+ "8290": 939875776.0,
+ "8295": 981224832.0,
+ "8300": 973482496.0,
+ "8305": 977951104.0,
+ "8310": 951317440.0,
+ "8315": 938032768.0,
+ "8320": 977546752.0,
+ "8325": 967701504.0,
+ "8330": 990204544.0,
+ "8335": 975809792.0,
+ "8340": 947969280.0,
+ "8345": 970863680.0,
+ "8350": 969947264.0,
+ "8355": 975458560.0,
+ "8360": 979849472.0,
+ "8365": 933249216.0,
+ "8370": 965366272.0,
+ "8375": 979880128.0,
+ "8380": 965581184.0,
+ "8385": 972620288.0,
+ "8390": 962944768.0,
+ "8395": 951483264.0,
+ "8400": 972189632.0,
+ "8405": 951416576.0,
+ "8410": 960852864.0,
+ "8415": 965647872.0,
+ "8420": 941496960.0,
+ "8425": 968092928.0,
+ "8430": 961037312.0,
+ "8435": 965993344.0,
+ "8440": 969732736.0,
+ "8445": 952751424.0,
+ "8450": 984203904.0,
+ "8455": 990432000.0,
+ "8460": 968853440.0,
+ "8465": 967392640.0,
+ "8470": 963083008.0,
+ "8475": 942786624.0,
+ "8480": 987434176.0,
+ "8485": 980034176.0,
+ "8490": 992329600.0,
+ "8495": 971754112.0,
+ "8500": 951823360.0,
+ "8505": 983228800.0,
+ "8510": 974309568.0,
+ "8515": 969046784.0,
+ "8520": 961730944.0,
+ "8525": 944922304.0,
+ "8530": 984660608.0,
+ "8535": 978670656.0,
+ "8540": 968110080.0,
+ "8545": 968524480.0,
+ "8550": 942347392.0,
+ "8555": 971930752.0,
+ "8560": 958590336.0,
+ "8565": 975812096.0,
+ "8570": 974790272.0,
+ "8575": 971485312.0,
+ "8580": 932314048.0,
+ "8585": 965844224.0,
+ "8590": 979204736.0,
+ "8595": 979228992.0,
+ "8600": 984273152.0,
+ "8605": 957989312.0,
+ "8610": 984168384.0,
+ "8615": 977626048.0,
+ "8620": 963301952.0,
+ "8625": 979589952.0,
+ "8630": 943409024.0,
+ "8635": 961711936.0,
+ "8640": 973623232.0,
+ "8645": 970575104.0,
+ "8650": 969548608.0,
+ "8655": 971083968.0,
+ "8660": 944791360.0,
+ "8665": 986806400.0,
+ "8670": 960723904.0,
+ "8675": 974538176.0,
+ "8680": 962363392.0,
+ "8685": 955619648.0,
+ "8690": 978563840.0,
+ "8695": 968917632.0,
+ "8700": 973037184.0,
+ "8705": 973570880.0,
+ "8710": 947093184.0,
+ "8715": 973675392.0,
+ "8720": 958168000.0,
+ "8725": 978732480.0,
+ "8730": 986091264.0,
+ "8735": 952203264.0,
+ "8740": 940976448.0,
+ "8745": 988046464.0,
+ "8750": 971900736.0,
+ "8755": 971639616.0,
+ "8760": 965755008.0,
+ "8765": 934692160.0,
+ "8770": 986510272.0,
+ "8775": 969679296.0,
+ "8780": 967393920.0,
+ "8785": 962431168.0,
+ "8790": 947946368.0,
+ "8795": 969419392.0,
+ "8800": 971061312.0,
+ "8805": 973534976.0,
+ "8810": 983247360.0,
+ "8815": 951791424.0,
+ "8820": 940190528.0,
+ "8825": 964691840.0,
+ "8830": 981237952.0,
+ "8835": 971841408.0,
+ "8840": 979946560.0,
+ "8845": 951308864.0,
+ "8850": 987092992.0,
+ "8855": 971287168.0,
+ "8860": 962172032.0,
+ "8865": 956633280.0,
+ "8870": 946021632.0,
+ "8875": 968481408.0,
+ "8880": 983755776.0,
+ "8885": 971736384.0,
+ "8890": 969784896.0,
+ "8895": 952910336.0,
+ "8900": 961620672.0,
+ "8905": 976778112.0,
+ "8910": 981939072.0,
+ "8915": 980891648.0,
+ "8920": 968024512.0,
+ "8925": 940251392.0,
+ "8930": 970136064.0,
+ "8935": 963917888.0,
+ "8940": 977583360.0,
+ "8945": 981544192.0,
+ "8950": 946004416.0,
+ "8955": 972768192.0,
+ "8960": 973947008.0,
+ "8965": 973383936.0,
+ "8970": 966398272.0,
+ "8975": 936781568.0,
+ "8980": 953136576.0,
+ "8985": 977411200.0,
+ "8990": 967375936.0,
+ "8995": 980774528.0,
+ "9000": 952728192.0,
+ "9005": 950697216.0,
+ "9010": 975551808.0,
+ "9015": 982901760.0,
+ "9020": 959389056.0,
+ "9025": 979940736.0,
+ "9030": 953776512.0,
+ "9035": 968922560.0,
+ "9040": 978552640.0,
+ "9045": 968831296.0,
+ "9050": 982790464.0,
+ "9055": 948045888.0,
+ "9060": 956589184.0,
+ "9065": 970172672.0,
+ "9070": 967684480.0,
+ "9075": 981152384.0,
+ "9080": 952602048.0,
+ "9085": 971467520.0,
+ "9090": 963557248.0,
+ "9095": 967994880.0,
+ "9100": 974781440.0,
+ "9105": 960009792.0,
+ "9110": 947825408.0,
+ "9115": 956785664.0,
+ "9120": 985655040.0,
+ "9125": 963481472.0,
+ "9130": 958727936.0,
+ "9135": 951879232.0,
+ "9140": 967393984.0,
+ "9145": 977101056.0,
+ "9150": 987110720.0,
+ "9155": 977189632.0,
+ "9160": 957952704.0,
+ "9165": 950256768.0,
+ "9170": 988456064.0,
+ "9175": 971693632.0,
+ "9180": 967584320.0,
+ "9185": 955140416.0,
+ "9190": 957140928.0,
+ "9195": 966212352.0,
+ "9200": 969115840.0,
+ "9205": 967428224.0,
+ "9210": 984534912.0,
+ "9215": 932014528.0,
+ "9220": 949587904.0,
+ "9225": 971095424.0,
+ "9230": 971012288.0,
+ "9235": 971575616.0,
+ "9240": 959574272.0,
+ "9245": 963990912.0,
+ "9250": 961552512.0,
+ "9255": 983019392.0,
+ "9260": 979394944.0,
+ "9265": 952805504.0,
+ "9270": 949326336.0,
+ "9275": 978637248.0,
+ "9280": 977361344.0,
+ "9285": 962731200.0,
+ "9290": 979255680.0,
+ "9295": 958574656.0,
+ "9300": 965053312.0,
+ "9305": 969046400.0,
+ "9310": 973026560.0,
+ "9315": 976292288.0,
+ "9320": 947985280.0,
+ "9325": 979776640.0,
+ "9330": 977718976.0,
+ "9335": 975414848.0,
+ "9340": 960187968.0,
+ "9345": 943341952.0,
+ "9350": 952640128.0,
+ "9355": 963598208.0,
+ "9360": 960270848.0,
+ "9365": 983214848.0,
+ "9370": 982509056.0,
+ "9375": 942057024.0,
+ "9380": 982602944.0,
+ "9385": 985542528.0,
+ "9390": 973357952.0,
+ "9395": 978024960.0,
+ "9400": 938125376.0,
+ "9405": 968465088.0,
+ "9410": 981829696.0,
+ "9415": 991810624.0,
+ "9420": 960258176.0,
+ "9425": 956645376.0,
+ "9430": 939027904.0,
+ "9435": 974855168.0,
+ "9440": 959027456.0,
+ "9445": 973696192.0,
+ "9450": 961227776.0,
+ "9455": 946047232.0,
+ "9460": 978455936.0,
+ "9465": 988237952.0,
+ "9470": 963345856.0,
+ "9475": 983982464.0,
+ "9480": 931407360.0,
+ "9485": 987430464.0,
+ "9490": 963495168.0,
+ "9495": 972477824.0,
+ "9500": 982245312.0,
+ "9505": 970089216.0,
+ "9510": 964443776.0,
+ "9515": 957060544.0,
+ "9520": 948360320.0,
+ "9525": 965533056.0,
+ "9530": 958222848.0,
+ "9535": 951184000.0,
+ "9540": 954452096.0,
+ "9545": 979665792.0,
+ "9550": 955705152.0,
+ "9555": 953111552.0,
+ "9560": 958396416.0,
+ "9565": 970089344.0,
+ "9570": 977259456.0,
+ "9575": 958986112.0,
+ "9580": 963197568.0,
+ "9585": 946283648.0,
+ "9590": 947963904.0,
+ "9595": 966430592.0,
+ "9600": 984959040.0,
+ "9605": 985143936.0,
+ "9610": 943481984.0,
+ "9615": 952715008.0,
+ "9620": 980440576.0,
+ "9625": 978303936.0,
+ "9630": 969868160.0,
+ "9635": 974525504.0,
+ "9640": 940122816.0,
+ "9645": 962069952.0,
+ "9650": 971067328.0,
+ "9655": 987521088.0,
+ "9660": 962704576.0,
+ "9665": 949898624.0,
+ "9670": 966470528.0,
+ "9675": 963383936.0,
+ "9680": 964843136.0,
+ "9685": 986562112.0,
+ "9690": 939724096.0,
+ "9695": 950472256.0,
+ "9700": 975001984.0,
+ "9705": 972711936.0,
+ "9710": 967509760.0,
+ "9715": 971305408.0,
+ "9720": 940441792.0,
+ "9725": 966294528.0,
+ "9730": 974027264.0,
+ "9735": 973989248.0,
+ "9740": 971271680.0,
+ "9745": 951061952.0,
+ "9750": 979642304.0,
+ "9755": 970022464.0,
+ "9760": 967782144.0,
+ "9765": 963702016.0,
+ "9770": 952406272.0,
+ "9775": 956486400.0,
+ "9780": 970119744.0,
+ "9785": 958688960.0,
+ "9790": 960858752.0,
+ "9795": 958029440.0,
+ "9800": 949218624.0,
+ "9805": 962086592.0,
+ "9810": 978418688.0,
+ "9815": 977606464.0,
+ "9820": 982410176.0,
+ "9825": 939265280.0,
+ "9830": 969135040.0,
+ "9835": 972511872.0,
+ "9840": 971504128.0,
+ "9845": 967136512.0,
+ "9850": 946839424.0,
+ "9855": 956824128.0,
+ "9860": 986868992.0,
+ "9865": 969709248.0,
+ "9870": 990372224.0,
+ "9875": 957135744.0,
+ "9880": 931092352.0,
+ "9885": 963617216.0,
+ "9890": 972564736.0,
+ "9895": 983526656.0,
+ "9900": 956239424.0,
+ "9905": 938825600.0,
+ "9910": 978737920.0,
+ "9915": 973480128.0,
+ "9920": 944352576.0,
+ "9925": 962596864.0,
+ "9930": 947479232.0,
+ "9935": 960746688.0,
+ "9940": 965980864.0,
+ "9945": 959010752.0,
+ "9950": 964129216.0,
+ "9955": 943475520.0,
+ "9960": 966686464.0,
+ "9965": 983719680.0,
+ "9970": 966804288.0,
+ "9975": 963722560.0,
+ "9980": 980734592.0,
+ "9985": 942341056.0,
+ "9990": 976442240.0,
+ "9995": 982719104.0,
+ "10000": 971864128.0,
+ "10005": 969013952.0,
+ "10010": 944136192.0,
+ "10015": 982651648.0,
+ "10020": 978235776.0,
+ "10025": 979660352.0,
+ "10030": 971208704.0,
+ "10035": 946398464.0,
+ "10040": 949993920.0,
+ "10045": 977929088.0,
+ "10050": 985968512.0,
+ "10055": 990260032.0,
+ "10060": 958923136.0,
+ "10065": 947308672.0,
+ "10070": 966687808.0,
+ "10075": 979044608.0,
+ "10080": 971424832.0,
+ "10085": 974488128.0,
+ "10090": 943751168.0,
+ "10095": 962277376.0,
+ "10100": 971509952.0,
+ "10105": 975740800.0,
+ "10110": 971944448.0,
+ "10115": 948612864.0,
+ "10120": 962398656.0,
+ "10125": 973998848.0,
+ "10130": 979987200.0,
+ "10135": 972261248.0,
+ "10140": 957413696.0,
+ "10145": 933813248.0,
+ "10150": 973622848.0,
+ "10155": 969274304.0,
+ "10160": 961962624.0,
+ "10165": 974813312.0,
+ "10170": 944318784.0,
+ "10175": 978795520.0,
+ "10180": 983504768.0,
+ "10185": 978501632.0,
+ "10190": 954944512.0,
+ "10195": 936623232.0,
+ "10200": 987885440.0,
+ "10205": 972258816.0,
+ "10210": 966509376.0,
+ "10215": 975802752.0,
+ "10220": 948321088.0,
+ "10225": 950461120.0,
+ "10230": 975630208.0,
+ "10235": 953782016.0,
+ "10240": 969370176.0,
+ "10245": 961576192.0,
+ "10250": 936081152.0,
+ "10255": 979319424.0,
+ "10260": 964128640.0,
+ "10265": 967120192.0,
+ "10270": 968498816.0,
+ "10275": 935868800.0,
+ "10280": 969242688.0,
+ "10285": 995937280.0,
+ "10290": 979259136.0,
+ "10295": 981503616.0,
+ "10300": 951791680.0,
+ "10305": 971754560.0,
+ "10310": 959965824.0,
+ "10315": 971077952.0,
+ "10320": 984925632.0,
+ "10325": 983082624.0,
+ "10330": 934657152.0,
+ "10335": 976220352.0,
+ "10340": 957051712.0,
+ "10345": 973646080.0,
+ "10350": 984548224.0,
+ "10355": 941791872.0,
+ "10360": 961714240.0,
+ "10365": 974072704.0,
+ "10370": 980395520.0,
+ "10375": 969789888.0,
+ "10380": 961306368.0,
+ "10385": 955071936.0,
+ "10390": 990365952.0,
+ "10395": 964464000.0,
+ "10400": 960446144.0,
+ "10405": 949563072.0,
+ "10410": 954754304.0,
+ "10415": 975151232.0,
+ "10420": 967101120.0,
+ "10425": 968904128.0,
+ "10430": 964671872.0,
+ "10435": 962459584.0,
+ "10440": 971827584.0,
+ "10445": 972036800.0,
+ "10450": 974739712.0,
+ "10455": 966332928.0,
+ "10460": 948443328.0,
+ "10465": 971633920.0,
+ "10470": 972473984.0,
+ "10475": 979072128.0,
+ "10480": 997084800.0,
+ "10485": 949607552.0,
+ "10490": 934736512.0,
+ "10495": 969033856.0,
+ "10500": 977803776.0,
+ "10505": 958570240.0,
+ "10510": 951081088.0,
+ "10515": 953656832.0,
+ "10520": 971768384.0,
+ "10525": 969009472.0,
+ "10530": 970215680.0,
+ "10535": 986292992.0,
+ "10540": 946977152.0,
+ "10545": 970127104.0,
+ "10550": 969372992.0,
+ "10555": 959300800.0,
+ "10560": 975720832.0,
+ "10565": 961216448.0,
+ "10570": 968959360.0,
+ "10575": 973174272.0,
+ "10580": 960182720.0,
+ "10585": 973309440.0,
+ "10590": 951782144.0,
+ "10595": 956492288.0,
+ "10600": 967455680.0,
+ "10605": 986403584.0,
+ "10610": 966269568.0,
+ "10615": 976502656.0,
+ "10620": 940588992.0,
+ "10625": 965036864.0,
+ "10630": 967383488.0,
+ "10635": 973025920.0,
+ "10640": 974566400.0,
+ "10645": 948472256.0,
+ "10650": 966032512.0,
+ "10655": 985539008.0,
+ "10660": 976460928.0,
+ "10665": 967101120.0,
+ "10670": 954918464.0,
+ "10675": 933718272.0,
+ "10680": 985944832.0,
+ "10685": 990967680.0,
+ "10690": 963776000.0,
+ "10695": 971578624.0,
+ "10700": 949710976.0,
+ "10705": 977786368.0,
+ "10710": 968243712.0,
+ "10715": 966750336.0,
+ "10720": 966038400.0,
+ "10725": 943961472.0,
+ "10730": 980036160.0,
+ "10735": 960563328.0,
+ "10740": 971111424.0,
+ "10745": 983819328.0,
+ "10750": 981627072.0,
+ "10755": 944956224.0,
+ "10760": 969697536.0,
+ "10765": 972653184.0,
+ "10770": 973592640.0,
+ "10775": 958354432.0,
+ "10780": 949396992.0,
+ "10785": 953595776.0,
+ "10790": 969862784.0,
+ "10795": 960249728.0,
+ "10800": 971706432.0,
+ "10805": 950981632.0,
+ "10810": 973867008.0,
+ "10815": 959902272.0,
+ "10820": 971104384.0,
+ "10825": 967088000.0,
+ "10830": 956843200.0,
+ "10835": 962953856.0,
+ "10840": 970556096.0,
+ "10845": 963811072.0,
+ "10850": 957572096.0,
+ "10855": 967361920.0,
+ "10860": 950321024.0,
+ "10865": 963889536.0,
+ "10870": 983368896.0,
+ "10875": 982371968.0,
+ "10880": 958217856.0,
+ "10885": 954330624.0,
+ "10890": 973060864.0,
+ "10895": 973396480.0,
+ "10900": 970089920.0,
+ "10905": 964838592.0,
+ "10910": 938643072.0,
+ "10915": 960478720.0,
+ "10920": 983431808.0,
+ "10925": 969440640.0,
+ "10930": 968268288.0,
+ "10935": 962588992.0,
+ "10940": 953930048.0,
+ "10945": 964470912.0,
+ "10950": 971827840.0,
+ "10955": 966066944.0,
+ "10960": 971449152.0,
+ "10965": 966221568.0,
+ "10970": 983031360.0,
+ "10975": 965673280.0,
+ "10980": 974312320.0,
+ "10985": 986228352.0,
+ "10990": 950760320.0,
+ "10995": 963204800.0,
+ "11000": 985212672.0,
+ "11005": 978412288.0,
+ "11010": 971355776.0,
+ "11015": 969480960.0,
+ "11020": 947788928.0,
+ "11025": 959772544.0,
+ "11030": 977959936.0,
+ "11035": 975451008.0,
+ "11040": 986277504.0,
+ "11045": 956634432.0,
+ "11050": 973260672.0,
+ "11055": 974149504.0,
+ "11060": 962016128.0,
+ "11065": 985504512.0,
+ "11070": 949753280.0,
+ "11075": 976394624.0,
+ "11080": 972031872.0,
+ "11085": 967020416.0,
+ "11090": 976336384.0,
+ "11095": 946732032.0,
+ "11100": 966014272.0,
+ "11105": 973917504.0,
+ "11110": 980682176.0,
+ "11115": 967836672.0,
+ "11120": 957161280.0,
+ "11125": 956646656.0,
+ "11130": 975539968.0,
+ "11135": 979452032.0,
+ "11140": 964636736.0,
+ "11145": 966265792.0,
+ "11150": 935727232.0,
+ "11155": 976467648.0,
+ "11160": 983961664.0,
+ "11165": 982399360.0,
+ "11170": 977720064.0,
+ "11175": 958108096.0,
+ "11180": 962306240.0,
+ "11185": 971891456.0,
+ "11190": 979924416.0,
+ "11195": 985255872.0,
+ "11200": 983025472.0,
+ "11205": 941737600.0,
+ "11210": 984678912.0,
+ "11215": 967432192.0,
+ "11220": 983352000.0,
+ "11225": 961579904.0,
+ "11230": 953286272.0,
+ "11235": 981753920.0,
+ "11240": 977473344.0,
+ "11245": 965594752.0,
+ "11250": 969379072.0,
+ "11255": 959675136.0,
+ "11260": 979628352.0,
+ "11265": 963742080.0,
+ "11270": 981387392.0,
+ "11275": 968132480.0,
+ "11280": 955867648.0,
+ "11285": 953773632.0,
+ "11290": 956513920.0,
+ "11295": 968320320.0,
+ "11300": 962413952.0,
+ "11305": 958477056.0,
+ "11310": 946143616.0,
+ "11315": 983343232.0,
+ "11320": 964742208.0,
+ "11325": 980844992.0,
+ "11330": 975686656.0,
+ "11335": 952358528.0,
+ "11340": 970141248.0,
+ "11345": 970000320.0,
+ "11350": 981713920.0,
+ "11355": 982531008.0,
+ "11360": 941237888.0,
+ "11365": 970640192.0,
+ "11370": 978903424.0,
+ "11375": 975301568.0,
+ "11380": 968416256.0,
+ "11385": 958673088.0,
+ "11390": 937971968.0,
+ "11395": 977420160.0,
+ "11400": 973551360.0,
+ "11405": 961555584.0,
+ "11410": 966291712.0,
+ "11415": 929213632.0,
+ "11420": 964718848.0,
+ "11425": 981182528.0,
+ "11430": 978574592.0,
+ "11435": 970325888.0,
+ "11440": 944973696.0,
+ "11445": 975068032.0,
+ "11450": 984427392.0,
+ "11455": 971364352.0,
+ "11460": 965071360.0,
+ "11465": 960150656.0,
+ "11470": 954908800.0,
+ "11475": 972813184.0,
+ "11480": 956431552.0,
+ "11485": 977088960.0,
+ "11490": 986184192.0,
+ "11495": 959146176.0,
+ "11500": 969768896.0,
+ "11505": 964037120.0,
+ "11510": 976603392.0,
+ "11515": 977965248.0,
+ "11520": 954036416.0,
+ "11525": 976051392.0,
+ "11530": 976575744.0,
+ "11535": 979724928.0,
+ "11540": 974337408.0,
+ "11545": 953335872.0,
+ "11550": 952790336.0,
+ "11555": 981364224.0,
+ "11560": 984814912.0,
+ "11565": 965119296.0,
+ "11570": 966388096.0,
+ "11575": 951670144.0,
+ "11580": 976271040.0,
+ "11585": 977377152.0,
+ "11590": 969153536.0,
+ "11595": 976633088.0,
+ "11600": 946475968.0,
+ "11605": 973212096.0,
+ "11610": 982445120.0,
+ "11615": 972059136.0,
+ "11620": 969179328.0,
+ "11625": 949418368.0,
+ "11630": 937577728.0,
+ "11635": 973618176.0,
+ "11640": 981198336.0,
+ "11645": 980141696.0,
+ "11650": 971794176.0,
+ "11655": 956067968.0,
+ "11660": 980601088.0,
+ "11665": 958364288.0,
+ "11670": 982631232.0,
+ "11675": 972254400.0,
+ "11680": 956150912.0,
+ "11685": 982832256.0,
+ "11690": 968553088.0,
+ "11695": 968086272.0,
+ "11700": 973659712.0,
+ "11705": 956149376.0,
+ "11710": 964658176.0,
+ "11715": 983306432.0,
+ "11720": 983734848.0,
+ "11725": 965216768.0,
+ "11730": 955435456.0,
+ "11735": 942289920.0,
+ "11740": 973779456.0,
+ "11745": 971285248.0,
+ "11750": 961599424.0,
+ "11755": 963397824.0,
+ "11760": 950131200.0,
+ "11765": 983766400.0,
+ "11770": 984208512.0,
+ "11775": 975523712.0,
+ "11780": 985478464.0,
+ "11785": 947161984.0,
+ "11790": 972493312.0,
+ "11795": 970462400.0,
+ "11800": 973178368.0,
+ "11805": 986620480.0,
+ "11810": 967751488.0,
+ "11815": 955799168.0,
+ "11820": 973347712.0,
+ "11825": 970646336.0,
+ "11830": 974624640.0,
+ "11835": 961679552.0,
+ "11840": 944528640.0,
+ "11845": 980866432.0,
+ "11850": 974564800.0,
+ "11855": 978016640.0,
+ "11860": 971869824.0,
+ "11865": 938508672.0,
+ "11870": 940174400.0,
+ "11875": 990073216.0,
+ "11880": 972232128.0,
+ "11885": 962926592.0,
+ "11890": 970329856.0,
+ "11895": 965331392.0,
+ "11900": 979368768.0,
+ "11905": 961579072.0,
+ "11910": 983739904.0,
+ "11915": 989931712.0,
+ "11920": 944844672.0,
+ "11925": 993967360.0,
+ "11930": 964656640.0,
+ "11935": 963318144.0,
+ "11940": 976990208.0,
+ "11945": 943877888.0,
+ "11950": 977614336.0,
+ "11955": 979017984.0,
+ "11960": 972299648.0,
+ "11965": 976420736.0,
+ "11970": 963196544.0,
+ "11975": 963264192.0,
+ "11980": 977523584.0,
+ "11985": 953295936.0,
+ "11990": 968557184.0,
+ "11995": 964949696.0,
+ "12000": 958245696.0,
+ "12005": 974362816.0,
+ "12010": 979133120.0,
+ "12015": 971844096.0,
+ "12020": 973046400.0,
+ "12025": 934406400.0,
+ "12030": 969001216.0,
+ "12035": 984151808.0,
+ "12040": 977262208.0,
+ "12045": 981731776.0,
+ "12050": 931274816.0,
+ "12055": 938311872.0,
+ "12060": 974067200.0,
+ "12065": 965477696.0,
+ "12070": 968006528.0,
+ "12075": 949658880.0,
+ "12080": 953243712.0,
+ "12085": 972513344.0,
+ "12090": 963833920.0,
+ "12095": 963602496.0,
+ "12100": 976480832.0,
+ "12105": 950655424.0,
+ "12110": 972107904.0,
+ "12115": 968212608.0,
+ "12120": 986542656.0,
+ "12125": 980539648.0,
+ "12130": 941737792.0,
+ "12135": 955712512.0,
+ "12140": 976076864.0,
+ "12145": 979760640.0,
+ "12150": 979903040.0,
+ "12155": 961891328.0,
+ "12160": 946421376.0,
+ "12165": 968674944.0,
+ "12170": 964216192.0,
+ "12175": 967224192.0,
+ "12180": 974769664.0,
+ "12185": 952382400.0,
+ "12190": 988016576.0,
+ "12195": 970187072.0,
+ "12200": 964551104.0,
+ "12205": 968434560.0,
+ "12210": 938996224.0,
+ "12215": 996909184.0,
+ "12220": 969778496.0,
+ "12225": 979236736.0,
+ "12230": 980062912.0,
+ "12235": 949854784.0,
+ "12240": 963612544.0,
+ "12245": 965285248.0,
+ "12250": 976217216.0,
+ "12255": 968066560.0,
+ "12260": 983371456.0,
+ "12265": 931721536.0,
+ "12270": 966209920.0,
+ "12275": 979566720.0,
+ "12280": 977469952.0,
+ "12285": 970346432.0,
+ "12290": 929181248.0,
+ "12295": 976800384.0,
+ "12300": 985978496.0,
+ "12305": 969918592.0,
+ "12310": 986289600.0,
+ "12315": 936202304.0,
+ "12320": 958148352.0,
+ "12325": 966634880.0,
+ "12330": 968319488.0,
+ "12335": 963973120.0,
+ "12340": 957507840.0,
+ "12345": 944285056.0,
+ "12350": 966565632.0,
+ "12355": 975757952.0,
+ "12360": 978686464.0,
+ "12365": 964769792.0,
+ "12370": 948300992.0,
+ "12375": 963864256.0,
+ "12380": 964849664.0,
+ "12385": 972869120.0,
+ "12390": 961833792.0,
+ "12395": 961544256.0,
+ "12400": 975505536.0,
+ "12405": 976412672.0,
+ "12410": 953583168.0,
+ "12415": 963226112.0,
+ "12420": 944420736.0,
+ "12425": 949603456.0,
+ "12430": 972833536.0,
+ "12435": 969607552.0,
+ "12440": 962386112.0,
+ "12445": 952637696.0,
+ "12450": 947956480.0,
+ "12455": 981740352.0,
+ "12460": 974357440.0,
+ "12465": 954756736.0,
+ "12470": 980836608.0,
+ "12475": 958470144.0,
+ "12480": 967282176.0,
+ "12485": 978194176.0,
+ "12490": 974255104.0,
+ "12495": 969619136.0,
+ "12500": 961844416.0,
+ "12505": 943830272.0,
+ "12510": 961026880.0,
+ "12515": 969812928.0,
+ "12520": 974083712.0,
+ "12525": 972377536.0,
+ "12530": 944719296.0,
+ "12535": 976303296.0,
+ "12540": 965709056.0,
+ "12545": 971849664.0,
+ "12550": 969844672.0,
+ "12555": 941324672.0,
+ "12560": 964666432.0,
+ "12565": 947400448.0,
+ "12570": 974493568.0,
+ "12575": 963104192.0,
+ "12580": 958128896.0,
+ "12585": 963636992.0,
+ "12590": 965789568.0,
+ "12595": 978658816.0,
+ "12600": 981959552.0,
+ "12605": 949359232.0,
+ "12610": 938076928.0,
+ "12615": 962718720.0,
+ "12620": 961005440.0,
+ "12625": 966630720.0,
+ "12630": 970775872.0,
+ "12635": 962372032.0,
+ "12640": 978419712.0,
+ "12645": 968940544.0,
+ "12650": 970174592.0,
+ "12655": 964155264.0,
+ "12660": 932281664.0,
+ "12665": 956939712.0,
+ "12670": 986266368.0,
+ "12675": 965740032.0,
+ "12680": 960978944.0,
+ "12685": 951439744.0,
+ "12690": 944876992.0,
+ "12695": 978375488.0,
+ "12700": 984995712.0,
+ "12705": 959030784.0,
+ "12710": 968442880.0,
+ "12715": 956452160.0,
+ "12720": "nan",
+ "12725": "nan",
+ "12730": "nan",
+ "12735": "nan",
+ "12740": "nan",
+ "12745": "nan",
+ "12750": "nan",
+ "12755": "nan",
+ "12760": "nan",
+ "12765": "nan",
+ "12770": "nan",
+ "12775": "nan",
+ "12780": "nan",
+ "12785": "nan",
+ "12790": "nan",
+ "12795": "nan",
+ "12800": "nan",
+ "12805": "nan",
+ "12810": "nan",
+ "12815": "nan",
+ "12820": "nan",
+ "12825": "nan",
+ "12830": "nan",
+ "12835": "nan",
+ "12840": "nan",
+ "12845": "nan",
+ "12850": "nan",
+ "12855": "nan",
+ "12860": "nan",
+ "12865": "nan",
+ "12870": "nan",
+ "12875": "nan",
+ "12880": "nan",
+ "12885": "nan",
+ "12890": "nan",
+ "12895": "nan",
+ "12900": "nan",
+ "12905": "nan",
+ "12910": "nan",
+ "12915": "nan",
+ "12920": "nan",
+ "12925": "nan",
+ "12930": "nan",
+ "12935": "nan",
+ "12940": "nan",
+ "12945": "nan",
+ "12950": "nan",
+ "12955": "nan",
+ "12960": "nan",
+ "12965": "nan",
+ "12970": "nan",
+ "12975": "nan",
+ "12980": "nan",
+ "12985": "nan",
+ "12990": "nan",
+ "12995": "nan",
+ "13000": "nan"
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": 13218789376.0,
+ "5": 13218789376.0,
+ "10": 13218789376.0,
+ "15": 13218789376.0,
+ "20": 13218789376.0,
+ "25": 13218789376.0,
+ "30": 13218789376.0,
+ "35": 13218789376.0,
+ "40": 13218789376.0,
+ "45": 13218789376.0,
+ "50": 13218789376.0,
+ "55": 13218789376.0,
+ "60": 13218789376.0,
+ "65": 13218789376.0,
+ "70": 13218789376.0,
+ "75": 13218789376.0,
+ "80": 13218789376.0,
+ "85": 13218789376.0,
+ "90": 13218789376.0,
+ "95": 13218789376.0,
+ "100": 13218789376.0,
+ "105": 13218789376.0,
+ "110": 13218789376.0,
+ "115": 13218789376.0,
+ "120": 13218789376.0,
+ "125": 13218789376.0,
+ "130": 13218789376.0,
+ "135": 13218789376.0,
+ "140": 13218789376.0,
+ "145": 13218789376.0,
+ "150": 13218789376.0,
+ "155": 13218789376.0,
+ "160": 13218789376.0,
+ "165": 13218789376.0,
+ "170": 13218789376.0,
+ "175": 13218789376.0,
+ "180": 13218789376.0,
+ "185": 13218789376.0,
+ "190": 13218789376.0,
+ "195": 13218789376.0,
+ "200": 13218789376.0,
+ "205": 13218789376.0,
+ "210": 13218789376.0,
+ "215": 13218789376.0,
+ "220": 13218789376.0,
+ "225": 13218789376.0,
+ "230": 13218789376.0,
+ "235": 13218789376.0,
+ "240": 13218789376.0,
+ "245": 13218789376.0,
+ "250": 13218789376.0,
+ "255": 13218789376.0,
+ "260": 13218789376.0,
+ "265": 13218789376.0,
+ "270": 13218789376.0,
+ "275": 13218789376.0,
+ "280": 13218789376.0,
+ "285": 13218789376.0,
+ "290": 13218789376.0,
+ "295": 13218789376.0,
+ "300": 13218789376.0,
+ "305": 13218789376.0,
+ "310": 13218789376.0,
+ "315": 13218789376.0,
+ "320": 13218789376.0,
+ "325": 13218789376.0,
+ "330": 13218789376.0,
+ "335": 13218789376.0,
+ "340": 13218789376.0,
+ "345": 13218789376.0,
+ "350": 13218789376.0,
+ "355": 13218789376.0,
+ "360": 13218789376.0,
+ "365": 13218789376.0,
+ "370": 13218789376.0,
+ "375": 13218789376.0,
+ "380": 13218789376.0,
+ "385": 13218789376.0,
+ "390": 13218789376.0,
+ "395": 13218789376.0,
+ "400": 13218789376.0,
+ "405": 13218789376.0,
+ "410": 13218789376.0,
+ "415": 13218789376.0,
+ "420": 13218789376.0,
+ "425": 13218789376.0,
+ "430": 13218789376.0,
+ "435": 13218789376.0,
+ "440": 13218789376.0,
+ "445": 13218789376.0,
+ "450": 13218789376.0,
+ "455": 13218789376.0,
+ "460": 13218789376.0,
+ "465": 13218789376.0,
+ "470": 13218789376.0,
+ "475": 13218789376.0,
+ "480": 13218789376.0,
+ "485": 13218789376.0,
+ "490": 13218789376.0,
+ "495": 13218789376.0,
+ "500": 13218789376.0,
+ "505": 13218789376.0,
+ "510": 13218789376.0,
+ "515": 13218789376.0,
+ "520": 13218789376.0,
+ "525": 13218789376.0,
+ "530": 13218789376.0,
+ "535": 13218789376.0,
+ "540": 13218789376.0,
+ "545": 13218789376.0,
+ "550": 13218789376.0,
+ "555": 13218789376.0,
+ "560": 13218789376.0,
+ "565": 13218789376.0,
+ "570": 13218789376.0,
+ "575": 13218789376.0,
+ "580": 13218789376.0,
+ "585": 13218789376.0,
+ "590": 13218789376.0,
+ "595": 13218789376.0,
+ "600": 13218789376.0,
+ "605": 13218789376.0,
+ "610": 13218789376.0,
+ "615": 13218789376.0,
+ "620": 13218789376.0,
+ "625": 13218789376.0,
+ "630": 13218789376.0,
+ "635": 13218789376.0,
+ "640": 13218789376.0,
+ "645": 13218789376.0,
+ "650": 13218789376.0,
+ "655": 13218789376.0,
+ "660": 13218789376.0,
+ "665": 13218789376.0,
+ "670": 13218789376.0,
+ "675": 13218789376.0,
+ "680": 13218789376.0,
+ "685": 13218789376.0,
+ "690": 13218789376.0,
+ "695": 13218789376.0,
+ "700": 13218789376.0,
+ "705": 13218789376.0,
+ "710": 13218789376.0,
+ "715": 13218789376.0,
+ "720": 13218789376.0,
+ "725": 13218789376.0,
+ "730": 13218789376.0,
+ "735": 13218789376.0,
+ "740": 13218789376.0,
+ "745": 13218789376.0,
+ "750": 13218789376.0,
+ "755": 13218789376.0,
+ "760": 13218789376.0,
+ "765": 13218789376.0,
+ "770": 13218789376.0,
+ "775": 13218789376.0,
+ "780": 13218789376.0,
+ "785": 13218789376.0,
+ "790": 13218789376.0,
+ "795": 13218789376.0,
+ "800": 13218789376.0,
+ "805": 13218789376.0,
+ "810": 13218789376.0,
+ "815": 13218789376.0,
+ "820": 13218789376.0,
+ "825": 13218789376.0,
+ "830": 13218789376.0,
+ "835": 13218789376.0,
+ "840": 13218789376.0,
+ "845": 13218789376.0,
+ "850": 13218789376.0,
+ "855": 13218789376.0,
+ "860": 13218789376.0,
+ "865": 13218789376.0,
+ "870": 13218789376.0,
+ "875": 13218789376.0,
+ "880": 13218789376.0,
+ "885": 13218789376.0,
+ "890": 13218789376.0,
+ "895": 13218789376.0,
+ "900": 13218789376.0,
+ "905": 13218789376.0,
+ "910": 13218789376.0,
+ "915": 13218789376.0,
+ "920": 13218789376.0,
+ "925": 13218789376.0,
+ "930": 13218789376.0,
+ "935": 13218789376.0,
+ "940": 13218789376.0,
+ "945": 13218789376.0,
+ "950": 13218789376.0,
+ "955": 13218789376.0,
+ "960": 13218789376.0,
+ "965": 13218789376.0,
+ "970": 13218789376.0,
+ "975": 13218789376.0,
+ "980": 13218789376.0,
+ "985": 13218789376.0,
+ "990": 13218789376.0,
+ "995": 13218789376.0,
+ "1000": 13218789376.0,
+ "1005": 13218789376.0,
+ "1010": 13218789376.0,
+ "1015": 13218789376.0,
+ "1020": 13218789376.0,
+ "1025": 13218789376.0,
+ "1030": 13218789376.0,
+ "1035": 13218789376.0,
+ "1040": 13218789376.0,
+ "1045": 13218789376.0,
+ "1050": 13218789376.0,
+ "1055": 13218789376.0,
+ "1060": 13218789376.0,
+ "1065": 13218789376.0,
+ "1070": 13218789376.0,
+ "1075": 13218789376.0,
+ "1080": 13218789376.0,
+ "1085": 13218789376.0,
+ "1090": 13218789376.0,
+ "1095": 13218789376.0,
+ "1100": 13218789376.0,
+ "1105": 13218789376.0,
+ "1110": 13218789376.0,
+ "1115": 13218789376.0,
+ "1120": 13218789376.0,
+ "1125": 13218789376.0,
+ "1130": 13218789376.0,
+ "1135": 13218789376.0,
+ "1140": 13218789376.0,
+ "1145": 13218789376.0,
+ "1150": 13218789376.0,
+ "1155": 13218789376.0,
+ "1160": 13218789376.0,
+ "1165": 13218789376.0,
+ "1170": 13218789376.0,
+ "1175": 13218789376.0,
+ "1180": 13218789376.0,
+ "1185": 13218789376.0,
+ "1190": 13218789376.0,
+ "1195": 13218789376.0,
+ "1200": 13218789376.0,
+ "1205": 13218789376.0,
+ "1210": 13218789376.0,
+ "1215": 13218789376.0,
+ "1220": 13218789376.0,
+ "1225": 13218789376.0,
+ "1230": 13218789376.0,
+ "1235": 13218789376.0,
+ "1240": 13218789376.0,
+ "1245": 13218789376.0,
+ "1250": 13218789376.0,
+ "1255": 13218789376.0,
+ "1260": 13218789376.0,
+ "1265": 13218789376.0,
+ "1270": 13218789376.0,
+ "1275": 13218789376.0,
+ "1280": 13218789376.0,
+ "1285": 13218789376.0,
+ "1290": 13218789376.0,
+ "1295": 13218789376.0,
+ "1300": 13218789376.0,
+ "1305": 13218789376.0,
+ "1310": 13218789376.0,
+ "1315": 13218789376.0,
+ "1320": 13218789376.0,
+ "1325": 13218789376.0,
+ "1330": 13218789376.0,
+ "1335": 13218789376.0,
+ "1340": 13218789376.0,
+ "1345": 13218789376.0,
+ "1350": 13218789376.0,
+ "1355": 13218789376.0,
+ "1360": 13218789376.0,
+ "1365": 13218789376.0,
+ "1370": 13218789376.0,
+ "1375": 13218789376.0,
+ "1380": 13218789376.0,
+ "1385": 13218789376.0,
+ "1390": 13218789376.0,
+ "1395": 13218789376.0,
+ "1400": 13218789376.0,
+ "1405": 13218789376.0,
+ "1410": 13218789376.0,
+ "1415": 13218789376.0,
+ "1420": 13218789376.0,
+ "1425": 13218789376.0,
+ "1430": 13218789376.0,
+ "1435": 13218789376.0,
+ "1440": 13218789376.0,
+ "1445": 13218789376.0,
+ "1450": 13218789376.0,
+ "1455": 13218789376.0,
+ "1460": 13218789376.0,
+ "1465": 13218789376.0,
+ "1470": 13218789376.0,
+ "1475": 13218789376.0,
+ "1480": 13218789376.0,
+ "1485": 13218789376.0,
+ "1490": 13218789376.0,
+ "1495": 13218789376.0,
+ "1500": 13218789376.0,
+ "1505": 13218789376.0,
+ "1510": 13218789376.0,
+ "1515": 13218789376.0,
+ "1520": 13218789376.0,
+ "1525": 13218789376.0,
+ "1530": 13218789376.0,
+ "1535": 13218789376.0,
+ "1540": 13218789376.0,
+ "1545": 13218789376.0,
+ "1550": 13218789376.0,
+ "1555": 13218789376.0,
+ "1560": 13218789376.0,
+ "1565": 13218789376.0,
+ "1570": 13218789376.0,
+ "1575": 13218789376.0,
+ "1580": 13218789376.0,
+ "1585": 13218789376.0,
+ "1590": 13218789376.0,
+ "1595": 13218789376.0,
+ "1600": 13218789376.0,
+ "1605": 13218789376.0,
+ "1610": 13218789376.0,
+ "1615": 13218789376.0,
+ "1620": 13218789376.0,
+ "1625": 13218789376.0,
+ "1630": 13218789376.0,
+ "1635": 13218789376.0,
+ "1640": 13218789376.0,
+ "1645": 13218789376.0,
+ "1650": 13218789376.0,
+ "1655": 13218789376.0,
+ "1660": 13218789376.0,
+ "1665": 13218789376.0,
+ "1670": 13218789376.0,
+ "1675": 13218789376.0,
+ "1680": 13218789376.0,
+ "1685": 13218789376.0,
+ "1690": 13218789376.0,
+ "1695": 13218789376.0,
+ "1700": 13218789376.0,
+ "1705": 13218789376.0,
+ "1710": 13218789376.0,
+ "1715": 13218789376.0,
+ "1720": 13218789376.0,
+ "1725": 13218789376.0,
+ "1730": 13218789376.0,
+ "1735": 13218789376.0,
+ "1740": 13218789376.0,
+ "1745": 13218789376.0,
+ "1750": 13218789376.0,
+ "1755": 13218789376.0,
+ "1760": 13218789376.0,
+ "1765": 13218789376.0,
+ "1770": 13218789376.0,
+ "1775": 13218789376.0,
+ "1780": 13218789376.0,
+ "1785": 13218789376.0,
+ "1790": 13218789376.0,
+ "1795": 13218789376.0,
+ "1800": 13218789376.0,
+ "1805": 13218789376.0,
+ "1810": 13218789376.0,
+ "1815": 13218789376.0,
+ "1820": 13218789376.0,
+ "1825": 13218789376.0,
+ "1830": 13218789376.0,
+ "1835": 13218789376.0,
+ "1840": 13218789376.0,
+ "1845": 13218789376.0,
+ "1850": 13218789376.0,
+ "1855": 13218789376.0,
+ "1860": 13218789376.0,
+ "1865": 13218789376.0,
+ "1870": 13218789376.0,
+ "1875": 13218789376.0,
+ "1880": 13218789376.0,
+ "1885": 13218789376.0,
+ "1890": 13218789376.0,
+ "1895": 13218789376.0,
+ "1900": 13218789376.0,
+ "1905": 13218789376.0,
+ "1910": 13218789376.0,
+ "1915": 13218789376.0,
+ "1920": 13218789376.0,
+ "1925": 13218789376.0,
+ "1930": 13218789376.0,
+ "1935": 13218789376.0,
+ "1940": 13218789376.0,
+ "1945": 13218789376.0,
+ "1950": 13218789376.0,
+ "1955": 13218789376.0,
+ "1960": 13218789376.0,
+ "1965": 13218789376.0,
+ "1970": 13218789376.0,
+ "1975": 13218789376.0,
+ "1980": 13218789376.0,
+ "1985": 13218789376.0,
+ "1990": 13218789376.0,
+ "1995": 13218789376.0,
+ "2000": 13218789376.0,
+ "2005": 13218789376.0,
+ "2010": 13218789376.0,
+ "2015": 13218789376.0,
+ "2020": 13218789376.0,
+ "2025": 13218789376.0,
+ "2030": 13218789376.0,
+ "2035": 13218789376.0,
+ "2040": 13218789376.0,
+ "2045": 13218789376.0,
+ "2050": 13218789376.0,
+ "2055": 13218789376.0,
+ "2060": 13218789376.0,
+ "2065": 13218789376.0,
+ "2070": 13218789376.0,
+ "2075": 13218789376.0,
+ "2080": 13218789376.0,
+ "2085": 13218789376.0,
+ "2090": 13218789376.0,
+ "2095": 13218789376.0,
+ "2100": 13218789376.0,
+ "2105": 13218789376.0,
+ "2110": 13218789376.0,
+ "2115": 13218789376.0,
+ "2120": 13218789376.0,
+ "2125": 13218789376.0,
+ "2130": 13218789376.0,
+ "2135": 13218789376.0,
+ "2140": 13218789376.0,
+ "2145": 13218789376.0,
+ "2150": 13218789376.0,
+ "2155": 13218789376.0,
+ "2160": 13218789376.0,
+ "2165": 13218789376.0,
+ "2170": 13218789376.0,
+ "2175": 13218789376.0,
+ "2180": 13218789376.0,
+ "2185": 13218789376.0,
+ "2190": 13218789376.0,
+ "2195": 13218789376.0,
+ "2200": 13218789376.0,
+ "2205": 13218789376.0,
+ "2210": 13218789376.0,
+ "2215": 13218789376.0,
+ "2220": 13218789376.0,
+ "2225": 13218789376.0,
+ "2230": 13218789376.0,
+ "2235": 13218789376.0,
+ "2240": 13218789376.0,
+ "2245": 13218789376.0,
+ "2250": 13218789376.0,
+ "2255": 13218789376.0,
+ "2260": 13218789376.0,
+ "2265": 13218789376.0,
+ "2270": 13218789376.0,
+ "2275": 13218789376.0,
+ "2280": 13218789376.0,
+ "2285": 13218789376.0,
+ "2290": 13218789376.0,
+ "2295": 13218789376.0,
+ "2300": 13218789376.0,
+ "2305": 13218789376.0,
+ "2310": 13218789376.0,
+ "2315": 13218789376.0,
+ "2320": 13218789376.0,
+ "2325": 13218789376.0,
+ "2330": 13218789376.0,
+ "2335": 13218789376.0,
+ "2340": 13218789376.0,
+ "2345": 13218789376.0,
+ "2350": 13218789376.0,
+ "2355": 13218789376.0,
+ "2360": 13218789376.0,
+ "2365": 13218789376.0,
+ "2370": 13218789376.0,
+ "2375": 13218789376.0,
+ "2380": 13218789376.0,
+ "2385": 13218789376.0,
+ "2390": 13218789376.0,
+ "2395": 13218789376.0,
+ "2400": 13218789376.0,
+ "2405": 13218789376.0,
+ "2410": 13218789376.0,
+ "2415": 13218789376.0,
+ "2420": 13218789376.0,
+ "2425": 13218789376.0,
+ "2430": 13218789376.0,
+ "2435": 13218789376.0,
+ "2440": 13218789376.0,
+ "2445": 13218789376.0,
+ "2450": 13218789376.0,
+ "2455": 13218789376.0,
+ "2460": 13218789376.0,
+ "2465": 13218789376.0,
+ "2470": 13218789376.0,
+ "2475": 13218789376.0,
+ "2480": 13218789376.0,
+ "2485": 13218789376.0,
+ "2490": 13218789376.0,
+ "2495": 13218789376.0,
+ "2500": 13218789376.0,
+ "2505": 13218789376.0,
+ "2510": 13218789376.0,
+ "2515": 13218789376.0,
+ "2520": 13218789376.0,
+ "2525": 13218789376.0,
+ "2530": 13218789376.0,
+ "2535": 13218789376.0,
+ "2540": 13218789376.0,
+ "2545": 13218789376.0,
+ "2550": 13218789376.0,
+ "2555": 13218789376.0,
+ "2560": 13218789376.0,
+ "2565": 13218789376.0,
+ "2570": 13218789376.0,
+ "2575": 13218789376.0,
+ "2580": 13218789376.0,
+ "2585": 13218789376.0,
+ "2590": 13218789376.0,
+ "2595": 13218789376.0,
+ "2600": 13218789376.0,
+ "2605": 13218789376.0,
+ "2610": 13218789376.0,
+ "2615": 13218789376.0,
+ "2620": 13218789376.0,
+ "2625": 13218789376.0,
+ "2630": 13218789376.0,
+ "2635": 13218789376.0,
+ "2640": 13218789376.0,
+ "2645": 13218789376.0,
+ "2650": 13218789376.0,
+ "2655": 13218789376.0,
+ "2660": 13218789376.0,
+ "2665": 13218789376.0,
+ "2670": 13218789376.0,
+ "2675": 13218789376.0,
+ "2680": 13218789376.0,
+ "2685": 13218789376.0,
+ "2690": 13218789376.0,
+ "2695": 13218789376.0,
+ "2700": 13218789376.0,
+ "2705": 13218789376.0,
+ "2710": 13218789376.0,
+ "2715": 13218789376.0,
+ "2720": 13218789376.0,
+ "2725": 13218789376.0,
+ "2730": 13218789376.0,
+ "2735": 13218789376.0,
+ "2740": 13218789376.0,
+ "2745": 13218789376.0,
+ "2750": 13218789376.0,
+ "2755": 13218789376.0,
+ "2760": 13218789376.0,
+ "2765": 13218789376.0,
+ "2770": 13218789376.0,
+ "2775": 13218789376.0,
+ "2780": 13218789376.0,
+ "2785": 13218789376.0,
+ "2790": 13218789376.0,
+ "2795": 13218789376.0,
+ "2800": 13218789376.0,
+ "2805": 13218789376.0,
+ "2810": 13218789376.0,
+ "2815": 13218789376.0,
+ "2820": 13218789376.0,
+ "2825": 13218789376.0,
+ "2830": 13218789376.0,
+ "2835": 13218789376.0,
+ "2840": 13218789376.0,
+ "2845": 13218789376.0,
+ "2850": 13218789376.0,
+ "2855": 13218789376.0,
+ "2860": 13218789376.0,
+ "2865": 13218789376.0,
+ "2870": 13218789376.0,
+ "2875": 13218789376.0,
+ "2880": 13218789376.0,
+ "2885": 13218789376.0,
+ "2890": 13218789376.0,
+ "2895": 13218789376.0,
+ "2900": 13218789376.0,
+ "2905": 13218789376.0,
+ "2910": 13218789376.0,
+ "2915": 13218789376.0,
+ "2920": 13218789376.0,
+ "2925": 13218789376.0,
+ "2930": 13218789376.0,
+ "2935": 13218789376.0,
+ "2940": 13218789376.0,
+ "2945": 13218789376.0,
+ "2950": 13218789376.0,
+ "2955": 13218789376.0,
+ "2960": 13218789376.0,
+ "2965": 13218789376.0,
+ "2970": 13218789376.0,
+ "2975": 13218789376.0,
+ "2980": 13218789376.0,
+ "2985": 13218789376.0,
+ "2990": 13218789376.0,
+ "2995": 13218789376.0,
+ "3000": 13218789376.0,
+ "3005": 13218789376.0,
+ "3010": 13218789376.0,
+ "3015": 13218789376.0,
+ "3020": 13218789376.0,
+ "3025": 13218789376.0,
+ "3030": 13218789376.0,
+ "3035": 13218789376.0,
+ "3040": 13218789376.0,
+ "3045": 13218789376.0,
+ "3050": 13218789376.0,
+ "3055": 13218789376.0,
+ "3060": 13218789376.0,
+ "3065": 13218789376.0,
+ "3070": 13218789376.0,
+ "3075": 13218789376.0,
+ "3080": 13218789376.0,
+ "3085": 13218789376.0,
+ "3090": 13218789376.0,
+ "3095": 13218789376.0,
+ "3100": 13218789376.0,
+ "3105": 13218789376.0,
+ "3110": 13218789376.0,
+ "3115": 13218789376.0,
+ "3120": 13218789376.0,
+ "3125": 13218789376.0,
+ "3130": 13218789376.0,
+ "3135": 13218789376.0,
+ "3140": 13218789376.0,
+ "3145": 13218789376.0,
+ "3150": 13218789376.0,
+ "3155": 13218789376.0,
+ "3160": 13218789376.0,
+ "3165": 13218789376.0,
+ "3170": 13218789376.0,
+ "3175": 13218789376.0,
+ "3180": 13218789376.0,
+ "3185": 13218789376.0,
+ "3190": 13218789376.0,
+ "3195": 13218789376.0,
+ "3200": 13218789376.0,
+ "3205": 13218789376.0,
+ "3210": 13218789376.0,
+ "3215": 13218789376.0,
+ "3220": 13218789376.0,
+ "3225": 13218789376.0,
+ "3230": 13218789376.0,
+ "3235": 13218789376.0,
+ "3240": 13218789376.0,
+ "3245": 13218789376.0,
+ "3250": 13218789376.0,
+ "3255": 13218789376.0,
+ "3260": 13218789376.0,
+ "3265": 13218789376.0,
+ "3270": 13218789376.0,
+ "3275": 13218789376.0,
+ "3280": 13218789376.0,
+ "3285": 13218789376.0,
+ "3290": 13218789376.0,
+ "3295": 13218789376.0,
+ "3300": 13218789376.0,
+ "3305": 13218789376.0,
+ "3310": 13218789376.0,
+ "3315": 13218789376.0,
+ "3320": 13218789376.0,
+ "3325": 13218789376.0,
+ "3330": 13218789376.0,
+ "3335": 13218789376.0,
+ "3340": 13218789376.0,
+ "3345": 13218789376.0,
+ "3350": 13218789376.0,
+ "3355": 13218789376.0,
+ "3360": 13218789376.0,
+ "3365": 13218789376.0,
+ "3370": 13218789376.0,
+ "3375": 13218789376.0,
+ "3380": 13218789376.0,
+ "3385": 13218789376.0,
+ "3390": 13218789376.0,
+ "3395": 13218789376.0,
+ "3400": 13218789376.0,
+ "3405": 13218789376.0,
+ "3410": 13218789376.0,
+ "3415": 13218789376.0,
+ "3420": 13218789376.0,
+ "3425": 13218789376.0,
+ "3430": 13218789376.0,
+ "3435": 13218789376.0,
+ "3440": 13218789376.0,
+ "3445": 13218789376.0,
+ "3450": 13218789376.0,
+ "3455": 13218789376.0,
+ "3460": 13218789376.0,
+ "3465": 13218789376.0,
+ "3470": 13218789376.0,
+ "3475": 13218789376.0,
+ "3480": 13218789376.0,
+ "3485": 13218789376.0,
+ "3490": 13218789376.0,
+ "3495": 13218789376.0,
+ "3500": 13218789376.0,
+ "3505": 13218789376.0,
+ "3510": 13218789376.0,
+ "3515": 13218789376.0,
+ "3520": 13218789376.0,
+ "3525": 13218789376.0,
+ "3530": 13218789376.0,
+ "3535": 13218789376.0,
+ "3540": 13218789376.0,
+ "3545": 13218789376.0,
+ "3550": 13218789376.0,
+ "3555": 13218789376.0,
+ "3560": 13218789376.0,
+ "3565": 13218789376.0,
+ "3570": 13218789376.0,
+ "3575": 13218789376.0,
+ "3580": 13218789376.0,
+ "3585": 13218789376.0,
+ "3590": 13218789376.0,
+ "3595": 13218789376.0,
+ "3600": 13218789376.0,
+ "3605": 13218789376.0,
+ "3610": 13218789376.0,
+ "3615": 13218789376.0,
+ "3620": 13218789376.0,
+ "3625": 13218789376.0,
+ "3630": 13218789376.0,
+ "3635": 13218789376.0,
+ "3640": 13218789376.0,
+ "3645": 13218789376.0,
+ "3650": 13218789376.0,
+ "3655": 13218789376.0,
+ "3660": 13218789376.0,
+ "3665": 13218789376.0,
+ "3670": 13218789376.0,
+ "3675": 13218789376.0,
+ "3680": 13218789376.0,
+ "3685": 13218789376.0,
+ "3690": 13218789376.0,
+ "3695": 13218789376.0,
+ "3700": 13218789376.0,
+ "3705": 13218789376.0,
+ "3710": 13218789376.0,
+ "3715": 13218789376.0,
+ "3720": 13218789376.0,
+ "3725": 13218789376.0,
+ "3730": 13218789376.0,
+ "3735": 13218789376.0,
+ "3740": 13218789376.0,
+ "3745": 13218789376.0,
+ "3750": 13218789376.0,
+ "3755": 13218789376.0,
+ "3760": 13218789376.0,
+ "3765": 13218789376.0,
+ "3770": 13218789376.0,
+ "3775": 13218789376.0,
+ "3780": 13218789376.0,
+ "3785": 13218789376.0,
+ "3790": 13218789376.0,
+ "3795": 13218789376.0,
+ "3800": 13218789376.0,
+ "3805": 13218789376.0,
+ "3810": 13218789376.0,
+ "3815": 13218789376.0,
+ "3820": 13218789376.0,
+ "3825": 13218789376.0,
+ "3830": 13218789376.0,
+ "3835": 13218789376.0,
+ "3840": 13218789376.0,
+ "3845": 13218789376.0,
+ "3850": 13218789376.0,
+ "3855": 13218789376.0,
+ "3860": 13218789376.0,
+ "3865": 13218789376.0,
+ "3870": 13218789376.0,
+ "3875": 13218789376.0,
+ "3880": 13218789376.0,
+ "3885": 13218789376.0,
+ "3890": 13218789376.0,
+ "3895": 13218789376.0,
+ "3900": 13218789376.0,
+ "3905": 13218789376.0,
+ "3910": 13218789376.0,
+ "3915": 13218789376.0,
+ "3920": 13218789376.0,
+ "3925": 13218789376.0,
+ "3930": 13218789376.0,
+ "3935": 13218789376.0,
+ "3940": 13218789376.0,
+ "3945": 13218789376.0,
+ "3950": 13218789376.0,
+ "3955": 13218789376.0,
+ "3960": 13218789376.0,
+ "3965": 13218789376.0,
+ "3970": 13218789376.0,
+ "3975": 13218789376.0,
+ "3980": 13218789376.0,
+ "3985": 13218789376.0,
+ "3990": 13218789376.0,
+ "3995": 13218789376.0,
+ "4000": 13218789376.0,
+ "4005": 13218789376.0,
+ "4010": 13218789376.0,
+ "4015": 13218789376.0,
+ "4020": 13218789376.0,
+ "4025": 13218789376.0,
+ "4030": 13218789376.0,
+ "4035": 13218789376.0,
+ "4040": 13218789376.0,
+ "4045": 13218789376.0,
+ "4050": 13218789376.0,
+ "4055": 13218789376.0,
+ "4060": 13218789376.0,
+ "4065": 13218789376.0,
+ "4070": 13218789376.0,
+ "4075": 13218789376.0,
+ "4080": 13218789376.0,
+ "4085": 13218789376.0,
+ "4090": 13218789376.0,
+ "4095": 13218789376.0,
+ "4100": 13218789376.0,
+ "4105": 13218789376.0,
+ "4110": 13218789376.0,
+ "4115": 13218789376.0,
+ "4120": 13218789376.0,
+ "4125": 13218789376.0,
+ "4130": 13218789376.0,
+ "4135": 13218789376.0,
+ "4140": 13218789376.0,
+ "4145": 13218789376.0,
+ "4150": 13218789376.0,
+ "4155": 13218789376.0,
+ "4160": 13218789376.0,
+ "4165": 13218789376.0,
+ "4170": 13218789376.0,
+ "4175": 13218789376.0,
+ "4180": 13218789376.0,
+ "4185": 13218789376.0,
+ "4190": 13218789376.0,
+ "4195": 13218789376.0,
+ "4200": 13218789376.0,
+ "4205": 13218789376.0,
+ "4210": 13218789376.0,
+ "4215": 13218789376.0,
+ "4220": 13218789376.0,
+ "4225": 13218789376.0,
+ "4230": 13218789376.0,
+ "4235": 13218789376.0,
+ "4240": 13218789376.0,
+ "4245": 13218789376.0,
+ "4250": 13218789376.0,
+ "4255": 13218789376.0,
+ "4260": 13218789376.0,
+ "4265": 13218789376.0,
+ "4270": 13218789376.0,
+ "4275": 13218789376.0,
+ "4280": 13218789376.0,
+ "4285": 13218789376.0,
+ "4290": 13218789376.0,
+ "4295": 13218789376.0,
+ "4300": 13218789376.0,
+ "4305": 13218789376.0,
+ "4310": 13218789376.0,
+ "4315": 13218789376.0,
+ "4320": 13218789376.0,
+ "4325": 13218789376.0,
+ "4330": 13218789376.0,
+ "4335": 13218789376.0,
+ "4340": 13218789376.0,
+ "4345": 13218789376.0,
+ "4350": 13218789376.0,
+ "4355": 13218789376.0,
+ "4360": 13218789376.0,
+ "4365": 13218789376.0,
+ "4370": 13218789376.0,
+ "4375": 13218789376.0,
+ "4380": 13218789376.0,
+ "4385": 13218789376.0,
+ "4390": 13218789376.0,
+ "4395": 13218789376.0,
+ "4400": 13218789376.0,
+ "4405": 13218789376.0,
+ "4410": 13218789376.0,
+ "4415": 13218789376.0,
+ "4420": 13218789376.0,
+ "4425": 13218789376.0,
+ "4430": 13218789376.0,
+ "4435": 13218789376.0,
+ "4440": 13218789376.0,
+ "4445": 13218789376.0,
+ "4450": 13218789376.0,
+ "4455": 13218789376.0,
+ "4460": 13218789376.0,
+ "4465": 13218789376.0,
+ "4470": 13218789376.0,
+ "4475": 13218789376.0,
+ "4480": 13218789376.0,
+ "4485": 13218789376.0,
+ "4490": 13218789376.0,
+ "4495": 13218789376.0,
+ "4500": 13218789376.0,
+ "4505": 13218789376.0,
+ "4510": 13218789376.0,
+ "4515": 13218789376.0,
+ "4520": 13218789376.0,
+ "4525": 13218789376.0,
+ "4530": 13218789376.0,
+ "4535": 13218789376.0,
+ "4540": 13218789376.0,
+ "4545": 13218789376.0,
+ "4550": 13218789376.0,
+ "4555": 13218789376.0,
+ "4560": 13218789376.0,
+ "4565": 13218789376.0,
+ "4570": 13218789376.0,
+ "4575": 13218789376.0,
+ "4580": 13218789376.0,
+ "4585": 13218789376.0,
+ "4590": 13218789376.0,
+ "4595": 13218789376.0,
+ "4600": 13218789376.0,
+ "4605": 13218789376.0,
+ "4610": 13218789376.0,
+ "4615": 13218789376.0,
+ "4620": 13218789376.0,
+ "4625": 13218789376.0,
+ "4630": 13218789376.0,
+ "4635": 13218789376.0,
+ "4640": 13218789376.0,
+ "4645": 13218789376.0,
+ "4650": 13218789376.0,
+ "4655": 13218789376.0,
+ "4660": 13218789376.0,
+ "4665": 13218789376.0,
+ "4670": 13218789376.0,
+ "4675": 13218789376.0,
+ "4680": 13218789376.0,
+ "4685": 13218789376.0,
+ "4690": 13218789376.0,
+ "4695": 13218789376.0,
+ "4700": 13218789376.0,
+ "4705": 13218789376.0,
+ "4710": 13218789376.0,
+ "4715": 13218789376.0,
+ "4720": 13218789376.0,
+ "4725": 13218789376.0,
+ "4730": 13218789376.0,
+ "4735": 13218789376.0,
+ "4740": 13218789376.0,
+ "4745": 13218789376.0,
+ "4750": 13218789376.0,
+ "4755": 13218789376.0,
+ "4760": 13218789376.0,
+ "4765": 13218789376.0,
+ "4770": 13218789376.0,
+ "4775": 13218789376.0,
+ "4780": 13218789376.0,
+ "4785": 13218789376.0,
+ "4790": 13218789376.0,
+ "4795": 13218789376.0,
+ "4800": 13218789376.0,
+ "4805": 13218789376.0,
+ "4810": 13218789376.0,
+ "4815": 13218789376.0,
+ "4820": 13218789376.0,
+ "4825": 13218789376.0,
+ "4830": 13218789376.0,
+ "4835": 13218789376.0,
+ "4840": 13218789376.0,
+ "4845": 13218789376.0,
+ "4850": 13218789376.0,
+ "4855": 13218789376.0,
+ "4860": 13218789376.0,
+ "4865": 13218789376.0,
+ "4870": 13218789376.0,
+ "4875": 13218789376.0,
+ "4880": 13218789376.0,
+ "4885": 13218789376.0,
+ "4890": 13218789376.0,
+ "4895": 13218789376.0,
+ "4900": 13218789376.0,
+ "4905": 13218789376.0,
+ "4910": 13218789376.0,
+ "4915": 13218789376.0,
+ "4920": 13218789376.0,
+ "4925": 13218789376.0,
+ "4930": 13218789376.0,
+ "4935": 13218789376.0,
+ "4940": 13218789376.0,
+ "4945": 13218789376.0,
+ "4950": 13218789376.0,
+ "4955": 13218789376.0,
+ "4960": 13218789376.0,
+ "4965": 13218789376.0,
+ "4970": 13218789376.0,
+ "4975": 13218789376.0,
+ "4980": 13218789376.0,
+ "4985": 13218789376.0,
+ "4990": 13218789376.0,
+ "4995": 13218789376.0,
+ "5000": 13218789376.0,
+ "5005": 13218789376.0,
+ "5010": 13218789376.0,
+ "5015": 13218789376.0,
+ "5020": 13218789376.0,
+ "5025": 13218789376.0,
+ "5030": 13218789376.0,
+ "5035": 13218789376.0,
+ "5040": 13218789376.0,
+ "5045": 13218789376.0,
+ "5050": 13218789376.0,
+ "5055": 13218789376.0,
+ "5060": 13218789376.0,
+ "5065": 13218789376.0,
+ "5070": 13218789376.0,
+ "5075": 13218789376.0,
+ "5080": 13218789376.0,
+ "5085": 13218789376.0,
+ "5090": 13218789376.0,
+ "5095": 13218789376.0,
+ "5100": 13218789376.0,
+ "5105": 13218789376.0,
+ "5110": 13218789376.0,
+ "5115": 13218789376.0,
+ "5120": 13218789376.0,
+ "5125": 13218789376.0,
+ "5130": 13218789376.0,
+ "5135": 13218789376.0,
+ "5140": 13218789376.0,
+ "5145": 13218789376.0,
+ "5150": 13218789376.0,
+ "5155": 13218789376.0,
+ "5160": 13218789376.0,
+ "5165": 13218789376.0,
+ "5170": 13218789376.0,
+ "5175": 13218789376.0,
+ "5180": 13218789376.0,
+ "5185": 13218789376.0,
+ "5190": 13218789376.0,
+ "5195": 13218789376.0,
+ "5200": 13218789376.0,
+ "5205": 13218789376.0,
+ "5210": 13218789376.0,
+ "5215": 13218789376.0,
+ "5220": 13218789376.0,
+ "5225": 13218789376.0,
+ "5230": 13218789376.0,
+ "5235": 13218789376.0,
+ "5240": 13218789376.0,
+ "5245": 13218789376.0,
+ "5250": 13218789376.0,
+ "5255": 13218789376.0,
+ "5260": 13218789376.0,
+ "5265": 13218789376.0,
+ "5270": 13218789376.0,
+ "5275": 13218789376.0,
+ "5280": 13218789376.0,
+ "5285": 13218789376.0,
+ "5290": 13218789376.0,
+ "5295": 13218789376.0,
+ "5300": 13218789376.0,
+ "5305": 13218789376.0,
+ "5310": 13218789376.0,
+ "5315": 13218789376.0,
+ "5320": 13218789376.0,
+ "5325": 13218789376.0,
+ "5330": 13218789376.0,
+ "5335": 13218789376.0,
+ "5340": 13218789376.0,
+ "5345": 13218789376.0,
+ "5350": 13218789376.0,
+ "5355": 13218789376.0,
+ "5360": 13218789376.0,
+ "5365": 13218789376.0,
+ "5370": 13218789376.0,
+ "5375": 13218789376.0,
+ "5380": 13218789376.0,
+ "5385": 13218789376.0,
+ "5390": 13218789376.0,
+ "5395": 13218789376.0,
+ "5400": 13218789376.0,
+ "5405": 13218789376.0,
+ "5410": 13218789376.0,
+ "5415": 13218789376.0,
+ "5420": 13218789376.0,
+ "5425": 13218789376.0,
+ "5430": 13218789376.0,
+ "5435": 13218789376.0,
+ "5440": 13218789376.0,
+ "5445": 13218789376.0,
+ "5450": 13218789376.0,
+ "5455": 13218789376.0,
+ "5460": 13218789376.0,
+ "5465": 13218789376.0,
+ "5470": 13218789376.0,
+ "5475": 13218789376.0,
+ "5480": 13218789376.0,
+ "5485": 13218789376.0,
+ "5490": 13218789376.0,
+ "5495": 13218789376.0,
+ "5500": 13218789376.0,
+ "5505": 13218789376.0,
+ "5510": 13218789376.0,
+ "5515": 13218789376.0,
+ "5520": 13218789376.0,
+ "5525": 13218789376.0,
+ "5530": 13218789376.0,
+ "5535": 13218789376.0,
+ "5540": 13218789376.0,
+ "5545": 13218789376.0,
+ "5550": 13218789376.0,
+ "5555": 13218789376.0,
+ "5560": 13218789376.0,
+ "5565": 13218789376.0,
+ "5570": 13218789376.0,
+ "5575": 13218789376.0,
+ "5580": 13218789376.0,
+ "5585": 13218789376.0,
+ "5590": 13218789376.0,
+ "5595": 13218789376.0,
+ "5600": 13218789376.0,
+ "5605": 13218789376.0,
+ "5610": 13218789376.0,
+ "5615": 13218789376.0,
+ "5620": 13218789376.0,
+ "5625": 13218789376.0,
+ "5630": 13218789376.0,
+ "5635": 13218789376.0,
+ "5640": 13218789376.0,
+ "5645": 13218789376.0,
+ "5650": 13218789376.0,
+ "5655": 13218789376.0,
+ "5660": 13218789376.0,
+ "5665": 13218789376.0,
+ "5670": 13218789376.0,
+ "5675": 13218789376.0,
+ "5680": 13218789376.0,
+ "5685": 13218789376.0,
+ "5690": 13218789376.0,
+ "5695": 13218789376.0,
+ "5700": 13218789376.0,
+ "5705": 13218789376.0,
+ "5710": 13218789376.0,
+ "5715": 13218789376.0,
+ "5720": 13218789376.0,
+ "5725": 13218789376.0,
+ "5730": 13218789376.0,
+ "5735": 13218789376.0,
+ "5740": 13218789376.0,
+ "5745": 13218789376.0,
+ "5750": 13218789376.0,
+ "5755": 13218789376.0,
+ "5760": 13218789376.0,
+ "5765": 13218789376.0,
+ "5770": 13218789376.0,
+ "5775": 13218789376.0,
+ "5780": 13218789376.0,
+ "5785": 13218789376.0,
+ "5790": 13218789376.0,
+ "5795": 13218789376.0,
+ "5800": 13218789376.0,
+ "5805": 13218789376.0,
+ "5810": 13218789376.0,
+ "5815": 13218789376.0,
+ "5820": 13218789376.0,
+ "5825": 13218789376.0,
+ "5830": 13218789376.0,
+ "5835": 13218789376.0,
+ "5840": 13218789376.0,
+ "5845": 13218789376.0,
+ "5850": 13218789376.0,
+ "5855": 13218789376.0,
+ "5860": 13218789376.0,
+ "5865": 13218789376.0,
+ "5870": 13218789376.0,
+ "5875": 13218789376.0,
+ "5880": 13218789376.0,
+ "5885": 13218789376.0,
+ "5890": 13218789376.0,
+ "5895": 13218789376.0,
+ "5900": 13218789376.0,
+ "5905": 13218789376.0,
+ "5910": 13218789376.0,
+ "5915": 13218789376.0,
+ "5920": 13218789376.0,
+ "5925": 13218789376.0,
+ "5930": 13218789376.0,
+ "5935": 13218789376.0,
+ "5940": 13218789376.0,
+ "5945": 13218789376.0,
+ "5950": 13218789376.0,
+ "5955": 13218789376.0,
+ "5960": 13218789376.0,
+ "5965": 13218789376.0,
+ "5970": 13218789376.0,
+ "5975": 13218789376.0,
+ "5980": 13218789376.0,
+ "5985": 13218789376.0,
+ "5990": 13218789376.0,
+ "5995": 13218789376.0,
+ "6000": 13218789376.0,
+ "6005": 13218789376.0,
+ "6010": 13218789376.0,
+ "6015": 13218789376.0,
+ "6020": 13218789376.0,
+ "6025": 13218789376.0,
+ "6030": 13218789376.0,
+ "6035": 13218789376.0,
+ "6040": 13218789376.0,
+ "6045": 13218789376.0,
+ "6050": 13218789376.0,
+ "6055": 13218789376.0,
+ "6060": 13218789376.0,
+ "6065": 13218789376.0,
+ "6070": 13218789376.0,
+ "6075": 13218789376.0,
+ "6080": 13218789376.0,
+ "6085": 13218789376.0,
+ "6090": 13218789376.0,
+ "6095": 13218789376.0,
+ "6100": 13218789376.0,
+ "6105": 13218789376.0,
+ "6110": 13218789376.0,
+ "6115": 13218789376.0,
+ "6120": 13218789376.0,
+ "6125": 13218789376.0,
+ "6130": 13218789376.0,
+ "6135": 13218789376.0,
+ "6140": 13218789376.0,
+ "6145": 13218789376.0,
+ "6150": 13218789376.0,
+ "6155": 13218789376.0,
+ "6160": 13218789376.0,
+ "6165": 13218789376.0,
+ "6170": 13218789376.0,
+ "6175": 13218789376.0,
+ "6180": 13218789376.0,
+ "6185": 13218789376.0,
+ "6190": 13218789376.0,
+ "6195": 13218789376.0,
+ "6200": 13218789376.0,
+ "6205": 13218789376.0,
+ "6210": 13218789376.0,
+ "6215": 13218789376.0,
+ "6220": 13218789376.0,
+ "6225": 13218789376.0,
+ "6230": 13218789376.0,
+ "6235": 13218789376.0,
+ "6240": 13218789376.0,
+ "6245": 13218789376.0,
+ "6250": 13218789376.0,
+ "6255": 13218789376.0,
+ "6260": 13218789376.0,
+ "6265": 13218789376.0,
+ "6270": 13218789376.0,
+ "6275": 13218789376.0,
+ "6280": 13218789376.0,
+ "6285": 13218789376.0,
+ "6290": 13218789376.0,
+ "6295": 13218789376.0,
+ "6300": 13218789376.0,
+ "6305": 13218789376.0,
+ "6310": 13218789376.0,
+ "6315": 13218789376.0,
+ "6320": 13218789376.0,
+ "6325": 13218789376.0,
+ "6330": 13218789376.0,
+ "6335": 13218789376.0,
+ "6340": 13218789376.0,
+ "6345": 13218789376.0,
+ "6350": 13218789376.0,
+ "6355": 13218789376.0,
+ "6360": 13218789376.0,
+ "6365": 13218789376.0,
+ "6370": 13218789376.0,
+ "6375": 13218789376.0,
+ "6380": 13218789376.0,
+ "6385": 13218789376.0,
+ "6390": 13218789376.0,
+ "6395": 13218789376.0,
+ "6400": 13218789376.0,
+ "6405": 13218789376.0,
+ "6410": 13218789376.0,
+ "6415": 13218789376.0,
+ "6420": 13218789376.0,
+ "6425": 13218789376.0,
+ "6430": 13218789376.0,
+ "6435": 13218789376.0,
+ "6440": 13218789376.0,
+ "6445": 13218789376.0,
+ "6450": 13218789376.0,
+ "6455": 13218789376.0,
+ "6460": 13218789376.0,
+ "6465": 13218789376.0,
+ "6470": 13218789376.0,
+ "6475": 13218789376.0,
+ "6480": 13218789376.0,
+ "6485": 13218789376.0,
+ "6490": 13218789376.0,
+ "6495": 13218789376.0,
+ "6500": 13218789376.0,
+ "6505": 13218789376.0,
+ "6510": 13218789376.0,
+ "6515": 13218789376.0,
+ "6520": 13218789376.0,
+ "6525": 13218789376.0,
+ "6530": 13218789376.0,
+ "6535": 13218789376.0,
+ "6540": 13218789376.0,
+ "6545": 13218789376.0,
+ "6550": 13218789376.0,
+ "6555": 13218789376.0,
+ "6560": 13218789376.0,
+ "6565": 13218789376.0,
+ "6570": 13218789376.0,
+ "6575": 13218789376.0,
+ "6580": 13218789376.0,
+ "6585": 13218789376.0,
+ "6590": 13218789376.0,
+ "6595": 13218789376.0,
+ "6600": 13218789376.0,
+ "6605": 13218789376.0,
+ "6610": 13218789376.0,
+ "6615": 13218789376.0,
+ "6620": 13218789376.0,
+ "6625": 13218789376.0,
+ "6630": 13218789376.0,
+ "6635": 13218789376.0,
+ "6640": 13218789376.0,
+ "6645": 13218789376.0,
+ "6650": 13218789376.0,
+ "6655": 13218789376.0,
+ "6660": 13218789376.0,
+ "6665": 13218789376.0,
+ "6670": 13218789376.0,
+ "6675": 13218789376.0,
+ "6680": 13218789376.0,
+ "6685": 13218789376.0,
+ "6690": 13218789376.0,
+ "6695": 13218789376.0,
+ "6700": 13218789376.0,
+ "6705": 13218789376.0,
+ "6710": 13218789376.0,
+ "6715": 13218789376.0,
+ "6720": 13218789376.0,
+ "6725": 13218789376.0,
+ "6730": 13218789376.0,
+ "6735": 13218789376.0,
+ "6740": 13218789376.0,
+ "6745": 13218789376.0,
+ "6750": 13218789376.0,
+ "6755": 13218789376.0,
+ "6760": 13218789376.0,
+ "6765": 13218789376.0,
+ "6770": 13218789376.0,
+ "6775": 13218789376.0,
+ "6780": 13218789376.0,
+ "6785": 13218789376.0,
+ "6790": 13218789376.0,
+ "6795": 13218789376.0,
+ "6800": 13218789376.0,
+ "6805": 13218789376.0,
+ "6810": 13218789376.0,
+ "6815": 13218789376.0,
+ "6820": 13218789376.0,
+ "6825": 13218789376.0,
+ "6830": 13218789376.0,
+ "6835": 13218789376.0,
+ "6840": 13218789376.0,
+ "6845": 13218789376.0,
+ "6850": 13218789376.0,
+ "6855": 13218789376.0,
+ "6860": 13218789376.0,
+ "6865": 13218789376.0,
+ "6870": 13218789376.0,
+ "6875": 13218789376.0,
+ "6880": 13218789376.0,
+ "6885": 13218789376.0,
+ "6890": 13218789376.0,
+ "6895": 13218789376.0,
+ "6900": 13218789376.0,
+ "6905": 13218789376.0,
+ "6910": 13218789376.0,
+ "6915": 13218789376.0,
+ "6920": 13218789376.0,
+ "6925": 13218789376.0,
+ "6930": 13218789376.0,
+ "6935": 13218789376.0,
+ "6940": 13218789376.0,
+ "6945": 13218789376.0,
+ "6950": 13218789376.0,
+ "6955": 13218789376.0,
+ "6960": 13218789376.0,
+ "6965": 13218789376.0,
+ "6970": 13218789376.0,
+ "6975": 13218789376.0,
+ "6980": 13218789376.0,
+ "6985": 13218789376.0,
+ "6990": 13218789376.0,
+ "6995": 13218789376.0,
+ "7000": 13218789376.0,
+ "7005": 13218789376.0,
+ "7010": 13218789376.0,
+ "7015": 13218789376.0,
+ "7020": 13218789376.0,
+ "7025": 13218789376.0,
+ "7030": 13218789376.0,
+ "7035": 13218789376.0,
+ "7040": 13218789376.0,
+ "7045": 13218789376.0,
+ "7050": 13218789376.0,
+ "7055": 13218789376.0,
+ "7060": 13218789376.0,
+ "7065": 13218789376.0,
+ "7070": 13218789376.0,
+ "7075": 13218789376.0,
+ "7080": 13218789376.0,
+ "7085": 13218789376.0,
+ "7090": 13218789376.0,
+ "7095": 13218789376.0,
+ "7100": 13218789376.0,
+ "7105": 13218789376.0,
+ "7110": 13218789376.0,
+ "7115": 13218789376.0,
+ "7120": 13218789376.0,
+ "7125": 13218789376.0,
+ "7130": 13218789376.0,
+ "7135": 13218789376.0,
+ "7140": 13218789376.0,
+ "7145": 13218789376.0,
+ "7150": 13218789376.0,
+ "7155": 13218789376.0,
+ "7160": 13218789376.0,
+ "7165": 13218789376.0,
+ "7170": 13218789376.0,
+ "7175": 13218789376.0,
+ "7180": 13218789376.0,
+ "7185": 13218789376.0,
+ "7190": 13218789376.0,
+ "7195": 13218789376.0,
+ "7200": 13218789376.0,
+ "7205": 13218789376.0,
+ "7210": 13218789376.0,
+ "7215": 13218789376.0,
+ "7220": 13218789376.0,
+ "7225": 13218789376.0,
+ "7230": 13218789376.0,
+ "7235": 13218789376.0,
+ "7240": 13218789376.0,
+ "7245": 13218789376.0,
+ "7250": 13218789376.0,
+ "7255": 13218789376.0,
+ "7260": 13218789376.0,
+ "7265": 13218789376.0,
+ "7270": 13218789376.0,
+ "7275": 13218789376.0,
+ "7280": 13218789376.0,
+ "7285": 13218789376.0,
+ "7290": 13218789376.0,
+ "7295": 13218789376.0,
+ "7300": 13218789376.0,
+ "7305": 13218789376.0,
+ "7310": 13218789376.0,
+ "7315": 13218789376.0,
+ "7320": 13218789376.0,
+ "7325": 13218789376.0,
+ "7330": 13218789376.0,
+ "7335": 13218789376.0,
+ "7340": 13218789376.0,
+ "7345": 13218789376.0,
+ "7350": 13218789376.0,
+ "7355": 13218789376.0,
+ "7360": 13218789376.0,
+ "7365": 13218789376.0,
+ "7370": 13218789376.0,
+ "7375": 13218789376.0,
+ "7380": 13218789376.0,
+ "7385": 13218789376.0,
+ "7390": 13218789376.0,
+ "7395": 13218789376.0,
+ "7400": 13218789376.0,
+ "7405": 13218789376.0,
+ "7410": 13218789376.0,
+ "7415": 13218789376.0,
+ "7420": 13218789376.0,
+ "7425": 13218789376.0,
+ "7430": 13218789376.0,
+ "7435": 13218789376.0,
+ "7440": 13218789376.0,
+ "7445": 13218789376.0,
+ "7450": 13218789376.0,
+ "7455": 13218789376.0,
+ "7460": 13218789376.0,
+ "7465": 13218789376.0,
+ "7470": 13218789376.0,
+ "7475": 13218789376.0,
+ "7480": 13218789376.0,
+ "7485": 13218789376.0,
+ "7490": 13218789376.0,
+ "7495": 13218789376.0,
+ "7500": 13218789376.0,
+ "7505": 13218789376.0,
+ "7510": 13218789376.0,
+ "7515": 13218789376.0,
+ "7520": 13218789376.0,
+ "7525": 13218789376.0,
+ "7530": 13218789376.0,
+ "7535": 13218789376.0,
+ "7540": 13218789376.0,
+ "7545": 13218789376.0,
+ "7550": 13218789376.0,
+ "7555": 13218789376.0,
+ "7560": 13218789376.0,
+ "7565": 13218789376.0,
+ "7570": 13218789376.0,
+ "7575": 13218789376.0,
+ "7580": 13218789376.0,
+ "7585": 13218789376.0,
+ "7590": 13218789376.0,
+ "7595": 13218789376.0,
+ "7600": 13218789376.0,
+ "7605": 13218789376.0,
+ "7610": 13218789376.0,
+ "7615": 13218789376.0,
+ "7620": 13218789376.0,
+ "7625": 13218789376.0,
+ "7630": 13218789376.0,
+ "7635": 13218789376.0,
+ "7640": 13218789376.0,
+ "7645": 13218789376.0,
+ "7650": 13218789376.0,
+ "7655": 13218789376.0,
+ "7660": 13218789376.0,
+ "7665": 13218789376.0,
+ "7670": 13218789376.0,
+ "7675": 13218789376.0,
+ "7680": 13218789376.0,
+ "7685": 13218789376.0,
+ "7690": 13218789376.0,
+ "7695": 13218789376.0,
+ "7700": 13218789376.0,
+ "7705": 13218789376.0,
+ "7710": 13218789376.0,
+ "7715": 13218789376.0,
+ "7720": 13218789376.0,
+ "7725": 13218789376.0,
+ "7730": 13218789376.0,
+ "7735": 13218789376.0,
+ "7740": 13218789376.0,
+ "7745": 13218789376.0,
+ "7750": 13218789376.0,
+ "7755": 13218789376.0,
+ "7760": 13218789376.0,
+ "7765": 13218789376.0,
+ "7770": 13218789376.0,
+ "7775": 13218789376.0,
+ "7780": 13218789376.0,
+ "7785": 13218789376.0,
+ "7790": 13218789376.0,
+ "7795": 13218789376.0,
+ "7800": 13218789376.0,
+ "7805": 13218789376.0,
+ "7810": 13218789376.0,
+ "7815": 13218789376.0,
+ "7820": 13218789376.0,
+ "7825": 13218789376.0,
+ "7830": 13218789376.0,
+ "7835": 13218789376.0,
+ "7840": 13218789376.0,
+ "7845": 13218789376.0,
+ "7850": 13218789376.0,
+ "7855": 13218789376.0,
+ "7860": 13218789376.0,
+ "7865": 13218789376.0,
+ "7870": 13218789376.0,
+ "7875": 13218789376.0,
+ "7880": 13218789376.0,
+ "7885": 13218789376.0,
+ "7890": 13218789376.0,
+ "7895": 13218789376.0,
+ "7900": 13218789376.0,
+ "7905": 13218789376.0,
+ "7910": 13218789376.0,
+ "7915": 13218789376.0,
+ "7920": 13218789376.0,
+ "7925": 13218789376.0,
+ "7930": 13218789376.0,
+ "7935": 13218789376.0,
+ "7940": 13218789376.0,
+ "7945": 13218789376.0,
+ "7950": 13218789376.0,
+ "7955": 13218789376.0,
+ "7960": 13218789376.0,
+ "7965": 13218789376.0,
+ "7970": 13218789376.0,
+ "7975": 13218789376.0,
+ "7980": 13218789376.0,
+ "7985": 13218789376.0,
+ "7990": 13218789376.0,
+ "7995": 13218789376.0,
+ "8000": 13218789376.0,
+ "8005": 13218789376.0,
+ "8010": 13218789376.0,
+ "8015": 13218789376.0,
+ "8020": 13218789376.0,
+ "8025": 13218789376.0,
+ "8030": 13218789376.0,
+ "8035": 13218789376.0,
+ "8040": 13218789376.0,
+ "8045": 13218789376.0,
+ "8050": 13218789376.0,
+ "8055": 13218789376.0,
+ "8060": 13218789376.0,
+ "8065": 13218789376.0,
+ "8070": 13218789376.0,
+ "8075": 13218789376.0,
+ "8080": 13218789376.0,
+ "8085": 13218789376.0,
+ "8090": 13218789376.0,
+ "8095": 13218789376.0,
+ "8100": 13218789376.0,
+ "8105": 13218789376.0,
+ "8110": 13218789376.0,
+ "8115": 13218789376.0,
+ "8120": 13218789376.0,
+ "8125": 13218789376.0,
+ "8130": 13218789376.0,
+ "8135": 13218789376.0,
+ "8140": 13218789376.0,
+ "8145": 13218789376.0,
+ "8150": 13218789376.0,
+ "8155": 13218789376.0,
+ "8160": 13218789376.0,
+ "8165": 13218789376.0,
+ "8170": 13218789376.0,
+ "8175": 13218789376.0,
+ "8180": 13218789376.0,
+ "8185": 13218789376.0,
+ "8190": 13218789376.0,
+ "8195": 13218789376.0,
+ "8200": 13218789376.0,
+ "8205": 13218789376.0,
+ "8210": 13218789376.0,
+ "8215": 13218789376.0,
+ "8220": 13218789376.0,
+ "8225": 13218789376.0,
+ "8230": 13218789376.0,
+ "8235": 13218789376.0,
+ "8240": 13218789376.0,
+ "8245": 13218789376.0,
+ "8250": 13218789376.0,
+ "8255": 13218789376.0,
+ "8260": 13218789376.0,
+ "8265": 13218789376.0,
+ "8270": 13218789376.0,
+ "8275": 13218789376.0,
+ "8280": 13218789376.0,
+ "8285": 13218789376.0,
+ "8290": 13218789376.0,
+ "8295": 13218789376.0,
+ "8300": 13218789376.0,
+ "8305": 13218789376.0,
+ "8310": 13218789376.0,
+ "8315": 13218789376.0,
+ "8320": 13218789376.0,
+ "8325": 13218789376.0,
+ "8330": 13218789376.0,
+ "8335": 13218789376.0,
+ "8340": 13218789376.0,
+ "8345": 13218789376.0,
+ "8350": 13218789376.0,
+ "8355": 13218789376.0,
+ "8360": 13218789376.0,
+ "8365": 13218789376.0,
+ "8370": 13218789376.0,
+ "8375": 13218789376.0,
+ "8380": 13218789376.0,
+ "8385": 13218789376.0,
+ "8390": 13218789376.0,
+ "8395": 13218789376.0,
+ "8400": 13218789376.0,
+ "8405": 13218789376.0,
+ "8410": 13218789376.0,
+ "8415": 13218789376.0,
+ "8420": 13218789376.0,
+ "8425": 13218789376.0,
+ "8430": 13218789376.0,
+ "8435": 13218789376.0,
+ "8440": 13218789376.0,
+ "8445": 13218789376.0,
+ "8450": 13218789376.0,
+ "8455": 13218789376.0,
+ "8460": 13218789376.0,
+ "8465": 13218789376.0,
+ "8470": 13218789376.0,
+ "8475": 13218789376.0,
+ "8480": 13218789376.0,
+ "8485": 13218789376.0,
+ "8490": 13218789376.0,
+ "8495": 13218789376.0,
+ "8500": 13218789376.0,
+ "8505": 13218789376.0,
+ "8510": 13218789376.0,
+ "8515": 13218789376.0,
+ "8520": 13218789376.0,
+ "8525": 13218789376.0,
+ "8530": 13218789376.0,
+ "8535": 13218789376.0,
+ "8540": 13218789376.0,
+ "8545": 13218789376.0,
+ "8550": 13218789376.0,
+ "8555": 13218789376.0,
+ "8560": 13218789376.0,
+ "8565": 13218789376.0,
+ "8570": 13218789376.0,
+ "8575": 13218789376.0,
+ "8580": 13218789376.0,
+ "8585": 13218789376.0,
+ "8590": 13218789376.0,
+ "8595": 13218789376.0,
+ "8600": 13218789376.0,
+ "8605": 13218789376.0,
+ "8610": 13218789376.0,
+ "8615": 13218789376.0,
+ "8620": 13218789376.0,
+ "8625": 13218789376.0,
+ "8630": 13218789376.0,
+ "8635": 13218789376.0,
+ "8640": 13218789376.0,
+ "8645": 13218789376.0,
+ "8650": 13218789376.0,
+ "8655": 13218789376.0,
+ "8660": 13218789376.0,
+ "8665": 13218789376.0,
+ "8670": 13218789376.0,
+ "8675": 13218789376.0,
+ "8680": 13218789376.0,
+ "8685": 13218789376.0,
+ "8690": 13218789376.0,
+ "8695": 13218789376.0,
+ "8700": 13218789376.0,
+ "8705": 13218789376.0,
+ "8710": 13218789376.0,
+ "8715": 13218789376.0,
+ "8720": 13218789376.0,
+ "8725": 13218789376.0,
+ "8730": 13218789376.0,
+ "8735": 13218789376.0,
+ "8740": 13218789376.0,
+ "8745": 13218789376.0,
+ "8750": 13218789376.0,
+ "8755": 13218789376.0,
+ "8760": 13218789376.0,
+ "8765": 13218789376.0,
+ "8770": 13218789376.0,
+ "8775": 13218789376.0,
+ "8780": 13218789376.0,
+ "8785": 13218789376.0,
+ "8790": 13218789376.0,
+ "8795": 13218789376.0,
+ "8800": 13218789376.0,
+ "8805": 13218789376.0,
+ "8810": 13218789376.0,
+ "8815": 13218789376.0,
+ "8820": 13218789376.0,
+ "8825": 13218789376.0,
+ "8830": 13218789376.0,
+ "8835": 13218789376.0,
+ "8840": 13218789376.0,
+ "8845": 13218789376.0,
+ "8850": 13218789376.0,
+ "8855": 13218789376.0,
+ "8860": 13218789376.0,
+ "8865": 13218789376.0,
+ "8870": 13218789376.0,
+ "8875": 13218789376.0,
+ "8880": 13218789376.0,
+ "8885": 13218789376.0,
+ "8890": 13218789376.0,
+ "8895": 13218789376.0,
+ "8900": 13218789376.0,
+ "8905": 13218789376.0,
+ "8910": 13218789376.0,
+ "8915": 13218789376.0,
+ "8920": 13218789376.0,
+ "8925": 13218789376.0,
+ "8930": 13218789376.0,
+ "8935": 13218789376.0,
+ "8940": 13218789376.0,
+ "8945": 13218789376.0,
+ "8950": 13218789376.0,
+ "8955": 13218789376.0,
+ "8960": 13218789376.0,
+ "8965": 13218789376.0,
+ "8970": 13218789376.0,
+ "8975": 13218789376.0,
+ "8980": 13218789376.0,
+ "8985": 13218789376.0,
+ "8990": 13218789376.0,
+ "8995": 13218789376.0,
+ "9000": 13218789376.0,
+ "9005": 13218789376.0,
+ "9010": 13218789376.0,
+ "9015": 13218789376.0,
+ "9020": 13218789376.0,
+ "9025": 13218789376.0,
+ "9030": 13218789376.0,
+ "9035": 13218789376.0,
+ "9040": 13218789376.0,
+ "9045": 13218789376.0,
+ "9050": 13218789376.0,
+ "9055": 13218789376.0,
+ "9060": 13218789376.0,
+ "9065": 13218789376.0,
+ "9070": 13218789376.0,
+ "9075": 13218789376.0,
+ "9080": 13218789376.0,
+ "9085": 13218789376.0,
+ "9090": 13218789376.0,
+ "9095": 13218789376.0,
+ "9100": 13218789376.0,
+ "9105": 13218789376.0,
+ "9110": 13218789376.0,
+ "9115": 13218789376.0,
+ "9120": 13218789376.0,
+ "9125": 13218789376.0,
+ "9130": 13218789376.0,
+ "9135": 13218789376.0,
+ "9140": 13218789376.0,
+ "9145": 13218789376.0,
+ "9150": 13218789376.0,
+ "9155": 13218789376.0,
+ "9160": 13218789376.0,
+ "9165": 13218789376.0,
+ "9170": 13218789376.0,
+ "9175": 13218789376.0,
+ "9180": 13218789376.0,
+ "9185": 13218789376.0,
+ "9190": 13218789376.0,
+ "9195": 13218789376.0,
+ "9200": 13218789376.0,
+ "9205": 13218789376.0,
+ "9210": 13218789376.0,
+ "9215": 13218789376.0,
+ "9220": 13218789376.0,
+ "9225": 13218789376.0,
+ "9230": 13218789376.0,
+ "9235": 13218789376.0,
+ "9240": 13218789376.0,
+ "9245": 13218789376.0,
+ "9250": 13218789376.0,
+ "9255": 13218789376.0,
+ "9260": 13218789376.0,
+ "9265": 13218789376.0,
+ "9270": 13218789376.0,
+ "9275": 13218789376.0,
+ "9280": 13218789376.0,
+ "9285": 13218789376.0,
+ "9290": 13218789376.0,
+ "9295": 13218789376.0,
+ "9300": 13218789376.0,
+ "9305": 13218789376.0,
+ "9310": 13218789376.0,
+ "9315": 13218789376.0,
+ "9320": 13218789376.0,
+ "9325": 13218789376.0,
+ "9330": 13218789376.0,
+ "9335": 13218789376.0,
+ "9340": 13218789376.0,
+ "9345": 13218789376.0,
+ "9350": 13218789376.0,
+ "9355": 13218789376.0,
+ "9360": 13218789376.0,
+ "9365": 13218789376.0,
+ "9370": 13218789376.0,
+ "9375": 13218789376.0,
+ "9380": 13218789376.0,
+ "9385": 13218789376.0,
+ "9390": 13218789376.0,
+ "9395": 13218789376.0,
+ "9400": 13218789376.0,
+ "9405": 13218789376.0,
+ "9410": 13218789376.0,
+ "9415": 13218789376.0,
+ "9420": 13218789376.0,
+ "9425": 13218789376.0,
+ "9430": 13218789376.0,
+ "9435": 13218789376.0,
+ "9440": 13218789376.0,
+ "9445": 13218789376.0,
+ "9450": 13218789376.0,
+ "9455": 13218789376.0,
+ "9460": 13218789376.0,
+ "9465": 13218789376.0,
+ "9470": 13218789376.0,
+ "9475": 13218789376.0,
+ "9480": 13218789376.0,
+ "9485": 13218789376.0,
+ "9490": 13218789376.0,
+ "9495": 13218789376.0,
+ "9500": 13218789376.0,
+ "9505": 13218789376.0,
+ "9510": 13218789376.0,
+ "9515": 13218789376.0,
+ "9520": 13218789376.0,
+ "9525": 13218789376.0,
+ "9530": 13218789376.0,
+ "9535": 13218789376.0,
+ "9540": 13218789376.0,
+ "9545": 13218789376.0,
+ "9550": 13218789376.0,
+ "9555": 13218789376.0,
+ "9560": 13218789376.0,
+ "9565": 13218789376.0,
+ "9570": 13218789376.0,
+ "9575": 13218789376.0,
+ "9580": 13218789376.0,
+ "9585": 13218789376.0,
+ "9590": 13218789376.0,
+ "9595": 13218789376.0,
+ "9600": 13218789376.0,
+ "9605": 13218789376.0,
+ "9610": 13218789376.0,
+ "9615": 13218789376.0,
+ "9620": 13218789376.0,
+ "9625": 13218789376.0,
+ "9630": 13218789376.0,
+ "9635": 13218789376.0,
+ "9640": 13218789376.0,
+ "9645": 13218789376.0,
+ "9650": 13218789376.0,
+ "9655": 13218789376.0,
+ "9660": 13218789376.0,
+ "9665": 13218789376.0,
+ "9670": 13218789376.0,
+ "9675": 13218789376.0,
+ "9680": 13218789376.0,
+ "9685": 13218789376.0,
+ "9690": 13218789376.0,
+ "9695": 13218789376.0,
+ "9700": 13218789376.0,
+ "9705": 13218789376.0,
+ "9710": 13218789376.0,
+ "9715": 13218789376.0,
+ "9720": 13218789376.0,
+ "9725": 13218789376.0,
+ "9730": 13218789376.0,
+ "9735": 13218789376.0,
+ "9740": 13218789376.0,
+ "9745": 13218789376.0,
+ "9750": 13218789376.0,
+ "9755": 13218789376.0,
+ "9760": 13218789376.0,
+ "9765": 13218789376.0,
+ "9770": 13218789376.0,
+ "9775": 13218789376.0,
+ "9780": 13218789376.0,
+ "9785": 13218789376.0,
+ "9790": 13218789376.0,
+ "9795": 13218789376.0,
+ "9800": 13218789376.0,
+ "9805": 13218789376.0,
+ "9810": 13218789376.0,
+ "9815": 13218789376.0,
+ "9820": 13218789376.0,
+ "9825": 13218789376.0,
+ "9830": 13218789376.0,
+ "9835": 13218789376.0,
+ "9840": 13218789376.0,
+ "9845": 13218789376.0,
+ "9850": 13218789376.0,
+ "9855": 13218789376.0,
+ "9860": 13218789376.0,
+ "9865": 13218789376.0,
+ "9870": 13218789376.0,
+ "9875": 13218789376.0,
+ "9880": 13218789376.0,
+ "9885": 13218789376.0,
+ "9890": 13218789376.0,
+ "9895": 13218789376.0,
+ "9900": 13218789376.0,
+ "9905": 13218789376.0,
+ "9910": 13218789376.0,
+ "9915": 13218789376.0,
+ "9920": 13218789376.0,
+ "9925": 13218789376.0,
+ "9930": 13218789376.0,
+ "9935": 13218789376.0,
+ "9940": 13218789376.0,
+ "9945": 13218789376.0,
+ "9950": 13218789376.0,
+ "9955": 13218789376.0,
+ "9960": 13218789376.0,
+ "9965": 13218789376.0,
+ "9970": 13218789376.0,
+ "9975": 13218789376.0,
+ "9980": 13218789376.0,
+ "9985": 13218789376.0,
+ "9990": 13218789376.0,
+ "9995": 13218789376.0,
+ "10000": 13218789376.0,
+ "10005": 13218789376.0,
+ "10010": 13218789376.0,
+ "10015": 13218789376.0,
+ "10020": 13218789376.0,
+ "10025": 13218789376.0,
+ "10030": 13218789376.0,
+ "10035": 13218789376.0,
+ "10040": 13218789376.0,
+ "10045": 13218789376.0,
+ "10050": 13218789376.0,
+ "10055": 13218789376.0,
+ "10060": 13218789376.0,
+ "10065": 13218789376.0,
+ "10070": 13218789376.0,
+ "10075": 13218789376.0,
+ "10080": 13218789376.0,
+ "10085": 13218789376.0,
+ "10090": 13218789376.0,
+ "10095": 13218789376.0,
+ "10100": 13218789376.0,
+ "10105": 13218789376.0,
+ "10110": 13218789376.0,
+ "10115": 13218789376.0,
+ "10120": 13218789376.0,
+ "10125": 13218789376.0,
+ "10130": 13218789376.0,
+ "10135": 13218789376.0,
+ "10140": 13218789376.0,
+ "10145": 13218789376.0,
+ "10150": 13218789376.0,
+ "10155": 13218789376.0,
+ "10160": 13218789376.0,
+ "10165": 13218789376.0,
+ "10170": 13218789376.0,
+ "10175": 13218789376.0,
+ "10180": 13218789376.0,
+ "10185": 13218789376.0,
+ "10190": 13218789376.0,
+ "10195": 13218789376.0,
+ "10200": 13218789376.0,
+ "10205": 13218789376.0,
+ "10210": 13218789376.0,
+ "10215": 13218789376.0,
+ "10220": 13218789376.0,
+ "10225": 13218789376.0,
+ "10230": 13218789376.0,
+ "10235": 13218789376.0,
+ "10240": 13218789376.0,
+ "10245": 13218789376.0,
+ "10250": 13218789376.0,
+ "10255": 13218789376.0,
+ "10260": 13218789376.0,
+ "10265": 13218789376.0,
+ "10270": 13218789376.0,
+ "10275": 13218789376.0,
+ "10280": 13218789376.0,
+ "10285": 13218789376.0,
+ "10290": 13218789376.0,
+ "10295": 13218789376.0,
+ "10300": 13218789376.0,
+ "10305": 13218789376.0,
+ "10310": 13218789376.0,
+ "10315": 13218789376.0,
+ "10320": 13218789376.0,
+ "10325": 13218789376.0,
+ "10330": 13218789376.0,
+ "10335": 13218789376.0,
+ "10340": 13218789376.0,
+ "10345": 13218789376.0,
+ "10350": 13218789376.0,
+ "10355": 13218789376.0,
+ "10360": 13218789376.0,
+ "10365": 13218789376.0,
+ "10370": 13218789376.0,
+ "10375": 13218789376.0,
+ "10380": 13218789376.0,
+ "10385": 13218789376.0,
+ "10390": 13218789376.0,
+ "10395": 13218789376.0,
+ "10400": 13218789376.0,
+ "10405": 13218789376.0,
+ "10410": 13218789376.0,
+ "10415": 13218789376.0,
+ "10420": 13218789376.0,
+ "10425": 13218789376.0,
+ "10430": 13218789376.0,
+ "10435": 13218789376.0,
+ "10440": 13218789376.0,
+ "10445": 13218789376.0,
+ "10450": 13218789376.0,
+ "10455": 13218789376.0,
+ "10460": 13218789376.0,
+ "10465": 13218789376.0,
+ "10470": 13218789376.0,
+ "10475": 13218789376.0,
+ "10480": 13218789376.0,
+ "10485": 13218789376.0,
+ "10490": 13218789376.0,
+ "10495": 13218789376.0,
+ "10500": 13218789376.0,
+ "10505": 13218789376.0,
+ "10510": 13218789376.0,
+ "10515": 13218789376.0,
+ "10520": 13218789376.0,
+ "10525": 13218789376.0,
+ "10530": 13218789376.0,
+ "10535": 13218789376.0,
+ "10540": 13218789376.0,
+ "10545": 13218789376.0,
+ "10550": 13218789376.0,
+ "10555": 13218789376.0,
+ "10560": 13218789376.0,
+ "10565": 13218789376.0,
+ "10570": 13218789376.0,
+ "10575": 13218789376.0,
+ "10580": 13218789376.0,
+ "10585": 13218789376.0,
+ "10590": 13218789376.0,
+ "10595": 13218789376.0,
+ "10600": 13218789376.0,
+ "10605": 13218789376.0,
+ "10610": 13218789376.0,
+ "10615": 13218789376.0,
+ "10620": 13218789376.0,
+ "10625": 13218789376.0,
+ "10630": 13218789376.0,
+ "10635": 13218789376.0,
+ "10640": 13218789376.0,
+ "10645": 13218789376.0,
+ "10650": 13218789376.0,
+ "10655": 13218789376.0,
+ "10660": 13218789376.0,
+ "10665": 13218789376.0,
+ "10670": 13218789376.0,
+ "10675": 13218789376.0,
+ "10680": 13218789376.0,
+ "10685": 13218789376.0,
+ "10690": 13218789376.0,
+ "10695": 13218789376.0,
+ "10700": 13218789376.0,
+ "10705": 13218789376.0,
+ "10710": 13218789376.0,
+ "10715": 13218789376.0,
+ "10720": 13218789376.0,
+ "10725": 13218789376.0,
+ "10730": 13218789376.0,
+ "10735": 13218789376.0,
+ "10740": 13218789376.0,
+ "10745": 13218789376.0,
+ "10750": 13218789376.0,
+ "10755": 13218789376.0,
+ "10760": 13218789376.0,
+ "10765": 13218789376.0,
+ "10770": 13218789376.0,
+ "10775": 13218789376.0,
+ "10780": 13218789376.0,
+ "10785": 13218789376.0,
+ "10790": 13218789376.0,
+ "10795": 13218789376.0,
+ "10800": 13218789376.0,
+ "10805": 13218789376.0,
+ "10810": 13218789376.0,
+ "10815": 13218789376.0,
+ "10820": 13218789376.0,
+ "10825": 13218789376.0,
+ "10830": 13218789376.0,
+ "10835": 13218789376.0,
+ "10840": 13218789376.0,
+ "10845": 13218789376.0,
+ "10850": 13218789376.0,
+ "10855": 13218789376.0,
+ "10860": 13218789376.0,
+ "10865": 13218789376.0,
+ "10870": 13218789376.0,
+ "10875": 13218789376.0,
+ "10880": 13218789376.0,
+ "10885": 13218789376.0,
+ "10890": 13218789376.0,
+ "10895": 13218789376.0,
+ "10900": 13218789376.0,
+ "10905": 13218789376.0,
+ "10910": 13218789376.0,
+ "10915": 13218789376.0,
+ "10920": 13218789376.0,
+ "10925": 13218789376.0,
+ "10930": 13218789376.0,
+ "10935": 13218789376.0,
+ "10940": 13218789376.0,
+ "10945": 13218789376.0,
+ "10950": 13218789376.0,
+ "10955": 13218789376.0,
+ "10960": 13218789376.0,
+ "10965": 13218789376.0,
+ "10970": 13218789376.0,
+ "10975": 13218789376.0,
+ "10980": 13218789376.0,
+ "10985": 13218789376.0,
+ "10990": 13218789376.0,
+ "10995": 13218789376.0,
+ "11000": 13218789376.0,
+ "11005": 13218789376.0,
+ "11010": 13218789376.0,
+ "11015": 13218789376.0,
+ "11020": 13218789376.0,
+ "11025": 13218789376.0,
+ "11030": 13218789376.0,
+ "11035": 13218789376.0,
+ "11040": 13218789376.0,
+ "11045": 13218789376.0,
+ "11050": 13218789376.0,
+ "11055": 13218789376.0,
+ "11060": 13218789376.0,
+ "11065": 13218789376.0,
+ "11070": 13218789376.0,
+ "11075": 13218789376.0,
+ "11080": 13218789376.0,
+ "11085": 13218789376.0,
+ "11090": 13218789376.0,
+ "11095": 13218789376.0,
+ "11100": 13218789376.0,
+ "11105": 13218789376.0,
+ "11110": 13218789376.0,
+ "11115": 13218789376.0,
+ "11120": 13218789376.0,
+ "11125": 13218789376.0,
+ "11130": 13218789376.0,
+ "11135": 13218789376.0,
+ "11140": 13218789376.0,
+ "11145": 13218789376.0,
+ "11150": 13218789376.0,
+ "11155": 13218789376.0,
+ "11160": 13218789376.0,
+ "11165": 13218789376.0,
+ "11170": 13218789376.0,
+ "11175": 13218789376.0,
+ "11180": 13218789376.0,
+ "11185": 13218789376.0,
+ "11190": 13218789376.0,
+ "11195": 13218789376.0,
+ "11200": 13218789376.0,
+ "11205": 13218789376.0,
+ "11210": 13218789376.0,
+ "11215": 13218789376.0,
+ "11220": 13218789376.0,
+ "11225": 13218789376.0,
+ "11230": 13218789376.0,
+ "11235": 13218789376.0,
+ "11240": 13218789376.0,
+ "11245": 13218789376.0,
+ "11250": 13218789376.0,
+ "11255": 13218789376.0,
+ "11260": 13218789376.0,
+ "11265": 13218789376.0,
+ "11270": 13218789376.0,
+ "11275": 13218789376.0,
+ "11280": 13218789376.0,
+ "11285": 13218789376.0,
+ "11290": 13218789376.0,
+ "11295": 13218789376.0,
+ "11300": 13218789376.0,
+ "11305": 13218789376.0,
+ "11310": 13218789376.0,
+ "11315": 13218789376.0,
+ "11320": 13218789376.0,
+ "11325": 13218789376.0,
+ "11330": 13218789376.0,
+ "11335": 13218789376.0,
+ "11340": 13218789376.0,
+ "11345": 13218789376.0,
+ "11350": 13218789376.0,
+ "11355": 13218789376.0,
+ "11360": 13218789376.0,
+ "11365": 13218789376.0,
+ "11370": 13218789376.0,
+ "11375": 13218789376.0,
+ "11380": 13218789376.0,
+ "11385": 13218789376.0,
+ "11390": 13218789376.0,
+ "11395": 13218789376.0,
+ "11400": 13218789376.0,
+ "11405": 13218789376.0,
+ "11410": 13218789376.0,
+ "11415": 13218789376.0,
+ "11420": 13218789376.0,
+ "11425": 13218789376.0,
+ "11430": 13218789376.0,
+ "11435": 13218789376.0,
+ "11440": 13218789376.0,
+ "11445": 13218789376.0,
+ "11450": 13218789376.0,
+ "11455": 13218789376.0,
+ "11460": 13218789376.0,
+ "11465": 13218789376.0,
+ "11470": 13218789376.0,
+ "11475": 13218789376.0,
+ "11480": 13218789376.0,
+ "11485": 13218789376.0,
+ "11490": 13218789376.0,
+ "11495": 13218789376.0,
+ "11500": 13218789376.0,
+ "11505": 13218789376.0,
+ "11510": 13218789376.0,
+ "11515": 13218789376.0,
+ "11520": 13218789376.0,
+ "11525": 13218789376.0,
+ "11530": 13218789376.0,
+ "11535": 13218789376.0,
+ "11540": 13218789376.0,
+ "11545": 13218789376.0,
+ "11550": 13218789376.0,
+ "11555": 13218789376.0,
+ "11560": 13218789376.0,
+ "11565": 13218789376.0,
+ "11570": 13218789376.0,
+ "11575": 13218789376.0,
+ "11580": 13218789376.0,
+ "11585": 13218789376.0,
+ "11590": 13218789376.0,
+ "11595": 13218789376.0,
+ "11600": 13218789376.0,
+ "11605": 13218789376.0,
+ "11610": 13218789376.0,
+ "11615": 13218789376.0,
+ "11620": 13218789376.0,
+ "11625": 13218789376.0,
+ "11630": 13218789376.0,
+ "11635": 13218789376.0,
+ "11640": 13218789376.0,
+ "11645": 13218789376.0,
+ "11650": 13218789376.0,
+ "11655": 13218789376.0,
+ "11660": 13218789376.0,
+ "11665": 13218789376.0,
+ "11670": 13218789376.0,
+ "11675": 13218789376.0,
+ "11680": 13218789376.0,
+ "11685": 13218789376.0,
+ "11690": 13218789376.0,
+ "11695": 13218789376.0,
+ "11700": 13218789376.0,
+ "11705": 13218789376.0,
+ "11710": 13218789376.0,
+ "11715": 13218789376.0,
+ "11720": 13218789376.0,
+ "11725": 13218789376.0,
+ "11730": 13218789376.0,
+ "11735": 13218789376.0,
+ "11740": 13218789376.0,
+ "11745": 13218789376.0,
+ "11750": 13218789376.0,
+ "11755": 13218789376.0,
+ "11760": 13218789376.0,
+ "11765": 13218789376.0,
+ "11770": 13218789376.0,
+ "11775": 13218789376.0,
+ "11780": 13218789376.0,
+ "11785": 13218789376.0,
+ "11790": 13218789376.0,
+ "11795": 13218789376.0,
+ "11800": 13218789376.0,
+ "11805": 13218789376.0,
+ "11810": 13218789376.0,
+ "11815": 13218789376.0,
+ "11820": 13218789376.0,
+ "11825": 13218789376.0,
+ "11830": 13218789376.0,
+ "11835": 13218789376.0,
+ "11840": 13218789376.0,
+ "11845": 13218789376.0,
+ "11850": 13218789376.0,
+ "11855": 13218789376.0,
+ "11860": 13218789376.0,
+ "11865": 13218789376.0,
+ "11870": 13218789376.0,
+ "11875": 13218789376.0,
+ "11880": 13218789376.0,
+ "11885": 13218789376.0,
+ "11890": 13218789376.0,
+ "11895": 13218789376.0,
+ "11900": 13218789376.0,
+ "11905": 13218789376.0,
+ "11910": 13218789376.0,
+ "11915": 13218789376.0,
+ "11920": 13218789376.0,
+ "11925": 13218789376.0,
+ "11930": 13218789376.0,
+ "11935": 13218789376.0,
+ "11940": 13218789376.0,
+ "11945": 13218789376.0,
+ "11950": 13218789376.0,
+ "11955": 13218789376.0,
+ "11960": 13218789376.0,
+ "11965": 13218789376.0,
+ "11970": 13218789376.0,
+ "11975": 13218789376.0,
+ "11980": 13218789376.0,
+ "11985": 13218789376.0,
+ "11990": 13218789376.0,
+ "11995": 13218789376.0,
+ "12000": 13218789376.0,
+ "12005": 13218789376.0,
+ "12010": 13218789376.0,
+ "12015": 13218789376.0,
+ "12020": 13218789376.0,
+ "12025": 13218789376.0,
+ "12030": 13218789376.0,
+ "12035": 13218789376.0,
+ "12040": 13218789376.0,
+ "12045": 13218789376.0,
+ "12050": 13218789376.0,
+ "12055": 13218789376.0,
+ "12060": 13218789376.0,
+ "12065": 13218789376.0,
+ "12070": 13218789376.0,
+ "12075": 13218789376.0,
+ "12080": 13218789376.0,
+ "12085": 13218789376.0,
+ "12090": 13218789376.0,
+ "12095": 13218789376.0,
+ "12100": 13218789376.0,
+ "12105": 13218789376.0,
+ "12110": 13218789376.0,
+ "12115": 13218789376.0,
+ "12120": 13218789376.0,
+ "12125": 13218789376.0,
+ "12130": 13218789376.0,
+ "12135": 13218789376.0,
+ "12140": 13218789376.0,
+ "12145": 13218789376.0,
+ "12150": 13218789376.0,
+ "12155": 13218789376.0,
+ "12160": 13218789376.0,
+ "12165": 13218789376.0,
+ "12170": 13218789376.0,
+ "12175": 13218789376.0,
+ "12180": 13218789376.0,
+ "12185": 13218789376.0,
+ "12190": 13218789376.0,
+ "12195": 13218789376.0,
+ "12200": 13218789376.0,
+ "12205": 13218789376.0,
+ "12210": 13218789376.0,
+ "12215": 13218789376.0,
+ "12220": 13218789376.0,
+ "12225": 13218789376.0,
+ "12230": 13218789376.0,
+ "12235": 13218789376.0,
+ "12240": 13218789376.0,
+ "12245": 13218789376.0,
+ "12250": 13218789376.0,
+ "12255": 13218789376.0,
+ "12260": 13218789376.0,
+ "12265": 13218789376.0,
+ "12270": 13218789376.0,
+ "12275": 13218789376.0,
+ "12280": 13218789376.0,
+ "12285": 13218789376.0,
+ "12290": 13218789376.0,
+ "12295": 13218789376.0,
+ "12300": 13218789376.0,
+ "12305": 13218789376.0,
+ "12310": 13218789376.0,
+ "12315": 13218789376.0,
+ "12320": 13218789376.0,
+ "12325": 13218789376.0,
+ "12330": 13218789376.0,
+ "12335": 13218789376.0,
+ "12340": 13218789376.0,
+ "12345": 13218789376.0,
+ "12350": 13218789376.0,
+ "12355": 13218789376.0,
+ "12360": 13218789376.0,
+ "12365": 13218789376.0,
+ "12370": 13218789376.0,
+ "12375": 13218789376.0,
+ "12380": 13218789376.0,
+ "12385": 13218789376.0,
+ "12390": 13218789376.0,
+ "12395": 13218789376.0,
+ "12400": 13218789376.0,
+ "12405": 13218789376.0,
+ "12410": 13218789376.0,
+ "12415": 13218789376.0,
+ "12420": 13218789376.0,
+ "12425": 13218789376.0,
+ "12430": 13218789376.0,
+ "12435": 13218789376.0,
+ "12440": 13218789376.0,
+ "12445": 13218789376.0,
+ "12450": 13218789376.0,
+ "12455": 13218789376.0,
+ "12460": 13218789376.0,
+ "12465": 13218789376.0,
+ "12470": 13218789376.0,
+ "12475": 13218789376.0,
+ "12480": 13218789376.0,
+ "12485": 13218789376.0,
+ "12490": 13218789376.0,
+ "12495": 13218789376.0,
+ "12500": 13218789376.0,
+ "12505": 13218789376.0,
+ "12510": 13218789376.0,
+ "12515": 13218789376.0,
+ "12520": 13218789376.0,
+ "12525": 13218789376.0,
+ "12530": 13218789376.0,
+ "12535": 13218789376.0,
+ "12540": 13218789376.0,
+ "12545": 13218789376.0,
+ "12550": 13218789376.0,
+ "12555": 13218789376.0,
+ "12560": 13218789376.0,
+ "12565": 13218789376.0,
+ "12570": 13218789376.0,
+ "12575": 13218789376.0,
+ "12580": 13218789376.0,
+ "12585": 13218789376.0,
+ "12590": 13218789376.0,
+ "12595": 13218789376.0,
+ "12600": 13218789376.0,
+ "12605": 13218789376.0,
+ "12610": 13218789376.0,
+ "12615": 13218789376.0,
+ "12620": 13218789376.0,
+ "12625": 13218789376.0,
+ "12630": 13218789376.0,
+ "12635": 13218789376.0,
+ "12640": 13218789376.0,
+ "12645": 13218789376.0,
+ "12650": 13218789376.0,
+ "12655": 13218789376.0,
+ "12660": 13218789376.0,
+ "12665": 13218789376.0,
+ "12670": 13218789376.0,
+ "12675": 13218789376.0,
+ "12680": 13218789376.0,
+ "12685": 13218789376.0,
+ "12690": 13218789376.0,
+ "12695": 13218789376.0,
+ "12700": 13218789376.0,
+ "12705": 13218789376.0,
+ "12710": 13218789376.0,
+ "12715": 13218789376.0,
+ "12720": "nan",
+ "12725": "nan",
+ "12730": "nan",
+ "12735": "nan",
+ "12740": "nan",
+ "12745": "nan",
+ "12750": "nan",
+ "12755": "nan",
+ "12760": "nan",
+ "12765": "nan",
+ "12770": "nan",
+ "12775": "nan",
+ "12780": "nan",
+ "12785": "nan",
+ "12790": "nan",
+ "12795": "nan",
+ "12800": "nan",
+ "12805": "nan",
+ "12810": "nan",
+ "12815": "nan",
+ "12820": "nan",
+ "12825": "nan",
+ "12830": "nan",
+ "12835": "nan",
+ "12840": "nan",
+ "12845": "nan",
+ "12850": "nan",
+ "12855": "nan",
+ "12860": "nan",
+ "12865": "nan",
+ "12870": "nan",
+ "12875": "nan",
+ "12880": "nan",
+ "12885": "nan",
+ "12890": "nan",
+ "12895": "nan",
+ "12900": "nan",
+ "12905": "nan",
+ "12910": "nan",
+ "12915": "nan",
+ "12920": "nan",
+ "12925": "nan",
+ "12930": "nan",
+ "12935": "nan",
+ "12940": "nan",
+ "12945": "nan",
+ "12950": "nan",
+ "12955": "nan",
+ "12960": "nan",
+ "12965": "nan",
+ "12970": "nan",
+ "12975": "nan",
+ "12980": "nan",
+ "12985": "nan",
+ "12990": "nan",
+ "12995": "nan",
+ "13000": "nan"
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": 27608694784.0,
+ "5": 28107184128.0,
+ "10": 28107184128.0,
+ "15": 28107184128.0,
+ "20": 28107184128.0,
+ "25": 28107184128.0,
+ "30": 28107184128.0,
+ "35": 28107184128.0,
+ "40": 28107184128.0,
+ "45": 28107184128.0,
+ "50": 28107184128.0,
+ "55": 28107184128.0,
+ "60": 28107184128.0,
+ "65": 28107184128.0,
+ "70": 28107184128.0,
+ "75": 28107184128.0,
+ "80": 28107184128.0,
+ "85": 28107184128.0,
+ "90": 28107184128.0,
+ "95": 28107184128.0,
+ "100": 28107184128.0,
+ "105": 28107184128.0,
+ "110": 28107184128.0,
+ "115": 28107184128.0,
+ "120": 28107184128.0,
+ "125": 28107184128.0,
+ "130": 28107184128.0,
+ "135": 28107184128.0,
+ "140": 28107184128.0,
+ "145": 28107184128.0,
+ "150": 28107184128.0,
+ "155": 28107184128.0,
+ "160": 28107184128.0,
+ "165": 28107184128.0,
+ "170": 28107184128.0,
+ "175": 28107184128.0,
+ "180": 28107184128.0,
+ "185": 28107184128.0,
+ "190": 28107184128.0,
+ "195": 28107184128.0,
+ "200": 28107184128.0,
+ "205": 28107184128.0,
+ "210": 28107184128.0,
+ "215": 28107184128.0,
+ "220": 28107184128.0,
+ "225": 28107184128.0,
+ "230": 28107184128.0,
+ "235": 28107184128.0,
+ "240": 28107184128.0,
+ "245": 28107184128.0,
+ "250": 28107184128.0,
+ "255": 28107184128.0,
+ "260": 28107184128.0,
+ "265": 28107184128.0,
+ "270": 28107184128.0,
+ "275": 28107184128.0,
+ "280": 28107184128.0,
+ "285": 28107184128.0,
+ "290": 28107184128.0,
+ "295": 28107184128.0,
+ "300": 28107184128.0,
+ "305": 28107184128.0,
+ "310": 28107184128.0,
+ "315": 28107184128.0,
+ "320": 28107184128.0,
+ "325": 28107184128.0,
+ "330": 28107184128.0,
+ "335": 28107184128.0,
+ "340": 28107184128.0,
+ "345": 28107184128.0,
+ "350": 28107184128.0,
+ "355": 28107184128.0,
+ "360": 28107184128.0,
+ "365": 28107184128.0,
+ "370": 28107184128.0,
+ "375": 28107184128.0,
+ "380": 28107184128.0,
+ "385": 28107184128.0,
+ "390": 28107184128.0,
+ "395": 28107184128.0,
+ "400": 28107184128.0,
+ "405": 28107184128.0,
+ "410": 28107184128.0,
+ "415": 28107184128.0,
+ "420": 28107184128.0,
+ "425": 28107184128.0,
+ "430": 28107184128.0,
+ "435": 28107184128.0,
+ "440": 28107184128.0,
+ "445": 28107184128.0,
+ "450": 28107184128.0,
+ "455": 28107184128.0,
+ "460": 28107184128.0,
+ "465": 28107184128.0,
+ "470": 28107184128.0,
+ "475": 28107184128.0,
+ "480": 28107184128.0,
+ "485": 28107184128.0,
+ "490": 28107184128.0,
+ "495": 28107184128.0,
+ "500": 28107184128.0,
+ "505": 28107184128.0,
+ "510": 28107184128.0,
+ "515": 28107184128.0,
+ "520": 28107184128.0,
+ "525": 28107184128.0,
+ "530": 28107184128.0,
+ "535": 28107184128.0,
+ "540": 28107184128.0,
+ "545": 28107184128.0,
+ "550": 28107184128.0,
+ "555": 28107184128.0,
+ "560": 28107184128.0,
+ "565": 28107184128.0,
+ "570": 28107184128.0,
+ "575": 28107184128.0,
+ "580": 28107184128.0,
+ "585": 28107184128.0,
+ "590": 28107184128.0,
+ "595": 28107184128.0,
+ "600": 28107184128.0,
+ "605": 28107184128.0,
+ "610": 28107184128.0,
+ "615": 28107184128.0,
+ "620": 28107184128.0,
+ "625": 28107184128.0,
+ "630": 28107184128.0,
+ "635": 28107184128.0,
+ "640": 28107184128.0,
+ "645": 28107184128.0,
+ "650": 28107184128.0,
+ "655": 28107184128.0,
+ "660": 28107184128.0,
+ "665": 28107184128.0,
+ "670": 28107184128.0,
+ "675": 28107184128.0,
+ "680": 28107184128.0,
+ "685": 28107184128.0,
+ "690": 28107184128.0,
+ "695": 28107184128.0,
+ "700": 28107184128.0,
+ "705": 28107184128.0,
+ "710": 28107184128.0,
+ "715": 28107184128.0,
+ "720": 28107184128.0,
+ "725": 28107184128.0,
+ "730": 28107184128.0,
+ "735": 28107184128.0,
+ "740": 28107184128.0,
+ "745": 28107184128.0,
+ "750": 28107184128.0,
+ "755": 28107184128.0,
+ "760": 28107184128.0,
+ "765": 28107184128.0,
+ "770": 28107184128.0,
+ "775": 28107184128.0,
+ "780": 28107184128.0,
+ "785": 28107184128.0,
+ "790": 28107184128.0,
+ "795": 28107184128.0,
+ "800": 28107184128.0,
+ "805": 28107184128.0,
+ "810": 28107184128.0,
+ "815": 28107184128.0,
+ "820": 28107184128.0,
+ "825": 28107184128.0,
+ "830": 28107184128.0,
+ "835": 28107184128.0,
+ "840": 28107184128.0,
+ "845": 28107184128.0,
+ "850": 28107184128.0,
+ "855": 28107184128.0,
+ "860": 28107184128.0,
+ "865": 28107184128.0,
+ "870": 28107184128.0,
+ "875": 28107184128.0,
+ "880": 28107184128.0,
+ "885": 28107184128.0,
+ "890": 28107184128.0,
+ "895": 28107184128.0,
+ "900": 28107184128.0,
+ "905": 28107184128.0,
+ "910": 28107184128.0,
+ "915": 28107184128.0,
+ "920": 28107184128.0,
+ "925": 28107184128.0,
+ "930": 28107184128.0,
+ "935": 28107184128.0,
+ "940": 28107184128.0,
+ "945": 28107184128.0,
+ "950": 28107184128.0,
+ "955": 28107184128.0,
+ "960": 28107184128.0,
+ "965": 28107184128.0,
+ "970": 28107184128.0,
+ "975": 28107184128.0,
+ "980": 28107184128.0,
+ "985": 28107184128.0,
+ "990": 28107184128.0,
+ "995": 28107184128.0,
+ "1000": 28107184128.0,
+ "1005": 28107184128.0,
+ "1010": 28107184128.0,
+ "1015": 28107184128.0,
+ "1020": 28107184128.0,
+ "1025": 28107184128.0,
+ "1030": 28107184128.0,
+ "1035": 28107184128.0,
+ "1040": 28107184128.0,
+ "1045": 28107184128.0,
+ "1050": 28107184128.0,
+ "1055": 28107184128.0,
+ "1060": 28107184128.0,
+ "1065": 28107184128.0,
+ "1070": 28107184128.0,
+ "1075": 28107184128.0,
+ "1080": 28107184128.0,
+ "1085": 28107184128.0,
+ "1090": 28107184128.0,
+ "1095": 28107184128.0,
+ "1100": 28107184128.0,
+ "1105": 28107184128.0,
+ "1110": 28107184128.0,
+ "1115": 28107184128.0,
+ "1120": 28107184128.0,
+ "1125": 28107184128.0,
+ "1130": 28107184128.0,
+ "1135": 28107184128.0,
+ "1140": 28107184128.0,
+ "1145": 28107184128.0,
+ "1150": 28107184128.0,
+ "1155": 28107184128.0,
+ "1160": 28107184128.0,
+ "1165": 28107184128.0,
+ "1170": 28107184128.0,
+ "1175": 28107184128.0,
+ "1180": 28107184128.0,
+ "1185": 28107184128.0,
+ "1190": 28107184128.0,
+ "1195": 28107184128.0,
+ "1200": 28107184128.0,
+ "1205": 28107184128.0,
+ "1210": 28107184128.0,
+ "1215": 28107184128.0,
+ "1220": 28107184128.0,
+ "1225": 28107184128.0,
+ "1230": 28107184128.0,
+ "1235": 28107184128.0,
+ "1240": 28107184128.0,
+ "1245": 28107184128.0,
+ "1250": 28107184128.0,
+ "1255": 28107184128.0,
+ "1260": 28107184128.0,
+ "1265": 28107184128.0,
+ "1270": 28107184128.0,
+ "1275": 28107184128.0,
+ "1280": 28107184128.0,
+ "1285": 28107184128.0,
+ "1290": 28107184128.0,
+ "1295": 28107184128.0,
+ "1300": 28107184128.0,
+ "1305": 28107184128.0,
+ "1310": 28107184128.0,
+ "1315": 28107184128.0,
+ "1320": 28107184128.0,
+ "1325": 28107184128.0,
+ "1330": 28107184128.0,
+ "1335": 28107184128.0,
+ "1340": 28107184128.0,
+ "1345": 28107184128.0,
+ "1350": 28107184128.0,
+ "1355": 28107184128.0,
+ "1360": 28107184128.0,
+ "1365": 28107184128.0,
+ "1370": 28107184128.0,
+ "1375": 28107184128.0,
+ "1380": 28107184128.0,
+ "1385": 28107184128.0,
+ "1390": 28107184128.0,
+ "1395": 28107184128.0,
+ "1400": 28107184128.0,
+ "1405": 28107184128.0,
+ "1410": 28107184128.0,
+ "1415": 28107184128.0,
+ "1420": 28107184128.0,
+ "1425": 28107184128.0,
+ "1430": 28107184128.0,
+ "1435": 28107184128.0,
+ "1440": 28107184128.0,
+ "1445": 28107184128.0,
+ "1450": 28107184128.0,
+ "1455": 28107184128.0,
+ "1460": 28107184128.0,
+ "1465": 28107184128.0,
+ "1470": 28107184128.0,
+ "1475": 28107184128.0,
+ "1480": 28107184128.0,
+ "1485": 28107184128.0,
+ "1490": 28107184128.0,
+ "1495": 28107184128.0,
+ "1500": 28107184128.0,
+ "1505": 28107184128.0,
+ "1510": 28107184128.0,
+ "1515": 28107184128.0,
+ "1520": 28107184128.0,
+ "1525": 28107184128.0,
+ "1530": 28107184128.0,
+ "1535": 28107184128.0,
+ "1540": 28107184128.0,
+ "1545": 28107184128.0,
+ "1550": 28107184128.0,
+ "1555": 28107184128.0,
+ "1560": 28107184128.0,
+ "1565": 28107184128.0,
+ "1570": 28107184128.0,
+ "1575": 28107184128.0,
+ "1580": 28107184128.0,
+ "1585": 28107184128.0,
+ "1590": 28107184128.0,
+ "1595": 28107184128.0,
+ "1600": 28107184128.0,
+ "1605": 28107184128.0,
+ "1610": 28107184128.0,
+ "1615": 28107184128.0,
+ "1620": 28107184128.0,
+ "1625": 28107184128.0,
+ "1630": 28107184128.0,
+ "1635": 28107184128.0,
+ "1640": 28107184128.0,
+ "1645": 28107184128.0,
+ "1650": 28107184128.0,
+ "1655": 28107184128.0,
+ "1660": 28107184128.0,
+ "1665": 28107184128.0,
+ "1670": 28107184128.0,
+ "1675": 28107184128.0,
+ "1680": 28107184128.0,
+ "1685": 28107184128.0,
+ "1690": 28107184128.0,
+ "1695": 28107184128.0,
+ "1700": 28107184128.0,
+ "1705": 28107184128.0,
+ "1710": 28107184128.0,
+ "1715": 28107184128.0,
+ "1720": 28107184128.0,
+ "1725": 28107184128.0,
+ "1730": 28107184128.0,
+ "1735": 28107184128.0,
+ "1740": 28107184128.0,
+ "1745": 28107184128.0,
+ "1750": 28107184128.0,
+ "1755": 28107184128.0,
+ "1760": 28107184128.0,
+ "1765": 28107184128.0,
+ "1770": 28107184128.0,
+ "1775": 28107184128.0,
+ "1780": 28107184128.0,
+ "1785": 28107184128.0,
+ "1790": 28107184128.0,
+ "1795": 28107184128.0,
+ "1800": 28107184128.0,
+ "1805": 28107184128.0,
+ "1810": 28107184128.0,
+ "1815": 28107184128.0,
+ "1820": 28107184128.0,
+ "1825": 28107184128.0,
+ "1830": 28107184128.0,
+ "1835": 28107184128.0,
+ "1840": 28107184128.0,
+ "1845": 28107184128.0,
+ "1850": 28107184128.0,
+ "1855": 28107184128.0,
+ "1860": 28107184128.0,
+ "1865": 28107184128.0,
+ "1870": 28107184128.0,
+ "1875": 28107184128.0,
+ "1880": 28107184128.0,
+ "1885": 28107184128.0,
+ "1890": 28107184128.0,
+ "1895": 28107184128.0,
+ "1900": 28107184128.0,
+ "1905": 28107184128.0,
+ "1910": 28107184128.0,
+ "1915": 28107184128.0,
+ "1920": 28107184128.0,
+ "1925": 28107184128.0,
+ "1930": 28107184128.0,
+ "1935": 28107184128.0,
+ "1940": 28107184128.0,
+ "1945": 28107184128.0,
+ "1950": 28107184128.0,
+ "1955": 28107184128.0,
+ "1960": 28107184128.0,
+ "1965": 28107184128.0,
+ "1970": 28107184128.0,
+ "1975": 28107184128.0,
+ "1980": 28107184128.0,
+ "1985": 28107184128.0,
+ "1990": 28107184128.0,
+ "1995": 28107184128.0,
+ "2000": 28107184128.0,
+ "2005": 28107184128.0,
+ "2010": 28107184128.0,
+ "2015": 28107184128.0,
+ "2020": 28107184128.0,
+ "2025": 28107184128.0,
+ "2030": 28107184128.0,
+ "2035": 28107184128.0,
+ "2040": 28107184128.0,
+ "2045": 28107184128.0,
+ "2050": 28107184128.0,
+ "2055": 28107184128.0,
+ "2060": 28107184128.0,
+ "2065": 28107184128.0,
+ "2070": 28107184128.0,
+ "2075": 28107184128.0,
+ "2080": 28107184128.0,
+ "2085": 28107184128.0,
+ "2090": 28107184128.0,
+ "2095": 28107184128.0,
+ "2100": 28107184128.0,
+ "2105": 28107184128.0,
+ "2110": 28107184128.0,
+ "2115": 28107184128.0,
+ "2120": 28107184128.0,
+ "2125": 28107184128.0,
+ "2130": 28107184128.0,
+ "2135": 28107184128.0,
+ "2140": 28107184128.0,
+ "2145": 28107184128.0,
+ "2150": 28107184128.0,
+ "2155": 28107184128.0,
+ "2160": 28107184128.0,
+ "2165": 28107184128.0,
+ "2170": 28107184128.0,
+ "2175": 28107184128.0,
+ "2180": 28107184128.0,
+ "2185": 28107184128.0,
+ "2190": 28107184128.0,
+ "2195": 28107184128.0,
+ "2200": 28107184128.0,
+ "2205": 28107184128.0,
+ "2210": 28107184128.0,
+ "2215": 28107184128.0,
+ "2220": 28107184128.0,
+ "2225": 28107184128.0,
+ "2230": 28107184128.0,
+ "2235": 28107184128.0,
+ "2240": 28107184128.0,
+ "2245": 28107184128.0,
+ "2250": 28107184128.0,
+ "2255": 28107184128.0,
+ "2260": 28107184128.0,
+ "2265": 28107184128.0,
+ "2270": 28107184128.0,
+ "2275": 28107184128.0,
+ "2280": 28107184128.0,
+ "2285": 28107184128.0,
+ "2290": 28107184128.0,
+ "2295": 28107184128.0,
+ "2300": 28107184128.0,
+ "2305": 28107184128.0,
+ "2310": 28107184128.0,
+ "2315": 28107184128.0,
+ "2320": 28107184128.0,
+ "2325": 28107184128.0,
+ "2330": 28107184128.0,
+ "2335": 28107184128.0,
+ "2340": 28107184128.0,
+ "2345": 28107184128.0,
+ "2350": 28107184128.0,
+ "2355": 28107184128.0,
+ "2360": 28107184128.0,
+ "2365": 28107184128.0,
+ "2370": 28107184128.0,
+ "2375": 28107184128.0,
+ "2380": 28107184128.0,
+ "2385": 28107184128.0,
+ "2390": 28107184128.0,
+ "2395": 28107184128.0,
+ "2400": 28107184128.0,
+ "2405": 28107184128.0,
+ "2410": 28107184128.0,
+ "2415": 28107184128.0,
+ "2420": 28107184128.0,
+ "2425": 28107184128.0,
+ "2430": 28107184128.0,
+ "2435": 28107184128.0,
+ "2440": 28107184128.0,
+ "2445": 28107184128.0,
+ "2450": 28107184128.0,
+ "2455": 28107184128.0,
+ "2460": 28107184128.0,
+ "2465": 28107184128.0,
+ "2470": 28107184128.0,
+ "2475": 28107184128.0,
+ "2480": 28107184128.0,
+ "2485": 28107184128.0,
+ "2490": 28107184128.0,
+ "2495": 28107184128.0,
+ "2500": 28107184128.0,
+ "2505": 28107184128.0,
+ "2510": 28107184128.0,
+ "2515": 28107184128.0,
+ "2520": 28107184128.0,
+ "2525": 28107184128.0,
+ "2530": 28107184128.0,
+ "2535": 28107184128.0,
+ "2540": 28107184128.0,
+ "2545": 28107184128.0,
+ "2550": 28107184128.0,
+ "2555": 28107184128.0,
+ "2560": 28107184128.0,
+ "2565": 28107184128.0,
+ "2570": 28107184128.0,
+ "2575": 28107184128.0,
+ "2580": 28107184128.0,
+ "2585": 28107184128.0,
+ "2590": 28107184128.0,
+ "2595": 28107184128.0,
+ "2600": 28107184128.0,
+ "2605": 28107184128.0,
+ "2610": 28107184128.0,
+ "2615": 28107184128.0,
+ "2620": 28107184128.0,
+ "2625": 28107184128.0,
+ "2630": 28107184128.0,
+ "2635": 28107184128.0,
+ "2640": 28107184128.0,
+ "2645": 28107184128.0,
+ "2650": 28107184128.0,
+ "2655": 28107184128.0,
+ "2660": 28107184128.0,
+ "2665": 28107184128.0,
+ "2670": 28107184128.0,
+ "2675": 28107184128.0,
+ "2680": 28107184128.0,
+ "2685": 28107184128.0,
+ "2690": 28107184128.0,
+ "2695": 28107184128.0,
+ "2700": 28107184128.0,
+ "2705": 28107184128.0,
+ "2710": 28107184128.0,
+ "2715": 28107184128.0,
+ "2720": 28107184128.0,
+ "2725": 28107184128.0,
+ "2730": 28107184128.0,
+ "2735": 28107184128.0,
+ "2740": 28107184128.0,
+ "2745": 28107184128.0,
+ "2750": 28107184128.0,
+ "2755": 28107184128.0,
+ "2760": 28107184128.0,
+ "2765": 28107184128.0,
+ "2770": 28107184128.0,
+ "2775": 28107184128.0,
+ "2780": 28107184128.0,
+ "2785": 28107184128.0,
+ "2790": 28107184128.0,
+ "2795": 28107184128.0,
+ "2800": 28107184128.0,
+ "2805": 28107184128.0,
+ "2810": 28107184128.0,
+ "2815": 28107184128.0,
+ "2820": 28107184128.0,
+ "2825": 28107184128.0,
+ "2830": 28107184128.0,
+ "2835": 28107184128.0,
+ "2840": 28107184128.0,
+ "2845": 28107184128.0,
+ "2850": 28107184128.0,
+ "2855": 28107184128.0,
+ "2860": 28107184128.0,
+ "2865": 28107184128.0,
+ "2870": 28107184128.0,
+ "2875": 28107184128.0,
+ "2880": 28107184128.0,
+ "2885": 28107184128.0,
+ "2890": 28107184128.0,
+ "2895": 28107184128.0,
+ "2900": 28107184128.0,
+ "2905": 28107184128.0,
+ "2910": 28107184128.0,
+ "2915": 28107184128.0,
+ "2920": 28107184128.0,
+ "2925": 28107184128.0,
+ "2930": 28107184128.0,
+ "2935": 28107184128.0,
+ "2940": 28107184128.0,
+ "2945": 28107184128.0,
+ "2950": 28107184128.0,
+ "2955": 28107184128.0,
+ "2960": 28107184128.0,
+ "2965": 28107184128.0,
+ "2970": 28107184128.0,
+ "2975": 28107184128.0,
+ "2980": 28107184128.0,
+ "2985": 28107184128.0,
+ "2990": 28107184128.0,
+ "2995": 28107184128.0,
+ "3000": 28107184128.0,
+ "3005": 28107184128.0,
+ "3010": 28107184128.0,
+ "3015": 28107184128.0,
+ "3020": 28107184128.0,
+ "3025": 28107184128.0,
+ "3030": 28107184128.0,
+ "3035": 28107184128.0,
+ "3040": 28107184128.0,
+ "3045": 28107184128.0,
+ "3050": 28107184128.0,
+ "3055": 28107184128.0,
+ "3060": 28107184128.0,
+ "3065": 28107184128.0,
+ "3070": 28107184128.0,
+ "3075": 28107184128.0,
+ "3080": 28107184128.0,
+ "3085": 28107184128.0,
+ "3090": 28107184128.0,
+ "3095": 28107184128.0,
+ "3100": 28107184128.0,
+ "3105": 28107184128.0,
+ "3110": 28107184128.0,
+ "3115": 28107184128.0,
+ "3120": 28107184128.0,
+ "3125": 28107184128.0,
+ "3130": 28107184128.0,
+ "3135": 28107184128.0,
+ "3140": 28107184128.0,
+ "3145": 28107184128.0,
+ "3150": 28107184128.0,
+ "3155": 28107184128.0,
+ "3160": 28107184128.0,
+ "3165": 28107184128.0,
+ "3170": 28107184128.0,
+ "3175": 28107184128.0,
+ "3180": 28107184128.0,
+ "3185": 28107184128.0,
+ "3190": 28107184128.0,
+ "3195": 28107184128.0,
+ "3200": 28107184128.0,
+ "3205": 28107184128.0,
+ "3210": 28107184128.0,
+ "3215": 28107184128.0,
+ "3220": 28107184128.0,
+ "3225": 28107184128.0,
+ "3230": 28107184128.0,
+ "3235": 28107184128.0,
+ "3240": 28107184128.0,
+ "3245": 28107184128.0,
+ "3250": 28107184128.0,
+ "3255": 28107184128.0,
+ "3260": 28107184128.0,
+ "3265": 28107184128.0,
+ "3270": 28107184128.0,
+ "3275": 28107184128.0,
+ "3280": 28107184128.0,
+ "3285": 28107184128.0,
+ "3290": 28107184128.0,
+ "3295": 28107184128.0,
+ "3300": 28107184128.0,
+ "3305": 28107184128.0,
+ "3310": 28107184128.0,
+ "3315": 28107184128.0,
+ "3320": 28107184128.0,
+ "3325": 28107184128.0,
+ "3330": 28107184128.0,
+ "3335": 28107184128.0,
+ "3340": 28107184128.0,
+ "3345": 28107184128.0,
+ "3350": 28107184128.0,
+ "3355": 28107184128.0,
+ "3360": 28107184128.0,
+ "3365": 28107184128.0,
+ "3370": 28107184128.0,
+ "3375": 28107184128.0,
+ "3380": 28107184128.0,
+ "3385": 28107184128.0,
+ "3390": 28107184128.0,
+ "3395": 28107184128.0,
+ "3400": 28107184128.0,
+ "3405": 28107184128.0,
+ "3410": 28107184128.0,
+ "3415": 28107184128.0,
+ "3420": 28107184128.0,
+ "3425": 28107184128.0,
+ "3430": 28107184128.0,
+ "3435": 28107184128.0,
+ "3440": 28107184128.0,
+ "3445": 28107184128.0,
+ "3450": 28107184128.0,
+ "3455": 28107184128.0,
+ "3460": 28107184128.0,
+ "3465": 28107184128.0,
+ "3470": 28107184128.0,
+ "3475": 28107184128.0,
+ "3480": 28107184128.0,
+ "3485": 28107184128.0,
+ "3490": 28107184128.0,
+ "3495": 28107184128.0,
+ "3500": 28107184128.0,
+ "3505": 28107184128.0,
+ "3510": 28107184128.0,
+ "3515": 28107184128.0,
+ "3520": 28107184128.0,
+ "3525": 28107184128.0,
+ "3530": 28107184128.0,
+ "3535": 28107184128.0,
+ "3540": 28107184128.0,
+ "3545": 28107184128.0,
+ "3550": 28107184128.0,
+ "3555": 28107184128.0,
+ "3560": 28107184128.0,
+ "3565": 28107184128.0,
+ "3570": 28107184128.0,
+ "3575": 28107184128.0,
+ "3580": 28107184128.0,
+ "3585": 28107184128.0,
+ "3590": 28107184128.0,
+ "3595": 28107184128.0,
+ "3600": 28107184128.0,
+ "3605": 28107184128.0,
+ "3610": 28107184128.0,
+ "3615": 28107184128.0,
+ "3620": 28107184128.0,
+ "3625": 28107184128.0,
+ "3630": 28107184128.0,
+ "3635": 28107184128.0,
+ "3640": 28107184128.0,
+ "3645": 28107184128.0,
+ "3650": 28107184128.0,
+ "3655": 28107184128.0,
+ "3660": 28107184128.0,
+ "3665": 28107184128.0,
+ "3670": 28107184128.0,
+ "3675": 28107184128.0,
+ "3680": 28107184128.0,
+ "3685": 28107184128.0,
+ "3690": 28107184128.0,
+ "3695": 28107184128.0,
+ "3700": 28107184128.0,
+ "3705": 28107184128.0,
+ "3710": 28107184128.0,
+ "3715": 28107184128.0,
+ "3720": 28107184128.0,
+ "3725": 28107184128.0,
+ "3730": 28107184128.0,
+ "3735": 28107184128.0,
+ "3740": 28107184128.0,
+ "3745": 28107184128.0,
+ "3750": 28107184128.0,
+ "3755": 28107184128.0,
+ "3760": 28107184128.0,
+ "3765": 28107184128.0,
+ "3770": 28107184128.0,
+ "3775": 28107184128.0,
+ "3780": 28107184128.0,
+ "3785": 28107184128.0,
+ "3790": 28107184128.0,
+ "3795": 28107184128.0,
+ "3800": 28107184128.0,
+ "3805": 28107184128.0,
+ "3810": 28107184128.0,
+ "3815": 28107184128.0,
+ "3820": 28107184128.0,
+ "3825": 28107184128.0,
+ "3830": 28107184128.0,
+ "3835": 28107184128.0,
+ "3840": 28107184128.0,
+ "3845": 28107184128.0,
+ "3850": 28107184128.0,
+ "3855": 28107184128.0,
+ "3860": 28107184128.0,
+ "3865": 28107184128.0,
+ "3870": 28107184128.0,
+ "3875": 28107184128.0,
+ "3880": 28107184128.0,
+ "3885": 28107184128.0,
+ "3890": 28107184128.0,
+ "3895": 28107184128.0,
+ "3900": 28107184128.0,
+ "3905": 28107184128.0,
+ "3910": 28107184128.0,
+ "3915": 28107184128.0,
+ "3920": 28107184128.0,
+ "3925": 28107184128.0,
+ "3930": 28107184128.0,
+ "3935": 28107184128.0,
+ "3940": 28107184128.0,
+ "3945": 28107184128.0,
+ "3950": 28107184128.0,
+ "3955": 28107184128.0,
+ "3960": 28107184128.0,
+ "3965": 28107184128.0,
+ "3970": 28107184128.0,
+ "3975": 28107184128.0,
+ "3980": 28107184128.0,
+ "3985": 28107184128.0,
+ "3990": 28107184128.0,
+ "3995": 28107184128.0,
+ "4000": 28107184128.0,
+ "4005": 28107184128.0,
+ "4010": 28107184128.0,
+ "4015": 28107184128.0,
+ "4020": 28107184128.0,
+ "4025": 28107184128.0,
+ "4030": 28107184128.0,
+ "4035": 28107184128.0,
+ "4040": 28107184128.0,
+ "4045": 28107184128.0,
+ "4050": 28107184128.0,
+ "4055": 28107184128.0,
+ "4060": 28107184128.0,
+ "4065": 28107184128.0,
+ "4070": 28107184128.0,
+ "4075": 28107184128.0,
+ "4080": 28107184128.0,
+ "4085": 28107184128.0,
+ "4090": 28107184128.0,
+ "4095": 28107184128.0,
+ "4100": 28107184128.0,
+ "4105": 28107184128.0,
+ "4110": 28107184128.0,
+ "4115": 28107184128.0,
+ "4120": 28107184128.0,
+ "4125": 28107184128.0,
+ "4130": 28107184128.0,
+ "4135": 28107184128.0,
+ "4140": 28107184128.0,
+ "4145": 28107184128.0,
+ "4150": 28107184128.0,
+ "4155": 28107184128.0,
+ "4160": 28107184128.0,
+ "4165": 28107184128.0,
+ "4170": 28107184128.0,
+ "4175": 28107184128.0,
+ "4180": 28107184128.0,
+ "4185": 28107184128.0,
+ "4190": 28107184128.0,
+ "4195": 28107184128.0,
+ "4200": 28107184128.0,
+ "4205": 28107184128.0,
+ "4210": 28107184128.0,
+ "4215": 28107184128.0,
+ "4220": 28107184128.0,
+ "4225": 28107184128.0,
+ "4230": 28107184128.0,
+ "4235": 28107184128.0,
+ "4240": 28107184128.0,
+ "4245": 28107184128.0,
+ "4250": 28107184128.0,
+ "4255": 28107184128.0,
+ "4260": 28107184128.0,
+ "4265": 28107184128.0,
+ "4270": 28107184128.0,
+ "4275": 28107184128.0,
+ "4280": 28107184128.0,
+ "4285": 28107184128.0,
+ "4290": 28107184128.0,
+ "4295": 28107184128.0,
+ "4300": 28107184128.0,
+ "4305": 28107184128.0,
+ "4310": 28107184128.0,
+ "4315": 28107184128.0,
+ "4320": 28107184128.0,
+ "4325": 28107184128.0,
+ "4330": 28107184128.0,
+ "4335": 28107184128.0,
+ "4340": 28107184128.0,
+ "4345": 28107184128.0,
+ "4350": 28107184128.0,
+ "4355": 28107184128.0,
+ "4360": 28107184128.0,
+ "4365": 28107184128.0,
+ "4370": 28107184128.0,
+ "4375": 28107184128.0,
+ "4380": 28107184128.0,
+ "4385": 28107184128.0,
+ "4390": 28107184128.0,
+ "4395": 28107184128.0,
+ "4400": 28107184128.0,
+ "4405": 28107184128.0,
+ "4410": 28107184128.0,
+ "4415": 28107184128.0,
+ "4420": 28107184128.0,
+ "4425": 28107184128.0,
+ "4430": 28107184128.0,
+ "4435": 28107184128.0,
+ "4440": 28107184128.0,
+ "4445": 28107184128.0,
+ "4450": 28107184128.0,
+ "4455": 28107184128.0,
+ "4460": 28107184128.0,
+ "4465": 28107184128.0,
+ "4470": 28107184128.0,
+ "4475": 28107184128.0,
+ "4480": 28107184128.0,
+ "4485": 28107184128.0,
+ "4490": 28107184128.0,
+ "4495": 28107184128.0,
+ "4500": 28107184128.0,
+ "4505": 28107184128.0,
+ "4510": 28107184128.0,
+ "4515": 28107184128.0,
+ "4520": 28107184128.0,
+ "4525": 28107184128.0,
+ "4530": 28107184128.0,
+ "4535": 28107184128.0,
+ "4540": 28107184128.0,
+ "4545": 28107184128.0,
+ "4550": 28107184128.0,
+ "4555": 28107184128.0,
+ "4560": 28107184128.0,
+ "4565": 28107184128.0,
+ "4570": 28107184128.0,
+ "4575": 28107184128.0,
+ "4580": 28107184128.0,
+ "4585": 28107184128.0,
+ "4590": 28107184128.0,
+ "4595": 28107184128.0,
+ "4600": 28107184128.0,
+ "4605": 28107184128.0,
+ "4610": 28107184128.0,
+ "4615": 28107184128.0,
+ "4620": 28107184128.0,
+ "4625": 28107184128.0,
+ "4630": 28107184128.0,
+ "4635": 28107184128.0,
+ "4640": 28107184128.0,
+ "4645": 28107184128.0,
+ "4650": 28107184128.0,
+ "4655": 28107184128.0,
+ "4660": 28107184128.0,
+ "4665": 28107184128.0,
+ "4670": 28107184128.0,
+ "4675": 28107184128.0,
+ "4680": 28107184128.0,
+ "4685": 28107184128.0,
+ "4690": 28107184128.0,
+ "4695": 28107184128.0,
+ "4700": 28107184128.0,
+ "4705": 28107184128.0,
+ "4710": 28107184128.0,
+ "4715": 28107184128.0,
+ "4720": 28107184128.0,
+ "4725": 28107184128.0,
+ "4730": 28107184128.0,
+ "4735": 28107184128.0,
+ "4740": 28107184128.0,
+ "4745": 28107184128.0,
+ "4750": 28107184128.0,
+ "4755": 28107184128.0,
+ "4760": 28107184128.0,
+ "4765": 28107184128.0,
+ "4770": 28107184128.0,
+ "4775": 28107184128.0,
+ "4780": 28107184128.0,
+ "4785": 28107184128.0,
+ "4790": 28107184128.0,
+ "4795": 28107184128.0,
+ "4800": 28107184128.0,
+ "4805": 28107184128.0,
+ "4810": 28107184128.0,
+ "4815": 28107184128.0,
+ "4820": 28107184128.0,
+ "4825": 28107184128.0,
+ "4830": 28107184128.0,
+ "4835": 28107184128.0,
+ "4840": 28107184128.0,
+ "4845": 28107184128.0,
+ "4850": 28107184128.0,
+ "4855": 28107184128.0,
+ "4860": 28107184128.0,
+ "4865": 28107184128.0,
+ "4870": 28107184128.0,
+ "4875": 28107184128.0,
+ "4880": 28107184128.0,
+ "4885": 28107184128.0,
+ "4890": 28107184128.0,
+ "4895": 28107184128.0,
+ "4900": 28107184128.0,
+ "4905": 28107184128.0,
+ "4910": 28107184128.0,
+ "4915": 28107184128.0,
+ "4920": 28107184128.0,
+ "4925": 28107184128.0,
+ "4930": 28107184128.0,
+ "4935": 28107184128.0,
+ "4940": 28107184128.0,
+ "4945": 28107184128.0,
+ "4950": 28107184128.0,
+ "4955": 28107184128.0,
+ "4960": 28107184128.0,
+ "4965": 28107184128.0,
+ "4970": 28107184128.0,
+ "4975": 28107184128.0,
+ "4980": 28107184128.0,
+ "4985": 28107184128.0,
+ "4990": 28107184128.0,
+ "4995": 28107184128.0,
+ "5000": 28107184128.0,
+ "5005": 28107184128.0,
+ "5010": 28107184128.0,
+ "5015": 28107184128.0,
+ "5020": 28107184128.0,
+ "5025": 28107184128.0,
+ "5030": 28107184128.0,
+ "5035": 28107184128.0,
+ "5040": 28107184128.0,
+ "5045": 28107184128.0,
+ "5050": 28107184128.0,
+ "5055": 28107184128.0,
+ "5060": 28107184128.0,
+ "5065": 28107184128.0,
+ "5070": 28107184128.0,
+ "5075": 28107184128.0,
+ "5080": 28107184128.0,
+ "5085": 28107184128.0,
+ "5090": 28107184128.0,
+ "5095": 28107184128.0,
+ "5100": 28107184128.0,
+ "5105": 28107184128.0,
+ "5110": 28107184128.0,
+ "5115": 28107184128.0,
+ "5120": 28107184128.0,
+ "5125": 28107184128.0,
+ "5130": 28107184128.0,
+ "5135": 28107184128.0,
+ "5140": 28107184128.0,
+ "5145": 28107184128.0,
+ "5150": 28107184128.0,
+ "5155": 28107184128.0,
+ "5160": 28107184128.0,
+ "5165": 28107184128.0,
+ "5170": 28107184128.0,
+ "5175": 28107184128.0,
+ "5180": 28107184128.0,
+ "5185": 28107184128.0,
+ "5190": 28107184128.0,
+ "5195": 28107184128.0,
+ "5200": 28107184128.0,
+ "5205": 28107184128.0,
+ "5210": 28107184128.0,
+ "5215": 28107184128.0,
+ "5220": 28107184128.0,
+ "5225": 28107184128.0,
+ "5230": 28107184128.0,
+ "5235": 28107184128.0,
+ "5240": 28107184128.0,
+ "5245": 28107184128.0,
+ "5250": 28107184128.0,
+ "5255": 28107184128.0,
+ "5260": 28107184128.0,
+ "5265": 28107184128.0,
+ "5270": 28107184128.0,
+ "5275": 28107184128.0,
+ "5280": 28107184128.0,
+ "5285": 28107184128.0,
+ "5290": 28107184128.0,
+ "5295": 28107184128.0,
+ "5300": 28107184128.0,
+ "5305": 28107184128.0,
+ "5310": 28107184128.0,
+ "5315": 28107184128.0,
+ "5320": 28107184128.0,
+ "5325": 28107184128.0,
+ "5330": 28107184128.0,
+ "5335": 28107184128.0,
+ "5340": 28107184128.0,
+ "5345": 28107184128.0,
+ "5350": 28107184128.0,
+ "5355": 28107184128.0,
+ "5360": 28107184128.0,
+ "5365": 28107184128.0,
+ "5370": 28107184128.0,
+ "5375": 28107184128.0,
+ "5380": 28107184128.0,
+ "5385": 28107184128.0,
+ "5390": 28107184128.0,
+ "5395": 28107184128.0,
+ "5400": 28107184128.0,
+ "5405": 28107184128.0,
+ "5410": 28107184128.0,
+ "5415": 28107184128.0,
+ "5420": 28107184128.0,
+ "5425": 28107184128.0,
+ "5430": 28107184128.0,
+ "5435": 28107184128.0,
+ "5440": 28107184128.0,
+ "5445": 28107184128.0,
+ "5450": 28107184128.0,
+ "5455": 28107184128.0,
+ "5460": 28107184128.0,
+ "5465": 28107184128.0,
+ "5470": 28107184128.0,
+ "5475": 28107184128.0,
+ "5480": 28107184128.0,
+ "5485": 28107184128.0,
+ "5490": 28107184128.0,
+ "5495": 28107184128.0,
+ "5500": 28107184128.0,
+ "5505": 28107184128.0,
+ "5510": 28107184128.0,
+ "5515": 28107184128.0,
+ "5520": 28107184128.0,
+ "5525": 28107184128.0,
+ "5530": 28107184128.0,
+ "5535": 28107184128.0,
+ "5540": 28107184128.0,
+ "5545": 28107184128.0,
+ "5550": 28107184128.0,
+ "5555": 28107184128.0,
+ "5560": 28107184128.0,
+ "5565": 28107184128.0,
+ "5570": 28107184128.0,
+ "5575": 28107184128.0,
+ "5580": 28107184128.0,
+ "5585": 28107184128.0,
+ "5590": 28107184128.0,
+ "5595": 28107184128.0,
+ "5600": 28107184128.0,
+ "5605": 28107184128.0,
+ "5610": 28107184128.0,
+ "5615": 28107184128.0,
+ "5620": 28107184128.0,
+ "5625": 28107184128.0,
+ "5630": 28107184128.0,
+ "5635": 28107184128.0,
+ "5640": 28107184128.0,
+ "5645": 28107184128.0,
+ "5650": 28107184128.0,
+ "5655": 28107184128.0,
+ "5660": 28107184128.0,
+ "5665": 28107184128.0,
+ "5670": 28107184128.0,
+ "5675": 28107184128.0,
+ "5680": 28107184128.0,
+ "5685": 28107184128.0,
+ "5690": 28107184128.0,
+ "5695": 28107184128.0,
+ "5700": 28107184128.0,
+ "5705": 28107184128.0,
+ "5710": 28107184128.0,
+ "5715": 28107184128.0,
+ "5720": 28107184128.0,
+ "5725": 28107184128.0,
+ "5730": 28107184128.0,
+ "5735": 28107184128.0,
+ "5740": 28107184128.0,
+ "5745": 28107184128.0,
+ "5750": 28107184128.0,
+ "5755": 28107184128.0,
+ "5760": 28107184128.0,
+ "5765": 28107184128.0,
+ "5770": 28107184128.0,
+ "5775": 28107184128.0,
+ "5780": 28107184128.0,
+ "5785": 28107184128.0,
+ "5790": 28107184128.0,
+ "5795": 28107184128.0,
+ "5800": 28107184128.0,
+ "5805": 28107184128.0,
+ "5810": 28107184128.0,
+ "5815": 28107184128.0,
+ "5820": 28107184128.0,
+ "5825": 28107184128.0,
+ "5830": 28107184128.0,
+ "5835": 28107184128.0,
+ "5840": 28107184128.0,
+ "5845": 28107184128.0,
+ "5850": 28107184128.0,
+ "5855": 28107184128.0,
+ "5860": 28107184128.0,
+ "5865": 28107184128.0,
+ "5870": 28107184128.0,
+ "5875": 28107184128.0,
+ "5880": 28107184128.0,
+ "5885": 28107184128.0,
+ "5890": 28107184128.0,
+ "5895": 28107184128.0,
+ "5900": 28107184128.0,
+ "5905": 28107184128.0,
+ "5910": 28107184128.0,
+ "5915": 28107184128.0,
+ "5920": 28107184128.0,
+ "5925": 28107184128.0,
+ "5930": 28107184128.0,
+ "5935": 28107184128.0,
+ "5940": 28107184128.0,
+ "5945": 28107184128.0,
+ "5950": 28107184128.0,
+ "5955": 28107184128.0,
+ "5960": 28107184128.0,
+ "5965": 28107184128.0,
+ "5970": 28107184128.0,
+ "5975": 28107184128.0,
+ "5980": 28107184128.0,
+ "5985": 28107184128.0,
+ "5990": 28107184128.0,
+ "5995": 28107184128.0,
+ "6000": 28107184128.0,
+ "6005": 28107184128.0,
+ "6010": 28107184128.0,
+ "6015": 28107184128.0,
+ "6020": 28107184128.0,
+ "6025": 28107184128.0,
+ "6030": 28107184128.0,
+ "6035": 28107184128.0,
+ "6040": 28107184128.0,
+ "6045": 28107184128.0,
+ "6050": 28107184128.0,
+ "6055": 28107184128.0,
+ "6060": 28107184128.0,
+ "6065": 28107184128.0,
+ "6070": 28107184128.0,
+ "6075": 28107184128.0,
+ "6080": 28107184128.0,
+ "6085": 28107184128.0,
+ "6090": 28107184128.0,
+ "6095": 28107184128.0,
+ "6100": 28107184128.0,
+ "6105": 28107184128.0,
+ "6110": 28107184128.0,
+ "6115": 28107184128.0,
+ "6120": 28107184128.0,
+ "6125": 28107184128.0,
+ "6130": 28107184128.0,
+ "6135": 28107184128.0,
+ "6140": 28107184128.0,
+ "6145": 28107184128.0,
+ "6150": 28107184128.0,
+ "6155": 28107184128.0,
+ "6160": 28107184128.0,
+ "6165": 28107184128.0,
+ "6170": 28107184128.0,
+ "6175": 28107184128.0,
+ "6180": 28107184128.0,
+ "6185": 28107184128.0,
+ "6190": 28107184128.0,
+ "6195": 28107184128.0,
+ "6200": 28107184128.0,
+ "6205": 28107184128.0,
+ "6210": 28107184128.0,
+ "6215": 28107184128.0,
+ "6220": 28107184128.0,
+ "6225": 28107184128.0,
+ "6230": 28107184128.0,
+ "6235": 28107184128.0,
+ "6240": 28107184128.0,
+ "6245": 28107184128.0,
+ "6250": 28107184128.0,
+ "6255": 28107184128.0,
+ "6260": 28107184128.0,
+ "6265": 28107184128.0,
+ "6270": 28107184128.0,
+ "6275": 28107184128.0,
+ "6280": 28107184128.0,
+ "6285": 28107184128.0,
+ "6290": 28107184128.0,
+ "6295": 28107184128.0,
+ "6300": 28107184128.0,
+ "6305": 28107184128.0,
+ "6310": 28107184128.0,
+ "6315": 28107184128.0,
+ "6320": 28107184128.0,
+ "6325": 28107184128.0,
+ "6330": 28107184128.0,
+ "6335": 28107184128.0,
+ "6340": 28107184128.0,
+ "6345": 28107184128.0,
+ "6350": 28107184128.0,
+ "6355": 28107184128.0,
+ "6360": 28107184128.0,
+ "6365": 28107184128.0,
+ "6370": 28107184128.0,
+ "6375": 28107184128.0,
+ "6380": 28107184128.0,
+ "6385": 28107184128.0,
+ "6390": 28107184128.0,
+ "6395": 28107184128.0,
+ "6400": 28107184128.0,
+ "6405": 28107184128.0,
+ "6410": 28107184128.0,
+ "6415": 28107184128.0,
+ "6420": 28107184128.0,
+ "6425": 28107184128.0,
+ "6430": 28107184128.0,
+ "6435": 28107184128.0,
+ "6440": 28107184128.0,
+ "6445": 28107184128.0,
+ "6450": 28107184128.0,
+ "6455": 28107184128.0,
+ "6460": 28107184128.0,
+ "6465": 28107184128.0,
+ "6470": 28107184128.0,
+ "6475": 28107184128.0,
+ "6480": 28107184128.0,
+ "6485": 28107184128.0,
+ "6490": 28107184128.0,
+ "6495": 28107184128.0,
+ "6500": 28107184128.0,
+ "6505": 28107184128.0,
+ "6510": 28107184128.0,
+ "6515": 28107184128.0,
+ "6520": 28107184128.0,
+ "6525": 28107184128.0,
+ "6530": 28107184128.0,
+ "6535": 28107184128.0,
+ "6540": 28107184128.0,
+ "6545": 28107184128.0,
+ "6550": 28107184128.0,
+ "6555": 28107184128.0,
+ "6560": 28107184128.0,
+ "6565": 28107184128.0,
+ "6570": 28107184128.0,
+ "6575": 28107184128.0,
+ "6580": 28107184128.0,
+ "6585": 28107184128.0,
+ "6590": 28107184128.0,
+ "6595": 28107184128.0,
+ "6600": 28107184128.0,
+ "6605": 28107184128.0,
+ "6610": 28107184128.0,
+ "6615": 28107184128.0,
+ "6620": 28107184128.0,
+ "6625": 28107184128.0,
+ "6630": 28107184128.0,
+ "6635": 28107184128.0,
+ "6640": 28107184128.0,
+ "6645": 28107184128.0,
+ "6650": 28107184128.0,
+ "6655": 28107184128.0,
+ "6660": 28107184128.0,
+ "6665": 28107184128.0,
+ "6670": 28107184128.0,
+ "6675": 28107184128.0,
+ "6680": 28107184128.0,
+ "6685": 28107184128.0,
+ "6690": 28107184128.0,
+ "6695": 28107184128.0,
+ "6700": 28107184128.0,
+ "6705": 28107184128.0,
+ "6710": 28107184128.0,
+ "6715": 28107184128.0,
+ "6720": 28107184128.0,
+ "6725": 28107184128.0,
+ "6730": 28107184128.0,
+ "6735": 28107184128.0,
+ "6740": 28107184128.0,
+ "6745": 28107184128.0,
+ "6750": 28107184128.0,
+ "6755": 28107184128.0,
+ "6760": 28107184128.0,
+ "6765": 28107184128.0,
+ "6770": 28107184128.0,
+ "6775": 28107184128.0,
+ "6780": 28107184128.0,
+ "6785": 28107184128.0,
+ "6790": 28107184128.0,
+ "6795": 28107184128.0,
+ "6800": 28107184128.0,
+ "6805": 28107184128.0,
+ "6810": 28107184128.0,
+ "6815": 28107184128.0,
+ "6820": 28107184128.0,
+ "6825": 28107184128.0,
+ "6830": 28107184128.0,
+ "6835": 28107184128.0,
+ "6840": 28107184128.0,
+ "6845": 28107184128.0,
+ "6850": 28107184128.0,
+ "6855": 28107184128.0,
+ "6860": 28107184128.0,
+ "6865": 28107184128.0,
+ "6870": 28107184128.0,
+ "6875": 28107184128.0,
+ "6880": 28107184128.0,
+ "6885": 28107184128.0,
+ "6890": 28107184128.0,
+ "6895": 28107184128.0,
+ "6900": 28107184128.0,
+ "6905": 28107184128.0,
+ "6910": 28107184128.0,
+ "6915": 28107184128.0,
+ "6920": 28107184128.0,
+ "6925": 28107184128.0,
+ "6930": 28107184128.0,
+ "6935": 28107184128.0,
+ "6940": 28107184128.0,
+ "6945": 28107184128.0,
+ "6950": 28107184128.0,
+ "6955": 28107184128.0,
+ "6960": 28107184128.0,
+ "6965": 28107184128.0,
+ "6970": 28107184128.0,
+ "6975": 28107184128.0,
+ "6980": 28107184128.0,
+ "6985": 28107184128.0,
+ "6990": 28107184128.0,
+ "6995": 28107184128.0,
+ "7000": 28107184128.0,
+ "7005": 28107184128.0,
+ "7010": 28107184128.0,
+ "7015": 28107184128.0,
+ "7020": 28107184128.0,
+ "7025": 28107184128.0,
+ "7030": 28107184128.0,
+ "7035": 28107184128.0,
+ "7040": 28107184128.0,
+ "7045": 28107184128.0,
+ "7050": 28107184128.0,
+ "7055": 28107184128.0,
+ "7060": 28107184128.0,
+ "7065": 28107184128.0,
+ "7070": 28107184128.0,
+ "7075": 28107184128.0,
+ "7080": 28107184128.0,
+ "7085": 28107184128.0,
+ "7090": 28107184128.0,
+ "7095": 28107184128.0,
+ "7100": 28107184128.0,
+ "7105": 28107184128.0,
+ "7110": 28107184128.0,
+ "7115": 28107184128.0,
+ "7120": 28107184128.0,
+ "7125": 28107184128.0,
+ "7130": 28107184128.0,
+ "7135": 28107184128.0,
+ "7140": 28107184128.0,
+ "7145": 28107184128.0,
+ "7150": 28107184128.0,
+ "7155": 28107184128.0,
+ "7160": 28107184128.0,
+ "7165": 28107184128.0,
+ "7170": 28107184128.0,
+ "7175": 28107184128.0,
+ "7180": 28107184128.0,
+ "7185": 28107184128.0,
+ "7190": 28107184128.0,
+ "7195": 28107184128.0,
+ "7200": 28107184128.0,
+ "7205": 28107184128.0,
+ "7210": 28107184128.0,
+ "7215": 28107184128.0,
+ "7220": 28107184128.0,
+ "7225": 28107184128.0,
+ "7230": 28107184128.0,
+ "7235": 28107184128.0,
+ "7240": 28107184128.0,
+ "7245": 28107184128.0,
+ "7250": 28107184128.0,
+ "7255": 28107184128.0,
+ "7260": 28107184128.0,
+ "7265": 28107184128.0,
+ "7270": 28107184128.0,
+ "7275": 28107184128.0,
+ "7280": 28107184128.0,
+ "7285": 28107184128.0,
+ "7290": 28107184128.0,
+ "7295": 28107184128.0,
+ "7300": 28107184128.0,
+ "7305": 28107184128.0,
+ "7310": 28107184128.0,
+ "7315": 28107184128.0,
+ "7320": 28107184128.0,
+ "7325": 28107184128.0,
+ "7330": 28107184128.0,
+ "7335": 28107184128.0,
+ "7340": 28107184128.0,
+ "7345": 28107184128.0,
+ "7350": 28107184128.0,
+ "7355": 28107184128.0,
+ "7360": 28107184128.0,
+ "7365": 28107184128.0,
+ "7370": 28107184128.0,
+ "7375": 28107184128.0,
+ "7380": 28107184128.0,
+ "7385": 28107184128.0,
+ "7390": 28107184128.0,
+ "7395": 28107184128.0,
+ "7400": 28107184128.0,
+ "7405": 28107184128.0,
+ "7410": 28107184128.0,
+ "7415": 28107184128.0,
+ "7420": 28107184128.0,
+ "7425": 28107184128.0,
+ "7430": 28107184128.0,
+ "7435": 28107184128.0,
+ "7440": 28107184128.0,
+ "7445": 28107184128.0,
+ "7450": 28107184128.0,
+ "7455": 28107184128.0,
+ "7460": 28107184128.0,
+ "7465": 28107184128.0,
+ "7470": 28107184128.0,
+ "7475": 28107184128.0,
+ "7480": 28107184128.0,
+ "7485": 28107184128.0,
+ "7490": 28107184128.0,
+ "7495": 28107184128.0,
+ "7500": 28107184128.0,
+ "7505": 28107184128.0,
+ "7510": 28107184128.0,
+ "7515": 28107184128.0,
+ "7520": 28107184128.0,
+ "7525": 28107184128.0,
+ "7530": 28107184128.0,
+ "7535": 28107184128.0,
+ "7540": 28107184128.0,
+ "7545": 28107184128.0,
+ "7550": 28107184128.0,
+ "7555": 28107184128.0,
+ "7560": 28107184128.0,
+ "7565": 28107184128.0,
+ "7570": 28107184128.0,
+ "7575": 28107184128.0,
+ "7580": 28107184128.0,
+ "7585": 28107184128.0,
+ "7590": 28107184128.0,
+ "7595": 28107184128.0,
+ "7600": 28107184128.0,
+ "7605": 28107184128.0,
+ "7610": 28107184128.0,
+ "7615": 28107184128.0,
+ "7620": 28107184128.0,
+ "7625": 28107184128.0,
+ "7630": 28107184128.0,
+ "7635": 28107184128.0,
+ "7640": 28107184128.0,
+ "7645": 28107184128.0,
+ "7650": 28107184128.0,
+ "7655": 28107184128.0,
+ "7660": 28107184128.0,
+ "7665": 28107184128.0,
+ "7670": 28107184128.0,
+ "7675": 28107184128.0,
+ "7680": 28107184128.0,
+ "7685": 28107184128.0,
+ "7690": 28107184128.0,
+ "7695": 28107184128.0,
+ "7700": 28107184128.0,
+ "7705": 28107184128.0,
+ "7710": 28107184128.0,
+ "7715": 28107184128.0,
+ "7720": 28107184128.0,
+ "7725": 28107184128.0,
+ "7730": 28107184128.0,
+ "7735": 28107184128.0,
+ "7740": 28107184128.0,
+ "7745": 28107184128.0,
+ "7750": 28107184128.0,
+ "7755": 28107184128.0,
+ "7760": 28107184128.0,
+ "7765": 28107184128.0,
+ "7770": 28107184128.0,
+ "7775": 28107184128.0,
+ "7780": 28107184128.0,
+ "7785": 28107184128.0,
+ "7790": 28107184128.0,
+ "7795": 28107184128.0,
+ "7800": 28107184128.0,
+ "7805": 28107184128.0,
+ "7810": 28107184128.0,
+ "7815": 28107184128.0,
+ "7820": 28107184128.0,
+ "7825": 28107184128.0,
+ "7830": 28107184128.0,
+ "7835": 28107184128.0,
+ "7840": 28107184128.0,
+ "7845": 28107184128.0,
+ "7850": 28107184128.0,
+ "7855": 28107184128.0,
+ "7860": 28107184128.0,
+ "7865": 28107184128.0,
+ "7870": 28107184128.0,
+ "7875": 28107184128.0,
+ "7880": 28107184128.0,
+ "7885": 28107184128.0,
+ "7890": 28107184128.0,
+ "7895": 28107184128.0,
+ "7900": 28107184128.0,
+ "7905": 28107184128.0,
+ "7910": 28107184128.0,
+ "7915": 28107184128.0,
+ "7920": 28107184128.0,
+ "7925": 28107184128.0,
+ "7930": 28107184128.0,
+ "7935": 28107184128.0,
+ "7940": 28107184128.0,
+ "7945": 28107184128.0,
+ "7950": 28107184128.0,
+ "7955": 28107184128.0,
+ "7960": 28107184128.0,
+ "7965": 28107184128.0,
+ "7970": 28107184128.0,
+ "7975": 28107184128.0,
+ "7980": 28107184128.0,
+ "7985": 28107184128.0,
+ "7990": 28107184128.0,
+ "7995": 28107184128.0,
+ "8000": 28107184128.0,
+ "8005": 28107184128.0,
+ "8010": 28107184128.0,
+ "8015": 28107184128.0,
+ "8020": 28107184128.0,
+ "8025": 28107184128.0,
+ "8030": 28107184128.0,
+ "8035": 28107184128.0,
+ "8040": 28107184128.0,
+ "8045": 28107184128.0,
+ "8050": 28107184128.0,
+ "8055": 28107184128.0,
+ "8060": 28107184128.0,
+ "8065": 28107184128.0,
+ "8070": 28107184128.0,
+ "8075": 28107184128.0,
+ "8080": 28107184128.0,
+ "8085": 28107184128.0,
+ "8090": 28107184128.0,
+ "8095": 28107184128.0,
+ "8100": 28107184128.0,
+ "8105": 28107184128.0,
+ "8110": 28107184128.0,
+ "8115": 28107184128.0,
+ "8120": 28107184128.0,
+ "8125": 28107184128.0,
+ "8130": 28107184128.0,
+ "8135": 28107184128.0,
+ "8140": 28107184128.0,
+ "8145": 28107184128.0,
+ "8150": 28107184128.0,
+ "8155": 28107184128.0,
+ "8160": 28107184128.0,
+ "8165": 28107184128.0,
+ "8170": 28107184128.0,
+ "8175": 28107184128.0,
+ "8180": 28107184128.0,
+ "8185": 28107184128.0,
+ "8190": 28107184128.0,
+ "8195": 28107184128.0,
+ "8200": 28107184128.0,
+ "8205": 28107184128.0,
+ "8210": 28107184128.0,
+ "8215": 28107184128.0,
+ "8220": 28107184128.0,
+ "8225": 28107184128.0,
+ "8230": 28107184128.0,
+ "8235": 28107184128.0,
+ "8240": 28107184128.0,
+ "8245": 28107184128.0,
+ "8250": 28107184128.0,
+ "8255": 28107184128.0,
+ "8260": 28107184128.0,
+ "8265": 28107184128.0,
+ "8270": 28107184128.0,
+ "8275": 28107184128.0,
+ "8280": 28107184128.0,
+ "8285": 28107184128.0,
+ "8290": 28107184128.0,
+ "8295": 28107184128.0,
+ "8300": 28107184128.0,
+ "8305": 28107184128.0,
+ "8310": 28107184128.0,
+ "8315": 28107184128.0,
+ "8320": 28107184128.0,
+ "8325": 28107184128.0,
+ "8330": 28107184128.0,
+ "8335": 28107184128.0,
+ "8340": 28107184128.0,
+ "8345": 28107184128.0,
+ "8350": 28107184128.0,
+ "8355": 28107184128.0,
+ "8360": 28107184128.0,
+ "8365": 28107184128.0,
+ "8370": 28107184128.0,
+ "8375": 28107184128.0,
+ "8380": 28107184128.0,
+ "8385": 28107184128.0,
+ "8390": 28107184128.0,
+ "8395": 28107184128.0,
+ "8400": 28107184128.0,
+ "8405": 28107184128.0,
+ "8410": 28107184128.0,
+ "8415": 28107184128.0,
+ "8420": 28107184128.0,
+ "8425": 28107184128.0,
+ "8430": 28107184128.0,
+ "8435": 28107184128.0,
+ "8440": 28107184128.0,
+ "8445": 28107184128.0,
+ "8450": 28107184128.0,
+ "8455": 28107184128.0,
+ "8460": 28107184128.0,
+ "8465": 28107184128.0,
+ "8470": 28107184128.0,
+ "8475": 28107184128.0,
+ "8480": 28107184128.0,
+ "8485": 28107184128.0,
+ "8490": 28107184128.0,
+ "8495": 28107184128.0,
+ "8500": 28107184128.0,
+ "8505": 28107184128.0,
+ "8510": 28107184128.0,
+ "8515": 28107184128.0,
+ "8520": 28107184128.0,
+ "8525": 28107184128.0,
+ "8530": 28107184128.0,
+ "8535": 28107184128.0,
+ "8540": 28107184128.0,
+ "8545": 28107184128.0,
+ "8550": 28107184128.0,
+ "8555": 28107184128.0,
+ "8560": 28107184128.0,
+ "8565": 28107184128.0,
+ "8570": 28107184128.0,
+ "8575": 28107184128.0,
+ "8580": 28107184128.0,
+ "8585": 28107184128.0,
+ "8590": 28107184128.0,
+ "8595": 28107184128.0,
+ "8600": 28107184128.0,
+ "8605": 28107184128.0,
+ "8610": 28107184128.0,
+ "8615": 28107184128.0,
+ "8620": 28107184128.0,
+ "8625": 28107184128.0,
+ "8630": 28107184128.0,
+ "8635": 28107184128.0,
+ "8640": 28107184128.0,
+ "8645": 28107184128.0,
+ "8650": 28107184128.0,
+ "8655": 28107184128.0,
+ "8660": 28107184128.0,
+ "8665": 28107184128.0,
+ "8670": 28107184128.0,
+ "8675": 28107184128.0,
+ "8680": 28107184128.0,
+ "8685": 28107184128.0,
+ "8690": 28107184128.0,
+ "8695": 28107184128.0,
+ "8700": 28107184128.0,
+ "8705": 28107184128.0,
+ "8710": 28107184128.0,
+ "8715": 28107184128.0,
+ "8720": 28107184128.0,
+ "8725": 28107184128.0,
+ "8730": 28107184128.0,
+ "8735": 28107184128.0,
+ "8740": 28107184128.0,
+ "8745": 28107184128.0,
+ "8750": 28107184128.0,
+ "8755": 28107184128.0,
+ "8760": 28107184128.0,
+ "8765": 28107184128.0,
+ "8770": 28107184128.0,
+ "8775": 28107184128.0,
+ "8780": 28107184128.0,
+ "8785": 28107184128.0,
+ "8790": 28107184128.0,
+ "8795": 28107184128.0,
+ "8800": 28107184128.0,
+ "8805": 28107184128.0,
+ "8810": 28107184128.0,
+ "8815": 28107184128.0,
+ "8820": 28107184128.0,
+ "8825": 28107184128.0,
+ "8830": 28107184128.0,
+ "8835": 28107184128.0,
+ "8840": 28107184128.0,
+ "8845": 28107184128.0,
+ "8850": 28107184128.0,
+ "8855": 28107184128.0,
+ "8860": 28107184128.0,
+ "8865": 28107184128.0,
+ "8870": 28107184128.0,
+ "8875": 28107184128.0,
+ "8880": 28107184128.0,
+ "8885": 28107184128.0,
+ "8890": 28107184128.0,
+ "8895": 28107184128.0,
+ "8900": 28107184128.0,
+ "8905": 28107184128.0,
+ "8910": 28107184128.0,
+ "8915": 28107184128.0,
+ "8920": 28107184128.0,
+ "8925": 28107184128.0,
+ "8930": 28107184128.0,
+ "8935": 28107184128.0,
+ "8940": 28107184128.0,
+ "8945": 28107184128.0,
+ "8950": 28107184128.0,
+ "8955": 28107184128.0,
+ "8960": 28107184128.0,
+ "8965": 28107184128.0,
+ "8970": 28107184128.0,
+ "8975": 28107184128.0,
+ "8980": 28107184128.0,
+ "8985": 28107184128.0,
+ "8990": 28107184128.0,
+ "8995": 28107184128.0,
+ "9000": 28107184128.0,
+ "9005": 28107184128.0,
+ "9010": 28107184128.0,
+ "9015": 28107184128.0,
+ "9020": 28107184128.0,
+ "9025": 28107184128.0,
+ "9030": 28107184128.0,
+ "9035": 28107184128.0,
+ "9040": 28107184128.0,
+ "9045": 28107184128.0,
+ "9050": 28107184128.0,
+ "9055": 28107184128.0,
+ "9060": 28107184128.0,
+ "9065": 28107184128.0,
+ "9070": 28107184128.0,
+ "9075": 28107184128.0,
+ "9080": 28107184128.0,
+ "9085": 28107184128.0,
+ "9090": 28107184128.0,
+ "9095": 28107184128.0,
+ "9100": 28107184128.0,
+ "9105": 28107184128.0,
+ "9110": 28107184128.0,
+ "9115": 28107184128.0,
+ "9120": 28107184128.0,
+ "9125": 28107184128.0,
+ "9130": 28107184128.0,
+ "9135": 28107184128.0,
+ "9140": 28107184128.0,
+ "9145": 28107184128.0,
+ "9150": 28107184128.0,
+ "9155": 28107184128.0,
+ "9160": 28107184128.0,
+ "9165": 28107184128.0,
+ "9170": 28107184128.0,
+ "9175": 28107184128.0,
+ "9180": 28107184128.0,
+ "9185": 28107184128.0,
+ "9190": 28107184128.0,
+ "9195": 28107184128.0,
+ "9200": 28107184128.0,
+ "9205": 28107184128.0,
+ "9210": 28107184128.0,
+ "9215": 28107184128.0,
+ "9220": 28107184128.0,
+ "9225": 28107184128.0,
+ "9230": 28107184128.0,
+ "9235": 28107184128.0,
+ "9240": 28107184128.0,
+ "9245": 28107184128.0,
+ "9250": 28107184128.0,
+ "9255": 28107184128.0,
+ "9260": 28107184128.0,
+ "9265": 28107184128.0,
+ "9270": 28107184128.0,
+ "9275": 28107184128.0,
+ "9280": 28107184128.0,
+ "9285": 28107184128.0,
+ "9290": 28107184128.0,
+ "9295": 28107184128.0,
+ "9300": 28107184128.0,
+ "9305": 28107184128.0,
+ "9310": 28107184128.0,
+ "9315": 28107184128.0,
+ "9320": 28107184128.0,
+ "9325": 28107184128.0,
+ "9330": 28107184128.0,
+ "9335": 28107184128.0,
+ "9340": 28107184128.0,
+ "9345": 28107184128.0,
+ "9350": 28107184128.0,
+ "9355": 28107184128.0,
+ "9360": 28107184128.0,
+ "9365": 28107184128.0,
+ "9370": 28107184128.0,
+ "9375": 28107184128.0,
+ "9380": 28107184128.0,
+ "9385": 28107184128.0,
+ "9390": 28107184128.0,
+ "9395": 28107184128.0,
+ "9400": 28107184128.0,
+ "9405": 28107184128.0,
+ "9410": 28107184128.0,
+ "9415": 28107184128.0,
+ "9420": 28107184128.0,
+ "9425": 28107184128.0,
+ "9430": 28107184128.0,
+ "9435": 28107184128.0,
+ "9440": 28107184128.0,
+ "9445": 28107184128.0,
+ "9450": 28107184128.0,
+ "9455": 28107184128.0,
+ "9460": 28107184128.0,
+ "9465": 28107184128.0,
+ "9470": 28107184128.0,
+ "9475": 28107184128.0,
+ "9480": 28107184128.0,
+ "9485": 28107184128.0,
+ "9490": 28107184128.0,
+ "9495": 28107184128.0,
+ "9500": 28107184128.0,
+ "9505": 28107184128.0,
+ "9510": 28107184128.0,
+ "9515": 28107184128.0,
+ "9520": 28107184128.0,
+ "9525": 28107184128.0,
+ "9530": 28107184128.0,
+ "9535": 28107184128.0,
+ "9540": 28107184128.0,
+ "9545": 28107184128.0,
+ "9550": 28107184128.0,
+ "9555": 28107184128.0,
+ "9560": 28107184128.0,
+ "9565": 28107184128.0,
+ "9570": 28107184128.0,
+ "9575": 28107184128.0,
+ "9580": 28107184128.0,
+ "9585": 28107184128.0,
+ "9590": 28107184128.0,
+ "9595": 28107184128.0,
+ "9600": 28107184128.0,
+ "9605": 28107184128.0,
+ "9610": 28107184128.0,
+ "9615": 28107184128.0,
+ "9620": 28107184128.0,
+ "9625": 28107184128.0,
+ "9630": 28107184128.0,
+ "9635": 28107184128.0,
+ "9640": 28107184128.0,
+ "9645": 28107184128.0,
+ "9650": 28107184128.0,
+ "9655": 28107184128.0,
+ "9660": 28107184128.0,
+ "9665": 28107184128.0,
+ "9670": 28107184128.0,
+ "9675": 28107184128.0,
+ "9680": 28107184128.0,
+ "9685": 28107184128.0,
+ "9690": 28107184128.0,
+ "9695": 28107184128.0,
+ "9700": 28107184128.0,
+ "9705": 28107184128.0,
+ "9710": 28107184128.0,
+ "9715": 28107184128.0,
+ "9720": 28107184128.0,
+ "9725": 28107184128.0,
+ "9730": 28107184128.0,
+ "9735": 28107184128.0,
+ "9740": 28107184128.0,
+ "9745": 28107184128.0,
+ "9750": 28107184128.0,
+ "9755": 28107184128.0,
+ "9760": 28107184128.0,
+ "9765": 28107184128.0,
+ "9770": 28107184128.0,
+ "9775": 28107184128.0,
+ "9780": 28107184128.0,
+ "9785": 28107184128.0,
+ "9790": 28107184128.0,
+ "9795": 28107184128.0,
+ "9800": 28107184128.0,
+ "9805": 28107184128.0,
+ "9810": 28107184128.0,
+ "9815": 28107184128.0,
+ "9820": 28107184128.0,
+ "9825": 28107184128.0,
+ "9830": 28107184128.0,
+ "9835": 28107184128.0,
+ "9840": 28107184128.0,
+ "9845": 28107184128.0,
+ "9850": 28107184128.0,
+ "9855": 28107184128.0,
+ "9860": 28107184128.0,
+ "9865": 28107184128.0,
+ "9870": 28107184128.0,
+ "9875": 28107184128.0,
+ "9880": 28107184128.0,
+ "9885": 28107184128.0,
+ "9890": 28107184128.0,
+ "9895": 28107184128.0,
+ "9900": 28107184128.0,
+ "9905": 28107184128.0,
+ "9910": 28107184128.0,
+ "9915": 28107184128.0,
+ "9920": 28107184128.0,
+ "9925": 28107184128.0,
+ "9930": 28107184128.0,
+ "9935": 28107184128.0,
+ "9940": 28107184128.0,
+ "9945": 28107184128.0,
+ "9950": 28107184128.0,
+ "9955": 28107184128.0,
+ "9960": 28107184128.0,
+ "9965": 28107184128.0,
+ "9970": 28107184128.0,
+ "9975": 28107184128.0,
+ "9980": 28107184128.0,
+ "9985": 28107184128.0,
+ "9990": 28107184128.0,
+ "9995": 28107184128.0,
+ "10000": 28107184128.0,
+ "10005": 28107184128.0,
+ "10010": 28107184128.0,
+ "10015": 28107184128.0,
+ "10020": 28107184128.0,
+ "10025": 28107184128.0,
+ "10030": 28107184128.0,
+ "10035": 28107184128.0,
+ "10040": 28107184128.0,
+ "10045": 28107184128.0,
+ "10050": 28107184128.0,
+ "10055": 28107184128.0,
+ "10060": 28107184128.0,
+ "10065": 28107184128.0,
+ "10070": 28107184128.0,
+ "10075": 28107184128.0,
+ "10080": 28107184128.0,
+ "10085": 28107184128.0,
+ "10090": 28107184128.0,
+ "10095": 28107184128.0,
+ "10100": 28107184128.0,
+ "10105": 28107184128.0,
+ "10110": 28107184128.0,
+ "10115": 28107184128.0,
+ "10120": 28107184128.0,
+ "10125": 28107184128.0,
+ "10130": 28107184128.0,
+ "10135": 28107184128.0,
+ "10140": 28107184128.0,
+ "10145": 28107184128.0,
+ "10150": 28107184128.0,
+ "10155": 28107184128.0,
+ "10160": 28107184128.0,
+ "10165": 28107184128.0,
+ "10170": 28107184128.0,
+ "10175": 28107184128.0,
+ "10180": 28107184128.0,
+ "10185": 28107184128.0,
+ "10190": 28107184128.0,
+ "10195": 28107184128.0,
+ "10200": 28107184128.0,
+ "10205": 28107184128.0,
+ "10210": 28107184128.0,
+ "10215": 28107184128.0,
+ "10220": 28107184128.0,
+ "10225": 28107184128.0,
+ "10230": 28107184128.0,
+ "10235": 28107184128.0,
+ "10240": 28107184128.0,
+ "10245": 28107184128.0,
+ "10250": 28107184128.0,
+ "10255": 28107184128.0,
+ "10260": 28107184128.0,
+ "10265": 28107184128.0,
+ "10270": 28107184128.0,
+ "10275": 28107184128.0,
+ "10280": 28107184128.0,
+ "10285": 28107184128.0,
+ "10290": 28107184128.0,
+ "10295": 28107184128.0,
+ "10300": 28107184128.0,
+ "10305": 28107184128.0,
+ "10310": 28107184128.0,
+ "10315": 28107184128.0,
+ "10320": 28107184128.0,
+ "10325": 28107184128.0,
+ "10330": 28107184128.0,
+ "10335": 28107184128.0,
+ "10340": 28107184128.0,
+ "10345": 28107184128.0,
+ "10350": 28107184128.0,
+ "10355": 28107184128.0,
+ "10360": 28107184128.0,
+ "10365": 28107184128.0,
+ "10370": 28107184128.0,
+ "10375": 28107184128.0,
+ "10380": 28107184128.0,
+ "10385": 28107184128.0,
+ "10390": 28107184128.0,
+ "10395": 28107184128.0,
+ "10400": 28107184128.0,
+ "10405": 28107184128.0,
+ "10410": 28107184128.0,
+ "10415": 28107184128.0,
+ "10420": 28107184128.0,
+ "10425": 28107184128.0,
+ "10430": 28107184128.0,
+ "10435": 28107184128.0,
+ "10440": 28107184128.0,
+ "10445": 28107184128.0,
+ "10450": 28107184128.0,
+ "10455": 28107184128.0,
+ "10460": 28107184128.0,
+ "10465": 28107184128.0,
+ "10470": 28107184128.0,
+ "10475": 28107184128.0,
+ "10480": 28107184128.0,
+ "10485": 28107184128.0,
+ "10490": 28107184128.0,
+ "10495": 28107184128.0,
+ "10500": 28107184128.0,
+ "10505": 28107184128.0,
+ "10510": 28107184128.0,
+ "10515": 28107184128.0,
+ "10520": 28107184128.0,
+ "10525": 28107184128.0,
+ "10530": 28107184128.0,
+ "10535": 28107184128.0,
+ "10540": 28107184128.0,
+ "10545": 28107184128.0,
+ "10550": 28107184128.0,
+ "10555": 28107184128.0,
+ "10560": 28107184128.0,
+ "10565": 28107184128.0,
+ "10570": 28107184128.0,
+ "10575": 28107184128.0,
+ "10580": 28107184128.0,
+ "10585": 28107184128.0,
+ "10590": 28107184128.0,
+ "10595": 28107184128.0,
+ "10600": 28107184128.0,
+ "10605": 28107184128.0,
+ "10610": 28107184128.0,
+ "10615": 28107184128.0,
+ "10620": 28107184128.0,
+ "10625": 28107184128.0,
+ "10630": 28107184128.0,
+ "10635": 28107184128.0,
+ "10640": 28107184128.0,
+ "10645": 28107184128.0,
+ "10650": 28107184128.0,
+ "10655": 28107184128.0,
+ "10660": 28107184128.0,
+ "10665": 28107184128.0,
+ "10670": 28107184128.0,
+ "10675": 28107184128.0,
+ "10680": 28107184128.0,
+ "10685": 28107184128.0,
+ "10690": 28107184128.0,
+ "10695": 28107184128.0,
+ "10700": 28107184128.0,
+ "10705": 28107184128.0,
+ "10710": 28107184128.0,
+ "10715": 28107184128.0,
+ "10720": 28107184128.0,
+ "10725": 28107184128.0,
+ "10730": 28107184128.0,
+ "10735": 28107184128.0,
+ "10740": 28107184128.0,
+ "10745": 28107184128.0,
+ "10750": 28107184128.0,
+ "10755": 28107184128.0,
+ "10760": 28107184128.0,
+ "10765": 28107184128.0,
+ "10770": 28107184128.0,
+ "10775": 28107184128.0,
+ "10780": 28107184128.0,
+ "10785": 28107184128.0,
+ "10790": 28107184128.0,
+ "10795": 28107184128.0,
+ "10800": 28107184128.0,
+ "10805": 28107184128.0,
+ "10810": 28107184128.0,
+ "10815": 28107184128.0,
+ "10820": 28107184128.0,
+ "10825": 28107184128.0,
+ "10830": 28107184128.0,
+ "10835": 28107184128.0,
+ "10840": 28107184128.0,
+ "10845": 28107184128.0,
+ "10850": 28107184128.0,
+ "10855": 28107184128.0,
+ "10860": 28107184128.0,
+ "10865": 28107184128.0,
+ "10870": 28107184128.0,
+ "10875": 28107184128.0,
+ "10880": 28107184128.0,
+ "10885": 28107184128.0,
+ "10890": 28107184128.0,
+ "10895": 28107184128.0,
+ "10900": 28107184128.0,
+ "10905": 28107184128.0,
+ "10910": 28107184128.0,
+ "10915": 28107184128.0,
+ "10920": 28107184128.0,
+ "10925": 28107184128.0,
+ "10930": 28107184128.0,
+ "10935": 28107184128.0,
+ "10940": 28107184128.0,
+ "10945": 28107184128.0,
+ "10950": 28107184128.0,
+ "10955": 28107184128.0,
+ "10960": 28107184128.0,
+ "10965": 28107184128.0,
+ "10970": 28107184128.0,
+ "10975": 28107184128.0,
+ "10980": 28107184128.0,
+ "10985": 28107184128.0,
+ "10990": 28107184128.0,
+ "10995": 28107184128.0,
+ "11000": 28107184128.0,
+ "11005": 28107184128.0,
+ "11010": 28107184128.0,
+ "11015": 28107184128.0,
+ "11020": 28107184128.0,
+ "11025": 28107184128.0,
+ "11030": 28107184128.0,
+ "11035": 28107184128.0,
+ "11040": 28107184128.0,
+ "11045": 28107184128.0,
+ "11050": 28107184128.0,
+ "11055": 28107184128.0,
+ "11060": 28107184128.0,
+ "11065": 28107184128.0,
+ "11070": 28107184128.0,
+ "11075": 28107184128.0,
+ "11080": 28107184128.0,
+ "11085": 28107184128.0,
+ "11090": 28107184128.0,
+ "11095": 28107184128.0,
+ "11100": 28107184128.0,
+ "11105": 28107184128.0,
+ "11110": 28107184128.0,
+ "11115": 28107184128.0,
+ "11120": 28107184128.0,
+ "11125": 28107184128.0,
+ "11130": 28107184128.0,
+ "11135": 28107184128.0,
+ "11140": 28107184128.0,
+ "11145": 28107184128.0,
+ "11150": 28107184128.0,
+ "11155": 28107184128.0,
+ "11160": 28107184128.0,
+ "11165": 28107184128.0,
+ "11170": 28107184128.0,
+ "11175": 28107184128.0,
+ "11180": 28107184128.0,
+ "11185": 28107184128.0,
+ "11190": 28107184128.0,
+ "11195": 28107184128.0,
+ "11200": 28107184128.0,
+ "11205": 28107184128.0,
+ "11210": 28107184128.0,
+ "11215": 28107184128.0,
+ "11220": 28107184128.0,
+ "11225": 28107184128.0,
+ "11230": 28107184128.0,
+ "11235": 28107184128.0,
+ "11240": 28107184128.0,
+ "11245": 28107184128.0,
+ "11250": 28107184128.0,
+ "11255": 28107184128.0,
+ "11260": 28107184128.0,
+ "11265": 28107184128.0,
+ "11270": 28107184128.0,
+ "11275": 28107184128.0,
+ "11280": 28107184128.0,
+ "11285": 28107184128.0,
+ "11290": 28107184128.0,
+ "11295": 28107184128.0,
+ "11300": 28107184128.0,
+ "11305": 28107184128.0,
+ "11310": 28107184128.0,
+ "11315": 28107184128.0,
+ "11320": 28107184128.0,
+ "11325": 28107184128.0,
+ "11330": 28107184128.0,
+ "11335": 28107184128.0,
+ "11340": 28107184128.0,
+ "11345": 28107184128.0,
+ "11350": 28107184128.0,
+ "11355": 28107184128.0,
+ "11360": 28107184128.0,
+ "11365": 28107184128.0,
+ "11370": 28107184128.0,
+ "11375": 28107184128.0,
+ "11380": 28107184128.0,
+ "11385": 28107184128.0,
+ "11390": 28107184128.0,
+ "11395": 28107184128.0,
+ "11400": 28107184128.0,
+ "11405": 28107184128.0,
+ "11410": 28107184128.0,
+ "11415": 28107184128.0,
+ "11420": 28107184128.0,
+ "11425": 28107184128.0,
+ "11430": 28107184128.0,
+ "11435": 28107184128.0,
+ "11440": 28107184128.0,
+ "11445": 28107184128.0,
+ "11450": 28107184128.0,
+ "11455": 28107184128.0,
+ "11460": 28107184128.0,
+ "11465": 28107184128.0,
+ "11470": 28107184128.0,
+ "11475": 28107184128.0,
+ "11480": 28107184128.0,
+ "11485": 28107184128.0,
+ "11490": 28107184128.0,
+ "11495": 28107184128.0,
+ "11500": 28107184128.0,
+ "11505": 28107184128.0,
+ "11510": 28107184128.0,
+ "11515": 28107184128.0,
+ "11520": 28107184128.0,
+ "11525": 28107184128.0,
+ "11530": 28107184128.0,
+ "11535": 28107184128.0,
+ "11540": 28107184128.0,
+ "11545": 28107184128.0,
+ "11550": 28107184128.0,
+ "11555": 28107184128.0,
+ "11560": 28107184128.0,
+ "11565": 28107184128.0,
+ "11570": 28107184128.0,
+ "11575": 28107184128.0,
+ "11580": 28107184128.0,
+ "11585": 28107184128.0,
+ "11590": 28107184128.0,
+ "11595": 28107184128.0,
+ "11600": 28107184128.0,
+ "11605": 28107184128.0,
+ "11610": 28107184128.0,
+ "11615": 28107184128.0,
+ "11620": 28107184128.0,
+ "11625": 28107184128.0,
+ "11630": 28107184128.0,
+ "11635": 28107184128.0,
+ "11640": 28107184128.0,
+ "11645": 28107184128.0,
+ "11650": 28107184128.0,
+ "11655": 28107184128.0,
+ "11660": 28107184128.0,
+ "11665": 28107184128.0,
+ "11670": 28107184128.0,
+ "11675": 28107184128.0,
+ "11680": 28107184128.0,
+ "11685": 28107184128.0,
+ "11690": 28107184128.0,
+ "11695": 28107184128.0,
+ "11700": 28107184128.0,
+ "11705": 28107184128.0,
+ "11710": 28107184128.0,
+ "11715": 28107184128.0,
+ "11720": 28107184128.0,
+ "11725": 28107184128.0,
+ "11730": 28107184128.0,
+ "11735": 28107184128.0,
+ "11740": 28107184128.0,
+ "11745": 28107184128.0,
+ "11750": 28107184128.0,
+ "11755": 28107184128.0,
+ "11760": 28107184128.0,
+ "11765": 28107184128.0,
+ "11770": 28107184128.0,
+ "11775": 28107184128.0,
+ "11780": 28107184128.0,
+ "11785": 28107184128.0,
+ "11790": 28107184128.0,
+ "11795": 28107184128.0,
+ "11800": 28107184128.0,
+ "11805": 28107184128.0,
+ "11810": 28107184128.0,
+ "11815": 28107184128.0,
+ "11820": 28107184128.0,
+ "11825": 28107184128.0,
+ "11830": 28107184128.0,
+ "11835": 28107184128.0,
+ "11840": 28107184128.0,
+ "11845": 28107184128.0,
+ "11850": 28107184128.0,
+ "11855": 28107184128.0,
+ "11860": 28107184128.0,
+ "11865": 28107184128.0,
+ "11870": 28107184128.0,
+ "11875": 28107184128.0,
+ "11880": 28107184128.0,
+ "11885": 28107184128.0,
+ "11890": 28107184128.0,
+ "11895": 28107184128.0,
+ "11900": 28107184128.0,
+ "11905": 28107184128.0,
+ "11910": 28107184128.0,
+ "11915": 28107184128.0,
+ "11920": 28107184128.0,
+ "11925": 28107184128.0,
+ "11930": 28107184128.0,
+ "11935": 28107184128.0,
+ "11940": 28107184128.0,
+ "11945": 28107184128.0,
+ "11950": 28107184128.0,
+ "11955": 28107184128.0,
+ "11960": 28107184128.0,
+ "11965": 28107184128.0,
+ "11970": 28107184128.0,
+ "11975": 28107184128.0,
+ "11980": 28107184128.0,
+ "11985": 28107184128.0,
+ "11990": 28107184128.0,
+ "11995": 28107184128.0,
+ "12000": 28107184128.0,
+ "12005": 28107184128.0,
+ "12010": 28107184128.0,
+ "12015": 28107184128.0,
+ "12020": 28107184128.0,
+ "12025": 28107184128.0,
+ "12030": 28107184128.0,
+ "12035": 28107184128.0,
+ "12040": 28107184128.0,
+ "12045": 28107184128.0,
+ "12050": 28107184128.0,
+ "12055": 28107184128.0,
+ "12060": 28107184128.0,
+ "12065": 28107184128.0,
+ "12070": 28107184128.0,
+ "12075": 28107184128.0,
+ "12080": 28107184128.0,
+ "12085": 28107184128.0,
+ "12090": 28107184128.0,
+ "12095": 28107184128.0,
+ "12100": 28107184128.0,
+ "12105": 28107184128.0,
+ "12110": 28107184128.0,
+ "12115": 28107184128.0,
+ "12120": 28107184128.0,
+ "12125": 28107184128.0,
+ "12130": 28107184128.0,
+ "12135": 28107184128.0,
+ "12140": 28107184128.0,
+ "12145": 28107184128.0,
+ "12150": 28107184128.0,
+ "12155": 28107184128.0,
+ "12160": 28107184128.0,
+ "12165": 28107184128.0,
+ "12170": 28107184128.0,
+ "12175": 28107184128.0,
+ "12180": 28107184128.0,
+ "12185": 28107184128.0,
+ "12190": 28107184128.0,
+ "12195": 28107184128.0,
+ "12200": 28107184128.0,
+ "12205": 28107184128.0,
+ "12210": 28107184128.0,
+ "12215": 28107184128.0,
+ "12220": 28107184128.0,
+ "12225": 28107184128.0,
+ "12230": 28107184128.0,
+ "12235": 28107184128.0,
+ "12240": 28107184128.0,
+ "12245": 28107184128.0,
+ "12250": 28107184128.0,
+ "12255": 28107184128.0,
+ "12260": 28107184128.0,
+ "12265": 28107184128.0,
+ "12270": 28107184128.0,
+ "12275": 28107184128.0,
+ "12280": 28107184128.0,
+ "12285": 28107184128.0,
+ "12290": 28107184128.0,
+ "12295": 28107184128.0,
+ "12300": 28107184128.0,
+ "12305": 28107184128.0,
+ "12310": 28107184128.0,
+ "12315": 28107184128.0,
+ "12320": 28107184128.0,
+ "12325": 28107184128.0,
+ "12330": 28107184128.0,
+ "12335": 28107184128.0,
+ "12340": 28107184128.0,
+ "12345": 28107184128.0,
+ "12350": 28107184128.0,
+ "12355": 28107184128.0,
+ "12360": 28107184128.0,
+ "12365": 28107184128.0,
+ "12370": 28107184128.0,
+ "12375": 28107184128.0,
+ "12380": 28107184128.0,
+ "12385": 28107184128.0,
+ "12390": 28107184128.0,
+ "12395": 28107184128.0,
+ "12400": 28107184128.0,
+ "12405": 28107184128.0,
+ "12410": 28107184128.0,
+ "12415": 28107184128.0,
+ "12420": 28107184128.0,
+ "12425": 28107184128.0,
+ "12430": 28107184128.0,
+ "12435": 28107184128.0,
+ "12440": 28107184128.0,
+ "12445": 28107184128.0,
+ "12450": 28107184128.0,
+ "12455": 28107184128.0,
+ "12460": 28107184128.0,
+ "12465": 28107184128.0,
+ "12470": 28107184128.0,
+ "12475": 28107184128.0,
+ "12480": 28107184128.0,
+ "12485": 28107184128.0,
+ "12490": 28107184128.0,
+ "12495": 28107184128.0,
+ "12500": 28107184128.0,
+ "12505": 28107184128.0,
+ "12510": 28107184128.0,
+ "12515": 28107184128.0,
+ "12520": 28107184128.0,
+ "12525": 28107184128.0,
+ "12530": 28107184128.0,
+ "12535": 28107184128.0,
+ "12540": 28107184128.0,
+ "12545": 28107184128.0,
+ "12550": 28107184128.0,
+ "12555": 28107184128.0,
+ "12560": 28107184128.0,
+ "12565": 28107184128.0,
+ "12570": 28107184128.0,
+ "12575": 28107184128.0,
+ "12580": 28107184128.0,
+ "12585": 28107184128.0,
+ "12590": 28107184128.0,
+ "12595": 28107184128.0,
+ "12600": 28107184128.0,
+ "12605": 28107184128.0,
+ "12610": 28107184128.0,
+ "12615": 28107184128.0,
+ "12620": 28107184128.0,
+ "12625": 28107184128.0,
+ "12630": 28107184128.0,
+ "12635": 28107184128.0,
+ "12640": 28107184128.0,
+ "12645": 28107184128.0,
+ "12650": 28107184128.0,
+ "12655": 28107184128.0,
+ "12660": 28107184128.0,
+ "12665": 28107184128.0,
+ "12670": 28107184128.0,
+ "12675": 28107184128.0,
+ "12680": 28107184128.0,
+ "12685": 28107184128.0,
+ "12690": 28107184128.0,
+ "12695": 28107184128.0,
+ "12700": 28107184128.0,
+ "12705": 28107184128.0,
+ "12710": 28107184128.0,
+ "12715": 28107184128.0,
+ "12720": "nan",
+ "12725": "nan",
+ "12730": "nan",
+ "12735": "nan",
+ "12740": "nan",
+ "12745": "nan",
+ "12750": "nan",
+ "12755": "nan",
+ "12760": "nan",
+ "12765": "nan",
+ "12770": "nan",
+ "12775": "nan",
+ "12780": "nan",
+ "12785": "nan",
+ "12790": "nan",
+ "12795": "nan",
+ "12800": "nan",
+ "12805": "nan",
+ "12810": "nan",
+ "12815": "nan",
+ "12820": "nan",
+ "12825": "nan",
+ "12830": "nan",
+ "12835": "nan",
+ "12840": "nan",
+ "12845": "nan",
+ "12850": "nan",
+ "12855": "nan",
+ "12860": "nan",
+ "12865": "nan",
+ "12870": "nan",
+ "12875": "nan",
+ "12880": "nan",
+ "12885": "nan",
+ "12890": "nan",
+ "12895": "nan",
+ "12900": "nan",
+ "12905": "nan",
+ "12910": "nan",
+ "12915": "nan",
+ "12920": "nan",
+ "12925": "nan",
+ "12930": "nan",
+ "12935": "nan",
+ "12940": "nan",
+ "12945": "nan",
+ "12950": "nan",
+ "12955": "nan",
+ "12960": "nan",
+ "12965": "nan",
+ "12970": "nan",
+ "12975": "nan",
+ "12980": "nan",
+ "12985": "nan",
+ "12990": "nan",
+ "12995": "nan",
+ "13000": "nan"
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": "nan",
+ "5": "nan",
+ "10": "nan",
+ "15": "nan",
+ "20": "nan",
+ "25": "nan",
+ "30": "nan",
+ "35": "nan",
+ "40": "nan",
+ "45": "nan",
+ "50": "nan",
+ "55": "nan",
+ "60": "nan",
+ "65": "nan",
+ "70": "nan",
+ "75": "nan",
+ "80": "nan",
+ "85": "nan",
+ "90": "nan",
+ "95": "nan",
+ "100": 1.13737,
+ "105": "nan",
+ "110": "nan",
+ "115": "nan",
+ "120": "nan",
+ "125": "nan",
+ "130": "nan",
+ "135": "nan",
+ "140": "nan",
+ "145": "nan",
+ "150": "nan",
+ "155": "nan",
+ "160": "nan",
+ "165": "nan",
+ "170": "nan",
+ "175": "nan",
+ "180": "nan",
+ "185": "nan",
+ "190": "nan",
+ "195": "nan",
+ "200": 0.92163,
+ "205": "nan",
+ "210": "nan",
+ "215": "nan",
+ "220": "nan",
+ "225": "nan",
+ "230": "nan",
+ "235": "nan",
+ "240": "nan",
+ "245": "nan",
+ "250": "nan",
+ "255": "nan",
+ "260": "nan",
+ "265": "nan",
+ "270": "nan",
+ "275": "nan",
+ "280": "nan",
+ "285": "nan",
+ "290": "nan",
+ "295": "nan",
+ "300": 0.91017,
+ "305": "nan",
+ "310": "nan",
+ "315": "nan",
+ "320": "nan",
+ "325": "nan",
+ "330": "nan",
+ "335": "nan",
+ "340": "nan",
+ "345": "nan",
+ "350": "nan",
+ "355": "nan",
+ "360": "nan",
+ "365": "nan",
+ "370": "nan",
+ "375": "nan",
+ "380": "nan",
+ "385": "nan",
+ "390": "nan",
+ "395": "nan",
+ "400": 0.92356,
+ "405": "nan",
+ "410": "nan",
+ "415": "nan",
+ "420": "nan",
+ "425": "nan",
+ "430": "nan",
+ "435": "nan",
+ "440": "nan",
+ "445": "nan",
+ "450": "nan",
+ "455": "nan",
+ "460": "nan",
+ "465": "nan",
+ "470": "nan",
+ "475": "nan",
+ "480": "nan",
+ "485": "nan",
+ "490": "nan",
+ "495": "nan",
+ "500": 0.92399,
+ "505": "nan",
+ "510": "nan",
+ "515": "nan",
+ "520": "nan",
+ "525": "nan",
+ "530": "nan",
+ "535": "nan",
+ "540": "nan",
+ "545": "nan",
+ "550": "nan",
+ "555": "nan",
+ "560": "nan",
+ "565": "nan",
+ "570": "nan",
+ "575": "nan",
+ "580": "nan",
+ "585": "nan",
+ "590": "nan",
+ "595": "nan",
+ "600": 0.91892,
+ "605": "nan",
+ "610": "nan",
+ "615": "nan",
+ "620": "nan",
+ "625": "nan",
+ "630": "nan",
+ "635": "nan",
+ "640": "nan",
+ "645": "nan",
+ "650": "nan",
+ "655": "nan",
+ "660": "nan",
+ "665": "nan",
+ "670": "nan",
+ "675": "nan",
+ "680": "nan",
+ "685": "nan",
+ "690": "nan",
+ "695": "nan",
+ "700": 0.92118,
+ "705": "nan",
+ "710": "nan",
+ "715": "nan",
+ "720": "nan",
+ "725": "nan",
+ "730": "nan",
+ "735": "nan",
+ "740": "nan",
+ "745": "nan",
+ "750": "nan",
+ "755": "nan",
+ "760": "nan",
+ "765": "nan",
+ "770": "nan",
+ "775": "nan",
+ "780": "nan",
+ "785": "nan",
+ "790": "nan",
+ "795": "nan",
+ "800": 0.91859,
+ "805": "nan",
+ "810": "nan",
+ "815": "nan",
+ "820": "nan",
+ "825": "nan",
+ "830": "nan",
+ "835": "nan",
+ "840": "nan",
+ "845": "nan",
+ "850": "nan",
+ "855": "nan",
+ "860": "nan",
+ "865": "nan",
+ "870": "nan",
+ "875": "nan",
+ "880": "nan",
+ "885": "nan",
+ "890": "nan",
+ "895": "nan",
+ "900": 0.91514,
+ "905": "nan",
+ "910": "nan",
+ "915": "nan",
+ "920": "nan",
+ "925": "nan",
+ "930": "nan",
+ "935": "nan",
+ "940": "nan",
+ "945": "nan",
+ "950": "nan",
+ "955": "nan",
+ "960": "nan",
+ "965": "nan",
+ "970": "nan",
+ "975": "nan",
+ "980": "nan",
+ "985": "nan",
+ "990": "nan",
+ "995": "nan",
+ "1000": 0.92291,
+ "1005": "nan",
+ "1010": "nan",
+ "1015": "nan",
+ "1020": "nan",
+ "1025": "nan",
+ "1030": "nan",
+ "1035": "nan",
+ "1040": "nan",
+ "1045": "nan",
+ "1050": "nan",
+ "1055": "nan",
+ "1060": "nan",
+ "1065": "nan",
+ "1070": "nan",
+ "1075": "nan",
+ "1080": "nan",
+ "1085": "nan",
+ "1090": "nan",
+ "1095": "nan",
+ "1100": 0.92819,
+ "1105": "nan",
+ "1110": "nan",
+ "1115": "nan",
+ "1120": "nan",
+ "1125": "nan",
+ "1130": "nan",
+ "1135": "nan",
+ "1140": "nan",
+ "1145": "nan",
+ "1150": "nan",
+ "1155": "nan",
+ "1160": "nan",
+ "1165": "nan",
+ "1170": "nan",
+ "1175": "nan",
+ "1180": "nan",
+ "1185": "nan",
+ "1190": "nan",
+ "1195": "nan",
+ "1200": 0.9181,
+ "1205": "nan",
+ "1210": "nan",
+ "1215": "nan",
+ "1220": "nan",
+ "1225": "nan",
+ "1230": "nan",
+ "1235": "nan",
+ "1240": "nan",
+ "1245": "nan",
+ "1250": "nan",
+ "1255": "nan",
+ "1260": "nan",
+ "1265": "nan",
+ "1270": "nan",
+ "1275": "nan",
+ "1280": "nan",
+ "1285": "nan",
+ "1290": "nan",
+ "1295": "nan",
+ "1300": 0.98022,
+ "1305": "nan",
+ "1310": "nan",
+ "1315": "nan",
+ "1320": "nan",
+ "1325": "nan",
+ "1330": "nan",
+ "1335": "nan",
+ "1340": "nan",
+ "1345": "nan",
+ "1350": "nan",
+ "1355": "nan",
+ "1360": "nan",
+ "1365": "nan",
+ "1370": "nan",
+ "1375": "nan",
+ "1380": "nan",
+ "1385": "nan",
+ "1390": "nan",
+ "1395": "nan",
+ "1400": 1.01105,
+ "1405": "nan",
+ "1410": "nan",
+ "1415": "nan",
+ "1420": "nan",
+ "1425": "nan",
+ "1430": "nan",
+ "1435": "nan",
+ "1440": "nan",
+ "1445": "nan",
+ "1450": "nan",
+ "1455": "nan",
+ "1460": "nan",
+ "1465": "nan",
+ "1470": "nan",
+ "1475": "nan",
+ "1480": "nan",
+ "1485": "nan",
+ "1490": "nan",
+ "1495": "nan",
+ "1500": 0.91346,
+ "1505": "nan",
+ "1510": "nan",
+ "1515": "nan",
+ "1520": "nan",
+ "1525": "nan",
+ "1530": "nan",
+ "1535": "nan",
+ "1540": "nan",
+ "1545": "nan",
+ "1550": "nan",
+ "1555": "nan",
+ "1560": "nan",
+ "1565": "nan",
+ "1570": "nan",
+ "1575": "nan",
+ "1580": "nan",
+ "1585": "nan",
+ "1590": "nan",
+ "1595": "nan",
+ "1600": 0.91787,
+ "1605": "nan",
+ "1610": "nan",
+ "1615": "nan",
+ "1620": "nan",
+ "1625": "nan",
+ "1630": "nan",
+ "1635": "nan",
+ "1640": "nan",
+ "1645": "nan",
+ "1650": "nan",
+ "1655": "nan",
+ "1660": "nan",
+ "1665": "nan",
+ "1670": "nan",
+ "1675": "nan",
+ "1680": "nan",
+ "1685": "nan",
+ "1690": "nan",
+ "1695": "nan",
+ "1700": 0.91742,
+ "1705": "nan",
+ "1710": "nan",
+ "1715": "nan",
+ "1720": "nan",
+ "1725": "nan",
+ "1730": "nan",
+ "1735": "nan",
+ "1740": "nan",
+ "1745": "nan",
+ "1750": "nan",
+ "1755": "nan",
+ "1760": "nan",
+ "1765": "nan",
+ "1770": "nan",
+ "1775": "nan",
+ "1780": "nan",
+ "1785": "nan",
+ "1790": "nan",
+ "1795": "nan",
+ "1800": 1.00231,
+ "1805": "nan",
+ "1810": "nan",
+ "1815": "nan",
+ "1820": "nan",
+ "1825": "nan",
+ "1830": "nan",
+ "1835": "nan",
+ "1840": "nan",
+ "1845": "nan",
+ "1850": "nan",
+ "1855": "nan",
+ "1860": "nan",
+ "1865": "nan",
+ "1870": "nan",
+ "1875": "nan",
+ "1880": "nan",
+ "1885": "nan",
+ "1890": "nan",
+ "1895": "nan",
+ "1900": 0.91484,
+ "1905": "nan",
+ "1910": "nan",
+ "1915": "nan",
+ "1920": "nan",
+ "1925": "nan",
+ "1930": "nan",
+ "1935": "nan",
+ "1940": "nan",
+ "1945": "nan",
+ "1950": "nan",
+ "1955": "nan",
+ "1960": "nan",
+ "1965": "nan",
+ "1970": "nan",
+ "1975": "nan",
+ "1980": "nan",
+ "1985": "nan",
+ "1990": "nan",
+ "1995": "nan",
+ "2000": 0.92229,
+ "2005": "nan",
+ "2010": "nan",
+ "2015": "nan",
+ "2020": "nan",
+ "2025": "nan",
+ "2030": "nan",
+ "2035": "nan",
+ "2040": "nan",
+ "2045": "nan",
+ "2050": "nan",
+ "2055": "nan",
+ "2060": "nan",
+ "2065": "nan",
+ "2070": "nan",
+ "2075": "nan",
+ "2080": "nan",
+ "2085": "nan",
+ "2090": "nan",
+ "2095": "nan",
+ "2100": 0.91845,
+ "2105": "nan",
+ "2110": "nan",
+ "2115": "nan",
+ "2120": "nan",
+ "2125": "nan",
+ "2130": "nan",
+ "2135": "nan",
+ "2140": "nan",
+ "2145": "nan",
+ "2150": "nan",
+ "2155": "nan",
+ "2160": "nan",
+ "2165": "nan",
+ "2170": "nan",
+ "2175": "nan",
+ "2180": "nan",
+ "2185": "nan",
+ "2190": "nan",
+ "2195": "nan",
+ "2200": 0.96226,
+ "2205": "nan",
+ "2210": "nan",
+ "2215": "nan",
+ "2220": "nan",
+ "2225": "nan",
+ "2230": "nan",
+ "2235": "nan",
+ "2240": "nan",
+ "2245": "nan",
+ "2250": "nan",
+ "2255": "nan",
+ "2260": "nan",
+ "2265": "nan",
+ "2270": "nan",
+ "2275": "nan",
+ "2280": "nan",
+ "2285": "nan",
+ "2290": "nan",
+ "2295": "nan",
+ "2300": 1.01963,
+ "2305": "nan",
+ "2310": "nan",
+ "2315": "nan",
+ "2320": "nan",
+ "2325": "nan",
+ "2330": "nan",
+ "2335": "nan",
+ "2340": "nan",
+ "2345": "nan",
+ "2350": "nan",
+ "2355": "nan",
+ "2360": "nan",
+ "2365": "nan",
+ "2370": "nan",
+ "2375": "nan",
+ "2380": "nan",
+ "2385": "nan",
+ "2390": "nan",
+ "2395": "nan",
+ "2400": 0.90953,
+ "2405": "nan",
+ "2410": "nan",
+ "2415": "nan",
+ "2420": "nan",
+ "2425": "nan",
+ "2430": "nan",
+ "2435": "nan",
+ "2440": "nan",
+ "2445": "nan",
+ "2450": "nan",
+ "2455": "nan",
+ "2460": "nan",
+ "2465": "nan",
+ "2470": "nan",
+ "2475": "nan",
+ "2480": "nan",
+ "2485": "nan",
+ "2490": "nan",
+ "2495": "nan",
+ "2500": 0.91481,
+ "2505": "nan",
+ "2510": "nan",
+ "2515": "nan",
+ "2520": "nan",
+ "2525": "nan",
+ "2530": "nan",
+ "2535": "nan",
+ "2540": "nan",
+ "2545": "nan",
+ "2550": "nan",
+ "2555": "nan",
+ "2560": "nan",
+ "2565": "nan",
+ "2570": "nan",
+ "2575": "nan",
+ "2580": "nan",
+ "2585": "nan",
+ "2590": "nan",
+ "2595": "nan",
+ "2600": 0.91625,
+ "2605": "nan",
+ "2610": "nan",
+ "2615": "nan",
+ "2620": "nan",
+ "2625": "nan",
+ "2630": "nan",
+ "2635": "nan",
+ "2640": "nan",
+ "2645": "nan",
+ "2650": "nan",
+ "2655": "nan",
+ "2660": "nan",
+ "2665": "nan",
+ "2670": "nan",
+ "2675": "nan",
+ "2680": "nan",
+ "2685": "nan",
+ "2690": "nan",
+ "2695": "nan",
+ "2700": 1.01096,
+ "2705": "nan",
+ "2710": "nan",
+ "2715": "nan",
+ "2720": "nan",
+ "2725": "nan",
+ "2730": "nan",
+ "2735": "nan",
+ "2740": "nan",
+ "2745": "nan",
+ "2750": "nan",
+ "2755": "nan",
+ "2760": "nan",
+ "2765": "nan",
+ "2770": "nan",
+ "2775": "nan",
+ "2780": "nan",
+ "2785": "nan",
+ "2790": "nan",
+ "2795": "nan",
+ "2800": 0.89726,
+ "2805": "nan",
+ "2810": "nan",
+ "2815": "nan",
+ "2820": "nan",
+ "2825": "nan",
+ "2830": "nan",
+ "2835": "nan",
+ "2840": "nan",
+ "2845": "nan",
+ "2850": "nan",
+ "2855": "nan",
+ "2860": "nan",
+ "2865": "nan",
+ "2870": "nan",
+ "2875": "nan",
+ "2880": "nan",
+ "2885": "nan",
+ "2890": "nan",
+ "2895": "nan",
+ "2900": 0.89875,
+ "2905": "nan",
+ "2910": "nan",
+ "2915": "nan",
+ "2920": "nan",
+ "2925": "nan",
+ "2930": "nan",
+ "2935": "nan",
+ "2940": "nan",
+ "2945": "nan",
+ "2950": "nan",
+ "2955": "nan",
+ "2960": "nan",
+ "2965": "nan",
+ "2970": "nan",
+ "2975": "nan",
+ "2980": "nan",
+ "2985": "nan",
+ "2990": "nan",
+ "2995": "nan",
+ "3000": 0.90963,
+ "3005": "nan",
+ "3010": "nan",
+ "3015": "nan",
+ "3020": "nan",
+ "3025": "nan",
+ "3030": "nan",
+ "3035": "nan",
+ "3040": "nan",
+ "3045": "nan",
+ "3050": "nan",
+ "3055": "nan",
+ "3060": "nan",
+ "3065": "nan",
+ "3070": "nan",
+ "3075": "nan",
+ "3080": "nan",
+ "3085": "nan",
+ "3090": "nan",
+ "3095": "nan",
+ "3100": 0.96015,
+ "3105": "nan",
+ "3110": "nan",
+ "3115": "nan",
+ "3120": "nan",
+ "3125": "nan",
+ "3130": "nan",
+ "3135": "nan",
+ "3140": "nan",
+ "3145": "nan",
+ "3150": "nan",
+ "3155": "nan",
+ "3160": "nan",
+ "3165": "nan",
+ "3170": "nan",
+ "3175": "nan",
+ "3180": "nan",
+ "3185": "nan",
+ "3190": "nan",
+ "3195": "nan",
+ "3200": 1.01137,
+ "3205": "nan",
+ "3210": "nan",
+ "3215": "nan",
+ "3220": "nan",
+ "3225": "nan",
+ "3230": "nan",
+ "3235": "nan",
+ "3240": "nan",
+ "3245": "nan",
+ "3250": "nan",
+ "3255": "nan",
+ "3260": "nan",
+ "3265": "nan",
+ "3270": "nan",
+ "3275": "nan",
+ "3280": "nan",
+ "3285": "nan",
+ "3290": "nan",
+ "3295": "nan",
+ "3300": 0.91337,
+ "3305": "nan",
+ "3310": "nan",
+ "3315": "nan",
+ "3320": "nan",
+ "3325": "nan",
+ "3330": "nan",
+ "3335": "nan",
+ "3340": "nan",
+ "3345": "nan",
+ "3350": "nan",
+ "3355": "nan",
+ "3360": "nan",
+ "3365": "nan",
+ "3370": "nan",
+ "3375": "nan",
+ "3380": "nan",
+ "3385": "nan",
+ "3390": "nan",
+ "3395": "nan",
+ "3400": 0.91177,
+ "3405": "nan",
+ "3410": "nan",
+ "3415": "nan",
+ "3420": "nan",
+ "3425": "nan",
+ "3430": "nan",
+ "3435": "nan",
+ "3440": "nan",
+ "3445": "nan",
+ "3450": "nan",
+ "3455": "nan",
+ "3460": "nan",
+ "3465": "nan",
+ "3470": "nan",
+ "3475": "nan",
+ "3480": "nan",
+ "3485": "nan",
+ "3490": "nan",
+ "3495": "nan",
+ "3500": 0.91124,
+ "3505": "nan",
+ "3510": "nan",
+ "3515": "nan",
+ "3520": "nan",
+ "3525": "nan",
+ "3530": "nan",
+ "3535": "nan",
+ "3540": "nan",
+ "3545": "nan",
+ "3550": "nan",
+ "3555": "nan",
+ "3560": "nan",
+ "3565": "nan",
+ "3570": "nan",
+ "3575": "nan",
+ "3580": "nan",
+ "3585": "nan",
+ "3590": "nan",
+ "3595": "nan",
+ "3600": 0.91795,
+ "3605": "nan",
+ "3610": "nan",
+ "3615": "nan",
+ "3620": "nan",
+ "3625": "nan",
+ "3630": "nan",
+ "3635": "nan",
+ "3640": "nan",
+ "3645": "nan",
+ "3650": "nan",
+ "3655": "nan",
+ "3660": "nan",
+ "3665": "nan",
+ "3670": "nan",
+ "3675": "nan",
+ "3680": "nan",
+ "3685": "nan",
+ "3690": "nan",
+ "3695": "nan",
+ "3700": 0.99608,
+ "3705": "nan",
+ "3710": "nan",
+ "3715": "nan",
+ "3720": "nan",
+ "3725": "nan",
+ "3730": "nan",
+ "3735": "nan",
+ "3740": "nan",
+ "3745": "nan",
+ "3750": "nan",
+ "3755": "nan",
+ "3760": "nan",
+ "3765": "nan",
+ "3770": "nan",
+ "3775": "nan",
+ "3780": "nan",
+ "3785": "nan",
+ "3790": "nan",
+ "3795": "nan",
+ "3800": 0.89009,
+ "3805": "nan",
+ "3810": "nan",
+ "3815": "nan",
+ "3820": "nan",
+ "3825": "nan",
+ "3830": "nan",
+ "3835": "nan",
+ "3840": "nan",
+ "3845": "nan",
+ "3850": "nan",
+ "3855": "nan",
+ "3860": "nan",
+ "3865": "nan",
+ "3870": "nan",
+ "3875": "nan",
+ "3880": "nan",
+ "3885": "nan",
+ "3890": "nan",
+ "3895": "nan",
+ "3900": 0.88671,
+ "3905": "nan",
+ "3910": "nan",
+ "3915": "nan",
+ "3920": "nan",
+ "3925": "nan",
+ "3930": "nan",
+ "3935": "nan",
+ "3940": "nan",
+ "3945": "nan",
+ "3950": "nan",
+ "3955": "nan",
+ "3960": "nan",
+ "3965": "nan",
+ "3970": "nan",
+ "3975": "nan",
+ "3980": "nan",
+ "3985": "nan",
+ "3990": "nan",
+ "3995": "nan",
+ "4000": 0.96428,
+ "4005": "nan",
+ "4010": "nan",
+ "4015": "nan",
+ "4020": "nan",
+ "4025": "nan",
+ "4030": "nan",
+ "4035": "nan",
+ "4040": "nan",
+ "4045": "nan",
+ "4050": "nan",
+ "4055": "nan",
+ "4060": "nan",
+ "4065": "nan",
+ "4070": "nan",
+ "4075": "nan",
+ "4080": "nan",
+ "4085": "nan",
+ "4090": "nan",
+ "4095": "nan",
+ "4100": 1.01022,
+ "4105": "nan",
+ "4110": "nan",
+ "4115": "nan",
+ "4120": "nan",
+ "4125": "nan",
+ "4130": "nan",
+ "4135": "nan",
+ "4140": "nan",
+ "4145": "nan",
+ "4150": "nan",
+ "4155": "nan",
+ "4160": "nan",
+ "4165": "nan",
+ "4170": "nan",
+ "4175": "nan",
+ "4180": "nan",
+ "4185": "nan",
+ "4190": "nan",
+ "4195": "nan",
+ "4200": 0.91862,
+ "4205": "nan",
+ "4210": "nan",
+ "4215": "nan",
+ "4220": "nan",
+ "4225": "nan",
+ "4230": "nan",
+ "4235": "nan",
+ "4240": "nan",
+ "4245": "nan",
+ "4250": "nan",
+ "4255": "nan",
+ "4260": "nan",
+ "4265": "nan",
+ "4270": "nan",
+ "4275": "nan",
+ "4280": "nan",
+ "4285": "nan",
+ "4290": "nan",
+ "4295": "nan",
+ "4300": 0.91817,
+ "4305": "nan",
+ "4310": "nan",
+ "4315": "nan",
+ "4320": "nan",
+ "4325": "nan",
+ "4330": "nan",
+ "4335": "nan",
+ "4340": "nan",
+ "4345": "nan",
+ "4350": "nan",
+ "4355": "nan",
+ "4360": "nan",
+ "4365": "nan",
+ "4370": "nan",
+ "4375": "nan",
+ "4380": "nan",
+ "4385": "nan",
+ "4390": "nan",
+ "4395": "nan",
+ "4400": 0.91961,
+ "4405": "nan",
+ "4410": "nan",
+ "4415": "nan",
+ "4420": "nan",
+ "4425": "nan",
+ "4430": "nan",
+ "4435": "nan",
+ "4440": "nan",
+ "4445": "nan",
+ "4450": "nan",
+ "4455": "nan",
+ "4460": "nan",
+ "4465": "nan",
+ "4470": "nan",
+ "4475": "nan",
+ "4480": "nan",
+ "4485": "nan",
+ "4490": "nan",
+ "4495": "nan",
+ "4500": 0.91494,
+ "4505": "nan",
+ "4510": "nan",
+ "4515": "nan",
+ "4520": "nan",
+ "4525": "nan",
+ "4530": "nan",
+ "4535": "nan",
+ "4540": "nan",
+ "4545": "nan",
+ "4550": "nan",
+ "4555": "nan",
+ "4560": "nan",
+ "4565": "nan",
+ "4570": "nan",
+ "4575": "nan",
+ "4580": "nan",
+ "4585": "nan",
+ "4590": "nan",
+ "4595": "nan",
+ "4600": 0.97166,
+ "4605": "nan",
+ "4610": "nan",
+ "4615": "nan",
+ "4620": "nan",
+ "4625": "nan",
+ "4630": "nan",
+ "4635": "nan",
+ "4640": "nan",
+ "4645": "nan",
+ "4650": "nan",
+ "4655": "nan",
+ "4660": "nan",
+ "4665": "nan",
+ "4670": "nan",
+ "4675": "nan",
+ "4680": "nan",
+ "4685": "nan",
+ "4690": "nan",
+ "4695": "nan",
+ "4700": 0.89156,
+ "4705": "nan",
+ "4710": "nan",
+ "4715": "nan",
+ "4720": "nan",
+ "4725": "nan",
+ "4730": "nan",
+ "4735": "nan",
+ "4740": "nan",
+ "4745": "nan",
+ "4750": "nan",
+ "4755": "nan",
+ "4760": "nan",
+ "4765": "nan",
+ "4770": "nan",
+ "4775": "nan",
+ "4780": "nan",
+ "4785": "nan",
+ "4790": "nan",
+ "4795": "nan",
+ "4800": 0.91581,
+ "4805": "nan",
+ "4810": "nan",
+ "4815": "nan",
+ "4820": "nan",
+ "4825": "nan",
+ "4830": "nan",
+ "4835": "nan",
+ "4840": "nan",
+ "4845": "nan",
+ "4850": "nan",
+ "4855": "nan",
+ "4860": "nan",
+ "4865": "nan",
+ "4870": "nan",
+ "4875": "nan",
+ "4880": "nan",
+ "4885": "nan",
+ "4890": "nan",
+ "4895": "nan",
+ "4900": 0.98462,
+ "4905": "nan",
+ "4910": "nan",
+ "4915": "nan",
+ "4920": "nan",
+ "4925": "nan",
+ "4930": "nan",
+ "4935": "nan",
+ "4940": "nan",
+ "4945": "nan",
+ "4950": "nan",
+ "4955": "nan",
+ "4960": "nan",
+ "4965": "nan",
+ "4970": "nan",
+ "4975": "nan",
+ "4980": "nan",
+ "4985": "nan",
+ "4990": "nan",
+ "4995": "nan",
+ "5000": 1.0238,
+ "5005": "nan",
+ "5010": "nan",
+ "5015": "nan",
+ "5020": "nan",
+ "5025": "nan",
+ "5030": "nan",
+ "5035": "nan",
+ "5040": "nan",
+ "5045": "nan",
+ "5050": "nan",
+ "5055": "nan",
+ "5060": "nan",
+ "5065": "nan",
+ "5070": "nan",
+ "5075": "nan",
+ "5080": "nan",
+ "5085": "nan",
+ "5090": "nan",
+ "5095": "nan",
+ "5100": 0.91893,
+ "5105": "nan",
+ "5110": "nan",
+ "5115": "nan",
+ "5120": "nan",
+ "5125": "nan",
+ "5130": "nan",
+ "5135": "nan",
+ "5140": "nan",
+ "5145": "nan",
+ "5150": "nan",
+ "5155": "nan",
+ "5160": "nan",
+ "5165": "nan",
+ "5170": "nan",
+ "5175": "nan",
+ "5180": "nan",
+ "5185": "nan",
+ "5190": "nan",
+ "5195": "nan",
+ "5200": 0.9139,
+ "5205": "nan",
+ "5210": "nan",
+ "5215": "nan",
+ "5220": "nan",
+ "5225": "nan",
+ "5230": "nan",
+ "5235": "nan",
+ "5240": "nan",
+ "5245": "nan",
+ "5250": "nan",
+ "5255": "nan",
+ "5260": "nan",
+ "5265": "nan",
+ "5270": "nan",
+ "5275": "nan",
+ "5280": "nan",
+ "5285": "nan",
+ "5290": "nan",
+ "5295": "nan",
+ "5300": 0.9158,
+ "5305": "nan",
+ "5310": "nan",
+ "5315": "nan",
+ "5320": "nan",
+ "5325": "nan",
+ "5330": "nan",
+ "5335": "nan",
+ "5340": "nan",
+ "5345": "nan",
+ "5350": "nan",
+ "5355": "nan",
+ "5360": "nan",
+ "5365": "nan",
+ "5370": "nan",
+ "5375": "nan",
+ "5380": "nan",
+ "5385": "nan",
+ "5390": "nan",
+ "5395": "nan",
+ "5400": 0.91724,
+ "5405": "nan",
+ "5410": "nan",
+ "5415": "nan",
+ "5420": "nan",
+ "5425": "nan",
+ "5430": "nan",
+ "5435": "nan",
+ "5440": "nan",
+ "5445": "nan",
+ "5450": "nan",
+ "5455": "nan",
+ "5460": "nan",
+ "5465": "nan",
+ "5470": "nan",
+ "5475": "nan",
+ "5480": "nan",
+ "5485": "nan",
+ "5490": "nan",
+ "5495": "nan",
+ "5500": 1.00127,
+ "5505": "nan",
+ "5510": "nan",
+ "5515": "nan",
+ "5520": "nan",
+ "5525": "nan",
+ "5530": "nan",
+ "5535": "nan",
+ "5540": "nan",
+ "5545": "nan",
+ "5550": "nan",
+ "5555": "nan",
+ "5560": "nan",
+ "5565": "nan",
+ "5570": "nan",
+ "5575": "nan",
+ "5580": "nan",
+ "5585": "nan",
+ "5590": "nan",
+ "5595": "nan",
+ "5600": 0.91763,
+ "5605": "nan",
+ "5610": "nan",
+ "5615": "nan",
+ "5620": "nan",
+ "5625": "nan",
+ "5630": "nan",
+ "5635": "nan",
+ "5640": "nan",
+ "5645": "nan",
+ "5650": "nan",
+ "5655": "nan",
+ "5660": "nan",
+ "5665": "nan",
+ "5670": "nan",
+ "5675": "nan",
+ "5680": "nan",
+ "5685": "nan",
+ "5690": "nan",
+ "5695": "nan",
+ "5700": 0.91302,
+ "5705": "nan",
+ "5710": "nan",
+ "5715": "nan",
+ "5720": "nan",
+ "5725": "nan",
+ "5730": "nan",
+ "5735": "nan",
+ "5740": "nan",
+ "5745": "nan",
+ "5750": "nan",
+ "5755": "nan",
+ "5760": "nan",
+ "5765": "nan",
+ "5770": "nan",
+ "5775": "nan",
+ "5780": "nan",
+ "5785": "nan",
+ "5790": "nan",
+ "5795": "nan",
+ "5800": 0.9736,
+ "5805": "nan",
+ "5810": "nan",
+ "5815": "nan",
+ "5820": "nan",
+ "5825": "nan",
+ "5830": "nan",
+ "5835": "nan",
+ "5840": "nan",
+ "5845": "nan",
+ "5850": "nan",
+ "5855": "nan",
+ "5860": "nan",
+ "5865": "nan",
+ "5870": "nan",
+ "5875": "nan",
+ "5880": "nan",
+ "5885": "nan",
+ "5890": "nan",
+ "5895": "nan",
+ "5900": 1.0057,
+ "5905": "nan",
+ "5910": "nan",
+ "5915": "nan",
+ "5920": "nan",
+ "5925": "nan",
+ "5930": "nan",
+ "5935": "nan",
+ "5940": "nan",
+ "5945": "nan",
+ "5950": "nan",
+ "5955": "nan",
+ "5960": "nan",
+ "5965": "nan",
+ "5970": "nan",
+ "5975": "nan",
+ "5980": "nan",
+ "5985": "nan",
+ "5990": "nan",
+ "5995": "nan",
+ "6000": 0.9012,
+ "6005": "nan",
+ "6010": "nan",
+ "6015": "nan",
+ "6020": "nan",
+ "6025": "nan",
+ "6030": "nan",
+ "6035": "nan",
+ "6040": "nan",
+ "6045": "nan",
+ "6050": "nan",
+ "6055": "nan",
+ "6060": "nan",
+ "6065": "nan",
+ "6070": "nan",
+ "6075": "nan",
+ "6080": "nan",
+ "6085": "nan",
+ "6090": "nan",
+ "6095": "nan",
+ "6100": 0.89269,
+ "6105": "nan",
+ "6110": "nan",
+ "6115": "nan",
+ "6120": "nan",
+ "6125": "nan",
+ "6130": "nan",
+ "6135": "nan",
+ "6140": "nan",
+ "6145": "nan",
+ "6150": "nan",
+ "6155": "nan",
+ "6160": "nan",
+ "6165": "nan",
+ "6170": "nan",
+ "6175": "nan",
+ "6180": "nan",
+ "6185": "nan",
+ "6190": "nan",
+ "6195": "nan",
+ "6200": 0.89637,
+ "6205": "nan",
+ "6210": "nan",
+ "6215": "nan",
+ "6220": "nan",
+ "6225": "nan",
+ "6230": "nan",
+ "6235": "nan",
+ "6240": "nan",
+ "6245": "nan",
+ "6250": "nan",
+ "6255": "nan",
+ "6260": "nan",
+ "6265": "nan",
+ "6270": "nan",
+ "6275": "nan",
+ "6280": "nan",
+ "6285": "nan",
+ "6290": "nan",
+ "6295": "nan",
+ "6300": 1.01218,
+ "6305": "nan",
+ "6310": "nan",
+ "6315": "nan",
+ "6320": "nan",
+ "6325": "nan",
+ "6330": "nan",
+ "6335": "nan",
+ "6340": "nan",
+ "6345": "nan",
+ "6350": "nan",
+ "6355": "nan",
+ "6360": "nan",
+ "6365": "nan",
+ "6370": "nan",
+ "6375": "nan",
+ "6380": "nan",
+ "6385": "nan",
+ "6390": "nan",
+ "6395": "nan",
+ "6400": 0.92171,
+ "6405": "nan",
+ "6410": "nan",
+ "6415": "nan",
+ "6420": "nan",
+ "6425": "nan",
+ "6430": "nan",
+ "6435": "nan",
+ "6440": "nan",
+ "6445": "nan",
+ "6450": "nan",
+ "6455": "nan",
+ "6460": "nan",
+ "6465": "nan",
+ "6470": "nan",
+ "6475": "nan",
+ "6480": "nan",
+ "6485": "nan",
+ "6490": "nan",
+ "6495": "nan",
+ "6500": 0.9199,
+ "6505": "nan",
+ "6510": "nan",
+ "6515": "nan",
+ "6520": "nan",
+ "6525": "nan",
+ "6530": "nan",
+ "6535": "nan",
+ "6540": "nan",
+ "6545": "nan",
+ "6550": "nan",
+ "6555": "nan",
+ "6560": "nan",
+ "6565": "nan",
+ "6570": "nan",
+ "6575": "nan",
+ "6580": "nan",
+ "6585": "nan",
+ "6590": "nan",
+ "6595": "nan",
+ "6600": 0.88888,
+ "6605": "nan",
+ "6610": "nan",
+ "6615": "nan",
+ "6620": "nan",
+ "6625": "nan",
+ "6630": "nan",
+ "6635": "nan",
+ "6640": "nan",
+ "6645": "nan",
+ "6650": "nan",
+ "6655": "nan",
+ "6660": "nan",
+ "6665": "nan",
+ "6670": "nan",
+ "6675": "nan",
+ "6680": "nan",
+ "6685": "nan",
+ "6690": "nan",
+ "6695": "nan",
+ "6700": 0.95756,
+ "6705": "nan",
+ "6710": "nan",
+ "6715": "nan",
+ "6720": "nan",
+ "6725": "nan",
+ "6730": "nan",
+ "6735": "nan",
+ "6740": "nan",
+ "6745": "nan",
+ "6750": "nan",
+ "6755": "nan",
+ "6760": "nan",
+ "6765": "nan",
+ "6770": "nan",
+ "6775": "nan",
+ "6780": "nan",
+ "6785": "nan",
+ "6790": "nan",
+ "6795": "nan",
+ "6800": 0.98858,
+ "6805": "nan",
+ "6810": "nan",
+ "6815": "nan",
+ "6820": "nan",
+ "6825": "nan",
+ "6830": "nan",
+ "6835": "nan",
+ "6840": "nan",
+ "6845": "nan",
+ "6850": "nan",
+ "6855": "nan",
+ "6860": "nan",
+ "6865": "nan",
+ "6870": "nan",
+ "6875": "nan",
+ "6880": "nan",
+ "6885": "nan",
+ "6890": "nan",
+ "6895": "nan",
+ "6900": 0.92019,
+ "6905": "nan",
+ "6910": "nan",
+ "6915": "nan",
+ "6920": "nan",
+ "6925": "nan",
+ "6930": "nan",
+ "6935": "nan",
+ "6940": "nan",
+ "6945": "nan",
+ "6950": "nan",
+ "6955": "nan",
+ "6960": "nan",
+ "6965": "nan",
+ "6970": "nan",
+ "6975": "nan",
+ "6980": "nan",
+ "6985": "nan",
+ "6990": "nan",
+ "6995": "nan",
+ "7000": 0.92142,
+ "7005": "nan",
+ "7010": "nan",
+ "7015": "nan",
+ "7020": "nan",
+ "7025": "nan",
+ "7030": "nan",
+ "7035": "nan",
+ "7040": "nan",
+ "7045": "nan",
+ "7050": "nan",
+ "7055": "nan",
+ "7060": "nan",
+ "7065": "nan",
+ "7070": "nan",
+ "7075": "nan",
+ "7080": "nan",
+ "7085": "nan",
+ "7090": "nan",
+ "7095": "nan",
+ "7100": 0.91628,
+ "7105": "nan",
+ "7110": "nan",
+ "7115": "nan",
+ "7120": "nan",
+ "7125": "nan",
+ "7130": "nan",
+ "7135": "nan",
+ "7140": "nan",
+ "7145": "nan",
+ "7150": "nan",
+ "7155": "nan",
+ "7160": "nan",
+ "7165": "nan",
+ "7170": "nan",
+ "7175": "nan",
+ "7180": "nan",
+ "7185": "nan",
+ "7190": "nan",
+ "7195": "nan",
+ "7200": 1.02044,
+ "7205": "nan",
+ "7210": "nan",
+ "7215": "nan",
+ "7220": "nan",
+ "7225": "nan",
+ "7230": "nan",
+ "7235": "nan",
+ "7240": "nan",
+ "7245": "nan",
+ "7250": "nan",
+ "7255": "nan",
+ "7260": "nan",
+ "7265": "nan",
+ "7270": "nan",
+ "7275": "nan",
+ "7280": "nan",
+ "7285": "nan",
+ "7290": "nan",
+ "7295": "nan",
+ "7300": 0.92189,
+ "7305": "nan",
+ "7310": "nan",
+ "7315": "nan",
+ "7320": "nan",
+ "7325": "nan",
+ "7330": "nan",
+ "7335": "nan",
+ "7340": "nan",
+ "7345": "nan",
+ "7350": "nan",
+ "7355": "nan",
+ "7360": "nan",
+ "7365": "nan",
+ "7370": "nan",
+ "7375": "nan",
+ "7380": "nan",
+ "7385": "nan",
+ "7390": "nan",
+ "7395": "nan",
+ "7400": 0.92339,
+ "7405": "nan",
+ "7410": "nan",
+ "7415": "nan",
+ "7420": "nan",
+ "7425": "nan",
+ "7430": "nan",
+ "7435": "nan",
+ "7440": "nan",
+ "7445": "nan",
+ "7450": "nan",
+ "7455": "nan",
+ "7460": "nan",
+ "7465": "nan",
+ "7470": "nan",
+ "7475": "nan",
+ "7480": "nan",
+ "7485": "nan",
+ "7490": "nan",
+ "7495": "nan",
+ "7500": 0.92441,
+ "7505": "nan",
+ "7510": "nan",
+ "7515": "nan",
+ "7520": "nan",
+ "7525": "nan",
+ "7530": "nan",
+ "7535": "nan",
+ "7540": "nan",
+ "7545": "nan",
+ "7550": "nan",
+ "7555": "nan",
+ "7560": "nan",
+ "7565": "nan",
+ "7570": "nan",
+ "7575": "nan",
+ "7580": "nan",
+ "7585": "nan",
+ "7590": "nan",
+ "7595": "nan",
+ "7600": 0.98985,
+ "7605": "nan",
+ "7610": "nan",
+ "7615": "nan",
+ "7620": "nan",
+ "7625": "nan",
+ "7630": "nan",
+ "7635": "nan",
+ "7640": "nan",
+ "7645": "nan",
+ "7650": "nan",
+ "7655": "nan",
+ "7660": "nan",
+ "7665": "nan",
+ "7670": "nan",
+ "7675": "nan",
+ "7680": "nan",
+ "7685": "nan",
+ "7690": "nan",
+ "7695": "nan",
+ "7700": 0.9984,
+ "7705": "nan",
+ "7710": "nan",
+ "7715": "nan",
+ "7720": "nan",
+ "7725": "nan",
+ "7730": "nan",
+ "7735": "nan",
+ "7740": "nan",
+ "7745": "nan",
+ "7750": "nan",
+ "7755": "nan",
+ "7760": "nan",
+ "7765": "nan",
+ "7770": "nan",
+ "7775": "nan",
+ "7780": "nan",
+ "7785": "nan",
+ "7790": "nan",
+ "7795": "nan",
+ "7800": 0.92652,
+ "7805": "nan",
+ "7810": "nan",
+ "7815": "nan",
+ "7820": "nan",
+ "7825": "nan",
+ "7830": "nan",
+ "7835": "nan",
+ "7840": "nan",
+ "7845": "nan",
+ "7850": "nan",
+ "7855": "nan",
+ "7860": "nan",
+ "7865": "nan",
+ "7870": "nan",
+ "7875": "nan",
+ "7880": "nan",
+ "7885": "nan",
+ "7890": "nan",
+ "7895": "nan",
+ "7900": 0.92853,
+ "7905": "nan",
+ "7910": "nan",
+ "7915": "nan",
+ "7920": "nan",
+ "7925": "nan",
+ "7930": "nan",
+ "7935": "nan",
+ "7940": "nan",
+ "7945": "nan",
+ "7950": "nan",
+ "7955": "nan",
+ "7960": "nan",
+ "7965": "nan",
+ "7970": "nan",
+ "7975": "nan",
+ "7980": "nan",
+ "7985": "nan",
+ "7990": "nan",
+ "7995": "nan",
+ "8000": 0.92936,
+ "8005": "nan",
+ "8010": "nan",
+ "8015": "nan",
+ "8020": "nan",
+ "8025": "nan",
+ "8030": "nan",
+ "8035": "nan",
+ "8040": "nan",
+ "8045": "nan",
+ "8050": "nan",
+ "8055": "nan",
+ "8060": "nan",
+ "8065": "nan",
+ "8070": "nan",
+ "8075": "nan",
+ "8080": "nan",
+ "8085": "nan",
+ "8090": "nan",
+ "8095": "nan",
+ "8100": 1.02351,
+ "8105": "nan",
+ "8110": "nan",
+ "8115": "nan",
+ "8120": "nan",
+ "8125": "nan",
+ "8130": "nan",
+ "8135": "nan",
+ "8140": "nan",
+ "8145": "nan",
+ "8150": "nan",
+ "8155": "nan",
+ "8160": "nan",
+ "8165": "nan",
+ "8170": "nan",
+ "8175": "nan",
+ "8180": "nan",
+ "8185": "nan",
+ "8190": "nan",
+ "8195": "nan",
+ "8200": 0.92463,
+ "8205": "nan",
+ "8210": "nan",
+ "8215": "nan",
+ "8220": "nan",
+ "8225": "nan",
+ "8230": "nan",
+ "8235": "nan",
+ "8240": "nan",
+ "8245": "nan",
+ "8250": "nan",
+ "8255": "nan",
+ "8260": "nan",
+ "8265": "nan",
+ "8270": "nan",
+ "8275": "nan",
+ "8280": "nan",
+ "8285": "nan",
+ "8290": "nan",
+ "8295": "nan",
+ "8300": 0.92647,
+ "8305": "nan",
+ "8310": "nan",
+ "8315": "nan",
+ "8320": "nan",
+ "8325": "nan",
+ "8330": "nan",
+ "8335": "nan",
+ "8340": "nan",
+ "8345": "nan",
+ "8350": "nan",
+ "8355": "nan",
+ "8360": "nan",
+ "8365": "nan",
+ "8370": "nan",
+ "8375": "nan",
+ "8380": "nan",
+ "8385": "nan",
+ "8390": "nan",
+ "8395": "nan",
+ "8400": 0.96739,
+ "8405": "nan",
+ "8410": "nan",
+ "8415": "nan",
+ "8420": "nan",
+ "8425": "nan",
+ "8430": "nan",
+ "8435": "nan",
+ "8440": "nan",
+ "8445": "nan",
+ "8450": "nan",
+ "8455": "nan",
+ "8460": "nan",
+ "8465": "nan",
+ "8470": "nan",
+ "8475": "nan",
+ "8480": "nan",
+ "8485": "nan",
+ "8490": "nan",
+ "8495": "nan",
+ "8500": 0.99499,
+ "8505": "nan",
+ "8510": "nan",
+ "8515": "nan",
+ "8520": "nan",
+ "8525": "nan",
+ "8530": "nan",
+ "8535": "nan",
+ "8540": "nan",
+ "8545": "nan",
+ "8550": "nan",
+ "8555": "nan",
+ "8560": "nan",
+ "8565": "nan",
+ "8570": "nan",
+ "8575": "nan",
+ "8580": "nan",
+ "8585": "nan",
+ "8590": "nan",
+ "8595": "nan",
+ "8600": 0.90151,
+ "8605": "nan",
+ "8610": "nan",
+ "8615": "nan",
+ "8620": "nan",
+ "8625": "nan",
+ "8630": "nan",
+ "8635": "nan",
+ "8640": "nan",
+ "8645": "nan",
+ "8650": "nan",
+ "8655": "nan",
+ "8660": "nan",
+ "8665": "nan",
+ "8670": "nan",
+ "8675": "nan",
+ "8680": "nan",
+ "8685": "nan",
+ "8690": "nan",
+ "8695": "nan",
+ "8700": 0.91227,
+ "8705": "nan",
+ "8710": "nan",
+ "8715": "nan",
+ "8720": "nan",
+ "8725": "nan",
+ "8730": "nan",
+ "8735": "nan",
+ "8740": "nan",
+ "8745": "nan",
+ "8750": "nan",
+ "8755": "nan",
+ "8760": "nan",
+ "8765": "nan",
+ "8770": "nan",
+ "8775": "nan",
+ "8780": "nan",
+ "8785": "nan",
+ "8790": "nan",
+ "8795": "nan",
+ "8800": 0.9268,
+ "8805": "nan",
+ "8810": "nan",
+ "8815": "nan",
+ "8820": "nan",
+ "8825": "nan",
+ "8830": "nan",
+ "8835": "nan",
+ "8840": "nan",
+ "8845": "nan",
+ "8850": "nan",
+ "8855": "nan",
+ "8860": "nan",
+ "8865": "nan",
+ "8870": "nan",
+ "8875": "nan",
+ "8880": "nan",
+ "8885": "nan",
+ "8890": "nan",
+ "8895": "nan",
+ "8900": 0.92834,
+ "8905": "nan",
+ "8910": "nan",
+ "8915": "nan",
+ "8920": "nan",
+ "8925": "nan",
+ "8930": "nan",
+ "8935": "nan",
+ "8940": "nan",
+ "8945": "nan",
+ "8950": "nan",
+ "8955": "nan",
+ "8960": "nan",
+ "8965": "nan",
+ "8970": "nan",
+ "8975": "nan",
+ "8980": "nan",
+ "8985": "nan",
+ "8990": "nan",
+ "8995": "nan",
+ "9000": 1.00636,
+ "9005": "nan",
+ "9010": "nan",
+ "9015": "nan",
+ "9020": "nan",
+ "9025": "nan",
+ "9030": "nan",
+ "9035": "nan",
+ "9040": "nan",
+ "9045": "nan",
+ "9050": "nan",
+ "9055": "nan",
+ "9060": "nan",
+ "9065": "nan",
+ "9070": "nan",
+ "9075": "nan",
+ "9080": "nan",
+ "9085": "nan",
+ "9090": "nan",
+ "9095": "nan",
+ "9100": 0.92408,
+ "9105": "nan",
+ "9110": "nan",
+ "9115": "nan",
+ "9120": "nan",
+ "9125": "nan",
+ "9130": "nan",
+ "9135": "nan",
+ "9140": "nan",
+ "9145": "nan",
+ "9150": "nan",
+ "9155": "nan",
+ "9160": "nan",
+ "9165": "nan",
+ "9170": "nan",
+ "9175": "nan",
+ "9180": "nan",
+ "9185": "nan",
+ "9190": "nan",
+ "9195": "nan",
+ "9200": 0.8975,
+ "9205": "nan",
+ "9210": "nan",
+ "9215": "nan",
+ "9220": "nan",
+ "9225": "nan",
+ "9230": "nan",
+ "9235": "nan",
+ "9240": "nan",
+ "9245": "nan",
+ "9250": "nan",
+ "9255": "nan",
+ "9260": "nan",
+ "9265": "nan",
+ "9270": "nan",
+ "9275": "nan",
+ "9280": "nan",
+ "9285": "nan",
+ "9290": "nan",
+ "9295": "nan",
+ "9300": 0.96478,
+ "9305": "nan",
+ "9310": "nan",
+ "9315": "nan",
+ "9320": "nan",
+ "9325": "nan",
+ "9330": "nan",
+ "9335": "nan",
+ "9340": "nan",
+ "9345": "nan",
+ "9350": "nan",
+ "9355": "nan",
+ "9360": "nan",
+ "9365": "nan",
+ "9370": "nan",
+ "9375": "nan",
+ "9380": "nan",
+ "9385": "nan",
+ "9390": "nan",
+ "9395": "nan",
+ "9400": 0.98208,
+ "9405": "nan",
+ "9410": "nan",
+ "9415": "nan",
+ "9420": "nan",
+ "9425": "nan",
+ "9430": "nan",
+ "9435": "nan",
+ "9440": "nan",
+ "9445": "nan",
+ "9450": "nan",
+ "9455": "nan",
+ "9460": "nan",
+ "9465": "nan",
+ "9470": "nan",
+ "9475": "nan",
+ "9480": "nan",
+ "9485": "nan",
+ "9490": "nan",
+ "9495": "nan",
+ "9500": 0.92094,
+ "9505": "nan",
+ "9510": "nan",
+ "9515": "nan",
+ "9520": "nan",
+ "9525": "nan",
+ "9530": "nan",
+ "9535": "nan",
+ "9540": "nan",
+ "9545": "nan",
+ "9550": "nan",
+ "9555": "nan",
+ "9560": "nan",
+ "9565": "nan",
+ "9570": "nan",
+ "9575": "nan",
+ "9580": "nan",
+ "9585": "nan",
+ "9590": "nan",
+ "9595": "nan",
+ "9600": 0.91916,
+ "9605": "nan",
+ "9610": "nan",
+ "9615": "nan",
+ "9620": "nan",
+ "9625": "nan",
+ "9630": "nan",
+ "9635": "nan",
+ "9640": "nan",
+ "9645": "nan",
+ "9650": "nan",
+ "9655": "nan",
+ "9660": "nan",
+ "9665": "nan",
+ "9670": "nan",
+ "9675": "nan",
+ "9680": "nan",
+ "9685": "nan",
+ "9690": "nan",
+ "9695": "nan",
+ "9700": 0.90531,
+ "9705": "nan",
+ "9710": "nan",
+ "9715": "nan",
+ "9720": "nan",
+ "9725": "nan",
+ "9730": "nan",
+ "9735": "nan",
+ "9740": "nan",
+ "9745": "nan",
+ "9750": "nan",
+ "9755": "nan",
+ "9760": "nan",
+ "9765": "nan",
+ "9770": "nan",
+ "9775": "nan",
+ "9780": "nan",
+ "9785": "nan",
+ "9790": "nan",
+ "9795": "nan",
+ "9800": 0.99629,
+ "9805": "nan",
+ "9810": "nan",
+ "9815": "nan",
+ "9820": "nan",
+ "9825": "nan",
+ "9830": "nan",
+ "9835": "nan",
+ "9840": "nan",
+ "9845": "nan",
+ "9850": "nan",
+ "9855": "nan",
+ "9860": "nan",
+ "9865": "nan",
+ "9870": "nan",
+ "9875": "nan",
+ "9880": "nan",
+ "9885": "nan",
+ "9890": "nan",
+ "9895": "nan",
+ "9900": 0.91989,
+ "9905": "nan",
+ "9910": "nan",
+ "9915": "nan",
+ "9920": "nan",
+ "9925": "nan",
+ "9930": "nan",
+ "9935": "nan",
+ "9940": "nan",
+ "9945": "nan",
+ "9950": "nan",
+ "9955": "nan",
+ "9960": "nan",
+ "9965": "nan",
+ "9970": "nan",
+ "9975": "nan",
+ "9980": "nan",
+ "9985": "nan",
+ "9990": "nan",
+ "9995": "nan",
+ "10000": 0.90473,
+ "10005": "nan",
+ "10010": "nan",
+ "10015": "nan",
+ "10020": "nan",
+ "10025": "nan",
+ "10030": "nan",
+ "10035": "nan",
+ "10040": "nan",
+ "10045": "nan",
+ "10050": "nan",
+ "10055": "nan",
+ "10060": "nan",
+ "10065": "nan",
+ "10070": "nan",
+ "10075": "nan",
+ "10080": "nan",
+ "10085": "nan",
+ "10090": "nan",
+ "10095": "nan",
+ "10100": 0.92605,
+ "10105": "nan",
+ "10110": "nan",
+ "10115": "nan",
+ "10120": "nan",
+ "10125": "nan",
+ "10130": "nan",
+ "10135": "nan",
+ "10140": "nan",
+ "10145": "nan",
+ "10150": "nan",
+ "10155": "nan",
+ "10160": "nan",
+ "10165": "nan",
+ "10170": "nan",
+ "10175": "nan",
+ "10180": "nan",
+ "10185": "nan",
+ "10190": "nan",
+ "10195": "nan",
+ "10200": 0.98226,
+ "10205": "nan",
+ "10210": "nan",
+ "10215": "nan",
+ "10220": "nan",
+ "10225": "nan",
+ "10230": "nan",
+ "10235": "nan",
+ "10240": "nan",
+ "10245": "nan",
+ "10250": "nan",
+ "10255": "nan",
+ "10260": "nan",
+ "10265": "nan",
+ "10270": "nan",
+ "10275": "nan",
+ "10280": "nan",
+ "10285": "nan",
+ "10290": "nan",
+ "10295": "nan",
+ "10300": 1.00905,
+ "10305": "nan",
+ "10310": "nan",
+ "10315": "nan",
+ "10320": "nan",
+ "10325": "nan",
+ "10330": "nan",
+ "10335": "nan",
+ "10340": "nan",
+ "10345": "nan",
+ "10350": "nan",
+ "10355": "nan",
+ "10360": "nan",
+ "10365": "nan",
+ "10370": "nan",
+ "10375": "nan",
+ "10380": "nan",
+ "10385": "nan",
+ "10390": "nan",
+ "10395": "nan",
+ "10400": 0.92156,
+ "10405": "nan",
+ "10410": "nan",
+ "10415": "nan",
+ "10420": "nan",
+ "10425": "nan",
+ "10430": "nan",
+ "10435": "nan",
+ "10440": "nan",
+ "10445": "nan",
+ "10450": "nan",
+ "10455": "nan",
+ "10460": "nan",
+ "10465": "nan",
+ "10470": "nan",
+ "10475": "nan",
+ "10480": "nan",
+ "10485": "nan",
+ "10490": "nan",
+ "10495": "nan",
+ "10500": 0.92591,
+ "10505": "nan",
+ "10510": "nan",
+ "10515": "nan",
+ "10520": "nan",
+ "10525": "nan",
+ "10530": "nan",
+ "10535": "nan",
+ "10540": "nan",
+ "10545": "nan",
+ "10550": "nan",
+ "10555": "nan",
+ "10560": "nan",
+ "10565": "nan",
+ "10570": "nan",
+ "10575": "nan",
+ "10580": "nan",
+ "10585": "nan",
+ "10590": "nan",
+ "10595": "nan",
+ "10600": 0.92578,
+ "10605": "nan",
+ "10610": "nan",
+ "10615": "nan",
+ "10620": "nan",
+ "10625": "nan",
+ "10630": "nan",
+ "10635": "nan",
+ "10640": "nan",
+ "10645": "nan",
+ "10650": "nan",
+ "10655": "nan",
+ "10660": "nan",
+ "10665": "nan",
+ "10670": "nan",
+ "10675": "nan",
+ "10680": "nan",
+ "10685": "nan",
+ "10690": "nan",
+ "10695": "nan",
+ "10700": 1.00913,
+ "10705": "nan",
+ "10710": "nan",
+ "10715": "nan",
+ "10720": "nan",
+ "10725": "nan",
+ "10730": "nan",
+ "10735": "nan",
+ "10740": "nan",
+ "10745": "nan",
+ "10750": "nan",
+ "10755": "nan",
+ "10760": "nan",
+ "10765": "nan",
+ "10770": "nan",
+ "10775": "nan",
+ "10780": "nan",
+ "10785": "nan",
+ "10790": "nan",
+ "10795": "nan",
+ "10800": 0.92582,
+ "10805": "nan",
+ "10810": "nan",
+ "10815": "nan",
+ "10820": "nan",
+ "10825": "nan",
+ "10830": "nan",
+ "10835": "nan",
+ "10840": "nan",
+ "10845": "nan",
+ "10850": "nan",
+ "10855": "nan",
+ "10860": "nan",
+ "10865": "nan",
+ "10870": "nan",
+ "10875": "nan",
+ "10880": "nan",
+ "10885": "nan",
+ "10890": "nan",
+ "10895": "nan",
+ "10900": 0.92772,
+ "10905": "nan",
+ "10910": "nan",
+ "10915": "nan",
+ "10920": "nan",
+ "10925": "nan",
+ "10930": "nan",
+ "10935": "nan",
+ "10940": "nan",
+ "10945": "nan",
+ "10950": "nan",
+ "10955": "nan",
+ "10960": "nan",
+ "10965": "nan",
+ "10970": "nan",
+ "10975": "nan",
+ "10980": "nan",
+ "10985": "nan",
+ "10990": "nan",
+ "10995": "nan",
+ "11000": 0.91979,
+ "11005": "nan",
+ "11010": "nan",
+ "11015": "nan",
+ "11020": "nan",
+ "11025": "nan",
+ "11030": "nan",
+ "11035": "nan",
+ "11040": "nan",
+ "11045": "nan",
+ "11050": "nan",
+ "11055": "nan",
+ "11060": "nan",
+ "11065": "nan",
+ "11070": "nan",
+ "11075": "nan",
+ "11080": "nan",
+ "11085": "nan",
+ "11090": "nan",
+ "11095": "nan",
+ "11100": 1.03705,
+ "11105": "nan",
+ "11110": "nan",
+ "11115": "nan",
+ "11120": "nan",
+ "11125": "nan",
+ "11130": "nan",
+ "11135": "nan",
+ "11140": "nan",
+ "11145": "nan",
+ "11150": "nan",
+ "11155": "nan",
+ "11160": "nan",
+ "11165": "nan",
+ "11170": "nan",
+ "11175": "nan",
+ "11180": "nan",
+ "11185": "nan",
+ "11190": "nan",
+ "11195": "nan",
+ "11200": 0.90885,
+ "11205": "nan",
+ "11210": "nan",
+ "11215": "nan",
+ "11220": "nan",
+ "11225": "nan",
+ "11230": "nan",
+ "11235": "nan",
+ "11240": "nan",
+ "11245": "nan",
+ "11250": "nan",
+ "11255": "nan",
+ "11260": "nan",
+ "11265": "nan",
+ "11270": "nan",
+ "11275": "nan",
+ "11280": "nan",
+ "11285": "nan",
+ "11290": "nan",
+ "11295": "nan",
+ "11300": 0.91622,
+ "11305": "nan",
+ "11310": "nan",
+ "11315": "nan",
+ "11320": "nan",
+ "11325": "nan",
+ "11330": "nan",
+ "11335": "nan",
+ "11340": "nan",
+ "11345": "nan",
+ "11350": "nan",
+ "11355": "nan",
+ "11360": "nan",
+ "11365": "nan",
+ "11370": "nan",
+ "11375": "nan",
+ "11380": "nan",
+ "11385": "nan",
+ "11390": "nan",
+ "11395": "nan",
+ "11400": 0.91818,
+ "11405": "nan",
+ "11410": "nan",
+ "11415": "nan",
+ "11420": "nan",
+ "11425": "nan",
+ "11430": "nan",
+ "11435": "nan",
+ "11440": "nan",
+ "11445": "nan",
+ "11450": "nan",
+ "11455": "nan",
+ "11460": "nan",
+ "11465": "nan",
+ "11470": "nan",
+ "11475": "nan",
+ "11480": "nan",
+ "11485": "nan",
+ "11490": "nan",
+ "11495": "nan",
+ "11500": 0.92193,
+ "11505": "nan",
+ "11510": "nan",
+ "11515": "nan",
+ "11520": "nan",
+ "11525": "nan",
+ "11530": "nan",
+ "11535": "nan",
+ "11540": "nan",
+ "11545": "nan",
+ "11550": "nan",
+ "11555": "nan",
+ "11560": "nan",
+ "11565": "nan",
+ "11570": "nan",
+ "11575": "nan",
+ "11580": "nan",
+ "11585": "nan",
+ "11590": "nan",
+ "11595": "nan",
+ "11600": 1.00741,
+ "11605": "nan",
+ "11610": "nan",
+ "11615": "nan",
+ "11620": "nan",
+ "11625": "nan",
+ "11630": "nan",
+ "11635": "nan",
+ "11640": "nan",
+ "11645": "nan",
+ "11650": "nan",
+ "11655": "nan",
+ "11660": "nan",
+ "11665": "nan",
+ "11670": "nan",
+ "11675": "nan",
+ "11680": "nan",
+ "11685": "nan",
+ "11690": "nan",
+ "11695": "nan",
+ "11700": 0.92603,
+ "11705": "nan",
+ "11710": "nan",
+ "11715": "nan",
+ "11720": "nan",
+ "11725": "nan",
+ "11730": "nan",
+ "11735": "nan",
+ "11740": "nan",
+ "11745": "nan",
+ "11750": "nan",
+ "11755": "nan",
+ "11760": "nan",
+ "11765": "nan",
+ "11770": "nan",
+ "11775": "nan",
+ "11780": "nan",
+ "11785": "nan",
+ "11790": "nan",
+ "11795": "nan",
+ "11800": 0.92723,
+ "11805": "nan",
+ "11810": "nan",
+ "11815": "nan",
+ "11820": "nan",
+ "11825": "nan",
+ "11830": "nan",
+ "11835": "nan",
+ "11840": "nan",
+ "11845": "nan",
+ "11850": "nan",
+ "11855": "nan",
+ "11860": "nan",
+ "11865": "nan",
+ "11870": "nan",
+ "11875": "nan",
+ "11880": "nan",
+ "11885": "nan",
+ "11890": "nan",
+ "11895": "nan",
+ "11900": 0.92352,
+ "11905": "nan",
+ "11910": "nan",
+ "11915": "nan",
+ "11920": "nan",
+ "11925": "nan",
+ "11930": "nan",
+ "11935": "nan",
+ "11940": "nan",
+ "11945": "nan",
+ "11950": "nan",
+ "11955": "nan",
+ "11960": "nan",
+ "11965": "nan",
+ "11970": "nan",
+ "11975": "nan",
+ "11980": "nan",
+ "11985": "nan",
+ "11990": "nan",
+ "11995": "nan",
+ "12000": 1.05482,
+ "12005": "nan",
+ "12010": "nan",
+ "12015": "nan",
+ "12020": "nan",
+ "12025": "nan",
+ "12030": "nan",
+ "12035": "nan",
+ "12040": "nan",
+ "12045": "nan",
+ "12050": "nan",
+ "12055": "nan",
+ "12060": "nan",
+ "12065": "nan",
+ "12070": "nan",
+ "12075": "nan",
+ "12080": "nan",
+ "12085": "nan",
+ "12090": "nan",
+ "12095": "nan",
+ "12100": 0.94597,
+ "12105": "nan",
+ "12110": "nan",
+ "12115": "nan",
+ "12120": "nan",
+ "12125": "nan",
+ "12130": "nan",
+ "12135": "nan",
+ "12140": "nan",
+ "12145": "nan",
+ "12150": "nan",
+ "12155": "nan",
+ "12160": "nan",
+ "12165": "nan",
+ "12170": "nan",
+ "12175": "nan",
+ "12180": "nan",
+ "12185": "nan",
+ "12190": "nan",
+ "12195": "nan",
+ "12200": 0.91078,
+ "12205": "nan",
+ "12210": "nan",
+ "12215": "nan",
+ "12220": "nan",
+ "12225": "nan",
+ "12230": "nan",
+ "12235": "nan",
+ "12240": "nan",
+ "12245": "nan",
+ "12250": "nan",
+ "12255": "nan",
+ "12260": "nan",
+ "12265": "nan",
+ "12270": "nan",
+ "12275": "nan",
+ "12280": "nan",
+ "12285": "nan",
+ "12290": "nan",
+ "12295": "nan",
+ "12300": 0.89248,
+ "12305": "nan",
+ "12310": "nan",
+ "12315": "nan",
+ "12320": "nan",
+ "12325": "nan",
+ "12330": "nan",
+ "12335": "nan",
+ "12340": "nan",
+ "12345": "nan",
+ "12350": "nan",
+ "12355": "nan",
+ "12360": "nan",
+ "12365": "nan",
+ "12370": "nan",
+ "12375": "nan",
+ "12380": "nan",
+ "12385": "nan",
+ "12390": "nan",
+ "12395": "nan",
+ "12400": 0.9801,
+ "12405": "nan",
+ "12410": "nan",
+ "12415": "nan",
+ "12420": "nan",
+ "12425": "nan",
+ "12430": "nan",
+ "12435": "nan",
+ "12440": "nan",
+ "12445": "nan",
+ "12450": "nan",
+ "12455": "nan",
+ "12460": "nan",
+ "12465": "nan",
+ "12470": "nan",
+ "12475": "nan",
+ "12480": "nan",
+ "12485": "nan",
+ "12490": "nan",
+ "12495": "nan",
+ "12500": 0.90225,
+ "12505": "nan",
+ "12510": "nan",
+ "12515": "nan",
+ "12520": "nan",
+ "12525": "nan",
+ "12530": "nan",
+ "12535": "nan",
+ "12540": "nan",
+ "12545": "nan",
+ "12550": "nan",
+ "12555": "nan",
+ "12560": "nan",
+ "12565": "nan",
+ "12570": "nan",
+ "12575": "nan",
+ "12580": "nan",
+ "12585": "nan",
+ "12590": "nan",
+ "12595": "nan",
+ "12600": 0.92251,
+ "12605": "nan",
+ "12610": "nan",
+ "12615": "nan",
+ "12620": "nan",
+ "12625": "nan",
+ "12630": "nan",
+ "12635": "nan",
+ "12640": "nan",
+ "12645": "nan",
+ "12650": "nan",
+ "12655": "nan",
+ "12660": "nan",
+ "12665": "nan",
+ "12670": "nan",
+ "12675": "nan",
+ "12680": "nan",
+ "12685": "nan",
+ "12690": "nan",
+ "12695": "nan",
+ "12700": 0.91943,
+ "12705": "nan",
+ "12710": "nan",
+ "12715": "nan",
+ "12720": "nan",
+ "12725": "nan",
+ "12730": "nan",
+ "12735": "nan",
+ "12740": "nan",
+ "12745": "nan",
+ "12750": "nan",
+ "12755": "nan",
+ "12760": "nan",
+ "12765": "nan",
+ "12770": "nan",
+ "12775": "nan",
+ "12780": "nan",
+ "12785": "nan",
+ "12790": "nan",
+ "12795": "nan",
+ "12800": "nan",
+ "12805": "nan",
+ "12810": "nan",
+ "12815": "nan",
+ "12820": "nan",
+ "12825": "nan",
+ "12830": "nan",
+ "12835": "nan",
+ "12840": "nan",
+ "12845": "nan",
+ "12850": "nan",
+ "12855": "nan",
+ "12860": "nan",
+ "12865": "nan",
+ "12870": "nan",
+ "12875": "nan",
+ "12880": "nan",
+ "12885": "nan",
+ "12890": "nan",
+ "12895": "nan",
+ "12900": "nan",
+ "12905": "nan",
+ "12910": "nan",
+ "12915": "nan",
+ "12920": "nan",
+ "12925": "nan",
+ "12930": "nan",
+ "12935": "nan",
+ "12940": "nan",
+ "12945": "nan",
+ "12950": "nan",
+ "12955": "nan",
+ "12960": "nan",
+ "12965": "nan",
+ "12970": "nan",
+ "12975": "nan",
+ "12980": "nan",
+ "12985": "nan",
+ "12990": "nan",
+ "12995": "nan",
+ "13000": "nan"
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm/golden_values_dev_dgx_h100.json
index 02bcf7fe698..9405a616f22 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm/golden_values_dev_dgx_h100.json
@@ -1 +1,13042 @@
-{"lm loss": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": 12.98419, "5": 12.93858, "10": 12.06404, "15": 11.97882, "20": 10.53588, "25": 10.11952, "30": 9.7286, "35": 9.44173, "40": 9.2373, "45": 9.03763, "50": 8.85277, "55": 8.64259, "60": 8.60098, "65": 8.50179, "70": 8.41326, "75": 8.31346, "80": 8.16921, "85": 8.09253, "90": 7.97894, "95": 7.91859, "100": 7.82704, "105": 7.71191, "110": 7.62418, "115": 7.52685, "120": 7.48107, "125": 7.48004, "130": 7.33364, "135": 7.26758, "140": 7.23146, "145": 7.04647, "150": 7.17621, "155": 7.00383, "160": 6.89968, "165": 6.91293, "170": 6.84228, "175": 6.85916, "180": 6.81429, "185": 6.7203, "190": 6.66124, "195": 6.59364, "200": 6.64046, "205": 6.64305, "210": 6.5179, "215": 6.51519, "220": 6.51027, "225": 6.46653, "230": 6.47574, "235": 6.42409, "240": 6.36976, "245": 6.3778, "250": 6.29868, "255": 6.43438, "260": 6.34377, "265": 6.28803, "270": 6.23364, "275": 6.26123, "280": 6.19076, "285": 6.19886, "290": 6.15022, "295": 6.12619, "300": 6.11141, "305": 6.01886, "310": 6.08556, "315": 6.07169, "320": 5.99243, "325": 5.93189, "330": 5.99792, "335": 6.0145, "340": 5.93453, "345": 5.92339, "350": 5.87179, "355": 5.84258, "360": 5.85866, "365": 5.81752, "370": 5.80407, "375": 5.80516, "380": 5.85848, "385": 5.78993, "390": 5.81141, "395": 5.68051, "400": 5.66121, "405": 5.68906, "410": 5.66202, "415": 5.70461, "420": 5.63851, "425": 5.66062, "430": 5.62802, "435": 5.56913, "440": 5.62147, "445": 5.52803, "450": 5.58428, "455": 5.5123, "460": 5.49325, "465": 5.56828, "470": 5.54845, "475": 5.49678, "480": 5.46247, "485": 5.49185, "490": 5.47566, "495": 5.47856, "500": 5.42533, "505": 5.38883, "510": 5.44319, "515": 5.42148, "520": 5.47608, "525": 5.31477, "530": 5.33216, "535": 5.36, "540": 5.33276, "545": 5.41314, "550": 5.37099, "555": 5.23374, "560": 5.32665, "565": 5.27809, "570": 5.25324, "575": 5.28184, "580": 5.23593, "585": 5.21762, "590": 5.22346, "595": 5.22561, "600": 5.26751, "605": 5.22896, "610": 5.2012, "615": 5.18737, "620": 5.19543, "625": 5.19655, "630": 5.14985, "635": 5.12452, "640": 5.09298, "645": 5.13279, "650": 5.14481, "655": 5.11963, "660": 5.0475, "665": 5.1142, "670": 5.04119, "675": 5.01723, "680": 5.05635, "685": 5.00678, "690": 5.01633, "695": 4.96228, "700": 4.97301, "705": 4.95571, "710": 4.97305, "715": 4.87719, "720": 4.85764, "725": 4.80769, "730": 4.84352, "735": 4.82916, "740": 4.8644, "745": 4.74895, "750": 4.75764, "755": 4.8023, "760": 4.78257, "765": 4.76428, "770": 4.69615, "775": 4.69212, "780": 4.684, "785": 4.7405, "790": 4.67498, "795": 4.64675, "800": 4.61184, "805": 4.61203, "810": 4.65393, "815": 4.60253, "820": 4.62914, "825": 4.58486, "830": 4.57946, "835": 4.56275, "840": 4.48603, "845": 4.50743, "850": 4.45704, "855": 4.51258, "860": 4.43583, "865": 4.52116, "870": 4.47717, "875": 4.38345, "880": 4.41849, "885": 4.38985, "890": 4.43389, "895": 4.42652, "900": 4.39808, "905": 4.32618, "910": 4.35391, "915": 4.34171, "920": 4.38377, "925": 4.38353, "930": 4.30961, "935": 4.29021, "940": 4.36235, "945": 4.31265, "950": 4.35051, "955": 4.28087, "960": 4.19218, "965": 4.27537, "970": 4.26236, "975": 4.24397, "980": 4.22146, "985": 4.17523, "990": 4.13237, "995": 4.18304, "1000": 4.2308, "1005": 4.18908, "1010": 4.17127, "1015": 4.13273, "1020": 4.15962, "1025": 4.22451, "1030": 4.1247, "1035": 4.10028, "1040": 4.13078, "1045": 4.11598, "1050": 4.15985, "1055": 4.0973, "1060": 4.1011, "1065": 4.06848, "1070": 4.05654, "1075": 4.07977, "1080": 4.07117, "1085": 4.06586, "1090": 4.02567, "1095": 4.09923, "1100": 4.06572, "1105": 4.071, "1110": 4.03046, "1115": 4.00398, "1120": 4.00356, "1125": 4.00318, "1130": 4.04694, "1135": 4.01101, "1140": 4.00143, "1145": 3.93633, "1150": 4.03948, "1155": 4.00619, "1160": 3.9851, "1165": 3.86882, "1170": 3.92684, "1175": 3.93038, "1180": 3.95878, "1185": 3.96184, "1190": 3.93034, "1195": 3.94125, "1200": 3.89932, "1205": 3.87266, "1210": 3.97831, "1215": 3.84899, "1220": 3.87263, "1225": 3.81413, "1230": 3.92846, "1235": 3.89312, "1240": 3.87537, "1245": 3.80079, "1250": 3.82885, "1255": 3.8493, "1260": 3.8874, "1265": 3.79336, "1270": 3.88167, "1275": 3.8455, "1280": 3.82143, "1285": 3.85735, "1290": 3.87159, "1295": 3.84912, "1300": 3.81711, "1305": 3.82495, "1310": 3.82551, "1315": 3.81192, "1320": 3.81405, "1325": 3.72789, "1330": 3.79529, "1335": 3.76889, "1340": 3.76123, "1345": 3.74704, "1350": 3.73516, "1355": 3.76572, "1360": 3.73919, "1365": 3.73441, "1370": 3.73284, "1375": 3.74202, "1380": 3.74909, "1385": 3.7541, "1390": 3.66502, "1395": 3.73883, "1400": 3.72777, "1405": 3.66183, "1410": 3.67095, "1415": 3.64909, "1420": 3.7012, "1425": 3.71121, "1430": 3.6707, "1435": 3.65988, "1440": 3.63587, "1445": 3.67497, "1450": 3.67892, "1455": 3.65003, "1460": 3.64827, "1465": 3.69085, "1470": 3.63258, "1475": 3.6929, "1480": 3.65433, "1485": 3.65353, "1490": 3.61514, "1495": 3.60996, "1500": 3.65344, "1505": 3.69268, "1510": 3.55544, "1515": 3.60731, "1520": 3.63573, "1525": 3.59446, "1530": 3.59436, "1535": 3.59439, "1540": 3.59806, "1545": 3.59944, "1550": 3.56476, "1555": 3.56897, "1560": 3.61095, "1565": 3.62486, "1570": 3.58908, "1575": 3.54769, "1580": 3.59493, "1585": 3.58838, "1590": 3.47011, "1595": 3.51425, "1600": 3.50212, "1605": 3.55557, "1610": 3.56864, "1615": 3.50712, "1620": 3.52052, "1625": 3.47381, "1630": 3.50275, "1635": 3.55197, "1640": 3.51791, "1645": 3.54635, "1650": 3.49031, "1655": 3.4704, "1660": 3.52131, "1665": 3.45881, "1670": 3.51236, "1675": 3.50045, "1680": 3.46681, "1685": 3.49078, "1690": 3.48353, "1695": 3.49776, "1700": 3.46185, "1705": 3.40868, "1710": 3.48728, "1715": 3.49675, "1720": 3.43132, "1725": 3.44122, "1730": 3.42961, "1735": 3.4644, "1740": 3.45584, "1745": 3.43534, "1750": 3.41748, "1755": 3.42246, "1760": 3.37607, "1765": 3.42468, "1770": 3.43261, "1775": 3.379, "1780": 3.4355, "1785": 3.42306, "1790": 3.37807, "1795": 3.40594, "1800": 3.34684, "1805": 3.39025, "1810": 3.34654, "1815": 3.4221, "1820": 3.41309, "1825": 3.38102, "1830": 3.32801, "1835": 3.42581, "1840": 3.38756, "1845": 3.42483, "1850": 3.39575, "1855": 3.37642, "1860": 3.35084, "1865": 3.38387, "1870": 3.29854, "1875": 3.45603, "1880": 3.34133, "1885": 3.36396, "1890": 3.34216, "1895": 3.39809, "1900": 3.36794, "1905": 3.30189, "1910": 3.32844, "1915": 3.3151, "1920": 3.36278, "1925": 3.33318, "1930": 3.31578, "1935": 3.31145, "1940": 3.36438, "1945": 3.26186, "1950": 3.40063, "1955": 3.30708, "1960": 3.31486, "1965": 3.28405, "1970": 3.29999, "1975": 3.33744, "1980": 3.34165, "1985": 3.23762, "1990": 3.32593, "1995": 3.28362, "2000": 3.27303, "2005": 3.26618, "2010": 3.28661, "2015": 3.22715, "2020": 3.27479, "2025": 3.27135, "2030": 3.27231, "2035": 3.29709, "2040": 3.26992, "2045": 3.23395, "2050": 3.27144, "2055": 3.32457, "2060": 3.28203, "2065": 3.24424, "2070": 3.30416, "2075": 3.24361, "2080": 3.22085, "2085": 3.30421, "2090": 3.15415, "2095": 3.29493, "2100": 3.23056, "2105": 3.19586, "2110": 3.20308, "2115": 3.24137, "2120": 3.18045, "2125": 3.21484, "2130": 3.22503, "2135": 3.27763, "2140": 3.1948, "2145": 3.21466, "2150": 3.20867, "2155": 3.2365, "2160": 3.20615, "2165": 3.25296, "2170": 3.23909, "2175": 3.17384, "2180": 3.22049, "2185": 3.25755, "2190": 3.24863, "2195": 3.15892, "2200": 3.2045, "2205": 3.17192, "2210": 3.12378, "2215": 3.19067, "2220": 3.19593, "2225": 3.18949, "2230": 3.13791, "2235": 3.19063, "2240": 3.21593, "2245": 3.1871, "2250": 3.21088, "2255": 3.1585, "2260": 3.14105, "2265": 3.23218, "2270": 3.21271, "2275": 3.1566, "2280": 3.17614, "2285": 3.16486, "2290": 3.17692, "2295": 3.20197, "2300": 3.13847, "2305": 3.15963, "2310": 3.12614, "2315": 3.06316, "2320": 3.11512, "2325": 3.17382, "2330": 3.12397, "2335": 3.12649, "2340": 3.17195, "2345": 3.12462, "2350": 3.129, "2355": 3.11726, "2360": 3.16031, "2365": 3.09266, "2370": 3.15197, "2375": 3.13019, "2380": 3.11082, "2385": 3.09359, "2390": 3.09567, "2395": 3.09807, "2400": 3.09966, "2405": 3.10436, "2410": 3.09007, "2415": 3.09491, "2420": 3.08537, "2425": 3.07877, "2430": 3.08079, "2435": 3.06761, "2440": 3.08574, "2445": 3.05747, "2450": 3.12167, "2455": 3.15832, "2460": 3.08596, "2465": 3.07656, "2470": 3.03663, "2475": 3.06421, "2480": 3.10252, "2485": 3.06485, "2490": 3.06573, "2495": 3.08845, "2500": 3.05671, "2505": 3.105, "2510": 3.12399, "2515": 3.0532, "2520": 3.07806, "2525": 3.02426, "2530": 3.04842, "2535": 3.09401, "2540": 3.07984, "2545": 3.05538, "2550": 3.00469, "2555": 3.07001, "2560": 3.04403, "2565": 3.12, "2570": 3.00976, "2575": 3.0601, "2580": 3.08548, "2585": 3.02156, "2590": 3.06606, "2595": 2.99925, "2600": 3.0841, "2605": 3.06879, "2610": 3.05401, "2615": 3.06935, "2620": 2.99191, "2625": 3.01384, "2630": 3.03627, "2635": 3.05041, "2640": 3.01088, "2645": 3.05612, "2650": 3.02233, "2655": 2.98756, "2660": 3.01604, "2665": 3.03817, "2670": 2.98547, "2675": 2.97442, "2680": 3.00378, "2685": 3.00171, "2690": 2.99912, "2695": 2.99265, "2700": 3.03079, "2705": 2.98376, "2710": 2.97975, "2715": 2.96047, "2720": 3.02663, "2725": 2.99565, "2730": 3.05827, "2735": 3.04913, "2740": 3.02027, "2745": 3.02502, "2750": 3.02065, "2755": 2.96792, "2760": 2.99447, "2765": 3.00785, "2770": 2.98958, "2775": 2.99278, "2780": 3.02294, "2785": 2.95383, "2790": 2.96474, "2795": 2.95595, "2800": 2.98985, "2805": 2.94051, "2810": 2.99046, "2815": 2.95976, "2820": 3.0756, "2825": 3.03639, "2830": 3.01855, "2835": 2.92175, "2840": 2.92574, "2845": 2.96102, "2850": 2.96997, "2855": 2.96207, "2860": 2.94977, "2865": 2.91535, "2870": 2.99202, "2875": 2.92084, "2880": 2.96303, "2885": 2.91779, "2890": 2.98572, "2895": 2.93253, "2900": 2.95289, "2905": 3.00499, "2910": 2.92994, "2915": 2.94325, "2920": 2.95516, "2925": 2.94427, "2930": 2.95621, "2935": 2.94005, "2940": 2.94552, "2945": 2.9075, "2950": 2.97913, "2955": 2.91177, "2960": 2.97029, "2965": 2.87292, "2970": 2.96107, "2975": 2.99603, "2980": 2.94257, "2985": 3.04155, "2990": 2.93897, "2995": 2.87114, "3000": 2.9422, "3005": 2.89655, "3010": 2.93538, "3015": 2.91032, "3020": 2.91995, "3025": 2.91883, "3030": 2.92686, "3035": 2.95815, "3040": 2.9312, "3045": 2.83504, "3050": 2.8988, "3055": 2.89613, "3060": 2.92461, "3065": 2.92459, "3070": 2.88159, "3075": 2.86953, "3080": 2.9243, "3085": 2.90325, "3090": 2.91754, "3095": 2.92816, "3100": 2.86703, "3105": 2.92918, "3110": 2.90236, "3115": 2.94681, "3120": 2.95312, "3125": 2.86217, "3130": 2.93048, "3135": 2.92489, "3140": 2.87699, "3145": 2.91715, "3150": 2.85701, "3155": 2.8442, "3160": 2.83887, "3165": 2.84564, "3170": 2.89213, "3175": 2.90452, "3180": 2.85788, "3185": 2.89571, "3190": 2.90627, "3195": 2.92723, "3200": 2.92789, "3205": 2.85912, "3210": 2.86987, "3215": 2.91563, "3220": 2.87374, "3225": 2.86935, "3230": 2.815, "3235": 2.87434, "3240": 2.8734, "3245": 2.90299, "3250": 2.86289, "3255": 2.8503, "3260": 2.85959, "3265": 2.86936, "3270": 2.85223, "3275": 2.86681, "3280": 2.79974, "3285": 2.81123, "3290": 2.86928, "3295": 2.92038, "3300": 2.87938, "3305": 2.86113, "3310": 2.85785, "3315": 2.80615, "3320": 2.8258, "3325": 2.82403, "3330": 2.82839, "3335": 2.8465, "3340": 2.82742, "3345": 2.84707, "3350": 2.84121, "3355": 2.85933, "3360": 2.79899, "3365": 2.85514, "3370": 2.84537, "3375": 2.84431, "3380": 2.84971, "3385": 2.87822, "3390": 2.8682, "3395": 2.81029, "3400": 2.78472, "3405": 2.82816, "3410": 2.84591, "3415": 2.86134, "3420": 2.82742, "3425": 2.81129, "3430": 2.82982, "3435": 2.8906, "3440": 2.81795, "3445": 2.86973, "3450": 2.81507, "3455": 2.7888, "3460": 2.8203, "3465": 2.84691, "3470": 2.83475, "3475": 2.7758, "3480": 2.84173, "3485": 2.82043, "3490": 2.8933, "3495": 2.84901, "3500": 2.84084, "3505": 2.82498, "3510": 2.81285, "3515": 2.83554, "3520": 2.77982, "3525": 2.80208, "3530": 2.84998, "3535": 2.78436, "3540": 2.83771, "3545": 2.81048, "3550": 2.79684, "3555": 2.8181, "3560": 2.82828, "3565": 2.82912, "3570": 2.80393, "3575": 2.80372, "3580": 2.82073, "3585": 2.83541, "3590": 2.8298, "3595": 2.77921, "3600": 2.74897, "3605": 2.79083, "3610": 2.8488, "3615": 2.75492, "3620": 2.80351, "3625": 2.88338, "3630": 2.77314, "3635": 2.78625, "3640": 2.78253, "3645": 2.76926, "3650": 2.80301, "3655": 2.81469, "3660": 2.76524, "3665": 2.7858, "3670": 2.77457, "3675": 2.77711, "3680": 2.80733, "3685": 2.80194, "3690": 2.8055, "3695": 2.81135, "3700": 2.78752, "3705": 2.78408, "3710": 2.75166, "3715": 2.80201, "3720": 2.79409, "3725": 2.78884, "3730": 2.84096, "3735": 2.80547, "3740": 2.74952, "3745": 2.78986, "3750": 2.8022, "3755": 2.79576, "3760": 2.75757, "3765": 2.75586, "3770": 2.75989, "3775": 2.76991, "3780": 2.76383, "3785": 2.7793, "3790": 2.74202, "3795": 2.79511, "3800": 2.80269, "3805": 2.75159, "3810": 2.80354, "3815": 2.76482, "3820": 2.78758, "3825": 2.73331, "3830": 2.74563, "3835": 2.81464, "3840": 2.72812, "3845": 2.71424, "3850": 2.77453, "3855": 2.71774, "3860": 2.80173, "3865": 2.75469, "3870": 2.77531, "3875": 2.75779, "3880": 2.78968, "3885": 2.78424, "3890": 2.74541, "3895": 2.79804, "3900": 2.76127, "3905": 2.72353, "3910": 2.74147, "3915": 2.75183, "3920": 2.79462, "3925": 2.77792, "3930": 2.70759, "3935": 2.73982, "3940": 2.75131, "3945": 2.74267, "3950": 2.725, "3955": 2.77958, "3960": 2.75991, "3965": 2.74216, "3970": 2.75653, "3975": 2.72552, "3980": 2.73817, "3985": 2.75045, "3990": 2.69347, "3995": 2.78059, "4000": 2.73558, "4005": 2.7658, "4010": 2.70885, "4015": 2.72538, "4020": 2.74956, "4025": 2.733, "4030": 2.65924, "4035": 2.69455, "4040": 2.74652, "4045": 2.74857, "4050": 2.78817, "4055": 2.7239, "4060": 2.71419, "4065": 2.6515, "4070": 2.80691, "4075": 2.75748, "4080": 2.71884, "4085": 2.74977, "4090": 2.67836, "4095": 2.69073, "4100": 2.7114, "4105": 2.73822, "4110": 2.72956, "4115": 2.70127, "4120": 2.73267, "4125": 2.70389, "4130": 2.69553, "4135": 2.6893, "4140": 2.68057, "4145": 2.77973, "4150": 2.70801, "4155": 2.73792, "4160": 2.76329, "4165": 2.72099, "4170": 2.67438, "4175": 2.71828, "4180": 2.72666, "4185": 2.72916, "4190": 2.73806, "4195": 2.70222, "4200": 2.71066, "4205": 2.73922, "4210": 2.67309, "4215": 2.66565, "4220": 2.65928, "4225": 2.70242, "4230": 2.71409, "4235": 2.7326, "4240": 2.70413, "4245": 2.69658, "4250": 2.71363, "4255": 2.64837, "4260": 2.7266, "4265": 2.73863, "4270": 2.72157, "4275": 2.68943, "4280": 2.70233, "4285": 2.73114, "4290": 2.68765, "4295": 2.69223, "4300": 2.69956, "4305": 2.70313, "4310": 2.73003, "4315": 2.71191, "4320": 2.69906, "4325": 2.70557, "4330": 2.7106, "4335": 2.69172, "4340": 2.6976, "4345": 2.72675, "4350": 2.67431, "4355": 2.69349, "4360": 2.71041, "4365": 2.78314, "4370": 2.73369, "4375": 2.74431, "4380": 2.71504, "4385": 2.69901, "4390": 2.70191, "4395": 2.75058, "4400": 2.66501, "4405": 2.66954, "4410": 2.68278, "4415": 2.70442, "4420": 2.7077, "4425": 2.72158, "4430": 2.69277, "4435": 2.68035, "4440": 2.69127, "4445": 2.67896, "4450": 2.65272, "4455": 2.69044, "4460": 2.70302, "4465": 2.70631, "4470": 2.6731, "4475": 2.68761, "4480": 2.65626, "4485": 2.69968, "4490": 2.65279, "4495": 2.70894, "4500": 2.70235, "4505": 2.69674, "4510": 2.64893, "4515": 2.70162, "4520": 2.66802, "4525": 2.66816, "4530": 2.6736, "4535": 2.67113, "4540": 2.70729, "4545": 2.65603, "4550": 2.70177, "4555": 2.68009, "4560": 2.65895, "4565": 2.63898, "4570": 2.6402, "4575": 2.66692, "4580": 2.68858, "4585": 2.68157, "4590": 2.61727, "4595": 2.66336, "4600": 2.67638, "4605": 2.68094, "4610": 2.66704, "4615": 2.66234, "4620": 2.65727, "4625": 2.68721, "4630": 2.6742, "4635": 2.64708, "4640": 2.69575, "4645": 2.64774, "4650": 2.7018, "4655": 2.70661, "4660": 2.67838, "4665": 2.68918, "4670": 2.67556, "4675": 2.68422, "4680": 2.66596, "4685": 2.65737, "4690": 2.70912, "4695": 2.65528, "4700": 2.67174, "4705": 2.65146, "4710": 2.68366, "4715": 2.64964, "4720": 2.72485, "4725": 2.62902, "4730": 2.65213, "4735": 2.68926, "4740": 2.64614, "4745": 2.65553, "4750": 2.65737, "4755": 2.65793, "4760": 2.66632, "4765": 2.64489, "4770": 2.62202, "4775": 2.65194, "4780": 2.65519, "4785": 2.68655, "4790": 2.65201, "4795": 2.67305, "4800": 2.62427, "4805": 2.64099, "4810": 2.65942, "4815": 2.65033, "4820": 2.6668, "4825": 2.65019, "4830": 2.6151, "4835": 2.64832, "4840": 2.65513, "4845": 2.6348, "4850": 2.62482, "4855": 2.60337, "4860": 2.65757, "4865": 2.62536, "4870": 2.63972, "4875": 2.61897, "4880": 2.62576, "4885": 2.62632, "4890": 2.67912, "4895": 2.65961, "4900": 2.618, "4905": 2.61823, "4910": 2.63845, "4915": 2.61463, "4920": 2.65397, "4925": 2.64838, "4930": 2.57129, "4935": 2.65193, "4940": 2.63034, "4945": 2.63777, "4950": 2.62825, "4955": 2.61794, "4960": 2.61856, "4965": 2.65951, "4970": 2.6008, "4975": 2.65676, "4980": 2.62049, "4985": 2.63225, "4990": 2.65645, "4995": 2.58184, "5000": 2.6621, "5005": 2.6658, "5010": 2.68112, "5015": 2.63396, "5020": 2.64091, "5025": 2.68726, "5030": 2.64362, "5035": 2.61873, "5040": 2.62248, "5045": 2.60699, "5050": 2.62641, "5055": 2.65014, "5060": 2.64375, "5065": 2.68893, "5070": 2.60617, "5075": 2.61421, "5080": 2.61231, "5085": 2.60499, "5090": 2.59441, "5095": 2.65086, "5100": 2.64984, "5105": 2.61053, "5110": 2.66408, "5115": 2.62171, "5120": 2.67055, "5125": 2.6309, "5130": 2.615, "5135": 2.61462, "5140": 2.57424, "5145": 2.62966, "5150": 2.63646, "5155": 2.61887, "5160": 2.66278, "5165": 2.58409, "5170": 2.59136, "5175": 2.62185, "5180": 2.60659, "5185": 2.62099, "5190": 2.6266, "5195": 2.67047, "5200": 2.5968, "5205": 2.60868, "5210": 2.60701, "5215": 2.64792, "5220": 2.58826, "5225": 2.55166, "5230": 2.6359, "5235": 2.61417, "5240": 2.62802, "5245": 2.64006, "5250": 2.61297, "5255": 2.62612, "5260": 2.5619, "5265": 2.59802, "5270": 2.58865, "5275": 2.61781, "5280": 2.61032, "5285": 2.60442, "5290": 2.63245, "5295": 2.62071, "5300": 2.57979, "5305": 2.59834, "5310": 2.60591, "5315": 2.5881, "5320": 2.61539, "5325": 2.64615, "5330": 2.6015, "5335": 2.58439, "5340": 2.56291, "5345": 2.65819, "5350": 2.62526, "5355": 2.57953, "5360": 2.59528, "5365": 2.62373, "5370": 2.61518, "5375": 2.63002, "5380": 2.58083, "5385": 2.56502, "5390": 2.58666, "5395": 2.61597, "5400": 2.60909, "5405": 2.54774, "5410": 2.61298, "5415": 2.59619, "5420": 2.61443, "5425": 2.62678, "5430": 2.62674, "5435": 2.57707, "5440": 2.58734, "5445": 2.633, "5450": 2.6473, "5455": 2.61252, "5460": 2.59272, "5465": 2.60502, "5470": 2.60189, "5475": 2.62728, "5480": 2.58753, "5485": 2.59002, "5490": 2.57733, "5495": 2.57075, "5500": 2.56937, "5505": 2.61715, "5510": 2.62664, "5515": 2.58137, "5520": 2.55697, "5525": 2.5859, "5530": 2.66433, "5535": 2.62339, "5540": 2.57109, "5545": 2.59633, "5550": 2.54936, "5555": 2.57342, "5560": 2.56447, "5565": 2.60758, "5570": 2.65168, "5575": 2.63138, "5580": 2.57564, "5585": 2.59822, "5590": 2.56185, "5595": 2.58521, "5600": 2.55512, "5605": 2.59879, "5610": 2.58291, "5615": 2.58198, "5620": 2.58123, "5625": 2.55147, "5630": 2.57081, "5635": 2.63484, "5640": 2.59425, "5645": 2.56995, "5650": 2.58004, "5655": 2.54766, "5660": 2.55881, "5665": 2.58604, "5670": 2.56686, "5675": 2.60728, "5680": 2.52861, "5685": 2.56813, "5690": 2.6039, "5695": 2.55782, "5700": 2.59695, "5705": 2.596, "5710": 2.57921, "5715": 2.58424, "5720": 2.53643, "5725": 2.6038, "5730": 2.57366, "5735": 2.61087, "5740": 2.59519, "5745": 2.56, "5750": 2.54216, "5755": 2.55997, "5760": 2.62481, "5765": 2.56328, "5770": 2.5429, "5775": 2.58373, "5780": 2.57701, "5785": 2.53911, "5790": 2.56461, "5795": 2.60179, "5800": 2.54494, "5805": 2.53531, "5810": 2.55658, "5815": 2.52456, "5820": 2.59694, "5825": 2.50599, "5830": 2.49558, "5835": 2.59597, "5840": 2.53979, "5845": 2.5528, "5850": 2.61315, "5855": 2.5102, "5860": 2.56169, "5865": 2.51778, "5870": 2.57574, "5875": 2.60723, "5880": 2.58596, "5885": 2.56757, "5890": 2.58608, "5895": 2.55562, "5900": 2.61651, "5905": 2.55716, "5910": 2.59828, "5915": 2.61008, "5920": 2.58733, "5925": 2.55324, "5930": 2.57568, "5935": 2.55168, "5940": 2.57131, "5945": 2.5204, "5950": 2.55562, "5955": 2.586, "5960": 2.56741, "5965": 2.62046, "5970": 2.55594, "5975": 2.58503, "5980": 2.55843, "5985": 2.56032, "5990": 2.55653, "5995": 2.55873, "6000": 2.55658, "6005": 2.51961, "6010": 2.5612, "6015": 2.52607, "6020": 2.53453, "6025": 2.55768, "6030": 2.6046, "6035": 2.54228, "6040": 2.54868, "6045": 2.49077, "6050": 2.5963, "6055": 2.5204, "6060": 2.54409, "6065": 2.52518, "6070": 2.52918, "6075": 2.5364, "6080": 2.53607, "6085": 2.59714, "6090": 2.57034, "6095": 2.53592, "6100": 2.5428, "6105": 2.52487, "6110": 2.55483, "6115": 2.58495, "6120": 2.55695, "6125": 2.53683, "6130": 2.47322, "6135": 2.5563, "6140": 2.55589, "6145": 2.55739, "6150": 2.52565, "6155": 2.50872, "6160": 2.54299, "6165": 2.57304, "6170": 2.54638, "6175": 2.60079, "6180": 2.51196, "6185": 2.55194, "6190": 2.49345, "6195": 2.57854, "6200": 2.55164, "6205": 2.5377, "6210": 2.52088, "6215": 2.51358, "6220": 2.56539, "6225": 2.51406, "6230": 2.51072, "6235": 2.56268, "6240": 2.55115, "6245": 2.52327, "6250": 2.53069, "6255": 2.57365, "6260": 2.52537, "6265": 2.57441, "6270": 2.52397, "6275": 2.56565, "6280": 2.52297, "6285": 2.5207, "6290": 2.51982, "6295": 2.50722, "6300": 2.55559, "6305": 2.52486, "6310": 2.51259, "6315": 2.53731, "6320": 2.4894, "6325": 2.59818, "6330": 2.555, "6335": 2.51085, "6340": 2.51313, "6345": 2.55702, "6350": 2.556, "6355": 2.52448, "6360": 2.52293, "6365": 2.48409, "6370": 2.53563, "6375": 2.49779, "6380": 2.56282, "6385": 2.58189, "6390": 2.50441, "6395": 2.55121, "6400": 2.5086, "6405": 2.5278, "6410": 2.51466, "6415": 2.52482, "6420": 2.54258, "6425": 2.53509, "6430": 2.57978, "6435": 2.54444, "6440": 2.53907, "6445": 2.53125, "6450": 2.53474, "6455": 2.52399, "6460": 2.51849, "6465": 2.56225, "6470": 2.52104, "6475": 2.52654, "6480": 2.48826, "6485": 2.52861, "6490": 2.50978, "6495": 2.49978, "6500": 2.52402, "6505": 2.49432, "6510": 2.54199, "6515": 2.5101, "6520": 2.51003, "6525": 2.49503, "6530": 2.54392, "6535": 2.53282, "6540": 2.53291, "6545": 2.56194, "6550": 2.50127, "6555": 2.55627, "6560": 2.51016, "6565": 2.52281, "6570": 2.58445, "6575": 2.52324, "6580": 2.49815, "6585": 2.50823, "6590": 2.5097, "6595": 2.49807, "6600": 2.49539, "6605": 2.54253, "6610": 2.4797, "6615": 2.56766, "6620": 2.53402, "6625": 2.51202, "6630": 2.51431, "6635": 2.47464, "6640": 2.54106, "6645": 2.59681, "6650": 2.51024, "6655": 2.4983, "6660": 2.57419, "6665": 2.52156, "6670": 2.5674, "6675": 2.46861, "6680": 2.54697, "6685": 2.53564, "6690": 2.51427, "6695": 2.48573, "6700": 2.52463, "6705": 2.52218, "6710": 2.49347, "6715": 2.51687, "6720": 2.50996, "6725": 2.52089, "6730": 2.52013, "6735": 2.4825, "6740": 2.51535, "6745": 2.49672, "6750": 2.55754, "6755": 2.47484, "6760": 2.54212, "6765": 2.48878, "6770": 2.51847, "6775": 2.50828, "6780": 2.53878, "6785": 2.47177, "6790": 2.54553, "6795": 2.49868, "6800": 2.52671, "6805": 2.51099, "6810": 2.50296, "6815": 2.52064, "6820": 2.48696, "6825": 2.5071, "6830": 2.54063, "6835": 2.50678, "6840": 2.50885, "6845": 2.52492, "6850": 2.47583, "6855": 2.512, "6860": 2.50239, "6865": 2.49001, "6870": 2.55392, "6875": 2.47561, "6880": 2.55072, "6885": 2.47892, "6890": 2.54905, "6895": 2.50384, "6900": 2.49072, "6905": 2.51205, "6910": 2.5215, "6915": 2.51823, "6920": 2.5328, "6925": 2.54741, "6930": 2.49289, "6935": 2.521, "6940": 2.50604, "6945": 2.46237, "6950": 2.48628, "6955": 2.5288, "6960": 2.51952, "6965": 2.49196, "6970": 2.47065, "6975": 2.52409, "6980": 2.45258, "6985": 2.51631, "6990": 2.52932, "6995": 2.46179, "7000": 2.49172, "7005": 2.47011, "7010": 2.47632, "7015": 2.51983, "7020": 2.46705, "7025": 2.45424, "7030": 2.48487, "7035": 2.47988, "7040": 2.50783, "7045": 2.52359, "7050": 2.52831, "7055": 2.44161, "7060": 2.47409, "7065": 2.48138, "7070": 2.48981, "7075": 2.49452, "7080": 2.53479, "7085": 2.48717, "7090": 2.47618, "7095": 2.4999, "7100": 2.51585, "7105": 2.4884, "7110": 2.487, "7115": 2.50558, "7120": 2.47286, "7125": 2.46376, "7130": 2.48693, "7135": 2.51456, "7140": 2.50032, "7145": 2.49769, "7150": 2.51016, "7155": 2.50401, "7160": 2.47274, "7165": 2.45638, "7170": 2.50459, "7175": 2.50355, "7180": 2.50497, "7185": 2.48172, "7190": 2.46296, "7195": 2.46639, "7200": 2.50998, "7205": 2.49029, "7210": 2.44246, "7215": 2.47885, "7220": 2.4456, "7225": 2.51269, "7230": 2.50805, "7235": 2.48249, "7240": 2.47867, "7245": 2.50035, "7250": 2.50922, "7255": 2.49324, "7260": 2.46058, "7265": 2.45308, "7270": 2.47086, "7275": 2.49781, "7280": 2.49343, "7285": 2.42363, "7290": 2.47944, "7295": 2.48626, "7300": 2.41751, "7305": 2.44554, "7310": 2.44899, "7315": 2.48986, "7320": 2.48389, "7325": 2.45917, "7330": 2.4893, "7335": 2.47688, "7340": 2.46486, "7345": 2.49515, "7350": 2.5106, "7355": 2.49669, "7360": 2.48037, "7365": 2.46906, "7370": 2.47138, "7375": 2.4508, "7380": 2.49622, "7385": 2.48448, "7390": 2.47337, "7395": 2.47339, "7400": 2.48169, "7405": 2.43994, "7410": 2.48078, "7415": 2.47113, "7420": 2.49398, "7425": 2.45774, "7430": 2.52358, "7435": 2.49185, "7440": 2.52151, "7445": 2.5101, "7450": 2.4751, "7455": 2.45401, "7460": 2.46474, "7465": 2.47685, "7470": 2.44899, "7475": 2.45681, "7480": 2.51145, "7485": 2.45042, "7490": 2.47478, "7495": 2.48246, "7500": 2.49584, "7505": 2.44104, "7510": 2.43501, "7515": 2.41997, "7520": 2.49389, "7525": 2.49884, "7530": 2.47668, "7535": 2.4601, "7540": 2.47288, "7545": 2.47471, "7550": 2.49181, "7555": 2.45487, "7560": 2.42922, "7565": 2.51106, "7570": 2.4857, "7575": 2.439, "7580": 2.45825, "7585": 2.48256, "7590": 2.48193, "7595": 2.46508, "7600": 2.46362, "7605": 2.44863, "7610": 2.44948, "7615": 2.42526, "7620": 2.54441, "7625": 2.47879, "7630": 2.42526, "7635": 2.42739, "7640": 2.45364, "7645": 2.47151, "7650": 2.46303, "7655": 2.48304, "7660": 2.4532, "7665": 2.4342, "7670": 2.4426, "7675": 2.45588, "7680": 2.48517, "7685": 2.43208, "7690": 2.48, "7695": 2.45485, "7700": 2.48159, "7705": 2.49878, "7710": 2.49483, "7715": 2.44384, "7720": 2.4696, "7725": 2.47981, "7730": 2.45864, "7735": 2.47057, "7740": 2.43882, "7745": 2.45157, "7750": 2.43921, "7755": 2.46722, "7760": 2.45122, "7765": 2.45511, "7770": 2.47144, "7775": 2.45332, "7780": 2.41653, "7785": 2.44516, "7790": 2.48285, "7795": 2.44125, "7800": 2.46355, "7805": 2.48202, "7810": 2.50258, "7815": 2.48733, "7820": 2.44788, "7825": 2.51471, "7830": 2.45477, "7835": 2.4697, "7840": 2.47907, "7845": 2.46064, "7850": 2.41717, "7855": 2.47244, "7860": 2.49887, "7865": 2.42434, "7870": 2.46693, "7875": 2.44544, "7880": 2.45287, "7885": 2.46023, "7890": 2.47026, "7895": 2.44872, "7900": 2.4404, "7905": 2.43773, "7910": 2.42565, "7915": 2.48107, "7920": 2.47699, "7925": 2.4218, "7930": 2.47199, "7935": 2.44975, "7940": 2.42126, "7945": 2.46977, "7950": 2.44424, "7955": 2.4204, "7960": 2.49038, "7965": 2.5188, "7970": 2.52207, "7975": 2.44798, "7980": 2.44076, "7985": 2.46872, "7990": 2.43169, "7995": 2.46954, "8000": 2.43641, "8005": 2.41891, "8010": 2.45749, "8015": 2.46841, "8020": 2.48116, "8025": 2.47363, "8030": 2.45173, "8035": 2.47071, "8040": 2.41983, "8045": 2.45333, "8050": 2.44721, "8055": 2.42302, "8060": 2.44253, "8065": 2.46158, "8070": 2.4567, "8075": 2.46077, "8080": 2.44618, "8085": 2.44085, "8090": 2.42787, "8095": 2.42397, "8100": 2.43904, "8105": 2.49479, "8110": 2.43878, "8115": 2.58899, "8120": 2.49362, "8125": 2.47876, "8130": 2.45879, "8135": 2.4574, "8140": 2.44166, "8145": 2.42774, "8150": 2.42089, "8155": 2.48312, "8160": 2.45131, "8165": 2.43947, "8170": 2.43326, "8175": 2.42092, "8180": 2.4946, "8185": 2.42477, "8190": 2.46908, "8195": 2.45732, "8200": 2.44651, "8205": 2.44406, "8210": 2.43096, "8215": 2.44122, "8220": 2.43556, "8225": 2.41067, "8230": 2.44055, "8235": 2.46438, "8240": 2.42694, "8245": 2.44767, "8250": 2.44524, "8255": 2.43772, "8260": 2.43153, "8265": 2.42903, "8270": 2.4363, "8275": 2.44197, "8280": 2.39831, "8285": 2.4405, "8290": 2.48021, "8295": 2.44762, "8300": 2.45931, "8305": 2.40847, "8310": 2.43461, "8315": 2.45616, "8320": 2.40422, "8325": 2.39725, "8330": 2.43986, "8335": 2.44684, "8340": 2.49212, "8345": 2.44942, "8350": 2.45049, "8355": 2.40704, "8360": 2.40131, "8365": 2.45443, "8370": 2.45427, "8375": 2.42518, "8380": 2.41939, "8385": 2.42541, "8390": 2.4387, "8395": 2.44193, "8400": 2.44114, "8405": 2.49132, "8410": 2.4383, "8415": 2.43519, "8420": 2.41861, "8425": 2.44324, "8430": 2.46253, "8435": 2.40559, "8440": 2.45227, "8445": 2.45999, "8450": 2.40867, "8455": 2.46028, "8460": 2.45495, "8465": 2.43629, "8470": 2.40854, "8475": 2.47887, "8480": 2.40222, "8485": 2.41392, "8490": 2.46612, "8495": 2.43613, "8500": 2.44492, "8505": 2.40329, "8510": 2.40218, "8515": 2.42871, "8520": 2.42574, "8525": 2.49152, "8530": 2.3746, "8535": 2.40109, "8540": 2.48679, "8545": 2.3811, "8550": 2.43875, "8555": 2.4514, "8560": 2.47019, "8565": 2.42055, "8570": 2.43185, "8575": 2.44959, "8580": 2.44124, "8585": 2.42059, "8590": 2.4038, "8595": 2.42895, "8600": 2.41116, "8605": 2.49131, "8610": 2.42052, "8615": 2.38808, "8620": 2.45039, "8625": 2.42523, "8630": 2.45471, "8635": 2.4509, "8640": 2.43534, "8645": 2.47406, "8650": 2.42305, "8655": 2.45293, "8660": 2.45576, "8665": 2.38622, "8670": 2.41139, "8675": 2.42943, "8680": 2.44841, "8685": 2.43079, "8690": 2.41017, "8695": 2.44311, "8700": 2.43428, "8705": 2.42016, "8710": 2.42854, "8715": 2.44862, "8720": 2.47696, "8725": 2.41012, "8730": 2.39278, "8735": 2.43505, "8740": 2.43198, "8745": 2.39801, "8750": 2.43609, "8755": 2.42381, "8760": 2.40031, "8765": 2.43541, "8770": 2.40569, "8775": 2.43812, "8780": 2.42153, "8785": 2.47144, "8790": 2.42041, "8795": 2.41876, "8800": 2.41592, "8805": 2.40548, "8810": 2.41139, "8815": 2.47509, "8820": 2.45362, "8825": 2.4241, "8830": 2.38744, "8835": 2.42258, "8840": 2.39347, "8845": 2.42679, "8850": 2.43485, "8855": 2.4044, "8860": 2.42715, "8865": 2.42631, "8870": 2.43391, "8875": 2.44152, "8880": 2.41099, "8885": 2.39514, "8890": 2.44614, "8895": 2.42902, "8900": 2.41354, "8905": 2.40085, "8910": 2.4019, "8915": 2.4163, "8920": 2.43454, "8925": 2.46713, "8930": 2.41511, "8935": 2.40784, "8940": 2.38869, "8945": 2.39353, "8950": 2.41789, "8955": 2.39534, "8960": 2.43426, "8965": 2.41798, "8970": 2.40536, "8975": 2.47767, "8980": 2.44109, "8985": 2.37482, "8990": 2.41061, "8995": 2.416, "9000": 2.45568, "9005": 2.41279, "9010": 2.37662, "9015": 2.41141, "9020": 2.40089, "9025": 2.3701, "9030": 2.40026, "9035": 2.4243, "9040": 2.42079, "9045": 2.41805, "9050": 2.39505, "9055": 2.41785, "9060": 2.41922, "9065": 2.40527, "9070": 2.44454, "9075": 2.39395, "9080": 2.43398, "9085": 2.4136, "9090": 2.41293, "9095": 2.39793, "9100": 2.40135, "9105": 2.35782, "9110": 2.46451, "9115": 2.41499, "9120": 2.40368, "9125": 2.45804, "9130": 2.39387, "9135": 2.44878, "9140": 2.43562, "9145": 2.42684, "9150": 2.42505, "9155": 2.3752, "9160": 2.41724, "9165": 2.42569, "9170": 2.37359, "9175": 2.41857, "9180": 2.37803, "9185": 2.43942, "9190": 2.41281, "9195": 2.40662, "9200": 2.39186, "9205": 2.44999, "9210": 2.36248, "9215": 2.46363, "9220": 2.44779, "9225": 2.3828, "9230": 2.44575, "9235": 2.39772, "9240": 2.40182, "9245": 2.43796, "9250": 2.43806, "9255": 2.4326, "9260": 2.38813, "9265": 2.43977, "9270": 2.43657, "9275": 2.39535, "9280": 2.39074, "9285": 2.42225, "9290": 2.40437, "9295": 2.38603, "9300": 2.42495, "9305": 2.40579, "9310": 2.41555, "9315": 2.41153, "9320": 2.44493, "9325": 2.37049, "9330": 2.40434, "9335": 2.36191, "9340": 2.40835, "9345": 2.41458, "9350": 2.44039, "9355": 2.47763, "9360": 2.43745, "9365": 2.38821, "9370": 2.43648, "9375": 2.43331, "9380": 2.35346, "9385": 2.39958, "9390": 2.38109, "9395": 2.38731, "9400": 2.44471, "9405": 2.41259, "9410": 2.39756, "9415": 2.43759, "9420": 2.4441, "9425": 2.43656, "9430": 2.45071, "9435": 2.41453, "9440": 2.47761, "9445": 2.37622, "9450": 2.39383, "9455": 2.40249, "9460": 2.38597, "9465": 2.3775, "9470": 2.38205, "9475": 2.36454, "9480": 2.43551, "9485": 2.38642, "9490": 2.4204, "9495": 2.38165, "9500": 2.36325, "9505": 2.4296, "9510": 2.39916, "9515": 2.43096, "9520": 2.41792, "9525": 2.38898, "9530": 2.45385, "9535": 2.40151, "9540": 2.41839, "9545": 2.37813, "9550": 2.42143, "9555": 2.39054, "9560": 2.42191, "9565": 2.40523, "9570": 2.37157, "9575": 2.41109, "9580": 2.39564, "9585": 2.42353, "9590": 2.42924, "9595": 2.44777, "9600": 2.39117, "9605": 2.38431, "9610": 2.42142, "9615": 2.41558, "9620": 2.41413, "9625": 2.44723, "9630": 2.39712, "9635": 2.40396, "9640": 2.44817, "9645": 2.4109, "9650": 2.39894, "9655": 2.37366, "9660": 2.42329, "9665": 2.39029, "9670": 2.38274, "9675": 2.35662, "9680": 2.39869, "9685": 2.40199, "9690": 2.46804, "9695": 2.38133, "9700": 2.37698, "9705": 2.38453, "9710": 2.36554, "9715": 2.38868, "9720": 2.43552, "9725": 2.4413, "9730": 2.42919, "9735": 2.38684, "9740": 2.38077, "9745": 2.42676, "9750": 2.3991, "9755": 2.40788, "9760": 2.41084, "9765": 2.37036, "9770": 2.43675, "9775": 2.40145, "9780": 2.36196, "9785": 2.40085, "9790": 2.40714, "9795": 2.3593, "9800": 2.39629, "9805": 2.40561, "9810": 2.41066, "9815": 2.37884, "9820": 2.37671, "9825": 2.40364, "9830": 2.42194, "9835": 2.3861, "9840": 2.41457, "9845": 2.36502, "9850": 2.39824, "9855": 2.39496, "9860": 2.3972, "9865": 2.38197, "9870": 2.39342, "9875": 2.38398, "9880": 2.45319, "9885": 2.39313, "9890": 2.35399, "9895": 2.32116, "9900": 2.3962, "9905": 2.42494, "9910": 2.35642, "9915": 2.36473, "9920": 2.41154, "9925": 2.39863, "9930": 2.38182, "9935": 2.35063, "9940": 2.38377, "9945": 2.37842, "9950": 2.40342, "9955": 2.44928, "9960": 2.43108, "9965": 2.35851, "9970": 2.41017, "9975": 2.38564, "9980": 2.33084, "9985": 2.40772, "9990": 2.39761, "9995": 2.39543, "10000": 2.36621, "10005": 2.37213, "10010": 2.38256, "10015": 2.44495, "10020": 2.36326, "10025": 2.38851, "10030": 2.38817, "10035": 2.40993, "10040": 2.40515, "10045": 2.3831, "10050": 2.34965, "10055": 2.36805, "10060": 2.42146, "10065": 2.37528, "10070": 2.42235, "10075": 2.37088, "10080": 2.36211, "10085": 2.36918, "10090": 2.34573, "10095": 2.40221, "10100": 2.31408, "10105": 2.38253, "10110": 2.40897, "10115": 2.38736, "10120": 2.35801, "10125": 2.37033, "10130": 2.36037, "10135": 2.38382, "10140": 2.4139, "10145": 2.40714, "10150": 2.37532, "10155": 2.39536, "10160": 2.36205, "10165": 2.38369, "10170": 2.4236, "10175": 2.32447, "10180": 2.39651, "10185": 2.3824, "10190": 2.44396, "10195": 2.40416, "10200": 2.38955, "10205": 2.38797, "10210": 2.36805, "10215": 2.34261, "10220": 2.41843, "10225": 2.43079, "10230": 2.35627, "10235": 2.38764, "10240": 2.37226, "10245": 2.39117, "10250": 2.38838, "10255": 2.41316, "10260": 2.33469, "10265": 2.34846, "10270": 2.34979, "10275": 2.3717, "10280": 2.4513, "10285": 2.35906, "10290": 2.3861, "10295": 2.375, "10300": 2.36936, "10305": 2.41578, "10310": 2.38877, "10315": 2.36095, "10320": 2.36607, "10325": 2.36094, "10330": 2.41247, "10335": 2.36135, "10340": 2.41934, "10345": 2.36966, "10350": 2.35686, "10355": 2.39609, "10360": 2.37338, "10365": 2.36225, "10370": 2.34061, "10375": 2.3585, "10380": 2.41953, "10385": 2.40576, "10390": 2.38058, "10395": 2.35968, "10400": 2.37919, "10405": 2.34877, "10410": 2.3389, "10415": 2.41664, "10420": 2.37924, "10425": 2.32522, "10430": 2.35941, "10435": 2.37129, "10440": 2.3711, "10445": 2.35949, "10450": 2.36154, "10455": 2.38113, "10460": 2.38064, "10465": 2.30273, "10470": 2.3577, "10475": 2.37958, "10480": 2.36276, "10485": 2.36137, "10490": 2.41283, "10495": 2.36502, "10500": 2.36277, "10505": 2.37018, "10510": 2.38172, "10515": 2.37393, "10520": 2.40259, "10525": 2.39024, "10530": 2.39211, "10535": 2.35551, "10540": 2.40461, "10545": 2.35856, "10550": 2.37752, "10555": 2.35793, "10560": 2.34025, "10565": 2.37346, "10570": 2.37536, "10575": 2.3535, "10580": 2.37788, "10585": 2.36682, "10590": 2.37817, "10595": 2.37713, "10600": 2.33146, "10605": 2.3724, "10610": 2.36498, "10615": 2.36379, "10620": 2.34659, "10625": 2.41843, "10630": 2.36855, "10635": 2.32266, "10640": 2.36413, "10645": 2.42158, "10650": 2.36174, "10655": 2.30869, "10660": 2.34689, "10665": 2.39981, "10670": 2.31617, "10675": 2.41612, "10680": 2.35445, "10685": 2.28871, "10690": 2.38456, "10695": 2.33038, "10700": 2.38407, "10705": 2.38432, "10710": 2.34313, "10715": 2.3828, "10720": 2.32518, "10725": 2.35278, "10730": 2.34872, "10735": 2.35338, "10740": 2.31849, "10745": 2.33808, "10750": 2.33362, "10755": 2.4041, "10760": 2.36431, "10765": 2.33591, "10770": 2.36802, "10775": 2.38746, "10780": 2.36985, "10785": 2.39167, "10790": 2.34599, "10795": 2.38556, "10800": 2.32491, "10805": 2.39755, "10810": 2.37536, "10815": 2.35431, "10820": 2.34323, "10825": 2.37192, "10830": 2.33781, "10835": 2.3477, "10840": 2.32993, "10845": 2.38645, "10850": 2.33282, "10855": 2.36654, "10860": 2.33304, "10865": 2.32192, "10870": 2.32311, "10875": 2.30406, "10880": 2.39356, "10885": 2.40455, "10890": 2.36115, "10895": 2.37301, "10900": 2.33176, "10905": 2.31266, "10910": 2.40728, "10915": 2.37119, "10920": 2.37413, "10925": 2.36306, "10930": 2.31881, "10935": 2.36035, "10940": 2.35501, "10945": 2.34689, "10950": 2.36286, "10955": 2.3644, "10960": 2.30987, "10965": 2.3635, "10970": 2.35624, "10975": 2.40775, "10980": 2.37303, "10985": 2.3427, "10990": 2.39729, "10995": 2.36387, "11000": 2.33714, "11005": 2.36117, "11010": 2.34243, "11015": 2.32557, "11020": 2.3346, "11025": 2.36577, "11030": 2.34044, "11035": 2.31307, "11040": 2.31887, "11045": 2.31738, "11050": 2.31805, "11055": 2.28859, "11060": 2.33998, "11065": 2.31013, "11070": 2.39402, "11075": 2.32015, "11080": 2.35427, "11085": 2.33669, "11090": 2.34632, "11095": 2.37084, "11100": 2.32912, "11105": 2.31663, "11110": 2.36288, "11115": 2.37225, "11120": 2.38139, "11125": 2.31341, "11130": 2.34997, "11135": 2.3336, "11140": 2.37217, "11145": 2.35107, "11150": 2.39612, "11155": 2.34114, "11160": 2.3659, "11165": 2.36388, "11170": 2.34098, "11175": 2.33474, "11180": 2.37348, "11185": 2.31203, "11190": 2.27804, "11195": 2.32819, "11200": 2.34726, "11205": 2.36258, "11210": 2.33385, "11215": 2.31927, "11220": 2.34329, "11225": 2.37141, "11230": 2.36569, "11235": 2.32069, "11240": 2.34092, "11245": 2.35748, "11250": 2.3324, "11255": 2.33515, "11260": 2.35577, "11265": 2.38918, "11270": 2.28782, "11275": 2.31519, "11280": 2.36893, "11285": 2.29387, "11290": 2.34639, "11295": 2.3655, "11300": 2.38111, "11305": 2.33495, "11310": 2.32963, "11315": 2.29825, "11320": 2.30482, "11325": 2.31462, "11330": 2.35421, "11335": 2.33831, "11340": 2.30841, "11345": 2.31278, "11350": 2.29588, "11355": 2.3219, "11360": 2.35153, "11365": 2.29378, "11370": 2.35263, "11375": 2.32804, "11380": 2.34006, "11385": 2.34763, "11390": 2.33477, "11395": 2.28732, "11400": 2.30981, "11405": 2.35647, "11410": 2.35502, "11415": 2.38458, "11420": 2.35172, "11425": 2.30761, "11430": 2.36718, "11435": 2.36201, "11440": 2.34796, "11445": 2.36318, "11450": 2.32182, "11455": 2.30476, "11460": 2.35092, "11465": 2.34386, "11470": 2.37434, "11475": 2.31342, "11480": 2.32527, "11485": 2.30987, "11490": 2.34568, "11495": 2.406, "11500": 2.33937, "11505": 2.35014, "11510": 2.36223, "11515": 2.32176, "11520": 2.30507, "11525": 2.36152, "11530": 2.31469, "11535": 2.32196, "11540": 2.34627, "11545": 2.34321, "11550": 2.36438, "11555": 2.32533, "11560": 2.34981, "11565": 2.34125, "11570": 2.34916, "11575": 2.29628, "11580": 2.32931, "11585": 2.35173, "11590": 2.36158, "11595": 2.33454, "11600": 2.35704, "11605": 2.3235, "11610": 2.36089, "11615": 2.35899, "11620": 2.29569, "11625": 2.2757, "11630": 2.32782, "11635": 2.34204, "11640": 2.30488, "11645": 2.30751, "11650": 2.32628, "11655": 2.35114, "11660": 2.33566, "11665": 2.32994, "11670": 2.30002, "11675": 2.29666, "11680": 2.32542, "11685": 2.33637, "11690": 2.34433, "11695": 2.31688, "11700": 2.32535, "11705": 2.3009, "11710": 2.34479, "11715": 2.31575, "11720": 2.29844, "11725": 2.33988, "11730": 2.30403, "11735": 2.32822, "11740": 2.27122, "11745": 2.31714, "11750": 2.32793, "11755": 2.35133, "11760": 2.31357, "11765": 2.3378, "11770": 2.27597, "11775": 2.32591, "11780": 2.25511, "11785": 2.2973, "11790": 2.31403, "11795": 2.32024, "11800": 2.3345, "11805": 2.30403, "11810": 2.30398, "11815": 2.33078, "11820": 2.32015, "11825": 2.36083, "11830": 2.31663, "11835": 2.33741, "11840": 2.34081, "11845": 2.31727, "11850": 2.30496, "11855": 2.31403, "11860": 2.34333, "11865": 2.35836, "11870": 2.37861, "11875": 2.28155, "11880": 2.29163, "11885": 2.33553, "11890": 2.29241, "11895": 2.29059, "11900": 2.33401, "11905": 2.31769, "11910": 2.27783, "11915": 2.31082, "11920": 2.33519, "11925": 2.30272, "11930": 2.30681, "11935": 2.31569, "11940": 2.3175, "11945": 2.34208, "11950": 2.29773, "11955": 2.31327, "11960": 2.33576, "11965": 2.29584, "11970": 2.28204, "11975": 2.33575, "11980": 2.30612, "11985": 2.2776, "11990": 2.30416, "11995": 2.33013, "12000": 2.32323, "12005": 2.32565, "12010": 2.2884, "12015": 2.30861, "12020": 2.32922, "12025": 2.33525, "12030": 2.31246, "12035": 2.33617, "12040": 2.3154, "12045": 2.3126, "12050": 2.30835, "12055": 2.33352, "12060": 2.29764, "12065": 2.32975, "12070": 2.30319, "12075": 2.2775, "12080": 2.35063, "12085": 2.33812, "12090": 2.33359, "12095": 2.28176, "12100": 2.31543, "12105": 2.30903, "12110": 2.33029, "12115": 2.3036, "12120": 2.30606, "12125": 2.29484, "12130": 2.30409, "12135": 2.32842, "12140": 2.29591, "12145": 2.25622, "12150": 2.26125, "12155": 2.34249, "12160": 2.35771, "12165": 2.31914, "12170": 2.3336, "12175": 2.3412, "12180": 2.33054, "12185": 2.34135, "12190": 2.33375, "12195": 2.29767, "12200": 2.30036, "12205": 2.32225, "12210": 2.35697, "12215": 2.30437, "12220": 2.2987, "12225": 2.24241, "12230": 2.33348, "12235": 2.33945, "12240": 2.32345, "12245": 2.28764, "12250": 2.27397, "12255": 2.33706, "12260": 2.31368, "12265": 2.34287, "12270": 2.31292, "12275": 2.31361, "12280": 2.31869, "12285": 2.28631, "12290": 2.31074, "12295": 2.26654, "12300": 2.32931, "12305": 2.26821, "12310": 2.28768, "12315": 2.3543, "12320": 2.2963, "12325": 2.32045, "12330": 2.30113, "12335": 2.3194, "12340": 2.34117, "12345": 2.36885, "12350": 2.34318, "12355": 2.30683, "12360": 2.31344, "12365": 2.32933, "12370": 2.29273, "12375": 2.29957, "12380": 2.29184, "12385": 2.29061, "12390": 2.25018, "12395": 2.30421, "12400": 2.29905, "12405": 2.31088, "12410": 2.30419, "12415": 2.28306, "12420": 2.31729, "12425": 2.30099, "12430": 2.31571, "12435": 2.30048, "12440": 2.33123, "12445": 2.3202, "12450": 2.30745, "12455": 2.24018, "12460": 2.33488, "12465": 2.36363, "12470": 2.27626, "12475": 2.27276, "12480": 2.29139, "12485": 2.30632, "12490": 2.33128, "12495": 2.26961, "12500": 2.32122, "12505": 2.3351, "12510": 2.35582, "12515": 2.27062, "12520": 2.31971, "12525": 2.28653, "12530": 2.32054, "12535": 2.27138, "12540": 2.28491, "12545": 2.29049, "12550": 2.31572, "12555": 2.32333, "12560": 2.30023, "12565": 2.3353, "12570": 2.27829, "12575": 2.29941, "12580": 2.31153, "12585": 2.29201, "12590": 2.33455, "12595": 2.3227, "12600": 2.28167, "12605": 2.31996, "12610": 2.3631, "12615": 2.30567, "12620": 2.33322, "12625": 2.32935, "12630": 2.29885, "12635": 2.33561, "12640": 2.29568, "12645": 2.27902, "12650": 2.32556, "12655": 2.2647, "12660": 2.34199, "12665": 2.31843, "12670": 2.3097, "12675": 2.31886, "12680": 2.27525, "12685": 2.3664, "12690": 2.30452, "12695": 2.33199, "12700": 2.29244, "12705": 2.30628, "12710": 2.30837, "12715": 2.28749, "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}, "num-zeros": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": 956236544.0, "5": 967337600.0, "10": 971388224.0, "15": 946439424.0, "20": 961330240.0, "25": 1083876480.0, "30": 1211133312.0, "35": 1297707520.0, "40": 1271785728.0, "45": 1175048064.0, "50": 1126729728.0, "55": 1083975424.0, "60": 1045060608.0, "65": 1026047360.0, "70": 995721280.0, "75": 986257152.0, "80": 1010241664.0, "85": 1006739968.0, "90": 988780736.0, "95": 959700032.0, "100": 971861632.0, "105": 980754624.0, "110": 977222528.0, "115": 978430848.0, "120": 961162432.0, "125": 942469184.0, "130": 977095104.0, "135": 966160128.0, "140": 963476928.0, "145": 976512384.0, "150": 921597184.0, "155": 968134336.0, "160": 956383232.0, "165": 959869952.0, "170": 974372224.0, "175": 949013120.0, "180": 946688448.0, "185": 972006784.0, "190": 969055488.0, "195": 985121664.0, "200": 945774592.0, "205": 958353792.0, "210": 979445248.0, "215": 967478208.0, "220": 956423424.0, "225": 962400768.0, "230": 948177792.0, "235": 965221120.0, "240": 966072192.0, "245": 969161216.0, "250": 974435968.0, "255": 925063296.0, "260": 965635968.0, "265": 970660352.0, "270": 959131264.0, "275": 954001216.0, "280": 963427648.0, "285": 945777408.0, "290": 974124544.0, "295": 966704640.0, "300": 967140096.0, "305": 964514048.0, "310": 940354048.0, "315": 967404800.0, "320": 969006080.0, "325": 980552832.0, "330": 972090752.0, "335": 946865984.0, "340": 966598784.0, "345": 973025856.0, "350": 973918720.0, "355": 963261696.0, "360": 948351680.0, "365": 964821248.0, "370": 962952704.0, "375": 958446848.0, "380": 947153280.0, "385": 955988608.0, "390": 945399616.0, "395": 970423552.0, "400": 979770112.0, "405": 968344320.0, "410": 970058752.0, "415": 953158528.0, "420": 943569920.0, "425": 954774144.0, "430": 962663232.0, "435": 977082240.0, "440": 954811392.0, "445": 971894272.0, "450": 963512576.0, "455": 973134720.0, "460": 983714688.0, "465": 945280512.0, "470": 942055616.0, "475": 967007104.0, "480": 966107264.0, "485": 976414528.0, "490": 962538880.0, "495": 945454464.0, "500": 964454656.0, "505": 986005440.0, "510": 965682944.0, "515": 943411584.0, "520": 945017408.0, "525": 971262208.0, "530": 971890688.0, "535": 979140352.0, "540": 969531264.0, "545": 954116608.0, "550": 951267584.0, "555": 987219456.0, "560": 960428288.0, "565": 966616320.0, "570": 975727488.0, "575": 927224960.0, "580": 970694528.0, "585": 961176064.0, "590": 972967040.0, "595": 963682816.0, "600": 937079168.0, "605": 951470208.0, "610": 963360768.0, "615": 970009728.0, "620": 976472192.0, "625": 949579776.0, "630": 954445504.0, "635": 986042816.0, "640": 980980992.0, "645": 955010560.0, "650": 958545664.0, "655": 951656640.0, "660": 961043712.0, "665": 967552000.0, "670": 962514304.0, "675": 968337536.0, "680": 965619200.0, "685": 962871040.0, "690": 961921088.0, "695": 954770368.0, "700": 970340608.0, "705": 945512640.0, "710": 943885440.0, "715": 973357568.0, "720": 968369856.0, "725": 978489984.0, "730": 952195008.0, "735": 948813952.0, "740": 955633408.0, "745": 975866880.0, "750": 981235072.0, "755": 962156608.0, "760": 951964800.0, "765": 967343616.0, "770": 976148096.0, "775": 970544000.0, "780": 977540928.0, "785": 931529024.0, "790": 960441536.0, "795": 964582016.0, "800": 967022848.0, "805": 962321024.0, "810": 940969344.0, "815": 949037568.0, "820": 953181440.0, "825": 954502400.0, "830": 976442240.0, "835": 956073344.0, "840": 948401920.0, "845": 965153024.0, "850": 966029248.0, "855": 960904384.0, "860": 976027200.0, "865": 938157824.0, "870": 966414016.0, "875": 972314880.0, "880": 963120896.0, "885": 967745600.0, "890": 949967872.0, "895": 960019072.0, "900": 974229696.0, "905": 963968256.0, "910": 958435072.0, "915": 956354560.0, "920": 943974592.0, "925": 960833728.0, "930": 978845952.0, "935": 971073664.0, "940": 960905792.0, "945": 945063040.0, "950": 957423360.0, "955": 979035520.0, "960": 983589248.0, "965": 966165824.0, "970": 951228672.0, "975": 961577344.0, "980": 968071040.0, "985": 968991872.0, "990": 984393024.0, "995": 953291264.0, "1000": 934780480.0, "1005": 960147328.0, "1010": 971538624.0, "1015": 985184896.0, "1020": 962780928.0, "1025": 935009408.0, "1030": 974679936.0, "1035": 964992384.0, "1040": 980464256.0, "1045": 960826496.0, "1050": 955197824.0, "1055": 957780352.0, "1060": 967748800.0, "1065": 967116352.0, "1070": 966600064.0, "1075": 950061696.0, "1080": 954508544.0, "1085": 967251712.0, "1090": 977132800.0, "1095": 961237632.0, "1100": 979613568.0, "1105": 953365120.0, "1110": 965954176.0, "1115": 966986944.0, "1120": 970350592.0, "1125": 965707776.0, "1130": 954942400.0, "1135": 965843328.0, "1140": 965176384.0, "1145": 970988224.0, "1150": 955556864.0, "1155": 930578432.0, "1160": 957774208.0, "1165": 978124736.0, "1170": 974299520.0, "1175": 973059648.0, "1180": 973083648.0, "1185": 947344640.0, "1190": 964793216.0, "1195": 953138560.0, "1200": 972843136.0, "1205": 988478656.0, "1210": 931126784.0, "1215": 968647040.0, "1220": 969160960.0, "1225": 975950656.0, "1230": 967331712.0, "1235": 943446912.0, "1240": 955853952.0, "1245": 981503488.0, "1250": 966111808.0, "1255": 973676032.0, "1260": 946497280.0, "1265": 963997824.0, "1270": 960489024.0, "1275": 973615104.0, "1280": 961112576.0, "1285": 957580480.0, "1290": 952528768.0, "1295": 971610240.0, "1300": 968862464.0, "1305": 963739136.0, "1310": 963334656.0, "1315": 943553408.0, "1320": 966307200.0, "1325": 989784960.0, "1330": 969508992.0, "1335": 972302464.0, "1340": 972269440.0, "1345": 960658304.0, "1350": 968639296.0, "1355": 955853312.0, "1360": 971822144.0, "1365": 960387584.0, "1370": 948791872.0, "1375": 973533376.0, "1380": 953470208.0, "1385": 969146880.0, "1390": 975720640.0, "1395": 931673984.0, "1400": 945854848.0, "1405": 976753536.0, "1410": 974510336.0, "1415": 967573760.0, "1420": 966747328.0, "1425": 937378560.0, "1430": 973916608.0, "1435": 978335552.0, "1440": 964178304.0, "1445": 958058240.0, "1450": 946147712.0, "1455": 983922304.0, "1460": 968651136.0, "1465": 948745152.0, "1470": 984243328.0, "1475": 943906048.0, "1480": 963975488.0, "1485": 957349376.0, "1490": 961261888.0, "1495": 980539648.0, "1500": 958332032.0, "1505": 942866816.0, "1510": 984180096.0, "1515": 959094528.0, "1520": 959105408.0, "1525": 952786816.0, "1530": 957741312.0, "1535": 949428928.0, "1540": 971088256.0, "1545": 963132352.0, "1550": 978666752.0, "1555": 952320512.0, "1560": 980089984.0, "1565": 967314048.0, "1570": 973844352.0, "1575": 975494912.0, "1580": 941862656.0, "1585": 970028352.0, "1590": 983822208.0, "1595": 948631616.0, "1600": 967442560.0, "1605": 952451328.0, "1610": 969616512.0, "1615": 983146496.0, "1620": 968019200.0, "1625": 970715776.0, "1630": 962887360.0, "1635": 942311936.0, "1640": 981612224.0, "1645": 973977856.0, "1650": 974188224.0, "1655": 967265024.0, "1660": 940687744.0, "1665": 961704448.0, "1670": 962902016.0, "1675": 971280896.0, "1680": 980879232.0, "1685": 944416192.0, "1690": 964688128.0, "1695": 965644992.0, "1700": 966342336.0, "1705": 985200000.0, "1710": 978354304.0, "1715": 943210880.0, "1720": 977089408.0, "1725": 965870208.0, "1730": 968968960.0, "1735": 965088000.0, "1740": 949713280.0, "1745": 970012352.0, "1750": 959681728.0, "1755": 960085440.0, "1760": 966381376.0, "1765": 951816192.0, "1770": 954665728.0, "1775": 973752064.0, "1780": 970534272.0, "1785": 968824960.0, "1790": 950235520.0, "1795": 945131072.0, "1800": 984666816.0, "1805": 987163520.0, "1810": 977766656.0, "1815": 948004480.0, "1820": 949209216.0, "1825": 978853632.0, "1830": 966362368.0, "1835": 964133632.0, "1840": 972320128.0, "1845": 935415808.0, "1850": 952497792.0, "1855": 980048640.0, "1860": 975866880.0, "1865": 958966528.0, "1870": 958949056.0, "1875": 932593408.0, "1880": 973574016.0, "1885": 978843264.0, "1890": 971358720.0, "1895": 959212288.0, "1900": 947394432.0, "1905": 981829952.0, "1910": 969126912.0, "1915": 970040704.0, "1920": 975597056.0, "1925": 960496512.0, "1930": 977922304.0, "1935": 963250432.0, "1940": 952460928.0, "1945": 981338176.0, "1950": 939172864.0, "1955": 960604416.0, "1960": 970031744.0, "1965": 981176000.0, "1970": 962045120.0, "1975": 952822016.0, "1980": 936847360.0, "1985": 975938432.0, "1990": 965965696.0, "1995": 962609920.0, "2000": 960553984.0, "2005": 954497728.0, "2010": 975579776.0, "2015": 991802112.0, "2020": 975433408.0, "2025": 974303936.0, "2030": 952084736.0, "2035": 967847680.0, "2040": 987457536.0, "2045": 976480064.0, "2050": 984702464.0, "2055": 942839488.0, "2060": 942593920.0, "2065": 966208768.0, "2070": 969622528.0, "2075": 980553536.0, "2080": 977598080.0, "2085": 939635968.0, "2090": 969872256.0, "2095": 961274880.0, "2100": 976719168.0, "2105": 972537920.0, "2110": 959901568.0, "2115": 956875264.0, "2120": 977482304.0, "2125": 962566784.0, "2130": 979618496.0, "2135": 950537408.0, "2140": 946996544.0, "2145": 962273920.0, "2150": 973404416.0, "2155": 972690944.0, "2160": 970314560.0, "2165": 948644160.0, "2170": 961541696.0, "2175": 969377216.0, "2180": 969329920.0, "2185": 947446592.0, "2190": 940480960.0, "2195": 986085952.0, "2200": 961861248.0, "2205": 978924672.0, "2210": 964100864.0, "2215": 963502336.0, "2220": 951311104.0, "2225": 969315776.0, "2230": 976331328.0, "2235": 974025920.0, "2240": 975493888.0, "2245": 960230784.0, "2250": 967640192.0, "2255": 969129984.0, "2260": 975065024.0, "2265": 968258688.0, "2270": 951744768.0, "2275": 962766848.0, "2280": 969640064.0, "2285": 971692992.0, "2290": 962889344.0, "2295": 931409280.0, "2300": 959906048.0, "2305": 970426560.0, "2310": 967444864.0, "2315": 970905792.0, "2320": 975590848.0, "2325": 938587264.0, "2330": 988438528.0, "2335": 977489408.0, "2340": 964596352.0, "2345": 964166528.0, "2350": 947555712.0, "2355": 977029568.0, "2360": 966899072.0, "2365": 977297728.0, "2370": 965072640.0, "2375": 953966272.0, "2380": 962918912.0, "2385": 967194496.0, "2390": 963077248.0, "2395": 974465792.0, "2400": 958410816.0, "2405": 968119552.0, "2410": 951586112.0, "2415": 965904256.0, "2420": 966516160.0, "2425": 959045632.0, "2430": 956685952.0, "2435": 961389184.0, "2440": 959755904.0, "2445": 970891392.0, "2450": 961996736.0, "2455": 922721216.0, "2460": 951953536.0, "2465": 955730432.0, "2470": 972570496.0, "2475": 973812992.0, "2480": 943895296.0, "2485": 944184064.0, "2490": 972411136.0, "2495": 974451712.0, "2500": 973910080.0, "2505": 958492032.0, "2510": 939510912.0, "2515": 979553728.0, "2520": 970473792.0, "2525": 964390784.0, "2530": 955799168.0, "2535": 936598144.0, "2540": 969027648.0, "2545": 970385024.0, "2550": 969462528.0, "2555": 969439040.0, "2560": 964978304.0, "2565": 959763712.0, "2570": 985176704.0, "2575": 957426112.0, "2580": 967424512.0, "2585": 966022400.0, "2590": 956354944.0, "2595": 981830400.0, "2600": 959530880.0, "2605": 962999168.0, "2610": 965972864.0, "2615": 951924992.0, "2620": 971241216.0, "2625": 976456064.0, "2630": 974409984.0, "2635": 948071296.0, "2640": 948137088.0, "2645": 963036736.0, "2650": 953982912.0, "2655": 977112448.0, "2660": 949622976.0, "2665": 953929024.0, "2670": 959064064.0, "2675": 979275904.0, "2680": 961395840.0, "2685": 970701952.0, "2690": 965222016.0, "2695": 943553536.0, "2700": 969425664.0, "2705": 978961792.0, "2710": 971810560.0, "2715": 990813952.0, "2720": 942649152.0, "2725": 967955328.0, "2730": 955466496.0, "2735": 970672704.0, "2740": 977921536.0, "2745": 932280000.0, "2750": 947856384.0, "2755": 956317184.0, "2760": 981697664.0, "2765": 966112192.0, "2770": 948914688.0, "2775": 935830272.0, "2780": 964777088.0, "2785": 969570176.0, "2790": 974273152.0, "2795": 966886144.0, "2800": 944388096.0, "2805": 964353920.0, "2810": 969610752.0, "2815": 975845248.0, "2820": 963081600.0, "2825": 937627392.0, "2830": 956738368.0, "2835": 986321024.0, "2840": 961756672.0, "2845": 967505920.0, "2850": 951714816.0, "2855": 962091520.0, "2860": 954242304.0, "2865": 955881216.0, "2870": 944662848.0, "2875": 974662784.0, "2880": 968199936.0, "2885": 981081984.0, "2890": 953454080.0, "2895": 957178304.0, "2900": 964989440.0, "2905": 931707648.0, "2910": 955730688.0, "2915": 979477120.0, "2920": 970492544.0, "2925": 964975680.0, "2930": 964046592.0, "2935": 940140416.0, "2940": 964912384.0, "2945": 989149952.0, "2950": 965209088.0, "2955": 965104256.0, "2960": 933161472.0, "2965": 968794496.0, "2970": 973034688.0, "2975": 958092288.0, "2980": 964497280.0, "2985": 937267584.0, "2990": 951254720.0, "2995": 978315264.0, "3000": 969275392.0, "3005": 974686080.0, "3010": 950235008.0, "3015": 943841024.0, "3020": 958440960.0, "3025": 975186560.0, "3030": 965018624.0, "3035": 963454464.0, "3040": 952131776.0, "3045": 989793472.0, "3050": 965545728.0, "3055": 982520704.0, "3060": 971227712.0, "3065": 943915648.0, "3070": 978409024.0, "3075": 975204992.0, "3080": 960992896.0, "3085": 962352128.0, "3090": 945953664.0, "3095": 938116032.0, "3100": 972929088.0, "3105": 961989120.0, "3110": 970657152.0, "3115": 963390528.0, "3120": 947116032.0, "3125": 972720640.0, "3130": 952973312.0, "3135": 966041920.0, "3140": 968488896.0, "3145": 937852288.0, "3150": 975009280.0, "3155": 976815488.0, "3160": 969629184.0, "3165": 982194944.0, "3170": 937959936.0, "3175": 953825280.0, "3180": 983810176.0, "3185": 965169536.0, "3190": 968482880.0, "3195": 950933888.0, "3200": 945101440.0, "3205": 959863232.0, "3210": 957486336.0, "3215": 958020096.0, "3220": 968129792.0, "3225": 935614144.0, "3230": 962588672.0, "3235": 975775488.0, "3240": 962621440.0, "3245": 981274368.0, "3250": 943260544.0, "3255": 954599424.0, "3260": 980363648.0, "3265": 963619968.0, "3270": 965162624.0, "3275": 959731072.0, "3280": 967046720.0, "3285": 982478144.0, "3290": 947689408.0, "3295": 966419840.0, "3300": 959165632.0, "3305": 949131008.0, "3310": 979510144.0, "3315": 964283008.0, "3320": 969207296.0, "3325": 956193280.0, "3330": 941167104.0, "3335": 964973184.0, "3340": 956900736.0, "3345": 972500224.0, "3350": 964576192.0, "3355": 943346176.0, "3360": 970037632.0, "3365": 969453952.0, "3370": 954774976.0, "3375": 958676288.0, "3380": 971463168.0, "3385": 947973376.0, "3390": 965793024.0, "3395": 978390016.0, "3400": 978127360.0, "3405": 976724032.0, "3410": 924193664.0, "3415": 955424256.0, "3420": 971820416.0, "3425": 977167488.0, "3430": 973837696.0, "3435": 936071680.0, "3440": 970502464.0, "3445": 957303232.0, "3450": 959840896.0, "3455": 963864256.0, "3460": 967881856.0, "3465": 931318976.0, "3470": 952348224.0, "3475": 973704384.0, "3480": 959738112.0, "3485": 979960640.0, "3490": 944670592.0, "3495": 953904576.0, "3500": 969330432.0, "3505": 964348416.0, "3510": 971222656.0, "3515": 955946112.0, "3520": 958733120.0, "3525": 971914240.0, "3530": 964124672.0, "3535": 983192192.0, "3540": 937489664.0, "3545": 944730496.0, "3550": 984462784.0, "3555": 978057984.0, "3560": 974372992.0, "3565": 968800128.0, "3570": 946694528.0, "3575": 976104640.0, "3580": 977494272.0, "3585": 954568832.0, "3590": 956425856.0, "3595": 951444800.0, "3600": 988996800.0, "3605": 962008448.0, "3610": 965055104.0, "3615": 974635648.0, "3620": 954888384.0, "3625": 939515392.0, "3630": 990148736.0, "3635": 971435712.0, "3640": 976025856.0, "3645": 961489664.0, "3650": 945804160.0, "3655": 965782784.0, "3660": 976208000.0, "3665": 964021248.0, "3670": 977431936.0, "3675": 943476096.0, "3680": 958185152.0, "3685": 964285312.0, "3690": 982093952.0, "3695": 963125248.0, "3700": 950570176.0, "3705": 947345792.0, "3710": 982355328.0, "3715": 972682240.0, "3720": 976138624.0, "3725": 964038272.0, "3730": 948851328.0, "3735": 967079424.0, "3740": 960973568.0, "3745": 969334272.0, "3750": 963949056.0, "3755": 953432832.0, "3760": 976641920.0, "3765": 979841280.0, "3770": 972360320.0, "3775": 972374720.0, "3780": 952585664.0, "3785": 960236800.0, "3790": 985598848.0, "3795": 969195392.0, "3800": 957877504.0, "3805": 972408192.0, "3810": 954517952.0, "3815": 974566528.0, "3820": 963019648.0, "3825": 962070528.0, "3830": 969394304.0, "3835": 934744960.0, "3840": 971255808.0, "3845": 986849792.0, "3850": 968873536.0, "3855": 965248640.0, "3860": 948057600.0, "3865": 975049216.0, "3870": 985098432.0, "3875": 983043072.0, "3880": 963615616.0, "3885": 953012224.0, "3890": 960296960.0, "3895": 960591616.0, "3900": 984926464.0, "3905": 976222592.0, "3910": 987360512.0, "3915": 946017792.0, "3920": 974867328.0, "3925": 961248384.0, "3930": 976790208.0, "3935": 978925824.0, "3940": 950292672.0, "3945": 960260864.0, "3950": 974185152.0, "3955": 972967808.0, "3960": 974078848.0, "3965": 950861696.0, "3970": 980692992.0, "3975": 960750208.0, "3980": 977519808.0, "3985": 962949440.0, "3990": 972755328.0, "3995": 953714176.0, "4000": 975011968.0, "4005": 971658752.0, "4010": 978420992.0, "4015": 971491968.0, "4020": 950311040.0, "4025": 968433024.0, "4030": 997935360.0, "4035": 978548864.0, "4040": 959813248.0, "4045": 939664448.0, "4050": 944718720.0, "4055": 980958720.0, "4060": 977682816.0, "4065": 975706624.0, "4070": 942152832.0, "4075": 945765376.0, "4080": 988765632.0, "4085": 962079872.0, "4090": 983356736.0, "4095": 986958720.0, "4100": 957224448.0, "4105": 954073984.0, "4110": 966488192.0, "4115": 976042432.0, "4120": 983531776.0, "4125": 960054336.0, "4130": 967278976.0, "4135": 971431296.0, "4140": 963171200.0, "4145": 956175616.0, "4150": 960332672.0, "4155": 946216576.0, "4160": 968451328.0, "4165": 970327424.0, "4170": 971946880.0, "4175": 955862272.0, "4180": 940997632.0, "4185": 968318592.0, "4190": 968003712.0, "4195": 989223168.0, "4200": 962678912.0, "4205": 960595072.0, "4210": 971851136.0, "4215": 974145536.0, "4220": 981144704.0, "4225": 975196160.0, "4230": 952703104.0, "4235": 958493312.0, "4240": 966824448.0, "4245": 961756800.0, "4250": 965845632.0, "4255": 958268032.0, "4260": 949545344.0, "4265": 964137856.0, "4270": 978408896.0, "4275": 975397504.0, "4280": 962672064.0, "4285": 951451136.0, "4290": 980029952.0, "4295": 968842816.0, "4300": 958299456.0, "4305": 966802048.0, "4310": 939613376.0, "4315": 949416704.0, "4320": 984522240.0, "4325": 982547712.0, "4330": 974746240.0, "4335": 949459840.0, "4340": 959478400.0, "4345": 956644096.0, "4350": 979960320.0, "4355": 968869888.0, "4360": 966313728.0, "4365": 941208704.0, "4370": 969495552.0, "4375": 972982336.0, "4380": 966147840.0, "4385": 972060544.0, "4390": 954141120.0, "4395": 951856512.0, "4400": 973629184.0, "4405": 972441728.0, "4410": 967908480.0, "4415": 958935232.0, "4420": 960777856.0, "4425": 976416000.0, "4430": 965905664.0, "4435": 975864704.0, "4440": 962282496.0, "4445": 954799872.0, "4450": 978294912.0, "4455": 960203776.0, "4460": 968579008.0, "4465": 968805760.0, "4470": 944127232.0, "4475": 951973056.0, "4480": 978824832.0, "4485": 968265728.0, "4490": 956975616.0, "4495": 938925248.0, "4500": 953215488.0, "4505": 977129344.0, "4510": 978700416.0, "4515": 962493568.0, "4520": 958825024.0, "4525": 958101760.0, "4530": 964428608.0, "4535": 976641024.0, "4540": 976846784.0, "4545": 970169408.0, "4550": 953223680.0, "4555": 959591040.0, "4560": 972497920.0, "4565": 973442560.0, "4570": 978909568.0, "4575": 957760000.0, "4580": 963142016.0, "4585": 957381888.0, "4590": 986530176.0, "4595": 960189824.0, "4600": 952267648.0, "4605": 959297664.0, "4610": 963698304.0, "4615": 957950912.0, "4620": 960255936.0, "4625": 973786624.0, "4630": 944507648.0, "4635": 977024896.0, "4640": 960310272.0, "4645": 981993856.0, "4650": 962400128.0, "4655": 939457792.0, "4660": 964001280.0, "4665": 962540544.0, "4670": 976671872.0, "4675": 963514432.0, "4680": 957488448.0, "4685": 949674432.0, "4690": 956962240.0, "4695": 969926912.0, "4700": 961214016.0, "4705": 970781312.0, "4710": 934493248.0, "4715": 970475008.0, "4720": 966361920.0, "4725": 980303616.0, "4730": 965826240.0, "4735": 937871360.0, "4740": 960124288.0, "4745": 975997376.0, "4750": 967957760.0, "4755": 984959744.0, "4760": 959123648.0, "4765": 955261056.0, "4770": 958621440.0, "4775": 991092608.0, "4780": 976856000.0, "4785": 967582336.0, "4790": 943756160.0, "4795": 955861760.0, "4800": 967745984.0, "4805": 976565888.0, "4810": 965160448.0, "4815": 957969408.0, "4820": 973993216.0, "4825": 961505920.0, "4830": 962638336.0, "4835": 972543936.0, "4840": 948913920.0, "4845": 965716608.0, "4850": 960305024.0, "4855": 964130624.0, "4860": 963051008.0, "4865": 967532352.0, "4870": 957207424.0, "4875": 983574528.0, "4880": 957085120.0, "4885": 977052480.0, "4890": 959740928.0, "4895": 942152192.0, "4900": 973700352.0, "4905": 975213952.0, "4910": 969220608.0, "4915": 970053248.0, "4920": 941169024.0, "4925": 954809344.0, "4930": 977034624.0, "4935": 963750016.0, "4940": 972586496.0, "4945": 960056448.0, "4950": 940793856.0, "4955": 968036480.0, "4960": 976800640.0, "4965": 961022464.0, "4970": 958665472.0, "4975": 933775168.0, "4980": 960824704.0, "4985": 963012160.0, "4990": 963588736.0, "4995": 986307968.0, "5000": 940776000.0, "5005": 968887552.0, "5010": 970307776.0, "5015": 965233408.0, "5020": 966702336.0, "5025": 949471744.0, "5030": 953490560.0, "5035": 967352704.0, "5040": 955689856.0, "5045": 969129856.0, "5050": 953417984.0, "5055": 954811520.0, "5060": 963029248.0, "5065": 952195072.0, "5070": 973597056.0, "5075": 978581632.0, "5080": 942828736.0, "5085": 965863040.0, "5090": 972857088.0, "5095": 964397952.0, "5100": 958318016.0, "5105": 965353728.0, "5110": 950401920.0, "5115": 972347520.0, "5120": 960424960.0, "5125": 969755008.0, "5130": 938795456.0, "5135": 943659008.0, "5140": 969888896.0, "5145": 968688768.0, "5150": 970601216.0, "5155": 972635008.0, "5160": 926551872.0, "5165": 961591552.0, "5170": 966873472.0, "5175": 966086400.0, "5180": 963656192.0, "5185": 930802688.0, "5190": 949852992.0, "5195": 972422016.0, "5200": 973758656.0, "5205": 968249664.0, "5210": 960527936.0, "5215": 928815680.0, "5220": 979174208.0, "5225": 984780416.0, "5230": 975060672.0, "5235": 975055232.0, "5240": 944314112.0, "5245": 970834048.0, "5250": 972426624.0, "5255": 966895296.0, "5260": 976679744.0, "5265": 942270592.0, "5270": 969202176.0, "5275": 970073344.0, "5280": 962825728.0, "5285": 964072064.0, "5290": 932501824.0, "5295": 951762944.0, "5300": 975588288.0, "5305": 951853504.0, "5310": 968040960.0, "5315": 955817472.0, "5320": 950921984.0, "5325": 973040384.0, "5330": 967843264.0, "5335": 967536384.0, "5340": 966503424.0, "5345": 962961344.0, "5350": 978937216.0, "5355": 972202560.0, "5360": 963892288.0, "5365": 965246464.0, "5370": 947824896.0, "5375": 948820864.0, "5380": 967210752.0, "5385": 980540416.0, "5390": 965312832.0, "5395": 955160000.0, "5400": 948286464.0, "5405": 974365952.0, "5410": 967844224.0, "5415": 976074688.0, "5420": 967431680.0, "5425": 937412096.0, "5430": 963919744.0, "5435": 971948544.0, "5440": 969128192.0, "5445": 957501184.0, "5450": 919431040.0, "5455": 952052800.0, "5460": 962315264.0, "5465": 978905088.0, "5470": 981031104.0, "5475": 941610304.0, "5480": 955755072.0, "5485": 964902912.0, "5490": 976036992.0, "5495": 962805120.0, "5500": 971218240.0, "5505": 957001728.0, "5510": 968610432.0, "5515": 945466624.0, "5520": 963181952.0, "5525": 975932480.0, "5530": 936635520.0, "5535": 970693824.0, "5540": 960295936.0, "5545": 972084800.0, "5550": 967898240.0, "5555": 955968960.0, "5560": 954520832.0, "5565": 968862848.0, "5570": 945186112.0, "5575": 960539584.0, "5580": 960563456.0, "5585": 959470720.0, "5590": 977668224.0, "5595": 975194496.0, "5600": 963009472.0, "5605": 964201216.0, "5610": 943157760.0, "5615": 966515904.0, "5620": 963224448.0, "5625": 982398656.0, "5630": 976073984.0, "5635": 957347520.0, "5640": 951455488.0, "5645": 967805568.0, "5650": 979181056.0, "5655": 983510912.0, "5660": 956493952.0, "5665": 953574656.0, "5670": 966097408.0, "5675": 967701184.0, "5680": 978598848.0, "5685": 962008576.0, "5690": 935908928.0, "5695": 963739648.0, "5700": 952464384.0, "5705": 974525376.0, "5710": 971341376.0, "5715": 946147648.0, "5720": 974991360.0, "5725": 967473664.0, "5730": 978561792.0, "5735": 964979712.0, "5740": 943573056.0, "5745": 971142016.0, "5750": 981951168.0, "5755": 956754944.0, "5760": 963695168.0, "5765": 957707648.0, "5770": 955747584.0, "5775": 970847104.0, "5780": 962811840.0, "5785": 970673664.0, "5790": 974652672.0, "5795": 949802368.0, "5800": 965999232.0, "5805": 968774272.0, "5810": 975986176.0, "5815": 970103936.0, "5820": 936377408.0, "5825": 969266816.0, "5830": 977614464.0, "5835": 974980224.0, "5840": 963061120.0, "5845": 968774464.0, "5850": 942897536.0, "5855": 975999104.0, "5860": 979537600.0, "5865": 978369280.0, "5870": 968714112.0, "5875": 942328320.0, "5880": 964508224.0, "5885": 974806656.0, "5890": 972671104.0, "5895": 965681920.0, "5900": 941482880.0, "5905": 961766528.0, "5910": 958568832.0, "5915": 968174464.0, "5920": 977468032.0, "5925": 959468800.0, "5930": 946750080.0, "5935": 952334656.0, "5940": 977662144.0, "5945": 984818560.0, "5950": 980689536.0, "5955": 935094464.0, "5960": 961735296.0, "5965": 965938176.0, "5970": 970612096.0, "5975": 961933888.0, "5980": 958344832.0, "5985": 964562816.0, "5990": 973676288.0, "5995": 955919488.0, "6000": 955633216.0, "6005": 961366784.0, "6010": 952707072.0, "6015": 974539328.0, "6020": 978265920.0, "6025": 972226688.0, "6030": 955311744.0, "6035": 946954368.0, "6040": 962641920.0, "6045": 983743552.0, "6050": 956515200.0, "6055": 963445888.0, "6060": 945767488.0, "6065": 958443776.0, "6070": 978390272.0, "6075": 977988416.0, "6080": 957530304.0, "6085": 947641408.0, "6090": 953645632.0, "6095": 964770560.0, "6100": 979886720.0, "6105": 971011904.0, "6110": 961816128.0, "6115": 943814016.0, "6120": 968573824.0, "6125": 960703232.0, "6130": 984040320.0, "6135": 960994432.0, "6140": 958720704.0, "6145": 971225728.0, "6150": 968493312.0, "6155": 974930688.0, "6160": 977216128.0, "6165": 952742848.0, "6170": 951129728.0, "6175": 963358592.0, "6180": 969549568.0, "6185": 966320320.0, "6190": 963731264.0, "6195": 947266752.0, "6200": 969378240.0, "6205": 967161728.0, "6210": 959339264.0, "6215": 973147776.0, "6220": 936374912.0, "6225": 978715520.0, "6230": 976146816.0, "6235": 971740992.0, "6240": 966006336.0, "6245": 956249728.0, "6250": 956449920.0, "6255": 973363584.0, "6260": 978804800.0, "6265": 974941952.0, "6270": 958855808.0, "6275": 963713152.0, "6280": 973141120.0, "6285": 966123136.0, "6290": 971040256.0, "6295": 987490560.0, "6300": 947640064.0, "6305": 964795456.0, "6310": 979017216.0, "6315": 978443392.0, "6320": 971724672.0, "6325": 923002624.0, "6330": 959366784.0, "6335": 974940864.0, "6340": 984824576.0, "6345": 966835456.0, "6350": 944574016.0, "6355": 957990272.0, "6360": 972711552.0, "6365": 972207744.0, "6370": 958966784.0, "6375": 967142144.0, "6380": 951399104.0, "6385": 973584896.0, "6390": 965514880.0, "6395": 975032064.0, "6400": 983945472.0, "6405": 944065408.0, "6410": 977178496.0, "6415": 971635776.0, "6420": 956726592.0, "6425": 960937728.0, "6430": 957525120.0, "6435": 960333440.0, "6440": 968713088.0, "6445": 973479168.0, "6450": 974637056.0, "6455": 962142208.0, "6460": 940994496.0, "6465": 974482944.0, "6470": 979911936.0, "6475": 960847808.0, "6480": 967532032.0, "6485": 948559616.0, "6490": 970748032.0, "6495": 988369024.0, "6500": 980468864.0, "6505": 972158336.0, "6510": 951648576.0, "6515": 957565440.0, "6520": 979061952.0, "6525": 978903424.0, "6530": 973271744.0, "6535": 967829056.0, "6540": 950159040.0, "6545": 966294144.0, "6550": 979335168.0, "6555": 967119872.0, "6560": 975391104.0, "6565": 949645696.0, "6570": 952068224.0, "6575": 962553728.0, "6580": 975679424.0, "6585": 979544832.0, "6590": 949212544.0, "6595": 961471616.0, "6600": 961353856.0, "6605": 961755520.0, "6610": 985212480.0, "6615": 959518336.0, "6620": 944576256.0, "6625": 971028736.0, "6630": 971564928.0, "6635": 964103936.0, "6640": 959857152.0, "6645": 951077504.0, "6650": 978674944.0, "6655": 965949440.0, "6660": 968814080.0, "6665": 969002112.0, "6670": 932999424.0, "6675": 970736128.0, "6680": 969016064.0, "6685": 958784384.0, "6690": 956215552.0, "6695": 955745920.0, "6700": 962135936.0, "6705": 979365824.0, "6710": 971098240.0, "6715": 966874944.0, "6720": 974162048.0, "6725": 941768192.0, "6730": 979399488.0, "6735": 994709376.0, "6740": 976356224.0, "6745": 974602752.0, "6750": 939272320.0, "6755": 977649344.0, "6760": 969757888.0, "6765": 978454848.0, "6770": 975513728.0, "6775": 943523520.0, "6780": 947283584.0, "6785": 975371712.0, "6790": 960607104.0, "6795": 976217984.0, "6800": 973344640.0, "6805": 946806016.0, "6810": 958265856.0, "6815": 970929792.0, "6820": 978086528.0, "6825": 969192704.0, "6830": 950435072.0, "6835": 981464192.0, "6840": 983022336.0, "6845": 948763840.0, "6850": 965465152.0, "6855": 954199552.0, "6860": 979123968.0, "6865": 983975808.0, "6870": 964842560.0, "6875": 978847808.0, "6880": 950371200.0, "6885": 958582016.0, "6890": 960484032.0, "6895": 965665280.0, "6900": 985370880.0, "6905": 968478592.0, "6910": 950097088.0, "6915": 971060736.0, "6920": 967166720.0, "6925": 965180672.0, "6930": 964715648.0, "6935": 952122112.0, "6940": 962920704.0, "6945": 986470144.0, "6950": 973350272.0, "6955": 964715136.0, "6960": 940248960.0, "6965": 974503680.0, "6970": 978554240.0, "6975": 985114880.0, "6980": 982851072.0, "6985": 959949376.0, "6990": 945298944.0, "6995": 987557120.0, "7000": 963329344.0, "7005": 962922240.0, "7010": 985144320.0, "7015": 945447424.0, "7020": 982884608.0, "7025": 968840640.0, "7030": 953537472.0, "7035": 982810432.0, "7040": 950520320.0, "7045": 956041600.0, "7050": 960403712.0, "7055": 963929728.0, "7060": 976999040.0, "7065": 968391296.0, "7070": 953547264.0, "7075": 956559360.0, "7080": 969124864.0, "7085": 965868800.0, "7090": 969521920.0, "7095": 960078592.0, "7100": 973651200.0, "7105": 973332672.0, "7110": 970254848.0, "7115": 958633088.0, "7120": 948865536.0, "7125": 963163584.0, "7130": 971421376.0, "7135": 964540096.0, "7140": 961681152.0, "7145": 930416448.0, "7150": 946280064.0, "7155": 991092864.0, "7160": 968442496.0, "7165": 956888320.0, "7170": 968275328.0, "7175": 955828224.0, "7180": 958441536.0, "7185": 984880256.0, "7190": 978735936.0, "7195": 973712000.0, "7200": 935905536.0, "7205": 957525760.0, "7210": 967114624.0, "7215": 969707264.0, "7220": 982219584.0, "7225": 928936768.0, "7230": 949560960.0, "7235": 967251712.0, "7240": 966968064.0, "7245": 967454976.0, "7250": 949502336.0, "7255": 957363968.0, "7260": 970114816.0, "7265": 974961664.0, "7270": 959874240.0, "7275": 959286784.0, "7280": 957121920.0, "7285": 977575808.0, "7290": 977249920.0, "7295": 962901120.0, "7300": 975499904.0, "7305": 964022528.0, "7310": 977515520.0, "7315": 966809600.0, "7320": 974552768.0, "7325": 966906752.0, "7330": 959690880.0, "7335": 964049280.0, "7340": 977556864.0, "7345": 967668224.0, "7350": 984804864.0, "7355": 959674816.0, "7360": 948842240.0, "7365": 972772864.0, "7370": 982593664.0, "7375": 963567424.0, "7380": 964284224.0, "7385": 948615488.0, "7390": 964192512.0, "7395": 958787008.0, "7400": 970242816.0, "7405": 988116736.0, "7410": 952423488.0, "7415": 950935744.0, "7420": 967472640.0, "7425": 982705664.0, "7430": 965871552.0, "7435": 973294080.0, "7440": 937228160.0, "7445": 969074752.0, "7450": 980608832.0, "7455": 971587712.0, "7460": 972749056.0, "7465": 939573760.0, "7470": 972011648.0, "7475": 958500480.0, "7480": 969529792.0, "7485": 961637568.0, "7490": 934760704.0, "7495": 957297216.0, "7500": 969548416.0, "7505": 970380928.0, "7510": 972589184.0, "7515": 979619840.0, "7520": 951796224.0, "7525": 970763840.0, "7530": 954655104.0, "7535": 971887616.0, "7540": 979952832.0, "7545": 959493248.0, "7550": 960599936.0, "7555": 960564352.0, "7560": 970209920.0, "7565": 955267200.0, "7570": 942667904.0, "7575": 966060032.0, "7580": 982740480.0, "7585": 979190784.0, "7590": 970155264.0, "7595": 950188416.0, "7600": 946546432.0, "7605": 982722432.0, "7610": 969487360.0, "7615": 988970624.0, "7620": 957136000.0, "7625": 941577856.0, "7630": 971809152.0, "7635": 984756608.0, "7640": 983937792.0, "7645": 968197120.0, "7650": 959357504.0, "7655": 962682368.0, "7660": 969198976.0, "7665": 978073088.0, "7670": 975279104.0, "7675": 975891840.0, "7680": 943167616.0, "7685": 960475136.0, "7690": 975843968.0, "7695": 982086400.0, "7700": 979923648.0, "7705": 940595776.0, "7710": 974811648.0, "7715": 979946496.0, "7720": 968216448.0, "7725": 960576640.0, "7730": 943583104.0, "7735": 968598400.0, "7740": 980697600.0, "7745": 964667008.0, "7750": 963965568.0, "7755": 960341056.0, "7760": 970667072.0, "7765": 971220096.0, "7770": 962730624.0, "7775": 981588800.0, "7780": 965073280.0, "7785": 959776384.0, "7790": 968147968.0, "7795": 969118208.0, "7800": 971586880.0, "7805": 968712128.0, "7810": 946156608.0, "7815": 963601664.0, "7820": 974369664.0, "7825": 963930944.0, "7830": 957420864.0, "7835": 949820864.0, "7840": 957576448.0, "7845": 954299264.0, "7850": 980140416.0, "7855": 987100288.0, "7860": 947203712.0, "7865": 949597632.0, "7870": 965653760.0, "7875": 976341632.0, "7880": 968749184.0, "7885": 969863296.0, "7890": 951979520.0, "7895": 974744576.0, "7900": 964075264.0, "7905": 964628544.0, "7910": 966224768.0, "7915": 943623808.0, "7920": 951251584.0, "7925": 969683840.0, "7930": 965018496.0, "7935": 984522112.0, "7940": 965260992.0, "7945": 950920512.0, "7950": 961778944.0, "7955": 980819072.0, "7960": 964107328.0, "7965": 952800768.0, "7970": 952096960.0, "7975": 969954944.0, "7980": 965058752.0, "7985": 959497728.0, "7990": 968288768.0, "7995": 947074368.0, "8000": 962595712.0, "8005": 980875264.0, "8010": 965703040.0, "8015": 982795648.0, "8020": 960636544.0, "8025": 965519616.0, "8030": 958643200.0, "8035": 975716096.0, "8040": 960827648.0, "8045": 948395264.0, "8050": 959831808.0, "8055": 979617792.0, "8060": 969592128.0, "8065": 958394752.0, "8070": 964066944.0, "8075": 942266240.0, "8080": 966035328.0, "8085": 966815936.0, "8090": 983700160.0, "8095": 988871424.0, "8100": 966531968.0, "8105": 944438272.0, "8110": 969326016.0, "8115": 985228672.0, "8120": 974833408.0, "8125": 964005120.0, "8130": 966272000.0, "8135": 967624576.0, "8140": 963686848.0, "8145": 994976768.0, "8150": 973166016.0, "8155": 938390528.0, "8160": 964462464.0, "8165": 972803200.0, "8170": 968497280.0, "8175": 961587008.0, "8180": 936029440.0, "8185": 962625536.0, "8190": 967799296.0, "8195": 977385088.0, "8200": 956367296.0, "8205": 960566528.0, "8210": 946495424.0, "8215": 982005248.0, "8220": 988443520.0, "8225": 966243584.0, "8230": 962552576.0, "8235": 934131712.0, "8240": 980267904.0, "8245": 976606848.0, "8250": 964327808.0, "8255": 977492864.0, "8260": 956833664.0, "8265": 982957440.0, "8270": 952836608.0, "8275": 974283968.0, "8280": 974906560.0, "8285": 953985664.0, "8290": 940194816.0, "8295": 981360128.0, "8300": 972952832.0, "8305": 978368320.0, "8310": 951095936.0, "8315": 937922048.0, "8320": 977484544.0, "8325": 967872768.0, "8330": 990116800.0, "8335": 975746048.0, "8340": 947366912.0, "8345": 970641408.0, "8350": 970082176.0, "8355": 975014080.0, "8360": 979651456.0, "8365": 932855680.0, "8370": 965537344.0, "8375": 979732736.0, "8380": 965482496.0, "8385": 972889472.0, "8390": 962502912.0, "8395": 951003840.0, "8400": 972739968.0, "8405": 951808384.0, "8410": 960912000.0, "8415": 965867904.0, "8420": 941925888.0, "8425": 968447872.0, "8430": 961416704.0, "8435": 966249344.0, "8440": 969510272.0, "8445": 952921344.0, "8450": 984742912.0, "8455": 990518400.0, "8460": 969086848.0, "8465": 967798656.0, "8470": 963598464.0, "8475": 942921920.0, "8480": 987605888.0, "8485": 979799936.0, "8490": 991849856.0, "8495": 971815552.0, "8500": 951760768.0, "8505": 982982848.0, "8510": 974371200.0, "8515": 969206912.0, "8520": 961827968.0, "8525": 944996096.0, "8530": 984721152.0, "8535": 978411520.0, "8540": 968342592.0, "8545": 969125440.0, "8550": 942408448.0, "8555": 971549056.0, "8560": 958775296.0, "8565": 975676160.0, "8570": 975305216.0, "8575": 971852992.0, "8580": 932583232.0, "8585": 966065856.0, "8590": 978933760.0, "8595": 979387904.0, "8600": 983792768.0, "8605": 958356416.0, "8610": 984069888.0, "8615": 978067776.0, "8620": 963535168.0, "8625": 979909120.0, "8630": 943580032.0, "8635": 961797632.0, "8640": 973745600.0, "8645": 970784128.0, "8650": 969289152.0, "8655": 970653440.0, "8660": 944484096.0, "8665": 986977728.0, "8670": 960353920.0, "8675": 974610176.0, "8680": 962718976.0, "8685": 956147136.0, "8690": 978612864.0, "8695": 969139072.0, "8700": 973135360.0, "8705": 973914176.0, "8710": 947435776.0, "8715": 973736320.0, "8720": 958622976.0, "8725": 978719488.0, "8730": 985894400.0, "8735": 952583040.0, "8740": 940201728.0, "8745": 987763456.0, "8750": 972207744.0, "8755": 971134720.0, "8760": 965569152.0, "8765": 934519872.0, "8770": 986656640.0, "8775": 969789440.0, "8780": 967920512.0, "8785": 962639488.0, "8790": 947921664.0, "8795": 969775296.0, "8800": 971220608.0, "8805": 973559168.0, "8810": 983161280.0, "8815": 951065856.0, "8820": 939478016.0, "8825": 964494336.0, "8830": 981089472.0, "8835": 971889408.0, "8840": 979835520.0, "8845": 951616384.0, "8850": 987153920.0, "8855": 971335296.0, "8860": 962222080.0, "8865": 957359360.0, "8870": 946242816.0, "8875": 968628096.0, "8880": 984173184.0, "8885": 971110144.0, "8890": 970299648.0, "8895": 952971136.0, "8900": 962246528.0, "8905": 977392000.0, "8910": 981876416.0, "8915": 981149952.0, "8920": 968258432.0, "8925": 940189184.0, "8930": 970787456.0, "8935": 963634560.0, "8940": 978025664.0, "8945": 982356352.0, "8950": 946274176.0, "8955": 972928128.0, "8960": 974032128.0, "8965": 973961216.0, "8970": 966361216.0, "8975": 937321600.0, "8980": 953099648.0, "8985": 977878528.0, "8990": 967166592.0, "8995": 980283904.0, "9000": 952421184.0, "9005": 950292544.0, "9010": 974935552.0, "9015": 982668672.0, "9020": 959278656.0, "9025": 979055040.0, "9030": 953936640.0, "9035": 968749312.0, "9040": 978270080.0, "9045": 968843136.0, "9050": 983417600.0, "9055": 947885952.0, "9060": 956699776.0, "9065": 970246528.0, "9070": 968015744.0, "9075": 981225856.0, "9080": 952541632.0, "9085": 971319168.0, "9090": 963789184.0, "9095": 968313984.0, "9100": 974584320.0, "9105": 960032896.0, "9110": 947321664.0, "9115": 956833728.0, "9120": 985899904.0, "9125": 963026176.0, "9130": 958457216.0, "9135": 951989056.0, "9140": 967565824.0, "9145": 977433728.0, "9150": 987305408.0, "9155": 976649408.0, "9160": 958050816.0, "9165": 950957248.0, "9170": 988702272.0, "9175": 971913280.0, "9180": 967854400.0, "9185": 955127680.0, "9190": 957263744.0, "9195": 966003584.0, "9200": 968856960.0, "9205": 967330048.0, "9210": 984179584.0, "9215": 931743808.0, "9220": 949808960.0, "9225": 971440256.0, "9230": 971281792.0, "9235": 971857152.0, "9240": 959917376.0, "9245": 963584128.0, "9250": 961416384.0, "9255": 983241472.0, "9260": 979566336.0, "9265": 953039104.0, "9270": 949474624.0, "9275": 978502016.0, "9280": 978025536.0, "9285": 962828800.0, "9290": 979390080.0, "9295": 958548480.0, "9300": 965876352.0, "9305": 969599232.0, "9310": 973283008.0, "9315": 976451392.0, "9320": 948304512.0, "9325": 979749696.0, "9330": 977926784.0, "9335": 975525504.0, "9340": 960336000.0, "9345": 943464832.0, "9350": 952835072.0, "9355": 962850048.0, "9360": 960675328.0, "9365": 983816320.0, "9370": 983035904.0, "9375": 942080896.0, "9380": 982540928.0, "9385": 985259136.0, "9390": 973406272.0, "9395": 978528128.0, "9400": 938038400.0, "9405": 968500672.0, "9410": 981791488.0, "9415": 991945472.0, "9420": 960625728.0, "9425": 956681216.0, "9430": 938695808.0, "9435": 974362368.0, "9440": 959727872.0, "9445": 973720576.0, "9450": 961877760.0, "9455": 946303872.0, "9460": 978086272.0, "9465": 988617984.0, "9470": 963615872.0, "9475": 983908608.0, "9480": 930854528.0, "9485": 987221248.0, "9490": 963974912.0, "9495": 972857088.0, "9500": 982392960.0, "9505": 970286080.0, "9510": 964873536.0, "9515": 957183296.0, "9520": 948641664.0, "9525": 965336064.0, "9530": 958567296.0, "9535": 950963840.0, "9540": 954501120.0, "9545": 979935296.0, "9550": 955384704.0, "9555": 953296192.0, "9560": 958726208.0, "9565": 969930112.0, "9570": 977751168.0, "9575": 958849792.0, "9580": 963257728.0, "9585": 946197184.0, "9590": 948135936.0, "9595": 967007808.0, "9600": 985117952.0, "9605": 985499648.0, "9610": 943959808.0, "9615": 952912128.0, "9620": 980920192.0, "9625": 978524736.0, "9630": 969671168.0, "9635": 974868544.0, "9640": 940772416.0, "9645": 962475008.0, "9650": 970857536.0, "9655": 987496960.0, "9660": 963394176.0, "9665": 950327872.0, "9670": 965817856.0, "9675": 963579264.0, "9680": 965384064.0, "9685": 986598272.0, "9690": 940596864.0, "9695": 950521728.0, "9700": 975714688.0, "9705": 972896256.0, "9710": 967299968.0, "9715": 971403392.0, "9720": 940613632.0, "9725": 966514816.0, "9730": 974099584.0, "9735": 974345792.0, "9740": 971516928.0, "9745": 951220736.0, "9750": 979370880.0, "9755": 970170432.0, "9760": 968237888.0, "9765": 963835520.0, "9770": 952652160.0, "9775": 956682880.0, "9780": 970721984.0, "9785": 958959232.0, "9790": 961043072.0, "9795": 958779200.0, "9800": 949918656.0, "9805": 962651200.0, "9810": 979093888.0, "9815": 978146816.0, "9820": 982841088.0, "9825": 939730944.0, "9830": 969614208.0, "9835": 973272832.0, "9840": 971945664.0, "9845": 967603328.0, "9850": 947232896.0, "9855": 956896512.0, "9860": 987801728.0, "9865": 970385664.0, "9870": 990310144.0, "9875": 957380096.0, "9880": 931362176.0, "9885": 963678464.0, "9890": 972811648.0, "9895": 984054016.0, "9900": 956595136.0, "9905": 939303808.0, "9910": 979107072.0, "9915": 973996800.0, "9920": 943946432.0, "9925": 963187328.0, "9930": 948020224.0, "9935": 960573120.0, "9940": 965856512.0, "9945": 958998016.0, "9950": 964584192.0, "9955": 943733120.0, "9960": 966844160.0, "9965": 983732096.0, "9970": 966840192.0, "9975": 964040640.0, "9980": 980881024.0, "9985": 942746240.0, "9990": 976134400.0, "9995": 982950848.0, "10000": 972073152.0, "10005": 970193472.0, "10010": 944380480.0, "10015": 983265344.0, "10020": 977865472.0, "10025": 979868544.0, "10030": 971490816.0, "10035": 946263296.0, "10040": 950534016.0, "10045": 977546880.0, "10050": 986017280.0, "10055": 990492800.0, "10060": 958996032.0, "10065": 947517312.0, "10070": 966895616.0, "10075": 979683904.0, "10080": 971953920.0, "10085": 974879744.0, "10090": 944216960.0, "10095": 962977344.0, "10100": 972381952.0, "10105": 976354432.0, "10110": 972128768.0, "10115": 948919680.0, "10120": 962852480.0, "10125": 974293120.0, "10130": 980737472.0, "10135": 972335104.0, "10140": 957843264.0, "10145": 934671872.0, "10150": 973965568.0, "10155": 970306112.0, "10160": 962491456.0, "10165": 975341248.0, "10170": 944624384.0, "10175": 979643712.0, "10180": 984008448.0, "10185": 978870144.0, "10190": 955877376.0, "10195": 937261120.0, "10200": 988253760.0, "10205": 973401856.0, "10210": 966901120.0, "10215": 976049664.0, "10220": 948799872.0, "10225": 950572096.0, "10230": 976120896.0, "10235": 954421632.0, "10240": 969850752.0, "10245": 962265472.0, "10250": 936756480.0, "10255": 979774976.0, "10260": 965000704.0, "10265": 967563712.0, "10270": 969297920.0, "10275": 935944256.0, "10280": 969526272.0, "10285": 996465152.0, "10290": 979762816.0, "10295": 981662912.0, "10300": 952271936.0, "10305": 972024256.0, "10310": 960359872.0, "10315": 971605760.0, "10320": 985354304.0, "10325": 983302336.0, "10330": 935148288.0, "10335": 976392064.0, "10340": 957603840.0, "10345": 973044352.0, "10350": 984707136.0, "10355": 942479296.0, "10360": 962279040.0, "10365": 973641856.0, "10370": 980432768.0, "10375": 970343296.0, "10380": 962080384.0, "10385": 955687296.0, "10390": 990783104.0, "10395": 965164608.0, "10400": 960470208.0, "10405": 950214848.0, "10410": 955491392.0, "10415": 975924736.0, "10420": 967248320.0, "10425": 969875328.0, "10430": 965126272.0, "10435": 962680768.0, "10440": 972024064.0, "10445": 972467456.0, "10450": 974949504.0, "10455": 965864704.0, "10460": 948726272.0, "10465": 971534464.0, "10470": 972756736.0, "10475": 979392128.0, "10480": 997292352.0, "10485": 949631936.0, "10490": 935104896.0, "10495": 969599424.0, "10500": 978688704.0, "10505": 959342784.0, "10510": 951008000.0, "10515": 954223744.0, "10520": 972150016.0, "10525": 969942528.0, "10530": 970425728.0, "10535": 986576256.0, "10540": 946829632.0, "10545": 970484032.0, "10550": 969371968.0, "10555": 959521856.0, "10560": 976274496.0, "10565": 960798208.0, "10570": 968688128.0, "10575": 973272576.0, "10580": 961017472.0, "10585": 973457024.0, "10590": 952053568.0, "10595": 956331776.0, "10600": 967935552.0, "10605": 986576256.0, "10610": 966417408.0, "10615": 976957568.0, "10620": 940933888.0, "10625": 965306432.0, "10630": 968022272.0, "10635": 973333888.0, "10640": 974664448.0, "10645": 948582400.0, "10650": 966388224.0, "10655": 985562624.0, "10660": 976682624.0, "10665": 967088256.0, "10670": 955226368.0, "10675": 934529920.0, "10680": 986153344.0, "10685": 991102656.0, "10690": 963886208.0, "10695": 971933632.0, "10700": 950091520.0, "10705": 978240128.0, "10710": 968317184.0, "10715": 967450432.0, "10720": 966357824.0, "10725": 944490816.0, "10730": 980318592.0, "10735": 961117952.0, "10740": 971283392.0, "10745": 984630528.0, "10750": 981762816.0, "10755": 945191296.0, "10760": 969882304.0, "10765": 972886400.0, "10770": 974268608.0, "10775": 959067392.0, "10780": 949520384.0, "10785": 953706304.0, "10790": 970157568.0, "10795": 960631552.0, "10800": 972050368.0, "10805": 951460864.0, "10810": 974235456.0, "10815": 959804160.0, "10820": 971302656.0, "10825": 967211072.0, "10830": 957016128.0, "10835": 963139136.0, "10840": 971035008.0, "10845": 964268160.0, "10850": 958162432.0, "10855": 967657344.0, "10860": 950849536.0, "10865": 964061696.0, "10870": 983627200.0, "10875": 982016640.0, "10880": 958659648.0, "10885": 954981888.0, "10890": 973122560.0, "10895": 973655744.0, "10900": 970546048.0, "10905": 965184256.0, "10910": 939048192.0, "10915": 960749824.0, "10920": 983653376.0, "10925": 970068160.0, "10930": 968771200.0, "10935": 963228480.0, "10940": 954249408.0, "10945": 964532608.0, "10950": 972466880.0, "10955": 966621248.0, "10960": 972285056.0, "10965": 966333184.0, "10970": 983572160.0, "10975": 965330496.0, "10980": 974669248.0, "10985": 986818496.0, "10990": 950797760.0, "10995": 963598784.0, "11000": 985495104.0, "11005": 978671168.0, "11010": 971614464.0, "11015": 970071232.0, "11020": 948195648.0, "11025": 960105088.0, "11030": 978168768.0, "11035": 976017024.0, "11040": 986523264.0, "11045": 956708480.0, "11050": 973395968.0, "11055": 974051968.0, "11060": 962164544.0, "11065": 985712768.0, "11070": 949791424.0, "11075": 976565888.0, "11080": 972315712.0, "11085": 967328576.0, "11090": 976399296.0, "11095": 946696448.0, "11100": 966199040.0, "11105": 974421504.0, "11110": 981198912.0, "11115": 968108160.0, "11120": 957518656.0, "11125": 956979840.0, "11130": 975786432.0, "11135": 979636544.0, "11140": 964944832.0, "11145": 966499008.0, "11150": 935518400.0, "11155": 976579008.0, "11160": 984367232.0, "11165": 982289792.0, "11170": 978113472.0, "11175": 958084864.0, "11180": 962589888.0, "11185": 972260672.0, "11190": 979666368.0, "11195": 985502784.0, "11200": 983014336.0, "11205": 942426240.0, "11210": 984802368.0, "11215": 967690816.0, "11220": 983476928.0, "11225": 961985728.0, "11230": 953398272.0, "11235": 981841280.0, "11240": 977805568.0, "11245": 966530176.0, "11250": 969466304.0, "11255": 960572544.0, "11260": 980096576.0, "11265": 963926720.0, "11270": 981695936.0, "11275": 968525888.0, "11280": 955905088.0, "11285": 953700224.0, "11290": 956489152.0, "11295": 968197568.0, "11300": 962513216.0, "11305": 958759872.0, "11310": 946304256.0, "11315": 983036096.0, "11320": 964828480.0, "11325": 980906304.0, "11330": 975476608.0, "11335": 952186816.0, "11340": 970596800.0, "11345": 969926080.0, "11350": 981628736.0, "11355": 981905088.0, "11360": 940723328.0, "11365": 970750592.0, "11370": 978978432.0, "11375": 975338432.0, "11380": 968256960.0, "11385": 958096384.0, "11390": 937948288.0, "11395": 977494080.0, "11400": 973515520.0, "11405": 961359424.0, "11410": 966143616.0, "11415": 929202368.0, "11420": 964768960.0, "11425": 981196352.0, "11430": 978636864.0, "11435": 970153280.0, "11440": 945072704.0, "11445": 975241024.0, "11450": 984735296.0, "11455": 971426176.0, "11460": 965182016.0, "11465": 960090176.0, "11470": 955191296.0, "11475": 972691072.0, "11480": 956542272.0, "11485": 977076864.0, "11490": 986332352.0, "11495": 959121344.0, "11500": 969424704.0, "11505": 964024640.0, "11510": 976702848.0, "11515": 977904064.0, "11520": 953963584.0, "11525": 976039360.0, "11530": 976686784.0, "11535": 979809792.0, "11540": 974141760.0, "11545": 953644288.0, "11550": 953295552.0, "11555": 981560640.0, "11560": 984532352.0, "11565": 965181312.0, "11570": 966487424.0, "11575": 950896832.0, "11580": 976062592.0, "11585": 977550784.0, "11590": 969314368.0, "11595": 976719232.0, "11600": 946047104.0, "11605": 973359168.0, "11610": 982457984.0, "11615": 972010048.0, "11620": 969363904.0, "11625": 949111040.0, "11630": 937578176.0, "11635": 973667008.0, "11640": 981259456.0, "11645": 980106048.0, "11650": 971758144.0, "11655": 956204288.0, "11660": 980712192.0, "11665": 958265664.0, "11670": 982618880.0, "11675": 972427200.0, "11680": 956445568.0, "11685": 982955712.0, "11690": 968614528.0, "11695": 968085632.0, "11700": 973819008.0, "11705": 956432640.0, "11710": 964584640.0, "11715": 983208448.0, "11720": 983636224.0, "11725": 965204032.0, "11730": 955695040.0, "11735": 942941376.0, "11740": 973829824.0, "11745": 971260672.0, "11750": 961624256.0, "11755": 963534976.0, "11760": 950291904.0, "11765": 983877632.0, "11770": 984810368.0, "11775": 975671936.0, "11780": 985430336.0, "11785": 947272512.0, "11790": 972444352.0, "11795": 970670464.0, "11800": 973251520.0, "11805": 986780480.0, "11810": 967591808.0, "11815": 955862848.0, "11820": 973765952.0, "11825": 970671296.0, "11830": 974883776.0, "11835": 961826368.0, "11840": 944467904.0, "11845": 980681344.0, "11850": 974601536.0, "11855": 977943744.0, "11860": 971600192.0, "11865": 938631104.0, "11870": 940150208.0, "11875": 990036736.0, "11880": 971605184.0, "11885": 962938432.0, "11890": 970330560.0, "11895": 965356416.0, "11900": 978791360.0, "11905": 961690240.0, "11910": 983653824.0, "11915": 989882688.0, "11920": 944918016.0, "11925": 994335296.0, "11930": 964853504.0, "11935": 963527104.0, "11940": 977089344.0, "11945": 944859968.0, "11950": 977822912.0, "11955": 979190400.0, "11960": 972201664.0, "11965": 976359488.0, "11970": 963048064.0, "11975": 963239936.0, "11980": 977609536.0, "11985": 953135424.0, "11990": 968692864.0, "11995": 965206144.0, "12000": 958650816.0, "12005": 974558720.0, "12010": 979488320.0, "12015": 972212928.0, "12020": 972972992.0, "12025": 934714048.0, "12030": 969111104.0, "12035": 984089664.0, "12040": 977507648.0, "12045": 981645056.0, "12050": 931286400.0, "12055": 938790208.0, "12060": 974349248.0, "12065": 965845696.0, "12070": 968239104.0, "12075": 949940224.0, "12080": 953231552.0, "12085": 972342592.0, "12090": 964042304.0, "12095": 963552832.0, "12100": 976701248.0, "12105": 950225216.0, "12110": 972205568.0, "12115": 968115136.0, "12120": 986359296.0, "12125": 980675264.0, "12130": 941589504.0, "12135": 955591040.0, "12140": 975756032.0, "12145": 979390528.0, "12150": 979435776.0, "12155": 961767936.0, "12160": 946323264.0, "12165": 968612864.0, "12170": 964129152.0, "12175": 967543936.0, "12180": 974942848.0, "12185": 952774592.0, "12190": 988261760.0, "12195": 970261312.0, "12200": 964981312.0, "12205": 968767232.0, "12210": 939144320.0, "12215": 996663488.0, "12220": 970037696.0, "12225": 979618880.0, "12230": 980198144.0, "12235": 950224576.0, "12240": 963441344.0, "12245": 965754240.0, "12250": 976793792.0, "12255": 968140288.0, "12260": 983346688.0, "12265": 931758592.0, "12270": 966457472.0, "12275": 979665408.0, "12280": 977629696.0, "12285": 970432320.0, "12290": 929095296.0, "12295": 976750144.0, "12300": 985683008.0, "12305": 969918016.0, "12310": 986105792.0, "12315": 936237952.0, "12320": 957828032.0, "12325": 966696000.0, "12330": 968270016.0, "12335": 963800896.0, "12340": 957433344.0, "12345": 944247872.0, "12350": 966701760.0, "12355": 975709440.0, "12360": 978832832.0, "12365": 964561280.0, "12370": 948990016.0, "12375": 963593536.0, "12380": 964789056.0, "12385": 973126912.0, "12390": 961501056.0, "12395": 961629824.0, "12400": 975307712.0, "12405": 976388032.0, "12410": 953154688.0, "12415": 962821184.0, "12420": 943854144.0, "12425": 948976640.0, "12430": 972157696.0, "12435": 969004352.0, "12440": 961893696.0, "12445": 951751744.0, "12450": 947021888.0, "12455": 981248448.0, "12460": 973989696.0, "12465": 954462784.0, "12470": 981144320.0, "12475": 958569728.0, "12480": 967084864.0, "12485": 978267776.0, "12490": 974168192.0, "12495": 969692160.0, "12500": 961573632.0, "12505": 943534528.0, "12510": 961013504.0, "12515": 969566080.0, "12520": 974194304.0, "12525": 972107840.0, "12530": 944608640.0, "12535": 976414272.0, "12540": 965917440.0, "12545": 972033856.0, "12550": 969671552.0, "12555": 941300736.0, "12560": 964469312.0, "12565": 947620608.0, "12570": 974443840.0, "12575": 963055232.0, "12580": 958104128.0, "12585": 964300352.0, "12590": 965924288.0, "12595": 978732672.0, "12600": 982093952.0, "12605": 949125824.0, "12610": 937745344.0, "12615": 962779264.0, "12620": 961201664.0, "12625": 966531136.0, "12630": 971046272.0, "12635": 962236416.0, "12640": 978541696.0, "12645": 969369920.0, "12650": 970161664.0, "12655": 964217216.0, "12660": 932390336.0, "12665": 956865664.0, "12670": 986180352.0, "12675": 965566464.0, "12680": 961126528.0, "12685": 951304256.0, "12690": 945491456.0, "12695": 978387648.0, "12700": 985277888.0, "12705": 958784640.0, "12710": 968294144.0, "12715": 956280512.0, "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}, "mem-allocated-bytes": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": 12795811840.0, "5": 12795811840.0, "10": 12795811840.0, "15": 12795811840.0, "20": 12795811840.0, "25": 12795811840.0, "30": 12795811840.0, "35": 12795811840.0, "40": 12795811840.0, "45": 12795811840.0, "50": 12795811840.0, "55": 12795811840.0, "60": 12795811840.0, "65": 12795811840.0, "70": 12795811840.0, "75": 12795811840.0, "80": 12795811840.0, "85": 12795811840.0, "90": 12795811840.0, "95": 12795811840.0, "100": 12795811840.0, "105": 12795811840.0, "110": 12795811840.0, "115": 12795811840.0, "120": 12795811840.0, "125": 12795811840.0, "130": 12795811840.0, "135": 12795811840.0, "140": 12795811840.0, "145": 12795811840.0, "150": 12795811840.0, "155": 12795811840.0, "160": 12795811840.0, "165": 12795811840.0, "170": 12795811840.0, "175": 12795811840.0, "180": 12795811840.0, "185": 12795811840.0, "190": 12795811840.0, "195": 12795811840.0, "200": 12795811840.0, "205": 12795811840.0, "210": 12795811840.0, "215": 12795811840.0, "220": 12795811840.0, "225": 12795811840.0, "230": 12795811840.0, "235": 12795811840.0, "240": 12795811840.0, "245": 12795811840.0, "250": 12795811840.0, "255": 12795811840.0, "260": 12795811840.0, "265": 12795811840.0, "270": 12795811840.0, "275": 12795811840.0, "280": 12795811840.0, "285": 12795811840.0, "290": 12795811840.0, "295": 12795811840.0, "300": 12795811840.0, "305": 12795811840.0, "310": 12795811840.0, "315": 12795811840.0, "320": 12795811840.0, "325": 12795811840.0, "330": 12795811840.0, "335": 12795811840.0, "340": 12795811840.0, "345": 12795811840.0, "350": 12795811840.0, "355": 12795811840.0, "360": 12795811840.0, "365": 12795811840.0, "370": 12795811840.0, "375": 12795811840.0, "380": 12795811840.0, "385": 12795811840.0, "390": 12795811840.0, "395": 12795811840.0, "400": 12795811840.0, "405": 12795811840.0, "410": 12795811840.0, "415": 12795811840.0, "420": 12795811840.0, "425": 12795811840.0, "430": 12795811840.0, "435": 12795811840.0, "440": 12795811840.0, "445": 12795811840.0, "450": 12795811840.0, "455": 12795811840.0, "460": 12795811840.0, "465": 12795811840.0, "470": 12795811840.0, "475": 12795811840.0, "480": 12795811840.0, "485": 12795811840.0, "490": 12795811840.0, "495": 12795811840.0, "500": 12795811840.0, "505": 12795811840.0, "510": 12795811840.0, "515": 12795811840.0, "520": 12795811840.0, "525": 12795811840.0, "530": 12795811840.0, "535": 12795811840.0, "540": 12795811840.0, "545": 12795811840.0, "550": 12795811840.0, "555": 12795811840.0, "560": 12795811840.0, "565": 12795811840.0, "570": 12795811840.0, "575": 12795811840.0, "580": 12795811840.0, "585": 12795811840.0, "590": 12795811840.0, "595": 12795811840.0, "600": 12795811840.0, "605": 12795811840.0, "610": 12795811840.0, "615": 12795811840.0, "620": 12795811840.0, "625": 12795811840.0, "630": 12795811840.0, "635": 12795811840.0, "640": 12795811840.0, "645": 12795811840.0, "650": 12795811840.0, "655": 12795811840.0, "660": 12795811840.0, "665": 12795811840.0, "670": 12795811840.0, "675": 12795811840.0, "680": 12795811840.0, "685": 12795811840.0, "690": 12795811840.0, "695": 12795811840.0, "700": 12795811840.0, "705": 12795811840.0, "710": 12795811840.0, "715": 12795811840.0, "720": 12795811840.0, "725": 12795811840.0, "730": 12795811840.0, "735": 12795811840.0, "740": 12795811840.0, "745": 12795811840.0, "750": 12795811840.0, "755": 12795811840.0, "760": 12795811840.0, "765": 12795811840.0, "770": 12795811840.0, "775": 12795811840.0, "780": 12795811840.0, "785": 12795811840.0, "790": 12795811840.0, "795": 12795811840.0, "800": 12795811840.0, "805": 12795811840.0, "810": 12795811840.0, "815": 12795811840.0, "820": 12795811840.0, "825": 12795811840.0, "830": 12795811840.0, "835": 12795811840.0, "840": 12795811840.0, "845": 12795811840.0, "850": 12795811840.0, "855": 12795811840.0, "860": 12795811840.0, "865": 12795811840.0, "870": 12795811840.0, "875": 12795811840.0, "880": 12795811840.0, "885": 12795811840.0, "890": 12795811840.0, "895": 12795811840.0, "900": 12795811840.0, "905": 12795811840.0, "910": 12795811840.0, "915": 12795811840.0, "920": 12795811840.0, "925": 12795811840.0, "930": 12795811840.0, "935": 12795811840.0, "940": 12795811840.0, "945": 12795811840.0, "950": 12795811840.0, "955": 12795811840.0, "960": 12795811840.0, "965": 12795811840.0, "970": 12795811840.0, "975": 12795811840.0, "980": 12795811840.0, "985": 12795811840.0, "990": 12795811840.0, "995": 12795811840.0, "1000": 12795811840.0, "1005": 12795811840.0, "1010": 12795811840.0, "1015": 12795811840.0, "1020": 12795811840.0, "1025": 12795811840.0, "1030": 12795811840.0, "1035": 12795811840.0, "1040": 12795811840.0, "1045": 12795811840.0, "1050": 12795811840.0, "1055": 12795811840.0, "1060": 12795811840.0, "1065": 12795811840.0, "1070": 12795811840.0, "1075": 12795811840.0, "1080": 12795811840.0, "1085": 12795811840.0, "1090": 12795811840.0, "1095": 12795811840.0, "1100": 12795811840.0, "1105": 12795811840.0, "1110": 12795811840.0, "1115": 12795811840.0, "1120": 12795811840.0, "1125": 12795811840.0, "1130": 12795811840.0, "1135": 12795811840.0, "1140": 12795811840.0, "1145": 12795811840.0, "1150": 12795811840.0, "1155": 12795811840.0, "1160": 12795811840.0, "1165": 12795811840.0, "1170": 12795811840.0, "1175": 12795811840.0, "1180": 12795811840.0, "1185": 12795811840.0, "1190": 12795811840.0, "1195": 12795811840.0, "1200": 12795811840.0, "1205": 12795811840.0, "1210": 12795811840.0, "1215": 12795811840.0, "1220": 12795811840.0, "1225": 12795811840.0, "1230": 12795811840.0, "1235": 12795811840.0, "1240": 12795811840.0, "1245": 12795811840.0, "1250": 12795811840.0, "1255": 12795811840.0, "1260": 12795811840.0, "1265": 12795811840.0, "1270": 12795811840.0, "1275": 12795811840.0, "1280": 12795811840.0, "1285": 12795811840.0, "1290": 12795811840.0, "1295": 12795811840.0, "1300": 12795811840.0, "1305": 12795811840.0, "1310": 12795811840.0, "1315": 12795811840.0, "1320": 12795811840.0, "1325": 12795811840.0, "1330": 12795811840.0, "1335": 12795811840.0, "1340": 12795811840.0, "1345": 12795811840.0, "1350": 12795811840.0, "1355": 12795811840.0, "1360": 12795811840.0, "1365": 12795811840.0, "1370": 12795811840.0, "1375": 12795811840.0, "1380": 12795811840.0, "1385": 12795811840.0, "1390": 12795811840.0, "1395": 12795811840.0, "1400": 12795811840.0, "1405": 12795811840.0, "1410": 12795811840.0, "1415": 12795811840.0, "1420": 12795811840.0, "1425": 12795811840.0, "1430": 12795811840.0, "1435": 12795811840.0, "1440": 12795811840.0, "1445": 12795811840.0, "1450": 12795811840.0, "1455": 12795811840.0, "1460": 12795811840.0, "1465": 12795811840.0, "1470": 12795811840.0, "1475": 12795811840.0, "1480": 12795811840.0, "1485": 12795811840.0, "1490": 12795811840.0, "1495": 12795811840.0, "1500": 12795811840.0, "1505": 12795811840.0, "1510": 12795811840.0, "1515": 12795811840.0, "1520": 12795811840.0, "1525": 12795811840.0, "1530": 12795811840.0, "1535": 12795811840.0, "1540": 12795811840.0, "1545": 12795811840.0, "1550": 12795811840.0, "1555": 12795811840.0, "1560": 12795811840.0, "1565": 12795811840.0, "1570": 12795811840.0, "1575": 12795811840.0, "1580": 12795811840.0, "1585": 12795811840.0, "1590": 12795811840.0, "1595": 12795811840.0, "1600": 12795811840.0, "1605": 12795811840.0, "1610": 12795811840.0, "1615": 12795811840.0, "1620": 12795811840.0, "1625": 12795811840.0, "1630": 12795811840.0, "1635": 12795811840.0, "1640": 12795811840.0, "1645": 12795811840.0, "1650": 12795811840.0, "1655": 12795811840.0, "1660": 12795811840.0, "1665": 12795811840.0, "1670": 12795811840.0, "1675": 12795811840.0, "1680": 12795811840.0, "1685": 12795811840.0, "1690": 12795811840.0, "1695": 12795811840.0, "1700": 12795811840.0, "1705": 12795811840.0, "1710": 12795811840.0, "1715": 12795811840.0, "1720": 12795811840.0, "1725": 12795811840.0, "1730": 12795811840.0, "1735": 12795811840.0, "1740": 12795811840.0, "1745": 12795811840.0, "1750": 12795811840.0, "1755": 12795811840.0, "1760": 12795811840.0, "1765": 12795811840.0, "1770": 12795811840.0, "1775": 12795811840.0, "1780": 12795811840.0, "1785": 12795811840.0, "1790": 12795811840.0, "1795": 12795811840.0, "1800": 12795811840.0, "1805": 12795811840.0, "1810": 12795811840.0, "1815": 12795811840.0, "1820": 12795811840.0, "1825": 12795811840.0, "1830": 12795811840.0, "1835": 12795811840.0, "1840": 12795811840.0, "1845": 12795811840.0, "1850": 12795811840.0, "1855": 12795811840.0, "1860": 12795811840.0, "1865": 12795811840.0, "1870": 12795811840.0, "1875": 12795811840.0, "1880": 12795811840.0, "1885": 12795811840.0, "1890": 12795811840.0, "1895": 12795811840.0, "1900": 12795811840.0, "1905": 12795811840.0, "1910": 12795811840.0, "1915": 12795811840.0, "1920": 12795811840.0, "1925": 12795811840.0, "1930": 12795811840.0, "1935": 12795811840.0, "1940": 12795811840.0, "1945": 12795811840.0, "1950": 12795811840.0, "1955": 12795811840.0, "1960": 12795811840.0, "1965": 12795811840.0, "1970": 12795811840.0, "1975": 12795811840.0, "1980": 12795811840.0, "1985": 12795811840.0, "1990": 12795811840.0, "1995": 12795811840.0, "2000": 12795811840.0, "2005": 12795811840.0, "2010": 12795811840.0, "2015": 12795811840.0, "2020": 12795811840.0, "2025": 12795811840.0, "2030": 12795811840.0, "2035": 12795811840.0, "2040": 12795811840.0, "2045": 12795811840.0, "2050": 12795811840.0, "2055": 12795811840.0, "2060": 12795811840.0, "2065": 12795811840.0, "2070": 12795811840.0, "2075": 12795811840.0, "2080": 12795811840.0, "2085": 12795811840.0, "2090": 12795811840.0, "2095": 12795811840.0, "2100": 12795811840.0, "2105": 12795811840.0, "2110": 12795811840.0, "2115": 12795811840.0, "2120": 12795811840.0, "2125": 12795811840.0, "2130": 12795811840.0, "2135": 12795811840.0, "2140": 12795811840.0, "2145": 12795811840.0, "2150": 12795811840.0, "2155": 12795811840.0, "2160": 12795811840.0, "2165": 12795811840.0, "2170": 12795811840.0, "2175": 12795811840.0, "2180": 12795811840.0, "2185": 12795811840.0, "2190": 12795811840.0, "2195": 12795811840.0, "2200": 12795811840.0, "2205": 12795811840.0, "2210": 12795811840.0, "2215": 12795811840.0, "2220": 12795811840.0, "2225": 12795811840.0, "2230": 12795811840.0, "2235": 12795811840.0, "2240": 12795811840.0, "2245": 12795811840.0, "2250": 12795811840.0, "2255": 12795811840.0, "2260": 12795811840.0, "2265": 12795811840.0, "2270": 12795811840.0, "2275": 12795811840.0, "2280": 12795811840.0, "2285": 12795811840.0, "2290": 12795811840.0, "2295": 12795811840.0, "2300": 12795811840.0, "2305": 12795811840.0, "2310": 12795811840.0, "2315": 12795811840.0, "2320": 12795811840.0, "2325": 12795811840.0, "2330": 12795811840.0, "2335": 12795811840.0, "2340": 12795811840.0, "2345": 12795811840.0, "2350": 12795811840.0, "2355": 12795811840.0, "2360": 12795811840.0, "2365": 12795811840.0, "2370": 12795811840.0, "2375": 12795811840.0, "2380": 12795811840.0, "2385": 12795811840.0, "2390": 12795811840.0, "2395": 12795811840.0, "2400": 12795811840.0, "2405": 12795811840.0, "2410": 12795811840.0, "2415": 12795811840.0, "2420": 12795811840.0, "2425": 12795811840.0, "2430": 12795811840.0, "2435": 12795811840.0, "2440": 12795811840.0, "2445": 12795811840.0, "2450": 12795811840.0, "2455": 12795811840.0, "2460": 12795811840.0, "2465": 12795811840.0, "2470": 12795811840.0, "2475": 12795811840.0, "2480": 12795811840.0, "2485": 12795811840.0, "2490": 12795811840.0, "2495": 12795811840.0, "2500": 12795811840.0, "2505": 12795811840.0, "2510": 12795811840.0, "2515": 12795811840.0, "2520": 12795811840.0, "2525": 12795811840.0, "2530": 12795811840.0, "2535": 12795811840.0, "2540": 12795811840.0, "2545": 12795811840.0, "2550": 12795811840.0, "2555": 12795811840.0, "2560": 12795811840.0, "2565": 12795811840.0, "2570": 12795811840.0, "2575": 12795811840.0, "2580": 12795811840.0, "2585": 12795811840.0, "2590": 12795811840.0, "2595": 12795811840.0, "2600": 12795811840.0, "2605": 12795811840.0, "2610": 12795811840.0, "2615": 12795811840.0, "2620": 12795811840.0, "2625": 12795811840.0, "2630": 12795811840.0, "2635": 12795811840.0, "2640": 12795811840.0, "2645": 12795811840.0, "2650": 12795811840.0, "2655": 12795811840.0, "2660": 12795811840.0, "2665": 12795811840.0, "2670": 12795811840.0, "2675": 12795811840.0, "2680": 12795811840.0, "2685": 12795811840.0, "2690": 12795811840.0, "2695": 12795811840.0, "2700": 12795811840.0, "2705": 12795811840.0, "2710": 12795811840.0, "2715": 12795811840.0, "2720": 12795811840.0, "2725": 12795811840.0, "2730": 12795811840.0, "2735": 12795811840.0, "2740": 12795811840.0, "2745": 12795811840.0, "2750": 12795811840.0, "2755": 12795811840.0, "2760": 12795811840.0, "2765": 12795811840.0, "2770": 12795811840.0, "2775": 12795811840.0, "2780": 12795811840.0, "2785": 12795811840.0, "2790": 12795811840.0, "2795": 12795811840.0, "2800": 12795811840.0, "2805": 12795811840.0, "2810": 12795811840.0, "2815": 12795811840.0, "2820": 12795811840.0, "2825": 12795811840.0, "2830": 12795811840.0, "2835": 12795811840.0, "2840": 12795811840.0, "2845": 12795811840.0, "2850": 12795811840.0, "2855": 12795811840.0, "2860": 12795811840.0, "2865": 12795811840.0, "2870": 12795811840.0, "2875": 12795811840.0, "2880": 12795811840.0, "2885": 12795811840.0, "2890": 12795811840.0, "2895": 12795811840.0, "2900": 12795811840.0, "2905": 12795811840.0, "2910": 12795811840.0, "2915": 12795811840.0, "2920": 12795811840.0, "2925": 12795811840.0, "2930": 12795811840.0, "2935": 12795811840.0, "2940": 12795811840.0, "2945": 12795811840.0, "2950": 12795811840.0, "2955": 12795811840.0, "2960": 12795811840.0, "2965": 12795811840.0, "2970": 12795811840.0, "2975": 12795811840.0, "2980": 12795811840.0, "2985": 12795811840.0, "2990": 12795811840.0, "2995": 12795811840.0, "3000": 12795811840.0, "3005": 12795811840.0, "3010": 12795811840.0, "3015": 12795811840.0, "3020": 12795811840.0, "3025": 12795811840.0, "3030": 12795811840.0, "3035": 12795811840.0, "3040": 12795811840.0, "3045": 12795811840.0, "3050": 12795811840.0, "3055": 12795811840.0, "3060": 12795811840.0, "3065": 12795811840.0, "3070": 12795811840.0, "3075": 12795811840.0, "3080": 12795811840.0, "3085": 12795811840.0, "3090": 12795811840.0, "3095": 12795811840.0, "3100": 12795811840.0, "3105": 12795811840.0, "3110": 12795811840.0, "3115": 12795811840.0, "3120": 12795811840.0, "3125": 12795811840.0, "3130": 12795811840.0, "3135": 12795811840.0, "3140": 12795811840.0, "3145": 12795811840.0, "3150": 12795811840.0, "3155": 12795811840.0, "3160": 12795811840.0, "3165": 12795811840.0, "3170": 12795811840.0, "3175": 12795811840.0, "3180": 12795811840.0, "3185": 12795811840.0, "3190": 12795811840.0, "3195": 12795811840.0, "3200": 12795811840.0, "3205": 12795811840.0, "3210": 12795811840.0, "3215": 12795811840.0, "3220": 12795811840.0, "3225": 12795811840.0, "3230": 12795811840.0, "3235": 12795811840.0, "3240": 12795811840.0, "3245": 12795811840.0, "3250": 12795811840.0, "3255": 12795811840.0, "3260": 12795811840.0, "3265": 12795811840.0, "3270": 12795811840.0, "3275": 12795811840.0, "3280": 12795811840.0, "3285": 12795811840.0, "3290": 12795811840.0, "3295": 12795811840.0, "3300": 12795811840.0, "3305": 12795811840.0, "3310": 12795811840.0, "3315": 12795811840.0, "3320": 12795811840.0, "3325": 12795811840.0, "3330": 12795811840.0, "3335": 12795811840.0, "3340": 12795811840.0, "3345": 12795811840.0, "3350": 12795811840.0, "3355": 12795811840.0, "3360": 12795811840.0, "3365": 12795811840.0, "3370": 12795811840.0, "3375": 12795811840.0, "3380": 12795811840.0, "3385": 12795811840.0, "3390": 12795811840.0, "3395": 12795811840.0, "3400": 12795811840.0, "3405": 12795811840.0, "3410": 12795811840.0, "3415": 12795811840.0, "3420": 12795811840.0, "3425": 12795811840.0, "3430": 12795811840.0, "3435": 12795811840.0, "3440": 12795811840.0, "3445": 12795811840.0, "3450": 12795811840.0, "3455": 12795811840.0, "3460": 12795811840.0, "3465": 12795811840.0, "3470": 12795811840.0, "3475": 12795811840.0, "3480": 12795811840.0, "3485": 12795811840.0, "3490": 12795811840.0, "3495": 12795811840.0, "3500": 12795811840.0, "3505": 12795811840.0, "3510": 12795811840.0, "3515": 12795811840.0, "3520": 12795811840.0, "3525": 12795811840.0, "3530": 12795811840.0, "3535": 12795811840.0, "3540": 12795811840.0, "3545": 12795811840.0, "3550": 12795811840.0, "3555": 12795811840.0, "3560": 12795811840.0, "3565": 12795811840.0, "3570": 12795789312.0, "3575": 12795789312.0, "3580": 12795789312.0, "3585": 12795789312.0, "3590": 12795789312.0, "3595": 12795789312.0, "3600": 12795789312.0, "3605": 12795789312.0, "3610": 12795789312.0, "3615": 12795789312.0, "3620": 12795789312.0, "3625": 12795789312.0, "3630": 12795789312.0, "3635": 12795789312.0, "3640": 12795789312.0, "3645": 12795789312.0, "3650": 12795789312.0, "3655": 12795789312.0, "3660": 12795789312.0, "3665": 12795789312.0, "3670": 12795789312.0, "3675": 12795789312.0, "3680": 12795789312.0, "3685": 12795789312.0, "3690": 12795789312.0, "3695": 12795789312.0, "3700": 12795789312.0, "3705": 12795789312.0, "3710": 12795789312.0, "3715": 12795789312.0, "3720": 12795789312.0, "3725": 12795789312.0, "3730": 12795789312.0, "3735": 12795789312.0, "3740": 12795789312.0, "3745": 12795789312.0, "3750": 12795789312.0, "3755": 12795789312.0, "3760": 12795789312.0, "3765": 12795789312.0, "3770": 12795789312.0, "3775": 12795789312.0, "3780": 12795789312.0, "3785": 12795789312.0, "3790": 12795789312.0, "3795": 12795789312.0, "3800": 12795789312.0, "3805": 12795789312.0, "3810": 12795789312.0, "3815": 12795789312.0, "3820": 12795789312.0, "3825": 12795789312.0, "3830": 12795789312.0, "3835": 12795789312.0, "3840": 12795789312.0, "3845": 12795789312.0, "3850": 12795789312.0, "3855": 12795789312.0, "3860": 12795789312.0, "3865": 12795789312.0, "3870": 12795789312.0, "3875": 12795789312.0, "3880": 12795789312.0, "3885": 12795789312.0, "3890": 12795789312.0, "3895": 12795789312.0, "3900": 12795789312.0, "3905": 12795789312.0, "3910": 12795789312.0, "3915": 12795789312.0, "3920": 12795789312.0, "3925": 12795789312.0, "3930": 12795789312.0, "3935": 12795789312.0, "3940": 12795789312.0, "3945": 12795789312.0, "3950": 12795789312.0, "3955": 12795789312.0, "3960": 12795789312.0, "3965": 12795789312.0, "3970": 12795789312.0, "3975": 12795789312.0, "3980": 12795789312.0, "3985": 12795789312.0, "3990": 12795789312.0, "3995": 12795789312.0, "4000": 12795789312.0, "4005": 12795789312.0, "4010": 12795789312.0, "4015": 12795789312.0, "4020": 12795789312.0, "4025": 12795789312.0, "4030": 12795789312.0, "4035": 12795789312.0, "4040": 12795789312.0, "4045": 12795789312.0, "4050": 12795789312.0, "4055": 12795789312.0, "4060": 12795789312.0, "4065": 12795789312.0, "4070": 12795789312.0, "4075": 12795789312.0, "4080": 12795789312.0, "4085": 12795789312.0, "4090": 12795789312.0, "4095": 12795789312.0, "4100": 12795789312.0, "4105": 12795789312.0, "4110": 12795789312.0, "4115": 12795789312.0, "4120": 12795789312.0, "4125": 12795789312.0, "4130": 12795789312.0, "4135": 12795789312.0, "4140": 12795789312.0, "4145": 12795789312.0, "4150": 12795789312.0, "4155": 12795789312.0, "4160": 12795789312.0, "4165": 12795789312.0, "4170": 12795789312.0, "4175": 12795789312.0, "4180": 12795789312.0, "4185": 12795789312.0, "4190": 12795789312.0, "4195": 12795789312.0, "4200": 12795789312.0, "4205": 12795789312.0, "4210": 12795789312.0, "4215": 12795789312.0, "4220": 12795789312.0, "4225": 12795789312.0, "4230": 12795789312.0, "4235": 12795789312.0, "4240": 12795789312.0, "4245": 12795789312.0, "4250": 12795789312.0, "4255": 12795789312.0, "4260": 12795789312.0, "4265": 12795789312.0, "4270": 12795789312.0, "4275": 12795789312.0, "4280": 12795789312.0, "4285": 12795789312.0, "4290": 12795789312.0, "4295": 12795789312.0, "4300": 12795789312.0, "4305": 12795789312.0, "4310": 12795789312.0, "4315": 12795789312.0, "4320": 12795789312.0, "4325": 12795789312.0, "4330": 12795789312.0, "4335": 12795789312.0, "4340": 12795789312.0, "4345": 12795789312.0, "4350": 12795789312.0, "4355": 12795789312.0, "4360": 12795789312.0, "4365": 12795789312.0, "4370": 12795789312.0, "4375": 12795789312.0, "4380": 12795789312.0, "4385": 12795789312.0, "4390": 12795789312.0, "4395": 12795789312.0, "4400": 12795789312.0, "4405": 12795789312.0, "4410": 12795789312.0, "4415": 12795789312.0, "4420": 12795789312.0, "4425": 12795789312.0, "4430": 12795789312.0, "4435": 12795789312.0, "4440": 12795789312.0, "4445": 12795789312.0, "4450": 12795789312.0, "4455": 12795789312.0, "4460": 12795789312.0, "4465": 12795789312.0, "4470": 12795789312.0, "4475": 12795789312.0, "4480": 12795789312.0, "4485": 12795789312.0, "4490": 12795789312.0, "4495": 12795789312.0, "4500": 12795789312.0, "4505": 12795789312.0, "4510": 12795789312.0, "4515": 12795789312.0, "4520": 12795789312.0, "4525": 12795789312.0, "4530": 12795789312.0, "4535": 12795789312.0, "4540": 12795789312.0, "4545": 12795789312.0, "4550": 12795789312.0, "4555": 12795789312.0, "4560": 12795789312.0, "4565": 12795789312.0, "4570": 12795789312.0, "4575": 12795789312.0, "4580": 12795789312.0, "4585": 12795789312.0, "4590": 12795789312.0, "4595": 12795789312.0, "4600": 12795789312.0, "4605": 12795789312.0, "4610": 12795789312.0, "4615": 12795789312.0, "4620": 12795789312.0, "4625": 12795789312.0, "4630": 12795789312.0, "4635": 12795789312.0, "4640": 12795789312.0, "4645": 12795789312.0, "4650": 12795789312.0, "4655": 12795789312.0, "4660": 12795789312.0, "4665": 12795789312.0, "4670": 12795789312.0, "4675": 12795789312.0, "4680": 12795789312.0, "4685": 12795789312.0, "4690": 12795789312.0, "4695": 12795789312.0, "4700": 12795789312.0, "4705": 12795789312.0, "4710": 12795789312.0, "4715": 12795789312.0, "4720": 12795789312.0, "4725": 12795789312.0, "4730": 12795789312.0, "4735": 12795789312.0, "4740": 12795789312.0, "4745": 12795789312.0, "4750": 12795789312.0, "4755": 12795789312.0, "4760": 12795789312.0, "4765": 12795789312.0, "4770": 12795789312.0, "4775": 12795789312.0, "4780": 12795789312.0, "4785": 12795789312.0, "4790": 12795789312.0, "4795": 12795789312.0, "4800": 12795789312.0, "4805": 12795789312.0, "4810": 12795789312.0, "4815": 12795789312.0, "4820": 12795789312.0, "4825": 12795789312.0, "4830": 12795789312.0, "4835": 12795789312.0, "4840": 12795789312.0, "4845": 12795789312.0, "4850": 12795789312.0, "4855": 12795789312.0, "4860": 12795789312.0, "4865": 12795789312.0, "4870": 12795789312.0, "4875": 12795789312.0, "4880": 12795789312.0, "4885": 12795789312.0, "4890": 12795789312.0, "4895": 12795789312.0, "4900": 12795789312.0, "4905": 12795789312.0, "4910": 12795789312.0, "4915": 12795789312.0, "4920": 12795789312.0, "4925": 12795789312.0, "4930": 12795789312.0, "4935": 12795789312.0, "4940": 12795789312.0, "4945": 12795789312.0, "4950": 12795789312.0, "4955": 12795789312.0, "4960": 12795789312.0, "4965": 12795789312.0, "4970": 12795789312.0, "4975": 12795789312.0, "4980": 12795789312.0, "4985": 12795789312.0, "4990": 12795789312.0, "4995": 12795789312.0, "5000": 12795789312.0, "5005": 12795789312.0, "5010": 12795789312.0, "5015": 12795789312.0, "5020": 12795789312.0, "5025": 12795789312.0, "5030": 12795789312.0, "5035": 12795789312.0, "5040": 12795789312.0, "5045": 12795789312.0, "5050": 12795789312.0, "5055": 12795789312.0, "5060": 12795789312.0, "5065": 12795789312.0, "5070": 12795789312.0, "5075": 12795789312.0, "5080": 12795789312.0, "5085": 12795789312.0, "5090": 12795789312.0, "5095": 12795789312.0, "5100": 12795789312.0, "5105": 12795789312.0, "5110": 12795789312.0, "5115": 12795789312.0, "5120": 12795789312.0, "5125": 12795789312.0, "5130": 12795789312.0, "5135": 12795789312.0, "5140": 12795789312.0, "5145": 12795789312.0, "5150": 12795789312.0, "5155": 12795789312.0, "5160": 12795789312.0, "5165": 12795789312.0, "5170": 12795789312.0, "5175": 12795789312.0, "5180": 12795789312.0, "5185": 12795789312.0, "5190": 12795789312.0, "5195": 12795789312.0, "5200": 12795789312.0, "5205": 12795789312.0, "5210": 12795789312.0, "5215": 12795789312.0, "5220": 12795789312.0, "5225": 12795789312.0, "5230": 12795789312.0, "5235": 12795789312.0, "5240": 12795789312.0, "5245": 12795789312.0, "5250": 12795789312.0, "5255": 12795789312.0, "5260": 12795789312.0, "5265": 12795789312.0, "5270": 12795789312.0, "5275": 12795789312.0, "5280": 12795789312.0, "5285": 12795789312.0, "5290": 12795789312.0, "5295": 12795789312.0, "5300": 12795789312.0, "5305": 12795789312.0, "5310": 12795789312.0, "5315": 12795789312.0, "5320": 12795789312.0, "5325": 12795789312.0, "5330": 12795789312.0, "5335": 12795789312.0, "5340": 12795789312.0, "5345": 12795789312.0, "5350": 12795789312.0, "5355": 12795789312.0, "5360": 12795789312.0, "5365": 12795789312.0, "5370": 12795789312.0, "5375": 12795789312.0, "5380": 12795789312.0, "5385": 12795789312.0, "5390": 12795789312.0, "5395": 12795789312.0, "5400": 12795789312.0, "5405": 12795789312.0, "5410": 12795789312.0, "5415": 12795789312.0, "5420": 12795789312.0, "5425": 12795789312.0, "5430": 12795789312.0, "5435": 12795789312.0, "5440": 12795789312.0, "5445": 12795789312.0, "5450": 12795789312.0, "5455": 12795789312.0, "5460": 12795789312.0, "5465": 12795789312.0, "5470": 12795789312.0, "5475": 12795789312.0, "5480": 12795789312.0, "5485": 12795789312.0, "5490": 12795789312.0, "5495": 12795789312.0, "5500": 12795789312.0, "5505": 12795789312.0, "5510": 12795789312.0, "5515": 12795789312.0, "5520": 12795789312.0, "5525": 12795789312.0, "5530": 12795789312.0, "5535": 12795789312.0, "5540": 12795789312.0, "5545": 12795789312.0, "5550": 12795789312.0, "5555": 12795789312.0, "5560": 12795789312.0, "5565": 12795789312.0, "5570": 12795789312.0, "5575": 12795789312.0, "5580": 12795789312.0, "5585": 12795789312.0, "5590": 12795789312.0, "5595": 12795789312.0, "5600": 12795789312.0, "5605": 12795789312.0, "5610": 12795789312.0, "5615": 12795789312.0, "5620": 12795789312.0, "5625": 12795789312.0, "5630": 12795789312.0, "5635": 12795789312.0, "5640": 12795789312.0, "5645": 12795789312.0, "5650": 12795789312.0, "5655": 12795789312.0, "5660": 12795789312.0, "5665": 12795789312.0, "5670": 12795789312.0, "5675": 12795789312.0, "5680": 12795789312.0, "5685": 12795789312.0, "5690": 12795789312.0, "5695": 12795789312.0, "5700": 12795789312.0, "5705": 12795789312.0, "5710": 12795789312.0, "5715": 12795789312.0, "5720": 12795789312.0, "5725": 12795789312.0, "5730": 12795789312.0, "5735": 12795789312.0, "5740": 12795789312.0, "5745": 12795789312.0, "5750": 12795789312.0, "5755": 12795789312.0, "5760": 12795789312.0, "5765": 12795789312.0, "5770": 12795789312.0, "5775": 12795789312.0, "5780": 12795789312.0, "5785": 12795789312.0, "5790": 12795789312.0, "5795": 12795789312.0, "5800": 12795789312.0, "5805": 12795789312.0, "5810": 12795789312.0, "5815": 12795789312.0, "5820": 12795789312.0, "5825": 12795789312.0, "5830": 12795789312.0, "5835": 12795789312.0, "5840": 12795789312.0, "5845": 12795789312.0, "5850": 12795789312.0, "5855": 12795789312.0, "5860": 12795789312.0, "5865": 12795789312.0, "5870": 12795789312.0, "5875": 12795789312.0, "5880": 12795789312.0, "5885": 12795789312.0, "5890": 12795789312.0, "5895": 12795789312.0, "5900": 12795789312.0, "5905": 12795789312.0, "5910": 12795789312.0, "5915": 12795789312.0, "5920": 12795789312.0, "5925": 12795789312.0, "5930": 12795789312.0, "5935": 12795789312.0, "5940": 12795789312.0, "5945": 12795789312.0, "5950": 12795789312.0, "5955": 12795789312.0, "5960": 12795789312.0, "5965": 12795789312.0, "5970": 12795789312.0, "5975": 12795789312.0, "5980": 12795789312.0, "5985": 12795789312.0, "5990": 12795789312.0, "5995": 12795789312.0, "6000": 12795789312.0, "6005": 12795789312.0, "6010": 12795789312.0, "6015": 12795789312.0, "6020": 12795789312.0, "6025": 12795789312.0, "6030": 12795789312.0, "6035": 12795789312.0, "6040": 12795789312.0, "6045": 12795789312.0, "6050": 12795789312.0, "6055": 12795789312.0, "6060": 12795789312.0, "6065": 12795789312.0, "6070": 12795789312.0, "6075": 12795789312.0, "6080": 12795789312.0, "6085": 12795789312.0, "6090": 12795789312.0, "6095": 12795789312.0, "6100": 12795789312.0, "6105": 12795789312.0, "6110": 12795789312.0, "6115": 12795789312.0, "6120": 12795789312.0, "6125": 12795789312.0, "6130": 12795789312.0, "6135": 12795789312.0, "6140": 12795789312.0, "6145": 12795789312.0, "6150": 12795789312.0, "6155": 12795789312.0, "6160": 12795789312.0, "6165": 12795789312.0, "6170": 12795789312.0, "6175": 12795789312.0, "6180": 12795789312.0, "6185": 12795789312.0, "6190": 12795789312.0, "6195": 12795789312.0, "6200": 12795789312.0, "6205": 12795789312.0, "6210": 12795789312.0, "6215": 12795789312.0, "6220": 12795789312.0, "6225": 12795789312.0, "6230": 12795789312.0, "6235": 12795789312.0, "6240": 12795789312.0, "6245": 12795789312.0, "6250": 12795789312.0, "6255": 12795789312.0, "6260": 12795789312.0, "6265": 12795789312.0, "6270": 12795789312.0, "6275": 12795789312.0, "6280": 12795789312.0, "6285": 12795789312.0, "6290": 12795789312.0, "6295": 12795789312.0, "6300": 12795789312.0, "6305": 12795789312.0, "6310": 12795789312.0, "6315": 12795789312.0, "6320": 12795789312.0, "6325": 12795789312.0, "6330": 12795789312.0, "6335": 12795789312.0, "6340": 12795789312.0, "6345": 12795789312.0, "6350": 12795789312.0, "6355": 12795789312.0, "6360": 12795789312.0, "6365": 12795789312.0, "6370": 12795789312.0, "6375": 12795789312.0, "6380": 12795789312.0, "6385": 12795789312.0, "6390": 12795789312.0, "6395": 12795789312.0, "6400": 12795789312.0, "6405": 12795789312.0, "6410": 12795789312.0, "6415": 12795789312.0, "6420": 12795789312.0, "6425": 12795789312.0, "6430": 12795789312.0, "6435": 12795789312.0, "6440": 12795789312.0, "6445": 12795789312.0, "6450": 12795789312.0, "6455": 12795789312.0, "6460": 12795789312.0, "6465": 12795789312.0, "6470": 12795789312.0, "6475": 12795789312.0, "6480": 12795789312.0, "6485": 12795789312.0, "6490": 12795789312.0, "6495": 12795789312.0, "6500": 12795789312.0, "6505": 12795789312.0, "6510": 12795789312.0, "6515": 12795789312.0, "6520": 12795789312.0, "6525": 12795789312.0, "6530": 12795789312.0, "6535": 12795789312.0, "6540": 12795789312.0, "6545": 12795789312.0, "6550": 12795789312.0, "6555": 12795789312.0, "6560": 12795789312.0, "6565": 12795789312.0, "6570": 12795789312.0, "6575": 12795789312.0, "6580": 12795789312.0, "6585": 12795789312.0, "6590": 12795789312.0, "6595": 12795789312.0, "6600": 12795789312.0, "6605": 12795789312.0, "6610": 12795789312.0, "6615": 12795789312.0, "6620": 12795789312.0, "6625": 12795789312.0, "6630": 12795789312.0, "6635": 12795789312.0, "6640": 12795789312.0, "6645": 12795789312.0, "6650": 12795789312.0, "6655": 12795789312.0, "6660": 12795789312.0, "6665": 12795789312.0, "6670": 12795789312.0, "6675": 12795789312.0, "6680": 12795789312.0, "6685": 12795789312.0, "6690": 12795789312.0, "6695": 12795789312.0, "6700": 12795789312.0, "6705": 12795789312.0, "6710": 12795789312.0, "6715": 12795789312.0, "6720": 12795789312.0, "6725": 12795789312.0, "6730": 12795789312.0, "6735": 12795789312.0, "6740": 12795789312.0, "6745": 12795789312.0, "6750": 12795789312.0, "6755": 12795789312.0, "6760": 12795789312.0, "6765": 12795789312.0, "6770": 12795789312.0, "6775": 12795789312.0, "6780": 12795789312.0, "6785": 12795789312.0, "6790": 12795789312.0, "6795": 12795789312.0, "6800": 12795789312.0, "6805": 12795789312.0, "6810": 12795789312.0, "6815": 12795789312.0, "6820": 12795789312.0, "6825": 12795789312.0, "6830": 12795789312.0, "6835": 12795789312.0, "6840": 12795789312.0, "6845": 12795789312.0, "6850": 12795789312.0, "6855": 12795789312.0, "6860": 12795789312.0, "6865": 12795789312.0, "6870": 12795789312.0, "6875": 12795789312.0, "6880": 12795789312.0, "6885": 12795789312.0, "6890": 12795789312.0, "6895": 12795789312.0, "6900": 12795789312.0, "6905": 12795789312.0, "6910": 12795789312.0, "6915": 12795789312.0, "6920": 12795789312.0, "6925": 12795789312.0, "6930": 12795789312.0, "6935": 12795789312.0, "6940": 12795789312.0, "6945": 12795789312.0, "6950": 12795789312.0, "6955": 12795789312.0, "6960": 12795789312.0, "6965": 12795789312.0, "6970": 12795789312.0, "6975": 12795789312.0, "6980": 12795789312.0, "6985": 12795789312.0, "6990": 12795789312.0, "6995": 12795789312.0, "7000": 12795789312.0, "7005": 12795789312.0, "7010": 12795789312.0, "7015": 12795789312.0, "7020": 12795789312.0, "7025": 12795789312.0, "7030": 12795789312.0, "7035": 12795789312.0, "7040": 12795789312.0, "7045": 12795789312.0, "7050": 12795789312.0, "7055": 12795789312.0, "7060": 12795789312.0, "7065": 12795789312.0, "7070": 12795789312.0, "7075": 12795789312.0, "7080": 12795789312.0, "7085": 12795789312.0, "7090": 12795789312.0, "7095": 12795789312.0, "7100": 12795789312.0, "7105": 12795789312.0, "7110": 12795789312.0, "7115": 12795789312.0, "7120": 12795789312.0, "7125": 12795789312.0, "7130": 12795789312.0, "7135": 12795789312.0, "7140": 12795789312.0, "7145": 12795789312.0, "7150": 12795789312.0, "7155": 12795789312.0, "7160": 12795789312.0, "7165": 12795789312.0, "7170": 12795789312.0, "7175": 12795789312.0, "7180": 12795789312.0, "7185": 12795789312.0, "7190": 12795789312.0, "7195": 12795789312.0, "7200": 12795789312.0, "7205": 12795789312.0, "7210": 12795789312.0, "7215": 12795789312.0, "7220": 12795789312.0, "7225": 12795789312.0, "7230": 12795789312.0, "7235": 12795789312.0, "7240": 12795789312.0, "7245": 12795789312.0, "7250": 12795789312.0, "7255": 12795789312.0, "7260": 12795789312.0, "7265": 12795789312.0, "7270": 12795789312.0, "7275": 12795789312.0, "7280": 12795789312.0, "7285": 12795789312.0, "7290": 12795789312.0, "7295": 12795789312.0, "7300": 12795789312.0, "7305": 12795789312.0, "7310": 12795789312.0, "7315": 12795789312.0, "7320": 12795789312.0, "7325": 12795789312.0, "7330": 12795789312.0, "7335": 12795789312.0, "7340": 12795789312.0, "7345": 12795789312.0, "7350": 12795789312.0, "7355": 12795789312.0, "7360": 12795789312.0, "7365": 12795789312.0, "7370": 12795789312.0, "7375": 12795789312.0, "7380": 12795789312.0, "7385": 12795789312.0, "7390": 12795789312.0, "7395": 12795789312.0, "7400": 12795789312.0, "7405": 12795789312.0, "7410": 12795789312.0, "7415": 12795789312.0, "7420": 12795789312.0, "7425": 12795789312.0, "7430": 12795789312.0, "7435": 12795789312.0, "7440": 12795789312.0, "7445": 12795789312.0, "7450": 12795789312.0, "7455": 12795789312.0, "7460": 12795789312.0, "7465": 12795789312.0, "7470": 12795789312.0, "7475": 12795789312.0, "7480": 12795789312.0, "7485": 12795789312.0, "7490": 12795789312.0, "7495": 12795789312.0, "7500": 12795789312.0, "7505": 12795789312.0, "7510": 12795789312.0, "7515": 12795789312.0, "7520": 12795789312.0, "7525": 12795789312.0, "7530": 12795789312.0, "7535": 12795789312.0, "7540": 12795789312.0, "7545": 12795789312.0, "7550": 12795789312.0, "7555": 12795789312.0, "7560": 12795789312.0, "7565": 12795789312.0, "7570": 12795789312.0, "7575": 12795789312.0, "7580": 12795789312.0, "7585": 12795789312.0, "7590": 12795789312.0, "7595": 12795789312.0, "7600": 12795789312.0, "7605": 12795789312.0, "7610": 12795789312.0, "7615": 12795789312.0, "7620": 12795789312.0, "7625": 12795789312.0, "7630": 12795789312.0, "7635": 12795789312.0, "7640": 12795789312.0, "7645": 12795789312.0, "7650": 12795789312.0, "7655": 12795789312.0, "7660": 12795789312.0, "7665": 12795789312.0, "7670": 12795789312.0, "7675": 12795789312.0, "7680": 12795789312.0, "7685": 12795789312.0, "7690": 12795789312.0, "7695": 12795789312.0, "7700": 12795789312.0, "7705": 12795789312.0, "7710": 12795789312.0, "7715": 12795789312.0, "7720": 12795789312.0, "7725": 12795789312.0, "7730": 12795789312.0, "7735": 12795789312.0, "7740": 12795789312.0, "7745": 12795789312.0, "7750": 12795789312.0, "7755": 12795789312.0, "7760": 12795789312.0, "7765": 12795789312.0, "7770": 12795789312.0, "7775": 12795789312.0, "7780": 12795789312.0, "7785": 12795789312.0, "7790": 12795789312.0, "7795": 12795789312.0, "7800": 12795789312.0, "7805": 12795789312.0, "7810": 12795789312.0, "7815": 12795789312.0, "7820": 12795789312.0, "7825": 12795789312.0, "7830": 12795789312.0, "7835": 12795789312.0, "7840": 12795789312.0, "7845": 12795789312.0, "7850": 12795789312.0, "7855": 12795789312.0, "7860": 12795789312.0, "7865": 12795789312.0, "7870": 12795789312.0, "7875": 12795789312.0, "7880": 12795789312.0, "7885": 12795789312.0, "7890": 12795789312.0, "7895": 12795789312.0, "7900": 12795789312.0, "7905": 12795789312.0, "7910": 12795789312.0, "7915": 12795789312.0, "7920": 12795789312.0, "7925": 12795789312.0, "7930": 12795789312.0, "7935": 12795789312.0, "7940": 12795789312.0, "7945": 12795789312.0, "7950": 12795789312.0, "7955": 12795789312.0, "7960": 12795789312.0, "7965": 12795789312.0, "7970": 12795789312.0, "7975": 12795789312.0, "7980": 12795789312.0, "7985": 12795789312.0, "7990": 12795789312.0, "7995": 12795789312.0, "8000": 12795789312.0, "8005": 12795789312.0, "8010": 12795789312.0, "8015": 12795789312.0, "8020": 12795789312.0, "8025": 12795789312.0, "8030": 12795789312.0, "8035": 12795789312.0, "8040": 12795789312.0, "8045": 12795789312.0, "8050": 12795789312.0, "8055": 12795789312.0, "8060": 12795789312.0, "8065": 12795789312.0, "8070": 12795789312.0, "8075": 12795789312.0, "8080": 12795789312.0, "8085": 12795789312.0, "8090": 12795789312.0, "8095": 12795789312.0, "8100": 12795789312.0, "8105": 12795789312.0, "8110": 12795789312.0, "8115": 12795789312.0, "8120": 12795789312.0, "8125": 12795789312.0, "8130": 12795789312.0, "8135": 12795789312.0, "8140": 12795789312.0, "8145": 12795789312.0, "8150": 12795789312.0, "8155": 12795789312.0, "8160": 12795789312.0, "8165": 12795789312.0, "8170": 12795789312.0, "8175": 12795789312.0, "8180": 12795789312.0, "8185": 12795789312.0, "8190": 12795789312.0, "8195": 12795789312.0, "8200": 12795789312.0, "8205": 12795789312.0, "8210": 12795789312.0, "8215": 12795789312.0, "8220": 12795789312.0, "8225": 12795789312.0, "8230": 12795789312.0, "8235": 12795789312.0, "8240": 12795789312.0, "8245": 12795789312.0, "8250": 12795789312.0, "8255": 12795789312.0, "8260": 12795789312.0, "8265": 12795789312.0, "8270": 12795789312.0, "8275": 12795789312.0, "8280": 12795789312.0, "8285": 12795789312.0, "8290": 12795789312.0, "8295": 12795789312.0, "8300": 12795789312.0, "8305": 12795789312.0, "8310": 12795789312.0, "8315": 12795789312.0, "8320": 12795789312.0, "8325": 12795789312.0, "8330": 12795789312.0, "8335": 12795789312.0, "8340": 12795789312.0, "8345": 12795789312.0, "8350": 12795789312.0, "8355": 12795789312.0, "8360": 12795789312.0, "8365": 12795789312.0, "8370": 12795789312.0, "8375": 12795789312.0, "8380": 12795789312.0, "8385": 12795789312.0, "8390": 12795789312.0, "8395": 12795789312.0, "8400": 12795789312.0, "8405": 12795789312.0, "8410": 12795789312.0, "8415": 12795789312.0, "8420": 12795789312.0, "8425": 12795789312.0, "8430": 12795789312.0, "8435": 12795789312.0, "8440": 12795789312.0, "8445": 12795789312.0, "8450": 12795789312.0, "8455": 12795789312.0, "8460": 12795789312.0, "8465": 12795789312.0, "8470": 12795789312.0, "8475": 12795789312.0, "8480": 12795789312.0, "8485": 12795789312.0, "8490": 12795789312.0, "8495": 12795789312.0, "8500": 12795789312.0, "8505": 12795789312.0, "8510": 12795789312.0, "8515": 12795789312.0, "8520": 12795789312.0, "8525": 12795789312.0, "8530": 12795789312.0, "8535": 12795789312.0, "8540": 12795789312.0, "8545": 12795789312.0, "8550": 12795789312.0, "8555": 12795789312.0, "8560": 12795789312.0, "8565": 12795789312.0, "8570": 12795789312.0, "8575": 12795789312.0, "8580": 12795789312.0, "8585": 12795789312.0, "8590": 12795789312.0, "8595": 12795789312.0, "8600": 12795789312.0, "8605": 12795789312.0, "8610": 12795789312.0, "8615": 12795789312.0, "8620": 12795789312.0, "8625": 12795789312.0, "8630": 12795789312.0, "8635": 12795789312.0, "8640": 12795789312.0, "8645": 12795789312.0, "8650": 12795789312.0, "8655": 12795789312.0, "8660": 12795789312.0, "8665": 12795789312.0, "8670": 12795789312.0, "8675": 12795789312.0, "8680": 12795789312.0, "8685": 12795789312.0, "8690": 12795789312.0, "8695": 12795789312.0, "8700": 12795789312.0, "8705": 12795789312.0, "8710": 12795789312.0, "8715": 12795789312.0, "8720": 12795789312.0, "8725": 12795789312.0, "8730": 12795789312.0, "8735": 12795789312.0, "8740": 12795789312.0, "8745": 12795789312.0, "8750": 12795789312.0, "8755": 12795789312.0, "8760": 12795789312.0, "8765": 12795789312.0, "8770": 12795789312.0, "8775": 12795789312.0, "8780": 12795789312.0, "8785": 12795789312.0, "8790": 12795789312.0, "8795": 12795789312.0, "8800": 12795789312.0, "8805": 12795789312.0, "8810": 12795789312.0, "8815": 12795789312.0, "8820": 12795789312.0, "8825": 12795789312.0, "8830": 12795789312.0, "8835": 12795789312.0, "8840": 12795789312.0, "8845": 12795789312.0, "8850": 12795789312.0, "8855": 12795789312.0, "8860": 12795789312.0, "8865": 12795789312.0, "8870": 12795789312.0, "8875": 12795789312.0, "8880": 12795789312.0, "8885": 12795789312.0, "8890": 12795789312.0, "8895": 12795789312.0, "8900": 12795789312.0, "8905": 12795789312.0, "8910": 12795789312.0, "8915": 12795789312.0, "8920": 12795789312.0, "8925": 12795789312.0, "8930": 12795789312.0, "8935": 12795789312.0, "8940": 12795789312.0, "8945": 12795789312.0, "8950": 12795789312.0, "8955": 12795789312.0, "8960": 12795789312.0, "8965": 12795789312.0, "8970": 12795789312.0, "8975": 12795789312.0, "8980": 12795789312.0, "8985": 12795789312.0, "8990": 12795789312.0, "8995": 12795789312.0, "9000": 12795789312.0, "9005": 12795789312.0, "9010": 12795789312.0, "9015": 12795789312.0, "9020": 12795789312.0, "9025": 12795789312.0, "9030": 12795789312.0, "9035": 12795789312.0, "9040": 12795789312.0, "9045": 12795789312.0, "9050": 12795789312.0, "9055": 12795789312.0, "9060": 12795789312.0, "9065": 12795789312.0, "9070": 12795789312.0, "9075": 12795789312.0, "9080": 12795789312.0, "9085": 12795789312.0, "9090": 12795789312.0, "9095": 12795789312.0, "9100": 12795789312.0, "9105": 12795789312.0, "9110": 12795789312.0, "9115": 12795789312.0, "9120": 12795789312.0, "9125": 12795789312.0, "9130": 12795789312.0, "9135": 12795789312.0, "9140": 12795789312.0, "9145": 12795789312.0, "9150": 12795789312.0, "9155": 12795789312.0, "9160": 12795789312.0, "9165": 12795789312.0, "9170": 12795789312.0, "9175": 12795789312.0, "9180": 12795789312.0, "9185": 12795789312.0, "9190": 12795789312.0, "9195": 12795789312.0, "9200": 12795789312.0, "9205": 12795789312.0, "9210": 12795789312.0, "9215": 12795789312.0, "9220": 12795789312.0, "9225": 12795789312.0, "9230": 12795789312.0, "9235": 12795789312.0, "9240": 12795789312.0, "9245": 12795789312.0, "9250": 12795789312.0, "9255": 12795789312.0, "9260": 12795789312.0, "9265": 12795789312.0, "9270": 12795789312.0, "9275": 12795789312.0, "9280": 12795789312.0, "9285": 12795789312.0, "9290": 12795789312.0, "9295": 12795789312.0, "9300": 12795789312.0, "9305": 12795789312.0, "9310": 12795789312.0, "9315": 12795789312.0, "9320": 12795789312.0, "9325": 12795789312.0, "9330": 12795789312.0, "9335": 12795789312.0, "9340": 12795789312.0, "9345": 12795789312.0, "9350": 12795789312.0, "9355": 12795789312.0, "9360": 12795789312.0, "9365": 12795789312.0, "9370": 12795789312.0, "9375": 12795789312.0, "9380": 12795789312.0, "9385": 12795789312.0, "9390": 12795789312.0, "9395": 12795789312.0, "9400": 12795789312.0, "9405": 12795789312.0, "9410": 12795789312.0, "9415": 12795789312.0, "9420": 12795789312.0, "9425": 12795789312.0, "9430": 12795789312.0, "9435": 12795789312.0, "9440": 12795789312.0, "9445": 12795789312.0, "9450": 12795789312.0, "9455": 12795789312.0, "9460": 12795789312.0, "9465": 12795789312.0, "9470": 12795789312.0, "9475": 12795789312.0, "9480": 12795789312.0, "9485": 12795789312.0, "9490": 12795789312.0, "9495": 12795789312.0, "9500": 12795789312.0, "9505": 12795789312.0, "9510": 12795789312.0, "9515": 12795789312.0, "9520": 12795789312.0, "9525": 12795789312.0, "9530": 12795789312.0, "9535": 12795789312.0, "9540": 12795789312.0, "9545": 12795789312.0, "9550": 12795789312.0, "9555": 12795789312.0, "9560": 12795789312.0, "9565": 12795789312.0, "9570": 12795789312.0, "9575": 12795789312.0, "9580": 12795789312.0, "9585": 12795789312.0, "9590": 12795789312.0, "9595": 12795789312.0, "9600": 12795789312.0, "9605": 12795789312.0, "9610": 12795789312.0, "9615": 12795789312.0, "9620": 12795789312.0, "9625": 12795789312.0, "9630": 12795789312.0, "9635": 12795789312.0, "9640": 12795789312.0, "9645": 12795789312.0, "9650": 12795789312.0, "9655": 12795789312.0, "9660": 12795789312.0, "9665": 12795789312.0, "9670": 12795789312.0, "9675": 12795789312.0, "9680": 12795789312.0, "9685": 12795789312.0, "9690": 12795789312.0, "9695": 12795789312.0, "9700": 12795789312.0, "9705": 12795789312.0, "9710": 12795789312.0, "9715": 12795789312.0, "9720": 12795789312.0, "9725": 12795789312.0, "9730": 12795789312.0, "9735": 12795789312.0, "9740": 12795789312.0, "9745": 12795789312.0, "9750": 12795789312.0, "9755": 12795789312.0, "9760": 12795789312.0, "9765": 12795789312.0, "9770": 12795789312.0, "9775": 12795789312.0, "9780": 12795789312.0, "9785": 12795789312.0, "9790": 12795789312.0, "9795": 12795789312.0, "9800": 12795789312.0, "9805": 12795789312.0, "9810": 12795789312.0, "9815": 12795789312.0, "9820": 12795789312.0, "9825": 12795789312.0, "9830": 12795789312.0, "9835": 12795789312.0, "9840": 12795789312.0, "9845": 12795789312.0, "9850": 12795789312.0, "9855": 12795789312.0, "9860": 12795789312.0, "9865": 12795789312.0, "9870": 12795789312.0, "9875": 12795789312.0, "9880": 12795789312.0, "9885": 12795789312.0, "9890": 12795789312.0, "9895": 12795789312.0, "9900": 12795789312.0, "9905": 12795789312.0, "9910": 12795789312.0, "9915": 12795789312.0, "9920": 12795789312.0, "9925": 12795789312.0, "9930": 12795789312.0, "9935": 12795789312.0, "9940": 12795789312.0, "9945": 12795789312.0, "9950": 12795789312.0, "9955": 12795789312.0, "9960": 12795789312.0, "9965": 12795789312.0, "9970": 12795789312.0, "9975": 12795789312.0, "9980": 12795789312.0, "9985": 12795789312.0, "9990": 12795789312.0, "9995": 12795789312.0, "10000": 12795789312.0, "10005": 12795789312.0, "10010": 12795789312.0, "10015": 12795789312.0, "10020": 12795789312.0, "10025": 12795789312.0, "10030": 12795789312.0, "10035": 12795789312.0, "10040": 12795789312.0, "10045": 12795789312.0, "10050": 12795789312.0, "10055": 12795789312.0, "10060": 12795789312.0, "10065": 12795789312.0, "10070": 12795789312.0, "10075": 12795789312.0, "10080": 12795789312.0, "10085": 12795789312.0, "10090": 12795789312.0, "10095": 12795789312.0, "10100": 12795789312.0, "10105": 12795789312.0, "10110": 12795789312.0, "10115": 12795789312.0, "10120": 12795789312.0, "10125": 12795789312.0, "10130": 12795789312.0, "10135": 12795789312.0, "10140": 12795789312.0, "10145": 12795789312.0, "10150": 12795789312.0, "10155": 12795789312.0, "10160": 12795789312.0, "10165": 12795789312.0, "10170": 12795789312.0, "10175": 12795789312.0, "10180": 12795789312.0, "10185": 12795789312.0, "10190": 12795789312.0, "10195": 12795789312.0, "10200": 12795789312.0, "10205": 12795789312.0, "10210": 12795789312.0, "10215": 12795789312.0, "10220": 12795789312.0, "10225": 12795789312.0, "10230": 12795789312.0, "10235": 12795789312.0, "10240": 12795789312.0, "10245": 12795789312.0, "10250": 12795789312.0, "10255": 12795789312.0, "10260": 12795789312.0, "10265": 12795789312.0, "10270": 12795789312.0, "10275": 12795789312.0, "10280": 12795789312.0, "10285": 12795789312.0, "10290": 12795789312.0, "10295": 12795789312.0, "10300": 12795789312.0, "10305": 12795789312.0, "10310": 12795789312.0, "10315": 12795789312.0, "10320": 12795789312.0, "10325": 12795789312.0, "10330": 12795789312.0, "10335": 12795789312.0, "10340": 12795789312.0, "10345": 12795789312.0, "10350": 12795789312.0, "10355": 12795789312.0, "10360": 12795789312.0, "10365": 12795789312.0, "10370": 12795789312.0, "10375": 12795789312.0, "10380": 12795789312.0, "10385": 12795789312.0, "10390": 12795789312.0, "10395": 12795789312.0, "10400": 12795789312.0, "10405": 12795789312.0, "10410": 12795789312.0, "10415": 12795789312.0, "10420": 12795789312.0, "10425": 12795789312.0, "10430": 12795789312.0, "10435": 12795789312.0, "10440": 12795789312.0, "10445": 12795789312.0, "10450": 12795789312.0, "10455": 12795789312.0, "10460": 12795789312.0, "10465": 12795789312.0, "10470": 12795789312.0, "10475": 12795789312.0, "10480": 12795789312.0, "10485": 12795789312.0, "10490": 12795789312.0, "10495": 12795789312.0, "10500": 12795789312.0, "10505": 12795789312.0, "10510": 12795789312.0, "10515": 12795789312.0, "10520": 12795789312.0, "10525": 12795789312.0, "10530": 12795789312.0, "10535": 12795789312.0, "10540": 12795789312.0, "10545": 12795789312.0, "10550": 12795789312.0, "10555": 12795789312.0, "10560": 12795789312.0, "10565": 12795789312.0, "10570": 12795789312.0, "10575": 12795789312.0, "10580": 12795789312.0, "10585": 12795789312.0, "10590": 12795789312.0, "10595": 12795789312.0, "10600": 12795789312.0, "10605": 12795789312.0, "10610": 12795789312.0, "10615": 12795789312.0, "10620": 12795789312.0, "10625": 12795789312.0, "10630": 12795789312.0, "10635": 12795789312.0, "10640": 12795789312.0, "10645": 12795789312.0, "10650": 12795789312.0, "10655": 12795789312.0, "10660": 12795789312.0, "10665": 12795789312.0, "10670": 12795789312.0, "10675": 12795789312.0, "10680": 12795789312.0, "10685": 12795789312.0, "10690": 12795789312.0, "10695": 12795789312.0, "10700": 12795789312.0, "10705": 12795789312.0, "10710": 12795789312.0, "10715": 12795789312.0, "10720": 12795789312.0, "10725": 12795789312.0, "10730": 12795789312.0, "10735": 12795789312.0, "10740": 12795789312.0, "10745": 12795789312.0, "10750": 12795789312.0, "10755": 12795789312.0, "10760": 12795789312.0, "10765": 12795789312.0, "10770": 12795789312.0, "10775": 12795789312.0, "10780": 12795789312.0, "10785": 12795789312.0, "10790": 12795789312.0, "10795": 12795789312.0, "10800": 12795789312.0, "10805": 12795789312.0, "10810": 12795789312.0, "10815": 12795789312.0, "10820": 12795789312.0, "10825": 12795789312.0, "10830": 12795789312.0, "10835": 12795789312.0, "10840": 12795789312.0, "10845": 12795789312.0, "10850": 12795789312.0, "10855": 12795789312.0, "10860": 12795789312.0, "10865": 12795789312.0, "10870": 12795789312.0, "10875": 12795789312.0, "10880": 12795789312.0, "10885": 12795789312.0, "10890": 12795789312.0, "10895": 12795789312.0, "10900": 12795789312.0, "10905": 12795789312.0, "10910": 12795789312.0, "10915": 12795789312.0, "10920": 12795789312.0, "10925": 12795789312.0, "10930": 12795789312.0, "10935": 12795789312.0, "10940": 12795789312.0, "10945": 12795789312.0, "10950": 12795789312.0, "10955": 12795789312.0, "10960": 12795789312.0, "10965": 12795789312.0, "10970": 12795789312.0, "10975": 12795789312.0, "10980": 12795789312.0, "10985": 12795789312.0, "10990": 12795789312.0, "10995": 12795789312.0, "11000": 12795789312.0, "11005": 12795789312.0, "11010": 12795789312.0, "11015": 12795789312.0, "11020": 12795789312.0, "11025": 12795789312.0, "11030": 12795789312.0, "11035": 12795789312.0, "11040": 12795789312.0, "11045": 12795789312.0, "11050": 12795789312.0, "11055": 12795789312.0, "11060": 12795789312.0, "11065": 12795789312.0, "11070": 12795789312.0, "11075": 12795789312.0, "11080": 12795789312.0, "11085": 12795789312.0, "11090": 12795789312.0, "11095": 12795789312.0, "11100": 12795789312.0, "11105": 12795789312.0, "11110": 12795789312.0, "11115": 12795789312.0, "11120": 12795789312.0, "11125": 12795789312.0, "11130": 12795789312.0, "11135": 12795789312.0, "11140": 12795789312.0, "11145": 12795789312.0, "11150": 12795789312.0, "11155": 12795789312.0, "11160": 12795789312.0, "11165": 12795789312.0, "11170": 12795789312.0, "11175": 12795789312.0, "11180": 12795789312.0, "11185": 12795789312.0, "11190": 12795789312.0, "11195": 12795789312.0, "11200": 12795789312.0, "11205": 12795789312.0, "11210": 12795789312.0, "11215": 12795789312.0, "11220": 12795789312.0, "11225": 12795789312.0, "11230": 12795789312.0, "11235": 12795789312.0, "11240": 12795789312.0, "11245": 12795789312.0, "11250": 12795789312.0, "11255": 12795789312.0, "11260": 12795789312.0, "11265": 12795789312.0, "11270": 12795789312.0, "11275": 12795789312.0, "11280": 12795789312.0, "11285": 12795789312.0, "11290": 12795789312.0, "11295": 12795789312.0, "11300": 12795789312.0, "11305": 12795789312.0, "11310": 12795789312.0, "11315": 12795789312.0, "11320": 12795789312.0, "11325": 12795789312.0, "11330": 12795789312.0, "11335": 12795789312.0, "11340": 12795789312.0, "11345": 12795789312.0, "11350": 12795789312.0, "11355": 12795789312.0, "11360": 12795789312.0, "11365": 12795789312.0, "11370": 12795789312.0, "11375": 12795789312.0, "11380": 12795789312.0, "11385": 12795789312.0, "11390": 12795789312.0, "11395": 12795789312.0, "11400": 12795789312.0, "11405": 12795789312.0, "11410": 12795789312.0, "11415": 12795789312.0, "11420": 12795789312.0, "11425": 12795789312.0, "11430": 12795789312.0, "11435": 12795789312.0, "11440": 12795789312.0, "11445": 12795789312.0, "11450": 12795789312.0, "11455": 12795789312.0, "11460": 12795789312.0, "11465": 12795789312.0, "11470": 12795789312.0, "11475": 12795789312.0, "11480": 12795789312.0, "11485": 12795789312.0, "11490": 12795789312.0, "11495": 12795789312.0, "11500": 12795789312.0, "11505": 12795789312.0, "11510": 12795789312.0, "11515": 12795789312.0, "11520": 12795789312.0, "11525": 12795789312.0, "11530": 12795789312.0, "11535": 12795789312.0, "11540": 12795789312.0, "11545": 12795789312.0, "11550": 12795789312.0, "11555": 12795789312.0, "11560": 12795789312.0, "11565": 12795789312.0, "11570": 12795789312.0, "11575": 12795789312.0, "11580": 12795789312.0, "11585": 12795789312.0, "11590": 12795789312.0, "11595": 12795789312.0, "11600": 12795789312.0, "11605": 12795789312.0, "11610": 12795789312.0, "11615": 12795789312.0, "11620": 12795789312.0, "11625": 12795789312.0, "11630": 12795789312.0, "11635": 12795789312.0, "11640": 12795789312.0, "11645": 12795789312.0, "11650": 12795789312.0, "11655": 12795789312.0, "11660": 12795789312.0, "11665": 12795789312.0, "11670": 12795789312.0, "11675": 12795789312.0, "11680": 12795789312.0, "11685": 12795789312.0, "11690": 12795789312.0, "11695": 12795789312.0, "11700": 12795789312.0, "11705": 12795789312.0, "11710": 12795789312.0, "11715": 12795789312.0, "11720": 12795789312.0, "11725": 12795789312.0, "11730": 12795789312.0, "11735": 12795789312.0, "11740": 12795789312.0, "11745": 12795789312.0, "11750": 12795789312.0, "11755": 12795789312.0, "11760": 12795789312.0, "11765": 12795789312.0, "11770": 12795789312.0, "11775": 12795789312.0, "11780": 12795789312.0, "11785": 12795789312.0, "11790": 12795789312.0, "11795": 12795789312.0, "11800": 12795789312.0, "11805": 12795789312.0, "11810": 12795789312.0, "11815": 12795789312.0, "11820": 12795789312.0, "11825": 12795789312.0, "11830": 12795789312.0, "11835": 12795789312.0, "11840": 12795789312.0, "11845": 12795789312.0, "11850": 12795789312.0, "11855": 12795789312.0, "11860": 12795789312.0, "11865": 12795789312.0, "11870": 12795789312.0, "11875": 12795789312.0, "11880": 12795789312.0, "11885": 12795789312.0, "11890": 12795789312.0, "11895": 12795789312.0, "11900": 12795789312.0, "11905": 12795789312.0, "11910": 12795789312.0, "11915": 12795789312.0, "11920": 12795789312.0, "11925": 12795789312.0, "11930": 12795789312.0, "11935": 12795789312.0, "11940": 12795789312.0, "11945": 12795789312.0, "11950": 12795789312.0, "11955": 12795789312.0, "11960": 12795789312.0, "11965": 12795789312.0, "11970": 12795789312.0, "11975": 12795789312.0, "11980": 12795789312.0, "11985": 12795789312.0, "11990": 12795789312.0, "11995": 12795789312.0, "12000": 12795789312.0, "12005": 12795789312.0, "12010": 12795789312.0, "12015": 12795789312.0, "12020": 12795789312.0, "12025": 12795789312.0, "12030": 12795789312.0, "12035": 12795789312.0, "12040": 12795789312.0, "12045": 12795789312.0, "12050": 12795789312.0, "12055": 12795789312.0, "12060": 12795789312.0, "12065": 12795789312.0, "12070": 12795789312.0, "12075": 12795789312.0, "12080": 12795789312.0, "12085": 12795789312.0, "12090": 12795789312.0, "12095": 12795789312.0, "12100": 12795789312.0, "12105": 12795789312.0, "12110": 12795789312.0, "12115": 12795789312.0, "12120": 12795789312.0, "12125": 12795789312.0, "12130": 12795789312.0, "12135": 12795789312.0, "12140": 12795789312.0, "12145": 12795789312.0, "12150": 12795789312.0, "12155": 12795789312.0, "12160": 12795789312.0, "12165": 12795789312.0, "12170": 12795789312.0, "12175": 12795789312.0, "12180": 12795789312.0, "12185": 12795789312.0, "12190": 12795789312.0, "12195": 12795789312.0, "12200": 12795789312.0, "12205": 12795789312.0, "12210": 12795789312.0, "12215": 12795789312.0, "12220": 12795789312.0, "12225": 12795789312.0, "12230": 12795789312.0, "12235": 12795789312.0, "12240": 12795789312.0, "12245": 12795789312.0, "12250": 12795789312.0, "12255": 12795789312.0, "12260": 12795789312.0, "12265": 12795789312.0, "12270": 12795789312.0, "12275": 12795789312.0, "12280": 12795789312.0, "12285": 12795789312.0, "12290": 12795789312.0, "12295": 12795789312.0, "12300": 12795789312.0, "12305": 12795789312.0, "12310": 12795789312.0, "12315": 12795789312.0, "12320": 12795789312.0, "12325": 12795789312.0, "12330": 12795789312.0, "12335": 12795789312.0, "12340": 12795789312.0, "12345": 12795789312.0, "12350": 12795789312.0, "12355": 12795789312.0, "12360": 12795789312.0, "12365": 12795789312.0, "12370": 12795789312.0, "12375": 12795789312.0, "12380": 12795789312.0, "12385": 12795789312.0, "12390": 12795789312.0, "12395": 12795789312.0, "12400": 12795789312.0, "12405": 12795789312.0, "12410": 12795789312.0, "12415": 12795789312.0, "12420": 12795789312.0, "12425": 12795789312.0, "12430": 12795789312.0, "12435": 12795789312.0, "12440": 12795789312.0, "12445": 12795789312.0, "12450": 12795789312.0, "12455": 12795789312.0, "12460": 12795789312.0, "12465": 12795789312.0, "12470": 12795789312.0, "12475": 12795789312.0, "12480": 12795789312.0, "12485": 12795789312.0, "12490": 12795789312.0, "12495": 12795789312.0, "12500": 12795789312.0, "12505": 12795789312.0, "12510": 12795789312.0, "12515": 12795789312.0, "12520": 12795789312.0, "12525": 12795789312.0, "12530": 12795789312.0, "12535": 12795789312.0, "12540": 12795789312.0, "12545": 12795789312.0, "12550": 12795789312.0, "12555": 12795789312.0, "12560": 12795789312.0, "12565": 12795789312.0, "12570": 12795789312.0, "12575": 12795789312.0, "12580": 12795789312.0, "12585": 12795789312.0, "12590": 12795789312.0, "12595": 12795789312.0, "12600": 12795789312.0, "12605": 12795789312.0, "12610": 12795789312.0, "12615": 12795789312.0, "12620": 12795789312.0, "12625": 12795789312.0, "12630": 12795789312.0, "12635": 12795789312.0, "12640": 12795789312.0, "12645": 12795789312.0, "12650": 12795789312.0, "12655": 12795789312.0, "12660": 12795789312.0, "12665": 12795789312.0, "12670": 12795789312.0, "12675": 12795789312.0, "12680": 12795789312.0, "12685": 12795789312.0, "12690": 12795789312.0, "12695": 12795789312.0, "12700": 12795789312.0, "12705": 12795789312.0, "12710": 12795789312.0, "12715": 12795789312.0, "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}, "mem-max-allocated-bytes": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": 27991298048.0, "5": 28489385984.0, "10": 28489385984.0, "15": 28489385984.0, "20": 28489385984.0, "25": 28489385984.0, "30": 28489385984.0, "35": 28489385984.0, "40": 28489385984.0, "45": 28489385984.0, "50": 28489385984.0, "55": 28489385984.0, "60": 28489385984.0, "65": 28489385984.0, "70": 28489385984.0, "75": 28489385984.0, "80": 28489385984.0, "85": 28489385984.0, "90": 28489385984.0, "95": 28489385984.0, "100": 28489385984.0, "105": 28489385984.0, "110": 28489385984.0, "115": 28489385984.0, "120": 28489385984.0, "125": 28489385984.0, "130": 28489385984.0, "135": 28489385984.0, "140": 28489385984.0, "145": 28489385984.0, "150": 28489385984.0, "155": 28489385984.0, "160": 28489385984.0, "165": 28489385984.0, "170": 28489385984.0, "175": 28489385984.0, "180": 28489385984.0, "185": 28489385984.0, "190": 28489385984.0, "195": 28489385984.0, "200": 28489385984.0, "205": 28489385984.0, "210": 28489385984.0, "215": 28489385984.0, "220": 28489385984.0, "225": 28489385984.0, "230": 28489385984.0, "235": 28489385984.0, "240": 28489385984.0, "245": 28489385984.0, "250": 28489385984.0, "255": 28489385984.0, "260": 28489385984.0, "265": 28489385984.0, "270": 28489385984.0, "275": 28489385984.0, "280": 28489385984.0, "285": 28489385984.0, "290": 28489385984.0, "295": 28489385984.0, "300": 28489385984.0, "305": 28489385984.0, "310": 28489385984.0, "315": 28489385984.0, "320": 28489385984.0, "325": 28489385984.0, "330": 28489385984.0, "335": 28489385984.0, "340": 28489385984.0, "345": 28489385984.0, "350": 28489385984.0, "355": 28489385984.0, "360": 28489385984.0, "365": 28489385984.0, "370": 28489385984.0, "375": 28489385984.0, "380": 28489385984.0, "385": 28489385984.0, "390": 28489385984.0, "395": 28489385984.0, "400": 28489385984.0, "405": 28489385984.0, "410": 28489385984.0, "415": 28489385984.0, "420": 28489385984.0, "425": 28489385984.0, "430": 28489385984.0, "435": 28489385984.0, "440": 28489385984.0, "445": 28489385984.0, "450": 28489385984.0, "455": 28489385984.0, "460": 28489385984.0, "465": 28489385984.0, "470": 28489385984.0, "475": 28489385984.0, "480": 28489385984.0, "485": 28489385984.0, "490": 28489385984.0, "495": 28489385984.0, "500": 28489385984.0, "505": 28489385984.0, "510": 28489385984.0, "515": 28489385984.0, "520": 28489385984.0, "525": 28489385984.0, "530": 28489385984.0, "535": 28489385984.0, "540": 28489385984.0, "545": 28489385984.0, "550": 28489385984.0, "555": 28489385984.0, "560": 28489385984.0, "565": 28489385984.0, "570": 28489385984.0, "575": 28489385984.0, "580": 28489385984.0, "585": 28489385984.0, "590": 28489385984.0, "595": 28489385984.0, "600": 28489385984.0, "605": 28489385984.0, "610": 28489385984.0, "615": 28489385984.0, "620": 28489385984.0, "625": 28489385984.0, "630": 28489385984.0, "635": 28489385984.0, "640": 28489385984.0, "645": 28489385984.0, "650": 28489385984.0, "655": 28489385984.0, "660": 28489385984.0, "665": 28489385984.0, "670": 28489385984.0, "675": 28489385984.0, "680": 28489385984.0, "685": 28489385984.0, "690": 28489385984.0, "695": 28489385984.0, "700": 28489385984.0, "705": 28489385984.0, "710": 28489385984.0, "715": 28489385984.0, "720": 28489385984.0, "725": 28489385984.0, "730": 28489385984.0, "735": 28489385984.0, "740": 28489385984.0, "745": 28489385984.0, "750": 28489385984.0, "755": 28489385984.0, "760": 28489385984.0, "765": 28489385984.0, "770": 28489385984.0, "775": 28489385984.0, "780": 28489385984.0, "785": 28489385984.0, "790": 28489385984.0, "795": 28489385984.0, "800": 28489385984.0, "805": 28489385984.0, "810": 28489385984.0, "815": 28489385984.0, "820": 28489385984.0, "825": 28489385984.0, "830": 28489385984.0, "835": 28489385984.0, "840": 28489385984.0, "845": 28489385984.0, "850": 28489385984.0, "855": 28489385984.0, "860": 28489385984.0, "865": 28489385984.0, "870": 28489385984.0, "875": 28489385984.0, "880": 28489385984.0, "885": 28489385984.0, "890": 28489385984.0, "895": 28489385984.0, "900": 28489385984.0, "905": 28489385984.0, "910": 28489385984.0, "915": 28489385984.0, "920": 28489385984.0, "925": 28489385984.0, "930": 28489385984.0, "935": 28489385984.0, "940": 28489385984.0, "945": 28489385984.0, "950": 28489385984.0, "955": 28489385984.0, "960": 28489385984.0, "965": 28489385984.0, "970": 28489385984.0, "975": 28489385984.0, "980": 28489385984.0, "985": 28489385984.0, "990": 28489385984.0, "995": 28489385984.0, "1000": 28489385984.0, "1005": 28489385984.0, "1010": 28489385984.0, "1015": 28489385984.0, "1020": 28489385984.0, "1025": 28489385984.0, "1030": 28489385984.0, "1035": 28489385984.0, "1040": 28489385984.0, "1045": 28489385984.0, "1050": 28489385984.0, "1055": 28489385984.0, "1060": 28489385984.0, "1065": 28489385984.0, "1070": 28489385984.0, "1075": 28489385984.0, "1080": 28489385984.0, "1085": 28489385984.0, "1090": 28489385984.0, "1095": 28489385984.0, "1100": 28489385984.0, "1105": 28489385984.0, "1110": 28489385984.0, "1115": 28489385984.0, "1120": 28489385984.0, "1125": 28489385984.0, "1130": 28489385984.0, "1135": 28489385984.0, "1140": 28489385984.0, "1145": 28489385984.0, "1150": 28489385984.0, "1155": 28489385984.0, "1160": 28489385984.0, "1165": 28489385984.0, "1170": 28489385984.0, "1175": 28489385984.0, "1180": 28489385984.0, "1185": 28489385984.0, "1190": 28489385984.0, "1195": 28489385984.0, "1200": 28489385984.0, "1205": 28489385984.0, "1210": 28489385984.0, "1215": 28489385984.0, "1220": 28489385984.0, "1225": 28489385984.0, "1230": 28489385984.0, "1235": 28489385984.0, "1240": 28489385984.0, "1245": 28489385984.0, "1250": 28489385984.0, "1255": 28489385984.0, "1260": 28489385984.0, "1265": 28489385984.0, "1270": 28489385984.0, "1275": 28489385984.0, "1280": 28489385984.0, "1285": 28489385984.0, "1290": 28489385984.0, "1295": 28489385984.0, "1300": 28489385984.0, "1305": 28489385984.0, "1310": 28489385984.0, "1315": 28489385984.0, "1320": 28489385984.0, "1325": 28489385984.0, "1330": 28489385984.0, "1335": 28489385984.0, "1340": 28489385984.0, "1345": 28489385984.0, "1350": 28489385984.0, "1355": 28489385984.0, "1360": 28489385984.0, "1365": 28489385984.0, "1370": 28489385984.0, "1375": 28489385984.0, "1380": 28489385984.0, "1385": 28489385984.0, "1390": 28489385984.0, "1395": 28489385984.0, "1400": 28489385984.0, "1405": 28489385984.0, "1410": 28489385984.0, "1415": 28489385984.0, "1420": 28489385984.0, "1425": 28489385984.0, "1430": 28489385984.0, "1435": 28489385984.0, "1440": 28489385984.0, "1445": 28489385984.0, "1450": 28489385984.0, "1455": 28489385984.0, "1460": 28489385984.0, "1465": 28489385984.0, "1470": 28489385984.0, "1475": 28489385984.0, "1480": 28489385984.0, "1485": 28489385984.0, "1490": 28489385984.0, "1495": 28489385984.0, "1500": 28489385984.0, "1505": 28489385984.0, "1510": 28489385984.0, "1515": 28489385984.0, "1520": 28489385984.0, "1525": 28489385984.0, "1530": 28489385984.0, "1535": 28489385984.0, "1540": 28489385984.0, "1545": 28489385984.0, "1550": 28489385984.0, "1555": 28489385984.0, "1560": 28489385984.0, "1565": 28489385984.0, "1570": 28489385984.0, "1575": 28489385984.0, "1580": 28489385984.0, "1585": 28489385984.0, "1590": 28489385984.0, "1595": 28489385984.0, "1600": 28489385984.0, "1605": 28489385984.0, "1610": 28489385984.0, "1615": 28489385984.0, "1620": 28489385984.0, "1625": 28489385984.0, "1630": 28489385984.0, "1635": 28489385984.0, "1640": 28489385984.0, "1645": 28489385984.0, "1650": 28489385984.0, "1655": 28489385984.0, "1660": 28489385984.0, "1665": 28489385984.0, "1670": 28489385984.0, "1675": 28489385984.0, "1680": 28489385984.0, "1685": 28489385984.0, "1690": 28489385984.0, "1695": 28489385984.0, "1700": 28489385984.0, "1705": 28489385984.0, "1710": 28489385984.0, "1715": 28489385984.0, "1720": 28489385984.0, "1725": 28489385984.0, "1730": 28489385984.0, "1735": 28489385984.0, "1740": 28489385984.0, "1745": 28489385984.0, "1750": 28489385984.0, "1755": 28489385984.0, "1760": 28489385984.0, "1765": 28489385984.0, "1770": 28489385984.0, "1775": 28489385984.0, "1780": 28489385984.0, "1785": 28489385984.0, "1790": 28489385984.0, "1795": 28489385984.0, "1800": 28489385984.0, "1805": 28489385984.0, "1810": 28489385984.0, "1815": 28489385984.0, "1820": 28489385984.0, "1825": 28489385984.0, "1830": 28489385984.0, "1835": 28489385984.0, "1840": 28489385984.0, "1845": 28489385984.0, "1850": 28489385984.0, "1855": 28489385984.0, "1860": 28489385984.0, "1865": 28489385984.0, "1870": 28489385984.0, "1875": 28489385984.0, "1880": 28489385984.0, "1885": 28489385984.0, "1890": 28489385984.0, "1895": 28489385984.0, "1900": 28489385984.0, "1905": 28489385984.0, "1910": 28489385984.0, "1915": 28489385984.0, "1920": 28489385984.0, "1925": 28489385984.0, "1930": 28489385984.0, "1935": 28489385984.0, "1940": 28489385984.0, "1945": 28489385984.0, "1950": 28489385984.0, "1955": 28489385984.0, "1960": 28489385984.0, "1965": 28489385984.0, "1970": 28489385984.0, "1975": 28489385984.0, "1980": 28489385984.0, "1985": 28489385984.0, "1990": 28489385984.0, "1995": 28489385984.0, "2000": 28489385984.0, "2005": 28489385984.0, "2010": 28489385984.0, "2015": 28489385984.0, "2020": 28489385984.0, "2025": 28489385984.0, "2030": 28489385984.0, "2035": 28489385984.0, "2040": 28489385984.0, "2045": 28489385984.0, "2050": 28489385984.0, "2055": 28489385984.0, "2060": 28489385984.0, "2065": 28489385984.0, "2070": 28489385984.0, "2075": 28489385984.0, "2080": 28489385984.0, "2085": 28489385984.0, "2090": 28489385984.0, "2095": 28489385984.0, "2100": 28489385984.0, "2105": 28489385984.0, "2110": 28489385984.0, "2115": 28489385984.0, "2120": 28489385984.0, "2125": 28489385984.0, "2130": 28489385984.0, "2135": 28489385984.0, "2140": 28489385984.0, "2145": 28489385984.0, "2150": 28489385984.0, "2155": 28489385984.0, "2160": 28489385984.0, "2165": 28489385984.0, "2170": 28489385984.0, "2175": 28489385984.0, "2180": 28489385984.0, "2185": 28489385984.0, "2190": 28489385984.0, "2195": 28489385984.0, "2200": 28489385984.0, "2205": 28489385984.0, "2210": 28489385984.0, "2215": 28489385984.0, "2220": 28489385984.0, "2225": 28489385984.0, "2230": 28489385984.0, "2235": 28489385984.0, "2240": 28489385984.0, "2245": 28489385984.0, "2250": 28489385984.0, "2255": 28489385984.0, "2260": 28489385984.0, "2265": 28489385984.0, "2270": 28489385984.0, "2275": 28489385984.0, "2280": 28489385984.0, "2285": 28489385984.0, "2290": 28489385984.0, "2295": 28489385984.0, "2300": 28489385984.0, "2305": 28489385984.0, "2310": 28489385984.0, "2315": 28489385984.0, "2320": 28489385984.0, "2325": 28489385984.0, "2330": 28489385984.0, "2335": 28489385984.0, "2340": 28489385984.0, "2345": 28489385984.0, "2350": 28489385984.0, "2355": 28489385984.0, "2360": 28489385984.0, "2365": 28489385984.0, "2370": 28489385984.0, "2375": 28489385984.0, "2380": 28489385984.0, "2385": 28489385984.0, "2390": 28489385984.0, "2395": 28489385984.0, "2400": 28489385984.0, "2405": 28489385984.0, "2410": 28489385984.0, "2415": 28489385984.0, "2420": 28489385984.0, "2425": 28489385984.0, "2430": 28489385984.0, "2435": 28489385984.0, "2440": 28489385984.0, "2445": 28489385984.0, "2450": 28489385984.0, "2455": 28489385984.0, "2460": 28489385984.0, "2465": 28489385984.0, "2470": 28489385984.0, "2475": 28489385984.0, "2480": 28489385984.0, "2485": 28489385984.0, "2490": 28489385984.0, "2495": 28489385984.0, "2500": 28489385984.0, "2505": 28489385984.0, "2510": 28489385984.0, "2515": 28489385984.0, "2520": 28489385984.0, "2525": 28489385984.0, "2530": 28489385984.0, "2535": 28489385984.0, "2540": 28489385984.0, "2545": 28489385984.0, "2550": 28489385984.0, "2555": 28489385984.0, "2560": 28489385984.0, "2565": 28489385984.0, "2570": 28489385984.0, "2575": 28489385984.0, "2580": 28489385984.0, "2585": 28489385984.0, "2590": 28489385984.0, "2595": 28489385984.0, "2600": 28489385984.0, "2605": 28489385984.0, "2610": 28489385984.0, "2615": 28489385984.0, "2620": 28489385984.0, "2625": 28489385984.0, "2630": 28489385984.0, "2635": 28489385984.0, "2640": 28489385984.0, "2645": 28489385984.0, "2650": 28489385984.0, "2655": 28489385984.0, "2660": 28489385984.0, "2665": 28489385984.0, "2670": 28489385984.0, "2675": 28489385984.0, "2680": 28489385984.0, "2685": 28489385984.0, "2690": 28489385984.0, "2695": 28489385984.0, "2700": 28489385984.0, "2705": 28489385984.0, "2710": 28489385984.0, "2715": 28489385984.0, "2720": 28489385984.0, "2725": 28489385984.0, "2730": 28489385984.0, "2735": 28489385984.0, "2740": 28489385984.0, "2745": 28489385984.0, "2750": 28489385984.0, "2755": 28489385984.0, "2760": 28489385984.0, "2765": 28489385984.0, "2770": 28489385984.0, "2775": 28489385984.0, "2780": 28489385984.0, "2785": 28489385984.0, "2790": 28489385984.0, "2795": 28489385984.0, "2800": 28489385984.0, "2805": 28489385984.0, "2810": 28489385984.0, "2815": 28489385984.0, "2820": 28489385984.0, "2825": 28489385984.0, "2830": 28489385984.0, "2835": 28489385984.0, "2840": 28489385984.0, "2845": 28489385984.0, "2850": 28489385984.0, "2855": 28489385984.0, "2860": 28489385984.0, "2865": 28489385984.0, "2870": 28489385984.0, "2875": 28489385984.0, "2880": 28489385984.0, "2885": 28489385984.0, "2890": 28489385984.0, "2895": 28489385984.0, "2900": 28489385984.0, "2905": 28489385984.0, "2910": 28489385984.0, "2915": 28489385984.0, "2920": 28489385984.0, "2925": 28489385984.0, "2930": 28489385984.0, "2935": 28489385984.0, "2940": 28489385984.0, "2945": 28489385984.0, "2950": 28489385984.0, "2955": 28489385984.0, "2960": 28489385984.0, "2965": 28489385984.0, "2970": 28489385984.0, "2975": 28489385984.0, "2980": 28489385984.0, "2985": 28489385984.0, "2990": 28489385984.0, "2995": 28489385984.0, "3000": 28489385984.0, "3005": 28489385984.0, "3010": 28489385984.0, "3015": 28489385984.0, "3020": 28489385984.0, "3025": 28489385984.0, "3030": 28489385984.0, "3035": 28489385984.0, "3040": 28489385984.0, "3045": 28489385984.0, "3050": 28489385984.0, "3055": 28489385984.0, "3060": 28489385984.0, "3065": 28489385984.0, "3070": 28489385984.0, "3075": 28489385984.0, "3080": 28489385984.0, "3085": 28489385984.0, "3090": 28489385984.0, "3095": 28489385984.0, "3100": 28489385984.0, "3105": 28489385984.0, "3110": 28489385984.0, "3115": 28489385984.0, "3120": 28489385984.0, "3125": 28489385984.0, "3130": 28489385984.0, "3135": 28489385984.0, "3140": 28489385984.0, "3145": 28489385984.0, "3150": 28489385984.0, "3155": 28489385984.0, "3160": 28489385984.0, "3165": 28489385984.0, "3170": 28489385984.0, "3175": 28489385984.0, "3180": 28489385984.0, "3185": 28489385984.0, "3190": 28489385984.0, "3195": 28489385984.0, "3200": 28489385984.0, "3205": 28489385984.0, "3210": 28489385984.0, "3215": 28489385984.0, "3220": 28489385984.0, "3225": 28489385984.0, "3230": 28489385984.0, "3235": 28489385984.0, "3240": 28489385984.0, "3245": 28489385984.0, "3250": 28489385984.0, "3255": 28489385984.0, "3260": 28489385984.0, "3265": 28489385984.0, "3270": 28489385984.0, "3275": 28489385984.0, "3280": 28489385984.0, "3285": 28489385984.0, "3290": 28489385984.0, "3295": 28489385984.0, "3300": 28489385984.0, "3305": 28489385984.0, "3310": 28489385984.0, "3315": 28489385984.0, "3320": 28489385984.0, "3325": 28489385984.0, "3330": 28489385984.0, "3335": 28489385984.0, "3340": 28489385984.0, "3345": 28489385984.0, "3350": 28489385984.0, "3355": 28489385984.0, "3360": 28489385984.0, "3365": 28489385984.0, "3370": 28489385984.0, "3375": 28489385984.0, "3380": 28489385984.0, "3385": 28489385984.0, "3390": 28489385984.0, "3395": 28489385984.0, "3400": 28489385984.0, "3405": 28489385984.0, "3410": 28489385984.0, "3415": 28489385984.0, "3420": 28489385984.0, "3425": 28489385984.0, "3430": 28489385984.0, "3435": 28489385984.0, "3440": 28489385984.0, "3445": 28489385984.0, "3450": 28489385984.0, "3455": 28489385984.0, "3460": 28489385984.0, "3465": 28489385984.0, "3470": 28489385984.0, "3475": 28489385984.0, "3480": 28489385984.0, "3485": 28489385984.0, "3490": 28489385984.0, "3495": 28489385984.0, "3500": 28489385984.0, "3505": 28489385984.0, "3510": 28489385984.0, "3515": 28489385984.0, "3520": 28489385984.0, "3525": 28489385984.0, "3530": 28489385984.0, "3535": 28489385984.0, "3540": 28489385984.0, "3545": 28489385984.0, "3550": 28489385984.0, "3555": 28489385984.0, "3560": 28489385984.0, "3565": 28489385984.0, "3570": 28489568256.0, "3575": 28489568256.0, "3580": 28489568256.0, "3585": 28489568256.0, "3590": 28489568256.0, "3595": 28489568256.0, "3600": 28489568256.0, "3605": 28489568256.0, "3610": 28489568256.0, "3615": 28489568256.0, "3620": 28489568256.0, "3625": 28489568256.0, "3630": 28489568256.0, "3635": 28489568256.0, "3640": 28489568256.0, "3645": 28489568256.0, "3650": 28489568256.0, "3655": 28489568256.0, "3660": 28489568256.0, "3665": 28489568256.0, "3670": 28489568256.0, "3675": 28489568256.0, "3680": 28489568256.0, "3685": 28489568256.0, "3690": 28489568256.0, "3695": 28489568256.0, "3700": 28489568256.0, "3705": 28489568256.0, "3710": 28489568256.0, "3715": 28489568256.0, "3720": 28489568256.0, "3725": 28489568256.0, "3730": 28489568256.0, "3735": 28489568256.0, "3740": 28489568256.0, "3745": 28489568256.0, "3750": 28489568256.0, "3755": 28489568256.0, "3760": 28489568256.0, "3765": 28489568256.0, "3770": 28489568256.0, "3775": 28489568256.0, "3780": 28489568256.0, "3785": 28489568256.0, "3790": 28489568256.0, "3795": 28489568256.0, "3800": 28489568256.0, "3805": 28489568256.0, "3810": 28489568256.0, "3815": 28489568256.0, "3820": 28489568256.0, "3825": 28489568256.0, "3830": 28489568256.0, "3835": 28489568256.0, "3840": 28489568256.0, "3845": 28489568256.0, "3850": 28489568256.0, "3855": 28489568256.0, "3860": 28489568256.0, "3865": 28489568256.0, "3870": 28489568256.0, "3875": 28489568256.0, "3880": 28489568256.0, "3885": 28489568256.0, "3890": 28489568256.0, "3895": 28489568256.0, "3900": 28489568256.0, "3905": 28489568256.0, "3910": 28489568256.0, "3915": 28489568256.0, "3920": 28489568256.0, "3925": 28489568256.0, "3930": 28489568256.0, "3935": 28489568256.0, "3940": 28489568256.0, "3945": 28489568256.0, "3950": 28489568256.0, "3955": 28489568256.0, "3960": 28489568256.0, "3965": 28489568256.0, "3970": 28489568256.0, "3975": 28489568256.0, "3980": 28489568256.0, "3985": 28489568256.0, "3990": 28489568256.0, "3995": 28489568256.0, "4000": 28489568256.0, "4005": 28489568256.0, "4010": 28489568256.0, "4015": 28489568256.0, "4020": 28489568256.0, "4025": 28489568256.0, "4030": 28489568256.0, "4035": 28489568256.0, "4040": 28489568256.0, "4045": 28489568256.0, "4050": 28489568256.0, "4055": 28489568256.0, "4060": 28489568256.0, "4065": 28489568256.0, "4070": 28489568256.0, "4075": 28489568256.0, "4080": 28489568256.0, "4085": 28489568256.0, "4090": 28489568256.0, "4095": 28489568256.0, "4100": 28489568256.0, "4105": 28489568256.0, "4110": 28489568256.0, "4115": 28489568256.0, "4120": 28489568256.0, "4125": 28489568256.0, "4130": 28489568256.0, "4135": 28489568256.0, "4140": 28489568256.0, "4145": 28489568256.0, "4150": 28489568256.0, "4155": 28489568256.0, "4160": 28489568256.0, "4165": 28489568256.0, "4170": 28489568256.0, "4175": 28489568256.0, "4180": 28489568256.0, "4185": 28489568256.0, "4190": 28489568256.0, "4195": 28489568256.0, "4200": 28489568256.0, "4205": 28489568256.0, "4210": 28489568256.0, "4215": 28489568256.0, "4220": 28489568256.0, "4225": 28489568256.0, "4230": 28489568256.0, "4235": 28489568256.0, "4240": 28489568256.0, "4245": 28489568256.0, "4250": 28489568256.0, "4255": 28489568256.0, "4260": 28489568256.0, "4265": 28489568256.0, "4270": 28489568256.0, "4275": 28489568256.0, "4280": 28489568256.0, "4285": 28489568256.0, "4290": 28489568256.0, "4295": 28489568256.0, "4300": 28489568256.0, "4305": 28489568256.0, "4310": 28489568256.0, "4315": 28489568256.0, "4320": 28489568256.0, "4325": 28489568256.0, "4330": 28489568256.0, "4335": 28489568256.0, "4340": 28489568256.0, "4345": 28489568256.0, "4350": 28489568256.0, "4355": 28489568256.0, "4360": 28489568256.0, "4365": 28489568256.0, "4370": 28489568256.0, "4375": 28489568256.0, "4380": 28489568256.0, "4385": 28489568256.0, "4390": 28489568256.0, "4395": 28489568256.0, "4400": 28489568256.0, "4405": 28489568256.0, "4410": 28489568256.0, "4415": 28489568256.0, "4420": 28489568256.0, "4425": 28489568256.0, "4430": 28489568256.0, "4435": 28489568256.0, "4440": 28489568256.0, "4445": 28489568256.0, "4450": 28489568256.0, "4455": 28489568256.0, "4460": 28489568256.0, "4465": 28489568256.0, "4470": 28489568256.0, "4475": 28489568256.0, "4480": 28489568256.0, "4485": 28489568256.0, "4490": 28489568256.0, "4495": 28489568256.0, "4500": 28489568256.0, "4505": 28489568256.0, "4510": 28489568256.0, "4515": 28489568256.0, "4520": 28489568256.0, "4525": 28489568256.0, "4530": 28489568256.0, "4535": 28489568256.0, "4540": 28489568256.0, "4545": 28489568256.0, "4550": 28489568256.0, "4555": 28489568256.0, "4560": 28489568256.0, "4565": 28489568256.0, "4570": 28489568256.0, "4575": 28489568256.0, "4580": 28489568256.0, "4585": 28489568256.0, "4590": 28489568256.0, "4595": 28489568256.0, "4600": 28489568256.0, "4605": 28489568256.0, "4610": 28489568256.0, "4615": 28489568256.0, "4620": 28489568256.0, "4625": 28489568256.0, "4630": 28489568256.0, "4635": 28489568256.0, "4640": 28489568256.0, "4645": 28489568256.0, "4650": 28489568256.0, "4655": 28489568256.0, "4660": 28489568256.0, "4665": 28489568256.0, "4670": 28489568256.0, "4675": 28489568256.0, "4680": 28489568256.0, "4685": 28489568256.0, "4690": 28489568256.0, "4695": 28489568256.0, "4700": 28489568256.0, "4705": 28489568256.0, "4710": 28489568256.0, "4715": 28489568256.0, "4720": 28489568256.0, "4725": 28489568256.0, "4730": 28489568256.0, "4735": 28489568256.0, "4740": 28489568256.0, "4745": 28489568256.0, "4750": 28489568256.0, "4755": 28489568256.0, "4760": 28489568256.0, "4765": 28489568256.0, "4770": 28489568256.0, "4775": 28489568256.0, "4780": 28489568256.0, "4785": 28489568256.0, "4790": 28489568256.0, "4795": 28489568256.0, "4800": 28489568256.0, "4805": 28489568256.0, "4810": 28489568256.0, "4815": 28489568256.0, "4820": 28489568256.0, "4825": 28489568256.0, "4830": 28489568256.0, "4835": 28489568256.0, "4840": 28489568256.0, "4845": 28489568256.0, "4850": 28489568256.0, "4855": 28489568256.0, "4860": 28489568256.0, "4865": 28489568256.0, "4870": 28489568256.0, "4875": 28489568256.0, "4880": 28489568256.0, "4885": 28489568256.0, "4890": 28489568256.0, "4895": 28489568256.0, "4900": 28489568256.0, "4905": 28489568256.0, "4910": 28489568256.0, "4915": 28489568256.0, "4920": 28489568256.0, "4925": 28489568256.0, "4930": 28489568256.0, "4935": 28489568256.0, "4940": 28489568256.0, "4945": 28489568256.0, "4950": 28489568256.0, "4955": 28489568256.0, "4960": 28489568256.0, "4965": 28489568256.0, "4970": 28489568256.0, "4975": 28489568256.0, "4980": 28489568256.0, "4985": 28489568256.0, "4990": 28489568256.0, "4995": 28489568256.0, "5000": 28489568256.0, "5005": 28489568256.0, "5010": 28489568256.0, "5015": 28489568256.0, "5020": 28489568256.0, "5025": 28489568256.0, "5030": 28489568256.0, "5035": 28489568256.0, "5040": 28489568256.0, "5045": 28489568256.0, "5050": 28489568256.0, "5055": 28489568256.0, "5060": 28489568256.0, "5065": 28489568256.0, "5070": 28489568256.0, "5075": 28489568256.0, "5080": 28489568256.0, "5085": 28489568256.0, "5090": 28489568256.0, "5095": 28489568256.0, "5100": 28489568256.0, "5105": 28489568256.0, "5110": 28489568256.0, "5115": 28489568256.0, "5120": 28489568256.0, "5125": 28489568256.0, "5130": 28489568256.0, "5135": 28489568256.0, "5140": 28489568256.0, "5145": 28489568256.0, "5150": 28489568256.0, "5155": 28489568256.0, "5160": 28489568256.0, "5165": 28489568256.0, "5170": 28489568256.0, "5175": 28489568256.0, "5180": 28489568256.0, "5185": 28489568256.0, "5190": 28489568256.0, "5195": 28489568256.0, "5200": 28489568256.0, "5205": 28489568256.0, "5210": 28489568256.0, "5215": 28489568256.0, "5220": 28489568256.0, "5225": 28489568256.0, "5230": 28489568256.0, "5235": 28489568256.0, "5240": 28489568256.0, "5245": 28489568256.0, "5250": 28489568256.0, "5255": 28489568256.0, "5260": 28489568256.0, "5265": 28489568256.0, "5270": 28489568256.0, "5275": 28489568256.0, "5280": 28489568256.0, "5285": 28489568256.0, "5290": 28489568256.0, "5295": 28489568256.0, "5300": 28489568256.0, "5305": 28489568256.0, "5310": 28489568256.0, "5315": 28489568256.0, "5320": 28489568256.0, "5325": 28489568256.0, "5330": 28489568256.0, "5335": 28489568256.0, "5340": 28489568256.0, "5345": 28489568256.0, "5350": 28489568256.0, "5355": 28489568256.0, "5360": 28489568256.0, "5365": 28489568256.0, "5370": 28489568256.0, "5375": 28489568256.0, "5380": 28489568256.0, "5385": 28489568256.0, "5390": 28489568256.0, "5395": 28489568256.0, "5400": 28489568256.0, "5405": 28489568256.0, "5410": 28489568256.0, "5415": 28489568256.0, "5420": 28489568256.0, "5425": 28489568256.0, "5430": 28489568256.0, "5435": 28489568256.0, "5440": 28489568256.0, "5445": 28489568256.0, "5450": 28489568256.0, "5455": 28489568256.0, "5460": 28489568256.0, "5465": 28489568256.0, "5470": 28489568256.0, "5475": 28489568256.0, "5480": 28489568256.0, "5485": 28489568256.0, "5490": 28489568256.0, "5495": 28489568256.0, "5500": 28489568256.0, "5505": 28489568256.0, "5510": 28489568256.0, "5515": 28489568256.0, "5520": 28489568256.0, "5525": 28489568256.0, "5530": 28489568256.0, "5535": 28489568256.0, "5540": 28489568256.0, "5545": 28489568256.0, "5550": 28489568256.0, "5555": 28489568256.0, "5560": 28489568256.0, "5565": 28489568256.0, "5570": 28489568256.0, "5575": 28489568256.0, "5580": 28489568256.0, "5585": 28489568256.0, "5590": 28489568256.0, "5595": 28489568256.0, "5600": 28489568256.0, "5605": 28489568256.0, "5610": 28489568256.0, "5615": 28489568256.0, "5620": 28489568256.0, "5625": 28489568256.0, "5630": 28489568256.0, "5635": 28489568256.0, "5640": 28489568256.0, "5645": 28489568256.0, "5650": 28489568256.0, "5655": 28489568256.0, "5660": 28489568256.0, "5665": 28489568256.0, "5670": 28489568256.0, "5675": 28489568256.0, "5680": 28489568256.0, "5685": 28489568256.0, "5690": 28489568256.0, "5695": 28489568256.0, "5700": 28489568256.0, "5705": 28489568256.0, "5710": 28489568256.0, "5715": 28489568256.0, "5720": 28489568256.0, "5725": 28489568256.0, "5730": 28489568256.0, "5735": 28489568256.0, "5740": 28489568256.0, "5745": 28489568256.0, "5750": 28489568256.0, "5755": 28489568256.0, "5760": 28489568256.0, "5765": 28489568256.0, "5770": 28489568256.0, "5775": 28489568256.0, "5780": 28489568256.0, "5785": 28489568256.0, "5790": 28489568256.0, "5795": 28489568256.0, "5800": 28489568256.0, "5805": 28489568256.0, "5810": 28489568256.0, "5815": 28489568256.0, "5820": 28489568256.0, "5825": 28489568256.0, "5830": 28489568256.0, "5835": 28489568256.0, "5840": 28489568256.0, "5845": 28489568256.0, "5850": 28489568256.0, "5855": 28489568256.0, "5860": 28489568256.0, "5865": 28489568256.0, "5870": 28489568256.0, "5875": 28489568256.0, "5880": 28489568256.0, "5885": 28489568256.0, "5890": 28489568256.0, "5895": 28489568256.0, "5900": 28489568256.0, "5905": 28489568256.0, "5910": 28489568256.0, "5915": 28489568256.0, "5920": 28489568256.0, "5925": 28489568256.0, "5930": 28489568256.0, "5935": 28489568256.0, "5940": 28489568256.0, "5945": 28489568256.0, "5950": 28489568256.0, "5955": 28489568256.0, "5960": 28489568256.0, "5965": 28489568256.0, "5970": 28489568256.0, "5975": 28489568256.0, "5980": 28489568256.0, "5985": 28489568256.0, "5990": 28489568256.0, "5995": 28489568256.0, "6000": 28489568256.0, "6005": 28489568256.0, "6010": 28489568256.0, "6015": 28489568256.0, "6020": 28489568256.0, "6025": 28489568256.0, "6030": 28489568256.0, "6035": 28489568256.0, "6040": 28489568256.0, "6045": 28489568256.0, "6050": 28489568256.0, "6055": 28489568256.0, "6060": 28489568256.0, "6065": 28489568256.0, "6070": 28489568256.0, "6075": 28489568256.0, "6080": 28489568256.0, "6085": 28489568256.0, "6090": 28489568256.0, "6095": 28489568256.0, "6100": 28489568256.0, "6105": 28489568256.0, "6110": 28489568256.0, "6115": 28489568256.0, "6120": 28489568256.0, "6125": 28489568256.0, "6130": 28489568256.0, "6135": 28489568256.0, "6140": 28489568256.0, "6145": 28489568256.0, "6150": 28489568256.0, "6155": 28489568256.0, "6160": 28489568256.0, "6165": 28489568256.0, "6170": 28489568256.0, "6175": 28489568256.0, "6180": 28489568256.0, "6185": 28489568256.0, "6190": 28489568256.0, "6195": 28489568256.0, "6200": 28489568256.0, "6205": 28489568256.0, "6210": 28489568256.0, "6215": 28489568256.0, "6220": 28489568256.0, "6225": 28489568256.0, "6230": 28489568256.0, "6235": 28489568256.0, "6240": 28489568256.0, "6245": 28489568256.0, "6250": 28489568256.0, "6255": 28489568256.0, "6260": 28489568256.0, "6265": 28489568256.0, "6270": 28489568256.0, "6275": 28489568256.0, "6280": 28489568256.0, "6285": 28489568256.0, "6290": 28489568256.0, "6295": 28489568256.0, "6300": 28489568256.0, "6305": 28489568256.0, "6310": 28489568256.0, "6315": 28489568256.0, "6320": 28489568256.0, "6325": 28489568256.0, "6330": 28489568256.0, "6335": 28489568256.0, "6340": 28489568256.0, "6345": 28489568256.0, "6350": 28489568256.0, "6355": 28489568256.0, "6360": 28489568256.0, "6365": 28489568256.0, "6370": 28489568256.0, "6375": 28489568256.0, "6380": 28489568256.0, "6385": 28489568256.0, "6390": 28489568256.0, "6395": 28489568256.0, "6400": 28489568256.0, "6405": 28489568256.0, "6410": 28489568256.0, "6415": 28489568256.0, "6420": 28489568256.0, "6425": 28489568256.0, "6430": 28489568256.0, "6435": 28489568256.0, "6440": 28489568256.0, "6445": 28489568256.0, "6450": 28489568256.0, "6455": 28489568256.0, "6460": 28489568256.0, "6465": 28489568256.0, "6470": 28489568256.0, "6475": 28489568256.0, "6480": 28489568256.0, "6485": 28489568256.0, "6490": 28489568256.0, "6495": 28489568256.0, "6500": 28489568256.0, "6505": 28489568256.0, "6510": 28489568256.0, "6515": 28489568256.0, "6520": 28489568256.0, "6525": 28489568256.0, "6530": 28489568256.0, "6535": 28489568256.0, "6540": 28489568256.0, "6545": 28489568256.0, "6550": 28489568256.0, "6555": 28489568256.0, "6560": 28489568256.0, "6565": 28489568256.0, "6570": 28489568256.0, "6575": 28489568256.0, "6580": 28489568256.0, "6585": 28489568256.0, "6590": 28489568256.0, "6595": 28489568256.0, "6600": 28489568256.0, "6605": 28489568256.0, "6610": 28489568256.0, "6615": 28489568256.0, "6620": 28489568256.0, "6625": 28489568256.0, "6630": 28489568256.0, "6635": 28489568256.0, "6640": 28489568256.0, "6645": 28489568256.0, "6650": 28489568256.0, "6655": 28489568256.0, "6660": 28489568256.0, "6665": 28489568256.0, "6670": 28489568256.0, "6675": 28489568256.0, "6680": 28489568256.0, "6685": 28489568256.0, "6690": 28489568256.0, "6695": 28489568256.0, "6700": 28489568256.0, "6705": 28489568256.0, "6710": 28489568256.0, "6715": 28489568256.0, "6720": 28489568256.0, "6725": 28489568256.0, "6730": 28489568256.0, "6735": 28489568256.0, "6740": 28489568256.0, "6745": 28489568256.0, "6750": 28489568256.0, "6755": 28489568256.0, "6760": 28489568256.0, "6765": 28489568256.0, "6770": 28489568256.0, "6775": 28489568256.0, "6780": 28489568256.0, "6785": 28489568256.0, "6790": 28489568256.0, "6795": 28489568256.0, "6800": 28489568256.0, "6805": 28489568256.0, "6810": 28489568256.0, "6815": 28489568256.0, "6820": 28489568256.0, "6825": 28489568256.0, "6830": 28489568256.0, "6835": 28489568256.0, "6840": 28489568256.0, "6845": 28489568256.0, "6850": 28489568256.0, "6855": 28489568256.0, "6860": 28489568256.0, "6865": 28489568256.0, "6870": 28489568256.0, "6875": 28489568256.0, "6880": 28489568256.0, "6885": 28489568256.0, "6890": 28489568256.0, "6895": 28489568256.0, "6900": 28489568256.0, "6905": 28489568256.0, "6910": 28489568256.0, "6915": 28489568256.0, "6920": 28489568256.0, "6925": 28489568256.0, "6930": 28489568256.0, "6935": 28489568256.0, "6940": 28489568256.0, "6945": 28489568256.0, "6950": 28489568256.0, "6955": 28489568256.0, "6960": 28489568256.0, "6965": 28489568256.0, "6970": 28489568256.0, "6975": 28489568256.0, "6980": 28489568256.0, "6985": 28489568256.0, "6990": 28489568256.0, "6995": 28489568256.0, "7000": 28489568256.0, "7005": 28489568256.0, "7010": 28489568256.0, "7015": 28489568256.0, "7020": 28489568256.0, "7025": 28489568256.0, "7030": 28489568256.0, "7035": 28489568256.0, "7040": 28489568256.0, "7045": 28489568256.0, "7050": 28489568256.0, "7055": 28489568256.0, "7060": 28489568256.0, "7065": 28489568256.0, "7070": 28489568256.0, "7075": 28489568256.0, "7080": 28489568256.0, "7085": 28489568256.0, "7090": 28489568256.0, "7095": 28489568256.0, "7100": 28489568256.0, "7105": 28489568256.0, "7110": 28489568256.0, "7115": 28489568256.0, "7120": 28489568256.0, "7125": 28489568256.0, "7130": 28489568256.0, "7135": 28489568256.0, "7140": 28489568256.0, "7145": 28489568256.0, "7150": 28489568256.0, "7155": 28489568256.0, "7160": 28489568256.0, "7165": 28489568256.0, "7170": 28489568256.0, "7175": 28489568256.0, "7180": 28489568256.0, "7185": 28489568256.0, "7190": 28489568256.0, "7195": 28489568256.0, "7200": 28489568256.0, "7205": 28489568256.0, "7210": 28489568256.0, "7215": 28489568256.0, "7220": 28489568256.0, "7225": 28489568256.0, "7230": 28489568256.0, "7235": 28489568256.0, "7240": 28489568256.0, "7245": 28489568256.0, "7250": 28489568256.0, "7255": 28489568256.0, "7260": 28489568256.0, "7265": 28489568256.0, "7270": 28489568256.0, "7275": 28489568256.0, "7280": 28489568256.0, "7285": 28489568256.0, "7290": 28489568256.0, "7295": 28489568256.0, "7300": 28489568256.0, "7305": 28489568256.0, "7310": 28489568256.0, "7315": 28489568256.0, "7320": 28489568256.0, "7325": 28489568256.0, "7330": 28489568256.0, "7335": 28489568256.0, "7340": 28489568256.0, "7345": 28489568256.0, "7350": 28489568256.0, "7355": 28489568256.0, "7360": 28489568256.0, "7365": 28489568256.0, "7370": 28489568256.0, "7375": 28489568256.0, "7380": 28489568256.0, "7385": 28489568256.0, "7390": 28489568256.0, "7395": 28489568256.0, "7400": 28489568256.0, "7405": 28489568256.0, "7410": 28489568256.0, "7415": 28489568256.0, "7420": 28489568256.0, "7425": 28489568256.0, "7430": 28489568256.0, "7435": 28489568256.0, "7440": 28489568256.0, "7445": 28489568256.0, "7450": 28489568256.0, "7455": 28489568256.0, "7460": 28489568256.0, "7465": 28489568256.0, "7470": 28489568256.0, "7475": 28489568256.0, "7480": 28489568256.0, "7485": 28489568256.0, "7490": 28489568256.0, "7495": 28489568256.0, "7500": 28489568256.0, "7505": 28489568256.0, "7510": 28489568256.0, "7515": 28489568256.0, "7520": 28489568256.0, "7525": 28489568256.0, "7530": 28489568256.0, "7535": 28489568256.0, "7540": 28489568256.0, "7545": 28489568256.0, "7550": 28489568256.0, "7555": 28489568256.0, "7560": 28489568256.0, "7565": 28489568256.0, "7570": 28489568256.0, "7575": 28489568256.0, "7580": 28489568256.0, "7585": 28489568256.0, "7590": 28489568256.0, "7595": 28489568256.0, "7600": 28489568256.0, "7605": 28489568256.0, "7610": 28489568256.0, "7615": 28489568256.0, "7620": 28489568256.0, "7625": 28489568256.0, "7630": 28489568256.0, "7635": 28489568256.0, "7640": 28489568256.0, "7645": 28489568256.0, "7650": 28489568256.0, "7655": 28489568256.0, "7660": 28489568256.0, "7665": 28489568256.0, "7670": 28489568256.0, "7675": 28489568256.0, "7680": 28489568256.0, "7685": 28489568256.0, "7690": 28489568256.0, "7695": 28489568256.0, "7700": 28489568256.0, "7705": 28489568256.0, "7710": 28489568256.0, "7715": 28489568256.0, "7720": 28489568256.0, "7725": 28489568256.0, "7730": 28489568256.0, "7735": 28489568256.0, "7740": 28489568256.0, "7745": 28489568256.0, "7750": 28489568256.0, "7755": 28489568256.0, "7760": 28489568256.0, "7765": 28489568256.0, "7770": 28489568256.0, "7775": 28489568256.0, "7780": 28489568256.0, "7785": 28489568256.0, "7790": 28489568256.0, "7795": 28489568256.0, "7800": 28489568256.0, "7805": 28489568256.0, "7810": 28489568256.0, "7815": 28489568256.0, "7820": 28489568256.0, "7825": 28489568256.0, "7830": 28489568256.0, "7835": 28489568256.0, "7840": 28489568256.0, "7845": 28489568256.0, "7850": 28489568256.0, "7855": 28489568256.0, "7860": 28489568256.0, "7865": 28489568256.0, "7870": 28489568256.0, "7875": 28489568256.0, "7880": 28489568256.0, "7885": 28489568256.0, "7890": 28489568256.0, "7895": 28489568256.0, "7900": 28489568256.0, "7905": 28489568256.0, "7910": 28489568256.0, "7915": 28489568256.0, "7920": 28489568256.0, "7925": 28489568256.0, "7930": 28489568256.0, "7935": 28489568256.0, "7940": 28489568256.0, "7945": 28489568256.0, "7950": 28489568256.0, "7955": 28489568256.0, "7960": 28489568256.0, "7965": 28489568256.0, "7970": 28489568256.0, "7975": 28489568256.0, "7980": 28489568256.0, "7985": 28489568256.0, "7990": 28489568256.0, "7995": 28489568256.0, "8000": 28489568256.0, "8005": 28489568256.0, "8010": 28489568256.0, "8015": 28489568256.0, "8020": 28489568256.0, "8025": 28489568256.0, "8030": 28489568256.0, "8035": 28489568256.0, "8040": 28489568256.0, "8045": 28489568256.0, "8050": 28489568256.0, "8055": 28489568256.0, "8060": 28489568256.0, "8065": 28489568256.0, "8070": 28489568256.0, "8075": 28489568256.0, "8080": 28489568256.0, "8085": 28489568256.0, "8090": 28489568256.0, "8095": 28489568256.0, "8100": 28489568256.0, "8105": 28489568256.0, "8110": 28489568256.0, "8115": 28489568256.0, "8120": 28489568256.0, "8125": 28489568256.0, "8130": 28489568256.0, "8135": 28489568256.0, "8140": 28489568256.0, "8145": 28489568256.0, "8150": 28489568256.0, "8155": 28489568256.0, "8160": 28489568256.0, "8165": 28489568256.0, "8170": 28489568256.0, "8175": 28489568256.0, "8180": 28489568256.0, "8185": 28489568256.0, "8190": 28489568256.0, "8195": 28489568256.0, "8200": 28489568256.0, "8205": 28489568256.0, "8210": 28489568256.0, "8215": 28489568256.0, "8220": 28489568256.0, "8225": 28489568256.0, "8230": 28489568256.0, "8235": 28489568256.0, "8240": 28489568256.0, "8245": 28489568256.0, "8250": 28489568256.0, "8255": 28489568256.0, "8260": 28489568256.0, "8265": 28489568256.0, "8270": 28489568256.0, "8275": 28489568256.0, "8280": 28489568256.0, "8285": 28489568256.0, "8290": 28489568256.0, "8295": 28489568256.0, "8300": 28489568256.0, "8305": 28489568256.0, "8310": 28489568256.0, "8315": 28489568256.0, "8320": 28489568256.0, "8325": 28489568256.0, "8330": 28489568256.0, "8335": 28489568256.0, "8340": 28489568256.0, "8345": 28489568256.0, "8350": 28489568256.0, "8355": 28489568256.0, "8360": 28489568256.0, "8365": 28489568256.0, "8370": 28489568256.0, "8375": 28489568256.0, "8380": 28489568256.0, "8385": 28489568256.0, "8390": 28489568256.0, "8395": 28489568256.0, "8400": 28489568256.0, "8405": 28489568256.0, "8410": 28489568256.0, "8415": 28489568256.0, "8420": 28489568256.0, "8425": 28489568256.0, "8430": 28489568256.0, "8435": 28489568256.0, "8440": 28489568256.0, "8445": 28489568256.0, "8450": 28489568256.0, "8455": 28489568256.0, "8460": 28489568256.0, "8465": 28489568256.0, "8470": 28489568256.0, "8475": 28489568256.0, "8480": 28489568256.0, "8485": 28489568256.0, "8490": 28489568256.0, "8495": 28489568256.0, "8500": 28489568256.0, "8505": 28489568256.0, "8510": 28489568256.0, "8515": 28489568256.0, "8520": 28489568256.0, "8525": 28489568256.0, "8530": 28489568256.0, "8535": 28489568256.0, "8540": 28489568256.0, "8545": 28489568256.0, "8550": 28489568256.0, "8555": 28489568256.0, "8560": 28489568256.0, "8565": 28489568256.0, "8570": 28489568256.0, "8575": 28489568256.0, "8580": 28489568256.0, "8585": 28489568256.0, "8590": 28489568256.0, "8595": 28489568256.0, "8600": 28489568256.0, "8605": 28489568256.0, "8610": 28489568256.0, "8615": 28489568256.0, "8620": 28489568256.0, "8625": 28489568256.0, "8630": 28489568256.0, "8635": 28489568256.0, "8640": 28489568256.0, "8645": 28489568256.0, "8650": 28489568256.0, "8655": 28489568256.0, "8660": 28489568256.0, "8665": 28489568256.0, "8670": 28489568256.0, "8675": 28489568256.0, "8680": 28489568256.0, "8685": 28489568256.0, "8690": 28489568256.0, "8695": 28489568256.0, "8700": 28489568256.0, "8705": 28489568256.0, "8710": 28489568256.0, "8715": 28489568256.0, "8720": 28489568256.0, "8725": 28489568256.0, "8730": 28489568256.0, "8735": 28489568256.0, "8740": 28489568256.0, "8745": 28489568256.0, "8750": 28489568256.0, "8755": 28489568256.0, "8760": 28489568256.0, "8765": 28489568256.0, "8770": 28489568256.0, "8775": 28489568256.0, "8780": 28489568256.0, "8785": 28489568256.0, "8790": 28489568256.0, "8795": 28489568256.0, "8800": 28489568256.0, "8805": 28489568256.0, "8810": 28489568256.0, "8815": 28489568256.0, "8820": 28489568256.0, "8825": 28489568256.0, "8830": 28489568256.0, "8835": 28489568256.0, "8840": 28489568256.0, "8845": 28489568256.0, "8850": 28489568256.0, "8855": 28489568256.0, "8860": 28489568256.0, "8865": 28489568256.0, "8870": 28489568256.0, "8875": 28489568256.0, "8880": 28489568256.0, "8885": 28489568256.0, "8890": 28489568256.0, "8895": 28489568256.0, "8900": 28489568256.0, "8905": 28489568256.0, "8910": 28489568256.0, "8915": 28489568256.0, "8920": 28489568256.0, "8925": 28489568256.0, "8930": 28489568256.0, "8935": 28489568256.0, "8940": 28489568256.0, "8945": 28489568256.0, "8950": 28489568256.0, "8955": 28489568256.0, "8960": 28489568256.0, "8965": 28489568256.0, "8970": 28489568256.0, "8975": 28489568256.0, "8980": 28489568256.0, "8985": 28489568256.0, "8990": 28489568256.0, "8995": 28489568256.0, "9000": 28489568256.0, "9005": 28489568256.0, "9010": 28489568256.0, "9015": 28489568256.0, "9020": 28489568256.0, "9025": 28489568256.0, "9030": 28489568256.0, "9035": 28489568256.0, "9040": 28489568256.0, "9045": 28489568256.0, "9050": 28489568256.0, "9055": 28489568256.0, "9060": 28489568256.0, "9065": 28489568256.0, "9070": 28489568256.0, "9075": 28489568256.0, "9080": 28489568256.0, "9085": 28489568256.0, "9090": 28489568256.0, "9095": 28489568256.0, "9100": 28489568256.0, "9105": 28489568256.0, "9110": 28489568256.0, "9115": 28489568256.0, "9120": 28489568256.0, "9125": 28489568256.0, "9130": 28489568256.0, "9135": 28489568256.0, "9140": 28489568256.0, "9145": 28489568256.0, "9150": 28489568256.0, "9155": 28489568256.0, "9160": 28489568256.0, "9165": 28489568256.0, "9170": 28489568256.0, "9175": 28489568256.0, "9180": 28489568256.0, "9185": 28489568256.0, "9190": 28489568256.0, "9195": 28489568256.0, "9200": 28489568256.0, "9205": 28489568256.0, "9210": 28489568256.0, "9215": 28489568256.0, "9220": 28489568256.0, "9225": 28489568256.0, "9230": 28489568256.0, "9235": 28489568256.0, "9240": 28489568256.0, "9245": 28489568256.0, "9250": 28489568256.0, "9255": 28489568256.0, "9260": 28489568256.0, "9265": 28489568256.0, "9270": 28489568256.0, "9275": 28489568256.0, "9280": 28489568256.0, "9285": 28489568256.0, "9290": 28489568256.0, "9295": 28489568256.0, "9300": 28489568256.0, "9305": 28489568256.0, "9310": 28489568256.0, "9315": 28489568256.0, "9320": 28489568256.0, "9325": 28489568256.0, "9330": 28489568256.0, "9335": 28489568256.0, "9340": 28489568256.0, "9345": 28489568256.0, "9350": 28489568256.0, "9355": 28489568256.0, "9360": 28489568256.0, "9365": 28489568256.0, "9370": 28489568256.0, "9375": 28489568256.0, "9380": 28489568256.0, "9385": 28489568256.0, "9390": 28489568256.0, "9395": 28489568256.0, "9400": 28489568256.0, "9405": 28489568256.0, "9410": 28489568256.0, "9415": 28489568256.0, "9420": 28489568256.0, "9425": 28489568256.0, "9430": 28489568256.0, "9435": 28489568256.0, "9440": 28489568256.0, "9445": 28489568256.0, "9450": 28489568256.0, "9455": 28489568256.0, "9460": 28489568256.0, "9465": 28489568256.0, "9470": 28489568256.0, "9475": 28489568256.0, "9480": 28489568256.0, "9485": 28489568256.0, "9490": 28489568256.0, "9495": 28489568256.0, "9500": 28489568256.0, "9505": 28489568256.0, "9510": 28489568256.0, "9515": 28489568256.0, "9520": 28489568256.0, "9525": 28489568256.0, "9530": 28489568256.0, "9535": 28489568256.0, "9540": 28489568256.0, "9545": 28489568256.0, "9550": 28489568256.0, "9555": 28489568256.0, "9560": 28489568256.0, "9565": 28489568256.0, "9570": 28489568256.0, "9575": 28489568256.0, "9580": 28489568256.0, "9585": 28489568256.0, "9590": 28489568256.0, "9595": 28489568256.0, "9600": 28489568256.0, "9605": 28489568256.0, "9610": 28489568256.0, "9615": 28489568256.0, "9620": 28489568256.0, "9625": 28489568256.0, "9630": 28489568256.0, "9635": 28489568256.0, "9640": 28489568256.0, "9645": 28489568256.0, "9650": 28489568256.0, "9655": 28489568256.0, "9660": 28489568256.0, "9665": 28489568256.0, "9670": 28489568256.0, "9675": 28489568256.0, "9680": 28489568256.0, "9685": 28489568256.0, "9690": 28489568256.0, "9695": 28489568256.0, "9700": 28489568256.0, "9705": 28489568256.0, "9710": 28489568256.0, "9715": 28489568256.0, "9720": 28489568256.0, "9725": 28489568256.0, "9730": 28489568256.0, "9735": 28489568256.0, "9740": 28489568256.0, "9745": 28489568256.0, "9750": 28489568256.0, "9755": 28489568256.0, "9760": 28489568256.0, "9765": 28489568256.0, "9770": 28489568256.0, "9775": 28489568256.0, "9780": 28489568256.0, "9785": 28489568256.0, "9790": 28489568256.0, "9795": 28489568256.0, "9800": 28489568256.0, "9805": 28489568256.0, "9810": 28489568256.0, "9815": 28489568256.0, "9820": 28489568256.0, "9825": 28489568256.0, "9830": 28489568256.0, "9835": 28489568256.0, "9840": 28489568256.0, "9845": 28489568256.0, "9850": 28489568256.0, "9855": 28489568256.0, "9860": 28489568256.0, "9865": 28489568256.0, "9870": 28489568256.0, "9875": 28489568256.0, "9880": 28489568256.0, "9885": 28489568256.0, "9890": 28489568256.0, "9895": 28489568256.0, "9900": 28489568256.0, "9905": 28489568256.0, "9910": 28489568256.0, "9915": 28489568256.0, "9920": 28489568256.0, "9925": 28489568256.0, "9930": 28489568256.0, "9935": 28489568256.0, "9940": 28489568256.0, "9945": 28489568256.0, "9950": 28489568256.0, "9955": 28489568256.0, "9960": 28489568256.0, "9965": 28489568256.0, "9970": 28489568256.0, "9975": 28489568256.0, "9980": 28489568256.0, "9985": 28489568256.0, "9990": 28489568256.0, "9995": 28489568256.0, "10000": 28489568256.0, "10005": 28489568256.0, "10010": 28489568256.0, "10015": 28489568256.0, "10020": 28489568256.0, "10025": 28489568256.0, "10030": 28489568256.0, "10035": 28489568256.0, "10040": 28489568256.0, "10045": 28489568256.0, "10050": 28489568256.0, "10055": 28489568256.0, "10060": 28489568256.0, "10065": 28489568256.0, "10070": 28489568256.0, "10075": 28489568256.0, "10080": 28489568256.0, "10085": 28489568256.0, "10090": 28489568256.0, "10095": 28489568256.0, "10100": 28489568256.0, "10105": 28489568256.0, "10110": 28489568256.0, "10115": 28489568256.0, "10120": 28489568256.0, "10125": 28489568256.0, "10130": 28489568256.0, "10135": 28489568256.0, "10140": 28489568256.0, "10145": 28489568256.0, "10150": 28489568256.0, "10155": 28489568256.0, "10160": 28489568256.0, "10165": 28489568256.0, "10170": 28489568256.0, "10175": 28489568256.0, "10180": 28489568256.0, "10185": 28489568256.0, "10190": 28489568256.0, "10195": 28489568256.0, "10200": 28489568256.0, "10205": 28489568256.0, "10210": 28489568256.0, "10215": 28489568256.0, "10220": 28489568256.0, "10225": 28489568256.0, "10230": 28489568256.0, "10235": 28489568256.0, "10240": 28489568256.0, "10245": 28489568256.0, "10250": 28489568256.0, "10255": 28489568256.0, "10260": 28489568256.0, "10265": 28489568256.0, "10270": 28489568256.0, "10275": 28489568256.0, "10280": 28489568256.0, "10285": 28489568256.0, "10290": 28489568256.0, "10295": 28489568256.0, "10300": 28489568256.0, "10305": 28489568256.0, "10310": 28489568256.0, "10315": 28489568256.0, "10320": 28489568256.0, "10325": 28489568256.0, "10330": 28489568256.0, "10335": 28489568256.0, "10340": 28489568256.0, "10345": 28489568256.0, "10350": 28489568256.0, "10355": 28489568256.0, "10360": 28489568256.0, "10365": 28489568256.0, "10370": 28489568256.0, "10375": 28489568256.0, "10380": 28489568256.0, "10385": 28489568256.0, "10390": 28489568256.0, "10395": 28489568256.0, "10400": 28489568256.0, "10405": 28489568256.0, "10410": 28489568256.0, "10415": 28489568256.0, "10420": 28489568256.0, "10425": 28489568256.0, "10430": 28489568256.0, "10435": 28489568256.0, "10440": 28489568256.0, "10445": 28489568256.0, "10450": 28489568256.0, "10455": 28489568256.0, "10460": 28489568256.0, "10465": 28489568256.0, "10470": 28489568256.0, "10475": 28489568256.0, "10480": 28489568256.0, "10485": 28489568256.0, "10490": 28489568256.0, "10495": 28489568256.0, "10500": 28489568256.0, "10505": 28489568256.0, "10510": 28489568256.0, "10515": 28489568256.0, "10520": 28489568256.0, "10525": 28489568256.0, "10530": 28489568256.0, "10535": 28489568256.0, "10540": 28489568256.0, "10545": 28489568256.0, "10550": 28489568256.0, "10555": 28489568256.0, "10560": 28489568256.0, "10565": 28489568256.0, "10570": 28489568256.0, "10575": 28489568256.0, "10580": 28489568256.0, "10585": 28489568256.0, "10590": 28489568256.0, "10595": 28489568256.0, "10600": 28489568256.0, "10605": 28489568256.0, "10610": 28489568256.0, "10615": 28489568256.0, "10620": 28489568256.0, "10625": 28489568256.0, "10630": 28489568256.0, "10635": 28489568256.0, "10640": 28489568256.0, "10645": 28489568256.0, "10650": 28489568256.0, "10655": 28489568256.0, "10660": 28489568256.0, "10665": 28489568256.0, "10670": 28489568256.0, "10675": 28489568256.0, "10680": 28489568256.0, "10685": 28489568256.0, "10690": 28489568256.0, "10695": 28489568256.0, "10700": 28489568256.0, "10705": 28489568256.0, "10710": 28489568256.0, "10715": 28489568256.0, "10720": 28489568256.0, "10725": 28489568256.0, "10730": 28489568256.0, "10735": 28489568256.0, "10740": 28489568256.0, "10745": 28489568256.0, "10750": 28489568256.0, "10755": 28489568256.0, "10760": 28489568256.0, "10765": 28489568256.0, "10770": 28489568256.0, "10775": 28489568256.0, "10780": 28489568256.0, "10785": 28489568256.0, "10790": 28489568256.0, "10795": 28489568256.0, "10800": 28489568256.0, "10805": 28489568256.0, "10810": 28489568256.0, "10815": 28489568256.0, "10820": 28489568256.0, "10825": 28489568256.0, "10830": 28489568256.0, "10835": 28489568256.0, "10840": 28489568256.0, "10845": 28489568256.0, "10850": 28489568256.0, "10855": 28489568256.0, "10860": 28489568256.0, "10865": 28489568256.0, "10870": 28489568256.0, "10875": 28489568256.0, "10880": 28489568256.0, "10885": 28489568256.0, "10890": 28489568256.0, "10895": 28489568256.0, "10900": 28489568256.0, "10905": 28489568256.0, "10910": 28489568256.0, "10915": 28489568256.0, "10920": 28489568256.0, "10925": 28489568256.0, "10930": 28489568256.0, "10935": 28489568256.0, "10940": 28489568256.0, "10945": 28489568256.0, "10950": 28489568256.0, "10955": 28489568256.0, "10960": 28489568256.0, "10965": 28489568256.0, "10970": 28489568256.0, "10975": 28489568256.0, "10980": 28489568256.0, "10985": 28489568256.0, "10990": 28489568256.0, "10995": 28489568256.0, "11000": 28489568256.0, "11005": 28489568256.0, "11010": 28489568256.0, "11015": 28489568256.0, "11020": 28489568256.0, "11025": 28489568256.0, "11030": 28489568256.0, "11035": 28489568256.0, "11040": 28489568256.0, "11045": 28489568256.0, "11050": 28489568256.0, "11055": 28489568256.0, "11060": 28489568256.0, "11065": 28489568256.0, "11070": 28489568256.0, "11075": 28489568256.0, "11080": 28489568256.0, "11085": 28489568256.0, "11090": 28489568256.0, "11095": 28489568256.0, "11100": 28489568256.0, "11105": 28489568256.0, "11110": 28489568256.0, "11115": 28489568256.0, "11120": 28489568256.0, "11125": 28489568256.0, "11130": 28489568256.0, "11135": 28489568256.0, "11140": 28489568256.0, "11145": 28489568256.0, "11150": 28489568256.0, "11155": 28489568256.0, "11160": 28489568256.0, "11165": 28489568256.0, "11170": 28489568256.0, "11175": 28489568256.0, "11180": 28489568256.0, "11185": 28489568256.0, "11190": 28489568256.0, "11195": 28489568256.0, "11200": 28489568256.0, "11205": 28489568256.0, "11210": 28489568256.0, "11215": 28489568256.0, "11220": 28489568256.0, "11225": 28489568256.0, "11230": 28489568256.0, "11235": 28489568256.0, "11240": 28489568256.0, "11245": 28489568256.0, "11250": 28489568256.0, "11255": 28489568256.0, "11260": 28489568256.0, "11265": 28489568256.0, "11270": 28489568256.0, "11275": 28489568256.0, "11280": 28489568256.0, "11285": 28489568256.0, "11290": 28489568256.0, "11295": 28489568256.0, "11300": 28489568256.0, "11305": 28489568256.0, "11310": 28489568256.0, "11315": 28489568256.0, "11320": 28489568256.0, "11325": 28489568256.0, "11330": 28489568256.0, "11335": 28489568256.0, "11340": 28489568256.0, "11345": 28489568256.0, "11350": 28489568256.0, "11355": 28489568256.0, "11360": 28489568256.0, "11365": 28489568256.0, "11370": 28489568256.0, "11375": 28489568256.0, "11380": 28489568256.0, "11385": 28489568256.0, "11390": 28489568256.0, "11395": 28489568256.0, "11400": 28489568256.0, "11405": 28489568256.0, "11410": 28489568256.0, "11415": 28489568256.0, "11420": 28489568256.0, "11425": 28489568256.0, "11430": 28489568256.0, "11435": 28489568256.0, "11440": 28489568256.0, "11445": 28489568256.0, "11450": 28489568256.0, "11455": 28489568256.0, "11460": 28489568256.0, "11465": 28489568256.0, "11470": 28489568256.0, "11475": 28489568256.0, "11480": 28489568256.0, "11485": 28489568256.0, "11490": 28489568256.0, "11495": 28489568256.0, "11500": 28489568256.0, "11505": 28489568256.0, "11510": 28489568256.0, "11515": 28489568256.0, "11520": 28489568256.0, "11525": 28489568256.0, "11530": 28489568256.0, "11535": 28489568256.0, "11540": 28489568256.0, "11545": 28489568256.0, "11550": 28489568256.0, "11555": 28489568256.0, "11560": 28489568256.0, "11565": 28489568256.0, "11570": 28489568256.0, "11575": 28489568256.0, "11580": 28489568256.0, "11585": 28489568256.0, "11590": 28489568256.0, "11595": 28489568256.0, "11600": 28489568256.0, "11605": 28489568256.0, "11610": 28489568256.0, "11615": 28489568256.0, "11620": 28489568256.0, "11625": 28489568256.0, "11630": 28489568256.0, "11635": 28489568256.0, "11640": 28489568256.0, "11645": 28489568256.0, "11650": 28489568256.0, "11655": 28489568256.0, "11660": 28489568256.0, "11665": 28489568256.0, "11670": 28489568256.0, "11675": 28489568256.0, "11680": 28489568256.0, "11685": 28489568256.0, "11690": 28489568256.0, "11695": 28489568256.0, "11700": 28489568256.0, "11705": 28489568256.0, "11710": 28489568256.0, "11715": 28489568256.0, "11720": 28489568256.0, "11725": 28489568256.0, "11730": 28489568256.0, "11735": 28489568256.0, "11740": 28489568256.0, "11745": 28489568256.0, "11750": 28489568256.0, "11755": 28489568256.0, "11760": 28489568256.0, "11765": 28489568256.0, "11770": 28489568256.0, "11775": 28489568256.0, "11780": 28489568256.0, "11785": 28489568256.0, "11790": 28489568256.0, "11795": 28489568256.0, "11800": 28489568256.0, "11805": 28489568256.0, "11810": 28489568256.0, "11815": 28489568256.0, "11820": 28489568256.0, "11825": 28489568256.0, "11830": 28489568256.0, "11835": 28489568256.0, "11840": 28489568256.0, "11845": 28489568256.0, "11850": 28489568256.0, "11855": 28489568256.0, "11860": 28489568256.0, "11865": 28489568256.0, "11870": 28489568256.0, "11875": 28489568256.0, "11880": 28489568256.0, "11885": 28489568256.0, "11890": 28489568256.0, "11895": 28489568256.0, "11900": 28489568256.0, "11905": 28489568256.0, "11910": 28489568256.0, "11915": 28489568256.0, "11920": 28489568256.0, "11925": 28489568256.0, "11930": 28489568256.0, "11935": 28489568256.0, "11940": 28489568256.0, "11945": 28489568256.0, "11950": 28489568256.0, "11955": 28489568256.0, "11960": 28489568256.0, "11965": 28489568256.0, "11970": 28489568256.0, "11975": 28489568256.0, "11980": 28489568256.0, "11985": 28489568256.0, "11990": 28489568256.0, "11995": 28489568256.0, "12000": 28489568256.0, "12005": 28489568256.0, "12010": 28489568256.0, "12015": 28489568256.0, "12020": 28489568256.0, "12025": 28489568256.0, "12030": 28489568256.0, "12035": 28489568256.0, "12040": 28489568256.0, "12045": 28489568256.0, "12050": 28489568256.0, "12055": 28489568256.0, "12060": 28489568256.0, "12065": 28489568256.0, "12070": 28489568256.0, "12075": 28489568256.0, "12080": 28489568256.0, "12085": 28489568256.0, "12090": 28489568256.0, "12095": 28489568256.0, "12100": 28489568256.0, "12105": 28489568256.0, "12110": 28489568256.0, "12115": 28489568256.0, "12120": 28489568256.0, "12125": 28489568256.0, "12130": 28489568256.0, "12135": 28489568256.0, "12140": 28489568256.0, "12145": 28489568256.0, "12150": 28489568256.0, "12155": 28489568256.0, "12160": 28489568256.0, "12165": 28489568256.0, "12170": 28489568256.0, "12175": 28489568256.0, "12180": 28489568256.0, "12185": 28489568256.0, "12190": 28489568256.0, "12195": 28489568256.0, "12200": 28489568256.0, "12205": 28489568256.0, "12210": 28489568256.0, "12215": 28489568256.0, "12220": 28489568256.0, "12225": 28489568256.0, "12230": 28489568256.0, "12235": 28489568256.0, "12240": 28489568256.0, "12245": 28489568256.0, "12250": 28489568256.0, "12255": 28489568256.0, "12260": 28489568256.0, "12265": 28489568256.0, "12270": 28489568256.0, "12275": 28489568256.0, "12280": 28489568256.0, "12285": 28489568256.0, "12290": 28489568256.0, "12295": 28489568256.0, "12300": 28489568256.0, "12305": 28489568256.0, "12310": 28489568256.0, "12315": 28489568256.0, "12320": 28489568256.0, "12325": 28489568256.0, "12330": 28489568256.0, "12335": 28489568256.0, "12340": 28489568256.0, "12345": 28489568256.0, "12350": 28489568256.0, "12355": 28489568256.0, "12360": 28489568256.0, "12365": 28489568256.0, "12370": 28489568256.0, "12375": 28489568256.0, "12380": 28489568256.0, "12385": 28489568256.0, "12390": 28489568256.0, "12395": 28489568256.0, "12400": 28489568256.0, "12405": 28489568256.0, "12410": 28489568256.0, "12415": 28489568256.0, "12420": 28489568256.0, "12425": 28489568256.0, "12430": 28489568256.0, "12435": 28489568256.0, "12440": 28489568256.0, "12445": 28489568256.0, "12450": 28489568256.0, "12455": 28489568256.0, "12460": 28489568256.0, "12465": 28489568256.0, "12470": 28489568256.0, "12475": 28489568256.0, "12480": 28489568256.0, "12485": 28489568256.0, "12490": 28489568256.0, "12495": 28489568256.0, "12500": 28489568256.0, "12505": 28489568256.0, "12510": 28489568256.0, "12515": 28489568256.0, "12520": 28489568256.0, "12525": 28489568256.0, "12530": 28489568256.0, "12535": 28489568256.0, "12540": 28489568256.0, "12545": 28489568256.0, "12550": 28489568256.0, "12555": 28489568256.0, "12560": 28489568256.0, "12565": 28489568256.0, "12570": 28489568256.0, "12575": 28489568256.0, "12580": 28489568256.0, "12585": 28489568256.0, "12590": 28489568256.0, "12595": 28489568256.0, "12600": 28489568256.0, "12605": 28489568256.0, "12610": 28489568256.0, "12615": 28489568256.0, "12620": 28489568256.0, "12625": 28489568256.0, "12630": 28489568256.0, "12635": 28489568256.0, "12640": 28489568256.0, "12645": 28489568256.0, "12650": 28489568256.0, "12655": 28489568256.0, "12660": 28489568256.0, "12665": 28489568256.0, "12670": 28489568256.0, "12675": 28489568256.0, "12680": 28489568256.0, "12685": 28489568256.0, "12690": 28489568256.0, "12695": 28489568256.0, "12700": 28489568256.0, "12705": 28489568256.0, "12710": 28489568256.0, "12715": 28489568256.0, "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}, "iteration-time": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": "nan", "5": "nan", "10": "nan", "15": "nan", "20": "nan", "25": "nan", "30": "nan", "35": "nan", "40": "nan", "45": "nan", "50": "nan", "55": "nan", "60": "nan", "65": "nan", "70": "nan", "75": "nan", "80": "nan", "85": "nan", "90": "nan", "95": "nan", "100": 3.5554, "105": "nan", "110": "nan", "115": "nan", "120": "nan", "125": "nan", "130": "nan", "135": "nan", "140": "nan", "145": "nan", "150": "nan", "155": "nan", "160": "nan", "165": "nan", "170": "nan", "175": "nan", "180": "nan", "185": "nan", "190": "nan", "195": "nan", "200": 3.47185, "205": "nan", "210": "nan", "215": "nan", "220": "nan", "225": "nan", "230": "nan", "235": "nan", "240": "nan", "245": "nan", "250": "nan", "255": "nan", "260": "nan", "265": "nan", "270": "nan", "275": "nan", "280": "nan", "285": "nan", "290": "nan", "295": "nan", "300": 3.45692, "305": "nan", "310": "nan", "315": "nan", "320": "nan", "325": "nan", "330": "nan", "335": "nan", "340": "nan", "345": "nan", "350": "nan", "355": "nan", "360": "nan", "365": "nan", "370": "nan", "375": "nan", "380": "nan", "385": "nan", "390": "nan", "395": "nan", "400": 3.45471, "405": "nan", "410": "nan", "415": "nan", "420": "nan", "425": "nan", "430": "nan", "435": "nan", "440": "nan", "445": "nan", "450": "nan", "455": "nan", "460": "nan", "465": "nan", "470": "nan", "475": "nan", "480": "nan", "485": "nan", "490": "nan", "495": "nan", "500": 3.45467, "505": "nan", "510": "nan", "515": "nan", "520": "nan", "525": "nan", "530": "nan", "535": "nan", "540": "nan", "545": "nan", "550": "nan", "555": "nan", "560": "nan", "565": "nan", "570": "nan", "575": "nan", "580": "nan", "585": "nan", "590": "nan", "595": "nan", "600": 3.4543, "605": "nan", "610": "nan", "615": "nan", "620": "nan", "625": "nan", "630": "nan", "635": "nan", "640": "nan", "645": "nan", "650": "nan", "655": "nan", "660": "nan", "665": "nan", "670": "nan", "675": "nan", "680": "nan", "685": "nan", "690": "nan", "695": "nan", "700": 3.45264, "705": "nan", "710": "nan", "715": "nan", "720": "nan", "725": "nan", "730": "nan", "735": "nan", "740": "nan", "745": "nan", "750": "nan", "755": "nan", "760": "nan", "765": "nan", "770": "nan", "775": "nan", "780": "nan", "785": "nan", "790": "nan", "795": "nan", "800": 3.45125, "805": "nan", "810": "nan", "815": "nan", "820": "nan", "825": "nan", "830": "nan", "835": "nan", "840": "nan", "845": "nan", "850": "nan", "855": "nan", "860": "nan", "865": "nan", "870": "nan", "875": "nan", "880": "nan", "885": "nan", "890": "nan", "895": "nan", "900": 3.44668, "905": "nan", "910": "nan", "915": "nan", "920": "nan", "925": "nan", "930": "nan", "935": "nan", "940": "nan", "945": "nan", "950": "nan", "955": "nan", "960": "nan", "965": "nan", "970": "nan", "975": "nan", "980": "nan", "985": "nan", "990": "nan", "995": "nan", "1000": 3.44035, "1005": "nan", "1010": "nan", "1015": "nan", "1020": "nan", "1025": "nan", "1030": "nan", "1035": "nan", "1040": "nan", "1045": "nan", "1050": "nan", "1055": "nan", "1060": "nan", "1065": "nan", "1070": "nan", "1075": "nan", "1080": "nan", "1085": "nan", "1090": "nan", "1095": "nan", "1100": 3.43442, "1105": "nan", "1110": "nan", "1115": "nan", "1120": "nan", "1125": "nan", "1130": "nan", "1135": "nan", "1140": "nan", "1145": "nan", "1150": "nan", "1155": "nan", "1160": "nan", "1165": "nan", "1170": "nan", "1175": "nan", "1180": "nan", "1185": "nan", "1190": "nan", "1195": "nan", "1200": 3.4306, "1205": "nan", "1210": "nan", "1215": "nan", "1220": "nan", "1225": "nan", "1230": "nan", "1235": "nan", "1240": "nan", "1245": "nan", "1250": "nan", "1255": "nan", "1260": "nan", "1265": "nan", "1270": "nan", "1275": "nan", "1280": "nan", "1285": "nan", "1290": "nan", "1295": "nan", "1300": 3.42464, "1305": "nan", "1310": "nan", "1315": "nan", "1320": "nan", "1325": "nan", "1330": "nan", "1335": "nan", "1340": "nan", "1345": "nan", "1350": "nan", "1355": "nan", "1360": "nan", "1365": "nan", "1370": "nan", "1375": "nan", "1380": "nan", "1385": "nan", "1390": "nan", "1395": "nan", "1400": 3.42155, "1405": "nan", "1410": "nan", "1415": "nan", "1420": "nan", "1425": "nan", "1430": "nan", "1435": "nan", "1440": "nan", "1445": "nan", "1450": "nan", "1455": "nan", "1460": "nan", "1465": "nan", "1470": "nan", "1475": "nan", "1480": "nan", "1485": "nan", "1490": "nan", "1495": "nan", "1500": 3.4201, "1505": "nan", "1510": "nan", "1515": "nan", "1520": "nan", "1525": "nan", "1530": "nan", "1535": "nan", "1540": "nan", "1545": "nan", "1550": "nan", "1555": "nan", "1560": "nan", "1565": "nan", "1570": "nan", "1575": "nan", "1580": "nan", "1585": "nan", "1590": "nan", "1595": "nan", "1600": 3.41703, "1605": "nan", "1610": "nan", "1615": "nan", "1620": "nan", "1625": "nan", "1630": "nan", "1635": "nan", "1640": "nan", "1645": "nan", "1650": "nan", "1655": "nan", "1660": "nan", "1665": "nan", "1670": "nan", "1675": "nan", "1680": "nan", "1685": "nan", "1690": "nan", "1695": "nan", "1700": 3.41482, "1705": "nan", "1710": "nan", "1715": "nan", "1720": "nan", "1725": "nan", "1730": "nan", "1735": "nan", "1740": "nan", "1745": "nan", "1750": "nan", "1755": "nan", "1760": "nan", "1765": "nan", "1770": "nan", "1775": "nan", "1780": "nan", "1785": "nan", "1790": "nan", "1795": "nan", "1800": 3.41352, "1805": "nan", "1810": "nan", "1815": "nan", "1820": "nan", "1825": "nan", "1830": "nan", "1835": "nan", "1840": "nan", "1845": "nan", "1850": "nan", "1855": "nan", "1860": "nan", "1865": "nan", "1870": "nan", "1875": "nan", "1880": "nan", "1885": "nan", "1890": "nan", "1895": "nan", "1900": 3.4128, "1905": "nan", "1910": "nan", "1915": "nan", "1920": "nan", "1925": "nan", "1930": "nan", "1935": "nan", "1940": "nan", "1945": "nan", "1950": "nan", "1955": "nan", "1960": "nan", "1965": "nan", "1970": "nan", "1975": "nan", "1980": "nan", "1985": "nan", "1990": "nan", "1995": "nan", "2000": 3.40994, "2005": "nan", "2010": "nan", "2015": "nan", "2020": "nan", "2025": "nan", "2030": "nan", "2035": "nan", "2040": "nan", "2045": "nan", "2050": "nan", "2055": "nan", "2060": "nan", "2065": "nan", "2070": "nan", "2075": "nan", "2080": "nan", "2085": "nan", "2090": "nan", "2095": "nan", "2100": 3.40878, "2105": "nan", "2110": "nan", "2115": "nan", "2120": "nan", "2125": "nan", "2130": "nan", "2135": "nan", "2140": "nan", "2145": "nan", "2150": "nan", "2155": "nan", "2160": "nan", "2165": "nan", "2170": "nan", "2175": "nan", "2180": "nan", "2185": "nan", "2190": "nan", "2195": "nan", "2200": 3.40862, "2205": "nan", "2210": "nan", "2215": "nan", "2220": "nan", "2225": "nan", "2230": "nan", "2235": "nan", "2240": "nan", "2245": "nan", "2250": "nan", "2255": "nan", "2260": "nan", "2265": "nan", "2270": "nan", "2275": "nan", "2280": "nan", "2285": "nan", "2290": "nan", "2295": "nan", "2300": 3.41459, "2305": "nan", "2310": "nan", "2315": "nan", "2320": "nan", "2325": "nan", "2330": "nan", "2335": "nan", "2340": "nan", "2345": "nan", "2350": "nan", "2355": "nan", "2360": "nan", "2365": "nan", "2370": "nan", "2375": "nan", "2380": "nan", "2385": "nan", "2390": "nan", "2395": "nan", "2400": 3.40662, "2405": "nan", "2410": "nan", "2415": "nan", "2420": "nan", "2425": "nan", "2430": "nan", "2435": "nan", "2440": "nan", "2445": "nan", "2450": "nan", "2455": "nan", "2460": "nan", "2465": "nan", "2470": "nan", "2475": "nan", "2480": "nan", "2485": "nan", "2490": "nan", "2495": "nan", "2500": 3.40543, "2505": "nan", "2510": "nan", "2515": "nan", "2520": "nan", "2525": "nan", "2530": "nan", "2535": "nan", "2540": "nan", "2545": "nan", "2550": "nan", "2555": "nan", "2560": "nan", "2565": "nan", "2570": "nan", "2575": "nan", "2580": "nan", "2585": "nan", "2590": "nan", "2595": "nan", "2600": 3.40484, "2605": "nan", "2610": "nan", "2615": "nan", "2620": "nan", "2625": "nan", "2630": "nan", "2635": "nan", "2640": "nan", "2645": "nan", "2650": "nan", "2655": "nan", "2660": "nan", "2665": "nan", "2670": "nan", "2675": "nan", "2680": "nan", "2685": "nan", "2690": "nan", "2695": "nan", "2700": 3.40448, "2705": "nan", "2710": "nan", "2715": "nan", "2720": "nan", "2725": "nan", "2730": "nan", "2735": "nan", "2740": "nan", "2745": "nan", "2750": "nan", "2755": "nan", "2760": "nan", "2765": "nan", "2770": "nan", "2775": "nan", "2780": "nan", "2785": "nan", "2790": "nan", "2795": "nan", "2800": 3.403, "2805": "nan", "2810": "nan", "2815": "nan", "2820": "nan", "2825": "nan", "2830": "nan", "2835": "nan", "2840": "nan", "2845": "nan", "2850": "nan", "2855": "nan", "2860": "nan", "2865": "nan", "2870": "nan", "2875": "nan", "2880": "nan", "2885": "nan", "2890": "nan", "2895": "nan", "2900": 3.40346, "2905": "nan", "2910": "nan", "2915": "nan", "2920": "nan", "2925": "nan", "2930": "nan", "2935": "nan", "2940": "nan", "2945": "nan", "2950": "nan", "2955": "nan", "2960": "nan", "2965": "nan", "2970": "nan", "2975": "nan", "2980": "nan", "2985": "nan", "2990": "nan", "2995": "nan", "3000": 3.4023, "3005": "nan", "3010": "nan", "3015": "nan", "3020": "nan", "3025": "nan", "3030": "nan", "3035": "nan", "3040": "nan", "3045": "nan", "3050": "nan", "3055": "nan", "3060": "nan", "3065": "nan", "3070": "nan", "3075": "nan", "3080": "nan", "3085": "nan", "3090": "nan", "3095": "nan", "3100": 3.40069, "3105": "nan", "3110": "nan", "3115": "nan", "3120": "nan", "3125": "nan", "3130": "nan", "3135": "nan", "3140": "nan", "3145": "nan", "3150": "nan", "3155": "nan", "3160": "nan", "3165": "nan", "3170": "nan", "3175": "nan", "3180": "nan", "3185": "nan", "3190": "nan", "3195": "nan", "3200": 3.40162, "3205": "nan", "3210": "nan", "3215": "nan", "3220": "nan", "3225": "nan", "3230": "nan", "3235": "nan", "3240": "nan", "3245": "nan", "3250": "nan", "3255": "nan", "3260": "nan", "3265": "nan", "3270": "nan", "3275": "nan", "3280": "nan", "3285": "nan", "3290": "nan", "3295": "nan", "3300": 3.40071, "3305": "nan", "3310": "nan", "3315": "nan", "3320": "nan", "3325": "nan", "3330": "nan", "3335": "nan", "3340": "nan", "3345": "nan", "3350": "nan", "3355": "nan", "3360": "nan", "3365": "nan", "3370": "nan", "3375": "nan", "3380": "nan", "3385": "nan", "3390": "nan", "3395": "nan", "3400": 3.40058, "3405": "nan", "3410": "nan", "3415": "nan", "3420": "nan", "3425": "nan", "3430": "nan", "3435": "nan", "3440": "nan", "3445": "nan", "3450": "nan", "3455": "nan", "3460": "nan", "3465": "nan", "3470": "nan", "3475": "nan", "3480": "nan", "3485": "nan", "3490": "nan", "3495": "nan", "3500": 3.39993, "3505": "nan", "3510": "nan", "3515": "nan", "3520": "nan", "3525": "nan", "3530": "nan", "3535": "nan", "3540": "nan", "3545": "nan", "3550": "nan", "3555": "nan", "3560": "nan", "3565": "nan", "3570": "nan", "3575": "nan", "3580": "nan", "3585": "nan", "3590": "nan", "3595": "nan", "3600": 3.62689, "3605": "nan", "3610": "nan", "3615": "nan", "3620": "nan", "3625": "nan", "3630": "nan", "3635": "nan", "3640": "nan", "3645": "nan", "3650": "nan", "3655": "nan", "3660": "nan", "3665": "nan", "3670": "nan", "3675": "nan", "3680": "nan", "3685": "nan", "3690": "nan", "3695": "nan", "3700": 3.40474, "3705": "nan", "3710": "nan", "3715": "nan", "3720": "nan", "3725": "nan", "3730": "nan", "3735": "nan", "3740": "nan", "3745": "nan", "3750": "nan", "3755": "nan", "3760": "nan", "3765": "nan", "3770": "nan", "3775": "nan", "3780": "nan", "3785": "nan", "3790": "nan", "3795": "nan", "3800": 3.40515, "3805": "nan", "3810": "nan", "3815": "nan", "3820": "nan", "3825": "nan", "3830": "nan", "3835": "nan", "3840": "nan", "3845": "nan", "3850": "nan", "3855": "nan", "3860": "nan", "3865": "nan", "3870": "nan", "3875": "nan", "3880": "nan", "3885": "nan", "3890": "nan", "3895": "nan", "3900": 3.40436, "3905": "nan", "3910": "nan", "3915": "nan", "3920": "nan", "3925": "nan", "3930": "nan", "3935": "nan", "3940": "nan", "3945": "nan", "3950": "nan", "3955": "nan", "3960": "nan", "3965": "nan", "3970": "nan", "3975": "nan", "3980": "nan", "3985": "nan", "3990": "nan", "3995": "nan", "4000": 3.40304, "4005": "nan", "4010": "nan", "4015": "nan", "4020": "nan", "4025": "nan", "4030": "nan", "4035": "nan", "4040": "nan", "4045": "nan", "4050": "nan", "4055": "nan", "4060": "nan", "4065": "nan", "4070": "nan", "4075": "nan", "4080": "nan", "4085": "nan", "4090": "nan", "4095": "nan", "4100": 3.40076, "4105": "nan", "4110": "nan", "4115": "nan", "4120": "nan", "4125": "nan", "4130": "nan", "4135": "nan", "4140": "nan", "4145": "nan", "4150": "nan", "4155": "nan", "4160": "nan", "4165": "nan", "4170": "nan", "4175": "nan", "4180": "nan", "4185": "nan", "4190": "nan", "4195": "nan", "4200": 3.40222, "4205": "nan", "4210": "nan", "4215": "nan", "4220": "nan", "4225": "nan", "4230": "nan", "4235": "nan", "4240": "nan", "4245": "nan", "4250": "nan", "4255": "nan", "4260": "nan", "4265": "nan", "4270": "nan", "4275": "nan", "4280": "nan", "4285": "nan", "4290": "nan", "4295": "nan", "4300": 3.40241, "4305": "nan", "4310": "nan", "4315": "nan", "4320": "nan", "4325": "nan", "4330": "nan", "4335": "nan", "4340": "nan", "4345": "nan", "4350": "nan", "4355": "nan", "4360": "nan", "4365": "nan", "4370": "nan", "4375": "nan", "4380": "nan", "4385": "nan", "4390": "nan", "4395": "nan", "4400": 3.40228, "4405": "nan", "4410": "nan", "4415": "nan", "4420": "nan", "4425": "nan", "4430": "nan", "4435": "nan", "4440": "nan", "4445": "nan", "4450": "nan", "4455": "nan", "4460": "nan", "4465": "nan", "4470": "nan", "4475": "nan", "4480": "nan", "4485": "nan", "4490": "nan", "4495": "nan", "4500": 3.40282, "4505": "nan", "4510": "nan", "4515": "nan", "4520": "nan", "4525": "nan", "4530": "nan", "4535": "nan", "4540": "nan", "4545": "nan", "4550": "nan", "4555": "nan", "4560": "nan", "4565": "nan", "4570": "nan", "4575": "nan", "4580": "nan", "4585": "nan", "4590": "nan", "4595": "nan", "4600": 3.40214, "4605": "nan", "4610": "nan", "4615": "nan", "4620": "nan", "4625": "nan", "4630": "nan", "4635": "nan", "4640": "nan", "4645": "nan", "4650": "nan", "4655": "nan", "4660": "nan", "4665": "nan", "4670": "nan", "4675": "nan", "4680": "nan", "4685": "nan", "4690": "nan", "4695": "nan", "4700": 3.40155, "4705": "nan", "4710": "nan", "4715": "nan", "4720": "nan", "4725": "nan", "4730": "nan", "4735": "nan", "4740": "nan", "4745": "nan", "4750": "nan", "4755": "nan", "4760": "nan", "4765": "nan", "4770": "nan", "4775": "nan", "4780": "nan", "4785": "nan", "4790": "nan", "4795": "nan", "4800": 3.4016, "4805": "nan", "4810": "nan", "4815": "nan", "4820": "nan", "4825": "nan", "4830": "nan", "4835": "nan", "4840": "nan", "4845": "nan", "4850": "nan", "4855": "nan", "4860": "nan", "4865": "nan", "4870": "nan", "4875": "nan", "4880": "nan", "4885": "nan", "4890": "nan", "4895": "nan", "4900": 3.40208, "4905": "nan", "4910": "nan", "4915": "nan", "4920": "nan", "4925": "nan", "4930": "nan", "4935": "nan", "4940": "nan", "4945": "nan", "4950": "nan", "4955": "nan", "4960": "nan", "4965": "nan", "4970": "nan", "4975": "nan", "4980": "nan", "4985": "nan", "4990": "nan", "4995": "nan", "5000": 3.40265, "5005": "nan", "5010": "nan", "5015": "nan", "5020": "nan", "5025": "nan", "5030": "nan", "5035": "nan", "5040": "nan", "5045": "nan", "5050": "nan", "5055": "nan", "5060": "nan", "5065": "nan", "5070": "nan", "5075": "nan", "5080": "nan", "5085": "nan", "5090": "nan", "5095": "nan", "5100": 3.3986, "5105": "nan", "5110": "nan", "5115": "nan", "5120": "nan", "5125": "nan", "5130": "nan", "5135": "nan", "5140": "nan", "5145": "nan", "5150": "nan", "5155": "nan", "5160": "nan", "5165": "nan", "5170": "nan", "5175": "nan", "5180": "nan", "5185": "nan", "5190": "nan", "5195": "nan", "5200": 3.39887, "5205": "nan", "5210": "nan", "5215": "nan", "5220": "nan", "5225": "nan", "5230": "nan", "5235": "nan", "5240": "nan", "5245": "nan", "5250": "nan", "5255": "nan", "5260": "nan", "5265": "nan", "5270": "nan", "5275": "nan", "5280": "nan", "5285": "nan", "5290": "nan", "5295": "nan", "5300": 3.3991, "5305": "nan", "5310": "nan", "5315": "nan", "5320": "nan", "5325": "nan", "5330": "nan", "5335": "nan", "5340": "nan", "5345": "nan", "5350": "nan", "5355": "nan", "5360": "nan", "5365": "nan", "5370": "nan", "5375": "nan", "5380": "nan", "5385": "nan", "5390": "nan", "5395": "nan", "5400": 3.40087, "5405": "nan", "5410": "nan", "5415": "nan", "5420": "nan", "5425": "nan", "5430": "nan", "5435": "nan", "5440": "nan", "5445": "nan", "5450": "nan", "5455": "nan", "5460": "nan", "5465": "nan", "5470": "nan", "5475": "nan", "5480": "nan", "5485": "nan", "5490": "nan", "5495": "nan", "5500": 3.40055, "5505": "nan", "5510": "nan", "5515": "nan", "5520": "nan", "5525": "nan", "5530": "nan", "5535": "nan", "5540": "nan", "5545": "nan", "5550": "nan", "5555": "nan", "5560": "nan", "5565": "nan", "5570": "nan", "5575": "nan", "5580": "nan", "5585": "nan", "5590": "nan", "5595": "nan", "5600": 3.40101, "5605": "nan", "5610": "nan", "5615": "nan", "5620": "nan", "5625": "nan", "5630": "nan", "5635": "nan", "5640": "nan", "5645": "nan", "5650": "nan", "5655": "nan", "5660": "nan", "5665": "nan", "5670": "nan", "5675": "nan", "5680": "nan", "5685": "nan", "5690": "nan", "5695": "nan", "5700": 3.4007, "5705": "nan", "5710": "nan", "5715": "nan", "5720": "nan", "5725": "nan", "5730": "nan", "5735": "nan", "5740": "nan", "5745": "nan", "5750": "nan", "5755": "nan", "5760": "nan", "5765": "nan", "5770": "nan", "5775": "nan", "5780": "nan", "5785": "nan", "5790": "nan", "5795": "nan", "5800": 3.40177, "5805": "nan", "5810": "nan", "5815": "nan", "5820": "nan", "5825": "nan", "5830": "nan", "5835": "nan", "5840": "nan", "5845": "nan", "5850": "nan", "5855": "nan", "5860": "nan", "5865": "nan", "5870": "nan", "5875": "nan", "5880": "nan", "5885": "nan", "5890": "nan", "5895": "nan", "5900": 3.40093, "5905": "nan", "5910": "nan", "5915": "nan", "5920": "nan", "5925": "nan", "5930": "nan", "5935": "nan", "5940": "nan", "5945": "nan", "5950": "nan", "5955": "nan", "5960": "nan", "5965": "nan", "5970": "nan", "5975": "nan", "5980": "nan", "5985": "nan", "5990": "nan", "5995": "nan", "6000": 3.40207, "6005": "nan", "6010": "nan", "6015": "nan", "6020": "nan", "6025": "nan", "6030": "nan", "6035": "nan", "6040": "nan", "6045": "nan", "6050": "nan", "6055": "nan", "6060": "nan", "6065": "nan", "6070": "nan", "6075": "nan", "6080": "nan", "6085": "nan", "6090": "nan", "6095": "nan", "6100": 3.40047, "6105": "nan", "6110": "nan", "6115": "nan", "6120": "nan", "6125": "nan", "6130": "nan", "6135": "nan", "6140": "nan", "6145": "nan", "6150": "nan", "6155": "nan", "6160": "nan", "6165": "nan", "6170": "nan", "6175": "nan", "6180": "nan", "6185": "nan", "6190": "nan", "6195": "nan", "6200": 3.40254, "6205": "nan", "6210": "nan", "6215": "nan", "6220": "nan", "6225": "nan", "6230": "nan", "6235": "nan", "6240": "nan", "6245": "nan", "6250": "nan", "6255": "nan", "6260": "nan", "6265": "nan", "6270": "nan", "6275": "nan", "6280": "nan", "6285": "nan", "6290": "nan", "6295": "nan", "6300": 3.4024, "6305": "nan", "6310": "nan", "6315": "nan", "6320": "nan", "6325": "nan", "6330": "nan", "6335": "nan", "6340": "nan", "6345": "nan", "6350": "nan", "6355": "nan", "6360": "nan", "6365": "nan", "6370": "nan", "6375": "nan", "6380": "nan", "6385": "nan", "6390": "nan", "6395": "nan", "6400": 3.40281, "6405": "nan", "6410": "nan", "6415": "nan", "6420": "nan", "6425": "nan", "6430": "nan", "6435": "nan", "6440": "nan", "6445": "nan", "6450": "nan", "6455": "nan", "6460": "nan", "6465": "nan", "6470": "nan", "6475": "nan", "6480": "nan", "6485": "nan", "6490": "nan", "6495": "nan", "6500": 3.40268, "6505": "nan", "6510": "nan", "6515": "nan", "6520": "nan", "6525": "nan", "6530": "nan", "6535": "nan", "6540": "nan", "6545": "nan", "6550": "nan", "6555": "nan", "6560": "nan", "6565": "nan", "6570": "nan", "6575": "nan", "6580": "nan", "6585": "nan", "6590": "nan", "6595": "nan", "6600": 3.40166, "6605": "nan", "6610": "nan", "6615": "nan", "6620": "nan", "6625": "nan", "6630": "nan", "6635": "nan", "6640": "nan", "6645": "nan", "6650": "nan", "6655": "nan", "6660": "nan", "6665": "nan", "6670": "nan", "6675": "nan", "6680": "nan", "6685": "nan", "6690": "nan", "6695": "nan", "6700": 3.40129, "6705": "nan", "6710": "nan", "6715": "nan", "6720": "nan", "6725": "nan", "6730": "nan", "6735": "nan", "6740": "nan", "6745": "nan", "6750": "nan", "6755": "nan", "6760": "nan", "6765": "nan", "6770": "nan", "6775": "nan", "6780": "nan", "6785": "nan", "6790": "nan", "6795": "nan", "6800": 3.40047, "6805": "nan", "6810": "nan", "6815": "nan", "6820": "nan", "6825": "nan", "6830": "nan", "6835": "nan", "6840": "nan", "6845": "nan", "6850": "nan", "6855": "nan", "6860": "nan", "6865": "nan", "6870": "nan", "6875": "nan", "6880": "nan", "6885": "nan", "6890": "nan", "6895": "nan", "6900": 3.40079, "6905": "nan", "6910": "nan", "6915": "nan", "6920": "nan", "6925": "nan", "6930": "nan", "6935": "nan", "6940": "nan", "6945": "nan", "6950": "nan", "6955": "nan", "6960": "nan", "6965": "nan", "6970": "nan", "6975": "nan", "6980": "nan", "6985": "nan", "6990": "nan", "6995": "nan", "7000": 3.40034, "7005": "nan", "7010": "nan", "7015": "nan", "7020": "nan", "7025": "nan", "7030": "nan", "7035": "nan", "7040": "nan", "7045": "nan", "7050": "nan", "7055": "nan", "7060": "nan", "7065": "nan", "7070": "nan", "7075": "nan", "7080": "nan", "7085": "nan", "7090": "nan", "7095": "nan", "7100": 3.39893, "7105": "nan", "7110": "nan", "7115": "nan", "7120": "nan", "7125": "nan", "7130": "nan", "7135": "nan", "7140": "nan", "7145": "nan", "7150": "nan", "7155": "nan", "7160": "nan", "7165": "nan", "7170": "nan", "7175": "nan", "7180": "nan", "7185": "nan", "7190": "nan", "7195": "nan", "7200": 3.40035, "7205": "nan", "7210": "nan", "7215": "nan", "7220": "nan", "7225": "nan", "7230": "nan", "7235": "nan", "7240": "nan", "7245": "nan", "7250": "nan", "7255": "nan", "7260": "nan", "7265": "nan", "7270": "nan", "7275": "nan", "7280": "nan", "7285": "nan", "7290": "nan", "7295": "nan", "7300": 3.39965, "7305": "nan", "7310": "nan", "7315": "nan", "7320": "nan", "7325": "nan", "7330": "nan", "7335": "nan", "7340": "nan", "7345": "nan", "7350": "nan", "7355": "nan", "7360": "nan", "7365": "nan", "7370": "nan", "7375": "nan", "7380": "nan", "7385": "nan", "7390": "nan", "7395": "nan", "7400": 3.40073, "7405": "nan", "7410": "nan", "7415": "nan", "7420": "nan", "7425": "nan", "7430": "nan", "7435": "nan", "7440": "nan", "7445": "nan", "7450": "nan", "7455": "nan", "7460": "nan", "7465": "nan", "7470": "nan", "7475": "nan", "7480": "nan", "7485": "nan", "7490": "nan", "7495": "nan", "7500": 3.40077, "7505": "nan", "7510": "nan", "7515": "nan", "7520": "nan", "7525": "nan", "7530": "nan", "7535": "nan", "7540": "nan", "7545": "nan", "7550": "nan", "7555": "nan", "7560": "nan", "7565": "nan", "7570": "nan", "7575": "nan", "7580": "nan", "7585": "nan", "7590": "nan", "7595": "nan", "7600": 3.39989, "7605": "nan", "7610": "nan", "7615": "nan", "7620": "nan", "7625": "nan", "7630": "nan", "7635": "nan", "7640": "nan", "7645": "nan", "7650": "nan", "7655": "nan", "7660": "nan", "7665": "nan", "7670": "nan", "7675": "nan", "7680": "nan", "7685": "nan", "7690": "nan", "7695": "nan", "7700": 3.3988, "7705": "nan", "7710": "nan", "7715": "nan", "7720": "nan", "7725": "nan", "7730": "nan", "7735": "nan", "7740": "nan", "7745": "nan", "7750": "nan", "7755": "nan", "7760": "nan", "7765": "nan", "7770": "nan", "7775": "nan", "7780": "nan", "7785": "nan", "7790": "nan", "7795": "nan", "7800": 3.3995, "7805": "nan", "7810": "nan", "7815": "nan", "7820": "nan", "7825": "nan", "7830": "nan", "7835": "nan", "7840": "nan", "7845": "nan", "7850": "nan", "7855": "nan", "7860": "nan", "7865": "nan", "7870": "nan", "7875": "nan", "7880": "nan", "7885": "nan", "7890": "nan", "7895": "nan", "7900": 3.39928, "7905": "nan", "7910": "nan", "7915": "nan", "7920": "nan", "7925": "nan", "7930": "nan", "7935": "nan", "7940": "nan", "7945": "nan", "7950": "nan", "7955": "nan", "7960": "nan", "7965": "nan", "7970": "nan", "7975": "nan", "7980": "nan", "7985": "nan", "7990": "nan", "7995": "nan", "8000": 3.39834, "8005": "nan", "8010": "nan", "8015": "nan", "8020": "nan", "8025": "nan", "8030": "nan", "8035": "nan", "8040": "nan", "8045": "nan", "8050": "nan", "8055": "nan", "8060": "nan", "8065": "nan", "8070": "nan", "8075": "nan", "8080": "nan", "8085": "nan", "8090": "nan", "8095": "nan", "8100": 3.39716, "8105": "nan", "8110": "nan", "8115": "nan", "8120": "nan", "8125": "nan", "8130": "nan", "8135": "nan", "8140": "nan", "8145": "nan", "8150": "nan", "8155": "nan", "8160": "nan", "8165": "nan", "8170": "nan", "8175": "nan", "8180": "nan", "8185": "nan", "8190": "nan", "8195": "nan", "8200": 3.39809, "8205": "nan", "8210": "nan", "8215": "nan", "8220": "nan", "8225": "nan", "8230": "nan", "8235": "nan", "8240": "nan", "8245": "nan", "8250": "nan", "8255": "nan", "8260": "nan", "8265": "nan", "8270": "nan", "8275": "nan", "8280": "nan", "8285": "nan", "8290": "nan", "8295": "nan", "8300": 3.39856, "8305": "nan", "8310": "nan", "8315": "nan", "8320": "nan", "8325": "nan", "8330": "nan", "8335": "nan", "8340": "nan", "8345": "nan", "8350": "nan", "8355": "nan", "8360": "nan", "8365": "nan", "8370": "nan", "8375": "nan", "8380": "nan", "8385": "nan", "8390": "nan", "8395": "nan", "8400": 3.39822, "8405": "nan", "8410": "nan", "8415": "nan", "8420": "nan", "8425": "nan", "8430": "nan", "8435": "nan", "8440": "nan", "8445": "nan", "8450": "nan", "8455": "nan", "8460": "nan", "8465": "nan", "8470": "nan", "8475": "nan", "8480": "nan", "8485": "nan", "8490": "nan", "8495": "nan", "8500": 3.39927, "8505": "nan", "8510": "nan", "8515": "nan", "8520": "nan", "8525": "nan", "8530": "nan", "8535": "nan", "8540": "nan", "8545": "nan", "8550": "nan", "8555": "nan", "8560": "nan", "8565": "nan", "8570": "nan", "8575": "nan", "8580": "nan", "8585": "nan", "8590": "nan", "8595": "nan", "8600": 3.40049, "8605": "nan", "8610": "nan", "8615": "nan", "8620": "nan", "8625": "nan", "8630": "nan", "8635": "nan", "8640": "nan", "8645": "nan", "8650": "nan", "8655": "nan", "8660": "nan", "8665": "nan", "8670": "nan", "8675": "nan", "8680": "nan", "8685": "nan", "8690": "nan", "8695": "nan", "8700": 3.39943, "8705": "nan", "8710": "nan", "8715": "nan", "8720": "nan", "8725": "nan", "8730": "nan", "8735": "nan", "8740": "nan", "8745": "nan", "8750": "nan", "8755": "nan", "8760": "nan", "8765": "nan", "8770": "nan", "8775": "nan", "8780": "nan", "8785": "nan", "8790": "nan", "8795": "nan", "8800": 3.3999, "8805": "nan", "8810": "nan", "8815": "nan", "8820": "nan", "8825": "nan", "8830": "nan", "8835": "nan", "8840": "nan", "8845": "nan", "8850": "nan", "8855": "nan", "8860": "nan", "8865": "nan", "8870": "nan", "8875": "nan", "8880": "nan", "8885": "nan", "8890": "nan", "8895": "nan", "8900": 3.39978, "8905": "nan", "8910": "nan", "8915": "nan", "8920": "nan", "8925": "nan", "8930": "nan", "8935": "nan", "8940": "nan", "8945": "nan", "8950": "nan", "8955": "nan", "8960": "nan", "8965": "nan", "8970": "nan", "8975": "nan", "8980": "nan", "8985": "nan", "8990": "nan", "8995": "nan", "9000": 3.40051, "9005": "nan", "9010": "nan", "9015": "nan", "9020": "nan", "9025": "nan", "9030": "nan", "9035": "nan", "9040": "nan", "9045": "nan", "9050": "nan", "9055": "nan", "9060": "nan", "9065": "nan", "9070": "nan", "9075": "nan", "9080": "nan", "9085": "nan", "9090": "nan", "9095": "nan", "9100": 3.39875, "9105": "nan", "9110": "nan", "9115": "nan", "9120": "nan", "9125": "nan", "9130": "nan", "9135": "nan", "9140": "nan", "9145": "nan", "9150": "nan", "9155": "nan", "9160": "nan", "9165": "nan", "9170": "nan", "9175": "nan", "9180": "nan", "9185": "nan", "9190": "nan", "9195": "nan", "9200": 3.39972, "9205": "nan", "9210": "nan", "9215": "nan", "9220": "nan", "9225": "nan", "9230": "nan", "9235": "nan", "9240": "nan", "9245": "nan", "9250": "nan", "9255": "nan", "9260": "nan", "9265": "nan", "9270": "nan", "9275": "nan", "9280": "nan", "9285": "nan", "9290": "nan", "9295": "nan", "9300": 3.40071, "9305": "nan", "9310": "nan", "9315": "nan", "9320": "nan", "9325": "nan", "9330": "nan", "9335": "nan", "9340": "nan", "9345": "nan", "9350": "nan", "9355": "nan", "9360": "nan", "9365": "nan", "9370": "nan", "9375": "nan", "9380": "nan", "9385": "nan", "9390": "nan", "9395": "nan", "9400": 3.40005, "9405": "nan", "9410": "nan", "9415": "nan", "9420": "nan", "9425": "nan", "9430": "nan", "9435": "nan", "9440": "nan", "9445": "nan", "9450": "nan", "9455": "nan", "9460": "nan", "9465": "nan", "9470": "nan", "9475": "nan", "9480": "nan", "9485": "nan", "9490": "nan", "9495": "nan", "9500": 3.39896, "9505": "nan", "9510": "nan", "9515": "nan", "9520": "nan", "9525": "nan", "9530": "nan", "9535": "nan", "9540": "nan", "9545": "nan", "9550": "nan", "9555": "nan", "9560": "nan", "9565": "nan", "9570": "nan", "9575": "nan", "9580": "nan", "9585": "nan", "9590": "nan", "9595": "nan", "9600": 3.39953, "9605": "nan", "9610": "nan", "9615": "nan", "9620": "nan", "9625": "nan", "9630": "nan", "9635": "nan", "9640": "nan", "9645": "nan", "9650": "nan", "9655": "nan", "9660": "nan", "9665": "nan", "9670": "nan", "9675": "nan", "9680": "nan", "9685": "nan", "9690": "nan", "9695": "nan", "9700": 3.39866, "9705": "nan", "9710": "nan", "9715": "nan", "9720": "nan", "9725": "nan", "9730": "nan", "9735": "nan", "9740": "nan", "9745": "nan", "9750": "nan", "9755": "nan", "9760": "nan", "9765": "nan", "9770": "nan", "9775": "nan", "9780": "nan", "9785": "nan", "9790": "nan", "9795": "nan", "9800": 3.40009, "9805": "nan", "9810": "nan", "9815": "nan", "9820": "nan", "9825": "nan", "9830": "nan", "9835": "nan", "9840": "nan", "9845": "nan", "9850": "nan", "9855": "nan", "9860": "nan", "9865": "nan", "9870": "nan", "9875": "nan", "9880": "nan", "9885": "nan", "9890": "nan", "9895": "nan", "9900": 3.39979, "9905": "nan", "9910": "nan", "9915": "nan", "9920": "nan", "9925": "nan", "9930": "nan", "9935": "nan", "9940": "nan", "9945": "nan", "9950": "nan", "9955": "nan", "9960": "nan", "9965": "nan", "9970": "nan", "9975": "nan", "9980": "nan", "9985": "nan", "9990": "nan", "9995": "nan", "10000": 3.3996, "10005": "nan", "10010": "nan", "10015": "nan", "10020": "nan", "10025": "nan", "10030": "nan", "10035": "nan", "10040": "nan", "10045": "nan", "10050": "nan", "10055": "nan", "10060": "nan", "10065": "nan", "10070": "nan", "10075": "nan", "10080": "nan", "10085": "nan", "10090": "nan", "10095": "nan", "10100": 3.39815, "10105": "nan", "10110": "nan", "10115": "nan", "10120": "nan", "10125": "nan", "10130": "nan", "10135": "nan", "10140": "nan", "10145": "nan", "10150": "nan", "10155": "nan", "10160": "nan", "10165": "nan", "10170": "nan", "10175": "nan", "10180": "nan", "10185": "nan", "10190": "nan", "10195": "nan", "10200": 3.3996, "10205": "nan", "10210": "nan", "10215": "nan", "10220": "nan", "10225": "nan", "10230": "nan", "10235": "nan", "10240": "nan", "10245": "nan", "10250": "nan", "10255": "nan", "10260": "nan", "10265": "nan", "10270": "nan", "10275": "nan", "10280": "nan", "10285": "nan", "10290": "nan", "10295": "nan", "10300": 3.40067, "10305": "nan", "10310": "nan", "10315": "nan", "10320": "nan", "10325": "nan", "10330": "nan", "10335": "nan", "10340": "nan", "10345": "nan", "10350": "nan", "10355": "nan", "10360": "nan", "10365": "nan", "10370": "nan", "10375": "nan", "10380": "nan", "10385": "nan", "10390": "nan", "10395": "nan", "10400": 3.39985, "10405": "nan", "10410": "nan", "10415": "nan", "10420": "nan", "10425": "nan", "10430": "nan", "10435": "nan", "10440": "nan", "10445": "nan", "10450": "nan", "10455": "nan", "10460": "nan", "10465": "nan", "10470": "nan", "10475": "nan", "10480": "nan", "10485": "nan", "10490": "nan", "10495": "nan", "10500": 3.39923, "10505": "nan", "10510": "nan", "10515": "nan", "10520": "nan", "10525": "nan", "10530": "nan", "10535": "nan", "10540": "nan", "10545": "nan", "10550": "nan", "10555": "nan", "10560": "nan", "10565": "nan", "10570": "nan", "10575": "nan", "10580": "nan", "10585": "nan", "10590": "nan", "10595": "nan", "10600": 3.40087, "10605": "nan", "10610": "nan", "10615": "nan", "10620": "nan", "10625": "nan", "10630": "nan", "10635": "nan", "10640": "nan", "10645": "nan", "10650": "nan", "10655": "nan", "10660": "nan", "10665": "nan", "10670": "nan", "10675": "nan", "10680": "nan", "10685": "nan", "10690": "nan", "10695": "nan", "10700": 3.40098, "10705": "nan", "10710": "nan", "10715": "nan", "10720": "nan", "10725": "nan", "10730": "nan", "10735": "nan", "10740": "nan", "10745": "nan", "10750": "nan", "10755": "nan", "10760": "nan", "10765": "nan", "10770": "nan", "10775": "nan", "10780": "nan", "10785": "nan", "10790": "nan", "10795": "nan", "10800": 3.39973, "10805": "nan", "10810": "nan", "10815": "nan", "10820": "nan", "10825": "nan", "10830": "nan", "10835": "nan", "10840": "nan", "10845": "nan", "10850": "nan", "10855": "nan", "10860": "nan", "10865": "nan", "10870": "nan", "10875": "nan", "10880": "nan", "10885": "nan", "10890": "nan", "10895": "nan", "10900": 3.66029, "10905": "nan", "10910": "nan", "10915": "nan", "10920": "nan", "10925": "nan", "10930": "nan", "10935": "nan", "10940": "nan", "10945": "nan", "10950": "nan", "10955": "nan", "10960": "nan", "10965": "nan", "10970": "nan", "10975": "nan", "10980": "nan", "10985": "nan", "10990": "nan", "10995": "nan", "11000": 3.38756, "11005": "nan", "11010": "nan", "11015": "nan", "11020": "nan", "11025": "nan", "11030": "nan", "11035": "nan", "11040": "nan", "11045": "nan", "11050": "nan", "11055": "nan", "11060": "nan", "11065": "nan", "11070": "nan", "11075": "nan", "11080": "nan", "11085": "nan", "11090": "nan", "11095": "nan", "11100": 3.38656, "11105": "nan", "11110": "nan", "11115": "nan", "11120": "nan", "11125": "nan", "11130": "nan", "11135": "nan", "11140": "nan", "11145": "nan", "11150": "nan", "11155": "nan", "11160": "nan", "11165": "nan", "11170": "nan", "11175": "nan", "11180": "nan", "11185": "nan", "11190": "nan", "11195": "nan", "11200": 3.38765, "11205": "nan", "11210": "nan", "11215": "nan", "11220": "nan", "11225": "nan", "11230": "nan", "11235": "nan", "11240": "nan", "11245": "nan", "11250": "nan", "11255": "nan", "11260": "nan", "11265": "nan", "11270": "nan", "11275": "nan", "11280": "nan", "11285": "nan", "11290": "nan", "11295": "nan", "11300": 3.38835, "11305": "nan", "11310": "nan", "11315": "nan", "11320": "nan", "11325": "nan", "11330": "nan", "11335": "nan", "11340": "nan", "11345": "nan", "11350": "nan", "11355": "nan", "11360": "nan", "11365": "nan", "11370": "nan", "11375": "nan", "11380": "nan", "11385": "nan", "11390": "nan", "11395": "nan", "11400": 3.38844, "11405": "nan", "11410": "nan", "11415": "nan", "11420": "nan", "11425": "nan", "11430": "nan", "11435": "nan", "11440": "nan", "11445": "nan", "11450": "nan", "11455": "nan", "11460": "nan", "11465": "nan", "11470": "nan", "11475": "nan", "11480": "nan", "11485": "nan", "11490": "nan", "11495": "nan", "11500": 3.38793, "11505": "nan", "11510": "nan", "11515": "nan", "11520": "nan", "11525": "nan", "11530": "nan", "11535": "nan", "11540": "nan", "11545": "nan", "11550": "nan", "11555": "nan", "11560": "nan", "11565": "nan", "11570": "nan", "11575": "nan", "11580": "nan", "11585": "nan", "11590": "nan", "11595": "nan", "11600": 3.38725, "11605": "nan", "11610": "nan", "11615": "nan", "11620": "nan", "11625": "nan", "11630": "nan", "11635": "nan", "11640": "nan", "11645": "nan", "11650": "nan", "11655": "nan", "11660": "nan", "11665": "nan", "11670": "nan", "11675": "nan", "11680": "nan", "11685": "nan", "11690": "nan", "11695": "nan", "11700": 3.38738, "11705": "nan", "11710": "nan", "11715": "nan", "11720": "nan", "11725": "nan", "11730": "nan", "11735": "nan", "11740": "nan", "11745": "nan", "11750": "nan", "11755": "nan", "11760": "nan", "11765": "nan", "11770": "nan", "11775": "nan", "11780": "nan", "11785": "nan", "11790": "nan", "11795": "nan", "11800": 3.38748, "11805": "nan", "11810": "nan", "11815": "nan", "11820": "nan", "11825": "nan", "11830": "nan", "11835": "nan", "11840": "nan", "11845": "nan", "11850": "nan", "11855": "nan", "11860": "nan", "11865": "nan", "11870": "nan", "11875": "nan", "11880": "nan", "11885": "nan", "11890": "nan", "11895": "nan", "11900": 3.38839, "11905": "nan", "11910": "nan", "11915": "nan", "11920": "nan", "11925": "nan", "11930": "nan", "11935": "nan", "11940": "nan", "11945": "nan", "11950": "nan", "11955": "nan", "11960": "nan", "11965": "nan", "11970": "nan", "11975": "nan", "11980": "nan", "11985": "nan", "11990": "nan", "11995": "nan", "12000": 3.38814, "12005": "nan", "12010": "nan", "12015": "nan", "12020": "nan", "12025": "nan", "12030": "nan", "12035": "nan", "12040": "nan", "12045": "nan", "12050": "nan", "12055": "nan", "12060": "nan", "12065": "nan", "12070": "nan", "12075": "nan", "12080": "nan", "12085": "nan", "12090": "nan", "12095": "nan", "12100": 3.38677, "12105": "nan", "12110": "nan", "12115": "nan", "12120": "nan", "12125": "nan", "12130": "nan", "12135": "nan", "12140": "nan", "12145": "nan", "12150": "nan", "12155": "nan", "12160": "nan", "12165": "nan", "12170": "nan", "12175": "nan", "12180": "nan", "12185": "nan", "12190": "nan", "12195": "nan", "12200": 3.38679, "12205": "nan", "12210": "nan", "12215": "nan", "12220": "nan", "12225": "nan", "12230": "nan", "12235": "nan", "12240": "nan", "12245": "nan", "12250": "nan", "12255": "nan", "12260": "nan", "12265": "nan", "12270": "nan", "12275": "nan", "12280": "nan", "12285": "nan", "12290": "nan", "12295": "nan", "12300": 3.38609, "12305": "nan", "12310": "nan", "12315": "nan", "12320": "nan", "12325": "nan", "12330": "nan", "12335": "nan", "12340": "nan", "12345": "nan", "12350": "nan", "12355": "nan", "12360": "nan", "12365": "nan", "12370": "nan", "12375": "nan", "12380": "nan", "12385": "nan", "12390": "nan", "12395": "nan", "12400": 3.38665, "12405": "nan", "12410": "nan", "12415": "nan", "12420": "nan", "12425": "nan", "12430": "nan", "12435": "nan", "12440": "nan", "12445": "nan", "12450": "nan", "12455": "nan", "12460": "nan", "12465": "nan", "12470": "nan", "12475": "nan", "12480": "nan", "12485": "nan", "12490": "nan", "12495": "nan", "12500": 3.38727, "12505": "nan", "12510": "nan", "12515": "nan", "12520": "nan", "12525": "nan", "12530": "nan", "12535": "nan", "12540": "nan", "12545": "nan", "12550": "nan", "12555": "nan", "12560": "nan", "12565": "nan", "12570": "nan", "12575": "nan", "12580": "nan", "12585": "nan", "12590": "nan", "12595": "nan", "12600": 3.38752, "12605": "nan", "12610": "nan", "12615": "nan", "12620": "nan", "12625": "nan", "12630": "nan", "12635": "nan", "12640": "nan", "12645": "nan", "12650": "nan", "12655": "nan", "12660": "nan", "12665": "nan", "12670": "nan", "12675": "nan", "12680": "nan", "12685": "nan", "12690": "nan", "12695": "nan", "12700": 3.38807, "12705": "nan", "12710": "nan", "12715": "nan", "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}}
\ No newline at end of file
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": 12.95636,
+ "5": 12.91493,
+ "10": 12.06071,
+ "15": 11.399,
+ "20": 10.42783,
+ "25": 9.97708,
+ "30": 9.62862,
+ "35": 9.36622,
+ "40": 9.17249,
+ "45": 9.00253,
+ "50": 8.8424,
+ "55": 8.63901,
+ "60": 8.60961,
+ "65": 8.47692,
+ "70": 8.46329,
+ "75": 8.28991,
+ "80": 8.16093,
+ "85": 8.10818,
+ "90": 7.92141,
+ "95": 7.90423,
+ "100": 7.79003,
+ "105": 7.66332,
+ "110": 7.53152,
+ "115": 7.44265,
+ "120": 7.47216,
+ "125": 7.42219,
+ "130": 7.27255,
+ "135": 7.27337,
+ "140": 7.17596,
+ "145": 7.06835,
+ "150": 7.15145,
+ "155": 7.02838,
+ "160": 6.92535,
+ "165": 6.89994,
+ "170": 6.81047,
+ "175": 6.86004,
+ "180": 6.81466,
+ "185": 6.75648,
+ "190": 6.68105,
+ "195": 6.63303,
+ "200": 6.67654,
+ "205": 6.69053,
+ "210": 6.55683,
+ "215": 6.54183,
+ "220": 6.50946,
+ "225": 6.48416,
+ "230": 6.51062,
+ "235": 6.43666,
+ "240": 6.38442,
+ "245": 6.35429,
+ "250": 6.32154,
+ "255": 6.45338,
+ "260": 6.34969,
+ "265": 6.26896,
+ "270": 6.29578,
+ "275": 6.26435,
+ "280": 6.21097,
+ "285": 6.24029,
+ "290": 6.17061,
+ "295": 6.11197,
+ "300": 6.10341,
+ "305": 6.02939,
+ "310": 6.08873,
+ "315": 6.08978,
+ "320": 5.98033,
+ "325": 5.94023,
+ "330": 6.00151,
+ "335": 6.03605,
+ "340": 5.95411,
+ "345": 5.93918,
+ "350": 5.90365,
+ "355": 5.86832,
+ "360": 5.86866,
+ "365": 5.85156,
+ "370": 5.83353,
+ "375": 5.84167,
+ "380": 5.8501,
+ "385": 5.78541,
+ "390": 5.79439,
+ "395": 5.69008,
+ "400": 5.67247,
+ "405": 5.68516,
+ "410": 5.67869,
+ "415": 5.72763,
+ "420": 5.64672,
+ "425": 5.66169,
+ "430": 5.63737,
+ "435": 5.58968,
+ "440": 5.61896,
+ "445": 5.5645,
+ "450": 5.58508,
+ "455": 5.50743,
+ "460": 5.50847,
+ "465": 5.59451,
+ "470": 5.60347,
+ "475": 5.51168,
+ "480": 5.48287,
+ "485": 5.49811,
+ "490": 5.46891,
+ "495": 5.47163,
+ "500": 5.4091,
+ "505": 5.38671,
+ "510": 5.45239,
+ "515": 5.4142,
+ "520": 5.43017,
+ "525": 5.29804,
+ "530": 5.33295,
+ "535": 5.32694,
+ "540": 5.34564,
+ "545": 5.3566,
+ "550": 5.372,
+ "555": 5.21825,
+ "560": 5.35828,
+ "565": 5.29055,
+ "570": 5.25088,
+ "575": 5.28196,
+ "580": 5.21238,
+ "585": 5.22679,
+ "590": 5.23078,
+ "595": 5.20361,
+ "600": 5.26226,
+ "605": 5.20734,
+ "610": 5.20433,
+ "615": 5.16551,
+ "620": 5.1828,
+ "625": 5.21942,
+ "630": 5.14048,
+ "635": 5.10207,
+ "640": 5.06829,
+ "645": 5.11226,
+ "650": 5.13069,
+ "655": 5.11083,
+ "660": 5.05053,
+ "665": 5.07037,
+ "670": 5.03239,
+ "675": 5.01416,
+ "680": 5.00235,
+ "685": 4.98481,
+ "690": 4.98266,
+ "695": 4.94418,
+ "700": 4.96036,
+ "705": 4.91447,
+ "710": 4.94457,
+ "715": 4.84337,
+ "720": 4.82105,
+ "725": 4.76373,
+ "730": 4.83281,
+ "735": 4.78972,
+ "740": 4.78672,
+ "745": 4.69179,
+ "750": 4.71771,
+ "755": 4.77161,
+ "760": 4.76703,
+ "765": 4.69822,
+ "770": 4.67903,
+ "775": 4.64475,
+ "780": 4.65639,
+ "785": 4.72058,
+ "790": 4.63092,
+ "795": 4.57774,
+ "800": 4.56798,
+ "805": 4.58802,
+ "810": 4.63352,
+ "815": 4.57408,
+ "820": 4.6076,
+ "825": 4.56989,
+ "830": 4.561,
+ "835": 4.51372,
+ "840": 4.4407,
+ "845": 4.46506,
+ "850": 4.43154,
+ "855": 4.47148,
+ "860": 4.42231,
+ "865": 4.45963,
+ "870": 4.43819,
+ "875": 4.34074,
+ "880": 4.4007,
+ "885": 4.35925,
+ "890": 4.41003,
+ "895": 4.41996,
+ "900": 4.36598,
+ "905": 4.30337,
+ "910": 4.32788,
+ "915": 4.3191,
+ "920": 4.34354,
+ "925": 4.36176,
+ "930": 4.27417,
+ "935": 4.29383,
+ "940": 4.31971,
+ "945": 4.29013,
+ "950": 4.32454,
+ "955": 4.2605,
+ "960": 4.16766,
+ "965": 4.26398,
+ "970": 4.2642,
+ "975": 4.21835,
+ "980": 4.20542,
+ "985": 4.16834,
+ "990": 4.13159,
+ "995": 4.15896,
+ "1000": 4.20986,
+ "1005": 4.17818,
+ "1010": 4.14706,
+ "1015": 4.12779,
+ "1020": 4.14585,
+ "1025": 4.20083,
+ "1030": 4.10606,
+ "1035": 4.06602,
+ "1040": 4.104,
+ "1045": 4.1058,
+ "1050": 4.1194,
+ "1055": 4.07941,
+ "1060": 4.10741,
+ "1065": 4.05729,
+ "1070": 4.04856,
+ "1075": 4.06438,
+ "1080": 4.06666,
+ "1085": 4.06126,
+ "1090": 4.03111,
+ "1095": 4.09404,
+ "1100": 4.05366,
+ "1105": 4.06449,
+ "1110": 4.03061,
+ "1115": 4.00934,
+ "1120": 3.98834,
+ "1125": 3.98655,
+ "1130": 4.03219,
+ "1135": 4.01012,
+ "1140": 4.00354,
+ "1145": 3.94596,
+ "1150": 4.03253,
+ "1155": 3.98732,
+ "1160": 3.95928,
+ "1165": 3.86152,
+ "1170": 3.92565,
+ "1175": 3.94727,
+ "1180": 3.94594,
+ "1185": 3.98008,
+ "1190": 3.92521,
+ "1195": 3.93802,
+ "1200": 3.86632,
+ "1205": 3.87409,
+ "1210": 3.99026,
+ "1215": 3.82709,
+ "1220": 3.86889,
+ "1225": 3.80788,
+ "1230": 3.90481,
+ "1235": 3.89023,
+ "1240": 3.86358,
+ "1245": 3.80325,
+ "1250": 3.83985,
+ "1255": 3.87091,
+ "1260": 3.88382,
+ "1265": 3.79458,
+ "1270": 3.86062,
+ "1275": 3.83631,
+ "1280": 3.81782,
+ "1285": 3.84386,
+ "1290": 3.86067,
+ "1295": 3.84325,
+ "1300": 3.82563,
+ "1305": 3.81469,
+ "1310": 3.81207,
+ "1315": 3.81033,
+ "1320": 3.8231,
+ "1325": 3.70845,
+ "1330": 3.78885,
+ "1335": 3.77725,
+ "1340": 3.75347,
+ "1345": 3.75107,
+ "1350": 3.71799,
+ "1355": 3.76936,
+ "1360": 3.72335,
+ "1365": 3.72556,
+ "1370": 3.73231,
+ "1375": 3.74918,
+ "1380": 3.75889,
+ "1385": 3.73782,
+ "1390": 3.64839,
+ "1395": 3.74238,
+ "1400": 3.72646,
+ "1405": 3.66878,
+ "1410": 3.66425,
+ "1415": 3.65271,
+ "1420": 3.69535,
+ "1425": 3.70138,
+ "1430": 3.67121,
+ "1435": 3.66402,
+ "1440": 3.65193,
+ "1445": 3.68875,
+ "1450": 3.67616,
+ "1455": 3.63441,
+ "1460": 3.64566,
+ "1465": 3.67157,
+ "1470": 3.63262,
+ "1475": 3.69518,
+ "1480": 3.65146,
+ "1485": 3.66092,
+ "1490": 3.62758,
+ "1495": 3.59427,
+ "1500": 3.62478,
+ "1505": 3.68785,
+ "1510": 3.563,
+ "1515": 3.60002,
+ "1520": 3.62112,
+ "1525": 3.60225,
+ "1530": 3.58683,
+ "1535": 3.57992,
+ "1540": 3.60666,
+ "1545": 3.59612,
+ "1550": 3.55628,
+ "1555": 3.56998,
+ "1560": 3.60483,
+ "1565": 3.60236,
+ "1570": 3.60117,
+ "1575": 3.54502,
+ "1580": 3.58519,
+ "1585": 3.57087,
+ "1590": 3.45241,
+ "1595": 3.50264,
+ "1600": 3.49859,
+ "1605": 3.54796,
+ "1610": 3.57472,
+ "1615": 3.49729,
+ "1620": 3.51578,
+ "1625": 3.46192,
+ "1630": 3.49755,
+ "1635": 3.53974,
+ "1640": 3.5315,
+ "1645": 3.53664,
+ "1650": 3.49268,
+ "1655": 3.4598,
+ "1660": 3.51996,
+ "1665": 3.44574,
+ "1670": 3.50193,
+ "1675": 3.48185,
+ "1680": 3.4727,
+ "1685": 3.47728,
+ "1690": 3.47809,
+ "1695": 3.49119,
+ "1700": 3.44575,
+ "1705": 3.39167,
+ "1710": 3.49469,
+ "1715": 3.49059,
+ "1720": 3.42628,
+ "1725": 3.42554,
+ "1730": 3.4079,
+ "1735": 3.45263,
+ "1740": 3.45054,
+ "1745": 3.43655,
+ "1750": 3.39646,
+ "1755": 3.39273,
+ "1760": 3.38735,
+ "1765": 3.4222,
+ "1770": 3.44622,
+ "1775": 3.387,
+ "1780": 3.43414,
+ "1785": 3.40568,
+ "1790": 3.37632,
+ "1795": 3.39987,
+ "1800": 3.33185,
+ "1805": 3.3833,
+ "1810": 3.32286,
+ "1815": 3.41563,
+ "1820": 3.40441,
+ "1825": 3.36911,
+ "1830": 3.32658,
+ "1835": 3.4159,
+ "1840": 3.38999,
+ "1845": 3.41729,
+ "1850": 3.37087,
+ "1855": 3.36447,
+ "1860": 3.33,
+ "1865": 3.3828,
+ "1870": 3.29994,
+ "1875": 3.42203,
+ "1880": 3.31553,
+ "1885": 3.34452,
+ "1890": 3.34191,
+ "1895": 3.38708,
+ "1900": 3.37396,
+ "1905": 3.3097,
+ "1910": 3.32571,
+ "1915": 3.31274,
+ "1920": 3.34648,
+ "1925": 3.32051,
+ "1930": 3.30099,
+ "1935": 3.30725,
+ "1940": 3.35696,
+ "1945": 3.25663,
+ "1950": 3.4137,
+ "1955": 3.29079,
+ "1960": 3.29664,
+ "1965": 3.26156,
+ "1970": 3.29433,
+ "1975": 3.3407,
+ "1980": 3.33342,
+ "1985": 3.23349,
+ "1990": 3.29114,
+ "1995": 3.28958,
+ "2000": 3.27902,
+ "2005": 3.27718,
+ "2010": 3.26011,
+ "2015": 3.21811,
+ "2020": 3.25285,
+ "2025": 3.26578,
+ "2030": 3.27801,
+ "2035": 3.3008,
+ "2040": 3.25364,
+ "2045": 3.22173,
+ "2050": 3.27331,
+ "2055": 3.31203,
+ "2060": 3.29841,
+ "2065": 3.22375,
+ "2070": 3.27847,
+ "2075": 3.23278,
+ "2080": 3.21071,
+ "2085": 3.27722,
+ "2090": 3.1336,
+ "2095": 3.27283,
+ "2100": 3.22452,
+ "2105": 3.19481,
+ "2110": 3.19679,
+ "2115": 3.23269,
+ "2120": 3.17039,
+ "2125": 3.20328,
+ "2130": 3.21145,
+ "2135": 3.27257,
+ "2140": 3.18877,
+ "2145": 3.19442,
+ "2150": 3.20116,
+ "2155": 3.23563,
+ "2160": 3.19845,
+ "2165": 3.24654,
+ "2170": 3.2216,
+ "2175": 3.16252,
+ "2180": 3.21306,
+ "2185": 3.24681,
+ "2190": 3.22862,
+ "2195": 3.14645,
+ "2200": 3.19104,
+ "2205": 3.15804,
+ "2210": 3.11856,
+ "2215": 3.18638,
+ "2220": 3.18386,
+ "2225": 3.16933,
+ "2230": 3.13649,
+ "2235": 3.17995,
+ "2240": 3.21834,
+ "2245": 3.1834,
+ "2250": 3.19663,
+ "2255": 3.12028,
+ "2260": 3.14221,
+ "2265": 3.20381,
+ "2270": 3.19267,
+ "2275": 3.16805,
+ "2280": 3.16867,
+ "2285": 3.1601,
+ "2290": 3.16301,
+ "2295": 3.18761,
+ "2300": 3.12966,
+ "2305": 3.17206,
+ "2310": 3.11553,
+ "2315": 3.06971,
+ "2320": 3.12085,
+ "2325": 3.16434,
+ "2330": 3.11423,
+ "2335": 3.11076,
+ "2340": 3.15588,
+ "2345": 3.10477,
+ "2350": 3.11732,
+ "2355": 3.11185,
+ "2360": 3.15506,
+ "2365": 3.11076,
+ "2370": 3.16356,
+ "2375": 3.14222,
+ "2380": 3.10402,
+ "2385": 3.0741,
+ "2390": 3.09007,
+ "2395": 3.07964,
+ "2400": 3.0826,
+ "2405": 3.08587,
+ "2410": 3.07576,
+ "2415": 3.07852,
+ "2420": 3.06533,
+ "2425": 3.07275,
+ "2430": 3.06933,
+ "2435": 3.0564,
+ "2440": 3.08532,
+ "2445": 3.05739,
+ "2450": 3.11279,
+ "2455": 3.15238,
+ "2460": 3.07391,
+ "2465": 3.07127,
+ "2470": 3.03425,
+ "2475": 3.0807,
+ "2480": 3.08629,
+ "2485": 3.0577,
+ "2490": 3.05332,
+ "2495": 3.07752,
+ "2500": 3.06183,
+ "2505": 3.11062,
+ "2510": 3.11173,
+ "2515": 3.04133,
+ "2520": 3.06465,
+ "2525": 3.01687,
+ "2530": 3.03591,
+ "2535": 3.07924,
+ "2540": 3.0737,
+ "2545": 3.04184,
+ "2550": 2.99916,
+ "2555": 3.07523,
+ "2560": 3.05422,
+ "2565": 3.10948,
+ "2570": 2.99967,
+ "2575": 3.04042,
+ "2580": 3.07793,
+ "2585": 3.01288,
+ "2590": 3.06395,
+ "2595": 2.99387,
+ "2600": 3.05698,
+ "2605": 3.04109,
+ "2610": 3.05189,
+ "2615": 3.05276,
+ "2620": 2.98134,
+ "2625": 3.00912,
+ "2630": 3.02901,
+ "2635": 3.05103,
+ "2640": 3.01503,
+ "2645": 3.04801,
+ "2650": 3.00882,
+ "2655": 2.98542,
+ "2660": 3.00801,
+ "2665": 3.03349,
+ "2670": 2.97627,
+ "2675": 2.96043,
+ "2680": 2.99062,
+ "2685": 3.00007,
+ "2690": 2.99293,
+ "2695": 2.98783,
+ "2700": 3.02201,
+ "2705": 2.97427,
+ "2710": 2.96634,
+ "2715": 2.95921,
+ "2720": 3.02806,
+ "2725": 3.00529,
+ "2730": 3.0322,
+ "2735": 3.02478,
+ "2740": 2.9826,
+ "2745": 3.00739,
+ "2750": 3.00234,
+ "2755": 2.96406,
+ "2760": 2.9934,
+ "2765": 3.00331,
+ "2770": 2.97181,
+ "2775": 2.98931,
+ "2780": 3.01168,
+ "2785": 2.94404,
+ "2790": 2.95915,
+ "2795": 2.9483,
+ "2800": 2.95122,
+ "2805": 2.92887,
+ "2810": 2.97394,
+ "2815": 2.95273,
+ "2820": 3.09901,
+ "2825": 3.09029,
+ "2830": 3.06294,
+ "2835": 2.99075,
+ "2840": 2.99097,
+ "2845": 3.16374,
+ "2850": 3.0727,
+ "2855": 2.9904,
+ "2860": 2.95406,
+ "2865": 2.90905,
+ "2870": 2.97507,
+ "2875": 2.91077,
+ "2880": 2.94853,
+ "2885": 2.92014,
+ "2890": 2.97403,
+ "2895": 2.92299,
+ "2900": 2.94051,
+ "2905": 3.04074,
+ "2910": 2.94425,
+ "2915": 2.96669,
+ "2920": 2.95973,
+ "2925": 2.94592,
+ "2930": 2.95038,
+ "2935": 2.93537,
+ "2940": 2.93072,
+ "2945": 3.05785,
+ "2950": 3.04355,
+ "2955": 2.9389,
+ "2960": 2.99183,
+ "2965": 2.87016,
+ "2970": 2.95512,
+ "2975": 2.98818,
+ "2980": 2.93644,
+ "2985": 3.03092,
+ "2990": 2.93432,
+ "2995": 2.86558,
+ "3000": 2.93207,
+ "3005": 2.89421,
+ "3010": 2.93131,
+ "3015": 2.93536,
+ "3020": 2.98292,
+ "3025": 2.97341,
+ "3030": 2.94829,
+ "3035": 2.9713,
+ "3040": 2.91863,
+ "3045": 2.83181,
+ "3050": 2.8976,
+ "3055": 2.89038,
+ "3060": 2.92509,
+ "3065": 2.91825,
+ "3070": 2.88122,
+ "3075": 2.87986,
+ "3080": 2.96134,
+ "3085": 2.92236,
+ "3090": 2.93503,
+ "3095": 2.92516,
+ "3100": 2.86898,
+ "3105": 2.93926,
+ "3110": 2.90416,
+ "3115": 2.9415,
+ "3120": 2.9498,
+ "3125": 2.85617,
+ "3130": 2.93084,
+ "3135": 2.92732,
+ "3140": 2.87396,
+ "3145": 2.92342,
+ "3150": 2.86658,
+ "3155": 2.88336,
+ "3160": 2.84999,
+ "3165": 2.86518,
+ "3170": 2.90374,
+ "3175": 2.90679,
+ "3180": 2.86664,
+ "3185": 2.98102,
+ "3190": 2.92651,
+ "3195": 2.93655,
+ "3200": 2.92553,
+ "3205": 2.86435,
+ "3210": 2.87237,
+ "3215": 2.91522,
+ "3220": 2.87693,
+ "3225": 2.87047,
+ "3230": 2.81633,
+ "3235": 2.87742,
+ "3240": 2.881,
+ "3245": 2.90693,
+ "3250": 2.86631,
+ "3255": 2.85231,
+ "3260": 2.87001,
+ "3265": 2.87376,
+ "3270": 2.84904,
+ "3275": 2.86952,
+ "3280": 2.81176,
+ "3285": 2.81325,
+ "3290": 2.87177,
+ "3295": 2.90468,
+ "3300": 2.87538,
+ "3305": 2.86924,
+ "3310": 2.86384,
+ "3315": 2.81591,
+ "3320": 2.83934,
+ "3325": 2.83244,
+ "3330": 2.8359,
+ "3335": 2.84976,
+ "3340": 2.83631,
+ "3345": 2.85001,
+ "3350": 2.89341,
+ "3355": 2.88029,
+ "3360": 2.80501,
+ "3365": 2.85662,
+ "3370": 2.85298,
+ "3375": 2.84567,
+ "3380": 2.85507,
+ "3385": 2.87967,
+ "3390": 2.8654,
+ "3395": 2.81392,
+ "3400": 2.7912,
+ "3405": 2.83445,
+ "3410": 2.84915,
+ "3415": 2.86658,
+ "3420": 2.82691,
+ "3425": 2.81422,
+ "3430": 2.83318,
+ "3435": 2.89964,
+ "3440": 2.82094,
+ "3445": 2.87176,
+ "3450": 2.82231,
+ "3455": 2.79427,
+ "3460": 2.82113,
+ "3465": 2.85549,
+ "3470": 2.84379,
+ "3475": 2.7827,
+ "3480": 2.84358,
+ "3485": 2.82723,
+ "3490": 2.89648,
+ "3495": 2.85358,
+ "3500": 2.83932,
+ "3505": 2.82976,
+ "3510": 2.81722,
+ "3515": 2.83947,
+ "3520": 2.7798,
+ "3525": 2.8076,
+ "3530": 2.86312,
+ "3535": 2.7916,
+ "3540": 2.84405,
+ "3545": 2.81728,
+ "3550": 2.80222,
+ "3555": 2.82111,
+ "3560": 2.82831,
+ "3565": 2.83064,
+ "3570": 2.80758,
+ "3575": 2.8108,
+ "3580": 2.82354,
+ "3585": 2.83897,
+ "3590": 2.83314,
+ "3595": 2.79006,
+ "3600": 2.75763,
+ "3605": 2.79839,
+ "3610": 2.85366,
+ "3615": 2.75968,
+ "3620": 2.81153,
+ "3625": 2.89012,
+ "3630": 2.78512,
+ "3635": 2.78962,
+ "3640": 2.79249,
+ "3645": 2.77318,
+ "3650": 2.80654,
+ "3655": 2.82567,
+ "3660": 2.77181,
+ "3665": 2.78827,
+ "3670": 2.76987,
+ "3675": 2.77982,
+ "3680": 2.81722,
+ "3685": 2.807,
+ "3690": 2.81012,
+ "3695": 2.81602,
+ "3700": 2.79246,
+ "3705": 2.78977,
+ "3710": 2.75621,
+ "3715": 2.80769,
+ "3720": 2.80295,
+ "3725": 2.79689,
+ "3730": 2.84284,
+ "3735": 2.80302,
+ "3740": 2.75403,
+ "3745": 2.79871,
+ "3750": 2.80835,
+ "3755": 2.80365,
+ "3760": 2.76297,
+ "3765": 2.76114,
+ "3770": 2.76425,
+ "3775": 2.77515,
+ "3780": 2.76611,
+ "3785": 2.78273,
+ "3790": 2.74585,
+ "3795": 2.7943,
+ "3800": 2.80919,
+ "3805": 2.75606,
+ "3810": 2.80348,
+ "3815": 2.76783,
+ "3820": 2.78665,
+ "3825": 2.74368,
+ "3830": 2.74754,
+ "3835": 2.81858,
+ "3840": 2.72874,
+ "3845": 2.72322,
+ "3850": 2.78,
+ "3855": 2.72854,
+ "3860": 2.80843,
+ "3865": 2.75384,
+ "3870": 2.77843,
+ "3875": 2.75435,
+ "3880": 2.78959,
+ "3885": 2.78704,
+ "3890": 2.74772,
+ "3895": 2.80198,
+ "3900": 2.76665,
+ "3905": 2.7273,
+ "3910": 2.74855,
+ "3915": 2.75398,
+ "3920": 2.79669,
+ "3925": 2.78392,
+ "3930": 2.71057,
+ "3935": 2.74346,
+ "3940": 2.75667,
+ "3945": 2.74619,
+ "3950": 2.73229,
+ "3955": 2.78862,
+ "3960": 2.76604,
+ "3965": 2.74261,
+ "3970": 2.76202,
+ "3975": 2.72885,
+ "3980": 2.74356,
+ "3985": 2.753,
+ "3990": 2.69532,
+ "3995": 2.78674,
+ "4000": 2.73839,
+ "4005": 2.77018,
+ "4010": 2.71275,
+ "4015": 2.72469,
+ "4020": 2.75812,
+ "4025": 2.73947,
+ "4030": 2.66471,
+ "4035": 2.70312,
+ "4040": 2.7551,
+ "4045": 2.75154,
+ "4050": 2.79476,
+ "4055": 2.72564,
+ "4060": 2.71823,
+ "4065": 2.65515,
+ "4070": 2.81106,
+ "4075": 2.7593,
+ "4080": 2.7224,
+ "4085": 2.7526,
+ "4090": 2.68408,
+ "4095": 2.69465,
+ "4100": 2.71413,
+ "4105": 2.74242,
+ "4110": 2.73879,
+ "4115": 2.70781,
+ "4120": 2.72904,
+ "4125": 2.706,
+ "4130": 2.70256,
+ "4135": 2.69364,
+ "4140": 2.68685,
+ "4145": 2.78573,
+ "4150": 2.71248,
+ "4155": 2.74287,
+ "4160": 2.7636,
+ "4165": 2.72423,
+ "4170": 2.68299,
+ "4175": 2.72506,
+ "4180": 2.73013,
+ "4185": 2.73505,
+ "4190": 2.74113,
+ "4195": 2.69908,
+ "4200": 2.70931,
+ "4205": 2.74563,
+ "4210": 2.68107,
+ "4215": 2.67252,
+ "4220": 2.66454,
+ "4225": 2.70808,
+ "4230": 2.71983,
+ "4235": 2.73343,
+ "4240": 2.70888,
+ "4245": 2.69989,
+ "4250": 2.71677,
+ "4255": 2.65608,
+ "4260": 2.7308,
+ "4265": 2.74762,
+ "4270": 2.73075,
+ "4275": 2.6981,
+ "4280": 2.70824,
+ "4285": 2.73731,
+ "4290": 2.69096,
+ "4295": 2.69773,
+ "4300": 2.70521,
+ "4305": 2.70287,
+ "4310": 2.73612,
+ "4315": 2.71699,
+ "4320": 2.70088,
+ "4325": 2.70647,
+ "4330": 2.71085,
+ "4335": 2.69263,
+ "4340": 2.69943,
+ "4345": 2.72678,
+ "4350": 2.67936,
+ "4355": 2.69716,
+ "4360": 2.717,
+ "4365": 2.79214,
+ "4370": 2.73775,
+ "4375": 2.74574,
+ "4380": 2.70719,
+ "4385": 2.70003,
+ "4390": 2.70588,
+ "4395": 2.75693,
+ "4400": 2.66771,
+ "4405": 2.66938,
+ "4410": 2.6866,
+ "4415": 2.70735,
+ "4420": 2.70696,
+ "4425": 2.72098,
+ "4430": 2.69315,
+ "4435": 2.68322,
+ "4440": 2.6986,
+ "4445": 2.68318,
+ "4450": 2.65777,
+ "4455": 2.66913,
+ "4460": 2.68987,
+ "4465": 2.70288,
+ "4470": 2.67093,
+ "4475": 2.69006,
+ "4480": 2.65881,
+ "4485": 2.70227,
+ "4490": 2.65665,
+ "4495": 2.71197,
+ "4500": 2.70447,
+ "4505": 2.70028,
+ "4510": 2.65488,
+ "4515": 2.70383,
+ "4520": 2.67082,
+ "4525": 2.67161,
+ "4530": 2.67935,
+ "4535": 2.67779,
+ "4540": 2.71224,
+ "4545": 2.66111,
+ "4550": 2.70285,
+ "4555": 2.68549,
+ "4560": 2.65973,
+ "4565": 2.64283,
+ "4570": 2.64437,
+ "4575": 2.66817,
+ "4580": 2.69161,
+ "4585": 2.6866,
+ "4590": 2.61902,
+ "4595": 2.66724,
+ "4600": 2.68012,
+ "4605": 2.67797,
+ "4610": 2.65878,
+ "4615": 2.66405,
+ "4620": 2.65841,
+ "4625": 2.68759,
+ "4630": 2.67207,
+ "4635": 2.6467,
+ "4640": 2.69593,
+ "4645": 2.64943,
+ "4650": 2.70482,
+ "4655": 2.70974,
+ "4660": 2.678,
+ "4665": 2.68927,
+ "4670": 2.67818,
+ "4675": 2.69201,
+ "4680": 2.66723,
+ "4685": 2.66091,
+ "4690": 2.70694,
+ "4695": 2.65664,
+ "4700": 2.67563,
+ "4705": 2.6541,
+ "4710": 2.67809,
+ "4715": 2.65279,
+ "4720": 2.72611,
+ "4725": 2.62991,
+ "4730": 2.65465,
+ "4735": 2.68665,
+ "4740": 2.64349,
+ "4745": 2.65464,
+ "4750": 2.6425,
+ "4755": 2.65742,
+ "4760": 2.66679,
+ "4765": 2.64631,
+ "4770": 2.62442,
+ "4775": 2.65996,
+ "4780": 2.65947,
+ "4785": 2.69218,
+ "4790": 2.65164,
+ "4795": 2.67687,
+ "4800": 2.6313,
+ "4805": 2.6446,
+ "4810": 2.66621,
+ "4815": 2.64665,
+ "4820": 2.6782,
+ "4825": 2.6539,
+ "4830": 2.61864,
+ "4835": 2.65011,
+ "4840": 2.65638,
+ "4845": 2.64014,
+ "4850": 2.62687,
+ "4855": 2.60115,
+ "4860": 2.65274,
+ "4865": 2.62849,
+ "4870": 2.6455,
+ "4875": 2.62328,
+ "4880": 2.6283,
+ "4885": 2.62736,
+ "4890": 2.6794,
+ "4895": 2.65936,
+ "4900": 2.61724,
+ "4905": 2.61961,
+ "4910": 2.63978,
+ "4915": 2.61489,
+ "4920": 2.65096,
+ "4925": 2.6502,
+ "4930": 2.57167,
+ "4935": 2.65086,
+ "4940": 2.63371,
+ "4945": 2.63866,
+ "4950": 2.63093,
+ "4955": 2.61847,
+ "4960": 2.62102,
+ "4965": 2.65956,
+ "4970": 2.60049,
+ "4975": 2.65887,
+ "4980": 2.62253,
+ "4985": 2.63045,
+ "4990": 2.65896,
+ "4995": 2.58449,
+ "5000": 2.66172,
+ "5005": 2.66467,
+ "5010": 2.68321,
+ "5015": 2.63657,
+ "5020": 2.64187,
+ "5025": 2.68752,
+ "5030": 2.64287,
+ "5035": 2.61752,
+ "5040": 2.62143,
+ "5045": 2.6017,
+ "5050": 2.62827,
+ "5055": 2.6529,
+ "5060": 2.64519,
+ "5065": 2.68833,
+ "5070": 2.60442,
+ "5075": 2.61853,
+ "5080": 2.61208,
+ "5085": 2.604,
+ "5090": 2.59066,
+ "5095": 2.65156,
+ "5100": 2.6482,
+ "5105": 2.61142,
+ "5110": 2.66463,
+ "5115": 2.61773,
+ "5120": 2.6851,
+ "5125": 2.6328,
+ "5130": 2.61414,
+ "5135": 2.61072,
+ "5140": 2.57728,
+ "5145": 2.62747,
+ "5150": 2.63552,
+ "5155": 2.61666,
+ "5160": 2.66422,
+ "5165": 2.58419,
+ "5170": 2.59264,
+ "5175": 2.61755,
+ "5180": 2.60594,
+ "5185": 2.62077,
+ "5190": 2.62566,
+ "5195": 2.67043,
+ "5200": 2.60106,
+ "5205": 2.60577,
+ "5210": 2.60577,
+ "5215": 2.64683,
+ "5220": 2.58739,
+ "5225": 2.55124,
+ "5230": 2.63497,
+ "5235": 2.61953,
+ "5240": 2.63299,
+ "5245": 2.64007,
+ "5250": 2.62651,
+ "5255": 2.62996,
+ "5260": 2.56167,
+ "5265": 2.59636,
+ "5270": 2.58715,
+ "5275": 2.61558,
+ "5280": 2.6101,
+ "5285": 2.60177,
+ "5290": 2.63161,
+ "5295": 2.62124,
+ "5300": 2.57873,
+ "5305": 2.59839,
+ "5310": 2.61352,
+ "5315": 2.58879,
+ "5320": 2.61649,
+ "5325": 2.6449,
+ "5330": 2.606,
+ "5335": 2.5852,
+ "5340": 2.56323,
+ "5345": 2.65839,
+ "5350": 2.6253,
+ "5355": 2.58005,
+ "5360": 2.59537,
+ "5365": 2.62193,
+ "5370": 2.616,
+ "5375": 2.62869,
+ "5380": 2.57994,
+ "5385": 2.56513,
+ "5390": 2.58667,
+ "5395": 2.61896,
+ "5400": 2.60742,
+ "5405": 2.54859,
+ "5410": 2.61326,
+ "5415": 2.5968,
+ "5420": 2.61305,
+ "5425": 2.62522,
+ "5430": 2.62714,
+ "5435": 2.57646,
+ "5440": 2.58691,
+ "5445": 2.62928,
+ "5450": 2.64904,
+ "5455": 2.61295,
+ "5460": 2.59303,
+ "5465": 2.60946,
+ "5470": 2.60013,
+ "5475": 2.62549,
+ "5480": 2.59024,
+ "5485": 2.59298,
+ "5490": 2.57902,
+ "5495": 2.57297,
+ "5500": 2.56984,
+ "5505": 2.6179,
+ "5510": 2.62675,
+ "5515": 2.58442,
+ "5520": 2.55635,
+ "5525": 2.58644,
+ "5530": 2.66454,
+ "5535": 2.62243,
+ "5540": 2.57094,
+ "5545": 2.59629,
+ "5550": 2.55008,
+ "5555": 2.57413,
+ "5560": 2.56392,
+ "5565": 2.60633,
+ "5570": 2.65231,
+ "5575": 2.62956,
+ "5580": 2.57293,
+ "5585": 2.59512,
+ "5590": 2.56253,
+ "5595": 2.58488,
+ "5600": 2.55316,
+ "5605": 2.60174,
+ "5610": 2.58344,
+ "5615": 2.58383,
+ "5620": 2.5828,
+ "5625": 2.55159,
+ "5630": 2.57274,
+ "5635": 2.63391,
+ "5640": 2.59709,
+ "5645": 2.57394,
+ "5650": 2.58002,
+ "5655": 2.54968,
+ "5660": 2.55884,
+ "5665": 2.58772,
+ "5670": 2.56974,
+ "5675": 2.60699,
+ "5680": 2.52871,
+ "5685": 2.57074,
+ "5690": 2.60289,
+ "5695": 2.56061,
+ "5700": 2.59647,
+ "5705": 2.59767,
+ "5710": 2.57979,
+ "5715": 2.58478,
+ "5720": 2.53602,
+ "5725": 2.60163,
+ "5730": 2.57478,
+ "5735": 2.61025,
+ "5740": 2.59376,
+ "5745": 2.55786,
+ "5750": 2.54163,
+ "5755": 2.56148,
+ "5760": 2.61763,
+ "5765": 2.55839,
+ "5770": 2.54209,
+ "5775": 2.5851,
+ "5780": 2.57898,
+ "5785": 2.54052,
+ "5790": 2.56441,
+ "5795": 2.6016,
+ "5800": 2.54469,
+ "5805": 2.53479,
+ "5810": 2.55723,
+ "5815": 2.52569,
+ "5820": 2.59764,
+ "5825": 2.50389,
+ "5830": 2.49824,
+ "5835": 2.59839,
+ "5840": 2.54067,
+ "5845": 2.5546,
+ "5850": 2.61487,
+ "5855": 2.51063,
+ "5860": 2.56026,
+ "5865": 2.51866,
+ "5870": 2.57501,
+ "5875": 2.61024,
+ "5880": 2.58742,
+ "5885": 2.56735,
+ "5890": 2.58353,
+ "5895": 2.55592,
+ "5900": 2.61408,
+ "5905": 2.55554,
+ "5910": 2.59722,
+ "5915": 2.60946,
+ "5920": 2.58692,
+ "5925": 2.54513,
+ "5930": 2.58538,
+ "5935": 2.55227,
+ "5940": 2.57041,
+ "5945": 2.51784,
+ "5950": 2.55423,
+ "5955": 2.58456,
+ "5960": 2.56321,
+ "5965": 2.61861,
+ "5970": 2.54969,
+ "5975": 2.57895,
+ "5980": 2.55865,
+ "5985": 2.56052,
+ "5990": 2.55619,
+ "5995": 2.55747,
+ "6000": 2.55354,
+ "6005": 2.52322,
+ "6010": 2.56029,
+ "6015": 2.524,
+ "6020": 2.53447,
+ "6025": 2.55788,
+ "6030": 2.60493,
+ "6035": 2.54194,
+ "6040": 2.54995,
+ "6045": 2.49165,
+ "6050": 2.5951,
+ "6055": 2.51904,
+ "6060": 2.54539,
+ "6065": 2.52515,
+ "6070": 2.52835,
+ "6075": 2.53731,
+ "6080": 2.53486,
+ "6085": 2.59864,
+ "6090": 2.5708,
+ "6095": 2.53577,
+ "6100": 2.5429,
+ "6105": 2.52445,
+ "6110": 2.5555,
+ "6115": 2.58453,
+ "6120": 2.55777,
+ "6125": 2.5404,
+ "6130": 2.47406,
+ "6135": 2.55687,
+ "6140": 2.5547,
+ "6145": 2.55742,
+ "6150": 2.52523,
+ "6155": 2.50878,
+ "6160": 2.53933,
+ "6165": 2.5709,
+ "6170": 2.54328,
+ "6175": 2.60016,
+ "6180": 2.52416,
+ "6185": 2.55328,
+ "6190": 2.49156,
+ "6195": 2.57673,
+ "6200": 2.55024,
+ "6205": 2.53605,
+ "6210": 2.51872,
+ "6215": 2.5131,
+ "6220": 2.5647,
+ "6225": 2.51278,
+ "6230": 2.51147,
+ "6235": 2.56204,
+ "6240": 2.55176,
+ "6245": 2.52293,
+ "6250": 2.53194,
+ "6255": 2.57179,
+ "6260": 2.52397,
+ "6265": 2.57464,
+ "6270": 2.5253,
+ "6275": 2.56546,
+ "6280": 2.52265,
+ "6285": 2.52082,
+ "6290": 2.52178,
+ "6295": 2.50489,
+ "6300": 2.55651,
+ "6305": 2.5255,
+ "6310": 2.51251,
+ "6315": 2.53866,
+ "6320": 2.48965,
+ "6325": 2.59676,
+ "6330": 2.5548,
+ "6335": 2.51247,
+ "6340": 2.51081,
+ "6345": 2.55459,
+ "6350": 2.55485,
+ "6355": 2.5242,
+ "6360": 2.52125,
+ "6365": 2.48319,
+ "6370": 2.53517,
+ "6375": 2.49198,
+ "6380": 2.55811,
+ "6385": 2.57549,
+ "6390": 2.50397,
+ "6395": 2.55156,
+ "6400": 2.50904,
+ "6405": 2.52939,
+ "6410": 2.52069,
+ "6415": 2.52703,
+ "6420": 2.54256,
+ "6425": 2.5347,
+ "6430": 2.57884,
+ "6435": 2.54441,
+ "6440": 2.5365,
+ "6445": 2.52925,
+ "6450": 2.53175,
+ "6455": 2.52274,
+ "6460": 2.51748,
+ "6465": 2.5608,
+ "6470": 2.51903,
+ "6475": 2.52435,
+ "6480": 2.48637,
+ "6485": 2.52979,
+ "6490": 2.50961,
+ "6495": 2.49958,
+ "6500": 2.52332,
+ "6505": 2.49435,
+ "6510": 2.5431,
+ "6515": 2.51011,
+ "6520": 2.50992,
+ "6525": 2.49432,
+ "6530": 2.54454,
+ "6535": 2.53285,
+ "6540": 2.53196,
+ "6545": 2.56057,
+ "6550": 2.50472,
+ "6555": 2.55543,
+ "6560": 2.51041,
+ "6565": 2.52139,
+ "6570": 2.58448,
+ "6575": 2.52272,
+ "6580": 2.50045,
+ "6585": 2.50883,
+ "6590": 2.51066,
+ "6595": 2.49931,
+ "6600": 2.48752,
+ "6605": 2.53809,
+ "6610": 2.47784,
+ "6615": 2.56592,
+ "6620": 2.53317,
+ "6625": 2.51215,
+ "6630": 2.51366,
+ "6635": 2.47267,
+ "6640": 2.53835,
+ "6645": 2.59583,
+ "6650": 2.50952,
+ "6655": 2.49788,
+ "6660": 2.57415,
+ "6665": 2.51974,
+ "6670": 2.56822,
+ "6675": 2.46804,
+ "6680": 2.54779,
+ "6685": 2.53524,
+ "6690": 2.51328,
+ "6695": 2.48745,
+ "6700": 2.52449,
+ "6705": 2.51812,
+ "6710": 2.49315,
+ "6715": 2.51956,
+ "6720": 2.51086,
+ "6725": 2.52243,
+ "6730": 2.51941,
+ "6735": 2.48358,
+ "6740": 2.51226,
+ "6745": 2.4971,
+ "6750": 2.55699,
+ "6755": 2.47508,
+ "6760": 2.54106,
+ "6765": 2.48797,
+ "6770": 2.51814,
+ "6775": 2.50699,
+ "6780": 2.53925,
+ "6785": 2.47179,
+ "6790": 2.54293,
+ "6795": 2.499,
+ "6800": 2.52666,
+ "6805": 2.51089,
+ "6810": 2.50481,
+ "6815": 2.52269,
+ "6820": 2.48597,
+ "6825": 2.50306,
+ "6830": 2.54052,
+ "6835": 2.50686,
+ "6840": 2.51012,
+ "6845": 2.5258,
+ "6850": 2.47578,
+ "6855": 2.51497,
+ "6860": 2.50403,
+ "6865": 2.49004,
+ "6870": 2.55532,
+ "6875": 2.4756,
+ "6880": 2.55005,
+ "6885": 2.47667,
+ "6890": 2.54499,
+ "6895": 2.50058,
+ "6900": 2.48911,
+ "6905": 2.49914,
+ "6910": 2.51865,
+ "6915": 2.51876,
+ "6920": 2.53284,
+ "6925": 2.54186,
+ "6930": 2.49003,
+ "6935": 2.51954,
+ "6940": 2.50123,
+ "6945": 2.46259,
+ "6950": 2.48298,
+ "6955": 2.52635,
+ "6960": 2.51943,
+ "6965": 2.49462,
+ "6970": 2.47092,
+ "6975": 2.52187,
+ "6980": 2.45145,
+ "6985": 2.51525,
+ "6990": 2.53123,
+ "6995": 2.46232,
+ "7000": 2.48871,
+ "7005": 2.47014,
+ "7010": 2.47431,
+ "7015": 2.51846,
+ "7020": 2.46743,
+ "7025": 2.45259,
+ "7030": 2.48539,
+ "7035": 2.47883,
+ "7040": 2.50494,
+ "7045": 2.51969,
+ "7050": 2.52672,
+ "7055": 2.44446,
+ "7060": 2.47361,
+ "7065": 2.48553,
+ "7070": 2.49028,
+ "7075": 2.49411,
+ "7080": 2.53854,
+ "7085": 2.48557,
+ "7090": 2.48272,
+ "7095": 2.50214,
+ "7100": 2.51549,
+ "7105": 2.48863,
+ "7110": 2.48625,
+ "7115": 2.5057,
+ "7120": 2.47446,
+ "7125": 2.46362,
+ "7130": 2.48515,
+ "7135": 2.51314,
+ "7140": 2.49783,
+ "7145": 2.49951,
+ "7150": 2.50942,
+ "7155": 2.50244,
+ "7160": 2.47497,
+ "7165": 2.45734,
+ "7170": 2.50625,
+ "7175": 2.50449,
+ "7180": 2.50343,
+ "7185": 2.4821,
+ "7190": 2.46138,
+ "7195": 2.46516,
+ "7200": 2.50961,
+ "7205": 2.48844,
+ "7210": 2.44288,
+ "7215": 2.48064,
+ "7220": 2.4432,
+ "7225": 2.51318,
+ "7230": 2.50885,
+ "7235": 2.48565,
+ "7240": 2.47827,
+ "7245": 2.49869,
+ "7250": 2.50766,
+ "7255": 2.49096,
+ "7260": 2.45665,
+ "7265": 2.45398,
+ "7270": 2.471,
+ "7275": 2.50056,
+ "7280": 2.49436,
+ "7285": 2.42318,
+ "7290": 2.48025,
+ "7295": 2.48765,
+ "7300": 2.41781,
+ "7305": 2.44566,
+ "7310": 2.44742,
+ "7315": 2.49088,
+ "7320": 2.48414,
+ "7325": 2.45913,
+ "7330": 2.48837,
+ "7335": 2.47377,
+ "7340": 2.46217,
+ "7345": 2.49547,
+ "7350": 2.50972,
+ "7355": 2.49521,
+ "7360": 2.47975,
+ "7365": 2.46757,
+ "7370": 2.47133,
+ "7375": 2.44779,
+ "7380": 2.49143,
+ "7385": 2.48328,
+ "7390": 2.47244,
+ "7395": 2.47201,
+ "7400": 2.47965,
+ "7405": 2.44114,
+ "7410": 2.4806,
+ "7415": 2.46985,
+ "7420": 2.4921,
+ "7425": 2.45657,
+ "7430": 2.5236,
+ "7435": 2.49195,
+ "7440": 2.51827,
+ "7445": 2.50779,
+ "7450": 2.47245,
+ "7455": 2.45529,
+ "7460": 2.46275,
+ "7465": 2.47587,
+ "7470": 2.45102,
+ "7475": 2.45675,
+ "7480": 2.51057,
+ "7485": 2.44969,
+ "7490": 2.47412,
+ "7495": 2.48024,
+ "7500": 2.49436,
+ "7505": 2.43994,
+ "7510": 2.43541,
+ "7515": 2.41913,
+ "7520": 2.49605,
+ "7525": 2.49704,
+ "7530": 2.47612,
+ "7535": 2.46046,
+ "7540": 2.47199,
+ "7545": 2.47311,
+ "7550": 2.4891,
+ "7555": 2.45305,
+ "7560": 2.42768,
+ "7565": 2.50895,
+ "7570": 2.48445,
+ "7575": 2.43768,
+ "7580": 2.45675,
+ "7585": 2.48217,
+ "7590": 2.48097,
+ "7595": 2.46221,
+ "7600": 2.46178,
+ "7605": 2.44643,
+ "7610": 2.45041,
+ "7615": 2.42542,
+ "7620": 2.54338,
+ "7625": 2.48067,
+ "7630": 2.42332,
+ "7635": 2.42574,
+ "7640": 2.45345,
+ "7645": 2.47356,
+ "7650": 2.46236,
+ "7655": 2.48328,
+ "7660": 2.45205,
+ "7665": 2.43265,
+ "7670": 2.43945,
+ "7675": 2.45593,
+ "7680": 2.48501,
+ "7685": 2.43122,
+ "7690": 2.47924,
+ "7695": 2.45353,
+ "7700": 2.48295,
+ "7705": 2.49849,
+ "7710": 2.49353,
+ "7715": 2.4439,
+ "7720": 2.46796,
+ "7725": 2.47949,
+ "7730": 2.45614,
+ "7735": 2.46957,
+ "7740": 2.43649,
+ "7745": 2.44912,
+ "7750": 2.43696,
+ "7755": 2.46558,
+ "7760": 2.4507,
+ "7765": 2.45439,
+ "7770": 2.46929,
+ "7775": 2.45158,
+ "7780": 2.41614,
+ "7785": 2.44384,
+ "7790": 2.48079,
+ "7795": 2.4398,
+ "7800": 2.46271,
+ "7805": 2.48027,
+ "7810": 2.50129,
+ "7815": 2.48639,
+ "7820": 2.44516,
+ "7825": 2.51324,
+ "7830": 2.45153,
+ "7835": 2.46659,
+ "7840": 2.47993,
+ "7845": 2.45911,
+ "7850": 2.41588,
+ "7855": 2.4717,
+ "7860": 2.49983,
+ "7865": 2.42282,
+ "7870": 2.46628,
+ "7875": 2.44702,
+ "7880": 2.45404,
+ "7885": 2.46099,
+ "7890": 2.47047,
+ "7895": 2.44714,
+ "7900": 2.43952,
+ "7905": 2.43594,
+ "7910": 2.42458,
+ "7915": 2.48098,
+ "7920": 2.47635,
+ "7925": 2.42092,
+ "7930": 2.47004,
+ "7935": 2.44984,
+ "7940": 2.4209,
+ "7945": 2.4679,
+ "7950": 2.44269,
+ "7955": 2.41681,
+ "7960": 2.48704,
+ "7965": 2.5161,
+ "7970": 2.52106,
+ "7975": 2.44905,
+ "7980": 2.44048,
+ "7985": 2.46706,
+ "7990": 2.4318,
+ "7995": 2.46886,
+ "8000": 2.43524,
+ "8005": 2.41715,
+ "8010": 2.45855,
+ "8015": 2.47016,
+ "8020": 2.4817,
+ "8025": 2.47442,
+ "8030": 2.45168,
+ "8035": 2.46987,
+ "8040": 2.41869,
+ "8045": 2.4532,
+ "8050": 2.44608,
+ "8055": 2.42567,
+ "8060": 2.44377,
+ "8065": 2.46148,
+ "8070": 2.45678,
+ "8075": 2.45758,
+ "8080": 2.44368,
+ "8085": 2.44074,
+ "8090": 2.425,
+ "8095": 2.42411,
+ "8100": 2.43875,
+ "8105": 2.49528,
+ "8110": 2.43881,
+ "8115": 2.4442,
+ "8120": 2.46682,
+ "8125": 2.468,
+ "8130": 2.45181,
+ "8135": 2.45124,
+ "8140": 2.43943,
+ "8145": 2.4251,
+ "8150": 2.42115,
+ "8155": 2.48618,
+ "8160": 2.45302,
+ "8165": 2.44147,
+ "8170": 2.43445,
+ "8175": 2.42116,
+ "8180": 2.49462,
+ "8185": 2.42506,
+ "8190": 2.46774,
+ "8195": 2.4559,
+ "8200": 2.44665,
+ "8205": 2.44464,
+ "8210": 2.43127,
+ "8215": 2.43962,
+ "8220": 2.43461,
+ "8225": 2.40807,
+ "8230": 2.43921,
+ "8235": 2.46438,
+ "8240": 2.42629,
+ "8245": 2.44606,
+ "8250": 2.44335,
+ "8255": 2.43703,
+ "8260": 2.43286,
+ "8265": 2.43363,
+ "8270": 2.43733,
+ "8275": 2.44725,
+ "8280": 2.40445,
+ "8285": 2.44626,
+ "8290": 2.48807,
+ "8295": 2.4556,
+ "8300": 2.46545,
+ "8305": 2.41882,
+ "8310": 2.44521,
+ "8315": 2.4652,
+ "8320": 2.40775,
+ "8325": 2.40185,
+ "8330": 2.44465,
+ "8335": 2.454,
+ "8340": 2.4989,
+ "8345": 2.45642,
+ "8350": 2.45938,
+ "8355": 2.41812,
+ "8360": 2.41252,
+ "8365": 2.46233,
+ "8370": 2.45956,
+ "8375": 2.43404,
+ "8380": 2.42861,
+ "8385": 2.43299,
+ "8390": 2.4462,
+ "8395": 2.44992,
+ "8400": 2.44991,
+ "8405": 2.49879,
+ "8410": 2.44773,
+ "8415": 2.44365,
+ "8420": 2.42653,
+ "8425": 2.44808,
+ "8430": 2.47166,
+ "8435": 2.4148,
+ "8440": 2.46205,
+ "8445": 2.4719,
+ "8450": 2.41712,
+ "8455": 2.47322,
+ "8460": 2.46703,
+ "8465": 2.45532,
+ "8470": 2.44631,
+ "8475": 2.50009,
+ "8480": 2.41626,
+ "8485": 2.42485,
+ "8490": 2.47607,
+ "8495": 2.44617,
+ "8500": 2.45344,
+ "8505": 2.41523,
+ "8510": 2.41142,
+ "8515": 2.43755,
+ "8520": 2.43349,
+ "8525": 2.50425,
+ "8530": 2.38186,
+ "8535": 2.41091,
+ "8540": 2.49491,
+ "8545": 2.39141,
+ "8550": 2.44924,
+ "8555": 2.46223,
+ "8560": 2.48222,
+ "8565": 2.43498,
+ "8570": 2.44426,
+ "8575": 2.45864,
+ "8580": 2.45188,
+ "8585": 2.43229,
+ "8590": 2.41478,
+ "8595": 2.43822,
+ "8600": 2.42252,
+ "8605": 2.50219,
+ "8610": 2.43046,
+ "8615": 2.39848,
+ "8620": 2.46033,
+ "8625": 2.43683,
+ "8630": 2.46564,
+ "8635": 2.46509,
+ "8640": 2.44573,
+ "8645": 2.48476,
+ "8650": 2.43317,
+ "8655": 2.46437,
+ "8660": 2.46688,
+ "8665": 2.39792,
+ "8670": 2.4204,
+ "8675": 2.44134,
+ "8680": 2.45917,
+ "8685": 2.44287,
+ "8690": 2.42337,
+ "8695": 2.4576,
+ "8700": 2.44685,
+ "8705": 2.43517,
+ "8710": 2.43956,
+ "8715": 2.45926,
+ "8720": 2.48768,
+ "8725": 2.42018,
+ "8730": 2.40411,
+ "8735": 2.4458,
+ "8740": 2.44124,
+ "8745": 2.40867,
+ "8750": 2.44728,
+ "8755": 2.43508,
+ "8760": 2.41316,
+ "8765": 2.44751,
+ "8770": 2.41594,
+ "8775": 2.45002,
+ "8780": 2.43082,
+ "8785": 2.4826,
+ "8790": 2.43308,
+ "8795": 2.43349,
+ "8800": 2.4321,
+ "8805": 2.41894,
+ "8810": 2.42481,
+ "8815": 2.487,
+ "8820": 2.46651,
+ "8825": 2.4393,
+ "8830": 2.39711,
+ "8835": 2.43377,
+ "8840": 2.40495,
+ "8845": 2.44022,
+ "8850": 2.44733,
+ "8855": 2.41504,
+ "8860": 2.44057,
+ "8865": 2.43912,
+ "8870": 2.44762,
+ "8875": 2.45022,
+ "8880": 2.42391,
+ "8885": 2.40582,
+ "8890": 2.45757,
+ "8895": 2.43989,
+ "8900": 2.42516,
+ "8905": 2.41312,
+ "8910": 2.41171,
+ "8915": 2.43051,
+ "8920": 2.44575,
+ "8925": 2.47902,
+ "8930": 2.42743,
+ "8935": 2.42201,
+ "8940": 2.40161,
+ "8945": 2.40426,
+ "8950": 2.42938,
+ "8955": 2.40734,
+ "8960": 2.44555,
+ "8965": 2.42863,
+ "8970": 2.41756,
+ "8975": 2.49059,
+ "8980": 2.45369,
+ "8985": 2.3862,
+ "8990": 2.42051,
+ "8995": 2.42916,
+ "9000": 2.46813,
+ "9005": 2.42341,
+ "9010": 2.3862,
+ "9015": 2.41938,
+ "9020": 2.41034,
+ "9025": 2.38041,
+ "9030": 2.41146,
+ "9035": 2.43664,
+ "9040": 2.43479,
+ "9045": 2.43227,
+ "9050": 2.40905,
+ "9055": 2.43064,
+ "9060": 2.43608,
+ "9065": 2.41987,
+ "9070": 2.45671,
+ "9075": 2.40626,
+ "9080": 2.45053,
+ "9085": 2.4361,
+ "9090": 2.42635,
+ "9095": 2.41044,
+ "9100": 2.41349,
+ "9105": 2.37061,
+ "9110": 2.47783,
+ "9115": 2.42768,
+ "9120": 2.4153,
+ "9125": 2.47081,
+ "9130": 2.40666,
+ "9135": 2.45957,
+ "9140": 2.44595,
+ "9145": 2.43789,
+ "9150": 2.43689,
+ "9155": 2.38451,
+ "9160": 2.42955,
+ "9165": 2.43724,
+ "9170": 2.3856,
+ "9175": 2.43,
+ "9180": 2.38849,
+ "9185": 2.44997,
+ "9190": 2.42357,
+ "9195": 2.4078,
+ "9200": 2.40419,
+ "9205": 2.46075,
+ "9210": 2.37469,
+ "9215": 2.47828,
+ "9220": 2.46007,
+ "9225": 2.39401,
+ "9230": 2.45725,
+ "9235": 2.40825,
+ "9240": 2.41321,
+ "9245": 2.44871,
+ "9250": 2.43983,
+ "9255": 2.44156,
+ "9260": 2.39854,
+ "9265": 2.44978,
+ "9270": 2.44664,
+ "9275": 2.40469,
+ "9280": 2.39981,
+ "9285": 2.43354,
+ "9290": 2.41628,
+ "9295": 2.39626,
+ "9300": 2.43389,
+ "9305": 2.41395,
+ "9310": 2.42504,
+ "9315": 2.41992,
+ "9320": 2.45402,
+ "9325": 2.38111,
+ "9330": 2.41516,
+ "9335": 2.37044,
+ "9340": 2.4179,
+ "9345": 2.42705,
+ "9350": 2.45019,
+ "9355": 2.48641,
+ "9360": 2.44802,
+ "9365": 2.39905,
+ "9370": 2.44623,
+ "9375": 2.44502,
+ "9380": 2.36429,
+ "9385": 2.41094,
+ "9390": 2.39295,
+ "9395": 2.398,
+ "9400": 2.45369,
+ "9405": 2.42353,
+ "9410": 2.40802,
+ "9415": 2.44816,
+ "9420": 2.45482,
+ "9425": 2.44258,
+ "9430": 2.45777,
+ "9435": 2.42475,
+ "9440": 2.48693,
+ "9445": 2.38593,
+ "9450": 2.4031,
+ "9455": 2.41359,
+ "9460": 2.39628,
+ "9465": 2.38987,
+ "9470": 2.39178,
+ "9475": 2.37518,
+ "9480": 2.44372,
+ "9485": 2.39835,
+ "9490": 2.43096,
+ "9495": 2.39337,
+ "9500": 2.37474,
+ "9505": 2.43992,
+ "9510": 2.40976,
+ "9515": 2.44038,
+ "9520": 2.42911,
+ "9525": 2.39978,
+ "9530": 2.46462,
+ "9535": 2.41098,
+ "9540": 2.42979,
+ "9545": 2.38866,
+ "9550": 2.43326,
+ "9555": 2.40096,
+ "9560": 2.43313,
+ "9565": 2.41693,
+ "9570": 2.38286,
+ "9575": 2.4218,
+ "9580": 2.40624,
+ "9585": 2.43284,
+ "9590": 2.4387,
+ "9595": 2.45826,
+ "9600": 2.40192,
+ "9605": 2.39415,
+ "9610": 2.43148,
+ "9615": 2.42381,
+ "9620": 2.42343,
+ "9625": 2.45384,
+ "9630": 2.40573,
+ "9635": 2.4104,
+ "9640": 2.45515,
+ "9645": 2.41908,
+ "9650": 2.40681,
+ "9655": 2.38176,
+ "9660": 2.43323,
+ "9665": 2.39773,
+ "9670": 2.39086,
+ "9675": 2.36759,
+ "9680": 2.40689,
+ "9685": 2.40759,
+ "9690": 2.47137,
+ "9695": 2.39035,
+ "9700": 2.38642,
+ "9705": 2.39295,
+ "9710": 2.37507,
+ "9715": 2.39687,
+ "9720": 2.44489,
+ "9725": 2.45193,
+ "9730": 2.44046,
+ "9735": 2.39598,
+ "9740": 2.39167,
+ "9745": 2.43511,
+ "9750": 2.40675,
+ "9755": 2.41637,
+ "9760": 2.42082,
+ "9765": 2.37763,
+ "9770": 2.4769,
+ "9775": 2.40938,
+ "9780": 2.37101,
+ "9785": 2.41029,
+ "9790": 2.41665,
+ "9795": 2.36761,
+ "9800": 2.40508,
+ "9805": 2.41261,
+ "9810": 2.41612,
+ "9815": 2.38943,
+ "9820": 2.38493,
+ "9825": 2.41288,
+ "9830": 2.42952,
+ "9835": 2.39301,
+ "9840": 2.42326,
+ "9845": 2.37134,
+ "9850": 2.40705,
+ "9855": 2.40435,
+ "9860": 2.39835,
+ "9865": 2.39051,
+ "9870": 2.39483,
+ "9875": 2.38932,
+ "9880": 2.45964,
+ "9885": 2.40149,
+ "9890": 2.3625,
+ "9895": 2.32983,
+ "9900": 2.40524,
+ "9905": 2.44036,
+ "9910": 2.36752,
+ "9915": 2.37375,
+ "9920": 2.42036,
+ "9925": 2.40752,
+ "9930": 2.39138,
+ "9935": 2.35829,
+ "9940": 2.39175,
+ "9945": 2.38731,
+ "9950": 2.41161,
+ "9955": 2.4562,
+ "9960": 2.44029,
+ "9965": 2.36548,
+ "9970": 2.41841,
+ "9975": 2.39346,
+ "9980": 2.34202,
+ "9985": 2.41527,
+ "9990": 2.40476,
+ "9995": 2.40551,
+ "10000": 2.3779,
+ "10005": 2.38222,
+ "10010": 2.39155,
+ "10015": 2.45278,
+ "10020": 2.37287,
+ "10025": 2.39614,
+ "10030": 2.39671,
+ "10035": 2.41747,
+ "10040": 2.41168,
+ "10045": 2.391,
+ "10050": 2.35587,
+ "10055": 2.37631,
+ "10060": 2.42816,
+ "10065": 2.38202,
+ "10070": 2.43219,
+ "10075": 2.38058,
+ "10080": 2.37107,
+ "10085": 2.37994,
+ "10090": 2.35547,
+ "10095": 2.41061,
+ "10100": 2.3218,
+ "10105": 2.39037,
+ "10110": 2.41856,
+ "10115": 2.39594,
+ "10120": 2.36629,
+ "10125": 2.37945,
+ "10130": 2.36734,
+ "10135": 2.39077,
+ "10140": 2.42128,
+ "10145": 2.41628,
+ "10150": 2.38441,
+ "10155": 2.40315,
+ "10160": 2.3701,
+ "10165": 2.39226,
+ "10170": 2.43082,
+ "10175": 2.33481,
+ "10180": 2.40418,
+ "10185": 2.39205,
+ "10190": 2.45277,
+ "10195": 2.41277,
+ "10200": 2.39905,
+ "10205": 2.39664,
+ "10210": 2.37582,
+ "10215": 2.35082,
+ "10220": 2.42833,
+ "10225": 2.438,
+ "10230": 2.36343,
+ "10235": 2.39441,
+ "10240": 2.38042,
+ "10245": 2.39776,
+ "10250": 2.39615,
+ "10255": 2.42117,
+ "10260": 2.34215,
+ "10265": 2.35597,
+ "10270": 2.35839,
+ "10275": 2.38034,
+ "10280": 2.45804,
+ "10285": 2.36609,
+ "10290": 2.39509,
+ "10295": 2.38161,
+ "10300": 2.376,
+ "10305": 2.42265,
+ "10310": 2.39724,
+ "10315": 2.36745,
+ "10320": 2.37409,
+ "10325": 2.36891,
+ "10330": 2.41989,
+ "10335": 2.36912,
+ "10340": 2.42622,
+ "10345": 2.37725,
+ "10350": 2.36505,
+ "10355": 2.40497,
+ "10360": 2.38079,
+ "10365": 2.36896,
+ "10370": 2.35332,
+ "10375": 2.36773,
+ "10380": 2.42902,
+ "10385": 2.41607,
+ "10390": 2.38923,
+ "10395": 2.36549,
+ "10400": 2.38611,
+ "10405": 2.36126,
+ "10410": 2.35109,
+ "10415": 2.42503,
+ "10420": 2.3871,
+ "10425": 2.33401,
+ "10430": 2.36713,
+ "10435": 2.37802,
+ "10440": 2.37871,
+ "10445": 2.36671,
+ "10450": 2.36821,
+ "10455": 2.38669,
+ "10460": 2.38756,
+ "10465": 2.30956,
+ "10470": 2.36511,
+ "10475": 2.3866,
+ "10480": 2.37089,
+ "10485": 2.36836,
+ "10490": 2.42144,
+ "10495": 2.37354,
+ "10500": 2.37199,
+ "10505": 2.37777,
+ "10510": 2.39,
+ "10515": 2.37919,
+ "10520": 2.41012,
+ "10525": 2.39687,
+ "10530": 2.39908,
+ "10535": 2.36141,
+ "10540": 2.41268,
+ "10545": 2.36489,
+ "10550": 2.38305,
+ "10555": 2.36463,
+ "10560": 2.34744,
+ "10565": 2.38069,
+ "10570": 2.38206,
+ "10575": 2.36644,
+ "10580": 2.38522,
+ "10585": 2.37539,
+ "10590": 2.38616,
+ "10595": 2.38394,
+ "10600": 2.33869,
+ "10605": 2.37852,
+ "10610": 2.37214,
+ "10615": 2.37005,
+ "10620": 2.35367,
+ "10625": 2.42649,
+ "10630": 2.37613,
+ "10635": 2.3312,
+ "10640": 2.37069,
+ "10645": 2.42878,
+ "10650": 2.36823,
+ "10655": 2.31441,
+ "10660": 2.35337,
+ "10665": 2.4063,
+ "10670": 2.32304,
+ "10675": 2.42238,
+ "10680": 2.36118,
+ "10685": 2.29504,
+ "10690": 2.39175,
+ "10695": 2.33674,
+ "10700": 2.39022,
+ "10705": 2.39174,
+ "10710": 2.34831,
+ "10715": 2.38812,
+ "10720": 2.33134,
+ "10725": 2.35906,
+ "10730": 2.35495,
+ "10735": 2.36061,
+ "10740": 2.32528,
+ "10745": 2.34563,
+ "10750": 2.34057,
+ "10755": 2.41066,
+ "10760": 2.37111,
+ "10765": 2.34177,
+ "10770": 2.3764,
+ "10775": 2.39158,
+ "10780": 2.37592,
+ "10785": 2.40194,
+ "10790": 2.35381,
+ "10795": 2.39226,
+ "10800": 2.32822,
+ "10805": 2.4021,
+ "10810": 2.38039,
+ "10815": 2.35987,
+ "10820": 2.34926,
+ "10825": 2.37741,
+ "10830": 2.34371,
+ "10835": 2.35353,
+ "10840": 2.33485,
+ "10845": 2.39224,
+ "10850": 2.33793,
+ "10855": 2.36948,
+ "10860": 2.33964,
+ "10865": 2.32843,
+ "10870": 2.33078,
+ "10875": 2.30989,
+ "10880": 2.40035,
+ "10885": 2.41176,
+ "10890": 2.36855,
+ "10895": 2.37788,
+ "10900": 2.3384,
+ "10905": 2.31887,
+ "10910": 2.41223,
+ "10915": 2.37677,
+ "10920": 2.38028,
+ "10925": 2.36814,
+ "10930": 2.32494,
+ "10935": 2.36648,
+ "10940": 2.35886,
+ "10945": 2.35128,
+ "10950": 2.36889,
+ "10955": 2.36934,
+ "10960": 2.31708,
+ "10965": 2.36916,
+ "10970": 2.36263,
+ "10975": 2.41456,
+ "10980": 2.38086,
+ "10985": 2.34865,
+ "10990": 2.4034,
+ "10995": 2.37075,
+ "11000": 2.34255,
+ "11005": 2.36778,
+ "11010": 2.34924,
+ "11015": 2.33315,
+ "11020": 2.33903,
+ "11025": 2.37143,
+ "11030": 2.34476,
+ "11035": 2.31967,
+ "11040": 2.32289,
+ "11045": 2.32388,
+ "11050": 2.32252,
+ "11055": 2.29585,
+ "11060": 2.34538,
+ "11065": 2.31536,
+ "11070": 2.40051,
+ "11075": 2.32509,
+ "11080": 2.36017,
+ "11085": 2.34352,
+ "11090": 2.35209,
+ "11095": 2.37641,
+ "11100": 2.33449,
+ "11105": 2.32154,
+ "11110": 2.368,
+ "11115": 2.37793,
+ "11120": 2.38619,
+ "11125": 2.32229,
+ "11130": 2.35716,
+ "11135": 2.33918,
+ "11140": 2.37855,
+ "11145": 2.35579,
+ "11150": 2.40131,
+ "11155": 2.34705,
+ "11160": 2.37239,
+ "11165": 2.36885,
+ "11170": 2.3465,
+ "11175": 2.34021,
+ "11180": 2.37972,
+ "11185": 2.31673,
+ "11190": 2.28317,
+ "11195": 2.33373,
+ "11200": 2.35134,
+ "11205": 2.36714,
+ "11210": 2.33799,
+ "11215": 2.32437,
+ "11220": 2.3481,
+ "11225": 2.37627,
+ "11230": 2.37091,
+ "11235": 2.32348,
+ "11240": 2.3466,
+ "11245": 2.36177,
+ "11250": 2.33705,
+ "11255": 2.34193,
+ "11260": 2.36182,
+ "11265": 2.39408,
+ "11270": 2.29316,
+ "11275": 2.31872,
+ "11280": 2.37433,
+ "11285": 2.29595,
+ "11290": 2.35146,
+ "11295": 2.36999,
+ "11300": 2.38682,
+ "11305": 2.34013,
+ "11310": 2.3354,
+ "11315": 2.30317,
+ "11320": 2.30974,
+ "11325": 2.3208,
+ "11330": 2.35828,
+ "11335": 2.34307,
+ "11340": 2.3129,
+ "11345": 2.31769,
+ "11350": 2.30069,
+ "11355": 2.3251,
+ "11360": 2.35679,
+ "11365": 2.29948,
+ "11370": 2.35751,
+ "11375": 2.33197,
+ "11380": 2.34491,
+ "11385": 2.35247,
+ "11390": 2.33803,
+ "11395": 2.29059,
+ "11400": 2.31536,
+ "11405": 2.3598,
+ "11410": 2.35964,
+ "11415": 2.39106,
+ "11420": 2.35695,
+ "11425": 2.31353,
+ "11430": 2.3736,
+ "11435": 2.3667,
+ "11440": 2.35263,
+ "11445": 2.36807,
+ "11450": 2.32665,
+ "11455": 2.30943,
+ "11460": 2.35563,
+ "11465": 2.34819,
+ "11470": 2.38002,
+ "11475": 2.319,
+ "11480": 2.33041,
+ "11485": 2.31503,
+ "11490": 2.35057,
+ "11495": 2.41059,
+ "11500": 2.34525,
+ "11505": 2.35499,
+ "11510": 2.36763,
+ "11515": 2.32598,
+ "11520": 2.30859,
+ "11525": 2.36509,
+ "11530": 2.31744,
+ "11535": 2.32663,
+ "11540": 2.35006,
+ "11545": 2.34784,
+ "11550": 2.36943,
+ "11555": 2.33007,
+ "11560": 2.35288,
+ "11565": 2.34403,
+ "11570": 2.35346,
+ "11575": 2.30052,
+ "11580": 2.33268,
+ "11585": 2.35658,
+ "11590": 2.36633,
+ "11595": 2.33968,
+ "11600": 2.36249,
+ "11605": 2.32611,
+ "11610": 2.3654,
+ "11615": 2.36389,
+ "11620": 2.30009,
+ "11625": 2.28266,
+ "11630": 2.33544,
+ "11635": 2.34875,
+ "11640": 2.30973,
+ "11645": 2.31232,
+ "11650": 2.33028,
+ "11655": 2.35572,
+ "11660": 2.34011,
+ "11665": 2.33427,
+ "11670": 2.3041,
+ "11675": 2.30115,
+ "11680": 2.32859,
+ "11685": 2.34041,
+ "11690": 2.34791,
+ "11695": 2.32158,
+ "11700": 2.32899,
+ "11705": 2.30663,
+ "11710": 2.34874,
+ "11715": 2.31887,
+ "11720": 2.30331,
+ "11725": 2.34343,
+ "11730": 2.30734,
+ "11735": 2.33175,
+ "11740": 2.27598,
+ "11745": 2.32117,
+ "11750": 2.33154,
+ "11755": 2.35613,
+ "11760": 2.31593,
+ "11765": 2.34029,
+ "11770": 2.28007,
+ "11775": 2.32874,
+ "11780": 2.25882,
+ "11785": 2.29998,
+ "11790": 2.31839,
+ "11795": 2.32436,
+ "11800": 2.33862,
+ "11805": 2.30782,
+ "11810": 2.30838,
+ "11815": 2.33529,
+ "11820": 2.32509,
+ "11825": 2.36473,
+ "11830": 2.32136,
+ "11835": 2.34203,
+ "11840": 2.34565,
+ "11845": 2.32162,
+ "11850": 2.30761,
+ "11855": 2.31613,
+ "11860": 2.34632,
+ "11865": 2.36175,
+ "11870": 2.3825,
+ "11875": 2.28407,
+ "11880": 2.29477,
+ "11885": 2.34011,
+ "11890": 2.29485,
+ "11895": 2.29405,
+ "11900": 2.33591,
+ "11905": 2.32263,
+ "11910": 2.28085,
+ "11915": 2.31397,
+ "11920": 2.33712,
+ "11925": 2.30656,
+ "11930": 2.31046,
+ "11935": 2.3206,
+ "11940": 2.32098,
+ "11945": 2.34487,
+ "11950": 2.3033,
+ "11955": 2.31832,
+ "11960": 2.34126,
+ "11965": 2.29979,
+ "11970": 2.28637,
+ "11975": 2.33975,
+ "11980": 2.31076,
+ "11985": 2.28178,
+ "11990": 2.30755,
+ "11995": 2.33395,
+ "12000": 2.32869,
+ "12005": 2.32829,
+ "12010": 2.29237,
+ "12015": 2.31401,
+ "12020": 2.33204,
+ "12025": 2.33978,
+ "12030": 2.31521,
+ "12035": 2.33999,
+ "12040": 2.31862,
+ "12045": 2.31428,
+ "12050": 2.31132,
+ "12055": 2.33564,
+ "12060": 2.30019,
+ "12065": 2.33242,
+ "12070": 2.30696,
+ "12075": 2.28019,
+ "12080": 2.35436,
+ "12085": 2.34253,
+ "12090": 2.33556,
+ "12095": 2.2845,
+ "12100": 2.31882,
+ "12105": 2.31267,
+ "12110": 2.3337,
+ "12115": 2.30828,
+ "12120": 2.30846,
+ "12125": 2.29721,
+ "12130": 2.30882,
+ "12135": 2.33247,
+ "12140": 2.29946,
+ "12145": 2.25894,
+ "12150": 2.2623,
+ "12155": 2.34462,
+ "12160": 2.36042,
+ "12165": 2.3217,
+ "12170": 2.33654,
+ "12175": 2.34429,
+ "12180": 2.33378,
+ "12185": 2.34364,
+ "12190": 2.33822,
+ "12195": 2.3013,
+ "12200": 2.30315,
+ "12205": 2.32625,
+ "12210": 2.35973,
+ "12215": 2.30677,
+ "12220": 2.30311,
+ "12225": 2.25141,
+ "12230": 2.337,
+ "12235": 2.34191,
+ "12240": 2.32761,
+ "12245": 2.28907,
+ "12250": 2.27562,
+ "12255": 2.33792,
+ "12260": 2.3162,
+ "12265": 2.34477,
+ "12270": 2.31496,
+ "12275": 2.31706,
+ "12280": 2.32245,
+ "12285": 2.2891,
+ "12290": 2.31391,
+ "12295": 2.26998,
+ "12300": 2.33169,
+ "12305": 2.27166,
+ "12310": 2.29137,
+ "12315": 2.35763,
+ "12320": 2.29959,
+ "12325": 2.32374,
+ "12330": 2.30301,
+ "12335": 2.32281,
+ "12340": 2.34365,
+ "12345": 2.36988,
+ "12350": 2.34598,
+ "12355": 2.30802,
+ "12360": 2.31641,
+ "12365": 2.33362,
+ "12370": 2.29476,
+ "12375": 2.3034,
+ "12380": 2.29451,
+ "12385": 2.29368,
+ "12390": 2.25148,
+ "12395": 2.30792,
+ "12400": 2.30262,
+ "12405": 2.31508,
+ "12410": 2.30746,
+ "12415": 2.28713,
+ "12420": 2.3211,
+ "12425": 2.30387,
+ "12430": 2.31915,
+ "12435": 2.30225,
+ "12440": 2.33631,
+ "12445": 2.3229,
+ "12450": 2.31029,
+ "12455": 2.24284,
+ "12460": 2.3393,
+ "12465": 2.36818,
+ "12470": 2.27992,
+ "12475": 2.27607,
+ "12480": 2.29448,
+ "12485": 2.3091,
+ "12490": 2.33361,
+ "12495": 2.27164,
+ "12500": 2.32378,
+ "12505": 2.33909,
+ "12510": 2.35961,
+ "12515": 2.27361,
+ "12520": 2.32348,
+ "12525": 2.28983,
+ "12530": 2.32402,
+ "12535": 2.27622,
+ "12540": 2.2884,
+ "12545": 2.29316,
+ "12550": 2.31887,
+ "12555": 2.32698,
+ "12560": 2.30354,
+ "12565": 2.33766,
+ "12570": 2.28173,
+ "12575": 2.30324,
+ "12580": 2.31441,
+ "12585": 2.29584,
+ "12590": 2.33735,
+ "12595": 2.327,
+ "12600": 2.28524,
+ "12605": 2.32325,
+ "12610": 2.36557,
+ "12615": 2.30924,
+ "12620": 2.33596,
+ "12625": 2.33366,
+ "12630": 2.30138,
+ "12635": 2.33851,
+ "12640": 2.29753,
+ "12645": 2.28239,
+ "12650": 2.32994,
+ "12655": 2.26839,
+ "12660": 2.34397,
+ "12665": 2.32017,
+ "12670": 2.31271,
+ "12675": 2.32157,
+ "12680": 2.27735,
+ "12685": 2.36821,
+ "12690": 2.30671,
+ "12695": 2.33358,
+ "12700": 2.29629,
+ "12705": 2.31071,
+ "12710": 2.31098,
+ "12715": 2.2899,
+ "12720": "nan",
+ "12725": "nan",
+ "12730": "nan",
+ "12735": "nan",
+ "12740": "nan",
+ "12745": "nan",
+ "12750": "nan",
+ "12755": "nan",
+ "12760": "nan",
+ "12765": "nan",
+ "12770": "nan",
+ "12775": "nan",
+ "12780": "nan",
+ "12785": "nan",
+ "12790": "nan",
+ "12795": "nan",
+ "12800": "nan",
+ "12805": "nan",
+ "12810": "nan",
+ "12815": "nan",
+ "12820": "nan",
+ "12825": "nan",
+ "12830": "nan",
+ "12835": "nan",
+ "12840": "nan",
+ "12845": "nan",
+ "12850": "nan",
+ "12855": "nan",
+ "12860": "nan",
+ "12865": "nan",
+ "12870": "nan",
+ "12875": "nan",
+ "12880": "nan",
+ "12885": "nan",
+ "12890": "nan",
+ "12895": "nan",
+ "12900": "nan",
+ "12905": "nan",
+ "12910": "nan",
+ "12915": "nan",
+ "12920": "nan",
+ "12925": "nan",
+ "12930": "nan",
+ "12935": "nan",
+ "12940": "nan",
+ "12945": "nan",
+ "12950": "nan",
+ "12955": "nan",
+ "12960": "nan",
+ "12965": "nan",
+ "12970": "nan",
+ "12975": "nan",
+ "12980": "nan",
+ "12985": "nan",
+ "12990": "nan",
+ "12995": "nan",
+ "13000": "nan"
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": 956238208.0,
+ "5": 967339520.0,
+ "10": 971388032.0,
+ "15": 946517440.0,
+ "20": 957353792.0,
+ "25": 1064191488.0,
+ "30": 1177591296.0,
+ "35": 1231325824.0,
+ "40": 1192786176.0,
+ "45": 1126079744.0,
+ "50": 1114080640.0,
+ "55": 1080115072.0,
+ "60": 1029360512.0,
+ "65": 1011742400.0,
+ "70": 989809536.0,
+ "75": 988686592.0,
+ "80": 1010954752.0,
+ "85": 1002030784.0,
+ "90": 982093760.0,
+ "95": 957666304.0,
+ "100": 970864192.0,
+ "105": 978298944.0,
+ "110": 976436928.0,
+ "115": 977307072.0,
+ "120": 961413888.0,
+ "125": 941839232.0,
+ "130": 974677184.0,
+ "135": 964893696.0,
+ "140": 962928448.0,
+ "145": 976848256.0,
+ "150": 921837824.0,
+ "155": 968182848.0,
+ "160": 956372224.0,
+ "165": 959838592.0,
+ "170": 974363456.0,
+ "175": 948986496.0,
+ "180": 946722496.0,
+ "185": 971990144.0,
+ "190": 969058048.0,
+ "195": 985124672.0,
+ "200": 945767360.0,
+ "205": 958344704.0,
+ "210": 979438400.0,
+ "215": 967482112.0,
+ "220": 956425280.0,
+ "225": 962397312.0,
+ "230": 948180160.0,
+ "235": 965227648.0,
+ "240": 966065152.0,
+ "245": 969164800.0,
+ "250": 974440512.0,
+ "255": 925058112.0,
+ "260": 965630592.0,
+ "265": 970669824.0,
+ "270": 959129600.0,
+ "275": 953994048.0,
+ "280": 963424832.0,
+ "285": 945779840.0,
+ "290": 974125184.0,
+ "295": 966700352.0,
+ "300": 967154304.0,
+ "305": 964506304.0,
+ "310": 940357760.0,
+ "315": 967397376.0,
+ "320": 968995200.0,
+ "325": 980551744.0,
+ "330": 972091328.0,
+ "335": 946866432.0,
+ "340": 966592192.0,
+ "345": 973017408.0,
+ "350": 973919360.0,
+ "355": 963260736.0,
+ "360": 948351488.0,
+ "365": 964818176.0,
+ "370": 962952768.0,
+ "375": 958446016.0,
+ "380": 947153536.0,
+ "385": 955985536.0,
+ "390": 945399552.0,
+ "395": 970421056.0,
+ "400": 979772928.0,
+ "405": 968348544.0,
+ "410": 970068352.0,
+ "415": 953155840.0,
+ "420": 943570304.0,
+ "425": 954774784.0,
+ "430": 962661376.0,
+ "435": 977075392.0,
+ "440": 954810496.0,
+ "445": 971890432.0,
+ "450": 963508416.0,
+ "455": 973133696.0,
+ "460": 983712384.0,
+ "465": 945280320.0,
+ "470": 942051776.0,
+ "475": 966991488.0,
+ "480": 966099968.0,
+ "485": 976414720.0,
+ "490": 962540800.0,
+ "495": 945464320.0,
+ "500": 964455104.0,
+ "505": 986008192.0,
+ "510": 965685696.0,
+ "515": 943408256.0,
+ "520": 945017408.0,
+ "525": 971264512.0,
+ "530": 971888128.0,
+ "535": 979137408.0,
+ "540": 969533056.0,
+ "545": 954128704.0,
+ "550": 951265920.0,
+ "555": 987223104.0,
+ "560": 960427264.0,
+ "565": 966615680.0,
+ "570": 975726336.0,
+ "575": 927225856.0,
+ "580": 970697984.0,
+ "585": 961173760.0,
+ "590": 972966208.0,
+ "595": 963684544.0,
+ "600": 937076416.0,
+ "605": 951474048.0,
+ "610": 963361152.0,
+ "615": 970013248.0,
+ "620": 976473088.0,
+ "625": 949583744.0,
+ "630": 954443136.0,
+ "635": 986045888.0,
+ "640": 980977024.0,
+ "645": 955010560.0,
+ "650": 958551424.0,
+ "655": 951653504.0,
+ "660": 961039360.0,
+ "665": 967552640.0,
+ "670": 962513984.0,
+ "675": 968332992.0,
+ "680": 965619840.0,
+ "685": 962863104.0,
+ "690": 961919872.0,
+ "695": 954768768.0,
+ "700": 970332928.0,
+ "705": 945515520.0,
+ "710": 943882176.0,
+ "715": 973357120.0,
+ "720": 968366464.0,
+ "725": 978490112.0,
+ "730": 952193408.0,
+ "735": 948811776.0,
+ "740": 955636096.0,
+ "745": 975864320.0,
+ "750": 981233792.0,
+ "755": 962160000.0,
+ "760": 951964672.0,
+ "765": 967346560.0,
+ "770": 976147456.0,
+ "775": 970547200.0,
+ "780": 977539392.0,
+ "785": 931527936.0,
+ "790": 960440448.0,
+ "795": 964584576.0,
+ "800": 967023104.0,
+ "805": 962318144.0,
+ "810": 940972672.0,
+ "815": 949034688.0,
+ "820": 953181824.0,
+ "825": 954501440.0,
+ "830": 976444416.0,
+ "835": 956077248.0,
+ "840": 948405120.0,
+ "845": 965154560.0,
+ "850": 966026368.0,
+ "855": 960905088.0,
+ "860": 976024960.0,
+ "865": 938161792.0,
+ "870": 966415680.0,
+ "875": 972314176.0,
+ "880": 963123712.0,
+ "885": 967742976.0,
+ "890": 949969792.0,
+ "895": 960017536.0,
+ "900": 974231296.0,
+ "905": 963968512.0,
+ "910": 958436352.0,
+ "915": 956353472.0,
+ "920": 943975552.0,
+ "925": 960830592.0,
+ "930": 978849408.0,
+ "935": 971072064.0,
+ "940": 960906432.0,
+ "945": 945062400.0,
+ "950": 957425856.0,
+ "955": 979037056.0,
+ "960": 983589376.0,
+ "965": 966165568.0,
+ "970": 951229760.0,
+ "975": 961578752.0,
+ "980": 968072448.0,
+ "985": 968988672.0,
+ "990": 984390272.0,
+ "995": 953292224.0,
+ "1000": 934781056.0,
+ "1005": 960145408.0,
+ "1010": 971541504.0,
+ "1015": 985182592.0,
+ "1020": 962781376.0,
+ "1025": 935010688.0,
+ "1030": 974681152.0,
+ "1035": 964994816.0,
+ "1040": 980464704.0,
+ "1045": 960823424.0,
+ "1050": 955203136.0,
+ "1055": 957780224.0,
+ "1060": 967749120.0,
+ "1065": 967115840.0,
+ "1070": 966599040.0,
+ "1075": 950060096.0,
+ "1080": 954510784.0,
+ "1085": 967251200.0,
+ "1090": 977132032.0,
+ "1095": 961234688.0,
+ "1100": 979610240.0,
+ "1105": 953364992.0,
+ "1110": 965953472.0,
+ "1115": 966983552.0,
+ "1120": 970352384.0,
+ "1125": 965708416.0,
+ "1130": 954942720.0,
+ "1135": 965842176.0,
+ "1140": 965173632.0,
+ "1145": 970989120.0,
+ "1150": 955556928.0,
+ "1155": 930576768.0,
+ "1160": 957775488.0,
+ "1165": 978123264.0,
+ "1170": 974302976.0,
+ "1175": 973057920.0,
+ "1180": 973084416.0,
+ "1185": 947341952.0,
+ "1190": 964793728.0,
+ "1195": 953137856.0,
+ "1200": 972846528.0,
+ "1205": 988477760.0,
+ "1210": 931125568.0,
+ "1215": 968646656.0,
+ "1220": 969161280.0,
+ "1225": 975950208.0,
+ "1230": 967334912.0,
+ "1235": 943444608.0,
+ "1240": 955857152.0,
+ "1245": 981503872.0,
+ "1250": 966110016.0,
+ "1255": 973670208.0,
+ "1260": 946495936.0,
+ "1265": 963997568.0,
+ "1270": 960487872.0,
+ "1275": 973615360.0,
+ "1280": 961114240.0,
+ "1285": 957583232.0,
+ "1290": 952528512.0,
+ "1295": 971613312.0,
+ "1300": 968863424.0,
+ "1305": 963741504.0,
+ "1310": 963336512.0,
+ "1315": 943555584.0,
+ "1320": 966306176.0,
+ "1325": 989786432.0,
+ "1330": 969505408.0,
+ "1335": 972299840.0,
+ "1340": 972270080.0,
+ "1345": 960657920.0,
+ "1350": 968638208.0,
+ "1355": 955854464.0,
+ "1360": 971823616.0,
+ "1365": 960387584.0,
+ "1370": 948791680.0,
+ "1375": 973532224.0,
+ "1380": 953470464.0,
+ "1385": 969146880.0,
+ "1390": 975720128.0,
+ "1395": 931674240.0,
+ "1400": 945856640.0,
+ "1405": 976751872.0,
+ "1410": 974512768.0,
+ "1415": 967571456.0,
+ "1420": 966746752.0,
+ "1425": 937378368.0,
+ "1430": 973914368.0,
+ "1435": 978332160.0,
+ "1440": 964179456.0,
+ "1445": 958055552.0,
+ "1450": 946146944.0,
+ "1455": 983924288.0,
+ "1460": 968651392.0,
+ "1465": 948746496.0,
+ "1470": 984241856.0,
+ "1475": 943900608.0,
+ "1480": 963974400.0,
+ "1485": 957349376.0,
+ "1490": 961258496.0,
+ "1495": 980540928.0,
+ "1500": 958331584.0,
+ "1505": 942866048.0,
+ "1510": 984178560.0,
+ "1515": 959093376.0,
+ "1520": 959104704.0,
+ "1525": 952783360.0,
+ "1530": 957743040.0,
+ "1535": 949428736.0,
+ "1540": 971085056.0,
+ "1545": 963133696.0,
+ "1550": 978667328.0,
+ "1555": 952317312.0,
+ "1560": 980089472.0,
+ "1565": 967314944.0,
+ "1570": 973839744.0,
+ "1575": 975492992.0,
+ "1580": 941861248.0,
+ "1585": 970028864.0,
+ "1590": 983823488.0,
+ "1595": 948633216.0,
+ "1600": 967443968.0,
+ "1605": 952450496.0,
+ "1610": 969617920.0,
+ "1615": 983145344.0,
+ "1620": 968020096.0,
+ "1625": 970717120.0,
+ "1630": 962888064.0,
+ "1635": 942311232.0,
+ "1640": 981610496.0,
+ "1645": 973977088.0,
+ "1650": 974184000.0,
+ "1655": 967265408.0,
+ "1660": 940687488.0,
+ "1665": 961703552.0,
+ "1670": 962901440.0,
+ "1675": 971281536.0,
+ "1680": 980877568.0,
+ "1685": 944417920.0,
+ "1690": 964689600.0,
+ "1695": 965642112.0,
+ "1700": 966343488.0,
+ "1705": 985201152.0,
+ "1710": 978353792.0,
+ "1715": 943211392.0,
+ "1720": 977089408.0,
+ "1725": 965872128.0,
+ "1730": 968971648.0,
+ "1735": 965088512.0,
+ "1740": 949712064.0,
+ "1745": 970011968.0,
+ "1750": 959683200.0,
+ "1755": 960089920.0,
+ "1760": 966377728.0,
+ "1765": 951817856.0,
+ "1770": 954664448.0,
+ "1775": 973750144.0,
+ "1780": 970533248.0,
+ "1785": 968826368.0,
+ "1790": 950233728.0,
+ "1795": 945131904.0,
+ "1800": 984666240.0,
+ "1805": 987162112.0,
+ "1810": 977770368.0,
+ "1815": 948006400.0,
+ "1820": 949209216.0,
+ "1825": 978853696.0,
+ "1830": 966362240.0,
+ "1835": 964132608.0,
+ "1840": 972319936.0,
+ "1845": 935414656.0,
+ "1850": 952499648.0,
+ "1855": 980053248.0,
+ "1860": 975868864.0,
+ "1865": 958964992.0,
+ "1870": 958950144.0,
+ "1875": 932595392.0,
+ "1880": 973575168.0,
+ "1885": 978846208.0,
+ "1890": 971357888.0,
+ "1895": 959212672.0,
+ "1900": 947393664.0,
+ "1905": 981829888.0,
+ "1910": 969124096.0,
+ "1915": 970039296.0,
+ "1920": 975599104.0,
+ "1925": 960495488.0,
+ "1930": 977922880.0,
+ "1935": 963250944.0,
+ "1940": 952460480.0,
+ "1945": 981337984.0,
+ "1950": 939169664.0,
+ "1955": 960603840.0,
+ "1960": 970032896.0,
+ "1965": 981177088.0,
+ "1970": 962046720.0,
+ "1975": 952823296.0,
+ "1980": 936849600.0,
+ "1985": 975939584.0,
+ "1990": 965967104.0,
+ "1995": 962612416.0,
+ "2000": 960554496.0,
+ "2005": 954497600.0,
+ "2010": 975579648.0,
+ "2015": 991803584.0,
+ "2020": 975435776.0,
+ "2025": 974302912.0,
+ "2030": 952084224.0,
+ "2035": 967847424.0,
+ "2040": 987457792.0,
+ "2045": 976480512.0,
+ "2050": 984702144.0,
+ "2055": 942837120.0,
+ "2060": 942592896.0,
+ "2065": 966210304.0,
+ "2070": 969622720.0,
+ "2075": 980553856.0,
+ "2080": 977597952.0,
+ "2085": 939639680.0,
+ "2090": 969873216.0,
+ "2095": 961275776.0,
+ "2100": 976720384.0,
+ "2105": 972537024.0,
+ "2110": 959903872.0,
+ "2115": 956877504.0,
+ "2120": 977480704.0,
+ "2125": 962566912.0,
+ "2130": 979617600.0,
+ "2135": 950536512.0,
+ "2140": 946996608.0,
+ "2145": 962277696.0,
+ "2150": 973402816.0,
+ "2155": 972688448.0,
+ "2160": 970314624.0,
+ "2165": 948642112.0,
+ "2170": 961542144.0,
+ "2175": 969378304.0,
+ "2180": 969329600.0,
+ "2185": 947446784.0,
+ "2190": 940480256.0,
+ "2195": 986087808.0,
+ "2200": 961862976.0,
+ "2205": 978924800.0,
+ "2210": 964102208.0,
+ "2215": 963502016.0,
+ "2220": 951310976.0,
+ "2225": 969315328.0,
+ "2230": 976330496.0,
+ "2235": 974026368.0,
+ "2240": 975492992.0,
+ "2245": 960230656.0,
+ "2250": 967638976.0,
+ "2255": 969131008.0,
+ "2260": 975063808.0,
+ "2265": 968257920.0,
+ "2270": 951742208.0,
+ "2275": 962765824.0,
+ "2280": 969640000.0,
+ "2285": 971691776.0,
+ "2290": 962890432.0,
+ "2295": 931409920.0,
+ "2300": 959904512.0,
+ "2305": 970424640.0,
+ "2310": 967445248.0,
+ "2315": 970905920.0,
+ "2320": 975589760.0,
+ "2325": 938586752.0,
+ "2330": 988438784.0,
+ "2335": 977489024.0,
+ "2340": 964596224.0,
+ "2345": 964166144.0,
+ "2350": 947555520.0,
+ "2355": 977030400.0,
+ "2360": 966899712.0,
+ "2365": 977296448.0,
+ "2370": 965070912.0,
+ "2375": 953965184.0,
+ "2380": 962916736.0,
+ "2385": 967196672.0,
+ "2390": 963075712.0,
+ "2395": 974466432.0,
+ "2400": 958410624.0,
+ "2405": 968119936.0,
+ "2410": 951586176.0,
+ "2415": 965904768.0,
+ "2420": 966517120.0,
+ "2425": 959045440.0,
+ "2430": 956684800.0,
+ "2435": 961388736.0,
+ "2440": 959755328.0,
+ "2445": 970890624.0,
+ "2450": 961997568.0,
+ "2455": 922720640.0,
+ "2460": 951952192.0,
+ "2465": 955730496.0,
+ "2470": 972570496.0,
+ "2475": 973810432.0,
+ "2480": 943894016.0,
+ "2485": 944184832.0,
+ "2490": 972410240.0,
+ "2495": 974449664.0,
+ "2500": 973909696.0,
+ "2505": 958489024.0,
+ "2510": 939508608.0,
+ "2515": 979552768.0,
+ "2520": 970472000.0,
+ "2525": 964389632.0,
+ "2530": 955799232.0,
+ "2535": 936598464.0,
+ "2540": 969026752.0,
+ "2545": 970384192.0,
+ "2550": 969460160.0,
+ "2555": 969439360.0,
+ "2560": 964976768.0,
+ "2565": 959763200.0,
+ "2570": 985174784.0,
+ "2575": 957423360.0,
+ "2580": 967423488.0,
+ "2585": 966023040.0,
+ "2590": 956354240.0,
+ "2595": 981828864.0,
+ "2600": 959531392.0,
+ "2605": 962999680.0,
+ "2610": 965970240.0,
+ "2615": 951924032.0,
+ "2620": 971241088.0,
+ "2625": 976456064.0,
+ "2630": 974410624.0,
+ "2635": 948071296.0,
+ "2640": 948137344.0,
+ "2645": 963038912.0,
+ "2650": 953982016.0,
+ "2655": 977112640.0,
+ "2660": 949622528.0,
+ "2665": 953929472.0,
+ "2670": 959064064.0,
+ "2675": 979275776.0,
+ "2680": 961396224.0,
+ "2685": 970702208.0,
+ "2690": 965220736.0,
+ "2695": 943553856.0,
+ "2700": 969424960.0,
+ "2705": 978961792.0,
+ "2710": 971810688.0,
+ "2715": 990813824.0,
+ "2720": 942648768.0,
+ "2725": 967953984.0,
+ "2730": 955465408.0,
+ "2735": 970673536.0,
+ "2740": 977922688.0,
+ "2745": 932280064.0,
+ "2750": 947857536.0,
+ "2755": 956317824.0,
+ "2760": 981696768.0,
+ "2765": 966111104.0,
+ "2770": 948915200.0,
+ "2775": 935830528.0,
+ "2780": 964777472.0,
+ "2785": 969568256.0,
+ "2790": 974273920.0,
+ "2795": 966884736.0,
+ "2800": 944390464.0,
+ "2805": 964354176.0,
+ "2810": 969611264.0,
+ "2815": 975844864.0,
+ "2820": 963083968.0,
+ "2825": 937626688.0,
+ "2830": 956737920.0,
+ "2835": 986317312.0,
+ "2840": 961753792.0,
+ "2845": 967497664.0,
+ "2850": 951711232.0,
+ "2855": 962089856.0,
+ "2860": 954239488.0,
+ "2865": 955880832.0,
+ "2870": 944660864.0,
+ "2875": 974662912.0,
+ "2880": 968200320.0,
+ "2885": 981080832.0,
+ "2890": 953454592.0,
+ "2895": 957177728.0,
+ "2900": 964988928.0,
+ "2905": 931698048.0,
+ "2910": 955728512.0,
+ "2915": 979473920.0,
+ "2920": 970491520.0,
+ "2925": 964976000.0,
+ "2930": 964046976.0,
+ "2935": 940140672.0,
+ "2940": 964910592.0,
+ "2945": 989140480.0,
+ "2950": 965204224.0,
+ "2955": 965114496.0,
+ "2960": 933158144.0,
+ "2965": 968804352.0,
+ "2970": 973033920.0,
+ "2975": 958091392.0,
+ "2980": 964495744.0,
+ "2985": 937267456.0,
+ "2990": 951252864.0,
+ "2995": 978314880.0,
+ "3000": 969275520.0,
+ "3005": 974685760.0,
+ "3010": 950235136.0,
+ "3015": 943853312.0,
+ "3020": 958437760.0,
+ "3025": 975181440.0,
+ "3030": 965028672.0,
+ "3035": 963452096.0,
+ "3040": 952132480.0,
+ "3045": 989792000.0,
+ "3050": 965544960.0,
+ "3055": 982520192.0,
+ "3060": 971238464.0,
+ "3065": 943916032.0,
+ "3070": 978408704.0,
+ "3075": 975205632.0,
+ "3080": 960989888.0,
+ "3085": 962361792.0,
+ "3090": 945962176.0,
+ "3095": 938115328.0,
+ "3100": 972928448.0,
+ "3105": 961988096.0,
+ "3110": 970655872.0,
+ "3115": 963388544.0,
+ "3120": 947128256.0,
+ "3125": 972719360.0,
+ "3130": 952985088.0,
+ "3135": 966041088.0,
+ "3140": 968488576.0,
+ "3145": 937850816.0,
+ "3150": 975009664.0,
+ "3155": 976812608.0,
+ "3160": 969628160.0,
+ "3165": 982192576.0,
+ "3170": 937971968.0,
+ "3175": 953836928.0,
+ "3180": 983808256.0,
+ "3185": 965174912.0,
+ "3190": 968492480.0,
+ "3195": 950942784.0,
+ "3200": 945113472.0,
+ "3205": 959861760.0,
+ "3210": 957485632.0,
+ "3215": 958032128.0,
+ "3220": 968128512.0,
+ "3225": 935613824.0,
+ "3230": 962601216.0,
+ "3235": 975774720.0,
+ "3240": 962621184.0,
+ "3245": 981274048.0,
+ "3250": 943271424.0,
+ "3255": 954597632.0,
+ "3260": 980361600.0,
+ "3265": 963631488.0,
+ "3270": 965175424.0,
+ "3275": 959744000.0,
+ "3280": 967056512.0,
+ "3285": 982489216.0,
+ "3290": 947700992.0,
+ "3295": 966433792.0,
+ "3300": 959177984.0,
+ "3305": 949142144.0,
+ "3310": 979510912.0,
+ "3315": 964295424.0,
+ "3320": 969218304.0,
+ "3325": 956205440.0,
+ "3330": 941178752.0,
+ "3335": 964985728.0,
+ "3340": 956900928.0,
+ "3345": 972501248.0,
+ "3350": 964575232.0,
+ "3355": 943358976.0,
+ "3360": 970036352.0,
+ "3365": 969466496.0,
+ "3370": 954762048.0,
+ "3375": 958689664.0,
+ "3380": 971475328.0,
+ "3385": 947986048.0,
+ "3390": 965792896.0,
+ "3395": 978402432.0,
+ "3400": 978138368.0,
+ "3405": 976735744.0,
+ "3410": 924216960.0,
+ "3415": 955448704.0,
+ "3420": 971845184.0,
+ "3425": 977180160.0,
+ "3430": 973849984.0,
+ "3435": 936096256.0,
+ "3440": 970528896.0,
+ "3445": 957327680.0,
+ "3450": 959851968.0,
+ "3455": 963876096.0,
+ "3460": 967894400.0,
+ "3465": 931343616.0,
+ "3470": 952360448.0,
+ "3475": 973728512.0,
+ "3480": 959751104.0,
+ "3485": 979959616.0,
+ "3490": 944682496.0,
+ "3495": 953930240.0,
+ "3500": 969341440.0,
+ "3505": 964360704.0,
+ "3510": 971246336.0,
+ "3515": 955971584.0,
+ "3520": 958758016.0,
+ "3525": 971938944.0,
+ "3530": 964149248.0,
+ "3535": 983204352.0,
+ "3540": 937514688.0,
+ "3545": 944754752.0,
+ "3550": 984487296.0,
+ "3555": 978070784.0,
+ "3560": 974397760.0,
+ "3565": 968825088.0,
+ "3570": 946719488.0,
+ "3575": 976129728.0,
+ "3580": 977517568.0,
+ "3585": 954579904.0,
+ "3590": 956450816.0,
+ "3595": 951467904.0,
+ "3600": 989007872.0,
+ "3605": 962031488.0,
+ "3610": 965078016.0,
+ "3615": 974658560.0,
+ "3620": 954911104.0,
+ "3625": 939539072.0,
+ "3630": 990173440.0,
+ "3635": 971459328.0,
+ "3640": 976048512.0,
+ "3645": 961512832.0,
+ "3650": 945826304.0,
+ "3655": 965806464.0,
+ "3660": 976218624.0,
+ "3665": 964045056.0,
+ "3670": 977455744.0,
+ "3675": 943498112.0,
+ "3680": 958208192.0,
+ "3685": 964297664.0,
+ "3690": 982105408.0,
+ "3695": 963148416.0,
+ "3700": 950593920.0,
+ "3705": 947368832.0,
+ "3710": 982379520.0,
+ "3715": 972704896.0,
+ "3720": 976150400.0,
+ "3725": 964061952.0,
+ "3730": 948862464.0,
+ "3735": 967102976.0,
+ "3740": 960972544.0,
+ "3745": 969346176.0,
+ "3750": 963958912.0,
+ "3755": 953444224.0,
+ "3760": 976653312.0,
+ "3765": 979853376.0,
+ "3770": 972371136.0,
+ "3775": 972385152.0,
+ "3780": 952596608.0,
+ "3785": 960248896.0,
+ "3790": 985609984.0,
+ "3795": 969208064.0,
+ "3800": 957888896.0,
+ "3805": 972420160.0,
+ "3810": 954529792.0,
+ "3815": 974579008.0,
+ "3820": 963032192.0,
+ "3825": 962083584.0,
+ "3830": 969407232.0,
+ "3835": 934758144.0,
+ "3840": 971268928.0,
+ "3845": 986861568.0,
+ "3850": 968872576.0,
+ "3855": 965259136.0,
+ "3860": 948069504.0,
+ "3865": 975060864.0,
+ "3870": 985098240.0,
+ "3875": 983041536.0,
+ "3880": 963614976.0,
+ "3885": 953023296.0,
+ "3890": 960283264.0,
+ "3895": 960602304.0,
+ "3900": 984924288.0,
+ "3905": 976209664.0,
+ "3910": 987348480.0,
+ "3915": 946018304.0,
+ "3920": 974866688.0,
+ "3925": 961247104.0,
+ "3930": 976776832.0,
+ "3935": 978912448.0,
+ "3940": 950291456.0,
+ "3945": 960272512.0,
+ "3950": 974197120.0,
+ "3955": 972954688.0,
+ "3960": 974078400.0,
+ "3965": 950873408.0,
+ "3970": 980679552.0,
+ "3975": 960724736.0,
+ "3980": 977530624.0,
+ "3985": 962948480.0,
+ "3990": 972742784.0,
+ "3995": 953701248.0,
+ "4000": 974987648.0,
+ "4005": 971658240.0,
+ "4010": 978408000.0,
+ "4015": 971490816.0,
+ "4020": 950321984.0,
+ "4025": 968433152.0,
+ "4030": 997947904.0,
+ "4035": 978560256.0,
+ "4040": 959811840.0,
+ "4045": 939663424.0,
+ "4050": 944705664.0,
+ "4055": 980972864.0,
+ "4060": 977696256.0,
+ "4065": 975729856.0,
+ "4070": 942163712.0,
+ "4075": 945752576.0,
+ "4080": 988740096.0,
+ "4085": 962092544.0,
+ "4090": 983357312.0,
+ "4095": 986957760.0,
+ "4100": 957199552.0,
+ "4105": 954098048.0,
+ "4110": 966498048.0,
+ "4115": 976029248.0,
+ "4120": 983531264.0,
+ "4125": 960040704.0,
+ "4130": 967276416.0,
+ "4135": 971442944.0,
+ "4140": 963169920.0,
+ "4145": 956187008.0,
+ "4150": 960281664.0,
+ "4155": 946204096.0,
+ "4160": 968426560.0,
+ "4165": 970325504.0,
+ "4170": 971982400.0,
+ "4175": 955860736.0,
+ "4180": 940972928.0,
+ "4185": 968293120.0,
+ "4190": 967991424.0,
+ "4195": 989223296.0,
+ "4200": 962689152.0,
+ "4205": 960581376.0,
+ "4210": 971849664.0,
+ "4215": 974144832.0,
+ "4220": 981168128.0,
+ "4225": 975219904.0,
+ "4230": 952725888.0,
+ "4235": 958493120.0,
+ "4240": 966823744.0,
+ "4245": 961732352.0,
+ "4250": 965845184.0,
+ "4255": 958291712.0,
+ "4260": 949594240.0,
+ "4265": 964124672.0,
+ "4270": 978406720.0,
+ "4275": 975433024.0,
+ "4280": 962683264.0,
+ "4285": 951462912.0,
+ "4290": 980053632.0,
+ "4295": 968853248.0,
+ "4300": 958285696.0,
+ "4305": 966815232.0,
+ "4310": 939637760.0,
+ "4315": 949402880.0,
+ "4320": 984559808.0,
+ "4325": 982559360.0,
+ "4330": 974745216.0,
+ "4335": 949470976.0,
+ "4340": 959514176.0,
+ "4345": 956618752.0,
+ "4350": 979947008.0,
+ "4355": 968881664.0,
+ "4360": 966325632.0,
+ "4365": 941257088.0,
+ "4370": 969506304.0,
+ "4375": 972982016.0,
+ "4380": 966147840.0,
+ "4385": 972036224.0,
+ "4390": 954165312.0,
+ "4395": 951879808.0,
+ "4400": 973676928.0,
+ "4405": 972490496.0,
+ "4410": 967920192.0,
+ "4415": 958996864.0,
+ "4420": 960802432.0,
+ "4425": 976427904.0,
+ "4430": 965954752.0,
+ "4435": 975925632.0,
+ "4440": 962366848.0,
+ "4445": 954836544.0,
+ "4450": 978343552.0,
+ "4455": 960242112.0,
+ "4460": 968629632.0,
+ "4465": 968842496.0,
+ "4470": 944164096.0,
+ "4475": 952008256.0,
+ "4480": 978873920.0,
+ "4485": 968328192.0,
+ "4490": 957074432.0,
+ "4495": 938975104.0,
+ "4500": 953265152.0,
+ "4505": 977166336.0,
+ "4510": 978761984.0,
+ "4515": 962492608.0,
+ "4520": 958836352.0,
+ "4525": 958138496.0,
+ "4530": 964428224.0,
+ "4535": 976629376.0,
+ "4540": 976870912.0,
+ "4545": 970145152.0,
+ "4550": 953234560.0,
+ "4555": 959602432.0,
+ "4560": 972460416.0,
+ "4565": 973453824.0,
+ "4570": 978908928.0,
+ "4575": 957797248.0,
+ "4580": 963180096.0,
+ "4585": 957393088.0,
+ "4590": 986579968.0,
+ "4595": 960189568.0,
+ "4600": 952266496.0,
+ "4605": 959322112.0,
+ "4610": 963734336.0,
+ "4615": 957987648.0,
+ "4620": 960280832.0,
+ "4625": 973823104.0,
+ "4630": 944532608.0,
+ "4635": 977049728.0,
+ "4640": 960298112.0,
+ "4645": 982029440.0,
+ "4650": 962399104.0,
+ "4655": 939445376.0,
+ "4660": 964037248.0,
+ "4665": 962602240.0,
+ "4670": 976670592.0,
+ "4675": 963501504.0,
+ "4680": 957462656.0,
+ "4685": 949722432.0,
+ "4690": 956974336.0,
+ "4695": 969988224.0,
+ "4700": 961213696.0,
+ "4705": 970804928.0,
+ "4710": 934505216.0,
+ "4715": 970426496.0,
+ "4720": 966386432.0,
+ "4725": 980291648.0,
+ "4730": 965899776.0,
+ "4735": 937883712.0,
+ "4740": 960124416.0,
+ "4745": 976034240.0,
+ "4750": 967996672.0,
+ "4755": 984984640.0,
+ "4760": 959160704.0,
+ "4765": 955260480.0,
+ "4770": 958633280.0,
+ "4775": 991091904.0,
+ "4780": 976806144.0,
+ "4785": 967569792.0,
+ "4790": 943731520.0,
+ "4795": 955836096.0,
+ "4800": 967734144.0,
+ "4805": 976504192.0,
+ "4810": 965124096.0,
+ "4815": 958006016.0,
+ "4820": 973992064.0,
+ "4825": 961493376.0,
+ "4830": 962612992.0,
+ "4835": 972519872.0,
+ "4840": 948900928.0,
+ "4845": 965727232.0,
+ "4850": 960317312.0,
+ "4855": 964131072.0,
+ "4860": 963038528.0,
+ "4865": 967482752.0,
+ "4870": 957157120.0,
+ "4875": 983549312.0,
+ "4880": 957072064.0,
+ "4885": 977004032.0,
+ "4890": 959752640.0,
+ "4895": 942237760.0,
+ "4900": 973663168.0,
+ "4905": 975250944.0,
+ "4910": 969207680.0,
+ "4915": 970077824.0,
+ "4920": 941168832.0,
+ "4925": 954797120.0,
+ "4930": 977033472.0,
+ "4935": 963763264.0,
+ "4940": 972636032.0,
+ "4945": 960056448.0,
+ "4950": 940842816.0,
+ "4955": 968036096.0,
+ "4960": 976849664.0,
+ "4965": 960984576.0,
+ "4970": 958615232.0,
+ "4975": 933738496.0,
+ "4980": 960849216.0,
+ "4985": 963012608.0,
+ "4990": 963514752.0,
+ "4995": 986270592.0,
+ "5000": 940714560.0,
+ "5005": 968912192.0,
+ "5010": 970295808.0,
+ "5015": 965196928.0,
+ "5020": 966677760.0,
+ "5025": 949434880.0,
+ "5030": 953527168.0,
+ "5035": 967317120.0,
+ "5040": 955677312.0,
+ "5045": 969128704.0,
+ "5050": 953368448.0,
+ "5055": 954787584.0,
+ "5060": 962992448.0,
+ "5065": 952158720.0,
+ "5070": 973622016.0,
+ "5075": 978544512.0,
+ "5080": 942841984.0,
+ "5085": 965826688.0,
+ "5090": 972869760.0,
+ "5095": 964386304.0,
+ "5100": 958317952.0,
+ "5105": 965304704.0,
+ "5110": 950426368.0,
+ "5115": 972334464.0,
+ "5120": 960412736.0,
+ "5125": 969669760.0,
+ "5130": 938722560.0,
+ "5135": 943648384.0,
+ "5140": 969888448.0,
+ "5145": 968602496.0,
+ "5150": 970576256.0,
+ "5155": 972561344.0,
+ "5160": 926526656.0,
+ "5165": 961605120.0,
+ "5170": 966812736.0,
+ "5175": 965999104.0,
+ "5180": 963667584.0,
+ "5185": 930790400.0,
+ "5190": 949767296.0,
+ "5195": 972459776.0,
+ "5200": 973758400.0,
+ "5205": 968213504.0,
+ "5210": 960515072.0,
+ "5215": 928754176.0,
+ "5220": 979235328.0,
+ "5225": 984792384.0,
+ "5230": 975061632.0,
+ "5235": 975080128.0,
+ "5240": 944338688.0,
+ "5245": 970809856.0,
+ "5250": 972437376.0,
+ "5255": 966945472.0,
+ "5260": 976754048.0,
+ "5265": 942271616.0,
+ "5270": 969238528.0,
+ "5275": 970073280.0,
+ "5280": 962813184.0,
+ "5285": 964085376.0,
+ "5290": 932502848.0,
+ "5295": 951775104.0,
+ "5300": 975637440.0,
+ "5305": 951879040.0,
+ "5310": 968053120.0,
+ "5315": 955768512.0,
+ "5320": 950946944.0,
+ "5325": 973089792.0,
+ "5330": 967782400.0,
+ "5335": 967548864.0,
+ "5340": 966491200.0,
+ "5345": 962986112.0,
+ "5350": 978936576.0,
+ "5355": 972314176.0,
+ "5360": 963844096.0,
+ "5365": 965283712.0,
+ "5370": 947850112.0,
+ "5375": 948870336.0,
+ "5380": 967211968.0,
+ "5385": 980515840.0,
+ "5390": 965325824.0,
+ "5395": 955184768.0,
+ "5400": 948286848.0,
+ "5405": 974268288.0,
+ "5410": 967819648.0,
+ "5415": 975976960.0,
+ "5420": 967506496.0,
+ "5425": 937351168.0,
+ "5430": 963858112.0,
+ "5435": 971875392.0,
+ "5440": 969079424.0,
+ "5445": 957489408.0,
+ "5450": 919444096.0,
+ "5455": 952065152.0,
+ "5460": 962363968.0,
+ "5465": 978856256.0,
+ "5470": 980933120.0,
+ "5475": 941573760.0,
+ "5480": 955804288.0,
+ "5485": 964902528.0,
+ "5490": 975961920.0,
+ "5495": 962769856.0,
+ "5500": 971131520.0,
+ "5505": 956915200.0,
+ "5510": 968537216.0,
+ "5515": 945405696.0,
+ "5520": 963158016.0,
+ "5525": 975882624.0,
+ "5530": 936598528.0,
+ "5535": 970608704.0,
+ "5540": 960245440.0,
+ "5545": 971925632.0,
+ "5550": 967860992.0,
+ "5555": 955919872.0,
+ "5560": 954497024.0,
+ "5565": 968752896.0,
+ "5570": 945162944.0,
+ "5575": 960589440.0,
+ "5580": 960575872.0,
+ "5585": 959483520.0,
+ "5590": 977668864.0,
+ "5595": 975245568.0,
+ "5600": 962985088.0,
+ "5605": 964116352.0,
+ "5610": 943157952.0,
+ "5615": 966602176.0,
+ "5620": 963311040.0,
+ "5625": 982288320.0,
+ "5630": 976050624.0,
+ "5635": 957225216.0,
+ "5640": 951468544.0,
+ "5645": 967830464.0,
+ "5650": 979133120.0,
+ "5655": 983462400.0,
+ "5660": 956445056.0,
+ "5665": 953438656.0,
+ "5670": 966098048.0,
+ "5675": 967653504.0,
+ "5680": 978696960.0,
+ "5685": 962081664.0,
+ "5690": 935846976.0,
+ "5695": 963701312.0,
+ "5700": 952427904.0,
+ "5705": 974415040.0,
+ "5710": 971206592.0,
+ "5715": 946098688.0,
+ "5720": 974992128.0,
+ "5725": 967437760.0,
+ "5730": 978488448.0,
+ "5735": 964857088.0,
+ "5740": 943624576.0,
+ "5745": 971192576.0,
+ "5750": 981903232.0,
+ "5755": 956694400.0,
+ "5760": 963635840.0,
+ "5765": 957695488.0,
+ "5770": 955612928.0,
+ "5775": 970811008.0,
+ "5780": 962763520.0,
+ "5785": 970576384.0,
+ "5790": 974567808.0,
+ "5795": 949716224.0,
+ "5800": 965927104.0,
+ "5805": 968762112.0,
+ "5810": 976023424.0,
+ "5815": 970054208.0,
+ "5820": 936427008.0,
+ "5825": 969278720.0,
+ "5830": 977675520.0,
+ "5835": 975029120.0,
+ "5840": 963087488.0,
+ "5845": 968739776.0,
+ "5850": 942983296.0,
+ "5855": 975974912.0,
+ "5860": 979586240.0,
+ "5865": 978395392.0,
+ "5870": 968875392.0,
+ "5875": 942253952.0,
+ "5880": 964460160.0,
+ "5885": 974721536.0,
+ "5890": 972585792.0,
+ "5895": 965718720.0,
+ "5900": 941322880.0,
+ "5905": 961816384.0,
+ "5910": 958508224.0,
+ "5915": 968174272.0,
+ "5920": 977432704.0,
+ "5925": 959568128.0,
+ "5930": 946687488.0,
+ "5935": 952420608.0,
+ "5940": 977686528.0,
+ "5945": 984781184.0,
+ "5950": 980727168.0,
+ "5955": 935069312.0,
+ "5960": 961687360.0,
+ "5965": 966000512.0,
+ "5970": 970723136.0,
+ "5975": 961933056.0,
+ "5980": 958259328.0,
+ "5985": 964539008.0,
+ "5990": 973566464.0,
+ "5995": 955920064.0,
+ "6000": 955633408.0,
+ "6005": 961454336.0,
+ "6010": 952744448.0,
+ "6015": 974466048.0,
+ "6020": 978364544.0,
+ "6025": 972190464.0,
+ "6030": 955422784.0,
+ "6035": 946979392.0,
+ "6040": 962839104.0,
+ "6045": 983682560.0,
+ "6050": 956576768.0,
+ "6055": 963457920.0,
+ "6060": 945840704.0,
+ "6065": 958443456.0,
+ "6070": 978391168.0,
+ "6075": 978098944.0,
+ "6080": 957542208.0,
+ "6085": 947739520.0,
+ "6090": 953768064.0,
+ "6095": 964672384.0,
+ "6100": 979961024.0,
+ "6105": 971023872.0,
+ "6110": 961767296.0,
+ "6115": 943790080.0,
+ "6120": 968561856.0,
+ "6125": 960789888.0,
+ "6130": 983979392.0,
+ "6135": 961018880.0,
+ "6140": 958671488.0,
+ "6145": 971176064.0,
+ "6150": 968591040.0,
+ "6155": 974967424.0,
+ "6160": 977216128.0,
+ "6165": 952743744.0,
+ "6170": 951068800.0,
+ "6175": 963383488.0,
+ "6180": 969463616.0,
+ "6185": 966344704.0,
+ "6190": 963743872.0,
+ "6195": 947204864.0,
+ "6200": 969329280.0,
+ "6205": 967223872.0,
+ "6210": 959204928.0,
+ "6215": 973012224.0,
+ "6220": 936449280.0,
+ "6225": 978666880.0,
+ "6230": 976060672.0,
+ "6235": 971802752.0,
+ "6240": 965944192.0,
+ "6245": 956029440.0,
+ "6250": 956646784.0,
+ "6255": 973400000.0,
+ "6260": 978902144.0,
+ "6265": 974880832.0,
+ "6270": 958830848.0,
+ "6275": 963811456.0,
+ "6280": 973239808.0,
+ "6285": 966086208.0,
+ "6290": 971004032.0,
+ "6295": 987564544.0,
+ "6300": 947518144.0,
+ "6305": 964881664.0,
+ "6310": 979127360.0,
+ "6315": 978491904.0,
+ "6320": 971846912.0,
+ "6325": 923003392.0,
+ "6330": 959219840.0,
+ "6335": 974940416.0,
+ "6340": 984750784.0,
+ "6345": 966922112.0,
+ "6350": 944686016.0,
+ "6355": 958149952.0,
+ "6360": 972846848.0,
+ "6365": 972207808.0,
+ "6370": 959152064.0,
+ "6375": 967179008.0,
+ "6380": 951498688.0,
+ "6385": 973743872.0,
+ "6390": 965576576.0,
+ "6395": 975377600.0,
+ "6400": 984092928.0,
+ "6405": 944003712.0,
+ "6410": 977387776.0,
+ "6415": 971562688.0,
+ "6420": 956554560.0,
+ "6425": 961257536.0,
+ "6430": 957499520.0,
+ "6435": 960148992.0,
+ "6440": 968996800.0,
+ "6445": 973577728.0,
+ "6450": 974256256.0,
+ "6455": 962215552.0,
+ "6460": 941143040.0,
+ "6465": 974594240.0,
+ "6470": 979986688.0,
+ "6475": 960787200.0,
+ "6480": 967680768.0,
+ "6485": 948695872.0,
+ "6490": 970736896.0,
+ "6495": 988356416.0,
+ "6500": 980444224.0,
+ "6505": 972048768.0,
+ "6510": 951538560.0,
+ "6515": 957553536.0,
+ "6520": 978988160.0,
+ "6525": 979075968.0,
+ "6530": 973432448.0,
+ "6535": 967927680.0,
+ "6540": 950344576.0,
+ "6545": 966282304.0,
+ "6550": 979666944.0,
+ "6555": 967156480.0,
+ "6560": 975378752.0,
+ "6565": 949388480.0,
+ "6570": 952153856.0,
+ "6575": 962491968.0,
+ "6580": 975828224.0,
+ "6585": 979201600.0,
+ "6590": 949150784.0,
+ "6595": 961570816.0,
+ "6600": 961156800.0,
+ "6605": 961387776.0,
+ "6610": 985139264.0,
+ "6615": 959604992.0,
+ "6620": 944527488.0,
+ "6625": 970979072.0,
+ "6630": 971344128.0,
+ "6635": 964018688.0,
+ "6640": 959735296.0,
+ "6645": 951066496.0,
+ "6650": 978983360.0,
+ "6655": 965888896.0,
+ "6660": 968655296.0,
+ "6665": 969051328.0,
+ "6670": 932925440.0,
+ "6675": 970809600.0,
+ "6680": 968930432.0,
+ "6685": 958834048.0,
+ "6690": 956239360.0,
+ "6695": 955414528.0,
+ "6700": 961951744.0,
+ "6705": 979267840.0,
+ "6710": 971185792.0,
+ "6715": 966765440.0,
+ "6720": 973915840.0,
+ "6725": 941977280.0,
+ "6730": 979326080.0,
+ "6735": 994512256.0,
+ "6740": 976712000.0,
+ "6745": 974664320.0,
+ "6750": 939125120.0,
+ "6755": 977587840.0,
+ "6760": 969794048.0,
+ "6765": 978431104.0,
+ "6770": 975390848.0,
+ "6775": 943240896.0,
+ "6780": 947394304.0,
+ "6785": 975457408.0,
+ "6790": 960606208.0,
+ "6795": 976119744.0,
+ "6800": 973160064.0,
+ "6805": 946682880.0,
+ "6810": 958523648.0,
+ "6815": 970806784.0,
+ "6820": 977803264.0,
+ "6825": 969525248.0,
+ "6830": 950521664.0,
+ "6835": 981722624.0,
+ "6840": 983268352.0,
+ "6845": 949033536.0,
+ "6850": 965785344.0,
+ "6855": 954186880.0,
+ "6860": 979099776.0,
+ "6865": 984086016.0,
+ "6870": 965062656.0,
+ "6875": 978762112.0,
+ "6880": 950469248.0,
+ "6885": 958533248.0,
+ "6890": 960226240.0,
+ "6895": 965886592.0,
+ "6900": 985432576.0,
+ "6905": 968809536.0,
+ "6910": 950022976.0,
+ "6915": 970876928.0,
+ "6920": 967301760.0,
+ "6925": 965291136.0,
+ "6930": 964642368.0,
+ "6935": 952024256.0,
+ "6940": 963106176.0,
+ "6945": 986617216.0,
+ "6950": 973548096.0,
+ "6955": 964838400.0,
+ "6960": 940152704.0,
+ "6965": 974565248.0,
+ "6970": 978542720.0,
+ "6975": 985066496.0,
+ "6980": 982483584.0,
+ "6985": 959913728.0,
+ "6990": 945569024.0,
+ "6995": 987496320.0,
+ "7000": 963293504.0,
+ "7005": 962922496.0,
+ "7010": 984873920.0,
+ "7015": 945472128.0,
+ "7020": 982970688.0,
+ "7025": 968730752.0,
+ "7030": 953267840.0,
+ "7035": 983032576.0,
+ "7040": 950545536.0,
+ "7045": 955794944.0,
+ "7050": 959998336.0,
+ "7055": 963843712.0,
+ "7060": 976703872.0,
+ "7065": 968243456.0,
+ "7070": 953732288.0,
+ "7075": 956633088.0,
+ "7080": 969051392.0,
+ "7085": 965646976.0,
+ "7090": 969275648.0,
+ "7095": 959794944.0,
+ "7100": 973258112.0,
+ "7105": 973233792.0,
+ "7110": 970156224.0,
+ "7115": 958695360.0,
+ "7120": 949100544.0,
+ "7125": 963213760.0,
+ "7130": 971458688.0,
+ "7135": 964614656.0,
+ "7140": 961472768.0,
+ "7145": 930478144.0,
+ "7150": 946305344.0,
+ "7155": 990945728.0,
+ "7160": 968504640.0,
+ "7165": 957035648.0,
+ "7170": 968091136.0,
+ "7175": 955889536.0,
+ "7180": 958098304.0,
+ "7185": 984769280.0,
+ "7190": 978724544.0,
+ "7195": 973823360.0,
+ "7200": 935906432.0,
+ "7205": 957574976.0,
+ "7210": 967287296.0,
+ "7215": 969486848.0,
+ "7220": 982282368.0,
+ "7225": 928875264.0,
+ "7230": 949709248.0,
+ "7235": 967103360.0,
+ "7240": 966992512.0,
+ "7245": 967701120.0,
+ "7250": 949504448.0,
+ "7255": 957437248.0,
+ "7260": 970348672.0,
+ "7265": 975011200.0,
+ "7270": 959861440.0,
+ "7275": 959238720.0,
+ "7280": 957073536.0,
+ "7285": 977612800.0,
+ "7290": 977410304.0,
+ "7295": 963012224.0,
+ "7300": 975696000.0,
+ "7305": 964281216.0,
+ "7310": 977590784.0,
+ "7315": 966920512.0,
+ "7320": 974589952.0,
+ "7325": 967030144.0,
+ "7330": 959814144.0,
+ "7335": 964098432.0,
+ "7340": 977865216.0,
+ "7345": 967964416.0,
+ "7350": 984903808.0,
+ "7355": 959835136.0,
+ "7360": 949175360.0,
+ "7365": 972711808.0,
+ "7370": 982594432.0,
+ "7375": 963297920.0,
+ "7380": 964137600.0,
+ "7385": 948664704.0,
+ "7390": 963972352.0,
+ "7395": 958382528.0,
+ "7400": 969862656.0,
+ "7405": 987809920.0,
+ "7410": 952313344.0,
+ "7415": 950431680.0,
+ "7420": 967091520.0,
+ "7425": 983148160.0,
+ "7430": 965896640.0,
+ "7435": 972987520.0,
+ "7440": 937056448.0,
+ "7445": 968792960.0,
+ "7450": 980130752.0,
+ "7455": 971354240.0,
+ "7460": 972147072.0,
+ "7465": 939438976.0,
+ "7470": 971876800.0,
+ "7475": 958106752.0,
+ "7480": 969431232.0,
+ "7485": 961637632.0,
+ "7490": 934427904.0,
+ "7495": 957212288.0,
+ "7500": 969499392.0,
+ "7505": 970369536.0,
+ "7510": 972331904.0,
+ "7515": 979583680.0,
+ "7520": 951895680.0,
+ "7525": 970542720.0,
+ "7530": 954594944.0,
+ "7535": 971456640.0,
+ "7540": 979399808.0,
+ "7545": 959210880.0,
+ "7550": 960539264.0,
+ "7555": 960183744.0,
+ "7560": 970406144.0,
+ "7565": 955121152.0,
+ "7570": 942681216.0,
+ "7575": 965667328.0,
+ "7580": 982016448.0,
+ "7585": 978957312.0,
+ "7590": 970352896.0,
+ "7595": 949944064.0,
+ "7600": 946166208.0,
+ "7605": 982330368.0,
+ "7610": 969105344.0,
+ "7615": 988675072.0,
+ "7620": 957024576.0,
+ "7625": 941271296.0,
+ "7630": 971452736.0,
+ "7635": 984388096.0,
+ "7640": 983484032.0,
+ "7645": 968295040.0,
+ "7650": 959136256.0,
+ "7655": 962572608.0,
+ "7660": 969101888.0,
+ "7665": 978367872.0,
+ "7670": 975598336.0,
+ "7675": 975485568.0,
+ "7680": 942860288.0,
+ "7685": 960278336.0,
+ "7690": 975426304.0,
+ "7695": 982429888.0,
+ "7700": 979395712.0,
+ "7705": 940767936.0,
+ "7710": 974726144.0,
+ "7715": 979320064.0,
+ "7720": 967701888.0,
+ "7725": 960431040.0,
+ "7730": 943571072.0,
+ "7735": 968205120.0,
+ "7740": 980144320.0,
+ "7745": 964028096.0,
+ "7750": 963510528.0,
+ "7755": 959923712.0,
+ "7760": 970409472.0,
+ "7765": 970840064.0,
+ "7770": 962608256.0,
+ "7775": 981822592.0,
+ "7780": 964937536.0,
+ "7785": 959813888.0,
+ "7790": 968418688.0,
+ "7795": 968810368.0,
+ "7800": 971942400.0,
+ "7805": 968393600.0,
+ "7810": 945837056.0,
+ "7815": 963393664.0,
+ "7820": 974554240.0,
+ "7825": 963661440.0,
+ "7830": 957249792.0,
+ "7835": 949943296.0,
+ "7840": 957195520.0,
+ "7845": 954041216.0,
+ "7850": 979637248.0,
+ "7855": 986769152.0,
+ "7860": 947535296.0,
+ "7865": 949192448.0,
+ "7870": 965347520.0,
+ "7875": 975739648.0,
+ "7880": 968699904.0,
+ "7885": 969383808.0,
+ "7890": 951943040.0,
+ "7895": 974806144.0,
+ "7900": 963731392.0,
+ "7905": 963952640.0,
+ "7910": 965562240.0,
+ "7915": 943316480.0,
+ "7920": 951016704.0,
+ "7925": 969537152.0,
+ "7930": 964735232.0,
+ "7935": 984412288.0,
+ "7940": 964486016.0,
+ "7945": 950846144.0,
+ "7950": 962123008.0,
+ "7955": 980081408.0,
+ "7960": 963652928.0,
+ "7965": 953426176.0,
+ "7970": 951790080.0,
+ "7975": 969476032.0,
+ "7980": 965341184.0,
+ "7985": 959473280.0,
+ "7990": 968080384.0,
+ "7995": 946841344.0,
+ "8000": 962742912.0,
+ "8005": 980642304.0,
+ "8010": 965851136.0,
+ "8015": 982918656.0,
+ "8020": 961115904.0,
+ "8025": 965420992.0,
+ "8030": 958410048.0,
+ "8035": 975936640.0,
+ "8040": 960876928.0,
+ "8045": 948345920.0,
+ "8050": 959722624.0,
+ "8055": 979580480.0,
+ "8060": 969802624.0,
+ "8065": 958418816.0,
+ "8070": 964214784.0,
+ "8075": 942438528.0,
+ "8080": 966293376.0,
+ "8085": 966952576.0,
+ "8090": 983725632.0,
+ "8095": 988883968.0,
+ "8100": 966654784.0,
+ "8105": 944475136.0,
+ "8110": 969154432.0,
+ "8115": 985881472.0,
+ "8120": 975193216.0,
+ "8125": 964510016.0,
+ "8130": 966592896.0,
+ "8135": 968277376.0,
+ "8140": 964192384.0,
+ "8145": 995580288.0,
+ "8150": 973731712.0,
+ "8155": 938881344.0,
+ "8160": 964549120.0,
+ "8165": 973467840.0,
+ "8170": 968669248.0,
+ "8175": 961785344.0,
+ "8180": 936312960.0,
+ "8185": 963032448.0,
+ "8190": 968045440.0,
+ "8195": 977582464.0,
+ "8200": 956846720.0,
+ "8205": 960801152.0,
+ "8210": 946900800.0,
+ "8215": 982422912.0,
+ "8220": 988640640.0,
+ "8225": 966550912.0,
+ "8230": 962577728.0,
+ "8235": 934266880.0,
+ "8240": 980121088.0,
+ "8245": 976742656.0,
+ "8250": 964438272.0,
+ "8255": 977690240.0,
+ "8260": 957215808.0,
+ "8265": 982982016.0,
+ "8270": 953045504.0,
+ "8275": 974443520.0,
+ "8280": 974709696.0,
+ "8285": 953580736.0,
+ "8290": 940207104.0,
+ "8295": 981334400.0,
+ "8300": 973592832.0,
+ "8305": 978208704.0,
+ "8310": 951476736.0,
+ "8315": 938155648.0,
+ "8320": 977608256.0,
+ "8325": 968131584.0,
+ "8330": 990118272.0,
+ "8335": 975845952.0,
+ "8340": 947797120.0,
+ "8345": 971108608.0,
+ "8350": 970389568.0,
+ "8355": 975309696.0,
+ "8360": 979959168.0,
+ "8365": 933125760.0,
+ "8370": 965870144.0,
+ "8375": 979941504.0,
+ "8380": 965457536.0,
+ "8385": 972902272.0,
+ "8390": 962932800.0,
+ "8395": 951285504.0,
+ "8400": 972582528.0,
+ "8405": 951932416.0,
+ "8410": 960913088.0,
+ "8415": 965733120.0,
+ "8420": 942024128.0,
+ "8425": 968350080.0,
+ "8430": 960937984.0,
+ "8435": 966275200.0,
+ "8440": 969976960.0,
+ "8445": 953032704.0,
+ "8450": 984572288.0,
+ "8455": 990529984.0,
+ "8460": 969012352.0,
+ "8465": 967267968.0,
+ "8470": 963153024.0,
+ "8475": 942747136.0,
+ "8480": 987456448.0,
+ "8485": 979909888.0,
+ "8490": 992218368.0,
+ "8495": 971653952.0,
+ "8500": 951453312.0,
+ "8505": 983301760.0,
+ "8510": 973866432.0,
+ "8515": 968949184.0,
+ "8520": 961717248.0,
+ "8525": 945032064.0,
+ "8530": 984401920.0,
+ "8535": 978362240.0,
+ "8540": 967715968.0,
+ "8545": 968670464.0,
+ "8550": 942272768.0,
+ "8555": 971942272.0,
+ "8560": 958429504.0,
+ "8565": 975786240.0,
+ "8570": 974826944.0,
+ "8575": 971790976.0,
+ "8580": 932055168.0,
+ "8585": 965917312.0,
+ "8590": 978957568.0,
+ "8595": 979130816.0,
+ "8600": 984002176.0,
+ "8605": 958012032.0,
+ "8610": 983798656.0,
+ "8615": 977747648.0,
+ "8620": 963337984.0,
+ "8625": 979429248.0,
+ "8630": 943333760.0,
+ "8635": 962054912.0,
+ "8640": 973241024.0,
+ "8645": 970513088.0,
+ "8650": 969657984.0,
+ "8655": 970996480.0,
+ "8660": 944544640.0,
+ "8665": 986559104.0,
+ "8670": 960599424.0,
+ "8675": 974340096.0,
+ "8680": 962877824.0,
+ "8685": 955741440.0,
+ "8690": 978440576.0,
+ "8695": 968964672.0,
+ "8700": 972962240.0,
+ "8705": 973851264.0,
+ "8710": 946968384.0,
+ "8715": 973256064.0,
+ "8720": 958203392.0,
+ "8725": 978976448.0,
+ "8730": 985659904.0,
+ "8735": 952336768.0,
+ "8740": 940581440.0,
+ "8745": 987553920.0,
+ "8750": 972231552.0,
+ "8755": 971589760.0,
+ "8760": 965322944.0,
+ "8765": 935034496.0,
+ "8770": 986508864.0,
+ "8775": 969788416.0,
+ "8780": 967404160.0,
+ "8785": 961988480.0,
+ "8790": 947342144.0,
+ "8795": 969540800.0,
+ "8800": 970839296.0,
+ "8805": 973361792.0,
+ "8810": 983391808.0,
+ "8815": 951703552.0,
+ "8820": 939856512.0,
+ "8825": 964356992.0,
+ "8830": 981113152.0,
+ "8835": 971531776.0,
+ "8840": 979490752.0,
+ "8845": 951368832.0,
+ "8850": 986906880.0,
+ "8855": 970990464.0,
+ "8860": 962109568.0,
+ "8865": 957197952.0,
+ "8870": 945626752.0,
+ "8875": 968480512.0,
+ "8880": 984062848.0,
+ "8885": 971587712.0,
+ "8890": 970177344.0,
+ "8895": 952798592.0,
+ "8900": 961704064.0,
+ "8905": 976776384.0,
+ "8910": 982036032.0,
+ "8915": 980878848.0,
+ "8920": 967569408.0,
+ "8925": 940163840.0,
+ "8930": 970577664.0,
+ "8935": 964259456.0,
+ "8940": 977594496.0,
+ "8945": 981714688.0,
+ "8950": 945977984.0,
+ "8955": 972681344.0,
+ "8960": 973207680.0,
+ "8965": 973566848.0,
+ "8970": 966348032.0,
+ "8975": 937087360.0,
+ "8980": 952802688.0,
+ "8985": 977877056.0,
+ "8990": 967091328.0,
+ "8995": 980577152.0,
+ "9000": 952173120.0,
+ "9005": 950807680.0,
+ "9010": 975107072.0,
+ "9015": 982421376.0,
+ "9020": 958921984.0,
+ "9025": 979619712.0,
+ "9030": 953713792.0,
+ "9035": 968417408.0,
+ "9040": 978206976.0,
+ "9045": 968768384.0,
+ "9050": 983084800.0,
+ "9055": 947871872.0,
+ "9060": 956280064.0,
+ "9065": 970011136.0,
+ "9070": 967853952.0,
+ "9075": 980916864.0,
+ "9080": 952196160.0,
+ "9085": 971256000.0,
+ "9090": 963370752.0,
+ "9095": 968275072.0,
+ "9100": 974460544.0,
+ "9105": 959749504.0,
+ "9110": 947873152.0,
+ "9115": 956488704.0,
+ "9120": 985580608.0,
+ "9125": 963025472.0,
+ "9130": 958505088.0,
+ "9135": 951311040.0,
+ "9140": 967061120.0,
+ "9145": 976830080.0,
+ "9150": 986997120.0,
+ "9155": 976930304.0,
+ "9160": 957827584.0,
+ "9165": 950181760.0,
+ "9170": 988529408.0,
+ "9175": 971285120.0,
+ "9180": 967484544.0,
+ "9185": 955065728.0,
+ "9190": 956905472.0,
+ "9195": 965805952.0,
+ "9200": 968609984.0,
+ "9205": 967243008.0,
+ "9210": 984521600.0,
+ "9215": 931692928.0,
+ "9220": 949709184.0,
+ "9225": 970934144.0,
+ "9230": 971047936.0,
+ "9235": 971696896.0,
+ "9240": 959609856.0,
+ "9245": 963866752.0,
+ "9250": 961353856.0,
+ "9255": 983177472.0,
+ "9260": 979159808.0,
+ "9265": 952472192.0,
+ "9270": 949325440.0,
+ "9275": 978180928.0,
+ "9280": 977863936.0,
+ "9285": 962335808.0,
+ "9290": 978959232.0,
+ "9295": 958535552.0,
+ "9300": 965616128.0,
+ "9305": 969044416.0,
+ "9310": 972999680.0,
+ "9315": 976352576.0,
+ "9320": 947984000.0,
+ "9325": 979480000.0,
+ "9330": 977961408.0,
+ "9335": 975289920.0,
+ "9340": 960088576.0,
+ "9345": 943327488.0,
+ "9350": 952514624.0,
+ "9355": 963388864.0,
+ "9360": 960489728.0,
+ "9365": 983742016.0,
+ "9370": 982422016.0,
+ "9375": 942030720.0,
+ "9380": 983055936.0,
+ "9385": 985392704.0,
+ "9390": 972963648.0,
+ "9395": 978623872.0,
+ "9400": 938136000.0,
+ "9405": 968425920.0,
+ "9410": 981618496.0,
+ "9415": 991820352.0,
+ "9420": 960342272.0,
+ "9425": 956876736.0,
+ "9430": 938878208.0,
+ "9435": 974569728.0,
+ "9440": 959602432.0,
+ "9445": 974199232.0,
+ "9450": 961384512.0,
+ "9455": 945848000.0,
+ "9460": 978146560.0,
+ "9465": 987977856.0,
+ "9470": 963087040.0,
+ "9475": 983735936.0,
+ "9480": 931197056.0,
+ "9485": 987662144.0,
+ "9490": 963667136.0,
+ "9495": 972328000.0,
+ "9500": 982451968.0,
+ "9505": 969941120.0,
+ "9510": 964244992.0,
+ "9515": 957158080.0,
+ "9520": 948370816.0,
+ "9525": 965322688.0,
+ "9530": 958603072.0,
+ "9535": 951367040.0,
+ "9540": 954254144.0,
+ "9545": 979345920.0,
+ "9550": 955704832.0,
+ "9555": 953158784.0,
+ "9560": 958148288.0,
+ "9565": 969842688.0,
+ "9570": 977689152.0,
+ "9575": 959007424.0,
+ "9580": 962802048.0,
+ "9585": 945914624.0,
+ "9590": 948319296.0,
+ "9595": 966821760.0,
+ "9600": 984637504.0,
+ "9605": 985018624.0,
+ "9610": 943615744.0,
+ "9615": 952553984.0,
+ "9620": 980919424.0,
+ "9625": 978670592.0,
+ "9630": 969793152.0,
+ "9635": 974880576.0,
+ "9640": 940403584.0,
+ "9645": 962338560.0,
+ "9650": 971139200.0,
+ "9655": 987188544.0,
+ "9660": 962729856.0,
+ "9665": 949970880.0,
+ "9670": 966321856.0,
+ "9675": 963186560.0,
+ "9680": 964929664.0,
+ "9685": 986634432.0,
+ "9690": 940214400.0,
+ "9695": 950507520.0,
+ "9700": 975627584.0,
+ "9705": 972612928.0,
+ "9710": 966856576.0,
+ "9715": 971143616.0,
+ "9720": 940513856.0,
+ "9725": 965912128.0,
+ "9730": 973877888.0,
+ "9735": 974172608.0,
+ "9740": 971110656.0,
+ "9745": 950961280.0,
+ "9750": 979701376.0,
+ "9755": 970057856.0,
+ "9760": 967854336.0,
+ "9765": 963810240.0,
+ "9770": 952441600.0,
+ "9775": 956632576.0,
+ "9780": 970278784.0,
+ "9785": 958686592.0,
+ "9790": 961127424.0,
+ "9795": 958470784.0,
+ "9800": 949352448.0,
+ "9805": 962490880.0,
+ "9810": 978601856.0,
+ "9815": 977788800.0,
+ "9820": 982581120.0,
+ "9825": 939349376.0,
+ "9830": 969404736.0,
+ "9835": 972829568.0,
+ "9840": 971145920.0,
+ "9845": 967146880.0,
+ "9850": 946664960.0,
+ "9855": 957276608.0,
+ "9860": 987617664.0,
+ "9865": 970002112.0,
+ "9870": 990063104.0,
+ "9875": 957023168.0,
+ "9880": 930919552.0,
+ "9885": 963369984.0,
+ "9890": 972256000.0,
+ "9895": 983795200.0,
+ "9900": 956852096.0,
+ "9905": 938860864.0,
+ "9910": 978612544.0,
+ "9915": 973442944.0,
+ "9920": 944400512.0,
+ "9925": 962460544.0,
+ "9930": 947575808.0,
+ "9935": 960768768.0,
+ "9940": 965991360.0,
+ "9945": 958542464.0,
+ "9950": 963943552.0,
+ "9955": 943303040.0,
+ "9960": 966928320.0,
+ "9965": 983459328.0,
+ "9970": 966433344.0,
+ "9975": 963658880.0,
+ "9980": 980524416.0,
+ "9985": 942412992.0,
+ "9990": 976514688.0,
+ "9995": 982421696.0,
+ "10000": 971874304.0,
+ "10005": 969540544.0,
+ "10010": 944219008.0,
+ "10015": 983176832.0,
+ "10020": 978159744.0,
+ "10025": 979523840.0,
+ "10030": 971365760.0,
+ "10035": 946237248.0,
+ "10040": 949943616.0,
+ "10045": 977963200.0,
+ "10050": 985770112.0,
+ "10055": 990172800.0,
+ "10060": 958786176.0,
+ "10065": 947122944.0,
+ "10070": 966979840.0,
+ "10075": 979154176.0,
+ "10080": 971521536.0,
+ "10085": 974608640.0,
+ "10090": 943429632.0,
+ "10095": 962804608.0,
+ "10100": 971998656.0,
+ "10105": 975960064.0,
+ "10110": 971894144.0,
+ "10115": 948832640.0,
+ "10120": 962580992.0,
+ "10125": 973813568.0,
+ "10130": 980220160.0,
+ "10135": 972049792.0,
+ "10140": 957743104.0,
+ "10145": 933921280.0,
+ "10150": 973387264.0,
+ "10155": 969433024.0,
+ "10160": 962035072.0,
+ "10165": 974995968.0,
+ "10170": 944365376.0,
+ "10175": 978781568.0,
+ "10180": 983454592.0,
+ "10185": 978413888.0,
+ "10190": 955347840.0,
+ "10195": 936952960.0,
+ "10200": 987896064.0,
+ "10205": 972773312.0,
+ "10210": 966531840.0,
+ "10215": 975826432.0,
+ "10220": 948246272.0,
+ "10225": 950423040.0,
+ "10230": 975369984.0,
+ "10235": 953940928.0,
+ "10240": 969345664.0,
+ "10245": 961612224.0,
+ "10250": 936153472.0,
+ "10255": 979515648.0,
+ "10260": 964372224.0,
+ "10265": 967377216.0,
+ "10270": 968830336.0,
+ "10275": 935979648.0,
+ "10280": 969560320.0,
+ "10285": 995739008.0,
+ "10290": 978790400.0,
+ "10295": 981477056.0,
+ "10300": 951900864.0,
+ "10305": 971789952.0,
+ "10310": 960075584.0,
+ "10315": 971224192.0,
+ "10320": 984960512.0,
+ "10325": 983215680.0,
+ "10330": 934961856.0,
+ "10335": 976403328.0,
+ "10340": 957454976.0,
+ "10345": 973779776.0,
+ "10350": 984533888.0,
+ "10355": 941692160.0,
+ "10360": 961896384.0,
+ "10365": 974009280.0,
+ "10370": 980271744.0,
+ "10375": 969911424.0,
+ "10380": 961242688.0,
+ "10385": 955119680.0,
+ "10390": 990167424.0,
+ "10395": 964697344.0,
+ "10400": 960800384.0,
+ "10405": 949794880.0,
+ "10410": 954973696.0,
+ "10415": 975369536.0,
+ "10420": 967024896.0,
+ "10425": 969123584.0,
+ "10430": 964412416.0,
+ "10435": 962679232.0,
+ "10440": 971555456.0,
+ "10445": 972121216.0,
+ "10450": 974775936.0,
+ "10455": 966035712.0,
+ "10460": 948282112.0,
+ "10465": 971189376.0,
+ "10470": 972312448.0,
+ "10475": 978973184.0,
+ "10480": 997033344.0,
+ "10485": 949347968.0,
+ "10490": 934452480.0,
+ "10495": 968799488.0,
+ "10500": 978096896.0,
+ "10505": 958678784.0,
+ "10510": 950661504.0,
+ "10515": 953816064.0,
+ "10520": 971619456.0,
+ "10525": 969523584.0,
+ "10530": 969956416.0,
+ "10535": 985959616.0,
+ "10540": 946655296.0,
+ "10545": 970224256.0,
+ "10550": 969050048.0,
+ "10555": 959127040.0,
+ "10560": 975928192.0,
+ "10565": 961004864.0,
+ "10570": 968760576.0,
+ "10575": 972361920.0,
+ "10580": 960450944.0,
+ "10585": 973098496.0,
+ "10590": 951276864.0,
+ "10595": 955899584.0,
+ "10600": 967134464.0,
+ "10605": 985775616.0,
+ "10610": 965887488.0,
+ "10615": 976377984.0,
+ "10620": 940772224.0,
+ "10625": 964654080.0,
+ "10630": 967061632.0,
+ "10635": 972742464.0,
+ "10640": 974269696.0,
+ "10645": 948248320.0,
+ "10650": 965526656.0,
+ "10655": 985549248.0,
+ "10660": 976126848.0,
+ "10665": 966806592.0,
+ "10670": 954781504.0,
+ "10675": 934158336.0,
+ "10680": 985955456.0,
+ "10685": 990879552.0,
+ "10690": 963724160.0,
+ "10695": 971465600.0,
+ "10700": 949488000.0,
+ "10705": 977846336.0,
+ "10710": 967774464.0,
+ "10715": 966883840.0,
+ "10720": 965741184.0,
+ "10725": 943910976.0,
+ "10730": 979763648.0,
+ "10735": 960526080.0,
+ "10740": 970887680.0,
+ "10745": 984161792.0,
+ "10750": 981011392.0,
+ "10755": 944759616.0,
+ "10760": 969277952.0,
+ "10765": 972565888.0,
+ "10770": 973246528.0,
+ "10775": 958365120.0,
+ "10780": 949088128.0,
+ "10785": 953260736.0,
+ "10790": 969640576.0,
+ "10795": 959977984.0,
+ "10800": 971914112.0,
+ "10805": 950906048.0,
+ "10810": 973705856.0,
+ "10815": 959373952.0,
+ "10820": 970980544.0,
+ "10825": 966693888.0,
+ "10830": 956609088.0,
+ "10835": 962903104.0,
+ "10840": 970530304.0,
+ "10845": 963528000.0,
+ "10850": 957743936.0,
+ "10855": 967336832.0,
+ "10860": 950516224.0,
+ "10865": 963973696.0,
+ "10870": 983059712.0,
+ "10875": 981940800.0,
+ "10880": 958105920.0,
+ "10885": 954574784.0,
+ "10890": 972692032.0,
+ "10895": 973026752.0,
+ "10900": 970016512.0,
+ "10905": 964408192.0,
+ "10910": 938543424.0,
+ "10915": 960121856.0,
+ "10920": 983112512.0,
+ "10925": 969525632.0,
+ "10930": 968056832.0,
+ "10935": 962330112.0,
+ "10940": 953695936.0,
+ "10945": 964656000.0,
+ "10950": 971912704.0,
+ "10955": 966274688.0,
+ "10960": 971239296.0,
+ "10965": 965974272.0,
+ "10970": 983152896.0,
+ "10975": 965500992.0,
+ "10980": 974347648.0,
+ "10985": 986153152.0,
+ "10990": 950734720.0,
+ "10995": 962957632.0,
+ "11000": 984645632.0,
+ "11005": 978005888.0,
+ "11010": 971084736.0,
+ "11015": 969540928.0,
+ "11020": 947443584.0,
+ "11025": 959759232.0,
+ "11030": 977565888.0,
+ "11035": 975462656.0,
+ "11040": 985919232.0,
+ "11045": 956191168.0,
+ "11050": 973025536.0,
+ "11055": 974123904.0,
+ "11060": 961535168.0,
+ "11065": 984926080.0,
+ "11070": 949335168.0,
+ "11075": 976048448.0,
+ "11080": 971612864.0,
+ "11085": 966872192.0,
+ "11090": 975660032.0,
+ "11095": 946314240.0,
+ "11100": 965521216.0,
+ "11105": 973891520.0,
+ "11110": 980607680.0,
+ "11115": 967578496.0,
+ "11120": 956988800.0,
+ "11125": 956350720.0,
+ "11130": 974764480.0,
+ "11135": 978995136.0,
+ "11140": 964107200.0,
+ "11145": 965760512.0,
+ "11150": 935356672.0,
+ "11155": 975383808.0,
+ "11160": 983591616.0,
+ "11165": 982103040.0,
+ "11170": 977214464.0,
+ "11175": 957652544.0,
+ "11180": 961923264.0,
+ "11185": 971484736.0,
+ "11190": 979394880.0,
+ "11195": 984898496.0,
+ "11200": 982384512.0,
+ "11205": 941871616.0,
+ "11210": 984210368.0,
+ "11215": 966940032.0,
+ "11220": 982933440.0,
+ "11225": 961345216.0,
+ "11230": 952794112.0,
+ "11235": 981397952.0,
+ "11240": 977288064.0,
+ "11245": 965789632.0,
+ "11250": 969070912.0,
+ "11255": 960030016.0,
+ "11260": 979480128.0,
+ "11265": 963433024.0,
+ "11270": 981152768.0,
+ "11275": 968032896.0,
+ "11280": 955902592.0,
+ "11285": 953195008.0,
+ "11290": 956118656.0,
+ "11295": 967631296.0,
+ "11300": 961921920.0,
+ "11305": 958093568.0,
+ "11310": 946019264.0,
+ "11315": 982887104.0,
+ "11320": 963966656.0,
+ "11325": 980683776.0,
+ "11330": 975045376.0,
+ "11335": 951385408.0,
+ "11340": 970287360.0,
+ "11345": 969715584.0,
+ "11350": 981270144.0,
+ "11355": 981865920.0,
+ "11360": 940830400.0,
+ "11365": 970416256.0,
+ "11370": 978532352.0,
+ "11375": 975066432.0,
+ "11380": 968255104.0,
+ "11385": 958610240.0,
+ "11390": 937860224.0,
+ "11395": 976816384.0,
+ "11400": 973094784.0,
+ "11405": 960792256.0,
+ "11410": 965934016.0,
+ "11415": 929101632.0,
+ "11420": 964656704.0,
+ "11425": 980900288.0,
+ "11430": 978352256.0,
+ "11435": 969930496.0,
+ "11440": 944960256.0,
+ "11445": 974945024.0,
+ "11450": 984193536.0,
+ "11455": 970920192.0,
+ "11460": 964788032.0,
+ "11465": 959779840.0,
+ "11470": 954796800.0,
+ "11475": 972394944.0,
+ "11480": 956111104.0,
+ "11485": 976547776.0,
+ "11490": 986096896.0,
+ "11495": 958885632.0,
+ "11500": 970159296.0,
+ "11505": 963629376.0,
+ "11510": 976196288.0,
+ "11515": 977754432.0,
+ "11520": 953592832.0,
+ "11525": 975656448.0,
+ "11530": 976241664.0,
+ "11535": 979771648.0,
+ "11540": 974225728.0,
+ "11545": 953371520.0,
+ "11550": 952960320.0,
+ "11555": 981597184.0,
+ "11560": 984420032.0,
+ "11565": 965007424.0,
+ "11570": 966005824.0,
+ "11575": 951323968.0,
+ "11580": 975949760.0,
+ "11585": 976971456.0,
+ "11590": 969004672.0,
+ "11595": 976372352.0,
+ "11600": 946117504.0,
+ "11605": 972976512.0,
+ "11610": 982099648.0,
+ "11615": 971848576.0,
+ "11620": 968932096.0,
+ "11625": 948544000.0,
+ "11630": 937082240.0,
+ "11635": 973246784.0,
+ "11640": 980778496.0,
+ "11645": 979550272.0,
+ "11650": 971571200.0,
+ "11655": 955918720.0,
+ "11660": 980377984.0,
+ "11665": 957771840.0,
+ "11670": 982370944.0,
+ "11675": 972031296.0,
+ "11680": 956038336.0,
+ "11685": 982695680.0,
+ "11690": 968206848.0,
+ "11695": 967654080.0,
+ "11700": 973386624.0,
+ "11705": 955828160.0,
+ "11710": 964362176.0,
+ "11715": 982948608.0,
+ "11720": 983559424.0,
+ "11725": 965129216.0,
+ "11730": 955433728.0,
+ "11735": 942620864.0,
+ "11740": 973654592.0,
+ "11745": 970718144.0,
+ "11750": 961559872.0,
+ "11755": 962806464.0,
+ "11760": 949884096.0,
+ "11765": 983433536.0,
+ "11770": 984378496.0,
+ "11775": 975203712.0,
+ "11780": 985170496.0,
+ "11785": 946901760.0,
+ "11790": 972012608.0,
+ "11795": 970299840.0,
+ "11800": 972807616.0,
+ "11805": 986521728.0,
+ "11810": 966950464.0,
+ "11815": 955589632.0,
+ "11820": 973517952.0,
+ "11825": 970349952.0,
+ "11830": 974794944.0,
+ "11835": 961566464.0,
+ "11840": 944378944.0,
+ "11845": 980594432.0,
+ "11850": 974513472.0,
+ "11855": 977893120.0,
+ "11860": 971389120.0,
+ "11865": 938261120.0,
+ "11870": 939964992.0,
+ "11875": 989777216.0,
+ "11880": 971971968.0,
+ "11885": 962679040.0,
+ "11890": 970033472.0,
+ "11895": 965194944.0,
+ "11900": 979072192.0,
+ "11905": 961049536.0,
+ "11910": 983430528.0,
+ "11915": 989806592.0,
+ "11920": 944166464.0,
+ "11925": 993816768.0,
+ "11930": 964211648.0,
+ "11935": 963032704.0,
+ "11940": 976558272.0,
+ "11945": 944637120.0,
+ "11950": 977490048.0,
+ "11955": 978905216.0,
+ "11960": 972040448.0,
+ "11965": 975989504.0,
+ "11970": 963020480.0,
+ "11975": 962954624.0,
+ "11980": 977423104.0,
+ "11985": 953010752.0,
+ "11990": 968248320.0,
+ "11995": 964736704.0,
+ "12000": 957960448.0,
+ "12005": 973929920.0,
+ "12010": 978787456.0,
+ "12015": 971681344.0,
+ "12020": 972846592.0,
+ "12025": 934010752.0,
+ "12030": 968579776.0,
+ "12035": 983657024.0,
+ "12040": 977184192.0,
+ "12045": 981385152.0,
+ "12050": 930805504.0,
+ "12055": 938148672.0,
+ "12060": 973965760.0,
+ "12065": 965572288.0,
+ "12070": 967819456.0,
+ "12075": 949533120.0,
+ "12080": 952959104.0,
+ "12085": 972328000.0,
+ "12090": 963512384.0,
+ "12095": 963268736.0,
+ "12100": 976624896.0,
+ "12105": 950333184.0,
+ "12110": 971577536.0,
+ "12115": 967854912.0,
+ "12120": 986356032.0,
+ "12125": 980610816.0,
+ "12130": 941366528.0,
+ "12135": 955427904.0,
+ "12140": 975803584.0,
+ "12145": 979451008.0,
+ "12150": 979544768.0,
+ "12155": 961569664.0,
+ "12160": 946185792.0,
+ "12165": 968329536.0,
+ "12170": 963709248.0,
+ "12175": 967258816.0,
+ "12180": 974632384.0,
+ "12185": 952415552.0,
+ "12190": 987879168.0,
+ "12195": 970233024.0,
+ "12200": 964549376.0,
+ "12205": 968334528.0,
+ "12210": 938908800.0,
+ "12215": 996464640.0,
+ "12220": 969469248.0,
+ "12225": 979370624.0,
+ "12230": 979961792.0,
+ "12235": 950000896.0,
+ "12240": 963697088.0,
+ "12245": 965775680.0,
+ "12250": 976681984.0,
+ "12255": 968433280.0,
+ "12260": 983664320.0,
+ "12265": 931892032.0,
+ "12270": 966171776.0,
+ "12275": 979552512.0,
+ "12280": 977688960.0,
+ "12285": 970429632.0,
+ "12290": 929315456.0,
+ "12295": 976772032.0,
+ "12300": 986049088.0,
+ "12305": 969584256.0,
+ "12310": 986005376.0,
+ "12315": 936027456.0,
+ "12320": 957642496.0,
+ "12325": 966202240.0,
+ "12330": 968292608.0,
+ "12335": 963687360.0,
+ "12340": 957247872.0,
+ "12345": 944271424.0,
+ "12350": 966649600.0,
+ "12355": 975486080.0,
+ "12360": 978634688.0,
+ "12365": 964201856.0,
+ "12370": 948778368.0,
+ "12375": 963602944.0,
+ "12380": 964688384.0,
+ "12385": 972793664.0,
+ "12390": 961485760.0,
+ "12395": 961271168.0,
+ "12400": 975342592.0,
+ "12405": 976299904.0,
+ "12410": 953299008.0,
+ "12415": 962706688.0,
+ "12420": 944036736.0,
+ "12425": 948974592.0,
+ "12430": 972069376.0,
+ "12435": 968670592.0,
+ "12440": 961658048.0,
+ "12445": 951909632.0,
+ "12450": 947131008.0,
+ "12455": 980976832.0,
+ "12460": 973901696.0,
+ "12465": 954326144.0,
+ "12470": 980931712.0,
+ "12475": 958062720.0,
+ "12480": 966837376.0,
+ "12485": 978056896.0,
+ "12490": 973895936.0,
+ "12495": 969407936.0,
+ "12500": 961141120.0,
+ "12505": 943213824.0,
+ "12510": 960766080.0,
+ "12515": 969638080.0,
+ "12520": 973971200.0,
+ "12525": 972228096.0,
+ "12530": 944434304.0,
+ "12535": 976337728.0,
+ "12540": 965435392.0,
+ "12545": 971699456.0,
+ "12550": 969423872.0,
+ "12555": 940893248.0,
+ "12560": 963999680.0,
+ "12565": 947261568.0,
+ "12570": 974171520.0,
+ "12575": 962966784.0,
+ "12580": 957904960.0,
+ "12585": 964310592.0,
+ "12590": 965663040.0,
+ "12595": 978496832.0,
+ "12600": 981415744.0,
+ "12605": 949112192.0,
+ "12610": 937705728.0,
+ "12615": 962690944.0,
+ "12620": 960966784.0,
+ "12625": 966271936.0,
+ "12630": 970736512.0,
+ "12635": 962160064.0,
+ "12640": 978256960.0,
+ "12645": 969491328.0,
+ "12650": 970048960.0,
+ "12655": 964006464.0,
+ "12660": 932254144.0,
+ "12665": 956728320.0,
+ "12670": 986153664.0,
+ "12675": 965503808.0,
+ "12680": 960963200.0,
+ "12685": 951228032.0,
+ "12690": 945266304.0,
+ "12695": 978347904.0,
+ "12700": 985300224.0,
+ "12705": 958770112.0,
+ "12710": 968391296.0,
+ "12715": 956547968.0,
+ "12720": "nan",
+ "12725": "nan",
+ "12730": "nan",
+ "12735": "nan",
+ "12740": "nan",
+ "12745": "nan",
+ "12750": "nan",
+ "12755": "nan",
+ "12760": "nan",
+ "12765": "nan",
+ "12770": "nan",
+ "12775": "nan",
+ "12780": "nan",
+ "12785": "nan",
+ "12790": "nan",
+ "12795": "nan",
+ "12800": "nan",
+ "12805": "nan",
+ "12810": "nan",
+ "12815": "nan",
+ "12820": "nan",
+ "12825": "nan",
+ "12830": "nan",
+ "12835": "nan",
+ "12840": "nan",
+ "12845": "nan",
+ "12850": "nan",
+ "12855": "nan",
+ "12860": "nan",
+ "12865": "nan",
+ "12870": "nan",
+ "12875": "nan",
+ "12880": "nan",
+ "12885": "nan",
+ "12890": "nan",
+ "12895": "nan",
+ "12900": "nan",
+ "12905": "nan",
+ "12910": "nan",
+ "12915": "nan",
+ "12920": "nan",
+ "12925": "nan",
+ "12930": "nan",
+ "12935": "nan",
+ "12940": "nan",
+ "12945": "nan",
+ "12950": "nan",
+ "12955": "nan",
+ "12960": "nan",
+ "12965": "nan",
+ "12970": "nan",
+ "12975": "nan",
+ "12980": "nan",
+ "12985": "nan",
+ "12990": "nan",
+ "12995": "nan",
+ "13000": "nan"
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": 13268858880.0,
+ "5": 13268858880.0,
+ "10": 13268858880.0,
+ "15": 13268858880.0,
+ "20": 13268858880.0,
+ "25": 13268858880.0,
+ "30": 13268858880.0,
+ "35": 13268858880.0,
+ "40": 13268858880.0,
+ "45": 13268858880.0,
+ "50": 13268858880.0,
+ "55": 13268858880.0,
+ "60": 13268858880.0,
+ "65": 13268858880.0,
+ "70": 13268858880.0,
+ "75": 13268858880.0,
+ "80": 13268858880.0,
+ "85": 13268858880.0,
+ "90": 13268858880.0,
+ "95": 13268858880.0,
+ "100": 13268858880.0,
+ "105": 13268858880.0,
+ "110": 13268858880.0,
+ "115": 13268858880.0,
+ "120": 13268858880.0,
+ "125": 13268858880.0,
+ "130": 13268858880.0,
+ "135": 13268858880.0,
+ "140": 13268858880.0,
+ "145": 13268858880.0,
+ "150": 13268858880.0,
+ "155": 13268858880.0,
+ "160": 13268858880.0,
+ "165": 13268858880.0,
+ "170": 13268858880.0,
+ "175": 13268858880.0,
+ "180": 13268858880.0,
+ "185": 13268858880.0,
+ "190": 13268858880.0,
+ "195": 13268858880.0,
+ "200": 13268858880.0,
+ "205": 13268858880.0,
+ "210": 13268858880.0,
+ "215": 13268858880.0,
+ "220": 13268858880.0,
+ "225": 13268858880.0,
+ "230": 13268858880.0,
+ "235": 13268858880.0,
+ "240": 13268858880.0,
+ "245": 13268858880.0,
+ "250": 13268858880.0,
+ "255": 13268858880.0,
+ "260": 13268858880.0,
+ "265": 13268858880.0,
+ "270": 13268858880.0,
+ "275": 13268858880.0,
+ "280": 13268858880.0,
+ "285": 13268858880.0,
+ "290": 13268858880.0,
+ "295": 13268858880.0,
+ "300": 13268858880.0,
+ "305": 13268858880.0,
+ "310": 13268858880.0,
+ "315": 13268858880.0,
+ "320": 13268858880.0,
+ "325": 13268858880.0,
+ "330": 13268858880.0,
+ "335": 13268858880.0,
+ "340": 13268858880.0,
+ "345": 13268858880.0,
+ "350": 13268858880.0,
+ "355": 13268858880.0,
+ "360": 13268858880.0,
+ "365": 13268858880.0,
+ "370": 13268858880.0,
+ "375": 13268858880.0,
+ "380": 13268858880.0,
+ "385": 13268858880.0,
+ "390": 13268858880.0,
+ "395": 13268858880.0,
+ "400": 13268858880.0,
+ "405": 13268858880.0,
+ "410": 13268858880.0,
+ "415": 13268858880.0,
+ "420": 13268858880.0,
+ "425": 13268858880.0,
+ "430": 13268858880.0,
+ "435": 13268858880.0,
+ "440": 13268858880.0,
+ "445": 13268858880.0,
+ "450": 13268858880.0,
+ "455": 13268858880.0,
+ "460": 13268858880.0,
+ "465": 13268858880.0,
+ "470": 13268858880.0,
+ "475": 13268858880.0,
+ "480": 13268858880.0,
+ "485": 13268858880.0,
+ "490": 13268858880.0,
+ "495": 13268858880.0,
+ "500": 13268858880.0,
+ "505": 13268858880.0,
+ "510": 13268858880.0,
+ "515": 13268858880.0,
+ "520": 13268858880.0,
+ "525": 13268858880.0,
+ "530": 13268858880.0,
+ "535": 13268858880.0,
+ "540": 13268858880.0,
+ "545": 13268858880.0,
+ "550": 13268858880.0,
+ "555": 13268858880.0,
+ "560": 13268858880.0,
+ "565": 13268858880.0,
+ "570": 13268858880.0,
+ "575": 13268858880.0,
+ "580": 13268858880.0,
+ "585": 13268858880.0,
+ "590": 13268858880.0,
+ "595": 13268858880.0,
+ "600": 13268858880.0,
+ "605": 13268858880.0,
+ "610": 13268858880.0,
+ "615": 13268858880.0,
+ "620": 13268858880.0,
+ "625": 13268858880.0,
+ "630": 13268858880.0,
+ "635": 13268858880.0,
+ "640": 13268858880.0,
+ "645": 13268858880.0,
+ "650": 13268858880.0,
+ "655": 13268858880.0,
+ "660": 13268858880.0,
+ "665": 13268858880.0,
+ "670": 13268858880.0,
+ "675": 13268858880.0,
+ "680": 13268858880.0,
+ "685": 13268858880.0,
+ "690": 13268858880.0,
+ "695": 13268858880.0,
+ "700": 13268858880.0,
+ "705": 13268858880.0,
+ "710": 13268858880.0,
+ "715": 13268858880.0,
+ "720": 13268858880.0,
+ "725": 13268858880.0,
+ "730": 13268858880.0,
+ "735": 13268858880.0,
+ "740": 13268858880.0,
+ "745": 13268858880.0,
+ "750": 13268858880.0,
+ "755": 13268858880.0,
+ "760": 13268858880.0,
+ "765": 13268858880.0,
+ "770": 13268858880.0,
+ "775": 13268858880.0,
+ "780": 13268858880.0,
+ "785": 13268858880.0,
+ "790": 13268858880.0,
+ "795": 13268858880.0,
+ "800": 13268858880.0,
+ "805": 13268858880.0,
+ "810": 13268858880.0,
+ "815": 13268858880.0,
+ "820": 13268858880.0,
+ "825": 13268858880.0,
+ "830": 13268858880.0,
+ "835": 13268858880.0,
+ "840": 13268858880.0,
+ "845": 13268858880.0,
+ "850": 13268858880.0,
+ "855": 13268858880.0,
+ "860": 13268858880.0,
+ "865": 13268858880.0,
+ "870": 13268858880.0,
+ "875": 13268858880.0,
+ "880": 13268858880.0,
+ "885": 13268858880.0,
+ "890": 13268858880.0,
+ "895": 13268858880.0,
+ "900": 13268858880.0,
+ "905": 13268858880.0,
+ "910": 13268858880.0,
+ "915": 13268858880.0,
+ "920": 13268858880.0,
+ "925": 13268858880.0,
+ "930": 13268858880.0,
+ "935": 13268858880.0,
+ "940": 13268858880.0,
+ "945": 13268858880.0,
+ "950": 13268858880.0,
+ "955": 13268858880.0,
+ "960": 13268858880.0,
+ "965": 13268858880.0,
+ "970": 13268858880.0,
+ "975": 13268858880.0,
+ "980": 13268858880.0,
+ "985": 13268858880.0,
+ "990": 13268858880.0,
+ "995": 13268858880.0,
+ "1000": 13268858880.0,
+ "1005": 13268858880.0,
+ "1010": 13268858880.0,
+ "1015": 13268858880.0,
+ "1020": 13268858880.0,
+ "1025": 13268858880.0,
+ "1030": 13268858880.0,
+ "1035": 13268858880.0,
+ "1040": 13268858880.0,
+ "1045": 13268858880.0,
+ "1050": 13268858880.0,
+ "1055": 13268858880.0,
+ "1060": 13268858880.0,
+ "1065": 13268858880.0,
+ "1070": 13268858880.0,
+ "1075": 13268858880.0,
+ "1080": 13268858880.0,
+ "1085": 13268858880.0,
+ "1090": 13268858880.0,
+ "1095": 13268858880.0,
+ "1100": 13268858880.0,
+ "1105": 13268858880.0,
+ "1110": 13268858880.0,
+ "1115": 13268858880.0,
+ "1120": 13268858880.0,
+ "1125": 13268858880.0,
+ "1130": 13268858880.0,
+ "1135": 13268858880.0,
+ "1140": 13268858880.0,
+ "1145": 13268858880.0,
+ "1150": 13268858880.0,
+ "1155": 13268858880.0,
+ "1160": 13268858880.0,
+ "1165": 13268858880.0,
+ "1170": 13268858880.0,
+ "1175": 13268858880.0,
+ "1180": 13268858880.0,
+ "1185": 13268858880.0,
+ "1190": 13268858880.0,
+ "1195": 13268858880.0,
+ "1200": 13268858880.0,
+ "1205": 13268858880.0,
+ "1210": 13268858880.0,
+ "1215": 13268858880.0,
+ "1220": 13268858880.0,
+ "1225": 13268858880.0,
+ "1230": 13268858880.0,
+ "1235": 13268858880.0,
+ "1240": 13268858880.0,
+ "1245": 13268858880.0,
+ "1250": 13268858880.0,
+ "1255": 13268858880.0,
+ "1260": 13268858880.0,
+ "1265": 13268858880.0,
+ "1270": 13268858880.0,
+ "1275": 13268858880.0,
+ "1280": 13268858880.0,
+ "1285": 13268858880.0,
+ "1290": 13268858880.0,
+ "1295": 13268858880.0,
+ "1300": 13268858880.0,
+ "1305": 13268858880.0,
+ "1310": 13268858880.0,
+ "1315": 13268858880.0,
+ "1320": 13268858880.0,
+ "1325": 13268858880.0,
+ "1330": 13268858880.0,
+ "1335": 13268858880.0,
+ "1340": 13268858880.0,
+ "1345": 13268858880.0,
+ "1350": 13268858880.0,
+ "1355": 13268858880.0,
+ "1360": 13268858880.0,
+ "1365": 13268858880.0,
+ "1370": 13268858880.0,
+ "1375": 13268858880.0,
+ "1380": 13268858880.0,
+ "1385": 13268858880.0,
+ "1390": 13268858880.0,
+ "1395": 13268858880.0,
+ "1400": 13268858880.0,
+ "1405": 13268858880.0,
+ "1410": 13268858880.0,
+ "1415": 13268858880.0,
+ "1420": 13268858880.0,
+ "1425": 13268858880.0,
+ "1430": 13268858880.0,
+ "1435": 13268858880.0,
+ "1440": 13268858880.0,
+ "1445": 13268858880.0,
+ "1450": 13268858880.0,
+ "1455": 13268858880.0,
+ "1460": 13268858880.0,
+ "1465": 13268858880.0,
+ "1470": 13268858880.0,
+ "1475": 13268858880.0,
+ "1480": 13268858880.0,
+ "1485": 13268858880.0,
+ "1490": 13268858880.0,
+ "1495": 13268858880.0,
+ "1500": 13268858880.0,
+ "1505": 13268858880.0,
+ "1510": 13268858880.0,
+ "1515": 13268858880.0,
+ "1520": 13268858880.0,
+ "1525": 13268858880.0,
+ "1530": 13268858880.0,
+ "1535": 13268858880.0,
+ "1540": 13268858880.0,
+ "1545": 13268858880.0,
+ "1550": 13268858880.0,
+ "1555": 13268858880.0,
+ "1560": 13268858880.0,
+ "1565": 13268858880.0,
+ "1570": 13268858880.0,
+ "1575": 13268858880.0,
+ "1580": 13268858880.0,
+ "1585": 13268858880.0,
+ "1590": 13268858880.0,
+ "1595": 13268858880.0,
+ "1600": 13268858880.0,
+ "1605": 13268858880.0,
+ "1610": 13268858880.0,
+ "1615": 13268858880.0,
+ "1620": 13268858880.0,
+ "1625": 13268858880.0,
+ "1630": 13268858880.0,
+ "1635": 13268858880.0,
+ "1640": 13268858880.0,
+ "1645": 13268858880.0,
+ "1650": 13268858880.0,
+ "1655": 13268858880.0,
+ "1660": 13268858880.0,
+ "1665": 13268858880.0,
+ "1670": 13268858880.0,
+ "1675": 13268858880.0,
+ "1680": 13268858880.0,
+ "1685": 13268858880.0,
+ "1690": 13268858880.0,
+ "1695": 13268858880.0,
+ "1700": 13268858880.0,
+ "1705": 13268858880.0,
+ "1710": 13268858880.0,
+ "1715": 13268858880.0,
+ "1720": 13268858880.0,
+ "1725": 13268858880.0,
+ "1730": 13268858880.0,
+ "1735": 13268858880.0,
+ "1740": 13268858880.0,
+ "1745": 13268858880.0,
+ "1750": 13268858880.0,
+ "1755": 13268858880.0,
+ "1760": 13268858880.0,
+ "1765": 13268858880.0,
+ "1770": 13268858880.0,
+ "1775": 13268858880.0,
+ "1780": 13268858880.0,
+ "1785": 13268858880.0,
+ "1790": 13268858880.0,
+ "1795": 13268858880.0,
+ "1800": 13268858880.0,
+ "1805": 13268858880.0,
+ "1810": 13268858880.0,
+ "1815": 13268858880.0,
+ "1820": 13268858880.0,
+ "1825": 13268858880.0,
+ "1830": 13268858880.0,
+ "1835": 13268858880.0,
+ "1840": 13268858880.0,
+ "1845": 13268858880.0,
+ "1850": 13268858880.0,
+ "1855": 13268858880.0,
+ "1860": 13268858880.0,
+ "1865": 13268858880.0,
+ "1870": 13268858880.0,
+ "1875": 13268858880.0,
+ "1880": 13268858880.0,
+ "1885": 13268858880.0,
+ "1890": 13268858880.0,
+ "1895": 13268858880.0,
+ "1900": 13268858880.0,
+ "1905": 13268858880.0,
+ "1910": 13268858880.0,
+ "1915": 13268858880.0,
+ "1920": 13268858880.0,
+ "1925": 13268858880.0,
+ "1930": 13268858880.0,
+ "1935": 13268858880.0,
+ "1940": 13268858880.0,
+ "1945": 13268858880.0,
+ "1950": 13268858880.0,
+ "1955": 13268858880.0,
+ "1960": 13268858880.0,
+ "1965": 13268858880.0,
+ "1970": 13268858880.0,
+ "1975": 13268858880.0,
+ "1980": 13268858880.0,
+ "1985": 13268858880.0,
+ "1990": 13268858880.0,
+ "1995": 13268858880.0,
+ "2000": 13268858880.0,
+ "2005": 13268858880.0,
+ "2010": 13268858880.0,
+ "2015": 13268858880.0,
+ "2020": 13268858880.0,
+ "2025": 13268858880.0,
+ "2030": 13268858880.0,
+ "2035": 13268858880.0,
+ "2040": 13268858880.0,
+ "2045": 13268858880.0,
+ "2050": 13268858880.0,
+ "2055": 13268858880.0,
+ "2060": 13268858880.0,
+ "2065": 13268858880.0,
+ "2070": 13268858880.0,
+ "2075": 13268858880.0,
+ "2080": 13268858880.0,
+ "2085": 13268858880.0,
+ "2090": 13268858880.0,
+ "2095": 13268858880.0,
+ "2100": 13268858880.0,
+ "2105": 13268858880.0,
+ "2110": 13268858880.0,
+ "2115": 13268858880.0,
+ "2120": 13268858880.0,
+ "2125": 13268858880.0,
+ "2130": 13268858880.0,
+ "2135": 13268858880.0,
+ "2140": 13268858880.0,
+ "2145": 13268858880.0,
+ "2150": 13268858880.0,
+ "2155": 13268858880.0,
+ "2160": 13268858880.0,
+ "2165": 13268858880.0,
+ "2170": 13268858880.0,
+ "2175": 13268858880.0,
+ "2180": 13268858880.0,
+ "2185": 13268858880.0,
+ "2190": 13268858880.0,
+ "2195": 13268858880.0,
+ "2200": 13268858880.0,
+ "2205": 13268858880.0,
+ "2210": 13268858880.0,
+ "2215": 13268858880.0,
+ "2220": 13268858880.0,
+ "2225": 13268858880.0,
+ "2230": 13268858880.0,
+ "2235": 13268858880.0,
+ "2240": 13268858880.0,
+ "2245": 13268858880.0,
+ "2250": 13268858880.0,
+ "2255": 13268858880.0,
+ "2260": 13268858880.0,
+ "2265": 13268858880.0,
+ "2270": 13268858880.0,
+ "2275": 13268858880.0,
+ "2280": 13268858880.0,
+ "2285": 13268858880.0,
+ "2290": 13268858880.0,
+ "2295": 13268858880.0,
+ "2300": 13268858880.0,
+ "2305": 13268858880.0,
+ "2310": 13268858880.0,
+ "2315": 13268858880.0,
+ "2320": 13268858880.0,
+ "2325": 13268858880.0,
+ "2330": 13268858880.0,
+ "2335": 13268858880.0,
+ "2340": 13268858880.0,
+ "2345": 13268858880.0,
+ "2350": 13268858880.0,
+ "2355": 13268858880.0,
+ "2360": 13268858880.0,
+ "2365": 13268858880.0,
+ "2370": 13268858880.0,
+ "2375": 13268858880.0,
+ "2380": 13268858880.0,
+ "2385": 13268858880.0,
+ "2390": 13268858880.0,
+ "2395": 13268858880.0,
+ "2400": 13268858880.0,
+ "2405": 13268858880.0,
+ "2410": 13268858880.0,
+ "2415": 13268858880.0,
+ "2420": 13268858880.0,
+ "2425": 13268858880.0,
+ "2430": 13268858880.0,
+ "2435": 13268858880.0,
+ "2440": 13268858880.0,
+ "2445": 13268858880.0,
+ "2450": 13268858880.0,
+ "2455": 13268858880.0,
+ "2460": 13268858880.0,
+ "2465": 13268858880.0,
+ "2470": 13268858880.0,
+ "2475": 13268858880.0,
+ "2480": 13268858880.0,
+ "2485": 13268858880.0,
+ "2490": 13268858880.0,
+ "2495": 13268858880.0,
+ "2500": 13268858880.0,
+ "2505": 13268858880.0,
+ "2510": 13268858880.0,
+ "2515": 13268858880.0,
+ "2520": 13268858880.0,
+ "2525": 13268858880.0,
+ "2530": 13268858880.0,
+ "2535": 13268858880.0,
+ "2540": 13268858880.0,
+ "2545": 13268858880.0,
+ "2550": 13268858880.0,
+ "2555": 13268858880.0,
+ "2560": 13268858880.0,
+ "2565": 13268858880.0,
+ "2570": 13268858880.0,
+ "2575": 13268858880.0,
+ "2580": 13268858880.0,
+ "2585": 13268858880.0,
+ "2590": 13268858880.0,
+ "2595": 13268858880.0,
+ "2600": 13268858880.0,
+ "2605": 13268858880.0,
+ "2610": 13268858880.0,
+ "2615": 13268858880.0,
+ "2620": 13268858880.0,
+ "2625": 13268858880.0,
+ "2630": 13268858880.0,
+ "2635": 13268858880.0,
+ "2640": 13268858880.0,
+ "2645": 13268858880.0,
+ "2650": 13268858880.0,
+ "2655": 13268858880.0,
+ "2660": 13268858880.0,
+ "2665": 13268858880.0,
+ "2670": 13268858880.0,
+ "2675": 13268858880.0,
+ "2680": 13268858880.0,
+ "2685": 13268858880.0,
+ "2690": 13268858880.0,
+ "2695": 13268858880.0,
+ "2700": 13268858880.0,
+ "2705": 13268858880.0,
+ "2710": 13268858880.0,
+ "2715": 13268858880.0,
+ "2720": 13268858880.0,
+ "2725": 13268858880.0,
+ "2730": 13268858880.0,
+ "2735": 13268858880.0,
+ "2740": 13268858880.0,
+ "2745": 13268858880.0,
+ "2750": 13268858880.0,
+ "2755": 13268858880.0,
+ "2760": 13268858880.0,
+ "2765": 13268858880.0,
+ "2770": 13268858880.0,
+ "2775": 13268858880.0,
+ "2780": 13268858880.0,
+ "2785": 13268858880.0,
+ "2790": 13268858880.0,
+ "2795": 13268858880.0,
+ "2800": 13268858880.0,
+ "2805": 13268858880.0,
+ "2810": 13268858880.0,
+ "2815": 13268858880.0,
+ "2820": 13268858880.0,
+ "2825": 13268858880.0,
+ "2830": 13268858880.0,
+ "2835": 13268858880.0,
+ "2840": 13268858880.0,
+ "2845": 13268858880.0,
+ "2850": 13268858880.0,
+ "2855": 13268858880.0,
+ "2860": 13268858880.0,
+ "2865": 13268858880.0,
+ "2870": 13268858880.0,
+ "2875": 13268858880.0,
+ "2880": 13268858880.0,
+ "2885": 13268858880.0,
+ "2890": 13268858880.0,
+ "2895": 13268858880.0,
+ "2900": 13268858880.0,
+ "2905": 13268858880.0,
+ "2910": 13268858880.0,
+ "2915": 13268858880.0,
+ "2920": 13268858880.0,
+ "2925": 13268858880.0,
+ "2930": 13268858880.0,
+ "2935": 13268858880.0,
+ "2940": 13268858880.0,
+ "2945": 13268858880.0,
+ "2950": 13268858880.0,
+ "2955": 13268858880.0,
+ "2960": 13268858880.0,
+ "2965": 13268858880.0,
+ "2970": 13268858880.0,
+ "2975": 13268858880.0,
+ "2980": 13268858880.0,
+ "2985": 13268858880.0,
+ "2990": 13268858880.0,
+ "2995": 13268858880.0,
+ "3000": 13268858880.0,
+ "3005": 13268858880.0,
+ "3010": 13268858880.0,
+ "3015": 13268858880.0,
+ "3020": 13268858880.0,
+ "3025": 13268858880.0,
+ "3030": 13268858880.0,
+ "3035": 13268858880.0,
+ "3040": 13268858880.0,
+ "3045": 13268858880.0,
+ "3050": 13268858880.0,
+ "3055": 13268858880.0,
+ "3060": 13268858880.0,
+ "3065": 13268858880.0,
+ "3070": 13268858880.0,
+ "3075": 13268858880.0,
+ "3080": 13268858880.0,
+ "3085": 13268858880.0,
+ "3090": 13268858880.0,
+ "3095": 13268858880.0,
+ "3100": 13268858880.0,
+ "3105": 13268858880.0,
+ "3110": 13268858880.0,
+ "3115": 13268858880.0,
+ "3120": 13268858880.0,
+ "3125": 13268858880.0,
+ "3130": 13268858880.0,
+ "3135": 13268858880.0,
+ "3140": 13268858880.0,
+ "3145": 13268858880.0,
+ "3150": 13268858880.0,
+ "3155": 13268858880.0,
+ "3160": 13268858880.0,
+ "3165": 13268858880.0,
+ "3170": 13268858880.0,
+ "3175": 13268858880.0,
+ "3180": 13268858880.0,
+ "3185": 13268858880.0,
+ "3190": 13268858880.0,
+ "3195": 13268858880.0,
+ "3200": 13268858880.0,
+ "3205": 13268858880.0,
+ "3210": 13268858880.0,
+ "3215": 13268858880.0,
+ "3220": 13268858880.0,
+ "3225": 13268858880.0,
+ "3230": 13268858880.0,
+ "3235": 13268858880.0,
+ "3240": 13268858880.0,
+ "3245": 13268858880.0,
+ "3250": 13268858880.0,
+ "3255": 13268858880.0,
+ "3260": 13268858880.0,
+ "3265": 13268858880.0,
+ "3270": 13268858880.0,
+ "3275": 13268858880.0,
+ "3280": 13268858880.0,
+ "3285": 13268858880.0,
+ "3290": 13268858880.0,
+ "3295": 13268858880.0,
+ "3300": 13268858880.0,
+ "3305": 13268858880.0,
+ "3310": 13268858880.0,
+ "3315": 13268858880.0,
+ "3320": 13268858880.0,
+ "3325": 13268858880.0,
+ "3330": 13268858880.0,
+ "3335": 13268858880.0,
+ "3340": 13268858880.0,
+ "3345": 13268858880.0,
+ "3350": 13268858880.0,
+ "3355": 13268858880.0,
+ "3360": 13268858880.0,
+ "3365": 13268858880.0,
+ "3370": 13268858880.0,
+ "3375": 13268858880.0,
+ "3380": 13268858880.0,
+ "3385": 13268858880.0,
+ "3390": 13268858880.0,
+ "3395": 13268858880.0,
+ "3400": 13268858880.0,
+ "3405": 13268858880.0,
+ "3410": 13268858880.0,
+ "3415": 13268858880.0,
+ "3420": 13268858880.0,
+ "3425": 13268858880.0,
+ "3430": 13268858880.0,
+ "3435": 13268858880.0,
+ "3440": 13268858880.0,
+ "3445": 13268858880.0,
+ "3450": 13268858880.0,
+ "3455": 13268858880.0,
+ "3460": 13268858880.0,
+ "3465": 13268858880.0,
+ "3470": 13268858880.0,
+ "3475": 13268858880.0,
+ "3480": 13268858880.0,
+ "3485": 13268858880.0,
+ "3490": 13268858880.0,
+ "3495": 13268858880.0,
+ "3500": 13268858880.0,
+ "3505": 13268858880.0,
+ "3510": 13268858880.0,
+ "3515": 13268858880.0,
+ "3520": 13268858880.0,
+ "3525": 13268858880.0,
+ "3530": 13268858880.0,
+ "3535": 13268858880.0,
+ "3540": 13268858880.0,
+ "3545": 13268858880.0,
+ "3550": 13268858880.0,
+ "3555": 13268858880.0,
+ "3560": 13268858880.0,
+ "3565": 13268858880.0,
+ "3570": 13268858880.0,
+ "3575": 13268858880.0,
+ "3580": 13268858880.0,
+ "3585": 13268858880.0,
+ "3590": 13268858880.0,
+ "3595": 13268858880.0,
+ "3600": 13268858880.0,
+ "3605": 13268858880.0,
+ "3610": 13268858880.0,
+ "3615": 13268858880.0,
+ "3620": 13268858880.0,
+ "3625": 13268858880.0,
+ "3630": 13268858880.0,
+ "3635": 13268858880.0,
+ "3640": 13268858880.0,
+ "3645": 13268858880.0,
+ "3650": 13268858880.0,
+ "3655": 13268858880.0,
+ "3660": 13268858880.0,
+ "3665": 13268858880.0,
+ "3670": 13268858880.0,
+ "3675": 13268858880.0,
+ "3680": 13268858880.0,
+ "3685": 13268858880.0,
+ "3690": 13268858880.0,
+ "3695": 13268858880.0,
+ "3700": 13268858880.0,
+ "3705": 13268858880.0,
+ "3710": 13268858880.0,
+ "3715": 13268858880.0,
+ "3720": 13268858880.0,
+ "3725": 13268858880.0,
+ "3730": 13268858880.0,
+ "3735": 13268858880.0,
+ "3740": 13268858880.0,
+ "3745": 13268858880.0,
+ "3750": 13268858880.0,
+ "3755": 13268858880.0,
+ "3760": 13268858880.0,
+ "3765": 13268858880.0,
+ "3770": 13268858880.0,
+ "3775": 13268858880.0,
+ "3780": 13268858880.0,
+ "3785": 13268858880.0,
+ "3790": 13268858880.0,
+ "3795": 13268858880.0,
+ "3800": 13268858880.0,
+ "3805": 13268858880.0,
+ "3810": 13268858880.0,
+ "3815": 13268858880.0,
+ "3820": 13268858880.0,
+ "3825": 13268858880.0,
+ "3830": 13268858880.0,
+ "3835": 13268858880.0,
+ "3840": 13268858880.0,
+ "3845": 13268858880.0,
+ "3850": 13268858880.0,
+ "3855": 13268858880.0,
+ "3860": 13268858880.0,
+ "3865": 13268858880.0,
+ "3870": 13268858880.0,
+ "3875": 13268858880.0,
+ "3880": 13268858880.0,
+ "3885": 13268858880.0,
+ "3890": 13268858880.0,
+ "3895": 13268858880.0,
+ "3900": 13268858880.0,
+ "3905": 13268858880.0,
+ "3910": 13268858880.0,
+ "3915": 13268858880.0,
+ "3920": 13268858880.0,
+ "3925": 13268858880.0,
+ "3930": 13268858880.0,
+ "3935": 13268858880.0,
+ "3940": 13268858880.0,
+ "3945": 13268858880.0,
+ "3950": 13268858880.0,
+ "3955": 13268858880.0,
+ "3960": 13268858880.0,
+ "3965": 13268858880.0,
+ "3970": 13268858880.0,
+ "3975": 13268858880.0,
+ "3980": 13268858880.0,
+ "3985": 13268858880.0,
+ "3990": 13268858880.0,
+ "3995": 13268858880.0,
+ "4000": 13268858880.0,
+ "4005": 13268858880.0,
+ "4010": 13268858880.0,
+ "4015": 13268858880.0,
+ "4020": 13268858880.0,
+ "4025": 13268858880.0,
+ "4030": 13268858880.0,
+ "4035": 13268858880.0,
+ "4040": 13268858880.0,
+ "4045": 13268858880.0,
+ "4050": 13268858880.0,
+ "4055": 13268858880.0,
+ "4060": 13268858880.0,
+ "4065": 13268858880.0,
+ "4070": 13268858880.0,
+ "4075": 13268858880.0,
+ "4080": 13268858880.0,
+ "4085": 13268858880.0,
+ "4090": 13268858880.0,
+ "4095": 13268858880.0,
+ "4100": 13268858880.0,
+ "4105": 13268858880.0,
+ "4110": 13268858880.0,
+ "4115": 13268858880.0,
+ "4120": 13268858880.0,
+ "4125": 13268858880.0,
+ "4130": 13268858880.0,
+ "4135": 13268858880.0,
+ "4140": 13268858880.0,
+ "4145": 13268858880.0,
+ "4150": 13268858880.0,
+ "4155": 13268858880.0,
+ "4160": 13268858880.0,
+ "4165": 13268858880.0,
+ "4170": 13268858880.0,
+ "4175": 13268858880.0,
+ "4180": 13268858880.0,
+ "4185": 13268858880.0,
+ "4190": 13268858880.0,
+ "4195": 13268858880.0,
+ "4200": 13268858880.0,
+ "4205": 13268858880.0,
+ "4210": 13268858880.0,
+ "4215": 13268858880.0,
+ "4220": 13268858880.0,
+ "4225": 13268858880.0,
+ "4230": 13268858880.0,
+ "4235": 13268858880.0,
+ "4240": 13268858880.0,
+ "4245": 13268858880.0,
+ "4250": 13268858880.0,
+ "4255": 13268858880.0,
+ "4260": 13268858880.0,
+ "4265": 13268858880.0,
+ "4270": 13268858880.0,
+ "4275": 13268858880.0,
+ "4280": 13268858880.0,
+ "4285": 13268858880.0,
+ "4290": 13268858880.0,
+ "4295": 13268858880.0,
+ "4300": 13268858880.0,
+ "4305": 13268858880.0,
+ "4310": 13268858880.0,
+ "4315": 13268858880.0,
+ "4320": 13268858880.0,
+ "4325": 13268858880.0,
+ "4330": 13268858880.0,
+ "4335": 13268858880.0,
+ "4340": 13268858880.0,
+ "4345": 13268858880.0,
+ "4350": 13268858880.0,
+ "4355": 13268858880.0,
+ "4360": 13268858880.0,
+ "4365": 13268858880.0,
+ "4370": 13268858880.0,
+ "4375": 13268858880.0,
+ "4380": 13268858880.0,
+ "4385": 13268858880.0,
+ "4390": 13268858880.0,
+ "4395": 13268858880.0,
+ "4400": 13268858880.0,
+ "4405": 13268858880.0,
+ "4410": 13268858880.0,
+ "4415": 13268858880.0,
+ "4420": 13268858880.0,
+ "4425": 13268858880.0,
+ "4430": 13268858880.0,
+ "4435": 13268858880.0,
+ "4440": 13268858880.0,
+ "4445": 13268858880.0,
+ "4450": 13268858880.0,
+ "4455": 13268858880.0,
+ "4460": 13268858880.0,
+ "4465": 13268858880.0,
+ "4470": 13268858880.0,
+ "4475": 13268858880.0,
+ "4480": 13268858880.0,
+ "4485": 13268858880.0,
+ "4490": 13268858880.0,
+ "4495": 13268858880.0,
+ "4500": 13268858880.0,
+ "4505": 13268858880.0,
+ "4510": 13268858880.0,
+ "4515": 13268858880.0,
+ "4520": 13268858880.0,
+ "4525": 13268858880.0,
+ "4530": 13268858880.0,
+ "4535": 13268858880.0,
+ "4540": 13268858880.0,
+ "4545": 13268858880.0,
+ "4550": 13268858880.0,
+ "4555": 13268858880.0,
+ "4560": 13268858880.0,
+ "4565": 13268858880.0,
+ "4570": 13268858880.0,
+ "4575": 13268858880.0,
+ "4580": 13268858880.0,
+ "4585": 13268858880.0,
+ "4590": 13268858880.0,
+ "4595": 13268858880.0,
+ "4600": 13268858880.0,
+ "4605": 13268858880.0,
+ "4610": 13268858880.0,
+ "4615": 13268858880.0,
+ "4620": 13268858880.0,
+ "4625": 13268858880.0,
+ "4630": 13268858880.0,
+ "4635": 13268858880.0,
+ "4640": 13268858880.0,
+ "4645": 13268858880.0,
+ "4650": 13268858880.0,
+ "4655": 13268858880.0,
+ "4660": 13268858880.0,
+ "4665": 13268858880.0,
+ "4670": 13268858880.0,
+ "4675": 13268858880.0,
+ "4680": 13268858880.0,
+ "4685": 13268858880.0,
+ "4690": 13268858880.0,
+ "4695": 13268858880.0,
+ "4700": 13268858880.0,
+ "4705": 13268858880.0,
+ "4710": 13268858880.0,
+ "4715": 13268858880.0,
+ "4720": 13268858880.0,
+ "4725": 13268858880.0,
+ "4730": 13268858880.0,
+ "4735": 13268858880.0,
+ "4740": 13268858880.0,
+ "4745": 13268858880.0,
+ "4750": 13268858880.0,
+ "4755": 13268858880.0,
+ "4760": 13268858880.0,
+ "4765": 13268858880.0,
+ "4770": 13268858880.0,
+ "4775": 13268858880.0,
+ "4780": 13268858880.0,
+ "4785": 13268858880.0,
+ "4790": 13268858880.0,
+ "4795": 13268858880.0,
+ "4800": 13268858880.0,
+ "4805": 13268858880.0,
+ "4810": 13268858880.0,
+ "4815": 13268858880.0,
+ "4820": 13268858880.0,
+ "4825": 13268858880.0,
+ "4830": 13268858880.0,
+ "4835": 13268858880.0,
+ "4840": 13268858880.0,
+ "4845": 13268858880.0,
+ "4850": 13268858880.0,
+ "4855": 13268858880.0,
+ "4860": 13268858880.0,
+ "4865": 13268858880.0,
+ "4870": 13268858880.0,
+ "4875": 13268858880.0,
+ "4880": 13268858880.0,
+ "4885": 13268858880.0,
+ "4890": 13268858880.0,
+ "4895": 13268858880.0,
+ "4900": 13268858880.0,
+ "4905": 13268858880.0,
+ "4910": 13268858880.0,
+ "4915": 13268858880.0,
+ "4920": 13268858880.0,
+ "4925": 13268858880.0,
+ "4930": 13268858880.0,
+ "4935": 13268858880.0,
+ "4940": 13268858880.0,
+ "4945": 13268858880.0,
+ "4950": 13268858880.0,
+ "4955": 13268858880.0,
+ "4960": 13268858880.0,
+ "4965": 13268858880.0,
+ "4970": 13268858880.0,
+ "4975": 13268858880.0,
+ "4980": 13268858880.0,
+ "4985": 13268858880.0,
+ "4990": 13268858880.0,
+ "4995": 13268858880.0,
+ "5000": 13268858880.0,
+ "5005": 13268858880.0,
+ "5010": 13268858880.0,
+ "5015": 13268858880.0,
+ "5020": 13268858880.0,
+ "5025": 13268858880.0,
+ "5030": 13268858880.0,
+ "5035": 13268858880.0,
+ "5040": 13268858880.0,
+ "5045": 13268858880.0,
+ "5050": 13268858880.0,
+ "5055": 13268858880.0,
+ "5060": 13268858880.0,
+ "5065": 13268858880.0,
+ "5070": 13268858880.0,
+ "5075": 13268858880.0,
+ "5080": 13268858880.0,
+ "5085": 13268858880.0,
+ "5090": 13268858880.0,
+ "5095": 13268858880.0,
+ "5100": 13268858880.0,
+ "5105": 13268858880.0,
+ "5110": 13268858880.0,
+ "5115": 13268858880.0,
+ "5120": 13268858880.0,
+ "5125": 13268858880.0,
+ "5130": 13268858880.0,
+ "5135": 13268858880.0,
+ "5140": 13268858880.0,
+ "5145": 13268858880.0,
+ "5150": 13268858880.0,
+ "5155": 13268858880.0,
+ "5160": 13268858880.0,
+ "5165": 13268858880.0,
+ "5170": 13268858880.0,
+ "5175": 13268858880.0,
+ "5180": 13268858880.0,
+ "5185": 13268858880.0,
+ "5190": 13268858880.0,
+ "5195": 13268858880.0,
+ "5200": 13268858880.0,
+ "5205": 13268858880.0,
+ "5210": 13268858880.0,
+ "5215": 13268858880.0,
+ "5220": 13268858880.0,
+ "5225": 13268858880.0,
+ "5230": 13268858880.0,
+ "5235": 13268858880.0,
+ "5240": 13268858880.0,
+ "5245": 13268858880.0,
+ "5250": 13268858880.0,
+ "5255": 13268858880.0,
+ "5260": 13268858880.0,
+ "5265": 13268858880.0,
+ "5270": 13268858880.0,
+ "5275": 13268858880.0,
+ "5280": 13268858880.0,
+ "5285": 13268858880.0,
+ "5290": 13268858880.0,
+ "5295": 13268858880.0,
+ "5300": 13268858880.0,
+ "5305": 13268858880.0,
+ "5310": 13268858880.0,
+ "5315": 13268858880.0,
+ "5320": 13268858880.0,
+ "5325": 13268858880.0,
+ "5330": 13268858880.0,
+ "5335": 13268858880.0,
+ "5340": 13268858880.0,
+ "5345": 13268858880.0,
+ "5350": 13268858880.0,
+ "5355": 13268858880.0,
+ "5360": 13268858880.0,
+ "5365": 13268858880.0,
+ "5370": 13268858880.0,
+ "5375": 13268858880.0,
+ "5380": 13268858880.0,
+ "5385": 13268858880.0,
+ "5390": 13268858880.0,
+ "5395": 13268858880.0,
+ "5400": 13268858880.0,
+ "5405": 13268858880.0,
+ "5410": 13268858880.0,
+ "5415": 13268858880.0,
+ "5420": 13268858880.0,
+ "5425": 13268858880.0,
+ "5430": 13268858880.0,
+ "5435": 13268858880.0,
+ "5440": 13268858880.0,
+ "5445": 13268858880.0,
+ "5450": 13268858880.0,
+ "5455": 13268858880.0,
+ "5460": 13268858880.0,
+ "5465": 13268858880.0,
+ "5470": 13268858880.0,
+ "5475": 13268858880.0,
+ "5480": 13268858880.0,
+ "5485": 13268858880.0,
+ "5490": 13268858880.0,
+ "5495": 13268858880.0,
+ "5500": 13268858880.0,
+ "5505": 13268858880.0,
+ "5510": 13268858880.0,
+ "5515": 13268858880.0,
+ "5520": 13268858880.0,
+ "5525": 13268858880.0,
+ "5530": 13268858880.0,
+ "5535": 13268858880.0,
+ "5540": 13268858880.0,
+ "5545": 13268858880.0,
+ "5550": 13268858880.0,
+ "5555": 13268858880.0,
+ "5560": 13268858880.0,
+ "5565": 13268858880.0,
+ "5570": 13268858880.0,
+ "5575": 13268858880.0,
+ "5580": 13268858880.0,
+ "5585": 13268858880.0,
+ "5590": 13268858880.0,
+ "5595": 13268858880.0,
+ "5600": 13268858880.0,
+ "5605": 13268858880.0,
+ "5610": 13268858880.0,
+ "5615": 13268858880.0,
+ "5620": 13268858880.0,
+ "5625": 13268858880.0,
+ "5630": 13268858880.0,
+ "5635": 13268858880.0,
+ "5640": 13268858880.0,
+ "5645": 13268858880.0,
+ "5650": 13268858880.0,
+ "5655": 13268858880.0,
+ "5660": 13268858880.0,
+ "5665": 13268858880.0,
+ "5670": 13268858880.0,
+ "5675": 13268858880.0,
+ "5680": 13268858880.0,
+ "5685": 13268858880.0,
+ "5690": 13268858880.0,
+ "5695": 13268858880.0,
+ "5700": 13268858880.0,
+ "5705": 13268858880.0,
+ "5710": 13268858880.0,
+ "5715": 13268858880.0,
+ "5720": 13268858880.0,
+ "5725": 13268858880.0,
+ "5730": 13268858880.0,
+ "5735": 13268858880.0,
+ "5740": 13268858880.0,
+ "5745": 13268858880.0,
+ "5750": 13268858880.0,
+ "5755": 13268858880.0,
+ "5760": 13268858880.0,
+ "5765": 13268858880.0,
+ "5770": 13268858880.0,
+ "5775": 13268858880.0,
+ "5780": 13268858880.0,
+ "5785": 13268858880.0,
+ "5790": 13268858880.0,
+ "5795": 13268858880.0,
+ "5800": 13268858880.0,
+ "5805": 13268858880.0,
+ "5810": 13268858880.0,
+ "5815": 13268858880.0,
+ "5820": 13268858880.0,
+ "5825": 13268858880.0,
+ "5830": 13268858880.0,
+ "5835": 13268858880.0,
+ "5840": 13268858880.0,
+ "5845": 13268858880.0,
+ "5850": 13268858880.0,
+ "5855": 13268858880.0,
+ "5860": 13268858880.0,
+ "5865": 13268858880.0,
+ "5870": 13268858880.0,
+ "5875": 13268858880.0,
+ "5880": 13268858880.0,
+ "5885": 13268858880.0,
+ "5890": 13268858880.0,
+ "5895": 13268858880.0,
+ "5900": 13268858880.0,
+ "5905": 13268858880.0,
+ "5910": 13268858880.0,
+ "5915": 13268858880.0,
+ "5920": 13268858880.0,
+ "5925": 13268858880.0,
+ "5930": 13268858880.0,
+ "5935": 13268858880.0,
+ "5940": 13268858880.0,
+ "5945": 13268858880.0,
+ "5950": 13268858880.0,
+ "5955": 13268858880.0,
+ "5960": 13268858880.0,
+ "5965": 13268858880.0,
+ "5970": 13268858880.0,
+ "5975": 13268858880.0,
+ "5980": 13268858880.0,
+ "5985": 13268858880.0,
+ "5990": 13268858880.0,
+ "5995": 13268858880.0,
+ "6000": 13268858880.0,
+ "6005": 13268858880.0,
+ "6010": 13268858880.0,
+ "6015": 13268858880.0,
+ "6020": 13268858880.0,
+ "6025": 13268858880.0,
+ "6030": 13268858880.0,
+ "6035": 13268858880.0,
+ "6040": 13268858880.0,
+ "6045": 13268858880.0,
+ "6050": 13268858880.0,
+ "6055": 13268858880.0,
+ "6060": 13268858880.0,
+ "6065": 13268858880.0,
+ "6070": 13268858880.0,
+ "6075": 13268858880.0,
+ "6080": 13268858880.0,
+ "6085": 13268858880.0,
+ "6090": 13268858880.0,
+ "6095": 13268858880.0,
+ "6100": 13268858880.0,
+ "6105": 13268858880.0,
+ "6110": 13268858880.0,
+ "6115": 13268858880.0,
+ "6120": 13268858880.0,
+ "6125": 13268858880.0,
+ "6130": 13268858880.0,
+ "6135": 13268858880.0,
+ "6140": 13268858880.0,
+ "6145": 13268858880.0,
+ "6150": 13268858880.0,
+ "6155": 13268858880.0,
+ "6160": 13268858880.0,
+ "6165": 13268858880.0,
+ "6170": 13268858880.0,
+ "6175": 13268858880.0,
+ "6180": 13268858880.0,
+ "6185": 13268858880.0,
+ "6190": 13268858880.0,
+ "6195": 13268858880.0,
+ "6200": 13268858880.0,
+ "6205": 13268858880.0,
+ "6210": 13268858880.0,
+ "6215": 13268858880.0,
+ "6220": 13268858880.0,
+ "6225": 13268858880.0,
+ "6230": 13268858880.0,
+ "6235": 13268858880.0,
+ "6240": 13268858880.0,
+ "6245": 13268858880.0,
+ "6250": 13268858880.0,
+ "6255": 13268858880.0,
+ "6260": 13268858880.0,
+ "6265": 13268858880.0,
+ "6270": 13268858880.0,
+ "6275": 13268858880.0,
+ "6280": 13268858880.0,
+ "6285": 13268858880.0,
+ "6290": 13268858880.0,
+ "6295": 13268858880.0,
+ "6300": 13268858880.0,
+ "6305": 13268858880.0,
+ "6310": 13268858880.0,
+ "6315": 13268858880.0,
+ "6320": 13268858880.0,
+ "6325": 13268858880.0,
+ "6330": 13268858880.0,
+ "6335": 13268858880.0,
+ "6340": 13268858880.0,
+ "6345": 13268858880.0,
+ "6350": 13268858880.0,
+ "6355": 13268858880.0,
+ "6360": 13268858880.0,
+ "6365": 13268858880.0,
+ "6370": 13268858880.0,
+ "6375": 13268858880.0,
+ "6380": 13268858880.0,
+ "6385": 13268858880.0,
+ "6390": 13268858880.0,
+ "6395": 13268858880.0,
+ "6400": 13268858880.0,
+ "6405": 13268858880.0,
+ "6410": 13268858880.0,
+ "6415": 13268858880.0,
+ "6420": 13268858880.0,
+ "6425": 13268858880.0,
+ "6430": 13268858880.0,
+ "6435": 13268858880.0,
+ "6440": 13268858880.0,
+ "6445": 13268858880.0,
+ "6450": 13268858880.0,
+ "6455": 13268858880.0,
+ "6460": 13268858880.0,
+ "6465": 13268858880.0,
+ "6470": 13268858880.0,
+ "6475": 13268858880.0,
+ "6480": 13268858880.0,
+ "6485": 13268858880.0,
+ "6490": 13268858880.0,
+ "6495": 13268858880.0,
+ "6500": 13268858880.0,
+ "6505": 13268858880.0,
+ "6510": 13268858880.0,
+ "6515": 13268858880.0,
+ "6520": 13268858880.0,
+ "6525": 13268858880.0,
+ "6530": 13268858880.0,
+ "6535": 13268858880.0,
+ "6540": 13268858880.0,
+ "6545": 13268858880.0,
+ "6550": 13268858880.0,
+ "6555": 13268858880.0,
+ "6560": 13268858880.0,
+ "6565": 13268858880.0,
+ "6570": 13268858880.0,
+ "6575": 13268858880.0,
+ "6580": 13268858880.0,
+ "6585": 13268858880.0,
+ "6590": 13268858880.0,
+ "6595": 13268858880.0,
+ "6600": 13268858880.0,
+ "6605": 13268858880.0,
+ "6610": 13268858880.0,
+ "6615": 13268858880.0,
+ "6620": 13268858880.0,
+ "6625": 13268858880.0,
+ "6630": 13268858880.0,
+ "6635": 13268858880.0,
+ "6640": 13268858880.0,
+ "6645": 13268858880.0,
+ "6650": 13268858880.0,
+ "6655": 13268858880.0,
+ "6660": 13268858880.0,
+ "6665": 13268858880.0,
+ "6670": 13268858880.0,
+ "6675": 13268858880.0,
+ "6680": 13268858880.0,
+ "6685": 13268858880.0,
+ "6690": 13268858880.0,
+ "6695": 13268858880.0,
+ "6700": 13268858880.0,
+ "6705": 13268858880.0,
+ "6710": 13268858880.0,
+ "6715": 13268858880.0,
+ "6720": 13268858880.0,
+ "6725": 13268858880.0,
+ "6730": 13268858880.0,
+ "6735": 13268858880.0,
+ "6740": 13268858880.0,
+ "6745": 13268858880.0,
+ "6750": 13268858880.0,
+ "6755": 13268858880.0,
+ "6760": 13268858880.0,
+ "6765": 13268858880.0,
+ "6770": 13268858880.0,
+ "6775": 13268858880.0,
+ "6780": 13268858880.0,
+ "6785": 13268858880.0,
+ "6790": 13268858880.0,
+ "6795": 13268858880.0,
+ "6800": 13268858880.0,
+ "6805": 13268858880.0,
+ "6810": 13268858880.0,
+ "6815": 13268858880.0,
+ "6820": 13268858880.0,
+ "6825": 13268858880.0,
+ "6830": 13268858880.0,
+ "6835": 13268858880.0,
+ "6840": 13268858880.0,
+ "6845": 13268858880.0,
+ "6850": 13268858880.0,
+ "6855": 13268858880.0,
+ "6860": 13268858880.0,
+ "6865": 13268858880.0,
+ "6870": 13268858880.0,
+ "6875": 13268858880.0,
+ "6880": 13268858880.0,
+ "6885": 13268858880.0,
+ "6890": 13268858880.0,
+ "6895": 13268858880.0,
+ "6900": 13268858880.0,
+ "6905": 13268858880.0,
+ "6910": 13268858880.0,
+ "6915": 13268858880.0,
+ "6920": 13268858880.0,
+ "6925": 13268858880.0,
+ "6930": 13268858880.0,
+ "6935": 13268858880.0,
+ "6940": 13268858880.0,
+ "6945": 13268858880.0,
+ "6950": 13268858880.0,
+ "6955": 13268858880.0,
+ "6960": 13268858880.0,
+ "6965": 13268858880.0,
+ "6970": 13268858880.0,
+ "6975": 13268858880.0,
+ "6980": 13268858880.0,
+ "6985": 13268858880.0,
+ "6990": 13268858880.0,
+ "6995": 13268858880.0,
+ "7000": 13268858880.0,
+ "7005": 13268858880.0,
+ "7010": 13268858880.0,
+ "7015": 13268858880.0,
+ "7020": 13268858880.0,
+ "7025": 13268858880.0,
+ "7030": 13268858880.0,
+ "7035": 13268858880.0,
+ "7040": 13268858880.0,
+ "7045": 13268858880.0,
+ "7050": 13268858880.0,
+ "7055": 13268858880.0,
+ "7060": 13268858880.0,
+ "7065": 13268858880.0,
+ "7070": 13268858880.0,
+ "7075": 13268858880.0,
+ "7080": 13268858880.0,
+ "7085": 13268858880.0,
+ "7090": 13268858880.0,
+ "7095": 13268858880.0,
+ "7100": 13268858880.0,
+ "7105": 13268858880.0,
+ "7110": 13268858880.0,
+ "7115": 13268858880.0,
+ "7120": 13268858880.0,
+ "7125": 13268858880.0,
+ "7130": 13268858880.0,
+ "7135": 13268858880.0,
+ "7140": 13268858880.0,
+ "7145": 13268858880.0,
+ "7150": 13268858880.0,
+ "7155": 13268858880.0,
+ "7160": 13268858880.0,
+ "7165": 13268858880.0,
+ "7170": 13268858880.0,
+ "7175": 13268858880.0,
+ "7180": 13268858880.0,
+ "7185": 13268858880.0,
+ "7190": 13268858880.0,
+ "7195": 13268858880.0,
+ "7200": 13268858880.0,
+ "7205": 13268858880.0,
+ "7210": 13268858880.0,
+ "7215": 13268858880.0,
+ "7220": 13268858880.0,
+ "7225": 13268858880.0,
+ "7230": 13268858880.0,
+ "7235": 13268858880.0,
+ "7240": 13268858880.0,
+ "7245": 13268858880.0,
+ "7250": 13268858880.0,
+ "7255": 13268858880.0,
+ "7260": 13268858880.0,
+ "7265": 13268858880.0,
+ "7270": 13268858880.0,
+ "7275": 13268858880.0,
+ "7280": 13268858880.0,
+ "7285": 13268858880.0,
+ "7290": 13268858880.0,
+ "7295": 13268858880.0,
+ "7300": 13268858880.0,
+ "7305": 13268858880.0,
+ "7310": 13268858880.0,
+ "7315": 13268858880.0,
+ "7320": 13268858880.0,
+ "7325": 13268858880.0,
+ "7330": 13268858880.0,
+ "7335": 13268858880.0,
+ "7340": 13268858880.0,
+ "7345": 13268858880.0,
+ "7350": 13268858880.0,
+ "7355": 13268858880.0,
+ "7360": 13268858880.0,
+ "7365": 13268858880.0,
+ "7370": 13268858880.0,
+ "7375": 13268858880.0,
+ "7380": 13268858880.0,
+ "7385": 13268858880.0,
+ "7390": 13268858880.0,
+ "7395": 13268858880.0,
+ "7400": 13268858880.0,
+ "7405": 13268858880.0,
+ "7410": 13268858880.0,
+ "7415": 13268858880.0,
+ "7420": 13268858880.0,
+ "7425": 13268858880.0,
+ "7430": 13268858880.0,
+ "7435": 13268858880.0,
+ "7440": 13268858880.0,
+ "7445": 13268858880.0,
+ "7450": 13268858880.0,
+ "7455": 13268858880.0,
+ "7460": 13268858880.0,
+ "7465": 13268858880.0,
+ "7470": 13268858880.0,
+ "7475": 13268858880.0,
+ "7480": 13268858880.0,
+ "7485": 13268858880.0,
+ "7490": 13268858880.0,
+ "7495": 13268858880.0,
+ "7500": 13268858880.0,
+ "7505": 13268858880.0,
+ "7510": 13268858880.0,
+ "7515": 13268858880.0,
+ "7520": 13268858880.0,
+ "7525": 13268858880.0,
+ "7530": 13268858880.0,
+ "7535": 13268858880.0,
+ "7540": 13268858880.0,
+ "7545": 13268858880.0,
+ "7550": 13268858880.0,
+ "7555": 13268858880.0,
+ "7560": 13268858880.0,
+ "7565": 13268858880.0,
+ "7570": 13268858880.0,
+ "7575": 13268858880.0,
+ "7580": 13268858880.0,
+ "7585": 13268858880.0,
+ "7590": 13268858880.0,
+ "7595": 13268858880.0,
+ "7600": 13268858880.0,
+ "7605": 13268858880.0,
+ "7610": 13268858880.0,
+ "7615": 13268858880.0,
+ "7620": 13268858880.0,
+ "7625": 13268858880.0,
+ "7630": 13268858880.0,
+ "7635": 13268858880.0,
+ "7640": 13268858880.0,
+ "7645": 13268858880.0,
+ "7650": 13268858880.0,
+ "7655": 13268858880.0,
+ "7660": 13268858880.0,
+ "7665": 13268858880.0,
+ "7670": 13268858880.0,
+ "7675": 13268858880.0,
+ "7680": 13268858880.0,
+ "7685": 13268858880.0,
+ "7690": 13268858880.0,
+ "7695": 13268858880.0,
+ "7700": 13268858880.0,
+ "7705": 13268858880.0,
+ "7710": 13268858880.0,
+ "7715": 13268858880.0,
+ "7720": 13268858880.0,
+ "7725": 13268858880.0,
+ "7730": 13268858880.0,
+ "7735": 13268858880.0,
+ "7740": 13268858880.0,
+ "7745": 13268858880.0,
+ "7750": 13268858880.0,
+ "7755": 13268858880.0,
+ "7760": 13268858880.0,
+ "7765": 13268858880.0,
+ "7770": 13268858880.0,
+ "7775": 13268858880.0,
+ "7780": 13268858880.0,
+ "7785": 13268858880.0,
+ "7790": 13268858880.0,
+ "7795": 13268858880.0,
+ "7800": 13268858880.0,
+ "7805": 13268858880.0,
+ "7810": 13268858880.0,
+ "7815": 13268858880.0,
+ "7820": 13268858880.0,
+ "7825": 13268858880.0,
+ "7830": 13268858880.0,
+ "7835": 13268858880.0,
+ "7840": 13268858880.0,
+ "7845": 13268858880.0,
+ "7850": 13268858880.0,
+ "7855": 13268858880.0,
+ "7860": 13268858880.0,
+ "7865": 13268858880.0,
+ "7870": 13268858880.0,
+ "7875": 13268858880.0,
+ "7880": 13268858880.0,
+ "7885": 13268858880.0,
+ "7890": 13268858880.0,
+ "7895": 13268858880.0,
+ "7900": 13268858880.0,
+ "7905": 13268858880.0,
+ "7910": 13268858880.0,
+ "7915": 13268858880.0,
+ "7920": 13268858880.0,
+ "7925": 13268858880.0,
+ "7930": 13268858880.0,
+ "7935": 13268858880.0,
+ "7940": 13268858880.0,
+ "7945": 13268858880.0,
+ "7950": 13268858880.0,
+ "7955": 13268858880.0,
+ "7960": 13268858880.0,
+ "7965": 13268858880.0,
+ "7970": 13268858880.0,
+ "7975": 13268858880.0,
+ "7980": 13268858880.0,
+ "7985": 13268858880.0,
+ "7990": 13268858880.0,
+ "7995": 13268858880.0,
+ "8000": 13268858880.0,
+ "8005": 13268858880.0,
+ "8010": 13268858880.0,
+ "8015": 13268858880.0,
+ "8020": 13268858880.0,
+ "8025": 13268858880.0,
+ "8030": 13268858880.0,
+ "8035": 13268858880.0,
+ "8040": 13268858880.0,
+ "8045": 13268858880.0,
+ "8050": 13268858880.0,
+ "8055": 13268858880.0,
+ "8060": 13268858880.0,
+ "8065": 13268858880.0,
+ "8070": 13268858880.0,
+ "8075": 13268858880.0,
+ "8080": 13268858880.0,
+ "8085": 13268858880.0,
+ "8090": 13268858880.0,
+ "8095": 13268858880.0,
+ "8100": 13268858880.0,
+ "8105": 13268858880.0,
+ "8110": 13268858880.0,
+ "8115": 13268858880.0,
+ "8120": 13268858880.0,
+ "8125": 13268858880.0,
+ "8130": 13268858880.0,
+ "8135": 13268858880.0,
+ "8140": 13268858880.0,
+ "8145": 13268858880.0,
+ "8150": 13268858880.0,
+ "8155": 13268858880.0,
+ "8160": 13268858880.0,
+ "8165": 13268858880.0,
+ "8170": 13268858880.0,
+ "8175": 13268858880.0,
+ "8180": 13268858880.0,
+ "8185": 13268858880.0,
+ "8190": 13268858880.0,
+ "8195": 13268858880.0,
+ "8200": 13268858880.0,
+ "8205": 13268858880.0,
+ "8210": 13268858880.0,
+ "8215": 13268858880.0,
+ "8220": 13268858880.0,
+ "8225": 13268858880.0,
+ "8230": 13268858880.0,
+ "8235": 13268858880.0,
+ "8240": 13268858880.0,
+ "8245": 13268858880.0,
+ "8250": 13268858880.0,
+ "8255": 13268858880.0,
+ "8260": 13269043200.0,
+ "8265": 13269043200.0,
+ "8270": 13269043200.0,
+ "8275": 13269043200.0,
+ "8280": 13269043200.0,
+ "8285": 13269043200.0,
+ "8290": 13269043200.0,
+ "8295": 13269043200.0,
+ "8300": 13269043200.0,
+ "8305": 13269043200.0,
+ "8310": 13269043200.0,
+ "8315": 13269043200.0,
+ "8320": 13269043200.0,
+ "8325": 13269043200.0,
+ "8330": 13269043200.0,
+ "8335": 13269043200.0,
+ "8340": 13269043200.0,
+ "8345": 13269043200.0,
+ "8350": 13269043200.0,
+ "8355": 13269043200.0,
+ "8360": 13269043200.0,
+ "8365": 13269043200.0,
+ "8370": 13269043200.0,
+ "8375": 13269043200.0,
+ "8380": 13269043200.0,
+ "8385": 13269043200.0,
+ "8390": 13269043200.0,
+ "8395": 13269043200.0,
+ "8400": 13269043200.0,
+ "8405": 13269043200.0,
+ "8410": 13269043200.0,
+ "8415": 13269043200.0,
+ "8420": 13269043200.0,
+ "8425": 13269043200.0,
+ "8430": 13269043200.0,
+ "8435": 13269043200.0,
+ "8440": 13269043200.0,
+ "8445": 13269043200.0,
+ "8450": 13269043200.0,
+ "8455": 13269043200.0,
+ "8460": 13269043200.0,
+ "8465": 13269043200.0,
+ "8470": 13269043200.0,
+ "8475": 13269043200.0,
+ "8480": 13269043200.0,
+ "8485": 13269043200.0,
+ "8490": 13269043200.0,
+ "8495": 13269043200.0,
+ "8500": 13269043200.0,
+ "8505": 13269043200.0,
+ "8510": 13269043200.0,
+ "8515": 13269043200.0,
+ "8520": 13269043200.0,
+ "8525": 13269043200.0,
+ "8530": 13269043200.0,
+ "8535": 13269043200.0,
+ "8540": 13269043200.0,
+ "8545": 13269043200.0,
+ "8550": 13269043200.0,
+ "8555": 13269043200.0,
+ "8560": 13269043200.0,
+ "8565": 13269043200.0,
+ "8570": 13269043200.0,
+ "8575": 13269043200.0,
+ "8580": 13269043200.0,
+ "8585": 13269043200.0,
+ "8590": 13269043200.0,
+ "8595": 13269043200.0,
+ "8600": 13269043200.0,
+ "8605": 13269043200.0,
+ "8610": 13269043200.0,
+ "8615": 13269043200.0,
+ "8620": 13269043200.0,
+ "8625": 13269043200.0,
+ "8630": 13269043200.0,
+ "8635": 13269043200.0,
+ "8640": 13269043200.0,
+ "8645": 13269043200.0,
+ "8650": 13269043200.0,
+ "8655": 13269043200.0,
+ "8660": 13269043200.0,
+ "8665": 13269043200.0,
+ "8670": 13269043200.0,
+ "8675": 13269043200.0,
+ "8680": 13269043200.0,
+ "8685": 13269043200.0,
+ "8690": 13269043200.0,
+ "8695": 13269043200.0,
+ "8700": 13269043200.0,
+ "8705": 13269043200.0,
+ "8710": 13269043200.0,
+ "8715": 13269043200.0,
+ "8720": 13269043200.0,
+ "8725": 13269043200.0,
+ "8730": 13269043200.0,
+ "8735": 13269043200.0,
+ "8740": 13269043200.0,
+ "8745": 13269043200.0,
+ "8750": 13269043200.0,
+ "8755": 13269043200.0,
+ "8760": 13269043200.0,
+ "8765": 13269043200.0,
+ "8770": 13269043200.0,
+ "8775": 13269043200.0,
+ "8780": 13269043200.0,
+ "8785": 13269043200.0,
+ "8790": 13269043200.0,
+ "8795": 13269043200.0,
+ "8800": 13269043200.0,
+ "8805": 13269043200.0,
+ "8810": 13269043200.0,
+ "8815": 13269043200.0,
+ "8820": 13269043200.0,
+ "8825": 13269043200.0,
+ "8830": 13269043200.0,
+ "8835": 13269043200.0,
+ "8840": 13269043200.0,
+ "8845": 13269043200.0,
+ "8850": 13269043200.0,
+ "8855": 13269043200.0,
+ "8860": 13269043200.0,
+ "8865": 13269043200.0,
+ "8870": 13269043200.0,
+ "8875": 13269043200.0,
+ "8880": 13269043200.0,
+ "8885": 13269043200.0,
+ "8890": 13269043200.0,
+ "8895": 13269043200.0,
+ "8900": 13269043200.0,
+ "8905": 13269043200.0,
+ "8910": 13269043200.0,
+ "8915": 13269043200.0,
+ "8920": 13269043200.0,
+ "8925": 13269043200.0,
+ "8930": 13269043200.0,
+ "8935": 13269043200.0,
+ "8940": 13269043200.0,
+ "8945": 13269043200.0,
+ "8950": 13269043200.0,
+ "8955": 13269043200.0,
+ "8960": 13269043200.0,
+ "8965": 13269043200.0,
+ "8970": 13269043200.0,
+ "8975": 13269043200.0,
+ "8980": 13269043200.0,
+ "8985": 13269043200.0,
+ "8990": 13269043200.0,
+ "8995": 13269043200.0,
+ "9000": 13269043200.0,
+ "9005": 13269043200.0,
+ "9010": 13269043200.0,
+ "9015": 13269043200.0,
+ "9020": 13269043200.0,
+ "9025": 13269043200.0,
+ "9030": 13269043200.0,
+ "9035": 13269043200.0,
+ "9040": 13269043200.0,
+ "9045": 13269043200.0,
+ "9050": 13269043200.0,
+ "9055": 13269043200.0,
+ "9060": 13269043200.0,
+ "9065": 13269043200.0,
+ "9070": 13269043200.0,
+ "9075": 13269043200.0,
+ "9080": 13269043200.0,
+ "9085": 13269043200.0,
+ "9090": 13269043200.0,
+ "9095": 13269043200.0,
+ "9100": 13269043200.0,
+ "9105": 13269043200.0,
+ "9110": 13269043200.0,
+ "9115": 13269043200.0,
+ "9120": 13269043200.0,
+ "9125": 13269043200.0,
+ "9130": 13269043200.0,
+ "9135": 13269043200.0,
+ "9140": 13269043200.0,
+ "9145": 13269043200.0,
+ "9150": 13269043200.0,
+ "9155": 13269043200.0,
+ "9160": 13269043200.0,
+ "9165": 13269043200.0,
+ "9170": 13269043200.0,
+ "9175": 13269043200.0,
+ "9180": 13269043200.0,
+ "9185": 13269043200.0,
+ "9190": 13269043200.0,
+ "9195": 13269043200.0,
+ "9200": 13269043200.0,
+ "9205": 13269043200.0,
+ "9210": 13269043200.0,
+ "9215": 13269043200.0,
+ "9220": 13269043200.0,
+ "9225": 13269043200.0,
+ "9230": 13269043200.0,
+ "9235": 13269043200.0,
+ "9240": 13269043200.0,
+ "9245": 13269043200.0,
+ "9250": 13269043200.0,
+ "9255": 13269043200.0,
+ "9260": 13269043200.0,
+ "9265": 13269043200.0,
+ "9270": 13269043200.0,
+ "9275": 13269043200.0,
+ "9280": 13269043200.0,
+ "9285": 13269043200.0,
+ "9290": 13269043200.0,
+ "9295": 13269043200.0,
+ "9300": 13269043200.0,
+ "9305": 13269043200.0,
+ "9310": 13269043200.0,
+ "9315": 13269043200.0,
+ "9320": 13269043200.0,
+ "9325": 13269043200.0,
+ "9330": 13269043200.0,
+ "9335": 13269043200.0,
+ "9340": 13269043200.0,
+ "9345": 13269043200.0,
+ "9350": 13269043200.0,
+ "9355": 13269043200.0,
+ "9360": 13269043200.0,
+ "9365": 13269043200.0,
+ "9370": 13269043200.0,
+ "9375": 13269043200.0,
+ "9380": 13269043200.0,
+ "9385": 13269043200.0,
+ "9390": 13269043200.0,
+ "9395": 13269043200.0,
+ "9400": 13269043200.0,
+ "9405": 13269043200.0,
+ "9410": 13269043200.0,
+ "9415": 13269043200.0,
+ "9420": 13269043200.0,
+ "9425": 13269043200.0,
+ "9430": 13269043200.0,
+ "9435": 13269043200.0,
+ "9440": 13269043200.0,
+ "9445": 13269043200.0,
+ "9450": 13269043200.0,
+ "9455": 13269043200.0,
+ "9460": 13269043200.0,
+ "9465": 13269043200.0,
+ "9470": 13269043200.0,
+ "9475": 13269043200.0,
+ "9480": 13269043200.0,
+ "9485": 13269043200.0,
+ "9490": 13269043200.0,
+ "9495": 13269043200.0,
+ "9500": 13269043200.0,
+ "9505": 13269043200.0,
+ "9510": 13269043200.0,
+ "9515": 13269043200.0,
+ "9520": 13269043200.0,
+ "9525": 13269043200.0,
+ "9530": 13269043200.0,
+ "9535": 13269043200.0,
+ "9540": 13269043200.0,
+ "9545": 13269043200.0,
+ "9550": 13269043200.0,
+ "9555": 13269043200.0,
+ "9560": 13269043200.0,
+ "9565": 13269043200.0,
+ "9570": 13269043200.0,
+ "9575": 13269043200.0,
+ "9580": 13269043200.0,
+ "9585": 13269043200.0,
+ "9590": 13269043200.0,
+ "9595": 13269043200.0,
+ "9600": 13269043200.0,
+ "9605": 13269043200.0,
+ "9610": 13269043200.0,
+ "9615": 13269043200.0,
+ "9620": 13269043200.0,
+ "9625": 13269043200.0,
+ "9630": 13269043200.0,
+ "9635": 13269043200.0,
+ "9640": 13269043200.0,
+ "9645": 13269043200.0,
+ "9650": 13269043200.0,
+ "9655": 13269043200.0,
+ "9660": 13269043200.0,
+ "9665": 13269043200.0,
+ "9670": 13269043200.0,
+ "9675": 13269043200.0,
+ "9680": 13269043200.0,
+ "9685": 13269043200.0,
+ "9690": 13269043200.0,
+ "9695": 13269043200.0,
+ "9700": 13269043200.0,
+ "9705": 13269043200.0,
+ "9710": 13269043200.0,
+ "9715": 13269043200.0,
+ "9720": 13269043200.0,
+ "9725": 13269043200.0,
+ "9730": 13269043200.0,
+ "9735": 13269043200.0,
+ "9740": 13269043200.0,
+ "9745": 13269043200.0,
+ "9750": 13269043200.0,
+ "9755": 13269043200.0,
+ "9760": 13269043200.0,
+ "9765": 13269043200.0,
+ "9770": 13269043200.0,
+ "9775": 13269043200.0,
+ "9780": 13269043200.0,
+ "9785": 13269043200.0,
+ "9790": 13269043200.0,
+ "9795": 13269043200.0,
+ "9800": 13269043200.0,
+ "9805": 13269043200.0,
+ "9810": 13269043200.0,
+ "9815": 13269043200.0,
+ "9820": 13269043200.0,
+ "9825": 13269043200.0,
+ "9830": 13269043200.0,
+ "9835": 13269043200.0,
+ "9840": 13269043200.0,
+ "9845": 13269043200.0,
+ "9850": 13269043200.0,
+ "9855": 13269043200.0,
+ "9860": 13269043200.0,
+ "9865": 13269043200.0,
+ "9870": 13269043200.0,
+ "9875": 13269043200.0,
+ "9880": 13269043200.0,
+ "9885": 13269043200.0,
+ "9890": 13269043200.0,
+ "9895": 13269043200.0,
+ "9900": 13269043200.0,
+ "9905": 13269043200.0,
+ "9910": 13269043200.0,
+ "9915": 13269043200.0,
+ "9920": 13269043200.0,
+ "9925": 13269043200.0,
+ "9930": 13269043200.0,
+ "9935": 13269043200.0,
+ "9940": 13269043200.0,
+ "9945": 13269043200.0,
+ "9950": 13269043200.0,
+ "9955": 13269043200.0,
+ "9960": 13269043200.0,
+ "9965": 13269043200.0,
+ "9970": 13269043200.0,
+ "9975": 13269043200.0,
+ "9980": 13269043200.0,
+ "9985": 13269043200.0,
+ "9990": 13269043200.0,
+ "9995": 13269043200.0,
+ "10000": 13269043200.0,
+ "10005": 13269043200.0,
+ "10010": 13269043200.0,
+ "10015": 13269043200.0,
+ "10020": 13269043200.0,
+ "10025": 13269043200.0,
+ "10030": 13269043200.0,
+ "10035": 13269043200.0,
+ "10040": 13269043200.0,
+ "10045": 13269043200.0,
+ "10050": 13269043200.0,
+ "10055": 13269043200.0,
+ "10060": 13269043200.0,
+ "10065": 13269043200.0,
+ "10070": 13269043200.0,
+ "10075": 13269043200.0,
+ "10080": 13269043200.0,
+ "10085": 13269043200.0,
+ "10090": 13269043200.0,
+ "10095": 13269043200.0,
+ "10100": 13269043200.0,
+ "10105": 13269043200.0,
+ "10110": 13269043200.0,
+ "10115": 13269043200.0,
+ "10120": 13269043200.0,
+ "10125": 13269043200.0,
+ "10130": 13269043200.0,
+ "10135": 13269043200.0,
+ "10140": 13269043200.0,
+ "10145": 13269043200.0,
+ "10150": 13269043200.0,
+ "10155": 13269043200.0,
+ "10160": 13269043200.0,
+ "10165": 13269043200.0,
+ "10170": 13269043200.0,
+ "10175": 13269043200.0,
+ "10180": 13269043200.0,
+ "10185": 13269043200.0,
+ "10190": 13269043200.0,
+ "10195": 13269043200.0,
+ "10200": 13269043200.0,
+ "10205": 13269043200.0,
+ "10210": 13269043200.0,
+ "10215": 13269043200.0,
+ "10220": 13269043200.0,
+ "10225": 13269043200.0,
+ "10230": 13269043200.0,
+ "10235": 13269043200.0,
+ "10240": 13269043200.0,
+ "10245": 13269043200.0,
+ "10250": 13269043200.0,
+ "10255": 13269043200.0,
+ "10260": 13269043200.0,
+ "10265": 13269043200.0,
+ "10270": 13269043200.0,
+ "10275": 13269043200.0,
+ "10280": 13269043200.0,
+ "10285": 13269043200.0,
+ "10290": 13269043200.0,
+ "10295": 13269043200.0,
+ "10300": 13269043200.0,
+ "10305": 13269043200.0,
+ "10310": 13269043200.0,
+ "10315": 13269043200.0,
+ "10320": 13269043200.0,
+ "10325": 13269043200.0,
+ "10330": 13269043200.0,
+ "10335": 13269043200.0,
+ "10340": 13269043200.0,
+ "10345": 13269043200.0,
+ "10350": 13269043200.0,
+ "10355": 13269043200.0,
+ "10360": 13269043200.0,
+ "10365": 13269043200.0,
+ "10370": 13269043200.0,
+ "10375": 13269043200.0,
+ "10380": 13269043200.0,
+ "10385": 13269043200.0,
+ "10390": 13269043200.0,
+ "10395": 13269043200.0,
+ "10400": 13269043200.0,
+ "10405": 13269043200.0,
+ "10410": 13269043200.0,
+ "10415": 13269043200.0,
+ "10420": 13269043200.0,
+ "10425": 13269043200.0,
+ "10430": 13269043200.0,
+ "10435": 13269043200.0,
+ "10440": 13269043200.0,
+ "10445": 13269043200.0,
+ "10450": 13269043200.0,
+ "10455": 13269043200.0,
+ "10460": 13269043200.0,
+ "10465": 13269043200.0,
+ "10470": 13269043200.0,
+ "10475": 13269043200.0,
+ "10480": 13269043200.0,
+ "10485": 13269043200.0,
+ "10490": 13269043200.0,
+ "10495": 13269043200.0,
+ "10500": 13269043200.0,
+ "10505": 13269043200.0,
+ "10510": 13269043200.0,
+ "10515": 13269043200.0,
+ "10520": 13269043200.0,
+ "10525": 13269043200.0,
+ "10530": 13269043200.0,
+ "10535": 13269043200.0,
+ "10540": 13269043200.0,
+ "10545": 13269043200.0,
+ "10550": 13269043200.0,
+ "10555": 13269043200.0,
+ "10560": 13269043200.0,
+ "10565": 13269043200.0,
+ "10570": 13269043200.0,
+ "10575": 13269043200.0,
+ "10580": 13269043200.0,
+ "10585": 13269043200.0,
+ "10590": 13269043200.0,
+ "10595": 13269043200.0,
+ "10600": 13269043200.0,
+ "10605": 13269043200.0,
+ "10610": 13269043200.0,
+ "10615": 13269043200.0,
+ "10620": 13269043200.0,
+ "10625": 13269043200.0,
+ "10630": 13269043200.0,
+ "10635": 13269043200.0,
+ "10640": 13269043200.0,
+ "10645": 13269043200.0,
+ "10650": 13269043200.0,
+ "10655": 13269043200.0,
+ "10660": 13269043200.0,
+ "10665": 13269043200.0,
+ "10670": 13269043200.0,
+ "10675": 13269043200.0,
+ "10680": 13269043200.0,
+ "10685": 13269043200.0,
+ "10690": 13269043200.0,
+ "10695": 13269043200.0,
+ "10700": 13269043200.0,
+ "10705": 13269043200.0,
+ "10710": 13269043200.0,
+ "10715": 13269043200.0,
+ "10720": 13269043200.0,
+ "10725": 13269043200.0,
+ "10730": 13269043200.0,
+ "10735": 13269043200.0,
+ "10740": 13269043200.0,
+ "10745": 13269043200.0,
+ "10750": 13269043200.0,
+ "10755": 13269043200.0,
+ "10760": 13269043200.0,
+ "10765": 13269043200.0,
+ "10770": 13269043200.0,
+ "10775": 13269043200.0,
+ "10780": 13269043200.0,
+ "10785": 13269043200.0,
+ "10790": 13269043200.0,
+ "10795": 13269043200.0,
+ "10800": 13269043200.0,
+ "10805": 13269043200.0,
+ "10810": 13269043200.0,
+ "10815": 13269043200.0,
+ "10820": 13269043200.0,
+ "10825": 13269043200.0,
+ "10830": 13269043200.0,
+ "10835": 13269043200.0,
+ "10840": 13269043200.0,
+ "10845": 13269043200.0,
+ "10850": 13269043200.0,
+ "10855": 13269043200.0,
+ "10860": 13269043200.0,
+ "10865": 13269043200.0,
+ "10870": 13269043200.0,
+ "10875": 13269043200.0,
+ "10880": 13269043200.0,
+ "10885": 13269043200.0,
+ "10890": 13269043200.0,
+ "10895": 13269043200.0,
+ "10900": 13269043200.0,
+ "10905": 13269043200.0,
+ "10910": 13269043200.0,
+ "10915": 13269043200.0,
+ "10920": 13269043200.0,
+ "10925": 13269043200.0,
+ "10930": 13269043200.0,
+ "10935": 13269043200.0,
+ "10940": 13269043200.0,
+ "10945": 13269043200.0,
+ "10950": 13269043200.0,
+ "10955": 13269043200.0,
+ "10960": 13269043200.0,
+ "10965": 13269043200.0,
+ "10970": 13269043200.0,
+ "10975": 13269043200.0,
+ "10980": 13269043200.0,
+ "10985": 13269043200.0,
+ "10990": 13269043200.0,
+ "10995": 13269043200.0,
+ "11000": 13269043200.0,
+ "11005": 13269043200.0,
+ "11010": 13269043200.0,
+ "11015": 13269043200.0,
+ "11020": 13269043200.0,
+ "11025": 13269043200.0,
+ "11030": 13269043200.0,
+ "11035": 13269043200.0,
+ "11040": 13269043200.0,
+ "11045": 13269043200.0,
+ "11050": 13269043200.0,
+ "11055": 13269043200.0,
+ "11060": 13269043200.0,
+ "11065": 13269043200.0,
+ "11070": 13269043200.0,
+ "11075": 13269043200.0,
+ "11080": 13269043200.0,
+ "11085": 13269043200.0,
+ "11090": 13269043200.0,
+ "11095": 13269043200.0,
+ "11100": 13269043200.0,
+ "11105": 13269043200.0,
+ "11110": 13269043200.0,
+ "11115": 13269043200.0,
+ "11120": 13269043200.0,
+ "11125": 13269043200.0,
+ "11130": 13269043200.0,
+ "11135": 13269043200.0,
+ "11140": 13269043200.0,
+ "11145": 13269043200.0,
+ "11150": 13269043200.0,
+ "11155": 13269043200.0,
+ "11160": 13269043200.0,
+ "11165": 13269043200.0,
+ "11170": 13269043200.0,
+ "11175": 13269043200.0,
+ "11180": 13269043200.0,
+ "11185": 13269043200.0,
+ "11190": 13269043200.0,
+ "11195": 13269043200.0,
+ "11200": 13269043200.0,
+ "11205": 13269043200.0,
+ "11210": 13269043200.0,
+ "11215": 13269043200.0,
+ "11220": 13269043200.0,
+ "11225": 13269043200.0,
+ "11230": 13269043200.0,
+ "11235": 13269043200.0,
+ "11240": 13269043200.0,
+ "11245": 13269043200.0,
+ "11250": 13269043200.0,
+ "11255": 13269043200.0,
+ "11260": 13269043200.0,
+ "11265": 13269043200.0,
+ "11270": 13269043200.0,
+ "11275": 13269043200.0,
+ "11280": 13269043200.0,
+ "11285": 13269043200.0,
+ "11290": 13269043200.0,
+ "11295": 13269043200.0,
+ "11300": 13269043200.0,
+ "11305": 13269043200.0,
+ "11310": 13269043200.0,
+ "11315": 13269043200.0,
+ "11320": 13269043200.0,
+ "11325": 13269043200.0,
+ "11330": 13269043200.0,
+ "11335": 13269043200.0,
+ "11340": 13269043200.0,
+ "11345": 13269043200.0,
+ "11350": 13269043200.0,
+ "11355": 13269043200.0,
+ "11360": 13269043200.0,
+ "11365": 13269043200.0,
+ "11370": 13269043200.0,
+ "11375": 13269043200.0,
+ "11380": 13269043200.0,
+ "11385": 13269043200.0,
+ "11390": 13269043200.0,
+ "11395": 13269043200.0,
+ "11400": 13269043200.0,
+ "11405": 13269043200.0,
+ "11410": 13269043200.0,
+ "11415": 13269043200.0,
+ "11420": 13269043200.0,
+ "11425": 13269043200.0,
+ "11430": 13269043200.0,
+ "11435": 13269043200.0,
+ "11440": 13269043200.0,
+ "11445": 13269043200.0,
+ "11450": 13269043200.0,
+ "11455": 13269043200.0,
+ "11460": 13269043200.0,
+ "11465": 13269043200.0,
+ "11470": 13269043200.0,
+ "11475": 13269043200.0,
+ "11480": 13269043200.0,
+ "11485": 13269043200.0,
+ "11490": 13269043200.0,
+ "11495": 13269043200.0,
+ "11500": 13269043200.0,
+ "11505": 13269043200.0,
+ "11510": 13269043200.0,
+ "11515": 13269043200.0,
+ "11520": 13269043200.0,
+ "11525": 13269043200.0,
+ "11530": 13269043200.0,
+ "11535": 13269043200.0,
+ "11540": 13269043200.0,
+ "11545": 13269043200.0,
+ "11550": 13269043200.0,
+ "11555": 13269043200.0,
+ "11560": 13269043200.0,
+ "11565": 13269043200.0,
+ "11570": 13269043200.0,
+ "11575": 13269043200.0,
+ "11580": 13269043200.0,
+ "11585": 13269043200.0,
+ "11590": 13269043200.0,
+ "11595": 13269043200.0,
+ "11600": 13269043200.0,
+ "11605": 13269043200.0,
+ "11610": 13269043200.0,
+ "11615": 13269043200.0,
+ "11620": 13269043200.0,
+ "11625": 13269043200.0,
+ "11630": 13269043200.0,
+ "11635": 13269043200.0,
+ "11640": 13269043200.0,
+ "11645": 13269043200.0,
+ "11650": 13269043200.0,
+ "11655": 13269043200.0,
+ "11660": 13269043200.0,
+ "11665": 13269043200.0,
+ "11670": 13269043200.0,
+ "11675": 13269043200.0,
+ "11680": 13269043200.0,
+ "11685": 13269043200.0,
+ "11690": 13269043200.0,
+ "11695": 13269043200.0,
+ "11700": 13269043200.0,
+ "11705": 13269043200.0,
+ "11710": 13269043200.0,
+ "11715": 13269043200.0,
+ "11720": 13269043200.0,
+ "11725": 13269043200.0,
+ "11730": 13269043200.0,
+ "11735": 13269043200.0,
+ "11740": 13269043200.0,
+ "11745": 13269043200.0,
+ "11750": 13269043200.0,
+ "11755": 13269043200.0,
+ "11760": 13269043200.0,
+ "11765": 13269043200.0,
+ "11770": 13269043200.0,
+ "11775": 13269043200.0,
+ "11780": 13269043200.0,
+ "11785": 13269043200.0,
+ "11790": 13269043200.0,
+ "11795": 13269043200.0,
+ "11800": 13269043200.0,
+ "11805": 13269043200.0,
+ "11810": 13269043200.0,
+ "11815": 13269043200.0,
+ "11820": 13269043200.0,
+ "11825": 13269043200.0,
+ "11830": 13269043200.0,
+ "11835": 13269043200.0,
+ "11840": 13269043200.0,
+ "11845": 13269043200.0,
+ "11850": 13269043200.0,
+ "11855": 13269043200.0,
+ "11860": 13269043200.0,
+ "11865": 13269043200.0,
+ "11870": 13269043200.0,
+ "11875": 13269043200.0,
+ "11880": 13269043200.0,
+ "11885": 13269043200.0,
+ "11890": 13269043200.0,
+ "11895": 13269043200.0,
+ "11900": 13269043200.0,
+ "11905": 13269043200.0,
+ "11910": 13269043200.0,
+ "11915": 13269043200.0,
+ "11920": 13269043200.0,
+ "11925": 13269043200.0,
+ "11930": 13269043200.0,
+ "11935": 13269043200.0,
+ "11940": 13269043200.0,
+ "11945": 13269043200.0,
+ "11950": 13269043200.0,
+ "11955": 13269043200.0,
+ "11960": 13269043200.0,
+ "11965": 13269043200.0,
+ "11970": 13269043200.0,
+ "11975": 13269043200.0,
+ "11980": 13269043200.0,
+ "11985": 13269043200.0,
+ "11990": 13269043200.0,
+ "11995": 13269043200.0,
+ "12000": 13269043200.0,
+ "12005": 13269043200.0,
+ "12010": 13269043200.0,
+ "12015": 13269043200.0,
+ "12020": 13269043200.0,
+ "12025": 13269043200.0,
+ "12030": 13269043200.0,
+ "12035": 13269043200.0,
+ "12040": 13269043200.0,
+ "12045": 13269043200.0,
+ "12050": 13269043200.0,
+ "12055": 13269043200.0,
+ "12060": 13269043200.0,
+ "12065": 13269043200.0,
+ "12070": 13269043200.0,
+ "12075": 13269043200.0,
+ "12080": 13269043200.0,
+ "12085": 13269043200.0,
+ "12090": 13269043200.0,
+ "12095": 13269043200.0,
+ "12100": 13269043200.0,
+ "12105": 13269043200.0,
+ "12110": 13269043200.0,
+ "12115": 13269043200.0,
+ "12120": 13269043200.0,
+ "12125": 13269043200.0,
+ "12130": 13269043200.0,
+ "12135": 13269043200.0,
+ "12140": 13269043200.0,
+ "12145": 13269043200.0,
+ "12150": 13269043200.0,
+ "12155": 13269043200.0,
+ "12160": 13269043200.0,
+ "12165": 13269043200.0,
+ "12170": 13269043200.0,
+ "12175": 13269043200.0,
+ "12180": 13269043200.0,
+ "12185": 13269043200.0,
+ "12190": 13269043200.0,
+ "12195": 13269043200.0,
+ "12200": 13269043200.0,
+ "12205": 13269043200.0,
+ "12210": 13269043200.0,
+ "12215": 13269043200.0,
+ "12220": 13269043200.0,
+ "12225": 13269043200.0,
+ "12230": 13269043200.0,
+ "12235": 13269043200.0,
+ "12240": 13269043200.0,
+ "12245": 13269043200.0,
+ "12250": 13269043200.0,
+ "12255": 13269043200.0,
+ "12260": 13269043200.0,
+ "12265": 13269043200.0,
+ "12270": 13269043200.0,
+ "12275": 13269043200.0,
+ "12280": 13269043200.0,
+ "12285": 13269043200.0,
+ "12290": 13269043200.0,
+ "12295": 13269043200.0,
+ "12300": 13269043200.0,
+ "12305": 13269043200.0,
+ "12310": 13269043200.0,
+ "12315": 13269043200.0,
+ "12320": 13269043200.0,
+ "12325": 13269043200.0,
+ "12330": 13269043200.0,
+ "12335": 13269043200.0,
+ "12340": 13269043200.0,
+ "12345": 13269043200.0,
+ "12350": 13269043200.0,
+ "12355": 13269043200.0,
+ "12360": 13269043200.0,
+ "12365": 13269043200.0,
+ "12370": 13269043200.0,
+ "12375": 13269043200.0,
+ "12380": 13269043200.0,
+ "12385": 13269043200.0,
+ "12390": 13269043200.0,
+ "12395": 13269043200.0,
+ "12400": 13269043200.0,
+ "12405": 13269043200.0,
+ "12410": 13269043200.0,
+ "12415": 13269043200.0,
+ "12420": 13269043200.0,
+ "12425": 13269043200.0,
+ "12430": 13269043200.0,
+ "12435": 13269043200.0,
+ "12440": 13269043200.0,
+ "12445": 13269043200.0,
+ "12450": 13269043200.0,
+ "12455": 13269043200.0,
+ "12460": 13269043200.0,
+ "12465": 13269043200.0,
+ "12470": 13269043200.0,
+ "12475": 13269043200.0,
+ "12480": 13269043200.0,
+ "12485": 13269043200.0,
+ "12490": 13269043200.0,
+ "12495": 13269043200.0,
+ "12500": 13269043200.0,
+ "12505": 13269043200.0,
+ "12510": 13269043200.0,
+ "12515": 13269043200.0,
+ "12520": 13269043200.0,
+ "12525": 13269043200.0,
+ "12530": 13269043200.0,
+ "12535": 13269043200.0,
+ "12540": 13269043200.0,
+ "12545": 13269043200.0,
+ "12550": 13269043200.0,
+ "12555": 13269043200.0,
+ "12560": 13269043200.0,
+ "12565": 13269043200.0,
+ "12570": 13269043200.0,
+ "12575": 13269043200.0,
+ "12580": 13269043200.0,
+ "12585": 13269043200.0,
+ "12590": 13269043200.0,
+ "12595": 13269043200.0,
+ "12600": 13269043200.0,
+ "12605": 13269043200.0,
+ "12610": 13269043200.0,
+ "12615": 13269043200.0,
+ "12620": 13269043200.0,
+ "12625": 13269043200.0,
+ "12630": 13269043200.0,
+ "12635": 13269043200.0,
+ "12640": 13269043200.0,
+ "12645": 13269043200.0,
+ "12650": 13269043200.0,
+ "12655": 13269043200.0,
+ "12660": 13269043200.0,
+ "12665": 13269043200.0,
+ "12670": 13269043200.0,
+ "12675": 13269043200.0,
+ "12680": 13269043200.0,
+ "12685": 13269043200.0,
+ "12690": 13269043200.0,
+ "12695": 13269043200.0,
+ "12700": 13269043200.0,
+ "12705": 13269043200.0,
+ "12710": 13269043200.0,
+ "12715": 13269043200.0,
+ "12720": "nan",
+ "12725": "nan",
+ "12730": "nan",
+ "12735": "nan",
+ "12740": "nan",
+ "12745": "nan",
+ "12750": "nan",
+ "12755": "nan",
+ "12760": "nan",
+ "12765": "nan",
+ "12770": "nan",
+ "12775": "nan",
+ "12780": "nan",
+ "12785": "nan",
+ "12790": "nan",
+ "12795": "nan",
+ "12800": "nan",
+ "12805": "nan",
+ "12810": "nan",
+ "12815": "nan",
+ "12820": "nan",
+ "12825": "nan",
+ "12830": "nan",
+ "12835": "nan",
+ "12840": "nan",
+ "12845": "nan",
+ "12850": "nan",
+ "12855": "nan",
+ "12860": "nan",
+ "12865": "nan",
+ "12870": "nan",
+ "12875": "nan",
+ "12880": "nan",
+ "12885": "nan",
+ "12890": "nan",
+ "12895": "nan",
+ "12900": "nan",
+ "12905": "nan",
+ "12910": "nan",
+ "12915": "nan",
+ "12920": "nan",
+ "12925": "nan",
+ "12930": "nan",
+ "12935": "nan",
+ "12940": "nan",
+ "12945": "nan",
+ "12950": "nan",
+ "12955": "nan",
+ "12960": "nan",
+ "12965": "nan",
+ "12970": "nan",
+ "12975": "nan",
+ "12980": "nan",
+ "12985": "nan",
+ "12990": "nan",
+ "12995": "nan",
+ "13000": "nan"
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": 27658764288.0,
+ "5": 28157253632.0,
+ "10": 28157253632.0,
+ "15": 28157253632.0,
+ "20": 28157253632.0,
+ "25": 28157253632.0,
+ "30": 28157253632.0,
+ "35": 28157253632.0,
+ "40": 28157253632.0,
+ "45": 28157253632.0,
+ "50": 28157253632.0,
+ "55": 28157253632.0,
+ "60": 28157253632.0,
+ "65": 28157253632.0,
+ "70": 28157253632.0,
+ "75": 28157253632.0,
+ "80": 28157253632.0,
+ "85": 28157253632.0,
+ "90": 28157253632.0,
+ "95": 28157253632.0,
+ "100": 28157253632.0,
+ "105": 28157253632.0,
+ "110": 28157253632.0,
+ "115": 28157253632.0,
+ "120": 28157253632.0,
+ "125": 28157253632.0,
+ "130": 28157253632.0,
+ "135": 28157253632.0,
+ "140": 28157253632.0,
+ "145": 28157253632.0,
+ "150": 28157253632.0,
+ "155": 28157253632.0,
+ "160": 28157253632.0,
+ "165": 28157253632.0,
+ "170": 28157253632.0,
+ "175": 28157253632.0,
+ "180": 28157253632.0,
+ "185": 28157253632.0,
+ "190": 28157253632.0,
+ "195": 28157253632.0,
+ "200": 28157253632.0,
+ "205": 28157253632.0,
+ "210": 28157253632.0,
+ "215": 28157253632.0,
+ "220": 28157253632.0,
+ "225": 28157253632.0,
+ "230": 28157253632.0,
+ "235": 28157253632.0,
+ "240": 28157253632.0,
+ "245": 28157253632.0,
+ "250": 28157253632.0,
+ "255": 28157253632.0,
+ "260": 28157253632.0,
+ "265": 28157253632.0,
+ "270": 28157253632.0,
+ "275": 28157253632.0,
+ "280": 28157253632.0,
+ "285": 28157253632.0,
+ "290": 28157253632.0,
+ "295": 28157253632.0,
+ "300": 28157253632.0,
+ "305": 28157253632.0,
+ "310": 28157253632.0,
+ "315": 28157253632.0,
+ "320": 28157253632.0,
+ "325": 28157253632.0,
+ "330": 28157253632.0,
+ "335": 28157253632.0,
+ "340": 28157253632.0,
+ "345": 28157253632.0,
+ "350": 28157253632.0,
+ "355": 28157253632.0,
+ "360": 28157253632.0,
+ "365": 28157253632.0,
+ "370": 28157253632.0,
+ "375": 28157253632.0,
+ "380": 28157253632.0,
+ "385": 28157253632.0,
+ "390": 28157253632.0,
+ "395": 28157253632.0,
+ "400": 28157253632.0,
+ "405": 28157253632.0,
+ "410": 28157253632.0,
+ "415": 28157253632.0,
+ "420": 28157253632.0,
+ "425": 28157253632.0,
+ "430": 28157253632.0,
+ "435": 28157253632.0,
+ "440": 28157253632.0,
+ "445": 28157253632.0,
+ "450": 28157253632.0,
+ "455": 28157253632.0,
+ "460": 28157253632.0,
+ "465": 28157253632.0,
+ "470": 28157253632.0,
+ "475": 28157253632.0,
+ "480": 28157253632.0,
+ "485": 28157253632.0,
+ "490": 28157253632.0,
+ "495": 28157253632.0,
+ "500": 28157253632.0,
+ "505": 28157253632.0,
+ "510": 28157253632.0,
+ "515": 28157253632.0,
+ "520": 28157253632.0,
+ "525": 28157253632.0,
+ "530": 28157253632.0,
+ "535": 28157253632.0,
+ "540": 28157253632.0,
+ "545": 28157253632.0,
+ "550": 28157253632.0,
+ "555": 28157253632.0,
+ "560": 28157253632.0,
+ "565": 28157253632.0,
+ "570": 28157253632.0,
+ "575": 28157253632.0,
+ "580": 28157253632.0,
+ "585": 28157253632.0,
+ "590": 28157253632.0,
+ "595": 28157253632.0,
+ "600": 28157253632.0,
+ "605": 28157253632.0,
+ "610": 28157253632.0,
+ "615": 28157253632.0,
+ "620": 28157253632.0,
+ "625": 28157253632.0,
+ "630": 28157253632.0,
+ "635": 28157253632.0,
+ "640": 28157253632.0,
+ "645": 28157253632.0,
+ "650": 28157253632.0,
+ "655": 28157253632.0,
+ "660": 28157253632.0,
+ "665": 28157253632.0,
+ "670": 28157253632.0,
+ "675": 28157253632.0,
+ "680": 28157253632.0,
+ "685": 28157253632.0,
+ "690": 28157253632.0,
+ "695": 28157253632.0,
+ "700": 28157253632.0,
+ "705": 28157253632.0,
+ "710": 28157253632.0,
+ "715": 28157253632.0,
+ "720": 28157253632.0,
+ "725": 28157253632.0,
+ "730": 28157253632.0,
+ "735": 28157253632.0,
+ "740": 28157253632.0,
+ "745": 28157253632.0,
+ "750": 28157253632.0,
+ "755": 28157253632.0,
+ "760": 28157253632.0,
+ "765": 28157253632.0,
+ "770": 28157253632.0,
+ "775": 28157253632.0,
+ "780": 28157253632.0,
+ "785": 28157253632.0,
+ "790": 28157253632.0,
+ "795": 28157253632.0,
+ "800": 28157253632.0,
+ "805": 28157253632.0,
+ "810": 28157253632.0,
+ "815": 28157253632.0,
+ "820": 28157253632.0,
+ "825": 28157253632.0,
+ "830": 28157253632.0,
+ "835": 28157253632.0,
+ "840": 28157253632.0,
+ "845": 28157253632.0,
+ "850": 28157253632.0,
+ "855": 28157253632.0,
+ "860": 28157253632.0,
+ "865": 28157253632.0,
+ "870": 28157253632.0,
+ "875": 28157253632.0,
+ "880": 28157253632.0,
+ "885": 28157253632.0,
+ "890": 28157253632.0,
+ "895": 28157253632.0,
+ "900": 28157253632.0,
+ "905": 28157253632.0,
+ "910": 28157253632.0,
+ "915": 28157253632.0,
+ "920": 28157253632.0,
+ "925": 28157253632.0,
+ "930": 28157253632.0,
+ "935": 28157253632.0,
+ "940": 28157253632.0,
+ "945": 28157253632.0,
+ "950": 28157253632.0,
+ "955": 28157253632.0,
+ "960": 28157253632.0,
+ "965": 28157253632.0,
+ "970": 28157253632.0,
+ "975": 28157253632.0,
+ "980": 28157253632.0,
+ "985": 28157253632.0,
+ "990": 28157253632.0,
+ "995": 28157253632.0,
+ "1000": 28157253632.0,
+ "1005": 28157253632.0,
+ "1010": 28157253632.0,
+ "1015": 28157253632.0,
+ "1020": 28157253632.0,
+ "1025": 28157253632.0,
+ "1030": 28157253632.0,
+ "1035": 28157253632.0,
+ "1040": 28157253632.0,
+ "1045": 28157253632.0,
+ "1050": 28157253632.0,
+ "1055": 28157253632.0,
+ "1060": 28157253632.0,
+ "1065": 28157253632.0,
+ "1070": 28157253632.0,
+ "1075": 28157253632.0,
+ "1080": 28157253632.0,
+ "1085": 28157253632.0,
+ "1090": 28157253632.0,
+ "1095": 28157253632.0,
+ "1100": 28157253632.0,
+ "1105": 28157253632.0,
+ "1110": 28157253632.0,
+ "1115": 28157253632.0,
+ "1120": 28157253632.0,
+ "1125": 28157253632.0,
+ "1130": 28157253632.0,
+ "1135": 28157253632.0,
+ "1140": 28157253632.0,
+ "1145": 28157253632.0,
+ "1150": 28157253632.0,
+ "1155": 28157253632.0,
+ "1160": 28157253632.0,
+ "1165": 28157253632.0,
+ "1170": 28157253632.0,
+ "1175": 28157253632.0,
+ "1180": 28157253632.0,
+ "1185": 28157253632.0,
+ "1190": 28157253632.0,
+ "1195": 28157253632.0,
+ "1200": 28157253632.0,
+ "1205": 28157253632.0,
+ "1210": 28157253632.0,
+ "1215": 28157253632.0,
+ "1220": 28157253632.0,
+ "1225": 28157253632.0,
+ "1230": 28157253632.0,
+ "1235": 28157253632.0,
+ "1240": 28157253632.0,
+ "1245": 28157253632.0,
+ "1250": 28157253632.0,
+ "1255": 28157253632.0,
+ "1260": 28157253632.0,
+ "1265": 28157253632.0,
+ "1270": 28157253632.0,
+ "1275": 28157253632.0,
+ "1280": 28157253632.0,
+ "1285": 28157253632.0,
+ "1290": 28157253632.0,
+ "1295": 28157253632.0,
+ "1300": 28157253632.0,
+ "1305": 28157253632.0,
+ "1310": 28157253632.0,
+ "1315": 28157253632.0,
+ "1320": 28157253632.0,
+ "1325": 28157253632.0,
+ "1330": 28157253632.0,
+ "1335": 28157253632.0,
+ "1340": 28157253632.0,
+ "1345": 28157253632.0,
+ "1350": 28157253632.0,
+ "1355": 28157253632.0,
+ "1360": 28157253632.0,
+ "1365": 28157253632.0,
+ "1370": 28157253632.0,
+ "1375": 28157253632.0,
+ "1380": 28157253632.0,
+ "1385": 28157253632.0,
+ "1390": 28157253632.0,
+ "1395": 28157253632.0,
+ "1400": 28157253632.0,
+ "1405": 28157253632.0,
+ "1410": 28157253632.0,
+ "1415": 28157253632.0,
+ "1420": 28157253632.0,
+ "1425": 28157253632.0,
+ "1430": 28157253632.0,
+ "1435": 28157253632.0,
+ "1440": 28157253632.0,
+ "1445": 28157253632.0,
+ "1450": 28157253632.0,
+ "1455": 28157253632.0,
+ "1460": 28157253632.0,
+ "1465": 28157253632.0,
+ "1470": 28157253632.0,
+ "1475": 28157253632.0,
+ "1480": 28157253632.0,
+ "1485": 28157253632.0,
+ "1490": 28157253632.0,
+ "1495": 28157253632.0,
+ "1500": 28157253632.0,
+ "1505": 28157253632.0,
+ "1510": 28157253632.0,
+ "1515": 28157253632.0,
+ "1520": 28157253632.0,
+ "1525": 28157253632.0,
+ "1530": 28157253632.0,
+ "1535": 28157253632.0,
+ "1540": 28157253632.0,
+ "1545": 28157253632.0,
+ "1550": 28157253632.0,
+ "1555": 28157253632.0,
+ "1560": 28157253632.0,
+ "1565": 28157253632.0,
+ "1570": 28157253632.0,
+ "1575": 28157253632.0,
+ "1580": 28157253632.0,
+ "1585": 28157253632.0,
+ "1590": 28157253632.0,
+ "1595": 28157253632.0,
+ "1600": 28157253632.0,
+ "1605": 28157253632.0,
+ "1610": 28157253632.0,
+ "1615": 28157253632.0,
+ "1620": 28157253632.0,
+ "1625": 28157253632.0,
+ "1630": 28157253632.0,
+ "1635": 28157253632.0,
+ "1640": 28157253632.0,
+ "1645": 28157253632.0,
+ "1650": 28157253632.0,
+ "1655": 28157253632.0,
+ "1660": 28157253632.0,
+ "1665": 28157253632.0,
+ "1670": 28157253632.0,
+ "1675": 28157253632.0,
+ "1680": 28157253632.0,
+ "1685": 28157253632.0,
+ "1690": 28157253632.0,
+ "1695": 28157253632.0,
+ "1700": 28157253632.0,
+ "1705": 28157253632.0,
+ "1710": 28157253632.0,
+ "1715": 28157253632.0,
+ "1720": 28157253632.0,
+ "1725": 28157253632.0,
+ "1730": 28157253632.0,
+ "1735": 28157253632.0,
+ "1740": 28157253632.0,
+ "1745": 28157253632.0,
+ "1750": 28157253632.0,
+ "1755": 28157253632.0,
+ "1760": 28157253632.0,
+ "1765": 28157253632.0,
+ "1770": 28157253632.0,
+ "1775": 28157253632.0,
+ "1780": 28157253632.0,
+ "1785": 28157253632.0,
+ "1790": 28157253632.0,
+ "1795": 28157253632.0,
+ "1800": 28157253632.0,
+ "1805": 28157253632.0,
+ "1810": 28157253632.0,
+ "1815": 28157253632.0,
+ "1820": 28157253632.0,
+ "1825": 28157253632.0,
+ "1830": 28157253632.0,
+ "1835": 28157253632.0,
+ "1840": 28157253632.0,
+ "1845": 28157253632.0,
+ "1850": 28157253632.0,
+ "1855": 28157253632.0,
+ "1860": 28157253632.0,
+ "1865": 28157253632.0,
+ "1870": 28157253632.0,
+ "1875": 28157253632.0,
+ "1880": 28157253632.0,
+ "1885": 28157253632.0,
+ "1890": 28157253632.0,
+ "1895": 28157253632.0,
+ "1900": 28157253632.0,
+ "1905": 28157253632.0,
+ "1910": 28157253632.0,
+ "1915": 28157253632.0,
+ "1920": 28157253632.0,
+ "1925": 28157253632.0,
+ "1930": 28157253632.0,
+ "1935": 28157253632.0,
+ "1940": 28157253632.0,
+ "1945": 28157253632.0,
+ "1950": 28157253632.0,
+ "1955": 28157253632.0,
+ "1960": 28157253632.0,
+ "1965": 28157253632.0,
+ "1970": 28157253632.0,
+ "1975": 28157253632.0,
+ "1980": 28157253632.0,
+ "1985": 28157253632.0,
+ "1990": 28157253632.0,
+ "1995": 28157253632.0,
+ "2000": 28157253632.0,
+ "2005": 28157253632.0,
+ "2010": 28157253632.0,
+ "2015": 28157253632.0,
+ "2020": 28157253632.0,
+ "2025": 28157253632.0,
+ "2030": 28157253632.0,
+ "2035": 28157253632.0,
+ "2040": 28157253632.0,
+ "2045": 28157253632.0,
+ "2050": 28157253632.0,
+ "2055": 28157253632.0,
+ "2060": 28157253632.0,
+ "2065": 28157253632.0,
+ "2070": 28157253632.0,
+ "2075": 28157253632.0,
+ "2080": 28157253632.0,
+ "2085": 28157253632.0,
+ "2090": 28157253632.0,
+ "2095": 28157253632.0,
+ "2100": 28157253632.0,
+ "2105": 28157253632.0,
+ "2110": 28157253632.0,
+ "2115": 28157253632.0,
+ "2120": 28157253632.0,
+ "2125": 28157253632.0,
+ "2130": 28157253632.0,
+ "2135": 28157253632.0,
+ "2140": 28157253632.0,
+ "2145": 28157253632.0,
+ "2150": 28157253632.0,
+ "2155": 28157253632.0,
+ "2160": 28157253632.0,
+ "2165": 28157253632.0,
+ "2170": 28157253632.0,
+ "2175": 28157253632.0,
+ "2180": 28157253632.0,
+ "2185": 28157253632.0,
+ "2190": 28157253632.0,
+ "2195": 28157253632.0,
+ "2200": 28157253632.0,
+ "2205": 28157253632.0,
+ "2210": 28157253632.0,
+ "2215": 28157253632.0,
+ "2220": 28157253632.0,
+ "2225": 28157253632.0,
+ "2230": 28157253632.0,
+ "2235": 28157253632.0,
+ "2240": 28157253632.0,
+ "2245": 28157253632.0,
+ "2250": 28157253632.0,
+ "2255": 28157253632.0,
+ "2260": 28157253632.0,
+ "2265": 28157253632.0,
+ "2270": 28157253632.0,
+ "2275": 28157253632.0,
+ "2280": 28157253632.0,
+ "2285": 28157253632.0,
+ "2290": 28157253632.0,
+ "2295": 28157253632.0,
+ "2300": 28157253632.0,
+ "2305": 28157253632.0,
+ "2310": 28157253632.0,
+ "2315": 28157253632.0,
+ "2320": 28157253632.0,
+ "2325": 28157253632.0,
+ "2330": 28157253632.0,
+ "2335": 28157253632.0,
+ "2340": 28157253632.0,
+ "2345": 28157253632.0,
+ "2350": 28157253632.0,
+ "2355": 28157253632.0,
+ "2360": 28157253632.0,
+ "2365": 28157253632.0,
+ "2370": 28157253632.0,
+ "2375": 28157253632.0,
+ "2380": 28157253632.0,
+ "2385": 28157253632.0,
+ "2390": 28157253632.0,
+ "2395": 28157253632.0,
+ "2400": 28157253632.0,
+ "2405": 28157253632.0,
+ "2410": 28157253632.0,
+ "2415": 28157253632.0,
+ "2420": 28157253632.0,
+ "2425": 28157253632.0,
+ "2430": 28157253632.0,
+ "2435": 28157253632.0,
+ "2440": 28157253632.0,
+ "2445": 28157253632.0,
+ "2450": 28157253632.0,
+ "2455": 28157253632.0,
+ "2460": 28157253632.0,
+ "2465": 28157253632.0,
+ "2470": 28157253632.0,
+ "2475": 28157253632.0,
+ "2480": 28157253632.0,
+ "2485": 28157253632.0,
+ "2490": 28157253632.0,
+ "2495": 28157253632.0,
+ "2500": 28157253632.0,
+ "2505": 28157253632.0,
+ "2510": 28157253632.0,
+ "2515": 28157253632.0,
+ "2520": 28157253632.0,
+ "2525": 28157253632.0,
+ "2530": 28157253632.0,
+ "2535": 28157253632.0,
+ "2540": 28157253632.0,
+ "2545": 28157253632.0,
+ "2550": 28157253632.0,
+ "2555": 28157253632.0,
+ "2560": 28157253632.0,
+ "2565": 28157253632.0,
+ "2570": 28157253632.0,
+ "2575": 28157253632.0,
+ "2580": 28157253632.0,
+ "2585": 28157253632.0,
+ "2590": 28157253632.0,
+ "2595": 28157253632.0,
+ "2600": 28157253632.0,
+ "2605": 28157253632.0,
+ "2610": 28157253632.0,
+ "2615": 28157253632.0,
+ "2620": 28157253632.0,
+ "2625": 28157253632.0,
+ "2630": 28157253632.0,
+ "2635": 28157253632.0,
+ "2640": 28157253632.0,
+ "2645": 28157253632.0,
+ "2650": 28157253632.0,
+ "2655": 28157253632.0,
+ "2660": 28157253632.0,
+ "2665": 28157253632.0,
+ "2670": 28157253632.0,
+ "2675": 28157253632.0,
+ "2680": 28157253632.0,
+ "2685": 28157253632.0,
+ "2690": 28157253632.0,
+ "2695": 28157253632.0,
+ "2700": 28157253632.0,
+ "2705": 28157253632.0,
+ "2710": 28157253632.0,
+ "2715": 28157253632.0,
+ "2720": 28157253632.0,
+ "2725": 28157253632.0,
+ "2730": 28157253632.0,
+ "2735": 28157253632.0,
+ "2740": 28157253632.0,
+ "2745": 28157253632.0,
+ "2750": 28157253632.0,
+ "2755": 28157253632.0,
+ "2760": 28157253632.0,
+ "2765": 28157253632.0,
+ "2770": 28157253632.0,
+ "2775": 28157253632.0,
+ "2780": 28157253632.0,
+ "2785": 28157253632.0,
+ "2790": 28157253632.0,
+ "2795": 28157253632.0,
+ "2800": 28157253632.0,
+ "2805": 28157253632.0,
+ "2810": 28157253632.0,
+ "2815": 28157253632.0,
+ "2820": 28157253632.0,
+ "2825": 28157253632.0,
+ "2830": 28157253632.0,
+ "2835": 28157253632.0,
+ "2840": 28157253632.0,
+ "2845": 28157253632.0,
+ "2850": 28157253632.0,
+ "2855": 28157253632.0,
+ "2860": 28157253632.0,
+ "2865": 28157253632.0,
+ "2870": 28157253632.0,
+ "2875": 28157253632.0,
+ "2880": 28157253632.0,
+ "2885": 28157253632.0,
+ "2890": 28157253632.0,
+ "2895": 28157253632.0,
+ "2900": 28157253632.0,
+ "2905": 28157253632.0,
+ "2910": 28157253632.0,
+ "2915": 28157253632.0,
+ "2920": 28157253632.0,
+ "2925": 28157253632.0,
+ "2930": 28157253632.0,
+ "2935": 28157253632.0,
+ "2940": 28157253632.0,
+ "2945": 28157253632.0,
+ "2950": 28157253632.0,
+ "2955": 28157253632.0,
+ "2960": 28157253632.0,
+ "2965": 28157253632.0,
+ "2970": 28157253632.0,
+ "2975": 28157253632.0,
+ "2980": 28157253632.0,
+ "2985": 28157253632.0,
+ "2990": 28157253632.0,
+ "2995": 28157253632.0,
+ "3000": 28157253632.0,
+ "3005": 28157253632.0,
+ "3010": 28157253632.0,
+ "3015": 28157253632.0,
+ "3020": 28157253632.0,
+ "3025": 28157253632.0,
+ "3030": 28157253632.0,
+ "3035": 28157253632.0,
+ "3040": 28157253632.0,
+ "3045": 28157253632.0,
+ "3050": 28157253632.0,
+ "3055": 28157253632.0,
+ "3060": 28157253632.0,
+ "3065": 28157253632.0,
+ "3070": 28157253632.0,
+ "3075": 28157253632.0,
+ "3080": 28157253632.0,
+ "3085": 28157253632.0,
+ "3090": 28157253632.0,
+ "3095": 28157253632.0,
+ "3100": 28157253632.0,
+ "3105": 28157253632.0,
+ "3110": 28157253632.0,
+ "3115": 28157253632.0,
+ "3120": 28157253632.0,
+ "3125": 28157253632.0,
+ "3130": 28157253632.0,
+ "3135": 28157253632.0,
+ "3140": 28157253632.0,
+ "3145": 28157253632.0,
+ "3150": 28157253632.0,
+ "3155": 28157253632.0,
+ "3160": 28157253632.0,
+ "3165": 28157253632.0,
+ "3170": 28157253632.0,
+ "3175": 28157253632.0,
+ "3180": 28157253632.0,
+ "3185": 28157253632.0,
+ "3190": 28157253632.0,
+ "3195": 28157253632.0,
+ "3200": 28157253632.0,
+ "3205": 28157253632.0,
+ "3210": 28157253632.0,
+ "3215": 28157253632.0,
+ "3220": 28157253632.0,
+ "3225": 28157253632.0,
+ "3230": 28157253632.0,
+ "3235": 28157253632.0,
+ "3240": 28157253632.0,
+ "3245": 28157253632.0,
+ "3250": 28157253632.0,
+ "3255": 28157253632.0,
+ "3260": 28157253632.0,
+ "3265": 28157253632.0,
+ "3270": 28157253632.0,
+ "3275": 28157253632.0,
+ "3280": 28157253632.0,
+ "3285": 28157253632.0,
+ "3290": 28157253632.0,
+ "3295": 28157253632.0,
+ "3300": 28157253632.0,
+ "3305": 28157253632.0,
+ "3310": 28157253632.0,
+ "3315": 28157253632.0,
+ "3320": 28157253632.0,
+ "3325": 28157253632.0,
+ "3330": 28157253632.0,
+ "3335": 28157253632.0,
+ "3340": 28157253632.0,
+ "3345": 28157253632.0,
+ "3350": 28157253632.0,
+ "3355": 28157253632.0,
+ "3360": 28157253632.0,
+ "3365": 28157253632.0,
+ "3370": 28157253632.0,
+ "3375": 28157253632.0,
+ "3380": 28157253632.0,
+ "3385": 28157253632.0,
+ "3390": 28157253632.0,
+ "3395": 28157253632.0,
+ "3400": 28157253632.0,
+ "3405": 28157253632.0,
+ "3410": 28157253632.0,
+ "3415": 28157253632.0,
+ "3420": 28157253632.0,
+ "3425": 28157253632.0,
+ "3430": 28157253632.0,
+ "3435": 28157253632.0,
+ "3440": 28157253632.0,
+ "3445": 28157253632.0,
+ "3450": 28157253632.0,
+ "3455": 28157253632.0,
+ "3460": 28157253632.0,
+ "3465": 28157253632.0,
+ "3470": 28157253632.0,
+ "3475": 28157253632.0,
+ "3480": 28157253632.0,
+ "3485": 28157253632.0,
+ "3490": 28157253632.0,
+ "3495": 28157253632.0,
+ "3500": 28157253632.0,
+ "3505": 28157253632.0,
+ "3510": 28157253632.0,
+ "3515": 28157253632.0,
+ "3520": 28157253632.0,
+ "3525": 28157253632.0,
+ "3530": 28157253632.0,
+ "3535": 28157253632.0,
+ "3540": 28157253632.0,
+ "3545": 28157253632.0,
+ "3550": 28157253632.0,
+ "3555": 28157253632.0,
+ "3560": 28157253632.0,
+ "3565": 28157253632.0,
+ "3570": 28157253632.0,
+ "3575": 28157253632.0,
+ "3580": 28157253632.0,
+ "3585": 28157253632.0,
+ "3590": 28157253632.0,
+ "3595": 28157253632.0,
+ "3600": 28157253632.0,
+ "3605": 28157253632.0,
+ "3610": 28157253632.0,
+ "3615": 28157253632.0,
+ "3620": 28157253632.0,
+ "3625": 28157253632.0,
+ "3630": 28157253632.0,
+ "3635": 28157253632.0,
+ "3640": 28157253632.0,
+ "3645": 28157253632.0,
+ "3650": 28157253632.0,
+ "3655": 28157253632.0,
+ "3660": 28157253632.0,
+ "3665": 28157253632.0,
+ "3670": 28157253632.0,
+ "3675": 28157253632.0,
+ "3680": 28157253632.0,
+ "3685": 28157253632.0,
+ "3690": 28157253632.0,
+ "3695": 28157253632.0,
+ "3700": 28157253632.0,
+ "3705": 28157253632.0,
+ "3710": 28157253632.0,
+ "3715": 28157253632.0,
+ "3720": 28157253632.0,
+ "3725": 28157253632.0,
+ "3730": 28157253632.0,
+ "3735": 28157253632.0,
+ "3740": 28157253632.0,
+ "3745": 28157253632.0,
+ "3750": 28157253632.0,
+ "3755": 28157253632.0,
+ "3760": 28157253632.0,
+ "3765": 28157253632.0,
+ "3770": 28157253632.0,
+ "3775": 28157253632.0,
+ "3780": 28157253632.0,
+ "3785": 28157253632.0,
+ "3790": 28157253632.0,
+ "3795": 28157253632.0,
+ "3800": 28157253632.0,
+ "3805": 28157253632.0,
+ "3810": 28157253632.0,
+ "3815": 28157253632.0,
+ "3820": 28157253632.0,
+ "3825": 28157253632.0,
+ "3830": 28157253632.0,
+ "3835": 28157253632.0,
+ "3840": 28157253632.0,
+ "3845": 28157253632.0,
+ "3850": 28157253632.0,
+ "3855": 28157253632.0,
+ "3860": 28157253632.0,
+ "3865": 28157253632.0,
+ "3870": 28157253632.0,
+ "3875": 28157253632.0,
+ "3880": 28157253632.0,
+ "3885": 28157253632.0,
+ "3890": 28157253632.0,
+ "3895": 28157253632.0,
+ "3900": 28157253632.0,
+ "3905": 28157253632.0,
+ "3910": 28157253632.0,
+ "3915": 28157253632.0,
+ "3920": 28157253632.0,
+ "3925": 28157253632.0,
+ "3930": 28157253632.0,
+ "3935": 28157253632.0,
+ "3940": 28157253632.0,
+ "3945": 28157253632.0,
+ "3950": 28157253632.0,
+ "3955": 28157253632.0,
+ "3960": 28157253632.0,
+ "3965": 28157253632.0,
+ "3970": 28157253632.0,
+ "3975": 28157253632.0,
+ "3980": 28157253632.0,
+ "3985": 28157253632.0,
+ "3990": 28157253632.0,
+ "3995": 28157253632.0,
+ "4000": 28157253632.0,
+ "4005": 28157253632.0,
+ "4010": 28157253632.0,
+ "4015": 28157253632.0,
+ "4020": 28157253632.0,
+ "4025": 28157253632.0,
+ "4030": 28157253632.0,
+ "4035": 28157253632.0,
+ "4040": 28157253632.0,
+ "4045": 28157253632.0,
+ "4050": 28157253632.0,
+ "4055": 28157253632.0,
+ "4060": 28157253632.0,
+ "4065": 28157253632.0,
+ "4070": 28157253632.0,
+ "4075": 28157253632.0,
+ "4080": 28157253632.0,
+ "4085": 28157253632.0,
+ "4090": 28157253632.0,
+ "4095": 28157253632.0,
+ "4100": 28157253632.0,
+ "4105": 28157253632.0,
+ "4110": 28157253632.0,
+ "4115": 28157253632.0,
+ "4120": 28157253632.0,
+ "4125": 28157253632.0,
+ "4130": 28157253632.0,
+ "4135": 28157253632.0,
+ "4140": 28157253632.0,
+ "4145": 28157253632.0,
+ "4150": 28157253632.0,
+ "4155": 28157253632.0,
+ "4160": 28157253632.0,
+ "4165": 28157253632.0,
+ "4170": 28157253632.0,
+ "4175": 28157253632.0,
+ "4180": 28157253632.0,
+ "4185": 28157253632.0,
+ "4190": 28157253632.0,
+ "4195": 28157253632.0,
+ "4200": 28157253632.0,
+ "4205": 28157253632.0,
+ "4210": 28157253632.0,
+ "4215": 28157253632.0,
+ "4220": 28157253632.0,
+ "4225": 28157253632.0,
+ "4230": 28157253632.0,
+ "4235": 28157253632.0,
+ "4240": 28157253632.0,
+ "4245": 28157253632.0,
+ "4250": 28157253632.0,
+ "4255": 28157253632.0,
+ "4260": 28157253632.0,
+ "4265": 28157253632.0,
+ "4270": 28157253632.0,
+ "4275": 28157253632.0,
+ "4280": 28157253632.0,
+ "4285": 28157253632.0,
+ "4290": 28157253632.0,
+ "4295": 28157253632.0,
+ "4300": 28157253632.0,
+ "4305": 28157253632.0,
+ "4310": 28157253632.0,
+ "4315": 28157253632.0,
+ "4320": 28157253632.0,
+ "4325": 28157253632.0,
+ "4330": 28157253632.0,
+ "4335": 28157253632.0,
+ "4340": 28157253632.0,
+ "4345": 28157253632.0,
+ "4350": 28157253632.0,
+ "4355": 28157253632.0,
+ "4360": 28157253632.0,
+ "4365": 28157253632.0,
+ "4370": 28157253632.0,
+ "4375": 28157253632.0,
+ "4380": 28157253632.0,
+ "4385": 28157253632.0,
+ "4390": 28157253632.0,
+ "4395": 28157253632.0,
+ "4400": 28157253632.0,
+ "4405": 28157253632.0,
+ "4410": 28157253632.0,
+ "4415": 28157253632.0,
+ "4420": 28157253632.0,
+ "4425": 28157253632.0,
+ "4430": 28157253632.0,
+ "4435": 28157253632.0,
+ "4440": 28157253632.0,
+ "4445": 28157253632.0,
+ "4450": 28157253632.0,
+ "4455": 28157253632.0,
+ "4460": 28157253632.0,
+ "4465": 28157253632.0,
+ "4470": 28157253632.0,
+ "4475": 28157253632.0,
+ "4480": 28157253632.0,
+ "4485": 28157253632.0,
+ "4490": 28157253632.0,
+ "4495": 28157253632.0,
+ "4500": 28157253632.0,
+ "4505": 28157253632.0,
+ "4510": 28157253632.0,
+ "4515": 28157253632.0,
+ "4520": 28157253632.0,
+ "4525": 28157253632.0,
+ "4530": 28157253632.0,
+ "4535": 28157253632.0,
+ "4540": 28157253632.0,
+ "4545": 28157253632.0,
+ "4550": 28157253632.0,
+ "4555": 28157253632.0,
+ "4560": 28157253632.0,
+ "4565": 28157253632.0,
+ "4570": 28157253632.0,
+ "4575": 28157253632.0,
+ "4580": 28157253632.0,
+ "4585": 28157253632.0,
+ "4590": 28157253632.0,
+ "4595": 28157253632.0,
+ "4600": 28157253632.0,
+ "4605": 28157253632.0,
+ "4610": 28157253632.0,
+ "4615": 28157253632.0,
+ "4620": 28157253632.0,
+ "4625": 28157253632.0,
+ "4630": 28157253632.0,
+ "4635": 28157253632.0,
+ "4640": 28157253632.0,
+ "4645": 28157253632.0,
+ "4650": 28157253632.0,
+ "4655": 28157253632.0,
+ "4660": 28157253632.0,
+ "4665": 28157253632.0,
+ "4670": 28157253632.0,
+ "4675": 28157253632.0,
+ "4680": 28157253632.0,
+ "4685": 28157253632.0,
+ "4690": 28157253632.0,
+ "4695": 28157253632.0,
+ "4700": 28157253632.0,
+ "4705": 28157253632.0,
+ "4710": 28157253632.0,
+ "4715": 28157253632.0,
+ "4720": 28157253632.0,
+ "4725": 28157253632.0,
+ "4730": 28157253632.0,
+ "4735": 28157253632.0,
+ "4740": 28157253632.0,
+ "4745": 28157253632.0,
+ "4750": 28157253632.0,
+ "4755": 28157253632.0,
+ "4760": 28157253632.0,
+ "4765": 28157253632.0,
+ "4770": 28157253632.0,
+ "4775": 28157253632.0,
+ "4780": 28157253632.0,
+ "4785": 28157253632.0,
+ "4790": 28157253632.0,
+ "4795": 28157253632.0,
+ "4800": 28157253632.0,
+ "4805": 28157253632.0,
+ "4810": 28157253632.0,
+ "4815": 28157253632.0,
+ "4820": 28157253632.0,
+ "4825": 28157253632.0,
+ "4830": 28157253632.0,
+ "4835": 28157253632.0,
+ "4840": 28157253632.0,
+ "4845": 28157253632.0,
+ "4850": 28157253632.0,
+ "4855": 28157253632.0,
+ "4860": 28157253632.0,
+ "4865": 28157253632.0,
+ "4870": 28157253632.0,
+ "4875": 28157253632.0,
+ "4880": 28157253632.0,
+ "4885": 28157253632.0,
+ "4890": 28157253632.0,
+ "4895": 28157253632.0,
+ "4900": 28157253632.0,
+ "4905": 28157253632.0,
+ "4910": 28157253632.0,
+ "4915": 28157253632.0,
+ "4920": 28157253632.0,
+ "4925": 28157253632.0,
+ "4930": 28157253632.0,
+ "4935": 28157253632.0,
+ "4940": 28157253632.0,
+ "4945": 28157253632.0,
+ "4950": 28157253632.0,
+ "4955": 28157253632.0,
+ "4960": 28157253632.0,
+ "4965": 28157253632.0,
+ "4970": 28157253632.0,
+ "4975": 28157253632.0,
+ "4980": 28157253632.0,
+ "4985": 28157253632.0,
+ "4990": 28157253632.0,
+ "4995": 28157253632.0,
+ "5000": 28157253632.0,
+ "5005": 28157253632.0,
+ "5010": 28157253632.0,
+ "5015": 28157253632.0,
+ "5020": 28157253632.0,
+ "5025": 28157253632.0,
+ "5030": 28157253632.0,
+ "5035": 28157253632.0,
+ "5040": 28157253632.0,
+ "5045": 28157253632.0,
+ "5050": 28157253632.0,
+ "5055": 28157253632.0,
+ "5060": 28157253632.0,
+ "5065": 28157253632.0,
+ "5070": 28157253632.0,
+ "5075": 28157253632.0,
+ "5080": 28157253632.0,
+ "5085": 28157253632.0,
+ "5090": 28157253632.0,
+ "5095": 28157253632.0,
+ "5100": 28157253632.0,
+ "5105": 28157253632.0,
+ "5110": 28157253632.0,
+ "5115": 28157253632.0,
+ "5120": 28157253632.0,
+ "5125": 28157253632.0,
+ "5130": 28157253632.0,
+ "5135": 28157253632.0,
+ "5140": 28157253632.0,
+ "5145": 28157253632.0,
+ "5150": 28157253632.0,
+ "5155": 28157253632.0,
+ "5160": 28157253632.0,
+ "5165": 28157253632.0,
+ "5170": 28157253632.0,
+ "5175": 28157253632.0,
+ "5180": 28157253632.0,
+ "5185": 28157253632.0,
+ "5190": 28157253632.0,
+ "5195": 28157253632.0,
+ "5200": 28157253632.0,
+ "5205": 28157253632.0,
+ "5210": 28157253632.0,
+ "5215": 28157253632.0,
+ "5220": 28157253632.0,
+ "5225": 28157253632.0,
+ "5230": 28157253632.0,
+ "5235": 28157253632.0,
+ "5240": 28157253632.0,
+ "5245": 28157253632.0,
+ "5250": 28157253632.0,
+ "5255": 28157253632.0,
+ "5260": 28157253632.0,
+ "5265": 28157253632.0,
+ "5270": 28157253632.0,
+ "5275": 28157253632.0,
+ "5280": 28157253632.0,
+ "5285": 28157253632.0,
+ "5290": 28157253632.0,
+ "5295": 28157253632.0,
+ "5300": 28157253632.0,
+ "5305": 28157253632.0,
+ "5310": 28157253632.0,
+ "5315": 28157253632.0,
+ "5320": 28157253632.0,
+ "5325": 28157253632.0,
+ "5330": 28157253632.0,
+ "5335": 28157253632.0,
+ "5340": 28157253632.0,
+ "5345": 28157253632.0,
+ "5350": 28157253632.0,
+ "5355": 28157253632.0,
+ "5360": 28157253632.0,
+ "5365": 28157253632.0,
+ "5370": 28157253632.0,
+ "5375": 28157253632.0,
+ "5380": 28157253632.0,
+ "5385": 28157253632.0,
+ "5390": 28157253632.0,
+ "5395": 28157253632.0,
+ "5400": 28157253632.0,
+ "5405": 28157253632.0,
+ "5410": 28157253632.0,
+ "5415": 28157253632.0,
+ "5420": 28157253632.0,
+ "5425": 28157253632.0,
+ "5430": 28157253632.0,
+ "5435": 28157253632.0,
+ "5440": 28157253632.0,
+ "5445": 28157253632.0,
+ "5450": 28157253632.0,
+ "5455": 28157253632.0,
+ "5460": 28157253632.0,
+ "5465": 28157253632.0,
+ "5470": 28157253632.0,
+ "5475": 28157253632.0,
+ "5480": 28157253632.0,
+ "5485": 28157253632.0,
+ "5490": 28157253632.0,
+ "5495": 28157253632.0,
+ "5500": 28157253632.0,
+ "5505": 28157253632.0,
+ "5510": 28157253632.0,
+ "5515": 28157253632.0,
+ "5520": 28157253632.0,
+ "5525": 28157253632.0,
+ "5530": 28157253632.0,
+ "5535": 28157253632.0,
+ "5540": 28157253632.0,
+ "5545": 28157253632.0,
+ "5550": 28157253632.0,
+ "5555": 28157253632.0,
+ "5560": 28157253632.0,
+ "5565": 28157253632.0,
+ "5570": 28157253632.0,
+ "5575": 28157253632.0,
+ "5580": 28157253632.0,
+ "5585": 28157253632.0,
+ "5590": 28157253632.0,
+ "5595": 28157253632.0,
+ "5600": 28157253632.0,
+ "5605": 28157253632.0,
+ "5610": 28157253632.0,
+ "5615": 28157253632.0,
+ "5620": 28157253632.0,
+ "5625": 28157253632.0,
+ "5630": 28157253632.0,
+ "5635": 28157253632.0,
+ "5640": 28157253632.0,
+ "5645": 28157253632.0,
+ "5650": 28157253632.0,
+ "5655": 28157253632.0,
+ "5660": 28157253632.0,
+ "5665": 28157253632.0,
+ "5670": 28157253632.0,
+ "5675": 28157253632.0,
+ "5680": 28157253632.0,
+ "5685": 28157253632.0,
+ "5690": 28157253632.0,
+ "5695": 28157253632.0,
+ "5700": 28157253632.0,
+ "5705": 28157253632.0,
+ "5710": 28157253632.0,
+ "5715": 28157253632.0,
+ "5720": 28157253632.0,
+ "5725": 28157253632.0,
+ "5730": 28157253632.0,
+ "5735": 28157253632.0,
+ "5740": 28157253632.0,
+ "5745": 28157253632.0,
+ "5750": 28157253632.0,
+ "5755": 28157253632.0,
+ "5760": 28157253632.0,
+ "5765": 28157253632.0,
+ "5770": 28157253632.0,
+ "5775": 28157253632.0,
+ "5780": 28157253632.0,
+ "5785": 28157253632.0,
+ "5790": 28157253632.0,
+ "5795": 28157253632.0,
+ "5800": 28157253632.0,
+ "5805": 28157253632.0,
+ "5810": 28157253632.0,
+ "5815": 28157253632.0,
+ "5820": 28157253632.0,
+ "5825": 28157253632.0,
+ "5830": 28157253632.0,
+ "5835": 28157253632.0,
+ "5840": 28157253632.0,
+ "5845": 28157253632.0,
+ "5850": 28157253632.0,
+ "5855": 28157253632.0,
+ "5860": 28157253632.0,
+ "5865": 28157253632.0,
+ "5870": 28157253632.0,
+ "5875": 28157253632.0,
+ "5880": 28157253632.0,
+ "5885": 28157253632.0,
+ "5890": 28157253632.0,
+ "5895": 28157253632.0,
+ "5900": 28157253632.0,
+ "5905": 28157253632.0,
+ "5910": 28157253632.0,
+ "5915": 28157253632.0,
+ "5920": 28157253632.0,
+ "5925": 28157253632.0,
+ "5930": 28157253632.0,
+ "5935": 28157253632.0,
+ "5940": 28157253632.0,
+ "5945": 28157253632.0,
+ "5950": 28157253632.0,
+ "5955": 28157253632.0,
+ "5960": 28157253632.0,
+ "5965": 28157253632.0,
+ "5970": 28157253632.0,
+ "5975": 28157253632.0,
+ "5980": 28157253632.0,
+ "5985": 28157253632.0,
+ "5990": 28157253632.0,
+ "5995": 28157253632.0,
+ "6000": 28157253632.0,
+ "6005": 28157253632.0,
+ "6010": 28157253632.0,
+ "6015": 28157253632.0,
+ "6020": 28157253632.0,
+ "6025": 28157253632.0,
+ "6030": 28157253632.0,
+ "6035": 28157253632.0,
+ "6040": 28157253632.0,
+ "6045": 28157253632.0,
+ "6050": 28157253632.0,
+ "6055": 28157253632.0,
+ "6060": 28157253632.0,
+ "6065": 28157253632.0,
+ "6070": 28157253632.0,
+ "6075": 28157253632.0,
+ "6080": 28157253632.0,
+ "6085": 28157253632.0,
+ "6090": 28157253632.0,
+ "6095": 28157253632.0,
+ "6100": 28157253632.0,
+ "6105": 28157253632.0,
+ "6110": 28157253632.0,
+ "6115": 28157253632.0,
+ "6120": 28157253632.0,
+ "6125": 28157253632.0,
+ "6130": 28157253632.0,
+ "6135": 28157253632.0,
+ "6140": 28157253632.0,
+ "6145": 28157253632.0,
+ "6150": 28157253632.0,
+ "6155": 28157253632.0,
+ "6160": 28157253632.0,
+ "6165": 28157253632.0,
+ "6170": 28157253632.0,
+ "6175": 28157253632.0,
+ "6180": 28157253632.0,
+ "6185": 28157253632.0,
+ "6190": 28157253632.0,
+ "6195": 28157253632.0,
+ "6200": 28157253632.0,
+ "6205": 28157253632.0,
+ "6210": 28157253632.0,
+ "6215": 28157253632.0,
+ "6220": 28157253632.0,
+ "6225": 28157253632.0,
+ "6230": 28157253632.0,
+ "6235": 28157253632.0,
+ "6240": 28157253632.0,
+ "6245": 28157253632.0,
+ "6250": 28157253632.0,
+ "6255": 28157253632.0,
+ "6260": 28157253632.0,
+ "6265": 28157253632.0,
+ "6270": 28157253632.0,
+ "6275": 28157253632.0,
+ "6280": 28157253632.0,
+ "6285": 28157253632.0,
+ "6290": 28157253632.0,
+ "6295": 28157253632.0,
+ "6300": 28157253632.0,
+ "6305": 28157253632.0,
+ "6310": 28157253632.0,
+ "6315": 28157253632.0,
+ "6320": 28157253632.0,
+ "6325": 28157253632.0,
+ "6330": 28157253632.0,
+ "6335": 28157253632.0,
+ "6340": 28157253632.0,
+ "6345": 28157253632.0,
+ "6350": 28157253632.0,
+ "6355": 28157253632.0,
+ "6360": 28157253632.0,
+ "6365": 28157253632.0,
+ "6370": 28157253632.0,
+ "6375": 28157253632.0,
+ "6380": 28157253632.0,
+ "6385": 28157253632.0,
+ "6390": 28157253632.0,
+ "6395": 28157253632.0,
+ "6400": 28157253632.0,
+ "6405": 28157253632.0,
+ "6410": 28157253632.0,
+ "6415": 28157253632.0,
+ "6420": 28157253632.0,
+ "6425": 28157253632.0,
+ "6430": 28157253632.0,
+ "6435": 28157253632.0,
+ "6440": 28157253632.0,
+ "6445": 28157253632.0,
+ "6450": 28157253632.0,
+ "6455": 28157253632.0,
+ "6460": 28157253632.0,
+ "6465": 28157253632.0,
+ "6470": 28157253632.0,
+ "6475": 28157253632.0,
+ "6480": 28157253632.0,
+ "6485": 28157253632.0,
+ "6490": 28157253632.0,
+ "6495": 28157253632.0,
+ "6500": 28157253632.0,
+ "6505": 28157253632.0,
+ "6510": 28157253632.0,
+ "6515": 28157253632.0,
+ "6520": 28157253632.0,
+ "6525": 28157253632.0,
+ "6530": 28157253632.0,
+ "6535": 28157253632.0,
+ "6540": 28157253632.0,
+ "6545": 28157253632.0,
+ "6550": 28157253632.0,
+ "6555": 28157253632.0,
+ "6560": 28157253632.0,
+ "6565": 28157253632.0,
+ "6570": 28157253632.0,
+ "6575": 28157253632.0,
+ "6580": 28157253632.0,
+ "6585": 28157253632.0,
+ "6590": 28157253632.0,
+ "6595": 28157253632.0,
+ "6600": 28157253632.0,
+ "6605": 28157253632.0,
+ "6610": 28157253632.0,
+ "6615": 28157253632.0,
+ "6620": 28157253632.0,
+ "6625": 28157253632.0,
+ "6630": 28157253632.0,
+ "6635": 28157253632.0,
+ "6640": 28157253632.0,
+ "6645": 28157253632.0,
+ "6650": 28157253632.0,
+ "6655": 28157253632.0,
+ "6660": 28157253632.0,
+ "6665": 28157253632.0,
+ "6670": 28157253632.0,
+ "6675": 28157253632.0,
+ "6680": 28157253632.0,
+ "6685": 28157253632.0,
+ "6690": 28157253632.0,
+ "6695": 28157253632.0,
+ "6700": 28157253632.0,
+ "6705": 28157253632.0,
+ "6710": 28157253632.0,
+ "6715": 28157253632.0,
+ "6720": 28157253632.0,
+ "6725": 28157253632.0,
+ "6730": 28157253632.0,
+ "6735": 28157253632.0,
+ "6740": 28157253632.0,
+ "6745": 28157253632.0,
+ "6750": 28157253632.0,
+ "6755": 28157253632.0,
+ "6760": 28157253632.0,
+ "6765": 28157253632.0,
+ "6770": 28157253632.0,
+ "6775": 28157253632.0,
+ "6780": 28157253632.0,
+ "6785": 28157253632.0,
+ "6790": 28157253632.0,
+ "6795": 28157253632.0,
+ "6800": 28157253632.0,
+ "6805": 28157253632.0,
+ "6810": 28157253632.0,
+ "6815": 28157253632.0,
+ "6820": 28157253632.0,
+ "6825": 28157253632.0,
+ "6830": 28157253632.0,
+ "6835": 28157253632.0,
+ "6840": 28157253632.0,
+ "6845": 28157253632.0,
+ "6850": 28157253632.0,
+ "6855": 28157253632.0,
+ "6860": 28157253632.0,
+ "6865": 28157253632.0,
+ "6870": 28157253632.0,
+ "6875": 28157253632.0,
+ "6880": 28157253632.0,
+ "6885": 28157253632.0,
+ "6890": 28157253632.0,
+ "6895": 28157253632.0,
+ "6900": 28157253632.0,
+ "6905": 28157253632.0,
+ "6910": 28157253632.0,
+ "6915": 28157253632.0,
+ "6920": 28157253632.0,
+ "6925": 28157253632.0,
+ "6930": 28157253632.0,
+ "6935": 28157253632.0,
+ "6940": 28157253632.0,
+ "6945": 28157253632.0,
+ "6950": 28157253632.0,
+ "6955": 28157253632.0,
+ "6960": 28157253632.0,
+ "6965": 28157253632.0,
+ "6970": 28157253632.0,
+ "6975": 28157253632.0,
+ "6980": 28157253632.0,
+ "6985": 28157253632.0,
+ "6990": 28157253632.0,
+ "6995": 28157253632.0,
+ "7000": 28157253632.0,
+ "7005": 28157253632.0,
+ "7010": 28157253632.0,
+ "7015": 28157253632.0,
+ "7020": 28157253632.0,
+ "7025": 28157253632.0,
+ "7030": 28157253632.0,
+ "7035": 28157253632.0,
+ "7040": 28157253632.0,
+ "7045": 28157253632.0,
+ "7050": 28157253632.0,
+ "7055": 28157253632.0,
+ "7060": 28157253632.0,
+ "7065": 28157253632.0,
+ "7070": 28157253632.0,
+ "7075": 28157253632.0,
+ "7080": 28157253632.0,
+ "7085": 28157253632.0,
+ "7090": 28157253632.0,
+ "7095": 28157253632.0,
+ "7100": 28157253632.0,
+ "7105": 28157253632.0,
+ "7110": 28157253632.0,
+ "7115": 28157253632.0,
+ "7120": 28157253632.0,
+ "7125": 28157253632.0,
+ "7130": 28157253632.0,
+ "7135": 28157253632.0,
+ "7140": 28157253632.0,
+ "7145": 28157253632.0,
+ "7150": 28157253632.0,
+ "7155": 28157253632.0,
+ "7160": 28157253632.0,
+ "7165": 28157253632.0,
+ "7170": 28157253632.0,
+ "7175": 28157253632.0,
+ "7180": 28157253632.0,
+ "7185": 28157253632.0,
+ "7190": 28157253632.0,
+ "7195": 28157253632.0,
+ "7200": 28157253632.0,
+ "7205": 28157253632.0,
+ "7210": 28157253632.0,
+ "7215": 28157253632.0,
+ "7220": 28157253632.0,
+ "7225": 28157253632.0,
+ "7230": 28157253632.0,
+ "7235": 28157253632.0,
+ "7240": 28157253632.0,
+ "7245": 28157253632.0,
+ "7250": 28157253632.0,
+ "7255": 28157253632.0,
+ "7260": 28157253632.0,
+ "7265": 28157253632.0,
+ "7270": 28157253632.0,
+ "7275": 28157253632.0,
+ "7280": 28157253632.0,
+ "7285": 28157253632.0,
+ "7290": 28157253632.0,
+ "7295": 28157253632.0,
+ "7300": 28157253632.0,
+ "7305": 28157253632.0,
+ "7310": 28157253632.0,
+ "7315": 28157253632.0,
+ "7320": 28157253632.0,
+ "7325": 28157253632.0,
+ "7330": 28157253632.0,
+ "7335": 28157253632.0,
+ "7340": 28157253632.0,
+ "7345": 28157253632.0,
+ "7350": 28157253632.0,
+ "7355": 28157253632.0,
+ "7360": 28157253632.0,
+ "7365": 28157253632.0,
+ "7370": 28157253632.0,
+ "7375": 28157253632.0,
+ "7380": 28157253632.0,
+ "7385": 28157253632.0,
+ "7390": 28157253632.0,
+ "7395": 28157253632.0,
+ "7400": 28157253632.0,
+ "7405": 28157253632.0,
+ "7410": 28157253632.0,
+ "7415": 28157253632.0,
+ "7420": 28157253632.0,
+ "7425": 28157253632.0,
+ "7430": 28157253632.0,
+ "7435": 28157253632.0,
+ "7440": 28157253632.0,
+ "7445": 28157253632.0,
+ "7450": 28157253632.0,
+ "7455": 28157253632.0,
+ "7460": 28157253632.0,
+ "7465": 28157253632.0,
+ "7470": 28157253632.0,
+ "7475": 28157253632.0,
+ "7480": 28157253632.0,
+ "7485": 28157253632.0,
+ "7490": 28157253632.0,
+ "7495": 28157253632.0,
+ "7500": 28157253632.0,
+ "7505": 28157253632.0,
+ "7510": 28157253632.0,
+ "7515": 28157253632.0,
+ "7520": 28157253632.0,
+ "7525": 28157253632.0,
+ "7530": 28157253632.0,
+ "7535": 28157253632.0,
+ "7540": 28157253632.0,
+ "7545": 28157253632.0,
+ "7550": 28157253632.0,
+ "7555": 28157253632.0,
+ "7560": 28157253632.0,
+ "7565": 28157253632.0,
+ "7570": 28157253632.0,
+ "7575": 28157253632.0,
+ "7580": 28157253632.0,
+ "7585": 28157253632.0,
+ "7590": 28157253632.0,
+ "7595": 28157253632.0,
+ "7600": 28157253632.0,
+ "7605": 28157253632.0,
+ "7610": 28157253632.0,
+ "7615": 28157253632.0,
+ "7620": 28157253632.0,
+ "7625": 28157253632.0,
+ "7630": 28157253632.0,
+ "7635": 28157253632.0,
+ "7640": 28157253632.0,
+ "7645": 28157253632.0,
+ "7650": 28157253632.0,
+ "7655": 28157253632.0,
+ "7660": 28157253632.0,
+ "7665": 28157253632.0,
+ "7670": 28157253632.0,
+ "7675": 28157253632.0,
+ "7680": 28157253632.0,
+ "7685": 28157253632.0,
+ "7690": 28157253632.0,
+ "7695": 28157253632.0,
+ "7700": 28157253632.0,
+ "7705": 28157253632.0,
+ "7710": 28157253632.0,
+ "7715": 28157253632.0,
+ "7720": 28157253632.0,
+ "7725": 28157253632.0,
+ "7730": 28157253632.0,
+ "7735": 28157253632.0,
+ "7740": 28157253632.0,
+ "7745": 28157253632.0,
+ "7750": 28157253632.0,
+ "7755": 28157253632.0,
+ "7760": 28157253632.0,
+ "7765": 28157253632.0,
+ "7770": 28157253632.0,
+ "7775": 28157253632.0,
+ "7780": 28157253632.0,
+ "7785": 28157253632.0,
+ "7790": 28157253632.0,
+ "7795": 28157253632.0,
+ "7800": 28157253632.0,
+ "7805": 28157253632.0,
+ "7810": 28157253632.0,
+ "7815": 28157253632.0,
+ "7820": 28157253632.0,
+ "7825": 28157253632.0,
+ "7830": 28157253632.0,
+ "7835": 28157253632.0,
+ "7840": 28157253632.0,
+ "7845": 28157253632.0,
+ "7850": 28157253632.0,
+ "7855": 28157253632.0,
+ "7860": 28157253632.0,
+ "7865": 28157253632.0,
+ "7870": 28157253632.0,
+ "7875": 28157253632.0,
+ "7880": 28157253632.0,
+ "7885": 28157253632.0,
+ "7890": 28157253632.0,
+ "7895": 28157253632.0,
+ "7900": 28157253632.0,
+ "7905": 28157253632.0,
+ "7910": 28157253632.0,
+ "7915": 28157253632.0,
+ "7920": 28157253632.0,
+ "7925": 28157253632.0,
+ "7930": 28157253632.0,
+ "7935": 28157253632.0,
+ "7940": 28157253632.0,
+ "7945": 28157253632.0,
+ "7950": 28157253632.0,
+ "7955": 28157253632.0,
+ "7960": 28157253632.0,
+ "7965": 28157253632.0,
+ "7970": 28157253632.0,
+ "7975": 28157253632.0,
+ "7980": 28157253632.0,
+ "7985": 28157253632.0,
+ "7990": 28157253632.0,
+ "7995": 28157253632.0,
+ "8000": 28157253632.0,
+ "8005": 28157253632.0,
+ "8010": 28157253632.0,
+ "8015": 28157253632.0,
+ "8020": 28157253632.0,
+ "8025": 28157253632.0,
+ "8030": 28157253632.0,
+ "8035": 28157253632.0,
+ "8040": 28157253632.0,
+ "8045": 28157253632.0,
+ "8050": 28157253632.0,
+ "8055": 28157253632.0,
+ "8060": 28157253632.0,
+ "8065": 28157253632.0,
+ "8070": 28157253632.0,
+ "8075": 28157253632.0,
+ "8080": 28157253632.0,
+ "8085": 28157253632.0,
+ "8090": 28157253632.0,
+ "8095": 28157253632.0,
+ "8100": 28157253632.0,
+ "8105": 28157253632.0,
+ "8110": 28157253632.0,
+ "8115": 28157253632.0,
+ "8120": 28157253632.0,
+ "8125": 28157253632.0,
+ "8130": 28157253632.0,
+ "8135": 28157253632.0,
+ "8140": 28157253632.0,
+ "8145": 28157253632.0,
+ "8150": 28157253632.0,
+ "8155": 28157253632.0,
+ "8160": 28157253632.0,
+ "8165": 28157253632.0,
+ "8170": 28157253632.0,
+ "8175": 28157253632.0,
+ "8180": 28157253632.0,
+ "8185": 28157253632.0,
+ "8190": 28157253632.0,
+ "8195": 28157253632.0,
+ "8200": 28157253632.0,
+ "8205": 28157253632.0,
+ "8210": 28157253632.0,
+ "8215": 28157253632.0,
+ "8220": 28157253632.0,
+ "8225": 28157253632.0,
+ "8230": 28157253632.0,
+ "8235": 28157253632.0,
+ "8240": 28157253632.0,
+ "8245": 28157253632.0,
+ "8250": 28157253632.0,
+ "8255": 28157253632.0,
+ "8260": 28159016960.0,
+ "8265": 28159016960.0,
+ "8270": 28159016960.0,
+ "8275": 28159016960.0,
+ "8280": 28159016960.0,
+ "8285": 28159016960.0,
+ "8290": 28159016960.0,
+ "8295": 28159016960.0,
+ "8300": 28159016960.0,
+ "8305": 28159016960.0,
+ "8310": 28159016960.0,
+ "8315": 28159016960.0,
+ "8320": 28159016960.0,
+ "8325": 28159016960.0,
+ "8330": 28159016960.0,
+ "8335": 28159016960.0,
+ "8340": 28159016960.0,
+ "8345": 28159016960.0,
+ "8350": 28159016960.0,
+ "8355": 28159016960.0,
+ "8360": 28159016960.0,
+ "8365": 28159016960.0,
+ "8370": 28159016960.0,
+ "8375": 28159016960.0,
+ "8380": 28159016960.0,
+ "8385": 28159016960.0,
+ "8390": 28159016960.0,
+ "8395": 28159016960.0,
+ "8400": 28159016960.0,
+ "8405": 28159016960.0,
+ "8410": 28159016960.0,
+ "8415": 28159016960.0,
+ "8420": 28159016960.0,
+ "8425": 28159016960.0,
+ "8430": 28159016960.0,
+ "8435": 28159016960.0,
+ "8440": 28159016960.0,
+ "8445": 28159016960.0,
+ "8450": 28159016960.0,
+ "8455": 28159016960.0,
+ "8460": 28159016960.0,
+ "8465": 28159016960.0,
+ "8470": 28159016960.0,
+ "8475": 28159016960.0,
+ "8480": 28159016960.0,
+ "8485": 28159016960.0,
+ "8490": 28159016960.0,
+ "8495": 28159016960.0,
+ "8500": 28159016960.0,
+ "8505": 28159016960.0,
+ "8510": 28159016960.0,
+ "8515": 28159016960.0,
+ "8520": 28159016960.0,
+ "8525": 28159016960.0,
+ "8530": 28159016960.0,
+ "8535": 28159016960.0,
+ "8540": 28159016960.0,
+ "8545": 28159016960.0,
+ "8550": 28159016960.0,
+ "8555": 28159016960.0,
+ "8560": 28159016960.0,
+ "8565": 28159016960.0,
+ "8570": 28159016960.0,
+ "8575": 28159016960.0,
+ "8580": 28159016960.0,
+ "8585": 28159016960.0,
+ "8590": 28159016960.0,
+ "8595": 28159016960.0,
+ "8600": 28159016960.0,
+ "8605": 28159016960.0,
+ "8610": 28159016960.0,
+ "8615": 28159016960.0,
+ "8620": 28159016960.0,
+ "8625": 28159016960.0,
+ "8630": 28159016960.0,
+ "8635": 28159016960.0,
+ "8640": 28159016960.0,
+ "8645": 28159016960.0,
+ "8650": 28159016960.0,
+ "8655": 28159016960.0,
+ "8660": 28159016960.0,
+ "8665": 28159016960.0,
+ "8670": 28159016960.0,
+ "8675": 28159016960.0,
+ "8680": 28159016960.0,
+ "8685": 28159016960.0,
+ "8690": 28159016960.0,
+ "8695": 28159016960.0,
+ "8700": 28159016960.0,
+ "8705": 28159016960.0,
+ "8710": 28159016960.0,
+ "8715": 28159016960.0,
+ "8720": 28159016960.0,
+ "8725": 28159016960.0,
+ "8730": 28159016960.0,
+ "8735": 28159016960.0,
+ "8740": 28159016960.0,
+ "8745": 28159016960.0,
+ "8750": 28159016960.0,
+ "8755": 28159016960.0,
+ "8760": 28159016960.0,
+ "8765": 28159016960.0,
+ "8770": 28159016960.0,
+ "8775": 28159016960.0,
+ "8780": 28159016960.0,
+ "8785": 28159016960.0,
+ "8790": 28159016960.0,
+ "8795": 28159016960.0,
+ "8800": 28159016960.0,
+ "8805": 28159016960.0,
+ "8810": 28159016960.0,
+ "8815": 28159016960.0,
+ "8820": 28159016960.0,
+ "8825": 28159016960.0,
+ "8830": 28159016960.0,
+ "8835": 28159016960.0,
+ "8840": 28159016960.0,
+ "8845": 28159016960.0,
+ "8850": 28159016960.0,
+ "8855": 28159016960.0,
+ "8860": 28159016960.0,
+ "8865": 28159016960.0,
+ "8870": 28159016960.0,
+ "8875": 28159016960.0,
+ "8880": 28159016960.0,
+ "8885": 28159016960.0,
+ "8890": 28159016960.0,
+ "8895": 28159016960.0,
+ "8900": 28159016960.0,
+ "8905": 28159016960.0,
+ "8910": 28159016960.0,
+ "8915": 28159016960.0,
+ "8920": 28159016960.0,
+ "8925": 28159016960.0,
+ "8930": 28159016960.0,
+ "8935": 28159016960.0,
+ "8940": 28159016960.0,
+ "8945": 28159016960.0,
+ "8950": 28159016960.0,
+ "8955": 28159016960.0,
+ "8960": 28159016960.0,
+ "8965": 28159016960.0,
+ "8970": 28159016960.0,
+ "8975": 28159016960.0,
+ "8980": 28159016960.0,
+ "8985": 28159016960.0,
+ "8990": 28159016960.0,
+ "8995": 28159016960.0,
+ "9000": 28159016960.0,
+ "9005": 28160194560.0,
+ "9010": 28160194560.0,
+ "9015": 28160194560.0,
+ "9020": 28160194560.0,
+ "9025": 28160194560.0,
+ "9030": 28160194560.0,
+ "9035": 28160194560.0,
+ "9040": 28160194560.0,
+ "9045": 28160194560.0,
+ "9050": 28160194560.0,
+ "9055": 28160194560.0,
+ "9060": 28160194560.0,
+ "9065": 28160194560.0,
+ "9070": 28160194560.0,
+ "9075": 28160194560.0,
+ "9080": 28160194560.0,
+ "9085": 28160194560.0,
+ "9090": 28160194560.0,
+ "9095": 28160194560.0,
+ "9100": 28160194560.0,
+ "9105": 28160194560.0,
+ "9110": 28160194560.0,
+ "9115": 28160194560.0,
+ "9120": 28160194560.0,
+ "9125": 28160194560.0,
+ "9130": 28160194560.0,
+ "9135": 28160194560.0,
+ "9140": 28160194560.0,
+ "9145": 28160194560.0,
+ "9150": 28160194560.0,
+ "9155": 28160194560.0,
+ "9160": 28160194560.0,
+ "9165": 28160194560.0,
+ "9170": 28160194560.0,
+ "9175": 28160194560.0,
+ "9180": 28160194560.0,
+ "9185": 28160194560.0,
+ "9190": 28160194560.0,
+ "9195": 28160194560.0,
+ "9200": 28160194560.0,
+ "9205": 28160194560.0,
+ "9210": 28160194560.0,
+ "9215": 28160194560.0,
+ "9220": 28160194560.0,
+ "9225": 28160194560.0,
+ "9230": 28160194560.0,
+ "9235": 28160194560.0,
+ "9240": 28160194560.0,
+ "9245": 28160194560.0,
+ "9250": 28160194560.0,
+ "9255": 28160194560.0,
+ "9260": 28160194560.0,
+ "9265": 28160194560.0,
+ "9270": 28160194560.0,
+ "9275": 28160194560.0,
+ "9280": 28160194560.0,
+ "9285": 28160194560.0,
+ "9290": 28160194560.0,
+ "9295": 28160194560.0,
+ "9300": 28160194560.0,
+ "9305": 28160194560.0,
+ "9310": 28160194560.0,
+ "9315": 28160194560.0,
+ "9320": 28160194560.0,
+ "9325": 28160194560.0,
+ "9330": 28160194560.0,
+ "9335": 28160194560.0,
+ "9340": 28160194560.0,
+ "9345": 28160194560.0,
+ "9350": 28160194560.0,
+ "9355": 28160194560.0,
+ "9360": 28160194560.0,
+ "9365": 28160194560.0,
+ "9370": 28160194560.0,
+ "9375": 28160194560.0,
+ "9380": 28160194560.0,
+ "9385": 28160194560.0,
+ "9390": 28160194560.0,
+ "9395": 28160194560.0,
+ "9400": 28160194560.0,
+ "9405": 28160194560.0,
+ "9410": 28160194560.0,
+ "9415": 28160194560.0,
+ "9420": 28160194560.0,
+ "9425": 28160194560.0,
+ "9430": 28160194560.0,
+ "9435": 28160194560.0,
+ "9440": 28160194560.0,
+ "9445": 28160194560.0,
+ "9450": 28160194560.0,
+ "9455": 28160194560.0,
+ "9460": 28160194560.0,
+ "9465": 28160194560.0,
+ "9470": 28160194560.0,
+ "9475": 28160194560.0,
+ "9480": 28160194560.0,
+ "9485": 28160194560.0,
+ "9490": 28160194560.0,
+ "9495": 28160194560.0,
+ "9500": 28160194560.0,
+ "9505": 28160194560.0,
+ "9510": 28160194560.0,
+ "9515": 28160194560.0,
+ "9520": 28160194560.0,
+ "9525": 28160194560.0,
+ "9530": 28160194560.0,
+ "9535": 28160194560.0,
+ "9540": 28160194560.0,
+ "9545": 28160194560.0,
+ "9550": 28160194560.0,
+ "9555": 28160194560.0,
+ "9560": 28160194560.0,
+ "9565": 28160194560.0,
+ "9570": 28160194560.0,
+ "9575": 28160194560.0,
+ "9580": 28160194560.0,
+ "9585": 28160194560.0,
+ "9590": 28160194560.0,
+ "9595": 28160194560.0,
+ "9600": 28160194560.0,
+ "9605": 28160194560.0,
+ "9610": 28160194560.0,
+ "9615": 28160194560.0,
+ "9620": 28160194560.0,
+ "9625": 28160194560.0,
+ "9630": 28160194560.0,
+ "9635": 28160194560.0,
+ "9640": 28160194560.0,
+ "9645": 28160194560.0,
+ "9650": 28160194560.0,
+ "9655": 28160194560.0,
+ "9660": 28160194560.0,
+ "9665": 28160194560.0,
+ "9670": 28160194560.0,
+ "9675": 28160194560.0,
+ "9680": 28160194560.0,
+ "9685": 28160194560.0,
+ "9690": 28160194560.0,
+ "9695": 28160194560.0,
+ "9700": 28160194560.0,
+ "9705": 28160194560.0,
+ "9710": 28160194560.0,
+ "9715": 28160194560.0,
+ "9720": 28160194560.0,
+ "9725": 28160194560.0,
+ "9730": 28160194560.0,
+ "9735": 28160194560.0,
+ "9740": 28160194560.0,
+ "9745": 28160194560.0,
+ "9750": 28160194560.0,
+ "9755": 28160194560.0,
+ "9760": 28160194560.0,
+ "9765": 28160194560.0,
+ "9770": 28160194560.0,
+ "9775": 28160194560.0,
+ "9780": 28160194560.0,
+ "9785": 28160194560.0,
+ "9790": 28160194560.0,
+ "9795": 28160194560.0,
+ "9800": 28160194560.0,
+ "9805": 28160194560.0,
+ "9810": 28160194560.0,
+ "9815": 28160194560.0,
+ "9820": 28160194560.0,
+ "9825": 28160194560.0,
+ "9830": 28160194560.0,
+ "9835": 28160194560.0,
+ "9840": 28160194560.0,
+ "9845": 28160194560.0,
+ "9850": 28160194560.0,
+ "9855": 28160194560.0,
+ "9860": 28160194560.0,
+ "9865": 28160194560.0,
+ "9870": 28160194560.0,
+ "9875": 28160194560.0,
+ "9880": 28160194560.0,
+ "9885": 28160194560.0,
+ "9890": 28160194560.0,
+ "9895": 28160194560.0,
+ "9900": 28160194560.0,
+ "9905": 28160194560.0,
+ "9910": 28160194560.0,
+ "9915": 28160194560.0,
+ "9920": 28160194560.0,
+ "9925": 28160194560.0,
+ "9930": 28160194560.0,
+ "9935": 28160194560.0,
+ "9940": 28160194560.0,
+ "9945": 28160194560.0,
+ "9950": 28160194560.0,
+ "9955": 28160194560.0,
+ "9960": 28160194560.0,
+ "9965": 28160194560.0,
+ "9970": 28160194560.0,
+ "9975": 28160194560.0,
+ "9980": 28160194560.0,
+ "9985": 28160194560.0,
+ "9990": 28160194560.0,
+ "9995": 28160194560.0,
+ "10000": 28160194560.0,
+ "10005": 28160194560.0,
+ "10010": 28160194560.0,
+ "10015": 28160194560.0,
+ "10020": 28160194560.0,
+ "10025": 28160194560.0,
+ "10030": 28160194560.0,
+ "10035": 28160194560.0,
+ "10040": 28160194560.0,
+ "10045": 28160194560.0,
+ "10050": 28160194560.0,
+ "10055": 28160194560.0,
+ "10060": 28160194560.0,
+ "10065": 28160194560.0,
+ "10070": 28160194560.0,
+ "10075": 28160194560.0,
+ "10080": 28160194560.0,
+ "10085": 28160194560.0,
+ "10090": 28160194560.0,
+ "10095": 28160194560.0,
+ "10100": 28160194560.0,
+ "10105": 28160194560.0,
+ "10110": 28160194560.0,
+ "10115": 28160194560.0,
+ "10120": 28160194560.0,
+ "10125": 28160194560.0,
+ "10130": 28160194560.0,
+ "10135": 28160194560.0,
+ "10140": 28160194560.0,
+ "10145": 28160194560.0,
+ "10150": 28160194560.0,
+ "10155": 28160194560.0,
+ "10160": 28160194560.0,
+ "10165": 28160194560.0,
+ "10170": 28160194560.0,
+ "10175": 28160194560.0,
+ "10180": 28160194560.0,
+ "10185": 28160194560.0,
+ "10190": 28160194560.0,
+ "10195": 28160194560.0,
+ "10200": 28160194560.0,
+ "10205": 28160194560.0,
+ "10210": 28160194560.0,
+ "10215": 28160194560.0,
+ "10220": 28160194560.0,
+ "10225": 28160194560.0,
+ "10230": 28160194560.0,
+ "10235": 28160194560.0,
+ "10240": 28160194560.0,
+ "10245": 28160194560.0,
+ "10250": 28160194560.0,
+ "10255": 28160194560.0,
+ "10260": 28160194560.0,
+ "10265": 28160194560.0,
+ "10270": 28160194560.0,
+ "10275": 28160194560.0,
+ "10280": 28160194560.0,
+ "10285": 28160194560.0,
+ "10290": 28160194560.0,
+ "10295": 28160194560.0,
+ "10300": 28160194560.0,
+ "10305": 28160194560.0,
+ "10310": 28160194560.0,
+ "10315": 28160194560.0,
+ "10320": 28160194560.0,
+ "10325": 28160194560.0,
+ "10330": 28160194560.0,
+ "10335": 28160194560.0,
+ "10340": 28160194560.0,
+ "10345": 28160194560.0,
+ "10350": 28160194560.0,
+ "10355": 28160194560.0,
+ "10360": 28160194560.0,
+ "10365": 28160194560.0,
+ "10370": 28160194560.0,
+ "10375": 28160194560.0,
+ "10380": 28160194560.0,
+ "10385": 28160194560.0,
+ "10390": 28160194560.0,
+ "10395": 28160194560.0,
+ "10400": 28160194560.0,
+ "10405": 28160194560.0,
+ "10410": 28160194560.0,
+ "10415": 28160194560.0,
+ "10420": 28160194560.0,
+ "10425": 28160194560.0,
+ "10430": 28160194560.0,
+ "10435": 28160194560.0,
+ "10440": 28160194560.0,
+ "10445": 28160194560.0,
+ "10450": 28160194560.0,
+ "10455": 28160194560.0,
+ "10460": 28160194560.0,
+ "10465": 28160194560.0,
+ "10470": 28160194560.0,
+ "10475": 28160194560.0,
+ "10480": 28160194560.0,
+ "10485": 28160194560.0,
+ "10490": 28160194560.0,
+ "10495": 28160194560.0,
+ "10500": 28160194560.0,
+ "10505": 28160194560.0,
+ "10510": 28160194560.0,
+ "10515": 28160194560.0,
+ "10520": 28160194560.0,
+ "10525": 28160194560.0,
+ "10530": 28160194560.0,
+ "10535": 28160194560.0,
+ "10540": 28160194560.0,
+ "10545": 28160194560.0,
+ "10550": 28160194560.0,
+ "10555": 28160194560.0,
+ "10560": 28160194560.0,
+ "10565": 28160194560.0,
+ "10570": 28160194560.0,
+ "10575": 28160194560.0,
+ "10580": 28160194560.0,
+ "10585": 28160194560.0,
+ "10590": 28160194560.0,
+ "10595": 28160194560.0,
+ "10600": 28160194560.0,
+ "10605": 28160194560.0,
+ "10610": 28160194560.0,
+ "10615": 28160194560.0,
+ "10620": 28160194560.0,
+ "10625": 28160194560.0,
+ "10630": 28160194560.0,
+ "10635": 28160194560.0,
+ "10640": 28160194560.0,
+ "10645": 28160194560.0,
+ "10650": 28160194560.0,
+ "10655": 28160194560.0,
+ "10660": 28160194560.0,
+ "10665": 28160194560.0,
+ "10670": 28160194560.0,
+ "10675": 28160194560.0,
+ "10680": 28160194560.0,
+ "10685": 28160194560.0,
+ "10690": 28160194560.0,
+ "10695": 28160194560.0,
+ "10700": 28160194560.0,
+ "10705": 28160194560.0,
+ "10710": 28160194560.0,
+ "10715": 28160194560.0,
+ "10720": 28160194560.0,
+ "10725": 28160194560.0,
+ "10730": 28160194560.0,
+ "10735": 28160194560.0,
+ "10740": 28160194560.0,
+ "10745": 28160194560.0,
+ "10750": 28160194560.0,
+ "10755": 28160194560.0,
+ "10760": 28160194560.0,
+ "10765": 28160194560.0,
+ "10770": 28160194560.0,
+ "10775": 28160194560.0,
+ "10780": 28160194560.0,
+ "10785": 28160194560.0,
+ "10790": 28160194560.0,
+ "10795": 28160194560.0,
+ "10800": 28160194560.0,
+ "10805": 28160194560.0,
+ "10810": 28160194560.0,
+ "10815": 28160194560.0,
+ "10820": 28160194560.0,
+ "10825": 28160194560.0,
+ "10830": 28160194560.0,
+ "10835": 28160194560.0,
+ "10840": 28160194560.0,
+ "10845": 28160194560.0,
+ "10850": 28160194560.0,
+ "10855": 28160194560.0,
+ "10860": 28160194560.0,
+ "10865": 28160194560.0,
+ "10870": 28160194560.0,
+ "10875": 28160194560.0,
+ "10880": 28160194560.0,
+ "10885": 28160194560.0,
+ "10890": 28160194560.0,
+ "10895": 28160194560.0,
+ "10900": 28160194560.0,
+ "10905": 28160194560.0,
+ "10910": 28160194560.0,
+ "10915": 28160194560.0,
+ "10920": 28160194560.0,
+ "10925": 28160194560.0,
+ "10930": 28160194560.0,
+ "10935": 28160194560.0,
+ "10940": 28160194560.0,
+ "10945": 28160194560.0,
+ "10950": 28160194560.0,
+ "10955": 28160194560.0,
+ "10960": 28160194560.0,
+ "10965": 28160194560.0,
+ "10970": 28160194560.0,
+ "10975": 28160194560.0,
+ "10980": 28160194560.0,
+ "10985": 28160194560.0,
+ "10990": 28160194560.0,
+ "10995": 28160194560.0,
+ "11000": 28160194560.0,
+ "11005": 28160194560.0,
+ "11010": 28160194560.0,
+ "11015": 28160194560.0,
+ "11020": 28160194560.0,
+ "11025": 28160194560.0,
+ "11030": 28160194560.0,
+ "11035": 28160194560.0,
+ "11040": 28160194560.0,
+ "11045": 28160194560.0,
+ "11050": 28160194560.0,
+ "11055": 28160194560.0,
+ "11060": 28160194560.0,
+ "11065": 28160194560.0,
+ "11070": 28160194560.0,
+ "11075": 28160194560.0,
+ "11080": 28160194560.0,
+ "11085": 28160194560.0,
+ "11090": 28160194560.0,
+ "11095": 28160194560.0,
+ "11100": 28160194560.0,
+ "11105": 28160194560.0,
+ "11110": 28160194560.0,
+ "11115": 28160194560.0,
+ "11120": 28160194560.0,
+ "11125": 28160194560.0,
+ "11130": 28160194560.0,
+ "11135": 28160194560.0,
+ "11140": 28160194560.0,
+ "11145": 28160194560.0,
+ "11150": 28160194560.0,
+ "11155": 28160194560.0,
+ "11160": 28160194560.0,
+ "11165": 28160194560.0,
+ "11170": 28160194560.0,
+ "11175": 28160194560.0,
+ "11180": 28160194560.0,
+ "11185": 28160194560.0,
+ "11190": 28160194560.0,
+ "11195": 28160194560.0,
+ "11200": 28160194560.0,
+ "11205": 28160194560.0,
+ "11210": 28160194560.0,
+ "11215": 28160194560.0,
+ "11220": 28160194560.0,
+ "11225": 28160194560.0,
+ "11230": 28160194560.0,
+ "11235": 28160194560.0,
+ "11240": 28160194560.0,
+ "11245": 28160194560.0,
+ "11250": 28160194560.0,
+ "11255": 28160194560.0,
+ "11260": 28160194560.0,
+ "11265": 28160194560.0,
+ "11270": 28160194560.0,
+ "11275": 28160194560.0,
+ "11280": 28160194560.0,
+ "11285": 28160194560.0,
+ "11290": 28160194560.0,
+ "11295": 28160194560.0,
+ "11300": 28160194560.0,
+ "11305": 28160194560.0,
+ "11310": 28160194560.0,
+ "11315": 28160194560.0,
+ "11320": 28160194560.0,
+ "11325": 28160194560.0,
+ "11330": 28160194560.0,
+ "11335": 28160194560.0,
+ "11340": 28160194560.0,
+ "11345": 28160194560.0,
+ "11350": 28160194560.0,
+ "11355": 28160194560.0,
+ "11360": 28160194560.0,
+ "11365": 28160194560.0,
+ "11370": 28160194560.0,
+ "11375": 28160194560.0,
+ "11380": 28160194560.0,
+ "11385": 28160194560.0,
+ "11390": 28160194560.0,
+ "11395": 28160194560.0,
+ "11400": 28160194560.0,
+ "11405": 28160194560.0,
+ "11410": 28160194560.0,
+ "11415": 28160194560.0,
+ "11420": 28160194560.0,
+ "11425": 28160194560.0,
+ "11430": 28160194560.0,
+ "11435": 28160194560.0,
+ "11440": 28160194560.0,
+ "11445": 28160194560.0,
+ "11450": 28160194560.0,
+ "11455": 28160194560.0,
+ "11460": 28160194560.0,
+ "11465": 28160194560.0,
+ "11470": 28160194560.0,
+ "11475": 28160194560.0,
+ "11480": 28160194560.0,
+ "11485": 28160194560.0,
+ "11490": 28160194560.0,
+ "11495": 28160194560.0,
+ "11500": 28160194560.0,
+ "11505": 28160194560.0,
+ "11510": 28160194560.0,
+ "11515": 28160194560.0,
+ "11520": 28160194560.0,
+ "11525": 28160194560.0,
+ "11530": 28160194560.0,
+ "11535": 28160194560.0,
+ "11540": 28160194560.0,
+ "11545": 28160194560.0,
+ "11550": 28160194560.0,
+ "11555": 28160194560.0,
+ "11560": 28160194560.0,
+ "11565": 28160194560.0,
+ "11570": 28160194560.0,
+ "11575": 28160194560.0,
+ "11580": 28160194560.0,
+ "11585": 28160194560.0,
+ "11590": 28160194560.0,
+ "11595": 28160194560.0,
+ "11600": 28160194560.0,
+ "11605": 28160194560.0,
+ "11610": 28160194560.0,
+ "11615": 28160194560.0,
+ "11620": 28160194560.0,
+ "11625": 28160194560.0,
+ "11630": 28160194560.0,
+ "11635": 28160194560.0,
+ "11640": 28160194560.0,
+ "11645": 28160194560.0,
+ "11650": 28160194560.0,
+ "11655": 28160194560.0,
+ "11660": 28160194560.0,
+ "11665": 28160194560.0,
+ "11670": 28160194560.0,
+ "11675": 28160194560.0,
+ "11680": 28160194560.0,
+ "11685": 28160194560.0,
+ "11690": 28160194560.0,
+ "11695": 28160194560.0,
+ "11700": 28160194560.0,
+ "11705": 28160194560.0,
+ "11710": 28160194560.0,
+ "11715": 28160194560.0,
+ "11720": 28160194560.0,
+ "11725": 28160194560.0,
+ "11730": 28160194560.0,
+ "11735": 28160194560.0,
+ "11740": 28160194560.0,
+ "11745": 28160194560.0,
+ "11750": 28160194560.0,
+ "11755": 28160194560.0,
+ "11760": 28160194560.0,
+ "11765": 28160194560.0,
+ "11770": 28160194560.0,
+ "11775": 28160194560.0,
+ "11780": 28160194560.0,
+ "11785": 28160194560.0,
+ "11790": 28160194560.0,
+ "11795": 28160194560.0,
+ "11800": 28160194560.0,
+ "11805": 28160194560.0,
+ "11810": 28160194560.0,
+ "11815": 28160194560.0,
+ "11820": 28160194560.0,
+ "11825": 28160194560.0,
+ "11830": 28160194560.0,
+ "11835": 28160194560.0,
+ "11840": 28160194560.0,
+ "11845": 28160194560.0,
+ "11850": 28160194560.0,
+ "11855": 28160194560.0,
+ "11860": 28160194560.0,
+ "11865": 28160194560.0,
+ "11870": 28160194560.0,
+ "11875": 28160194560.0,
+ "11880": 28160194560.0,
+ "11885": 28160194560.0,
+ "11890": 28160194560.0,
+ "11895": 28160194560.0,
+ "11900": 28160194560.0,
+ "11905": 28160194560.0,
+ "11910": 28160194560.0,
+ "11915": 28160194560.0,
+ "11920": 28160194560.0,
+ "11925": 28160194560.0,
+ "11930": 28160194560.0,
+ "11935": 28160194560.0,
+ "11940": 28160194560.0,
+ "11945": 28160194560.0,
+ "11950": 28160194560.0,
+ "11955": 28160194560.0,
+ "11960": 28160194560.0,
+ "11965": 28160194560.0,
+ "11970": 28160194560.0,
+ "11975": 28160194560.0,
+ "11980": 28160194560.0,
+ "11985": 28160194560.0,
+ "11990": 28160194560.0,
+ "11995": 28160194560.0,
+ "12000": 28160194560.0,
+ "12005": 28160194560.0,
+ "12010": 28160194560.0,
+ "12015": 28160194560.0,
+ "12020": 28160194560.0,
+ "12025": 28160194560.0,
+ "12030": 28160194560.0,
+ "12035": 28160194560.0,
+ "12040": 28160194560.0,
+ "12045": 28160194560.0,
+ "12050": 28160194560.0,
+ "12055": 28160194560.0,
+ "12060": 28160194560.0,
+ "12065": 28160194560.0,
+ "12070": 28160194560.0,
+ "12075": 28160194560.0,
+ "12080": 28160194560.0,
+ "12085": 28160194560.0,
+ "12090": 28160194560.0,
+ "12095": 28160194560.0,
+ "12100": 28160194560.0,
+ "12105": 28160194560.0,
+ "12110": 28160194560.0,
+ "12115": 28160194560.0,
+ "12120": 28160194560.0,
+ "12125": 28160194560.0,
+ "12130": 28160194560.0,
+ "12135": 28160194560.0,
+ "12140": 28160194560.0,
+ "12145": 28160194560.0,
+ "12150": 28160194560.0,
+ "12155": 28160194560.0,
+ "12160": 28160194560.0,
+ "12165": 28160194560.0,
+ "12170": 28160194560.0,
+ "12175": 28160194560.0,
+ "12180": 28160194560.0,
+ "12185": 28160194560.0,
+ "12190": 28160194560.0,
+ "12195": 28160194560.0,
+ "12200": 28160194560.0,
+ "12205": 28160194560.0,
+ "12210": 28160194560.0,
+ "12215": 28160194560.0,
+ "12220": 28160194560.0,
+ "12225": 28160194560.0,
+ "12230": 28160194560.0,
+ "12235": 28160194560.0,
+ "12240": 28160194560.0,
+ "12245": 28160194560.0,
+ "12250": 28160194560.0,
+ "12255": 28160194560.0,
+ "12260": 28160194560.0,
+ "12265": 28160194560.0,
+ "12270": 28160194560.0,
+ "12275": 28160194560.0,
+ "12280": 28160194560.0,
+ "12285": 28160194560.0,
+ "12290": 28160194560.0,
+ "12295": 28160194560.0,
+ "12300": 28160194560.0,
+ "12305": 28160194560.0,
+ "12310": 28160194560.0,
+ "12315": 28160194560.0,
+ "12320": 28160194560.0,
+ "12325": 28160194560.0,
+ "12330": 28160194560.0,
+ "12335": 28160194560.0,
+ "12340": 28160194560.0,
+ "12345": 28160194560.0,
+ "12350": 28160194560.0,
+ "12355": 28160194560.0,
+ "12360": 28160194560.0,
+ "12365": 28160194560.0,
+ "12370": 28160194560.0,
+ "12375": 28160194560.0,
+ "12380": 28160194560.0,
+ "12385": 28160194560.0,
+ "12390": 28160194560.0,
+ "12395": 28160194560.0,
+ "12400": 28160194560.0,
+ "12405": 28160194560.0,
+ "12410": 28160194560.0,
+ "12415": 28160194560.0,
+ "12420": 28160194560.0,
+ "12425": 28160194560.0,
+ "12430": 28160194560.0,
+ "12435": 28160194560.0,
+ "12440": 28160194560.0,
+ "12445": 28160194560.0,
+ "12450": 28160194560.0,
+ "12455": 28160194560.0,
+ "12460": 28160194560.0,
+ "12465": 28160194560.0,
+ "12470": 28160194560.0,
+ "12475": 28160194560.0,
+ "12480": 28160194560.0,
+ "12485": 28160194560.0,
+ "12490": 28160194560.0,
+ "12495": 28160194560.0,
+ "12500": 28160194560.0,
+ "12505": 28160194560.0,
+ "12510": 28160194560.0,
+ "12515": 28160194560.0,
+ "12520": 28160194560.0,
+ "12525": 28160194560.0,
+ "12530": 28160194560.0,
+ "12535": 28160194560.0,
+ "12540": 28160194560.0,
+ "12545": 28160194560.0,
+ "12550": 28160194560.0,
+ "12555": 28160194560.0,
+ "12560": 28160194560.0,
+ "12565": 28160194560.0,
+ "12570": 28160194560.0,
+ "12575": 28160194560.0,
+ "12580": 28160194560.0,
+ "12585": 28160194560.0,
+ "12590": 28160194560.0,
+ "12595": 28160194560.0,
+ "12600": 28160194560.0,
+ "12605": 28160194560.0,
+ "12610": 28160194560.0,
+ "12615": 28160194560.0,
+ "12620": 28160194560.0,
+ "12625": 28160194560.0,
+ "12630": 28160194560.0,
+ "12635": 28160194560.0,
+ "12640": 28160194560.0,
+ "12645": 28160194560.0,
+ "12650": 28160194560.0,
+ "12655": 28160194560.0,
+ "12660": 28160194560.0,
+ "12665": 28160194560.0,
+ "12670": 28160194560.0,
+ "12675": 28160194560.0,
+ "12680": 28160194560.0,
+ "12685": 28160194560.0,
+ "12690": 28160194560.0,
+ "12695": 28160194560.0,
+ "12700": 28160194560.0,
+ "12705": 28160194560.0,
+ "12710": 28160194560.0,
+ "12715": 28160194560.0,
+ "12720": "nan",
+ "12725": "nan",
+ "12730": "nan",
+ "12735": "nan",
+ "12740": "nan",
+ "12745": "nan",
+ "12750": "nan",
+ "12755": "nan",
+ "12760": "nan",
+ "12765": "nan",
+ "12770": "nan",
+ "12775": "nan",
+ "12780": "nan",
+ "12785": "nan",
+ "12790": "nan",
+ "12795": "nan",
+ "12800": "nan",
+ "12805": "nan",
+ "12810": "nan",
+ "12815": "nan",
+ "12820": "nan",
+ "12825": "nan",
+ "12830": "nan",
+ "12835": "nan",
+ "12840": "nan",
+ "12845": "nan",
+ "12850": "nan",
+ "12855": "nan",
+ "12860": "nan",
+ "12865": "nan",
+ "12870": "nan",
+ "12875": "nan",
+ "12880": "nan",
+ "12885": "nan",
+ "12890": "nan",
+ "12895": "nan",
+ "12900": "nan",
+ "12905": "nan",
+ "12910": "nan",
+ "12915": "nan",
+ "12920": "nan",
+ "12925": "nan",
+ "12930": "nan",
+ "12935": "nan",
+ "12940": "nan",
+ "12945": "nan",
+ "12950": "nan",
+ "12955": "nan",
+ "12960": "nan",
+ "12965": "nan",
+ "12970": "nan",
+ "12975": "nan",
+ "12980": "nan",
+ "12985": "nan",
+ "12990": "nan",
+ "12995": "nan",
+ "13000": "nan"
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": "nan",
+ "5": "nan",
+ "10": "nan",
+ "15": "nan",
+ "20": "nan",
+ "25": "nan",
+ "30": "nan",
+ "35": "nan",
+ "40": "nan",
+ "45": "nan",
+ "50": "nan",
+ "55": "nan",
+ "60": "nan",
+ "65": "nan",
+ "70": "nan",
+ "75": "nan",
+ "80": "nan",
+ "85": "nan",
+ "90": "nan",
+ "95": "nan",
+ "100": 1.58468,
+ "105": "nan",
+ "110": "nan",
+ "115": "nan",
+ "120": "nan",
+ "125": "nan",
+ "130": "nan",
+ "135": "nan",
+ "140": "nan",
+ "145": "nan",
+ "150": "nan",
+ "155": "nan",
+ "160": "nan",
+ "165": "nan",
+ "170": "nan",
+ "175": "nan",
+ "180": "nan",
+ "185": "nan",
+ "190": "nan",
+ "195": "nan",
+ "200": 1.46052,
+ "205": "nan",
+ "210": "nan",
+ "215": "nan",
+ "220": "nan",
+ "225": "nan",
+ "230": "nan",
+ "235": "nan",
+ "240": "nan",
+ "245": "nan",
+ "250": "nan",
+ "255": "nan",
+ "260": "nan",
+ "265": "nan",
+ "270": "nan",
+ "275": "nan",
+ "280": "nan",
+ "285": "nan",
+ "290": "nan",
+ "295": "nan",
+ "300": 1.48144,
+ "305": "nan",
+ "310": "nan",
+ "315": "nan",
+ "320": "nan",
+ "325": "nan",
+ "330": "nan",
+ "335": "nan",
+ "340": "nan",
+ "345": "nan",
+ "350": "nan",
+ "355": "nan",
+ "360": "nan",
+ "365": "nan",
+ "370": "nan",
+ "375": "nan",
+ "380": "nan",
+ "385": "nan",
+ "390": "nan",
+ "395": "nan",
+ "400": 1.5294,
+ "405": "nan",
+ "410": "nan",
+ "415": "nan",
+ "420": "nan",
+ "425": "nan",
+ "430": "nan",
+ "435": "nan",
+ "440": "nan",
+ "445": "nan",
+ "450": "nan",
+ "455": "nan",
+ "460": "nan",
+ "465": "nan",
+ "470": "nan",
+ "475": "nan",
+ "480": "nan",
+ "485": "nan",
+ "490": "nan",
+ "495": "nan",
+ "500": 1.52806,
+ "505": "nan",
+ "510": "nan",
+ "515": "nan",
+ "520": "nan",
+ "525": "nan",
+ "530": "nan",
+ "535": "nan",
+ "540": "nan",
+ "545": "nan",
+ "550": "nan",
+ "555": "nan",
+ "560": "nan",
+ "565": "nan",
+ "570": "nan",
+ "575": "nan",
+ "580": "nan",
+ "585": "nan",
+ "590": "nan",
+ "595": "nan",
+ "600": 1.69372,
+ "605": "nan",
+ "610": "nan",
+ "615": "nan",
+ "620": "nan",
+ "625": "nan",
+ "630": "nan",
+ "635": "nan",
+ "640": "nan",
+ "645": "nan",
+ "650": "nan",
+ "655": "nan",
+ "660": "nan",
+ "665": "nan",
+ "670": "nan",
+ "675": "nan",
+ "680": "nan",
+ "685": "nan",
+ "690": "nan",
+ "695": "nan",
+ "700": 1.45115,
+ "705": "nan",
+ "710": "nan",
+ "715": "nan",
+ "720": "nan",
+ "725": "nan",
+ "730": "nan",
+ "735": "nan",
+ "740": "nan",
+ "745": "nan",
+ "750": "nan",
+ "755": "nan",
+ "760": "nan",
+ "765": "nan",
+ "770": "nan",
+ "775": "nan",
+ "780": "nan",
+ "785": "nan",
+ "790": "nan",
+ "795": "nan",
+ "800": 1.56133,
+ "805": "nan",
+ "810": "nan",
+ "815": "nan",
+ "820": "nan",
+ "825": "nan",
+ "830": "nan",
+ "835": "nan",
+ "840": "nan",
+ "845": "nan",
+ "850": "nan",
+ "855": "nan",
+ "860": "nan",
+ "865": "nan",
+ "870": "nan",
+ "875": "nan",
+ "880": "nan",
+ "885": "nan",
+ "890": "nan",
+ "895": "nan",
+ "900": 1.44623,
+ "905": "nan",
+ "910": "nan",
+ "915": "nan",
+ "920": "nan",
+ "925": "nan",
+ "930": "nan",
+ "935": "nan",
+ "940": "nan",
+ "945": "nan",
+ "950": "nan",
+ "955": "nan",
+ "960": "nan",
+ "965": "nan",
+ "970": "nan",
+ "975": "nan",
+ "980": "nan",
+ "985": "nan",
+ "990": "nan",
+ "995": "nan",
+ "1000": 1.45115,
+ "1005": "nan",
+ "1010": "nan",
+ "1015": "nan",
+ "1020": "nan",
+ "1025": "nan",
+ "1030": "nan",
+ "1035": "nan",
+ "1040": "nan",
+ "1045": "nan",
+ "1050": "nan",
+ "1055": "nan",
+ "1060": "nan",
+ "1065": "nan",
+ "1070": "nan",
+ "1075": "nan",
+ "1080": "nan",
+ "1085": "nan",
+ "1090": "nan",
+ "1095": "nan",
+ "1100": 1.44011,
+ "1105": "nan",
+ "1110": "nan",
+ "1115": "nan",
+ "1120": "nan",
+ "1125": "nan",
+ "1130": "nan",
+ "1135": "nan",
+ "1140": "nan",
+ "1145": "nan",
+ "1150": "nan",
+ "1155": "nan",
+ "1160": "nan",
+ "1165": "nan",
+ "1170": "nan",
+ "1175": "nan",
+ "1180": "nan",
+ "1185": "nan",
+ "1190": "nan",
+ "1195": "nan",
+ "1200": 1.44546,
+ "1205": "nan",
+ "1210": "nan",
+ "1215": "nan",
+ "1220": "nan",
+ "1225": "nan",
+ "1230": "nan",
+ "1235": "nan",
+ "1240": "nan",
+ "1245": "nan",
+ "1250": "nan",
+ "1255": "nan",
+ "1260": "nan",
+ "1265": "nan",
+ "1270": "nan",
+ "1275": "nan",
+ "1280": "nan",
+ "1285": "nan",
+ "1290": "nan",
+ "1295": "nan",
+ "1300": 1.43704,
+ "1305": "nan",
+ "1310": "nan",
+ "1315": "nan",
+ "1320": "nan",
+ "1325": "nan",
+ "1330": "nan",
+ "1335": "nan",
+ "1340": "nan",
+ "1345": "nan",
+ "1350": "nan",
+ "1355": "nan",
+ "1360": "nan",
+ "1365": "nan",
+ "1370": "nan",
+ "1375": "nan",
+ "1380": "nan",
+ "1385": "nan",
+ "1390": "nan",
+ "1395": "nan",
+ "1400": 1.47549,
+ "1405": "nan",
+ "1410": "nan",
+ "1415": "nan",
+ "1420": "nan",
+ "1425": "nan",
+ "1430": "nan",
+ "1435": "nan",
+ "1440": "nan",
+ "1445": "nan",
+ "1450": "nan",
+ "1455": "nan",
+ "1460": "nan",
+ "1465": "nan",
+ "1470": "nan",
+ "1475": "nan",
+ "1480": "nan",
+ "1485": "nan",
+ "1490": "nan",
+ "1495": "nan",
+ "1500": 1.5242,
+ "1505": "nan",
+ "1510": "nan",
+ "1515": "nan",
+ "1520": "nan",
+ "1525": "nan",
+ "1530": "nan",
+ "1535": "nan",
+ "1540": "nan",
+ "1545": "nan",
+ "1550": "nan",
+ "1555": "nan",
+ "1560": "nan",
+ "1565": "nan",
+ "1570": "nan",
+ "1575": "nan",
+ "1580": "nan",
+ "1585": "nan",
+ "1590": "nan",
+ "1595": "nan",
+ "1600": 1.44191,
+ "1605": "nan",
+ "1610": "nan",
+ "1615": "nan",
+ "1620": "nan",
+ "1625": "nan",
+ "1630": "nan",
+ "1635": "nan",
+ "1640": "nan",
+ "1645": "nan",
+ "1650": "nan",
+ "1655": "nan",
+ "1660": "nan",
+ "1665": "nan",
+ "1670": "nan",
+ "1675": "nan",
+ "1680": "nan",
+ "1685": "nan",
+ "1690": "nan",
+ "1695": "nan",
+ "1700": 1.43495,
+ "1705": "nan",
+ "1710": "nan",
+ "1715": "nan",
+ "1720": "nan",
+ "1725": "nan",
+ "1730": "nan",
+ "1735": "nan",
+ "1740": "nan",
+ "1745": "nan",
+ "1750": "nan",
+ "1755": "nan",
+ "1760": "nan",
+ "1765": "nan",
+ "1770": "nan",
+ "1775": "nan",
+ "1780": "nan",
+ "1785": "nan",
+ "1790": "nan",
+ "1795": "nan",
+ "1800": 1.4335,
+ "1805": "nan",
+ "1810": "nan",
+ "1815": "nan",
+ "1820": "nan",
+ "1825": "nan",
+ "1830": "nan",
+ "1835": "nan",
+ "1840": "nan",
+ "1845": "nan",
+ "1850": "nan",
+ "1855": "nan",
+ "1860": "nan",
+ "1865": "nan",
+ "1870": "nan",
+ "1875": "nan",
+ "1880": "nan",
+ "1885": "nan",
+ "1890": "nan",
+ "1895": "nan",
+ "1900": 1.46027,
+ "1905": "nan",
+ "1910": "nan",
+ "1915": "nan",
+ "1920": "nan",
+ "1925": "nan",
+ "1930": "nan",
+ "1935": "nan",
+ "1940": "nan",
+ "1945": "nan",
+ "1950": "nan",
+ "1955": "nan",
+ "1960": "nan",
+ "1965": "nan",
+ "1970": "nan",
+ "1975": "nan",
+ "1980": "nan",
+ "1985": "nan",
+ "1990": "nan",
+ "1995": "nan",
+ "2000": 1.43986,
+ "2005": "nan",
+ "2010": "nan",
+ "2015": "nan",
+ "2020": "nan",
+ "2025": "nan",
+ "2030": "nan",
+ "2035": "nan",
+ "2040": "nan",
+ "2045": "nan",
+ "2050": "nan",
+ "2055": "nan",
+ "2060": "nan",
+ "2065": "nan",
+ "2070": "nan",
+ "2075": "nan",
+ "2080": "nan",
+ "2085": "nan",
+ "2090": "nan",
+ "2095": "nan",
+ "2100": 1.47505,
+ "2105": "nan",
+ "2110": "nan",
+ "2115": "nan",
+ "2120": "nan",
+ "2125": "nan",
+ "2130": "nan",
+ "2135": "nan",
+ "2140": "nan",
+ "2145": "nan",
+ "2150": "nan",
+ "2155": "nan",
+ "2160": "nan",
+ "2165": "nan",
+ "2170": "nan",
+ "2175": "nan",
+ "2180": "nan",
+ "2185": "nan",
+ "2190": "nan",
+ "2195": "nan",
+ "2200": 1.43441,
+ "2205": "nan",
+ "2210": "nan",
+ "2215": "nan",
+ "2220": "nan",
+ "2225": "nan",
+ "2230": "nan",
+ "2235": "nan",
+ "2240": "nan",
+ "2245": "nan",
+ "2250": "nan",
+ "2255": "nan",
+ "2260": "nan",
+ "2265": "nan",
+ "2270": "nan",
+ "2275": "nan",
+ "2280": "nan",
+ "2285": "nan",
+ "2290": "nan",
+ "2295": "nan",
+ "2300": 1.54495,
+ "2305": "nan",
+ "2310": "nan",
+ "2315": "nan",
+ "2320": "nan",
+ "2325": "nan",
+ "2330": "nan",
+ "2335": "nan",
+ "2340": "nan",
+ "2345": "nan",
+ "2350": "nan",
+ "2355": "nan",
+ "2360": "nan",
+ "2365": "nan",
+ "2370": "nan",
+ "2375": "nan",
+ "2380": "nan",
+ "2385": "nan",
+ "2390": "nan",
+ "2395": "nan",
+ "2400": 1.51503,
+ "2405": "nan",
+ "2410": "nan",
+ "2415": "nan",
+ "2420": "nan",
+ "2425": "nan",
+ "2430": "nan",
+ "2435": "nan",
+ "2440": "nan",
+ "2445": "nan",
+ "2450": "nan",
+ "2455": "nan",
+ "2460": "nan",
+ "2465": "nan",
+ "2470": "nan",
+ "2475": "nan",
+ "2480": "nan",
+ "2485": "nan",
+ "2490": "nan",
+ "2495": "nan",
+ "2500": 1.43246,
+ "2505": "nan",
+ "2510": "nan",
+ "2515": "nan",
+ "2520": "nan",
+ "2525": "nan",
+ "2530": "nan",
+ "2535": "nan",
+ "2540": "nan",
+ "2545": "nan",
+ "2550": "nan",
+ "2555": "nan",
+ "2560": "nan",
+ "2565": "nan",
+ "2570": "nan",
+ "2575": "nan",
+ "2580": "nan",
+ "2585": "nan",
+ "2590": "nan",
+ "2595": "nan",
+ "2600": 1.47287,
+ "2605": "nan",
+ "2610": "nan",
+ "2615": "nan",
+ "2620": "nan",
+ "2625": "nan",
+ "2630": "nan",
+ "2635": "nan",
+ "2640": "nan",
+ "2645": "nan",
+ "2650": "nan",
+ "2655": "nan",
+ "2660": "nan",
+ "2665": "nan",
+ "2670": "nan",
+ "2675": "nan",
+ "2680": "nan",
+ "2685": "nan",
+ "2690": "nan",
+ "2695": "nan",
+ "2700": 1.42953,
+ "2705": "nan",
+ "2710": "nan",
+ "2715": "nan",
+ "2720": "nan",
+ "2725": "nan",
+ "2730": "nan",
+ "2735": "nan",
+ "2740": "nan",
+ "2745": "nan",
+ "2750": "nan",
+ "2755": "nan",
+ "2760": "nan",
+ "2765": "nan",
+ "2770": "nan",
+ "2775": "nan",
+ "2780": "nan",
+ "2785": "nan",
+ "2790": "nan",
+ "2795": "nan",
+ "2800": 1.46691,
+ "2805": "nan",
+ "2810": "nan",
+ "2815": "nan",
+ "2820": "nan",
+ "2825": "nan",
+ "2830": "nan",
+ "2835": "nan",
+ "2840": "nan",
+ "2845": "nan",
+ "2850": "nan",
+ "2855": "nan",
+ "2860": "nan",
+ "2865": "nan",
+ "2870": "nan",
+ "2875": "nan",
+ "2880": "nan",
+ "2885": "nan",
+ "2890": "nan",
+ "2895": "nan",
+ "2900": 1.43076,
+ "2905": "nan",
+ "2910": "nan",
+ "2915": "nan",
+ "2920": "nan",
+ "2925": "nan",
+ "2930": "nan",
+ "2935": "nan",
+ "2940": "nan",
+ "2945": "nan",
+ "2950": "nan",
+ "2955": "nan",
+ "2960": "nan",
+ "2965": "nan",
+ "2970": "nan",
+ "2975": "nan",
+ "2980": "nan",
+ "2985": "nan",
+ "2990": "nan",
+ "2995": "nan",
+ "3000": 1.64091,
+ "3005": "nan",
+ "3010": "nan",
+ "3015": "nan",
+ "3020": "nan",
+ "3025": "nan",
+ "3030": "nan",
+ "3035": "nan",
+ "3040": "nan",
+ "3045": "nan",
+ "3050": "nan",
+ "3055": "nan",
+ "3060": "nan",
+ "3065": "nan",
+ "3070": "nan",
+ "3075": "nan",
+ "3080": "nan",
+ "3085": "nan",
+ "3090": "nan",
+ "3095": "nan",
+ "3100": 1.59233,
+ "3105": "nan",
+ "3110": "nan",
+ "3115": "nan",
+ "3120": "nan",
+ "3125": "nan",
+ "3130": "nan",
+ "3135": "nan",
+ "3140": "nan",
+ "3145": "nan",
+ "3150": "nan",
+ "3155": "nan",
+ "3160": "nan",
+ "3165": "nan",
+ "3170": "nan",
+ "3175": "nan",
+ "3180": "nan",
+ "3185": "nan",
+ "3190": "nan",
+ "3195": "nan",
+ "3200": 1.43772,
+ "3205": "nan",
+ "3210": "nan",
+ "3215": "nan",
+ "3220": "nan",
+ "3225": "nan",
+ "3230": "nan",
+ "3235": "nan",
+ "3240": "nan",
+ "3245": "nan",
+ "3250": "nan",
+ "3255": "nan",
+ "3260": "nan",
+ "3265": "nan",
+ "3270": "nan",
+ "3275": "nan",
+ "3280": "nan",
+ "3285": "nan",
+ "3290": "nan",
+ "3295": "nan",
+ "3300": 1.46407,
+ "3305": "nan",
+ "3310": "nan",
+ "3315": "nan",
+ "3320": "nan",
+ "3325": "nan",
+ "3330": "nan",
+ "3335": "nan",
+ "3340": "nan",
+ "3345": "nan",
+ "3350": "nan",
+ "3355": "nan",
+ "3360": "nan",
+ "3365": "nan",
+ "3370": "nan",
+ "3375": "nan",
+ "3380": "nan",
+ "3385": "nan",
+ "3390": "nan",
+ "3395": "nan",
+ "3400": 1.42524,
+ "3405": "nan",
+ "3410": "nan",
+ "3415": "nan",
+ "3420": "nan",
+ "3425": "nan",
+ "3430": "nan",
+ "3435": "nan",
+ "3440": "nan",
+ "3445": "nan",
+ "3450": "nan",
+ "3455": "nan",
+ "3460": "nan",
+ "3465": "nan",
+ "3470": "nan",
+ "3475": "nan",
+ "3480": "nan",
+ "3485": "nan",
+ "3490": "nan",
+ "3495": "nan",
+ "3500": 1.4277,
+ "3505": "nan",
+ "3510": "nan",
+ "3515": "nan",
+ "3520": "nan",
+ "3525": "nan",
+ "3530": "nan",
+ "3535": "nan",
+ "3540": "nan",
+ "3545": "nan",
+ "3550": "nan",
+ "3555": "nan",
+ "3560": "nan",
+ "3565": "nan",
+ "3570": "nan",
+ "3575": "nan",
+ "3580": "nan",
+ "3585": "nan",
+ "3590": "nan",
+ "3595": "nan",
+ "3600": 1.43193,
+ "3605": "nan",
+ "3610": "nan",
+ "3615": "nan",
+ "3620": "nan",
+ "3625": "nan",
+ "3630": "nan",
+ "3635": "nan",
+ "3640": "nan",
+ "3645": "nan",
+ "3650": "nan",
+ "3655": "nan",
+ "3660": "nan",
+ "3665": "nan",
+ "3670": "nan",
+ "3675": "nan",
+ "3680": "nan",
+ "3685": "nan",
+ "3690": "nan",
+ "3695": "nan",
+ "3700": 1.54239,
+ "3705": "nan",
+ "3710": "nan",
+ "3715": "nan",
+ "3720": "nan",
+ "3725": "nan",
+ "3730": "nan",
+ "3735": "nan",
+ "3740": "nan",
+ "3745": "nan",
+ "3750": "nan",
+ "3755": "nan",
+ "3760": "nan",
+ "3765": "nan",
+ "3770": "nan",
+ "3775": "nan",
+ "3780": "nan",
+ "3785": "nan",
+ "3790": "nan",
+ "3795": "nan",
+ "3800": 1.51705,
+ "3805": "nan",
+ "3810": "nan",
+ "3815": "nan",
+ "3820": "nan",
+ "3825": "nan",
+ "3830": "nan",
+ "3835": "nan",
+ "3840": "nan",
+ "3845": "nan",
+ "3850": "nan",
+ "3855": "nan",
+ "3860": "nan",
+ "3865": "nan",
+ "3870": "nan",
+ "3875": "nan",
+ "3880": "nan",
+ "3885": "nan",
+ "3890": "nan",
+ "3895": "nan",
+ "3900": 1.67534,
+ "3905": "nan",
+ "3910": "nan",
+ "3915": "nan",
+ "3920": "nan",
+ "3925": "nan",
+ "3930": "nan",
+ "3935": "nan",
+ "3940": "nan",
+ "3945": "nan",
+ "3950": "nan",
+ "3955": "nan",
+ "3960": "nan",
+ "3965": "nan",
+ "3970": "nan",
+ "3975": "nan",
+ "3980": "nan",
+ "3985": "nan",
+ "3990": "nan",
+ "3995": "nan",
+ "4000": 1.67907,
+ "4005": "nan",
+ "4010": "nan",
+ "4015": "nan",
+ "4020": "nan",
+ "4025": "nan",
+ "4030": "nan",
+ "4035": "nan",
+ "4040": "nan",
+ "4045": "nan",
+ "4050": "nan",
+ "4055": "nan",
+ "4060": "nan",
+ "4065": "nan",
+ "4070": "nan",
+ "4075": "nan",
+ "4080": "nan",
+ "4085": "nan",
+ "4090": "nan",
+ "4095": "nan",
+ "4100": 1.4593,
+ "4105": "nan",
+ "4110": "nan",
+ "4115": "nan",
+ "4120": "nan",
+ "4125": "nan",
+ "4130": "nan",
+ "4135": "nan",
+ "4140": "nan",
+ "4145": "nan",
+ "4150": "nan",
+ "4155": "nan",
+ "4160": "nan",
+ "4165": "nan",
+ "4170": "nan",
+ "4175": "nan",
+ "4180": "nan",
+ "4185": "nan",
+ "4190": "nan",
+ "4195": "nan",
+ "4200": 1.42454,
+ "4205": "nan",
+ "4210": "nan",
+ "4215": "nan",
+ "4220": "nan",
+ "4225": "nan",
+ "4230": "nan",
+ "4235": "nan",
+ "4240": "nan",
+ "4245": "nan",
+ "4250": "nan",
+ "4255": "nan",
+ "4260": "nan",
+ "4265": "nan",
+ "4270": "nan",
+ "4275": "nan",
+ "4280": "nan",
+ "4285": "nan",
+ "4290": "nan",
+ "4295": "nan",
+ "4300": 1.45808,
+ "4305": "nan",
+ "4310": "nan",
+ "4315": "nan",
+ "4320": "nan",
+ "4325": "nan",
+ "4330": "nan",
+ "4335": "nan",
+ "4340": "nan",
+ "4345": "nan",
+ "4350": "nan",
+ "4355": "nan",
+ "4360": "nan",
+ "4365": "nan",
+ "4370": "nan",
+ "4375": "nan",
+ "4380": "nan",
+ "4385": "nan",
+ "4390": "nan",
+ "4395": "nan",
+ "4400": 1.4385,
+ "4405": "nan",
+ "4410": "nan",
+ "4415": "nan",
+ "4420": "nan",
+ "4425": "nan",
+ "4430": "nan",
+ "4435": "nan",
+ "4440": "nan",
+ "4445": "nan",
+ "4450": "nan",
+ "4455": "nan",
+ "4460": "nan",
+ "4465": "nan",
+ "4470": "nan",
+ "4475": "nan",
+ "4480": "nan",
+ "4485": "nan",
+ "4490": "nan",
+ "4495": "nan",
+ "4500": 1.43321,
+ "4505": "nan",
+ "4510": "nan",
+ "4515": "nan",
+ "4520": "nan",
+ "4525": "nan",
+ "4530": "nan",
+ "4535": "nan",
+ "4540": "nan",
+ "4545": "nan",
+ "4550": "nan",
+ "4555": "nan",
+ "4560": "nan",
+ "4565": "nan",
+ "4570": "nan",
+ "4575": "nan",
+ "4580": "nan",
+ "4585": "nan",
+ "4590": "nan",
+ "4595": "nan",
+ "4600": 3.29055,
+ "4605": "nan",
+ "4610": "nan",
+ "4615": "nan",
+ "4620": "nan",
+ "4625": "nan",
+ "4630": "nan",
+ "4635": "nan",
+ "4640": "nan",
+ "4645": "nan",
+ "4650": "nan",
+ "4655": "nan",
+ "4660": "nan",
+ "4665": "nan",
+ "4670": "nan",
+ "4675": "nan",
+ "4680": "nan",
+ "4685": "nan",
+ "4690": "nan",
+ "4695": "nan",
+ "4700": 1.42713,
+ "4705": "nan",
+ "4710": "nan",
+ "4715": "nan",
+ "4720": "nan",
+ "4725": "nan",
+ "4730": "nan",
+ "4735": "nan",
+ "4740": "nan",
+ "4745": "nan",
+ "4750": "nan",
+ "4755": "nan",
+ "4760": "nan",
+ "4765": "nan",
+ "4770": "nan",
+ "4775": "nan",
+ "4780": "nan",
+ "4785": "nan",
+ "4790": "nan",
+ "4795": "nan",
+ "4800": 1.47366,
+ "4805": "nan",
+ "4810": "nan",
+ "4815": "nan",
+ "4820": "nan",
+ "4825": "nan",
+ "4830": "nan",
+ "4835": "nan",
+ "4840": "nan",
+ "4845": "nan",
+ "4850": "nan",
+ "4855": "nan",
+ "4860": "nan",
+ "4865": "nan",
+ "4870": "nan",
+ "4875": "nan",
+ "4880": "nan",
+ "4885": "nan",
+ "4890": "nan",
+ "4895": "nan",
+ "4900": 1.42583,
+ "4905": "nan",
+ "4910": "nan",
+ "4915": "nan",
+ "4920": "nan",
+ "4925": "nan",
+ "4930": "nan",
+ "4935": "nan",
+ "4940": "nan",
+ "4945": "nan",
+ "4950": "nan",
+ "4955": "nan",
+ "4960": "nan",
+ "4965": "nan",
+ "4970": "nan",
+ "4975": "nan",
+ "4980": "nan",
+ "4985": "nan",
+ "4990": "nan",
+ "4995": "nan",
+ "5000": 1.51639,
+ "5005": "nan",
+ "5010": "nan",
+ "5015": "nan",
+ "5020": "nan",
+ "5025": "nan",
+ "5030": "nan",
+ "5035": "nan",
+ "5040": "nan",
+ "5045": "nan",
+ "5050": "nan",
+ "5055": "nan",
+ "5060": "nan",
+ "5065": "nan",
+ "5070": "nan",
+ "5075": "nan",
+ "5080": "nan",
+ "5085": "nan",
+ "5090": "nan",
+ "5095": "nan",
+ "5100": 1.4372,
+ "5105": "nan",
+ "5110": "nan",
+ "5115": "nan",
+ "5120": "nan",
+ "5125": "nan",
+ "5130": "nan",
+ "5135": "nan",
+ "5140": "nan",
+ "5145": "nan",
+ "5150": "nan",
+ "5155": "nan",
+ "5160": "nan",
+ "5165": "nan",
+ "5170": "nan",
+ "5175": "nan",
+ "5180": "nan",
+ "5185": "nan",
+ "5190": "nan",
+ "5195": "nan",
+ "5200": 1.432,
+ "5205": "nan",
+ "5210": "nan",
+ "5215": "nan",
+ "5220": "nan",
+ "5225": "nan",
+ "5230": "nan",
+ "5235": "nan",
+ "5240": "nan",
+ "5245": "nan",
+ "5250": "nan",
+ "5255": "nan",
+ "5260": "nan",
+ "5265": "nan",
+ "5270": "nan",
+ "5275": "nan",
+ "5280": "nan",
+ "5285": "nan",
+ "5290": "nan",
+ "5295": "nan",
+ "5300": 1.50922,
+ "5305": "nan",
+ "5310": "nan",
+ "5315": "nan",
+ "5320": "nan",
+ "5325": "nan",
+ "5330": "nan",
+ "5335": "nan",
+ "5340": "nan",
+ "5345": "nan",
+ "5350": "nan",
+ "5355": "nan",
+ "5360": "nan",
+ "5365": "nan",
+ "5370": "nan",
+ "5375": "nan",
+ "5380": "nan",
+ "5385": "nan",
+ "5390": "nan",
+ "5395": "nan",
+ "5400": 1.50495,
+ "5405": "nan",
+ "5410": "nan",
+ "5415": "nan",
+ "5420": "nan",
+ "5425": "nan",
+ "5430": "nan",
+ "5435": "nan",
+ "5440": "nan",
+ "5445": "nan",
+ "5450": "nan",
+ "5455": "nan",
+ "5460": "nan",
+ "5465": "nan",
+ "5470": "nan",
+ "5475": "nan",
+ "5480": "nan",
+ "5485": "nan",
+ "5490": "nan",
+ "5495": "nan",
+ "5500": 1.43826,
+ "5505": "nan",
+ "5510": "nan",
+ "5515": "nan",
+ "5520": "nan",
+ "5525": "nan",
+ "5530": "nan",
+ "5535": "nan",
+ "5540": "nan",
+ "5545": "nan",
+ "5550": "nan",
+ "5555": "nan",
+ "5560": "nan",
+ "5565": "nan",
+ "5570": "nan",
+ "5575": "nan",
+ "5580": "nan",
+ "5585": "nan",
+ "5590": "nan",
+ "5595": "nan",
+ "5600": 1.43314,
+ "5605": "nan",
+ "5610": "nan",
+ "5615": "nan",
+ "5620": "nan",
+ "5625": "nan",
+ "5630": "nan",
+ "5635": "nan",
+ "5640": "nan",
+ "5645": "nan",
+ "5650": "nan",
+ "5655": "nan",
+ "5660": "nan",
+ "5665": "nan",
+ "5670": "nan",
+ "5675": "nan",
+ "5680": "nan",
+ "5685": "nan",
+ "5690": "nan",
+ "5695": "nan",
+ "5700": 1.42578,
+ "5705": "nan",
+ "5710": "nan",
+ "5715": "nan",
+ "5720": "nan",
+ "5725": "nan",
+ "5730": "nan",
+ "5735": "nan",
+ "5740": "nan",
+ "5745": "nan",
+ "5750": "nan",
+ "5755": "nan",
+ "5760": "nan",
+ "5765": "nan",
+ "5770": "nan",
+ "5775": "nan",
+ "5780": "nan",
+ "5785": "nan",
+ "5790": "nan",
+ "5795": "nan",
+ "5800": 1.54238,
+ "5805": "nan",
+ "5810": "nan",
+ "5815": "nan",
+ "5820": "nan",
+ "5825": "nan",
+ "5830": "nan",
+ "5835": "nan",
+ "5840": "nan",
+ "5845": "nan",
+ "5850": "nan",
+ "5855": "nan",
+ "5860": "nan",
+ "5865": "nan",
+ "5870": "nan",
+ "5875": "nan",
+ "5880": "nan",
+ "5885": "nan",
+ "5890": "nan",
+ "5895": "nan",
+ "5900": 1.66057,
+ "5905": "nan",
+ "5910": "nan",
+ "5915": "nan",
+ "5920": "nan",
+ "5925": "nan",
+ "5930": "nan",
+ "5935": "nan",
+ "5940": "nan",
+ "5945": "nan",
+ "5950": "nan",
+ "5955": "nan",
+ "5960": "nan",
+ "5965": "nan",
+ "5970": "nan",
+ "5975": "nan",
+ "5980": "nan",
+ "5985": "nan",
+ "5990": "nan",
+ "5995": "nan",
+ "6000": 1.43351,
+ "6005": "nan",
+ "6010": "nan",
+ "6015": "nan",
+ "6020": "nan",
+ "6025": "nan",
+ "6030": "nan",
+ "6035": "nan",
+ "6040": "nan",
+ "6045": "nan",
+ "6050": "nan",
+ "6055": "nan",
+ "6060": "nan",
+ "6065": "nan",
+ "6070": "nan",
+ "6075": "nan",
+ "6080": "nan",
+ "6085": "nan",
+ "6090": "nan",
+ "6095": "nan",
+ "6100": 1.54984,
+ "6105": "nan",
+ "6110": "nan",
+ "6115": "nan",
+ "6120": "nan",
+ "6125": "nan",
+ "6130": "nan",
+ "6135": "nan",
+ "6140": "nan",
+ "6145": "nan",
+ "6150": "nan",
+ "6155": "nan",
+ "6160": "nan",
+ "6165": "nan",
+ "6170": "nan",
+ "6175": "nan",
+ "6180": "nan",
+ "6185": "nan",
+ "6190": "nan",
+ "6195": "nan",
+ "6200": 1.51837,
+ "6205": "nan",
+ "6210": "nan",
+ "6215": "nan",
+ "6220": "nan",
+ "6225": "nan",
+ "6230": "nan",
+ "6235": "nan",
+ "6240": "nan",
+ "6245": "nan",
+ "6250": "nan",
+ "6255": "nan",
+ "6260": "nan",
+ "6265": "nan",
+ "6270": "nan",
+ "6275": "nan",
+ "6280": "nan",
+ "6285": "nan",
+ "6290": "nan",
+ "6295": "nan",
+ "6300": 1.43501,
+ "6305": "nan",
+ "6310": "nan",
+ "6315": "nan",
+ "6320": "nan",
+ "6325": "nan",
+ "6330": "nan",
+ "6335": "nan",
+ "6340": "nan",
+ "6345": "nan",
+ "6350": "nan",
+ "6355": "nan",
+ "6360": "nan",
+ "6365": "nan",
+ "6370": "nan",
+ "6375": "nan",
+ "6380": "nan",
+ "6385": "nan",
+ "6390": "nan",
+ "6395": "nan",
+ "6400": 1.44369,
+ "6405": "nan",
+ "6410": "nan",
+ "6415": "nan",
+ "6420": "nan",
+ "6425": "nan",
+ "6430": "nan",
+ "6435": "nan",
+ "6440": "nan",
+ "6445": "nan",
+ "6450": "nan",
+ "6455": "nan",
+ "6460": "nan",
+ "6465": "nan",
+ "6470": "nan",
+ "6475": "nan",
+ "6480": "nan",
+ "6485": "nan",
+ "6490": "nan",
+ "6495": "nan",
+ "6500": 1.69151,
+ "6505": "nan",
+ "6510": "nan",
+ "6515": "nan",
+ "6520": "nan",
+ "6525": "nan",
+ "6530": "nan",
+ "6535": "nan",
+ "6540": "nan",
+ "6545": "nan",
+ "6550": "nan",
+ "6555": "nan",
+ "6560": "nan",
+ "6565": "nan",
+ "6570": "nan",
+ "6575": "nan",
+ "6580": "nan",
+ "6585": "nan",
+ "6590": "nan",
+ "6595": "nan",
+ "6600": 1.73692,
+ "6605": "nan",
+ "6610": "nan",
+ "6615": "nan",
+ "6620": "nan",
+ "6625": "nan",
+ "6630": "nan",
+ "6635": "nan",
+ "6640": "nan",
+ "6645": "nan",
+ "6650": "nan",
+ "6655": "nan",
+ "6660": "nan",
+ "6665": "nan",
+ "6670": "nan",
+ "6675": "nan",
+ "6680": "nan",
+ "6685": "nan",
+ "6690": "nan",
+ "6695": "nan",
+ "6700": 1.50479,
+ "6705": "nan",
+ "6710": "nan",
+ "6715": "nan",
+ "6720": "nan",
+ "6725": "nan",
+ "6730": "nan",
+ "6735": "nan",
+ "6740": "nan",
+ "6745": "nan",
+ "6750": "nan",
+ "6755": "nan",
+ "6760": "nan",
+ "6765": "nan",
+ "6770": "nan",
+ "6775": "nan",
+ "6780": "nan",
+ "6785": "nan",
+ "6790": "nan",
+ "6795": "nan",
+ "6800": 1.42689,
+ "6805": "nan",
+ "6810": "nan",
+ "6815": "nan",
+ "6820": "nan",
+ "6825": "nan",
+ "6830": "nan",
+ "6835": "nan",
+ "6840": "nan",
+ "6845": "nan",
+ "6850": "nan",
+ "6855": "nan",
+ "6860": "nan",
+ "6865": "nan",
+ "6870": "nan",
+ "6875": "nan",
+ "6880": "nan",
+ "6885": "nan",
+ "6890": "nan",
+ "6895": "nan",
+ "6900": 1.50703,
+ "6905": "nan",
+ "6910": "nan",
+ "6915": "nan",
+ "6920": "nan",
+ "6925": "nan",
+ "6930": "nan",
+ "6935": "nan",
+ "6940": "nan",
+ "6945": "nan",
+ "6950": "nan",
+ "6955": "nan",
+ "6960": "nan",
+ "6965": "nan",
+ "6970": "nan",
+ "6975": "nan",
+ "6980": "nan",
+ "6985": "nan",
+ "6990": "nan",
+ "6995": "nan",
+ "7000": 1.4532,
+ "7005": "nan",
+ "7010": "nan",
+ "7015": "nan",
+ "7020": "nan",
+ "7025": "nan",
+ "7030": "nan",
+ "7035": "nan",
+ "7040": "nan",
+ "7045": "nan",
+ "7050": "nan",
+ "7055": "nan",
+ "7060": "nan",
+ "7065": "nan",
+ "7070": "nan",
+ "7075": "nan",
+ "7080": "nan",
+ "7085": "nan",
+ "7090": "nan",
+ "7095": "nan",
+ "7100": 1.51593,
+ "7105": "nan",
+ "7110": "nan",
+ "7115": "nan",
+ "7120": "nan",
+ "7125": "nan",
+ "7130": "nan",
+ "7135": "nan",
+ "7140": "nan",
+ "7145": "nan",
+ "7150": "nan",
+ "7155": "nan",
+ "7160": "nan",
+ "7165": "nan",
+ "7170": "nan",
+ "7175": "nan",
+ "7180": "nan",
+ "7185": "nan",
+ "7190": "nan",
+ "7195": "nan",
+ "7200": 1.45753,
+ "7205": "nan",
+ "7210": "nan",
+ "7215": "nan",
+ "7220": "nan",
+ "7225": "nan",
+ "7230": "nan",
+ "7235": "nan",
+ "7240": "nan",
+ "7245": "nan",
+ "7250": "nan",
+ "7255": "nan",
+ "7260": "nan",
+ "7265": "nan",
+ "7270": "nan",
+ "7275": "nan",
+ "7280": "nan",
+ "7285": "nan",
+ "7290": "nan",
+ "7295": "nan",
+ "7300": 1.61042,
+ "7305": "nan",
+ "7310": "nan",
+ "7315": "nan",
+ "7320": "nan",
+ "7325": "nan",
+ "7330": "nan",
+ "7335": "nan",
+ "7340": "nan",
+ "7345": "nan",
+ "7350": "nan",
+ "7355": "nan",
+ "7360": "nan",
+ "7365": "nan",
+ "7370": "nan",
+ "7375": "nan",
+ "7380": "nan",
+ "7385": "nan",
+ "7390": "nan",
+ "7395": "nan",
+ "7400": 1.43351,
+ "7405": "nan",
+ "7410": "nan",
+ "7415": "nan",
+ "7420": "nan",
+ "7425": "nan",
+ "7430": "nan",
+ "7435": "nan",
+ "7440": "nan",
+ "7445": "nan",
+ "7450": "nan",
+ "7455": "nan",
+ "7460": "nan",
+ "7465": "nan",
+ "7470": "nan",
+ "7475": "nan",
+ "7480": "nan",
+ "7485": "nan",
+ "7490": "nan",
+ "7495": "nan",
+ "7500": 1.45693,
+ "7505": "nan",
+ "7510": "nan",
+ "7515": "nan",
+ "7520": "nan",
+ "7525": "nan",
+ "7530": "nan",
+ "7535": "nan",
+ "7540": "nan",
+ "7545": "nan",
+ "7550": "nan",
+ "7555": "nan",
+ "7560": "nan",
+ "7565": "nan",
+ "7570": "nan",
+ "7575": "nan",
+ "7580": "nan",
+ "7585": "nan",
+ "7590": "nan",
+ "7595": "nan",
+ "7600": 1.43171,
+ "7605": "nan",
+ "7610": "nan",
+ "7615": "nan",
+ "7620": "nan",
+ "7625": "nan",
+ "7630": "nan",
+ "7635": "nan",
+ "7640": "nan",
+ "7645": "nan",
+ "7650": "nan",
+ "7655": "nan",
+ "7660": "nan",
+ "7665": "nan",
+ "7670": "nan",
+ "7675": "nan",
+ "7680": "nan",
+ "7685": "nan",
+ "7690": "nan",
+ "7695": "nan",
+ "7700": 1.42897,
+ "7705": "nan",
+ "7710": "nan",
+ "7715": "nan",
+ "7720": "nan",
+ "7725": "nan",
+ "7730": "nan",
+ "7735": "nan",
+ "7740": "nan",
+ "7745": "nan",
+ "7750": "nan",
+ "7755": "nan",
+ "7760": "nan",
+ "7765": "nan",
+ "7770": "nan",
+ "7775": "nan",
+ "7780": "nan",
+ "7785": "nan",
+ "7790": "nan",
+ "7795": "nan",
+ "7800": 1.44919,
+ "7805": "nan",
+ "7810": "nan",
+ "7815": "nan",
+ "7820": "nan",
+ "7825": "nan",
+ "7830": "nan",
+ "7835": "nan",
+ "7840": "nan",
+ "7845": "nan",
+ "7850": "nan",
+ "7855": "nan",
+ "7860": "nan",
+ "7865": "nan",
+ "7870": "nan",
+ "7875": "nan",
+ "7880": "nan",
+ "7885": "nan",
+ "7890": "nan",
+ "7895": "nan",
+ "7900": 1.45149,
+ "7905": "nan",
+ "7910": "nan",
+ "7915": "nan",
+ "7920": "nan",
+ "7925": "nan",
+ "7930": "nan",
+ "7935": "nan",
+ "7940": "nan",
+ "7945": "nan",
+ "7950": "nan",
+ "7955": "nan",
+ "7960": "nan",
+ "7965": "nan",
+ "7970": "nan",
+ "7975": "nan",
+ "7980": "nan",
+ "7985": "nan",
+ "7990": "nan",
+ "7995": "nan",
+ "8000": 1.51543,
+ "8005": "nan",
+ "8010": "nan",
+ "8015": "nan",
+ "8020": "nan",
+ "8025": "nan",
+ "8030": "nan",
+ "8035": "nan",
+ "8040": "nan",
+ "8045": "nan",
+ "8050": "nan",
+ "8055": "nan",
+ "8060": "nan",
+ "8065": "nan",
+ "8070": "nan",
+ "8075": "nan",
+ "8080": "nan",
+ "8085": "nan",
+ "8090": "nan",
+ "8095": "nan",
+ "8100": 1.46342,
+ "8105": "nan",
+ "8110": "nan",
+ "8115": "nan",
+ "8120": "nan",
+ "8125": "nan",
+ "8130": "nan",
+ "8135": "nan",
+ "8140": "nan",
+ "8145": "nan",
+ "8150": "nan",
+ "8155": "nan",
+ "8160": "nan",
+ "8165": "nan",
+ "8170": "nan",
+ "8175": "nan",
+ "8180": "nan",
+ "8185": "nan",
+ "8190": "nan",
+ "8195": "nan",
+ "8200": 1.59038,
+ "8205": "nan",
+ "8210": "nan",
+ "8215": "nan",
+ "8220": "nan",
+ "8225": "nan",
+ "8230": "nan",
+ "8235": "nan",
+ "8240": "nan",
+ "8245": "nan",
+ "8250": "nan",
+ "8255": "nan",
+ "8260": "nan",
+ "8265": "nan",
+ "8270": "nan",
+ "8275": "nan",
+ "8280": "nan",
+ "8285": "nan",
+ "8290": "nan",
+ "8295": "nan",
+ "8300": 1.72439,
+ "8305": "nan",
+ "8310": "nan",
+ "8315": "nan",
+ "8320": "nan",
+ "8325": "nan",
+ "8330": "nan",
+ "8335": "nan",
+ "8340": "nan",
+ "8345": "nan",
+ "8350": "nan",
+ "8355": "nan",
+ "8360": "nan",
+ "8365": "nan",
+ "8370": "nan",
+ "8375": "nan",
+ "8380": "nan",
+ "8385": "nan",
+ "8390": "nan",
+ "8395": "nan",
+ "8400": 1.43549,
+ "8405": "nan",
+ "8410": "nan",
+ "8415": "nan",
+ "8420": "nan",
+ "8425": "nan",
+ "8430": "nan",
+ "8435": "nan",
+ "8440": "nan",
+ "8445": "nan",
+ "8450": "nan",
+ "8455": "nan",
+ "8460": "nan",
+ "8465": "nan",
+ "8470": "nan",
+ "8475": "nan",
+ "8480": "nan",
+ "8485": "nan",
+ "8490": "nan",
+ "8495": "nan",
+ "8500": 1.51344,
+ "8505": "nan",
+ "8510": "nan",
+ "8515": "nan",
+ "8520": "nan",
+ "8525": "nan",
+ "8530": "nan",
+ "8535": "nan",
+ "8540": "nan",
+ "8545": "nan",
+ "8550": "nan",
+ "8555": "nan",
+ "8560": "nan",
+ "8565": "nan",
+ "8570": "nan",
+ "8575": "nan",
+ "8580": "nan",
+ "8585": "nan",
+ "8590": "nan",
+ "8595": "nan",
+ "8600": 1.43389,
+ "8605": "nan",
+ "8610": "nan",
+ "8615": "nan",
+ "8620": "nan",
+ "8625": "nan",
+ "8630": "nan",
+ "8635": "nan",
+ "8640": "nan",
+ "8645": "nan",
+ "8650": "nan",
+ "8655": "nan",
+ "8660": "nan",
+ "8665": "nan",
+ "8670": "nan",
+ "8675": "nan",
+ "8680": "nan",
+ "8685": "nan",
+ "8690": "nan",
+ "8695": "nan",
+ "8700": 1.42937,
+ "8705": "nan",
+ "8710": "nan",
+ "8715": "nan",
+ "8720": "nan",
+ "8725": "nan",
+ "8730": "nan",
+ "8735": "nan",
+ "8740": "nan",
+ "8745": "nan",
+ "8750": "nan",
+ "8755": "nan",
+ "8760": "nan",
+ "8765": "nan",
+ "8770": "nan",
+ "8775": "nan",
+ "8780": "nan",
+ "8785": "nan",
+ "8790": "nan",
+ "8795": "nan",
+ "8800": 1.43364,
+ "8805": "nan",
+ "8810": "nan",
+ "8815": "nan",
+ "8820": "nan",
+ "8825": "nan",
+ "8830": "nan",
+ "8835": "nan",
+ "8840": "nan",
+ "8845": "nan",
+ "8850": "nan",
+ "8855": "nan",
+ "8860": "nan",
+ "8865": "nan",
+ "8870": "nan",
+ "8875": "nan",
+ "8880": "nan",
+ "8885": "nan",
+ "8890": "nan",
+ "8895": "nan",
+ "8900": 1.42823,
+ "8905": "nan",
+ "8910": "nan",
+ "8915": "nan",
+ "8920": "nan",
+ "8925": "nan",
+ "8930": "nan",
+ "8935": "nan",
+ "8940": "nan",
+ "8945": "nan",
+ "8950": "nan",
+ "8955": "nan",
+ "8960": "nan",
+ "8965": "nan",
+ "8970": "nan",
+ "8975": "nan",
+ "8980": "nan",
+ "8985": "nan",
+ "8990": "nan",
+ "8995": "nan",
+ "9000": 1.42555,
+ "9005": "nan",
+ "9010": "nan",
+ "9015": "nan",
+ "9020": "nan",
+ "9025": "nan",
+ "9030": "nan",
+ "9035": "nan",
+ "9040": "nan",
+ "9045": "nan",
+ "9050": "nan",
+ "9055": "nan",
+ "9060": "nan",
+ "9065": "nan",
+ "9070": "nan",
+ "9075": "nan",
+ "9080": "nan",
+ "9085": "nan",
+ "9090": "nan",
+ "9095": "nan",
+ "9100": 1.47893,
+ "9105": "nan",
+ "9110": "nan",
+ "9115": "nan",
+ "9120": "nan",
+ "9125": "nan",
+ "9130": "nan",
+ "9135": "nan",
+ "9140": "nan",
+ "9145": "nan",
+ "9150": "nan",
+ "9155": "nan",
+ "9160": "nan",
+ "9165": "nan",
+ "9170": "nan",
+ "9175": "nan",
+ "9180": "nan",
+ "9185": "nan",
+ "9190": "nan",
+ "9195": "nan",
+ "9200": 1.54458,
+ "9205": "nan",
+ "9210": "nan",
+ "9215": "nan",
+ "9220": "nan",
+ "9225": "nan",
+ "9230": "nan",
+ "9235": "nan",
+ "9240": "nan",
+ "9245": "nan",
+ "9250": "nan",
+ "9255": "nan",
+ "9260": "nan",
+ "9265": "nan",
+ "9270": "nan",
+ "9275": "nan",
+ "9280": "nan",
+ "9285": "nan",
+ "9290": "nan",
+ "9295": "nan",
+ "9300": 1.51369,
+ "9305": "nan",
+ "9310": "nan",
+ "9315": "nan",
+ "9320": "nan",
+ "9325": "nan",
+ "9330": "nan",
+ "9335": "nan",
+ "9340": "nan",
+ "9345": "nan",
+ "9350": "nan",
+ "9355": "nan",
+ "9360": "nan",
+ "9365": "nan",
+ "9370": "nan",
+ "9375": "nan",
+ "9380": "nan",
+ "9385": "nan",
+ "9390": "nan",
+ "9395": "nan",
+ "9400": 1.59273,
+ "9405": "nan",
+ "9410": "nan",
+ "9415": "nan",
+ "9420": "nan",
+ "9425": "nan",
+ "9430": "nan",
+ "9435": "nan",
+ "9440": "nan",
+ "9445": "nan",
+ "9450": "nan",
+ "9455": "nan",
+ "9460": "nan",
+ "9465": "nan",
+ "9470": "nan",
+ "9475": "nan",
+ "9480": "nan",
+ "9485": "nan",
+ "9490": "nan",
+ "9495": "nan",
+ "9500": 1.51633,
+ "9505": "nan",
+ "9510": "nan",
+ "9515": "nan",
+ "9520": "nan",
+ "9525": "nan",
+ "9530": "nan",
+ "9535": "nan",
+ "9540": "nan",
+ "9545": "nan",
+ "9550": "nan",
+ "9555": "nan",
+ "9560": "nan",
+ "9565": "nan",
+ "9570": "nan",
+ "9575": "nan",
+ "9580": "nan",
+ "9585": "nan",
+ "9590": "nan",
+ "9595": "nan",
+ "9600": 1.42217,
+ "9605": "nan",
+ "9610": "nan",
+ "9615": "nan",
+ "9620": "nan",
+ "9625": "nan",
+ "9630": "nan",
+ "9635": "nan",
+ "9640": "nan",
+ "9645": "nan",
+ "9650": "nan",
+ "9655": "nan",
+ "9660": "nan",
+ "9665": "nan",
+ "9670": "nan",
+ "9675": "nan",
+ "9680": "nan",
+ "9685": "nan",
+ "9690": "nan",
+ "9695": "nan",
+ "9700": 1.46834,
+ "9705": "nan",
+ "9710": "nan",
+ "9715": "nan",
+ "9720": "nan",
+ "9725": "nan",
+ "9730": "nan",
+ "9735": "nan",
+ "9740": "nan",
+ "9745": "nan",
+ "9750": "nan",
+ "9755": "nan",
+ "9760": "nan",
+ "9765": "nan",
+ "9770": "nan",
+ "9775": "nan",
+ "9780": "nan",
+ "9785": "nan",
+ "9790": "nan",
+ "9795": "nan",
+ "9800": 2.31936,
+ "9805": "nan",
+ "9810": "nan",
+ "9815": "nan",
+ "9820": "nan",
+ "9825": "nan",
+ "9830": "nan",
+ "9835": "nan",
+ "9840": "nan",
+ "9845": "nan",
+ "9850": "nan",
+ "9855": "nan",
+ "9860": "nan",
+ "9865": "nan",
+ "9870": "nan",
+ "9875": "nan",
+ "9880": "nan",
+ "9885": "nan",
+ "9890": "nan",
+ "9895": "nan",
+ "9900": 1.5164,
+ "9905": "nan",
+ "9910": "nan",
+ "9915": "nan",
+ "9920": "nan",
+ "9925": "nan",
+ "9930": "nan",
+ "9935": "nan",
+ "9940": "nan",
+ "9945": "nan",
+ "9950": "nan",
+ "9955": "nan",
+ "9960": "nan",
+ "9965": "nan",
+ "9970": "nan",
+ "9975": "nan",
+ "9980": "nan",
+ "9985": "nan",
+ "9990": "nan",
+ "9995": "nan",
+ "10000": 1.42494,
+ "10005": "nan",
+ "10010": "nan",
+ "10015": "nan",
+ "10020": "nan",
+ "10025": "nan",
+ "10030": "nan",
+ "10035": "nan",
+ "10040": "nan",
+ "10045": "nan",
+ "10050": "nan",
+ "10055": "nan",
+ "10060": "nan",
+ "10065": "nan",
+ "10070": "nan",
+ "10075": "nan",
+ "10080": "nan",
+ "10085": "nan",
+ "10090": "nan",
+ "10095": "nan",
+ "10100": 1.43004,
+ "10105": "nan",
+ "10110": "nan",
+ "10115": "nan",
+ "10120": "nan",
+ "10125": "nan",
+ "10130": "nan",
+ "10135": "nan",
+ "10140": "nan",
+ "10145": "nan",
+ "10150": "nan",
+ "10155": "nan",
+ "10160": "nan",
+ "10165": "nan",
+ "10170": "nan",
+ "10175": "nan",
+ "10180": "nan",
+ "10185": "nan",
+ "10190": "nan",
+ "10195": "nan",
+ "10200": 1.4274,
+ "10205": "nan",
+ "10210": "nan",
+ "10215": "nan",
+ "10220": "nan",
+ "10225": "nan",
+ "10230": "nan",
+ "10235": "nan",
+ "10240": "nan",
+ "10245": "nan",
+ "10250": "nan",
+ "10255": "nan",
+ "10260": "nan",
+ "10265": "nan",
+ "10270": "nan",
+ "10275": "nan",
+ "10280": "nan",
+ "10285": "nan",
+ "10290": "nan",
+ "10295": "nan",
+ "10300": 1.43038,
+ "10305": "nan",
+ "10310": "nan",
+ "10315": "nan",
+ "10320": "nan",
+ "10325": "nan",
+ "10330": "nan",
+ "10335": "nan",
+ "10340": "nan",
+ "10345": "nan",
+ "10350": "nan",
+ "10355": "nan",
+ "10360": "nan",
+ "10365": "nan",
+ "10370": "nan",
+ "10375": "nan",
+ "10380": "nan",
+ "10385": "nan",
+ "10390": "nan",
+ "10395": "nan",
+ "10400": 1.43012,
+ "10405": "nan",
+ "10410": "nan",
+ "10415": "nan",
+ "10420": "nan",
+ "10425": "nan",
+ "10430": "nan",
+ "10435": "nan",
+ "10440": "nan",
+ "10445": "nan",
+ "10450": "nan",
+ "10455": "nan",
+ "10460": "nan",
+ "10465": "nan",
+ "10470": "nan",
+ "10475": "nan",
+ "10480": "nan",
+ "10485": "nan",
+ "10490": "nan",
+ "10495": "nan",
+ "10500": 1.43136,
+ "10505": "nan",
+ "10510": "nan",
+ "10515": "nan",
+ "10520": "nan",
+ "10525": "nan",
+ "10530": "nan",
+ "10535": "nan",
+ "10540": "nan",
+ "10545": "nan",
+ "10550": "nan",
+ "10555": "nan",
+ "10560": "nan",
+ "10565": "nan",
+ "10570": "nan",
+ "10575": "nan",
+ "10580": "nan",
+ "10585": "nan",
+ "10590": "nan",
+ "10595": "nan",
+ "10600": 1.43207,
+ "10605": "nan",
+ "10610": "nan",
+ "10615": "nan",
+ "10620": "nan",
+ "10625": "nan",
+ "10630": "nan",
+ "10635": "nan",
+ "10640": "nan",
+ "10645": "nan",
+ "10650": "nan",
+ "10655": "nan",
+ "10660": "nan",
+ "10665": "nan",
+ "10670": "nan",
+ "10675": "nan",
+ "10680": "nan",
+ "10685": "nan",
+ "10690": "nan",
+ "10695": "nan",
+ "10700": 1.43277,
+ "10705": "nan",
+ "10710": "nan",
+ "10715": "nan",
+ "10720": "nan",
+ "10725": "nan",
+ "10730": "nan",
+ "10735": "nan",
+ "10740": "nan",
+ "10745": "nan",
+ "10750": "nan",
+ "10755": "nan",
+ "10760": "nan",
+ "10765": "nan",
+ "10770": "nan",
+ "10775": "nan",
+ "10780": "nan",
+ "10785": "nan",
+ "10790": "nan",
+ "10795": "nan",
+ "10800": 1.59276,
+ "10805": "nan",
+ "10810": "nan",
+ "10815": "nan",
+ "10820": "nan",
+ "10825": "nan",
+ "10830": "nan",
+ "10835": "nan",
+ "10840": "nan",
+ "10845": "nan",
+ "10850": "nan",
+ "10855": "nan",
+ "10860": "nan",
+ "10865": "nan",
+ "10870": "nan",
+ "10875": "nan",
+ "10880": "nan",
+ "10885": "nan",
+ "10890": "nan",
+ "10895": "nan",
+ "10900": 1.43482,
+ "10905": "nan",
+ "10910": "nan",
+ "10915": "nan",
+ "10920": "nan",
+ "10925": "nan",
+ "10930": "nan",
+ "10935": "nan",
+ "10940": "nan",
+ "10945": "nan",
+ "10950": "nan",
+ "10955": "nan",
+ "10960": "nan",
+ "10965": "nan",
+ "10970": "nan",
+ "10975": "nan",
+ "10980": "nan",
+ "10985": "nan",
+ "10990": "nan",
+ "10995": "nan",
+ "11000": 1.43307,
+ "11005": "nan",
+ "11010": "nan",
+ "11015": "nan",
+ "11020": "nan",
+ "11025": "nan",
+ "11030": "nan",
+ "11035": "nan",
+ "11040": "nan",
+ "11045": "nan",
+ "11050": "nan",
+ "11055": "nan",
+ "11060": "nan",
+ "11065": "nan",
+ "11070": "nan",
+ "11075": "nan",
+ "11080": "nan",
+ "11085": "nan",
+ "11090": "nan",
+ "11095": "nan",
+ "11100": 1.45438,
+ "11105": "nan",
+ "11110": "nan",
+ "11115": "nan",
+ "11120": "nan",
+ "11125": "nan",
+ "11130": "nan",
+ "11135": "nan",
+ "11140": "nan",
+ "11145": "nan",
+ "11150": "nan",
+ "11155": "nan",
+ "11160": "nan",
+ "11165": "nan",
+ "11170": "nan",
+ "11175": "nan",
+ "11180": "nan",
+ "11185": "nan",
+ "11190": "nan",
+ "11195": "nan",
+ "11200": 1.91248,
+ "11205": "nan",
+ "11210": "nan",
+ "11215": "nan",
+ "11220": "nan",
+ "11225": "nan",
+ "11230": "nan",
+ "11235": "nan",
+ "11240": "nan",
+ "11245": "nan",
+ "11250": "nan",
+ "11255": "nan",
+ "11260": "nan",
+ "11265": "nan",
+ "11270": "nan",
+ "11275": "nan",
+ "11280": "nan",
+ "11285": "nan",
+ "11290": "nan",
+ "11295": "nan",
+ "11300": 1.66415,
+ "11305": "nan",
+ "11310": "nan",
+ "11315": "nan",
+ "11320": "nan",
+ "11325": "nan",
+ "11330": "nan",
+ "11335": "nan",
+ "11340": "nan",
+ "11345": "nan",
+ "11350": "nan",
+ "11355": "nan",
+ "11360": "nan",
+ "11365": "nan",
+ "11370": "nan",
+ "11375": "nan",
+ "11380": "nan",
+ "11385": "nan",
+ "11390": "nan",
+ "11395": "nan",
+ "11400": 1.42797,
+ "11405": "nan",
+ "11410": "nan",
+ "11415": "nan",
+ "11420": "nan",
+ "11425": "nan",
+ "11430": "nan",
+ "11435": "nan",
+ "11440": "nan",
+ "11445": "nan",
+ "11450": "nan",
+ "11455": "nan",
+ "11460": "nan",
+ "11465": "nan",
+ "11470": "nan",
+ "11475": "nan",
+ "11480": "nan",
+ "11485": "nan",
+ "11490": "nan",
+ "11495": "nan",
+ "11500": 1.42875,
+ "11505": "nan",
+ "11510": "nan",
+ "11515": "nan",
+ "11520": "nan",
+ "11525": "nan",
+ "11530": "nan",
+ "11535": "nan",
+ "11540": "nan",
+ "11545": "nan",
+ "11550": "nan",
+ "11555": "nan",
+ "11560": "nan",
+ "11565": "nan",
+ "11570": "nan",
+ "11575": "nan",
+ "11580": "nan",
+ "11585": "nan",
+ "11590": "nan",
+ "11595": "nan",
+ "11600": 1.42955,
+ "11605": "nan",
+ "11610": "nan",
+ "11615": "nan",
+ "11620": "nan",
+ "11625": "nan",
+ "11630": "nan",
+ "11635": "nan",
+ "11640": "nan",
+ "11645": "nan",
+ "11650": "nan",
+ "11655": "nan",
+ "11660": "nan",
+ "11665": "nan",
+ "11670": "nan",
+ "11675": "nan",
+ "11680": "nan",
+ "11685": "nan",
+ "11690": "nan",
+ "11695": "nan",
+ "11700": 1.45905,
+ "11705": "nan",
+ "11710": "nan",
+ "11715": "nan",
+ "11720": "nan",
+ "11725": "nan",
+ "11730": "nan",
+ "11735": "nan",
+ "11740": "nan",
+ "11745": "nan",
+ "11750": "nan",
+ "11755": "nan",
+ "11760": "nan",
+ "11765": "nan",
+ "11770": "nan",
+ "11775": "nan",
+ "11780": "nan",
+ "11785": "nan",
+ "11790": "nan",
+ "11795": "nan",
+ "11800": 1.42993,
+ "11805": "nan",
+ "11810": "nan",
+ "11815": "nan",
+ "11820": "nan",
+ "11825": "nan",
+ "11830": "nan",
+ "11835": "nan",
+ "11840": "nan",
+ "11845": "nan",
+ "11850": "nan",
+ "11855": "nan",
+ "11860": "nan",
+ "11865": "nan",
+ "11870": "nan",
+ "11875": "nan",
+ "11880": "nan",
+ "11885": "nan",
+ "11890": "nan",
+ "11895": "nan",
+ "11900": 1.43196,
+ "11905": "nan",
+ "11910": "nan",
+ "11915": "nan",
+ "11920": "nan",
+ "11925": "nan",
+ "11930": "nan",
+ "11935": "nan",
+ "11940": "nan",
+ "11945": "nan",
+ "11950": "nan",
+ "11955": "nan",
+ "11960": "nan",
+ "11965": "nan",
+ "11970": "nan",
+ "11975": "nan",
+ "11980": "nan",
+ "11985": "nan",
+ "11990": "nan",
+ "11995": "nan",
+ "12000": 1.45532,
+ "12005": "nan",
+ "12010": "nan",
+ "12015": "nan",
+ "12020": "nan",
+ "12025": "nan",
+ "12030": "nan",
+ "12035": "nan",
+ "12040": "nan",
+ "12045": "nan",
+ "12050": "nan",
+ "12055": "nan",
+ "12060": "nan",
+ "12065": "nan",
+ "12070": "nan",
+ "12075": "nan",
+ "12080": "nan",
+ "12085": "nan",
+ "12090": "nan",
+ "12095": "nan",
+ "12100": 1.43662,
+ "12105": "nan",
+ "12110": "nan",
+ "12115": "nan",
+ "12120": "nan",
+ "12125": "nan",
+ "12130": "nan",
+ "12135": "nan",
+ "12140": "nan",
+ "12145": "nan",
+ "12150": "nan",
+ "12155": "nan",
+ "12160": "nan",
+ "12165": "nan",
+ "12170": "nan",
+ "12175": "nan",
+ "12180": "nan",
+ "12185": "nan",
+ "12190": "nan",
+ "12195": "nan",
+ "12200": 1.42854,
+ "12205": "nan",
+ "12210": "nan",
+ "12215": "nan",
+ "12220": "nan",
+ "12225": "nan",
+ "12230": "nan",
+ "12235": "nan",
+ "12240": "nan",
+ "12245": "nan",
+ "12250": "nan",
+ "12255": "nan",
+ "12260": "nan",
+ "12265": "nan",
+ "12270": "nan",
+ "12275": "nan",
+ "12280": "nan",
+ "12285": "nan",
+ "12290": "nan",
+ "12295": "nan",
+ "12300": 1.46417,
+ "12305": "nan",
+ "12310": "nan",
+ "12315": "nan",
+ "12320": "nan",
+ "12325": "nan",
+ "12330": "nan",
+ "12335": "nan",
+ "12340": "nan",
+ "12345": "nan",
+ "12350": "nan",
+ "12355": "nan",
+ "12360": "nan",
+ "12365": "nan",
+ "12370": "nan",
+ "12375": "nan",
+ "12380": "nan",
+ "12385": "nan",
+ "12390": "nan",
+ "12395": "nan",
+ "12400": 1.43423,
+ "12405": "nan",
+ "12410": "nan",
+ "12415": "nan",
+ "12420": "nan",
+ "12425": "nan",
+ "12430": "nan",
+ "12435": "nan",
+ "12440": "nan",
+ "12445": "nan",
+ "12450": "nan",
+ "12455": "nan",
+ "12460": "nan",
+ "12465": "nan",
+ "12470": "nan",
+ "12475": "nan",
+ "12480": "nan",
+ "12485": "nan",
+ "12490": "nan",
+ "12495": "nan",
+ "12500": 1.43958,
+ "12505": "nan",
+ "12510": "nan",
+ "12515": "nan",
+ "12520": "nan",
+ "12525": "nan",
+ "12530": "nan",
+ "12535": "nan",
+ "12540": "nan",
+ "12545": "nan",
+ "12550": "nan",
+ "12555": "nan",
+ "12560": "nan",
+ "12565": "nan",
+ "12570": "nan",
+ "12575": "nan",
+ "12580": "nan",
+ "12585": "nan",
+ "12590": "nan",
+ "12595": "nan",
+ "12600": 1.46497,
+ "12605": "nan",
+ "12610": "nan",
+ "12615": "nan",
+ "12620": "nan",
+ "12625": "nan",
+ "12630": "nan",
+ "12635": "nan",
+ "12640": "nan",
+ "12645": "nan",
+ "12650": "nan",
+ "12655": "nan",
+ "12660": "nan",
+ "12665": "nan",
+ "12670": "nan",
+ "12675": "nan",
+ "12680": "nan",
+ "12685": "nan",
+ "12690": "nan",
+ "12695": "nan",
+ "12700": 1.5955,
+ "12705": "nan",
+ "12710": "nan",
+ "12715": "nan",
+ "12720": "nan",
+ "12725": "nan",
+ "12730": "nan",
+ "12735": "nan",
+ "12740": "nan",
+ "12745": "nan",
+ "12750": "nan",
+ "12755": "nan",
+ "12760": "nan",
+ "12765": "nan",
+ "12770": "nan",
+ "12775": "nan",
+ "12780": "nan",
+ "12785": "nan",
+ "12790": "nan",
+ "12795": "nan",
+ "12800": "nan",
+ "12805": "nan",
+ "12810": "nan",
+ "12815": "nan",
+ "12820": "nan",
+ "12825": "nan",
+ "12830": "nan",
+ "12835": "nan",
+ "12840": "nan",
+ "12845": "nan",
+ "12850": "nan",
+ "12855": "nan",
+ "12860": "nan",
+ "12865": "nan",
+ "12870": "nan",
+ "12875": "nan",
+ "12880": "nan",
+ "12885": "nan",
+ "12890": "nan",
+ "12895": "nan",
+ "12900": "nan",
+ "12905": "nan",
+ "12910": "nan",
+ "12915": "nan",
+ "12920": "nan",
+ "12925": "nan",
+ "12930": "nan",
+ "12935": "nan",
+ "12940": "nan",
+ "12945": "nan",
+ "12950": "nan",
+ "12955": "nan",
+ "12960": "nan",
+ "12965": "nan",
+ "12970": "nan",
+ "12975": "nan",
+ "12980": "nan",
+ "12985": "nan",
+ "12990": "nan",
+ "12995": "nan",
+ "13000": "nan"
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm/model_config.yaml
index 87a4fccb347..3bde7ac0d2d 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm/model_config.yaml
@@ -71,6 +71,7 @@ MODEL_ARGS:
--load: ${CHECKPOINT_LOAD_PATH}
--save: ${CHECKPOINT_SAVE_PATH}
--save-interval: 1000
+ --save-retain-interval: 5000
# Add initialization args
--init-method-std: 0.0134
# Add logging args
@@ -89,6 +90,8 @@ MODEL_ARGS:
--bf16: true
--exit-interval: 13000
--wandb-save-dir: ${WANDB_SAVE_PATH}
+ --async-save: true
+ --use-persistent-ckpt-worker: true
METRICS:
- "iteration-time"
- "lm loss"
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm_gb200/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm_gb200/golden_values_dev_dgx_gb200.json
new file mode 100644
index 00000000000..e05f23ef303
--- /dev/null
+++ b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm_gb200/golden_values_dev_dgx_gb200.json
@@ -0,0 +1,13042 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": 13.00341,
+ "5": 12.95282,
+ "10": 12.10778,
+ "15": 11.54743,
+ "20": 10.38064,
+ "25": 9.96943,
+ "30": 9.61583,
+ "35": 9.35993,
+ "40": 9.18388,
+ "45": 9.04655,
+ "50": 8.88708,
+ "55": 8.73169,
+ "60": 8.64595,
+ "65": 8.56844,
+ "70": 8.47431,
+ "75": 8.29862,
+ "80": 8.15455,
+ "85": 8.13037,
+ "90": 8.03999,
+ "95": 7.94008,
+ "100": 7.81135,
+ "105": 7.68743,
+ "110": 7.59148,
+ "115": 7.48955,
+ "120": 7.46567,
+ "125": 7.41631,
+ "130": 7.2757,
+ "135": 7.23718,
+ "140": 7.1914,
+ "145": 7.04633,
+ "150": 7.18433,
+ "155": 7.03577,
+ "160": 6.94043,
+ "165": 6.93135,
+ "170": 6.81691,
+ "175": 6.84664,
+ "180": 6.8163,
+ "185": 6.7551,
+ "190": 6.69768,
+ "195": 6.63881,
+ "200": 6.65089,
+ "205": 6.61819,
+ "210": 6.53871,
+ "215": 6.50275,
+ "220": 6.51297,
+ "225": 6.49057,
+ "230": 6.528,
+ "235": 6.46104,
+ "240": 6.3991,
+ "245": 6.39504,
+ "250": 6.34376,
+ "255": 6.45426,
+ "260": 6.35163,
+ "265": 6.30593,
+ "270": 6.2449,
+ "275": 6.24094,
+ "280": 6.22016,
+ "285": 6.22853,
+ "290": 6.18891,
+ "295": 6.13549,
+ "300": 6.12886,
+ "305": 6.05304,
+ "310": 6.11723,
+ "315": 6.08704,
+ "320": 5.99557,
+ "325": 5.9427,
+ "330": 5.9916,
+ "335": 6.01094,
+ "340": 5.943,
+ "345": 5.92627,
+ "350": 5.91033,
+ "355": 5.8613,
+ "360": 5.90035,
+ "365": 5.86533,
+ "370": 5.83734,
+ "375": 5.87837,
+ "380": 5.84845,
+ "385": 5.78602,
+ "390": 5.79911,
+ "395": 5.69522,
+ "400": 5.66258,
+ "405": 5.6747,
+ "410": 5.66221,
+ "415": 5.69803,
+ "420": 5.67149,
+ "425": 5.68823,
+ "430": 5.63923,
+ "435": 5.565,
+ "440": 5.61082,
+ "445": 5.52916,
+ "450": 5.60319,
+ "455": 5.51312,
+ "460": 5.4984,
+ "465": 5.59438,
+ "470": 5.56489,
+ "475": 5.48861,
+ "480": 5.48945,
+ "485": 5.50778,
+ "490": 5.48381,
+ "495": 5.46072,
+ "500": 5.40169,
+ "505": 5.35826,
+ "510": 5.45831,
+ "515": 5.42488,
+ "520": 5.44311,
+ "525": 5.28781,
+ "530": 5.32698,
+ "535": 5.32999,
+ "540": 5.3123,
+ "545": 5.37638,
+ "550": 5.35248,
+ "555": 5.21556,
+ "560": 5.32644,
+ "565": 5.27412,
+ "570": 5.27391,
+ "575": 5.30158,
+ "580": 5.2278,
+ "585": 5.19805,
+ "590": 5.20458,
+ "595": 5.22492,
+ "600": 5.26217,
+ "605": 5.22824,
+ "610": 5.21735,
+ "615": 5.16884,
+ "620": 5.17672,
+ "625": 5.19402,
+ "630": 5.1403,
+ "635": 5.13381,
+ "640": 5.08111,
+ "645": 5.11608,
+ "650": 5.12425,
+ "655": 5.11953,
+ "660": 5.0382,
+ "665": 5.08316,
+ "670": 5.02529,
+ "675": 5.02258,
+ "680": 5.01892,
+ "685": 4.97919,
+ "690": 4.99517,
+ "695": 4.96847,
+ "700": 4.95202,
+ "705": 4.94585,
+ "710": 4.95931,
+ "715": 4.86635,
+ "720": 4.85227,
+ "725": 4.81006,
+ "730": 4.85128,
+ "735": 4.8154,
+ "740": 4.83909,
+ "745": 4.69365,
+ "750": 4.74319,
+ "755": 4.77271,
+ "760": 4.76402,
+ "765": 4.73205,
+ "770": 4.67087,
+ "775": 4.65419,
+ "780": 4.67296,
+ "785": 4.76815,
+ "790": 4.65704,
+ "795": 4.66153,
+ "800": 4.61386,
+ "805": 4.59823,
+ "810": 4.62179,
+ "815": 4.57296,
+ "820": 4.60773,
+ "825": 4.57845,
+ "830": 4.56525,
+ "835": 4.5459,
+ "840": 4.45395,
+ "845": 4.46403,
+ "850": 4.43813,
+ "855": 4.49986,
+ "860": 4.43175,
+ "865": 4.4862,
+ "870": 4.45556,
+ "875": 4.34881,
+ "880": 4.40215,
+ "885": 4.37846,
+ "890": 4.41409,
+ "895": 4.4217,
+ "900": 4.37113,
+ "905": 4.32814,
+ "910": 4.35962,
+ "915": 4.3318,
+ "920": 4.37059,
+ "925": 4.37232,
+ "930": 4.2975,
+ "935": 4.28259,
+ "940": 4.34895,
+ "945": 4.29285,
+ "950": 4.35589,
+ "955": 4.2374,
+ "960": 4.17782,
+ "965": 4.28699,
+ "970": 4.2605,
+ "975": 4.22283,
+ "980": 4.21392,
+ "985": 4.15862,
+ "990": 4.14997,
+ "995": 4.17141,
+ "1000": 4.24118,
+ "1005": 4.17774,
+ "1010": 4.1699,
+ "1015": 4.12546,
+ "1020": 4.14741,
+ "1025": 4.21766,
+ "1030": 4.1176,
+ "1035": 4.09191,
+ "1040": 4.09265,
+ "1045": 4.09146,
+ "1050": 4.13107,
+ "1055": 4.08531,
+ "1060": 4.09951,
+ "1065": 4.06018,
+ "1070": 4.05114,
+ "1075": 4.05617,
+ "1080": 4.05555,
+ "1085": 4.08029,
+ "1090": 4.0071,
+ "1095": 4.0907,
+ "1100": 4.05758,
+ "1105": 4.07469,
+ "1110": 4.02177,
+ "1115": 4.01434,
+ "1120": 3.99358,
+ "1125": 3.98306,
+ "1130": 4.05322,
+ "1135": 4.0092,
+ "1140": 3.99072,
+ "1145": 3.92195,
+ "1150": 4.03579,
+ "1155": 3.99701,
+ "1160": 3.97067,
+ "1165": 3.86814,
+ "1170": 3.92222,
+ "1175": 3.92804,
+ "1180": 3.95933,
+ "1185": 3.96391,
+ "1190": 3.92944,
+ "1195": 3.93933,
+ "1200": 3.8774,
+ "1205": 3.85232,
+ "1210": 3.98673,
+ "1215": 3.83918,
+ "1220": 3.84435,
+ "1225": 3.80402,
+ "1230": 3.90757,
+ "1235": 3.88788,
+ "1240": 3.86858,
+ "1245": 3.79653,
+ "1250": 3.8226,
+ "1255": 3.86487,
+ "1260": 3.88944,
+ "1265": 3.8054,
+ "1270": 3.8725,
+ "1275": 3.84069,
+ "1280": 3.8272,
+ "1285": 3.84101,
+ "1290": 3.86734,
+ "1295": 3.84344,
+ "1300": 3.81557,
+ "1305": 3.82235,
+ "1310": 3.81946,
+ "1315": 3.81858,
+ "1320": 3.82101,
+ "1325": 3.71406,
+ "1330": 3.78974,
+ "1335": 3.75815,
+ "1340": 3.74693,
+ "1345": 3.74855,
+ "1350": 3.72971,
+ "1355": 3.76468,
+ "1360": 3.72924,
+ "1365": 3.72618,
+ "1370": 3.745,
+ "1375": 3.74466,
+ "1380": 3.75533,
+ "1385": 3.74749,
+ "1390": 3.66162,
+ "1395": 3.74044,
+ "1400": 3.73268,
+ "1405": 3.66171,
+ "1410": 3.675,
+ "1415": 3.65416,
+ "1420": 3.68871,
+ "1425": 3.7079,
+ "1430": 3.67178,
+ "1435": 3.65934,
+ "1440": 3.64023,
+ "1445": 3.69206,
+ "1450": 3.68166,
+ "1455": 3.66296,
+ "1460": 3.65139,
+ "1465": 3.68083,
+ "1470": 3.62366,
+ "1475": 3.6855,
+ "1480": 3.67027,
+ "1485": 3.65899,
+ "1490": 3.62592,
+ "1495": 3.60214,
+ "1500": 3.64218,
+ "1505": 3.68503,
+ "1510": 3.55738,
+ "1515": 3.60821,
+ "1520": 3.63821,
+ "1525": 3.59664,
+ "1530": 3.58203,
+ "1535": 3.59168,
+ "1540": 3.6284,
+ "1545": 3.60374,
+ "1550": 3.5589,
+ "1555": 3.58528,
+ "1560": 3.60845,
+ "1565": 3.61971,
+ "1570": 3.59263,
+ "1575": 3.56569,
+ "1580": 3.58413,
+ "1585": 3.57017,
+ "1590": 3.47604,
+ "1595": 3.50827,
+ "1600": 3.49282,
+ "1605": 3.53851,
+ "1610": 3.57015,
+ "1615": 3.50241,
+ "1620": 3.52246,
+ "1625": 3.47492,
+ "1630": 3.50009,
+ "1635": 3.54559,
+ "1640": 3.5397,
+ "1645": 3.53653,
+ "1650": 3.52508,
+ "1655": 3.48883,
+ "1660": 3.52548,
+ "1665": 3.48788,
+ "1670": 3.5136,
+ "1675": 3.49709,
+ "1680": 3.46921,
+ "1685": 3.49078,
+ "1690": 3.47354,
+ "1695": 3.48898,
+ "1700": 3.46982,
+ "1705": 3.39023,
+ "1710": 3.50693,
+ "1715": 3.49841,
+ "1720": 3.43695,
+ "1725": 3.42268,
+ "1730": 3.41636,
+ "1735": 3.45692,
+ "1740": 3.46242,
+ "1745": 3.45952,
+ "1750": 3.42376,
+ "1755": 3.42622,
+ "1760": 3.39111,
+ "1765": 3.43137,
+ "1770": 3.4422,
+ "1775": 3.38271,
+ "1780": 3.41979,
+ "1785": 3.41667,
+ "1790": 3.38863,
+ "1795": 3.40825,
+ "1800": 3.34462,
+ "1805": 3.39529,
+ "1810": 3.32295,
+ "1815": 3.42738,
+ "1820": 3.41333,
+ "1825": 3.375,
+ "1830": 3.33549,
+ "1835": 3.43637,
+ "1840": 3.40535,
+ "1845": 3.42467,
+ "1850": 3.38812,
+ "1855": 3.36965,
+ "1860": 3.33865,
+ "1865": 3.38199,
+ "1870": 3.30933,
+ "1875": 3.43508,
+ "1880": 3.3395,
+ "1885": 3.35459,
+ "1890": 3.33612,
+ "1895": 3.39941,
+ "1900": 3.37561,
+ "1905": 3.30827,
+ "1910": 3.32745,
+ "1915": 3.30559,
+ "1920": 3.35831,
+ "1925": 3.33716,
+ "1930": 3.32272,
+ "1935": 3.32229,
+ "1940": 3.38342,
+ "1945": 3.27978,
+ "1950": 3.40609,
+ "1955": 3.28996,
+ "1960": 3.28625,
+ "1965": 3.27158,
+ "1970": 3.29364,
+ "1975": 3.3382,
+ "1980": 3.33403,
+ "1985": 3.24406,
+ "1990": 3.30765,
+ "1995": 3.27854,
+ "2000": 3.27484,
+ "2005": 3.26184,
+ "2010": 3.26534,
+ "2015": 3.21842,
+ "2020": 3.27379,
+ "2025": 3.26704,
+ "2030": 3.28585,
+ "2035": 3.30676,
+ "2040": 3.25351,
+ "2045": 3.24455,
+ "2050": 3.27847,
+ "2055": 3.32351,
+ "2060": 3.28264,
+ "2065": 3.24592,
+ "2070": 3.28814,
+ "2075": 3.23618,
+ "2080": 3.23069,
+ "2085": 3.27266,
+ "2090": 3.14481,
+ "2095": 3.27316,
+ "2100": 3.239,
+ "2105": 3.19514,
+ "2110": 3.21434,
+ "2115": 3.22623,
+ "2120": 3.17749,
+ "2125": 3.20847,
+ "2130": 3.21248,
+ "2135": 3.28745,
+ "2140": 3.20611,
+ "2145": 3.19872,
+ "2150": 3.21887,
+ "2155": 3.23068,
+ "2160": 3.18815,
+ "2165": 3.22776,
+ "2170": 3.21354,
+ "2175": 3.16799,
+ "2180": 3.23529,
+ "2185": 3.24865,
+ "2190": 3.23197,
+ "2195": 3.16199,
+ "2200": 3.19947,
+ "2205": 3.16674,
+ "2210": 3.12196,
+ "2215": 3.1938,
+ "2220": 3.19794,
+ "2225": 3.18644,
+ "2230": 3.12773,
+ "2235": 3.16594,
+ "2240": 3.20496,
+ "2245": 3.17049,
+ "2250": 3.19927,
+ "2255": 3.12229,
+ "2260": 3.13331,
+ "2265": 3.20933,
+ "2270": 3.18357,
+ "2275": 3.13915,
+ "2280": 3.17223,
+ "2285": 3.16447,
+ "2290": 3.16657,
+ "2295": 3.1927,
+ "2300": 3.13208,
+ "2305": 3.18964,
+ "2310": 3.13932,
+ "2315": 3.07633,
+ "2320": 3.11821,
+ "2325": 3.16623,
+ "2330": 3.12536,
+ "2335": 3.12162,
+ "2340": 3.14297,
+ "2345": 3.10963,
+ "2350": 3.11784,
+ "2355": 3.10625,
+ "2360": 3.16104,
+ "2365": 3.10584,
+ "2370": 3.14549,
+ "2375": 3.17332,
+ "2380": 3.1178,
+ "2385": 3.10236,
+ "2390": 3.08947,
+ "2395": 3.08137,
+ "2400": 3.08073,
+ "2405": 3.08587,
+ "2410": 3.08314,
+ "2415": 3.0791,
+ "2420": 3.07113,
+ "2425": 3.07964,
+ "2430": 3.07021,
+ "2435": 3.07029,
+ "2440": 3.08337,
+ "2445": 3.04968,
+ "2450": 3.12731,
+ "2455": 3.15086,
+ "2460": 3.07874,
+ "2465": 3.08421,
+ "2470": 3.06043,
+ "2475": 3.06934,
+ "2480": 3.09621,
+ "2485": 3.04953,
+ "2490": 3.04445,
+ "2495": 3.05866,
+ "2500": 3.05497,
+ "2505": 3.08808,
+ "2510": 3.15669,
+ "2515": 3.0582,
+ "2520": 3.10236,
+ "2525": 3.021,
+ "2530": 3.03681,
+ "2535": 3.08312,
+ "2540": 3.06587,
+ "2545": 3.05372,
+ "2550": 2.99619,
+ "2555": 3.06456,
+ "2560": 3.03578,
+ "2565": 3.09463,
+ "2570": 3.00563,
+ "2575": 3.04436,
+ "2580": 3.084,
+ "2585": 3.01523,
+ "2590": 3.06732,
+ "2595": 3.00023,
+ "2600": 3.08861,
+ "2605": 3.14402,
+ "2610": 3.08533,
+ "2615": 3.07656,
+ "2620": 3.00603,
+ "2625": 3.01343,
+ "2630": 3.03246,
+ "2635": 3.0523,
+ "2640": 3.00846,
+ "2645": 3.05301,
+ "2650": 3.01792,
+ "2655": 2.98999,
+ "2660": 3.01495,
+ "2665": 3.04128,
+ "2670": 2.9958,
+ "2675": 2.9649,
+ "2680": 2.99141,
+ "2685": 2.99883,
+ "2690": 2.99734,
+ "2695": 2.98598,
+ "2700": 3.03046,
+ "2705": 2.9834,
+ "2710": 2.96809,
+ "2715": 2.96337,
+ "2720": 3.0264,
+ "2725": 2.98626,
+ "2730": 3.03826,
+ "2735": 3.02965,
+ "2740": 2.99202,
+ "2745": 3.02839,
+ "2750": 3.00926,
+ "2755": 2.97903,
+ "2760": 2.99403,
+ "2765": 3.00181,
+ "2770": 2.97472,
+ "2775": 2.9923,
+ "2780": 3.0061,
+ "2785": 2.95659,
+ "2790": 2.95657,
+ "2795": 2.94303,
+ "2800": 2.95448,
+ "2805": 2.93332,
+ "2810": 2.98427,
+ "2815": 2.96695,
+ "2820": 3.00231,
+ "2825": 3.0147,
+ "2830": 2.99007,
+ "2835": 2.90983,
+ "2840": 2.92834,
+ "2845": 2.99026,
+ "2850": 2.97232,
+ "2855": 2.96179,
+ "2860": 2.94476,
+ "2865": 2.91085,
+ "2870": 2.9888,
+ "2875": 2.91323,
+ "2880": 2.94565,
+ "2885": 2.91637,
+ "2890": 2.97445,
+ "2895": 2.92644,
+ "2900": 2.94865,
+ "2905": 3.03353,
+ "2910": 2.91723,
+ "2915": 2.93144,
+ "2920": 2.94797,
+ "2925": 2.93537,
+ "2930": 2.95144,
+ "2935": 2.9369,
+ "2940": 2.96325,
+ "2945": 2.91214,
+ "2950": 2.98922,
+ "2955": 2.90926,
+ "2960": 2.96584,
+ "2965": 2.88683,
+ "2970": 2.95718,
+ "2975": 2.98277,
+ "2980": 2.9367,
+ "2985": 3.03343,
+ "2990": 2.93541,
+ "2995": 2.86261,
+ "3000": 2.92534,
+ "3005": 2.88407,
+ "3010": 2.93414,
+ "3015": 2.92309,
+ "3020": 2.977,
+ "3025": 2.95352,
+ "3030": 2.93532,
+ "3035": 2.95868,
+ "3040": 2.91521,
+ "3045": 2.83891,
+ "3050": 2.8977,
+ "3055": 2.88904,
+ "3060": 2.92244,
+ "3065": 2.91471,
+ "3070": 2.91449,
+ "3075": 2.88932,
+ "3080": 2.92122,
+ "3085": 2.89336,
+ "3090": 2.92476,
+ "3095": 2.92179,
+ "3100": 2.86189,
+ "3105": 2.92443,
+ "3110": 2.89386,
+ "3115": 2.93361,
+ "3120": 2.95895,
+ "3125": 2.85462,
+ "3130": 2.92521,
+ "3135": 2.93995,
+ "3140": 2.87634,
+ "3145": 2.91792,
+ "3150": 2.85868,
+ "3155": 2.84568,
+ "3160": 2.84763,
+ "3165": 2.84155,
+ "3170": 2.88845,
+ "3175": 2.91094,
+ "3180": 2.8661,
+ "3185": 2.89866,
+ "3190": 2.91812,
+ "3195": 2.92615,
+ "3200": 2.95465,
+ "3205": 2.86572,
+ "3210": 2.87823,
+ "3215": 2.91074,
+ "3220": 2.86578,
+ "3225": 2.86915,
+ "3230": 2.80969,
+ "3235": 2.86948,
+ "3240": 2.86986,
+ "3245": 2.89771,
+ "3250": 2.85186,
+ "3255": 2.84589,
+ "3260": 2.85824,
+ "3265": 2.87471,
+ "3270": 2.84718,
+ "3275": 2.86791,
+ "3280": 2.79556,
+ "3285": 2.8117,
+ "3290": 2.86902,
+ "3295": 2.89815,
+ "3300": 2.87745,
+ "3305": 2.86306,
+ "3310": 2.85809,
+ "3315": 2.81091,
+ "3320": 2.8372,
+ "3325": 2.84637,
+ "3330": 2.83284,
+ "3335": 2.84257,
+ "3340": 2.82204,
+ "3345": 2.84111,
+ "3350": 2.84457,
+ "3355": 2.85755,
+ "3360": 2.79436,
+ "3365": 2.8589,
+ "3370": 2.85077,
+ "3375": 2.83647,
+ "3380": 2.84805,
+ "3385": 2.88398,
+ "3390": 2.87033,
+ "3395": 2.80786,
+ "3400": 2.78177,
+ "3405": 2.83488,
+ "3410": 2.85095,
+ "3415": 2.86165,
+ "3420": 2.82119,
+ "3425": 2.8102,
+ "3430": 2.8363,
+ "3435": 2.89596,
+ "3440": 2.8135,
+ "3445": 2.86467,
+ "3450": 2.81786,
+ "3455": 2.7874,
+ "3460": 2.81131,
+ "3465": 2.84613,
+ "3470": 2.83749,
+ "3475": 2.77325,
+ "3480": 2.84036,
+ "3485": 2.82948,
+ "3490": 2.89206,
+ "3495": 2.85003,
+ "3500": 2.83297,
+ "3505": 2.82069,
+ "3510": 2.81196,
+ "3515": 2.83997,
+ "3520": 2.77942,
+ "3525": 2.80249,
+ "3530": 2.85547,
+ "3535": 2.78481,
+ "3540": 2.83765,
+ "3545": 2.80967,
+ "3550": 2.79672,
+ "3555": 2.82248,
+ "3560": 2.82367,
+ "3565": 2.8292,
+ "3570": 2.80511,
+ "3575": 2.80236,
+ "3580": 2.81895,
+ "3585": 2.83346,
+ "3590": 2.83136,
+ "3595": 2.77898,
+ "3600": 2.74846,
+ "3605": 2.79314,
+ "3610": 2.84596,
+ "3615": 2.7513,
+ "3620": 2.80426,
+ "3625": 2.88432,
+ "3630": 2.77648,
+ "3635": 2.78896,
+ "3640": 2.78215,
+ "3645": 2.7715,
+ "3650": 2.80647,
+ "3655": 2.82016,
+ "3660": 2.7692,
+ "3665": 2.78341,
+ "3670": 2.76989,
+ "3675": 2.77234,
+ "3680": 2.8071,
+ "3685": 2.80012,
+ "3690": 2.80812,
+ "3695": 2.81329,
+ "3700": 2.79421,
+ "3705": 2.78765,
+ "3710": 2.7547,
+ "3715": 2.80455,
+ "3720": 2.79133,
+ "3725": 2.79596,
+ "3730": 2.83694,
+ "3735": 2.79754,
+ "3740": 2.75406,
+ "3745": 2.79016,
+ "3750": 2.80417,
+ "3755": 2.7959,
+ "3760": 2.75894,
+ "3765": 2.75271,
+ "3770": 2.76679,
+ "3775": 2.77229,
+ "3780": 2.75917,
+ "3785": 2.7853,
+ "3790": 2.74166,
+ "3795": 2.79163,
+ "3800": 2.8017,
+ "3805": 2.74854,
+ "3810": 2.80189,
+ "3815": 2.76902,
+ "3820": 2.79012,
+ "3825": 2.73474,
+ "3830": 2.74634,
+ "3835": 2.81817,
+ "3840": 2.72688,
+ "3845": 2.72236,
+ "3850": 2.77266,
+ "3855": 2.72027,
+ "3860": 2.80478,
+ "3865": 2.75292,
+ "3870": 2.77339,
+ "3875": 2.75733,
+ "3880": 2.78558,
+ "3885": 2.78699,
+ "3890": 2.74303,
+ "3895": 2.7996,
+ "3900": 2.7629,
+ "3905": 2.72091,
+ "3910": 2.74733,
+ "3915": 2.758,
+ "3920": 2.7967,
+ "3925": 2.77615,
+ "3930": 2.71012,
+ "3935": 2.73961,
+ "3940": 2.75239,
+ "3945": 2.74262,
+ "3950": 2.73683,
+ "3955": 2.78055,
+ "3960": 2.76323,
+ "3965": 2.74184,
+ "3970": 2.75793,
+ "3975": 2.73131,
+ "3980": 2.73729,
+ "3985": 2.74485,
+ "3990": 2.69287,
+ "3995": 2.7834,
+ "4000": 2.73648,
+ "4005": 2.76984,
+ "4010": 2.70933,
+ "4015": 2.72406,
+ "4020": 2.75112,
+ "4025": 2.73253,
+ "4030": 2.66097,
+ "4035": 2.69403,
+ "4040": 2.75338,
+ "4045": 2.74949,
+ "4050": 2.79346,
+ "4055": 2.72174,
+ "4060": 2.7159,
+ "4065": 2.65176,
+ "4070": 2.81153,
+ "4075": 2.75849,
+ "4080": 2.71956,
+ "4085": 2.75067,
+ "4090": 2.67987,
+ "4095": 2.69015,
+ "4100": 2.71058,
+ "4105": 2.73706,
+ "4110": 2.72798,
+ "4115": 2.70336,
+ "4120": 2.72875,
+ "4125": 2.7011,
+ "4130": 2.69669,
+ "4135": 2.68835,
+ "4140": 2.68559,
+ "4145": 2.78223,
+ "4150": 2.71009,
+ "4155": 2.73996,
+ "4160": 2.76212,
+ "4165": 2.72153,
+ "4170": 2.67355,
+ "4175": 2.71994,
+ "4180": 2.727,
+ "4185": 2.72905,
+ "4190": 2.73546,
+ "4195": 2.69401,
+ "4200": 2.70509,
+ "4205": 2.74168,
+ "4210": 2.67815,
+ "4215": 2.66585,
+ "4220": 2.65921,
+ "4225": 2.70225,
+ "4230": 2.72856,
+ "4235": 2.73236,
+ "4240": 2.70478,
+ "4245": 2.69813,
+ "4250": 2.70927,
+ "4255": 2.65015,
+ "4260": 2.72399,
+ "4265": 2.73256,
+ "4270": 2.72096,
+ "4275": 2.68966,
+ "4280": 2.70256,
+ "4285": 2.7317,
+ "4290": 2.68551,
+ "4295": 2.69037,
+ "4300": 2.69925,
+ "4305": 2.69945,
+ "4310": 2.72829,
+ "4315": 2.71156,
+ "4320": 2.69748,
+ "4325": 2.7069,
+ "4330": 2.70852,
+ "4335": 2.69007,
+ "4340": 2.69885,
+ "4345": 2.72196,
+ "4350": 2.67329,
+ "4355": 2.6918,
+ "4360": 2.71448,
+ "4365": 2.78418,
+ "4370": 2.73535,
+ "4375": 2.74071,
+ "4380": 2.70083,
+ "4385": 2.69955,
+ "4390": 2.7011,
+ "4395": 2.75061,
+ "4400": 2.66448,
+ "4405": 2.66584,
+ "4410": 2.68248,
+ "4415": 2.70539,
+ "4420": 2.70577,
+ "4425": 2.72028,
+ "4430": 2.69184,
+ "4435": 2.68123,
+ "4440": 2.69505,
+ "4445": 2.67984,
+ "4450": 2.65395,
+ "4455": 2.66652,
+ "4460": 2.68943,
+ "4465": 2.69819,
+ "4470": 2.67145,
+ "4475": 2.68587,
+ "4480": 2.65553,
+ "4485": 2.69963,
+ "4490": 2.65294,
+ "4495": 2.71037,
+ "4500": 2.70324,
+ "4505": 2.69683,
+ "4510": 2.64896,
+ "4515": 2.69797,
+ "4520": 2.66783,
+ "4525": 2.67012,
+ "4530": 2.67298,
+ "4535": 2.66959,
+ "4540": 2.70704,
+ "4545": 2.65362,
+ "4550": 2.70111,
+ "4555": 2.68032,
+ "4560": 2.65997,
+ "4565": 2.6396,
+ "4570": 2.63886,
+ "4575": 2.66714,
+ "4580": 2.68937,
+ "4585": 2.68084,
+ "4590": 2.61504,
+ "4595": 2.66219,
+ "4600": 2.67647,
+ "4605": 2.68095,
+ "4610": 2.66659,
+ "4615": 2.66484,
+ "4620": 2.65798,
+ "4625": 2.71498,
+ "4630": 2.67833,
+ "4635": 2.64579,
+ "4640": 2.69257,
+ "4645": 2.64881,
+ "4650": 2.69874,
+ "4655": 2.70611,
+ "4660": 2.67285,
+ "4665": 2.68494,
+ "4670": 2.67239,
+ "4675": 2.68509,
+ "4680": 2.66516,
+ "4685": 2.6554,
+ "4690": 2.70637,
+ "4695": 2.65837,
+ "4700": 2.67207,
+ "4705": 2.6526,
+ "4710": 2.67753,
+ "4715": 2.65051,
+ "4720": 2.72219,
+ "4725": 2.62879,
+ "4730": 2.65138,
+ "4735": 2.6864,
+ "4740": 2.64033,
+ "4745": 2.64943,
+ "4750": 2.63903,
+ "4755": 2.6536,
+ "4760": 2.66257,
+ "4765": 2.64101,
+ "4770": 2.62193,
+ "4775": 2.65287,
+ "4780": 2.6554,
+ "4785": 2.69084,
+ "4790": 2.64804,
+ "4795": 2.66928,
+ "4800": 2.62567,
+ "4805": 2.63985,
+ "4810": 2.661,
+ "4815": 2.64809,
+ "4820": 2.67082,
+ "4825": 2.65124,
+ "4830": 2.61626,
+ "4835": 2.64733,
+ "4840": 2.65426,
+ "4845": 2.63739,
+ "4850": 2.62545,
+ "4855": 2.60194,
+ "4860": 2.64972,
+ "4865": 2.62682,
+ "4870": 2.63799,
+ "4875": 2.61811,
+ "4880": 2.62652,
+ "4885": 2.62543,
+ "4890": 2.68013,
+ "4895": 2.66042,
+ "4900": 2.61535,
+ "4905": 2.61912,
+ "4910": 2.63653,
+ "4915": 2.61524,
+ "4920": 2.65467,
+ "4925": 2.64999,
+ "4930": 2.56915,
+ "4935": 2.64924,
+ "4940": 2.63292,
+ "4945": 2.63792,
+ "4950": 2.62476,
+ "4955": 2.61732,
+ "4960": 2.6167,
+ "4965": 2.66171,
+ "4970": 2.59848,
+ "4975": 2.65287,
+ "4980": 2.6187,
+ "4985": 2.63335,
+ "4990": 2.6586,
+ "4995": 2.57974,
+ "5000": 2.65898,
+ "5005": 2.66578,
+ "5010": 2.68301,
+ "5015": 2.63322,
+ "5020": 2.63924,
+ "5025": 2.6866,
+ "5030": 2.64456,
+ "5035": 2.61559,
+ "5040": 2.62024,
+ "5045": 2.60287,
+ "5050": 2.6231,
+ "5055": 2.64947,
+ "5060": 2.6452,
+ "5065": 2.6881,
+ "5070": 2.60595,
+ "5075": 2.61447,
+ "5080": 2.60665,
+ "5085": 2.60357,
+ "5090": 2.59045,
+ "5095": 2.64908,
+ "5100": 2.64606,
+ "5105": 2.61042,
+ "5110": 2.66253,
+ "5115": 2.61674,
+ "5120": 2.6721,
+ "5125": 2.62882,
+ "5130": 2.6133,
+ "5135": 2.61257,
+ "5140": 2.57323,
+ "5145": 2.62739,
+ "5150": 2.63655,
+ "5155": 2.61744,
+ "5160": 2.6636,
+ "5165": 2.58271,
+ "5170": 2.58972,
+ "5175": 2.61655,
+ "5180": 2.60404,
+ "5185": 2.61939,
+ "5190": 2.62404,
+ "5195": 2.66753,
+ "5200": 2.5989,
+ "5205": 2.60446,
+ "5210": 2.60475,
+ "5215": 2.64605,
+ "5220": 2.58627,
+ "5225": 2.5519,
+ "5230": 2.6321,
+ "5235": 2.61744,
+ "5240": 2.64291,
+ "5245": 2.63621,
+ "5250": 2.60218,
+ "5255": 2.6186,
+ "5260": 2.55727,
+ "5265": 2.59614,
+ "5270": 2.58806,
+ "5275": 2.61658,
+ "5280": 2.60897,
+ "5285": 2.60361,
+ "5290": 2.63324,
+ "5295": 2.62086,
+ "5300": 2.57816,
+ "5305": 2.59805,
+ "5310": 2.61289,
+ "5315": 2.5858,
+ "5320": 2.61592,
+ "5325": 2.6441,
+ "5330": 2.60238,
+ "5335": 2.58083,
+ "5340": 2.56386,
+ "5345": 2.65745,
+ "5350": 2.62017,
+ "5355": 2.5774,
+ "5360": 2.59964,
+ "5365": 2.62005,
+ "5370": 2.61244,
+ "5375": 2.62598,
+ "5380": 2.57586,
+ "5385": 2.56169,
+ "5390": 2.58533,
+ "5395": 2.61918,
+ "5400": 2.60517,
+ "5405": 2.54618,
+ "5410": 2.60967,
+ "5415": 2.595,
+ "5420": 2.61096,
+ "5425": 2.62258,
+ "5430": 2.62722,
+ "5435": 2.5764,
+ "5440": 2.58535,
+ "5445": 2.62872,
+ "5450": 2.64623,
+ "5455": 2.609,
+ "5460": 2.59057,
+ "5465": 2.60606,
+ "5470": 2.59632,
+ "5475": 2.62477,
+ "5480": 2.58787,
+ "5485": 2.58859,
+ "5490": 2.57747,
+ "5495": 2.57015,
+ "5500": 2.56782,
+ "5505": 2.61543,
+ "5510": 2.62353,
+ "5515": 2.58253,
+ "5520": 2.55721,
+ "5525": 2.58387,
+ "5530": 2.66211,
+ "5535": 2.62005,
+ "5540": 2.57052,
+ "5545": 2.59608,
+ "5550": 2.54903,
+ "5555": 2.571,
+ "5560": 2.56272,
+ "5565": 2.60562,
+ "5570": 2.65097,
+ "5575": 2.62977,
+ "5580": 2.57224,
+ "5585": 2.59618,
+ "5590": 2.56443,
+ "5595": 2.58267,
+ "5600": 2.55357,
+ "5605": 2.59978,
+ "5610": 2.58046,
+ "5615": 2.58067,
+ "5620": 2.58075,
+ "5625": 2.55005,
+ "5630": 2.56908,
+ "5635": 2.63034,
+ "5640": 2.59338,
+ "5645": 2.56893,
+ "5650": 2.57608,
+ "5655": 2.54821,
+ "5660": 2.55792,
+ "5665": 2.58591,
+ "5670": 2.56547,
+ "5675": 2.60454,
+ "5680": 2.5262,
+ "5685": 2.56681,
+ "5690": 2.60176,
+ "5695": 2.55937,
+ "5700": 2.59684,
+ "5705": 2.59636,
+ "5710": 2.57638,
+ "5715": 2.58318,
+ "5720": 2.53262,
+ "5725": 2.60064,
+ "5730": 2.57304,
+ "5735": 2.60717,
+ "5740": 2.59194,
+ "5745": 2.55793,
+ "5750": 2.53836,
+ "5755": 2.5567,
+ "5760": 2.61443,
+ "5765": 2.55588,
+ "5770": 2.53851,
+ "5775": 2.58782,
+ "5780": 2.57719,
+ "5785": 2.53776,
+ "5790": 2.56308,
+ "5795": 2.60024,
+ "5800": 2.54175,
+ "5805": 2.53318,
+ "5810": 2.55626,
+ "5815": 2.5244,
+ "5820": 2.59728,
+ "5825": 2.50527,
+ "5830": 2.49522,
+ "5835": 2.59468,
+ "5840": 2.53947,
+ "5845": 2.55127,
+ "5850": 2.61142,
+ "5855": 2.50793,
+ "5860": 2.55927,
+ "5865": 2.51711,
+ "5870": 2.57272,
+ "5875": 2.60591,
+ "5880": 2.58271,
+ "5885": 2.56594,
+ "5890": 2.58158,
+ "5895": 2.55213,
+ "5900": 2.61216,
+ "5905": 2.55585,
+ "5910": 2.59736,
+ "5915": 2.61237,
+ "5920": 2.58654,
+ "5925": 2.54725,
+ "5930": 2.62155,
+ "5935": 2.58179,
+ "5940": 2.58196,
+ "5945": 2.52214,
+ "5950": 2.55771,
+ "5955": 2.58453,
+ "5960": 2.56145,
+ "5965": 2.61505,
+ "5970": 2.54895,
+ "5975": 2.57663,
+ "5980": 2.55629,
+ "5985": 2.55842,
+ "5990": 2.55443,
+ "5995": 2.55641,
+ "6000": 2.55159,
+ "6005": 2.52054,
+ "6010": 2.55823,
+ "6015": 2.52099,
+ "6020": 2.53292,
+ "6025": 2.55713,
+ "6030": 2.60153,
+ "6035": 2.54241,
+ "6040": 2.54726,
+ "6045": 2.48874,
+ "6050": 2.59262,
+ "6055": 2.51567,
+ "6060": 2.54341,
+ "6065": 2.52354,
+ "6070": 2.52746,
+ "6075": 2.53647,
+ "6080": 2.53381,
+ "6085": 2.59542,
+ "6090": 2.56894,
+ "6095": 2.53307,
+ "6100": 2.54002,
+ "6105": 2.52343,
+ "6110": 2.5534,
+ "6115": 2.58294,
+ "6120": 2.55434,
+ "6125": 2.53926,
+ "6130": 2.47204,
+ "6135": 2.55321,
+ "6140": 2.5535,
+ "6145": 2.55533,
+ "6150": 2.52311,
+ "6155": 2.50907,
+ "6160": 2.53915,
+ "6165": 2.57049,
+ "6170": 2.54242,
+ "6175": 2.59879,
+ "6180": 2.50969,
+ "6185": 2.54989,
+ "6190": 2.49232,
+ "6195": 2.57901,
+ "6200": 2.55006,
+ "6205": 2.53567,
+ "6210": 2.51847,
+ "6215": 2.51389,
+ "6220": 2.56254,
+ "6225": 2.51182,
+ "6230": 2.50777,
+ "6235": 2.55912,
+ "6240": 2.54898,
+ "6245": 2.52163,
+ "6250": 2.53375,
+ "6255": 2.5742,
+ "6260": 2.52313,
+ "6265": 2.57224,
+ "6270": 2.52333,
+ "6275": 2.5617,
+ "6280": 2.52013,
+ "6285": 2.51858,
+ "6290": 2.51624,
+ "6295": 2.50566,
+ "6300": 2.55453,
+ "6305": 2.52441,
+ "6310": 2.51112,
+ "6315": 2.53522,
+ "6320": 2.48781,
+ "6325": 2.5957,
+ "6330": 2.55401,
+ "6335": 2.50915,
+ "6340": 2.50909,
+ "6345": 2.55416,
+ "6350": 2.55378,
+ "6355": 2.52109,
+ "6360": 2.51957,
+ "6365": 2.48297,
+ "6370": 2.53494,
+ "6375": 2.49306,
+ "6380": 2.55558,
+ "6385": 2.57647,
+ "6390": 2.50421,
+ "6395": 2.54995,
+ "6400": 2.50619,
+ "6405": 2.52515,
+ "6410": 2.5145,
+ "6415": 2.523,
+ "6420": 2.5404,
+ "6425": 2.53299,
+ "6430": 2.57518,
+ "6435": 2.54369,
+ "6440": 2.53393,
+ "6445": 2.52557,
+ "6450": 2.53037,
+ "6455": 2.51956,
+ "6460": 2.51424,
+ "6465": 2.55742,
+ "6470": 2.51587,
+ "6475": 2.52288,
+ "6480": 2.48485,
+ "6485": 2.52568,
+ "6490": 2.50531,
+ "6495": 2.49851,
+ "6500": 2.52156,
+ "6505": 2.49322,
+ "6510": 2.54091,
+ "6515": 2.50769,
+ "6520": 2.50847,
+ "6525": 2.49126,
+ "6530": 2.54068,
+ "6535": 2.53196,
+ "6540": 2.5298,
+ "6545": 2.55935,
+ "6550": 2.50141,
+ "6555": 2.55393,
+ "6560": 2.50851,
+ "6565": 2.51974,
+ "6570": 2.58456,
+ "6575": 2.5206,
+ "6580": 2.49613,
+ "6585": 2.50468,
+ "6590": 2.50784,
+ "6595": 2.49551,
+ "6600": 2.48492,
+ "6605": 2.53761,
+ "6610": 2.47813,
+ "6615": 2.56643,
+ "6620": 2.53331,
+ "6625": 2.50918,
+ "6630": 2.50884,
+ "6635": 2.46955,
+ "6640": 2.53625,
+ "6645": 2.59366,
+ "6650": 2.50727,
+ "6655": 2.49491,
+ "6660": 2.57096,
+ "6665": 2.51861,
+ "6670": 2.5669,
+ "6675": 2.46451,
+ "6680": 2.54552,
+ "6685": 2.53303,
+ "6690": 2.51066,
+ "6695": 2.48338,
+ "6700": 2.5201,
+ "6705": 2.51556,
+ "6710": 2.48936,
+ "6715": 2.51426,
+ "6720": 2.50704,
+ "6725": 2.51943,
+ "6730": 2.51595,
+ "6735": 2.47971,
+ "6740": 2.51256,
+ "6745": 2.49237,
+ "6750": 2.55432,
+ "6755": 2.47171,
+ "6760": 2.5398,
+ "6765": 2.48511,
+ "6770": 2.51673,
+ "6775": 2.50938,
+ "6780": 2.53691,
+ "6785": 2.46942,
+ "6790": 2.5404,
+ "6795": 2.49747,
+ "6800": 2.526,
+ "6805": 2.50951,
+ "6810": 2.50246,
+ "6815": 2.51992,
+ "6820": 2.48493,
+ "6825": 2.50145,
+ "6830": 2.53699,
+ "6835": 2.50583,
+ "6840": 2.5074,
+ "6845": 2.52166,
+ "6850": 2.47062,
+ "6855": 2.50963,
+ "6860": 2.50181,
+ "6865": 2.48488,
+ "6870": 2.54928,
+ "6875": 2.47554,
+ "6880": 2.55063,
+ "6885": 2.47562,
+ "6890": 2.54406,
+ "6895": 2.49869,
+ "6900": 2.48472,
+ "6905": 2.49732,
+ "6910": 2.51658,
+ "6915": 2.51538,
+ "6920": 2.52945,
+ "6925": 2.53592,
+ "6930": 2.48743,
+ "6935": 2.51601,
+ "6940": 2.49733,
+ "6945": 2.45735,
+ "6950": 2.48159,
+ "6955": 2.52529,
+ "6960": 2.51942,
+ "6965": 2.49038,
+ "6970": 2.46776,
+ "6975": 2.51893,
+ "6980": 2.45119,
+ "6985": 2.51435,
+ "6990": 2.52681,
+ "6995": 2.46008,
+ "7000": 2.4857,
+ "7005": 2.4667,
+ "7010": 2.47022,
+ "7015": 2.51594,
+ "7020": 2.46517,
+ "7025": 2.44841,
+ "7030": 2.48208,
+ "7035": 2.47701,
+ "7040": 2.50402,
+ "7045": 2.51891,
+ "7050": 2.52336,
+ "7055": 2.4399,
+ "7060": 2.47602,
+ "7065": 2.48595,
+ "7070": 2.49079,
+ "7075": 2.49162,
+ "7080": 2.53301,
+ "7085": 2.48237,
+ "7090": 2.473,
+ "7095": 2.49968,
+ "7100": 2.51235,
+ "7105": 2.48475,
+ "7110": 2.48398,
+ "7115": 2.5008,
+ "7120": 2.46802,
+ "7125": 2.45993,
+ "7130": 2.48174,
+ "7135": 2.51009,
+ "7140": 2.49524,
+ "7145": 2.49547,
+ "7150": 2.50838,
+ "7155": 2.50549,
+ "7160": 2.47267,
+ "7165": 2.45441,
+ "7170": 2.50323,
+ "7175": 2.50138,
+ "7180": 2.50287,
+ "7185": 2.47888,
+ "7190": 2.45754,
+ "7195": 2.46282,
+ "7200": 2.50582,
+ "7205": 2.48673,
+ "7210": 2.44143,
+ "7215": 2.4774,
+ "7220": 2.44056,
+ "7225": 2.50992,
+ "7230": 2.50819,
+ "7235": 2.48137,
+ "7240": 2.47521,
+ "7245": 2.49699,
+ "7250": 2.50435,
+ "7255": 2.48907,
+ "7260": 2.4559,
+ "7265": 2.44754,
+ "7270": 2.46985,
+ "7275": 2.49711,
+ "7280": 2.49068,
+ "7285": 2.42028,
+ "7290": 2.4765,
+ "7295": 2.48576,
+ "7300": 2.41537,
+ "7305": 2.44172,
+ "7310": 2.44612,
+ "7315": 2.48688,
+ "7320": 2.48061,
+ "7325": 2.45471,
+ "7330": 2.48741,
+ "7335": 2.47103,
+ "7340": 2.45999,
+ "7345": 2.49065,
+ "7350": 2.50759,
+ "7355": 2.49204,
+ "7360": 2.47672,
+ "7365": 2.46675,
+ "7370": 2.46718,
+ "7375": 2.44736,
+ "7380": 2.49247,
+ "7385": 2.48245,
+ "7390": 2.47143,
+ "7395": 2.47076,
+ "7400": 2.47826,
+ "7405": 2.43636,
+ "7410": 2.47594,
+ "7415": 2.46762,
+ "7420": 2.49047,
+ "7425": 2.45309,
+ "7430": 2.51923,
+ "7435": 2.48835,
+ "7440": 2.51712,
+ "7445": 2.50649,
+ "7450": 2.47023,
+ "7455": 2.4505,
+ "7460": 2.46232,
+ "7465": 2.47224,
+ "7470": 2.44654,
+ "7475": 2.45263,
+ "7480": 2.50696,
+ "7485": 2.44731,
+ "7490": 2.47151,
+ "7495": 2.47897,
+ "7500": 2.49176,
+ "7505": 2.43714,
+ "7510": 2.4327,
+ "7515": 2.41782,
+ "7520": 2.48958,
+ "7525": 2.49304,
+ "7530": 2.47239,
+ "7535": 2.45795,
+ "7540": 2.47005,
+ "7545": 2.47142,
+ "7550": 2.48676,
+ "7555": 2.45126,
+ "7560": 2.42516,
+ "7565": 2.50493,
+ "7570": 2.48155,
+ "7575": 2.43534,
+ "7580": 2.45593,
+ "7585": 2.47794,
+ "7590": 2.47723,
+ "7595": 2.45983,
+ "7600": 2.4619,
+ "7605": 2.44428,
+ "7610": 2.44666,
+ "7615": 2.42334,
+ "7620": 2.54396,
+ "7625": 2.47914,
+ "7630": 2.42239,
+ "7635": 2.42463,
+ "7640": 2.44931,
+ "7645": 2.47013,
+ "7650": 2.45978,
+ "7655": 2.48267,
+ "7660": 2.45011,
+ "7665": 2.43052,
+ "7670": 2.43917,
+ "7675": 2.45127,
+ "7680": 2.4822,
+ "7685": 2.42789,
+ "7690": 2.47748,
+ "7695": 2.45207,
+ "7700": 2.47852,
+ "7705": 2.49472,
+ "7710": 2.4906,
+ "7715": 2.43857,
+ "7720": 2.46528,
+ "7725": 2.47882,
+ "7730": 2.45384,
+ "7735": 2.46895,
+ "7740": 2.43524,
+ "7745": 2.44409,
+ "7750": 2.43493,
+ "7755": 2.46349,
+ "7760": 2.44734,
+ "7765": 2.45062,
+ "7770": 2.46527,
+ "7775": 2.44968,
+ "7780": 2.41367,
+ "7785": 2.44115,
+ "7790": 2.47809,
+ "7795": 2.43666,
+ "7800": 2.45672,
+ "7805": 2.47745,
+ "7810": 2.49856,
+ "7815": 2.48425,
+ "7820": 2.44372,
+ "7825": 2.51041,
+ "7830": 2.44874,
+ "7835": 2.46354,
+ "7840": 2.47479,
+ "7845": 2.45731,
+ "7850": 2.41361,
+ "7855": 2.46795,
+ "7860": 2.49498,
+ "7865": 2.42043,
+ "7870": 2.46323,
+ "7875": 2.44232,
+ "7880": 2.44989,
+ "7885": 2.45571,
+ "7890": 2.46629,
+ "7895": 2.44233,
+ "7900": 2.43437,
+ "7905": 2.43129,
+ "7910": 2.41956,
+ "7915": 2.47833,
+ "7920": 2.47132,
+ "7925": 2.41801,
+ "7930": 2.46833,
+ "7935": 2.44775,
+ "7940": 2.42007,
+ "7945": 2.46791,
+ "7950": 2.4404,
+ "7955": 2.4146,
+ "7960": 2.48514,
+ "7965": 2.51375,
+ "7970": 2.5174,
+ "7975": 2.44637,
+ "7980": 2.43831,
+ "7985": 2.46457,
+ "7990": 2.4281,
+ "7995": 2.46689,
+ "8000": 2.43426,
+ "8005": 2.4158,
+ "8010": 2.45426,
+ "8015": 2.46634,
+ "8020": 2.47876,
+ "8025": 2.47029,
+ "8030": 2.4497,
+ "8035": 2.46716,
+ "8040": 2.41566,
+ "8045": 2.45052,
+ "8050": 2.44515,
+ "8055": 2.42304,
+ "8060": 2.43892,
+ "8065": 2.45749,
+ "8070": 2.45333,
+ "8075": 2.45444,
+ "8080": 2.44209,
+ "8085": 2.43781,
+ "8090": 2.4222,
+ "8095": 2.42035,
+ "8100": 2.43701,
+ "8105": 2.49226,
+ "8110": 2.43554,
+ "8115": 2.44107,
+ "8120": 2.46446,
+ "8125": 2.46397,
+ "8130": 2.45005,
+ "8135": 2.44859,
+ "8140": 2.43719,
+ "8145": 2.42285,
+ "8150": 2.41929,
+ "8155": 2.48306,
+ "8160": 2.44961,
+ "8165": 2.43799,
+ "8170": 2.43144,
+ "8175": 2.4191,
+ "8180": 2.49169,
+ "8185": 2.42127,
+ "8190": 2.46436,
+ "8195": 2.45449,
+ "8200": 2.44219,
+ "8205": 2.44226,
+ "8210": 2.42747,
+ "8215": 2.43706,
+ "8220": 2.43181,
+ "8225": 2.40713,
+ "8230": 2.43716,
+ "8235": 2.46065,
+ "8240": 2.42535,
+ "8245": 2.4454,
+ "8250": 2.44079,
+ "8255": 2.4331,
+ "8260": 2.43059,
+ "8265": 2.4271,
+ "8270": 2.43176,
+ "8275": 2.43875,
+ "8280": 2.3943,
+ "8285": 2.43559,
+ "8290": 2.47695,
+ "8295": 2.44448,
+ "8300": 2.45474,
+ "8305": 2.40521,
+ "8310": 2.4311,
+ "8315": 2.45391,
+ "8320": 2.3955,
+ "8325": 2.3912,
+ "8330": 2.43328,
+ "8335": 2.44265,
+ "8340": 2.48794,
+ "8345": 2.45065,
+ "8350": 2.45145,
+ "8355": 2.41099,
+ "8360": 2.40381,
+ "8365": 2.45603,
+ "8370": 2.45802,
+ "8375": 2.42948,
+ "8380": 2.42394,
+ "8385": 2.43031,
+ "8390": 2.4431,
+ "8395": 2.44939,
+ "8400": 2.44584,
+ "8405": 2.495,
+ "8410": 2.44327,
+ "8415": 2.43894,
+ "8420": 2.42215,
+ "8425": 2.44395,
+ "8430": 2.46704,
+ "8435": 2.41158,
+ "8440": 2.4595,
+ "8445": 2.46637,
+ "8450": 2.41361,
+ "8455": 2.46609,
+ "8460": 2.46112,
+ "8465": 2.44125,
+ "8470": 2.41471,
+ "8475": 2.48334,
+ "8480": 2.41021,
+ "8485": 2.42432,
+ "8490": 2.47209,
+ "8495": 2.44598,
+ "8500": 2.45224,
+ "8505": 2.42377,
+ "8510": 2.41131,
+ "8515": 2.43868,
+ "8520": 2.43196,
+ "8525": 2.49903,
+ "8530": 2.38038,
+ "8535": 2.40733,
+ "8540": 2.49469,
+ "8545": 2.3894,
+ "8550": 2.44785,
+ "8555": 2.45932,
+ "8560": 2.47841,
+ "8565": 2.43193,
+ "8570": 2.44137,
+ "8575": 2.45682,
+ "8580": 2.44831,
+ "8585": 2.42889,
+ "8590": 2.41034,
+ "8595": 2.43429,
+ "8600": 2.42179,
+ "8605": 2.5001,
+ "8610": 2.4355,
+ "8615": 2.39538,
+ "8620": 2.45793,
+ "8625": 2.43491,
+ "8630": 2.46953,
+ "8635": 2.46026,
+ "8640": 2.44563,
+ "8645": 2.48328,
+ "8650": 2.4289,
+ "8655": 2.46099,
+ "8660": 2.46544,
+ "8665": 2.39596,
+ "8670": 2.4187,
+ "8675": 2.43748,
+ "8680": 2.45725,
+ "8685": 2.43903,
+ "8690": 2.41756,
+ "8695": 2.44994,
+ "8700": 2.44424,
+ "8705": 2.43203,
+ "8710": 2.43799,
+ "8715": 2.45796,
+ "8720": 2.4885,
+ "8725": 2.42286,
+ "8730": 2.40182,
+ "8735": 2.44324,
+ "8740": 2.43971,
+ "8745": 2.40735,
+ "8750": 2.44644,
+ "8755": 2.43268,
+ "8760": 2.40882,
+ "8765": 2.44368,
+ "8770": 2.41346,
+ "8775": 2.44728,
+ "8780": 2.42897,
+ "8785": 2.48105,
+ "8790": 2.42913,
+ "8795": 2.4275,
+ "8800": 2.42507,
+ "8805": 2.41547,
+ "8810": 2.41989,
+ "8815": 2.48573,
+ "8820": 2.46245,
+ "8825": 2.43508,
+ "8830": 2.39363,
+ "8835": 2.43059,
+ "8840": 2.4011,
+ "8845": 2.43561,
+ "8850": 2.4432,
+ "8855": 2.41208,
+ "8860": 2.43692,
+ "8865": 2.4368,
+ "8870": 2.44614,
+ "8875": 2.44694,
+ "8880": 2.41968,
+ "8885": 2.40435,
+ "8890": 2.45636,
+ "8895": 2.43825,
+ "8900": 2.42221,
+ "8905": 2.41086,
+ "8910": 2.40878,
+ "8915": 2.42762,
+ "8920": 2.44256,
+ "8925": 2.47644,
+ "8930": 2.42537,
+ "8935": 2.42038,
+ "8940": 2.39746,
+ "8945": 2.40294,
+ "8950": 2.42585,
+ "8955": 2.40426,
+ "8960": 2.4429,
+ "8965": 2.42534,
+ "8970": 2.41287,
+ "8975": 2.48503,
+ "8980": 2.44965,
+ "8985": 2.38255,
+ "8990": 2.41786,
+ "8995": 2.42593,
+ "9000": 2.46556,
+ "9005": 2.42295,
+ "9010": 2.38472,
+ "9015": 2.41606,
+ "9020": 2.40591,
+ "9025": 2.37666,
+ "9030": 2.41023,
+ "9035": 2.43334,
+ "9040": 2.43041,
+ "9045": 2.42903,
+ "9050": 2.40571,
+ "9055": 2.42869,
+ "9060": 2.43178,
+ "9065": 2.41519,
+ "9070": 2.45462,
+ "9075": 2.40713,
+ "9080": 2.44279,
+ "9085": 2.42534,
+ "9090": 2.42125,
+ "9095": 2.40854,
+ "9100": 2.41025,
+ "9105": 2.367,
+ "9110": 2.47453,
+ "9115": 2.42448,
+ "9120": 2.41281,
+ "9125": 2.46625,
+ "9130": 2.40301,
+ "9135": 2.45728,
+ "9140": 2.44304,
+ "9145": 2.43611,
+ "9150": 2.43471,
+ "9155": 2.38526,
+ "9160": 2.42692,
+ "9165": 2.43405,
+ "9170": 2.38388,
+ "9175": 2.42789,
+ "9180": 2.38445,
+ "9185": 2.4481,
+ "9190": 2.42039,
+ "9195": 2.39805,
+ "9200": 2.40085,
+ "9205": 2.45829,
+ "9210": 2.371,
+ "9215": 2.47166,
+ "9220": 2.45642,
+ "9225": 2.39429,
+ "9230": 2.45657,
+ "9235": 2.40768,
+ "9240": 2.41027,
+ "9245": 2.44666,
+ "9250": 2.43485,
+ "9255": 2.43757,
+ "9260": 2.39426,
+ "9265": 2.44798,
+ "9270": 2.44482,
+ "9275": 2.40359,
+ "9280": 2.39705,
+ "9285": 2.43007,
+ "9290": 2.41426,
+ "9295": 2.39299,
+ "9300": 2.43289,
+ "9305": 2.41279,
+ "9310": 2.423,
+ "9315": 2.41942,
+ "9320": 2.45204,
+ "9325": 2.37844,
+ "9330": 2.41244,
+ "9335": 2.36996,
+ "9340": 2.41408,
+ "9345": 2.42371,
+ "9350": 2.44769,
+ "9355": 2.48475,
+ "9360": 2.44528,
+ "9365": 2.39717,
+ "9370": 2.44369,
+ "9375": 2.44154,
+ "9380": 2.36344,
+ "9385": 2.40893,
+ "9390": 2.392,
+ "9395": 2.39609,
+ "9400": 2.45217,
+ "9405": 2.42157,
+ "9410": 2.40685,
+ "9415": 2.4464,
+ "9420": 2.45271,
+ "9425": 2.44004,
+ "9430": 2.45598,
+ "9435": 2.42225,
+ "9440": 2.48766,
+ "9445": 2.38406,
+ "9450": 2.403,
+ "9455": 2.41116,
+ "9460": 2.39528,
+ "9465": 2.38808,
+ "9470": 2.39304,
+ "9475": 2.37421,
+ "9480": 2.44231,
+ "9485": 2.3953,
+ "9490": 2.429,
+ "9495": 2.39024,
+ "9500": 2.37127,
+ "9505": 2.43715,
+ "9510": 2.40592,
+ "9515": 2.43922,
+ "9520": 2.4267,
+ "9525": 2.39745,
+ "9530": 2.46261,
+ "9535": 2.4094,
+ "9540": 2.42584,
+ "9545": 2.38754,
+ "9550": 2.43208,
+ "9555": 2.40052,
+ "9560": 2.43543,
+ "9565": 2.42095,
+ "9570": 2.3827,
+ "9575": 2.421,
+ "9580": 2.40541,
+ "9585": 2.43312,
+ "9590": 2.43539,
+ "9595": 2.45662,
+ "9600": 2.39949,
+ "9605": 2.39163,
+ "9610": 2.42764,
+ "9615": 2.42109,
+ "9620": 2.42093,
+ "9625": 2.45314,
+ "9630": 2.40624,
+ "9635": 2.41073,
+ "9640": 2.45484,
+ "9645": 2.41927,
+ "9650": 2.40829,
+ "9655": 2.38,
+ "9660": 2.43229,
+ "9665": 2.39547,
+ "9670": 2.38951,
+ "9675": 2.36541,
+ "9680": 2.40451,
+ "9685": 2.40399,
+ "9690": 2.47182,
+ "9695": 2.3897,
+ "9700": 2.38473,
+ "9705": 2.39081,
+ "9710": 2.37373,
+ "9715": 2.39466,
+ "9720": 2.44223,
+ "9725": 2.44947,
+ "9730": 2.43603,
+ "9735": 2.39314,
+ "9740": 2.38934,
+ "9745": 2.43447,
+ "9750": 2.40659,
+ "9755": 2.41677,
+ "9760": 2.41903,
+ "9765": 2.3765,
+ "9770": 2.45819,
+ "9775": 2.40841,
+ "9780": 2.37191,
+ "9785": 2.40974,
+ "9790": 2.41715,
+ "9795": 2.36864,
+ "9800": 2.40489,
+ "9805": 2.41293,
+ "9810": 2.41636,
+ "9815": 2.38765,
+ "9820": 2.38452,
+ "9825": 2.41458,
+ "9830": 2.42939,
+ "9835": 2.39324,
+ "9840": 2.42203,
+ "9845": 2.36981,
+ "9850": 2.40762,
+ "9855": 2.40309,
+ "9860": 2.39844,
+ "9865": 2.38777,
+ "9870": 2.39425,
+ "9875": 2.38784,
+ "9880": 2.45814,
+ "9885": 2.40039,
+ "9890": 2.36126,
+ "9895": 2.32977,
+ "9900": 2.40372,
+ "9905": 2.43103,
+ "9910": 2.36341,
+ "9915": 2.37042,
+ "9920": 2.41821,
+ "9925": 2.40625,
+ "9930": 2.39008,
+ "9935": 2.35818,
+ "9940": 2.39734,
+ "9945": 2.38754,
+ "9950": 2.4102,
+ "9955": 2.45576,
+ "9960": 2.43796,
+ "9965": 2.36266,
+ "9970": 2.41282,
+ "9975": 2.39226,
+ "9980": 2.34002,
+ "9985": 2.4136,
+ "9990": 2.40407,
+ "9995": 2.40251,
+ "10000": 2.37503,
+ "10005": 2.38016,
+ "10010": 2.38897,
+ "10015": 2.45152,
+ "10020": 2.37135,
+ "10025": 2.39485,
+ "10030": 2.39555,
+ "10035": 2.41697,
+ "10040": 2.41008,
+ "10045": 2.39015,
+ "10050": 2.3608,
+ "10055": 2.37497,
+ "10060": 2.42662,
+ "10065": 2.381,
+ "10070": 2.42931,
+ "10075": 2.37803,
+ "10080": 2.36864,
+ "10085": 2.37765,
+ "10090": 2.35414,
+ "10095": 2.40824,
+ "10100": 2.32065,
+ "10105": 2.38938,
+ "10110": 2.41723,
+ "10115": 2.3932,
+ "10120": 2.3649,
+ "10125": 2.3777,
+ "10130": 2.36629,
+ "10135": 2.38894,
+ "10140": 2.41886,
+ "10145": 2.41313,
+ "10150": 2.38364,
+ "10155": 2.40171,
+ "10160": 2.36831,
+ "10165": 2.38929,
+ "10170": 2.42951,
+ "10175": 2.33254,
+ "10180": 2.40139,
+ "10185": 2.38961,
+ "10190": 2.44963,
+ "10195": 2.40903,
+ "10200": 2.39636,
+ "10205": 2.39541,
+ "10210": 2.37545,
+ "10215": 2.35138,
+ "10220": 2.42686,
+ "10225": 2.4374,
+ "10230": 2.36151,
+ "10235": 2.39413,
+ "10240": 2.37745,
+ "10245": 2.39586,
+ "10250": 2.39498,
+ "10255": 2.42063,
+ "10260": 2.34179,
+ "10265": 2.35595,
+ "10270": 2.35696,
+ "10275": 2.37742,
+ "10280": 2.45578,
+ "10285": 2.36405,
+ "10290": 2.39142,
+ "10295": 2.37906,
+ "10300": 2.37381,
+ "10305": 2.42262,
+ "10310": 2.39511,
+ "10315": 2.36789,
+ "10320": 2.3721,
+ "10325": 2.36743,
+ "10330": 2.41904,
+ "10335": 2.36838,
+ "10340": 2.42816,
+ "10345": 2.37986,
+ "10350": 2.36588,
+ "10355": 2.40402,
+ "10360": 2.38027,
+ "10365": 2.36618,
+ "10370": 2.34647,
+ "10375": 2.36068,
+ "10380": 2.42359,
+ "10385": 2.41044,
+ "10390": 2.38547,
+ "10395": 2.36378,
+ "10400": 2.3835,
+ "10405": 2.35412,
+ "10410": 2.34482,
+ "10415": 2.42134,
+ "10420": 2.38596,
+ "10425": 2.33277,
+ "10430": 2.36637,
+ "10435": 2.37673,
+ "10440": 2.37865,
+ "10445": 2.36491,
+ "10450": 2.3669,
+ "10455": 2.38595,
+ "10460": 2.38657,
+ "10465": 2.30837,
+ "10470": 2.36369,
+ "10475": 2.38619,
+ "10480": 2.36847,
+ "10485": 2.3661,
+ "10490": 2.42007,
+ "10495": 2.37239,
+ "10500": 2.37043,
+ "10505": 2.37665,
+ "10510": 2.38893,
+ "10515": 2.3786,
+ "10520": 2.40715,
+ "10525": 2.39544,
+ "10530": 2.39751,
+ "10535": 2.36068,
+ "10540": 2.41035,
+ "10545": 2.36388,
+ "10550": 2.38204,
+ "10555": 2.36345,
+ "10560": 2.34633,
+ "10565": 2.37874,
+ "10570": 2.38037,
+ "10575": 2.36022,
+ "10580": 2.38294,
+ "10585": 2.37306,
+ "10590": 2.38416,
+ "10595": 2.38213,
+ "10600": 2.33865,
+ "10605": 2.37753,
+ "10610": 2.37058,
+ "10615": 2.37039,
+ "10620": 2.35512,
+ "10625": 2.42587,
+ "10630": 2.37959,
+ "10635": 2.32939,
+ "10640": 2.37025,
+ "10645": 2.42582,
+ "10650": 2.36625,
+ "10655": 2.31323,
+ "10660": 2.35436,
+ "10665": 2.40571,
+ "10670": 2.32294,
+ "10675": 2.42284,
+ "10680": 2.36013,
+ "10685": 2.29341,
+ "10690": 2.38755,
+ "10695": 2.33602,
+ "10700": 2.39021,
+ "10705": 2.39103,
+ "10710": 2.34839,
+ "10715": 2.38791,
+ "10720": 2.33122,
+ "10725": 2.35822,
+ "10730": 2.35545,
+ "10735": 2.35889,
+ "10740": 2.32328,
+ "10745": 2.3436,
+ "10750": 2.33874,
+ "10755": 2.40853,
+ "10760": 2.37013,
+ "10765": 2.34207,
+ "10770": 2.3764,
+ "10775": 2.39275,
+ "10780": 2.37572,
+ "10785": 2.39828,
+ "10790": 2.35249,
+ "10795": 2.39336,
+ "10800": 2.32898,
+ "10805": 2.40193,
+ "10810": 2.38061,
+ "10815": 2.35979,
+ "10820": 2.34858,
+ "10825": 2.37631,
+ "10830": 2.34337,
+ "10835": 2.35361,
+ "10840": 2.33499,
+ "10845": 2.39216,
+ "10850": 2.33795,
+ "10855": 2.36908,
+ "10860": 2.33711,
+ "10865": 2.32743,
+ "10870": 2.32848,
+ "10875": 2.30977,
+ "10880": 2.39976,
+ "10885": 2.40825,
+ "10890": 2.36578,
+ "10895": 2.37644,
+ "10900": 2.33747,
+ "10905": 2.31849,
+ "10910": 2.41102,
+ "10915": 2.37581,
+ "10920": 2.37944,
+ "10925": 2.36681,
+ "10930": 2.32353,
+ "10935": 2.366,
+ "10940": 2.35829,
+ "10945": 2.35268,
+ "10950": 2.36612,
+ "10955": 2.36831,
+ "10960": 2.31609,
+ "10965": 2.36861,
+ "10970": 2.36195,
+ "10975": 2.41427,
+ "10980": 2.37916,
+ "10985": 2.34718,
+ "10990": 2.40203,
+ "10995": 2.36813,
+ "11000": 2.3431,
+ "11005": 2.36593,
+ "11010": 2.34815,
+ "11015": 2.33058,
+ "11020": 2.33971,
+ "11025": 2.37059,
+ "11030": 2.34495,
+ "11035": 2.32056,
+ "11040": 2.32299,
+ "11045": 2.32437,
+ "11050": 2.32271,
+ "11055": 2.29492,
+ "11060": 2.34574,
+ "11065": 2.31429,
+ "11070": 2.3997,
+ "11075": 2.32488,
+ "11080": 2.35909,
+ "11085": 2.34215,
+ "11090": 2.35097,
+ "11095": 2.37591,
+ "11100": 2.33353,
+ "11105": 2.32079,
+ "11110": 2.36825,
+ "11115": 2.37722,
+ "11120": 2.38574,
+ "11125": 2.32049,
+ "11130": 2.35432,
+ "11135": 2.33798,
+ "11140": 2.3756,
+ "11145": 2.35296,
+ "11150": 2.39873,
+ "11155": 2.34565,
+ "11160": 2.36975,
+ "11165": 2.36888,
+ "11170": 2.34562,
+ "11175": 2.33842,
+ "11180": 2.377,
+ "11185": 2.31587,
+ "11190": 2.28274,
+ "11195": 2.33321,
+ "11200": 2.35058,
+ "11205": 2.36546,
+ "11210": 2.33584,
+ "11215": 2.32266,
+ "11220": 2.34696,
+ "11225": 2.37525,
+ "11230": 2.37025,
+ "11235": 2.32278,
+ "11240": 2.34703,
+ "11245": 2.36117,
+ "11250": 2.33601,
+ "11255": 2.33861,
+ "11260": 2.36043,
+ "11265": 2.39195,
+ "11270": 2.29246,
+ "11275": 2.31756,
+ "11280": 2.37339,
+ "11285": 2.29474,
+ "11290": 2.35128,
+ "11295": 2.36883,
+ "11300": 2.38525,
+ "11305": 2.33833,
+ "11310": 2.335,
+ "11315": 2.30246,
+ "11320": 2.30888,
+ "11325": 2.31935,
+ "11330": 2.35739,
+ "11335": 2.34267,
+ "11340": 2.31199,
+ "11345": 2.31657,
+ "11350": 2.29981,
+ "11355": 2.32629,
+ "11360": 2.35455,
+ "11365": 2.29683,
+ "11370": 2.35491,
+ "11375": 2.33083,
+ "11380": 2.34514,
+ "11385": 2.35134,
+ "11390": 2.33916,
+ "11395": 2.2904,
+ "11400": 2.31403,
+ "11405": 2.35915,
+ "11410": 2.35981,
+ "11415": 2.3881,
+ "11420": 2.35419,
+ "11425": 2.3118,
+ "11430": 2.37189,
+ "11435": 2.36599,
+ "11440": 2.35189,
+ "11445": 2.36747,
+ "11450": 2.32505,
+ "11455": 2.31027,
+ "11460": 2.35577,
+ "11465": 2.34641,
+ "11470": 2.37697,
+ "11475": 2.31611,
+ "11480": 2.32987,
+ "11485": 2.3146,
+ "11490": 2.34959,
+ "11495": 2.41145,
+ "11500": 2.34337,
+ "11505": 2.35276,
+ "11510": 2.36718,
+ "11515": 2.32509,
+ "11520": 2.30839,
+ "11525": 2.36337,
+ "11530": 2.3194,
+ "11535": 2.32609,
+ "11540": 2.35006,
+ "11545": 2.34693,
+ "11550": 2.36786,
+ "11555": 2.32993,
+ "11560": 2.3536,
+ "11565": 2.34458,
+ "11570": 2.3536,
+ "11575": 2.3011,
+ "11580": 2.33205,
+ "11585": 2.35602,
+ "11590": 2.36689,
+ "11595": 2.33751,
+ "11600": 2.36086,
+ "11605": 2.32743,
+ "11610": 2.36471,
+ "11615": 2.36332,
+ "11620": 2.29974,
+ "11625": 2.28036,
+ "11630": 2.32987,
+ "11635": 2.34628,
+ "11640": 2.30899,
+ "11645": 2.31323,
+ "11650": 2.33117,
+ "11655": 2.35585,
+ "11660": 2.34024,
+ "11665": 2.33286,
+ "11670": 2.30287,
+ "11675": 2.30075,
+ "11680": 2.32815,
+ "11685": 2.33901,
+ "11690": 2.34716,
+ "11695": 2.3197,
+ "11700": 2.32709,
+ "11705": 2.30459,
+ "11710": 2.34705,
+ "11715": 2.31826,
+ "11720": 2.30389,
+ "11725": 2.34199,
+ "11730": 2.30779,
+ "11735": 2.33209,
+ "11740": 2.2753,
+ "11745": 2.32144,
+ "11750": 2.3336,
+ "11755": 2.35456,
+ "11760": 2.31473,
+ "11765": 2.33841,
+ "11770": 2.27811,
+ "11775": 2.32897,
+ "11780": 2.25799,
+ "11785": 2.30024,
+ "11790": 2.31493,
+ "11795": 2.32371,
+ "11800": 2.33793,
+ "11805": 2.30621,
+ "11810": 2.30786,
+ "11815": 2.3343,
+ "11820": 2.32101,
+ "11825": 2.36172,
+ "11830": 2.32071,
+ "11835": 2.34063,
+ "11840": 2.34274,
+ "11845": 2.32181,
+ "11850": 2.30581,
+ "11855": 2.31644,
+ "11860": 2.34571,
+ "11865": 2.36074,
+ "11870": 2.38141,
+ "11875": 2.2838,
+ "11880": 2.29616,
+ "11885": 2.34139,
+ "11890": 2.29488,
+ "11895": 2.29572,
+ "11900": 2.33763,
+ "11905": 2.32182,
+ "11910": 2.28009,
+ "11915": 2.31065,
+ "11920": 2.33704,
+ "11925": 2.30583,
+ "11930": 2.30791,
+ "11935": 2.31945,
+ "11940": 2.31995,
+ "11945": 2.34572,
+ "11950": 2.30214,
+ "11955": 2.31571,
+ "11960": 2.33963,
+ "11965": 2.30105,
+ "11970": 2.30187,
+ "11975": 2.3488,
+ "11980": 2.31378,
+ "11985": 2.28272,
+ "11990": 2.30796,
+ "11995": 2.33162,
+ "12000": 2.32448,
+ "12005": 2.32719,
+ "12010": 2.29018,
+ "12015": 2.31336,
+ "12020": 2.32951,
+ "12025": 2.33643,
+ "12030": 2.31428,
+ "12035": 2.33848,
+ "12040": 2.31854,
+ "12045": 2.31434,
+ "12050": 2.31215,
+ "12055": 2.3352,
+ "12060": 2.30159,
+ "12065": 2.33219,
+ "12070": 2.30837,
+ "12075": 2.28037,
+ "12080": 2.35467,
+ "12085": 2.34152,
+ "12090": 2.33354,
+ "12095": 2.28306,
+ "12100": 2.31673,
+ "12105": 2.31069,
+ "12110": 2.33069,
+ "12115": 2.30641,
+ "12120": 2.30704,
+ "12125": 2.29609,
+ "12130": 2.30622,
+ "12135": 2.33037,
+ "12140": 2.29688,
+ "12145": 2.25958,
+ "12150": 2.26153,
+ "12155": 2.34294,
+ "12160": 2.35915,
+ "12165": 2.32051,
+ "12170": 2.334,
+ "12175": 2.34314,
+ "12180": 2.3318,
+ "12185": 2.34303,
+ "12190": 2.33577,
+ "12195": 2.30093,
+ "12200": 2.30246,
+ "12205": 2.32658,
+ "12210": 2.35972,
+ "12215": 2.30653,
+ "12220": 2.30191,
+ "12225": 2.24709,
+ "12230": 2.33677,
+ "12235": 2.34065,
+ "12240": 2.32782,
+ "12245": 2.2935,
+ "12250": 2.27763,
+ "12255": 2.3395,
+ "12260": 2.31679,
+ "12265": 2.34478,
+ "12270": 2.31554,
+ "12275": 2.31677,
+ "12280": 2.32253,
+ "12285": 2.28952,
+ "12290": 2.31433,
+ "12295": 2.26884,
+ "12300": 2.33108,
+ "12305": 2.27093,
+ "12310": 2.28974,
+ "12315": 2.35756,
+ "12320": 2.29797,
+ "12325": 2.32217,
+ "12330": 2.30328,
+ "12335": 2.32209,
+ "12340": 2.34279,
+ "12345": 2.37023,
+ "12350": 2.34617,
+ "12355": 2.30866,
+ "12360": 2.31576,
+ "12365": 2.33299,
+ "12370": 2.29597,
+ "12375": 2.30396,
+ "12380": 2.30039,
+ "12385": 2.29488,
+ "12390": 2.25304,
+ "12395": 2.30748,
+ "12400": 2.30213,
+ "12405": 2.31418,
+ "12410": 2.30832,
+ "12415": 2.2868,
+ "12420": 2.32194,
+ "12425": 2.30362,
+ "12430": 2.31855,
+ "12435": 2.30193,
+ "12440": 2.33517,
+ "12445": 2.3228,
+ "12450": 2.30936,
+ "12455": 2.24374,
+ "12460": 2.33755,
+ "12465": 2.36582,
+ "12470": 2.27944,
+ "12475": 2.27513,
+ "12480": 2.29532,
+ "12485": 2.30917,
+ "12490": 2.33321,
+ "12495": 2.27309,
+ "12500": 2.32322,
+ "12505": 2.33899,
+ "12510": 2.35918,
+ "12515": 2.27269,
+ "12520": 2.32282,
+ "12525": 2.28953,
+ "12530": 2.32246,
+ "12535": 2.27508,
+ "12540": 2.28833,
+ "12545": 2.29304,
+ "12550": 2.31847,
+ "12555": 2.32567,
+ "12560": 2.30247,
+ "12565": 2.33702,
+ "12570": 2.27975,
+ "12575": 2.30359,
+ "12580": 2.31404,
+ "12585": 2.29346,
+ "12590": 2.33723,
+ "12595": 2.32629,
+ "12600": 2.28432,
+ "12605": 2.32186,
+ "12610": 2.36725,
+ "12615": 2.30853,
+ "12620": 2.33633,
+ "12625": 2.33404,
+ "12630": 2.30017,
+ "12635": 2.33795,
+ "12640": 2.29794,
+ "12645": 2.28149,
+ "12650": 2.32885,
+ "12655": 2.26803,
+ "12660": 2.34509,
+ "12665": 2.32031,
+ "12670": 2.31308,
+ "12675": 2.32177,
+ "12680": 2.2781,
+ "12685": 2.36828,
+ "12690": 2.30518,
+ "12695": 2.33467,
+ "12700": 2.29576,
+ "12705": 2.30963,
+ "12710": 2.31082,
+ "12715": 2.2908,
+ "12720": 2.3163,
+ "12725": 2.27788,
+ "12730": 2.34119,
+ "12735": 2.3003,
+ "12740": 2.34099,
+ "12745": 2.28836,
+ "12750": 2.27593,
+ "12755": 2.28477,
+ "12760": 2.27021,
+ "12765": 2.34085,
+ "12770": 2.32873,
+ "12775": 2.26616,
+ "12780": 2.32288,
+ "12785": 2.30308,
+ "12790": 2.3082,
+ "12795": 2.32298,
+ "12800": 2.29883,
+ "12805": 2.31865,
+ "12810": 2.28331,
+ "12815": 2.30037,
+ "12820": 2.32651,
+ "12825": 2.32484,
+ "12830": 2.29196,
+ "12835": 2.2712,
+ "12840": 2.27357,
+ "12845": 2.31533,
+ "12850": 2.27968,
+ "12855": 2.2743,
+ "12860": 2.27381,
+ "12865": 2.31935,
+ "12870": 2.2679,
+ "12875": 2.34222,
+ "12880": 2.32036,
+ "12885": 2.28574,
+ "12890": 2.31145,
+ "12895": 2.24813,
+ "12900": 2.32757,
+ "12905": 2.31794,
+ "12910": 2.29019,
+ "12915": 2.28757,
+ "12920": 2.30176,
+ "12925": 2.30293,
+ "12930": 2.27492,
+ "12935": 2.24416,
+ "12940": 2.26089,
+ "12945": 2.31287,
+ "12950": 2.28712,
+ "12955": 2.32917,
+ "12960": 2.31954,
+ "12965": 2.29119,
+ "12970": 2.27432,
+ "12975": 2.27007,
+ "12980": 2.33407,
+ "12985": 2.28074,
+ "12990": 2.2834,
+ "12995": 2.27493,
+ "13000": 2.25272
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": 956238080.0,
+ "5": 967340032.0,
+ "10": 971387136.0,
+ "15": 946583872.0,
+ "20": 958988544.0,
+ "25": 1069888064.0,
+ "30": 1195545216.0,
+ "35": 1265919104.0,
+ "40": 1225385472.0,
+ "45": 1143984000.0,
+ "50": 1123953792.0,
+ "55": 1097520640.0,
+ "60": 1062098688.0,
+ "65": 1045100544.0,
+ "70": 1013369024.0,
+ "75": 1002651456.0,
+ "80": 1017073408.0,
+ "85": 1010896064.0,
+ "90": 990290304.0,
+ "95": 963963648.0,
+ "100": 977244864.0,
+ "105": 986322432.0,
+ "110": 979685888.0,
+ "115": 981374592.0,
+ "120": 963850304.0,
+ "125": 943482752.0,
+ "130": 976245632.0,
+ "135": 967085504.0,
+ "140": 963599104.0,
+ "145": 977430784.0,
+ "150": 922111680.0,
+ "155": 968683328.0,
+ "160": 956706432.0,
+ "165": 960015936.0,
+ "170": 974511872.0,
+ "175": 949081792.0,
+ "180": 946759872.0,
+ "185": 972029696.0,
+ "190": 969055616.0,
+ "195": 985147648.0,
+ "200": 945774912.0,
+ "205": 958352384.0,
+ "210": 979447552.0,
+ "215": 967492032.0,
+ "220": 956426880.0,
+ "225": 962403968.0,
+ "230": 948176128.0,
+ "235": 965226688.0,
+ "240": 966071360.0,
+ "245": 969163776.0,
+ "250": 974432512.0,
+ "255": 925067072.0,
+ "260": 965635840.0,
+ "265": 970669952.0,
+ "270": 959135488.0,
+ "275": 954000256.0,
+ "280": 963429120.0,
+ "285": 945788160.0,
+ "290": 974126784.0,
+ "295": 966698112.0,
+ "300": 967156352.0,
+ "305": 964511808.0,
+ "310": 940348864.0,
+ "315": 967400384.0,
+ "320": 969008576.0,
+ "325": 980561216.0,
+ "330": 972092288.0,
+ "335": 946870592.0,
+ "340": 966600576.0,
+ "345": 973021056.0,
+ "350": 973923136.0,
+ "355": 963257600.0,
+ "360": 948344576.0,
+ "365": 964819904.0,
+ "370": 962946624.0,
+ "375": 958442880.0,
+ "380": 947151296.0,
+ "385": 955998848.0,
+ "390": 945404416.0,
+ "395": 970418944.0,
+ "400": 979779136.0,
+ "405": 968349248.0,
+ "410": 970067008.0,
+ "415": 953161536.0,
+ "420": 943564992.0,
+ "425": 954775168.0,
+ "430": 962666112.0,
+ "435": 977082048.0,
+ "440": 954810112.0,
+ "445": 971891200.0,
+ "450": 963511232.0,
+ "455": 973135936.0,
+ "460": 983716480.0,
+ "465": 945283008.0,
+ "470": 942053952.0,
+ "475": 967004544.0,
+ "480": 966101760.0,
+ "485": 976409792.0,
+ "490": 962541952.0,
+ "495": 945466304.0,
+ "500": 964461504.0,
+ "505": 986015680.0,
+ "510": 965678144.0,
+ "515": 943414080.0,
+ "520": 945023552.0,
+ "525": 971265600.0,
+ "530": 971890944.0,
+ "535": 979139904.0,
+ "540": 969535488.0,
+ "545": 954128896.0,
+ "550": 951268864.0,
+ "555": 987222080.0,
+ "560": 960432128.0,
+ "565": 966618752.0,
+ "570": 975727808.0,
+ "575": 927225344.0,
+ "580": 970699200.0,
+ "585": 961180864.0,
+ "590": 972971072.0,
+ "595": 963684608.0,
+ "600": 937080064.0,
+ "605": 951475072.0,
+ "610": 963360768.0,
+ "615": 970011584.0,
+ "620": 976474304.0,
+ "625": 949584512.0,
+ "630": 954446592.0,
+ "635": 986046080.0,
+ "640": 980983168.0,
+ "645": 955012928.0,
+ "650": 958553792.0,
+ "655": 951656512.0,
+ "660": 961046016.0,
+ "665": 967560320.0,
+ "670": 962522496.0,
+ "675": 968339520.0,
+ "680": 965623104.0,
+ "685": 962871360.0,
+ "690": 961923584.0,
+ "695": 954770176.0,
+ "700": 970341312.0,
+ "705": 945518784.0,
+ "710": 943886144.0,
+ "715": 973361344.0,
+ "720": 968367616.0,
+ "725": 978494976.0,
+ "730": 952192512.0,
+ "735": 948816064.0,
+ "740": 955635584.0,
+ "745": 975871552.0,
+ "750": 981237760.0,
+ "755": 962162368.0,
+ "760": 951962368.0,
+ "765": 967347712.0,
+ "770": 976154176.0,
+ "775": 970551040.0,
+ "780": 977544192.0,
+ "785": 931529728.0,
+ "790": 960442048.0,
+ "795": 964582592.0,
+ "800": 967021440.0,
+ "805": 962323648.0,
+ "810": 940978496.0,
+ "815": 949040832.0,
+ "820": 953188160.0,
+ "825": 954505152.0,
+ "830": 976443904.0,
+ "835": 956075968.0,
+ "840": 948409856.0,
+ "845": 965160128.0,
+ "850": 966029504.0,
+ "855": 960906944.0,
+ "860": 976028736.0,
+ "865": 938162112.0,
+ "870": 966417792.0,
+ "875": 972318848.0,
+ "880": 963125504.0,
+ "885": 967747776.0,
+ "890": 949973248.0,
+ "895": 960018048.0,
+ "900": 974236608.0,
+ "905": 963970048.0,
+ "910": 958435200.0,
+ "915": 956354752.0,
+ "920": 943976576.0,
+ "925": 960833920.0,
+ "930": 978850304.0,
+ "935": 971075776.0,
+ "940": 960910144.0,
+ "945": 945065728.0,
+ "950": 957428160.0,
+ "955": 979041664.0,
+ "960": 983593216.0,
+ "965": 966167488.0,
+ "970": 951231552.0,
+ "975": 961577728.0,
+ "980": 968070720.0,
+ "985": 968992768.0,
+ "990": 984393024.0,
+ "995": 953293440.0,
+ "1000": 934779904.0,
+ "1005": 960148352.0,
+ "1010": 971540288.0,
+ "1015": 985186304.0,
+ "1020": 962782912.0,
+ "1025": 935013120.0,
+ "1030": 974679936.0,
+ "1035": 964995008.0,
+ "1040": 980467776.0,
+ "1045": 960829376.0,
+ "1050": 955202368.0,
+ "1055": 957782592.0,
+ "1060": 967751808.0,
+ "1065": 967118208.0,
+ "1070": 966603264.0,
+ "1075": 950063424.0,
+ "1080": 954511424.0,
+ "1085": 967251008.0,
+ "1090": 977138048.0,
+ "1095": 961238464.0,
+ "1100": 979612672.0,
+ "1105": 953365568.0,
+ "1110": 965954240.0,
+ "1115": 966987456.0,
+ "1120": 970352640.0,
+ "1125": 965710784.0,
+ "1130": 954943808.0,
+ "1135": 965844160.0,
+ "1140": 965175104.0,
+ "1145": 970989056.0,
+ "1150": 955558656.0,
+ "1155": 930577344.0,
+ "1160": 957775040.0,
+ "1165": 978124608.0,
+ "1170": 974303296.0,
+ "1175": 973060928.0,
+ "1180": 973080320.0,
+ "1185": 947347520.0,
+ "1190": 964794304.0,
+ "1195": 953141632.0,
+ "1200": 972847232.0,
+ "1205": 988483584.0,
+ "1210": 931126400.0,
+ "1215": 968644928.0,
+ "1220": 969165504.0,
+ "1225": 975950912.0,
+ "1230": 967337728.0,
+ "1235": 943447168.0,
+ "1240": 955857920.0,
+ "1245": 981507520.0,
+ "1250": 966114240.0,
+ "1255": 973674176.0,
+ "1260": 946497152.0,
+ "1265": 963998336.0,
+ "1270": 960488896.0,
+ "1275": 973616576.0,
+ "1280": 961115904.0,
+ "1285": 957585216.0,
+ "1290": 952528384.0,
+ "1295": 971612736.0,
+ "1300": 968865216.0,
+ "1305": 963741120.0,
+ "1310": 963336384.0,
+ "1315": 943555136.0,
+ "1320": 966307456.0,
+ "1325": 989785088.0,
+ "1330": 969508864.0,
+ "1335": 972301312.0,
+ "1340": 972270592.0,
+ "1345": 960659136.0,
+ "1350": 968641536.0,
+ "1355": 955854144.0,
+ "1360": 971822784.0,
+ "1365": 960388032.0,
+ "1370": 948792640.0,
+ "1375": 973534080.0,
+ "1380": 953469888.0,
+ "1385": 969149696.0,
+ "1390": 975719552.0,
+ "1395": 931676416.0,
+ "1400": 945855808.0,
+ "1405": 976752576.0,
+ "1410": 974511040.0,
+ "1415": 967572800.0,
+ "1420": 966748096.0,
+ "1425": 937379392.0,
+ "1430": 973914176.0,
+ "1435": 978336320.0,
+ "1440": 964179712.0,
+ "1445": 958056512.0,
+ "1450": 946147584.0,
+ "1455": 983922368.0,
+ "1460": 968652928.0,
+ "1465": 948746496.0,
+ "1470": 984244032.0,
+ "1475": 943905600.0,
+ "1480": 963975488.0,
+ "1485": 957348928.0,
+ "1490": 961262592.0,
+ "1495": 980541184.0,
+ "1500": 958332800.0,
+ "1505": 942868800.0,
+ "1510": 984180352.0,
+ "1515": 959092928.0,
+ "1520": 959104896.0,
+ "1525": 952785152.0,
+ "1530": 957745792.0,
+ "1535": 949430720.0,
+ "1540": 971088768.0,
+ "1545": 963134464.0,
+ "1550": 978667072.0,
+ "1555": 952319808.0,
+ "1560": 980089344.0,
+ "1565": 967318144.0,
+ "1570": 973843520.0,
+ "1575": 975494400.0,
+ "1580": 941863296.0,
+ "1585": 970028736.0,
+ "1590": 983821504.0,
+ "1595": 948633600.0,
+ "1600": 967447104.0,
+ "1605": 952451840.0,
+ "1610": 969619840.0,
+ "1615": 983147136.0,
+ "1620": 968020800.0,
+ "1625": 970717568.0,
+ "1630": 962888000.0,
+ "1635": 942312064.0,
+ "1640": 981610368.0,
+ "1645": 973979456.0,
+ "1650": 974182080.0,
+ "1655": 967266112.0,
+ "1660": 940687616.0,
+ "1665": 961702720.0,
+ "1670": 962901440.0,
+ "1675": 971280704.0,
+ "1680": 980880064.0,
+ "1685": 944417472.0,
+ "1690": 964689344.0,
+ "1695": 965643264.0,
+ "1700": 966343424.0,
+ "1705": 985202240.0,
+ "1710": 978352384.0,
+ "1715": 943212864.0,
+ "1720": 977088768.0,
+ "1725": 965873088.0,
+ "1730": 968971904.0,
+ "1735": 965088192.0,
+ "1740": 949713280.0,
+ "1745": 970010048.0,
+ "1750": 959683008.0,
+ "1755": 960087424.0,
+ "1760": 966380160.0,
+ "1765": 951815360.0,
+ "1770": 954665536.0,
+ "1775": 973750656.0,
+ "1780": 970533952.0,
+ "1785": 968824832.0,
+ "1790": 950235328.0,
+ "1795": 945133568.0,
+ "1800": 984667200.0,
+ "1805": 987163392.0,
+ "1810": 977767744.0,
+ "1815": 948005376.0,
+ "1820": 949208704.0,
+ "1825": 978855168.0,
+ "1830": 966361984.0,
+ "1835": 964133568.0,
+ "1840": 972318272.0,
+ "1845": 935416064.0,
+ "1850": 952499712.0,
+ "1855": 980051136.0,
+ "1860": 975869696.0,
+ "1865": 958966336.0,
+ "1870": 958947648.0,
+ "1875": 932594688.0,
+ "1880": 973575936.0,
+ "1885": 978846272.0,
+ "1890": 971360448.0,
+ "1895": 959212416.0,
+ "1900": 947394496.0,
+ "1905": 981830720.0,
+ "1910": 969124480.0,
+ "1915": 970041280.0,
+ "1920": 975598592.0,
+ "1925": 960495360.0,
+ "1930": 977921984.0,
+ "1935": 963249792.0,
+ "1940": 952459264.0,
+ "1945": 981337728.0,
+ "1950": 939172864.0,
+ "1955": 960605120.0,
+ "1960": 970032384.0,
+ "1965": 981176640.0,
+ "1970": 962044928.0,
+ "1975": 952823552.0,
+ "1980": 936849344.0,
+ "1985": 975937600.0,
+ "1990": 965965824.0,
+ "1995": 962611456.0,
+ "2000": 960554432.0,
+ "2005": 954497664.0,
+ "2010": 975581632.0,
+ "2015": 991804288.0,
+ "2020": 975434496.0,
+ "2025": 974303168.0,
+ "2030": 952084352.0,
+ "2035": 967846400.0,
+ "2040": 987459776.0,
+ "2045": 976479808.0,
+ "2050": 984701376.0,
+ "2055": 942838080.0,
+ "2060": 942594816.0,
+ "2065": 966208896.0,
+ "2070": 969623552.0,
+ "2075": 980553664.0,
+ "2080": 977598528.0,
+ "2085": 939639040.0,
+ "2090": 969873088.0,
+ "2095": 961277824.0,
+ "2100": 976718912.0,
+ "2105": 972536000.0,
+ "2110": 959902720.0,
+ "2115": 956880896.0,
+ "2120": 977481088.0,
+ "2125": 962567168.0,
+ "2130": 979619200.0,
+ "2135": 950537472.0,
+ "2140": 946994944.0,
+ "2145": 962275776.0,
+ "2150": 973404800.0,
+ "2155": 972691712.0,
+ "2160": 970314560.0,
+ "2165": 948642688.0,
+ "2170": 961543744.0,
+ "2175": 969378496.0,
+ "2180": 969328832.0,
+ "2185": 947448512.0,
+ "2190": 940480576.0,
+ "2195": 986085824.0,
+ "2200": 961860928.0,
+ "2205": 978924032.0,
+ "2210": 964101632.0,
+ "2215": 963501312.0,
+ "2220": 951310976.0,
+ "2225": 969316288.0,
+ "2230": 976331264.0,
+ "2235": 974027648.0,
+ "2240": 975494848.0,
+ "2245": 960232448.0,
+ "2250": 967638912.0,
+ "2255": 969133696.0,
+ "2260": 975065152.0,
+ "2265": 968258688.0,
+ "2270": 951744000.0,
+ "2275": 962769088.0,
+ "2280": 969640896.0,
+ "2285": 971692928.0,
+ "2290": 962891712.0,
+ "2295": 931408128.0,
+ "2300": 959906432.0,
+ "2305": 970424000.0,
+ "2310": 967444800.0,
+ "2315": 970905600.0,
+ "2320": 975589568.0,
+ "2325": 938586112.0,
+ "2330": 988437312.0,
+ "2335": 977487872.0,
+ "2340": 964596544.0,
+ "2345": 964166976.0,
+ "2350": 947555008.0,
+ "2355": 977030720.0,
+ "2360": 966899968.0,
+ "2365": 977296704.0,
+ "2370": 965071488.0,
+ "2375": 953964864.0,
+ "2380": 962917504.0,
+ "2385": 967194624.0,
+ "2390": 963075712.0,
+ "2395": 974465728.0,
+ "2400": 958409856.0,
+ "2405": 968120768.0,
+ "2410": 951586368.0,
+ "2415": 965904640.0,
+ "2420": 966518592.0,
+ "2425": 959044096.0,
+ "2430": 956685632.0,
+ "2435": 961387072.0,
+ "2440": 959755712.0,
+ "2445": 970891200.0,
+ "2450": 961997184.0,
+ "2455": 922721472.0,
+ "2460": 951953600.0,
+ "2465": 955730176.0,
+ "2470": 972569600.0,
+ "2475": 973811968.0,
+ "2480": 943893696.0,
+ "2485": 944184576.0,
+ "2490": 972411904.0,
+ "2495": 974451840.0,
+ "2500": 973910720.0,
+ "2505": 958492032.0,
+ "2510": 939506176.0,
+ "2515": 979553408.0,
+ "2520": 970471232.0,
+ "2525": 964389504.0,
+ "2530": 955799232.0,
+ "2535": 936597056.0,
+ "2540": 969026240.0,
+ "2545": 970384896.0,
+ "2550": 969461824.0,
+ "2555": 969440320.0,
+ "2560": 964978752.0,
+ "2565": 959765632.0,
+ "2570": 985176512.0,
+ "2575": 957423872.0,
+ "2580": 967424448.0,
+ "2585": 966024128.0,
+ "2590": 956354176.0,
+ "2595": 981829504.0,
+ "2600": 959530816.0,
+ "2605": 963000064.0,
+ "2610": 965972672.0,
+ "2615": 951924608.0,
+ "2620": 971241600.0,
+ "2625": 976454720.0,
+ "2630": 974410240.0,
+ "2635": 948070336.0,
+ "2640": 948137152.0,
+ "2645": 963037632.0,
+ "2650": 953983616.0,
+ "2655": 977112000.0,
+ "2660": 949621312.0,
+ "2665": 953928768.0,
+ "2670": 959061312.0,
+ "2675": 979275520.0,
+ "2680": 961395776.0,
+ "2685": 970699392.0,
+ "2690": 965219712.0,
+ "2695": 943553472.0,
+ "2700": 969424704.0,
+ "2705": 978960128.0,
+ "2710": 971810880.0,
+ "2715": 990814208.0,
+ "2720": 942648320.0,
+ "2725": 967955200.0,
+ "2730": 955465088.0,
+ "2735": 970675456.0,
+ "2740": 977921472.0,
+ "2745": 932278016.0,
+ "2750": 947856704.0,
+ "2755": 956316160.0,
+ "2760": 981697344.0,
+ "2765": 966111552.0,
+ "2770": 948914240.0,
+ "2775": 935832064.0,
+ "2780": 964779008.0,
+ "2785": 969569024.0,
+ "2790": 974273344.0,
+ "2795": 966886080.0,
+ "2800": 944388800.0,
+ "2805": 964353280.0,
+ "2810": 969610560.0,
+ "2815": 975843712.0,
+ "2820": 963085120.0,
+ "2825": 937628928.0,
+ "2830": 956740800.0,
+ "2835": 986321856.0,
+ "2840": 961758592.0,
+ "2845": 967507136.0,
+ "2850": 951716928.0,
+ "2855": 962093312.0,
+ "2860": 954243328.0,
+ "2865": 955882624.0,
+ "2870": 944663104.0,
+ "2875": 974666176.0,
+ "2880": 968202880.0,
+ "2885": 981082112.0,
+ "2890": 953454720.0,
+ "2895": 957178432.0,
+ "2900": 964989696.0,
+ "2905": 931707328.0,
+ "2910": 955731136.0,
+ "2915": 979476608.0,
+ "2920": 970492352.0,
+ "2925": 964976512.0,
+ "2930": 964048000.0,
+ "2935": 940141248.0,
+ "2940": 964910592.0,
+ "2945": 989150720.0,
+ "2950": 965209344.0,
+ "2955": 965104128.0,
+ "2960": 933161216.0,
+ "2965": 968792896.0,
+ "2970": 973033792.0,
+ "2975": 958093376.0,
+ "2980": 964496064.0,
+ "2985": 937269504.0,
+ "2990": 951254272.0,
+ "2995": 978315584.0,
+ "3000": 969275840.0,
+ "3005": 974686656.0,
+ "3010": 950234432.0,
+ "3015": 943841920.0,
+ "3020": 958440064.0,
+ "3025": 975183488.0,
+ "3030": 965017472.0,
+ "3035": 963453120.0,
+ "3040": 952134208.0,
+ "3045": 989792448.0,
+ "3050": 965544448.0,
+ "3055": 982520448.0,
+ "3060": 971226688.0,
+ "3065": 943915264.0,
+ "3070": 978404096.0,
+ "3075": 975203264.0,
+ "3080": 960992192.0,
+ "3085": 962351424.0,
+ "3090": 945950912.0,
+ "3095": 938114880.0,
+ "3100": 972928192.0,
+ "3105": 961988992.0,
+ "3110": 970655424.0,
+ "3115": 963389504.0,
+ "3120": 947128768.0,
+ "3125": 972720192.0,
+ "3130": 952973376.0,
+ "3135": 966052480.0,
+ "3140": 968489152.0,
+ "3145": 937863808.0,
+ "3150": 975022016.0,
+ "3155": 976827328.0,
+ "3160": 969638848.0,
+ "3165": 982208128.0,
+ "3170": 937973824.0,
+ "3175": 953837824.0,
+ "3180": 983822016.0,
+ "3185": 965180032.0,
+ "3190": 968492992.0,
+ "3195": 950932608.0,
+ "3200": 945101888.0,
+ "3205": 959863168.0,
+ "3210": 957499136.0,
+ "3215": 958020096.0,
+ "3220": 968141504.0,
+ "3225": 935613824.0,
+ "3230": 962601024.0,
+ "3235": 975775616.0,
+ "3240": 962621248.0,
+ "3245": 981274432.0,
+ "3250": 943259840.0,
+ "3255": 954598016.0,
+ "3260": 980361792.0,
+ "3265": 963619776.0,
+ "3270": 965163968.0,
+ "3275": 959731584.0,
+ "3280": 967045376.0,
+ "3285": 982476544.0,
+ "3290": 947688192.0,
+ "3295": 966433984.0,
+ "3300": 959165312.0,
+ "3305": 949143104.0,
+ "3310": 979521984.0,
+ "3315": 964281984.0,
+ "3320": 969219584.0,
+ "3325": 956191744.0,
+ "3330": 941167296.0,
+ "3335": 964973440.0,
+ "3340": 956900864.0,
+ "3345": 972513408.0,
+ "3350": 964588608.0,
+ "3355": 943345984.0,
+ "3360": 970050432.0,
+ "3365": 969466368.0,
+ "3370": 954773824.0,
+ "3375": 958677248.0,
+ "3380": 971463488.0,
+ "3385": 947984960.0,
+ "3390": 965793216.0,
+ "3395": 978403904.0,
+ "3400": 978127360.0,
+ "3405": 976723392.0,
+ "3410": 924192960.0,
+ "3415": 955436608.0,
+ "3420": 971820160.0,
+ "3425": 977168064.0,
+ "3430": 973838272.0,
+ "3435": 936070976.0,
+ "3440": 970516352.0,
+ "3445": 957316160.0,
+ "3450": 959853696.0,
+ "3455": 963864832.0,
+ "3460": 967882432.0,
+ "3465": 931319424.0,
+ "3470": 952348032.0,
+ "3475": 973715840.0,
+ "3480": 959751232.0,
+ "3485": 979958976.0,
+ "3490": 944683072.0,
+ "3495": 953916800.0,
+ "3500": 969328960.0,
+ "3505": 964360256.0,
+ "3510": 971233984.0,
+ "3515": 955946048.0,
+ "3520": 958745536.0,
+ "3525": 971913920.0,
+ "3530": 964149568.0,
+ "3535": 983190912.0,
+ "3540": 937501696.0,
+ "3545": 944730624.0,
+ "3550": 984461952.0,
+ "3555": 978069632.0,
+ "3560": 974385792.0,
+ "3565": 968812480.0,
+ "3570": 946694976.0,
+ "3575": 976105408.0,
+ "3580": 977492672.0,
+ "3585": 954568320.0,
+ "3590": 956426432.0,
+ "3595": 951442432.0,
+ "3600": 989021440.0,
+ "3605": 962032256.0,
+ "3610": 965078784.0,
+ "3615": 974659840.0,
+ "3620": 954900608.0,
+ "3625": 939539776.0,
+ "3630": 990173952.0,
+ "3635": 971447552.0,
+ "3640": 976036160.0,
+ "3645": 961500352.0,
+ "3650": 945814912.0,
+ "3655": 965794688.0,
+ "3660": 976218944.0,
+ "3665": 964044096.0,
+ "3670": 977455488.0,
+ "3675": 943500288.0,
+ "3680": 958197248.0,
+ "3685": 964322368.0,
+ "3690": 982116352.0,
+ "3695": 963135872.0,
+ "3700": 950594112.0,
+ "3705": 947356928.0,
+ "3710": 982367424.0,
+ "3715": 972693184.0,
+ "3720": 976162880.0,
+ "3725": 964049664.0,
+ "3730": 948886912.0,
+ "3735": 967103040.0,
+ "3740": 960971584.0,
+ "3745": 969346752.0,
+ "3750": 963960384.0,
+ "3755": 953457280.0,
+ "3760": 976665664.0,
+ "3765": 979865984.0,
+ "3770": 972408448.0,
+ "3775": 972409856.0,
+ "3780": 952609792.0,
+ "3785": 960271808.0,
+ "3790": 985622400.0,
+ "3795": 969232512.0,
+ "3800": 957901184.0,
+ "3805": 972444928.0,
+ "3810": 954543040.0,
+ "3815": 974602816.0,
+ "3820": 963042944.0,
+ "3825": 962095936.0,
+ "3830": 969406976.0,
+ "3835": 934782272.0,
+ "3840": 971305280.0,
+ "3845": 986897088.0,
+ "3850": 968909376.0,
+ "3855": 965296576.0,
+ "3860": 948093120.0,
+ "3865": 975085440.0,
+ "3870": 985135488.0,
+ "3875": 983066688.0,
+ "3880": 963639552.0,
+ "3885": 953048768.0,
+ "3890": 960333312.0,
+ "3895": 960638528.0,
+ "3900": 984961664.0,
+ "3905": 976260992.0,
+ "3910": 987385664.0,
+ "3915": 946040960.0,
+ "3920": 974902976.0,
+ "3925": 961246784.0,
+ "3930": 976813696.0,
+ "3935": 978937344.0,
+ "3940": 950328512.0,
+ "3945": 960297792.0,
+ "3950": 974209216.0,
+ "3955": 973004032.0,
+ "3960": 974091392.0,
+ "3965": 950911104.0,
+ "3970": 980704448.0,
+ "3975": 960761728.0,
+ "3980": 977567296.0,
+ "3985": 962984768.0,
+ "3990": 972778688.0,
+ "3995": 953726016.0,
+ "4000": 975036928.0,
+ "4005": 971682560.0,
+ "4010": 978420736.0,
+ "4015": 971515904.0,
+ "4020": 950322624.0,
+ "4025": 968457920.0,
+ "4030": 997960128.0,
+ "4035": 978585536.0,
+ "4040": 959836352.0,
+ "4045": 939688704.0,
+ "4050": 944754048.0,
+ "4055": 980983936.0,
+ "4060": 977721280.0,
+ "4065": 975754688.0,
+ "4070": 942176640.0,
+ "4075": 945789504.0,
+ "4080": 988777984.0,
+ "4085": 962116096.0,
+ "4090": 983380864.0,
+ "4095": 986958144.0,
+ "4100": 957224128.0,
+ "4105": 954110144.0,
+ "4110": 966512064.0,
+ "4115": 976089664.0,
+ "4120": 983568064.0,
+ "4125": 960091328.0,
+ "4130": 967314368.0,
+ "4135": 971478784.0,
+ "4140": 963194432.0,
+ "4145": 956223232.0,
+ "4150": 960331264.0,
+ "4155": 946229376.0,
+ "4160": 968462272.0,
+ "4165": 970375872.0,
+ "4170": 971996608.0,
+ "4175": 955898688.0,
+ "4180": 940997248.0,
+ "4185": 968330880.0,
+ "4190": 968065536.0,
+ "4195": 989248000.0,
+ "4200": 962714368.0,
+ "4205": 960631936.0,
+ "4210": 971886912.0,
+ "4215": 974218432.0,
+ "4220": 981169152.0,
+ "4225": 975281600.0,
+ "4230": 952752000.0,
+ "4235": 958494592.0,
+ "4240": 966835840.0,
+ "4245": 961769280.0,
+ "4250": 965881792.0,
+ "4255": 958291840.0,
+ "4260": 949594432.0,
+ "4265": 964174656.0,
+ "4270": 978444864.0,
+ "4275": 975458496.0,
+ "4280": 962708672.0,
+ "4285": 951525632.0,
+ "4290": 980078464.0,
+ "4295": 968878976.0,
+ "4300": 958323456.0,
+ "4305": 966863424.0,
+ "4310": 939626304.0,
+ "4315": 949453440.0,
+ "4320": 984584704.0,
+ "4325": 982595840.0,
+ "4330": 974758592.0,
+ "4335": 949495936.0,
+ "4340": 959551360.0,
+ "4345": 956668224.0,
+ "4350": 979996544.0,
+ "4355": 968881728.0,
+ "4360": 966350784.0,
+ "4365": 941257536.0,
+ "4370": 969507648.0,
+ "4375": 973019648.0,
+ "4380": 966172160.0,
+ "4385": 972059840.0,
+ "4390": 954176768.0,
+ "4395": 951881344.0,
+ "4400": 973664960.0,
+ "4405": 972478272.0,
+ "4410": 967944960.0,
+ "4415": 958997824.0,
+ "4420": 960802944.0,
+ "4425": 976476608.0,
+ "4430": 965991232.0,
+ "4435": 975949952.0,
+ "4440": 962355520.0,
+ "4445": 954861760.0,
+ "4450": 978343616.0,
+ "4455": 960255360.0,
+ "4460": 968592576.0,
+ "4465": 968867648.0,
+ "4470": 944164544.0,
+ "4475": 952033600.0,
+ "4480": 978825088.0,
+ "4485": 968327936.0,
+ "4490": 957061568.0,
+ "4495": 938987136.0,
+ "4500": 953277568.0,
+ "4505": 977216384.0,
+ "4510": 978774336.0,
+ "4515": 962517312.0,
+ "4520": 958886784.0,
+ "4525": 958138688.0,
+ "4530": 964440064.0,
+ "4535": 976629952.0,
+ "4540": 976896576.0,
+ "4545": 970207040.0,
+ "4550": 953247680.0,
+ "4555": 959589760.0,
+ "4560": 972558784.0,
+ "4565": 973528896.0,
+ "4570": 978958400.0,
+ "4575": 957859072.0,
+ "4580": 963252672.0,
+ "4585": 957492672.0,
+ "4590": 986605120.0,
+ "4595": 960249792.0,
+ "4600": 952290240.0,
+ "4605": 959323264.0,
+ "4610": 963772992.0,
+ "4615": 958011968.0,
+ "4620": 960280000.0,
+ "4625": 973847424.0,
+ "4630": 944542784.0,
+ "4635": 977037120.0,
+ "4640": 960322432.0,
+ "4645": 982028736.0,
+ "4650": 962448384.0,
+ "4655": 939470080.0,
+ "4660": 964038144.0,
+ "4665": 962602048.0,
+ "4670": 976684032.0,
+ "4675": 963502784.0,
+ "4680": 957512448.0,
+ "4685": 949685760.0,
+ "4690": 957023680.0,
+ "4695": 969999936.0,
+ "4700": 961262720.0,
+ "4705": 970817664.0,
+ "4710": 934542016.0,
+ "4715": 970512448.0,
+ "4720": 966385600.0,
+ "4725": 980377472.0,
+ "4730": 965886784.0,
+ "4735": 937945472.0,
+ "4740": 960173696.0,
+ "4745": 976083008.0,
+ "4750": 968033408.0,
+ "4755": 984996800.0,
+ "4760": 959208960.0,
+ "4765": 955261632.0,
+ "4770": 958682304.0,
+ "4775": 991141824.0,
+ "4780": 976893440.0,
+ "4785": 967557376.0,
+ "4790": 943744320.0,
+ "4795": 955874496.0,
+ "4800": 967758592.0,
+ "4805": 976603392.0,
+ "4810": 965222336.0,
+ "4815": 958017856.0,
+ "4820": 974004736.0,
+ "4825": 961568064.0,
+ "4830": 962649344.0,
+ "4835": 972532288.0,
+ "4840": 948962816.0,
+ "4845": 965753600.0,
+ "4850": 960304768.0,
+ "4855": 964155392.0,
+ "4860": 963088448.0,
+ "4865": 967507520.0,
+ "4870": 957243328.0,
+ "4875": 983586560.0,
+ "4880": 957120896.0,
+ "4885": 977051840.0,
+ "4890": 959728640.0,
+ "4895": 942213312.0,
+ "4900": 973712448.0,
+ "4905": 975263296.0,
+ "4910": 969245184.0,
+ "4915": 970126848.0,
+ "4920": 941157120.0,
+ "4925": 954796928.0,
+ "4930": 977095424.0,
+ "4935": 963799872.0,
+ "4940": 972672192.0,
+ "4945": 960043968.0,
+ "4950": 940819328.0,
+ "4955": 968135488.0,
+ "4960": 976886912.0,
+ "4965": 961008576.0,
+ "4970": 958641152.0,
+ "4975": 933811648.0,
+ "4980": 960811648.0,
+ "4985": 962987584.0,
+ "4990": 963552704.0,
+ "4995": 986332416.0,
+ "5000": 940726912.0,
+ "5005": 968911744.0,
+ "5010": 970296000.0,
+ "5015": 965281536.0,
+ "5020": 966727808.0,
+ "5025": 949447936.0,
+ "5030": 953528000.0,
+ "5035": 967340992.0,
+ "5040": 955750528.0,
+ "5045": 969178432.0,
+ "5050": 953416960.0,
+ "5055": 954847552.0,
+ "5060": 963016640.0,
+ "5065": 952219520.0,
+ "5070": 973596928.0,
+ "5075": 978580288.0,
+ "5080": 942915840.0,
+ "5085": 965863168.0,
+ "5090": 972954688.0,
+ "5095": 964495680.0,
+ "5100": 958391744.0,
+ "5105": 965378432.0,
+ "5110": 950450304.0,
+ "5115": 972383744.0,
+ "5120": 960498688.0,
+ "5125": 969818112.0,
+ "5130": 938856960.0,
+ "5135": 943769728.0,
+ "5140": 969998720.0,
+ "5145": 968688320.0,
+ "5150": 970638016.0,
+ "5155": 972684352.0,
+ "5160": 926600576.0,
+ "5165": 961653824.0,
+ "5170": 966886592.0,
+ "5175": 966135424.0,
+ "5180": 963778176.0,
+ "5185": 930925824.0,
+ "5190": 949877312.0,
+ "5195": 972509376.0,
+ "5200": 973820480.0,
+ "5205": 968226240.0,
+ "5210": 960588800.0,
+ "5215": 928828352.0,
+ "5220": 979211648.0,
+ "5225": 984853376.0,
+ "5230": 975098112.0,
+ "5235": 975166272.0,
+ "5240": 944424000.0,
+ "5245": 970895616.0,
+ "5250": 972524992.0,
+ "5255": 967043968.0,
+ "5260": 976828224.0,
+ "5265": 942295872.0,
+ "5270": 969262976.0,
+ "5275": 970172224.0,
+ "5280": 962924480.0,
+ "5285": 964121920.0,
+ "5290": 932636480.0,
+ "5295": 951861696.0,
+ "5300": 975661696.0,
+ "5305": 951977344.0,
+ "5310": 968126208.0,
+ "5315": 955854912.0,
+ "5320": 951044352.0,
+ "5325": 973089472.0,
+ "5330": 967843968.0,
+ "5335": 967658560.0,
+ "5340": 966540608.0,
+ "5345": 963034432.0,
+ "5350": 978924928.0,
+ "5355": 972276800.0,
+ "5360": 963953792.0,
+ "5365": 965295616.0,
+ "5370": 947885760.0,
+ "5375": 948857408.0,
+ "5380": 967261056.0,
+ "5385": 980552896.0,
+ "5390": 965374656.0,
+ "5395": 955270720.0,
+ "5400": 948336320.0,
+ "5405": 974415488.0,
+ "5410": 967942912.0,
+ "5415": 976186048.0,
+ "5420": 967567296.0,
+ "5425": 937473920.0,
+ "5430": 963955776.0,
+ "5435": 971935552.0,
+ "5440": 969176448.0,
+ "5445": 957549888.0,
+ "5450": 919479488.0,
+ "5455": 952137920.0,
+ "5460": 962474048.0,
+ "5465": 978991744.0,
+ "5470": 981105024.0,
+ "5475": 941709440.0,
+ "5480": 955890624.0,
+ "5485": 964975616.0,
+ "5490": 976048192.0,
+ "5495": 962842752.0,
+ "5500": 971242368.0,
+ "5505": 957038208.0,
+ "5510": 968672256.0,
+ "5515": 945577600.0,
+ "5520": 963244224.0,
+ "5525": 976030976.0,
+ "5530": 936696128.0,
+ "5535": 970744320.0,
+ "5540": 960467968.0,
+ "5545": 972048640.0,
+ "5550": 967934400.0,
+ "5555": 956030528.0,
+ "5560": 954643968.0,
+ "5565": 968900736.0,
+ "5570": 945284928.0,
+ "5575": 960662528.0,
+ "5580": 960723584.0,
+ "5585": 959679296.0,
+ "5590": 977815808.0,
+ "5595": 975294272.0,
+ "5600": 963034688.0,
+ "5605": 964226176.0,
+ "5610": 943317056.0,
+ "5615": 966761792.0,
+ "5620": 963470272.0,
+ "5625": 982496320.0,
+ "5630": 976173696.0,
+ "5635": 957421824.0,
+ "5640": 951529728.0,
+ "5645": 967940672.0,
+ "5650": 979305152.0,
+ "5655": 983511104.0,
+ "5660": 956505792.0,
+ "5665": 953573440.0,
+ "5670": 966159360.0,
+ "5675": 967726848.0,
+ "5680": 978733632.0,
+ "5685": 962265728.0,
+ "5690": 935982144.0,
+ "5695": 963812800.0,
+ "5700": 952574784.0,
+ "5705": 974573952.0,
+ "5710": 971366464.0,
+ "5715": 946183872.0,
+ "5720": 975249536.0,
+ "5725": 967683328.0,
+ "5730": 978697088.0,
+ "5735": 964979776.0,
+ "5740": 943771008.0,
+ "5745": 971302656.0,
+ "5750": 981938752.0,
+ "5755": 956827840.0,
+ "5760": 963793216.0,
+ "5765": 957817344.0,
+ "5770": 955846912.0,
+ "5775": 970907584.0,
+ "5780": 962910272.0,
+ "5785": 970846912.0,
+ "5790": 974812096.0,
+ "5795": 949986752.0,
+ "5800": 966024000.0,
+ "5805": 968910016.0,
+ "5810": 976257088.0,
+ "5815": 970212928.0,
+ "5820": 936525120.0,
+ "5825": 969462848.0,
+ "5830": 977847744.0,
+ "5835": 975127104.0,
+ "5840": 963246912.0,
+ "5845": 968886464.0,
+ "5850": 943217536.0,
+ "5855": 976109632.0,
+ "5860": 979697024.0,
+ "5865": 978493312.0,
+ "5870": 969035584.0,
+ "5875": 942328512.0,
+ "5880": 964645184.0,
+ "5885": 974978304.0,
+ "5890": 972782528.0,
+ "5895": 965742272.0,
+ "5900": 941323200.0,
+ "5905": 961926272.0,
+ "5910": 958679232.0,
+ "5915": 968394880.0,
+ "5920": 977357184.0,
+ "5925": 959603648.0,
+ "5930": 946782592.0,
+ "5935": 952536448.0,
+ "5940": 977672320.0,
+ "5945": 984805440.0,
+ "5950": 980811648.0,
+ "5955": 935093120.0,
+ "5960": 961796544.0,
+ "5965": 965925568.0,
+ "5970": 970723584.0,
+ "5975": 961994688.0,
+ "5980": 958246336.0,
+ "5985": 964747136.0,
+ "5990": 973615680.0,
+ "5995": 955981184.0,
+ "6000": 955657408.0,
+ "6005": 961391744.0,
+ "6010": 952842624.0,
+ "6015": 974538304.0,
+ "6020": 978425088.0,
+ "6025": 972251520.0,
+ "6030": 955397632.0,
+ "6035": 946990912.0,
+ "6040": 962666624.0,
+ "6045": 983866368.0,
+ "6050": 956613312.0,
+ "6055": 963544384.0,
+ "6060": 945853376.0,
+ "6065": 958468352.0,
+ "6070": 978476992.0,
+ "6075": 978099392.0,
+ "6080": 957580032.0,
+ "6085": 947629952.0,
+ "6090": 953756864.0,
+ "6095": 964684416.0,
+ "6100": 979986240.0,
+ "6105": 971232960.0,
+ "6110": 961901760.0,
+ "6115": 943888064.0,
+ "6120": 968721600.0,
+ "6125": 960715392.0,
+ "6130": 984039872.0,
+ "6135": 961130112.0,
+ "6140": 958892800.0,
+ "6145": 971373376.0,
+ "6150": 968702208.0,
+ "6155": 975126592.0,
+ "6160": 977339200.0,
+ "6165": 952867264.0,
+ "6170": 951228544.0,
+ "6175": 963519104.0,
+ "6180": 969550528.0,
+ "6185": 966491968.0,
+ "6190": 963780352.0,
+ "6195": 947291328.0,
+ "6200": 969574720.0,
+ "6205": 967247872.0,
+ "6210": 959437568.0,
+ "6215": 973270144.0,
+ "6220": 936559552.0,
+ "6225": 978935936.0,
+ "6230": 976292864.0,
+ "6235": 971864256.0,
+ "6240": 966338176.0,
+ "6245": 956215040.0,
+ "6250": 956610304.0,
+ "6255": 973375424.0,
+ "6260": 979036928.0,
+ "6265": 975261248.0,
+ "6270": 958978432.0,
+ "6275": 963970880.0,
+ "6280": 973386048.0,
+ "6285": 966258304.0,
+ "6290": 971150720.0,
+ "6295": 987576320.0,
+ "6300": 947701696.0,
+ "6305": 965041344.0,
+ "6310": 979398464.0,
+ "6315": 978688448.0,
+ "6320": 971932800.0,
+ "6325": 923113408.0,
+ "6330": 959514880.0,
+ "6335": 975051584.0,
+ "6340": 985315712.0,
+ "6345": 967045312.0,
+ "6350": 944820736.0,
+ "6355": 958321984.0,
+ "6360": 973018880.0,
+ "6365": 972404736.0,
+ "6370": 959630848.0,
+ "6375": 967485888.0,
+ "6380": 951362560.0,
+ "6385": 973794304.0,
+ "6390": 965735168.0,
+ "6395": 975303168.0,
+ "6400": 984314880.0,
+ "6405": 944422144.0,
+ "6410": 977732160.0,
+ "6415": 971686144.0,
+ "6420": 956788672.0,
+ "6425": 961306944.0,
+ "6430": 957955392.0,
+ "6435": 960505792.0,
+ "6440": 969020736.0,
+ "6445": 973872960.0,
+ "6450": 974612672.0,
+ "6455": 962486912.0,
+ "6460": 941388416.0,
+ "6465": 974655232.0,
+ "6470": 980206848.0,
+ "6475": 961267648.0,
+ "6480": 967865536.0,
+ "6485": 949004032.0,
+ "6490": 971191744.0,
+ "6495": 988602816.0,
+ "6500": 980802176.0,
+ "6505": 972330112.0,
+ "6510": 951785472.0,
+ "6515": 958069824.0,
+ "6520": 979073536.0,
+ "6525": 979088768.0,
+ "6530": 973617152.0,
+ "6535": 968222464.0,
+ "6540": 950516544.0,
+ "6545": 966441216.0,
+ "6550": 979716160.0,
+ "6555": 967439040.0,
+ "6560": 975563136.0,
+ "6565": 949855168.0,
+ "6570": 952179392.0,
+ "6575": 962750912.0,
+ "6580": 976233664.0,
+ "6585": 979741568.0,
+ "6590": 949495808.0,
+ "6595": 961951488.0,
+ "6600": 961342144.0,
+ "6605": 961682496.0,
+ "6610": 985247936.0,
+ "6615": 959825984.0,
+ "6620": 944747840.0,
+ "6625": 971176192.0,
+ "6630": 971516352.0,
+ "6635": 964240256.0,
+ "6640": 959857344.0,
+ "6645": 951373696.0,
+ "6650": 979180224.0,
+ "6655": 966146880.0,
+ "6660": 968717120.0,
+ "6665": 969174720.0,
+ "6670": 933046912.0,
+ "6675": 970933056.0,
+ "6680": 969175936.0,
+ "6685": 958931968.0,
+ "6690": 956460480.0,
+ "6695": 955560704.0,
+ "6700": 962393856.0,
+ "6705": 979231488.0,
+ "6710": 971147776.0,
+ "6715": 967157952.0,
+ "6720": 974236864.0,
+ "6725": 941929344.0,
+ "6730": 979473536.0,
+ "6735": 994820544.0,
+ "6740": 976934144.0,
+ "6745": 974799808.0,
+ "6750": 939161920.0,
+ "6755": 977895424.0,
+ "6760": 969918144.0,
+ "6765": 978639552.0,
+ "6770": 975587712.0,
+ "6775": 943585856.0,
+ "6780": 947430528.0,
+ "6785": 975530944.0,
+ "6790": 960889152.0,
+ "6795": 976414464.0,
+ "6800": 973516736.0,
+ "6805": 946867008.0,
+ "6810": 958437888.0,
+ "6815": 970782784.0,
+ "6820": 977999680.0,
+ "6825": 969315520.0,
+ "6830": 950730304.0,
+ "6835": 981673024.0,
+ "6840": 983132672.0,
+ "6845": 948936192.0,
+ "6850": 965809856.0,
+ "6855": 954064128.0,
+ "6860": 979357824.0,
+ "6865": 984061696.0,
+ "6870": 965100352.0,
+ "6875": 978933952.0,
+ "6880": 950666368.0,
+ "6885": 958557376.0,
+ "6890": 960656704.0,
+ "6895": 966243456.0,
+ "6900": 985345984.0,
+ "6905": 968969856.0,
+ "6910": 950196224.0,
+ "6915": 971099264.0,
+ "6920": 967596416.0,
+ "6925": 965513152.0,
+ "6930": 964852416.0,
+ "6935": 952245952.0,
+ "6940": 963241728.0,
+ "6945": 986666560.0,
+ "6950": 973768960.0,
+ "6955": 964924608.0,
+ "6960": 940274688.0,
+ "6965": 975019648.0,
+ "6970": 978641728.0,
+ "6975": 985239040.0,
+ "6980": 982655040.0,
+ "6985": 960294400.0,
+ "6990": 945704448.0,
+ "6995": 987766336.0,
+ "7000": 963317440.0,
+ "7005": 963045760.0,
+ "7010": 985353984.0,
+ "7015": 945669120.0,
+ "7020": 983129344.0,
+ "7025": 968988800.0,
+ "7030": 953562688.0,
+ "7035": 983118720.0,
+ "7040": 950766208.0,
+ "7045": 956286336.0,
+ "7050": 960452992.0,
+ "7055": 963769600.0,
+ "7060": 976814528.0,
+ "7065": 968477184.0,
+ "7070": 953817920.0,
+ "7075": 957039744.0,
+ "7080": 969383040.0,
+ "7085": 966052928.0,
+ "7090": 969731456.0,
+ "7095": 960300288.0,
+ "7100": 973872448.0,
+ "7105": 973209856.0,
+ "7110": 970513728.0,
+ "7115": 958916480.0,
+ "7120": 949518784.0,
+ "7125": 963153856.0,
+ "7130": 971680640.0,
+ "7135": 964676800.0,
+ "7140": 961595904.0,
+ "7145": 931069184.0,
+ "7150": 946833152.0,
+ "7155": 990945280.0,
+ "7160": 968663872.0,
+ "7165": 956926400.0,
+ "7170": 968423552.0,
+ "7175": 955864320.0,
+ "7180": 958331200.0,
+ "7185": 985273280.0,
+ "7190": 979117824.0,
+ "7195": 974032448.0,
+ "7200": 936447232.0,
+ "7205": 958005888.0,
+ "7210": 967372992.0,
+ "7215": 969622400.0,
+ "7220": 982491200.0,
+ "7225": 929305152.0,
+ "7230": 950090048.0,
+ "7235": 967215296.0,
+ "7240": 967251712.0,
+ "7245": 967860032.0,
+ "7250": 950094976.0,
+ "7255": 958040960.0,
+ "7260": 970705600.0,
+ "7265": 975243968.0,
+ "7270": 960156928.0,
+ "7275": 959656320.0,
+ "7280": 957553536.0,
+ "7285": 978387136.0,
+ "7290": 977863936.0,
+ "7295": 963392320.0,
+ "7300": 975868224.0,
+ "7305": 964588352.0,
+ "7310": 977836352.0,
+ "7315": 967215872.0,
+ "7320": 974848256.0,
+ "7325": 967496768.0,
+ "7330": 960097600.0,
+ "7335": 964578688.0,
+ "7340": 978332544.0,
+ "7345": 968062592.0,
+ "7350": 985297856.0,
+ "7355": 960130752.0,
+ "7360": 949274304.0,
+ "7365": 973093760.0,
+ "7370": 982913920.0,
+ "7375": 963555648.0,
+ "7380": 964689792.0,
+ "7385": 949157376.0,
+ "7390": 964574592.0,
+ "7395": 958898240.0,
+ "7400": 970427456.0,
+ "7405": 988104640.0,
+ "7410": 952865920.0,
+ "7415": 950934528.0,
+ "7420": 967226752.0,
+ "7425": 983172800.0,
+ "7430": 966265728.0,
+ "7435": 973491840.0,
+ "7440": 937683712.0,
+ "7445": 969259264.0,
+ "7450": 980769024.0,
+ "7455": 971858368.0,
+ "7460": 972529408.0,
+ "7465": 940188544.0,
+ "7470": 972282496.0,
+ "7475": 958389696.0,
+ "7480": 969505344.0,
+ "7485": 961847040.0,
+ "7490": 934846592.0,
+ "7495": 957507200.0,
+ "7500": 969769152.0,
+ "7505": 970161536.0,
+ "7510": 972406464.0,
+ "7515": 980210048.0,
+ "7520": 952473600.0,
+ "7525": 970801600.0,
+ "7530": 954925952.0,
+ "7535": 971567936.0,
+ "7540": 979904512.0,
+ "7545": 959591808.0,
+ "7550": 961140800.0,
+ "7555": 960895552.0,
+ "7560": 970885632.0,
+ "7565": 955157248.0,
+ "7570": 943331520.0,
+ "7575": 966109824.0,
+ "7580": 982409792.0,
+ "7585": 979290048.0,
+ "7590": 970979008.0,
+ "7595": 950336384.0,
+ "7600": 946411712.0,
+ "7605": 982649152.0,
+ "7610": 969955072.0,
+ "7615": 989327744.0,
+ "7620": 957356672.0,
+ "7625": 941577536.0,
+ "7630": 971881984.0,
+ "7635": 984658624.0,
+ "7640": 983926016.0,
+ "7645": 968503552.0,
+ "7650": 959528896.0,
+ "7655": 962977792.0,
+ "7660": 969519488.0,
+ "7665": 978576896.0,
+ "7670": 975844800.0,
+ "7675": 976001216.0,
+ "7680": 942945920.0,
+ "7685": 960549504.0,
+ "7690": 975622592.0,
+ "7695": 982725568.0,
+ "7700": 980058816.0,
+ "7705": 941456384.0,
+ "7710": 975279744.0,
+ "7715": 979628288.0,
+ "7720": 968339712.0,
+ "7725": 960700608.0,
+ "7730": 943977216.0,
+ "7735": 968476032.0,
+ "7740": 980832192.0,
+ "7745": 964593600.0,
+ "7750": 964505920.0,
+ "7755": 959959936.0,
+ "7760": 971035968.0,
+ "7765": 971331968.0,
+ "7770": 962621184.0,
+ "7775": 982228992.0,
+ "7780": 965195904.0,
+ "7785": 959752704.0,
+ "7790": 968591680.0,
+ "7795": 968859456.0,
+ "7800": 972188736.0,
+ "7805": 968762240.0,
+ "7810": 946316224.0,
+ "7815": 964008448.0,
+ "7820": 974652096.0,
+ "7825": 963857664.0,
+ "7830": 957421888.0,
+ "7835": 950374080.0,
+ "7840": 957441088.0,
+ "7845": 954201344.0,
+ "7850": 980128000.0,
+ "7855": 986990592.0,
+ "7860": 947953088.0,
+ "7865": 949598656.0,
+ "7870": 965851456.0,
+ "7875": 976366912.0,
+ "7880": 969056576.0,
+ "7885": 969827328.0,
+ "7890": 952287488.0,
+ "7895": 975297408.0,
+ "7900": 963977472.0,
+ "7905": 964432576.0,
+ "7910": 966139520.0,
+ "7915": 943821248.0,
+ "7920": 951435520.0,
+ "7925": 969931584.0,
+ "7930": 965301248.0,
+ "7935": 984633472.0,
+ "7940": 965187648.0,
+ "7945": 950834240.0,
+ "7950": 962713728.0,
+ "7955": 980597632.0,
+ "7960": 964009728.0,
+ "7965": 953464256.0,
+ "7970": 952121408.0,
+ "7975": 970176768.0,
+ "7980": 965820288.0,
+ "7985": 959510336.0,
+ "7990": 968583680.0,
+ "7995": 947223296.0,
+ "8000": 963087744.0,
+ "8005": 981245184.0,
+ "8010": 966256320.0,
+ "8015": 983078976.0,
+ "8020": 961325504.0,
+ "8025": 965814784.0,
+ "8030": 958815232.0,
+ "8035": 976182784.0,
+ "8040": 961184576.0,
+ "8045": 948370368.0,
+ "8050": 960090304.0,
+ "8055": 979949632.0,
+ "8060": 970317824.0,
+ "8065": 958616896.0,
+ "8070": 964572224.0,
+ "8075": 942979840.0,
+ "8080": 966354688.0,
+ "8085": 967456384.0,
+ "8090": 984068800.0,
+ "8095": 989130304.0,
+ "8100": 967122880.0,
+ "8105": 944880832.0,
+ "8110": 969854208.0,
+ "8115": 986200512.0,
+ "8120": 975353792.0,
+ "8125": 964977664.0,
+ "8130": 966949824.0,
+ "8135": 968265216.0,
+ "8140": 964500160.0,
+ "8145": 995874752.0,
+ "8150": 973817600.0,
+ "8155": 939275776.0,
+ "8160": 965212608.0,
+ "8165": 973688512.0,
+ "8170": 969087680.0,
+ "8175": 962153344.0,
+ "8180": 936964480.0,
+ "8185": 963486208.0,
+ "8190": 968119488.0,
+ "8195": 978061376.0,
+ "8200": 956982272.0,
+ "8205": 960910912.0,
+ "8210": 947123008.0,
+ "8215": 982891392.0,
+ "8220": 989217856.0,
+ "8225": 967091328.0,
+ "8230": 962860608.0,
+ "8235": 934980480.0,
+ "8240": 980883072.0,
+ "8245": 976853696.0,
+ "8250": 964513536.0,
+ "8255": 978243904.0,
+ "8260": 957165888.0,
+ "8265": 983486016.0,
+ "8270": 953168128.0,
+ "8275": 974862144.0,
+ "8280": 975042112.0,
+ "8285": 954023424.0,
+ "8290": 940649664.0,
+ "8295": 981948480.0,
+ "8300": 973790592.0,
+ "8305": 978307968.0,
+ "8310": 951945216.0,
+ "8315": 938364608.0,
+ "8320": 977977344.0,
+ "8325": 968107648.0,
+ "8330": 990904448.0,
+ "8335": 976668800.0,
+ "8340": 948313088.0,
+ "8345": 971268096.0,
+ "8350": 970833216.0,
+ "8355": 975592512.0,
+ "8360": 980242240.0,
+ "8365": 933298112.0,
+ "8370": 965809280.0,
+ "8375": 980285440.0,
+ "8380": 965519296.0,
+ "8385": 972964096.0,
+ "8390": 963276160.0,
+ "8395": 951641792.0,
+ "8400": 972950208.0,
+ "8405": 952006016.0,
+ "8410": 961048640.0,
+ "8415": 966224960.0,
+ "8420": 942073216.0,
+ "8425": 968608576.0,
+ "8430": 961269632.0,
+ "8435": 966016768.0,
+ "8440": 969989952.0,
+ "8445": 953303168.0,
+ "8450": 984719936.0,
+ "8455": 990886144.0,
+ "8460": 969110848.0,
+ "8465": 967810816.0,
+ "8470": 963340288.0,
+ "8475": 943511616.0,
+ "8480": 987741120.0,
+ "8485": 980317376.0,
+ "8490": 992612032.0,
+ "8495": 972245632.0,
+ "8500": 951823360.0,
+ "8505": 983793408.0,
+ "8510": 974002240.0,
+ "8515": 969145216.0,
+ "8520": 962195840.0,
+ "8525": 945413312.0,
+ "8530": 984573440.0,
+ "8535": 978608128.0,
+ "8540": 968331200.0,
+ "8545": 969014720.0,
+ "8550": 942421056.0,
+ "8555": 972137728.0,
+ "8560": 958995008.0,
+ "8565": 976032000.0,
+ "8570": 974998080.0,
+ "8575": 971534528.0,
+ "8580": 932411712.0,
+ "8585": 965795392.0,
+ "8590": 979154688.0,
+ "8595": 979621440.0,
+ "8600": 984124800.0,
+ "8605": 958319616.0,
+ "8610": 984389056.0,
+ "8615": 977724160.0,
+ "8620": 963289728.0,
+ "8625": 979552576.0,
+ "8630": 943285120.0,
+ "8635": 962324608.0,
+ "8640": 973621184.0,
+ "8645": 970942976.0,
+ "8650": 969644480.0,
+ "8655": 971046080.0,
+ "8660": 944863936.0,
+ "8665": 986915840.0,
+ "8670": 960930944.0,
+ "8675": 974807040.0,
+ "8680": 962730432.0,
+ "8685": 955913472.0,
+ "8690": 978796160.0,
+ "8695": 969175104.0,
+ "8700": 973282112.0,
+ "8705": 973926784.0,
+ "8710": 947349248.0,
+ "8715": 973834112.0,
+ "8720": 958732160.0,
+ "8725": 979211392.0,
+ "8730": 986078336.0,
+ "8735": 952288384.0,
+ "8740": 940913152.0,
+ "8745": 987934528.0,
+ "8750": 972318080.0,
+ "8755": 971958080.0,
+ "8760": 965827136.0,
+ "8765": 934961664.0,
+ "8770": 986889536.0,
+ "8775": 969984320.0,
+ "8780": 967712512.0,
+ "8785": 962688384.0,
+ "8790": 947760128.0,
+ "8795": 969675584.0,
+ "8800": 970863680.0,
+ "8805": 973828480.0,
+ "8810": 983933760.0,
+ "8815": 951961600.0,
+ "8820": 940164672.0,
+ "8825": 964555200.0,
+ "8830": 981470848.0,
+ "8835": 971778240.0,
+ "8840": 979651200.0,
+ "8845": 951320384.0,
+ "8850": 987141504.0,
+ "8855": 971372864.0,
+ "8860": 962404800.0,
+ "8865": 957297472.0,
+ "8870": 946156288.0,
+ "8875": 969020672.0,
+ "8880": 984369856.0,
+ "8885": 971441792.0,
+ "8890": 970361408.0,
+ "8895": 953044608.0,
+ "8900": 961975936.0,
+ "8905": 977023360.0,
+ "8910": 982158720.0,
+ "8915": 981124992.0,
+ "8920": 968220160.0,
+ "8925": 940594688.0,
+ "8930": 970846912.0,
+ "8935": 964150656.0,
+ "8940": 977951488.0,
+ "8945": 982379648.0,
+ "8950": 946076352.0,
+ "8955": 972927552.0,
+ "8960": 973859840.0,
+ "8965": 974120640.0,
+ "8970": 966680000.0,
+ "8975": 937468160.0,
+ "8980": 953369728.0,
+ "8985": 977766912.0,
+ "8990": 967620608.0,
+ "8995": 980945856.0,
+ "9000": 952394752.0,
+ "9005": 950709376.0,
+ "9010": 975378752.0,
+ "9015": 982950080.0,
+ "9020": 959475008.0,
+ "9025": 979791872.0,
+ "9030": 954120320.0,
+ "9035": 968614656.0,
+ "9040": 978674880.0,
+ "9045": 968964160.0,
+ "9050": 983280640.0,
+ "9055": 948192640.0,
+ "9060": 956759744.0,
+ "9065": 969974400.0,
+ "9070": 967757056.0,
+ "9075": 981028672.0,
+ "9080": 952994304.0,
+ "9085": 971687424.0,
+ "9090": 963874496.0,
+ "9095": 968373952.0,
+ "9100": 974891776.0,
+ "9105": 960536704.0,
+ "9110": 948230272.0,
+ "9115": 956895488.0,
+ "9120": 986010176.0,
+ "9125": 963197504.0,
+ "9130": 958861888.0,
+ "9135": 951827776.0,
+ "9140": 967368448.0,
+ "9145": 977617280.0,
+ "9150": 986985472.0,
+ "9155": 977385792.0,
+ "9160": 958147904.0,
+ "9165": 950599232.0,
+ "9170": 988430720.0,
+ "9175": 971827456.0,
+ "9180": 967657152.0,
+ "9185": 955335872.0,
+ "9190": 957127552.0,
+ "9195": 966261376.0,
+ "9200": 969175040.0,
+ "9205": 967636352.0,
+ "9210": 985002240.0,
+ "9215": 931816768.0,
+ "9220": 949942976.0,
+ "9225": 971389696.0,
+ "9230": 970999680.0,
+ "9235": 971870080.0,
+ "9240": 960051776.0,
+ "9245": 964111808.0,
+ "9250": 961931648.0,
+ "9255": 983252672.0,
+ "9260": 979394752.0,
+ "9265": 952509760.0,
+ "9270": 949681856.0,
+ "9275": 978722560.0,
+ "9280": 978049664.0,
+ "9285": 962680960.0,
+ "9290": 979242752.0,
+ "9295": 958904064.0,
+ "9300": 966046592.0,
+ "9305": 969462592.0,
+ "9310": 973282752.0,
+ "9315": 976167552.0,
+ "9320": 948389184.0,
+ "9325": 979737216.0,
+ "9330": 978122432.0,
+ "9335": 975671488.0,
+ "9340": 960371840.0,
+ "9345": 943573440.0,
+ "9350": 953153920.0,
+ "9355": 963375808.0,
+ "9360": 960797696.0,
+ "9365": 983508224.0,
+ "9370": 982987328.0,
+ "9375": 942067008.0,
+ "9380": 983326144.0,
+ "9385": 985749440.0,
+ "9390": 973308096.0,
+ "9395": 978894592.0,
+ "9400": 938332160.0,
+ "9405": 968683456.0,
+ "9410": 982171328.0,
+ "9415": 992238848.0,
+ "9420": 960673344.0,
+ "9425": 957097856.0,
+ "9430": 939234816.0,
+ "9435": 974951872.0,
+ "9440": 959689472.0,
+ "9445": 974297664.0,
+ "9450": 962110144.0,
+ "9455": 946217088.0,
+ "9460": 978465920.0,
+ "9465": 988615680.0,
+ "9470": 963602304.0,
+ "9475": 983955968.0,
+ "9480": 931651008.0,
+ "9485": 987649984.0,
+ "9490": 963923968.0,
+ "9495": 972783104.0,
+ "9500": 982760384.0,
+ "9505": 970407616.0,
+ "9510": 964983552.0,
+ "9515": 957084480.0,
+ "9520": 948395456.0,
+ "9525": 965495424.0,
+ "9530": 958616256.0,
+ "9535": 951551872.0,
+ "9540": 954032704.0,
+ "9545": 979505152.0,
+ "9550": 956158400.0,
+ "9555": 953539968.0,
+ "9560": 958480064.0,
+ "9565": 969926848.0,
+ "9570": 977774464.0,
+ "9575": 959007552.0,
+ "9580": 963107904.0,
+ "9585": 946810048.0,
+ "9590": 948810432.0,
+ "9595": 967363264.0,
+ "9600": 985067136.0,
+ "9605": 985239360.0,
+ "9610": 943946880.0,
+ "9615": 953020480.0,
+ "9620": 981447552.0,
+ "9625": 978904192.0,
+ "9630": 970259840.0,
+ "9635": 975088384.0,
+ "9640": 940783936.0,
+ "9645": 962349824.0,
+ "9650": 971519744.0,
+ "9655": 987470656.0,
+ "9660": 963466368.0,
+ "9665": 950093504.0,
+ "9670": 966320192.0,
+ "9675": 963284160.0,
+ "9680": 965297472.0,
+ "9685": 986536832.0,
+ "9690": 940841216.0,
+ "9695": 950778496.0,
+ "9700": 976180736.0,
+ "9705": 972956480.0,
+ "9710": 967815424.0,
+ "9715": 971697664.0,
+ "9720": 940869888.0,
+ "9725": 966477312.0,
+ "9730": 974320320.0,
+ "9735": 974762368.0,
+ "9740": 971725312.0,
+ "9745": 951060672.0,
+ "9750": 979997248.0,
+ "9755": 970451264.0,
+ "9760": 968346944.0,
+ "9765": 964130816.0,
+ "9770": 952710976.0,
+ "9775": 956865600.0,
+ "9780": 970389120.0,
+ "9785": 959043072.0,
+ "9790": 961323648.0,
+ "9795": 958544896.0,
+ "9800": 949609152.0,
+ "9805": 962440768.0,
+ "9810": 978688000.0,
+ "9815": 977997760.0,
+ "9820": 982998976.0,
+ "9825": 939509760.0,
+ "9830": 969416512.0,
+ "9835": 973184704.0,
+ "9840": 971895552.0,
+ "9845": 967552704.0,
+ "9850": 947278592.0,
+ "9855": 957841536.0,
+ "9860": 987531456.0,
+ "9865": 970396032.0,
+ "9870": 990394624.0,
+ "9875": 957255616.0,
+ "9880": 931373248.0,
+ "9885": 963726400.0,
+ "9890": 972710848.0,
+ "9895": 984065280.0,
+ "9900": 956629824.0,
+ "9905": 939315648.0,
+ "9910": 979056000.0,
+ "9915": 974277760.0,
+ "9920": 944916160.0,
+ "9925": 962988416.0,
+ "9930": 947846144.0,
+ "9935": 961026496.0,
+ "9940": 965854848.0,
+ "9945": 958762688.0,
+ "9950": 964496192.0,
+ "9955": 943559104.0,
+ "9960": 967015296.0,
+ "9965": 984074496.0,
+ "9970": 966937216.0,
+ "9975": 964346240.0,
+ "9980": 980855808.0,
+ "9985": 942708160.0,
+ "9990": 976956480.0,
+ "9995": 982753280.0,
+ "10000": 971923776.0,
+ "10005": 970081216.0,
+ "10010": 944760576.0,
+ "10015": 983815808.0,
+ "10020": 978859200.0,
+ "10025": 979745280.0,
+ "10030": 971783936.0,
+ "10035": 947023168.0,
+ "10040": 950827904.0,
+ "10045": 978282816.0,
+ "10050": 986212160.0,
+ "10055": 990541760.0,
+ "10060": 958798592.0,
+ "10065": 947921472.0,
+ "10070": 967496640.0,
+ "10075": 979866880.0,
+ "10080": 971877184.0,
+ "10085": 975039104.0,
+ "10090": 944118272.0,
+ "10095": 963111040.0,
+ "10100": 972613056.0,
+ "10105": 976340736.0,
+ "10110": 972299136.0,
+ "10115": 949200960.0,
+ "10120": 962728896.0,
+ "10125": 974463808.0,
+ "10130": 981079104.0,
+ "10135": 972727488.0,
+ "10140": 958456576.0,
+ "10145": 934757632.0,
+ "10150": 974173632.0,
+ "10155": 970071040.0,
+ "10160": 962587904.0,
+ "10165": 975340032.0,
+ "10170": 944992576.0,
+ "10175": 979764288.0,
+ "10180": 984166848.0,
+ "10185": 979299584.0,
+ "10190": 956011072.0,
+ "10195": 937762816.0,
+ "10200": 988375040.0,
+ "10205": 973486912.0,
+ "10210": 967071936.0,
+ "10215": 976244288.0,
+ "10220": 948392768.0,
+ "10225": 950509120.0,
+ "10230": 975910400.0,
+ "10235": 954518208.0,
+ "10240": 969725760.0,
+ "10245": 962262720.0,
+ "10250": 936595328.0,
+ "10255": 979858880.0,
+ "10260": 965036480.0,
+ "10265": 967979456.0,
+ "10270": 969013376.0,
+ "10275": 936557568.0,
+ "10280": 970089984.0,
+ "10285": 996574272.0,
+ "10290": 979919936.0,
+ "10295": 982067648.0,
+ "10300": 952589312.0,
+ "10305": 972220096.0,
+ "10310": 960615680.0,
+ "10315": 971949120.0,
+ "10320": 985623808.0,
+ "10325": 983756096.0,
+ "10330": 935342912.0,
+ "10335": 976575872.0,
+ "10340": 957602304.0,
+ "10345": 974258688.0,
+ "10350": 984902144.0,
+ "10355": 942600832.0,
+ "10360": 962436736.0,
+ "10365": 974586944.0,
+ "10370": 981020352.0,
+ "10375": 970280320.0,
+ "10380": 962189312.0,
+ "10385": 955427904.0,
+ "10390": 991076992.0,
+ "10395": 965397312.0,
+ "10400": 961206656.0,
+ "10405": 950568768.0,
+ "10410": 955784576.0,
+ "10415": 976330368.0,
+ "10420": 967640256.0,
+ "10425": 970082752.0,
+ "10430": 965444544.0,
+ "10435": 963393344.0,
+ "10440": 972564096.0,
+ "10445": 972711104.0,
+ "10450": 975586240.0,
+ "10455": 966761280.0,
+ "10460": 949044608.0,
+ "10465": 972012864.0,
+ "10470": 973235136.0,
+ "10475": 979907264.0,
+ "10480": 997795200.0,
+ "10485": 949619008.0,
+ "10490": 935300480.0,
+ "10495": 969611904.0,
+ "10500": 978921280.0,
+ "10505": 959660992.0,
+ "10510": 951633856.0,
+ "10515": 954111680.0,
+ "10520": 972589952.0,
+ "10525": 969646400.0,
+ "10530": 970841536.0,
+ "10535": 986954368.0,
+ "10540": 947295104.0,
+ "10545": 971010944.0,
+ "10550": 969935232.0,
+ "10555": 959974656.0,
+ "10560": 976629184.0,
+ "10565": 961878592.0,
+ "10570": 969657536.0,
+ "10575": 973664384.0,
+ "10580": 961126784.0,
+ "10585": 973761984.0,
+ "10590": 952334656.0,
+ "10595": 957006080.0,
+ "10600": 967797504.0,
+ "10605": 986734464.0,
+ "10610": 966871040.0,
+ "10615": 977264000.0,
+ "10620": 941644608.0,
+ "10625": 965452224.0,
+ "10630": 968266048.0,
+ "10635": 973553152.0,
+ "10640": 974994624.0,
+ "10645": 949035328.0,
+ "10650": 966545152.0,
+ "10655": 986163456.0,
+ "10660": 977134784.0,
+ "10665": 967246080.0,
+ "10670": 955432448.0,
+ "10675": 934489536.0,
+ "10680": 986790528.0,
+ "10685": 991420352.0,
+ "10690": 964497600.0,
+ "10695": 972595136.0,
+ "10700": 950187648.0,
+ "10705": 978509504.0,
+ "10710": 968830912.0,
+ "10715": 967620672.0,
+ "10720": 966650496.0,
+ "10725": 944721472.0,
+ "10730": 980659840.0,
+ "10735": 961373248.0,
+ "10740": 971771328.0,
+ "10745": 985008576.0,
+ "10750": 981981696.0,
+ "10755": 945716672.0,
+ "10760": 970052160.0,
+ "10765": 973155328.0,
+ "10770": 974475456.0,
+ "10775": 959065152.0,
+ "10780": 949775680.0,
+ "10785": 954158976.0,
+ "10790": 970536768.0,
+ "10795": 960812480.0,
+ "10800": 972540160.0,
+ "10805": 951692608.0,
+ "10810": 974430272.0,
+ "10815": 960269632.0,
+ "10820": 971717568.0,
+ "10825": 967418816.0,
+ "10830": 957221696.0,
+ "10835": 963639936.0,
+ "10840": 971205888.0,
+ "10845": 964413312.0,
+ "10850": 958640000.0,
+ "10855": 968024512.0,
+ "10860": 951093888.0,
+ "10865": 964574848.0,
+ "10870": 983797696.0,
+ "10875": 982923584.0,
+ "10880": 958486080.0,
+ "10885": 955116224.0,
+ "10890": 973588480.0,
+ "10895": 973971840.0,
+ "10900": 970937984.0,
+ "10905": 965514560.0,
+ "10910": 939441472.0,
+ "10915": 961079296.0,
+ "10920": 984131648.0,
+ "10925": 970323776.0,
+ "10930": 969175744.0,
+ "10935": 963301760.0,
+ "10940": 954394688.0,
+ "10945": 965563776.0,
+ "10950": 973128640.0,
+ "10955": 967246784.0,
+ "10960": 972552896.0,
+ "10965": 966833984.0,
+ "10970": 983963392.0,
+ "10975": 966287232.0,
+ "10980": 975342720.0,
+ "10985": 987025792.0,
+ "10990": 951644096.0,
+ "10995": 963978240.0,
+ "11000": 985935360.0,
+ "11005": 979086848.0,
+ "11010": 972104448.0,
+ "11015": 970536832.0,
+ "11020": 948057984.0,
+ "11025": 960816512.0,
+ "11030": 978449600.0,
+ "11035": 976138624.0,
+ "11040": 986853184.0,
+ "11045": 957369600.0,
+ "11050": 973873792.0,
+ "11055": 975254464.0,
+ "11060": 962811776.0,
+ "11065": 985970368.0,
+ "11070": 950293056.0,
+ "11075": 977031296.0,
+ "11080": 972596224.0,
+ "11085": 967855104.0,
+ "11090": 976950464.0,
+ "11095": 947014080.0,
+ "11100": 966197568.0,
+ "11105": 974603968.0,
+ "11110": 981418624.0,
+ "11115": 968486848.0,
+ "11120": 958033216.0,
+ "11125": 957333376.0,
+ "11130": 976226496.0,
+ "11135": 979965312.0,
+ "11140": 965299264.0,
+ "11145": 966830400.0,
+ "11150": 936524416.0,
+ "11155": 976638336.0,
+ "11160": 984635776.0,
+ "11165": 983012480.0,
+ "11170": 978529920.0,
+ "11175": 958660352.0,
+ "11180": 962894464.0,
+ "11185": 972480192.0,
+ "11190": 980341504.0,
+ "11195": 985512704.0,
+ "11200": 983381056.0,
+ "11205": 942584384.0,
+ "11210": 985304320.0,
+ "11215": 967911168.0,
+ "11220": 983659648.0,
+ "11225": 962193280.0,
+ "11230": 953703808.0,
+ "11235": 982258240.0,
+ "11240": 977915200.0,
+ "11245": 966858368.0,
+ "11250": 969759808.0,
+ "11255": 960890560.0,
+ "11260": 980377856.0,
+ "11265": 964428800.0,
+ "11270": 982000896.0,
+ "11275": 968942592.0,
+ "11280": 956529920.0,
+ "11285": 954079872.0,
+ "11290": 957163328.0,
+ "11295": 968884800.0,
+ "11300": 963188032.0,
+ "11305": 959323840.0,
+ "11310": 946978176.0,
+ "11315": 983673792.0,
+ "11320": 965318976.0,
+ "11325": 981629568.0,
+ "11330": 976249728.0,
+ "11335": 952934336.0,
+ "11340": 971284032.0,
+ "11345": 970465728.0,
+ "11350": 982093824.0,
+ "11355": 982960896.0,
+ "11360": 941900288.0,
+ "11365": 971190976.0,
+ "11370": 979491968.0,
+ "11375": 976025856.0,
+ "11380": 968820864.0,
+ "11385": 959336256.0,
+ "11390": 938646720.0,
+ "11395": 977983680.0,
+ "11400": 973858112.0,
+ "11405": 961935744.0,
+ "11410": 966806528.0,
+ "11415": 929777408.0,
+ "11420": 965160384.0,
+ "11425": 981550848.0,
+ "11430": 979212288.0,
+ "11435": 970729728.0,
+ "11440": 945697280.0,
+ "11445": 975559552.0,
+ "11450": 984857152.0,
+ "11455": 971634304.0,
+ "11460": 965512128.0,
+ "11465": 960628416.0,
+ "11470": 955201728.0,
+ "11475": 973266304.0,
+ "11480": 956934656.0,
+ "11485": 977665152.0,
+ "11490": 987202432.0,
+ "11495": 959512640.0,
+ "11500": 970934080.0,
+ "11505": 964563584.0,
+ "11510": 977252928.0,
+ "11515": 978639616.0,
+ "11520": 954465344.0,
+ "11525": 976774656.0,
+ "11530": 977371968.0,
+ "11535": 980325376.0,
+ "11540": 974974656.0,
+ "11545": 954059392.0,
+ "11550": 953760768.0,
+ "11555": 982529344.0,
+ "11560": 985169536.0,
+ "11565": 965645568.0,
+ "11570": 967013632.0,
+ "11575": 952418368.0,
+ "11580": 976773568.0,
+ "11585": 977635008.0,
+ "11590": 969803136.0,
+ "11595": 977281984.0,
+ "11600": 946867904.0,
+ "11605": 973934848.0,
+ "11610": 982750144.0,
+ "11615": 972768256.0,
+ "11620": 969840832.0,
+ "11625": 949821696.0,
+ "11630": 938129920.0,
+ "11635": 974156672.0,
+ "11640": 981822656.0,
+ "11645": 980619520.0,
+ "11650": 972432320.0,
+ "11655": 956571072.0,
+ "11660": 981361024.0,
+ "11665": 958829312.0,
+ "11670": 982936768.0,
+ "11675": 973014976.0,
+ "11680": 956714432.0,
+ "11685": 983334272.0,
+ "11690": 969116032.0,
+ "11695": 968036288.0,
+ "11700": 974173376.0,
+ "11705": 956800064.0,
+ "11710": 965369920.0,
+ "11715": 983784000.0,
+ "11720": 984323008.0,
+ "11725": 965977664.0,
+ "11730": 956037824.0,
+ "11735": 943149120.0,
+ "11740": 974220992.0,
+ "11745": 971688640.0,
+ "11750": 962298304.0,
+ "11755": 964072960.0,
+ "11760": 950646400.0,
+ "11765": 984367936.0,
+ "11770": 985153088.0,
+ "11775": 975965056.0,
+ "11780": 985772224.0,
+ "11785": 947688640.0,
+ "11790": 973230528.0,
+ "11795": 971111488.0,
+ "11800": 973656384.0,
+ "11805": 987233472.0,
+ "11810": 968240704.0,
+ "11815": 956327424.0,
+ "11820": 974046080.0,
+ "11825": 971406976.0,
+ "11830": 975546112.0,
+ "11835": 962241792.0,
+ "11840": 944970496.0,
+ "11845": 981491008.0,
+ "11850": 975104640.0,
+ "11855": 978532480.0,
+ "11860": 972385280.0,
+ "11865": 939096384.0,
+ "11870": 940701376.0,
+ "11875": 990513856.0,
+ "11880": 972906624.0,
+ "11885": 963429056.0,
+ "11890": 970857216.0,
+ "11895": 965527488.0,
+ "11900": 979883712.0,
+ "11905": 962068800.0,
+ "11910": 984083072.0,
+ "11915": 990557376.0,
+ "11920": 945026432.0,
+ "11925": 994653632.0,
+ "11930": 965404352.0,
+ "11935": 963992512.0,
+ "11940": 977430528.0,
+ "11945": 945362048.0,
+ "11950": 978226752.0,
+ "11955": 979741056.0,
+ "11960": 972926400.0,
+ "11965": 976653248.0,
+ "11970": 963649344.0,
+ "11975": 963654656.0,
+ "11980": 978148672.0,
+ "11985": 953660672.0,
+ "11990": 969205632.0,
+ "11995": 965646848.0,
+ "12000": 959189376.0,
+ "12005": 975245760.0,
+ "12010": 980040512.0,
+ "12015": 972961472.0,
+ "12020": 974015936.0,
+ "12025": 935486528.0,
+ "12030": 969883456.0,
+ "12035": 984936448.0,
+ "12040": 978279104.0,
+ "12045": 982466944.0,
+ "12050": 932071488.0,
+ "12055": 939453504.0,
+ "12060": 975133888.0,
+ "12065": 966175552.0,
+ "12070": 968532864.0,
+ "12075": 950454848.0,
+ "12080": 954126336.0,
+ "12085": 973089216.0,
+ "12090": 964544896.0,
+ "12095": 964301056.0,
+ "12100": 977560000.0,
+ "12105": 951131328.0,
+ "12110": 972670528.0,
+ "12115": 968593472.0,
+ "12120": 987082368.0,
+ "12125": 981458944.0,
+ "12130": 942360832.0,
+ "12135": 956325184.0,
+ "12140": 976590336.0,
+ "12145": 980090304.0,
+ "12150": 980381248.0,
+ "12155": 962713536.0,
+ "12160": 947316608.0,
+ "12165": 968991616.0,
+ "12170": 965062464.0,
+ "12175": 968575040.0,
+ "12180": 976120320.0,
+ "12185": 954075520.0,
+ "12190": 989391104.0,
+ "12195": 971708800.0,
+ "12200": 965827264.0,
+ "12205": 969711040.0,
+ "12210": 940248576.0,
+ "12215": 997989184.0,
+ "12220": 970995200.0,
+ "12225": 980366208.0,
+ "12230": 980909120.0,
+ "12235": 950726656.0,
+ "12240": 964435264.0,
+ "12245": 966266944.0,
+ "12250": 977518208.0,
+ "12255": 968924928.0,
+ "12260": 984033792.0,
+ "12265": 932555840.0,
+ "12270": 967046272.0,
+ "12275": 980228736.0,
+ "12280": 978388800.0,
+ "12285": 971154624.0,
+ "12290": 929978688.0,
+ "12295": 977657280.0,
+ "12300": 986762304.0,
+ "12305": 970652864.0,
+ "12310": 986926592.0,
+ "12315": 936949504.0,
+ "12320": 958171456.0,
+ "12325": 967223808.0,
+ "12330": 968969344.0,
+ "12335": 964536128.0,
+ "12340": 958341056.0,
+ "12345": 944836288.0,
+ "12350": 967498368.0,
+ "12355": 976371072.0,
+ "12360": 979569024.0,
+ "12365": 965419264.0,
+ "12370": 949564672.0,
+ "12375": 964426688.0,
+ "12380": 965351744.0,
+ "12385": 973481536.0,
+ "12390": 962323136.0,
+ "12395": 962068928.0,
+ "12400": 975785216.0,
+ "12405": 976753664.0,
+ "12410": 953826496.0,
+ "12415": 963161280.0,
+ "12420": 944512320.0,
+ "12425": 949133632.0,
+ "12430": 972276608.0,
+ "12435": 969186816.0,
+ "12440": 961963200.0,
+ "12445": 952227776.0,
+ "12450": 947719424.0,
+ "12455": 981553664.0,
+ "12460": 973975168.0,
+ "12465": 954619008.0,
+ "12470": 981484928.0,
+ "12475": 958971712.0,
+ "12480": 967487808.0,
+ "12485": 978829888.0,
+ "12490": 974300928.0,
+ "12495": 970095616.0,
+ "12500": 962149312.0,
+ "12505": 944269184.0,
+ "12510": 961737600.0,
+ "12515": 969895232.0,
+ "12520": 974657408.0,
+ "12525": 972976704.0,
+ "12530": 945367232.0,
+ "12535": 977345216.0,
+ "12540": 966331136.0,
+ "12545": 972619968.0,
+ "12550": 970357760.0,
+ "12555": 941690496.0,
+ "12560": 965253184.0,
+ "12565": 948048128.0,
+ "12570": 975043136.0,
+ "12575": 963776704.0,
+ "12580": 958666560.0,
+ "12585": 964961536.0,
+ "12590": 966634624.0,
+ "12595": 979503232.0,
+ "12600": 982656704.0,
+ "12605": 949885952.0,
+ "12610": 938553984.0,
+ "12615": 963255936.0,
+ "12620": 961752256.0,
+ "12625": 967008512.0,
+ "12630": 971583232.0,
+ "12635": 962909312.0,
+ "12640": 978944512.0,
+ "12645": 969957952.0,
+ "12650": 970774272.0,
+ "12655": 964693888.0,
+ "12660": 932868224.0,
+ "12665": 957452096.0,
+ "12670": 986767872.0,
+ "12675": 966118336.0,
+ "12680": 961677440.0,
+ "12685": 951879616.0,
+ "12690": 945942400.0,
+ "12695": 978999424.0,
+ "12700": 985706368.0,
+ "12705": 959433984.0,
+ "12710": 969079488.0,
+ "12715": 957051264.0,
+ "12720": 977087616.0,
+ "12725": 965928640.0,
+ "12730": 969980352.0,
+ "12735": 987452160.0,
+ "12740": 937968896.0,
+ "12745": 971461312.0,
+ "12750": 974890176.0,
+ "12755": 980713920.0,
+ "12760": 970725312.0,
+ "12765": 942710592.0,
+ "12770": 952649088.0,
+ "12775": 951886656.0,
+ "12780": 969741312.0,
+ "12785": 956598016.0,
+ "12790": 963494144.0,
+ "12795": 953530688.0,
+ "12800": 963390720.0,
+ "12805": 973287872.0,
+ "12810": 974244416.0,
+ "12815": 953176896.0,
+ "12820": 941838656.0,
+ "12825": 967136384.0,
+ "12830": 999296576.0,
+ "12835": 977431744.0,
+ "12840": 963033920.0,
+ "12845": 942053952.0,
+ "12850": 959577152.0,
+ "12855": 961839488.0,
+ "12860": 972850048.0,
+ "12865": 978494912.0,
+ "12870": 974999104.0,
+ "12875": 956488640.0,
+ "12880": 968016832.0,
+ "12885": 981818176.0,
+ "12890": 960462464.0,
+ "12895": 970178176.0,
+ "12900": 938071232.0,
+ "12905": 968847744.0,
+ "12910": 982008960.0,
+ "12915": 975883520.0,
+ "12920": 956247808.0,
+ "12925": 949779136.0,
+ "12930": 960950592.0,
+ "12935": 991529088.0,
+ "12940": 968458944.0,
+ "12945": 976237952.0,
+ "12950": 971959936.0,
+ "12955": 956432064.0,
+ "12960": 978052416.0,
+ "12965": 961617664.0,
+ "12970": 962469760.0,
+ "12975": 960424000.0,
+ "12980": 938356224.0,
+ "12985": 963543808.0,
+ "12990": 968678016.0,
+ "12995": 976098752.0,
+ "13000": 980572992.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": 14650828800.0,
+ "5": 14650828800.0,
+ "10": 14650828800.0,
+ "15": 14650828800.0,
+ "20": 14650828800.0,
+ "25": 14650828800.0,
+ "30": 14650828800.0,
+ "35": 14650828800.0,
+ "40": 14650828800.0,
+ "45": 14650828800.0,
+ "50": 14650828800.0,
+ "55": 14650828800.0,
+ "60": 14650828800.0,
+ "65": 14650828800.0,
+ "70": 14650828800.0,
+ "75": 14650828800.0,
+ "80": 14650828800.0,
+ "85": 14650828800.0,
+ "90": 14650828800.0,
+ "95": 14650828800.0,
+ "100": 14650828800.0,
+ "105": 14650828800.0,
+ "110": 14650828800.0,
+ "115": 14650828800.0,
+ "120": 14650828800.0,
+ "125": 14650828800.0,
+ "130": 14650828800.0,
+ "135": 14650828800.0,
+ "140": 14650828800.0,
+ "145": 14650828800.0,
+ "150": 14650828800.0,
+ "155": 14650828800.0,
+ "160": 14650828800.0,
+ "165": 14650828800.0,
+ "170": 14650828800.0,
+ "175": 14650828800.0,
+ "180": 14650828800.0,
+ "185": 14650828800.0,
+ "190": 14650828800.0,
+ "195": 14650828800.0,
+ "200": 14650828800.0,
+ "205": 14650828800.0,
+ "210": 14650828800.0,
+ "215": 14650828800.0,
+ "220": 14650828800.0,
+ "225": 14650828800.0,
+ "230": 14650828800.0,
+ "235": 14650828800.0,
+ "240": 14650828800.0,
+ "245": 14650828800.0,
+ "250": 14650828800.0,
+ "255": 14650828800.0,
+ "260": 14650828800.0,
+ "265": 14650828800.0,
+ "270": 14650828800.0,
+ "275": 14650828800.0,
+ "280": 14650828800.0,
+ "285": 14650828800.0,
+ "290": 14650828800.0,
+ "295": 14650828800.0,
+ "300": 14650828800.0,
+ "305": 14650828800.0,
+ "310": 14650828800.0,
+ "315": 14650828800.0,
+ "320": 14650828800.0,
+ "325": 14650828800.0,
+ "330": 14650828800.0,
+ "335": 14650828800.0,
+ "340": 14650828800.0,
+ "345": 14650828800.0,
+ "350": 14650828800.0,
+ "355": 14650828800.0,
+ "360": 14650828800.0,
+ "365": 14650828800.0,
+ "370": 14650828800.0,
+ "375": 14650828800.0,
+ "380": 14650828800.0,
+ "385": 14650828800.0,
+ "390": 14650828800.0,
+ "395": 14650828800.0,
+ "400": 14650828800.0,
+ "405": 14650828800.0,
+ "410": 14650828800.0,
+ "415": 14650828800.0,
+ "420": 14650828800.0,
+ "425": 14650828800.0,
+ "430": 14650828800.0,
+ "435": 14650828800.0,
+ "440": 14650828800.0,
+ "445": 14650828800.0,
+ "450": 14650828800.0,
+ "455": 14650828800.0,
+ "460": 14650828800.0,
+ "465": 14650828800.0,
+ "470": 14650828800.0,
+ "475": 14650828800.0,
+ "480": 14650828800.0,
+ "485": 14650828800.0,
+ "490": 14650828800.0,
+ "495": 14650828800.0,
+ "500": 14650828800.0,
+ "505": 14650828800.0,
+ "510": 14650828800.0,
+ "515": 14650828800.0,
+ "520": 14650828800.0,
+ "525": 14650828800.0,
+ "530": 14650828800.0,
+ "535": 14650828800.0,
+ "540": 14650828800.0,
+ "545": 14650828800.0,
+ "550": 14650828800.0,
+ "555": 14650828800.0,
+ "560": 14650828800.0,
+ "565": 14650828800.0,
+ "570": 14650828800.0,
+ "575": 14650828800.0,
+ "580": 14650828800.0,
+ "585": 14650828800.0,
+ "590": 14650828800.0,
+ "595": 14650828800.0,
+ "600": 14650828800.0,
+ "605": 14650828800.0,
+ "610": 14650828800.0,
+ "615": 14650828800.0,
+ "620": 14650828800.0,
+ "625": 14650828800.0,
+ "630": 14650828800.0,
+ "635": 14650828800.0,
+ "640": 14650828800.0,
+ "645": 14650828800.0,
+ "650": 14650828800.0,
+ "655": 14650828800.0,
+ "660": 14650828800.0,
+ "665": 14650828800.0,
+ "670": 14650828800.0,
+ "675": 14650828800.0,
+ "680": 14650828800.0,
+ "685": 14650828800.0,
+ "690": 14650828800.0,
+ "695": 14650828800.0,
+ "700": 14650828800.0,
+ "705": 14650828800.0,
+ "710": 14650828800.0,
+ "715": 14650828800.0,
+ "720": 14650828800.0,
+ "725": 14650828800.0,
+ "730": 14650828800.0,
+ "735": 14650828800.0,
+ "740": 14650828800.0,
+ "745": 14650828800.0,
+ "750": 14650828800.0,
+ "755": 14650828800.0,
+ "760": 14650828800.0,
+ "765": 14650828800.0,
+ "770": 14650828800.0,
+ "775": 14650828800.0,
+ "780": 14650828800.0,
+ "785": 14650828800.0,
+ "790": 14650828800.0,
+ "795": 14650828800.0,
+ "800": 14650828800.0,
+ "805": 14650828800.0,
+ "810": 14650828800.0,
+ "815": 14650828800.0,
+ "820": 14650828800.0,
+ "825": 14650828800.0,
+ "830": 14650828800.0,
+ "835": 14650828800.0,
+ "840": 14650828800.0,
+ "845": 14650828800.0,
+ "850": 14650828800.0,
+ "855": 14650828800.0,
+ "860": 14650828800.0,
+ "865": 14650828800.0,
+ "870": 14650828800.0,
+ "875": 14650828800.0,
+ "880": 14650828800.0,
+ "885": 14650828800.0,
+ "890": 14650828800.0,
+ "895": 14650828800.0,
+ "900": 14650828800.0,
+ "905": 14650828800.0,
+ "910": 14650828800.0,
+ "915": 14650828800.0,
+ "920": 14650828800.0,
+ "925": 14650828800.0,
+ "930": 14650828800.0,
+ "935": 14650828800.0,
+ "940": 14650828800.0,
+ "945": 14650828800.0,
+ "950": 14650828800.0,
+ "955": 14650828800.0,
+ "960": 14650828800.0,
+ "965": 14650828800.0,
+ "970": 14650828800.0,
+ "975": 14650828800.0,
+ "980": 14650828800.0,
+ "985": 14650828800.0,
+ "990": 14650828800.0,
+ "995": 14650828800.0,
+ "1000": 14650828800.0,
+ "1005": 14650828800.0,
+ "1010": 14650828800.0,
+ "1015": 14650828800.0,
+ "1020": 14650828800.0,
+ "1025": 14650828800.0,
+ "1030": 14650828800.0,
+ "1035": 14650828800.0,
+ "1040": 14650828800.0,
+ "1045": 14650828800.0,
+ "1050": 14650828800.0,
+ "1055": 14650828800.0,
+ "1060": 14650828800.0,
+ "1065": 14650828800.0,
+ "1070": 14650828800.0,
+ "1075": 14650828800.0,
+ "1080": 14650828800.0,
+ "1085": 14650828800.0,
+ "1090": 14650828800.0,
+ "1095": 14650828800.0,
+ "1100": 14650828800.0,
+ "1105": 14650828800.0,
+ "1110": 14650828800.0,
+ "1115": 14650828800.0,
+ "1120": 14650828800.0,
+ "1125": 14650828800.0,
+ "1130": 14650828800.0,
+ "1135": 14650828800.0,
+ "1140": 14650828800.0,
+ "1145": 14650828800.0,
+ "1150": 14650828800.0,
+ "1155": 14650828800.0,
+ "1160": 14650828800.0,
+ "1165": 14650828800.0,
+ "1170": 14650828800.0,
+ "1175": 14650828800.0,
+ "1180": 14650828800.0,
+ "1185": 14650828800.0,
+ "1190": 14650828800.0,
+ "1195": 14650828800.0,
+ "1200": 14650828800.0,
+ "1205": 14650828800.0,
+ "1210": 14650828800.0,
+ "1215": 14650828800.0,
+ "1220": 14650828800.0,
+ "1225": 14650828800.0,
+ "1230": 14650828800.0,
+ "1235": 14650828800.0,
+ "1240": 14650828800.0,
+ "1245": 14650828800.0,
+ "1250": 14650828800.0,
+ "1255": 14650828800.0,
+ "1260": 14650828800.0,
+ "1265": 14650828800.0,
+ "1270": 14650828800.0,
+ "1275": 14650828800.0,
+ "1280": 14650828800.0,
+ "1285": 14650828800.0,
+ "1290": 14650828800.0,
+ "1295": 14650828800.0,
+ "1300": 14650828800.0,
+ "1305": 14650828800.0,
+ "1310": 14650828800.0,
+ "1315": 14650828800.0,
+ "1320": 14650828800.0,
+ "1325": 14650828800.0,
+ "1330": 14650828800.0,
+ "1335": 14650828800.0,
+ "1340": 14650828800.0,
+ "1345": 14650828800.0,
+ "1350": 14650828800.0,
+ "1355": 14650828800.0,
+ "1360": 14650828800.0,
+ "1365": 14650828800.0,
+ "1370": 14650828800.0,
+ "1375": 14650828800.0,
+ "1380": 14650828800.0,
+ "1385": 14650828800.0,
+ "1390": 14650828800.0,
+ "1395": 14650828800.0,
+ "1400": 14650828800.0,
+ "1405": 14650828800.0,
+ "1410": 14650828800.0,
+ "1415": 14650828800.0,
+ "1420": 14650828800.0,
+ "1425": 14650828800.0,
+ "1430": 14650828800.0,
+ "1435": 14650828800.0,
+ "1440": 14650828800.0,
+ "1445": 14650828800.0,
+ "1450": 14650828800.0,
+ "1455": 14650828800.0,
+ "1460": 14650828800.0,
+ "1465": 14650828800.0,
+ "1470": 14650828800.0,
+ "1475": 14650828800.0,
+ "1480": 14650828800.0,
+ "1485": 14650828800.0,
+ "1490": 14650828800.0,
+ "1495": 14650828800.0,
+ "1500": 14650828800.0,
+ "1505": 14650828800.0,
+ "1510": 14650828800.0,
+ "1515": 14650828800.0,
+ "1520": 14650828800.0,
+ "1525": 14650828800.0,
+ "1530": 14650828800.0,
+ "1535": 14650828800.0,
+ "1540": 14650828800.0,
+ "1545": 14650828800.0,
+ "1550": 14650828800.0,
+ "1555": 14650828800.0,
+ "1560": 14650828800.0,
+ "1565": 14650828800.0,
+ "1570": 14650828800.0,
+ "1575": 14650828800.0,
+ "1580": 14650828800.0,
+ "1585": 14650828800.0,
+ "1590": 14650828800.0,
+ "1595": 14650828800.0,
+ "1600": 14650828800.0,
+ "1605": 14650828800.0,
+ "1610": 14650828800.0,
+ "1615": 14650828800.0,
+ "1620": 14650828800.0,
+ "1625": 14650828800.0,
+ "1630": 14650828800.0,
+ "1635": 14650828800.0,
+ "1640": 14650828800.0,
+ "1645": 14650828800.0,
+ "1650": 14650828800.0,
+ "1655": 14650828800.0,
+ "1660": 14650828800.0,
+ "1665": 14650828800.0,
+ "1670": 14650828800.0,
+ "1675": 14650828800.0,
+ "1680": 14650828800.0,
+ "1685": 14650828800.0,
+ "1690": 14650828800.0,
+ "1695": 14650828800.0,
+ "1700": 14650828800.0,
+ "1705": 14650828800.0,
+ "1710": 14650828800.0,
+ "1715": 14650828800.0,
+ "1720": 14650828800.0,
+ "1725": 14650828800.0,
+ "1730": 14650828800.0,
+ "1735": 14650828800.0,
+ "1740": 14650828800.0,
+ "1745": 14650828800.0,
+ "1750": 14650828800.0,
+ "1755": 14650828800.0,
+ "1760": 14650828800.0,
+ "1765": 14650828800.0,
+ "1770": 14650828800.0,
+ "1775": 14650828800.0,
+ "1780": 14650828800.0,
+ "1785": 14650828800.0,
+ "1790": 14650828800.0,
+ "1795": 14650828800.0,
+ "1800": 14650828800.0,
+ "1805": 14650828800.0,
+ "1810": 14650828800.0,
+ "1815": 14650828800.0,
+ "1820": 14650828800.0,
+ "1825": 14650828800.0,
+ "1830": 14650828800.0,
+ "1835": 14650828800.0,
+ "1840": 14650828800.0,
+ "1845": 14650828800.0,
+ "1850": 14650828800.0,
+ "1855": 14650828800.0,
+ "1860": 14650828800.0,
+ "1865": 14650828800.0,
+ "1870": 14650828800.0,
+ "1875": 14650828800.0,
+ "1880": 14650828800.0,
+ "1885": 14650828800.0,
+ "1890": 14650828800.0,
+ "1895": 14650828800.0,
+ "1900": 14650828800.0,
+ "1905": 14650828800.0,
+ "1910": 14650828800.0,
+ "1915": 14650828800.0,
+ "1920": 14650828800.0,
+ "1925": 14650828800.0,
+ "1930": 14650828800.0,
+ "1935": 14650828800.0,
+ "1940": 14650828800.0,
+ "1945": 14650828800.0,
+ "1950": 14650828800.0,
+ "1955": 14650828800.0,
+ "1960": 14650828800.0,
+ "1965": 14650828800.0,
+ "1970": 14650828800.0,
+ "1975": 14650828800.0,
+ "1980": 14650828800.0,
+ "1985": 14650828800.0,
+ "1990": 14650828800.0,
+ "1995": 14650828800.0,
+ "2000": 14650828800.0,
+ "2005": 14650828800.0,
+ "2010": 14650828800.0,
+ "2015": 14650828800.0,
+ "2020": 14650828800.0,
+ "2025": 14650828800.0,
+ "2030": 14650828800.0,
+ "2035": 14650828800.0,
+ "2040": 14650828800.0,
+ "2045": 14650828800.0,
+ "2050": 14650828800.0,
+ "2055": 14650828800.0,
+ "2060": 14650828800.0,
+ "2065": 14650828800.0,
+ "2070": 14650828800.0,
+ "2075": 14650828800.0,
+ "2080": 14650828800.0,
+ "2085": 14650828800.0,
+ "2090": 14650828800.0,
+ "2095": 14650828800.0,
+ "2100": 14650828800.0,
+ "2105": 14650828800.0,
+ "2110": 14650828800.0,
+ "2115": 14650828800.0,
+ "2120": 14650828800.0,
+ "2125": 14650828800.0,
+ "2130": 14650828800.0,
+ "2135": 14650828800.0,
+ "2140": 14650828800.0,
+ "2145": 14650828800.0,
+ "2150": 14650828800.0,
+ "2155": 14650828800.0,
+ "2160": 14650828800.0,
+ "2165": 14650828800.0,
+ "2170": 14650828800.0,
+ "2175": 14650828800.0,
+ "2180": 14650828800.0,
+ "2185": 14650828800.0,
+ "2190": 14650828800.0,
+ "2195": 14650828800.0,
+ "2200": 14650828800.0,
+ "2205": 14650828800.0,
+ "2210": 14650828800.0,
+ "2215": 14650828800.0,
+ "2220": 14650828800.0,
+ "2225": 14650828800.0,
+ "2230": 14650828800.0,
+ "2235": 14650828800.0,
+ "2240": 14650828800.0,
+ "2245": 14650828800.0,
+ "2250": 14650828800.0,
+ "2255": 14650828800.0,
+ "2260": 14650828800.0,
+ "2265": 14650828800.0,
+ "2270": 14650828800.0,
+ "2275": 14650828800.0,
+ "2280": 14650828800.0,
+ "2285": 14650828800.0,
+ "2290": 14650828800.0,
+ "2295": 14650828800.0,
+ "2300": 14650828800.0,
+ "2305": 14650828800.0,
+ "2310": 14650828800.0,
+ "2315": 14650828800.0,
+ "2320": 14650828800.0,
+ "2325": 14650828800.0,
+ "2330": 14650828800.0,
+ "2335": 14650828800.0,
+ "2340": 14650828800.0,
+ "2345": 14650828800.0,
+ "2350": 14650828800.0,
+ "2355": 14650828800.0,
+ "2360": 14650828800.0,
+ "2365": 14650828800.0,
+ "2370": 14650828800.0,
+ "2375": 14650828800.0,
+ "2380": 14650828800.0,
+ "2385": 14650828800.0,
+ "2390": 14650828800.0,
+ "2395": 14650828800.0,
+ "2400": 14650828800.0,
+ "2405": 14650828800.0,
+ "2410": 14650828800.0,
+ "2415": 14650828800.0,
+ "2420": 14650828800.0,
+ "2425": 14650828800.0,
+ "2430": 14650828800.0,
+ "2435": 14650828800.0,
+ "2440": 14650828800.0,
+ "2445": 14650828800.0,
+ "2450": 14650828800.0,
+ "2455": 14650828800.0,
+ "2460": 14650828800.0,
+ "2465": 14650828800.0,
+ "2470": 14650828800.0,
+ "2475": 14650828800.0,
+ "2480": 14650828800.0,
+ "2485": 14650828800.0,
+ "2490": 14650828800.0,
+ "2495": 14650828800.0,
+ "2500": 14650828800.0,
+ "2505": 14650828800.0,
+ "2510": 14650828800.0,
+ "2515": 14650828800.0,
+ "2520": 14650828800.0,
+ "2525": 14650828800.0,
+ "2530": 14650828800.0,
+ "2535": 14650828800.0,
+ "2540": 14650828800.0,
+ "2545": 14650828800.0,
+ "2550": 14650828800.0,
+ "2555": 14650828800.0,
+ "2560": 14650828800.0,
+ "2565": 14650828800.0,
+ "2570": 14650828800.0,
+ "2575": 14650828800.0,
+ "2580": 14650828800.0,
+ "2585": 14650828800.0,
+ "2590": 14650828800.0,
+ "2595": 14650828800.0,
+ "2600": 14650828800.0,
+ "2605": 14650828800.0,
+ "2610": 14650828800.0,
+ "2615": 14650828800.0,
+ "2620": 14650828800.0,
+ "2625": 14650828800.0,
+ "2630": 14650828800.0,
+ "2635": 14650828800.0,
+ "2640": 14650828800.0,
+ "2645": 14650828800.0,
+ "2650": 14650828800.0,
+ "2655": 14650828800.0,
+ "2660": 14650828800.0,
+ "2665": 14650828800.0,
+ "2670": 14650828800.0,
+ "2675": 14650828800.0,
+ "2680": 14650828800.0,
+ "2685": 14650828800.0,
+ "2690": 14650828800.0,
+ "2695": 14650828800.0,
+ "2700": 14650828800.0,
+ "2705": 14650828800.0,
+ "2710": 14650828800.0,
+ "2715": 14650828800.0,
+ "2720": 14650828800.0,
+ "2725": 14650828800.0,
+ "2730": 14650828800.0,
+ "2735": 14650828800.0,
+ "2740": 14650828800.0,
+ "2745": 14650828800.0,
+ "2750": 14650828800.0,
+ "2755": 14650828800.0,
+ "2760": 14650828800.0,
+ "2765": 14650828800.0,
+ "2770": 14650828800.0,
+ "2775": 14650828800.0,
+ "2780": 14650828800.0,
+ "2785": 14650828800.0,
+ "2790": 14650828800.0,
+ "2795": 14650828800.0,
+ "2800": 14650828800.0,
+ "2805": 14650828800.0,
+ "2810": 14650828800.0,
+ "2815": 14650828800.0,
+ "2820": 14650828800.0,
+ "2825": 14650828800.0,
+ "2830": 14650828800.0,
+ "2835": 14650828800.0,
+ "2840": 14650828800.0,
+ "2845": 14650828800.0,
+ "2850": 14650828800.0,
+ "2855": 14650828800.0,
+ "2860": 14650828800.0,
+ "2865": 14650828800.0,
+ "2870": 14650828800.0,
+ "2875": 14650828800.0,
+ "2880": 14650828800.0,
+ "2885": 14650828800.0,
+ "2890": 14650828800.0,
+ "2895": 14650828800.0,
+ "2900": 14650828800.0,
+ "2905": 14650828800.0,
+ "2910": 14650828800.0,
+ "2915": 14650828800.0,
+ "2920": 14650828800.0,
+ "2925": 14650828800.0,
+ "2930": 14650828800.0,
+ "2935": 14650828800.0,
+ "2940": 14650828800.0,
+ "2945": 14650828800.0,
+ "2950": 14650828800.0,
+ "2955": 14650828800.0,
+ "2960": 14650828800.0,
+ "2965": 14650828800.0,
+ "2970": 14650828800.0,
+ "2975": 14650828800.0,
+ "2980": 14650828800.0,
+ "2985": 14650828800.0,
+ "2990": 14650828800.0,
+ "2995": 14650828800.0,
+ "3000": 14650828800.0,
+ "3005": 14650828800.0,
+ "3010": 14650828800.0,
+ "3015": 14650828800.0,
+ "3020": 14650828800.0,
+ "3025": 14650828800.0,
+ "3030": 14650828800.0,
+ "3035": 14650828800.0,
+ "3040": 14650828800.0,
+ "3045": 14650828800.0,
+ "3050": 14650828800.0,
+ "3055": 14650828800.0,
+ "3060": 14650828800.0,
+ "3065": 14650828800.0,
+ "3070": 14650828800.0,
+ "3075": 14650828800.0,
+ "3080": 14650828800.0,
+ "3085": 14650828800.0,
+ "3090": 14650828800.0,
+ "3095": 14650828800.0,
+ "3100": 14650828800.0,
+ "3105": 14650828800.0,
+ "3110": 14650828800.0,
+ "3115": 14650828800.0,
+ "3120": 14650828800.0,
+ "3125": 14650828800.0,
+ "3130": 14650828800.0,
+ "3135": 14650828800.0,
+ "3140": 14650828800.0,
+ "3145": 14650828800.0,
+ "3150": 14650828800.0,
+ "3155": 14650828800.0,
+ "3160": 14650828800.0,
+ "3165": 14650828800.0,
+ "3170": 14650828800.0,
+ "3175": 14650828800.0,
+ "3180": 14650828800.0,
+ "3185": 14650828800.0,
+ "3190": 14650828800.0,
+ "3195": 14650828800.0,
+ "3200": 14650828800.0,
+ "3205": 14650828800.0,
+ "3210": 14650828800.0,
+ "3215": 14650828800.0,
+ "3220": 14650828800.0,
+ "3225": 14650828800.0,
+ "3230": 14650828800.0,
+ "3235": 14650828800.0,
+ "3240": 14650828800.0,
+ "3245": 14650828800.0,
+ "3250": 14650828800.0,
+ "3255": 14650828800.0,
+ "3260": 14650828800.0,
+ "3265": 14650828800.0,
+ "3270": 14650828800.0,
+ "3275": 14650828800.0,
+ "3280": 14650828800.0,
+ "3285": 14650828800.0,
+ "3290": 14650828800.0,
+ "3295": 14650828800.0,
+ "3300": 14650828800.0,
+ "3305": 14650828800.0,
+ "3310": 14650828800.0,
+ "3315": 14650828800.0,
+ "3320": 14650828800.0,
+ "3325": 14650828800.0,
+ "3330": 14650828800.0,
+ "3335": 14650828800.0,
+ "3340": 14650828800.0,
+ "3345": 14650828800.0,
+ "3350": 14650828800.0,
+ "3355": 14650828800.0,
+ "3360": 14650828800.0,
+ "3365": 14650828800.0,
+ "3370": 14650828800.0,
+ "3375": 14650828800.0,
+ "3380": 14650828800.0,
+ "3385": 14650828800.0,
+ "3390": 14650828800.0,
+ "3395": 14650828800.0,
+ "3400": 14650828800.0,
+ "3405": 14650828800.0,
+ "3410": 14650828800.0,
+ "3415": 14650828800.0,
+ "3420": 14650828800.0,
+ "3425": 14650828800.0,
+ "3430": 14650828800.0,
+ "3435": 14650828800.0,
+ "3440": 14650828800.0,
+ "3445": 14650828800.0,
+ "3450": 14650828800.0,
+ "3455": 14650828800.0,
+ "3460": 14650828800.0,
+ "3465": 14650828800.0,
+ "3470": 14650828800.0,
+ "3475": 14650828800.0,
+ "3480": 14650828800.0,
+ "3485": 14650828800.0,
+ "3490": 14650828800.0,
+ "3495": 14650828800.0,
+ "3500": 14650828800.0,
+ "3505": 14650828800.0,
+ "3510": 14650828800.0,
+ "3515": 14650828800.0,
+ "3520": 14650828800.0,
+ "3525": 14650828800.0,
+ "3530": 14650828800.0,
+ "3535": 14650828800.0,
+ "3540": 14650828800.0,
+ "3545": 14650828800.0,
+ "3550": 14650828800.0,
+ "3555": 14650828800.0,
+ "3560": 14650828800.0,
+ "3565": 14650828800.0,
+ "3570": 14650828800.0,
+ "3575": 14650828800.0,
+ "3580": 14650828800.0,
+ "3585": 14650828800.0,
+ "3590": 14650828800.0,
+ "3595": 14650828800.0,
+ "3600": 14650828800.0,
+ "3605": 14650828800.0,
+ "3610": 14650828800.0,
+ "3615": 14650828800.0,
+ "3620": 14650828800.0,
+ "3625": 14650828800.0,
+ "3630": 14650828800.0,
+ "3635": 14650828800.0,
+ "3640": 14650828800.0,
+ "3645": 14650828800.0,
+ "3650": 14650828800.0,
+ "3655": 14650828800.0,
+ "3660": 14650828800.0,
+ "3665": 14650828800.0,
+ "3670": 14650828800.0,
+ "3675": 14650828800.0,
+ "3680": 14650828800.0,
+ "3685": 14650828800.0,
+ "3690": 14650828800.0,
+ "3695": 14650828800.0,
+ "3700": 14650828800.0,
+ "3705": 14650828800.0,
+ "3710": 14650828800.0,
+ "3715": 14650828800.0,
+ "3720": 14650828800.0,
+ "3725": 14650828800.0,
+ "3730": 14650828800.0,
+ "3735": 14650828800.0,
+ "3740": 14650828800.0,
+ "3745": 14650828800.0,
+ "3750": 14650828800.0,
+ "3755": 14650828800.0,
+ "3760": 14650828800.0,
+ "3765": 14650828800.0,
+ "3770": 14650828800.0,
+ "3775": 14650828800.0,
+ "3780": 14650828800.0,
+ "3785": 14650828800.0,
+ "3790": 14650828800.0,
+ "3795": 14650828800.0,
+ "3800": 14650828800.0,
+ "3805": 14650828800.0,
+ "3810": 14650828800.0,
+ "3815": 14650828800.0,
+ "3820": 14650828800.0,
+ "3825": 14650828800.0,
+ "3830": 14650828800.0,
+ "3835": 14650828800.0,
+ "3840": 14650828800.0,
+ "3845": 14650828800.0,
+ "3850": 14650828800.0,
+ "3855": 14650828800.0,
+ "3860": 14650828800.0,
+ "3865": 14650828800.0,
+ "3870": 14650828800.0,
+ "3875": 14650828800.0,
+ "3880": 14650828800.0,
+ "3885": 14650828800.0,
+ "3890": 14650828800.0,
+ "3895": 14650828800.0,
+ "3900": 14650828800.0,
+ "3905": 14650828800.0,
+ "3910": 14650828800.0,
+ "3915": 14650828800.0,
+ "3920": 14650828800.0,
+ "3925": 14650828800.0,
+ "3930": 14650828800.0,
+ "3935": 14650828800.0,
+ "3940": 14650828800.0,
+ "3945": 14650828800.0,
+ "3950": 14650828800.0,
+ "3955": 14650828800.0,
+ "3960": 14650828800.0,
+ "3965": 14650828800.0,
+ "3970": 14650828800.0,
+ "3975": 14650828800.0,
+ "3980": 14650828800.0,
+ "3985": 14650828800.0,
+ "3990": 14650828800.0,
+ "3995": 14650828800.0,
+ "4000": 14650828800.0,
+ "4005": 14650828800.0,
+ "4010": 14650828800.0,
+ "4015": 14650828800.0,
+ "4020": 14650828800.0,
+ "4025": 14650828800.0,
+ "4030": 14650828800.0,
+ "4035": 14650828800.0,
+ "4040": 14650828800.0,
+ "4045": 14650828800.0,
+ "4050": 14650828800.0,
+ "4055": 14650828800.0,
+ "4060": 14650828800.0,
+ "4065": 14650828800.0,
+ "4070": 14650828800.0,
+ "4075": 14650828800.0,
+ "4080": 14650828800.0,
+ "4085": 14650828800.0,
+ "4090": 14650828800.0,
+ "4095": 14650828800.0,
+ "4100": 14650828800.0,
+ "4105": 14650828800.0,
+ "4110": 14650828800.0,
+ "4115": 14650828800.0,
+ "4120": 14650828800.0,
+ "4125": 14650828800.0,
+ "4130": 14650828800.0,
+ "4135": 14650828800.0,
+ "4140": 14650828800.0,
+ "4145": 14650828800.0,
+ "4150": 14650828800.0,
+ "4155": 14650828800.0,
+ "4160": 14650828800.0,
+ "4165": 14650828800.0,
+ "4170": 14650828800.0,
+ "4175": 14650828800.0,
+ "4180": 14650828800.0,
+ "4185": 14650828800.0,
+ "4190": 14650828800.0,
+ "4195": 14650828800.0,
+ "4200": 14650828800.0,
+ "4205": 14650828800.0,
+ "4210": 14650828800.0,
+ "4215": 14650828800.0,
+ "4220": 14650828800.0,
+ "4225": 14650828800.0,
+ "4230": 14650828800.0,
+ "4235": 14650828800.0,
+ "4240": 14650828800.0,
+ "4245": 14650828800.0,
+ "4250": 14650828800.0,
+ "4255": 14650828800.0,
+ "4260": 14650828800.0,
+ "4265": 14650828800.0,
+ "4270": 14650828800.0,
+ "4275": 14650828800.0,
+ "4280": 14650828800.0,
+ "4285": 14650828800.0,
+ "4290": 14650828800.0,
+ "4295": 14650828800.0,
+ "4300": 14650828800.0,
+ "4305": 14650828800.0,
+ "4310": 14650828800.0,
+ "4315": 14650828800.0,
+ "4320": 14650828800.0,
+ "4325": 14650828800.0,
+ "4330": 14650828800.0,
+ "4335": 14650828800.0,
+ "4340": 14650828800.0,
+ "4345": 14650828800.0,
+ "4350": 14650828800.0,
+ "4355": 14650828800.0,
+ "4360": 14650828800.0,
+ "4365": 14650828800.0,
+ "4370": 14650828800.0,
+ "4375": 14650828800.0,
+ "4380": 14650828800.0,
+ "4385": 14650828800.0,
+ "4390": 14650828800.0,
+ "4395": 14650828800.0,
+ "4400": 14650828800.0,
+ "4405": 14650828800.0,
+ "4410": 14650828800.0,
+ "4415": 14650828800.0,
+ "4420": 14650828800.0,
+ "4425": 14650828800.0,
+ "4430": 14650828800.0,
+ "4435": 14650828800.0,
+ "4440": 14650828800.0,
+ "4445": 14650828800.0,
+ "4450": 14650828800.0,
+ "4455": 14650828800.0,
+ "4460": 14650828800.0,
+ "4465": 14650828800.0,
+ "4470": 14650828800.0,
+ "4475": 14650828800.0,
+ "4480": 14650828800.0,
+ "4485": 14650828800.0,
+ "4490": 14650828800.0,
+ "4495": 14650828800.0,
+ "4500": 14650828800.0,
+ "4505": 14650828800.0,
+ "4510": 14650828800.0,
+ "4515": 14650828800.0,
+ "4520": 14650828800.0,
+ "4525": 14650828800.0,
+ "4530": 14650828800.0,
+ "4535": 14650828800.0,
+ "4540": 14650828800.0,
+ "4545": 14650828800.0,
+ "4550": 14650828800.0,
+ "4555": 14650828800.0,
+ "4560": 14650828800.0,
+ "4565": 14650828800.0,
+ "4570": 14650828800.0,
+ "4575": 14650828800.0,
+ "4580": 14650828800.0,
+ "4585": 14650828800.0,
+ "4590": 14650828800.0,
+ "4595": 14650828800.0,
+ "4600": 14650828800.0,
+ "4605": 14650828800.0,
+ "4610": 14650828800.0,
+ "4615": 14650828800.0,
+ "4620": 14650828800.0,
+ "4625": 14650828800.0,
+ "4630": 14650828800.0,
+ "4635": 14650828800.0,
+ "4640": 14650828800.0,
+ "4645": 14650828800.0,
+ "4650": 14650828800.0,
+ "4655": 14650828800.0,
+ "4660": 14650828800.0,
+ "4665": 14650828800.0,
+ "4670": 14650828800.0,
+ "4675": 14650828800.0,
+ "4680": 14650828800.0,
+ "4685": 14650828800.0,
+ "4690": 14650828800.0,
+ "4695": 14650828800.0,
+ "4700": 14650828800.0,
+ "4705": 14650828800.0,
+ "4710": 14650828800.0,
+ "4715": 14650828800.0,
+ "4720": 14650828800.0,
+ "4725": 14650828800.0,
+ "4730": 14650828800.0,
+ "4735": 14650828800.0,
+ "4740": 14650828800.0,
+ "4745": 14650828800.0,
+ "4750": 14650828800.0,
+ "4755": 14650828800.0,
+ "4760": 14650828800.0,
+ "4765": 14650828800.0,
+ "4770": 14650828800.0,
+ "4775": 14650828800.0,
+ "4780": 14650828800.0,
+ "4785": 14650828800.0,
+ "4790": 14650828800.0,
+ "4795": 14650828800.0,
+ "4800": 14650828800.0,
+ "4805": 14650828800.0,
+ "4810": 14650828800.0,
+ "4815": 14650828800.0,
+ "4820": 14650828800.0,
+ "4825": 14650828800.0,
+ "4830": 14650828800.0,
+ "4835": 14650828800.0,
+ "4840": 14650828800.0,
+ "4845": 14650828800.0,
+ "4850": 14650828800.0,
+ "4855": 14650828800.0,
+ "4860": 14650828800.0,
+ "4865": 14650828800.0,
+ "4870": 14650828800.0,
+ "4875": 14650828800.0,
+ "4880": 14650828800.0,
+ "4885": 14650828800.0,
+ "4890": 14650828800.0,
+ "4895": 14650828800.0,
+ "4900": 14650828800.0,
+ "4905": 14650828800.0,
+ "4910": 14650828800.0,
+ "4915": 14650828800.0,
+ "4920": 14650828800.0,
+ "4925": 14650828800.0,
+ "4930": 14650828800.0,
+ "4935": 14650828800.0,
+ "4940": 14650828800.0,
+ "4945": 14650828800.0,
+ "4950": 14650828800.0,
+ "4955": 14650828800.0,
+ "4960": 14650828800.0,
+ "4965": 14650828800.0,
+ "4970": 14650828800.0,
+ "4975": 14650828800.0,
+ "4980": 14650828800.0,
+ "4985": 14650828800.0,
+ "4990": 14650828800.0,
+ "4995": 14650828800.0,
+ "5000": 14650828800.0,
+ "5005": 14650828800.0,
+ "5010": 14650828800.0,
+ "5015": 14650828800.0,
+ "5020": 14650828800.0,
+ "5025": 14650828800.0,
+ "5030": 14650828800.0,
+ "5035": 14650828800.0,
+ "5040": 14650828800.0,
+ "5045": 14650828800.0,
+ "5050": 14650828800.0,
+ "5055": 14650828800.0,
+ "5060": 14650828800.0,
+ "5065": 14650828800.0,
+ "5070": 14650828800.0,
+ "5075": 14650828800.0,
+ "5080": 14650828800.0,
+ "5085": 14650828800.0,
+ "5090": 14650828800.0,
+ "5095": 14650828800.0,
+ "5100": 14650828800.0,
+ "5105": 14650828800.0,
+ "5110": 14650828800.0,
+ "5115": 14650828800.0,
+ "5120": 14650828800.0,
+ "5125": 14650828800.0,
+ "5130": 14650828800.0,
+ "5135": 14650828800.0,
+ "5140": 14650828800.0,
+ "5145": 14650828800.0,
+ "5150": 14650828800.0,
+ "5155": 14650828800.0,
+ "5160": 14650828800.0,
+ "5165": 14650828800.0,
+ "5170": 14650828800.0,
+ "5175": 14650828800.0,
+ "5180": 14650828800.0,
+ "5185": 14650828800.0,
+ "5190": 14650828800.0,
+ "5195": 14650828800.0,
+ "5200": 14650828800.0,
+ "5205": 14650828800.0,
+ "5210": 14650828800.0,
+ "5215": 14650828800.0,
+ "5220": 14650828800.0,
+ "5225": 14650828800.0,
+ "5230": 14650828800.0,
+ "5235": 14650828800.0,
+ "5240": 14650828800.0,
+ "5245": 14650828800.0,
+ "5250": 14650828800.0,
+ "5255": 14650828800.0,
+ "5260": 14650828800.0,
+ "5265": 14650828800.0,
+ "5270": 14650828800.0,
+ "5275": 14650828800.0,
+ "5280": 14650828800.0,
+ "5285": 14650828800.0,
+ "5290": 14650828800.0,
+ "5295": 14650828800.0,
+ "5300": 14650828800.0,
+ "5305": 14650828800.0,
+ "5310": 14650828800.0,
+ "5315": 14650828800.0,
+ "5320": 14650828800.0,
+ "5325": 14650828800.0,
+ "5330": 14650828800.0,
+ "5335": 14650828800.0,
+ "5340": 14650828800.0,
+ "5345": 14650828800.0,
+ "5350": 14650828800.0,
+ "5355": 14650828800.0,
+ "5360": 14650828800.0,
+ "5365": 14650828800.0,
+ "5370": 14650828800.0,
+ "5375": 14650828800.0,
+ "5380": 14650828800.0,
+ "5385": 14650828800.0,
+ "5390": 14650828800.0,
+ "5395": 14650828800.0,
+ "5400": 14650828800.0,
+ "5405": 14650828800.0,
+ "5410": 14650828800.0,
+ "5415": 14650828800.0,
+ "5420": 14650828800.0,
+ "5425": 14650828800.0,
+ "5430": 14650828800.0,
+ "5435": 14650828800.0,
+ "5440": 14650828800.0,
+ "5445": 14650828800.0,
+ "5450": 14650828800.0,
+ "5455": 14650828800.0,
+ "5460": 14650828800.0,
+ "5465": 14650828800.0,
+ "5470": 14650828800.0,
+ "5475": 14650828800.0,
+ "5480": 14650828800.0,
+ "5485": 14650828800.0,
+ "5490": 14650828800.0,
+ "5495": 14650828800.0,
+ "5500": 14650828800.0,
+ "5505": 14650828800.0,
+ "5510": 14650828800.0,
+ "5515": 14650828800.0,
+ "5520": 14650828800.0,
+ "5525": 14650828800.0,
+ "5530": 14650828800.0,
+ "5535": 14650828800.0,
+ "5540": 14650828800.0,
+ "5545": 14650828800.0,
+ "5550": 14650828800.0,
+ "5555": 14650828800.0,
+ "5560": 14650828800.0,
+ "5565": 14650828800.0,
+ "5570": 14650828800.0,
+ "5575": 14650828800.0,
+ "5580": 14650828800.0,
+ "5585": 14650828800.0,
+ "5590": 14650828800.0,
+ "5595": 14650828800.0,
+ "5600": 14650828800.0,
+ "5605": 14650828800.0,
+ "5610": 14650828800.0,
+ "5615": 14650828800.0,
+ "5620": 14650828800.0,
+ "5625": 14650828800.0,
+ "5630": 14650828800.0,
+ "5635": 14650828800.0,
+ "5640": 14650828800.0,
+ "5645": 14650828800.0,
+ "5650": 14650828800.0,
+ "5655": 14650828800.0,
+ "5660": 14650828800.0,
+ "5665": 14650828800.0,
+ "5670": 14650828800.0,
+ "5675": 14650828800.0,
+ "5680": 14650828800.0,
+ "5685": 14650828800.0,
+ "5690": 14650828800.0,
+ "5695": 14650828800.0,
+ "5700": 14650828800.0,
+ "5705": 14650828800.0,
+ "5710": 14650828800.0,
+ "5715": 14650828800.0,
+ "5720": 14650828800.0,
+ "5725": 14650828800.0,
+ "5730": 14650828800.0,
+ "5735": 14650828800.0,
+ "5740": 14650828800.0,
+ "5745": 14650828800.0,
+ "5750": 14650828800.0,
+ "5755": 14650828800.0,
+ "5760": 14650828800.0,
+ "5765": 14650828800.0,
+ "5770": 14650828800.0,
+ "5775": 14650828800.0,
+ "5780": 14650828800.0,
+ "5785": 14650828800.0,
+ "5790": 14650828800.0,
+ "5795": 14650828800.0,
+ "5800": 14650828800.0,
+ "5805": 14650828800.0,
+ "5810": 14650828800.0,
+ "5815": 14650828800.0,
+ "5820": 14650828800.0,
+ "5825": 14650828800.0,
+ "5830": 14650828800.0,
+ "5835": 14650828800.0,
+ "5840": 14650828800.0,
+ "5845": 14650828800.0,
+ "5850": 14650828800.0,
+ "5855": 14650828800.0,
+ "5860": 14650828800.0,
+ "5865": 14650828800.0,
+ "5870": 14650828800.0,
+ "5875": 14650828800.0,
+ "5880": 14650828800.0,
+ "5885": 14650828800.0,
+ "5890": 14650828800.0,
+ "5895": 14650828800.0,
+ "5900": 14650828800.0,
+ "5905": 14650828800.0,
+ "5910": 14650828800.0,
+ "5915": 14650828800.0,
+ "5920": 14650828800.0,
+ "5925": 14650828800.0,
+ "5930": 14650828800.0,
+ "5935": 14650828800.0,
+ "5940": 14650828800.0,
+ "5945": 14650828800.0,
+ "5950": 14650828800.0,
+ "5955": 14650828800.0,
+ "5960": 14650828800.0,
+ "5965": 14650828800.0,
+ "5970": 14650828800.0,
+ "5975": 14650828800.0,
+ "5980": 14650828800.0,
+ "5985": 14650828800.0,
+ "5990": 14650828800.0,
+ "5995": 14650828800.0,
+ "6000": 14650828800.0,
+ "6005": 14650828800.0,
+ "6010": 14650828800.0,
+ "6015": 14650828800.0,
+ "6020": 14650828800.0,
+ "6025": 14650828800.0,
+ "6030": 14650828800.0,
+ "6035": 14650828800.0,
+ "6040": 14650828800.0,
+ "6045": 14650828800.0,
+ "6050": 14650828800.0,
+ "6055": 14650828800.0,
+ "6060": 14650828800.0,
+ "6065": 14650828800.0,
+ "6070": 14650828800.0,
+ "6075": 14650828800.0,
+ "6080": 14650828800.0,
+ "6085": 14650828800.0,
+ "6090": 14650828800.0,
+ "6095": 14650828800.0,
+ "6100": 14650828800.0,
+ "6105": 14650828800.0,
+ "6110": 14650828800.0,
+ "6115": 14650828800.0,
+ "6120": 14650828800.0,
+ "6125": 14650828800.0,
+ "6130": 14650828800.0,
+ "6135": 14650828800.0,
+ "6140": 14650828800.0,
+ "6145": 14650828800.0,
+ "6150": 14650828800.0,
+ "6155": 14650828800.0,
+ "6160": 14650828800.0,
+ "6165": 14650828800.0,
+ "6170": 14650828800.0,
+ "6175": 14650828800.0,
+ "6180": 14650828800.0,
+ "6185": 14650828800.0,
+ "6190": 14650828800.0,
+ "6195": 14650828800.0,
+ "6200": 14650828800.0,
+ "6205": 14650828800.0,
+ "6210": 14650828800.0,
+ "6215": 14650828800.0,
+ "6220": 14650828800.0,
+ "6225": 14650828800.0,
+ "6230": 14650828800.0,
+ "6235": 14650828800.0,
+ "6240": 14650828800.0,
+ "6245": 14650828800.0,
+ "6250": 14650828800.0,
+ "6255": 14650828800.0,
+ "6260": 14650828800.0,
+ "6265": 14650828800.0,
+ "6270": 14650828800.0,
+ "6275": 14650828800.0,
+ "6280": 14650828800.0,
+ "6285": 14650828800.0,
+ "6290": 14650828800.0,
+ "6295": 14650828800.0,
+ "6300": 14650828800.0,
+ "6305": 14650828800.0,
+ "6310": 14650828800.0,
+ "6315": 14650828800.0,
+ "6320": 14650828800.0,
+ "6325": 14650828800.0,
+ "6330": 14650828800.0,
+ "6335": 14650828800.0,
+ "6340": 14650828800.0,
+ "6345": 14650828800.0,
+ "6350": 14650828800.0,
+ "6355": 14650828800.0,
+ "6360": 14650828800.0,
+ "6365": 14650828800.0,
+ "6370": 14650828800.0,
+ "6375": 14650828800.0,
+ "6380": 14650828800.0,
+ "6385": 14650828800.0,
+ "6390": 14650828800.0,
+ "6395": 14650828800.0,
+ "6400": 14650828800.0,
+ "6405": 14650828800.0,
+ "6410": 14650828800.0,
+ "6415": 14650828800.0,
+ "6420": 14650828800.0,
+ "6425": 14650828800.0,
+ "6430": 14650828800.0,
+ "6435": 14650828800.0,
+ "6440": 14650828800.0,
+ "6445": 14650828800.0,
+ "6450": 14650828800.0,
+ "6455": 14650828800.0,
+ "6460": 14650828800.0,
+ "6465": 14650828800.0,
+ "6470": 14650828800.0,
+ "6475": 14650828800.0,
+ "6480": 14650828800.0,
+ "6485": 14650828800.0,
+ "6490": 14650828800.0,
+ "6495": 14650828800.0,
+ "6500": 14650828800.0,
+ "6505": 14650828800.0,
+ "6510": 14650828800.0,
+ "6515": 14650828800.0,
+ "6520": 14650828800.0,
+ "6525": 14650828800.0,
+ "6530": 14650828800.0,
+ "6535": 14650828800.0,
+ "6540": 14650828800.0,
+ "6545": 14650828800.0,
+ "6550": 14650828800.0,
+ "6555": 14650828800.0,
+ "6560": 14650828800.0,
+ "6565": 14650828800.0,
+ "6570": 14650828800.0,
+ "6575": 14650828800.0,
+ "6580": 14650828800.0,
+ "6585": 14650828800.0,
+ "6590": 14650828800.0,
+ "6595": 14650828800.0,
+ "6600": 14650828800.0,
+ "6605": 14650828800.0,
+ "6610": 14650828800.0,
+ "6615": 14650828800.0,
+ "6620": 14650828800.0,
+ "6625": 14650828800.0,
+ "6630": 14650828800.0,
+ "6635": 14650828800.0,
+ "6640": 14650828800.0,
+ "6645": 14650828800.0,
+ "6650": 14650828800.0,
+ "6655": 14650828800.0,
+ "6660": 14650828800.0,
+ "6665": 14650828800.0,
+ "6670": 14650828800.0,
+ "6675": 14650828800.0,
+ "6680": 14650828800.0,
+ "6685": 14650828800.0,
+ "6690": 14650828800.0,
+ "6695": 14650828800.0,
+ "6700": 14650828800.0,
+ "6705": 14650828800.0,
+ "6710": 14650828800.0,
+ "6715": 14650828800.0,
+ "6720": 14650828800.0,
+ "6725": 14650828800.0,
+ "6730": 14650828800.0,
+ "6735": 14650828800.0,
+ "6740": 14650828800.0,
+ "6745": 14650828800.0,
+ "6750": 14650828800.0,
+ "6755": 14650828800.0,
+ "6760": 14650828800.0,
+ "6765": 14650828800.0,
+ "6770": 14650828800.0,
+ "6775": 14650828800.0,
+ "6780": 14650828800.0,
+ "6785": 14650828800.0,
+ "6790": 14650828800.0,
+ "6795": 14650828800.0,
+ "6800": 14650828800.0,
+ "6805": 14650828800.0,
+ "6810": 14650828800.0,
+ "6815": 14650828800.0,
+ "6820": 14650828800.0,
+ "6825": 14650828800.0,
+ "6830": 14650828800.0,
+ "6835": 14650828800.0,
+ "6840": 14650828800.0,
+ "6845": 14650828800.0,
+ "6850": 14650828800.0,
+ "6855": 14650828800.0,
+ "6860": 14650828800.0,
+ "6865": 14650828800.0,
+ "6870": 14650828800.0,
+ "6875": 14650828800.0,
+ "6880": 14650828800.0,
+ "6885": 14650828800.0,
+ "6890": 14650828800.0,
+ "6895": 14650828800.0,
+ "6900": 14650828800.0,
+ "6905": 14650828800.0,
+ "6910": 14650828800.0,
+ "6915": 14650828800.0,
+ "6920": 14650828800.0,
+ "6925": 14650828800.0,
+ "6930": 14650828800.0,
+ "6935": 14650828800.0,
+ "6940": 14650828800.0,
+ "6945": 14650828800.0,
+ "6950": 14650828800.0,
+ "6955": 14650828800.0,
+ "6960": 14650828800.0,
+ "6965": 14650828800.0,
+ "6970": 14650828800.0,
+ "6975": 14650828800.0,
+ "6980": 14650828800.0,
+ "6985": 14650828800.0,
+ "6990": 14650828800.0,
+ "6995": 14650828800.0,
+ "7000": 14650828800.0,
+ "7005": 14650828800.0,
+ "7010": 14650828800.0,
+ "7015": 14650828800.0,
+ "7020": 14650828800.0,
+ "7025": 14650828800.0,
+ "7030": 14650828800.0,
+ "7035": 14650828800.0,
+ "7040": 14650828800.0,
+ "7045": 14650828800.0,
+ "7050": 14650828800.0,
+ "7055": 14650828800.0,
+ "7060": 14650828800.0,
+ "7065": 14650828800.0,
+ "7070": 14650828800.0,
+ "7075": 14650828800.0,
+ "7080": 14650828800.0,
+ "7085": 14650828800.0,
+ "7090": 14650828800.0,
+ "7095": 14650828800.0,
+ "7100": 14650828800.0,
+ "7105": 14650828800.0,
+ "7110": 14650828800.0,
+ "7115": 14650828800.0,
+ "7120": 14650828800.0,
+ "7125": 14650828800.0,
+ "7130": 14650828800.0,
+ "7135": 14650828800.0,
+ "7140": 14650828800.0,
+ "7145": 14650828800.0,
+ "7150": 14650828800.0,
+ "7155": 14650828800.0,
+ "7160": 14650828800.0,
+ "7165": 14650828800.0,
+ "7170": 14650828800.0,
+ "7175": 14650828800.0,
+ "7180": 14650828800.0,
+ "7185": 14650828800.0,
+ "7190": 14650828800.0,
+ "7195": 14650828800.0,
+ "7200": 14650828800.0,
+ "7205": 14650828800.0,
+ "7210": 14650828800.0,
+ "7215": 14650828800.0,
+ "7220": 14650828800.0,
+ "7225": 14650828800.0,
+ "7230": 14650828800.0,
+ "7235": 14650828800.0,
+ "7240": 14650828800.0,
+ "7245": 14650828800.0,
+ "7250": 14650828800.0,
+ "7255": 14650828800.0,
+ "7260": 14650828800.0,
+ "7265": 14650828800.0,
+ "7270": 14650828800.0,
+ "7275": 14650828800.0,
+ "7280": 14650828800.0,
+ "7285": 14650828800.0,
+ "7290": 14650828800.0,
+ "7295": 14650828800.0,
+ "7300": 14650828800.0,
+ "7305": 14650828800.0,
+ "7310": 14650828800.0,
+ "7315": 14650828800.0,
+ "7320": 14650828800.0,
+ "7325": 14650828800.0,
+ "7330": 14650828800.0,
+ "7335": 14650828800.0,
+ "7340": 14650828800.0,
+ "7345": 14650828800.0,
+ "7350": 14650828800.0,
+ "7355": 14650828800.0,
+ "7360": 14650828800.0,
+ "7365": 14650828800.0,
+ "7370": 14650828800.0,
+ "7375": 14650828800.0,
+ "7380": 14650828800.0,
+ "7385": 14650828800.0,
+ "7390": 14650828800.0,
+ "7395": 14650828800.0,
+ "7400": 14650828800.0,
+ "7405": 14650828800.0,
+ "7410": 14650828800.0,
+ "7415": 14650828800.0,
+ "7420": 14650828800.0,
+ "7425": 14650828800.0,
+ "7430": 14650828800.0,
+ "7435": 14650828800.0,
+ "7440": 14650828800.0,
+ "7445": 14650828800.0,
+ "7450": 14650828800.0,
+ "7455": 14650828800.0,
+ "7460": 14650828800.0,
+ "7465": 14650828800.0,
+ "7470": 14650828800.0,
+ "7475": 14650828800.0,
+ "7480": 14650828800.0,
+ "7485": 14650828800.0,
+ "7490": 14650828800.0,
+ "7495": 14650828800.0,
+ "7500": 14650828800.0,
+ "7505": 14650828800.0,
+ "7510": 14650828800.0,
+ "7515": 14650828800.0,
+ "7520": 14650828800.0,
+ "7525": 14650828800.0,
+ "7530": 14650828800.0,
+ "7535": 14650828800.0,
+ "7540": 14650828800.0,
+ "7545": 14650828800.0,
+ "7550": 14650828800.0,
+ "7555": 14650828800.0,
+ "7560": 14650828800.0,
+ "7565": 14650828800.0,
+ "7570": 14650828800.0,
+ "7575": 14650828800.0,
+ "7580": 14650828800.0,
+ "7585": 14650828800.0,
+ "7590": 14650828800.0,
+ "7595": 14650828800.0,
+ "7600": 14650828800.0,
+ "7605": 14650828800.0,
+ "7610": 14650828800.0,
+ "7615": 14650828800.0,
+ "7620": 14650828800.0,
+ "7625": 14650828800.0,
+ "7630": 14650828800.0,
+ "7635": 14650828800.0,
+ "7640": 14650828800.0,
+ "7645": 14650828800.0,
+ "7650": 14650828800.0,
+ "7655": 14650828800.0,
+ "7660": 14650828800.0,
+ "7665": 14650828800.0,
+ "7670": 14650828800.0,
+ "7675": 14650828800.0,
+ "7680": 14650828800.0,
+ "7685": 14650828800.0,
+ "7690": 14650828800.0,
+ "7695": 14650828800.0,
+ "7700": 14650828800.0,
+ "7705": 14650828800.0,
+ "7710": 14650828800.0,
+ "7715": 14650828800.0,
+ "7720": 14650828800.0,
+ "7725": 14650828800.0,
+ "7730": 14650828800.0,
+ "7735": 14650828800.0,
+ "7740": 14650828800.0,
+ "7745": 14650828800.0,
+ "7750": 14650828800.0,
+ "7755": 14650828800.0,
+ "7760": 14650828800.0,
+ "7765": 14650828800.0,
+ "7770": 14650828800.0,
+ "7775": 14650828800.0,
+ "7780": 14650828800.0,
+ "7785": 14650828800.0,
+ "7790": 14650828800.0,
+ "7795": 14650828800.0,
+ "7800": 14650828800.0,
+ "7805": 14650828800.0,
+ "7810": 14650828800.0,
+ "7815": 14650828800.0,
+ "7820": 14650828800.0,
+ "7825": 14650828800.0,
+ "7830": 14650828800.0,
+ "7835": 14650828800.0,
+ "7840": 14650828800.0,
+ "7845": 14650828800.0,
+ "7850": 14650828800.0,
+ "7855": 14650828800.0,
+ "7860": 14650828800.0,
+ "7865": 14650828800.0,
+ "7870": 14650828800.0,
+ "7875": 14650828800.0,
+ "7880": 14650828800.0,
+ "7885": 14650828800.0,
+ "7890": 14650828800.0,
+ "7895": 14650828800.0,
+ "7900": 14650828800.0,
+ "7905": 14650828800.0,
+ "7910": 14650828800.0,
+ "7915": 14650828800.0,
+ "7920": 14650828800.0,
+ "7925": 14650828800.0,
+ "7930": 14650828800.0,
+ "7935": 14650828800.0,
+ "7940": 14650828800.0,
+ "7945": 14650828800.0,
+ "7950": 14650828800.0,
+ "7955": 14650828800.0,
+ "7960": 14650828800.0,
+ "7965": 14650828800.0,
+ "7970": 14650828800.0,
+ "7975": 14650828800.0,
+ "7980": 14650828800.0,
+ "7985": 14650828800.0,
+ "7990": 14650828800.0,
+ "7995": 14650828800.0,
+ "8000": 14650828800.0,
+ "8005": 14650828800.0,
+ "8010": 14650828800.0,
+ "8015": 14650828800.0,
+ "8020": 14650828800.0,
+ "8025": 14650828800.0,
+ "8030": 14650828800.0,
+ "8035": 14650828800.0,
+ "8040": 14650828800.0,
+ "8045": 14650828800.0,
+ "8050": 14650828800.0,
+ "8055": 14650828800.0,
+ "8060": 14650828800.0,
+ "8065": 14650828800.0,
+ "8070": 14650828800.0,
+ "8075": 14650828800.0,
+ "8080": 14650828800.0,
+ "8085": 14650828800.0,
+ "8090": 14650828800.0,
+ "8095": 14650828800.0,
+ "8100": 14650828800.0,
+ "8105": 14650828800.0,
+ "8110": 14650828800.0,
+ "8115": 14650828800.0,
+ "8120": 14650828800.0,
+ "8125": 14650828800.0,
+ "8130": 14650828800.0,
+ "8135": 14650828800.0,
+ "8140": 14650828800.0,
+ "8145": 14650828800.0,
+ "8150": 14650828800.0,
+ "8155": 14650828800.0,
+ "8160": 14650828800.0,
+ "8165": 14650828800.0,
+ "8170": 14650828800.0,
+ "8175": 14650828800.0,
+ "8180": 14650828800.0,
+ "8185": 14650828800.0,
+ "8190": 14650828800.0,
+ "8195": 14650828800.0,
+ "8200": 14650828800.0,
+ "8205": 14650828800.0,
+ "8210": 14650828800.0,
+ "8215": 14650828800.0,
+ "8220": 14650828800.0,
+ "8225": 14650828800.0,
+ "8230": 14650828800.0,
+ "8235": 14650828800.0,
+ "8240": 14650828800.0,
+ "8245": 14650828800.0,
+ "8250": 14650828800.0,
+ "8255": 14650828800.0,
+ "8260": 14650828800.0,
+ "8265": 14650828800.0,
+ "8270": 14650828800.0,
+ "8275": 14650828800.0,
+ "8280": 14650828800.0,
+ "8285": 14650828800.0,
+ "8290": 14650828800.0,
+ "8295": 14650828800.0,
+ "8300": 14650828800.0,
+ "8305": 14650828800.0,
+ "8310": 14650828800.0,
+ "8315": 14650828800.0,
+ "8320": 14650828800.0,
+ "8325": 14650828800.0,
+ "8330": 14650828800.0,
+ "8335": 14650828800.0,
+ "8340": 14650828800.0,
+ "8345": 14650828800.0,
+ "8350": 14650828800.0,
+ "8355": 14650828800.0,
+ "8360": 14650828800.0,
+ "8365": 14650828800.0,
+ "8370": 14650828800.0,
+ "8375": 14650828800.0,
+ "8380": 14650828800.0,
+ "8385": 14650828800.0,
+ "8390": 14650828800.0,
+ "8395": 14650828800.0,
+ "8400": 14650828800.0,
+ "8405": 14650828800.0,
+ "8410": 14650828800.0,
+ "8415": 14650828800.0,
+ "8420": 14650828800.0,
+ "8425": 14650828800.0,
+ "8430": 14650828800.0,
+ "8435": 14650828800.0,
+ "8440": 14650828800.0,
+ "8445": 14650828800.0,
+ "8450": 14650828800.0,
+ "8455": 14650828800.0,
+ "8460": 14650828800.0,
+ "8465": 14650828800.0,
+ "8470": 14650828800.0,
+ "8475": 14650828800.0,
+ "8480": 14650828800.0,
+ "8485": 14650828800.0,
+ "8490": 14650828800.0,
+ "8495": 14650828800.0,
+ "8500": 14650828800.0,
+ "8505": 14650828800.0,
+ "8510": 14650828800.0,
+ "8515": 14650828800.0,
+ "8520": 14650828800.0,
+ "8525": 14650828800.0,
+ "8530": 14650828800.0,
+ "8535": 14650828800.0,
+ "8540": 14650828800.0,
+ "8545": 14650828800.0,
+ "8550": 14650828800.0,
+ "8555": 14650828800.0,
+ "8560": 14650828800.0,
+ "8565": 14650828800.0,
+ "8570": 14650828800.0,
+ "8575": 14650828800.0,
+ "8580": 14650828800.0,
+ "8585": 14650828800.0,
+ "8590": 14650828800.0,
+ "8595": 14650828800.0,
+ "8600": 14650828800.0,
+ "8605": 14650828800.0,
+ "8610": 14650828800.0,
+ "8615": 14650828800.0,
+ "8620": 14650828800.0,
+ "8625": 14650828800.0,
+ "8630": 14650828800.0,
+ "8635": 14650828800.0,
+ "8640": 14650828800.0,
+ "8645": 14650828800.0,
+ "8650": 14650828800.0,
+ "8655": 14650828800.0,
+ "8660": 14650828800.0,
+ "8665": 14650828800.0,
+ "8670": 14650828800.0,
+ "8675": 14650828800.0,
+ "8680": 14650828800.0,
+ "8685": 14650828800.0,
+ "8690": 14650828800.0,
+ "8695": 14650828800.0,
+ "8700": 14650828800.0,
+ "8705": 14650828800.0,
+ "8710": 14650828800.0,
+ "8715": 14650828800.0,
+ "8720": 14650828800.0,
+ "8725": 14650828800.0,
+ "8730": 14650828800.0,
+ "8735": 14650828800.0,
+ "8740": 14650828800.0,
+ "8745": 14650828800.0,
+ "8750": 14650828800.0,
+ "8755": 14650828800.0,
+ "8760": 14650828800.0,
+ "8765": 14650828800.0,
+ "8770": 14650828800.0,
+ "8775": 14650828800.0,
+ "8780": 14650828800.0,
+ "8785": 14650828800.0,
+ "8790": 14650828800.0,
+ "8795": 14650828800.0,
+ "8800": 14650828800.0,
+ "8805": 14650828800.0,
+ "8810": 14650828800.0,
+ "8815": 14650828800.0,
+ "8820": 14650828800.0,
+ "8825": 14650828800.0,
+ "8830": 14650828800.0,
+ "8835": 14650828800.0,
+ "8840": 14650828800.0,
+ "8845": 14650828800.0,
+ "8850": 14650828800.0,
+ "8855": 14650828800.0,
+ "8860": 14650828800.0,
+ "8865": 14650828800.0,
+ "8870": 14650828800.0,
+ "8875": 14650828800.0,
+ "8880": 14650828800.0,
+ "8885": 14650828800.0,
+ "8890": 14650828800.0,
+ "8895": 14650828800.0,
+ "8900": 14650828800.0,
+ "8905": 14650828800.0,
+ "8910": 14650828800.0,
+ "8915": 14650828800.0,
+ "8920": 14650828800.0,
+ "8925": 14650828800.0,
+ "8930": 14650828800.0,
+ "8935": 14650828800.0,
+ "8940": 14650828800.0,
+ "8945": 14650828800.0,
+ "8950": 14650828800.0,
+ "8955": 14650828800.0,
+ "8960": 14650828800.0,
+ "8965": 14650828800.0,
+ "8970": 14650828800.0,
+ "8975": 14650828800.0,
+ "8980": 14650828800.0,
+ "8985": 14650828800.0,
+ "8990": 14650828800.0,
+ "8995": 14650828800.0,
+ "9000": 14650828800.0,
+ "9005": 14650828800.0,
+ "9010": 14650828800.0,
+ "9015": 14650828800.0,
+ "9020": 14650828800.0,
+ "9025": 14650828800.0,
+ "9030": 14650828800.0,
+ "9035": 14650828800.0,
+ "9040": 14650828800.0,
+ "9045": 14650828800.0,
+ "9050": 14650828800.0,
+ "9055": 14650828800.0,
+ "9060": 14650828800.0,
+ "9065": 14650828800.0,
+ "9070": 14650828800.0,
+ "9075": 14650828800.0,
+ "9080": 14650828800.0,
+ "9085": 14650828800.0,
+ "9090": 14650828800.0,
+ "9095": 14650828800.0,
+ "9100": 14650828800.0,
+ "9105": 14650828800.0,
+ "9110": 14650828800.0,
+ "9115": 14650828800.0,
+ "9120": 14650828800.0,
+ "9125": 14650828800.0,
+ "9130": 14650828800.0,
+ "9135": 14650828800.0,
+ "9140": 14650828800.0,
+ "9145": 14650828800.0,
+ "9150": 14650828800.0,
+ "9155": 14650828800.0,
+ "9160": 14650828800.0,
+ "9165": 14650828800.0,
+ "9170": 14650828800.0,
+ "9175": 14650828800.0,
+ "9180": 14650828800.0,
+ "9185": 14650828800.0,
+ "9190": 14650828800.0,
+ "9195": 14650828800.0,
+ "9200": 14650828800.0,
+ "9205": 14650828800.0,
+ "9210": 14650828800.0,
+ "9215": 14650828800.0,
+ "9220": 14650828800.0,
+ "9225": 14650828800.0,
+ "9230": 14650828800.0,
+ "9235": 14650828800.0,
+ "9240": 14650828800.0,
+ "9245": 14650828800.0,
+ "9250": 14650828800.0,
+ "9255": 14650828800.0,
+ "9260": 14650828800.0,
+ "9265": 14650828800.0,
+ "9270": 14650828800.0,
+ "9275": 14650828800.0,
+ "9280": 14650828800.0,
+ "9285": 14650828800.0,
+ "9290": 14650828800.0,
+ "9295": 14650828800.0,
+ "9300": 14650828800.0,
+ "9305": 14650828800.0,
+ "9310": 14650828800.0,
+ "9315": 14650828800.0,
+ "9320": 14650828800.0,
+ "9325": 14650828800.0,
+ "9330": 14650828800.0,
+ "9335": 14650828800.0,
+ "9340": 14650828800.0,
+ "9345": 14650828800.0,
+ "9350": 14650828800.0,
+ "9355": 14650828800.0,
+ "9360": 14650828800.0,
+ "9365": 14650828800.0,
+ "9370": 14650828800.0,
+ "9375": 14650828800.0,
+ "9380": 14650828800.0,
+ "9385": 14650828800.0,
+ "9390": 14650828800.0,
+ "9395": 14650828800.0,
+ "9400": 14650828800.0,
+ "9405": 14650828800.0,
+ "9410": 14650828800.0,
+ "9415": 14650828800.0,
+ "9420": 14650828800.0,
+ "9425": 14650828800.0,
+ "9430": 14650828800.0,
+ "9435": 14650828800.0,
+ "9440": 14650828800.0,
+ "9445": 14650828800.0,
+ "9450": 14650828800.0,
+ "9455": 14650828800.0,
+ "9460": 14650828800.0,
+ "9465": 14650828800.0,
+ "9470": 14650828800.0,
+ "9475": 14650828800.0,
+ "9480": 14650828800.0,
+ "9485": 14650828800.0,
+ "9490": 14650828800.0,
+ "9495": 14650828800.0,
+ "9500": 14650828800.0,
+ "9505": 14650828800.0,
+ "9510": 14650828800.0,
+ "9515": 14650828800.0,
+ "9520": 14650828800.0,
+ "9525": 14650828800.0,
+ "9530": 14650828800.0,
+ "9535": 14650828800.0,
+ "9540": 14650828800.0,
+ "9545": 14650828800.0,
+ "9550": 14650828800.0,
+ "9555": 14650828800.0,
+ "9560": 14650828800.0,
+ "9565": 14650828800.0,
+ "9570": 14650828800.0,
+ "9575": 14650828800.0,
+ "9580": 14650828800.0,
+ "9585": 14650828800.0,
+ "9590": 14650828800.0,
+ "9595": 14650828800.0,
+ "9600": 14650828800.0,
+ "9605": 14650828800.0,
+ "9610": 14650828800.0,
+ "9615": 14650828800.0,
+ "9620": 14650828800.0,
+ "9625": 14650828800.0,
+ "9630": 14650828800.0,
+ "9635": 14650828800.0,
+ "9640": 14650828800.0,
+ "9645": 14650828800.0,
+ "9650": 14650828800.0,
+ "9655": 14650828800.0,
+ "9660": 14650828800.0,
+ "9665": 14650828800.0,
+ "9670": 14650828800.0,
+ "9675": 14650828800.0,
+ "9680": 14650828800.0,
+ "9685": 14650828800.0,
+ "9690": 14650828800.0,
+ "9695": 14650828800.0,
+ "9700": 14650828800.0,
+ "9705": 14650828800.0,
+ "9710": 14650828800.0,
+ "9715": 14650828800.0,
+ "9720": 14650828800.0,
+ "9725": 14650828800.0,
+ "9730": 14650828800.0,
+ "9735": 14650828800.0,
+ "9740": 14650828800.0,
+ "9745": 14650828800.0,
+ "9750": 14650828800.0,
+ "9755": 14650828800.0,
+ "9760": 14650828800.0,
+ "9765": 14650828800.0,
+ "9770": 14650828800.0,
+ "9775": 14650828800.0,
+ "9780": 14650828800.0,
+ "9785": 14650828800.0,
+ "9790": 14650828800.0,
+ "9795": 14650828800.0,
+ "9800": 14650828800.0,
+ "9805": 14650828800.0,
+ "9810": 14650828800.0,
+ "9815": 14650828800.0,
+ "9820": 14650828800.0,
+ "9825": 14650828800.0,
+ "9830": 14650828800.0,
+ "9835": 14650828800.0,
+ "9840": 14650828800.0,
+ "9845": 14650828800.0,
+ "9850": 14650828800.0,
+ "9855": 14650828800.0,
+ "9860": 14650828800.0,
+ "9865": 14650828800.0,
+ "9870": 14650828800.0,
+ "9875": 14650828800.0,
+ "9880": 14650828800.0,
+ "9885": 14650828800.0,
+ "9890": 14650828800.0,
+ "9895": 14650828800.0,
+ "9900": 14650828800.0,
+ "9905": 14650828800.0,
+ "9910": 14650828800.0,
+ "9915": 14650828800.0,
+ "9920": 14650828800.0,
+ "9925": 14650828800.0,
+ "9930": 14650828800.0,
+ "9935": 14650828800.0,
+ "9940": 14650828800.0,
+ "9945": 14650828800.0,
+ "9950": 14650828800.0,
+ "9955": 14650828800.0,
+ "9960": 14650828800.0,
+ "9965": 14650828800.0,
+ "9970": 14650828800.0,
+ "9975": 14650828800.0,
+ "9980": 14650828800.0,
+ "9985": 14650828800.0,
+ "9990": 14650828800.0,
+ "9995": 14650828800.0,
+ "10000": 14650828800.0,
+ "10005": 14650828800.0,
+ "10010": 14650828800.0,
+ "10015": 14650828800.0,
+ "10020": 14650828800.0,
+ "10025": 14650828800.0,
+ "10030": 14650828800.0,
+ "10035": 14650828800.0,
+ "10040": 14650828800.0,
+ "10045": 14650828800.0,
+ "10050": 14650828800.0,
+ "10055": 14650828800.0,
+ "10060": 14650828800.0,
+ "10065": 14650828800.0,
+ "10070": 14650828800.0,
+ "10075": 14650828800.0,
+ "10080": 14650828800.0,
+ "10085": 14650828800.0,
+ "10090": 14650828800.0,
+ "10095": 14650828800.0,
+ "10100": 14650828800.0,
+ "10105": 14650828800.0,
+ "10110": 14650828800.0,
+ "10115": 14650828800.0,
+ "10120": 14650828800.0,
+ "10125": 14650828800.0,
+ "10130": 14650828800.0,
+ "10135": 14650828800.0,
+ "10140": 14650828800.0,
+ "10145": 14650828800.0,
+ "10150": 14650828800.0,
+ "10155": 14650828800.0,
+ "10160": 14650828800.0,
+ "10165": 14650828800.0,
+ "10170": 14650828800.0,
+ "10175": 14650828800.0,
+ "10180": 14650828800.0,
+ "10185": 14650828800.0,
+ "10190": 14650828800.0,
+ "10195": 14650828800.0,
+ "10200": 14650828800.0,
+ "10205": 14650828800.0,
+ "10210": 14650828800.0,
+ "10215": 14650828800.0,
+ "10220": 14650828800.0,
+ "10225": 14650828800.0,
+ "10230": 14650828800.0,
+ "10235": 14650828800.0,
+ "10240": 14650828800.0,
+ "10245": 14650828800.0,
+ "10250": 14650828800.0,
+ "10255": 14650828800.0,
+ "10260": 14650828800.0,
+ "10265": 14650828800.0,
+ "10270": 14650828800.0,
+ "10275": 14650828800.0,
+ "10280": 14650828800.0,
+ "10285": 14650828800.0,
+ "10290": 14650828800.0,
+ "10295": 14650828800.0,
+ "10300": 14650828800.0,
+ "10305": 14650828800.0,
+ "10310": 14650828800.0,
+ "10315": 14650828800.0,
+ "10320": 14650828800.0,
+ "10325": 14650828800.0,
+ "10330": 14650828800.0,
+ "10335": 14650828800.0,
+ "10340": 14650828800.0,
+ "10345": 14650828800.0,
+ "10350": 14650828800.0,
+ "10355": 14650828800.0,
+ "10360": 14650828800.0,
+ "10365": 14650828800.0,
+ "10370": 14650828800.0,
+ "10375": 14650828800.0,
+ "10380": 14650828800.0,
+ "10385": 14650828800.0,
+ "10390": 14650828800.0,
+ "10395": 14650828800.0,
+ "10400": 14650828800.0,
+ "10405": 14650828800.0,
+ "10410": 14650828800.0,
+ "10415": 14650828800.0,
+ "10420": 14650828800.0,
+ "10425": 14650828800.0,
+ "10430": 14650828800.0,
+ "10435": 14650828800.0,
+ "10440": 14650828800.0,
+ "10445": 14650828800.0,
+ "10450": 14650828800.0,
+ "10455": 14650828800.0,
+ "10460": 14650828800.0,
+ "10465": 14650828800.0,
+ "10470": 14650828800.0,
+ "10475": 14650828800.0,
+ "10480": 14650828800.0,
+ "10485": 14650828800.0,
+ "10490": 14650828800.0,
+ "10495": 14650828800.0,
+ "10500": 14650828800.0,
+ "10505": 14650828800.0,
+ "10510": 14650828800.0,
+ "10515": 14650828800.0,
+ "10520": 14650828800.0,
+ "10525": 14650828800.0,
+ "10530": 14650828800.0,
+ "10535": 14650828800.0,
+ "10540": 14650828800.0,
+ "10545": 14650828800.0,
+ "10550": 14650828800.0,
+ "10555": 14650828800.0,
+ "10560": 14650828800.0,
+ "10565": 14650828800.0,
+ "10570": 14650828800.0,
+ "10575": 14650828800.0,
+ "10580": 14650828800.0,
+ "10585": 14650828800.0,
+ "10590": 14650828800.0,
+ "10595": 14650828800.0,
+ "10600": 14650828800.0,
+ "10605": 14650828800.0,
+ "10610": 14650828800.0,
+ "10615": 14650828800.0,
+ "10620": 14650828800.0,
+ "10625": 14650828800.0,
+ "10630": 14650828800.0,
+ "10635": 14650828800.0,
+ "10640": 14650828800.0,
+ "10645": 14650828800.0,
+ "10650": 14650828800.0,
+ "10655": 14650828800.0,
+ "10660": 14650828800.0,
+ "10665": 14650828800.0,
+ "10670": 14650828800.0,
+ "10675": 14650828800.0,
+ "10680": 14650828800.0,
+ "10685": 14650828800.0,
+ "10690": 14650828800.0,
+ "10695": 14650828800.0,
+ "10700": 14650828800.0,
+ "10705": 14650828800.0,
+ "10710": 14650828800.0,
+ "10715": 14650828800.0,
+ "10720": 14650828800.0,
+ "10725": 14650828800.0,
+ "10730": 14650828800.0,
+ "10735": 14650828800.0,
+ "10740": 14650828800.0,
+ "10745": 14650828800.0,
+ "10750": 14650828800.0,
+ "10755": 14650828800.0,
+ "10760": 14650828800.0,
+ "10765": 14650828800.0,
+ "10770": 14650828800.0,
+ "10775": 14650828800.0,
+ "10780": 14650828800.0,
+ "10785": 14650828800.0,
+ "10790": 14650828800.0,
+ "10795": 14650828800.0,
+ "10800": 14650828800.0,
+ "10805": 14650828800.0,
+ "10810": 14650828800.0,
+ "10815": 14650828800.0,
+ "10820": 14650828800.0,
+ "10825": 14650828800.0,
+ "10830": 14650828800.0,
+ "10835": 14650828800.0,
+ "10840": 14650828800.0,
+ "10845": 14650828800.0,
+ "10850": 14650828800.0,
+ "10855": 14650828800.0,
+ "10860": 14650828800.0,
+ "10865": 14650828800.0,
+ "10870": 14650828800.0,
+ "10875": 14650828800.0,
+ "10880": 14650828800.0,
+ "10885": 14650828800.0,
+ "10890": 14650828800.0,
+ "10895": 14650828800.0,
+ "10900": 14650828800.0,
+ "10905": 14650828800.0,
+ "10910": 14650828800.0,
+ "10915": 14650828800.0,
+ "10920": 14650828800.0,
+ "10925": 14650828800.0,
+ "10930": 14650828800.0,
+ "10935": 14650828800.0,
+ "10940": 14650828800.0,
+ "10945": 14650828800.0,
+ "10950": 14650828800.0,
+ "10955": 14650828800.0,
+ "10960": 14650828800.0,
+ "10965": 14650828800.0,
+ "10970": 14650828800.0,
+ "10975": 14650828800.0,
+ "10980": 14650828800.0,
+ "10985": 14650828800.0,
+ "10990": 14650828800.0,
+ "10995": 14650828800.0,
+ "11000": 14650828800.0,
+ "11005": 14650828800.0,
+ "11010": 14650828800.0,
+ "11015": 14650828800.0,
+ "11020": 14650828800.0,
+ "11025": 14650828800.0,
+ "11030": 14650828800.0,
+ "11035": 14650828800.0,
+ "11040": 14650828800.0,
+ "11045": 14650828800.0,
+ "11050": 14650828800.0,
+ "11055": 14650828800.0,
+ "11060": 14650828800.0,
+ "11065": 14650828800.0,
+ "11070": 14650828800.0,
+ "11075": 14650828800.0,
+ "11080": 14650828800.0,
+ "11085": 14650828800.0,
+ "11090": 14650828800.0,
+ "11095": 14650828800.0,
+ "11100": 14650828800.0,
+ "11105": 14650828800.0,
+ "11110": 14650828800.0,
+ "11115": 14650828800.0,
+ "11120": 14650828800.0,
+ "11125": 14650828800.0,
+ "11130": 14650828800.0,
+ "11135": 14650828800.0,
+ "11140": 14650828800.0,
+ "11145": 14650828800.0,
+ "11150": 14650828800.0,
+ "11155": 14650828800.0,
+ "11160": 14650828800.0,
+ "11165": 14650828800.0,
+ "11170": 14650828800.0,
+ "11175": 14650828800.0,
+ "11180": 14650828800.0,
+ "11185": 14650828800.0,
+ "11190": 14650828800.0,
+ "11195": 14650828800.0,
+ "11200": 14650828800.0,
+ "11205": 14650828800.0,
+ "11210": 14650828800.0,
+ "11215": 14650828800.0,
+ "11220": 14650828800.0,
+ "11225": 14650828800.0,
+ "11230": 14650828800.0,
+ "11235": 14650828800.0,
+ "11240": 14650828800.0,
+ "11245": 14650828800.0,
+ "11250": 14650828800.0,
+ "11255": 14650828800.0,
+ "11260": 14650828800.0,
+ "11265": 14650828800.0,
+ "11270": 14650828800.0,
+ "11275": 14650828800.0,
+ "11280": 14650828800.0,
+ "11285": 14650828800.0,
+ "11290": 14650828800.0,
+ "11295": 14650828800.0,
+ "11300": 14650828800.0,
+ "11305": 14650828800.0,
+ "11310": 14650828800.0,
+ "11315": 14650828800.0,
+ "11320": 14650828800.0,
+ "11325": 14650828800.0,
+ "11330": 14650828800.0,
+ "11335": 14650828800.0,
+ "11340": 14650828800.0,
+ "11345": 14650828800.0,
+ "11350": 14650828800.0,
+ "11355": 14650828800.0,
+ "11360": 14650828800.0,
+ "11365": 14650828800.0,
+ "11370": 14650828800.0,
+ "11375": 14650828800.0,
+ "11380": 14650828800.0,
+ "11385": 14650828800.0,
+ "11390": 14650828800.0,
+ "11395": 14650828800.0,
+ "11400": 14650828800.0,
+ "11405": 14650828800.0,
+ "11410": 14650828800.0,
+ "11415": 14650828800.0,
+ "11420": 14650828800.0,
+ "11425": 14650828800.0,
+ "11430": 14650828800.0,
+ "11435": 14650828800.0,
+ "11440": 14650828800.0,
+ "11445": 14650828800.0,
+ "11450": 14650828800.0,
+ "11455": 14650828800.0,
+ "11460": 14650828800.0,
+ "11465": 14650828800.0,
+ "11470": 14650828800.0,
+ "11475": 14650828800.0,
+ "11480": 14650828800.0,
+ "11485": 14650828800.0,
+ "11490": 14650828800.0,
+ "11495": 14650828800.0,
+ "11500": 14650828800.0,
+ "11505": 14650828800.0,
+ "11510": 14650828800.0,
+ "11515": 14650828800.0,
+ "11520": 14650828800.0,
+ "11525": 14650828800.0,
+ "11530": 14650828800.0,
+ "11535": 14650828800.0,
+ "11540": 14650828800.0,
+ "11545": 14650828800.0,
+ "11550": 14650828800.0,
+ "11555": 14650828800.0,
+ "11560": 14650828800.0,
+ "11565": 14650828800.0,
+ "11570": 14650828800.0,
+ "11575": 14650828800.0,
+ "11580": 14650828800.0,
+ "11585": 14650828800.0,
+ "11590": 14650828800.0,
+ "11595": 14650828800.0,
+ "11600": 14650828800.0,
+ "11605": 14650828800.0,
+ "11610": 14650828800.0,
+ "11615": 14650828800.0,
+ "11620": 14650828800.0,
+ "11625": 14650828800.0,
+ "11630": 14650828800.0,
+ "11635": 14650828800.0,
+ "11640": 14650828800.0,
+ "11645": 14650828800.0,
+ "11650": 14650828800.0,
+ "11655": 14650828800.0,
+ "11660": 14650828800.0,
+ "11665": 14650828800.0,
+ "11670": 14650828800.0,
+ "11675": 14650828800.0,
+ "11680": 14650828800.0,
+ "11685": 14650828800.0,
+ "11690": 14650828800.0,
+ "11695": 14650828800.0,
+ "11700": 14650828800.0,
+ "11705": 14650828800.0,
+ "11710": 14650828800.0,
+ "11715": 14650828800.0,
+ "11720": 14650828800.0,
+ "11725": 14650828800.0,
+ "11730": 14650828800.0,
+ "11735": 14650828800.0,
+ "11740": 14650828800.0,
+ "11745": 14650828800.0,
+ "11750": 14650828800.0,
+ "11755": 14650828800.0,
+ "11760": 14650828800.0,
+ "11765": 14650828800.0,
+ "11770": 14650828800.0,
+ "11775": 14650828800.0,
+ "11780": 14650828800.0,
+ "11785": 14650828800.0,
+ "11790": 14650828800.0,
+ "11795": 14650828800.0,
+ "11800": 14650828800.0,
+ "11805": 14650828800.0,
+ "11810": 14650828800.0,
+ "11815": 14650828800.0,
+ "11820": 14650828800.0,
+ "11825": 14650828800.0,
+ "11830": 14650828800.0,
+ "11835": 14650828800.0,
+ "11840": 14650828800.0,
+ "11845": 14650828800.0,
+ "11850": 14650828800.0,
+ "11855": 14650828800.0,
+ "11860": 14650828800.0,
+ "11865": 14650828800.0,
+ "11870": 14650828800.0,
+ "11875": 14650828800.0,
+ "11880": 14650828800.0,
+ "11885": 14650828800.0,
+ "11890": 14650828800.0,
+ "11895": 14650828800.0,
+ "11900": 14650828800.0,
+ "11905": 14650828800.0,
+ "11910": 14650828800.0,
+ "11915": 14650828800.0,
+ "11920": 14650828800.0,
+ "11925": 14650828800.0,
+ "11930": 14650828800.0,
+ "11935": 14650828800.0,
+ "11940": 14650828800.0,
+ "11945": 14650828800.0,
+ "11950": 14650828800.0,
+ "11955": 14650828800.0,
+ "11960": 14650828800.0,
+ "11965": 14650828800.0,
+ "11970": 14650828800.0,
+ "11975": 14650828800.0,
+ "11980": 14650828800.0,
+ "11985": 14650828800.0,
+ "11990": 14650828800.0,
+ "11995": 14650828800.0,
+ "12000": 14650828800.0,
+ "12005": 14650828800.0,
+ "12010": 14650828800.0,
+ "12015": 14650828800.0,
+ "12020": 14650828800.0,
+ "12025": 14650828800.0,
+ "12030": 14650828800.0,
+ "12035": 14650828800.0,
+ "12040": 14650828800.0,
+ "12045": 14650828800.0,
+ "12050": 14650828800.0,
+ "12055": 14650828800.0,
+ "12060": 14650828800.0,
+ "12065": 14650828800.0,
+ "12070": 14650828800.0,
+ "12075": 14650828800.0,
+ "12080": 14650828800.0,
+ "12085": 14650828800.0,
+ "12090": 14650828800.0,
+ "12095": 14650828800.0,
+ "12100": 14650828800.0,
+ "12105": 14650828800.0,
+ "12110": 14650828800.0,
+ "12115": 14650828800.0,
+ "12120": 14650828800.0,
+ "12125": 14650828800.0,
+ "12130": 14650828800.0,
+ "12135": 14650828800.0,
+ "12140": 14650828800.0,
+ "12145": 14650828800.0,
+ "12150": 14650828800.0,
+ "12155": 14650828800.0,
+ "12160": 14650828800.0,
+ "12165": 14650828800.0,
+ "12170": 14650828800.0,
+ "12175": 14650828800.0,
+ "12180": 14650828800.0,
+ "12185": 14650828800.0,
+ "12190": 14650828800.0,
+ "12195": 14650828800.0,
+ "12200": 14650828800.0,
+ "12205": 14650828800.0,
+ "12210": 14650828800.0,
+ "12215": 14650828800.0,
+ "12220": 14650828800.0,
+ "12225": 14650828800.0,
+ "12230": 14650828800.0,
+ "12235": 14650828800.0,
+ "12240": 14650828800.0,
+ "12245": 14650828800.0,
+ "12250": 14650828800.0,
+ "12255": 14650828800.0,
+ "12260": 14650828800.0,
+ "12265": 14650828800.0,
+ "12270": 14650828800.0,
+ "12275": 14650828800.0,
+ "12280": 14650828800.0,
+ "12285": 14650828800.0,
+ "12290": 14650828800.0,
+ "12295": 14650828800.0,
+ "12300": 14650828800.0,
+ "12305": 14650828800.0,
+ "12310": 14650828800.0,
+ "12315": 14650828800.0,
+ "12320": 14650828800.0,
+ "12325": 14650828800.0,
+ "12330": 14650828800.0,
+ "12335": 14650828800.0,
+ "12340": 14650828800.0,
+ "12345": 14650828800.0,
+ "12350": 14650828800.0,
+ "12355": 14650828800.0,
+ "12360": 14650828800.0,
+ "12365": 14650828800.0,
+ "12370": 14650828800.0,
+ "12375": 14650828800.0,
+ "12380": 14650828800.0,
+ "12385": 14650828800.0,
+ "12390": 14650828800.0,
+ "12395": 14650828800.0,
+ "12400": 14650828800.0,
+ "12405": 14650828800.0,
+ "12410": 14650828800.0,
+ "12415": 14650828800.0,
+ "12420": 14650828800.0,
+ "12425": 14650828800.0,
+ "12430": 14650828800.0,
+ "12435": 14650828800.0,
+ "12440": 14650828800.0,
+ "12445": 14650828800.0,
+ "12450": 14650828800.0,
+ "12455": 14650828800.0,
+ "12460": 14650828800.0,
+ "12465": 14650828800.0,
+ "12470": 14650828800.0,
+ "12475": 14650828800.0,
+ "12480": 14650828800.0,
+ "12485": 14650828800.0,
+ "12490": 14650828800.0,
+ "12495": 14650828800.0,
+ "12500": 14650828800.0,
+ "12505": 14650828800.0,
+ "12510": 14650828800.0,
+ "12515": 14650828800.0,
+ "12520": 14650828800.0,
+ "12525": 14650828800.0,
+ "12530": 14650828800.0,
+ "12535": 14650828800.0,
+ "12540": 14650828800.0,
+ "12545": 14650828800.0,
+ "12550": 14650828800.0,
+ "12555": 14650828800.0,
+ "12560": 14650828800.0,
+ "12565": 14650828800.0,
+ "12570": 14650828800.0,
+ "12575": 14650828800.0,
+ "12580": 14650828800.0,
+ "12585": 14650828800.0,
+ "12590": 14650828800.0,
+ "12595": 14650828800.0,
+ "12600": 14650828800.0,
+ "12605": 14650828800.0,
+ "12610": 14650828800.0,
+ "12615": 14650828800.0,
+ "12620": 14650828800.0,
+ "12625": 14650828800.0,
+ "12630": 14650828800.0,
+ "12635": 14650828800.0,
+ "12640": 14650828800.0,
+ "12645": 14650828800.0,
+ "12650": 14650828800.0,
+ "12655": 14650828800.0,
+ "12660": 14650828800.0,
+ "12665": 14650828800.0,
+ "12670": 14650828800.0,
+ "12675": 14650828800.0,
+ "12680": 14650828800.0,
+ "12685": 14650828800.0,
+ "12690": 14650828800.0,
+ "12695": 14650828800.0,
+ "12700": 14650828800.0,
+ "12705": 14650828800.0,
+ "12710": 14650828800.0,
+ "12715": 14650828800.0,
+ "12720": 14650828800.0,
+ "12725": 14650828800.0,
+ "12730": 14650828800.0,
+ "12735": 14650828800.0,
+ "12740": 14650828800.0,
+ "12745": 14650828800.0,
+ "12750": 14650828800.0,
+ "12755": 14650828800.0,
+ "12760": 14650828800.0,
+ "12765": 14650828800.0,
+ "12770": 14650828800.0,
+ "12775": 14650828800.0,
+ "12780": 14650828800.0,
+ "12785": 14650828800.0,
+ "12790": 14650828800.0,
+ "12795": 14650828800.0,
+ "12800": 14650828800.0,
+ "12805": 14650828800.0,
+ "12810": 14650828800.0,
+ "12815": 14650828800.0,
+ "12820": 14650828800.0,
+ "12825": 14650828800.0,
+ "12830": 14650828800.0,
+ "12835": 14650828800.0,
+ "12840": 14650828800.0,
+ "12845": 14650828800.0,
+ "12850": 14650828800.0,
+ "12855": 14650828800.0,
+ "12860": 14650828800.0,
+ "12865": 14650828800.0,
+ "12870": 14650828800.0,
+ "12875": 14650828800.0,
+ "12880": 14650828800.0,
+ "12885": 14650828800.0,
+ "12890": 14650828800.0,
+ "12895": 14650828800.0,
+ "12900": 14650828800.0,
+ "12905": 14650828800.0,
+ "12910": 14650828800.0,
+ "12915": 14650828800.0,
+ "12920": 14650828800.0,
+ "12925": 14650828800.0,
+ "12930": 14650828800.0,
+ "12935": 14650828800.0,
+ "12940": 14650828800.0,
+ "12945": 14650828800.0,
+ "12950": 14650828800.0,
+ "12955": 14650828800.0,
+ "12960": 14650828800.0,
+ "12965": 14650828800.0,
+ "12970": 14650828800.0,
+ "12975": 14650828800.0,
+ "12980": 14650828800.0,
+ "12985": 14650828800.0,
+ "12990": 14650828800.0,
+ "12995": 14650828800.0,
+ "13000": 14650828800.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": 33457934336.0,
+ "5": 34434887680.0,
+ "10": 34434887680.0,
+ "15": 34434887680.0,
+ "20": 34434887680.0,
+ "25": 34434887680.0,
+ "30": 34434887680.0,
+ "35": 34434887680.0,
+ "40": 34434887680.0,
+ "45": 34434887680.0,
+ "50": 34434887680.0,
+ "55": 34434887680.0,
+ "60": 34434887680.0,
+ "65": 34434887680.0,
+ "70": 34434887680.0,
+ "75": 34434887680.0,
+ "80": 34434887680.0,
+ "85": 34434887680.0,
+ "90": 34434887680.0,
+ "95": 34434887680.0,
+ "100": 34434887680.0,
+ "105": 34434887680.0,
+ "110": 34434887680.0,
+ "115": 34434887680.0,
+ "120": 34434887680.0,
+ "125": 34434887680.0,
+ "130": 34434887680.0,
+ "135": 34434887680.0,
+ "140": 34434887680.0,
+ "145": 34434887680.0,
+ "150": 34434887680.0,
+ "155": 34434887680.0,
+ "160": 34434887680.0,
+ "165": 34434887680.0,
+ "170": 34434887680.0,
+ "175": 34434887680.0,
+ "180": 34434887680.0,
+ "185": 34434887680.0,
+ "190": 34434887680.0,
+ "195": 34434887680.0,
+ "200": 34434887680.0,
+ "205": 34434887680.0,
+ "210": 34434887680.0,
+ "215": 34434887680.0,
+ "220": 34434887680.0,
+ "225": 34434887680.0,
+ "230": 34434887680.0,
+ "235": 34434887680.0,
+ "240": 34434887680.0,
+ "245": 34434887680.0,
+ "250": 34434887680.0,
+ "255": 34434887680.0,
+ "260": 34434887680.0,
+ "265": 34434887680.0,
+ "270": 34434887680.0,
+ "275": 34434887680.0,
+ "280": 34434887680.0,
+ "285": 34434887680.0,
+ "290": 34434887680.0,
+ "295": 34434887680.0,
+ "300": 34434887680.0,
+ "305": 34434887680.0,
+ "310": 34434887680.0,
+ "315": 34434887680.0,
+ "320": 34434887680.0,
+ "325": 34434887680.0,
+ "330": 34434887680.0,
+ "335": 34434887680.0,
+ "340": 34434887680.0,
+ "345": 34434887680.0,
+ "350": 34434887680.0,
+ "355": 34434887680.0,
+ "360": 34434887680.0,
+ "365": 34434887680.0,
+ "370": 34434887680.0,
+ "375": 34434887680.0,
+ "380": 34434887680.0,
+ "385": 34434887680.0,
+ "390": 34434887680.0,
+ "395": 34434887680.0,
+ "400": 34434887680.0,
+ "405": 34434887680.0,
+ "410": 34434887680.0,
+ "415": 34434887680.0,
+ "420": 34434887680.0,
+ "425": 34434887680.0,
+ "430": 34434887680.0,
+ "435": 34434887680.0,
+ "440": 34434887680.0,
+ "445": 34434887680.0,
+ "450": 34434887680.0,
+ "455": 34434887680.0,
+ "460": 34434887680.0,
+ "465": 34434887680.0,
+ "470": 34434887680.0,
+ "475": 34434887680.0,
+ "480": 34434887680.0,
+ "485": 34434887680.0,
+ "490": 34434887680.0,
+ "495": 34434887680.0,
+ "500": 34434887680.0,
+ "505": 34434887680.0,
+ "510": 34434887680.0,
+ "515": 34434887680.0,
+ "520": 34434887680.0,
+ "525": 34434887680.0,
+ "530": 34434887680.0,
+ "535": 34434887680.0,
+ "540": 34434887680.0,
+ "545": 34434887680.0,
+ "550": 34434887680.0,
+ "555": 34434887680.0,
+ "560": 34434887680.0,
+ "565": 34434887680.0,
+ "570": 34434887680.0,
+ "575": 34434887680.0,
+ "580": 34434887680.0,
+ "585": 34434887680.0,
+ "590": 34434887680.0,
+ "595": 34434887680.0,
+ "600": 34434887680.0,
+ "605": 34434887680.0,
+ "610": 34434887680.0,
+ "615": 34434887680.0,
+ "620": 34434887680.0,
+ "625": 34434887680.0,
+ "630": 34434887680.0,
+ "635": 34434887680.0,
+ "640": 34434887680.0,
+ "645": 34434887680.0,
+ "650": 34434887680.0,
+ "655": 34434887680.0,
+ "660": 34434887680.0,
+ "665": 34434887680.0,
+ "670": 34434887680.0,
+ "675": 34434887680.0,
+ "680": 34434887680.0,
+ "685": 34434887680.0,
+ "690": 34434887680.0,
+ "695": 34434887680.0,
+ "700": 34434887680.0,
+ "705": 34434887680.0,
+ "710": 34434887680.0,
+ "715": 34434887680.0,
+ "720": 34434887680.0,
+ "725": 34434887680.0,
+ "730": 34434887680.0,
+ "735": 34434887680.0,
+ "740": 34434887680.0,
+ "745": 34434887680.0,
+ "750": 34434887680.0,
+ "755": 34434887680.0,
+ "760": 34434887680.0,
+ "765": 34434887680.0,
+ "770": 34434887680.0,
+ "775": 34434887680.0,
+ "780": 34434887680.0,
+ "785": 34434887680.0,
+ "790": 34434887680.0,
+ "795": 34434887680.0,
+ "800": 34434887680.0,
+ "805": 34434887680.0,
+ "810": 34434887680.0,
+ "815": 34434887680.0,
+ "820": 34434887680.0,
+ "825": 34434887680.0,
+ "830": 34434887680.0,
+ "835": 34434887680.0,
+ "840": 34434887680.0,
+ "845": 34434887680.0,
+ "850": 34434887680.0,
+ "855": 34434887680.0,
+ "860": 34434887680.0,
+ "865": 34434887680.0,
+ "870": 34434887680.0,
+ "875": 34434887680.0,
+ "880": 34434887680.0,
+ "885": 34434887680.0,
+ "890": 34434887680.0,
+ "895": 34434887680.0,
+ "900": 34434887680.0,
+ "905": 34434887680.0,
+ "910": 34434887680.0,
+ "915": 34434887680.0,
+ "920": 34434887680.0,
+ "925": 34434887680.0,
+ "930": 34434887680.0,
+ "935": 34434887680.0,
+ "940": 34434887680.0,
+ "945": 34434887680.0,
+ "950": 34434887680.0,
+ "955": 34434887680.0,
+ "960": 34434887680.0,
+ "965": 34434887680.0,
+ "970": 34434887680.0,
+ "975": 34434887680.0,
+ "980": 34434887680.0,
+ "985": 34434887680.0,
+ "990": 34434887680.0,
+ "995": 34434887680.0,
+ "1000": 34434887680.0,
+ "1005": 34434887680.0,
+ "1010": 34434887680.0,
+ "1015": 34434887680.0,
+ "1020": 34434887680.0,
+ "1025": 34434887680.0,
+ "1030": 34434887680.0,
+ "1035": 34434887680.0,
+ "1040": 34434887680.0,
+ "1045": 34434887680.0,
+ "1050": 34434887680.0,
+ "1055": 34434887680.0,
+ "1060": 34434887680.0,
+ "1065": 34434887680.0,
+ "1070": 34434887680.0,
+ "1075": 34434887680.0,
+ "1080": 34434887680.0,
+ "1085": 34434887680.0,
+ "1090": 34434887680.0,
+ "1095": 34434887680.0,
+ "1100": 34434887680.0,
+ "1105": 34434887680.0,
+ "1110": 34434887680.0,
+ "1115": 34434887680.0,
+ "1120": 34434887680.0,
+ "1125": 34434887680.0,
+ "1130": 34434887680.0,
+ "1135": 34434887680.0,
+ "1140": 34434887680.0,
+ "1145": 34434887680.0,
+ "1150": 34434887680.0,
+ "1155": 34434887680.0,
+ "1160": 34434887680.0,
+ "1165": 34434887680.0,
+ "1170": 34434887680.0,
+ "1175": 34434887680.0,
+ "1180": 34434887680.0,
+ "1185": 34434887680.0,
+ "1190": 34434887680.0,
+ "1195": 34434887680.0,
+ "1200": 34434887680.0,
+ "1205": 34434887680.0,
+ "1210": 34434887680.0,
+ "1215": 34434887680.0,
+ "1220": 34434887680.0,
+ "1225": 34434887680.0,
+ "1230": 34434887680.0,
+ "1235": 34434887680.0,
+ "1240": 34434887680.0,
+ "1245": 34434887680.0,
+ "1250": 34434887680.0,
+ "1255": 34434887680.0,
+ "1260": 34434887680.0,
+ "1265": 34434887680.0,
+ "1270": 34434887680.0,
+ "1275": 34434887680.0,
+ "1280": 34434887680.0,
+ "1285": 34434887680.0,
+ "1290": 34434887680.0,
+ "1295": 34434887680.0,
+ "1300": 34434887680.0,
+ "1305": 34434887680.0,
+ "1310": 34434887680.0,
+ "1315": 34434887680.0,
+ "1320": 34434887680.0,
+ "1325": 34434887680.0,
+ "1330": 34434887680.0,
+ "1335": 34434887680.0,
+ "1340": 34434887680.0,
+ "1345": 34434887680.0,
+ "1350": 34434887680.0,
+ "1355": 34434887680.0,
+ "1360": 34434887680.0,
+ "1365": 34434887680.0,
+ "1370": 34434887680.0,
+ "1375": 34434887680.0,
+ "1380": 34434887680.0,
+ "1385": 34434887680.0,
+ "1390": 34434887680.0,
+ "1395": 34434887680.0,
+ "1400": 34434887680.0,
+ "1405": 34434887680.0,
+ "1410": 34434887680.0,
+ "1415": 34434887680.0,
+ "1420": 34434887680.0,
+ "1425": 34434887680.0,
+ "1430": 34434887680.0,
+ "1435": 34434887680.0,
+ "1440": 34434887680.0,
+ "1445": 34434887680.0,
+ "1450": 34434887680.0,
+ "1455": 34434887680.0,
+ "1460": 34434887680.0,
+ "1465": 34434887680.0,
+ "1470": 34434887680.0,
+ "1475": 34434887680.0,
+ "1480": 34434887680.0,
+ "1485": 34434887680.0,
+ "1490": 34434887680.0,
+ "1495": 34434887680.0,
+ "1500": 34434887680.0,
+ "1505": 34434887680.0,
+ "1510": 34434887680.0,
+ "1515": 34434887680.0,
+ "1520": 34434887680.0,
+ "1525": 34434887680.0,
+ "1530": 34434887680.0,
+ "1535": 34434887680.0,
+ "1540": 34434887680.0,
+ "1545": 34434887680.0,
+ "1550": 34434887680.0,
+ "1555": 34434887680.0,
+ "1560": 34434887680.0,
+ "1565": 34434887680.0,
+ "1570": 34434887680.0,
+ "1575": 34434887680.0,
+ "1580": 34434887680.0,
+ "1585": 34434887680.0,
+ "1590": 34434887680.0,
+ "1595": 34434887680.0,
+ "1600": 34434887680.0,
+ "1605": 34434887680.0,
+ "1610": 34434887680.0,
+ "1615": 34434887680.0,
+ "1620": 34434887680.0,
+ "1625": 34434887680.0,
+ "1630": 34434887680.0,
+ "1635": 34434887680.0,
+ "1640": 34434887680.0,
+ "1645": 34434887680.0,
+ "1650": 34434887680.0,
+ "1655": 34434887680.0,
+ "1660": 34434887680.0,
+ "1665": 34434887680.0,
+ "1670": 34434887680.0,
+ "1675": 34434887680.0,
+ "1680": 34434887680.0,
+ "1685": 34434887680.0,
+ "1690": 34434887680.0,
+ "1695": 34434887680.0,
+ "1700": 34434887680.0,
+ "1705": 34434887680.0,
+ "1710": 34434887680.0,
+ "1715": 34434887680.0,
+ "1720": 34434887680.0,
+ "1725": 34434887680.0,
+ "1730": 34434887680.0,
+ "1735": 34434887680.0,
+ "1740": 34434887680.0,
+ "1745": 34434887680.0,
+ "1750": 34434887680.0,
+ "1755": 34434887680.0,
+ "1760": 34434887680.0,
+ "1765": 34434887680.0,
+ "1770": 34434887680.0,
+ "1775": 34434887680.0,
+ "1780": 34434887680.0,
+ "1785": 34434887680.0,
+ "1790": 34434887680.0,
+ "1795": 34434887680.0,
+ "1800": 34434887680.0,
+ "1805": 34434887680.0,
+ "1810": 34434887680.0,
+ "1815": 34434887680.0,
+ "1820": 34434887680.0,
+ "1825": 34434887680.0,
+ "1830": 34434887680.0,
+ "1835": 34434887680.0,
+ "1840": 34434887680.0,
+ "1845": 34434887680.0,
+ "1850": 34434887680.0,
+ "1855": 34434887680.0,
+ "1860": 34434887680.0,
+ "1865": 34434887680.0,
+ "1870": 34434887680.0,
+ "1875": 34434887680.0,
+ "1880": 34434887680.0,
+ "1885": 34434887680.0,
+ "1890": 34434887680.0,
+ "1895": 34434887680.0,
+ "1900": 34434887680.0,
+ "1905": 34434887680.0,
+ "1910": 34434887680.0,
+ "1915": 34434887680.0,
+ "1920": 34434887680.0,
+ "1925": 34434887680.0,
+ "1930": 34434887680.0,
+ "1935": 34434887680.0,
+ "1940": 34434887680.0,
+ "1945": 34434887680.0,
+ "1950": 34434887680.0,
+ "1955": 34434887680.0,
+ "1960": 34434887680.0,
+ "1965": 34434887680.0,
+ "1970": 34434887680.0,
+ "1975": 34434887680.0,
+ "1980": 34434887680.0,
+ "1985": 34434887680.0,
+ "1990": 34434887680.0,
+ "1995": 34434887680.0,
+ "2000": 34434887680.0,
+ "2005": 34434887680.0,
+ "2010": 34434887680.0,
+ "2015": 34434887680.0,
+ "2020": 34434887680.0,
+ "2025": 34434887680.0,
+ "2030": 34434887680.0,
+ "2035": 34434887680.0,
+ "2040": 34434887680.0,
+ "2045": 34434887680.0,
+ "2050": 34434887680.0,
+ "2055": 34434887680.0,
+ "2060": 34434887680.0,
+ "2065": 34434887680.0,
+ "2070": 34434887680.0,
+ "2075": 34434887680.0,
+ "2080": 34434887680.0,
+ "2085": 34434887680.0,
+ "2090": 34434887680.0,
+ "2095": 34434887680.0,
+ "2100": 34434887680.0,
+ "2105": 34434887680.0,
+ "2110": 34434887680.0,
+ "2115": 34434887680.0,
+ "2120": 34434887680.0,
+ "2125": 34434887680.0,
+ "2130": 34434887680.0,
+ "2135": 34434887680.0,
+ "2140": 34434887680.0,
+ "2145": 34434887680.0,
+ "2150": 34434887680.0,
+ "2155": 34434887680.0,
+ "2160": 34434887680.0,
+ "2165": 34434887680.0,
+ "2170": 34434887680.0,
+ "2175": 34434887680.0,
+ "2180": 34434887680.0,
+ "2185": 34434887680.0,
+ "2190": 34434887680.0,
+ "2195": 34434887680.0,
+ "2200": 34434887680.0,
+ "2205": 34434887680.0,
+ "2210": 34434887680.0,
+ "2215": 34434887680.0,
+ "2220": 34434887680.0,
+ "2225": 34434887680.0,
+ "2230": 34434887680.0,
+ "2235": 34434887680.0,
+ "2240": 34434887680.0,
+ "2245": 34434887680.0,
+ "2250": 34434887680.0,
+ "2255": 34434887680.0,
+ "2260": 34434887680.0,
+ "2265": 34434887680.0,
+ "2270": 34434887680.0,
+ "2275": 34434887680.0,
+ "2280": 34434887680.0,
+ "2285": 34434887680.0,
+ "2290": 34434887680.0,
+ "2295": 34434887680.0,
+ "2300": 34434887680.0,
+ "2305": 34434887680.0,
+ "2310": 34434887680.0,
+ "2315": 34434887680.0,
+ "2320": 34434887680.0,
+ "2325": 34434887680.0,
+ "2330": 34434887680.0,
+ "2335": 34434887680.0,
+ "2340": 34434887680.0,
+ "2345": 34434887680.0,
+ "2350": 34434887680.0,
+ "2355": 34434887680.0,
+ "2360": 34434887680.0,
+ "2365": 34434887680.0,
+ "2370": 34434887680.0,
+ "2375": 34434887680.0,
+ "2380": 34434887680.0,
+ "2385": 34434887680.0,
+ "2390": 34434887680.0,
+ "2395": 34434887680.0,
+ "2400": 34434887680.0,
+ "2405": 34434887680.0,
+ "2410": 34434887680.0,
+ "2415": 34434887680.0,
+ "2420": 34434887680.0,
+ "2425": 34434887680.0,
+ "2430": 34434887680.0,
+ "2435": 34434887680.0,
+ "2440": 34434887680.0,
+ "2445": 34434887680.0,
+ "2450": 34434887680.0,
+ "2455": 34434887680.0,
+ "2460": 34434887680.0,
+ "2465": 34434887680.0,
+ "2470": 34434887680.0,
+ "2475": 34434887680.0,
+ "2480": 34434887680.0,
+ "2485": 34434887680.0,
+ "2490": 34434887680.0,
+ "2495": 34434887680.0,
+ "2500": 34434887680.0,
+ "2505": 34434887680.0,
+ "2510": 34434887680.0,
+ "2515": 34434887680.0,
+ "2520": 34434887680.0,
+ "2525": 34434887680.0,
+ "2530": 34434887680.0,
+ "2535": 34434887680.0,
+ "2540": 34434887680.0,
+ "2545": 34434887680.0,
+ "2550": 34434887680.0,
+ "2555": 34434887680.0,
+ "2560": 34434887680.0,
+ "2565": 34434887680.0,
+ "2570": 34434887680.0,
+ "2575": 34434887680.0,
+ "2580": 34434887680.0,
+ "2585": 34434887680.0,
+ "2590": 34434887680.0,
+ "2595": 34434887680.0,
+ "2600": 34434887680.0,
+ "2605": 34434887680.0,
+ "2610": 34434887680.0,
+ "2615": 34434887680.0,
+ "2620": 34434887680.0,
+ "2625": 34434887680.0,
+ "2630": 34434887680.0,
+ "2635": 34434887680.0,
+ "2640": 34434887680.0,
+ "2645": 34434887680.0,
+ "2650": 34434887680.0,
+ "2655": 34434887680.0,
+ "2660": 34434887680.0,
+ "2665": 34434887680.0,
+ "2670": 34434887680.0,
+ "2675": 34434887680.0,
+ "2680": 34434887680.0,
+ "2685": 34434887680.0,
+ "2690": 34434887680.0,
+ "2695": 34434887680.0,
+ "2700": 34434887680.0,
+ "2705": 34434887680.0,
+ "2710": 34434887680.0,
+ "2715": 34434887680.0,
+ "2720": 34434887680.0,
+ "2725": 34434887680.0,
+ "2730": 34434887680.0,
+ "2735": 34434887680.0,
+ "2740": 34434887680.0,
+ "2745": 34434887680.0,
+ "2750": 34434887680.0,
+ "2755": 34434887680.0,
+ "2760": 34434887680.0,
+ "2765": 34434887680.0,
+ "2770": 34434887680.0,
+ "2775": 34434887680.0,
+ "2780": 34434887680.0,
+ "2785": 34434887680.0,
+ "2790": 34434887680.0,
+ "2795": 34434887680.0,
+ "2800": 34434887680.0,
+ "2805": 34434887680.0,
+ "2810": 34434887680.0,
+ "2815": 34434887680.0,
+ "2820": 34434887680.0,
+ "2825": 34434887680.0,
+ "2830": 34434887680.0,
+ "2835": 34434887680.0,
+ "2840": 34434887680.0,
+ "2845": 34434887680.0,
+ "2850": 34434887680.0,
+ "2855": 34434887680.0,
+ "2860": 34434887680.0,
+ "2865": 34434887680.0,
+ "2870": 34434887680.0,
+ "2875": 34434887680.0,
+ "2880": 34434887680.0,
+ "2885": 34434887680.0,
+ "2890": 34434887680.0,
+ "2895": 34434887680.0,
+ "2900": 34434887680.0,
+ "2905": 34434887680.0,
+ "2910": 34434887680.0,
+ "2915": 34434887680.0,
+ "2920": 34434887680.0,
+ "2925": 34434887680.0,
+ "2930": 34434887680.0,
+ "2935": 34434887680.0,
+ "2940": 34434887680.0,
+ "2945": 34434887680.0,
+ "2950": 34434887680.0,
+ "2955": 34434887680.0,
+ "2960": 34434887680.0,
+ "2965": 34434887680.0,
+ "2970": 34434887680.0,
+ "2975": 34434887680.0,
+ "2980": 34434887680.0,
+ "2985": 34434887680.0,
+ "2990": 34434887680.0,
+ "2995": 34434887680.0,
+ "3000": 34434887680.0,
+ "3005": 34434887680.0,
+ "3010": 34434887680.0,
+ "3015": 34434887680.0,
+ "3020": 34434887680.0,
+ "3025": 34434887680.0,
+ "3030": 34434887680.0,
+ "3035": 34434887680.0,
+ "3040": 34434887680.0,
+ "3045": 34434887680.0,
+ "3050": 34434887680.0,
+ "3055": 34434887680.0,
+ "3060": 34434887680.0,
+ "3065": 34434887680.0,
+ "3070": 34434887680.0,
+ "3075": 34434887680.0,
+ "3080": 34434887680.0,
+ "3085": 34434887680.0,
+ "3090": 34434887680.0,
+ "3095": 34434887680.0,
+ "3100": 34434887680.0,
+ "3105": 34434887680.0,
+ "3110": 34434887680.0,
+ "3115": 34434887680.0,
+ "3120": 34434887680.0,
+ "3125": 34434887680.0,
+ "3130": 34434887680.0,
+ "3135": 34434887680.0,
+ "3140": 34434887680.0,
+ "3145": 34434887680.0,
+ "3150": 34434887680.0,
+ "3155": 34434887680.0,
+ "3160": 34434887680.0,
+ "3165": 34434887680.0,
+ "3170": 34434887680.0,
+ "3175": 34434887680.0,
+ "3180": 34434887680.0,
+ "3185": 34434887680.0,
+ "3190": 34434887680.0,
+ "3195": 34434887680.0,
+ "3200": 34434887680.0,
+ "3205": 34434887680.0,
+ "3210": 34434887680.0,
+ "3215": 34434887680.0,
+ "3220": 34434887680.0,
+ "3225": 34434887680.0,
+ "3230": 34434887680.0,
+ "3235": 34434887680.0,
+ "3240": 34434887680.0,
+ "3245": 34434887680.0,
+ "3250": 34434887680.0,
+ "3255": 34434887680.0,
+ "3260": 34434887680.0,
+ "3265": 34434887680.0,
+ "3270": 34434887680.0,
+ "3275": 34434887680.0,
+ "3280": 34434887680.0,
+ "3285": 34434887680.0,
+ "3290": 34434887680.0,
+ "3295": 34434887680.0,
+ "3300": 34434887680.0,
+ "3305": 34434887680.0,
+ "3310": 34434887680.0,
+ "3315": 34434887680.0,
+ "3320": 34434887680.0,
+ "3325": 34434887680.0,
+ "3330": 34434887680.0,
+ "3335": 34434887680.0,
+ "3340": 34434887680.0,
+ "3345": 34434887680.0,
+ "3350": 34434887680.0,
+ "3355": 34434887680.0,
+ "3360": 34434887680.0,
+ "3365": 34434887680.0,
+ "3370": 34434887680.0,
+ "3375": 34434887680.0,
+ "3380": 34434887680.0,
+ "3385": 34434887680.0,
+ "3390": 34434887680.0,
+ "3395": 34434887680.0,
+ "3400": 34434887680.0,
+ "3405": 34434887680.0,
+ "3410": 34434887680.0,
+ "3415": 34434887680.0,
+ "3420": 34434887680.0,
+ "3425": 34434887680.0,
+ "3430": 34434887680.0,
+ "3435": 34434887680.0,
+ "3440": 34434887680.0,
+ "3445": 34434887680.0,
+ "3450": 34434887680.0,
+ "3455": 34434887680.0,
+ "3460": 34434887680.0,
+ "3465": 34434887680.0,
+ "3470": 34434887680.0,
+ "3475": 34434887680.0,
+ "3480": 34434887680.0,
+ "3485": 34434887680.0,
+ "3490": 34434887680.0,
+ "3495": 34434887680.0,
+ "3500": 34434887680.0,
+ "3505": 34434887680.0,
+ "3510": 34434887680.0,
+ "3515": 34434887680.0,
+ "3520": 34434887680.0,
+ "3525": 34434887680.0,
+ "3530": 34434887680.0,
+ "3535": 34434887680.0,
+ "3540": 34434887680.0,
+ "3545": 34434887680.0,
+ "3550": 34434887680.0,
+ "3555": 34434887680.0,
+ "3560": 34434887680.0,
+ "3565": 34434887680.0,
+ "3570": 34434887680.0,
+ "3575": 34434887680.0,
+ "3580": 34434887680.0,
+ "3585": 34434887680.0,
+ "3590": 34434887680.0,
+ "3595": 34434887680.0,
+ "3600": 34434887680.0,
+ "3605": 34434887680.0,
+ "3610": 34434887680.0,
+ "3615": 34434887680.0,
+ "3620": 34434887680.0,
+ "3625": 34434887680.0,
+ "3630": 34434887680.0,
+ "3635": 34434887680.0,
+ "3640": 34434887680.0,
+ "3645": 34434887680.0,
+ "3650": 34434887680.0,
+ "3655": 34434887680.0,
+ "3660": 34434887680.0,
+ "3665": 34434887680.0,
+ "3670": 34434887680.0,
+ "3675": 34434887680.0,
+ "3680": 34434887680.0,
+ "3685": 34434887680.0,
+ "3690": 34434887680.0,
+ "3695": 34434887680.0,
+ "3700": 34434887680.0,
+ "3705": 34434887680.0,
+ "3710": 34434887680.0,
+ "3715": 34434887680.0,
+ "3720": 34434887680.0,
+ "3725": 34434887680.0,
+ "3730": 34434887680.0,
+ "3735": 34434887680.0,
+ "3740": 34434887680.0,
+ "3745": 34434887680.0,
+ "3750": 34434887680.0,
+ "3755": 34434887680.0,
+ "3760": 34434887680.0,
+ "3765": 34434887680.0,
+ "3770": 34434887680.0,
+ "3775": 34434887680.0,
+ "3780": 34434887680.0,
+ "3785": 34434887680.0,
+ "3790": 34434887680.0,
+ "3795": 34434887680.0,
+ "3800": 34434887680.0,
+ "3805": 34434887680.0,
+ "3810": 34434887680.0,
+ "3815": 34434887680.0,
+ "3820": 34434887680.0,
+ "3825": 34434887680.0,
+ "3830": 34434887680.0,
+ "3835": 34434887680.0,
+ "3840": 34434887680.0,
+ "3845": 34434887680.0,
+ "3850": 34434887680.0,
+ "3855": 34434887680.0,
+ "3860": 34434887680.0,
+ "3865": 34434887680.0,
+ "3870": 34434887680.0,
+ "3875": 34434887680.0,
+ "3880": 34434887680.0,
+ "3885": 34434887680.0,
+ "3890": 34434887680.0,
+ "3895": 34434887680.0,
+ "3900": 34434887680.0,
+ "3905": 34434887680.0,
+ "3910": 34434887680.0,
+ "3915": 34434887680.0,
+ "3920": 34434887680.0,
+ "3925": 34434887680.0,
+ "3930": 34434887680.0,
+ "3935": 34434887680.0,
+ "3940": 34434887680.0,
+ "3945": 34434887680.0,
+ "3950": 34434887680.0,
+ "3955": 34434887680.0,
+ "3960": 34434887680.0,
+ "3965": 34434887680.0,
+ "3970": 34434887680.0,
+ "3975": 34434887680.0,
+ "3980": 34434887680.0,
+ "3985": 34434887680.0,
+ "3990": 34434887680.0,
+ "3995": 34434887680.0,
+ "4000": 34434887680.0,
+ "4005": 34434887680.0,
+ "4010": 34434887680.0,
+ "4015": 34434887680.0,
+ "4020": 34434887680.0,
+ "4025": 34434887680.0,
+ "4030": 34434887680.0,
+ "4035": 34434887680.0,
+ "4040": 34434887680.0,
+ "4045": 34434887680.0,
+ "4050": 34434887680.0,
+ "4055": 34434887680.0,
+ "4060": 34434887680.0,
+ "4065": 34434887680.0,
+ "4070": 34434887680.0,
+ "4075": 34434887680.0,
+ "4080": 34434887680.0,
+ "4085": 34434887680.0,
+ "4090": 34434887680.0,
+ "4095": 34434887680.0,
+ "4100": 34434887680.0,
+ "4105": 34434887680.0,
+ "4110": 34434887680.0,
+ "4115": 34434887680.0,
+ "4120": 34434887680.0,
+ "4125": 34434887680.0,
+ "4130": 34434887680.0,
+ "4135": 34434887680.0,
+ "4140": 34434887680.0,
+ "4145": 34434887680.0,
+ "4150": 34434887680.0,
+ "4155": 34434887680.0,
+ "4160": 34434887680.0,
+ "4165": 34434887680.0,
+ "4170": 34434887680.0,
+ "4175": 34434887680.0,
+ "4180": 34434887680.0,
+ "4185": 34434887680.0,
+ "4190": 34434887680.0,
+ "4195": 34434887680.0,
+ "4200": 34434887680.0,
+ "4205": 34434887680.0,
+ "4210": 34434887680.0,
+ "4215": 34434887680.0,
+ "4220": 34434887680.0,
+ "4225": 34434887680.0,
+ "4230": 34434887680.0,
+ "4235": 34434887680.0,
+ "4240": 34434887680.0,
+ "4245": 34434887680.0,
+ "4250": 34434887680.0,
+ "4255": 34434887680.0,
+ "4260": 34434887680.0,
+ "4265": 34434887680.0,
+ "4270": 34434887680.0,
+ "4275": 34434887680.0,
+ "4280": 34434887680.0,
+ "4285": 34434887680.0,
+ "4290": 34434887680.0,
+ "4295": 34434887680.0,
+ "4300": 34434887680.0,
+ "4305": 34434887680.0,
+ "4310": 34434887680.0,
+ "4315": 34434887680.0,
+ "4320": 34434887680.0,
+ "4325": 34434887680.0,
+ "4330": 34434887680.0,
+ "4335": 34434887680.0,
+ "4340": 34434887680.0,
+ "4345": 34434887680.0,
+ "4350": 34434887680.0,
+ "4355": 34434887680.0,
+ "4360": 34434887680.0,
+ "4365": 34434887680.0,
+ "4370": 34434887680.0,
+ "4375": 34434887680.0,
+ "4380": 34434887680.0,
+ "4385": 34434887680.0,
+ "4390": 34434887680.0,
+ "4395": 34434887680.0,
+ "4400": 34434887680.0,
+ "4405": 34434887680.0,
+ "4410": 34434887680.0,
+ "4415": 34434887680.0,
+ "4420": 34434887680.0,
+ "4425": 34434887680.0,
+ "4430": 34434887680.0,
+ "4435": 34434887680.0,
+ "4440": 34434887680.0,
+ "4445": 34434887680.0,
+ "4450": 34434887680.0,
+ "4455": 34434887680.0,
+ "4460": 34434887680.0,
+ "4465": 34434887680.0,
+ "4470": 34434887680.0,
+ "4475": 34434887680.0,
+ "4480": 34434887680.0,
+ "4485": 34434887680.0,
+ "4490": 34434887680.0,
+ "4495": 34434887680.0,
+ "4500": 34434887680.0,
+ "4505": 34434887680.0,
+ "4510": 34434887680.0,
+ "4515": 34434887680.0,
+ "4520": 34434887680.0,
+ "4525": 34434887680.0,
+ "4530": 34434887680.0,
+ "4535": 34434887680.0,
+ "4540": 34434887680.0,
+ "4545": 34434887680.0,
+ "4550": 34434887680.0,
+ "4555": 34434887680.0,
+ "4560": 34434887680.0,
+ "4565": 34434887680.0,
+ "4570": 34434887680.0,
+ "4575": 34434887680.0,
+ "4580": 34434887680.0,
+ "4585": 34434887680.0,
+ "4590": 34434887680.0,
+ "4595": 34434887680.0,
+ "4600": 34434887680.0,
+ "4605": 34434887680.0,
+ "4610": 34434887680.0,
+ "4615": 34434887680.0,
+ "4620": 34434887680.0,
+ "4625": 34434887680.0,
+ "4630": 34434887680.0,
+ "4635": 34434887680.0,
+ "4640": 34434887680.0,
+ "4645": 34434887680.0,
+ "4650": 34434887680.0,
+ "4655": 34434887680.0,
+ "4660": 34434887680.0,
+ "4665": 34434887680.0,
+ "4670": 34434887680.0,
+ "4675": 34434887680.0,
+ "4680": 34434887680.0,
+ "4685": 34434887680.0,
+ "4690": 34434887680.0,
+ "4695": 34434887680.0,
+ "4700": 34434887680.0,
+ "4705": 34434887680.0,
+ "4710": 34434887680.0,
+ "4715": 34434887680.0,
+ "4720": 34434887680.0,
+ "4725": 34434887680.0,
+ "4730": 34434887680.0,
+ "4735": 34434887680.0,
+ "4740": 34434887680.0,
+ "4745": 34434887680.0,
+ "4750": 34434887680.0,
+ "4755": 34434887680.0,
+ "4760": 34434887680.0,
+ "4765": 34434887680.0,
+ "4770": 34434887680.0,
+ "4775": 34434887680.0,
+ "4780": 34434887680.0,
+ "4785": 34434887680.0,
+ "4790": 34434887680.0,
+ "4795": 34434887680.0,
+ "4800": 34434887680.0,
+ "4805": 34434887680.0,
+ "4810": 34434887680.0,
+ "4815": 34434887680.0,
+ "4820": 34434887680.0,
+ "4825": 34434887680.0,
+ "4830": 34434887680.0,
+ "4835": 34434887680.0,
+ "4840": 34434887680.0,
+ "4845": 34434887680.0,
+ "4850": 34434887680.0,
+ "4855": 34434887680.0,
+ "4860": 34434887680.0,
+ "4865": 34434887680.0,
+ "4870": 34434887680.0,
+ "4875": 34434887680.0,
+ "4880": 34434887680.0,
+ "4885": 34434887680.0,
+ "4890": 34434887680.0,
+ "4895": 34434887680.0,
+ "4900": 34434887680.0,
+ "4905": 34434887680.0,
+ "4910": 34434887680.0,
+ "4915": 34434887680.0,
+ "4920": 34434887680.0,
+ "4925": 34434887680.0,
+ "4930": 34434887680.0,
+ "4935": 34434887680.0,
+ "4940": 34434887680.0,
+ "4945": 34434887680.0,
+ "4950": 34434887680.0,
+ "4955": 34434887680.0,
+ "4960": 34434887680.0,
+ "4965": 34434887680.0,
+ "4970": 34434887680.0,
+ "4975": 34434887680.0,
+ "4980": 34434887680.0,
+ "4985": 34434887680.0,
+ "4990": 34434887680.0,
+ "4995": 34434887680.0,
+ "5000": 34434887680.0,
+ "5005": 34434887680.0,
+ "5010": 34434887680.0,
+ "5015": 34434887680.0,
+ "5020": 34434887680.0,
+ "5025": 34434887680.0,
+ "5030": 34434887680.0,
+ "5035": 34434887680.0,
+ "5040": 34434887680.0,
+ "5045": 34434887680.0,
+ "5050": 34434887680.0,
+ "5055": 34434887680.0,
+ "5060": 34434887680.0,
+ "5065": 34434887680.0,
+ "5070": 34434887680.0,
+ "5075": 34434887680.0,
+ "5080": 34434887680.0,
+ "5085": 34434887680.0,
+ "5090": 34434887680.0,
+ "5095": 34434887680.0,
+ "5100": 34434887680.0,
+ "5105": 34434887680.0,
+ "5110": 34434887680.0,
+ "5115": 34434887680.0,
+ "5120": 34434887680.0,
+ "5125": 34434887680.0,
+ "5130": 34434887680.0,
+ "5135": 34434887680.0,
+ "5140": 34434887680.0,
+ "5145": 34434887680.0,
+ "5150": 34434887680.0,
+ "5155": 34434887680.0,
+ "5160": 34434887680.0,
+ "5165": 34434887680.0,
+ "5170": 34434887680.0,
+ "5175": 34434887680.0,
+ "5180": 34434887680.0,
+ "5185": 34434887680.0,
+ "5190": 34434887680.0,
+ "5195": 34434887680.0,
+ "5200": 34434887680.0,
+ "5205": 34434887680.0,
+ "5210": 34434887680.0,
+ "5215": 34434887680.0,
+ "5220": 34434887680.0,
+ "5225": 34434887680.0,
+ "5230": 34434887680.0,
+ "5235": 34434887680.0,
+ "5240": 34434887680.0,
+ "5245": 34434887680.0,
+ "5250": 34434887680.0,
+ "5255": 34434887680.0,
+ "5260": 34434887680.0,
+ "5265": 34434887680.0,
+ "5270": 34434887680.0,
+ "5275": 34434887680.0,
+ "5280": 34434887680.0,
+ "5285": 34434887680.0,
+ "5290": 34434887680.0,
+ "5295": 34434887680.0,
+ "5300": 34434887680.0,
+ "5305": 34434887680.0,
+ "5310": 34434887680.0,
+ "5315": 34434887680.0,
+ "5320": 34434887680.0,
+ "5325": 34434887680.0,
+ "5330": 34434887680.0,
+ "5335": 34434887680.0,
+ "5340": 34434887680.0,
+ "5345": 34434887680.0,
+ "5350": 34434887680.0,
+ "5355": 34434887680.0,
+ "5360": 34434887680.0,
+ "5365": 34434887680.0,
+ "5370": 34434887680.0,
+ "5375": 34434887680.0,
+ "5380": 34434887680.0,
+ "5385": 34434887680.0,
+ "5390": 34434887680.0,
+ "5395": 34434887680.0,
+ "5400": 34434887680.0,
+ "5405": 34434887680.0,
+ "5410": 34434887680.0,
+ "5415": 34434887680.0,
+ "5420": 34434887680.0,
+ "5425": 34434887680.0,
+ "5430": 34434887680.0,
+ "5435": 34434887680.0,
+ "5440": 34434887680.0,
+ "5445": 34434887680.0,
+ "5450": 34434887680.0,
+ "5455": 34434887680.0,
+ "5460": 34434887680.0,
+ "5465": 34434887680.0,
+ "5470": 34434887680.0,
+ "5475": 34434887680.0,
+ "5480": 34434887680.0,
+ "5485": 34434887680.0,
+ "5490": 34434887680.0,
+ "5495": 34434887680.0,
+ "5500": 34434887680.0,
+ "5505": 34434887680.0,
+ "5510": 34434887680.0,
+ "5515": 34434887680.0,
+ "5520": 34434887680.0,
+ "5525": 34434887680.0,
+ "5530": 34434887680.0,
+ "5535": 34434887680.0,
+ "5540": 34434887680.0,
+ "5545": 34434887680.0,
+ "5550": 34434887680.0,
+ "5555": 34434887680.0,
+ "5560": 34434887680.0,
+ "5565": 34434887680.0,
+ "5570": 34434887680.0,
+ "5575": 34434887680.0,
+ "5580": 34434887680.0,
+ "5585": 34434887680.0,
+ "5590": 34434887680.0,
+ "5595": 34434887680.0,
+ "5600": 34434887680.0,
+ "5605": 34434887680.0,
+ "5610": 34434887680.0,
+ "5615": 34434887680.0,
+ "5620": 34434887680.0,
+ "5625": 34434887680.0,
+ "5630": 34434887680.0,
+ "5635": 34434887680.0,
+ "5640": 34434887680.0,
+ "5645": 34434887680.0,
+ "5650": 34434887680.0,
+ "5655": 34434887680.0,
+ "5660": 34434887680.0,
+ "5665": 34434887680.0,
+ "5670": 34434887680.0,
+ "5675": 34434887680.0,
+ "5680": 34434887680.0,
+ "5685": 34434887680.0,
+ "5690": 34434887680.0,
+ "5695": 34434887680.0,
+ "5700": 34434887680.0,
+ "5705": 34434887680.0,
+ "5710": 34434887680.0,
+ "5715": 34434887680.0,
+ "5720": 34434887680.0,
+ "5725": 34434887680.0,
+ "5730": 34434887680.0,
+ "5735": 34434887680.0,
+ "5740": 34434887680.0,
+ "5745": 34434887680.0,
+ "5750": 34434887680.0,
+ "5755": 34434887680.0,
+ "5760": 34434887680.0,
+ "5765": 34434887680.0,
+ "5770": 34434887680.0,
+ "5775": 34434887680.0,
+ "5780": 34434887680.0,
+ "5785": 34434887680.0,
+ "5790": 34434887680.0,
+ "5795": 34434887680.0,
+ "5800": 34434887680.0,
+ "5805": 34434887680.0,
+ "5810": 34434887680.0,
+ "5815": 34434887680.0,
+ "5820": 34434887680.0,
+ "5825": 34434887680.0,
+ "5830": 34434887680.0,
+ "5835": 34434887680.0,
+ "5840": 34434887680.0,
+ "5845": 34434887680.0,
+ "5850": 34434887680.0,
+ "5855": 34434887680.0,
+ "5860": 34434887680.0,
+ "5865": 34434887680.0,
+ "5870": 34434887680.0,
+ "5875": 34434887680.0,
+ "5880": 34434887680.0,
+ "5885": 34434887680.0,
+ "5890": 34434887680.0,
+ "5895": 34434887680.0,
+ "5900": 34434887680.0,
+ "5905": 34434887680.0,
+ "5910": 34434887680.0,
+ "5915": 34434887680.0,
+ "5920": 34434887680.0,
+ "5925": 34434887680.0,
+ "5930": 34434887680.0,
+ "5935": 34434887680.0,
+ "5940": 34434887680.0,
+ "5945": 34434887680.0,
+ "5950": 34434887680.0,
+ "5955": 34434887680.0,
+ "5960": 34434887680.0,
+ "5965": 34434887680.0,
+ "5970": 34434887680.0,
+ "5975": 34434887680.0,
+ "5980": 34434887680.0,
+ "5985": 34434887680.0,
+ "5990": 34434887680.0,
+ "5995": 34434887680.0,
+ "6000": 34434887680.0,
+ "6005": 34434887680.0,
+ "6010": 34434887680.0,
+ "6015": 34434887680.0,
+ "6020": 34434887680.0,
+ "6025": 34434887680.0,
+ "6030": 34434887680.0,
+ "6035": 34434887680.0,
+ "6040": 34434887680.0,
+ "6045": 34434887680.0,
+ "6050": 34434887680.0,
+ "6055": 34434887680.0,
+ "6060": 34434887680.0,
+ "6065": 34434887680.0,
+ "6070": 34434887680.0,
+ "6075": 34434887680.0,
+ "6080": 34434887680.0,
+ "6085": 34434887680.0,
+ "6090": 34434887680.0,
+ "6095": 34434887680.0,
+ "6100": 34434887680.0,
+ "6105": 34434887680.0,
+ "6110": 34434887680.0,
+ "6115": 34434887680.0,
+ "6120": 34434887680.0,
+ "6125": 34434887680.0,
+ "6130": 34434887680.0,
+ "6135": 34434887680.0,
+ "6140": 34434887680.0,
+ "6145": 34434887680.0,
+ "6150": 34434887680.0,
+ "6155": 34434887680.0,
+ "6160": 34434887680.0,
+ "6165": 34434887680.0,
+ "6170": 34434887680.0,
+ "6175": 34434887680.0,
+ "6180": 34434887680.0,
+ "6185": 34434887680.0,
+ "6190": 34434887680.0,
+ "6195": 34434887680.0,
+ "6200": 34434887680.0,
+ "6205": 34434887680.0,
+ "6210": 34434887680.0,
+ "6215": 34434887680.0,
+ "6220": 34434887680.0,
+ "6225": 34434887680.0,
+ "6230": 34434887680.0,
+ "6235": 34434887680.0,
+ "6240": 34434887680.0,
+ "6245": 34434887680.0,
+ "6250": 34434887680.0,
+ "6255": 34434887680.0,
+ "6260": 34434887680.0,
+ "6265": 34434887680.0,
+ "6270": 34434887680.0,
+ "6275": 34434887680.0,
+ "6280": 34434887680.0,
+ "6285": 34434887680.0,
+ "6290": 34434887680.0,
+ "6295": 34434887680.0,
+ "6300": 34434887680.0,
+ "6305": 34434887680.0,
+ "6310": 34434887680.0,
+ "6315": 34434887680.0,
+ "6320": 34434887680.0,
+ "6325": 34434887680.0,
+ "6330": 34434887680.0,
+ "6335": 34434887680.0,
+ "6340": 34434887680.0,
+ "6345": 34434887680.0,
+ "6350": 34434887680.0,
+ "6355": 34434887680.0,
+ "6360": 34434887680.0,
+ "6365": 34434887680.0,
+ "6370": 34434887680.0,
+ "6375": 34434887680.0,
+ "6380": 34434887680.0,
+ "6385": 34434887680.0,
+ "6390": 34434887680.0,
+ "6395": 34434887680.0,
+ "6400": 34434887680.0,
+ "6405": 34434887680.0,
+ "6410": 34434887680.0,
+ "6415": 34434887680.0,
+ "6420": 34434887680.0,
+ "6425": 34434887680.0,
+ "6430": 34434887680.0,
+ "6435": 34434887680.0,
+ "6440": 34434887680.0,
+ "6445": 34434887680.0,
+ "6450": 34434887680.0,
+ "6455": 34434887680.0,
+ "6460": 34434887680.0,
+ "6465": 34434887680.0,
+ "6470": 34434887680.0,
+ "6475": 34434887680.0,
+ "6480": 34434887680.0,
+ "6485": 34434887680.0,
+ "6490": 34434887680.0,
+ "6495": 34434887680.0,
+ "6500": 34434887680.0,
+ "6505": 34434887680.0,
+ "6510": 34434887680.0,
+ "6515": 34434887680.0,
+ "6520": 34434887680.0,
+ "6525": 34434887680.0,
+ "6530": 34434887680.0,
+ "6535": 34434887680.0,
+ "6540": 34434887680.0,
+ "6545": 34434887680.0,
+ "6550": 34434887680.0,
+ "6555": 34434887680.0,
+ "6560": 34434887680.0,
+ "6565": 34434887680.0,
+ "6570": 34434887680.0,
+ "6575": 34434887680.0,
+ "6580": 34434887680.0,
+ "6585": 34434887680.0,
+ "6590": 34434887680.0,
+ "6595": 34434887680.0,
+ "6600": 34434887680.0,
+ "6605": 34434887680.0,
+ "6610": 34434887680.0,
+ "6615": 34434887680.0,
+ "6620": 34434887680.0,
+ "6625": 34434887680.0,
+ "6630": 34434887680.0,
+ "6635": 34434887680.0,
+ "6640": 34434887680.0,
+ "6645": 34434887680.0,
+ "6650": 34434887680.0,
+ "6655": 34434887680.0,
+ "6660": 34434887680.0,
+ "6665": 34434887680.0,
+ "6670": 34434887680.0,
+ "6675": 34434887680.0,
+ "6680": 34434887680.0,
+ "6685": 34434887680.0,
+ "6690": 34434887680.0,
+ "6695": 34434887680.0,
+ "6700": 34434887680.0,
+ "6705": 34434887680.0,
+ "6710": 34434887680.0,
+ "6715": 34434887680.0,
+ "6720": 34434887680.0,
+ "6725": 34434887680.0,
+ "6730": 34434887680.0,
+ "6735": 34434887680.0,
+ "6740": 34434887680.0,
+ "6745": 34434887680.0,
+ "6750": 34434887680.0,
+ "6755": 34434887680.0,
+ "6760": 34434887680.0,
+ "6765": 34434887680.0,
+ "6770": 34434887680.0,
+ "6775": 34434887680.0,
+ "6780": 34434887680.0,
+ "6785": 34434887680.0,
+ "6790": 34434887680.0,
+ "6795": 34434887680.0,
+ "6800": 34434887680.0,
+ "6805": 34434887680.0,
+ "6810": 34434887680.0,
+ "6815": 34434887680.0,
+ "6820": 34434887680.0,
+ "6825": 34434887680.0,
+ "6830": 34434887680.0,
+ "6835": 34434887680.0,
+ "6840": 34434887680.0,
+ "6845": 34434887680.0,
+ "6850": 34434887680.0,
+ "6855": 34434887680.0,
+ "6860": 34434887680.0,
+ "6865": 34434887680.0,
+ "6870": 34434887680.0,
+ "6875": 34434887680.0,
+ "6880": 34434887680.0,
+ "6885": 34434887680.0,
+ "6890": 34434887680.0,
+ "6895": 34434887680.0,
+ "6900": 34434887680.0,
+ "6905": 34434887680.0,
+ "6910": 34434887680.0,
+ "6915": 34434887680.0,
+ "6920": 34434887680.0,
+ "6925": 34434887680.0,
+ "6930": 34434887680.0,
+ "6935": 34434887680.0,
+ "6940": 34434887680.0,
+ "6945": 34434887680.0,
+ "6950": 34434887680.0,
+ "6955": 34434887680.0,
+ "6960": 34434887680.0,
+ "6965": 34434887680.0,
+ "6970": 34434887680.0,
+ "6975": 34434887680.0,
+ "6980": 34434887680.0,
+ "6985": 34434887680.0,
+ "6990": 34434887680.0,
+ "6995": 34434887680.0,
+ "7000": 34434887680.0,
+ "7005": 34434887680.0,
+ "7010": 34434887680.0,
+ "7015": 34434887680.0,
+ "7020": 34434887680.0,
+ "7025": 34434887680.0,
+ "7030": 34434887680.0,
+ "7035": 34434887680.0,
+ "7040": 34434887680.0,
+ "7045": 34434887680.0,
+ "7050": 34434887680.0,
+ "7055": 34434887680.0,
+ "7060": 34434887680.0,
+ "7065": 34434887680.0,
+ "7070": 34434887680.0,
+ "7075": 34434887680.0,
+ "7080": 34434887680.0,
+ "7085": 34434887680.0,
+ "7090": 34434887680.0,
+ "7095": 34434887680.0,
+ "7100": 34434887680.0,
+ "7105": 34434887680.0,
+ "7110": 34434887680.0,
+ "7115": 34434887680.0,
+ "7120": 34434887680.0,
+ "7125": 34434887680.0,
+ "7130": 34434887680.0,
+ "7135": 34434887680.0,
+ "7140": 34434887680.0,
+ "7145": 34434887680.0,
+ "7150": 34434887680.0,
+ "7155": 34434887680.0,
+ "7160": 34434887680.0,
+ "7165": 34434887680.0,
+ "7170": 34434887680.0,
+ "7175": 34434887680.0,
+ "7180": 34434887680.0,
+ "7185": 34434887680.0,
+ "7190": 34434887680.0,
+ "7195": 34434887680.0,
+ "7200": 34434887680.0,
+ "7205": 34434887680.0,
+ "7210": 34434887680.0,
+ "7215": 34434887680.0,
+ "7220": 34434887680.0,
+ "7225": 34434887680.0,
+ "7230": 34434887680.0,
+ "7235": 34434887680.0,
+ "7240": 34434887680.0,
+ "7245": 34434887680.0,
+ "7250": 34434887680.0,
+ "7255": 34434887680.0,
+ "7260": 34434887680.0,
+ "7265": 34434887680.0,
+ "7270": 34434887680.0,
+ "7275": 34434887680.0,
+ "7280": 34434887680.0,
+ "7285": 34434887680.0,
+ "7290": 34434887680.0,
+ "7295": 34434887680.0,
+ "7300": 34434887680.0,
+ "7305": 34434887680.0,
+ "7310": 34434887680.0,
+ "7315": 34434887680.0,
+ "7320": 34434887680.0,
+ "7325": 34434887680.0,
+ "7330": 34434887680.0,
+ "7335": 34434887680.0,
+ "7340": 34434887680.0,
+ "7345": 34434887680.0,
+ "7350": 34434887680.0,
+ "7355": 34434887680.0,
+ "7360": 34434887680.0,
+ "7365": 34434887680.0,
+ "7370": 34434887680.0,
+ "7375": 34434887680.0,
+ "7380": 34434887680.0,
+ "7385": 34434887680.0,
+ "7390": 34434887680.0,
+ "7395": 34434887680.0,
+ "7400": 34434887680.0,
+ "7405": 34434887680.0,
+ "7410": 34434887680.0,
+ "7415": 34434887680.0,
+ "7420": 34434887680.0,
+ "7425": 34434887680.0,
+ "7430": 34434887680.0,
+ "7435": 34434887680.0,
+ "7440": 34434887680.0,
+ "7445": 34434887680.0,
+ "7450": 34434887680.0,
+ "7455": 34434887680.0,
+ "7460": 34434887680.0,
+ "7465": 34434887680.0,
+ "7470": 34434887680.0,
+ "7475": 34434887680.0,
+ "7480": 34434887680.0,
+ "7485": 34434887680.0,
+ "7490": 34434887680.0,
+ "7495": 34434887680.0,
+ "7500": 34434887680.0,
+ "7505": 34434887680.0,
+ "7510": 34434887680.0,
+ "7515": 34434887680.0,
+ "7520": 34434887680.0,
+ "7525": 34434887680.0,
+ "7530": 34434887680.0,
+ "7535": 34434887680.0,
+ "7540": 34434887680.0,
+ "7545": 34434887680.0,
+ "7550": 34434887680.0,
+ "7555": 34434887680.0,
+ "7560": 34434887680.0,
+ "7565": 34434887680.0,
+ "7570": 34434887680.0,
+ "7575": 34434887680.0,
+ "7580": 34434887680.0,
+ "7585": 34434887680.0,
+ "7590": 34434887680.0,
+ "7595": 34434887680.0,
+ "7600": 34434887680.0,
+ "7605": 34434887680.0,
+ "7610": 34434887680.0,
+ "7615": 34434887680.0,
+ "7620": 34434887680.0,
+ "7625": 34434887680.0,
+ "7630": 34434887680.0,
+ "7635": 34434887680.0,
+ "7640": 34434887680.0,
+ "7645": 34434887680.0,
+ "7650": 34434887680.0,
+ "7655": 34434887680.0,
+ "7660": 34434887680.0,
+ "7665": 34434887680.0,
+ "7670": 34434887680.0,
+ "7675": 34434887680.0,
+ "7680": 34434887680.0,
+ "7685": 34434887680.0,
+ "7690": 34434887680.0,
+ "7695": 34434887680.0,
+ "7700": 34434887680.0,
+ "7705": 34434887680.0,
+ "7710": 34434887680.0,
+ "7715": 34434887680.0,
+ "7720": 34434887680.0,
+ "7725": 34434887680.0,
+ "7730": 34434887680.0,
+ "7735": 34434887680.0,
+ "7740": 34434887680.0,
+ "7745": 34434887680.0,
+ "7750": 34434887680.0,
+ "7755": 34434887680.0,
+ "7760": 34434887680.0,
+ "7765": 34434887680.0,
+ "7770": 34434887680.0,
+ "7775": 34434887680.0,
+ "7780": 34434887680.0,
+ "7785": 34434887680.0,
+ "7790": 34434887680.0,
+ "7795": 34434887680.0,
+ "7800": 34434887680.0,
+ "7805": 34434887680.0,
+ "7810": 34434887680.0,
+ "7815": 34434887680.0,
+ "7820": 34434887680.0,
+ "7825": 34434887680.0,
+ "7830": 34434887680.0,
+ "7835": 34434887680.0,
+ "7840": 34434887680.0,
+ "7845": 34434887680.0,
+ "7850": 34434887680.0,
+ "7855": 34434887680.0,
+ "7860": 34434887680.0,
+ "7865": 34434887680.0,
+ "7870": 34434887680.0,
+ "7875": 34434887680.0,
+ "7880": 34434887680.0,
+ "7885": 34434887680.0,
+ "7890": 34434887680.0,
+ "7895": 34434887680.0,
+ "7900": 34434887680.0,
+ "7905": 34434887680.0,
+ "7910": 34434887680.0,
+ "7915": 34434887680.0,
+ "7920": 34434887680.0,
+ "7925": 34434887680.0,
+ "7930": 34434887680.0,
+ "7935": 34434887680.0,
+ "7940": 34434887680.0,
+ "7945": 34434887680.0,
+ "7950": 34434887680.0,
+ "7955": 34434887680.0,
+ "7960": 34434887680.0,
+ "7965": 34434887680.0,
+ "7970": 34434887680.0,
+ "7975": 34434887680.0,
+ "7980": 34434887680.0,
+ "7985": 34434887680.0,
+ "7990": 34434887680.0,
+ "7995": 34434887680.0,
+ "8000": 34434887680.0,
+ "8005": 34434887680.0,
+ "8010": 34434887680.0,
+ "8015": 34434887680.0,
+ "8020": 34434887680.0,
+ "8025": 34434887680.0,
+ "8030": 34434887680.0,
+ "8035": 34434887680.0,
+ "8040": 34434887680.0,
+ "8045": 34434887680.0,
+ "8050": 34434887680.0,
+ "8055": 34434887680.0,
+ "8060": 34434887680.0,
+ "8065": 34434887680.0,
+ "8070": 34434887680.0,
+ "8075": 34434887680.0,
+ "8080": 34434887680.0,
+ "8085": 34434887680.0,
+ "8090": 34434887680.0,
+ "8095": 34434887680.0,
+ "8100": 34434887680.0,
+ "8105": 34434887680.0,
+ "8110": 34434887680.0,
+ "8115": 34434887680.0,
+ "8120": 34434887680.0,
+ "8125": 34434887680.0,
+ "8130": 34434887680.0,
+ "8135": 34434887680.0,
+ "8140": 34434887680.0,
+ "8145": 34434887680.0,
+ "8150": 34434887680.0,
+ "8155": 34434887680.0,
+ "8160": 34434887680.0,
+ "8165": 34434887680.0,
+ "8170": 34434887680.0,
+ "8175": 34434887680.0,
+ "8180": 34434887680.0,
+ "8185": 34434887680.0,
+ "8190": 34434887680.0,
+ "8195": 34434887680.0,
+ "8200": 34434887680.0,
+ "8205": 34434887680.0,
+ "8210": 34434887680.0,
+ "8215": 34434887680.0,
+ "8220": 34434887680.0,
+ "8225": 34434887680.0,
+ "8230": 34434887680.0,
+ "8235": 34434887680.0,
+ "8240": 34434887680.0,
+ "8245": 34434887680.0,
+ "8250": 34434887680.0,
+ "8255": 34434887680.0,
+ "8260": 34434887680.0,
+ "8265": 34434887680.0,
+ "8270": 34434887680.0,
+ "8275": 34434887680.0,
+ "8280": 34434887680.0,
+ "8285": 34434887680.0,
+ "8290": 34434887680.0,
+ "8295": 34434887680.0,
+ "8300": 34434887680.0,
+ "8305": 34434887680.0,
+ "8310": 34434887680.0,
+ "8315": 34434887680.0,
+ "8320": 34434887680.0,
+ "8325": 34434887680.0,
+ "8330": 34434887680.0,
+ "8335": 34434887680.0,
+ "8340": 34434887680.0,
+ "8345": 34434887680.0,
+ "8350": 34434887680.0,
+ "8355": 34434887680.0,
+ "8360": 34434887680.0,
+ "8365": 34434887680.0,
+ "8370": 34434887680.0,
+ "8375": 34434887680.0,
+ "8380": 34434887680.0,
+ "8385": 34434887680.0,
+ "8390": 34434887680.0,
+ "8395": 34434887680.0,
+ "8400": 34434887680.0,
+ "8405": 34434887680.0,
+ "8410": 34434887680.0,
+ "8415": 34434887680.0,
+ "8420": 34434887680.0,
+ "8425": 34434887680.0,
+ "8430": 34434887680.0,
+ "8435": 34434887680.0,
+ "8440": 34434887680.0,
+ "8445": 34434887680.0,
+ "8450": 34434887680.0,
+ "8455": 34434887680.0,
+ "8460": 34434887680.0,
+ "8465": 34434887680.0,
+ "8470": 34434887680.0,
+ "8475": 34434887680.0,
+ "8480": 34434887680.0,
+ "8485": 34434887680.0,
+ "8490": 34434887680.0,
+ "8495": 34434887680.0,
+ "8500": 34434887680.0,
+ "8505": 34434887680.0,
+ "8510": 34434887680.0,
+ "8515": 34434887680.0,
+ "8520": 34434887680.0,
+ "8525": 34434887680.0,
+ "8530": 34434887680.0,
+ "8535": 34434887680.0,
+ "8540": 34434887680.0,
+ "8545": 34434887680.0,
+ "8550": 34434887680.0,
+ "8555": 34434887680.0,
+ "8560": 34434887680.0,
+ "8565": 34434887680.0,
+ "8570": 34434887680.0,
+ "8575": 34434887680.0,
+ "8580": 34434887680.0,
+ "8585": 34434887680.0,
+ "8590": 34434887680.0,
+ "8595": 34434887680.0,
+ "8600": 34434887680.0,
+ "8605": 34434887680.0,
+ "8610": 34434887680.0,
+ "8615": 34434887680.0,
+ "8620": 34434887680.0,
+ "8625": 34434887680.0,
+ "8630": 34434887680.0,
+ "8635": 34434887680.0,
+ "8640": 34434887680.0,
+ "8645": 34434887680.0,
+ "8650": 34434887680.0,
+ "8655": 34434887680.0,
+ "8660": 34434887680.0,
+ "8665": 34434887680.0,
+ "8670": 34434887680.0,
+ "8675": 34434887680.0,
+ "8680": 34434887680.0,
+ "8685": 34434887680.0,
+ "8690": 34434887680.0,
+ "8695": 34434887680.0,
+ "8700": 34434887680.0,
+ "8705": 34434887680.0,
+ "8710": 34434887680.0,
+ "8715": 34434887680.0,
+ "8720": 34434887680.0,
+ "8725": 34434887680.0,
+ "8730": 34434887680.0,
+ "8735": 34434887680.0,
+ "8740": 34434887680.0,
+ "8745": 34434887680.0,
+ "8750": 34434887680.0,
+ "8755": 34434887680.0,
+ "8760": 34434887680.0,
+ "8765": 34434887680.0,
+ "8770": 34434887680.0,
+ "8775": 34434887680.0,
+ "8780": 34434887680.0,
+ "8785": 34434887680.0,
+ "8790": 34434887680.0,
+ "8795": 34434887680.0,
+ "8800": 34434887680.0,
+ "8805": 34434887680.0,
+ "8810": 34434887680.0,
+ "8815": 34434887680.0,
+ "8820": 34434887680.0,
+ "8825": 34434887680.0,
+ "8830": 34434887680.0,
+ "8835": 34434887680.0,
+ "8840": 34434887680.0,
+ "8845": 34434887680.0,
+ "8850": 34434887680.0,
+ "8855": 34434887680.0,
+ "8860": 34434887680.0,
+ "8865": 34434887680.0,
+ "8870": 34434887680.0,
+ "8875": 34434887680.0,
+ "8880": 34434887680.0,
+ "8885": 34434887680.0,
+ "8890": 34434887680.0,
+ "8895": 34434887680.0,
+ "8900": 34434887680.0,
+ "8905": 34434887680.0,
+ "8910": 34434887680.0,
+ "8915": 34434887680.0,
+ "8920": 34434887680.0,
+ "8925": 34434887680.0,
+ "8930": 34434887680.0,
+ "8935": 34434887680.0,
+ "8940": 34434887680.0,
+ "8945": 34434887680.0,
+ "8950": 34434887680.0,
+ "8955": 34434887680.0,
+ "8960": 34434887680.0,
+ "8965": 34434887680.0,
+ "8970": 34434887680.0,
+ "8975": 34434887680.0,
+ "8980": 34434887680.0,
+ "8985": 34434887680.0,
+ "8990": 34434887680.0,
+ "8995": 34434887680.0,
+ "9000": 34434887680.0,
+ "9005": 34434887680.0,
+ "9010": 34434887680.0,
+ "9015": 34434887680.0,
+ "9020": 34434887680.0,
+ "9025": 34434887680.0,
+ "9030": 34434887680.0,
+ "9035": 34434887680.0,
+ "9040": 34434887680.0,
+ "9045": 34434887680.0,
+ "9050": 34434887680.0,
+ "9055": 34434887680.0,
+ "9060": 34434887680.0,
+ "9065": 34434887680.0,
+ "9070": 34434887680.0,
+ "9075": 34434887680.0,
+ "9080": 34434887680.0,
+ "9085": 34434887680.0,
+ "9090": 34434887680.0,
+ "9095": 34434887680.0,
+ "9100": 34434887680.0,
+ "9105": 34434887680.0,
+ "9110": 34434887680.0,
+ "9115": 34434887680.0,
+ "9120": 34434887680.0,
+ "9125": 34434887680.0,
+ "9130": 34434887680.0,
+ "9135": 34434887680.0,
+ "9140": 34434887680.0,
+ "9145": 34434887680.0,
+ "9150": 34434887680.0,
+ "9155": 34434887680.0,
+ "9160": 34434887680.0,
+ "9165": 34434887680.0,
+ "9170": 34434887680.0,
+ "9175": 34434887680.0,
+ "9180": 34434887680.0,
+ "9185": 34434887680.0,
+ "9190": 34434887680.0,
+ "9195": 34434887680.0,
+ "9200": 34434887680.0,
+ "9205": 34434887680.0,
+ "9210": 34434887680.0,
+ "9215": 34434887680.0,
+ "9220": 34434887680.0,
+ "9225": 34434887680.0,
+ "9230": 34434887680.0,
+ "9235": 34434887680.0,
+ "9240": 34434887680.0,
+ "9245": 34434887680.0,
+ "9250": 34434887680.0,
+ "9255": 34434887680.0,
+ "9260": 34434887680.0,
+ "9265": 34434887680.0,
+ "9270": 34434887680.0,
+ "9275": 34434887680.0,
+ "9280": 34434887680.0,
+ "9285": 34434887680.0,
+ "9290": 34434887680.0,
+ "9295": 34434887680.0,
+ "9300": 34434887680.0,
+ "9305": 34434887680.0,
+ "9310": 34434887680.0,
+ "9315": 34434887680.0,
+ "9320": 34434887680.0,
+ "9325": 34434887680.0,
+ "9330": 34434887680.0,
+ "9335": 34434887680.0,
+ "9340": 34434887680.0,
+ "9345": 34434887680.0,
+ "9350": 34434887680.0,
+ "9355": 34434887680.0,
+ "9360": 34434887680.0,
+ "9365": 34434887680.0,
+ "9370": 34434887680.0,
+ "9375": 34434887680.0,
+ "9380": 34434887680.0,
+ "9385": 34434887680.0,
+ "9390": 34434887680.0,
+ "9395": 34434887680.0,
+ "9400": 34434887680.0,
+ "9405": 34434887680.0,
+ "9410": 34434887680.0,
+ "9415": 34434887680.0,
+ "9420": 34434887680.0,
+ "9425": 34434887680.0,
+ "9430": 34434887680.0,
+ "9435": 34434887680.0,
+ "9440": 34434887680.0,
+ "9445": 34434887680.0,
+ "9450": 34434887680.0,
+ "9455": 34434887680.0,
+ "9460": 34434887680.0,
+ "9465": 34434887680.0,
+ "9470": 34434887680.0,
+ "9475": 34434887680.0,
+ "9480": 34434887680.0,
+ "9485": 34434887680.0,
+ "9490": 34434887680.0,
+ "9495": 34434887680.0,
+ "9500": 34434887680.0,
+ "9505": 34434887680.0,
+ "9510": 34434887680.0,
+ "9515": 34434887680.0,
+ "9520": 34434887680.0,
+ "9525": 34434887680.0,
+ "9530": 34434887680.0,
+ "9535": 34434887680.0,
+ "9540": 34434887680.0,
+ "9545": 34434887680.0,
+ "9550": 34434887680.0,
+ "9555": 34434887680.0,
+ "9560": 34434887680.0,
+ "9565": 34434887680.0,
+ "9570": 34434887680.0,
+ "9575": 34434887680.0,
+ "9580": 34434887680.0,
+ "9585": 34434887680.0,
+ "9590": 34434887680.0,
+ "9595": 34434887680.0,
+ "9600": 34434887680.0,
+ "9605": 34434887680.0,
+ "9610": 34434887680.0,
+ "9615": 34434887680.0,
+ "9620": 34434887680.0,
+ "9625": 34434887680.0,
+ "9630": 34434887680.0,
+ "9635": 34434887680.0,
+ "9640": 34434887680.0,
+ "9645": 34434887680.0,
+ "9650": 34434887680.0,
+ "9655": 34434887680.0,
+ "9660": 34434887680.0,
+ "9665": 34434887680.0,
+ "9670": 34434887680.0,
+ "9675": 34434887680.0,
+ "9680": 34434887680.0,
+ "9685": 34434887680.0,
+ "9690": 34434887680.0,
+ "9695": 34434887680.0,
+ "9700": 34434887680.0,
+ "9705": 34434887680.0,
+ "9710": 34434887680.0,
+ "9715": 34434887680.0,
+ "9720": 34434887680.0,
+ "9725": 34434887680.0,
+ "9730": 34434887680.0,
+ "9735": 34434887680.0,
+ "9740": 34434887680.0,
+ "9745": 34434887680.0,
+ "9750": 34434887680.0,
+ "9755": 34434887680.0,
+ "9760": 34434887680.0,
+ "9765": 34434887680.0,
+ "9770": 34434887680.0,
+ "9775": 34434887680.0,
+ "9780": 34434887680.0,
+ "9785": 34434887680.0,
+ "9790": 34434887680.0,
+ "9795": 34434887680.0,
+ "9800": 34434887680.0,
+ "9805": 34434887680.0,
+ "9810": 34434887680.0,
+ "9815": 34434887680.0,
+ "9820": 34434887680.0,
+ "9825": 34434887680.0,
+ "9830": 34434887680.0,
+ "9835": 34434887680.0,
+ "9840": 34434887680.0,
+ "9845": 34434887680.0,
+ "9850": 34434887680.0,
+ "9855": 34434887680.0,
+ "9860": 34434887680.0,
+ "9865": 34434887680.0,
+ "9870": 34434887680.0,
+ "9875": 34434887680.0,
+ "9880": 34434887680.0,
+ "9885": 34434887680.0,
+ "9890": 34434887680.0,
+ "9895": 34434887680.0,
+ "9900": 34434887680.0,
+ "9905": 34434887680.0,
+ "9910": 34434887680.0,
+ "9915": 34434887680.0,
+ "9920": 34434887680.0,
+ "9925": 34434887680.0,
+ "9930": 34434887680.0,
+ "9935": 34434887680.0,
+ "9940": 34434887680.0,
+ "9945": 34434887680.0,
+ "9950": 34434887680.0,
+ "9955": 34434887680.0,
+ "9960": 34434887680.0,
+ "9965": 34434887680.0,
+ "9970": 34434887680.0,
+ "9975": 34434887680.0,
+ "9980": 34434887680.0,
+ "9985": 34434887680.0,
+ "9990": 34434887680.0,
+ "9995": 34434887680.0,
+ "10000": 34434887680.0,
+ "10005": 34434887680.0,
+ "10010": 34434887680.0,
+ "10015": 34434887680.0,
+ "10020": 34434887680.0,
+ "10025": 34434887680.0,
+ "10030": 34434887680.0,
+ "10035": 34434887680.0,
+ "10040": 34434887680.0,
+ "10045": 34434887680.0,
+ "10050": 34434887680.0,
+ "10055": 34434887680.0,
+ "10060": 34434887680.0,
+ "10065": 34434887680.0,
+ "10070": 34434887680.0,
+ "10075": 34434887680.0,
+ "10080": 34434887680.0,
+ "10085": 34434887680.0,
+ "10090": 34434887680.0,
+ "10095": 34434887680.0,
+ "10100": 34434887680.0,
+ "10105": 34434887680.0,
+ "10110": 34434887680.0,
+ "10115": 34434887680.0,
+ "10120": 34434887680.0,
+ "10125": 34434887680.0,
+ "10130": 34434887680.0,
+ "10135": 34434887680.0,
+ "10140": 34434887680.0,
+ "10145": 34434887680.0,
+ "10150": 34434887680.0,
+ "10155": 34434887680.0,
+ "10160": 34434887680.0,
+ "10165": 34434887680.0,
+ "10170": 34434887680.0,
+ "10175": 34434887680.0,
+ "10180": 34434887680.0,
+ "10185": 34434887680.0,
+ "10190": 34434887680.0,
+ "10195": 34434887680.0,
+ "10200": 34434887680.0,
+ "10205": 34434887680.0,
+ "10210": 34434887680.0,
+ "10215": 34434887680.0,
+ "10220": 34434887680.0,
+ "10225": 34434887680.0,
+ "10230": 34434887680.0,
+ "10235": 34434887680.0,
+ "10240": 34434887680.0,
+ "10245": 34434887680.0,
+ "10250": 34434887680.0,
+ "10255": 34434887680.0,
+ "10260": 34434887680.0,
+ "10265": 34434887680.0,
+ "10270": 34434887680.0,
+ "10275": 34434887680.0,
+ "10280": 34434887680.0,
+ "10285": 34434887680.0,
+ "10290": 34434887680.0,
+ "10295": 34434887680.0,
+ "10300": 34434887680.0,
+ "10305": 34434887680.0,
+ "10310": 34434887680.0,
+ "10315": 34434887680.0,
+ "10320": 34434887680.0,
+ "10325": 34434887680.0,
+ "10330": 34434887680.0,
+ "10335": 34434887680.0,
+ "10340": 34434887680.0,
+ "10345": 34434887680.0,
+ "10350": 34434887680.0,
+ "10355": 34434887680.0,
+ "10360": 34434887680.0,
+ "10365": 34434887680.0,
+ "10370": 34434887680.0,
+ "10375": 34434887680.0,
+ "10380": 34434887680.0,
+ "10385": 34434887680.0,
+ "10390": 34434887680.0,
+ "10395": 34434887680.0,
+ "10400": 34434887680.0,
+ "10405": 34434887680.0,
+ "10410": 34434887680.0,
+ "10415": 34434887680.0,
+ "10420": 34434887680.0,
+ "10425": 34434887680.0,
+ "10430": 34434887680.0,
+ "10435": 34434887680.0,
+ "10440": 34434887680.0,
+ "10445": 34434887680.0,
+ "10450": 34434887680.0,
+ "10455": 34434887680.0,
+ "10460": 34434887680.0,
+ "10465": 34434887680.0,
+ "10470": 34434887680.0,
+ "10475": 34434887680.0,
+ "10480": 34434887680.0,
+ "10485": 34434887680.0,
+ "10490": 34434887680.0,
+ "10495": 34434887680.0,
+ "10500": 34434887680.0,
+ "10505": 34434887680.0,
+ "10510": 34434887680.0,
+ "10515": 34434887680.0,
+ "10520": 34434887680.0,
+ "10525": 34434887680.0,
+ "10530": 34434887680.0,
+ "10535": 34434887680.0,
+ "10540": 34434887680.0,
+ "10545": 34434887680.0,
+ "10550": 34434887680.0,
+ "10555": 34434887680.0,
+ "10560": 34434887680.0,
+ "10565": 34434887680.0,
+ "10570": 34434887680.0,
+ "10575": 34434887680.0,
+ "10580": 34434887680.0,
+ "10585": 34434887680.0,
+ "10590": 34434887680.0,
+ "10595": 34434887680.0,
+ "10600": 34434887680.0,
+ "10605": 34434887680.0,
+ "10610": 34434887680.0,
+ "10615": 34434887680.0,
+ "10620": 34434887680.0,
+ "10625": 34434887680.0,
+ "10630": 34434887680.0,
+ "10635": 34434887680.0,
+ "10640": 34434887680.0,
+ "10645": 34434887680.0,
+ "10650": 34434887680.0,
+ "10655": 34434887680.0,
+ "10660": 34434887680.0,
+ "10665": 34434887680.0,
+ "10670": 34434887680.0,
+ "10675": 34434887680.0,
+ "10680": 34434887680.0,
+ "10685": 34434887680.0,
+ "10690": 34434887680.0,
+ "10695": 34434887680.0,
+ "10700": 34434887680.0,
+ "10705": 34434887680.0,
+ "10710": 34434887680.0,
+ "10715": 34434887680.0,
+ "10720": 34434887680.0,
+ "10725": 34434887680.0,
+ "10730": 34434887680.0,
+ "10735": 34434887680.0,
+ "10740": 34434887680.0,
+ "10745": 34434887680.0,
+ "10750": 34434887680.0,
+ "10755": 34434887680.0,
+ "10760": 34434887680.0,
+ "10765": 34434887680.0,
+ "10770": 34434887680.0,
+ "10775": 34434887680.0,
+ "10780": 34434887680.0,
+ "10785": 34434887680.0,
+ "10790": 34434887680.0,
+ "10795": 34434887680.0,
+ "10800": 34434887680.0,
+ "10805": 34434887680.0,
+ "10810": 34434887680.0,
+ "10815": 34434887680.0,
+ "10820": 34434887680.0,
+ "10825": 34434887680.0,
+ "10830": 34434887680.0,
+ "10835": 34434887680.0,
+ "10840": 34434887680.0,
+ "10845": 34434887680.0,
+ "10850": 34434887680.0,
+ "10855": 34434887680.0,
+ "10860": 34434887680.0,
+ "10865": 34434887680.0,
+ "10870": 34434887680.0,
+ "10875": 34434887680.0,
+ "10880": 34434887680.0,
+ "10885": 34434887680.0,
+ "10890": 34434887680.0,
+ "10895": 34434887680.0,
+ "10900": 34434887680.0,
+ "10905": 34434887680.0,
+ "10910": 34434887680.0,
+ "10915": 34434887680.0,
+ "10920": 34434887680.0,
+ "10925": 34434887680.0,
+ "10930": 34434887680.0,
+ "10935": 34434887680.0,
+ "10940": 34434887680.0,
+ "10945": 34434887680.0,
+ "10950": 34434887680.0,
+ "10955": 34434887680.0,
+ "10960": 34434887680.0,
+ "10965": 34434887680.0,
+ "10970": 34434887680.0,
+ "10975": 34434887680.0,
+ "10980": 34434887680.0,
+ "10985": 34434887680.0,
+ "10990": 34434887680.0,
+ "10995": 34434887680.0,
+ "11000": 34434887680.0,
+ "11005": 34434887680.0,
+ "11010": 34434887680.0,
+ "11015": 34434887680.0,
+ "11020": 34434887680.0,
+ "11025": 34434887680.0,
+ "11030": 34434887680.0,
+ "11035": 34434887680.0,
+ "11040": 34434887680.0,
+ "11045": 34434887680.0,
+ "11050": 34434887680.0,
+ "11055": 34434887680.0,
+ "11060": 34434887680.0,
+ "11065": 34434887680.0,
+ "11070": 34434887680.0,
+ "11075": 34434887680.0,
+ "11080": 34434887680.0,
+ "11085": 34434887680.0,
+ "11090": 34434887680.0,
+ "11095": 34434887680.0,
+ "11100": 34434887680.0,
+ "11105": 34434887680.0,
+ "11110": 34434887680.0,
+ "11115": 34434887680.0,
+ "11120": 34434887680.0,
+ "11125": 34434887680.0,
+ "11130": 34434887680.0,
+ "11135": 34434887680.0,
+ "11140": 34434887680.0,
+ "11145": 34434887680.0,
+ "11150": 34434887680.0,
+ "11155": 34434887680.0,
+ "11160": 34434887680.0,
+ "11165": 34434887680.0,
+ "11170": 34434887680.0,
+ "11175": 34434887680.0,
+ "11180": 34434887680.0,
+ "11185": 34434887680.0,
+ "11190": 34434887680.0,
+ "11195": 34434887680.0,
+ "11200": 34434887680.0,
+ "11205": 34434887680.0,
+ "11210": 34434887680.0,
+ "11215": 34434887680.0,
+ "11220": 34434887680.0,
+ "11225": 34434887680.0,
+ "11230": 34434887680.0,
+ "11235": 34434887680.0,
+ "11240": 34434887680.0,
+ "11245": 34434887680.0,
+ "11250": 34434887680.0,
+ "11255": 34434887680.0,
+ "11260": 34434887680.0,
+ "11265": 34434887680.0,
+ "11270": 34434887680.0,
+ "11275": 34434887680.0,
+ "11280": 34434887680.0,
+ "11285": 34434887680.0,
+ "11290": 34434887680.0,
+ "11295": 34434887680.0,
+ "11300": 34434887680.0,
+ "11305": 34434887680.0,
+ "11310": 34434887680.0,
+ "11315": 34434887680.0,
+ "11320": 34434887680.0,
+ "11325": 34434887680.0,
+ "11330": 34434887680.0,
+ "11335": 34434887680.0,
+ "11340": 34434887680.0,
+ "11345": 34434887680.0,
+ "11350": 34434887680.0,
+ "11355": 34434887680.0,
+ "11360": 34434887680.0,
+ "11365": 34434887680.0,
+ "11370": 34434887680.0,
+ "11375": 34434887680.0,
+ "11380": 34434887680.0,
+ "11385": 34434887680.0,
+ "11390": 34434887680.0,
+ "11395": 34434887680.0,
+ "11400": 34434887680.0,
+ "11405": 34434887680.0,
+ "11410": 34434887680.0,
+ "11415": 34434887680.0,
+ "11420": 34434887680.0,
+ "11425": 34434887680.0,
+ "11430": 34434887680.0,
+ "11435": 34434887680.0,
+ "11440": 34434887680.0,
+ "11445": 34434887680.0,
+ "11450": 34434887680.0,
+ "11455": 34434887680.0,
+ "11460": 34434887680.0,
+ "11465": 34434887680.0,
+ "11470": 34434887680.0,
+ "11475": 34434887680.0,
+ "11480": 34434887680.0,
+ "11485": 34434887680.0,
+ "11490": 34434887680.0,
+ "11495": 34434887680.0,
+ "11500": 34434887680.0,
+ "11505": 34434887680.0,
+ "11510": 34434887680.0,
+ "11515": 34434887680.0,
+ "11520": 34434887680.0,
+ "11525": 34434887680.0,
+ "11530": 34434887680.0,
+ "11535": 34434887680.0,
+ "11540": 34434887680.0,
+ "11545": 34434887680.0,
+ "11550": 34434887680.0,
+ "11555": 34434887680.0,
+ "11560": 34434887680.0,
+ "11565": 34434887680.0,
+ "11570": 34434887680.0,
+ "11575": 34434887680.0,
+ "11580": 34434887680.0,
+ "11585": 34434887680.0,
+ "11590": 34434887680.0,
+ "11595": 34434887680.0,
+ "11600": 34434887680.0,
+ "11605": 34434887680.0,
+ "11610": 34434887680.0,
+ "11615": 34434887680.0,
+ "11620": 34434887680.0,
+ "11625": 34434887680.0,
+ "11630": 34434887680.0,
+ "11635": 34434887680.0,
+ "11640": 34434887680.0,
+ "11645": 34434887680.0,
+ "11650": 34434887680.0,
+ "11655": 34434887680.0,
+ "11660": 34434887680.0,
+ "11665": 34434887680.0,
+ "11670": 34434887680.0,
+ "11675": 34434887680.0,
+ "11680": 34434887680.0,
+ "11685": 34434887680.0,
+ "11690": 34434887680.0,
+ "11695": 34434887680.0,
+ "11700": 34434887680.0,
+ "11705": 34434887680.0,
+ "11710": 34434887680.0,
+ "11715": 34434887680.0,
+ "11720": 34434887680.0,
+ "11725": 34434887680.0,
+ "11730": 34434887680.0,
+ "11735": 34434887680.0,
+ "11740": 34434887680.0,
+ "11745": 34434887680.0,
+ "11750": 34434887680.0,
+ "11755": 34434887680.0,
+ "11760": 34434887680.0,
+ "11765": 34434887680.0,
+ "11770": 34434887680.0,
+ "11775": 34434887680.0,
+ "11780": 34434887680.0,
+ "11785": 34434887680.0,
+ "11790": 34434887680.0,
+ "11795": 34434887680.0,
+ "11800": 34434887680.0,
+ "11805": 34434887680.0,
+ "11810": 34434887680.0,
+ "11815": 34434887680.0,
+ "11820": 34434887680.0,
+ "11825": 34434887680.0,
+ "11830": 34434887680.0,
+ "11835": 34434887680.0,
+ "11840": 34434887680.0,
+ "11845": 34434887680.0,
+ "11850": 34434887680.0,
+ "11855": 34434887680.0,
+ "11860": 34434887680.0,
+ "11865": 34434887680.0,
+ "11870": 34434887680.0,
+ "11875": 34434887680.0,
+ "11880": 34434887680.0,
+ "11885": 34434887680.0,
+ "11890": 34434887680.0,
+ "11895": 34434887680.0,
+ "11900": 34434887680.0,
+ "11905": 34434887680.0,
+ "11910": 34434887680.0,
+ "11915": 34434887680.0,
+ "11920": 34434887680.0,
+ "11925": 34434887680.0,
+ "11930": 34434887680.0,
+ "11935": 34434887680.0,
+ "11940": 34434887680.0,
+ "11945": 34434887680.0,
+ "11950": 34434887680.0,
+ "11955": 34434887680.0,
+ "11960": 34434887680.0,
+ "11965": 34434887680.0,
+ "11970": 34434887680.0,
+ "11975": 34434887680.0,
+ "11980": 34434887680.0,
+ "11985": 34434887680.0,
+ "11990": 34434887680.0,
+ "11995": 34434887680.0,
+ "12000": 34434887680.0,
+ "12005": 34434887680.0,
+ "12010": 34434887680.0,
+ "12015": 34434887680.0,
+ "12020": 34434887680.0,
+ "12025": 34434887680.0,
+ "12030": 34434887680.0,
+ "12035": 34434887680.0,
+ "12040": 34434887680.0,
+ "12045": 34434887680.0,
+ "12050": 34434887680.0,
+ "12055": 34434887680.0,
+ "12060": 34434887680.0,
+ "12065": 34434887680.0,
+ "12070": 34434887680.0,
+ "12075": 34434887680.0,
+ "12080": 34434887680.0,
+ "12085": 34434887680.0,
+ "12090": 34434887680.0,
+ "12095": 34434887680.0,
+ "12100": 34434887680.0,
+ "12105": 34434887680.0,
+ "12110": 34434887680.0,
+ "12115": 34434887680.0,
+ "12120": 34434887680.0,
+ "12125": 34434887680.0,
+ "12130": 34434887680.0,
+ "12135": 34434887680.0,
+ "12140": 34434887680.0,
+ "12145": 34434887680.0,
+ "12150": 34434887680.0,
+ "12155": 34434887680.0,
+ "12160": 34434887680.0,
+ "12165": 34434887680.0,
+ "12170": 34434887680.0,
+ "12175": 34434887680.0,
+ "12180": 34434887680.0,
+ "12185": 34434887680.0,
+ "12190": 34434887680.0,
+ "12195": 34434887680.0,
+ "12200": 34434887680.0,
+ "12205": 34434887680.0,
+ "12210": 34434887680.0,
+ "12215": 34434887680.0,
+ "12220": 34434887680.0,
+ "12225": 34434887680.0,
+ "12230": 34434887680.0,
+ "12235": 34434887680.0,
+ "12240": 34434887680.0,
+ "12245": 34434887680.0,
+ "12250": 34434887680.0,
+ "12255": 34434887680.0,
+ "12260": 34434887680.0,
+ "12265": 34434887680.0,
+ "12270": 34434887680.0,
+ "12275": 34434887680.0,
+ "12280": 34434887680.0,
+ "12285": 34434887680.0,
+ "12290": 34434887680.0,
+ "12295": 34434887680.0,
+ "12300": 34434887680.0,
+ "12305": 34434887680.0,
+ "12310": 34434887680.0,
+ "12315": 34434887680.0,
+ "12320": 34434887680.0,
+ "12325": 34434887680.0,
+ "12330": 34434887680.0,
+ "12335": 34434887680.0,
+ "12340": 34434887680.0,
+ "12345": 34434887680.0,
+ "12350": 34434887680.0,
+ "12355": 34434887680.0,
+ "12360": 34434887680.0,
+ "12365": 34434887680.0,
+ "12370": 34434887680.0,
+ "12375": 34434887680.0,
+ "12380": 34434887680.0,
+ "12385": 34434887680.0,
+ "12390": 34434887680.0,
+ "12395": 34434887680.0,
+ "12400": 34434887680.0,
+ "12405": 34434887680.0,
+ "12410": 34434887680.0,
+ "12415": 34434887680.0,
+ "12420": 34434887680.0,
+ "12425": 34434887680.0,
+ "12430": 34434887680.0,
+ "12435": 34434887680.0,
+ "12440": 34434887680.0,
+ "12445": 34434887680.0,
+ "12450": 34434887680.0,
+ "12455": 34434887680.0,
+ "12460": 34434887680.0,
+ "12465": 34434887680.0,
+ "12470": 34434887680.0,
+ "12475": 34434887680.0,
+ "12480": 34434887680.0,
+ "12485": 34434887680.0,
+ "12490": 34434887680.0,
+ "12495": 34434887680.0,
+ "12500": 34434887680.0,
+ "12505": 34434887680.0,
+ "12510": 34434887680.0,
+ "12515": 34434887680.0,
+ "12520": 34434887680.0,
+ "12525": 34434887680.0,
+ "12530": 34434887680.0,
+ "12535": 34434887680.0,
+ "12540": 34434887680.0,
+ "12545": 34434887680.0,
+ "12550": 34434887680.0,
+ "12555": 34434887680.0,
+ "12560": 34434887680.0,
+ "12565": 34434887680.0,
+ "12570": 34434887680.0,
+ "12575": 34434887680.0,
+ "12580": 34434887680.0,
+ "12585": 34434887680.0,
+ "12590": 34434887680.0,
+ "12595": 34434887680.0,
+ "12600": 34434887680.0,
+ "12605": 34434887680.0,
+ "12610": 34434887680.0,
+ "12615": 34434887680.0,
+ "12620": 34434887680.0,
+ "12625": 34434887680.0,
+ "12630": 34434887680.0,
+ "12635": 34434887680.0,
+ "12640": 34434887680.0,
+ "12645": 34434887680.0,
+ "12650": 34434887680.0,
+ "12655": 34434887680.0,
+ "12660": 34434887680.0,
+ "12665": 34434887680.0,
+ "12670": 34434887680.0,
+ "12675": 34434887680.0,
+ "12680": 34434887680.0,
+ "12685": 34434887680.0,
+ "12690": 34434887680.0,
+ "12695": 34434887680.0,
+ "12700": 34434887680.0,
+ "12705": 34434887680.0,
+ "12710": 34434887680.0,
+ "12715": 34434887680.0,
+ "12720": 34434887680.0,
+ "12725": 34434887680.0,
+ "12730": 34434887680.0,
+ "12735": 34434887680.0,
+ "12740": 34434887680.0,
+ "12745": 34434887680.0,
+ "12750": 34434887680.0,
+ "12755": 34434887680.0,
+ "12760": 34434887680.0,
+ "12765": 34434887680.0,
+ "12770": 34434887680.0,
+ "12775": 34434887680.0,
+ "12780": 34434887680.0,
+ "12785": 34434887680.0,
+ "12790": 34434887680.0,
+ "12795": 34434887680.0,
+ "12800": 34434887680.0,
+ "12805": 34434887680.0,
+ "12810": 34434887680.0,
+ "12815": 34434887680.0,
+ "12820": 34434887680.0,
+ "12825": 34434887680.0,
+ "12830": 34434887680.0,
+ "12835": 34434887680.0,
+ "12840": 34434887680.0,
+ "12845": 34434887680.0,
+ "12850": 34434887680.0,
+ "12855": 34434887680.0,
+ "12860": 34434887680.0,
+ "12865": 34434887680.0,
+ "12870": 34434887680.0,
+ "12875": 34434887680.0,
+ "12880": 34434887680.0,
+ "12885": 34434887680.0,
+ "12890": 34434887680.0,
+ "12895": 34434887680.0,
+ "12900": 34434887680.0,
+ "12905": 34434887680.0,
+ "12910": 34434887680.0,
+ "12915": 34434887680.0,
+ "12920": 34434887680.0,
+ "12925": 34434887680.0,
+ "12930": 34434887680.0,
+ "12935": 34434887680.0,
+ "12940": 34434887680.0,
+ "12945": 34434887680.0,
+ "12950": 34434887680.0,
+ "12955": 34434887680.0,
+ "12960": 34434887680.0,
+ "12965": 34434887680.0,
+ "12970": 34434887680.0,
+ "12975": 34434887680.0,
+ "12980": 34434887680.0,
+ "12985": 34434887680.0,
+ "12990": 34434887680.0,
+ "12995": 34434887680.0,
+ "13000": 34434887680.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 13000,
+ "step_interval": 5,
+ "values": {
+ "1": "nan",
+ "5": "nan",
+ "10": "nan",
+ "15": "nan",
+ "20": "nan",
+ "25": "nan",
+ "30": "nan",
+ "35": "nan",
+ "40": "nan",
+ "45": "nan",
+ "50": "nan",
+ "55": "nan",
+ "60": "nan",
+ "65": "nan",
+ "70": "nan",
+ "75": "nan",
+ "80": "nan",
+ "85": "nan",
+ "90": "nan",
+ "95": "nan",
+ "100": 1.52575,
+ "105": "nan",
+ "110": "nan",
+ "115": "nan",
+ "120": "nan",
+ "125": "nan",
+ "130": "nan",
+ "135": "nan",
+ "140": "nan",
+ "145": "nan",
+ "150": "nan",
+ "155": "nan",
+ "160": "nan",
+ "165": "nan",
+ "170": "nan",
+ "175": "nan",
+ "180": "nan",
+ "185": "nan",
+ "190": "nan",
+ "195": "nan",
+ "200": 1.22241,
+ "205": "nan",
+ "210": "nan",
+ "215": "nan",
+ "220": "nan",
+ "225": "nan",
+ "230": "nan",
+ "235": "nan",
+ "240": "nan",
+ "245": "nan",
+ "250": "nan",
+ "255": "nan",
+ "260": "nan",
+ "265": "nan",
+ "270": "nan",
+ "275": "nan",
+ "280": "nan",
+ "285": "nan",
+ "290": "nan",
+ "295": "nan",
+ "300": 1.68719,
+ "305": "nan",
+ "310": "nan",
+ "315": "nan",
+ "320": "nan",
+ "325": "nan",
+ "330": "nan",
+ "335": "nan",
+ "340": "nan",
+ "345": "nan",
+ "350": "nan",
+ "355": "nan",
+ "360": "nan",
+ "365": "nan",
+ "370": "nan",
+ "375": "nan",
+ "380": "nan",
+ "385": "nan",
+ "390": "nan",
+ "395": "nan",
+ "400": 2.3415,
+ "405": "nan",
+ "410": "nan",
+ "415": "nan",
+ "420": "nan",
+ "425": "nan",
+ "430": "nan",
+ "435": "nan",
+ "440": "nan",
+ "445": "nan",
+ "450": "nan",
+ "455": "nan",
+ "460": "nan",
+ "465": "nan",
+ "470": "nan",
+ "475": "nan",
+ "480": "nan",
+ "485": "nan",
+ "490": "nan",
+ "495": "nan",
+ "500": 1.22139,
+ "505": "nan",
+ "510": "nan",
+ "515": "nan",
+ "520": "nan",
+ "525": "nan",
+ "530": "nan",
+ "535": "nan",
+ "540": "nan",
+ "545": "nan",
+ "550": "nan",
+ "555": "nan",
+ "560": "nan",
+ "565": "nan",
+ "570": "nan",
+ "575": "nan",
+ "580": "nan",
+ "585": "nan",
+ "590": "nan",
+ "595": "nan",
+ "600": 1.64801,
+ "605": "nan",
+ "610": "nan",
+ "615": "nan",
+ "620": "nan",
+ "625": "nan",
+ "630": "nan",
+ "635": "nan",
+ "640": "nan",
+ "645": "nan",
+ "650": "nan",
+ "655": "nan",
+ "660": "nan",
+ "665": "nan",
+ "670": "nan",
+ "675": "nan",
+ "680": "nan",
+ "685": "nan",
+ "690": "nan",
+ "695": "nan",
+ "700": 1.59272,
+ "705": "nan",
+ "710": "nan",
+ "715": "nan",
+ "720": "nan",
+ "725": "nan",
+ "730": "nan",
+ "735": "nan",
+ "740": "nan",
+ "745": "nan",
+ "750": "nan",
+ "755": "nan",
+ "760": "nan",
+ "765": "nan",
+ "770": "nan",
+ "775": "nan",
+ "780": "nan",
+ "785": "nan",
+ "790": "nan",
+ "795": "nan",
+ "800": 1.2195,
+ "805": "nan",
+ "810": "nan",
+ "815": "nan",
+ "820": "nan",
+ "825": "nan",
+ "830": "nan",
+ "835": "nan",
+ "840": "nan",
+ "845": "nan",
+ "850": "nan",
+ "855": "nan",
+ "860": "nan",
+ "865": "nan",
+ "870": "nan",
+ "875": "nan",
+ "880": "nan",
+ "885": "nan",
+ "890": "nan",
+ "895": "nan",
+ "900": 1.62873,
+ "905": "nan",
+ "910": "nan",
+ "915": "nan",
+ "920": "nan",
+ "925": "nan",
+ "930": "nan",
+ "935": "nan",
+ "940": "nan",
+ "945": "nan",
+ "950": "nan",
+ "955": "nan",
+ "960": "nan",
+ "965": "nan",
+ "970": "nan",
+ "975": "nan",
+ "980": "nan",
+ "985": "nan",
+ "990": "nan",
+ "995": "nan",
+ "1000": 1.64961,
+ "1005": "nan",
+ "1010": "nan",
+ "1015": "nan",
+ "1020": "nan",
+ "1025": "nan",
+ "1030": "nan",
+ "1035": "nan",
+ "1040": "nan",
+ "1045": "nan",
+ "1050": "nan",
+ "1055": "nan",
+ "1060": "nan",
+ "1065": "nan",
+ "1070": "nan",
+ "1075": "nan",
+ "1080": "nan",
+ "1085": "nan",
+ "1090": "nan",
+ "1095": "nan",
+ "1100": 1.62131,
+ "1105": "nan",
+ "1110": "nan",
+ "1115": "nan",
+ "1120": "nan",
+ "1125": "nan",
+ "1130": "nan",
+ "1135": "nan",
+ "1140": "nan",
+ "1145": "nan",
+ "1150": "nan",
+ "1155": "nan",
+ "1160": "nan",
+ "1165": "nan",
+ "1170": "nan",
+ "1175": "nan",
+ "1180": "nan",
+ "1185": "nan",
+ "1190": "nan",
+ "1195": "nan",
+ "1200": 1.46576,
+ "1205": "nan",
+ "1210": "nan",
+ "1215": "nan",
+ "1220": "nan",
+ "1225": "nan",
+ "1230": "nan",
+ "1235": "nan",
+ "1240": "nan",
+ "1245": "nan",
+ "1250": "nan",
+ "1255": "nan",
+ "1260": "nan",
+ "1265": "nan",
+ "1270": "nan",
+ "1275": "nan",
+ "1280": "nan",
+ "1285": "nan",
+ "1290": "nan",
+ "1295": "nan",
+ "1300": 1.52481,
+ "1305": "nan",
+ "1310": "nan",
+ "1315": "nan",
+ "1320": "nan",
+ "1325": "nan",
+ "1330": "nan",
+ "1335": "nan",
+ "1340": "nan",
+ "1345": "nan",
+ "1350": "nan",
+ "1355": "nan",
+ "1360": "nan",
+ "1365": "nan",
+ "1370": "nan",
+ "1375": "nan",
+ "1380": "nan",
+ "1385": "nan",
+ "1390": "nan",
+ "1395": "nan",
+ "1400": 1.18935,
+ "1405": "nan",
+ "1410": "nan",
+ "1415": "nan",
+ "1420": "nan",
+ "1425": "nan",
+ "1430": "nan",
+ "1435": "nan",
+ "1440": "nan",
+ "1445": "nan",
+ "1450": "nan",
+ "1455": "nan",
+ "1460": "nan",
+ "1465": "nan",
+ "1470": "nan",
+ "1475": "nan",
+ "1480": "nan",
+ "1485": "nan",
+ "1490": "nan",
+ "1495": "nan",
+ "1500": 1.81193,
+ "1505": "nan",
+ "1510": "nan",
+ "1515": "nan",
+ "1520": "nan",
+ "1525": "nan",
+ "1530": "nan",
+ "1535": "nan",
+ "1540": "nan",
+ "1545": "nan",
+ "1550": "nan",
+ "1555": "nan",
+ "1560": "nan",
+ "1565": "nan",
+ "1570": "nan",
+ "1575": "nan",
+ "1580": "nan",
+ "1585": "nan",
+ "1590": "nan",
+ "1595": "nan",
+ "1600": 1.18361,
+ "1605": "nan",
+ "1610": "nan",
+ "1615": "nan",
+ "1620": "nan",
+ "1625": "nan",
+ "1630": "nan",
+ "1635": "nan",
+ "1640": "nan",
+ "1645": "nan",
+ "1650": "nan",
+ "1655": "nan",
+ "1660": "nan",
+ "1665": "nan",
+ "1670": "nan",
+ "1675": "nan",
+ "1680": "nan",
+ "1685": "nan",
+ "1690": "nan",
+ "1695": "nan",
+ "1700": 1.8225,
+ "1705": "nan",
+ "1710": "nan",
+ "1715": "nan",
+ "1720": "nan",
+ "1725": "nan",
+ "1730": "nan",
+ "1735": "nan",
+ "1740": "nan",
+ "1745": "nan",
+ "1750": "nan",
+ "1755": "nan",
+ "1760": "nan",
+ "1765": "nan",
+ "1770": "nan",
+ "1775": "nan",
+ "1780": "nan",
+ "1785": "nan",
+ "1790": "nan",
+ "1795": "nan",
+ "1800": 1.17987,
+ "1805": "nan",
+ "1810": "nan",
+ "1815": "nan",
+ "1820": "nan",
+ "1825": "nan",
+ "1830": "nan",
+ "1835": "nan",
+ "1840": "nan",
+ "1845": "nan",
+ "1850": "nan",
+ "1855": "nan",
+ "1860": "nan",
+ "1865": "nan",
+ "1870": "nan",
+ "1875": "nan",
+ "1880": "nan",
+ "1885": "nan",
+ "1890": "nan",
+ "1895": "nan",
+ "1900": 1.51305,
+ "1905": "nan",
+ "1910": "nan",
+ "1915": "nan",
+ "1920": "nan",
+ "1925": "nan",
+ "1930": "nan",
+ "1935": "nan",
+ "1940": "nan",
+ "1945": "nan",
+ "1950": "nan",
+ "1955": "nan",
+ "1960": "nan",
+ "1965": "nan",
+ "1970": "nan",
+ "1975": "nan",
+ "1980": "nan",
+ "1985": "nan",
+ "1990": "nan",
+ "1995": "nan",
+ "2000": 1.50011,
+ "2005": "nan",
+ "2010": "nan",
+ "2015": "nan",
+ "2020": "nan",
+ "2025": "nan",
+ "2030": "nan",
+ "2035": "nan",
+ "2040": "nan",
+ "2045": "nan",
+ "2050": "nan",
+ "2055": "nan",
+ "2060": "nan",
+ "2065": "nan",
+ "2070": "nan",
+ "2075": "nan",
+ "2080": "nan",
+ "2085": "nan",
+ "2090": "nan",
+ "2095": "nan",
+ "2100": 1.57718,
+ "2105": "nan",
+ "2110": "nan",
+ "2115": "nan",
+ "2120": "nan",
+ "2125": "nan",
+ "2130": "nan",
+ "2135": "nan",
+ "2140": "nan",
+ "2145": "nan",
+ "2150": "nan",
+ "2155": "nan",
+ "2160": "nan",
+ "2165": "nan",
+ "2170": "nan",
+ "2175": "nan",
+ "2180": "nan",
+ "2185": "nan",
+ "2190": "nan",
+ "2195": "nan",
+ "2200": 1.45196,
+ "2205": "nan",
+ "2210": "nan",
+ "2215": "nan",
+ "2220": "nan",
+ "2225": "nan",
+ "2230": "nan",
+ "2235": "nan",
+ "2240": "nan",
+ "2245": "nan",
+ "2250": "nan",
+ "2255": "nan",
+ "2260": "nan",
+ "2265": "nan",
+ "2270": "nan",
+ "2275": "nan",
+ "2280": "nan",
+ "2285": "nan",
+ "2290": "nan",
+ "2295": "nan",
+ "2300": 1.26751,
+ "2305": "nan",
+ "2310": "nan",
+ "2315": "nan",
+ "2320": "nan",
+ "2325": "nan",
+ "2330": "nan",
+ "2335": "nan",
+ "2340": "nan",
+ "2345": "nan",
+ "2350": "nan",
+ "2355": "nan",
+ "2360": "nan",
+ "2365": "nan",
+ "2370": "nan",
+ "2375": "nan",
+ "2380": "nan",
+ "2385": "nan",
+ "2390": "nan",
+ "2395": "nan",
+ "2400": 1.67576,
+ "2405": "nan",
+ "2410": "nan",
+ "2415": "nan",
+ "2420": "nan",
+ "2425": "nan",
+ "2430": "nan",
+ "2435": "nan",
+ "2440": "nan",
+ "2445": "nan",
+ "2450": "nan",
+ "2455": "nan",
+ "2460": "nan",
+ "2465": "nan",
+ "2470": "nan",
+ "2475": "nan",
+ "2480": "nan",
+ "2485": "nan",
+ "2490": "nan",
+ "2495": "nan",
+ "2500": 1.45152,
+ "2505": "nan",
+ "2510": "nan",
+ "2515": "nan",
+ "2520": "nan",
+ "2525": "nan",
+ "2530": "nan",
+ "2535": "nan",
+ "2540": "nan",
+ "2545": "nan",
+ "2550": "nan",
+ "2555": "nan",
+ "2560": "nan",
+ "2565": "nan",
+ "2570": "nan",
+ "2575": "nan",
+ "2580": "nan",
+ "2585": "nan",
+ "2590": "nan",
+ "2595": "nan",
+ "2600": 1.17207,
+ "2605": "nan",
+ "2610": "nan",
+ "2615": "nan",
+ "2620": "nan",
+ "2625": "nan",
+ "2630": "nan",
+ "2635": "nan",
+ "2640": "nan",
+ "2645": "nan",
+ "2650": "nan",
+ "2655": "nan",
+ "2660": "nan",
+ "2665": "nan",
+ "2670": "nan",
+ "2675": "nan",
+ "2680": "nan",
+ "2685": "nan",
+ "2690": "nan",
+ "2695": "nan",
+ "2700": 1.47473,
+ "2705": "nan",
+ "2710": "nan",
+ "2715": "nan",
+ "2720": "nan",
+ "2725": "nan",
+ "2730": "nan",
+ "2735": "nan",
+ "2740": "nan",
+ "2745": "nan",
+ "2750": "nan",
+ "2755": "nan",
+ "2760": "nan",
+ "2765": "nan",
+ "2770": "nan",
+ "2775": "nan",
+ "2780": "nan",
+ "2785": "nan",
+ "2790": "nan",
+ "2795": "nan",
+ "2800": 1.46554,
+ "2805": "nan",
+ "2810": "nan",
+ "2815": "nan",
+ "2820": "nan",
+ "2825": "nan",
+ "2830": "nan",
+ "2835": "nan",
+ "2840": "nan",
+ "2845": "nan",
+ "2850": "nan",
+ "2855": "nan",
+ "2860": "nan",
+ "2865": "nan",
+ "2870": "nan",
+ "2875": "nan",
+ "2880": "nan",
+ "2885": "nan",
+ "2890": "nan",
+ "2895": "nan",
+ "2900": 1.47107,
+ "2905": "nan",
+ "2910": "nan",
+ "2915": "nan",
+ "2920": "nan",
+ "2925": "nan",
+ "2930": "nan",
+ "2935": "nan",
+ "2940": "nan",
+ "2945": "nan",
+ "2950": "nan",
+ "2955": "nan",
+ "2960": "nan",
+ "2965": "nan",
+ "2970": "nan",
+ "2975": "nan",
+ "2980": "nan",
+ "2985": "nan",
+ "2990": "nan",
+ "2995": "nan",
+ "3000": 1.47313,
+ "3005": "nan",
+ "3010": "nan",
+ "3015": "nan",
+ "3020": "nan",
+ "3025": "nan",
+ "3030": "nan",
+ "3035": "nan",
+ "3040": "nan",
+ "3045": "nan",
+ "3050": "nan",
+ "3055": "nan",
+ "3060": "nan",
+ "3065": "nan",
+ "3070": "nan",
+ "3075": "nan",
+ "3080": "nan",
+ "3085": "nan",
+ "3090": "nan",
+ "3095": "nan",
+ "3100": 1.58134,
+ "3105": "nan",
+ "3110": "nan",
+ "3115": "nan",
+ "3120": "nan",
+ "3125": "nan",
+ "3130": "nan",
+ "3135": "nan",
+ "3140": "nan",
+ "3145": "nan",
+ "3150": "nan",
+ "3155": "nan",
+ "3160": "nan",
+ "3165": "nan",
+ "3170": "nan",
+ "3175": "nan",
+ "3180": "nan",
+ "3185": "nan",
+ "3190": "nan",
+ "3195": "nan",
+ "3200": 1.47784,
+ "3205": "nan",
+ "3210": "nan",
+ "3215": "nan",
+ "3220": "nan",
+ "3225": "nan",
+ "3230": "nan",
+ "3235": "nan",
+ "3240": "nan",
+ "3245": "nan",
+ "3250": "nan",
+ "3255": "nan",
+ "3260": "nan",
+ "3265": "nan",
+ "3270": "nan",
+ "3275": "nan",
+ "3280": "nan",
+ "3285": "nan",
+ "3290": "nan",
+ "3295": "nan",
+ "3300": 1.53281,
+ "3305": "nan",
+ "3310": "nan",
+ "3315": "nan",
+ "3320": "nan",
+ "3325": "nan",
+ "3330": "nan",
+ "3335": "nan",
+ "3340": "nan",
+ "3345": "nan",
+ "3350": "nan",
+ "3355": "nan",
+ "3360": "nan",
+ "3365": "nan",
+ "3370": "nan",
+ "3375": "nan",
+ "3380": "nan",
+ "3385": "nan",
+ "3390": "nan",
+ "3395": "nan",
+ "3400": 1.16771,
+ "3405": "nan",
+ "3410": "nan",
+ "3415": "nan",
+ "3420": "nan",
+ "3425": "nan",
+ "3430": "nan",
+ "3435": "nan",
+ "3440": "nan",
+ "3445": "nan",
+ "3450": "nan",
+ "3455": "nan",
+ "3460": "nan",
+ "3465": "nan",
+ "3470": "nan",
+ "3475": "nan",
+ "3480": "nan",
+ "3485": "nan",
+ "3490": "nan",
+ "3495": "nan",
+ "3500": 1.7677,
+ "3505": "nan",
+ "3510": "nan",
+ "3515": "nan",
+ "3520": "nan",
+ "3525": "nan",
+ "3530": "nan",
+ "3535": "nan",
+ "3540": "nan",
+ "3545": "nan",
+ "3550": "nan",
+ "3555": "nan",
+ "3560": "nan",
+ "3565": "nan",
+ "3570": "nan",
+ "3575": "nan",
+ "3580": "nan",
+ "3585": "nan",
+ "3590": "nan",
+ "3595": "nan",
+ "3600": 1.16778,
+ "3605": "nan",
+ "3610": "nan",
+ "3615": "nan",
+ "3620": "nan",
+ "3625": "nan",
+ "3630": "nan",
+ "3635": "nan",
+ "3640": "nan",
+ "3645": "nan",
+ "3650": "nan",
+ "3655": "nan",
+ "3660": "nan",
+ "3665": "nan",
+ "3670": "nan",
+ "3675": "nan",
+ "3680": "nan",
+ "3685": "nan",
+ "3690": "nan",
+ "3695": "nan",
+ "3700": 1.52902,
+ "3705": "nan",
+ "3710": "nan",
+ "3715": "nan",
+ "3720": "nan",
+ "3725": "nan",
+ "3730": "nan",
+ "3735": "nan",
+ "3740": "nan",
+ "3745": "nan",
+ "3750": "nan",
+ "3755": "nan",
+ "3760": "nan",
+ "3765": "nan",
+ "3770": "nan",
+ "3775": "nan",
+ "3780": "nan",
+ "3785": "nan",
+ "3790": "nan",
+ "3795": "nan",
+ "3800": 1.45421,
+ "3805": "nan",
+ "3810": "nan",
+ "3815": "nan",
+ "3820": "nan",
+ "3825": "nan",
+ "3830": "nan",
+ "3835": "nan",
+ "3840": "nan",
+ "3845": "nan",
+ "3850": "nan",
+ "3855": "nan",
+ "3860": "nan",
+ "3865": "nan",
+ "3870": "nan",
+ "3875": "nan",
+ "3880": "nan",
+ "3885": "nan",
+ "3890": "nan",
+ "3895": "nan",
+ "3900": 1.48924,
+ "3905": "nan",
+ "3910": "nan",
+ "3915": "nan",
+ "3920": "nan",
+ "3925": "nan",
+ "3930": "nan",
+ "3935": "nan",
+ "3940": "nan",
+ "3945": "nan",
+ "3950": "nan",
+ "3955": "nan",
+ "3960": "nan",
+ "3965": "nan",
+ "3970": "nan",
+ "3975": "nan",
+ "3980": "nan",
+ "3985": "nan",
+ "3990": "nan",
+ "3995": "nan",
+ "4000": 1.49248,
+ "4005": "nan",
+ "4010": "nan",
+ "4015": "nan",
+ "4020": "nan",
+ "4025": "nan",
+ "4030": "nan",
+ "4035": "nan",
+ "4040": "nan",
+ "4045": "nan",
+ "4050": "nan",
+ "4055": "nan",
+ "4060": "nan",
+ "4065": "nan",
+ "4070": "nan",
+ "4075": "nan",
+ "4080": "nan",
+ "4085": "nan",
+ "4090": "nan",
+ "4095": "nan",
+ "4100": 1.24661,
+ "4105": "nan",
+ "4110": "nan",
+ "4115": "nan",
+ "4120": "nan",
+ "4125": "nan",
+ "4130": "nan",
+ "4135": "nan",
+ "4140": "nan",
+ "4145": "nan",
+ "4150": "nan",
+ "4155": "nan",
+ "4160": "nan",
+ "4165": "nan",
+ "4170": "nan",
+ "4175": "nan",
+ "4180": "nan",
+ "4185": "nan",
+ "4190": "nan",
+ "4195": "nan",
+ "4200": 1.49468,
+ "4205": "nan",
+ "4210": "nan",
+ "4215": "nan",
+ "4220": "nan",
+ "4225": "nan",
+ "4230": "nan",
+ "4235": "nan",
+ "4240": "nan",
+ "4245": "nan",
+ "4250": "nan",
+ "4255": "nan",
+ "4260": "nan",
+ "4265": "nan",
+ "4270": "nan",
+ "4275": "nan",
+ "4280": "nan",
+ "4285": "nan",
+ "4290": "nan",
+ "4295": "nan",
+ "4300": 1.16555,
+ "4305": "nan",
+ "4310": "nan",
+ "4315": "nan",
+ "4320": "nan",
+ "4325": "nan",
+ "4330": "nan",
+ "4335": "nan",
+ "4340": "nan",
+ "4345": "nan",
+ "4350": "nan",
+ "4355": "nan",
+ "4360": "nan",
+ "4365": "nan",
+ "4370": "nan",
+ "4375": "nan",
+ "4380": "nan",
+ "4385": "nan",
+ "4390": "nan",
+ "4395": "nan",
+ "4400": 1.47217,
+ "4405": "nan",
+ "4410": "nan",
+ "4415": "nan",
+ "4420": "nan",
+ "4425": "nan",
+ "4430": "nan",
+ "4435": "nan",
+ "4440": "nan",
+ "4445": "nan",
+ "4450": "nan",
+ "4455": "nan",
+ "4460": "nan",
+ "4465": "nan",
+ "4470": "nan",
+ "4475": "nan",
+ "4480": "nan",
+ "4485": "nan",
+ "4490": "nan",
+ "4495": "nan",
+ "4500": 1.16663,
+ "4505": "nan",
+ "4510": "nan",
+ "4515": "nan",
+ "4520": "nan",
+ "4525": "nan",
+ "4530": "nan",
+ "4535": "nan",
+ "4540": "nan",
+ "4545": "nan",
+ "4550": "nan",
+ "4555": "nan",
+ "4560": "nan",
+ "4565": "nan",
+ "4570": "nan",
+ "4575": "nan",
+ "4580": "nan",
+ "4585": "nan",
+ "4590": "nan",
+ "4595": "nan",
+ "4600": 1.43693,
+ "4605": "nan",
+ "4610": "nan",
+ "4615": "nan",
+ "4620": "nan",
+ "4625": "nan",
+ "4630": "nan",
+ "4635": "nan",
+ "4640": "nan",
+ "4645": "nan",
+ "4650": "nan",
+ "4655": "nan",
+ "4660": "nan",
+ "4665": "nan",
+ "4670": "nan",
+ "4675": "nan",
+ "4680": "nan",
+ "4685": "nan",
+ "4690": "nan",
+ "4695": "nan",
+ "4700": 1.16482,
+ "4705": "nan",
+ "4710": "nan",
+ "4715": "nan",
+ "4720": "nan",
+ "4725": "nan",
+ "4730": "nan",
+ "4735": "nan",
+ "4740": "nan",
+ "4745": "nan",
+ "4750": "nan",
+ "4755": "nan",
+ "4760": "nan",
+ "4765": "nan",
+ "4770": "nan",
+ "4775": "nan",
+ "4780": "nan",
+ "4785": "nan",
+ "4790": "nan",
+ "4795": "nan",
+ "4800": 1.48482,
+ "4805": "nan",
+ "4810": "nan",
+ "4815": "nan",
+ "4820": "nan",
+ "4825": "nan",
+ "4830": "nan",
+ "4835": "nan",
+ "4840": "nan",
+ "4845": "nan",
+ "4850": "nan",
+ "4855": "nan",
+ "4860": "nan",
+ "4865": "nan",
+ "4870": "nan",
+ "4875": "nan",
+ "4880": "nan",
+ "4885": "nan",
+ "4890": "nan",
+ "4895": "nan",
+ "4900": 1.16433,
+ "4905": "nan",
+ "4910": "nan",
+ "4915": "nan",
+ "4920": "nan",
+ "4925": "nan",
+ "4930": "nan",
+ "4935": "nan",
+ "4940": "nan",
+ "4945": "nan",
+ "4950": "nan",
+ "4955": "nan",
+ "4960": "nan",
+ "4965": "nan",
+ "4970": "nan",
+ "4975": "nan",
+ "4980": "nan",
+ "4985": "nan",
+ "4990": "nan",
+ "4995": "nan",
+ "5000": 1.46317,
+ "5005": "nan",
+ "5010": "nan",
+ "5015": "nan",
+ "5020": "nan",
+ "5025": "nan",
+ "5030": "nan",
+ "5035": "nan",
+ "5040": "nan",
+ "5045": "nan",
+ "5050": "nan",
+ "5055": "nan",
+ "5060": "nan",
+ "5065": "nan",
+ "5070": "nan",
+ "5075": "nan",
+ "5080": "nan",
+ "5085": "nan",
+ "5090": "nan",
+ "5095": "nan",
+ "5100": 1.25189,
+ "5105": "nan",
+ "5110": "nan",
+ "5115": "nan",
+ "5120": "nan",
+ "5125": "nan",
+ "5130": "nan",
+ "5135": "nan",
+ "5140": "nan",
+ "5145": "nan",
+ "5150": "nan",
+ "5155": "nan",
+ "5160": "nan",
+ "5165": "nan",
+ "5170": "nan",
+ "5175": "nan",
+ "5180": "nan",
+ "5185": "nan",
+ "5190": "nan",
+ "5195": "nan",
+ "5200": 1.16439,
+ "5205": "nan",
+ "5210": "nan",
+ "5215": "nan",
+ "5220": "nan",
+ "5225": "nan",
+ "5230": "nan",
+ "5235": "nan",
+ "5240": "nan",
+ "5245": "nan",
+ "5250": "nan",
+ "5255": "nan",
+ "5260": "nan",
+ "5265": "nan",
+ "5270": "nan",
+ "5275": "nan",
+ "5280": "nan",
+ "5285": "nan",
+ "5290": "nan",
+ "5295": "nan",
+ "5300": 1.50195,
+ "5305": "nan",
+ "5310": "nan",
+ "5315": "nan",
+ "5320": "nan",
+ "5325": "nan",
+ "5330": "nan",
+ "5335": "nan",
+ "5340": "nan",
+ "5345": "nan",
+ "5350": "nan",
+ "5355": "nan",
+ "5360": "nan",
+ "5365": "nan",
+ "5370": "nan",
+ "5375": "nan",
+ "5380": "nan",
+ "5385": "nan",
+ "5390": "nan",
+ "5395": "nan",
+ "5400": 1.16391,
+ "5405": "nan",
+ "5410": "nan",
+ "5415": "nan",
+ "5420": "nan",
+ "5425": "nan",
+ "5430": "nan",
+ "5435": "nan",
+ "5440": "nan",
+ "5445": "nan",
+ "5450": "nan",
+ "5455": "nan",
+ "5460": "nan",
+ "5465": "nan",
+ "5470": "nan",
+ "5475": "nan",
+ "5480": "nan",
+ "5485": "nan",
+ "5490": "nan",
+ "5495": "nan",
+ "5500": 1.49666,
+ "5505": "nan",
+ "5510": "nan",
+ "5515": "nan",
+ "5520": "nan",
+ "5525": "nan",
+ "5530": "nan",
+ "5535": "nan",
+ "5540": "nan",
+ "5545": "nan",
+ "5550": "nan",
+ "5555": "nan",
+ "5560": "nan",
+ "5565": "nan",
+ "5570": "nan",
+ "5575": "nan",
+ "5580": "nan",
+ "5585": "nan",
+ "5590": "nan",
+ "5595": "nan",
+ "5600": 1.16446,
+ "5605": "nan",
+ "5610": "nan",
+ "5615": "nan",
+ "5620": "nan",
+ "5625": "nan",
+ "5630": "nan",
+ "5635": "nan",
+ "5640": "nan",
+ "5645": "nan",
+ "5650": "nan",
+ "5655": "nan",
+ "5660": "nan",
+ "5665": "nan",
+ "5670": "nan",
+ "5675": "nan",
+ "5680": "nan",
+ "5685": "nan",
+ "5690": "nan",
+ "5695": "nan",
+ "5700": 1.42544,
+ "5705": "nan",
+ "5710": "nan",
+ "5715": "nan",
+ "5720": "nan",
+ "5725": "nan",
+ "5730": "nan",
+ "5735": "nan",
+ "5740": "nan",
+ "5745": "nan",
+ "5750": "nan",
+ "5755": "nan",
+ "5760": "nan",
+ "5765": "nan",
+ "5770": "nan",
+ "5775": "nan",
+ "5780": "nan",
+ "5785": "nan",
+ "5790": "nan",
+ "5795": "nan",
+ "5800": 1.18903,
+ "5805": "nan",
+ "5810": "nan",
+ "5815": "nan",
+ "5820": "nan",
+ "5825": "nan",
+ "5830": "nan",
+ "5835": "nan",
+ "5840": "nan",
+ "5845": "nan",
+ "5850": "nan",
+ "5855": "nan",
+ "5860": "nan",
+ "5865": "nan",
+ "5870": "nan",
+ "5875": "nan",
+ "5880": "nan",
+ "5885": "nan",
+ "5890": "nan",
+ "5895": "nan",
+ "5900": 1.16376,
+ "5905": "nan",
+ "5910": "nan",
+ "5915": "nan",
+ "5920": "nan",
+ "5925": "nan",
+ "5930": "nan",
+ "5935": "nan",
+ "5940": "nan",
+ "5945": "nan",
+ "5950": "nan",
+ "5955": "nan",
+ "5960": "nan",
+ "5965": "nan",
+ "5970": "nan",
+ "5975": "nan",
+ "5980": "nan",
+ "5985": "nan",
+ "5990": "nan",
+ "5995": "nan",
+ "6000": 1.48894,
+ "6005": "nan",
+ "6010": "nan",
+ "6015": "nan",
+ "6020": "nan",
+ "6025": "nan",
+ "6030": "nan",
+ "6035": "nan",
+ "6040": "nan",
+ "6045": "nan",
+ "6050": "nan",
+ "6055": "nan",
+ "6060": "nan",
+ "6065": "nan",
+ "6070": "nan",
+ "6075": "nan",
+ "6080": "nan",
+ "6085": "nan",
+ "6090": "nan",
+ "6095": "nan",
+ "6100": 1.56874,
+ "6105": "nan",
+ "6110": "nan",
+ "6115": "nan",
+ "6120": "nan",
+ "6125": "nan",
+ "6130": "nan",
+ "6135": "nan",
+ "6140": "nan",
+ "6145": "nan",
+ "6150": "nan",
+ "6155": "nan",
+ "6160": "nan",
+ "6165": "nan",
+ "6170": "nan",
+ "6175": "nan",
+ "6180": "nan",
+ "6185": "nan",
+ "6190": "nan",
+ "6195": "nan",
+ "6200": 1.16367,
+ "6205": "nan",
+ "6210": "nan",
+ "6215": "nan",
+ "6220": "nan",
+ "6225": "nan",
+ "6230": "nan",
+ "6235": "nan",
+ "6240": "nan",
+ "6245": "nan",
+ "6250": "nan",
+ "6255": "nan",
+ "6260": "nan",
+ "6265": "nan",
+ "6270": "nan",
+ "6275": "nan",
+ "6280": "nan",
+ "6285": "nan",
+ "6290": "nan",
+ "6295": "nan",
+ "6300": 1.49027,
+ "6305": "nan",
+ "6310": "nan",
+ "6315": "nan",
+ "6320": "nan",
+ "6325": "nan",
+ "6330": "nan",
+ "6335": "nan",
+ "6340": "nan",
+ "6345": "nan",
+ "6350": "nan",
+ "6355": "nan",
+ "6360": "nan",
+ "6365": "nan",
+ "6370": "nan",
+ "6375": "nan",
+ "6380": "nan",
+ "6385": "nan",
+ "6390": "nan",
+ "6395": "nan",
+ "6400": 1.16275,
+ "6405": "nan",
+ "6410": "nan",
+ "6415": "nan",
+ "6420": "nan",
+ "6425": "nan",
+ "6430": "nan",
+ "6435": "nan",
+ "6440": "nan",
+ "6445": "nan",
+ "6450": "nan",
+ "6455": "nan",
+ "6460": "nan",
+ "6465": "nan",
+ "6470": "nan",
+ "6475": "nan",
+ "6480": "nan",
+ "6485": "nan",
+ "6490": "nan",
+ "6495": "nan",
+ "6500": 1.78937,
+ "6505": "nan",
+ "6510": "nan",
+ "6515": "nan",
+ "6520": "nan",
+ "6525": "nan",
+ "6530": "nan",
+ "6535": "nan",
+ "6540": "nan",
+ "6545": "nan",
+ "6550": "nan",
+ "6555": "nan",
+ "6560": "nan",
+ "6565": "nan",
+ "6570": "nan",
+ "6575": "nan",
+ "6580": "nan",
+ "6585": "nan",
+ "6590": "nan",
+ "6595": "nan",
+ "6600": 1.16414,
+ "6605": "nan",
+ "6610": "nan",
+ "6615": "nan",
+ "6620": "nan",
+ "6625": "nan",
+ "6630": "nan",
+ "6635": "nan",
+ "6640": "nan",
+ "6645": "nan",
+ "6650": "nan",
+ "6655": "nan",
+ "6660": "nan",
+ "6665": "nan",
+ "6670": "nan",
+ "6675": "nan",
+ "6680": "nan",
+ "6685": "nan",
+ "6690": "nan",
+ "6695": "nan",
+ "6700": 1.16369,
+ "6705": "nan",
+ "6710": "nan",
+ "6715": "nan",
+ "6720": "nan",
+ "6725": "nan",
+ "6730": "nan",
+ "6735": "nan",
+ "6740": "nan",
+ "6745": "nan",
+ "6750": "nan",
+ "6755": "nan",
+ "6760": "nan",
+ "6765": "nan",
+ "6770": "nan",
+ "6775": "nan",
+ "6780": "nan",
+ "6785": "nan",
+ "6790": "nan",
+ "6795": "nan",
+ "6800": 1.87916,
+ "6805": "nan",
+ "6810": "nan",
+ "6815": "nan",
+ "6820": "nan",
+ "6825": "nan",
+ "6830": "nan",
+ "6835": "nan",
+ "6840": "nan",
+ "6845": "nan",
+ "6850": "nan",
+ "6855": "nan",
+ "6860": "nan",
+ "6865": "nan",
+ "6870": "nan",
+ "6875": "nan",
+ "6880": "nan",
+ "6885": "nan",
+ "6890": "nan",
+ "6895": "nan",
+ "6900": 1.74932,
+ "6905": "nan",
+ "6910": "nan",
+ "6915": "nan",
+ "6920": "nan",
+ "6925": "nan",
+ "6930": "nan",
+ "6935": "nan",
+ "6940": "nan",
+ "6945": "nan",
+ "6950": "nan",
+ "6955": "nan",
+ "6960": "nan",
+ "6965": "nan",
+ "6970": "nan",
+ "6975": "nan",
+ "6980": "nan",
+ "6985": "nan",
+ "6990": "nan",
+ "6995": "nan",
+ "7000": 1.26768,
+ "7005": "nan",
+ "7010": "nan",
+ "7015": "nan",
+ "7020": "nan",
+ "7025": "nan",
+ "7030": "nan",
+ "7035": "nan",
+ "7040": "nan",
+ "7045": "nan",
+ "7050": "nan",
+ "7055": "nan",
+ "7060": "nan",
+ "7065": "nan",
+ "7070": "nan",
+ "7075": "nan",
+ "7080": "nan",
+ "7085": "nan",
+ "7090": "nan",
+ "7095": "nan",
+ "7100": 1.56361,
+ "7105": "nan",
+ "7110": "nan",
+ "7115": "nan",
+ "7120": "nan",
+ "7125": "nan",
+ "7130": "nan",
+ "7135": "nan",
+ "7140": "nan",
+ "7145": "nan",
+ "7150": "nan",
+ "7155": "nan",
+ "7160": "nan",
+ "7165": "nan",
+ "7170": "nan",
+ "7175": "nan",
+ "7180": "nan",
+ "7185": "nan",
+ "7190": "nan",
+ "7195": "nan",
+ "7200": 1.46637,
+ "7205": "nan",
+ "7210": "nan",
+ "7215": "nan",
+ "7220": "nan",
+ "7225": "nan",
+ "7230": "nan",
+ "7235": "nan",
+ "7240": "nan",
+ "7245": "nan",
+ "7250": "nan",
+ "7255": "nan",
+ "7260": "nan",
+ "7265": "nan",
+ "7270": "nan",
+ "7275": "nan",
+ "7280": "nan",
+ "7285": "nan",
+ "7290": "nan",
+ "7295": "nan",
+ "7300": 1.47082,
+ "7305": "nan",
+ "7310": "nan",
+ "7315": "nan",
+ "7320": "nan",
+ "7325": "nan",
+ "7330": "nan",
+ "7335": "nan",
+ "7340": "nan",
+ "7345": "nan",
+ "7350": "nan",
+ "7355": "nan",
+ "7360": "nan",
+ "7365": "nan",
+ "7370": "nan",
+ "7375": "nan",
+ "7380": "nan",
+ "7385": "nan",
+ "7390": "nan",
+ "7395": "nan",
+ "7400": 1.46294,
+ "7405": "nan",
+ "7410": "nan",
+ "7415": "nan",
+ "7420": "nan",
+ "7425": "nan",
+ "7430": "nan",
+ "7435": "nan",
+ "7440": "nan",
+ "7445": "nan",
+ "7450": "nan",
+ "7455": "nan",
+ "7460": "nan",
+ "7465": "nan",
+ "7470": "nan",
+ "7475": "nan",
+ "7480": "nan",
+ "7485": "nan",
+ "7490": "nan",
+ "7495": "nan",
+ "7500": 1.48069,
+ "7505": "nan",
+ "7510": "nan",
+ "7515": "nan",
+ "7520": "nan",
+ "7525": "nan",
+ "7530": "nan",
+ "7535": "nan",
+ "7540": "nan",
+ "7545": "nan",
+ "7550": "nan",
+ "7555": "nan",
+ "7560": "nan",
+ "7565": "nan",
+ "7570": "nan",
+ "7575": "nan",
+ "7580": "nan",
+ "7585": "nan",
+ "7590": "nan",
+ "7595": "nan",
+ "7600": 1.44277,
+ "7605": "nan",
+ "7610": "nan",
+ "7615": "nan",
+ "7620": "nan",
+ "7625": "nan",
+ "7630": "nan",
+ "7635": "nan",
+ "7640": "nan",
+ "7645": "nan",
+ "7650": "nan",
+ "7655": "nan",
+ "7660": "nan",
+ "7665": "nan",
+ "7670": "nan",
+ "7675": "nan",
+ "7680": "nan",
+ "7685": "nan",
+ "7690": "nan",
+ "7695": "nan",
+ "7700": 1.44688,
+ "7705": "nan",
+ "7710": "nan",
+ "7715": "nan",
+ "7720": "nan",
+ "7725": "nan",
+ "7730": "nan",
+ "7735": "nan",
+ "7740": "nan",
+ "7745": "nan",
+ "7750": "nan",
+ "7755": "nan",
+ "7760": "nan",
+ "7765": "nan",
+ "7770": "nan",
+ "7775": "nan",
+ "7780": "nan",
+ "7785": "nan",
+ "7790": "nan",
+ "7795": "nan",
+ "7800": 1.46189,
+ "7805": "nan",
+ "7810": "nan",
+ "7815": "nan",
+ "7820": "nan",
+ "7825": "nan",
+ "7830": "nan",
+ "7835": "nan",
+ "7840": "nan",
+ "7845": "nan",
+ "7850": "nan",
+ "7855": "nan",
+ "7860": "nan",
+ "7865": "nan",
+ "7870": "nan",
+ "7875": "nan",
+ "7880": "nan",
+ "7885": "nan",
+ "7890": "nan",
+ "7895": "nan",
+ "7900": 1.48119,
+ "7905": "nan",
+ "7910": "nan",
+ "7915": "nan",
+ "7920": "nan",
+ "7925": "nan",
+ "7930": "nan",
+ "7935": "nan",
+ "7940": "nan",
+ "7945": "nan",
+ "7950": "nan",
+ "7955": "nan",
+ "7960": "nan",
+ "7965": "nan",
+ "7970": "nan",
+ "7975": "nan",
+ "7980": "nan",
+ "7985": "nan",
+ "7990": "nan",
+ "7995": "nan",
+ "8000": 1.16319,
+ "8005": "nan",
+ "8010": "nan",
+ "8015": "nan",
+ "8020": "nan",
+ "8025": "nan",
+ "8030": "nan",
+ "8035": "nan",
+ "8040": "nan",
+ "8045": "nan",
+ "8050": "nan",
+ "8055": "nan",
+ "8060": "nan",
+ "8065": "nan",
+ "8070": "nan",
+ "8075": "nan",
+ "8080": "nan",
+ "8085": "nan",
+ "8090": "nan",
+ "8095": "nan",
+ "8100": 1.56123,
+ "8105": "nan",
+ "8110": "nan",
+ "8115": "nan",
+ "8120": "nan",
+ "8125": "nan",
+ "8130": "nan",
+ "8135": "nan",
+ "8140": "nan",
+ "8145": "nan",
+ "8150": "nan",
+ "8155": "nan",
+ "8160": "nan",
+ "8165": "nan",
+ "8170": "nan",
+ "8175": "nan",
+ "8180": "nan",
+ "8185": "nan",
+ "8190": "nan",
+ "8195": "nan",
+ "8200": 1.47947,
+ "8205": "nan",
+ "8210": "nan",
+ "8215": "nan",
+ "8220": "nan",
+ "8225": "nan",
+ "8230": "nan",
+ "8235": "nan",
+ "8240": "nan",
+ "8245": "nan",
+ "8250": "nan",
+ "8255": "nan",
+ "8260": "nan",
+ "8265": "nan",
+ "8270": "nan",
+ "8275": "nan",
+ "8280": "nan",
+ "8285": "nan",
+ "8290": "nan",
+ "8295": "nan",
+ "8300": 1.16329,
+ "8305": "nan",
+ "8310": "nan",
+ "8315": "nan",
+ "8320": "nan",
+ "8325": "nan",
+ "8330": "nan",
+ "8335": "nan",
+ "8340": "nan",
+ "8345": "nan",
+ "8350": "nan",
+ "8355": "nan",
+ "8360": "nan",
+ "8365": "nan",
+ "8370": "nan",
+ "8375": "nan",
+ "8380": "nan",
+ "8385": "nan",
+ "8390": "nan",
+ "8395": "nan",
+ "8400": 1.76645,
+ "8405": "nan",
+ "8410": "nan",
+ "8415": "nan",
+ "8420": "nan",
+ "8425": "nan",
+ "8430": "nan",
+ "8435": "nan",
+ "8440": "nan",
+ "8445": "nan",
+ "8450": "nan",
+ "8455": "nan",
+ "8460": "nan",
+ "8465": "nan",
+ "8470": "nan",
+ "8475": "nan",
+ "8480": "nan",
+ "8485": "nan",
+ "8490": "nan",
+ "8495": "nan",
+ "8500": 1.24745,
+ "8505": "nan",
+ "8510": "nan",
+ "8515": "nan",
+ "8520": "nan",
+ "8525": "nan",
+ "8530": "nan",
+ "8535": "nan",
+ "8540": "nan",
+ "8545": "nan",
+ "8550": "nan",
+ "8555": "nan",
+ "8560": "nan",
+ "8565": "nan",
+ "8570": "nan",
+ "8575": "nan",
+ "8580": "nan",
+ "8585": "nan",
+ "8590": "nan",
+ "8595": "nan",
+ "8600": 1.50472,
+ "8605": "nan",
+ "8610": "nan",
+ "8615": "nan",
+ "8620": "nan",
+ "8625": "nan",
+ "8630": "nan",
+ "8635": "nan",
+ "8640": "nan",
+ "8645": "nan",
+ "8650": "nan",
+ "8655": "nan",
+ "8660": "nan",
+ "8665": "nan",
+ "8670": "nan",
+ "8675": "nan",
+ "8680": "nan",
+ "8685": "nan",
+ "8690": "nan",
+ "8695": "nan",
+ "8700": 1.2351,
+ "8705": "nan",
+ "8710": "nan",
+ "8715": "nan",
+ "8720": "nan",
+ "8725": "nan",
+ "8730": "nan",
+ "8735": "nan",
+ "8740": "nan",
+ "8745": "nan",
+ "8750": "nan",
+ "8755": "nan",
+ "8760": "nan",
+ "8765": "nan",
+ "8770": "nan",
+ "8775": "nan",
+ "8780": "nan",
+ "8785": "nan",
+ "8790": "nan",
+ "8795": "nan",
+ "8800": 1.23483,
+ "8805": "nan",
+ "8810": "nan",
+ "8815": "nan",
+ "8820": "nan",
+ "8825": "nan",
+ "8830": "nan",
+ "8835": "nan",
+ "8840": "nan",
+ "8845": "nan",
+ "8850": "nan",
+ "8855": "nan",
+ "8860": "nan",
+ "8865": "nan",
+ "8870": "nan",
+ "8875": "nan",
+ "8880": "nan",
+ "8885": "nan",
+ "8890": "nan",
+ "8895": "nan",
+ "8900": 1.44634,
+ "8905": "nan",
+ "8910": "nan",
+ "8915": "nan",
+ "8920": "nan",
+ "8925": "nan",
+ "8930": "nan",
+ "8935": "nan",
+ "8940": "nan",
+ "8945": "nan",
+ "8950": "nan",
+ "8955": "nan",
+ "8960": "nan",
+ "8965": "nan",
+ "8970": "nan",
+ "8975": "nan",
+ "8980": "nan",
+ "8985": "nan",
+ "8990": "nan",
+ "8995": "nan",
+ "9000": 1.28102,
+ "9005": "nan",
+ "9010": "nan",
+ "9015": "nan",
+ "9020": "nan",
+ "9025": "nan",
+ "9030": "nan",
+ "9035": "nan",
+ "9040": "nan",
+ "9045": "nan",
+ "9050": "nan",
+ "9055": "nan",
+ "9060": "nan",
+ "9065": "nan",
+ "9070": "nan",
+ "9075": "nan",
+ "9080": "nan",
+ "9085": "nan",
+ "9090": "nan",
+ "9095": "nan",
+ "9100": 1.9791,
+ "9105": "nan",
+ "9110": "nan",
+ "9115": "nan",
+ "9120": "nan",
+ "9125": "nan",
+ "9130": "nan",
+ "9135": "nan",
+ "9140": "nan",
+ "9145": "nan",
+ "9150": "nan",
+ "9155": "nan",
+ "9160": "nan",
+ "9165": "nan",
+ "9170": "nan",
+ "9175": "nan",
+ "9180": "nan",
+ "9185": "nan",
+ "9190": "nan",
+ "9195": "nan",
+ "9200": 1.23709,
+ "9205": "nan",
+ "9210": "nan",
+ "9215": "nan",
+ "9220": "nan",
+ "9225": "nan",
+ "9230": "nan",
+ "9235": "nan",
+ "9240": "nan",
+ "9245": "nan",
+ "9250": "nan",
+ "9255": "nan",
+ "9260": "nan",
+ "9265": "nan",
+ "9270": "nan",
+ "9275": "nan",
+ "9280": "nan",
+ "9285": "nan",
+ "9290": "nan",
+ "9295": "nan",
+ "9300": 1.23759,
+ "9305": "nan",
+ "9310": "nan",
+ "9315": "nan",
+ "9320": "nan",
+ "9325": "nan",
+ "9330": "nan",
+ "9335": "nan",
+ "9340": "nan",
+ "9345": "nan",
+ "9350": "nan",
+ "9355": "nan",
+ "9360": "nan",
+ "9365": "nan",
+ "9370": "nan",
+ "9375": "nan",
+ "9380": "nan",
+ "9385": "nan",
+ "9390": "nan",
+ "9395": "nan",
+ "9400": 1.99173,
+ "9405": "nan",
+ "9410": "nan",
+ "9415": "nan",
+ "9420": "nan",
+ "9425": "nan",
+ "9430": "nan",
+ "9435": "nan",
+ "9440": "nan",
+ "9445": "nan",
+ "9450": "nan",
+ "9455": "nan",
+ "9460": "nan",
+ "9465": "nan",
+ "9470": "nan",
+ "9475": "nan",
+ "9480": "nan",
+ "9485": "nan",
+ "9490": "nan",
+ "9495": "nan",
+ "9500": 1.53391,
+ "9505": "nan",
+ "9510": "nan",
+ "9515": "nan",
+ "9520": "nan",
+ "9525": "nan",
+ "9530": "nan",
+ "9535": "nan",
+ "9540": "nan",
+ "9545": "nan",
+ "9550": "nan",
+ "9555": "nan",
+ "9560": "nan",
+ "9565": "nan",
+ "9570": "nan",
+ "9575": "nan",
+ "9580": "nan",
+ "9585": "nan",
+ "9590": "nan",
+ "9595": "nan",
+ "9600": 1.59234,
+ "9605": "nan",
+ "9610": "nan",
+ "9615": "nan",
+ "9620": "nan",
+ "9625": "nan",
+ "9630": "nan",
+ "9635": "nan",
+ "9640": "nan",
+ "9645": "nan",
+ "9650": "nan",
+ "9655": "nan",
+ "9660": "nan",
+ "9665": "nan",
+ "9670": "nan",
+ "9675": "nan",
+ "9680": "nan",
+ "9685": "nan",
+ "9690": "nan",
+ "9695": "nan",
+ "9700": 1.23829,
+ "9705": "nan",
+ "9710": "nan",
+ "9715": "nan",
+ "9720": "nan",
+ "9725": "nan",
+ "9730": "nan",
+ "9735": "nan",
+ "9740": "nan",
+ "9745": "nan",
+ "9750": "nan",
+ "9755": "nan",
+ "9760": "nan",
+ "9765": "nan",
+ "9770": "nan",
+ "9775": "nan",
+ "9780": "nan",
+ "9785": "nan",
+ "9790": "nan",
+ "9795": "nan",
+ "9800": 1.94369,
+ "9805": "nan",
+ "9810": "nan",
+ "9815": "nan",
+ "9820": "nan",
+ "9825": "nan",
+ "9830": "nan",
+ "9835": "nan",
+ "9840": "nan",
+ "9845": "nan",
+ "9850": "nan",
+ "9855": "nan",
+ "9860": "nan",
+ "9865": "nan",
+ "9870": "nan",
+ "9875": "nan",
+ "9880": "nan",
+ "9885": "nan",
+ "9890": "nan",
+ "9895": "nan",
+ "9900": 1.54022,
+ "9905": "nan",
+ "9910": "nan",
+ "9915": "nan",
+ "9920": "nan",
+ "9925": "nan",
+ "9930": "nan",
+ "9935": "nan",
+ "9940": "nan",
+ "9945": "nan",
+ "9950": "nan",
+ "9955": "nan",
+ "9960": "nan",
+ "9965": "nan",
+ "9970": "nan",
+ "9975": "nan",
+ "9980": "nan",
+ "9985": "nan",
+ "9990": "nan",
+ "9995": "nan",
+ "10000": 1.61649,
+ "10005": "nan",
+ "10010": "nan",
+ "10015": "nan",
+ "10020": "nan",
+ "10025": "nan",
+ "10030": "nan",
+ "10035": "nan",
+ "10040": "nan",
+ "10045": "nan",
+ "10050": "nan",
+ "10055": "nan",
+ "10060": "nan",
+ "10065": "nan",
+ "10070": "nan",
+ "10075": "nan",
+ "10080": "nan",
+ "10085": "nan",
+ "10090": "nan",
+ "10095": "nan",
+ "10100": 1.85307,
+ "10105": "nan",
+ "10110": "nan",
+ "10115": "nan",
+ "10120": "nan",
+ "10125": "nan",
+ "10130": "nan",
+ "10135": "nan",
+ "10140": "nan",
+ "10145": "nan",
+ "10150": "nan",
+ "10155": "nan",
+ "10160": "nan",
+ "10165": "nan",
+ "10170": "nan",
+ "10175": "nan",
+ "10180": "nan",
+ "10185": "nan",
+ "10190": "nan",
+ "10195": "nan",
+ "10200": 1.3129,
+ "10205": "nan",
+ "10210": "nan",
+ "10215": "nan",
+ "10220": "nan",
+ "10225": "nan",
+ "10230": "nan",
+ "10235": "nan",
+ "10240": "nan",
+ "10245": "nan",
+ "10250": "nan",
+ "10255": "nan",
+ "10260": "nan",
+ "10265": "nan",
+ "10270": "nan",
+ "10275": "nan",
+ "10280": "nan",
+ "10285": "nan",
+ "10290": "nan",
+ "10295": "nan",
+ "10300": 1.57995,
+ "10305": "nan",
+ "10310": "nan",
+ "10315": "nan",
+ "10320": "nan",
+ "10325": "nan",
+ "10330": "nan",
+ "10335": "nan",
+ "10340": "nan",
+ "10345": "nan",
+ "10350": "nan",
+ "10355": "nan",
+ "10360": "nan",
+ "10365": "nan",
+ "10370": "nan",
+ "10375": "nan",
+ "10380": "nan",
+ "10385": "nan",
+ "10390": "nan",
+ "10395": "nan",
+ "10400": 1.55986,
+ "10405": "nan",
+ "10410": "nan",
+ "10415": "nan",
+ "10420": "nan",
+ "10425": "nan",
+ "10430": "nan",
+ "10435": "nan",
+ "10440": "nan",
+ "10445": "nan",
+ "10450": "nan",
+ "10455": "nan",
+ "10460": "nan",
+ "10465": "nan",
+ "10470": "nan",
+ "10475": "nan",
+ "10480": "nan",
+ "10485": "nan",
+ "10490": "nan",
+ "10495": "nan",
+ "10500": 1.54981,
+ "10505": "nan",
+ "10510": "nan",
+ "10515": "nan",
+ "10520": "nan",
+ "10525": "nan",
+ "10530": "nan",
+ "10535": "nan",
+ "10540": "nan",
+ "10545": "nan",
+ "10550": "nan",
+ "10555": "nan",
+ "10560": "nan",
+ "10565": "nan",
+ "10570": "nan",
+ "10575": "nan",
+ "10580": "nan",
+ "10585": "nan",
+ "10590": "nan",
+ "10595": "nan",
+ "10600": 1.61207,
+ "10605": "nan",
+ "10610": "nan",
+ "10615": "nan",
+ "10620": "nan",
+ "10625": "nan",
+ "10630": "nan",
+ "10635": "nan",
+ "10640": "nan",
+ "10645": "nan",
+ "10650": "nan",
+ "10655": "nan",
+ "10660": "nan",
+ "10665": "nan",
+ "10670": "nan",
+ "10675": "nan",
+ "10680": "nan",
+ "10685": "nan",
+ "10690": "nan",
+ "10695": "nan",
+ "10700": 1.56274,
+ "10705": "nan",
+ "10710": "nan",
+ "10715": "nan",
+ "10720": "nan",
+ "10725": "nan",
+ "10730": "nan",
+ "10735": "nan",
+ "10740": "nan",
+ "10745": "nan",
+ "10750": "nan",
+ "10755": "nan",
+ "10760": "nan",
+ "10765": "nan",
+ "10770": "nan",
+ "10775": "nan",
+ "10780": "nan",
+ "10785": "nan",
+ "10790": "nan",
+ "10795": "nan",
+ "10800": 1.60491,
+ "10805": "nan",
+ "10810": "nan",
+ "10815": "nan",
+ "10820": "nan",
+ "10825": "nan",
+ "10830": "nan",
+ "10835": "nan",
+ "10840": "nan",
+ "10845": "nan",
+ "10850": "nan",
+ "10855": "nan",
+ "10860": "nan",
+ "10865": "nan",
+ "10870": "nan",
+ "10875": "nan",
+ "10880": "nan",
+ "10885": "nan",
+ "10890": "nan",
+ "10895": "nan",
+ "10900": 1.54037,
+ "10905": "nan",
+ "10910": "nan",
+ "10915": "nan",
+ "10920": "nan",
+ "10925": "nan",
+ "10930": "nan",
+ "10935": "nan",
+ "10940": "nan",
+ "10945": "nan",
+ "10950": "nan",
+ "10955": "nan",
+ "10960": "nan",
+ "10965": "nan",
+ "10970": "nan",
+ "10975": "nan",
+ "10980": "nan",
+ "10985": "nan",
+ "10990": "nan",
+ "10995": "nan",
+ "11000": 1.53164,
+ "11005": "nan",
+ "11010": "nan",
+ "11015": "nan",
+ "11020": "nan",
+ "11025": "nan",
+ "11030": "nan",
+ "11035": "nan",
+ "11040": "nan",
+ "11045": "nan",
+ "11050": "nan",
+ "11055": "nan",
+ "11060": "nan",
+ "11065": "nan",
+ "11070": "nan",
+ "11075": "nan",
+ "11080": "nan",
+ "11085": "nan",
+ "11090": "nan",
+ "11095": "nan",
+ "11100": 1.6827,
+ "11105": "nan",
+ "11110": "nan",
+ "11115": "nan",
+ "11120": "nan",
+ "11125": "nan",
+ "11130": "nan",
+ "11135": "nan",
+ "11140": "nan",
+ "11145": "nan",
+ "11150": "nan",
+ "11155": "nan",
+ "11160": "nan",
+ "11165": "nan",
+ "11170": "nan",
+ "11175": "nan",
+ "11180": "nan",
+ "11185": "nan",
+ "11190": "nan",
+ "11195": "nan",
+ "11200": 1.24755,
+ "11205": "nan",
+ "11210": "nan",
+ "11215": "nan",
+ "11220": "nan",
+ "11225": "nan",
+ "11230": "nan",
+ "11235": "nan",
+ "11240": "nan",
+ "11245": "nan",
+ "11250": "nan",
+ "11255": "nan",
+ "11260": "nan",
+ "11265": "nan",
+ "11270": "nan",
+ "11275": "nan",
+ "11280": "nan",
+ "11285": "nan",
+ "11290": "nan",
+ "11295": "nan",
+ "11300": 1.58769,
+ "11305": "nan",
+ "11310": "nan",
+ "11315": "nan",
+ "11320": "nan",
+ "11325": "nan",
+ "11330": "nan",
+ "11335": "nan",
+ "11340": "nan",
+ "11345": "nan",
+ "11350": "nan",
+ "11355": "nan",
+ "11360": "nan",
+ "11365": "nan",
+ "11370": "nan",
+ "11375": "nan",
+ "11380": "nan",
+ "11385": "nan",
+ "11390": "nan",
+ "11395": "nan",
+ "11400": 1.24695,
+ "11405": "nan",
+ "11410": "nan",
+ "11415": "nan",
+ "11420": "nan",
+ "11425": "nan",
+ "11430": "nan",
+ "11435": "nan",
+ "11440": "nan",
+ "11445": "nan",
+ "11450": "nan",
+ "11455": "nan",
+ "11460": "nan",
+ "11465": "nan",
+ "11470": "nan",
+ "11475": "nan",
+ "11480": "nan",
+ "11485": "nan",
+ "11490": "nan",
+ "11495": "nan",
+ "11500": 1.26924,
+ "11505": "nan",
+ "11510": "nan",
+ "11515": "nan",
+ "11520": "nan",
+ "11525": "nan",
+ "11530": "nan",
+ "11535": "nan",
+ "11540": "nan",
+ "11545": "nan",
+ "11550": "nan",
+ "11555": "nan",
+ "11560": "nan",
+ "11565": "nan",
+ "11570": "nan",
+ "11575": "nan",
+ "11580": "nan",
+ "11585": "nan",
+ "11590": "nan",
+ "11595": "nan",
+ "11600": 1.58092,
+ "11605": "nan",
+ "11610": "nan",
+ "11615": "nan",
+ "11620": "nan",
+ "11625": "nan",
+ "11630": "nan",
+ "11635": "nan",
+ "11640": "nan",
+ "11645": "nan",
+ "11650": "nan",
+ "11655": "nan",
+ "11660": "nan",
+ "11665": "nan",
+ "11670": "nan",
+ "11675": "nan",
+ "11680": "nan",
+ "11685": "nan",
+ "11690": "nan",
+ "11695": "nan",
+ "11700": 1.2414,
+ "11705": "nan",
+ "11710": "nan",
+ "11715": "nan",
+ "11720": "nan",
+ "11725": "nan",
+ "11730": "nan",
+ "11735": "nan",
+ "11740": "nan",
+ "11745": "nan",
+ "11750": "nan",
+ "11755": "nan",
+ "11760": "nan",
+ "11765": "nan",
+ "11770": "nan",
+ "11775": "nan",
+ "11780": "nan",
+ "11785": "nan",
+ "11790": "nan",
+ "11795": "nan",
+ "11800": 1.55653,
+ "11805": "nan",
+ "11810": "nan",
+ "11815": "nan",
+ "11820": "nan",
+ "11825": "nan",
+ "11830": "nan",
+ "11835": "nan",
+ "11840": "nan",
+ "11845": "nan",
+ "11850": "nan",
+ "11855": "nan",
+ "11860": "nan",
+ "11865": "nan",
+ "11870": "nan",
+ "11875": "nan",
+ "11880": "nan",
+ "11885": "nan",
+ "11890": "nan",
+ "11895": "nan",
+ "11900": 1.24347,
+ "11905": "nan",
+ "11910": "nan",
+ "11915": "nan",
+ "11920": "nan",
+ "11925": "nan",
+ "11930": "nan",
+ "11935": "nan",
+ "11940": "nan",
+ "11945": "nan",
+ "11950": "nan",
+ "11955": "nan",
+ "11960": "nan",
+ "11965": "nan",
+ "11970": "nan",
+ "11975": "nan",
+ "11980": "nan",
+ "11985": "nan",
+ "11990": "nan",
+ "11995": "nan",
+ "12000": 1.26258,
+ "12005": "nan",
+ "12010": "nan",
+ "12015": "nan",
+ "12020": "nan",
+ "12025": "nan",
+ "12030": "nan",
+ "12035": "nan",
+ "12040": "nan",
+ "12045": "nan",
+ "12050": "nan",
+ "12055": "nan",
+ "12060": "nan",
+ "12065": "nan",
+ "12070": "nan",
+ "12075": "nan",
+ "12080": "nan",
+ "12085": "nan",
+ "12090": "nan",
+ "12095": "nan",
+ "12100": 1.31137,
+ "12105": "nan",
+ "12110": "nan",
+ "12115": "nan",
+ "12120": "nan",
+ "12125": "nan",
+ "12130": "nan",
+ "12135": "nan",
+ "12140": "nan",
+ "12145": "nan",
+ "12150": "nan",
+ "12155": "nan",
+ "12160": "nan",
+ "12165": "nan",
+ "12170": "nan",
+ "12175": "nan",
+ "12180": "nan",
+ "12185": "nan",
+ "12190": "nan",
+ "12195": "nan",
+ "12200": 1.2351,
+ "12205": "nan",
+ "12210": "nan",
+ "12215": "nan",
+ "12220": "nan",
+ "12225": "nan",
+ "12230": "nan",
+ "12235": "nan",
+ "12240": "nan",
+ "12245": "nan",
+ "12250": "nan",
+ "12255": "nan",
+ "12260": "nan",
+ "12265": "nan",
+ "12270": "nan",
+ "12275": "nan",
+ "12280": "nan",
+ "12285": "nan",
+ "12290": "nan",
+ "12295": "nan",
+ "12300": 1.23767,
+ "12305": "nan",
+ "12310": "nan",
+ "12315": "nan",
+ "12320": "nan",
+ "12325": "nan",
+ "12330": "nan",
+ "12335": "nan",
+ "12340": "nan",
+ "12345": "nan",
+ "12350": "nan",
+ "12355": "nan",
+ "12360": "nan",
+ "12365": "nan",
+ "12370": "nan",
+ "12375": "nan",
+ "12380": "nan",
+ "12385": "nan",
+ "12390": "nan",
+ "12395": "nan",
+ "12400": 1.50314,
+ "12405": "nan",
+ "12410": "nan",
+ "12415": "nan",
+ "12420": "nan",
+ "12425": "nan",
+ "12430": "nan",
+ "12435": "nan",
+ "12440": "nan",
+ "12445": "nan",
+ "12450": "nan",
+ "12455": "nan",
+ "12460": "nan",
+ "12465": "nan",
+ "12470": "nan",
+ "12475": "nan",
+ "12480": "nan",
+ "12485": "nan",
+ "12490": "nan",
+ "12495": "nan",
+ "12500": 1.44228,
+ "12505": "nan",
+ "12510": "nan",
+ "12515": "nan",
+ "12520": "nan",
+ "12525": "nan",
+ "12530": "nan",
+ "12535": "nan",
+ "12540": "nan",
+ "12545": "nan",
+ "12550": "nan",
+ "12555": "nan",
+ "12560": "nan",
+ "12565": "nan",
+ "12570": "nan",
+ "12575": "nan",
+ "12580": "nan",
+ "12585": "nan",
+ "12590": "nan",
+ "12595": "nan",
+ "12600": 1.14538,
+ "12605": "nan",
+ "12610": "nan",
+ "12615": "nan",
+ "12620": "nan",
+ "12625": "nan",
+ "12630": "nan",
+ "12635": "nan",
+ "12640": "nan",
+ "12645": "nan",
+ "12650": "nan",
+ "12655": "nan",
+ "12660": "nan",
+ "12665": "nan",
+ "12670": "nan",
+ "12675": "nan",
+ "12680": "nan",
+ "12685": "nan",
+ "12690": "nan",
+ "12695": "nan",
+ "12700": 1.47378,
+ "12705": "nan",
+ "12710": "nan",
+ "12715": "nan",
+ "12720": "nan",
+ "12725": "nan",
+ "12730": "nan",
+ "12735": "nan",
+ "12740": "nan",
+ "12745": "nan",
+ "12750": "nan",
+ "12755": "nan",
+ "12760": "nan",
+ "12765": "nan",
+ "12770": "nan",
+ "12775": "nan",
+ "12780": "nan",
+ "12785": "nan",
+ "12790": "nan",
+ "12795": "nan",
+ "12800": 1.14584,
+ "12805": "nan",
+ "12810": "nan",
+ "12815": "nan",
+ "12820": "nan",
+ "12825": "nan",
+ "12830": "nan",
+ "12835": "nan",
+ "12840": "nan",
+ "12845": "nan",
+ "12850": "nan",
+ "12855": "nan",
+ "12860": "nan",
+ "12865": "nan",
+ "12870": "nan",
+ "12875": "nan",
+ "12880": "nan",
+ "12885": "nan",
+ "12890": "nan",
+ "12895": "nan",
+ "12900": 1.16693,
+ "12905": "nan",
+ "12910": "nan",
+ "12915": "nan",
+ "12920": "nan",
+ "12925": "nan",
+ "12930": "nan",
+ "12935": "nan",
+ "12940": "nan",
+ "12945": "nan",
+ "12950": "nan",
+ "12955": "nan",
+ "12960": "nan",
+ "12965": "nan",
+ "12970": "nan",
+ "12975": "nan",
+ "12980": "nan",
+ "12985": "nan",
+ "12990": "nan",
+ "12995": "nan",
+ "13000": 1.47183
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm_gb200/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm_gb200/golden_values_dev_dgx_h100.json
new file mode 100644
index 00000000000..02bcf7fe698
--- /dev/null
+++ b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm_gb200/golden_values_dev_dgx_h100.json
@@ -0,0 +1 @@
+{"lm loss": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": 12.98419, "5": 12.93858, "10": 12.06404, "15": 11.97882, "20": 10.53588, "25": 10.11952, "30": 9.7286, "35": 9.44173, "40": 9.2373, "45": 9.03763, "50": 8.85277, "55": 8.64259, "60": 8.60098, "65": 8.50179, "70": 8.41326, "75": 8.31346, "80": 8.16921, "85": 8.09253, "90": 7.97894, "95": 7.91859, "100": 7.82704, "105": 7.71191, "110": 7.62418, "115": 7.52685, "120": 7.48107, "125": 7.48004, "130": 7.33364, "135": 7.26758, "140": 7.23146, "145": 7.04647, "150": 7.17621, "155": 7.00383, "160": 6.89968, "165": 6.91293, "170": 6.84228, "175": 6.85916, "180": 6.81429, "185": 6.7203, "190": 6.66124, "195": 6.59364, "200": 6.64046, "205": 6.64305, "210": 6.5179, "215": 6.51519, "220": 6.51027, "225": 6.46653, "230": 6.47574, "235": 6.42409, "240": 6.36976, "245": 6.3778, "250": 6.29868, "255": 6.43438, "260": 6.34377, "265": 6.28803, "270": 6.23364, "275": 6.26123, "280": 6.19076, "285": 6.19886, "290": 6.15022, "295": 6.12619, "300": 6.11141, "305": 6.01886, "310": 6.08556, "315": 6.07169, "320": 5.99243, "325": 5.93189, "330": 5.99792, "335": 6.0145, "340": 5.93453, "345": 5.92339, "350": 5.87179, "355": 5.84258, "360": 5.85866, "365": 5.81752, "370": 5.80407, "375": 5.80516, "380": 5.85848, "385": 5.78993, "390": 5.81141, "395": 5.68051, "400": 5.66121, "405": 5.68906, "410": 5.66202, "415": 5.70461, "420": 5.63851, "425": 5.66062, "430": 5.62802, "435": 5.56913, "440": 5.62147, "445": 5.52803, "450": 5.58428, "455": 5.5123, "460": 5.49325, "465": 5.56828, "470": 5.54845, "475": 5.49678, "480": 5.46247, "485": 5.49185, "490": 5.47566, "495": 5.47856, "500": 5.42533, "505": 5.38883, "510": 5.44319, "515": 5.42148, "520": 5.47608, "525": 5.31477, "530": 5.33216, "535": 5.36, "540": 5.33276, "545": 5.41314, "550": 5.37099, "555": 5.23374, "560": 5.32665, "565": 5.27809, "570": 5.25324, "575": 5.28184, "580": 5.23593, "585": 5.21762, "590": 5.22346, "595": 5.22561, "600": 5.26751, "605": 5.22896, "610": 5.2012, "615": 5.18737, "620": 5.19543, "625": 5.19655, "630": 5.14985, "635": 5.12452, "640": 5.09298, "645": 5.13279, "650": 5.14481, "655": 5.11963, "660": 5.0475, "665": 5.1142, "670": 5.04119, "675": 5.01723, "680": 5.05635, "685": 5.00678, "690": 5.01633, "695": 4.96228, "700": 4.97301, "705": 4.95571, "710": 4.97305, "715": 4.87719, "720": 4.85764, "725": 4.80769, "730": 4.84352, "735": 4.82916, "740": 4.8644, "745": 4.74895, "750": 4.75764, "755": 4.8023, "760": 4.78257, "765": 4.76428, "770": 4.69615, "775": 4.69212, "780": 4.684, "785": 4.7405, "790": 4.67498, "795": 4.64675, "800": 4.61184, "805": 4.61203, "810": 4.65393, "815": 4.60253, "820": 4.62914, "825": 4.58486, "830": 4.57946, "835": 4.56275, "840": 4.48603, "845": 4.50743, "850": 4.45704, "855": 4.51258, "860": 4.43583, "865": 4.52116, "870": 4.47717, "875": 4.38345, "880": 4.41849, "885": 4.38985, "890": 4.43389, "895": 4.42652, "900": 4.39808, "905": 4.32618, "910": 4.35391, "915": 4.34171, "920": 4.38377, "925": 4.38353, "930": 4.30961, "935": 4.29021, "940": 4.36235, "945": 4.31265, "950": 4.35051, "955": 4.28087, "960": 4.19218, "965": 4.27537, "970": 4.26236, "975": 4.24397, "980": 4.22146, "985": 4.17523, "990": 4.13237, "995": 4.18304, "1000": 4.2308, "1005": 4.18908, "1010": 4.17127, "1015": 4.13273, "1020": 4.15962, "1025": 4.22451, "1030": 4.1247, "1035": 4.10028, "1040": 4.13078, "1045": 4.11598, "1050": 4.15985, "1055": 4.0973, "1060": 4.1011, "1065": 4.06848, "1070": 4.05654, "1075": 4.07977, "1080": 4.07117, "1085": 4.06586, "1090": 4.02567, "1095": 4.09923, "1100": 4.06572, "1105": 4.071, "1110": 4.03046, "1115": 4.00398, "1120": 4.00356, "1125": 4.00318, "1130": 4.04694, "1135": 4.01101, "1140": 4.00143, "1145": 3.93633, "1150": 4.03948, "1155": 4.00619, "1160": 3.9851, "1165": 3.86882, "1170": 3.92684, "1175": 3.93038, "1180": 3.95878, "1185": 3.96184, "1190": 3.93034, "1195": 3.94125, "1200": 3.89932, "1205": 3.87266, "1210": 3.97831, "1215": 3.84899, "1220": 3.87263, "1225": 3.81413, "1230": 3.92846, "1235": 3.89312, "1240": 3.87537, "1245": 3.80079, "1250": 3.82885, "1255": 3.8493, "1260": 3.8874, "1265": 3.79336, "1270": 3.88167, "1275": 3.8455, "1280": 3.82143, "1285": 3.85735, "1290": 3.87159, "1295": 3.84912, "1300": 3.81711, "1305": 3.82495, "1310": 3.82551, "1315": 3.81192, "1320": 3.81405, "1325": 3.72789, "1330": 3.79529, "1335": 3.76889, "1340": 3.76123, "1345": 3.74704, "1350": 3.73516, "1355": 3.76572, "1360": 3.73919, "1365": 3.73441, "1370": 3.73284, "1375": 3.74202, "1380": 3.74909, "1385": 3.7541, "1390": 3.66502, "1395": 3.73883, "1400": 3.72777, "1405": 3.66183, "1410": 3.67095, "1415": 3.64909, "1420": 3.7012, "1425": 3.71121, "1430": 3.6707, "1435": 3.65988, "1440": 3.63587, "1445": 3.67497, "1450": 3.67892, "1455": 3.65003, "1460": 3.64827, "1465": 3.69085, "1470": 3.63258, "1475": 3.6929, "1480": 3.65433, "1485": 3.65353, "1490": 3.61514, "1495": 3.60996, "1500": 3.65344, "1505": 3.69268, "1510": 3.55544, "1515": 3.60731, "1520": 3.63573, "1525": 3.59446, "1530": 3.59436, "1535": 3.59439, "1540": 3.59806, "1545": 3.59944, "1550": 3.56476, "1555": 3.56897, "1560": 3.61095, "1565": 3.62486, "1570": 3.58908, "1575": 3.54769, "1580": 3.59493, "1585": 3.58838, "1590": 3.47011, "1595": 3.51425, "1600": 3.50212, "1605": 3.55557, "1610": 3.56864, "1615": 3.50712, "1620": 3.52052, "1625": 3.47381, "1630": 3.50275, "1635": 3.55197, "1640": 3.51791, "1645": 3.54635, "1650": 3.49031, "1655": 3.4704, "1660": 3.52131, "1665": 3.45881, "1670": 3.51236, "1675": 3.50045, "1680": 3.46681, "1685": 3.49078, "1690": 3.48353, "1695": 3.49776, "1700": 3.46185, "1705": 3.40868, "1710": 3.48728, "1715": 3.49675, "1720": 3.43132, "1725": 3.44122, "1730": 3.42961, "1735": 3.4644, "1740": 3.45584, "1745": 3.43534, "1750": 3.41748, "1755": 3.42246, "1760": 3.37607, "1765": 3.42468, "1770": 3.43261, "1775": 3.379, "1780": 3.4355, "1785": 3.42306, "1790": 3.37807, "1795": 3.40594, "1800": 3.34684, "1805": 3.39025, "1810": 3.34654, "1815": 3.4221, "1820": 3.41309, "1825": 3.38102, "1830": 3.32801, "1835": 3.42581, "1840": 3.38756, "1845": 3.42483, "1850": 3.39575, "1855": 3.37642, "1860": 3.35084, "1865": 3.38387, "1870": 3.29854, "1875": 3.45603, "1880": 3.34133, "1885": 3.36396, "1890": 3.34216, "1895": 3.39809, "1900": 3.36794, "1905": 3.30189, "1910": 3.32844, "1915": 3.3151, "1920": 3.36278, "1925": 3.33318, "1930": 3.31578, "1935": 3.31145, "1940": 3.36438, "1945": 3.26186, "1950": 3.40063, "1955": 3.30708, "1960": 3.31486, "1965": 3.28405, "1970": 3.29999, "1975": 3.33744, "1980": 3.34165, "1985": 3.23762, "1990": 3.32593, "1995": 3.28362, "2000": 3.27303, "2005": 3.26618, "2010": 3.28661, "2015": 3.22715, "2020": 3.27479, "2025": 3.27135, "2030": 3.27231, "2035": 3.29709, "2040": 3.26992, "2045": 3.23395, "2050": 3.27144, "2055": 3.32457, "2060": 3.28203, "2065": 3.24424, "2070": 3.30416, "2075": 3.24361, "2080": 3.22085, "2085": 3.30421, "2090": 3.15415, "2095": 3.29493, "2100": 3.23056, "2105": 3.19586, "2110": 3.20308, "2115": 3.24137, "2120": 3.18045, "2125": 3.21484, "2130": 3.22503, "2135": 3.27763, "2140": 3.1948, "2145": 3.21466, "2150": 3.20867, "2155": 3.2365, "2160": 3.20615, "2165": 3.25296, "2170": 3.23909, "2175": 3.17384, "2180": 3.22049, "2185": 3.25755, "2190": 3.24863, "2195": 3.15892, "2200": 3.2045, "2205": 3.17192, "2210": 3.12378, "2215": 3.19067, "2220": 3.19593, "2225": 3.18949, "2230": 3.13791, "2235": 3.19063, "2240": 3.21593, "2245": 3.1871, "2250": 3.21088, "2255": 3.1585, "2260": 3.14105, "2265": 3.23218, "2270": 3.21271, "2275": 3.1566, "2280": 3.17614, "2285": 3.16486, "2290": 3.17692, "2295": 3.20197, "2300": 3.13847, "2305": 3.15963, "2310": 3.12614, "2315": 3.06316, "2320": 3.11512, "2325": 3.17382, "2330": 3.12397, "2335": 3.12649, "2340": 3.17195, "2345": 3.12462, "2350": 3.129, "2355": 3.11726, "2360": 3.16031, "2365": 3.09266, "2370": 3.15197, "2375": 3.13019, "2380": 3.11082, "2385": 3.09359, "2390": 3.09567, "2395": 3.09807, "2400": 3.09966, "2405": 3.10436, "2410": 3.09007, "2415": 3.09491, "2420": 3.08537, "2425": 3.07877, "2430": 3.08079, "2435": 3.06761, "2440": 3.08574, "2445": 3.05747, "2450": 3.12167, "2455": 3.15832, "2460": 3.08596, "2465": 3.07656, "2470": 3.03663, "2475": 3.06421, "2480": 3.10252, "2485": 3.06485, "2490": 3.06573, "2495": 3.08845, "2500": 3.05671, "2505": 3.105, "2510": 3.12399, "2515": 3.0532, "2520": 3.07806, "2525": 3.02426, "2530": 3.04842, "2535": 3.09401, "2540": 3.07984, "2545": 3.05538, "2550": 3.00469, "2555": 3.07001, "2560": 3.04403, "2565": 3.12, "2570": 3.00976, "2575": 3.0601, "2580": 3.08548, "2585": 3.02156, "2590": 3.06606, "2595": 2.99925, "2600": 3.0841, "2605": 3.06879, "2610": 3.05401, "2615": 3.06935, "2620": 2.99191, "2625": 3.01384, "2630": 3.03627, "2635": 3.05041, "2640": 3.01088, "2645": 3.05612, "2650": 3.02233, "2655": 2.98756, "2660": 3.01604, "2665": 3.03817, "2670": 2.98547, "2675": 2.97442, "2680": 3.00378, "2685": 3.00171, "2690": 2.99912, "2695": 2.99265, "2700": 3.03079, "2705": 2.98376, "2710": 2.97975, "2715": 2.96047, "2720": 3.02663, "2725": 2.99565, "2730": 3.05827, "2735": 3.04913, "2740": 3.02027, "2745": 3.02502, "2750": 3.02065, "2755": 2.96792, "2760": 2.99447, "2765": 3.00785, "2770": 2.98958, "2775": 2.99278, "2780": 3.02294, "2785": 2.95383, "2790": 2.96474, "2795": 2.95595, "2800": 2.98985, "2805": 2.94051, "2810": 2.99046, "2815": 2.95976, "2820": 3.0756, "2825": 3.03639, "2830": 3.01855, "2835": 2.92175, "2840": 2.92574, "2845": 2.96102, "2850": 2.96997, "2855": 2.96207, "2860": 2.94977, "2865": 2.91535, "2870": 2.99202, "2875": 2.92084, "2880": 2.96303, "2885": 2.91779, "2890": 2.98572, "2895": 2.93253, "2900": 2.95289, "2905": 3.00499, "2910": 2.92994, "2915": 2.94325, "2920": 2.95516, "2925": 2.94427, "2930": 2.95621, "2935": 2.94005, "2940": 2.94552, "2945": 2.9075, "2950": 2.97913, "2955": 2.91177, "2960": 2.97029, "2965": 2.87292, "2970": 2.96107, "2975": 2.99603, "2980": 2.94257, "2985": 3.04155, "2990": 2.93897, "2995": 2.87114, "3000": 2.9422, "3005": 2.89655, "3010": 2.93538, "3015": 2.91032, "3020": 2.91995, "3025": 2.91883, "3030": 2.92686, "3035": 2.95815, "3040": 2.9312, "3045": 2.83504, "3050": 2.8988, "3055": 2.89613, "3060": 2.92461, "3065": 2.92459, "3070": 2.88159, "3075": 2.86953, "3080": 2.9243, "3085": 2.90325, "3090": 2.91754, "3095": 2.92816, "3100": 2.86703, "3105": 2.92918, "3110": 2.90236, "3115": 2.94681, "3120": 2.95312, "3125": 2.86217, "3130": 2.93048, "3135": 2.92489, "3140": 2.87699, "3145": 2.91715, "3150": 2.85701, "3155": 2.8442, "3160": 2.83887, "3165": 2.84564, "3170": 2.89213, "3175": 2.90452, "3180": 2.85788, "3185": 2.89571, "3190": 2.90627, "3195": 2.92723, "3200": 2.92789, "3205": 2.85912, "3210": 2.86987, "3215": 2.91563, "3220": 2.87374, "3225": 2.86935, "3230": 2.815, "3235": 2.87434, "3240": 2.8734, "3245": 2.90299, "3250": 2.86289, "3255": 2.8503, "3260": 2.85959, "3265": 2.86936, "3270": 2.85223, "3275": 2.86681, "3280": 2.79974, "3285": 2.81123, "3290": 2.86928, "3295": 2.92038, "3300": 2.87938, "3305": 2.86113, "3310": 2.85785, "3315": 2.80615, "3320": 2.8258, "3325": 2.82403, "3330": 2.82839, "3335": 2.8465, "3340": 2.82742, "3345": 2.84707, "3350": 2.84121, "3355": 2.85933, "3360": 2.79899, "3365": 2.85514, "3370": 2.84537, "3375": 2.84431, "3380": 2.84971, "3385": 2.87822, "3390": 2.8682, "3395": 2.81029, "3400": 2.78472, "3405": 2.82816, "3410": 2.84591, "3415": 2.86134, "3420": 2.82742, "3425": 2.81129, "3430": 2.82982, "3435": 2.8906, "3440": 2.81795, "3445": 2.86973, "3450": 2.81507, "3455": 2.7888, "3460": 2.8203, "3465": 2.84691, "3470": 2.83475, "3475": 2.7758, "3480": 2.84173, "3485": 2.82043, "3490": 2.8933, "3495": 2.84901, "3500": 2.84084, "3505": 2.82498, "3510": 2.81285, "3515": 2.83554, "3520": 2.77982, "3525": 2.80208, "3530": 2.84998, "3535": 2.78436, "3540": 2.83771, "3545": 2.81048, "3550": 2.79684, "3555": 2.8181, "3560": 2.82828, "3565": 2.82912, "3570": 2.80393, "3575": 2.80372, "3580": 2.82073, "3585": 2.83541, "3590": 2.8298, "3595": 2.77921, "3600": 2.74897, "3605": 2.79083, "3610": 2.8488, "3615": 2.75492, "3620": 2.80351, "3625": 2.88338, "3630": 2.77314, "3635": 2.78625, "3640": 2.78253, "3645": 2.76926, "3650": 2.80301, "3655": 2.81469, "3660": 2.76524, "3665": 2.7858, "3670": 2.77457, "3675": 2.77711, "3680": 2.80733, "3685": 2.80194, "3690": 2.8055, "3695": 2.81135, "3700": 2.78752, "3705": 2.78408, "3710": 2.75166, "3715": 2.80201, "3720": 2.79409, "3725": 2.78884, "3730": 2.84096, "3735": 2.80547, "3740": 2.74952, "3745": 2.78986, "3750": 2.8022, "3755": 2.79576, "3760": 2.75757, "3765": 2.75586, "3770": 2.75989, "3775": 2.76991, "3780": 2.76383, "3785": 2.7793, "3790": 2.74202, "3795": 2.79511, "3800": 2.80269, "3805": 2.75159, "3810": 2.80354, "3815": 2.76482, "3820": 2.78758, "3825": 2.73331, "3830": 2.74563, "3835": 2.81464, "3840": 2.72812, "3845": 2.71424, "3850": 2.77453, "3855": 2.71774, "3860": 2.80173, "3865": 2.75469, "3870": 2.77531, "3875": 2.75779, "3880": 2.78968, "3885": 2.78424, "3890": 2.74541, "3895": 2.79804, "3900": 2.76127, "3905": 2.72353, "3910": 2.74147, "3915": 2.75183, "3920": 2.79462, "3925": 2.77792, "3930": 2.70759, "3935": 2.73982, "3940": 2.75131, "3945": 2.74267, "3950": 2.725, "3955": 2.77958, "3960": 2.75991, "3965": 2.74216, "3970": 2.75653, "3975": 2.72552, "3980": 2.73817, "3985": 2.75045, "3990": 2.69347, "3995": 2.78059, "4000": 2.73558, "4005": 2.7658, "4010": 2.70885, "4015": 2.72538, "4020": 2.74956, "4025": 2.733, "4030": 2.65924, "4035": 2.69455, "4040": 2.74652, "4045": 2.74857, "4050": 2.78817, "4055": 2.7239, "4060": 2.71419, "4065": 2.6515, "4070": 2.80691, "4075": 2.75748, "4080": 2.71884, "4085": 2.74977, "4090": 2.67836, "4095": 2.69073, "4100": 2.7114, "4105": 2.73822, "4110": 2.72956, "4115": 2.70127, "4120": 2.73267, "4125": 2.70389, "4130": 2.69553, "4135": 2.6893, "4140": 2.68057, "4145": 2.77973, "4150": 2.70801, "4155": 2.73792, "4160": 2.76329, "4165": 2.72099, "4170": 2.67438, "4175": 2.71828, "4180": 2.72666, "4185": 2.72916, "4190": 2.73806, "4195": 2.70222, "4200": 2.71066, "4205": 2.73922, "4210": 2.67309, "4215": 2.66565, "4220": 2.65928, "4225": 2.70242, "4230": 2.71409, "4235": 2.7326, "4240": 2.70413, "4245": 2.69658, "4250": 2.71363, "4255": 2.64837, "4260": 2.7266, "4265": 2.73863, "4270": 2.72157, "4275": 2.68943, "4280": 2.70233, "4285": 2.73114, "4290": 2.68765, "4295": 2.69223, "4300": 2.69956, "4305": 2.70313, "4310": 2.73003, "4315": 2.71191, "4320": 2.69906, "4325": 2.70557, "4330": 2.7106, "4335": 2.69172, "4340": 2.6976, "4345": 2.72675, "4350": 2.67431, "4355": 2.69349, "4360": 2.71041, "4365": 2.78314, "4370": 2.73369, "4375": 2.74431, "4380": 2.71504, "4385": 2.69901, "4390": 2.70191, "4395": 2.75058, "4400": 2.66501, "4405": 2.66954, "4410": 2.68278, "4415": 2.70442, "4420": 2.7077, "4425": 2.72158, "4430": 2.69277, "4435": 2.68035, "4440": 2.69127, "4445": 2.67896, "4450": 2.65272, "4455": 2.69044, "4460": 2.70302, "4465": 2.70631, "4470": 2.6731, "4475": 2.68761, "4480": 2.65626, "4485": 2.69968, "4490": 2.65279, "4495": 2.70894, "4500": 2.70235, "4505": 2.69674, "4510": 2.64893, "4515": 2.70162, "4520": 2.66802, "4525": 2.66816, "4530": 2.6736, "4535": 2.67113, "4540": 2.70729, "4545": 2.65603, "4550": 2.70177, "4555": 2.68009, "4560": 2.65895, "4565": 2.63898, "4570": 2.6402, "4575": 2.66692, "4580": 2.68858, "4585": 2.68157, "4590": 2.61727, "4595": 2.66336, "4600": 2.67638, "4605": 2.68094, "4610": 2.66704, "4615": 2.66234, "4620": 2.65727, "4625": 2.68721, "4630": 2.6742, "4635": 2.64708, "4640": 2.69575, "4645": 2.64774, "4650": 2.7018, "4655": 2.70661, "4660": 2.67838, "4665": 2.68918, "4670": 2.67556, "4675": 2.68422, "4680": 2.66596, "4685": 2.65737, "4690": 2.70912, "4695": 2.65528, "4700": 2.67174, "4705": 2.65146, "4710": 2.68366, "4715": 2.64964, "4720": 2.72485, "4725": 2.62902, "4730": 2.65213, "4735": 2.68926, "4740": 2.64614, "4745": 2.65553, "4750": 2.65737, "4755": 2.65793, "4760": 2.66632, "4765": 2.64489, "4770": 2.62202, "4775": 2.65194, "4780": 2.65519, "4785": 2.68655, "4790": 2.65201, "4795": 2.67305, "4800": 2.62427, "4805": 2.64099, "4810": 2.65942, "4815": 2.65033, "4820": 2.6668, "4825": 2.65019, "4830": 2.6151, "4835": 2.64832, "4840": 2.65513, "4845": 2.6348, "4850": 2.62482, "4855": 2.60337, "4860": 2.65757, "4865": 2.62536, "4870": 2.63972, "4875": 2.61897, "4880": 2.62576, "4885": 2.62632, "4890": 2.67912, "4895": 2.65961, "4900": 2.618, "4905": 2.61823, "4910": 2.63845, "4915": 2.61463, "4920": 2.65397, "4925": 2.64838, "4930": 2.57129, "4935": 2.65193, "4940": 2.63034, "4945": 2.63777, "4950": 2.62825, "4955": 2.61794, "4960": 2.61856, "4965": 2.65951, "4970": 2.6008, "4975": 2.65676, "4980": 2.62049, "4985": 2.63225, "4990": 2.65645, "4995": 2.58184, "5000": 2.6621, "5005": 2.6658, "5010": 2.68112, "5015": 2.63396, "5020": 2.64091, "5025": 2.68726, "5030": 2.64362, "5035": 2.61873, "5040": 2.62248, "5045": 2.60699, "5050": 2.62641, "5055": 2.65014, "5060": 2.64375, "5065": 2.68893, "5070": 2.60617, "5075": 2.61421, "5080": 2.61231, "5085": 2.60499, "5090": 2.59441, "5095": 2.65086, "5100": 2.64984, "5105": 2.61053, "5110": 2.66408, "5115": 2.62171, "5120": 2.67055, "5125": 2.6309, "5130": 2.615, "5135": 2.61462, "5140": 2.57424, "5145": 2.62966, "5150": 2.63646, "5155": 2.61887, "5160": 2.66278, "5165": 2.58409, "5170": 2.59136, "5175": 2.62185, "5180": 2.60659, "5185": 2.62099, "5190": 2.6266, "5195": 2.67047, "5200": 2.5968, "5205": 2.60868, "5210": 2.60701, "5215": 2.64792, "5220": 2.58826, "5225": 2.55166, "5230": 2.6359, "5235": 2.61417, "5240": 2.62802, "5245": 2.64006, "5250": 2.61297, "5255": 2.62612, "5260": 2.5619, "5265": 2.59802, "5270": 2.58865, "5275": 2.61781, "5280": 2.61032, "5285": 2.60442, "5290": 2.63245, "5295": 2.62071, "5300": 2.57979, "5305": 2.59834, "5310": 2.60591, "5315": 2.5881, "5320": 2.61539, "5325": 2.64615, "5330": 2.6015, "5335": 2.58439, "5340": 2.56291, "5345": 2.65819, "5350": 2.62526, "5355": 2.57953, "5360": 2.59528, "5365": 2.62373, "5370": 2.61518, "5375": 2.63002, "5380": 2.58083, "5385": 2.56502, "5390": 2.58666, "5395": 2.61597, "5400": 2.60909, "5405": 2.54774, "5410": 2.61298, "5415": 2.59619, "5420": 2.61443, "5425": 2.62678, "5430": 2.62674, "5435": 2.57707, "5440": 2.58734, "5445": 2.633, "5450": 2.6473, "5455": 2.61252, "5460": 2.59272, "5465": 2.60502, "5470": 2.60189, "5475": 2.62728, "5480": 2.58753, "5485": 2.59002, "5490": 2.57733, "5495": 2.57075, "5500": 2.56937, "5505": 2.61715, "5510": 2.62664, "5515": 2.58137, "5520": 2.55697, "5525": 2.5859, "5530": 2.66433, "5535": 2.62339, "5540": 2.57109, "5545": 2.59633, "5550": 2.54936, "5555": 2.57342, "5560": 2.56447, "5565": 2.60758, "5570": 2.65168, "5575": 2.63138, "5580": 2.57564, "5585": 2.59822, "5590": 2.56185, "5595": 2.58521, "5600": 2.55512, "5605": 2.59879, "5610": 2.58291, "5615": 2.58198, "5620": 2.58123, "5625": 2.55147, "5630": 2.57081, "5635": 2.63484, "5640": 2.59425, "5645": 2.56995, "5650": 2.58004, "5655": 2.54766, "5660": 2.55881, "5665": 2.58604, "5670": 2.56686, "5675": 2.60728, "5680": 2.52861, "5685": 2.56813, "5690": 2.6039, "5695": 2.55782, "5700": 2.59695, "5705": 2.596, "5710": 2.57921, "5715": 2.58424, "5720": 2.53643, "5725": 2.6038, "5730": 2.57366, "5735": 2.61087, "5740": 2.59519, "5745": 2.56, "5750": 2.54216, "5755": 2.55997, "5760": 2.62481, "5765": 2.56328, "5770": 2.5429, "5775": 2.58373, "5780": 2.57701, "5785": 2.53911, "5790": 2.56461, "5795": 2.60179, "5800": 2.54494, "5805": 2.53531, "5810": 2.55658, "5815": 2.52456, "5820": 2.59694, "5825": 2.50599, "5830": 2.49558, "5835": 2.59597, "5840": 2.53979, "5845": 2.5528, "5850": 2.61315, "5855": 2.5102, "5860": 2.56169, "5865": 2.51778, "5870": 2.57574, "5875": 2.60723, "5880": 2.58596, "5885": 2.56757, "5890": 2.58608, "5895": 2.55562, "5900": 2.61651, "5905": 2.55716, "5910": 2.59828, "5915": 2.61008, "5920": 2.58733, "5925": 2.55324, "5930": 2.57568, "5935": 2.55168, "5940": 2.57131, "5945": 2.5204, "5950": 2.55562, "5955": 2.586, "5960": 2.56741, "5965": 2.62046, "5970": 2.55594, "5975": 2.58503, "5980": 2.55843, "5985": 2.56032, "5990": 2.55653, "5995": 2.55873, "6000": 2.55658, "6005": 2.51961, "6010": 2.5612, "6015": 2.52607, "6020": 2.53453, "6025": 2.55768, "6030": 2.6046, "6035": 2.54228, "6040": 2.54868, "6045": 2.49077, "6050": 2.5963, "6055": 2.5204, "6060": 2.54409, "6065": 2.52518, "6070": 2.52918, "6075": 2.5364, "6080": 2.53607, "6085": 2.59714, "6090": 2.57034, "6095": 2.53592, "6100": 2.5428, "6105": 2.52487, "6110": 2.55483, "6115": 2.58495, "6120": 2.55695, "6125": 2.53683, "6130": 2.47322, "6135": 2.5563, "6140": 2.55589, "6145": 2.55739, "6150": 2.52565, "6155": 2.50872, "6160": 2.54299, "6165": 2.57304, "6170": 2.54638, "6175": 2.60079, "6180": 2.51196, "6185": 2.55194, "6190": 2.49345, "6195": 2.57854, "6200": 2.55164, "6205": 2.5377, "6210": 2.52088, "6215": 2.51358, "6220": 2.56539, "6225": 2.51406, "6230": 2.51072, "6235": 2.56268, "6240": 2.55115, "6245": 2.52327, "6250": 2.53069, "6255": 2.57365, "6260": 2.52537, "6265": 2.57441, "6270": 2.52397, "6275": 2.56565, "6280": 2.52297, "6285": 2.5207, "6290": 2.51982, "6295": 2.50722, "6300": 2.55559, "6305": 2.52486, "6310": 2.51259, "6315": 2.53731, "6320": 2.4894, "6325": 2.59818, "6330": 2.555, "6335": 2.51085, "6340": 2.51313, "6345": 2.55702, "6350": 2.556, "6355": 2.52448, "6360": 2.52293, "6365": 2.48409, "6370": 2.53563, "6375": 2.49779, "6380": 2.56282, "6385": 2.58189, "6390": 2.50441, "6395": 2.55121, "6400": 2.5086, "6405": 2.5278, "6410": 2.51466, "6415": 2.52482, "6420": 2.54258, "6425": 2.53509, "6430": 2.57978, "6435": 2.54444, "6440": 2.53907, "6445": 2.53125, "6450": 2.53474, "6455": 2.52399, "6460": 2.51849, "6465": 2.56225, "6470": 2.52104, "6475": 2.52654, "6480": 2.48826, "6485": 2.52861, "6490": 2.50978, "6495": 2.49978, "6500": 2.52402, "6505": 2.49432, "6510": 2.54199, "6515": 2.5101, "6520": 2.51003, "6525": 2.49503, "6530": 2.54392, "6535": 2.53282, "6540": 2.53291, "6545": 2.56194, "6550": 2.50127, "6555": 2.55627, "6560": 2.51016, "6565": 2.52281, "6570": 2.58445, "6575": 2.52324, "6580": 2.49815, "6585": 2.50823, "6590": 2.5097, "6595": 2.49807, "6600": 2.49539, "6605": 2.54253, "6610": 2.4797, "6615": 2.56766, "6620": 2.53402, "6625": 2.51202, "6630": 2.51431, "6635": 2.47464, "6640": 2.54106, "6645": 2.59681, "6650": 2.51024, "6655": 2.4983, "6660": 2.57419, "6665": 2.52156, "6670": 2.5674, "6675": 2.46861, "6680": 2.54697, "6685": 2.53564, "6690": 2.51427, "6695": 2.48573, "6700": 2.52463, "6705": 2.52218, "6710": 2.49347, "6715": 2.51687, "6720": 2.50996, "6725": 2.52089, "6730": 2.52013, "6735": 2.4825, "6740": 2.51535, "6745": 2.49672, "6750": 2.55754, "6755": 2.47484, "6760": 2.54212, "6765": 2.48878, "6770": 2.51847, "6775": 2.50828, "6780": 2.53878, "6785": 2.47177, "6790": 2.54553, "6795": 2.49868, "6800": 2.52671, "6805": 2.51099, "6810": 2.50296, "6815": 2.52064, "6820": 2.48696, "6825": 2.5071, "6830": 2.54063, "6835": 2.50678, "6840": 2.50885, "6845": 2.52492, "6850": 2.47583, "6855": 2.512, "6860": 2.50239, "6865": 2.49001, "6870": 2.55392, "6875": 2.47561, "6880": 2.55072, "6885": 2.47892, "6890": 2.54905, "6895": 2.50384, "6900": 2.49072, "6905": 2.51205, "6910": 2.5215, "6915": 2.51823, "6920": 2.5328, "6925": 2.54741, "6930": 2.49289, "6935": 2.521, "6940": 2.50604, "6945": 2.46237, "6950": 2.48628, "6955": 2.5288, "6960": 2.51952, "6965": 2.49196, "6970": 2.47065, "6975": 2.52409, "6980": 2.45258, "6985": 2.51631, "6990": 2.52932, "6995": 2.46179, "7000": 2.49172, "7005": 2.47011, "7010": 2.47632, "7015": 2.51983, "7020": 2.46705, "7025": 2.45424, "7030": 2.48487, "7035": 2.47988, "7040": 2.50783, "7045": 2.52359, "7050": 2.52831, "7055": 2.44161, "7060": 2.47409, "7065": 2.48138, "7070": 2.48981, "7075": 2.49452, "7080": 2.53479, "7085": 2.48717, "7090": 2.47618, "7095": 2.4999, "7100": 2.51585, "7105": 2.4884, "7110": 2.487, "7115": 2.50558, "7120": 2.47286, "7125": 2.46376, "7130": 2.48693, "7135": 2.51456, "7140": 2.50032, "7145": 2.49769, "7150": 2.51016, "7155": 2.50401, "7160": 2.47274, "7165": 2.45638, "7170": 2.50459, "7175": 2.50355, "7180": 2.50497, "7185": 2.48172, "7190": 2.46296, "7195": 2.46639, "7200": 2.50998, "7205": 2.49029, "7210": 2.44246, "7215": 2.47885, "7220": 2.4456, "7225": 2.51269, "7230": 2.50805, "7235": 2.48249, "7240": 2.47867, "7245": 2.50035, "7250": 2.50922, "7255": 2.49324, "7260": 2.46058, "7265": 2.45308, "7270": 2.47086, "7275": 2.49781, "7280": 2.49343, "7285": 2.42363, "7290": 2.47944, "7295": 2.48626, "7300": 2.41751, "7305": 2.44554, "7310": 2.44899, "7315": 2.48986, "7320": 2.48389, "7325": 2.45917, "7330": 2.4893, "7335": 2.47688, "7340": 2.46486, "7345": 2.49515, "7350": 2.5106, "7355": 2.49669, "7360": 2.48037, "7365": 2.46906, "7370": 2.47138, "7375": 2.4508, "7380": 2.49622, "7385": 2.48448, "7390": 2.47337, "7395": 2.47339, "7400": 2.48169, "7405": 2.43994, "7410": 2.48078, "7415": 2.47113, "7420": 2.49398, "7425": 2.45774, "7430": 2.52358, "7435": 2.49185, "7440": 2.52151, "7445": 2.5101, "7450": 2.4751, "7455": 2.45401, "7460": 2.46474, "7465": 2.47685, "7470": 2.44899, "7475": 2.45681, "7480": 2.51145, "7485": 2.45042, "7490": 2.47478, "7495": 2.48246, "7500": 2.49584, "7505": 2.44104, "7510": 2.43501, "7515": 2.41997, "7520": 2.49389, "7525": 2.49884, "7530": 2.47668, "7535": 2.4601, "7540": 2.47288, "7545": 2.47471, "7550": 2.49181, "7555": 2.45487, "7560": 2.42922, "7565": 2.51106, "7570": 2.4857, "7575": 2.439, "7580": 2.45825, "7585": 2.48256, "7590": 2.48193, "7595": 2.46508, "7600": 2.46362, "7605": 2.44863, "7610": 2.44948, "7615": 2.42526, "7620": 2.54441, "7625": 2.47879, "7630": 2.42526, "7635": 2.42739, "7640": 2.45364, "7645": 2.47151, "7650": 2.46303, "7655": 2.48304, "7660": 2.4532, "7665": 2.4342, "7670": 2.4426, "7675": 2.45588, "7680": 2.48517, "7685": 2.43208, "7690": 2.48, "7695": 2.45485, "7700": 2.48159, "7705": 2.49878, "7710": 2.49483, "7715": 2.44384, "7720": 2.4696, "7725": 2.47981, "7730": 2.45864, "7735": 2.47057, "7740": 2.43882, "7745": 2.45157, "7750": 2.43921, "7755": 2.46722, "7760": 2.45122, "7765": 2.45511, "7770": 2.47144, "7775": 2.45332, "7780": 2.41653, "7785": 2.44516, "7790": 2.48285, "7795": 2.44125, "7800": 2.46355, "7805": 2.48202, "7810": 2.50258, "7815": 2.48733, "7820": 2.44788, "7825": 2.51471, "7830": 2.45477, "7835": 2.4697, "7840": 2.47907, "7845": 2.46064, "7850": 2.41717, "7855": 2.47244, "7860": 2.49887, "7865": 2.42434, "7870": 2.46693, "7875": 2.44544, "7880": 2.45287, "7885": 2.46023, "7890": 2.47026, "7895": 2.44872, "7900": 2.4404, "7905": 2.43773, "7910": 2.42565, "7915": 2.48107, "7920": 2.47699, "7925": 2.4218, "7930": 2.47199, "7935": 2.44975, "7940": 2.42126, "7945": 2.46977, "7950": 2.44424, "7955": 2.4204, "7960": 2.49038, "7965": 2.5188, "7970": 2.52207, "7975": 2.44798, "7980": 2.44076, "7985": 2.46872, "7990": 2.43169, "7995": 2.46954, "8000": 2.43641, "8005": 2.41891, "8010": 2.45749, "8015": 2.46841, "8020": 2.48116, "8025": 2.47363, "8030": 2.45173, "8035": 2.47071, "8040": 2.41983, "8045": 2.45333, "8050": 2.44721, "8055": 2.42302, "8060": 2.44253, "8065": 2.46158, "8070": 2.4567, "8075": 2.46077, "8080": 2.44618, "8085": 2.44085, "8090": 2.42787, "8095": 2.42397, "8100": 2.43904, "8105": 2.49479, "8110": 2.43878, "8115": 2.58899, "8120": 2.49362, "8125": 2.47876, "8130": 2.45879, "8135": 2.4574, "8140": 2.44166, "8145": 2.42774, "8150": 2.42089, "8155": 2.48312, "8160": 2.45131, "8165": 2.43947, "8170": 2.43326, "8175": 2.42092, "8180": 2.4946, "8185": 2.42477, "8190": 2.46908, "8195": 2.45732, "8200": 2.44651, "8205": 2.44406, "8210": 2.43096, "8215": 2.44122, "8220": 2.43556, "8225": 2.41067, "8230": 2.44055, "8235": 2.46438, "8240": 2.42694, "8245": 2.44767, "8250": 2.44524, "8255": 2.43772, "8260": 2.43153, "8265": 2.42903, "8270": 2.4363, "8275": 2.44197, "8280": 2.39831, "8285": 2.4405, "8290": 2.48021, "8295": 2.44762, "8300": 2.45931, "8305": 2.40847, "8310": 2.43461, "8315": 2.45616, "8320": 2.40422, "8325": 2.39725, "8330": 2.43986, "8335": 2.44684, "8340": 2.49212, "8345": 2.44942, "8350": 2.45049, "8355": 2.40704, "8360": 2.40131, "8365": 2.45443, "8370": 2.45427, "8375": 2.42518, "8380": 2.41939, "8385": 2.42541, "8390": 2.4387, "8395": 2.44193, "8400": 2.44114, "8405": 2.49132, "8410": 2.4383, "8415": 2.43519, "8420": 2.41861, "8425": 2.44324, "8430": 2.46253, "8435": 2.40559, "8440": 2.45227, "8445": 2.45999, "8450": 2.40867, "8455": 2.46028, "8460": 2.45495, "8465": 2.43629, "8470": 2.40854, "8475": 2.47887, "8480": 2.40222, "8485": 2.41392, "8490": 2.46612, "8495": 2.43613, "8500": 2.44492, "8505": 2.40329, "8510": 2.40218, "8515": 2.42871, "8520": 2.42574, "8525": 2.49152, "8530": 2.3746, "8535": 2.40109, "8540": 2.48679, "8545": 2.3811, "8550": 2.43875, "8555": 2.4514, "8560": 2.47019, "8565": 2.42055, "8570": 2.43185, "8575": 2.44959, "8580": 2.44124, "8585": 2.42059, "8590": 2.4038, "8595": 2.42895, "8600": 2.41116, "8605": 2.49131, "8610": 2.42052, "8615": 2.38808, "8620": 2.45039, "8625": 2.42523, "8630": 2.45471, "8635": 2.4509, "8640": 2.43534, "8645": 2.47406, "8650": 2.42305, "8655": 2.45293, "8660": 2.45576, "8665": 2.38622, "8670": 2.41139, "8675": 2.42943, "8680": 2.44841, "8685": 2.43079, "8690": 2.41017, "8695": 2.44311, "8700": 2.43428, "8705": 2.42016, "8710": 2.42854, "8715": 2.44862, "8720": 2.47696, "8725": 2.41012, "8730": 2.39278, "8735": 2.43505, "8740": 2.43198, "8745": 2.39801, "8750": 2.43609, "8755": 2.42381, "8760": 2.40031, "8765": 2.43541, "8770": 2.40569, "8775": 2.43812, "8780": 2.42153, "8785": 2.47144, "8790": 2.42041, "8795": 2.41876, "8800": 2.41592, "8805": 2.40548, "8810": 2.41139, "8815": 2.47509, "8820": 2.45362, "8825": 2.4241, "8830": 2.38744, "8835": 2.42258, "8840": 2.39347, "8845": 2.42679, "8850": 2.43485, "8855": 2.4044, "8860": 2.42715, "8865": 2.42631, "8870": 2.43391, "8875": 2.44152, "8880": 2.41099, "8885": 2.39514, "8890": 2.44614, "8895": 2.42902, "8900": 2.41354, "8905": 2.40085, "8910": 2.4019, "8915": 2.4163, "8920": 2.43454, "8925": 2.46713, "8930": 2.41511, "8935": 2.40784, "8940": 2.38869, "8945": 2.39353, "8950": 2.41789, "8955": 2.39534, "8960": 2.43426, "8965": 2.41798, "8970": 2.40536, "8975": 2.47767, "8980": 2.44109, "8985": 2.37482, "8990": 2.41061, "8995": 2.416, "9000": 2.45568, "9005": 2.41279, "9010": 2.37662, "9015": 2.41141, "9020": 2.40089, "9025": 2.3701, "9030": 2.40026, "9035": 2.4243, "9040": 2.42079, "9045": 2.41805, "9050": 2.39505, "9055": 2.41785, "9060": 2.41922, "9065": 2.40527, "9070": 2.44454, "9075": 2.39395, "9080": 2.43398, "9085": 2.4136, "9090": 2.41293, "9095": 2.39793, "9100": 2.40135, "9105": 2.35782, "9110": 2.46451, "9115": 2.41499, "9120": 2.40368, "9125": 2.45804, "9130": 2.39387, "9135": 2.44878, "9140": 2.43562, "9145": 2.42684, "9150": 2.42505, "9155": 2.3752, "9160": 2.41724, "9165": 2.42569, "9170": 2.37359, "9175": 2.41857, "9180": 2.37803, "9185": 2.43942, "9190": 2.41281, "9195": 2.40662, "9200": 2.39186, "9205": 2.44999, "9210": 2.36248, "9215": 2.46363, "9220": 2.44779, "9225": 2.3828, "9230": 2.44575, "9235": 2.39772, "9240": 2.40182, "9245": 2.43796, "9250": 2.43806, "9255": 2.4326, "9260": 2.38813, "9265": 2.43977, "9270": 2.43657, "9275": 2.39535, "9280": 2.39074, "9285": 2.42225, "9290": 2.40437, "9295": 2.38603, "9300": 2.42495, "9305": 2.40579, "9310": 2.41555, "9315": 2.41153, "9320": 2.44493, "9325": 2.37049, "9330": 2.40434, "9335": 2.36191, "9340": 2.40835, "9345": 2.41458, "9350": 2.44039, "9355": 2.47763, "9360": 2.43745, "9365": 2.38821, "9370": 2.43648, "9375": 2.43331, "9380": 2.35346, "9385": 2.39958, "9390": 2.38109, "9395": 2.38731, "9400": 2.44471, "9405": 2.41259, "9410": 2.39756, "9415": 2.43759, "9420": 2.4441, "9425": 2.43656, "9430": 2.45071, "9435": 2.41453, "9440": 2.47761, "9445": 2.37622, "9450": 2.39383, "9455": 2.40249, "9460": 2.38597, "9465": 2.3775, "9470": 2.38205, "9475": 2.36454, "9480": 2.43551, "9485": 2.38642, "9490": 2.4204, "9495": 2.38165, "9500": 2.36325, "9505": 2.4296, "9510": 2.39916, "9515": 2.43096, "9520": 2.41792, "9525": 2.38898, "9530": 2.45385, "9535": 2.40151, "9540": 2.41839, "9545": 2.37813, "9550": 2.42143, "9555": 2.39054, "9560": 2.42191, "9565": 2.40523, "9570": 2.37157, "9575": 2.41109, "9580": 2.39564, "9585": 2.42353, "9590": 2.42924, "9595": 2.44777, "9600": 2.39117, "9605": 2.38431, "9610": 2.42142, "9615": 2.41558, "9620": 2.41413, "9625": 2.44723, "9630": 2.39712, "9635": 2.40396, "9640": 2.44817, "9645": 2.4109, "9650": 2.39894, "9655": 2.37366, "9660": 2.42329, "9665": 2.39029, "9670": 2.38274, "9675": 2.35662, "9680": 2.39869, "9685": 2.40199, "9690": 2.46804, "9695": 2.38133, "9700": 2.37698, "9705": 2.38453, "9710": 2.36554, "9715": 2.38868, "9720": 2.43552, "9725": 2.4413, "9730": 2.42919, "9735": 2.38684, "9740": 2.38077, "9745": 2.42676, "9750": 2.3991, "9755": 2.40788, "9760": 2.41084, "9765": 2.37036, "9770": 2.43675, "9775": 2.40145, "9780": 2.36196, "9785": 2.40085, "9790": 2.40714, "9795": 2.3593, "9800": 2.39629, "9805": 2.40561, "9810": 2.41066, "9815": 2.37884, "9820": 2.37671, "9825": 2.40364, "9830": 2.42194, "9835": 2.3861, "9840": 2.41457, "9845": 2.36502, "9850": 2.39824, "9855": 2.39496, "9860": 2.3972, "9865": 2.38197, "9870": 2.39342, "9875": 2.38398, "9880": 2.45319, "9885": 2.39313, "9890": 2.35399, "9895": 2.32116, "9900": 2.3962, "9905": 2.42494, "9910": 2.35642, "9915": 2.36473, "9920": 2.41154, "9925": 2.39863, "9930": 2.38182, "9935": 2.35063, "9940": 2.38377, "9945": 2.37842, "9950": 2.40342, "9955": 2.44928, "9960": 2.43108, "9965": 2.35851, "9970": 2.41017, "9975": 2.38564, "9980": 2.33084, "9985": 2.40772, "9990": 2.39761, "9995": 2.39543, "10000": 2.36621, "10005": 2.37213, "10010": 2.38256, "10015": 2.44495, "10020": 2.36326, "10025": 2.38851, "10030": 2.38817, "10035": 2.40993, "10040": 2.40515, "10045": 2.3831, "10050": 2.34965, "10055": 2.36805, "10060": 2.42146, "10065": 2.37528, "10070": 2.42235, "10075": 2.37088, "10080": 2.36211, "10085": 2.36918, "10090": 2.34573, "10095": 2.40221, "10100": 2.31408, "10105": 2.38253, "10110": 2.40897, "10115": 2.38736, "10120": 2.35801, "10125": 2.37033, "10130": 2.36037, "10135": 2.38382, "10140": 2.4139, "10145": 2.40714, "10150": 2.37532, "10155": 2.39536, "10160": 2.36205, "10165": 2.38369, "10170": 2.4236, "10175": 2.32447, "10180": 2.39651, "10185": 2.3824, "10190": 2.44396, "10195": 2.40416, "10200": 2.38955, "10205": 2.38797, "10210": 2.36805, "10215": 2.34261, "10220": 2.41843, "10225": 2.43079, "10230": 2.35627, "10235": 2.38764, "10240": 2.37226, "10245": 2.39117, "10250": 2.38838, "10255": 2.41316, "10260": 2.33469, "10265": 2.34846, "10270": 2.34979, "10275": 2.3717, "10280": 2.4513, "10285": 2.35906, "10290": 2.3861, "10295": 2.375, "10300": 2.36936, "10305": 2.41578, "10310": 2.38877, "10315": 2.36095, "10320": 2.36607, "10325": 2.36094, "10330": 2.41247, "10335": 2.36135, "10340": 2.41934, "10345": 2.36966, "10350": 2.35686, "10355": 2.39609, "10360": 2.37338, "10365": 2.36225, "10370": 2.34061, "10375": 2.3585, "10380": 2.41953, "10385": 2.40576, "10390": 2.38058, "10395": 2.35968, "10400": 2.37919, "10405": 2.34877, "10410": 2.3389, "10415": 2.41664, "10420": 2.37924, "10425": 2.32522, "10430": 2.35941, "10435": 2.37129, "10440": 2.3711, "10445": 2.35949, "10450": 2.36154, "10455": 2.38113, "10460": 2.38064, "10465": 2.30273, "10470": 2.3577, "10475": 2.37958, "10480": 2.36276, "10485": 2.36137, "10490": 2.41283, "10495": 2.36502, "10500": 2.36277, "10505": 2.37018, "10510": 2.38172, "10515": 2.37393, "10520": 2.40259, "10525": 2.39024, "10530": 2.39211, "10535": 2.35551, "10540": 2.40461, "10545": 2.35856, "10550": 2.37752, "10555": 2.35793, "10560": 2.34025, "10565": 2.37346, "10570": 2.37536, "10575": 2.3535, "10580": 2.37788, "10585": 2.36682, "10590": 2.37817, "10595": 2.37713, "10600": 2.33146, "10605": 2.3724, "10610": 2.36498, "10615": 2.36379, "10620": 2.34659, "10625": 2.41843, "10630": 2.36855, "10635": 2.32266, "10640": 2.36413, "10645": 2.42158, "10650": 2.36174, "10655": 2.30869, "10660": 2.34689, "10665": 2.39981, "10670": 2.31617, "10675": 2.41612, "10680": 2.35445, "10685": 2.28871, "10690": 2.38456, "10695": 2.33038, "10700": 2.38407, "10705": 2.38432, "10710": 2.34313, "10715": 2.3828, "10720": 2.32518, "10725": 2.35278, "10730": 2.34872, "10735": 2.35338, "10740": 2.31849, "10745": 2.33808, "10750": 2.33362, "10755": 2.4041, "10760": 2.36431, "10765": 2.33591, "10770": 2.36802, "10775": 2.38746, "10780": 2.36985, "10785": 2.39167, "10790": 2.34599, "10795": 2.38556, "10800": 2.32491, "10805": 2.39755, "10810": 2.37536, "10815": 2.35431, "10820": 2.34323, "10825": 2.37192, "10830": 2.33781, "10835": 2.3477, "10840": 2.32993, "10845": 2.38645, "10850": 2.33282, "10855": 2.36654, "10860": 2.33304, "10865": 2.32192, "10870": 2.32311, "10875": 2.30406, "10880": 2.39356, "10885": 2.40455, "10890": 2.36115, "10895": 2.37301, "10900": 2.33176, "10905": 2.31266, "10910": 2.40728, "10915": 2.37119, "10920": 2.37413, "10925": 2.36306, "10930": 2.31881, "10935": 2.36035, "10940": 2.35501, "10945": 2.34689, "10950": 2.36286, "10955": 2.3644, "10960": 2.30987, "10965": 2.3635, "10970": 2.35624, "10975": 2.40775, "10980": 2.37303, "10985": 2.3427, "10990": 2.39729, "10995": 2.36387, "11000": 2.33714, "11005": 2.36117, "11010": 2.34243, "11015": 2.32557, "11020": 2.3346, "11025": 2.36577, "11030": 2.34044, "11035": 2.31307, "11040": 2.31887, "11045": 2.31738, "11050": 2.31805, "11055": 2.28859, "11060": 2.33998, "11065": 2.31013, "11070": 2.39402, "11075": 2.32015, "11080": 2.35427, "11085": 2.33669, "11090": 2.34632, "11095": 2.37084, "11100": 2.32912, "11105": 2.31663, "11110": 2.36288, "11115": 2.37225, "11120": 2.38139, "11125": 2.31341, "11130": 2.34997, "11135": 2.3336, "11140": 2.37217, "11145": 2.35107, "11150": 2.39612, "11155": 2.34114, "11160": 2.3659, "11165": 2.36388, "11170": 2.34098, "11175": 2.33474, "11180": 2.37348, "11185": 2.31203, "11190": 2.27804, "11195": 2.32819, "11200": 2.34726, "11205": 2.36258, "11210": 2.33385, "11215": 2.31927, "11220": 2.34329, "11225": 2.37141, "11230": 2.36569, "11235": 2.32069, "11240": 2.34092, "11245": 2.35748, "11250": 2.3324, "11255": 2.33515, "11260": 2.35577, "11265": 2.38918, "11270": 2.28782, "11275": 2.31519, "11280": 2.36893, "11285": 2.29387, "11290": 2.34639, "11295": 2.3655, "11300": 2.38111, "11305": 2.33495, "11310": 2.32963, "11315": 2.29825, "11320": 2.30482, "11325": 2.31462, "11330": 2.35421, "11335": 2.33831, "11340": 2.30841, "11345": 2.31278, "11350": 2.29588, "11355": 2.3219, "11360": 2.35153, "11365": 2.29378, "11370": 2.35263, "11375": 2.32804, "11380": 2.34006, "11385": 2.34763, "11390": 2.33477, "11395": 2.28732, "11400": 2.30981, "11405": 2.35647, "11410": 2.35502, "11415": 2.38458, "11420": 2.35172, "11425": 2.30761, "11430": 2.36718, "11435": 2.36201, "11440": 2.34796, "11445": 2.36318, "11450": 2.32182, "11455": 2.30476, "11460": 2.35092, "11465": 2.34386, "11470": 2.37434, "11475": 2.31342, "11480": 2.32527, "11485": 2.30987, "11490": 2.34568, "11495": 2.406, "11500": 2.33937, "11505": 2.35014, "11510": 2.36223, "11515": 2.32176, "11520": 2.30507, "11525": 2.36152, "11530": 2.31469, "11535": 2.32196, "11540": 2.34627, "11545": 2.34321, "11550": 2.36438, "11555": 2.32533, "11560": 2.34981, "11565": 2.34125, "11570": 2.34916, "11575": 2.29628, "11580": 2.32931, "11585": 2.35173, "11590": 2.36158, "11595": 2.33454, "11600": 2.35704, "11605": 2.3235, "11610": 2.36089, "11615": 2.35899, "11620": 2.29569, "11625": 2.2757, "11630": 2.32782, "11635": 2.34204, "11640": 2.30488, "11645": 2.30751, "11650": 2.32628, "11655": 2.35114, "11660": 2.33566, "11665": 2.32994, "11670": 2.30002, "11675": 2.29666, "11680": 2.32542, "11685": 2.33637, "11690": 2.34433, "11695": 2.31688, "11700": 2.32535, "11705": 2.3009, "11710": 2.34479, "11715": 2.31575, "11720": 2.29844, "11725": 2.33988, "11730": 2.30403, "11735": 2.32822, "11740": 2.27122, "11745": 2.31714, "11750": 2.32793, "11755": 2.35133, "11760": 2.31357, "11765": 2.3378, "11770": 2.27597, "11775": 2.32591, "11780": 2.25511, "11785": 2.2973, "11790": 2.31403, "11795": 2.32024, "11800": 2.3345, "11805": 2.30403, "11810": 2.30398, "11815": 2.33078, "11820": 2.32015, "11825": 2.36083, "11830": 2.31663, "11835": 2.33741, "11840": 2.34081, "11845": 2.31727, "11850": 2.30496, "11855": 2.31403, "11860": 2.34333, "11865": 2.35836, "11870": 2.37861, "11875": 2.28155, "11880": 2.29163, "11885": 2.33553, "11890": 2.29241, "11895": 2.29059, "11900": 2.33401, "11905": 2.31769, "11910": 2.27783, "11915": 2.31082, "11920": 2.33519, "11925": 2.30272, "11930": 2.30681, "11935": 2.31569, "11940": 2.3175, "11945": 2.34208, "11950": 2.29773, "11955": 2.31327, "11960": 2.33576, "11965": 2.29584, "11970": 2.28204, "11975": 2.33575, "11980": 2.30612, "11985": 2.2776, "11990": 2.30416, "11995": 2.33013, "12000": 2.32323, "12005": 2.32565, "12010": 2.2884, "12015": 2.30861, "12020": 2.32922, "12025": 2.33525, "12030": 2.31246, "12035": 2.33617, "12040": 2.3154, "12045": 2.3126, "12050": 2.30835, "12055": 2.33352, "12060": 2.29764, "12065": 2.32975, "12070": 2.30319, "12075": 2.2775, "12080": 2.35063, "12085": 2.33812, "12090": 2.33359, "12095": 2.28176, "12100": 2.31543, "12105": 2.30903, "12110": 2.33029, "12115": 2.3036, "12120": 2.30606, "12125": 2.29484, "12130": 2.30409, "12135": 2.32842, "12140": 2.29591, "12145": 2.25622, "12150": 2.26125, "12155": 2.34249, "12160": 2.35771, "12165": 2.31914, "12170": 2.3336, "12175": 2.3412, "12180": 2.33054, "12185": 2.34135, "12190": 2.33375, "12195": 2.29767, "12200": 2.30036, "12205": 2.32225, "12210": 2.35697, "12215": 2.30437, "12220": 2.2987, "12225": 2.24241, "12230": 2.33348, "12235": 2.33945, "12240": 2.32345, "12245": 2.28764, "12250": 2.27397, "12255": 2.33706, "12260": 2.31368, "12265": 2.34287, "12270": 2.31292, "12275": 2.31361, "12280": 2.31869, "12285": 2.28631, "12290": 2.31074, "12295": 2.26654, "12300": 2.32931, "12305": 2.26821, "12310": 2.28768, "12315": 2.3543, "12320": 2.2963, "12325": 2.32045, "12330": 2.30113, "12335": 2.3194, "12340": 2.34117, "12345": 2.36885, "12350": 2.34318, "12355": 2.30683, "12360": 2.31344, "12365": 2.32933, "12370": 2.29273, "12375": 2.29957, "12380": 2.29184, "12385": 2.29061, "12390": 2.25018, "12395": 2.30421, "12400": 2.29905, "12405": 2.31088, "12410": 2.30419, "12415": 2.28306, "12420": 2.31729, "12425": 2.30099, "12430": 2.31571, "12435": 2.30048, "12440": 2.33123, "12445": 2.3202, "12450": 2.30745, "12455": 2.24018, "12460": 2.33488, "12465": 2.36363, "12470": 2.27626, "12475": 2.27276, "12480": 2.29139, "12485": 2.30632, "12490": 2.33128, "12495": 2.26961, "12500": 2.32122, "12505": 2.3351, "12510": 2.35582, "12515": 2.27062, "12520": 2.31971, "12525": 2.28653, "12530": 2.32054, "12535": 2.27138, "12540": 2.28491, "12545": 2.29049, "12550": 2.31572, "12555": 2.32333, "12560": 2.30023, "12565": 2.3353, "12570": 2.27829, "12575": 2.29941, "12580": 2.31153, "12585": 2.29201, "12590": 2.33455, "12595": 2.3227, "12600": 2.28167, "12605": 2.31996, "12610": 2.3631, "12615": 2.30567, "12620": 2.33322, "12625": 2.32935, "12630": 2.29885, "12635": 2.33561, "12640": 2.29568, "12645": 2.27902, "12650": 2.32556, "12655": 2.2647, "12660": 2.34199, "12665": 2.31843, "12670": 2.3097, "12675": 2.31886, "12680": 2.27525, "12685": 2.3664, "12690": 2.30452, "12695": 2.33199, "12700": 2.29244, "12705": 2.30628, "12710": 2.30837, "12715": 2.28749, "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}, "num-zeros": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": 956236544.0, "5": 967337600.0, "10": 971388224.0, "15": 946439424.0, "20": 961330240.0, "25": 1083876480.0, "30": 1211133312.0, "35": 1297707520.0, "40": 1271785728.0, "45": 1175048064.0, "50": 1126729728.0, "55": 1083975424.0, "60": 1045060608.0, "65": 1026047360.0, "70": 995721280.0, "75": 986257152.0, "80": 1010241664.0, "85": 1006739968.0, "90": 988780736.0, "95": 959700032.0, "100": 971861632.0, "105": 980754624.0, "110": 977222528.0, "115": 978430848.0, "120": 961162432.0, "125": 942469184.0, "130": 977095104.0, "135": 966160128.0, "140": 963476928.0, "145": 976512384.0, "150": 921597184.0, "155": 968134336.0, "160": 956383232.0, "165": 959869952.0, "170": 974372224.0, "175": 949013120.0, "180": 946688448.0, "185": 972006784.0, "190": 969055488.0, "195": 985121664.0, "200": 945774592.0, "205": 958353792.0, "210": 979445248.0, "215": 967478208.0, "220": 956423424.0, "225": 962400768.0, "230": 948177792.0, "235": 965221120.0, "240": 966072192.0, "245": 969161216.0, "250": 974435968.0, "255": 925063296.0, "260": 965635968.0, "265": 970660352.0, "270": 959131264.0, "275": 954001216.0, "280": 963427648.0, "285": 945777408.0, "290": 974124544.0, "295": 966704640.0, "300": 967140096.0, "305": 964514048.0, "310": 940354048.0, "315": 967404800.0, "320": 969006080.0, "325": 980552832.0, "330": 972090752.0, "335": 946865984.0, "340": 966598784.0, "345": 973025856.0, "350": 973918720.0, "355": 963261696.0, "360": 948351680.0, "365": 964821248.0, "370": 962952704.0, "375": 958446848.0, "380": 947153280.0, "385": 955988608.0, "390": 945399616.0, "395": 970423552.0, "400": 979770112.0, "405": 968344320.0, "410": 970058752.0, "415": 953158528.0, "420": 943569920.0, "425": 954774144.0, "430": 962663232.0, "435": 977082240.0, "440": 954811392.0, "445": 971894272.0, "450": 963512576.0, "455": 973134720.0, "460": 983714688.0, "465": 945280512.0, "470": 942055616.0, "475": 967007104.0, "480": 966107264.0, "485": 976414528.0, "490": 962538880.0, "495": 945454464.0, "500": 964454656.0, "505": 986005440.0, "510": 965682944.0, "515": 943411584.0, "520": 945017408.0, "525": 971262208.0, "530": 971890688.0, "535": 979140352.0, "540": 969531264.0, "545": 954116608.0, "550": 951267584.0, "555": 987219456.0, "560": 960428288.0, "565": 966616320.0, "570": 975727488.0, "575": 927224960.0, "580": 970694528.0, "585": 961176064.0, "590": 972967040.0, "595": 963682816.0, "600": 937079168.0, "605": 951470208.0, "610": 963360768.0, "615": 970009728.0, "620": 976472192.0, "625": 949579776.0, "630": 954445504.0, "635": 986042816.0, "640": 980980992.0, "645": 955010560.0, "650": 958545664.0, "655": 951656640.0, "660": 961043712.0, "665": 967552000.0, "670": 962514304.0, "675": 968337536.0, "680": 965619200.0, "685": 962871040.0, "690": 961921088.0, "695": 954770368.0, "700": 970340608.0, "705": 945512640.0, "710": 943885440.0, "715": 973357568.0, "720": 968369856.0, "725": 978489984.0, "730": 952195008.0, "735": 948813952.0, "740": 955633408.0, "745": 975866880.0, "750": 981235072.0, "755": 962156608.0, "760": 951964800.0, "765": 967343616.0, "770": 976148096.0, "775": 970544000.0, "780": 977540928.0, "785": 931529024.0, "790": 960441536.0, "795": 964582016.0, "800": 967022848.0, "805": 962321024.0, "810": 940969344.0, "815": 949037568.0, "820": 953181440.0, "825": 954502400.0, "830": 976442240.0, "835": 956073344.0, "840": 948401920.0, "845": 965153024.0, "850": 966029248.0, "855": 960904384.0, "860": 976027200.0, "865": 938157824.0, "870": 966414016.0, "875": 972314880.0, "880": 963120896.0, "885": 967745600.0, "890": 949967872.0, "895": 960019072.0, "900": 974229696.0, "905": 963968256.0, "910": 958435072.0, "915": 956354560.0, "920": 943974592.0, "925": 960833728.0, "930": 978845952.0, "935": 971073664.0, "940": 960905792.0, "945": 945063040.0, "950": 957423360.0, "955": 979035520.0, "960": 983589248.0, "965": 966165824.0, "970": 951228672.0, "975": 961577344.0, "980": 968071040.0, "985": 968991872.0, "990": 984393024.0, "995": 953291264.0, "1000": 934780480.0, "1005": 960147328.0, "1010": 971538624.0, "1015": 985184896.0, "1020": 962780928.0, "1025": 935009408.0, "1030": 974679936.0, "1035": 964992384.0, "1040": 980464256.0, "1045": 960826496.0, "1050": 955197824.0, "1055": 957780352.0, "1060": 967748800.0, "1065": 967116352.0, "1070": 966600064.0, "1075": 950061696.0, "1080": 954508544.0, "1085": 967251712.0, "1090": 977132800.0, "1095": 961237632.0, "1100": 979613568.0, "1105": 953365120.0, "1110": 965954176.0, "1115": 966986944.0, "1120": 970350592.0, "1125": 965707776.0, "1130": 954942400.0, "1135": 965843328.0, "1140": 965176384.0, "1145": 970988224.0, "1150": 955556864.0, "1155": 930578432.0, "1160": 957774208.0, "1165": 978124736.0, "1170": 974299520.0, "1175": 973059648.0, "1180": 973083648.0, "1185": 947344640.0, "1190": 964793216.0, "1195": 953138560.0, "1200": 972843136.0, "1205": 988478656.0, "1210": 931126784.0, "1215": 968647040.0, "1220": 969160960.0, "1225": 975950656.0, "1230": 967331712.0, "1235": 943446912.0, "1240": 955853952.0, "1245": 981503488.0, "1250": 966111808.0, "1255": 973676032.0, "1260": 946497280.0, "1265": 963997824.0, "1270": 960489024.0, "1275": 973615104.0, "1280": 961112576.0, "1285": 957580480.0, "1290": 952528768.0, "1295": 971610240.0, "1300": 968862464.0, "1305": 963739136.0, "1310": 963334656.0, "1315": 943553408.0, "1320": 966307200.0, "1325": 989784960.0, "1330": 969508992.0, "1335": 972302464.0, "1340": 972269440.0, "1345": 960658304.0, "1350": 968639296.0, "1355": 955853312.0, "1360": 971822144.0, "1365": 960387584.0, "1370": 948791872.0, "1375": 973533376.0, "1380": 953470208.0, "1385": 969146880.0, "1390": 975720640.0, "1395": 931673984.0, "1400": 945854848.0, "1405": 976753536.0, "1410": 974510336.0, "1415": 967573760.0, "1420": 966747328.0, "1425": 937378560.0, "1430": 973916608.0, "1435": 978335552.0, "1440": 964178304.0, "1445": 958058240.0, "1450": 946147712.0, "1455": 983922304.0, "1460": 968651136.0, "1465": 948745152.0, "1470": 984243328.0, "1475": 943906048.0, "1480": 963975488.0, "1485": 957349376.0, "1490": 961261888.0, "1495": 980539648.0, "1500": 958332032.0, "1505": 942866816.0, "1510": 984180096.0, "1515": 959094528.0, "1520": 959105408.0, "1525": 952786816.0, "1530": 957741312.0, "1535": 949428928.0, "1540": 971088256.0, "1545": 963132352.0, "1550": 978666752.0, "1555": 952320512.0, "1560": 980089984.0, "1565": 967314048.0, "1570": 973844352.0, "1575": 975494912.0, "1580": 941862656.0, "1585": 970028352.0, "1590": 983822208.0, "1595": 948631616.0, "1600": 967442560.0, "1605": 952451328.0, "1610": 969616512.0, "1615": 983146496.0, "1620": 968019200.0, "1625": 970715776.0, "1630": 962887360.0, "1635": 942311936.0, "1640": 981612224.0, "1645": 973977856.0, "1650": 974188224.0, "1655": 967265024.0, "1660": 940687744.0, "1665": 961704448.0, "1670": 962902016.0, "1675": 971280896.0, "1680": 980879232.0, "1685": 944416192.0, "1690": 964688128.0, "1695": 965644992.0, "1700": 966342336.0, "1705": 985200000.0, "1710": 978354304.0, "1715": 943210880.0, "1720": 977089408.0, "1725": 965870208.0, "1730": 968968960.0, "1735": 965088000.0, "1740": 949713280.0, "1745": 970012352.0, "1750": 959681728.0, "1755": 960085440.0, "1760": 966381376.0, "1765": 951816192.0, "1770": 954665728.0, "1775": 973752064.0, "1780": 970534272.0, "1785": 968824960.0, "1790": 950235520.0, "1795": 945131072.0, "1800": 984666816.0, "1805": 987163520.0, "1810": 977766656.0, "1815": 948004480.0, "1820": 949209216.0, "1825": 978853632.0, "1830": 966362368.0, "1835": 964133632.0, "1840": 972320128.0, "1845": 935415808.0, "1850": 952497792.0, "1855": 980048640.0, "1860": 975866880.0, "1865": 958966528.0, "1870": 958949056.0, "1875": 932593408.0, "1880": 973574016.0, "1885": 978843264.0, "1890": 971358720.0, "1895": 959212288.0, "1900": 947394432.0, "1905": 981829952.0, "1910": 969126912.0, "1915": 970040704.0, "1920": 975597056.0, "1925": 960496512.0, "1930": 977922304.0, "1935": 963250432.0, "1940": 952460928.0, "1945": 981338176.0, "1950": 939172864.0, "1955": 960604416.0, "1960": 970031744.0, "1965": 981176000.0, "1970": 962045120.0, "1975": 952822016.0, "1980": 936847360.0, "1985": 975938432.0, "1990": 965965696.0, "1995": 962609920.0, "2000": 960553984.0, "2005": 954497728.0, "2010": 975579776.0, "2015": 991802112.0, "2020": 975433408.0, "2025": 974303936.0, "2030": 952084736.0, "2035": 967847680.0, "2040": 987457536.0, "2045": 976480064.0, "2050": 984702464.0, "2055": 942839488.0, "2060": 942593920.0, "2065": 966208768.0, "2070": 969622528.0, "2075": 980553536.0, "2080": 977598080.0, "2085": 939635968.0, "2090": 969872256.0, "2095": 961274880.0, "2100": 976719168.0, "2105": 972537920.0, "2110": 959901568.0, "2115": 956875264.0, "2120": 977482304.0, "2125": 962566784.0, "2130": 979618496.0, "2135": 950537408.0, "2140": 946996544.0, "2145": 962273920.0, "2150": 973404416.0, "2155": 972690944.0, "2160": 970314560.0, "2165": 948644160.0, "2170": 961541696.0, "2175": 969377216.0, "2180": 969329920.0, "2185": 947446592.0, "2190": 940480960.0, "2195": 986085952.0, "2200": 961861248.0, "2205": 978924672.0, "2210": 964100864.0, "2215": 963502336.0, "2220": 951311104.0, "2225": 969315776.0, "2230": 976331328.0, "2235": 974025920.0, "2240": 975493888.0, "2245": 960230784.0, "2250": 967640192.0, "2255": 969129984.0, "2260": 975065024.0, "2265": 968258688.0, "2270": 951744768.0, "2275": 962766848.0, "2280": 969640064.0, "2285": 971692992.0, "2290": 962889344.0, "2295": 931409280.0, "2300": 959906048.0, "2305": 970426560.0, "2310": 967444864.0, "2315": 970905792.0, "2320": 975590848.0, "2325": 938587264.0, "2330": 988438528.0, "2335": 977489408.0, "2340": 964596352.0, "2345": 964166528.0, "2350": 947555712.0, "2355": 977029568.0, "2360": 966899072.0, "2365": 977297728.0, "2370": 965072640.0, "2375": 953966272.0, "2380": 962918912.0, "2385": 967194496.0, "2390": 963077248.0, "2395": 974465792.0, "2400": 958410816.0, "2405": 968119552.0, "2410": 951586112.0, "2415": 965904256.0, "2420": 966516160.0, "2425": 959045632.0, "2430": 956685952.0, "2435": 961389184.0, "2440": 959755904.0, "2445": 970891392.0, "2450": 961996736.0, "2455": 922721216.0, "2460": 951953536.0, "2465": 955730432.0, "2470": 972570496.0, "2475": 973812992.0, "2480": 943895296.0, "2485": 944184064.0, "2490": 972411136.0, "2495": 974451712.0, "2500": 973910080.0, "2505": 958492032.0, "2510": 939510912.0, "2515": 979553728.0, "2520": 970473792.0, "2525": 964390784.0, "2530": 955799168.0, "2535": 936598144.0, "2540": 969027648.0, "2545": 970385024.0, "2550": 969462528.0, "2555": 969439040.0, "2560": 964978304.0, "2565": 959763712.0, "2570": 985176704.0, "2575": 957426112.0, "2580": 967424512.0, "2585": 966022400.0, "2590": 956354944.0, "2595": 981830400.0, "2600": 959530880.0, "2605": 962999168.0, "2610": 965972864.0, "2615": 951924992.0, "2620": 971241216.0, "2625": 976456064.0, "2630": 974409984.0, "2635": 948071296.0, "2640": 948137088.0, "2645": 963036736.0, "2650": 953982912.0, "2655": 977112448.0, "2660": 949622976.0, "2665": 953929024.0, "2670": 959064064.0, "2675": 979275904.0, "2680": 961395840.0, "2685": 970701952.0, "2690": 965222016.0, "2695": 943553536.0, "2700": 969425664.0, "2705": 978961792.0, "2710": 971810560.0, "2715": 990813952.0, "2720": 942649152.0, "2725": 967955328.0, "2730": 955466496.0, "2735": 970672704.0, "2740": 977921536.0, "2745": 932280000.0, "2750": 947856384.0, "2755": 956317184.0, "2760": 981697664.0, "2765": 966112192.0, "2770": 948914688.0, "2775": 935830272.0, "2780": 964777088.0, "2785": 969570176.0, "2790": 974273152.0, "2795": 966886144.0, "2800": 944388096.0, "2805": 964353920.0, "2810": 969610752.0, "2815": 975845248.0, "2820": 963081600.0, "2825": 937627392.0, "2830": 956738368.0, "2835": 986321024.0, "2840": 961756672.0, "2845": 967505920.0, "2850": 951714816.0, "2855": 962091520.0, "2860": 954242304.0, "2865": 955881216.0, "2870": 944662848.0, "2875": 974662784.0, "2880": 968199936.0, "2885": 981081984.0, "2890": 953454080.0, "2895": 957178304.0, "2900": 964989440.0, "2905": 931707648.0, "2910": 955730688.0, "2915": 979477120.0, "2920": 970492544.0, "2925": 964975680.0, "2930": 964046592.0, "2935": 940140416.0, "2940": 964912384.0, "2945": 989149952.0, "2950": 965209088.0, "2955": 965104256.0, "2960": 933161472.0, "2965": 968794496.0, "2970": 973034688.0, "2975": 958092288.0, "2980": 964497280.0, "2985": 937267584.0, "2990": 951254720.0, "2995": 978315264.0, "3000": 969275392.0, "3005": 974686080.0, "3010": 950235008.0, "3015": 943841024.0, "3020": 958440960.0, "3025": 975186560.0, "3030": 965018624.0, "3035": 963454464.0, "3040": 952131776.0, "3045": 989793472.0, "3050": 965545728.0, "3055": 982520704.0, "3060": 971227712.0, "3065": 943915648.0, "3070": 978409024.0, "3075": 975204992.0, "3080": 960992896.0, "3085": 962352128.0, "3090": 945953664.0, "3095": 938116032.0, "3100": 972929088.0, "3105": 961989120.0, "3110": 970657152.0, "3115": 963390528.0, "3120": 947116032.0, "3125": 972720640.0, "3130": 952973312.0, "3135": 966041920.0, "3140": 968488896.0, "3145": 937852288.0, "3150": 975009280.0, "3155": 976815488.0, "3160": 969629184.0, "3165": 982194944.0, "3170": 937959936.0, "3175": 953825280.0, "3180": 983810176.0, "3185": 965169536.0, "3190": 968482880.0, "3195": 950933888.0, "3200": 945101440.0, "3205": 959863232.0, "3210": 957486336.0, "3215": 958020096.0, "3220": 968129792.0, "3225": 935614144.0, "3230": 962588672.0, "3235": 975775488.0, "3240": 962621440.0, "3245": 981274368.0, "3250": 943260544.0, "3255": 954599424.0, "3260": 980363648.0, "3265": 963619968.0, "3270": 965162624.0, "3275": 959731072.0, "3280": 967046720.0, "3285": 982478144.0, "3290": 947689408.0, "3295": 966419840.0, "3300": 959165632.0, "3305": 949131008.0, "3310": 979510144.0, "3315": 964283008.0, "3320": 969207296.0, "3325": 956193280.0, "3330": 941167104.0, "3335": 964973184.0, "3340": 956900736.0, "3345": 972500224.0, "3350": 964576192.0, "3355": 943346176.0, "3360": 970037632.0, "3365": 969453952.0, "3370": 954774976.0, "3375": 958676288.0, "3380": 971463168.0, "3385": 947973376.0, "3390": 965793024.0, "3395": 978390016.0, "3400": 978127360.0, "3405": 976724032.0, "3410": 924193664.0, "3415": 955424256.0, "3420": 971820416.0, "3425": 977167488.0, "3430": 973837696.0, "3435": 936071680.0, "3440": 970502464.0, "3445": 957303232.0, "3450": 959840896.0, "3455": 963864256.0, "3460": 967881856.0, "3465": 931318976.0, "3470": 952348224.0, "3475": 973704384.0, "3480": 959738112.0, "3485": 979960640.0, "3490": 944670592.0, "3495": 953904576.0, "3500": 969330432.0, "3505": 964348416.0, "3510": 971222656.0, "3515": 955946112.0, "3520": 958733120.0, "3525": 971914240.0, "3530": 964124672.0, "3535": 983192192.0, "3540": 937489664.0, "3545": 944730496.0, "3550": 984462784.0, "3555": 978057984.0, "3560": 974372992.0, "3565": 968800128.0, "3570": 946694528.0, "3575": 976104640.0, "3580": 977494272.0, "3585": 954568832.0, "3590": 956425856.0, "3595": 951444800.0, "3600": 988996800.0, "3605": 962008448.0, "3610": 965055104.0, "3615": 974635648.0, "3620": 954888384.0, "3625": 939515392.0, "3630": 990148736.0, "3635": 971435712.0, "3640": 976025856.0, "3645": 961489664.0, "3650": 945804160.0, "3655": 965782784.0, "3660": 976208000.0, "3665": 964021248.0, "3670": 977431936.0, "3675": 943476096.0, "3680": 958185152.0, "3685": 964285312.0, "3690": 982093952.0, "3695": 963125248.0, "3700": 950570176.0, "3705": 947345792.0, "3710": 982355328.0, "3715": 972682240.0, "3720": 976138624.0, "3725": 964038272.0, "3730": 948851328.0, "3735": 967079424.0, "3740": 960973568.0, "3745": 969334272.0, "3750": 963949056.0, "3755": 953432832.0, "3760": 976641920.0, "3765": 979841280.0, "3770": 972360320.0, "3775": 972374720.0, "3780": 952585664.0, "3785": 960236800.0, "3790": 985598848.0, "3795": 969195392.0, "3800": 957877504.0, "3805": 972408192.0, "3810": 954517952.0, "3815": 974566528.0, "3820": 963019648.0, "3825": 962070528.0, "3830": 969394304.0, "3835": 934744960.0, "3840": 971255808.0, "3845": 986849792.0, "3850": 968873536.0, "3855": 965248640.0, "3860": 948057600.0, "3865": 975049216.0, "3870": 985098432.0, "3875": 983043072.0, "3880": 963615616.0, "3885": 953012224.0, "3890": 960296960.0, "3895": 960591616.0, "3900": 984926464.0, "3905": 976222592.0, "3910": 987360512.0, "3915": 946017792.0, "3920": 974867328.0, "3925": 961248384.0, "3930": 976790208.0, "3935": 978925824.0, "3940": 950292672.0, "3945": 960260864.0, "3950": 974185152.0, "3955": 972967808.0, "3960": 974078848.0, "3965": 950861696.0, "3970": 980692992.0, "3975": 960750208.0, "3980": 977519808.0, "3985": 962949440.0, "3990": 972755328.0, "3995": 953714176.0, "4000": 975011968.0, "4005": 971658752.0, "4010": 978420992.0, "4015": 971491968.0, "4020": 950311040.0, "4025": 968433024.0, "4030": 997935360.0, "4035": 978548864.0, "4040": 959813248.0, "4045": 939664448.0, "4050": 944718720.0, "4055": 980958720.0, "4060": 977682816.0, "4065": 975706624.0, "4070": 942152832.0, "4075": 945765376.0, "4080": 988765632.0, "4085": 962079872.0, "4090": 983356736.0, "4095": 986958720.0, "4100": 957224448.0, "4105": 954073984.0, "4110": 966488192.0, "4115": 976042432.0, "4120": 983531776.0, "4125": 960054336.0, "4130": 967278976.0, "4135": 971431296.0, "4140": 963171200.0, "4145": 956175616.0, "4150": 960332672.0, "4155": 946216576.0, "4160": 968451328.0, "4165": 970327424.0, "4170": 971946880.0, "4175": 955862272.0, "4180": 940997632.0, "4185": 968318592.0, "4190": 968003712.0, "4195": 989223168.0, "4200": 962678912.0, "4205": 960595072.0, "4210": 971851136.0, "4215": 974145536.0, "4220": 981144704.0, "4225": 975196160.0, "4230": 952703104.0, "4235": 958493312.0, "4240": 966824448.0, "4245": 961756800.0, "4250": 965845632.0, "4255": 958268032.0, "4260": 949545344.0, "4265": 964137856.0, "4270": 978408896.0, "4275": 975397504.0, "4280": 962672064.0, "4285": 951451136.0, "4290": 980029952.0, "4295": 968842816.0, "4300": 958299456.0, "4305": 966802048.0, "4310": 939613376.0, "4315": 949416704.0, "4320": 984522240.0, "4325": 982547712.0, "4330": 974746240.0, "4335": 949459840.0, "4340": 959478400.0, "4345": 956644096.0, "4350": 979960320.0, "4355": 968869888.0, "4360": 966313728.0, "4365": 941208704.0, "4370": 969495552.0, "4375": 972982336.0, "4380": 966147840.0, "4385": 972060544.0, "4390": 954141120.0, "4395": 951856512.0, "4400": 973629184.0, "4405": 972441728.0, "4410": 967908480.0, "4415": 958935232.0, "4420": 960777856.0, "4425": 976416000.0, "4430": 965905664.0, "4435": 975864704.0, "4440": 962282496.0, "4445": 954799872.0, "4450": 978294912.0, "4455": 960203776.0, "4460": 968579008.0, "4465": 968805760.0, "4470": 944127232.0, "4475": 951973056.0, "4480": 978824832.0, "4485": 968265728.0, "4490": 956975616.0, "4495": 938925248.0, "4500": 953215488.0, "4505": 977129344.0, "4510": 978700416.0, "4515": 962493568.0, "4520": 958825024.0, "4525": 958101760.0, "4530": 964428608.0, "4535": 976641024.0, "4540": 976846784.0, "4545": 970169408.0, "4550": 953223680.0, "4555": 959591040.0, "4560": 972497920.0, "4565": 973442560.0, "4570": 978909568.0, "4575": 957760000.0, "4580": 963142016.0, "4585": 957381888.0, "4590": 986530176.0, "4595": 960189824.0, "4600": 952267648.0, "4605": 959297664.0, "4610": 963698304.0, "4615": 957950912.0, "4620": 960255936.0, "4625": 973786624.0, "4630": 944507648.0, "4635": 977024896.0, "4640": 960310272.0, "4645": 981993856.0, "4650": 962400128.0, "4655": 939457792.0, "4660": 964001280.0, "4665": 962540544.0, "4670": 976671872.0, "4675": 963514432.0, "4680": 957488448.0, "4685": 949674432.0, "4690": 956962240.0, "4695": 969926912.0, "4700": 961214016.0, "4705": 970781312.0, "4710": 934493248.0, "4715": 970475008.0, "4720": 966361920.0, "4725": 980303616.0, "4730": 965826240.0, "4735": 937871360.0, "4740": 960124288.0, "4745": 975997376.0, "4750": 967957760.0, "4755": 984959744.0, "4760": 959123648.0, "4765": 955261056.0, "4770": 958621440.0, "4775": 991092608.0, "4780": 976856000.0, "4785": 967582336.0, "4790": 943756160.0, "4795": 955861760.0, "4800": 967745984.0, "4805": 976565888.0, "4810": 965160448.0, "4815": 957969408.0, "4820": 973993216.0, "4825": 961505920.0, "4830": 962638336.0, "4835": 972543936.0, "4840": 948913920.0, "4845": 965716608.0, "4850": 960305024.0, "4855": 964130624.0, "4860": 963051008.0, "4865": 967532352.0, "4870": 957207424.0, "4875": 983574528.0, "4880": 957085120.0, "4885": 977052480.0, "4890": 959740928.0, "4895": 942152192.0, "4900": 973700352.0, "4905": 975213952.0, "4910": 969220608.0, "4915": 970053248.0, "4920": 941169024.0, "4925": 954809344.0, "4930": 977034624.0, "4935": 963750016.0, "4940": 972586496.0, "4945": 960056448.0, "4950": 940793856.0, "4955": 968036480.0, "4960": 976800640.0, "4965": 961022464.0, "4970": 958665472.0, "4975": 933775168.0, "4980": 960824704.0, "4985": 963012160.0, "4990": 963588736.0, "4995": 986307968.0, "5000": 940776000.0, "5005": 968887552.0, "5010": 970307776.0, "5015": 965233408.0, "5020": 966702336.0, "5025": 949471744.0, "5030": 953490560.0, "5035": 967352704.0, "5040": 955689856.0, "5045": 969129856.0, "5050": 953417984.0, "5055": 954811520.0, "5060": 963029248.0, "5065": 952195072.0, "5070": 973597056.0, "5075": 978581632.0, "5080": 942828736.0, "5085": 965863040.0, "5090": 972857088.0, "5095": 964397952.0, "5100": 958318016.0, "5105": 965353728.0, "5110": 950401920.0, "5115": 972347520.0, "5120": 960424960.0, "5125": 969755008.0, "5130": 938795456.0, "5135": 943659008.0, "5140": 969888896.0, "5145": 968688768.0, "5150": 970601216.0, "5155": 972635008.0, "5160": 926551872.0, "5165": 961591552.0, "5170": 966873472.0, "5175": 966086400.0, "5180": 963656192.0, "5185": 930802688.0, "5190": 949852992.0, "5195": 972422016.0, "5200": 973758656.0, "5205": 968249664.0, "5210": 960527936.0, "5215": 928815680.0, "5220": 979174208.0, "5225": 984780416.0, "5230": 975060672.0, "5235": 975055232.0, "5240": 944314112.0, "5245": 970834048.0, "5250": 972426624.0, "5255": 966895296.0, "5260": 976679744.0, "5265": 942270592.0, "5270": 969202176.0, "5275": 970073344.0, "5280": 962825728.0, "5285": 964072064.0, "5290": 932501824.0, "5295": 951762944.0, "5300": 975588288.0, "5305": 951853504.0, "5310": 968040960.0, "5315": 955817472.0, "5320": 950921984.0, "5325": 973040384.0, "5330": 967843264.0, "5335": 967536384.0, "5340": 966503424.0, "5345": 962961344.0, "5350": 978937216.0, "5355": 972202560.0, "5360": 963892288.0, "5365": 965246464.0, "5370": 947824896.0, "5375": 948820864.0, "5380": 967210752.0, "5385": 980540416.0, "5390": 965312832.0, "5395": 955160000.0, "5400": 948286464.0, "5405": 974365952.0, "5410": 967844224.0, "5415": 976074688.0, "5420": 967431680.0, "5425": 937412096.0, "5430": 963919744.0, "5435": 971948544.0, "5440": 969128192.0, "5445": 957501184.0, "5450": 919431040.0, "5455": 952052800.0, "5460": 962315264.0, "5465": 978905088.0, "5470": 981031104.0, "5475": 941610304.0, "5480": 955755072.0, "5485": 964902912.0, "5490": 976036992.0, "5495": 962805120.0, "5500": 971218240.0, "5505": 957001728.0, "5510": 968610432.0, "5515": 945466624.0, "5520": 963181952.0, "5525": 975932480.0, "5530": 936635520.0, "5535": 970693824.0, "5540": 960295936.0, "5545": 972084800.0, "5550": 967898240.0, "5555": 955968960.0, "5560": 954520832.0, "5565": 968862848.0, "5570": 945186112.0, "5575": 960539584.0, "5580": 960563456.0, "5585": 959470720.0, "5590": 977668224.0, "5595": 975194496.0, "5600": 963009472.0, "5605": 964201216.0, "5610": 943157760.0, "5615": 966515904.0, "5620": 963224448.0, "5625": 982398656.0, "5630": 976073984.0, "5635": 957347520.0, "5640": 951455488.0, "5645": 967805568.0, "5650": 979181056.0, "5655": 983510912.0, "5660": 956493952.0, "5665": 953574656.0, "5670": 966097408.0, "5675": 967701184.0, "5680": 978598848.0, "5685": 962008576.0, "5690": 935908928.0, "5695": 963739648.0, "5700": 952464384.0, "5705": 974525376.0, "5710": 971341376.0, "5715": 946147648.0, "5720": 974991360.0, "5725": 967473664.0, "5730": 978561792.0, "5735": 964979712.0, "5740": 943573056.0, "5745": 971142016.0, "5750": 981951168.0, "5755": 956754944.0, "5760": 963695168.0, "5765": 957707648.0, "5770": 955747584.0, "5775": 970847104.0, "5780": 962811840.0, "5785": 970673664.0, "5790": 974652672.0, "5795": 949802368.0, "5800": 965999232.0, "5805": 968774272.0, "5810": 975986176.0, "5815": 970103936.0, "5820": 936377408.0, "5825": 969266816.0, "5830": 977614464.0, "5835": 974980224.0, "5840": 963061120.0, "5845": 968774464.0, "5850": 942897536.0, "5855": 975999104.0, "5860": 979537600.0, "5865": 978369280.0, "5870": 968714112.0, "5875": 942328320.0, "5880": 964508224.0, "5885": 974806656.0, "5890": 972671104.0, "5895": 965681920.0, "5900": 941482880.0, "5905": 961766528.0, "5910": 958568832.0, "5915": 968174464.0, "5920": 977468032.0, "5925": 959468800.0, "5930": 946750080.0, "5935": 952334656.0, "5940": 977662144.0, "5945": 984818560.0, "5950": 980689536.0, "5955": 935094464.0, "5960": 961735296.0, "5965": 965938176.0, "5970": 970612096.0, "5975": 961933888.0, "5980": 958344832.0, "5985": 964562816.0, "5990": 973676288.0, "5995": 955919488.0, "6000": 955633216.0, "6005": 961366784.0, "6010": 952707072.0, "6015": 974539328.0, "6020": 978265920.0, "6025": 972226688.0, "6030": 955311744.0, "6035": 946954368.0, "6040": 962641920.0, "6045": 983743552.0, "6050": 956515200.0, "6055": 963445888.0, "6060": 945767488.0, "6065": 958443776.0, "6070": 978390272.0, "6075": 977988416.0, "6080": 957530304.0, "6085": 947641408.0, "6090": 953645632.0, "6095": 964770560.0, "6100": 979886720.0, "6105": 971011904.0, "6110": 961816128.0, "6115": 943814016.0, "6120": 968573824.0, "6125": 960703232.0, "6130": 984040320.0, "6135": 960994432.0, "6140": 958720704.0, "6145": 971225728.0, "6150": 968493312.0, "6155": 974930688.0, "6160": 977216128.0, "6165": 952742848.0, "6170": 951129728.0, "6175": 963358592.0, "6180": 969549568.0, "6185": 966320320.0, "6190": 963731264.0, "6195": 947266752.0, "6200": 969378240.0, "6205": 967161728.0, "6210": 959339264.0, "6215": 973147776.0, "6220": 936374912.0, "6225": 978715520.0, "6230": 976146816.0, "6235": 971740992.0, "6240": 966006336.0, "6245": 956249728.0, "6250": 956449920.0, "6255": 973363584.0, "6260": 978804800.0, "6265": 974941952.0, "6270": 958855808.0, "6275": 963713152.0, "6280": 973141120.0, "6285": 966123136.0, "6290": 971040256.0, "6295": 987490560.0, "6300": 947640064.0, "6305": 964795456.0, "6310": 979017216.0, "6315": 978443392.0, "6320": 971724672.0, "6325": 923002624.0, "6330": 959366784.0, "6335": 974940864.0, "6340": 984824576.0, "6345": 966835456.0, "6350": 944574016.0, "6355": 957990272.0, "6360": 972711552.0, "6365": 972207744.0, "6370": 958966784.0, "6375": 967142144.0, "6380": 951399104.0, "6385": 973584896.0, "6390": 965514880.0, "6395": 975032064.0, "6400": 983945472.0, "6405": 944065408.0, "6410": 977178496.0, "6415": 971635776.0, "6420": 956726592.0, "6425": 960937728.0, "6430": 957525120.0, "6435": 960333440.0, "6440": 968713088.0, "6445": 973479168.0, "6450": 974637056.0, "6455": 962142208.0, "6460": 940994496.0, "6465": 974482944.0, "6470": 979911936.0, "6475": 960847808.0, "6480": 967532032.0, "6485": 948559616.0, "6490": 970748032.0, "6495": 988369024.0, "6500": 980468864.0, "6505": 972158336.0, "6510": 951648576.0, "6515": 957565440.0, "6520": 979061952.0, "6525": 978903424.0, "6530": 973271744.0, "6535": 967829056.0, "6540": 950159040.0, "6545": 966294144.0, "6550": 979335168.0, "6555": 967119872.0, "6560": 975391104.0, "6565": 949645696.0, "6570": 952068224.0, "6575": 962553728.0, "6580": 975679424.0, "6585": 979544832.0, "6590": 949212544.0, "6595": 961471616.0, "6600": 961353856.0, "6605": 961755520.0, "6610": 985212480.0, "6615": 959518336.0, "6620": 944576256.0, "6625": 971028736.0, "6630": 971564928.0, "6635": 964103936.0, "6640": 959857152.0, "6645": 951077504.0, "6650": 978674944.0, "6655": 965949440.0, "6660": 968814080.0, "6665": 969002112.0, "6670": 932999424.0, "6675": 970736128.0, "6680": 969016064.0, "6685": 958784384.0, "6690": 956215552.0, "6695": 955745920.0, "6700": 962135936.0, "6705": 979365824.0, "6710": 971098240.0, "6715": 966874944.0, "6720": 974162048.0, "6725": 941768192.0, "6730": 979399488.0, "6735": 994709376.0, "6740": 976356224.0, "6745": 974602752.0, "6750": 939272320.0, "6755": 977649344.0, "6760": 969757888.0, "6765": 978454848.0, "6770": 975513728.0, "6775": 943523520.0, "6780": 947283584.0, "6785": 975371712.0, "6790": 960607104.0, "6795": 976217984.0, "6800": 973344640.0, "6805": 946806016.0, "6810": 958265856.0, "6815": 970929792.0, "6820": 978086528.0, "6825": 969192704.0, "6830": 950435072.0, "6835": 981464192.0, "6840": 983022336.0, "6845": 948763840.0, "6850": 965465152.0, "6855": 954199552.0, "6860": 979123968.0, "6865": 983975808.0, "6870": 964842560.0, "6875": 978847808.0, "6880": 950371200.0, "6885": 958582016.0, "6890": 960484032.0, "6895": 965665280.0, "6900": 985370880.0, "6905": 968478592.0, "6910": 950097088.0, "6915": 971060736.0, "6920": 967166720.0, "6925": 965180672.0, "6930": 964715648.0, "6935": 952122112.0, "6940": 962920704.0, "6945": 986470144.0, "6950": 973350272.0, "6955": 964715136.0, "6960": 940248960.0, "6965": 974503680.0, "6970": 978554240.0, "6975": 985114880.0, "6980": 982851072.0, "6985": 959949376.0, "6990": 945298944.0, "6995": 987557120.0, "7000": 963329344.0, "7005": 962922240.0, "7010": 985144320.0, "7015": 945447424.0, "7020": 982884608.0, "7025": 968840640.0, "7030": 953537472.0, "7035": 982810432.0, "7040": 950520320.0, "7045": 956041600.0, "7050": 960403712.0, "7055": 963929728.0, "7060": 976999040.0, "7065": 968391296.0, "7070": 953547264.0, "7075": 956559360.0, "7080": 969124864.0, "7085": 965868800.0, "7090": 969521920.0, "7095": 960078592.0, "7100": 973651200.0, "7105": 973332672.0, "7110": 970254848.0, "7115": 958633088.0, "7120": 948865536.0, "7125": 963163584.0, "7130": 971421376.0, "7135": 964540096.0, "7140": 961681152.0, "7145": 930416448.0, "7150": 946280064.0, "7155": 991092864.0, "7160": 968442496.0, "7165": 956888320.0, "7170": 968275328.0, "7175": 955828224.0, "7180": 958441536.0, "7185": 984880256.0, "7190": 978735936.0, "7195": 973712000.0, "7200": 935905536.0, "7205": 957525760.0, "7210": 967114624.0, "7215": 969707264.0, "7220": 982219584.0, "7225": 928936768.0, "7230": 949560960.0, "7235": 967251712.0, "7240": 966968064.0, "7245": 967454976.0, "7250": 949502336.0, "7255": 957363968.0, "7260": 970114816.0, "7265": 974961664.0, "7270": 959874240.0, "7275": 959286784.0, "7280": 957121920.0, "7285": 977575808.0, "7290": 977249920.0, "7295": 962901120.0, "7300": 975499904.0, "7305": 964022528.0, "7310": 977515520.0, "7315": 966809600.0, "7320": 974552768.0, "7325": 966906752.0, "7330": 959690880.0, "7335": 964049280.0, "7340": 977556864.0, "7345": 967668224.0, "7350": 984804864.0, "7355": 959674816.0, "7360": 948842240.0, "7365": 972772864.0, "7370": 982593664.0, "7375": 963567424.0, "7380": 964284224.0, "7385": 948615488.0, "7390": 964192512.0, "7395": 958787008.0, "7400": 970242816.0, "7405": 988116736.0, "7410": 952423488.0, "7415": 950935744.0, "7420": 967472640.0, "7425": 982705664.0, "7430": 965871552.0, "7435": 973294080.0, "7440": 937228160.0, "7445": 969074752.0, "7450": 980608832.0, "7455": 971587712.0, "7460": 972749056.0, "7465": 939573760.0, "7470": 972011648.0, "7475": 958500480.0, "7480": 969529792.0, "7485": 961637568.0, "7490": 934760704.0, "7495": 957297216.0, "7500": 969548416.0, "7505": 970380928.0, "7510": 972589184.0, "7515": 979619840.0, "7520": 951796224.0, "7525": 970763840.0, "7530": 954655104.0, "7535": 971887616.0, "7540": 979952832.0, "7545": 959493248.0, "7550": 960599936.0, "7555": 960564352.0, "7560": 970209920.0, "7565": 955267200.0, "7570": 942667904.0, "7575": 966060032.0, "7580": 982740480.0, "7585": 979190784.0, "7590": 970155264.0, "7595": 950188416.0, "7600": 946546432.0, "7605": 982722432.0, "7610": 969487360.0, "7615": 988970624.0, "7620": 957136000.0, "7625": 941577856.0, "7630": 971809152.0, "7635": 984756608.0, "7640": 983937792.0, "7645": 968197120.0, "7650": 959357504.0, "7655": 962682368.0, "7660": 969198976.0, "7665": 978073088.0, "7670": 975279104.0, "7675": 975891840.0, "7680": 943167616.0, "7685": 960475136.0, "7690": 975843968.0, "7695": 982086400.0, "7700": 979923648.0, "7705": 940595776.0, "7710": 974811648.0, "7715": 979946496.0, "7720": 968216448.0, "7725": 960576640.0, "7730": 943583104.0, "7735": 968598400.0, "7740": 980697600.0, "7745": 964667008.0, "7750": 963965568.0, "7755": 960341056.0, "7760": 970667072.0, "7765": 971220096.0, "7770": 962730624.0, "7775": 981588800.0, "7780": 965073280.0, "7785": 959776384.0, "7790": 968147968.0, "7795": 969118208.0, "7800": 971586880.0, "7805": 968712128.0, "7810": 946156608.0, "7815": 963601664.0, "7820": 974369664.0, "7825": 963930944.0, "7830": 957420864.0, "7835": 949820864.0, "7840": 957576448.0, "7845": 954299264.0, "7850": 980140416.0, "7855": 987100288.0, "7860": 947203712.0, "7865": 949597632.0, "7870": 965653760.0, "7875": 976341632.0, "7880": 968749184.0, "7885": 969863296.0, "7890": 951979520.0, "7895": 974744576.0, "7900": 964075264.0, "7905": 964628544.0, "7910": 966224768.0, "7915": 943623808.0, "7920": 951251584.0, "7925": 969683840.0, "7930": 965018496.0, "7935": 984522112.0, "7940": 965260992.0, "7945": 950920512.0, "7950": 961778944.0, "7955": 980819072.0, "7960": 964107328.0, "7965": 952800768.0, "7970": 952096960.0, "7975": 969954944.0, "7980": 965058752.0, "7985": 959497728.0, "7990": 968288768.0, "7995": 947074368.0, "8000": 962595712.0, "8005": 980875264.0, "8010": 965703040.0, "8015": 982795648.0, "8020": 960636544.0, "8025": 965519616.0, "8030": 958643200.0, "8035": 975716096.0, "8040": 960827648.0, "8045": 948395264.0, "8050": 959831808.0, "8055": 979617792.0, "8060": 969592128.0, "8065": 958394752.0, "8070": 964066944.0, "8075": 942266240.0, "8080": 966035328.0, "8085": 966815936.0, "8090": 983700160.0, "8095": 988871424.0, "8100": 966531968.0, "8105": 944438272.0, "8110": 969326016.0, "8115": 985228672.0, "8120": 974833408.0, "8125": 964005120.0, "8130": 966272000.0, "8135": 967624576.0, "8140": 963686848.0, "8145": 994976768.0, "8150": 973166016.0, "8155": 938390528.0, "8160": 964462464.0, "8165": 972803200.0, "8170": 968497280.0, "8175": 961587008.0, "8180": 936029440.0, "8185": 962625536.0, "8190": 967799296.0, "8195": 977385088.0, "8200": 956367296.0, "8205": 960566528.0, "8210": 946495424.0, "8215": 982005248.0, "8220": 988443520.0, "8225": 966243584.0, "8230": 962552576.0, "8235": 934131712.0, "8240": 980267904.0, "8245": 976606848.0, "8250": 964327808.0, "8255": 977492864.0, "8260": 956833664.0, "8265": 982957440.0, "8270": 952836608.0, "8275": 974283968.0, "8280": 974906560.0, "8285": 953985664.0, "8290": 940194816.0, "8295": 981360128.0, "8300": 972952832.0, "8305": 978368320.0, "8310": 951095936.0, "8315": 937922048.0, "8320": 977484544.0, "8325": 967872768.0, "8330": 990116800.0, "8335": 975746048.0, "8340": 947366912.0, "8345": 970641408.0, "8350": 970082176.0, "8355": 975014080.0, "8360": 979651456.0, "8365": 932855680.0, "8370": 965537344.0, "8375": 979732736.0, "8380": 965482496.0, "8385": 972889472.0, "8390": 962502912.0, "8395": 951003840.0, "8400": 972739968.0, "8405": 951808384.0, "8410": 960912000.0, "8415": 965867904.0, "8420": 941925888.0, "8425": 968447872.0, "8430": 961416704.0, "8435": 966249344.0, "8440": 969510272.0, "8445": 952921344.0, "8450": 984742912.0, "8455": 990518400.0, "8460": 969086848.0, "8465": 967798656.0, "8470": 963598464.0, "8475": 942921920.0, "8480": 987605888.0, "8485": 979799936.0, "8490": 991849856.0, "8495": 971815552.0, "8500": 951760768.0, "8505": 982982848.0, "8510": 974371200.0, "8515": 969206912.0, "8520": 961827968.0, "8525": 944996096.0, "8530": 984721152.0, "8535": 978411520.0, "8540": 968342592.0, "8545": 969125440.0, "8550": 942408448.0, "8555": 971549056.0, "8560": 958775296.0, "8565": 975676160.0, "8570": 975305216.0, "8575": 971852992.0, "8580": 932583232.0, "8585": 966065856.0, "8590": 978933760.0, "8595": 979387904.0, "8600": 983792768.0, "8605": 958356416.0, "8610": 984069888.0, "8615": 978067776.0, "8620": 963535168.0, "8625": 979909120.0, "8630": 943580032.0, "8635": 961797632.0, "8640": 973745600.0, "8645": 970784128.0, "8650": 969289152.0, "8655": 970653440.0, "8660": 944484096.0, "8665": 986977728.0, "8670": 960353920.0, "8675": 974610176.0, "8680": 962718976.0, "8685": 956147136.0, "8690": 978612864.0, "8695": 969139072.0, "8700": 973135360.0, "8705": 973914176.0, "8710": 947435776.0, "8715": 973736320.0, "8720": 958622976.0, "8725": 978719488.0, "8730": 985894400.0, "8735": 952583040.0, "8740": 940201728.0, "8745": 987763456.0, "8750": 972207744.0, "8755": 971134720.0, "8760": 965569152.0, "8765": 934519872.0, "8770": 986656640.0, "8775": 969789440.0, "8780": 967920512.0, "8785": 962639488.0, "8790": 947921664.0, "8795": 969775296.0, "8800": 971220608.0, "8805": 973559168.0, "8810": 983161280.0, "8815": 951065856.0, "8820": 939478016.0, "8825": 964494336.0, "8830": 981089472.0, "8835": 971889408.0, "8840": 979835520.0, "8845": 951616384.0, "8850": 987153920.0, "8855": 971335296.0, "8860": 962222080.0, "8865": 957359360.0, "8870": 946242816.0, "8875": 968628096.0, "8880": 984173184.0, "8885": 971110144.0, "8890": 970299648.0, "8895": 952971136.0, "8900": 962246528.0, "8905": 977392000.0, "8910": 981876416.0, "8915": 981149952.0, "8920": 968258432.0, "8925": 940189184.0, "8930": 970787456.0, "8935": 963634560.0, "8940": 978025664.0, "8945": 982356352.0, "8950": 946274176.0, "8955": 972928128.0, "8960": 974032128.0, "8965": 973961216.0, "8970": 966361216.0, "8975": 937321600.0, "8980": 953099648.0, "8985": 977878528.0, "8990": 967166592.0, "8995": 980283904.0, "9000": 952421184.0, "9005": 950292544.0, "9010": 974935552.0, "9015": 982668672.0, "9020": 959278656.0, "9025": 979055040.0, "9030": 953936640.0, "9035": 968749312.0, "9040": 978270080.0, "9045": 968843136.0, "9050": 983417600.0, "9055": 947885952.0, "9060": 956699776.0, "9065": 970246528.0, "9070": 968015744.0, "9075": 981225856.0, "9080": 952541632.0, "9085": 971319168.0, "9090": 963789184.0, "9095": 968313984.0, "9100": 974584320.0, "9105": 960032896.0, "9110": 947321664.0, "9115": 956833728.0, "9120": 985899904.0, "9125": 963026176.0, "9130": 958457216.0, "9135": 951989056.0, "9140": 967565824.0, "9145": 977433728.0, "9150": 987305408.0, "9155": 976649408.0, "9160": 958050816.0, "9165": 950957248.0, "9170": 988702272.0, "9175": 971913280.0, "9180": 967854400.0, "9185": 955127680.0, "9190": 957263744.0, "9195": 966003584.0, "9200": 968856960.0, "9205": 967330048.0, "9210": 984179584.0, "9215": 931743808.0, "9220": 949808960.0, "9225": 971440256.0, "9230": 971281792.0, "9235": 971857152.0, "9240": 959917376.0, "9245": 963584128.0, "9250": 961416384.0, "9255": 983241472.0, "9260": 979566336.0, "9265": 953039104.0, "9270": 949474624.0, "9275": 978502016.0, "9280": 978025536.0, "9285": 962828800.0, "9290": 979390080.0, "9295": 958548480.0, "9300": 965876352.0, "9305": 969599232.0, "9310": 973283008.0, "9315": 976451392.0, "9320": 948304512.0, "9325": 979749696.0, "9330": 977926784.0, "9335": 975525504.0, "9340": 960336000.0, "9345": 943464832.0, "9350": 952835072.0, "9355": 962850048.0, "9360": 960675328.0, "9365": 983816320.0, "9370": 983035904.0, "9375": 942080896.0, "9380": 982540928.0, "9385": 985259136.0, "9390": 973406272.0, "9395": 978528128.0, "9400": 938038400.0, "9405": 968500672.0, "9410": 981791488.0, "9415": 991945472.0, "9420": 960625728.0, "9425": 956681216.0, "9430": 938695808.0, "9435": 974362368.0, "9440": 959727872.0, "9445": 973720576.0, "9450": 961877760.0, "9455": 946303872.0, "9460": 978086272.0, "9465": 988617984.0, "9470": 963615872.0, "9475": 983908608.0, "9480": 930854528.0, "9485": 987221248.0, "9490": 963974912.0, "9495": 972857088.0, "9500": 982392960.0, "9505": 970286080.0, "9510": 964873536.0, "9515": 957183296.0, "9520": 948641664.0, "9525": 965336064.0, "9530": 958567296.0, "9535": 950963840.0, "9540": 954501120.0, "9545": 979935296.0, "9550": 955384704.0, "9555": 953296192.0, "9560": 958726208.0, "9565": 969930112.0, "9570": 977751168.0, "9575": 958849792.0, "9580": 963257728.0, "9585": 946197184.0, "9590": 948135936.0, "9595": 967007808.0, "9600": 985117952.0, "9605": 985499648.0, "9610": 943959808.0, "9615": 952912128.0, "9620": 980920192.0, "9625": 978524736.0, "9630": 969671168.0, "9635": 974868544.0, "9640": 940772416.0, "9645": 962475008.0, "9650": 970857536.0, "9655": 987496960.0, "9660": 963394176.0, "9665": 950327872.0, "9670": 965817856.0, "9675": 963579264.0, "9680": 965384064.0, "9685": 986598272.0, "9690": 940596864.0, "9695": 950521728.0, "9700": 975714688.0, "9705": 972896256.0, "9710": 967299968.0, "9715": 971403392.0, "9720": 940613632.0, "9725": 966514816.0, "9730": 974099584.0, "9735": 974345792.0, "9740": 971516928.0, "9745": 951220736.0, "9750": 979370880.0, "9755": 970170432.0, "9760": 968237888.0, "9765": 963835520.0, "9770": 952652160.0, "9775": 956682880.0, "9780": 970721984.0, "9785": 958959232.0, "9790": 961043072.0, "9795": 958779200.0, "9800": 949918656.0, "9805": 962651200.0, "9810": 979093888.0, "9815": 978146816.0, "9820": 982841088.0, "9825": 939730944.0, "9830": 969614208.0, "9835": 973272832.0, "9840": 971945664.0, "9845": 967603328.0, "9850": 947232896.0, "9855": 956896512.0, "9860": 987801728.0, "9865": 970385664.0, "9870": 990310144.0, "9875": 957380096.0, "9880": 931362176.0, "9885": 963678464.0, "9890": 972811648.0, "9895": 984054016.0, "9900": 956595136.0, "9905": 939303808.0, "9910": 979107072.0, "9915": 973996800.0, "9920": 943946432.0, "9925": 963187328.0, "9930": 948020224.0, "9935": 960573120.0, "9940": 965856512.0, "9945": 958998016.0, "9950": 964584192.0, "9955": 943733120.0, "9960": 966844160.0, "9965": 983732096.0, "9970": 966840192.0, "9975": 964040640.0, "9980": 980881024.0, "9985": 942746240.0, "9990": 976134400.0, "9995": 982950848.0, "10000": 972073152.0, "10005": 970193472.0, "10010": 944380480.0, "10015": 983265344.0, "10020": 977865472.0, "10025": 979868544.0, "10030": 971490816.0, "10035": 946263296.0, "10040": 950534016.0, "10045": 977546880.0, "10050": 986017280.0, "10055": 990492800.0, "10060": 958996032.0, "10065": 947517312.0, "10070": 966895616.0, "10075": 979683904.0, "10080": 971953920.0, "10085": 974879744.0, "10090": 944216960.0, "10095": 962977344.0, "10100": 972381952.0, "10105": 976354432.0, "10110": 972128768.0, "10115": 948919680.0, "10120": 962852480.0, "10125": 974293120.0, "10130": 980737472.0, "10135": 972335104.0, "10140": 957843264.0, "10145": 934671872.0, "10150": 973965568.0, "10155": 970306112.0, "10160": 962491456.0, "10165": 975341248.0, "10170": 944624384.0, "10175": 979643712.0, "10180": 984008448.0, "10185": 978870144.0, "10190": 955877376.0, "10195": 937261120.0, "10200": 988253760.0, "10205": 973401856.0, "10210": 966901120.0, "10215": 976049664.0, "10220": 948799872.0, "10225": 950572096.0, "10230": 976120896.0, "10235": 954421632.0, "10240": 969850752.0, "10245": 962265472.0, "10250": 936756480.0, "10255": 979774976.0, "10260": 965000704.0, "10265": 967563712.0, "10270": 969297920.0, "10275": 935944256.0, "10280": 969526272.0, "10285": 996465152.0, "10290": 979762816.0, "10295": 981662912.0, "10300": 952271936.0, "10305": 972024256.0, "10310": 960359872.0, "10315": 971605760.0, "10320": 985354304.0, "10325": 983302336.0, "10330": 935148288.0, "10335": 976392064.0, "10340": 957603840.0, "10345": 973044352.0, "10350": 984707136.0, "10355": 942479296.0, "10360": 962279040.0, "10365": 973641856.0, "10370": 980432768.0, "10375": 970343296.0, "10380": 962080384.0, "10385": 955687296.0, "10390": 990783104.0, "10395": 965164608.0, "10400": 960470208.0, "10405": 950214848.0, "10410": 955491392.0, "10415": 975924736.0, "10420": 967248320.0, "10425": 969875328.0, "10430": 965126272.0, "10435": 962680768.0, "10440": 972024064.0, "10445": 972467456.0, "10450": 974949504.0, "10455": 965864704.0, "10460": 948726272.0, "10465": 971534464.0, "10470": 972756736.0, "10475": 979392128.0, "10480": 997292352.0, "10485": 949631936.0, "10490": 935104896.0, "10495": 969599424.0, "10500": 978688704.0, "10505": 959342784.0, "10510": 951008000.0, "10515": 954223744.0, "10520": 972150016.0, "10525": 969942528.0, "10530": 970425728.0, "10535": 986576256.0, "10540": 946829632.0, "10545": 970484032.0, "10550": 969371968.0, "10555": 959521856.0, "10560": 976274496.0, "10565": 960798208.0, "10570": 968688128.0, "10575": 973272576.0, "10580": 961017472.0, "10585": 973457024.0, "10590": 952053568.0, "10595": 956331776.0, "10600": 967935552.0, "10605": 986576256.0, "10610": 966417408.0, "10615": 976957568.0, "10620": 940933888.0, "10625": 965306432.0, "10630": 968022272.0, "10635": 973333888.0, "10640": 974664448.0, "10645": 948582400.0, "10650": 966388224.0, "10655": 985562624.0, "10660": 976682624.0, "10665": 967088256.0, "10670": 955226368.0, "10675": 934529920.0, "10680": 986153344.0, "10685": 991102656.0, "10690": 963886208.0, "10695": 971933632.0, "10700": 950091520.0, "10705": 978240128.0, "10710": 968317184.0, "10715": 967450432.0, "10720": 966357824.0, "10725": 944490816.0, "10730": 980318592.0, "10735": 961117952.0, "10740": 971283392.0, "10745": 984630528.0, "10750": 981762816.0, "10755": 945191296.0, "10760": 969882304.0, "10765": 972886400.0, "10770": 974268608.0, "10775": 959067392.0, "10780": 949520384.0, "10785": 953706304.0, "10790": 970157568.0, "10795": 960631552.0, "10800": 972050368.0, "10805": 951460864.0, "10810": 974235456.0, "10815": 959804160.0, "10820": 971302656.0, "10825": 967211072.0, "10830": 957016128.0, "10835": 963139136.0, "10840": 971035008.0, "10845": 964268160.0, "10850": 958162432.0, "10855": 967657344.0, "10860": 950849536.0, "10865": 964061696.0, "10870": 983627200.0, "10875": 982016640.0, "10880": 958659648.0, "10885": 954981888.0, "10890": 973122560.0, "10895": 973655744.0, "10900": 970546048.0, "10905": 965184256.0, "10910": 939048192.0, "10915": 960749824.0, "10920": 983653376.0, "10925": 970068160.0, "10930": 968771200.0, "10935": 963228480.0, "10940": 954249408.0, "10945": 964532608.0, "10950": 972466880.0, "10955": 966621248.0, "10960": 972285056.0, "10965": 966333184.0, "10970": 983572160.0, "10975": 965330496.0, "10980": 974669248.0, "10985": 986818496.0, "10990": 950797760.0, "10995": 963598784.0, "11000": 985495104.0, "11005": 978671168.0, "11010": 971614464.0, "11015": 970071232.0, "11020": 948195648.0, "11025": 960105088.0, "11030": 978168768.0, "11035": 976017024.0, "11040": 986523264.0, "11045": 956708480.0, "11050": 973395968.0, "11055": 974051968.0, "11060": 962164544.0, "11065": 985712768.0, "11070": 949791424.0, "11075": 976565888.0, "11080": 972315712.0, "11085": 967328576.0, "11090": 976399296.0, "11095": 946696448.0, "11100": 966199040.0, "11105": 974421504.0, "11110": 981198912.0, "11115": 968108160.0, "11120": 957518656.0, "11125": 956979840.0, "11130": 975786432.0, "11135": 979636544.0, "11140": 964944832.0, "11145": 966499008.0, "11150": 935518400.0, "11155": 976579008.0, "11160": 984367232.0, "11165": 982289792.0, "11170": 978113472.0, "11175": 958084864.0, "11180": 962589888.0, "11185": 972260672.0, "11190": 979666368.0, "11195": 985502784.0, "11200": 983014336.0, "11205": 942426240.0, "11210": 984802368.0, "11215": 967690816.0, "11220": 983476928.0, "11225": 961985728.0, "11230": 953398272.0, "11235": 981841280.0, "11240": 977805568.0, "11245": 966530176.0, "11250": 969466304.0, "11255": 960572544.0, "11260": 980096576.0, "11265": 963926720.0, "11270": 981695936.0, "11275": 968525888.0, "11280": 955905088.0, "11285": 953700224.0, "11290": 956489152.0, "11295": 968197568.0, "11300": 962513216.0, "11305": 958759872.0, "11310": 946304256.0, "11315": 983036096.0, "11320": 964828480.0, "11325": 980906304.0, "11330": 975476608.0, "11335": 952186816.0, "11340": 970596800.0, "11345": 969926080.0, "11350": 981628736.0, "11355": 981905088.0, "11360": 940723328.0, "11365": 970750592.0, "11370": 978978432.0, "11375": 975338432.0, "11380": 968256960.0, "11385": 958096384.0, "11390": 937948288.0, "11395": 977494080.0, "11400": 973515520.0, "11405": 961359424.0, "11410": 966143616.0, "11415": 929202368.0, "11420": 964768960.0, "11425": 981196352.0, "11430": 978636864.0, "11435": 970153280.0, "11440": 945072704.0, "11445": 975241024.0, "11450": 984735296.0, "11455": 971426176.0, "11460": 965182016.0, "11465": 960090176.0, "11470": 955191296.0, "11475": 972691072.0, "11480": 956542272.0, "11485": 977076864.0, "11490": 986332352.0, "11495": 959121344.0, "11500": 969424704.0, "11505": 964024640.0, "11510": 976702848.0, "11515": 977904064.0, "11520": 953963584.0, "11525": 976039360.0, "11530": 976686784.0, "11535": 979809792.0, "11540": 974141760.0, "11545": 953644288.0, "11550": 953295552.0, "11555": 981560640.0, "11560": 984532352.0, "11565": 965181312.0, "11570": 966487424.0, "11575": 950896832.0, "11580": 976062592.0, "11585": 977550784.0, "11590": 969314368.0, "11595": 976719232.0, "11600": 946047104.0, "11605": 973359168.0, "11610": 982457984.0, "11615": 972010048.0, "11620": 969363904.0, "11625": 949111040.0, "11630": 937578176.0, "11635": 973667008.0, "11640": 981259456.0, "11645": 980106048.0, "11650": 971758144.0, "11655": 956204288.0, "11660": 980712192.0, "11665": 958265664.0, "11670": 982618880.0, "11675": 972427200.0, "11680": 956445568.0, "11685": 982955712.0, "11690": 968614528.0, "11695": 968085632.0, "11700": 973819008.0, "11705": 956432640.0, "11710": 964584640.0, "11715": 983208448.0, "11720": 983636224.0, "11725": 965204032.0, "11730": 955695040.0, "11735": 942941376.0, "11740": 973829824.0, "11745": 971260672.0, "11750": 961624256.0, "11755": 963534976.0, "11760": 950291904.0, "11765": 983877632.0, "11770": 984810368.0, "11775": 975671936.0, "11780": 985430336.0, "11785": 947272512.0, "11790": 972444352.0, "11795": 970670464.0, "11800": 973251520.0, "11805": 986780480.0, "11810": 967591808.0, "11815": 955862848.0, "11820": 973765952.0, "11825": 970671296.0, "11830": 974883776.0, "11835": 961826368.0, "11840": 944467904.0, "11845": 980681344.0, "11850": 974601536.0, "11855": 977943744.0, "11860": 971600192.0, "11865": 938631104.0, "11870": 940150208.0, "11875": 990036736.0, "11880": 971605184.0, "11885": 962938432.0, "11890": 970330560.0, "11895": 965356416.0, "11900": 978791360.0, "11905": 961690240.0, "11910": 983653824.0, "11915": 989882688.0, "11920": 944918016.0, "11925": 994335296.0, "11930": 964853504.0, "11935": 963527104.0, "11940": 977089344.0, "11945": 944859968.0, "11950": 977822912.0, "11955": 979190400.0, "11960": 972201664.0, "11965": 976359488.0, "11970": 963048064.0, "11975": 963239936.0, "11980": 977609536.0, "11985": 953135424.0, "11990": 968692864.0, "11995": 965206144.0, "12000": 958650816.0, "12005": 974558720.0, "12010": 979488320.0, "12015": 972212928.0, "12020": 972972992.0, "12025": 934714048.0, "12030": 969111104.0, "12035": 984089664.0, "12040": 977507648.0, "12045": 981645056.0, "12050": 931286400.0, "12055": 938790208.0, "12060": 974349248.0, "12065": 965845696.0, "12070": 968239104.0, "12075": 949940224.0, "12080": 953231552.0, "12085": 972342592.0, "12090": 964042304.0, "12095": 963552832.0, "12100": 976701248.0, "12105": 950225216.0, "12110": 972205568.0, "12115": 968115136.0, "12120": 986359296.0, "12125": 980675264.0, "12130": 941589504.0, "12135": 955591040.0, "12140": 975756032.0, "12145": 979390528.0, "12150": 979435776.0, "12155": 961767936.0, "12160": 946323264.0, "12165": 968612864.0, "12170": 964129152.0, "12175": 967543936.0, "12180": 974942848.0, "12185": 952774592.0, "12190": 988261760.0, "12195": 970261312.0, "12200": 964981312.0, "12205": 968767232.0, "12210": 939144320.0, "12215": 996663488.0, "12220": 970037696.0, "12225": 979618880.0, "12230": 980198144.0, "12235": 950224576.0, "12240": 963441344.0, "12245": 965754240.0, "12250": 976793792.0, "12255": 968140288.0, "12260": 983346688.0, "12265": 931758592.0, "12270": 966457472.0, "12275": 979665408.0, "12280": 977629696.0, "12285": 970432320.0, "12290": 929095296.0, "12295": 976750144.0, "12300": 985683008.0, "12305": 969918016.0, "12310": 986105792.0, "12315": 936237952.0, "12320": 957828032.0, "12325": 966696000.0, "12330": 968270016.0, "12335": 963800896.0, "12340": 957433344.0, "12345": 944247872.0, "12350": 966701760.0, "12355": 975709440.0, "12360": 978832832.0, "12365": 964561280.0, "12370": 948990016.0, "12375": 963593536.0, "12380": 964789056.0, "12385": 973126912.0, "12390": 961501056.0, "12395": 961629824.0, "12400": 975307712.0, "12405": 976388032.0, "12410": 953154688.0, "12415": 962821184.0, "12420": 943854144.0, "12425": 948976640.0, "12430": 972157696.0, "12435": 969004352.0, "12440": 961893696.0, "12445": 951751744.0, "12450": 947021888.0, "12455": 981248448.0, "12460": 973989696.0, "12465": 954462784.0, "12470": 981144320.0, "12475": 958569728.0, "12480": 967084864.0, "12485": 978267776.0, "12490": 974168192.0, "12495": 969692160.0, "12500": 961573632.0, "12505": 943534528.0, "12510": 961013504.0, "12515": 969566080.0, "12520": 974194304.0, "12525": 972107840.0, "12530": 944608640.0, "12535": 976414272.0, "12540": 965917440.0, "12545": 972033856.0, "12550": 969671552.0, "12555": 941300736.0, "12560": 964469312.0, "12565": 947620608.0, "12570": 974443840.0, "12575": 963055232.0, "12580": 958104128.0, "12585": 964300352.0, "12590": 965924288.0, "12595": 978732672.0, "12600": 982093952.0, "12605": 949125824.0, "12610": 937745344.0, "12615": 962779264.0, "12620": 961201664.0, "12625": 966531136.0, "12630": 971046272.0, "12635": 962236416.0, "12640": 978541696.0, "12645": 969369920.0, "12650": 970161664.0, "12655": 964217216.0, "12660": 932390336.0, "12665": 956865664.0, "12670": 986180352.0, "12675": 965566464.0, "12680": 961126528.0, "12685": 951304256.0, "12690": 945491456.0, "12695": 978387648.0, "12700": 985277888.0, "12705": 958784640.0, "12710": 968294144.0, "12715": 956280512.0, "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}, "mem-allocated-bytes": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": 12795811840.0, "5": 12795811840.0, "10": 12795811840.0, "15": 12795811840.0, "20": 12795811840.0, "25": 12795811840.0, "30": 12795811840.0, "35": 12795811840.0, "40": 12795811840.0, "45": 12795811840.0, "50": 12795811840.0, "55": 12795811840.0, "60": 12795811840.0, "65": 12795811840.0, "70": 12795811840.0, "75": 12795811840.0, "80": 12795811840.0, "85": 12795811840.0, "90": 12795811840.0, "95": 12795811840.0, "100": 12795811840.0, "105": 12795811840.0, "110": 12795811840.0, "115": 12795811840.0, "120": 12795811840.0, "125": 12795811840.0, "130": 12795811840.0, "135": 12795811840.0, "140": 12795811840.0, "145": 12795811840.0, "150": 12795811840.0, "155": 12795811840.0, "160": 12795811840.0, "165": 12795811840.0, "170": 12795811840.0, "175": 12795811840.0, "180": 12795811840.0, "185": 12795811840.0, "190": 12795811840.0, "195": 12795811840.0, "200": 12795811840.0, "205": 12795811840.0, "210": 12795811840.0, "215": 12795811840.0, "220": 12795811840.0, "225": 12795811840.0, "230": 12795811840.0, "235": 12795811840.0, "240": 12795811840.0, "245": 12795811840.0, "250": 12795811840.0, "255": 12795811840.0, "260": 12795811840.0, "265": 12795811840.0, "270": 12795811840.0, "275": 12795811840.0, "280": 12795811840.0, "285": 12795811840.0, "290": 12795811840.0, "295": 12795811840.0, "300": 12795811840.0, "305": 12795811840.0, "310": 12795811840.0, "315": 12795811840.0, "320": 12795811840.0, "325": 12795811840.0, "330": 12795811840.0, "335": 12795811840.0, "340": 12795811840.0, "345": 12795811840.0, "350": 12795811840.0, "355": 12795811840.0, "360": 12795811840.0, "365": 12795811840.0, "370": 12795811840.0, "375": 12795811840.0, "380": 12795811840.0, "385": 12795811840.0, "390": 12795811840.0, "395": 12795811840.0, "400": 12795811840.0, "405": 12795811840.0, "410": 12795811840.0, "415": 12795811840.0, "420": 12795811840.0, "425": 12795811840.0, "430": 12795811840.0, "435": 12795811840.0, "440": 12795811840.0, "445": 12795811840.0, "450": 12795811840.0, "455": 12795811840.0, "460": 12795811840.0, "465": 12795811840.0, "470": 12795811840.0, "475": 12795811840.0, "480": 12795811840.0, "485": 12795811840.0, "490": 12795811840.0, "495": 12795811840.0, "500": 12795811840.0, "505": 12795811840.0, "510": 12795811840.0, "515": 12795811840.0, "520": 12795811840.0, "525": 12795811840.0, "530": 12795811840.0, "535": 12795811840.0, "540": 12795811840.0, "545": 12795811840.0, "550": 12795811840.0, "555": 12795811840.0, "560": 12795811840.0, "565": 12795811840.0, "570": 12795811840.0, "575": 12795811840.0, "580": 12795811840.0, "585": 12795811840.0, "590": 12795811840.0, "595": 12795811840.0, "600": 12795811840.0, "605": 12795811840.0, "610": 12795811840.0, "615": 12795811840.0, "620": 12795811840.0, "625": 12795811840.0, "630": 12795811840.0, "635": 12795811840.0, "640": 12795811840.0, "645": 12795811840.0, "650": 12795811840.0, "655": 12795811840.0, "660": 12795811840.0, "665": 12795811840.0, "670": 12795811840.0, "675": 12795811840.0, "680": 12795811840.0, "685": 12795811840.0, "690": 12795811840.0, "695": 12795811840.0, "700": 12795811840.0, "705": 12795811840.0, "710": 12795811840.0, "715": 12795811840.0, "720": 12795811840.0, "725": 12795811840.0, "730": 12795811840.0, "735": 12795811840.0, "740": 12795811840.0, "745": 12795811840.0, "750": 12795811840.0, "755": 12795811840.0, "760": 12795811840.0, "765": 12795811840.0, "770": 12795811840.0, "775": 12795811840.0, "780": 12795811840.0, "785": 12795811840.0, "790": 12795811840.0, "795": 12795811840.0, "800": 12795811840.0, "805": 12795811840.0, "810": 12795811840.0, "815": 12795811840.0, "820": 12795811840.0, "825": 12795811840.0, "830": 12795811840.0, "835": 12795811840.0, "840": 12795811840.0, "845": 12795811840.0, "850": 12795811840.0, "855": 12795811840.0, "860": 12795811840.0, "865": 12795811840.0, "870": 12795811840.0, "875": 12795811840.0, "880": 12795811840.0, "885": 12795811840.0, "890": 12795811840.0, "895": 12795811840.0, "900": 12795811840.0, "905": 12795811840.0, "910": 12795811840.0, "915": 12795811840.0, "920": 12795811840.0, "925": 12795811840.0, "930": 12795811840.0, "935": 12795811840.0, "940": 12795811840.0, "945": 12795811840.0, "950": 12795811840.0, "955": 12795811840.0, "960": 12795811840.0, "965": 12795811840.0, "970": 12795811840.0, "975": 12795811840.0, "980": 12795811840.0, "985": 12795811840.0, "990": 12795811840.0, "995": 12795811840.0, "1000": 12795811840.0, "1005": 12795811840.0, "1010": 12795811840.0, "1015": 12795811840.0, "1020": 12795811840.0, "1025": 12795811840.0, "1030": 12795811840.0, "1035": 12795811840.0, "1040": 12795811840.0, "1045": 12795811840.0, "1050": 12795811840.0, "1055": 12795811840.0, "1060": 12795811840.0, "1065": 12795811840.0, "1070": 12795811840.0, "1075": 12795811840.0, "1080": 12795811840.0, "1085": 12795811840.0, "1090": 12795811840.0, "1095": 12795811840.0, "1100": 12795811840.0, "1105": 12795811840.0, "1110": 12795811840.0, "1115": 12795811840.0, "1120": 12795811840.0, "1125": 12795811840.0, "1130": 12795811840.0, "1135": 12795811840.0, "1140": 12795811840.0, "1145": 12795811840.0, "1150": 12795811840.0, "1155": 12795811840.0, "1160": 12795811840.0, "1165": 12795811840.0, "1170": 12795811840.0, "1175": 12795811840.0, "1180": 12795811840.0, "1185": 12795811840.0, "1190": 12795811840.0, "1195": 12795811840.0, "1200": 12795811840.0, "1205": 12795811840.0, "1210": 12795811840.0, "1215": 12795811840.0, "1220": 12795811840.0, "1225": 12795811840.0, "1230": 12795811840.0, "1235": 12795811840.0, "1240": 12795811840.0, "1245": 12795811840.0, "1250": 12795811840.0, "1255": 12795811840.0, "1260": 12795811840.0, "1265": 12795811840.0, "1270": 12795811840.0, "1275": 12795811840.0, "1280": 12795811840.0, "1285": 12795811840.0, "1290": 12795811840.0, "1295": 12795811840.0, "1300": 12795811840.0, "1305": 12795811840.0, "1310": 12795811840.0, "1315": 12795811840.0, "1320": 12795811840.0, "1325": 12795811840.0, "1330": 12795811840.0, "1335": 12795811840.0, "1340": 12795811840.0, "1345": 12795811840.0, "1350": 12795811840.0, "1355": 12795811840.0, "1360": 12795811840.0, "1365": 12795811840.0, "1370": 12795811840.0, "1375": 12795811840.0, "1380": 12795811840.0, "1385": 12795811840.0, "1390": 12795811840.0, "1395": 12795811840.0, "1400": 12795811840.0, "1405": 12795811840.0, "1410": 12795811840.0, "1415": 12795811840.0, "1420": 12795811840.0, "1425": 12795811840.0, "1430": 12795811840.0, "1435": 12795811840.0, "1440": 12795811840.0, "1445": 12795811840.0, "1450": 12795811840.0, "1455": 12795811840.0, "1460": 12795811840.0, "1465": 12795811840.0, "1470": 12795811840.0, "1475": 12795811840.0, "1480": 12795811840.0, "1485": 12795811840.0, "1490": 12795811840.0, "1495": 12795811840.0, "1500": 12795811840.0, "1505": 12795811840.0, "1510": 12795811840.0, "1515": 12795811840.0, "1520": 12795811840.0, "1525": 12795811840.0, "1530": 12795811840.0, "1535": 12795811840.0, "1540": 12795811840.0, "1545": 12795811840.0, "1550": 12795811840.0, "1555": 12795811840.0, "1560": 12795811840.0, "1565": 12795811840.0, "1570": 12795811840.0, "1575": 12795811840.0, "1580": 12795811840.0, "1585": 12795811840.0, "1590": 12795811840.0, "1595": 12795811840.0, "1600": 12795811840.0, "1605": 12795811840.0, "1610": 12795811840.0, "1615": 12795811840.0, "1620": 12795811840.0, "1625": 12795811840.0, "1630": 12795811840.0, "1635": 12795811840.0, "1640": 12795811840.0, "1645": 12795811840.0, "1650": 12795811840.0, "1655": 12795811840.0, "1660": 12795811840.0, "1665": 12795811840.0, "1670": 12795811840.0, "1675": 12795811840.0, "1680": 12795811840.0, "1685": 12795811840.0, "1690": 12795811840.0, "1695": 12795811840.0, "1700": 12795811840.0, "1705": 12795811840.0, "1710": 12795811840.0, "1715": 12795811840.0, "1720": 12795811840.0, "1725": 12795811840.0, "1730": 12795811840.0, "1735": 12795811840.0, "1740": 12795811840.0, "1745": 12795811840.0, "1750": 12795811840.0, "1755": 12795811840.0, "1760": 12795811840.0, "1765": 12795811840.0, "1770": 12795811840.0, "1775": 12795811840.0, "1780": 12795811840.0, "1785": 12795811840.0, "1790": 12795811840.0, "1795": 12795811840.0, "1800": 12795811840.0, "1805": 12795811840.0, "1810": 12795811840.0, "1815": 12795811840.0, "1820": 12795811840.0, "1825": 12795811840.0, "1830": 12795811840.0, "1835": 12795811840.0, "1840": 12795811840.0, "1845": 12795811840.0, "1850": 12795811840.0, "1855": 12795811840.0, "1860": 12795811840.0, "1865": 12795811840.0, "1870": 12795811840.0, "1875": 12795811840.0, "1880": 12795811840.0, "1885": 12795811840.0, "1890": 12795811840.0, "1895": 12795811840.0, "1900": 12795811840.0, "1905": 12795811840.0, "1910": 12795811840.0, "1915": 12795811840.0, "1920": 12795811840.0, "1925": 12795811840.0, "1930": 12795811840.0, "1935": 12795811840.0, "1940": 12795811840.0, "1945": 12795811840.0, "1950": 12795811840.0, "1955": 12795811840.0, "1960": 12795811840.0, "1965": 12795811840.0, "1970": 12795811840.0, "1975": 12795811840.0, "1980": 12795811840.0, "1985": 12795811840.0, "1990": 12795811840.0, "1995": 12795811840.0, "2000": 12795811840.0, "2005": 12795811840.0, "2010": 12795811840.0, "2015": 12795811840.0, "2020": 12795811840.0, "2025": 12795811840.0, "2030": 12795811840.0, "2035": 12795811840.0, "2040": 12795811840.0, "2045": 12795811840.0, "2050": 12795811840.0, "2055": 12795811840.0, "2060": 12795811840.0, "2065": 12795811840.0, "2070": 12795811840.0, "2075": 12795811840.0, "2080": 12795811840.0, "2085": 12795811840.0, "2090": 12795811840.0, "2095": 12795811840.0, "2100": 12795811840.0, "2105": 12795811840.0, "2110": 12795811840.0, "2115": 12795811840.0, "2120": 12795811840.0, "2125": 12795811840.0, "2130": 12795811840.0, "2135": 12795811840.0, "2140": 12795811840.0, "2145": 12795811840.0, "2150": 12795811840.0, "2155": 12795811840.0, "2160": 12795811840.0, "2165": 12795811840.0, "2170": 12795811840.0, "2175": 12795811840.0, "2180": 12795811840.0, "2185": 12795811840.0, "2190": 12795811840.0, "2195": 12795811840.0, "2200": 12795811840.0, "2205": 12795811840.0, "2210": 12795811840.0, "2215": 12795811840.0, "2220": 12795811840.0, "2225": 12795811840.0, "2230": 12795811840.0, "2235": 12795811840.0, "2240": 12795811840.0, "2245": 12795811840.0, "2250": 12795811840.0, "2255": 12795811840.0, "2260": 12795811840.0, "2265": 12795811840.0, "2270": 12795811840.0, "2275": 12795811840.0, "2280": 12795811840.0, "2285": 12795811840.0, "2290": 12795811840.0, "2295": 12795811840.0, "2300": 12795811840.0, "2305": 12795811840.0, "2310": 12795811840.0, "2315": 12795811840.0, "2320": 12795811840.0, "2325": 12795811840.0, "2330": 12795811840.0, "2335": 12795811840.0, "2340": 12795811840.0, "2345": 12795811840.0, "2350": 12795811840.0, "2355": 12795811840.0, "2360": 12795811840.0, "2365": 12795811840.0, "2370": 12795811840.0, "2375": 12795811840.0, "2380": 12795811840.0, "2385": 12795811840.0, "2390": 12795811840.0, "2395": 12795811840.0, "2400": 12795811840.0, "2405": 12795811840.0, "2410": 12795811840.0, "2415": 12795811840.0, "2420": 12795811840.0, "2425": 12795811840.0, "2430": 12795811840.0, "2435": 12795811840.0, "2440": 12795811840.0, "2445": 12795811840.0, "2450": 12795811840.0, "2455": 12795811840.0, "2460": 12795811840.0, "2465": 12795811840.0, "2470": 12795811840.0, "2475": 12795811840.0, "2480": 12795811840.0, "2485": 12795811840.0, "2490": 12795811840.0, "2495": 12795811840.0, "2500": 12795811840.0, "2505": 12795811840.0, "2510": 12795811840.0, "2515": 12795811840.0, "2520": 12795811840.0, "2525": 12795811840.0, "2530": 12795811840.0, "2535": 12795811840.0, "2540": 12795811840.0, "2545": 12795811840.0, "2550": 12795811840.0, "2555": 12795811840.0, "2560": 12795811840.0, "2565": 12795811840.0, "2570": 12795811840.0, "2575": 12795811840.0, "2580": 12795811840.0, "2585": 12795811840.0, "2590": 12795811840.0, "2595": 12795811840.0, "2600": 12795811840.0, "2605": 12795811840.0, "2610": 12795811840.0, "2615": 12795811840.0, "2620": 12795811840.0, "2625": 12795811840.0, "2630": 12795811840.0, "2635": 12795811840.0, "2640": 12795811840.0, "2645": 12795811840.0, "2650": 12795811840.0, "2655": 12795811840.0, "2660": 12795811840.0, "2665": 12795811840.0, "2670": 12795811840.0, "2675": 12795811840.0, "2680": 12795811840.0, "2685": 12795811840.0, "2690": 12795811840.0, "2695": 12795811840.0, "2700": 12795811840.0, "2705": 12795811840.0, "2710": 12795811840.0, "2715": 12795811840.0, "2720": 12795811840.0, "2725": 12795811840.0, "2730": 12795811840.0, "2735": 12795811840.0, "2740": 12795811840.0, "2745": 12795811840.0, "2750": 12795811840.0, "2755": 12795811840.0, "2760": 12795811840.0, "2765": 12795811840.0, "2770": 12795811840.0, "2775": 12795811840.0, "2780": 12795811840.0, "2785": 12795811840.0, "2790": 12795811840.0, "2795": 12795811840.0, "2800": 12795811840.0, "2805": 12795811840.0, "2810": 12795811840.0, "2815": 12795811840.0, "2820": 12795811840.0, "2825": 12795811840.0, "2830": 12795811840.0, "2835": 12795811840.0, "2840": 12795811840.0, "2845": 12795811840.0, "2850": 12795811840.0, "2855": 12795811840.0, "2860": 12795811840.0, "2865": 12795811840.0, "2870": 12795811840.0, "2875": 12795811840.0, "2880": 12795811840.0, "2885": 12795811840.0, "2890": 12795811840.0, "2895": 12795811840.0, "2900": 12795811840.0, "2905": 12795811840.0, "2910": 12795811840.0, "2915": 12795811840.0, "2920": 12795811840.0, "2925": 12795811840.0, "2930": 12795811840.0, "2935": 12795811840.0, "2940": 12795811840.0, "2945": 12795811840.0, "2950": 12795811840.0, "2955": 12795811840.0, "2960": 12795811840.0, "2965": 12795811840.0, "2970": 12795811840.0, "2975": 12795811840.0, "2980": 12795811840.0, "2985": 12795811840.0, "2990": 12795811840.0, "2995": 12795811840.0, "3000": 12795811840.0, "3005": 12795811840.0, "3010": 12795811840.0, "3015": 12795811840.0, "3020": 12795811840.0, "3025": 12795811840.0, "3030": 12795811840.0, "3035": 12795811840.0, "3040": 12795811840.0, "3045": 12795811840.0, "3050": 12795811840.0, "3055": 12795811840.0, "3060": 12795811840.0, "3065": 12795811840.0, "3070": 12795811840.0, "3075": 12795811840.0, "3080": 12795811840.0, "3085": 12795811840.0, "3090": 12795811840.0, "3095": 12795811840.0, "3100": 12795811840.0, "3105": 12795811840.0, "3110": 12795811840.0, "3115": 12795811840.0, "3120": 12795811840.0, "3125": 12795811840.0, "3130": 12795811840.0, "3135": 12795811840.0, "3140": 12795811840.0, "3145": 12795811840.0, "3150": 12795811840.0, "3155": 12795811840.0, "3160": 12795811840.0, "3165": 12795811840.0, "3170": 12795811840.0, "3175": 12795811840.0, "3180": 12795811840.0, "3185": 12795811840.0, "3190": 12795811840.0, "3195": 12795811840.0, "3200": 12795811840.0, "3205": 12795811840.0, "3210": 12795811840.0, "3215": 12795811840.0, "3220": 12795811840.0, "3225": 12795811840.0, "3230": 12795811840.0, "3235": 12795811840.0, "3240": 12795811840.0, "3245": 12795811840.0, "3250": 12795811840.0, "3255": 12795811840.0, "3260": 12795811840.0, "3265": 12795811840.0, "3270": 12795811840.0, "3275": 12795811840.0, "3280": 12795811840.0, "3285": 12795811840.0, "3290": 12795811840.0, "3295": 12795811840.0, "3300": 12795811840.0, "3305": 12795811840.0, "3310": 12795811840.0, "3315": 12795811840.0, "3320": 12795811840.0, "3325": 12795811840.0, "3330": 12795811840.0, "3335": 12795811840.0, "3340": 12795811840.0, "3345": 12795811840.0, "3350": 12795811840.0, "3355": 12795811840.0, "3360": 12795811840.0, "3365": 12795811840.0, "3370": 12795811840.0, "3375": 12795811840.0, "3380": 12795811840.0, "3385": 12795811840.0, "3390": 12795811840.0, "3395": 12795811840.0, "3400": 12795811840.0, "3405": 12795811840.0, "3410": 12795811840.0, "3415": 12795811840.0, "3420": 12795811840.0, "3425": 12795811840.0, "3430": 12795811840.0, "3435": 12795811840.0, "3440": 12795811840.0, "3445": 12795811840.0, "3450": 12795811840.0, "3455": 12795811840.0, "3460": 12795811840.0, "3465": 12795811840.0, "3470": 12795811840.0, "3475": 12795811840.0, "3480": 12795811840.0, "3485": 12795811840.0, "3490": 12795811840.0, "3495": 12795811840.0, "3500": 12795811840.0, "3505": 12795811840.0, "3510": 12795811840.0, "3515": 12795811840.0, "3520": 12795811840.0, "3525": 12795811840.0, "3530": 12795811840.0, "3535": 12795811840.0, "3540": 12795811840.0, "3545": 12795811840.0, "3550": 12795811840.0, "3555": 12795811840.0, "3560": 12795811840.0, "3565": 12795811840.0, "3570": 12795789312.0, "3575": 12795789312.0, "3580": 12795789312.0, "3585": 12795789312.0, "3590": 12795789312.0, "3595": 12795789312.0, "3600": 12795789312.0, "3605": 12795789312.0, "3610": 12795789312.0, "3615": 12795789312.0, "3620": 12795789312.0, "3625": 12795789312.0, "3630": 12795789312.0, "3635": 12795789312.0, "3640": 12795789312.0, "3645": 12795789312.0, "3650": 12795789312.0, "3655": 12795789312.0, "3660": 12795789312.0, "3665": 12795789312.0, "3670": 12795789312.0, "3675": 12795789312.0, "3680": 12795789312.0, "3685": 12795789312.0, "3690": 12795789312.0, "3695": 12795789312.0, "3700": 12795789312.0, "3705": 12795789312.0, "3710": 12795789312.0, "3715": 12795789312.0, "3720": 12795789312.0, "3725": 12795789312.0, "3730": 12795789312.0, "3735": 12795789312.0, "3740": 12795789312.0, "3745": 12795789312.0, "3750": 12795789312.0, "3755": 12795789312.0, "3760": 12795789312.0, "3765": 12795789312.0, "3770": 12795789312.0, "3775": 12795789312.0, "3780": 12795789312.0, "3785": 12795789312.0, "3790": 12795789312.0, "3795": 12795789312.0, "3800": 12795789312.0, "3805": 12795789312.0, "3810": 12795789312.0, "3815": 12795789312.0, "3820": 12795789312.0, "3825": 12795789312.0, "3830": 12795789312.0, "3835": 12795789312.0, "3840": 12795789312.0, "3845": 12795789312.0, "3850": 12795789312.0, "3855": 12795789312.0, "3860": 12795789312.0, "3865": 12795789312.0, "3870": 12795789312.0, "3875": 12795789312.0, "3880": 12795789312.0, "3885": 12795789312.0, "3890": 12795789312.0, "3895": 12795789312.0, "3900": 12795789312.0, "3905": 12795789312.0, "3910": 12795789312.0, "3915": 12795789312.0, "3920": 12795789312.0, "3925": 12795789312.0, "3930": 12795789312.0, "3935": 12795789312.0, "3940": 12795789312.0, "3945": 12795789312.0, "3950": 12795789312.0, "3955": 12795789312.0, "3960": 12795789312.0, "3965": 12795789312.0, "3970": 12795789312.0, "3975": 12795789312.0, "3980": 12795789312.0, "3985": 12795789312.0, "3990": 12795789312.0, "3995": 12795789312.0, "4000": 12795789312.0, "4005": 12795789312.0, "4010": 12795789312.0, "4015": 12795789312.0, "4020": 12795789312.0, "4025": 12795789312.0, "4030": 12795789312.0, "4035": 12795789312.0, "4040": 12795789312.0, "4045": 12795789312.0, "4050": 12795789312.0, "4055": 12795789312.0, "4060": 12795789312.0, "4065": 12795789312.0, "4070": 12795789312.0, "4075": 12795789312.0, "4080": 12795789312.0, "4085": 12795789312.0, "4090": 12795789312.0, "4095": 12795789312.0, "4100": 12795789312.0, "4105": 12795789312.0, "4110": 12795789312.0, "4115": 12795789312.0, "4120": 12795789312.0, "4125": 12795789312.0, "4130": 12795789312.0, "4135": 12795789312.0, "4140": 12795789312.0, "4145": 12795789312.0, "4150": 12795789312.0, "4155": 12795789312.0, "4160": 12795789312.0, "4165": 12795789312.0, "4170": 12795789312.0, "4175": 12795789312.0, "4180": 12795789312.0, "4185": 12795789312.0, "4190": 12795789312.0, "4195": 12795789312.0, "4200": 12795789312.0, "4205": 12795789312.0, "4210": 12795789312.0, "4215": 12795789312.0, "4220": 12795789312.0, "4225": 12795789312.0, "4230": 12795789312.0, "4235": 12795789312.0, "4240": 12795789312.0, "4245": 12795789312.0, "4250": 12795789312.0, "4255": 12795789312.0, "4260": 12795789312.0, "4265": 12795789312.0, "4270": 12795789312.0, "4275": 12795789312.0, "4280": 12795789312.0, "4285": 12795789312.0, "4290": 12795789312.0, "4295": 12795789312.0, "4300": 12795789312.0, "4305": 12795789312.0, "4310": 12795789312.0, "4315": 12795789312.0, "4320": 12795789312.0, "4325": 12795789312.0, "4330": 12795789312.0, "4335": 12795789312.0, "4340": 12795789312.0, "4345": 12795789312.0, "4350": 12795789312.0, "4355": 12795789312.0, "4360": 12795789312.0, "4365": 12795789312.0, "4370": 12795789312.0, "4375": 12795789312.0, "4380": 12795789312.0, "4385": 12795789312.0, "4390": 12795789312.0, "4395": 12795789312.0, "4400": 12795789312.0, "4405": 12795789312.0, "4410": 12795789312.0, "4415": 12795789312.0, "4420": 12795789312.0, "4425": 12795789312.0, "4430": 12795789312.0, "4435": 12795789312.0, "4440": 12795789312.0, "4445": 12795789312.0, "4450": 12795789312.0, "4455": 12795789312.0, "4460": 12795789312.0, "4465": 12795789312.0, "4470": 12795789312.0, "4475": 12795789312.0, "4480": 12795789312.0, "4485": 12795789312.0, "4490": 12795789312.0, "4495": 12795789312.0, "4500": 12795789312.0, "4505": 12795789312.0, "4510": 12795789312.0, "4515": 12795789312.0, "4520": 12795789312.0, "4525": 12795789312.0, "4530": 12795789312.0, "4535": 12795789312.0, "4540": 12795789312.0, "4545": 12795789312.0, "4550": 12795789312.0, "4555": 12795789312.0, "4560": 12795789312.0, "4565": 12795789312.0, "4570": 12795789312.0, "4575": 12795789312.0, "4580": 12795789312.0, "4585": 12795789312.0, "4590": 12795789312.0, "4595": 12795789312.0, "4600": 12795789312.0, "4605": 12795789312.0, "4610": 12795789312.0, "4615": 12795789312.0, "4620": 12795789312.0, "4625": 12795789312.0, "4630": 12795789312.0, "4635": 12795789312.0, "4640": 12795789312.0, "4645": 12795789312.0, "4650": 12795789312.0, "4655": 12795789312.0, "4660": 12795789312.0, "4665": 12795789312.0, "4670": 12795789312.0, "4675": 12795789312.0, "4680": 12795789312.0, "4685": 12795789312.0, "4690": 12795789312.0, "4695": 12795789312.0, "4700": 12795789312.0, "4705": 12795789312.0, "4710": 12795789312.0, "4715": 12795789312.0, "4720": 12795789312.0, "4725": 12795789312.0, "4730": 12795789312.0, "4735": 12795789312.0, "4740": 12795789312.0, "4745": 12795789312.0, "4750": 12795789312.0, "4755": 12795789312.0, "4760": 12795789312.0, "4765": 12795789312.0, "4770": 12795789312.0, "4775": 12795789312.0, "4780": 12795789312.0, "4785": 12795789312.0, "4790": 12795789312.0, "4795": 12795789312.0, "4800": 12795789312.0, "4805": 12795789312.0, "4810": 12795789312.0, "4815": 12795789312.0, "4820": 12795789312.0, "4825": 12795789312.0, "4830": 12795789312.0, "4835": 12795789312.0, "4840": 12795789312.0, "4845": 12795789312.0, "4850": 12795789312.0, "4855": 12795789312.0, "4860": 12795789312.0, "4865": 12795789312.0, "4870": 12795789312.0, "4875": 12795789312.0, "4880": 12795789312.0, "4885": 12795789312.0, "4890": 12795789312.0, "4895": 12795789312.0, "4900": 12795789312.0, "4905": 12795789312.0, "4910": 12795789312.0, "4915": 12795789312.0, "4920": 12795789312.0, "4925": 12795789312.0, "4930": 12795789312.0, "4935": 12795789312.0, "4940": 12795789312.0, "4945": 12795789312.0, "4950": 12795789312.0, "4955": 12795789312.0, "4960": 12795789312.0, "4965": 12795789312.0, "4970": 12795789312.0, "4975": 12795789312.0, "4980": 12795789312.0, "4985": 12795789312.0, "4990": 12795789312.0, "4995": 12795789312.0, "5000": 12795789312.0, "5005": 12795789312.0, "5010": 12795789312.0, "5015": 12795789312.0, "5020": 12795789312.0, "5025": 12795789312.0, "5030": 12795789312.0, "5035": 12795789312.0, "5040": 12795789312.0, "5045": 12795789312.0, "5050": 12795789312.0, "5055": 12795789312.0, "5060": 12795789312.0, "5065": 12795789312.0, "5070": 12795789312.0, "5075": 12795789312.0, "5080": 12795789312.0, "5085": 12795789312.0, "5090": 12795789312.0, "5095": 12795789312.0, "5100": 12795789312.0, "5105": 12795789312.0, "5110": 12795789312.0, "5115": 12795789312.0, "5120": 12795789312.0, "5125": 12795789312.0, "5130": 12795789312.0, "5135": 12795789312.0, "5140": 12795789312.0, "5145": 12795789312.0, "5150": 12795789312.0, "5155": 12795789312.0, "5160": 12795789312.0, "5165": 12795789312.0, "5170": 12795789312.0, "5175": 12795789312.0, "5180": 12795789312.0, "5185": 12795789312.0, "5190": 12795789312.0, "5195": 12795789312.0, "5200": 12795789312.0, "5205": 12795789312.0, "5210": 12795789312.0, "5215": 12795789312.0, "5220": 12795789312.0, "5225": 12795789312.0, "5230": 12795789312.0, "5235": 12795789312.0, "5240": 12795789312.0, "5245": 12795789312.0, "5250": 12795789312.0, "5255": 12795789312.0, "5260": 12795789312.0, "5265": 12795789312.0, "5270": 12795789312.0, "5275": 12795789312.0, "5280": 12795789312.0, "5285": 12795789312.0, "5290": 12795789312.0, "5295": 12795789312.0, "5300": 12795789312.0, "5305": 12795789312.0, "5310": 12795789312.0, "5315": 12795789312.0, "5320": 12795789312.0, "5325": 12795789312.0, "5330": 12795789312.0, "5335": 12795789312.0, "5340": 12795789312.0, "5345": 12795789312.0, "5350": 12795789312.0, "5355": 12795789312.0, "5360": 12795789312.0, "5365": 12795789312.0, "5370": 12795789312.0, "5375": 12795789312.0, "5380": 12795789312.0, "5385": 12795789312.0, "5390": 12795789312.0, "5395": 12795789312.0, "5400": 12795789312.0, "5405": 12795789312.0, "5410": 12795789312.0, "5415": 12795789312.0, "5420": 12795789312.0, "5425": 12795789312.0, "5430": 12795789312.0, "5435": 12795789312.0, "5440": 12795789312.0, "5445": 12795789312.0, "5450": 12795789312.0, "5455": 12795789312.0, "5460": 12795789312.0, "5465": 12795789312.0, "5470": 12795789312.0, "5475": 12795789312.0, "5480": 12795789312.0, "5485": 12795789312.0, "5490": 12795789312.0, "5495": 12795789312.0, "5500": 12795789312.0, "5505": 12795789312.0, "5510": 12795789312.0, "5515": 12795789312.0, "5520": 12795789312.0, "5525": 12795789312.0, "5530": 12795789312.0, "5535": 12795789312.0, "5540": 12795789312.0, "5545": 12795789312.0, "5550": 12795789312.0, "5555": 12795789312.0, "5560": 12795789312.0, "5565": 12795789312.0, "5570": 12795789312.0, "5575": 12795789312.0, "5580": 12795789312.0, "5585": 12795789312.0, "5590": 12795789312.0, "5595": 12795789312.0, "5600": 12795789312.0, "5605": 12795789312.0, "5610": 12795789312.0, "5615": 12795789312.0, "5620": 12795789312.0, "5625": 12795789312.0, "5630": 12795789312.0, "5635": 12795789312.0, "5640": 12795789312.0, "5645": 12795789312.0, "5650": 12795789312.0, "5655": 12795789312.0, "5660": 12795789312.0, "5665": 12795789312.0, "5670": 12795789312.0, "5675": 12795789312.0, "5680": 12795789312.0, "5685": 12795789312.0, "5690": 12795789312.0, "5695": 12795789312.0, "5700": 12795789312.0, "5705": 12795789312.0, "5710": 12795789312.0, "5715": 12795789312.0, "5720": 12795789312.0, "5725": 12795789312.0, "5730": 12795789312.0, "5735": 12795789312.0, "5740": 12795789312.0, "5745": 12795789312.0, "5750": 12795789312.0, "5755": 12795789312.0, "5760": 12795789312.0, "5765": 12795789312.0, "5770": 12795789312.0, "5775": 12795789312.0, "5780": 12795789312.0, "5785": 12795789312.0, "5790": 12795789312.0, "5795": 12795789312.0, "5800": 12795789312.0, "5805": 12795789312.0, "5810": 12795789312.0, "5815": 12795789312.0, "5820": 12795789312.0, "5825": 12795789312.0, "5830": 12795789312.0, "5835": 12795789312.0, "5840": 12795789312.0, "5845": 12795789312.0, "5850": 12795789312.0, "5855": 12795789312.0, "5860": 12795789312.0, "5865": 12795789312.0, "5870": 12795789312.0, "5875": 12795789312.0, "5880": 12795789312.0, "5885": 12795789312.0, "5890": 12795789312.0, "5895": 12795789312.0, "5900": 12795789312.0, "5905": 12795789312.0, "5910": 12795789312.0, "5915": 12795789312.0, "5920": 12795789312.0, "5925": 12795789312.0, "5930": 12795789312.0, "5935": 12795789312.0, "5940": 12795789312.0, "5945": 12795789312.0, "5950": 12795789312.0, "5955": 12795789312.0, "5960": 12795789312.0, "5965": 12795789312.0, "5970": 12795789312.0, "5975": 12795789312.0, "5980": 12795789312.0, "5985": 12795789312.0, "5990": 12795789312.0, "5995": 12795789312.0, "6000": 12795789312.0, "6005": 12795789312.0, "6010": 12795789312.0, "6015": 12795789312.0, "6020": 12795789312.0, "6025": 12795789312.0, "6030": 12795789312.0, "6035": 12795789312.0, "6040": 12795789312.0, "6045": 12795789312.0, "6050": 12795789312.0, "6055": 12795789312.0, "6060": 12795789312.0, "6065": 12795789312.0, "6070": 12795789312.0, "6075": 12795789312.0, "6080": 12795789312.0, "6085": 12795789312.0, "6090": 12795789312.0, "6095": 12795789312.0, "6100": 12795789312.0, "6105": 12795789312.0, "6110": 12795789312.0, "6115": 12795789312.0, "6120": 12795789312.0, "6125": 12795789312.0, "6130": 12795789312.0, "6135": 12795789312.0, "6140": 12795789312.0, "6145": 12795789312.0, "6150": 12795789312.0, "6155": 12795789312.0, "6160": 12795789312.0, "6165": 12795789312.0, "6170": 12795789312.0, "6175": 12795789312.0, "6180": 12795789312.0, "6185": 12795789312.0, "6190": 12795789312.0, "6195": 12795789312.0, "6200": 12795789312.0, "6205": 12795789312.0, "6210": 12795789312.0, "6215": 12795789312.0, "6220": 12795789312.0, "6225": 12795789312.0, "6230": 12795789312.0, "6235": 12795789312.0, "6240": 12795789312.0, "6245": 12795789312.0, "6250": 12795789312.0, "6255": 12795789312.0, "6260": 12795789312.0, "6265": 12795789312.0, "6270": 12795789312.0, "6275": 12795789312.0, "6280": 12795789312.0, "6285": 12795789312.0, "6290": 12795789312.0, "6295": 12795789312.0, "6300": 12795789312.0, "6305": 12795789312.0, "6310": 12795789312.0, "6315": 12795789312.0, "6320": 12795789312.0, "6325": 12795789312.0, "6330": 12795789312.0, "6335": 12795789312.0, "6340": 12795789312.0, "6345": 12795789312.0, "6350": 12795789312.0, "6355": 12795789312.0, "6360": 12795789312.0, "6365": 12795789312.0, "6370": 12795789312.0, "6375": 12795789312.0, "6380": 12795789312.0, "6385": 12795789312.0, "6390": 12795789312.0, "6395": 12795789312.0, "6400": 12795789312.0, "6405": 12795789312.0, "6410": 12795789312.0, "6415": 12795789312.0, "6420": 12795789312.0, "6425": 12795789312.0, "6430": 12795789312.0, "6435": 12795789312.0, "6440": 12795789312.0, "6445": 12795789312.0, "6450": 12795789312.0, "6455": 12795789312.0, "6460": 12795789312.0, "6465": 12795789312.0, "6470": 12795789312.0, "6475": 12795789312.0, "6480": 12795789312.0, "6485": 12795789312.0, "6490": 12795789312.0, "6495": 12795789312.0, "6500": 12795789312.0, "6505": 12795789312.0, "6510": 12795789312.0, "6515": 12795789312.0, "6520": 12795789312.0, "6525": 12795789312.0, "6530": 12795789312.0, "6535": 12795789312.0, "6540": 12795789312.0, "6545": 12795789312.0, "6550": 12795789312.0, "6555": 12795789312.0, "6560": 12795789312.0, "6565": 12795789312.0, "6570": 12795789312.0, "6575": 12795789312.0, "6580": 12795789312.0, "6585": 12795789312.0, "6590": 12795789312.0, "6595": 12795789312.0, "6600": 12795789312.0, "6605": 12795789312.0, "6610": 12795789312.0, "6615": 12795789312.0, "6620": 12795789312.0, "6625": 12795789312.0, "6630": 12795789312.0, "6635": 12795789312.0, "6640": 12795789312.0, "6645": 12795789312.0, "6650": 12795789312.0, "6655": 12795789312.0, "6660": 12795789312.0, "6665": 12795789312.0, "6670": 12795789312.0, "6675": 12795789312.0, "6680": 12795789312.0, "6685": 12795789312.0, "6690": 12795789312.0, "6695": 12795789312.0, "6700": 12795789312.0, "6705": 12795789312.0, "6710": 12795789312.0, "6715": 12795789312.0, "6720": 12795789312.0, "6725": 12795789312.0, "6730": 12795789312.0, "6735": 12795789312.0, "6740": 12795789312.0, "6745": 12795789312.0, "6750": 12795789312.0, "6755": 12795789312.0, "6760": 12795789312.0, "6765": 12795789312.0, "6770": 12795789312.0, "6775": 12795789312.0, "6780": 12795789312.0, "6785": 12795789312.0, "6790": 12795789312.0, "6795": 12795789312.0, "6800": 12795789312.0, "6805": 12795789312.0, "6810": 12795789312.0, "6815": 12795789312.0, "6820": 12795789312.0, "6825": 12795789312.0, "6830": 12795789312.0, "6835": 12795789312.0, "6840": 12795789312.0, "6845": 12795789312.0, "6850": 12795789312.0, "6855": 12795789312.0, "6860": 12795789312.0, "6865": 12795789312.0, "6870": 12795789312.0, "6875": 12795789312.0, "6880": 12795789312.0, "6885": 12795789312.0, "6890": 12795789312.0, "6895": 12795789312.0, "6900": 12795789312.0, "6905": 12795789312.0, "6910": 12795789312.0, "6915": 12795789312.0, "6920": 12795789312.0, "6925": 12795789312.0, "6930": 12795789312.0, "6935": 12795789312.0, "6940": 12795789312.0, "6945": 12795789312.0, "6950": 12795789312.0, "6955": 12795789312.0, "6960": 12795789312.0, "6965": 12795789312.0, "6970": 12795789312.0, "6975": 12795789312.0, "6980": 12795789312.0, "6985": 12795789312.0, "6990": 12795789312.0, "6995": 12795789312.0, "7000": 12795789312.0, "7005": 12795789312.0, "7010": 12795789312.0, "7015": 12795789312.0, "7020": 12795789312.0, "7025": 12795789312.0, "7030": 12795789312.0, "7035": 12795789312.0, "7040": 12795789312.0, "7045": 12795789312.0, "7050": 12795789312.0, "7055": 12795789312.0, "7060": 12795789312.0, "7065": 12795789312.0, "7070": 12795789312.0, "7075": 12795789312.0, "7080": 12795789312.0, "7085": 12795789312.0, "7090": 12795789312.0, "7095": 12795789312.0, "7100": 12795789312.0, "7105": 12795789312.0, "7110": 12795789312.0, "7115": 12795789312.0, "7120": 12795789312.0, "7125": 12795789312.0, "7130": 12795789312.0, "7135": 12795789312.0, "7140": 12795789312.0, "7145": 12795789312.0, "7150": 12795789312.0, "7155": 12795789312.0, "7160": 12795789312.0, "7165": 12795789312.0, "7170": 12795789312.0, "7175": 12795789312.0, "7180": 12795789312.0, "7185": 12795789312.0, "7190": 12795789312.0, "7195": 12795789312.0, "7200": 12795789312.0, "7205": 12795789312.0, "7210": 12795789312.0, "7215": 12795789312.0, "7220": 12795789312.0, "7225": 12795789312.0, "7230": 12795789312.0, "7235": 12795789312.0, "7240": 12795789312.0, "7245": 12795789312.0, "7250": 12795789312.0, "7255": 12795789312.0, "7260": 12795789312.0, "7265": 12795789312.0, "7270": 12795789312.0, "7275": 12795789312.0, "7280": 12795789312.0, "7285": 12795789312.0, "7290": 12795789312.0, "7295": 12795789312.0, "7300": 12795789312.0, "7305": 12795789312.0, "7310": 12795789312.0, "7315": 12795789312.0, "7320": 12795789312.0, "7325": 12795789312.0, "7330": 12795789312.0, "7335": 12795789312.0, "7340": 12795789312.0, "7345": 12795789312.0, "7350": 12795789312.0, "7355": 12795789312.0, "7360": 12795789312.0, "7365": 12795789312.0, "7370": 12795789312.0, "7375": 12795789312.0, "7380": 12795789312.0, "7385": 12795789312.0, "7390": 12795789312.0, "7395": 12795789312.0, "7400": 12795789312.0, "7405": 12795789312.0, "7410": 12795789312.0, "7415": 12795789312.0, "7420": 12795789312.0, "7425": 12795789312.0, "7430": 12795789312.0, "7435": 12795789312.0, "7440": 12795789312.0, "7445": 12795789312.0, "7450": 12795789312.0, "7455": 12795789312.0, "7460": 12795789312.0, "7465": 12795789312.0, "7470": 12795789312.0, "7475": 12795789312.0, "7480": 12795789312.0, "7485": 12795789312.0, "7490": 12795789312.0, "7495": 12795789312.0, "7500": 12795789312.0, "7505": 12795789312.0, "7510": 12795789312.0, "7515": 12795789312.0, "7520": 12795789312.0, "7525": 12795789312.0, "7530": 12795789312.0, "7535": 12795789312.0, "7540": 12795789312.0, "7545": 12795789312.0, "7550": 12795789312.0, "7555": 12795789312.0, "7560": 12795789312.0, "7565": 12795789312.0, "7570": 12795789312.0, "7575": 12795789312.0, "7580": 12795789312.0, "7585": 12795789312.0, "7590": 12795789312.0, "7595": 12795789312.0, "7600": 12795789312.0, "7605": 12795789312.0, "7610": 12795789312.0, "7615": 12795789312.0, "7620": 12795789312.0, "7625": 12795789312.0, "7630": 12795789312.0, "7635": 12795789312.0, "7640": 12795789312.0, "7645": 12795789312.0, "7650": 12795789312.0, "7655": 12795789312.0, "7660": 12795789312.0, "7665": 12795789312.0, "7670": 12795789312.0, "7675": 12795789312.0, "7680": 12795789312.0, "7685": 12795789312.0, "7690": 12795789312.0, "7695": 12795789312.0, "7700": 12795789312.0, "7705": 12795789312.0, "7710": 12795789312.0, "7715": 12795789312.0, "7720": 12795789312.0, "7725": 12795789312.0, "7730": 12795789312.0, "7735": 12795789312.0, "7740": 12795789312.0, "7745": 12795789312.0, "7750": 12795789312.0, "7755": 12795789312.0, "7760": 12795789312.0, "7765": 12795789312.0, "7770": 12795789312.0, "7775": 12795789312.0, "7780": 12795789312.0, "7785": 12795789312.0, "7790": 12795789312.0, "7795": 12795789312.0, "7800": 12795789312.0, "7805": 12795789312.0, "7810": 12795789312.0, "7815": 12795789312.0, "7820": 12795789312.0, "7825": 12795789312.0, "7830": 12795789312.0, "7835": 12795789312.0, "7840": 12795789312.0, "7845": 12795789312.0, "7850": 12795789312.0, "7855": 12795789312.0, "7860": 12795789312.0, "7865": 12795789312.0, "7870": 12795789312.0, "7875": 12795789312.0, "7880": 12795789312.0, "7885": 12795789312.0, "7890": 12795789312.0, "7895": 12795789312.0, "7900": 12795789312.0, "7905": 12795789312.0, "7910": 12795789312.0, "7915": 12795789312.0, "7920": 12795789312.0, "7925": 12795789312.0, "7930": 12795789312.0, "7935": 12795789312.0, "7940": 12795789312.0, "7945": 12795789312.0, "7950": 12795789312.0, "7955": 12795789312.0, "7960": 12795789312.0, "7965": 12795789312.0, "7970": 12795789312.0, "7975": 12795789312.0, "7980": 12795789312.0, "7985": 12795789312.0, "7990": 12795789312.0, "7995": 12795789312.0, "8000": 12795789312.0, "8005": 12795789312.0, "8010": 12795789312.0, "8015": 12795789312.0, "8020": 12795789312.0, "8025": 12795789312.0, "8030": 12795789312.0, "8035": 12795789312.0, "8040": 12795789312.0, "8045": 12795789312.0, "8050": 12795789312.0, "8055": 12795789312.0, "8060": 12795789312.0, "8065": 12795789312.0, "8070": 12795789312.0, "8075": 12795789312.0, "8080": 12795789312.0, "8085": 12795789312.0, "8090": 12795789312.0, "8095": 12795789312.0, "8100": 12795789312.0, "8105": 12795789312.0, "8110": 12795789312.0, "8115": 12795789312.0, "8120": 12795789312.0, "8125": 12795789312.0, "8130": 12795789312.0, "8135": 12795789312.0, "8140": 12795789312.0, "8145": 12795789312.0, "8150": 12795789312.0, "8155": 12795789312.0, "8160": 12795789312.0, "8165": 12795789312.0, "8170": 12795789312.0, "8175": 12795789312.0, "8180": 12795789312.0, "8185": 12795789312.0, "8190": 12795789312.0, "8195": 12795789312.0, "8200": 12795789312.0, "8205": 12795789312.0, "8210": 12795789312.0, "8215": 12795789312.0, "8220": 12795789312.0, "8225": 12795789312.0, "8230": 12795789312.0, "8235": 12795789312.0, "8240": 12795789312.0, "8245": 12795789312.0, "8250": 12795789312.0, "8255": 12795789312.0, "8260": 12795789312.0, "8265": 12795789312.0, "8270": 12795789312.0, "8275": 12795789312.0, "8280": 12795789312.0, "8285": 12795789312.0, "8290": 12795789312.0, "8295": 12795789312.0, "8300": 12795789312.0, "8305": 12795789312.0, "8310": 12795789312.0, "8315": 12795789312.0, "8320": 12795789312.0, "8325": 12795789312.0, "8330": 12795789312.0, "8335": 12795789312.0, "8340": 12795789312.0, "8345": 12795789312.0, "8350": 12795789312.0, "8355": 12795789312.0, "8360": 12795789312.0, "8365": 12795789312.0, "8370": 12795789312.0, "8375": 12795789312.0, "8380": 12795789312.0, "8385": 12795789312.0, "8390": 12795789312.0, "8395": 12795789312.0, "8400": 12795789312.0, "8405": 12795789312.0, "8410": 12795789312.0, "8415": 12795789312.0, "8420": 12795789312.0, "8425": 12795789312.0, "8430": 12795789312.0, "8435": 12795789312.0, "8440": 12795789312.0, "8445": 12795789312.0, "8450": 12795789312.0, "8455": 12795789312.0, "8460": 12795789312.0, "8465": 12795789312.0, "8470": 12795789312.0, "8475": 12795789312.0, "8480": 12795789312.0, "8485": 12795789312.0, "8490": 12795789312.0, "8495": 12795789312.0, "8500": 12795789312.0, "8505": 12795789312.0, "8510": 12795789312.0, "8515": 12795789312.0, "8520": 12795789312.0, "8525": 12795789312.0, "8530": 12795789312.0, "8535": 12795789312.0, "8540": 12795789312.0, "8545": 12795789312.0, "8550": 12795789312.0, "8555": 12795789312.0, "8560": 12795789312.0, "8565": 12795789312.0, "8570": 12795789312.0, "8575": 12795789312.0, "8580": 12795789312.0, "8585": 12795789312.0, "8590": 12795789312.0, "8595": 12795789312.0, "8600": 12795789312.0, "8605": 12795789312.0, "8610": 12795789312.0, "8615": 12795789312.0, "8620": 12795789312.0, "8625": 12795789312.0, "8630": 12795789312.0, "8635": 12795789312.0, "8640": 12795789312.0, "8645": 12795789312.0, "8650": 12795789312.0, "8655": 12795789312.0, "8660": 12795789312.0, "8665": 12795789312.0, "8670": 12795789312.0, "8675": 12795789312.0, "8680": 12795789312.0, "8685": 12795789312.0, "8690": 12795789312.0, "8695": 12795789312.0, "8700": 12795789312.0, "8705": 12795789312.0, "8710": 12795789312.0, "8715": 12795789312.0, "8720": 12795789312.0, "8725": 12795789312.0, "8730": 12795789312.0, "8735": 12795789312.0, "8740": 12795789312.0, "8745": 12795789312.0, "8750": 12795789312.0, "8755": 12795789312.0, "8760": 12795789312.0, "8765": 12795789312.0, "8770": 12795789312.0, "8775": 12795789312.0, "8780": 12795789312.0, "8785": 12795789312.0, "8790": 12795789312.0, "8795": 12795789312.0, "8800": 12795789312.0, "8805": 12795789312.0, "8810": 12795789312.0, "8815": 12795789312.0, "8820": 12795789312.0, "8825": 12795789312.0, "8830": 12795789312.0, "8835": 12795789312.0, "8840": 12795789312.0, "8845": 12795789312.0, "8850": 12795789312.0, "8855": 12795789312.0, "8860": 12795789312.0, "8865": 12795789312.0, "8870": 12795789312.0, "8875": 12795789312.0, "8880": 12795789312.0, "8885": 12795789312.0, "8890": 12795789312.0, "8895": 12795789312.0, "8900": 12795789312.0, "8905": 12795789312.0, "8910": 12795789312.0, "8915": 12795789312.0, "8920": 12795789312.0, "8925": 12795789312.0, "8930": 12795789312.0, "8935": 12795789312.0, "8940": 12795789312.0, "8945": 12795789312.0, "8950": 12795789312.0, "8955": 12795789312.0, "8960": 12795789312.0, "8965": 12795789312.0, "8970": 12795789312.0, "8975": 12795789312.0, "8980": 12795789312.0, "8985": 12795789312.0, "8990": 12795789312.0, "8995": 12795789312.0, "9000": 12795789312.0, "9005": 12795789312.0, "9010": 12795789312.0, "9015": 12795789312.0, "9020": 12795789312.0, "9025": 12795789312.0, "9030": 12795789312.0, "9035": 12795789312.0, "9040": 12795789312.0, "9045": 12795789312.0, "9050": 12795789312.0, "9055": 12795789312.0, "9060": 12795789312.0, "9065": 12795789312.0, "9070": 12795789312.0, "9075": 12795789312.0, "9080": 12795789312.0, "9085": 12795789312.0, "9090": 12795789312.0, "9095": 12795789312.0, "9100": 12795789312.0, "9105": 12795789312.0, "9110": 12795789312.0, "9115": 12795789312.0, "9120": 12795789312.0, "9125": 12795789312.0, "9130": 12795789312.0, "9135": 12795789312.0, "9140": 12795789312.0, "9145": 12795789312.0, "9150": 12795789312.0, "9155": 12795789312.0, "9160": 12795789312.0, "9165": 12795789312.0, "9170": 12795789312.0, "9175": 12795789312.0, "9180": 12795789312.0, "9185": 12795789312.0, "9190": 12795789312.0, "9195": 12795789312.0, "9200": 12795789312.0, "9205": 12795789312.0, "9210": 12795789312.0, "9215": 12795789312.0, "9220": 12795789312.0, "9225": 12795789312.0, "9230": 12795789312.0, "9235": 12795789312.0, "9240": 12795789312.0, "9245": 12795789312.0, "9250": 12795789312.0, "9255": 12795789312.0, "9260": 12795789312.0, "9265": 12795789312.0, "9270": 12795789312.0, "9275": 12795789312.0, "9280": 12795789312.0, "9285": 12795789312.0, "9290": 12795789312.0, "9295": 12795789312.0, "9300": 12795789312.0, "9305": 12795789312.0, "9310": 12795789312.0, "9315": 12795789312.0, "9320": 12795789312.0, "9325": 12795789312.0, "9330": 12795789312.0, "9335": 12795789312.0, "9340": 12795789312.0, "9345": 12795789312.0, "9350": 12795789312.0, "9355": 12795789312.0, "9360": 12795789312.0, "9365": 12795789312.0, "9370": 12795789312.0, "9375": 12795789312.0, "9380": 12795789312.0, "9385": 12795789312.0, "9390": 12795789312.0, "9395": 12795789312.0, "9400": 12795789312.0, "9405": 12795789312.0, "9410": 12795789312.0, "9415": 12795789312.0, "9420": 12795789312.0, "9425": 12795789312.0, "9430": 12795789312.0, "9435": 12795789312.0, "9440": 12795789312.0, "9445": 12795789312.0, "9450": 12795789312.0, "9455": 12795789312.0, "9460": 12795789312.0, "9465": 12795789312.0, "9470": 12795789312.0, "9475": 12795789312.0, "9480": 12795789312.0, "9485": 12795789312.0, "9490": 12795789312.0, "9495": 12795789312.0, "9500": 12795789312.0, "9505": 12795789312.0, "9510": 12795789312.0, "9515": 12795789312.0, "9520": 12795789312.0, "9525": 12795789312.0, "9530": 12795789312.0, "9535": 12795789312.0, "9540": 12795789312.0, "9545": 12795789312.0, "9550": 12795789312.0, "9555": 12795789312.0, "9560": 12795789312.0, "9565": 12795789312.0, "9570": 12795789312.0, "9575": 12795789312.0, "9580": 12795789312.0, "9585": 12795789312.0, "9590": 12795789312.0, "9595": 12795789312.0, "9600": 12795789312.0, "9605": 12795789312.0, "9610": 12795789312.0, "9615": 12795789312.0, "9620": 12795789312.0, "9625": 12795789312.0, "9630": 12795789312.0, "9635": 12795789312.0, "9640": 12795789312.0, "9645": 12795789312.0, "9650": 12795789312.0, "9655": 12795789312.0, "9660": 12795789312.0, "9665": 12795789312.0, "9670": 12795789312.0, "9675": 12795789312.0, "9680": 12795789312.0, "9685": 12795789312.0, "9690": 12795789312.0, "9695": 12795789312.0, "9700": 12795789312.0, "9705": 12795789312.0, "9710": 12795789312.0, "9715": 12795789312.0, "9720": 12795789312.0, "9725": 12795789312.0, "9730": 12795789312.0, "9735": 12795789312.0, "9740": 12795789312.0, "9745": 12795789312.0, "9750": 12795789312.0, "9755": 12795789312.0, "9760": 12795789312.0, "9765": 12795789312.0, "9770": 12795789312.0, "9775": 12795789312.0, "9780": 12795789312.0, "9785": 12795789312.0, "9790": 12795789312.0, "9795": 12795789312.0, "9800": 12795789312.0, "9805": 12795789312.0, "9810": 12795789312.0, "9815": 12795789312.0, "9820": 12795789312.0, "9825": 12795789312.0, "9830": 12795789312.0, "9835": 12795789312.0, "9840": 12795789312.0, "9845": 12795789312.0, "9850": 12795789312.0, "9855": 12795789312.0, "9860": 12795789312.0, "9865": 12795789312.0, "9870": 12795789312.0, "9875": 12795789312.0, "9880": 12795789312.0, "9885": 12795789312.0, "9890": 12795789312.0, "9895": 12795789312.0, "9900": 12795789312.0, "9905": 12795789312.0, "9910": 12795789312.0, "9915": 12795789312.0, "9920": 12795789312.0, "9925": 12795789312.0, "9930": 12795789312.0, "9935": 12795789312.0, "9940": 12795789312.0, "9945": 12795789312.0, "9950": 12795789312.0, "9955": 12795789312.0, "9960": 12795789312.0, "9965": 12795789312.0, "9970": 12795789312.0, "9975": 12795789312.0, "9980": 12795789312.0, "9985": 12795789312.0, "9990": 12795789312.0, "9995": 12795789312.0, "10000": 12795789312.0, "10005": 12795789312.0, "10010": 12795789312.0, "10015": 12795789312.0, "10020": 12795789312.0, "10025": 12795789312.0, "10030": 12795789312.0, "10035": 12795789312.0, "10040": 12795789312.0, "10045": 12795789312.0, "10050": 12795789312.0, "10055": 12795789312.0, "10060": 12795789312.0, "10065": 12795789312.0, "10070": 12795789312.0, "10075": 12795789312.0, "10080": 12795789312.0, "10085": 12795789312.0, "10090": 12795789312.0, "10095": 12795789312.0, "10100": 12795789312.0, "10105": 12795789312.0, "10110": 12795789312.0, "10115": 12795789312.0, "10120": 12795789312.0, "10125": 12795789312.0, "10130": 12795789312.0, "10135": 12795789312.0, "10140": 12795789312.0, "10145": 12795789312.0, "10150": 12795789312.0, "10155": 12795789312.0, "10160": 12795789312.0, "10165": 12795789312.0, "10170": 12795789312.0, "10175": 12795789312.0, "10180": 12795789312.0, "10185": 12795789312.0, "10190": 12795789312.0, "10195": 12795789312.0, "10200": 12795789312.0, "10205": 12795789312.0, "10210": 12795789312.0, "10215": 12795789312.0, "10220": 12795789312.0, "10225": 12795789312.0, "10230": 12795789312.0, "10235": 12795789312.0, "10240": 12795789312.0, "10245": 12795789312.0, "10250": 12795789312.0, "10255": 12795789312.0, "10260": 12795789312.0, "10265": 12795789312.0, "10270": 12795789312.0, "10275": 12795789312.0, "10280": 12795789312.0, "10285": 12795789312.0, "10290": 12795789312.0, "10295": 12795789312.0, "10300": 12795789312.0, "10305": 12795789312.0, "10310": 12795789312.0, "10315": 12795789312.0, "10320": 12795789312.0, "10325": 12795789312.0, "10330": 12795789312.0, "10335": 12795789312.0, "10340": 12795789312.0, "10345": 12795789312.0, "10350": 12795789312.0, "10355": 12795789312.0, "10360": 12795789312.0, "10365": 12795789312.0, "10370": 12795789312.0, "10375": 12795789312.0, "10380": 12795789312.0, "10385": 12795789312.0, "10390": 12795789312.0, "10395": 12795789312.0, "10400": 12795789312.0, "10405": 12795789312.0, "10410": 12795789312.0, "10415": 12795789312.0, "10420": 12795789312.0, "10425": 12795789312.0, "10430": 12795789312.0, "10435": 12795789312.0, "10440": 12795789312.0, "10445": 12795789312.0, "10450": 12795789312.0, "10455": 12795789312.0, "10460": 12795789312.0, "10465": 12795789312.0, "10470": 12795789312.0, "10475": 12795789312.0, "10480": 12795789312.0, "10485": 12795789312.0, "10490": 12795789312.0, "10495": 12795789312.0, "10500": 12795789312.0, "10505": 12795789312.0, "10510": 12795789312.0, "10515": 12795789312.0, "10520": 12795789312.0, "10525": 12795789312.0, "10530": 12795789312.0, "10535": 12795789312.0, "10540": 12795789312.0, "10545": 12795789312.0, "10550": 12795789312.0, "10555": 12795789312.0, "10560": 12795789312.0, "10565": 12795789312.0, "10570": 12795789312.0, "10575": 12795789312.0, "10580": 12795789312.0, "10585": 12795789312.0, "10590": 12795789312.0, "10595": 12795789312.0, "10600": 12795789312.0, "10605": 12795789312.0, "10610": 12795789312.0, "10615": 12795789312.0, "10620": 12795789312.0, "10625": 12795789312.0, "10630": 12795789312.0, "10635": 12795789312.0, "10640": 12795789312.0, "10645": 12795789312.0, "10650": 12795789312.0, "10655": 12795789312.0, "10660": 12795789312.0, "10665": 12795789312.0, "10670": 12795789312.0, "10675": 12795789312.0, "10680": 12795789312.0, "10685": 12795789312.0, "10690": 12795789312.0, "10695": 12795789312.0, "10700": 12795789312.0, "10705": 12795789312.0, "10710": 12795789312.0, "10715": 12795789312.0, "10720": 12795789312.0, "10725": 12795789312.0, "10730": 12795789312.0, "10735": 12795789312.0, "10740": 12795789312.0, "10745": 12795789312.0, "10750": 12795789312.0, "10755": 12795789312.0, "10760": 12795789312.0, "10765": 12795789312.0, "10770": 12795789312.0, "10775": 12795789312.0, "10780": 12795789312.0, "10785": 12795789312.0, "10790": 12795789312.0, "10795": 12795789312.0, "10800": 12795789312.0, "10805": 12795789312.0, "10810": 12795789312.0, "10815": 12795789312.0, "10820": 12795789312.0, "10825": 12795789312.0, "10830": 12795789312.0, "10835": 12795789312.0, "10840": 12795789312.0, "10845": 12795789312.0, "10850": 12795789312.0, "10855": 12795789312.0, "10860": 12795789312.0, "10865": 12795789312.0, "10870": 12795789312.0, "10875": 12795789312.0, "10880": 12795789312.0, "10885": 12795789312.0, "10890": 12795789312.0, "10895": 12795789312.0, "10900": 12795789312.0, "10905": 12795789312.0, "10910": 12795789312.0, "10915": 12795789312.0, "10920": 12795789312.0, "10925": 12795789312.0, "10930": 12795789312.0, "10935": 12795789312.0, "10940": 12795789312.0, "10945": 12795789312.0, "10950": 12795789312.0, "10955": 12795789312.0, "10960": 12795789312.0, "10965": 12795789312.0, "10970": 12795789312.0, "10975": 12795789312.0, "10980": 12795789312.0, "10985": 12795789312.0, "10990": 12795789312.0, "10995": 12795789312.0, "11000": 12795789312.0, "11005": 12795789312.0, "11010": 12795789312.0, "11015": 12795789312.0, "11020": 12795789312.0, "11025": 12795789312.0, "11030": 12795789312.0, "11035": 12795789312.0, "11040": 12795789312.0, "11045": 12795789312.0, "11050": 12795789312.0, "11055": 12795789312.0, "11060": 12795789312.0, "11065": 12795789312.0, "11070": 12795789312.0, "11075": 12795789312.0, "11080": 12795789312.0, "11085": 12795789312.0, "11090": 12795789312.0, "11095": 12795789312.0, "11100": 12795789312.0, "11105": 12795789312.0, "11110": 12795789312.0, "11115": 12795789312.0, "11120": 12795789312.0, "11125": 12795789312.0, "11130": 12795789312.0, "11135": 12795789312.0, "11140": 12795789312.0, "11145": 12795789312.0, "11150": 12795789312.0, "11155": 12795789312.0, "11160": 12795789312.0, "11165": 12795789312.0, "11170": 12795789312.0, "11175": 12795789312.0, "11180": 12795789312.0, "11185": 12795789312.0, "11190": 12795789312.0, "11195": 12795789312.0, "11200": 12795789312.0, "11205": 12795789312.0, "11210": 12795789312.0, "11215": 12795789312.0, "11220": 12795789312.0, "11225": 12795789312.0, "11230": 12795789312.0, "11235": 12795789312.0, "11240": 12795789312.0, "11245": 12795789312.0, "11250": 12795789312.0, "11255": 12795789312.0, "11260": 12795789312.0, "11265": 12795789312.0, "11270": 12795789312.0, "11275": 12795789312.0, "11280": 12795789312.0, "11285": 12795789312.0, "11290": 12795789312.0, "11295": 12795789312.0, "11300": 12795789312.0, "11305": 12795789312.0, "11310": 12795789312.0, "11315": 12795789312.0, "11320": 12795789312.0, "11325": 12795789312.0, "11330": 12795789312.0, "11335": 12795789312.0, "11340": 12795789312.0, "11345": 12795789312.0, "11350": 12795789312.0, "11355": 12795789312.0, "11360": 12795789312.0, "11365": 12795789312.0, "11370": 12795789312.0, "11375": 12795789312.0, "11380": 12795789312.0, "11385": 12795789312.0, "11390": 12795789312.0, "11395": 12795789312.0, "11400": 12795789312.0, "11405": 12795789312.0, "11410": 12795789312.0, "11415": 12795789312.0, "11420": 12795789312.0, "11425": 12795789312.0, "11430": 12795789312.0, "11435": 12795789312.0, "11440": 12795789312.0, "11445": 12795789312.0, "11450": 12795789312.0, "11455": 12795789312.0, "11460": 12795789312.0, "11465": 12795789312.0, "11470": 12795789312.0, "11475": 12795789312.0, "11480": 12795789312.0, "11485": 12795789312.0, "11490": 12795789312.0, "11495": 12795789312.0, "11500": 12795789312.0, "11505": 12795789312.0, "11510": 12795789312.0, "11515": 12795789312.0, "11520": 12795789312.0, "11525": 12795789312.0, "11530": 12795789312.0, "11535": 12795789312.0, "11540": 12795789312.0, "11545": 12795789312.0, "11550": 12795789312.0, "11555": 12795789312.0, "11560": 12795789312.0, "11565": 12795789312.0, "11570": 12795789312.0, "11575": 12795789312.0, "11580": 12795789312.0, "11585": 12795789312.0, "11590": 12795789312.0, "11595": 12795789312.0, "11600": 12795789312.0, "11605": 12795789312.0, "11610": 12795789312.0, "11615": 12795789312.0, "11620": 12795789312.0, "11625": 12795789312.0, "11630": 12795789312.0, "11635": 12795789312.0, "11640": 12795789312.0, "11645": 12795789312.0, "11650": 12795789312.0, "11655": 12795789312.0, "11660": 12795789312.0, "11665": 12795789312.0, "11670": 12795789312.0, "11675": 12795789312.0, "11680": 12795789312.0, "11685": 12795789312.0, "11690": 12795789312.0, "11695": 12795789312.0, "11700": 12795789312.0, "11705": 12795789312.0, "11710": 12795789312.0, "11715": 12795789312.0, "11720": 12795789312.0, "11725": 12795789312.0, "11730": 12795789312.0, "11735": 12795789312.0, "11740": 12795789312.0, "11745": 12795789312.0, "11750": 12795789312.0, "11755": 12795789312.0, "11760": 12795789312.0, "11765": 12795789312.0, "11770": 12795789312.0, "11775": 12795789312.0, "11780": 12795789312.0, "11785": 12795789312.0, "11790": 12795789312.0, "11795": 12795789312.0, "11800": 12795789312.0, "11805": 12795789312.0, "11810": 12795789312.0, "11815": 12795789312.0, "11820": 12795789312.0, "11825": 12795789312.0, "11830": 12795789312.0, "11835": 12795789312.0, "11840": 12795789312.0, "11845": 12795789312.0, "11850": 12795789312.0, "11855": 12795789312.0, "11860": 12795789312.0, "11865": 12795789312.0, "11870": 12795789312.0, "11875": 12795789312.0, "11880": 12795789312.0, "11885": 12795789312.0, "11890": 12795789312.0, "11895": 12795789312.0, "11900": 12795789312.0, "11905": 12795789312.0, "11910": 12795789312.0, "11915": 12795789312.0, "11920": 12795789312.0, "11925": 12795789312.0, "11930": 12795789312.0, "11935": 12795789312.0, "11940": 12795789312.0, "11945": 12795789312.0, "11950": 12795789312.0, "11955": 12795789312.0, "11960": 12795789312.0, "11965": 12795789312.0, "11970": 12795789312.0, "11975": 12795789312.0, "11980": 12795789312.0, "11985": 12795789312.0, "11990": 12795789312.0, "11995": 12795789312.0, "12000": 12795789312.0, "12005": 12795789312.0, "12010": 12795789312.0, "12015": 12795789312.0, "12020": 12795789312.0, "12025": 12795789312.0, "12030": 12795789312.0, "12035": 12795789312.0, "12040": 12795789312.0, "12045": 12795789312.0, "12050": 12795789312.0, "12055": 12795789312.0, "12060": 12795789312.0, "12065": 12795789312.0, "12070": 12795789312.0, "12075": 12795789312.0, "12080": 12795789312.0, "12085": 12795789312.0, "12090": 12795789312.0, "12095": 12795789312.0, "12100": 12795789312.0, "12105": 12795789312.0, "12110": 12795789312.0, "12115": 12795789312.0, "12120": 12795789312.0, "12125": 12795789312.0, "12130": 12795789312.0, "12135": 12795789312.0, "12140": 12795789312.0, "12145": 12795789312.0, "12150": 12795789312.0, "12155": 12795789312.0, "12160": 12795789312.0, "12165": 12795789312.0, "12170": 12795789312.0, "12175": 12795789312.0, "12180": 12795789312.0, "12185": 12795789312.0, "12190": 12795789312.0, "12195": 12795789312.0, "12200": 12795789312.0, "12205": 12795789312.0, "12210": 12795789312.0, "12215": 12795789312.0, "12220": 12795789312.0, "12225": 12795789312.0, "12230": 12795789312.0, "12235": 12795789312.0, "12240": 12795789312.0, "12245": 12795789312.0, "12250": 12795789312.0, "12255": 12795789312.0, "12260": 12795789312.0, "12265": 12795789312.0, "12270": 12795789312.0, "12275": 12795789312.0, "12280": 12795789312.0, "12285": 12795789312.0, "12290": 12795789312.0, "12295": 12795789312.0, "12300": 12795789312.0, "12305": 12795789312.0, "12310": 12795789312.0, "12315": 12795789312.0, "12320": 12795789312.0, "12325": 12795789312.0, "12330": 12795789312.0, "12335": 12795789312.0, "12340": 12795789312.0, "12345": 12795789312.0, "12350": 12795789312.0, "12355": 12795789312.0, "12360": 12795789312.0, "12365": 12795789312.0, "12370": 12795789312.0, "12375": 12795789312.0, "12380": 12795789312.0, "12385": 12795789312.0, "12390": 12795789312.0, "12395": 12795789312.0, "12400": 12795789312.0, "12405": 12795789312.0, "12410": 12795789312.0, "12415": 12795789312.0, "12420": 12795789312.0, "12425": 12795789312.0, "12430": 12795789312.0, "12435": 12795789312.0, "12440": 12795789312.0, "12445": 12795789312.0, "12450": 12795789312.0, "12455": 12795789312.0, "12460": 12795789312.0, "12465": 12795789312.0, "12470": 12795789312.0, "12475": 12795789312.0, "12480": 12795789312.0, "12485": 12795789312.0, "12490": 12795789312.0, "12495": 12795789312.0, "12500": 12795789312.0, "12505": 12795789312.0, "12510": 12795789312.0, "12515": 12795789312.0, "12520": 12795789312.0, "12525": 12795789312.0, "12530": 12795789312.0, "12535": 12795789312.0, "12540": 12795789312.0, "12545": 12795789312.0, "12550": 12795789312.0, "12555": 12795789312.0, "12560": 12795789312.0, "12565": 12795789312.0, "12570": 12795789312.0, "12575": 12795789312.0, "12580": 12795789312.0, "12585": 12795789312.0, "12590": 12795789312.0, "12595": 12795789312.0, "12600": 12795789312.0, "12605": 12795789312.0, "12610": 12795789312.0, "12615": 12795789312.0, "12620": 12795789312.0, "12625": 12795789312.0, "12630": 12795789312.0, "12635": 12795789312.0, "12640": 12795789312.0, "12645": 12795789312.0, "12650": 12795789312.0, "12655": 12795789312.0, "12660": 12795789312.0, "12665": 12795789312.0, "12670": 12795789312.0, "12675": 12795789312.0, "12680": 12795789312.0, "12685": 12795789312.0, "12690": 12795789312.0, "12695": 12795789312.0, "12700": 12795789312.0, "12705": 12795789312.0, "12710": 12795789312.0, "12715": 12795789312.0, "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}, "mem-max-allocated-bytes": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": 27991298048.0, "5": 28489385984.0, "10": 28489385984.0, "15": 28489385984.0, "20": 28489385984.0, "25": 28489385984.0, "30": 28489385984.0, "35": 28489385984.0, "40": 28489385984.0, "45": 28489385984.0, "50": 28489385984.0, "55": 28489385984.0, "60": 28489385984.0, "65": 28489385984.0, "70": 28489385984.0, "75": 28489385984.0, "80": 28489385984.0, "85": 28489385984.0, "90": 28489385984.0, "95": 28489385984.0, "100": 28489385984.0, "105": 28489385984.0, "110": 28489385984.0, "115": 28489385984.0, "120": 28489385984.0, "125": 28489385984.0, "130": 28489385984.0, "135": 28489385984.0, "140": 28489385984.0, "145": 28489385984.0, "150": 28489385984.0, "155": 28489385984.0, "160": 28489385984.0, "165": 28489385984.0, "170": 28489385984.0, "175": 28489385984.0, "180": 28489385984.0, "185": 28489385984.0, "190": 28489385984.0, "195": 28489385984.0, "200": 28489385984.0, "205": 28489385984.0, "210": 28489385984.0, "215": 28489385984.0, "220": 28489385984.0, "225": 28489385984.0, "230": 28489385984.0, "235": 28489385984.0, "240": 28489385984.0, "245": 28489385984.0, "250": 28489385984.0, "255": 28489385984.0, "260": 28489385984.0, "265": 28489385984.0, "270": 28489385984.0, "275": 28489385984.0, "280": 28489385984.0, "285": 28489385984.0, "290": 28489385984.0, "295": 28489385984.0, "300": 28489385984.0, "305": 28489385984.0, "310": 28489385984.0, "315": 28489385984.0, "320": 28489385984.0, "325": 28489385984.0, "330": 28489385984.0, "335": 28489385984.0, "340": 28489385984.0, "345": 28489385984.0, "350": 28489385984.0, "355": 28489385984.0, "360": 28489385984.0, "365": 28489385984.0, "370": 28489385984.0, "375": 28489385984.0, "380": 28489385984.0, "385": 28489385984.0, "390": 28489385984.0, "395": 28489385984.0, "400": 28489385984.0, "405": 28489385984.0, "410": 28489385984.0, "415": 28489385984.0, "420": 28489385984.0, "425": 28489385984.0, "430": 28489385984.0, "435": 28489385984.0, "440": 28489385984.0, "445": 28489385984.0, "450": 28489385984.0, "455": 28489385984.0, "460": 28489385984.0, "465": 28489385984.0, "470": 28489385984.0, "475": 28489385984.0, "480": 28489385984.0, "485": 28489385984.0, "490": 28489385984.0, "495": 28489385984.0, "500": 28489385984.0, "505": 28489385984.0, "510": 28489385984.0, "515": 28489385984.0, "520": 28489385984.0, "525": 28489385984.0, "530": 28489385984.0, "535": 28489385984.0, "540": 28489385984.0, "545": 28489385984.0, "550": 28489385984.0, "555": 28489385984.0, "560": 28489385984.0, "565": 28489385984.0, "570": 28489385984.0, "575": 28489385984.0, "580": 28489385984.0, "585": 28489385984.0, "590": 28489385984.0, "595": 28489385984.0, "600": 28489385984.0, "605": 28489385984.0, "610": 28489385984.0, "615": 28489385984.0, "620": 28489385984.0, "625": 28489385984.0, "630": 28489385984.0, "635": 28489385984.0, "640": 28489385984.0, "645": 28489385984.0, "650": 28489385984.0, "655": 28489385984.0, "660": 28489385984.0, "665": 28489385984.0, "670": 28489385984.0, "675": 28489385984.0, "680": 28489385984.0, "685": 28489385984.0, "690": 28489385984.0, "695": 28489385984.0, "700": 28489385984.0, "705": 28489385984.0, "710": 28489385984.0, "715": 28489385984.0, "720": 28489385984.0, "725": 28489385984.0, "730": 28489385984.0, "735": 28489385984.0, "740": 28489385984.0, "745": 28489385984.0, "750": 28489385984.0, "755": 28489385984.0, "760": 28489385984.0, "765": 28489385984.0, "770": 28489385984.0, "775": 28489385984.0, "780": 28489385984.0, "785": 28489385984.0, "790": 28489385984.0, "795": 28489385984.0, "800": 28489385984.0, "805": 28489385984.0, "810": 28489385984.0, "815": 28489385984.0, "820": 28489385984.0, "825": 28489385984.0, "830": 28489385984.0, "835": 28489385984.0, "840": 28489385984.0, "845": 28489385984.0, "850": 28489385984.0, "855": 28489385984.0, "860": 28489385984.0, "865": 28489385984.0, "870": 28489385984.0, "875": 28489385984.0, "880": 28489385984.0, "885": 28489385984.0, "890": 28489385984.0, "895": 28489385984.0, "900": 28489385984.0, "905": 28489385984.0, "910": 28489385984.0, "915": 28489385984.0, "920": 28489385984.0, "925": 28489385984.0, "930": 28489385984.0, "935": 28489385984.0, "940": 28489385984.0, "945": 28489385984.0, "950": 28489385984.0, "955": 28489385984.0, "960": 28489385984.0, "965": 28489385984.0, "970": 28489385984.0, "975": 28489385984.0, "980": 28489385984.0, "985": 28489385984.0, "990": 28489385984.0, "995": 28489385984.0, "1000": 28489385984.0, "1005": 28489385984.0, "1010": 28489385984.0, "1015": 28489385984.0, "1020": 28489385984.0, "1025": 28489385984.0, "1030": 28489385984.0, "1035": 28489385984.0, "1040": 28489385984.0, "1045": 28489385984.0, "1050": 28489385984.0, "1055": 28489385984.0, "1060": 28489385984.0, "1065": 28489385984.0, "1070": 28489385984.0, "1075": 28489385984.0, "1080": 28489385984.0, "1085": 28489385984.0, "1090": 28489385984.0, "1095": 28489385984.0, "1100": 28489385984.0, "1105": 28489385984.0, "1110": 28489385984.0, "1115": 28489385984.0, "1120": 28489385984.0, "1125": 28489385984.0, "1130": 28489385984.0, "1135": 28489385984.0, "1140": 28489385984.0, "1145": 28489385984.0, "1150": 28489385984.0, "1155": 28489385984.0, "1160": 28489385984.0, "1165": 28489385984.0, "1170": 28489385984.0, "1175": 28489385984.0, "1180": 28489385984.0, "1185": 28489385984.0, "1190": 28489385984.0, "1195": 28489385984.0, "1200": 28489385984.0, "1205": 28489385984.0, "1210": 28489385984.0, "1215": 28489385984.0, "1220": 28489385984.0, "1225": 28489385984.0, "1230": 28489385984.0, "1235": 28489385984.0, "1240": 28489385984.0, "1245": 28489385984.0, "1250": 28489385984.0, "1255": 28489385984.0, "1260": 28489385984.0, "1265": 28489385984.0, "1270": 28489385984.0, "1275": 28489385984.0, "1280": 28489385984.0, "1285": 28489385984.0, "1290": 28489385984.0, "1295": 28489385984.0, "1300": 28489385984.0, "1305": 28489385984.0, "1310": 28489385984.0, "1315": 28489385984.0, "1320": 28489385984.0, "1325": 28489385984.0, "1330": 28489385984.0, "1335": 28489385984.0, "1340": 28489385984.0, "1345": 28489385984.0, "1350": 28489385984.0, "1355": 28489385984.0, "1360": 28489385984.0, "1365": 28489385984.0, "1370": 28489385984.0, "1375": 28489385984.0, "1380": 28489385984.0, "1385": 28489385984.0, "1390": 28489385984.0, "1395": 28489385984.0, "1400": 28489385984.0, "1405": 28489385984.0, "1410": 28489385984.0, "1415": 28489385984.0, "1420": 28489385984.0, "1425": 28489385984.0, "1430": 28489385984.0, "1435": 28489385984.0, "1440": 28489385984.0, "1445": 28489385984.0, "1450": 28489385984.0, "1455": 28489385984.0, "1460": 28489385984.0, "1465": 28489385984.0, "1470": 28489385984.0, "1475": 28489385984.0, "1480": 28489385984.0, "1485": 28489385984.0, "1490": 28489385984.0, "1495": 28489385984.0, "1500": 28489385984.0, "1505": 28489385984.0, "1510": 28489385984.0, "1515": 28489385984.0, "1520": 28489385984.0, "1525": 28489385984.0, "1530": 28489385984.0, "1535": 28489385984.0, "1540": 28489385984.0, "1545": 28489385984.0, "1550": 28489385984.0, "1555": 28489385984.0, "1560": 28489385984.0, "1565": 28489385984.0, "1570": 28489385984.0, "1575": 28489385984.0, "1580": 28489385984.0, "1585": 28489385984.0, "1590": 28489385984.0, "1595": 28489385984.0, "1600": 28489385984.0, "1605": 28489385984.0, "1610": 28489385984.0, "1615": 28489385984.0, "1620": 28489385984.0, "1625": 28489385984.0, "1630": 28489385984.0, "1635": 28489385984.0, "1640": 28489385984.0, "1645": 28489385984.0, "1650": 28489385984.0, "1655": 28489385984.0, "1660": 28489385984.0, "1665": 28489385984.0, "1670": 28489385984.0, "1675": 28489385984.0, "1680": 28489385984.0, "1685": 28489385984.0, "1690": 28489385984.0, "1695": 28489385984.0, "1700": 28489385984.0, "1705": 28489385984.0, "1710": 28489385984.0, "1715": 28489385984.0, "1720": 28489385984.0, "1725": 28489385984.0, "1730": 28489385984.0, "1735": 28489385984.0, "1740": 28489385984.0, "1745": 28489385984.0, "1750": 28489385984.0, "1755": 28489385984.0, "1760": 28489385984.0, "1765": 28489385984.0, "1770": 28489385984.0, "1775": 28489385984.0, "1780": 28489385984.0, "1785": 28489385984.0, "1790": 28489385984.0, "1795": 28489385984.0, "1800": 28489385984.0, "1805": 28489385984.0, "1810": 28489385984.0, "1815": 28489385984.0, "1820": 28489385984.0, "1825": 28489385984.0, "1830": 28489385984.0, "1835": 28489385984.0, "1840": 28489385984.0, "1845": 28489385984.0, "1850": 28489385984.0, "1855": 28489385984.0, "1860": 28489385984.0, "1865": 28489385984.0, "1870": 28489385984.0, "1875": 28489385984.0, "1880": 28489385984.0, "1885": 28489385984.0, "1890": 28489385984.0, "1895": 28489385984.0, "1900": 28489385984.0, "1905": 28489385984.0, "1910": 28489385984.0, "1915": 28489385984.0, "1920": 28489385984.0, "1925": 28489385984.0, "1930": 28489385984.0, "1935": 28489385984.0, "1940": 28489385984.0, "1945": 28489385984.0, "1950": 28489385984.0, "1955": 28489385984.0, "1960": 28489385984.0, "1965": 28489385984.0, "1970": 28489385984.0, "1975": 28489385984.0, "1980": 28489385984.0, "1985": 28489385984.0, "1990": 28489385984.0, "1995": 28489385984.0, "2000": 28489385984.0, "2005": 28489385984.0, "2010": 28489385984.0, "2015": 28489385984.0, "2020": 28489385984.0, "2025": 28489385984.0, "2030": 28489385984.0, "2035": 28489385984.0, "2040": 28489385984.0, "2045": 28489385984.0, "2050": 28489385984.0, "2055": 28489385984.0, "2060": 28489385984.0, "2065": 28489385984.0, "2070": 28489385984.0, "2075": 28489385984.0, "2080": 28489385984.0, "2085": 28489385984.0, "2090": 28489385984.0, "2095": 28489385984.0, "2100": 28489385984.0, "2105": 28489385984.0, "2110": 28489385984.0, "2115": 28489385984.0, "2120": 28489385984.0, "2125": 28489385984.0, "2130": 28489385984.0, "2135": 28489385984.0, "2140": 28489385984.0, "2145": 28489385984.0, "2150": 28489385984.0, "2155": 28489385984.0, "2160": 28489385984.0, "2165": 28489385984.0, "2170": 28489385984.0, "2175": 28489385984.0, "2180": 28489385984.0, "2185": 28489385984.0, "2190": 28489385984.0, "2195": 28489385984.0, "2200": 28489385984.0, "2205": 28489385984.0, "2210": 28489385984.0, "2215": 28489385984.0, "2220": 28489385984.0, "2225": 28489385984.0, "2230": 28489385984.0, "2235": 28489385984.0, "2240": 28489385984.0, "2245": 28489385984.0, "2250": 28489385984.0, "2255": 28489385984.0, "2260": 28489385984.0, "2265": 28489385984.0, "2270": 28489385984.0, "2275": 28489385984.0, "2280": 28489385984.0, "2285": 28489385984.0, "2290": 28489385984.0, "2295": 28489385984.0, "2300": 28489385984.0, "2305": 28489385984.0, "2310": 28489385984.0, "2315": 28489385984.0, "2320": 28489385984.0, "2325": 28489385984.0, "2330": 28489385984.0, "2335": 28489385984.0, "2340": 28489385984.0, "2345": 28489385984.0, "2350": 28489385984.0, "2355": 28489385984.0, "2360": 28489385984.0, "2365": 28489385984.0, "2370": 28489385984.0, "2375": 28489385984.0, "2380": 28489385984.0, "2385": 28489385984.0, "2390": 28489385984.0, "2395": 28489385984.0, "2400": 28489385984.0, "2405": 28489385984.0, "2410": 28489385984.0, "2415": 28489385984.0, "2420": 28489385984.0, "2425": 28489385984.0, "2430": 28489385984.0, "2435": 28489385984.0, "2440": 28489385984.0, "2445": 28489385984.0, "2450": 28489385984.0, "2455": 28489385984.0, "2460": 28489385984.0, "2465": 28489385984.0, "2470": 28489385984.0, "2475": 28489385984.0, "2480": 28489385984.0, "2485": 28489385984.0, "2490": 28489385984.0, "2495": 28489385984.0, "2500": 28489385984.0, "2505": 28489385984.0, "2510": 28489385984.0, "2515": 28489385984.0, "2520": 28489385984.0, "2525": 28489385984.0, "2530": 28489385984.0, "2535": 28489385984.0, "2540": 28489385984.0, "2545": 28489385984.0, "2550": 28489385984.0, "2555": 28489385984.0, "2560": 28489385984.0, "2565": 28489385984.0, "2570": 28489385984.0, "2575": 28489385984.0, "2580": 28489385984.0, "2585": 28489385984.0, "2590": 28489385984.0, "2595": 28489385984.0, "2600": 28489385984.0, "2605": 28489385984.0, "2610": 28489385984.0, "2615": 28489385984.0, "2620": 28489385984.0, "2625": 28489385984.0, "2630": 28489385984.0, "2635": 28489385984.0, "2640": 28489385984.0, "2645": 28489385984.0, "2650": 28489385984.0, "2655": 28489385984.0, "2660": 28489385984.0, "2665": 28489385984.0, "2670": 28489385984.0, "2675": 28489385984.0, "2680": 28489385984.0, "2685": 28489385984.0, "2690": 28489385984.0, "2695": 28489385984.0, "2700": 28489385984.0, "2705": 28489385984.0, "2710": 28489385984.0, "2715": 28489385984.0, "2720": 28489385984.0, "2725": 28489385984.0, "2730": 28489385984.0, "2735": 28489385984.0, "2740": 28489385984.0, "2745": 28489385984.0, "2750": 28489385984.0, "2755": 28489385984.0, "2760": 28489385984.0, "2765": 28489385984.0, "2770": 28489385984.0, "2775": 28489385984.0, "2780": 28489385984.0, "2785": 28489385984.0, "2790": 28489385984.0, "2795": 28489385984.0, "2800": 28489385984.0, "2805": 28489385984.0, "2810": 28489385984.0, "2815": 28489385984.0, "2820": 28489385984.0, "2825": 28489385984.0, "2830": 28489385984.0, "2835": 28489385984.0, "2840": 28489385984.0, "2845": 28489385984.0, "2850": 28489385984.0, "2855": 28489385984.0, "2860": 28489385984.0, "2865": 28489385984.0, "2870": 28489385984.0, "2875": 28489385984.0, "2880": 28489385984.0, "2885": 28489385984.0, "2890": 28489385984.0, "2895": 28489385984.0, "2900": 28489385984.0, "2905": 28489385984.0, "2910": 28489385984.0, "2915": 28489385984.0, "2920": 28489385984.0, "2925": 28489385984.0, "2930": 28489385984.0, "2935": 28489385984.0, "2940": 28489385984.0, "2945": 28489385984.0, "2950": 28489385984.0, "2955": 28489385984.0, "2960": 28489385984.0, "2965": 28489385984.0, "2970": 28489385984.0, "2975": 28489385984.0, "2980": 28489385984.0, "2985": 28489385984.0, "2990": 28489385984.0, "2995": 28489385984.0, "3000": 28489385984.0, "3005": 28489385984.0, "3010": 28489385984.0, "3015": 28489385984.0, "3020": 28489385984.0, "3025": 28489385984.0, "3030": 28489385984.0, "3035": 28489385984.0, "3040": 28489385984.0, "3045": 28489385984.0, "3050": 28489385984.0, "3055": 28489385984.0, "3060": 28489385984.0, "3065": 28489385984.0, "3070": 28489385984.0, "3075": 28489385984.0, "3080": 28489385984.0, "3085": 28489385984.0, "3090": 28489385984.0, "3095": 28489385984.0, "3100": 28489385984.0, "3105": 28489385984.0, "3110": 28489385984.0, "3115": 28489385984.0, "3120": 28489385984.0, "3125": 28489385984.0, "3130": 28489385984.0, "3135": 28489385984.0, "3140": 28489385984.0, "3145": 28489385984.0, "3150": 28489385984.0, "3155": 28489385984.0, "3160": 28489385984.0, "3165": 28489385984.0, "3170": 28489385984.0, "3175": 28489385984.0, "3180": 28489385984.0, "3185": 28489385984.0, "3190": 28489385984.0, "3195": 28489385984.0, "3200": 28489385984.0, "3205": 28489385984.0, "3210": 28489385984.0, "3215": 28489385984.0, "3220": 28489385984.0, "3225": 28489385984.0, "3230": 28489385984.0, "3235": 28489385984.0, "3240": 28489385984.0, "3245": 28489385984.0, "3250": 28489385984.0, "3255": 28489385984.0, "3260": 28489385984.0, "3265": 28489385984.0, "3270": 28489385984.0, "3275": 28489385984.0, "3280": 28489385984.0, "3285": 28489385984.0, "3290": 28489385984.0, "3295": 28489385984.0, "3300": 28489385984.0, "3305": 28489385984.0, "3310": 28489385984.0, "3315": 28489385984.0, "3320": 28489385984.0, "3325": 28489385984.0, "3330": 28489385984.0, "3335": 28489385984.0, "3340": 28489385984.0, "3345": 28489385984.0, "3350": 28489385984.0, "3355": 28489385984.0, "3360": 28489385984.0, "3365": 28489385984.0, "3370": 28489385984.0, "3375": 28489385984.0, "3380": 28489385984.0, "3385": 28489385984.0, "3390": 28489385984.0, "3395": 28489385984.0, "3400": 28489385984.0, "3405": 28489385984.0, "3410": 28489385984.0, "3415": 28489385984.0, "3420": 28489385984.0, "3425": 28489385984.0, "3430": 28489385984.0, "3435": 28489385984.0, "3440": 28489385984.0, "3445": 28489385984.0, "3450": 28489385984.0, "3455": 28489385984.0, "3460": 28489385984.0, "3465": 28489385984.0, "3470": 28489385984.0, "3475": 28489385984.0, "3480": 28489385984.0, "3485": 28489385984.0, "3490": 28489385984.0, "3495": 28489385984.0, "3500": 28489385984.0, "3505": 28489385984.0, "3510": 28489385984.0, "3515": 28489385984.0, "3520": 28489385984.0, "3525": 28489385984.0, "3530": 28489385984.0, "3535": 28489385984.0, "3540": 28489385984.0, "3545": 28489385984.0, "3550": 28489385984.0, "3555": 28489385984.0, "3560": 28489385984.0, "3565": 28489385984.0, "3570": 28489568256.0, "3575": 28489568256.0, "3580": 28489568256.0, "3585": 28489568256.0, "3590": 28489568256.0, "3595": 28489568256.0, "3600": 28489568256.0, "3605": 28489568256.0, "3610": 28489568256.0, "3615": 28489568256.0, "3620": 28489568256.0, "3625": 28489568256.0, "3630": 28489568256.0, "3635": 28489568256.0, "3640": 28489568256.0, "3645": 28489568256.0, "3650": 28489568256.0, "3655": 28489568256.0, "3660": 28489568256.0, "3665": 28489568256.0, "3670": 28489568256.0, "3675": 28489568256.0, "3680": 28489568256.0, "3685": 28489568256.0, "3690": 28489568256.0, "3695": 28489568256.0, "3700": 28489568256.0, "3705": 28489568256.0, "3710": 28489568256.0, "3715": 28489568256.0, "3720": 28489568256.0, "3725": 28489568256.0, "3730": 28489568256.0, "3735": 28489568256.0, "3740": 28489568256.0, "3745": 28489568256.0, "3750": 28489568256.0, "3755": 28489568256.0, "3760": 28489568256.0, "3765": 28489568256.0, "3770": 28489568256.0, "3775": 28489568256.0, "3780": 28489568256.0, "3785": 28489568256.0, "3790": 28489568256.0, "3795": 28489568256.0, "3800": 28489568256.0, "3805": 28489568256.0, "3810": 28489568256.0, "3815": 28489568256.0, "3820": 28489568256.0, "3825": 28489568256.0, "3830": 28489568256.0, "3835": 28489568256.0, "3840": 28489568256.0, "3845": 28489568256.0, "3850": 28489568256.0, "3855": 28489568256.0, "3860": 28489568256.0, "3865": 28489568256.0, "3870": 28489568256.0, "3875": 28489568256.0, "3880": 28489568256.0, "3885": 28489568256.0, "3890": 28489568256.0, "3895": 28489568256.0, "3900": 28489568256.0, "3905": 28489568256.0, "3910": 28489568256.0, "3915": 28489568256.0, "3920": 28489568256.0, "3925": 28489568256.0, "3930": 28489568256.0, "3935": 28489568256.0, "3940": 28489568256.0, "3945": 28489568256.0, "3950": 28489568256.0, "3955": 28489568256.0, "3960": 28489568256.0, "3965": 28489568256.0, "3970": 28489568256.0, "3975": 28489568256.0, "3980": 28489568256.0, "3985": 28489568256.0, "3990": 28489568256.0, "3995": 28489568256.0, "4000": 28489568256.0, "4005": 28489568256.0, "4010": 28489568256.0, "4015": 28489568256.0, "4020": 28489568256.0, "4025": 28489568256.0, "4030": 28489568256.0, "4035": 28489568256.0, "4040": 28489568256.0, "4045": 28489568256.0, "4050": 28489568256.0, "4055": 28489568256.0, "4060": 28489568256.0, "4065": 28489568256.0, "4070": 28489568256.0, "4075": 28489568256.0, "4080": 28489568256.0, "4085": 28489568256.0, "4090": 28489568256.0, "4095": 28489568256.0, "4100": 28489568256.0, "4105": 28489568256.0, "4110": 28489568256.0, "4115": 28489568256.0, "4120": 28489568256.0, "4125": 28489568256.0, "4130": 28489568256.0, "4135": 28489568256.0, "4140": 28489568256.0, "4145": 28489568256.0, "4150": 28489568256.0, "4155": 28489568256.0, "4160": 28489568256.0, "4165": 28489568256.0, "4170": 28489568256.0, "4175": 28489568256.0, "4180": 28489568256.0, "4185": 28489568256.0, "4190": 28489568256.0, "4195": 28489568256.0, "4200": 28489568256.0, "4205": 28489568256.0, "4210": 28489568256.0, "4215": 28489568256.0, "4220": 28489568256.0, "4225": 28489568256.0, "4230": 28489568256.0, "4235": 28489568256.0, "4240": 28489568256.0, "4245": 28489568256.0, "4250": 28489568256.0, "4255": 28489568256.0, "4260": 28489568256.0, "4265": 28489568256.0, "4270": 28489568256.0, "4275": 28489568256.0, "4280": 28489568256.0, "4285": 28489568256.0, "4290": 28489568256.0, "4295": 28489568256.0, "4300": 28489568256.0, "4305": 28489568256.0, "4310": 28489568256.0, "4315": 28489568256.0, "4320": 28489568256.0, "4325": 28489568256.0, "4330": 28489568256.0, "4335": 28489568256.0, "4340": 28489568256.0, "4345": 28489568256.0, "4350": 28489568256.0, "4355": 28489568256.0, "4360": 28489568256.0, "4365": 28489568256.0, "4370": 28489568256.0, "4375": 28489568256.0, "4380": 28489568256.0, "4385": 28489568256.0, "4390": 28489568256.0, "4395": 28489568256.0, "4400": 28489568256.0, "4405": 28489568256.0, "4410": 28489568256.0, "4415": 28489568256.0, "4420": 28489568256.0, "4425": 28489568256.0, "4430": 28489568256.0, "4435": 28489568256.0, "4440": 28489568256.0, "4445": 28489568256.0, "4450": 28489568256.0, "4455": 28489568256.0, "4460": 28489568256.0, "4465": 28489568256.0, "4470": 28489568256.0, "4475": 28489568256.0, "4480": 28489568256.0, "4485": 28489568256.0, "4490": 28489568256.0, "4495": 28489568256.0, "4500": 28489568256.0, "4505": 28489568256.0, "4510": 28489568256.0, "4515": 28489568256.0, "4520": 28489568256.0, "4525": 28489568256.0, "4530": 28489568256.0, "4535": 28489568256.0, "4540": 28489568256.0, "4545": 28489568256.0, "4550": 28489568256.0, "4555": 28489568256.0, "4560": 28489568256.0, "4565": 28489568256.0, "4570": 28489568256.0, "4575": 28489568256.0, "4580": 28489568256.0, "4585": 28489568256.0, "4590": 28489568256.0, "4595": 28489568256.0, "4600": 28489568256.0, "4605": 28489568256.0, "4610": 28489568256.0, "4615": 28489568256.0, "4620": 28489568256.0, "4625": 28489568256.0, "4630": 28489568256.0, "4635": 28489568256.0, "4640": 28489568256.0, "4645": 28489568256.0, "4650": 28489568256.0, "4655": 28489568256.0, "4660": 28489568256.0, "4665": 28489568256.0, "4670": 28489568256.0, "4675": 28489568256.0, "4680": 28489568256.0, "4685": 28489568256.0, "4690": 28489568256.0, "4695": 28489568256.0, "4700": 28489568256.0, "4705": 28489568256.0, "4710": 28489568256.0, "4715": 28489568256.0, "4720": 28489568256.0, "4725": 28489568256.0, "4730": 28489568256.0, "4735": 28489568256.0, "4740": 28489568256.0, "4745": 28489568256.0, "4750": 28489568256.0, "4755": 28489568256.0, "4760": 28489568256.0, "4765": 28489568256.0, "4770": 28489568256.0, "4775": 28489568256.0, "4780": 28489568256.0, "4785": 28489568256.0, "4790": 28489568256.0, "4795": 28489568256.0, "4800": 28489568256.0, "4805": 28489568256.0, "4810": 28489568256.0, "4815": 28489568256.0, "4820": 28489568256.0, "4825": 28489568256.0, "4830": 28489568256.0, "4835": 28489568256.0, "4840": 28489568256.0, "4845": 28489568256.0, "4850": 28489568256.0, "4855": 28489568256.0, "4860": 28489568256.0, "4865": 28489568256.0, "4870": 28489568256.0, "4875": 28489568256.0, "4880": 28489568256.0, "4885": 28489568256.0, "4890": 28489568256.0, "4895": 28489568256.0, "4900": 28489568256.0, "4905": 28489568256.0, "4910": 28489568256.0, "4915": 28489568256.0, "4920": 28489568256.0, "4925": 28489568256.0, "4930": 28489568256.0, "4935": 28489568256.0, "4940": 28489568256.0, "4945": 28489568256.0, "4950": 28489568256.0, "4955": 28489568256.0, "4960": 28489568256.0, "4965": 28489568256.0, "4970": 28489568256.0, "4975": 28489568256.0, "4980": 28489568256.0, "4985": 28489568256.0, "4990": 28489568256.0, "4995": 28489568256.0, "5000": 28489568256.0, "5005": 28489568256.0, "5010": 28489568256.0, "5015": 28489568256.0, "5020": 28489568256.0, "5025": 28489568256.0, "5030": 28489568256.0, "5035": 28489568256.0, "5040": 28489568256.0, "5045": 28489568256.0, "5050": 28489568256.0, "5055": 28489568256.0, "5060": 28489568256.0, "5065": 28489568256.0, "5070": 28489568256.0, "5075": 28489568256.0, "5080": 28489568256.0, "5085": 28489568256.0, "5090": 28489568256.0, "5095": 28489568256.0, "5100": 28489568256.0, "5105": 28489568256.0, "5110": 28489568256.0, "5115": 28489568256.0, "5120": 28489568256.0, "5125": 28489568256.0, "5130": 28489568256.0, "5135": 28489568256.0, "5140": 28489568256.0, "5145": 28489568256.0, "5150": 28489568256.0, "5155": 28489568256.0, "5160": 28489568256.0, "5165": 28489568256.0, "5170": 28489568256.0, "5175": 28489568256.0, "5180": 28489568256.0, "5185": 28489568256.0, "5190": 28489568256.0, "5195": 28489568256.0, "5200": 28489568256.0, "5205": 28489568256.0, "5210": 28489568256.0, "5215": 28489568256.0, "5220": 28489568256.0, "5225": 28489568256.0, "5230": 28489568256.0, "5235": 28489568256.0, "5240": 28489568256.0, "5245": 28489568256.0, "5250": 28489568256.0, "5255": 28489568256.0, "5260": 28489568256.0, "5265": 28489568256.0, "5270": 28489568256.0, "5275": 28489568256.0, "5280": 28489568256.0, "5285": 28489568256.0, "5290": 28489568256.0, "5295": 28489568256.0, "5300": 28489568256.0, "5305": 28489568256.0, "5310": 28489568256.0, "5315": 28489568256.0, "5320": 28489568256.0, "5325": 28489568256.0, "5330": 28489568256.0, "5335": 28489568256.0, "5340": 28489568256.0, "5345": 28489568256.0, "5350": 28489568256.0, "5355": 28489568256.0, "5360": 28489568256.0, "5365": 28489568256.0, "5370": 28489568256.0, "5375": 28489568256.0, "5380": 28489568256.0, "5385": 28489568256.0, "5390": 28489568256.0, "5395": 28489568256.0, "5400": 28489568256.0, "5405": 28489568256.0, "5410": 28489568256.0, "5415": 28489568256.0, "5420": 28489568256.0, "5425": 28489568256.0, "5430": 28489568256.0, "5435": 28489568256.0, "5440": 28489568256.0, "5445": 28489568256.0, "5450": 28489568256.0, "5455": 28489568256.0, "5460": 28489568256.0, "5465": 28489568256.0, "5470": 28489568256.0, "5475": 28489568256.0, "5480": 28489568256.0, "5485": 28489568256.0, "5490": 28489568256.0, "5495": 28489568256.0, "5500": 28489568256.0, "5505": 28489568256.0, "5510": 28489568256.0, "5515": 28489568256.0, "5520": 28489568256.0, "5525": 28489568256.0, "5530": 28489568256.0, "5535": 28489568256.0, "5540": 28489568256.0, "5545": 28489568256.0, "5550": 28489568256.0, "5555": 28489568256.0, "5560": 28489568256.0, "5565": 28489568256.0, "5570": 28489568256.0, "5575": 28489568256.0, "5580": 28489568256.0, "5585": 28489568256.0, "5590": 28489568256.0, "5595": 28489568256.0, "5600": 28489568256.0, "5605": 28489568256.0, "5610": 28489568256.0, "5615": 28489568256.0, "5620": 28489568256.0, "5625": 28489568256.0, "5630": 28489568256.0, "5635": 28489568256.0, "5640": 28489568256.0, "5645": 28489568256.0, "5650": 28489568256.0, "5655": 28489568256.0, "5660": 28489568256.0, "5665": 28489568256.0, "5670": 28489568256.0, "5675": 28489568256.0, "5680": 28489568256.0, "5685": 28489568256.0, "5690": 28489568256.0, "5695": 28489568256.0, "5700": 28489568256.0, "5705": 28489568256.0, "5710": 28489568256.0, "5715": 28489568256.0, "5720": 28489568256.0, "5725": 28489568256.0, "5730": 28489568256.0, "5735": 28489568256.0, "5740": 28489568256.0, "5745": 28489568256.0, "5750": 28489568256.0, "5755": 28489568256.0, "5760": 28489568256.0, "5765": 28489568256.0, "5770": 28489568256.0, "5775": 28489568256.0, "5780": 28489568256.0, "5785": 28489568256.0, "5790": 28489568256.0, "5795": 28489568256.0, "5800": 28489568256.0, "5805": 28489568256.0, "5810": 28489568256.0, "5815": 28489568256.0, "5820": 28489568256.0, "5825": 28489568256.0, "5830": 28489568256.0, "5835": 28489568256.0, "5840": 28489568256.0, "5845": 28489568256.0, "5850": 28489568256.0, "5855": 28489568256.0, "5860": 28489568256.0, "5865": 28489568256.0, "5870": 28489568256.0, "5875": 28489568256.0, "5880": 28489568256.0, "5885": 28489568256.0, "5890": 28489568256.0, "5895": 28489568256.0, "5900": 28489568256.0, "5905": 28489568256.0, "5910": 28489568256.0, "5915": 28489568256.0, "5920": 28489568256.0, "5925": 28489568256.0, "5930": 28489568256.0, "5935": 28489568256.0, "5940": 28489568256.0, "5945": 28489568256.0, "5950": 28489568256.0, "5955": 28489568256.0, "5960": 28489568256.0, "5965": 28489568256.0, "5970": 28489568256.0, "5975": 28489568256.0, "5980": 28489568256.0, "5985": 28489568256.0, "5990": 28489568256.0, "5995": 28489568256.0, "6000": 28489568256.0, "6005": 28489568256.0, "6010": 28489568256.0, "6015": 28489568256.0, "6020": 28489568256.0, "6025": 28489568256.0, "6030": 28489568256.0, "6035": 28489568256.0, "6040": 28489568256.0, "6045": 28489568256.0, "6050": 28489568256.0, "6055": 28489568256.0, "6060": 28489568256.0, "6065": 28489568256.0, "6070": 28489568256.0, "6075": 28489568256.0, "6080": 28489568256.0, "6085": 28489568256.0, "6090": 28489568256.0, "6095": 28489568256.0, "6100": 28489568256.0, "6105": 28489568256.0, "6110": 28489568256.0, "6115": 28489568256.0, "6120": 28489568256.0, "6125": 28489568256.0, "6130": 28489568256.0, "6135": 28489568256.0, "6140": 28489568256.0, "6145": 28489568256.0, "6150": 28489568256.0, "6155": 28489568256.0, "6160": 28489568256.0, "6165": 28489568256.0, "6170": 28489568256.0, "6175": 28489568256.0, "6180": 28489568256.0, "6185": 28489568256.0, "6190": 28489568256.0, "6195": 28489568256.0, "6200": 28489568256.0, "6205": 28489568256.0, "6210": 28489568256.0, "6215": 28489568256.0, "6220": 28489568256.0, "6225": 28489568256.0, "6230": 28489568256.0, "6235": 28489568256.0, "6240": 28489568256.0, "6245": 28489568256.0, "6250": 28489568256.0, "6255": 28489568256.0, "6260": 28489568256.0, "6265": 28489568256.0, "6270": 28489568256.0, "6275": 28489568256.0, "6280": 28489568256.0, "6285": 28489568256.0, "6290": 28489568256.0, "6295": 28489568256.0, "6300": 28489568256.0, "6305": 28489568256.0, "6310": 28489568256.0, "6315": 28489568256.0, "6320": 28489568256.0, "6325": 28489568256.0, "6330": 28489568256.0, "6335": 28489568256.0, "6340": 28489568256.0, "6345": 28489568256.0, "6350": 28489568256.0, "6355": 28489568256.0, "6360": 28489568256.0, "6365": 28489568256.0, "6370": 28489568256.0, "6375": 28489568256.0, "6380": 28489568256.0, "6385": 28489568256.0, "6390": 28489568256.0, "6395": 28489568256.0, "6400": 28489568256.0, "6405": 28489568256.0, "6410": 28489568256.0, "6415": 28489568256.0, "6420": 28489568256.0, "6425": 28489568256.0, "6430": 28489568256.0, "6435": 28489568256.0, "6440": 28489568256.0, "6445": 28489568256.0, "6450": 28489568256.0, "6455": 28489568256.0, "6460": 28489568256.0, "6465": 28489568256.0, "6470": 28489568256.0, "6475": 28489568256.0, "6480": 28489568256.0, "6485": 28489568256.0, "6490": 28489568256.0, "6495": 28489568256.0, "6500": 28489568256.0, "6505": 28489568256.0, "6510": 28489568256.0, "6515": 28489568256.0, "6520": 28489568256.0, "6525": 28489568256.0, "6530": 28489568256.0, "6535": 28489568256.0, "6540": 28489568256.0, "6545": 28489568256.0, "6550": 28489568256.0, "6555": 28489568256.0, "6560": 28489568256.0, "6565": 28489568256.0, "6570": 28489568256.0, "6575": 28489568256.0, "6580": 28489568256.0, "6585": 28489568256.0, "6590": 28489568256.0, "6595": 28489568256.0, "6600": 28489568256.0, "6605": 28489568256.0, "6610": 28489568256.0, "6615": 28489568256.0, "6620": 28489568256.0, "6625": 28489568256.0, "6630": 28489568256.0, "6635": 28489568256.0, "6640": 28489568256.0, "6645": 28489568256.0, "6650": 28489568256.0, "6655": 28489568256.0, "6660": 28489568256.0, "6665": 28489568256.0, "6670": 28489568256.0, "6675": 28489568256.0, "6680": 28489568256.0, "6685": 28489568256.0, "6690": 28489568256.0, "6695": 28489568256.0, "6700": 28489568256.0, "6705": 28489568256.0, "6710": 28489568256.0, "6715": 28489568256.0, "6720": 28489568256.0, "6725": 28489568256.0, "6730": 28489568256.0, "6735": 28489568256.0, "6740": 28489568256.0, "6745": 28489568256.0, "6750": 28489568256.0, "6755": 28489568256.0, "6760": 28489568256.0, "6765": 28489568256.0, "6770": 28489568256.0, "6775": 28489568256.0, "6780": 28489568256.0, "6785": 28489568256.0, "6790": 28489568256.0, "6795": 28489568256.0, "6800": 28489568256.0, "6805": 28489568256.0, "6810": 28489568256.0, "6815": 28489568256.0, "6820": 28489568256.0, "6825": 28489568256.0, "6830": 28489568256.0, "6835": 28489568256.0, "6840": 28489568256.0, "6845": 28489568256.0, "6850": 28489568256.0, "6855": 28489568256.0, "6860": 28489568256.0, "6865": 28489568256.0, "6870": 28489568256.0, "6875": 28489568256.0, "6880": 28489568256.0, "6885": 28489568256.0, "6890": 28489568256.0, "6895": 28489568256.0, "6900": 28489568256.0, "6905": 28489568256.0, "6910": 28489568256.0, "6915": 28489568256.0, "6920": 28489568256.0, "6925": 28489568256.0, "6930": 28489568256.0, "6935": 28489568256.0, "6940": 28489568256.0, "6945": 28489568256.0, "6950": 28489568256.0, "6955": 28489568256.0, "6960": 28489568256.0, "6965": 28489568256.0, "6970": 28489568256.0, "6975": 28489568256.0, "6980": 28489568256.0, "6985": 28489568256.0, "6990": 28489568256.0, "6995": 28489568256.0, "7000": 28489568256.0, "7005": 28489568256.0, "7010": 28489568256.0, "7015": 28489568256.0, "7020": 28489568256.0, "7025": 28489568256.0, "7030": 28489568256.0, "7035": 28489568256.0, "7040": 28489568256.0, "7045": 28489568256.0, "7050": 28489568256.0, "7055": 28489568256.0, "7060": 28489568256.0, "7065": 28489568256.0, "7070": 28489568256.0, "7075": 28489568256.0, "7080": 28489568256.0, "7085": 28489568256.0, "7090": 28489568256.0, "7095": 28489568256.0, "7100": 28489568256.0, "7105": 28489568256.0, "7110": 28489568256.0, "7115": 28489568256.0, "7120": 28489568256.0, "7125": 28489568256.0, "7130": 28489568256.0, "7135": 28489568256.0, "7140": 28489568256.0, "7145": 28489568256.0, "7150": 28489568256.0, "7155": 28489568256.0, "7160": 28489568256.0, "7165": 28489568256.0, "7170": 28489568256.0, "7175": 28489568256.0, "7180": 28489568256.0, "7185": 28489568256.0, "7190": 28489568256.0, "7195": 28489568256.0, "7200": 28489568256.0, "7205": 28489568256.0, "7210": 28489568256.0, "7215": 28489568256.0, "7220": 28489568256.0, "7225": 28489568256.0, "7230": 28489568256.0, "7235": 28489568256.0, "7240": 28489568256.0, "7245": 28489568256.0, "7250": 28489568256.0, "7255": 28489568256.0, "7260": 28489568256.0, "7265": 28489568256.0, "7270": 28489568256.0, "7275": 28489568256.0, "7280": 28489568256.0, "7285": 28489568256.0, "7290": 28489568256.0, "7295": 28489568256.0, "7300": 28489568256.0, "7305": 28489568256.0, "7310": 28489568256.0, "7315": 28489568256.0, "7320": 28489568256.0, "7325": 28489568256.0, "7330": 28489568256.0, "7335": 28489568256.0, "7340": 28489568256.0, "7345": 28489568256.0, "7350": 28489568256.0, "7355": 28489568256.0, "7360": 28489568256.0, "7365": 28489568256.0, "7370": 28489568256.0, "7375": 28489568256.0, "7380": 28489568256.0, "7385": 28489568256.0, "7390": 28489568256.0, "7395": 28489568256.0, "7400": 28489568256.0, "7405": 28489568256.0, "7410": 28489568256.0, "7415": 28489568256.0, "7420": 28489568256.0, "7425": 28489568256.0, "7430": 28489568256.0, "7435": 28489568256.0, "7440": 28489568256.0, "7445": 28489568256.0, "7450": 28489568256.0, "7455": 28489568256.0, "7460": 28489568256.0, "7465": 28489568256.0, "7470": 28489568256.0, "7475": 28489568256.0, "7480": 28489568256.0, "7485": 28489568256.0, "7490": 28489568256.0, "7495": 28489568256.0, "7500": 28489568256.0, "7505": 28489568256.0, "7510": 28489568256.0, "7515": 28489568256.0, "7520": 28489568256.0, "7525": 28489568256.0, "7530": 28489568256.0, "7535": 28489568256.0, "7540": 28489568256.0, "7545": 28489568256.0, "7550": 28489568256.0, "7555": 28489568256.0, "7560": 28489568256.0, "7565": 28489568256.0, "7570": 28489568256.0, "7575": 28489568256.0, "7580": 28489568256.0, "7585": 28489568256.0, "7590": 28489568256.0, "7595": 28489568256.0, "7600": 28489568256.0, "7605": 28489568256.0, "7610": 28489568256.0, "7615": 28489568256.0, "7620": 28489568256.0, "7625": 28489568256.0, "7630": 28489568256.0, "7635": 28489568256.0, "7640": 28489568256.0, "7645": 28489568256.0, "7650": 28489568256.0, "7655": 28489568256.0, "7660": 28489568256.0, "7665": 28489568256.0, "7670": 28489568256.0, "7675": 28489568256.0, "7680": 28489568256.0, "7685": 28489568256.0, "7690": 28489568256.0, "7695": 28489568256.0, "7700": 28489568256.0, "7705": 28489568256.0, "7710": 28489568256.0, "7715": 28489568256.0, "7720": 28489568256.0, "7725": 28489568256.0, "7730": 28489568256.0, "7735": 28489568256.0, "7740": 28489568256.0, "7745": 28489568256.0, "7750": 28489568256.0, "7755": 28489568256.0, "7760": 28489568256.0, "7765": 28489568256.0, "7770": 28489568256.0, "7775": 28489568256.0, "7780": 28489568256.0, "7785": 28489568256.0, "7790": 28489568256.0, "7795": 28489568256.0, "7800": 28489568256.0, "7805": 28489568256.0, "7810": 28489568256.0, "7815": 28489568256.0, "7820": 28489568256.0, "7825": 28489568256.0, "7830": 28489568256.0, "7835": 28489568256.0, "7840": 28489568256.0, "7845": 28489568256.0, "7850": 28489568256.0, "7855": 28489568256.0, "7860": 28489568256.0, "7865": 28489568256.0, "7870": 28489568256.0, "7875": 28489568256.0, "7880": 28489568256.0, "7885": 28489568256.0, "7890": 28489568256.0, "7895": 28489568256.0, "7900": 28489568256.0, "7905": 28489568256.0, "7910": 28489568256.0, "7915": 28489568256.0, "7920": 28489568256.0, "7925": 28489568256.0, "7930": 28489568256.0, "7935": 28489568256.0, "7940": 28489568256.0, "7945": 28489568256.0, "7950": 28489568256.0, "7955": 28489568256.0, "7960": 28489568256.0, "7965": 28489568256.0, "7970": 28489568256.0, "7975": 28489568256.0, "7980": 28489568256.0, "7985": 28489568256.0, "7990": 28489568256.0, "7995": 28489568256.0, "8000": 28489568256.0, "8005": 28489568256.0, "8010": 28489568256.0, "8015": 28489568256.0, "8020": 28489568256.0, "8025": 28489568256.0, "8030": 28489568256.0, "8035": 28489568256.0, "8040": 28489568256.0, "8045": 28489568256.0, "8050": 28489568256.0, "8055": 28489568256.0, "8060": 28489568256.0, "8065": 28489568256.0, "8070": 28489568256.0, "8075": 28489568256.0, "8080": 28489568256.0, "8085": 28489568256.0, "8090": 28489568256.0, "8095": 28489568256.0, "8100": 28489568256.0, "8105": 28489568256.0, "8110": 28489568256.0, "8115": 28489568256.0, "8120": 28489568256.0, "8125": 28489568256.0, "8130": 28489568256.0, "8135": 28489568256.0, "8140": 28489568256.0, "8145": 28489568256.0, "8150": 28489568256.0, "8155": 28489568256.0, "8160": 28489568256.0, "8165": 28489568256.0, "8170": 28489568256.0, "8175": 28489568256.0, "8180": 28489568256.0, "8185": 28489568256.0, "8190": 28489568256.0, "8195": 28489568256.0, "8200": 28489568256.0, "8205": 28489568256.0, "8210": 28489568256.0, "8215": 28489568256.0, "8220": 28489568256.0, "8225": 28489568256.0, "8230": 28489568256.0, "8235": 28489568256.0, "8240": 28489568256.0, "8245": 28489568256.0, "8250": 28489568256.0, "8255": 28489568256.0, "8260": 28489568256.0, "8265": 28489568256.0, "8270": 28489568256.0, "8275": 28489568256.0, "8280": 28489568256.0, "8285": 28489568256.0, "8290": 28489568256.0, "8295": 28489568256.0, "8300": 28489568256.0, "8305": 28489568256.0, "8310": 28489568256.0, "8315": 28489568256.0, "8320": 28489568256.0, "8325": 28489568256.0, "8330": 28489568256.0, "8335": 28489568256.0, "8340": 28489568256.0, "8345": 28489568256.0, "8350": 28489568256.0, "8355": 28489568256.0, "8360": 28489568256.0, "8365": 28489568256.0, "8370": 28489568256.0, "8375": 28489568256.0, "8380": 28489568256.0, "8385": 28489568256.0, "8390": 28489568256.0, "8395": 28489568256.0, "8400": 28489568256.0, "8405": 28489568256.0, "8410": 28489568256.0, "8415": 28489568256.0, "8420": 28489568256.0, "8425": 28489568256.0, "8430": 28489568256.0, "8435": 28489568256.0, "8440": 28489568256.0, "8445": 28489568256.0, "8450": 28489568256.0, "8455": 28489568256.0, "8460": 28489568256.0, "8465": 28489568256.0, "8470": 28489568256.0, "8475": 28489568256.0, "8480": 28489568256.0, "8485": 28489568256.0, "8490": 28489568256.0, "8495": 28489568256.0, "8500": 28489568256.0, "8505": 28489568256.0, "8510": 28489568256.0, "8515": 28489568256.0, "8520": 28489568256.0, "8525": 28489568256.0, "8530": 28489568256.0, "8535": 28489568256.0, "8540": 28489568256.0, "8545": 28489568256.0, "8550": 28489568256.0, "8555": 28489568256.0, "8560": 28489568256.0, "8565": 28489568256.0, "8570": 28489568256.0, "8575": 28489568256.0, "8580": 28489568256.0, "8585": 28489568256.0, "8590": 28489568256.0, "8595": 28489568256.0, "8600": 28489568256.0, "8605": 28489568256.0, "8610": 28489568256.0, "8615": 28489568256.0, "8620": 28489568256.0, "8625": 28489568256.0, "8630": 28489568256.0, "8635": 28489568256.0, "8640": 28489568256.0, "8645": 28489568256.0, "8650": 28489568256.0, "8655": 28489568256.0, "8660": 28489568256.0, "8665": 28489568256.0, "8670": 28489568256.0, "8675": 28489568256.0, "8680": 28489568256.0, "8685": 28489568256.0, "8690": 28489568256.0, "8695": 28489568256.0, "8700": 28489568256.0, "8705": 28489568256.0, "8710": 28489568256.0, "8715": 28489568256.0, "8720": 28489568256.0, "8725": 28489568256.0, "8730": 28489568256.0, "8735": 28489568256.0, "8740": 28489568256.0, "8745": 28489568256.0, "8750": 28489568256.0, "8755": 28489568256.0, "8760": 28489568256.0, "8765": 28489568256.0, "8770": 28489568256.0, "8775": 28489568256.0, "8780": 28489568256.0, "8785": 28489568256.0, "8790": 28489568256.0, "8795": 28489568256.0, "8800": 28489568256.0, "8805": 28489568256.0, "8810": 28489568256.0, "8815": 28489568256.0, "8820": 28489568256.0, "8825": 28489568256.0, "8830": 28489568256.0, "8835": 28489568256.0, "8840": 28489568256.0, "8845": 28489568256.0, "8850": 28489568256.0, "8855": 28489568256.0, "8860": 28489568256.0, "8865": 28489568256.0, "8870": 28489568256.0, "8875": 28489568256.0, "8880": 28489568256.0, "8885": 28489568256.0, "8890": 28489568256.0, "8895": 28489568256.0, "8900": 28489568256.0, "8905": 28489568256.0, "8910": 28489568256.0, "8915": 28489568256.0, "8920": 28489568256.0, "8925": 28489568256.0, "8930": 28489568256.0, "8935": 28489568256.0, "8940": 28489568256.0, "8945": 28489568256.0, "8950": 28489568256.0, "8955": 28489568256.0, "8960": 28489568256.0, "8965": 28489568256.0, "8970": 28489568256.0, "8975": 28489568256.0, "8980": 28489568256.0, "8985": 28489568256.0, "8990": 28489568256.0, "8995": 28489568256.0, "9000": 28489568256.0, "9005": 28489568256.0, "9010": 28489568256.0, "9015": 28489568256.0, "9020": 28489568256.0, "9025": 28489568256.0, "9030": 28489568256.0, "9035": 28489568256.0, "9040": 28489568256.0, "9045": 28489568256.0, "9050": 28489568256.0, "9055": 28489568256.0, "9060": 28489568256.0, "9065": 28489568256.0, "9070": 28489568256.0, "9075": 28489568256.0, "9080": 28489568256.0, "9085": 28489568256.0, "9090": 28489568256.0, "9095": 28489568256.0, "9100": 28489568256.0, "9105": 28489568256.0, "9110": 28489568256.0, "9115": 28489568256.0, "9120": 28489568256.0, "9125": 28489568256.0, "9130": 28489568256.0, "9135": 28489568256.0, "9140": 28489568256.0, "9145": 28489568256.0, "9150": 28489568256.0, "9155": 28489568256.0, "9160": 28489568256.0, "9165": 28489568256.0, "9170": 28489568256.0, "9175": 28489568256.0, "9180": 28489568256.0, "9185": 28489568256.0, "9190": 28489568256.0, "9195": 28489568256.0, "9200": 28489568256.0, "9205": 28489568256.0, "9210": 28489568256.0, "9215": 28489568256.0, "9220": 28489568256.0, "9225": 28489568256.0, "9230": 28489568256.0, "9235": 28489568256.0, "9240": 28489568256.0, "9245": 28489568256.0, "9250": 28489568256.0, "9255": 28489568256.0, "9260": 28489568256.0, "9265": 28489568256.0, "9270": 28489568256.0, "9275": 28489568256.0, "9280": 28489568256.0, "9285": 28489568256.0, "9290": 28489568256.0, "9295": 28489568256.0, "9300": 28489568256.0, "9305": 28489568256.0, "9310": 28489568256.0, "9315": 28489568256.0, "9320": 28489568256.0, "9325": 28489568256.0, "9330": 28489568256.0, "9335": 28489568256.0, "9340": 28489568256.0, "9345": 28489568256.0, "9350": 28489568256.0, "9355": 28489568256.0, "9360": 28489568256.0, "9365": 28489568256.0, "9370": 28489568256.0, "9375": 28489568256.0, "9380": 28489568256.0, "9385": 28489568256.0, "9390": 28489568256.0, "9395": 28489568256.0, "9400": 28489568256.0, "9405": 28489568256.0, "9410": 28489568256.0, "9415": 28489568256.0, "9420": 28489568256.0, "9425": 28489568256.0, "9430": 28489568256.0, "9435": 28489568256.0, "9440": 28489568256.0, "9445": 28489568256.0, "9450": 28489568256.0, "9455": 28489568256.0, "9460": 28489568256.0, "9465": 28489568256.0, "9470": 28489568256.0, "9475": 28489568256.0, "9480": 28489568256.0, "9485": 28489568256.0, "9490": 28489568256.0, "9495": 28489568256.0, "9500": 28489568256.0, "9505": 28489568256.0, "9510": 28489568256.0, "9515": 28489568256.0, "9520": 28489568256.0, "9525": 28489568256.0, "9530": 28489568256.0, "9535": 28489568256.0, "9540": 28489568256.0, "9545": 28489568256.0, "9550": 28489568256.0, "9555": 28489568256.0, "9560": 28489568256.0, "9565": 28489568256.0, "9570": 28489568256.0, "9575": 28489568256.0, "9580": 28489568256.0, "9585": 28489568256.0, "9590": 28489568256.0, "9595": 28489568256.0, "9600": 28489568256.0, "9605": 28489568256.0, "9610": 28489568256.0, "9615": 28489568256.0, "9620": 28489568256.0, "9625": 28489568256.0, "9630": 28489568256.0, "9635": 28489568256.0, "9640": 28489568256.0, "9645": 28489568256.0, "9650": 28489568256.0, "9655": 28489568256.0, "9660": 28489568256.0, "9665": 28489568256.0, "9670": 28489568256.0, "9675": 28489568256.0, "9680": 28489568256.0, "9685": 28489568256.0, "9690": 28489568256.0, "9695": 28489568256.0, "9700": 28489568256.0, "9705": 28489568256.0, "9710": 28489568256.0, "9715": 28489568256.0, "9720": 28489568256.0, "9725": 28489568256.0, "9730": 28489568256.0, "9735": 28489568256.0, "9740": 28489568256.0, "9745": 28489568256.0, "9750": 28489568256.0, "9755": 28489568256.0, "9760": 28489568256.0, "9765": 28489568256.0, "9770": 28489568256.0, "9775": 28489568256.0, "9780": 28489568256.0, "9785": 28489568256.0, "9790": 28489568256.0, "9795": 28489568256.0, "9800": 28489568256.0, "9805": 28489568256.0, "9810": 28489568256.0, "9815": 28489568256.0, "9820": 28489568256.0, "9825": 28489568256.0, "9830": 28489568256.0, "9835": 28489568256.0, "9840": 28489568256.0, "9845": 28489568256.0, "9850": 28489568256.0, "9855": 28489568256.0, "9860": 28489568256.0, "9865": 28489568256.0, "9870": 28489568256.0, "9875": 28489568256.0, "9880": 28489568256.0, "9885": 28489568256.0, "9890": 28489568256.0, "9895": 28489568256.0, "9900": 28489568256.0, "9905": 28489568256.0, "9910": 28489568256.0, "9915": 28489568256.0, "9920": 28489568256.0, "9925": 28489568256.0, "9930": 28489568256.0, "9935": 28489568256.0, "9940": 28489568256.0, "9945": 28489568256.0, "9950": 28489568256.0, "9955": 28489568256.0, "9960": 28489568256.0, "9965": 28489568256.0, "9970": 28489568256.0, "9975": 28489568256.0, "9980": 28489568256.0, "9985": 28489568256.0, "9990": 28489568256.0, "9995": 28489568256.0, "10000": 28489568256.0, "10005": 28489568256.0, "10010": 28489568256.0, "10015": 28489568256.0, "10020": 28489568256.0, "10025": 28489568256.0, "10030": 28489568256.0, "10035": 28489568256.0, "10040": 28489568256.0, "10045": 28489568256.0, "10050": 28489568256.0, "10055": 28489568256.0, "10060": 28489568256.0, "10065": 28489568256.0, "10070": 28489568256.0, "10075": 28489568256.0, "10080": 28489568256.0, "10085": 28489568256.0, "10090": 28489568256.0, "10095": 28489568256.0, "10100": 28489568256.0, "10105": 28489568256.0, "10110": 28489568256.0, "10115": 28489568256.0, "10120": 28489568256.0, "10125": 28489568256.0, "10130": 28489568256.0, "10135": 28489568256.0, "10140": 28489568256.0, "10145": 28489568256.0, "10150": 28489568256.0, "10155": 28489568256.0, "10160": 28489568256.0, "10165": 28489568256.0, "10170": 28489568256.0, "10175": 28489568256.0, "10180": 28489568256.0, "10185": 28489568256.0, "10190": 28489568256.0, "10195": 28489568256.0, "10200": 28489568256.0, "10205": 28489568256.0, "10210": 28489568256.0, "10215": 28489568256.0, "10220": 28489568256.0, "10225": 28489568256.0, "10230": 28489568256.0, "10235": 28489568256.0, "10240": 28489568256.0, "10245": 28489568256.0, "10250": 28489568256.0, "10255": 28489568256.0, "10260": 28489568256.0, "10265": 28489568256.0, "10270": 28489568256.0, "10275": 28489568256.0, "10280": 28489568256.0, "10285": 28489568256.0, "10290": 28489568256.0, "10295": 28489568256.0, "10300": 28489568256.0, "10305": 28489568256.0, "10310": 28489568256.0, "10315": 28489568256.0, "10320": 28489568256.0, "10325": 28489568256.0, "10330": 28489568256.0, "10335": 28489568256.0, "10340": 28489568256.0, "10345": 28489568256.0, "10350": 28489568256.0, "10355": 28489568256.0, "10360": 28489568256.0, "10365": 28489568256.0, "10370": 28489568256.0, "10375": 28489568256.0, "10380": 28489568256.0, "10385": 28489568256.0, "10390": 28489568256.0, "10395": 28489568256.0, "10400": 28489568256.0, "10405": 28489568256.0, "10410": 28489568256.0, "10415": 28489568256.0, "10420": 28489568256.0, "10425": 28489568256.0, "10430": 28489568256.0, "10435": 28489568256.0, "10440": 28489568256.0, "10445": 28489568256.0, "10450": 28489568256.0, "10455": 28489568256.0, "10460": 28489568256.0, "10465": 28489568256.0, "10470": 28489568256.0, "10475": 28489568256.0, "10480": 28489568256.0, "10485": 28489568256.0, "10490": 28489568256.0, "10495": 28489568256.0, "10500": 28489568256.0, "10505": 28489568256.0, "10510": 28489568256.0, "10515": 28489568256.0, "10520": 28489568256.0, "10525": 28489568256.0, "10530": 28489568256.0, "10535": 28489568256.0, "10540": 28489568256.0, "10545": 28489568256.0, "10550": 28489568256.0, "10555": 28489568256.0, "10560": 28489568256.0, "10565": 28489568256.0, "10570": 28489568256.0, "10575": 28489568256.0, "10580": 28489568256.0, "10585": 28489568256.0, "10590": 28489568256.0, "10595": 28489568256.0, "10600": 28489568256.0, "10605": 28489568256.0, "10610": 28489568256.0, "10615": 28489568256.0, "10620": 28489568256.0, "10625": 28489568256.0, "10630": 28489568256.0, "10635": 28489568256.0, "10640": 28489568256.0, "10645": 28489568256.0, "10650": 28489568256.0, "10655": 28489568256.0, "10660": 28489568256.0, "10665": 28489568256.0, "10670": 28489568256.0, "10675": 28489568256.0, "10680": 28489568256.0, "10685": 28489568256.0, "10690": 28489568256.0, "10695": 28489568256.0, "10700": 28489568256.0, "10705": 28489568256.0, "10710": 28489568256.0, "10715": 28489568256.0, "10720": 28489568256.0, "10725": 28489568256.0, "10730": 28489568256.0, "10735": 28489568256.0, "10740": 28489568256.0, "10745": 28489568256.0, "10750": 28489568256.0, "10755": 28489568256.0, "10760": 28489568256.0, "10765": 28489568256.0, "10770": 28489568256.0, "10775": 28489568256.0, "10780": 28489568256.0, "10785": 28489568256.0, "10790": 28489568256.0, "10795": 28489568256.0, "10800": 28489568256.0, "10805": 28489568256.0, "10810": 28489568256.0, "10815": 28489568256.0, "10820": 28489568256.0, "10825": 28489568256.0, "10830": 28489568256.0, "10835": 28489568256.0, "10840": 28489568256.0, "10845": 28489568256.0, "10850": 28489568256.0, "10855": 28489568256.0, "10860": 28489568256.0, "10865": 28489568256.0, "10870": 28489568256.0, "10875": 28489568256.0, "10880": 28489568256.0, "10885": 28489568256.0, "10890": 28489568256.0, "10895": 28489568256.0, "10900": 28489568256.0, "10905": 28489568256.0, "10910": 28489568256.0, "10915": 28489568256.0, "10920": 28489568256.0, "10925": 28489568256.0, "10930": 28489568256.0, "10935": 28489568256.0, "10940": 28489568256.0, "10945": 28489568256.0, "10950": 28489568256.0, "10955": 28489568256.0, "10960": 28489568256.0, "10965": 28489568256.0, "10970": 28489568256.0, "10975": 28489568256.0, "10980": 28489568256.0, "10985": 28489568256.0, "10990": 28489568256.0, "10995": 28489568256.0, "11000": 28489568256.0, "11005": 28489568256.0, "11010": 28489568256.0, "11015": 28489568256.0, "11020": 28489568256.0, "11025": 28489568256.0, "11030": 28489568256.0, "11035": 28489568256.0, "11040": 28489568256.0, "11045": 28489568256.0, "11050": 28489568256.0, "11055": 28489568256.0, "11060": 28489568256.0, "11065": 28489568256.0, "11070": 28489568256.0, "11075": 28489568256.0, "11080": 28489568256.0, "11085": 28489568256.0, "11090": 28489568256.0, "11095": 28489568256.0, "11100": 28489568256.0, "11105": 28489568256.0, "11110": 28489568256.0, "11115": 28489568256.0, "11120": 28489568256.0, "11125": 28489568256.0, "11130": 28489568256.0, "11135": 28489568256.0, "11140": 28489568256.0, "11145": 28489568256.0, "11150": 28489568256.0, "11155": 28489568256.0, "11160": 28489568256.0, "11165": 28489568256.0, "11170": 28489568256.0, "11175": 28489568256.0, "11180": 28489568256.0, "11185": 28489568256.0, "11190": 28489568256.0, "11195": 28489568256.0, "11200": 28489568256.0, "11205": 28489568256.0, "11210": 28489568256.0, "11215": 28489568256.0, "11220": 28489568256.0, "11225": 28489568256.0, "11230": 28489568256.0, "11235": 28489568256.0, "11240": 28489568256.0, "11245": 28489568256.0, "11250": 28489568256.0, "11255": 28489568256.0, "11260": 28489568256.0, "11265": 28489568256.0, "11270": 28489568256.0, "11275": 28489568256.0, "11280": 28489568256.0, "11285": 28489568256.0, "11290": 28489568256.0, "11295": 28489568256.0, "11300": 28489568256.0, "11305": 28489568256.0, "11310": 28489568256.0, "11315": 28489568256.0, "11320": 28489568256.0, "11325": 28489568256.0, "11330": 28489568256.0, "11335": 28489568256.0, "11340": 28489568256.0, "11345": 28489568256.0, "11350": 28489568256.0, "11355": 28489568256.0, "11360": 28489568256.0, "11365": 28489568256.0, "11370": 28489568256.0, "11375": 28489568256.0, "11380": 28489568256.0, "11385": 28489568256.0, "11390": 28489568256.0, "11395": 28489568256.0, "11400": 28489568256.0, "11405": 28489568256.0, "11410": 28489568256.0, "11415": 28489568256.0, "11420": 28489568256.0, "11425": 28489568256.0, "11430": 28489568256.0, "11435": 28489568256.0, "11440": 28489568256.0, "11445": 28489568256.0, "11450": 28489568256.0, "11455": 28489568256.0, "11460": 28489568256.0, "11465": 28489568256.0, "11470": 28489568256.0, "11475": 28489568256.0, "11480": 28489568256.0, "11485": 28489568256.0, "11490": 28489568256.0, "11495": 28489568256.0, "11500": 28489568256.0, "11505": 28489568256.0, "11510": 28489568256.0, "11515": 28489568256.0, "11520": 28489568256.0, "11525": 28489568256.0, "11530": 28489568256.0, "11535": 28489568256.0, "11540": 28489568256.0, "11545": 28489568256.0, "11550": 28489568256.0, "11555": 28489568256.0, "11560": 28489568256.0, "11565": 28489568256.0, "11570": 28489568256.0, "11575": 28489568256.0, "11580": 28489568256.0, "11585": 28489568256.0, "11590": 28489568256.0, "11595": 28489568256.0, "11600": 28489568256.0, "11605": 28489568256.0, "11610": 28489568256.0, "11615": 28489568256.0, "11620": 28489568256.0, "11625": 28489568256.0, "11630": 28489568256.0, "11635": 28489568256.0, "11640": 28489568256.0, "11645": 28489568256.0, "11650": 28489568256.0, "11655": 28489568256.0, "11660": 28489568256.0, "11665": 28489568256.0, "11670": 28489568256.0, "11675": 28489568256.0, "11680": 28489568256.0, "11685": 28489568256.0, "11690": 28489568256.0, "11695": 28489568256.0, "11700": 28489568256.0, "11705": 28489568256.0, "11710": 28489568256.0, "11715": 28489568256.0, "11720": 28489568256.0, "11725": 28489568256.0, "11730": 28489568256.0, "11735": 28489568256.0, "11740": 28489568256.0, "11745": 28489568256.0, "11750": 28489568256.0, "11755": 28489568256.0, "11760": 28489568256.0, "11765": 28489568256.0, "11770": 28489568256.0, "11775": 28489568256.0, "11780": 28489568256.0, "11785": 28489568256.0, "11790": 28489568256.0, "11795": 28489568256.0, "11800": 28489568256.0, "11805": 28489568256.0, "11810": 28489568256.0, "11815": 28489568256.0, "11820": 28489568256.0, "11825": 28489568256.0, "11830": 28489568256.0, "11835": 28489568256.0, "11840": 28489568256.0, "11845": 28489568256.0, "11850": 28489568256.0, "11855": 28489568256.0, "11860": 28489568256.0, "11865": 28489568256.0, "11870": 28489568256.0, "11875": 28489568256.0, "11880": 28489568256.0, "11885": 28489568256.0, "11890": 28489568256.0, "11895": 28489568256.0, "11900": 28489568256.0, "11905": 28489568256.0, "11910": 28489568256.0, "11915": 28489568256.0, "11920": 28489568256.0, "11925": 28489568256.0, "11930": 28489568256.0, "11935": 28489568256.0, "11940": 28489568256.0, "11945": 28489568256.0, "11950": 28489568256.0, "11955": 28489568256.0, "11960": 28489568256.0, "11965": 28489568256.0, "11970": 28489568256.0, "11975": 28489568256.0, "11980": 28489568256.0, "11985": 28489568256.0, "11990": 28489568256.0, "11995": 28489568256.0, "12000": 28489568256.0, "12005": 28489568256.0, "12010": 28489568256.0, "12015": 28489568256.0, "12020": 28489568256.0, "12025": 28489568256.0, "12030": 28489568256.0, "12035": 28489568256.0, "12040": 28489568256.0, "12045": 28489568256.0, "12050": 28489568256.0, "12055": 28489568256.0, "12060": 28489568256.0, "12065": 28489568256.0, "12070": 28489568256.0, "12075": 28489568256.0, "12080": 28489568256.0, "12085": 28489568256.0, "12090": 28489568256.0, "12095": 28489568256.0, "12100": 28489568256.0, "12105": 28489568256.0, "12110": 28489568256.0, "12115": 28489568256.0, "12120": 28489568256.0, "12125": 28489568256.0, "12130": 28489568256.0, "12135": 28489568256.0, "12140": 28489568256.0, "12145": 28489568256.0, "12150": 28489568256.0, "12155": 28489568256.0, "12160": 28489568256.0, "12165": 28489568256.0, "12170": 28489568256.0, "12175": 28489568256.0, "12180": 28489568256.0, "12185": 28489568256.0, "12190": 28489568256.0, "12195": 28489568256.0, "12200": 28489568256.0, "12205": 28489568256.0, "12210": 28489568256.0, "12215": 28489568256.0, "12220": 28489568256.0, "12225": 28489568256.0, "12230": 28489568256.0, "12235": 28489568256.0, "12240": 28489568256.0, "12245": 28489568256.0, "12250": 28489568256.0, "12255": 28489568256.0, "12260": 28489568256.0, "12265": 28489568256.0, "12270": 28489568256.0, "12275": 28489568256.0, "12280": 28489568256.0, "12285": 28489568256.0, "12290": 28489568256.0, "12295": 28489568256.0, "12300": 28489568256.0, "12305": 28489568256.0, "12310": 28489568256.0, "12315": 28489568256.0, "12320": 28489568256.0, "12325": 28489568256.0, "12330": 28489568256.0, "12335": 28489568256.0, "12340": 28489568256.0, "12345": 28489568256.0, "12350": 28489568256.0, "12355": 28489568256.0, "12360": 28489568256.0, "12365": 28489568256.0, "12370": 28489568256.0, "12375": 28489568256.0, "12380": 28489568256.0, "12385": 28489568256.0, "12390": 28489568256.0, "12395": 28489568256.0, "12400": 28489568256.0, "12405": 28489568256.0, "12410": 28489568256.0, "12415": 28489568256.0, "12420": 28489568256.0, "12425": 28489568256.0, "12430": 28489568256.0, "12435": 28489568256.0, "12440": 28489568256.0, "12445": 28489568256.0, "12450": 28489568256.0, "12455": 28489568256.0, "12460": 28489568256.0, "12465": 28489568256.0, "12470": 28489568256.0, "12475": 28489568256.0, "12480": 28489568256.0, "12485": 28489568256.0, "12490": 28489568256.0, "12495": 28489568256.0, "12500": 28489568256.0, "12505": 28489568256.0, "12510": 28489568256.0, "12515": 28489568256.0, "12520": 28489568256.0, "12525": 28489568256.0, "12530": 28489568256.0, "12535": 28489568256.0, "12540": 28489568256.0, "12545": 28489568256.0, "12550": 28489568256.0, "12555": 28489568256.0, "12560": 28489568256.0, "12565": 28489568256.0, "12570": 28489568256.0, "12575": 28489568256.0, "12580": 28489568256.0, "12585": 28489568256.0, "12590": 28489568256.0, "12595": 28489568256.0, "12600": 28489568256.0, "12605": 28489568256.0, "12610": 28489568256.0, "12615": 28489568256.0, "12620": 28489568256.0, "12625": 28489568256.0, "12630": 28489568256.0, "12635": 28489568256.0, "12640": 28489568256.0, "12645": 28489568256.0, "12650": 28489568256.0, "12655": 28489568256.0, "12660": 28489568256.0, "12665": 28489568256.0, "12670": 28489568256.0, "12675": 28489568256.0, "12680": 28489568256.0, "12685": 28489568256.0, "12690": 28489568256.0, "12695": 28489568256.0, "12700": 28489568256.0, "12705": 28489568256.0, "12710": 28489568256.0, "12715": 28489568256.0, "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}, "iteration-time": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": "nan", "5": "nan", "10": "nan", "15": "nan", "20": "nan", "25": "nan", "30": "nan", "35": "nan", "40": "nan", "45": "nan", "50": "nan", "55": "nan", "60": "nan", "65": "nan", "70": "nan", "75": "nan", "80": "nan", "85": "nan", "90": "nan", "95": "nan", "100": 3.5554, "105": "nan", "110": "nan", "115": "nan", "120": "nan", "125": "nan", "130": "nan", "135": "nan", "140": "nan", "145": "nan", "150": "nan", "155": "nan", "160": "nan", "165": "nan", "170": "nan", "175": "nan", "180": "nan", "185": "nan", "190": "nan", "195": "nan", "200": 3.47185, "205": "nan", "210": "nan", "215": "nan", "220": "nan", "225": "nan", "230": "nan", "235": "nan", "240": "nan", "245": "nan", "250": "nan", "255": "nan", "260": "nan", "265": "nan", "270": "nan", "275": "nan", "280": "nan", "285": "nan", "290": "nan", "295": "nan", "300": 3.45692, "305": "nan", "310": "nan", "315": "nan", "320": "nan", "325": "nan", "330": "nan", "335": "nan", "340": "nan", "345": "nan", "350": "nan", "355": "nan", "360": "nan", "365": "nan", "370": "nan", "375": "nan", "380": "nan", "385": "nan", "390": "nan", "395": "nan", "400": 3.45471, "405": "nan", "410": "nan", "415": "nan", "420": "nan", "425": "nan", "430": "nan", "435": "nan", "440": "nan", "445": "nan", "450": "nan", "455": "nan", "460": "nan", "465": "nan", "470": "nan", "475": "nan", "480": "nan", "485": "nan", "490": "nan", "495": "nan", "500": 3.45467, "505": "nan", "510": "nan", "515": "nan", "520": "nan", "525": "nan", "530": "nan", "535": "nan", "540": "nan", "545": "nan", "550": "nan", "555": "nan", "560": "nan", "565": "nan", "570": "nan", "575": "nan", "580": "nan", "585": "nan", "590": "nan", "595": "nan", "600": 3.4543, "605": "nan", "610": "nan", "615": "nan", "620": "nan", "625": "nan", "630": "nan", "635": "nan", "640": "nan", "645": "nan", "650": "nan", "655": "nan", "660": "nan", "665": "nan", "670": "nan", "675": "nan", "680": "nan", "685": "nan", "690": "nan", "695": "nan", "700": 3.45264, "705": "nan", "710": "nan", "715": "nan", "720": "nan", "725": "nan", "730": "nan", "735": "nan", "740": "nan", "745": "nan", "750": "nan", "755": "nan", "760": "nan", "765": "nan", "770": "nan", "775": "nan", "780": "nan", "785": "nan", "790": "nan", "795": "nan", "800": 3.45125, "805": "nan", "810": "nan", "815": "nan", "820": "nan", "825": "nan", "830": "nan", "835": "nan", "840": "nan", "845": "nan", "850": "nan", "855": "nan", "860": "nan", "865": "nan", "870": "nan", "875": "nan", "880": "nan", "885": "nan", "890": "nan", "895": "nan", "900": 3.44668, "905": "nan", "910": "nan", "915": "nan", "920": "nan", "925": "nan", "930": "nan", "935": "nan", "940": "nan", "945": "nan", "950": "nan", "955": "nan", "960": "nan", "965": "nan", "970": "nan", "975": "nan", "980": "nan", "985": "nan", "990": "nan", "995": "nan", "1000": 3.44035, "1005": "nan", "1010": "nan", "1015": "nan", "1020": "nan", "1025": "nan", "1030": "nan", "1035": "nan", "1040": "nan", "1045": "nan", "1050": "nan", "1055": "nan", "1060": "nan", "1065": "nan", "1070": "nan", "1075": "nan", "1080": "nan", "1085": "nan", "1090": "nan", "1095": "nan", "1100": 3.43442, "1105": "nan", "1110": "nan", "1115": "nan", "1120": "nan", "1125": "nan", "1130": "nan", "1135": "nan", "1140": "nan", "1145": "nan", "1150": "nan", "1155": "nan", "1160": "nan", "1165": "nan", "1170": "nan", "1175": "nan", "1180": "nan", "1185": "nan", "1190": "nan", "1195": "nan", "1200": 3.4306, "1205": "nan", "1210": "nan", "1215": "nan", "1220": "nan", "1225": "nan", "1230": "nan", "1235": "nan", "1240": "nan", "1245": "nan", "1250": "nan", "1255": "nan", "1260": "nan", "1265": "nan", "1270": "nan", "1275": "nan", "1280": "nan", "1285": "nan", "1290": "nan", "1295": "nan", "1300": 3.42464, "1305": "nan", "1310": "nan", "1315": "nan", "1320": "nan", "1325": "nan", "1330": "nan", "1335": "nan", "1340": "nan", "1345": "nan", "1350": "nan", "1355": "nan", "1360": "nan", "1365": "nan", "1370": "nan", "1375": "nan", "1380": "nan", "1385": "nan", "1390": "nan", "1395": "nan", "1400": 3.42155, "1405": "nan", "1410": "nan", "1415": "nan", "1420": "nan", "1425": "nan", "1430": "nan", "1435": "nan", "1440": "nan", "1445": "nan", "1450": "nan", "1455": "nan", "1460": "nan", "1465": "nan", "1470": "nan", "1475": "nan", "1480": "nan", "1485": "nan", "1490": "nan", "1495": "nan", "1500": 3.4201, "1505": "nan", "1510": "nan", "1515": "nan", "1520": "nan", "1525": "nan", "1530": "nan", "1535": "nan", "1540": "nan", "1545": "nan", "1550": "nan", "1555": "nan", "1560": "nan", "1565": "nan", "1570": "nan", "1575": "nan", "1580": "nan", "1585": "nan", "1590": "nan", "1595": "nan", "1600": 3.41703, "1605": "nan", "1610": "nan", "1615": "nan", "1620": "nan", "1625": "nan", "1630": "nan", "1635": "nan", "1640": "nan", "1645": "nan", "1650": "nan", "1655": "nan", "1660": "nan", "1665": "nan", "1670": "nan", "1675": "nan", "1680": "nan", "1685": "nan", "1690": "nan", "1695": "nan", "1700": 3.41482, "1705": "nan", "1710": "nan", "1715": "nan", "1720": "nan", "1725": "nan", "1730": "nan", "1735": "nan", "1740": "nan", "1745": "nan", "1750": "nan", "1755": "nan", "1760": "nan", "1765": "nan", "1770": "nan", "1775": "nan", "1780": "nan", "1785": "nan", "1790": "nan", "1795": "nan", "1800": 3.41352, "1805": "nan", "1810": "nan", "1815": "nan", "1820": "nan", "1825": "nan", "1830": "nan", "1835": "nan", "1840": "nan", "1845": "nan", "1850": "nan", "1855": "nan", "1860": "nan", "1865": "nan", "1870": "nan", "1875": "nan", "1880": "nan", "1885": "nan", "1890": "nan", "1895": "nan", "1900": 3.4128, "1905": "nan", "1910": "nan", "1915": "nan", "1920": "nan", "1925": "nan", "1930": "nan", "1935": "nan", "1940": "nan", "1945": "nan", "1950": "nan", "1955": "nan", "1960": "nan", "1965": "nan", "1970": "nan", "1975": "nan", "1980": "nan", "1985": "nan", "1990": "nan", "1995": "nan", "2000": 3.40994, "2005": "nan", "2010": "nan", "2015": "nan", "2020": "nan", "2025": "nan", "2030": "nan", "2035": "nan", "2040": "nan", "2045": "nan", "2050": "nan", "2055": "nan", "2060": "nan", "2065": "nan", "2070": "nan", "2075": "nan", "2080": "nan", "2085": "nan", "2090": "nan", "2095": "nan", "2100": 3.40878, "2105": "nan", "2110": "nan", "2115": "nan", "2120": "nan", "2125": "nan", "2130": "nan", "2135": "nan", "2140": "nan", "2145": "nan", "2150": "nan", "2155": "nan", "2160": "nan", "2165": "nan", "2170": "nan", "2175": "nan", "2180": "nan", "2185": "nan", "2190": "nan", "2195": "nan", "2200": 3.40862, "2205": "nan", "2210": "nan", "2215": "nan", "2220": "nan", "2225": "nan", "2230": "nan", "2235": "nan", "2240": "nan", "2245": "nan", "2250": "nan", "2255": "nan", "2260": "nan", "2265": "nan", "2270": "nan", "2275": "nan", "2280": "nan", "2285": "nan", "2290": "nan", "2295": "nan", "2300": 3.41459, "2305": "nan", "2310": "nan", "2315": "nan", "2320": "nan", "2325": "nan", "2330": "nan", "2335": "nan", "2340": "nan", "2345": "nan", "2350": "nan", "2355": "nan", "2360": "nan", "2365": "nan", "2370": "nan", "2375": "nan", "2380": "nan", "2385": "nan", "2390": "nan", "2395": "nan", "2400": 3.40662, "2405": "nan", "2410": "nan", "2415": "nan", "2420": "nan", "2425": "nan", "2430": "nan", "2435": "nan", "2440": "nan", "2445": "nan", "2450": "nan", "2455": "nan", "2460": "nan", "2465": "nan", "2470": "nan", "2475": "nan", "2480": "nan", "2485": "nan", "2490": "nan", "2495": "nan", "2500": 3.40543, "2505": "nan", "2510": "nan", "2515": "nan", "2520": "nan", "2525": "nan", "2530": "nan", "2535": "nan", "2540": "nan", "2545": "nan", "2550": "nan", "2555": "nan", "2560": "nan", "2565": "nan", "2570": "nan", "2575": "nan", "2580": "nan", "2585": "nan", "2590": "nan", "2595": "nan", "2600": 3.40484, "2605": "nan", "2610": "nan", "2615": "nan", "2620": "nan", "2625": "nan", "2630": "nan", "2635": "nan", "2640": "nan", "2645": "nan", "2650": "nan", "2655": "nan", "2660": "nan", "2665": "nan", "2670": "nan", "2675": "nan", "2680": "nan", "2685": "nan", "2690": "nan", "2695": "nan", "2700": 3.40448, "2705": "nan", "2710": "nan", "2715": "nan", "2720": "nan", "2725": "nan", "2730": "nan", "2735": "nan", "2740": "nan", "2745": "nan", "2750": "nan", "2755": "nan", "2760": "nan", "2765": "nan", "2770": "nan", "2775": "nan", "2780": "nan", "2785": "nan", "2790": "nan", "2795": "nan", "2800": 3.403, "2805": "nan", "2810": "nan", "2815": "nan", "2820": "nan", "2825": "nan", "2830": "nan", "2835": "nan", "2840": "nan", "2845": "nan", "2850": "nan", "2855": "nan", "2860": "nan", "2865": "nan", "2870": "nan", "2875": "nan", "2880": "nan", "2885": "nan", "2890": "nan", "2895": "nan", "2900": 3.40346, "2905": "nan", "2910": "nan", "2915": "nan", "2920": "nan", "2925": "nan", "2930": "nan", "2935": "nan", "2940": "nan", "2945": "nan", "2950": "nan", "2955": "nan", "2960": "nan", "2965": "nan", "2970": "nan", "2975": "nan", "2980": "nan", "2985": "nan", "2990": "nan", "2995": "nan", "3000": 3.4023, "3005": "nan", "3010": "nan", "3015": "nan", "3020": "nan", "3025": "nan", "3030": "nan", "3035": "nan", "3040": "nan", "3045": "nan", "3050": "nan", "3055": "nan", "3060": "nan", "3065": "nan", "3070": "nan", "3075": "nan", "3080": "nan", "3085": "nan", "3090": "nan", "3095": "nan", "3100": 3.40069, "3105": "nan", "3110": "nan", "3115": "nan", "3120": "nan", "3125": "nan", "3130": "nan", "3135": "nan", "3140": "nan", "3145": "nan", "3150": "nan", "3155": "nan", "3160": "nan", "3165": "nan", "3170": "nan", "3175": "nan", "3180": "nan", "3185": "nan", "3190": "nan", "3195": "nan", "3200": 3.40162, "3205": "nan", "3210": "nan", "3215": "nan", "3220": "nan", "3225": "nan", "3230": "nan", "3235": "nan", "3240": "nan", "3245": "nan", "3250": "nan", "3255": "nan", "3260": "nan", "3265": "nan", "3270": "nan", "3275": "nan", "3280": "nan", "3285": "nan", "3290": "nan", "3295": "nan", "3300": 3.40071, "3305": "nan", "3310": "nan", "3315": "nan", "3320": "nan", "3325": "nan", "3330": "nan", "3335": "nan", "3340": "nan", "3345": "nan", "3350": "nan", "3355": "nan", "3360": "nan", "3365": "nan", "3370": "nan", "3375": "nan", "3380": "nan", "3385": "nan", "3390": "nan", "3395": "nan", "3400": 3.40058, "3405": "nan", "3410": "nan", "3415": "nan", "3420": "nan", "3425": "nan", "3430": "nan", "3435": "nan", "3440": "nan", "3445": "nan", "3450": "nan", "3455": "nan", "3460": "nan", "3465": "nan", "3470": "nan", "3475": "nan", "3480": "nan", "3485": "nan", "3490": "nan", "3495": "nan", "3500": 3.39993, "3505": "nan", "3510": "nan", "3515": "nan", "3520": "nan", "3525": "nan", "3530": "nan", "3535": "nan", "3540": "nan", "3545": "nan", "3550": "nan", "3555": "nan", "3560": "nan", "3565": "nan", "3570": "nan", "3575": "nan", "3580": "nan", "3585": "nan", "3590": "nan", "3595": "nan", "3600": 3.62689, "3605": "nan", "3610": "nan", "3615": "nan", "3620": "nan", "3625": "nan", "3630": "nan", "3635": "nan", "3640": "nan", "3645": "nan", "3650": "nan", "3655": "nan", "3660": "nan", "3665": "nan", "3670": "nan", "3675": "nan", "3680": "nan", "3685": "nan", "3690": "nan", "3695": "nan", "3700": 3.40474, "3705": "nan", "3710": "nan", "3715": "nan", "3720": "nan", "3725": "nan", "3730": "nan", "3735": "nan", "3740": "nan", "3745": "nan", "3750": "nan", "3755": "nan", "3760": "nan", "3765": "nan", "3770": "nan", "3775": "nan", "3780": "nan", "3785": "nan", "3790": "nan", "3795": "nan", "3800": 3.40515, "3805": "nan", "3810": "nan", "3815": "nan", "3820": "nan", "3825": "nan", "3830": "nan", "3835": "nan", "3840": "nan", "3845": "nan", "3850": "nan", "3855": "nan", "3860": "nan", "3865": "nan", "3870": "nan", "3875": "nan", "3880": "nan", "3885": "nan", "3890": "nan", "3895": "nan", "3900": 3.40436, "3905": "nan", "3910": "nan", "3915": "nan", "3920": "nan", "3925": "nan", "3930": "nan", "3935": "nan", "3940": "nan", "3945": "nan", "3950": "nan", "3955": "nan", "3960": "nan", "3965": "nan", "3970": "nan", "3975": "nan", "3980": "nan", "3985": "nan", "3990": "nan", "3995": "nan", "4000": 3.40304, "4005": "nan", "4010": "nan", "4015": "nan", "4020": "nan", "4025": "nan", "4030": "nan", "4035": "nan", "4040": "nan", "4045": "nan", "4050": "nan", "4055": "nan", "4060": "nan", "4065": "nan", "4070": "nan", "4075": "nan", "4080": "nan", "4085": "nan", "4090": "nan", "4095": "nan", "4100": 3.40076, "4105": "nan", "4110": "nan", "4115": "nan", "4120": "nan", "4125": "nan", "4130": "nan", "4135": "nan", "4140": "nan", "4145": "nan", "4150": "nan", "4155": "nan", "4160": "nan", "4165": "nan", "4170": "nan", "4175": "nan", "4180": "nan", "4185": "nan", "4190": "nan", "4195": "nan", "4200": 3.40222, "4205": "nan", "4210": "nan", "4215": "nan", "4220": "nan", "4225": "nan", "4230": "nan", "4235": "nan", "4240": "nan", "4245": "nan", "4250": "nan", "4255": "nan", "4260": "nan", "4265": "nan", "4270": "nan", "4275": "nan", "4280": "nan", "4285": "nan", "4290": "nan", "4295": "nan", "4300": 3.40241, "4305": "nan", "4310": "nan", "4315": "nan", "4320": "nan", "4325": "nan", "4330": "nan", "4335": "nan", "4340": "nan", "4345": "nan", "4350": "nan", "4355": "nan", "4360": "nan", "4365": "nan", "4370": "nan", "4375": "nan", "4380": "nan", "4385": "nan", "4390": "nan", "4395": "nan", "4400": 3.40228, "4405": "nan", "4410": "nan", "4415": "nan", "4420": "nan", "4425": "nan", "4430": "nan", "4435": "nan", "4440": "nan", "4445": "nan", "4450": "nan", "4455": "nan", "4460": "nan", "4465": "nan", "4470": "nan", "4475": "nan", "4480": "nan", "4485": "nan", "4490": "nan", "4495": "nan", "4500": 3.40282, "4505": "nan", "4510": "nan", "4515": "nan", "4520": "nan", "4525": "nan", "4530": "nan", "4535": "nan", "4540": "nan", "4545": "nan", "4550": "nan", "4555": "nan", "4560": "nan", "4565": "nan", "4570": "nan", "4575": "nan", "4580": "nan", "4585": "nan", "4590": "nan", "4595": "nan", "4600": 3.40214, "4605": "nan", "4610": "nan", "4615": "nan", "4620": "nan", "4625": "nan", "4630": "nan", "4635": "nan", "4640": "nan", "4645": "nan", "4650": "nan", "4655": "nan", "4660": "nan", "4665": "nan", "4670": "nan", "4675": "nan", "4680": "nan", "4685": "nan", "4690": "nan", "4695": "nan", "4700": 3.40155, "4705": "nan", "4710": "nan", "4715": "nan", "4720": "nan", "4725": "nan", "4730": "nan", "4735": "nan", "4740": "nan", "4745": "nan", "4750": "nan", "4755": "nan", "4760": "nan", "4765": "nan", "4770": "nan", "4775": "nan", "4780": "nan", "4785": "nan", "4790": "nan", "4795": "nan", "4800": 3.4016, "4805": "nan", "4810": "nan", "4815": "nan", "4820": "nan", "4825": "nan", "4830": "nan", "4835": "nan", "4840": "nan", "4845": "nan", "4850": "nan", "4855": "nan", "4860": "nan", "4865": "nan", "4870": "nan", "4875": "nan", "4880": "nan", "4885": "nan", "4890": "nan", "4895": "nan", "4900": 3.40208, "4905": "nan", "4910": "nan", "4915": "nan", "4920": "nan", "4925": "nan", "4930": "nan", "4935": "nan", "4940": "nan", "4945": "nan", "4950": "nan", "4955": "nan", "4960": "nan", "4965": "nan", "4970": "nan", "4975": "nan", "4980": "nan", "4985": "nan", "4990": "nan", "4995": "nan", "5000": 3.40265, "5005": "nan", "5010": "nan", "5015": "nan", "5020": "nan", "5025": "nan", "5030": "nan", "5035": "nan", "5040": "nan", "5045": "nan", "5050": "nan", "5055": "nan", "5060": "nan", "5065": "nan", "5070": "nan", "5075": "nan", "5080": "nan", "5085": "nan", "5090": "nan", "5095": "nan", "5100": 3.3986, "5105": "nan", "5110": "nan", "5115": "nan", "5120": "nan", "5125": "nan", "5130": "nan", "5135": "nan", "5140": "nan", "5145": "nan", "5150": "nan", "5155": "nan", "5160": "nan", "5165": "nan", "5170": "nan", "5175": "nan", "5180": "nan", "5185": "nan", "5190": "nan", "5195": "nan", "5200": 3.39887, "5205": "nan", "5210": "nan", "5215": "nan", "5220": "nan", "5225": "nan", "5230": "nan", "5235": "nan", "5240": "nan", "5245": "nan", "5250": "nan", "5255": "nan", "5260": "nan", "5265": "nan", "5270": "nan", "5275": "nan", "5280": "nan", "5285": "nan", "5290": "nan", "5295": "nan", "5300": 3.3991, "5305": "nan", "5310": "nan", "5315": "nan", "5320": "nan", "5325": "nan", "5330": "nan", "5335": "nan", "5340": "nan", "5345": "nan", "5350": "nan", "5355": "nan", "5360": "nan", "5365": "nan", "5370": "nan", "5375": "nan", "5380": "nan", "5385": "nan", "5390": "nan", "5395": "nan", "5400": 3.40087, "5405": "nan", "5410": "nan", "5415": "nan", "5420": "nan", "5425": "nan", "5430": "nan", "5435": "nan", "5440": "nan", "5445": "nan", "5450": "nan", "5455": "nan", "5460": "nan", "5465": "nan", "5470": "nan", "5475": "nan", "5480": "nan", "5485": "nan", "5490": "nan", "5495": "nan", "5500": 3.40055, "5505": "nan", "5510": "nan", "5515": "nan", "5520": "nan", "5525": "nan", "5530": "nan", "5535": "nan", "5540": "nan", "5545": "nan", "5550": "nan", "5555": "nan", "5560": "nan", "5565": "nan", "5570": "nan", "5575": "nan", "5580": "nan", "5585": "nan", "5590": "nan", "5595": "nan", "5600": 3.40101, "5605": "nan", "5610": "nan", "5615": "nan", "5620": "nan", "5625": "nan", "5630": "nan", "5635": "nan", "5640": "nan", "5645": "nan", "5650": "nan", "5655": "nan", "5660": "nan", "5665": "nan", "5670": "nan", "5675": "nan", "5680": "nan", "5685": "nan", "5690": "nan", "5695": "nan", "5700": 3.4007, "5705": "nan", "5710": "nan", "5715": "nan", "5720": "nan", "5725": "nan", "5730": "nan", "5735": "nan", "5740": "nan", "5745": "nan", "5750": "nan", "5755": "nan", "5760": "nan", "5765": "nan", "5770": "nan", "5775": "nan", "5780": "nan", "5785": "nan", "5790": "nan", "5795": "nan", "5800": 3.40177, "5805": "nan", "5810": "nan", "5815": "nan", "5820": "nan", "5825": "nan", "5830": "nan", "5835": "nan", "5840": "nan", "5845": "nan", "5850": "nan", "5855": "nan", "5860": "nan", "5865": "nan", "5870": "nan", "5875": "nan", "5880": "nan", "5885": "nan", "5890": "nan", "5895": "nan", "5900": 3.40093, "5905": "nan", "5910": "nan", "5915": "nan", "5920": "nan", "5925": "nan", "5930": "nan", "5935": "nan", "5940": "nan", "5945": "nan", "5950": "nan", "5955": "nan", "5960": "nan", "5965": "nan", "5970": "nan", "5975": "nan", "5980": "nan", "5985": "nan", "5990": "nan", "5995": "nan", "6000": 3.40207, "6005": "nan", "6010": "nan", "6015": "nan", "6020": "nan", "6025": "nan", "6030": "nan", "6035": "nan", "6040": "nan", "6045": "nan", "6050": "nan", "6055": "nan", "6060": "nan", "6065": "nan", "6070": "nan", "6075": "nan", "6080": "nan", "6085": "nan", "6090": "nan", "6095": "nan", "6100": 3.40047, "6105": "nan", "6110": "nan", "6115": "nan", "6120": "nan", "6125": "nan", "6130": "nan", "6135": "nan", "6140": "nan", "6145": "nan", "6150": "nan", "6155": "nan", "6160": "nan", "6165": "nan", "6170": "nan", "6175": "nan", "6180": "nan", "6185": "nan", "6190": "nan", "6195": "nan", "6200": 3.40254, "6205": "nan", "6210": "nan", "6215": "nan", "6220": "nan", "6225": "nan", "6230": "nan", "6235": "nan", "6240": "nan", "6245": "nan", "6250": "nan", "6255": "nan", "6260": "nan", "6265": "nan", "6270": "nan", "6275": "nan", "6280": "nan", "6285": "nan", "6290": "nan", "6295": "nan", "6300": 3.4024, "6305": "nan", "6310": "nan", "6315": "nan", "6320": "nan", "6325": "nan", "6330": "nan", "6335": "nan", "6340": "nan", "6345": "nan", "6350": "nan", "6355": "nan", "6360": "nan", "6365": "nan", "6370": "nan", "6375": "nan", "6380": "nan", "6385": "nan", "6390": "nan", "6395": "nan", "6400": 3.40281, "6405": "nan", "6410": "nan", "6415": "nan", "6420": "nan", "6425": "nan", "6430": "nan", "6435": "nan", "6440": "nan", "6445": "nan", "6450": "nan", "6455": "nan", "6460": "nan", "6465": "nan", "6470": "nan", "6475": "nan", "6480": "nan", "6485": "nan", "6490": "nan", "6495": "nan", "6500": 3.40268, "6505": "nan", "6510": "nan", "6515": "nan", "6520": "nan", "6525": "nan", "6530": "nan", "6535": "nan", "6540": "nan", "6545": "nan", "6550": "nan", "6555": "nan", "6560": "nan", "6565": "nan", "6570": "nan", "6575": "nan", "6580": "nan", "6585": "nan", "6590": "nan", "6595": "nan", "6600": 3.40166, "6605": "nan", "6610": "nan", "6615": "nan", "6620": "nan", "6625": "nan", "6630": "nan", "6635": "nan", "6640": "nan", "6645": "nan", "6650": "nan", "6655": "nan", "6660": "nan", "6665": "nan", "6670": "nan", "6675": "nan", "6680": "nan", "6685": "nan", "6690": "nan", "6695": "nan", "6700": 3.40129, "6705": "nan", "6710": "nan", "6715": "nan", "6720": "nan", "6725": "nan", "6730": "nan", "6735": "nan", "6740": "nan", "6745": "nan", "6750": "nan", "6755": "nan", "6760": "nan", "6765": "nan", "6770": "nan", "6775": "nan", "6780": "nan", "6785": "nan", "6790": "nan", "6795": "nan", "6800": 3.40047, "6805": "nan", "6810": "nan", "6815": "nan", "6820": "nan", "6825": "nan", "6830": "nan", "6835": "nan", "6840": "nan", "6845": "nan", "6850": "nan", "6855": "nan", "6860": "nan", "6865": "nan", "6870": "nan", "6875": "nan", "6880": "nan", "6885": "nan", "6890": "nan", "6895": "nan", "6900": 3.40079, "6905": "nan", "6910": "nan", "6915": "nan", "6920": "nan", "6925": "nan", "6930": "nan", "6935": "nan", "6940": "nan", "6945": "nan", "6950": "nan", "6955": "nan", "6960": "nan", "6965": "nan", "6970": "nan", "6975": "nan", "6980": "nan", "6985": "nan", "6990": "nan", "6995": "nan", "7000": 3.40034, "7005": "nan", "7010": "nan", "7015": "nan", "7020": "nan", "7025": "nan", "7030": "nan", "7035": "nan", "7040": "nan", "7045": "nan", "7050": "nan", "7055": "nan", "7060": "nan", "7065": "nan", "7070": "nan", "7075": "nan", "7080": "nan", "7085": "nan", "7090": "nan", "7095": "nan", "7100": 3.39893, "7105": "nan", "7110": "nan", "7115": "nan", "7120": "nan", "7125": "nan", "7130": "nan", "7135": "nan", "7140": "nan", "7145": "nan", "7150": "nan", "7155": "nan", "7160": "nan", "7165": "nan", "7170": "nan", "7175": "nan", "7180": "nan", "7185": "nan", "7190": "nan", "7195": "nan", "7200": 3.40035, "7205": "nan", "7210": "nan", "7215": "nan", "7220": "nan", "7225": "nan", "7230": "nan", "7235": "nan", "7240": "nan", "7245": "nan", "7250": "nan", "7255": "nan", "7260": "nan", "7265": "nan", "7270": "nan", "7275": "nan", "7280": "nan", "7285": "nan", "7290": "nan", "7295": "nan", "7300": 3.39965, "7305": "nan", "7310": "nan", "7315": "nan", "7320": "nan", "7325": "nan", "7330": "nan", "7335": "nan", "7340": "nan", "7345": "nan", "7350": "nan", "7355": "nan", "7360": "nan", "7365": "nan", "7370": "nan", "7375": "nan", "7380": "nan", "7385": "nan", "7390": "nan", "7395": "nan", "7400": 3.40073, "7405": "nan", "7410": "nan", "7415": "nan", "7420": "nan", "7425": "nan", "7430": "nan", "7435": "nan", "7440": "nan", "7445": "nan", "7450": "nan", "7455": "nan", "7460": "nan", "7465": "nan", "7470": "nan", "7475": "nan", "7480": "nan", "7485": "nan", "7490": "nan", "7495": "nan", "7500": 3.40077, "7505": "nan", "7510": "nan", "7515": "nan", "7520": "nan", "7525": "nan", "7530": "nan", "7535": "nan", "7540": "nan", "7545": "nan", "7550": "nan", "7555": "nan", "7560": "nan", "7565": "nan", "7570": "nan", "7575": "nan", "7580": "nan", "7585": "nan", "7590": "nan", "7595": "nan", "7600": 3.39989, "7605": "nan", "7610": "nan", "7615": "nan", "7620": "nan", "7625": "nan", "7630": "nan", "7635": "nan", "7640": "nan", "7645": "nan", "7650": "nan", "7655": "nan", "7660": "nan", "7665": "nan", "7670": "nan", "7675": "nan", "7680": "nan", "7685": "nan", "7690": "nan", "7695": "nan", "7700": 3.3988, "7705": "nan", "7710": "nan", "7715": "nan", "7720": "nan", "7725": "nan", "7730": "nan", "7735": "nan", "7740": "nan", "7745": "nan", "7750": "nan", "7755": "nan", "7760": "nan", "7765": "nan", "7770": "nan", "7775": "nan", "7780": "nan", "7785": "nan", "7790": "nan", "7795": "nan", "7800": 3.3995, "7805": "nan", "7810": "nan", "7815": "nan", "7820": "nan", "7825": "nan", "7830": "nan", "7835": "nan", "7840": "nan", "7845": "nan", "7850": "nan", "7855": "nan", "7860": "nan", "7865": "nan", "7870": "nan", "7875": "nan", "7880": "nan", "7885": "nan", "7890": "nan", "7895": "nan", "7900": 3.39928, "7905": "nan", "7910": "nan", "7915": "nan", "7920": "nan", "7925": "nan", "7930": "nan", "7935": "nan", "7940": "nan", "7945": "nan", "7950": "nan", "7955": "nan", "7960": "nan", "7965": "nan", "7970": "nan", "7975": "nan", "7980": "nan", "7985": "nan", "7990": "nan", "7995": "nan", "8000": 3.39834, "8005": "nan", "8010": "nan", "8015": "nan", "8020": "nan", "8025": "nan", "8030": "nan", "8035": "nan", "8040": "nan", "8045": "nan", "8050": "nan", "8055": "nan", "8060": "nan", "8065": "nan", "8070": "nan", "8075": "nan", "8080": "nan", "8085": "nan", "8090": "nan", "8095": "nan", "8100": 3.39716, "8105": "nan", "8110": "nan", "8115": "nan", "8120": "nan", "8125": "nan", "8130": "nan", "8135": "nan", "8140": "nan", "8145": "nan", "8150": "nan", "8155": "nan", "8160": "nan", "8165": "nan", "8170": "nan", "8175": "nan", "8180": "nan", "8185": "nan", "8190": "nan", "8195": "nan", "8200": 3.39809, "8205": "nan", "8210": "nan", "8215": "nan", "8220": "nan", "8225": "nan", "8230": "nan", "8235": "nan", "8240": "nan", "8245": "nan", "8250": "nan", "8255": "nan", "8260": "nan", "8265": "nan", "8270": "nan", "8275": "nan", "8280": "nan", "8285": "nan", "8290": "nan", "8295": "nan", "8300": 3.39856, "8305": "nan", "8310": "nan", "8315": "nan", "8320": "nan", "8325": "nan", "8330": "nan", "8335": "nan", "8340": "nan", "8345": "nan", "8350": "nan", "8355": "nan", "8360": "nan", "8365": "nan", "8370": "nan", "8375": "nan", "8380": "nan", "8385": "nan", "8390": "nan", "8395": "nan", "8400": 3.39822, "8405": "nan", "8410": "nan", "8415": "nan", "8420": "nan", "8425": "nan", "8430": "nan", "8435": "nan", "8440": "nan", "8445": "nan", "8450": "nan", "8455": "nan", "8460": "nan", "8465": "nan", "8470": "nan", "8475": "nan", "8480": "nan", "8485": "nan", "8490": "nan", "8495": "nan", "8500": 3.39927, "8505": "nan", "8510": "nan", "8515": "nan", "8520": "nan", "8525": "nan", "8530": "nan", "8535": "nan", "8540": "nan", "8545": "nan", "8550": "nan", "8555": "nan", "8560": "nan", "8565": "nan", "8570": "nan", "8575": "nan", "8580": "nan", "8585": "nan", "8590": "nan", "8595": "nan", "8600": 3.40049, "8605": "nan", "8610": "nan", "8615": "nan", "8620": "nan", "8625": "nan", "8630": "nan", "8635": "nan", "8640": "nan", "8645": "nan", "8650": "nan", "8655": "nan", "8660": "nan", "8665": "nan", "8670": "nan", "8675": "nan", "8680": "nan", "8685": "nan", "8690": "nan", "8695": "nan", "8700": 3.39943, "8705": "nan", "8710": "nan", "8715": "nan", "8720": "nan", "8725": "nan", "8730": "nan", "8735": "nan", "8740": "nan", "8745": "nan", "8750": "nan", "8755": "nan", "8760": "nan", "8765": "nan", "8770": "nan", "8775": "nan", "8780": "nan", "8785": "nan", "8790": "nan", "8795": "nan", "8800": 3.3999, "8805": "nan", "8810": "nan", "8815": "nan", "8820": "nan", "8825": "nan", "8830": "nan", "8835": "nan", "8840": "nan", "8845": "nan", "8850": "nan", "8855": "nan", "8860": "nan", "8865": "nan", "8870": "nan", "8875": "nan", "8880": "nan", "8885": "nan", "8890": "nan", "8895": "nan", "8900": 3.39978, "8905": "nan", "8910": "nan", "8915": "nan", "8920": "nan", "8925": "nan", "8930": "nan", "8935": "nan", "8940": "nan", "8945": "nan", "8950": "nan", "8955": "nan", "8960": "nan", "8965": "nan", "8970": "nan", "8975": "nan", "8980": "nan", "8985": "nan", "8990": "nan", "8995": "nan", "9000": 3.40051, "9005": "nan", "9010": "nan", "9015": "nan", "9020": "nan", "9025": "nan", "9030": "nan", "9035": "nan", "9040": "nan", "9045": "nan", "9050": "nan", "9055": "nan", "9060": "nan", "9065": "nan", "9070": "nan", "9075": "nan", "9080": "nan", "9085": "nan", "9090": "nan", "9095": "nan", "9100": 3.39875, "9105": "nan", "9110": "nan", "9115": "nan", "9120": "nan", "9125": "nan", "9130": "nan", "9135": "nan", "9140": "nan", "9145": "nan", "9150": "nan", "9155": "nan", "9160": "nan", "9165": "nan", "9170": "nan", "9175": "nan", "9180": "nan", "9185": "nan", "9190": "nan", "9195": "nan", "9200": 3.39972, "9205": "nan", "9210": "nan", "9215": "nan", "9220": "nan", "9225": "nan", "9230": "nan", "9235": "nan", "9240": "nan", "9245": "nan", "9250": "nan", "9255": "nan", "9260": "nan", "9265": "nan", "9270": "nan", "9275": "nan", "9280": "nan", "9285": "nan", "9290": "nan", "9295": "nan", "9300": 3.40071, "9305": "nan", "9310": "nan", "9315": "nan", "9320": "nan", "9325": "nan", "9330": "nan", "9335": "nan", "9340": "nan", "9345": "nan", "9350": "nan", "9355": "nan", "9360": "nan", "9365": "nan", "9370": "nan", "9375": "nan", "9380": "nan", "9385": "nan", "9390": "nan", "9395": "nan", "9400": 3.40005, "9405": "nan", "9410": "nan", "9415": "nan", "9420": "nan", "9425": "nan", "9430": "nan", "9435": "nan", "9440": "nan", "9445": "nan", "9450": "nan", "9455": "nan", "9460": "nan", "9465": "nan", "9470": "nan", "9475": "nan", "9480": "nan", "9485": "nan", "9490": "nan", "9495": "nan", "9500": 3.39896, "9505": "nan", "9510": "nan", "9515": "nan", "9520": "nan", "9525": "nan", "9530": "nan", "9535": "nan", "9540": "nan", "9545": "nan", "9550": "nan", "9555": "nan", "9560": "nan", "9565": "nan", "9570": "nan", "9575": "nan", "9580": "nan", "9585": "nan", "9590": "nan", "9595": "nan", "9600": 3.39953, "9605": "nan", "9610": "nan", "9615": "nan", "9620": "nan", "9625": "nan", "9630": "nan", "9635": "nan", "9640": "nan", "9645": "nan", "9650": "nan", "9655": "nan", "9660": "nan", "9665": "nan", "9670": "nan", "9675": "nan", "9680": "nan", "9685": "nan", "9690": "nan", "9695": "nan", "9700": 3.39866, "9705": "nan", "9710": "nan", "9715": "nan", "9720": "nan", "9725": "nan", "9730": "nan", "9735": "nan", "9740": "nan", "9745": "nan", "9750": "nan", "9755": "nan", "9760": "nan", "9765": "nan", "9770": "nan", "9775": "nan", "9780": "nan", "9785": "nan", "9790": "nan", "9795": "nan", "9800": 3.40009, "9805": "nan", "9810": "nan", "9815": "nan", "9820": "nan", "9825": "nan", "9830": "nan", "9835": "nan", "9840": "nan", "9845": "nan", "9850": "nan", "9855": "nan", "9860": "nan", "9865": "nan", "9870": "nan", "9875": "nan", "9880": "nan", "9885": "nan", "9890": "nan", "9895": "nan", "9900": 3.39979, "9905": "nan", "9910": "nan", "9915": "nan", "9920": "nan", "9925": "nan", "9930": "nan", "9935": "nan", "9940": "nan", "9945": "nan", "9950": "nan", "9955": "nan", "9960": "nan", "9965": "nan", "9970": "nan", "9975": "nan", "9980": "nan", "9985": "nan", "9990": "nan", "9995": "nan", "10000": 3.3996, "10005": "nan", "10010": "nan", "10015": "nan", "10020": "nan", "10025": "nan", "10030": "nan", "10035": "nan", "10040": "nan", "10045": "nan", "10050": "nan", "10055": "nan", "10060": "nan", "10065": "nan", "10070": "nan", "10075": "nan", "10080": "nan", "10085": "nan", "10090": "nan", "10095": "nan", "10100": 3.39815, "10105": "nan", "10110": "nan", "10115": "nan", "10120": "nan", "10125": "nan", "10130": "nan", "10135": "nan", "10140": "nan", "10145": "nan", "10150": "nan", "10155": "nan", "10160": "nan", "10165": "nan", "10170": "nan", "10175": "nan", "10180": "nan", "10185": "nan", "10190": "nan", "10195": "nan", "10200": 3.3996, "10205": "nan", "10210": "nan", "10215": "nan", "10220": "nan", "10225": "nan", "10230": "nan", "10235": "nan", "10240": "nan", "10245": "nan", "10250": "nan", "10255": "nan", "10260": "nan", "10265": "nan", "10270": "nan", "10275": "nan", "10280": "nan", "10285": "nan", "10290": "nan", "10295": "nan", "10300": 3.40067, "10305": "nan", "10310": "nan", "10315": "nan", "10320": "nan", "10325": "nan", "10330": "nan", "10335": "nan", "10340": "nan", "10345": "nan", "10350": "nan", "10355": "nan", "10360": "nan", "10365": "nan", "10370": "nan", "10375": "nan", "10380": "nan", "10385": "nan", "10390": "nan", "10395": "nan", "10400": 3.39985, "10405": "nan", "10410": "nan", "10415": "nan", "10420": "nan", "10425": "nan", "10430": "nan", "10435": "nan", "10440": "nan", "10445": "nan", "10450": "nan", "10455": "nan", "10460": "nan", "10465": "nan", "10470": "nan", "10475": "nan", "10480": "nan", "10485": "nan", "10490": "nan", "10495": "nan", "10500": 3.39923, "10505": "nan", "10510": "nan", "10515": "nan", "10520": "nan", "10525": "nan", "10530": "nan", "10535": "nan", "10540": "nan", "10545": "nan", "10550": "nan", "10555": "nan", "10560": "nan", "10565": "nan", "10570": "nan", "10575": "nan", "10580": "nan", "10585": "nan", "10590": "nan", "10595": "nan", "10600": 3.40087, "10605": "nan", "10610": "nan", "10615": "nan", "10620": "nan", "10625": "nan", "10630": "nan", "10635": "nan", "10640": "nan", "10645": "nan", "10650": "nan", "10655": "nan", "10660": "nan", "10665": "nan", "10670": "nan", "10675": "nan", "10680": "nan", "10685": "nan", "10690": "nan", "10695": "nan", "10700": 3.40098, "10705": "nan", "10710": "nan", "10715": "nan", "10720": "nan", "10725": "nan", "10730": "nan", "10735": "nan", "10740": "nan", "10745": "nan", "10750": "nan", "10755": "nan", "10760": "nan", "10765": "nan", "10770": "nan", "10775": "nan", "10780": "nan", "10785": "nan", "10790": "nan", "10795": "nan", "10800": 3.39973, "10805": "nan", "10810": "nan", "10815": "nan", "10820": "nan", "10825": "nan", "10830": "nan", "10835": "nan", "10840": "nan", "10845": "nan", "10850": "nan", "10855": "nan", "10860": "nan", "10865": "nan", "10870": "nan", "10875": "nan", "10880": "nan", "10885": "nan", "10890": "nan", "10895": "nan", "10900": 3.66029, "10905": "nan", "10910": "nan", "10915": "nan", "10920": "nan", "10925": "nan", "10930": "nan", "10935": "nan", "10940": "nan", "10945": "nan", "10950": "nan", "10955": "nan", "10960": "nan", "10965": "nan", "10970": "nan", "10975": "nan", "10980": "nan", "10985": "nan", "10990": "nan", "10995": "nan", "11000": 3.38756, "11005": "nan", "11010": "nan", "11015": "nan", "11020": "nan", "11025": "nan", "11030": "nan", "11035": "nan", "11040": "nan", "11045": "nan", "11050": "nan", "11055": "nan", "11060": "nan", "11065": "nan", "11070": "nan", "11075": "nan", "11080": "nan", "11085": "nan", "11090": "nan", "11095": "nan", "11100": 3.38656, "11105": "nan", "11110": "nan", "11115": "nan", "11120": "nan", "11125": "nan", "11130": "nan", "11135": "nan", "11140": "nan", "11145": "nan", "11150": "nan", "11155": "nan", "11160": "nan", "11165": "nan", "11170": "nan", "11175": "nan", "11180": "nan", "11185": "nan", "11190": "nan", "11195": "nan", "11200": 3.38765, "11205": "nan", "11210": "nan", "11215": "nan", "11220": "nan", "11225": "nan", "11230": "nan", "11235": "nan", "11240": "nan", "11245": "nan", "11250": "nan", "11255": "nan", "11260": "nan", "11265": "nan", "11270": "nan", "11275": "nan", "11280": "nan", "11285": "nan", "11290": "nan", "11295": "nan", "11300": 3.38835, "11305": "nan", "11310": "nan", "11315": "nan", "11320": "nan", "11325": "nan", "11330": "nan", "11335": "nan", "11340": "nan", "11345": "nan", "11350": "nan", "11355": "nan", "11360": "nan", "11365": "nan", "11370": "nan", "11375": "nan", "11380": "nan", "11385": "nan", "11390": "nan", "11395": "nan", "11400": 3.38844, "11405": "nan", "11410": "nan", "11415": "nan", "11420": "nan", "11425": "nan", "11430": "nan", "11435": "nan", "11440": "nan", "11445": "nan", "11450": "nan", "11455": "nan", "11460": "nan", "11465": "nan", "11470": "nan", "11475": "nan", "11480": "nan", "11485": "nan", "11490": "nan", "11495": "nan", "11500": 3.38793, "11505": "nan", "11510": "nan", "11515": "nan", "11520": "nan", "11525": "nan", "11530": "nan", "11535": "nan", "11540": "nan", "11545": "nan", "11550": "nan", "11555": "nan", "11560": "nan", "11565": "nan", "11570": "nan", "11575": "nan", "11580": "nan", "11585": "nan", "11590": "nan", "11595": "nan", "11600": 3.38725, "11605": "nan", "11610": "nan", "11615": "nan", "11620": "nan", "11625": "nan", "11630": "nan", "11635": "nan", "11640": "nan", "11645": "nan", "11650": "nan", "11655": "nan", "11660": "nan", "11665": "nan", "11670": "nan", "11675": "nan", "11680": "nan", "11685": "nan", "11690": "nan", "11695": "nan", "11700": 3.38738, "11705": "nan", "11710": "nan", "11715": "nan", "11720": "nan", "11725": "nan", "11730": "nan", "11735": "nan", "11740": "nan", "11745": "nan", "11750": "nan", "11755": "nan", "11760": "nan", "11765": "nan", "11770": "nan", "11775": "nan", "11780": "nan", "11785": "nan", "11790": "nan", "11795": "nan", "11800": 3.38748, "11805": "nan", "11810": "nan", "11815": "nan", "11820": "nan", "11825": "nan", "11830": "nan", "11835": "nan", "11840": "nan", "11845": "nan", "11850": "nan", "11855": "nan", "11860": "nan", "11865": "nan", "11870": "nan", "11875": "nan", "11880": "nan", "11885": "nan", "11890": "nan", "11895": "nan", "11900": 3.38839, "11905": "nan", "11910": "nan", "11915": "nan", "11920": "nan", "11925": "nan", "11930": "nan", "11935": "nan", "11940": "nan", "11945": "nan", "11950": "nan", "11955": "nan", "11960": "nan", "11965": "nan", "11970": "nan", "11975": "nan", "11980": "nan", "11985": "nan", "11990": "nan", "11995": "nan", "12000": 3.38814, "12005": "nan", "12010": "nan", "12015": "nan", "12020": "nan", "12025": "nan", "12030": "nan", "12035": "nan", "12040": "nan", "12045": "nan", "12050": "nan", "12055": "nan", "12060": "nan", "12065": "nan", "12070": "nan", "12075": "nan", "12080": "nan", "12085": "nan", "12090": "nan", "12095": "nan", "12100": 3.38677, "12105": "nan", "12110": "nan", "12115": "nan", "12120": "nan", "12125": "nan", "12130": "nan", "12135": "nan", "12140": "nan", "12145": "nan", "12150": "nan", "12155": "nan", "12160": "nan", "12165": "nan", "12170": "nan", "12175": "nan", "12180": "nan", "12185": "nan", "12190": "nan", "12195": "nan", "12200": 3.38679, "12205": "nan", "12210": "nan", "12215": "nan", "12220": "nan", "12225": "nan", "12230": "nan", "12235": "nan", "12240": "nan", "12245": "nan", "12250": "nan", "12255": "nan", "12260": "nan", "12265": "nan", "12270": "nan", "12275": "nan", "12280": "nan", "12285": "nan", "12290": "nan", "12295": "nan", "12300": 3.38609, "12305": "nan", "12310": "nan", "12315": "nan", "12320": "nan", "12325": "nan", "12330": "nan", "12335": "nan", "12340": "nan", "12345": "nan", "12350": "nan", "12355": "nan", "12360": "nan", "12365": "nan", "12370": "nan", "12375": "nan", "12380": "nan", "12385": "nan", "12390": "nan", "12395": "nan", "12400": 3.38665, "12405": "nan", "12410": "nan", "12415": "nan", "12420": "nan", "12425": "nan", "12430": "nan", "12435": "nan", "12440": "nan", "12445": "nan", "12450": "nan", "12455": "nan", "12460": "nan", "12465": "nan", "12470": "nan", "12475": "nan", "12480": "nan", "12485": "nan", "12490": "nan", "12495": "nan", "12500": 3.38727, "12505": "nan", "12510": "nan", "12515": "nan", "12520": "nan", "12525": "nan", "12530": "nan", "12535": "nan", "12540": "nan", "12545": "nan", "12550": "nan", "12555": "nan", "12560": "nan", "12565": "nan", "12570": "nan", "12575": "nan", "12580": "nan", "12585": "nan", "12590": "nan", "12595": "nan", "12600": 3.38752, "12605": "nan", "12610": "nan", "12615": "nan", "12620": "nan", "12625": "nan", "12630": "nan", "12635": "nan", "12640": "nan", "12645": "nan", "12650": "nan", "12655": "nan", "12660": "nan", "12665": "nan", "12670": "nan", "12675": "nan", "12680": "nan", "12685": "nan", "12690": "nan", "12695": "nan", "12700": 3.38807, "12705": "nan", "12710": "nan", "12715": "nan", "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm_gb200/golden_values_lts_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm_gb200/golden_values_lts_dgx_a100.json
new file mode 100644
index 00000000000..379036a5d12
--- /dev/null
+++ b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm_gb200/golden_values_lts_dgx_a100.json
@@ -0,0 +1 @@
+{"lm loss": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": 12.98419, "5": 12.93854, "10": 12.06404, "15": 11.97879, "20": 10.53584, "25": 10.11956, "30": 9.7286, "35": 9.44176, "40": 9.23735, "45": 9.03751, "50": 8.85055, "55": 8.63984, "60": 8.64552, "65": 8.5235, "70": 8.478, "75": 8.3676, "80": 8.15365, "85": 8.15327, "90": 8.03161, "95": 7.95115, "100": 7.84388, "105": 7.72026, "110": 7.54733, "115": 7.45663, "120": 7.46178, "125": 7.47933, "130": 7.29598, "135": 7.26374, "140": 7.18817, "145": 7.03912, "150": 7.16593, "155": 7.03177, "160": 6.9153, "165": 6.90426, "170": 6.82329, "175": 6.89163, "180": 6.83373, "185": 6.7479, "190": 6.70023, "195": 6.63022, "200": 6.69259, "205": 6.62944, "210": 6.51496, "215": 6.49432, "220": 6.50588, "225": 6.49262, "230": 6.49489, "235": 6.48163, "240": 6.36127, "245": 6.36637, "250": 6.29142, "255": 6.45496, "260": 6.34018, "265": 6.26489, "270": 6.22894, "275": 6.23729, "280": 6.18967, "285": 6.20291, "290": 6.17708, "295": 6.13232, "300": 6.10682, "305": 6.00908, "310": 6.05252, "315": 6.05204, "320": 5.96463, "325": 5.91966, "330": 6.0284, "335": 5.99858, "340": 5.95439, "345": 5.93592, "350": 5.91905, "355": 5.86171, "360": 5.86274, "365": 5.82901, "370": 5.79297, "375": 5.80909, "380": 5.84904, "385": 5.80185, "390": 5.77686, "395": 5.68835, "400": 5.63684, "405": 5.66531, "410": 5.66265, "415": 5.72537, "420": 5.65949, "425": 5.68078, "430": 5.63808, "435": 5.57811, "440": 5.59213, "445": 5.5255, "450": 5.57672, "455": 5.51795, "460": 5.50422, "465": 5.5785, "470": 5.56096, "475": 5.48941, "480": 5.49418, "485": 5.50116, "490": 5.45593, "495": 5.46922, "500": 5.42565, "505": 5.41648, "510": 5.43832, "515": 5.42661, "520": 5.42935, "525": 5.29381, "530": 5.32998, "535": 5.31902, "540": 5.34353, "545": 5.39054, "550": 5.37019, "555": 5.19836, "560": 5.35374, "565": 5.30845, "570": 5.25365, "575": 5.31206, "580": 5.23746, "585": 5.20838, "590": 5.21845, "595": 5.23516, "600": 5.26955, "605": 5.21896, "610": 5.2345, "615": 5.20262, "620": 5.21905, "625": 5.1986, "630": 5.14834, "635": 5.13713, "640": 5.09747, "645": 5.1618, "650": 5.16552, "655": 5.15803, "660": 5.07055, "665": 5.10628, "670": 5.08585, "675": 5.05517, "680": 5.04384, "685": 5.03952, "690": 5.04772, "695": 5.00275, "700": 4.99979, "705": 4.95596, "710": 4.98915, "715": 4.92713, "720": 4.88403, "725": 4.84387, "730": 4.89609, "735": 4.86372, "740": 4.90625, "745": 4.76089, "750": 4.77974, "755": 4.83335, "760": 4.81568, "765": 4.76505, "770": 4.7234, "775": 4.6921, "780": 4.7432, "785": 4.78676, "790": 4.67828, "795": 4.64866, "800": 4.63612, "805": 4.64195, "810": 4.65077, "815": 4.63084, "820": 4.64482, "825": 4.61571, "830": 4.59387, "835": 4.56863, "840": 4.48927, "845": 4.49634, "850": 4.47352, "855": 4.52409, "860": 4.46251, "865": 4.52223, "870": 4.48175, "875": 4.39297, "880": 4.44361, "885": 4.4111, "890": 4.43659, "895": 4.41378, "900": 4.3919, "905": 4.35935, "910": 4.36354, "915": 4.34282, "920": 4.39946, "925": 4.41667, "930": 4.31531, "935": 4.30369, "940": 4.35342, "945": 4.30303, "950": 4.34463, "955": 4.26527, "960": 4.19654, "965": 4.29565, "970": 4.28642, "975": 4.23045, "980": 4.22564, "985": 4.1684, "990": 4.15028, "995": 4.17776, "1000": 4.23154, "1005": 4.18887, "1010": 4.16924, "1015": 4.11841, "1020": 4.15687, "1025": 4.21713, "1030": 4.13768, "1035": 4.10871, "1040": 4.118, "1045": 4.10979, "1050": 4.15517, "1055": 4.09572, "1060": 4.1093, "1065": 4.0737, "1070": 4.06643, "1075": 4.08655, "1080": 4.09213, "1085": 4.07259, "1090": 4.01677, "1095": 4.09659, "1100": 4.04859, "1105": 4.07084, "1110": 4.03263, "1115": 4.01534, "1120": 4.00246, "1125": 4.01029, "1130": 4.06057, "1135": 4.00668, "1140": 4.01528, "1145": 3.944, "1150": 4.02955, "1155": 4.00187, "1160": 3.97146, "1165": 3.86545, "1170": 3.91556, "1175": 3.95241, "1180": 3.9746, "1185": 3.9952, "1190": 3.94499, "1195": 3.96181, "1200": 3.8893, "1205": 3.8679, "1210": 3.98189, "1215": 3.81925, "1220": 3.86366, "1225": 3.81826, "1230": 3.91477, "1235": 3.90783, "1240": 3.89088, "1245": 3.7878, "1250": 3.83462, "1255": 3.86647, "1260": 3.90676, "1265": 3.79799, "1270": 3.87717, "1275": 3.83332, "1280": 3.82518, "1285": 3.84272, "1290": 3.88049, "1295": 3.84309, "1300": 3.82132, "1305": 3.8325, "1310": 3.81275, "1315": 3.80311, "1320": 3.81776, "1325": 3.71836, "1330": 3.7917, "1335": 3.74949, "1340": 3.75432, "1345": 3.75185, "1350": 3.72246, "1355": 3.76982, "1360": 3.73827, "1365": 3.72479, "1370": 3.73938, "1375": 3.73694, "1380": 3.75033, "1385": 3.75324, "1390": 3.66352, "1395": 3.74143, "1400": 3.73229, "1405": 3.66226, "1410": 3.65804, "1415": 3.66281, "1420": 3.69529, "1425": 3.71298, "1430": 3.68032, "1435": 3.66567, "1440": 3.62691, "1445": 3.68062, "1450": 3.66766, "1455": 3.62753, "1460": 3.65479, "1465": 3.66938, "1470": 3.62148, "1475": 3.69775, "1480": 3.65738, "1485": 3.66245, "1490": 3.63265, "1495": 3.61492, "1500": 3.6367, "1505": 3.68281, "1510": 3.54566, "1515": 3.609, "1520": 3.63666, "1525": 3.60047, "1530": 3.58381, "1535": 3.5969, "1540": 3.59479, "1545": 3.61209, "1550": 3.55084, "1555": 3.60011, "1560": 3.61885, "1565": 3.61562, "1570": 3.59161, "1575": 3.54997, "1580": 3.58264, "1585": 3.57028, "1590": 3.46649, "1595": 3.51916, "1600": 3.51438, "1605": 3.56399, "1610": 3.56561, "1615": 3.48914, "1620": 3.51872, "1625": 3.46038, "1630": 3.50057, "1635": 3.54094, "1640": 3.52781, "1645": 3.53869, "1650": 3.49558, "1655": 3.47362, "1660": 3.53058, "1665": 3.4694, "1670": 3.51604, "1675": 3.49294, "1680": 3.47856, "1685": 3.47172, "1690": 3.48567, "1695": 3.48595, "1700": 3.46718, "1705": 3.40405, "1710": 3.4904, "1715": 3.50163, "1720": 3.43217, "1725": 3.43389, "1730": 3.44993, "1735": 3.46203, "1740": 3.45825, "1745": 3.45364, "1750": 3.4018, "1755": 3.41959, "1760": 3.38236, "1765": 3.42143, "1770": 3.42391, "1775": 3.38829, "1780": 3.43133, "1785": 3.42531, "1790": 3.38061, "1795": 3.40413, "1800": 3.34505, "1805": 3.40411, "1810": 3.3443, "1815": 3.41721, "1820": 3.41298, "1825": 3.41394, "1830": 3.34507, "1835": 3.44327, "1840": 3.40026, "1845": 3.42225, "1850": 3.3737, "1855": 3.37379, "1860": 3.34718, "1865": 3.39774, "1870": 3.31941, "1875": 3.44184, "1880": 3.34583, "1885": 3.33834, "1890": 3.33625, "1895": 3.40145, "1900": 3.36496, "1905": 3.32271, "1910": 3.3368, "1915": 3.32331, "1920": 3.36555, "1925": 3.33897, "1930": 3.31467, "1935": 3.31123, "1940": 3.3762, "1945": 3.27313, "1950": 3.41256, "1955": 3.30548, "1960": 3.29785, "1965": 3.27936, "1970": 3.29856, "1975": 3.35566, "1980": 3.33591, "1985": 3.24229, "1990": 3.30028, "1995": 3.28316, "2000": 3.28226, "2005": 3.26535, "2010": 3.27352, "2015": 3.2365, "2020": 3.27501, "2025": 3.27742, "2030": 3.29047, "2035": 3.29225, "2040": 3.23929, "2045": 3.2491, "2050": 3.27456, "2055": 3.33373, "2060": 3.29439, "2065": 3.23928, "2070": 3.29227, "2075": 3.2624, "2080": 3.23888, "2085": 3.27623, "2090": 3.13844, "2095": 3.28702, "2100": 3.23688, "2105": 3.19771, "2110": 3.20277, "2115": 3.2382, "2120": 3.1813, "2125": 3.22025, "2130": 3.22372, "2135": 3.29164, "2140": 3.20319, "2145": 3.20474, "2150": 3.21527, "2155": 3.23198, "2160": 3.19331, "2165": 3.25562, "2170": 3.22466, "2175": 3.20239, "2180": 3.25351, "2185": 3.25349, "2190": 3.24601, "2195": 3.17461, "2200": 3.19592, "2205": 3.16638, "2210": 3.11756, "2215": 3.18822, "2220": 3.19477, "2225": 3.17024, "2230": 3.12954, "2235": 3.17891, "2240": 3.21397, "2245": 3.18137, "2250": 3.19957, "2255": 3.12336, "2260": 3.13509, "2265": 3.22906, "2270": 3.18616, "2275": 3.1529, "2280": 3.18794, "2285": 3.16881, "2290": 3.16738, "2295": 3.20792, "2300": 3.14094, "2305": 3.16694, "2310": 3.12982, "2315": 3.0574, "2320": 3.12693, "2325": 3.18302, "2330": 3.13863, "2335": 3.13173, "2340": 3.16429, "2345": 3.11609, "2350": 3.12193, "2355": 3.12102, "2360": 3.16057, "2365": 3.14069, "2370": 3.1566, "2375": 3.13306, "2380": 3.1089, "2385": 3.07598, "2390": 3.09423, "2395": 3.0923, "2400": 3.09049, "2405": 3.10312, "2410": 3.08738, "2415": 3.0859, "2420": 3.07232, "2425": 3.07515, "2430": 3.08315, "2435": 3.0666, "2440": 3.09293, "2445": 3.05738, "2450": 3.12724, "2455": 3.1668, "2460": 3.08258, "2465": 3.07557, "2470": 3.03871, "2475": 3.07295, "2480": 3.09401, "2485": 3.04631, "2490": 3.05694, "2495": 3.07352, "2500": 3.07374, "2505": 3.09757, "2510": 3.11425, "2515": 3.04708, "2520": 3.0804, "2525": 3.02461, "2530": 3.04704, "2535": 3.08948, "2540": 3.07683, "2545": 3.05223, "2550": 3.0346, "2555": 3.106, "2560": 3.05236, "2565": 3.10772, "2570": 3.01402, "2575": 3.04644, "2580": 3.07514, "2585": 3.02095, "2590": 3.06575, "2595": 2.99715, "2600": 3.06315, "2605": 3.04568, "2610": 3.04704, "2615": 3.05832, "2620": 2.98626, "2625": 3.00956, "2630": 3.03577, "2635": 3.05424, "2640": 3.01527, "2645": 3.05906, "2650": 3.02519, "2655": 3.00005, "2660": 3.00566, "2665": 3.03857, "2670": 2.9869, "2675": 2.96406, "2680": 2.99471, "2685": 3.00999, "2690": 3.00193, "2695": 2.99167, "2700": 3.02505, "2705": 2.98729, "2710": 2.97342, "2715": 2.96605, "2720": 3.02717, "2725": 3.00415, "2730": 3.02744, "2735": 2.98433, "2740": 2.98866, "2745": 3.00764, "2750": 3.01004, "2755": 2.97318, "2760": 3.00963, "2765": 3.01277, "2770": 2.98399, "2775": 2.99786, "2780": 3.0332, "2785": 2.95954, "2790": 2.96222, "2795": 2.97034, "2800": 2.98127, "2805": 2.93987, "2810": 2.983, "2815": 2.9543, "2820": 2.98172, "2825": 3.00651, "2830": 2.98887, "2835": 2.91425, "2840": 2.92476, "2845": 2.95821, "2850": 2.96948, "2855": 2.97557, "2860": 2.96429, "2865": 2.91763, "2870": 2.98977, "2875": 2.9248, "2880": 2.95945, "2885": 2.91822, "2890": 2.97801, "2895": 2.92798, "2900": 2.94851, "2905": 3.01025, "2910": 2.90885, "2915": 2.95254, "2920": 2.97185, "2925": 2.95597, "2930": 2.94982, "2935": 2.93409, "2940": 2.93873, "2945": 2.9082, "2950": 2.97314, "2955": 2.90745, "2960": 2.96696, "2965": 2.86293, "2970": 2.95934, "2975": 2.99626, "2980": 2.93691, "2985": 3.03654, "2990": 2.93999, "2995": 2.87144, "3000": 2.92971, "3005": 2.88913, "3010": 2.93432, "3015": 2.91276, "3020": 2.91232, "3025": 2.92393, "3030": 2.92045, "3035": 2.95576, "3040": 2.91549, "3045": 2.83698, "3050": 2.89779, "3055": 2.90445, "3060": 2.92488, "3065": 2.92719, "3070": 2.87846, "3075": 2.86687, "3080": 2.92258, "3085": 2.89966, "3090": 2.91396, "3095": 2.92199, "3100": 2.87182, "3105": 2.92802, "3110": 2.91863, "3115": 2.9403, "3120": 2.9518, "3125": 2.85751, "3130": 2.93151, "3135": 2.92635, "3140": 2.87276, "3145": 2.92375, "3150": 2.85484, "3155": 2.8485, "3160": 2.83637, "3165": 2.84033, "3170": 2.88849, "3175": 2.90239, "3180": 2.85043, "3185": 2.89154, "3190": 2.90425, "3195": 2.92291, "3200": 2.92063, "3205": 2.86226, "3210": 2.86891, "3215": 2.91036, "3220": 2.87807, "3225": 2.86421, "3230": 2.81586, "3235": 2.87333, "3240": 2.87156, "3245": 2.90126, "3250": 2.85213, "3255": 2.84713, "3260": 2.86187, "3265": 2.86787, "3270": 2.84284, "3275": 2.86909, "3280": 2.79921, "3285": 2.81195, "3290": 2.86578, "3295": 2.89818, "3300": 2.87821, "3305": 2.85947, "3310": 2.85651, "3315": 2.81024, "3320": 2.82854, "3325": 2.8296, "3330": 2.82729, "3335": 2.84636, "3340": 2.83138, "3345": 2.85448, "3350": 2.85382, "3355": 2.86154, "3360": 2.8033, "3365": 2.8523, "3370": 2.84468, "3375": 2.8454, "3380": 2.85596, "3385": 2.87939, "3390": 2.85981, "3395": 2.80661, "3400": 2.78593, "3405": 2.83043, "3410": 2.84374, "3415": 2.85748, "3420": 2.82452, "3425": 2.80324, "3430": 2.8249, "3435": 2.88819, "3440": 2.81499, "3445": 2.87505, "3450": 2.81623, "3455": 2.79143, "3460": 2.81576, "3465": 2.84564, "3470": 2.83662, "3475": 2.78218, "3480": 2.84288, "3485": 2.82039, "3490": 2.89267, "3495": 2.84976, "3500": 2.83815, "3505": 2.82178, "3510": 2.81512, "3515": 2.83844, "3520": 2.77805, "3525": 2.80344, "3530": 2.84921, "3535": 2.78161, "3540": 2.83674, "3545": 2.80973, "3550": 2.79547, "3555": 2.81863, "3560": 2.82078, "3565": 2.82666, "3570": 2.80124, "3575": 2.79967, "3580": 2.82045, "3585": 2.83294, "3590": 2.82697, "3595": 2.78815, "3600": 2.74915, "3605": 2.7873, "3610": 2.84627, "3615": 2.7499, "3620": 2.8026, "3625": 2.88969, "3630": 2.78406, "3635": 2.78796, "3640": 2.7812, "3645": 2.76636, "3650": 2.80296, "3655": 2.82044, "3660": 2.76688, "3665": 2.78201, "3670": 2.76782, "3675": 2.77556, "3680": 2.80739, "3685": 2.80119, "3690": 2.80155, "3695": 2.80812, "3700": 2.78648, "3705": 2.78205, "3710": 2.74897, "3715": 2.80626, "3720": 2.79273, "3725": 2.80205, "3730": 2.84576, "3735": 2.79732, "3740": 2.74582, "3745": 2.78748, "3750": 2.80839, "3755": 2.78946, "3760": 2.75624, "3765": 2.7529, "3770": 2.76053, "3775": 2.77003, "3780": 2.75736, "3785": 2.77603, "3790": 2.74395, "3795": 2.79193, "3800": 2.79977, "3805": 2.74927, "3810": 2.8012, "3815": 2.75973, "3820": 2.78378, "3825": 2.72979, "3830": 2.74412, "3835": 2.81283, "3840": 2.72346, "3845": 2.71207, "3850": 2.77208, "3855": 2.71862, "3860": 2.79999, "3865": 2.74819, "3870": 2.7788, "3875": 2.75589, "3880": 2.78591, "3885": 2.78703, "3890": 2.745, "3895": 2.79656, "3900": 2.76217, "3905": 2.7204, "3910": 2.74251, "3915": 2.75354, "3920": 2.7922, "3925": 2.779, "3930": 2.70818, "3935": 2.73844, "3940": 2.75097, "3945": 2.7403, "3950": 2.71938, "3955": 2.77595, "3960": 2.7594, "3965": 2.74095, "3970": 2.75582, "3975": 2.72197, "3980": 2.73797, "3985": 2.74501, "3990": 2.69182, "3995": 2.77927, "4000": 2.73467, "4005": 2.76845, "4010": 2.70729, "4015": 2.72138, "4020": 2.7543, "4025": 2.73235, "4030": 2.65702, "4035": 2.69604, "4040": 2.7478, "4045": 2.74602, "4050": 2.79108, "4055": 2.72052, "4060": 2.7132, "4065": 2.65055, "4070": 2.80438, "4075": 2.75742, "4080": 2.71911, "4085": 2.74853, "4090": 2.67511, "4095": 2.68776, "4100": 2.71367, "4105": 2.73756, "4110": 2.72971, "4115": 2.70539, "4120": 2.73419, "4125": 2.70252, "4130": 2.69562, "4135": 2.68553, "4140": 2.6792, "4145": 2.77962, "4150": 2.71146, "4155": 2.73921, "4160": 2.75831, "4165": 2.71903, "4170": 2.67374, "4175": 2.71681, "4180": 2.72466, "4185": 2.72363, "4190": 2.74097, "4195": 2.69297, "4200": 2.70506, "4205": 2.74554, "4210": 2.67484, "4215": 2.66292, "4220": 2.65885, "4225": 2.70625, "4230": 2.71947, "4235": 2.73163, "4240": 2.71, "4245": 2.69905, "4250": 2.71229, "4255": 2.6486, "4260": 2.7231, "4265": 2.74057, "4270": 2.72269, "4275": 2.68914, "4280": 2.70013, "4285": 2.73156, "4290": 2.68402, "4295": 2.69061, "4300": 2.69891, "4305": 2.70215, "4310": 2.73018, "4315": 2.70831, "4320": 2.70146, "4325": 2.70427, "4330": 2.71103, "4335": 2.68951, "4340": 2.69679, "4345": 2.72847, "4350": 2.67635, "4355": 2.69369, "4360": 2.71057, "4365": 2.78505, "4370": 2.73161, "4375": 2.7416, "4380": 2.70035, "4385": 2.69729, "4390": 2.6976, "4395": 2.75157, "4400": 2.6658, "4405": 2.66826, "4410": 2.68381, "4415": 2.70238, "4420": 2.70549, "4425": 2.72132, "4430": 2.69374, "4435": 2.68373, "4440": 2.69348, "4445": 2.67973, "4450": 2.65166, "4455": 2.66887, "4460": 2.6881, "4465": 2.69814, "4470": 2.67338, "4475": 2.68749, "4480": 2.65676, "4485": 2.69941, "4490": 2.65189, "4495": 2.70897, "4500": 2.70239, "4505": 2.69521, "4510": 2.64919, "4515": 2.70031, "4520": 2.66829, "4525": 2.66609, "4530": 2.67206, "4535": 2.67259, "4540": 2.70728, "4545": 2.65612, "4550": 2.70083, "4555": 2.67856, "4560": 2.65513, "4565": 2.63823, "4570": 2.64122, "4575": 2.6646, "4580": 2.68762, "4585": 2.68234, "4590": 2.61701, "4595": 2.66208, "4600": 2.6793, "4605": 2.67662, "4610": 2.65261, "4615": 2.66058, "4620": 2.65881, "4625": 2.68537, "4630": 2.67218, "4635": 2.64354, "4640": 2.69408, "4645": 2.64915, "4650": 2.70258, "4655": 2.71207, "4660": 2.67684, "4665": 2.68847, "4670": 2.67711, "4675": 2.68695, "4680": 2.66399, "4685": 2.65574, "4690": 2.70135, "4695": 2.65603, "4700": 2.67474, "4705": 2.64927, "4710": 2.67454, "4715": 2.6445, "4720": 2.72006, "4725": 2.62873, "4730": 2.64853, "4735": 2.68453, "4740": 2.64342, "4745": 2.65059, "4750": 2.63888, "4755": 2.65157, "4760": 2.65996, "4765": 2.64629, "4770": 2.62414, "4775": 2.65418, "4780": 2.65916, "4785": 2.69171, "4790": 2.64922, "4795": 2.67239, "4800": 2.62814, "4805": 2.64031, "4810": 2.66308, "4815": 2.64414, "4820": 2.66856, "4825": 2.64936, "4830": 2.61301, "4835": 2.64828, "4840": 2.65719, "4845": 2.63018, "4850": 2.62249, "4855": 2.59762, "4860": 2.65006, "4865": 2.62728, "4870": 2.63739, "4875": 2.61869, "4880": 2.62597, "4885": 2.62554, "4890": 2.6792, "4895": 2.659, "4900": 2.61682, "4905": 2.62007, "4910": 2.63757, "4915": 2.612, "4920": 2.65399, "4925": 2.64822, "4930": 2.57057, "4935": 2.65116, "4940": 2.63208, "4945": 2.63853, "4950": 2.6253, "4955": 2.61842, "4960": 2.61614, "4965": 2.65867, "4970": 2.59725, "4975": 2.65839, "4980": 2.61913, "4985": 2.63529, "4990": 2.65578, "4995": 2.58058, "5000": 2.66073, "5005": 2.66345, "5010": 2.68227, "5015": 2.63294, "5020": 2.63988, "5025": 2.68591, "5030": 2.64819, "5035": 2.61651, "5040": 2.61892, "5045": 2.60431, "5050": 2.62445, "5055": 2.64904, "5060": 2.64327, "5065": 2.68471, "5070": 2.60524, "5075": 2.61458, "5080": 2.60662, "5085": 2.60264, "5090": 2.59015, "5095": 2.64761, "5100": 2.6457, "5105": 2.60793, "5110": 2.66334, "5115": 2.62502, "5120": 2.67169, "5125": 2.62758, "5130": 2.61382, "5135": 2.61793, "5140": 2.57598, "5145": 2.62766, "5150": 2.63527, "5155": 2.6155, "5160": 2.65966, "5165": 2.58483, "5170": 2.58978, "5175": 2.61727, "5180": 2.60571, "5185": 2.62183, "5190": 2.62827, "5195": 2.67109, "5200": 2.59759, "5205": 2.60702, "5210": 2.60617, "5215": 2.64994, "5220": 2.58854, "5225": 2.55622, "5230": 2.63303, "5235": 2.61375, "5240": 2.63238, "5245": 2.62967, "5250": 2.5932, "5255": 2.61614, "5260": 2.55787, "5265": 2.59927, "5270": 2.59007, "5275": 2.61657, "5280": 2.61119, "5285": 2.60465, "5290": 2.63616, "5295": 2.62334, "5300": 2.58036, "5305": 2.59688, "5310": 2.60926, "5315": 2.58728, "5320": 2.61359, "5325": 2.6452, "5330": 2.6021, "5335": 2.58401, "5340": 2.56341, "5345": 2.65571, "5350": 2.61767, "5355": 2.58122, "5360": 2.59757, "5365": 2.61791, "5370": 2.61383, "5375": 2.62936, "5380": 2.57845, "5385": 2.56235, "5390": 2.58632, "5395": 2.61671, "5400": 2.60674, "5405": 2.54636, "5410": 2.61005, "5415": 2.59464, "5420": 2.61139, "5425": 2.625, "5430": 2.6265, "5435": 2.5765, "5440": 2.58814, "5445": 2.63079, "5450": 2.64534, "5455": 2.6101, "5460": 2.5903, "5465": 2.6052, "5470": 2.59691, "5475": 2.62442, "5480": 2.58886, "5485": 2.5915, "5490": 2.57494, "5495": 2.56951, "5500": 2.56777, "5505": 2.61617, "5510": 2.62567, "5515": 2.58374, "5520": 2.56088, "5525": 2.58583, "5530": 2.66352, "5535": 2.62035, "5540": 2.56974, "5545": 2.59511, "5550": 2.54965, "5555": 2.57459, "5560": 2.56453, "5565": 2.60605, "5570": 2.65634, "5575": 2.63541, "5580": 2.57175, "5585": 2.5945, "5590": 2.55928, "5595": 2.58276, "5600": 2.55517, "5605": 2.60104, "5610": 2.58094, "5615": 2.58249, "5620": 2.58036, "5625": 2.55089, "5630": 2.57187, "5635": 2.6324, "5640": 2.59548, "5645": 2.57131, "5650": 2.57908, "5655": 2.54814, "5660": 2.56269, "5665": 2.58485, "5670": 2.56733, "5675": 2.61086, "5680": 2.5279, "5685": 2.5673, "5690": 2.60169, "5695": 2.56032, "5700": 2.59817, "5705": 2.59773, "5710": 2.57825, "5715": 2.58556, "5720": 2.53545, "5725": 2.60141, "5730": 2.57545, "5735": 2.60941, "5740": 2.59452, "5745": 2.55803, "5750": 2.53903, "5755": 2.55715, "5760": 2.61525, "5765": 2.56038, "5770": 2.5412, "5775": 2.58539, "5780": 2.57794, "5785": 2.53954, "5790": 2.56402, "5795": 2.60037, "5800": 2.54586, "5805": 2.53368, "5810": 2.55739, "5815": 2.52458, "5820": 2.59565, "5825": 2.50396, "5830": 2.49812, "5835": 2.59879, "5840": 2.54027, "5845": 2.55119, "5850": 2.61034, "5855": 2.5078, "5860": 2.56077, "5865": 2.51887, "5870": 2.57531, "5875": 2.60847, "5880": 2.58556, "5885": 2.56695, "5890": 2.58322, "5895": 2.55606, "5900": 2.61399, "5905": 2.55561, "5910": 2.59622, "5915": 2.60833, "5920": 2.58814, "5925": 2.53547, "5930": 2.57614, "5935": 2.55624, "5940": 2.57051, "5945": 2.51758, "5950": 2.5549, "5955": 2.58364, "5960": 2.56774, "5965": 2.61916, "5970": 2.54972, "5975": 2.58058, "5980": 2.55777, "5985": 2.5587, "5990": 2.55442, "5995": 2.55786, "6000": 2.5541, "6005": 2.51959, "6010": 2.5612, "6015": 2.5234, "6020": 2.53535, "6025": 2.55845, "6030": 2.60267, "6035": 2.54217, "6040": 2.54797, "6045": 2.49021, "6050": 2.59325, "6055": 2.51806, "6060": 2.54365, "6065": 2.52512, "6070": 2.52866, "6075": 2.5355, "6080": 2.53264, "6085": 2.59598, "6090": 2.56805, "6095": 2.53269, "6100": 2.53973, "6105": 2.52014, "6110": 2.5548, "6115": 2.5854, "6120": 2.55512, "6125": 2.53857, "6130": 2.47414, "6135": 2.5559, "6140": 2.55561, "6145": 2.55497, "6150": 2.52352, "6155": 2.50641, "6160": 2.54021, "6165": 2.57308, "6170": 2.54839, "6175": 2.60252, "6180": 2.51088, "6185": 2.54973, "6190": 2.49031, "6195": 2.57918, "6200": 2.55088, "6205": 2.53612, "6210": 2.51902, "6215": 2.51336, "6220": 2.56408, "6225": 2.51443, "6230": 2.51059, "6235": 2.5608, "6240": 2.55012, "6245": 2.5247, "6250": 2.53168, "6255": 2.57932, "6260": 2.52354, "6265": 2.57262, "6270": 2.52446, "6275": 2.56295, "6280": 2.52126, "6285": 2.52066, "6290": 2.52096, "6295": 2.50424, "6300": 2.55533, "6305": 2.52427, "6310": 2.51089, "6315": 2.53744, "6320": 2.488, "6325": 2.59779, "6330": 2.55532, "6335": 2.51175, "6340": 2.51285, "6345": 2.55419, "6350": 2.55594, "6355": 2.52344, "6360": 2.52101, "6365": 2.48499, "6370": 2.53525, "6375": 2.49254, "6380": 2.55757, "6385": 2.57399, "6390": 2.50202, "6395": 2.55089, "6400": 2.50993, "6405": 2.5282, "6410": 2.5186, "6415": 2.52701, "6420": 2.54208, "6425": 2.53505, "6430": 2.57601, "6435": 2.54343, "6440": 2.53596, "6445": 2.52707, "6450": 2.53253, "6455": 2.5206, "6460": 2.51748, "6465": 2.56168, "6470": 2.5182, "6475": 2.52367, "6480": 2.48721, "6485": 2.52891, "6490": 2.50941, "6495": 2.49869, "6500": 2.52288, "6505": 2.49655, "6510": 2.5435, "6515": 2.51061, "6520": 2.51072, "6525": 2.4951, "6530": 2.54405, "6535": 2.53179, "6540": 2.52806, "6545": 2.56063, "6550": 2.50377, "6555": 2.5567, "6560": 2.50957, "6565": 2.52066, "6570": 2.5846, "6575": 2.52194, "6580": 2.49953, "6585": 2.50756, "6590": 2.5088, "6595": 2.49707, "6600": 2.48904, "6605": 2.53959, "6610": 2.47704, "6615": 2.56692, "6620": 2.53266, "6625": 2.50914, "6630": 2.51237, "6635": 2.47298, "6640": 2.53895, "6645": 2.59549, "6650": 2.51039, "6655": 2.5003, "6660": 2.57417, "6665": 2.52229, "6670": 2.56842, "6675": 2.46804, "6680": 2.54777, "6685": 2.53423, "6690": 2.51271, "6695": 2.48576, "6700": 2.52379, "6705": 2.51865, "6710": 2.49131, "6715": 2.51508, "6720": 2.51046, "6725": 2.51877, "6730": 2.51764, "6735": 2.48391, "6740": 2.51343, "6745": 2.49511, "6750": 2.55883, "6755": 2.47532, "6760": 2.54102, "6765": 2.48848, "6770": 2.51699, "6775": 2.50924, "6780": 2.53742, "6785": 2.47155, "6790": 2.54353, "6795": 2.49747, "6800": 2.52506, "6805": 2.51084, "6810": 2.50457, "6815": 2.52164, "6820": 2.48634, "6825": 2.5058, "6830": 2.54057, "6835": 2.5071, "6840": 2.50862, "6845": 2.52459, "6850": 2.47366, "6855": 2.5114, "6860": 2.50121, "6865": 2.48697, "6870": 2.55404, "6875": 2.47466, "6880": 2.55074, "6885": 2.47742, "6890": 2.5456, "6895": 2.50044, "6900": 2.48513, "6905": 2.49727, "6910": 2.51927, "6915": 2.51524, "6920": 2.53222, "6925": 2.54381, "6930": 2.49006, "6935": 2.52259, "6940": 2.49956, "6945": 2.46052, "6950": 2.48424, "6955": 2.52708, "6960": 2.51994, "6965": 2.493, "6970": 2.47002, "6975": 2.52124, "6980": 2.45179, "6985": 2.51469, "6990": 2.53168, "6995": 2.46205, "7000": 2.48721, "7005": 2.47085, "7010": 2.47458, "7015": 2.52239, "7020": 2.46871, "7025": 2.455, "7030": 2.48512, "7035": 2.47891, "7040": 2.5055, "7045": 2.52023, "7050": 2.52604, "7055": 2.44159, "7060": 2.47055, "7065": 2.4812, "7070": 2.49094, "7075": 2.49419, "7080": 2.5339, "7085": 2.48474, "7090": 2.47483, "7095": 2.50175, "7100": 2.51611, "7105": 2.48909, "7110": 2.48875, "7115": 2.50654, "7120": 2.47919, "7125": 2.4648, "7130": 2.48826, "7135": 2.51398, "7140": 2.49836, "7145": 2.49968, "7150": 2.51209, "7155": 2.50995, "7160": 2.47585, "7165": 2.45844, "7170": 2.5053, "7175": 2.50179, "7180": 2.50705, "7185": 2.48036, "7190": 2.46179, "7195": 2.46554, "7200": 2.5094, "7205": 2.48845, "7210": 2.44336, "7215": 2.48072, "7220": 2.44332, "7225": 2.51135, "7230": 2.50786, "7235": 2.48259, "7240": 2.47784, "7245": 2.49888, "7250": 2.50767, "7255": 2.49251, "7260": 2.45824, "7265": 2.45066, "7270": 2.47062, "7275": 2.50235, "7280": 2.49435, "7285": 2.42685, "7290": 2.48191, "7295": 2.48837, "7300": 2.41828, "7305": 2.44602, "7310": 2.44836, "7315": 2.49166, "7320": 2.48459, "7325": 2.45973, "7330": 2.49034, "7335": 2.47382, "7340": 2.46396, "7345": 2.4941, "7350": 2.5103, "7355": 2.49639, "7360": 2.48014, "7365": 2.46719, "7370": 2.4737, "7375": 2.44973, "7380": 2.49351, "7385": 2.48455, "7390": 2.47135, "7395": 2.477, "7400": 2.48731, "7405": 2.44334, "7410": 2.48259, "7415": 2.46975, "7420": 2.49233, "7425": 2.45642, "7430": 2.52265, "7435": 2.49165, "7440": 2.52011, "7445": 2.50649, "7450": 2.47346, "7455": 2.46263, "7460": 2.46431, "7465": 2.48535, "7470": 2.45144, "7475": 2.45815, "7480": 2.51387, "7485": 2.45117, "7490": 2.47472, "7495": 2.47889, "7500": 2.49782, "7505": 2.44041, "7510": 2.43588, "7515": 2.42224, "7520": 2.49392, "7525": 2.49662, "7530": 2.47567, "7535": 2.45999, "7540": 2.47286, "7545": 2.47379, "7550": 2.49017, "7555": 2.45369, "7560": 2.42765, "7565": 2.50931, "7570": 2.48419, "7575": 2.43912, "7580": 2.45901, "7585": 2.48319, "7590": 2.48184, "7595": 2.46424, "7600": 2.46287, "7605": 2.44744, "7610": 2.44904, "7615": 2.42801, "7620": 2.54382, "7625": 2.47981, "7630": 2.42404, "7635": 2.42598, "7640": 2.45333, "7645": 2.47382, "7650": 2.46343, "7655": 2.48648, "7660": 2.4512, "7665": 2.43238, "7670": 2.44036, "7675": 2.45618, "7680": 2.48615, "7685": 2.43166, "7690": 2.48187, "7695": 2.4547, "7700": 2.48168, "7705": 2.51497, "7710": 2.50597, "7715": 2.4453, "7720": 2.47122, "7725": 2.48144, "7730": 2.45802, "7735": 2.49851, "7740": 2.44105, "7745": 2.45131, "7750": 2.44125, "7755": 2.47226, "7760": 2.45319, "7765": 2.45587, "7770": 2.47059, "7775": 2.4538, "7780": 2.41754, "7785": 2.44561, "7790": 2.48633, "7795": 2.44163, "7800": 2.4703, "7805": 2.48729, "7810": 2.50584, "7815": 2.49634, "7820": 2.45047, "7825": 2.51645, "7830": 2.45243, "7835": 2.46939, "7840": 2.47968, "7845": 2.4632, "7850": 2.41717, "7855": 2.47372, "7860": 2.5075, "7865": 2.42692, "7870": 2.47598, "7875": 2.44861, "7880": 2.45488, "7885": 2.46205, "7890": 2.47234, "7895": 2.45238, "7900": 2.44179, "7905": 2.43688, "7910": 2.426, "7915": 2.48366, "7920": 2.47574, "7925": 2.4225, "7930": 2.47225, "7935": 2.45119, "7940": 2.424, "7945": 2.47139, "7950": 2.44612, "7955": 2.41981, "7960": 2.49324, "7965": 2.53135, "7970": 2.52904, "7975": 2.45162, "7980": 2.44412, "7985": 2.46923, "7990": 2.43335, "7995": 2.46941, "8000": 2.43681, "8005": 2.41849, "8010": 2.45817, "8015": 2.47029, "8020": 2.4825, "8025": 2.47607, "8030": 2.45105, "8035": 2.47127, "8040": 2.42237, "8045": 2.45296, "8050": 2.4482, "8055": 2.42424, "8060": 2.44611, "8065": 2.46189, "8070": 2.45993, "8075": 2.45852, "8080": 2.4462, "8085": 2.43951, "8090": 2.4255, "8095": 2.42228, "8100": 2.43833, "8105": 2.49489, "8110": 2.43712, "8115": 2.44338, "8120": 2.46793, "8125": 2.46836, "8130": 2.45375, "8135": 2.45377, "8140": 2.43826, "8145": 2.42645, "8150": 2.42267, "8155": 2.4863, "8160": 2.45488, "8165": 2.4436, "8170": 2.43613, "8175": 2.42318, "8180": 2.49704, "8185": 2.4269, "8190": 2.46935, "8195": 2.4585, "8200": 2.44837, "8205": 2.44505, "8210": 2.43355, "8215": 2.44056, "8220": 2.4358, "8225": 2.41015, "8230": 2.44176, "8235": 2.4652, "8240": 2.42683, "8245": 2.44875, "8250": 2.44572, "8255": 2.4414, "8260": 2.43328, "8265": 2.42888, "8270": 2.43309, "8275": 2.44149, "8280": 2.39924, "8285": 2.4396, "8290": 2.48206, "8295": 2.4474, "8300": 2.45867, "8305": 2.40775, "8310": 2.43515, "8315": 2.46058, "8320": 2.40073, "8325": 2.39433, "8330": 2.43694, "8335": 2.44522, "8340": 2.49104, "8345": 2.4478, "8350": 2.44978, "8355": 2.40749, "8360": 2.40123, "8365": 2.45247, "8370": 2.45194, "8375": 2.42273, "8380": 2.41941, "8385": 2.42385, "8390": 2.4346, "8395": 2.43997, "8400": 2.43946, "8405": 2.48993, "8410": 2.437, "8415": 2.43406, "8420": 2.41749, "8425": 2.43899, "8430": 2.46013, "8435": 2.40557, "8440": 2.45224, "8445": 2.45724, "8450": 2.40482, "8455": 2.46, "8460": 2.45352, "8465": 2.4356, "8470": 2.40868, "8475": 2.47827, "8480": 2.40162, "8485": 2.41279, "8490": 2.46472, "8495": 2.43598, "8500": 2.44504, "8505": 2.40324, "8510": 2.40392, "8515": 2.42848, "8520": 2.42568, "8525": 2.4921, "8530": 2.37281, "8535": 2.40013, "8540": 2.48538, "8545": 2.38102, "8550": 2.43805, "8555": 2.44969, "8560": 2.47001, "8565": 2.42051, "8570": 2.43016, "8575": 2.44673, "8580": 2.44107, "8585": 2.4206, "8590": 2.40199, "8595": 2.42668, "8600": 2.41302, "8605": 2.49039, "8610": 2.42012, "8615": 2.38793, "8620": 2.44803, "8625": 2.42382, "8630": 2.45408, "8635": 2.44896, "8640": 2.43494, "8645": 2.4736, "8650": 2.42011, "8655": 2.45238, "8660": 2.45516, "8665": 2.38524, "8670": 2.40887, "8675": 2.4285, "8680": 2.44765, "8685": 2.4292, "8690": 2.41052, "8695": 2.44169, "8700": 2.43279, "8705": 2.41873, "8710": 2.42723, "8715": 2.44835, "8720": 2.47546, "8725": 2.40943, "8730": 2.3906, "8735": 2.43279, "8740": 2.42918, "8745": 2.39996, "8750": 2.43474, "8755": 2.42434, "8760": 2.40027, "8765": 2.43461, "8770": 2.40495, "8775": 2.43724, "8780": 2.42018, "8785": 2.47035, "8790": 2.41985, "8795": 2.41766, "8800": 2.41504, "8805": 2.40682, "8810": 2.41125, "8815": 2.47313, "8820": 2.45249, "8825": 2.42464, "8830": 2.38678, "8835": 2.42251, "8840": 2.39423, "8845": 2.42654, "8850": 2.43398, "8855": 2.40351, "8860": 2.42788, "8865": 2.42683, "8870": 2.43891, "8875": 2.43765, "8880": 2.41181, "8885": 2.39358, "8890": 2.44536, "8895": 2.42582, "8900": 2.41128, "8905": 2.40287, "8910": 2.40178, "8915": 2.41899, "8920": 2.43296, "8925": 2.46577, "8930": 2.4143, "8935": 2.40926, "8940": 2.38869, "8945": 2.3922, "8950": 2.41756, "8955": 2.39479, "8960": 2.43345, "8965": 2.41805, "8970": 2.40426, "8975": 2.47319, "8980": 2.43981, "8985": 2.37288, "8990": 2.4072, "8995": 2.41768, "9000": 2.45601, "9005": 2.41214, "9010": 2.37503, "9015": 2.40997, "9020": 2.3985, "9025": 2.37013, "9030": 2.39819, "9035": 2.4235, "9040": 2.42034, "9045": 2.41944, "9050": 2.39591, "9055": 2.41865, "9060": 2.41835, "9065": 2.40424, "9070": 2.44544, "9075": 2.39609, "9080": 2.43408, "9085": 2.41287, "9090": 2.41241, "9095": 2.39607, "9100": 2.4007, "9105": 2.35839, "9110": 2.46492, "9115": 2.41577, "9120": 2.405, "9125": 2.458, "9130": 2.39476, "9135": 2.44823, "9140": 2.43426, "9145": 2.43038, "9150": 2.42521, "9155": 2.37592, "9160": 2.41896, "9165": 2.42461, "9170": 2.37306, "9175": 2.41974, "9180": 2.37725, "9185": 2.43877, "9190": 2.41309, "9195": 2.38919, "9200": 2.39263, "9205": 2.45158, "9210": 2.36253, "9215": 2.46302, "9220": 2.44836, "9225": 2.38351, "9230": 2.44641, "9235": 2.39743, "9240": 2.40233, "9245": 2.43533, "9250": 2.42643, "9255": 2.42876, "9260": 2.38508, "9265": 2.43838, "9270": 2.43511, "9275": 2.39568, "9280": 2.39071, "9285": 2.42371, "9290": 2.40431, "9295": 2.38539, "9300": 2.42356, "9305": 2.40497, "9310": 2.41467, "9315": 2.40958, "9320": 2.44478, "9325": 2.37057, "9330": 2.40175, "9335": 2.36064, "9340": 2.40709, "9345": 2.41361, "9350": 2.43767, "9355": 2.47543, "9360": 2.4371, "9365": 2.38709, "9370": 2.43617, "9375": 2.43237, "9380": 2.35257, "9385": 2.40086, "9390": 2.38081, "9395": 2.3885, "9400": 2.44434, "9405": 2.41269, "9410": 2.39825, "9415": 2.43724, "9420": 2.44459, "9425": 2.43199, "9430": 2.44775, "9435": 2.41468, "9440": 2.4757, "9445": 2.3748, "9450": 2.39361, "9455": 2.40348, "9460": 2.38595, "9465": 2.37795, "9470": 2.38158, "9475": 2.36591, "9480": 2.43326, "9485": 2.38642, "9490": 2.41941, "9495": 2.38123, "9500": 2.36329, "9505": 2.42967, "9510": 2.39832, "9515": 2.42896, "9520": 2.41821, "9525": 2.39098, "9530": 2.45386, "9535": 2.40282, "9540": 2.41807, "9545": 2.37646, "9550": 2.42227, "9555": 2.38928, "9560": 2.42056, "9565": 2.40473, "9570": 2.37127, "9575": 2.41035, "9580": 2.39592, "9585": 2.42186, "9590": 2.42769, "9595": 2.44835, "9600": 2.39217, "9605": 2.38532, "9610": 2.41965, "9615": 2.41204, "9620": 2.41154, "9625": 2.44483, "9630": 2.395, "9635": 2.40098, "9640": 2.44575, "9645": 2.41172, "9650": 2.39957, "9655": 2.37414, "9660": 2.42292, "9665": 2.38785, "9670": 2.38317, "9675": 2.35775, "9680": 2.39654, "9685": 2.39503, "9690": 2.4613, "9695": 2.38031, "9700": 2.37568, "9705": 2.38353, "9710": 2.36601, "9715": 2.38749, "9720": 2.4332, "9725": 2.44039, "9730": 2.42955, "9735": 2.38794, "9740": 2.37969, "9745": 2.42437, "9750": 2.3981, "9755": 2.4077, "9760": 2.41101, "9765": 2.36724, "9770": 2.4342, "9775": 2.40115, "9780": 2.36209, "9785": 2.40063, "9790": 2.40782, "9795": 2.35982, "9800": 2.39646, "9805": 2.40576, "9810": 2.40871, "9815": 2.37765, "9820": 2.37557, "9825": 2.40604, "9830": 2.42044, "9835": 2.38417, "9840": 2.41278, "9845": 2.36206, "9850": 2.39915, "9855": 2.39447, "9860": 2.38855, "9865": 2.37969, "9870": 2.38592, "9875": 2.38177, "9880": 2.45139, "9885": 2.39245, "9890": 2.35063, "9895": 2.31931, "9900": 2.3952, "9905": 2.42474, "9910": 2.35507, "9915": 2.36324, "9920": 2.41087, "9925": 2.3985, "9930": 2.38397, "9935": 2.35059, "9940": 2.38385, "9945": 2.3797, "9950": 2.40362, "9955": 2.44626, "9960": 2.4258, "9965": 2.35484, "9970": 2.40776, "9975": 2.38445, "9980": 2.3311, "9985": 2.4051, "9990": 2.39694, "9995": 2.39526, "10000": 2.36696, "10005": 2.37191, "10010": 2.38253, "10015": 2.44412, "10020": 2.36322, "10025": 2.38777, "10030": 2.38744, "10035": 2.40886, "10040": 2.40309, "10045": 2.38187, "10050": 2.34835, "10055": 2.36929, "10060": 2.41862, "10065": 2.37479, "10070": 2.423, "10075": 2.37255, "10080": 2.36332, "10085": 2.37108, "10090": 2.34595, "10095": 2.40124, "10100": 2.32131, "10105": 2.38462, "10110": 2.41161, "10115": 2.38754, "10120": 2.35903, "10125": 2.37092, "10130": 2.36022, "10135": 2.38243, "10140": 2.41271, "10145": 2.40607, "10150": 2.37539, "10155": 2.39406, "10160": 2.36042, "10165": 2.3842, "10170": 2.42368, "10175": 2.32305, "10180": 2.39352, "10185": 2.38375, "10190": 2.44188, "10195": 2.40305, "10200": 2.39015, "10205": 2.38822, "10210": 2.36991, "10215": 2.34364, "10220": 2.41795, "10225": 2.43037, "10230": 2.35534, "10235": 2.38619, "10240": 2.37205, "10245": 2.39035, "10250": 2.38646, "10255": 2.41642, "10260": 2.33424, "10265": 2.34821, "10270": 2.35061, "10275": 2.36955, "10280": 2.44822, "10285": 2.35754, "10290": 2.38389, "10295": 2.3723, "10300": 2.36828, "10305": 2.41571, "10310": 2.3877, "10315": 2.36013, "10320": 2.36624, "10325": 2.36267, "10330": 2.41312, "10335": 2.36153, "10340": 2.41836, "10345": 2.36889, "10350": 2.35672, "10355": 2.39532, "10360": 2.37312, "10365": 2.36074, "10370": 2.33915, "10375": 2.35562, "10380": 2.41668, "10385": 2.40583, "10390": 2.38095, "10395": 2.3576, "10400": 2.37548, "10405": 2.34822, "10410": 2.3381, "10415": 2.41431, "10420": 2.38041, "10425": 2.32562, "10430": 2.36014, "10435": 2.36947, "10440": 2.37091, "10445": 2.3601, "10450": 2.36009, "10455": 2.37843, "10460": 2.38055, "10465": 2.30313, "10470": 2.35764, "10475": 2.37919, "10480": 2.36187, "10485": 2.35885, "10490": 2.4119, "10495": 2.36592, "10500": 2.36273, "10505": 2.36961, "10510": 2.3816, "10515": 2.37346, "10520": 2.40183, "10525": 2.389, "10530": 2.39058, "10535": 2.35539, "10540": 2.40525, "10545": 2.35703, "10550": 2.37657, "10555": 2.35803, "10560": 2.3405, "10565": 2.37135, "10570": 2.37477, "10575": 2.35743, "10580": 2.37782, "10585": 2.36809, "10590": 2.37791, "10595": 2.37689, "10600": 2.33089, "10605": 2.3717, "10610": 2.36437, "10615": 2.36241, "10620": 2.34737, "10625": 2.41733, "10630": 2.36755, "10635": 2.32224, "10640": 2.36202, "10645": 2.42137, "10650": 2.3603, "10655": 2.31025, "10660": 2.34676, "10665": 2.39855, "10670": 2.31403, "10675": 2.41662, "10680": 2.35394, "10685": 2.28619, "10690": 2.38317, "10695": 2.32897, "10700": 2.38276, "10705": 2.38404, "10710": 2.34229, "10715": 2.38181, "10720": 2.32585, "10725": 2.35132, "10730": 2.34862, "10735": 2.3518, "10740": 2.31777, "10745": 2.33753, "10750": 2.33397, "10755": 2.40347, "10760": 2.36377, "10765": 2.33563, "10770": 2.36832, "10775": 2.38431, "10780": 2.36873, "10785": 2.39064, "10790": 2.34523, "10795": 2.38497, "10800": 2.32139, "10805": 2.39484, "10810": 2.37409, "10815": 2.35325, "10820": 2.34132, "10825": 2.36976, "10830": 2.33704, "10835": 2.347, "10840": 2.32835, "10845": 2.38518, "10850": 2.33101, "10855": 2.3619, "10860": 2.33071, "10865": 2.32059, "10870": 2.32243, "10875": 2.30352, "10880": 2.39351, "10885": 2.4031, "10890": 2.36036, "10895": 2.37107, "10900": 2.33074, "10905": 2.3132, "10910": 2.40682, "10915": 2.37125, "10920": 2.37395, "10925": 2.36178, "10930": 2.31785, "10935": 2.35984, "10940": 2.35325, "10945": 2.34609, "10950": 2.36031, "10955": 2.36317, "10960": 2.31043, "10965": 2.36158, "10970": 2.35657, "10975": 2.40855, "10980": 2.37438, "10985": 2.34339, "10990": 2.39794, "10995": 2.36358, "11000": 2.3367, "11005": 2.36071, "11010": 2.34173, "11015": 2.32617, "11020": 2.33259, "11025": 2.36576, "11030": 2.33866, "11035": 2.31271, "11040": 2.31651, "11045": 2.31822, "11050": 2.31809, "11055": 2.28913, "11060": 2.33868, "11065": 2.30809, "11070": 2.39307, "11075": 2.31783, "11080": 2.35458, "11085": 2.3363, "11090": 2.34633, "11095": 2.3718, "11100": 2.32898, "11105": 2.31552, "11110": 2.36164, "11115": 2.37242, "11120": 2.38066, "11125": 2.31475, "11130": 2.3499, "11135": 2.33245, "11140": 2.37221, "11145": 2.34935, "11150": 2.39376, "11155": 2.33996, "11160": 2.3647, "11165": 2.36272, "11170": 2.34016, "11175": 2.33391, "11180": 2.3723, "11185": 2.31334, "11190": 2.27753, "11195": 2.32826, "11200": 2.34711, "11205": 2.36211, "11210": 2.33021, "11215": 2.31819, "11220": 2.34183, "11225": 2.37052, "11230": 2.36505, "11235": 2.31953, "11240": 2.34095, "11245": 2.35575, "11250": 2.33117, "11255": 2.33525, "11260": 2.35568, "11265": 2.38813, "11270": 2.28869, "11275": 2.31457, "11280": 2.36943, "11285": 2.28996, "11290": 2.34586, "11295": 2.36394, "11300": 2.37973, "11305": 2.3352, "11310": 2.33011, "11315": 2.29941, "11320": 2.30556, "11325": 2.31524, "11330": 2.35389, "11335": 2.33811, "11340": 2.30715, "11345": 2.31278, "11350": 2.29619, "11355": 2.31948, "11360": 2.35099, "11365": 2.29275, "11370": 2.35148, "11375": 2.32687, "11380": 2.33972, "11385": 2.34686, "11390": 2.33406, "11395": 2.28592, "11400": 2.30948, "11405": 2.35421, "11410": 2.35465, "11415": 2.38449, "11420": 2.35035, "11425": 2.30772, "11430": 2.36681, "11435": 2.36061, "11440": 2.34732, "11445": 2.36229, "11450": 2.32147, "11455": 2.30472, "11460": 2.35122, "11465": 2.34332, "11470": 2.37307, "11475": 2.31161, "11480": 2.32382, "11485": 2.30846, "11490": 2.344, "11495": 2.40648, "11500": 2.33935, "11505": 2.34918, "11510": 2.36196, "11515": 2.32145, "11520": 2.3043, "11525": 2.36115, "11530": 2.31264, "11535": 2.3215, "11540": 2.34504, "11545": 2.34375, "11550": 2.3636, "11555": 2.32437, "11560": 2.34788, "11565": 2.33894, "11570": 2.34963, "11575": 2.29524, "11580": 2.32749, "11585": 2.35175, "11590": 2.3618, "11595": 2.33486, "11600": 2.3573, "11605": 2.32205, "11610": 2.35986, "11615": 2.36006, "11620": 2.29629, "11625": 2.27613, "11630": 2.32588, "11635": 2.34225, "11640": 2.30389, "11645": 2.30695, "11650": 2.32765, "11655": 2.35115, "11660": 2.33559, "11665": 2.32862, "11670": 2.29989, "11675": 2.29611, "11680": 2.3245, "11685": 2.335, "11690": 2.34341, "11695": 2.31618, "11700": 2.32498, "11705": 2.29971, "11710": 2.34418, "11715": 2.31355, "11720": 2.29829, "11725": 2.33913, "11730": 2.30649, "11735": 2.32753, "11740": 2.27272, "11745": 2.31682, "11750": 2.32586, "11755": 2.3504, "11760": 2.31072, "11765": 2.34059, "11770": 2.27741, "11775": 2.32428, "11780": 2.25457, "11785": 2.29632, "11790": 2.32161, "11795": 2.32148, "11800": 2.33441, "11805": 2.30224, "11810": 2.30358, "11815": 2.32855, "11820": 2.31768, "11825": 2.35799, "11830": 2.3157, "11835": 2.33627, "11840": 2.33933, "11845": 2.31625, "11850": 2.30279, "11855": 2.31274, "11860": 2.34249, "11865": 2.35695, "11870": 2.37806, "11875": 2.27972, "11880": 2.29164, "11885": 2.33582, "11890": 2.29146, "11895": 2.28855, "11900": 2.33216, "11905": 2.32131, "11910": 2.27663, "11915": 2.31183, "11920": 2.33333, "11925": 2.30178, "11930": 2.30522, "11935": 2.31612, "11940": 2.31579, "11945": 2.34065, "11950": 2.29853, "11955": 2.31246, "11960": 2.33665, "11965": 2.29384, "11970": 2.28198, "11975": 2.33532, "11980": 2.30525, "11985": 2.27675, "11990": 2.30305, "11995": 2.32936, "12000": 2.3243, "12005": 2.32389, "12010": 2.2876, "12015": 2.30885, "12020": 2.32609, "12025": 2.3339, "12030": 2.31036, "12035": 2.33548, "12040": 2.31504, "12045": 2.30961, "12050": 2.30776, "12055": 2.33205, "12060": 2.29709, "12065": 2.32997, "12070": 2.30215, "12075": 2.27525, "12080": 2.35012, "12085": 2.33805, "12090": 2.33153, "12095": 2.28064, "12100": 2.31437, "12105": 2.3082, "12110": 2.3286, "12115": 2.30272, "12120": 2.30476, "12125": 2.29299, "12130": 2.30209, "12135": 2.32731, "12140": 2.29487, "12145": 2.25587, "12150": 2.25942, "12155": 2.34139, "12160": 2.3564, "12165": 2.31895, "12170": 2.33202, "12175": 2.34176, "12180": 2.32919, "12185": 2.3392, "12190": 2.33408, "12195": 2.29739, "12200": 2.29971, "12205": 2.32199, "12210": 2.35582, "12215": 2.30236, "12220": 2.298, "12225": 2.24251, "12230": 2.33254, "12235": 2.33789, "12240": 2.32363, "12245": 2.28607, "12250": 2.27217, "12255": 2.33627, "12260": 2.31277, "12265": 2.34134, "12270": 2.31196, "12275": 2.31345, "12280": 2.31816, "12285": 2.28538, "12290": 2.30977, "12295": 2.26494, "12300": 2.32846, "12305": 2.26819, "12310": 2.2931, "12315": 2.38537, "12320": 2.3108, "12325": 2.32611, "12330": 2.30252, "12335": 2.32201, "12340": 2.33972, "12345": 2.36736, "12350": 2.34235, "12355": 2.3039, "12360": 2.31292, "12365": 2.32841, "12370": 2.29028, "12375": 2.29855, "12380": 2.29177, "12385": 2.28958, "12390": 2.24749, "12395": 2.30262, "12400": 2.29816, "12405": 2.30947, "12410": 2.30218, "12415": 2.28121, "12420": 2.31576, "12425": 2.29993, "12430": 2.31452, "12435": 2.29916, "12440": 2.33133, "12445": 2.31918, "12450": 2.30543, "12455": 2.24083, "12460": 2.33478, "12465": 2.36325, "12470": 2.27585, "12475": 2.27386, "12480": 2.29137, "12485": 2.30584, "12490": 2.33022, "12495": 2.26913, "12500": 2.3199, "12505": 2.33541, "12510": 2.3559, "12515": 2.27055, "12520": 2.31974, "12525": 2.28748, "12530": 2.32122, "12535": 2.27213, "12540": 2.28628, "12545": 2.29073, "12550": 2.31647, "12555": 2.32346, "12560": 2.30163, "12565": 2.33608, "12570": 2.2774, "12575": 2.30021, "12580": 2.31011, "12585": 2.29127, "12590": 2.33476, "12595": 2.32359, "12600": 2.28129, "12605": 2.31883, "12610": 2.36317, "12615": 2.30591, "12620": 2.33262, "12625": 2.33055, "12630": 2.2973, "12635": 2.33522, "12640": 2.29483, "12645": 2.27955, "12650": 2.32509, "12655": 2.2649, "12660": 2.34164, "12665": 2.31762, "12670": 2.30944, "12675": 2.31773, "12680": 2.27484, "12685": 2.36538, "12690": 2.30258, "12695": 2.3316, "12700": 2.29242, "12705": 2.3066, "12710": 2.30648, "12715": 2.28622, "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}, "num-zeros": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": 956235968.0, "5": 967337088.0, "10": 971385344.0, "15": 946451584.0, "20": 961454144.0, "25": 1083495040.0, "30": 1210886784.0, "35": 1297400576.0, "40": 1271467008.0, "45": 1175317120.0, "50": 1127123712.0, "55": 1082697856.0, "60": 1044925888.0, "65": 1035741120.0, "70": 1015001472.0, "75": 1009854336.0, "80": 1028703936.0, "85": 1019290368.0, "90": 990809344.0, "95": 964848896.0, "100": 973471680.0, "105": 983119424.0, "110": 977565504.0, "115": 977218048.0, "120": 961220416.0, "125": 943671872.0, "130": 976736960.0, "135": 964989632.0, "140": 963116864.0, "145": 976472832.0, "150": 921307328.0, "155": 968129664.0, "160": 956322560.0, "165": 959833728.0, "170": 974347456.0, "175": 949087808.0, "180": 946701504.0, "185": 972012288.0, "190": 969080384.0, "195": 985148288.0, "200": 945763584.0, "205": 958347008.0, "210": 979438848.0, "215": 967497024.0, "220": 956425216.0, "225": 962393664.0, "230": 948174848.0, "235": 965217152.0, "240": 966068608.0, "245": 969152000.0, "250": 974439936.0, "255": 925058944.0, "260": 965631168.0, "265": 970669376.0, "270": 959136512.0, "275": 953999296.0, "280": 963425664.0, "285": 945783936.0, "290": 974119040.0, "295": 966704000.0, "300": 967153024.0, "305": 964512128.0, "310": 940359680.0, "315": 967399488.0, "320": 969007936.0, "325": 980555456.0, "330": 972089216.0, "335": 946860544.0, "340": 966592384.0, "345": 973019520.0, "350": 973912320.0, "355": 963257280.0, "360": 948348032.0, "365": 964815680.0, "370": 962952960.0, "375": 958450432.0, "380": 947149312.0, "385": 955992704.0, "390": 945401920.0, "395": 970418112.0, "400": 979775872.0, "405": 968349696.0, "410": 970067456.0, "415": 953155072.0, "420": 943562368.0, "425": 954772160.0, "430": 962659264.0, "435": 977076736.0, "440": 954808960.0, "445": 971890816.0, "450": 963511680.0, "455": 973132288.0, "460": 983708288.0, "465": 945281152.0, "470": 942057856.0, "475": 967007104.0, "480": 966098304.0, "485": 976406464.0, "490": 962540544.0, "495": 945459840.0, "500": 964454784.0, "505": 986003136.0, "510": 965679232.0, "515": 943408000.0, "520": 945020992.0, "525": 971262848.0, "530": 971888448.0, "535": 979136128.0, "540": 969531136.0, "545": 954123392.0, "550": 951268096.0, "555": 987224704.0, "560": 960424256.0, "565": 966614016.0, "570": 975728384.0, "575": 927223872.0, "580": 970694784.0, "585": 961176832.0, "590": 972965888.0, "595": 963684352.0, "600": 937077568.0, "605": 951474304.0, "610": 963361024.0, "615": 970007808.0, "620": 976466176.0, "625": 949580224.0, "630": 954443904.0, "635": 986043648.0, "640": 980977216.0, "645": 955008832.0, "650": 958547584.0, "655": 951651520.0, "660": 961042688.0, "665": 967554944.0, "670": 962511744.0, "675": 968338048.0, "680": 965622016.0, "685": 962869120.0, "690": 961921152.0, "695": 954770816.0, "700": 970338624.0, "705": 945517632.0, "710": 943884160.0, "715": 973356416.0, "720": 968367744.0, "725": 978491840.0, "730": 952192896.0, "735": 948810368.0, "740": 955631744.0, "745": 975870976.0, "750": 981237376.0, "755": 962160384.0, "760": 951962496.0, "765": 967347840.0, "770": 976146048.0, "775": 970547840.0, "780": 977538112.0, "785": 931529792.0, "790": 960443264.0, "795": 964581248.0, "800": 967022144.0, "805": 962318720.0, "810": 940976448.0, "815": 949040000.0, "820": 953185856.0, "825": 954501760.0, "830": 976443776.0, "835": 956075520.0, "840": 948406912.0, "845": 965154752.0, "850": 966026688.0, "855": 960904960.0, "860": 976025088.0, "865": 938160704.0, "870": 966414976.0, "875": 972314304.0, "880": 963119744.0, "885": 967742720.0, "890": 949969728.0, "895": 960017792.0, "900": 974230272.0, "905": 963968064.0, "910": 958436608.0, "915": 956353920.0, "920": 943970368.0, "925": 960829184.0, "930": 978849664.0, "935": 971071104.0, "940": 960907136.0, "945": 945064832.0, "950": 957427008.0, "955": 979039232.0, "960": 983589376.0, "965": 966165696.0, "970": 951225600.0, "975": 961578240.0, "980": 968070528.0, "985": 968992256.0, "990": 984392960.0, "995": 953292032.0, "1000": 934783040.0, "1005": 960146944.0, "1010": 971539968.0, "1015": 985186560.0, "1020": 962781312.0, "1025": 935010304.0, "1030": 974676416.0, "1035": 964993152.0, "1040": 980464704.0, "1045": 960830464.0, "1050": 955200064.0, "1055": 957782272.0, "1060": 967746944.0, "1065": 967115264.0, "1070": 966600192.0, "1075": 950056448.0, "1080": 954506240.0, "1085": 967250688.0, "1090": 977131776.0, "1095": 961235584.0, "1100": 979611776.0, "1105": 953366144.0, "1110": 965955584.0, "1115": 966984832.0, "1120": 970350528.0, "1125": 965706496.0, "1130": 954942208.0, "1135": 965843328.0, "1140": 965175360.0, "1145": 970988544.0, "1150": 955558144.0, "1155": 930579520.0, "1160": 957773824.0, "1165": 978125312.0, "1170": 974302848.0, "1175": 973057600.0, "1180": 973081216.0, "1185": 947342912.0, "1190": 964792960.0, "1195": 953134464.0, "1200": 972844672.0, "1205": 988479168.0, "1210": 931123264.0, "1215": 968647744.0, "1220": 969164992.0, "1225": 975950272.0, "1230": 967333824.0, "1235": 943446336.0, "1240": 955852416.0, "1245": 981507264.0, "1250": 966111104.0, "1255": 973672960.0, "1260": 946495744.0, "1265": 963997568.0, "1270": 960487744.0, "1275": 973616640.0, "1280": 961114880.0, "1285": 957581440.0, "1290": 952528640.0, "1295": 971612608.0, "1300": 968863744.0, "1305": 963737728.0, "1310": 963337664.0, "1315": 943555840.0, "1320": 966307008.0, "1325": 989786496.0, "1330": 969506304.0, "1335": 972303232.0, "1340": 972271104.0, "1345": 960659072.0, "1350": 968639552.0, "1355": 955854208.0, "1360": 971819456.0, "1365": 960387456.0, "1370": 948791360.0, "1375": 973533696.0, "1380": 953470208.0, "1385": 969144704.0, "1390": 975719424.0, "1395": 931675264.0, "1400": 945856192.0, "1405": 976752768.0, "1410": 974512064.0, "1415": 967568832.0, "1420": 966747200.0, "1425": 937379200.0, "1430": 973916160.0, "1435": 978333952.0, "1440": 964180352.0, "1445": 958059264.0, "1450": 946148608.0, "1455": 983924608.0, "1460": 968649984.0, "1465": 948747072.0, "1470": 984244864.0, "1475": 943903360.0, "1480": 963976576.0, "1485": 957349120.0, "1490": 961261568.0, "1495": 980539520.0, "1500": 958334144.0, "1505": 942866752.0, "1510": 984180608.0, "1515": 959093888.0, "1520": 959103936.0, "1525": 952784896.0, "1530": 957744896.0, "1535": 949431168.0, "1540": 971087680.0, "1545": 963135040.0, "1550": 978667264.0, "1555": 952316672.0, "1560": 980089536.0, "1565": 967317376.0, "1570": 973843712.0, "1575": 975494784.0, "1580": 941864320.0, "1585": 970030528.0, "1590": 983822592.0, "1595": 948635392.0, "1600": 967446016.0, "1605": 952449280.0, "1610": 969617664.0, "1615": 983150208.0, "1620": 968020608.0, "1625": 970718336.0, "1630": 962888320.0, "1635": 942312576.0, "1640": 981613056.0, "1645": 973977984.0, "1650": 974185728.0, "1655": 967264192.0, "1660": 940689024.0, "1665": 961702848.0, "1670": 962902848.0, "1675": 971280896.0, "1680": 980879872.0, "1685": 944418816.0, "1690": 964687872.0, "1695": 965643264.0, "1700": 966339584.0, "1705": 985202688.0, "1710": 978353024.0, "1715": 943211136.0, "1720": 977088448.0, "1725": 965872448.0, "1730": 968966784.0, "1735": 965089536.0, "1740": 949714624.0, "1745": 970010176.0, "1750": 959685056.0, "1755": 960087040.0, "1760": 966380800.0, "1765": 951817088.0, "1770": 954666944.0, "1775": 973752064.0, "1780": 970536448.0, "1785": 968825728.0, "1790": 950235648.0, "1795": 945133184.0, "1800": 984667712.0, "1805": 987163072.0, "1810": 977768000.0, "1815": 948006400.0, "1820": 949209344.0, "1825": 978852352.0, "1830": 966361856.0, "1835": 964132864.0, "1840": 972318784.0, "1845": 935415808.0, "1850": 952497536.0, "1855": 980050944.0, "1860": 975868288.0, "1865": 958966272.0, "1870": 958948864.0, "1875": 932594944.0, "1880": 973575936.0, "1885": 978847744.0, "1890": 971359168.0, "1895": 959211776.0, "1900": 947394304.0, "1905": 981827712.0, "1910": 969127936.0, "1915": 970041728.0, "1920": 975598080.0, "1925": 960496640.0, "1930": 977926144.0, "1935": 963251456.0, "1940": 952460800.0, "1945": 981337920.0, "1950": 939171584.0, "1955": 960606208.0, "1960": 970031616.0, "1965": 981176768.0, "1970": 962045312.0, "1975": 952824064.0, "1980": 936849920.0, "1985": 975939456.0, "1990": 965967936.0, "1995": 962612608.0, "2000": 960553984.0, "2005": 954497792.0, "2010": 975579392.0, "2015": 991802880.0, "2020": 975435840.0, "2025": 974304768.0, "2030": 952083776.0, "2035": 967850432.0, "2040": 987460096.0, "2045": 976479616.0, "2050": 984702208.0, "2055": 942837248.0, "2060": 942594688.0, "2065": 966210176.0, "2070": 969622912.0, "2075": 980554880.0, "2080": 977600384.0, "2085": 939640128.0, "2090": 969874880.0, "2095": 961277312.0, "2100": 976718976.0, "2105": 972537856.0, "2110": 959904000.0, "2115": 956877184.0, "2120": 977480320.0, "2125": 962566272.0, "2130": 979618624.0, "2135": 950537280.0, "2140": 946995456.0, "2145": 962275584.0, "2150": 973404032.0, "2155": 972689216.0, "2160": 970315008.0, "2165": 948644224.0, "2170": 961545088.0, "2175": 969376640.0, "2180": 969329408.0, "2185": 947447424.0, "2190": 940481024.0, "2195": 986085888.0, "2200": 961860352.0, "2205": 978924032.0, "2210": 964102656.0, "2215": 963501952.0, "2220": 951311104.0, "2225": 969316672.0, "2230": 976331136.0, "2235": 974024704.0, "2240": 975494208.0, "2245": 960232000.0, "2250": 967640384.0, "2255": 969132672.0, "2260": 975065216.0, "2265": 968258304.0, "2270": 951745664.0, "2275": 962768640.0, "2280": 969640064.0, "2285": 971693632.0, "2290": 962890944.0, "2295": 931408256.0, "2300": 959906688.0, "2305": 970426496.0, "2310": 967444800.0, "2315": 970906880.0, "2320": 975589248.0, "2325": 938586496.0, "2330": 988435968.0, "2335": 977489152.0, "2340": 964596224.0, "2345": 964167680.0, "2350": 947555456.0, "2355": 977029504.0, "2360": 966898688.0, "2365": 977293056.0, "2370": 965073024.0, "2375": 953966016.0, "2380": 962917184.0, "2385": 967195264.0, "2390": 963078016.0, "2395": 974465536.0, "2400": 958411584.0, "2405": 968120896.0, "2410": 951585920.0, "2415": 965904960.0, "2420": 966518208.0, "2425": 959045248.0, "2430": 956686080.0, "2435": 961390080.0, "2440": 959755648.0, "2445": 970890176.0, "2450": 961998976.0, "2455": 922721024.0, "2460": 951954560.0, "2465": 955732224.0, "2470": 972571904.0, "2475": 973812736.0, "2480": 943896704.0, "2485": 944186560.0, "2490": 972411968.0, "2495": 974452672.0, "2500": 973908096.0, "2505": 958492352.0, "2510": 939510144.0, "2515": 979553408.0, "2520": 970473600.0, "2525": 964389504.0, "2530": 955798848.0, "2535": 936598656.0, "2540": 969027968.0, "2545": 970385088.0, "2550": 969458560.0, "2555": 969437056.0, "2560": 964977664.0, "2565": 959764544.0, "2570": 985176320.0, "2575": 957425664.0, "2580": 967424768.0, "2585": 966023424.0, "2590": 956355072.0, "2595": 981829760.0, "2600": 959532160.0, "2605": 963000960.0, "2610": 965972608.0, "2615": 951925504.0, "2620": 971242688.0, "2625": 976457408.0, "2630": 974411584.0, "2635": 948070912.0, "2640": 948137728.0, "2645": 963038848.0, "2650": 953984064.0, "2655": 977112064.0, "2660": 949623424.0, "2665": 953930112.0, "2670": 959063552.0, "2675": 979276736.0, "2680": 961397312.0, "2685": 970702592.0, "2690": 965221952.0, "2695": 943553664.0, "2700": 969425280.0, "2705": 978961536.0, "2710": 971812224.0, "2715": 990814976.0, "2720": 942649536.0, "2725": 967955904.0, "2730": 955468160.0, "2735": 970675840.0, "2740": 977921792.0, "2745": 932281024.0, "2750": 947857152.0, "2755": 956317440.0, "2760": 981697344.0, "2765": 966111488.0, "2770": 948915456.0, "2775": 935830656.0, "2780": 964776512.0, "2785": 969569152.0, "2790": 974273664.0, "2795": 966883712.0, "2800": 944388224.0, "2805": 964353152.0, "2810": 969609344.0, "2815": 975844352.0, "2820": 963087232.0, "2825": 937629952.0, "2830": 956742464.0, "2835": 986322752.0, "2840": 961759360.0, "2845": 967508416.0, "2850": 951716480.0, "2855": 962092736.0, "2860": 954241920.0, "2865": 955882368.0, "2870": 944664064.0, "2875": 974664000.0, "2880": 968201984.0, "2885": 981081664.0, "2890": 953455872.0, "2895": 957179008.0, "2900": 964989824.0, "2905": 931709056.0, "2910": 955731584.0, "2915": 979476608.0, "2920": 970491904.0, "2925": 964976000.0, "2930": 964048128.0, "2935": 940141696.0, "2940": 964911104.0, "2945": 989150336.0, "2950": 965209728.0, "2955": 965104768.0, "2960": 933162112.0, "2965": 968794816.0, "2970": 973035264.0, "2975": 958093440.0, "2980": 964497344.0, "2985": 937268864.0, "2990": 951255424.0, "2995": 978316288.0, "3000": 969275776.0, "3005": 974687936.0, "3010": 950235520.0, "3015": 943841152.0, "3020": 958442752.0, "3025": 975186112.0, "3030": 965018240.0, "3035": 963454336.0, "3040": 952134656.0, "3045": 989793408.0, "3050": 965545216.0, "3055": 982521216.0, "3060": 971227520.0, "3065": 943917312.0, "3070": 978408576.0, "3075": 975206208.0, "3080": 960992768.0, "3085": 962353280.0, "3090": 945954304.0, "3095": 938116096.0, "3100": 972929856.0, "3105": 961990400.0, "3110": 970657024.0, "3115": 963390336.0, "3120": 947117312.0, "3125": 972721408.0, "3130": 952974080.0, "3135": 966042624.0, "3140": 968488512.0, "3145": 937851392.0, "3150": 975010176.0, "3155": 976815360.0, "3160": 969628032.0, "3165": 982195584.0, "3170": 937961088.0, "3175": 953825024.0, "3180": 983810048.0, "3185": 965169920.0, "3190": 968483200.0, "3195": 950932864.0, "3200": 945102400.0, "3205": 959863552.0, "3210": 957487488.0, "3215": 958020352.0, "3220": 968129408.0, "3225": 935614592.0, "3230": 962590080.0, "3235": 975776960.0, "3240": 962621952.0, "3245": 981274880.0, "3250": 943260608.0, "3255": 954600064.0, "3260": 980362752.0, "3265": 963620416.0, "3270": 965164928.0, "3275": 959731712.0, "3280": 967046400.0, "3285": 982477824.0, "3290": 947690368.0, "3295": 966423168.0, "3300": 959165952.0, "3305": 949131776.0, "3310": 979510592.0, "3315": 964283520.0, "3320": 969208960.0, "3325": 956193856.0, "3330": 941167872.0, "3335": 964986176.0, "3340": 956913792.0, "3345": 972511872.0, "3350": 964575744.0, "3355": 943359232.0, "3360": 970037504.0, "3365": 969467904.0, "3370": 954763392.0, "3375": 958676096.0, "3380": 971474944.0, "3385": 947985280.0, "3390": 965793152.0, "3395": 978403520.0, "3400": 978138752.0, "3405": 976737152.0, "3410": 924193216.0, "3415": 955437440.0, "3420": 971832960.0, "3425": 977168768.0, "3430": 973837696.0, "3435": 936085504.0, "3440": 970517120.0, "3445": 957314816.0, "3450": 959853440.0, "3455": 963865600.0, "3460": 967895040.0, "3465": 931319360.0, "3470": 952348352.0, "3475": 973716096.0, "3480": 959751168.0, "3485": 979959296.0, "3490": 944684224.0, "3495": 953917952.0, "3500": 969330176.0, "3505": 964360704.0, "3510": 971235456.0, "3515": 955957888.0, "3520": 958747264.0, "3525": 971915392.0, "3530": 964137984.0, "3535": 983204736.0, "3540": 937501952.0, "3545": 944743040.0, "3550": 984463872.0, "3555": 978071424.0, "3560": 974386048.0, "3565": 968813568.0, "3570": 946708224.0, "3575": 976105344.0, "3580": 977506752.0, "3585": 954580608.0, "3590": 956439552.0, "3595": 951455232.0, "3600": 989009856.0, "3605": 962020160.0, "3610": 965066816.0, "3615": 974649280.0, "3620": 954900992.0, "3625": 939529280.0, "3630": 990161536.0, "3635": 971447488.0, "3640": 976038912.0, "3645": 961502208.0, "3650": 945816192.0, "3655": 965782720.0, "3660": 976207936.0, "3665": 964032640.0, "3670": 977444352.0, "3675": 943488640.0, "3680": 958197888.0, "3685": 964298624.0, "3690": 982118400.0, "3695": 963126016.0, "3700": 950571520.0, "3705": 947357440.0, "3710": 982380160.0, "3715": 972682368.0, "3720": 976139008.0, "3725": 964051584.0, "3730": 948861888.0, "3735": 967079296.0, "3740": 960960896.0, "3745": 969322432.0, "3750": 963935872.0, "3755": 953421824.0, "3760": 976654464.0, "3765": 979841536.0, "3770": 972372736.0, "3775": 972386432.0, "3780": 952599040.0, "3785": 960225664.0, "3790": 985586240.0, "3795": 969183360.0, "3800": 957865856.0, "3805": 972396288.0, "3810": 954517824.0, "3815": 974567808.0, "3820": 963020416.0, "3825": 962059712.0, "3830": 969383808.0, "3835": 934745984.0, "3840": 971270528.0, "3845": 986838144.0, "3850": 968850240.0, "3855": 965260864.0, "3860": 948069888.0, "3865": 975062528.0, "3870": 985111936.0, "3875": 983055232.0, "3880": 963640448.0, "3885": 953024000.0, "3890": 960298240.0, "3895": 960592000.0, "3900": 984901760.0, "3905": 976210880.0, "3910": 987348992.0, "3915": 946005888.0, "3920": 974879616.0, "3925": 961236352.0, "3930": 976790336.0, "3935": 978926464.0, "3940": 950304448.0, "3945": 960273664.0, "3950": 974172928.0, "3955": 972980352.0, "3960": 974078592.0, "3965": 950874176.0, "3970": 980656000.0, "3975": 960726272.0, "3980": 977520000.0, "3985": 962949120.0, "3990": 972755776.0, "3995": 953701632.0, "4000": 974988416.0, "4005": 971658112.0, "4010": 978397184.0, "4015": 971468288.0, "4020": 950297600.0, "4025": 968408576.0, "4030": 997923840.0, "4035": 978537152.0, "4040": 959788800.0, "4045": 939639808.0, "4050": 944694400.0, "4055": 980948672.0, "4060": 977659136.0, "4065": 975707200.0, "4070": 942140416.0, "4075": 945753088.0, "4080": 988741376.0, "4085": 962081024.0, "4090": 983332928.0, "4095": 986920832.0, "4100": 957176512.0, "4105": 954074624.0, "4110": 966463104.0, "4115": 976016960.0, "4120": 983506944.0, "4125": 960017024.0, "4130": 967253824.0, "4135": 971444032.0, "4140": 963146944.0, "4145": 956162816.0, "4150": 960282624.0, "4155": 946204480.0, "4160": 968402560.0, "4165": 970302720.0, "4170": 971934848.0, "4175": 955849792.0, "4180": 940962176.0, "4185": 968281088.0, "4190": 967966848.0, "4195": 989187008.0, "4200": 962630144.0, "4205": 960569792.0, "4210": 971813760.0, "4215": 974121088.0, "4220": 981120704.0, "4225": 975209024.0, "4230": 952702400.0, "4235": 958494080.0, "4240": 966811392.0, "4245": 961732736.0, "4250": 965857920.0, "4255": 958267136.0, "4260": 949557632.0, "4265": 964149376.0, "4270": 978395840.0, "4275": 975422464.0, "4280": 962672256.0, "4285": 951463296.0, "4290": 980017408.0, "4295": 968855296.0, "4300": 958286336.0, "4305": 966765632.0, "4310": 939614592.0, "4315": 949429696.0, "4320": 984536512.0, "4325": 982573056.0, "4330": 974772096.0, "4335": 949447040.0, "4340": 959503680.0, "4345": 956644096.0, "4350": 979935744.0, "4355": 968870784.0, "4360": 966338816.0, "4365": 941246208.0, "4370": 969508288.0, "4375": 973008000.0, "4380": 966160768.0, "4385": 972036992.0, "4390": 954165440.0, "4395": 951869056.0, "4400": 973665280.0, "4405": 972466432.0, "4410": 967921280.0, "4415": 958960768.0, "4420": 960791744.0, "4425": 976440576.0, "4430": 965930368.0, "4435": 975914240.0, "4440": 962343424.0, "4445": 954825536.0, "4450": 978332672.0, "4455": 960256384.0, "4460": 968593856.0, "4465": 968818944.0, "4470": 944152448.0, "4475": 952010240.0, "4480": 978837312.0, "4485": 968292032.0, "4490": 957038208.0, "4495": 938951296.0, "4500": 953253376.0, "4505": 977203456.0, "4510": 978762368.0, "4515": 962505600.0, "4520": 958837632.0, "4525": 958139520.0, "4530": 964429824.0, "4535": 976604928.0, "4540": 976823488.0, "4545": 970194560.0, "4550": 953223488.0, "4555": 959566720.0, "4560": 972485376.0, "4565": 973467520.0, "4570": 978886400.0, "4575": 957835072.0, "4580": 963191936.0, "4585": 957370752.0, "4590": 986568064.0, "4595": 960176896.0, "4600": 952267392.0, "4605": 959311296.0, "4610": 963711616.0, "4615": 957987520.0, "4620": 960268672.0, "4625": 973823488.0, "4630": 944495104.0, "4635": 977037952.0, "4640": 960311040.0, "4645": 982017600.0, "4650": 962374528.0, "4655": 939483264.0, "4660": 964001792.0, "4665": 962602752.0, "4670": 976696704.0, "4675": 963491392.0, "4680": 957513600.0, "4685": 949686656.0, "4690": 956986752.0, "4695": 969964032.0, "4700": 961238272.0, "4705": 970782208.0, "4710": 934518272.0, "4715": 970476096.0, "4720": 966337728.0, "4725": 980328128.0, "4730": 965875840.0, "4735": 937908736.0, "4740": 960161536.0, "4745": 976046720.0, "4750": 967984512.0, "4755": 984947328.0, "4760": 959111680.0, "4765": 955199488.0, "4770": 958583872.0, "4775": 991080576.0, "4780": 976794752.0, "4785": 967571200.0, "4790": 943707328.0, "4795": 955813760.0, "4800": 967686272.0, "4805": 976542592.0, "4810": 965124864.0, "4815": 957983104.0, "4820": 973980928.0, "4825": 961481728.0, "4830": 962651264.0, "4835": 972545408.0, "4840": 948877888.0, "4845": 965765760.0, "4850": 960305536.0, "4855": 964131584.0, "4860": 963063808.0, "4865": 967482752.0, "4870": 957158656.0, "4875": 983550592.0, "4880": 957085056.0, "4885": 977004544.0, "4890": 959764928.0, "4895": 942128512.0, "4900": 973663296.0, "4905": 975227264.0, "4910": 969246080.0, "4915": 970115456.0, "4920": 941182656.0, "4925": 954773120.0, "4930": 977023104.0, "4935": 963751424.0, "4940": 972623808.0, "4945": 960032128.0, "4950": 940732480.0, "4955": 967988096.0, "4960": 976728320.0, "4965": 960948608.0, "4970": 958567680.0, "4975": 933676800.0, "4980": 960813184.0, "4985": 962951104.0, "4990": 963516288.0, "4995": 986247936.0, "5000": 940654592.0, "5005": 968864192.0, "5010": 970259840.0, "5015": 965172608.0, "5020": 966617280.0, "5025": 949312896.0, "5030": 953465344.0, "5035": 967256000.0, "5040": 955628928.0, "5045": 969080000.0, "5050": 953309056.0, "5055": 954762624.0, "5060": 962993024.0, "5065": 952085120.0, "5070": 973511552.0, "5075": 978483328.0, "5080": 942842112.0, "5085": 965766400.0, "5090": 972783680.0, "5095": 964338176.0, "5100": 958294016.0, "5105": 965281664.0, "5110": 950365824.0, "5115": 972247808.0, "5120": 960340608.0, "5125": 969731968.0, "5130": 938710656.0, "5135": 943623424.0, "5140": 969840640.0, "5145": 968529344.0, "5150": 970528000.0, "5155": 972525248.0, "5160": 926502016.0, "5165": 961518464.0, "5170": 966824960.0, "5175": 966024192.0, "5180": 963595328.0, "5185": 930779008.0, "5190": 949693952.0, "5195": 972360320.0, "5200": 973649152.0, "5205": 968176128.0, "5210": 960442304.0, "5215": 928693888.0, "5220": 979112384.0, "5225": 984693504.0, "5230": 974975616.0, "5235": 974982464.0, "5240": 944278144.0, "5245": 970774656.0, "5250": 972390464.0, "5255": 966848128.0, "5260": 976729728.0, "5265": 942283776.0, "5270": 969202048.0, "5275": 970061632.0, "5280": 962764928.0, "5285": 964023296.0, "5290": 932465792.0, "5295": 951714688.0, "5300": 975563520.0, "5305": 951904128.0, "5310": 968004736.0, "5315": 955755904.0, "5320": 950885888.0, "5325": 973016320.0, "5330": 967732992.0, "5335": 967475008.0, "5340": 966369536.0, "5345": 962912512.0, "5350": 978913280.0, "5355": 972191104.0, "5360": 963819136.0, "5365": 965161280.0, "5370": 947777152.0, "5375": 948748416.0, "5380": 967102656.0, "5385": 980430272.0, "5390": 965252480.0, "5395": 955025536.0, "5400": 948263040.0, "5405": 974207168.0, "5410": 967794944.0, "5415": 975952640.0, "5420": 967297152.0, "5425": 937265984.0, "5430": 963686528.0, "5435": 971789568.0, "5440": 968956736.0, "5445": 957306432.0, "5450": 919370368.0, "5455": 951930496.0, "5460": 962315008.0, "5465": 978783104.0, "5470": 980883712.0, "5475": 941501568.0, "5480": 955719168.0, "5485": 964816768.0, "5490": 975815424.0, "5495": 962707968.0, "5500": 971107648.0, "5505": 956916224.0, "5510": 968451392.0, "5515": 945405568.0, "5520": 963071872.0, "5525": 975771904.0, "5530": 936475648.0, "5535": 970547968.0, "5540": 960222464.0, "5545": 971888384.0, "5550": 967786624.0, "5555": 955772544.0, "5560": 954374016.0, "5565": 968741248.0, "5570": 945064448.0, "5575": 960441600.0, "5580": 960465280.0, "5585": 959385472.0, "5590": 977594752.0, "5595": 975097728.0, "5600": 962936320.0, "5605": 964017920.0, "5610": 943072576.0, "5615": 966467584.0, "5620": 963213440.0, "5625": 982251392.0, "5630": 975903616.0, "5635": 957188352.0, "5640": 951321920.0, "5645": 967769728.0, "5650": 979010112.0, "5655": 983338496.0, "5660": 956309184.0, "5665": 953303680.0, "5670": 965950464.0, "5675": 967530304.0, "5680": 978475840.0, "5685": 961873792.0, "5690": 935811520.0, "5695": 963629312.0, "5700": 952340480.0, "5705": 974340992.0, "5710": 971034880.0, "5715": 945939328.0, "5720": 974930560.0, "5725": 967241344.0, "5730": 978353792.0, "5735": 964685568.0, "5740": 943451840.0, "5745": 971057664.0, "5750": 981706368.0, "5755": 956582976.0, "5760": 963500032.0, "5765": 957572224.0, "5770": 955539072.0, "5775": 970688512.0, "5780": 962628288.0, "5785": 970453440.0, "5790": 974456832.0, "5795": 949655936.0, "5800": 965840256.0, "5805": 968615424.0, "5810": 975876352.0, "5815": 969980864.0, "5820": 936254464.0, "5825": 969106496.0, "5830": 977602112.0, "5835": 974857920.0, "5840": 962964480.0, "5845": 968640960.0, "5850": 942897792.0, "5855": 976024384.0, "5860": 979439552.0, "5865": 978148352.0, "5870": 968604800.0, "5875": 942081792.0, "5880": 964361984.0, "5885": 974635264.0, "5890": 972499392.0, "5895": 965435904.0, "5900": 941175808.0, "5905": 961730432.0, "5910": 958446336.0, "5915": 967928576.0, "5920": 977235520.0, "5925": 959469824.0, "5930": 946577664.0, "5935": 952298048.0, "5940": 977465600.0, "5945": 984830720.0, "5950": 980530304.0, "5955": 934909760.0, "5960": 961539776.0, "5965": 965803520.0, "5970": 970514560.0, "5975": 961675264.0, "5980": 958086400.0, "5985": 964439936.0, "5990": 973381760.0, "5995": 955833856.0, "6000": 955473024.0, "6005": 961281280.0, "6010": 952609664.0, "6015": 974269184.0, "6020": 978020224.0, "6025": 971871104.0, "6030": 955287680.0, "6035": 946819712.0, "6040": 962494592.0, "6045": 983547136.0, "6050": 956393088.0, "6055": 963359616.0, "6060": 945583360.0, "6065": 958321536.0, "6070": 978182400.0, "6075": 977890560.0, "6080": 957321728.0, "6085": 947495808.0, "6090": 953634816.0, "6095": 964598656.0, "6100": 979702976.0, "6105": 971048960.0, "6110": 961632384.0, "6115": 943507840.0, "6120": 968476288.0, "6125": 960494720.0, "6130": 983671360.0, "6135": 960847104.0, "6140": 958474944.0, "6145": 970942592.0, "6150": 968419072.0, "6155": 974943104.0, "6160": 977314944.0, "6165": 952410816.0, "6170": 951019392.0, "6175": 963261056.0, "6180": 969317440.0, "6185": 966209792.0, "6190": 963584192.0, "6195": 946898560.0, "6200": 969195008.0, "6205": 966867072.0, "6210": 959094080.0, "6215": 972791104.0, "6220": 936153920.0, "6225": 978469696.0, "6230": 975851584.0, "6235": 971556544.0, "6240": 965723904.0, "6245": 955868672.0, "6250": 956253568.0, "6255": 973080192.0, "6260": 978570880.0, "6265": 974795136.0, "6270": 958560384.0, "6275": 963615744.0, "6280": 972920512.0, "6285": 966135296.0, "6290": 970845184.0, "6295": 987233024.0, "6300": 947370624.0, "6305": 964771840.0, "6310": 979152448.0, "6315": 978344704.0, "6320": 971638464.0, "6325": 922708992.0, "6330": 958826368.0, "6335": 974829952.0, "6340": 984530432.0, "6345": 966922880.0, "6350": 944525952.0, "6355": 957990144.0, "6360": 972515456.0, "6365": 972268800.0, "6370": 958820096.0, "6375": 966932544.0, "6380": 951265920.0, "6385": 973684224.0, "6390": 965368832.0, "6395": 974959488.0, "6400": 984229888.0, "6405": 944126656.0, "6410": 977154048.0, "6415": 971514048.0, "6420": 956530048.0, "6425": 961085056.0, "6430": 957341312.0, "6435": 960001984.0, "6440": 968825216.0, "6445": 973343936.0, "6450": 974355456.0, "6455": 962167616.0, "6460": 941044864.0, "6465": 974176192.0, "6470": 979789696.0, "6475": 960836480.0, "6480": 967656576.0, "6485": 948487424.0, "6490": 970896896.0, "6495": 988185216.0, "6500": 980162816.0, "6505": 971974080.0, "6510": 951329088.0, "6515": 957639488.0, "6520": 978534080.0, "6525": 978818432.0, "6530": 973112832.0, "6535": 967681664.0, "6540": 949926464.0, "6545": 966122880.0, "6550": 979186688.0, "6555": 966898560.0, "6560": 975305344.0, "6565": 949327232.0, "6570": 951871552.0, "6575": 962025216.0, "6580": 975459712.0, "6585": 979436096.0, "6590": 948819648.0, "6595": 961398912.0, "6600": 961035520.0, "6605": 961387712.0, "6610": 985261952.0, "6615": 959604864.0, "6620": 944392448.0, "6625": 970623296.0, "6630": 971319040.0, "6635": 963969536.0, "6640": 959549568.0, "6645": 950709504.0, "6650": 978527872.0, "6655": 965593984.0, "6660": 968298752.0, "6665": 968645504.0, "6670": 932542592.0, "6675": 970478976.0, "6680": 968697088.0, "6685": 958428416.0, "6690": 955883648.0, "6695": 955193088.0, "6700": 961816384.0, "6705": 978898816.0, "6710": 970680768.0, "6715": 966518592.0, "6720": 973696320.0, "6725": 941461440.0, "6730": 979068352.0, "6735": 994181056.0, "6740": 976271104.0, "6745": 974234752.0, "6750": 938818176.0, "6755": 977293312.0, "6760": 969438336.0, "6765": 978112192.0, "6770": 975145600.0, "6775": 943069184.0, "6780": 946927488.0, "6785": 975065088.0, "6790": 960090624.0, "6795": 975750784.0, "6800": 972730752.0, "6805": 946265728.0, "6810": 958129664.0, "6815": 970426432.0, "6820": 977582592.0, "6825": 969082048.0, "6830": 950103424.0, "6835": 981242752.0, "6840": 982863232.0, "6845": 948763904.0, "6850": 965380416.0, "6855": 953757312.0, "6860": 978584576.0, "6865": 983644672.0, "6870": 964683136.0, "6875": 978492672.0, "6880": 950138368.0, "6885": 958115968.0, "6890": 959858496.0, "6895": 965740160.0, "6900": 984915840.0, "6905": 968171648.0, "6910": 949790208.0, "6915": 970620288.0, "6920": 966797952.0, "6925": 964972416.0, "6930": 964311040.0, "6935": 951827008.0, "6940": 962700032.0, "6945": 986297920.0, "6950": 973154752.0, "6955": 964482624.0, "6960": 939806464.0, "6965": 974418496.0, "6970": 978198976.0, "6975": 984772032.0, "6980": 982077952.0, "6985": 959446400.0, "6990": 945089984.0, "6995": 986954752.0, "7000": 962949312.0, "7005": 962664640.0, "7010": 984653184.0, "7015": 945128576.0, "7020": 982638336.0, "7025": 968189888.0, "7030": 953218688.0, "7035": 982478784.0, "7040": 950287616.0, "7045": 955659904.0, "7050": 959973824.0, "7055": 963561472.0, "7060": 976520448.0, "7065": 967826304.0, "7070": 953400576.0, "7075": 956437120.0, "7080": 968756864.0, "7085": 965636224.0, "7090": 969043584.0, "7095": 959624448.0, "7100": 973197760.0, "7105": 973048704.0, "7110": 969799936.0, "7115": 958092672.0, "7120": 948744576.0, "7125": 962832896.0, "7130": 971299840.0, "7135": 964393216.0, "7140": 961213824.0, "7145": 930183232.0, "7150": 945960320.0, "7155": 990784256.0, "7160": 968134912.0, "7165": 956493696.0, "7170": 968041856.0, "7175": 955532928.0, "7180": 957827072.0, "7185": 984449472.0, "7190": 978477632.0, "7195": 973381504.0, "7200": 935623360.0, "7205": 957230336.0, "7210": 966881216.0, "7215": 969215616.0, "7220": 981925376.0, "7225": 928530432.0, "7230": 949353088.0, "7235": 966760576.0, "7240": 966575168.0, "7245": 967134912.0, "7250": 949025088.0, "7255": 956946368.0, "7260": 969807872.0, "7265": 974691648.0, "7270": 959566592.0, "7275": 958942848.0, "7280": 956765824.0, "7285": 977538560.0, "7290": 976734080.0, "7295": 962754880.0, "7300": 975082176.0, "7305": 963739776.0, "7310": 976988032.0, "7315": 966514048.0, "7320": 974233408.0, "7325": 966526080.0, "7330": 959260544.0, "7335": 963692864.0, "7340": 977422592.0, "7345": 967423744.0, "7350": 984486528.0, "7355": 959355520.0, "7360": 948462720.0, "7365": 972380288.0, "7370": 982188928.0, "7375": 963174656.0, "7380": 963633280.0, "7385": 948295808.0, "7390": 963566400.0, "7395": 958074624.0, "7400": 969333440.0, "7405": 987281280.0, "7410": 951600128.0, "7415": 950185728.0, "7420": 966599296.0, "7425": 982484800.0, "7430": 965429312.0, "7435": 972606592.0, "7440": 936859200.0, "7445": 968497024.0, "7450": 979957696.0, "7455": 970923776.0, "7460": 971840896.0, "7465": 939178432.0, "7470": 971494592.0, "7475": 957699840.0, "7480": 968804352.0, "7485": 961318528.0, "7490": 933789312.0, "7495": 957100544.0, "7500": 969006592.0, "7505": 969803648.0, "7510": 971704192.0, "7515": 979311680.0, "7520": 951624960.0, "7525": 970088000.0, "7530": 953869440.0, "7535": 971075776.0, "7540": 979153728.0, "7545": 958988480.0, "7550": 960133120.0, "7555": 959912320.0, "7560": 969914368.0, "7565": 954751808.0, "7570": 942337152.0, "7575": 965335296.0, "7580": 982003200.0, "7585": 978710656.0, "7590": 969922368.0, "7595": 949709184.0, "7600": 945771776.0, "7605": 982120576.0, "7610": 968713216.0, "7615": 988171520.0, "7620": 956914048.0, "7625": 940939520.0, "7630": 971317248.0, "7635": 984252672.0, "7640": 983151168.0, "7645": 967816320.0, "7650": 958914176.0, "7655": 962227200.0, "7660": 968720896.0, "7665": 977950912.0, "7670": 975216896.0, "7675": 975006848.0, "7680": 942233088.0, "7685": 959380480.0, "7690": 975130240.0, "7695": 982011904.0, "7700": 978902976.0, "7705": 940606208.0, "7710": 974454784.0, "7715": 979171456.0, "7720": 967490560.0, "7725": 960158592.0, "7730": 942943488.0, "7735": 967944832.0, "7740": 980179072.0, "7745": 963843328.0, "7750": 963459328.0, "7755": 959614848.0, "7760": 970433344.0, "7765": 970703552.0, "7770": 962102848.0, "7775": 981846464.0, "7780": 964457472.0, "7785": 959517056.0, "7790": 967987840.0, "7795": 968687424.0, "7800": 971671040.0, "7805": 968158592.0, "7810": 945835136.0, "7815": 963331200.0, "7820": 974466304.0, "7825": 963474560.0, "7830": 957260288.0, "7835": 949782656.0, "7840": 957269312.0, "7845": 954002944.0, "7850": 979745088.0, "7855": 986607936.0, "7860": 947287040.0, "7865": 949166208.0, "7870": 965038592.0, "7875": 975639808.0, "7880": 968440192.0, "7885": 969100288.0, "7890": 951917056.0, "7895": 974263360.0, "7900": 963606656.0, "7905": 963902464.0, "7910": 965879680.0, "7915": 943390208.0, "7920": 950807040.0, "7925": 969449856.0, "7930": 964844544.0, "7935": 984472576.0, "7940": 964350400.0, "7945": 950747264.0, "7950": 962036352.0, "7955": 979737344.0, "7960": 963687936.0, "7965": 953212928.0, "7970": 951432448.0, "7975": 969081344.0, "7980": 965377920.0, "7985": 959348736.0, "7990": 968055040.0, "7995": 946779136.0, "8000": 962594304.0, "8005": 980605056.0, "8010": 965702976.0, "8015": 982808256.0, "8020": 960389952.0, "8025": 965309888.0, "8030": 958262208.0, "8035": 975592640.0, "8040": 960643584.0, "8045": 948272384.0, "8050": 959290432.0, "8055": 979224768.0, "8060": 969468672.0, "8065": 957939904.0, "8070": 963845696.0, "8075": 941970688.0, "8080": 965985472.0, "8085": 966864320.0, "8090": 983528064.0, "8095": 988650688.0, "8100": 966078208.0, "8105": 944512128.0, "8110": 968908992.0, "8115": 985180608.0, "8120": 974701888.0, "8125": 964153664.0, "8130": 966322432.0, "8135": 967723904.0, "8140": 963798528.0, "8145": 995247616.0, "8150": 973142080.0, "8155": 938476544.0, "8160": 964315008.0, "8165": 972925952.0, "8170": 968275520.0, "8175": 961599808.0, "8180": 936139392.0, "8185": 962736896.0, "8190": 968033088.0, "8195": 977397632.0, "8200": 956563840.0, "8205": 960812480.0, "8210": 946741568.0, "8215": 982398592.0, "8220": 987755072.0, "8225": 966280064.0, "8230": 962282688.0, "8235": 933738944.0, "8240": 980304576.0, "8245": 976189632.0, "8250": 963750400.0, "8255": 977001088.0, "8260": 956452416.0, "8265": 982710592.0, "8270": 952492096.0, "8275": 973853312.0, "8280": 974218048.0, "8285": 953334080.0, "8290": 939714688.0, "8295": 981089472.0, "8300": 973026240.0, "8305": 978061504.0, "8310": 950924096.0, "8315": 937711936.0, "8320": 977374080.0, "8325": 967837376.0, "8330": 990203584.0, "8335": 975599104.0, "8340": 947317568.0, "8345": 970703104.0, "8350": 970132096.0, "8355": 974915968.0, "8360": 979210496.0, "8365": 932621952.0, "8370": 965304512.0, "8375": 979830976.0, "8380": 965015744.0, "8385": 972730560.0, "8390": 962613248.0, "8395": 951077120.0, "8400": 972409664.0, "8405": 951415680.0, "8410": 960789632.0, "8415": 965573312.0, "8420": 941582016.0, "8425": 968081152.0, "8430": 961159616.0, "8435": 966042432.0, "8440": 969658688.0, "8445": 952713344.0, "8450": 984510784.0, "8455": 990051520.0, "8460": 968656960.0, "8465": 967147136.0, "8470": 962985088.0, "8475": 943118272.0, "8480": 986979264.0, "8485": 979689920.0, "8490": 992280000.0, "8495": 971471552.0, "8500": 951317824.0, "8505": 983350528.0, "8510": 974186112.0, "8515": 968862976.0, "8520": 961618560.0, "8525": 945070592.0, "8530": 984119616.0, "8535": 978264000.0, "8540": 967839104.0, "8545": 968719616.0, "8550": 941721408.0, "8555": 971721728.0, "8560": 958210112.0, "8565": 975541440.0, "8570": 974850816.0, "8575": 971165696.0, "8580": 932030912.0, "8585": 965721024.0, "8590": 978774592.0, "8595": 979179264.0, "8600": 983534976.0, "8605": 957472768.0, "8610": 983406848.0, "8615": 977662720.0, "8620": 963278016.0, "8625": 979344320.0, "8630": 943469248.0, "8635": 961638848.0, "8640": 973401088.0, "8645": 970304896.0, "8650": 969363520.0, "8655": 970457216.0, "8660": 943783488.0, "8665": 986105984.0, "8670": 960734720.0, "8675": 973824192.0, "8680": 962276288.0, "8685": 955741760.0, "8690": 978501312.0, "8695": 968820032.0, "8700": 972532608.0, "8705": 973766784.0, "8710": 946858560.0, "8715": 973281984.0, "8720": 958450816.0, "8725": 979051520.0, "8730": 985599936.0, "8735": 952338240.0, "8740": 940889408.0, "8745": 987271872.0, "8750": 972048384.0, "8755": 971578176.0, "8760": 965483200.0, "8765": 934360448.0, "8770": 986362240.0, "8775": 969740736.0, "8780": 966950400.0, "8785": 961987648.0, "8790": 947565056.0, "8795": 969517056.0, "8800": 970803392.0, "8805": 972994112.0, "8810": 983429888.0, "8815": 951016960.0, "8820": 939808704.0, "8825": 964408448.0, "8830": 981249472.0, "8835": 971398336.0, "8840": 979160512.0, "8845": 951234560.0, "8850": 986674560.0, "8855": 970377472.0, "8860": 962000896.0, "8865": 956620736.0, "8870": 945714880.0, "8875": 968444160.0, "8880": 983889920.0, "8885": 971269056.0, "8890": 969427200.0, "8895": 952763456.0, "8900": 961386176.0, "8905": 976606592.0, "8910": 981717440.0, "8915": 980645504.0, "8920": 968074112.0, "8925": 939992256.0, "8930": 970258432.0, "8935": 964040384.0, "8940": 977644800.0, "8945": 981826944.0, "8950": 945562496.0, "8955": 972497152.0, "8960": 973356992.0, "8965": 973678848.0, "8970": 966311424.0, "8975": 936927552.0, "8980": 952829248.0, "8985": 977632320.0, "8990": 967399424.0, "8995": 980307840.0, "9000": 952199232.0, "9005": 950403776.0, "9010": 974875392.0, "9015": 982753728.0, "9020": 958935488.0, "9025": 979374144.0, "9030": 953690688.0, "9035": 968675712.0, "9040": 978208384.0, "9045": 968412032.0, "9050": 983012544.0, "9055": 947799488.0, "9060": 956368192.0, "9065": 969361984.0, "9070": 967990592.0, "9075": 980648000.0, "9080": 952455488.0, "9085": 971552512.0, "9090": 963642816.0, "9095": 968240320.0, "9100": 974375744.0, "9105": 960292096.0, "9110": 947543104.0, "9115": 956514368.0, "9120": 985151168.0, "9125": 963050368.0, "9130": 958395136.0, "9135": 951643648.0, "9140": 967074432.0, "9145": 976978944.0, "9150": 986789952.0, "9155": 976907008.0, "9160": 957621504.0, "9165": 950526656.0, "9170": 988443776.0, "9175": 971262400.0, "9180": 967509504.0, "9185": 955042112.0, "9190": 956366976.0, "9195": 965770368.0, "9200": 968622848.0, "9205": 967231744.0, "9210": 984253184.0, "9215": 931731776.0, "9220": 949083264.0, "9225": 970972416.0, "9230": 970827200.0, "9235": 971550848.0, "9240": 959819392.0, "9245": 963536256.0, "9250": 961576256.0, "9255": 982627072.0, "9260": 979369344.0, "9265": 952769152.0, "9270": 948946560.0, "9275": 978587840.0, "9280": 977583360.0, "9285": 962705472.0, "9290": 978850560.0, "9295": 958658624.0, "9300": 965618304.0, "9305": 968911488.0, "9310": 972879232.0, "9315": 976034368.0, "9320": 947948352.0, "9325": 979431360.0, "9330": 977558272.0, "9335": 975487744.0, "9340": 960151872.0, "9345": 943218176.0, "9350": 952676288.0, "9355": 963291136.0, "9360": 960074112.0, "9365": 983534848.0, "9370": 982557952.0, "9375": 942044672.0, "9380": 982945920.0, "9385": 985381568.0, "9390": 973099712.0, "9395": 978541248.0, "9400": 937756160.0, "9405": 968082816.0, "9410": 981275392.0, "9415": 991699072.0, "9420": 960244800.0, "9425": 956695680.0, "9430": 938879872.0, "9435": 974510464.0, "9440": 959322368.0, "9445": 973635584.0, "9450": 961485184.0, "9455": 945813120.0, "9460": 978135680.0, "9465": 988016576.0, "9470": 962977216.0, "9475": 983614016.0, "9480": 931015168.0, "9485": 986877504.0, "9490": 963520000.0, "9495": 972526464.0, "9500": 982355008.0, "9505": 970113472.0, "9510": 964358720.0, "9515": 956691264.0, "9520": 947893248.0, "9525": 965359680.0, "9530": 958138048.0, "9535": 951097600.0, "9540": 953849536.0, "9545": 979579200.0, "9550": 955691648.0, "9555": 952532928.0, "9560": 958223360.0, "9565": 969425920.0, "9570": 977369856.0, "9575": 959033984.0, "9580": 963097536.0, "9585": 945779776.0, "9590": 948295360.0, "9595": 966492160.0, "9600": 984344000.0, "9605": 984934912.0, "9610": 943517952.0, "9615": 952482496.0, "9620": 980674816.0, "9625": 978574272.0, "9630": 970052544.0, "9635": 974733184.0, "9640": 940245440.0, "9645": 962402112.0, "9650": 971201664.0, "9655": 987483968.0, "9660": 963258752.0, "9665": 949972864.0, "9670": 966334592.0, "9675": 963075968.0, "9680": 965003840.0, "9685": 986401984.0, "9690": 940337664.0, "9695": 950509504.0, "9700": 975088128.0, "9705": 972504064.0, "9710": 967558912.0, "9715": 971305216.0, "9720": 940514752.0, "9725": 966097152.0, "9730": 973731968.0, "9735": 974370176.0, "9740": 971295680.0, "9745": 950816192.0, "9750": 979727232.0, "9755": 970292480.0, "9760": 968175296.0, "9765": 963811392.0, "9770": 952258304.0, "9775": 956670528.0, "9780": 970353408.0, "9785": 958627264.0, "9790": 961264512.0, "9795": 958201472.0, "9800": 948971520.0, "9805": 962036992.0, "9810": 978381568.0, "9815": 977262656.0, "9820": 982496960.0, "9825": 939141376.0, "9830": 969073216.0, "9835": 972339072.0, "9840": 971208320.0, "9845": 966656960.0, "9850": 946518784.0, "9855": 956995328.0, "9860": 986979584.0, "9865": 970027904.0, "9870": 989916864.0, "9875": 956925696.0, "9880": 931276800.0, "9885": 963470400.0, "9890": 972405568.0, "9895": 983489792.0, "9900": 956755968.0, "9905": 938556160.0, "9910": 978602368.0, "9915": 973099200.0, "9920": 944426496.0, "9925": 962916992.0, "9930": 947183744.0, "9935": 960278272.0, "9940": 965697280.0, "9945": 958382720.0, "9950": 963625856.0, "9955": 942997056.0, "9960": 966882304.0, "9965": 983350592.0, "9970": 966386496.0, "9975": 963881344.0, "9980": 980328512.0, "9985": 941824384.0, "9990": 976809984.0, "9995": 982129024.0, "10000": 971851776.0, "10005": 969861376.0, "10010": 943877440.0, "10015": 982577152.0, "10020": 977804032.0, "10025": 979549888.0, "10030": 971244672.0, "10035": 946533312.0, "10040": 950485760.0, "10045": 978173696.0, "10050": 985758592.0, "10055": 990346368.0, "10060": 959069696.0, "10065": 947062784.0, "10070": 966931584.0, "10075": 979401280.0, "10080": 971622080.0, "10085": 974745856.0, "10090": 944008832.0, "10095": 963125376.0, "10100": 972146944.0, "10105": 976034752.0, "10110": 971772800.0, "10115": 948353792.0, "10120": 962507264.0, "10125": 974194752.0, "10130": 980564736.0, "10135": 972567680.0, "10140": 957806016.0, "10145": 933887232.0, "10150": 973855872.0, "10155": 969285952.0, "10160": 961618304.0, "10165": 974885760.0, "10170": 944427776.0, "10175": 978892608.0, "10180": 983664000.0, "10185": 978746752.0, "10190": 955519744.0, "10195": 937150144.0, "10200": 988093952.0, "10205": 972676352.0, "10210": 966740672.0, "10215": 975630720.0, "10220": 948295552.0, "10225": 950289280.0, "10230": 975223168.0, "10235": 953831744.0, "10240": 969616640.0, "10245": 961772032.0, "10250": 936264064.0, "10255": 979504128.0, "10260": 964644352.0, "10265": 967489600.0, "10270": 968425088.0, "10275": 936103808.0, "10280": 969561856.0, "10285": 996083968.0, "10290": 979454656.0, "10295": 981540224.0, "10300": 951828480.0, "10305": 971864512.0, "10310": 960064768.0, "10315": 971187840.0, "10320": 985120896.0, "10325": 983215936.0, "10330": 934926464.0, "10335": 976317696.0, "10340": 957468800.0, "10345": 973596928.0, "10350": 984756096.0, "10355": 941803008.0, "10360": 961872512.0, "10365": 974268416.0, "10370": 980346176.0, "10375": 969875584.0, "10380": 961527616.0, "10385": 955072512.0, "10390": 990463232.0, "10395": 964845440.0, "10400": 960777984.0, "10405": 949747968.0, "10410": 955098176.0, "10415": 976146624.0, "10420": 967212096.0, "10425": 969751808.0, "10430": 964795648.0, "10435": 963036736.0, "10440": 971963200.0, "10445": 972183296.0, "10450": 975122048.0, "10455": 966331904.0, "10460": 948480896.0, "10465": 971374464.0, "10470": 972572672.0, "10475": 979157888.0, "10480": 997170048.0, "10485": 949275904.0, "10490": 934796800.0, "10495": 969120896.0, "10500": 977939456.0, "10505": 958974848.0, "10510": 950564864.0, "10515": 953891648.0, "10520": 972026624.0, "10525": 969414016.0, "10530": 970264896.0, "10535": 986317568.0, "10540": 946952448.0, "10545": 970374144.0, "10550": 968708032.0, "10555": 959557632.0, "10560": 975881664.0, "10565": 961363648.0, "10570": 968627200.0, "10575": 972498368.0, "10580": 960586752.0, "10585": 973333312.0, "10590": 951660288.0, "10595": 956430016.0, "10600": 967505280.0, "10605": 986588288.0, "10610": 966405248.0, "10615": 976526336.0, "10620": 940774144.0, "10625": 964938240.0, "10630": 967592064.0, "10635": 973064768.0, "10640": 974382592.0, "10645": 948458688.0, "10650": 966118656.0, "10655": 985083136.0, "10660": 976596736.0, "10665": 967187456.0, "10670": 954858240.0, "10675": 934123776.0, "10680": 986080640.0, "10685": 990783424.0, "10690": 963935488.0, "10695": 971897600.0, "10700": 949662464.0, "10705": 977799040.0, "10710": 968120576.0, "10715": 966837376.0, "10720": 966149760.0, "10725": 944047616.0, "10730": 980048704.0, "10735": 960773376.0, "10740": 971234048.0, "10745": 984200064.0, "10750": 981234944.0, "10755": 944244992.0, "10760": 969661632.0, "10765": 972555392.0, "10770": 973764864.0, "10775": 958674688.0, "10780": 949434368.0, "10785": 953571456.0, "10790": 970121216.0, "10795": 960127488.0, "10800": 972161280.0, "10805": 950822848.0, "10810": 973682880.0, "10815": 959656832.0, "10820": 971093696.0, "10825": 967113792.0, "10830": 956819456.0, "10835": 963015552.0, "10840": 970409024.0, "10845": 963739136.0, "10850": 957549312.0, "10855": 967510912.0, "10860": 950518400.0, "10865": 964160128.0, "10870": 983381376.0, "10875": 982274560.0, "10880": 958610752.0, "10885": 954466816.0, "10890": 972840192.0, "10895": 973359552.0, "10900": 970362240.0, "10905": 964840448.0, "10910": 938557824.0, "10915": 960615424.0, "10920": 982842368.0, "10925": 969796480.0, "10930": 968551104.0, "10935": 962492544.0, "10940": 953943040.0, "10945": 964767936.0, "10950": 972467008.0, "10955": 966633216.0, "10960": 971941824.0, "10965": 966455552.0, "10970": 983437248.0, "10975": 965796672.0, "10980": 974656832.0, "10985": 986572608.0, "10990": 950958336.0, "10995": 963353408.0, "11000": 985286784.0, "11005": 978548224.0, "11010": 971455360.0, "11015": 969726400.0, "11020": 947938112.0, "11025": 959834624.0, "11030": 977837824.0, "11035": 975095872.0, "11040": 986142784.0, "11045": 956475200.0, "11050": 973064832.0, "11055": 974445952.0, "11060": 962065280.0, "11065": 985161344.0, "11070": 949631488.0, "11075": 976197760.0, "11080": 971823744.0, "11085": 967095040.0, "11090": 976461504.0, "11095": 946438144.0, "11100": 965375424.0, "11105": 973585792.0, "11110": 980633600.0, "11115": 967886656.0, "11120": 956744384.0, "11125": 956475200.0, "11130": 975504896.0, "11135": 979451712.0, "11140": 964612992.0, "11145": 966377088.0, "11150": 935937152.0, "11155": 975939584.0, "11160": 983999424.0, "11165": 982400320.0, "11170": 977339264.0, "11175": 957764608.0, "11180": 962147136.0, "11185": 971844672.0, "11190": 979692288.0, "11195": 985085376.0, "11200": 982817216.0, "11205": 942217408.0, "11210": 984703488.0, "11215": 967396928.0, "11220": 983169280.0, "11225": 961691264.0, "11230": 953189568.0, "11235": 981706240.0, "11240": 977425152.0, "11245": 966073920.0, "11250": 969233408.0, "11255": 960376256.0, "11260": 979703296.0, "11265": 963607232.0, "11270": 981118656.0, "11275": 968047232.0, "11280": 955794176.0, "11285": 953405440.0, "11290": 956623360.0, "11295": 968271680.0, "11300": 962673344.0, "11305": 958624960.0, "11310": 946341184.0, "11315": 982999936.0, "11320": 964742912.0, "11325": 981029568.0, "11330": 975564288.0, "11335": 952309056.0, "11340": 970375616.0, "11345": 969803648.0, "11350": 981444096.0, "11355": 982385024.0, "11360": 941016704.0, "11365": 970431360.0, "11370": 978805312.0, "11375": 975313792.0, "11380": 968478016.0, "11385": 958625152.0, "11390": 938120832.0, "11395": 977384192.0, "11400": 972740288.0, "11405": 961470400.0, "11410": 966144128.0, "11415": 929225856.0, "11420": 964547904.0, "11425": 980999168.0, "11430": 978502144.0, "11435": 970006400.0, "11440": 945134656.0, "11445": 974983936.0, "11450": 984464448.0, "11455": 971217024.0, "11460": 964937088.0, "11465": 959659776.0, "11470": 954982528.0, "11475": 972653632.0, "11480": 956542208.0, "11485": 977052928.0, "11490": 986172224.0, "11495": 959133824.0, "11500": 970309248.0, "11505": 963864960.0, "11510": 976505280.0, "11515": 978026624.0, "11520": 954061696.0, "11525": 975806016.0, "11530": 976686720.0, "11535": 979970048.0, "11540": 974399040.0, "11545": 953188288.0, "11550": 953074304.0, "11555": 981696128.0, "11560": 984556736.0, "11565": 965242048.0, "11570": 966375872.0, "11575": 951523968.0, "11580": 975902976.0, "11585": 977428032.0, "11590": 969376384.0, "11595": 976214208.0, "11600": 946488640.0, "11605": 973334272.0, "11610": 982199616.0, "11615": 971997440.0, "11620": 968577856.0, "11625": 948852096.0, "11630": 937541248.0, "11635": 973606016.0, "11640": 981160768.0, "11645": 980069504.0, "11650": 971782464.0, "11655": 956044224.0, "11660": 980441920.0, "11665": 958240768.0, "11670": 982643776.0, "11675": 972390592.0, "11680": 956322304.0, "11685": 982734016.0, "11690": 968614464.0, "11695": 967558592.0, "11700": 973842752.0, "11705": 956309760.0, "11710": 964708032.0, "11715": 983208768.0, "11720": 983538048.0, "11725": 965265600.0, "11730": 955214208.0, "11735": 942646336.0, "11740": 973792512.0, "11745": 971088320.0, "11750": 961672064.0, "11755": 963374080.0, "11760": 949543168.0, "11765": 983755008.0, "11770": 984528384.0, "11775": 975438144.0, "11780": 984888832.0, "11785": 946940288.0, "11790": 972493440.0, "11795": 970387968.0, "11800": 973227264.0, "11805": 986472960.0, "11810": 967530304.0, "11815": 955616128.0, "11820": 973433920.0, "11825": 970646848.0, "11830": 974846080.0, "11835": 961579392.0, "11840": 944504512.0, "11845": 980497536.0, "11850": 974454528.0, "11855": 977918336.0, "11860": 971612352.0, "11865": 938262400.0, "11870": 940126208.0, "11875": 990048704.0, "11880": 972206784.0, "11885": 962901568.0, "11890": 970318208.0, "11895": 965208192.0, "11900": 979393920.0, "11905": 961566912.0, "11910": 982719680.0, "11915": 989857600.0, "11920": 944868608.0, "11925": 994162944.0, "11930": 964190528.0, "11935": 963465152.0, "11940": 976622080.0, "11945": 944763008.0, "11950": 977159808.0, "11955": 979128640.0, "11960": 972399168.0, "11965": 976260672.0, "11970": 963060544.0, "11975": 963129408.0, "11980": 977634432.0, "11985": 953073920.0, "11990": 968742400.0, "11995": 965156352.0, "12000": 958418112.0, "12005": 973932800.0, "12010": 979537920.0, "12015": 972212736.0, "12020": 973267520.0, "12025": 934456768.0, "12030": 969246336.0, "12035": 984077504.0, "12040": 977777472.0, "12045": 981645632.0, "12050": 931299456.0, "12055": 938852096.0, "12060": 974410240.0, "12065": 965723008.0, "12070": 968326208.0, "12075": 949990592.0, "12080": 953194240.0, "12085": 972673152.0, "12090": 963907776.0, "12095": 963627072.0, "12100": 976849088.0, "12105": 950520256.0, "12110": 971690368.0, "12115": 968300352.0, "12120": 986518208.0, "12125": 980798080.0, "12130": 941565184.0, "12135": 955700864.0, "12140": 976100864.0, "12145": 979711616.0, "12150": 979730944.0, "12155": 962112384.0, "12160": 946594176.0, "12165": 968588032.0, "12170": 964645760.0, "12175": 967863936.0, "12180": 975310656.0, "12185": 953131520.0, "12190": 988619136.0, "12195": 970961088.0, "12200": 965117120.0, "12205": 968926720.0, "12210": 939426496.0, "12215": 997290688.0, "12220": 970357184.0, "12225": 979974272.0, "12230": 980481408.0, "12235": 950285504.0, "12240": 963896832.0, "12245": 966011648.0, "12250": 976881216.0, "12255": 968239744.0, "12260": 983813760.0, "12265": 932017664.0, "12270": 966679040.0, "12275": 979923008.0, "12280": 977409216.0, "12285": 970702528.0, "12290": 929600256.0, "12295": 977118656.0, "12300": 986039360.0, "12305": 970126848.0, "12310": 986129472.0, "12315": 936110464.0, "12320": 957900288.0, "12325": 966485952.0, "12330": 968416512.0, "12335": 963885568.0, "12340": 957481216.0, "12345": 944333504.0, "12350": 966714496.0, "12355": 975720896.0, "12360": 978980352.0, "12365": 964351680.0, "12370": 949002240.0, "12375": 963753920.0, "12380": 964518144.0, "12385": 973139392.0, "12390": 961403136.0, "12395": 961530944.0, "12400": 975517120.0, "12405": 976523520.0, "12410": 953484928.0, "12415": 963238144.0, "12420": 944371136.0, "12425": 949470016.0, "12430": 972612736.0, "12435": 968709440.0, "12440": 962225600.0, "12445": 952293120.0, "12450": 947771904.0, "12455": 981396992.0, "12460": 974272512.0, "12465": 954487232.0, "12470": 981144320.0, "12475": 958704384.0, "12480": 967121920.0, "12485": 978366016.0, "12490": 974278784.0, "12495": 969692096.0, "12500": 961659904.0, "12505": 943534912.0, "12510": 960928768.0, "12515": 969530304.0, "12520": 973764160.0, "12525": 972291392.0, "12530": 944744576.0, "12535": 976634496.0, "12540": 965682880.0, "12545": 972057792.0, "12550": 969770432.0, "12555": 941177664.0, "12560": 964738560.0, "12565": 947584320.0, "12570": 974419712.0, "12575": 962993280.0, "12580": 958078592.0, "12585": 964422976.0, "12590": 965961088.0, "12595": 978854528.0, "12600": 981995776.0, "12605": 949260736.0, "12610": 937854656.0, "12615": 962619712.0, "12620": 961337152.0, "12625": 966494016.0, "12630": 970714112.0, "12635": 962162688.0, "12640": 978283392.0, "12645": 969272128.0, "12650": 970076032.0, "12655": 964130240.0, "12660": 932255680.0, "12665": 956938560.0, "12670": 986216960.0, "12675": 965529856.0, "12680": 961113536.0, "12685": 951218048.0, "12690": 945392256.0, "12695": 978288768.0, "12700": 985215808.0, "12705": 958807744.0, "12710": 968417792.0, "12715": 956500736.0, "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}, "mem-allocated-bytes": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": 12698293248.0, "5": 12698293248.0, "10": 12698293248.0, "15": 12698293248.0, "20": 12698293248.0, "25": 12698293248.0, "30": 12698293248.0, "35": 12698293248.0, "40": 12698293248.0, "45": 12698293248.0, "50": 12698293248.0, "55": 12698293248.0, "60": 12698293248.0, "65": 12698293248.0, "70": 12698293248.0, "75": 12698293248.0, "80": 12698293248.0, "85": 12698293248.0, "90": 12698293248.0, "95": 12698293248.0, "100": 12698293248.0, "105": 12698293248.0, "110": 12698293248.0, "115": 12698293248.0, "120": 12698293248.0, "125": 12698293248.0, "130": 12698293248.0, "135": 12698293248.0, "140": 12698293248.0, "145": 12698293248.0, "150": 12698293248.0, "155": 12698293248.0, "160": 12698293248.0, "165": 12698293248.0, "170": 12698293248.0, "175": 12698293248.0, "180": 12698293248.0, "185": 12698293248.0, "190": 12698293248.0, "195": 12698293248.0, "200": 12698293248.0, "205": 12698293248.0, "210": 12698293248.0, "215": 12698293248.0, "220": 12698293248.0, "225": 12698293248.0, "230": 12698293248.0, "235": 12698293248.0, "240": 12698293248.0, "245": 12698293248.0, "250": 12698293248.0, "255": 12698293248.0, "260": 12698293248.0, "265": 12698293248.0, "270": 12698293248.0, "275": 12698293248.0, "280": 12698293248.0, "285": 12698293248.0, "290": 12698293248.0, "295": 12698293248.0, "300": 12698293248.0, "305": 12698293248.0, "310": 12698293248.0, "315": 12698293248.0, "320": 12698293248.0, "325": 12698293248.0, "330": 12698293248.0, "335": 12698293248.0, "340": 12698293248.0, "345": 12698293248.0, "350": 12698293248.0, "355": 12698293248.0, "360": 12698293248.0, "365": 12698293248.0, "370": 12698293248.0, "375": 12698293248.0, "380": 12698293248.0, "385": 12698293248.0, "390": 12698293248.0, "395": 12698293248.0, "400": 12698293248.0, "405": 12698293248.0, "410": 12698293248.0, "415": 12698293248.0, "420": 12698293248.0, "425": 12698293248.0, "430": 12698293248.0, "435": 12698293248.0, "440": 12698293248.0, "445": 12698293248.0, "450": 12698293248.0, "455": 12698293248.0, "460": 12698293248.0, "465": 12698293248.0, "470": 12698293248.0, "475": 12698293248.0, "480": 12698293248.0, "485": 12698293248.0, "490": 12698293248.0, "495": 12698293248.0, "500": 12698293248.0, "505": 12698293248.0, "510": 12698293248.0, "515": 12698293248.0, "520": 12698293248.0, "525": 12698293248.0, "530": 12698293248.0, "535": 12698293248.0, "540": 12698293248.0, "545": 12698293248.0, "550": 12698293248.0, "555": 12698293248.0, "560": 12698293248.0, "565": 12698293248.0, "570": 12698293248.0, "575": 12698293248.0, "580": 12698293248.0, "585": 12698293248.0, "590": 12698293248.0, "595": 12698293248.0, "600": 12698293248.0, "605": 12698293248.0, "610": 12698293248.0, "615": 12698293248.0, "620": 12698293248.0, "625": 12698293248.0, "630": 12698293248.0, "635": 12698293248.0, "640": 12698293248.0, "645": 12698293248.0, "650": 12698293248.0, "655": 12698293248.0, "660": 12698293248.0, "665": 12698293248.0, "670": 12698293248.0, "675": 12698293248.0, "680": 12698293248.0, "685": 12698293248.0, "690": 12698293248.0, "695": 12698293248.0, "700": 12698293248.0, "705": 12698293248.0, "710": 12698293248.0, "715": 12698293248.0, "720": 12698293248.0, "725": 12698293248.0, "730": 12698293248.0, "735": 12698293248.0, "740": 12698293248.0, "745": 12698293248.0, "750": 12698293248.0, "755": 12698293248.0, "760": 12698293248.0, "765": 12698293248.0, "770": 12698293248.0, "775": 12698293248.0, "780": 12698293248.0, "785": 12698293248.0, "790": 12698293248.0, "795": 12698293248.0, "800": 12698293248.0, "805": 12698293248.0, "810": 12698293248.0, "815": 12698293248.0, "820": 12698293248.0, "825": 12698293248.0, "830": 12698293248.0, "835": 12698293248.0, "840": 12698293248.0, "845": 12698293248.0, "850": 12698293248.0, "855": 12698293248.0, "860": 12698293248.0, "865": 12698293248.0, "870": 12698293248.0, "875": 12698293248.0, "880": 12698293248.0, "885": 12698293248.0, "890": 12698293248.0, "895": 12698293248.0, "900": 12698293248.0, "905": 12698293248.0, "910": 12698293248.0, "915": 12698293248.0, "920": 12698293248.0, "925": 12698293248.0, "930": 12698293248.0, "935": 12698293248.0, "940": 12698293248.0, "945": 12698293248.0, "950": 12698293248.0, "955": 12698293248.0, "960": 12698293248.0, "965": 12698293248.0, "970": 12698293248.0, "975": 12698293248.0, "980": 12698293248.0, "985": 12698293248.0, "990": 12698293248.0, "995": 12698293248.0, "1000": 12698293248.0, "1005": 12698293248.0, "1010": 12698293248.0, "1015": 12698293248.0, "1020": 12698293248.0, "1025": 12698293248.0, "1030": 12698293248.0, "1035": 12698293248.0, "1040": 12698293248.0, "1045": 12698293248.0, "1050": 12698293248.0, "1055": 12698293248.0, "1060": 12698293248.0, "1065": 12698293248.0, "1070": 12698293248.0, "1075": 12698293248.0, "1080": 12698293248.0, "1085": 12698293248.0, "1090": 12698293248.0, "1095": 12698293248.0, "1100": 12698293248.0, "1105": 12698293248.0, "1110": 12698293248.0, "1115": 12698293248.0, "1120": 12698293248.0, "1125": 12698293248.0, "1130": 12698293248.0, "1135": 12698293248.0, "1140": 12698293248.0, "1145": 12698293248.0, "1150": 12698293248.0, "1155": 12698293248.0, "1160": 12698293248.0, "1165": 12698293248.0, "1170": 12698293248.0, "1175": 12698293248.0, "1180": 12698293248.0, "1185": 12698293248.0, "1190": 12698293248.0, "1195": 12698293248.0, "1200": 12698293248.0, "1205": 12698293248.0, "1210": 12698293248.0, "1215": 12698293248.0, "1220": 12698293248.0, "1225": 12698293248.0, "1230": 12698293248.0, "1235": 12698293248.0, "1240": 12698293248.0, "1245": 12698293248.0, "1250": 12698293248.0, "1255": 12698293248.0, "1260": 12698293248.0, "1265": 12698293248.0, "1270": 12698293248.0, "1275": 12698293248.0, "1280": 12698293248.0, "1285": 12698293248.0, "1290": 12698293248.0, "1295": 12698293248.0, "1300": 12698293248.0, "1305": 12698293248.0, "1310": 12698293248.0, "1315": 12698293248.0, "1320": 12698293248.0, "1325": 12698293248.0, "1330": 12698293248.0, "1335": 12698293248.0, "1340": 12698293248.0, "1345": 12698293248.0, "1350": 12698293248.0, "1355": 12698293248.0, "1360": 12698293248.0, "1365": 12698293248.0, "1370": 12698293248.0, "1375": 12698293248.0, "1380": 12698293248.0, "1385": 12698293248.0, "1390": 12698293248.0, "1395": 12698293248.0, "1400": 12698293248.0, "1405": 12698293248.0, "1410": 12698293248.0, "1415": 12698293248.0, "1420": 12698293248.0, "1425": 12698293248.0, "1430": 12698293248.0, "1435": 12698293248.0, "1440": 12698293248.0, "1445": 12698293248.0, "1450": 12698293248.0, "1455": 12698293248.0, "1460": 12698293248.0, "1465": 12698293248.0, "1470": 12698293248.0, "1475": 12698293248.0, "1480": 12698293248.0, "1485": 12698293248.0, "1490": 12698293248.0, "1495": 12698293248.0, "1500": 12698293248.0, "1505": 12698293248.0, "1510": 12698293248.0, "1515": 12698293248.0, "1520": 12698293248.0, "1525": 12698293248.0, "1530": 12698293248.0, "1535": 12698293248.0, "1540": 12698293248.0, "1545": 12698293248.0, "1550": 12698293248.0, "1555": 12698293248.0, "1560": 12698293248.0, "1565": 12698293248.0, "1570": 12698293248.0, "1575": 12698293248.0, "1580": 12698293248.0, "1585": 12698293248.0, "1590": 12698293248.0, "1595": 12698293248.0, "1600": 12698293248.0, "1605": 12698293248.0, "1610": 12698293248.0, "1615": 12698293248.0, "1620": 12698293248.0, "1625": 12698293248.0, "1630": 12698293248.0, "1635": 12698293248.0, "1640": 12698293248.0, "1645": 12698293248.0, "1650": 12698293248.0, "1655": 12698293248.0, "1660": 12698293248.0, "1665": 12698293248.0, "1670": 12698293248.0, "1675": 12698293248.0, "1680": 12698293248.0, "1685": 12698293248.0, "1690": 12698293248.0, "1695": 12698293248.0, "1700": 12698293248.0, "1705": 12698293248.0, "1710": 12698293248.0, "1715": 12698293248.0, "1720": 12698293248.0, "1725": 12698293248.0, "1730": 12698293248.0, "1735": 12698293248.0, "1740": 12698293248.0, "1745": 12698293248.0, "1750": 12698293248.0, "1755": 12698293248.0, "1760": 12698293248.0, "1765": 12698293248.0, "1770": 12698293248.0, "1775": 12698293248.0, "1780": 12698293248.0, "1785": 12698293248.0, "1790": 12698293248.0, "1795": 12698293248.0, "1800": 12698293248.0, "1805": 12698293248.0, "1810": 12698293248.0, "1815": 12698293248.0, "1820": 12698293248.0, "1825": 12698293248.0, "1830": 12698293248.0, "1835": 12698293248.0, "1840": 12698293248.0, "1845": 12698293248.0, "1850": 12698293248.0, "1855": 12698293248.0, "1860": 12698293248.0, "1865": 12698293248.0, "1870": 12698293248.0, "1875": 12698293248.0, "1880": 12698293248.0, "1885": 12698293248.0, "1890": 12698293248.0, "1895": 12698293248.0, "1900": 12698293248.0, "1905": 12698293248.0, "1910": 12698293248.0, "1915": 12698293248.0, "1920": 12698293248.0, "1925": 12698293248.0, "1930": 12698293248.0, "1935": 12698293248.0, "1940": 12698293248.0, "1945": 12698293248.0, "1950": 12698293248.0, "1955": 12698293248.0, "1960": 12698293248.0, "1965": 12698293248.0, "1970": 12698293248.0, "1975": 12698293248.0, "1980": 12698293248.0, "1985": 12698293248.0, "1990": 12698293248.0, "1995": 12698293248.0, "2000": 12698293248.0, "2005": 12698293248.0, "2010": 12698293248.0, "2015": 12698293248.0, "2020": 12698293248.0, "2025": 12698293248.0, "2030": 12698293248.0, "2035": 12698293248.0, "2040": 12698293248.0, "2045": 12698293248.0, "2050": 12698293248.0, "2055": 12698293248.0, "2060": 12698293248.0, "2065": 12698293248.0, "2070": 12698293248.0, "2075": 12698293248.0, "2080": 12698293248.0, "2085": 12698293248.0, "2090": 12698293248.0, "2095": 12698293248.0, "2100": 12698293248.0, "2105": 12698293248.0, "2110": 12698293248.0, "2115": 12698293248.0, "2120": 12698293248.0, "2125": 12698293248.0, "2130": 12698293248.0, "2135": 12698293248.0, "2140": 12698293248.0, "2145": 12698293248.0, "2150": 12698293248.0, "2155": 12698293248.0, "2160": 12698293248.0, "2165": 12698293248.0, "2170": 12698293248.0, "2175": 12698293248.0, "2180": 12698293248.0, "2185": 12698293248.0, "2190": 12698293248.0, "2195": 12698293248.0, "2200": 12698293248.0, "2205": 12698293248.0, "2210": 12698293248.0, "2215": 12698293248.0, "2220": 12698293248.0, "2225": 12698293248.0, "2230": 12698293248.0, "2235": 12698293248.0, "2240": 12698293248.0, "2245": 12698293248.0, "2250": 12698293248.0, "2255": 12698293248.0, "2260": 12698293248.0, "2265": 12698293248.0, "2270": 12698293248.0, "2275": 12698293248.0, "2280": 12698293248.0, "2285": 12698293248.0, "2290": 12698293248.0, "2295": 12698293248.0, "2300": 12698293248.0, "2305": 12698293248.0, "2310": 12698293248.0, "2315": 12698293248.0, "2320": 12698293248.0, "2325": 12698293248.0, "2330": 12698293248.0, "2335": 12698293248.0, "2340": 12698293248.0, "2345": 12698293248.0, "2350": 12698293248.0, "2355": 12698293248.0, "2360": 12698293248.0, "2365": 12698293248.0, "2370": 12698293248.0, "2375": 12698293248.0, "2380": 12698293248.0, "2385": 12698293248.0, "2390": 12698293248.0, "2395": 12698293248.0, "2400": 12698293248.0, "2405": 12698293248.0, "2410": 12698293248.0, "2415": 12698293248.0, "2420": 12698293248.0, "2425": 12698293248.0, "2430": 12698293248.0, "2435": 12698293248.0, "2440": 12698293248.0, "2445": 12698293248.0, "2450": 12698293248.0, "2455": 12698293248.0, "2460": 12698293248.0, "2465": 12698293248.0, "2470": 12698293248.0, "2475": 12698293248.0, "2480": 12698293248.0, "2485": 12698293248.0, "2490": 12698293248.0, "2495": 12698293248.0, "2500": 12698293248.0, "2505": 12698293248.0, "2510": 12698293248.0, "2515": 12698293248.0, "2520": 12698293248.0, "2525": 12698293248.0, "2530": 12698293248.0, "2535": 12698293248.0, "2540": 12698293248.0, "2545": 12698293248.0, "2550": 12698293248.0, "2555": 12698293248.0, "2560": 12698293248.0, "2565": 12698293248.0, "2570": 12698293248.0, "2575": 12698293248.0, "2580": 12698293248.0, "2585": 12698293248.0, "2590": 12698293248.0, "2595": 12698293248.0, "2600": 12698293248.0, "2605": 12698293248.0, "2610": 12698293248.0, "2615": 12698293248.0, "2620": 12698293248.0, "2625": 12698293248.0, "2630": 12698293248.0, "2635": 12698293248.0, "2640": 12698293248.0, "2645": 12698293248.0, "2650": 12698293248.0, "2655": 12698293248.0, "2660": 12698293248.0, "2665": 12698293248.0, "2670": 12698293248.0, "2675": 12698293248.0, "2680": 12698293248.0, "2685": 12698293248.0, "2690": 12698293248.0, "2695": 12698293248.0, "2700": 12698293248.0, "2705": 12698293248.0, "2710": 12698293248.0, "2715": 12698293248.0, "2720": 12698293248.0, "2725": 12698293248.0, "2730": 12698293248.0, "2735": 12698293248.0, "2740": 12698293248.0, "2745": 12698293248.0, "2750": 12698293248.0, "2755": 12698293248.0, "2760": 12698293248.0, "2765": 12698293248.0, "2770": 12698293248.0, "2775": 12698293248.0, "2780": 12698293248.0, "2785": 12698293248.0, "2790": 12698293248.0, "2795": 12698293248.0, "2800": 12698293248.0, "2805": 12698293248.0, "2810": 12698293248.0, "2815": 12698293248.0, "2820": 12698293248.0, "2825": 12698293248.0, "2830": 12698293248.0, "2835": 12698293248.0, "2840": 12698293248.0, "2845": 12698293248.0, "2850": 12698293248.0, "2855": 12698293248.0, "2860": 12698293248.0, "2865": 12698293248.0, "2870": 12698293248.0, "2875": 12698293248.0, "2880": 12698293248.0, "2885": 12698293248.0, "2890": 12698293248.0, "2895": 12698293248.0, "2900": 12698293248.0, "2905": 12698293248.0, "2910": 12698293248.0, "2915": 12698293248.0, "2920": 12698293248.0, "2925": 12698293248.0, "2930": 12698293248.0, "2935": 12698293248.0, "2940": 12698293248.0, "2945": 12698293248.0, "2950": 12698293248.0, "2955": 12698293248.0, "2960": 12698293248.0, "2965": 12698293248.0, "2970": 12698293248.0, "2975": 12698293248.0, "2980": 12698293248.0, "2985": 12698293248.0, "2990": 12698293248.0, "2995": 12698293248.0, "3000": 12698293248.0, "3005": 12698293248.0, "3010": 12698293248.0, "3015": 12698293248.0, "3020": 12698293248.0, "3025": 12698293248.0, "3030": 12698293248.0, "3035": 12698293248.0, "3040": 12698293248.0, "3045": 12698293248.0, "3050": 12698293248.0, "3055": 12698293248.0, "3060": 12698293248.0, "3065": 12698293248.0, "3070": 12698293248.0, "3075": 12698293248.0, "3080": 12698293248.0, "3085": 12698293248.0, "3090": 12698293248.0, "3095": 12698293248.0, "3100": 12698293248.0, "3105": 12698293248.0, "3110": 12698293248.0, "3115": 12698293248.0, "3120": 12698293248.0, "3125": 12698293248.0, "3130": 12698293248.0, "3135": 12698293248.0, "3140": 12698293248.0, "3145": 12698293248.0, "3150": 12698293248.0, "3155": 12698293248.0, "3160": 12698293248.0, "3165": 12698293248.0, "3170": 12698293248.0, "3175": 12698293248.0, "3180": 12698293248.0, "3185": 12698293248.0, "3190": 12698293248.0, "3195": 12698293248.0, "3200": 12698293248.0, "3205": 12698293248.0, "3210": 12698293248.0, "3215": 12698293248.0, "3220": 12698293248.0, "3225": 12698293248.0, "3230": 12698293248.0, "3235": 12698293248.0, "3240": 12698293248.0, "3245": 12698293248.0, "3250": 12698293248.0, "3255": 12698293248.0, "3260": 12698293248.0, "3265": 12698293248.0, "3270": 12698293248.0, "3275": 12698293248.0, "3280": 12698293248.0, "3285": 12698293248.0, "3290": 12698293248.0, "3295": 12698293248.0, "3300": 12698293248.0, "3305": 12698293248.0, "3310": 12698293248.0, "3315": 12698293248.0, "3320": 12698293248.0, "3325": 12698293248.0, "3330": 12698293248.0, "3335": 12698293248.0, "3340": 12698293248.0, "3345": 12698293248.0, "3350": 12698293248.0, "3355": 12698293248.0, "3360": 12698293248.0, "3365": 12698293248.0, "3370": 12698293248.0, "3375": 12698293248.0, "3380": 12698293248.0, "3385": 12698293248.0, "3390": 12698293248.0, "3395": 12698293248.0, "3400": 12698293248.0, "3405": 12698293248.0, "3410": 12698293248.0, "3415": 12698293248.0, "3420": 12698293248.0, "3425": 12698293248.0, "3430": 12698293248.0, "3435": 12698293248.0, "3440": 12698293248.0, "3445": 12698293248.0, "3450": 12698293248.0, "3455": 12698293248.0, "3460": 12698293248.0, "3465": 12698293248.0, "3470": 12698293248.0, "3475": 12698293248.0, "3480": 12698293248.0, "3485": 12698293248.0, "3490": 12698293248.0, "3495": 12698293248.0, "3500": 12698293248.0, "3505": 12698293248.0, "3510": 12698293248.0, "3515": 12698293248.0, "3520": 12698293248.0, "3525": 12698293248.0, "3530": 12698293248.0, "3535": 12698293248.0, "3540": 12698293248.0, "3545": 12698293248.0, "3550": 12698293248.0, "3555": 12698293248.0, "3560": 12698293248.0, "3565": 12698293248.0, "3570": 12698293248.0, "3575": 12698293248.0, "3580": 12698293248.0, "3585": 12698293248.0, "3590": 12698293248.0, "3595": 12698293248.0, "3600": 12698293248.0, "3605": 12698293248.0, "3610": 12698293248.0, "3615": 12698293248.0, "3620": 12698293248.0, "3625": 12698293248.0, "3630": 12698293248.0, "3635": 12698293248.0, "3640": 12698293248.0, "3645": 12698293248.0, "3650": 12698492928.0, "3655": 12698492928.0, "3660": 12698492928.0, "3665": 12698492928.0, "3670": 12698492928.0, "3675": 12698492928.0, "3680": 12698492928.0, "3685": 12698492928.0, "3690": 12698492928.0, "3695": 12698492928.0, "3700": 12698492928.0, "3705": 12698492928.0, "3710": 12698492928.0, "3715": 12698492928.0, "3720": 12698492928.0, "3725": 12698492928.0, "3730": 12698492928.0, "3735": 12698492928.0, "3740": 12698492928.0, "3745": 12698492928.0, "3750": 12698492928.0, "3755": 12698492928.0, "3760": 12698492928.0, "3765": 12698492928.0, "3770": 12698492928.0, "3775": 12698492928.0, "3780": 12698492928.0, "3785": 12698492928.0, "3790": 12698492928.0, "3795": 12698492928.0, "3800": 12698492928.0, "3805": 12698492928.0, "3810": 12698492928.0, "3815": 12698492928.0, "3820": 12698492928.0, "3825": 12698492928.0, "3830": 12698492928.0, "3835": 12698492928.0, "3840": 12698492928.0, "3845": 12698492928.0, "3850": 12698492928.0, "3855": 12698492928.0, "3860": 12698492928.0, "3865": 12698492928.0, "3870": 12698492928.0, "3875": 12698492928.0, "3880": 12698492928.0, "3885": 12698492928.0, "3890": 12698492928.0, "3895": 12698492928.0, "3900": 12698492928.0, "3905": 12698492928.0, "3910": 12698492928.0, "3915": 12698492928.0, "3920": 12698492928.0, "3925": 12698492928.0, "3930": 12698492928.0, "3935": 12698492928.0, "3940": 12698492928.0, "3945": 12698492928.0, "3950": 12698492928.0, "3955": 12698492928.0, "3960": 12698492928.0, "3965": 12698492928.0, "3970": 12698492928.0, "3975": 12698492928.0, "3980": 12698492928.0, "3985": 12698492928.0, "3990": 12698492928.0, "3995": 12698492928.0, "4000": 12698492928.0, "4005": 12698492928.0, "4010": 12698492928.0, "4015": 12698492928.0, "4020": 12698492928.0, "4025": 12698492928.0, "4030": 12698492928.0, "4035": 12698492928.0, "4040": 12698492928.0, "4045": 12698492928.0, "4050": 12698492928.0, "4055": 12698492928.0, "4060": 12698492928.0, "4065": 12698492928.0, "4070": 12698492928.0, "4075": 12698492928.0, "4080": 12698492928.0, "4085": 12698492928.0, "4090": 12698492928.0, "4095": 12698492928.0, "4100": 12698492928.0, "4105": 12698492928.0, "4110": 12698492928.0, "4115": 12698492928.0, "4120": 12698492928.0, "4125": 12698492928.0, "4130": 12698492928.0, "4135": 12698492928.0, "4140": 12698492928.0, "4145": 12698492928.0, "4150": 12698492928.0, "4155": 12698492928.0, "4160": 12698492928.0, "4165": 12698492928.0, "4170": 12698492928.0, "4175": 12698492928.0, "4180": 12698492928.0, "4185": 12698492928.0, "4190": 12698492928.0, "4195": 12698492928.0, "4200": 12698492928.0, "4205": 12698492928.0, "4210": 12698492928.0, "4215": 12698492928.0, "4220": 12698492928.0, "4225": 12698492928.0, "4230": 12698492928.0, "4235": 12698492928.0, "4240": 12698492928.0, "4245": 12698492928.0, "4250": 12698492928.0, "4255": 12698492928.0, "4260": 12698492928.0, "4265": 12698492928.0, "4270": 12698492928.0, "4275": 12698492928.0, "4280": 12698492928.0, "4285": 12698492928.0, "4290": 12698492928.0, "4295": 12698492928.0, "4300": 12698492928.0, "4305": 12698492928.0, "4310": 12698492928.0, "4315": 12698492928.0, "4320": 12698492928.0, "4325": 12698492928.0, "4330": 12698492928.0, "4335": 12698492928.0, "4340": 12698492928.0, "4345": 12698492928.0, "4350": 12698492928.0, "4355": 12698492928.0, "4360": 12698492928.0, "4365": 12698492928.0, "4370": 12698492928.0, "4375": 12698492928.0, "4380": 12698492928.0, "4385": 12698492928.0, "4390": 12698492928.0, "4395": 12698492928.0, "4400": 12698492928.0, "4405": 12698492928.0, "4410": 12698492928.0, "4415": 12698492928.0, "4420": 12698492928.0, "4425": 12698492928.0, "4430": 12698492928.0, "4435": 12698492928.0, "4440": 12698492928.0, "4445": 12698492928.0, "4450": 12698492928.0, "4455": 12698492928.0, "4460": 12698492928.0, "4465": 12698492928.0, "4470": 12698492928.0, "4475": 12698492928.0, "4480": 12698492928.0, "4485": 12698492928.0, "4490": 12698492928.0, "4495": 12698492928.0, "4500": 12698492928.0, "4505": 12698492928.0, "4510": 12698492928.0, "4515": 12698492928.0, "4520": 12698492928.0, "4525": 12698492928.0, "4530": 12698492928.0, "4535": 12698492928.0, "4540": 12698492928.0, "4545": 12698492928.0, "4550": 12698492928.0, "4555": 12698492928.0, "4560": 12698492928.0, "4565": 12698492928.0, "4570": 12698492928.0, "4575": 12698492928.0, "4580": 12698492928.0, "4585": 12698492928.0, "4590": 12698492928.0, "4595": 12698492928.0, "4600": 12698492928.0, "4605": 12698492928.0, "4610": 12698492928.0, "4615": 12698492928.0, "4620": 12698492928.0, "4625": 12698492928.0, "4630": 12698492928.0, "4635": 12698492928.0, "4640": 12698492928.0, "4645": 12698492928.0, "4650": 12698492928.0, "4655": 12698492928.0, "4660": 12698492928.0, "4665": 12698492928.0, "4670": 12698492928.0, "4675": 12698492928.0, "4680": 12698492928.0, "4685": 12698492928.0, "4690": 12698492928.0, "4695": 12698492928.0, "4700": 12698492928.0, "4705": 12698492928.0, "4710": 12698492928.0, "4715": 12698492928.0, "4720": 12698492928.0, "4725": 12698492928.0, "4730": 12698492928.0, "4735": 12698492928.0, "4740": 12698492928.0, "4745": 12698492928.0, "4750": 12698492928.0, "4755": 12698492928.0, "4760": 12698492928.0, "4765": 12698492928.0, "4770": 12698492928.0, "4775": 12698492928.0, "4780": 12698492928.0, "4785": 12698492928.0, "4790": 12698492928.0, "4795": 12698492928.0, "4800": 12698492928.0, "4805": 12698492928.0, "4810": 12698492928.0, "4815": 12698492928.0, "4820": 12698492928.0, "4825": 12698492928.0, "4830": 12698492928.0, "4835": 12698492928.0, "4840": 12698492928.0, "4845": 12698492928.0, "4850": 12698492928.0, "4855": 12698492928.0, "4860": 12698492928.0, "4865": 12698492928.0, "4870": 12698492928.0, "4875": 12698492928.0, "4880": 12698492928.0, "4885": 12698492928.0, "4890": 12698492928.0, "4895": 12698492928.0, "4900": 12698492928.0, "4905": 12698492928.0, "4910": 12698492928.0, "4915": 12698492928.0, "4920": 12698492928.0, "4925": 12698492928.0, "4930": 12698492928.0, "4935": 12698492928.0, "4940": 12698492928.0, "4945": 12698492928.0, "4950": 12698492928.0, "4955": 12698492928.0, "4960": 12698492928.0, "4965": 12698492928.0, "4970": 12698492928.0, "4975": 12698492928.0, "4980": 12698492928.0, "4985": 12698492928.0, "4990": 12698492928.0, "4995": 12698492928.0, "5000": 12698492928.0, "5005": 12698492928.0, "5010": 12698492928.0, "5015": 12698492928.0, "5020": 12698492928.0, "5025": 12698492928.0, "5030": 12698492928.0, "5035": 12698492928.0, "5040": 12698492928.0, "5045": 12698492928.0, "5050": 12698492928.0, "5055": 12698492928.0, "5060": 12698492928.0, "5065": 12698492928.0, "5070": 12698492928.0, "5075": 12698492928.0, "5080": 12698492928.0, "5085": 12698492928.0, "5090": 12698492928.0, "5095": 12698492928.0, "5100": 12698492928.0, "5105": 12698492928.0, "5110": 12698492928.0, "5115": 12698492928.0, "5120": 12698492928.0, "5125": 12698492928.0, "5130": 12698492928.0, "5135": 12698492928.0, "5140": 12698492928.0, "5145": 12698492928.0, "5150": 12698492928.0, "5155": 12698492928.0, "5160": 12698492928.0, "5165": 12698492928.0, "5170": 12698492928.0, "5175": 12698492928.0, "5180": 12698492928.0, "5185": 12698492928.0, "5190": 12698492928.0, "5195": 12698492928.0, "5200": 12698492928.0, "5205": 12698492928.0, "5210": 12698492928.0, "5215": 12698492928.0, "5220": 12698492928.0, "5225": 12698492928.0, "5230": 12698492928.0, "5235": 12698492928.0, "5240": 12698492928.0, "5245": 12698492928.0, "5250": 12698492928.0, "5255": 12698492928.0, "5260": 12698492928.0, "5265": 12698492928.0, "5270": 12698492928.0, "5275": 12698492928.0, "5280": 12698492928.0, "5285": 12698492928.0, "5290": 12698492928.0, "5295": 12698492928.0, "5300": 12698492928.0, "5305": 12698492928.0, "5310": 12698492928.0, "5315": 12698492928.0, "5320": 12698492928.0, "5325": 12698492928.0, "5330": 12698492928.0, "5335": 12698492928.0, "5340": 12698492928.0, "5345": 12698492928.0, "5350": 12698492928.0, "5355": 12698492928.0, "5360": 12698492928.0, "5365": 12698492928.0, "5370": 12698492928.0, "5375": 12698492928.0, "5380": 12698492928.0, "5385": 12698492928.0, "5390": 12698492928.0, "5395": 12698492928.0, "5400": 12698492928.0, "5405": 12698492928.0, "5410": 12698492928.0, "5415": 12698492928.0, "5420": 12698492928.0, "5425": 12698492928.0, "5430": 12698492928.0, "5435": 12698492928.0, "5440": 12698492928.0, "5445": 12698492928.0, "5450": 12698492928.0, "5455": 12698492928.0, "5460": 12698492928.0, "5465": 12698492928.0, "5470": 12698492928.0, "5475": 12698492928.0, "5480": 12698492928.0, "5485": 12698492928.0, "5490": 12698492928.0, "5495": 12698492928.0, "5500": 12698492928.0, "5505": 12698492928.0, "5510": 12698492928.0, "5515": 12698492928.0, "5520": 12698492928.0, "5525": 12698492928.0, "5530": 12698492928.0, "5535": 12698492928.0, "5540": 12698492928.0, "5545": 12698492928.0, "5550": 12698492928.0, "5555": 12698492928.0, "5560": 12698492928.0, "5565": 12698492928.0, "5570": 12698492928.0, "5575": 12698492928.0, "5580": 12698492928.0, "5585": 12698492928.0, "5590": 12698492928.0, "5595": 12698492928.0, "5600": 12698492928.0, "5605": 12698492928.0, "5610": 12698492928.0, "5615": 12698492928.0, "5620": 12698492928.0, "5625": 12698492928.0, "5630": 12698492928.0, "5635": 12698492928.0, "5640": 12698492928.0, "5645": 12698492928.0, "5650": 12698492928.0, "5655": 12698492928.0, "5660": 12698492928.0, "5665": 12698492928.0, "5670": 12698492928.0, "5675": 12698492928.0, "5680": 12698492928.0, "5685": 12698492928.0, "5690": 12698492928.0, "5695": 12698492928.0, "5700": 12698492928.0, "5705": 12698492928.0, "5710": 12698492928.0, "5715": 12698492928.0, "5720": 12698492928.0, "5725": 12698492928.0, "5730": 12698492928.0, "5735": 12698492928.0, "5740": 12698492928.0, "5745": 12698492928.0, "5750": 12698492928.0, "5755": 12698492928.0, "5760": 12698492928.0, "5765": 12698492928.0, "5770": 12698492928.0, "5775": 12698492928.0, "5780": 12698492928.0, "5785": 12698492928.0, "5790": 12698492928.0, "5795": 12698492928.0, "5800": 12698492928.0, "5805": 12698492928.0, "5810": 12698492928.0, "5815": 12698492928.0, "5820": 12698492928.0, "5825": 12698492928.0, "5830": 12698492928.0, "5835": 12698492928.0, "5840": 12698492928.0, "5845": 12698492928.0, "5850": 12698492928.0, "5855": 12698492928.0, "5860": 12698492928.0, "5865": 12698492928.0, "5870": 12698492928.0, "5875": 12698492928.0, "5880": 12698492928.0, "5885": 12698492928.0, "5890": 12698492928.0, "5895": 12698492928.0, "5900": 12698492928.0, "5905": 12698492928.0, "5910": 12698492928.0, "5915": 12698492928.0, "5920": 12698492928.0, "5925": 12698492928.0, "5930": 12698492928.0, "5935": 12698492928.0, "5940": 12698492928.0, "5945": 12698492928.0, "5950": 12698492928.0, "5955": 12698492928.0, "5960": 12698492928.0, "5965": 12698492928.0, "5970": 12698492928.0, "5975": 12698492928.0, "5980": 12698492928.0, "5985": 12698492928.0, "5990": 12698492928.0, "5995": 12698492928.0, "6000": 12698492928.0, "6005": 12698492928.0, "6010": 12698492928.0, "6015": 12698492928.0, "6020": 12698492928.0, "6025": 12698492928.0, "6030": 12698492928.0, "6035": 12698492928.0, "6040": 12698492928.0, "6045": 12698492928.0, "6050": 12698492928.0, "6055": 12698492928.0, "6060": 12698492928.0, "6065": 12698492928.0, "6070": 12698492928.0, "6075": 12698492928.0, "6080": 12698492928.0, "6085": 12698492928.0, "6090": 12698492928.0, "6095": 12698492928.0, "6100": 12698492928.0, "6105": 12698492928.0, "6110": 12698492928.0, "6115": 12698492928.0, "6120": 12698492928.0, "6125": 12698492928.0, "6130": 12698492928.0, "6135": 12698492928.0, "6140": 12698492928.0, "6145": 12698492928.0, "6150": 12698492928.0, "6155": 12698492928.0, "6160": 12698492928.0, "6165": 12698492928.0, "6170": 12698492928.0, "6175": 12698492928.0, "6180": 12698492928.0, "6185": 12698492928.0, "6190": 12698492928.0, "6195": 12698492928.0, "6200": 12698492928.0, "6205": 12698492928.0, "6210": 12698492928.0, "6215": 12698492928.0, "6220": 12698492928.0, "6225": 12698492928.0, "6230": 12698492928.0, "6235": 12698492928.0, "6240": 12698492928.0, "6245": 12698492928.0, "6250": 12698492928.0, "6255": 12698492928.0, "6260": 12698492928.0, "6265": 12698492928.0, "6270": 12698492928.0, "6275": 12698492928.0, "6280": 12698492928.0, "6285": 12698492928.0, "6290": 12698492928.0, "6295": 12698492928.0, "6300": 12698492928.0, "6305": 12698492928.0, "6310": 12698492928.0, "6315": 12698492928.0, "6320": 12698492928.0, "6325": 12698492928.0, "6330": 12698492928.0, "6335": 12698492928.0, "6340": 12698492928.0, "6345": 12698492928.0, "6350": 12698492928.0, "6355": 12698492928.0, "6360": 12698492928.0, "6365": 12698492928.0, "6370": 12698492928.0, "6375": 12698492928.0, "6380": 12698492928.0, "6385": 12698492928.0, "6390": 12698492928.0, "6395": 12698492928.0, "6400": 12698492928.0, "6405": 12698492928.0, "6410": 12698492928.0, "6415": 12698492928.0, "6420": 12698492928.0, "6425": 12698492928.0, "6430": 12698492928.0, "6435": 12698492928.0, "6440": 12698492928.0, "6445": 12698492928.0, "6450": 12698492928.0, "6455": 12698492928.0, "6460": 12698492928.0, "6465": 12698492928.0, "6470": 12698492928.0, "6475": 12698492928.0, "6480": 12698492928.0, "6485": 12698492928.0, "6490": 12698492928.0, "6495": 12698492928.0, "6500": 12698492928.0, "6505": 12698492928.0, "6510": 12698492928.0, "6515": 12698492928.0, "6520": 12698492928.0, "6525": 12698492928.0, "6530": 12698492928.0, "6535": 12698492928.0, "6540": 12698492928.0, "6545": 12698492928.0, "6550": 12698492928.0, "6555": 12698492928.0, "6560": 12698492928.0, "6565": 12698492928.0, "6570": 12698492928.0, "6575": 12698492928.0, "6580": 12698492928.0, "6585": 12698492928.0, "6590": 12698492928.0, "6595": 12698492928.0, "6600": 12698492928.0, "6605": 12698492928.0, "6610": 12698492928.0, "6615": 12698492928.0, "6620": 12698492928.0, "6625": 12698492928.0, "6630": 12698492928.0, "6635": 12698492928.0, "6640": 12698492928.0, "6645": 12698492928.0, "6650": 12698492928.0, "6655": 12698492928.0, "6660": 12698492928.0, "6665": 12698492928.0, "6670": 12698492928.0, "6675": 12698492928.0, "6680": 12698492928.0, "6685": 12698492928.0, "6690": 12698492928.0, "6695": 12698492928.0, "6700": 12698492928.0, "6705": 12698492928.0, "6710": 12698492928.0, "6715": 12698492928.0, "6720": 12698492928.0, "6725": 12698492928.0, "6730": 12698492928.0, "6735": 12698492928.0, "6740": 12698492928.0, "6745": 12698492928.0, "6750": 12698492928.0, "6755": 12698492928.0, "6760": 12698492928.0, "6765": 12698492928.0, "6770": 12698492928.0, "6775": 12698492928.0, "6780": 12698492928.0, "6785": 12698492928.0, "6790": 12698492928.0, "6795": 12698492928.0, "6800": 12698492928.0, "6805": 12698492928.0, "6810": 12698492928.0, "6815": 12698492928.0, "6820": 12698492928.0, "6825": 12698492928.0, "6830": 12698492928.0, "6835": 12698492928.0, "6840": 12698492928.0, "6845": 12698492928.0, "6850": 12698492928.0, "6855": 12698492928.0, "6860": 12698492928.0, "6865": 12698492928.0, "6870": 12698492928.0, "6875": 12698492928.0, "6880": 12698492928.0, "6885": 12698492928.0, "6890": 12698492928.0, "6895": 12698492928.0, "6900": 12698492928.0, "6905": 12698492928.0, "6910": 12698492928.0, "6915": 12698492928.0, "6920": 12698492928.0, "6925": 12698492928.0, "6930": 12698492928.0, "6935": 12698492928.0, "6940": 12698492928.0, "6945": 12698492928.0, "6950": 12698492928.0, "6955": 12698492928.0, "6960": 12698492928.0, "6965": 12698492928.0, "6970": 12698492928.0, "6975": 12698492928.0, "6980": 12698492928.0, "6985": 12698492928.0, "6990": 12698492928.0, "6995": 12698492928.0, "7000": 12698492928.0, "7005": 12698492928.0, "7010": 12698492928.0, "7015": 12698492928.0, "7020": 12698492928.0, "7025": 12698492928.0, "7030": 12698492928.0, "7035": 12698492928.0, "7040": 12698492928.0, "7045": 12698492928.0, "7050": 12698492928.0, "7055": 12698492928.0, "7060": 12698492928.0, "7065": 12698492928.0, "7070": 12698492928.0, "7075": 12698492928.0, "7080": 12698492928.0, "7085": 12698492928.0, "7090": 12698492928.0, "7095": 12698492928.0, "7100": 12698492928.0, "7105": 12698492928.0, "7110": 12698492928.0, "7115": 12698492928.0, "7120": 12698492928.0, "7125": 12698492928.0, "7130": 12698492928.0, "7135": 12698492928.0, "7140": 12698492928.0, "7145": 12698492928.0, "7150": 12698492928.0, "7155": 12698492928.0, "7160": 12698492928.0, "7165": 12698492928.0, "7170": 12698492928.0, "7175": 12698492928.0, "7180": 12698492928.0, "7185": 12698492928.0, "7190": 12698492928.0, "7195": 12698492928.0, "7200": 12698492928.0, "7205": 12698492928.0, "7210": 12698492928.0, "7215": 12698492928.0, "7220": 12698492928.0, "7225": 12698492928.0, "7230": 12698492928.0, "7235": 12698492928.0, "7240": 12698492928.0, "7245": 12698492928.0, "7250": 12698492928.0, "7255": 12698492928.0, "7260": 12698492928.0, "7265": 12698492928.0, "7270": 12698492928.0, "7275": 12698492928.0, "7280": 12698492928.0, "7285": 12698492928.0, "7290": 12698492928.0, "7295": 12698492928.0, "7300": 12698492928.0, "7305": 12698492928.0, "7310": 12698492928.0, "7315": 12698492928.0, "7320": 12698492928.0, "7325": 12698492928.0, "7330": 12698492928.0, "7335": 12698492928.0, "7340": 12698492928.0, "7345": 12698492928.0, "7350": 12698492928.0, "7355": 12698492928.0, "7360": 12698492928.0, "7365": 12698492928.0, "7370": 12698492928.0, "7375": 12698492928.0, "7380": 12698492928.0, "7385": 12698492928.0, "7390": 12698492928.0, "7395": 12698492928.0, "7400": 12698492928.0, "7405": 12698492928.0, "7410": 12698492928.0, "7415": 12698492928.0, "7420": 12698492928.0, "7425": 12698492928.0, "7430": 12698492928.0, "7435": 12698492928.0, "7440": 12698492928.0, "7445": 12698492928.0, "7450": 12698492928.0, "7455": 12698492928.0, "7460": 12698492928.0, "7465": 12698492928.0, "7470": 12698492928.0, "7475": 12698492928.0, "7480": 12698492928.0, "7485": 12698492928.0, "7490": 12698492928.0, "7495": 12698492928.0, "7500": 12698492928.0, "7505": 12698492928.0, "7510": 12698492928.0, "7515": 12698492928.0, "7520": 12698492928.0, "7525": 12698492928.0, "7530": 12698492928.0, "7535": 12698492928.0, "7540": 12698492928.0, "7545": 12698492928.0, "7550": 12698492928.0, "7555": 12698492928.0, "7560": 12698492928.0, "7565": 12698492928.0, "7570": 12698492928.0, "7575": 12698492928.0, "7580": 12698492928.0, "7585": 12698492928.0, "7590": 12698492928.0, "7595": 12698492928.0, "7600": 12698492928.0, "7605": 12698492928.0, "7610": 12698492928.0, "7615": 12698492928.0, "7620": 12698492928.0, "7625": 12698492928.0, "7630": 12698492928.0, "7635": 12698492928.0, "7640": 12698492928.0, "7645": 12698492928.0, "7650": 12698492928.0, "7655": 12698492928.0, "7660": 12698492928.0, "7665": 12698492928.0, "7670": 12698492928.0, "7675": 12698492928.0, "7680": 12698492928.0, "7685": 12698492928.0, "7690": 12698492928.0, "7695": 12698492928.0, "7700": 12698492928.0, "7705": 12698492928.0, "7710": 12698492928.0, "7715": 12698492928.0, "7720": 12698492928.0, "7725": 12698492928.0, "7730": 12698492928.0, "7735": 12698492928.0, "7740": 12698492928.0, "7745": 12698492928.0, "7750": 12698492928.0, "7755": 12698492928.0, "7760": 12698492928.0, "7765": 12698492928.0, "7770": 12698492928.0, "7775": 12698492928.0, "7780": 12698492928.0, "7785": 12698492928.0, "7790": 12698492928.0, "7795": 12698492928.0, "7800": 12698492928.0, "7805": 12698492928.0, "7810": 12698492928.0, "7815": 12698492928.0, "7820": 12698492928.0, "7825": 12698492928.0, "7830": 12698492928.0, "7835": 12698492928.0, "7840": 12698492928.0, "7845": 12698492928.0, "7850": 12698492928.0, "7855": 12698492928.0, "7860": 12698492928.0, "7865": 12698492928.0, "7870": 12698492928.0, "7875": 12698492928.0, "7880": 12698492928.0, "7885": 12698492928.0, "7890": 12698492928.0, "7895": 12698492928.0, "7900": 12698492928.0, "7905": 12698492928.0, "7910": 12698492928.0, "7915": 12698492928.0, "7920": 12698492928.0, "7925": 12698492928.0, "7930": 12698492928.0, "7935": 12698492928.0, "7940": 12698492928.0, "7945": 12698492928.0, "7950": 12698492928.0, "7955": 12698492928.0, "7960": 12698492928.0, "7965": 12698492928.0, "7970": 12698492928.0, "7975": 12698492928.0, "7980": 12698492928.0, "7985": 12698492928.0, "7990": 12698492928.0, "7995": 12698492928.0, "8000": 12698492928.0, "8005": 12698492928.0, "8010": 12698492928.0, "8015": 12698492928.0, "8020": 12698492928.0, "8025": 12698492928.0, "8030": 12698492928.0, "8035": 12698492928.0, "8040": 12698492928.0, "8045": 12698492928.0, "8050": 12698492928.0, "8055": 12698492928.0, "8060": 12698492928.0, "8065": 12698492928.0, "8070": 12698492928.0, "8075": 12698492928.0, "8080": 12698492928.0, "8085": 12698492928.0, "8090": 12698492928.0, "8095": 12698492928.0, "8100": 12698492928.0, "8105": 12698492928.0, "8110": 12698492928.0, "8115": 12698492928.0, "8120": 12698492928.0, "8125": 12698492928.0, "8130": 12698492928.0, "8135": 12698492928.0, "8140": 12698492928.0, "8145": 12698492928.0, "8150": 12698492928.0, "8155": 12698492928.0, "8160": 12698492928.0, "8165": 12698492928.0, "8170": 12698492928.0, "8175": 12698492928.0, "8180": 12698492928.0, "8185": 12698492928.0, "8190": 12698492928.0, "8195": 12698492928.0, "8200": 12698492928.0, "8205": 12698492928.0, "8210": 12698492928.0, "8215": 12698492928.0, "8220": 12698492928.0, "8225": 12698492928.0, "8230": 12698492928.0, "8235": 12698492928.0, "8240": 12698492928.0, "8245": 12698492928.0, "8250": 12698492928.0, "8255": 12698492928.0, "8260": 12698492928.0, "8265": 12698492928.0, "8270": 12698492928.0, "8275": 12698492928.0, "8280": 12698492928.0, "8285": 12698492928.0, "8290": 12698492928.0, "8295": 12698492928.0, "8300": 12698492928.0, "8305": 12698492928.0, "8310": 12698492928.0, "8315": 12698492928.0, "8320": 12698492928.0, "8325": 12698492928.0, "8330": 12698492928.0, "8335": 12698492928.0, "8340": 12698492928.0, "8345": 12698492928.0, "8350": 12698492928.0, "8355": 12698492928.0, "8360": 12698492928.0, "8365": 12698492928.0, "8370": 12698492928.0, "8375": 12698492928.0, "8380": 12698492928.0, "8385": 12698492928.0, "8390": 12698492928.0, "8395": 12698492928.0, "8400": 12698492928.0, "8405": 12698492928.0, "8410": 12698492928.0, "8415": 12698492928.0, "8420": 12698492928.0, "8425": 12698492928.0, "8430": 12698492928.0, "8435": 12698492928.0, "8440": 12698492928.0, "8445": 12698492928.0, "8450": 12698492928.0, "8455": 12698492928.0, "8460": 12698492928.0, "8465": 12698492928.0, "8470": 12698492928.0, "8475": 12698492928.0, "8480": 12698492928.0, "8485": 12698492928.0, "8490": 12698492928.0, "8495": 12698492928.0, "8500": 12698492928.0, "8505": 12698492928.0, "8510": 12698492928.0, "8515": 12698492928.0, "8520": 12698492928.0, "8525": 12698492928.0, "8530": 12698492928.0, "8535": 12698492928.0, "8540": 12698492928.0, "8545": 12698492928.0, "8550": 12698492928.0, "8555": 12698492928.0, "8560": 12698492928.0, "8565": 12698492928.0, "8570": 12698492928.0, "8575": 12698492928.0, "8580": 12698492928.0, "8585": 12698492928.0, "8590": 12698492928.0, "8595": 12698492928.0, "8600": 12698492928.0, "8605": 12698492928.0, "8610": 12698492928.0, "8615": 12698492928.0, "8620": 12698492928.0, "8625": 12698492928.0, "8630": 12698492928.0, "8635": 12698492928.0, "8640": 12698492928.0, "8645": 12698492928.0, "8650": 12698492928.0, "8655": 12698492928.0, "8660": 12698492928.0, "8665": 12698492928.0, "8670": 12698492928.0, "8675": 12698492928.0, "8680": 12698492928.0, "8685": 12698492928.0, "8690": 12698492928.0, "8695": 12698492928.0, "8700": 12698492928.0, "8705": 12698492928.0, "8710": 12698492928.0, "8715": 12698492928.0, "8720": 12698492928.0, "8725": 12698492928.0, "8730": 12698492928.0, "8735": 12698492928.0, "8740": 12698492928.0, "8745": 12698492928.0, "8750": 12698492928.0, "8755": 12698492928.0, "8760": 12698492928.0, "8765": 12698492928.0, "8770": 12698492928.0, "8775": 12698492928.0, "8780": 12698492928.0, "8785": 12698492928.0, "8790": 12698492928.0, "8795": 12698492928.0, "8800": 12698492928.0, "8805": 12698492928.0, "8810": 12698492928.0, "8815": 12698492928.0, "8820": 12698492928.0, "8825": 12698492928.0, "8830": 12698492928.0, "8835": 12698492928.0, "8840": 12698492928.0, "8845": 12698492928.0, "8850": 12698492928.0, "8855": 12698492928.0, "8860": 12698492928.0, "8865": 12698492928.0, "8870": 12698492928.0, "8875": 12698492928.0, "8880": 12698492928.0, "8885": 12698492928.0, "8890": 12698492928.0, "8895": 12698492928.0, "8900": 12698492928.0, "8905": 12698492928.0, "8910": 12698492928.0, "8915": 12698492928.0, "8920": 12698492928.0, "8925": 12698492928.0, "8930": 12698492928.0, "8935": 12698492928.0, "8940": 12698492928.0, "8945": 12698492928.0, "8950": 12698492928.0, "8955": 12698492928.0, "8960": 12698492928.0, "8965": 12698492928.0, "8970": 12698492928.0, "8975": 12698492928.0, "8980": 12698492928.0, "8985": 12698492928.0, "8990": 12698492928.0, "8995": 12698492928.0, "9000": 12698492928.0, "9005": 12698492928.0, "9010": 12698492928.0, "9015": 12698492928.0, "9020": 12698492928.0, "9025": 12698492928.0, "9030": 12698492928.0, "9035": 12698492928.0, "9040": 12698492928.0, "9045": 12698492928.0, "9050": 12698492928.0, "9055": 12698492928.0, "9060": 12698492928.0, "9065": 12698492928.0, "9070": 12698492928.0, "9075": 12698492928.0, "9080": 12698492928.0, "9085": 12698492928.0, "9090": 12698492928.0, "9095": 12698492928.0, "9100": 12698492928.0, "9105": 12698492928.0, "9110": 12698492928.0, "9115": 12698492928.0, "9120": 12698492928.0, "9125": 12698492928.0, "9130": 12698492928.0, "9135": 12698492928.0, "9140": 12698492928.0, "9145": 12698492928.0, "9150": 12698492928.0, "9155": 12698492928.0, "9160": 12698492928.0, "9165": 12698492928.0, "9170": 12698492928.0, "9175": 12698492928.0, "9180": 12698492928.0, "9185": 12698492928.0, "9190": 12698492928.0, "9195": 12698492928.0, "9200": 12698492928.0, "9205": 12698492928.0, "9210": 12698492928.0, "9215": 12698492928.0, "9220": 12698492928.0, "9225": 12698492928.0, "9230": 12698492928.0, "9235": 12698492928.0, "9240": 12698492928.0, "9245": 12698492928.0, "9250": 12698492928.0, "9255": 12698492928.0, "9260": 12698492928.0, "9265": 12698492928.0, "9270": 12698492928.0, "9275": 12698492928.0, "9280": 12698492928.0, "9285": 12698492928.0, "9290": 12698492928.0, "9295": 12698492928.0, "9300": 12698492928.0, "9305": 12698492928.0, "9310": 12698492928.0, "9315": 12698492928.0, "9320": 12698492928.0, "9325": 12698492928.0, "9330": 12698492928.0, "9335": 12698492928.0, "9340": 12698492928.0, "9345": 12698492928.0, "9350": 12698492928.0, "9355": 12698492928.0, "9360": 12698492928.0, "9365": 12698492928.0, "9370": 12698492928.0, "9375": 12698492928.0, "9380": 12698492928.0, "9385": 12698492928.0, "9390": 12698492928.0, "9395": 12698492928.0, "9400": 12698492928.0, "9405": 12698492928.0, "9410": 12698492928.0, "9415": 12698492928.0, "9420": 12698492928.0, "9425": 12698492928.0, "9430": 12698492928.0, "9435": 12698492928.0, "9440": 12698492928.0, "9445": 12698492928.0, "9450": 12698492928.0, "9455": 12698492928.0, "9460": 12698492928.0, "9465": 12698492928.0, "9470": 12698492928.0, "9475": 12698492928.0, "9480": 12698492928.0, "9485": 12698492928.0, "9490": 12698492928.0, "9495": 12698492928.0, "9500": 12698492928.0, "9505": 12698492928.0, "9510": 12698492928.0, "9515": 12698492928.0, "9520": 12698492928.0, "9525": 12698492928.0, "9530": 12698492928.0, "9535": 12698492928.0, "9540": 12698492928.0, "9545": 12698492928.0, "9550": 12698492928.0, "9555": 12698492928.0, "9560": 12698492928.0, "9565": 12698492928.0, "9570": 12698492928.0, "9575": 12698492928.0, "9580": 12698492928.0, "9585": 12698492928.0, "9590": 12698492928.0, "9595": 12698492928.0, "9600": 12698492928.0, "9605": 12698492928.0, "9610": 12698492928.0, "9615": 12698492928.0, "9620": 12698492928.0, "9625": 12698492928.0, "9630": 12698492928.0, "9635": 12698492928.0, "9640": 12698492928.0, "9645": 12698492928.0, "9650": 12698492928.0, "9655": 12698492928.0, "9660": 12698492928.0, "9665": 12698492928.0, "9670": 12698492928.0, "9675": 12698492928.0, "9680": 12698492928.0, "9685": 12698492928.0, "9690": 12698492928.0, "9695": 12698492928.0, "9700": 12698492928.0, "9705": 12698492928.0, "9710": 12698492928.0, "9715": 12698492928.0, "9720": 12698492928.0, "9725": 12698492928.0, "9730": 12698492928.0, "9735": 12698492928.0, "9740": 12698492928.0, "9745": 12698492928.0, "9750": 12698492928.0, "9755": 12698492928.0, "9760": 12698492928.0, "9765": 12698492928.0, "9770": 12698492928.0, "9775": 12698492928.0, "9780": 12698492928.0, "9785": 12698492928.0, "9790": 12698492928.0, "9795": 12698492928.0, "9800": 12698492928.0, "9805": 12698492928.0, "9810": 12698492928.0, "9815": 12698492928.0, "9820": 12698492928.0, "9825": 12698492928.0, "9830": 12698492928.0, "9835": 12698492928.0, "9840": 12698492928.0, "9845": 12698492928.0, "9850": 12698492928.0, "9855": 12698492928.0, "9860": 12698492928.0, "9865": 12698492928.0, "9870": 12698492928.0, "9875": 12698492928.0, "9880": 12698492928.0, "9885": 12698492928.0, "9890": 12698492928.0, "9895": 12698492928.0, "9900": 12698492928.0, "9905": 12698492928.0, "9910": 12698492928.0, "9915": 12698492928.0, "9920": 12698492928.0, "9925": 12698492928.0, "9930": 12698492928.0, "9935": 12698492928.0, "9940": 12698492928.0, "9945": 12698492928.0, "9950": 12698492928.0, "9955": 12698492928.0, "9960": 12698492928.0, "9965": 12698492928.0, "9970": 12698492928.0, "9975": 12698492928.0, "9980": 12698492928.0, "9985": 12698492928.0, "9990": 12698492928.0, "9995": 12698492928.0, "10000": 12698492928.0, "10005": 12698492928.0, "10010": 12698492928.0, "10015": 12698492928.0, "10020": 12698492928.0, "10025": 12698492928.0, "10030": 12698492928.0, "10035": 12698492928.0, "10040": 12698492928.0, "10045": 12698492928.0, "10050": 12698492928.0, "10055": 12698492928.0, "10060": 12698492928.0, "10065": 12698492928.0, "10070": 12698492928.0, "10075": 12698492928.0, "10080": 12698492928.0, "10085": 12698492928.0, "10090": 12698492928.0, "10095": 12698492928.0, "10100": 12698492928.0, "10105": 12698492928.0, "10110": 12698492928.0, "10115": 12698492928.0, "10120": 12698492928.0, "10125": 12698492928.0, "10130": 12698492928.0, "10135": 12698492928.0, "10140": 12698492928.0, "10145": 12698492928.0, "10150": 12698492928.0, "10155": 12698492928.0, "10160": 12698492928.0, "10165": 12698492928.0, "10170": 12698492928.0, "10175": 12698492928.0, "10180": 12698492928.0, "10185": 12698492928.0, "10190": 12698492928.0, "10195": 12698492928.0, "10200": 12698492928.0, "10205": 12698492928.0, "10210": 12698492928.0, "10215": 12698492928.0, "10220": 12698492928.0, "10225": 12698492928.0, "10230": 12698492928.0, "10235": 12698492928.0, "10240": 12698492928.0, "10245": 12698492928.0, "10250": 12698492928.0, "10255": 12698492928.0, "10260": 12698492928.0, "10265": 12698492928.0, "10270": 12698492928.0, "10275": 12698492928.0, "10280": 12698492928.0, "10285": 12698492928.0, "10290": 12698492928.0, "10295": 12698492928.0, "10300": 12698492928.0, "10305": 12698492928.0, "10310": 12698492928.0, "10315": 12698492928.0, "10320": 12698492928.0, "10325": 12698492928.0, "10330": 12698492928.0, "10335": 12698492928.0, "10340": 12698492928.0, "10345": 12698492928.0, "10350": 12698492928.0, "10355": 12698492928.0, "10360": 12698492928.0, "10365": 12698492928.0, "10370": 12698492928.0, "10375": 12698492928.0, "10380": 12698492928.0, "10385": 12698492928.0, "10390": 12698492928.0, "10395": 12698492928.0, "10400": 12698492928.0, "10405": 12698492928.0, "10410": 12698492928.0, "10415": 12698492928.0, "10420": 12698492928.0, "10425": 12698492928.0, "10430": 12698492928.0, "10435": 12698492928.0, "10440": 12698492928.0, "10445": 12698492928.0, "10450": 12698492928.0, "10455": 12698492928.0, "10460": 12698492928.0, "10465": 12698492928.0, "10470": 12698492928.0, "10475": 12698492928.0, "10480": 12698492928.0, "10485": 12698492928.0, "10490": 12698492928.0, "10495": 12698492928.0, "10500": 12698492928.0, "10505": 12698492928.0, "10510": 12698492928.0, "10515": 12698492928.0, "10520": 12698492928.0, "10525": 12698492928.0, "10530": 12698492928.0, "10535": 12698492928.0, "10540": 12698492928.0, "10545": 12698492928.0, "10550": 12698492928.0, "10555": 12698492928.0, "10560": 12698492928.0, "10565": 12698492928.0, "10570": 12698492928.0, "10575": 12698492928.0, "10580": 12698492928.0, "10585": 12698492928.0, "10590": 12698492928.0, "10595": 12698492928.0, "10600": 12698492928.0, "10605": 12698492928.0, "10610": 12698492928.0, "10615": 12698492928.0, "10620": 12698492928.0, "10625": 12698492928.0, "10630": 12698492928.0, "10635": 12698492928.0, "10640": 12698492928.0, "10645": 12698492928.0, "10650": 12698492928.0, "10655": 12698492928.0, "10660": 12698492928.0, "10665": 12698492928.0, "10670": 12698492928.0, "10675": 12698492928.0, "10680": 12698492928.0, "10685": 12698492928.0, "10690": 12698492928.0, "10695": 12698492928.0, "10700": 12698492928.0, "10705": 12698492928.0, "10710": 12698492928.0, "10715": 12698492928.0, "10720": 12698492928.0, "10725": 12698492928.0, "10730": 12698492928.0, "10735": 12698492928.0, "10740": 12698492928.0, "10745": 12698492928.0, "10750": 12698492928.0, "10755": 12698492928.0, "10760": 12698492928.0, "10765": 12698492928.0, "10770": 12698492928.0, "10775": 12698492928.0, "10780": 12698492928.0, "10785": 12698492928.0, "10790": 12698492928.0, "10795": 12698492928.0, "10800": 12698492928.0, "10805": 12698492928.0, "10810": 12698492928.0, "10815": 12698492928.0, "10820": 12698492928.0, "10825": 12698492928.0, "10830": 12698492928.0, "10835": 12698492928.0, "10840": 12698492928.0, "10845": 12698492928.0, "10850": 12698492928.0, "10855": 12698492928.0, "10860": 12698492928.0, "10865": 12698492928.0, "10870": 12698492928.0, "10875": 12698492928.0, "10880": 12698492928.0, "10885": 12698492928.0, "10890": 12698492928.0, "10895": 12698492928.0, "10900": 12698492928.0, "10905": 12698492928.0, "10910": 12698492928.0, "10915": 12698492928.0, "10920": 12698492928.0, "10925": 12698492928.0, "10930": 12698492928.0, "10935": 12698492928.0, "10940": 12698492928.0, "10945": 12698492928.0, "10950": 12698492928.0, "10955": 12698492928.0, "10960": 12698492928.0, "10965": 12698492928.0, "10970": 12698492928.0, "10975": 12698492928.0, "10980": 12698492928.0, "10985": 12698492928.0, "10990": 12698492928.0, "10995": 12698492928.0, "11000": 12698492928.0, "11005": 12698492928.0, "11010": 12698492928.0, "11015": 12698492928.0, "11020": 12698492928.0, "11025": 12698492928.0, "11030": 12698492928.0, "11035": 12698492928.0, "11040": 12698492928.0, "11045": 12698492928.0, "11050": 12698492928.0, "11055": 12698492928.0, "11060": 12698492928.0, "11065": 12698492928.0, "11070": 12698492928.0, "11075": 12698492928.0, "11080": 12698492928.0, "11085": 12698492928.0, "11090": 12698492928.0, "11095": 12698492928.0, "11100": 12698492928.0, "11105": 12698492928.0, "11110": 12698492928.0, "11115": 12698492928.0, "11120": 12698492928.0, "11125": 12698492928.0, "11130": 12698492928.0, "11135": 12698492928.0, "11140": 12698492928.0, "11145": 12698492928.0, "11150": 12698492928.0, "11155": 12698492928.0, "11160": 12698492928.0, "11165": 12698492928.0, "11170": 12698492928.0, "11175": 12698492928.0, "11180": 12698492928.0, "11185": 12698492928.0, "11190": 12698492928.0, "11195": 12698492928.0, "11200": 12698492928.0, "11205": 12698492928.0, "11210": 12698492928.0, "11215": 12698492928.0, "11220": 12698492928.0, "11225": 12698492928.0, "11230": 12698492928.0, "11235": 12698492928.0, "11240": 12698492928.0, "11245": 12698492928.0, "11250": 12698492928.0, "11255": 12698492928.0, "11260": 12698492928.0, "11265": 12698492928.0, "11270": 12698492928.0, "11275": 12698492928.0, "11280": 12698492928.0, "11285": 12698492928.0, "11290": 12698492928.0, "11295": 12698492928.0, "11300": 12698492928.0, "11305": 12698492928.0, "11310": 12698492928.0, "11315": 12698492928.0, "11320": 12698492928.0, "11325": 12698492928.0, "11330": 12698492928.0, "11335": 12698492928.0, "11340": 12698492928.0, "11345": 12698492928.0, "11350": 12698492928.0, "11355": 12698492928.0, "11360": 12698492928.0, "11365": 12698492928.0, "11370": 12698492928.0, "11375": 12698492928.0, "11380": 12698492928.0, "11385": 12698492928.0, "11390": 12698492928.0, "11395": 12698492928.0, "11400": 12698492928.0, "11405": 12698492928.0, "11410": 12698492928.0, "11415": 12698492928.0, "11420": 12698492928.0, "11425": 12698492928.0, "11430": 12698492928.0, "11435": 12698492928.0, "11440": 12698492928.0, "11445": 12698492928.0, "11450": 12698492928.0, "11455": 12698492928.0, "11460": 12698492928.0, "11465": 12698492928.0, "11470": 12698492928.0, "11475": 12698492928.0, "11480": 12698492928.0, "11485": 12698492928.0, "11490": 12698492928.0, "11495": 12698492928.0, "11500": 12698492928.0, "11505": 12698492928.0, "11510": 12698492928.0, "11515": 12698492928.0, "11520": 12698492928.0, "11525": 12698492928.0, "11530": 12698492928.0, "11535": 12698492928.0, "11540": 12698492928.0, "11545": 12698492928.0, "11550": 12698492928.0, "11555": 12698492928.0, "11560": 12698492928.0, "11565": 12698492928.0, "11570": 12698492928.0, "11575": 12698492928.0, "11580": 12698492928.0, "11585": 12698492928.0, "11590": 12698492928.0, "11595": 12698492928.0, "11600": 12698492928.0, "11605": 12698492928.0, "11610": 12698492928.0, "11615": 12698492928.0, "11620": 12698492928.0, "11625": 12698492928.0, "11630": 12698492928.0, "11635": 12698492928.0, "11640": 12698492928.0, "11645": 12698492928.0, "11650": 12698492928.0, "11655": 12698492928.0, "11660": 12698492928.0, "11665": 12698492928.0, "11670": 12698492928.0, "11675": 12698492928.0, "11680": 12698492928.0, "11685": 12698492928.0, "11690": 12698492928.0, "11695": 12698492928.0, "11700": 12698492928.0, "11705": 12698492928.0, "11710": 12698492928.0, "11715": 12698492928.0, "11720": 12698492928.0, "11725": 12698492928.0, "11730": 12698492928.0, "11735": 12698492928.0, "11740": 12698492928.0, "11745": 12698492928.0, "11750": 12698492928.0, "11755": 12698492928.0, "11760": 12698492928.0, "11765": 12698492928.0, "11770": 12698492928.0, "11775": 12698492928.0, "11780": 12698492928.0, "11785": 12698492928.0, "11790": 12698492928.0, "11795": 12698492928.0, "11800": 12698492928.0, "11805": 12698492928.0, "11810": 12698492928.0, "11815": 12698492928.0, "11820": 12698492928.0, "11825": 12698492928.0, "11830": 12698492928.0, "11835": 12698492928.0, "11840": 12698492928.0, "11845": 12698492928.0, "11850": 12698492928.0, "11855": 12698492928.0, "11860": 12698492928.0, "11865": 12698492928.0, "11870": 12698492928.0, "11875": 12698492928.0, "11880": 12698492928.0, "11885": 12698492928.0, "11890": 12698492928.0, "11895": 12698492928.0, "11900": 12698492928.0, "11905": 12698492928.0, "11910": 12698492928.0, "11915": 12698492928.0, "11920": 12698492928.0, "11925": 12698492928.0, "11930": 12698492928.0, "11935": 12698492928.0, "11940": 12698492928.0, "11945": 12698492928.0, "11950": 12698492928.0, "11955": 12698492928.0, "11960": 12698492928.0, "11965": 12698492928.0, "11970": 12698492928.0, "11975": 12698492928.0, "11980": 12698492928.0, "11985": 12698492928.0, "11990": 12698492928.0, "11995": 12698492928.0, "12000": 12698492928.0, "12005": 12698492928.0, "12010": 12698492928.0, "12015": 12698492928.0, "12020": 12698492928.0, "12025": 12698492928.0, "12030": 12698492928.0, "12035": 12698492928.0, "12040": 12698492928.0, "12045": 12698492928.0, "12050": 12698492928.0, "12055": 12698492928.0, "12060": 12698492928.0, "12065": 12698492928.0, "12070": 12698492928.0, "12075": 12698492928.0, "12080": 12698492928.0, "12085": 12698492928.0, "12090": 12698492928.0, "12095": 12698492928.0, "12100": 12698492928.0, "12105": 12698492928.0, "12110": 12698492928.0, "12115": 12698492928.0, "12120": 12698492928.0, "12125": 12698492928.0, "12130": 12698492928.0, "12135": 12698492928.0, "12140": 12698492928.0, "12145": 12698492928.0, "12150": 12698492928.0, "12155": 12698492928.0, "12160": 12698492928.0, "12165": 12698492928.0, "12170": 12698492928.0, "12175": 12698492928.0, "12180": 12698492928.0, "12185": 12698492928.0, "12190": 12698492928.0, "12195": 12698492928.0, "12200": 12698492928.0, "12205": 12698492928.0, "12210": 12698492928.0, "12215": 12698492928.0, "12220": 12698492928.0, "12225": 12698492928.0, "12230": 12698492928.0, "12235": 12698492928.0, "12240": 12698492928.0, "12245": 12698492928.0, "12250": 12698492928.0, "12255": 12698492928.0, "12260": 12698492928.0, "12265": 12698492928.0, "12270": 12698492928.0, "12275": 12698492928.0, "12280": 12698492928.0, "12285": 12698492928.0, "12290": 12698492928.0, "12295": 12698492928.0, "12300": 12698492928.0, "12305": 12698492928.0, "12310": 12698492928.0, "12315": 12698492928.0, "12320": 12698492928.0, "12325": 12698492928.0, "12330": 12698492928.0, "12335": 12698492928.0, "12340": 12698492928.0, "12345": 12698492928.0, "12350": 12698492928.0, "12355": 12698492928.0, "12360": 12698492928.0, "12365": 12698492928.0, "12370": 12698492928.0, "12375": 12698492928.0, "12380": 12698492928.0, "12385": 12698492928.0, "12390": 12698492928.0, "12395": 12698492928.0, "12400": 12698492928.0, "12405": 12698492928.0, "12410": 12698492928.0, "12415": 12698492928.0, "12420": 12698492928.0, "12425": 12698492928.0, "12430": 12698492928.0, "12435": 12698492928.0, "12440": 12698492928.0, "12445": 12698492928.0, "12450": 12698492928.0, "12455": 12698492928.0, "12460": 12698492928.0, "12465": 12698492928.0, "12470": 12698492928.0, "12475": 12698492928.0, "12480": 12698492928.0, "12485": 12698492928.0, "12490": 12698492928.0, "12495": 12698492928.0, "12500": 12698492928.0, "12505": 12698492928.0, "12510": 12698492928.0, "12515": 12698492928.0, "12520": 12698492928.0, "12525": 12698492928.0, "12530": 12698492928.0, "12535": 12698492928.0, "12540": 12698492928.0, "12545": 12698492928.0, "12550": 12698492928.0, "12555": 12698492928.0, "12560": 12698492928.0, "12565": 12698492928.0, "12570": 12698492928.0, "12575": 12698492928.0, "12580": 12698492928.0, "12585": 12698492928.0, "12590": 12698492928.0, "12595": 12698492928.0, "12600": 12698492928.0, "12605": 12698492928.0, "12610": 12698492928.0, "12615": 12698492928.0, "12620": 12698492928.0, "12625": 12698492928.0, "12630": 12698492928.0, "12635": 12698492928.0, "12640": 12698492928.0, "12645": 12698492928.0, "12650": 12698492928.0, "12655": 12698492928.0, "12660": 12698492928.0, "12665": 12698492928.0, "12670": 12698492928.0, "12675": 12698492928.0, "12680": 12698492928.0, "12685": 12698492928.0, "12690": 12698492928.0, "12695": 12698492928.0, "12700": 12698492928.0, "12705": 12698492928.0, "12710": 12698492928.0, "12715": 12698492928.0, "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}, "mem-max-allocated-bytes": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": 27893811200.0, "5": 28391901184.0, "10": 28391901184.0, "15": 28391901184.0, "20": 28391901184.0, "25": 28391901184.0, "30": 28391901184.0, "35": 28391901184.0, "40": 28391901184.0, "45": 28391901184.0, "50": 28391901184.0, "55": 28391901184.0, "60": 28391901184.0, "65": 28391901184.0, "70": 28391901184.0, "75": 28391901184.0, "80": 28391901184.0, "85": 28391901184.0, "90": 28391901184.0, "95": 28391901184.0, "100": 28391901184.0, "105": 28391901184.0, "110": 28391901184.0, "115": 28391901184.0, "120": 28391901184.0, "125": 28391901184.0, "130": 28391901184.0, "135": 28391901184.0, "140": 28391901184.0, "145": 28391901184.0, "150": 28391901184.0, "155": 28391901184.0, "160": 28391901184.0, "165": 28391901184.0, "170": 28391901184.0, "175": 28391901184.0, "180": 28391901184.0, "185": 28391901184.0, "190": 28391901184.0, "195": 28391901184.0, "200": 28391901184.0, "205": 28391901184.0, "210": 28391901184.0, "215": 28391901184.0, "220": 28391901184.0, "225": 28391901184.0, "230": 28391901184.0, "235": 28391901184.0, "240": 28391901184.0, "245": 28391901184.0, "250": 28391901184.0, "255": 28391901184.0, "260": 28391901184.0, "265": 28391901184.0, "270": 28391901184.0, "275": 28391901184.0, "280": 28391901184.0, "285": 28391901184.0, "290": 28391901184.0, "295": 28391901184.0, "300": 28391901184.0, "305": 28391901184.0, "310": 28391901184.0, "315": 28391901184.0, "320": 28391901184.0, "325": 28391901184.0, "330": 28391901184.0, "335": 28391901184.0, "340": 28391901184.0, "345": 28391901184.0, "350": 28391901184.0, "355": 28391901184.0, "360": 28391901184.0, "365": 28391901184.0, "370": 28391901184.0, "375": 28391901184.0, "380": 28391901184.0, "385": 28391901184.0, "390": 28391901184.0, "395": 28391901184.0, "400": 28391901184.0, "405": 28391901184.0, "410": 28391901184.0, "415": 28391901184.0, "420": 28391901184.0, "425": 28391901184.0, "430": 28391901184.0, "435": 28391901184.0, "440": 28391901184.0, "445": 28391901184.0, "450": 28391901184.0, "455": 28391901184.0, "460": 28391901184.0, "465": 28391901184.0, "470": 28391901184.0, "475": 28391901184.0, "480": 28391901184.0, "485": 28391901184.0, "490": 28391901184.0, "495": 28391901184.0, "500": 28391901184.0, "505": 28391901184.0, "510": 28391901184.0, "515": 28391901184.0, "520": 28391901184.0, "525": 28391901184.0, "530": 28391901184.0, "535": 28391901184.0, "540": 28391901184.0, "545": 28391901184.0, "550": 28391901184.0, "555": 28391901184.0, "560": 28391901184.0, "565": 28391901184.0, "570": 28391901184.0, "575": 28391901184.0, "580": 28391901184.0, "585": 28391901184.0, "590": 28391901184.0, "595": 28391901184.0, "600": 28391901184.0, "605": 28391901184.0, "610": 28391901184.0, "615": 28391901184.0, "620": 28391901184.0, "625": 28391901184.0, "630": 28391901184.0, "635": 28391901184.0, "640": 28391901184.0, "645": 28391901184.0, "650": 28391901184.0, "655": 28391901184.0, "660": 28391901184.0, "665": 28391901184.0, "670": 28391901184.0, "675": 28391901184.0, "680": 28391901184.0, "685": 28391901184.0, "690": 28391901184.0, "695": 28391901184.0, "700": 28391901184.0, "705": 28391901184.0, "710": 28391901184.0, "715": 28391901184.0, "720": 28391901184.0, "725": 28391901184.0, "730": 28391901184.0, "735": 28391901184.0, "740": 28391901184.0, "745": 28391901184.0, "750": 28391901184.0, "755": 28391901184.0, "760": 28391901184.0, "765": 28391901184.0, "770": 28391901184.0, "775": 28391901184.0, "780": 28391901184.0, "785": 28391901184.0, "790": 28391901184.0, "795": 28391901184.0, "800": 28391901184.0, "805": 28391901184.0, "810": 28391901184.0, "815": 28391901184.0, "820": 28391901184.0, "825": 28391901184.0, "830": 28391901184.0, "835": 28391901184.0, "840": 28391901184.0, "845": 28391901184.0, "850": 28391901184.0, "855": 28391901184.0, "860": 28391901184.0, "865": 28391901184.0, "870": 28391901184.0, "875": 28391901184.0, "880": 28391901184.0, "885": 28391901184.0, "890": 28391901184.0, "895": 28391901184.0, "900": 28391901184.0, "905": 28391901184.0, "910": 28391901184.0, "915": 28391901184.0, "920": 28391901184.0, "925": 28391901184.0, "930": 28391901184.0, "935": 28391901184.0, "940": 28391901184.0, "945": 28391901184.0, "950": 28391901184.0, "955": 28391901184.0, "960": 28391901184.0, "965": 28391901184.0, "970": 28391901184.0, "975": 28391901184.0, "980": 28391901184.0, "985": 28391901184.0, "990": 28391901184.0, "995": 28391901184.0, "1000": 28391901184.0, "1005": 28391901184.0, "1010": 28391901184.0, "1015": 28391901184.0, "1020": 28391901184.0, "1025": 28391901184.0, "1030": 28391901184.0, "1035": 28391901184.0, "1040": 28391901184.0, "1045": 28391901184.0, "1050": 28391901184.0, "1055": 28391901184.0, "1060": 28391901184.0, "1065": 28391901184.0, "1070": 28391901184.0, "1075": 28391901184.0, "1080": 28391901184.0, "1085": 28391901184.0, "1090": 28391901184.0, "1095": 28391901184.0, "1100": 28391901184.0, "1105": 28391901184.0, "1110": 28391901184.0, "1115": 28391901184.0, "1120": 28391901184.0, "1125": 28391901184.0, "1130": 28391901184.0, "1135": 28391901184.0, "1140": 28391901184.0, "1145": 28391901184.0, "1150": 28391901184.0, "1155": 28391901184.0, "1160": 28391901184.0, "1165": 28391901184.0, "1170": 28391901184.0, "1175": 28391901184.0, "1180": 28391901184.0, "1185": 28391901184.0, "1190": 28391901184.0, "1195": 28391901184.0, "1200": 28391901184.0, "1205": 28391901184.0, "1210": 28391901184.0, "1215": 28391901184.0, "1220": 28391901184.0, "1225": 28391901184.0, "1230": 28391901184.0, "1235": 28391901184.0, "1240": 28391901184.0, "1245": 28391901184.0, "1250": 28391901184.0, "1255": 28391901184.0, "1260": 28391901184.0, "1265": 28391901184.0, "1270": 28391901184.0, "1275": 28391901184.0, "1280": 28391901184.0, "1285": 28391901184.0, "1290": 28391901184.0, "1295": 28391901184.0, "1300": 28391901184.0, "1305": 28391901184.0, "1310": 28391901184.0, "1315": 28391901184.0, "1320": 28391901184.0, "1325": 28391901184.0, "1330": 28391901184.0, "1335": 28391901184.0, "1340": 28391901184.0, "1345": 28391901184.0, "1350": 28391901184.0, "1355": 28391901184.0, "1360": 28391901184.0, "1365": 28391901184.0, "1370": 28391901184.0, "1375": 28391901184.0, "1380": 28391901184.0, "1385": 28391901184.0, "1390": 28391901184.0, "1395": 28391901184.0, "1400": 28391901184.0, "1405": 28391901184.0, "1410": 28391901184.0, "1415": 28391901184.0, "1420": 28391901184.0, "1425": 28391901184.0, "1430": 28391901184.0, "1435": 28391901184.0, "1440": 28391901184.0, "1445": 28391901184.0, "1450": 28391901184.0, "1455": 28391901184.0, "1460": 28391901184.0, "1465": 28391901184.0, "1470": 28391901184.0, "1475": 28391901184.0, "1480": 28391901184.0, "1485": 28391901184.0, "1490": 28391901184.0, "1495": 28391901184.0, "1500": 28391901184.0, "1505": 28391901184.0, "1510": 28391901184.0, "1515": 28391901184.0, "1520": 28391901184.0, "1525": 28391901184.0, "1530": 28391901184.0, "1535": 28391901184.0, "1540": 28391901184.0, "1545": 28391901184.0, "1550": 28391901184.0, "1555": 28391901184.0, "1560": 28391901184.0, "1565": 28391901184.0, "1570": 28391901184.0, "1575": 28391901184.0, "1580": 28391901184.0, "1585": 28391901184.0, "1590": 28391901184.0, "1595": 28391901184.0, "1600": 28391901184.0, "1605": 28391901184.0, "1610": 28391901184.0, "1615": 28391901184.0, "1620": 28391901184.0, "1625": 28391901184.0, "1630": 28391901184.0, "1635": 28391901184.0, "1640": 28391901184.0, "1645": 28391901184.0, "1650": 28391901184.0, "1655": 28391901184.0, "1660": 28391901184.0, "1665": 28391901184.0, "1670": 28391901184.0, "1675": 28391901184.0, "1680": 28391901184.0, "1685": 28391901184.0, "1690": 28391901184.0, "1695": 28391901184.0, "1700": 28391901184.0, "1705": 28391901184.0, "1710": 28391901184.0, "1715": 28391901184.0, "1720": 28391901184.0, "1725": 28391901184.0, "1730": 28391901184.0, "1735": 28391901184.0, "1740": 28391901184.0, "1745": 28391901184.0, "1750": 28391901184.0, "1755": 28391901184.0, "1760": 28391901184.0, "1765": 28391901184.0, "1770": 28391901184.0, "1775": 28391901184.0, "1780": 28391901184.0, "1785": 28391901184.0, "1790": 28391901184.0, "1795": 28391901184.0, "1800": 28391901184.0, "1805": 28391901184.0, "1810": 28391901184.0, "1815": 28391901184.0, "1820": 28391901184.0, "1825": 28391901184.0, "1830": 28391901184.0, "1835": 28391901184.0, "1840": 28391901184.0, "1845": 28391901184.0, "1850": 28391901184.0, "1855": 28391901184.0, "1860": 28391901184.0, "1865": 28391901184.0, "1870": 28391901184.0, "1875": 28391901184.0, "1880": 28391901184.0, "1885": 28391901184.0, "1890": 28391901184.0, "1895": 28391901184.0, "1900": 28391901184.0, "1905": 28391901184.0, "1910": 28391901184.0, "1915": 28391901184.0, "1920": 28391901184.0, "1925": 28391901184.0, "1930": 28391901184.0, "1935": 28391901184.0, "1940": 28391901184.0, "1945": 28391901184.0, "1950": 28391901184.0, "1955": 28391901184.0, "1960": 28391901184.0, "1965": 28391901184.0, "1970": 28391901184.0, "1975": 28391901184.0, "1980": 28391901184.0, "1985": 28391901184.0, "1990": 28391901184.0, "1995": 28391901184.0, "2000": 28391901184.0, "2005": 28391901184.0, "2010": 28391901184.0, "2015": 28391901184.0, "2020": 28391901184.0, "2025": 28391901184.0, "2030": 28391901184.0, "2035": 28391901184.0, "2040": 28391901184.0, "2045": 28391901184.0, "2050": 28391901184.0, "2055": 28391901184.0, "2060": 28391901184.0, "2065": 28391901184.0, "2070": 28391901184.0, "2075": 28391901184.0, "2080": 28391901184.0, "2085": 28391901184.0, "2090": 28391901184.0, "2095": 28391901184.0, "2100": 28391901184.0, "2105": 28391901184.0, "2110": 28391901184.0, "2115": 28391901184.0, "2120": 28391901184.0, "2125": 28391901184.0, "2130": 28391901184.0, "2135": 28391901184.0, "2140": 28391901184.0, "2145": 28391901184.0, "2150": 28391901184.0, "2155": 28391901184.0, "2160": 28391901184.0, "2165": 28391901184.0, "2170": 28391901184.0, "2175": 28391901184.0, "2180": 28391901184.0, "2185": 28391901184.0, "2190": 28391901184.0, "2195": 28391901184.0, "2200": 28391901184.0, "2205": 28391901184.0, "2210": 28391901184.0, "2215": 28391901184.0, "2220": 28391901184.0, "2225": 28391901184.0, "2230": 28391901184.0, "2235": 28391901184.0, "2240": 28391901184.0, "2245": 28391901184.0, "2250": 28391901184.0, "2255": 28391901184.0, "2260": 28391901184.0, "2265": 28391901184.0, "2270": 28391901184.0, "2275": 28391901184.0, "2280": 28391901184.0, "2285": 28391901184.0, "2290": 28391901184.0, "2295": 28391901184.0, "2300": 28391901184.0, "2305": 28391901184.0, "2310": 28391901184.0, "2315": 28391901184.0, "2320": 28391901184.0, "2325": 28391901184.0, "2330": 28391901184.0, "2335": 28391901184.0, "2340": 28391901184.0, "2345": 28391901184.0, "2350": 28391901184.0, "2355": 28391901184.0, "2360": 28391901184.0, "2365": 28391901184.0, "2370": 28391901184.0, "2375": 28391901184.0, "2380": 28391901184.0, "2385": 28391901184.0, "2390": 28391901184.0, "2395": 28391901184.0, "2400": 28391901184.0, "2405": 28391901184.0, "2410": 28391901184.0, "2415": 28391901184.0, "2420": 28391901184.0, "2425": 28391901184.0, "2430": 28391901184.0, "2435": 28391901184.0, "2440": 28391901184.0, "2445": 28391901184.0, "2450": 28391901184.0, "2455": 28391901184.0, "2460": 28391901184.0, "2465": 28391901184.0, "2470": 28391901184.0, "2475": 28391901184.0, "2480": 28391901184.0, "2485": 28391901184.0, "2490": 28391901184.0, "2495": 28391901184.0, "2500": 28391901184.0, "2505": 28391901184.0, "2510": 28391901184.0, "2515": 28391901184.0, "2520": 28391901184.0, "2525": 28391901184.0, "2530": 28391901184.0, "2535": 28391901184.0, "2540": 28391901184.0, "2545": 28391901184.0, "2550": 28391901184.0, "2555": 28391901184.0, "2560": 28391901184.0, "2565": 28391901184.0, "2570": 28391901184.0, "2575": 28391901184.0, "2580": 28391901184.0, "2585": 28391901184.0, "2590": 28391901184.0, "2595": 28391901184.0, "2600": 28391901184.0, "2605": 28391901184.0, "2610": 28391901184.0, "2615": 28391901184.0, "2620": 28391901184.0, "2625": 28391901184.0, "2630": 28391901184.0, "2635": 28391901184.0, "2640": 28391901184.0, "2645": 28391901184.0, "2650": 28391901184.0, "2655": 28391901184.0, "2660": 28391901184.0, "2665": 28391901184.0, "2670": 28391901184.0, "2675": 28391901184.0, "2680": 28391901184.0, "2685": 28391901184.0, "2690": 28391901184.0, "2695": 28391901184.0, "2700": 28391901184.0, "2705": 28391901184.0, "2710": 28391901184.0, "2715": 28391901184.0, "2720": 28391901184.0, "2725": 28391901184.0, "2730": 28391901184.0, "2735": 28391901184.0, "2740": 28391901184.0, "2745": 28391901184.0, "2750": 28391901184.0, "2755": 28391901184.0, "2760": 28391901184.0, "2765": 28391901184.0, "2770": 28391901184.0, "2775": 28391901184.0, "2780": 28391901184.0, "2785": 28391901184.0, "2790": 28391901184.0, "2795": 28391901184.0, "2800": 28391901184.0, "2805": 28391901184.0, "2810": 28391901184.0, "2815": 28391901184.0, "2820": 28391901184.0, "2825": 28391901184.0, "2830": 28391901184.0, "2835": 28391901184.0, "2840": 28391901184.0, "2845": 28391901184.0, "2850": 28391901184.0, "2855": 28391901184.0, "2860": 28391901184.0, "2865": 28391901184.0, "2870": 28391901184.0, "2875": 28391901184.0, "2880": 28391901184.0, "2885": 28391901184.0, "2890": 28391901184.0, "2895": 28391901184.0, "2900": 28391901184.0, "2905": 28391901184.0, "2910": 28391901184.0, "2915": 28391901184.0, "2920": 28391901184.0, "2925": 28391901184.0, "2930": 28391901184.0, "2935": 28391901184.0, "2940": 28391901184.0, "2945": 28391901184.0, "2950": 28391901184.0, "2955": 28391901184.0, "2960": 28391901184.0, "2965": 28391901184.0, "2970": 28391901184.0, "2975": 28391901184.0, "2980": 28391901184.0, "2985": 28391901184.0, "2990": 28391901184.0, "2995": 28391901184.0, "3000": 28391901184.0, "3005": 28391901184.0, "3010": 28391901184.0, "3015": 28391901184.0, "3020": 28391901184.0, "3025": 28391901184.0, "3030": 28391901184.0, "3035": 28391901184.0, "3040": 28391901184.0, "3045": 28391901184.0, "3050": 28391901184.0, "3055": 28391901184.0, "3060": 28391901184.0, "3065": 28391901184.0, "3070": 28391901184.0, "3075": 28391901184.0, "3080": 28391901184.0, "3085": 28391901184.0, "3090": 28391901184.0, "3095": 28391901184.0, "3100": 28391901184.0, "3105": 28391901184.0, "3110": 28391901184.0, "3115": 28391901184.0, "3120": 28391901184.0, "3125": 28391901184.0, "3130": 28391901184.0, "3135": 28391901184.0, "3140": 28391901184.0, "3145": 28391901184.0, "3150": 28391901184.0, "3155": 28391901184.0, "3160": 28391901184.0, "3165": 28391901184.0, "3170": 28391901184.0, "3175": 28391901184.0, "3180": 28391901184.0, "3185": 28391901184.0, "3190": 28391901184.0, "3195": 28391901184.0, "3200": 28391901184.0, "3205": 28391901184.0, "3210": 28391901184.0, "3215": 28391901184.0, "3220": 28391901184.0, "3225": 28391901184.0, "3230": 28391901184.0, "3235": 28391901184.0, "3240": 28391901184.0, "3245": 28391901184.0, "3250": 28391901184.0, "3255": 28391901184.0, "3260": 28391901184.0, "3265": 28391901184.0, "3270": 28391901184.0, "3275": 28391901184.0, "3280": 28391901184.0, "3285": 28391901184.0, "3290": 28391901184.0, "3295": 28391901184.0, "3300": 28391901184.0, "3305": 28391901184.0, "3310": 28391901184.0, "3315": 28391901184.0, "3320": 28391901184.0, "3325": 28391901184.0, "3330": 28391901184.0, "3335": 28391901184.0, "3340": 28391901184.0, "3345": 28391901184.0, "3350": 28391901184.0, "3355": 28391901184.0, "3360": 28391901184.0, "3365": 28391901184.0, "3370": 28391901184.0, "3375": 28391901184.0, "3380": 28391901184.0, "3385": 28391901184.0, "3390": 28391901184.0, "3395": 28391901184.0, "3400": 28391901184.0, "3405": 28391901184.0, "3410": 28391901184.0, "3415": 28391901184.0, "3420": 28391901184.0, "3425": 28391901184.0, "3430": 28391901184.0, "3435": 28391901184.0, "3440": 28391901184.0, "3445": 28391901184.0, "3450": 28391901184.0, "3455": 28391901184.0, "3460": 28391901184.0, "3465": 28391901184.0, "3470": 28391901184.0, "3475": 28391901184.0, "3480": 28391901184.0, "3485": 28391901184.0, "3490": 28391901184.0, "3495": 28391901184.0, "3500": 28391901184.0, "3505": 28391901184.0, "3510": 28391901184.0, "3515": 28391901184.0, "3520": 28391901184.0, "3525": 28391901184.0, "3530": 28391901184.0, "3535": 28391901184.0, "3540": 28391901184.0, "3545": 28391901184.0, "3550": 28391901184.0, "3555": 28391901184.0, "3560": 28391901184.0, "3565": 28391901184.0, "3570": 28391901184.0, "3575": 28391901184.0, "3580": 28391901184.0, "3585": 28391901184.0, "3590": 28391901184.0, "3595": 28391901184.0, "3600": 28391901184.0, "3605": 28391901184.0, "3610": 28391901184.0, "3615": 28391901184.0, "3620": 28391901184.0, "3625": 28391901184.0, "3630": 28391901184.0, "3635": 28391901184.0, "3640": 28391901184.0, "3645": 28391901184.0, "3650": 28391712768.0, "3655": 28391712768.0, "3660": 28391712768.0, "3665": 28391712768.0, "3670": 28391712768.0, "3675": 28391712768.0, "3680": 28391712768.0, "3685": 28391712768.0, "3690": 28391712768.0, "3695": 28391712768.0, "3700": 28391712768.0, "3705": 28391712768.0, "3710": 28391712768.0, "3715": 28391712768.0, "3720": 28391712768.0, "3725": 28391712768.0, "3730": 28391712768.0, "3735": 28391712768.0, "3740": 28391712768.0, "3745": 28391712768.0, "3750": 28391712768.0, "3755": 28391712768.0, "3760": 28391712768.0, "3765": 28391712768.0, "3770": 28391712768.0, "3775": 28391712768.0, "3780": 28391712768.0, "3785": 28391712768.0, "3790": 28391712768.0, "3795": 28391712768.0, "3800": 28391712768.0, "3805": 28391712768.0, "3810": 28391712768.0, "3815": 28391712768.0, "3820": 28391712768.0, "3825": 28391712768.0, "3830": 28391712768.0, "3835": 28391712768.0, "3840": 28391712768.0, "3845": 28391712768.0, "3850": 28391712768.0, "3855": 28391712768.0, "3860": 28391712768.0, "3865": 28391712768.0, "3870": 28391712768.0, "3875": 28391712768.0, "3880": 28391712768.0, "3885": 28391712768.0, "3890": 28391712768.0, "3895": 28391712768.0, "3900": 28391712768.0, "3905": 28391712768.0, "3910": 28391712768.0, "3915": 28391712768.0, "3920": 28391712768.0, "3925": 28391712768.0, "3930": 28391712768.0, "3935": 28391712768.0, "3940": 28391712768.0, "3945": 28391712768.0, "3950": 28391712768.0, "3955": 28391712768.0, "3960": 28391712768.0, "3965": 28391712768.0, "3970": 28391712768.0, "3975": 28391712768.0, "3980": 28391712768.0, "3985": 28391712768.0, "3990": 28391712768.0, "3995": 28391712768.0, "4000": 28391712768.0, "4005": 28391712768.0, "4010": 28391712768.0, "4015": 28391712768.0, "4020": 28391712768.0, "4025": 28391712768.0, "4030": 28391712768.0, "4035": 28391712768.0, "4040": 28391712768.0, "4045": 28391712768.0, "4050": 28391712768.0, "4055": 28391712768.0, "4060": 28391712768.0, "4065": 28391712768.0, "4070": 28391712768.0, "4075": 28391712768.0, "4080": 28391712768.0, "4085": 28391712768.0, "4090": 28391712768.0, "4095": 28391712768.0, "4100": 28391712768.0, "4105": 28391712768.0, "4110": 28391712768.0, "4115": 28391712768.0, "4120": 28391712768.0, "4125": 28391712768.0, "4130": 28391712768.0, "4135": 28391712768.0, "4140": 28391712768.0, "4145": 28391712768.0, "4150": 28391712768.0, "4155": 28391712768.0, "4160": 28391712768.0, "4165": 28391712768.0, "4170": 28391712768.0, "4175": 28391712768.0, "4180": 28391712768.0, "4185": 28391712768.0, "4190": 28391712768.0, "4195": 28391712768.0, "4200": 28391712768.0, "4205": 28391712768.0, "4210": 28391712768.0, "4215": 28391712768.0, "4220": 28391712768.0, "4225": 28391712768.0, "4230": 28391712768.0, "4235": 28391712768.0, "4240": 28391712768.0, "4245": 28391712768.0, "4250": 28391712768.0, "4255": 28391712768.0, "4260": 28391712768.0, "4265": 28391712768.0, "4270": 28391712768.0, "4275": 28391712768.0, "4280": 28391712768.0, "4285": 28391712768.0, "4290": 28391712768.0, "4295": 28391712768.0, "4300": 28391712768.0, "4305": 28391712768.0, "4310": 28391712768.0, "4315": 28391712768.0, "4320": 28391712768.0, "4325": 28391712768.0, "4330": 28391712768.0, "4335": 28391712768.0, "4340": 28391712768.0, "4345": 28391712768.0, "4350": 28391712768.0, "4355": 28391712768.0, "4360": 28391712768.0, "4365": 28391712768.0, "4370": 28391712768.0, "4375": 28391712768.0, "4380": 28391712768.0, "4385": 28391712768.0, "4390": 28391712768.0, "4395": 28391712768.0, "4400": 28391712768.0, "4405": 28391712768.0, "4410": 28391712768.0, "4415": 28391712768.0, "4420": 28391712768.0, "4425": 28391712768.0, "4430": 28391712768.0, "4435": 28391712768.0, "4440": 28391712768.0, "4445": 28391712768.0, "4450": 28391712768.0, "4455": 28391712768.0, "4460": 28391712768.0, "4465": 28391712768.0, "4470": 28391712768.0, "4475": 28391712768.0, "4480": 28391712768.0, "4485": 28391712768.0, "4490": 28391712768.0, "4495": 28391712768.0, "4500": 28391712768.0, "4505": 28391712768.0, "4510": 28391712768.0, "4515": 28391712768.0, "4520": 28391712768.0, "4525": 28391712768.0, "4530": 28391712768.0, "4535": 28391712768.0, "4540": 28391712768.0, "4545": 28391712768.0, "4550": 28391712768.0, "4555": 28391712768.0, "4560": 28391712768.0, "4565": 28391712768.0, "4570": 28391712768.0, "4575": 28391712768.0, "4580": 28391712768.0, "4585": 28391712768.0, "4590": 28391712768.0, "4595": 28391712768.0, "4600": 28391712768.0, "4605": 28391712768.0, "4610": 28391712768.0, "4615": 28391712768.0, "4620": 28391712768.0, "4625": 28391712768.0, "4630": 28391712768.0, "4635": 28391712768.0, "4640": 28391712768.0, "4645": 28391712768.0, "4650": 28391712768.0, "4655": 28391712768.0, "4660": 28391712768.0, "4665": 28391712768.0, "4670": 28391712768.0, "4675": 28391712768.0, "4680": 28391712768.0, "4685": 28391712768.0, "4690": 28391712768.0, "4695": 28391712768.0, "4700": 28391712768.0, "4705": 28391712768.0, "4710": 28391712768.0, "4715": 28391712768.0, "4720": 28391712768.0, "4725": 28391712768.0, "4730": 28391712768.0, "4735": 28391712768.0, "4740": 28391712768.0, "4745": 28391712768.0, "4750": 28391712768.0, "4755": 28391712768.0, "4760": 28391712768.0, "4765": 28391712768.0, "4770": 28391712768.0, "4775": 28391712768.0, "4780": 28391712768.0, "4785": 28391712768.0, "4790": 28391712768.0, "4795": 28391712768.0, "4800": 28391712768.0, "4805": 28391712768.0, "4810": 28391712768.0, "4815": 28391712768.0, "4820": 28391712768.0, "4825": 28391712768.0, "4830": 28391712768.0, "4835": 28391712768.0, "4840": 28391712768.0, "4845": 28391712768.0, "4850": 28391712768.0, "4855": 28391712768.0, "4860": 28391712768.0, "4865": 28391712768.0, "4870": 28391712768.0, "4875": 28391712768.0, "4880": 28391712768.0, "4885": 28391712768.0, "4890": 28391712768.0, "4895": 28391712768.0, "4900": 28391712768.0, "4905": 28391712768.0, "4910": 28391712768.0, "4915": 28391712768.0, "4920": 28391712768.0, "4925": 28391712768.0, "4930": 28391712768.0, "4935": 28391712768.0, "4940": 28391712768.0, "4945": 28391712768.0, "4950": 28391712768.0, "4955": 28391712768.0, "4960": 28391712768.0, "4965": 28391712768.0, "4970": 28391712768.0, "4975": 28391712768.0, "4980": 28391712768.0, "4985": 28391712768.0, "4990": 28391712768.0, "4995": 28391712768.0, "5000": 28391712768.0, "5005": 28391712768.0, "5010": 28391712768.0, "5015": 28391712768.0, "5020": 28391712768.0, "5025": 28391712768.0, "5030": 28391712768.0, "5035": 28391712768.0, "5040": 28391712768.0, "5045": 28391712768.0, "5050": 28391712768.0, "5055": 28391712768.0, "5060": 28391712768.0, "5065": 28391712768.0, "5070": 28391712768.0, "5075": 28391712768.0, "5080": 28391712768.0, "5085": 28391712768.0, "5090": 28391712768.0, "5095": 28391712768.0, "5100": 28391712768.0, "5105": 28391712768.0, "5110": 28391712768.0, "5115": 28391712768.0, "5120": 28391712768.0, "5125": 28391712768.0, "5130": 28391712768.0, "5135": 28391712768.0, "5140": 28391712768.0, "5145": 28391712768.0, "5150": 28391712768.0, "5155": 28391712768.0, "5160": 28391712768.0, "5165": 28391712768.0, "5170": 28391712768.0, "5175": 28391712768.0, "5180": 28391712768.0, "5185": 28391712768.0, "5190": 28391712768.0, "5195": 28391712768.0, "5200": 28391712768.0, "5205": 28391712768.0, "5210": 28391712768.0, "5215": 28391712768.0, "5220": 28391712768.0, "5225": 28391712768.0, "5230": 28391712768.0, "5235": 28391712768.0, "5240": 28391712768.0, "5245": 28391712768.0, "5250": 28391712768.0, "5255": 28391712768.0, "5260": 28391712768.0, "5265": 28391712768.0, "5270": 28391712768.0, "5275": 28391712768.0, "5280": 28391712768.0, "5285": 28391712768.0, "5290": 28391712768.0, "5295": 28391712768.0, "5300": 28391712768.0, "5305": 28391712768.0, "5310": 28391712768.0, "5315": 28391712768.0, "5320": 28391712768.0, "5325": 28391712768.0, "5330": 28391712768.0, "5335": 28391712768.0, "5340": 28391712768.0, "5345": 28391712768.0, "5350": 28391712768.0, "5355": 28391712768.0, "5360": 28391712768.0, "5365": 28391712768.0, "5370": 28391712768.0, "5375": 28391712768.0, "5380": 28391712768.0, "5385": 28391712768.0, "5390": 28391712768.0, "5395": 28391712768.0, "5400": 28391712768.0, "5405": 28391712768.0, "5410": 28391712768.0, "5415": 28391712768.0, "5420": 28391712768.0, "5425": 28391712768.0, "5430": 28391712768.0, "5435": 28391712768.0, "5440": 28391712768.0, "5445": 28391712768.0, "5450": 28391712768.0, "5455": 28391712768.0, "5460": 28391712768.0, "5465": 28391712768.0, "5470": 28391712768.0, "5475": 28391712768.0, "5480": 28391712768.0, "5485": 28391712768.0, "5490": 28391712768.0, "5495": 28391712768.0, "5500": 28391712768.0, "5505": 28391712768.0, "5510": 28391712768.0, "5515": 28391712768.0, "5520": 28391712768.0, "5525": 28391712768.0, "5530": 28391712768.0, "5535": 28391712768.0, "5540": 28391712768.0, "5545": 28391712768.0, "5550": 28391712768.0, "5555": 28391712768.0, "5560": 28391712768.0, "5565": 28391712768.0, "5570": 28391712768.0, "5575": 28391712768.0, "5580": 28391712768.0, "5585": 28391712768.0, "5590": 28391712768.0, "5595": 28391712768.0, "5600": 28391712768.0, "5605": 28391712768.0, "5610": 28391712768.0, "5615": 28391712768.0, "5620": 28391712768.0, "5625": 28391712768.0, "5630": 28391712768.0, "5635": 28391712768.0, "5640": 28391712768.0, "5645": 28391712768.0, "5650": 28391712768.0, "5655": 28391712768.0, "5660": 28391712768.0, "5665": 28391712768.0, "5670": 28391712768.0, "5675": 28391712768.0, "5680": 28391712768.0, "5685": 28391712768.0, "5690": 28391712768.0, "5695": 28391712768.0, "5700": 28391712768.0, "5705": 28391712768.0, "5710": 28391712768.0, "5715": 28391712768.0, "5720": 28391712768.0, "5725": 28391712768.0, "5730": 28391712768.0, "5735": 28391712768.0, "5740": 28391712768.0, "5745": 28391712768.0, "5750": 28391712768.0, "5755": 28391712768.0, "5760": 28391712768.0, "5765": 28391712768.0, "5770": 28391712768.0, "5775": 28391712768.0, "5780": 28391712768.0, "5785": 28391712768.0, "5790": 28391712768.0, "5795": 28391712768.0, "5800": 28391712768.0, "5805": 28391712768.0, "5810": 28391712768.0, "5815": 28391712768.0, "5820": 28391712768.0, "5825": 28391712768.0, "5830": 28391712768.0, "5835": 28391712768.0, "5840": 28391712768.0, "5845": 28391712768.0, "5850": 28391712768.0, "5855": 28391712768.0, "5860": 28391712768.0, "5865": 28391712768.0, "5870": 28391712768.0, "5875": 28391712768.0, "5880": 28391712768.0, "5885": 28391712768.0, "5890": 28391712768.0, "5895": 28391712768.0, "5900": 28391712768.0, "5905": 28391712768.0, "5910": 28391712768.0, "5915": 28391712768.0, "5920": 28391712768.0, "5925": 28391712768.0, "5930": 28391712768.0, "5935": 28391712768.0, "5940": 28391712768.0, "5945": 28391712768.0, "5950": 28391712768.0, "5955": 28391712768.0, "5960": 28391712768.0, "5965": 28391712768.0, "5970": 28391712768.0, "5975": 28391712768.0, "5980": 28391712768.0, "5985": 28391712768.0, "5990": 28391712768.0, "5995": 28391712768.0, "6000": 28391712768.0, "6005": 28391712768.0, "6010": 28391712768.0, "6015": 28391712768.0, "6020": 28391712768.0, "6025": 28391712768.0, "6030": 28391712768.0, "6035": 28391712768.0, "6040": 28391712768.0, "6045": 28391712768.0, "6050": 28391712768.0, "6055": 28391712768.0, "6060": 28391712768.0, "6065": 28391712768.0, "6070": 28391712768.0, "6075": 28391712768.0, "6080": 28391712768.0, "6085": 28391712768.0, "6090": 28391712768.0, "6095": 28391712768.0, "6100": 28391712768.0, "6105": 28391712768.0, "6110": 28391712768.0, "6115": 28391712768.0, "6120": 28391712768.0, "6125": 28391712768.0, "6130": 28391712768.0, "6135": 28391712768.0, "6140": 28391712768.0, "6145": 28391712768.0, "6150": 28391712768.0, "6155": 28391712768.0, "6160": 28391712768.0, "6165": 28391712768.0, "6170": 28391712768.0, "6175": 28391712768.0, "6180": 28391712768.0, "6185": 28391712768.0, "6190": 28391712768.0, "6195": 28391712768.0, "6200": 28391712768.0, "6205": 28391712768.0, "6210": 28391712768.0, "6215": 28391712768.0, "6220": 28391712768.0, "6225": 28391712768.0, "6230": 28391712768.0, "6235": 28391712768.0, "6240": 28391712768.0, "6245": 28391712768.0, "6250": 28391712768.0, "6255": 28391712768.0, "6260": 28391712768.0, "6265": 28391712768.0, "6270": 28391712768.0, "6275": 28391712768.0, "6280": 28391712768.0, "6285": 28391712768.0, "6290": 28391712768.0, "6295": 28391712768.0, "6300": 28391712768.0, "6305": 28391712768.0, "6310": 28391712768.0, "6315": 28391712768.0, "6320": 28391712768.0, "6325": 28391712768.0, "6330": 28391712768.0, "6335": 28391712768.0, "6340": 28391712768.0, "6345": 28391712768.0, "6350": 28391712768.0, "6355": 28391712768.0, "6360": 28391712768.0, "6365": 28391712768.0, "6370": 28391712768.0, "6375": 28391712768.0, "6380": 28391712768.0, "6385": 28391712768.0, "6390": 28391712768.0, "6395": 28391712768.0, "6400": 28391712768.0, "6405": 28391712768.0, "6410": 28391712768.0, "6415": 28391712768.0, "6420": 28391712768.0, "6425": 28391712768.0, "6430": 28391712768.0, "6435": 28391712768.0, "6440": 28391712768.0, "6445": 28391712768.0, "6450": 28391712768.0, "6455": 28391712768.0, "6460": 28391712768.0, "6465": 28391712768.0, "6470": 28391712768.0, "6475": 28391712768.0, "6480": 28391712768.0, "6485": 28391712768.0, "6490": 28391712768.0, "6495": 28391712768.0, "6500": 28391712768.0, "6505": 28391712768.0, "6510": 28391712768.0, "6515": 28391712768.0, "6520": 28391712768.0, "6525": 28391712768.0, "6530": 28391712768.0, "6535": 28391712768.0, "6540": 28391712768.0, "6545": 28391712768.0, "6550": 28391712768.0, "6555": 28391712768.0, "6560": 28391712768.0, "6565": 28391712768.0, "6570": 28391712768.0, "6575": 28391712768.0, "6580": 28391712768.0, "6585": 28391712768.0, "6590": 28391712768.0, "6595": 28391712768.0, "6600": 28391712768.0, "6605": 28391712768.0, "6610": 28391712768.0, "6615": 28391712768.0, "6620": 28391712768.0, "6625": 28391712768.0, "6630": 28391712768.0, "6635": 28391712768.0, "6640": 28391712768.0, "6645": 28391712768.0, "6650": 28391712768.0, "6655": 28391712768.0, "6660": 28391712768.0, "6665": 28391712768.0, "6670": 28391712768.0, "6675": 28391712768.0, "6680": 28391712768.0, "6685": 28391712768.0, "6690": 28391712768.0, "6695": 28391712768.0, "6700": 28391712768.0, "6705": 28391712768.0, "6710": 28391712768.0, "6715": 28391712768.0, "6720": 28391712768.0, "6725": 28391712768.0, "6730": 28391712768.0, "6735": 28391712768.0, "6740": 28391712768.0, "6745": 28391712768.0, "6750": 28391712768.0, "6755": 28391712768.0, "6760": 28391712768.0, "6765": 28391712768.0, "6770": 28391712768.0, "6775": 28391712768.0, "6780": 28391712768.0, "6785": 28391712768.0, "6790": 28391712768.0, "6795": 28391712768.0, "6800": 28391712768.0, "6805": 28391712768.0, "6810": 28391712768.0, "6815": 28391712768.0, "6820": 28391712768.0, "6825": 28391712768.0, "6830": 28391712768.0, "6835": 28391712768.0, "6840": 28391712768.0, "6845": 28391712768.0, "6850": 28391712768.0, "6855": 28391712768.0, "6860": 28391712768.0, "6865": 28391712768.0, "6870": 28391712768.0, "6875": 28391712768.0, "6880": 28391712768.0, "6885": 28391712768.0, "6890": 28391712768.0, "6895": 28391712768.0, "6900": 28391712768.0, "6905": 28391712768.0, "6910": 28391712768.0, "6915": 28391712768.0, "6920": 28391712768.0, "6925": 28391712768.0, "6930": 28391712768.0, "6935": 28391712768.0, "6940": 28391712768.0, "6945": 28391712768.0, "6950": 28391712768.0, "6955": 28391712768.0, "6960": 28391712768.0, "6965": 28391712768.0, "6970": 28391712768.0, "6975": 28391712768.0, "6980": 28391712768.0, "6985": 28391712768.0, "6990": 28391712768.0, "6995": 28391712768.0, "7000": 28391712768.0, "7005": 28391712768.0, "7010": 28391712768.0, "7015": 28391712768.0, "7020": 28391712768.0, "7025": 28391712768.0, "7030": 28391712768.0, "7035": 28391712768.0, "7040": 28391712768.0, "7045": 28391712768.0, "7050": 28391712768.0, "7055": 28391712768.0, "7060": 28391712768.0, "7065": 28391712768.0, "7070": 28391712768.0, "7075": 28391712768.0, "7080": 28391712768.0, "7085": 28391712768.0, "7090": 28391712768.0, "7095": 28391712768.0, "7100": 28391712768.0, "7105": 28391712768.0, "7110": 28391712768.0, "7115": 28391712768.0, "7120": 28391712768.0, "7125": 28391712768.0, "7130": 28391712768.0, "7135": 28391712768.0, "7140": 28391712768.0, "7145": 28391712768.0, "7150": 28391712768.0, "7155": 28391712768.0, "7160": 28391712768.0, "7165": 28391712768.0, "7170": 28391712768.0, "7175": 28391712768.0, "7180": 28391712768.0, "7185": 28391712768.0, "7190": 28391712768.0, "7195": 28391712768.0, "7200": 28391712768.0, "7205": 28391712768.0, "7210": 28391712768.0, "7215": 28391712768.0, "7220": 28391712768.0, "7225": 28391712768.0, "7230": 28391712768.0, "7235": 28391712768.0, "7240": 28391712768.0, "7245": 28391712768.0, "7250": 28391712768.0, "7255": 28391712768.0, "7260": 28391712768.0, "7265": 28391712768.0, "7270": 28391712768.0, "7275": 28391712768.0, "7280": 28391712768.0, "7285": 28391712768.0, "7290": 28391712768.0, "7295": 28391712768.0, "7300": 28391712768.0, "7305": 28391712768.0, "7310": 28391712768.0, "7315": 28391712768.0, "7320": 28391712768.0, "7325": 28391712768.0, "7330": 28391712768.0, "7335": 28391712768.0, "7340": 28391712768.0, "7345": 28391712768.0, "7350": 28391712768.0, "7355": 28391712768.0, "7360": 28391712768.0, "7365": 28391712768.0, "7370": 28391712768.0, "7375": 28391712768.0, "7380": 28391712768.0, "7385": 28391712768.0, "7390": 28391712768.0, "7395": 28391712768.0, "7400": 28391712768.0, "7405": 28391712768.0, "7410": 28391712768.0, "7415": 28391712768.0, "7420": 28391712768.0, "7425": 28391712768.0, "7430": 28391712768.0, "7435": 28391712768.0, "7440": 28391712768.0, "7445": 28391712768.0, "7450": 28391712768.0, "7455": 28391712768.0, "7460": 28391712768.0, "7465": 28391712768.0, "7470": 28391712768.0, "7475": 28391712768.0, "7480": 28391712768.0, "7485": 28391712768.0, "7490": 28391712768.0, "7495": 28391712768.0, "7500": 28391712768.0, "7505": 28391712768.0, "7510": 28391712768.0, "7515": 28391712768.0, "7520": 28391712768.0, "7525": 28391712768.0, "7530": 28391712768.0, "7535": 28391712768.0, "7540": 28391712768.0, "7545": 28391712768.0, "7550": 28391712768.0, "7555": 28391712768.0, "7560": 28391712768.0, "7565": 28391712768.0, "7570": 28391712768.0, "7575": 28391712768.0, "7580": 28391712768.0, "7585": 28391712768.0, "7590": 28391712768.0, "7595": 28391712768.0, "7600": 28391712768.0, "7605": 28391712768.0, "7610": 28391712768.0, "7615": 28391712768.0, "7620": 28391712768.0, "7625": 28391712768.0, "7630": 28391712768.0, "7635": 28391712768.0, "7640": 28391712768.0, "7645": 28391712768.0, "7650": 28391712768.0, "7655": 28391712768.0, "7660": 28391712768.0, "7665": 28391712768.0, "7670": 28391712768.0, "7675": 28391712768.0, "7680": 28391712768.0, "7685": 28391712768.0, "7690": 28391712768.0, "7695": 28391712768.0, "7700": 28391712768.0, "7705": 28391712768.0, "7710": 28391712768.0, "7715": 28391712768.0, "7720": 28391712768.0, "7725": 28391712768.0, "7730": 28391712768.0, "7735": 28391712768.0, "7740": 28391712768.0, "7745": 28391712768.0, "7750": 28391712768.0, "7755": 28391712768.0, "7760": 28391712768.0, "7765": 28391712768.0, "7770": 28391712768.0, "7775": 28391712768.0, "7780": 28391712768.0, "7785": 28391712768.0, "7790": 28391712768.0, "7795": 28391712768.0, "7800": 28391712768.0, "7805": 28391712768.0, "7810": 28391712768.0, "7815": 28391712768.0, "7820": 28391712768.0, "7825": 28391712768.0, "7830": 28391712768.0, "7835": 28391712768.0, "7840": 28391712768.0, "7845": 28391712768.0, "7850": 28391712768.0, "7855": 28391712768.0, "7860": 28391712768.0, "7865": 28391712768.0, "7870": 28391712768.0, "7875": 28391712768.0, "7880": 28391712768.0, "7885": 28391712768.0, "7890": 28391712768.0, "7895": 28391712768.0, "7900": 28391712768.0, "7905": 28391712768.0, "7910": 28391712768.0, "7915": 28391712768.0, "7920": 28391712768.0, "7925": 28391712768.0, "7930": 28391712768.0, "7935": 28391712768.0, "7940": 28391712768.0, "7945": 28391712768.0, "7950": 28391712768.0, "7955": 28391712768.0, "7960": 28391712768.0, "7965": 28391712768.0, "7970": 28391712768.0, "7975": 28391712768.0, "7980": 28391712768.0, "7985": 28391712768.0, "7990": 28391712768.0, "7995": 28391712768.0, "8000": 28391712768.0, "8005": 28391712768.0, "8010": 28391712768.0, "8015": 28391712768.0, "8020": 28391712768.0, "8025": 28391712768.0, "8030": 28391712768.0, "8035": 28391712768.0, "8040": 28391712768.0, "8045": 28391712768.0, "8050": 28391712768.0, "8055": 28391712768.0, "8060": 28391712768.0, "8065": 28391712768.0, "8070": 28391712768.0, "8075": 28391712768.0, "8080": 28391712768.0, "8085": 28391712768.0, "8090": 28391712768.0, "8095": 28391712768.0, "8100": 28391712768.0, "8105": 28391712768.0, "8110": 28391712768.0, "8115": 28391712768.0, "8120": 28391712768.0, "8125": 28391712768.0, "8130": 28391712768.0, "8135": 28391712768.0, "8140": 28391712768.0, "8145": 28391712768.0, "8150": 28391712768.0, "8155": 28391712768.0, "8160": 28391712768.0, "8165": 28391712768.0, "8170": 28391712768.0, "8175": 28391712768.0, "8180": 28391712768.0, "8185": 28391712768.0, "8190": 28391712768.0, "8195": 28391712768.0, "8200": 28391712768.0, "8205": 28391712768.0, "8210": 28391712768.0, "8215": 28391712768.0, "8220": 28391712768.0, "8225": 28391712768.0, "8230": 28391712768.0, "8235": 28391712768.0, "8240": 28391712768.0, "8245": 28391712768.0, "8250": 28391712768.0, "8255": 28391712768.0, "8260": 28391712768.0, "8265": 28391712768.0, "8270": 28391712768.0, "8275": 28391712768.0, "8280": 28391712768.0, "8285": 28391712768.0, "8290": 28391712768.0, "8295": 28391712768.0, "8300": 28391712768.0, "8305": 28391712768.0, "8310": 28391712768.0, "8315": 28391712768.0, "8320": 28391712768.0, "8325": 28391712768.0, "8330": 28391712768.0, "8335": 28391712768.0, "8340": 28391712768.0, "8345": 28391712768.0, "8350": 28391712768.0, "8355": 28391712768.0, "8360": 28391712768.0, "8365": 28391712768.0, "8370": 28391712768.0, "8375": 28391712768.0, "8380": 28391712768.0, "8385": 28391712768.0, "8390": 28391712768.0, "8395": 28391712768.0, "8400": 28391712768.0, "8405": 28391712768.0, "8410": 28391712768.0, "8415": 28391712768.0, "8420": 28391712768.0, "8425": 28391712768.0, "8430": 28391712768.0, "8435": 28391712768.0, "8440": 28391712768.0, "8445": 28391712768.0, "8450": 28391712768.0, "8455": 28391712768.0, "8460": 28391712768.0, "8465": 28391712768.0, "8470": 28391712768.0, "8475": 28391712768.0, "8480": 28391712768.0, "8485": 28391712768.0, "8490": 28391712768.0, "8495": 28391712768.0, "8500": 28391712768.0, "8505": 28391712768.0, "8510": 28391712768.0, "8515": 28391712768.0, "8520": 28391712768.0, "8525": 28391712768.0, "8530": 28391712768.0, "8535": 28391712768.0, "8540": 28391712768.0, "8545": 28391712768.0, "8550": 28391712768.0, "8555": 28391712768.0, "8560": 28391712768.0, "8565": 28391712768.0, "8570": 28391712768.0, "8575": 28391712768.0, "8580": 28391712768.0, "8585": 28391712768.0, "8590": 28391712768.0, "8595": 28391712768.0, "8600": 28391712768.0, "8605": 28391712768.0, "8610": 28391712768.0, "8615": 28391712768.0, "8620": 28391712768.0, "8625": 28391712768.0, "8630": 28391712768.0, "8635": 28391712768.0, "8640": 28391712768.0, "8645": 28391712768.0, "8650": 28391712768.0, "8655": 28391712768.0, "8660": 28391712768.0, "8665": 28391712768.0, "8670": 28391712768.0, "8675": 28391712768.0, "8680": 28391712768.0, "8685": 28391712768.0, "8690": 28391712768.0, "8695": 28391712768.0, "8700": 28391712768.0, "8705": 28391712768.0, "8710": 28391712768.0, "8715": 28391712768.0, "8720": 28391712768.0, "8725": 28391712768.0, "8730": 28391712768.0, "8735": 28391712768.0, "8740": 28391712768.0, "8745": 28391712768.0, "8750": 28391712768.0, "8755": 28391712768.0, "8760": 28391712768.0, "8765": 28391712768.0, "8770": 28391712768.0, "8775": 28391712768.0, "8780": 28391712768.0, "8785": 28391712768.0, "8790": 28391712768.0, "8795": 28391712768.0, "8800": 28391712768.0, "8805": 28391712768.0, "8810": 28391712768.0, "8815": 28391712768.0, "8820": 28391712768.0, "8825": 28391712768.0, "8830": 28391712768.0, "8835": 28391712768.0, "8840": 28391712768.0, "8845": 28391712768.0, "8850": 28391712768.0, "8855": 28391712768.0, "8860": 28391712768.0, "8865": 28391712768.0, "8870": 28391712768.0, "8875": 28391712768.0, "8880": 28391712768.0, "8885": 28391712768.0, "8890": 28391712768.0, "8895": 28391712768.0, "8900": 28391712768.0, "8905": 28391712768.0, "8910": 28391712768.0, "8915": 28391712768.0, "8920": 28391712768.0, "8925": 28391712768.0, "8930": 28391712768.0, "8935": 28391712768.0, "8940": 28391712768.0, "8945": 28391712768.0, "8950": 28391712768.0, "8955": 28391712768.0, "8960": 28391712768.0, "8965": 28391712768.0, "8970": 28391712768.0, "8975": 28391712768.0, "8980": 28391712768.0, "8985": 28391712768.0, "8990": 28391712768.0, "8995": 28391712768.0, "9000": 28391712768.0, "9005": 28391712768.0, "9010": 28391712768.0, "9015": 28391712768.0, "9020": 28391712768.0, "9025": 28391712768.0, "9030": 28391712768.0, "9035": 28391712768.0, "9040": 28391712768.0, "9045": 28391712768.0, "9050": 28391712768.0, "9055": 28391712768.0, "9060": 28391712768.0, "9065": 28391712768.0, "9070": 28391712768.0, "9075": 28391712768.0, "9080": 28391712768.0, "9085": 28391712768.0, "9090": 28391712768.0, "9095": 28391712768.0, "9100": 28391712768.0, "9105": 28391712768.0, "9110": 28391712768.0, "9115": 28391712768.0, "9120": 28391712768.0, "9125": 28391712768.0, "9130": 28391712768.0, "9135": 28391712768.0, "9140": 28391712768.0, "9145": 28391712768.0, "9150": 28391712768.0, "9155": 28391712768.0, "9160": 28391712768.0, "9165": 28391712768.0, "9170": 28391712768.0, "9175": 28391712768.0, "9180": 28391712768.0, "9185": 28391712768.0, "9190": 28391712768.0, "9195": 28391712768.0, "9200": 28391712768.0, "9205": 28391712768.0, "9210": 28391712768.0, "9215": 28391712768.0, "9220": 28391712768.0, "9225": 28391712768.0, "9230": 28391712768.0, "9235": 28391712768.0, "9240": 28391712768.0, "9245": 28391712768.0, "9250": 28391712768.0, "9255": 28391712768.0, "9260": 28391712768.0, "9265": 28391712768.0, "9270": 28391712768.0, "9275": 28391712768.0, "9280": 28391712768.0, "9285": 28391712768.0, "9290": 28391712768.0, "9295": 28391712768.0, "9300": 28391712768.0, "9305": 28391712768.0, "9310": 28391712768.0, "9315": 28391712768.0, "9320": 28391712768.0, "9325": 28391712768.0, "9330": 28391712768.0, "9335": 28391712768.0, "9340": 28391712768.0, "9345": 28391712768.0, "9350": 28391712768.0, "9355": 28391712768.0, "9360": 28391712768.0, "9365": 28391712768.0, "9370": 28391712768.0, "9375": 28391712768.0, "9380": 28391712768.0, "9385": 28391712768.0, "9390": 28391712768.0, "9395": 28391712768.0, "9400": 28391712768.0, "9405": 28391712768.0, "9410": 28391712768.0, "9415": 28391712768.0, "9420": 28391712768.0, "9425": 28391712768.0, "9430": 28391712768.0, "9435": 28391712768.0, "9440": 28391712768.0, "9445": 28391712768.0, "9450": 28391712768.0, "9455": 28391712768.0, "9460": 28391712768.0, "9465": 28391712768.0, "9470": 28391712768.0, "9475": 28391712768.0, "9480": 28391712768.0, "9485": 28391712768.0, "9490": 28391712768.0, "9495": 28391712768.0, "9500": 28391712768.0, "9505": 28391712768.0, "9510": 28391712768.0, "9515": 28391712768.0, "9520": 28391712768.0, "9525": 28391712768.0, "9530": 28391712768.0, "9535": 28391712768.0, "9540": 28391712768.0, "9545": 28391712768.0, "9550": 28391712768.0, "9555": 28391712768.0, "9560": 28391712768.0, "9565": 28391712768.0, "9570": 28391712768.0, "9575": 28391712768.0, "9580": 28391712768.0, "9585": 28391712768.0, "9590": 28391712768.0, "9595": 28391712768.0, "9600": 28391712768.0, "9605": 28391712768.0, "9610": 28391712768.0, "9615": 28391712768.0, "9620": 28391712768.0, "9625": 28391712768.0, "9630": 28391712768.0, "9635": 28391712768.0, "9640": 28391712768.0, "9645": 28391712768.0, "9650": 28391712768.0, "9655": 28391712768.0, "9660": 28391712768.0, "9665": 28391712768.0, "9670": 28391712768.0, "9675": 28391712768.0, "9680": 28391712768.0, "9685": 28391712768.0, "9690": 28391712768.0, "9695": 28391712768.0, "9700": 28391712768.0, "9705": 28391712768.0, "9710": 28391712768.0, "9715": 28391712768.0, "9720": 28391712768.0, "9725": 28391712768.0, "9730": 28391712768.0, "9735": 28391712768.0, "9740": 28391712768.0, "9745": 28391712768.0, "9750": 28391712768.0, "9755": 28391712768.0, "9760": 28391712768.0, "9765": 28391712768.0, "9770": 28391712768.0, "9775": 28391712768.0, "9780": 28391712768.0, "9785": 28391712768.0, "9790": 28391712768.0, "9795": 28391712768.0, "9800": 28391712768.0, "9805": 28391712768.0, "9810": 28391712768.0, "9815": 28391712768.0, "9820": 28391712768.0, "9825": 28391712768.0, "9830": 28391712768.0, "9835": 28391712768.0, "9840": 28391712768.0, "9845": 28391712768.0, "9850": 28391712768.0, "9855": 28391712768.0, "9860": 28391712768.0, "9865": 28391712768.0, "9870": 28391712768.0, "9875": 28391712768.0, "9880": 28391712768.0, "9885": 28391712768.0, "9890": 28391712768.0, "9895": 28391712768.0, "9900": 28391712768.0, "9905": 28391712768.0, "9910": 28391712768.0, "9915": 28391712768.0, "9920": 28391712768.0, "9925": 28391712768.0, "9930": 28391712768.0, "9935": 28391712768.0, "9940": 28391712768.0, "9945": 28391712768.0, "9950": 28391712768.0, "9955": 28391712768.0, "9960": 28391712768.0, "9965": 28391712768.0, "9970": 28391712768.0, "9975": 28391712768.0, "9980": 28391712768.0, "9985": 28391712768.0, "9990": 28391712768.0, "9995": 28391712768.0, "10000": 28391712768.0, "10005": 28391712768.0, "10010": 28391712768.0, "10015": 28391712768.0, "10020": 28391712768.0, "10025": 28391712768.0, "10030": 28391712768.0, "10035": 28391712768.0, "10040": 28391712768.0, "10045": 28391712768.0, "10050": 28391712768.0, "10055": 28391712768.0, "10060": 28391712768.0, "10065": 28391712768.0, "10070": 28391712768.0, "10075": 28391712768.0, "10080": 28391712768.0, "10085": 28391712768.0, "10090": 28391712768.0, "10095": 28391712768.0, "10100": 28391712768.0, "10105": 28391712768.0, "10110": 28391712768.0, "10115": 28391712768.0, "10120": 28391712768.0, "10125": 28391712768.0, "10130": 28391712768.0, "10135": 28391712768.0, "10140": 28391712768.0, "10145": 28391712768.0, "10150": 28391712768.0, "10155": 28391712768.0, "10160": 28391712768.0, "10165": 28391712768.0, "10170": 28391712768.0, "10175": 28391712768.0, "10180": 28391712768.0, "10185": 28391712768.0, "10190": 28391712768.0, "10195": 28391712768.0, "10200": 28391712768.0, "10205": 28391712768.0, "10210": 28391712768.0, "10215": 28391712768.0, "10220": 28391712768.0, "10225": 28391712768.0, "10230": 28391712768.0, "10235": 28391712768.0, "10240": 28391712768.0, "10245": 28391712768.0, "10250": 28391712768.0, "10255": 28391712768.0, "10260": 28391712768.0, "10265": 28391712768.0, "10270": 28391712768.0, "10275": 28391712768.0, "10280": 28391712768.0, "10285": 28391712768.0, "10290": 28391712768.0, "10295": 28391712768.0, "10300": 28391712768.0, "10305": 28391712768.0, "10310": 28391712768.0, "10315": 28391712768.0, "10320": 28391712768.0, "10325": 28391712768.0, "10330": 28391712768.0, "10335": 28391712768.0, "10340": 28391712768.0, "10345": 28391712768.0, "10350": 28391712768.0, "10355": 28391712768.0, "10360": 28391712768.0, "10365": 28391712768.0, "10370": 28391712768.0, "10375": 28391712768.0, "10380": 28391712768.0, "10385": 28391712768.0, "10390": 28391712768.0, "10395": 28391712768.0, "10400": 28391712768.0, "10405": 28391712768.0, "10410": 28391712768.0, "10415": 28391712768.0, "10420": 28391712768.0, "10425": 28391712768.0, "10430": 28391712768.0, "10435": 28391712768.0, "10440": 28391712768.0, "10445": 28391712768.0, "10450": 28391712768.0, "10455": 28391712768.0, "10460": 28391712768.0, "10465": 28391712768.0, "10470": 28391712768.0, "10475": 28391712768.0, "10480": 28391712768.0, "10485": 28391712768.0, "10490": 28391712768.0, "10495": 28391712768.0, "10500": 28391712768.0, "10505": 28391712768.0, "10510": 28391712768.0, "10515": 28391712768.0, "10520": 28391712768.0, "10525": 28391712768.0, "10530": 28391712768.0, "10535": 28391712768.0, "10540": 28391712768.0, "10545": 28391712768.0, "10550": 28391712768.0, "10555": 28391712768.0, "10560": 28391712768.0, "10565": 28391712768.0, "10570": 28391712768.0, "10575": 28391712768.0, "10580": 28391712768.0, "10585": 28391712768.0, "10590": 28391712768.0, "10595": 28391712768.0, "10600": 28391712768.0, "10605": 28391712768.0, "10610": 28391712768.0, "10615": 28391712768.0, "10620": 28391712768.0, "10625": 28391712768.0, "10630": 28391712768.0, "10635": 28391712768.0, "10640": 28391712768.0, "10645": 28391712768.0, "10650": 28391712768.0, "10655": 28391712768.0, "10660": 28391712768.0, "10665": 28391712768.0, "10670": 28391712768.0, "10675": 28391712768.0, "10680": 28391712768.0, "10685": 28391712768.0, "10690": 28391712768.0, "10695": 28391712768.0, "10700": 28391712768.0, "10705": 28391712768.0, "10710": 28391712768.0, "10715": 28391712768.0, "10720": 28391712768.0, "10725": 28391712768.0, "10730": 28391712768.0, "10735": 28391712768.0, "10740": 28391712768.0, "10745": 28391712768.0, "10750": 28391712768.0, "10755": 28391712768.0, "10760": 28391712768.0, "10765": 28391712768.0, "10770": 28391712768.0, "10775": 28391712768.0, "10780": 28391712768.0, "10785": 28391712768.0, "10790": 28391712768.0, "10795": 28391712768.0, "10800": 28391712768.0, "10805": 28391712768.0, "10810": 28391712768.0, "10815": 28391712768.0, "10820": 28391712768.0, "10825": 28391712768.0, "10830": 28391712768.0, "10835": 28391712768.0, "10840": 28391712768.0, "10845": 28391712768.0, "10850": 28391712768.0, "10855": 28391712768.0, "10860": 28391712768.0, "10865": 28391712768.0, "10870": 28391712768.0, "10875": 28391712768.0, "10880": 28391712768.0, "10885": 28391712768.0, "10890": 28391712768.0, "10895": 28391712768.0, "10900": 28391712768.0, "10905": 28391712768.0, "10910": 28391712768.0, "10915": 28391712768.0, "10920": 28391712768.0, "10925": 28391712768.0, "10930": 28391712768.0, "10935": 28391712768.0, "10940": 28391712768.0, "10945": 28391712768.0, "10950": 28391712768.0, "10955": 28391712768.0, "10960": 28391712768.0, "10965": 28391712768.0, "10970": 28391712768.0, "10975": 28391712768.0, "10980": 28391712768.0, "10985": 28391712768.0, "10990": 28391712768.0, "10995": 28391712768.0, "11000": 28391712768.0, "11005": 28391712768.0, "11010": 28391712768.0, "11015": 28391712768.0, "11020": 28391712768.0, "11025": 28391712768.0, "11030": 28391712768.0, "11035": 28391712768.0, "11040": 28391712768.0, "11045": 28391712768.0, "11050": 28391712768.0, "11055": 28391712768.0, "11060": 28391712768.0, "11065": 28391712768.0, "11070": 28391712768.0, "11075": 28391712768.0, "11080": 28391712768.0, "11085": 28391712768.0, "11090": 28391712768.0, "11095": 28391712768.0, "11100": 28391712768.0, "11105": 28391712768.0, "11110": 28391712768.0, "11115": 28391712768.0, "11120": 28391712768.0, "11125": 28391712768.0, "11130": 28391712768.0, "11135": 28391712768.0, "11140": 28391712768.0, "11145": 28391712768.0, "11150": 28391712768.0, "11155": 28391712768.0, "11160": 28391712768.0, "11165": 28391712768.0, "11170": 28391712768.0, "11175": 28391712768.0, "11180": 28391712768.0, "11185": 28391712768.0, "11190": 28391712768.0, "11195": 28391712768.0, "11200": 28391712768.0, "11205": 28391712768.0, "11210": 28391712768.0, "11215": 28391712768.0, "11220": 28391712768.0, "11225": 28391712768.0, "11230": 28391712768.0, "11235": 28391712768.0, "11240": 28391712768.0, "11245": 28391712768.0, "11250": 28391712768.0, "11255": 28391712768.0, "11260": 28391712768.0, "11265": 28391712768.0, "11270": 28391712768.0, "11275": 28391712768.0, "11280": 28391712768.0, "11285": 28391712768.0, "11290": 28391712768.0, "11295": 28391712768.0, "11300": 28391712768.0, "11305": 28391712768.0, "11310": 28391712768.0, "11315": 28391712768.0, "11320": 28391712768.0, "11325": 28391712768.0, "11330": 28391712768.0, "11335": 28391712768.0, "11340": 28391712768.0, "11345": 28391712768.0, "11350": 28391712768.0, "11355": 28391712768.0, "11360": 28391712768.0, "11365": 28391712768.0, "11370": 28391712768.0, "11375": 28391712768.0, "11380": 28391712768.0, "11385": 28391712768.0, "11390": 28391712768.0, "11395": 28391712768.0, "11400": 28391712768.0, "11405": 28391712768.0, "11410": 28391712768.0, "11415": 28391712768.0, "11420": 28391712768.0, "11425": 28391712768.0, "11430": 28391712768.0, "11435": 28391712768.0, "11440": 28391712768.0, "11445": 28391712768.0, "11450": 28391712768.0, "11455": 28391712768.0, "11460": 28391712768.0, "11465": 28391712768.0, "11470": 28391712768.0, "11475": 28391712768.0, "11480": 28391712768.0, "11485": 28391712768.0, "11490": 28391712768.0, "11495": 28391712768.0, "11500": 28391712768.0, "11505": 28391712768.0, "11510": 28391712768.0, "11515": 28391712768.0, "11520": 28391712768.0, "11525": 28391712768.0, "11530": 28391712768.0, "11535": 28391712768.0, "11540": 28391712768.0, "11545": 28391712768.0, "11550": 28391712768.0, "11555": 28391712768.0, "11560": 28391712768.0, "11565": 28391712768.0, "11570": 28391712768.0, "11575": 28391712768.0, "11580": 28391712768.0, "11585": 28391712768.0, "11590": 28391712768.0, "11595": 28391712768.0, "11600": 28391712768.0, "11605": 28391712768.0, "11610": 28391712768.0, "11615": 28391712768.0, "11620": 28391712768.0, "11625": 28391712768.0, "11630": 28391712768.0, "11635": 28391712768.0, "11640": 28391712768.0, "11645": 28391712768.0, "11650": 28391712768.0, "11655": 28391712768.0, "11660": 28391712768.0, "11665": 28391712768.0, "11670": 28391712768.0, "11675": 28391712768.0, "11680": 28391712768.0, "11685": 28391712768.0, "11690": 28391712768.0, "11695": 28391712768.0, "11700": 28391712768.0, "11705": 28391712768.0, "11710": 28391712768.0, "11715": 28391712768.0, "11720": 28391712768.0, "11725": 28391712768.0, "11730": 28391712768.0, "11735": 28391712768.0, "11740": 28391712768.0, "11745": 28391712768.0, "11750": 28391712768.0, "11755": 28391712768.0, "11760": 28391712768.0, "11765": 28391712768.0, "11770": 28391712768.0, "11775": 28391712768.0, "11780": 28391712768.0, "11785": 28391712768.0, "11790": 28391712768.0, "11795": 28391712768.0, "11800": 28391712768.0, "11805": 28391712768.0, "11810": 28391712768.0, "11815": 28391712768.0, "11820": 28391712768.0, "11825": 28391712768.0, "11830": 28391712768.0, "11835": 28391712768.0, "11840": 28391712768.0, "11845": 28391712768.0, "11850": 28391712768.0, "11855": 28391712768.0, "11860": 28391712768.0, "11865": 28391712768.0, "11870": 28391712768.0, "11875": 28391712768.0, "11880": 28391712768.0, "11885": 28391712768.0, "11890": 28391712768.0, "11895": 28391712768.0, "11900": 28391712768.0, "11905": 28391712768.0, "11910": 28391712768.0, "11915": 28391712768.0, "11920": 28391712768.0, "11925": 28391712768.0, "11930": 28391712768.0, "11935": 28391712768.0, "11940": 28391712768.0, "11945": 28391712768.0, "11950": 28391712768.0, "11955": 28391712768.0, "11960": 28391712768.0, "11965": 28391712768.0, "11970": 28391712768.0, "11975": 28391712768.0, "11980": 28391712768.0, "11985": 28391712768.0, "11990": 28391712768.0, "11995": 28391712768.0, "12000": 28391712768.0, "12005": 28391712768.0, "12010": 28391712768.0, "12015": 28391712768.0, "12020": 28391712768.0, "12025": 28391712768.0, "12030": 28391712768.0, "12035": 28391712768.0, "12040": 28391712768.0, "12045": 28391712768.0, "12050": 28391712768.0, "12055": 28391712768.0, "12060": 28391712768.0, "12065": 28391712768.0, "12070": 28391712768.0, "12075": 28391712768.0, "12080": 28391712768.0, "12085": 28391712768.0, "12090": 28391712768.0, "12095": 28391712768.0, "12100": 28391712768.0, "12105": 28391712768.0, "12110": 28391712768.0, "12115": 28391712768.0, "12120": 28391712768.0, "12125": 28391712768.0, "12130": 28391712768.0, "12135": 28391712768.0, "12140": 28391712768.0, "12145": 28391712768.0, "12150": 28391712768.0, "12155": 28391712768.0, "12160": 28391712768.0, "12165": 28391712768.0, "12170": 28391712768.0, "12175": 28391712768.0, "12180": 28391712768.0, "12185": 28391712768.0, "12190": 28391712768.0, "12195": 28391712768.0, "12200": 28391712768.0, "12205": 28391712768.0, "12210": 28391712768.0, "12215": 28391712768.0, "12220": 28391712768.0, "12225": 28391712768.0, "12230": 28391712768.0, "12235": 28391712768.0, "12240": 28391712768.0, "12245": 28391712768.0, "12250": 28391712768.0, "12255": 28391712768.0, "12260": 28391712768.0, "12265": 28391712768.0, "12270": 28391712768.0, "12275": 28391712768.0, "12280": 28391712768.0, "12285": 28391712768.0, "12290": 28391712768.0, "12295": 28391712768.0, "12300": 28391712768.0, "12305": 28391712768.0, "12310": 28391712768.0, "12315": 28391712768.0, "12320": 28391712768.0, "12325": 28391712768.0, "12330": 28391712768.0, "12335": 28391712768.0, "12340": 28391712768.0, "12345": 28391712768.0, "12350": 28391712768.0, "12355": 28391712768.0, "12360": 28391712768.0, "12365": 28391712768.0, "12370": 28391712768.0, "12375": 28391712768.0, "12380": 28391712768.0, "12385": 28391712768.0, "12390": 28391712768.0, "12395": 28391712768.0, "12400": 28391712768.0, "12405": 28391712768.0, "12410": 28391712768.0, "12415": 28391712768.0, "12420": 28391712768.0, "12425": 28391712768.0, "12430": 28391712768.0, "12435": 28391712768.0, "12440": 28391712768.0, "12445": 28391712768.0, "12450": 28391712768.0, "12455": 28391712768.0, "12460": 28391712768.0, "12465": 28391712768.0, "12470": 28391712768.0, "12475": 28391712768.0, "12480": 28391712768.0, "12485": 28391712768.0, "12490": 28391712768.0, "12495": 28391712768.0, "12500": 28391712768.0, "12505": 28391712768.0, "12510": 28391712768.0, "12515": 28391712768.0, "12520": 28391712768.0, "12525": 28391712768.0, "12530": 28391712768.0, "12535": 28391712768.0, "12540": 28391712768.0, "12545": 28391712768.0, "12550": 28391712768.0, "12555": 28391712768.0, "12560": 28391712768.0, "12565": 28391712768.0, "12570": 28391712768.0, "12575": 28391712768.0, "12580": 28391712768.0, "12585": 28391712768.0, "12590": 28391712768.0, "12595": 28391712768.0, "12600": 28391712768.0, "12605": 28391712768.0, "12610": 28391712768.0, "12615": 28391712768.0, "12620": 28391712768.0, "12625": 28391712768.0, "12630": 28391712768.0, "12635": 28391712768.0, "12640": 28391712768.0, "12645": 28391712768.0, "12650": 28391712768.0, "12655": 28391712768.0, "12660": 28391712768.0, "12665": 28391712768.0, "12670": 28391712768.0, "12675": 28391712768.0, "12680": 28391712768.0, "12685": 28391712768.0, "12690": 28391712768.0, "12695": 28391712768.0, "12700": 28391712768.0, "12705": 28391712768.0, "12710": 28391712768.0, "12715": 28391712768.0, "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}, "iteration-time": {"start_step": 1, "end_step": 13000, "step_interval": 5, "values": {"1": "nan", "5": "nan", "10": "nan", "15": "nan", "20": "nan", "25": "nan", "30": "nan", "35": "nan", "40": "nan", "45": "nan", "50": "nan", "55": "nan", "60": "nan", "65": "nan", "70": "nan", "75": "nan", "80": "nan", "85": "nan", "90": "nan", "95": "nan", "100": 3.55899, "105": "nan", "110": "nan", "115": "nan", "120": "nan", "125": "nan", "130": "nan", "135": "nan", "140": "nan", "145": "nan", "150": "nan", "155": "nan", "160": "nan", "165": "nan", "170": "nan", "175": "nan", "180": "nan", "185": "nan", "190": "nan", "195": "nan", "200": 3.44029, "205": "nan", "210": "nan", "215": "nan", "220": "nan", "225": "nan", "230": "nan", "235": "nan", "240": "nan", "245": "nan", "250": "nan", "255": "nan", "260": "nan", "265": "nan", "270": "nan", "275": "nan", "280": "nan", "285": "nan", "290": "nan", "295": "nan", "300": 3.44419, "305": "nan", "310": "nan", "315": "nan", "320": "nan", "325": "nan", "330": "nan", "335": "nan", "340": "nan", "345": "nan", "350": "nan", "355": "nan", "360": "nan", "365": "nan", "370": "nan", "375": "nan", "380": "nan", "385": "nan", "390": "nan", "395": "nan", "400": 3.44595, "405": "nan", "410": "nan", "415": "nan", "420": "nan", "425": "nan", "430": "nan", "435": "nan", "440": "nan", "445": "nan", "450": "nan", "455": "nan", "460": "nan", "465": "nan", "470": "nan", "475": "nan", "480": "nan", "485": "nan", "490": "nan", "495": "nan", "500": 3.4477, "505": "nan", "510": "nan", "515": "nan", "520": "nan", "525": "nan", "530": "nan", "535": "nan", "540": "nan", "545": "nan", "550": "nan", "555": "nan", "560": "nan", "565": "nan", "570": "nan", "575": "nan", "580": "nan", "585": "nan", "590": "nan", "595": "nan", "600": 3.44601, "605": "nan", "610": "nan", "615": "nan", "620": "nan", "625": "nan", "630": "nan", "635": "nan", "640": "nan", "645": "nan", "650": "nan", "655": "nan", "660": "nan", "665": "nan", "670": "nan", "675": "nan", "680": "nan", "685": "nan", "690": "nan", "695": "nan", "700": 3.4446, "705": "nan", "710": "nan", "715": "nan", "720": "nan", "725": "nan", "730": "nan", "735": "nan", "740": "nan", "745": "nan", "750": "nan", "755": "nan", "760": "nan", "765": "nan", "770": "nan", "775": "nan", "780": "nan", "785": "nan", "790": "nan", "795": "nan", "800": 3.44149, "805": "nan", "810": "nan", "815": "nan", "820": "nan", "825": "nan", "830": "nan", "835": "nan", "840": "nan", "845": "nan", "850": "nan", "855": "nan", "860": "nan", "865": "nan", "870": "nan", "875": "nan", "880": "nan", "885": "nan", "890": "nan", "895": "nan", "900": 3.43766, "905": "nan", "910": "nan", "915": "nan", "920": "nan", "925": "nan", "930": "nan", "935": "nan", "940": "nan", "945": "nan", "950": "nan", "955": "nan", "960": "nan", "965": "nan", "970": "nan", "975": "nan", "980": "nan", "985": "nan", "990": "nan", "995": "nan", "1000": 3.43127, "1005": "nan", "1010": "nan", "1015": "nan", "1020": "nan", "1025": "nan", "1030": "nan", "1035": "nan", "1040": "nan", "1045": "nan", "1050": "nan", "1055": "nan", "1060": "nan", "1065": "nan", "1070": "nan", "1075": "nan", "1080": "nan", "1085": "nan", "1090": "nan", "1095": "nan", "1100": 3.42513, "1105": "nan", "1110": "nan", "1115": "nan", "1120": "nan", "1125": "nan", "1130": "nan", "1135": "nan", "1140": "nan", "1145": "nan", "1150": "nan", "1155": "nan", "1160": "nan", "1165": "nan", "1170": "nan", "1175": "nan", "1180": "nan", "1185": "nan", "1190": "nan", "1195": "nan", "1200": 3.41859, "1205": "nan", "1210": "nan", "1215": "nan", "1220": "nan", "1225": "nan", "1230": "nan", "1235": "nan", "1240": "nan", "1245": "nan", "1250": "nan", "1255": "nan", "1260": "nan", "1265": "nan", "1270": "nan", "1275": "nan", "1280": "nan", "1285": "nan", "1290": "nan", "1295": "nan", "1300": 3.41493, "1305": "nan", "1310": "nan", "1315": "nan", "1320": "nan", "1325": "nan", "1330": "nan", "1335": "nan", "1340": "nan", "1345": "nan", "1350": "nan", "1355": "nan", "1360": "nan", "1365": "nan", "1370": "nan", "1375": "nan", "1380": "nan", "1385": "nan", "1390": "nan", "1395": "nan", "1400": 3.41144, "1405": "nan", "1410": "nan", "1415": "nan", "1420": "nan", "1425": "nan", "1430": "nan", "1435": "nan", "1440": "nan", "1445": "nan", "1450": "nan", "1455": "nan", "1460": "nan", "1465": "nan", "1470": "nan", "1475": "nan", "1480": "nan", "1485": "nan", "1490": "nan", "1495": "nan", "1500": 3.40875, "1505": "nan", "1510": "nan", "1515": "nan", "1520": "nan", "1525": "nan", "1530": "nan", "1535": "nan", "1540": "nan", "1545": "nan", "1550": "nan", "1555": "nan", "1560": "nan", "1565": "nan", "1570": "nan", "1575": "nan", "1580": "nan", "1585": "nan", "1590": "nan", "1595": "nan", "1600": 3.40598, "1605": "nan", "1610": "nan", "1615": "nan", "1620": "nan", "1625": "nan", "1630": "nan", "1635": "nan", "1640": "nan", "1645": "nan", "1650": "nan", "1655": "nan", "1660": "nan", "1665": "nan", "1670": "nan", "1675": "nan", "1680": "nan", "1685": "nan", "1690": "nan", "1695": "nan", "1700": 3.4038, "1705": "nan", "1710": "nan", "1715": "nan", "1720": "nan", "1725": "nan", "1730": "nan", "1735": "nan", "1740": "nan", "1745": "nan", "1750": "nan", "1755": "nan", "1760": "nan", "1765": "nan", "1770": "nan", "1775": "nan", "1780": "nan", "1785": "nan", "1790": "nan", "1795": "nan", "1800": 3.40301, "1805": "nan", "1810": "nan", "1815": "nan", "1820": "nan", "1825": "nan", "1830": "nan", "1835": "nan", "1840": "nan", "1845": "nan", "1850": "nan", "1855": "nan", "1860": "nan", "1865": "nan", "1870": "nan", "1875": "nan", "1880": "nan", "1885": "nan", "1890": "nan", "1895": "nan", "1900": 3.40033, "1905": "nan", "1910": "nan", "1915": "nan", "1920": "nan", "1925": "nan", "1930": "nan", "1935": "nan", "1940": "nan", "1945": "nan", "1950": "nan", "1955": "nan", "1960": "nan", "1965": "nan", "1970": "nan", "1975": "nan", "1980": "nan", "1985": "nan", "1990": "nan", "1995": "nan", "2000": 3.3983, "2005": "nan", "2010": "nan", "2015": "nan", "2020": "nan", "2025": "nan", "2030": "nan", "2035": "nan", "2040": "nan", "2045": "nan", "2050": "nan", "2055": "nan", "2060": "nan", "2065": "nan", "2070": "nan", "2075": "nan", "2080": "nan", "2085": "nan", "2090": "nan", "2095": "nan", "2100": 3.3967, "2105": "nan", "2110": "nan", "2115": "nan", "2120": "nan", "2125": "nan", "2130": "nan", "2135": "nan", "2140": "nan", "2145": "nan", "2150": "nan", "2155": "nan", "2160": "nan", "2165": "nan", "2170": "nan", "2175": "nan", "2180": "nan", "2185": "nan", "2190": "nan", "2195": "nan", "2200": 3.39751, "2205": "nan", "2210": "nan", "2215": "nan", "2220": "nan", "2225": "nan", "2230": "nan", "2235": "nan", "2240": "nan", "2245": "nan", "2250": "nan", "2255": "nan", "2260": "nan", "2265": "nan", "2270": "nan", "2275": "nan", "2280": "nan", "2285": "nan", "2290": "nan", "2295": "nan", "2300": 3.39609, "2305": "nan", "2310": "nan", "2315": "nan", "2320": "nan", "2325": "nan", "2330": "nan", "2335": "nan", "2340": "nan", "2345": "nan", "2350": "nan", "2355": "nan", "2360": "nan", "2365": "nan", "2370": "nan", "2375": "nan", "2380": "nan", "2385": "nan", "2390": "nan", "2395": "nan", "2400": 3.39469, "2405": "nan", "2410": "nan", "2415": "nan", "2420": "nan", "2425": "nan", "2430": "nan", "2435": "nan", "2440": "nan", "2445": "nan", "2450": "nan", "2455": "nan", "2460": "nan", "2465": "nan", "2470": "nan", "2475": "nan", "2480": "nan", "2485": "nan", "2490": "nan", "2495": "nan", "2500": 3.39297, "2505": "nan", "2510": "nan", "2515": "nan", "2520": "nan", "2525": "nan", "2530": "nan", "2535": "nan", "2540": "nan", "2545": "nan", "2550": "nan", "2555": "nan", "2560": "nan", "2565": "nan", "2570": "nan", "2575": "nan", "2580": "nan", "2585": "nan", "2590": "nan", "2595": "nan", "2600": 3.39215, "2605": "nan", "2610": "nan", "2615": "nan", "2620": "nan", "2625": "nan", "2630": "nan", "2635": "nan", "2640": "nan", "2645": "nan", "2650": "nan", "2655": "nan", "2660": "nan", "2665": "nan", "2670": "nan", "2675": "nan", "2680": "nan", "2685": "nan", "2690": "nan", "2695": "nan", "2700": 3.39252, "2705": "nan", "2710": "nan", "2715": "nan", "2720": "nan", "2725": "nan", "2730": "nan", "2735": "nan", "2740": "nan", "2745": "nan", "2750": "nan", "2755": "nan", "2760": "nan", "2765": "nan", "2770": "nan", "2775": "nan", "2780": "nan", "2785": "nan", "2790": "nan", "2795": "nan", "2800": 3.39244, "2805": "nan", "2810": "nan", "2815": "nan", "2820": "nan", "2825": "nan", "2830": "nan", "2835": "nan", "2840": "nan", "2845": "nan", "2850": "nan", "2855": "nan", "2860": "nan", "2865": "nan", "2870": "nan", "2875": "nan", "2880": "nan", "2885": "nan", "2890": "nan", "2895": "nan", "2900": 3.39099, "2905": "nan", "2910": "nan", "2915": "nan", "2920": "nan", "2925": "nan", "2930": "nan", "2935": "nan", "2940": "nan", "2945": "nan", "2950": "nan", "2955": "nan", "2960": "nan", "2965": "nan", "2970": "nan", "2975": "nan", "2980": "nan", "2985": "nan", "2990": "nan", "2995": "nan", "3000": 3.39125, "3005": "nan", "3010": "nan", "3015": "nan", "3020": "nan", "3025": "nan", "3030": "nan", "3035": "nan", "3040": "nan", "3045": "nan", "3050": "nan", "3055": "nan", "3060": "nan", "3065": "nan", "3070": "nan", "3075": "nan", "3080": "nan", "3085": "nan", "3090": "nan", "3095": "nan", "3100": 3.39087, "3105": "nan", "3110": "nan", "3115": "nan", "3120": "nan", "3125": "nan", "3130": "nan", "3135": "nan", "3140": "nan", "3145": "nan", "3150": "nan", "3155": "nan", "3160": "nan", "3165": "nan", "3170": "nan", "3175": "nan", "3180": "nan", "3185": "nan", "3190": "nan", "3195": "nan", "3200": 3.3894, "3205": "nan", "3210": "nan", "3215": "nan", "3220": "nan", "3225": "nan", "3230": "nan", "3235": "nan", "3240": "nan", "3245": "nan", "3250": "nan", "3255": "nan", "3260": "nan", "3265": "nan", "3270": "nan", "3275": "nan", "3280": "nan", "3285": "nan", "3290": "nan", "3295": "nan", "3300": 3.39035, "3305": "nan", "3310": "nan", "3315": "nan", "3320": "nan", "3325": "nan", "3330": "nan", "3335": "nan", "3340": "nan", "3345": "nan", "3350": "nan", "3355": "nan", "3360": "nan", "3365": "nan", "3370": "nan", "3375": "nan", "3380": "nan", "3385": "nan", "3390": "nan", "3395": "nan", "3400": 3.39004, "3405": "nan", "3410": "nan", "3415": "nan", "3420": "nan", "3425": "nan", "3430": "nan", "3435": "nan", "3440": "nan", "3445": "nan", "3450": "nan", "3455": "nan", "3460": "nan", "3465": "nan", "3470": "nan", "3475": "nan", "3480": "nan", "3485": "nan", "3490": "nan", "3495": "nan", "3500": 3.38966, "3505": "nan", "3510": "nan", "3515": "nan", "3520": "nan", "3525": "nan", "3530": "nan", "3535": "nan", "3540": "nan", "3545": "nan", "3550": "nan", "3555": "nan", "3560": "nan", "3565": "nan", "3570": "nan", "3575": "nan", "3580": "nan", "3585": "nan", "3590": "nan", "3595": "nan", "3600": 3.38952, "3605": "nan", "3610": "nan", "3615": "nan", "3620": "nan", "3625": "nan", "3630": "nan", "3635": "nan", "3640": "nan", "3645": "nan", "3650": "nan", "3655": "nan", "3660": "nan", "3665": "nan", "3670": "nan", "3675": "nan", "3680": "nan", "3685": "nan", "3690": "nan", "3695": "nan", "3700": 3.3871, "3705": "nan", "3710": "nan", "3715": "nan", "3720": "nan", "3725": "nan", "3730": "nan", "3735": "nan", "3740": "nan", "3745": "nan", "3750": "nan", "3755": "nan", "3760": "nan", "3765": "nan", "3770": "nan", "3775": "nan", "3780": "nan", "3785": "nan", "3790": "nan", "3795": "nan", "3800": 3.38626, "3805": "nan", "3810": "nan", "3815": "nan", "3820": "nan", "3825": "nan", "3830": "nan", "3835": "nan", "3840": "nan", "3845": "nan", "3850": "nan", "3855": "nan", "3860": "nan", "3865": "nan", "3870": "nan", "3875": "nan", "3880": "nan", "3885": "nan", "3890": "nan", "3895": "nan", "3900": 3.38708, "3905": "nan", "3910": "nan", "3915": "nan", "3920": "nan", "3925": "nan", "3930": "nan", "3935": "nan", "3940": "nan", "3945": "nan", "3950": "nan", "3955": "nan", "3960": "nan", "3965": "nan", "3970": "nan", "3975": "nan", "3980": "nan", "3985": "nan", "3990": "nan", "3995": "nan", "4000": 3.38632, "4005": "nan", "4010": "nan", "4015": "nan", "4020": "nan", "4025": "nan", "4030": "nan", "4035": "nan", "4040": "nan", "4045": "nan", "4050": "nan", "4055": "nan", "4060": "nan", "4065": "nan", "4070": "nan", "4075": "nan", "4080": "nan", "4085": "nan", "4090": "nan", "4095": "nan", "4100": 3.38441, "4105": "nan", "4110": "nan", "4115": "nan", "4120": "nan", "4125": "nan", "4130": "nan", "4135": "nan", "4140": "nan", "4145": "nan", "4150": "nan", "4155": "nan", "4160": "nan", "4165": "nan", "4170": "nan", "4175": "nan", "4180": "nan", "4185": "nan", "4190": "nan", "4195": "nan", "4200": 3.38457, "4205": "nan", "4210": "nan", "4215": "nan", "4220": "nan", "4225": "nan", "4230": "nan", "4235": "nan", "4240": "nan", "4245": "nan", "4250": "nan", "4255": "nan", "4260": "nan", "4265": "nan", "4270": "nan", "4275": "nan", "4280": "nan", "4285": "nan", "4290": "nan", "4295": "nan", "4300": 3.38457, "4305": "nan", "4310": "nan", "4315": "nan", "4320": "nan", "4325": "nan", "4330": "nan", "4335": "nan", "4340": "nan", "4345": "nan", "4350": "nan", "4355": "nan", "4360": "nan", "4365": "nan", "4370": "nan", "4375": "nan", "4380": "nan", "4385": "nan", "4390": "nan", "4395": "nan", "4400": 3.38389, "4405": "nan", "4410": "nan", "4415": "nan", "4420": "nan", "4425": "nan", "4430": "nan", "4435": "nan", "4440": "nan", "4445": "nan", "4450": "nan", "4455": "nan", "4460": "nan", "4465": "nan", "4470": "nan", "4475": "nan", "4480": "nan", "4485": "nan", "4490": "nan", "4495": "nan", "4500": 3.38314, "4505": "nan", "4510": "nan", "4515": "nan", "4520": "nan", "4525": "nan", "4530": "nan", "4535": "nan", "4540": "nan", "4545": "nan", "4550": "nan", "4555": "nan", "4560": "nan", "4565": "nan", "4570": "nan", "4575": "nan", "4580": "nan", "4585": "nan", "4590": "nan", "4595": "nan", "4600": 3.38093, "4605": "nan", "4610": "nan", "4615": "nan", "4620": "nan", "4625": "nan", "4630": "nan", "4635": "nan", "4640": "nan", "4645": "nan", "4650": "nan", "4655": "nan", "4660": "nan", "4665": "nan", "4670": "nan", "4675": "nan", "4680": "nan", "4685": "nan", "4690": "nan", "4695": "nan", "4700": 3.38075, "4705": "nan", "4710": "nan", "4715": "nan", "4720": "nan", "4725": "nan", "4730": "nan", "4735": "nan", "4740": "nan", "4745": "nan", "4750": "nan", "4755": "nan", "4760": "nan", "4765": "nan", "4770": "nan", "4775": "nan", "4780": "nan", "4785": "nan", "4790": "nan", "4795": "nan", "4800": 3.38137, "4805": "nan", "4810": "nan", "4815": "nan", "4820": "nan", "4825": "nan", "4830": "nan", "4835": "nan", "4840": "nan", "4845": "nan", "4850": "nan", "4855": "nan", "4860": "nan", "4865": "nan", "4870": "nan", "4875": "nan", "4880": "nan", "4885": "nan", "4890": "nan", "4895": "nan", "4900": 3.38055, "4905": "nan", "4910": "nan", "4915": "nan", "4920": "nan", "4925": "nan", "4930": "nan", "4935": "nan", "4940": "nan", "4945": "nan", "4950": "nan", "4955": "nan", "4960": "nan", "4965": "nan", "4970": "nan", "4975": "nan", "4980": "nan", "4985": "nan", "4990": "nan", "4995": "nan", "5000": 3.38053, "5005": "nan", "5010": "nan", "5015": "nan", "5020": "nan", "5025": "nan", "5030": "nan", "5035": "nan", "5040": "nan", "5045": "nan", "5050": "nan", "5055": "nan", "5060": "nan", "5065": "nan", "5070": "nan", "5075": "nan", "5080": "nan", "5085": "nan", "5090": "nan", "5095": "nan", "5100": 3.3822, "5105": "nan", "5110": "nan", "5115": "nan", "5120": "nan", "5125": "nan", "5130": "nan", "5135": "nan", "5140": "nan", "5145": "nan", "5150": "nan", "5155": "nan", "5160": "nan", "5165": "nan", "5170": "nan", "5175": "nan", "5180": "nan", "5185": "nan", "5190": "nan", "5195": "nan", "5200": 3.38192, "5205": "nan", "5210": "nan", "5215": "nan", "5220": "nan", "5225": "nan", "5230": "nan", "5235": "nan", "5240": "nan", "5245": "nan", "5250": "nan", "5255": "nan", "5260": "nan", "5265": "nan", "5270": "nan", "5275": "nan", "5280": "nan", "5285": "nan", "5290": "nan", "5295": "nan", "5300": 3.38156, "5305": "nan", "5310": "nan", "5315": "nan", "5320": "nan", "5325": "nan", "5330": "nan", "5335": "nan", "5340": "nan", "5345": "nan", "5350": "nan", "5355": "nan", "5360": "nan", "5365": "nan", "5370": "nan", "5375": "nan", "5380": "nan", "5385": "nan", "5390": "nan", "5395": "nan", "5400": 3.38069, "5405": "nan", "5410": "nan", "5415": "nan", "5420": "nan", "5425": "nan", "5430": "nan", "5435": "nan", "5440": "nan", "5445": "nan", "5450": "nan", "5455": "nan", "5460": "nan", "5465": "nan", "5470": "nan", "5475": "nan", "5480": "nan", "5485": "nan", "5490": "nan", "5495": "nan", "5500": 3.38347, "5505": "nan", "5510": "nan", "5515": "nan", "5520": "nan", "5525": "nan", "5530": "nan", "5535": "nan", "5540": "nan", "5545": "nan", "5550": "nan", "5555": "nan", "5560": "nan", "5565": "nan", "5570": "nan", "5575": "nan", "5580": "nan", "5585": "nan", "5590": "nan", "5595": "nan", "5600": 3.38222, "5605": "nan", "5610": "nan", "5615": "nan", "5620": "nan", "5625": "nan", "5630": "nan", "5635": "nan", "5640": "nan", "5645": "nan", "5650": "nan", "5655": "nan", "5660": "nan", "5665": "nan", "5670": "nan", "5675": "nan", "5680": "nan", "5685": "nan", "5690": "nan", "5695": "nan", "5700": 3.38243, "5705": "nan", "5710": "nan", "5715": "nan", "5720": "nan", "5725": "nan", "5730": "nan", "5735": "nan", "5740": "nan", "5745": "nan", "5750": "nan", "5755": "nan", "5760": "nan", "5765": "nan", "5770": "nan", "5775": "nan", "5780": "nan", "5785": "nan", "5790": "nan", "5795": "nan", "5800": 3.38216, "5805": "nan", "5810": "nan", "5815": "nan", "5820": "nan", "5825": "nan", "5830": "nan", "5835": "nan", "5840": "nan", "5845": "nan", "5850": "nan", "5855": "nan", "5860": "nan", "5865": "nan", "5870": "nan", "5875": "nan", "5880": "nan", "5885": "nan", "5890": "nan", "5895": "nan", "5900": 3.38049, "5905": "nan", "5910": "nan", "5915": "nan", "5920": "nan", "5925": "nan", "5930": "nan", "5935": "nan", "5940": "nan", "5945": "nan", "5950": "nan", "5955": "nan", "5960": "nan", "5965": "nan", "5970": "nan", "5975": "nan", "5980": "nan", "5985": "nan", "5990": "nan", "5995": "nan", "6000": 3.38023, "6005": "nan", "6010": "nan", "6015": "nan", "6020": "nan", "6025": "nan", "6030": "nan", "6035": "nan", "6040": "nan", "6045": "nan", "6050": "nan", "6055": "nan", "6060": "nan", "6065": "nan", "6070": "nan", "6075": "nan", "6080": "nan", "6085": "nan", "6090": "nan", "6095": "nan", "6100": 3.37904, "6105": "nan", "6110": "nan", "6115": "nan", "6120": "nan", "6125": "nan", "6130": "nan", "6135": "nan", "6140": "nan", "6145": "nan", "6150": "nan", "6155": "nan", "6160": "nan", "6165": "nan", "6170": "nan", "6175": "nan", "6180": "nan", "6185": "nan", "6190": "nan", "6195": "nan", "6200": 3.38026, "6205": "nan", "6210": "nan", "6215": "nan", "6220": "nan", "6225": "nan", "6230": "nan", "6235": "nan", "6240": "nan", "6245": "nan", "6250": "nan", "6255": "nan", "6260": "nan", "6265": "nan", "6270": "nan", "6275": "nan", "6280": "nan", "6285": "nan", "6290": "nan", "6295": "nan", "6300": 3.38003, "6305": "nan", "6310": "nan", "6315": "nan", "6320": "nan", "6325": "nan", "6330": "nan", "6335": "nan", "6340": "nan", "6345": "nan", "6350": "nan", "6355": "nan", "6360": "nan", "6365": "nan", "6370": "nan", "6375": "nan", "6380": "nan", "6385": "nan", "6390": "nan", "6395": "nan", "6400": 3.37878, "6405": "nan", "6410": "nan", "6415": "nan", "6420": "nan", "6425": "nan", "6430": "nan", "6435": "nan", "6440": "nan", "6445": "nan", "6450": "nan", "6455": "nan", "6460": "nan", "6465": "nan", "6470": "nan", "6475": "nan", "6480": "nan", "6485": "nan", "6490": "nan", "6495": "nan", "6500": 3.38006, "6505": "nan", "6510": "nan", "6515": "nan", "6520": "nan", "6525": "nan", "6530": "nan", "6535": "nan", "6540": "nan", "6545": "nan", "6550": "nan", "6555": "nan", "6560": "nan", "6565": "nan", "6570": "nan", "6575": "nan", "6580": "nan", "6585": "nan", "6590": "nan", "6595": "nan", "6600": 3.38036, "6605": "nan", "6610": "nan", "6615": "nan", "6620": "nan", "6625": "nan", "6630": "nan", "6635": "nan", "6640": "nan", "6645": "nan", "6650": "nan", "6655": "nan", "6660": "nan", "6665": "nan", "6670": "nan", "6675": "nan", "6680": "nan", "6685": "nan", "6690": "nan", "6695": "nan", "6700": 3.38116, "6705": "nan", "6710": "nan", "6715": "nan", "6720": "nan", "6725": "nan", "6730": "nan", "6735": "nan", "6740": "nan", "6745": "nan", "6750": "nan", "6755": "nan", "6760": "nan", "6765": "nan", "6770": "nan", "6775": "nan", "6780": "nan", "6785": "nan", "6790": "nan", "6795": "nan", "6800": 3.39788, "6805": "nan", "6810": "nan", "6815": "nan", "6820": "nan", "6825": "nan", "6830": "nan", "6835": "nan", "6840": "nan", "6845": "nan", "6850": "nan", "6855": "nan", "6860": "nan", "6865": "nan", "6870": "nan", "6875": "nan", "6880": "nan", "6885": "nan", "6890": "nan", "6895": "nan", "6900": 3.39841, "6905": "nan", "6910": "nan", "6915": "nan", "6920": "nan", "6925": "nan", "6930": "nan", "6935": "nan", "6940": "nan", "6945": "nan", "6950": "nan", "6955": "nan", "6960": "nan", "6965": "nan", "6970": "nan", "6975": "nan", "6980": "nan", "6985": "nan", "6990": "nan", "6995": "nan", "7000": 3.39845, "7005": "nan", "7010": "nan", "7015": "nan", "7020": "nan", "7025": "nan", "7030": "nan", "7035": "nan", "7040": "nan", "7045": "nan", "7050": "nan", "7055": "nan", "7060": "nan", "7065": "nan", "7070": "nan", "7075": "nan", "7080": "nan", "7085": "nan", "7090": "nan", "7095": "nan", "7100": 3.37973, "7105": "nan", "7110": "nan", "7115": "nan", "7120": "nan", "7125": "nan", "7130": "nan", "7135": "nan", "7140": "nan", "7145": "nan", "7150": "nan", "7155": "nan", "7160": "nan", "7165": "nan", "7170": "nan", "7175": "nan", "7180": "nan", "7185": "nan", "7190": "nan", "7195": "nan", "7200": 3.3813, "7205": "nan", "7210": "nan", "7215": "nan", "7220": "nan", "7225": "nan", "7230": "nan", "7235": "nan", "7240": "nan", "7245": "nan", "7250": "nan", "7255": "nan", "7260": "nan", "7265": "nan", "7270": "nan", "7275": "nan", "7280": "nan", "7285": "nan", "7290": "nan", "7295": "nan", "7300": 3.38135, "7305": "nan", "7310": "nan", "7315": "nan", "7320": "nan", "7325": "nan", "7330": "nan", "7335": "nan", "7340": "nan", "7345": "nan", "7350": "nan", "7355": "nan", "7360": "nan", "7365": "nan", "7370": "nan", "7375": "nan", "7380": "nan", "7385": "nan", "7390": "nan", "7395": "nan", "7400": 3.38258, "7405": "nan", "7410": "nan", "7415": "nan", "7420": "nan", "7425": "nan", "7430": "nan", "7435": "nan", "7440": "nan", "7445": "nan", "7450": "nan", "7455": "nan", "7460": "nan", "7465": "nan", "7470": "nan", "7475": "nan", "7480": "nan", "7485": "nan", "7490": "nan", "7495": "nan", "7500": 3.37993, "7505": "nan", "7510": "nan", "7515": "nan", "7520": "nan", "7525": "nan", "7530": "nan", "7535": "nan", "7540": "nan", "7545": "nan", "7550": "nan", "7555": "nan", "7560": "nan", "7565": "nan", "7570": "nan", "7575": "nan", "7580": "nan", "7585": "nan", "7590": "nan", "7595": "nan", "7600": 3.3791, "7605": "nan", "7610": "nan", "7615": "nan", "7620": "nan", "7625": "nan", "7630": "nan", "7635": "nan", "7640": "nan", "7645": "nan", "7650": "nan", "7655": "nan", "7660": "nan", "7665": "nan", "7670": "nan", "7675": "nan", "7680": "nan", "7685": "nan", "7690": "nan", "7695": "nan", "7700": 3.3801, "7705": "nan", "7710": "nan", "7715": "nan", "7720": "nan", "7725": "nan", "7730": "nan", "7735": "nan", "7740": "nan", "7745": "nan", "7750": "nan", "7755": "nan", "7760": "nan", "7765": "nan", "7770": "nan", "7775": "nan", "7780": "nan", "7785": "nan", "7790": "nan", "7795": "nan", "7800": 3.40629, "7805": "nan", "7810": "nan", "7815": "nan", "7820": "nan", "7825": "nan", "7830": "nan", "7835": "nan", "7840": "nan", "7845": "nan", "7850": "nan", "7855": "nan", "7860": "nan", "7865": "nan", "7870": "nan", "7875": "nan", "7880": "nan", "7885": "nan", "7890": "nan", "7895": "nan", "7900": 3.39663, "7905": "nan", "7910": "nan", "7915": "nan", "7920": "nan", "7925": "nan", "7930": "nan", "7935": "nan", "7940": "nan", "7945": "nan", "7950": "nan", "7955": "nan", "7960": "nan", "7965": "nan", "7970": "nan", "7975": "nan", "7980": "nan", "7985": "nan", "7990": "nan", "7995": "nan", "8000": 3.39421, "8005": "nan", "8010": "nan", "8015": "nan", "8020": "nan", "8025": "nan", "8030": "nan", "8035": "nan", "8040": "nan", "8045": "nan", "8050": "nan", "8055": "nan", "8060": "nan", "8065": "nan", "8070": "nan", "8075": "nan", "8080": "nan", "8085": "nan", "8090": "nan", "8095": "nan", "8100": 3.38433, "8105": "nan", "8110": "nan", "8115": "nan", "8120": "nan", "8125": "nan", "8130": "nan", "8135": "nan", "8140": "nan", "8145": "nan", "8150": "nan", "8155": "nan", "8160": "nan", "8165": "nan", "8170": "nan", "8175": "nan", "8180": "nan", "8185": "nan", "8190": "nan", "8195": "nan", "8200": 3.38564, "8205": "nan", "8210": "nan", "8215": "nan", "8220": "nan", "8225": "nan", "8230": "nan", "8235": "nan", "8240": "nan", "8245": "nan", "8250": "nan", "8255": "nan", "8260": "nan", "8265": "nan", "8270": "nan", "8275": "nan", "8280": "nan", "8285": "nan", "8290": "nan", "8295": "nan", "8300": 3.38548, "8305": "nan", "8310": "nan", "8315": "nan", "8320": "nan", "8325": "nan", "8330": "nan", "8335": "nan", "8340": "nan", "8345": "nan", "8350": "nan", "8355": "nan", "8360": "nan", "8365": "nan", "8370": "nan", "8375": "nan", "8380": "nan", "8385": "nan", "8390": "nan", "8395": "nan", "8400": 3.38652, "8405": "nan", "8410": "nan", "8415": "nan", "8420": "nan", "8425": "nan", "8430": "nan", "8435": "nan", "8440": "nan", "8445": "nan", "8450": "nan", "8455": "nan", "8460": "nan", "8465": "nan", "8470": "nan", "8475": "nan", "8480": "nan", "8485": "nan", "8490": "nan", "8495": "nan", "8500": 3.38537, "8505": "nan", "8510": "nan", "8515": "nan", "8520": "nan", "8525": "nan", "8530": "nan", "8535": "nan", "8540": "nan", "8545": "nan", "8550": "nan", "8555": "nan", "8560": "nan", "8565": "nan", "8570": "nan", "8575": "nan", "8580": "nan", "8585": "nan", "8590": "nan", "8595": "nan", "8600": 3.38441, "8605": "nan", "8610": "nan", "8615": "nan", "8620": "nan", "8625": "nan", "8630": "nan", "8635": "nan", "8640": "nan", "8645": "nan", "8650": "nan", "8655": "nan", "8660": "nan", "8665": "nan", "8670": "nan", "8675": "nan", "8680": "nan", "8685": "nan", "8690": "nan", "8695": "nan", "8700": 3.38569, "8705": "nan", "8710": "nan", "8715": "nan", "8720": "nan", "8725": "nan", "8730": "nan", "8735": "nan", "8740": "nan", "8745": "nan", "8750": "nan", "8755": "nan", "8760": "nan", "8765": "nan", "8770": "nan", "8775": "nan", "8780": "nan", "8785": "nan", "8790": "nan", "8795": "nan", "8800": 3.38471, "8805": "nan", "8810": "nan", "8815": "nan", "8820": "nan", "8825": "nan", "8830": "nan", "8835": "nan", "8840": "nan", "8845": "nan", "8850": "nan", "8855": "nan", "8860": "nan", "8865": "nan", "8870": "nan", "8875": "nan", "8880": "nan", "8885": "nan", "8890": "nan", "8895": "nan", "8900": 3.38583, "8905": "nan", "8910": "nan", "8915": "nan", "8920": "nan", "8925": "nan", "8930": "nan", "8935": "nan", "8940": "nan", "8945": "nan", "8950": "nan", "8955": "nan", "8960": "nan", "8965": "nan", "8970": "nan", "8975": "nan", "8980": "nan", "8985": "nan", "8990": "nan", "8995": "nan", "9000": 3.38381, "9005": "nan", "9010": "nan", "9015": "nan", "9020": "nan", "9025": "nan", "9030": "nan", "9035": "nan", "9040": "nan", "9045": "nan", "9050": "nan", "9055": "nan", "9060": "nan", "9065": "nan", "9070": "nan", "9075": "nan", "9080": "nan", "9085": "nan", "9090": "nan", "9095": "nan", "9100": 3.38317, "9105": "nan", "9110": "nan", "9115": "nan", "9120": "nan", "9125": "nan", "9130": "nan", "9135": "nan", "9140": "nan", "9145": "nan", "9150": "nan", "9155": "nan", "9160": "nan", "9165": "nan", "9170": "nan", "9175": "nan", "9180": "nan", "9185": "nan", "9190": "nan", "9195": "nan", "9200": 3.3843, "9205": "nan", "9210": "nan", "9215": "nan", "9220": "nan", "9225": "nan", "9230": "nan", "9235": "nan", "9240": "nan", "9245": "nan", "9250": "nan", "9255": "nan", "9260": "nan", "9265": "nan", "9270": "nan", "9275": "nan", "9280": "nan", "9285": "nan", "9290": "nan", "9295": "nan", "9300": 3.38399, "9305": "nan", "9310": "nan", "9315": "nan", "9320": "nan", "9325": "nan", "9330": "nan", "9335": "nan", "9340": "nan", "9345": "nan", "9350": "nan", "9355": "nan", "9360": "nan", "9365": "nan", "9370": "nan", "9375": "nan", "9380": "nan", "9385": "nan", "9390": "nan", "9395": "nan", "9400": 3.38416, "9405": "nan", "9410": "nan", "9415": "nan", "9420": "nan", "9425": "nan", "9430": "nan", "9435": "nan", "9440": "nan", "9445": "nan", "9450": "nan", "9455": "nan", "9460": "nan", "9465": "nan", "9470": "nan", "9475": "nan", "9480": "nan", "9485": "nan", "9490": "nan", "9495": "nan", "9500": 3.38376, "9505": "nan", "9510": "nan", "9515": "nan", "9520": "nan", "9525": "nan", "9530": "nan", "9535": "nan", "9540": "nan", "9545": "nan", "9550": "nan", "9555": "nan", "9560": "nan", "9565": "nan", "9570": "nan", "9575": "nan", "9580": "nan", "9585": "nan", "9590": "nan", "9595": "nan", "9600": 3.38327, "9605": "nan", "9610": "nan", "9615": "nan", "9620": "nan", "9625": "nan", "9630": "nan", "9635": "nan", "9640": "nan", "9645": "nan", "9650": "nan", "9655": "nan", "9660": "nan", "9665": "nan", "9670": "nan", "9675": "nan", "9680": "nan", "9685": "nan", "9690": "nan", "9695": "nan", "9700": 3.38432, "9705": "nan", "9710": "nan", "9715": "nan", "9720": "nan", "9725": "nan", "9730": "nan", "9735": "nan", "9740": "nan", "9745": "nan", "9750": "nan", "9755": "nan", "9760": "nan", "9765": "nan", "9770": "nan", "9775": "nan", "9780": "nan", "9785": "nan", "9790": "nan", "9795": "nan", "9800": 3.38427, "9805": "nan", "9810": "nan", "9815": "nan", "9820": "nan", "9825": "nan", "9830": "nan", "9835": "nan", "9840": "nan", "9845": "nan", "9850": "nan", "9855": "nan", "9860": "nan", "9865": "nan", "9870": "nan", "9875": "nan", "9880": "nan", "9885": "nan", "9890": "nan", "9895": "nan", "9900": 3.38415, "9905": "nan", "9910": "nan", "9915": "nan", "9920": "nan", "9925": "nan", "9930": "nan", "9935": "nan", "9940": "nan", "9945": "nan", "9950": "nan", "9955": "nan", "9960": "nan", "9965": "nan", "9970": "nan", "9975": "nan", "9980": "nan", "9985": "nan", "9990": "nan", "9995": "nan", "10000": 3.38426, "10005": "nan", "10010": "nan", "10015": "nan", "10020": "nan", "10025": "nan", "10030": "nan", "10035": "nan", "10040": "nan", "10045": "nan", "10050": "nan", "10055": "nan", "10060": "nan", "10065": "nan", "10070": "nan", "10075": "nan", "10080": "nan", "10085": "nan", "10090": "nan", "10095": "nan", "10100": 3.38377, "10105": "nan", "10110": "nan", "10115": "nan", "10120": "nan", "10125": "nan", "10130": "nan", "10135": "nan", "10140": "nan", "10145": "nan", "10150": "nan", "10155": "nan", "10160": "nan", "10165": "nan", "10170": "nan", "10175": "nan", "10180": "nan", "10185": "nan", "10190": "nan", "10195": "nan", "10200": 3.38456, "10205": "nan", "10210": "nan", "10215": "nan", "10220": "nan", "10225": "nan", "10230": "nan", "10235": "nan", "10240": "nan", "10245": "nan", "10250": "nan", "10255": "nan", "10260": "nan", "10265": "nan", "10270": "nan", "10275": "nan", "10280": "nan", "10285": "nan", "10290": "nan", "10295": "nan", "10300": 3.38487, "10305": "nan", "10310": "nan", "10315": "nan", "10320": "nan", "10325": "nan", "10330": "nan", "10335": "nan", "10340": "nan", "10345": "nan", "10350": "nan", "10355": "nan", "10360": "nan", "10365": "nan", "10370": "nan", "10375": "nan", "10380": "nan", "10385": "nan", "10390": "nan", "10395": "nan", "10400": 3.38413, "10405": "nan", "10410": "nan", "10415": "nan", "10420": "nan", "10425": "nan", "10430": "nan", "10435": "nan", "10440": "nan", "10445": "nan", "10450": "nan", "10455": "nan", "10460": "nan", "10465": "nan", "10470": "nan", "10475": "nan", "10480": "nan", "10485": "nan", "10490": "nan", "10495": "nan", "10500": 3.38411, "10505": "nan", "10510": "nan", "10515": "nan", "10520": "nan", "10525": "nan", "10530": "nan", "10535": "nan", "10540": "nan", "10545": "nan", "10550": "nan", "10555": "nan", "10560": "nan", "10565": "nan", "10570": "nan", "10575": "nan", "10580": "nan", "10585": "nan", "10590": "nan", "10595": "nan", "10600": 3.38408, "10605": "nan", "10610": "nan", "10615": "nan", "10620": "nan", "10625": "nan", "10630": "nan", "10635": "nan", "10640": "nan", "10645": "nan", "10650": "nan", "10655": "nan", "10660": "nan", "10665": "nan", "10670": "nan", "10675": "nan", "10680": "nan", "10685": "nan", "10690": "nan", "10695": "nan", "10700": 3.3839, "10705": "nan", "10710": "nan", "10715": "nan", "10720": "nan", "10725": "nan", "10730": "nan", "10735": "nan", "10740": "nan", "10745": "nan", "10750": "nan", "10755": "nan", "10760": "nan", "10765": "nan", "10770": "nan", "10775": "nan", "10780": "nan", "10785": "nan", "10790": "nan", "10795": "nan", "10800": 3.38381, "10805": "nan", "10810": "nan", "10815": "nan", "10820": "nan", "10825": "nan", "10830": "nan", "10835": "nan", "10840": "nan", "10845": "nan", "10850": "nan", "10855": "nan", "10860": "nan", "10865": "nan", "10870": "nan", "10875": "nan", "10880": "nan", "10885": "nan", "10890": "nan", "10895": "nan", "10900": 3.83861, "10905": "nan", "10910": "nan", "10915": "nan", "10920": "nan", "10925": "nan", "10930": "nan", "10935": "nan", "10940": "nan", "10945": "nan", "10950": "nan", "10955": "nan", "10960": "nan", "10965": "nan", "10970": "nan", "10975": "nan", "10980": "nan", "10985": "nan", "10990": "nan", "10995": "nan", "11000": 3.39347, "11005": "nan", "11010": "nan", "11015": "nan", "11020": "nan", "11025": "nan", "11030": "nan", "11035": "nan", "11040": "nan", "11045": "nan", "11050": "nan", "11055": "nan", "11060": "nan", "11065": "nan", "11070": "nan", "11075": "nan", "11080": "nan", "11085": "nan", "11090": "nan", "11095": "nan", "11100": 3.39189, "11105": "nan", "11110": "nan", "11115": "nan", "11120": "nan", "11125": "nan", "11130": "nan", "11135": "nan", "11140": "nan", "11145": "nan", "11150": "nan", "11155": "nan", "11160": "nan", "11165": "nan", "11170": "nan", "11175": "nan", "11180": "nan", "11185": "nan", "11190": "nan", "11195": "nan", "11200": 3.3918, "11205": "nan", "11210": "nan", "11215": "nan", "11220": "nan", "11225": "nan", "11230": "nan", "11235": "nan", "11240": "nan", "11245": "nan", "11250": "nan", "11255": "nan", "11260": "nan", "11265": "nan", "11270": "nan", "11275": "nan", "11280": "nan", "11285": "nan", "11290": "nan", "11295": "nan", "11300": 3.39238, "11305": "nan", "11310": "nan", "11315": "nan", "11320": "nan", "11325": "nan", "11330": "nan", "11335": "nan", "11340": "nan", "11345": "nan", "11350": "nan", "11355": "nan", "11360": "nan", "11365": "nan", "11370": "nan", "11375": "nan", "11380": "nan", "11385": "nan", "11390": "nan", "11395": "nan", "11400": 3.39244, "11405": "nan", "11410": "nan", "11415": "nan", "11420": "nan", "11425": "nan", "11430": "nan", "11435": "nan", "11440": "nan", "11445": "nan", "11450": "nan", "11455": "nan", "11460": "nan", "11465": "nan", "11470": "nan", "11475": "nan", "11480": "nan", "11485": "nan", "11490": "nan", "11495": "nan", "11500": 3.39253, "11505": "nan", "11510": "nan", "11515": "nan", "11520": "nan", "11525": "nan", "11530": "nan", "11535": "nan", "11540": "nan", "11545": "nan", "11550": "nan", "11555": "nan", "11560": "nan", "11565": "nan", "11570": "nan", "11575": "nan", "11580": "nan", "11585": "nan", "11590": "nan", "11595": "nan", "11600": 3.39379, "11605": "nan", "11610": "nan", "11615": "nan", "11620": "nan", "11625": "nan", "11630": "nan", "11635": "nan", "11640": "nan", "11645": "nan", "11650": "nan", "11655": "nan", "11660": "nan", "11665": "nan", "11670": "nan", "11675": "nan", "11680": "nan", "11685": "nan", "11690": "nan", "11695": "nan", "11700": 3.39314, "11705": "nan", "11710": "nan", "11715": "nan", "11720": "nan", "11725": "nan", "11730": "nan", "11735": "nan", "11740": "nan", "11745": "nan", "11750": "nan", "11755": "nan", "11760": "nan", "11765": "nan", "11770": "nan", "11775": "nan", "11780": "nan", "11785": "nan", "11790": "nan", "11795": "nan", "11800": 3.41793, "11805": "nan", "11810": "nan", "11815": "nan", "11820": "nan", "11825": "nan", "11830": "nan", "11835": "nan", "11840": "nan", "11845": "nan", "11850": "nan", "11855": "nan", "11860": "nan", "11865": "nan", "11870": "nan", "11875": "nan", "11880": "nan", "11885": "nan", "11890": "nan", "11895": "nan", "11900": 3.39269, "11905": "nan", "11910": "nan", "11915": "nan", "11920": "nan", "11925": "nan", "11930": "nan", "11935": "nan", "11940": "nan", "11945": "nan", "11950": "nan", "11955": "nan", "11960": "nan", "11965": "nan", "11970": "nan", "11975": "nan", "11980": "nan", "11985": "nan", "11990": "nan", "11995": "nan", "12000": 3.39317, "12005": "nan", "12010": "nan", "12015": "nan", "12020": "nan", "12025": "nan", "12030": "nan", "12035": "nan", "12040": "nan", "12045": "nan", "12050": "nan", "12055": "nan", "12060": "nan", "12065": "nan", "12070": "nan", "12075": "nan", "12080": "nan", "12085": "nan", "12090": "nan", "12095": "nan", "12100": 3.39129, "12105": "nan", "12110": "nan", "12115": "nan", "12120": "nan", "12125": "nan", "12130": "nan", "12135": "nan", "12140": "nan", "12145": "nan", "12150": "nan", "12155": "nan", "12160": "nan", "12165": "nan", "12170": "nan", "12175": "nan", "12180": "nan", "12185": "nan", "12190": "nan", "12195": "nan", "12200": 3.3923, "12205": "nan", "12210": "nan", "12215": "nan", "12220": "nan", "12225": "nan", "12230": "nan", "12235": "nan", "12240": "nan", "12245": "nan", "12250": "nan", "12255": "nan", "12260": "nan", "12265": "nan", "12270": "nan", "12275": "nan", "12280": "nan", "12285": "nan", "12290": "nan", "12295": "nan", "12300": 3.39197, "12305": "nan", "12310": "nan", "12315": "nan", "12320": "nan", "12325": "nan", "12330": "nan", "12335": "nan", "12340": "nan", "12345": "nan", "12350": "nan", "12355": "nan", "12360": "nan", "12365": "nan", "12370": "nan", "12375": "nan", "12380": "nan", "12385": "nan", "12390": "nan", "12395": "nan", "12400": 3.39164, "12405": "nan", "12410": "nan", "12415": "nan", "12420": "nan", "12425": "nan", "12430": "nan", "12435": "nan", "12440": "nan", "12445": "nan", "12450": "nan", "12455": "nan", "12460": "nan", "12465": "nan", "12470": "nan", "12475": "nan", "12480": "nan", "12485": "nan", "12490": "nan", "12495": "nan", "12500": 3.39073, "12505": "nan", "12510": "nan", "12515": "nan", "12520": "nan", "12525": "nan", "12530": "nan", "12535": "nan", "12540": "nan", "12545": "nan", "12550": "nan", "12555": "nan", "12560": "nan", "12565": "nan", "12570": "nan", "12575": "nan", "12580": "nan", "12585": "nan", "12590": "nan", "12595": "nan", "12600": 3.39126, "12605": "nan", "12610": "nan", "12615": "nan", "12620": "nan", "12625": "nan", "12630": "nan", "12635": "nan", "12640": "nan", "12645": "nan", "12650": "nan", "12655": "nan", "12660": "nan", "12665": "nan", "12670": "nan", "12675": "nan", "12680": "nan", "12685": "nan", "12690": "nan", "12695": "nan", "12700": 3.39109, "12705": "nan", "12710": "nan", "12715": "nan", "12720": "nan", "12725": "nan", "12730": "nan", "12735": "nan", "12740": "nan", "12745": "nan", "12750": "nan", "12755": "nan", "12760": "nan", "12765": "nan", "12770": "nan", "12775": "nan", "12780": "nan", "12785": "nan", "12790": "nan", "12795": "nan", "12800": "nan", "12805": "nan", "12810": "nan", "12815": "nan", "12820": "nan", "12825": "nan", "12830": "nan", "12835": "nan", "12840": "nan", "12845": "nan", "12850": "nan", "12855": "nan", "12860": "nan", "12865": "nan", "12870": "nan", "12875": "nan", "12880": "nan", "12885": "nan", "12890": "nan", "12895": "nan", "12900": "nan", "12905": "nan", "12910": "nan", "12915": "nan", "12920": "nan", "12925": "nan", "12930": "nan", "12935": "nan", "12940": "nan", "12945": "nan", "12950": "nan", "12955": "nan", "12960": "nan", "12965": "nan", "12970": "nan", "12975": "nan", "12980": "nan", "12985": "nan", "12990": "nan", "12995": "nan", "13000": "nan"}}}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm_gb200/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm_gb200/model_config.yaml
new file mode 100644
index 00000000000..b41e49d1185
--- /dev/null
+++ b/tests/functional_tests/test_cases/gpt/gpt3_15b_8t_release_sm_gb200/model_config.yaml
@@ -0,0 +1,102 @@
+ENV_VARS:
+ CUDA_DEVICE_MAX_CONNECTIONS: 1
+ NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1
+ NON_DETERMINSTIC_RESULTS: 1
+ NVTE_NORM_FWD_USE_CUDNN: 1
+ NVTE_NORM_BWD_USE_CUDNN: 1
+ NVTE_FUSED_ATTN: 1
+ PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
+ USE_MNNVL: 1
+TEST_TYPE: "release"
+MODEL_ARGS:
+ # Distributed args
+ --distributed-timeout-minutes: 60
+ --tensor-model-parallel-size: 4
+ --pipeline-model-parallel-size: 2
+ --num-layers-per-virtual-pipeline-stage: 8
+ --use-distributed-optimizer: true
+ --overlap-grad-reduce: true
+ --overlap-param-gather: true
+ --tp-comm-overlap: true
+ # Training args
+ --use-mcore-models: true
+ --sequence-parallel: true
+ --disable-bias-linear: true
+ --micro-batch-size: 4
+ --rampup-batch-size: "[384 384 97656250]"
+ --global-batch-size: 1152
+ --train-samples: 19531250
+ --manual-gc: true
+ --cross-entropy-loss-fusion: true
+ --cross-entropy-fusion-impl: te
+ # Transformer Engine args
+ --transformer-impl: transformer_engine
+ # Data args
+ --data-cache-path: ${DATA_CACHE_PATH}
+ --tokenizer-type: GPTSentencePieceTokenizer
+ --tokenizer-model: ${DATA_PATH}/utils/nemotron_2_256k.model
+ --data-path: $DATA_BLEND
+ --split: 99,1,0
+ --no-mmap-bin-files: true
+ --num-workers: 6
+ # Add network size args
+ --apply-layernorm-1p: true
+ --untie-embeddings-and-output-weights: true
+ --position-embedding-type: rope
+ --rotary-percent: 0.5
+ --squared-relu: true
+ --num-layers: 32
+ --hidden-size: 6144
+ --num-attention-heads: 48
+ --group-query-attention: true
+ --num-query-groups: 8
+ --seq-length: 4096
+ --max-position-embeddings: 4096
+ # Add regularization args
+ --attention-dropout: 0.0
+ --hidden-dropout: 0.0
+ --clip-grad: 1.0
+ --weight-decay: 0.1
+ # Add learning rate args
+ --lr-decay-samples: 1949218748
+ --lr-warmup-samples: 3906252
+ --lr: 4.5e-4
+ --min-lr: 4.5e-5
+ --decoupled-lr: 5.0e-4
+ --decoupled-min-lr: 4.5e-5
+ --lr-decay-style: cosine
+ --adam-beta1: 0.9
+ --adam-beta2: 0.95
+ # Add validation args
+ --eval-iters: 32
+ --eval-interval: 2000
+ # Add checkpointing args
+ --save: ${CHECKPOINT_SAVE_PATH}
+ --load: ${CHECKPOINT_LOAD_PATH}
+ --save-interval: 1000
+ --save-retain-interval: 5000
+ # Add initialization args
+ --init-method-std: 0.0134
+ # Add logging args
+ --log-timers-to-tensorboard: true
+ --log-memory-to-tensorboard: true
+ --log-num-zeros-in-grad: true
+ --log-params-norm: true
+ --log-validation-ppl-to-tensorboard: true
+ --log-throughput: true
+ --log-interval: 100
+ --tensorboard-dir: ${TENSORBOARD_PATH}
+ --wandb-project: megatron-core-release-runs
+ --wandb-entity: adlr
+ --wandb-exp-name: ${WANDB_EXPERIMENT}
+ # Add mixed precision args
+ --bf16: true
+ --exit-interval: 13000
+ --wandb-save-dir: ${WANDB_SAVE_PATH}
+ --async-save: true
+ --use-persistent-ckpt-worker: true
+METRICS:
+ - "iteration-time"
+ - "lm loss"
+ - "mem-allocated-bytes"
+ - "mem-max-allocated-bytes"
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_7b_tp1_pp4_memory_speed/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_7b_tp1_pp4_memory_speed/golden_values_dev_dgx_gb200.json
index 0174aaf4684..21b63e8effd 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_7b_tp1_pp4_memory_speed/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_7b_tp1_pp4_memory_speed/golden_values_dev_dgx_gb200.json
@@ -6,29 +6,29 @@
"values": {
"1": 12.59654,
"2": 12.60484,
- "3": 12.59799,
- "4": 12.59687,
- "5": 12.59285,
- "6": 12.59259,
- "7": 12.58011,
- "8": 12.54308,
- "9": 12.51049,
- "10": 12.49679,
- "11": 12.32875,
- "12": 12.29944,
- "13": 12.2346,
+ "3": 12.59797,
+ "4": 12.5969,
+ "5": 12.59289,
+ "6": 12.59265,
+ "7": 12.58015,
+ "8": 12.54318,
+ "9": 12.5105,
+ "10": 12.49672,
+ "11": 12.32881,
+ "12": 12.29939,
+ "13": 12.23473,
"14": 12.23325,
- "15": 11.81699,
- "16": 11.80131,
- "17": 11.76433,
- "18": 11.73986,
- "19": 11.6089,
- "20": 11.50642,
- "21": 11.26938,
- "22": 11.37967,
- "23": 11.288,
+ "15": 11.817,
+ "16": 11.80134,
+ "17": 11.76435,
+ "18": 11.73993,
+ "19": 11.60889,
+ "20": 11.50648,
+ "21": 11.26946,
+ "22": 11.37969,
+ "23": 11.28801,
"24": 11.16331,
- "25": 10.99891
+ "25": 10.9989
}
},
"num-zeros": {
@@ -36,31 +36,31 @@
"end_step": 25,
"step_interval": 1,
"values": {
- "1": 521037632.0,
- "2": 521666368.0,
- "3": 520934816.0,
- "4": 521227264.0,
- "5": 520996064.0,
- "6": 521371840.0,
- "7": 521420352.0,
- "8": 521057344.0,
- "9": 521461504.0,
- "10": 521178624.0,
- "11": 522279104.0,
- "12": 521439616.0,
- "13": 521475712.0,
- "14": 522445376.0,
- "15": 521592960.0,
- "16": 521416448.0,
- "17": 521026496.0,
- "18": 521277760.0,
- "19": 521154656.0,
- "20": 521134784.0,
- "21": 522907648.0,
- "22": 521590304.0,
- "23": 521352384.0,
- "24": 521424640.0,
- "25": 523543808.0
+ "1": 521038208.0,
+ "2": 521665504.0,
+ "3": 520934784.0,
+ "4": 521226912.0,
+ "5": 520995584.0,
+ "6": 521371136.0,
+ "7": 521420160.0,
+ "8": 521056672.0,
+ "9": 521461088.0,
+ "10": 521178048.0,
+ "11": 522280576.0,
+ "12": 521439168.0,
+ "13": 521475200.0,
+ "14": 522446240.0,
+ "15": 521590592.0,
+ "16": 521416064.0,
+ "17": 521027200.0,
+ "18": 521279360.0,
+ "19": 521153088.0,
+ "20": 521134144.0,
+ "21": 522908352.0,
+ "22": 521591200.0,
+ "23": 521351488.0,
+ "24": 521424000.0,
+ "25": 523543424.0
}
},
"mem-allocated-bytes": {
@@ -133,29 +133,29 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 3.93489,
+ "2": 10.37368,
"3": "nan",
- "4": 0.83885,
+ "4": 0.82471,
"5": "nan",
- "6": 0.86101,
+ "6": 0.81418,
"7": "nan",
- "8": 0.82617,
+ "8": 0.81547,
"9": "nan",
- "10": 0.8264,
+ "10": 0.82718,
"11": "nan",
- "12": 0.82456,
+ "12": 0.82851,
"13": "nan",
- "14": 0.82414,
+ "14": 0.81363,
"15": "nan",
- "16": 0.82604,
+ "16": 0.81374,
"17": "nan",
- "18": 0.83002,
+ "18": 0.81396,
"19": "nan",
- "20": 0.8234,
+ "20": 0.81346,
"21": "nan",
- "22": 0.82298,
+ "22": 0.81255,
"23": "nan",
- "24": 0.82311,
+ "24": 0.81353,
"25": "nan"
}
}
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_7b_tp1_pp4_memory_speed/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_7b_tp1_pp4_memory_speed/golden_values_dev_dgx_h100.json
index 2ed3bf0784f..1719b6e9578 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_7b_tp1_pp4_memory_speed/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_7b_tp1_pp4_memory_speed/golden_values_dev_dgx_h100.json
@@ -6,29 +6,29 @@
"values": {
"1": 12.58569,
"2": 12.58406,
- "3": 12.58486,
- "4": 12.58642,
- "5": 12.58279,
- "6": 12.57912,
- "7": 12.56177,
- "8": 12.52304,
- "9": 12.4966,
- "10": 12.4826,
- "11": 12.31462,
- "12": 12.272,
- "13": 12.20924,
- "14": 12.20094,
+ "3": 12.58488,
+ "4": 12.58632,
+ "5": 12.5828,
+ "6": 12.57908,
+ "7": 12.56192,
+ "8": 12.52305,
+ "9": 12.49668,
+ "10": 12.48245,
+ "11": 12.31444,
+ "12": 12.27191,
+ "13": 12.20938,
+ "14": 12.20089,
"15": 11.79651,
- "16": 11.78035,
- "17": 11.74188,
- "18": 11.71656,
- "19": 11.59074,
- "20": 11.47672,
- "21": 11.23784,
- "22": 11.3586,
- "23": 11.25768,
+ "16": 11.78043,
+ "17": 11.74179,
+ "18": 11.71657,
+ "19": 11.59068,
+ "20": 11.47674,
+ "21": 11.2379,
+ "22": 11.35857,
+ "23": 11.25769,
"24": 11.14081,
- "25": 10.97989
+ "25": 10.97993
}
},
"num-zeros": {
@@ -36,31 +36,31 @@
"end_step": 25,
"step_interval": 1,
"values": {
- "1": 521035392.0,
- "2": 521662624.0,
- "3": 520932992.0,
- "4": 521225120.0,
- "5": 520993600.0,
- "6": 521369824.0,
- "7": 521417344.0,
- "8": 521054784.0,
- "9": 521458592.0,
- "10": 521175520.0,
- "11": 522277376.0,
- "12": 521435904.0,
- "13": 521472640.0,
- "14": 522442496.0,
- "15": 521589568.0,
- "16": 521414080.0,
- "17": 521025696.0,
- "18": 521279168.0,
- "19": 521154400.0,
- "20": 521132352.0,
- "21": 522909696.0,
- "22": 521591904.0,
- "23": 521353504.0,
- "24": 521426496.0,
- "25": 523547008.0
+ "1": 521034848.0,
+ "2": 521662912.0,
+ "3": 520932320.0,
+ "4": 521225216.0,
+ "5": 520992768.0,
+ "6": 521369920.0,
+ "7": 521417152.0,
+ "8": 521055744.0,
+ "9": 521459008.0,
+ "10": 521175872.0,
+ "11": 522276992.0,
+ "12": 521435104.0,
+ "13": 521472960.0,
+ "14": 522443232.0,
+ "15": 521589504.0,
+ "16": 521413760.0,
+ "17": 521026112.0,
+ "18": 521279232.0,
+ "19": 521152640.0,
+ "20": 521132288.0,
+ "21": 522908864.0,
+ "22": 521591872.0,
+ "23": 521353024.0,
+ "24": 521427040.0,
+ "25": 523546112.0
}
},
"mem-allocated-bytes": {
@@ -101,30 +101,30 @@
"step_interval": 1,
"values": {
"1": 52730814464.0,
- "2": 60518424576.0,
- "3": 60518424576.0,
- "4": 60518424576.0,
- "5": 60518424576.0,
- "6": 60518424576.0,
- "7": 60518424576.0,
- "8": 60518424576.0,
- "9": 60518424576.0,
- "10": 60518424576.0,
- "11": 60518424576.0,
- "12": 60518424576.0,
- "13": 60518424576.0,
- "14": 60518424576.0,
- "15": 60518424576.0,
- "16": 60518424576.0,
- "17": 60518424576.0,
- "18": 60518424576.0,
- "19": 60518424576.0,
- "20": 60518424576.0,
- "21": 60518424576.0,
- "22": 60518424576.0,
- "23": 60518424576.0,
- "24": 60518424576.0,
- "25": 60518424576.0
+ "2": 60518313984.0,
+ "3": 60518313984.0,
+ "4": 60518313984.0,
+ "5": 60518313984.0,
+ "6": 60518313984.0,
+ "7": 60518313984.0,
+ "8": 60518313984.0,
+ "9": 60518313984.0,
+ "10": 60518313984.0,
+ "11": 60518313984.0,
+ "12": 60518313984.0,
+ "13": 60518313984.0,
+ "14": 60518313984.0,
+ "15": 60518313984.0,
+ "16": 60518313984.0,
+ "17": 60518313984.0,
+ "18": 60518313984.0,
+ "19": 60518313984.0,
+ "20": 60518313984.0,
+ "21": 60518313984.0,
+ "22": 60518313984.0,
+ "23": 60518313984.0,
+ "24": 60518313984.0,
+ "25": 60518313984.0
}
},
"iteration-time": {
@@ -133,29 +133,29 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 11.06832,
+ "2": 9.40588,
"3": "nan",
- "4": 1.16152,
+ "4": 1.14216,
"5": "nan",
- "6": 1.15069,
+ "6": 1.13426,
"7": "nan",
- "8": 1.15402,
+ "8": 1.13417,
"9": "nan",
- "10": 1.15412,
+ "10": 1.13556,
"11": "nan",
- "12": 1.15321,
+ "12": 1.13935,
"13": "nan",
- "14": 1.15624,
+ "14": 1.13873,
"15": "nan",
- "16": 1.1571,
+ "16": 1.13957,
"17": "nan",
- "18": 1.15577,
+ "18": 1.14035,
"19": "nan",
- "20": 1.15939,
+ "20": 1.13973,
"21": "nan",
- "22": 1.15675,
+ "22": 1.13936,
"23": "nan",
- "24": 1.15533,
+ "24": 1.13705,
"25": "nan"
}
}
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_7b_tp1_pp4_memory_speed/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_7b_tp1_pp4_memory_speed/model_config.yaml
index 52b7c68d384..e91d1105e83 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_7b_tp1_pp4_memory_speed/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_7b_tp1_pp4_memory_speed/model_config.yaml
@@ -58,6 +58,8 @@ MODEL_ARGS:
--bf16: true
--log-memory-to-tensorboard: true
--exit-interval: 25
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
METRICS:
- "iteration-time"
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_7b_tp4_pp1_memory_speed/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_7b_tp4_pp1_memory_speed/golden_values_dev_dgx_gb200.json
index ca51cd1bcb3..dc762c9b513 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_7b_tp4_pp1_memory_speed/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_7b_tp4_pp1_memory_speed/golden_values_dev_dgx_gb200.json
@@ -6,29 +6,29 @@
"values": {
"1": 12.61164,
"2": 12.60596,
- "3": 12.60278,
+ "3": 12.60284,
"4": 12.59692,
- "5": 12.5956,
- "6": 12.59777,
- "7": 12.58051,
- "8": 12.53845,
- "9": 12.51222,
- "10": 12.49859,
- "11": 12.32384,
- "12": 12.29418,
- "13": 12.23141,
- "14": 12.22824,
- "15": 11.82221,
- "16": 11.80412,
- "17": 11.76119,
- "18": 11.73708,
- "19": 11.61309,
+ "5": 12.59563,
+ "6": 12.59765,
+ "7": 12.58048,
+ "8": 12.53848,
+ "9": 12.51216,
+ "10": 12.4986,
+ "11": 12.32362,
+ "12": 12.29423,
+ "13": 12.23125,
+ "14": 12.22834,
+ "15": 11.82216,
+ "16": 11.80406,
+ "17": 11.76114,
+ "18": 11.7371,
+ "19": 11.61304,
"20": 11.50147,
- "21": 11.26475,
- "22": 11.37638,
- "23": 11.28398,
- "24": 11.1565,
- "25": 10.99865
+ "21": 11.26477,
+ "22": 11.37633,
+ "23": 11.28391,
+ "24": 11.15655,
+ "25": 10.99866
}
},
"num-zeros": {
@@ -36,31 +36,31 @@
"end_step": 25,
"step_interval": 1,
"values": {
- "1": 523049152.0,
- "2": 523677792.0,
- "3": 522947712.0,
- "4": 523241632.0,
- "5": 523021120.0,
- "6": 523374368.0,
- "7": 523437888.0,
- "8": 523083584.0,
- "9": 523470432.0,
- "10": 523196128.0,
- "11": 524297728.0,
- "12": 523455584.0,
- "13": 523501312.0,
- "14": 524479392.0,
- "15": 523634048.0,
- "16": 523462624.0,
- "17": 523079392.0,
- "18": 523360448.0,
- "19": 523209952.0,
- "20": 523228480.0,
- "21": 524938432.0,
- "22": 523660512.0,
- "23": 523415872.0,
- "24": 523485056.0,
- "25": 525638592.0
+ "1": 523050144.0,
+ "2": 523678816.0,
+ "3": 522945600.0,
+ "4": 523240640.0,
+ "5": 523021472.0,
+ "6": 523373120.0,
+ "7": 523437344.0,
+ "8": 523085504.0,
+ "9": 523469120.0,
+ "10": 523195520.0,
+ "11": 524297440.0,
+ "12": 523455616.0,
+ "13": 523497152.0,
+ "14": 524479520.0,
+ "15": 523635392.0,
+ "16": 523462432.0,
+ "17": 523079680.0,
+ "18": 523362816.0,
+ "19": 523208736.0,
+ "20": 523229056.0,
+ "21": 524937344.0,
+ "22": 523659200.0,
+ "23": 523415552.0,
+ "24": 523485568.0,
+ "25": 525640512.0
}
},
"mem-allocated-bytes": {
@@ -133,29 +133,29 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.88247,
+ "2": 5.70576,
"3": "nan",
- "4": 0.98359,
+ "4": 0.89304,
"5": "nan",
- "6": 0.91373,
+ "6": 0.89085,
"7": "nan",
- "8": 1.07044,
+ "8": 0.89054,
"9": "nan",
- "10": 0.91309,
+ "10": 0.88818,
"11": "nan",
- "12": 0.91579,
+ "12": 0.88741,
"13": "nan",
- "14": 0.90609,
+ "14": 0.88829,
"15": "nan",
- "16": 0.90906,
+ "16": 0.89204,
"17": "nan",
- "18": 0.91134,
+ "18": 0.8886,
"19": "nan",
- "20": 0.90623,
+ "20": 0.88626,
"21": "nan",
- "22": 0.91236,
+ "22": 0.8871,
"23": "nan",
- "24": 0.9145,
+ "24": 0.88991,
"25": "nan"
}
}
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_7b_tp4_pp1_memory_speed/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_7b_tp4_pp1_memory_speed/golden_values_dev_dgx_h100.json
index a05cc0a0778..c854a86514b 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_7b_tp4_pp1_memory_speed/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_7b_tp4_pp1_memory_speed/golden_values_dev_dgx_h100.json
@@ -4,31 +4,31 @@
"end_step": 25,
"step_interval": 1,
"values": {
- "1": 12.59715,
- "2": 12.60067,
- "3": 12.59727,
- "4": 12.60021,
- "5": 12.59013,
- "6": 12.58834,
- "7": 12.57605,
- "8": 12.5362,
- "9": 12.50745,
- "10": 12.49091,
- "11": 12.32614,
+ "1": 12.59719,
+ "2": 12.60069,
+ "3": 12.59724,
+ "4": 12.60016,
+ "5": 12.59009,
+ "6": 12.58836,
+ "7": 12.57598,
+ "8": 12.53622,
+ "9": 12.50748,
+ "10": 12.49077,
+ "11": 12.32622,
"12": 12.29366,
- "13": 12.22589,
- "14": 12.23023,
- "15": 11.82108,
- "16": 11.80586,
- "17": 11.77001,
- "18": 11.74946,
- "19": 11.62189,
- "20": 11.51704,
- "21": 11.27121,
- "22": 11.38966,
- "23": 11.29559,
- "24": 11.16591,
- "25": 11.00354
+ "13": 12.22602,
+ "14": 12.23012,
+ "15": 11.82095,
+ "16": 11.80585,
+ "17": 11.76996,
+ "18": 11.74947,
+ "19": 11.62191,
+ "20": 11.51706,
+ "21": 11.27116,
+ "22": 11.38967,
+ "23": 11.29564,
+ "24": 11.16596,
+ "25": 11.00353
}
},
"num-zeros": {
@@ -36,31 +36,31 @@
"end_step": 25,
"step_interval": 1,
"values": {
- "1": 523037536.0,
- "2": 523668064.0,
- "3": 522933056.0,
- "4": 523228480.0,
- "5": 523009792.0,
- "6": 523364320.0,
- "7": 523427840.0,
- "8": 523074688.0,
- "9": 523459232.0,
- "10": 523184992.0,
- "11": 524288736.0,
- "12": 523447712.0,
- "13": 523490112.0,
- "14": 524476096.0,
- "15": 523630496.0,
- "16": 523459232.0,
- "17": 523075936.0,
- "18": 523360192.0,
- "19": 523206816.0,
- "20": 523230848.0,
- "21": 524941248.0,
- "22": 523654464.0,
- "23": 523420576.0,
- "24": 523494720.0,
- "25": 525638016.0
+ "1": 523038112.0,
+ "2": 523667424.0,
+ "3": 522931840.0,
+ "4": 523230912.0,
+ "5": 523011200.0,
+ "6": 523366144.0,
+ "7": 523428576.0,
+ "8": 523073920.0,
+ "9": 523458656.0,
+ "10": 523184896.0,
+ "11": 524290432.0,
+ "12": 523448384.0,
+ "13": 523489472.0,
+ "14": 524476384.0,
+ "15": 523628288.0,
+ "16": 523460864.0,
+ "17": 523076928.0,
+ "18": 523361952.0,
+ "19": 523206976.0,
+ "20": 523228864.0,
+ "21": 524941632.0,
+ "22": 523658176.0,
+ "23": 523419360.0,
+ "24": 523494336.0,
+ "25": 525637664.0
}
},
"mem-allocated-bytes": {
@@ -133,29 +133,29 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 5.99154,
+ "2": 6.29925,
"3": "nan",
- "4": 1.10664,
+ "4": 1.10183,
"5": "nan",
- "6": 1.10108,
+ "6": 1.0959,
"7": "nan",
- "8": 1.09852,
+ "8": 1.09548,
"9": "nan",
- "10": 1.10395,
+ "10": 1.09665,
"11": "nan",
- "12": 1.13133,
+ "12": 1.09723,
"13": "nan",
- "14": 1.1009,
+ "14": 1.09908,
"15": "nan",
- "16": 1.10173,
+ "16": 1.09791,
"17": "nan",
- "18": 1.10058,
+ "18": 1.09734,
"19": "nan",
- "20": 1.10006,
+ "20": 1.0999,
"21": "nan",
- "22": 1.10081,
+ "22": 1.09871,
"23": "nan",
- "24": 1.09852,
+ "24": 1.09858,
"25": "nan"
}
}
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_7b_tp4_pp1_memory_speed/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_7b_tp4_pp1_memory_speed/model_config.yaml
index 3183afad2e0..96bc016ca97 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_7b_tp4_pp1_memory_speed/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_7b_tp4_pp1_memory_speed/model_config.yaml
@@ -56,6 +56,8 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
METRICS:
- "iteration-time"
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_disable/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_disable/model_config.yaml
index 2026f11ade2..7d946d05b0c 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_disable/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_disable/model_config.yaml
@@ -63,9 +63,9 @@ MODEL_ARGS:
--load: ${CHECKPOINT_LOAD_PATH}
# data settings
--data-cache-path: ${DATA_CACHE_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
# logging settings
--tensorboard-dir: ${TENSORBOARD_PATH}
--timing-log-level: 0
@@ -77,6 +77,8 @@ MODEL_ARGS:
--log-memory-to-tensorboard: true
# rerun settings
--rerun-mode: disabled
+ --async-save: true
+ --use-persistent-ckpt-worker: true
AFTER_SCRIPT: |
check_log() { if [[ -z $(grep -r $1 "$2" $LOG_DIR) ]]; then exit 1; else echo OK; fi }
check_log_not() { if [[ -z $(grep -r $1 "$2" $LOG_DIR) ]]; then echo OK; else exit 1; fi }
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_enable/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_enable/model_config.yaml
index 41cb6561429..ee557999f8e 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_enable/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_enable/model_config.yaml
@@ -62,9 +62,9 @@ MODEL_ARGS:
--load: ${CHECKPOINT_LOAD_PATH}
# data settings
--data-cache-path: ${DATA_CACHE_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
# logging settings
--tensorboard-dir: ${TENSORBOARD_PATH}
--timing-log-level: 0
@@ -76,6 +76,8 @@ MODEL_ARGS:
--log-memory-to-tensorboard: true
# rerun settings
--rerun-mode: validate_results
+ --async-save: true
+ --use-persistent-ckpt-worker: true
AFTER_SCRIPT: |
check_log() { if [[ -z $(grep -r $1 "$2" $LOG_DIR) ]]; then exit 1; else echo OK; fi }
check_log -F "WARNING:megatron.core.rerun_state_machine:Result validation enabled"
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_persistent_1/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_persistent_1/model_config.yaml
index cd590ff1554..daa5093f0a4 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_persistent_1/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_persistent_1/model_config.yaml
@@ -62,9 +62,9 @@ MODEL_ARGS:
--load: ${CHECKPOINT_LOAD_PATH}
# data settings
--data-cache-path: ${DATA_CACHE_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
# logging settings
--tensorboard-dir: ${TENSORBOARD_PATH}
--timing-log-level: 0
@@ -78,6 +78,8 @@ MODEL_ARGS:
--rerun-mode: validate_results
--error-injection-rate: 100
--error-injection-type: persistent_error
+ --async-save: true
+ --use-persistent-ckpt-worker: true
AFTER_SCRIPT: |
check_log() { if [[ -z $(grep -r $1 "$2" $LOG_DIR) ]]; then exit 1; else echo OK; fi }
check_log -F "Result validation enabled"
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_persistent_2/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_persistent_2/model_config.yaml
index f902393d049..d9e8a56f0de 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_persistent_2/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_persistent_2/model_config.yaml
@@ -62,9 +62,9 @@ MODEL_ARGS:
--load: ${CHECKPOINT_LOAD_PATH}
# data settings
--data-cache-path: ${DATA_CACHE_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
# logging settings
--tensorboard-dir: ${TENSORBOARD_PATH}
--timing-log-level: 0
@@ -76,6 +76,8 @@ MODEL_ARGS:
--log-memory-to-tensorboard: true
# rerun settings
--rerun-mode: validate_results
+ --async-save: true
+ --use-persistent-ckpt-worker: true
AFTER_SCRIPT: |
check_log() { if [[ -z $(grep -r $1 "$2" $LOG_DIR) ]]; then exit 1; else echo OK; fi }
check_log -F "WARNING:megatron.core.rerun_state_machine:Result validation enabled"
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_reshard/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_reshard/model_config.yaml
index 2e82cad10a8..7fbd18cfbe5 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_reshard/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_reshard/model_config.yaml
@@ -63,9 +63,9 @@ MODEL_ARGS:
--load: ${CHECKPOINT_LOAD_PATH}
# data settings
--data-cache-path: ${DATA_CACHE_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
# logging settings
--tensorboard-dir: ${TENSORBOARD_PATH}
--timing-log-level: 0
@@ -77,6 +77,8 @@ MODEL_ARGS:
--log-memory-to-tensorboard: true
# rerun settings
--rerun-mode: validate_results
+ --async-save: true
+ --use-persistent-ckpt-worker: true
AFTER_SCRIPT: |
check_log() { if [[ -z $(grep -r $1 "$2" $LOG_DIR) ]]; then exit 1; else echo OK; fi }
check_log -F "WARNING:megatron.core.rerun_state_machine:Result validation enabled"
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_resume/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_resume/model_config.yaml
index 0abd4db698e..f2516fd2d4b 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_resume/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_resume/model_config.yaml
@@ -62,9 +62,9 @@ MODEL_ARGS:
--load: ${CHECKPOINT_LOAD_PATH}
# data settings
--data-cache-path: ${DATA_CACHE_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
# logging settings
--tensorboard-dir: ${TENSORBOARD_PATH}
--timing-log-level: 0
@@ -76,6 +76,8 @@ MODEL_ARGS:
--log-memory-to-tensorboard: true
# rerun settings
--rerun-mode: validate_results
+ --async-save: true
+ --use-persistent-ckpt-worker: true
AFTER_SCRIPT: |
check_log() { if [[ -z $(grep -r $1 "$2" $LOG_DIR) ]]; then exit 1; else echo OK; fi }
check_log -F "successfully loaded checkpoint"
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_resume_check_grads/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_resume_check_grads/model_config.yaml
index aff4e8a6f3d..1a260774210 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_resume_check_grads/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_resume_check_grads/model_config.yaml
@@ -88,11 +88,13 @@ BASE_MODEL_ARGS: &BASE_MODEL_ARGS
--dist-ckpt-optim-fully-reshardable: true
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}/model/mcore_gpt/gpt3_4b_pyt/25.03.05_bf16_rerun-enabled_v2
+ --async-save: true
+ --use-persistent-ckpt-worker: true
# data settings
--data-cache-path: ${DATA_CACHE_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
# logging settings
--tensorboard-dir: ${TENSORBOARD_PATH}
--timing-log-level: 0
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_transient/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_transient/model_config.yaml
index b091c0ff9f6..47fd85b9d5c 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_transient/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_transient/model_config.yaml
@@ -63,9 +63,9 @@ MODEL_ARGS:
--exit-interval: 4
# data settings
--data-cache-path: ${DATA_CACHE_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
# logging settings
--tensorboard-dir: ${TENSORBOARD_PATH}
--timing-log-level: 0
@@ -79,6 +79,8 @@ MODEL_ARGS:
--rerun-mode: validate_results
--error-injection-rate: 100
--error-injection-type: transient_error
+ --async-save: true
+ --use-persistent-ckpt-worker: true
AFTER_SCRIPT: |
check_log() { if [[ -z $(grep -r $1 "$2" $LOG_DIR) ]]; then exit 1; else echo OK; fi }
check_log -F "WARNING:megatron.core.rerun_state_machine:Result validation enabled"
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset/golden_values_dev_dgx_gb200.json
index bac18297ae6..1597a12fc00 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset/golden_values_dev_dgx_gb200.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.77536,
+ "1": 10.77535,
"2": 10.78444,
- "3": 10.78593,
- "4": 10.7484,
- "5": 10.81554,
- "6": 10.82691,
- "7": 10.78469,
- "8": 10.77764,
+ "3": 10.78594,
+ "4": 10.74844,
+ "5": 10.81549,
+ "6": 10.82694,
+ "7": 10.78465,
+ "8": 10.77766,
"9": 10.78351,
"10": 10.74241,
- "11": 10.83031,
- "12": 10.80335,
- "13": 10.81653,
- "14": 10.82186,
- "15": 10.74223,
- "16": 10.75087,
- "17": 10.71888,
- "18": 10.74308,
- "19": 10.7407,
- "20": 10.63713,
- "21": 10.6277,
- "22": 10.48435,
- "23": 10.65701,
- "24": 10.52682,
- "25": 10.47546,
- "26": 10.54091,
- "27": 10.55554,
- "28": 10.52147,
+ "11": 10.8303,
+ "12": 10.80334,
+ "13": 10.81651,
+ "14": 10.82185,
+ "15": 10.7422,
+ "16": 10.75086,
+ "17": 10.71886,
+ "18": 10.74306,
+ "19": 10.74073,
+ "20": 10.63717,
+ "21": 10.62764,
+ "22": 10.48433,
+ "23": 10.657,
+ "24": 10.52681,
+ "25": 10.47547,
+ "26": 10.54093,
+ "27": 10.55549,
+ "28": 10.52151,
"29": 10.53465,
- "30": 10.30892,
- "31": 10.06663,
+ "30": 10.30894,
+ "31": 10.06666,
"32": 10.41746,
- "33": 10.42487,
- "34": 10.1739,
- "35": 10.22475,
- "36": 10.18282,
- "37": 10.29689,
+ "33": 10.42488,
+ "34": 10.17386,
+ "35": 10.2248,
+ "36": 10.18284,
+ "37": 10.29686,
"38": 10.14801,
"39": 10.36934,
- "40": 10.04004,
- "41": 10.10752,
- "42": 10.18198,
+ "40": 10.04006,
+ "41": 10.10749,
+ "42": 10.18199,
"43": 9.79649,
- "44": 9.91071,
- "45": 9.79715,
+ "44": 9.91069,
+ "45": 9.79712,
"46": 9.79411,
- "47": 10.11365,
- "48": 9.82516,
- "49": 9.50416,
- "50": 9.88698
+ "47": 10.11362,
+ "48": 9.82518,
+ "49": 9.50417,
+ "50": 9.887
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1625.0,
- "2": 1666.0,
- "3": 1695.0,
- "4": 1746.0,
- "5": 1977.0,
- "6": 1839.0,
- "7": 1894.0,
- "8": 1665.0,
- "9": 1929.0,
- "10": 1436.0,
- "11": 1794.0,
- "12": 1845.0,
- "13": 1976.0,
- "14": 1931.0,
- "15": 1971.0,
- "16": 2095.0,
- "17": 1805.0,
- "18": 1764.0,
- "19": 1753.0,
- "20": 1693.0,
- "21": 1872.0,
- "22": 1669.0,
- "23": 2113.0,
- "24": 1589.0,
- "25": 1679.0,
- "26": 1667.0,
- "27": 1779.0,
- "28": 2025.0,
- "29": 1940.0,
- "30": 1885.0,
- "31": 1623.0,
- "32": 1978.0,
- "33": 2203.0,
- "34": 1947.0,
- "35": 2040.0,
- "36": 2002.0,
- "37": 2346.0,
- "38": 2100.0,
- "39": 2479.0,
- "40": 2258.0,
- "41": 2347.0,
- "42": 2331.0,
- "43": 2125.0,
- "44": 2126.0,
- "45": 2130.0,
- "46": 2342.0,
- "47": 2550.0,
- "48": 2401.0,
- "49": 2216.0,
- "50": 2456.0
+ "1": 1597.0,
+ "2": 1627.0,
+ "3": 1645.0,
+ "4": 1760.0,
+ "5": 1863.0,
+ "6": 1829.0,
+ "7": 1908.0,
+ "8": 1642.0,
+ "9": 1885.0,
+ "10": 1444.0,
+ "11": 1844.0,
+ "12": 1833.0,
+ "13": 1903.0,
+ "14": 1906.0,
+ "15": 1984.0,
+ "16": 2015.0,
+ "17": 1820.0,
+ "18": 1763.0,
+ "19": 1723.0,
+ "20": 1681.0,
+ "21": 1875.0,
+ "22": 1677.0,
+ "23": 1981.0,
+ "24": 1571.0,
+ "25": 1588.0,
+ "26": 1659.0,
+ "27": 1732.0,
+ "28": 2035.0,
+ "29": 1977.0,
+ "30": 1932.0,
+ "31": 1579.0,
+ "32": 1890.0,
+ "33": 2186.0,
+ "34": 1984.0,
+ "35": 2000.0,
+ "36": 1941.0,
+ "37": 2334.0,
+ "38": 2132.0,
+ "39": 2533.0,
+ "40": 2156.0,
+ "41": 2318.0,
+ "42": 2339.0,
+ "43": 1998.0,
+ "44": 2097.0,
+ "45": 2178.0,
+ "46": 2287.0,
+ "47": 2409.0,
+ "48": 2319.0,
+ "49": 2104.0,
+ "50": 2433.0
}
},
"mem-allocated-bytes": {
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.46491,
- "3": 0.11014,
- "4": 0.09722,
- "5": 0.09703,
- "6": 0.09705,
- "7": 0.09593,
- "8": 0.09584,
- "9": 0.09505,
- "10": 0.0949,
- "11": 0.09504,
- "12": 0.09589,
- "13": 0.09506,
- "14": 0.09425,
- "15": 0.09404,
- "16": 0.09465,
- "17": 0.09237,
- "18": 0.09201,
- "19": 0.09159,
- "20": 0.09124,
- "21": 0.09092,
- "22": 0.09028,
- "23": 0.08966,
- "24": 0.08893,
- "25": 0.09042,
- "26": 0.09055,
- "27": 0.08889,
- "28": 0.08857,
- "29": 0.0884,
- "30": 0.08807,
- "31": 0.08777,
- "32": 0.08747,
- "33": 0.0876,
- "34": 0.08733,
- "35": 0.0886,
- "36": 0.08828,
- "37": 0.08789,
- "38": 0.08768,
- "39": 0.08819,
- "40": 0.08922,
- "41": 0.08797,
- "42": 0.0876,
- "43": 0.0868,
- "44": 0.08693,
- "45": 0.08661,
- "46": 0.08657,
- "47": 0.08769,
- "48": 0.08644,
- "49": 0.08681,
- "50": 0.08702
+ "2": 3.85031,
+ "3": 0.10579,
+ "4": 0.08811,
+ "5": 0.08828,
+ "6": 0.09023,
+ "7": 0.09011,
+ "8": 0.09009,
+ "9": 0.08883,
+ "10": 0.09032,
+ "11": 0.08909,
+ "12": 0.09028,
+ "13": 0.08981,
+ "14": 0.0908,
+ "15": 0.09035,
+ "16": 0.08825,
+ "17": 0.09092,
+ "18": 0.09041,
+ "19": 0.09012,
+ "20": 0.09006,
+ "21": 0.08995,
+ "22": 0.09051,
+ "23": 0.09078,
+ "24": 0.09133,
+ "25": 0.0906,
+ "26": 0.09043,
+ "27": 0.08991,
+ "28": 0.08972,
+ "29": 0.09046,
+ "30": 0.08921,
+ "31": 0.09085,
+ "32": 0.09076,
+ "33": 0.0898,
+ "34": 0.08988,
+ "35": 0.09085,
+ "36": 0.08951,
+ "37": 0.09036,
+ "38": 0.08966,
+ "39": 0.08995,
+ "40": 0.0898,
+ "41": 0.09082,
+ "42": 0.09019,
+ "43": 0.09295,
+ "44": 0.09078,
+ "45": 0.0912,
+ "46": 0.09208,
+ "47": 0.09077,
+ "48": 0.09093,
+ "49": 0.09052,
+ "50": 0.08959
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset/golden_values_dev_dgx_h100.json
index 036b53dabb1..90430e7bb49 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.89074,
- "2": 10.89234,
- "3": 10.89032,
- "4": 10.89221,
- "5": 10.89416,
- "6": 10.90226,
- "7": 10.8884,
- "8": 10.90211,
- "9": 10.90202,
- "10": 10.88512,
- "11": 10.87636,
- "12": 10.89499,
- "13": 10.89837,
- "14": 10.89182,
- "15": 10.85125,
- "16": 10.8534,
- "17": 10.82862,
- "18": 10.83653,
- "19": 10.82847,
- "20": 10.74583,
- "21": 10.73117,
- "22": 10.61256,
- "23": 10.72616,
- "24": 10.62932,
- "25": 10.59394,
- "26": 10.63357,
- "27": 10.63137,
- "28": 10.58201,
- "29": 10.58671,
- "30": 10.40936,
- "31": 10.15873,
- "32": 10.48319,
- "33": 10.46977,
- "34": 10.23978,
- "35": 10.28144,
- "36": 10.23894,
- "37": 10.35198,
- "38": 10.20565,
- "39": 10.40496,
- "40": 10.09271,
- "41": 10.16148,
- "42": 10.2231,
- "43": 9.84152,
- "44": 9.97329,
- "45": 9.84544,
- "46": 9.82102,
- "47": 10.14261,
- "48": 9.86553,
- "49": 9.54033,
- "50": 9.9169
+ "1": 10.93343,
+ "2": 10.92669,
+ "3": 10.92551,
+ "4": 10.92532,
+ "5": 10.93754,
+ "6": 10.92826,
+ "7": 10.92387,
+ "8": 10.92373,
+ "9": 10.92337,
+ "10": 10.92739,
+ "11": 10.9111,
+ "12": 10.92744,
+ "13": 10.89422,
+ "14": 10.89282,
+ "15": 10.8915,
+ "16": 10.87289,
+ "17": 10.86946,
+ "18": 10.86806,
+ "19": 10.85351,
+ "20": 10.80097,
+ "21": 10.7848,
+ "22": 10.7646,
+ "23": 10.76754,
+ "24": 10.75104,
+ "25": 10.74832,
+ "26": 10.71952,
+ "27": 10.68558,
+ "28": 10.61474,
+ "29": 10.58962,
+ "30": 10.5545,
+ "31": 10.56397,
+ "32": 10.54722,
+ "33": 10.51341,
+ "34": 10.48393,
+ "35": 10.4865,
+ "36": 10.46528,
+ "37": 10.42754,
+ "38": 10.42671,
+ "39": 10.39365,
+ "40": 10.37202,
+ "41": 10.35128,
+ "42": 10.3467,
+ "43": 10.32131,
+ "44": 10.30199,
+ "45": 10.30547,
+ "46": 10.26661,
+ "47": 10.25284,
+ "48": 10.2161,
+ "49": 10.20371,
+ "50": 10.21024
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1544.0,
- "2": 1729.0,
- "3": 1672.0,
- "4": 1807.0,
- "5": 1942.0,
- "6": 1736.0,
- "7": 1956.0,
- "8": 1716.0,
- "9": 2011.0,
- "10": 1385.0,
- "11": 1864.0,
- "12": 1767.0,
- "13": 2019.0,
- "14": 1787.0,
- "15": 1828.0,
- "16": 1908.0,
- "17": 1718.0,
- "18": 1602.0,
- "19": 1785.0,
- "20": 1679.0,
- "21": 1917.0,
- "22": 1712.0,
- "23": 2034.0,
- "24": 1752.0,
- "25": 1645.0,
- "26": 1820.0,
- "27": 1915.0,
- "28": 1996.0,
- "29": 2051.0,
- "30": 1890.0,
- "31": 1577.0,
- "32": 1886.0,
- "33": 2116.0,
- "34": 1912.0,
- "35": 2037.0,
- "36": 1924.0,
- "37": 2462.0,
- "38": 2241.0,
- "39": 2321.0,
- "40": 2221.0,
- "41": 2345.0,
- "42": 2386.0,
- "43": 2027.0,
- "44": 2211.0,
- "45": 2096.0,
- "46": 2285.0,
- "47": 2536.0,
- "48": 2289.0,
- "49": 2270.0,
- "50": 2421.0
+ "1": 1967.0,
+ "2": 1681.0,
+ "3": 1791.0,
+ "4": 1723.0,
+ "5": 1710.0,
+ "6": 1726.0,
+ "7": 2091.0,
+ "8": 1799.0,
+ "9": 1821.0,
+ "10": 1745.0,
+ "11": 1701.0,
+ "12": 1780.0,
+ "13": 1825.0,
+ "14": 1958.0,
+ "15": 1674.0,
+ "16": 1722.0,
+ "17": 1768.0,
+ "18": 1839.0,
+ "19": 1664.0,
+ "20": 1721.0,
+ "21": 1818.0,
+ "22": 1726.0,
+ "23": 1779.0,
+ "24": 1839.0,
+ "25": 1706.0,
+ "26": 1863.0,
+ "27": 1859.0,
+ "28": 1933.0,
+ "29": 2047.0,
+ "30": 1888.0,
+ "31": 2105.0,
+ "32": 2049.0,
+ "33": 2023.0,
+ "34": 2178.0,
+ "35": 2178.0,
+ "36": 2147.0,
+ "37": 2234.0,
+ "38": 2222.0,
+ "39": 2298.0,
+ "40": 2289.0,
+ "41": 2397.0,
+ "42": 2034.0,
+ "43": 2344.0,
+ "44": 2355.0,
+ "45": 2587.0,
+ "46": 2553.0,
+ "47": 2505.0,
+ "48": 2517.0,
+ "49": 2842.0,
+ "50": 2696.0
}
},
"mem-allocated-bytes": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 6.80579,
- "2": 0.08104,
- "3": 0.07547,
- "4": 0.05731,
- "5": 0.06226,
- "6": 0.05988,
- "7": 0.06566,
- "8": 0.06635,
- "9": 0.06593,
- "10": 0.06639,
- "11": 0.06591,
- "12": 0.06568,
- "13": 0.06504,
- "14": 0.06232,
- "15": 0.06162,
- "16": 0.05614,
- "17": 0.06083,
- "18": 0.05789,
- "19": 0.05867,
- "20": 0.05574,
- "21": 0.06043,
- "22": 0.05778,
- "23": 0.06166,
- "24": 0.05671,
- "25": 0.05765,
- "26": 0.05638,
- "27": 0.05601,
- "28": 0.05637,
- "29": 0.05497,
- "30": 0.05757,
- "31": 0.05556,
- "32": 0.05715,
- "33": 0.05761,
- "34": 0.05779,
- "35": 0.05996,
- "36": 0.05761,
- "37": 0.06454,
- "38": 0.0575,
- "39": 0.05802,
- "40": 0.05752,
- "41": 0.05904,
- "42": 0.05622,
- "43": 0.0555,
- "44": 0.05785,
- "45": 0.0578,
- "46": 0.05758,
- "47": 0.05729,
- "48": 0.05652,
- "49": 0.05619,
- "50": 0.05705
+ "1": "nan",
+ "2": 3.40372,
+ "3": 0.08131,
+ "4": 0.05699,
+ "5": 0.06412,
+ "6": 0.05797,
+ "7": 0.06213,
+ "8": 0.05676,
+ "9": 0.06079,
+ "10": 0.06363,
+ "11": 0.06483,
+ "12": 0.06023,
+ "13": 0.06524,
+ "14": 0.05665,
+ "15": 0.06255,
+ "16": 0.05651,
+ "17": 0.05918,
+ "18": 0.05645,
+ "19": 0.065,
+ "20": 0.0567,
+ "21": 0.06464,
+ "22": 0.05675,
+ "23": 0.06287,
+ "24": 0.0638,
+ "25": 0.06331,
+ "26": 0.06343,
+ "27": 0.06599,
+ "28": 0.05686,
+ "29": 0.06404,
+ "30": 0.05627,
+ "31": 0.06488,
+ "32": 0.05893,
+ "33": 0.06176,
+ "34": 0.0589,
+ "35": 0.05859,
+ "36": 0.05794,
+ "37": 0.05662,
+ "38": 0.05694,
+ "39": 0.05667,
+ "40": 0.05653,
+ "41": 0.05655,
+ "42": 0.05641,
+ "43": 0.05703,
+ "44": 0.05664,
+ "45": 0.05563,
+ "46": 0.05663,
+ "47": 0.05672,
+ "48": 0.05838,
+ "49": 0.06153,
+ "50": 0.05691
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset/model_config.yaml
index ddc8286573b..86039c4d7f2 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -53,4 +53,6 @@ MODEL_ARGS:
--fim-data: true
--fim-rate: 0.5
--fim-spm-rate: 0.5
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/golden_values_dev_dgx_gb200.json
index 8bcd3aa91d5..045c3d3b47a 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/golden_values_dev_dgx_gb200.json
@@ -7,52 +7,52 @@
"1": 10.77472,
"2": 10.7834,
"3": 10.783,
- "4": 10.74952,
- "5": 10.8207,
- "6": 10.8234,
- "7": 10.79076,
- "8": 10.78002,
- "9": 10.78621,
- "10": 10.74365,
+ "4": 10.74953,
+ "5": 10.82069,
+ "6": 10.82339,
+ "7": 10.79075,
+ "8": 10.78,
+ "9": 10.78617,
+ "10": 10.74367,
"11": 10.8322,
"12": 10.80441,
- "13": 10.8213,
- "14": 10.82574,
- "15": 10.74146,
- "16": 10.75035,
- "17": 10.72535,
- "18": 10.74231,
- "19": 10.7445,
- "20": 10.63706,
- "21": 10.63104,
- "22": 10.48032,
- "23": 10.65993,
- "24": 10.5253,
- "25": 10.47539,
- "26": 10.54133,
- "27": 10.5547,
- "28": 10.521,
+ "13": 10.82131,
+ "14": 10.82576,
+ "15": 10.74148,
+ "16": 10.75034,
+ "17": 10.72538,
+ "18": 10.74232,
+ "19": 10.74454,
+ "20": 10.63704,
+ "21": 10.63099,
+ "22": 10.48029,
+ "23": 10.65995,
+ "24": 10.52537,
+ "25": 10.47538,
+ "26": 10.54137,
+ "27": 10.55474,
+ "28": 10.52102,
"29": 10.53614,
- "30": 10.30519,
- "31": 10.06487,
- "32": 10.41559,
- "33": 10.42241,
- "34": 10.1741,
- "35": 10.22337,
- "36": 10.18522,
+ "30": 10.30518,
+ "31": 10.06489,
+ "32": 10.41554,
+ "33": 10.42245,
+ "34": 10.17407,
+ "35": 10.22339,
+ "36": 10.18526,
"37": 10.30398,
- "38": 10.14967,
+ "38": 10.14971,
"39": 10.37031,
- "40": 10.04015,
- "41": 10.10913,
- "42": 10.17951,
- "43": 9.79734,
+ "40": 10.04014,
+ "41": 10.10916,
+ "42": 10.17947,
+ "43": 9.79735,
"44": 9.90801,
- "45": 9.79837,
- "46": 9.79661,
- "47": 10.12063,
- "48": 9.82076,
- "49": 9.50507,
+ "45": 9.79833,
+ "46": 9.79662,
+ "47": 10.1206,
+ "48": 9.82074,
+ "49": 9.50511,
"50": 9.88047
}
},
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1603.0,
- "2": 1689.0,
- "3": 1616.0,
- "4": 1774.0,
- "5": 2059.0,
- "6": 1983.0,
- "7": 2102.0,
- "8": 1640.0,
- "9": 1877.0,
- "10": 1435.0,
- "11": 1981.0,
- "12": 1898.0,
- "13": 1949.0,
- "14": 1797.0,
- "15": 1923.0,
- "16": 1993.0,
- "17": 1804.0,
- "18": 1793.0,
- "19": 1808.0,
- "20": 1658.0,
- "21": 1881.0,
- "22": 1744.0,
- "23": 2029.0,
- "24": 1621.0,
- "25": 1550.0,
- "26": 1686.0,
- "27": 1794.0,
- "28": 1927.0,
- "29": 1974.0,
- "30": 1884.0,
- "31": 1610.0,
- "32": 1934.0,
- "33": 2098.0,
- "34": 1840.0,
- "35": 2033.0,
- "36": 2052.0,
- "37": 2302.0,
- "38": 2119.0,
- "39": 2421.0,
- "40": 2242.0,
- "41": 2339.0,
- "42": 2362.0,
- "43": 2065.0,
- "44": 2186.0,
- "45": 2266.0,
- "46": 2378.0,
- "47": 2504.0,
- "48": 2503.0,
- "49": 2303.0,
- "50": 2494.0
+ "1": 1542.0,
+ "2": 1772.0,
+ "3": 1677.0,
+ "4": 1763.0,
+ "5": 1987.0,
+ "6": 1880.0,
+ "7": 1948.0,
+ "8": 1686.0,
+ "9": 1930.0,
+ "10": 1437.0,
+ "11": 1928.0,
+ "12": 1829.0,
+ "13": 1980.0,
+ "14": 1810.0,
+ "15": 2006.0,
+ "16": 1885.0,
+ "17": 1765.0,
+ "18": 1742.0,
+ "19": 1788.0,
+ "20": 1717.0,
+ "21": 1880.0,
+ "22": 1707.0,
+ "23": 2116.0,
+ "24": 1644.0,
+ "25": 1581.0,
+ "26": 1664.0,
+ "27": 1810.0,
+ "28": 2052.0,
+ "29": 1954.0,
+ "30": 1943.0,
+ "31": 1590.0,
+ "32": 1906.0,
+ "33": 2122.0,
+ "34": 1865.0,
+ "35": 1994.0,
+ "36": 1902.0,
+ "37": 2380.0,
+ "38": 2161.0,
+ "39": 2414.0,
+ "40": 2260.0,
+ "41": 2308.0,
+ "42": 2275.0,
+ "43": 2109.0,
+ "44": 2189.0,
+ "45": 2236.0,
+ "46": 2437.0,
+ "47": 2581.0,
+ "48": 2351.0,
+ "49": 2345.0,
+ "50": 2524.0
}
},
"mem-allocated-bytes": {
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.34829,
- "3": 0.10511,
- "4": 0.09797,
- "5": 0.09705,
- "6": 0.09665,
- "7": 0.09616,
- "8": 0.09616,
- "9": 0.0968,
- "10": 0.09708,
- "11": 0.09598,
- "12": 0.09533,
- "13": 0.09549,
- "14": 0.09665,
- "15": 0.09303,
- "16": 0.0963,
- "17": 0.10058,
- "18": 0.09955,
- "19": 0.10067,
- "20": 0.10221,
- "21": 0.09941,
- "22": 0.09872,
- "23": 0.09975,
- "24": 0.10322,
- "25": 0.09837,
- "26": 0.09834,
- "27": 0.09843,
- "28": 0.09692,
- "29": 0.09907,
- "30": 0.09889,
- "31": 0.10064,
- "32": 0.09748,
- "33": 0.09927,
- "34": 0.09831,
- "35": 0.09862,
- "36": 0.09852,
- "37": 0.09869,
- "38": 0.09941,
- "39": 0.09945,
- "40": 0.10014,
- "41": 0.09934,
- "42": 0.10081,
- "43": 0.10148,
- "44": 0.09766,
- "45": 0.09746,
- "46": 0.09842,
- "47": 0.09924,
- "48": 0.09864,
- "49": 0.09829,
- "50": 0.09685
+ "2": 4.06847,
+ "3": 0.10126,
+ "4": 0.08879,
+ "5": 0.0895,
+ "6": 0.09253,
+ "7": 0.09257,
+ "8": 0.09092,
+ "9": 0.0912,
+ "10": 0.09343,
+ "11": 0.09132,
+ "12": 0.09098,
+ "13": 0.08924,
+ "14": 0.08868,
+ "15": 0.0917,
+ "16": 0.09022,
+ "17": 0.09175,
+ "18": 0.08931,
+ "19": 0.0903,
+ "20": 0.08975,
+ "21": 0.08914,
+ "22": 0.09136,
+ "23": 0.09031,
+ "24": 0.08986,
+ "25": 0.08928,
+ "26": 0.08905,
+ "27": 0.0893,
+ "28": 0.08978,
+ "29": 0.08991,
+ "30": 0.08929,
+ "31": 0.09073,
+ "32": 0.08895,
+ "33": 0.08888,
+ "34": 0.0889,
+ "35": 0.08867,
+ "36": 0.08814,
+ "37": 0.08834,
+ "38": 0.08834,
+ "39": 0.08804,
+ "40": 0.08849,
+ "41": 0.08911,
+ "42": 0.08844,
+ "43": 0.0897,
+ "44": 0.08853,
+ "45": 0.09005,
+ "46": 0.09453,
+ "47": 0.09155,
+ "48": 0.08894,
+ "49": 0.09025,
+ "50": 0.08973
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/golden_values_dev_dgx_h100.json
index 5718cc22850..65b36c76533 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.89592,
- "2": 10.89514,
- "3": 10.88761,
- "4": 10.88903,
- "5": 10.89131,
- "6": 10.90004,
- "7": 10.89143,
- "8": 10.89938,
- "9": 10.90231,
- "10": 10.88299,
- "11": 10.87827,
- "12": 10.89318,
- "13": 10.89818,
- "14": 10.89188,
- "15": 10.84786,
- "16": 10.85369,
- "17": 10.831,
- "18": 10.83994,
- "19": 10.82779,
- "20": 10.74925,
- "21": 10.73558,
- "22": 10.61567,
- "23": 10.72599,
- "24": 10.63027,
- "25": 10.59226,
- "26": 10.63312,
- "27": 10.63277,
- "28": 10.58231,
- "29": 10.58547,
- "30": 10.41136,
- "31": 10.15833,
- "32": 10.48326,
- "33": 10.46651,
- "34": 10.23801,
- "35": 10.28136,
- "36": 10.24029,
- "37": 10.3617,
- "38": 10.20342,
- "39": 10.404,
- "40": 10.09306,
- "41": 10.15805,
- "42": 10.21903,
- "43": 9.84274,
- "44": 9.97219,
- "45": 9.84149,
- "46": 9.82007,
- "47": 10.14934,
- "48": 9.85997,
- "49": 9.54155,
- "50": 9.91285
+ "1": 10.93399,
+ "2": 10.92476,
+ "3": 10.92682,
+ "4": 10.93022,
+ "5": 10.93698,
+ "6": 10.92965,
+ "7": 10.92253,
+ "8": 10.92212,
+ "9": 10.92457,
+ "10": 10.92887,
+ "11": 10.90984,
+ "12": 10.92854,
+ "13": 10.89636,
+ "14": 10.89646,
+ "15": 10.89151,
+ "16": 10.87387,
+ "17": 10.87034,
+ "18": 10.87171,
+ "19": 10.85327,
+ "20": 10.80495,
+ "21": 10.78369,
+ "22": 10.7641,
+ "23": 10.77106,
+ "24": 10.75054,
+ "25": 10.74644,
+ "26": 10.72229,
+ "27": 10.68746,
+ "28": 10.61507,
+ "29": 10.59232,
+ "30": 10.55595,
+ "31": 10.56461,
+ "32": 10.54469,
+ "33": 10.51566,
+ "34": 10.48262,
+ "35": 10.48464,
+ "36": 10.46506,
+ "37": 10.42808,
+ "38": 10.42575,
+ "39": 10.39426,
+ "40": 10.37399,
+ "41": 10.35094,
+ "42": 10.34772,
+ "43": 10.31921,
+ "44": 10.29862,
+ "45": 10.30319,
+ "46": 10.26406,
+ "47": 10.25192,
+ "48": 10.21502,
+ "49": 10.20173,
+ "50": 10.21029
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1581.0,
- "2": 1674.0,
- "3": 1724.0,
- "4": 1803.0,
- "5": 1962.0,
- "6": 1846.0,
- "7": 1864.0,
- "8": 1792.0,
- "9": 1848.0,
- "10": 1435.0,
- "11": 1868.0,
- "12": 1782.0,
- "13": 1874.0,
- "14": 1783.0,
- "15": 1944.0,
- "16": 1933.0,
- "17": 1807.0,
- "18": 1737.0,
- "19": 1822.0,
- "20": 1679.0,
- "21": 1808.0,
- "22": 1806.0,
- "23": 2077.0,
- "24": 1663.0,
- "25": 1645.0,
- "26": 1719.0,
- "27": 1925.0,
- "28": 2030.0,
- "29": 2042.0,
- "30": 1912.0,
- "31": 1603.0,
- "32": 1938.0,
- "33": 2158.0,
- "34": 1896.0,
- "35": 2023.0,
- "36": 1910.0,
- "37": 2330.0,
- "38": 2298.0,
- "39": 2498.0,
- "40": 2270.0,
- "41": 2464.0,
- "42": 2296.0,
- "43": 2042.0,
- "44": 2138.0,
- "45": 2152.0,
- "46": 2282.0,
- "47": 2529.0,
- "48": 2454.0,
- "49": 2358.0,
- "50": 2580.0
+ "1": 1881.0,
+ "2": 1708.0,
+ "3": 1776.0,
+ "4": 1728.0,
+ "5": 1710.0,
+ "6": 1765.0,
+ "7": 2082.0,
+ "8": 1791.0,
+ "9": 1899.0,
+ "10": 1705.0,
+ "11": 1734.0,
+ "12": 1770.0,
+ "13": 1899.0,
+ "14": 1951.0,
+ "15": 1709.0,
+ "16": 1758.0,
+ "17": 1820.0,
+ "18": 1822.0,
+ "19": 1665.0,
+ "20": 1751.0,
+ "21": 1884.0,
+ "22": 1740.0,
+ "23": 1732.0,
+ "24": 1856.0,
+ "25": 1700.0,
+ "26": 1864.0,
+ "27": 1881.0,
+ "28": 1877.0,
+ "29": 2015.0,
+ "30": 1949.0,
+ "31": 2007.0,
+ "32": 1996.0,
+ "33": 2003.0,
+ "34": 2092.0,
+ "35": 2147.0,
+ "36": 2077.0,
+ "37": 2329.0,
+ "38": 2112.0,
+ "39": 2253.0,
+ "40": 2219.0,
+ "41": 2432.0,
+ "42": 2115.0,
+ "43": 2384.0,
+ "44": 2287.0,
+ "45": 2608.0,
+ "46": 2434.0,
+ "47": 2518.0,
+ "48": 2549.0,
+ "49": 2788.0,
+ "50": 2655.0
}
},
"mem-allocated-bytes": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 8.63401,
- "2": 0.09023,
- "3": 0.07348,
- "4": 0.05746,
- "5": 0.05663,
- "6": 0.05755,
- "7": 0.0574,
- "8": 0.05838,
- "9": 0.05585,
- "10": 0.05739,
- "11": 0.05576,
- "12": 0.0561,
- "13": 0.05582,
- "14": 0.05815,
- "15": 0.05615,
- "16": 0.05649,
- "17": 0.05732,
- "18": 0.05614,
- "19": 0.05614,
- "20": 0.0565,
- "21": 0.05624,
- "22": 0.05712,
- "23": 0.05601,
- "24": 0.05772,
- "25": 0.05612,
- "26": 0.05714,
- "27": 0.05571,
- "28": 0.05803,
- "29": 0.0562,
- "30": 0.05628,
- "31": 0.05602,
- "32": 0.05667,
- "33": 0.05631,
- "34": 0.05631,
- "35": 0.05623,
- "36": 0.0565,
- "37": 0.05737,
- "38": 0.05733,
- "39": 0.05988,
- "40": 0.05739,
- "41": 0.05719,
- "42": 0.05699,
- "43": 0.05608,
- "44": 0.05867,
- "45": 0.05838,
- "46": 0.05842,
- "47": 0.05635,
- "48": 0.05732,
- "49": 0.0569,
- "50": 0.05736
+ "1": "nan",
+ "2": 3.37692,
+ "3": 0.07393,
+ "4": 0.05728,
+ "5": 0.05729,
+ "6": 0.05705,
+ "7": 0.0579,
+ "8": 0.05799,
+ "9": 0.05687,
+ "10": 0.05736,
+ "11": 0.05674,
+ "12": 0.05645,
+ "13": 0.05676,
+ "14": 0.05779,
+ "15": 0.05635,
+ "16": 0.05659,
+ "17": 0.05629,
+ "18": 0.0568,
+ "19": 0.05642,
+ "20": 0.05691,
+ "21": 0.05766,
+ "22": 0.05668,
+ "23": 0.05613,
+ "24": 0.05638,
+ "25": 0.05602,
+ "26": 0.05761,
+ "27": 0.05639,
+ "28": 0.05699,
+ "29": 0.05616,
+ "30": 0.05659,
+ "31": 0.05711,
+ "32": 0.05649,
+ "33": 0.05617,
+ "34": 0.0569,
+ "35": 0.05652,
+ "36": 0.0569,
+ "37": 0.05632,
+ "38": 0.05648,
+ "39": 0.0565,
+ "40": 0.05649,
+ "41": 0.0565,
+ "42": 0.05735,
+ "43": 0.05649,
+ "44": 0.05736,
+ "45": 0.05689,
+ "46": 0.05671,
+ "47": 0.05806,
+ "48": 0.05641,
+ "49": 0.05733,
+ "50": 0.05712
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/model_config.yaml
index 2d5e340fa6d..6a90ba0f943 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_frozen_resume_torch_dist_dist_optimizer/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_frozen_resume_torch_dist_dist_optimizer/model_config.yaml
index fc92d226b6d..8dae22b8852 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_frozen_resume_torch_dist_dist_optimizer/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_frozen_resume_torch_dist_dist_optimizer/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,5 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: frozen-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_fp8_no_model_parallel/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_mup/model_config.yaml
similarity index 65%
rename from tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_fp8_no_model_parallel/model_config.yaml
rename to tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_mup/model_config.yaml
index 12063418adf..ff2da3180fc 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_fp8_no_model_parallel/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_mup/model_config.yaml
@@ -1,7 +1,6 @@
ENV_VARS:
CUDA_DEVICE_MAX_CONNECTIONS: 1
NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
- NVTE_FUSED_ATTN: 0
NCCL_ALGO: Ring
CUBLAS_WORKSPACE_CONFIG: :4096:8
MODEL_ARGS:
@@ -13,18 +12,18 @@ MODEL_ARGS:
--log-validation-ppl-to-tensorboard: true
--log-timers-to-tensorboard: true
--tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
+ --micro-batch-size: 4
+ --global-batch-size: 32
--seq-length: 1024
--max-position-embeddings: 1024
- --train-iters: 2000
+ --train-iters: 50
--timing-log-level: 0
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -40,16 +39,21 @@ MODEL_ARGS:
--transformer-impl: transformer_engine
--tensor-model-parallel-size: 1
--pipeline-model-parallel-size: 1
+ --use-distributed-optimizer: true
+ --no-mmap-bin-files: true
--deterministic-mode: true
--no-gradient-accumulation-fusion: true
- --fp8-format: hybrid
- --fp8-amax-history-len: 1024
- --fp8-amax-compute-algo: max
--attention-softmax-in-fp32: true
+ --use-mcore-models: true
--ckpt-format: torch_dist
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
+ --dist-ckpt-strictness: log_all
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
+ --attention-backend: unfused
--log-memory-to-tensorboard: true
- --legacy-tokenizer: true
+ # MuP (Maximal Update Parameterization) args
+ --use-mup: true
+ --mup-base-hidden-size: 256
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_gb200.json
index eb5a06ac1fc..e0e49eea9e5 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_gb200.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.77447,
- "2": 10.78365,
- "3": 10.78346,
- "4": 10.74822,
- "5": 10.81983,
- "6": 10.82303,
- "7": 10.79055,
- "8": 10.77956,
- "9": 10.78595,
- "10": 10.74453,
- "11": 10.83267,
- "12": 10.80426,
- "13": 10.82082,
- "14": 10.82567,
- "15": 10.74206,
- "16": 10.74904,
- "17": 10.7252,
- "18": 10.74176,
- "19": 10.74412,
- "20": 10.63678,
- "21": 10.63055,
- "22": 10.47962,
- "23": 10.65976,
- "24": 10.52477,
- "25": 10.47552,
- "26": 10.54117,
- "27": 10.55491,
- "28": 10.52139,
- "29": 10.536,
- "30": 10.3053,
- "31": 10.0644,
- "32": 10.41569,
- "33": 10.42199,
- "34": 10.17393,
- "35": 10.22403,
- "36": 10.18498,
- "37": 10.30417,
- "38": 10.14995,
- "39": 10.37042,
- "40": 10.03994,
- "41": 10.10953,
- "42": 10.17937,
- "43": 9.79747,
- "44": 9.90812,
- "45": 9.79809,
- "46": 9.7966,
- "47": 10.12109,
- "48": 9.82083,
- "49": 9.50495,
- "50": 9.88025,
- "51": 9.83614,
- "52": 9.72315,
- "53": 10.05318,
- "54": 9.93747,
- "55": 9.87384,
- "56": 9.60449,
- "57": 9.4523,
- "58": 9.8188,
- "59": 9.5772,
- "60": 9.48534,
- "61": 9.68548,
- "62": 9.97906,
- "63": 9.36419,
- "64": 9.76203,
- "65": 8.94097,
- "66": 9.69475,
- "67": 9.36656,
- "68": 9.77745,
- "69": 9.79001,
- "70": 9.72374,
- "71": 9.62037,
- "72": 9.57423,
- "73": 9.48575,
- "74": 8.92729,
- "75": 9.41651,
- "76": 9.07747,
- "77": 10.05444,
- "78": 9.71914,
- "79": 9.37306,
- "80": 9.40003,
- "81": 9.47844,
- "82": 9.69867,
- "83": 9.31155,
- "84": 9.41457,
- "85": 9.61163,
- "86": 9.07418,
- "87": 9.5939,
- "88": 9.74928,
- "89": 9.5985,
- "90": 9.82761,
- "91": 9.33631,
- "92": 9.35805,
- "93": 9.08552,
- "94": 8.82786,
- "95": 9.5303,
- "96": 9.52663,
- "97": 9.30483,
- "98": 9.67007,
- "99": 8.89606,
- "100": 9.40702
+ "1": 10.88423,
+ "2": 10.88416,
+ "3": 10.88528,
+ "4": 10.88261,
+ "5": 10.87548,
+ "6": 10.87734,
+ "7": 10.879,
+ "8": 10.87645,
+ "9": 10.88486,
+ "10": 10.87621,
+ "11": 10.87476,
+ "12": 10.86386,
+ "13": 10.87812,
+ "14": 10.86681,
+ "15": 10.83808,
+ "16": 10.82848,
+ "17": 10.84395,
+ "18": 10.82617,
+ "19": 10.83047,
+ "20": 10.73064,
+ "21": 10.73759,
+ "22": 10.72457,
+ "23": 10.71463,
+ "24": 10.68917,
+ "25": 10.67538,
+ "26": 10.6774,
+ "27": 10.63676,
+ "28": 10.58126,
+ "29": 10.54374,
+ "30": 10.52133,
+ "31": 10.5163,
+ "32": 10.49665,
+ "33": 10.46954,
+ "34": 10.43405,
+ "35": 10.44016,
+ "36": 10.41287,
+ "37": 10.37738,
+ "38": 10.39311,
+ "39": 10.35163,
+ "40": 10.34759,
+ "41": 10.3237,
+ "42": 10.29736,
+ "43": 10.28799,
+ "44": 10.24486,
+ "45": 10.27234,
+ "46": 10.22855,
+ "47": 10.22098,
+ "48": 10.17294,
+ "49": 10.17269,
+ "50": 10.17278,
+ "51": 10.17699,
+ "52": 10.12977,
+ "53": 10.1403,
+ "54": 10.10481,
+ "55": 10.07099,
+ "56": 10.10607,
+ "57": 10.09841,
+ "58": 10.11271,
+ "59": 10.05585,
+ "60": 10.0721,
+ "61": 10.0265,
+ "62": 9.99707,
+ "63": 10.0708,
+ "64": 10.02752,
+ "65": 10.00261,
+ "66": 10.02862,
+ "67": 10.00298,
+ "68": 9.96497,
+ "69": 9.98892,
+ "70": 9.97585,
+ "71": 9.99935,
+ "72": 9.97694,
+ "73": 9.96491,
+ "74": 9.95752,
+ "75": 9.92626,
+ "76": 9.9652,
+ "77": 9.95721,
+ "78": 9.90484,
+ "79": 9.91039,
+ "80": 9.92921,
+ "81": 9.9448,
+ "82": 9.88814,
+ "83": 9.84981,
+ "84": 9.78546,
+ "85": 9.77733,
+ "86": 9.87461,
+ "87": 9.9087,
+ "88": 9.88617,
+ "89": 9.81989,
+ "90": 9.81011,
+ "91": 9.82231,
+ "92": 9.81413,
+ "93": 9.75326,
+ "94": 9.8254,
+ "95": 9.8214,
+ "96": 9.80428,
+ "97": 9.74114,
+ "98": 9.77482,
+ "99": 9.81969,
+ "100": 9.71392
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1531.0,
- "2": 1722.0,
- "3": 1589.0,
- "4": 1870.0,
- "5": 1992.0,
- "6": 1894.0,
- "7": 1954.0,
- "8": 1697.0,
- "9": 1855.0,
- "10": 1477.0,
- "11": 1889.0,
- "12": 1848.0,
- "13": 1973.0,
- "14": 1877.0,
- "15": 2015.0,
- "16": 1943.0,
- "17": 1772.0,
- "18": 1764.0,
- "19": 1782.0,
- "20": 1678.0,
- "21": 1906.0,
- "22": 1738.0,
- "23": 2057.0,
- "24": 1597.0,
- "25": 1567.0,
- "26": 1762.0,
- "27": 1932.0,
- "28": 1987.0,
- "29": 1936.0,
- "30": 1965.0,
- "31": 1554.0,
- "32": 1846.0,
- "33": 2148.0,
- "34": 1872.0,
- "35": 1985.0,
- "36": 1906.0,
- "37": 2245.0,
- "38": 2119.0,
- "39": 2495.0,
- "40": 2274.0,
- "41": 2236.0,
- "42": 2318.0,
- "43": 2068.0,
- "44": 2120.0,
- "45": 2265.0,
- "46": 2447.0,
- "47": 2584.0,
- "48": 2296.0,
- "49": 2252.0,
- "50": 2568.0,
- "51": 2650.0,
- "52": 2700.0,
- "53": 2863.0,
- "54": 2676.0,
- "55": 2390.0,
- "56": 2753.0,
- "57": 2430.0,
- "58": 2919.0,
- "59": 2831.0,
- "60": 2428.0,
- "61": 2932.0,
- "62": 2724.0,
- "63": 2579.0,
- "64": 2987.0,
- "65": 2506.0,
- "66": 2886.0,
- "67": 2871.0,
- "68": 2870.0,
- "69": 3001.0,
- "70": 3294.0,
- "71": 3043.0,
- "72": 2614.0,
- "73": 3054.0,
- "74": 2024.0,
- "75": 2507.0,
- "76": 3020.0,
- "77": 3253.0,
+ "1": 1843.0,
+ "2": 1668.0,
+ "3": 1804.0,
+ "4": 1839.0,
+ "5": 1800.0,
+ "6": 1703.0,
+ "7": 1950.0,
+ "8": 1670.0,
+ "9": 1881.0,
+ "10": 1793.0,
+ "11": 1706.0,
+ "12": 1754.0,
+ "13": 1779.0,
+ "14": 1882.0,
+ "15": 1688.0,
+ "16": 1732.0,
+ "17": 1900.0,
+ "18": 1809.0,
+ "19": 1708.0,
+ "20": 1778.0,
+ "21": 1828.0,
+ "22": 1721.0,
+ "23": 1782.0,
+ "24": 1909.0,
+ "25": 1761.0,
+ "26": 1903.0,
+ "27": 1951.0,
+ "28": 1855.0,
+ "29": 1964.0,
+ "30": 1873.0,
+ "31": 2052.0,
+ "32": 2061.0,
+ "33": 2011.0,
+ "34": 2157.0,
+ "35": 2129.0,
+ "36": 2033.0,
+ "37": 2385.0,
+ "38": 2156.0,
+ "39": 2262.0,
+ "40": 2421.0,
+ "41": 2477.0,
+ "42": 2070.0,
+ "43": 2443.0,
+ "44": 2274.0,
+ "45": 2751.0,
+ "46": 2516.0,
+ "47": 2603.0,
+ "48": 2637.0,
+ "49": 2960.0,
+ "50": 2753.0,
+ "51": 2666.0,
+ "52": 2841.0,
+ "53": 2716.0,
+ "54": 2956.0,
+ "55": 2733.0,
+ "56": 2928.0,
+ "57": 2406.0,
+ "58": 3892.0,
+ "59": 2968.0,
+ "60": 3037.0,
+ "61": 3035.0,
+ "62": 3439.0,
+ "63": 3493.0,
+ "64": 3889.0,
+ "65": 2772.0,
+ "66": 3290.0,
+ "67": 3964.0,
+ "68": 3668.0,
+ "69": 2992.0,
+ "70": 3513.0,
+ "71": 3117.0,
+ "72": 3085.0,
+ "73": 3609.0,
+ "74": 3439.0,
+ "75": 3295.0,
+ "76": 3289.0,
+ "77": 3733.0,
"78": 3230.0,
- "79": 3210.0,
- "80": 3252.0,
- "81": 3614.0,
- "82": 3395.0,
- "83": 2919.0,
- "84": 3296.0,
- "85": 3320.0,
- "86": 2865.0,
- "87": 3931.0,
- "88": 3240.0,
- "89": 3428.0,
- "90": 3127.0,
- "91": 2815.0,
- "92": 3098.0,
- "93": 2796.0,
- "94": 3324.0,
- "95": 3428.0,
- "96": 3541.0,
- "97": 3216.0,
- "98": 3705.0,
- "99": 3184.0,
- "100": 3073.0
+ "79": 3224.0,
+ "80": 3050.0,
+ "81": 3461.0,
+ "82": 2960.0,
+ "83": 3203.0,
+ "84": 3079.0,
+ "85": 2906.0,
+ "86": 3272.0,
+ "87": 2976.0,
+ "88": 3152.0,
+ "89": 3078.0,
+ "90": 3459.0,
+ "91": 2910.0,
+ "92": 3201.0,
+ "93": 3262.0,
+ "94": 3189.0,
+ "95": 3631.0,
+ "96": 3396.0,
+ "97": 3594.0,
+ "98": 3555.0,
+ "99": 3349.0,
+ "100": 3390.0
}
},
"mem-allocated-bytes": {
@@ -326,105 +326,105 @@
"step_interval": 1,
"values": {
"1": 2644459008.0,
- "2": 2741075456.0,
- "3": 2741075456.0,
- "4": 2741075456.0,
- "5": 2741075456.0,
- "6": 2741075456.0,
- "7": 2741075456.0,
- "8": 2741075456.0,
- "9": 2741075456.0,
- "10": 2741075456.0,
- "11": 2741075456.0,
- "12": 2741075456.0,
- "13": 2741075456.0,
- "14": 2741075456.0,
- "15": 2741075456.0,
- "16": 2741075456.0,
- "17": 2741075456.0,
- "18": 2741075456.0,
- "19": 2741075456.0,
- "20": 2741075456.0,
- "21": 2741075456.0,
- "22": 2741075456.0,
- "23": 2741075456.0,
- "24": 2741075456.0,
- "25": 2741075456.0,
- "26": 2741075456.0,
- "27": 2741075456.0,
- "28": 2741075456.0,
- "29": 2741075456.0,
- "30": 2741075456.0,
- "31": 2741075456.0,
- "32": 2741075456.0,
- "33": 2741075456.0,
- "34": 2741075456.0,
- "35": 2741075456.0,
- "36": 2741075456.0,
- "37": 2741075456.0,
- "38": 2741075456.0,
- "39": 2741075456.0,
- "40": 2741075456.0,
- "41": 2741075456.0,
- "42": 2741075456.0,
- "43": 2741075456.0,
- "44": 2741075456.0,
- "45": 2741075456.0,
- "46": 2741075456.0,
- "47": 2741075456.0,
- "48": 2741075456.0,
- "49": 2741075456.0,
- "50": 2741075456.0,
- "51": 2741075456.0,
- "52": 2741075456.0,
- "53": 2741075456.0,
- "54": 2741075456.0,
- "55": 2741075456.0,
- "56": 2741075456.0,
- "57": 2741075456.0,
- "58": 2741075456.0,
- "59": 2741075456.0,
- "60": 2741075456.0,
- "61": 2741075456.0,
- "62": 2741075456.0,
- "63": 2741075456.0,
- "64": 2741075456.0,
- "65": 2741075456.0,
- "66": 2741075456.0,
- "67": 2741075456.0,
- "68": 2741075456.0,
- "69": 2741075456.0,
- "70": 2741075456.0,
- "71": 2741075456.0,
- "72": 2741075456.0,
- "73": 2741075456.0,
- "74": 2741075456.0,
- "75": 2741075456.0,
- "76": 2741075456.0,
- "77": 2741075456.0,
- "78": 2741075456.0,
- "79": 2741075456.0,
- "80": 2741075456.0,
- "81": 2741075456.0,
- "82": 2741075456.0,
- "83": 2741075456.0,
- "84": 2741075456.0,
- "85": 2741075456.0,
- "86": 2741075456.0,
- "87": 2741075456.0,
- "88": 2741075456.0,
- "89": 2741075456.0,
- "90": 2741075456.0,
- "91": 2741075456.0,
- "92": 2741075456.0,
- "93": 2741075456.0,
- "94": 2741075456.0,
- "95": 2741075456.0,
- "96": 2741075456.0,
- "97": 2741075456.0,
- "98": 2741075456.0,
- "99": 2741075456.0,
- "100": 2741075456.0
+ "2": 2741977600.0,
+ "3": 2741977600.0,
+ "4": 2741977600.0,
+ "5": 2741977600.0,
+ "6": 2741977600.0,
+ "7": 2741977600.0,
+ "8": 2741977600.0,
+ "9": 2741977600.0,
+ "10": 2741977600.0,
+ "11": 2741977600.0,
+ "12": 2741977600.0,
+ "13": 2741977600.0,
+ "14": 2741977600.0,
+ "15": 2741977600.0,
+ "16": 2741977600.0,
+ "17": 2741977600.0,
+ "18": 2741977600.0,
+ "19": 2741977600.0,
+ "20": 2741977600.0,
+ "21": 2741977600.0,
+ "22": 2741977600.0,
+ "23": 2741977600.0,
+ "24": 2741977600.0,
+ "25": 2741977600.0,
+ "26": 2741977600.0,
+ "27": 2741977600.0,
+ "28": 2741977600.0,
+ "29": 2741977600.0,
+ "30": 2741977600.0,
+ "31": 2741977600.0,
+ "32": 2741977600.0,
+ "33": 2741977600.0,
+ "34": 2741977600.0,
+ "35": 2741977600.0,
+ "36": 2741977600.0,
+ "37": 2741977600.0,
+ "38": 2741977600.0,
+ "39": 2741977600.0,
+ "40": 2741977600.0,
+ "41": 2741977600.0,
+ "42": 2741977600.0,
+ "43": 2741977600.0,
+ "44": 2741977600.0,
+ "45": 2741977600.0,
+ "46": 2741977600.0,
+ "47": 2741977600.0,
+ "48": 2741977600.0,
+ "49": 2741977600.0,
+ "50": 2741977600.0,
+ "51": 2743026176.0,
+ "52": 2743026176.0,
+ "53": 2743026176.0,
+ "54": 2743026176.0,
+ "55": 2743026176.0,
+ "56": 2743026176.0,
+ "57": 2743026176.0,
+ "58": 2743026176.0,
+ "59": 2743026176.0,
+ "60": 2743026176.0,
+ "61": 2743026176.0,
+ "62": 2743026176.0,
+ "63": 2743026176.0,
+ "64": 2743026176.0,
+ "65": 2743026176.0,
+ "66": 2743026176.0,
+ "67": 2743026176.0,
+ "68": 2743026176.0,
+ "69": 2743026176.0,
+ "70": 2743026176.0,
+ "71": 2743026176.0,
+ "72": 2743026176.0,
+ "73": 2743026176.0,
+ "74": 2743026176.0,
+ "75": 2743026176.0,
+ "76": 2743026176.0,
+ "77": 2743026176.0,
+ "78": 2743026176.0,
+ "79": 2743026176.0,
+ "80": 2743026176.0,
+ "81": 2743026176.0,
+ "82": 2743026176.0,
+ "83": 2743026176.0,
+ "84": 2743026176.0,
+ "85": 2743026176.0,
+ "86": 2743026176.0,
+ "87": 2743026176.0,
+ "88": 2743026176.0,
+ "89": 2743026176.0,
+ "90": 2743026176.0,
+ "91": 2743026176.0,
+ "92": 2743026176.0,
+ "93": 2743026176.0,
+ "94": 2743026176.0,
+ "95": 2743026176.0,
+ "96": 2743026176.0,
+ "97": 2743026176.0,
+ "98": 2743026176.0,
+ "99": 2743026176.0,
+ "100": 2743026176.0
}
},
"iteration-time": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.38769,
- "3": 0.09884,
- "4": 0.08514,
- "5": 0.08435,
- "6": 0.08412,
- "7": 0.08558,
- "8": 0.08425,
- "9": 0.08436,
- "10": 0.08457,
- "11": 0.08469,
- "12": 0.0848,
- "13": 0.08487,
- "14": 0.08571,
- "15": 0.08487,
- "16": 0.08529,
- "17": 0.08559,
- "18": 0.0898,
- "19": 0.08482,
- "20": 0.08509,
- "21": 0.08527,
- "22": 0.08597,
- "23": 0.08592,
- "24": 0.08654,
- "25": 0.08608,
- "26": 0.08574,
- "27": 0.08542,
- "28": 0.0856,
- "29": 0.08581,
- "30": 0.08539,
- "31": 0.08675,
- "32": 0.08679,
- "33": 0.08699,
- "34": 0.08599,
- "35": 0.08568,
- "36": 0.08528,
- "37": 0.08527,
- "38": 0.08526,
- "39": 0.08614,
- "40": 0.08507,
- "41": 0.08552,
- "42": 0.08586,
- "43": 0.08568,
- "44": 0.0866,
- "45": 0.08692,
- "46": 0.08614,
- "47": 0.0859,
- "48": 0.0863,
- "49": 0.08723,
- "50": 0.08703,
- "51": 0.09195,
- "52": 0.0775,
- "53": 0.07822,
- "54": 0.07813,
- "55": 0.0784,
- "56": 0.07871,
- "57": 0.07816,
- "58": 0.07787,
- "59": 0.07958,
- "60": 0.07893,
- "61": 0.07873,
- "62": 0.07887,
- "63": 0.07945,
- "64": 0.07879,
- "65": 0.08059,
- "66": 0.08041,
- "67": 0.08127,
- "68": 0.07996,
- "69": 0.0799,
- "70": 0.07821,
- "71": 0.07712,
- "72": 0.07745,
- "73": 0.07774,
- "74": 0.07859,
- "75": 0.07741,
- "76": 0.07753,
- "77": 0.07725,
- "78": 0.07676,
- "79": 0.07838,
- "80": 0.07786,
- "81": 0.07743,
- "82": 0.07732,
- "83": 0.0773,
- "84": 0.07664,
- "85": 0.07753,
- "86": 0.07826,
- "87": 0.07764,
- "88": 0.07681,
- "89": 0.07911,
- "90": 0.07799,
- "91": 0.07796,
- "92": 0.07853,
- "93": 0.07736,
- "94": 0.07777,
- "95": 0.07791,
- "96": 0.07723,
- "97": 0.07753,
- "98": 0.07789,
- "99": 0.07782,
- "100": 0.07733
+ "2": 3.9286,
+ "3": 0.09526,
+ "4": 0.08266,
+ "5": 0.08209,
+ "6": 0.08064,
+ "7": 0.07972,
+ "8": 0.08118,
+ "9": 0.08097,
+ "10": 0.08133,
+ "11": 0.08132,
+ "12": 0.08026,
+ "13": 0.08115,
+ "14": 0.08074,
+ "15": 0.08116,
+ "16": 0.08103,
+ "17": 0.08059,
+ "18": 0.08038,
+ "19": 0.08055,
+ "20": 0.07833,
+ "21": 0.08049,
+ "22": 0.0799,
+ "23": 0.0808,
+ "24": 0.08065,
+ "25": 0.08048,
+ "26": 0.07992,
+ "27": 0.07921,
+ "28": 0.08124,
+ "29": 0.08049,
+ "30": 0.08361,
+ "31": 0.08141,
+ "32": 0.08124,
+ "33": 0.08111,
+ "34": 0.08202,
+ "35": 0.08168,
+ "36": 0.08137,
+ "37": 0.08106,
+ "38": 0.08305,
+ "39": 0.08158,
+ "40": 0.08186,
+ "41": 0.08042,
+ "42": 0.08178,
+ "43": 0.08078,
+ "44": 0.08351,
+ "45": 0.08217,
+ "46": 0.08121,
+ "47": 0.08152,
+ "48": 0.0819,
+ "49": 0.08255,
+ "50": 0.08107,
+ "51": 0.11901,
+ "52": 0.08229,
+ "53": 0.08218,
+ "54": 0.08159,
+ "55": 0.08127,
+ "56": 0.08154,
+ "57": 0.08228,
+ "58": 0.08252,
+ "59": 0.0823,
+ "60": 0.08201,
+ "61": 0.08222,
+ "62": 0.08163,
+ "63": 0.08177,
+ "64": 0.08204,
+ "65": 0.08257,
+ "66": 0.08186,
+ "67": 0.08108,
+ "68": 0.08143,
+ "69": 0.0818,
+ "70": 0.08223,
+ "71": 0.08242,
+ "72": 0.08251,
+ "73": 0.08331,
+ "74": 0.08246,
+ "75": 0.08308,
+ "76": 0.08189,
+ "77": 0.08213,
+ "78": 0.08067,
+ "79": 0.0804,
+ "80": 0.08118,
+ "81": 0.08216,
+ "82": 0.08164,
+ "83": 0.08201,
+ "84": 0.08275,
+ "85": 0.08133,
+ "86": 0.0822,
+ "87": 0.08023,
+ "88": 0.08212,
+ "89": 0.08275,
+ "90": 0.08111,
+ "91": 0.08184,
+ "92": 0.08214,
+ "93": 0.08154,
+ "94": 0.08222,
+ "95": 0.08216,
+ "96": 0.08161,
+ "97": 0.08176,
+ "98": 0.08154,
+ "99": 0.08061,
+ "100": 0.08168
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_h100.json
index 131bcbe928e..a4642c20d53 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.89631,
- "2": 10.89416,
- "3": 10.88786,
- "4": 10.8914,
- "5": 10.89154,
- "6": 10.90001,
- "7": 10.89184,
- "8": 10.89886,
- "9": 10.90208,
- "10": 10.88361,
- "11": 10.87816,
- "12": 10.89332,
- "13": 10.89816,
- "14": 10.89241,
- "15": 10.84798,
- "16": 10.854,
- "17": 10.83093,
- "18": 10.83991,
- "19": 10.82802,
- "20": 10.74822,
- "21": 10.73494,
- "22": 10.61719,
- "23": 10.72621,
- "24": 10.63177,
- "25": 10.5931,
- "26": 10.63365,
- "27": 10.63304,
- "28": 10.58259,
- "29": 10.58595,
- "30": 10.41201,
- "31": 10.15907,
- "32": 10.48362,
- "33": 10.46704,
- "34": 10.23815,
- "35": 10.28193,
- "36": 10.24052,
- "37": 10.36227,
- "38": 10.20306,
- "39": 10.40456,
- "40": 10.09271,
- "41": 10.15831,
- "42": 10.21934,
- "43": 9.8436,
- "44": 9.97299,
- "45": 9.84189,
- "46": 9.82017,
- "47": 10.14968,
- "48": 9.86021,
- "49": 9.54238,
- "50": 9.91347,
- "51": 9.85447,
- "52": 9.73936,
- "53": 10.07426,
- "54": 9.96915,
- "55": 9.88574,
- "56": 9.62437,
- "57": 9.4823,
- "58": 9.83483,
- "59": 9.58732,
- "60": 9.50245,
- "61": 9.69343,
- "62": 9.98806,
- "63": 9.39103,
- "64": 9.78021,
- "65": 8.94515,
- "66": 9.70494,
- "67": 9.37251,
- "68": 9.78329,
- "69": 9.79058,
- "70": 9.74454,
- "71": 9.62301,
- "72": 9.58458,
- "73": 9.50513,
- "74": 8.94312,
- "75": 9.42524,
- "76": 9.07601,
- "77": 10.06353,
- "78": 9.72308,
- "79": 9.37502,
- "80": 9.40453,
- "81": 9.47794,
- "82": 9.69667,
- "83": 9.3072,
- "84": 9.41526,
- "85": 9.61293,
- "86": 9.07195,
- "87": 9.5884,
- "88": 9.74762,
- "89": 9.59982,
- "90": 9.81672,
- "91": 9.3379,
- "92": 9.35605,
- "93": 9.07425,
- "94": 8.8351,
- "95": 9.5184,
- "96": 9.52391,
- "97": 9.30923,
- "98": 9.66743,
- "99": 8.88419,
- "100": 9.39924
+ "1": 10.93345,
+ "2": 10.92442,
+ "3": 10.92705,
+ "4": 10.93095,
+ "5": 10.93662,
+ "6": 10.92967,
+ "7": 10.92359,
+ "8": 10.92254,
+ "9": 10.92459,
+ "10": 10.92961,
+ "11": 10.90934,
+ "12": 10.92831,
+ "13": 10.89682,
+ "14": 10.8971,
+ "15": 10.89151,
+ "16": 10.87452,
+ "17": 10.87088,
+ "18": 10.8718,
+ "19": 10.85366,
+ "20": 10.80457,
+ "21": 10.78428,
+ "22": 10.7648,
+ "23": 10.77095,
+ "24": 10.75021,
+ "25": 10.74706,
+ "26": 10.72261,
+ "27": 10.68799,
+ "28": 10.61501,
+ "29": 10.5932,
+ "30": 10.55631,
+ "31": 10.5649,
+ "32": 10.54518,
+ "33": 10.51567,
+ "34": 10.4822,
+ "35": 10.485,
+ "36": 10.46532,
+ "37": 10.42863,
+ "38": 10.42601,
+ "39": 10.39456,
+ "40": 10.37451,
+ "41": 10.3512,
+ "42": 10.34808,
+ "43": 10.31966,
+ "44": 10.29913,
+ "45": 10.30316,
+ "46": 10.26446,
+ "47": 10.25218,
+ "48": 10.21542,
+ "49": 10.20204,
+ "50": 10.21075,
+ "51": 10.19945,
+ "52": 10.15805,
+ "53": 10.16368,
+ "54": 10.1283,
+ "55": 10.10242,
+ "56": 10.12754,
+ "57": 10.1158,
+ "58": 10.11962,
+ "59": 10.06698,
+ "60": 10.09293,
+ "61": 10.04268,
+ "62": 10.01398,
+ "63": 10.08129,
+ "64": 10.03224,
+ "65": 10.00035,
+ "66": 10.04315,
+ "67": 10.01555,
+ "68": 9.98465,
+ "69": 9.99674,
+ "70": 9.97884,
+ "71": 10.0032,
+ "72": 9.99334,
+ "73": 9.97486,
+ "74": 9.96895,
+ "75": 9.9335,
+ "76": 9.96001,
+ "77": 9.95966,
+ "78": 9.91404,
+ "79": 9.91448,
+ "80": 9.92779,
+ "81": 9.95362,
+ "82": 9.8908,
+ "83": 9.85459,
+ "84": 9.79532,
+ "85": 9.78078,
+ "86": 9.88654,
+ "87": 9.90729,
+ "88": 9.88176,
+ "89": 9.82133,
+ "90": 9.81597,
+ "91": 9.82611,
+ "92": 9.81639,
+ "93": 9.74789,
+ "94": 9.82694,
+ "95": 9.81238,
+ "96": 9.80068,
+ "97": 9.7458,
+ "98": 9.76761,
+ "99": 9.8208,
+ "100": 9.7046
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1483.0,
- "2": 1650.0,
- "3": 1681.0,
- "4": 1767.0,
- "5": 1903.0,
- "6": 1952.0,
- "7": 1967.0,
- "8": 1651.0,
- "9": 1886.0,
- "10": 1427.0,
- "11": 1897.0,
- "12": 1855.0,
- "13": 1941.0,
- "14": 1749.0,
- "15": 1901.0,
- "16": 1813.0,
- "17": 1710.0,
- "18": 1707.0,
- "19": 1819.0,
- "20": 1639.0,
- "21": 1880.0,
- "22": 1769.0,
- "23": 2016.0,
- "24": 1692.0,
- "25": 1672.0,
- "26": 1778.0,
- "27": 1861.0,
- "28": 1964.0,
- "29": 2021.0,
- "30": 1938.0,
- "31": 1645.0,
- "32": 1864.0,
- "33": 2150.0,
- "34": 1828.0,
- "35": 1982.0,
- "36": 1864.0,
- "37": 2355.0,
- "38": 2358.0,
- "39": 2385.0,
- "40": 2407.0,
- "41": 2501.0,
- "42": 2435.0,
- "43": 2033.0,
- "44": 2089.0,
- "45": 2210.0,
- "46": 2351.0,
- "47": 2502.0,
- "48": 2444.0,
- "49": 2302.0,
- "50": 2492.0,
- "51": 2598.0,
- "52": 2547.0,
- "53": 2957.0,
- "54": 2750.0,
- "55": 2372.0,
- "56": 2569.0,
- "57": 2395.0,
- "58": 2901.0,
- "59": 2741.0,
- "60": 2430.0,
- "61": 2868.0,
- "62": 2651.0,
- "63": 2507.0,
- "64": 3014.0,
- "65": 2683.0,
- "66": 2935.0,
- "67": 2783.0,
- "68": 2725.0,
- "69": 2788.0,
- "70": 3152.0,
- "71": 3026.0,
- "72": 2415.0,
- "73": 3122.0,
- "74": 1967.0,
- "75": 2581.0,
- "76": 3010.0,
- "77": 3294.0,
- "78": 3166.0,
- "79": 3150.0,
- "80": 3246.0,
- "81": 3566.0,
- "82": 3285.0,
- "83": 2817.0,
- "84": 3269.0,
- "85": 3425.0,
- "86": 2819.0,
- "87": 3577.0,
- "88": 3004.0,
- "89": 3323.0,
- "90": 3023.0,
- "91": 2661.0,
- "92": 3066.0,
- "93": 2691.0,
- "94": 3305.0,
- "95": 3403.0,
- "96": 3377.0,
- "97": 3242.0,
- "98": 3697.0,
- "99": 3112.0,
- "100": 3199.0
+ "1": 1860.0,
+ "2": 1723.0,
+ "3": 1744.0,
+ "4": 1706.0,
+ "5": 1728.0,
+ "6": 1819.0,
+ "7": 2115.0,
+ "8": 1842.0,
+ "9": 1822.0,
+ "10": 1770.0,
+ "11": 1734.0,
+ "12": 1679.0,
+ "13": 1770.0,
+ "14": 1931.0,
+ "15": 1653.0,
+ "16": 1783.0,
+ "17": 1832.0,
+ "18": 1735.0,
+ "19": 1762.0,
+ "20": 1714.0,
+ "21": 1836.0,
+ "22": 1727.0,
+ "23": 1754.0,
+ "24": 1759.0,
+ "25": 1747.0,
+ "26": 1912.0,
+ "27": 1847.0,
+ "28": 1943.0,
+ "29": 2000.0,
+ "30": 1930.0,
+ "31": 2072.0,
+ "32": 2032.0,
+ "33": 2014.0,
+ "34": 2136.0,
+ "35": 2114.0,
+ "36": 2098.0,
+ "37": 2397.0,
+ "38": 2157.0,
+ "39": 2259.0,
+ "40": 2350.0,
+ "41": 2440.0,
+ "42": 2122.0,
+ "43": 2439.0,
+ "44": 2215.0,
+ "45": 2535.0,
+ "46": 2552.0,
+ "47": 2417.0,
+ "48": 2589.0,
+ "49": 2853.0,
+ "50": 2585.0,
+ "51": 2568.0,
+ "52": 2681.0,
+ "53": 2591.0,
+ "54": 2826.0,
+ "55": 2541.0,
+ "56": 2707.0,
+ "57": 2220.0,
+ "58": 3568.0,
+ "59": 2948.0,
+ "60": 3002.0,
+ "61": 2827.0,
+ "62": 3194.0,
+ "63": 3323.0,
+ "64": 3791.0,
+ "65": 2593.0,
+ "66": 3088.0,
+ "67": 3720.0,
+ "68": 3432.0,
+ "69": 3048.0,
+ "70": 3272.0,
+ "71": 3340.0,
+ "72": 3185.0,
+ "73": 3640.0,
+ "74": 3503.0,
+ "75": 3238.0,
+ "76": 3508.0,
+ "77": 3802.0,
+ "78": 3448.0,
+ "79": 3414.0,
+ "80": 3309.0,
+ "81": 3704.0,
+ "82": 3260.0,
+ "83": 3205.0,
+ "84": 3020.0,
+ "85": 2918.0,
+ "86": 3055.0,
+ "87": 2884.0,
+ "88": 3115.0,
+ "89": 3105.0,
+ "90": 3936.0,
+ "91": 3090.0,
+ "92": 3218.0,
+ "93": 3442.0,
+ "94": 3329.0,
+ "95": 3262.0,
+ "96": 3439.0,
+ "97": 3701.0,
+ "98": 3464.0,
+ "99": 3162.0,
+ "100": 3314.0
}
},
"mem-allocated-bytes": {
@@ -375,56 +375,56 @@
"48": 2690743808.0,
"49": 2690743808.0,
"50": 2690743808.0,
- "51": 2690743808.0,
- "52": 2690743808.0,
- "53": 2690743808.0,
- "54": 2690743808.0,
- "55": 2690743808.0,
- "56": 2690743808.0,
- "57": 2690743808.0,
- "58": 2690743808.0,
- "59": 2690743808.0,
- "60": 2690743808.0,
- "61": 2690743808.0,
- "62": 2690743808.0,
- "63": 2690743808.0,
- "64": 2690743808.0,
- "65": 2690743808.0,
- "66": 2690743808.0,
- "67": 2690743808.0,
- "68": 2690743808.0,
- "69": 2690743808.0,
- "70": 2690743808.0,
- "71": 2690743808.0,
- "72": 2690743808.0,
- "73": 2690743808.0,
- "74": 2690743808.0,
- "75": 2690743808.0,
- "76": 2690743808.0,
- "77": 2690743808.0,
- "78": 2690743808.0,
- "79": 2690743808.0,
- "80": 2690743808.0,
- "81": 2690743808.0,
- "82": 2690743808.0,
- "83": 2690743808.0,
- "84": 2690743808.0,
- "85": 2690743808.0,
- "86": 2690743808.0,
- "87": 2690743808.0,
- "88": 2690743808.0,
- "89": 2690743808.0,
- "90": 2690743808.0,
- "91": 2690743808.0,
- "92": 2690743808.0,
- "93": 2690743808.0,
- "94": 2690743808.0,
- "95": 2690743808.0,
- "96": 2690743808.0,
- "97": 2690743808.0,
- "98": 2690743808.0,
- "99": 2690743808.0,
- "100": 2690743808.0
+ "51": 2691792384.0,
+ "52": 2691792384.0,
+ "53": 2691792384.0,
+ "54": 2691792384.0,
+ "55": 2691792384.0,
+ "56": 2691792384.0,
+ "57": 2691792384.0,
+ "58": 2691792384.0,
+ "59": 2691792384.0,
+ "60": 2691792384.0,
+ "61": 2691792384.0,
+ "62": 2691792384.0,
+ "63": 2691792384.0,
+ "64": 2691792384.0,
+ "65": 2691792384.0,
+ "66": 2691792384.0,
+ "67": 2691792384.0,
+ "68": 2691792384.0,
+ "69": 2691792384.0,
+ "70": 2691792384.0,
+ "71": 2691792384.0,
+ "72": 2691792384.0,
+ "73": 2691792384.0,
+ "74": 2691792384.0,
+ "75": 2691792384.0,
+ "76": 2691792384.0,
+ "77": 2691792384.0,
+ "78": 2691792384.0,
+ "79": 2691792384.0,
+ "80": 2691792384.0,
+ "81": 2691792384.0,
+ "82": 2691792384.0,
+ "83": 2691792384.0,
+ "84": 2691792384.0,
+ "85": 2691792384.0,
+ "86": 2691792384.0,
+ "87": 2691792384.0,
+ "88": 2691792384.0,
+ "89": 2691792384.0,
+ "90": 2691792384.0,
+ "91": 2691792384.0,
+ "92": 2691792384.0,
+ "93": 2691792384.0,
+ "94": 2691792384.0,
+ "95": 2691792384.0,
+ "96": 2691792384.0,
+ "97": 2691792384.0,
+ "98": 2691792384.0,
+ "99": 2691792384.0,
+ "100": 2691792384.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 6.85919,
- "2": 0.0831,
- "3": 0.08065,
- "4": 0.05861,
- "5": 0.04976,
- "6": 0.05045,
- "7": 0.04972,
- "8": 0.04911,
- "9": 0.04965,
- "10": 0.04942,
- "11": 0.04916,
- "12": 0.04915,
- "13": 0.04939,
- "14": 0.04993,
- "15": 0.04987,
- "16": 0.04906,
- "17": 0.05015,
- "18": 0.04924,
- "19": 0.05168,
- "20": 0.04963,
- "21": 0.05051,
- "22": 0.04948,
- "23": 0.05006,
- "24": 0.04939,
- "25": 0.05019,
- "26": 0.04951,
- "27": 0.05048,
- "28": 0.04917,
- "29": 0.05015,
- "30": 0.04921,
- "31": 0.04969,
- "32": 0.04894,
- "33": 0.04941,
- "34": 0.04938,
- "35": 0.04927,
- "36": 0.04942,
- "37": 0.04944,
- "38": 0.04973,
- "39": 0.04957,
- "40": 0.05016,
- "41": 0.04968,
- "42": 0.05042,
- "43": 0.0523,
- "44": 0.04956,
- "45": 0.04948,
- "46": 0.05093,
- "47": 0.0493,
- "48": 0.0498,
- "49": 0.05177,
- "50": 0.05032,
- "51": 0.05749,
- "52": 0.05013,
- "53": 0.0512,
- "54": 0.04935,
- "55": 0.04891,
- "56": 0.04976,
- "57": 0.04984,
- "58": 0.04964,
- "59": 0.05274,
- "60": 0.04962,
- "61": 0.05096,
- "62": 0.04934,
- "63": 0.04971,
- "64": 0.0503,
- "65": 0.05028,
- "66": 0.04991,
- "67": 0.04926,
- "68": 0.04848,
- "69": 0.0493,
- "70": 0.04943,
- "71": 0.04943,
- "72": 0.04852,
- "73": 0.04928,
- "74": 0.04895,
- "75": 0.04995,
- "76": 0.04877,
- "77": 0.0492,
- "78": 0.04886,
- "79": 0.04938,
- "80": 0.04894,
- "81": 0.04892,
- "82": 0.05016,
- "83": 0.04964,
- "84": 0.04956,
- "85": 0.04881,
- "86": 0.04999,
- "87": 0.04908,
- "88": 0.04838,
- "89": 0.04957,
- "90": 0.04882,
- "91": 0.04993,
- "92": 0.05004,
- "93": 0.05003,
- "94": 0.04961,
- "95": 0.05132,
- "96": 0.05071,
- "97": 0.04952,
- "98": 0.04851,
- "99": 0.05027,
- "100": 0.04988
+ "1": "nan",
+ "2": 3.9754,
+ "3": 0.07889,
+ "4": 0.06094,
+ "5": 0.06071,
+ "6": 0.0608,
+ "7": 0.0546,
+ "8": 0.05114,
+ "9": 0.05321,
+ "10": 0.05096,
+ "11": 0.05168,
+ "12": 0.05084,
+ "13": 0.05151,
+ "14": 0.0508,
+ "15": 0.05166,
+ "16": 0.05086,
+ "17": 0.0516,
+ "18": 0.05104,
+ "19": 0.05193,
+ "20": 0.05099,
+ "21": 0.05215,
+ "22": 0.05107,
+ "23": 0.05208,
+ "24": 0.05058,
+ "25": 0.05227,
+ "26": 0.05059,
+ "27": 0.05205,
+ "28": 0.05211,
+ "29": 0.05174,
+ "30": 0.05145,
+ "31": 0.05174,
+ "32": 0.05198,
+ "33": 0.05175,
+ "34": 0.05148,
+ "35": 0.05165,
+ "36": 0.05162,
+ "37": 0.05195,
+ "38": 0.05186,
+ "39": 0.05187,
+ "40": 0.05042,
+ "41": 0.05168,
+ "42": 0.05169,
+ "43": 0.05173,
+ "44": 0.05169,
+ "45": 0.05279,
+ "46": 0.05167,
+ "47": 0.05175,
+ "48": 0.05203,
+ "49": 0.05211,
+ "50": 0.05186,
+ "51": 0.06096,
+ "52": 0.05076,
+ "53": 0.05149,
+ "54": 0.05064,
+ "55": 0.05214,
+ "56": 0.05023,
+ "57": 0.05062,
+ "58": 0.05047,
+ "59": 0.05035,
+ "60": 0.0505,
+ "61": 0.05011,
+ "62": 0.05068,
+ "63": 0.05023,
+ "64": 0.05067,
+ "65": 0.05044,
+ "66": 0.05095,
+ "67": 0.05124,
+ "68": 0.05093,
+ "69": 0.05005,
+ "70": 0.05039,
+ "71": 0.05029,
+ "72": 0.05063,
+ "73": 0.05041,
+ "74": 0.05073,
+ "75": 0.05002,
+ "76": 0.0503,
+ "77": 0.05018,
+ "78": 0.0505,
+ "79": 0.0499,
+ "80": 0.05096,
+ "81": 0.05081,
+ "82": 0.0587,
+ "83": 0.05263,
+ "84": 0.05121,
+ "85": 0.05047,
+ "86": 0.05147,
+ "87": 0.04957,
+ "88": 0.0511,
+ "89": 0.0499,
+ "90": 0.05106,
+ "91": 0.04964,
+ "92": 0.05107,
+ "93": 0.0496,
+ "94": 0.05142,
+ "95": 0.05052,
+ "96": 0.05091,
+ "97": 0.04958,
+ "98": 0.05066,
+ "99": 0.05143,
+ "100": 0.05137
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer/model_config.yaml
index 82506115963..5859b7461f9 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,5 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer_no_mmap_bin_files/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer_no_mmap_bin_files/model_config.yaml
index 4a5bf3d8fc7..685ec4b3db7 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer_no_mmap_bin_files/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer_no_mmap_bin_files/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute/golden_values_dev_dgx_h100.json
index 05b11c3c8ee..49661b940bf 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.89631,
- "2": 10.89416,
- "3": 10.88786,
- "4": 10.8914,
- "5": 10.89154,
- "6": 10.90001,
- "7": 10.89184,
- "8": 10.89886,
- "9": 10.90208,
- "10": 10.88361,
- "11": 10.87817,
- "12": 10.89334,
- "13": 10.89814,
- "14": 10.89242,
- "15": 10.84803,
- "16": 10.85398,
- "17": 10.83097,
- "18": 10.83991,
- "19": 10.82801,
- "20": 10.74824,
- "21": 10.73496,
- "22": 10.61719,
- "23": 10.72621,
- "24": 10.63178,
- "25": 10.59309,
- "26": 10.63369,
- "27": 10.63304,
- "28": 10.58264,
- "29": 10.58594,
- "30": 10.41204,
- "31": 10.15899,
- "32": 10.48366,
- "33": 10.46706,
- "34": 10.23811,
- "35": 10.28189,
- "36": 10.24056,
- "37": 10.36219,
- "38": 10.20309,
- "39": 10.40454,
- "40": 10.09271,
- "41": 10.15835,
- "42": 10.21933,
- "43": 9.84358,
- "44": 9.97303,
- "45": 9.84194,
- "46": 9.82017,
- "47": 10.14969,
- "48": 9.86023,
- "49": 9.54235,
- "50": 9.91343,
- "51": 9.8545,
- "52": 9.7393,
- "53": 10.07426,
- "54": 9.96913,
- "55": 9.88574,
- "56": 9.62438,
- "57": 9.48229,
- "58": 9.83484,
- "59": 9.58731,
- "60": 9.50243,
- "61": 9.6934,
- "62": 9.988,
- "63": 9.39105,
- "64": 9.78022,
- "65": 8.94516,
- "66": 9.70492,
- "67": 9.37249,
- "68": 9.78328,
- "69": 9.79057,
- "70": 9.74451,
- "71": 9.62298,
- "72": 9.58457,
- "73": 9.50511,
- "74": 8.94308,
- "75": 9.42524,
- "76": 9.07602,
- "77": 10.06352,
- "78": 9.72307,
- "79": 9.37497,
- "80": 9.40454,
- "81": 9.4779,
- "82": 9.69669,
- "83": 9.30714,
- "84": 9.41525,
- "85": 9.61295,
- "86": 9.07198,
- "87": 9.58834,
- "88": 9.7476,
- "89": 9.59984,
- "90": 9.81672,
- "91": 9.33791,
- "92": 9.35608,
- "93": 9.07423,
- "94": 8.83511,
- "95": 9.51841,
- "96": 9.52391,
- "97": 9.30922,
- "98": 9.66746,
- "99": 8.88421,
- "100": 9.39923
+ "1": 10.93345,
+ "2": 10.92442,
+ "3": 10.92705,
+ "4": 10.93095,
+ "5": 10.93662,
+ "6": 10.92967,
+ "7": 10.92359,
+ "8": 10.92254,
+ "9": 10.92459,
+ "10": 10.92964,
+ "11": 10.90935,
+ "12": 10.9283,
+ "13": 10.89685,
+ "14": 10.89709,
+ "15": 10.89154,
+ "16": 10.87453,
+ "17": 10.87085,
+ "18": 10.87181,
+ "19": 10.85366,
+ "20": 10.80462,
+ "21": 10.78428,
+ "22": 10.76479,
+ "23": 10.77095,
+ "24": 10.75016,
+ "25": 10.74704,
+ "26": 10.72264,
+ "27": 10.68796,
+ "28": 10.61502,
+ "29": 10.59319,
+ "30": 10.55629,
+ "31": 10.56494,
+ "32": 10.54515,
+ "33": 10.51567,
+ "34": 10.48221,
+ "35": 10.48496,
+ "36": 10.46534,
+ "37": 10.42859,
+ "38": 10.42595,
+ "39": 10.39453,
+ "40": 10.37451,
+ "41": 10.35119,
+ "42": 10.34811,
+ "43": 10.31965,
+ "44": 10.29913,
+ "45": 10.30316,
+ "46": 10.26446,
+ "47": 10.25215,
+ "48": 10.21541,
+ "49": 10.20203,
+ "50": 10.21081,
+ "51": 10.19944,
+ "52": 10.15799,
+ "53": 10.16368,
+ "54": 10.12833,
+ "55": 10.10245,
+ "56": 10.12756,
+ "57": 10.11587,
+ "58": 10.11964,
+ "59": 10.06694,
+ "60": 10.09293,
+ "61": 10.04264,
+ "62": 10.01399,
+ "63": 10.08127,
+ "64": 10.03224,
+ "65": 10.00035,
+ "66": 10.04315,
+ "67": 10.01557,
+ "68": 9.98463,
+ "69": 9.99673,
+ "70": 9.97883,
+ "71": 10.00317,
+ "72": 9.99336,
+ "73": 9.97488,
+ "74": 9.96897,
+ "75": 9.93347,
+ "76": 9.96002,
+ "77": 9.95964,
+ "78": 9.91409,
+ "79": 9.91446,
+ "80": 9.92778,
+ "81": 9.95364,
+ "82": 9.8908,
+ "83": 9.85455,
+ "84": 9.79532,
+ "85": 9.7808,
+ "86": 9.88652,
+ "87": 9.90728,
+ "88": 9.88177,
+ "89": 9.82134,
+ "90": 9.81594,
+ "91": 9.82609,
+ "92": 9.81644,
+ "93": 9.74789,
+ "94": 9.82692,
+ "95": 9.81238,
+ "96": 9.80068,
+ "97": 9.74581,
+ "98": 9.76761,
+ "99": 9.8208,
+ "100": 9.70463
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1483.0,
- "2": 1650.0,
- "3": 1681.0,
- "4": 1767.0,
- "5": 1903.0,
- "6": 1952.0,
- "7": 1967.0,
- "8": 1651.0,
- "9": 1886.0,
- "10": 1427.0,
- "11": 1939.0,
- "12": 1778.0,
- "13": 1964.0,
- "14": 1762.0,
- "15": 1980.0,
- "16": 1923.0,
- "17": 1817.0,
- "18": 1783.0,
- "19": 1750.0,
- "20": 1588.0,
- "21": 1855.0,
- "22": 1641.0,
- "23": 2098.0,
- "24": 1679.0,
- "25": 1649.0,
- "26": 1806.0,
- "27": 1834.0,
- "28": 2042.0,
- "29": 2033.0,
- "30": 1984.0,
- "31": 1518.0,
- "32": 1954.0,
- "33": 2068.0,
- "34": 1900.0,
- "35": 1921.0,
- "36": 1965.0,
- "37": 2321.0,
- "38": 2340.0,
- "39": 2344.0,
- "40": 2367.0,
- "41": 2457.0,
- "42": 2367.0,
- "43": 2020.0,
- "44": 2135.0,
- "45": 2184.0,
- "46": 2310.0,
- "47": 2463.0,
- "48": 2450.0,
- "49": 2259.0,
- "50": 2444.0,
- "51": 2543.0,
- "52": 2613.0,
- "53": 2945.0,
- "54": 2713.0,
- "55": 2503.0,
- "56": 2692.0,
- "57": 2338.0,
- "58": 2961.0,
- "59": 2620.0,
- "60": 2367.0,
- "61": 2909.0,
- "62": 2728.0,
- "63": 2399.0,
- "64": 2909.0,
- "65": 2605.0,
- "66": 2983.0,
- "67": 2793.0,
- "68": 2663.0,
- "69": 2833.0,
- "70": 3135.0,
- "71": 2997.0,
- "72": 2464.0,
- "73": 3088.0,
- "74": 1970.0,
- "75": 2556.0,
- "76": 3064.0,
- "77": 3231.0,
- "78": 3097.0,
- "79": 3035.0,
- "80": 3301.0,
- "81": 3599.0,
- "82": 3215.0,
- "83": 2757.0,
- "84": 3130.0,
- "85": 3380.0,
- "86": 2742.0,
- "87": 3723.0,
- "88": 3066.0,
- "89": 3264.0,
- "90": 3198.0,
- "91": 2718.0,
- "92": 3070.0,
- "93": 2624.0,
- "94": 3301.0,
- "95": 3431.0,
- "96": 3358.0,
- "97": 3142.0,
- "98": 3704.0,
- "99": 3107.0,
- "100": 3089.0
+ "1": 1860.0,
+ "2": 1723.0,
+ "3": 1744.0,
+ "4": 1706.0,
+ "5": 1728.0,
+ "6": 1819.0,
+ "7": 2115.0,
+ "8": 1842.0,
+ "9": 1780.0,
+ "10": 1833.0,
+ "11": 1729.0,
+ "12": 1700.0,
+ "13": 1796.0,
+ "14": 1887.0,
+ "15": 1674.0,
+ "16": 1770.0,
+ "17": 1844.0,
+ "18": 1800.0,
+ "19": 1770.0,
+ "20": 1726.0,
+ "21": 1889.0,
+ "22": 1731.0,
+ "23": 1755.0,
+ "24": 1872.0,
+ "25": 1707.0,
+ "26": 1898.0,
+ "27": 1856.0,
+ "28": 1948.0,
+ "29": 2014.0,
+ "30": 1985.0,
+ "31": 2044.0,
+ "32": 2053.0,
+ "33": 1942.0,
+ "34": 2184.0,
+ "35": 2098.0,
+ "36": 2094.0,
+ "37": 2323.0,
+ "38": 2128.0,
+ "39": 2181.0,
+ "40": 2421.0,
+ "41": 2424.0,
+ "42": 2079.0,
+ "43": 2479.0,
+ "44": 2231.0,
+ "45": 2471.0,
+ "46": 2513.0,
+ "47": 2563.0,
+ "48": 2432.0,
+ "49": 2917.0,
+ "50": 2611.0,
+ "51": 2610.0,
+ "52": 2738.0,
+ "53": 2595.0,
+ "54": 2828.0,
+ "55": 2617.0,
+ "56": 2714.0,
+ "57": 2295.0,
+ "58": 3611.0,
+ "59": 2910.0,
+ "60": 2939.0,
+ "61": 2811.0,
+ "62": 3152.0,
+ "63": 3306.0,
+ "64": 3659.0,
+ "65": 2700.0,
+ "66": 3176.0,
+ "67": 3726.0,
+ "68": 3402.0,
+ "69": 3128.0,
+ "70": 3244.0,
+ "71": 3490.0,
+ "72": 3149.0,
+ "73": 3582.0,
+ "74": 3505.0,
+ "75": 3243.0,
+ "76": 3412.0,
+ "77": 3878.0,
+ "78": 3483.0,
+ "79": 3370.0,
+ "80": 3315.0,
+ "81": 3804.0,
+ "82": 3177.0,
+ "83": 3197.0,
+ "84": 2961.0,
+ "85": 2794.0,
+ "86": 3054.0,
+ "87": 2987.0,
+ "88": 3180.0,
+ "89": 3182.0,
+ "90": 3861.0,
+ "91": 2938.0,
+ "92": 3190.0,
+ "93": 3444.0,
+ "94": 3392.0,
+ "95": 3319.0,
+ "96": 3375.0,
+ "97": 3714.0,
+ "98": 3439.0,
+ "99": 3226.0,
+ "100": 3225.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1261849088.0,
- "2": 1261849088.0,
- "3": 1261849088.0,
- "4": 1261849088.0,
- "5": 1261849088.0,
- "6": 1261849088.0,
- "7": 1261849088.0,
- "8": 1261849088.0,
- "9": 1261849088.0,
- "10": 1261849088.0,
- "11": 1261849088.0,
- "12": 1261849088.0,
- "13": 1261849088.0,
- "14": 1261849088.0,
- "15": 1261849088.0,
- "16": 1261849088.0,
- "17": 1261849088.0,
- "18": 1261849088.0,
- "19": 1261849088.0,
- "20": 1261849088.0,
- "21": 1261849088.0,
- "22": 1261849088.0,
- "23": 1261849088.0,
- "24": 1261849088.0,
- "25": 1261849088.0,
- "26": 1261849088.0,
- "27": 1261849088.0,
- "28": 1261849088.0,
- "29": 1261849088.0,
- "30": 1261849088.0,
- "31": 1261849088.0,
- "32": 1261849088.0,
- "33": 1261849088.0,
- "34": 1261849088.0,
- "35": 1261849088.0,
- "36": 1261849088.0,
- "37": 1261849088.0,
- "38": 1261849088.0,
- "39": 1261849088.0,
- "40": 1261849088.0,
- "41": 1261849088.0,
- "42": 1261849088.0,
- "43": 1261849088.0,
- "44": 1261849088.0,
- "45": 1261849088.0,
- "46": 1261849088.0,
- "47": 1261849088.0,
- "48": 1261849088.0,
- "49": 1261849088.0,
- "50": 1261849088.0,
- "51": 1261849088.0,
- "52": 1261849088.0,
- "53": 1261849088.0,
- "54": 1261849088.0,
- "55": 1261849088.0,
- "56": 1261849088.0,
- "57": 1261849088.0,
- "58": 1261849088.0,
- "59": 1261849088.0,
- "60": 1261849088.0,
- "61": 1261849088.0,
- "62": 1261849088.0,
- "63": 1261849088.0,
- "64": 1261849088.0,
- "65": 1261849088.0,
- "66": 1261849088.0,
- "67": 1261849088.0,
- "68": 1261849088.0,
- "69": 1261849088.0,
- "70": 1261849088.0,
- "71": 1261849088.0,
- "72": 1261849088.0,
- "73": 1261849088.0,
- "74": 1261849088.0,
- "75": 1261849088.0,
- "76": 1261849088.0,
- "77": 1261849088.0,
- "78": 1261849088.0,
- "79": 1261849088.0,
- "80": 1261849088.0,
- "81": 1261849088.0,
- "82": 1261849088.0,
- "83": 1261849088.0,
- "84": 1261849088.0,
- "85": 1261849088.0,
- "86": 1261849088.0,
- "87": 1261849088.0,
- "88": 1261849088.0,
- "89": 1261849088.0,
- "90": 1261849088.0,
- "91": 1261849088.0,
- "92": 1261849088.0,
- "93": 1261849088.0,
- "94": 1261849088.0,
- "95": 1261849088.0,
- "96": 1261849088.0,
- "97": 1261849088.0,
- "98": 1261849088.0,
- "99": 1261849088.0,
- "100": 1261849088.0
+ "1": 1259751936.0,
+ "2": 1259751936.0,
+ "3": 1259751936.0,
+ "4": 1259751936.0,
+ "5": 1259751936.0,
+ "6": 1259751936.0,
+ "7": 1259751936.0,
+ "8": 1259751936.0,
+ "9": 1259751936.0,
+ "10": 1259751936.0,
+ "11": 1259751936.0,
+ "12": 1259751936.0,
+ "13": 1259751936.0,
+ "14": 1259751936.0,
+ "15": 1259751936.0,
+ "16": 1259751936.0,
+ "17": 1259751936.0,
+ "18": 1259751936.0,
+ "19": 1259751936.0,
+ "20": 1259751936.0,
+ "21": 1259751936.0,
+ "22": 1259751936.0,
+ "23": 1259751936.0,
+ "24": 1259751936.0,
+ "25": 1259751936.0,
+ "26": 1259751936.0,
+ "27": 1259751936.0,
+ "28": 1259751936.0,
+ "29": 1259751936.0,
+ "30": 1259751936.0,
+ "31": 1259751936.0,
+ "32": 1259751936.0,
+ "33": 1259751936.0,
+ "34": 1259751936.0,
+ "35": 1259751936.0,
+ "36": 1259751936.0,
+ "37": 1259751936.0,
+ "38": 1259751936.0,
+ "39": 1259751936.0,
+ "40": 1259751936.0,
+ "41": 1259751936.0,
+ "42": 1259751936.0,
+ "43": 1259751936.0,
+ "44": 1259751936.0,
+ "45": 1259751936.0,
+ "46": 1259751936.0,
+ "47": 1259751936.0,
+ "48": 1259751936.0,
+ "49": 1259751936.0,
+ "50": 1259751936.0,
+ "51": 1259751936.0,
+ "52": 1259751936.0,
+ "53": 1259751936.0,
+ "54": 1259751936.0,
+ "55": 1259751936.0,
+ "56": 1259751936.0,
+ "57": 1259751936.0,
+ "58": 1259751936.0,
+ "59": 1259751936.0,
+ "60": 1259751936.0,
+ "61": 1259751936.0,
+ "62": 1259751936.0,
+ "63": 1259751936.0,
+ "64": 1259751936.0,
+ "65": 1259751936.0,
+ "66": 1259751936.0,
+ "67": 1259751936.0,
+ "68": 1259751936.0,
+ "69": 1259751936.0,
+ "70": 1259751936.0,
+ "71": 1259751936.0,
+ "72": 1259751936.0,
+ "73": 1259751936.0,
+ "74": 1259751936.0,
+ "75": 1259751936.0,
+ "76": 1259751936.0,
+ "77": 1259751936.0,
+ "78": 1259751936.0,
+ "79": 1259751936.0,
+ "80": 1259751936.0,
+ "81": 1259751936.0,
+ "82": 1259751936.0,
+ "83": 1259751936.0,
+ "84": 1259751936.0,
+ "85": 1259751936.0,
+ "86": 1259751936.0,
+ "87": 1259751936.0,
+ "88": 1259751936.0,
+ "89": 1259751936.0,
+ "90": 1259751936.0,
+ "91": 1259751936.0,
+ "92": 1259751936.0,
+ "93": 1259751936.0,
+ "94": 1259751936.0,
+ "95": 1259751936.0,
+ "96": 1259751936.0,
+ "97": 1259751936.0,
+ "98": 1259751936.0,
+ "99": 1259751936.0,
+ "100": 1259751936.0
}
},
"mem-max-allocated-bytes": {
@@ -326,105 +326,105 @@
"step_interval": 1,
"values": {
"1": 2013853696.0,
- "2": 2563431424.0,
- "3": 2563431424.0,
- "4": 2563431424.0,
- "5": 2563431424.0,
- "6": 2563431424.0,
- "7": 2563431424.0,
- "8": 2563431424.0,
- "9": 2563431424.0,
- "10": 2563431424.0,
- "11": 2563431424.0,
- "12": 2563431424.0,
- "13": 2563431424.0,
- "14": 2563431424.0,
- "15": 2563431424.0,
- "16": 2563431424.0,
- "17": 2563431424.0,
- "18": 2563431424.0,
- "19": 2563431424.0,
- "20": 2563431424.0,
- "21": 2563431424.0,
- "22": 2563431424.0,
- "23": 2563431424.0,
- "24": 2563431424.0,
- "25": 2563431424.0,
- "26": 2563431424.0,
- "27": 2563431424.0,
- "28": 2563431424.0,
- "29": 2563431424.0,
- "30": 2563431424.0,
- "31": 2563431424.0,
- "32": 2563431424.0,
- "33": 2563431424.0,
- "34": 2563431424.0,
- "35": 2563431424.0,
- "36": 2563431424.0,
- "37": 2563431424.0,
- "38": 2563431424.0,
- "39": 2563431424.0,
- "40": 2563431424.0,
- "41": 2563431424.0,
- "42": 2563431424.0,
- "43": 2563431424.0,
- "44": 2563431424.0,
- "45": 2563431424.0,
- "46": 2563431424.0,
- "47": 2563431424.0,
- "48": 2563431424.0,
- "49": 2563431424.0,
- "50": 2563431424.0,
- "51": 2563431424.0,
- "52": 2563431424.0,
- "53": 2563431424.0,
- "54": 2563431424.0,
- "55": 2563431424.0,
- "56": 2563431424.0,
- "57": 2563431424.0,
- "58": 2563431424.0,
- "59": 2563431424.0,
- "60": 2563431424.0,
- "61": 2563431424.0,
- "62": 2563431424.0,
- "63": 2563431424.0,
- "64": 2563431424.0,
- "65": 2563431424.0,
- "66": 2563431424.0,
- "67": 2563431424.0,
- "68": 2563431424.0,
- "69": 2563431424.0,
- "70": 2563431424.0,
- "71": 2563431424.0,
- "72": 2563431424.0,
- "73": 2563431424.0,
- "74": 2563431424.0,
- "75": 2563431424.0,
- "76": 2563431424.0,
- "77": 2563431424.0,
- "78": 2563431424.0,
- "79": 2563431424.0,
- "80": 2563431424.0,
- "81": 2563431424.0,
- "82": 2563431424.0,
- "83": 2563431424.0,
- "84": 2563431424.0,
- "85": 2563431424.0,
- "86": 2563431424.0,
- "87": 2563431424.0,
- "88": 2563431424.0,
- "89": 2563431424.0,
- "90": 2563431424.0,
- "91": 2563431424.0,
- "92": 2563431424.0,
- "93": 2563431424.0,
- "94": 2563431424.0,
- "95": 2563431424.0,
- "96": 2563431424.0,
- "97": 2563431424.0,
- "98": 2563431424.0,
- "99": 2563431424.0,
- "100": 2563431424.0
+ "2": 2561334272.0,
+ "3": 2561334272.0,
+ "4": 2561334272.0,
+ "5": 2561334272.0,
+ "6": 2561334272.0,
+ "7": 2561334272.0,
+ "8": 2561334272.0,
+ "9": 2561334272.0,
+ "10": 2561334272.0,
+ "11": 2561334272.0,
+ "12": 2561334272.0,
+ "13": 2561334272.0,
+ "14": 2561334272.0,
+ "15": 2561334272.0,
+ "16": 2561334272.0,
+ "17": 2561334272.0,
+ "18": 2561334272.0,
+ "19": 2561334272.0,
+ "20": 2561334272.0,
+ "21": 2561334272.0,
+ "22": 2561334272.0,
+ "23": 2561334272.0,
+ "24": 2561334272.0,
+ "25": 2561334272.0,
+ "26": 2561334272.0,
+ "27": 2561334272.0,
+ "28": 2561334272.0,
+ "29": 2561334272.0,
+ "30": 2561334272.0,
+ "31": 2561334272.0,
+ "32": 2561334272.0,
+ "33": 2561334272.0,
+ "34": 2561334272.0,
+ "35": 2561334272.0,
+ "36": 2561334272.0,
+ "37": 2561334272.0,
+ "38": 2561334272.0,
+ "39": 2561334272.0,
+ "40": 2561334272.0,
+ "41": 2561334272.0,
+ "42": 2561334272.0,
+ "43": 2561334272.0,
+ "44": 2561334272.0,
+ "45": 2561334272.0,
+ "46": 2561334272.0,
+ "47": 2561334272.0,
+ "48": 2561334272.0,
+ "49": 2561334272.0,
+ "50": 2561334272.0,
+ "51": 2561334272.0,
+ "52": 2561334272.0,
+ "53": 2561334272.0,
+ "54": 2561334272.0,
+ "55": 2561334272.0,
+ "56": 2561334272.0,
+ "57": 2561334272.0,
+ "58": 2561334272.0,
+ "59": 2561334272.0,
+ "60": 2561334272.0,
+ "61": 2561334272.0,
+ "62": 2561334272.0,
+ "63": 2561334272.0,
+ "64": 2561334272.0,
+ "65": 2561334272.0,
+ "66": 2561334272.0,
+ "67": 2561334272.0,
+ "68": 2561334272.0,
+ "69": 2561334272.0,
+ "70": 2561334272.0,
+ "71": 2561334272.0,
+ "72": 2561334272.0,
+ "73": 2561334272.0,
+ "74": 2561334272.0,
+ "75": 2561334272.0,
+ "76": 2561334272.0,
+ "77": 2561334272.0,
+ "78": 2561334272.0,
+ "79": 2561334272.0,
+ "80": 2561334272.0,
+ "81": 2561334272.0,
+ "82": 2561334272.0,
+ "83": 2561334272.0,
+ "84": 2561334272.0,
+ "85": 2561334272.0,
+ "86": 2561334272.0,
+ "87": 2561334272.0,
+ "88": 2561334272.0,
+ "89": 2561334272.0,
+ "90": 2561334272.0,
+ "91": 2561334272.0,
+ "92": 2561334272.0,
+ "93": 2561334272.0,
+ "94": 2561334272.0,
+ "95": 2561334272.0,
+ "96": 2561334272.0,
+ "97": 2561334272.0,
+ "98": 2561334272.0,
+ "99": 2561334272.0,
+ "100": 2561334272.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 4.95057,
- "2": 0.11272,
- "3": 0.10006,
- "4": 0.0754,
- "5": 0.07446,
- "6": 0.07499,
- "7": 0.07451,
- "8": 0.07507,
- "9": 0.07406,
- "10": 0.07462,
- "11": 0.07387,
- "12": 0.07421,
- "13": 0.07426,
- "14": 0.075,
- "15": 0.07429,
- "16": 0.07394,
- "17": 0.07476,
- "18": 0.07498,
- "19": 0.07455,
- "20": 0.07456,
- "21": 0.07463,
- "22": 0.07473,
- "23": 0.07475,
- "24": 0.0743,
- "25": 0.07447,
- "26": 0.07414,
- "27": 0.07438,
- "28": 0.07665,
- "29": 0.07618,
- "30": 0.07525,
- "31": 0.07718,
- "32": 0.07452,
- "33": 0.07632,
- "34": 0.07594,
- "35": 0.0752,
- "36": 0.07788,
- "37": 0.07472,
- "38": 0.07514,
- "39": 0.07557,
- "40": 0.07528,
- "41": 0.07668,
- "42": 0.07829,
- "43": 0.07561,
- "44": 0.07525,
- "45": 0.07522,
- "46": 0.08858,
- "47": 0.09212,
- "48": 0.07649,
- "49": 0.07761,
- "50": 0.07534,
- "51": 0.0797,
- "52": 0.07601,
- "53": 0.07588,
- "54": 0.07564,
- "55": 0.07643,
- "56": 0.07613,
- "57": 0.07562,
- "58": 0.07558,
- "59": 0.07588,
- "60": 0.07563,
- "61": 0.07585,
- "62": 0.07578,
- "63": 0.07559,
- "64": 0.07502,
- "65": 0.07586,
- "66": 0.07503,
- "67": 0.0755,
- "68": 0.07448,
- "69": 0.07531,
- "70": 0.07481,
- "71": 0.07524,
- "72": 0.07712,
- "73": 0.07539,
- "74": 0.07566,
- "75": 0.07497,
- "76": 0.07458,
- "77": 0.07476,
- "78": 0.07547,
- "79": 0.07542,
- "80": 0.07549,
- "81": 0.07589,
- "82": 0.07548,
- "83": 0.07513,
- "84": 0.07494,
- "85": 0.07468,
- "86": 0.07522,
- "87": 0.07487,
- "88": 0.07533,
- "89": 0.07545,
- "90": 0.07496,
- "91": 0.07533,
- "92": 0.07435,
- "93": 0.07549,
- "94": 0.07465,
- "95": 0.07523,
- "96": 0.07531,
- "97": 0.07697,
- "98": 0.0768,
- "99": 0.07605,
- "100": 0.07588
+ "1": "nan",
+ "2": 2.94993,
+ "3": 0.09052,
+ "4": 0.07772,
+ "5": 0.07713,
+ "6": 0.07804,
+ "7": 0.07721,
+ "8": 0.07741,
+ "9": 0.07681,
+ "10": 0.07767,
+ "11": 0.07745,
+ "12": 0.07735,
+ "13": 0.07745,
+ "14": 0.07752,
+ "15": 0.0772,
+ "16": 0.07754,
+ "17": 0.07728,
+ "18": 0.07717,
+ "19": 0.07735,
+ "20": 0.07767,
+ "21": 0.07753,
+ "22": 0.07865,
+ "23": 0.07743,
+ "24": 0.07719,
+ "25": 0.07694,
+ "26": 0.07727,
+ "27": 0.07704,
+ "28": 0.07751,
+ "29": 0.07909,
+ "30": 0.07755,
+ "31": 0.07747,
+ "32": 0.07793,
+ "33": 0.07775,
+ "34": 0.07767,
+ "35": 0.07895,
+ "36": 0.07778,
+ "37": 0.07804,
+ "38": 0.07849,
+ "39": 0.07721,
+ "40": 0.0776,
+ "41": 0.07733,
+ "42": 0.07794,
+ "43": 0.07718,
+ "44": 0.0777,
+ "45": 0.07814,
+ "46": 0.07784,
+ "47": 0.07704,
+ "48": 0.07805,
+ "49": 0.07738,
+ "50": 0.07756,
+ "51": 0.08289,
+ "52": 0.07867,
+ "53": 0.0812,
+ "54": 0.07887,
+ "55": 0.07729,
+ "56": 0.07816,
+ "57": 0.07795,
+ "58": 0.0779,
+ "59": 0.07712,
+ "60": 0.07953,
+ "61": 0.07717,
+ "62": 0.08919,
+ "63": 0.09914,
+ "64": 0.08846,
+ "65": 0.07738,
+ "66": 0.07846,
+ "67": 0.07715,
+ "68": 0.07879,
+ "69": 0.07736,
+ "70": 0.07843,
+ "71": 0.077,
+ "72": 0.07912,
+ "73": 0.07737,
+ "74": 0.07771,
+ "75": 0.07722,
+ "76": 0.07756,
+ "77": 0.07718,
+ "78": 0.0779,
+ "79": 0.07912,
+ "80": 0.07774,
+ "81": 0.07745,
+ "82": 0.07779,
+ "83": 0.07725,
+ "84": 0.07785,
+ "85": 0.0776,
+ "86": 0.07765,
+ "87": 0.07716,
+ "88": 0.07776,
+ "89": 0.07819,
+ "90": 0.07801,
+ "91": 0.07798,
+ "92": 0.07759,
+ "93": 0.07715,
+ "94": 0.07869,
+ "95": 0.07761,
+ "96": 0.07808,
+ "97": 0.07687,
+ "98": 0.07996,
+ "99": 0.07725,
+ "100": 0.07871
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute/model_config.yaml
index 8a471f2238b..c3ca9477dd7 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_uniform_full_recompute/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_uniform_full_recompute/model_config.yaml
index a5dbe2157e5..8d8d30fc39c 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_uniform_full_recompute/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_uniform_full_recompute/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -53,4 +53,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_cp4_a2a_p2p_nondeterministic/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_cp4_a2a_p2p_nondeterministic/model_config.yaml
index 0310dbf2a64..99de5a5d98c 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_cp4_a2a_p2p_nondeterministic/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_cp4_a2a_p2p_nondeterministic/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: flash
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_cp4_a2a_p2p_nondeterministic/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_cp4_a2a_p2p_nondeterministic/model_config.yaml
index 6d6bf2b5b94..a173a0a5845 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_cp4_a2a_p2p_nondeterministic/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_cp4_a2a_p2p_nondeterministic/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -53,4 +53,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: flash
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings/golden_values_dev_dgx_h100.json
index 1f743e8c2e8..7947d44c6e2 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.85787,
- "2": 10.87336,
- "3": 10.86821,
- "4": 10.87255,
- "5": 10.87398,
- "6": 10.89631,
- "7": 10.86379,
- "8": 10.87834,
- "9": 10.87399,
- "10": 10.83714,
- "11": 10.86988,
- "12": 10.85947,
- "13": 10.87777,
- "14": 10.87924,
- "15": 10.81888,
- "16": 10.83058,
- "17": 10.78684,
- "18": 10.80146,
- "19": 10.79775,
- "20": 10.71155,
- "21": 10.6865,
- "22": 10.55277,
- "23": 10.7014,
- "24": 10.58527,
- "25": 10.52658,
- "26": 10.58299,
- "27": 10.59487,
- "28": 10.54787,
- "29": 10.55928,
- "30": 10.32818,
- "31": 10.08272,
- "32": 10.44699,
- "33": 10.42755,
- "34": 10.17932,
- "35": 10.24095,
- "36": 10.18094,
- "37": 10.32809,
- "38": 10.16727,
- "39": 10.37344,
- "40": 10.05079,
- "41": 10.10728,
- "42": 10.17799,
- "43": 9.77846,
- "44": 9.91207,
- "45": 9.77392,
- "46": 9.75431,
- "47": 10.09497,
- "48": 9.79523,
- "49": 9.46391,
- "50": 9.8673,
- "51": 9.80381,
- "52": 9.68202,
- "53": 10.02345,
- "54": 9.91634,
- "55": 9.82456,
- "56": 9.56974,
- "57": 9.42672,
- "58": 9.78081,
- "59": 9.53243,
- "60": 9.44593,
- "61": 9.64254,
- "62": 9.94293,
- "63": 9.31764,
- "64": 9.72548,
- "65": 8.88739,
- "66": 9.65691,
- "67": 9.31749,
- "68": 9.73495,
- "69": 9.74866,
- "70": 9.69625,
- "71": 9.57689,
- "72": 9.52422,
- "73": 9.45595,
- "74": 8.88269,
- "75": 9.37584,
- "76": 9.01136,
- "77": 10.02287,
- "78": 9.67963,
- "79": 9.33172,
- "80": 9.35826,
- "81": 9.43394,
- "82": 9.65054,
- "83": 9.25503,
- "84": 9.3714,
- "85": 9.5623,
- "86": 9.03489,
- "87": 9.54614,
- "88": 9.69785,
- "89": 9.54656,
- "90": 9.77624,
- "91": 9.2884,
- "92": 9.30662,
- "93": 9.02647,
- "94": 8.78837,
- "95": 9.48027,
- "96": 9.47974,
- "97": 9.25611,
- "98": 9.61949,
- "99": 8.83824,
- "100": 9.35135
+ "1": 10.96096,
+ "2": 10.94378,
+ "3": 10.93765,
+ "4": 10.95048,
+ "5": 10.9484,
+ "6": 10.94213,
+ "7": 10.93466,
+ "8": 10.9418,
+ "9": 10.94159,
+ "10": 10.9279,
+ "11": 10.93407,
+ "12": 10.94464,
+ "13": 10.92188,
+ "14": 10.92236,
+ "15": 10.89968,
+ "16": 10.89038,
+ "17": 10.88908,
+ "18": 10.88184,
+ "19": 10.87644,
+ "20": 10.80072,
+ "21": 10.78158,
+ "22": 10.7619,
+ "23": 10.76243,
+ "24": 10.73665,
+ "25": 10.73432,
+ "26": 10.72096,
+ "27": 10.68131,
+ "28": 10.59563,
+ "29": 10.57428,
+ "30": 10.55167,
+ "31": 10.55268,
+ "32": 10.5296,
+ "33": 10.493,
+ "34": 10.46576,
+ "35": 10.46775,
+ "36": 10.43429,
+ "37": 10.4056,
+ "38": 10.40708,
+ "39": 10.38089,
+ "40": 10.35375,
+ "41": 10.33107,
+ "42": 10.31723,
+ "43": 10.27722,
+ "44": 10.27454,
+ "45": 10.2706,
+ "46": 10.23924,
+ "47": 10.22012,
+ "48": 10.17383,
+ "49": 10.18286,
+ "50": 10.18248,
+ "51": 10.17725,
+ "52": 10.12542,
+ "53": 10.11926,
+ "54": 10.09854,
+ "55": 10.06504,
+ "56": 10.09598,
+ "57": 10.07024,
+ "58": 10.08098,
+ "59": 10.02379,
+ "60": 10.04444,
+ "61": 10.00536,
+ "62": 9.96186,
+ "63": 10.04441,
+ "64": 9.98732,
+ "65": 9.94124,
+ "66": 9.99569,
+ "67": 9.9633,
+ "68": 9.92303,
+ "69": 9.92539,
+ "70": 9.91447,
+ "71": 9.94786,
+ "72": 9.91509,
+ "73": 9.89099,
+ "74": 9.8886,
+ "75": 9.84715,
+ "76": 9.8959,
+ "77": 9.89055,
+ "78": 9.84139,
+ "79": 9.83901,
+ "80": 9.85627,
+ "81": 9.88218,
+ "82": 9.83192,
+ "83": 9.77673,
+ "84": 9.71179,
+ "85": 9.69853,
+ "86": 9.81118,
+ "87": 9.84718,
+ "88": 9.81946,
+ "89": 9.74333,
+ "90": 9.74339,
+ "91": 9.75548,
+ "92": 9.75111,
+ "93": 9.66383,
+ "94": 9.74694,
+ "95": 9.74365,
+ "96": 9.72109,
+ "97": 9.65219,
+ "98": 9.68133,
+ "99": 9.75591,
+ "100": 9.62788
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1858.0,
- "2": 1854.0,
- "3": 1803.0,
- "4": 1955.0,
- "5": 2000.0,
- "6": 2036.0,
- "7": 1932.0,
- "8": 1791.0,
- "9": 1935.0,
- "10": 1654.0,
- "11": 2080.0,
- "12": 1881.0,
- "13": 1977.0,
- "14": 2080.0,
- "15": 1957.0,
- "16": 1910.0,
- "17": 1974.0,
- "18": 1896.0,
- "19": 1955.0,
- "20": 1816.0,
- "21": 1906.0,
- "22": 1972.0,
- "23": 2062.0,
- "24": 1897.0,
- "25": 1830.0,
- "26": 1788.0,
- "27": 1849.0,
- "28": 2008.0,
- "29": 2128.0,
- "30": 1969.0,
- "31": 1630.0,
- "32": 2057.0,
- "33": 2171.0,
- "34": 1947.0,
- "35": 2097.0,
- "36": 1972.0,
- "37": 2348.0,
- "38": 2186.0,
- "39": 2378.0,
- "40": 2181.0,
- "41": 2326.0,
- "42": 2334.0,
- "43": 2219.0,
- "44": 2234.0,
- "45": 2231.0,
- "46": 2229.0,
- "47": 2449.0,
- "48": 2439.0,
- "49": 2159.0,
- "50": 2290.0,
- "51": 2514.0,
- "52": 2513.0,
- "53": 2894.0,
+ "1": 1958.0,
+ "2": 1885.0,
+ "3": 1969.0,
+ "4": 1999.0,
+ "5": 1867.0,
+ "6": 1911.0,
+ "7": 2177.0,
+ "8": 1891.0,
+ "9": 1929.0,
+ "10": 1805.0,
+ "11": 1962.0,
+ "12": 1807.0,
+ "13": 2014.0,
+ "14": 2004.0,
+ "15": 1722.0,
+ "16": 1911.0,
+ "17": 1881.0,
+ "18": 1958.0,
+ "19": 1921.0,
+ "20": 1825.0,
+ "21": 2063.0,
+ "22": 1883.0,
+ "23": 1983.0,
+ "24": 1920.0,
+ "25": 1933.0,
+ "26": 1989.0,
+ "27": 1976.0,
+ "28": 1888.0,
+ "29": 1950.0,
+ "30": 1859.0,
+ "31": 2141.0,
+ "32": 1926.0,
+ "33": 1962.0,
+ "34": 2175.0,
+ "35": 2166.0,
+ "36": 2189.0,
+ "37": 2293.0,
+ "38": 2254.0,
+ "39": 2165.0,
+ "40": 2413.0,
+ "41": 2387.0,
+ "42": 2116.0,
+ "43": 2361.0,
+ "44": 2275.0,
+ "45": 2400.0,
+ "46": 2482.0,
+ "47": 2433.0,
+ "48": 2526.0,
+ "49": 2714.0,
+ "50": 2432.0,
+ "51": 2487.0,
+ "52": 2530.0,
+ "53": 2516.0,
"54": 2656.0,
- "55": 2348.0,
- "56": 2506.0,
- "57": 2501.0,
- "58": 2770.0,
- "59": 2681.0,
- "60": 2434.0,
- "61": 2776.0,
- "62": 2596.0,
- "63": 2617.0,
- "64": 3012.0,
- "65": 2657.0,
- "66": 2947.0,
- "67": 3089.0,
- "68": 2818.0,
- "69": 2909.0,
- "70": 3025.0,
- "71": 2924.0,
- "72": 2702.0,
- "73": 2947.0,
- "74": 2306.0,
- "75": 2791.0,
- "76": 3093.0,
- "77": 3107.0,
- "78": 3134.0,
- "79": 3205.0,
- "80": 3123.0,
- "81": 3290.0,
- "82": 3172.0,
- "83": 2719.0,
- "84": 3328.0,
- "85": 3255.0,
- "86": 2546.0,
- "87": 3472.0,
- "88": 3068.0,
- "89": 2953.0,
- "90": 3300.0,
- "91": 3154.0,
- "92": 3061.0,
- "93": 2889.0,
- "94": 3535.0,
- "95": 3078.0,
- "96": 3181.0,
- "97": 3135.0,
- "98": 3569.0,
- "99": 3319.0,
- "100": 3223.0
+ "55": 2417.0,
+ "56": 2589.0,
+ "57": 2336.0,
+ "58": 3224.0,
+ "59": 2801.0,
+ "60": 2865.0,
+ "61": 2612.0,
+ "62": 2938.0,
+ "63": 3071.0,
+ "64": 3114.0,
+ "65": 2479.0,
+ "66": 2740.0,
+ "67": 3347.0,
+ "68": 3024.0,
+ "69": 2855.0,
+ "70": 2952.0,
+ "71": 2986.0,
+ "72": 3082.0,
+ "73": 3239.0,
+ "74": 3027.0,
+ "75": 3053.0,
+ "76": 3237.0,
+ "77": 3256.0,
+ "78": 3088.0,
+ "79": 2922.0,
+ "80": 2957.0,
+ "81": 3126.0,
+ "82": 3072.0,
+ "83": 2874.0,
+ "84": 2976.0,
+ "85": 2916.0,
+ "86": 3175.0,
+ "87": 2997.0,
+ "88": 3244.0,
+ "89": 3062.0,
+ "90": 3572.0,
+ "91": 2946.0,
+ "92": 3109.0,
+ "93": 2976.0,
+ "94": 3135.0,
+ "95": 3179.0,
+ "96": 3306.0,
+ "97": 3364.0,
+ "98": 3250.0,
+ "99": 3114.0,
+ "100": 2934.0
}
},
"mem-allocated-bytes": {
@@ -325,7 +325,7 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 2236675072.0,
+ "1": 2237723648.0,
"2": 2596141056.0,
"3": 2596141056.0,
"4": 2596141056.0,
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 7.66848,
- "2": 0.11896,
- "3": 0.09977,
- "4": 0.07967,
- "5": 0.07964,
- "6": 0.07997,
- "7": 0.08012,
- "8": 0.07951,
- "9": 0.08093,
- "10": 0.07978,
- "11": 0.07959,
- "12": 0.0801,
- "13": 0.08014,
- "14": 0.08001,
- "15": 0.08005,
- "16": 0.0803,
- "17": 0.0801,
- "18": 0.07861,
- "19": 0.07885,
- "20": 0.07921,
- "21": 0.07891,
- "22": 0.07852,
- "23": 0.07915,
- "24": 0.07938,
- "25": 0.08,
- "26": 0.0813,
- "27": 0.07978,
- "28": 0.07899,
- "29": 0.0798,
- "30": 0.08028,
- "31": 0.07891,
- "32": 0.07911,
- "33": 0.07848,
- "34": 0.07925,
- "35": 0.07821,
- "36": 0.07899,
- "37": 0.07887,
- "38": 0.07866,
- "39": 0.07853,
- "40": 0.08169,
- "41": 0.07849,
- "42": 0.07836,
- "43": 0.0786,
- "44": 0.07878,
- "45": 0.07828,
- "46": 0.07805,
- "47": 0.07784,
- "48": 0.07807,
- "49": 0.0787,
- "50": 0.0789,
- "51": 0.09689,
- "52": 0.08417,
- "53": 0.08482,
- "54": 0.08198,
- "55": 0.07942,
- "56": 0.07871,
- "57": 0.07976,
- "58": 0.07956,
- "59": 0.08,
- "60": 0.0792,
- "61": 0.07836,
- "62": 0.07989,
- "63": 0.0809,
- "64": 0.08148,
- "65": 0.08043,
- "66": 0.07986,
- "67": 0.08023,
- "68": 0.07899,
- "69": 0.07929,
- "70": 0.08168,
- "71": 0.08127,
- "72": 0.0786,
- "73": 0.07921,
- "74": 0.07909,
- "75": 0.0791,
- "76": 0.07958,
- "77": 0.07852,
- "78": 0.07999,
- "79": 0.07999,
- "80": 0.08194,
- "81": 0.07923,
- "82": 0.07928,
- "83": 0.07876,
- "84": 0.07871,
- "85": 0.08021,
- "86": 0.07922,
- "87": 0.07979,
- "88": 0.0797,
- "89": 0.08029,
- "90": 0.15516,
- "91": 0.11731,
- "92": 0.11011,
- "93": 0.14646,
- "94": 0.08003,
- "95": 0.08107,
- "96": 0.07984,
- "97": 0.07889,
- "98": 0.07881,
- "99": 0.07894,
- "100": 0.07813
+ "1": "nan",
+ "2": 5.09236,
+ "3": 0.11369,
+ "4": 0.0836,
+ "5": 0.08363,
+ "6": 0.08353,
+ "7": 0.08499,
+ "8": 0.08401,
+ "9": 0.08378,
+ "10": 0.08398,
+ "11": 0.08269,
+ "12": 0.08311,
+ "13": 0.08421,
+ "14": 0.08391,
+ "15": 0.08322,
+ "16": 0.08289,
+ "17": 0.08271,
+ "18": 0.08299,
+ "19": 0.08394,
+ "20": 0.08343,
+ "21": 0.08279,
+ "22": 0.08737,
+ "23": 0.08278,
+ "24": 0.08356,
+ "25": 0.08393,
+ "26": 0.08691,
+ "27": 0.08308,
+ "28": 0.08329,
+ "29": 0.08348,
+ "30": 0.0838,
+ "31": 0.08325,
+ "32": 0.084,
+ "33": 0.08338,
+ "34": 0.08332,
+ "35": 0.08339,
+ "36": 0.08319,
+ "37": 0.08381,
+ "38": 0.08369,
+ "39": 0.08365,
+ "40": 0.0841,
+ "41": 0.08363,
+ "42": 0.08359,
+ "43": 0.08156,
+ "44": 0.08844,
+ "45": 0.08687,
+ "46": 0.08493,
+ "47": 0.08389,
+ "48": 0.0829,
+ "49": 0.08484,
+ "50": 0.0876,
+ "51": 0.13296,
+ "52": 0.08685,
+ "53": 0.08742,
+ "54": 0.08748,
+ "55": 0.08101,
+ "56": 0.08854,
+ "57": 0.08722,
+ "58": 0.08724,
+ "59": 0.08693,
+ "60": 0.08819,
+ "61": 0.08829,
+ "62": 0.08759,
+ "63": 0.08811,
+ "64": 0.0867,
+ "65": 0.08124,
+ "66": 0.08187,
+ "67": 0.0822,
+ "68": 0.0821,
+ "69": 0.08201,
+ "70": 0.082,
+ "71": 0.08192,
+ "72": 0.08266,
+ "73": 0.0819,
+ "74": 0.08169,
+ "75": 0.0822,
+ "76": 0.08224,
+ "77": 0.08228,
+ "78": 0.08239,
+ "79": 0.08225,
+ "80": 0.08256,
+ "81": 0.08248,
+ "82": 0.0832,
+ "83": 0.08196,
+ "84": 0.08218,
+ "85": 0.08204,
+ "86": 0.08235,
+ "87": 0.08235,
+ "88": 0.082,
+ "89": 0.08171,
+ "90": 0.08191,
+ "91": 0.08193,
+ "92": 0.08185,
+ "93": 0.08235,
+ "94": 0.08227,
+ "95": 0.08229,
+ "96": 0.08213,
+ "97": 0.08197,
+ "98": 0.08215,
+ "99": 0.08239,
+ "100": 0.0822
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings/model_config.yaml
index 5e4131a43ca..9cde3247944 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -40,7 +40,6 @@ MODEL_ARGS:
--tensor-model-parallel-size: 1
--pipeline-model-parallel-size: 2
--position-embedding-type: rope
- --no-ckpt-fully-parallel-save: true
--deterministic-mode: true
--no-gradient-accumulation-fusion: true
--attention-softmax-in-fp32: true
@@ -51,4 +50,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion/golden_values_dev_dgx_h100.json
index b1c227e9ae3..fe8aeab6d9d 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.85902,
- "2": 10.87345,
- "3": 10.86919,
- "4": 10.87273,
- "5": 10.87389,
- "6": 10.89658,
- "7": 10.86387,
- "8": 10.87869,
- "9": 10.87439,
- "10": 10.83846,
- "11": 10.87012,
- "12": 10.86011,
- "13": 10.87824,
- "14": 10.87935,
- "15": 10.8191,
- "16": 10.83109,
- "17": 10.78722,
- "18": 10.80215,
- "19": 10.7983,
- "20": 10.71224,
- "21": 10.68683,
- "22": 10.55402,
- "23": 10.70111,
- "24": 10.58621,
- "25": 10.52673,
- "26": 10.5837,
- "27": 10.59499,
- "28": 10.54816,
- "29": 10.55965,
- "30": 10.32899,
- "31": 10.08331,
- "32": 10.44752,
- "33": 10.4278,
- "34": 10.1796,
- "35": 10.24121,
- "36": 10.18155,
- "37": 10.32827,
- "38": 10.16792,
- "39": 10.37357,
- "40": 10.05111,
- "41": 10.10708,
- "42": 10.17823,
- "43": 9.77867,
- "44": 9.91197,
- "45": 9.77404,
- "46": 9.75415,
- "47": 10.09501,
- "48": 9.79531,
- "49": 9.46422,
- "50": 9.86729,
- "51": 9.80375,
- "52": 9.68218,
- "53": 10.02348,
- "54": 9.91595,
- "55": 9.82442,
- "56": 9.56994,
- "57": 9.42628,
- "58": 9.78075,
- "59": 9.53254,
- "60": 9.44561,
- "61": 9.64249,
- "62": 9.94298,
- "63": 9.31745,
- "64": 9.7256,
- "65": 8.88735,
- "66": 9.65711,
- "67": 9.31747,
- "68": 9.73506,
- "69": 9.74863,
- "70": 9.69601,
- "71": 9.57682,
- "72": 9.52425,
- "73": 9.4558,
- "74": 8.8826,
- "75": 9.37563,
- "76": 9.01106,
- "77": 10.02278,
- "78": 9.6796,
- "79": 9.33171,
- "80": 9.35836,
- "81": 9.43399,
- "82": 9.65055,
- "83": 9.2551,
- "84": 9.37131,
- "85": 9.56237,
- "86": 9.0351,
- "87": 9.54617,
- "88": 9.69806,
- "89": 9.54657,
- "90": 9.77627,
- "91": 9.28858,
- "92": 9.30652,
- "93": 9.02646,
- "94": 8.7883,
- "95": 9.48041,
- "96": 9.47962,
- "97": 9.25545,
- "98": 9.61947,
- "99": 8.83854,
- "100": 9.35116
+ "1": 10.96078,
+ "2": 10.94368,
+ "3": 10.93755,
+ "4": 10.95045,
+ "5": 10.94855,
+ "6": 10.94271,
+ "7": 10.93509,
+ "8": 10.94175,
+ "9": 10.9418,
+ "10": 10.92772,
+ "11": 10.93415,
+ "12": 10.94481,
+ "13": 10.92209,
+ "14": 10.92274,
+ "15": 10.89968,
+ "16": 10.89042,
+ "17": 10.88898,
+ "18": 10.88214,
+ "19": 10.87681,
+ "20": 10.80092,
+ "21": 10.7814,
+ "22": 10.76173,
+ "23": 10.76218,
+ "24": 10.73668,
+ "25": 10.73447,
+ "26": 10.72102,
+ "27": 10.68102,
+ "28": 10.59586,
+ "29": 10.5744,
+ "30": 10.55199,
+ "31": 10.55319,
+ "32": 10.52963,
+ "33": 10.49301,
+ "34": 10.46587,
+ "35": 10.4678,
+ "36": 10.43443,
+ "37": 10.40559,
+ "38": 10.40696,
+ "39": 10.38092,
+ "40": 10.35372,
+ "41": 10.33115,
+ "42": 10.31725,
+ "43": 10.27748,
+ "44": 10.27477,
+ "45": 10.27092,
+ "46": 10.23937,
+ "47": 10.22002,
+ "48": 10.17409,
+ "49": 10.18285,
+ "50": 10.18251,
+ "51": 10.17734,
+ "52": 10.12546,
+ "53": 10.11937,
+ "54": 10.09849,
+ "55": 10.06507,
+ "56": 10.09605,
+ "57": 10.07033,
+ "58": 10.08106,
+ "59": 10.02397,
+ "60": 10.04465,
+ "61": 10.00554,
+ "62": 9.9619,
+ "63": 10.04448,
+ "64": 9.98728,
+ "65": 9.94154,
+ "66": 9.99578,
+ "67": 9.96346,
+ "68": 9.92329,
+ "69": 9.92558,
+ "70": 9.91456,
+ "71": 9.94801,
+ "72": 9.91519,
+ "73": 9.89109,
+ "74": 9.88874,
+ "75": 9.84731,
+ "76": 9.89603,
+ "77": 9.89058,
+ "78": 9.84152,
+ "79": 9.83897,
+ "80": 9.85635,
+ "81": 9.88224,
+ "82": 9.83208,
+ "83": 9.77655,
+ "84": 9.7121,
+ "85": 9.69859,
+ "86": 9.81129,
+ "87": 9.84706,
+ "88": 9.81953,
+ "89": 9.74315,
+ "90": 9.74319,
+ "91": 9.75538,
+ "92": 9.75092,
+ "93": 9.66354,
+ "94": 9.74705,
+ "95": 9.74357,
+ "96": 9.72111,
+ "97": 9.6524,
+ "98": 9.68128,
+ "99": 9.75571,
+ "100": 9.62797
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1862.0,
- "2": 1874.0,
- "3": 1748.0,
- "4": 1955.0,
- "5": 2050.0,
- "6": 1997.0,
- "7": 1967.0,
- "8": 1853.0,
- "9": 1965.0,
- "10": 1652.0,
- "11": 2042.0,
- "12": 1877.0,
- "13": 2076.0,
- "14": 1956.0,
- "15": 1953.0,
- "16": 1915.0,
- "17": 2045.0,
- "18": 1965.0,
- "19": 1988.0,
- "20": 1785.0,
- "21": 1941.0,
- "22": 1928.0,
- "23": 2112.0,
- "24": 1802.0,
- "25": 1933.0,
- "26": 1786.0,
- "27": 1945.0,
- "28": 2037.0,
- "29": 2119.0,
- "30": 2022.0,
- "31": 1699.0,
- "32": 2130.0,
- "33": 2187.0,
- "34": 1929.0,
- "35": 2092.0,
- "36": 2109.0,
- "37": 2362.0,
- "38": 2211.0,
- "39": 2383.0,
- "40": 2203.0,
- "41": 2288.0,
- "42": 2224.0,
- "43": 2150.0,
- "44": 2206.0,
- "45": 2187.0,
- "46": 2181.0,
- "47": 2260.0,
- "48": 2341.0,
- "49": 2210.0,
- "50": 2219.0,
- "51": 2508.0,
- "52": 2483.0,
- "53": 2959.0,
- "54": 2554.0,
- "55": 2408.0,
- "56": 2452.0,
- "57": 2528.0,
- "58": 2594.0,
- "59": 2750.0,
- "60": 2563.0,
- "61": 2794.0,
- "62": 2495.0,
- "63": 2493.0,
- "64": 2965.0,
- "65": 2569.0,
- "66": 2877.0,
- "67": 2969.0,
- "68": 2803.0,
- "69": 2944.0,
- "70": 3001.0,
- "71": 2867.0,
- "72": 2714.0,
- "73": 3017.0,
- "74": 2281.0,
- "75": 2774.0,
- "76": 2983.0,
- "77": 2955.0,
- "78": 3148.0,
- "79": 3076.0,
- "80": 2992.0,
- "81": 3255.0,
- "82": 3212.0,
- "83": 2809.0,
- "84": 3266.0,
- "85": 3188.0,
- "86": 2616.0,
- "87": 3492.0,
- "88": 3130.0,
- "89": 3020.0,
- "90": 3238.0,
- "91": 3106.0,
- "92": 3183.0,
- "93": 2960.0,
- "94": 3492.0,
- "95": 3112.0,
- "96": 3256.0,
- "97": 3055.0,
- "98": 3558.0,
- "99": 3196.0,
- "100": 3109.0
+ "1": 1951.0,
+ "2": 1836.0,
+ "3": 1829.0,
+ "4": 1977.0,
+ "5": 1855.0,
+ "6": 1949.0,
+ "7": 2113.0,
+ "8": 1918.0,
+ "9": 1917.0,
+ "10": 1879.0,
+ "11": 1898.0,
+ "12": 1855.0,
+ "13": 1993.0,
+ "14": 1963.0,
+ "15": 1830.0,
+ "16": 1834.0,
+ "17": 1862.0,
+ "18": 1921.0,
+ "19": 1951.0,
+ "20": 1803.0,
+ "21": 1938.0,
+ "22": 1942.0,
+ "23": 1907.0,
+ "24": 1944.0,
+ "25": 1867.0,
+ "26": 1979.0,
+ "27": 1989.0,
+ "28": 1871.0,
+ "29": 2010.0,
+ "30": 1870.0,
+ "31": 2144.0,
+ "32": 1920.0,
+ "33": 1965.0,
+ "34": 2072.0,
+ "35": 2086.0,
+ "36": 2071.0,
+ "37": 2282.0,
+ "38": 2141.0,
+ "39": 2184.0,
+ "40": 2371.0,
+ "41": 2347.0,
+ "42": 2112.0,
+ "43": 2355.0,
+ "44": 2332.0,
+ "45": 2484.0,
+ "46": 2513.0,
+ "47": 2428.0,
+ "48": 2614.0,
+ "49": 2868.0,
+ "50": 2505.0,
+ "51": 2437.0,
+ "52": 2516.0,
+ "53": 2550.0,
+ "54": 2646.0,
+ "55": 2463.0,
+ "56": 2620.0,
+ "57": 2377.0,
+ "58": 3132.0,
+ "59": 2845.0,
+ "60": 2837.0,
+ "61": 2673.0,
+ "62": 2874.0,
+ "63": 3098.0,
+ "64": 3190.0,
+ "65": 2558.0,
+ "66": 2845.0,
+ "67": 3258.0,
+ "68": 2931.0,
+ "69": 2880.0,
+ "70": 2961.0,
+ "71": 2901.0,
+ "72": 3087.0,
+ "73": 3297.0,
+ "74": 3024.0,
+ "75": 2902.0,
+ "76": 3226.0,
+ "77": 3261.0,
+ "78": 3067.0,
+ "79": 2946.0,
+ "80": 2975.0,
+ "81": 3235.0,
+ "82": 3084.0,
+ "83": 2974.0,
+ "84": 2866.0,
+ "85": 2947.0,
+ "86": 3183.0,
+ "87": 3042.0,
+ "88": 3275.0,
+ "89": 3019.0,
+ "90": 3411.0,
+ "91": 2894.0,
+ "92": 3121.0,
+ "93": 3029.0,
+ "94": 3302.0,
+ "95": 3083.0,
+ "96": 3302.0,
+ "97": 3218.0,
+ "98": 3121.0,
+ "99": 3023.0,
+ "100": 2930.0
}
},
"mem-allocated-bytes": {
@@ -325,7 +325,7 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 2239820800.0,
+ "1": 2240869376.0,
"2": 2599286784.0,
"3": 2599286784.0,
"4": 2599286784.0,
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 7.15273,
- "2": 0.12761,
- "3": 0.108,
- "4": 0.08804,
- "5": 0.08914,
- "6": 0.0872,
- "7": 0.0865,
- "8": 0.09025,
- "9": 0.09224,
- "10": 0.08785,
- "11": 0.08842,
- "12": 0.08678,
- "13": 0.08768,
- "14": 0.08732,
- "15": 0.08754,
- "16": 0.08689,
- "17": 0.08745,
- "18": 0.08749,
- "19": 0.08681,
- "20": 0.08755,
- "21": 0.08798,
- "22": 0.08687,
- "23": 0.0869,
- "24": 0.08666,
- "25": 0.08694,
- "26": 0.08728,
- "27": 0.08672,
- "28": 0.09131,
- "29": 0.09876,
- "30": 0.09345,
- "31": 0.0871,
- "32": 0.08745,
- "33": 0.0868,
- "34": 0.08664,
- "35": 0.08688,
- "36": 0.08685,
- "37": 0.08807,
- "38": 0.08807,
- "39": 0.09095,
- "40": 0.08728,
- "41": 0.08918,
- "42": 0.0874,
- "43": 0.08812,
- "44": 0.08765,
- "45": 0.08765,
- "46": 0.08695,
- "47": 0.08967,
- "48": 0.08734,
- "49": 0.08707,
- "50": 0.08818,
- "51": 0.09801,
- "52": 0.09366,
- "53": 0.09478,
- "54": 0.09027,
- "55": 0.08632,
- "56": 0.0857,
- "57": 0.08636,
- "58": 0.08585,
- "59": 0.08632,
- "60": 0.08559,
- "61": 0.08575,
- "62": 0.08716,
- "63": 0.08612,
- "64": 0.08569,
- "65": 0.0876,
- "66": 0.08587,
- "67": 0.0862,
- "68": 0.08594,
- "69": 0.0858,
- "70": 0.08668,
- "71": 0.08553,
- "72": 0.08961,
- "73": 0.09562,
- "74": 0.09156,
- "75": 0.0901,
- "76": 0.08615,
- "77": 0.08562,
- "78": 0.08664,
- "79": 0.08569,
- "80": 0.08621,
- "81": 0.08562,
- "82": 0.08601,
- "83": 0.08551,
- "84": 0.08569,
- "85": 0.08622,
- "86": 0.08639,
- "87": 0.08581,
- "88": 0.08569,
- "89": 0.08624,
- "90": 0.086,
- "91": 0.08602,
- "92": 0.08575,
- "93": 0.08626,
- "94": 0.0869,
- "95": 0.0867,
- "96": 0.0872,
- "97": 0.08727,
- "98": 0.08652,
- "99": 0.0867,
- "100": 0.08593
+ "1": "nan",
+ "2": 4.89526,
+ "3": 0.10937,
+ "4": 0.09062,
+ "5": 0.09018,
+ "6": 0.09011,
+ "7": 0.08972,
+ "8": 0.09092,
+ "9": 0.08947,
+ "10": 0.08976,
+ "11": 0.09014,
+ "12": 0.09787,
+ "13": 0.10092,
+ "14": 0.09841,
+ "15": 0.09142,
+ "16": 0.09294,
+ "17": 0.09177,
+ "18": 0.08946,
+ "19": 0.09259,
+ "20": 0.09206,
+ "21": 0.09371,
+ "22": 0.09206,
+ "23": 0.09206,
+ "24": 0.09026,
+ "25": 0.0893,
+ "26": 0.08943,
+ "27": 0.08942,
+ "28": 0.0893,
+ "29": 0.08927,
+ "30": 0.0894,
+ "31": 0.08915,
+ "32": 0.09409,
+ "33": 0.09036,
+ "34": 0.0908,
+ "35": 0.08883,
+ "36": 0.09011,
+ "37": 0.08962,
+ "38": 0.08961,
+ "39": 0.08912,
+ "40": 0.08979,
+ "41": 0.08942,
+ "42": 0.08955,
+ "43": 0.08935,
+ "44": 0.08987,
+ "45": 0.08959,
+ "46": 0.08945,
+ "47": 0.09005,
+ "48": 0.08975,
+ "49": 0.08931,
+ "50": 0.08929,
+ "51": 0.12301,
+ "52": 0.08736,
+ "53": 0.0888,
+ "54": 0.08895,
+ "55": 0.08987,
+ "56": 0.08948,
+ "57": 0.09021,
+ "58": 0.09079,
+ "59": 0.08914,
+ "60": 0.08963,
+ "61": 0.08936,
+ "62": 0.08957,
+ "63": 0.09096,
+ "64": 0.08773,
+ "65": 0.0896,
+ "66": 0.09287,
+ "67": 0.08937,
+ "68": 0.0891,
+ "69": 0.08928,
+ "70": 0.09031,
+ "71": 0.08941,
+ "72": 0.08903,
+ "73": 0.08891,
+ "74": 0.08947,
+ "75": 0.08874,
+ "76": 0.08895,
+ "77": 0.08791,
+ "78": 0.08908,
+ "79": 0.08889,
+ "80": 0.08837,
+ "81": 0.0891,
+ "82": 0.08905,
+ "83": 0.0891,
+ "84": 0.08917,
+ "85": 0.08889,
+ "86": 0.08863,
+ "87": 0.08867,
+ "88": 0.08899,
+ "89": 0.0896,
+ "90": 0.0889,
+ "91": 0.08907,
+ "92": 0.08836,
+ "93": 0.08881,
+ "94": 0.08858,
+ "95": 0.089,
+ "96": 0.09006,
+ "97": 0.08932,
+ "98": 0.08885,
+ "99": 0.08878,
+ "100": 0.08932
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion/model_config.yaml
index 37b5f5d7471..b4e07b5b5e1 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_rope_embeddings/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_rope_embeddings/model_config.yaml
index 7ddf65f4ca8..e80995f90a3 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_rope_embeddings/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_rope_embeddings/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_rope_embeddings_interleaved_no_fusion/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_rope_embeddings_interleaved_no_fusion/model_config.yaml
index b391387f9ff..98c7db9b9bd 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_rope_embeddings_interleaved_no_fusion/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_rope_embeddings_interleaved_no_fusion/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -53,4 +53,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_disable_bias_linear/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_disable_bias_linear/model_config.yaml
index 5415e3de96d..8d01a9132eb 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_disable_bias_linear/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_disable_bias_linear/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,5 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_frozen_resume_torch_dist_swiglu/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_frozen_resume_torch_dist_swiglu/model_config.yaml
index 8d372f5539d..16f9ba79fe4 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_frozen_resume_torch_dist_swiglu/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_frozen_resume_torch_dist_swiglu/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -53,4 +53,5 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: frozen-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_persistent_ckpt_disable_bias_linear/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_persistent_ckpt_disable_bias_linear/model_config.yaml
index 9a3947b5e71..89b883024df 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_persistent_ckpt_disable_bias_linear/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_persistent_ckpt_disable_bias_linear/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear/golden_values_dev_dgx_gb200.json
index 63c74381364..947616883e3 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear/golden_values_dev_dgx_gb200.json
@@ -6,104 +6,104 @@
"values": {
"1": 10.83582,
"2": 10.83571,
- "3": 10.83523,
+ "3": 10.83524,
"4": 10.79949,
"5": 10.84909,
- "6": 10.86563,
- "7": 10.82789,
+ "6": 10.86567,
+ "7": 10.82786,
"8": 10.8363,
"9": 10.83997,
- "10": 10.79865,
- "11": 10.8677,
- "12": 10.84994,
- "13": 10.85915,
- "14": 10.86874,
- "15": 10.80173,
- "16": 10.79183,
- "17": 10.77353,
- "18": 10.78739,
- "19": 10.78983,
- "20": 10.68446,
- "21": 10.6784,
- "22": 10.5257,
- "23": 10.70726,
- "24": 10.56551,
- "25": 10.51602,
- "26": 10.58017,
- "27": 10.58981,
+ "10": 10.79867,
+ "11": 10.86763,
+ "12": 10.84992,
+ "13": 10.85912,
+ "14": 10.8687,
+ "15": 10.80171,
+ "16": 10.79189,
+ "17": 10.77351,
+ "18": 10.78742,
+ "19": 10.78986,
+ "20": 10.68447,
+ "21": 10.67839,
+ "22": 10.52572,
+ "23": 10.70729,
+ "24": 10.56549,
+ "25": 10.51603,
+ "26": 10.58018,
+ "27": 10.58977,
"28": 10.54551,
- "29": 10.57726,
- "30": 10.34051,
- "31": 10.07051,
- "32": 10.44503,
- "33": 10.44293,
- "34": 10.19391,
- "35": 10.24261,
- "36": 10.19236,
- "37": 10.32969,
+ "29": 10.57724,
+ "30": 10.34049,
+ "31": 10.07056,
+ "32": 10.44508,
+ "33": 10.44289,
+ "34": 10.19394,
+ "35": 10.2426,
+ "36": 10.19235,
+ "37": 10.32972,
"38": 10.16551,
"39": 10.38729,
"40": 10.05174,
- "41": 10.12191,
- "42": 10.19259,
- "43": 9.8069,
- "44": 9.92475,
- "45": 9.80639,
- "46": 9.80145,
- "47": 10.12104,
- "48": 9.83127,
- "49": 9.50404,
- "50": 9.87954,
+ "41": 10.12185,
+ "42": 10.19258,
+ "43": 9.80694,
+ "44": 9.92474,
+ "45": 9.80636,
+ "46": 9.80144,
+ "47": 10.12106,
+ "48": 9.83126,
+ "49": 9.50406,
+ "50": 9.87955,
"51": 9.83807,
- "52": 9.72058,
- "53": 10.0568,
- "54": 9.95032,
- "55": 9.88328,
- "56": 9.60431,
+ "52": 9.72057,
+ "53": 10.05682,
+ "54": 9.95031,
+ "55": 9.88332,
+ "56": 9.60428,
"57": 9.45518,
- "58": 9.81927,
- "59": 9.58262,
+ "58": 9.81923,
+ "59": 9.58266,
"60": 9.48844,
- "61": 9.68577,
- "62": 9.97779,
+ "61": 9.68574,
+ "62": 9.9778,
"63": 9.36765,
- "64": 9.75913,
- "65": 8.9376,
- "66": 9.69257,
- "67": 9.36621,
- "68": 9.78303,
- "69": 9.79318,
- "70": 9.72699,
+ "64": 9.75912,
+ "65": 8.93762,
+ "66": 9.6926,
+ "67": 9.36619,
+ "68": 9.78309,
+ "69": 9.79315,
+ "70": 9.72695,
"71": 9.62875,
- "72": 9.58004,
+ "72": 9.58006,
"73": 9.487,
- "74": 8.92041,
- "75": 9.41128,
- "76": 9.07564,
- "77": 10.05848,
- "78": 9.72184,
- "79": 9.3732,
- "80": 9.40079,
- "81": 9.4792,
- "82": 9.69754,
- "83": 9.31037,
- "84": 9.41777,
- "85": 9.61194,
- "86": 9.07155,
- "87": 9.59661,
- "88": 9.74709,
- "89": 9.59667,
- "90": 9.82915,
- "91": 9.33725,
- "92": 9.3564,
- "93": 9.08552,
- "94": 8.82807,
- "95": 9.52842,
- "96": 9.52611,
- "97": 9.30632,
- "98": 9.66808,
- "99": 8.89461,
- "100": 9.40666
+ "74": 8.92045,
+ "75": 9.41127,
+ "76": 9.0757,
+ "77": 10.05849,
+ "78": 9.72185,
+ "79": 9.37321,
+ "80": 9.40082,
+ "81": 9.47926,
+ "82": 9.69753,
+ "83": 9.31033,
+ "84": 9.41773,
+ "85": 9.61195,
+ "86": 9.07158,
+ "87": 9.59659,
+ "88": 9.74711,
+ "89": 9.59669,
+ "90": 9.82914,
+ "91": 9.33728,
+ "92": 9.35642,
+ "93": 9.08554,
+ "94": 8.82803,
+ "95": 9.52843,
+ "96": 9.52607,
+ "97": 9.30634,
+ "98": 9.66809,
+ "99": 8.89459,
+ "100": 9.40668
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1536.0,
- "2": 1592.0,
- "3": 1551.0,
- "4": 1769.0,
- "5": 1824.0,
- "6": 1800.0,
- "7": 1734.0,
- "8": 1619.0,
- "9": 1829.0,
- "10": 1355.0,
- "11": 1911.0,
- "12": 1721.0,
- "13": 1913.0,
- "14": 1708.0,
- "15": 1919.0,
- "16": 1938.0,
- "17": 1740.0,
- "18": 1676.0,
- "19": 1743.0,
- "20": 1535.0,
- "21": 1797.0,
- "22": 1661.0,
- "23": 1887.0,
- "24": 1666.0,
- "25": 1633.0,
- "26": 1676.0,
- "27": 1740.0,
- "28": 1991.0,
- "29": 1918.0,
- "30": 1806.0,
- "31": 1588.0,
- "32": 1863.0,
- "33": 2126.0,
- "34": 1812.0,
- "35": 1976.0,
- "36": 1875.0,
- "37": 2301.0,
- "38": 2131.0,
- "39": 2351.0,
- "40": 2130.0,
- "41": 2391.0,
- "42": 2255.0,
- "43": 1975.0,
- "44": 2138.0,
- "45": 2208.0,
- "46": 2364.0,
- "47": 2564.0,
- "48": 2337.0,
- "49": 2142.0,
- "50": 2423.0,
- "51": 2546.0,
- "52": 2590.0,
- "53": 2879.0,
- "54": 2697.0,
- "55": 2316.0,
- "56": 2549.0,
- "57": 2261.0,
- "58": 2904.0,
- "59": 2740.0,
- "60": 2434.0,
- "61": 2801.0,
- "62": 2663.0,
- "63": 2502.0,
- "64": 2948.0,
- "65": 2644.0,
- "66": 2961.0,
- "67": 2813.0,
- "68": 2686.0,
- "69": 2912.0,
- "70": 3096.0,
- "71": 2854.0,
- "72": 2454.0,
- "73": 3081.0,
- "74": 1933.0,
- "75": 2465.0,
- "76": 3012.0,
- "77": 3163.0,
- "78": 2997.0,
- "79": 3089.0,
- "80": 3187.0,
- "81": 3500.0,
- "82": 3339.0,
- "83": 2705.0,
- "84": 3205.0,
- "85": 3033.0,
- "86": 2818.0,
- "87": 3671.0,
- "88": 3190.0,
- "89": 3336.0,
- "90": 3320.0,
- "91": 2698.0,
- "92": 3072.0,
- "93": 2750.0,
- "94": 3397.0,
- "95": 3317.0,
- "96": 3290.0,
- "97": 3116.0,
- "98": 3732.0,
- "99": 3049.0,
- "100": 2974.0
+ "1": 1501.0,
+ "2": 1576.0,
+ "3": 1604.0,
+ "4": 1778.0,
+ "5": 1880.0,
+ "6": 1803.0,
+ "7": 1789.0,
+ "8": 1669.0,
+ "9": 1811.0,
+ "10": 1392.0,
+ "11": 1835.0,
+ "12": 1663.0,
+ "13": 1855.0,
+ "14": 1841.0,
+ "15": 1874.0,
+ "16": 1844.0,
+ "17": 1738.0,
+ "18": 1724.0,
+ "19": 1742.0,
+ "20": 1612.0,
+ "21": 1751.0,
+ "22": 1713.0,
+ "23": 1964.0,
+ "24": 1632.0,
+ "25": 1570.0,
+ "26": 1675.0,
+ "27": 1752.0,
+ "28": 2026.0,
+ "29": 1938.0,
+ "30": 1848.0,
+ "31": 1557.0,
+ "32": 1926.0,
+ "33": 2052.0,
+ "34": 1880.0,
+ "35": 2022.0,
+ "36": 1926.0,
+ "37": 2344.0,
+ "38": 2202.0,
+ "39": 2285.0,
+ "40": 2225.0,
+ "41": 2328.0,
+ "42": 2200.0,
+ "43": 1984.0,
+ "44": 2142.0,
+ "45": 2173.0,
+ "46": 2308.0,
+ "47": 2592.0,
+ "48": 2460.0,
+ "49": 2242.0,
+ "50": 2383.0,
+ "51": 2489.0,
+ "52": 2497.0,
+ "53": 2875.0,
+ "54": 2654.0,
+ "55": 2317.0,
+ "56": 2599.0,
+ "57": 2299.0,
+ "58": 2830.0,
+ "59": 2784.0,
+ "60": 2437.0,
+ "61": 2916.0,
+ "62": 2599.0,
+ "63": 2388.0,
+ "64": 2785.0,
+ "65": 2677.0,
+ "66": 2972.0,
+ "67": 2797.0,
+ "68": 2752.0,
+ "69": 3049.0,
+ "70": 3087.0,
+ "71": 2952.0,
+ "72": 2411.0,
+ "73": 3099.0,
+ "74": 1975.0,
+ "75": 2614.0,
+ "76": 2941.0,
+ "77": 3166.0,
+ "78": 3123.0,
+ "79": 3085.0,
+ "80": 3198.0,
+ "81": 3350.0,
+ "82": 3322.0,
+ "83": 2858.0,
+ "84": 3125.0,
+ "85": 3194.0,
+ "86": 2777.0,
+ "87": 3602.0,
+ "88": 3006.0,
+ "89": 3267.0,
+ "90": 3133.0,
+ "91": 2753.0,
+ "92": 3113.0,
+ "93": 2714.0,
+ "94": 3364.0,
+ "95": 3273.0,
+ "96": 3202.0,
+ "97": 3124.0,
+ "98": 3716.0,
+ "99": 3121.0,
+ "100": 3131.0
}
},
"mem-allocated-bytes": {
@@ -325,7 +325,7 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1939785728.0,
+ "1": 1938737152.0,
"2": 2222434304.0,
"3": 2222434304.0,
"4": 2222434304.0,
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.20838,
- "3": 0.13042,
- "4": 0.11826,
- "5": 0.11718,
- "6": 0.11797,
- "7": 0.1177,
- "8": 0.11717,
- "9": 0.11846,
- "10": 0.11778,
- "11": 0.11712,
- "12": 0.11866,
- "13": 0.12004,
- "14": 0.11788,
- "15": 0.11787,
- "16": 0.1181,
- "17": 0.11903,
- "18": 0.11843,
- "19": 0.11754,
- "20": 0.11834,
- "21": 0.11897,
- "22": 0.12726,
- "23": 0.13834,
- "24": 0.15039,
- "25": 0.14107,
- "26": 0.14586,
- "27": 0.16343,
- "28": 0.2297,
- "29": 0.26681,
- "30": 0.19748,
- "31": 0.2586,
- "32": 0.12068,
- "33": 0.11944,
- "34": 0.11896,
- "35": 0.11984,
- "36": 0.11823,
- "37": 0.11997,
- "38": 0.11949,
- "39": 0.11877,
- "40": 0.11898,
- "41": 0.11996,
- "42": 0.11893,
- "43": 0.12547,
- "44": 0.13195,
- "45": 0.12144,
- "46": 0.11997,
- "47": 0.12005,
- "48": 0.11855,
- "49": 0.11944,
- "50": 0.11842,
- "51": 0.14635,
- "52": 0.12016,
- "53": 0.11762,
- "54": 0.11802,
- "55": 0.1184,
- "56": 0.11774,
- "57": 0.12181,
- "58": 0.11784,
- "59": 0.11936,
- "60": 0.11831,
- "61": 0.11819,
- "62": 0.11807,
- "63": 0.11828,
- "64": 0.11663,
- "65": 0.11901,
- "66": 0.1168,
- "67": 0.1167,
- "68": 0.12002,
- "69": 0.12016,
- "70": 0.1186,
- "71": 0.11772,
- "72": 0.1189,
- "73": 0.11915,
- "74": 0.11908,
- "75": 0.11898,
- "76": 0.11863,
- "77": 0.11869,
- "78": 0.11971,
- "79": 0.11843,
- "80": 0.1198,
- "81": 0.12003,
- "82": 0.11885,
- "83": 0.11905,
- "84": 0.12002,
- "85": 0.1192,
- "86": 0.11872,
- "87": 0.11777,
- "88": 0.11801,
- "89": 0.11864,
- "90": 0.11769,
- "91": 0.11692,
- "92": 0.12015,
- "93": 0.12072,
- "94": 0.11802,
- "95": 0.11798,
- "96": 0.12278,
- "97": 0.11941,
- "98": 0.1174,
- "99": 0.11816,
- "100": 0.12102
+ "2": 5.86605,
+ "3": 0.13527,
+ "4": 0.12296,
+ "5": 0.12443,
+ "6": 0.12222,
+ "7": 0.12431,
+ "8": 0.12301,
+ "9": 0.12262,
+ "10": 0.12615,
+ "11": 0.12479,
+ "12": 0.12612,
+ "13": 0.12396,
+ "14": 0.12727,
+ "15": 0.1273,
+ "16": 0.12644,
+ "17": 0.1273,
+ "18": 0.12789,
+ "19": 0.12832,
+ "20": 0.12576,
+ "21": 0.12724,
+ "22": 0.12793,
+ "23": 0.1258,
+ "24": 0.12564,
+ "25": 0.12595,
+ "26": 0.12455,
+ "27": 0.12657,
+ "28": 0.12588,
+ "29": 0.12712,
+ "30": 0.12691,
+ "31": 0.126,
+ "32": 0.12515,
+ "33": 0.12441,
+ "34": 0.12458,
+ "35": 0.12577,
+ "36": 0.12386,
+ "37": 0.12673,
+ "38": 0.1247,
+ "39": 0.12614,
+ "40": 0.12388,
+ "41": 0.12934,
+ "42": 0.12674,
+ "43": 0.12687,
+ "44": 0.1272,
+ "45": 0.1238,
+ "46": 0.12514,
+ "47": 0.12467,
+ "48": 0.12579,
+ "49": 0.12624,
+ "50": 0.12487,
+ "51": 0.16156,
+ "52": 0.1308,
+ "53": 0.12721,
+ "54": 0.12451,
+ "55": 0.12517,
+ "56": 0.12436,
+ "57": 0.12767,
+ "58": 0.12444,
+ "59": 0.12475,
+ "60": 0.12331,
+ "61": 0.12518,
+ "62": 0.12457,
+ "63": 0.12132,
+ "64": 0.12553,
+ "65": 0.12416,
+ "66": 0.12219,
+ "67": 0.12402,
+ "68": 0.12407,
+ "69": 0.12423,
+ "70": 0.12433,
+ "71": 0.12449,
+ "72": 0.12308,
+ "73": 0.12596,
+ "74": 0.12432,
+ "75": 0.12395,
+ "76": 0.12485,
+ "77": 0.12332,
+ "78": 0.12531,
+ "79": 0.1263,
+ "80": 0.12438,
+ "81": 0.1277,
+ "82": 0.12699,
+ "83": 0.12503,
+ "84": 0.12566,
+ "85": 0.12342,
+ "86": 0.12385,
+ "87": 0.12328,
+ "88": 0.12366,
+ "89": 0.12501,
+ "90": 0.1245,
+ "91": 0.12538,
+ "92": 0.12418,
+ "93": 0.12242,
+ "94": 0.12316,
+ "95": 0.12244,
+ "96": 0.12316,
+ "97": 0.12448,
+ "98": 0.12205,
+ "99": 0.12459,
+ "100": 0.12444
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear/golden_values_dev_dgx_h100.json
index 5b2aa3ce19c..9890b967344 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.85599,
- "2": 10.8648,
- "3": 10.87042,
- "4": 10.85288,
- "5": 10.88397,
- "6": 10.89184,
- "7": 10.86732,
- "8": 10.87057,
- "9": 10.87432,
- "10": 10.84185,
- "11": 10.87989,
- "12": 10.87417,
- "13": 10.87884,
- "14": 10.89184,
- "15": 10.82659,
- "16": 10.83027,
- "17": 10.80933,
- "18": 10.81431,
- "19": 10.8167,
- "20": 10.72165,
- "21": 10.70557,
- "22": 10.56881,
- "23": 10.72025,
- "24": 10.61194,
- "25": 10.55765,
- "26": 10.61149,
- "27": 10.62635,
- "28": 10.57155,
- "29": 10.58212,
- "30": 10.36267,
- "31": 10.11682,
- "32": 10.4682,
- "33": 10.45411,
- "34": 10.21121,
- "35": 10.27207,
- "36": 10.22246,
- "37": 10.34079,
- "38": 10.18964,
- "39": 10.40228,
- "40": 10.08758,
- "41": 10.13714,
- "42": 10.21175,
- "43": 9.82878,
- "44": 9.96255,
- "45": 9.82846,
- "46": 9.80952,
- "47": 10.13734,
- "48": 9.84349,
- "49": 9.52888,
- "50": 9.91046,
- "51": 9.85075,
- "52": 9.73181,
- "53": 10.06388,
- "54": 9.95432,
- "55": 9.87204,
- "56": 9.61823,
- "57": 9.47467,
- "58": 9.82802,
- "59": 9.57962,
- "60": 9.49074,
- "61": 9.68473,
- "62": 9.99245,
- "63": 9.38364,
- "64": 9.77766,
- "65": 8.94008,
- "66": 9.70099,
- "67": 9.3605,
- "68": 9.77766,
- "69": 9.78865,
- "70": 9.73813,
- "71": 9.61811,
- "72": 9.58068,
- "73": 9.4964,
- "74": 8.93812,
- "75": 9.42081,
- "76": 9.07416,
- "77": 10.06077,
- "78": 9.71952,
- "79": 9.37088,
- "80": 9.39874,
- "81": 9.47802,
- "82": 9.69299,
- "83": 9.30276,
- "84": 9.41548,
- "85": 9.60883,
- "86": 9.07461,
- "87": 9.58826,
- "88": 9.74392,
- "89": 9.5951,
- "90": 9.81217,
- "91": 9.33796,
- "92": 9.3534,
- "93": 9.07315,
- "94": 8.83127,
- "95": 9.51524,
- "96": 9.52183,
- "97": 9.31012,
- "98": 9.66532,
- "99": 8.88179,
- "100": 9.39375
+ "1": 10.95939,
+ "2": 10.94561,
+ "3": 10.94918,
+ "4": 10.94704,
+ "5": 10.9559,
+ "6": 10.95226,
+ "7": 10.95018,
+ "8": 10.95679,
+ "9": 10.94257,
+ "10": 10.94725,
+ "11": 10.9355,
+ "12": 10.94739,
+ "13": 10.91987,
+ "14": 10.92083,
+ "15": 10.9047,
+ "16": 10.89241,
+ "17": 10.89288,
+ "18": 10.89244,
+ "19": 10.87969,
+ "20": 10.8073,
+ "21": 10.78845,
+ "22": 10.77823,
+ "23": 10.78258,
+ "24": 10.75943,
+ "25": 10.75776,
+ "26": 10.73466,
+ "27": 10.6986,
+ "28": 10.61059,
+ "29": 10.57948,
+ "30": 10.56536,
+ "31": 10.56475,
+ "32": 10.55149,
+ "33": 10.5221,
+ "34": 10.47134,
+ "35": 10.48346,
+ "36": 10.46231,
+ "37": 10.42485,
+ "38": 10.4207,
+ "39": 10.39101,
+ "40": 10.37295,
+ "41": 10.35277,
+ "42": 10.35027,
+ "43": 10.31693,
+ "44": 10.30515,
+ "45": 10.29686,
+ "46": 10.2661,
+ "47": 10.25518,
+ "48": 10.21045,
+ "49": 10.20232,
+ "50": 10.21715,
+ "51": 10.21033,
+ "52": 10.15826,
+ "53": 10.15994,
+ "54": 10.13458,
+ "55": 10.11076,
+ "56": 10.12859,
+ "57": 10.11001,
+ "58": 10.12178,
+ "59": 10.06879,
+ "60": 10.08993,
+ "61": 10.04522,
+ "62": 10.01557,
+ "63": 10.08026,
+ "64": 10.03235,
+ "65": 10.0017,
+ "66": 10.04586,
+ "67": 10.01595,
+ "68": 9.98432,
+ "69": 9.9968,
+ "70": 9.98003,
+ "71": 10.00126,
+ "72": 9.98842,
+ "73": 9.97266,
+ "74": 9.96398,
+ "75": 9.92909,
+ "76": 9.95786,
+ "77": 9.95866,
+ "78": 9.91022,
+ "79": 9.90849,
+ "80": 9.92433,
+ "81": 9.94838,
+ "82": 9.88869,
+ "83": 9.85301,
+ "84": 9.79711,
+ "85": 9.77769,
+ "86": 9.88375,
+ "87": 9.90612,
+ "88": 9.87918,
+ "89": 9.81824,
+ "90": 9.8155,
+ "91": 9.81753,
+ "92": 9.81598,
+ "93": 9.74137,
+ "94": 9.8206,
+ "95": 9.80832,
+ "96": 9.79429,
+ "97": 9.73959,
+ "98": 9.76307,
+ "99": 9.81732,
+ "100": 9.70259
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1640.0,
- "2": 1738.0,
- "3": 1638.0,
- "4": 1810.0,
- "5": 1755.0,
- "6": 1681.0,
- "7": 1781.0,
- "8": 1502.0,
- "9": 1817.0,
- "10": 1394.0,
- "11": 1927.0,
- "12": 1691.0,
- "13": 1901.0,
- "14": 1631.0,
- "15": 1765.0,
- "16": 1864.0,
- "17": 1704.0,
- "18": 1771.0,
- "19": 1817.0,
- "20": 1831.0,
- "21": 1813.0,
- "22": 1673.0,
- "23": 2005.0,
- "24": 1553.0,
- "25": 1577.0,
- "26": 1656.0,
- "27": 1734.0,
- "28": 1896.0,
- "29": 2051.0,
- "30": 1897.0,
- "31": 1452.0,
- "32": 1785.0,
- "33": 2061.0,
- "34": 1857.0,
- "35": 1920.0,
- "36": 1990.0,
- "37": 2191.0,
- "38": 2142.0,
- "39": 2215.0,
- "40": 2166.0,
- "41": 2154.0,
- "42": 2148.0,
- "43": 1881.0,
- "44": 2066.0,
- "45": 1952.0,
- "46": 2217.0,
- "47": 2513.0,
- "48": 2356.0,
- "49": 2294.0,
- "50": 2140.0,
- "51": 2509.0,
- "52": 2528.0,
- "53": 2851.0,
- "54": 2747.0,
- "55": 2333.0,
- "56": 2724.0,
- "57": 2315.0,
- "58": 2754.0,
- "59": 2774.0,
- "60": 2336.0,
- "61": 2912.0,
- "62": 2415.0,
- "63": 2341.0,
- "64": 2837.0,
- "65": 2661.0,
- "66": 3000.0,
- "67": 2779.0,
- "68": 2691.0,
- "69": 2793.0,
- "70": 3183.0,
- "71": 2962.0,
- "72": 2393.0,
- "73": 2997.0,
- "74": 1935.0,
- "75": 2463.0,
- "76": 3065.0,
- "77": 3184.0,
- "78": 3154.0,
- "79": 3127.0,
- "80": 3286.0,
- "81": 3386.0,
- "82": 3128.0,
- "83": 2608.0,
- "84": 3079.0,
- "85": 3260.0,
- "86": 2687.0,
- "87": 3591.0,
- "88": 3035.0,
- "89": 3165.0,
- "90": 3166.0,
- "91": 2690.0,
- "92": 2897.0,
- "93": 2630.0,
- "94": 3348.0,
- "95": 3349.0,
- "96": 3288.0,
- "97": 3055.0,
- "98": 3516.0,
- "99": 3035.0,
- "100": 3109.0
+ "1": 1688.0,
+ "2": 1724.0,
+ "3": 1783.0,
+ "4": 1601.0,
+ "5": 1744.0,
+ "6": 1688.0,
+ "7": 1868.0,
+ "8": 1702.0,
+ "9": 1745.0,
+ "10": 1683.0,
+ "11": 1563.0,
+ "12": 1650.0,
+ "13": 1700.0,
+ "14": 1781.0,
+ "15": 1616.0,
+ "16": 1830.0,
+ "17": 1820.0,
+ "18": 1733.0,
+ "19": 1640.0,
+ "20": 1620.0,
+ "21": 1770.0,
+ "22": 1650.0,
+ "23": 1724.0,
+ "24": 1781.0,
+ "25": 1664.0,
+ "26": 1755.0,
+ "27": 1806.0,
+ "28": 1786.0,
+ "29": 1860.0,
+ "30": 1773.0,
+ "31": 1933.0,
+ "32": 1983.0,
+ "33": 1918.0,
+ "34": 2026.0,
+ "35": 1963.0,
+ "36": 1915.0,
+ "37": 2167.0,
+ "38": 2031.0,
+ "39": 2238.0,
+ "40": 2278.0,
+ "41": 2319.0,
+ "42": 1981.0,
+ "43": 2328.0,
+ "44": 2125.0,
+ "45": 2493.0,
+ "46": 2294.0,
+ "47": 2514.0,
+ "48": 2569.0,
+ "49": 2806.0,
+ "50": 2436.0,
+ "51": 2387.0,
+ "52": 2803.0,
+ "53": 2577.0,
+ "54": 2762.0,
+ "55": 2434.0,
+ "56": 2678.0,
+ "57": 2263.0,
+ "58": 3538.0,
+ "59": 2960.0,
+ "60": 2933.0,
+ "61": 2652.0,
+ "62": 3134.0,
+ "63": 3217.0,
+ "64": 3511.0,
+ "65": 2623.0,
+ "66": 3056.0,
+ "67": 3664.0,
+ "68": 3212.0,
+ "69": 2916.0,
+ "70": 3313.0,
+ "71": 3066.0,
+ "72": 2924.0,
+ "73": 3417.0,
+ "74": 3174.0,
+ "75": 2849.0,
+ "76": 3229.0,
+ "77": 3749.0,
+ "78": 3347.0,
+ "79": 3220.0,
+ "80": 2903.0,
+ "81": 3626.0,
+ "82": 2997.0,
+ "83": 3138.0,
+ "84": 3034.0,
+ "85": 2612.0,
+ "86": 2952.0,
+ "87": 2694.0,
+ "88": 3114.0,
+ "89": 3041.0,
+ "90": 3778.0,
+ "91": 3017.0,
+ "92": 2944.0,
+ "93": 3015.0,
+ "94": 3207.0,
+ "95": 3232.0,
+ "96": 3517.0,
+ "97": 3439.0,
+ "98": 3553.0,
+ "99": 3054.0,
+ "100": 2909.0
}
},
"mem-allocated-bytes": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 8.42141,
- "2": 0.12821,
- "3": 0.10969,
- "4": 0.08528,
- "5": 0.08609,
- "6": 0.08514,
- "7": 0.08511,
- "8": 0.08614,
- "9": 0.0853,
- "10": 0.08556,
- "11": 0.08506,
- "12": 0.08648,
- "13": 0.08513,
- "14": 0.08524,
- "15": 0.08502,
- "16": 0.08679,
- "17": 0.08617,
- "18": 0.08799,
- "19": 0.08587,
- "20": 0.08552,
- "21": 0.08665,
- "22": 0.08551,
- "23": 0.08517,
- "24": 0.08535,
- "25": 0.08579,
- "26": 0.08526,
- "27": 0.08602,
- "28": 0.08519,
- "29": 0.08544,
- "30": 0.08512,
- "31": 0.0856,
- "32": 0.08591,
- "33": 0.08561,
- "34": 0.08518,
- "35": 0.08492,
- "36": 0.08517,
- "37": 0.08548,
- "38": 0.08494,
- "39": 0.08594,
- "40": 0.08522,
- "41": 0.08599,
- "42": 0.0854,
- "43": 0.08536,
- "44": 0.0855,
- "45": 0.08648,
- "46": 0.088,
- "47": 0.08639,
- "48": 0.08682,
- "49": 0.08646,
- "50": 0.08529,
- "51": 0.09801,
- "52": 0.08949,
- "53": 0.08726,
- "54": 0.08702,
- "55": 0.08687,
- "56": 0.08692,
- "57": 0.08726,
- "58": 0.0871,
- "59": 0.08762,
- "60": 0.08729,
- "61": 0.08712,
- "62": 0.0868,
- "63": 0.08725,
- "64": 0.08676,
- "65": 0.08718,
- "66": 0.08682,
- "67": 0.08754,
- "68": 0.08695,
- "69": 0.08788,
- "70": 0.08724,
- "71": 0.08705,
- "72": 0.08759,
- "73": 0.08826,
- "74": 0.0871,
- "75": 0.08684,
- "76": 0.08689,
- "77": 0.08656,
- "78": 0.08667,
- "79": 0.08705,
- "80": 0.08727,
- "81": 0.0879,
- "82": 0.08956,
- "83": 0.08661,
- "84": 0.08671,
- "85": 0.08761,
- "86": 0.08652,
- "87": 0.08663,
- "88": 0.08663,
- "89": 0.08687,
- "90": 0.08718,
- "91": 0.0868,
- "92": 0.08665,
- "93": 0.08695,
- "94": 0.08685,
- "95": 0.08671,
- "96": 0.08669,
- "97": 0.08742,
- "98": 0.08628,
- "99": 0.08628,
- "100": 0.08651
+ "1": "nan",
+ "2": 5.96224,
+ "3": 0.10016,
+ "4": 0.08525,
+ "5": 0.08488,
+ "6": 0.08533,
+ "7": 0.08586,
+ "8": 0.08513,
+ "9": 0.08628,
+ "10": 0.08625,
+ "11": 0.08505,
+ "12": 0.08489,
+ "13": 0.08493,
+ "14": 0.0849,
+ "15": 0.08525,
+ "16": 0.08659,
+ "17": 0.08526,
+ "18": 0.085,
+ "19": 0.08506,
+ "20": 0.08535,
+ "21": 0.08579,
+ "22": 0.08486,
+ "23": 0.08522,
+ "24": 0.08499,
+ "25": 0.08491,
+ "26": 0.08499,
+ "27": 0.08503,
+ "28": 0.08627,
+ "29": 0.08596,
+ "30": 0.08495,
+ "31": 0.0851,
+ "32": 0.08521,
+ "33": 0.08506,
+ "34": 0.08533,
+ "35": 0.08539,
+ "36": 0.08667,
+ "37": 0.08651,
+ "38": 0.086,
+ "39": 0.08619,
+ "40": 0.08508,
+ "41": 0.08497,
+ "42": 0.08519,
+ "43": 0.08538,
+ "44": 0.08572,
+ "45": 0.08582,
+ "46": 0.08582,
+ "47": 0.08584,
+ "48": 0.08566,
+ "49": 0.08506,
+ "50": 0.08497,
+ "51": 0.12435,
+ "52": 0.08823,
+ "53": 0.08861,
+ "54": 0.08764,
+ "55": 0.08868,
+ "56": 0.0881,
+ "57": 0.0878,
+ "58": 0.08767,
+ "59": 0.08784,
+ "60": 0.08816,
+ "61": 0.0879,
+ "62": 0.08811,
+ "63": 0.08816,
+ "64": 0.08757,
+ "65": 0.0879,
+ "66": 0.08827,
+ "67": 0.08778,
+ "68": 0.08836,
+ "69": 0.08856,
+ "70": 0.08783,
+ "71": 0.08772,
+ "72": 0.0877,
+ "73": 0.08796,
+ "74": 0.08817,
+ "75": 0.08814,
+ "76": 0.08838,
+ "77": 0.08773,
+ "78": 0.0896,
+ "79": 0.08863,
+ "80": 0.08693,
+ "81": 0.08934,
+ "82": 0.08575,
+ "83": 0.08772,
+ "84": 0.08699,
+ "85": 0.08836,
+ "86": 0.08573,
+ "87": 0.08851,
+ "88": 0.08675,
+ "89": 0.08567,
+ "90": 0.0856,
+ "91": 0.08684,
+ "92": 0.08644,
+ "93": 0.08648,
+ "94": 0.08599,
+ "95": 0.08631,
+ "96": 0.08569,
+ "97": 0.08578,
+ "98": 0.0866,
+ "99": 0.08579,
+ "100": 0.08515
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear/model_config.yaml
index 7d069ce9ec6..305e7eabf98 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,5 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear/golden_values_dev_dgx_gb200.json
index a9a12874e97..463885b1cba 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear/golden_values_dev_dgx_gb200.json
@@ -6,104 +6,104 @@
"values": {
"1": 10.83582,
"2": 10.83571,
- "3": 10.83523,
+ "3": 10.83524,
"4": 10.79949,
"5": 10.84909,
- "6": 10.86563,
- "7": 10.82789,
+ "6": 10.86567,
+ "7": 10.82786,
"8": 10.8363,
"9": 10.83997,
- "10": 10.79865,
- "11": 10.8677,
- "12": 10.84994,
- "13": 10.85915,
- "14": 10.86874,
- "15": 10.80173,
- "16": 10.79183,
- "17": 10.77353,
- "18": 10.78739,
- "19": 10.78983,
- "20": 10.68446,
- "21": 10.6784,
- "22": 10.5257,
- "23": 10.70726,
- "24": 10.56551,
- "25": 10.51602,
- "26": 10.58017,
- "27": 10.58981,
+ "10": 10.79867,
+ "11": 10.86763,
+ "12": 10.84992,
+ "13": 10.85912,
+ "14": 10.8687,
+ "15": 10.80171,
+ "16": 10.79189,
+ "17": 10.77351,
+ "18": 10.78742,
+ "19": 10.78986,
+ "20": 10.68447,
+ "21": 10.67839,
+ "22": 10.52572,
+ "23": 10.70729,
+ "24": 10.56549,
+ "25": 10.51603,
+ "26": 10.58018,
+ "27": 10.58977,
"28": 10.54551,
- "29": 10.57726,
- "30": 10.34051,
- "31": 10.07051,
- "32": 10.44503,
- "33": 10.44293,
- "34": 10.19391,
- "35": 10.24261,
- "36": 10.19236,
- "37": 10.32969,
+ "29": 10.57724,
+ "30": 10.34049,
+ "31": 10.07056,
+ "32": 10.44508,
+ "33": 10.44289,
+ "34": 10.19394,
+ "35": 10.2426,
+ "36": 10.19235,
+ "37": 10.32972,
"38": 10.16551,
"39": 10.38729,
"40": 10.05174,
- "41": 10.12191,
- "42": 10.19259,
- "43": 9.8069,
- "44": 9.92475,
- "45": 9.80639,
- "46": 9.80145,
- "47": 10.12104,
- "48": 9.83127,
- "49": 9.50404,
- "50": 9.87954,
+ "41": 10.12185,
+ "42": 10.19258,
+ "43": 9.80694,
+ "44": 9.92474,
+ "45": 9.80636,
+ "46": 9.80144,
+ "47": 10.12106,
+ "48": 9.83126,
+ "49": 9.50406,
+ "50": 9.87955,
"51": 9.83807,
- "52": 9.72058,
- "53": 10.0568,
- "54": 9.95032,
- "55": 9.88328,
- "56": 9.60431,
+ "52": 9.72057,
+ "53": 10.05682,
+ "54": 9.95031,
+ "55": 9.88332,
+ "56": 9.60428,
"57": 9.45518,
- "58": 9.81927,
- "59": 9.58262,
+ "58": 9.81923,
+ "59": 9.58266,
"60": 9.48844,
- "61": 9.68577,
- "62": 9.97779,
+ "61": 9.68574,
+ "62": 9.9778,
"63": 9.36765,
- "64": 9.75913,
- "65": 8.9376,
- "66": 9.69257,
- "67": 9.36621,
- "68": 9.78303,
- "69": 9.79318,
- "70": 9.72699,
+ "64": 9.75912,
+ "65": 8.93762,
+ "66": 9.6926,
+ "67": 9.36619,
+ "68": 9.78309,
+ "69": 9.79315,
+ "70": 9.72695,
"71": 9.62875,
- "72": 9.58004,
+ "72": 9.58006,
"73": 9.487,
- "74": 8.92041,
- "75": 9.41128,
- "76": 9.07564,
- "77": 10.05848,
- "78": 9.72184,
- "79": 9.3732,
- "80": 9.40079,
- "81": 9.4792,
- "82": 9.69754,
- "83": 9.31037,
- "84": 9.41777,
- "85": 9.61194,
- "86": 9.07155,
- "87": 9.59661,
- "88": 9.74709,
- "89": 9.59667,
- "90": 9.82915,
- "91": 9.33725,
- "92": 9.3564,
- "93": 9.08552,
- "94": 8.82807,
- "95": 9.52842,
- "96": 9.52611,
- "97": 9.30632,
- "98": 9.66808,
- "99": 8.89461,
- "100": 9.40666
+ "74": 8.92045,
+ "75": 9.41127,
+ "76": 9.0757,
+ "77": 10.05849,
+ "78": 9.72185,
+ "79": 9.37321,
+ "80": 9.40082,
+ "81": 9.47926,
+ "82": 9.69753,
+ "83": 9.31033,
+ "84": 9.41773,
+ "85": 9.61195,
+ "86": 9.07158,
+ "87": 9.59659,
+ "88": 9.74711,
+ "89": 9.59669,
+ "90": 9.82914,
+ "91": 9.33728,
+ "92": 9.35642,
+ "93": 9.08554,
+ "94": 8.82803,
+ "95": 9.52843,
+ "96": 9.52607,
+ "97": 9.30634,
+ "98": 9.66809,
+ "99": 8.89459,
+ "100": 9.40668
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1536.0,
- "2": 1592.0,
- "3": 1551.0,
- "4": 1769.0,
- "5": 1824.0,
- "6": 1800.0,
- "7": 1734.0,
- "8": 1619.0,
- "9": 1829.0,
- "10": 1355.0,
- "11": 1911.0,
- "12": 1721.0,
- "13": 1913.0,
- "14": 1708.0,
- "15": 1919.0,
- "16": 1938.0,
- "17": 1740.0,
- "18": 1676.0,
- "19": 1743.0,
- "20": 1535.0,
- "21": 1797.0,
- "22": 1661.0,
- "23": 1887.0,
- "24": 1666.0,
- "25": 1633.0,
- "26": 1676.0,
- "27": 1740.0,
- "28": 1991.0,
- "29": 1918.0,
- "30": 1806.0,
- "31": 1588.0,
- "32": 1863.0,
- "33": 2126.0,
- "34": 1812.0,
- "35": 1976.0,
- "36": 1875.0,
- "37": 2301.0,
- "38": 2131.0,
- "39": 2351.0,
- "40": 2130.0,
- "41": 2391.0,
- "42": 2255.0,
- "43": 1975.0,
- "44": 2138.0,
- "45": 2208.0,
- "46": 2364.0,
- "47": 2564.0,
- "48": 2337.0,
- "49": 2142.0,
- "50": 2423.0,
- "51": 2546.0,
- "52": 2590.0,
- "53": 2879.0,
- "54": 2697.0,
- "55": 2316.0,
- "56": 2549.0,
- "57": 2261.0,
- "58": 2904.0,
- "59": 2740.0,
- "60": 2434.0,
- "61": 2801.0,
- "62": 2663.0,
- "63": 2502.0,
- "64": 2948.0,
- "65": 2644.0,
- "66": 2961.0,
- "67": 2813.0,
- "68": 2686.0,
- "69": 2912.0,
- "70": 3096.0,
- "71": 2854.0,
- "72": 2454.0,
- "73": 3081.0,
- "74": 1933.0,
- "75": 2465.0,
- "76": 3012.0,
- "77": 3163.0,
- "78": 2997.0,
- "79": 3089.0,
- "80": 3187.0,
- "81": 3500.0,
- "82": 3339.0,
- "83": 2705.0,
- "84": 3205.0,
- "85": 3033.0,
- "86": 2818.0,
- "87": 3671.0,
- "88": 3190.0,
- "89": 3336.0,
- "90": 3320.0,
- "91": 2698.0,
- "92": 3072.0,
- "93": 2750.0,
- "94": 3397.0,
- "95": 3317.0,
- "96": 3290.0,
- "97": 3116.0,
- "98": 3732.0,
- "99": 3049.0,
- "100": 2974.0
+ "1": 1501.0,
+ "2": 1576.0,
+ "3": 1604.0,
+ "4": 1778.0,
+ "5": 1880.0,
+ "6": 1803.0,
+ "7": 1789.0,
+ "8": 1669.0,
+ "9": 1811.0,
+ "10": 1392.0,
+ "11": 1835.0,
+ "12": 1663.0,
+ "13": 1855.0,
+ "14": 1841.0,
+ "15": 1874.0,
+ "16": 1844.0,
+ "17": 1738.0,
+ "18": 1724.0,
+ "19": 1742.0,
+ "20": 1612.0,
+ "21": 1751.0,
+ "22": 1713.0,
+ "23": 1964.0,
+ "24": 1632.0,
+ "25": 1570.0,
+ "26": 1675.0,
+ "27": 1752.0,
+ "28": 2026.0,
+ "29": 1938.0,
+ "30": 1848.0,
+ "31": 1557.0,
+ "32": 1926.0,
+ "33": 2052.0,
+ "34": 1880.0,
+ "35": 2022.0,
+ "36": 1926.0,
+ "37": 2344.0,
+ "38": 2202.0,
+ "39": 2285.0,
+ "40": 2225.0,
+ "41": 2328.0,
+ "42": 2200.0,
+ "43": 1984.0,
+ "44": 2142.0,
+ "45": 2173.0,
+ "46": 2308.0,
+ "47": 2592.0,
+ "48": 2460.0,
+ "49": 2242.0,
+ "50": 2383.0,
+ "51": 2489.0,
+ "52": 2497.0,
+ "53": 2875.0,
+ "54": 2654.0,
+ "55": 2317.0,
+ "56": 2599.0,
+ "57": 2299.0,
+ "58": 2830.0,
+ "59": 2784.0,
+ "60": 2437.0,
+ "61": 2916.0,
+ "62": 2599.0,
+ "63": 2388.0,
+ "64": 2785.0,
+ "65": 2677.0,
+ "66": 2972.0,
+ "67": 2797.0,
+ "68": 2752.0,
+ "69": 3049.0,
+ "70": 3087.0,
+ "71": 2952.0,
+ "72": 2411.0,
+ "73": 3099.0,
+ "74": 1975.0,
+ "75": 2614.0,
+ "76": 2941.0,
+ "77": 3166.0,
+ "78": 3123.0,
+ "79": 3085.0,
+ "80": 3198.0,
+ "81": 3350.0,
+ "82": 3322.0,
+ "83": 2858.0,
+ "84": 3125.0,
+ "85": 3194.0,
+ "86": 2777.0,
+ "87": 3602.0,
+ "88": 3006.0,
+ "89": 3267.0,
+ "90": 3133.0,
+ "91": 2753.0,
+ "92": 3113.0,
+ "93": 2714.0,
+ "94": 3364.0,
+ "95": 3273.0,
+ "96": 3202.0,
+ "97": 3124.0,
+ "98": 3716.0,
+ "99": 3121.0,
+ "100": 3131.0
}
},
"mem-allocated-bytes": {
@@ -325,7 +325,7 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1939785728.0,
+ "1": 1938737152.0,
"2": 2222434304.0,
"3": 2222434304.0,
"4": 2222434304.0,
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.22807,
- "3": 0.13601,
- "4": 0.12128,
- "5": 0.1198,
- "6": 0.1228,
- "7": 0.12056,
- "8": 0.11886,
- "9": 0.11944,
- "10": 0.11995,
- "11": 0.11935,
- "12": 0.11905,
- "13": 0.11975,
- "14": 0.12242,
- "15": 0.12061,
- "16": 0.12046,
- "17": 0.1208,
- "18": 0.12205,
- "19": 0.12427,
- "20": 0.12315,
- "21": 0.11965,
- "22": 0.12231,
- "23": 0.12286,
- "24": 0.12394,
- "25": 0.12377,
- "26": 0.12221,
- "27": 0.11936,
- "28": 0.11894,
- "29": 0.11945,
- "30": 0.12192,
- "31": 0.12571,
- "32": 0.12346,
- "33": 0.12413,
- "34": 0.12225,
- "35": 0.12328,
- "36": 0.12241,
- "37": 0.12432,
- "38": 0.12195,
- "39": 0.12262,
- "40": 0.12198,
- "41": 0.12396,
- "42": 0.12194,
- "43": 0.12435,
- "44": 0.12108,
- "45": 0.12326,
- "46": 0.1218,
- "47": 0.12308,
- "48": 0.12384,
- "49": 0.12795,
- "50": 0.12572,
- "51": 0.13502,
- "52": 0.13106,
- "53": 0.14515,
- "54": 0.12597,
- "55": 0.1249,
- "56": 0.12535,
- "57": 0.12569,
- "58": 0.12489,
- "59": 0.12862,
- "60": 0.12778,
- "61": 0.12731,
- "62": 0.12786,
- "63": 0.13022,
- "64": 0.12789,
- "65": 0.12838,
- "66": 0.12571,
- "67": 0.12651,
- "68": 0.12592,
- "69": 0.12663,
- "70": 0.12691,
- "71": 0.12636,
- "72": 0.12638,
- "73": 0.12671,
- "74": 0.12637,
- "75": 0.12602,
- "76": 0.12598,
- "77": 0.12554,
- "78": 0.12553,
- "79": 0.12501,
- "80": 0.13898,
- "81": 0.14589,
- "82": 0.14718,
- "83": 0.14665,
- "84": 0.16017,
- "85": 0.14231,
- "86": 0.15628,
- "87": 0.14055,
- "88": 0.13961,
- "89": 0.14878,
- "90": 0.14486,
- "91": 0.1432,
- "92": 0.14946,
- "93": 0.14581,
- "94": 0.1623,
- "95": 0.15638,
- "96": 0.12895,
- "97": 0.12907,
- "98": 0.12824,
- "99": 0.12741,
- "100": 0.12543
+ "2": 5.94954,
+ "3": 1.50291,
+ "4": 0.53566,
+ "5": 0.40704,
+ "6": 0.34894,
+ "7": 0.22372,
+ "8": 0.53898,
+ "9": 0.12291,
+ "10": 0.16074,
+ "11": 0.35135,
+ "12": 0.12732,
+ "13": 0.12357,
+ "14": 0.12383,
+ "15": 0.12737,
+ "16": 0.12362,
+ "17": 0.12343,
+ "18": 0.12473,
+ "19": 0.12595,
+ "20": 0.12604,
+ "21": 0.12429,
+ "22": 0.12556,
+ "23": 0.1253,
+ "24": 0.1263,
+ "25": 0.12594,
+ "26": 0.12525,
+ "27": 0.127,
+ "28": 0.12526,
+ "29": 0.12711,
+ "30": 0.12734,
+ "31": 0.12949,
+ "32": 0.12815,
+ "33": 0.12674,
+ "34": 0.12825,
+ "35": 0.12995,
+ "36": 0.12713,
+ "37": 0.12917,
+ "38": 0.12617,
+ "39": 0.12706,
+ "40": 0.1252,
+ "41": 0.12715,
+ "42": 0.12676,
+ "43": 0.12603,
+ "44": 0.1271,
+ "45": 0.12586,
+ "46": 0.1272,
+ "47": 0.1267,
+ "48": 0.12718,
+ "49": 0.12784,
+ "50": 0.1242,
+ "51": 0.1345,
+ "52": 0.1299,
+ "53": 0.14634,
+ "54": 0.12463,
+ "55": 0.12725,
+ "56": 0.12725,
+ "57": 0.12837,
+ "58": 0.12517,
+ "59": 0.12774,
+ "60": 0.1254,
+ "61": 0.12644,
+ "62": 0.12626,
+ "63": 0.12406,
+ "64": 0.12571,
+ "65": 0.12665,
+ "66": 0.12645,
+ "67": 0.12761,
+ "68": 0.12692,
+ "69": 0.12838,
+ "70": 0.12887,
+ "71": 0.1276,
+ "72": 0.12853,
+ "73": 0.12876,
+ "74": 0.12752,
+ "75": 0.12802,
+ "76": 0.12753,
+ "77": 0.12582,
+ "78": 0.12904,
+ "79": 0.12901,
+ "80": 0.12789,
+ "81": 0.12964,
+ "82": 0.1301,
+ "83": 0.1264,
+ "84": 0.12524,
+ "85": 0.1274,
+ "86": 0.12805,
+ "87": 0.12637,
+ "88": 0.12675,
+ "89": 0.12674,
+ "90": 0.12851,
+ "91": 0.12781,
+ "92": 0.12833,
+ "93": 0.1262,
+ "94": 0.1255,
+ "95": 0.12578,
+ "96": 0.12506,
+ "97": 0.12696,
+ "98": 0.12507,
+ "99": 0.12893,
+ "100": 0.12679
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear/golden_values_dev_dgx_h100.json
index 4fffaabca8a..f2e028a15e2 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.85599,
- "2": 10.8648,
- "3": 10.87042,
- "4": 10.85288,
- "5": 10.88397,
- "6": 10.89184,
- "7": 10.86732,
- "8": 10.87057,
- "9": 10.87432,
- "10": 10.84185,
- "11": 10.87989,
- "12": 10.87417,
- "13": 10.87884,
- "14": 10.89184,
- "15": 10.82659,
- "16": 10.83027,
- "17": 10.80933,
- "18": 10.81431,
- "19": 10.8167,
- "20": 10.72165,
- "21": 10.70557,
- "22": 10.56881,
- "23": 10.72025,
- "24": 10.61194,
- "25": 10.55765,
- "26": 10.61149,
- "27": 10.62635,
- "28": 10.57155,
- "29": 10.58212,
- "30": 10.36267,
- "31": 10.11682,
- "32": 10.4682,
- "33": 10.45411,
- "34": 10.21121,
- "35": 10.27207,
- "36": 10.22246,
- "37": 10.34079,
- "38": 10.18964,
- "39": 10.40228,
- "40": 10.08758,
- "41": 10.13714,
- "42": 10.21175,
- "43": 9.82878,
- "44": 9.96255,
- "45": 9.82846,
- "46": 9.80952,
- "47": 10.13734,
- "48": 9.84349,
- "49": 9.52888,
- "50": 9.91046,
- "51": 9.85075,
- "52": 9.73181,
- "53": 10.06388,
- "54": 9.95432,
- "55": 9.87204,
- "56": 9.61823,
- "57": 9.47467,
- "58": 9.82802,
- "59": 9.57962,
- "60": 9.49074,
- "61": 9.68473,
- "62": 9.99245,
- "63": 9.38364,
- "64": 9.77766,
- "65": 8.94008,
- "66": 9.70099,
- "67": 9.3605,
- "68": 9.77766,
- "69": 9.78865,
- "70": 9.73813,
- "71": 9.61811,
- "72": 9.58068,
- "73": 9.4964,
- "74": 8.93812,
- "75": 9.42081,
- "76": 9.07416,
- "77": 10.06077,
- "78": 9.71952,
- "79": 9.37088,
- "80": 9.39874,
- "81": 9.47802,
- "82": 9.69299,
- "83": 9.30276,
- "84": 9.41548,
- "85": 9.60883,
- "86": 9.07461,
- "87": 9.58826,
- "88": 9.74392,
- "89": 9.5951,
- "90": 9.81217,
- "91": 9.33796,
- "92": 9.3534,
- "93": 9.07315,
- "94": 8.83127,
- "95": 9.51524,
- "96": 9.52183,
- "97": 9.31012,
- "98": 9.66532,
- "99": 8.88179,
- "100": 9.39375
+ "1": 10.95939,
+ "2": 10.94561,
+ "3": 10.94918,
+ "4": 10.94704,
+ "5": 10.9559,
+ "6": 10.95226,
+ "7": 10.95018,
+ "8": 10.95679,
+ "9": 10.94257,
+ "10": 10.94725,
+ "11": 10.9355,
+ "12": 10.94739,
+ "13": 10.91987,
+ "14": 10.92083,
+ "15": 10.9047,
+ "16": 10.89241,
+ "17": 10.89288,
+ "18": 10.89244,
+ "19": 10.87969,
+ "20": 10.8073,
+ "21": 10.78845,
+ "22": 10.77823,
+ "23": 10.78258,
+ "24": 10.75943,
+ "25": 10.75776,
+ "26": 10.73466,
+ "27": 10.6986,
+ "28": 10.61059,
+ "29": 10.57948,
+ "30": 10.56536,
+ "31": 10.56475,
+ "32": 10.55149,
+ "33": 10.5221,
+ "34": 10.47134,
+ "35": 10.48346,
+ "36": 10.46231,
+ "37": 10.42485,
+ "38": 10.4207,
+ "39": 10.39101,
+ "40": 10.37295,
+ "41": 10.35277,
+ "42": 10.35027,
+ "43": 10.31693,
+ "44": 10.30515,
+ "45": 10.29686,
+ "46": 10.2661,
+ "47": 10.25518,
+ "48": 10.21045,
+ "49": 10.20232,
+ "50": 10.21715,
+ "51": 10.21033,
+ "52": 10.15826,
+ "53": 10.15994,
+ "54": 10.13458,
+ "55": 10.11076,
+ "56": 10.12859,
+ "57": 10.11001,
+ "58": 10.12178,
+ "59": 10.06879,
+ "60": 10.08993,
+ "61": 10.04522,
+ "62": 10.01557,
+ "63": 10.08026,
+ "64": 10.03235,
+ "65": 10.0017,
+ "66": 10.04586,
+ "67": 10.01595,
+ "68": 9.98432,
+ "69": 9.9968,
+ "70": 9.98003,
+ "71": 10.00126,
+ "72": 9.98842,
+ "73": 9.97266,
+ "74": 9.96398,
+ "75": 9.92909,
+ "76": 9.95786,
+ "77": 9.95866,
+ "78": 9.91022,
+ "79": 9.90849,
+ "80": 9.92433,
+ "81": 9.94838,
+ "82": 9.88869,
+ "83": 9.85301,
+ "84": 9.79711,
+ "85": 9.77769,
+ "86": 9.88375,
+ "87": 9.90612,
+ "88": 9.87918,
+ "89": 9.81824,
+ "90": 9.8155,
+ "91": 9.81753,
+ "92": 9.81598,
+ "93": 9.74137,
+ "94": 9.8206,
+ "95": 9.80832,
+ "96": 9.79429,
+ "97": 9.73959,
+ "98": 9.76307,
+ "99": 9.81732,
+ "100": 9.70259
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1640.0,
- "2": 1738.0,
- "3": 1638.0,
- "4": 1810.0,
- "5": 1755.0,
- "6": 1681.0,
- "7": 1781.0,
- "8": 1502.0,
- "9": 1817.0,
- "10": 1394.0,
- "11": 1927.0,
- "12": 1691.0,
- "13": 1901.0,
- "14": 1631.0,
- "15": 1765.0,
- "16": 1864.0,
- "17": 1704.0,
- "18": 1771.0,
- "19": 1817.0,
- "20": 1831.0,
- "21": 1813.0,
- "22": 1673.0,
- "23": 2005.0,
- "24": 1553.0,
- "25": 1577.0,
- "26": 1656.0,
- "27": 1734.0,
- "28": 1896.0,
- "29": 2051.0,
- "30": 1897.0,
- "31": 1452.0,
- "32": 1785.0,
- "33": 2061.0,
- "34": 1857.0,
- "35": 1920.0,
- "36": 1990.0,
- "37": 2191.0,
- "38": 2142.0,
- "39": 2215.0,
- "40": 2166.0,
- "41": 2154.0,
- "42": 2148.0,
- "43": 1881.0,
- "44": 2066.0,
- "45": 1952.0,
- "46": 2217.0,
- "47": 2513.0,
- "48": 2356.0,
- "49": 2294.0,
- "50": 2140.0,
- "51": 2509.0,
- "52": 2528.0,
- "53": 2851.0,
- "54": 2747.0,
- "55": 2333.0,
- "56": 2724.0,
- "57": 2315.0,
- "58": 2754.0,
- "59": 2774.0,
- "60": 2336.0,
- "61": 2912.0,
- "62": 2415.0,
- "63": 2341.0,
- "64": 2837.0,
- "65": 2661.0,
- "66": 3000.0,
- "67": 2779.0,
- "68": 2691.0,
- "69": 2793.0,
- "70": 3183.0,
- "71": 2962.0,
- "72": 2393.0,
- "73": 2997.0,
- "74": 1935.0,
- "75": 2463.0,
- "76": 3065.0,
- "77": 3184.0,
- "78": 3154.0,
- "79": 3127.0,
- "80": 3286.0,
- "81": 3386.0,
- "82": 3128.0,
- "83": 2608.0,
- "84": 3079.0,
- "85": 3260.0,
- "86": 2687.0,
- "87": 3591.0,
- "88": 3035.0,
- "89": 3165.0,
- "90": 3166.0,
- "91": 2690.0,
- "92": 2897.0,
- "93": 2630.0,
- "94": 3348.0,
- "95": 3349.0,
- "96": 3288.0,
- "97": 3055.0,
- "98": 3516.0,
- "99": 3035.0,
- "100": 3109.0
+ "1": 1688.0,
+ "2": 1724.0,
+ "3": 1783.0,
+ "4": 1601.0,
+ "5": 1744.0,
+ "6": 1688.0,
+ "7": 1868.0,
+ "8": 1702.0,
+ "9": 1745.0,
+ "10": 1683.0,
+ "11": 1563.0,
+ "12": 1650.0,
+ "13": 1700.0,
+ "14": 1781.0,
+ "15": 1616.0,
+ "16": 1830.0,
+ "17": 1820.0,
+ "18": 1733.0,
+ "19": 1640.0,
+ "20": 1620.0,
+ "21": 1770.0,
+ "22": 1650.0,
+ "23": 1724.0,
+ "24": 1781.0,
+ "25": 1664.0,
+ "26": 1755.0,
+ "27": 1806.0,
+ "28": 1786.0,
+ "29": 1860.0,
+ "30": 1773.0,
+ "31": 1933.0,
+ "32": 1983.0,
+ "33": 1918.0,
+ "34": 2026.0,
+ "35": 1963.0,
+ "36": 1915.0,
+ "37": 2167.0,
+ "38": 2031.0,
+ "39": 2238.0,
+ "40": 2278.0,
+ "41": 2319.0,
+ "42": 1981.0,
+ "43": 2328.0,
+ "44": 2125.0,
+ "45": 2493.0,
+ "46": 2294.0,
+ "47": 2514.0,
+ "48": 2569.0,
+ "49": 2806.0,
+ "50": 2436.0,
+ "51": 2387.0,
+ "52": 2803.0,
+ "53": 2577.0,
+ "54": 2762.0,
+ "55": 2434.0,
+ "56": 2678.0,
+ "57": 2263.0,
+ "58": 3538.0,
+ "59": 2960.0,
+ "60": 2933.0,
+ "61": 2652.0,
+ "62": 3134.0,
+ "63": 3217.0,
+ "64": 3511.0,
+ "65": 2623.0,
+ "66": 3056.0,
+ "67": 3664.0,
+ "68": 3212.0,
+ "69": 2916.0,
+ "70": 3313.0,
+ "71": 3066.0,
+ "72": 2924.0,
+ "73": 3417.0,
+ "74": 3174.0,
+ "75": 2849.0,
+ "76": 3229.0,
+ "77": 3749.0,
+ "78": 3347.0,
+ "79": 3220.0,
+ "80": 2903.0,
+ "81": 3626.0,
+ "82": 2997.0,
+ "83": 3138.0,
+ "84": 3034.0,
+ "85": 2612.0,
+ "86": 2952.0,
+ "87": 2694.0,
+ "88": 3114.0,
+ "89": 3041.0,
+ "90": 3778.0,
+ "91": 3017.0,
+ "92": 2944.0,
+ "93": 3015.0,
+ "94": 3207.0,
+ "95": 3232.0,
+ "96": 3517.0,
+ "97": 3439.0,
+ "98": 3553.0,
+ "99": 3054.0,
+ "100": 2909.0
}
},
"mem-allocated-bytes": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 8.07511,
- "2": 0.14681,
- "3": 0.10596,
- "4": 0.08711,
- "5": 0.0876,
- "6": 0.08568,
- "7": 0.08664,
- "8": 0.08587,
- "9": 0.08577,
- "10": 0.08621,
- "11": 0.08632,
- "12": 0.08547,
- "13": 0.08657,
- "14": 0.086,
- "15": 0.08713,
- "16": 0.08626,
- "17": 0.0867,
- "18": 0.08636,
- "19": 0.08698,
- "20": 0.08625,
- "21": 0.08785,
- "22": 0.08871,
- "23": 0.08659,
- "24": 0.08847,
- "25": 0.09629,
- "26": 0.09476,
- "27": 0.08553,
- "28": 0.08477,
- "29": 0.08431,
- "30": 0.08434,
- "31": 0.08557,
- "32": 0.08544,
- "33": 0.08488,
- "34": 0.08582,
- "35": 0.08395,
- "36": 0.08398,
- "37": 0.08559,
- "38": 0.08441,
- "39": 0.08418,
- "40": 0.08528,
- "41": 0.0861,
- "42": 0.08685,
- "43": 0.08626,
- "44": 0.08751,
- "45": 0.08791,
- "46": 0.087,
- "47": 0.08684,
- "48": 0.08803,
- "49": 0.08859,
- "50": 0.09019,
- "51": 0.10254,
- "52": 0.09302,
- "53": 0.10544,
- "54": 0.08758,
- "55": 0.0856,
- "56": 0.08575,
- "57": 0.08685,
- "58": 0.08631,
- "59": 0.08389,
- "60": 0.08441,
- "61": 0.08423,
- "62": 0.08509,
- "63": 0.08726,
- "64": 0.08594,
- "65": 0.08568,
- "66": 0.08392,
- "67": 0.08415,
- "68": 0.0849,
- "69": 0.08418,
- "70": 0.08396,
- "71": 0.08448,
- "72": 0.08498,
- "73": 0.08408,
- "74": 0.08475,
- "75": 0.08328,
- "76": 0.08384,
+ "1": "nan",
+ "2": 6.44339,
+ "3": 0.10706,
+ "4": 0.08531,
+ "5": 0.08367,
+ "6": 0.08479,
+ "7": 0.08459,
+ "8": 0.08402,
+ "9": 0.08392,
+ "10": 0.08433,
+ "11": 0.08379,
+ "12": 0.08462,
+ "13": 0.08495,
+ "14": 0.08452,
+ "15": 0.0842,
+ "16": 0.08543,
+ "17": 0.08453,
+ "18": 0.08425,
+ "19": 0.08419,
+ "20": 0.08437,
+ "21": 0.08426,
+ "22": 0.08441,
+ "23": 0.08453,
+ "24": 0.08416,
+ "25": 0.08623,
+ "26": 0.08834,
+ "27": 0.08458,
+ "28": 0.08427,
+ "29": 0.08416,
+ "30": 0.08429,
+ "31": 0.08395,
+ "32": 0.08447,
+ "33": 0.08418,
+ "34": 0.08393,
+ "35": 0.08491,
+ "36": 0.08421,
+ "37": 0.08392,
+ "38": 0.08401,
+ "39": 0.08398,
+ "40": 0.08377,
+ "41": 0.08579,
+ "42": 0.08516,
+ "43": 0.08679,
+ "44": 0.0841,
+ "45": 0.08361,
+ "46": 0.08383,
+ "47": 0.08533,
+ "48": 0.08808,
+ "49": 0.08425,
+ "50": 0.08384,
+ "51": 0.13315,
+ "52": 0.09345,
+ "53": 0.09138,
+ "54": 0.10572,
+ "55": 0.08925,
+ "56": 0.08835,
+ "57": 0.08684,
+ "58": 0.08658,
+ "59": 0.08447,
+ "60": 0.08419,
+ "61": 0.08671,
+ "62": 0.08499,
+ "63": 0.08482,
+ "64": 0.08459,
+ "65": 0.08443,
+ "66": 0.08603,
+ "67": 0.08456,
+ "68": 0.08403,
+ "69": 0.08527,
+ "70": 0.08415,
+ "71": 0.08381,
+ "72": 0.08609,
+ "73": 0.08423,
+ "74": 0.08393,
+ "75": 0.08414,
+ "76": 0.08428,
"77": 0.08424,
- "78": 0.08463,
- "79": 0.0841,
- "80": 0.08431,
- "81": 0.08441,
- "82": 0.0848,
- "83": 0.08442,
- "84": 0.08437,
- "85": 0.08486,
- "86": 0.08464,
- "87": 0.0837,
- "88": 0.0844,
- "89": 0.08503,
- "90": 0.08351,
- "91": 0.0839,
- "92": 0.08423,
- "93": 0.08472,
- "94": 0.08463,
- "95": 0.08455,
- "96": 0.08373,
- "97": 0.08396,
- "98": 0.08358,
- "99": 0.08466,
- "100": 0.08402
+ "78": 0.08433,
+ "79": 0.08397,
+ "80": 0.08418,
+ "81": 0.0841,
+ "82": 0.08391,
+ "83": 0.0848,
+ "84": 0.08425,
+ "85": 0.08414,
+ "86": 0.08391,
+ "87": 0.08365,
+ "88": 0.08417,
+ "89": 0.0841,
+ "90": 0.0845,
+ "91": 0.09075,
+ "92": 0.08419,
+ "93": 0.0842,
+ "94": 0.08383,
+ "95": 0.08374,
+ "96": 0.08433,
+ "97": 0.08453,
+ "98": 0.08411,
+ "99": 0.08412,
+ "100": 0.08379
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear/model_config.yaml
index ea882318c7e..3cf79eaf7d2 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_sequence_parallel/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_sequence_parallel/model_config.yaml
index d67dd6af765..15772459af3 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_sequence_parallel/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_sequence_parallel/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu/golden_values_dev_dgx_gb200.json
index 1fc5ef869c5..fe86682f4bc 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu/golden_values_dev_dgx_gb200.json
@@ -6,104 +6,104 @@
"values": {
"1": 10.74992,
"2": 10.77613,
- "3": 10.75714,
+ "3": 10.75715,
"4": 10.72305,
- "5": 10.80036,
- "6": 10.821,
+ "5": 10.80038,
+ "6": 10.82103,
"7": 10.77176,
- "8": 10.7988,
- "9": 10.77447,
- "10": 10.70645,
- "11": 10.8328,
+ "8": 10.79877,
+ "9": 10.77445,
+ "10": 10.70642,
+ "11": 10.83286,
"12": 10.81872,
- "13": 10.83078,
- "14": 10.83381,
- "15": 10.76396,
- "16": 10.76573,
- "17": 10.71925,
- "18": 10.76797,
- "19": 10.75316,
- "20": 10.70911,
- "21": 10.69217,
- "22": 10.56534,
- "23": 10.70907,
- "24": 10.6159,
- "25": 10.55058,
- "26": 10.62591,
- "27": 10.64705,
- "28": 10.63623,
- "29": 10.65641,
- "30": 10.43675,
- "31": 10.21912,
- "32": 10.5512,
- "33": 10.53381,
- "34": 10.31821,
+ "13": 10.83075,
+ "14": 10.8338,
+ "15": 10.76397,
+ "16": 10.76575,
+ "17": 10.71928,
+ "18": 10.768,
+ "19": 10.75318,
+ "20": 10.70918,
+ "21": 10.69214,
+ "22": 10.56531,
+ "23": 10.7091,
+ "24": 10.61591,
+ "25": 10.55061,
+ "26": 10.6259,
+ "27": 10.64706,
+ "28": 10.6362,
+ "29": 10.65644,
+ "30": 10.43679,
+ "31": 10.21909,
+ "32": 10.55114,
+ "33": 10.5338,
+ "34": 10.31817,
"35": 10.36833,
- "36": 10.3562,
- "37": 10.46302,
- "38": 10.33833,
- "39": 10.50306,
- "40": 10.23446,
- "41": 10.27335,
- "42": 10.3295,
- "43": 9.97414,
- "44": 10.1075,
- "45": 9.98853,
- "46": 9.95474,
- "47": 10.2514,
- "48": 10.01228,
- "49": 9.70796,
- "50": 10.05505,
+ "36": 10.35618,
+ "37": 10.46296,
+ "38": 10.33836,
+ "39": 10.50307,
+ "40": 10.23444,
+ "41": 10.2734,
+ "42": 10.32945,
+ "43": 9.97415,
+ "44": 10.10754,
+ "45": 9.9885,
+ "46": 9.95475,
+ "47": 10.25141,
+ "48": 10.01227,
+ "49": 9.70793,
+ "50": 10.05501,
"51": 9.9812,
- "52": 9.89198,
- "53": 10.19208,
+ "52": 9.89199,
+ "53": 10.19201,
"54": 10.09574,
- "55": 10.00506,
- "56": 9.78714,
- "57": 9.64607,
- "58": 9.9862,
- "59": 9.72684,
+ "55": 10.00502,
+ "56": 9.78719,
+ "57": 9.6461,
+ "58": 9.98626,
+ "59": 9.72683,
"60": 9.67172,
- "61": 9.80984,
+ "61": 9.80986,
"62": 10.11126,
- "63": 9.54877,
- "64": 9.90929,
- "65": 9.08735,
- "66": 9.84659,
- "67": 9.48264,
- "68": 9.89439,
- "69": 9.87695,
- "70": 9.82469,
- "71": 9.72751,
- "72": 9.72911,
- "73": 9.62051,
- "74": 9.11601,
- "75": 9.55057,
+ "63": 9.54873,
+ "64": 9.90931,
+ "65": 9.08736,
+ "66": 9.84658,
+ "67": 9.48259,
+ "68": 9.89433,
+ "69": 9.87692,
+ "70": 9.82465,
+ "71": 9.72749,
+ "72": 9.7291,
+ "73": 9.62049,
+ "74": 9.11605,
+ "75": 9.55059,
"76": 9.21504,
"77": 10.14893,
"78": 9.8138,
- "79": 9.47515,
- "80": 9.51582,
- "81": 9.58685,
- "82": 9.79026,
- "83": 9.45587,
+ "79": 9.4751,
+ "80": 9.51583,
+ "81": 9.58687,
+ "82": 9.79025,
+ "83": 9.45586,
"84": 9.50503,
"85": 9.71387,
- "86": 9.17463,
- "87": 9.66601,
- "88": 9.84354,
- "89": 9.70734,
- "90": 9.8955,
- "91": 9.48652,
- "92": 9.47023,
- "93": 9.21481,
- "94": 8.94327,
- "95": 9.6154,
- "96": 9.63634,
- "97": 9.37644,
- "98": 9.74975,
- "99": 9.01753,
- "100": 9.50515
+ "86": 9.17462,
+ "87": 9.666,
+ "88": 9.84355,
+ "89": 9.70736,
+ "90": 9.89548,
+ "91": 9.48655,
+ "92": 9.47022,
+ "93": 9.2148,
+ "94": 8.94328,
+ "95": 9.61538,
+ "96": 9.63633,
+ "97": 9.37646,
+ "98": 9.74974,
+ "99": 9.01759,
+ "100": 9.50514
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 2656.0,
- "2": 2663.0,
- "3": 2673.0,
- "4": 2426.0,
- "5": 2931.0,
- "6": 3062.0,
- "7": 2591.0,
- "8": 2693.0,
- "9": 2713.0,
- "10": 2502.0,
- "11": 2904.0,
- "12": 2792.0,
- "13": 2979.0,
- "14": 3000.0,
- "15": 2952.0,
- "16": 2860.0,
- "17": 2717.0,
- "18": 2802.0,
- "19": 2868.0,
- "20": 2620.0,
- "21": 2792.0,
- "22": 2532.0,
- "23": 2701.0,
- "24": 2580.0,
- "25": 2466.0,
- "26": 2839.0,
- "27": 2703.0,
- "28": 2719.0,
- "29": 2971.0,
- "30": 2755.0,
- "31": 2448.0,
- "32": 2670.0,
- "33": 2791.0,
- "34": 2439.0,
- "35": 2662.0,
- "36": 2496.0,
- "37": 2806.0,
- "38": 2697.0,
- "39": 2786.0,
- "40": 2539.0,
- "41": 2605.0,
- "42": 2640.0,
- "43": 2324.0,
- "44": 2548.0,
- "45": 2291.0,
- "46": 2437.0,
- "47": 2605.0,
- "48": 2395.0,
- "49": 2478.0,
- "50": 2633.0,
- "51": 2676.0,
- "52": 2581.0,
- "53": 2898.0,
- "54": 2849.0,
- "55": 2548.0,
- "56": 2661.0,
- "57": 2510.0,
- "58": 2758.0,
- "59": 2650.0,
- "60": 2242.0,
- "61": 2628.0,
- "62": 2899.0,
- "63": 2605.0,
- "64": 2939.0,
- "65": 2572.0,
- "66": 2896.0,
- "67": 2640.0,
- "68": 2709.0,
- "69": 2889.0,
- "70": 3012.0,
- "71": 2978.0,
- "72": 2536.0,
- "73": 2964.0,
- "74": 2163.0,
- "75": 2603.0,
- "76": 2974.0,
- "77": 3007.0,
- "78": 3138.0,
- "79": 3197.0,
- "80": 2984.0,
- "81": 3280.0,
- "82": 3341.0,
- "83": 2757.0,
- "84": 3399.0,
- "85": 3320.0,
- "86": 2882.0,
- "87": 3407.0,
- "88": 3278.0,
- "89": 3336.0,
- "90": 3322.0,
- "91": 2472.0,
- "92": 3061.0,
- "93": 2911.0,
- "94": 3005.0,
- "95": 2984.0,
- "96": 2991.0,
- "97": 3178.0,
- "98": 3343.0,
- "99": 2929.0,
- "100": 2588.0
+ "1": 2677.0,
+ "2": 2700.0,
+ "3": 2731.0,
+ "4": 2503.0,
+ "5": 2843.0,
+ "6": 2937.0,
+ "7": 2619.0,
+ "8": 2649.0,
+ "9": 2579.0,
+ "10": 2466.0,
+ "11": 2864.0,
+ "12": 2732.0,
+ "13": 2935.0,
+ "14": 2829.0,
+ "15": 2919.0,
+ "16": 2924.0,
+ "17": 2683.0,
+ "18": 2796.0,
+ "19": 2828.0,
+ "20": 2631.0,
+ "21": 2797.0,
+ "22": 2631.0,
+ "23": 2797.0,
+ "24": 2668.0,
+ "25": 2526.0,
+ "26": 2856.0,
+ "27": 2658.0,
+ "28": 2939.0,
+ "29": 3084.0,
+ "30": 2744.0,
+ "31": 2420.0,
+ "32": 2634.0,
+ "33": 2750.0,
+ "34": 2458.0,
+ "35": 2614.0,
+ "36": 2570.0,
+ "37": 2879.0,
+ "38": 2662.0,
+ "39": 2815.0,
+ "40": 2558.0,
+ "41": 2587.0,
+ "42": 2691.0,
+ "43": 2442.0,
+ "44": 2537.0,
+ "45": 2368.0,
+ "46": 2456.0,
+ "47": 2525.0,
+ "48": 2378.0,
+ "49": 2264.0,
+ "50": 2670.0,
+ "51": 2668.0,
+ "52": 2560.0,
+ "53": 2793.0,
+ "54": 2927.0,
+ "55": 2495.0,
+ "56": 2665.0,
+ "57": 2574.0,
+ "58": 2851.0,
+ "59": 2766.0,
+ "60": 2219.0,
+ "61": 2640.0,
+ "62": 2855.0,
+ "63": 2733.0,
+ "64": 3001.0,
+ "65": 2651.0,
+ "66": 2794.0,
+ "67": 2786.0,
+ "68": 2802.0,
+ "69": 2823.0,
+ "70": 2942.0,
+ "71": 2946.0,
+ "72": 2538.0,
+ "73": 2930.0,
+ "74": 2132.0,
+ "75": 2613.0,
+ "76": 2961.0,
+ "77": 2992.0,
+ "78": 3034.0,
+ "79": 3106.0,
+ "80": 3002.0,
+ "81": 3244.0,
+ "82": 3292.0,
+ "83": 2665.0,
+ "84": 3380.0,
+ "85": 3218.0,
+ "86": 2747.0,
+ "87": 3363.0,
+ "88": 3272.0,
+ "89": 3369.0,
+ "90": 3343.0,
+ "91": 2487.0,
+ "92": 2967.0,
+ "93": 2844.0,
+ "94": 2936.0,
+ "95": 3080.0,
+ "96": 3070.0,
+ "97": 3045.0,
+ "98": 3285.0,
+ "99": 2834.0,
+ "100": 2448.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 745732608.0,
- "2": 745732608.0,
- "3": 745732608.0,
- "4": 745732608.0,
- "5": 745732608.0,
- "6": 745732608.0,
- "7": 745732608.0,
- "8": 745732608.0,
- "9": 745732608.0,
- "10": 745732608.0,
- "11": 745732608.0,
- "12": 745732608.0,
- "13": 745732608.0,
- "14": 745732608.0,
- "15": 745732608.0,
- "16": 745732608.0,
- "17": 745732608.0,
- "18": 745732608.0,
- "19": 745732608.0,
- "20": 745732608.0,
- "21": 745732608.0,
- "22": 745732608.0,
- "23": 745732608.0,
- "24": 745732608.0,
- "25": 745732608.0,
- "26": 745732608.0,
- "27": 745732608.0,
- "28": 745732608.0,
- "29": 745732608.0,
- "30": 745732608.0,
- "31": 745732608.0,
- "32": 745732608.0,
- "33": 745732608.0,
- "34": 745732608.0,
- "35": 745732608.0,
- "36": 745732608.0,
- "37": 745732608.0,
- "38": 745732608.0,
- "39": 745732608.0,
- "40": 745732608.0,
- "41": 745732608.0,
- "42": 745732608.0,
- "43": 745732608.0,
- "44": 745732608.0,
- "45": 745732608.0,
- "46": 745732608.0,
- "47": 745732608.0,
- "48": 745732608.0,
- "49": 745732608.0,
- "50": 745732608.0,
- "51": 745732608.0,
- "52": 745732608.0,
- "53": 745732608.0,
- "54": 745732608.0,
- "55": 745732608.0,
- "56": 745732608.0,
- "57": 745732608.0,
- "58": 745732608.0,
- "59": 745732608.0,
- "60": 745732608.0,
- "61": 745732608.0,
- "62": 745732608.0,
- "63": 745732608.0,
- "64": 745732608.0,
- "65": 745732608.0,
- "66": 745732608.0,
- "67": 745732608.0,
- "68": 745732608.0,
- "69": 745732608.0,
- "70": 745732608.0,
- "71": 745732608.0,
- "72": 745732608.0,
- "73": 745732608.0,
- "74": 745732608.0,
- "75": 745732608.0,
- "76": 745732608.0,
- "77": 745732608.0,
- "78": 745732608.0,
- "79": 745732608.0,
- "80": 745732608.0,
- "81": 745732608.0,
- "82": 745732608.0,
- "83": 745732608.0,
- "84": 745732608.0,
- "85": 745732608.0,
- "86": 745732608.0,
- "87": 745732608.0,
- "88": 745732608.0,
- "89": 745732608.0,
- "90": 745732608.0,
- "91": 745732608.0,
- "92": 745732608.0,
- "93": 745732608.0,
- "94": 745732608.0,
- "95": 745732608.0,
- "96": 745732608.0,
- "97": 745732608.0,
- "98": 745732608.0,
- "99": 745732608.0,
- "100": 745732608.0
+ "1": 745077248.0,
+ "2": 745077248.0,
+ "3": 745077248.0,
+ "4": 745077248.0,
+ "5": 745077248.0,
+ "6": 745077248.0,
+ "7": 745077248.0,
+ "8": 745077248.0,
+ "9": 745077248.0,
+ "10": 745077248.0,
+ "11": 745077248.0,
+ "12": 745077248.0,
+ "13": 745077248.0,
+ "14": 745077248.0,
+ "15": 745077248.0,
+ "16": 745077248.0,
+ "17": 745077248.0,
+ "18": 745077248.0,
+ "19": 745077248.0,
+ "20": 745077248.0,
+ "21": 745077248.0,
+ "22": 745077248.0,
+ "23": 745077248.0,
+ "24": 745077248.0,
+ "25": 745077248.0,
+ "26": 745077248.0,
+ "27": 745077248.0,
+ "28": 745077248.0,
+ "29": 745077248.0,
+ "30": 745077248.0,
+ "31": 745077248.0,
+ "32": 745077248.0,
+ "33": 745077248.0,
+ "34": 745077248.0,
+ "35": 745077248.0,
+ "36": 745077248.0,
+ "37": 745077248.0,
+ "38": 745077248.0,
+ "39": 745077248.0,
+ "40": 745077248.0,
+ "41": 745077248.0,
+ "42": 745077248.0,
+ "43": 745077248.0,
+ "44": 745077248.0,
+ "45": 745077248.0,
+ "46": 745077248.0,
+ "47": 745077248.0,
+ "48": 745077248.0,
+ "49": 745077248.0,
+ "50": 745077248.0,
+ "51": 745077248.0,
+ "52": 745077248.0,
+ "53": 745077248.0,
+ "54": 745077248.0,
+ "55": 745077248.0,
+ "56": 745077248.0,
+ "57": 745077248.0,
+ "58": 745077248.0,
+ "59": 745077248.0,
+ "60": 745077248.0,
+ "61": 745077248.0,
+ "62": 745077248.0,
+ "63": 745077248.0,
+ "64": 745077248.0,
+ "65": 745077248.0,
+ "66": 745077248.0,
+ "67": 745077248.0,
+ "68": 745077248.0,
+ "69": 745077248.0,
+ "70": 745077248.0,
+ "71": 745077248.0,
+ "72": 745077248.0,
+ "73": 745077248.0,
+ "74": 745077248.0,
+ "75": 745077248.0,
+ "76": 745077248.0,
+ "77": 745077248.0,
+ "78": 745077248.0,
+ "79": 745077248.0,
+ "80": 745077248.0,
+ "81": 745077248.0,
+ "82": 745077248.0,
+ "83": 745077248.0,
+ "84": 745077248.0,
+ "85": 745077248.0,
+ "86": 745077248.0,
+ "87": 745077248.0,
+ "88": 745077248.0,
+ "89": 745077248.0,
+ "90": 745077248.0,
+ "91": 745077248.0,
+ "92": 745077248.0,
+ "93": 745077248.0,
+ "94": 745077248.0,
+ "95": 745077248.0,
+ "96": 745077248.0,
+ "97": 745077248.0,
+ "98": 745077248.0,
+ "99": 745077248.0,
+ "100": 745077248.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1940442112.0,
- "2": 2223151104.0,
- "3": 2223151104.0,
- "4": 2223151104.0,
- "5": 2223151104.0,
- "6": 2223151104.0,
- "7": 2223151104.0,
- "8": 2223151104.0,
- "9": 2223151104.0,
- "10": 2223151104.0,
- "11": 2223151104.0,
- "12": 2223151104.0,
- "13": 2223151104.0,
- "14": 2223151104.0,
- "15": 2223151104.0,
- "16": 2223151104.0,
- "17": 2223151104.0,
- "18": 2223151104.0,
- "19": 2223151104.0,
- "20": 2223151104.0,
- "21": 2223151104.0,
- "22": 2223151104.0,
- "23": 2223151104.0,
- "24": 2223151104.0,
- "25": 2223151104.0,
- "26": 2223151104.0,
- "27": 2223151104.0,
- "28": 2223151104.0,
- "29": 2223151104.0,
- "30": 2223151104.0,
- "31": 2223151104.0,
- "32": 2223151104.0,
- "33": 2223151104.0,
- "34": 2223151104.0,
- "35": 2223151104.0,
- "36": 2223151104.0,
- "37": 2223151104.0,
- "38": 2223151104.0,
- "39": 2223151104.0,
- "40": 2223151104.0,
- "41": 2223151104.0,
- "42": 2223151104.0,
- "43": 2223151104.0,
- "44": 2223151104.0,
- "45": 2223151104.0,
- "46": 2223151104.0,
- "47": 2223151104.0,
- "48": 2223151104.0,
- "49": 2223151104.0,
- "50": 2223151104.0,
- "51": 2223151104.0,
- "52": 2223151104.0,
- "53": 2223151104.0,
- "54": 2223151104.0,
- "55": 2223151104.0,
- "56": 2223151104.0,
- "57": 2223151104.0,
- "58": 2223151104.0,
- "59": 2223151104.0,
- "60": 2223151104.0,
- "61": 2223151104.0,
- "62": 2223151104.0,
- "63": 2223151104.0,
- "64": 2223151104.0,
- "65": 2223151104.0,
- "66": 2223151104.0,
- "67": 2223151104.0,
- "68": 2223151104.0,
- "69": 2223151104.0,
- "70": 2223151104.0,
- "71": 2223151104.0,
- "72": 2223151104.0,
- "73": 2223151104.0,
- "74": 2223151104.0,
- "75": 2223151104.0,
- "76": 2223151104.0,
- "77": 2223151104.0,
- "78": 2223151104.0,
- "79": 2223151104.0,
- "80": 2223151104.0,
- "81": 2223151104.0,
- "82": 2223151104.0,
- "83": 2223151104.0,
- "84": 2223151104.0,
- "85": 2223151104.0,
- "86": 2223151104.0,
- "87": 2223151104.0,
- "88": 2223151104.0,
- "89": 2223151104.0,
- "90": 2223151104.0,
- "91": 2223151104.0,
- "92": 2223151104.0,
- "93": 2223151104.0,
- "94": 2223151104.0,
- "95": 2223151104.0,
- "96": 2223151104.0,
- "97": 2223151104.0,
- "98": 2223151104.0,
- "99": 2223151104.0,
- "100": 2223151104.0
+ "1": 1939393536.0,
+ "2": 2220398592.0,
+ "3": 2220398592.0,
+ "4": 2220398592.0,
+ "5": 2220398592.0,
+ "6": 2220398592.0,
+ "7": 2220398592.0,
+ "8": 2220398592.0,
+ "9": 2220398592.0,
+ "10": 2220398592.0,
+ "11": 2220398592.0,
+ "12": 2220398592.0,
+ "13": 2220398592.0,
+ "14": 2220398592.0,
+ "15": 2220398592.0,
+ "16": 2220398592.0,
+ "17": 2220398592.0,
+ "18": 2220398592.0,
+ "19": 2220398592.0,
+ "20": 2220398592.0,
+ "21": 2220398592.0,
+ "22": 2220398592.0,
+ "23": 2220398592.0,
+ "24": 2220398592.0,
+ "25": 2220398592.0,
+ "26": 2220398592.0,
+ "27": 2220398592.0,
+ "28": 2220398592.0,
+ "29": 2220398592.0,
+ "30": 2220398592.0,
+ "31": 2220398592.0,
+ "32": 2220398592.0,
+ "33": 2220398592.0,
+ "34": 2220398592.0,
+ "35": 2220398592.0,
+ "36": 2220398592.0,
+ "37": 2220398592.0,
+ "38": 2220398592.0,
+ "39": 2220398592.0,
+ "40": 2220398592.0,
+ "41": 2220398592.0,
+ "42": 2220398592.0,
+ "43": 2220398592.0,
+ "44": 2220398592.0,
+ "45": 2220398592.0,
+ "46": 2220398592.0,
+ "47": 2220398592.0,
+ "48": 2220398592.0,
+ "49": 2220398592.0,
+ "50": 2220398592.0,
+ "51": 2220398592.0,
+ "52": 2220398592.0,
+ "53": 2220398592.0,
+ "54": 2220398592.0,
+ "55": 2220398592.0,
+ "56": 2220398592.0,
+ "57": 2220398592.0,
+ "58": 2220398592.0,
+ "59": 2220398592.0,
+ "60": 2220398592.0,
+ "61": 2220398592.0,
+ "62": 2220398592.0,
+ "63": 2220398592.0,
+ "64": 2220398592.0,
+ "65": 2220398592.0,
+ "66": 2220398592.0,
+ "67": 2220398592.0,
+ "68": 2220398592.0,
+ "69": 2220398592.0,
+ "70": 2220398592.0,
+ "71": 2220398592.0,
+ "72": 2220398592.0,
+ "73": 2220398592.0,
+ "74": 2220398592.0,
+ "75": 2220398592.0,
+ "76": 2220398592.0,
+ "77": 2220398592.0,
+ "78": 2220398592.0,
+ "79": 2220398592.0,
+ "80": 2220398592.0,
+ "81": 2220398592.0,
+ "82": 2220398592.0,
+ "83": 2220398592.0,
+ "84": 2220398592.0,
+ "85": 2220398592.0,
+ "86": 2220398592.0,
+ "87": 2220398592.0,
+ "88": 2220398592.0,
+ "89": 2220398592.0,
+ "90": 2220398592.0,
+ "91": 2220398592.0,
+ "92": 2220398592.0,
+ "93": 2220398592.0,
+ "94": 2220398592.0,
+ "95": 2220398592.0,
+ "96": 2220398592.0,
+ "97": 2220398592.0,
+ "98": 2220398592.0,
+ "99": 2220398592.0,
+ "100": 2220398592.0
}
},
"iteration-time": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.5568,
- "3": 0.14788,
- "4": 0.13602,
- "5": 0.13596,
- "6": 0.136,
- "7": 0.13621,
- "8": 0.13502,
- "9": 0.13408,
- "10": 0.23083,
- "11": 0.14377,
- "12": 0.14332,
- "13": 0.15453,
- "14": 0.15537,
- "15": 0.15549,
- "16": 0.15444,
- "17": 0.15453,
- "18": 0.15178,
- "19": 0.21432,
- "20": 0.15336,
- "21": 0.1534,
- "22": 0.15483,
- "23": 0.15395,
- "24": 0.15469,
- "25": 0.15447,
- "26": 0.15509,
- "27": 0.1545,
- "28": 0.15527,
- "29": 0.15593,
- "30": 0.15688,
- "31": 0.15659,
- "32": 0.15629,
- "33": 0.15533,
- "34": 0.155,
- "35": 0.15519,
- "36": 0.15784,
- "37": 0.15943,
- "38": 0.15552,
- "39": 0.15486,
- "40": 0.15539,
- "41": 0.15618,
- "42": 0.15569,
- "43": 0.15725,
- "44": 0.15522,
- "45": 0.1553,
- "46": 0.15719,
- "47": 0.15571,
- "48": 0.15568,
- "49": 0.15362,
- "50": 0.15495,
- "51": 0.18287,
- "52": 0.16115,
- "53": 0.15739,
- "54": 0.15665,
- "55": 0.15684,
- "56": 0.15658,
- "57": 0.15631,
- "58": 0.22153,
- "59": 0.15604,
- "60": 0.15313,
- "61": 0.15485,
- "62": 0.15518,
- "63": 0.15719,
- "64": 0.15757,
- "65": 0.15904,
- "66": 0.15846,
- "67": 0.15846,
- "68": 0.15754,
- "69": 0.15779,
- "70": 0.1589,
- "71": 0.16037,
- "72": 0.15778,
- "73": 0.15771,
- "74": 0.155,
- "75": 0.15611,
- "76": 0.15702,
- "77": 0.15564,
- "78": 0.15892,
- "79": 0.15669,
- "80": 0.15768,
- "81": 0.15805,
- "82": 0.15778,
- "83": 0.15674,
- "84": 0.15715,
- "85": 0.15834,
- "86": 0.15763,
- "87": 0.15855,
- "88": 0.15589,
- "89": 0.15616,
- "90": 0.15639,
- "91": 0.15722,
- "92": 0.15788,
- "93": 0.15597,
- "94": 0.15817,
- "95": 0.15819,
- "96": 0.15869,
- "97": 0.15875,
- "98": 0.15993,
- "99": 0.16297,
- "100": 0.16682
+ "2": 6.8425,
+ "3": 0.15357,
+ "4": 0.1409,
+ "5": 0.13808,
+ "6": 0.1382,
+ "7": 0.24407,
+ "8": 0.13904,
+ "9": 0.13868,
+ "10": 0.13899,
+ "11": 0.13745,
+ "12": 0.13793,
+ "13": 0.13808,
+ "14": 0.1368,
+ "15": 0.13736,
+ "16": 0.13801,
+ "17": 0.13947,
+ "18": 0.13945,
+ "19": 0.13791,
+ "20": 0.13947,
+ "21": 0.13849,
+ "22": 0.13877,
+ "23": 0.13852,
+ "24": 0.13794,
+ "25": 0.13904,
+ "26": 0.14025,
+ "27": 0.13916,
+ "28": 0.13997,
+ "29": 0.1407,
+ "30": 0.13911,
+ "31": 0.13955,
+ "32": 0.1446,
+ "33": 0.24847,
+ "34": 0.14784,
+ "35": 0.14131,
+ "36": 0.13933,
+ "37": 0.13988,
+ "38": 0.19634,
+ "39": 0.14058,
+ "40": 0.14008,
+ "41": 0.14147,
+ "42": 0.14265,
+ "43": 0.1426,
+ "44": 0.14006,
+ "45": 0.14114,
+ "46": 0.14113,
+ "47": 0.1398,
+ "48": 0.14109,
+ "49": 0.14027,
+ "50": 0.13929,
+ "51": 0.16842,
+ "52": 0.14006,
+ "53": 0.13988,
+ "54": 0.13768,
+ "55": 0.13634,
+ "56": 0.13659,
+ "57": 0.13814,
+ "58": 0.13574,
+ "59": 0.13686,
+ "60": 0.1366,
+ "61": 0.13869,
+ "62": 0.13965,
+ "63": 0.13601,
+ "64": 0.13824,
+ "65": 0.13849,
+ "66": 0.13984,
+ "67": 0.13968,
+ "68": 0.13808,
+ "69": 0.13877,
+ "70": 0.14016,
+ "71": 0.13855,
+ "72": 0.13617,
+ "73": 0.13908,
+ "74": 0.13686,
+ "75": 0.13824,
+ "76": 0.13807,
+ "77": 0.13594,
+ "78": 0.13948,
+ "79": 0.13994,
+ "80": 0.13781,
+ "81": 0.13907,
+ "82": 0.14148,
+ "83": 0.13897,
+ "84": 0.13796,
+ "85": 0.13672,
+ "86": 0.13869,
+ "87": 0.13816,
+ "88": 0.13546,
+ "89": 0.13874,
+ "90": 0.13865,
+ "91": 0.21702,
+ "92": 0.13637,
+ "93": 0.13856,
+ "94": 0.14121,
+ "95": 0.1418,
+ "96": 0.16074,
+ "97": 0.14631,
+ "98": 0.13758,
+ "99": 0.13713,
+ "100": 0.13749
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu/golden_values_dev_dgx_h100.json
index 42889e09b26..df7d09f1ff4 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.76985,
- "2": 10.81791,
- "3": 10.784,
- "4": 10.788,
- "5": 10.81927,
- "6": 10.84306,
- "7": 10.83464,
- "8": 10.8066,
- "9": 10.83359,
- "10": 10.73562,
- "11": 10.86814,
- "12": 10.85075,
- "13": 10.84505,
- "14": 10.87136,
- "15": 10.8218,
- "16": 10.80433,
- "17": 10.76124,
- "18": 10.80363,
- "19": 10.80599,
- "20": 10.74747,
- "21": 10.7254,
- "22": 10.60597,
- "23": 10.74387,
- "24": 10.65549,
- "25": 10.58002,
- "26": 10.64496,
- "27": 10.67191,
- "28": 10.66903,
- "29": 10.66652,
- "30": 10.46947,
- "31": 10.26264,
- "32": 10.56932,
- "33": 10.54232,
- "34": 10.36113,
- "35": 10.39558,
- "36": 10.36866,
- "37": 10.47523,
- "38": 10.33715,
- "39": 10.49947,
- "40": 10.23019,
- "41": 10.30905,
- "42": 10.33124,
- "43": 9.99091,
- "44": 10.09605,
- "45": 10.00787,
- "46": 9.96718,
- "47": 10.27077,
- "48": 10.01043,
- "49": 9.73437,
- "50": 10.04737,
- "51": 10.00084,
- "52": 9.89672,
- "53": 10.19876,
- "54": 10.09066,
- "55": 10.00567,
- "56": 9.77199,
- "57": 9.64533,
- "58": 9.98587,
- "59": 9.72608,
- "60": 9.6777,
- "61": 9.8157,
- "62": 10.092,
- "63": 9.54758,
- "64": 9.90438,
- "65": 9.09492,
- "66": 9.84068,
- "67": 9.48471,
- "68": 9.88996,
- "69": 9.87691,
- "70": 9.85294,
- "71": 9.73278,
- "72": 9.72558,
- "73": 9.63706,
- "74": 9.12334,
- "75": 9.55335,
- "76": 9.21765,
- "77": 10.15202,
- "78": 9.81465,
- "79": 9.47558,
- "80": 9.52073,
- "81": 9.5872,
- "82": 9.79125,
- "83": 9.44848,
- "84": 9.49585,
- "85": 9.72189,
- "86": 9.18037,
- "87": 9.66127,
- "88": 9.84359,
- "89": 9.71651,
- "90": 9.88102,
- "91": 9.48434,
- "92": 9.4705,
- "93": 9.20911,
- "94": 8.95382,
- "95": 9.60554,
- "96": 9.63976,
- "97": 9.38762,
- "98": 9.7573,
- "99": 9.0159,
- "100": 9.49925
+ "1": 10.94575,
+ "2": 10.93036,
+ "3": 10.93989,
+ "4": 10.92854,
+ "5": 10.94067,
+ "6": 10.92027,
+ "7": 10.91075,
+ "8": 10.91871,
+ "9": 10.93775,
+ "10": 10.92951,
+ "11": 10.91943,
+ "12": 10.92019,
+ "13": 10.89852,
+ "14": 10.89452,
+ "15": 10.90931,
+ "16": 10.88661,
+ "17": 10.88329,
+ "18": 10.90109,
+ "19": 10.8827,
+ "20": 10.83406,
+ "21": 10.83303,
+ "22": 10.82287,
+ "23": 10.8169,
+ "24": 10.8026,
+ "25": 10.7917,
+ "26": 10.78278,
+ "27": 10.78025,
+ "28": 10.71808,
+ "29": 10.68588,
+ "30": 10.68319,
+ "31": 10.66265,
+ "32": 10.64447,
+ "33": 10.62267,
+ "34": 10.57809,
+ "35": 10.58654,
+ "36": 10.5668,
+ "37": 10.53535,
+ "38": 10.54913,
+ "39": 10.49567,
+ "40": 10.49455,
+ "41": 10.4582,
+ "42": 10.43695,
+ "43": 10.41522,
+ "44": 10.40243,
+ "45": 10.39942,
+ "46": 10.35779,
+ "47": 10.34704,
+ "48": 10.29667,
+ "49": 10.30208,
+ "50": 10.29585,
+ "51": 10.2907,
+ "52": 10.23405,
+ "53": 10.23807,
+ "54": 10.20269,
+ "55": 10.17502,
+ "56": 10.19664,
+ "57": 10.17823,
+ "58": 10.18664,
+ "59": 10.12966,
+ "60": 10.14462,
+ "61": 10.09835,
+ "62": 10.06554,
+ "63": 10.13722,
+ "64": 10.0802,
+ "65": 10.04441,
+ "66": 10.08449,
+ "67": 10.05722,
+ "68": 10.01729,
+ "69": 10.03677,
+ "70": 10.01613,
+ "71": 10.0411,
+ "72": 10.02309,
+ "73": 9.99614,
+ "74": 9.99487,
+ "75": 9.95691,
+ "76": 9.98979,
+ "77": 9.98475,
+ "78": 9.94176,
+ "79": 9.94149,
+ "80": 9.94901,
+ "81": 9.97693,
+ "82": 9.92835,
+ "83": 9.88608,
+ "84": 9.83048,
+ "85": 9.81314,
+ "86": 9.92454,
+ "87": 9.94185,
+ "88": 9.90934,
+ "89": 9.86332,
+ "90": 9.85944,
+ "91": 9.86772,
+ "92": 9.85571,
+ "93": 9.79337,
+ "94": 9.86532,
+ "95": 9.8526,
+ "96": 9.8385,
+ "97": 9.78976,
+ "98": 9.81564,
+ "99": 9.86693,
+ "100": 9.75754
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 2680.0,
- "2": 2615.0,
- "3": 2642.0,
- "4": 2479.0,
- "5": 2971.0,
- "6": 2822.0,
- "7": 2833.0,
- "8": 2508.0,
- "9": 2922.0,
- "10": 2508.0,
- "11": 2917.0,
- "12": 2817.0,
- "13": 2935.0,
- "14": 2969.0,
- "15": 2679.0,
- "16": 2976.0,
- "17": 2609.0,
- "18": 2868.0,
- "19": 2790.0,
- "20": 2461.0,
- "21": 2636.0,
- "22": 2356.0,
- "23": 2798.0,
- "24": 2613.0,
- "25": 2640.0,
- "26": 2701.0,
- "27": 2761.0,
- "28": 2801.0,
- "29": 2971.0,
- "30": 2590.0,
- "31": 2307.0,
- "32": 2751.0,
- "33": 2881.0,
- "34": 2352.0,
- "35": 2480.0,
- "36": 2443.0,
- "37": 2748.0,
- "38": 2692.0,
- "39": 2709.0,
- "40": 2570.0,
- "41": 2752.0,
- "42": 2689.0,
- "43": 2381.0,
- "44": 2483.0,
- "45": 2397.0,
- "46": 2281.0,
- "47": 2684.0,
- "48": 2330.0,
- "49": 2293.0,
- "50": 2740.0,
- "51": 2575.0,
- "52": 2621.0,
- "53": 2891.0,
- "54": 2655.0,
- "55": 2559.0,
- "56": 2566.0,
- "57": 2471.0,
- "58": 2767.0,
- "59": 2529.0,
- "60": 2289.0,
- "61": 2642.0,
- "62": 2820.0,
- "63": 2654.0,
- "64": 3020.0,
- "65": 2687.0,
- "66": 2884.0,
- "67": 2666.0,
- "68": 2720.0,
- "69": 2738.0,
- "70": 3004.0,
- "71": 2816.0,
- "72": 2537.0,
- "73": 2826.0,
- "74": 2192.0,
- "75": 2647.0,
- "76": 3048.0,
- "77": 3019.0,
- "78": 3134.0,
- "79": 3092.0,
- "80": 3054.0,
- "81": 3298.0,
- "82": 3350.0,
- "83": 2597.0,
- "84": 3436.0,
- "85": 3350.0,
- "86": 2993.0,
- "87": 3509.0,
- "88": 3403.0,
- "89": 3490.0,
- "90": 3368.0,
- "91": 2461.0,
- "92": 2803.0,
- "93": 2933.0,
- "94": 2888.0,
- "95": 3138.0,
- "96": 3047.0,
- "97": 3016.0,
- "98": 3382.0,
- "99": 2995.0,
- "100": 2490.0
+ "1": 2934.0,
+ "2": 2872.0,
+ "3": 3049.0,
+ "4": 2828.0,
+ "5": 3061.0,
+ "6": 2933.0,
+ "7": 3149.0,
+ "8": 2726.0,
+ "9": 2823.0,
+ "10": 3016.0,
+ "11": 2957.0,
+ "12": 2757.0,
+ "13": 2936.0,
+ "14": 3051.0,
+ "15": 2883.0,
+ "16": 2857.0,
+ "17": 2960.0,
+ "18": 2847.0,
+ "19": 2855.0,
+ "20": 2795.0,
+ "21": 2863.0,
+ "22": 2794.0,
+ "23": 2933.0,
+ "24": 2925.0,
+ "25": 2786.0,
+ "26": 2812.0,
+ "27": 2832.0,
+ "28": 2882.0,
+ "29": 2955.0,
+ "30": 2866.0,
+ "31": 2809.0,
+ "32": 2744.0,
+ "33": 2541.0,
+ "34": 2824.0,
+ "35": 2686.0,
+ "36": 2694.0,
+ "37": 2660.0,
+ "38": 2718.0,
+ "39": 2688.0,
+ "40": 2689.0,
+ "41": 2564.0,
+ "42": 2464.0,
+ "43": 2536.0,
+ "44": 2638.0,
+ "45": 2806.0,
+ "46": 2605.0,
+ "47": 2594.0,
+ "48": 2713.0,
+ "49": 2613.0,
+ "50": 2605.0,
+ "51": 2697.0,
+ "52": 2691.0,
+ "53": 2633.0,
+ "54": 2821.0,
+ "55": 2569.0,
+ "56": 2906.0,
+ "57": 2709.0,
+ "58": 2987.0,
+ "59": 2819.0,
+ "60": 2834.0,
+ "61": 2796.0,
+ "62": 2877.0,
+ "63": 2999.0,
+ "64": 3126.0,
+ "65": 2797.0,
+ "66": 2920.0,
+ "67": 3398.0,
+ "68": 3118.0,
+ "69": 2982.0,
+ "70": 2976.0,
+ "71": 3172.0,
+ "72": 2997.0,
+ "73": 3157.0,
+ "74": 3087.0,
+ "75": 3007.0,
+ "76": 3353.0,
+ "77": 3536.0,
+ "78": 3306.0,
+ "79": 3308.0,
+ "80": 3040.0,
+ "81": 3267.0,
+ "82": 3247.0,
+ "83": 3206.0,
+ "84": 3166.0,
+ "85": 2863.0,
+ "86": 3146.0,
+ "87": 3080.0,
+ "88": 3245.0,
+ "89": 3182.0,
+ "90": 3575.0,
+ "91": 2827.0,
+ "92": 3217.0,
+ "93": 3110.0,
+ "94": 3462.0,
+ "95": 3390.0,
+ "96": 3694.0,
+ "97": 3510.0,
+ "98": 3549.0,
+ "99": 2994.0,
+ "100": 3311.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 744815104.0,
- "2": 744815104.0,
- "3": 744815104.0,
- "4": 744815104.0,
- "5": 744815104.0,
- "6": 744815104.0,
- "7": 744815104.0,
- "8": 744815104.0,
- "9": 744815104.0,
- "10": 744815104.0,
- "11": 744815104.0,
- "12": 744815104.0,
- "13": 744815104.0,
- "14": 744815104.0,
- "15": 744815104.0,
- "16": 744815104.0,
- "17": 744815104.0,
- "18": 744815104.0,
- "19": 744815104.0,
- "20": 744815104.0,
- "21": 744815104.0,
- "22": 744815104.0,
- "23": 744815104.0,
- "24": 744815104.0,
- "25": 744815104.0,
- "26": 744815104.0,
- "27": 744815104.0,
- "28": 744815104.0,
- "29": 744815104.0,
- "30": 744815104.0,
- "31": 744815104.0,
- "32": 744815104.0,
- "33": 744815104.0,
- "34": 744815104.0,
- "35": 744815104.0,
- "36": 744815104.0,
- "37": 744815104.0,
- "38": 744815104.0,
- "39": 744815104.0,
- "40": 744815104.0,
- "41": 744815104.0,
- "42": 744815104.0,
- "43": 744815104.0,
- "44": 744815104.0,
- "45": 744815104.0,
- "46": 744815104.0,
- "47": 744815104.0,
- "48": 744815104.0,
- "49": 744815104.0,
- "50": 744815104.0,
- "51": 744815104.0,
- "52": 744815104.0,
- "53": 744815104.0,
- "54": 744815104.0,
- "55": 744815104.0,
- "56": 744815104.0,
- "57": 744815104.0,
- "58": 744815104.0,
- "59": 744815104.0,
- "60": 744815104.0,
- "61": 744815104.0,
- "62": 744815104.0,
- "63": 744815104.0,
- "64": 744815104.0,
- "65": 744815104.0,
- "66": 744815104.0,
- "67": 744815104.0,
- "68": 744815104.0,
- "69": 744815104.0,
- "70": 744815104.0,
- "71": 744815104.0,
- "72": 744815104.0,
- "73": 744815104.0,
- "74": 744815104.0,
- "75": 744815104.0,
- "76": 744815104.0,
- "77": 744815104.0,
- "78": 744815104.0,
- "79": 744815104.0,
- "80": 744815104.0,
- "81": 744815104.0,
- "82": 744815104.0,
- "83": 744815104.0,
- "84": 744815104.0,
- "85": 744815104.0,
- "86": 744815104.0,
- "87": 744815104.0,
- "88": 744815104.0,
- "89": 744815104.0,
- "90": 744815104.0,
- "91": 744815104.0,
- "92": 744815104.0,
- "93": 744815104.0,
- "94": 744815104.0,
- "95": 744815104.0,
- "96": 744815104.0,
- "97": 744815104.0,
- "98": 744815104.0,
- "99": 744815104.0,
- "100": 744815104.0
+ "1": 745732608.0,
+ "2": 745732608.0,
+ "3": 745732608.0,
+ "4": 745732608.0,
+ "5": 745732608.0,
+ "6": 745732608.0,
+ "7": 745732608.0,
+ "8": 745732608.0,
+ "9": 745732608.0,
+ "10": 745732608.0,
+ "11": 745732608.0,
+ "12": 745732608.0,
+ "13": 745732608.0,
+ "14": 745732608.0,
+ "15": 745732608.0,
+ "16": 745732608.0,
+ "17": 745732608.0,
+ "18": 745732608.0,
+ "19": 745732608.0,
+ "20": 745732608.0,
+ "21": 745732608.0,
+ "22": 745732608.0,
+ "23": 745732608.0,
+ "24": 745732608.0,
+ "25": 745732608.0,
+ "26": 745732608.0,
+ "27": 745732608.0,
+ "28": 745732608.0,
+ "29": 745732608.0,
+ "30": 745732608.0,
+ "31": 745732608.0,
+ "32": 745732608.0,
+ "33": 745732608.0,
+ "34": 745732608.0,
+ "35": 745732608.0,
+ "36": 745732608.0,
+ "37": 745732608.0,
+ "38": 745732608.0,
+ "39": 745732608.0,
+ "40": 745732608.0,
+ "41": 745732608.0,
+ "42": 745732608.0,
+ "43": 745732608.0,
+ "44": 745732608.0,
+ "45": 745732608.0,
+ "46": 745732608.0,
+ "47": 745732608.0,
+ "48": 745732608.0,
+ "49": 745732608.0,
+ "50": 745732608.0,
+ "51": 745732608.0,
+ "52": 745732608.0,
+ "53": 745732608.0,
+ "54": 745732608.0,
+ "55": 745732608.0,
+ "56": 745732608.0,
+ "57": 745732608.0,
+ "58": 745732608.0,
+ "59": 745732608.0,
+ "60": 745732608.0,
+ "61": 745732608.0,
+ "62": 745732608.0,
+ "63": 745732608.0,
+ "64": 745732608.0,
+ "65": 745732608.0,
+ "66": 745732608.0,
+ "67": 745732608.0,
+ "68": 745732608.0,
+ "69": 745732608.0,
+ "70": 745732608.0,
+ "71": 745732608.0,
+ "72": 745732608.0,
+ "73": 745732608.0,
+ "74": 745732608.0,
+ "75": 745732608.0,
+ "76": 745732608.0,
+ "77": 745732608.0,
+ "78": 745732608.0,
+ "79": 745732608.0,
+ "80": 745732608.0,
+ "81": 745732608.0,
+ "82": 745732608.0,
+ "83": 745732608.0,
+ "84": 745732608.0,
+ "85": 745732608.0,
+ "86": 745732608.0,
+ "87": 745732608.0,
+ "88": 745732608.0,
+ "89": 745732608.0,
+ "90": 745732608.0,
+ "91": 745732608.0,
+ "92": 745732608.0,
+ "93": 745732608.0,
+ "94": 745732608.0,
+ "95": 745732608.0,
+ "96": 745732608.0,
+ "97": 745732608.0,
+ "98": 745732608.0,
+ "99": 745732608.0,
+ "100": 745732608.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1928907776.0,
- "2": 2210305536.0,
- "3": 2210305536.0,
- "4": 2210305536.0,
- "5": 2210305536.0,
- "6": 2210305536.0,
- "7": 2210305536.0,
- "8": 2210305536.0,
- "9": 2210305536.0,
- "10": 2210305536.0,
- "11": 2210305536.0,
- "12": 2210305536.0,
- "13": 2210305536.0,
- "14": 2210305536.0,
- "15": 2210305536.0,
- "16": 2210305536.0,
- "17": 2210305536.0,
- "18": 2210305536.0,
- "19": 2210305536.0,
- "20": 2210305536.0,
- "21": 2210305536.0,
- "22": 2210305536.0,
- "23": 2210305536.0,
- "24": 2210305536.0,
- "25": 2210305536.0,
- "26": 2210305536.0,
- "27": 2210305536.0,
- "28": 2210305536.0,
- "29": 2210305536.0,
- "30": 2210305536.0,
- "31": 2210305536.0,
- "32": 2210305536.0,
- "33": 2210305536.0,
- "34": 2210305536.0,
- "35": 2210305536.0,
- "36": 2210305536.0,
- "37": 2210305536.0,
- "38": 2210305536.0,
- "39": 2210305536.0,
- "40": 2210305536.0,
- "41": 2210305536.0,
- "42": 2210305536.0,
- "43": 2210305536.0,
- "44": 2210305536.0,
- "45": 2210305536.0,
- "46": 2210305536.0,
- "47": 2210305536.0,
- "48": 2210305536.0,
- "49": 2210305536.0,
- "50": 2210305536.0,
- "51": 2210305536.0,
- "52": 2210305536.0,
- "53": 2210305536.0,
- "54": 2210305536.0,
- "55": 2210305536.0,
- "56": 2210305536.0,
- "57": 2210305536.0,
- "58": 2210305536.0,
- "59": 2210305536.0,
- "60": 2210305536.0,
- "61": 2210305536.0,
- "62": 2210305536.0,
- "63": 2210305536.0,
- "64": 2210305536.0,
- "65": 2210305536.0,
- "66": 2210305536.0,
- "67": 2210305536.0,
- "68": 2210305536.0,
- "69": 2210305536.0,
- "70": 2210305536.0,
- "71": 2210305536.0,
- "72": 2210305536.0,
- "73": 2210305536.0,
- "74": 2210305536.0,
- "75": 2210305536.0,
- "76": 2210305536.0,
- "77": 2210305536.0,
- "78": 2210305536.0,
- "79": 2210305536.0,
- "80": 2210305536.0,
- "81": 2210305536.0,
- "82": 2210305536.0,
- "83": 2210305536.0,
- "84": 2210305536.0,
- "85": 2210305536.0,
- "86": 2210305536.0,
- "87": 2210305536.0,
- "88": 2210305536.0,
- "89": 2210305536.0,
- "90": 2210305536.0,
- "91": 2210305536.0,
- "92": 2210305536.0,
- "93": 2210305536.0,
- "94": 2210305536.0,
- "95": 2210305536.0,
- "96": 2210305536.0,
- "97": 2210305536.0,
- "98": 2210305536.0,
- "99": 2210305536.0,
- "100": 2210305536.0
+ "1": 1927859200.0,
+ "2": 2211485184.0,
+ "3": 2211485184.0,
+ "4": 2211485184.0,
+ "5": 2211485184.0,
+ "6": 2211485184.0,
+ "7": 2211485184.0,
+ "8": 2211485184.0,
+ "9": 2211485184.0,
+ "10": 2211485184.0,
+ "11": 2211485184.0,
+ "12": 2211485184.0,
+ "13": 2211485184.0,
+ "14": 2211485184.0,
+ "15": 2211485184.0,
+ "16": 2211485184.0,
+ "17": 2211485184.0,
+ "18": 2211485184.0,
+ "19": 2211485184.0,
+ "20": 2211485184.0,
+ "21": 2211485184.0,
+ "22": 2211485184.0,
+ "23": 2211485184.0,
+ "24": 2211485184.0,
+ "25": 2211485184.0,
+ "26": 2211485184.0,
+ "27": 2211485184.0,
+ "28": 2211485184.0,
+ "29": 2211485184.0,
+ "30": 2211485184.0,
+ "31": 2211485184.0,
+ "32": 2211485184.0,
+ "33": 2211485184.0,
+ "34": 2211485184.0,
+ "35": 2211485184.0,
+ "36": 2211485184.0,
+ "37": 2211485184.0,
+ "38": 2211485184.0,
+ "39": 2211485184.0,
+ "40": 2211485184.0,
+ "41": 2211485184.0,
+ "42": 2211485184.0,
+ "43": 2211485184.0,
+ "44": 2211485184.0,
+ "45": 2211485184.0,
+ "46": 2211485184.0,
+ "47": 2211485184.0,
+ "48": 2211485184.0,
+ "49": 2211485184.0,
+ "50": 2211485184.0,
+ "51": 2211485184.0,
+ "52": 2211485184.0,
+ "53": 2211485184.0,
+ "54": 2211485184.0,
+ "55": 2211485184.0,
+ "56": 2211485184.0,
+ "57": 2211485184.0,
+ "58": 2211485184.0,
+ "59": 2211485184.0,
+ "60": 2211485184.0,
+ "61": 2211485184.0,
+ "62": 2211485184.0,
+ "63": 2211485184.0,
+ "64": 2211485184.0,
+ "65": 2211485184.0,
+ "66": 2211485184.0,
+ "67": 2211485184.0,
+ "68": 2211485184.0,
+ "69": 2211485184.0,
+ "70": 2211485184.0,
+ "71": 2211485184.0,
+ "72": 2211485184.0,
+ "73": 2211485184.0,
+ "74": 2211485184.0,
+ "75": 2211485184.0,
+ "76": 2211485184.0,
+ "77": 2211485184.0,
+ "78": 2211485184.0,
+ "79": 2211485184.0,
+ "80": 2211485184.0,
+ "81": 2211485184.0,
+ "82": 2211485184.0,
+ "83": 2211485184.0,
+ "84": 2211485184.0,
+ "85": 2211485184.0,
+ "86": 2211485184.0,
+ "87": 2211485184.0,
+ "88": 2211485184.0,
+ "89": 2211485184.0,
+ "90": 2211485184.0,
+ "91": 2211485184.0,
+ "92": 2211485184.0,
+ "93": 2211485184.0,
+ "94": 2211485184.0,
+ "95": 2211485184.0,
+ "96": 2211485184.0,
+ "97": 2211485184.0,
+ "98": 2211485184.0,
+ "99": 2211485184.0,
+ "100": 2211485184.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 38.50475,
- "2": 0.14031,
- "3": 0.11652,
- "4": 0.09549,
- "5": 0.09354,
- "6": 0.09569,
- "7": 0.09409,
- "8": 0.09473,
- "9": 0.09388,
- "10": 0.09459,
- "11": 0.09596,
- "12": 0.09466,
- "13": 0.09509,
- "14": 0.09586,
- "15": 0.09314,
- "16": 0.09368,
- "17": 0.09468,
- "18": 0.09494,
- "19": 0.09289,
- "20": 0.09427,
- "21": 0.09599,
- "22": 0.09701,
- "23": 0.09665,
- "24": 0.09712,
- "25": 0.09542,
- "26": 0.09515,
- "27": 0.09642,
- "28": 0.09519,
- "29": 0.09691,
- "30": 0.09651,
- "31": 0.09742,
- "32": 0.09503,
- "33": 0.09471,
- "34": 0.09424,
- "35": 0.09574,
- "36": 0.09438,
- "37": 0.09509,
- "38": 0.09428,
- "39": 0.09484,
- "40": 0.09459,
- "41": 0.0951,
- "42": 0.09671,
- "43": 0.09633,
- "44": 0.09511,
- "45": 0.09592,
- "46": 0.09579,
- "47": 0.09614,
- "48": 0.09464,
- "49": 0.0958,
- "50": 0.09782,
- "51": 0.10564,
- "52": 0.09373,
- "53": 0.09475,
- "54": 0.09323,
- "55": 0.09237,
- "56": 0.09293,
- "57": 0.09228,
- "58": 0.0948,
- "59": 0.09906,
- "60": 0.10026,
- "61": 0.09961,
- "62": 0.09923,
- "63": 0.09889,
- "64": 0.09888,
- "65": 0.09925,
- "66": 0.1,
- "67": 0.09782,
- "68": 0.09891,
- "69": 0.09132,
- "70": 0.09102,
- "71": 0.091,
- "72": 0.09368,
- "73": 0.09219,
- "74": 0.09374,
- "75": 0.09232,
- "76": 0.09428,
- "77": 0.09256,
- "78": 0.09623,
- "79": 0.09624,
- "80": 0.09622,
- "81": 0.09668,
- "82": 0.09651,
- "83": 0.10042,
- "84": 0.09998,
- "85": 0.10102,
- "86": 0.09975,
- "87": 0.09955,
- "88": 0.10135,
- "89": 0.10038,
- "90": 0.09933,
- "91": 0.10071,
- "92": 0.09992,
- "93": 0.10054,
- "94": 0.09927,
- "95": 0.0998,
- "96": 0.101,
- "97": 0.09268,
- "98": 0.09188,
- "99": 0.09185,
- "100": 0.09107
+ "1": "nan",
+ "2": 7.46324,
+ "3": 0.1116,
+ "4": 0.09285,
+ "5": 0.09253,
+ "6": 0.09365,
+ "7": 0.09437,
+ "8": 0.09425,
+ "9": 0.09292,
+ "10": 0.09228,
+ "11": 0.09297,
+ "12": 0.0935,
+ "13": 0.09984,
+ "14": 0.09998,
+ "15": 0.10003,
+ "16": 0.09392,
+ "17": 0.09203,
+ "18": 0.09291,
+ "19": 0.09183,
+ "20": 0.09308,
+ "21": 0.09184,
+ "22": 0.09229,
+ "23": 0.09175,
+ "24": 0.09233,
+ "25": 0.09204,
+ "26": 0.09184,
+ "27": 0.09386,
+ "28": 0.09276,
+ "29": 0.09231,
+ "30": 0.09234,
+ "31": 0.09384,
+ "32": 0.09213,
+ "33": 0.09274,
+ "34": 0.0946,
+ "35": 0.09266,
+ "36": 0.09258,
+ "37": 0.09228,
+ "38": 0.09221,
+ "39": 0.09238,
+ "40": 0.09225,
+ "41": 0.09459,
+ "42": 0.09329,
+ "43": 0.09229,
+ "44": 0.09257,
+ "45": 0.09248,
+ "46": 0.09217,
+ "47": 0.09279,
+ "48": 0.09317,
+ "49": 0.09227,
+ "50": 0.09198,
+ "51": 0.1313,
+ "52": 0.09522,
+ "53": 0.09587,
+ "54": 0.09554,
+ "55": 0.09496,
+ "56": 0.09513,
+ "57": 0.095,
+ "58": 0.0971,
+ "59": 0.1019,
+ "60": 0.09949,
+ "61": 0.09893,
+ "62": 0.09465,
+ "63": 0.0931,
+ "64": 0.09354,
+ "65": 0.09516,
+ "66": 0.09657,
+ "67": 0.09436,
+ "68": 0.09426,
+ "69": 0.09439,
+ "70": 0.09309,
+ "71": 0.09326,
+ "72": 0.0926,
+ "73": 0.09269,
+ "74": 0.09224,
+ "75": 0.09283,
+ "76": 0.09274,
+ "77": 0.09356,
+ "78": 0.09325,
+ "79": 0.09252,
+ "80": 0.09217,
+ "81": 0.09211,
+ "82": 0.09217,
+ "83": 0.09255,
+ "84": 0.094,
+ "85": 0.09258,
+ "86": 0.09325,
+ "87": 0.09262,
+ "88": 0.09433,
+ "89": 0.09343,
+ "90": 0.09302,
+ "91": 0.09433,
+ "92": 0.09306,
+ "93": 0.09267,
+ "94": 0.09345,
+ "95": 0.09246,
+ "96": 0.09262,
+ "97": 0.09269,
+ "98": 0.09232,
+ "99": 0.09232,
+ "100": 0.09234
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu/model_config.yaml
index 1e25f4bd4e1..a39ef7f4f78 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,5 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs/golden_values_dev_dgx_h100.json
index 4943a180a1f..0a18434164d 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.91349,
- "2": 10.90719,
- "3": 10.91328,
- "4": 10.87838,
- "5": 10.91769,
- "6": 10.93821,
- "7": 10.90469,
- "8": 10.90393,
- "9": 10.90876,
- "10": 10.89645,
- "11": 10.92562,
- "12": 10.91891,
- "13": 10.91537,
- "14": 10.93343,
- "15": 10.86115,
- "16": 10.85374,
- "17": 10.82717,
- "18": 10.86544,
- "19": 10.86225,
- "20": 10.76737,
- "21": 10.74634,
- "22": 10.62228,
- "23": 10.76122,
- "24": 10.64732,
- "25": 10.59597,
- "26": 10.66352,
- "27": 10.6542,
- "28": 10.6077,
- "29": 10.62581,
- "30": 10.41591,
- "31": 10.16855,
- "32": 10.50267,
- "33": 10.50304,
- "34": 10.25481,
- "35": 10.31879,
- "36": 10.27167,
- "37": 10.37751,
- "38": 10.22122,
- "39": 10.44798,
- "40": 10.14166,
- "41": 10.1771,
- "42": 10.2426,
- "43": 9.87148,
- "44": 9.99875,
- "45": 9.88702,
- "46": 9.86139,
- "47": 10.18144,
- "48": 9.87873,
- "49": 9.58706,
- "50": 9.9542,
- "51": 9.8866,
- "52": 9.78429,
- "53": 10.10842,
- "54": 9.97368,
- "55": 9.89803,
- "56": 9.65427,
- "57": 9.52013,
- "58": 9.87297,
- "59": 9.6132,
- "60": 9.54967,
- "61": 9.70681,
- "62": 9.98533,
- "63": 9.41357,
- "64": 9.80966,
- "65": 8.97052,
- "66": 9.72773,
- "67": 9.39183,
- "68": 9.8084,
- "69": 9.82052,
- "70": 9.76655,
- "71": 9.63414,
- "72": 9.60485,
- "73": 9.52299,
- "74": 8.9718,
- "75": 9.42321,
- "76": 9.10113,
- "77": 10.0716,
- "78": 9.74266,
- "79": 9.40343,
- "80": 9.41333,
- "81": 9.49931,
- "82": 9.70236,
- "83": 9.33436,
- "84": 9.43774,
- "85": 9.63924,
- "86": 9.07931,
- "87": 9.60447,
- "88": 9.7824,
- "89": 9.62386,
- "90": 9.84241,
- "91": 9.35506,
- "92": 9.38398,
- "93": 9.09747,
- "94": 8.8471,
- "95": 9.5314,
- "96": 9.54263,
- "97": 9.32886,
- "98": 9.6926,
- "99": 8.89976,
- "100": 9.43124
+ "1": 10.9543,
+ "2": 10.97008,
+ "3": 10.96655,
+ "4": 10.96406,
+ "5": 10.96112,
+ "6": 10.95417,
+ "7": 10.9563,
+ "8": 10.95685,
+ "9": 10.96029,
+ "10": 10.95842,
+ "11": 10.95613,
+ "12": 10.93265,
+ "13": 10.95939,
+ "14": 10.94942,
+ "15": 10.92047,
+ "16": 10.90004,
+ "17": 10.91861,
+ "18": 10.90006,
+ "19": 10.91328,
+ "20": 10.82085,
+ "21": 10.81358,
+ "22": 10.80496,
+ "23": 10.78959,
+ "24": 10.75475,
+ "25": 10.7592,
+ "26": 10.7484,
+ "27": 10.69715,
+ "28": 10.63737,
+ "29": 10.61522,
+ "30": 10.5879,
+ "31": 10.5788,
+ "32": 10.55824,
+ "33": 10.52368,
+ "34": 10.50297,
+ "35": 10.48358,
+ "36": 10.46151,
+ "37": 10.45408,
+ "38": 10.45225,
+ "39": 10.4093,
+ "40": 10.40239,
+ "41": 10.37867,
+ "42": 10.32564,
+ "43": 10.31661,
+ "44": 10.27992,
+ "45": 10.30836,
+ "46": 10.24722,
+ "47": 10.23865,
+ "48": 10.2225,
+ "49": 10.2059,
+ "50": 10.18441,
+ "51": 10.19258,
+ "52": 10.15915,
+ "53": 10.16845,
+ "54": 10.11269,
+ "55": 10.08473,
+ "56": 10.10194,
+ "57": 10.12337,
+ "58": 10.10238,
+ "59": 10.0515,
+ "60": 10.08325,
+ "61": 10.02214,
+ "62": 9.99325,
+ "63": 10.06756,
+ "64": 10.02654,
+ "65": 10.02786,
+ "66": 10.02547,
+ "67": 9.99692,
+ "68": 9.97259,
+ "69": 10.02187,
+ "70": 9.98227,
+ "71": 9.98458,
+ "72": 10.00073,
+ "73": 9.96104,
+ "74": 9.95154,
+ "75": 9.91431,
+ "76": 9.95248,
+ "77": 9.94444,
+ "78": 9.91063,
+ "79": 9.90323,
+ "80": 9.91081,
+ "81": 9.9326,
+ "82": 9.86694,
+ "83": 9.84166,
+ "84": 9.79598,
+ "85": 9.76666,
+ "86": 9.88045,
+ "87": 9.86771,
+ "88": 9.86222,
+ "89": 9.8026,
+ "90": 9.78295,
+ "91": 9.82695,
+ "92": 9.79115,
+ "93": 9.72005,
+ "94": 9.81835,
+ "95": 9.79024,
+ "96": 9.8114,
+ "97": 9.74202,
+ "98": 9.76131,
+ "99": 9.79042,
+ "100": 9.70983
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 22727424.0,
- "2": 22925204.0,
- "3": 22596900.0,
- "4": 23219556.0,
- "5": 22714624.0,
- "6": 23021776.0,
- "7": 22771632.0,
- "8": 22926560.0,
- "9": 22842156.0,
- "10": 22918168.0,
- "11": 22500688.0,
- "12": 22459470.0,
- "13": 22917228.0,
- "14": 22387988.0,
- "15": 22821732.0,
- "16": 22830306.0,
- "17": 22819520.0,
- "18": 22582628.0,
- "19": 22618028.0,
- "20": 22693852.0,
- "21": 22739344.0,
- "22": 22799596.0,
- "23": 22539016.0,
- "24": 22770946.0,
- "25": 22819324.0,
- "26": 22547928.0,
- "27": 22468716.0,
- "28": 22453820.0,
- "29": 22529898.0,
- "30": 22631220.0,
- "31": 22955420.0,
- "32": 22585276.0,
- "33": 22558602.0,
- "34": 22835792.0,
- "35": 22788208.0,
- "36": 22589796.0,
- "37": 22496928.0,
- "38": 22896192.0,
- "39": 22801858.0,
- "40": 22657640.0,
- "41": 22658982.0,
- "42": 22667052.0,
- "43": 22975816.0,
- "44": 22747688.0,
- "45": 22674846.0,
- "46": 22884684.0,
- "47": 22633708.0,
- "48": 22928466.0,
- "49": 22728092.0,
- "50": 22905080.0,
- "51": 22791108.0,
- "52": 22748190.0,
- "53": 22924900.0,
- "54": 22840164.0,
- "55": 22518344.0,
- "56": 22877680.0,
- "57": 23113944.0,
- "58": 22846268.0,
- "59": 22716084.0,
- "60": 22742984.0,
- "61": 22724584.0,
- "62": 22672944.0,
- "63": 22846388.0,
- "64": 22823650.0,
- "65": 23061058.0,
- "66": 22729266.0,
- "67": 22908888.0,
- "68": 22610020.0,
- "69": 22583826.0,
- "70": 22829374.0,
- "71": 22748240.0,
- "72": 22654480.0,
- "73": 22741180.0,
- "74": 23047914.0,
- "75": 23054396.0,
- "76": 22900788.0,
- "77": 22271588.0,
- "78": 22789024.0,
- "79": 22743632.0,
- "80": 22706696.0,
- "81": 22891372.0,
- "82": 22777860.0,
- "83": 22840532.0,
- "84": 23010386.0,
- "85": 22711212.0,
- "86": 23103006.0,
- "87": 22734564.0,
- "88": 22637848.0,
- "89": 22497850.0,
- "90": 22972712.0,
- "91": 22767188.0,
- "92": 22808834.0,
- "93": 22659304.0,
- "94": 22911552.0,
- "95": 23047794.0,
- "96": 22829386.0,
- "97": 22608168.0,
- "98": 22762756.0,
- "99": 22905900.0,
- "100": 23015488.0
+ "1": 22962220.0,
+ "2": 22849880.0,
+ "3": 22709192.0,
+ "4": 22778682.0,
+ "5": 22783512.0,
+ "6": 22746660.0,
+ "7": 22872212.0,
+ "8": 22611264.0,
+ "9": 22757990.0,
+ "10": 22489100.0,
+ "11": 22753568.0,
+ "12": 22640822.0,
+ "13": 23322448.0,
+ "14": 22991688.0,
+ "15": 22717790.0,
+ "16": 22822824.0,
+ "17": 22929888.0,
+ "18": 22994636.0,
+ "19": 23084314.0,
+ "20": 22722876.0,
+ "21": 22919016.0,
+ "22": 22944468.0,
+ "23": 22628368.0,
+ "24": 22862112.0,
+ "25": 22636396.0,
+ "26": 23004736.0,
+ "27": 22802388.0,
+ "28": 22998952.0,
+ "29": 22979518.0,
+ "30": 22943884.0,
+ "31": 22907836.0,
+ "32": 22665794.0,
+ "33": 22740580.0,
+ "34": 23075970.0,
+ "35": 22750998.0,
+ "36": 22695464.0,
+ "37": 23103384.0,
+ "38": 22963994.0,
+ "39": 22984992.0,
+ "40": 22749568.0,
+ "41": 23064380.0,
+ "42": 22688536.0,
+ "43": 22987394.0,
+ "44": 22705448.0,
+ "45": 22847156.0,
+ "46": 22733644.0,
+ "47": 22849228.0,
+ "48": 22833464.0,
+ "49": 22889024.0,
+ "50": 22644544.0,
+ "51": 22699028.0,
+ "52": 22813184.0,
+ "53": 22961728.0,
+ "54": 22786038.0,
+ "55": 22922926.0,
+ "56": 22659904.0,
+ "57": 23193380.0,
+ "58": 22689146.0,
+ "59": 22841966.0,
+ "60": 23013882.0,
+ "61": 22672900.0,
+ "62": 22731948.0,
+ "63": 22632328.0,
+ "64": 22998984.0,
+ "65": 23198280.0,
+ "66": 22694556.0,
+ "67": 22955908.0,
+ "68": 22924840.0,
+ "69": 23150048.0,
+ "70": 22821720.0,
+ "71": 22739556.0,
+ "72": 23116436.0,
+ "73": 23133004.0,
+ "74": 22941168.0,
+ "75": 22880520.0,
+ "76": 22702682.0,
+ "77": 22980470.0,
+ "78": 22977228.0,
+ "79": 22824124.0,
+ "80": 22931384.0,
+ "81": 22834756.0,
+ "82": 22730576.0,
+ "83": 22735244.0,
+ "84": 23099118.0,
+ "85": 22922208.0,
+ "86": 23068192.0,
+ "87": 22373756.0,
+ "88": 22555678.0,
+ "89": 22727804.0,
+ "90": 22766028.0,
+ "91": 22919372.0,
+ "92": 22672728.0,
+ "93": 22646244.0,
+ "94": 23128424.0,
+ "95": 22695022.0,
+ "96": 22844930.0,
+ "97": 22831776.0,
+ "98": 22875330.0,
+ "99": 22636804.0,
+ "100": 23000018.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 747492864.0,
- "2": 747492864.0,
- "3": 747492864.0,
- "4": 747492864.0,
- "5": 747492864.0,
- "6": 747492864.0,
- "7": 747492864.0,
- "8": 747492864.0,
- "9": 747492864.0,
- "10": 747492864.0,
- "11": 747492864.0,
- "12": 747492864.0,
- "13": 747492864.0,
- "14": 747492864.0,
- "15": 747492864.0,
- "16": 747492864.0,
- "17": 747492864.0,
- "18": 747492864.0,
- "19": 747492864.0,
- "20": 747492864.0,
- "21": 747492864.0,
- "22": 747492864.0,
- "23": 747492864.0,
- "24": 747492864.0,
- "25": 747492864.0,
- "26": 747492864.0,
- "27": 747492864.0,
- "28": 747492864.0,
- "29": 747492864.0,
- "30": 747492864.0,
- "31": 747492864.0,
- "32": 747492864.0,
- "33": 747492864.0,
- "34": 747492864.0,
- "35": 747492864.0,
- "36": 747492864.0,
- "37": 747492864.0,
- "38": 747492864.0,
- "39": 747492864.0,
- "40": 747492864.0,
- "41": 747492864.0,
- "42": 747492864.0,
- "43": 747492864.0,
- "44": 747492864.0,
- "45": 747492864.0,
- "46": 747492864.0,
- "47": 747492864.0,
- "48": 747492864.0,
- "49": 747492864.0,
- "50": 747492864.0,
- "51": 747492864.0,
- "52": 747492864.0,
- "53": 747492864.0,
- "54": 747492864.0,
- "55": 747492864.0,
- "56": 747492864.0,
- "57": 747492864.0,
- "58": 747492864.0,
- "59": 747492864.0,
- "60": 747492864.0,
- "61": 747492864.0,
- "62": 747492864.0,
- "63": 747492864.0,
- "64": 747492864.0,
- "65": 747492864.0,
- "66": 747492864.0,
- "67": 747492864.0,
- "68": 747492864.0,
- "69": 747492864.0,
- "70": 747492864.0,
- "71": 747492864.0,
- "72": 747492864.0,
- "73": 747492864.0,
- "74": 747492864.0,
- "75": 747492864.0,
- "76": 747492864.0,
- "77": 747492864.0,
- "78": 747492864.0,
- "79": 747492864.0,
- "80": 747492864.0,
- "81": 747492864.0,
- "82": 747492864.0,
- "83": 747492864.0,
- "84": 747492864.0,
- "85": 747492864.0,
- "86": 747492864.0,
- "87": 747492864.0,
- "88": 747492864.0,
- "89": 747492864.0,
- "90": 747492864.0,
- "91": 747492864.0,
- "92": 747492864.0,
- "93": 747492864.0,
- "94": 747492864.0,
- "95": 747492864.0,
- "96": 747492864.0,
- "97": 747492864.0,
- "98": 747492864.0,
- "99": 747492864.0,
- "100": 747492864.0
+ "1": 745395712.0,
+ "2": 745395712.0,
+ "3": 745395712.0,
+ "4": 745395712.0,
+ "5": 745395712.0,
+ "6": 745395712.0,
+ "7": 745395712.0,
+ "8": 745395712.0,
+ "9": 745395712.0,
+ "10": 745395712.0,
+ "11": 745395712.0,
+ "12": 745395712.0,
+ "13": 745395712.0,
+ "14": 745395712.0,
+ "15": 745395712.0,
+ "16": 745395712.0,
+ "17": 745395712.0,
+ "18": 745395712.0,
+ "19": 745395712.0,
+ "20": 745395712.0,
+ "21": 745395712.0,
+ "22": 745395712.0,
+ "23": 745395712.0,
+ "24": 745395712.0,
+ "25": 745395712.0,
+ "26": 745395712.0,
+ "27": 745395712.0,
+ "28": 745395712.0,
+ "29": 745395712.0,
+ "30": 745395712.0,
+ "31": 745395712.0,
+ "32": 745395712.0,
+ "33": 745395712.0,
+ "34": 745395712.0,
+ "35": 745395712.0,
+ "36": 745395712.0,
+ "37": 745395712.0,
+ "38": 745395712.0,
+ "39": 745395712.0,
+ "40": 745395712.0,
+ "41": 745395712.0,
+ "42": 745395712.0,
+ "43": 745395712.0,
+ "44": 745395712.0,
+ "45": 745395712.0,
+ "46": 745395712.0,
+ "47": 745395712.0,
+ "48": 745395712.0,
+ "49": 745395712.0,
+ "50": 745395712.0,
+ "51": 745395712.0,
+ "52": 745395712.0,
+ "53": 745395712.0,
+ "54": 745395712.0,
+ "55": 745395712.0,
+ "56": 745395712.0,
+ "57": 745395712.0,
+ "58": 745395712.0,
+ "59": 745395712.0,
+ "60": 745395712.0,
+ "61": 745395712.0,
+ "62": 745395712.0,
+ "63": 745395712.0,
+ "64": 745395712.0,
+ "65": 745395712.0,
+ "66": 745395712.0,
+ "67": 745395712.0,
+ "68": 745395712.0,
+ "69": 745395712.0,
+ "70": 745395712.0,
+ "71": 745395712.0,
+ "72": 745395712.0,
+ "73": 745395712.0,
+ "74": 745395712.0,
+ "75": 745395712.0,
+ "76": 745395712.0,
+ "77": 745395712.0,
+ "78": 745395712.0,
+ "79": 745395712.0,
+ "80": 745395712.0,
+ "81": 745395712.0,
+ "82": 745395712.0,
+ "83": 745395712.0,
+ "84": 745395712.0,
+ "85": 745395712.0,
+ "86": 745395712.0,
+ "87": 745395712.0,
+ "88": 745395712.0,
+ "89": 745395712.0,
+ "90": 745395712.0,
+ "91": 745395712.0,
+ "92": 745395712.0,
+ "93": 745395712.0,
+ "94": 745395712.0,
+ "95": 745395712.0,
+ "96": 745395712.0,
+ "97": 745395712.0,
+ "98": 745395712.0,
+ "99": 745395712.0,
+ "100": 745395712.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1927341056.0,
- "2": 2212197376.0,
- "3": 2212197376.0,
- "4": 2212197376.0,
- "5": 2212197376.0,
- "6": 2212197376.0,
- "7": 2212197376.0,
- "8": 2212197376.0,
- "9": 2212197376.0,
- "10": 2212197376.0,
- "11": 2212197376.0,
- "12": 2212197376.0,
- "13": 2212197376.0,
- "14": 2212197376.0,
- "15": 2212197376.0,
- "16": 2212197376.0,
- "17": 2212197376.0,
- "18": 2212197376.0,
- "19": 2212197376.0,
- "20": 2212197376.0,
- "21": 2212197376.0,
- "22": 2212197376.0,
- "23": 2212197376.0,
- "24": 2212197376.0,
- "25": 2212197376.0,
- "26": 2212197376.0,
- "27": 2212197376.0,
- "28": 2212197376.0,
- "29": 2212197376.0,
- "30": 2212197376.0,
- "31": 2212197376.0,
- "32": 2212197376.0,
- "33": 2212197376.0,
- "34": 2212197376.0,
- "35": 2212197376.0,
- "36": 2212197376.0,
- "37": 2212197376.0,
- "38": 2212197376.0,
- "39": 2212197376.0,
- "40": 2212197376.0,
- "41": 2212197376.0,
- "42": 2212197376.0,
- "43": 2212197376.0,
- "44": 2212197376.0,
- "45": 2212197376.0,
- "46": 2212197376.0,
- "47": 2212197376.0,
- "48": 2212197376.0,
- "49": 2212197376.0,
- "50": 2212197376.0,
- "51": 2212197376.0,
- "52": 2212197376.0,
- "53": 2212197376.0,
- "54": 2212197376.0,
- "55": 2212197376.0,
- "56": 2212197376.0,
- "57": 2212197376.0,
- "58": 2212197376.0,
- "59": 2212197376.0,
- "60": 2212197376.0,
- "61": 2212197376.0,
- "62": 2212197376.0,
- "63": 2212197376.0,
- "64": 2212197376.0,
- "65": 2212197376.0,
- "66": 2212197376.0,
- "67": 2212197376.0,
- "68": 2212197376.0,
- "69": 2212197376.0,
- "70": 2212197376.0,
- "71": 2212197376.0,
- "72": 2212197376.0,
- "73": 2212197376.0,
- "74": 2212197376.0,
- "75": 2212197376.0,
- "76": 2212197376.0,
- "77": 2212197376.0,
- "78": 2212197376.0,
- "79": 2212197376.0,
- "80": 2212197376.0,
- "81": 2212197376.0,
- "82": 2212197376.0,
- "83": 2212197376.0,
- "84": 2212197376.0,
- "85": 2212197376.0,
- "86": 2212197376.0,
- "87": 2212197376.0,
- "88": 2212197376.0,
- "89": 2212197376.0,
- "90": 2212197376.0,
- "91": 2212197376.0,
- "92": 2212197376.0,
- "93": 2212197376.0,
- "94": 2212197376.0,
- "95": 2212197376.0,
- "96": 2212197376.0,
- "97": 2212197376.0,
- "98": 2212197376.0,
- "99": 2212197376.0,
- "100": 2212197376.0
+ "1": 1926292480.0,
+ "2": 2210100224.0,
+ "3": 2210100224.0,
+ "4": 2210100224.0,
+ "5": 2210100224.0,
+ "6": 2210100224.0,
+ "7": 2210100224.0,
+ "8": 2210100224.0,
+ "9": 2210100224.0,
+ "10": 2210100224.0,
+ "11": 2210100224.0,
+ "12": 2210100224.0,
+ "13": 2210100224.0,
+ "14": 2210100224.0,
+ "15": 2210100224.0,
+ "16": 2210100224.0,
+ "17": 2210100224.0,
+ "18": 2210100224.0,
+ "19": 2210100224.0,
+ "20": 2210100224.0,
+ "21": 2210100224.0,
+ "22": 2210100224.0,
+ "23": 2210100224.0,
+ "24": 2210100224.0,
+ "25": 2210100224.0,
+ "26": 2210100224.0,
+ "27": 2210100224.0,
+ "28": 2210100224.0,
+ "29": 2210100224.0,
+ "30": 2210100224.0,
+ "31": 2210100224.0,
+ "32": 2210100224.0,
+ "33": 2210100224.0,
+ "34": 2210100224.0,
+ "35": 2210100224.0,
+ "36": 2210100224.0,
+ "37": 2210100224.0,
+ "38": 2210100224.0,
+ "39": 2210100224.0,
+ "40": 2210100224.0,
+ "41": 2210100224.0,
+ "42": 2210100224.0,
+ "43": 2210100224.0,
+ "44": 2210100224.0,
+ "45": 2210100224.0,
+ "46": 2210100224.0,
+ "47": 2210100224.0,
+ "48": 2210100224.0,
+ "49": 2210100224.0,
+ "50": 2210100224.0,
+ "51": 2210100224.0,
+ "52": 2210100224.0,
+ "53": 2210100224.0,
+ "54": 2210100224.0,
+ "55": 2210100224.0,
+ "56": 2210100224.0,
+ "57": 2210100224.0,
+ "58": 2210100224.0,
+ "59": 2210100224.0,
+ "60": 2210100224.0,
+ "61": 2210100224.0,
+ "62": 2210100224.0,
+ "63": 2210100224.0,
+ "64": 2210100224.0,
+ "65": 2210100224.0,
+ "66": 2210100224.0,
+ "67": 2210100224.0,
+ "68": 2210100224.0,
+ "69": 2210100224.0,
+ "70": 2210100224.0,
+ "71": 2210100224.0,
+ "72": 2210100224.0,
+ "73": 2210100224.0,
+ "74": 2210100224.0,
+ "75": 2210100224.0,
+ "76": 2210100224.0,
+ "77": 2210100224.0,
+ "78": 2210100224.0,
+ "79": 2210100224.0,
+ "80": 2210100224.0,
+ "81": 2210100224.0,
+ "82": 2210100224.0,
+ "83": 2210100224.0,
+ "84": 2210100224.0,
+ "85": 2210100224.0,
+ "86": 2210100224.0,
+ "87": 2210100224.0,
+ "88": 2210100224.0,
+ "89": 2210100224.0,
+ "90": 2210100224.0,
+ "91": 2210100224.0,
+ "92": 2210100224.0,
+ "93": 2210100224.0,
+ "94": 2210100224.0,
+ "95": 2210100224.0,
+ "96": 2210100224.0,
+ "97": 2210100224.0,
+ "98": 2210100224.0,
+ "99": 2210100224.0,
+ "100": 2210100224.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 9.78643,
- "2": 0.13398,
- "3": 0.11557,
- "4": 0.09095,
- "5": 0.09137,
- "6": 0.09276,
- "7": 0.09034,
- "8": 0.09082,
- "9": 0.09002,
- "10": 0.09121,
- "11": 0.08989,
- "12": 0.0895,
- "13": 0.09015,
- "14": 0.09012,
- "15": 0.0903,
- "16": 0.09019,
- "17": 0.0907,
- "18": 0.09055,
- "19": 0.08988,
- "20": 0.08984,
- "21": 0.08951,
- "22": 0.0913,
- "23": 0.08972,
- "24": 0.08995,
- "25": 0.09008,
- "26": 0.08931,
- "27": 0.09055,
- "28": 0.08926,
- "29": 0.09028,
- "30": 0.09142,
- "31": 0.09085,
- "32": 0.09027,
- "33": 0.09061,
- "34": 0.08998,
- "35": 0.09113,
- "36": 0.09039,
- "37": 0.08973,
- "38": 0.09065,
- "39": 0.08993,
- "40": 0.09112,
- "41": 0.10695,
- "42": 0.11371,
- "43": 0.09964,
- "44": 0.09076,
- "45": 0.0899,
- "46": 0.09204,
- "47": 0.0904,
- "48": 0.08998,
- "49": 0.09097,
- "50": 0.08971,
- "51": 0.10825,
- "52": 0.097,
- "53": 0.09456,
- "54": 0.09109,
- "55": 0.09071,
- "56": 0.09099,
- "57": 0.09129,
- "58": 0.09159,
- "59": 0.09138,
- "60": 0.09089,
- "61": 0.09092,
- "62": 0.09153,
- "63": 0.09208,
- "64": 0.09107,
- "65": 0.0918,
- "66": 0.09116,
- "67": 0.09075,
- "68": 0.09166,
- "69": 0.0948,
- "70": 0.09166,
- "71": 0.09195,
- "72": 0.09271,
- "73": 0.09226,
- "74": 0.09271,
- "75": 0.09216,
- "76": 0.09129,
- "77": 0.09221,
- "78": 0.09252,
- "79": 0.09161,
- "80": 0.09144,
- "81": 0.09112,
- "82": 0.09152,
- "83": 0.09106,
- "84": 0.09137,
- "85": 0.09127,
- "86": 0.09136,
- "87": 0.09077,
- "88": 0.09362,
- "89": 0.09244,
- "90": 0.09162,
- "91": 0.09114,
- "92": 0.09065,
- "93": 0.0913,
- "94": 0.09071,
- "95": 0.09096,
- "96": 0.09066,
- "97": 0.09585,
- "98": 0.09148,
- "99": 0.09232,
- "100": 0.09229
+ "1": "nan",
+ "2": 6.31175,
+ "3": 0.11936,
+ "4": 0.09417,
+ "5": 0.09321,
+ "6": 0.09279,
+ "7": 0.09256,
+ "8": 0.09329,
+ "9": 0.09493,
+ "10": 0.093,
+ "11": 0.0927,
+ "12": 0.09164,
+ "13": 0.09135,
+ "14": 0.09096,
+ "15": 0.09096,
+ "16": 0.09229,
+ "17": 0.09097,
+ "18": 0.09104,
+ "19": 0.09051,
+ "20": 0.09154,
+ "21": 0.09096,
+ "22": 0.0915,
+ "23": 0.09092,
+ "24": 0.09094,
+ "25": 0.09057,
+ "26": 0.09095,
+ "27": 0.09157,
+ "28": 0.09065,
+ "29": 0.09146,
+ "30": 0.09078,
+ "31": 0.0923,
+ "32": 0.09142,
+ "33": 0.09135,
+ "34": 0.09083,
+ "35": 0.09086,
+ "36": 0.09323,
+ "37": 0.0928,
+ "38": 0.09281,
+ "39": 0.0926,
+ "40": 0.09228,
+ "41": 0.09296,
+ "42": 0.09248,
+ "43": 0.0948,
+ "44": 0.10259,
+ "45": 0.09307,
+ "46": 0.09027,
+ "47": 0.09032,
+ "48": 0.09124,
+ "49": 0.09095,
+ "50": 0.09089,
+ "51": 0.12578,
+ "52": 0.09563,
+ "53": 0.09257,
+ "54": 0.09229,
+ "55": 0.09351,
+ "56": 0.09268,
+ "57": 0.09289,
+ "58": 0.0945,
+ "59": 0.09515,
+ "60": 0.09697,
+ "61": 0.0956,
+ "62": 0.09438,
+ "63": 0.09506,
+ "64": 0.09457,
+ "65": 0.09405,
+ "66": 0.09513,
+ "67": 0.09524,
+ "68": 0.09415,
+ "69": 0.09451,
+ "70": 0.09308,
+ "71": 0.09324,
+ "72": 0.09323,
+ "73": 0.09431,
+ "74": 0.09396,
+ "75": 0.09368,
+ "76": 0.0945,
+ "77": 0.0943,
+ "78": 0.09495,
+ "79": 0.095,
+ "80": 0.09377,
+ "81": 0.09338,
+ "82": 0.09408,
+ "83": 0.09444,
+ "84": 0.09433,
+ "85": 0.09464,
+ "86": 0.09292,
+ "87": 0.09398,
+ "88": 0.09396,
+ "89": 0.09439,
+ "90": 0.09418,
+ "91": 0.09302,
+ "92": 0.09483,
+ "93": 0.09441,
+ "94": 0.09332,
+ "95": 0.09426,
+ "96": 0.0942,
+ "97": 0.09474,
+ "98": 0.09451,
+ "99": 0.09413,
+ "100": 0.09363
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs/model_config.yaml
index 2d734908089..5d1a4257402 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -50,4 +50,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_sequence_parallel/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_sequence_parallel/model_config.yaml
index 319164782fc..c201855b87a 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_sequence_parallel/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_sequence_parallel/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_swiglu/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_swiglu/model_config.yaml
index ea6f2520553..fcadf67b1c0 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_swiglu/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_swiglu/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,5 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_untie_embeddings_and_outputs/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_untie_embeddings_and_outputs/model_config.yaml
index ea8f4bb71d0..5b2bc318437 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_untie_embeddings_and_outputs/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_untie_embeddings_and_outputs/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -50,4 +50,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1/golden_values_dev_dgx_gb200.json
index 2610b7fe2f4..65904c58aca 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1/golden_values_dev_dgx_gb200.json
@@ -8,52 +8,52 @@
"2": 10.83322,
"3": 10.82737,
"4": 10.79588,
- "5": 10.85708,
- "6": 10.86392,
- "7": 10.8269,
- "8": 10.82589,
- "9": 10.83705,
- "10": 10.79716,
+ "5": 10.85705,
+ "6": 10.8639,
+ "7": 10.82692,
+ "8": 10.82592,
+ "9": 10.83704,
+ "10": 10.79717,
"11": 10.87851,
- "12": 10.85794,
- "13": 10.8537,
- "14": 10.87547,
- "15": 10.79179,
- "16": 10.80303,
- "17": 10.7745,
- "18": 10.804,
- "19": 10.79363,
- "20": 10.69591,
+ "12": 10.85796,
+ "13": 10.85375,
+ "14": 10.8755,
+ "15": 10.79176,
+ "16": 10.80297,
+ "17": 10.77451,
+ "18": 10.80401,
+ "19": 10.79366,
+ "20": 10.69587,
"21": 10.68551,
- "22": 10.53149,
- "23": 10.70658,
- "24": 10.57317,
- "25": 10.51546,
- "26": 10.59072,
- "27": 10.60736,
+ "22": 10.53152,
+ "23": 10.70657,
+ "24": 10.57319,
+ "25": 10.51544,
+ "26": 10.59074,
+ "27": 10.60737,
"28": 10.57024,
- "29": 10.58904,
- "30": 10.34679,
- "31": 10.07734,
- "32": 10.46319,
- "33": 10.45704,
- "34": 10.19923,
- "35": 10.25593,
+ "29": 10.58907,
+ "30": 10.34675,
+ "31": 10.07735,
+ "32": 10.46316,
+ "33": 10.45702,
+ "34": 10.19922,
+ "35": 10.25588,
"36": 10.21246,
- "37": 10.34688,
- "38": 10.18009,
+ "37": 10.34692,
+ "38": 10.18008,
"39": 10.408,
- "40": 10.07603,
- "41": 10.12932,
- "42": 10.21134,
+ "40": 10.07601,
+ "41": 10.12933,
+ "42": 10.21132,
"43": 9.81692,
- "44": 9.94028,
- "45": 9.81699,
- "46": 9.80606,
- "47": 10.12475,
- "48": 9.8405,
- "49": 9.50971,
- "50": 9.88934
+ "44": 9.94031,
+ "45": 9.81697,
+ "46": 9.80607,
+ "47": 10.12474,
+ "48": 9.84052,
+ "49": 9.50972,
+ "50": 9.88931
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1691.0,
- "2": 1553.0,
- "3": 1673.0,
- "4": 1760.0,
- "5": 1852.0,
- "6": 1861.0,
- "7": 1907.0,
- "8": 1712.0,
- "9": 1919.0,
- "10": 1427.0,
- "11": 1965.0,
- "12": 1742.0,
- "13": 1946.0,
- "14": 1903.0,
- "15": 1851.0,
- "16": 1804.0,
- "17": 1778.0,
- "18": 1702.0,
- "19": 1703.0,
- "20": 1706.0,
- "21": 1916.0,
- "22": 1698.0,
- "23": 2009.0,
- "24": 1606.0,
- "25": 1625.0,
- "26": 1722.0,
- "27": 1784.0,
- "28": 1981.0,
- "29": 1919.0,
- "30": 1948.0,
- "31": 1503.0,
- "32": 1904.0,
- "33": 2058.0,
- "34": 1737.0,
- "35": 1916.0,
- "36": 1980.0,
- "37": 2263.0,
- "38": 2121.0,
- "39": 2277.0,
- "40": 2021.0,
- "41": 2202.0,
- "42": 2340.0,
- "43": 1973.0,
- "44": 2006.0,
- "45": 2128.0,
- "46": 2132.0,
- "47": 2438.0,
- "48": 2286.0,
- "49": 2215.0,
- "50": 2337.0
+ "1": 1692.0,
+ "2": 1562.0,
+ "3": 1659.0,
+ "4": 1661.0,
+ "5": 1890.0,
+ "6": 1885.0,
+ "7": 1867.0,
+ "8": 1651.0,
+ "9": 1883.0,
+ "10": 1424.0,
+ "11": 1819.0,
+ "12": 1770.0,
+ "13": 1985.0,
+ "14": 1835.0,
+ "15": 1964.0,
+ "16": 1829.0,
+ "17": 1826.0,
+ "18": 1644.0,
+ "19": 1752.0,
+ "20": 1709.0,
+ "21": 1984.0,
+ "22": 1705.0,
+ "23": 1997.0,
+ "24": 1643.0,
+ "25": 1621.0,
+ "26": 1650.0,
+ "27": 1750.0,
+ "28": 1867.0,
+ "29": 1959.0,
+ "30": 2054.0,
+ "31": 1547.0,
+ "32": 1858.0,
+ "33": 2093.0,
+ "34": 1860.0,
+ "35": 1961.0,
+ "36": 1983.0,
+ "37": 2362.0,
+ "38": 2143.0,
+ "39": 2286.0,
+ "40": 2090.0,
+ "41": 2199.0,
+ "42": 2313.0,
+ "43": 1992.0,
+ "44": 2026.0,
+ "45": 2082.0,
+ "46": 2197.0,
+ "47": 2446.0,
+ "48": 2296.0,
+ "49": 2232.0,
+ "50": 2425.0
}
},
"mem-allocated-bytes": {
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.94258,
- "3": 0.12978,
- "4": 0.11688,
- "5": 0.11937,
- "6": 0.12093,
- "7": 0.12307,
- "8": 0.13062,
- "9": 0.12926,
- "10": 0.1228,
- "11": 0.12859,
- "12": 0.12404,
- "13": 0.12912,
- "14": 0.12318,
- "15": 0.12609,
- "16": 0.13327,
- "17": 0.12859,
- "18": 0.12957,
- "19": 0.12658,
- "20": 0.12929,
- "21": 0.12937,
- "22": 0.1298,
- "23": 0.12888,
- "24": 0.12917,
- "25": 0.1285,
- "26": 0.12864,
- "27": 0.13061,
- "28": 0.1272,
- "29": 0.12953,
- "30": 0.12693,
- "31": 0.13141,
- "32": 0.12786,
- "33": 0.12815,
- "34": 0.12937,
- "35": 0.12957,
- "36": 0.12737,
- "37": 0.1313,
- "38": 0.12977,
- "39": 0.12805,
- "40": 0.1298,
- "41": 0.1296,
- "42": 0.13074,
- "43": 0.12955,
- "44": 0.13171,
- "45": 0.13055,
- "46": 0.13271,
- "47": 0.13004,
- "48": 0.12873,
- "49": 0.13129,
- "50": 0.12858
+ "2": 9.03804,
+ "3": 0.13858,
+ "4": 0.12507,
+ "5": 0.12463,
+ "6": 0.12425,
+ "7": 0.12393,
+ "8": 0.12365,
+ "9": 0.12427,
+ "10": 0.12648,
+ "11": 0.12263,
+ "12": 0.12575,
+ "13": 0.12379,
+ "14": 0.12295,
+ "15": 0.12869,
+ "16": 0.12461,
+ "17": 0.12438,
+ "18": 0.12268,
+ "19": 0.12324,
+ "20": 0.12324,
+ "21": 0.12291,
+ "22": 0.12582,
+ "23": 0.12767,
+ "24": 0.12691,
+ "25": 0.12504,
+ "26": 0.12483,
+ "27": 0.12358,
+ "28": 0.1246,
+ "29": 0.12998,
+ "30": 0.1346,
+ "31": 0.12439,
+ "32": 0.12524,
+ "33": 0.12436,
+ "34": 0.12347,
+ "35": 0.12901,
+ "36": 0.12928,
+ "37": 0.13039,
+ "38": 0.12726,
+ "39": 0.1253,
+ "40": 0.12465,
+ "41": 0.12644,
+ "42": 0.12361,
+ "43": 0.12563,
+ "44": 0.12445,
+ "45": 0.12536,
+ "46": 0.12648,
+ "47": 0.12433,
+ "48": 0.12535,
+ "49": 0.12492,
+ "50": 0.12369
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1/golden_values_dev_dgx_h100.json
index 1c87eb73023..46ce6c8708c 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.84523,
- "2": 10.85412,
- "3": 10.85365,
- "4": 10.83867,
- "5": 10.87428,
- "6": 10.89334,
- "7": 10.8541,
- "8": 10.86232,
- "9": 10.86355,
- "10": 10.82858,
- "11": 10.88772,
- "12": 10.87148,
- "13": 10.87939,
- "14": 10.89122,
- "15": 10.81926,
- "16": 10.83064,
- "17": 10.79873,
- "18": 10.81769,
- "19": 10.8196,
- "20": 10.72749,
- "21": 10.70555,
- "22": 10.56395,
- "23": 10.7282,
- "24": 10.60841,
- "25": 10.55195,
- "26": 10.60869,
- "27": 10.62878,
- "28": 10.5827,
- "29": 10.59984,
- "30": 10.36504,
- "31": 10.12095,
- "32": 10.47626,
- "33": 10.46908,
- "34": 10.22325,
- "35": 10.27845,
- "36": 10.22879,
- "37": 10.35946,
- "38": 10.19333,
- "39": 10.41585,
- "40": 10.09773,
- "41": 10.15714,
- "42": 10.22441,
- "43": 9.8328,
- "44": 9.96934,
- "45": 9.84203,
- "46": 9.83023,
- "47": 10.15603,
- "48": 9.85506,
- "49": 9.54051,
- "50": 9.91254
+ "1": 10.96459,
+ "2": 10.95289,
+ "3": 10.95803,
+ "4": 10.95244,
+ "5": 10.95247,
+ "6": 10.95364,
+ "7": 10.95141,
+ "8": 10.95114,
+ "9": 10.9583,
+ "10": 10.94454,
+ "11": 10.93657,
+ "12": 10.94321,
+ "13": 10.94356,
+ "14": 10.94298,
+ "15": 10.91489,
+ "16": 10.90027,
+ "17": 10.91039,
+ "18": 10.9062,
+ "19": 10.90187,
+ "20": 10.80736,
+ "21": 10.79429,
+ "22": 10.80474,
+ "23": 10.78955,
+ "24": 10.77489,
+ "25": 10.76208,
+ "26": 10.75483,
+ "27": 10.71915,
+ "28": 10.63825,
+ "29": 10.6084,
+ "30": 10.5862,
+ "31": 10.59086,
+ "32": 10.57065,
+ "33": 10.53983,
+ "34": 10.49918,
+ "35": 10.49999,
+ "36": 10.48545,
+ "37": 10.44897,
+ "38": 10.45017,
+ "39": 10.41718,
+ "40": 10.39506,
+ "41": 10.37465,
+ "42": 10.3602,
+ "43": 10.33038,
+ "44": 10.3185,
+ "45": 10.31557,
+ "46": 10.27648,
+ "47": 10.26792,
+ "48": 10.21801,
+ "49": 10.21349,
+ "50": 10.22186
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1725.0,
- "2": 1664.0,
- "3": 1710.0,
- "4": 1712.0,
- "5": 1834.0,
- "6": 1743.0,
- "7": 1803.0,
- "8": 1744.0,
- "9": 1770.0,
- "10": 1478.0,
- "11": 1879.0,
- "12": 1696.0,
- "13": 1952.0,
- "14": 1732.0,
- "15": 1891.0,
- "16": 1872.0,
- "17": 1737.0,
- "18": 1744.0,
- "19": 1843.0,
- "20": 1639.0,
- "21": 1817.0,
- "22": 1615.0,
- "23": 1960.0,
- "24": 1646.0,
- "25": 1623.0,
- "26": 1671.0,
- "27": 1841.0,
- "28": 2009.0,
- "29": 1956.0,
- "30": 1882.0,
- "31": 1597.0,
- "32": 1921.0,
- "33": 2114.0,
- "34": 1828.0,
- "35": 2043.0,
- "36": 1947.0,
- "37": 2338.0,
- "38": 2227.0,
- "39": 2346.0,
- "40": 2168.0,
- "41": 2204.0,
- "42": 2247.0,
- "43": 2078.0,
- "44": 2064.0,
- "45": 2159.0,
- "46": 2489.0,
- "47": 2497.0,
- "48": 2305.0,
- "49": 2272.0,
- "50": 2319.0
+ "1": 1870.0,
+ "2": 1721.0,
+ "3": 1853.0,
+ "4": 1776.0,
+ "5": 1779.0,
+ "6": 1723.0,
+ "7": 1956.0,
+ "8": 1683.0,
+ "9": 1714.0,
+ "10": 1670.0,
+ "11": 1741.0,
+ "12": 1774.0,
+ "13": 1805.0,
+ "14": 1822.0,
+ "15": 1627.0,
+ "16": 1828.0,
+ "17": 1895.0,
+ "18": 1804.0,
+ "19": 1706.0,
+ "20": 1664.0,
+ "21": 1773.0,
+ "22": 1713.0,
+ "23": 1781.0,
+ "24": 1817.0,
+ "25": 1791.0,
+ "26": 1771.0,
+ "27": 1851.0,
+ "28": 1750.0,
+ "29": 1939.0,
+ "30": 1862.0,
+ "31": 2053.0,
+ "32": 1974.0,
+ "33": 1867.0,
+ "34": 2011.0,
+ "35": 2033.0,
+ "36": 1998.0,
+ "37": 2126.0,
+ "38": 2050.0,
+ "39": 2313.0,
+ "40": 2158.0,
+ "41": 2210.0,
+ "42": 1949.0,
+ "43": 2351.0,
+ "44": 2205.0,
+ "45": 2439.0,
+ "46": 2441.0,
+ "47": 2373.0,
+ "48": 2585.0,
+ "49": 2722.0,
+ "50": 2430.0
}
},
"mem-allocated-bytes": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 9.85525,
- "2": 0.11909,
- "3": 0.10687,
- "4": 0.08766,
- "5": 0.08696,
- "6": 0.08852,
- "7": 0.08705,
- "8": 0.0866,
- "9": 0.08968,
- "10": 0.09051,
- "11": 0.08988,
- "12": 0.08985,
- "13": 0.09145,
- "14": 0.09034,
- "15": 0.09081,
- "16": 0.09029,
- "17": 0.09013,
- "18": 0.09023,
- "19": 0.09004,
- "20": 0.09017,
- "21": 0.08987,
- "22": 0.09048,
- "23": 0.09047,
- "24": 0.08991,
- "25": 0.09343,
- "26": 0.0901,
- "27": 0.08989,
- "28": 0.09443,
- "29": 0.09097,
- "30": 0.09106,
- "31": 0.0927,
- "32": 0.08602,
- "33": 0.08691,
- "34": 0.08755,
- "35": 0.08733,
- "36": 0.08692,
- "37": 0.08659,
- "38": 0.08868,
- "39": 0.08692,
- "40": 0.08731,
- "41": 0.08817,
- "42": 0.08696,
- "43": 0.08838,
- "44": 0.08859,
- "45": 0.08767,
- "46": 0.0873,
- "47": 0.08882,
- "48": 0.08631,
- "49": 0.08619,
- "50": 0.0861
+ "1": "nan",
+ "2": 7.19155,
+ "3": 0.10374,
+ "4": 0.08959,
+ "5": 0.08955,
+ "6": 0.08875,
+ "7": 0.08894,
+ "8": 0.08859,
+ "9": 0.08929,
+ "10": 0.08861,
+ "11": 0.08912,
+ "12": 0.08929,
+ "13": 0.08875,
+ "14": 0.08897,
+ "15": 0.08846,
+ "16": 0.08936,
+ "17": 0.08864,
+ "18": 0.08883,
+ "19": 0.08895,
+ "20": 0.08882,
+ "21": 0.0889,
+ "22": 0.08874,
+ "23": 0.08875,
+ "24": 0.09104,
+ "25": 0.08799,
+ "26": 0.08831,
+ "27": 0.08867,
+ "28": 0.08892,
+ "29": 0.08845,
+ "30": 0.0902,
+ "31": 0.08948,
+ "32": 0.08881,
+ "33": 0.08658,
+ "34": 0.08761,
+ "35": 0.08847,
+ "36": 0.087,
+ "37": 0.09097,
+ "38": 0.08855,
+ "39": 0.08849,
+ "40": 0.08841,
+ "41": 0.08874,
+ "42": 0.08875,
+ "43": 0.08892,
+ "44": 0.08814,
+ "45": 0.08864,
+ "46": 0.08921,
+ "47": 0.08897,
+ "48": 0.08872,
+ "49": 0.08723,
+ "50": 0.09007
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1/model_config.yaml
index ea869246a7c..8d764f5a87a 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_calculate_per_token_loss/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_calculate_per_token_loss/model_config.yaml
index fff31764409..034339eef65 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_calculate_per_token_loss/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_calculate_per_token_loss/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_decoupled_lr/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_decoupled_lr/model_config.yaml
index 46ff13cb9a8..bf6775690cb 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_decoupled_lr/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_decoupled_lr/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce/model_config.yaml
index 5a1b1ce289d..43d9d059569 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
index 31ffc9c8111..153db6838cc 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -57,4 +57,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer/golden_values_dev_dgx_gb200.json
index 10988c85257..f14b7d93d81 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer/golden_values_dev_dgx_gb200.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.82558,
- "2": 10.83322,
- "3": 10.82737,
- "4": 10.79588,
- "5": 10.85708,
- "6": 10.86392,
- "7": 10.8269,
- "8": 10.82588,
- "9": 10.83699,
- "10": 10.79719,
- "11": 10.87851,
- "12": 10.85797,
- "13": 10.85368,
- "14": 10.87548,
- "15": 10.79177,
- "16": 10.80301,
- "17": 10.7745,
- "18": 10.80399,
- "19": 10.79365,
- "20": 10.69588,
- "21": 10.6855,
- "22": 10.53152,
- "23": 10.70658,
- "24": 10.57319,
- "25": 10.51545,
- "26": 10.59076,
- "27": 10.60738,
- "28": 10.57025,
- "29": 10.58904,
- "30": 10.34674,
- "31": 10.07736,
- "32": 10.46317,
- "33": 10.45705,
- "34": 10.19923,
- "35": 10.25593,
- "36": 10.21246,
- "37": 10.34689,
- "38": 10.18008,
- "39": 10.40796,
- "40": 10.07602,
- "41": 10.12935,
- "42": 10.21132,
- "43": 9.81692,
- "44": 9.94027,
- "45": 9.817,
- "46": 9.80608,
- "47": 10.12473,
- "48": 9.84047,
- "49": 9.50975,
- "50": 9.88932
+ "1": 10.94162,
+ "2": 10.93649,
+ "3": 10.93363,
+ "4": 10.94258,
+ "5": 10.93182,
+ "6": 10.93308,
+ "7": 10.93789,
+ "8": 10.9374,
+ "9": 10.93219,
+ "10": 10.93589,
+ "11": 10.92296,
+ "12": 10.9225,
+ "13": 10.92214,
+ "14": 10.91524,
+ "15": 10.89049,
+ "16": 10.88933,
+ "17": 10.89949,
+ "18": 10.87792,
+ "19": 10.87328,
+ "20": 10.79028,
+ "21": 10.78041,
+ "22": 10.7782,
+ "23": 10.77602,
+ "24": 10.73397,
+ "25": 10.74478,
+ "26": 10.72446,
+ "27": 10.69239,
+ "28": 10.62218,
+ "29": 10.60132,
+ "30": 10.57965,
+ "31": 10.56899,
+ "32": 10.5494,
+ "33": 10.52268,
+ "34": 10.48561,
+ "35": 10.49165,
+ "36": 10.47078,
+ "37": 10.43587,
+ "38": 10.4379,
+ "39": 10.40184,
+ "40": 10.39014,
+ "41": 10.37013,
+ "42": 10.34747,
+ "43": 10.32388,
+ "44": 10.2901,
+ "45": 10.30646,
+ "46": 10.26786,
+ "47": 10.2506,
+ "48": 10.20204,
+ "49": 10.20094,
+ "50": 10.21014
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1691.0,
- "2": 1553.0,
- "3": 1673.0,
- "4": 1760.0,
- "5": 1852.0,
- "6": 1861.0,
- "7": 1852.0,
- "8": 1755.0,
- "9": 1952.0,
- "10": 1427.0,
- "11": 1857.0,
- "12": 1820.0,
- "13": 1948.0,
- "14": 1828.0,
- "15": 1913.0,
- "16": 1881.0,
- "17": 1770.0,
- "18": 1683.0,
- "19": 1784.0,
- "20": 1714.0,
- "21": 1969.0,
- "22": 1701.0,
- "23": 1972.0,
- "24": 1545.0,
- "25": 1537.0,
- "26": 1650.0,
- "27": 1770.0,
- "28": 1889.0,
- "29": 1946.0,
- "30": 2031.0,
- "31": 1511.0,
- "32": 1848.0,
- "33": 2009.0,
- "34": 1749.0,
- "35": 1978.0,
- "36": 1926.0,
- "37": 2358.0,
- "38": 2036.0,
- "39": 2202.0,
- "40": 2015.0,
- "41": 2184.0,
- "42": 2304.0,
- "43": 2079.0,
- "44": 2042.0,
- "45": 2082.0,
- "46": 2206.0,
- "47": 2417.0,
- "48": 2284.0,
- "49": 2231.0,
- "50": 2430.0
+ "1": 1864.0,
+ "2": 1746.0,
+ "3": 1825.0,
+ "4": 1693.0,
+ "5": 1776.0,
+ "6": 1671.0,
+ "7": 2016.0,
+ "8": 1752.0,
+ "9": 1860.0,
+ "10": 1711.0,
+ "11": 1726.0,
+ "12": 1666.0,
+ "13": 1871.0,
+ "14": 1803.0,
+ "15": 1673.0,
+ "16": 1711.0,
+ "17": 1796.0,
+ "18": 1845.0,
+ "19": 1706.0,
+ "20": 1624.0,
+ "21": 1792.0,
+ "22": 1697.0,
+ "23": 1623.0,
+ "24": 1846.0,
+ "25": 1725.0,
+ "26": 1833.0,
+ "27": 1922.0,
+ "28": 1849.0,
+ "29": 2004.0,
+ "30": 1841.0,
+ "31": 2003.0,
+ "32": 1994.0,
+ "33": 1922.0,
+ "34": 1993.0,
+ "35": 2032.0,
+ "36": 1946.0,
+ "37": 2271.0,
+ "38": 2130.0,
+ "39": 2237.0,
+ "40": 2239.0,
+ "41": 2368.0,
+ "42": 1980.0,
+ "43": 2345.0,
+ "44": 2135.0,
+ "45": 2504.0,
+ "46": 2421.0,
+ "47": 2435.0,
+ "48": 2562.0,
+ "49": 2773.0,
+ "50": 2549.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 552193536.0,
- "2": 552193536.0,
- "3": 553242112.0,
- "4": 553242112.0,
- "5": 552193536.0,
- "6": 553242112.0,
- "7": 553242112.0,
- "8": 553242112.0,
- "9": 553242112.0,
- "10": 553242112.0,
- "11": 553242112.0,
- "12": 552193536.0,
- "13": 552193536.0,
- "14": 552193536.0,
- "15": 552193536.0,
- "16": 553242112.0,
- "17": 553242112.0,
- "18": 552193536.0,
- "19": 553242112.0,
- "20": 553242112.0,
- "21": 553242112.0,
- "22": 552193536.0,
- "23": 553242112.0,
- "24": 553242112.0,
- "25": 553242112.0,
- "26": 553242112.0,
- "27": 553242112.0,
- "28": 553242112.0,
- "29": 553242112.0,
- "30": 553242112.0,
- "31": 552193536.0,
- "32": 552193536.0,
- "33": 553242112.0,
- "34": 553242112.0,
- "35": 552193536.0,
- "36": 553242112.0,
- "37": 552193536.0,
- "38": 552193536.0,
- "39": 552193536.0,
- "40": 552193536.0,
- "41": 552193536.0,
- "42": 552193536.0,
- "43": 552193536.0,
- "44": 552193536.0,
- "45": 552193536.0,
- "46": 552193536.0,
- "47": 552193536.0,
- "48": 552193536.0,
- "49": 552193536.0,
- "50": 553242112.0
+ "1": 550224384.0,
+ "2": 550224384.0,
+ "3": 550224384.0,
+ "4": 550224384.0,
+ "5": 550224384.0,
+ "6": 550224384.0,
+ "7": 550224384.0,
+ "8": 550224384.0,
+ "9": 550224384.0,
+ "10": 550224384.0,
+ "11": 550224384.0,
+ "12": 550224384.0,
+ "13": 550224384.0,
+ "14": 550224384.0,
+ "15": 550224384.0,
+ "16": 550224384.0,
+ "17": 550224384.0,
+ "18": 550224384.0,
+ "19": 550224384.0,
+ "20": 550224384.0,
+ "21": 550224384.0,
+ "22": 550224384.0,
+ "23": 550224384.0,
+ "24": 550224384.0,
+ "25": 550224384.0,
+ "26": 550224384.0,
+ "27": 550224384.0,
+ "28": 550224384.0,
+ "29": 550224384.0,
+ "30": 550224384.0,
+ "31": 550224384.0,
+ "32": 550224384.0,
+ "33": 550224384.0,
+ "34": 550224384.0,
+ "35": 550224384.0,
+ "36": 550224384.0,
+ "37": 550224384.0,
+ "38": 550224384.0,
+ "39": 550224384.0,
+ "40": 550224384.0,
+ "41": 550224384.0,
+ "42": 550224384.0,
+ "43": 550224384.0,
+ "44": 550224384.0,
+ "45": 550224384.0,
+ "46": 550224384.0,
+ "47": 550224384.0,
+ "48": 550224384.0,
+ "49": 550224384.0,
+ "50": 550224384.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 3798208000.0,
- "2": 3942086144.0,
- "3": 3942086144.0,
- "4": 3942086144.0,
- "5": 3942086144.0,
- "6": 3942086144.0,
- "7": 3942086144.0,
- "8": 3942086144.0,
- "9": 3942086144.0,
- "10": 3942086144.0,
- "11": 3942086144.0,
- "12": 3942086144.0,
- "13": 3942086144.0,
- "14": 3942086144.0,
- "15": 3942086144.0,
- "16": 3942086144.0,
- "17": 3942086144.0,
- "18": 3942086144.0,
- "19": 3942086144.0,
- "20": 3942086144.0,
- "21": 3942086144.0,
- "22": 3942086144.0,
- "23": 3942086144.0,
- "24": 3942086144.0,
- "25": 3942086144.0,
- "26": 3942086144.0,
- "27": 3942086144.0,
- "28": 3942086144.0,
- "29": 3942086144.0,
- "30": 3942086144.0,
- "31": 3942086144.0,
- "32": 3942086144.0,
- "33": 3942086144.0,
- "34": 3942086144.0,
- "35": 3942086144.0,
- "36": 3942086144.0,
- "37": 3942086144.0,
- "38": 3942086144.0,
- "39": 3942086144.0,
- "40": 3942086144.0,
- "41": 3942086144.0,
- "42": 3942086144.0,
- "43": 3942086144.0,
- "44": 3942086144.0,
- "45": 3942086144.0,
- "46": 3942086144.0,
- "47": 3942086144.0,
- "48": 3942086144.0,
- "49": 3942086144.0,
- "50": 3942086144.0
+ "1": 3799125504.0,
+ "2": 3941952000.0,
+ "3": 3941952000.0,
+ "4": 3941952000.0,
+ "5": 3941952000.0,
+ "6": 3941952000.0,
+ "7": 3941952000.0,
+ "8": 3941952000.0,
+ "9": 3941952000.0,
+ "10": 3941952000.0,
+ "11": 3941952000.0,
+ "12": 3941952000.0,
+ "13": 3941952000.0,
+ "14": 3941952000.0,
+ "15": 3941952000.0,
+ "16": 3941952000.0,
+ "17": 3941952000.0,
+ "18": 3941952000.0,
+ "19": 3941952000.0,
+ "20": 3941952000.0,
+ "21": 3941952000.0,
+ "22": 3941952000.0,
+ "23": 3941952000.0,
+ "24": 3941952000.0,
+ "25": 3941952000.0,
+ "26": 3941952000.0,
+ "27": 3941952000.0,
+ "28": 3941952000.0,
+ "29": 3941952000.0,
+ "30": 3941952000.0,
+ "31": 3941952000.0,
+ "32": 3941952000.0,
+ "33": 3941952000.0,
+ "34": 3941952000.0,
+ "35": 3941952000.0,
+ "36": 3941952000.0,
+ "37": 3941952000.0,
+ "38": 3941952000.0,
+ "39": 3941952000.0,
+ "40": 3941952000.0,
+ "41": 3941952000.0,
+ "42": 3941952000.0,
+ "43": 3941952000.0,
+ "44": 3941952000.0,
+ "45": 3941952000.0,
+ "46": 3941952000.0,
+ "47": 3941952000.0,
+ "48": 3941952000.0,
+ "49": 3941952000.0,
+ "50": 3941952000.0
}
},
"iteration-time": {
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 3.84171,
- "3": 0.13294,
- "4": 0.11994,
- "5": 0.11682,
- "6": 0.11799,
- "7": 0.12021,
- "8": 0.11949,
- "9": 0.1195,
- "10": 0.12086,
- "11": 0.21563,
- "12": 0.12013,
- "13": 0.1204,
- "14": 0.1188,
- "15": 0.1192,
- "16": 0.11917,
- "17": 0.11999,
- "18": 0.12006,
- "19": 0.11965,
- "20": 0.12016,
- "21": 0.21525,
- "22": 0.11978,
- "23": 0.12009,
- "24": 0.12004,
- "25": 0.12129,
- "26": 0.12041,
- "27": 0.12075,
- "28": 0.12015,
- "29": 0.1204,
- "30": 0.12048,
- "31": 0.21709,
- "32": 0.12108,
- "33": 0.11972,
- "34": 0.12,
- "35": 0.11969,
- "36": 0.11944,
- "37": 0.11946,
- "38": 0.12056,
- "39": 0.12045,
- "40": 0.12052,
- "41": 0.21777,
- "42": 0.12063,
- "43": 0.12165,
- "44": 0.1204,
- "45": 0.12036,
- "46": 0.12154,
- "47": 0.12043,
- "48": 0.12145,
- "49": 0.12079,
- "50": 0.12035
+ "2": 9.89065,
+ "3": 0.14614,
+ "4": 0.12967,
+ "5": 0.1292,
+ "6": 0.13,
+ "7": 0.12828,
+ "8": 0.12581,
+ "9": 0.12705,
+ "10": 0.1297,
+ "11": 0.23265,
+ "12": 0.12914,
+ "13": 0.1295,
+ "14": 0.13002,
+ "15": 0.1255,
+ "16": 0.1297,
+ "17": 0.12966,
+ "18": 0.12919,
+ "19": 0.12694,
+ "20": 0.12893,
+ "21": 0.22645,
+ "22": 0.12597,
+ "23": 0.12819,
+ "24": 0.12909,
+ "25": 0.12751,
+ "26": 0.12792,
+ "27": 0.12804,
+ "28": 0.1278,
+ "29": 0.12758,
+ "30": 0.12639,
+ "31": 0.22482,
+ "32": 0.1275,
+ "33": 0.12843,
+ "34": 0.12769,
+ "35": 0.12551,
+ "36": 0.12822,
+ "37": 0.12855,
+ "38": 0.12795,
+ "39": 0.13015,
+ "40": 0.12784,
+ "41": 0.22528,
+ "42": 0.12969,
+ "43": 0.13135,
+ "44": 0.1286,
+ "45": 0.12841,
+ "46": 0.12974,
+ "47": 0.1293,
+ "48": 0.13082,
+ "49": 0.13005,
+ "50": 0.12961
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer/golden_values_dev_dgx_h100.json
index ea2f72181ea..d83ba03bbaa 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.84523,
- "2": 10.85412,
- "3": 10.85365,
- "4": 10.83867,
- "5": 10.87428,
- "6": 10.89334,
- "7": 10.8541,
- "8": 10.86235,
- "9": 10.86352,
- "10": 10.82859,
- "11": 10.88772,
- "12": 10.87148,
- "13": 10.87938,
- "14": 10.89123,
- "15": 10.81927,
- "16": 10.83063,
- "17": 10.79878,
- "18": 10.81771,
- "19": 10.81957,
- "20": 10.72749,
- "21": 10.70552,
- "22": 10.56396,
- "23": 10.72823,
- "24": 10.60839,
- "25": 10.55198,
- "26": 10.60868,
- "27": 10.62879,
- "28": 10.58271,
- "29": 10.59982,
- "30": 10.36511,
- "31": 10.12096,
- "32": 10.47628,
- "33": 10.46906,
- "34": 10.22326,
- "35": 10.27848,
- "36": 10.22883,
- "37": 10.35947,
- "38": 10.19331,
- "39": 10.41586,
- "40": 10.09773,
- "41": 10.15718,
- "42": 10.22441,
- "43": 9.83281,
- "44": 9.96935,
- "45": 9.84205,
- "46": 9.83017,
- "47": 10.15602,
- "48": 9.85503,
- "49": 9.54049,
- "50": 9.91258
+ "1": 10.96459,
+ "2": 10.95289,
+ "3": 10.95803,
+ "4": 10.95244,
+ "5": 10.95247,
+ "6": 10.9536,
+ "7": 10.95135,
+ "8": 10.95111,
+ "9": 10.95831,
+ "10": 10.94451,
+ "11": 10.93657,
+ "12": 10.94324,
+ "13": 10.94351,
+ "14": 10.943,
+ "15": 10.91487,
+ "16": 10.90029,
+ "17": 10.91041,
+ "18": 10.90618,
+ "19": 10.90184,
+ "20": 10.80738,
+ "21": 10.79428,
+ "22": 10.80474,
+ "23": 10.78949,
+ "24": 10.77488,
+ "25": 10.76205,
+ "26": 10.7548,
+ "27": 10.71915,
+ "28": 10.63825,
+ "29": 10.6084,
+ "30": 10.58625,
+ "31": 10.59088,
+ "32": 10.57067,
+ "33": 10.53986,
+ "34": 10.49921,
+ "35": 10.49999,
+ "36": 10.48548,
+ "37": 10.44895,
+ "38": 10.45018,
+ "39": 10.4172,
+ "40": 10.39509,
+ "41": 10.37462,
+ "42": 10.36019,
+ "43": 10.33041,
+ "44": 10.31847,
+ "45": 10.31554,
+ "46": 10.27645,
+ "47": 10.26792,
+ "48": 10.21802,
+ "49": 10.21351,
+ "50": 10.22185
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1725.0,
- "2": 1664.0,
- "3": 1710.0,
- "4": 1712.0,
- "5": 1834.0,
- "6": 1743.0,
- "7": 1803.0,
- "8": 1737.0,
- "9": 1779.0,
- "10": 1459.0,
- "11": 1898.0,
- "12": 1661.0,
- "13": 1860.0,
- "14": 1764.0,
- "15": 1886.0,
- "16": 1916.0,
- "17": 1773.0,
- "18": 1702.0,
- "19": 1742.0,
- "20": 1649.0,
- "21": 1899.0,
- "22": 1631.0,
- "23": 1960.0,
- "24": 1570.0,
- "25": 1647.0,
- "26": 1649.0,
- "27": 1811.0,
- "28": 1930.0,
- "29": 1910.0,
- "30": 1964.0,
- "31": 1536.0,
- "32": 1873.0,
- "33": 2191.0,
- "34": 1838.0,
- "35": 2017.0,
- "36": 1916.0,
- "37": 2345.0,
- "38": 2247.0,
- "39": 2374.0,
- "40": 2207.0,
- "41": 2246.0,
- "42": 2291.0,
- "43": 2027.0,
- "44": 2147.0,
- "45": 2164.0,
- "46": 2300.0,
- "47": 2418.0,
- "48": 2467.0,
- "49": 2255.0,
- "50": 2224.0
+ "1": 1870.0,
+ "2": 1721.0,
+ "3": 1853.0,
+ "4": 1776.0,
+ "5": 1840.0,
+ "6": 1726.0,
+ "7": 1933.0,
+ "8": 1747.0,
+ "9": 1726.0,
+ "10": 1779.0,
+ "11": 1670.0,
+ "12": 1708.0,
+ "13": 1870.0,
+ "14": 1846.0,
+ "15": 1641.0,
+ "16": 1776.0,
+ "17": 1826.0,
+ "18": 1827.0,
+ "19": 1686.0,
+ "20": 1737.0,
+ "21": 1783.0,
+ "22": 1737.0,
+ "23": 1644.0,
+ "24": 1851.0,
+ "25": 1706.0,
+ "26": 1758.0,
+ "27": 1862.0,
+ "28": 1756.0,
+ "29": 1868.0,
+ "30": 1847.0,
+ "31": 1971.0,
+ "32": 2035.0,
+ "33": 1955.0,
+ "34": 2005.0,
+ "35": 1989.0,
+ "36": 1975.0,
+ "37": 2130.0,
+ "38": 2095.0,
+ "39": 2136.0,
+ "40": 2279.0,
+ "41": 2361.0,
+ "42": 1916.0,
+ "43": 2368.0,
+ "44": 2058.0,
+ "45": 2461.0,
+ "46": 2314.0,
+ "47": 2303.0,
+ "48": 2632.0,
+ "49": 2796.0,
+ "50": 2516.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 553245184.0,
- "2": 553245184.0,
- "3": 553245184.0,
- "4": 553245184.0,
- "5": 553245184.0,
- "6": 553245184.0,
- "7": 553245184.0,
- "8": 553245184.0,
- "9": 553245184.0,
- "10": 553245184.0,
- "11": 553245184.0,
- "12": 553245184.0,
- "13": 553245184.0,
- "14": 553245184.0,
- "15": 553245184.0,
- "16": 553245184.0,
- "17": 553245184.0,
- "18": 553245184.0,
- "19": 553245184.0,
- "20": 553245184.0,
- "21": 553245184.0,
- "22": 553245184.0,
- "23": 553245184.0,
- "24": 553245184.0,
- "25": 553245184.0,
- "26": 553245184.0,
- "27": 553245184.0,
- "28": 553245184.0,
- "29": 553245184.0,
- "30": 553245184.0,
- "31": 553245184.0,
- "32": 553245184.0,
- "33": 553245184.0,
- "34": 553245184.0,
- "35": 553245184.0,
- "36": 553245184.0,
- "37": 553245184.0,
- "38": 553245184.0,
- "39": 553245184.0,
- "40": 553245184.0,
- "41": 553245184.0,
- "42": 553245184.0,
- "43": 553245184.0,
- "44": 553245184.0,
- "45": 553245184.0,
- "46": 553245184.0,
- "47": 553245184.0,
- "48": 553245184.0,
- "49": 553245184.0,
- "50": 553245184.0
+ "1": 552730112.0,
+ "2": 552730112.0,
+ "3": 552730112.0,
+ "4": 552730112.0,
+ "5": 552730112.0,
+ "6": 552730112.0,
+ "7": 552730112.0,
+ "8": 552730112.0,
+ "9": 552730112.0,
+ "10": 552730112.0,
+ "11": 552730112.0,
+ "12": 552730112.0,
+ "13": 552730112.0,
+ "14": 552730112.0,
+ "15": 552730112.0,
+ "16": 552730112.0,
+ "17": 552730112.0,
+ "18": 552730112.0,
+ "19": 552730112.0,
+ "20": 552730112.0,
+ "21": 552730112.0,
+ "22": 552730112.0,
+ "23": 552730112.0,
+ "24": 552730112.0,
+ "25": 552730112.0,
+ "26": 552730112.0,
+ "27": 552730112.0,
+ "28": 552730112.0,
+ "29": 552730112.0,
+ "30": 552730112.0,
+ "31": 552730112.0,
+ "32": 552730112.0,
+ "33": 552730112.0,
+ "34": 552730112.0,
+ "35": 552730112.0,
+ "36": 552730112.0,
+ "37": 552730112.0,
+ "38": 552730112.0,
+ "39": 552730112.0,
+ "40": 552730112.0,
+ "41": 552730112.0,
+ "42": 552730112.0,
+ "43": 552730112.0,
+ "44": 552730112.0,
+ "45": 552730112.0,
+ "46": 552730112.0,
+ "47": 552730112.0,
+ "48": 552730112.0,
+ "49": 552730112.0,
+ "50": 552730112.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 3798208000.0,
- "2": 3943137792.0,
- "3": 3943137792.0,
- "4": 3943137792.0,
- "5": 3943137792.0,
- "6": 3943137792.0,
- "7": 3943137792.0,
- "8": 3943137792.0,
- "9": 3943137792.0,
- "10": 3943137792.0,
- "11": 3943137792.0,
- "12": 3943137792.0,
- "13": 3943137792.0,
- "14": 3943137792.0,
- "15": 3943137792.0,
- "16": 3943137792.0,
- "17": 3943137792.0,
- "18": 3943137792.0,
- "19": 3943137792.0,
- "20": 3943137792.0,
- "21": 3943137792.0,
- "22": 3943137792.0,
- "23": 3943137792.0,
- "24": 3943137792.0,
- "25": 3943137792.0,
- "26": 3943137792.0,
- "27": 3943137792.0,
- "28": 3943137792.0,
- "29": 3943137792.0,
- "30": 3943137792.0,
- "31": 3943137792.0,
- "32": 3943137792.0,
- "33": 3943137792.0,
- "34": 3943137792.0,
- "35": 3943137792.0,
- "36": 3943137792.0,
- "37": 3943137792.0,
- "38": 3943137792.0,
- "39": 3943137792.0,
- "40": 3943137792.0,
- "41": 3943137792.0,
- "42": 3943137792.0,
- "43": 3943137792.0,
- "44": 3943137792.0,
- "45": 3943137792.0,
- "46": 3943137792.0,
- "47": 3943137792.0,
- "48": 3943137792.0,
- "49": 3943137792.0,
- "50": 3943137792.0
+ "1": 3799125504.0,
+ "2": 3942493696.0,
+ "3": 3942493696.0,
+ "4": 3942493696.0,
+ "5": 3942493696.0,
+ "6": 3942493696.0,
+ "7": 3942493696.0,
+ "8": 3942493696.0,
+ "9": 3942493696.0,
+ "10": 3942493696.0,
+ "11": 3942493696.0,
+ "12": 3942493696.0,
+ "13": 3942493696.0,
+ "14": 3942493696.0,
+ "15": 3942493696.0,
+ "16": 3942493696.0,
+ "17": 3942493696.0,
+ "18": 3942493696.0,
+ "19": 3942493696.0,
+ "20": 3942493696.0,
+ "21": 3942493696.0,
+ "22": 3942493696.0,
+ "23": 3942493696.0,
+ "24": 3942493696.0,
+ "25": 3942493696.0,
+ "26": 3942493696.0,
+ "27": 3942493696.0,
+ "28": 3942493696.0,
+ "29": 3942493696.0,
+ "30": 3942493696.0,
+ "31": 3942493696.0,
+ "32": 3942493696.0,
+ "33": 3942493696.0,
+ "34": 3942493696.0,
+ "35": 3942493696.0,
+ "36": 3942493696.0,
+ "37": 3942493696.0,
+ "38": 3942493696.0,
+ "39": 3942493696.0,
+ "40": 3942493696.0,
+ "41": 3942493696.0,
+ "42": 3942493696.0,
+ "43": 3942493696.0,
+ "44": 3942493696.0,
+ "45": 3942493696.0,
+ "46": 3942493696.0,
+ "47": 3942493696.0,
+ "48": 3942493696.0,
+ "49": 3942493696.0,
+ "50": 3942493696.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 13.97343,
- "2": 0.13214,
- "3": 0.11635,
- "4": 0.09459,
- "5": 0.0948,
- "6": 0.09321,
- "7": 0.09394,
- "8": 0.09525,
- "9": 0.09364,
- "10": 0.09321,
- "11": 0.22069,
- "12": 0.09263,
- "13": 0.09317,
- "14": 0.09315,
- "15": 0.09254,
- "16": 0.09554,
- "17": 0.09332,
- "18": 0.09352,
- "19": 0.09438,
- "20": 0.09298,
- "21": 0.22042,
- "22": 0.09282,
- "23": 0.09311,
- "24": 0.09404,
- "25": 0.09412,
- "26": 0.09311,
- "27": 0.09293,
- "28": 0.09243,
- "29": 0.09294,
- "30": 0.09541,
- "31": 0.22042,
- "32": 0.09422,
- "33": 0.09281,
- "34": 0.09264,
- "35": 0.09337,
- "36": 0.09247,
- "37": 0.09252,
- "38": 0.09352,
- "39": 0.09297,
- "40": 0.09265,
- "41": 0.22109,
- "42": 0.09577,
- "43": 0.09321,
- "44": 0.0937,
- "45": 0.09442,
- "46": 0.09283,
- "47": 0.09255,
- "48": 0.09325,
- "49": 0.09296,
- "50": 0.09323
+ "1": "nan",
+ "2": 9.24178,
+ "3": 0.12175,
+ "4": 0.09756,
+ "5": 0.09406,
+ "6": 0.09423,
+ "7": 0.09468,
+ "8": 0.09439,
+ "9": 0.09343,
+ "10": 0.09369,
+ "11": 0.21976,
+ "12": 0.09493,
+ "13": 0.09316,
+ "14": 0.09344,
+ "15": 0.08989,
+ "16": 0.09429,
+ "17": 0.09635,
+ "18": 0.0974,
+ "19": 0.09706,
+ "20": 0.09732,
+ "21": 0.22264,
+ "22": 0.09674,
+ "23": 0.09714,
+ "24": 0.09791,
+ "25": 0.09698,
+ "26": 0.09659,
+ "27": 0.0978,
+ "28": 0.09711,
+ "29": 0.09716,
+ "30": 0.09728,
+ "31": 0.22172,
+ "32": 0.09721,
+ "33": 0.09715,
+ "34": 0.09741,
+ "35": 0.0977,
+ "36": 0.09691,
+ "37": 0.09679,
+ "38": 0.09799,
+ "39": 0.09649,
+ "40": 0.09741,
+ "41": 0.22239,
+ "42": 0.09689,
+ "43": 0.09683,
+ "44": 0.09708,
+ "45": 0.09718,
+ "46": 0.09659,
+ "47": 0.0978,
+ "48": 0.09649,
+ "49": 0.09556,
+ "50": 0.09693
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer/model_config.yaml
index 0bd25e79735..28248046b44 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_untied/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_untied/model_config.yaml
index f6a72754edb..7c6c7ff6f9f 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_untied/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_untied/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -53,4 +53,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr/golden_values_dev_dgx_gb200.json
index a34edb3389a..4ce33e9f3b9 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr/golden_values_dev_dgx_gb200.json
@@ -7,103 +7,103 @@
"1": 10.82555,
"2": 10.83286,
"3": 10.82762,
- "4": 10.79573,
- "5": 10.85695,
- "6": 10.86391,
- "7": 10.82616,
- "8": 10.82544,
- "9": 10.83584,
- "10": 10.79629,
- "11": 10.8782,
- "12": 10.85821,
+ "4": 10.7957,
+ "5": 10.85697,
+ "6": 10.86388,
+ "7": 10.82617,
+ "8": 10.82543,
+ "9": 10.83586,
+ "10": 10.7963,
+ "11": 10.87822,
+ "12": 10.85823,
"13": 10.85418,
- "14": 10.87518,
- "15": 10.79205,
+ "14": 10.87517,
+ "15": 10.79204,
"16": 10.80305,
- "17": 10.77428,
- "18": 10.8046,
- "19": 10.79338,
- "20": 10.69563,
- "21": 10.68645,
- "22": 10.53149,
- "23": 10.70629,
- "24": 10.57273,
- "25": 10.5144,
- "26": 10.58993,
- "27": 10.60707,
+ "17": 10.77433,
+ "18": 10.80462,
+ "19": 10.79337,
+ "20": 10.69556,
+ "21": 10.68641,
+ "22": 10.53147,
+ "23": 10.70631,
+ "24": 10.57272,
+ "25": 10.51439,
+ "26": 10.58989,
+ "27": 10.60708,
"28": 10.57003,
- "29": 10.58929,
- "30": 10.34675,
- "31": 10.07709,
- "32": 10.46194,
- "33": 10.45484,
- "34": 10.19662,
- "35": 10.25291,
+ "29": 10.5893,
+ "30": 10.34669,
+ "31": 10.07712,
+ "32": 10.46192,
+ "33": 10.4548,
+ "34": 10.1966,
+ "35": 10.2529,
"36": 10.20971,
- "37": 10.34492,
- "38": 10.17789,
- "39": 10.4061,
- "40": 10.07414,
- "41": 10.12736,
- "42": 10.20823,
- "43": 9.81194,
- "44": 9.93354,
+ "37": 10.34489,
+ "38": 10.1779,
+ "39": 10.40615,
+ "40": 10.07413,
+ "41": 10.12733,
+ "42": 10.2082,
+ "43": 9.81191,
+ "44": 9.93355,
"45": 9.80953,
- "46": 9.79773,
- "47": 10.11569,
- "48": 9.83234,
- "49": 9.50281,
- "50": 9.88181,
- "51": 9.83458,
- "52": 9.71756,
- "53": 10.05126,
- "54": 9.94371,
- "55": 9.87457,
- "56": 9.6029,
+ "46": 9.79775,
+ "47": 10.11572,
+ "48": 9.83237,
+ "49": 9.50279,
+ "50": 9.8818,
+ "51": 9.8346,
+ "52": 9.71755,
+ "53": 10.05121,
+ "54": 9.94375,
+ "55": 9.87452,
+ "56": 9.60291,
"57": 9.45086,
- "58": 9.811,
+ "58": 9.81098,
"59": 9.56395,
- "60": 9.47155,
- "61": 9.66553,
- "62": 9.96353,
- "63": 9.34709,
- "64": 9.743,
- "65": 8.92136,
- "66": 9.67858,
- "67": 9.35222,
- "68": 9.76563,
- "69": 9.7774,
- "70": 9.70407,
- "71": 9.60099,
- "72": 9.5498,
+ "60": 9.47154,
+ "61": 9.66555,
+ "62": 9.96351,
+ "63": 9.34708,
+ "64": 9.74296,
+ "65": 8.92132,
+ "66": 9.67854,
+ "67": 9.3522,
+ "68": 9.76559,
+ "69": 9.77742,
+ "70": 9.70406,
+ "71": 9.601,
+ "72": 9.54984,
"73": 9.46046,
- "74": 8.89068,
- "75": 9.3874,
- "76": 9.04469,
- "77": 10.03647,
- "78": 9.6996,
- "79": 9.34722,
+ "74": 8.89067,
+ "75": 9.38738,
+ "76": 9.04468,
+ "77": 10.03651,
+ "78": 9.69957,
+ "79": 9.34723,
"80": 9.37822,
- "81": 9.45421,
- "82": 9.67529,
- "83": 9.28446,
- "84": 9.39113,
- "85": 9.58663,
- "86": 9.04694,
- "87": 9.56972,
- "88": 9.72085,
- "89": 9.5673,
+ "81": 9.4542,
+ "82": 9.67532,
+ "83": 9.28445,
+ "84": 9.39112,
+ "85": 9.58662,
+ "86": 9.04692,
+ "87": 9.5697,
+ "88": 9.72082,
+ "89": 9.56729,
"90": 9.79474,
- "91": 9.30448,
- "92": 9.32183,
- "93": 9.0517,
- "94": 8.79005,
- "95": 9.4918,
- "96": 9.48711,
- "97": 9.26589,
- "98": 9.62592,
+ "91": 9.30452,
+ "92": 9.32188,
+ "93": 9.05169,
+ "94": 8.79001,
+ "95": 9.49179,
+ "96": 9.48712,
+ "97": 9.2659,
+ "98": 9.62594,
"99": 8.85252,
- "100": 9.35907
+ "100": 9.35905
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1651.0,
- "2": 1716.0,
- "3": 1772.0,
- "4": 1774.0,
- "5": 1920.0,
- "6": 1864.0,
- "7": 1830.0,
- "8": 1695.0,
- "9": 1858.0,
- "10": 1367.0,
- "11": 1915.0,
- "12": 1797.0,
- "13": 1899.0,
- "14": 1769.0,
- "15": 1880.0,
- "16": 1806.0,
- "17": 1822.0,
- "18": 1686.0,
- "19": 1728.0,
- "20": 1667.0,
- "21": 1897.0,
- "22": 1703.0,
- "23": 1967.0,
- "24": 1595.0,
- "25": 1583.0,
- "26": 1684.0,
- "27": 1911.0,
- "28": 1969.0,
- "29": 1864.0,
- "30": 1943.0,
- "31": 1535.0,
- "32": 1895.0,
- "33": 2078.0,
- "34": 1739.0,
- "35": 1940.0,
- "36": 1919.0,
- "37": 2460.0,
- "38": 2107.0,
- "39": 2261.0,
- "40": 2059.0,
- "41": 2183.0,
- "42": 2269.0,
- "43": 1972.0,
- "44": 2040.0,
- "45": 2093.0,
- "46": 2140.0,
- "47": 2476.0,
- "48": 2311.0,
- "49": 2165.0,
- "50": 2411.0,
- "51": 2471.0,
- "52": 2670.0,
- "53": 2883.0,
- "54": 2589.0,
- "55": 2427.0,
- "56": 2774.0,
- "57": 2246.0,
- "58": 2994.0,
- "59": 2922.0,
- "60": 2416.0,
- "61": 2960.0,
- "62": 2646.0,
- "63": 2488.0,
- "64": 2956.0,
- "65": 2746.0,
- "66": 2864.0,
- "67": 2794.0,
- "68": 2703.0,
- "69": 2990.0,
- "70": 3012.0,
- "71": 2884.0,
- "72": 2536.0,
- "73": 3054.0,
- "74": 2100.0,
- "75": 2573.0,
- "76": 3076.0,
- "77": 3025.0,
- "78": 3014.0,
- "79": 3083.0,
- "80": 2989.0,
- "81": 3452.0,
- "82": 3253.0,
- "83": 2759.0,
- "84": 3186.0,
- "85": 3247.0,
- "86": 2624.0,
- "87": 3594.0,
- "88": 3009.0,
- "89": 3286.0,
- "90": 3354.0,
- "91": 2869.0,
- "92": 3156.0,
- "93": 2809.0,
- "94": 3350.0,
- "95": 3033.0,
- "96": 3323.0,
- "97": 3091.0,
- "98": 3356.0,
- "99": 3326.0,
- "100": 3144.0
+ "1": 1622.0,
+ "2": 1729.0,
+ "3": 1639.0,
+ "4": 1678.0,
+ "5": 1914.0,
+ "6": 1900.0,
+ "7": 1843.0,
+ "8": 1671.0,
+ "9": 1818.0,
+ "10": 1434.0,
+ "11": 1893.0,
+ "12": 1772.0,
+ "13": 1950.0,
+ "14": 1863.0,
+ "15": 1885.0,
+ "16": 1738.0,
+ "17": 1743.0,
+ "18": 1679.0,
+ "19": 1702.0,
+ "20": 1729.0,
+ "21": 1914.0,
+ "22": 1696.0,
+ "23": 1958.0,
+ "24": 1574.0,
+ "25": 1531.0,
+ "26": 1707.0,
+ "27": 1804.0,
+ "28": 1939.0,
+ "29": 1973.0,
+ "30": 2024.0,
+ "31": 1494.0,
+ "32": 1960.0,
+ "33": 1971.0,
+ "34": 1813.0,
+ "35": 1950.0,
+ "36": 2051.0,
+ "37": 2382.0,
+ "38": 2098.0,
+ "39": 2262.0,
+ "40": 2137.0,
+ "41": 2191.0,
+ "42": 2258.0,
+ "43": 2023.0,
+ "44": 2104.0,
+ "45": 2062.0,
+ "46": 2219.0,
+ "47": 2490.0,
+ "48": 2393.0,
+ "49": 2210.0,
+ "50": 2478.0,
+ "51": 2565.0,
+ "52": 2533.0,
+ "53": 2848.0,
+ "54": 2623.0,
+ "55": 2487.0,
+ "56": 2760.0,
+ "57": 2384.0,
+ "58": 2929.0,
+ "59": 2814.0,
+ "60": 2418.0,
+ "61": 2943.0,
+ "62": 2620.0,
+ "63": 2470.0,
+ "64": 2875.0,
+ "65": 2652.0,
+ "66": 3070.0,
+ "67": 2805.0,
+ "68": 2612.0,
+ "69": 3021.0,
+ "70": 3054.0,
+ "71": 2976.0,
+ "72": 2577.0,
+ "73": 2989.0,
+ "74": 2019.0,
+ "75": 2649.0,
+ "76": 3096.0,
+ "77": 3057.0,
+ "78": 2960.0,
+ "79": 3045.0,
+ "80": 3030.0,
+ "81": 3357.0,
+ "82": 3310.0,
+ "83": 2751.0,
+ "84": 3176.0,
+ "85": 3342.0,
+ "86": 2659.0,
+ "87": 3504.0,
+ "88": 3082.0,
+ "89": 3296.0,
+ "90": 3304.0,
+ "91": 2910.0,
+ "92": 3195.0,
+ "93": 2812.0,
+ "94": 3284.0,
+ "95": 3090.0,
+ "96": 3330.0,
+ "97": 3095.0,
+ "98": 3486.0,
+ "99": 3148.0,
+ "100": 3169.0
}
},
"mem-allocated-bytes": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.73603,
- "3": 0.12344,
- "4": 0.10783,
- "5": 0.10595,
- "6": 0.10649,
- "7": 0.10691,
- "8": 0.10679,
- "9": 0.10607,
- "10": 0.10675,
- "11": 0.10687,
- "12": 0.10636,
- "13": 0.10663,
- "14": 0.10668,
- "15": 0.10696,
- "16": 0.10672,
- "17": 0.10678,
- "18": 0.10603,
- "19": 0.10659,
- "20": 0.10684,
- "21": 0.10766,
- "22": 0.10849,
- "23": 0.10853,
- "24": 0.10805,
- "25": 0.10776,
- "26": 0.1069,
- "27": 0.10818,
- "28": 0.10669,
- "29": 0.10643,
- "30": 0.10634,
- "31": 0.10766,
- "32": 0.1076,
- "33": 0.10583,
- "34": 0.10631,
- "35": 0.10587,
- "36": 0.1054,
- "37": 0.10589,
- "38": 0.10633,
- "39": 0.10593,
- "40": 0.10674,
- "41": 0.10812,
- "42": 0.11127,
- "43": 0.11494,
- "44": 0.11409,
- "45": 0.11538,
- "46": 0.11702,
- "47": 0.1155,
- "48": 0.11481,
- "49": 0.11507,
- "50": 0.11401,
- "51": 0.11655,
- "52": 0.11513,
- "53": 0.11379,
- "54": 0.11378,
- "55": 0.11658,
- "56": 0.11792,
- "57": 0.11792,
- "58": 0.11715,
- "59": 0.11915,
- "60": 0.11642,
- "61": 0.11578,
- "62": 0.1171,
- "63": 0.11758,
- "64": 0.11517,
- "65": 0.11624,
- "66": 0.11434,
- "67": 0.11609,
- "68": 0.11506,
- "69": 0.11568,
- "70": 0.11661,
- "71": 0.11647,
- "72": 0.1166,
- "73": 0.11795,
- "74": 0.11661,
- "75": 0.11785,
- "76": 0.11659,
- "77": 0.11531,
- "78": 0.11705,
- "79": 0.11662,
- "80": 0.11765,
- "81": 0.11829,
- "82": 0.11742,
- "83": 0.11529,
- "84": 0.11678,
- "85": 0.11581,
- "86": 0.11703,
- "87": 0.11699,
- "88": 0.11641,
- "89": 0.11638,
- "90": 0.11586,
- "91": 0.11853,
- "92": 0.11725,
- "93": 0.1178,
- "94": 0.11647,
- "95": 0.11672,
- "96": 0.11702,
- "97": 0.11754,
- "98": 0.11614,
- "99": 0.11757,
- "100": 0.11708
+ "2": 8.98992,
+ "3": 0.1294,
+ "4": 0.11595,
+ "5": 0.11241,
+ "6": 0.11556,
+ "7": 0.11737,
+ "8": 0.11364,
+ "9": 0.11507,
+ "10": 0.117,
+ "11": 0.11605,
+ "12": 0.11662,
+ "13": 0.11519,
+ "14": 0.11521,
+ "15": 0.11573,
+ "16": 0.11538,
+ "17": 0.11465,
+ "18": 0.1166,
+ "19": 0.11515,
+ "20": 0.11437,
+ "21": 0.11558,
+ "22": 0.11522,
+ "23": 0.1153,
+ "24": 0.11438,
+ "25": 0.11808,
+ "26": 0.11687,
+ "27": 0.11525,
+ "28": 0.11383,
+ "29": 0.11673,
+ "30": 0.11524,
+ "31": 0.1166,
+ "32": 0.11702,
+ "33": 0.11405,
+ "34": 0.11492,
+ "35": 0.11579,
+ "36": 0.11598,
+ "37": 0.11831,
+ "38": 0.117,
+ "39": 0.11673,
+ "40": 0.11174,
+ "41": 0.11645,
+ "42": 0.11308,
+ "43": 0.11563,
+ "44": 0.11397,
+ "45": 0.11626,
+ "46": 0.11355,
+ "47": 0.11499,
+ "48": 0.11524,
+ "49": 0.11557,
+ "50": 0.11265,
+ "51": 0.11887,
+ "52": 0.11543,
+ "53": 0.1134,
+ "54": 0.11629,
+ "55": 0.11697,
+ "56": 0.11712,
+ "57": 0.11885,
+ "58": 0.11734,
+ "59": 0.11534,
+ "60": 0.11888,
+ "61": 0.11756,
+ "62": 0.11757,
+ "63": 0.11525,
+ "64": 0.11676,
+ "65": 0.1176,
+ "66": 0.11477,
+ "67": 0.11557,
+ "68": 0.116,
+ "69": 0.11786,
+ "70": 0.11593,
+ "71": 0.11704,
+ "72": 0.11671,
+ "73": 0.11593,
+ "74": 0.11743,
+ "75": 0.11579,
+ "76": 0.11805,
+ "77": 0.11433,
+ "78": 0.11717,
+ "79": 0.11771,
+ "80": 0.11637,
+ "81": 0.11676,
+ "82": 0.11803,
+ "83": 0.11703,
+ "84": 0.11777,
+ "85": 0.11644,
+ "86": 0.11704,
+ "87": 0.11621,
+ "88": 0.11725,
+ "89": 0.11643,
+ "90": 0.1164,
+ "91": 0.11606,
+ "92": 0.1162,
+ "93": 0.11808,
+ "94": 0.11939,
+ "95": 0.11748,
+ "96": 0.11697,
+ "97": 0.11629,
+ "98": 0.11519,
+ "99": 0.11719,
+ "100": 0.11973
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr/golden_values_dev_dgx_h100.json
index 2f0a7e29034..b7fda00303c 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.84517,
- "2": 10.85349,
- "3": 10.8539,
- "4": 10.83826,
- "5": 10.87422,
- "6": 10.89306,
- "7": 10.85452,
- "8": 10.8626,
- "9": 10.86463,
- "10": 10.82907,
- "11": 10.88787,
- "12": 10.87098,
- "13": 10.87914,
- "14": 10.89069,
- "15": 10.81973,
- "16": 10.83156,
- "17": 10.79863,
- "18": 10.8165,
- "19": 10.81889,
- "20": 10.72685,
- "21": 10.7058,
- "22": 10.5635,
- "23": 10.7279,
- "24": 10.6076,
- "25": 10.55128,
- "26": 10.60747,
- "27": 10.62771,
- "28": 10.5826,
- "29": 10.59962,
- "30": 10.36565,
- "31": 10.1199,
- "32": 10.47544,
- "33": 10.46636,
- "34": 10.22008,
- "35": 10.27436,
- "36": 10.2259,
- "37": 10.3573,
- "38": 10.19161,
- "39": 10.41342,
- "40": 10.09564,
- "41": 10.15513,
- "42": 10.22085,
- "43": 9.82792,
- "44": 9.96282,
- "45": 9.83422,
- "46": 9.8221,
- "47": 10.14764,
- "48": 9.84684,
- "49": 9.53373,
- "50": 9.90531,
- "51": 9.85118,
- "52": 9.73512,
- "53": 10.05864,
- "54": 9.94367,
- "55": 9.87297,
- "56": 9.61699,
- "57": 9.46751,
- "58": 9.82221,
- "59": 9.57334,
- "60": 9.48862,
- "61": 9.67922,
- "62": 9.97512,
- "63": 9.37044,
- "64": 9.76642,
- "65": 8.9343,
- "66": 9.69461,
- "67": 9.35362,
- "68": 9.76826,
- "69": 9.77678,
- "70": 9.72363,
- "71": 9.59894,
- "72": 9.56455,
- "73": 9.48329,
- "74": 8.92064,
- "75": 9.40392,
- "76": 9.05297,
- "77": 10.04178,
- "78": 9.69879,
- "79": 9.35126,
- "80": 9.38212,
- "81": 9.45864,
- "82": 9.67516,
- "83": 9.2841,
- "84": 9.39311,
- "85": 9.58936,
- "86": 9.05178,
- "87": 9.56418,
- "88": 9.71755,
- "89": 9.57129,
- "90": 9.78202,
- "91": 9.30611,
- "92": 9.32046,
- "93": 9.03939,
- "94": 8.7952,
- "95": 9.47908,
- "96": 9.48453,
- "97": 9.26989,
- "98": 9.62564,
- "99": 8.84254,
- "100": 9.3498
+ "1": 10.96464,
+ "2": 10.95235,
+ "3": 10.95856,
+ "4": 10.95193,
+ "5": 10.95264,
+ "6": 10.95378,
+ "7": 10.95159,
+ "8": 10.95135,
+ "9": 10.95868,
+ "10": 10.94451,
+ "11": 10.93685,
+ "12": 10.94337,
+ "13": 10.9431,
+ "14": 10.94283,
+ "15": 10.91492,
+ "16": 10.90026,
+ "17": 10.91106,
+ "18": 10.90591,
+ "19": 10.90126,
+ "20": 10.80774,
+ "21": 10.79441,
+ "22": 10.80464,
+ "23": 10.78957,
+ "24": 10.77418,
+ "25": 10.76058,
+ "26": 10.75392,
+ "27": 10.71868,
+ "28": 10.6386,
+ "29": 10.60796,
+ "30": 10.58596,
+ "31": 10.59039,
+ "32": 10.56925,
+ "33": 10.53722,
+ "34": 10.49523,
+ "35": 10.49634,
+ "36": 10.48326,
+ "37": 10.44676,
+ "38": 10.44823,
+ "39": 10.41532,
+ "40": 10.39307,
+ "41": 10.37226,
+ "42": 10.35598,
+ "43": 10.32383,
+ "44": 10.31085,
+ "45": 10.30675,
+ "46": 10.26596,
+ "47": 10.25821,
+ "48": 10.20969,
+ "49": 10.20573,
+ "50": 10.21349,
+ "51": 10.21324,
+ "52": 10.15841,
+ "53": 10.16271,
+ "54": 10.12808,
+ "55": 10.09863,
+ "56": 10.12107,
+ "57": 10.10886,
+ "58": 10.11351,
+ "59": 10.05656,
+ "60": 10.07839,
+ "61": 10.03376,
+ "62": 10.0011,
+ "63": 10.06758,
+ "64": 10.01685,
+ "65": 9.98862,
+ "66": 10.02937,
+ "67": 9.99986,
+ "68": 9.96571,
+ "69": 9.97411,
+ "70": 9.95973,
+ "71": 9.98127,
+ "72": 9.9666,
+ "73": 9.94914,
+ "74": 9.94057,
+ "75": 9.91021,
+ "76": 9.93991,
+ "77": 9.93686,
+ "78": 9.88869,
+ "79": 9.88963,
+ "80": 9.90385,
+ "81": 9.92955,
+ "82": 9.86789,
+ "83": 9.83326,
+ "84": 9.7745,
+ "85": 9.75501,
+ "86": 9.85588,
+ "87": 9.87807,
+ "88": 9.84949,
+ "89": 9.7871,
+ "90": 9.7852,
+ "91": 9.78365,
+ "92": 9.77827,
+ "93": 9.70912,
+ "94": 9.78144,
+ "95": 9.76788,
+ "96": 9.75263,
+ "97": 9.70031,
+ "98": 9.71822,
+ "99": 9.77351,
+ "100": 9.6534
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1655.0,
- "2": 1697.0,
- "3": 1773.0,
- "4": 1782.0,
- "5": 1897.0,
- "6": 1802.0,
- "7": 1874.0,
- "8": 1653.0,
- "9": 1814.0,
- "10": 1441.0,
- "11": 1909.0,
- "12": 1645.0,
- "13": 1931.0,
- "14": 1678.0,
- "15": 1918.0,
- "16": 1945.0,
- "17": 1707.0,
- "18": 1635.0,
- "19": 1720.0,
- "20": 1609.0,
- "21": 1813.0,
- "22": 1682.0,
- "23": 1908.0,
- "24": 1620.0,
- "25": 1563.0,
- "26": 1640.0,
- "27": 1775.0,
- "28": 1873.0,
- "29": 1969.0,
- "30": 1896.0,
- "31": 1588.0,
- "32": 1907.0,
- "33": 2180.0,
- "34": 1850.0,
- "35": 1987.0,
- "36": 1901.0,
- "37": 2358.0,
- "38": 2253.0,
- "39": 2364.0,
- "40": 2173.0,
- "41": 2234.0,
- "42": 2281.0,
- "43": 2027.0,
- "44": 2127.0,
- "45": 2170.0,
- "46": 2317.0,
- "47": 2438.0,
- "48": 2391.0,
- "49": 2276.0,
- "50": 2205.0,
- "51": 2647.0,
- "52": 2533.0,
- "53": 2935.0,
- "54": 2623.0,
- "55": 2386.0,
- "56": 2664.0,
- "57": 2391.0,
- "58": 2863.0,
- "59": 2758.0,
- "60": 2456.0,
- "61": 2865.0,
- "62": 2559.0,
- "63": 2463.0,
- "64": 3014.0,
- "65": 2526.0,
- "66": 3010.0,
- "67": 2723.0,
- "68": 2616.0,
- "69": 2739.0,
- "70": 3188.0,
- "71": 2919.0,
- "72": 2355.0,
- "73": 2921.0,
- "74": 1944.0,
- "75": 2454.0,
- "76": 3005.0,
- "77": 3204.0,
- "78": 3244.0,
- "79": 3047.0,
- "80": 3220.0,
- "81": 3492.0,
- "82": 3205.0,
- "83": 2692.0,
- "84": 3149.0,
- "85": 3256.0,
- "86": 2562.0,
- "87": 3753.0,
- "88": 2921.0,
- "89": 3239.0,
- "90": 3001.0,
- "91": 2656.0,
- "92": 3146.0,
- "93": 2642.0,
- "94": 3289.0,
- "95": 3324.0,
- "96": 3350.0,
- "97": 3079.0,
- "98": 3564.0,
- "99": 3215.0,
- "100": 3238.0
+ "1": 1876.0,
+ "2": 1608.0,
+ "3": 1840.0,
+ "4": 1776.0,
+ "5": 1738.0,
+ "6": 1754.0,
+ "7": 1820.0,
+ "8": 1741.0,
+ "9": 1763.0,
+ "10": 1831.0,
+ "11": 1719.0,
+ "12": 1721.0,
+ "13": 1823.0,
+ "14": 1908.0,
+ "15": 1638.0,
+ "16": 1807.0,
+ "17": 1849.0,
+ "18": 1907.0,
+ "19": 1697.0,
+ "20": 1766.0,
+ "21": 1856.0,
+ "22": 1740.0,
+ "23": 1751.0,
+ "24": 1936.0,
+ "25": 1726.0,
+ "26": 1660.0,
+ "27": 1817.0,
+ "28": 1699.0,
+ "29": 1956.0,
+ "30": 1835.0,
+ "31": 2040.0,
+ "32": 1949.0,
+ "33": 1902.0,
+ "34": 2003.0,
+ "35": 2190.0,
+ "36": 1999.0,
+ "37": 2269.0,
+ "38": 2091.0,
+ "39": 2123.0,
+ "40": 2255.0,
+ "41": 2303.0,
+ "42": 2006.0,
+ "43": 2337.0,
+ "44": 2148.0,
+ "45": 2533.0,
+ "46": 2427.0,
+ "47": 2355.0,
+ "48": 2503.0,
+ "49": 2798.0,
+ "50": 2522.0,
+ "51": 2511.0,
+ "52": 2675.0,
+ "53": 2488.0,
+ "54": 2696.0,
+ "55": 2402.0,
+ "56": 2719.0,
+ "57": 2213.0,
+ "58": 3568.0,
+ "59": 2848.0,
+ "60": 2927.0,
+ "61": 2635.0,
+ "62": 3128.0,
+ "63": 3208.0,
+ "64": 3586.0,
+ "65": 2627.0,
+ "66": 2953.0,
+ "67": 3779.0,
+ "68": 3405.0,
+ "69": 2940.0,
+ "70": 3324.0,
+ "71": 3249.0,
+ "72": 2914.0,
+ "73": 3396.0,
+ "74": 3436.0,
+ "75": 2999.0,
+ "76": 3254.0,
+ "77": 3550.0,
+ "78": 3368.0,
+ "79": 3362.0,
+ "80": 3052.0,
+ "81": 3480.0,
+ "82": 2934.0,
+ "83": 2995.0,
+ "84": 2909.0,
+ "85": 2673.0,
+ "86": 3051.0,
+ "87": 2939.0,
+ "88": 3104.0,
+ "89": 3197.0,
+ "90": 3666.0,
+ "91": 2889.0,
+ "92": 3003.0,
+ "93": 3148.0,
+ "94": 2984.0,
+ "95": 3451.0,
+ "96": 3409.0,
+ "97": 3460.0,
+ "98": 3482.0,
+ "99": 3094.0,
+ "100": 3127.0
}
},
"mem-allocated-bytes": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 9.86816,
- "2": 0.1216,
- "3": 0.1166,
- "4": 0.08589,
- "5": 0.08587,
- "6": 0.08491,
- "7": 0.0844,
- "8": 0.08084,
- "9": 0.07931,
- "10": 0.0798,
- "11": 0.07849,
- "12": 0.07832,
- "13": 0.0803,
- "14": 0.08035,
- "15": 0.07881,
- "16": 0.07881,
- "17": 0.08069,
- "18": 0.0794,
- "19": 0.07935,
- "20": 0.07915,
- "21": 0.07896,
- "22": 0.08062,
- "23": 0.08009,
- "24": 0.07923,
- "25": 0.07839,
- "26": 0.08166,
- "27": 0.07977,
- "28": 0.08005,
- "29": 0.08017,
- "30": 0.08118,
- "31": 0.0811,
- "32": 0.07964,
- "33": 0.08086,
- "34": 0.08069,
- "35": 0.07986,
- "36": 0.08098,
- "37": 0.07939,
- "38": 0.07947,
- "39": 0.07943,
- "40": 0.08028,
- "41": 0.07981,
- "42": 0.08016,
- "43": 0.08245,
- "44": 0.0799,
- "45": 0.08077,
- "46": 0.08028,
- "47": 0.07892,
- "48": 0.07997,
- "49": 0.08314,
- "50": 0.08027,
- "51": 0.08246,
- "52": 0.07991,
- "53": 0.08005,
- "54": 0.07954,
- "55": 0.07969,
- "56": 0.07938,
- "57": 0.07891,
- "58": 0.07987,
- "59": 0.0798,
- "60": 0.08057,
- "61": 0.07888,
- "62": 0.07914,
- "63": 0.07997,
- "64": 0.07986,
- "65": 0.07977,
- "66": 0.07953,
- "67": 0.07927,
- "68": 0.08003,
- "69": 0.08005,
- "70": 0.07926,
- "71": 0.07923,
- "72": 0.07966,
- "73": 0.08033,
- "74": 0.08038,
- "75": 0.07956,
- "76": 0.07935,
- "77": 0.07891,
- "78": 0.08007,
- "79": 0.08135,
- "80": 0.08025,
- "81": 0.07919,
- "82": 0.07932,
- "83": 0.07953,
- "84": 0.07937,
- "85": 0.0797,
- "86": 0.08168,
- "87": 0.08023,
- "88": 0.07957,
- "89": 0.08011,
- "90": 0.07975,
- "91": 0.08043,
- "92": 0.08179,
- "93": 0.08049,
- "94": 0.07951,
- "95": 0.08026,
- "96": 0.08,
- "97": 0.07948,
- "98": 0.0805,
- "99": 0.07879,
- "100": 0.07954
+ "1": "nan",
+ "2": 7.56814,
+ "3": 0.10619,
+ "4": 0.08033,
+ "5": 0.0808,
+ "6": 0.08084,
+ "7": 0.08095,
+ "8": 0.0807,
+ "9": 0.07994,
+ "10": 0.08005,
+ "11": 0.08106,
+ "12": 0.07993,
+ "13": 0.07921,
+ "14": 0.08247,
+ "15": 0.08144,
+ "16": 0.07818,
+ "17": 0.0804,
+ "18": 0.0807,
+ "19": 0.07975,
+ "20": 0.0793,
+ "21": 0.08029,
+ "22": 0.08012,
+ "23": 0.08013,
+ "24": 0.07971,
+ "25": 0.07927,
+ "26": 0.07965,
+ "27": 0.08096,
+ "28": 0.08004,
+ "29": 0.08139,
+ "30": 0.09128,
+ "31": 0.09161,
+ "32": 0.09021,
+ "33": 0.09147,
+ "34": 0.09147,
+ "35": 0.09111,
+ "36": 0.08057,
+ "37": 0.08069,
+ "38": 0.0799,
+ "39": 0.0797,
+ "40": 0.07838,
+ "41": 0.0784,
+ "42": 0.07819,
+ "43": 0.08049,
+ "44": 0.07909,
+ "45": 0.07982,
+ "46": 0.07852,
+ "47": 0.07896,
+ "48": 0.07853,
+ "49": 0.07864,
+ "50": 0.07995,
+ "51": 0.14736,
+ "52": 0.07916,
+ "53": 0.07876,
+ "54": 0.0807,
+ "55": 0.07827,
+ "56": 0.07973,
+ "57": 0.07822,
+ "58": 0.07904,
+ "59": 0.07985,
+ "60": 0.07836,
+ "61": 0.07955,
+ "62": 0.07998,
+ "63": 0.07907,
+ "64": 0.07957,
+ "65": 0.07933,
+ "66": 0.0801,
+ "67": 0.08009,
+ "68": 0.07922,
+ "69": 0.0801,
+ "70": 0.0796,
+ "71": 0.07976,
+ "72": 0.07965,
+ "73": 0.0802,
+ "74": 0.08056,
+ "75": 0.07969,
+ "76": 0.07954,
+ "77": 0.07992,
+ "78": 0.07911,
+ "79": 0.07963,
+ "80": 0.07839,
+ "81": 0.08049,
+ "82": 0.07814,
+ "83": 0.07997,
+ "84": 0.08335,
+ "85": 0.08135,
+ "86": 0.07828,
+ "87": 0.07964,
+ "88": 0.07949,
+ "89": 0.07875,
+ "90": 0.07861,
+ "91": 0.0783,
+ "92": 0.08012,
+ "93": 0.08192,
+ "94": 0.07835,
+ "95": 0.07993,
+ "96": 0.07909,
+ "97": 0.07833,
+ "98": 0.0784,
+ "99": 0.07862,
+ "100": 0.0786
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr/model_config.yaml
index d502c3e1fef..6bcce43a2db 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist/model_config.yaml
index edc9bc1ff2a..51cbf48c21b 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular # Usually ckpt-resume, but as a WAR to #513 set to regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss/golden_values_dev_dgx_gb200.json
index 0758fd3a8cf..64f3b7e5f89 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss/golden_values_dev_dgx_gb200.json
@@ -6,104 +6,104 @@
"values": {
"1": 10.82555,
"2": 10.83286,
- "3": 10.82763,
- "4": 10.79573,
- "5": 10.85699,
- "6": 10.8639,
+ "3": 10.82762,
+ "4": 10.79569,
+ "5": 10.85695,
+ "6": 10.86388,
"7": 10.82612,
"8": 10.82543,
- "9": 10.8359,
- "10": 10.79633,
- "11": 10.87819,
- "12": 10.85823,
+ "9": 10.83589,
+ "10": 10.79632,
+ "11": 10.8782,
+ "12": 10.85826,
"13": 10.85425,
- "14": 10.87526,
- "15": 10.79206,
- "16": 10.80309,
- "17": 10.77438,
- "18": 10.80484,
- "19": 10.79368,
- "20": 10.69574,
- "21": 10.68657,
- "22": 10.53162,
- "23": 10.70642,
- "24": 10.57336,
- "25": 10.51534,
- "26": 10.59088,
- "27": 10.60779,
- "28": 10.57051,
- "29": 10.58978,
- "30": 10.34722,
- "31": 10.07772,
+ "14": 10.87525,
+ "15": 10.79207,
+ "16": 10.80307,
+ "17": 10.7744,
+ "18": 10.80487,
+ "19": 10.7937,
+ "20": 10.69578,
+ "21": 10.68658,
+ "22": 10.53159,
+ "23": 10.70644,
+ "24": 10.57339,
+ "25": 10.51533,
+ "26": 10.59086,
+ "27": 10.60781,
+ "28": 10.57047,
+ "29": 10.58977,
+ "30": 10.34723,
+ "31": 10.07764,
"32": 10.46349,
- "33": 10.45726,
- "34": 10.19975,
- "35": 10.25642,
+ "33": 10.45724,
+ "34": 10.19974,
+ "35": 10.25638,
"36": 10.21264,
- "37": 10.34717,
- "38": 10.18011,
+ "37": 10.34716,
+ "38": 10.1801,
"39": 10.40833,
- "40": 10.07628,
- "41": 10.1297,
+ "40": 10.07629,
+ "41": 10.12976,
"42": 10.21174,
- "43": 9.8171,
+ "43": 9.81705,
"44": 9.94032,
- "45": 9.81748,
+ "45": 9.81746,
"46": 9.8063,
- "47": 10.12475,
- "48": 9.84049,
- "49": 9.51015,
- "50": 9.88941,
- "51": 9.8426,
- "52": 9.72578,
- "53": 10.05977,
- "54": 9.95226,
- "55": 9.88321,
- "56": 9.61276,
+ "47": 10.12473,
+ "48": 9.84048,
+ "49": 9.51014,
+ "50": 9.88943,
+ "51": 9.84256,
+ "52": 9.72576,
+ "53": 10.05974,
+ "54": 9.95227,
+ "55": 9.88316,
+ "56": 9.61277,
"57": 9.46222,
"58": 9.82313,
- "59": 9.57665,
+ "59": 9.57668,
"60": 9.48518,
- "61": 9.6788,
+ "61": 9.67879,
"62": 9.97777,
- "63": 9.36212,
- "64": 9.75714,
- "65": 8.93499,
- "66": 9.69281,
- "67": 9.36709,
- "68": 9.78179,
- "69": 9.79451,
- "70": 9.72295,
- "71": 9.62027,
- "72": 9.56974,
- "73": 9.481,
- "74": 8.91241,
- "75": 9.40906,
- "76": 9.06623,
- "77": 10.05808,
- "78": 9.72188,
- "79": 9.36927,
- "80": 9.40027,
- "81": 9.47702,
- "82": 9.69788,
- "83": 9.30742,
- "84": 9.41496,
- "85": 9.61115,
- "86": 9.07104,
+ "63": 9.36211,
+ "64": 9.75715,
+ "65": 8.93497,
+ "66": 9.6928,
+ "67": 9.3671,
+ "68": 9.78177,
+ "69": 9.79452,
+ "70": 9.72299,
+ "71": 9.62026,
+ "72": 9.56976,
+ "73": 9.48099,
+ "74": 8.91243,
+ "75": 9.40905,
+ "76": 9.06619,
+ "77": 10.0581,
+ "78": 9.72189,
+ "79": 9.36926,
+ "80": 9.40028,
+ "81": 9.47705,
+ "82": 9.69791,
+ "83": 9.30741,
+ "84": 9.41491,
+ "85": 9.61114,
+ "86": 9.07103,
"87": 9.59609,
- "88": 9.74908,
- "89": 9.5961,
- "90": 9.82722,
- "91": 9.3366,
- "92": 9.3558,
- "93": 9.08695,
- "94": 8.82752,
- "95": 9.53066,
- "96": 9.52759,
- "97": 9.30671,
- "98": 9.66909,
- "99": 8.89637,
- "100": 9.4052
+ "88": 9.74911,
+ "89": 9.59604,
+ "90": 9.82721,
+ "91": 9.33657,
+ "92": 9.35581,
+ "93": 9.08692,
+ "94": 8.82753,
+ "95": 9.53064,
+ "96": 9.52758,
+ "97": 9.30666,
+ "98": 9.66906,
+ "99": 8.89638,
+ "100": 9.40523
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1651.0,
- "2": 1716.0,
- "3": 1760.0,
- "4": 1771.0,
- "5": 1899.0,
- "6": 1905.0,
- "7": 1842.0,
- "8": 1706.0,
- "9": 1891.0,
- "10": 1543.0,
- "11": 1937.0,
- "12": 1794.0,
- "13": 1982.0,
- "14": 1727.0,
- "15": 1890.0,
- "16": 1746.0,
- "17": 1818.0,
- "18": 1651.0,
- "19": 1782.0,
- "20": 1698.0,
- "21": 1950.0,
- "22": 1702.0,
- "23": 1972.0,
- "24": 1551.0,
- "25": 1587.0,
- "26": 1773.0,
- "27": 1791.0,
- "28": 1858.0,
- "29": 1950.0,
- "30": 1951.0,
- "31": 1499.0,
- "32": 1823.0,
- "33": 2055.0,
- "34": 1788.0,
- "35": 1877.0,
- "36": 1933.0,
- "37": 2302.0,
- "38": 2181.0,
- "39": 2223.0,
- "40": 2009.0,
- "41": 2178.0,
- "42": 2185.0,
- "43": 2041.0,
- "44": 2069.0,
- "45": 2004.0,
- "46": 2212.0,
- "47": 2446.0,
- "48": 2290.0,
- "49": 2183.0,
- "50": 2323.0,
- "51": 2587.0,
- "52": 2574.0,
- "53": 2831.0,
- "54": 2602.0,
- "55": 2403.0,
- "56": 2822.0,
- "57": 2223.0,
- "58": 2954.0,
- "59": 2871.0,
- "60": 2518.0,
- "61": 2922.0,
- "62": 2677.0,
- "63": 2533.0,
- "64": 3023.0,
- "65": 2609.0,
- "66": 2960.0,
- "67": 2867.0,
- "68": 2652.0,
- "69": 3053.0,
- "70": 3011.0,
- "71": 2870.0,
- "72": 2460.0,
- "73": 3114.0,
- "74": 2017.0,
- "75": 2527.0,
- "76": 2954.0,
- "77": 2955.0,
- "78": 3055.0,
- "79": 3098.0,
- "80": 3047.0,
- "81": 3362.0,
- "82": 3296.0,
- "83": 2825.0,
- "84": 3113.0,
- "85": 3196.0,
- "86": 2666.0,
- "87": 3583.0,
- "88": 2985.0,
- "89": 3259.0,
- "90": 3220.0,
- "91": 2781.0,
- "92": 3090.0,
- "93": 2686.0,
- "94": 3474.0,
- "95": 3147.0,
- "96": 3418.0,
- "97": 3036.0,
- "98": 3411.0,
- "99": 3152.0,
- "100": 3098.0
+ "1": 1622.0,
+ "2": 1729.0,
+ "3": 1764.0,
+ "4": 1727.0,
+ "5": 1879.0,
+ "6": 1863.0,
+ "7": 1896.0,
+ "8": 1661.0,
+ "9": 1798.0,
+ "10": 1465.0,
+ "11": 1851.0,
+ "12": 1790.0,
+ "13": 1974.0,
+ "14": 1776.0,
+ "15": 1923.0,
+ "16": 1869.0,
+ "17": 1814.0,
+ "18": 1673.0,
+ "19": 1744.0,
+ "20": 1680.0,
+ "21": 1843.0,
+ "22": 1710.0,
+ "23": 2112.0,
+ "24": 1600.0,
+ "25": 1567.0,
+ "26": 1625.0,
+ "27": 1804.0,
+ "28": 1901.0,
+ "29": 1938.0,
+ "30": 1963.0,
+ "31": 1473.0,
+ "32": 1891.0,
+ "33": 2173.0,
+ "34": 1831.0,
+ "35": 1908.0,
+ "36": 1909.0,
+ "37": 2422.0,
+ "38": 2084.0,
+ "39": 2280.0,
+ "40": 2103.0,
+ "41": 2182.0,
+ "42": 2326.0,
+ "43": 1949.0,
+ "44": 2128.0,
+ "45": 2031.0,
+ "46": 2217.0,
+ "47": 2517.0,
+ "48": 2333.0,
+ "49": 2180.0,
+ "50": 2334.0,
+ "51": 2524.0,
+ "52": 2584.0,
+ "53": 2836.0,
+ "54": 2611.0,
+ "55": 2414.0,
+ "56": 2748.0,
+ "57": 2319.0,
+ "58": 2938.0,
+ "59": 2855.0,
+ "60": 2471.0,
+ "61": 2934.0,
+ "62": 2575.0,
+ "63": 2539.0,
+ "64": 2873.0,
+ "65": 2772.0,
+ "66": 2864.0,
+ "67": 2833.0,
+ "68": 2686.0,
+ "69": 3058.0,
+ "70": 3005.0,
+ "71": 2876.0,
+ "72": 2524.0,
+ "73": 2982.0,
+ "74": 2040.0,
+ "75": 2608.0,
+ "76": 2865.0,
+ "77": 3054.0,
+ "78": 2925.0,
+ "79": 3052.0,
+ "80": 2997.0,
+ "81": 3408.0,
+ "82": 3305.0,
+ "83": 2772.0,
+ "84": 3103.0,
+ "85": 3193.0,
+ "86": 2633.0,
+ "87": 3586.0,
+ "88": 3066.0,
+ "89": 3297.0,
+ "90": 3232.0,
+ "91": 2904.0,
+ "92": 3076.0,
+ "93": 2716.0,
+ "94": 3276.0,
+ "95": 3120.0,
+ "96": 3367.0,
+ "97": 3095.0,
+ "98": 3430.0,
+ "99": 3274.0,
+ "100": 3079.0
}
},
"mem-allocated-bytes": {
@@ -325,7 +325,7 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 2398288896.0,
+ "1": 2398288384.0,
"2": 2681049088.0,
"3": 2681049088.0,
"4": 2681049088.0,
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 3.0922,
- "3": 0.12788,
- "4": 0.11451,
- "5": 0.11407,
- "6": 0.11304,
- "7": 0.11565,
- "8": 0.11482,
- "9": 0.11841,
- "10": 0.11916,
- "11": 0.11884,
- "12": 0.11911,
- "13": 0.1155,
- "14": 0.12253,
- "15": 0.11369,
- "16": 0.11887,
- "17": 0.11433,
- "18": 0.12243,
- "19": 0.11544,
- "20": 0.11344,
- "21": 0.1254,
- "22": 0.11712,
- "23": 0.12494,
- "24": 0.12239,
- "25": 0.12344,
- "26": 0.11952,
- "27": 0.12117,
- "28": 0.11916,
- "29": 0.11974,
- "30": 0.11517,
- "31": 0.1219,
- "32": 0.12112,
- "33": 0.11997,
- "34": 0.1133,
- "35": 0.12245,
- "36": 0.12118,
- "37": 0.11239,
- "38": 0.12174,
- "39": 0.11964,
- "40": 0.11993,
- "41": 0.12013,
- "42": 0.12614,
- "43": 0.11697,
- "44": 0.11669,
- "45": 0.11781,
- "46": 0.11776,
- "47": 0.11182,
- "48": 0.1196,
- "49": 0.11814,
- "50": 0.11736,
- "51": 0.12093,
- "52": 0.1107,
- "53": 0.11502,
- "54": 0.11571,
- "55": 0.11493,
- "56": 0.11712,
- "57": 0.11663,
- "58": 0.11203,
- "59": 0.11604,
- "60": 0.11649,
- "61": 0.11616,
- "62": 0.11641,
- "63": 0.11603,
- "64": 0.11613,
- "65": 0.11708,
- "66": 0.11292,
- "67": 0.11356,
- "68": 0.11416,
- "69": 0.11305,
- "70": 0.11582,
- "71": 0.11552,
- "72": 0.11318,
- "73": 0.11798,
- "74": 0.11632,
- "75": 0.11624,
- "76": 0.11602,
- "77": 0.11547,
- "78": 0.11457,
- "79": 0.11402,
- "80": 0.11415,
- "81": 0.11627,
- "82": 0.11295,
- "83": 0.11397,
- "84": 0.11221,
- "85": 0.11326,
- "86": 0.11792,
- "87": 0.11391,
- "88": 0.11365,
- "89": 0.11478,
- "90": 0.11346,
- "91": 0.11213,
- "92": 0.11712,
- "93": 0.11574,
- "94": 0.11724,
- "95": 0.11254,
- "96": 0.11871,
- "97": 0.11957,
- "98": 0.11759,
- "99": 0.11864,
- "100": 0.11833
+ "2": 9.0179,
+ "3": 0.12425,
+ "4": 0.11171,
+ "5": 0.1113,
+ "6": 0.11136,
+ "7": 0.11152,
+ "8": 0.1128,
+ "9": 0.11283,
+ "10": 0.11178,
+ "11": 0.1114,
+ "12": 0.10919,
+ "13": 0.11062,
+ "14": 0.11052,
+ "15": 0.11204,
+ "16": 0.11101,
+ "17": 0.10801,
+ "18": 0.10961,
+ "19": 0.10805,
+ "20": 0.10908,
+ "21": 0.11181,
+ "22": 0.11019,
+ "23": 0.10842,
+ "24": 0.1101,
+ "25": 0.11377,
+ "26": 0.10767,
+ "27": 0.10828,
+ "28": 0.10775,
+ "29": 0.1111,
+ "30": 0.11204,
+ "31": 0.11206,
+ "32": 0.11166,
+ "33": 0.11093,
+ "34": 0.1093,
+ "35": 0.11115,
+ "36": 0.11082,
+ "37": 0.11056,
+ "38": 0.11028,
+ "39": 0.10835,
+ "40": 0.10939,
+ "41": 0.11079,
+ "42": 0.10956,
+ "43": 0.10806,
+ "44": 0.10932,
+ "45": 0.10799,
+ "46": 0.10768,
+ "47": 0.10905,
+ "48": 0.11353,
+ "49": 0.10994,
+ "50": 0.10866,
+ "51": 0.12844,
+ "52": 0.11643,
+ "53": 0.10864,
+ "54": 0.11347,
+ "55": 0.11478,
+ "56": 0.11082,
+ "57": 0.1112,
+ "58": 0.11325,
+ "59": 0.1109,
+ "60": 0.10898,
+ "61": 0.11011,
+ "62": 0.11005,
+ "63": 0.1101,
+ "64": 0.11097,
+ "65": 0.11278,
+ "66": 0.11156,
+ "67": 0.11122,
+ "68": 0.11125,
+ "69": 0.11008,
+ "70": 0.10837,
+ "71": 0.10856,
+ "72": 0.11244,
+ "73": 0.11132,
+ "74": 0.11053,
+ "75": 0.11064,
+ "76": 0.11148,
+ "77": 0.11101,
+ "78": 0.11284,
+ "79": 0.1119,
+ "80": 0.11215,
+ "81": 0.11292,
+ "82": 0.11191,
+ "83": 0.1108,
+ "84": 0.11285,
+ "85": 0.11227,
+ "86": 0.11126,
+ "87": 0.11007,
+ "88": 0.11445,
+ "89": 0.11142,
+ "90": 0.11159,
+ "91": 0.11062,
+ "92": 0.11088,
+ "93": 0.11224,
+ "94": 0.11204,
+ "95": 0.11256,
+ "96": 0.11345,
+ "97": 0.11247,
+ "98": 0.11327,
+ "99": 0.11177,
+ "100": 0.11086
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss/golden_values_dev_dgx_h100.json
index 126681fbe76..35ec9ab4d8b 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.84517,
- "2": 10.85349,
- "3": 10.8539,
- "4": 10.83825,
- "5": 10.87427,
- "6": 10.89307,
- "7": 10.85454,
- "8": 10.8626,
- "9": 10.86468,
- "10": 10.82907,
- "11": 10.88789,
- "12": 10.87095,
- "13": 10.87916,
- "14": 10.89079,
- "15": 10.81974,
- "16": 10.83162,
- "17": 10.79863,
- "18": 10.81667,
- "19": 10.81919,
- "20": 10.727,
- "21": 10.70594,
- "22": 10.56364,
- "23": 10.72802,
- "24": 10.60832,
- "25": 10.55217,
- "26": 10.60845,
- "27": 10.62847,
- "28": 10.5831,
- "29": 10.60012,
- "30": 10.36614,
- "31": 10.12044,
- "32": 10.47684,
- "33": 10.46873,
- "34": 10.22319,
- "35": 10.2778,
- "36": 10.22892,
- "37": 10.35949,
- "38": 10.19371,
- "39": 10.4155,
- "40": 10.0976,
- "41": 10.15737,
- "42": 10.22396,
- "43": 9.83286,
- "44": 9.96916,
- "45": 9.84196,
- "46": 9.83045,
- "47": 10.15628,
- "48": 9.85484,
- "49": 9.54086,
- "50": 9.9125,
- "51": 9.8587,
- "52": 9.74287,
- "53": 10.06647,
- "54": 9.95168,
- "55": 9.88096,
- "56": 9.62625,
- "57": 9.47766,
- "58": 9.8335,
- "59": 9.58522,
- "60": 9.50125,
- "61": 9.69186,
- "62": 9.98858,
- "63": 9.38478,
- "64": 9.78027,
- "65": 8.94761,
- "66": 9.70857,
- "67": 9.36847,
- "68": 9.78438,
- "69": 9.79407,
- "70": 9.7424,
- "71": 9.61808,
- "72": 9.58427,
- "73": 9.50347,
- "74": 8.9422,
- "75": 9.42532,
- "76": 9.07407,
- "77": 10.06351,
- "78": 9.7208,
- "79": 9.37296,
- "80": 9.40396,
- "81": 9.48168,
- "82": 9.69778,
- "83": 9.30711,
- "84": 9.41712,
- "85": 9.61405,
- "86": 9.07618,
- "87": 9.59088,
- "88": 9.7464,
- "89": 9.59987,
- "90": 9.81418,
- "91": 9.33775,
- "92": 9.35372,
- "93": 9.07397,
- "94": 8.8317,
- "95": 9.5173,
- "96": 9.52412,
- "97": 9.30995,
- "98": 9.66807,
- "99": 8.8859,
- "100": 9.39541
+ "1": 10.96464,
+ "2": 10.95235,
+ "3": 10.95854,
+ "4": 10.95197,
+ "5": 10.95261,
+ "6": 10.95382,
+ "7": 10.95159,
+ "8": 10.95132,
+ "9": 10.9587,
+ "10": 10.94456,
+ "11": 10.93682,
+ "12": 10.94339,
+ "13": 10.94312,
+ "14": 10.94293,
+ "15": 10.91494,
+ "16": 10.9003,
+ "17": 10.91111,
+ "18": 10.90616,
+ "19": 10.90162,
+ "20": 10.80787,
+ "21": 10.7946,
+ "22": 10.80476,
+ "23": 10.78979,
+ "24": 10.77498,
+ "25": 10.76171,
+ "26": 10.75518,
+ "27": 10.71954,
+ "28": 10.63912,
+ "29": 10.60849,
+ "30": 10.5865,
+ "31": 10.591,
+ "32": 10.5711,
+ "33": 10.5403,
+ "34": 10.49895,
+ "35": 10.50027,
+ "36": 10.48632,
+ "37": 10.44931,
+ "38": 10.45067,
+ "39": 10.41767,
+ "40": 10.39537,
+ "41": 10.37478,
+ "42": 10.3606,
+ "43": 10.33044,
+ "44": 10.31903,
+ "45": 10.31605,
+ "46": 10.27651,
+ "47": 10.26822,
+ "48": 10.21836,
+ "49": 10.21387,
+ "50": 10.22193,
+ "51": 10.22213,
+ "52": 10.1675,
+ "53": 10.17222,
+ "54": 10.13795,
+ "55": 10.10896,
+ "56": 10.13225,
+ "57": 10.12136,
+ "58": 10.12634,
+ "59": 10.07039,
+ "60": 10.09305,
+ "61": 10.04908,
+ "62": 10.01778,
+ "63": 10.08391,
+ "64": 10.03345,
+ "65": 10.00556,
+ "66": 10.04738,
+ "67": 10.01995,
+ "68": 9.98698,
+ "69": 9.99636,
+ "70": 9.9828,
+ "71": 10.00547,
+ "72": 9.99247,
+ "73": 9.97615,
+ "74": 9.96758,
+ "75": 9.93705,
+ "76": 9.96636,
+ "77": 9.96323,
+ "78": 9.91483,
+ "79": 9.91544,
+ "80": 9.92962,
+ "81": 9.95582,
+ "82": 9.89448,
+ "83": 9.85963,
+ "84": 9.80302,
+ "85": 9.78431,
+ "86": 9.88633,
+ "87": 9.90913,
+ "88": 9.88156,
+ "89": 9.82175,
+ "90": 9.82083,
+ "91": 9.82114,
+ "92": 9.81673,
+ "93": 9.74866,
+ "94": 9.82363,
+ "95": 9.81097,
+ "96": 9.79714,
+ "97": 9.74558,
+ "98": 9.76561,
+ "99": 9.82207,
+ "100": 9.70434
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1655.0,
- "2": 1697.0,
- "3": 1724.0,
- "4": 1720.0,
- "5": 1803.0,
- "6": 1772.0,
- "7": 1811.0,
- "8": 1678.0,
- "9": 1828.0,
- "10": 1448.0,
- "11": 1890.0,
- "12": 1657.0,
- "13": 1852.0,
- "14": 1717.0,
- "15": 1879.0,
- "16": 1921.0,
- "17": 1666.0,
- "18": 1729.0,
- "19": 1767.0,
- "20": 1657.0,
- "21": 1827.0,
- "22": 1594.0,
- "23": 1918.0,
- "24": 1622.0,
- "25": 1625.0,
- "26": 1649.0,
- "27": 1788.0,
- "28": 2030.0,
- "29": 1980.0,
- "30": 1882.0,
- "31": 1564.0,
- "32": 1918.0,
- "33": 2045.0,
- "34": 1884.0,
- "35": 1954.0,
- "36": 1910.0,
- "37": 2267.0,
- "38": 2195.0,
- "39": 2346.0,
- "40": 2191.0,
- "41": 2171.0,
- "42": 2246.0,
- "43": 1997.0,
- "44": 2156.0,
- "45": 2091.0,
- "46": 2439.0,
- "47": 2539.0,
- "48": 2418.0,
- "49": 2207.0,
- "50": 2189.0,
- "51": 2608.0,
- "52": 2444.0,
- "53": 2898.0,
- "54": 2664.0,
- "55": 2325.0,
- "56": 2614.0,
- "57": 2394.0,
- "58": 2812.0,
- "59": 2771.0,
- "60": 2361.0,
- "61": 2855.0,
- "62": 2675.0,
- "63": 2393.0,
- "64": 3014.0,
- "65": 2673.0,
- "66": 3051.0,
- "67": 2657.0,
- "68": 2662.0,
- "69": 2736.0,
- "70": 3139.0,
- "71": 2943.0,
- "72": 2293.0,
- "73": 2908.0,
- "74": 1887.0,
- "75": 2519.0,
- "76": 3060.0,
- "77": 3191.0,
- "78": 3211.0,
- "79": 3081.0,
- "80": 3205.0,
- "81": 3563.0,
- "82": 3201.0,
- "83": 2614.0,
- "84": 3162.0,
- "85": 3209.0,
- "86": 2660.0,
- "87": 3729.0,
- "88": 3002.0,
- "89": 3160.0,
- "90": 3168.0,
- "91": 2753.0,
- "92": 3258.0,
- "93": 2617.0,
- "94": 3341.0,
- "95": 3261.0,
- "96": 3370.0,
- "97": 3163.0,
- "98": 3566.0,
- "99": 3179.0,
- "100": 3135.0
+ "1": 1876.0,
+ "2": 1608.0,
+ "3": 1787.0,
+ "4": 1675.0,
+ "5": 1820.0,
+ "6": 1678.0,
+ "7": 1861.0,
+ "8": 1749.0,
+ "9": 1745.0,
+ "10": 1815.0,
+ "11": 1728.0,
+ "12": 1590.0,
+ "13": 1790.0,
+ "14": 1850.0,
+ "15": 1615.0,
+ "16": 1772.0,
+ "17": 1840.0,
+ "18": 1907.0,
+ "19": 1680.0,
+ "20": 1752.0,
+ "21": 1774.0,
+ "22": 1761.0,
+ "23": 1754.0,
+ "24": 1891.0,
+ "25": 1648.0,
+ "26": 1797.0,
+ "27": 1766.0,
+ "28": 1734.0,
+ "29": 1873.0,
+ "30": 1840.0,
+ "31": 2051.0,
+ "32": 2001.0,
+ "33": 1879.0,
+ "34": 1876.0,
+ "35": 2038.0,
+ "36": 1974.0,
+ "37": 2199.0,
+ "38": 2097.0,
+ "39": 2176.0,
+ "40": 2171.0,
+ "41": 2305.0,
+ "42": 2008.0,
+ "43": 2422.0,
+ "44": 2206.0,
+ "45": 2495.0,
+ "46": 2403.0,
+ "47": 2360.0,
+ "48": 2590.0,
+ "49": 2761.0,
+ "50": 2533.0,
+ "51": 2338.0,
+ "52": 2612.0,
+ "53": 2603.0,
+ "54": 2707.0,
+ "55": 2427.0,
+ "56": 2659.0,
+ "57": 2295.0,
+ "58": 3408.0,
+ "59": 2877.0,
+ "60": 2892.0,
+ "61": 2609.0,
+ "62": 3119.0,
+ "63": 3115.0,
+ "64": 3596.0,
+ "65": 2630.0,
+ "66": 3066.0,
+ "67": 3684.0,
+ "68": 3310.0,
+ "69": 3079.0,
+ "70": 3301.0,
+ "71": 3122.0,
+ "72": 2922.0,
+ "73": 3372.0,
+ "74": 3229.0,
+ "75": 3085.0,
+ "76": 3263.0,
+ "77": 3720.0,
+ "78": 3276.0,
+ "79": 3200.0,
+ "80": 3140.0,
+ "81": 3332.0,
+ "82": 2989.0,
+ "83": 2999.0,
+ "84": 2991.0,
+ "85": 2680.0,
+ "86": 3084.0,
+ "87": 2836.0,
+ "88": 3114.0,
+ "89": 3142.0,
+ "90": 3623.0,
+ "91": 2903.0,
+ "92": 2937.0,
+ "93": 3179.0,
+ "94": 2846.0,
+ "95": 3378.0,
+ "96": 3403.0,
+ "97": 3467.0,
+ "98": 3393.0,
+ "99": 3033.0,
+ "100": 3058.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 765318656.0,
- "2": 765318656.0,
- "3": 765318656.0,
- "4": 765318656.0,
- "5": 765318656.0,
- "6": 765318656.0,
- "7": 765318656.0,
- "8": 765318656.0,
- "9": 765318656.0,
- "10": 765318656.0,
- "11": 765318656.0,
- "12": 765318656.0,
- "13": 765318656.0,
- "14": 765318656.0,
- "15": 765318656.0,
- "16": 765318656.0,
- "17": 765318656.0,
- "18": 765318656.0,
- "19": 765318656.0,
- "20": 765318656.0,
- "21": 765318656.0,
- "22": 765318656.0,
- "23": 765318656.0,
- "24": 765318656.0,
- "25": 765318656.0,
- "26": 765318656.0,
- "27": 765318656.0,
- "28": 765318656.0,
- "29": 765318656.0,
- "30": 765318656.0,
- "31": 765318656.0,
- "32": 765318656.0,
- "33": 765318656.0,
- "34": 765318656.0,
- "35": 765318656.0,
- "36": 765318656.0,
- "37": 765318656.0,
- "38": 765318656.0,
- "39": 765318656.0,
- "40": 765318656.0,
- "41": 765318656.0,
- "42": 765318656.0,
- "43": 765318656.0,
- "44": 765318656.0,
- "45": 765318656.0,
- "46": 765318656.0,
- "47": 765318656.0,
- "48": 765318656.0,
- "49": 765318656.0,
- "50": 765318656.0,
- "51": 765318656.0,
- "52": 765318656.0,
- "53": 765318656.0,
- "54": 765318656.0,
- "55": 765318656.0,
- "56": 765318656.0,
- "57": 765318656.0,
- "58": 765318656.0,
- "59": 765318656.0,
- "60": 765318656.0,
- "61": 765318656.0,
- "62": 765318656.0,
- "63": 765318656.0,
- "64": 765318656.0,
- "65": 765318656.0,
- "66": 765318656.0,
- "67": 765318656.0,
- "68": 765318656.0,
- "69": 765318656.0,
- "70": 765318656.0,
- "71": 765318656.0,
- "72": 765318656.0,
- "73": 765318656.0,
- "74": 765318656.0,
- "75": 765318656.0,
- "76": 765318656.0,
- "77": 765318656.0,
- "78": 765318656.0,
- "79": 765318656.0,
- "80": 765318656.0,
- "81": 765318656.0,
- "82": 765318656.0,
- "83": 765318656.0,
- "84": 765318656.0,
- "85": 765318656.0,
- "86": 765318656.0,
- "87": 765318656.0,
- "88": 765318656.0,
- "89": 765318656.0,
- "90": 765318656.0,
- "91": 765318656.0,
- "92": 765318656.0,
- "93": 765318656.0,
- "94": 765318656.0,
- "95": 765318656.0,
- "96": 765318656.0,
- "97": 765318656.0,
- "98": 765318656.0,
- "99": 765318656.0,
- "100": 765318656.0
+ "1": 763221504.0,
+ "2": 763221504.0,
+ "3": 763221504.0,
+ "4": 763221504.0,
+ "5": 763221504.0,
+ "6": 763221504.0,
+ "7": 763221504.0,
+ "8": 763221504.0,
+ "9": 763221504.0,
+ "10": 763221504.0,
+ "11": 763221504.0,
+ "12": 763221504.0,
+ "13": 763221504.0,
+ "14": 763221504.0,
+ "15": 763221504.0,
+ "16": 763221504.0,
+ "17": 763221504.0,
+ "18": 763221504.0,
+ "19": 763221504.0,
+ "20": 763221504.0,
+ "21": 763221504.0,
+ "22": 763221504.0,
+ "23": 763221504.0,
+ "24": 763221504.0,
+ "25": 763221504.0,
+ "26": 763221504.0,
+ "27": 763221504.0,
+ "28": 763221504.0,
+ "29": 763221504.0,
+ "30": 763221504.0,
+ "31": 763221504.0,
+ "32": 763221504.0,
+ "33": 763221504.0,
+ "34": 763221504.0,
+ "35": 763221504.0,
+ "36": 763221504.0,
+ "37": 763221504.0,
+ "38": 763221504.0,
+ "39": 763221504.0,
+ "40": 763221504.0,
+ "41": 763221504.0,
+ "42": 763221504.0,
+ "43": 763221504.0,
+ "44": 763221504.0,
+ "45": 763221504.0,
+ "46": 763221504.0,
+ "47": 763221504.0,
+ "48": 763221504.0,
+ "49": 763221504.0,
+ "50": 763221504.0,
+ "51": 763221504.0,
+ "52": 763221504.0,
+ "53": 763221504.0,
+ "54": 763221504.0,
+ "55": 763221504.0,
+ "56": 763221504.0,
+ "57": 763221504.0,
+ "58": 763221504.0,
+ "59": 763221504.0,
+ "60": 763221504.0,
+ "61": 763221504.0,
+ "62": 763221504.0,
+ "63": 763221504.0,
+ "64": 763221504.0,
+ "65": 763221504.0,
+ "66": 763221504.0,
+ "67": 763221504.0,
+ "68": 763221504.0,
+ "69": 763221504.0,
+ "70": 763221504.0,
+ "71": 763221504.0,
+ "72": 763221504.0,
+ "73": 763221504.0,
+ "74": 763221504.0,
+ "75": 763221504.0,
+ "76": 763221504.0,
+ "77": 763221504.0,
+ "78": 763221504.0,
+ "79": 763221504.0,
+ "80": 763221504.0,
+ "81": 763221504.0,
+ "82": 763221504.0,
+ "83": 763221504.0,
+ "84": 763221504.0,
+ "85": 763221504.0,
+ "86": 763221504.0,
+ "87": 763221504.0,
+ "88": 763221504.0,
+ "89": 763221504.0,
+ "90": 763221504.0,
+ "91": 763221504.0,
+ "92": 763221504.0,
+ "93": 763221504.0,
+ "94": 763221504.0,
+ "95": 763221504.0,
+ "96": 763221504.0,
+ "97": 763221504.0,
+ "98": 763221504.0,
+ "99": 763221504.0,
+ "100": 763221504.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 2360539648.0,
- "2": 2645397504.0,
- "3": 2645397504.0,
- "4": 2645397504.0,
- "5": 2645397504.0,
- "6": 2645397504.0,
- "7": 2645397504.0,
- "8": 2645397504.0,
- "9": 2645397504.0,
- "10": 2645397504.0,
- "11": 2645397504.0,
- "12": 2645397504.0,
- "13": 2645397504.0,
- "14": 2645397504.0,
- "15": 2645397504.0,
- "16": 2645397504.0,
- "17": 2645397504.0,
- "18": 2645397504.0,
- "19": 2645397504.0,
- "20": 2645397504.0,
- "21": 2645397504.0,
- "22": 2645397504.0,
- "23": 2645397504.0,
- "24": 2645397504.0,
- "25": 2645397504.0,
- "26": 2645397504.0,
- "27": 2645397504.0,
- "28": 2645397504.0,
- "29": 2645397504.0,
- "30": 2645397504.0,
- "31": 2645397504.0,
- "32": 2645397504.0,
- "33": 2645397504.0,
- "34": 2645397504.0,
- "35": 2645397504.0,
- "36": 2645397504.0,
- "37": 2645397504.0,
- "38": 2645397504.0,
- "39": 2645397504.0,
- "40": 2645397504.0,
- "41": 2645397504.0,
- "42": 2645397504.0,
- "43": 2645397504.0,
- "44": 2645397504.0,
- "45": 2645397504.0,
- "46": 2645397504.0,
- "47": 2645397504.0,
- "48": 2645397504.0,
- "49": 2645397504.0,
- "50": 2645397504.0,
- "51": 2645397504.0,
- "52": 2645397504.0,
- "53": 2645397504.0,
- "54": 2645397504.0,
- "55": 2645397504.0,
- "56": 2645397504.0,
- "57": 2645397504.0,
- "58": 2645397504.0,
- "59": 2645397504.0,
- "60": 2645397504.0,
- "61": 2645397504.0,
- "62": 2645397504.0,
- "63": 2645397504.0,
- "64": 2645397504.0,
- "65": 2645397504.0,
- "66": 2645397504.0,
- "67": 2645397504.0,
- "68": 2645397504.0,
- "69": 2645397504.0,
- "70": 2645397504.0,
- "71": 2645397504.0,
- "72": 2645397504.0,
- "73": 2645397504.0,
- "74": 2645397504.0,
- "75": 2645397504.0,
- "76": 2645397504.0,
- "77": 2645397504.0,
- "78": 2645397504.0,
- "79": 2645397504.0,
- "80": 2645397504.0,
- "81": 2645397504.0,
- "82": 2645397504.0,
- "83": 2645397504.0,
- "84": 2645397504.0,
- "85": 2645397504.0,
- "86": 2645397504.0,
- "87": 2645397504.0,
- "88": 2645397504.0,
- "89": 2645397504.0,
- "90": 2645397504.0,
- "91": 2645397504.0,
- "92": 2645397504.0,
- "93": 2645397504.0,
- "94": 2645397504.0,
- "95": 2645397504.0,
- "96": 2645397504.0,
- "97": 2645397504.0,
- "98": 2645397504.0,
- "99": 2645397504.0,
- "100": 2645397504.0
+ "1": 2359491584.0,
+ "2": 2643300352.0,
+ "3": 2643300352.0,
+ "4": 2643300352.0,
+ "5": 2643300352.0,
+ "6": 2643300352.0,
+ "7": 2643300352.0,
+ "8": 2643300352.0,
+ "9": 2643300352.0,
+ "10": 2643300352.0,
+ "11": 2643300352.0,
+ "12": 2643300352.0,
+ "13": 2643300352.0,
+ "14": 2643300352.0,
+ "15": 2643300352.0,
+ "16": 2643300352.0,
+ "17": 2643300352.0,
+ "18": 2643300352.0,
+ "19": 2643300352.0,
+ "20": 2643300352.0,
+ "21": 2643300352.0,
+ "22": 2643300352.0,
+ "23": 2643300352.0,
+ "24": 2643300352.0,
+ "25": 2643300352.0,
+ "26": 2643300352.0,
+ "27": 2643300352.0,
+ "28": 2643300352.0,
+ "29": 2643300352.0,
+ "30": 2643300352.0,
+ "31": 2643300352.0,
+ "32": 2643300352.0,
+ "33": 2643300352.0,
+ "34": 2643300352.0,
+ "35": 2643300352.0,
+ "36": 2643300352.0,
+ "37": 2643300352.0,
+ "38": 2643300352.0,
+ "39": 2643300352.0,
+ "40": 2643300352.0,
+ "41": 2643300352.0,
+ "42": 2643300352.0,
+ "43": 2643300352.0,
+ "44": 2643300352.0,
+ "45": 2643300352.0,
+ "46": 2643300352.0,
+ "47": 2643300352.0,
+ "48": 2643300352.0,
+ "49": 2643300352.0,
+ "50": 2643300352.0,
+ "51": 2643300352.0,
+ "52": 2643300352.0,
+ "53": 2643300352.0,
+ "54": 2643300352.0,
+ "55": 2643300352.0,
+ "56": 2643300352.0,
+ "57": 2643300352.0,
+ "58": 2643300352.0,
+ "59": 2643300352.0,
+ "60": 2643300352.0,
+ "61": 2643300352.0,
+ "62": 2643300352.0,
+ "63": 2643300352.0,
+ "64": 2643300352.0,
+ "65": 2643300352.0,
+ "66": 2643300352.0,
+ "67": 2643300352.0,
+ "68": 2643300352.0,
+ "69": 2643300352.0,
+ "70": 2643300352.0,
+ "71": 2643300352.0,
+ "72": 2643300352.0,
+ "73": 2643300352.0,
+ "74": 2643300352.0,
+ "75": 2643300352.0,
+ "76": 2643300352.0,
+ "77": 2643300352.0,
+ "78": 2643300352.0,
+ "79": 2643300352.0,
+ "80": 2643300352.0,
+ "81": 2643300352.0,
+ "82": 2643300352.0,
+ "83": 2643300352.0,
+ "84": 2643300352.0,
+ "85": 2643300352.0,
+ "86": 2643300352.0,
+ "87": 2643300352.0,
+ "88": 2643300352.0,
+ "89": 2643300352.0,
+ "90": 2643300352.0,
+ "91": 2643300352.0,
+ "92": 2643300352.0,
+ "93": 2643300352.0,
+ "94": 2643300352.0,
+ "95": 2643300352.0,
+ "96": 2643300352.0,
+ "97": 2643300352.0,
+ "98": 2643300352.0,
+ "99": 2643300352.0,
+ "100": 2643300352.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 11.89927,
- "2": 0.1153,
- "3": 0.10368,
+ "1": "nan",
+ "2": 8.38432,
+ "3": 0.09734,
"4": 0.08198,
- "5": 0.0823,
- "6": 0.0813,
- "7": 0.08053,
- "8": 0.08097,
- "9": 0.08083,
- "10": 0.08105,
- "11": 0.08193,
- "12": 0.08083,
- "13": 0.08063,
- "14": 0.08095,
- "15": 0.08115,
- "16": 0.08099,
- "17": 0.08128,
- "18": 0.08134,
- "19": 0.08147,
- "20": 0.08174,
- "21": 0.08185,
- "22": 0.08175,
- "23": 0.08109,
- "24": 0.08065,
- "25": 0.08488,
- "26": 0.08433,
- "27": 0.08446,
- "28": 0.08482,
- "29": 0.08645,
- "30": 0.08469,
- "31": 0.08623,
- "32": 0.08474,
- "33": 0.08443,
- "34": 0.08442,
- "35": 0.08287,
- "36": 0.08188,
- "37": 0.08068,
- "38": 0.0808,
- "39": 0.08041,
- "40": 0.08119,
- "41": 0.08373,
- "42": 0.08116,
- "43": 0.08394,
- "44": 0.08252,
- "45": 0.08182,
- "46": 0.08217,
- "47": 0.08115,
- "48": 0.08122,
- "49": 0.08084,
- "50": 0.08062,
- "51": 0.09006,
- "52": 0.08529,
- "53": 0.08552,
- "54": 0.08335,
- "55": 0.08266,
- "56": 0.08016,
- "57": 0.08221,
- "58": 0.08,
- "59": 0.08121,
- "60": 0.08027,
- "61": 0.08342,
- "62": 0.08237,
- "63": 0.08269,
- "64": 0.0825,
- "65": 0.08238,
- "66": 0.08275,
- "67": 0.08276,
- "68": 0.08526,
- "69": 0.0814,
- "70": 0.08183,
- "71": 0.08214,
- "72": 0.08252,
- "73": 0.0824,
- "74": 0.08248,
- "75": 0.08211,
- "76": 0.0822,
- "77": 0.08148,
- "78": 0.08193,
- "79": 0.08271,
- "80": 0.082,
- "81": 0.08216,
- "82": 0.08205,
- "83": 0.0823,
- "84": 0.08236,
- "85": 0.08239,
- "86": 0.0805,
- "87": 0.07901,
- "88": 0.07985,
- "89": 0.07962,
- "90": 0.07883,
- "91": 0.07962,
- "92": 0.07909,
- "93": 0.07986,
- "94": 0.08107,
- "95": 0.08014,
- "96": 0.07993,
- "97": 0.08061,
- "98": 0.0808,
- "99": 0.07879,
- "100": 0.07901
+ "5": 0.08069,
+ "6": 0.08196,
+ "7": 0.0815,
+ "8": 0.08101,
+ "9": 0.08096,
+ "10": 0.08118,
+ "11": 0.08131,
+ "12": 0.08125,
+ "13": 0.08035,
+ "14": 0.07956,
+ "15": 0.0802,
+ "16": 0.08047,
+ "17": 0.08144,
+ "18": 0.08149,
+ "19": 0.0813,
+ "20": 0.0807,
+ "21": 0.08066,
+ "22": 0.08101,
+ "23": 0.08101,
+ "24": 0.08052,
+ "25": 0.07985,
+ "26": 0.08111,
+ "27": 0.08189,
+ "28": 0.08122,
+ "29": 0.08115,
+ "30": 0.08214,
+ "31": 0.0803,
+ "32": 0.07981,
+ "33": 0.08146,
+ "34": 0.08098,
+ "35": 0.08123,
+ "36": 0.08128,
+ "37": 0.08164,
+ "38": 0.08045,
+ "39": 0.08106,
+ "40": 0.08066,
+ "41": 0.08126,
+ "42": 0.08209,
+ "43": 0.08278,
+ "44": 0.081,
+ "45": 0.08112,
+ "46": 0.08114,
+ "47": 0.08052,
+ "48": 0.08124,
+ "49": 0.07844,
+ "50": 0.07803,
+ "51": 0.21518,
+ "52": 0.08006,
+ "53": 0.08132,
+ "54": 0.07937,
+ "55": 0.07938,
+ "56": 0.08037,
+ "57": 0.07881,
+ "58": 0.07976,
+ "59": 0.07801,
+ "60": 0.07793,
+ "61": 0.0794,
+ "62": 0.07936,
+ "63": 0.0794,
+ "64": 0.07825,
+ "65": 0.0793,
+ "66": 0.08006,
+ "67": 0.07846,
+ "68": 0.07965,
+ "69": 0.0785,
+ "70": 0.0781,
+ "71": 0.07967,
+ "72": 0.07963,
+ "73": 0.0793,
+ "74": 0.07934,
+ "75": 0.07967,
+ "76": 0.0787,
+ "77": 0.07839,
+ "78": 0.07971,
+ "79": 0.08017,
+ "80": 0.07879,
+ "81": 0.07826,
+ "82": 0.07757,
+ "83": 0.08025,
+ "84": 0.07976,
+ "85": 0.08071,
+ "86": 0.07857,
+ "87": 0.0823,
+ "88": 0.07931,
+ "89": 0.07898,
+ "90": 0.07914,
+ "91": 0.08228,
+ "92": 0.08057,
+ "93": 0.08023,
+ "94": 0.07889,
+ "95": 0.07852,
+ "96": 0.07986,
+ "97": 0.07889,
+ "98": 0.07887,
+ "99": 0.07843,
+ "100": 0.07827
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss/model_config.yaml
index 1b9c96b3f7d..6d77c3df361 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/golden_values_dev_dgx_gb200.json
index 28843c12217..a3165b8856a 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/golden_values_dev_dgx_gb200.json
@@ -6,104 +6,104 @@
"values": {
"1": 10.82555,
"2": 10.83286,
- "3": 10.82763,
- "4": 10.79573,
- "5": 10.85699,
- "6": 10.8639,
+ "3": 10.82762,
+ "4": 10.79569,
+ "5": 10.85695,
+ "6": 10.86388,
"7": 10.82612,
- "8": 10.82542,
- "9": 10.83587,
- "10": 10.79627,
- "11": 10.87822,
- "12": 10.85824,
- "13": 10.85426,
- "14": 10.87526,
- "15": 10.79208,
+ "8": 10.82543,
+ "9": 10.83589,
+ "10": 10.79632,
+ "11": 10.8782,
+ "12": 10.85826,
+ "13": 10.85425,
+ "14": 10.87525,
+ "15": 10.79207,
"16": 10.80307,
- "17": 10.77438,
+ "17": 10.7744,
"18": 10.80487,
- "19": 10.79369,
- "20": 10.69576,
- "21": 10.68654,
- "22": 10.53161,
- "23": 10.70646,
- "24": 10.57337,
- "25": 10.51533,
+ "19": 10.7937,
+ "20": 10.69579,
+ "21": 10.68655,
+ "22": 10.53162,
+ "23": 10.70645,
+ "24": 10.5734,
+ "25": 10.51531,
"26": 10.5909,
- "27": 10.60777,
- "28": 10.57049,
- "29": 10.58979,
- "30": 10.34722,
- "31": 10.07771,
- "32": 10.46349,
+ "27": 10.60778,
+ "28": 10.57052,
+ "29": 10.58974,
+ "30": 10.34723,
+ "31": 10.07765,
+ "32": 10.4635,
"33": 10.45722,
- "34": 10.19974,
- "35": 10.25643,
- "36": 10.21263,
- "37": 10.34718,
+ "34": 10.19977,
+ "35": 10.25639,
+ "36": 10.21261,
+ "37": 10.34715,
"38": 10.18009,
- "39": 10.40838,
- "40": 10.07629,
- "41": 10.1297,
- "42": 10.2117,
- "43": 9.81708,
- "44": 9.94034,
- "45": 9.81748,
- "46": 9.80633,
- "47": 10.12473,
- "48": 9.84047,
- "49": 9.51012,
- "50": 9.88943,
- "51": 9.84256,
- "52": 9.72573,
+ "39": 10.40837,
+ "40": 10.07626,
+ "41": 10.12969,
+ "42": 10.21172,
+ "43": 9.81709,
+ "44": 9.94035,
+ "45": 9.81749,
+ "46": 9.80632,
+ "47": 10.12471,
+ "48": 9.84046,
+ "49": 9.51013,
+ "50": 9.88941,
+ "51": 9.84258,
+ "52": 9.72577,
"53": 10.05974,
- "54": 9.95226,
- "55": 9.88318,
- "56": 9.61275,
- "57": 9.46219,
- "58": 9.8231,
- "59": 9.57666,
- "60": 9.48516,
- "61": 9.67876,
- "62": 9.97782,
- "63": 9.36212,
- "64": 9.75714,
- "65": 8.93494,
+ "54": 9.95228,
+ "55": 9.88317,
+ "56": 9.61277,
+ "57": 9.46223,
+ "58": 9.82312,
+ "59": 9.57667,
+ "60": 9.48518,
+ "61": 9.67881,
+ "62": 9.97778,
+ "63": 9.36213,
+ "64": 9.75717,
+ "65": 8.93497,
"66": 9.69283,
- "67": 9.36708,
+ "67": 9.36709,
"68": 9.78178,
- "69": 9.79452,
- "70": 9.72296,
- "71": 9.62031,
- "72": 9.56974,
- "73": 9.48101,
- "74": 8.91241,
- "75": 9.40905,
- "76": 9.06617,
+ "69": 9.79453,
+ "70": 9.72298,
+ "71": 9.62028,
+ "72": 9.56979,
+ "73": 9.48099,
+ "74": 8.91237,
+ "75": 9.40908,
+ "76": 9.06623,
"77": 10.05809,
- "78": 9.72194,
- "79": 9.36927,
- "80": 9.40029,
- "81": 9.47702,
- "82": 9.69787,
- "83": 9.30742,
- "84": 9.41492,
+ "78": 9.72192,
+ "79": 9.36926,
+ "80": 9.40026,
+ "81": 9.477,
+ "82": 9.69791,
+ "83": 9.30743,
+ "84": 9.41493,
"85": 9.61113,
- "86": 9.07103,
- "87": 9.5961,
- "88": 9.74909,
- "89": 9.59604,
+ "86": 9.07104,
+ "87": 9.59611,
+ "88": 9.74908,
+ "89": 9.5961,
"90": 9.82722,
- "91": 9.33657,
- "92": 9.35582,
- "93": 9.08689,
- "94": 8.82754,
+ "91": 9.33658,
+ "92": 9.3558,
+ "93": 9.08695,
+ "94": 8.82753,
"95": 9.53065,
- "96": 9.5276,
- "97": 9.30672,
- "98": 9.66905,
- "99": 8.89635,
- "100": 9.40525
+ "96": 9.52762,
+ "97": 9.30668,
+ "98": 9.66908,
+ "99": 8.89636,
+ "100": 9.40522
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1651.0,
- "2": 1716.0,
- "3": 1760.0,
- "4": 1771.0,
- "5": 1899.0,
- "6": 1905.0,
- "7": 1842.0,
- "8": 1667.0,
- "9": 1822.0,
- "10": 1434.0,
- "11": 1852.0,
- "12": 1741.0,
- "13": 1905.0,
- "14": 1841.0,
- "15": 1857.0,
- "16": 1841.0,
- "17": 1800.0,
- "18": 1666.0,
- "19": 1803.0,
- "20": 1800.0,
- "21": 1836.0,
- "22": 1688.0,
- "23": 1994.0,
- "24": 1641.0,
- "25": 1577.0,
- "26": 1676.0,
- "27": 1876.0,
- "28": 1970.0,
- "29": 1945.0,
- "30": 1916.0,
- "31": 1494.0,
- "32": 1868.0,
- "33": 2135.0,
- "34": 1740.0,
- "35": 1924.0,
- "36": 1854.0,
- "37": 2363.0,
- "38": 2164.0,
- "39": 2262.0,
- "40": 2081.0,
- "41": 2168.0,
- "42": 2247.0,
- "43": 2055.0,
- "44": 2070.0,
- "45": 1988.0,
- "46": 2208.0,
- "47": 2559.0,
- "48": 2287.0,
- "49": 2194.0,
- "50": 2303.0,
- "51": 2552.0,
- "52": 2565.0,
- "53": 2883.0,
- "54": 2710.0,
- "55": 2301.0,
- "56": 2798.0,
- "57": 2334.0,
- "58": 2979.0,
- "59": 2960.0,
- "60": 2451.0,
- "61": 2841.0,
- "62": 2577.0,
- "63": 2516.0,
- "64": 2907.0,
- "65": 2567.0,
- "66": 2862.0,
- "67": 2809.0,
- "68": 2609.0,
- "69": 2965.0,
- "70": 2985.0,
- "71": 2864.0,
- "72": 2613.0,
- "73": 3108.0,
- "74": 2048.0,
- "75": 2563.0,
- "76": 3046.0,
- "77": 3127.0,
- "78": 2959.0,
- "79": 3082.0,
- "80": 3025.0,
- "81": 3400.0,
- "82": 3223.0,
- "83": 2786.0,
- "84": 3180.0,
- "85": 3233.0,
- "86": 2611.0,
- "87": 3542.0,
- "88": 3084.0,
- "89": 3210.0,
- "90": 3271.0,
- "91": 2770.0,
- "92": 3220.0,
- "93": 2662.0,
- "94": 3405.0,
- "95": 3085.0,
- "96": 3336.0,
- "97": 3050.0,
- "98": 3421.0,
- "99": 3271.0,
- "100": 3079.0
+ "1": 1622.0,
+ "2": 1729.0,
+ "3": 1764.0,
+ "4": 1727.0,
+ "5": 1879.0,
+ "6": 1863.0,
+ "7": 1896.0,
+ "8": 1661.0,
+ "9": 1798.0,
+ "10": 1465.0,
+ "11": 1851.0,
+ "12": 1790.0,
+ "13": 1974.0,
+ "14": 1776.0,
+ "15": 1923.0,
+ "16": 1869.0,
+ "17": 1814.0,
+ "18": 1673.0,
+ "19": 1720.0,
+ "20": 1699.0,
+ "21": 1893.0,
+ "22": 1735.0,
+ "23": 1986.0,
+ "24": 1634.0,
+ "25": 1621.0,
+ "26": 1685.0,
+ "27": 1759.0,
+ "28": 1882.0,
+ "29": 1990.0,
+ "30": 2038.0,
+ "31": 1536.0,
+ "32": 1900.0,
+ "33": 2086.0,
+ "34": 1761.0,
+ "35": 1979.0,
+ "36": 1971.0,
+ "37": 2325.0,
+ "38": 2104.0,
+ "39": 2266.0,
+ "40": 2078.0,
+ "41": 2184.0,
+ "42": 2202.0,
+ "43": 2015.0,
+ "44": 2078.0,
+ "45": 2015.0,
+ "46": 2066.0,
+ "47": 2458.0,
+ "48": 2363.0,
+ "49": 2175.0,
+ "50": 2409.0,
+ "51": 2558.0,
+ "52": 2588.0,
+ "53": 2829.0,
+ "54": 2591.0,
+ "55": 2369.0,
+ "56": 2739.0,
+ "57": 2309.0,
+ "58": 2883.0,
+ "59": 2848.0,
+ "60": 2493.0,
+ "61": 2888.0,
+ "62": 2602.0,
+ "63": 2545.0,
+ "64": 2899.0,
+ "65": 2710.0,
+ "66": 3002.0,
+ "67": 2804.0,
+ "68": 2632.0,
+ "69": 2956.0,
+ "70": 2966.0,
+ "71": 2906.0,
+ "72": 2514.0,
+ "73": 3054.0,
+ "74": 2000.0,
+ "75": 2549.0,
+ "76": 2973.0,
+ "77": 3043.0,
+ "78": 3007.0,
+ "79": 3087.0,
+ "80": 2994.0,
+ "81": 3419.0,
+ "82": 3217.0,
+ "83": 2800.0,
+ "84": 3251.0,
+ "85": 3154.0,
+ "86": 2553.0,
+ "87": 3551.0,
+ "88": 3114.0,
+ "89": 3200.0,
+ "90": 3219.0,
+ "91": 2907.0,
+ "92": 3034.0,
+ "93": 2797.0,
+ "94": 3431.0,
+ "95": 3018.0,
+ "96": 3330.0,
+ "97": 3017.0,
+ "98": 3495.0,
+ "99": 3308.0,
+ "100": 3089.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 552325632.0,
- "2": 552325632.0,
- "3": 552325632.0,
- "4": 552325632.0,
- "5": 552325632.0,
- "6": 552325632.0,
- "7": 552325632.0,
- "8": 552325632.0,
- "9": 552325632.0,
- "10": 552325632.0,
- "11": 552325632.0,
- "12": 552325632.0,
- "13": 552325632.0,
- "14": 552325632.0,
- "15": 552325632.0,
- "16": 552325632.0,
- "17": 552325632.0,
- "18": 552325632.0,
- "19": 552325632.0,
- "20": 552325632.0,
- "21": 552325632.0,
- "22": 552325632.0,
- "23": 552325632.0,
- "24": 552325632.0,
- "25": 552325632.0,
- "26": 552325632.0,
- "27": 552325632.0,
- "28": 552325632.0,
- "29": 552325632.0,
- "30": 552325632.0,
- "31": 552325632.0,
- "32": 552325632.0,
- "33": 552325632.0,
- "34": 552325632.0,
- "35": 552325632.0,
- "36": 552325632.0,
- "37": 552325632.0,
- "38": 552325632.0,
- "39": 552325632.0,
- "40": 552325632.0,
- "41": 552325632.0,
- "42": 552325632.0,
- "43": 552325632.0,
- "44": 552325632.0,
- "45": 553374208.0,
- "46": 552325632.0,
- "47": 552325632.0,
- "48": 553374208.0,
- "49": 552325632.0,
- "50": 552325632.0,
- "51": 552325632.0,
- "52": 552325632.0,
- "53": 552325632.0,
- "54": 552325632.0,
- "55": 552325632.0,
- "56": 552325632.0,
- "57": 552325632.0,
- "58": 552325632.0,
- "59": 552325632.0,
- "60": 552325632.0,
- "61": 552325632.0,
- "62": 552325632.0,
- "63": 552325632.0,
- "64": 552325632.0,
- "65": 552325632.0,
- "66": 552325632.0,
- "67": 552325632.0,
- "68": 552325632.0,
- "69": 552325632.0,
- "70": 552325632.0,
- "71": 552325632.0,
- "72": 552325632.0,
- "73": 552325632.0,
- "74": 552325632.0,
- "75": 552325632.0,
- "76": 552325632.0,
- "77": 552325632.0,
- "78": 552325632.0,
- "79": 552325632.0,
- "80": 552325632.0,
- "81": 552325632.0,
- "82": 552325632.0,
- "83": 552325632.0,
- "84": 552325632.0,
- "85": 552325632.0,
- "86": 552325632.0,
- "87": 552325632.0,
- "88": 552325632.0,
- "89": 552325632.0,
- "90": 552325632.0,
- "91": 552325632.0,
- "92": 552325632.0,
- "93": 552325632.0,
- "94": 552325632.0,
- "95": 552325632.0,
- "96": 552325632.0,
- "97": 552325632.0,
- "98": 552325632.0,
- "99": 552325632.0,
- "100": 552325632.0
+ "1": 551273984.0,
+ "2": 551273984.0,
+ "3": 551273984.0,
+ "4": 551273984.0,
+ "5": 551273984.0,
+ "6": 551273984.0,
+ "7": 551273984.0,
+ "8": 551273984.0,
+ "9": 551273984.0,
+ "10": 551273984.0,
+ "11": 551273984.0,
+ "12": 551273984.0,
+ "13": 551273984.0,
+ "14": 551273984.0,
+ "15": 551273984.0,
+ "16": 551273984.0,
+ "17": 551273984.0,
+ "18": 551273984.0,
+ "19": 551273984.0,
+ "20": 551273984.0,
+ "21": 551273984.0,
+ "22": 551273984.0,
+ "23": 551273984.0,
+ "24": 551273984.0,
+ "25": 551273984.0,
+ "26": 551273984.0,
+ "27": 551273984.0,
+ "28": 551273984.0,
+ "29": 551273984.0,
+ "30": 551273984.0,
+ "31": 551273984.0,
+ "32": 551273984.0,
+ "33": 551273984.0,
+ "34": 551273984.0,
+ "35": 551273984.0,
+ "36": 551273984.0,
+ "37": 551273984.0,
+ "38": 551273984.0,
+ "39": 551273984.0,
+ "40": 551273984.0,
+ "41": 551273984.0,
+ "42": 551273984.0,
+ "43": 551273984.0,
+ "44": 551273984.0,
+ "45": 551273984.0,
+ "46": 551273984.0,
+ "47": 551273984.0,
+ "48": 551273984.0,
+ "49": 551273984.0,
+ "50": 551273984.0,
+ "51": 551273984.0,
+ "52": 551273984.0,
+ "53": 551273984.0,
+ "54": 551273984.0,
+ "55": 551273984.0,
+ "56": 551273984.0,
+ "57": 551273984.0,
+ "58": 551273984.0,
+ "59": 551273984.0,
+ "60": 551273984.0,
+ "61": 551273984.0,
+ "62": 551273984.0,
+ "63": 551273984.0,
+ "64": 551273984.0,
+ "65": 551273984.0,
+ "66": 551273984.0,
+ "67": 551273984.0,
+ "68": 551273984.0,
+ "69": 551273984.0,
+ "70": 551273984.0,
+ "71": 551273984.0,
+ "72": 551273984.0,
+ "73": 551273984.0,
+ "74": 551273984.0,
+ "75": 551273984.0,
+ "76": 551273984.0,
+ "77": 551273984.0,
+ "78": 551273984.0,
+ "79": 551273984.0,
+ "80": 551273984.0,
+ "81": 551273984.0,
+ "82": 551273984.0,
+ "83": 551273984.0,
+ "84": 551273984.0,
+ "85": 551273984.0,
+ "86": 551273984.0,
+ "87": 551273984.0,
+ "88": 551273984.0,
+ "89": 551273984.0,
+ "90": 551273984.0,
+ "91": 551273984.0,
+ "92": 551273984.0,
+ "93": 551273984.0,
+ "94": 551273984.0,
+ "95": 551273984.0,
+ "96": 551273984.0,
+ "97": 551273984.0,
+ "98": 551273984.0,
+ "99": 551273984.0,
+ "100": 551273984.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 2328239104.0,
- "2": 2471201792.0,
- "3": 2471201792.0,
- "4": 2471201792.0,
- "5": 2471201792.0,
- "6": 2471201792.0,
- "7": 2471201792.0,
- "8": 2471201792.0,
- "9": 2471201792.0,
- "10": 2471201792.0,
- "11": 2471201792.0,
- "12": 2471201792.0,
- "13": 2471201792.0,
- "14": 2471201792.0,
- "15": 2471201792.0,
- "16": 2471201792.0,
- "17": 2471201792.0,
- "18": 2471201792.0,
- "19": 2471201792.0,
- "20": 2471201792.0,
- "21": 2471201792.0,
- "22": 2471201792.0,
- "23": 2471201792.0,
- "24": 2471201792.0,
- "25": 2471201792.0,
- "26": 2471201792.0,
- "27": 2471201792.0,
- "28": 2471201792.0,
- "29": 2471201792.0,
- "30": 2471201792.0,
- "31": 2471201792.0,
- "32": 2471201792.0,
- "33": 2471201792.0,
- "34": 2471201792.0,
- "35": 2471201792.0,
- "36": 2471201792.0,
- "37": 2471201792.0,
- "38": 2471201792.0,
- "39": 2471201792.0,
- "40": 2471201792.0,
- "41": 2471201792.0,
- "42": 2471201792.0,
- "43": 2471201792.0,
- "44": 2471201792.0,
- "45": 2471201792.0,
- "46": 2471201792.0,
- "47": 2471201792.0,
- "48": 2471201792.0,
- "49": 2471201792.0,
- "50": 2471201792.0,
- "51": 2471201792.0,
- "52": 2471201792.0,
- "53": 2471201792.0,
- "54": 2471201792.0,
- "55": 2471201792.0,
- "56": 2471201792.0,
- "57": 2471201792.0,
- "58": 2471201792.0,
- "59": 2471201792.0,
- "60": 2471201792.0,
- "61": 2471201792.0,
- "62": 2471201792.0,
- "63": 2471201792.0,
- "64": 2471201792.0,
- "65": 2471201792.0,
- "66": 2471201792.0,
- "67": 2471201792.0,
- "68": 2471201792.0,
- "69": 2471201792.0,
- "70": 2471201792.0,
- "71": 2471201792.0,
- "72": 2471201792.0,
- "73": 2471201792.0,
- "74": 2471201792.0,
- "75": 2471201792.0,
- "76": 2471201792.0,
- "77": 2471201792.0,
- "78": 2471201792.0,
- "79": 2471201792.0,
- "80": 2471201792.0,
- "81": 2471201792.0,
- "82": 2471201792.0,
- "83": 2471201792.0,
- "84": 2471201792.0,
- "85": 2471201792.0,
- "86": 2471201792.0,
- "87": 2471201792.0,
- "88": 2471201792.0,
- "89": 2471201792.0,
- "90": 2471201792.0,
- "91": 2471201792.0,
- "92": 2471201792.0,
- "93": 2471201792.0,
- "94": 2471201792.0,
- "95": 2471201792.0,
- "96": 2471201792.0,
- "97": 2471201792.0,
- "98": 2471201792.0,
- "99": 2471201792.0,
- "100": 2471201792.0
+ "1": 2328238592.0,
+ "2": 2470149120.0,
+ "3": 2470149120.0,
+ "4": 2470149120.0,
+ "5": 2470149120.0,
+ "6": 2470149120.0,
+ "7": 2470149120.0,
+ "8": 2470149120.0,
+ "9": 2470149120.0,
+ "10": 2470149120.0,
+ "11": 2470149120.0,
+ "12": 2470149120.0,
+ "13": 2470149120.0,
+ "14": 2470149120.0,
+ "15": 2470149120.0,
+ "16": 2470149120.0,
+ "17": 2470149120.0,
+ "18": 2470149120.0,
+ "19": 2470149120.0,
+ "20": 2470149120.0,
+ "21": 2470149120.0,
+ "22": 2470149120.0,
+ "23": 2470149120.0,
+ "24": 2470149120.0,
+ "25": 2470149120.0,
+ "26": 2470149120.0,
+ "27": 2470149120.0,
+ "28": 2470149120.0,
+ "29": 2470149120.0,
+ "30": 2470149120.0,
+ "31": 2470149120.0,
+ "32": 2470149120.0,
+ "33": 2470149120.0,
+ "34": 2470149120.0,
+ "35": 2470149120.0,
+ "36": 2470149120.0,
+ "37": 2470149120.0,
+ "38": 2470149120.0,
+ "39": 2470149120.0,
+ "40": 2470149120.0,
+ "41": 2470149120.0,
+ "42": 2470149120.0,
+ "43": 2470149120.0,
+ "44": 2470149120.0,
+ "45": 2470149120.0,
+ "46": 2470149120.0,
+ "47": 2470149120.0,
+ "48": 2470149120.0,
+ "49": 2470149120.0,
+ "50": 2470149120.0,
+ "51": 2470149120.0,
+ "52": 2470149120.0,
+ "53": 2470149120.0,
+ "54": 2470149120.0,
+ "55": 2470149120.0,
+ "56": 2470149120.0,
+ "57": 2470149120.0,
+ "58": 2470149120.0,
+ "59": 2470149120.0,
+ "60": 2470149120.0,
+ "61": 2470149120.0,
+ "62": 2470149120.0,
+ "63": 2470149120.0,
+ "64": 2470149120.0,
+ "65": 2470149120.0,
+ "66": 2470149120.0,
+ "67": 2470149120.0,
+ "68": 2470149120.0,
+ "69": 2470149120.0,
+ "70": 2470149120.0,
+ "71": 2470149120.0,
+ "72": 2470149120.0,
+ "73": 2470149120.0,
+ "74": 2470149120.0,
+ "75": 2470149120.0,
+ "76": 2470149120.0,
+ "77": 2470149120.0,
+ "78": 2470149120.0,
+ "79": 2470149120.0,
+ "80": 2470149120.0,
+ "81": 2470149120.0,
+ "82": 2470149120.0,
+ "83": 2470149120.0,
+ "84": 2470149120.0,
+ "85": 2470149120.0,
+ "86": 2470149120.0,
+ "87": 2470149120.0,
+ "88": 2470149120.0,
+ "89": 2470149120.0,
+ "90": 2470149120.0,
+ "91": 2470149120.0,
+ "92": 2470149120.0,
+ "93": 2470149120.0,
+ "94": 2470149120.0,
+ "95": 2470149120.0,
+ "96": 2470149120.0,
+ "97": 2470149120.0,
+ "98": 2470149120.0,
+ "99": 2470149120.0,
+ "100": 2470149120.0
}
},
"iteration-time": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 3.33855,
- "3": 0.12562,
- "4": 0.10973,
- "5": 0.10864,
- "6": 0.10778,
- "7": 0.10885,
- "8": 0.10884,
- "9": 0.10877,
- "10": 0.10868,
- "11": 0.10997,
- "12": 0.10853,
- "13": 0.1086,
- "14": 0.10927,
- "15": 0.10879,
- "16": 0.10908,
- "17": 0.10873,
- "18": 0.10883,
- "19": 0.11028,
- "20": 0.11031,
- "21": 0.11086,
- "22": 0.10971,
- "23": 0.10987,
- "24": 0.1089,
- "25": 0.11118,
- "26": 0.10952,
- "27": 0.1165,
- "28": 0.11961,
- "29": 0.11977,
- "30": 0.11657,
- "31": 0.11728,
- "32": 0.11689,
- "33": 0.11642,
- "34": 0.11739,
- "35": 0.11665,
- "36": 0.11537,
- "37": 0.11552,
- "38": 0.11544,
- "39": 0.11538,
- "40": 0.11584,
- "41": 0.11597,
- "42": 0.11635,
- "43": 0.11593,
- "44": 0.11678,
- "45": 0.11608,
- "46": 0.11637,
- "47": 0.11572,
- "48": 0.11577,
- "49": 0.11481,
- "50": 0.11561,
- "51": 0.1213,
- "52": 0.10892,
- "53": 0.10742,
- "54": 0.10842,
- "55": 0.10806,
- "56": 0.10869,
- "57": 0.11057,
- "58": 0.108,
- "59": 0.10875,
- "60": 0.10969,
- "61": 0.1087,
- "62": 0.10795,
- "63": 0.1094,
- "64": 0.10922,
- "65": 0.11102,
- "66": 0.11016,
- "67": 0.10977,
- "68": 0.10988,
- "69": 0.11029,
- "70": 0.11078,
- "71": 0.11019,
- "72": 0.11727,
- "73": 0.11024,
- "74": 0.11054,
- "75": 0.10949,
- "76": 0.11384,
- "77": 0.11011,
- "78": 0.1101,
- "79": 0.10943,
- "80": 0.11059,
- "81": 0.11173,
- "82": 0.10987,
- "83": 0.1094,
- "84": 0.10956,
- "85": 0.11029,
- "86": 0.11179,
- "87": 0.10953,
- "88": 0.11045,
- "89": 0.1102,
- "90": 0.10897,
- "91": 0.11022,
- "92": 0.10965,
- "93": 0.11042,
- "94": 0.11158,
- "95": 0.11059,
- "96": 0.11046,
- "97": 0.11123,
- "98": 0.11055,
- "99": 0.11178,
- "100": 0.11266
+ "2": 9.34621,
+ "3": 0.13335,
+ "4": 0.11754,
+ "5": 0.11536,
+ "6": 0.11367,
+ "7": 0.11663,
+ "8": 0.11385,
+ "9": 0.11574,
+ "10": 0.11631,
+ "11": 0.11616,
+ "12": 0.11786,
+ "13": 0.11675,
+ "14": 0.12155,
+ "15": 0.11663,
+ "16": 0.11781,
+ "17": 0.11932,
+ "18": 0.11766,
+ "19": 0.11708,
+ "20": 0.11635,
+ "21": 0.11736,
+ "22": 0.11806,
+ "23": 0.11804,
+ "24": 0.1169,
+ "25": 0.11774,
+ "26": 0.11641,
+ "27": 0.11674,
+ "28": 0.1177,
+ "29": 0.11804,
+ "30": 0.11805,
+ "31": 0.11916,
+ "32": 0.11895,
+ "33": 0.11909,
+ "34": 0.1191,
+ "35": 0.11894,
+ "36": 0.11897,
+ "37": 0.11622,
+ "38": 0.11982,
+ "39": 0.1177,
+ "40": 0.1197,
+ "41": 0.11987,
+ "42": 0.11911,
+ "43": 0.11866,
+ "44": 0.11971,
+ "45": 0.11825,
+ "46": 0.1203,
+ "47": 0.11863,
+ "48": 0.1192,
+ "49": 0.11951,
+ "50": 0.11918,
+ "51": 0.13867,
+ "52": 0.12177,
+ "53": 0.12036,
+ "54": 0.1191,
+ "55": 0.11745,
+ "56": 0.11631,
+ "57": 0.11887,
+ "58": 0.11974,
+ "59": 0.12067,
+ "60": 0.1174,
+ "61": 0.12254,
+ "62": 0.11811,
+ "63": 0.11737,
+ "64": 0.11568,
+ "65": 0.11917,
+ "66": 0.1185,
+ "67": 0.1199,
+ "68": 0.11894,
+ "69": 0.11777,
+ "70": 0.11816,
+ "71": 0.11833,
+ "72": 0.11801,
+ "73": 0.11849,
+ "74": 0.11806,
+ "75": 0.11903,
+ "76": 0.11812,
+ "77": 0.11852,
+ "78": 0.11938,
+ "79": 0.1176,
+ "80": 0.11841,
+ "81": 0.11882,
+ "82": 0.11877,
+ "83": 0.11747,
+ "84": 0.11992,
+ "85": 0.12148,
+ "86": 0.11854,
+ "87": 0.1177,
+ "88": 0.1201,
+ "89": 0.11888,
+ "90": 0.1191,
+ "91": 0.11815,
+ "92": 0.1186,
+ "93": 0.1183,
+ "94": 0.11918,
+ "95": 0.13193,
+ "96": 0.13346,
+ "97": 0.12725,
+ "98": 0.13104,
+ "99": 0.12509,
+ "100": 0.12079
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/golden_values_dev_dgx_h100.json
index 82352c11781..4973ad2ae7c 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.84517,
- "2": 10.85349,
- "3": 10.8539,
- "4": 10.83825,
- "5": 10.87427,
- "6": 10.89307,
- "7": 10.85454,
- "8": 10.8626,
- "9": 10.86464,
- "10": 10.82906,
- "11": 10.88792,
- "12": 10.87099,
- "13": 10.87921,
- "14": 10.89078,
- "15": 10.81976,
- "16": 10.83158,
- "17": 10.79868,
- "18": 10.81672,
- "19": 10.81919,
- "20": 10.72701,
- "21": 10.70594,
- "22": 10.56367,
- "23": 10.72804,
- "24": 10.60832,
- "25": 10.5522,
- "26": 10.60853,
- "27": 10.62847,
- "28": 10.58306,
- "29": 10.60011,
- "30": 10.36616,
- "31": 10.12043,
- "32": 10.47685,
- "33": 10.46868,
- "34": 10.22316,
- "35": 10.27781,
- "36": 10.22892,
- "37": 10.35949,
- "38": 10.19369,
- "39": 10.41549,
- "40": 10.09758,
- "41": 10.1573,
- "42": 10.22398,
- "43": 9.83289,
- "44": 9.96912,
- "45": 9.84191,
- "46": 9.83041,
- "47": 10.15626,
- "48": 9.85486,
- "49": 9.54086,
- "50": 9.91248,
- "51": 9.85868,
- "52": 9.74284,
- "53": 10.06645,
- "54": 9.95167,
- "55": 9.88096,
- "56": 9.62626,
- "57": 9.47768,
- "58": 9.83346,
- "59": 9.58526,
- "60": 9.50125,
- "61": 9.69182,
- "62": 9.98853,
- "63": 9.38476,
- "64": 9.7803,
- "65": 8.94762,
- "66": 9.70856,
- "67": 9.36852,
- "68": 9.78439,
- "69": 9.79406,
- "70": 9.74241,
- "71": 9.61808,
- "72": 9.58428,
- "73": 9.5035,
- "74": 8.94221,
- "75": 9.42529,
- "76": 9.07408,
- "77": 10.06351,
- "78": 9.7208,
- "79": 9.37294,
- "80": 9.40396,
- "81": 9.48168,
- "82": 9.69778,
- "83": 9.30714,
- "84": 9.41712,
- "85": 9.61407,
- "86": 9.07615,
- "87": 9.59094,
- "88": 9.74641,
- "89": 9.59993,
- "90": 9.8142,
- "91": 9.33773,
- "92": 9.35373,
- "93": 9.07395,
- "94": 8.83173,
- "95": 9.51734,
- "96": 9.52415,
- "97": 9.30995,
- "98": 9.66805,
- "99": 8.88588,
- "100": 9.39538
+ "1": 10.96464,
+ "2": 10.95235,
+ "3": 10.95854,
+ "4": 10.95197,
+ "5": 10.95261,
+ "6": 10.95382,
+ "7": 10.95159,
+ "8": 10.95132,
+ "9": 10.9587,
+ "10": 10.94456,
+ "11": 10.93683,
+ "12": 10.94343,
+ "13": 10.94317,
+ "14": 10.94292,
+ "15": 10.91493,
+ "16": 10.90029,
+ "17": 10.91111,
+ "18": 10.90619,
+ "19": 10.90164,
+ "20": 10.80788,
+ "21": 10.79456,
+ "22": 10.80476,
+ "23": 10.78978,
+ "24": 10.775,
+ "25": 10.76168,
+ "26": 10.75516,
+ "27": 10.71957,
+ "28": 10.63914,
+ "29": 10.60849,
+ "30": 10.5865,
+ "31": 10.591,
+ "32": 10.57112,
+ "33": 10.54027,
+ "34": 10.49893,
+ "35": 10.50028,
+ "36": 10.48629,
+ "37": 10.44933,
+ "38": 10.45071,
+ "39": 10.4177,
+ "40": 10.39539,
+ "41": 10.37476,
+ "42": 10.36057,
+ "43": 10.33042,
+ "44": 10.31901,
+ "45": 10.31607,
+ "46": 10.2765,
+ "47": 10.26822,
+ "48": 10.21835,
+ "49": 10.21384,
+ "50": 10.22197,
+ "51": 10.22213,
+ "52": 10.16749,
+ "53": 10.17224,
+ "54": 10.13794,
+ "55": 10.10895,
+ "56": 10.13222,
+ "57": 10.12138,
+ "58": 10.12635,
+ "59": 10.07037,
+ "60": 10.09305,
+ "61": 10.04906,
+ "62": 10.0178,
+ "63": 10.08393,
+ "64": 10.03345,
+ "65": 10.0056,
+ "66": 10.04738,
+ "67": 10.01995,
+ "68": 9.987,
+ "69": 9.99636,
+ "70": 9.98278,
+ "71": 10.00544,
+ "72": 9.99249,
+ "73": 9.97617,
+ "74": 9.96757,
+ "75": 9.93705,
+ "76": 9.96637,
+ "77": 9.96324,
+ "78": 9.91485,
+ "79": 9.91547,
+ "80": 9.92964,
+ "81": 9.9558,
+ "82": 9.89448,
+ "83": 9.85961,
+ "84": 9.80303,
+ "85": 9.78429,
+ "86": 9.88633,
+ "87": 9.90913,
+ "88": 9.88156,
+ "89": 9.82172,
+ "90": 9.82086,
+ "91": 9.82114,
+ "92": 9.81672,
+ "93": 9.74867,
+ "94": 9.82366,
+ "95": 9.81096,
+ "96": 9.79717,
+ "97": 9.74553,
+ "98": 9.7656,
+ "99": 9.82207,
+ "100": 9.70437
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1655.0,
- "2": 1697.0,
- "3": 1724.0,
- "4": 1720.0,
- "5": 1803.0,
- "6": 1772.0,
- "7": 1811.0,
- "8": 1766.0,
- "9": 1750.0,
- "10": 1413.0,
- "11": 1861.0,
- "12": 1650.0,
- "13": 1895.0,
- "14": 1662.0,
- "15": 1951.0,
- "16": 1998.0,
- "17": 1798.0,
- "18": 1687.0,
- "19": 1856.0,
- "20": 1561.0,
- "21": 1882.0,
- "22": 1652.0,
- "23": 2075.0,
- "24": 1606.0,
- "25": 1665.0,
- "26": 1686.0,
- "27": 1839.0,
- "28": 2053.0,
- "29": 1907.0,
- "30": 1893.0,
- "31": 1581.0,
- "32": 1791.0,
- "33": 2149.0,
- "34": 1872.0,
- "35": 2010.0,
- "36": 1799.0,
- "37": 2311.0,
- "38": 2221.0,
- "39": 2261.0,
- "40": 2188.0,
- "41": 2204.0,
- "42": 2300.0,
- "43": 2001.0,
- "44": 2119.0,
- "45": 2126.0,
- "46": 2374.0,
- "47": 2468.0,
- "48": 2405.0,
- "49": 2247.0,
- "50": 2250.0,
- "51": 2607.0,
- "52": 2618.0,
- "53": 2828.0,
- "54": 2730.0,
- "55": 2351.0,
- "56": 2753.0,
- "57": 2323.0,
- "58": 2809.0,
- "59": 2721.0,
- "60": 2440.0,
- "61": 2875.0,
- "62": 2726.0,
- "63": 2444.0,
- "64": 3001.0,
- "65": 2602.0,
- "66": 2981.0,
- "67": 2676.0,
- "68": 2623.0,
- "69": 2802.0,
- "70": 3234.0,
- "71": 2902.0,
- "72": 2337.0,
- "73": 2856.0,
- "74": 1903.0,
- "75": 2388.0,
- "76": 3118.0,
- "77": 3108.0,
- "78": 3122.0,
- "79": 2994.0,
- "80": 3186.0,
- "81": 3470.0,
- "82": 3164.0,
- "83": 2726.0,
- "84": 3214.0,
- "85": 3262.0,
- "86": 2602.0,
- "87": 3658.0,
- "88": 2906.0,
- "89": 3054.0,
- "90": 3018.0,
- "91": 2690.0,
- "92": 3106.0,
- "93": 2701.0,
- "94": 3263.0,
- "95": 3426.0,
- "96": 3405.0,
- "97": 3087.0,
- "98": 3510.0,
- "99": 3148.0,
- "100": 3204.0
+ "1": 1876.0,
+ "2": 1608.0,
+ "3": 1787.0,
+ "4": 1675.0,
+ "5": 1820.0,
+ "6": 1678.0,
+ "7": 1861.0,
+ "8": 1749.0,
+ "9": 1745.0,
+ "10": 1815.0,
+ "11": 1687.0,
+ "12": 1707.0,
+ "13": 1793.0,
+ "14": 1934.0,
+ "15": 1596.0,
+ "16": 1789.0,
+ "17": 1804.0,
+ "18": 1887.0,
+ "19": 1732.0,
+ "20": 1743.0,
+ "21": 1701.0,
+ "22": 1737.0,
+ "23": 1763.0,
+ "24": 1893.0,
+ "25": 1668.0,
+ "26": 1818.0,
+ "27": 1714.0,
+ "28": 1802.0,
+ "29": 1882.0,
+ "30": 1794.0,
+ "31": 2054.0,
+ "32": 1856.0,
+ "33": 1964.0,
+ "34": 1906.0,
+ "35": 2093.0,
+ "36": 2092.0,
+ "37": 2153.0,
+ "38": 2118.0,
+ "39": 2128.0,
+ "40": 2240.0,
+ "41": 2216.0,
+ "42": 2011.0,
+ "43": 2345.0,
+ "44": 2251.0,
+ "45": 2452.0,
+ "46": 2385.0,
+ "47": 2315.0,
+ "48": 2605.0,
+ "49": 2784.0,
+ "50": 2452.0,
+ "51": 2363.0,
+ "52": 2745.0,
+ "53": 2553.0,
+ "54": 2662.0,
+ "55": 2488.0,
+ "56": 2667.0,
+ "57": 2231.0,
+ "58": 3620.0,
+ "59": 2972.0,
+ "60": 2910.0,
+ "61": 2690.0,
+ "62": 3078.0,
+ "63": 3201.0,
+ "64": 3564.0,
+ "65": 2704.0,
+ "66": 3007.0,
+ "67": 3658.0,
+ "68": 3419.0,
+ "69": 2956.0,
+ "70": 3369.0,
+ "71": 3228.0,
+ "72": 2840.0,
+ "73": 3343.0,
+ "74": 3291.0,
+ "75": 3072.0,
+ "76": 3254.0,
+ "77": 3602.0,
+ "78": 3210.0,
+ "79": 3275.0,
+ "80": 3018.0,
+ "81": 3339.0,
+ "82": 3037.0,
+ "83": 3045.0,
+ "84": 2995.0,
+ "85": 2769.0,
+ "86": 3150.0,
+ "87": 2842.0,
+ "88": 3058.0,
+ "89": 3194.0,
+ "90": 3657.0,
+ "91": 2888.0,
+ "92": 2904.0,
+ "93": 3116.0,
+ "94": 2942.0,
+ "95": 3370.0,
+ "96": 3525.0,
+ "97": 3443.0,
+ "98": 3405.0,
+ "99": 3124.0,
+ "100": 3034.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 551278080.0,
- "2": 551278080.0,
- "3": 551278080.0,
- "4": 551278080.0,
- "5": 551278080.0,
- "6": 551278080.0,
- "7": 551278080.0,
- "8": 551278080.0,
- "9": 551278080.0,
- "10": 551278080.0,
- "11": 551278080.0,
- "12": 551278080.0,
- "13": 551278080.0,
- "14": 551278080.0,
- "15": 551278080.0,
- "16": 551278080.0,
- "17": 551278080.0,
- "18": 551278080.0,
- "19": 551278080.0,
- "20": 551278080.0,
- "21": 551278080.0,
- "22": 551278080.0,
- "23": 551278080.0,
- "24": 551278080.0,
- "25": 551278080.0,
- "26": 551278080.0,
- "27": 551278080.0,
- "28": 551278080.0,
- "29": 551278080.0,
- "30": 551278080.0,
- "31": 551278080.0,
- "32": 551278080.0,
- "33": 551278080.0,
- "34": 551278080.0,
- "35": 551278080.0,
- "36": 551278080.0,
- "37": 551278080.0,
- "38": 551278080.0,
- "39": 551278080.0,
- "40": 551278080.0,
- "41": 551278080.0,
- "42": 551278080.0,
- "43": 551278080.0,
- "44": 551278080.0,
- "45": 551278080.0,
- "46": 551278080.0,
- "47": 551278080.0,
- "48": 551278080.0,
- "49": 551278080.0,
- "50": 551278080.0,
- "51": 551278080.0,
- "52": 551278080.0,
- "53": 551278080.0,
- "54": 551278080.0,
- "55": 551278080.0,
- "56": 551278080.0,
- "57": 551278080.0,
- "58": 551278080.0,
- "59": 551278080.0,
- "60": 551278080.0,
- "61": 551278080.0,
- "62": 551278080.0,
- "63": 551278080.0,
- "64": 551278080.0,
- "65": 551278080.0,
- "66": 551278080.0,
- "67": 551278080.0,
- "68": 551278080.0,
- "69": 551278080.0,
- "70": 551278080.0,
- "71": 551278080.0,
- "72": 551278080.0,
- "73": 551278080.0,
- "74": 551278080.0,
- "75": 551278080.0,
- "76": 551278080.0,
- "77": 551278080.0,
- "78": 551278080.0,
- "79": 551278080.0,
- "80": 551278080.0,
- "81": 551278080.0,
- "82": 551278080.0,
- "83": 551278080.0,
- "84": 551278080.0,
- "85": 551278080.0,
- "86": 551278080.0,
- "87": 551278080.0,
- "88": 551278080.0,
- "89": 551278080.0,
- "90": 551278080.0,
- "91": 551278080.0,
- "92": 551278080.0,
- "93": 551278080.0,
- "94": 551278080.0,
- "95": 551278080.0,
- "96": 551278080.0,
- "97": 551278080.0,
- "98": 551278080.0,
- "99": 551278080.0,
- "100": 551278080.0
+ "1": 552191488.0,
+ "2": 552191488.0,
+ "3": 552191488.0,
+ "4": 552191488.0,
+ "5": 552191488.0,
+ "6": 552191488.0,
+ "7": 552191488.0,
+ "8": 552191488.0,
+ "9": 552191488.0,
+ "10": 552191488.0,
+ "11": 552191488.0,
+ "12": 552191488.0,
+ "13": 552191488.0,
+ "14": 552191488.0,
+ "15": 552191488.0,
+ "16": 552191488.0,
+ "17": 552191488.0,
+ "18": 552191488.0,
+ "19": 552191488.0,
+ "20": 552191488.0,
+ "21": 552191488.0,
+ "22": 552191488.0,
+ "23": 552191488.0,
+ "24": 552191488.0,
+ "25": 552191488.0,
+ "26": 552191488.0,
+ "27": 552191488.0,
+ "28": 552191488.0,
+ "29": 552191488.0,
+ "30": 552191488.0,
+ "31": 552191488.0,
+ "32": 552191488.0,
+ "33": 552191488.0,
+ "34": 552191488.0,
+ "35": 552191488.0,
+ "36": 552191488.0,
+ "37": 552191488.0,
+ "38": 552191488.0,
+ "39": 552191488.0,
+ "40": 552191488.0,
+ "41": 552191488.0,
+ "42": 552191488.0,
+ "43": 552191488.0,
+ "44": 552191488.0,
+ "45": 552191488.0,
+ "46": 552191488.0,
+ "47": 552191488.0,
+ "48": 552191488.0,
+ "49": 552191488.0,
+ "50": 552191488.0,
+ "51": 552191488.0,
+ "52": 552191488.0,
+ "53": 552191488.0,
+ "54": 552191488.0,
+ "55": 553108992.0,
+ "56": 552191488.0,
+ "57": 552191488.0,
+ "58": 552191488.0,
+ "59": 552191488.0,
+ "60": 552191488.0,
+ "61": 552191488.0,
+ "62": 553108992.0,
+ "63": 552191488.0,
+ "64": 552191488.0,
+ "65": 552191488.0,
+ "66": 552191488.0,
+ "67": 552191488.0,
+ "68": 552191488.0,
+ "69": 552191488.0,
+ "70": 552191488.0,
+ "71": 552191488.0,
+ "72": 552191488.0,
+ "73": 552191488.0,
+ "74": 552191488.0,
+ "75": 552191488.0,
+ "76": 552191488.0,
+ "77": 552191488.0,
+ "78": 552191488.0,
+ "79": 552191488.0,
+ "80": 552191488.0,
+ "81": 552191488.0,
+ "82": 552191488.0,
+ "83": 552191488.0,
+ "84": 553108992.0,
+ "85": 552191488.0,
+ "86": 552191488.0,
+ "87": 552191488.0,
+ "88": 552191488.0,
+ "89": 553108992.0,
+ "90": 552191488.0,
+ "91": 552191488.0,
+ "92": 552191488.0,
+ "93": 552191488.0,
+ "94": 552191488.0,
+ "95": 552191488.0,
+ "96": 553108992.0,
+ "97": 553108992.0,
+ "98": 552191488.0,
+ "99": 552191488.0,
+ "100": 552191488.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 2289441792.0,
- "2": 2432405504.0,
- "3": 2432405504.0,
- "4": 2432405504.0,
- "5": 2432405504.0,
- "6": 2432405504.0,
- "7": 2432405504.0,
- "8": 2432405504.0,
- "9": 2432405504.0,
- "10": 2432405504.0,
- "11": 2432405504.0,
- "12": 2432405504.0,
- "13": 2432405504.0,
- "14": 2432405504.0,
- "15": 2432405504.0,
- "16": 2432405504.0,
- "17": 2432405504.0,
- "18": 2432405504.0,
- "19": 2432405504.0,
- "20": 2432405504.0,
- "21": 2432405504.0,
- "22": 2432405504.0,
- "23": 2432405504.0,
- "24": 2432405504.0,
- "25": 2432405504.0,
- "26": 2432405504.0,
- "27": 2432405504.0,
- "28": 2432405504.0,
- "29": 2432405504.0,
- "30": 2432405504.0,
- "31": 2432405504.0,
- "32": 2432405504.0,
- "33": 2432405504.0,
- "34": 2432405504.0,
- "35": 2432405504.0,
- "36": 2432405504.0,
- "37": 2432405504.0,
- "38": 2432405504.0,
- "39": 2432405504.0,
- "40": 2432405504.0,
- "41": 2432405504.0,
- "42": 2432405504.0,
- "43": 2432405504.0,
- "44": 2432405504.0,
- "45": 2432405504.0,
- "46": 2432405504.0,
- "47": 2432405504.0,
- "48": 2432405504.0,
- "49": 2432405504.0,
- "50": 2432405504.0,
- "51": 2432405504.0,
- "52": 2432405504.0,
- "53": 2432405504.0,
- "54": 2432405504.0,
- "55": 2432405504.0,
- "56": 2432405504.0,
- "57": 2432405504.0,
- "58": 2432405504.0,
- "59": 2432405504.0,
- "60": 2432405504.0,
- "61": 2432405504.0,
- "62": 2432405504.0,
- "63": 2432405504.0,
- "64": 2432405504.0,
- "65": 2432405504.0,
- "66": 2432405504.0,
- "67": 2432405504.0,
- "68": 2432405504.0,
- "69": 2432405504.0,
- "70": 2432405504.0,
- "71": 2432405504.0,
- "72": 2432405504.0,
- "73": 2432405504.0,
- "74": 2432405504.0,
- "75": 2432405504.0,
- "76": 2432405504.0,
- "77": 2432405504.0,
- "78": 2432405504.0,
- "79": 2432405504.0,
- "80": 2432405504.0,
- "81": 2432405504.0,
- "82": 2432405504.0,
- "83": 2432405504.0,
- "84": 2432405504.0,
- "85": 2432405504.0,
- "86": 2432405504.0,
- "87": 2432405504.0,
- "88": 2432405504.0,
- "89": 2432405504.0,
- "90": 2432405504.0,
- "91": 2432405504.0,
- "92": 2432405504.0,
- "93": 2432405504.0,
- "94": 2432405504.0,
- "95": 2432405504.0,
- "96": 2432405504.0,
- "97": 2432405504.0,
- "98": 2432405504.0,
- "99": 2432405504.0,
- "100": 2432405504.0
+ "1": 2290489856.0,
+ "2": 2433317888.0,
+ "3": 2433317888.0,
+ "4": 2433317888.0,
+ "5": 2433317888.0,
+ "6": 2433317888.0,
+ "7": 2433317888.0,
+ "8": 2433317888.0,
+ "9": 2433317888.0,
+ "10": 2433317888.0,
+ "11": 2433317888.0,
+ "12": 2433317888.0,
+ "13": 2433317888.0,
+ "14": 2433317888.0,
+ "15": 2433317888.0,
+ "16": 2433317888.0,
+ "17": 2433317888.0,
+ "18": 2433317888.0,
+ "19": 2433317888.0,
+ "20": 2433317888.0,
+ "21": 2433317888.0,
+ "22": 2433317888.0,
+ "23": 2433317888.0,
+ "24": 2433317888.0,
+ "25": 2433317888.0,
+ "26": 2433317888.0,
+ "27": 2433317888.0,
+ "28": 2433317888.0,
+ "29": 2433317888.0,
+ "30": 2433317888.0,
+ "31": 2433317888.0,
+ "32": 2433317888.0,
+ "33": 2433317888.0,
+ "34": 2433317888.0,
+ "35": 2433317888.0,
+ "36": 2433317888.0,
+ "37": 2433317888.0,
+ "38": 2433317888.0,
+ "39": 2433317888.0,
+ "40": 2433317888.0,
+ "41": 2433317888.0,
+ "42": 2433317888.0,
+ "43": 2433317888.0,
+ "44": 2433317888.0,
+ "45": 2433317888.0,
+ "46": 2433317888.0,
+ "47": 2433317888.0,
+ "48": 2433317888.0,
+ "49": 2433317888.0,
+ "50": 2433317888.0,
+ "51": 2434235392.0,
+ "52": 2434235392.0,
+ "53": 2434235392.0,
+ "54": 2434235392.0,
+ "55": 2434235392.0,
+ "56": 2434235392.0,
+ "57": 2434235392.0,
+ "58": 2434235392.0,
+ "59": 2434235392.0,
+ "60": 2434235392.0,
+ "61": 2434235392.0,
+ "62": 2434235392.0,
+ "63": 2434235392.0,
+ "64": 2434235392.0,
+ "65": 2434235392.0,
+ "66": 2434235392.0,
+ "67": 2434235392.0,
+ "68": 2434235392.0,
+ "69": 2434235392.0,
+ "70": 2434235392.0,
+ "71": 2434235392.0,
+ "72": 2434235392.0,
+ "73": 2434235392.0,
+ "74": 2434235392.0,
+ "75": 2434235392.0,
+ "76": 2434235392.0,
+ "77": 2434235392.0,
+ "78": 2434235392.0,
+ "79": 2434235392.0,
+ "80": 2434235392.0,
+ "81": 2434235392.0,
+ "82": 2434235392.0,
+ "83": 2434235392.0,
+ "84": 2434235392.0,
+ "85": 2434235392.0,
+ "86": 2434235392.0,
+ "87": 2434235392.0,
+ "88": 2434235392.0,
+ "89": 2434235392.0,
+ "90": 2434235392.0,
+ "91": 2434235392.0,
+ "92": 2434235392.0,
+ "93": 2434235392.0,
+ "94": 2434235392.0,
+ "95": 2434235392.0,
+ "96": 2434235392.0,
+ "97": 2434235392.0,
+ "98": 2434235392.0,
+ "99": 2434235392.0,
+ "100": 2434235392.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 12.06542,
- "2": 0.1206,
- "3": 0.10179,
- "4": 0.08257,
- "5": 0.08196,
- "6": 0.08184,
- "7": 0.08247,
- "8": 0.08147,
- "9": 0.08127,
- "10": 0.08228,
- "11": 0.0839,
- "12": 0.08236,
- "13": 0.08232,
- "14": 0.08218,
- "15": 0.08336,
- "16": 0.08213,
- "17": 0.08296,
- "18": 0.0816,
- "19": 0.08269,
- "20": 0.08138,
- "21": 0.08303,
- "22": 0.08243,
- "23": 0.08357,
- "24": 0.08151,
- "25": 0.08392,
- "26": 0.08247,
- "27": 0.08229,
- "28": 0.08279,
- "29": 0.08232,
- "30": 0.0824,
- "31": 0.08146,
- "32": 0.08912,
- "33": 0.08386,
- "34": 0.08198,
- "35": 0.08188,
- "36": 0.08394,
- "37": 0.08154,
- "38": 0.08111,
- "39": 0.08175,
- "40": 0.08143,
- "41": 0.08312,
- "42": 0.08219,
- "43": 0.08218,
- "44": 0.08316,
- "45": 0.08162,
- "46": 0.08265,
- "47": 0.08169,
- "48": 0.08346,
- "49": 0.08176,
- "50": 0.08213,
- "51": 0.09096,
- "52": 0.08501,
- "53": 0.08473,
- "54": 0.08165,
- "55": 0.08129,
- "56": 0.08244,
- "57": 0.08158,
- "58": 0.08104,
- "59": 0.08185,
- "60": 0.0834,
- "61": 0.08139,
- "62": 0.08134,
- "63": 0.086,
- "64": 0.08155,
- "65": 0.08326,
- "66": 0.08135,
- "67": 0.08434,
- "68": 0.0817,
- "69": 0.08297,
- "70": 0.08039,
- "71": 0.0801,
- "72": 0.07962,
- "73": 0.07979,
- "74": 0.08099,
- "75": 0.08004,
- "76": 0.07961,
- "77": 0.07959,
- "78": 0.08021,
- "79": 0.08102,
- "80": 0.07949,
- "81": 0.08018,
- "82": 0.08014,
- "83": 0.07929,
- "84": 0.07992,
- "85": 0.07982,
- "86": 0.08024,
- "87": 0.08054,
- "88": 0.08161,
- "89": 0.08084,
- "90": 0.08079,
- "91": 0.08239,
- "92": 0.08091,
- "93": 0.07966,
- "94": 0.08301,
- "95": 0.08124,
- "96": 0.08066,
- "97": 0.08098,
- "98": 0.08072,
- "99": 0.08164,
- "100": 0.08106
+ "1": "nan",
+ "2": 9.95426,
+ "3": 0.10465,
+ "4": 0.0838,
+ "5": 0.08579,
+ "6": 0.08156,
+ "7": 0.08079,
+ "8": 0.086,
+ "9": 0.08183,
+ "10": 0.08123,
+ "11": 0.08193,
+ "12": 0.07998,
+ "13": 0.07977,
+ "14": 0.08148,
+ "15": 0.08064,
+ "16": 0.07989,
+ "17": 0.08069,
+ "18": 0.08066,
+ "19": 0.08079,
+ "20": 0.0805,
+ "21": 0.08095,
+ "22": 0.08082,
+ "23": 0.08302,
+ "24": 0.08115,
+ "25": 0.08086,
+ "26": 0.08114,
+ "27": 0.08203,
+ "28": 0.08101,
+ "29": 0.08194,
+ "30": 0.08152,
+ "31": 0.08114,
+ "32": 0.08102,
+ "33": 0.08258,
+ "34": 0.0806,
+ "35": 0.0805,
+ "36": 0.08146,
+ "37": 0.08153,
+ "38": 0.08012,
+ "39": 0.0802,
+ "40": 0.08004,
+ "41": 0.08026,
+ "42": 0.07858,
+ "43": 0.08864,
+ "44": 0.08042,
+ "45": 0.0803,
+ "46": 0.08047,
+ "47": 0.08054,
+ "48": 0.08188,
+ "49": 0.08079,
+ "50": 0.08091,
+ "51": 0.12264,
+ "52": 0.08076,
+ "53": 0.08084,
+ "54": 0.0818,
+ "55": 0.08137,
+ "56": 0.08231,
+ "57": 0.08193,
+ "58": 0.08106,
+ "59": 0.08186,
+ "60": 0.08132,
+ "61": 0.08146,
+ "62": 0.08109,
+ "63": 0.08162,
+ "64": 0.08141,
+ "65": 0.08203,
+ "66": 0.08192,
+ "67": 0.08047,
+ "68": 0.08079,
+ "69": 0.08009,
+ "70": 0.08156,
+ "71": 0.08122,
+ "72": 0.08141,
+ "73": 0.08029,
+ "74": 0.08149,
+ "75": 0.08114,
+ "76": 0.08155,
+ "77": 0.08069,
+ "78": 0.08188,
+ "79": 0.08092,
+ "80": 0.08076,
+ "81": 0.08125,
+ "82": 0.08099,
+ "83": 0.08082,
+ "84": 0.08118,
+ "85": 0.08146,
+ "86": 0.08064,
+ "87": 0.08117,
+ "88": 0.08112,
+ "89": 0.08077,
+ "90": 0.08095,
+ "91": 0.08092,
+ "92": 0.08147,
+ "93": 0.08123,
+ "94": 0.08077,
+ "95": 0.08042,
+ "96": 0.08108,
+ "97": 0.08147,
+ "98": 0.08034,
+ "99": 0.08078,
+ "100": 0.08043
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/model_config.yaml
index ad7854aeacb..0a1f510fc8f 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
index af06fe06903..bc18bf3e2d8 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -58,4 +58,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular # Usually ckpt-resume, but as a WAR to #513 set to regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_untied/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_untied/model_config.yaml
index 035549f8fb6..a8194368c4e 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_untied/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_untied/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -55,4 +55,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular # Usually ckpt-resume, but as a WAR to #513 set to regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_tunable_overlap/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_tunable_overlap/model_config.yaml
index ef758e5639f..4dfc41a948f 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_tunable_overlap/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_tunable_overlap/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -53,4 +53,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular # Usually ckpt-resume, but as a WAR to #513 set to regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap/golden_values_dev_dgx_gb200.json
index 8a90b6fb7df..5e655485d52 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap/golden_values_dev_dgx_gb200.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.81848,
"2": 10.8198,
- "3": 10.83668,
- "4": 10.83525,
- "5": 10.84996,
- "6": 10.83445,
- "7": 10.82529,
- "8": 10.81514,
- "9": 10.87713,
+ "3": 10.83667,
+ "4": 10.8353,
+ "5": 10.85001,
+ "6": 10.83446,
+ "7": 10.8253,
+ "8": 10.81516,
+ "9": 10.87716,
"10": 10.88261,
- "11": 10.87195,
- "12": 10.8249,
- "13": 10.84823,
- "14": 10.81959,
- "15": 10.80596,
- "16": 10.80141,
- "17": 10.77143,
- "18": 10.78633,
- "19": 10.74566,
- "20": 10.62432,
- "21": 10.68067,
+ "11": 10.87191,
+ "12": 10.82486,
+ "13": 10.84827,
+ "14": 10.8196,
+ "15": 10.80595,
+ "16": 10.8014,
+ "17": 10.77144,
+ "18": 10.78637,
+ "19": 10.74572,
+ "20": 10.62431,
+ "21": 10.68065,
"22": 10.65086,
- "23": 10.76421,
- "24": 10.61849,
- "25": 10.46057,
- "26": 10.59622,
- "27": 10.54041,
+ "23": 10.76414,
+ "24": 10.61844,
+ "25": 10.46056,
+ "26": 10.59625,
+ "27": 10.54045,
"28": 10.44496,
- "29": 10.39552,
- "30": 10.40391,
- "31": 10.51272,
- "32": 10.32089,
- "33": 10.26353,
- "34": 10.46902,
- "35": 9.95972,
- "36": 10.11517,
- "37": 10.01099,
- "38": 10.38317,
- "39": 9.78588,
- "40": 10.10413,
- "41": 10.13151,
- "42": 10.02832,
- "43": 10.2098,
- "44": 10.07339,
- "45": 9.69361,
- "46": 9.99604,
- "47": 9.93464,
- "48": 9.67414,
- "49": 9.91775,
- "50": 9.93121
+ "29": 10.3955,
+ "30": 10.40393,
+ "31": 10.51269,
+ "32": 10.32093,
+ "33": 10.26351,
+ "34": 10.46901,
+ "35": 9.95973,
+ "36": 10.11518,
+ "37": 10.01096,
+ "38": 10.38318,
+ "39": 9.78592,
+ "40": 10.10416,
+ "41": 10.13149,
+ "42": 10.02829,
+ "43": 10.20978,
+ "44": 10.07337,
+ "45": 9.6936,
+ "46": 9.99607,
+ "47": 9.93467,
+ "48": 9.67415,
+ "49": 9.91776,
+ "50": 9.93114
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1118.0,
- "2": 1206.0,
- "3": 1308.0,
- "4": 1243.0,
- "5": 1256.0,
- "6": 1296.0,
- "7": 1259.0,
- "8": 1023.0,
- "9": 1295.0,
- "10": 1319.0,
- "11": 1282.0,
- "12": 1361.0,
- "13": 1336.0,
- "14": 1176.0,
- "15": 1188.0,
- "16": 1255.0,
- "17": 1182.0,
- "18": 1341.0,
- "19": 1043.0,
- "20": 1099.0,
- "21": 1248.0,
- "22": 1233.0,
- "23": 1369.0,
- "24": 1365.0,
- "25": 1073.0,
- "26": 1245.0,
- "27": 1211.0,
- "28": 1306.0,
- "29": 1317.0,
- "30": 1426.0,
- "31": 1476.0,
- "32": 1399.0,
- "33": 1444.0,
- "34": 1483.0,
- "35": 1242.0,
- "36": 1326.0,
- "37": 1447.0,
- "38": 1542.0,
- "39": 1342.0,
- "40": 1560.0,
- "41": 1611.0,
- "42": 1607.0,
- "43": 1651.0,
- "44": 1594.0,
- "45": 1499.0,
- "46": 1744.0,
- "47": 1571.0,
- "48": 1523.0,
- "49": 1629.0,
- "50": 1747.0
+ "1": 1076.0,
+ "2": 1208.0,
+ "3": 1258.0,
+ "4": 1277.0,
+ "5": 1246.0,
+ "6": 1250.0,
+ "7": 1209.0,
+ "8": 1001.0,
+ "9": 1360.0,
+ "10": 1325.0,
+ "11": 1245.0,
+ "12": 1358.0,
+ "13": 1303.0,
+ "14": 1131.0,
+ "15": 1174.0,
+ "16": 1254.0,
+ "17": 1207.0,
+ "18": 1343.0,
+ "19": 1091.0,
+ "20": 1062.0,
+ "21": 1233.0,
+ "22": 1257.0,
+ "23": 1376.0,
+ "24": 1275.0,
+ "25": 1134.0,
+ "26": 1179.0,
+ "27": 1217.0,
+ "28": 1307.0,
+ "29": 1321.0,
+ "30": 1372.0,
+ "31": 1475.0,
+ "32": 1394.0,
+ "33": 1486.0,
+ "34": 1522.0,
+ "35": 1129.0,
+ "36": 1305.0,
+ "37": 1428.0,
+ "38": 1593.0,
+ "39": 1392.0,
+ "40": 1482.0,
+ "41": 1626.0,
+ "42": 1594.0,
+ "43": 1737.0,
+ "44": 1511.0,
+ "45": 1489.0,
+ "46": 1774.0,
+ "47": 1605.0,
+ "48": 1596.0,
+ "49": 1712.0,
+ "50": 1740.0
}
},
"mem-allocated-bytes": {
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 3.1265,
- "3": 0.15779,
- "4": 0.14192,
- "5": 0.14446,
- "6": 0.14251,
- "7": 0.14375,
- "8": 0.1446,
- "9": 0.14351,
- "10": 0.14568,
- "11": 0.14477,
- "12": 0.14491,
- "13": 0.1447,
- "14": 0.14656,
- "15": 0.14652,
- "16": 0.14521,
- "17": 0.14638,
- "18": 0.14483,
- "19": 0.14549,
- "20": 0.14457,
- "21": 0.14306,
- "22": 0.14559,
- "23": 0.14596,
- "24": 0.14513,
- "25": 0.14367,
- "26": 0.14368,
- "27": 0.14398,
- "28": 0.14369,
- "29": 0.14435,
- "30": 0.14415,
- "31": 0.1433,
- "32": 0.14342,
- "33": 0.1441,
- "34": 0.14372,
- "35": 0.14431,
- "36": 0.1454,
- "37": 0.14634,
- "38": 0.14514,
- "39": 0.14529,
- "40": 0.14504,
- "41": 0.14496,
- "42": 0.14436,
- "43": 0.14492,
- "44": 0.14452,
- "45": 0.14629,
- "46": 0.14514,
- "47": 0.14578,
- "48": 0.1442,
- "49": 0.14396,
- "50": 0.14376
+ "2": 9.61562,
+ "3": 0.16844,
+ "4": 0.15723,
+ "5": 0.1525,
+ "6": 0.15368,
+ "7": 0.1564,
+ "8": 0.15395,
+ "9": 0.15394,
+ "10": 0.15187,
+ "11": 0.1507,
+ "12": 0.15595,
+ "13": 0.15908,
+ "14": 0.15006,
+ "15": 0.1512,
+ "16": 0.15414,
+ "17": 0.16741,
+ "18": 0.1517,
+ "19": 0.15049,
+ "20": 0.15743,
+ "21": 0.16405,
+ "22": 0.1515,
+ "23": 0.15504,
+ "24": 0.14877,
+ "25": 0.15039,
+ "26": 0.15018,
+ "27": 0.14901,
+ "28": 0.15124,
+ "29": 0.1496,
+ "30": 0.15239,
+ "31": 0.14986,
+ "32": 0.15144,
+ "33": 0.15057,
+ "34": 0.14938,
+ "35": 0.14965,
+ "36": 0.15083,
+ "37": 0.15057,
+ "38": 0.15276,
+ "39": 0.15348,
+ "40": 0.1583,
+ "41": 0.15028,
+ "42": 0.14947,
+ "43": 0.14843,
+ "44": 0.14972,
+ "45": 0.1515,
+ "46": 0.15337,
+ "47": 0.14819,
+ "48": 0.15122,
+ "49": 0.15663,
+ "50": 0.14862
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap/golden_values_dev_dgx_h100.json
index 399a2c50a8d..f7ffeea7f4b 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.83936,
- "2": 10.8442,
- "3": 10.86813,
- "4": 10.86022,
- "5": 10.87939,
- "6": 10.85969,
- "7": 10.86386,
- "8": 10.8444,
- "9": 10.88995,
- "10": 10.8926,
- "11": 10.89136,
- "12": 10.85312,
- "13": 10.87319,
- "14": 10.83805,
- "15": 10.83088,
- "16": 10.82011,
- "17": 10.79138,
- "18": 10.81055,
- "19": 10.77977,
- "20": 10.6635,
- "21": 10.69765,
- "22": 10.67421,
- "23": 10.77344,
- "24": 10.63919,
- "25": 10.50497,
- "26": 10.61911,
- "27": 10.56921,
- "28": 10.46859,
- "29": 10.41119,
- "30": 10.42916,
- "31": 10.52553,
- "32": 10.34942,
- "33": 10.2967,
- "34": 10.46909,
- "35": 9.99632,
- "36": 10.13945,
- "37": 10.0434,
- "38": 10.4139,
- "39": 9.80941,
- "40": 10.12495,
- "41": 10.14883,
- "42": 10.04042,
- "43": 10.22142,
- "44": 10.07348,
- "45": 9.71369,
- "46": 10.00449,
- "47": 9.94758,
- "48": 9.68856,
- "49": 9.93637,
- "50": 9.96042
+ "1": 10.96282,
+ "2": 10.95699,
+ "3": 10.95063,
+ "4": 10.95382,
+ "5": 10.95734,
+ "6": 10.9472,
+ "7": 10.95928,
+ "8": 10.95352,
+ "9": 10.94261,
+ "10": 10.95507,
+ "11": 10.94729,
+ "12": 10.94276,
+ "13": 10.93402,
+ "14": 10.9482,
+ "15": 10.91353,
+ "16": 10.88806,
+ "17": 10.89505,
+ "18": 10.89953,
+ "19": 10.89505,
+ "20": 10.82736,
+ "21": 10.80743,
+ "22": 10.80847,
+ "23": 10.79613,
+ "24": 10.76866,
+ "25": 10.7744,
+ "26": 10.75404,
+ "27": 10.70235,
+ "28": 10.62662,
+ "29": 10.61275,
+ "30": 10.59509,
+ "31": 10.58778,
+ "32": 10.55532,
+ "33": 10.53389,
+ "34": 10.51629,
+ "35": 10.50052,
+ "36": 10.48649,
+ "37": 10.4529,
+ "38": 10.42989,
+ "39": 10.40125,
+ "40": 10.39731,
+ "41": 10.39548,
+ "42": 10.34813,
+ "43": 10.33695,
+ "44": 10.2901,
+ "45": 10.30258,
+ "46": 10.28028,
+ "47": 10.26063,
+ "48": 10.24441,
+ "49": 10.20833,
+ "50": 10.18961
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1026.0,
- "2": 1184.0,
- "3": 1226.0,
- "4": 1248.0,
- "5": 1259.0,
- "6": 1421.0,
- "7": 1182.0,
- "8": 1036.0,
- "9": 1293.0,
- "10": 1319.0,
- "11": 1212.0,
- "12": 1373.0,
- "13": 1327.0,
- "14": 1121.0,
- "15": 1217.0,
- "16": 1163.0,
- "17": 1246.0,
- "18": 1280.0,
- "19": 1128.0,
- "20": 1019.0,
- "21": 1147.0,
- "22": 1156.0,
- "23": 1341.0,
- "24": 1312.0,
- "25": 1066.0,
- "26": 1138.0,
- "27": 1270.0,
- "28": 1260.0,
- "29": 1292.0,
- "30": 1532.0,
- "31": 1477.0,
- "32": 1460.0,
- "33": 1537.0,
- "34": 1513.0,
- "35": 1235.0,
- "36": 1316.0,
- "37": 1466.0,
- "38": 1564.0,
- "39": 1380.0,
- "40": 1513.0,
- "41": 1633.0,
- "42": 1509.0,
- "43": 1731.0,
- "44": 1636.0,
- "45": 1501.0,
- "46": 1884.0,
- "47": 1567.0,
- "48": 1631.0,
- "49": 1825.0,
- "50": 1639.0
+ "1": 1264.0,
+ "2": 1218.0,
+ "3": 1218.0,
+ "4": 1242.0,
+ "5": 1203.0,
+ "6": 1337.0,
+ "7": 1214.0,
+ "8": 1318.0,
+ "9": 1159.0,
+ "10": 1216.0,
+ "11": 1309.0,
+ "12": 1239.0,
+ "13": 1228.0,
+ "14": 1272.0,
+ "15": 1135.0,
+ "16": 1239.0,
+ "17": 1204.0,
+ "18": 1111.0,
+ "19": 1278.0,
+ "20": 1243.0,
+ "21": 1286.0,
+ "22": 1205.0,
+ "23": 1197.0,
+ "24": 1270.0,
+ "25": 1368.0,
+ "26": 1187.0,
+ "27": 1319.0,
+ "28": 1393.0,
+ "29": 1359.0,
+ "30": 1431.0,
+ "31": 1383.0,
+ "32": 1498.0,
+ "33": 1452.0,
+ "34": 1310.0,
+ "35": 1519.0,
+ "36": 1548.0,
+ "37": 1479.0,
+ "38": 1532.0,
+ "39": 1634.0,
+ "40": 1521.0,
+ "41": 1510.0,
+ "42": 1581.0,
+ "43": 1642.0,
+ "44": 1596.0,
+ "45": 1648.0,
+ "46": 1764.0,
+ "47": 1715.0,
+ "48": 1779.0,
+ "49": 1758.0,
+ "50": 1979.0
}
},
"mem-allocated-bytes": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.04337,
- "2": 0.16822,
- "3": 0.13237,
- "4": 0.10427,
- "5": 0.10319,
- "6": 0.10424,
- "7": 0.10225,
- "8": 0.10398,
- "9": 0.10251,
- "10": 0.10246,
- "11": 0.10345,
- "12": 0.103,
- "13": 0.10547,
- "14": 0.10352,
- "15": 0.10359,
- "16": 0.1027,
- "17": 0.10378,
- "18": 0.10313,
- "19": 0.10368,
- "20": 0.10223,
- "21": 0.10211,
- "22": 0.1031,
- "23": 0.10247,
- "24": 0.1027,
- "25": 0.10174,
- "26": 0.10084,
- "27": 0.10138,
- "28": 0.10076,
- "29": 0.10064,
- "30": 0.10061,
- "31": 0.10034,
- "32": 0.10099,
- "33": 0.10117,
- "34": 0.10033,
- "35": 0.10174,
- "36": 0.10259,
- "37": 0.1046,
- "38": 0.10281,
- "39": 0.10332,
- "40": 0.10619,
- "41": 0.10943,
- "42": 0.10864,
- "43": 0.10388,
- "44": 0.10366,
- "45": 0.10485,
- "46": 0.10446,
- "47": 0.10301,
- "48": 0.10412,
- "49": 0.10182,
- "50": 0.10428
+ "1": "nan",
+ "2": 6.9663,
+ "3": 0.13653,
+ "4": 0.10942,
+ "5": 0.10883,
+ "6": 0.11038,
+ "7": 0.10904,
+ "8": 0.11299,
+ "9": 0.10837,
+ "10": 0.11054,
+ "11": 0.1133,
+ "12": 0.10987,
+ "13": 0.11077,
+ "14": 0.11423,
+ "15": 0.10866,
+ "16": 0.10961,
+ "17": 0.10951,
+ "18": 0.10554,
+ "19": 0.10907,
+ "20": 0.11062,
+ "21": 0.10963,
+ "22": 0.11168,
+ "23": 0.1111,
+ "24": 0.11079,
+ "25": 0.10927,
+ "26": 0.11097,
+ "27": 0.10922,
+ "28": 0.11042,
+ "29": 0.10928,
+ "30": 0.10962,
+ "31": 0.1096,
+ "32": 0.11175,
+ "33": 0.10967,
+ "34": 0.10902,
+ "35": 0.10903,
+ "36": 0.10978,
+ "37": 0.11038,
+ "38": 0.11062,
+ "39": 0.10942,
+ "40": 0.11279,
+ "41": 0.11279,
+ "42": 0.11182,
+ "43": 0.11092,
+ "44": 0.1088,
+ "45": 0.1085,
+ "46": 0.10876,
+ "47": 0.11037,
+ "48": 0.10847,
+ "49": 0.10789,
+ "50": 0.10952
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap/model_config.yaml
index 06545179645..7dcdd335e8c 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -54,4 +54,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline/golden_values_dev_dgx_gb200.json
index da72109d85d..b59f2eab3b3 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline/golden_values_dev_dgx_gb200.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.93757,
"2": 10.92393,
- "3": 10.94318,
- "4": 10.93348,
+ "3": 10.94316,
+ "4": 10.93346,
"5": 10.93027,
- "6": 10.92214,
- "7": 10.9129,
- "8": 10.92494,
- "9": 10.94302,
+ "6": 10.92207,
+ "7": 10.91292,
+ "8": 10.92493,
+ "9": 10.94303,
"10": 10.92589,
- "11": 10.89715,
- "12": 10.91085,
- "13": 10.91359,
- "14": 10.90092,
- "15": 10.87211,
- "16": 10.86524,
- "17": 10.869,
- "18": 10.85374,
+ "11": 10.89718,
+ "12": 10.91088,
+ "13": 10.91361,
+ "14": 10.90096,
+ "15": 10.87207,
+ "16": 10.86525,
+ "17": 10.86902,
+ "18": 10.85373,
"19": 10.84295,
- "20": 10.76663,
- "21": 10.74374,
- "22": 10.67695,
- "23": 10.72701,
- "24": 10.66494,
- "25": 10.62546,
+ "20": 10.76659,
+ "21": 10.74373,
+ "22": 10.67694,
+ "23": 10.72697,
+ "24": 10.66498,
+ "25": 10.6255,
"26": 10.654,
- "27": 10.62035,
- "28": 10.56813,
+ "27": 10.62029,
+ "28": 10.56817,
"29": 10.56412,
- "30": 10.41005,
- "31": 10.21717,
+ "30": 10.41003,
+ "31": 10.21718,
"32": 10.46613,
- "33": 10.47136,
- "34": 10.26038,
- "35": 10.30272,
- "36": 10.264,
- "37": 10.35196,
- "38": 10.22183,
- "39": 10.38981,
- "40": 10.11089,
- "41": 10.13597,
- "42": 10.21619,
- "43": 9.89444,
- "44": 9.985,
- "45": 9.87317,
- "46": 9.86222,
+ "33": 10.47129,
+ "34": 10.26041,
+ "35": 10.30274,
+ "36": 10.26404,
+ "37": 10.35194,
+ "38": 10.22181,
+ "39": 10.38983,
+ "40": 10.11085,
+ "41": 10.13592,
+ "42": 10.21622,
+ "43": 9.89443,
+ "44": 9.98496,
+ "45": 9.8732,
+ "46": 9.86221,
"47": 10.13614,
- "48": 9.86196,
+ "48": 9.86198,
"49": 9.56912,
- "50": 9.91564
+ "50": 9.91565
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 22727242.0,
- "2": 22924896.0,
- "3": 22597216.0,
- "4": 23219716.0,
- "5": 22714012.0,
- "6": 23021178.0,
- "7": 22770808.0,
- "8": 22926716.0,
- "9": 22842500.0,
- "10": 22918960.0,
- "11": 22500834.0,
- "12": 22460340.0,
- "13": 22917536.0,
- "14": 22388990.0,
- "15": 22821224.0,
- "16": 22831266.0,
- "17": 22819108.0,
- "18": 22582264.0,
- "19": 22617384.0,
- "20": 22693436.0,
- "21": 22739352.0,
- "22": 22800104.0,
- "23": 22539998.0,
- "24": 22771512.0,
- "25": 22819132.0,
- "26": 22547588.0,
- "27": 22468844.0,
- "28": 22453516.0,
- "29": 22529320.0,
- "30": 22630996.0,
- "31": 22955520.0,
- "32": 22585756.0,
- "33": 22557744.0,
- "34": 22835696.0,
- "35": 22787828.0,
- "36": 22588412.0,
- "37": 22498040.0,
- "38": 22896082.0,
- "39": 22801992.0,
- "40": 22657536.0,
- "41": 22659220.0,
- "42": 22667844.0,
- "43": 22975904.0,
- "44": 22745960.0,
- "45": 22675400.0,
- "46": 22884844.0,
- "47": 22633716.0,
- "48": 22928608.0,
- "49": 22727282.0,
- "50": 22904808.0
+ "1": 22727352.0,
+ "2": 22924862.0,
+ "3": 22597228.0,
+ "4": 23219764.0,
+ "5": 22714064.0,
+ "6": 23021164.0,
+ "7": 22770838.0,
+ "8": 22926708.0,
+ "9": 22842636.0,
+ "10": 22918932.0,
+ "11": 22500860.0,
+ "12": 22460278.0,
+ "13": 22917508.0,
+ "14": 22388992.0,
+ "15": 22821332.0,
+ "16": 22831360.0,
+ "17": 22818988.0,
+ "18": 22582260.0,
+ "19": 22617512.0,
+ "20": 22693482.0,
+ "21": 22739376.0,
+ "22": 22800208.0,
+ "23": 22540044.0,
+ "24": 22771516.0,
+ "25": 22819082.0,
+ "26": 22547594.0,
+ "27": 22468792.0,
+ "28": 22453650.0,
+ "29": 22529422.0,
+ "30": 22631024.0,
+ "31": 22955468.0,
+ "32": 22585716.0,
+ "33": 22557726.0,
+ "34": 22835712.0,
+ "35": 22787748.0,
+ "36": 22588464.0,
+ "37": 22498104.0,
+ "38": 22895980.0,
+ "39": 22801960.0,
+ "40": 22657588.0,
+ "41": 22659166.0,
+ "42": 22667978.0,
+ "43": 22975850.0,
+ "44": 22745892.0,
+ "45": 22675412.0,
+ "46": 22884788.0,
+ "47": 22633592.0,
+ "48": 22928708.0,
+ "49": 22727260.0,
+ "50": 22904912.0
}
},
"mem-allocated-bytes": {
@@ -176,55 +176,55 @@
"step_interval": 1,
"values": {
"1": 2158025216.0,
- "2": 2416613888.0,
- "3": 2416613888.0,
- "4": 2416613888.0,
- "5": 2416613888.0,
- "6": 2416613888.0,
- "7": 2416613888.0,
- "8": 2416613888.0,
- "9": 2416613888.0,
- "10": 2416613888.0,
- "11": 2416613888.0,
- "12": 2416613888.0,
- "13": 2416613888.0,
- "14": 2416613888.0,
- "15": 2416613888.0,
- "16": 2416613888.0,
- "17": 2416613888.0,
- "18": 2416613888.0,
- "19": 2416613888.0,
- "20": 2416613888.0,
- "21": 2416613888.0,
- "22": 2416613888.0,
- "23": 2416613888.0,
- "24": 2416613888.0,
- "25": 2416613888.0,
- "26": 2416613888.0,
- "27": 2416613888.0,
- "28": 2416613888.0,
- "29": 2416613888.0,
- "30": 2416613888.0,
- "31": 2416613888.0,
- "32": 2416613888.0,
- "33": 2416613888.0,
- "34": 2416613888.0,
- "35": 2416613888.0,
- "36": 2416613888.0,
- "37": 2416613888.0,
- "38": 2416613888.0,
- "39": 2416613888.0,
- "40": 2416613888.0,
- "41": 2416613888.0,
- "42": 2416613888.0,
- "43": 2416613888.0,
- "44": 2416613888.0,
- "45": 2416613888.0,
- "46": 2416613888.0,
- "47": 2416613888.0,
- "48": 2416613888.0,
- "49": 2416613888.0,
- "50": 2416613888.0
+ "2": 2415566336.0,
+ "3": 2415566336.0,
+ "4": 2415566336.0,
+ "5": 2415566336.0,
+ "6": 2415566336.0,
+ "7": 2415566336.0,
+ "8": 2415566336.0,
+ "9": 2415566336.0,
+ "10": 2415566336.0,
+ "11": 2415566336.0,
+ "12": 2415566336.0,
+ "13": 2415566336.0,
+ "14": 2415566336.0,
+ "15": 2415566336.0,
+ "16": 2415566336.0,
+ "17": 2415566336.0,
+ "18": 2415566336.0,
+ "19": 2415566336.0,
+ "20": 2415566336.0,
+ "21": 2415566336.0,
+ "22": 2415566336.0,
+ "23": 2415566336.0,
+ "24": 2415566336.0,
+ "25": 2415566336.0,
+ "26": 2415566336.0,
+ "27": 2415566336.0,
+ "28": 2415566336.0,
+ "29": 2415566336.0,
+ "30": 2415566336.0,
+ "31": 2415566336.0,
+ "32": 2415566336.0,
+ "33": 2415566336.0,
+ "34": 2415566336.0,
+ "35": 2415566336.0,
+ "36": 2415566336.0,
+ "37": 2415566336.0,
+ "38": 2415566336.0,
+ "39": 2415566336.0,
+ "40": 2415566336.0,
+ "41": 2415566336.0,
+ "42": 2415566336.0,
+ "43": 2415566336.0,
+ "44": 2415566336.0,
+ "45": 2415566336.0,
+ "46": 2415566336.0,
+ "47": 2415566336.0,
+ "48": 2415566336.0,
+ "49": 2415566336.0,
+ "50": 2415566336.0
}
},
"iteration-time": {
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.44061,
- "3": 0.15375,
- "4": 0.14111,
- "5": 0.14053,
- "6": 0.14023,
- "7": 0.14152,
- "8": 0.14128,
- "9": 0.1417,
- "10": 0.14155,
- "11": 0.14076,
- "12": 0.1405,
- "13": 0.14129,
- "14": 0.14106,
- "15": 0.14101,
- "16": 0.14178,
- "17": 0.14173,
- "18": 0.14103,
- "19": 0.14094,
- "20": 0.14012,
- "21": 0.14153,
- "22": 0.14228,
- "23": 0.14118,
- "24": 0.14079,
- "25": 0.14034,
- "26": 0.14027,
- "27": 0.13947,
- "28": 0.13928,
- "29": 0.1398,
- "30": 0.14085,
- "31": 0.14179,
- "32": 0.13944,
- "33": 0.14174,
- "34": 0.1436,
- "35": 0.13902,
- "36": 0.13933,
- "37": 0.13922,
- "38": 0.13997,
- "39": 0.13881,
- "40": 0.13924,
- "41": 0.1392,
- "42": 0.14092,
- "43": 0.14136,
- "44": 0.14035,
- "45": 0.13841,
- "46": 0.1411,
- "47": 0.13878,
- "48": 0.14005,
- "49": 0.13925,
- "50": 0.13845
+ "2": 7.55862,
+ "3": 0.15424,
+ "4": 0.14447,
+ "5": 0.14487,
+ "6": 0.14649,
+ "7": 0.14905,
+ "8": 0.14737,
+ "9": 0.14546,
+ "10": 0.14639,
+ "11": 0.14694,
+ "12": 0.14764,
+ "13": 0.14783,
+ "14": 0.15046,
+ "15": 0.14727,
+ "16": 0.14715,
+ "17": 0.14912,
+ "18": 0.15267,
+ "19": 0.14754,
+ "20": 0.14944,
+ "21": 0.14739,
+ "22": 0.14765,
+ "23": 0.14985,
+ "24": 0.14828,
+ "25": 0.14802,
+ "26": 0.15122,
+ "27": 0.14967,
+ "28": 0.1501,
+ "29": 0.15013,
+ "30": 0.15048,
+ "31": 0.15048,
+ "32": 0.15018,
+ "33": 0.14712,
+ "34": 0.15007,
+ "35": 0.14904,
+ "36": 0.14828,
+ "37": 0.14709,
+ "38": 0.14853,
+ "39": 0.14847,
+ "40": 0.14939,
+ "41": 0.14997,
+ "42": 0.15,
+ "43": 0.15082,
+ "44": 0.15106,
+ "45": 0.14817,
+ "46": 0.14957,
+ "47": 0.14882,
+ "48": 0.22849,
+ "49": 0.14824,
+ "50": 0.14598
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline/golden_values_dev_dgx_h100.json
index 8a17375878f..71138d92a1e 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.88372,
- "2": 10.87208,
- "3": 10.8784,
- "4": 10.85806,
- "5": 10.88547,
- "6": 10.89556,
- "7": 10.88051,
- "8": 10.87687,
- "9": 10.868,
- "10": 10.86477,
- "11": 10.87779,
- "12": 10.8736,
- "13": 10.8617,
- "14": 10.88756,
- "15": 10.81334,
- "16": 10.8276,
- "17": 10.80766,
- "18": 10.81067,
- "19": 10.81127,
- "20": 10.71864,
- "21": 10.69427,
- "22": 10.58083,
- "23": 10.69548,
- "24": 10.60367,
- "25": 10.55396,
- "26": 10.61304,
- "27": 10.59026,
- "28": 10.54029,
- "29": 10.55687,
- "30": 10.36075,
- "31": 10.13943,
- "32": 10.44344,
- "33": 10.44459,
- "34": 10.21087,
- "35": 10.25855,
- "36": 10.22779,
- "37": 10.32843,
- "38": 10.18154,
- "39": 10.37655,
- "40": 10.0779,
- "41": 10.12618,
- "42": 10.19378,
- "43": 9.85406,
- "44": 9.94224,
- "45": 9.84493,
- "46": 9.831,
- "47": 10.13553,
- "48": 9.84455,
- "49": 9.5571,
- "50": 9.89982
+ "1": 10.89814,
+ "2": 10.89631,
+ "3": 10.90187,
+ "4": 10.89348,
+ "5": 10.89713,
+ "6": 10.88447,
+ "7": 10.90724,
+ "8": 10.88988,
+ "9": 10.90485,
+ "10": 10.89561,
+ "11": 10.88486,
+ "12": 10.87806,
+ "13": 10.89166,
+ "14": 10.8722,
+ "15": 10.85217,
+ "16": 10.8401,
+ "17": 10.85268,
+ "18": 10.83214,
+ "19": 10.83214,
+ "20": 10.7466,
+ "21": 10.74627,
+ "22": 10.72181,
+ "23": 10.70895,
+ "24": 10.68201,
+ "25": 10.67113,
+ "26": 10.66265,
+ "27": 10.60778,
+ "28": 10.55964,
+ "29": 10.52531,
+ "30": 10.49463,
+ "31": 10.48592,
+ "32": 10.47203,
+ "33": 10.42621,
+ "34": 10.41731,
+ "35": 10.40438,
+ "36": 10.37775,
+ "37": 10.34604,
+ "38": 10.35578,
+ "39": 10.31896,
+ "40": 10.30852,
+ "41": 10.28991,
+ "42": 10.24614,
+ "43": 10.2376,
+ "44": 10.21153,
+ "45": 10.22062,
+ "46": 10.17853,
+ "47": 10.17789,
+ "48": 10.12576,
+ "49": 10.1268,
+ "50": 10.12194
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 22726932.0,
- "2": 22924916.0,
- "3": 22597332.0,
- "4": 23219544.0,
- "5": 22713776.0,
- "6": 23021572.0,
- "7": 22771346.0,
- "8": 22926354.0,
- "9": 22842338.0,
- "10": 22918608.0,
- "11": 22500808.0,
- "12": 22460148.0,
- "13": 22917564.0,
- "14": 22389452.0,
- "15": 22821768.0,
- "16": 22831588.0,
- "17": 22819586.0,
- "18": 22582872.0,
- "19": 22618426.0,
- "20": 22693536.0,
- "21": 22739728.0,
- "22": 22800622.0,
- "23": 22539616.0,
- "24": 22771504.0,
- "25": 22819092.0,
- "26": 22547456.0,
- "27": 22468726.0,
- "28": 22453546.0,
- "29": 22529680.0,
- "30": 22630868.0,
- "31": 22955432.0,
- "32": 22585376.0,
- "33": 22557692.0,
- "34": 22835582.0,
- "35": 22788568.0,
- "36": 22588652.0,
- "37": 22497950.0,
- "38": 22895768.0,
- "39": 22801524.0,
- "40": 22657832.0,
- "41": 22659668.0,
- "42": 22667616.0,
- "43": 22975828.0,
- "44": 22746024.0,
- "45": 22674860.0,
- "46": 22884404.0,
- "47": 22633804.0,
- "48": 22928614.0,
- "49": 22728000.0,
- "50": 22904840.0
+ "1": 22962396.0,
+ "2": 22849532.0,
+ "3": 22709824.0,
+ "4": 22777498.0,
+ "5": 22782864.0,
+ "6": 22745206.0,
+ "7": 22871580.0,
+ "8": 22611072.0,
+ "9": 22758128.0,
+ "10": 22488868.0,
+ "11": 22753336.0,
+ "12": 22640716.0,
+ "13": 23323180.0,
+ "14": 22992022.0,
+ "15": 22717762.0,
+ "16": 22822348.0,
+ "17": 22928862.0,
+ "18": 22994820.0,
+ "19": 23085084.0,
+ "20": 22723054.0,
+ "21": 22917904.0,
+ "22": 22944892.0,
+ "23": 22628804.0,
+ "24": 22862054.0,
+ "25": 22636044.0,
+ "26": 23004428.0,
+ "27": 22801850.0,
+ "28": 22998952.0,
+ "29": 22979020.0,
+ "30": 22944072.0,
+ "31": 22908258.0,
+ "32": 22665236.0,
+ "33": 22740176.0,
+ "34": 23076388.0,
+ "35": 22749872.0,
+ "36": 22694514.0,
+ "37": 23104022.0,
+ "38": 22964652.0,
+ "39": 22985196.0,
+ "40": 22750676.0,
+ "41": 23064992.0,
+ "42": 22688700.0,
+ "43": 22987292.0,
+ "44": 22704620.0,
+ "45": 22847572.0,
+ "46": 22733980.0,
+ "47": 22850314.0,
+ "48": 22833360.0,
+ "49": 22888936.0,
+ "50": 22644684.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 687079936.0,
- "2": 687079936.0,
- "3": 687079936.0,
- "4": 687079936.0,
- "5": 687079936.0,
- "6": 687079936.0,
- "7": 687079936.0,
- "8": 687079936.0,
- "9": 687079936.0,
- "10": 687079936.0,
- "11": 687079936.0,
- "12": 687079936.0,
- "13": 687079936.0,
- "14": 687079936.0,
- "15": 687079936.0,
- "16": 687079936.0,
- "17": 687079936.0,
- "18": 687079936.0,
- "19": 687079936.0,
- "20": 687079936.0,
- "21": 687079936.0,
- "22": 687079936.0,
- "23": 687079936.0,
- "24": 687079936.0,
- "25": 687079936.0,
- "26": 687079936.0,
- "27": 687079936.0,
- "28": 687079936.0,
- "29": 687079936.0,
- "30": 687079936.0,
- "31": 687079936.0,
- "32": 687079936.0,
- "33": 687079936.0,
- "34": 687079936.0,
- "35": 687079936.0,
- "36": 687079936.0,
- "37": 687079936.0,
- "38": 687079936.0,
- "39": 687079936.0,
- "40": 687079936.0,
- "41": 687079936.0,
- "42": 687079936.0,
- "43": 687079936.0,
- "44": 687079936.0,
- "45": 687079936.0,
- "46": 687079936.0,
- "47": 687079936.0,
- "48": 687079936.0,
- "49": 687079936.0,
- "50": 687079936.0
+ "1": 688128512.0,
+ "2": 688128512.0,
+ "3": 688128512.0,
+ "4": 688128512.0,
+ "5": 688128512.0,
+ "6": 688128512.0,
+ "7": 688128512.0,
+ "8": 688128512.0,
+ "9": 688128512.0,
+ "10": 688128512.0,
+ "11": 688128512.0,
+ "12": 688128512.0,
+ "13": 688128512.0,
+ "14": 688128512.0,
+ "15": 688128512.0,
+ "16": 688128512.0,
+ "17": 688128512.0,
+ "18": 688128512.0,
+ "19": 688128512.0,
+ "20": 688128512.0,
+ "21": 688128512.0,
+ "22": 688128512.0,
+ "23": 688128512.0,
+ "24": 688128512.0,
+ "25": 688128512.0,
+ "26": 688128512.0,
+ "27": 688128512.0,
+ "28": 688128512.0,
+ "29": 688128512.0,
+ "30": 688128512.0,
+ "31": 688128512.0,
+ "32": 688128512.0,
+ "33": 688128512.0,
+ "34": 688128512.0,
+ "35": 688128512.0,
+ "36": 688128512.0,
+ "37": 688128512.0,
+ "38": 688128512.0,
+ "39": 688128512.0,
+ "40": 688128512.0,
+ "41": 688128512.0,
+ "42": 688128512.0,
+ "43": 688128512.0,
+ "44": 688128512.0,
+ "45": 688128512.0,
+ "46": 688128512.0,
+ "47": 688128512.0,
+ "48": 688128512.0,
+ "49": 688128512.0,
+ "50": 688128512.0
}
},
"mem-max-allocated-bytes": {
@@ -176,55 +176,55 @@
"step_interval": 1,
"values": {
"1": 2158025216.0,
- "2": 2414517760.0,
- "3": 2414517760.0,
- "4": 2414517760.0,
- "5": 2414517760.0,
- "6": 2414517760.0,
- "7": 2414517760.0,
- "8": 2414517760.0,
- "9": 2414517760.0,
- "10": 2414517760.0,
- "11": 2414517760.0,
- "12": 2414517760.0,
- "13": 2414517760.0,
- "14": 2414517760.0,
- "15": 2414517760.0,
- "16": 2414517760.0,
- "17": 2414517760.0,
- "18": 2414517760.0,
- "19": 2414517760.0,
- "20": 2414517760.0,
- "21": 2414517760.0,
- "22": 2414517760.0,
- "23": 2414517760.0,
- "24": 2414517760.0,
- "25": 2414517760.0,
- "26": 2414517760.0,
- "27": 2414517760.0,
- "28": 2414517760.0,
- "29": 2414517760.0,
- "30": 2414517760.0,
- "31": 2414517760.0,
- "32": 2414517760.0,
- "33": 2414517760.0,
- "34": 2414517760.0,
- "35": 2414517760.0,
- "36": 2414517760.0,
- "37": 2414517760.0,
- "38": 2414517760.0,
- "39": 2414517760.0,
- "40": 2414517760.0,
- "41": 2414517760.0,
- "42": 2414517760.0,
- "43": 2414517760.0,
- "44": 2414517760.0,
- "45": 2414517760.0,
- "46": 2414517760.0,
- "47": 2414517760.0,
- "48": 2414517760.0,
- "49": 2414517760.0,
- "50": 2414517760.0
+ "2": 2415566336.0,
+ "3": 2415566336.0,
+ "4": 2415566336.0,
+ "5": 2415566336.0,
+ "6": 2415566336.0,
+ "7": 2415566336.0,
+ "8": 2415566336.0,
+ "9": 2415566336.0,
+ "10": 2415566336.0,
+ "11": 2415566336.0,
+ "12": 2415566336.0,
+ "13": 2415566336.0,
+ "14": 2415566336.0,
+ "15": 2415566336.0,
+ "16": 2415566336.0,
+ "17": 2415566336.0,
+ "18": 2415566336.0,
+ "19": 2415566336.0,
+ "20": 2415566336.0,
+ "21": 2415566336.0,
+ "22": 2415566336.0,
+ "23": 2415566336.0,
+ "24": 2415566336.0,
+ "25": 2415566336.0,
+ "26": 2415566336.0,
+ "27": 2415566336.0,
+ "28": 2415566336.0,
+ "29": 2415566336.0,
+ "30": 2415566336.0,
+ "31": 2415566336.0,
+ "32": 2415566336.0,
+ "33": 2415566336.0,
+ "34": 2415566336.0,
+ "35": 2415566336.0,
+ "36": 2415566336.0,
+ "37": 2415566336.0,
+ "38": 2415566336.0,
+ "39": 2415566336.0,
+ "40": 2415566336.0,
+ "41": 2415566336.0,
+ "42": 2415566336.0,
+ "43": 2415566336.0,
+ "44": 2415566336.0,
+ "45": 2415566336.0,
+ "46": 2415566336.0,
+ "47": 2415566336.0,
+ "48": 2415566336.0,
+ "49": 2415566336.0,
+ "50": 2415566336.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 11.04447,
- "2": 0.15303,
- "3": 0.11363,
- "4": 0.09774,
- "5": 0.09666,
- "6": 0.0975,
- "7": 0.09718,
- "8": 0.09631,
- "9": 0.09764,
- "10": 0.0962,
- "11": 0.09842,
- "12": 0.09595,
- "13": 0.09748,
- "14": 0.09614,
- "15": 0.09539,
- "16": 0.09589,
- "17": 0.09791,
- "18": 0.0971,
- "19": 0.09598,
- "20": 0.09703,
- "21": 0.09477,
- "22": 0.09625,
- "23": 0.09521,
- "24": 0.09591,
- "25": 0.09662,
- "26": 0.09594,
- "27": 0.096,
- "28": 0.09633,
- "29": 0.09553,
- "30": 0.09789,
- "31": 0.09628,
- "32": 0.09629,
- "33": 0.09555,
- "34": 0.09528,
- "35": 0.09554,
- "36": 0.09515,
- "37": 0.09514,
- "38": 0.09534,
- "39": 0.0958,
- "40": 0.09495,
- "41": 0.09747,
- "42": 0.0951,
- "43": 0.09603,
- "44": 0.09547,
- "45": 0.09561,
- "46": 0.09761,
- "47": 0.09506,
- "48": 0.09637,
- "49": 0.09518,
- "50": 0.09512
+ "1": "nan",
+ "2": 6.17221,
+ "3": 0.11889,
+ "4": 0.10238,
+ "5": 0.10274,
+ "6": 0.09771,
+ "7": 0.09775,
+ "8": 0.09767,
+ "9": 0.09905,
+ "10": 0.09846,
+ "11": 0.09781,
+ "12": 0.09603,
+ "13": 0.09774,
+ "14": 0.09717,
+ "15": 0.09798,
+ "16": 0.0969,
+ "17": 0.0976,
+ "18": 0.09711,
+ "19": 0.09657,
+ "20": 0.09782,
+ "21": 0.09728,
+ "22": 0.09891,
+ "23": 0.098,
+ "24": 0.10019,
+ "25": 0.0979,
+ "26": 0.09813,
+ "27": 0.09822,
+ "28": 0.09772,
+ "29": 0.09913,
+ "30": 0.09916,
+ "31": 0.096,
+ "32": 0.09679,
+ "33": 0.09736,
+ "34": 0.09754,
+ "35": 0.09623,
+ "36": 0.09784,
+ "37": 0.0979,
+ "38": 0.0975,
+ "39": 0.0973,
+ "40": 0.09665,
+ "41": 0.0981,
+ "42": 0.09626,
+ "43": 0.09681,
+ "44": 0.09688,
+ "45": 0.09831,
+ "46": 0.09749,
+ "47": 0.09601,
+ "48": 0.09674,
+ "49": 0.09578,
+ "50": 0.09478
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline/model_config.yaml
index 8710e92a138..e068c864d10 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--decoder-last-pipeline-num-layers: 2
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split/golden_values_dev_dgx_h100.json
index bf7a46b3f3c..86f12bf9815 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.89824,
- "2": 10.90282,
- "3": 10.89982,
- "4": 10.86583,
- "5": 10.88993,
- "6": 10.9049,
- "7": 10.89182,
- "8": 10.90189,
- "9": 10.88632,
- "10": 10.88255,
- "11": 10.91544,
- "12": 10.90811,
- "13": 10.91696,
- "14": 10.92165,
- "15": 10.86969,
- "16": 10.8841,
- "17": 10.87056,
- "18": 10.88709,
- "19": 10.87706,
- "20": 10.84335,
- "21": 10.83631,
- "22": 10.76629,
- "23": 10.83029,
- "24": 10.79277,
- "25": 10.75377,
- "26": 10.78891,
- "27": 10.79166,
- "28": 10.74336,
- "29": 10.75965,
- "30": 10.62875,
- "31": 10.45418,
- "32": 10.68825,
- "33": 10.68615,
- "34": 10.52385,
- "35": 10.56066,
- "36": 10.53762,
- "37": 10.60286,
- "38": 10.46752,
- "39": 10.60804,
- "40": 10.36652,
- "41": 10.38788,
- "42": 10.45579,
- "43": 10.15865,
- "44": 10.24803,
- "45": 10.15385,
- "46": 10.13564,
- "47": 10.39205,
- "48": 10.1415,
- "49": 9.88025,
- "50": 10.18997
+ "1": 10.92679,
+ "2": 10.92501,
+ "3": 10.92521,
+ "4": 10.92422,
+ "5": 10.92547,
+ "6": 10.91033,
+ "7": 10.91228,
+ "8": 10.9229,
+ "9": 10.92921,
+ "10": 10.93096,
+ "11": 10.92107,
+ "12": 10.9162,
+ "13": 10.91442,
+ "14": 10.91553,
+ "15": 10.91878,
+ "16": 10.90576,
+ "17": 10.91144,
+ "18": 10.89352,
+ "19": 10.9045,
+ "20": 10.88092,
+ "21": 10.86586,
+ "22": 10.85397,
+ "23": 10.85621,
+ "24": 10.84645,
+ "25": 10.84779,
+ "26": 10.84328,
+ "27": 10.83813,
+ "28": 10.7638,
+ "29": 10.7529,
+ "30": 10.74291,
+ "31": 10.71652,
+ "32": 10.7293,
+ "33": 10.7083,
+ "34": 10.67989,
+ "35": 10.6798,
+ "36": 10.67755,
+ "37": 10.64942,
+ "38": 10.62141,
+ "39": 10.58148,
+ "40": 10.56724,
+ "41": 10.55467,
+ "42": 10.52441,
+ "43": 10.51692,
+ "44": 10.4963,
+ "45": 10.50698,
+ "46": 10.46293,
+ "47": 10.46403,
+ "48": 10.42071,
+ "49": 10.40805,
+ "50": 10.3905
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 22727178.0,
- "2": 22924812.0,
- "3": 22596704.0,
- "4": 23218766.0,
- "5": 22714208.0,
- "6": 23020316.0,
- "7": 22771086.0,
- "8": 22926440.0,
- "9": 22842352.0,
- "10": 22918036.0,
- "11": 22500516.0,
- "12": 22459304.0,
- "13": 22916284.0,
- "14": 22387532.0,
- "15": 22820856.0,
- "16": 22830090.0,
- "17": 22818880.0,
- "18": 22582012.0,
- "19": 22616784.0,
- "20": 22693674.0,
- "21": 22739360.0,
- "22": 22799250.0,
- "23": 22538774.0,
- "24": 22770954.0,
- "25": 22818024.0,
- "26": 22547278.0,
- "27": 22468476.0,
- "28": 22452228.0,
- "29": 22527980.0,
- "30": 22630720.0,
- "31": 22954516.0,
- "32": 22584820.0,
- "33": 22557266.0,
- "34": 22834728.0,
- "35": 22787216.0,
- "36": 22588668.0,
- "37": 22496474.0,
- "38": 22895320.0,
- "39": 22800062.0,
- "40": 22657316.0,
- "41": 22658142.0,
- "42": 22666692.0,
- "43": 22974950.0,
- "44": 22745468.0,
- "45": 22674868.0,
- "46": 22883238.0,
- "47": 22632908.0,
- "48": 22927884.0,
- "49": 22727252.0,
- "50": 22903748.0
+ "1": 22961676.0,
+ "2": 22849612.0,
+ "3": 22708396.0,
+ "4": 22776908.0,
+ "5": 22783084.0,
+ "6": 22744200.0,
+ "7": 22870900.0,
+ "8": 22610356.0,
+ "9": 22758668.0,
+ "10": 22488532.0,
+ "11": 22754076.0,
+ "12": 22641508.0,
+ "13": 23321640.0,
+ "14": 22991398.0,
+ "15": 22717960.0,
+ "16": 22822158.0,
+ "17": 22928176.0,
+ "18": 22993726.0,
+ "19": 23083638.0,
+ "20": 22722610.0,
+ "21": 22916936.0,
+ "22": 22944384.0,
+ "23": 22628000.0,
+ "24": 22861130.0,
+ "25": 22635252.0,
+ "26": 23004356.0,
+ "27": 22800372.0,
+ "28": 22998336.0,
+ "29": 22978000.0,
+ "30": 22943332.0,
+ "31": 22906864.0,
+ "32": 22665400.0,
+ "33": 22740010.0,
+ "34": 23074272.0,
+ "35": 22749910.0,
+ "36": 22694848.0,
+ "37": 23102524.0,
+ "38": 22963918.0,
+ "39": 22983688.0,
+ "40": 22748492.0,
+ "41": 23064376.0,
+ "42": 22687858.0,
+ "43": 22986648.0,
+ "44": 22703512.0,
+ "45": 22845966.0,
+ "46": 22732692.0,
+ "47": 22848302.0,
+ "48": 22832060.0,
+ "49": 22887408.0,
+ "50": 22643760.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 640822784.0,
- "2": 640822784.0,
- "3": 640822784.0,
- "4": 640822784.0,
- "5": 640822784.0,
- "6": 640822784.0,
- "7": 640822784.0,
- "8": 640822784.0,
- "9": 640822784.0,
- "10": 640822784.0,
- "11": 640822784.0,
- "12": 640822784.0,
- "13": 640822784.0,
- "14": 640822784.0,
- "15": 640822784.0,
- "16": 640822784.0,
- "17": 640822784.0,
- "18": 640822784.0,
- "19": 640822784.0,
- "20": 640822784.0,
- "21": 640822784.0,
- "22": 640822784.0,
- "23": 640822784.0,
- "24": 640822784.0,
- "25": 640822784.0,
- "26": 640822784.0,
- "27": 640822784.0,
- "28": 640822784.0,
- "29": 640822784.0,
- "30": 641740288.0,
- "31": 640822784.0,
- "32": 640822784.0,
- "33": 640822784.0,
- "34": 640822784.0,
- "35": 640822784.0,
- "36": 640822784.0,
- "37": 640822784.0,
- "38": 640822784.0,
- "39": 640822784.0,
- "40": 640822784.0,
- "41": 640822784.0,
- "42": 640822784.0,
- "43": 640822784.0,
- "44": 640822784.0,
- "45": 640822784.0,
- "46": 640822784.0,
- "47": 640822784.0,
- "48": 641740288.0,
- "49": 640822784.0,
- "50": 640822784.0
+ "1": 638725632.0,
+ "2": 638725632.0,
+ "3": 638725632.0,
+ "4": 638725632.0,
+ "5": 638725632.0,
+ "6": 638725632.0,
+ "7": 638725632.0,
+ "8": 638725632.0,
+ "9": 638725632.0,
+ "10": 638725632.0,
+ "11": 638725632.0,
+ "12": 638725632.0,
+ "13": 638725632.0,
+ "14": 638725632.0,
+ "15": 638725632.0,
+ "16": 638725632.0,
+ "17": 638725632.0,
+ "18": 638725632.0,
+ "19": 638725632.0,
+ "20": 638725632.0,
+ "21": 638725632.0,
+ "22": 638725632.0,
+ "23": 638725632.0,
+ "24": 638725632.0,
+ "25": 638725632.0,
+ "26": 638725632.0,
+ "27": 638725632.0,
+ "28": 638725632.0,
+ "29": 638725632.0,
+ "30": 638725632.0,
+ "31": 638725632.0,
+ "32": 638725632.0,
+ "33": 638725632.0,
+ "34": 638725632.0,
+ "35": 638725632.0,
+ "36": 638725632.0,
+ "37": 638725632.0,
+ "38": 638725632.0,
+ "39": 638725632.0,
+ "40": 638725632.0,
+ "41": 638725632.0,
+ "42": 638725632.0,
+ "43": 638725632.0,
+ "44": 638725632.0,
+ "45": 638725632.0,
+ "46": 638725632.0,
+ "47": 638725632.0,
+ "48": 638725632.0,
+ "49": 638725632.0,
+ "50": 638725632.0
}
},
"mem-max-allocated-bytes": {
@@ -176,55 +176,55 @@
"step_interval": 1,
"values": {
"1": 2610027008.0,
- "2": 2842349056.0,
- "3": 2842349056.0,
- "4": 2843266560.0,
- "5": 2843266560.0,
- "6": 2843266560.0,
- "7": 2843266560.0,
- "8": 2843266560.0,
- "9": 2843266560.0,
- "10": 2843266560.0,
- "11": 2843266560.0,
- "12": 2843266560.0,
- "13": 2843266560.0,
- "14": 2843266560.0,
- "15": 2843266560.0,
- "16": 2843266560.0,
- "17": 2843266560.0,
- "18": 2843266560.0,
- "19": 2843266560.0,
- "20": 2843266560.0,
- "21": 2843266560.0,
- "22": 2843266560.0,
- "23": 2843266560.0,
- "24": 2843266560.0,
- "25": 2843266560.0,
- "26": 2843266560.0,
- "27": 2843266560.0,
- "28": 2843266560.0,
- "29": 2843266560.0,
- "30": 2843266560.0,
- "31": 2843266560.0,
- "32": 2843266560.0,
- "33": 2843266560.0,
- "34": 2843266560.0,
- "35": 2843266560.0,
- "36": 2843266560.0,
- "37": 2843266560.0,
- "38": 2843266560.0,
- "39": 2843266560.0,
- "40": 2843266560.0,
- "41": 2843266560.0,
- "42": 2843266560.0,
- "43": 2843266560.0,
- "44": 2843266560.0,
- "45": 2843266560.0,
- "46": 2843266560.0,
- "47": 2843266560.0,
- "48": 2843266560.0,
- "49": 2843266560.0,
- "50": 2843266560.0
+ "2": 2840251904.0,
+ "3": 2841169408.0,
+ "4": 2841169408.0,
+ "5": 2841169408.0,
+ "6": 2841169408.0,
+ "7": 2841169408.0,
+ "8": 2841169408.0,
+ "9": 2841169408.0,
+ "10": 2841169408.0,
+ "11": 2841169408.0,
+ "12": 2841169408.0,
+ "13": 2841169408.0,
+ "14": 2841169408.0,
+ "15": 2841169408.0,
+ "16": 2841169408.0,
+ "17": 2841169408.0,
+ "18": 2841169408.0,
+ "19": 2841169408.0,
+ "20": 2841169408.0,
+ "21": 2841169408.0,
+ "22": 2841169408.0,
+ "23": 2841169408.0,
+ "24": 2841169408.0,
+ "25": 2841169408.0,
+ "26": 2841169408.0,
+ "27": 2841169408.0,
+ "28": 2841169408.0,
+ "29": 2841169408.0,
+ "30": 2841169408.0,
+ "31": 2841169408.0,
+ "32": 2841169408.0,
+ "33": 2841169408.0,
+ "34": 2841169408.0,
+ "35": 2841169408.0,
+ "36": 2841169408.0,
+ "37": 2841169408.0,
+ "38": 2841169408.0,
+ "39": 2841169408.0,
+ "40": 2841169408.0,
+ "41": 2841169408.0,
+ "42": 2841169408.0,
+ "43": 2841169408.0,
+ "44": 2841169408.0,
+ "45": 2841169408.0,
+ "46": 2841169408.0,
+ "47": 2841169408.0,
+ "48": 2841169408.0,
+ "49": 2841169408.0,
+ "50": 2841169408.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 11.63091,
- "2": 0.10057,
- "3": 0.08189,
- "4": 0.05797,
- "5": 0.05721,
- "6": 0.05698,
- "7": 0.05706,
- "8": 0.05717,
- "9": 0.05757,
- "10": 0.05769,
- "11": 0.05657,
- "12": 0.05708,
- "13": 0.05676,
- "14": 0.05712,
- "15": 0.05745,
- "16": 0.05704,
- "17": 0.05756,
- "18": 0.05699,
- "19": 0.05682,
- "20": 0.05715,
- "21": 0.0569,
- "22": 0.05766,
- "23": 0.0572,
- "24": 0.05719,
- "25": 0.05674,
- "26": 0.05685,
- "27": 0.05682,
- "28": 0.05657,
- "29": 0.0565,
- "30": 0.05693,
- "31": 0.05726,
- "32": 0.05673,
- "33": 0.05675,
- "34": 0.05664,
- "35": 0.05717,
- "36": 0.05653,
- "37": 0.05652,
- "38": 0.05671,
- "39": 0.05659,
- "40": 0.05731,
- "41": 0.05949,
- "42": 0.05669,
- "43": 0.05723,
- "44": 0.05695,
- "45": 0.05766,
- "46": 0.05736,
- "47": 0.05802,
- "48": 0.05662,
- "49": 0.05689,
- "50": 0.05838
+ "1": "nan",
+ "2": 6.61175,
+ "3": 0.07487,
+ "4": 0.05582,
+ "5": 0.05576,
+ "6": 0.05533,
+ "7": 0.05506,
+ "8": 0.05527,
+ "9": 0.0549,
+ "10": 0.05763,
+ "11": 0.05568,
+ "12": 0.05527,
+ "13": 0.05613,
+ "14": 0.05679,
+ "15": 0.05548,
+ "16": 0.05492,
+ "17": 0.05476,
+ "18": 0.05519,
+ "19": 0.05474,
+ "20": 0.0555,
+ "21": 0.05502,
+ "22": 0.05525,
+ "23": 0.0549,
+ "24": 0.05502,
+ "25": 0.05486,
+ "26": 0.05529,
+ "27": 0.05563,
+ "28": 0.05502,
+ "29": 0.05515,
+ "30": 0.05504,
+ "31": 0.05558,
+ "32": 0.05479,
+ "33": 0.05502,
+ "34": 0.05567,
+ "35": 0.05519,
+ "36": 0.05489,
+ "37": 0.05526,
+ "38": 0.05494,
+ "39": 0.05512,
+ "40": 0.05528,
+ "41": 0.05549,
+ "42": 0.05519,
+ "43": 0.05515,
+ "44": 0.0557,
+ "45": 0.05487,
+ "46": 0.055,
+ "47": 0.0548,
+ "48": 0.05523,
+ "49": 0.05535,
+ "50": 0.05557
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split/model_config.yaml
index d074a823ffd..b9e66c55bff 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -53,4 +53,6 @@ MODEL_ARGS:
--account-for-loss-in-pipeline-split: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_cp2_nondeterministic/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_cp2_nondeterministic/model_config.yaml
index 5394f9d0070..2f33dea359d 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_cp2_nondeterministic/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_cp2_nondeterministic/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: flash
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_cp2_nondeterministic/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_cp2_nondeterministic/model_config.yaml
index 4bd321b43da..0d803ff1aaf 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_cp2_nondeterministic/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_cp2_nondeterministic/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: flash
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: frozen-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_fsdp2_resume_torch_dist/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_fsdp2_resume_torch_dist/model_config.yaml
index 1229288b9be..9876606f2f7 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_fsdp2_resume_torch_dist/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_fsdp2_resume_torch_dist/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -43,7 +43,6 @@ MODEL_ARGS:
--untie-embeddings-and-output-weights: true
--deterministic-mode: true
--no-gradient-accumulation-fusion: true
- --no-async-tensor-model-parallel-allreduce: true
--attention-softmax-in-fp32: true
--use-checkpoint-opt_param-scheduler: true
--use-mcore-models: true
@@ -52,4 +51,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_gdn/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_gdn/model_config.yaml
index ee2c093e0ab..25bb4077a02 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_gdn/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_gdn/model_config.yaml
@@ -50,9 +50,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -79,4 +79,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_modelopt_distill_resume/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_modelopt_distill_resume/model_config.yaml
index 1171dfc454e..c75a5a81414 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_modelopt_distill_resume/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_modelopt_distill_resume/model_config.yaml
@@ -37,9 +37,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -66,4 +66,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_multi_dist_optimizer_instances/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_multi_dist_optimizer_instances/model_config.yaml
index 57bcdeb7ca6..e541610ae72 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_multi_dist_optimizer_instances/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_multi_dist_optimizer_instances/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -53,4 +53,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_cp2_nondeterministic/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_cp2_nondeterministic/model_config.yaml
index 3f427a04f9d..bd303906bd6 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_cp2_nondeterministic/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_cp2_nondeterministic/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: flash
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances/golden_values_dev_dgx_h100.json
index 13ad7566828..f5fd8427882 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.84466,
- "2": 10.84794,
- "3": 10.84923,
- "4": 10.8433,
- "5": 10.88246,
- "6": 10.8808,
- "7": 10.86574,
- "8": 10.85417,
- "9": 10.85542,
- "10": 10.81812,
- "11": 10.88726,
- "12": 10.86329,
- "13": 10.86656,
- "14": 10.884,
- "15": 10.8231,
- "16": 10.82809,
- "17": 10.79467,
- "18": 10.81466,
- "19": 10.80122,
- "20": 10.71614,
- "21": 10.69886,
- "22": 10.56738,
- "23": 10.71707,
- "24": 10.60503,
- "25": 10.55053,
- "26": 10.60941,
- "27": 10.62543,
- "28": 10.57767,
- "29": 10.59725,
- "30": 10.38488,
- "31": 10.15554,
- "32": 10.48231,
- "33": 10.4763,
- "34": 10.2393,
- "35": 10.29064,
- "36": 10.25146,
- "37": 10.35662,
- "38": 10.21142,
- "39": 10.42144,
- "40": 10.11569,
- "41": 10.16423,
- "42": 10.23644,
- "43": 9.86597,
- "44": 9.98146,
- "45": 9.86983,
- "46": 9.85349,
- "47": 10.16995,
- "48": 9.876,
- "49": 9.57237,
- "50": 9.92525,
- "51": 9.8709,
- "52": 9.7737,
- "53": 10.08149,
- "54": 9.97376,
- "55": 9.90036,
- "56": 9.64783,
- "57": 9.50136,
- "58": 9.85199,
- "59": 9.6034,
- "60": 9.50993,
- "61": 9.71315,
- "62": 9.99373,
- "63": 9.39358,
- "64": 9.78904,
- "65": 8.96358,
- "66": 9.71142,
- "67": 9.38175,
- "68": 9.79833,
- "69": 9.80889,
- "70": 9.75039,
- "71": 9.62004,
- "72": 9.59387,
- "73": 9.50631,
- "74": 8.94916,
- "75": 9.43188,
- "76": 9.08702,
- "77": 10.06886,
- "78": 9.73459,
- "79": 9.38325,
- "80": 9.41272,
- "81": 9.48499,
- "82": 9.70672,
- "83": 9.30939,
- "84": 9.42428,
- "85": 9.61991,
- "86": 9.07811,
- "87": 9.59541,
- "88": 9.75596,
- "89": 9.60274,
- "90": 9.82165,
- "91": 9.34268,
- "92": 9.35878,
- "93": 9.08116,
- "94": 8.83791,
- "95": 9.5238,
- "96": 9.53556,
- "97": 9.31807,
- "98": 9.68183,
- "99": 8.89422,
- "100": 9.40138
+ "1": 10.93132,
+ "2": 10.92661,
+ "3": 10.92492,
+ "4": 10.92081,
+ "5": 10.91909,
+ "6": 10.91583,
+ "7": 10.92398,
+ "8": 10.91668,
+ "9": 10.91511,
+ "10": 10.91953,
+ "11": 10.91089,
+ "12": 10.90468,
+ "13": 10.89957,
+ "14": 10.90081,
+ "15": 10.88314,
+ "16": 10.86333,
+ "17": 10.88212,
+ "18": 10.86579,
+ "19": 10.8688,
+ "20": 10.79009,
+ "21": 10.78749,
+ "22": 10.77487,
+ "23": 10.76339,
+ "24": 10.73373,
+ "25": 10.73323,
+ "26": 10.71703,
+ "27": 10.67915,
+ "28": 10.61679,
+ "29": 10.59026,
+ "30": 10.55955,
+ "31": 10.56384,
+ "32": 10.54875,
+ "33": 10.51301,
+ "34": 10.48527,
+ "35": 10.48192,
+ "36": 10.46066,
+ "37": 10.43615,
+ "38": 10.43666,
+ "39": 10.39487,
+ "40": 10.38845,
+ "41": 10.36652,
+ "42": 10.34954,
+ "43": 10.32453,
+ "44": 10.30276,
+ "45": 10.30963,
+ "46": 10.27269,
+ "47": 10.25873,
+ "48": 10.21288,
+ "49": 10.21687,
+ "50": 10.2101,
+ "51": 10.21225,
+ "52": 10.16485,
+ "53": 10.17138,
+ "54": 10.13369,
+ "55": 10.10841,
+ "56": 10.13285,
+ "57": 10.11976,
+ "58": 10.12939,
+ "59": 10.07404,
+ "60": 10.09745,
+ "61": 10.05118,
+ "62": 10.01669,
+ "63": 10.08732,
+ "64": 10.03654,
+ "65": 10.00961,
+ "66": 10.0475,
+ "67": 10.02252,
+ "68": 9.9873,
+ "69": 10.00132,
+ "70": 9.9863,
+ "71": 10.00761,
+ "72": 9.99373,
+ "73": 9.98421,
+ "74": 9.97134,
+ "75": 9.92975,
+ "76": 9.96821,
+ "77": 9.97253,
+ "78": 9.92117,
+ "79": 9.91774,
+ "80": 9.9356,
+ "81": 9.95704,
+ "82": 9.89748,
+ "83": 9.86594,
+ "84": 9.80667,
+ "85": 9.78948,
+ "86": 9.89884,
+ "87": 9.90957,
+ "88": 9.89145,
+ "89": 9.83321,
+ "90": 9.8267,
+ "91": 9.83688,
+ "92": 9.82594,
+ "93": 9.76052,
+ "94": 9.83255,
+ "95": 9.82136,
+ "96": 9.80914,
+ "97": 9.75581,
+ "98": 9.78098,
+ "99": 9.82253,
+ "100": 9.71473
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1713.0,
- "2": 1750.0,
- "3": 1744.0,
- "4": 1895.0,
- "5": 1839.0,
- "6": 1881.0,
- "7": 1850.0,
+ "1": 1749.0,
+ "2": 1763.0,
+ "3": 1797.0,
+ "4": 1864.0,
+ "5": 1778.0,
+ "6": 1808.0,
+ "7": 1884.0,
"8": 1743.0,
- "9": 1810.0,
- "10": 1452.0,
- "11": 1886.0,
- "12": 1752.0,
- "13": 1834.0,
- "14": 1774.0,
- "15": 1909.0,
- "16": 1803.0,
- "17": 1927.0,
- "18": 1765.0,
- "19": 1847.0,
- "20": 1707.0,
- "21": 1950.0,
- "22": 1794.0,
- "23": 1974.0,
- "24": 1676.0,
- "25": 1652.0,
- "26": 1774.0,
- "27": 1799.0,
- "28": 2135.0,
- "29": 2048.0,
- "30": 2032.0,
- "31": 1599.0,
- "32": 1929.0,
- "33": 2143.0,
- "34": 1874.0,
- "35": 1974.0,
- "36": 2011.0,
- "37": 2364.0,
- "38": 2199.0,
- "39": 2363.0,
- "40": 2239.0,
- "41": 2269.0,
- "42": 2228.0,
- "43": 1972.0,
- "44": 2070.0,
- "45": 2033.0,
- "46": 2357.0,
- "47": 2520.0,
- "48": 2316.0,
- "49": 2307.0,
- "50": 2302.0,
- "51": 2514.0,
- "52": 2430.0,
- "53": 2840.0,
- "54": 2677.0,
- "55": 2394.0,
- "56": 2601.0,
- "57": 2341.0,
- "58": 2837.0,
- "59": 2789.0,
- "60": 2425.0,
- "61": 2923.0,
- "62": 2591.0,
- "63": 2416.0,
- "64": 2937.0,
- "65": 2572.0,
- "66": 3008.0,
- "67": 2843.0,
- "68": 2761.0,
- "69": 2834.0,
- "70": 3108.0,
- "71": 2989.0,
- "72": 2316.0,
- "73": 2950.0,
- "74": 1899.0,
- "75": 2378.0,
- "76": 2962.0,
- "77": 3343.0,
- "78": 3183.0,
- "79": 2979.0,
- "80": 3209.0,
- "81": 3583.0,
- "82": 3160.0,
- "83": 2776.0,
- "84": 3242.0,
- "85": 3425.0,
- "86": 2720.0,
- "87": 3820.0,
- "88": 3050.0,
- "89": 3297.0,
- "90": 3069.0,
- "91": 2685.0,
- "92": 3061.0,
- "93": 2584.0,
- "94": 3338.0,
- "95": 3406.0,
- "96": 3389.0,
- "97": 3104.0,
- "98": 3583.0,
- "99": 3229.0,
- "100": 3225.0
+ "9": 1830.0,
+ "10": 1705.0,
+ "11": 1709.0,
+ "12": 1746.0,
+ "13": 1802.0,
+ "14": 1941.0,
+ "15": 1696.0,
+ "16": 1812.0,
+ "17": 1779.0,
+ "18": 1801.0,
+ "19": 1797.0,
+ "20": 1865.0,
+ "21": 1882.0,
+ "22": 1818.0,
+ "23": 1856.0,
+ "24": 1829.0,
+ "25": 1707.0,
+ "26": 1794.0,
+ "27": 1810.0,
+ "28": 1950.0,
+ "29": 1928.0,
+ "30": 1981.0,
+ "31": 2105.0,
+ "32": 1956.0,
+ "33": 1915.0,
+ "34": 2050.0,
+ "35": 2130.0,
+ "36": 2054.0,
+ "37": 2308.0,
+ "38": 2124.0,
+ "39": 2166.0,
+ "40": 2199.0,
+ "41": 2244.0,
+ "42": 1989.0,
+ "43": 2338.0,
+ "44": 2342.0,
+ "45": 2605.0,
+ "46": 2386.0,
+ "47": 2434.0,
+ "48": 2505.0,
+ "49": 2746.0,
+ "50": 2558.0,
+ "51": 2565.0,
+ "52": 2711.0,
+ "53": 2546.0,
+ "54": 2822.0,
+ "55": 2465.0,
+ "56": 2729.0,
+ "57": 2212.0,
+ "58": 3489.0,
+ "59": 2793.0,
+ "60": 2908.0,
+ "61": 2732.0,
+ "62": 3118.0,
+ "63": 3241.0,
+ "64": 3772.0,
+ "65": 2719.0,
+ "66": 2945.0,
+ "67": 3741.0,
+ "68": 3367.0,
+ "69": 2937.0,
+ "70": 3214.0,
+ "71": 3212.0,
+ "72": 3002.0,
+ "73": 3449.0,
+ "74": 3367.0,
+ "75": 3148.0,
+ "76": 3360.0,
+ "77": 3666.0,
+ "78": 3181.0,
+ "79": 3369.0,
+ "80": 3091.0,
+ "81": 3370.0,
+ "82": 3207.0,
+ "83": 3127.0,
+ "84": 3077.0,
+ "85": 2817.0,
+ "86": 3046.0,
+ "87": 3060.0,
+ "88": 3244.0,
+ "89": 3115.0,
+ "90": 4213.0,
+ "91": 2906.0,
+ "92": 3106.0,
+ "93": 3129.0,
+ "94": 3375.0,
+ "95": 3302.0,
+ "96": 3850.0,
+ "97": 3532.0,
+ "98": 3421.0,
+ "99": 3459.0,
+ "100": 3300.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 491224576.0,
- "2": 491224576.0,
- "3": 491224576.0,
- "4": 491224576.0,
- "5": 491224576.0,
- "6": 491224576.0,
- "7": 491224576.0,
- "8": 491224576.0,
- "9": 491224576.0,
- "10": 491224576.0,
- "11": 491224576.0,
- "12": 491224576.0,
- "13": 491224576.0,
- "14": 491224576.0,
- "15": 491224576.0,
- "16": 491224576.0,
- "17": 491224576.0,
- "18": 491224576.0,
- "19": 491224576.0,
- "20": 491224576.0,
- "21": 491224576.0,
- "22": 491224576.0,
- "23": 491224576.0,
- "24": 491224576.0,
- "25": 491224576.0,
- "26": 491224576.0,
- "27": 491224576.0,
- "28": 491224576.0,
- "29": 491224576.0,
- "30": 491224576.0,
- "31": 491224576.0,
- "32": 491224576.0,
- "33": 491224576.0,
- "34": 491224576.0,
- "35": 491224576.0,
- "36": 491224576.0,
- "37": 491224576.0,
- "38": 491224576.0,
- "39": 491224576.0,
- "40": 491224576.0,
- "41": 491224576.0,
- "42": 491224576.0,
- "43": 491224576.0,
- "44": 491224576.0,
- "45": 491224576.0,
- "46": 491224576.0,
- "47": 491224576.0,
- "48": 491224576.0,
- "49": 491224576.0,
- "50": 491224576.0,
- "51": 491224576.0,
- "52": 491224576.0,
- "53": 491224576.0,
- "54": 491224576.0,
- "55": 491224576.0,
- "56": 491224576.0,
- "57": 491224576.0,
- "58": 491224576.0,
- "59": 491224576.0,
- "60": 491224576.0,
- "61": 491224576.0,
- "62": 491224576.0,
- "63": 491224576.0,
- "64": 491224576.0,
- "65": 491224576.0,
- "66": 491224576.0,
- "67": 491224576.0,
- "68": 491224576.0,
- "69": 491224576.0,
- "70": 491224576.0,
- "71": 491224576.0,
- "72": 491224576.0,
- "73": 491224576.0,
- "74": 491224576.0,
- "75": 491224576.0,
- "76": 491224576.0,
- "77": 491224576.0,
- "78": 491224576.0,
- "79": 491224576.0,
- "80": 491224576.0,
- "81": 491224576.0,
- "82": 491224576.0,
- "83": 491224576.0,
- "84": 491224576.0,
- "85": 491224576.0,
- "86": 491224576.0,
- "87": 491224576.0,
- "88": 491224576.0,
- "89": 491224576.0,
- "90": 491224576.0,
- "91": 491224576.0,
- "92": 491224576.0,
- "93": 491224576.0,
- "94": 491224576.0,
- "95": 491224576.0,
- "96": 491224576.0,
- "97": 491224576.0,
- "98": 491224576.0,
- "99": 491224576.0,
- "100": 491224576.0
+ "1": 492011008.0,
+ "2": 492011008.0,
+ "3": 492011008.0,
+ "4": 492011008.0,
+ "5": 492011008.0,
+ "6": 492011008.0,
+ "7": 492011008.0,
+ "8": 492011008.0,
+ "9": 492011008.0,
+ "10": 492011008.0,
+ "11": 492011008.0,
+ "12": 492011008.0,
+ "13": 492011008.0,
+ "14": 492011008.0,
+ "15": 492011008.0,
+ "16": 492011008.0,
+ "17": 492011008.0,
+ "18": 492011008.0,
+ "19": 492011008.0,
+ "20": 492011008.0,
+ "21": 492011008.0,
+ "22": 492011008.0,
+ "23": 492011008.0,
+ "24": 492011008.0,
+ "25": 492011008.0,
+ "26": 492011008.0,
+ "27": 492011008.0,
+ "28": 492011008.0,
+ "29": 492011008.0,
+ "30": 492011008.0,
+ "31": 492011008.0,
+ "32": 492011008.0,
+ "33": 492011008.0,
+ "34": 492011008.0,
+ "35": 492011008.0,
+ "36": 492011008.0,
+ "37": 492011008.0,
+ "38": 492011008.0,
+ "39": 492011008.0,
+ "40": 492011008.0,
+ "41": 492011008.0,
+ "42": 492011008.0,
+ "43": 492011008.0,
+ "44": 492011008.0,
+ "45": 492011008.0,
+ "46": 492011008.0,
+ "47": 492011008.0,
+ "48": 492011008.0,
+ "49": 492011008.0,
+ "50": 492011008.0,
+ "51": 492011008.0,
+ "52": 492011008.0,
+ "53": 492011008.0,
+ "54": 492011008.0,
+ "55": 492011008.0,
+ "56": 492011008.0,
+ "57": 492011008.0,
+ "58": 492011008.0,
+ "59": 492011008.0,
+ "60": 492011008.0,
+ "61": 492011008.0,
+ "62": 492011008.0,
+ "63": 492011008.0,
+ "64": 492011008.0,
+ "65": 492011008.0,
+ "66": 492011008.0,
+ "67": 492011008.0,
+ "68": 492011008.0,
+ "69": 492011008.0,
+ "70": 492011008.0,
+ "71": 492011008.0,
+ "72": 492011008.0,
+ "73": 492011008.0,
+ "74": 492011008.0,
+ "75": 492011008.0,
+ "76": 492011008.0,
+ "77": 492011008.0,
+ "78": 492011008.0,
+ "79": 492011008.0,
+ "80": 492011008.0,
+ "81": 492011008.0,
+ "82": 492011008.0,
+ "83": 492011008.0,
+ "84": 492011008.0,
+ "85": 492011008.0,
+ "86": 492011008.0,
+ "87": 492011008.0,
+ "88": 492011008.0,
+ "89": 492011008.0,
+ "90": 492011008.0,
+ "91": 492011008.0,
+ "92": 492011008.0,
+ "93": 492011008.0,
+ "94": 492011008.0,
+ "95": 492011008.0,
+ "96": 492011008.0,
+ "97": 492011008.0,
+ "98": 492011008.0,
+ "99": 492011008.0,
+ "100": 492011008.0
}
},
"mem-max-allocated-bytes": {
@@ -326,105 +326,105 @@
"step_interval": 1,
"values": {
"1": 1553276416.0,
- "2": 1681702400.0,
- "3": 1681702400.0,
- "4": 1681702400.0,
- "5": 1681702400.0,
- "6": 1681702400.0,
- "7": 1681702400.0,
- "8": 1681702400.0,
- "9": 1681702400.0,
- "10": 1681702400.0,
- "11": 1681702400.0,
- "12": 1681702400.0,
- "13": 1681702400.0,
- "14": 1681702400.0,
- "15": 1681702400.0,
- "16": 1681702400.0,
- "17": 1681702400.0,
- "18": 1681702400.0,
- "19": 1681702400.0,
- "20": 1681702400.0,
- "21": 1681702400.0,
- "22": 1681702400.0,
- "23": 1681702400.0,
- "24": 1681702400.0,
- "25": 1681702400.0,
- "26": 1681702400.0,
- "27": 1681702400.0,
- "28": 1681702400.0,
- "29": 1681702400.0,
- "30": 1681702400.0,
- "31": 1681702400.0,
- "32": 1681702400.0,
- "33": 1681702400.0,
- "34": 1681702400.0,
- "35": 1681702400.0,
- "36": 1681702400.0,
- "37": 1681702400.0,
- "38": 1681702400.0,
- "39": 1681702400.0,
- "40": 1681702400.0,
- "41": 1681702400.0,
- "42": 1681702400.0,
- "43": 1681702400.0,
- "44": 1681702400.0,
- "45": 1681702400.0,
- "46": 1681702400.0,
- "47": 1681702400.0,
- "48": 1681702400.0,
- "49": 1681702400.0,
- "50": 1681702400.0,
- "51": 1681702400.0,
- "52": 1681702400.0,
- "53": 1681702400.0,
- "54": 1681702400.0,
- "55": 1681702400.0,
- "56": 1681702400.0,
- "57": 1681702400.0,
- "58": 1681702400.0,
- "59": 1681702400.0,
- "60": 1681702400.0,
- "61": 1681702400.0,
- "62": 1681702400.0,
- "63": 1681702400.0,
- "64": 1681702400.0,
- "65": 1681702400.0,
- "66": 1681702400.0,
- "67": 1681702400.0,
- "68": 1681702400.0,
- "69": 1681702400.0,
- "70": 1681702400.0,
- "71": 1681702400.0,
- "72": 1681702400.0,
- "73": 1681702400.0,
- "74": 1681702400.0,
- "75": 1681702400.0,
- "76": 1681702400.0,
- "77": 1681702400.0,
- "78": 1681702400.0,
- "79": 1681702400.0,
- "80": 1681702400.0,
- "81": 1681702400.0,
- "82": 1681702400.0,
- "83": 1681702400.0,
- "84": 1681702400.0,
- "85": 1681702400.0,
- "86": 1681702400.0,
- "87": 1681702400.0,
- "88": 1681702400.0,
- "89": 1681702400.0,
- "90": 1681702400.0,
- "91": 1681702400.0,
- "92": 1681702400.0,
- "93": 1681702400.0,
- "94": 1681702400.0,
- "95": 1681702400.0,
- "96": 1681702400.0,
- "97": 1681702400.0,
- "98": 1681702400.0,
- "99": 1681702400.0,
- "100": 1681702400.0
+ "2": 1682488832.0,
+ "3": 1682488832.0,
+ "4": 1682488832.0,
+ "5": 1682488832.0,
+ "6": 1682488832.0,
+ "7": 1682488832.0,
+ "8": 1682488832.0,
+ "9": 1682488832.0,
+ "10": 1682488832.0,
+ "11": 1682488832.0,
+ "12": 1682488832.0,
+ "13": 1682488832.0,
+ "14": 1682488832.0,
+ "15": 1682488832.0,
+ "16": 1682488832.0,
+ "17": 1682488832.0,
+ "18": 1682488832.0,
+ "19": 1682488832.0,
+ "20": 1682488832.0,
+ "21": 1682488832.0,
+ "22": 1682488832.0,
+ "23": 1682488832.0,
+ "24": 1682488832.0,
+ "25": 1682488832.0,
+ "26": 1682488832.0,
+ "27": 1682488832.0,
+ "28": 1682488832.0,
+ "29": 1682488832.0,
+ "30": 1682488832.0,
+ "31": 1682488832.0,
+ "32": 1682488832.0,
+ "33": 1682488832.0,
+ "34": 1682488832.0,
+ "35": 1682488832.0,
+ "36": 1682488832.0,
+ "37": 1682488832.0,
+ "38": 1682488832.0,
+ "39": 1682488832.0,
+ "40": 1682488832.0,
+ "41": 1682488832.0,
+ "42": 1682488832.0,
+ "43": 1682488832.0,
+ "44": 1682488832.0,
+ "45": 1682488832.0,
+ "46": 1682488832.0,
+ "47": 1682488832.0,
+ "48": 1682488832.0,
+ "49": 1682488832.0,
+ "50": 1682488832.0,
+ "51": 1684322304.0,
+ "52": 1684322304.0,
+ "53": 1684322304.0,
+ "54": 1684322304.0,
+ "55": 1684322304.0,
+ "56": 1684322304.0,
+ "57": 1684322304.0,
+ "58": 1684322304.0,
+ "59": 1684322304.0,
+ "60": 1684322304.0,
+ "61": 1684322304.0,
+ "62": 1684322304.0,
+ "63": 1684322304.0,
+ "64": 1684322304.0,
+ "65": 1684322304.0,
+ "66": 1684322304.0,
+ "67": 1684322304.0,
+ "68": 1684322304.0,
+ "69": 1684322304.0,
+ "70": 1684322304.0,
+ "71": 1684322304.0,
+ "72": 1684322304.0,
+ "73": 1684322304.0,
+ "74": 1684322304.0,
+ "75": 1684322304.0,
+ "76": 1684322304.0,
+ "77": 1684322304.0,
+ "78": 1684322304.0,
+ "79": 1684322304.0,
+ "80": 1684322304.0,
+ "81": 1684322304.0,
+ "82": 1684322304.0,
+ "83": 1684322304.0,
+ "84": 1684322304.0,
+ "85": 1684322304.0,
+ "86": 1684322304.0,
+ "87": 1684322304.0,
+ "88": 1684322304.0,
+ "89": 1684322304.0,
+ "90": 1684322304.0,
+ "91": 1684322304.0,
+ "92": 1684322304.0,
+ "93": 1684322304.0,
+ "94": 1684322304.0,
+ "95": 1684322304.0,
+ "96": 1684322304.0,
+ "97": 1684322304.0,
+ "98": 1684322304.0,
+ "99": 1684322304.0,
+ "100": 1684322304.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 13.28736,
- "2": 0.1399,
- "3": 0.12618,
- "4": 0.10709,
- "5": 0.11408,
- "6": 0.10894,
- "7": 0.10708,
- "8": 0.10773,
- "9": 0.10787,
- "10": 0.10884,
- "11": 0.10818,
- "12": 0.10774,
- "13": 0.1067,
- "14": 0.1065,
- "15": 0.10599,
- "16": 0.10552,
- "17": 0.10782,
- "18": 0.10913,
- "19": 0.10816,
- "20": 0.10759,
- "21": 0.108,
- "22": 0.10902,
- "23": 0.1076,
- "24": 0.1068,
- "25": 0.10674,
- "26": 0.10699,
- "27": 0.10678,
- "28": 0.10642,
- "29": 0.1066,
- "30": 0.10707,
- "31": 0.10794,
- "32": 0.10702,
- "33": 0.10586,
- "34": 0.10612,
- "35": 0.10628,
- "36": 0.10631,
- "37": 0.10573,
- "38": 0.10617,
- "39": 0.10563,
- "40": 0.1064,
- "41": 0.1059,
- "42": 0.1054,
- "43": 0.10691,
- "44": 0.10833,
- "45": 0.10638,
- "46": 0.10655,
- "47": 0.10676,
- "48": 0.10825,
- "49": 0.10534,
- "50": 0.10635,
- "51": 0.12108,
- "52": 0.12016,
- "53": 0.11315,
- "54": 0.10912,
- "55": 0.11556,
- "56": 0.10742,
- "57": 0.10784,
- "58": 0.11719,
- "59": 0.10791,
- "60": 0.10886,
- "61": 0.11563,
- "62": 0.10714,
- "63": 0.10967,
- "64": 0.11569,
- "65": 0.10753,
- "66": 0.1078,
- "67": 0.10545,
- "68": 0.10522,
- "69": 0.10496,
- "70": 0.10544,
- "71": 0.10719,
- "72": 0.10708,
- "73": 0.1062,
- "74": 0.10663,
- "75": 0.10766,
- "76": 0.10634,
- "77": 0.106,
- "78": 0.10757,
- "79": 0.10574,
- "80": 0.10548,
- "81": 0.1068,
- "82": 0.10639,
- "83": 0.10598,
- "84": 0.10693,
- "85": 0.10553,
- "86": 0.10606,
- "87": 0.10692,
- "88": 0.10564,
- "89": 0.10633,
- "90": 0.10625,
- "91": 0.10563,
- "92": 0.10508,
- "93": 0.10937,
- "94": 0.10519,
- "95": 0.10566,
- "96": 0.11009,
- "97": 0.10631,
- "98": 0.10595,
- "99": 0.10785,
- "100": 0.10678
+ "1": "nan",
+ "2": 6.81785,
+ "3": 0.1217,
+ "4": 0.10667,
+ "5": 0.10722,
+ "6": 0.10608,
+ "7": 0.10617,
+ "8": 0.10683,
+ "9": 0.10678,
+ "10": 0.107,
+ "11": 0.10656,
+ "12": 0.10744,
+ "13": 0.10621,
+ "14": 0.1069,
+ "15": 0.10627,
+ "16": 0.10766,
+ "17": 0.10662,
+ "18": 0.10581,
+ "19": 0.10787,
+ "20": 0.10641,
+ "21": 0.10555,
+ "22": 0.10661,
+ "23": 0.10668,
+ "24": 0.10869,
+ "25": 0.10581,
+ "26": 0.10618,
+ "27": 0.10675,
+ "28": 0.10689,
+ "29": 0.10705,
+ "30": 0.10679,
+ "31": 0.10617,
+ "32": 0.10676,
+ "33": 0.1105,
+ "34": 0.10647,
+ "35": 0.10623,
+ "36": 0.10597,
+ "37": 0.10603,
+ "38": 0.10695,
+ "39": 0.10637,
+ "40": 0.10613,
+ "41": 0.10636,
+ "42": 0.10584,
+ "43": 0.10891,
+ "44": 0.1079,
+ "45": 0.10744,
+ "46": 0.10704,
+ "47": 0.10654,
+ "48": 0.10684,
+ "49": 0.10703,
+ "50": 0.10637,
+ "51": 0.14819,
+ "52": 0.10705,
+ "53": 0.10902,
+ "54": 0.10788,
+ "55": 0.10684,
+ "56": 0.10746,
+ "57": 0.10757,
+ "58": 0.1072,
+ "59": 0.10688,
+ "60": 0.10686,
+ "61": 0.10764,
+ "62": 0.10733,
+ "63": 0.11646,
+ "64": 0.10801,
+ "65": 0.10715,
+ "66": 0.10704,
+ "67": 0.10824,
+ "68": 0.10674,
+ "69": 0.10728,
+ "70": 0.10724,
+ "71": 0.10735,
+ "72": 0.1069,
+ "73": 0.10696,
+ "74": 0.10737,
+ "75": 0.10741,
+ "76": 0.10806,
+ "77": 0.10676,
+ "78": 0.10692,
+ "79": 0.10744,
+ "80": 0.10661,
+ "81": 0.10891,
+ "82": 0.10834,
+ "83": 0.107,
+ "84": 0.10834,
+ "85": 0.10697,
+ "86": 0.10676,
+ "87": 0.10842,
+ "88": 0.10736,
+ "89": 0.10811,
+ "90": 0.10753,
+ "91": 0.10799,
+ "92": 0.10793,
+ "93": 0.10741,
+ "94": 0.10688,
+ "95": 0.10737,
+ "96": 0.10728,
+ "97": 0.10759,
+ "98": 0.10762,
+ "99": 0.10776,
+ "100": 0.10782
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances/model_config.yaml
index d3446e92c2e..42c361d5f8a 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -54,4 +54,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2/model_config.yaml
index 05b166f0a7b..11b6f1c57a1 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -50,4 +50,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2/golden_values_dev_dgx_h100.json
index 5b1ee17f8f6..441fc057cf5 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.86539,
- "2": 10.85871,
- "3": 10.86282,
- "4": 10.84007,
- "5": 10.87856,
- "6": 10.88852,
- "7": 10.86536,
- "8": 10.86015,
- "9": 10.85991,
- "10": 10.82982,
- "11": 10.88947,
- "12": 10.87511,
- "13": 10.87422,
- "14": 10.89675,
- "15": 10.82056,
- "16": 10.82497,
- "17": 10.78983,
- "18": 10.81029,
- "19": 10.80528,
- "20": 10.70396,
- "21": 10.6699,
- "22": 10.50641,
- "23": 10.69006,
- "24": 10.56312,
- "25": 10.49418,
- "26": 10.56627,
- "27": 10.58023,
- "28": 10.51571,
- "29": 10.55296,
- "30": 10.30551,
- "31": 10.02244,
- "32": 10.40618,
- "33": 10.39874,
- "34": 10.1377,
- "35": 10.20184,
- "36": 10.1605,
- "37": 10.28975,
- "38": 10.11483,
- "39": 10.361,
- "40": 10.01905,
- "41": 10.07292,
- "42": 10.14697,
- "43": 9.74684,
- "44": 9.87763,
- "45": 9.74962,
- "46": 9.73382,
- "47": 10.07536,
- "48": 9.78071,
- "49": 9.44783,
- "50": 9.8399
+ "1": 10.95937,
+ "2": 10.95597,
+ "3": 10.94131,
+ "4": 10.94667,
+ "5": 10.94232,
+ "6": 10.9384,
+ "7": 10.94616,
+ "8": 10.94246,
+ "9": 10.93783,
+ "10": 10.94034,
+ "11": 10.93147,
+ "12": 10.92721,
+ "13": 10.91651,
+ "14": 10.92439,
+ "15": 10.8876,
+ "16": 10.87828,
+ "17": 10.88554,
+ "18": 10.86856,
+ "19": 10.87826,
+ "20": 10.77113,
+ "21": 10.76393,
+ "22": 10.7496,
+ "23": 10.73998,
+ "24": 10.71066,
+ "25": 10.70289,
+ "26": 10.69316,
+ "27": 10.63313,
+ "28": 10.5774,
+ "29": 10.5472,
+ "30": 10.51617,
+ "31": 10.52164,
+ "32": 10.49521,
+ "33": 10.45901,
+ "34": 10.43664,
+ "35": 10.421,
+ "36": 10.40006,
+ "37": 10.37615,
+ "38": 10.38192,
+ "39": 10.3437,
+ "40": 10.33146,
+ "41": 10.30159,
+ "42": 10.27788,
+ "43": 10.25924,
+ "44": 10.2343,
+ "45": 10.24978,
+ "46": 10.20189,
+ "47": 10.19642,
+ "48": 10.1431,
+ "49": 10.14958,
+ "50": 10.1453
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 572.0,
- "2": 656.0,
- "3": 649.0,
- "4": 631.0,
- "5": 658.0,
- "6": 636.0,
- "7": 636.0,
- "8": 542.0,
- "9": 653.0,
- "10": 551.0,
- "11": 681.0,
- "12": 642.0,
- "13": 624.0,
+ "1": 602.0,
+ "2": 608.0,
+ "3": 625.0,
+ "4": 616.0,
+ "5": 644.0,
+ "6": 610.0,
+ "7": 647.0,
+ "8": 580.0,
+ "9": 624.0,
+ "10": 613.0,
+ "11": 631.0,
+ "12": 618.0,
+ "13": 636.0,
"14": 658.0,
- "15": 682.0,
- "16": 659.0,
- "17": 620.0,
- "18": 603.0,
- "19": 634.0,
- "20": 639.0,
- "21": 634.0,
- "22": 602.0,
- "23": 731.0,
- "24": 620.0,
- "25": 611.0,
- "26": 626.0,
- "27": 683.0,
- "28": 668.0,
- "29": 713.0,
- "30": 712.0,
- "31": 616.0,
- "32": 786.0,
- "33": 800.0,
- "34": 702.0,
- "35": 684.0,
- "36": 664.0,
- "37": 831.0,
- "38": 802.0,
- "39": 919.0,
- "40": 802.0,
- "41": 791.0,
- "42": 840.0,
- "43": 718.0,
- "44": 756.0,
- "45": 765.0,
- "46": 809.0,
- "47": 839.0,
- "48": 827.0,
- "49": 935.0,
- "50": 839.0
+ "15": 633.0,
+ "16": 619.0,
+ "17": 646.0,
+ "18": 641.0,
+ "19": 649.0,
+ "20": 612.0,
+ "21": 673.0,
+ "22": 643.0,
+ "23": 645.0,
+ "24": 683.0,
+ "25": 610.0,
+ "26": 651.0,
+ "27": 718.0,
+ "28": 683.0,
+ "29": 715.0,
+ "30": 736.0,
+ "31": 708.0,
+ "32": 716.0,
+ "33": 699.0,
+ "34": 741.0,
+ "35": 790.0,
+ "36": 727.0,
+ "37": 833.0,
+ "38": 778.0,
+ "39": 796.0,
+ "40": 843.0,
+ "41": 828.0,
+ "42": 748.0,
+ "43": 886.0,
+ "44": 825.0,
+ "45": 950.0,
+ "46": 873.0,
+ "47": 881.0,
+ "48": 983.0,
+ "49": 1051.0,
+ "50": 1022.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 510689792.0,
- "2": 510689792.0,
- "3": 510689792.0,
- "4": 510689792.0,
- "5": 510689792.0,
- "6": 510689792.0,
- "7": 510689792.0,
- "8": 510689792.0,
- "9": 510689792.0,
- "10": 510689792.0,
- "11": 510689792.0,
- "12": 510689792.0,
- "13": 510689792.0,
- "14": 510689792.0,
- "15": 510689792.0,
- "16": 510689792.0,
- "17": 510689792.0,
- "18": 510689792.0,
- "19": 510689792.0,
- "20": 510689792.0,
- "21": 510689792.0,
- "22": 510689792.0,
- "23": 510689792.0,
- "24": 510689792.0,
- "25": 510689792.0,
- "26": 510689792.0,
- "27": 510689792.0,
- "28": 510689792.0,
- "29": 510689792.0,
- "30": 510689792.0,
- "31": 510689792.0,
- "32": 510689792.0,
- "33": 510689792.0,
- "34": 510689792.0,
- "35": 510689792.0,
- "36": 510689792.0,
- "37": 510689792.0,
- "38": 510689792.0,
- "39": 510689792.0,
- "40": 510689792.0,
- "41": 510689792.0,
- "42": 510689792.0,
- "43": 510689792.0,
- "44": 510689792.0,
- "45": 510689792.0,
- "46": 510689792.0,
- "47": 510689792.0,
- "48": 510689792.0,
- "49": 510689792.0,
- "50": 510689792.0
+ "1": 511214080.0,
+ "2": 511214080.0,
+ "3": 511214080.0,
+ "4": 511214080.0,
+ "5": 511214080.0,
+ "6": 511214080.0,
+ "7": 511214080.0,
+ "8": 511214080.0,
+ "9": 511214080.0,
+ "10": 511214080.0,
+ "11": 511214080.0,
+ "12": 511214080.0,
+ "13": 511214080.0,
+ "14": 511214080.0,
+ "15": 511214080.0,
+ "16": 511214080.0,
+ "17": 511214080.0,
+ "18": 511214080.0,
+ "19": 511214080.0,
+ "20": 511214080.0,
+ "21": 511214080.0,
+ "22": 511214080.0,
+ "23": 511214080.0,
+ "24": 511214080.0,
+ "25": 511214080.0,
+ "26": 511214080.0,
+ "27": 511214080.0,
+ "28": 511214080.0,
+ "29": 511214080.0,
+ "30": 511214080.0,
+ "31": 511214080.0,
+ "32": 511214080.0,
+ "33": 511214080.0,
+ "34": 511214080.0,
+ "35": 511214080.0,
+ "36": 511214080.0,
+ "37": 511214080.0,
+ "38": 511214080.0,
+ "39": 511214080.0,
+ "40": 511214080.0,
+ "41": 511214080.0,
+ "42": 511214080.0,
+ "43": 511214080.0,
+ "44": 511214080.0,
+ "45": 511214080.0,
+ "46": 511214080.0,
+ "47": 511214080.0,
+ "48": 511214080.0,
+ "49": 511214080.0,
+ "50": 511214080.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 757802496.0,
- "2": 935777792.0,
- "3": 938397696.0,
- "4": 938397696.0,
- "5": 938397696.0,
- "6": 938397696.0,
- "7": 938397696.0,
- "8": 938397696.0,
- "9": 938397696.0,
- "10": 938398208.0,
- "11": 938398208.0,
- "12": 938398208.0,
- "13": 938398208.0,
- "14": 938398720.0,
- "15": 938398720.0,
- "16": 938398720.0,
- "17": 938398720.0,
- "18": 938398720.0,
- "19": 938398720.0,
- "20": 938398720.0,
- "21": 938398720.0,
- "22": 938398720.0,
- "23": 938398720.0,
- "24": 938398720.0,
- "25": 938399232.0,
- "26": 938399232.0,
- "27": 938399232.0,
- "28": 938399232.0,
- "29": 938399232.0,
- "30": 938399232.0,
- "31": 938399232.0,
- "32": 938399232.0,
- "33": 938399232.0,
- "34": 938399232.0,
- "35": 938399232.0,
- "36": 938399232.0,
- "37": 938399232.0,
- "38": 938399232.0,
- "39": 938399232.0,
- "40": 938399232.0,
- "41": 938399232.0,
- "42": 938399232.0,
- "43": 938399232.0,
- "44": 938399232.0,
- "45": 938399232.0,
- "46": 938399232.0,
- "47": 938399232.0,
- "48": 938399232.0,
- "49": 938399232.0,
- "50": 938399232.0
+ "1": 759896576.0,
+ "2": 935775232.0,
+ "3": 935777792.0,
+ "4": 935777792.0,
+ "5": 935777792.0,
+ "6": 935777792.0,
+ "7": 935777792.0,
+ "8": 935777792.0,
+ "9": 935777792.0,
+ "10": 935777792.0,
+ "11": 935777792.0,
+ "12": 935777792.0,
+ "13": 935777792.0,
+ "14": 935777792.0,
+ "15": 935777792.0,
+ "16": 935777792.0,
+ "17": 935777792.0,
+ "18": 935777792.0,
+ "19": 935777792.0,
+ "20": 935777792.0,
+ "21": 935777792.0,
+ "22": 935777792.0,
+ "23": 935777792.0,
+ "24": 935777792.0,
+ "25": 935777792.0,
+ "26": 935777792.0,
+ "27": 935777792.0,
+ "28": 935777792.0,
+ "29": 935777792.0,
+ "30": 935777792.0,
+ "31": 935777792.0,
+ "32": 935777792.0,
+ "33": 935777792.0,
+ "34": 935777792.0,
+ "35": 935777792.0,
+ "36": 935777792.0,
+ "37": 935777792.0,
+ "38": 935777792.0,
+ "39": 935777792.0,
+ "40": 935777792.0,
+ "41": 935777792.0,
+ "42": 935777792.0,
+ "43": 935777792.0,
+ "44": 935777792.0,
+ "45": 935777792.0,
+ "46": 935777792.0,
+ "47": 935777792.0,
+ "48": 935777792.0,
+ "49": 935777792.0,
+ "50": 935777792.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 35.36663,
- "2": 0.35208,
- "3": 0.32012,
- "4": 0.29736,
- "5": 0.30009,
- "6": 0.29722,
- "7": 0.29604,
- "8": 0.29598,
- "9": 0.30123,
- "10": 0.29278,
- "11": 0.29195,
- "12": 0.30003,
- "13": 0.2957,
- "14": 0.2935,
- "15": 0.29372,
- "16": 0.2984,
- "17": 0.29013,
- "18": 0.29041,
- "19": 0.2934,
- "20": 0.29454,
- "21": 0.2936,
- "22": 0.29663,
- "23": 0.29453,
- "24": 0.29404,
- "25": 0.2912,
- "26": 0.29009,
- "27": 0.29448,
- "28": 0.29043,
- "29": 0.29359,
- "30": 0.29413,
- "31": 0.29317,
- "32": 0.29247,
- "33": 0.29418,
- "34": 0.2938,
- "35": 0.29207,
- "36": 0.31485,
- "37": 0.29543,
- "38": 0.29402,
- "39": 0.29262,
- "40": 0.2957,
- "41": 0.29348,
- "42": 0.29242,
- "43": 0.29117,
- "44": 0.2927,
- "45": 0.29263,
- "46": 0.29024,
- "47": 0.29404,
- "48": 0.28901,
- "49": 0.28844,
- "50": 0.29053
+ "1": "nan",
+ "2": 9.17021,
+ "3": 0.3236,
+ "4": 0.29741,
+ "5": 0.29824,
+ "6": 0.29667,
+ "7": 0.29721,
+ "8": 0.29847,
+ "9": 0.29778,
+ "10": 0.29781,
+ "11": 0.29724,
+ "12": 0.29764,
+ "13": 0.29711,
+ "14": 0.29692,
+ "15": 0.29762,
+ "16": 0.30125,
+ "17": 0.29757,
+ "18": 0.29889,
+ "19": 0.30021,
+ "20": 0.29857,
+ "21": 0.29818,
+ "22": 0.29661,
+ "23": 0.2969,
+ "24": 0.29673,
+ "25": 0.29649,
+ "26": 0.29854,
+ "27": 0.29729,
+ "28": 0.29663,
+ "29": 0.29756,
+ "30": 0.29698,
+ "31": 0.2971,
+ "32": 0.29772,
+ "33": 0.29731,
+ "34": 0.29558,
+ "35": 0.29628,
+ "36": 0.29716,
+ "37": 0.2963,
+ "38": 0.29672,
+ "39": 0.29783,
+ "40": 0.29553,
+ "41": 0.29467,
+ "42": 0.29379,
+ "43": 0.2959,
+ "44": 0.29854,
+ "45": 0.29473,
+ "46": 0.2984,
+ "47": 0.29885,
+ "48": 0.29458,
+ "49": 0.29564,
+ "50": 0.29683
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2/model_config.yaml
index 70155c2ff81..56df7d07a0d 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -50,4 +50,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: flash
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss/golden_values_dev_dgx_h100.json
index f5628621ad5..96a6fd397a6 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.86539,
- "2": 10.85871,
- "3": 10.86282,
- "4": 10.84007,
- "5": 10.87856,
- "6": 10.88852,
- "7": 10.86536,
- "8": 10.86015,
- "9": 10.85991,
- "10": 10.82982,
- "11": 10.88947,
- "12": 10.87511,
- "13": 10.87422,
- "14": 10.89675,
- "15": 10.82056,
- "16": 10.82497,
- "17": 10.78983,
- "18": 10.81029,
- "19": 10.80528,
- "20": 10.70396,
- "21": 10.6699,
- "22": 10.50641,
- "23": 10.69006,
- "24": 10.56312,
- "25": 10.49418,
- "26": 10.56627,
- "27": 10.58023,
- "28": 10.51571,
- "29": 10.55296,
- "30": 10.30551,
- "31": 10.02244,
- "32": 10.40618,
- "33": 10.39874,
- "34": 10.1377,
- "35": 10.20184,
- "36": 10.1605,
- "37": 10.28975,
- "38": 10.11483,
- "39": 10.361,
- "40": 10.01905,
- "41": 10.07292,
- "42": 10.14697,
- "43": 9.74684,
- "44": 9.87763,
- "45": 9.74962,
- "46": 9.73382,
- "47": 10.07536,
- "48": 9.78071,
- "49": 9.44783,
- "50": 9.8399
+ "1": 10.95937,
+ "2": 10.95597,
+ "3": 10.94131,
+ "4": 10.94667,
+ "5": 10.94232,
+ "6": 10.9384,
+ "7": 10.94616,
+ "8": 10.94246,
+ "9": 10.93783,
+ "10": 10.94034,
+ "11": 10.93147,
+ "12": 10.92721,
+ "13": 10.91651,
+ "14": 10.92439,
+ "15": 10.8876,
+ "16": 10.87828,
+ "17": 10.88554,
+ "18": 10.86856,
+ "19": 10.87826,
+ "20": 10.77113,
+ "21": 10.76393,
+ "22": 10.7496,
+ "23": 10.73998,
+ "24": 10.71066,
+ "25": 10.70289,
+ "26": 10.69316,
+ "27": 10.63313,
+ "28": 10.5774,
+ "29": 10.5472,
+ "30": 10.51617,
+ "31": 10.52164,
+ "32": 10.49521,
+ "33": 10.45901,
+ "34": 10.43664,
+ "35": 10.421,
+ "36": 10.40006,
+ "37": 10.37615,
+ "38": 10.38192,
+ "39": 10.3437,
+ "40": 10.33146,
+ "41": 10.30159,
+ "42": 10.27788,
+ "43": 10.25924,
+ "44": 10.2343,
+ "45": 10.24978,
+ "46": 10.20189,
+ "47": 10.19642,
+ "48": 10.1431,
+ "49": 10.14958,
+ "50": 10.1453
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 572.0,
- "2": 656.0,
- "3": 649.0,
- "4": 631.0,
- "5": 658.0,
- "6": 636.0,
- "7": 636.0,
- "8": 542.0,
- "9": 653.0,
- "10": 551.0,
- "11": 681.0,
- "12": 642.0,
- "13": 624.0,
+ "1": 602.0,
+ "2": 608.0,
+ "3": 625.0,
+ "4": 616.0,
+ "5": 644.0,
+ "6": 610.0,
+ "7": 647.0,
+ "8": 580.0,
+ "9": 624.0,
+ "10": 613.0,
+ "11": 631.0,
+ "12": 618.0,
+ "13": 636.0,
"14": 658.0,
- "15": 682.0,
- "16": 659.0,
- "17": 620.0,
- "18": 603.0,
- "19": 634.0,
- "20": 639.0,
- "21": 634.0,
- "22": 602.0,
- "23": 731.0,
- "24": 620.0,
- "25": 611.0,
- "26": 626.0,
- "27": 683.0,
- "28": 668.0,
- "29": 713.0,
- "30": 712.0,
- "31": 616.0,
- "32": 786.0,
- "33": 800.0,
- "34": 702.0,
- "35": 684.0,
- "36": 664.0,
- "37": 831.0,
- "38": 802.0,
- "39": 919.0,
- "40": 802.0,
- "41": 791.0,
- "42": 840.0,
- "43": 718.0,
- "44": 756.0,
- "45": 765.0,
- "46": 809.0,
- "47": 839.0,
- "48": 827.0,
- "49": 935.0,
- "50": 839.0
+ "15": 633.0,
+ "16": 619.0,
+ "17": 646.0,
+ "18": 641.0,
+ "19": 649.0,
+ "20": 612.0,
+ "21": 673.0,
+ "22": 643.0,
+ "23": 645.0,
+ "24": 683.0,
+ "25": 610.0,
+ "26": 651.0,
+ "27": 718.0,
+ "28": 683.0,
+ "29": 715.0,
+ "30": 736.0,
+ "31": 708.0,
+ "32": 716.0,
+ "33": 699.0,
+ "34": 741.0,
+ "35": 790.0,
+ "36": 727.0,
+ "37": 833.0,
+ "38": 778.0,
+ "39": 796.0,
+ "40": 843.0,
+ "41": 828.0,
+ "42": 748.0,
+ "43": 886.0,
+ "44": 825.0,
+ "45": 950.0,
+ "46": 873.0,
+ "47": 881.0,
+ "48": 983.0,
+ "49": 1051.0,
+ "50": 1022.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 510689792.0,
- "2": 510689792.0,
- "3": 510689792.0,
- "4": 510689792.0,
- "5": 510689792.0,
- "6": 510689792.0,
- "7": 510689792.0,
- "8": 510689792.0,
- "9": 510689792.0,
- "10": 510689792.0,
- "11": 510689792.0,
- "12": 510689792.0,
- "13": 510689792.0,
- "14": 510689792.0,
- "15": 510689792.0,
- "16": 510689792.0,
- "17": 510689792.0,
- "18": 510689792.0,
- "19": 510689792.0,
- "20": 510689792.0,
- "21": 510689792.0,
- "22": 510689792.0,
- "23": 510689792.0,
- "24": 510689792.0,
- "25": 510689792.0,
- "26": 510689792.0,
- "27": 510689792.0,
- "28": 510689792.0,
- "29": 510689792.0,
- "30": 510689792.0,
- "31": 510689792.0,
- "32": 510689792.0,
- "33": 510689792.0,
- "34": 510689792.0,
- "35": 510689792.0,
- "36": 510689792.0,
- "37": 510689792.0,
- "38": 510689792.0,
- "39": 510689792.0,
- "40": 510689792.0,
- "41": 510689792.0,
- "42": 510689792.0,
- "43": 510689792.0,
- "44": 510689792.0,
- "45": 510689792.0,
- "46": 510689792.0,
- "47": 510689792.0,
- "48": 510689792.0,
- "49": 510689792.0,
- "50": 510689792.0
+ "1": 511214080.0,
+ "2": 511214080.0,
+ "3": 511214080.0,
+ "4": 511214080.0,
+ "5": 511214080.0,
+ "6": 511214080.0,
+ "7": 511214080.0,
+ "8": 511214080.0,
+ "9": 511214080.0,
+ "10": 511214080.0,
+ "11": 511214080.0,
+ "12": 511214080.0,
+ "13": 511214080.0,
+ "14": 511214080.0,
+ "15": 511214080.0,
+ "16": 511214080.0,
+ "17": 511214080.0,
+ "18": 511214080.0,
+ "19": 511214080.0,
+ "20": 511214080.0,
+ "21": 511214080.0,
+ "22": 511214080.0,
+ "23": 511214080.0,
+ "24": 511214080.0,
+ "25": 511214080.0,
+ "26": 511214080.0,
+ "27": 511214080.0,
+ "28": 511214080.0,
+ "29": 511214080.0,
+ "30": 511214080.0,
+ "31": 511214080.0,
+ "32": 511214080.0,
+ "33": 511214080.0,
+ "34": 511214080.0,
+ "35": 511214080.0,
+ "36": 511214080.0,
+ "37": 511214080.0,
+ "38": 511214080.0,
+ "39": 511214080.0,
+ "40": 511214080.0,
+ "41": 511214080.0,
+ "42": 511214080.0,
+ "43": 511214080.0,
+ "44": 511214080.0,
+ "45": 511214080.0,
+ "46": 511214080.0,
+ "47": 511214080.0,
+ "48": 511214080.0,
+ "49": 511214080.0,
+ "50": 511214080.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 755704320.0,
- "2": 938398720.0,
- "3": 938398720.0,
- "4": 938398720.0,
- "5": 938398720.0,
- "6": 938399232.0,
- "7": 938399232.0,
- "8": 938399232.0,
- "9": 938399232.0,
- "10": 938399232.0,
- "11": 938399232.0,
- "12": 938399232.0,
- "13": 938399232.0,
- "14": 938399232.0,
- "15": 938399232.0,
- "16": 938399232.0,
- "17": 938399232.0,
- "18": 938399232.0,
- "19": 938399232.0,
- "20": 938399232.0,
- "21": 938399232.0,
- "22": 938399232.0,
- "23": 938399232.0,
- "24": 938399232.0,
- "25": 938399232.0,
- "26": 938399232.0,
- "27": 938399232.0,
- "28": 938399232.0,
- "29": 938399232.0,
- "30": 938399232.0,
- "31": 938399232.0,
- "32": 938399232.0,
- "33": 938399232.0,
- "34": 938399232.0,
- "35": 938399232.0,
- "36": 938399232.0,
- "37": 938399232.0,
- "38": 938399232.0,
- "39": 938399232.0,
- "40": 938399232.0,
- "41": 938399232.0,
- "42": 938399232.0,
- "43": 938399232.0,
- "44": 938399232.0,
- "45": 938399232.0,
- "46": 938399232.0,
- "47": 938399232.0,
- "48": 938399232.0,
- "49": 938399232.0,
- "50": 938399232.0
+ "1": 757801984.0,
+ "2": 935777792.0,
+ "3": 935777792.0,
+ "4": 935777792.0,
+ "5": 935777792.0,
+ "6": 935777792.0,
+ "7": 935777792.0,
+ "8": 935777792.0,
+ "9": 935777792.0,
+ "10": 935777792.0,
+ "11": 935777792.0,
+ "12": 935777792.0,
+ "13": 935777792.0,
+ "14": 935777792.0,
+ "15": 935777792.0,
+ "16": 935777792.0,
+ "17": 935777792.0,
+ "18": 935777792.0,
+ "19": 935777792.0,
+ "20": 935777792.0,
+ "21": 935777792.0,
+ "22": 935777792.0,
+ "23": 935777792.0,
+ "24": 935777792.0,
+ "25": 935777792.0,
+ "26": 935777792.0,
+ "27": 935777792.0,
+ "28": 935777792.0,
+ "29": 935777792.0,
+ "30": 935777792.0,
+ "31": 936825856.0,
+ "32": 936825856.0,
+ "33": 936825856.0,
+ "34": 936825856.0,
+ "35": 936825856.0,
+ "36": 936825856.0,
+ "37": 936825856.0,
+ "38": 936825856.0,
+ "39": 936825856.0,
+ "40": 936825856.0,
+ "41": 936825856.0,
+ "42": 936825856.0,
+ "43": 936825856.0,
+ "44": 936825856.0,
+ "45": 936825856.0,
+ "46": 936825856.0,
+ "47": 936825856.0,
+ "48": 936825856.0,
+ "49": 936825856.0,
+ "50": 936825856.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 35.29813,
- "2": 0.37906,
- "3": 0.30948,
- "4": 0.2886,
- "5": 0.28858,
- "6": 0.29461,
- "7": 0.28328,
- "8": 0.28783,
- "9": 0.28448,
- "10": 0.28698,
- "11": 0.28404,
- "12": 0.28717,
- "13": 0.2828,
- "14": 0.2846,
- "15": 0.28648,
- "16": 0.28793,
- "17": 0.28473,
- "18": 0.28326,
- "19": 0.28524,
- "20": 0.29094,
- "21": 0.29401,
- "22": 0.28944,
- "23": 0.28693,
- "24": 0.29508,
- "25": 0.28683,
- "26": 0.28507,
- "27": 0.2849,
- "28": 0.28658,
- "29": 0.28518,
- "30": 0.28539,
- "31": 0.2829,
- "32": 0.28482,
- "33": 0.28454,
- "34": 0.28634,
- "35": 0.28739,
- "36": 0.28563,
- "37": 0.28401,
- "38": 0.28251,
- "39": 0.28156,
- "40": 0.28197,
- "41": 0.28236,
- "42": 0.27995,
- "43": 0.28293,
- "44": 0.28018,
- "45": 0.28419,
- "46": 0.28512,
- "47": 0.2818,
- "48": 0.28099,
- "49": 0.2831,
- "50": 0.28153
+ "1": "nan",
+ "2": 9.25152,
+ "3": 0.31529,
+ "4": 0.29809,
+ "5": 0.29564,
+ "6": 0.29773,
+ "7": 0.30146,
+ "8": 0.30715,
+ "9": 0.30716,
+ "10": 0.3102,
+ "11": 0.30552,
+ "12": 0.29683,
+ "13": 0.29565,
+ "14": 0.29578,
+ "15": 0.2929,
+ "16": 0.29323,
+ "17": 0.29367,
+ "18": 0.29382,
+ "19": 0.30407,
+ "20": 0.30543,
+ "21": 0.30472,
+ "22": 0.30468,
+ "23": 0.30434,
+ "24": 0.3055,
+ "25": 0.30394,
+ "26": 0.29844,
+ "27": 0.29422,
+ "28": 0.29421,
+ "29": 0.2939,
+ "30": 0.29373,
+ "31": 0.29428,
+ "32": 0.29216,
+ "33": 0.29359,
+ "34": 0.30409,
+ "35": 0.30316,
+ "36": 0.30365,
+ "37": 0.30415,
+ "38": 0.29751,
+ "39": 0.294,
+ "40": 0.30117,
+ "41": 0.30256,
+ "42": 0.3023,
+ "43": 0.30275,
+ "44": 0.30687,
+ "45": 0.30552,
+ "46": 0.30457,
+ "47": 0.30333,
+ "48": 0.30562,
+ "49": 0.30321,
+ "50": 0.30402
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss/model_config.yaml
index 92f4bfb1cdf..4aa6deabd64 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -50,4 +50,6 @@ MODEL_ARGS:
--attention-backend: flash
--log-memory-to-tensorboard: true
--calculate-per-token-loss: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_nondeterministic/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_nondeterministic/model_config.yaml
index 088ababb9cb..f1d41d4a22a 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_nondeterministic/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_nondeterministic/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -50,4 +50,6 @@ MODEL_ARGS:
--attention-backend: flash
--log-memory-to-tensorboard: true
--calculate-per-token-loss: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last/golden_values_dev_dgx_h100.json
index a470bf65873..b38a4e86cb7 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.86539,
- "2": 10.85871,
- "3": 10.86282,
- "4": 10.84007,
- "5": 10.87856,
- "6": 10.88852,
- "7": 10.86536,
- "8": 10.86015,
- "9": 10.85991,
- "10": 10.82982,
- "11": 10.88947,
- "12": 10.87511,
- "13": 10.87422,
- "14": 10.89675,
- "15": 10.82056,
- "16": 10.82497,
- "17": 10.78983,
- "18": 10.81029,
- "19": 10.80528,
- "20": 10.70396,
- "21": 10.6699,
- "22": 10.50641,
- "23": 10.69006,
- "24": 10.56312,
- "25": 10.49418,
- "26": 10.56627,
- "27": 10.58023,
- "28": 10.51571,
- "29": 10.55296,
- "30": 10.30551,
- "31": 10.02244,
- "32": 10.40618,
- "33": 10.39874,
- "34": 10.1377,
- "35": 10.20184,
- "36": 10.1605,
- "37": 10.28975,
- "38": 10.11483,
- "39": 10.361,
- "40": 10.01905,
- "41": 10.07292,
- "42": 10.14697,
- "43": 9.74684,
- "44": 9.87763,
- "45": 9.74962,
- "46": 9.73382,
- "47": 10.07536,
- "48": 9.78071,
- "49": 9.44783,
- "50": 9.8399
+ "1": 10.95937,
+ "2": 10.95597,
+ "3": 10.94131,
+ "4": 10.94667,
+ "5": 10.94232,
+ "6": 10.9384,
+ "7": 10.94616,
+ "8": 10.94246,
+ "9": 10.93783,
+ "10": 10.94034,
+ "11": 10.93147,
+ "12": 10.92721,
+ "13": 10.91651,
+ "14": 10.92439,
+ "15": 10.8876,
+ "16": 10.87828,
+ "17": 10.88554,
+ "18": 10.86856,
+ "19": 10.87826,
+ "20": 10.77113,
+ "21": 10.76393,
+ "22": 10.7496,
+ "23": 10.73998,
+ "24": 10.71066,
+ "25": 10.70289,
+ "26": 10.69316,
+ "27": 10.63313,
+ "28": 10.5774,
+ "29": 10.5472,
+ "30": 10.51617,
+ "31": 10.52164,
+ "32": 10.49521,
+ "33": 10.45901,
+ "34": 10.43664,
+ "35": 10.421,
+ "36": 10.40006,
+ "37": 10.37615,
+ "38": 10.38192,
+ "39": 10.3437,
+ "40": 10.33146,
+ "41": 10.30159,
+ "42": 10.27788,
+ "43": 10.25924,
+ "44": 10.2343,
+ "45": 10.24978,
+ "46": 10.20189,
+ "47": 10.19642,
+ "48": 10.1431,
+ "49": 10.14958,
+ "50": 10.1453
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 572.0,
- "2": 656.0,
- "3": 649.0,
- "4": 631.0,
- "5": 658.0,
- "6": 636.0,
- "7": 636.0,
- "8": 542.0,
- "9": 653.0,
- "10": 551.0,
- "11": 681.0,
- "12": 642.0,
- "13": 624.0,
+ "1": 602.0,
+ "2": 608.0,
+ "3": 625.0,
+ "4": 616.0,
+ "5": 644.0,
+ "6": 610.0,
+ "7": 647.0,
+ "8": 580.0,
+ "9": 624.0,
+ "10": 613.0,
+ "11": 631.0,
+ "12": 618.0,
+ "13": 636.0,
"14": 658.0,
- "15": 682.0,
- "16": 659.0,
- "17": 620.0,
- "18": 603.0,
- "19": 634.0,
- "20": 639.0,
- "21": 634.0,
- "22": 602.0,
- "23": 731.0,
- "24": 620.0,
- "25": 611.0,
- "26": 626.0,
- "27": 683.0,
- "28": 668.0,
- "29": 713.0,
- "30": 712.0,
- "31": 616.0,
- "32": 786.0,
- "33": 800.0,
- "34": 702.0,
- "35": 684.0,
- "36": 664.0,
- "37": 831.0,
- "38": 802.0,
- "39": 919.0,
- "40": 802.0,
- "41": 791.0,
- "42": 840.0,
- "43": 718.0,
- "44": 756.0,
- "45": 765.0,
- "46": 809.0,
- "47": 839.0,
- "48": 827.0,
- "49": 935.0,
- "50": 839.0
+ "15": 633.0,
+ "16": 619.0,
+ "17": 646.0,
+ "18": 641.0,
+ "19": 649.0,
+ "20": 612.0,
+ "21": 673.0,
+ "22": 643.0,
+ "23": 645.0,
+ "24": 683.0,
+ "25": 610.0,
+ "26": 651.0,
+ "27": 718.0,
+ "28": 683.0,
+ "29": 715.0,
+ "30": 736.0,
+ "31": 708.0,
+ "32": 716.0,
+ "33": 699.0,
+ "34": 741.0,
+ "35": 790.0,
+ "36": 727.0,
+ "37": 833.0,
+ "38": 778.0,
+ "39": 796.0,
+ "40": 843.0,
+ "41": 828.0,
+ "42": 748.0,
+ "43": 886.0,
+ "44": 825.0,
+ "45": 950.0,
+ "46": 873.0,
+ "47": 881.0,
+ "48": 983.0,
+ "49": 1051.0,
+ "50": 1022.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 510689792.0,
- "2": 510689792.0,
- "3": 510689792.0,
- "4": 510689792.0,
- "5": 510689792.0,
- "6": 510689792.0,
- "7": 510689792.0,
- "8": 510689792.0,
- "9": 510689792.0,
- "10": 510689792.0,
- "11": 510689792.0,
- "12": 510689792.0,
- "13": 510689792.0,
- "14": 510689792.0,
- "15": 510689792.0,
- "16": 510689792.0,
- "17": 510689792.0,
- "18": 510689792.0,
- "19": 510689792.0,
- "20": 510689792.0,
- "21": 510689792.0,
- "22": 510689792.0,
- "23": 510689792.0,
- "24": 510689792.0,
- "25": 510689792.0,
- "26": 510689792.0,
- "27": 510689792.0,
- "28": 510689792.0,
- "29": 510689792.0,
- "30": 510689792.0,
- "31": 510689792.0,
- "32": 510689792.0,
- "33": 510689792.0,
- "34": 510689792.0,
- "35": 510689792.0,
- "36": 510689792.0,
- "37": 510689792.0,
- "38": 510689792.0,
- "39": 510689792.0,
- "40": 510689792.0,
- "41": 510689792.0,
- "42": 510689792.0,
- "43": 510689792.0,
- "44": 510689792.0,
- "45": 510689792.0,
- "46": 510689792.0,
- "47": 510689792.0,
- "48": 510689792.0,
- "49": 510689792.0,
- "50": 510689792.0
+ "1": 511214080.0,
+ "2": 511214080.0,
+ "3": 511214080.0,
+ "4": 511214080.0,
+ "5": 511214080.0,
+ "6": 511214080.0,
+ "7": 511214080.0,
+ "8": 511214080.0,
+ "9": 511214080.0,
+ "10": 511214080.0,
+ "11": 511214080.0,
+ "12": 511214080.0,
+ "13": 511214080.0,
+ "14": 511214080.0,
+ "15": 511214080.0,
+ "16": 511214080.0,
+ "17": 511214080.0,
+ "18": 511214080.0,
+ "19": 511214080.0,
+ "20": 511214080.0,
+ "21": 511214080.0,
+ "22": 511214080.0,
+ "23": 511214080.0,
+ "24": 511214080.0,
+ "25": 511214080.0,
+ "26": 511214080.0,
+ "27": 511214080.0,
+ "28": 511214080.0,
+ "29": 511214080.0,
+ "30": 511214080.0,
+ "31": 511214080.0,
+ "32": 511214080.0,
+ "33": 511214080.0,
+ "34": 511214080.0,
+ "35": 511214080.0,
+ "36": 511214080.0,
+ "37": 511214080.0,
+ "38": 511214080.0,
+ "39": 511214080.0,
+ "40": 511214080.0,
+ "41": 511214080.0,
+ "42": 511214080.0,
+ "43": 511214080.0,
+ "44": 511214080.0,
+ "45": 511214080.0,
+ "46": 511214080.0,
+ "47": 511214080.0,
+ "48": 511214080.0,
+ "49": 511214080.0,
+ "50": 511214080.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 756752896.0,
- "2": 938398720.0,
- "3": 938398720.0,
- "4": 938398720.0,
- "5": 938398720.0,
- "6": 938398720.0,
- "7": 938398720.0,
- "8": 938398720.0,
- "9": 938398720.0,
- "10": 938398720.0,
- "11": 938398720.0,
- "12": 938398720.0,
- "13": 938398720.0,
- "14": 938398720.0,
- "15": 938398720.0,
- "16": 938399232.0,
- "17": 938399232.0,
- "18": 938399232.0,
- "19": 938399232.0,
- "20": 938399232.0,
- "21": 938399232.0,
- "22": 938399232.0,
- "23": 938399232.0,
- "24": 938399232.0,
- "25": 938399232.0,
- "26": 938399232.0,
- "27": 938399232.0,
- "28": 938399232.0,
- "29": 938399232.0,
- "30": 938399232.0,
- "31": 938399232.0,
- "32": 938399232.0,
- "33": 938399232.0,
- "34": 938399232.0,
- "35": 938399232.0,
- "36": 938399232.0,
- "37": 938399232.0,
- "38": 938399232.0,
- "39": 938399232.0,
- "40": 938399232.0,
- "41": 938399232.0,
- "42": 938399232.0,
- "43": 938399232.0,
- "44": 938399232.0,
- "45": 938399232.0,
- "46": 938399232.0,
- "47": 938399232.0,
- "48": 938399232.0,
- "49": 938399232.0,
- "50": 938399232.0
+ "1": 759896576.0,
+ "2": 935775232.0,
+ "3": 935776256.0,
+ "4": 935776256.0,
+ "5": 935776256.0,
+ "6": 935776256.0,
+ "7": 935776256.0,
+ "8": 935776256.0,
+ "9": 935776256.0,
+ "10": 935776256.0,
+ "11": 935776256.0,
+ "12": 935776256.0,
+ "13": 935777280.0,
+ "14": 935777280.0,
+ "15": 935777280.0,
+ "16": 935777280.0,
+ "17": 935777280.0,
+ "18": 935777280.0,
+ "19": 935777280.0,
+ "20": 935777280.0,
+ "21": 935777280.0,
+ "22": 935777280.0,
+ "23": 935777280.0,
+ "24": 935777280.0,
+ "25": 935777280.0,
+ "26": 935777280.0,
+ "27": 935777280.0,
+ "28": 935777280.0,
+ "29": 935777280.0,
+ "30": 935777280.0,
+ "31": 935777280.0,
+ "32": 935777280.0,
+ "33": 935777280.0,
+ "34": 935777280.0,
+ "35": 935777280.0,
+ "36": 935777280.0,
+ "37": 935777280.0,
+ "38": 935777280.0,
+ "39": 935777280.0,
+ "40": 935777280.0,
+ "41": 935777280.0,
+ "42": 935777280.0,
+ "43": 935777280.0,
+ "44": 935777280.0,
+ "45": 935777280.0,
+ "46": 935777280.0,
+ "47": 935777280.0,
+ "48": 935777280.0,
+ "49": 935777280.0,
+ "50": 935777280.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 37.38041,
- "2": 0.33426,
- "3": 0.30575,
- "4": 0.2855,
- "5": 0.28459,
- "6": 0.28581,
- "7": 0.28653,
- "8": 0.28649,
- "9": 0.28246,
- "10": 0.28538,
- "11": 0.28516,
- "12": 0.28331,
- "13": 0.28799,
- "14": 0.28438,
- "15": 0.28361,
- "16": 0.28315,
- "17": 0.2837,
- "18": 0.28279,
- "19": 0.28916,
- "20": 0.28613,
- "21": 0.2849,
- "22": 0.2837,
- "23": 0.2861,
- "24": 0.28551,
- "25": 0.28665,
- "26": 0.28308,
- "27": 0.28626,
- "28": 0.28139,
- "29": 0.28479,
- "30": 0.28557,
- "31": 0.28342,
- "32": 0.28058,
- "33": 0.2824,
- "34": 0.28129,
- "35": 0.28377,
- "36": 0.28273,
- "37": 0.28699,
- "38": 0.28388,
- "39": 0.28427,
- "40": 0.28442,
- "41": 0.28373,
- "42": 0.28177,
- "43": 0.28258,
- "44": 0.28237,
- "45": 0.2815,
- "46": 0.28503,
- "47": 0.2817,
- "48": 0.28433,
- "49": 0.28819,
- "50": 0.28371
+ "1": "nan",
+ "2": 10.6628,
+ "3": 0.32397,
+ "4": 0.30155,
+ "5": 0.30132,
+ "6": 0.30386,
+ "7": 0.30133,
+ "8": 0.30084,
+ "9": 0.30088,
+ "10": 0.29482,
+ "11": 0.29542,
+ "12": 0.29395,
+ "13": 0.29325,
+ "14": 0.2935,
+ "15": 0.29353,
+ "16": 0.29425,
+ "17": 0.294,
+ "18": 0.29539,
+ "19": 0.29461,
+ "20": 0.29402,
+ "21": 0.2938,
+ "22": 0.29397,
+ "23": 0.29592,
+ "24": 0.29366,
+ "25": 0.29324,
+ "26": 0.29479,
+ "27": 0.29459,
+ "28": 0.2944,
+ "29": 0.29489,
+ "30": 0.29506,
+ "31": 0.29432,
+ "32": 0.29461,
+ "33": 0.2951,
+ "34": 0.29546,
+ "35": 0.2946,
+ "36": 0.29893,
+ "37": 0.29547,
+ "38": 0.29576,
+ "39": 0.29658,
+ "40": 0.29466,
+ "41": 0.29385,
+ "42": 0.29533,
+ "43": 0.29583,
+ "44": 0.29319,
+ "45": 0.2948,
+ "46": 0.29468,
+ "47": 0.29469,
+ "48": 0.29379,
+ "49": 0.29317,
+ "50": 0.29558
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last/model_config.yaml
index 880d7fc7ce0..eb6ae776b50 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -53,4 +53,6 @@ MODEL_ARGS:
--use-tp-pp-dp-mapping: true
--expert-tensor-parallel-size: 4
--calculate-per-token-loss: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_nondeterministic_dp_last/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_nondeterministic_dp_last/model_config.yaml
index 013569c5882..98c4aefea36 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_nondeterministic_dp_last/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_nondeterministic_dp_last/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--use-tp-pp-dp-mapping: true
--expert-tensor-parallel-size: 4
--calculate-per-token-loss: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last/golden_values_dev_dgx_h100.json
index f78c3deb59d..2d126c9105b 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.86539,
- "2": 10.85871,
- "3": 10.86282,
- "4": 10.84007,
- "5": 10.87856,
- "6": 10.88852,
- "7": 10.86536,
- "8": 10.86015,
- "9": 10.85991,
- "10": 10.82982,
- "11": 10.88947,
- "12": 10.87511,
- "13": 10.87422,
- "14": 10.89675,
- "15": 10.82056,
- "16": 10.82497,
- "17": 10.78983,
- "18": 10.81029,
- "19": 10.80528,
- "20": 10.70396,
- "21": 10.6699,
- "22": 10.50641,
- "23": 10.69006,
- "24": 10.56312,
- "25": 10.49418,
- "26": 10.56627,
- "27": 10.58023,
- "28": 10.51571,
- "29": 10.55296,
- "30": 10.30551,
- "31": 10.02244,
- "32": 10.40618,
- "33": 10.39874,
- "34": 10.1377,
- "35": 10.20184,
- "36": 10.1605,
- "37": 10.28975,
- "38": 10.11483,
- "39": 10.361,
- "40": 10.01905,
- "41": 10.07292,
- "42": 10.14697,
- "43": 9.74684,
- "44": 9.87763,
- "45": 9.74962,
- "46": 9.73382,
- "47": 10.07536,
- "48": 9.78071,
- "49": 9.44783,
- "50": 9.8399
+ "1": 10.95937,
+ "2": 10.95597,
+ "3": 10.94131,
+ "4": 10.94667,
+ "5": 10.94232,
+ "6": 10.9384,
+ "7": 10.94616,
+ "8": 10.94246,
+ "9": 10.93783,
+ "10": 10.94034,
+ "11": 10.93147,
+ "12": 10.92721,
+ "13": 10.91651,
+ "14": 10.92439,
+ "15": 10.8876,
+ "16": 10.87828,
+ "17": 10.88554,
+ "18": 10.86856,
+ "19": 10.87826,
+ "20": 10.77113,
+ "21": 10.76393,
+ "22": 10.7496,
+ "23": 10.73998,
+ "24": 10.71066,
+ "25": 10.70289,
+ "26": 10.69316,
+ "27": 10.63313,
+ "28": 10.5774,
+ "29": 10.5472,
+ "30": 10.51617,
+ "31": 10.52164,
+ "32": 10.49521,
+ "33": 10.45901,
+ "34": 10.43664,
+ "35": 10.421,
+ "36": 10.40006,
+ "37": 10.37615,
+ "38": 10.38192,
+ "39": 10.3437,
+ "40": 10.33146,
+ "41": 10.30159,
+ "42": 10.27788,
+ "43": 10.25924,
+ "44": 10.2343,
+ "45": 10.24978,
+ "46": 10.20189,
+ "47": 10.19642,
+ "48": 10.1431,
+ "49": 10.14958,
+ "50": 10.1453
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 572.0,
- "2": 656.0,
- "3": 649.0,
- "4": 631.0,
- "5": 658.0,
- "6": 636.0,
- "7": 636.0,
- "8": 542.0,
- "9": 653.0,
- "10": 551.0,
- "11": 681.0,
- "12": 642.0,
- "13": 624.0,
+ "1": 602.0,
+ "2": 608.0,
+ "3": 625.0,
+ "4": 616.0,
+ "5": 644.0,
+ "6": 610.0,
+ "7": 647.0,
+ "8": 580.0,
+ "9": 624.0,
+ "10": 613.0,
+ "11": 631.0,
+ "12": 618.0,
+ "13": 636.0,
"14": 658.0,
- "15": 682.0,
- "16": 659.0,
- "17": 620.0,
- "18": 603.0,
- "19": 634.0,
- "20": 639.0,
- "21": 634.0,
- "22": 602.0,
- "23": 731.0,
- "24": 620.0,
- "25": 611.0,
- "26": 626.0,
- "27": 683.0,
- "28": 668.0,
- "29": 713.0,
- "30": 712.0,
- "31": 616.0,
- "32": 786.0,
- "33": 800.0,
- "34": 702.0,
- "35": 684.0,
- "36": 664.0,
- "37": 831.0,
- "38": 802.0,
- "39": 919.0,
- "40": 802.0,
- "41": 791.0,
- "42": 840.0,
- "43": 718.0,
- "44": 756.0,
- "45": 765.0,
- "46": 809.0,
- "47": 839.0,
- "48": 827.0,
- "49": 935.0,
- "50": 839.0
+ "15": 633.0,
+ "16": 619.0,
+ "17": 646.0,
+ "18": 641.0,
+ "19": 649.0,
+ "20": 612.0,
+ "21": 673.0,
+ "22": 643.0,
+ "23": 645.0,
+ "24": 683.0,
+ "25": 610.0,
+ "26": 651.0,
+ "27": 718.0,
+ "28": 683.0,
+ "29": 715.0,
+ "30": 736.0,
+ "31": 708.0,
+ "32": 716.0,
+ "33": 699.0,
+ "34": 741.0,
+ "35": 790.0,
+ "36": 727.0,
+ "37": 833.0,
+ "38": 778.0,
+ "39": 796.0,
+ "40": 843.0,
+ "41": 828.0,
+ "42": 748.0,
+ "43": 886.0,
+ "44": 825.0,
+ "45": 950.0,
+ "46": 873.0,
+ "47": 881.0,
+ "48": 983.0,
+ "49": 1051.0,
+ "50": 1022.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 512786944.0,
- "2": 512786944.0,
- "3": 512786944.0,
- "4": 512786944.0,
- "5": 512786944.0,
- "6": 512786944.0,
- "7": 512786944.0,
- "8": 512786944.0,
- "9": 512786944.0,
- "10": 512786944.0,
- "11": 512786944.0,
- "12": 512786944.0,
- "13": 512786944.0,
- "14": 512786944.0,
- "15": 512786944.0,
- "16": 512786944.0,
- "17": 512786944.0,
- "18": 512786944.0,
- "19": 512786944.0,
- "20": 512786944.0,
- "21": 512786944.0,
- "22": 512786944.0,
- "23": 512786944.0,
- "24": 512786944.0,
- "25": 512786944.0,
- "26": 512786944.0,
- "27": 512786944.0,
- "28": 512786944.0,
- "29": 512786944.0,
- "30": 512786944.0,
- "31": 512786944.0,
- "32": 512786944.0,
- "33": 512786944.0,
- "34": 512786944.0,
- "35": 512786944.0,
- "36": 512786944.0,
- "37": 512786944.0,
- "38": 512786944.0,
- "39": 512786944.0,
- "40": 512786944.0,
- "41": 512786944.0,
- "42": 512786944.0,
- "43": 512786944.0,
- "44": 512786944.0,
- "45": 512786944.0,
- "46": 512786944.0,
- "47": 512786944.0,
- "48": 512786944.0,
- "49": 512786944.0,
- "50": 512786944.0
+ "1": 511214080.0,
+ "2": 511214080.0,
+ "3": 511214080.0,
+ "4": 511214080.0,
+ "5": 511214080.0,
+ "6": 511214080.0,
+ "7": 511214080.0,
+ "8": 511214080.0,
+ "9": 511214080.0,
+ "10": 511214080.0,
+ "11": 511214080.0,
+ "12": 511214080.0,
+ "13": 511214080.0,
+ "14": 511214080.0,
+ "15": 511214080.0,
+ "16": 511214080.0,
+ "17": 511214080.0,
+ "18": 511214080.0,
+ "19": 511214080.0,
+ "20": 511214080.0,
+ "21": 511214080.0,
+ "22": 511214080.0,
+ "23": 511214080.0,
+ "24": 511214080.0,
+ "25": 511214080.0,
+ "26": 511214080.0,
+ "27": 511214080.0,
+ "28": 511214080.0,
+ "29": 511214080.0,
+ "30": 511214080.0,
+ "31": 511214080.0,
+ "32": 511214080.0,
+ "33": 511214080.0,
+ "34": 511214080.0,
+ "35": 511214080.0,
+ "36": 511214080.0,
+ "37": 511214080.0,
+ "38": 511214080.0,
+ "39": 511214080.0,
+ "40": 511214080.0,
+ "41": 511214080.0,
+ "42": 511214080.0,
+ "43": 511214080.0,
+ "44": 511214080.0,
+ "45": 511214080.0,
+ "46": 511214080.0,
+ "47": 511214080.0,
+ "48": 511214080.0,
+ "49": 511214080.0,
+ "50": 511214080.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 758851072.0,
- "2": 937350656.0,
- "3": 937350656.0,
- "4": 937350656.0,
- "5": 937350656.0,
- "6": 937350656.0,
- "7": 937350656.0,
- "8": 937350656.0,
- "9": 937350656.0,
- "10": 937350656.0,
- "11": 937350656.0,
- "12": 937350656.0,
- "13": 937350656.0,
- "14": 937350656.0,
- "15": 937350656.0,
- "16": 937350656.0,
- "17": 937350656.0,
- "18": 937350656.0,
- "19": 937350656.0,
- "20": 937350656.0,
- "21": 937350656.0,
- "22": 937350656.0,
- "23": 937350656.0,
- "24": 937350656.0,
- "25": 937350656.0,
- "26": 937350656.0,
- "27": 937350656.0,
- "28": 937350656.0,
- "29": 937350656.0,
- "30": 937350656.0,
- "31": 937350656.0,
- "32": 937350656.0,
- "33": 937350656.0,
- "34": 937350656.0,
- "35": 937350656.0,
- "36": 937350656.0,
- "37": 937350656.0,
- "38": 937350656.0,
- "39": 937350656.0,
- "40": 937350656.0,
- "41": 937350656.0,
- "42": 937350656.0,
- "43": 937350656.0,
- "44": 937350656.0,
- "45": 937350656.0,
- "46": 937350656.0,
- "47": 937350656.0,
- "48": 937350656.0,
- "49": 937350656.0,
- "50": 937350656.0
+ "1": 756753920.0,
+ "2": 936824320.0,
+ "3": 936824320.0,
+ "4": 936824320.0,
+ "5": 936824320.0,
+ "6": 936824320.0,
+ "7": 936824320.0,
+ "8": 936824320.0,
+ "9": 936824320.0,
+ "10": 936824320.0,
+ "11": 936824320.0,
+ "12": 936824320.0,
+ "13": 936824320.0,
+ "14": 936824320.0,
+ "15": 936824320.0,
+ "16": 936824320.0,
+ "17": 936824320.0,
+ "18": 936824320.0,
+ "19": 936824320.0,
+ "20": 936824320.0,
+ "21": 936824320.0,
+ "22": 936824320.0,
+ "23": 936824320.0,
+ "24": 936824320.0,
+ "25": 936824320.0,
+ "26": 936824320.0,
+ "27": 936824320.0,
+ "28": 936824320.0,
+ "29": 936824320.0,
+ "30": 936824320.0,
+ "31": 936824320.0,
+ "32": 936824320.0,
+ "33": 936824320.0,
+ "34": 936824320.0,
+ "35": 936824320.0,
+ "36": 936824320.0,
+ "37": 936824320.0,
+ "38": 936824320.0,
+ "39": 936824320.0,
+ "40": 936824320.0,
+ "41": 936824320.0,
+ "42": 936824320.0,
+ "43": 936824320.0,
+ "44": 936824320.0,
+ "45": 936824320.0,
+ "46": 936824320.0,
+ "47": 936824320.0,
+ "48": 936824320.0,
+ "49": 936824320.0,
+ "50": 936824320.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 35.82214,
- "2": 0.4147,
- "3": 0.32319,
- "4": 0.30032,
- "5": 0.30017,
- "6": 0.29443,
- "7": 0.29684,
- "8": 0.29654,
- "9": 0.29369,
- "10": 0.29144,
- "11": 0.29461,
- "12": 0.29494,
- "13": 0.2989,
- "14": 0.30075,
- "15": 0.30668,
- "16": 0.29656,
- "17": 0.29426,
- "18": 0.29342,
- "19": 0.29461,
- "20": 0.29689,
- "21": 0.29944,
- "22": 0.29592,
- "23": 0.29544,
- "24": 0.29391,
- "25": 0.29356,
- "26": 0.29086,
- "27": 0.29138,
- "28": 0.29613,
- "29": 0.29464,
- "30": 0.29623,
- "31": 0.29357,
- "32": 0.2941,
- "33": 0.29995,
- "34": 0.29721,
- "35": 0.29459,
- "36": 0.29391,
- "37": 0.29408,
- "38": 0.29673,
- "39": 0.2977,
- "40": 0.29439,
- "41": 0.29458,
- "42": 0.29561,
- "43": 0.29392,
- "44": 0.3078,
- "45": 0.29321,
- "46": 0.28828,
- "47": 0.28745,
- "48": 0.30287,
- "49": 0.28551,
- "50": 0.28747
+ "1": "nan",
+ "2": 9.663,
+ "3": 0.31101,
+ "4": 0.29378,
+ "5": 0.29536,
+ "6": 0.29318,
+ "7": 0.29023,
+ "8": 0.31228,
+ "9": 0.29377,
+ "10": 0.29238,
+ "11": 0.31033,
+ "12": 0.29245,
+ "13": 0.29248,
+ "14": 0.29722,
+ "15": 0.29237,
+ "16": 0.2925,
+ "17": 0.29261,
+ "18": 0.2925,
+ "19": 0.29247,
+ "20": 0.29597,
+ "21": 0.29945,
+ "22": 0.29493,
+ "23": 0.29363,
+ "24": 0.2942,
+ "25": 0.34279,
+ "26": 0.29184,
+ "27": 0.29217,
+ "28": 0.29367,
+ "29": 0.29287,
+ "30": 0.29283,
+ "31": 0.29262,
+ "32": 0.30918,
+ "33": 0.30312,
+ "34": 0.29337,
+ "35": 0.29867,
+ "36": 0.29307,
+ "37": 0.29414,
+ "38": 0.29626,
+ "39": 0.29258,
+ "40": 0.29239,
+ "41": 0.29221,
+ "42": 0.30855,
+ "43": 0.29955,
+ "44": 0.29581,
+ "45": 0.30432,
+ "46": 0.29276,
+ "47": 0.29452,
+ "48": 0.29853,
+ "49": 0.29311,
+ "50": 0.29453
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last/model_config.yaml
index f45345f9911..f798a99d703 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--log-memory-to-tensorboard: true
--use-tp-pp-dp-mapping: true
--expert-tensor-parallel-size: 4
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_nondeterministic_dp_last/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_nondeterministic_dp_last/model_config.yaml
index 8866fa67175..4b8aad6f105 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_nondeterministic_dp_last/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_nondeterministic_dp_last/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--log-memory-to-tensorboard: true
--use-tp-pp-dp-mapping: true
--expert-tensor-parallel-size: 4
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_nondeterministic/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_nondeterministic/model_config.yaml
index f4649e2d303..0216d5283d7 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_nondeterministic/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_nondeterministic/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: flash
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cross_entropy_loss_fusion/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cross_entropy_loss_fusion/model_config.yaml
index a77cd637800..2f73bf24f01 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cross_entropy_loss_fusion/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cross_entropy_loss_fusion/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -46,4 +46,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_ddp_average_in_collective/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_ddp_average_in_collective/model_config.yaml
index 9f416e74884..4eb20fa0683 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_ddp_average_in_collective/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_ddp_average_in_collective/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_defer_embedding_wgrad_compute/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_defer_embedding_wgrad_compute/model_config.yaml
index 2622612205a..bc37d8cd771 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_defer_embedding_wgrad_compute/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_defer_embedding_wgrad_compute/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -50,4 +50,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla/golden_values_dev_dgx_gb200.json
index 31729dd5fe5..047db9b77c3 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla/golden_values_dev_dgx_gb200.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.94863,
"2": 10.95748,
- "3": 10.95458,
- "4": 10.95314,
- "5": 10.94301,
- "6": 10.93709,
- "7": 10.94818,
- "8": 10.94698,
+ "3": 10.95457,
+ "4": 10.95313,
+ "5": 10.94299,
+ "6": 10.9371,
+ "7": 10.9482,
+ "8": 10.94697,
"9": 10.94866,
- "10": 10.95119,
+ "10": 10.95117,
"11": 10.9406,
- "12": 10.94105,
- "13": 10.94375,
+ "12": 10.94108,
+ "13": 10.94377,
"14": 10.94739,
- "15": 10.9429,
- "16": 10.93682,
- "17": 10.94182,
- "18": 10.93022,
+ "15": 10.94292,
+ "16": 10.93684,
+ "17": 10.94187,
+ "18": 10.93023,
"19": 10.93614,
- "20": 10.92135,
- "21": 10.91434,
- "22": 10.92114,
- "23": 10.92039,
- "24": 10.91062,
- "25": 10.91171,
- "26": 10.9101,
- "27": 10.90559,
+ "20": 10.92134,
+ "21": 10.91432,
+ "22": 10.92113,
+ "23": 10.92038,
+ "24": 10.91059,
+ "25": 10.9117,
+ "26": 10.91012,
+ "27": 10.90561,
"28": 10.87901,
- "29": 10.87862,
+ "29": 10.87865,
"30": 10.82431,
"31": 10.7917,
"32": 10.85763,
- "33": 10.85278,
- "34": 10.80465,
+ "33": 10.85279,
+ "34": 10.80469,
"35": 10.81124,
- "36": 10.79299,
- "37": 10.82161,
+ "36": 10.79298,
+ "37": 10.82159,
"38": 10.74654,
- "39": 10.79066,
- "40": 10.67639,
- "41": 10.71189,
- "42": 10.72663,
- "43": 10.58635,
- "44": 10.63487,
- "45": 10.59555,
- "46": 10.58202,
- "47": 10.67878,
- "48": 10.55683,
+ "39": 10.79069,
+ "40": 10.67643,
+ "41": 10.71186,
+ "42": 10.72666,
+ "43": 10.58633,
+ "44": 10.63486,
+ "45": 10.59559,
+ "46": 10.58203,
+ "47": 10.67874,
+ "48": 10.55684,
"49": 10.43321,
- "50": 10.57623
+ "50": 10.57624
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 22792076.0,
- "2": 22989660.0,
- "3": 22661158.0,
- "4": 23283080.0,
- "5": 22778860.0,
- "6": 23085232.0,
- "7": 22834892.0,
- "8": 22990502.0,
- "9": 22906480.0,
- "10": 22983488.0,
- "11": 22563552.0,
- "12": 22523694.0,
- "13": 22980968.0,
- "14": 22453154.0,
- "15": 22885546.0,
- "16": 22895028.0,
- "17": 22882956.0,
- "18": 22647168.0,
- "19": 22682056.0,
- "20": 22757444.0,
- "21": 22803808.0,
- "22": 22864026.0,
- "23": 22603204.0,
- "24": 22835232.0,
- "25": 22883270.0,
- "26": 22611998.0,
- "27": 22532132.0,
- "28": 22516960.0,
- "29": 22593572.0,
- "30": 22695024.0,
- "31": 23019244.0,
- "32": 22648204.0,
- "33": 22623192.0,
- "34": 22899922.0,
- "35": 22852560.0,
- "36": 22652964.0,
- "37": 22559866.0,
- "38": 22960222.0,
- "39": 22864432.0,
- "40": 22721420.0,
- "41": 22722086.0,
- "42": 22730128.0,
- "43": 23040178.0,
- "44": 22809816.0,
- "45": 22738252.0,
- "46": 22947510.0,
- "47": 22697018.0,
- "48": 22992168.0,
- "49": 22790946.0,
- "50": 22969044.0
+ "1": 22792008.0,
+ "2": 22989688.0,
+ "3": 22661140.0,
+ "4": 23283044.0,
+ "5": 22778806.0,
+ "6": 23085212.0,
+ "7": 22834896.0,
+ "8": 22990508.0,
+ "9": 22906460.0,
+ "10": 22983366.0,
+ "11": 22563548.0,
+ "12": 22523680.0,
+ "13": 22980988.0,
+ "14": 22453176.0,
+ "15": 22885472.0,
+ "16": 22894992.0,
+ "17": 22882968.0,
+ "18": 22647152.0,
+ "19": 22682046.0,
+ "20": 22757300.0,
+ "21": 22803894.0,
+ "22": 22864052.0,
+ "23": 22603212.0,
+ "24": 22835274.0,
+ "25": 22883332.0,
+ "26": 22612002.0,
+ "27": 22532080.0,
+ "28": 22516992.0,
+ "29": 22593624.0,
+ "30": 22694994.0,
+ "31": 23019280.0,
+ "32": 22648240.0,
+ "33": 22623122.0,
+ "34": 22900010.0,
+ "35": 22852524.0,
+ "36": 22652920.0,
+ "37": 22559892.0,
+ "38": 22960296.0,
+ "39": 22864440.0,
+ "40": 22721274.0,
+ "41": 22721990.0,
+ "42": 22730036.0,
+ "43": 23040186.0,
+ "44": 22809892.0,
+ "45": 22738188.0,
+ "46": 22947462.0,
+ "47": 22696952.0,
+ "48": 22992052.0,
+ "49": 22791010.0,
+ "50": 22969004.0
}
},
"mem-allocated-bytes": {
@@ -175,7 +175,7 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1122385408.0,
+ "1": 1123433984.0,
"2": 1245635072.0,
"3": 1245635072.0,
"4": 1245635072.0,
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.75152,
- "3": 0.1678,
- "4": 0.1543,
- "5": 0.15772,
- "6": 0.15798,
- "7": 0.15886,
- "8": 0.16038,
- "9": 0.15983,
- "10": 0.16009,
- "11": 0.15881,
- "12": 0.16004,
- "13": 0.15648,
- "14": 0.15396,
- "15": 0.15394,
- "16": 0.1544,
- "17": 0.15329,
- "18": 0.1539,
- "19": 0.15442,
- "20": 0.1521,
- "21": 0.15368,
- "22": 0.15287,
- "23": 0.15397,
- "24": 0.15553,
- "25": 0.15617,
- "26": 0.15925,
- "27": 0.145,
- "28": 0.14456,
- "29": 0.14869,
- "30": 0.15407,
- "31": 0.15556,
- "32": 0.15651,
- "33": 0.15726,
- "34": 0.1574,
- "35": 0.15981,
- "36": 0.16037,
- "37": 0.16044,
- "38": 0.15744,
- "39": 0.15875,
- "40": 0.15964,
- "41": 0.15984,
- "42": 0.1605,
- "43": 0.15901,
- "44": 0.16037,
- "45": 0.1616,
- "46": 0.16046,
- "47": 0.16125,
- "48": 0.16168,
- "49": 0.1611,
- "50": 0.15977
+ "2": 5.70034,
+ "3": 0.16043,
+ "4": 0.14515,
+ "5": 0.14665,
+ "6": 0.14743,
+ "7": 0.14773,
+ "8": 0.14897,
+ "9": 0.14739,
+ "10": 0.15002,
+ "11": 0.14783,
+ "12": 0.14932,
+ "13": 0.14731,
+ "14": 0.14893,
+ "15": 0.14704,
+ "16": 0.14861,
+ "17": 0.14821,
+ "18": 0.14808,
+ "19": 0.14853,
+ "20": 0.1507,
+ "21": 0.14953,
+ "22": 0.15066,
+ "23": 0.14924,
+ "24": 0.15017,
+ "25": 0.15085,
+ "26": 0.18249,
+ "27": 0.15492,
+ "28": 0.15306,
+ "29": 0.14968,
+ "30": 0.15095,
+ "31": 0.15033,
+ "32": 0.15059,
+ "33": 0.1505,
+ "34": 0.1502,
+ "35": 0.14853,
+ "36": 0.15099,
+ "37": 0.1478,
+ "38": 0.1494,
+ "39": 0.14947,
+ "40": 0.15074,
+ "41": 0.14956,
+ "42": 0.1512,
+ "43": 0.14912,
+ "44": 0.15691,
+ "45": 0.14931,
+ "46": 0.15116,
+ "47": 0.14871,
+ "48": 0.14949,
+ "49": 0.15211,
+ "50": 0.15069
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla/golden_values_dev_dgx_h100.json
index bb945f7d249..454d9ef840f 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.92337,
- "2": 10.91811,
- "3": 10.91506,
- "4": 10.92436,
- "5": 10.92089,
- "6": 10.92887,
- "7": 10.92681,
- "8": 10.91989,
- "9": 10.92227,
- "10": 10.92192,
- "11": 10.918,
- "12": 10.9238,
- "13": 10.92406,
- "14": 10.90862,
- "15": 10.92351,
- "16": 10.91807,
- "17": 10.9154,
- "18": 10.91265,
- "19": 10.9091,
- "20": 10.90031,
- "21": 10.8959,
- "22": 10.8828,
- "23": 10.89975,
- "24": 10.88437,
- "25": 10.87827,
- "26": 10.88155,
- "27": 10.88649,
- "28": 10.85679,
- "29": 10.85657,
- "30": 10.81423,
- "31": 10.76651,
- "32": 10.83131,
- "33": 10.83158,
- "34": 10.78071,
- "35": 10.78865,
- "36": 10.78003,
- "37": 10.80446,
- "38": 10.72434,
- "39": 10.78066,
- "40": 10.65927,
- "41": 10.69209,
- "42": 10.70974,
- "43": 10.56129,
- "44": 10.61371,
- "45": 10.56874,
- "46": 10.54454,
- "47": 10.66751,
- "48": 10.53792,
- "49": 10.40859,
- "50": 10.5542
+ "1": 10.92858,
+ "2": 10.92691,
+ "3": 10.9228,
+ "4": 10.92385,
+ "5": 10.92018,
+ "6": 10.92481,
+ "7": 10.91599,
+ "8": 10.92353,
+ "9": 10.91957,
+ "10": 10.92508,
+ "11": 10.92005,
+ "12": 10.92356,
+ "13": 10.91465,
+ "14": 10.91785,
+ "15": 10.91656,
+ "16": 10.92311,
+ "17": 10.91335,
+ "18": 10.91241,
+ "19": 10.90857,
+ "20": 10.90756,
+ "21": 10.89514,
+ "22": 10.89722,
+ "23": 10.89726,
+ "24": 10.89583,
+ "25": 10.90398,
+ "26": 10.89488,
+ "27": 10.8879,
+ "28": 10.86573,
+ "29": 10.85898,
+ "30": 10.85708,
+ "31": 10.8477,
+ "32": 10.86078,
+ "33": 10.84973,
+ "34": 10.83676,
+ "35": 10.84444,
+ "36": 10.83031,
+ "37": 10.82693,
+ "38": 10.8041,
+ "39": 10.77932,
+ "40": 10.77164,
+ "41": 10.75996,
+ "42": 10.75927,
+ "43": 10.75402,
+ "44": 10.74359,
+ "45": 10.7392,
+ "46": 10.71447,
+ "47": 10.71009,
+ "48": 10.68539,
+ "49": 10.69156,
+ "50": 10.6792
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 22791636.0,
- "2": 22989424.0,
- "3": 22661212.0,
- "4": 23283558.0,
- "5": 22778528.0,
- "6": 23085340.0,
- "7": 22834596.0,
- "8": 22990452.0,
- "9": 22906466.0,
- "10": 22983232.0,
- "11": 22564584.0,
- "12": 22524010.0,
- "13": 22981124.0,
- "14": 22453096.0,
- "15": 22886400.0,
- "16": 22895424.0,
- "17": 22883736.0,
- "18": 22647090.0,
- "19": 22682526.0,
- "20": 22758358.0,
- "21": 22804276.0,
- "22": 22863814.0,
- "23": 22603616.0,
- "24": 22835172.0,
- "25": 22883742.0,
- "26": 22611358.0,
- "27": 22532968.0,
- "28": 22517794.0,
- "29": 22593448.0,
- "30": 22695256.0,
- "31": 23019472.0,
- "32": 22648896.0,
- "33": 22622516.0,
- "34": 22899620.0,
- "35": 22851572.0,
- "36": 22653160.0,
- "37": 22560476.0,
- "38": 22960058.0,
- "39": 22865476.0,
- "40": 22721690.0,
- "41": 22723112.0,
- "42": 22730692.0,
- "43": 23039608.0,
- "44": 22809964.0,
- "45": 22738932.0,
- "46": 22948360.0,
- "47": 22696800.0,
- "48": 22992776.0,
- "49": 22791104.0,
- "50": 22968342.0
+ "1": 23025846.0,
+ "2": 22914698.0,
+ "3": 22773682.0,
+ "4": 22842060.0,
+ "5": 22847304.0,
+ "6": 22810108.0,
+ "7": 22935284.0,
+ "8": 22675112.0,
+ "9": 22822952.0,
+ "10": 22552496.0,
+ "11": 22818072.0,
+ "12": 22705452.0,
+ "13": 23386520.0,
+ "14": 23055528.0,
+ "15": 22782176.0,
+ "16": 22886556.0,
+ "17": 22992588.0,
+ "18": 23059260.0,
+ "19": 23148580.0,
+ "20": 22786812.0,
+ "21": 22982428.0,
+ "22": 23009472.0,
+ "23": 22692978.0,
+ "24": 22926428.0,
+ "25": 22699780.0,
+ "26": 23068192.0,
+ "27": 22865306.0,
+ "28": 23063146.0,
+ "29": 23043372.0,
+ "30": 23007928.0,
+ "31": 22972092.0,
+ "32": 22730576.0,
+ "33": 22804756.0,
+ "34": 23140008.0,
+ "35": 22814240.0,
+ "36": 22759076.0,
+ "37": 23167156.0,
+ "38": 23028990.0,
+ "39": 23048716.0,
+ "40": 22813580.0,
+ "41": 23128926.0,
+ "42": 22752828.0,
+ "43": 23050572.0,
+ "44": 22768242.0,
+ "45": 22910502.0,
+ "46": 22797120.0,
+ "47": 22912740.0,
+ "48": 22897256.0,
+ "49": 22951712.0,
+ "50": 22707582.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 387483136.0,
- "2": 387483136.0,
- "3": 387483136.0,
- "4": 387483136.0,
- "5": 387483136.0,
- "6": 387483136.0,
- "7": 387483136.0,
- "8": 387483136.0,
- "9": 387483136.0,
- "10": 387483136.0,
- "11": 387483136.0,
- "12": 387483136.0,
- "13": 387483136.0,
- "14": 387483136.0,
- "15": 387483136.0,
- "16": 387483136.0,
- "17": 387483136.0,
- "18": 387483136.0,
- "19": 387483136.0,
- "20": 387483136.0,
- "21": 387483136.0,
- "22": 387483136.0,
- "23": 387483136.0,
- "24": 387483136.0,
- "25": 387483136.0,
- "26": 387483136.0,
- "27": 387483136.0,
- "28": 387483136.0,
- "29": 387483136.0,
- "30": 387483136.0,
- "31": 387483136.0,
- "32": 387483136.0,
- "33": 387483136.0,
- "34": 387483136.0,
- "35": 387483136.0,
- "36": 387483136.0,
- "37": 387483136.0,
- "38": 387483136.0,
- "39": 387483136.0,
- "40": 387483136.0,
- "41": 387483136.0,
- "42": 387483136.0,
- "43": 387483136.0,
- "44": 387483136.0,
- "45": 387483136.0,
- "46": 387483136.0,
- "47": 387483136.0,
- "48": 387483136.0,
- "49": 387483136.0,
- "50": 387483136.0
+ "1": 388269568.0,
+ "2": 388269568.0,
+ "3": 388269568.0,
+ "4": 388269568.0,
+ "5": 388269568.0,
+ "6": 388269568.0,
+ "7": 388269568.0,
+ "8": 388269568.0,
+ "9": 388269568.0,
+ "10": 388269568.0,
+ "11": 388269568.0,
+ "12": 388269568.0,
+ "13": 388269568.0,
+ "14": 388269568.0,
+ "15": 388269568.0,
+ "16": 388269568.0,
+ "17": 388269568.0,
+ "18": 388269568.0,
+ "19": 388269568.0,
+ "20": 388269568.0,
+ "21": 388269568.0,
+ "22": 388269568.0,
+ "23": 388269568.0,
+ "24": 388269568.0,
+ "25": 388269568.0,
+ "26": 388269568.0,
+ "27": 388269568.0,
+ "28": 388269568.0,
+ "29": 388269568.0,
+ "30": 388269568.0,
+ "31": 388269568.0,
+ "32": 388269568.0,
+ "33": 388269568.0,
+ "34": 388269568.0,
+ "35": 388269568.0,
+ "36": 388269568.0,
+ "37": 388269568.0,
+ "38": 388269568.0,
+ "39": 388269568.0,
+ "40": 388269568.0,
+ "41": 388269568.0,
+ "42": 388269568.0,
+ "43": 388269568.0,
+ "44": 388269568.0,
+ "45": 388269568.0,
+ "46": 388269568.0,
+ "47": 388269568.0,
+ "48": 388269568.0,
+ "49": 388269568.0,
+ "50": 388269568.0
}
},
"mem-max-allocated-bytes": {
@@ -176,55 +176,55 @@
"step_interval": 1,
"values": {
"1": 1122385408.0,
- "2": 1245635072.0,
- "3": 1245635072.0,
- "4": 1245635072.0,
- "5": 1245635072.0,
- "6": 1245635072.0,
- "7": 1245635072.0,
- "8": 1245635072.0,
- "9": 1245635072.0,
- "10": 1245635072.0,
- "11": 1245635072.0,
- "12": 1245635072.0,
- "13": 1245635072.0,
- "14": 1245635072.0,
- "15": 1245635072.0,
- "16": 1245635072.0,
- "17": 1245635072.0,
- "18": 1245635072.0,
- "19": 1245635072.0,
- "20": 1245635072.0,
- "21": 1245635072.0,
- "22": 1245635072.0,
- "23": 1245635072.0,
- "24": 1245635072.0,
- "25": 1245635072.0,
- "26": 1245635072.0,
- "27": 1245635072.0,
- "28": 1245635072.0,
- "29": 1245635072.0,
- "30": 1245635072.0,
- "31": 1245635072.0,
- "32": 1245635072.0,
- "33": 1245635072.0,
- "34": 1245635072.0,
- "35": 1245635072.0,
- "36": 1245635072.0,
- "37": 1245635072.0,
- "38": 1245635072.0,
- "39": 1245635072.0,
- "40": 1245635072.0,
- "41": 1245635072.0,
- "42": 1245635072.0,
- "43": 1245635072.0,
- "44": 1245635072.0,
- "45": 1245635072.0,
- "46": 1245635072.0,
- "47": 1245635072.0,
- "48": 1245635072.0,
- "49": 1245635072.0,
- "50": 1245635072.0
+ "2": 1246421504.0,
+ "3": 1246421504.0,
+ "4": 1246421504.0,
+ "5": 1246421504.0,
+ "6": 1246421504.0,
+ "7": 1246421504.0,
+ "8": 1246421504.0,
+ "9": 1246421504.0,
+ "10": 1246421504.0,
+ "11": 1246421504.0,
+ "12": 1246421504.0,
+ "13": 1246421504.0,
+ "14": 1246421504.0,
+ "15": 1246421504.0,
+ "16": 1246421504.0,
+ "17": 1246421504.0,
+ "18": 1246421504.0,
+ "19": 1246421504.0,
+ "20": 1246421504.0,
+ "21": 1246421504.0,
+ "22": 1246421504.0,
+ "23": 1246421504.0,
+ "24": 1246421504.0,
+ "25": 1246421504.0,
+ "26": 1248256512.0,
+ "27": 1248256512.0,
+ "28": 1248256512.0,
+ "29": 1248256512.0,
+ "30": 1248256512.0,
+ "31": 1248256512.0,
+ "32": 1248256512.0,
+ "33": 1248256512.0,
+ "34": 1248256512.0,
+ "35": 1248256512.0,
+ "36": 1248256512.0,
+ "37": 1248256512.0,
+ "38": 1248256512.0,
+ "39": 1248256512.0,
+ "40": 1248256512.0,
+ "41": 1248256512.0,
+ "42": 1248256512.0,
+ "43": 1248256512.0,
+ "44": 1248256512.0,
+ "45": 1248256512.0,
+ "46": 1248256512.0,
+ "47": 1248256512.0,
+ "48": 1248256512.0,
+ "49": 1248256512.0,
+ "50": 1248256512.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 11.55479,
- "2": 0.135,
- "3": 0.11559,
- "4": 0.10311,
- "5": 0.10091,
- "6": 0.10054,
- "7": 0.10125,
- "8": 0.10194,
- "9": 0.10124,
- "10": 0.10175,
- "11": 0.10044,
- "12": 0.10706,
- "13": 0.10279,
- "14": 0.10111,
- "15": 0.10071,
- "16": 0.10185,
- "17": 0.10255,
- "18": 0.10134,
- "19": 0.10086,
- "20": 0.10058,
- "21": 0.10136,
- "22": 0.09986,
- "23": 0.10128,
- "24": 0.1004,
- "25": 0.10123,
- "26": 0.10374,
- "27": 0.09272,
- "28": 0.09193,
- "29": 0.09389,
- "30": 0.09165,
- "31": 0.09164,
- "32": 0.09201,
- "33": 0.09402,
- "34": 0.09129,
- "35": 0.09235,
- "36": 0.09303,
- "37": 0.09091,
- "38": 0.09089,
- "39": 0.09141,
- "40": 0.09122,
- "41": 0.0948,
- "42": 0.09477,
- "43": 0.09276,
- "44": 0.09423,
- "45": 0.09477,
- "46": 0.09451,
- "47": 0.0941,
- "48": 0.0934,
- "49": 0.09315,
- "50": 0.09366
+ "1": "nan",
+ "2": 5.61666,
+ "3": 0.12576,
+ "4": 0.09768,
+ "5": 0.09954,
+ "6": 0.10543,
+ "7": 0.09742,
+ "8": 0.0969,
+ "9": 0.10618,
+ "10": 0.09709,
+ "11": 0.09844,
+ "12": 0.10347,
+ "13": 0.09923,
+ "14": 0.09507,
+ "15": 0.0958,
+ "16": 0.09613,
+ "17": 0.09598,
+ "18": 0.09573,
+ "19": 0.09531,
+ "20": 0.09562,
+ "21": 0.0986,
+ "22": 0.09731,
+ "23": 0.0957,
+ "24": 0.09542,
+ "25": 0.09554,
+ "26": 0.13286,
+ "27": 0.0952,
+ "28": 0.09589,
+ "29": 0.09691,
+ "30": 0.09559,
+ "31": 0.09529,
+ "32": 0.09477,
+ "33": 0.09505,
+ "34": 0.09496,
+ "35": 0.09568,
+ "36": 0.09571,
+ "37": 0.09524,
+ "38": 0.09569,
+ "39": 0.09513,
+ "40": 0.09502,
+ "41": 0.09853,
+ "42": 0.09704,
+ "43": 0.09656,
+ "44": 0.09651,
+ "45": 0.09535,
+ "46": 0.09598,
+ "47": 0.09575,
+ "48": 0.10029,
+ "49": 0.09516,
+ "50": 0.09604
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla/model_config.yaml
index 00f01d3bac0..d8afdbf0756 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla/model_config.yaml
@@ -27,9 +27,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -56,4 +56,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_no_create_attention_mask_in_dataloader/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_no_create_attention_mask_in_dataloader/model_config.yaml
index d6384e7f604..023fc7b6e5b 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_no_create_attention_mask_in_dataloader/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_no_create_attention_mask_in_dataloader/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -50,4 +50,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_no_mmap_bin_files/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_no_mmap_bin_files/model_config.yaml
index d4939a8c2cf..9e0518f9ffd 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_no_mmap_bin_files/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_no_mmap_bin_files/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -50,4 +50,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_gb200.json
index caa1e54ee64..b7ced9b3ff9 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_gb200.json
@@ -6,104 +6,104 @@
"values": {
"1": 10.86897,
"2": 10.88544,
- "3": 10.86473,
- "4": 10.86826,
- "5": 10.87436,
+ "3": 10.86477,
+ "4": 10.86825,
+ "5": 10.87437,
"6": 10.89005,
- "7": 10.87769,
+ "7": 10.87768,
"8": 10.86364,
- "9": 10.88282,
- "10": 10.84687,
- "11": 10.87102,
- "12": 10.87345,
+ "9": 10.8828,
+ "10": 10.84683,
+ "11": 10.87103,
+ "12": 10.87356,
"13": 10.8814,
- "14": 10.8877,
- "15": 10.83869,
+ "14": 10.88767,
+ "15": 10.83862,
"16": 10.8239,
- "17": 10.80197,
- "18": 10.81094,
- "19": 10.82192,
- "20": 10.71791,
- "21": 10.68914,
- "22": 10.57271,
- "23": 10.7081,
+ "17": 10.80195,
+ "18": 10.81091,
+ "19": 10.82189,
+ "20": 10.71788,
+ "21": 10.6892,
+ "22": 10.57272,
+ "23": 10.70811,
"24": 10.59543,
- "25": 10.55292,
- "26": 10.61257,
- "27": 10.60051,
- "28": 10.56173,
- "29": 10.58089,
- "30": 10.35595,
- "31": 10.1182,
- "32": 10.44815,
- "33": 10.4542,
- "34": 10.21553,
- "35": 10.26124,
- "36": 10.20776,
- "37": 10.33673,
+ "25": 10.55295,
+ "26": 10.61259,
+ "27": 10.60052,
+ "28": 10.56168,
+ "29": 10.58085,
+ "30": 10.35597,
+ "31": 10.11819,
+ "32": 10.44814,
+ "33": 10.45418,
+ "34": 10.21548,
+ "35": 10.26125,
+ "36": 10.20779,
+ "37": 10.33671,
"38": 10.17741,
- "39": 10.39297,
- "40": 10.06349,
- "41": 10.13887,
- "42": 10.2056,
- "43": 9.82809,
- "44": 9.94547,
- "45": 9.82561,
- "46": 9.80186,
- "47": 10.14049,
- "48": 9.84276,
- "49": 9.52016,
- "50": 9.88454,
- "51": 9.84743,
- "52": 9.74209,
- "53": 10.05697,
- "54": 9.9505,
- "55": 9.88145,
- "56": 9.61274,
+ "39": 10.39294,
+ "40": 10.06346,
+ "41": 10.13891,
+ "42": 10.20558,
+ "43": 9.82807,
+ "44": 9.94542,
+ "45": 9.82557,
+ "46": 9.80182,
+ "47": 10.14053,
+ "48": 9.8428,
+ "49": 9.52012,
+ "50": 9.88456,
+ "51": 9.84742,
+ "52": 9.74207,
+ "53": 10.05695,
+ "54": 9.95053,
+ "55": 9.8814,
+ "56": 9.61275,
"57": 9.4687,
- "58": 9.82193,
+ "58": 9.82189,
"59": 9.57642,
- "60": 9.49762,
- "61": 9.69189,
+ "60": 9.4976,
+ "61": 9.69188,
"62": 9.9867,
- "63": 9.37512,
- "64": 9.76679,
- "65": 8.94648,
- "66": 9.7023,
- "67": 9.36326,
- "68": 9.7831,
- "69": 9.7986,
- "70": 9.7317,
- "71": 9.62571,
- "72": 9.58488,
- "73": 9.48967,
- "74": 8.9286,
- "75": 9.40862,
- "76": 9.07925,
- "77": 10.0594,
- "78": 9.72288,
- "79": 9.37784,
- "80": 9.40429,
- "81": 9.48309,
- "82": 9.7004,
- "83": 9.31595,
- "84": 9.41838,
- "85": 9.61685,
- "86": 9.07533,
- "87": 9.59616,
- "88": 9.75215,
- "89": 9.60184,
- "90": 9.82281,
- "91": 9.34037,
+ "63": 9.37509,
+ "64": 9.76682,
+ "65": 8.94649,
+ "66": 9.70228,
+ "67": 9.36325,
+ "68": 9.78311,
+ "69": 9.79862,
+ "70": 9.73172,
+ "71": 9.62574,
+ "72": 9.58486,
+ "73": 9.48964,
+ "74": 8.92859,
+ "75": 9.40865,
+ "76": 9.07927,
+ "77": 10.05938,
+ "78": 9.72283,
+ "79": 9.37782,
+ "80": 9.40428,
+ "81": 9.48312,
+ "82": 9.70038,
+ "83": 9.3159,
+ "84": 9.41834,
+ "85": 9.61684,
+ "86": 9.07537,
+ "87": 9.59617,
+ "88": 9.75218,
+ "89": 9.60187,
+ "90": 9.82286,
+ "91": 9.34039,
"92": 9.35854,
- "93": 9.08805,
- "94": 8.83037,
+ "93": 9.08806,
+ "94": 8.83038,
"95": 9.5266,
- "96": 9.53049,
- "97": 9.30389,
- "98": 9.67196,
- "99": 8.89637,
- "100": 9.40644
+ "96": 9.53051,
+ "97": 9.30391,
+ "98": 9.67199,
+ "99": 8.8964,
+ "100": 9.40645
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1621.0,
- "2": 1657.0,
- "3": 1580.0,
- "4": 1839.0,
- "5": 1862.0,
- "6": 1724.0,
- "7": 1714.0,
- "8": 1670.0,
- "9": 1762.0,
- "10": 1358.0,
- "11": 1734.0,
- "12": 1682.0,
- "13": 1761.0,
- "14": 1731.0,
- "15": 1788.0,
- "16": 1801.0,
- "17": 1866.0,
- "18": 1636.0,
- "19": 1709.0,
- "20": 1607.0,
- "21": 1821.0,
- "22": 1666.0,
- "23": 1991.0,
- "24": 1585.0,
- "25": 1587.0,
- "26": 1631.0,
- "27": 1714.0,
- "28": 1966.0,
- "29": 1997.0,
- "30": 1851.0,
- "31": 1581.0,
- "32": 1864.0,
- "33": 2107.0,
- "34": 1846.0,
- "35": 1982.0,
- "36": 1904.0,
- "37": 2373.0,
- "38": 2172.0,
- "39": 2343.0,
- "40": 2149.0,
- "41": 2331.0,
- "42": 2199.0,
- "43": 1914.0,
- "44": 2065.0,
- "45": 2081.0,
- "46": 2352.0,
- "47": 2497.0,
- "48": 2303.0,
- "49": 2346.0,
- "50": 2411.0,
- "51": 2491.0,
- "52": 2552.0,
- "53": 2980.0,
- "54": 2680.0,
- "55": 2274.0,
- "56": 2734.0,
- "57": 2319.0,
- "58": 2907.0,
- "59": 2886.0,
- "60": 2566.0,
- "61": 2855.0,
- "62": 2704.0,
- "63": 2370.0,
- "64": 2998.0,
- "65": 2563.0,
- "66": 2868.0,
- "67": 2762.0,
- "68": 2739.0,
- "69": 2730.0,
- "70": 3156.0,
- "71": 2803.0,
- "72": 2506.0,
- "73": 2896.0,
- "74": 1937.0,
- "75": 2450.0,
- "76": 2794.0,
- "77": 3047.0,
- "78": 3104.0,
- "79": 3069.0,
- "80": 3286.0,
- "81": 3543.0,
- "82": 3192.0,
- "83": 2614.0,
- "84": 3273.0,
- "85": 3111.0,
- "86": 2680.0,
- "87": 3654.0,
- "88": 3117.0,
- "89": 3351.0,
- "90": 3086.0,
- "91": 2721.0,
- "92": 3045.0,
- "93": 2672.0,
- "94": 3326.0,
- "95": 3125.0,
- "96": 3309.0,
- "97": 3208.0,
- "98": 3572.0,
- "99": 2980.0,
- "100": 3355.0
+ "1": 1592.0,
+ "2": 1638.0,
+ "3": 1658.0,
+ "4": 1737.0,
+ "5": 1836.0,
+ "6": 1740.0,
+ "7": 1780.0,
+ "8": 1628.0,
+ "9": 1785.0,
+ "10": 1372.0,
+ "11": 1738.0,
+ "12": 1745.0,
+ "13": 1808.0,
+ "14": 1709.0,
+ "15": 1798.0,
+ "16": 1765.0,
+ "17": 1875.0,
+ "18": 1580.0,
+ "19": 1744.0,
+ "20": 1573.0,
+ "21": 1835.0,
+ "22": 1649.0,
+ "23": 1993.0,
+ "24": 1600.0,
+ "25": 1527.0,
+ "26": 1592.0,
+ "27": 1818.0,
+ "28": 1889.0,
+ "29": 1909.0,
+ "30": 1889.0,
+ "31": 1635.0,
+ "32": 1847.0,
+ "33": 2066.0,
+ "34": 1761.0,
+ "35": 1861.0,
+ "36": 1916.0,
+ "37": 2298.0,
+ "38": 2162.0,
+ "39": 2226.0,
+ "40": 2127.0,
+ "41": 2226.0,
+ "42": 2204.0,
+ "43": 1844.0,
+ "44": 2106.0,
+ "45": 2056.0,
+ "46": 2238.0,
+ "47": 2476.0,
+ "48": 2289.0,
+ "49": 2209.0,
+ "50": 2363.0,
+ "51": 2520.0,
+ "52": 2558.0,
+ "53": 2942.0,
+ "54": 2681.0,
+ "55": 2327.0,
+ "56": 2615.0,
+ "57": 2270.0,
+ "58": 2908.0,
+ "59": 2839.0,
+ "60": 2480.0,
+ "61": 2826.0,
+ "62": 2734.0,
+ "63": 2456.0,
+ "64": 2959.0,
+ "65": 2545.0,
+ "66": 2920.0,
+ "67": 2728.0,
+ "68": 2711.0,
+ "69": 2809.0,
+ "70": 3161.0,
+ "71": 2910.0,
+ "72": 2486.0,
+ "73": 2907.0,
+ "74": 1911.0,
+ "75": 2411.0,
+ "76": 2887.0,
+ "77": 3233.0,
+ "78": 3181.0,
+ "79": 3168.0,
+ "80": 3234.0,
+ "81": 3589.0,
+ "82": 3171.0,
+ "83": 2616.0,
+ "84": 3228.0,
+ "85": 3154.0,
+ "86": 2732.0,
+ "87": 3792.0,
+ "88": 3096.0,
+ "89": 3401.0,
+ "90": 3040.0,
+ "91": 2717.0,
+ "92": 3139.0,
+ "93": 2742.0,
+ "94": 3101.0,
+ "95": 3195.0,
+ "96": 3298.0,
+ "97": 3343.0,
+ "98": 3542.0,
+ "99": 3040.0,
+ "100": 3293.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 514359808.0,
- "2": 514359808.0,
- "3": 514359808.0,
- "4": 514359808.0,
- "5": 514359808.0,
- "6": 514359808.0,
- "7": 514359808.0,
- "8": 514359808.0,
- "9": 514359808.0,
- "10": 514359808.0,
- "11": 514359808.0,
- "12": 514359808.0,
- "13": 514359808.0,
- "14": 514359808.0,
- "15": 514359808.0,
- "16": 514359808.0,
- "17": 514359808.0,
- "18": 514359808.0,
- "19": 514359808.0,
- "20": 514359808.0,
- "21": 514359808.0,
- "22": 514359808.0,
- "23": 514359808.0,
- "24": 514359808.0,
- "25": 514359808.0,
- "26": 514359808.0,
- "27": 514359808.0,
- "28": 514359808.0,
- "29": 514359808.0,
- "30": 514359808.0,
- "31": 514359808.0,
- "32": 514359808.0,
- "33": 514359808.0,
- "34": 514359808.0,
- "35": 514359808.0,
- "36": 514359808.0,
- "37": 514359808.0,
- "38": 514359808.0,
- "39": 514359808.0,
- "40": 514359808.0,
- "41": 514359808.0,
- "42": 514359808.0,
- "43": 514359808.0,
- "44": 514359808.0,
- "45": 514359808.0,
- "46": 514359808.0,
- "47": 514359808.0,
- "48": 514359808.0,
- "49": 514359808.0,
- "50": 514359808.0,
- "51": 514359808.0,
- "52": 514359808.0,
- "53": 514359808.0,
- "54": 514359808.0,
- "55": 514359808.0,
- "56": 514359808.0,
- "57": 514359808.0,
- "58": 514359808.0,
- "59": 514359808.0,
- "60": 514359808.0,
- "61": 514359808.0,
- "62": 514359808.0,
- "63": 514359808.0,
- "64": 514359808.0,
- "65": 514359808.0,
- "66": 514359808.0,
- "67": 514359808.0,
- "68": 514359808.0,
- "69": 514359808.0,
- "70": 514359808.0,
- "71": 514359808.0,
- "72": 514359808.0,
- "73": 514359808.0,
- "74": 514359808.0,
- "75": 514359808.0,
- "76": 514359808.0,
- "77": 514359808.0,
- "78": 514359808.0,
- "79": 514359808.0,
- "80": 514359808.0,
- "81": 514359808.0,
- "82": 514359808.0,
- "83": 514359808.0,
- "84": 514359808.0,
- "85": 514359808.0,
- "86": 514359808.0,
- "87": 514359808.0,
- "88": 514359808.0,
- "89": 514359808.0,
- "90": 514359808.0,
- "91": 514359808.0,
- "92": 514359808.0,
- "93": 514359808.0,
- "94": 514359808.0,
- "95": 514359808.0,
- "96": 514359808.0,
- "97": 514359808.0,
- "98": 514359808.0,
- "99": 514359808.0,
- "100": 514359808.0
+ "1": 514884096.0,
+ "2": 514884096.0,
+ "3": 514884096.0,
+ "4": 514884096.0,
+ "5": 514884096.0,
+ "6": 514884096.0,
+ "7": 514884096.0,
+ "8": 514884096.0,
+ "9": 514884096.0,
+ "10": 514884096.0,
+ "11": 514884096.0,
+ "12": 514884096.0,
+ "13": 514884096.0,
+ "14": 514884096.0,
+ "15": 514884096.0,
+ "16": 514884096.0,
+ "17": 514884096.0,
+ "18": 514884096.0,
+ "19": 514884096.0,
+ "20": 514884096.0,
+ "21": 514884096.0,
+ "22": 514884096.0,
+ "23": 514884096.0,
+ "24": 514884096.0,
+ "25": 514884096.0,
+ "26": 514884096.0,
+ "27": 514884096.0,
+ "28": 514884096.0,
+ "29": 514884096.0,
+ "30": 514884096.0,
+ "31": 514884096.0,
+ "32": 514884096.0,
+ "33": 514884096.0,
+ "34": 514884096.0,
+ "35": 514884096.0,
+ "36": 514884096.0,
+ "37": 514884096.0,
+ "38": 514884096.0,
+ "39": 514884096.0,
+ "40": 514884096.0,
+ "41": 514884096.0,
+ "42": 514884096.0,
+ "43": 514884096.0,
+ "44": 514884096.0,
+ "45": 514884096.0,
+ "46": 514884096.0,
+ "47": 514884096.0,
+ "48": 514884096.0,
+ "49": 514884096.0,
+ "50": 514884096.0,
+ "51": 514884096.0,
+ "52": 514884096.0,
+ "53": 514884096.0,
+ "54": 514884096.0,
+ "55": 514884096.0,
+ "56": 514884096.0,
+ "57": 514884096.0,
+ "58": 514884096.0,
+ "59": 514884096.0,
+ "60": 514884096.0,
+ "61": 514884096.0,
+ "62": 514884096.0,
+ "63": 514884096.0,
+ "64": 514884096.0,
+ "65": 514884096.0,
+ "66": 514884096.0,
+ "67": 514884096.0,
+ "68": 514884096.0,
+ "69": 514884096.0,
+ "70": 514884096.0,
+ "71": 514884096.0,
+ "72": 514884096.0,
+ "73": 514884096.0,
+ "74": 514884096.0,
+ "75": 514884096.0,
+ "76": 514884096.0,
+ "77": 514884096.0,
+ "78": 514884096.0,
+ "79": 514884096.0,
+ "80": 514884096.0,
+ "81": 514884096.0,
+ "82": 514884096.0,
+ "83": 514884096.0,
+ "84": 514884096.0,
+ "85": 514884096.0,
+ "86": 514884096.0,
+ "87": 514884096.0,
+ "88": 514884096.0,
+ "89": 514884096.0,
+ "90": 514884096.0,
+ "91": 514884096.0,
+ "92": 514884096.0,
+ "93": 514884096.0,
+ "94": 514884096.0,
+ "95": 514884096.0,
+ "96": 514884096.0,
+ "97": 514884096.0,
+ "98": 514884096.0,
+ "99": 514884096.0,
+ "100": 514884096.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1258060288.0,
- "2": 1437084160.0,
- "3": 1437084160.0,
- "4": 1437084160.0,
- "5": 1437084160.0,
- "6": 1437084160.0,
- "7": 1437084160.0,
- "8": 1437084160.0,
- "9": 1437084160.0,
- "10": 1437084160.0,
- "11": 1437084160.0,
- "12": 1437084160.0,
- "13": 1437084160.0,
- "14": 1437084160.0,
- "15": 1437084160.0,
- "16": 1437084160.0,
- "17": 1437084160.0,
- "18": 1437084160.0,
- "19": 1437084160.0,
- "20": 1437084160.0,
- "21": 1437084160.0,
- "22": 1437084160.0,
- "23": 1437084160.0,
- "24": 1437084160.0,
- "25": 1437084160.0,
- "26": 1437084160.0,
- "27": 1437084160.0,
- "28": 1437084160.0,
- "29": 1437084160.0,
- "30": 1437084160.0,
- "31": 1437084160.0,
- "32": 1437084160.0,
- "33": 1437084160.0,
- "34": 1437084160.0,
- "35": 1437084160.0,
- "36": 1437084160.0,
- "37": 1437084160.0,
- "38": 1437084160.0,
- "39": 1437084160.0,
- "40": 1437084160.0,
- "41": 1437084160.0,
- "42": 1437084160.0,
- "43": 1437084160.0,
- "44": 1437084160.0,
- "45": 1437084160.0,
- "46": 1437084160.0,
- "47": 1437084160.0,
- "48": 1437084160.0,
- "49": 1437084160.0,
- "50": 1437084160.0,
- "51": 1437084160.0,
- "52": 1437084160.0,
- "53": 1437084160.0,
- "54": 1437084160.0,
- "55": 1437084160.0,
- "56": 1437084160.0,
- "57": 1437084160.0,
- "58": 1437084160.0,
- "59": 1437084160.0,
- "60": 1437084160.0,
- "61": 1437084160.0,
- "62": 1437084160.0,
- "63": 1437084160.0,
- "64": 1437084160.0,
- "65": 1437084160.0,
- "66": 1437084160.0,
- "67": 1437084160.0,
- "68": 1437084160.0,
- "69": 1437084160.0,
- "70": 1437084160.0,
- "71": 1437084160.0,
- "72": 1437084160.0,
- "73": 1437084160.0,
- "74": 1437084160.0,
- "75": 1437084160.0,
- "76": 1437084160.0,
- "77": 1437084160.0,
- "78": 1437084160.0,
- "79": 1437084160.0,
- "80": 1437084160.0,
- "81": 1437084160.0,
- "82": 1437084160.0,
- "83": 1437084160.0,
- "84": 1437084160.0,
- "85": 1437084160.0,
- "86": 1437084160.0,
- "87": 1437084160.0,
- "88": 1437084160.0,
- "89": 1437084160.0,
- "90": 1437084160.0,
- "91": 1437084160.0,
- "92": 1437084160.0,
- "93": 1437084160.0,
- "94": 1437084160.0,
- "95": 1437084160.0,
- "96": 1437084160.0,
- "97": 1437084160.0,
- "98": 1437084160.0,
- "99": 1437084160.0,
- "100": 1437084160.0
+ "1": 1259108864.0,
+ "2": 1438394880.0,
+ "3": 1438394880.0,
+ "4": 1438394880.0,
+ "5": 1438394880.0,
+ "6": 1438394880.0,
+ "7": 1438394880.0,
+ "8": 1438394880.0,
+ "9": 1438394880.0,
+ "10": 1438394880.0,
+ "11": 1438394880.0,
+ "12": 1438394880.0,
+ "13": 1438394880.0,
+ "14": 1438394880.0,
+ "15": 1438394880.0,
+ "16": 1438394880.0,
+ "17": 1438394880.0,
+ "18": 1438394880.0,
+ "19": 1438394880.0,
+ "20": 1438394880.0,
+ "21": 1438394880.0,
+ "22": 1438394880.0,
+ "23": 1438394880.0,
+ "24": 1438394880.0,
+ "25": 1438394880.0,
+ "26": 1438394880.0,
+ "27": 1438394880.0,
+ "28": 1438394880.0,
+ "29": 1438394880.0,
+ "30": 1438394880.0,
+ "31": 1438394880.0,
+ "32": 1438394880.0,
+ "33": 1438394880.0,
+ "34": 1438394880.0,
+ "35": 1438394880.0,
+ "36": 1438394880.0,
+ "37": 1438394880.0,
+ "38": 1438394880.0,
+ "39": 1438394880.0,
+ "40": 1438394880.0,
+ "41": 1438394880.0,
+ "42": 1438394880.0,
+ "43": 1438394880.0,
+ "44": 1438394880.0,
+ "45": 1438394880.0,
+ "46": 1438394880.0,
+ "47": 1438394880.0,
+ "48": 1438394880.0,
+ "49": 1438394880.0,
+ "50": 1438394880.0,
+ "51": 1438394880.0,
+ "52": 1438394880.0,
+ "53": 1438394880.0,
+ "54": 1438394880.0,
+ "55": 1438394880.0,
+ "56": 1438394880.0,
+ "57": 1438394880.0,
+ "58": 1438394880.0,
+ "59": 1438394880.0,
+ "60": 1438394880.0,
+ "61": 1438394880.0,
+ "62": 1438394880.0,
+ "63": 1438394880.0,
+ "64": 1438394880.0,
+ "65": 1438394880.0,
+ "66": 1438394880.0,
+ "67": 1438394880.0,
+ "68": 1438394880.0,
+ "69": 1438394880.0,
+ "70": 1438394880.0,
+ "71": 1438394880.0,
+ "72": 1438394880.0,
+ "73": 1438394880.0,
+ "74": 1438394880.0,
+ "75": 1438394880.0,
+ "76": 1438394880.0,
+ "77": 1438394880.0,
+ "78": 1438394880.0,
+ "79": 1438394880.0,
+ "80": 1438394880.0,
+ "81": 1438394880.0,
+ "82": 1438394880.0,
+ "83": 1438394880.0,
+ "84": 1438394880.0,
+ "85": 1438394880.0,
+ "86": 1438394880.0,
+ "87": 1438394880.0,
+ "88": 1438394880.0,
+ "89": 1438394880.0,
+ "90": 1438394880.0,
+ "91": 1438394880.0,
+ "92": 1438394880.0,
+ "93": 1438394880.0,
+ "94": 1438394880.0,
+ "95": 1438394880.0,
+ "96": 1438394880.0,
+ "97": 1438394880.0,
+ "98": 1438394880.0,
+ "99": 1438394880.0,
+ "100": 1438394880.0
}
},
"iteration-time": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.48366,
- "3": 0.20961,
- "4": 0.19355,
- "5": 0.19146,
- "6": 0.19108,
- "7": 0.19236,
- "8": 0.19259,
- "9": 0.19267,
- "10": 0.19436,
- "11": 0.19257,
- "12": 0.19432,
- "13": 0.19332,
- "14": 0.19442,
- "15": 0.19393,
- "16": 0.19417,
- "17": 0.19555,
- "18": 0.19451,
- "19": 0.19452,
- "20": 0.19555,
- "21": 0.19375,
- "22": 0.19402,
- "23": 0.19539,
- "24": 0.19475,
- "25": 0.19576,
- "26": 0.19424,
- "27": 0.19514,
- "28": 0.19519,
- "29": 0.19578,
- "30": 0.19503,
- "31": 0.19394,
- "32": 0.19582,
- "33": 0.19444,
- "34": 0.19405,
- "35": 0.19498,
- "36": 0.19463,
- "37": 0.19572,
- "38": 0.19362,
- "39": 0.19492,
- "40": 0.19487,
- "41": 0.19497,
- "42": 0.19617,
- "43": 0.19571,
- "44": 0.19661,
- "45": 0.19634,
- "46": 0.19537,
- "47": 0.19646,
- "48": 0.19658,
- "49": 0.19727,
- "50": 0.19567,
- "51": 0.21203,
- "52": 0.19551,
- "53": 0.19415,
- "54": 0.19434,
- "55": 0.19584,
- "56": 0.19437,
- "57": 0.19536,
- "58": 0.20364,
- "59": 0.20029,
- "60": 0.1929,
- "61": 0.19274,
- "62": 0.19364,
- "63": 0.19667,
- "64": 0.19406,
- "65": 0.19781,
- "66": 0.19435,
- "67": 0.19308,
- "68": 0.1932,
- "69": 0.19478,
- "70": 0.19591,
- "71": 0.19922,
- "72": 0.19646,
- "73": 0.19646,
- "74": 0.19739,
- "75": 0.19817,
- "76": 0.20056,
- "77": 0.19655,
- "78": 0.19459,
- "79": 0.19478,
- "80": 0.19638,
- "81": 0.19329,
- "82": 0.19254,
- "83": 0.19379,
- "84": 0.19435,
- "85": 0.19517,
- "86": 0.19446,
- "87": 0.19464,
- "88": 0.19501,
- "89": 0.19544,
- "90": 0.19268,
- "91": 0.19425,
- "92": 0.1933,
- "93": 0.19366,
- "94": 0.19328,
- "95": 0.19408,
- "96": 0.19474,
- "97": 0.19719,
- "98": 0.19535,
- "99": 0.19604,
- "100": 0.19554
+ "2": 7.02396,
+ "3": 1.72471,
+ "4": 4.59112,
+ "5": 0.64605,
+ "6": 0.19769,
+ "7": 0.19652,
+ "8": 0.19968,
+ "9": 0.19671,
+ "10": 0.19901,
+ "11": 0.19969,
+ "12": 0.1998,
+ "13": 0.19841,
+ "14": 0.19825,
+ "15": 0.19777,
+ "16": 0.19822,
+ "17": 0.19797,
+ "18": 0.1996,
+ "19": 0.19784,
+ "20": 0.19764,
+ "21": 0.19694,
+ "22": 0.19945,
+ "23": 0.2371,
+ "24": 0.29995,
+ "25": 0.23957,
+ "26": 0.29727,
+ "27": 0.19838,
+ "28": 0.19795,
+ "29": 0.28338,
+ "30": 0.22699,
+ "31": 0.24258,
+ "32": 0.29474,
+ "33": 0.25539,
+ "34": 0.19968,
+ "35": 0.19808,
+ "36": 0.19822,
+ "37": 0.19897,
+ "38": 0.19895,
+ "39": 0.20247,
+ "40": 0.20143,
+ "41": 0.2013,
+ "42": 0.20244,
+ "43": 0.23533,
+ "44": 0.20223,
+ "45": 0.20033,
+ "46": 0.204,
+ "47": 0.20426,
+ "48": 0.23129,
+ "49": 0.22153,
+ "50": 0.2051,
+ "51": 0.22226,
+ "52": 0.199,
+ "53": 0.22428,
+ "54": 0.19788,
+ "55": 0.20032,
+ "56": 0.20098,
+ "57": 0.19989,
+ "58": 0.19967,
+ "59": 0.19858,
+ "60": 0.22128,
+ "61": 0.19988,
+ "62": 0.19702,
+ "63": 0.19878,
+ "64": 0.20043,
+ "65": 0.19873,
+ "66": 0.19929,
+ "67": 0.19902,
+ "68": 0.19908,
+ "69": 0.20073,
+ "70": 0.19874,
+ "71": 0.20014,
+ "72": 0.19866,
+ "73": 0.19911,
+ "74": 0.19875,
+ "75": 0.19788,
+ "76": 0.19803,
+ "77": 0.19842,
+ "78": 0.19894,
+ "79": 0.19804,
+ "80": 0.19881,
+ "81": 0.19809,
+ "82": 0.19856,
+ "83": 0.19924,
+ "84": 0.19721,
+ "85": 0.19922,
+ "86": 0.19726,
+ "87": 0.19809,
+ "88": 0.19884,
+ "89": 0.20014,
+ "90": 0.19844,
+ "91": 0.19909,
+ "92": 0.19796,
+ "93": 0.20176,
+ "94": 0.19878,
+ "95": 0.19855,
+ "96": 0.1997,
+ "97": 0.19858,
+ "98": 0.19901,
+ "99": 0.19906,
+ "100": 0.1993
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_h100.json
index aab9c0cb891..4685573b47a 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.85949,
- "2": 10.85553,
- "3": 10.86543,
- "4": 10.84553,
- "5": 10.88346,
- "6": 10.89431,
- "7": 10.87067,
- "8": 10.86979,
- "9": 10.86918,
- "10": 10.83886,
- "11": 10.8943,
- "12": 10.87983,
- "13": 10.87985,
- "14": 10.90321,
- "15": 10.84052,
- "16": 10.83787,
- "17": 10.80669,
- "18": 10.83026,
- "19": 10.82261,
- "20": 10.73193,
- "21": 10.70748,
- "22": 10.56005,
- "23": 10.72399,
- "24": 10.61114,
- "25": 10.54813,
- "26": 10.61329,
- "27": 10.63053,
- "28": 10.56646,
- "29": 10.59668,
- "30": 10.37137,
- "31": 10.11725,
- "32": 10.46127,
- "33": 10.45249,
- "34": 10.2169,
- "35": 10.27172,
- "36": 10.23119,
- "37": 10.34809,
- "38": 10.1884,
- "39": 10.41044,
- "40": 10.09425,
- "41": 10.14707,
- "42": 10.21242,
- "43": 9.84105,
- "44": 9.95918,
- "45": 9.84079,
- "46": 9.82479,
- "47": 10.13878,
- "48": 9.85831,
- "49": 9.54705,
- "50": 9.90875,
- "51": 9.8558,
- "52": 9.75237,
- "53": 10.07589,
- "54": 9.95688,
- "55": 9.88203,
- "56": 9.6313,
- "57": 9.48649,
- "58": 9.83109,
- "59": 9.58897,
- "60": 9.50643,
- "61": 9.70363,
- "62": 9.98286,
- "63": 9.38302,
- "64": 9.77901,
- "65": 8.95166,
- "66": 9.70158,
- "67": 9.37203,
- "68": 9.78849,
- "69": 9.79851,
- "70": 9.74737,
- "71": 9.61908,
- "72": 9.58502,
- "73": 9.49721,
- "74": 8.93927,
- "75": 9.42703,
- "76": 9.0802,
- "77": 10.06567,
- "78": 9.72893,
- "79": 9.3776,
- "80": 9.40982,
- "81": 9.47976,
- "82": 9.7018,
- "83": 9.30612,
- "84": 9.4209,
- "85": 9.61371,
- "86": 9.07649,
- "87": 9.5945,
- "88": 9.75068,
- "89": 9.60238,
- "90": 9.81898,
- "91": 9.33894,
- "92": 9.35716,
- "93": 9.07879,
- "94": 8.83503,
- "95": 9.52172,
- "96": 9.53003,
- "97": 9.31306,
- "98": 9.67783,
- "99": 8.89058,
- "100": 9.39725
+ "1": 10.95174,
+ "2": 10.94869,
+ "3": 10.94862,
+ "4": 10.94969,
+ "5": 10.94753,
+ "6": 10.94472,
+ "7": 10.9465,
+ "8": 10.94472,
+ "9": 10.94186,
+ "10": 10.94475,
+ "11": 10.94106,
+ "12": 10.93297,
+ "13": 10.92308,
+ "14": 10.92595,
+ "15": 10.89971,
+ "16": 10.89087,
+ "17": 10.89815,
+ "18": 10.88334,
+ "19": 10.88984,
+ "20": 10.79734,
+ "21": 10.79298,
+ "22": 10.77995,
+ "23": 10.76969,
+ "24": 10.74572,
+ "25": 10.73535,
+ "26": 10.72507,
+ "27": 10.67798,
+ "28": 10.61999,
+ "29": 10.59158,
+ "30": 10.56324,
+ "31": 10.56515,
+ "32": 10.54389,
+ "33": 10.50913,
+ "34": 10.48324,
+ "35": 10.46996,
+ "36": 10.44684,
+ "37": 10.42651,
+ "38": 10.43045,
+ "39": 10.39271,
+ "40": 10.38041,
+ "41": 10.35137,
+ "42": 10.32751,
+ "43": 10.31121,
+ "44": 10.28343,
+ "45": 10.29603,
+ "46": 10.2478,
+ "47": 10.24382,
+ "48": 10.19395,
+ "49": 10.19672,
+ "50": 10.18923,
+ "51": 10.19743,
+ "52": 10.15224,
+ "53": 10.15771,
+ "54": 10.12072,
+ "55": 10.09673,
+ "56": 10.1211,
+ "57": 10.11169,
+ "58": 10.12097,
+ "59": 10.06241,
+ "60": 10.09233,
+ "61": 10.04171,
+ "62": 10.00953,
+ "63": 10.07952,
+ "64": 10.03239,
+ "65": 10.00322,
+ "66": 10.03954,
+ "67": 10.01959,
+ "68": 9.9857,
+ "69": 10.002,
+ "70": 9.98146,
+ "71": 10.00456,
+ "72": 9.99623,
+ "73": 9.98759,
+ "74": 9.97288,
+ "75": 9.93232,
+ "76": 9.96721,
+ "77": 9.96948,
+ "78": 9.92086,
+ "79": 9.91733,
+ "80": 9.93737,
+ "81": 9.95724,
+ "82": 9.89589,
+ "83": 9.86221,
+ "84": 9.80483,
+ "85": 9.78639,
+ "86": 9.89556,
+ "87": 9.90872,
+ "88": 9.88854,
+ "89": 9.83331,
+ "90": 9.82424,
+ "91": 9.83931,
+ "92": 9.82246,
+ "93": 9.75869,
+ "94": 9.83492,
+ "95": 9.82147,
+ "96": 9.81159,
+ "97": 9.75623,
+ "98": 9.78246,
+ "99": 9.82336,
+ "100": 9.71586
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1688.0,
- "2": 1806.0,
- "3": 1675.0,
- "4": 1842.0,
- "5": 1909.0,
- "6": 1908.0,
- "7": 1783.0,
- "8": 1611.0,
- "9": 1753.0,
- "10": 1457.0,
- "11": 1880.0,
- "12": 1683.0,
- "13": 1907.0,
- "14": 1733.0,
- "15": 1930.0,
- "16": 1840.0,
- "17": 1892.0,
- "18": 1650.0,
- "19": 1790.0,
- "20": 1596.0,
- "21": 1765.0,
- "22": 1616.0,
- "23": 1974.0,
- "24": 1621.0,
- "25": 1557.0,
- "26": 1745.0,
- "27": 1722.0,
- "28": 1976.0,
- "29": 2068.0,
- "30": 1860.0,
- "31": 1536.0,
- "32": 1883.0,
- "33": 2071.0,
- "34": 1894.0,
- "35": 1902.0,
- "36": 1885.0,
- "37": 2231.0,
- "38": 2129.0,
- "39": 2333.0,
- "40": 2207.0,
- "41": 2193.0,
- "42": 2322.0,
- "43": 2015.0,
- "44": 2089.0,
- "45": 2095.0,
- "46": 2392.0,
- "47": 2430.0,
- "48": 2414.0,
- "49": 2340.0,
- "50": 2416.0,
- "51": 2613.0,
- "52": 2538.0,
- "53": 2792.0,
- "54": 2801.0,
- "55": 2216.0,
- "56": 2858.0,
- "57": 2381.0,
- "58": 2854.0,
- "59": 2787.0,
- "60": 2457.0,
- "61": 2941.0,
- "62": 2543.0,
- "63": 2408.0,
- "64": 2968.0,
- "65": 2472.0,
- "66": 2977.0,
- "67": 2839.0,
- "68": 2775.0,
- "69": 2832.0,
- "70": 3057.0,
- "71": 2909.0,
- "72": 2421.0,
- "73": 2982.0,
- "74": 1922.0,
- "75": 2474.0,
- "76": 3059.0,
- "77": 3177.0,
- "78": 3067.0,
- "79": 3052.0,
- "80": 3338.0,
- "81": 3644.0,
- "82": 3234.0,
- "83": 2798.0,
- "84": 3196.0,
- "85": 3324.0,
- "86": 2855.0,
- "87": 3820.0,
- "88": 2962.0,
- "89": 3379.0,
- "90": 3096.0,
- "91": 2857.0,
- "92": 3077.0,
- "93": 2693.0,
- "94": 3312.0,
- "95": 3399.0,
- "96": 3378.0,
- "97": 3030.0,
- "98": 3619.0,
- "99": 3160.0,
- "100": 3128.0
+ "1": 1733.0,
+ "2": 1654.0,
+ "3": 1807.0,
+ "4": 1633.0,
+ "5": 1831.0,
+ "6": 1678.0,
+ "7": 1895.0,
+ "8": 1671.0,
+ "9": 1673.0,
+ "10": 1708.0,
+ "11": 1553.0,
+ "12": 1705.0,
+ "13": 1834.0,
+ "14": 1862.0,
+ "15": 1629.0,
+ "16": 1711.0,
+ "17": 1804.0,
+ "18": 1759.0,
+ "19": 1724.0,
+ "20": 1631.0,
+ "21": 1731.0,
+ "22": 1689.0,
+ "23": 1823.0,
+ "24": 1768.0,
+ "25": 1639.0,
+ "26": 1778.0,
+ "27": 1847.0,
+ "28": 1833.0,
+ "29": 1889.0,
+ "30": 1796.0,
+ "31": 1941.0,
+ "32": 1911.0,
+ "33": 1876.0,
+ "34": 1967.0,
+ "35": 2017.0,
+ "36": 1911.0,
+ "37": 2142.0,
+ "38": 2077.0,
+ "39": 2168.0,
+ "40": 2277.0,
+ "41": 2228.0,
+ "42": 2042.0,
+ "43": 2283.0,
+ "44": 2229.0,
+ "45": 2449.0,
+ "46": 2431.0,
+ "47": 2507.0,
+ "48": 2573.0,
+ "49": 2802.0,
+ "50": 2595.0,
+ "51": 2496.0,
+ "52": 2589.0,
+ "53": 2666.0,
+ "54": 2777.0,
+ "55": 2571.0,
+ "56": 2742.0,
+ "57": 2260.0,
+ "58": 3560.0,
+ "59": 2980.0,
+ "60": 2999.0,
+ "61": 2639.0,
+ "62": 3174.0,
+ "63": 3364.0,
+ "64": 3609.0,
+ "65": 2773.0,
+ "66": 3041.0,
+ "67": 3905.0,
+ "68": 3439.0,
+ "69": 3016.0,
+ "70": 3321.0,
+ "71": 3199.0,
+ "72": 2945.0,
+ "73": 3486.0,
+ "74": 3218.0,
+ "75": 3115.0,
+ "76": 3388.0,
+ "77": 3690.0,
+ "78": 3383.0,
+ "79": 3277.0,
+ "80": 2844.0,
+ "81": 3319.0,
+ "82": 2759.0,
+ "83": 2962.0,
+ "84": 3000.0,
+ "85": 2744.0,
+ "86": 3058.0,
+ "87": 3003.0,
+ "88": 3063.0,
+ "89": 2947.0,
+ "90": 3757.0,
+ "91": 2785.0,
+ "92": 2967.0,
+ "93": 2920.0,
+ "94": 3199.0,
+ "95": 3313.0,
+ "96": 3580.0,
+ "97": 3554.0,
+ "98": 3283.0,
+ "99": 3298.0,
+ "100": 3386.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 516456960.0,
- "2": 516456960.0,
- "3": 516456960.0,
- "4": 516456960.0,
- "5": 516456960.0,
- "6": 516456960.0,
- "7": 516456960.0,
- "8": 516456960.0,
- "9": 516456960.0,
- "10": 516456960.0,
- "11": 516456960.0,
- "12": 516456960.0,
- "13": 516456960.0,
- "14": 516456960.0,
- "15": 516456960.0,
- "16": 516456960.0,
- "17": 516456960.0,
- "18": 516456960.0,
- "19": 516456960.0,
- "20": 516456960.0,
- "21": 516456960.0,
- "22": 516456960.0,
- "23": 516456960.0,
- "24": 516456960.0,
- "25": 516456960.0,
- "26": 516456960.0,
- "27": 516456960.0,
- "28": 516456960.0,
- "29": 516456960.0,
- "30": 516456960.0,
- "31": 516456960.0,
- "32": 516456960.0,
- "33": 516456960.0,
- "34": 516456960.0,
- "35": 516456960.0,
- "36": 516456960.0,
- "37": 516456960.0,
- "38": 516456960.0,
- "39": 516456960.0,
- "40": 516456960.0,
- "41": 516456960.0,
- "42": 516456960.0,
- "43": 516456960.0,
- "44": 516456960.0,
- "45": 516456960.0,
- "46": 516456960.0,
- "47": 516456960.0,
- "48": 516456960.0,
- "49": 516456960.0,
- "50": 516456960.0,
- "51": 516456960.0,
- "52": 516456960.0,
- "53": 516456960.0,
- "54": 516456960.0,
- "55": 516456960.0,
- "56": 516456960.0,
- "57": 516456960.0,
- "58": 516456960.0,
- "59": 516456960.0,
- "60": 516456960.0,
- "61": 516456960.0,
- "62": 516456960.0,
- "63": 516456960.0,
- "64": 516456960.0,
- "65": 516456960.0,
- "66": 516456960.0,
- "67": 516456960.0,
- "68": 516456960.0,
- "69": 516456960.0,
- "70": 516456960.0,
- "71": 516456960.0,
- "72": 516456960.0,
- "73": 516456960.0,
- "74": 516456960.0,
- "75": 516456960.0,
- "76": 516456960.0,
- "77": 516456960.0,
- "78": 516456960.0,
- "79": 516456960.0,
- "80": 516456960.0,
- "81": 516456960.0,
- "82": 516456960.0,
- "83": 516456960.0,
- "84": 516456960.0,
- "85": 516456960.0,
- "86": 516456960.0,
- "87": 516456960.0,
- "88": 516456960.0,
- "89": 516456960.0,
- "90": 516456960.0,
- "91": 516456960.0,
- "92": 516456960.0,
- "93": 516456960.0,
- "94": 516456960.0,
- "95": 516456960.0,
- "96": 516456960.0,
- "97": 516456960.0,
- "98": 516456960.0,
- "99": 516456960.0,
- "100": 516456960.0
+ "1": 517243392.0,
+ "2": 517243392.0,
+ "3": 517243392.0,
+ "4": 517243392.0,
+ "5": 517243392.0,
+ "6": 517243392.0,
+ "7": 517243392.0,
+ "8": 517243392.0,
+ "9": 517243392.0,
+ "10": 517243392.0,
+ "11": 517243392.0,
+ "12": 517243392.0,
+ "13": 517243392.0,
+ "14": 517243392.0,
+ "15": 517243392.0,
+ "16": 517243392.0,
+ "17": 517243392.0,
+ "18": 517243392.0,
+ "19": 517243392.0,
+ "20": 517243392.0,
+ "21": 517243392.0,
+ "22": 517243392.0,
+ "23": 517243392.0,
+ "24": 517243392.0,
+ "25": 517243392.0,
+ "26": 517243392.0,
+ "27": 517243392.0,
+ "28": 517243392.0,
+ "29": 517243392.0,
+ "30": 517243392.0,
+ "31": 517243392.0,
+ "32": 517243392.0,
+ "33": 517243392.0,
+ "34": 517243392.0,
+ "35": 517243392.0,
+ "36": 517243392.0,
+ "37": 517243392.0,
+ "38": 517243392.0,
+ "39": 517243392.0,
+ "40": 517243392.0,
+ "41": 517243392.0,
+ "42": 517243392.0,
+ "43": 517243392.0,
+ "44": 517243392.0,
+ "45": 517243392.0,
+ "46": 517243392.0,
+ "47": 517243392.0,
+ "48": 517243392.0,
+ "49": 517243392.0,
+ "50": 517243392.0,
+ "51": 517243392.0,
+ "52": 517243392.0,
+ "53": 517243392.0,
+ "54": 517243392.0,
+ "55": 517243392.0,
+ "56": 517243392.0,
+ "57": 517243392.0,
+ "58": 517243392.0,
+ "59": 517243392.0,
+ "60": 517243392.0,
+ "61": 517243392.0,
+ "62": 517243392.0,
+ "63": 517243392.0,
+ "64": 517243392.0,
+ "65": 517243392.0,
+ "66": 517243392.0,
+ "67": 517243392.0,
+ "68": 517243392.0,
+ "69": 517243392.0,
+ "70": 517243392.0,
+ "71": 517243392.0,
+ "72": 517243392.0,
+ "73": 517243392.0,
+ "74": 517243392.0,
+ "75": 517243392.0,
+ "76": 517243392.0,
+ "77": 517243392.0,
+ "78": 517243392.0,
+ "79": 517243392.0,
+ "80": 517243392.0,
+ "81": 517243392.0,
+ "82": 517243392.0,
+ "83": 517243392.0,
+ "84": 517243392.0,
+ "85": 517243392.0,
+ "86": 517243392.0,
+ "87": 517243392.0,
+ "88": 517243392.0,
+ "89": 517243392.0,
+ "90": 517243392.0,
+ "91": 517243392.0,
+ "92": 517243392.0,
+ "93": 517243392.0,
+ "94": 517243392.0,
+ "95": 517243392.0,
+ "96": 517243392.0,
+ "97": 517243392.0,
+ "98": 517243392.0,
+ "99": 517243392.0,
+ "100": 517243392.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1246525952.0,
- "2": 1426598400.0,
- "3": 1426598400.0,
- "4": 1426598400.0,
- "5": 1426598400.0,
- "6": 1426598400.0,
- "7": 1426598400.0,
- "8": 1426598400.0,
- "9": 1426598400.0,
- "10": 1426598400.0,
- "11": 1426598400.0,
- "12": 1426598400.0,
- "13": 1426598400.0,
- "14": 1426598400.0,
- "15": 1426598400.0,
- "16": 1426598400.0,
- "17": 1426598400.0,
- "18": 1426598400.0,
- "19": 1426598400.0,
- "20": 1426598400.0,
- "21": 1426598400.0,
- "22": 1426598400.0,
- "23": 1426598400.0,
- "24": 1426598400.0,
- "25": 1426598400.0,
- "26": 1426598400.0,
- "27": 1426598400.0,
- "28": 1426598400.0,
- "29": 1426598400.0,
- "30": 1426598400.0,
- "31": 1426598400.0,
- "32": 1426598400.0,
- "33": 1426598400.0,
- "34": 1426598400.0,
- "35": 1426598400.0,
- "36": 1426598400.0,
- "37": 1426598400.0,
- "38": 1426598400.0,
- "39": 1426598400.0,
- "40": 1426598400.0,
- "41": 1426598400.0,
- "42": 1426598400.0,
- "43": 1426598400.0,
- "44": 1426598400.0,
- "45": 1426598400.0,
- "46": 1426598400.0,
- "47": 1426598400.0,
- "48": 1426598400.0,
- "49": 1426598400.0,
- "50": 1426598400.0,
- "51": 1426598400.0,
- "52": 1426598400.0,
- "53": 1426598400.0,
- "54": 1426598400.0,
- "55": 1426598400.0,
- "56": 1426598400.0,
- "57": 1426598400.0,
- "58": 1426598400.0,
- "59": 1426598400.0,
- "60": 1426598400.0,
- "61": 1426598400.0,
- "62": 1426598400.0,
- "63": 1426598400.0,
- "64": 1426598400.0,
- "65": 1426598400.0,
- "66": 1426598400.0,
- "67": 1426598400.0,
- "68": 1426598400.0,
- "69": 1426598400.0,
- "70": 1426598400.0,
- "71": 1426598400.0,
- "72": 1426598400.0,
- "73": 1426598400.0,
- "74": 1426598400.0,
- "75": 1426598400.0,
- "76": 1426598400.0,
- "77": 1426598400.0,
- "78": 1426598400.0,
- "79": 1426598400.0,
- "80": 1426598400.0,
- "81": 1426598400.0,
- "82": 1426598400.0,
- "83": 1426598400.0,
- "84": 1426598400.0,
- "85": 1426598400.0,
- "86": 1426598400.0,
- "87": 1426598400.0,
- "88": 1426598400.0,
- "89": 1426598400.0,
- "90": 1426598400.0,
- "91": 1426598400.0,
- "92": 1426598400.0,
- "93": 1426598400.0,
- "94": 1426598400.0,
- "95": 1426598400.0,
- "96": 1426598400.0,
- "97": 1426598400.0,
- "98": 1426598400.0,
- "99": 1426598400.0,
- "100": 1426598400.0
+ "1": 1245477376.0,
+ "2": 1427384832.0,
+ "3": 1427384832.0,
+ "4": 1427384832.0,
+ "5": 1427384832.0,
+ "6": 1427384832.0,
+ "7": 1427384832.0,
+ "8": 1427384832.0,
+ "9": 1427384832.0,
+ "10": 1427384832.0,
+ "11": 1427384832.0,
+ "12": 1427384832.0,
+ "13": 1427384832.0,
+ "14": 1427384832.0,
+ "15": 1427384832.0,
+ "16": 1427384832.0,
+ "17": 1427384832.0,
+ "18": 1427384832.0,
+ "19": 1427384832.0,
+ "20": 1427384832.0,
+ "21": 1427384832.0,
+ "22": 1427384832.0,
+ "23": 1427384832.0,
+ "24": 1427384832.0,
+ "25": 1427384832.0,
+ "26": 1427384832.0,
+ "27": 1427384832.0,
+ "28": 1427384832.0,
+ "29": 1427384832.0,
+ "30": 1427384832.0,
+ "31": 1427384832.0,
+ "32": 1427384832.0,
+ "33": 1427384832.0,
+ "34": 1427384832.0,
+ "35": 1427384832.0,
+ "36": 1427384832.0,
+ "37": 1427384832.0,
+ "38": 1427384832.0,
+ "39": 1427384832.0,
+ "40": 1427384832.0,
+ "41": 1427384832.0,
+ "42": 1427384832.0,
+ "43": 1427384832.0,
+ "44": 1427384832.0,
+ "45": 1427384832.0,
+ "46": 1427384832.0,
+ "47": 1427384832.0,
+ "48": 1427384832.0,
+ "49": 1427384832.0,
+ "50": 1427384832.0,
+ "51": 1428171264.0,
+ "52": 1428171264.0,
+ "53": 1428171264.0,
+ "54": 1428171264.0,
+ "55": 1428171264.0,
+ "56": 1428171264.0,
+ "57": 1428171264.0,
+ "58": 1428171264.0,
+ "59": 1428171264.0,
+ "60": 1428171264.0,
+ "61": 1428171264.0,
+ "62": 1428171264.0,
+ "63": 1428171264.0,
+ "64": 1428171264.0,
+ "65": 1428171264.0,
+ "66": 1428171264.0,
+ "67": 1428171264.0,
+ "68": 1428171264.0,
+ "69": 1428171264.0,
+ "70": 1428171264.0,
+ "71": 1428171264.0,
+ "72": 1428171264.0,
+ "73": 1428171264.0,
+ "74": 1428171264.0,
+ "75": 1428171264.0,
+ "76": 1428171264.0,
+ "77": 1428171264.0,
+ "78": 1428171264.0,
+ "79": 1428171264.0,
+ "80": 1428171264.0,
+ "81": 1428171264.0,
+ "82": 1428171264.0,
+ "83": 1428171264.0,
+ "84": 1428171264.0,
+ "85": 1428171264.0,
+ "86": 1428171264.0,
+ "87": 1428171264.0,
+ "88": 1428171264.0,
+ "89": 1428171264.0,
+ "90": 1428171264.0,
+ "91": 1428171264.0,
+ "92": 1428171264.0,
+ "93": 1428171264.0,
+ "94": 1428171264.0,
+ "95": 1428171264.0,
+ "96": 1428171264.0,
+ "97": 1428171264.0,
+ "98": 1428171264.0,
+ "99": 1428171264.0,
+ "100": 1428171264.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 8.65189,
- "2": 0.17932,
- "3": 0.14636,
- "4": 0.12538,
- "5": 0.12402,
- "6": 0.12459,
- "7": 0.12481,
- "8": 0.12323,
- "9": 0.12314,
- "10": 0.12506,
- "11": 0.1247,
- "12": 0.124,
- "13": 0.12299,
- "14": 0.12337,
- "15": 0.12552,
- "16": 0.12432,
- "17": 0.12285,
- "18": 0.1235,
- "19": 0.12341,
- "20": 0.12389,
- "21": 0.12311,
- "22": 0.12402,
- "23": 0.12319,
- "24": 0.12321,
- "25": 0.12382,
- "26": 0.12336,
- "27": 0.12353,
- "28": 0.12251,
- "29": 0.12528,
- "30": 0.12437,
- "31": 0.12503,
- "32": 0.12365,
- "33": 0.1224,
- "34": 0.12436,
- "35": 0.12606,
- "36": 0.12382,
- "37": 0.12451,
- "38": 0.12292,
- "39": 0.1228,
- "40": 0.12355,
- "41": 0.12426,
- "42": 0.12483,
- "43": 0.12585,
- "44": 0.12964,
- "45": 0.12442,
- "46": 0.12437,
- "47": 0.12371,
- "48": 0.12305,
- "49": 0.12517,
- "50": 0.12295,
- "51": 0.14312,
- "52": 0.1306,
- "53": 0.12394,
- "54": 0.12469,
- "55": 0.12368,
- "56": 0.12394,
- "57": 0.12303,
- "58": 0.12356,
- "59": 0.12328,
- "60": 0.12317,
- "61": 0.12286,
- "62": 0.12321,
- "63": 0.12386,
- "64": 0.12303,
- "65": 0.12369,
- "66": 0.12284,
- "67": 0.12276,
- "68": 0.1233,
- "69": 0.12275,
- "70": 0.12331,
- "71": 0.12204,
- "72": 0.12226,
- "73": 0.12258,
- "74": 0.12222,
- "75": 0.12284,
- "76": 0.12277,
- "77": 0.12539,
- "78": 0.12356,
- "79": 0.1224,
- "80": 0.12283,
- "81": 0.12341,
- "82": 0.12375,
- "83": 0.1222,
- "84": 0.12248,
- "85": 0.12367,
- "86": 0.12361,
- "87": 0.12373,
- "88": 0.124,
- "89": 0.1217,
- "90": 0.12316,
- "91": 0.12421,
- "92": 0.12415,
- "93": 0.1244,
- "94": 0.12547,
- "95": 0.12292,
- "96": 0.12216,
- "97": 0.12313,
- "98": 0.12301,
- "99": 0.1248,
- "100": 0.12337
+ "1": "nan",
+ "2": 5.90924,
+ "3": 0.1503,
+ "4": 0.13338,
+ "5": 0.1361,
+ "6": 0.13457,
+ "7": 0.13172,
+ "8": 0.13424,
+ "9": 0.13302,
+ "10": 0.13349,
+ "11": 0.1335,
+ "12": 0.133,
+ "13": 0.1331,
+ "14": 0.13453,
+ "15": 0.13328,
+ "16": 0.13335,
+ "17": 0.13371,
+ "18": 0.13244,
+ "19": 0.13257,
+ "20": 0.13265,
+ "21": 0.13511,
+ "22": 0.13309,
+ "23": 0.13417,
+ "24": 0.13296,
+ "25": 0.13292,
+ "26": 0.133,
+ "27": 0.13261,
+ "28": 0.12739,
+ "29": 0.12854,
+ "30": 0.12774,
+ "31": 0.1285,
+ "32": 0.12737,
+ "33": 0.12764,
+ "34": 0.12986,
+ "35": 0.12791,
+ "36": 0.1284,
+ "37": 0.12767,
+ "38": 0.12932,
+ "39": 0.13681,
+ "40": 0.12838,
+ "41": 0.13488,
+ "42": 0.13129,
+ "43": 0.13008,
+ "44": 0.13364,
+ "45": 0.128,
+ "46": 0.13448,
+ "47": 0.12831,
+ "48": 0.12869,
+ "49": 0.12834,
+ "50": 0.12787,
+ "51": 0.14852,
+ "52": 0.12926,
+ "53": 0.13048,
+ "54": 0.13182,
+ "55": 0.1291,
+ "56": 0.12889,
+ "57": 0.13056,
+ "58": 0.12937,
+ "59": 0.12932,
+ "60": 0.12918,
+ "61": 0.13058,
+ "62": 0.12943,
+ "63": 0.12912,
+ "64": 0.12879,
+ "65": 0.1299,
+ "66": 0.12975,
+ "67": 0.12912,
+ "68": 0.12956,
+ "69": 0.12956,
+ "70": 0.12841,
+ "71": 0.12853,
+ "72": 0.13036,
+ "73": 0.12954,
+ "74": 0.12961,
+ "75": 0.1296,
+ "76": 0.12907,
+ "77": 0.13489,
+ "78": 0.16499,
+ "79": 0.1373,
+ "80": 0.13288,
+ "81": 0.13152,
+ "82": 0.12763,
+ "83": 0.12793,
+ "84": 0.12922,
+ "85": 0.12978,
+ "86": 0.13039,
+ "87": 0.13066,
+ "88": 0.13125,
+ "89": 0.13055,
+ "90": 0.13132,
+ "91": 0.13071,
+ "92": 0.13044,
+ "93": 0.13129,
+ "94": 0.13122,
+ "95": 0.13049,
+ "96": 0.13148,
+ "97": 0.12934,
+ "98": 0.13038,
+ "99": 0.12828,
+ "100": 0.13127
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist/model_config.yaml
index af4aa0bf4fc..9436fa2a5e6 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -49,4 +49,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cp2_nondeterministic/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cp2_nondeterministic/model_config.yaml
index 9fbe95431e0..2fe3d590c1b 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cp2_nondeterministic/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cp2_nondeterministic/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: flash
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cross_entropy_loss_fusion/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cross_entropy_loss_fusion/model_config.yaml
index 54d49da6c14..77789c90192 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cross_entropy_loss_fusion/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cross_entropy_loss_fusion/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -48,4 +48,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective/golden_values_dev_dgx_gb200.json
index 27eb21de0f0..09e346c2bca 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective/golden_values_dev_dgx_gb200.json
@@ -6,104 +6,104 @@
"values": {
"1": 10.86897,
"2": 10.88544,
- "3": 10.86473,
- "4": 10.86826,
- "5": 10.87436,
+ "3": 10.86477,
+ "4": 10.86825,
+ "5": 10.87437,
"6": 10.89005,
- "7": 10.87769,
+ "7": 10.87768,
"8": 10.86364,
- "9": 10.88282,
- "10": 10.84687,
- "11": 10.87102,
- "12": 10.87345,
+ "9": 10.8828,
+ "10": 10.84683,
+ "11": 10.87103,
+ "12": 10.87356,
"13": 10.8814,
- "14": 10.8877,
- "15": 10.83869,
+ "14": 10.88767,
+ "15": 10.83862,
"16": 10.8239,
- "17": 10.80197,
- "18": 10.81094,
- "19": 10.82192,
- "20": 10.71791,
- "21": 10.68914,
- "22": 10.57271,
- "23": 10.7081,
+ "17": 10.80195,
+ "18": 10.81091,
+ "19": 10.82189,
+ "20": 10.71788,
+ "21": 10.6892,
+ "22": 10.57272,
+ "23": 10.70811,
"24": 10.59543,
- "25": 10.55292,
- "26": 10.61257,
- "27": 10.60051,
- "28": 10.56173,
- "29": 10.58089,
- "30": 10.35595,
- "31": 10.1182,
- "32": 10.44815,
- "33": 10.4542,
- "34": 10.21553,
- "35": 10.26124,
- "36": 10.20776,
- "37": 10.33673,
+ "25": 10.55295,
+ "26": 10.61259,
+ "27": 10.60052,
+ "28": 10.56168,
+ "29": 10.58085,
+ "30": 10.35597,
+ "31": 10.11819,
+ "32": 10.44814,
+ "33": 10.45418,
+ "34": 10.21548,
+ "35": 10.26125,
+ "36": 10.20779,
+ "37": 10.33671,
"38": 10.17741,
- "39": 10.39297,
- "40": 10.06349,
- "41": 10.13887,
- "42": 10.2056,
- "43": 9.82809,
- "44": 9.94547,
- "45": 9.82561,
- "46": 9.80186,
- "47": 10.14049,
- "48": 9.84276,
- "49": 9.52016,
- "50": 9.88454,
- "51": 9.84743,
- "52": 9.74209,
- "53": 10.05697,
- "54": 9.9505,
- "55": 9.88145,
- "56": 9.61274,
+ "39": 10.39294,
+ "40": 10.06346,
+ "41": 10.13891,
+ "42": 10.20558,
+ "43": 9.82807,
+ "44": 9.94542,
+ "45": 9.82557,
+ "46": 9.80182,
+ "47": 10.14053,
+ "48": 9.8428,
+ "49": 9.52012,
+ "50": 9.88456,
+ "51": 9.84742,
+ "52": 9.74207,
+ "53": 10.05695,
+ "54": 9.95053,
+ "55": 9.8814,
+ "56": 9.61275,
"57": 9.4687,
- "58": 9.82193,
+ "58": 9.82189,
"59": 9.57642,
- "60": 9.49762,
- "61": 9.69189,
+ "60": 9.4976,
+ "61": 9.69188,
"62": 9.9867,
- "63": 9.37512,
- "64": 9.76679,
- "65": 8.94648,
- "66": 9.7023,
- "67": 9.36326,
- "68": 9.7831,
- "69": 9.7986,
- "70": 9.7317,
- "71": 9.62571,
- "72": 9.58488,
- "73": 9.48967,
- "74": 8.9286,
- "75": 9.40862,
- "76": 9.07925,
- "77": 10.0594,
- "78": 9.72288,
- "79": 9.37784,
- "80": 9.40429,
- "81": 9.48309,
- "82": 9.7004,
- "83": 9.31595,
- "84": 9.41838,
- "85": 9.61685,
- "86": 9.07533,
- "87": 9.59616,
- "88": 9.75215,
- "89": 9.60184,
- "90": 9.82281,
- "91": 9.34037,
+ "63": 9.37509,
+ "64": 9.76682,
+ "65": 8.94649,
+ "66": 9.70228,
+ "67": 9.36325,
+ "68": 9.78311,
+ "69": 9.79862,
+ "70": 9.73172,
+ "71": 9.62574,
+ "72": 9.58486,
+ "73": 9.48964,
+ "74": 8.92859,
+ "75": 9.40865,
+ "76": 9.07927,
+ "77": 10.05938,
+ "78": 9.72283,
+ "79": 9.37782,
+ "80": 9.40428,
+ "81": 9.48312,
+ "82": 9.70038,
+ "83": 9.3159,
+ "84": 9.41834,
+ "85": 9.61684,
+ "86": 9.07537,
+ "87": 9.59617,
+ "88": 9.75218,
+ "89": 9.60187,
+ "90": 9.82286,
+ "91": 9.34039,
"92": 9.35854,
- "93": 9.08805,
- "94": 8.83037,
+ "93": 9.08806,
+ "94": 8.83038,
"95": 9.5266,
- "96": 9.53049,
- "97": 9.30389,
- "98": 9.67196,
- "99": 8.89637,
- "100": 9.40644
+ "96": 9.53051,
+ "97": 9.30391,
+ "98": 9.67199,
+ "99": 8.8964,
+ "100": 9.40645
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1621.0,
- "2": 1657.0,
- "3": 1580.0,
- "4": 1839.0,
- "5": 1862.0,
- "6": 1724.0,
- "7": 1714.0,
- "8": 1670.0,
- "9": 1762.0,
- "10": 1358.0,
- "11": 1734.0,
- "12": 1682.0,
- "13": 1761.0,
- "14": 1731.0,
- "15": 1788.0,
- "16": 1801.0,
- "17": 1866.0,
- "18": 1636.0,
- "19": 1709.0,
- "20": 1607.0,
- "21": 1821.0,
- "22": 1666.0,
- "23": 1991.0,
- "24": 1585.0,
- "25": 1587.0,
- "26": 1631.0,
- "27": 1714.0,
- "28": 1966.0,
- "29": 1997.0,
- "30": 1851.0,
- "31": 1581.0,
- "32": 1864.0,
- "33": 2107.0,
- "34": 1846.0,
- "35": 1982.0,
- "36": 1904.0,
- "37": 2373.0,
- "38": 2172.0,
- "39": 2343.0,
- "40": 2149.0,
- "41": 2331.0,
- "42": 2199.0,
- "43": 1914.0,
- "44": 2065.0,
- "45": 2081.0,
- "46": 2352.0,
- "47": 2497.0,
- "48": 2303.0,
- "49": 2346.0,
- "50": 2411.0,
- "51": 2491.0,
- "52": 2552.0,
- "53": 2980.0,
- "54": 2680.0,
- "55": 2274.0,
- "56": 2734.0,
- "57": 2319.0,
- "58": 2907.0,
- "59": 2886.0,
- "60": 2566.0,
- "61": 2855.0,
- "62": 2704.0,
- "63": 2370.0,
- "64": 2998.0,
- "65": 2563.0,
- "66": 2868.0,
- "67": 2762.0,
- "68": 2739.0,
- "69": 2730.0,
- "70": 3156.0,
- "71": 2803.0,
- "72": 2506.0,
- "73": 2896.0,
- "74": 1937.0,
- "75": 2450.0,
- "76": 2794.0,
- "77": 3047.0,
- "78": 3104.0,
- "79": 3069.0,
- "80": 3286.0,
- "81": 3543.0,
- "82": 3192.0,
- "83": 2614.0,
- "84": 3273.0,
- "85": 3111.0,
- "86": 2680.0,
- "87": 3654.0,
- "88": 3117.0,
- "89": 3351.0,
- "90": 3086.0,
- "91": 2721.0,
- "92": 3045.0,
- "93": 2672.0,
- "94": 3326.0,
- "95": 3125.0,
- "96": 3309.0,
- "97": 3208.0,
- "98": 3572.0,
- "99": 2980.0,
- "100": 3355.0
+ "1": 1592.0,
+ "2": 1638.0,
+ "3": 1658.0,
+ "4": 1737.0,
+ "5": 1836.0,
+ "6": 1740.0,
+ "7": 1780.0,
+ "8": 1628.0,
+ "9": 1785.0,
+ "10": 1372.0,
+ "11": 1738.0,
+ "12": 1745.0,
+ "13": 1808.0,
+ "14": 1709.0,
+ "15": 1798.0,
+ "16": 1765.0,
+ "17": 1875.0,
+ "18": 1580.0,
+ "19": 1744.0,
+ "20": 1573.0,
+ "21": 1835.0,
+ "22": 1649.0,
+ "23": 1993.0,
+ "24": 1600.0,
+ "25": 1527.0,
+ "26": 1592.0,
+ "27": 1818.0,
+ "28": 1889.0,
+ "29": 1909.0,
+ "30": 1889.0,
+ "31": 1635.0,
+ "32": 1847.0,
+ "33": 2066.0,
+ "34": 1761.0,
+ "35": 1861.0,
+ "36": 1916.0,
+ "37": 2298.0,
+ "38": 2162.0,
+ "39": 2226.0,
+ "40": 2127.0,
+ "41": 2226.0,
+ "42": 2204.0,
+ "43": 1844.0,
+ "44": 2106.0,
+ "45": 2056.0,
+ "46": 2238.0,
+ "47": 2476.0,
+ "48": 2289.0,
+ "49": 2209.0,
+ "50": 2363.0,
+ "51": 2520.0,
+ "52": 2558.0,
+ "53": 2942.0,
+ "54": 2681.0,
+ "55": 2327.0,
+ "56": 2615.0,
+ "57": 2270.0,
+ "58": 2908.0,
+ "59": 2839.0,
+ "60": 2480.0,
+ "61": 2826.0,
+ "62": 2734.0,
+ "63": 2456.0,
+ "64": 2959.0,
+ "65": 2545.0,
+ "66": 2920.0,
+ "67": 2728.0,
+ "68": 2711.0,
+ "69": 2809.0,
+ "70": 3161.0,
+ "71": 2910.0,
+ "72": 2486.0,
+ "73": 2907.0,
+ "74": 1911.0,
+ "75": 2411.0,
+ "76": 2887.0,
+ "77": 3233.0,
+ "78": 3181.0,
+ "79": 3168.0,
+ "80": 3234.0,
+ "81": 3589.0,
+ "82": 3171.0,
+ "83": 2616.0,
+ "84": 3228.0,
+ "85": 3154.0,
+ "86": 2732.0,
+ "87": 3792.0,
+ "88": 3096.0,
+ "89": 3401.0,
+ "90": 3040.0,
+ "91": 2717.0,
+ "92": 3139.0,
+ "93": 2742.0,
+ "94": 3101.0,
+ "95": 3195.0,
+ "96": 3298.0,
+ "97": 3343.0,
+ "98": 3542.0,
+ "99": 3040.0,
+ "100": 3293.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 514359808.0,
- "2": 514359808.0,
- "3": 514359808.0,
- "4": 514359808.0,
- "5": 514359808.0,
- "6": 514359808.0,
- "7": 514359808.0,
- "8": 514359808.0,
- "9": 514359808.0,
- "10": 514359808.0,
- "11": 514359808.0,
- "12": 514359808.0,
- "13": 514359808.0,
- "14": 514359808.0,
- "15": 514359808.0,
- "16": 514359808.0,
- "17": 514359808.0,
- "18": 514359808.0,
- "19": 514359808.0,
- "20": 514359808.0,
- "21": 514359808.0,
- "22": 514359808.0,
- "23": 514359808.0,
- "24": 514359808.0,
- "25": 514359808.0,
- "26": 514359808.0,
- "27": 514359808.0,
- "28": 514359808.0,
- "29": 514359808.0,
- "30": 514359808.0,
- "31": 514359808.0,
- "32": 514359808.0,
- "33": 514359808.0,
- "34": 514359808.0,
- "35": 514359808.0,
- "36": 514359808.0,
- "37": 514359808.0,
- "38": 514359808.0,
- "39": 514359808.0,
- "40": 514359808.0,
- "41": 514359808.0,
- "42": 514359808.0,
- "43": 514359808.0,
- "44": 514359808.0,
- "45": 514359808.0,
- "46": 514359808.0,
- "47": 514359808.0,
- "48": 514359808.0,
- "49": 514359808.0,
- "50": 514359808.0,
- "51": 514359808.0,
- "52": 514359808.0,
- "53": 514359808.0,
- "54": 514359808.0,
- "55": 514359808.0,
- "56": 514359808.0,
- "57": 514359808.0,
- "58": 514359808.0,
- "59": 514359808.0,
- "60": 514359808.0,
- "61": 514359808.0,
- "62": 514359808.0,
- "63": 514359808.0,
- "64": 514359808.0,
- "65": 514359808.0,
- "66": 514359808.0,
- "67": 514359808.0,
- "68": 514359808.0,
- "69": 514359808.0,
- "70": 514359808.0,
- "71": 514359808.0,
- "72": 514359808.0,
- "73": 514359808.0,
- "74": 514359808.0,
- "75": 514359808.0,
- "76": 514359808.0,
- "77": 514359808.0,
- "78": 514359808.0,
- "79": 514359808.0,
- "80": 514359808.0,
- "81": 514359808.0,
- "82": 514359808.0,
- "83": 514359808.0,
- "84": 514359808.0,
- "85": 514359808.0,
- "86": 514359808.0,
- "87": 514359808.0,
- "88": 514359808.0,
- "89": 514359808.0,
- "90": 514359808.0,
- "91": 514359808.0,
- "92": 514359808.0,
- "93": 514359808.0,
- "94": 514359808.0,
- "95": 514359808.0,
- "96": 514359808.0,
- "97": 514359808.0,
- "98": 514359808.0,
- "99": 514359808.0,
- "100": 514359808.0
+ "1": 514884096.0,
+ "2": 514884096.0,
+ "3": 514884096.0,
+ "4": 514884096.0,
+ "5": 514884096.0,
+ "6": 514884096.0,
+ "7": 514884096.0,
+ "8": 514884096.0,
+ "9": 514884096.0,
+ "10": 514884096.0,
+ "11": 514884096.0,
+ "12": 514884096.0,
+ "13": 514884096.0,
+ "14": 514884096.0,
+ "15": 514884096.0,
+ "16": 514884096.0,
+ "17": 514884096.0,
+ "18": 514884096.0,
+ "19": 514884096.0,
+ "20": 514884096.0,
+ "21": 514884096.0,
+ "22": 514884096.0,
+ "23": 514884096.0,
+ "24": 514884096.0,
+ "25": 514884096.0,
+ "26": 514884096.0,
+ "27": 514884096.0,
+ "28": 514884096.0,
+ "29": 514884096.0,
+ "30": 514884096.0,
+ "31": 514884096.0,
+ "32": 514884096.0,
+ "33": 514884096.0,
+ "34": 514884096.0,
+ "35": 514884096.0,
+ "36": 514884096.0,
+ "37": 514884096.0,
+ "38": 514884096.0,
+ "39": 514884096.0,
+ "40": 514884096.0,
+ "41": 514884096.0,
+ "42": 514884096.0,
+ "43": 514884096.0,
+ "44": 514884096.0,
+ "45": 514884096.0,
+ "46": 514884096.0,
+ "47": 514884096.0,
+ "48": 514884096.0,
+ "49": 514884096.0,
+ "50": 514884096.0,
+ "51": 514884096.0,
+ "52": 514884096.0,
+ "53": 514884096.0,
+ "54": 514884096.0,
+ "55": 514884096.0,
+ "56": 514884096.0,
+ "57": 514884096.0,
+ "58": 514884096.0,
+ "59": 514884096.0,
+ "60": 514884096.0,
+ "61": 514884096.0,
+ "62": 514884096.0,
+ "63": 514884096.0,
+ "64": 514884096.0,
+ "65": 514884096.0,
+ "66": 514884096.0,
+ "67": 514884096.0,
+ "68": 514884096.0,
+ "69": 514884096.0,
+ "70": 514884096.0,
+ "71": 514884096.0,
+ "72": 514884096.0,
+ "73": 514884096.0,
+ "74": 514884096.0,
+ "75": 514884096.0,
+ "76": 514884096.0,
+ "77": 514884096.0,
+ "78": 514884096.0,
+ "79": 514884096.0,
+ "80": 514884096.0,
+ "81": 514884096.0,
+ "82": 514884096.0,
+ "83": 514884096.0,
+ "84": 514884096.0,
+ "85": 514884096.0,
+ "86": 514884096.0,
+ "87": 514884096.0,
+ "88": 514884096.0,
+ "89": 514884096.0,
+ "90": 514884096.0,
+ "91": 514884096.0,
+ "92": 514884096.0,
+ "93": 514884096.0,
+ "94": 514884096.0,
+ "95": 514884096.0,
+ "96": 514884096.0,
+ "97": 514884096.0,
+ "98": 514884096.0,
+ "99": 514884096.0,
+ "100": 514884096.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1258060288.0,
- "2": 1437084160.0,
- "3": 1437084160.0,
- "4": 1437084160.0,
- "5": 1437084160.0,
- "6": 1437084160.0,
- "7": 1437084160.0,
- "8": 1437084160.0,
- "9": 1437084160.0,
- "10": 1437084160.0,
- "11": 1437084160.0,
- "12": 1437084160.0,
- "13": 1437084160.0,
- "14": 1437084160.0,
- "15": 1437084160.0,
- "16": 1437084160.0,
- "17": 1437084160.0,
- "18": 1437084160.0,
- "19": 1437084160.0,
- "20": 1437084160.0,
- "21": 1437084160.0,
- "22": 1437084160.0,
- "23": 1437084160.0,
- "24": 1437084160.0,
- "25": 1437084160.0,
- "26": 1437084160.0,
- "27": 1437084160.0,
- "28": 1437084160.0,
- "29": 1437084160.0,
- "30": 1437084160.0,
- "31": 1437084160.0,
- "32": 1437084160.0,
- "33": 1437084160.0,
- "34": 1437084160.0,
- "35": 1437084160.0,
- "36": 1437084160.0,
- "37": 1437084160.0,
- "38": 1437084160.0,
- "39": 1437084160.0,
- "40": 1437084160.0,
- "41": 1437084160.0,
- "42": 1437084160.0,
- "43": 1437084160.0,
- "44": 1437084160.0,
- "45": 1437084160.0,
- "46": 1437084160.0,
- "47": 1437084160.0,
- "48": 1437084160.0,
- "49": 1437084160.0,
- "50": 1437084160.0,
- "51": 1437084160.0,
- "52": 1437084160.0,
- "53": 1437084160.0,
- "54": 1437084160.0,
- "55": 1437084160.0,
- "56": 1437084160.0,
- "57": 1437084160.0,
- "58": 1437084160.0,
- "59": 1437084160.0,
- "60": 1437084160.0,
- "61": 1437084160.0,
- "62": 1437084160.0,
- "63": 1437084160.0,
- "64": 1437084160.0,
- "65": 1437084160.0,
- "66": 1437084160.0,
- "67": 1437084160.0,
- "68": 1437084160.0,
- "69": 1437084160.0,
- "70": 1437084160.0,
- "71": 1437084160.0,
- "72": 1437084160.0,
- "73": 1437084160.0,
- "74": 1437084160.0,
- "75": 1437084160.0,
- "76": 1437084160.0,
- "77": 1437084160.0,
- "78": 1437084160.0,
- "79": 1437084160.0,
- "80": 1437084160.0,
- "81": 1437084160.0,
- "82": 1437084160.0,
- "83": 1437084160.0,
- "84": 1437084160.0,
- "85": 1437084160.0,
- "86": 1437084160.0,
- "87": 1437084160.0,
- "88": 1437084160.0,
- "89": 1437084160.0,
- "90": 1437084160.0,
- "91": 1437084160.0,
- "92": 1437084160.0,
- "93": 1437084160.0,
- "94": 1437084160.0,
- "95": 1437084160.0,
- "96": 1437084160.0,
- "97": 1437084160.0,
- "98": 1437084160.0,
- "99": 1437084160.0,
- "100": 1437084160.0
+ "1": 1259108864.0,
+ "2": 1438394880.0,
+ "3": 1438394880.0,
+ "4": 1438394880.0,
+ "5": 1438394880.0,
+ "6": 1438394880.0,
+ "7": 1438394880.0,
+ "8": 1438394880.0,
+ "9": 1438394880.0,
+ "10": 1438394880.0,
+ "11": 1438394880.0,
+ "12": 1438394880.0,
+ "13": 1438394880.0,
+ "14": 1438394880.0,
+ "15": 1438394880.0,
+ "16": 1438394880.0,
+ "17": 1438394880.0,
+ "18": 1438394880.0,
+ "19": 1438394880.0,
+ "20": 1438394880.0,
+ "21": 1438394880.0,
+ "22": 1438394880.0,
+ "23": 1438394880.0,
+ "24": 1438394880.0,
+ "25": 1438394880.0,
+ "26": 1438394880.0,
+ "27": 1438394880.0,
+ "28": 1438394880.0,
+ "29": 1438394880.0,
+ "30": 1438394880.0,
+ "31": 1438394880.0,
+ "32": 1438394880.0,
+ "33": 1438394880.0,
+ "34": 1438394880.0,
+ "35": 1438394880.0,
+ "36": 1438394880.0,
+ "37": 1438394880.0,
+ "38": 1438394880.0,
+ "39": 1438394880.0,
+ "40": 1438394880.0,
+ "41": 1438394880.0,
+ "42": 1438394880.0,
+ "43": 1438394880.0,
+ "44": 1438394880.0,
+ "45": 1438394880.0,
+ "46": 1438394880.0,
+ "47": 1438394880.0,
+ "48": 1438394880.0,
+ "49": 1438394880.0,
+ "50": 1438394880.0,
+ "51": 1438394880.0,
+ "52": 1438394880.0,
+ "53": 1438394880.0,
+ "54": 1438394880.0,
+ "55": 1438394880.0,
+ "56": 1438394880.0,
+ "57": 1438394880.0,
+ "58": 1438394880.0,
+ "59": 1438394880.0,
+ "60": 1438394880.0,
+ "61": 1438394880.0,
+ "62": 1438394880.0,
+ "63": 1438394880.0,
+ "64": 1438394880.0,
+ "65": 1438394880.0,
+ "66": 1438394880.0,
+ "67": 1438394880.0,
+ "68": 1438394880.0,
+ "69": 1438394880.0,
+ "70": 1438394880.0,
+ "71": 1438394880.0,
+ "72": 1438394880.0,
+ "73": 1438394880.0,
+ "74": 1438394880.0,
+ "75": 1438394880.0,
+ "76": 1438394880.0,
+ "77": 1438394880.0,
+ "78": 1438394880.0,
+ "79": 1438394880.0,
+ "80": 1438394880.0,
+ "81": 1438394880.0,
+ "82": 1438394880.0,
+ "83": 1438394880.0,
+ "84": 1438394880.0,
+ "85": 1438394880.0,
+ "86": 1438394880.0,
+ "87": 1438394880.0,
+ "88": 1438394880.0,
+ "89": 1438394880.0,
+ "90": 1438394880.0,
+ "91": 1438394880.0,
+ "92": 1438394880.0,
+ "93": 1438394880.0,
+ "94": 1438394880.0,
+ "95": 1438394880.0,
+ "96": 1438394880.0,
+ "97": 1438394880.0,
+ "98": 1438394880.0,
+ "99": 1438394880.0,
+ "100": 1438394880.0
}
},
"iteration-time": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.6815,
- "3": 0.23582,
- "4": 0.21969,
- "5": 0.22399,
- "6": 0.21848,
- "7": 0.21944,
- "8": 0.21989,
- "9": 0.22542,
- "10": 0.22685,
- "11": 0.22859,
- "12": 0.22734,
- "13": 0.22735,
- "14": 0.22682,
- "15": 0.22731,
- "16": 0.22724,
- "17": 0.22774,
- "18": 0.2253,
- "19": 0.21338,
- "20": 0.21612,
- "21": 0.22487,
- "22": 0.2609,
- "23": 0.34495,
- "24": 0.40538,
- "25": 0.27265,
- "26": 0.22852,
- "27": 0.23498,
- "28": 0.23458,
- "29": 0.2356,
- "30": 0.23223,
- "31": 0.23427,
- "32": 0.23193,
- "33": 0.23007,
- "34": 0.22762,
- "35": 0.22604,
- "36": 0.22153,
- "37": 0.21923,
- "38": 0.21718,
- "39": 0.2162,
- "40": 0.21653,
- "41": 0.21673,
- "42": 0.21416,
- "43": 0.21439,
- "44": 0.2141,
- "45": 0.21364,
- "46": 0.21263,
- "47": 0.2139,
- "48": 0.21445,
- "49": 0.21424,
- "50": 0.21381,
- "51": 0.21544,
- "52": 0.21075,
- "53": 0.21292,
- "54": 0.21407,
- "55": 0.2167,
- "56": 0.21877,
- "57": 0.21861,
- "58": 0.22087,
- "59": 0.21999,
- "60": 0.21884,
- "61": 0.21841,
- "62": 0.21988,
- "63": 0.21876,
- "64": 0.21811,
- "65": 0.21795,
- "66": 0.2197,
- "67": 0.22005,
- "68": 0.21994,
- "69": 0.21937,
- "70": 0.21964,
- "71": 0.22007,
- "72": 0.221,
- "73": 0.22145,
- "74": 0.22069,
- "75": 0.22126,
- "76": 0.21984,
- "77": 0.22096,
- "78": 0.2231,
- "79": 0.22168,
- "80": 0.21932,
- "81": 0.21748,
- "82": 0.21971,
- "83": 0.22113,
- "84": 0.22096,
- "85": 0.22316,
- "86": 0.22043,
- "87": 0.22198,
- "88": 0.2247,
- "89": 0.2219,
- "90": 0.22258,
- "91": 0.22224,
- "92": 0.22132,
- "93": 0.22182,
- "94": 0.22397,
- "95": 0.22547,
- "96": 0.22177,
- "97": 0.22282,
- "98": 0.22255,
- "99": 0.22417,
- "100": 0.22334
+ "2": 5.78191,
+ "3": 0.21543,
+ "4": 0.20431,
+ "5": 0.2048,
+ "6": 0.20252,
+ "7": 0.20042,
+ "8": 0.19829,
+ "9": 0.1981,
+ "10": 0.19894,
+ "11": 0.19941,
+ "12": 0.19791,
+ "13": 0.19665,
+ "14": 0.19775,
+ "15": 0.19771,
+ "16": 0.19844,
+ "17": 0.1973,
+ "18": 0.19675,
+ "19": 0.19775,
+ "20": 0.19752,
+ "21": 0.19736,
+ "22": 0.19856,
+ "23": 0.19791,
+ "24": 0.19728,
+ "25": 0.19919,
+ "26": 0.20211,
+ "27": 0.20996,
+ "28": 0.20609,
+ "29": 0.20893,
+ "30": 0.29002,
+ "31": 0.30297,
+ "32": 0.2644,
+ "33": 0.24762,
+ "34": 0.2283,
+ "35": 0.19894,
+ "36": 0.19904,
+ "37": 0.19825,
+ "38": 0.19956,
+ "39": 0.19723,
+ "40": 0.23122,
+ "41": 0.19962,
+ "42": 0.19911,
+ "43": 0.20105,
+ "44": 0.1992,
+ "45": 0.19775,
+ "46": 0.19912,
+ "47": 0.19899,
+ "48": 0.2034,
+ "49": 0.20022,
+ "50": 0.19798,
+ "51": 0.22615,
+ "52": 0.19944,
+ "53": 0.19734,
+ "54": 0.19576,
+ "55": 0.19493,
+ "56": 0.19695,
+ "57": 0.19635,
+ "58": 0.19686,
+ "59": 0.19616,
+ "60": 0.19825,
+ "61": 0.19728,
+ "62": 0.19652,
+ "63": 0.1972,
+ "64": 0.19799,
+ "65": 0.19902,
+ "66": 0.19916,
+ "67": 0.19906,
+ "68": 0.20142,
+ "69": 0.20147,
+ "70": 0.20041,
+ "71": 0.20146,
+ "72": 0.19828,
+ "73": 0.19865,
+ "74": 0.19876,
+ "75": 0.20114,
+ "76": 0.19884,
+ "77": 0.19884,
+ "78": 0.19712,
+ "79": 0.19797,
+ "80": 0.19724,
+ "81": 0.19755,
+ "82": 0.19734,
+ "83": 0.19936,
+ "84": 0.19723,
+ "85": 0.19891,
+ "86": 0.198,
+ "87": 0.19759,
+ "88": 0.19644,
+ "89": 0.1987,
+ "90": 0.19753,
+ "91": 0.20302,
+ "92": 0.19865,
+ "93": 0.19958,
+ "94": 0.19868,
+ "95": 0.1982,
+ "96": 0.19927,
+ "97": 0.19927,
+ "98": 0.19757,
+ "99": 0.19871,
+ "100": 0.201
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective/golden_values_dev_dgx_h100.json
index 6e4aa9e48e0..9f5fc44a034 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.85949,
- "2": 10.85553,
- "3": 10.86543,
- "4": 10.84553,
- "5": 10.88346,
- "6": 10.89431,
- "7": 10.87067,
- "8": 10.86979,
- "9": 10.86918,
- "10": 10.83886,
- "11": 10.8943,
- "12": 10.87983,
- "13": 10.87985,
- "14": 10.90321,
- "15": 10.84052,
- "16": 10.83787,
- "17": 10.80669,
- "18": 10.83026,
- "19": 10.82261,
- "20": 10.73193,
- "21": 10.70748,
- "22": 10.56005,
- "23": 10.72399,
- "24": 10.61114,
- "25": 10.54813,
- "26": 10.61329,
- "27": 10.63053,
- "28": 10.56646,
- "29": 10.59668,
- "30": 10.37137,
- "31": 10.11725,
- "32": 10.46127,
- "33": 10.45249,
- "34": 10.2169,
- "35": 10.27172,
- "36": 10.23119,
- "37": 10.34809,
- "38": 10.1884,
- "39": 10.41044,
- "40": 10.09425,
- "41": 10.14707,
- "42": 10.21242,
- "43": 9.84105,
- "44": 9.95918,
- "45": 9.84079,
- "46": 9.82479,
- "47": 10.13878,
- "48": 9.85831,
- "49": 9.54705,
- "50": 9.90875,
- "51": 9.8558,
- "52": 9.75237,
- "53": 10.07589,
- "54": 9.95688,
- "55": 9.88203,
- "56": 9.6313,
- "57": 9.48649,
- "58": 9.83109,
- "59": 9.58897,
- "60": 9.50643,
- "61": 9.70363,
- "62": 9.98286,
- "63": 9.38302,
- "64": 9.77901,
- "65": 8.95166,
- "66": 9.70158,
- "67": 9.37203,
- "68": 9.78849,
- "69": 9.79851,
- "70": 9.74737,
- "71": 9.61908,
- "72": 9.58502,
- "73": 9.49721,
- "74": 8.93927,
- "75": 9.42703,
- "76": 9.0802,
- "77": 10.06567,
- "78": 9.72893,
- "79": 9.3776,
- "80": 9.40982,
- "81": 9.47976,
- "82": 9.7018,
- "83": 9.30612,
- "84": 9.4209,
- "85": 9.61371,
- "86": 9.07649,
- "87": 9.5945,
- "88": 9.75068,
- "89": 9.60238,
- "90": 9.81898,
- "91": 9.33894,
- "92": 9.35716,
- "93": 9.07879,
- "94": 8.83503,
- "95": 9.52172,
- "96": 9.53003,
- "97": 9.31306,
- "98": 9.67783,
- "99": 8.89058,
- "100": 9.39725
+ "1": 10.95174,
+ "2": 10.94869,
+ "3": 10.94862,
+ "4": 10.94969,
+ "5": 10.94753,
+ "6": 10.94472,
+ "7": 10.9465,
+ "8": 10.94472,
+ "9": 10.94186,
+ "10": 10.94475,
+ "11": 10.94106,
+ "12": 10.93297,
+ "13": 10.92308,
+ "14": 10.92595,
+ "15": 10.89971,
+ "16": 10.89087,
+ "17": 10.89815,
+ "18": 10.88334,
+ "19": 10.88984,
+ "20": 10.79734,
+ "21": 10.79298,
+ "22": 10.77995,
+ "23": 10.76969,
+ "24": 10.74572,
+ "25": 10.73535,
+ "26": 10.72507,
+ "27": 10.67798,
+ "28": 10.61999,
+ "29": 10.59158,
+ "30": 10.56324,
+ "31": 10.56515,
+ "32": 10.54389,
+ "33": 10.50913,
+ "34": 10.48324,
+ "35": 10.46996,
+ "36": 10.44684,
+ "37": 10.42651,
+ "38": 10.43045,
+ "39": 10.39271,
+ "40": 10.38041,
+ "41": 10.35137,
+ "42": 10.32751,
+ "43": 10.31121,
+ "44": 10.28343,
+ "45": 10.29603,
+ "46": 10.2478,
+ "47": 10.24382,
+ "48": 10.19395,
+ "49": 10.19672,
+ "50": 10.18923,
+ "51": 10.19743,
+ "52": 10.15224,
+ "53": 10.15771,
+ "54": 10.12072,
+ "55": 10.09673,
+ "56": 10.1211,
+ "57": 10.11169,
+ "58": 10.12097,
+ "59": 10.06241,
+ "60": 10.09233,
+ "61": 10.04171,
+ "62": 10.00953,
+ "63": 10.07952,
+ "64": 10.03239,
+ "65": 10.00322,
+ "66": 10.03954,
+ "67": 10.01959,
+ "68": 9.9857,
+ "69": 10.002,
+ "70": 9.98146,
+ "71": 10.00456,
+ "72": 9.99623,
+ "73": 9.98759,
+ "74": 9.97288,
+ "75": 9.93232,
+ "76": 9.96721,
+ "77": 9.96948,
+ "78": 9.92086,
+ "79": 9.91733,
+ "80": 9.93737,
+ "81": 9.95724,
+ "82": 9.89589,
+ "83": 9.86221,
+ "84": 9.80483,
+ "85": 9.78639,
+ "86": 9.89556,
+ "87": 9.90872,
+ "88": 9.88854,
+ "89": 9.83331,
+ "90": 9.82424,
+ "91": 9.83931,
+ "92": 9.82246,
+ "93": 9.75869,
+ "94": 9.83492,
+ "95": 9.82147,
+ "96": 9.81159,
+ "97": 9.75623,
+ "98": 9.78246,
+ "99": 9.82336,
+ "100": 9.71586
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1688.0,
- "2": 1806.0,
- "3": 1675.0,
- "4": 1842.0,
- "5": 1909.0,
- "6": 1908.0,
- "7": 1783.0,
- "8": 1611.0,
- "9": 1753.0,
- "10": 1457.0,
- "11": 1880.0,
- "12": 1683.0,
- "13": 1907.0,
- "14": 1733.0,
- "15": 1930.0,
- "16": 1840.0,
- "17": 1892.0,
- "18": 1650.0,
- "19": 1790.0,
- "20": 1596.0,
- "21": 1765.0,
- "22": 1616.0,
- "23": 1974.0,
- "24": 1621.0,
- "25": 1557.0,
- "26": 1745.0,
- "27": 1722.0,
- "28": 1976.0,
- "29": 2068.0,
- "30": 1860.0,
- "31": 1536.0,
- "32": 1883.0,
- "33": 2071.0,
- "34": 1894.0,
- "35": 1902.0,
- "36": 1885.0,
- "37": 2231.0,
- "38": 2129.0,
- "39": 2333.0,
- "40": 2207.0,
- "41": 2193.0,
- "42": 2322.0,
- "43": 2015.0,
- "44": 2089.0,
- "45": 2095.0,
- "46": 2392.0,
- "47": 2430.0,
- "48": 2414.0,
- "49": 2340.0,
- "50": 2416.0,
- "51": 2613.0,
- "52": 2538.0,
- "53": 2792.0,
- "54": 2801.0,
- "55": 2216.0,
- "56": 2858.0,
- "57": 2381.0,
- "58": 2854.0,
- "59": 2787.0,
- "60": 2457.0,
- "61": 2941.0,
- "62": 2543.0,
- "63": 2408.0,
- "64": 2968.0,
- "65": 2472.0,
- "66": 2977.0,
- "67": 2839.0,
- "68": 2775.0,
- "69": 2832.0,
- "70": 3057.0,
- "71": 2909.0,
- "72": 2421.0,
- "73": 2982.0,
- "74": 1922.0,
- "75": 2474.0,
- "76": 3059.0,
- "77": 3177.0,
- "78": 3067.0,
- "79": 3052.0,
- "80": 3338.0,
- "81": 3644.0,
- "82": 3234.0,
- "83": 2798.0,
- "84": 3196.0,
- "85": 3324.0,
- "86": 2855.0,
- "87": 3820.0,
- "88": 2962.0,
- "89": 3379.0,
- "90": 3096.0,
- "91": 2857.0,
- "92": 3077.0,
- "93": 2693.0,
- "94": 3312.0,
- "95": 3399.0,
- "96": 3378.0,
- "97": 3030.0,
- "98": 3619.0,
- "99": 3160.0,
- "100": 3128.0
+ "1": 1733.0,
+ "2": 1654.0,
+ "3": 1807.0,
+ "4": 1633.0,
+ "5": 1831.0,
+ "6": 1678.0,
+ "7": 1895.0,
+ "8": 1671.0,
+ "9": 1673.0,
+ "10": 1708.0,
+ "11": 1553.0,
+ "12": 1705.0,
+ "13": 1834.0,
+ "14": 1862.0,
+ "15": 1629.0,
+ "16": 1711.0,
+ "17": 1804.0,
+ "18": 1759.0,
+ "19": 1724.0,
+ "20": 1631.0,
+ "21": 1731.0,
+ "22": 1689.0,
+ "23": 1823.0,
+ "24": 1768.0,
+ "25": 1639.0,
+ "26": 1778.0,
+ "27": 1847.0,
+ "28": 1833.0,
+ "29": 1889.0,
+ "30": 1796.0,
+ "31": 1941.0,
+ "32": 1911.0,
+ "33": 1876.0,
+ "34": 1967.0,
+ "35": 2017.0,
+ "36": 1911.0,
+ "37": 2142.0,
+ "38": 2077.0,
+ "39": 2168.0,
+ "40": 2277.0,
+ "41": 2228.0,
+ "42": 2042.0,
+ "43": 2283.0,
+ "44": 2229.0,
+ "45": 2449.0,
+ "46": 2431.0,
+ "47": 2507.0,
+ "48": 2573.0,
+ "49": 2802.0,
+ "50": 2595.0,
+ "51": 2496.0,
+ "52": 2589.0,
+ "53": 2666.0,
+ "54": 2777.0,
+ "55": 2571.0,
+ "56": 2742.0,
+ "57": 2260.0,
+ "58": 3560.0,
+ "59": 2980.0,
+ "60": 2999.0,
+ "61": 2639.0,
+ "62": 3174.0,
+ "63": 3364.0,
+ "64": 3609.0,
+ "65": 2773.0,
+ "66": 3041.0,
+ "67": 3905.0,
+ "68": 3439.0,
+ "69": 3016.0,
+ "70": 3321.0,
+ "71": 3199.0,
+ "72": 2945.0,
+ "73": 3486.0,
+ "74": 3218.0,
+ "75": 3115.0,
+ "76": 3388.0,
+ "77": 3690.0,
+ "78": 3383.0,
+ "79": 3277.0,
+ "80": 2844.0,
+ "81": 3319.0,
+ "82": 2759.0,
+ "83": 2962.0,
+ "84": 3000.0,
+ "85": 2744.0,
+ "86": 3058.0,
+ "87": 3003.0,
+ "88": 3063.0,
+ "89": 2947.0,
+ "90": 3757.0,
+ "91": 2785.0,
+ "92": 2967.0,
+ "93": 2920.0,
+ "94": 3199.0,
+ "95": 3313.0,
+ "96": 3580.0,
+ "97": 3554.0,
+ "98": 3283.0,
+ "99": 3298.0,
+ "100": 3386.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 516456960.0,
- "2": 516456960.0,
- "3": 516456960.0,
- "4": 516456960.0,
- "5": 516456960.0,
- "6": 516456960.0,
- "7": 516456960.0,
- "8": 516456960.0,
- "9": 516456960.0,
- "10": 516456960.0,
- "11": 516456960.0,
- "12": 516456960.0,
- "13": 516456960.0,
- "14": 516456960.0,
- "15": 516456960.0,
- "16": 516456960.0,
- "17": 516456960.0,
- "18": 516456960.0,
- "19": 516456960.0,
- "20": 516456960.0,
- "21": 516456960.0,
- "22": 516456960.0,
- "23": 516456960.0,
- "24": 516456960.0,
- "25": 516456960.0,
- "26": 516456960.0,
- "27": 516456960.0,
- "28": 516456960.0,
- "29": 516456960.0,
- "30": 516456960.0,
- "31": 516456960.0,
- "32": 516456960.0,
- "33": 516456960.0,
- "34": 516456960.0,
- "35": 516456960.0,
- "36": 516456960.0,
- "37": 516456960.0,
- "38": 516456960.0,
- "39": 516456960.0,
- "40": 516456960.0,
- "41": 516456960.0,
- "42": 516456960.0,
- "43": 516456960.0,
- "44": 516456960.0,
- "45": 516456960.0,
- "46": 516456960.0,
- "47": 516456960.0,
- "48": 516456960.0,
- "49": 516456960.0,
- "50": 516456960.0,
- "51": 516456960.0,
- "52": 516456960.0,
- "53": 516456960.0,
- "54": 516456960.0,
- "55": 516456960.0,
- "56": 516456960.0,
- "57": 516456960.0,
- "58": 516456960.0,
- "59": 516456960.0,
- "60": 516456960.0,
- "61": 516456960.0,
- "62": 516456960.0,
- "63": 516456960.0,
- "64": 516456960.0,
- "65": 516456960.0,
- "66": 516456960.0,
- "67": 516456960.0,
- "68": 516456960.0,
- "69": 516456960.0,
- "70": 516456960.0,
- "71": 516456960.0,
- "72": 516456960.0,
- "73": 516456960.0,
- "74": 516456960.0,
- "75": 516456960.0,
- "76": 516456960.0,
- "77": 516456960.0,
- "78": 516456960.0,
- "79": 516456960.0,
- "80": 516456960.0,
- "81": 516456960.0,
- "82": 516456960.0,
- "83": 516456960.0,
- "84": 516456960.0,
- "85": 516456960.0,
- "86": 516456960.0,
- "87": 516456960.0,
- "88": 516456960.0,
- "89": 516456960.0,
- "90": 516456960.0,
- "91": 516456960.0,
- "92": 516456960.0,
- "93": 516456960.0,
- "94": 516456960.0,
- "95": 516456960.0,
- "96": 516456960.0,
- "97": 516456960.0,
- "98": 516456960.0,
- "99": 516456960.0,
- "100": 516456960.0
+ "1": 517243392.0,
+ "2": 517243392.0,
+ "3": 517243392.0,
+ "4": 517243392.0,
+ "5": 517243392.0,
+ "6": 517243392.0,
+ "7": 517243392.0,
+ "8": 517243392.0,
+ "9": 517243392.0,
+ "10": 517243392.0,
+ "11": 517243392.0,
+ "12": 517243392.0,
+ "13": 517243392.0,
+ "14": 517243392.0,
+ "15": 517243392.0,
+ "16": 517243392.0,
+ "17": 517243392.0,
+ "18": 517243392.0,
+ "19": 517243392.0,
+ "20": 517243392.0,
+ "21": 517243392.0,
+ "22": 517243392.0,
+ "23": 517243392.0,
+ "24": 517243392.0,
+ "25": 517243392.0,
+ "26": 517243392.0,
+ "27": 517243392.0,
+ "28": 517243392.0,
+ "29": 517243392.0,
+ "30": 517243392.0,
+ "31": 517243392.0,
+ "32": 517243392.0,
+ "33": 517243392.0,
+ "34": 517243392.0,
+ "35": 517243392.0,
+ "36": 517243392.0,
+ "37": 517243392.0,
+ "38": 517243392.0,
+ "39": 517243392.0,
+ "40": 517243392.0,
+ "41": 517243392.0,
+ "42": 517243392.0,
+ "43": 517243392.0,
+ "44": 517243392.0,
+ "45": 517243392.0,
+ "46": 517243392.0,
+ "47": 517243392.0,
+ "48": 517243392.0,
+ "49": 517243392.0,
+ "50": 517243392.0,
+ "51": 517243392.0,
+ "52": 517243392.0,
+ "53": 517243392.0,
+ "54": 517243392.0,
+ "55": 517243392.0,
+ "56": 517243392.0,
+ "57": 517243392.0,
+ "58": 517243392.0,
+ "59": 517243392.0,
+ "60": 517243392.0,
+ "61": 517243392.0,
+ "62": 517243392.0,
+ "63": 517243392.0,
+ "64": 517243392.0,
+ "65": 517243392.0,
+ "66": 517243392.0,
+ "67": 517243392.0,
+ "68": 517243392.0,
+ "69": 517243392.0,
+ "70": 517243392.0,
+ "71": 517243392.0,
+ "72": 517243392.0,
+ "73": 517243392.0,
+ "74": 517243392.0,
+ "75": 517243392.0,
+ "76": 517243392.0,
+ "77": 517243392.0,
+ "78": 517243392.0,
+ "79": 517243392.0,
+ "80": 517243392.0,
+ "81": 517243392.0,
+ "82": 517243392.0,
+ "83": 517243392.0,
+ "84": 517243392.0,
+ "85": 517243392.0,
+ "86": 517243392.0,
+ "87": 517243392.0,
+ "88": 517243392.0,
+ "89": 517243392.0,
+ "90": 517243392.0,
+ "91": 517243392.0,
+ "92": 517243392.0,
+ "93": 517243392.0,
+ "94": 517243392.0,
+ "95": 517243392.0,
+ "96": 517243392.0,
+ "97": 517243392.0,
+ "98": 517243392.0,
+ "99": 517243392.0,
+ "100": 517243392.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1246525952.0,
- "2": 1426598400.0,
- "3": 1426598400.0,
- "4": 1426598400.0,
- "5": 1426598400.0,
- "6": 1426598400.0,
- "7": 1426598400.0,
- "8": 1426598400.0,
- "9": 1426598400.0,
- "10": 1426598400.0,
- "11": 1426598400.0,
- "12": 1426598400.0,
- "13": 1426598400.0,
- "14": 1426598400.0,
- "15": 1426598400.0,
- "16": 1426598400.0,
- "17": 1426598400.0,
- "18": 1426598400.0,
- "19": 1426598400.0,
- "20": 1426598400.0,
- "21": 1426598400.0,
- "22": 1426598400.0,
- "23": 1426598400.0,
- "24": 1426598400.0,
- "25": 1426598400.0,
- "26": 1426598400.0,
- "27": 1426598400.0,
- "28": 1426598400.0,
- "29": 1426598400.0,
- "30": 1426598400.0,
- "31": 1426598400.0,
- "32": 1426598400.0,
- "33": 1426598400.0,
- "34": 1426598400.0,
- "35": 1426598400.0,
- "36": 1426598400.0,
- "37": 1426598400.0,
- "38": 1426598400.0,
- "39": 1426598400.0,
- "40": 1426598400.0,
- "41": 1426598400.0,
- "42": 1426598400.0,
- "43": 1426598400.0,
- "44": 1426598400.0,
- "45": 1426598400.0,
- "46": 1426598400.0,
- "47": 1426598400.0,
- "48": 1426598400.0,
- "49": 1426598400.0,
- "50": 1426598400.0,
- "51": 1426598400.0,
- "52": 1426598400.0,
- "53": 1426598400.0,
- "54": 1426598400.0,
- "55": 1426598400.0,
- "56": 1426598400.0,
- "57": 1426598400.0,
- "58": 1426598400.0,
- "59": 1426598400.0,
- "60": 1426598400.0,
- "61": 1426598400.0,
- "62": 1426598400.0,
- "63": 1426598400.0,
- "64": 1426598400.0,
- "65": 1426598400.0,
- "66": 1426598400.0,
- "67": 1426598400.0,
- "68": 1426598400.0,
- "69": 1426598400.0,
- "70": 1426598400.0,
- "71": 1426598400.0,
- "72": 1426598400.0,
- "73": 1426598400.0,
- "74": 1426598400.0,
- "75": 1426598400.0,
- "76": 1426598400.0,
- "77": 1426598400.0,
- "78": 1426598400.0,
- "79": 1426598400.0,
- "80": 1426598400.0,
- "81": 1426598400.0,
- "82": 1426598400.0,
- "83": 1426598400.0,
- "84": 1426598400.0,
- "85": 1426598400.0,
- "86": 1426598400.0,
- "87": 1426598400.0,
- "88": 1426598400.0,
- "89": 1426598400.0,
- "90": 1426598400.0,
- "91": 1426598400.0,
- "92": 1426598400.0,
- "93": 1426598400.0,
- "94": 1426598400.0,
- "95": 1426598400.0,
- "96": 1426598400.0,
- "97": 1426598400.0,
- "98": 1426598400.0,
- "99": 1426598400.0,
- "100": 1426598400.0
+ "1": 1245477376.0,
+ "2": 1427384832.0,
+ "3": 1427384832.0,
+ "4": 1427384832.0,
+ "5": 1427384832.0,
+ "6": 1427384832.0,
+ "7": 1427384832.0,
+ "8": 1427384832.0,
+ "9": 1427384832.0,
+ "10": 1427384832.0,
+ "11": 1427384832.0,
+ "12": 1427384832.0,
+ "13": 1427384832.0,
+ "14": 1427384832.0,
+ "15": 1427384832.0,
+ "16": 1427384832.0,
+ "17": 1427384832.0,
+ "18": 1427384832.0,
+ "19": 1427384832.0,
+ "20": 1427384832.0,
+ "21": 1427384832.0,
+ "22": 1427384832.0,
+ "23": 1427384832.0,
+ "24": 1427384832.0,
+ "25": 1427384832.0,
+ "26": 1427384832.0,
+ "27": 1427384832.0,
+ "28": 1427384832.0,
+ "29": 1427384832.0,
+ "30": 1427384832.0,
+ "31": 1427384832.0,
+ "32": 1427384832.0,
+ "33": 1427384832.0,
+ "34": 1427384832.0,
+ "35": 1427384832.0,
+ "36": 1427384832.0,
+ "37": 1427384832.0,
+ "38": 1427384832.0,
+ "39": 1427384832.0,
+ "40": 1427384832.0,
+ "41": 1427384832.0,
+ "42": 1427384832.0,
+ "43": 1427384832.0,
+ "44": 1427384832.0,
+ "45": 1427384832.0,
+ "46": 1427384832.0,
+ "47": 1427384832.0,
+ "48": 1427384832.0,
+ "49": 1427384832.0,
+ "50": 1427384832.0,
+ "51": 1428171264.0,
+ "52": 1428171264.0,
+ "53": 1428171264.0,
+ "54": 1428171264.0,
+ "55": 1428171264.0,
+ "56": 1428171264.0,
+ "57": 1428171264.0,
+ "58": 1428171264.0,
+ "59": 1428171264.0,
+ "60": 1428171264.0,
+ "61": 1428171264.0,
+ "62": 1428171264.0,
+ "63": 1428171264.0,
+ "64": 1428171264.0,
+ "65": 1428171264.0,
+ "66": 1428171264.0,
+ "67": 1428171264.0,
+ "68": 1428171264.0,
+ "69": 1428171264.0,
+ "70": 1428171264.0,
+ "71": 1428171264.0,
+ "72": 1428171264.0,
+ "73": 1428171264.0,
+ "74": 1428171264.0,
+ "75": 1428171264.0,
+ "76": 1428171264.0,
+ "77": 1428171264.0,
+ "78": 1428171264.0,
+ "79": 1428171264.0,
+ "80": 1428171264.0,
+ "81": 1428171264.0,
+ "82": 1428171264.0,
+ "83": 1428171264.0,
+ "84": 1428171264.0,
+ "85": 1428171264.0,
+ "86": 1428171264.0,
+ "87": 1428171264.0,
+ "88": 1428171264.0,
+ "89": 1428171264.0,
+ "90": 1428171264.0,
+ "91": 1428171264.0,
+ "92": 1428171264.0,
+ "93": 1428171264.0,
+ "94": 1428171264.0,
+ "95": 1428171264.0,
+ "96": 1428171264.0,
+ "97": 1428171264.0,
+ "98": 1428171264.0,
+ "99": 1428171264.0,
+ "100": 1428171264.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 8.71736,
- "2": 0.17115,
- "3": 0.15694,
- "4": 0.13982,
- "5": 0.13869,
- "6": 0.1336,
- "7": 0.13504,
- "8": 0.13243,
- "9": 0.13367,
- "10": 0.13419,
- "11": 0.13733,
- "12": 0.13769,
- "13": 0.13945,
- "14": 0.13947,
- "15": 0.1359,
- "16": 0.13522,
- "17": 0.13429,
- "18": 0.13312,
- "19": 0.13374,
- "20": 0.13297,
- "21": 0.13311,
- "22": 0.13277,
- "23": 0.13534,
- "24": 0.13287,
- "25": 0.12793,
- "26": 0.12692,
- "27": 0.1283,
- "28": 0.13508,
- "29": 0.13475,
- "30": 0.1318,
- "31": 0.13396,
- "32": 0.13344,
- "33": 0.13398,
- "34": 0.13071,
- "35": 0.1284,
- "36": 0.12752,
- "37": 0.12689,
- "38": 0.12666,
- "39": 0.12799,
- "40": 0.12834,
- "41": 0.12686,
- "42": 0.12597,
- "43": 0.1242,
- "44": 0.12724,
- "45": 0.12459,
- "46": 0.12693,
- "47": 0.12473,
- "48": 0.12666,
- "49": 0.12677,
- "50": 0.12611,
- "51": 0.14947,
- "52": 0.12685,
- "53": 0.12533,
- "54": 0.12565,
- "55": 0.12664,
- "56": 0.12771,
- "57": 0.12644,
- "58": 0.12656,
- "59": 0.12707,
- "60": 0.12763,
- "61": 0.12599,
- "62": 0.12667,
- "63": 0.12558,
- "64": 0.12865,
- "65": 0.12684,
- "66": 0.12749,
- "67": 0.12671,
- "68": 0.12725,
- "69": 0.1267,
- "70": 0.1263,
- "71": 0.12741,
- "72": 0.12748,
- "73": 0.1278,
- "74": 0.12653,
- "75": 0.12606,
- "76": 0.12649,
- "77": 0.12666,
- "78": 0.12626,
- "79": 0.12702,
- "80": 0.12831,
- "81": 0.12686,
- "82": 0.12628,
- "83": 0.12693,
- "84": 0.12714,
- "85": 0.12632,
- "86": 0.12756,
- "87": 0.12631,
- "88": 0.12895,
- "89": 0.1284,
- "90": 0.12636,
- "91": 0.12805,
- "92": 0.12691,
- "93": 0.12665,
- "94": 0.12749,
- "95": 0.12697,
- "96": 0.12622,
- "97": 0.12701,
- "98": 0.12878,
- "99": 0.12567,
- "100": 0.12677
+ "1": "nan",
+ "2": 6.01523,
+ "3": 0.15045,
+ "4": 0.13171,
+ "5": 0.18136,
+ "6": 0.13713,
+ "7": 0.12894,
+ "8": 0.12981,
+ "9": 0.12827,
+ "10": 0.12966,
+ "11": 0.12933,
+ "12": 0.13227,
+ "13": 0.13068,
+ "14": 0.12952,
+ "15": 0.12852,
+ "16": 0.12949,
+ "17": 0.12915,
+ "18": 0.13203,
+ "19": 0.12705,
+ "20": 0.12904,
+ "21": 0.12982,
+ "22": 0.13055,
+ "23": 0.13014,
+ "24": 0.12978,
+ "25": 0.13006,
+ "26": 0.13104,
+ "27": 0.12883,
+ "28": 0.13148,
+ "29": 0.13038,
+ "30": 0.13032,
+ "31": 0.1291,
+ "32": 0.13108,
+ "33": 0.12902,
+ "34": 0.13352,
+ "35": 0.12925,
+ "36": 0.13068,
+ "37": 0.13224,
+ "38": 0.13025,
+ "39": 0.12967,
+ "40": 0.12971,
+ "41": 0.13003,
+ "42": 0.12981,
+ "43": 0.12924,
+ "44": 0.13001,
+ "45": 0.12917,
+ "46": 0.12945,
+ "47": 0.12893,
+ "48": 0.12984,
+ "49": 0.13153,
+ "50": 0.129,
+ "51": 0.1539,
+ "52": 0.13001,
+ "53": 0.13183,
+ "54": 0.13228,
+ "55": 0.13071,
+ "56": 0.13172,
+ "57": 0.13096,
+ "58": 0.13283,
+ "59": 0.13102,
+ "60": 0.13268,
+ "61": 0.13119,
+ "62": 0.1325,
+ "63": 0.13174,
+ "64": 0.13304,
+ "65": 0.13205,
+ "66": 0.13047,
+ "67": 0.13001,
+ "68": 0.13311,
+ "69": 0.13345,
+ "70": 0.13033,
+ "71": 0.13057,
+ "72": 0.13144,
+ "73": 0.13138,
+ "74": 0.12948,
+ "75": 0.13151,
+ "76": 0.13008,
+ "77": 0.12854,
+ "78": 0.12937,
+ "79": 0.1305,
+ "80": 0.13093,
+ "81": 0.12909,
+ "82": 0.13023,
+ "83": 0.13097,
+ "84": 0.12853,
+ "85": 0.12884,
+ "86": 0.12913,
+ "87": 0.13145,
+ "88": 0.13136,
+ "89": 0.12934,
+ "90": 0.12937,
+ "91": 0.13206,
+ "92": 0.12861,
+ "93": 0.12976,
+ "94": 0.13058,
+ "95": 0.13136,
+ "96": 0.13107,
+ "97": 0.13091,
+ "98": 0.13103,
+ "99": 0.13133,
+ "100": 0.13005
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective/model_config.yaml
index f906e5f8439..aacc077e937 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -50,4 +50,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute/golden_values_dev_dgx_h100.json
index f1a58884e99..2a650ce5a22 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.85949,
- "2": 10.85553,
- "3": 10.86544,
- "4": 10.84555,
- "5": 10.88343,
- "6": 10.89431,
- "7": 10.87069,
- "8": 10.86982,
- "9": 10.8692,
- "10": 10.83882,
- "11": 10.89437,
- "12": 10.8798,
- "13": 10.87986,
- "14": 10.90316,
- "15": 10.84047,
- "16": 10.83785,
- "17": 10.8067,
- "18": 10.83027,
- "19": 10.82265,
- "20": 10.73196,
- "21": 10.70751,
- "22": 10.56001,
- "23": 10.72404,
- "24": 10.61114,
- "25": 10.54812,
- "26": 10.61333,
- "27": 10.63051,
- "28": 10.56645,
- "29": 10.59672,
- "30": 10.37134,
- "31": 10.11723,
- "32": 10.46131,
- "33": 10.4525,
- "34": 10.21689,
- "35": 10.27171,
- "36": 10.2312,
- "37": 10.34809,
- "38": 10.18839,
- "39": 10.41045,
- "40": 10.09427,
- "41": 10.1471,
- "42": 10.21241,
- "43": 9.84107,
- "44": 9.95919,
- "45": 9.84081,
- "46": 9.82483,
- "47": 10.13877,
- "48": 9.85832,
- "49": 9.54703,
- "50": 9.90876,
- "51": 9.85581,
- "52": 9.75235,
- "53": 10.07582,
- "54": 9.95687,
- "55": 9.882,
- "56": 9.63137,
- "57": 9.48647,
- "58": 9.83111,
- "59": 9.58896,
- "60": 9.50647,
- "61": 9.70361,
- "62": 9.98283,
- "63": 9.38302,
- "64": 9.77906,
- "65": 8.95171,
- "66": 9.70162,
- "67": 9.372,
- "68": 9.78849,
- "69": 9.79851,
- "70": 9.74738,
- "71": 9.61908,
- "72": 9.58496,
- "73": 9.49723,
- "74": 8.93927,
- "75": 9.42706,
- "76": 9.08018,
- "77": 10.06566,
- "78": 9.72889,
- "79": 9.37757,
- "80": 9.40987,
- "81": 9.47974,
- "82": 9.70177,
- "83": 9.30611,
- "84": 9.42088,
- "85": 9.61376,
- "86": 9.07651,
- "87": 9.59452,
- "88": 9.75067,
- "89": 9.60239,
- "90": 9.81895,
- "91": 9.33895,
- "92": 9.35712,
- "93": 9.07879,
- "94": 8.83504,
- "95": 9.52168,
- "96": 9.53002,
- "97": 9.31306,
- "98": 9.67783,
- "99": 8.89053,
- "100": 9.39725
+ "1": 10.95174,
+ "2": 10.94869,
+ "3": 10.94863,
+ "4": 10.94972,
+ "5": 10.94752,
+ "6": 10.94473,
+ "7": 10.94653,
+ "8": 10.94474,
+ "9": 10.94189,
+ "10": 10.94477,
+ "11": 10.94104,
+ "12": 10.93297,
+ "13": 10.92309,
+ "14": 10.92595,
+ "15": 10.89973,
+ "16": 10.89082,
+ "17": 10.89814,
+ "18": 10.88334,
+ "19": 10.8898,
+ "20": 10.79729,
+ "21": 10.79296,
+ "22": 10.77999,
+ "23": 10.76965,
+ "24": 10.74571,
+ "25": 10.73535,
+ "26": 10.72504,
+ "27": 10.67796,
+ "28": 10.62001,
+ "29": 10.59162,
+ "30": 10.56331,
+ "31": 10.56512,
+ "32": 10.5439,
+ "33": 10.50915,
+ "34": 10.48322,
+ "35": 10.46993,
+ "36": 10.44681,
+ "37": 10.4265,
+ "38": 10.43043,
+ "39": 10.39271,
+ "40": 10.38041,
+ "41": 10.35139,
+ "42": 10.32749,
+ "43": 10.31123,
+ "44": 10.28348,
+ "45": 10.29601,
+ "46": 10.24783,
+ "47": 10.2438,
+ "48": 10.19392,
+ "49": 10.19672,
+ "50": 10.18926,
+ "51": 10.19743,
+ "52": 10.15224,
+ "53": 10.15771,
+ "54": 10.12072,
+ "55": 10.09676,
+ "56": 10.12112,
+ "57": 10.11171,
+ "58": 10.12098,
+ "59": 10.06239,
+ "60": 10.09232,
+ "61": 10.04176,
+ "62": 10.00955,
+ "63": 10.07951,
+ "64": 10.03234,
+ "65": 10.00322,
+ "66": 10.03952,
+ "67": 10.0196,
+ "68": 9.98576,
+ "69": 10.00194,
+ "70": 9.98144,
+ "71": 10.00457,
+ "72": 9.99621,
+ "73": 9.98763,
+ "74": 9.97288,
+ "75": 9.93232,
+ "76": 9.96724,
+ "77": 9.9695,
+ "78": 9.92082,
+ "79": 9.91733,
+ "80": 9.93739,
+ "81": 9.95723,
+ "82": 9.8959,
+ "83": 9.8622,
+ "84": 9.80485,
+ "85": 9.78638,
+ "86": 9.89558,
+ "87": 9.90872,
+ "88": 9.88855,
+ "89": 9.83327,
+ "90": 9.82428,
+ "91": 9.83931,
+ "92": 9.82241,
+ "93": 9.75867,
+ "94": 9.83496,
+ "95": 9.82146,
+ "96": 9.81159,
+ "97": 9.75623,
+ "98": 9.78244,
+ "99": 9.82336,
+ "100": 9.71588
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 71.0,
- "2": 65.0,
- "3": 68.0,
- "4": 57.0,
- "5": 55.0,
- "6": 70.0,
- "7": 73.0,
- "8": 58.0,
- "9": 66.0,
- "10": 65.0,
- "11": 58.0,
- "12": 77.0,
- "13": 50.0,
- "14": 65.0,
- "15": 71.0,
- "16": 68.0,
- "17": 58.0,
- "18": 57.0,
- "19": 68.0,
- "20": 61.0,
- "21": 65.0,
- "22": 57.0,
- "23": 83.0,
- "24": 58.0,
- "25": 66.0,
- "26": 63.0,
- "27": 80.0,
- "28": 82.0,
- "29": 72.0,
- "30": 71.0,
- "31": 68.0,
- "32": 75.0,
- "33": 85.0,
- "34": 63.0,
- "35": 69.0,
- "36": 58.0,
- "37": 83.0,
- "38": 65.0,
- "39": 68.0,
- "40": 81.0,
- "41": 72.0,
- "42": 76.0,
- "43": 84.0,
- "44": 85.0,
- "45": 85.0,
- "46": 79.0,
- "47": 81.0,
- "48": 68.0,
- "49": 89.0,
- "50": 80.0,
- "51": 70.0,
- "52": 81.0,
- "53": 95.0,
- "54": 101.0,
- "55": 58.0,
- "56": 90.0,
- "57": 83.0,
- "58": 90.0,
- "59": 79.0,
- "60": 84.0,
- "61": 92.0,
- "62": 102.0,
+ "1": 74.0,
+ "2": 75.0,
+ "3": 64.0,
+ "4": 85.0,
+ "5": 53.0,
+ "6": 73.0,
+ "7": 70.0,
+ "8": 69.0,
+ "9": 56.0,
+ "10": 54.0,
+ "11": 77.0,
+ "12": 82.0,
+ "13": 67.0,
+ "14": 70.0,
+ "15": 61.0,
+ "16": 73.0,
+ "17": 67.0,
+ "18": 76.0,
+ "19": 73.0,
+ "20": 70.0,
+ "21": 63.0,
+ "22": 73.0,
+ "23": 56.0,
+ "24": 55.0,
+ "25": 61.0,
+ "26": 57.0,
+ "27": 71.0,
+ "28": 64.0,
+ "29": 69.0,
+ "30": 69.0,
+ "31": 54.0,
+ "32": 79.0,
+ "33": 61.0,
+ "34": 72.0,
+ "35": 77.0,
+ "36": 88.0,
+ "37": 70.0,
+ "38": 75.0,
+ "39": 81.0,
+ "40": 69.0,
+ "41": 90.0,
+ "42": 66.0,
+ "43": 73.0,
+ "44": 83.0,
+ "45": 75.0,
+ "46": 68.0,
+ "47": 75.0,
+ "48": 88.0,
+ "49": 64.0,
+ "50": 96.0,
+ "51": 86.0,
+ "52": 90.0,
+ "53": 84.0,
+ "54": 74.0,
+ "55": 87.0,
+ "56": 79.0,
+ "57": 67.0,
+ "58": 75.0,
+ "59": 77.0,
+ "60": 86.0,
+ "61": 72.0,
+ "62": 98.0,
"63": 78.0,
- "64": 73.0,
- "65": 81.0,
- "66": 88.0,
- "67": 54.0,
- "68": 57.0,
- "69": 72.0,
- "70": 88.0,
- "71": 82.0,
- "72": 64.0,
- "73": 78.0,
- "74": 76.0,
- "75": 70.0,
- "76": 78.0,
- "77": 67.0,
- "78": 86.0,
- "79": 76.0,
- "80": 90.0,
- "81": 92.0,
- "82": 72.0,
- "83": 61.0,
- "84": 65.0,
- "85": 89.0,
- "86": 73.0,
- "87": 89.0,
- "88": 63.0,
- "89": 83.0,
- "90": 72.0,
- "91": 55.0,
- "92": 63.0,
- "93": 47.0,
- "94": 74.0,
- "95": 70.0,
- "96": 73.0,
- "97": 80.0,
- "98": 76.0,
- "99": 68.0,
- "100": 75.0
+ "64": 85.0,
+ "65": 78.0,
+ "66": 93.0,
+ "67": 66.0,
+ "68": 75.0,
+ "69": 85.0,
+ "70": 95.0,
+ "71": 85.0,
+ "72": 74.0,
+ "73": 95.0,
+ "74": 103.0,
+ "75": 84.0,
+ "76": 82.0,
+ "77": 82.0,
+ "78": 76.0,
+ "79": 94.0,
+ "80": 71.0,
+ "81": 91.0,
+ "82": 61.0,
+ "83": 74.0,
+ "84": 70.0,
+ "85": 57.0,
+ "86": 77.0,
+ "87": 70.0,
+ "88": 77.0,
+ "89": 74.0,
+ "90": 87.0,
+ "91": 75.0,
+ "92": 77.0,
+ "93": 77.0,
+ "94": 76.0,
+ "95": 93.0,
+ "96": 66.0,
+ "97": 96.0,
+ "98": 73.0,
+ "99": 83.0,
+ "100": 96.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 545423872.0,
- "2": 545423872.0,
- "3": 545423872.0,
- "4": 545423872.0,
- "5": 545423872.0,
- "6": 545423872.0,
- "7": 545423872.0,
- "8": 545423872.0,
- "9": 545423872.0,
- "10": 545423872.0,
- "11": 545423872.0,
- "12": 545423872.0,
- "13": 545423872.0,
- "14": 545423872.0,
- "15": 545423872.0,
- "16": 545423872.0,
- "17": 545423872.0,
- "18": 545423872.0,
- "19": 545423872.0,
- "20": 545423872.0,
- "21": 545423872.0,
- "22": 545423872.0,
- "23": 545423872.0,
- "24": 545423872.0,
- "25": 545423872.0,
- "26": 545423872.0,
- "27": 545423872.0,
- "28": 545423872.0,
- "29": 545423872.0,
- "30": 545423872.0,
- "31": 545423872.0,
- "32": 545423872.0,
- "33": 545423872.0,
- "34": 545423872.0,
- "35": 545423872.0,
- "36": 545423872.0,
- "37": 545423872.0,
- "38": 545423872.0,
- "39": 545423872.0,
- "40": 545423872.0,
- "41": 545423872.0,
- "42": 545423872.0,
- "43": 545423872.0,
- "44": 545423872.0,
- "45": 545423872.0,
- "46": 545423872.0,
- "47": 545423872.0,
- "48": 545423872.0,
- "49": 545423872.0,
- "50": 545423872.0,
- "51": 545423872.0,
- "52": 545423872.0,
- "53": 545423872.0,
- "54": 545423872.0,
- "55": 545423872.0,
- "56": 545423872.0,
- "57": 545423872.0,
- "58": 545423872.0,
- "59": 545423872.0,
- "60": 545423872.0,
- "61": 545423872.0,
- "62": 545423872.0,
- "63": 545423872.0,
- "64": 545423872.0,
- "65": 545423872.0,
- "66": 545423872.0,
- "67": 545423872.0,
- "68": 545423872.0,
- "69": 545423872.0,
- "70": 545423872.0,
- "71": 545423872.0,
- "72": 545423872.0,
- "73": 545423872.0,
- "74": 545423872.0,
- "75": 545423872.0,
- "76": 545423872.0,
- "77": 545423872.0,
- "78": 545423872.0,
- "79": 545423872.0,
- "80": 545423872.0,
- "81": 545423872.0,
- "82": 545423872.0,
- "83": 545423872.0,
- "84": 545423872.0,
- "85": 545423872.0,
- "86": 545423872.0,
- "87": 545423872.0,
- "88": 545423872.0,
- "89": 545423872.0,
- "90": 545423872.0,
- "91": 545423872.0,
- "92": 545423872.0,
- "93": 545423872.0,
- "94": 545423872.0,
- "95": 545423872.0,
- "96": 545423872.0,
- "97": 545423872.0,
- "98": 545423872.0,
- "99": 545423872.0,
- "100": 545423872.0
+ "1": 546210304.0,
+ "2": 546210304.0,
+ "3": 546210304.0,
+ "4": 546210304.0,
+ "5": 546210304.0,
+ "6": 546210304.0,
+ "7": 546210304.0,
+ "8": 546210304.0,
+ "9": 546210304.0,
+ "10": 546210304.0,
+ "11": 546210304.0,
+ "12": 546210304.0,
+ "13": 546210304.0,
+ "14": 546210304.0,
+ "15": 546210304.0,
+ "16": 546210304.0,
+ "17": 546210304.0,
+ "18": 546210304.0,
+ "19": 546210304.0,
+ "20": 546210304.0,
+ "21": 546210304.0,
+ "22": 546210304.0,
+ "23": 546210304.0,
+ "24": 546210304.0,
+ "25": 546210304.0,
+ "26": 546210304.0,
+ "27": 546210304.0,
+ "28": 546210304.0,
+ "29": 546210304.0,
+ "30": 546210304.0,
+ "31": 546210304.0,
+ "32": 546210304.0,
+ "33": 546210304.0,
+ "34": 546210304.0,
+ "35": 546210304.0,
+ "36": 546210304.0,
+ "37": 546210304.0,
+ "38": 546210304.0,
+ "39": 546210304.0,
+ "40": 546210304.0,
+ "41": 546210304.0,
+ "42": 546210304.0,
+ "43": 546210304.0,
+ "44": 546210304.0,
+ "45": 546210304.0,
+ "46": 546210304.0,
+ "47": 546210304.0,
+ "48": 546210304.0,
+ "49": 546210304.0,
+ "50": 546210304.0,
+ "51": 546210304.0,
+ "52": 546210304.0,
+ "53": 546210304.0,
+ "54": 546210304.0,
+ "55": 546210304.0,
+ "56": 546210304.0,
+ "57": 546210304.0,
+ "58": 546210304.0,
+ "59": 546210304.0,
+ "60": 546210304.0,
+ "61": 546210304.0,
+ "62": 546210304.0,
+ "63": 546210304.0,
+ "64": 546210304.0,
+ "65": 546210304.0,
+ "66": 546210304.0,
+ "67": 546210304.0,
+ "68": 546210304.0,
+ "69": 546210304.0,
+ "70": 546210304.0,
+ "71": 546210304.0,
+ "72": 546210304.0,
+ "73": 546210304.0,
+ "74": 546210304.0,
+ "75": 546210304.0,
+ "76": 546210304.0,
+ "77": 546210304.0,
+ "78": 546210304.0,
+ "79": 546210304.0,
+ "80": 546210304.0,
+ "81": 546210304.0,
+ "82": 546210304.0,
+ "83": 546210304.0,
+ "84": 546210304.0,
+ "85": 546210304.0,
+ "86": 546210304.0,
+ "87": 546210304.0,
+ "88": 546210304.0,
+ "89": 546210304.0,
+ "90": 546210304.0,
+ "91": 546210304.0,
+ "92": 546210304.0,
+ "93": 546210304.0,
+ "94": 546210304.0,
+ "95": 546210304.0,
+ "96": 546210304.0,
+ "97": 546210304.0,
+ "98": 546210304.0,
+ "99": 546210304.0,
+ "100": 546210304.0
}
},
"mem-max-allocated-bytes": {
@@ -326,105 +326,105 @@
"step_interval": 1,
"values": {
"1": 1713797632.0,
- "2": 1893870080.0,
- "3": 1893870080.0,
- "4": 1893870080.0,
- "5": 1893870080.0,
- "6": 1893870080.0,
- "7": 1893870080.0,
- "8": 1893870080.0,
- "9": 1893870080.0,
- "10": 1893870080.0,
- "11": 1893870080.0,
- "12": 1893870080.0,
- "13": 1893870080.0,
- "14": 1893870080.0,
- "15": 1893870080.0,
- "16": 1893870080.0,
- "17": 1893870080.0,
- "18": 1893870080.0,
- "19": 1893870080.0,
- "20": 1893870080.0,
- "21": 1893870080.0,
- "22": 1893870080.0,
- "23": 1893870080.0,
- "24": 1893870080.0,
- "25": 1893870080.0,
- "26": 1893870080.0,
- "27": 1893870080.0,
- "28": 1893870080.0,
- "29": 1893870080.0,
- "30": 1893870080.0,
- "31": 1893870080.0,
- "32": 1893870080.0,
- "33": 1893870080.0,
- "34": 1893870080.0,
- "35": 1893870080.0,
- "36": 1893870080.0,
- "37": 1893870080.0,
- "38": 1893870080.0,
- "39": 1893870080.0,
- "40": 1893870080.0,
- "41": 1893870080.0,
- "42": 1893870080.0,
- "43": 1893870080.0,
- "44": 1893870080.0,
- "45": 1893870080.0,
- "46": 1893870080.0,
- "47": 1893870080.0,
- "48": 1893870080.0,
- "49": 1893870080.0,
- "50": 1893870080.0,
- "51": 1893870080.0,
- "52": 1893870080.0,
- "53": 1893870080.0,
- "54": 1893870080.0,
- "55": 1893870080.0,
- "56": 1893870080.0,
- "57": 1893870080.0,
- "58": 1893870080.0,
- "59": 1893870080.0,
- "60": 1893870080.0,
- "61": 1893870080.0,
- "62": 1893870080.0,
- "63": 1893870080.0,
- "64": 1893870080.0,
- "65": 1893870080.0,
- "66": 1893870080.0,
- "67": 1893870080.0,
- "68": 1893870080.0,
- "69": 1893870080.0,
- "70": 1893870080.0,
- "71": 1893870080.0,
- "72": 1893870080.0,
- "73": 1893870080.0,
- "74": 1893870080.0,
- "75": 1893870080.0,
- "76": 1893870080.0,
- "77": 1893870080.0,
- "78": 1893870080.0,
- "79": 1893870080.0,
- "80": 1893870080.0,
- "81": 1893870080.0,
- "82": 1893870080.0,
- "83": 1893870080.0,
- "84": 1893870080.0,
- "85": 1893870080.0,
- "86": 1893870080.0,
- "87": 1893870080.0,
- "88": 1893870080.0,
- "89": 1893870080.0,
- "90": 1893870080.0,
- "91": 1893870080.0,
- "92": 1893870080.0,
- "93": 1893870080.0,
- "94": 1893870080.0,
- "95": 1893870080.0,
- "96": 1893870080.0,
- "97": 1893870080.0,
- "98": 1893870080.0,
- "99": 1893870080.0,
- "100": 1893870080.0
+ "2": 1895442944.0,
+ "3": 1895442944.0,
+ "4": 1895442944.0,
+ "5": 1895442944.0,
+ "6": 1895442944.0,
+ "7": 1895442944.0,
+ "8": 1895442944.0,
+ "9": 1895442944.0,
+ "10": 1895442944.0,
+ "11": 1895442944.0,
+ "12": 1895442944.0,
+ "13": 1895442944.0,
+ "14": 1895442944.0,
+ "15": 1895442944.0,
+ "16": 1895442944.0,
+ "17": 1895442944.0,
+ "18": 1895442944.0,
+ "19": 1895442944.0,
+ "20": 1895442944.0,
+ "21": 1895442944.0,
+ "22": 1895442944.0,
+ "23": 1895442944.0,
+ "24": 1895442944.0,
+ "25": 1895442944.0,
+ "26": 1895442944.0,
+ "27": 1895442944.0,
+ "28": 1895442944.0,
+ "29": 1895442944.0,
+ "30": 1895442944.0,
+ "31": 1895442944.0,
+ "32": 1895442944.0,
+ "33": 1895442944.0,
+ "34": 1895442944.0,
+ "35": 1895442944.0,
+ "36": 1895442944.0,
+ "37": 1895442944.0,
+ "38": 1895442944.0,
+ "39": 1895442944.0,
+ "40": 1895442944.0,
+ "41": 1895442944.0,
+ "42": 1895442944.0,
+ "43": 1895442944.0,
+ "44": 1895442944.0,
+ "45": 1895442944.0,
+ "46": 1895442944.0,
+ "47": 1895442944.0,
+ "48": 1895442944.0,
+ "49": 1895442944.0,
+ "50": 1895442944.0,
+ "51": 1895836160.0,
+ "52": 1895836160.0,
+ "53": 1895836160.0,
+ "54": 1895836160.0,
+ "55": 1895836160.0,
+ "56": 1895836160.0,
+ "57": 1895836160.0,
+ "58": 1895836160.0,
+ "59": 1895836160.0,
+ "60": 1895836160.0,
+ "61": 1895836160.0,
+ "62": 1895836160.0,
+ "63": 1895836160.0,
+ "64": 1895836160.0,
+ "65": 1895836160.0,
+ "66": 1895836160.0,
+ "67": 1895836160.0,
+ "68": 1895836160.0,
+ "69": 1895836160.0,
+ "70": 1895836160.0,
+ "71": 1895836160.0,
+ "72": 1895836160.0,
+ "73": 1895836160.0,
+ "74": 1895836160.0,
+ "75": 1895836160.0,
+ "76": 1895836160.0,
+ "77": 1895836160.0,
+ "78": 1895836160.0,
+ "79": 1895836160.0,
+ "80": 1895836160.0,
+ "81": 1895836160.0,
+ "82": 1895836160.0,
+ "83": 1895836160.0,
+ "84": 1895836160.0,
+ "85": 1895836160.0,
+ "86": 1895836160.0,
+ "87": 1895836160.0,
+ "88": 1895836160.0,
+ "89": 1895836160.0,
+ "90": 1895836160.0,
+ "91": 1895836160.0,
+ "92": 1895836160.0,
+ "93": 1895836160.0,
+ "94": 1895836160.0,
+ "95": 1895836160.0,
+ "96": 1895836160.0,
+ "97": 1895836160.0,
+ "98": 1895836160.0,
+ "99": 1895836160.0,
+ "100": 1895836160.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 8.61654,
- "2": 0.16646,
- "3": 0.14939,
- "4": 0.12694,
- "5": 0.1251,
- "6": 0.12545,
- "7": 0.12533,
- "8": 0.1271,
- "9": 0.1261,
- "10": 0.12491,
- "11": 0.12876,
- "12": 0.13422,
- "13": 0.13211,
- "14": 0.12395,
- "15": 0.12563,
- "16": 0.12703,
- "17": 0.1243,
- "18": 0.12651,
- "19": 0.12452,
- "20": 0.12538,
- "21": 0.1244,
- "22": 0.12395,
- "23": 0.12379,
- "24": 0.12455,
- "25": 0.12457,
- "26": 0.12444,
- "27": 0.12397,
- "28": 0.125,
- "29": 0.13321,
- "30": 0.13442,
- "31": 0.13329,
- "32": 0.12696,
- "33": 0.12493,
- "34": 0.12398,
- "35": 0.12918,
- "36": 0.13252,
- "37": 0.13148,
- "38": 0.13338,
- "39": 0.13083,
- "40": 0.13113,
- "41": 0.13061,
- "42": 0.1295,
- "43": 0.1305,
- "44": 0.13132,
- "45": 0.13148,
- "46": 0.13113,
- "47": 0.13116,
- "48": 0.12551,
- "49": 0.12779,
- "50": 0.12989,
- "51": 0.1367,
- "52": 0.13188,
- "53": 0.13008,
- "54": 0.13122,
- "55": 0.12979,
- "56": 0.12943,
- "57": 0.13002,
- "58": 0.12923,
- "59": 0.12984,
- "60": 0.13209,
- "61": 0.13094,
- "62": 0.13083,
- "63": 0.12826,
- "64": 0.13104,
- "65": 0.1292,
- "66": 0.12985,
- "67": 0.1295,
- "68": 0.12398,
- "69": 0.12509,
- "70": 0.12208,
- "71": 0.12371,
- "72": 0.12256,
- "73": 0.12266,
- "74": 0.12476,
- "75": 0.12866,
- "76": 0.12272,
- "77": 0.12403,
- "78": 0.12307,
- "79": 0.12209,
- "80": 0.12352,
- "81": 0.12155,
- "82": 0.12329,
- "83": 0.12201,
- "84": 0.12239,
- "85": 0.12414,
- "86": 0.12372,
- "87": 0.12357,
- "88": 0.12705,
- "89": 0.1249,
- "90": 0.12289,
- "91": 0.12523,
- "92": 0.51175,
- "93": 0.12454,
- "94": 0.12634,
- "95": 0.12226,
- "96": 0.12255,
- "97": 0.12357,
- "98": 0.12405,
- "99": 0.12419,
- "100": 0.12384
+ "1": "nan",
+ "2": 6.07458,
+ "3": 0.15644,
+ "4": 0.13031,
+ "5": 0.1289,
+ "6": 0.12927,
+ "7": 0.13043,
+ "8": 0.1306,
+ "9": 0.13116,
+ "10": 0.13131,
+ "11": 0.13126,
+ "12": 0.13025,
+ "13": 0.13161,
+ "14": 0.1307,
+ "15": 0.13106,
+ "16": 0.13023,
+ "17": 0.13016,
+ "18": 0.13043,
+ "19": 0.13006,
+ "20": 0.13036,
+ "21": 0.13047,
+ "22": 0.13377,
+ "23": 0.13791,
+ "24": 0.1363,
+ "25": 0.1319,
+ "26": 0.13277,
+ "27": 0.13029,
+ "28": 0.13138,
+ "29": 0.13156,
+ "30": 0.13044,
+ "31": 0.13042,
+ "32": 0.13038,
+ "33": 0.13266,
+ "34": 0.13086,
+ "35": 0.13102,
+ "36": 0.13173,
+ "37": 0.13119,
+ "38": 0.13113,
+ "39": 0.13159,
+ "40": 0.13125,
+ "41": 0.13097,
+ "42": 0.13678,
+ "43": 0.13175,
+ "44": 0.13188,
+ "45": 0.131,
+ "46": 0.13295,
+ "47": 0.13061,
+ "48": 0.13141,
+ "49": 0.1312,
+ "50": 0.13059,
+ "51": 0.15478,
+ "52": 0.12933,
+ "53": 0.13057,
+ "54": 0.13035,
+ "55": 0.12984,
+ "56": 0.12983,
+ "57": 0.12997,
+ "58": 0.12877,
+ "59": 0.12928,
+ "60": 0.13052,
+ "61": 0.13019,
+ "62": 0.13042,
+ "63": 0.13252,
+ "64": 0.13036,
+ "65": 0.12974,
+ "66": 0.12919,
+ "67": 0.13057,
+ "68": 0.13024,
+ "69": 0.13027,
+ "70": 0.13945,
+ "71": 0.13003,
+ "72": 0.13006,
+ "73": 0.12867,
+ "74": 0.12907,
+ "75": 0.12869,
+ "76": 0.13047,
+ "77": 0.12792,
+ "78": 0.12885,
+ "79": 0.12812,
+ "80": 0.12852,
+ "81": 0.12997,
+ "82": 0.12859,
+ "83": 0.12856,
+ "84": 0.12929,
+ "85": 0.12888,
+ "86": 0.13013,
+ "87": 0.12943,
+ "88": 0.13005,
+ "89": 0.1301,
+ "90": 0.131,
+ "91": 0.12992,
+ "92": 0.13003,
+ "93": 0.12957,
+ "94": 0.13084,
+ "95": 0.13026,
+ "96": 0.13054,
+ "97": 0.13037,
+ "98": 0.13013,
+ "99": 0.1303,
+ "100": 0.13006
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute/model_config.yaml
index 487227e5abd..f600f587ae9 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -50,4 +50,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_no_create_attention_mask_in_dataloader/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_no_create_attention_mask_in_dataloader/model_config.yaml
index ea5523e1d2a..58d9e2efbd2 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_no_create_attention_mask_in_dataloader/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_no_create_attention_mask_in_dataloader/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_no_mmap_bin_files/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_no_mmap_bin_files/model_config.yaml
index afbc17a0301..f1f26115d6b 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_no_mmap_bin_files/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_no_mmap_bin_files/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -50,4 +50,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_reshard_1x4xNone/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_reshard_1x4xNone/model_config.yaml
index bcbfdad6616..a413b7ebb0c 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_reshard_1x4xNone/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_reshard_1x4xNone/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -47,4 +47,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor/golden_values_dev_dgx_gb200.json
index 7dd5b31f34f..2471c5117cc 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor/golden_values_dev_dgx_gb200.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.87163,
+ "1": 10.87162,
"2": 10.87238,
- "3": 10.86215,
- "4": 10.84334,
+ "3": 10.86217,
+ "4": 10.84331,
"5": 10.8781,
- "6": 10.8937,
- "7": 10.87187,
- "8": 10.87789,
- "9": 10.86815,
- "10": 10.83758,
- "11": 10.87595,
- "12": 10.87605,
- "13": 10.89163,
+ "6": 10.89369,
+ "7": 10.87188,
+ "8": 10.87792,
+ "9": 10.86811,
+ "10": 10.83761,
+ "11": 10.87596,
+ "12": 10.87602,
+ "13": 10.89168,
"14": 10.89707,
- "15": 10.83373,
- "16": 10.82462,
- "17": 10.80227,
- "18": 10.82965,
- "19": 10.82299,
- "20": 10.73839,
- "21": 10.70969,
- "22": 10.5649,
- "23": 10.73038,
+ "15": 10.83377,
+ "16": 10.82464,
+ "17": 10.80226,
+ "18": 10.82961,
+ "19": 10.82301,
+ "20": 10.73842,
+ "21": 10.7097,
+ "22": 10.56485,
+ "23": 10.73039,
"24": 10.6062,
- "25": 10.55515,
- "26": 10.62333,
- "27": 10.61393,
+ "25": 10.55511,
+ "26": 10.62334,
+ "27": 10.61392,
"28": 10.57726,
- "29": 10.60204,
- "30": 10.38732,
- "31": 10.12791,
- "32": 10.4758,
- "33": 10.47238,
- "34": 10.22665,
- "35": 10.28584,
- "36": 10.23138,
- "37": 10.35035,
- "38": 10.19674,
+ "29": 10.60206,
+ "30": 10.38733,
+ "31": 10.12792,
+ "32": 10.47586,
+ "33": 10.47237,
+ "34": 10.22667,
+ "35": 10.2859,
+ "36": 10.23137,
+ "37": 10.35036,
+ "38": 10.19669,
"39": 10.40798,
- "40": 10.09496,
- "41": 10.13593,
- "42": 10.21728,
- "43": 9.84575,
- "44": 9.94965,
- "45": 9.83809,
- "46": 9.821,
+ "40": 10.095,
+ "41": 10.1359,
+ "42": 10.21731,
+ "43": 9.84574,
+ "44": 9.94964,
+ "45": 9.83808,
+ "46": 9.82101,
"47": 10.13316,
- "48": 9.85047,
- "49": 9.53,
- "50": 9.90689,
- "51": 9.85498,
- "52": 9.74731,
- "53": 10.06267,
- "54": 9.95301,
- "55": 9.88728,
+ "48": 9.85044,
+ "49": 9.53002,
+ "50": 9.90686,
+ "51": 9.85497,
+ "52": 9.7473,
+ "53": 10.06268,
+ "54": 9.95304,
+ "55": 9.88726,
"56": 9.6211,
"57": 9.47571,
- "58": 9.83152,
- "59": 9.58168,
- "60": 9.49439,
- "61": 9.68902,
- "62": 9.9857,
- "63": 9.37411,
+ "58": 9.83147,
+ "59": 9.5817,
+ "60": 9.49436,
+ "61": 9.68906,
+ "62": 9.98568,
+ "63": 9.3741,
"64": 9.7651,
- "65": 8.94171,
- "66": 9.69872,
- "67": 9.36899,
+ "65": 8.94174,
+ "66": 9.69878,
+ "67": 9.369,
"68": 9.78075,
"69": 9.79729,
- "70": 9.72884,
- "71": 9.62546,
- "72": 9.58193,
- "73": 9.48195,
- "74": 8.92206,
- "75": 9.4096,
- "76": 9.07711,
+ "70": 9.72882,
+ "71": 9.62545,
+ "72": 9.58195,
+ "73": 9.48193,
+ "74": 8.92205,
+ "75": 9.40961,
+ "76": 9.07704,
"77": 10.05905,
- "78": 9.7196,
- "79": 9.37915,
- "80": 9.39953,
- "81": 9.4826,
+ "78": 9.71961,
+ "79": 9.37914,
+ "80": 9.39952,
+ "81": 9.48265,
"82": 9.70045,
"83": 9.31347,
- "84": 9.41605,
- "85": 9.61616,
- "86": 9.07519,
- "87": 9.59811,
- "88": 9.75175,
- "89": 9.60152,
- "90": 9.82639,
- "91": 9.33477,
- "92": 9.3587,
- "93": 9.08591,
- "94": 8.82888,
+ "84": 9.41608,
+ "85": 9.61613,
+ "86": 9.07521,
+ "87": 9.59813,
+ "88": 9.75181,
+ "89": 9.60155,
+ "90": 9.82643,
+ "91": 9.33475,
+ "92": 9.35867,
+ "93": 9.08589,
+ "94": 8.82892,
"95": 9.52816,
- "96": 9.52866,
- "97": 9.30468,
- "98": 9.67128,
- "99": 8.89752,
- "100": 9.40653
+ "96": 9.52865,
+ "97": 9.30465,
+ "98": 9.67127,
+ "99": 8.89754,
+ "100": 9.40655
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1731.0,
- "2": 1804.0,
- "3": 1704.0,
- "4": 1768.0,
- "5": 2006.0,
- "6": 1918.0,
- "7": 1815.0,
- "8": 1654.0,
- "9": 1919.0,
- "10": 1481.0,
- "11": 1876.0,
- "12": 1795.0,
- "13": 1915.0,
- "14": 1830.0,
- "15": 2029.0,
- "16": 1948.0,
- "17": 1838.0,
- "18": 1747.0,
- "19": 1789.0,
- "20": 1771.0,
- "21": 1876.0,
- "22": 1854.0,
- "23": 2069.0,
- "24": 1684.0,
- "25": 1732.0,
- "26": 1803.0,
- "27": 1919.0,
- "28": 2095.0,
- "29": 2041.0,
- "30": 1919.0,
- "31": 1704.0,
- "32": 1869.0,
- "33": 2184.0,
- "34": 1846.0,
- "35": 1923.0,
- "36": 2071.0,
- "37": 2407.0,
- "38": 2209.0,
- "39": 2462.0,
- "40": 2275.0,
- "41": 2369.0,
- "42": 2305.0,
- "43": 2048.0,
- "44": 2171.0,
- "45": 2119.0,
- "46": 2287.0,
- "47": 2499.0,
- "48": 2361.0,
- "49": 2398.0,
- "50": 2321.0,
- "51": 2604.0,
- "52": 2579.0,
- "53": 3020.0,
- "54": 2705.0,
- "55": 2369.0,
- "56": 2752.0,
- "57": 2351.0,
- "58": 2902.0,
- "59": 2786.0,
- "60": 2511.0,
- "61": 2861.0,
- "62": 2715.0,
- "63": 2476.0,
- "64": 2944.0,
- "65": 2791.0,
- "66": 3095.0,
- "67": 2945.0,
- "68": 2853.0,
- "69": 2919.0,
- "70": 3113.0,
- "71": 2898.0,
- "72": 2554.0,
- "73": 3029.0,
- "74": 2044.0,
- "75": 2601.0,
- "76": 2957.0,
- "77": 3204.0,
- "78": 3197.0,
- "79": 3123.0,
- "80": 3255.0,
- "81": 3582.0,
- "82": 3338.0,
- "83": 2799.0,
- "84": 3225.0,
- "85": 3372.0,
- "86": 2818.0,
- "87": 3881.0,
- "88": 3040.0,
- "89": 3335.0,
- "90": 3256.0,
- "91": 2903.0,
- "92": 3202.0,
- "93": 2806.0,
- "94": 3422.0,
- "95": 3348.0,
- "96": 3594.0,
- "97": 3290.0,
- "98": 3746.0,
- "99": 3085.0,
- "100": 3366.0
+ "1": 1635.0,
+ "2": 1768.0,
+ "3": 1703.0,
+ "4": 1810.0,
+ "5": 2003.0,
+ "6": 1892.0,
+ "7": 1931.0,
+ "8": 1674.0,
+ "9": 1847.0,
+ "10": 1444.0,
+ "11": 1908.0,
+ "12": 1837.0,
+ "13": 1949.0,
+ "14": 1763.0,
+ "15": 2001.0,
+ "16": 1932.0,
+ "17": 1778.0,
+ "18": 1660.0,
+ "19": 1682.0,
+ "20": 1759.0,
+ "21": 1946.0,
+ "22": 1897.0,
+ "23": 2008.0,
+ "24": 1724.0,
+ "25": 1718.0,
+ "26": 1829.0,
+ "27": 1924.0,
+ "28": 2025.0,
+ "29": 2092.0,
+ "30": 2054.0,
+ "31": 1734.0,
+ "32": 2014.0,
+ "33": 2138.0,
+ "34": 1910.0,
+ "35": 2052.0,
+ "36": 2015.0,
+ "37": 2410.0,
+ "38": 2166.0,
+ "39": 2371.0,
+ "40": 2250.0,
+ "41": 2474.0,
+ "42": 2322.0,
+ "43": 2052.0,
+ "44": 2160.0,
+ "45": 2143.0,
+ "46": 2297.0,
+ "47": 2569.0,
+ "48": 2370.0,
+ "49": 2448.0,
+ "50": 2361.0,
+ "51": 2511.0,
+ "52": 2531.0,
+ "53": 3056.0,
+ "54": 2732.0,
+ "55": 2357.0,
+ "56": 2840.0,
+ "57": 2362.0,
+ "58": 2824.0,
+ "59": 2835.0,
+ "60": 2513.0,
+ "61": 3049.0,
+ "62": 2701.0,
+ "63": 2568.0,
+ "64": 3048.0,
+ "65": 2684.0,
+ "66": 3050.0,
+ "67": 2851.0,
+ "68": 2854.0,
+ "69": 2948.0,
+ "70": 3211.0,
+ "71": 2926.0,
+ "72": 2524.0,
+ "73": 3033.0,
+ "74": 2043.0,
+ "75": 2593.0,
+ "76": 2905.0,
+ "77": 3297.0,
+ "78": 3228.0,
+ "79": 3232.0,
+ "80": 3299.0,
+ "81": 3568.0,
+ "82": 3366.0,
+ "83": 2687.0,
+ "84": 3268.0,
+ "85": 3336.0,
+ "86": 2690.0,
+ "87": 3826.0,
+ "88": 3033.0,
+ "89": 3481.0,
+ "90": 3125.0,
+ "91": 2909.0,
+ "92": 3151.0,
+ "93": 2860.0,
+ "94": 3427.0,
+ "95": 3386.0,
+ "96": 3575.0,
+ "97": 3297.0,
+ "98": 3706.0,
+ "99": 3178.0,
+ "100": 3420.0
}
},
"mem-allocated-bytes": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 1.71105,
- "3": 0.22879,
- "4": 0.22169,
- "5": 0.21979,
- "6": 0.21933,
- "7": 0.21836,
- "8": 0.22054,
- "9": 0.22096,
- "10": 0.22079,
- "11": 0.22255,
- "12": 0.21905,
- "13": 0.22266,
- "14": 0.22261,
- "15": 0.22192,
- "16": 0.21928,
- "17": 0.22014,
- "18": 0.2213,
- "19": 0.22242,
- "20": 0.22097,
- "21": 0.21936,
- "22": 0.22091,
- "23": 0.22071,
- "24": 0.22306,
- "25": 0.22073,
- "26": 0.22028,
- "27": 0.22198,
- "28": 0.22294,
- "29": 0.22204,
- "30": 0.21896,
- "31": 0.22144,
- "32": 0.22279,
- "33": 0.22428,
- "34": 0.22247,
- "35": 0.22192,
- "36": 0.22242,
- "37": 0.22321,
- "38": 0.22186,
- "39": 0.22242,
- "40": 0.22098,
- "41": 0.22254,
- "42": 0.55234,
- "43": 0.22432,
- "44": 0.22103,
- "45": 0.22202,
- "46": 0.2216,
- "47": 0.22107,
- "48": 0.21878,
- "49": 0.22338,
- "50": 0.22181,
- "51": 0.22588,
- "52": 0.22221,
- "53": 0.22214,
- "54": 0.22059,
- "55": 0.22088,
- "56": 0.22231,
- "57": 0.2231,
- "58": 0.22228,
- "59": 0.22136,
- "60": 0.22087,
- "61": 0.22171,
- "62": 0.22165,
- "63": 0.22149,
- "64": 0.22165,
- "65": 0.22916,
- "66": 0.25667,
- "67": 0.22585,
- "68": 0.2212,
- "69": 0.22322,
- "70": 0.22332,
- "71": 0.22291,
- "72": 0.22074,
- "73": 0.2214,
- "74": 0.22287,
- "75": 0.21929,
- "76": 0.22246,
- "77": 0.22148,
- "78": 0.22442,
- "79": 0.22465,
- "80": 0.22859,
- "81": 0.22464,
- "82": 0.22391,
- "83": 0.22417,
- "84": 0.22202,
- "85": 0.22369,
- "86": 0.22224,
- "87": 0.22245,
- "88": 0.22255,
- "89": 0.22379,
- "90": 0.22356,
- "91": 0.22229,
- "92": 0.22297,
- "93": 0.22525,
- "94": 0.21956,
- "95": 0.22318,
- "96": 0.22361,
- "97": 0.22246,
- "98": 0.22326,
- "99": 0.22121,
- "100": 0.22214
+ "2": 3.47519,
+ "3": 0.23464,
+ "4": 0.22551,
+ "5": 0.22589,
+ "6": 0.22563,
+ "7": 0.22745,
+ "8": 0.22511,
+ "9": 0.22598,
+ "10": 0.2276,
+ "11": 0.22627,
+ "12": 0.2248,
+ "13": 0.22685,
+ "14": 0.22689,
+ "15": 0.22796,
+ "16": 0.22747,
+ "17": 0.22636,
+ "18": 0.2276,
+ "19": 0.22859,
+ "20": 0.22879,
+ "21": 0.22663,
+ "22": 0.22705,
+ "23": 0.22996,
+ "24": 0.22863,
+ "25": 0.22596,
+ "26": 0.22764,
+ "27": 0.2288,
+ "28": 0.22753,
+ "29": 0.23036,
+ "30": 0.2272,
+ "31": 0.22832,
+ "32": 0.23015,
+ "33": 0.22925,
+ "34": 0.2264,
+ "35": 0.22962,
+ "36": 0.228,
+ "37": 0.22671,
+ "38": 0.22839,
+ "39": 0.2276,
+ "40": 0.22772,
+ "41": 0.22842,
+ "42": 0.22878,
+ "43": 0.22806,
+ "44": 0.22909,
+ "45": 0.22908,
+ "46": 0.22891,
+ "47": 0.22584,
+ "48": 0.22899,
+ "49": 0.22909,
+ "50": 0.23058,
+ "51": 0.2353,
+ "52": 0.22737,
+ "53": 0.22819,
+ "54": 0.22792,
+ "55": 0.22942,
+ "56": 0.22934,
+ "57": 0.23061,
+ "58": 0.22975,
+ "59": 0.2291,
+ "60": 0.2292,
+ "61": 0.22916,
+ "62": 0.22792,
+ "63": 0.22698,
+ "64": 0.22881,
+ "65": 0.22606,
+ "66": 0.22804,
+ "67": 0.22786,
+ "68": 0.22928,
+ "69": 0.23139,
+ "70": 0.22847,
+ "71": 0.22889,
+ "72": 0.22839,
+ "73": 0.22809,
+ "74": 0.22741,
+ "75": 0.22906,
+ "76": 0.23698,
+ "77": 0.22921,
+ "78": 0.22663,
+ "79": 0.23187,
+ "80": 0.23018,
+ "81": 0.22995,
+ "82": 0.22897,
+ "83": 0.22956,
+ "84": 0.22742,
+ "85": 0.22817,
+ "86": 0.23042,
+ "87": 0.22896,
+ "88": 0.2286,
+ "89": 0.23053,
+ "90": 0.2294,
+ "91": 0.22555,
+ "92": 0.22838,
+ "93": 0.22865,
+ "94": 0.22866,
+ "95": 0.2295,
+ "96": 0.22867,
+ "97": 0.23017,
+ "98": 0.23013,
+ "99": 0.22868,
+ "100": 0.22733
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor/golden_values_dev_dgx_h100.json
index feb49a01aad..c79082a0ec6 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.84445,
- "2": 10.84755,
- "3": 10.84905,
- "4": 10.844,
- "5": 10.88133,
- "6": 10.88069,
- "7": 10.86435,
- "8": 10.85483,
- "9": 10.85577,
- "10": 10.81851,
- "11": 10.88835,
- "12": 10.86318,
- "13": 10.86739,
- "14": 10.88397,
- "15": 10.82443,
- "16": 10.82905,
- "17": 10.7953,
- "18": 10.81529,
- "19": 10.80121,
- "20": 10.71826,
- "21": 10.69956,
- "22": 10.56756,
- "23": 10.7171,
- "24": 10.60451,
- "25": 10.55018,
- "26": 10.60859,
- "27": 10.62013,
- "28": 10.57541,
- "29": 10.59599,
- "30": 10.38364,
- "31": 10.15409,
- "32": 10.48036,
- "33": 10.47379,
- "34": 10.23693,
- "35": 10.28857,
- "36": 10.24862,
- "37": 10.35357,
- "38": 10.20827,
- "39": 10.41871,
- "40": 10.11266,
- "41": 10.16079,
- "42": 10.23304,
- "43": 9.86146,
- "44": 9.97719,
- "45": 9.8651,
- "46": 9.8486,
- "47": 10.16607,
- "48": 9.87126,
- "49": 9.56738,
- "50": 9.92137,
- "51": 9.86682,
- "52": 9.7694,
- "53": 10.07839,
- "54": 9.96992,
- "55": 9.89678,
- "56": 9.64417,
- "57": 9.49737,
- "58": 9.84853,
- "59": 9.59973,
- "60": 9.5062,
- "61": 9.71028,
- "62": 9.99079,
- "63": 9.38989,
- "64": 9.78616,
- "65": 8.95963,
- "66": 9.70879,
- "67": 9.3791,
- "68": 9.79602,
- "69": 9.80692,
- "70": 9.74781,
- "71": 9.61777,
- "72": 9.59105,
- "73": 9.50417,
- "74": 8.94629,
- "75": 9.42953,
- "76": 9.08443,
- "77": 10.06697,
- "78": 9.73245,
- "79": 9.38132,
- "80": 9.41079,
- "81": 9.48315,
- "82": 9.70491,
- "83": 9.30719,
- "84": 9.42254,
- "85": 9.61799,
- "86": 9.07625,
- "87": 9.59384,
- "88": 9.75414,
- "89": 9.60107,
- "90": 9.8203,
- "91": 9.34086,
- "92": 9.35733,
- "93": 9.07939,
- "94": 8.83611,
- "95": 9.52231,
- "96": 9.53388,
- "97": 9.31636,
- "98": 9.68001,
- "99": 8.89242,
- "100": 9.3998
+ "1": 10.9313,
+ "2": 10.92649,
+ "3": 10.92492,
+ "4": 10.92049,
+ "5": 10.91862,
+ "6": 10.91576,
+ "7": 10.9247,
+ "8": 10.91818,
+ "9": 10.9151,
+ "10": 10.91994,
+ "11": 10.91158,
+ "12": 10.90479,
+ "13": 10.9005,
+ "14": 10.90129,
+ "15": 10.88356,
+ "16": 10.8637,
+ "17": 10.88305,
+ "18": 10.86569,
+ "19": 10.86865,
+ "20": 10.79163,
+ "21": 10.78749,
+ "22": 10.77557,
+ "23": 10.76378,
+ "24": 10.73311,
+ "25": 10.73327,
+ "26": 10.71729,
+ "27": 10.67374,
+ "28": 10.6152,
+ "29": 10.58911,
+ "30": 10.55842,
+ "31": 10.56337,
+ "32": 10.54728,
+ "33": 10.51063,
+ "34": 10.48413,
+ "35": 10.48015,
+ "36": 10.4581,
+ "37": 10.43335,
+ "38": 10.43476,
+ "39": 10.3924,
+ "40": 10.3866,
+ "41": 10.36418,
+ "42": 10.34636,
+ "43": 10.32165,
+ "44": 10.29971,
+ "45": 10.30674,
+ "46": 10.27,
+ "47": 10.25534,
+ "48": 10.20947,
+ "49": 10.21404,
+ "50": 10.2072,
+ "51": 10.20959,
+ "52": 10.16216,
+ "53": 10.16865,
+ "54": 10.13116,
+ "55": 10.10555,
+ "56": 10.13036,
+ "57": 10.11742,
+ "58": 10.12715,
+ "59": 10.07186,
+ "60": 10.09529,
+ "61": 10.04931,
+ "62": 10.01411,
+ "63": 10.08515,
+ "64": 10.03459,
+ "65": 10.00772,
+ "66": 10.04562,
+ "67": 10.02074,
+ "68": 9.98562,
+ "69": 9.99972,
+ "70": 9.98439,
+ "71": 10.00599,
+ "72": 9.9918,
+ "73": 9.98243,
+ "74": 9.96947,
+ "75": 9.92723,
+ "76": 9.96628,
+ "77": 9.97063,
+ "78": 9.91889,
+ "79": 9.91539,
+ "80": 9.93333,
+ "81": 9.9547,
+ "82": 9.89573,
+ "83": 9.86369,
+ "84": 9.80474,
+ "85": 9.78778,
+ "86": 9.89716,
+ "87": 9.90765,
+ "88": 9.88963,
+ "89": 9.83169,
+ "90": 9.82535,
+ "91": 9.83477,
+ "92": 9.82448,
+ "93": 9.7585,
+ "94": 9.83106,
+ "95": 9.81947,
+ "96": 9.80779,
+ "97": 9.75344,
+ "98": 9.77887,
+ "99": 9.82075,
+ "100": 9.71279
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1814.0,
- "2": 1725.0,
- "3": 1782.0,
- "4": 1955.0,
- "5": 1930.0,
- "6": 1875.0,
- "7": 1951.0,
- "8": 1800.0,
- "9": 1914.0,
- "10": 1495.0,
- "11": 1987.0,
- "12": 1811.0,
- "13": 2030.0,
- "14": 1930.0,
- "15": 1948.0,
- "16": 1933.0,
- "17": 1892.0,
- "18": 1781.0,
- "19": 1985.0,
- "20": 1812.0,
- "21": 2115.0,
- "22": 1885.0,
- "23": 2120.0,
- "24": 1814.0,
- "25": 1705.0,
- "26": 1815.0,
- "27": 1870.0,
- "28": 2162.0,
- "29": 2104.0,
- "30": 2061.0,
- "31": 1666.0,
- "32": 2010.0,
- "33": 2157.0,
- "34": 1918.0,
- "35": 2000.0,
- "36": 1966.0,
- "37": 2421.0,
- "38": 2318.0,
- "39": 2488.0,
- "40": 2213.0,
- "41": 2361.0,
- "42": 2330.0,
- "43": 2092.0,
- "44": 2184.0,
- "45": 2237.0,
- "46": 2311.0,
- "47": 2645.0,
- "48": 2374.0,
- "49": 2345.0,
- "50": 2357.0,
- "51": 2627.0,
- "52": 2530.0,
- "53": 2856.0,
- "54": 2776.0,
- "55": 2346.0,
- "56": 2679.0,
- "57": 2410.0,
- "58": 2990.0,
- "59": 2835.0,
- "60": 2502.0,
- "61": 2984.0,
- "62": 2692.0,
- "63": 2463.0,
- "64": 3009.0,
- "65": 2587.0,
- "66": 3126.0,
- "67": 2793.0,
- "68": 2665.0,
- "69": 2776.0,
- "70": 3135.0,
- "71": 3151.0,
- "72": 2424.0,
- "73": 2926.0,
- "74": 1921.0,
- "75": 2347.0,
- "76": 3026.0,
- "77": 3283.0,
- "78": 3224.0,
- "79": 3165.0,
- "80": 3311.0,
- "81": 3792.0,
- "82": 3279.0,
- "83": 2867.0,
- "84": 3381.0,
- "85": 3415.0,
- "86": 2962.0,
- "87": 3822.0,
- "88": 3311.0,
- "89": 3392.0,
- "90": 3184.0,
- "91": 2795.0,
- "92": 3121.0,
- "93": 2731.0,
- "94": 3503.0,
- "95": 3473.0,
- "96": 3465.0,
- "97": 3299.0,
- "98": 3663.0,
- "99": 3394.0,
- "100": 3235.0
+ "1": 1952.0,
+ "2": 1817.0,
+ "3": 1808.0,
+ "4": 1791.0,
+ "5": 1885.0,
+ "6": 1821.0,
+ "7": 2063.0,
+ "8": 1849.0,
+ "9": 1891.0,
+ "10": 1826.0,
+ "11": 1829.0,
+ "12": 1778.0,
+ "13": 1876.0,
+ "14": 1899.0,
+ "15": 1757.0,
+ "16": 1807.0,
+ "17": 1905.0,
+ "18": 1856.0,
+ "19": 1880.0,
+ "20": 1758.0,
+ "21": 2009.0,
+ "22": 1903.0,
+ "23": 1879.0,
+ "24": 1893.0,
+ "25": 1751.0,
+ "26": 1857.0,
+ "27": 1897.0,
+ "28": 1994.0,
+ "29": 2054.0,
+ "30": 1957.0,
+ "31": 2149.0,
+ "32": 2039.0,
+ "33": 2062.0,
+ "34": 2051.0,
+ "35": 2185.0,
+ "36": 2234.0,
+ "37": 2366.0,
+ "38": 2157.0,
+ "39": 2241.0,
+ "40": 2325.0,
+ "41": 2249.0,
+ "42": 2136.0,
+ "43": 2423.0,
+ "44": 2353.0,
+ "45": 2655.0,
+ "46": 2475.0,
+ "47": 2467.0,
+ "48": 2610.0,
+ "49": 2809.0,
+ "50": 2683.0,
+ "51": 2544.0,
+ "52": 2804.0,
+ "53": 2713.0,
+ "54": 2974.0,
+ "55": 2567.0,
+ "56": 2813.0,
+ "57": 2384.0,
+ "58": 3543.0,
+ "59": 2921.0,
+ "60": 3122.0,
+ "61": 2740.0,
+ "62": 3250.0,
+ "63": 3226.0,
+ "64": 3810.0,
+ "65": 2743.0,
+ "66": 3027.0,
+ "67": 3696.0,
+ "68": 3500.0,
+ "69": 3027.0,
+ "70": 3326.0,
+ "71": 3237.0,
+ "72": 3012.0,
+ "73": 3422.0,
+ "74": 3350.0,
+ "75": 3159.0,
+ "76": 3480.0,
+ "77": 3706.0,
+ "78": 3381.0,
+ "79": 3394.0,
+ "80": 3124.0,
+ "81": 3390.0,
+ "82": 3423.0,
+ "83": 3153.0,
+ "84": 3044.0,
+ "85": 2887.0,
+ "86": 3141.0,
+ "87": 3215.0,
+ "88": 3308.0,
+ "89": 3183.0,
+ "90": 4186.0,
+ "91": 3041.0,
+ "92": 3180.0,
+ "93": 3188.0,
+ "94": 3539.0,
+ "95": 3313.0,
+ "96": 3972.0,
+ "97": 3735.0,
+ "98": 3620.0,
+ "99": 3471.0,
+ "100": 3414.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 700103168.0,
- "2": 700103168.0,
- "3": 700103168.0,
- "4": 700103168.0,
- "5": 700103168.0,
- "6": 700103168.0,
- "7": 700103168.0,
- "8": 700103168.0,
- "9": 700103168.0,
- "10": 700103168.0,
- "11": 700103168.0,
- "12": 700103168.0,
- "13": 700103168.0,
- "14": 700103168.0,
- "15": 700103168.0,
- "16": 700103168.0,
- "17": 700103168.0,
- "18": 700103168.0,
- "19": 700103168.0,
- "20": 700103168.0,
- "21": 700103168.0,
- "22": 700103168.0,
- "23": 700103168.0,
- "24": 700103168.0,
- "25": 700103168.0,
- "26": 700103168.0,
- "27": 700103168.0,
- "28": 700103168.0,
- "29": 700103168.0,
- "30": 700103168.0,
- "31": 700103168.0,
- "32": 700103168.0,
- "33": 700103168.0,
- "34": 700103168.0,
- "35": 700103168.0,
- "36": 700103168.0,
- "37": 700103168.0,
- "38": 700103168.0,
- "39": 700103168.0,
- "40": 700103168.0,
- "41": 700103168.0,
- "42": 700103168.0,
- "43": 700103168.0,
- "44": 700103168.0,
- "45": 700103168.0,
- "46": 700103168.0,
- "47": 700103168.0,
- "48": 700103168.0,
- "49": 700103168.0,
- "50": 700103168.0,
- "51": 700103168.0,
- "52": 700103168.0,
- "53": 700103168.0,
- "54": 700103168.0,
- "55": 700103168.0,
- "56": 700103168.0,
- "57": 700103168.0,
- "58": 700103168.0,
- "59": 700103168.0,
- "60": 700103168.0,
- "61": 700103168.0,
- "62": 700103168.0,
- "63": 700103168.0,
- "64": 700103168.0,
- "65": 700103168.0,
- "66": 700103168.0,
- "67": 700103168.0,
- "68": 700103168.0,
- "69": 700103168.0,
- "70": 700103168.0,
- "71": 700103168.0,
- "72": 700103168.0,
- "73": 700103168.0,
- "74": 700103168.0,
- "75": 700103168.0,
- "76": 700103168.0,
- "77": 700103168.0,
- "78": 700103168.0,
- "79": 700103168.0,
- "80": 700103168.0,
- "81": 700103168.0,
- "82": 700103168.0,
- "83": 700103168.0,
- "84": 700103168.0,
- "85": 700103168.0,
- "86": 700103168.0,
- "87": 700103168.0,
- "88": 700103168.0,
- "89": 700103168.0,
- "90": 700103168.0,
- "91": 700103168.0,
- "92": 700103168.0,
- "93": 700103168.0,
- "94": 700103168.0,
- "95": 700103168.0,
- "96": 700103168.0,
- "97": 700103168.0,
- "98": 700103168.0,
- "99": 700103168.0,
- "100": 700103168.0
+ "1": 246999552.0,
+ "2": 246999552.0,
+ "3": 246999552.0,
+ "4": 246999552.0,
+ "5": 246999552.0,
+ "6": 246999552.0,
+ "7": 246999552.0,
+ "8": 246999552.0,
+ "9": 246999552.0,
+ "10": 246999552.0,
+ "11": 246999552.0,
+ "12": 246999552.0,
+ "13": 246999552.0,
+ "14": 246999552.0,
+ "15": 246999552.0,
+ "16": 246999552.0,
+ "17": 246999552.0,
+ "18": 246999552.0,
+ "19": 246999552.0,
+ "20": 246999552.0,
+ "21": 246999552.0,
+ "22": 246999552.0,
+ "23": 246999552.0,
+ "24": 246999552.0,
+ "25": 246999552.0,
+ "26": 246999552.0,
+ "27": 246999552.0,
+ "28": 246999552.0,
+ "29": 246999552.0,
+ "30": 246999552.0,
+ "31": 246999552.0,
+ "32": 246999552.0,
+ "33": 246999552.0,
+ "34": 246999552.0,
+ "35": 246999552.0,
+ "36": 246999552.0,
+ "37": 246999552.0,
+ "38": 246999552.0,
+ "39": 246999552.0,
+ "40": 246999552.0,
+ "41": 246999552.0,
+ "42": 246999552.0,
+ "43": 246999552.0,
+ "44": 246999552.0,
+ "45": 246999552.0,
+ "46": 246999552.0,
+ "47": 246999552.0,
+ "48": 246999552.0,
+ "49": 246999552.0,
+ "50": 246999552.0,
+ "51": 246999552.0,
+ "52": 246999552.0,
+ "53": 246999552.0,
+ "54": 246999552.0,
+ "55": 246999552.0,
+ "56": 246999552.0,
+ "57": 246999552.0,
+ "58": 246999552.0,
+ "59": 246999552.0,
+ "60": 246999552.0,
+ "61": 246999552.0,
+ "62": 246999552.0,
+ "63": 246999552.0,
+ "64": 246999552.0,
+ "65": 246999552.0,
+ "66": 246999552.0,
+ "67": 246999552.0,
+ "68": 246999552.0,
+ "69": 246999552.0,
+ "70": 246999552.0,
+ "71": 246999552.0,
+ "72": 246999552.0,
+ "73": 246999552.0,
+ "74": 246999552.0,
+ "75": 246999552.0,
+ "76": 246999552.0,
+ "77": 246999552.0,
+ "78": 246999552.0,
+ "79": 246999552.0,
+ "80": 246999552.0,
+ "81": 246999552.0,
+ "82": 246999552.0,
+ "83": 246999552.0,
+ "84": 246999552.0,
+ "85": 246999552.0,
+ "86": 246999552.0,
+ "87": 246999552.0,
+ "88": 246999552.0,
+ "89": 246999552.0,
+ "90": 246999552.0,
+ "91": 246999552.0,
+ "92": 246999552.0,
+ "93": 246999552.0,
+ "94": 246999552.0,
+ "95": 246999552.0,
+ "96": 246999552.0,
+ "97": 246999552.0,
+ "98": 246999552.0,
+ "99": 246999552.0,
+ "100": 246999552.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1956312576.0,
- "2": 1956313600.0,
- "3": 1956313600.0,
- "4": 1956313600.0,
- "5": 1956313600.0,
- "6": 1956313600.0,
- "7": 1956313600.0,
- "8": 1956313600.0,
- "9": 1956313600.0,
- "10": 1956313600.0,
- "11": 1956313600.0,
- "12": 1956313600.0,
- "13": 1956313600.0,
- "14": 1956313600.0,
- "15": 1956313600.0,
- "16": 1956313600.0,
- "17": 1956313600.0,
- "18": 1956313600.0,
- "19": 1956313600.0,
- "20": 1956313600.0,
- "21": 1956313600.0,
- "22": 1956313600.0,
- "23": 1956313600.0,
- "24": 1956313600.0,
- "25": 1956313600.0,
- "26": 1956313600.0,
- "27": 1956313600.0,
- "28": 1956313600.0,
- "29": 1956313600.0,
- "30": 1956313600.0,
- "31": 1956313600.0,
- "32": 1956313600.0,
- "33": 1956313600.0,
- "34": 1956313600.0,
- "35": 1956313600.0,
- "36": 1956313600.0,
- "37": 1956313600.0,
- "38": 1956313600.0,
- "39": 1956313600.0,
- "40": 1956313600.0,
- "41": 1956313600.0,
- "42": 1956313600.0,
- "43": 1956313600.0,
- "44": 1956313600.0,
- "45": 1956313600.0,
- "46": 1956313600.0,
- "47": 1956313600.0,
- "48": 1956313600.0,
- "49": 1956313600.0,
- "50": 1956313600.0,
- "51": 1956313600.0,
- "52": 1956313600.0,
- "53": 1956313600.0,
- "54": 1956313600.0,
- "55": 1956313600.0,
- "56": 1956313600.0,
- "57": 1956313600.0,
- "58": 1956313600.0,
- "59": 1956313600.0,
- "60": 1956313600.0,
- "61": 1956313600.0,
- "62": 1956313600.0,
- "63": 1956313600.0,
- "64": 1956313600.0,
- "65": 1956313600.0,
- "66": 1956313600.0,
- "67": 1956313600.0,
- "68": 1956313600.0,
- "69": 1956313600.0,
- "70": 1956313600.0,
- "71": 1956313600.0,
- "72": 1956313600.0,
- "73": 1956313600.0,
- "74": 1956313600.0,
- "75": 1956313600.0,
- "76": 1956313600.0,
- "77": 1956313600.0,
- "78": 1956313600.0,
- "79": 1956313600.0,
- "80": 1956313600.0,
- "81": 1956313600.0,
- "82": 1956313600.0,
- "83": 1956313600.0,
- "84": 1956313600.0,
- "85": 1956313600.0,
- "86": 1956313600.0,
- "87": 1956313600.0,
- "88": 1956313600.0,
- "89": 1956313600.0,
- "90": 1956313600.0,
- "91": 1956313600.0,
- "92": 1956313600.0,
- "93": 1956313600.0,
- "94": 1956313600.0,
- "95": 1956313600.0,
- "96": 1956313600.0,
- "97": 1956313600.0,
- "98": 1956313600.0,
- "99": 1956313600.0,
- "100": 1956313600.0
+ "1": 1503208960.0,
+ "2": 1503209984.0,
+ "3": 1503209984.0,
+ "4": 1503209984.0,
+ "5": 1503209984.0,
+ "6": 1503209984.0,
+ "7": 1503209984.0,
+ "8": 1503209984.0,
+ "9": 1503209984.0,
+ "10": 1503209984.0,
+ "11": 1503209984.0,
+ "12": 1503209984.0,
+ "13": 1503209984.0,
+ "14": 1503209984.0,
+ "15": 1503209984.0,
+ "16": 1503209984.0,
+ "17": 1503209984.0,
+ "18": 1503209984.0,
+ "19": 1503209984.0,
+ "20": 1503209984.0,
+ "21": 1503209984.0,
+ "22": 1503209984.0,
+ "23": 1503209984.0,
+ "24": 1503209984.0,
+ "25": 1503209984.0,
+ "26": 1503209984.0,
+ "27": 1503209984.0,
+ "28": 1503209984.0,
+ "29": 1503209984.0,
+ "30": 1503209984.0,
+ "31": 1503209984.0,
+ "32": 1503209984.0,
+ "33": 1503209984.0,
+ "34": 1503209984.0,
+ "35": 1503209984.0,
+ "36": 1503209984.0,
+ "37": 1503209984.0,
+ "38": 1503209984.0,
+ "39": 1503209984.0,
+ "40": 1503209984.0,
+ "41": 1503209984.0,
+ "42": 1503209984.0,
+ "43": 1503209984.0,
+ "44": 1503209984.0,
+ "45": 1503209984.0,
+ "46": 1503209984.0,
+ "47": 1503209984.0,
+ "48": 1503209984.0,
+ "49": 1503209984.0,
+ "50": 1503209984.0,
+ "51": 1503209984.0,
+ "52": 1503209984.0,
+ "53": 1503209984.0,
+ "54": 1503209984.0,
+ "55": 1503209984.0,
+ "56": 1503209984.0,
+ "57": 1503209984.0,
+ "58": 1503209984.0,
+ "59": 1503209984.0,
+ "60": 1503209984.0,
+ "61": 1503209984.0,
+ "62": 1503209984.0,
+ "63": 1503209984.0,
+ "64": 1503209984.0,
+ "65": 1503209984.0,
+ "66": 1503209984.0,
+ "67": 1503209984.0,
+ "68": 1503209984.0,
+ "69": 1503209984.0,
+ "70": 1503209984.0,
+ "71": 1503209984.0,
+ "72": 1503209984.0,
+ "73": 1503209984.0,
+ "74": 1503209984.0,
+ "75": 1503209984.0,
+ "76": 1503209984.0,
+ "77": 1503209984.0,
+ "78": 1503209984.0,
+ "79": 1503209984.0,
+ "80": 1503209984.0,
+ "81": 1503209984.0,
+ "82": 1503209984.0,
+ "83": 1503209984.0,
+ "84": 1503209984.0,
+ "85": 1503209984.0,
+ "86": 1503209984.0,
+ "87": 1503209984.0,
+ "88": 1503209984.0,
+ "89": 1503209984.0,
+ "90": 1503209984.0,
+ "91": 1503209984.0,
+ "92": 1503209984.0,
+ "93": 1503209984.0,
+ "94": 1503209984.0,
+ "95": 1503209984.0,
+ "96": 1503209984.0,
+ "97": 1503209984.0,
+ "98": 1503209984.0,
+ "99": 1503209984.0,
+ "100": 1503209984.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 4.9999,
- "2": 0.17604,
- "3": 0.16654,
- "4": 0.15324,
- "5": 0.14982,
- "6": 0.15181,
- "7": 0.15028,
- "8": 0.15021,
- "9": 0.14947,
- "10": 0.15037,
- "11": 0.15211,
- "12": 0.15245,
- "13": 0.1517,
- "14": 0.15044,
- "15": 0.15166,
- "16": 0.14955,
- "17": 0.15212,
- "18": 0.15368,
- "19": 0.15062,
- "20": 0.15093,
- "21": 0.1573,
- "22": 0.15817,
- "23": 0.14955,
- "24": 0.14912,
- "25": 0.15491,
- "26": 0.14937,
+ "1": "nan",
+ "2": 3.56324,
+ "3": 0.16988,
+ "4": 0.151,
+ "5": 0.1505,
+ "6": 0.15059,
+ "7": 0.1504,
+ "8": 0.14973,
+ "9": 0.14976,
+ "10": 0.15088,
+ "11": 0.15121,
+ "12": 0.14923,
+ "13": 0.14986,
+ "14": 0.14983,
+ "15": 0.14929,
+ "16": 0.14921,
+ "17": 0.15065,
+ "18": 0.15172,
+ "19": 0.14875,
+ "20": 0.1497,
+ "21": 0.14899,
+ "22": 0.14983,
+ "23": 0.15003,
+ "24": 0.15229,
+ "25": 0.14988,
+ "26": 0.15059,
"27": 0.15155,
- "28": 0.15055,
- "29": 0.14603,
- "30": 0.14602,
- "31": 0.14824,
- "32": 0.14477,
- "33": 0.14671,
- "34": 0.14693,
- "35": 0.14738,
- "36": 0.14504,
- "37": 0.14513,
- "38": 0.14512,
- "39": 0.14473,
- "40": 0.14614,
- "41": 0.14578,
- "42": 0.14684,
- "43": 0.14487,
- "44": 0.14547,
- "45": 0.145,
- "46": 0.14486,
- "47": 0.14751,
- "48": 0.14552,
- "49": 0.14493,
- "50": 0.14395,
- "51": 0.1521,
- "52": 0.14666,
- "53": 0.14801,
- "54": 0.14826,
- "55": 0.14557,
- "56": 0.15142,
- "57": 0.14933,
- "58": 0.14555,
- "59": 0.14614,
- "60": 0.15938,
- "61": 0.16219,
- "62": 0.14894,
- "63": 0.14392,
- "64": 0.14433,
- "65": 0.1452,
- "66": 0.14488,
- "67": 0.14508,
- "68": 0.14493,
- "69": 0.14702,
- "70": 0.14432,
- "71": 0.14412,
- "72": 0.14561,
- "73": 0.15534,
- "74": 0.14715,
- "75": 0.14564,
- "76": 0.146,
- "77": 0.14498,
- "78": 0.14433,
- "79": 0.14454,
- "80": 0.1457,
- "81": 0.14534,
- "82": 0.14499,
- "83": 0.14463,
- "84": 0.1456,
- "85": 0.14456,
- "86": 0.1456,
- "87": 0.14661,
- "88": 0.1469,
- "89": 0.14537,
- "90": 0.14515,
- "91": 0.14627,
- "92": 0.14607,
- "93": 0.14633,
- "94": 0.14863,
- "95": 0.14553,
- "96": 0.14487,
- "97": 0.14462,
- "98": 0.14685,
- "99": 0.14551,
- "100": 0.14614
+ "28": 0.14887,
+ "29": 0.15108,
+ "30": 0.14869,
+ "31": 0.1487,
+ "32": 0.14907,
+ "33": 0.14971,
+ "34": 0.14941,
+ "35": 0.14861,
+ "36": 0.1495,
+ "37": 0.15019,
+ "38": 0.14941,
+ "39": 0.14876,
+ "40": 0.15057,
+ "41": 0.15191,
+ "42": 0.14949,
+ "43": 0.15065,
+ "44": 0.1491,
+ "45": 0.14954,
+ "46": 0.14933,
+ "47": 0.14945,
+ "48": 0.15047,
+ "49": 0.14942,
+ "50": 0.14928,
+ "51": 0.18151,
+ "52": 0.15096,
+ "53": 0.17163,
+ "54": 0.17097,
+ "55": 0.1662,
+ "56": 0.14964,
+ "57": 0.14937,
+ "58": 0.14937,
+ "59": 0.14889,
+ "60": 0.15003,
+ "61": 0.15106,
+ "62": 0.14887,
+ "63": 0.14931,
+ "64": 0.14963,
+ "65": 0.14883,
+ "66": 0.14918,
+ "67": 0.14958,
+ "68": 0.14991,
+ "69": 0.15078,
+ "70": 0.15078,
+ "71": 0.14942,
+ "72": 0.14924,
+ "73": 0.14962,
+ "74": 0.14934,
+ "75": 0.14958,
+ "76": 0.15015,
+ "77": 0.14907,
+ "78": 0.15081,
+ "79": 0.1488,
+ "80": 0.14931,
+ "81": 0.14921,
+ "82": 0.14888,
+ "83": 0.15074,
+ "84": 0.14917,
+ "85": 0.14886,
+ "86": 0.14844,
+ "87": 0.14866,
+ "88": 0.14956,
+ "89": 0.14884,
+ "90": 0.14902,
+ "91": 0.1492,
+ "92": 0.14885,
+ "93": 0.14932,
+ "94": 0.14978,
+ "95": 0.14986,
+ "96": 0.14939,
+ "97": 0.14946,
+ "98": 0.15104,
+ "99": 0.15122,
+ "100": 0.14878
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor/model_config.yaml
index ecc62315f9f..f563037677e 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce/model_config.yaml
index 89c6943100e..d9cb8494444 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_gb200.json
index 85eca8a168b..5662faa717f 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_gb200.json
@@ -6,53 +6,53 @@
"values": {
"1": 10.84059,
"2": 10.85204,
- "3": 10.84133,
- "4": 10.84535,
- "5": 10.85551,
- "6": 10.86422,
- "7": 10.85246,
- "8": 10.84439,
- "9": 10.84792,
- "10": 10.81313,
- "11": 10.8561,
- "12": 10.84243,
- "13": 10.86076,
- "14": 10.8495,
- "15": 10.81525,
- "16": 10.80923,
- "17": 10.78383,
- "18": 10.79178,
- "19": 10.79409,
- "20": 10.70535,
- "21": 10.69778,
- "22": 10.58348,
+ "3": 10.84138,
+ "4": 10.84536,
+ "5": 10.85554,
+ "6": 10.86423,
+ "7": 10.8525,
+ "8": 10.84446,
+ "9": 10.84795,
+ "10": 10.81311,
+ "11": 10.85614,
+ "12": 10.84248,
+ "13": 10.86077,
+ "14": 10.84952,
+ "15": 10.81528,
+ "16": 10.80924,
+ "17": 10.78389,
+ "18": 10.79175,
+ "19": 10.79412,
+ "20": 10.70534,
+ "21": 10.69776,
+ "22": 10.58346,
"23": 10.69235,
- "24": 10.60608,
- "25": 10.56718,
- "26": 10.61425,
- "27": 10.60614,
- "28": 10.55901,
- "29": 10.56486,
- "30": 10.37865,
- "31": 10.16183,
- "32": 10.45519,
- "33": 10.45018,
- "34": 10.23984,
- "35": 10.27323,
- "36": 10.24226,
- "37": 10.34516,
- "38": 10.21732,
- "39": 10.39456,
- "40": 10.09506,
- "41": 10.15057,
- "42": 10.21211,
- "43": 9.87993,
- "44": 9.97831,
- "45": 9.85574,
- "46": 9.83355,
- "47": 10.14081,
- "48": 9.86387,
- "49": 9.55497,
+ "24": 10.6061,
+ "25": 10.5672,
+ "26": 10.61423,
+ "27": 10.60612,
+ "28": 10.55902,
+ "29": 10.56482,
+ "30": 10.37866,
+ "31": 10.16187,
+ "32": 10.4552,
+ "33": 10.45019,
+ "34": 10.23987,
+ "35": 10.27321,
+ "36": 10.24221,
+ "37": 10.34514,
+ "38": 10.21728,
+ "39": 10.39455,
+ "40": 10.09505,
+ "41": 10.15052,
+ "42": 10.21204,
+ "43": 9.87994,
+ "44": 9.97828,
+ "45": 9.85569,
+ "46": 9.83356,
+ "47": 10.1408,
+ "48": 9.86384,
+ "49": 9.55498,
"50": 9.91604
}
},
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1653.0,
- "2": 1752.0,
- "3": 1624.0,
- "4": 1773.0,
- "5": 2051.0,
- "6": 1884.0,
- "7": 1841.0,
- "8": 1684.0,
- "9": 1859.0,
- "10": 1545.0,
- "11": 1863.0,
- "12": 1746.0,
- "13": 2004.0,
- "14": 1896.0,
- "15": 1934.0,
- "16": 2001.0,
- "17": 1933.0,
- "18": 1793.0,
- "19": 1900.0,
- "20": 1792.0,
- "21": 2062.0,
- "22": 1795.0,
- "23": 1997.0,
- "24": 1666.0,
- "25": 1607.0,
- "26": 1745.0,
- "27": 1880.0,
- "28": 1887.0,
- "29": 2023.0,
- "30": 1964.0,
- "31": 1609.0,
- "32": 1793.0,
- "33": 2102.0,
- "34": 1891.0,
- "35": 1869.0,
- "36": 1984.0,
- "37": 2446.0,
- "38": 2088.0,
- "39": 2394.0,
- "40": 2182.0,
- "41": 2110.0,
- "42": 2180.0,
- "43": 1931.0,
- "44": 2082.0,
- "45": 2079.0,
- "46": 2189.0,
- "47": 2510.0,
- "48": 2197.0,
- "49": 2282.0,
- "50": 2160.0
+ "1": 1636.0,
+ "2": 1800.0,
+ "3": 1658.0,
+ "4": 1779.0,
+ "5": 2034.0,
+ "6": 1824.0,
+ "7": 1839.0,
+ "8": 1699.0,
+ "9": 1879.0,
+ "10": 1507.0,
+ "11": 1927.0,
+ "12": 1767.0,
+ "13": 2057.0,
+ "14": 1821.0,
+ "15": 1970.0,
+ "16": 2034.0,
+ "17": 1909.0,
+ "18": 1751.0,
+ "19": 1885.0,
+ "20": 1756.0,
+ "21": 2017.0,
+ "22": 1709.0,
+ "23": 2079.0,
+ "24": 1728.0,
+ "25": 1634.0,
+ "26": 1781.0,
+ "27": 1786.0,
+ "28": 1888.0,
+ "29": 1942.0,
+ "30": 1884.0,
+ "31": 1611.0,
+ "32": 1879.0,
+ "33": 2056.0,
+ "34": 1834.0,
+ "35": 1950.0,
+ "36": 2062.0,
+ "37": 2372.0,
+ "38": 2155.0,
+ "39": 2438.0,
+ "40": 2115.0,
+ "41": 2178.0,
+ "42": 2225.0,
+ "43": 1904.0,
+ "44": 2060.0,
+ "45": 1972.0,
+ "46": 2141.0,
+ "47": 2505.0,
+ "48": 2257.0,
+ "49": 2330.0,
+ "50": 2227.0
}
},
"mem-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1480436736.0,
- "2": 1542892032.0,
- "3": 1542892032.0,
- "4": 1542892032.0,
- "5": 1542892032.0,
- "6": 1542892032.0,
- "7": 1542892032.0,
- "8": 1542892032.0,
- "9": 1542892032.0,
- "10": 1542892032.0,
- "11": 1542892032.0,
- "12": 1542892032.0,
- "13": 1542892032.0,
- "14": 1542892032.0,
- "15": 1542892032.0,
- "16": 1542892032.0,
- "17": 1542892032.0,
- "18": 1542892032.0,
- "19": 1542892032.0,
- "20": 1542892032.0,
- "21": 1542892032.0,
- "22": 1542892032.0,
- "23": 1542892032.0,
- "24": 1542892032.0,
- "25": 1542892032.0,
- "26": 1542892032.0,
- "27": 1542892032.0,
- "28": 1542892032.0,
- "29": 1542892032.0,
- "30": 1542892032.0,
- "31": 1542892032.0,
- "32": 1542892032.0,
- "33": 1542892032.0,
- "34": 1542892032.0,
- "35": 1542892032.0,
- "36": 1542892032.0,
- "37": 1542892032.0,
- "38": 1542892032.0,
- "39": 1542892032.0,
- "40": 1542892032.0,
- "41": 1542892032.0,
- "42": 1542892032.0,
- "43": 1542892032.0,
- "44": 1542892032.0,
- "45": 1542892032.0,
- "46": 1542892032.0,
- "47": 1542892032.0,
- "48": 1542892032.0,
- "49": 1542892032.0,
- "50": 1542892032.0
+ "1": 1476898304.0,
+ "2": 1543416320.0,
+ "3": 1543416320.0,
+ "4": 1543416320.0,
+ "5": 1543416320.0,
+ "6": 1543416320.0,
+ "7": 1543416320.0,
+ "8": 1543416320.0,
+ "9": 1543416320.0,
+ "10": 1543416320.0,
+ "11": 1543416320.0,
+ "12": 1543416320.0,
+ "13": 1543416320.0,
+ "14": 1543416320.0,
+ "15": 1543416320.0,
+ "16": 1543416320.0,
+ "17": 1543416320.0,
+ "18": 1543416320.0,
+ "19": 1543416320.0,
+ "20": 1543416320.0,
+ "21": 1543416320.0,
+ "22": 1543416320.0,
+ "23": 1543416320.0,
+ "24": 1543416320.0,
+ "25": 1543416320.0,
+ "26": 1543416320.0,
+ "27": 1543416320.0,
+ "28": 1543416320.0,
+ "29": 1543416320.0,
+ "30": 1543416320.0,
+ "31": 1543416320.0,
+ "32": 1543416320.0,
+ "33": 1543416320.0,
+ "34": 1543416320.0,
+ "35": 1543416320.0,
+ "36": 1543416320.0,
+ "37": 1543416320.0,
+ "38": 1543416320.0,
+ "39": 1543416320.0,
+ "40": 1543416320.0,
+ "41": 1543416320.0,
+ "42": 1543416320.0,
+ "43": 1543416320.0,
+ "44": 1543416320.0,
+ "45": 1543416320.0,
+ "46": 1543416320.0,
+ "47": 1543416320.0,
+ "48": 1543416320.0,
+ "49": 1543416320.0,
+ "50": 1543416320.0
}
},
"iteration-time": {
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.48029,
- "3": 0.33127,
- "4": 0.31594,
- "5": 0.32143,
- "6": 0.31919,
- "7": 0.31884,
- "8": 0.32129,
- "9": 0.31988,
- "10": 0.32069,
- "11": 0.31907,
- "12": 0.31959,
- "13": 0.32211,
- "14": 0.31964,
- "15": 0.31855,
- "16": 0.32013,
- "17": 0.32004,
- "18": 0.31786,
- "19": 0.31755,
- "20": 0.31944,
- "21": 0.31998,
- "22": 0.32066,
- "23": 0.32079,
- "24": 0.31728,
- "25": 0.31689,
- "26": 0.31831,
- "27": 0.31727,
- "28": 0.31999,
- "29": 0.31997,
- "30": 0.31824,
- "31": 0.31724,
- "32": 0.33433,
- "33": 0.43748,
- "34": 0.63551,
- "35": 0.35878,
- "36": 0.31703,
- "37": 0.31709,
- "38": 0.32151,
- "39": 0.31762,
- "40": 0.3204,
- "41": 0.3741,
- "42": 0.37991,
- "43": 0.3738,
- "44": 0.38277,
- "45": 0.38,
- "46": 0.37409,
- "47": 0.36543,
- "48": 0.37113,
- "49": 0.36281,
- "50": 0.36274
+ "2": 4.53277,
+ "3": 0.36433,
+ "4": 0.35198,
+ "5": 0.35417,
+ "6": 0.35299,
+ "7": 0.35017,
+ "8": 0.34811,
+ "9": 0.35176,
+ "10": 0.35678,
+ "11": 0.35725,
+ "12": 0.35523,
+ "13": 0.35433,
+ "14": 0.35695,
+ "15": 0.35635,
+ "16": 0.35872,
+ "17": 0.35652,
+ "18": 0.35715,
+ "19": 0.35655,
+ "20": 0.35652,
+ "21": 0.35665,
+ "22": 0.35374,
+ "23": 0.35277,
+ "24": 0.35546,
+ "25": 0.35569,
+ "26": 0.35624,
+ "27": 0.35292,
+ "28": 0.34852,
+ "29": 0.35459,
+ "30": 0.35544,
+ "31": 0.35605,
+ "32": 0.3578,
+ "33": 0.35801,
+ "34": 0.35703,
+ "35": 0.35817,
+ "36": 0.35646,
+ "37": 0.35838,
+ "38": 0.36939,
+ "39": 0.36652,
+ "40": 0.36016,
+ "41": 0.35705,
+ "42": 0.35598,
+ "43": 0.35642,
+ "44": 0.35673,
+ "45": 0.36044,
+ "46": 0.35719,
+ "47": 0.35851,
+ "48": 0.35957,
+ "49": 0.35843,
+ "50": 0.35815
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_h100.json
index dcd92db1774..d28fefe2553 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.84269,
- "2": 10.85556,
- "3": 10.84446,
- "4": 10.84222,
- "5": 10.85859,
- "6": 10.86289,
- "7": 10.85166,
- "8": 10.84694,
- "9": 10.85648,
- "10": 10.8187,
- "11": 10.85952,
- "12": 10.8434,
- "13": 10.86329,
- "14": 10.85467,
- "15": 10.80947,
- "16": 10.81639,
- "17": 10.7887,
- "18": 10.79677,
- "19": 10.79127,
- "20": 10.70829,
- "21": 10.69425,
- "22": 10.58587,
- "23": 10.70272,
- "24": 10.60461,
- "25": 10.57071,
- "26": 10.62002,
- "27": 10.61414,
- "28": 10.56371,
- "29": 10.56749,
- "30": 10.39721,
- "31": 10.16567,
- "32": 10.45764,
- "33": 10.45152,
- "34": 10.23938,
- "35": 10.28311,
- "36": 10.24692,
- "37": 10.34247,
- "38": 10.2052,
- "39": 10.39167,
- "40": 10.09728,
- "41": 10.15266,
- "42": 10.21035,
- "43": 9.87733,
- "44": 9.98208,
- "45": 9.86184,
- "46": 9.83605,
- "47": 10.13379,
- "48": 9.87207,
- "49": 9.56144,
- "50": 9.91021
+ "1": 10.89756,
+ "2": 10.8907,
+ "3": 10.88309,
+ "4": 10.88974,
+ "5": 10.88254,
+ "6": 10.88652,
+ "7": 10.88675,
+ "8": 10.88256,
+ "9": 10.88485,
+ "10": 10.88068,
+ "11": 10.87902,
+ "12": 10.87772,
+ "13": 10.87244,
+ "14": 10.86856,
+ "15": 10.84592,
+ "16": 10.83856,
+ "17": 10.83758,
+ "18": 10.82716,
+ "19": 10.83324,
+ "20": 10.76017,
+ "21": 10.74131,
+ "22": 10.72493,
+ "23": 10.71936,
+ "24": 10.70187,
+ "25": 10.69475,
+ "26": 10.67761,
+ "27": 10.65131,
+ "28": 10.57392,
+ "29": 10.54996,
+ "30": 10.52271,
+ "31": 10.52536,
+ "32": 10.5134,
+ "33": 10.46557,
+ "34": 10.43746,
+ "35": 10.43744,
+ "36": 10.42114,
+ "37": 10.38939,
+ "38": 10.38941,
+ "39": 10.35682,
+ "40": 10.34745,
+ "41": 10.32253,
+ "42": 10.30527,
+ "43": 10.28371,
+ "44": 10.26292,
+ "45": 10.26627,
+ "46": 10.23666,
+ "47": 10.22118,
+ "48": 10.17263,
+ "49": 10.18151,
+ "50": 10.1779
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1669.0,
- "2": 1797.0,
- "3": 1769.0,
- "4": 1812.0,
- "5": 1956.0,
- "6": 1892.0,
- "7": 1848.0,
- "8": 1619.0,
- "9": 1899.0,
- "10": 1416.0,
- "11": 1910.0,
- "12": 1734.0,
- "13": 1952.0,
- "14": 1901.0,
- "15": 1958.0,
- "16": 1961.0,
- "17": 1919.0,
- "18": 1881.0,
- "19": 1883.0,
- "20": 1802.0,
- "21": 1931.0,
- "22": 1655.0,
- "23": 1993.0,
- "24": 1633.0,
- "25": 1767.0,
- "26": 1727.0,
- "27": 1709.0,
- "28": 1909.0,
- "29": 2062.0,
- "30": 1901.0,
- "31": 1678.0,
- "32": 1944.0,
- "33": 2164.0,
- "34": 1777.0,
- "35": 1938.0,
- "36": 1876.0,
- "37": 2428.0,
- "38": 2216.0,
- "39": 2329.0,
- "40": 2126.0,
- "41": 2312.0,
- "42": 2207.0,
- "43": 1975.0,
- "44": 2062.0,
- "45": 1927.0,
- "46": 2258.0,
- "47": 2545.0,
- "48": 2291.0,
- "49": 2254.0,
- "50": 2307.0
+ "1": 1841.0,
+ "2": 1782.0,
+ "3": 1741.0,
+ "4": 1707.0,
+ "5": 1875.0,
+ "6": 1802.0,
+ "7": 1934.0,
+ "8": 1730.0,
+ "9": 1768.0,
+ "10": 1790.0,
+ "11": 1708.0,
+ "12": 1779.0,
+ "13": 1664.0,
+ "14": 1862.0,
+ "15": 1746.0,
+ "16": 1808.0,
+ "17": 1792.0,
+ "18": 1785.0,
+ "19": 1677.0,
+ "20": 1696.0,
+ "21": 1880.0,
+ "22": 1739.0,
+ "23": 1816.0,
+ "24": 1871.0,
+ "25": 1752.0,
+ "26": 1792.0,
+ "27": 1867.0,
+ "28": 1841.0,
+ "29": 1956.0,
+ "30": 1785.0,
+ "31": 1940.0,
+ "32": 1984.0,
+ "33": 1990.0,
+ "34": 2025.0,
+ "35": 2083.0,
+ "36": 2042.0,
+ "37": 2329.0,
+ "38": 2156.0,
+ "39": 2296.0,
+ "40": 2364.0,
+ "41": 2347.0,
+ "42": 2091.0,
+ "43": 2441.0,
+ "44": 2216.0,
+ "45": 2657.0,
+ "46": 2380.0,
+ "47": 2496.0,
+ "48": 2776.0,
+ "49": 2988.0,
+ "50": 2644.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 299204096.0,
- "2": 299204096.0,
- "3": 299204096.0,
- "4": 299204096.0,
- "5": 299204096.0,
- "6": 299204096.0,
- "7": 299204096.0,
- "8": 299204096.0,
- "9": 299204096.0,
- "10": 299204096.0,
- "11": 299204096.0,
- "12": 299204096.0,
- "13": 299204096.0,
- "14": 299204096.0,
- "15": 299204096.0,
- "16": 299204096.0,
- "17": 299204096.0,
- "18": 299204096.0,
- "19": 299204096.0,
- "20": 299204096.0,
- "21": 299204096.0,
- "22": 299204096.0,
- "23": 299204096.0,
- "24": 299204096.0,
- "25": 299204096.0,
- "26": 299204096.0,
- "27": 299204096.0,
- "28": 299204096.0,
- "29": 299204096.0,
- "30": 299204096.0,
- "31": 299204096.0,
- "32": 299204096.0,
- "33": 299204096.0,
- "34": 299204096.0,
- "35": 299204096.0,
- "36": 299204096.0,
- "37": 299204096.0,
- "38": 299204096.0,
- "39": 299204096.0,
- "40": 299204096.0,
- "41": 299204096.0,
- "42": 299204096.0,
- "43": 299204096.0,
- "44": 299204096.0,
- "45": 299204096.0,
- "46": 299204096.0,
- "47": 299204096.0,
- "48": 299204096.0,
- "49": 299204096.0,
- "50": 299204096.0
+ "1": 299728384.0,
+ "2": 299728384.0,
+ "3": 299728384.0,
+ "4": 299728384.0,
+ "5": 299728384.0,
+ "6": 299728384.0,
+ "7": 299728384.0,
+ "8": 299728384.0,
+ "9": 299728384.0,
+ "10": 299728384.0,
+ "11": 299728384.0,
+ "12": 299728384.0,
+ "13": 299728384.0,
+ "14": 299728384.0,
+ "15": 299728384.0,
+ "16": 299728384.0,
+ "17": 299728384.0,
+ "18": 299728384.0,
+ "19": 299728384.0,
+ "20": 299728384.0,
+ "21": 299728384.0,
+ "22": 299728384.0,
+ "23": 299728384.0,
+ "24": 299728384.0,
+ "25": 299728384.0,
+ "26": 299728384.0,
+ "27": 299728384.0,
+ "28": 299728384.0,
+ "29": 299728384.0,
+ "30": 299728384.0,
+ "31": 299728384.0,
+ "32": 299728384.0,
+ "33": 299728384.0,
+ "34": 299728384.0,
+ "35": 299728384.0,
+ "36": 299728384.0,
+ "37": 299728384.0,
+ "38": 299728384.0,
+ "39": 299728384.0,
+ "40": 299728384.0,
+ "41": 299728384.0,
+ "42": 299728384.0,
+ "43": 299728384.0,
+ "44": 299728384.0,
+ "45": 299728384.0,
+ "46": 299728384.0,
+ "47": 299728384.0,
+ "48": 299728384.0,
+ "49": 299728384.0,
+ "50": 299728384.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1478995456.0,
- "2": 1545382400.0,
- "3": 1545382400.0,
- "4": 1545382400.0,
- "5": 1545382400.0,
- "6": 1545382400.0,
- "7": 1545382400.0,
- "8": 1545382400.0,
- "9": 1545382400.0,
- "10": 1545382400.0,
- "11": 1545382400.0,
- "12": 1545382400.0,
- "13": 1545382400.0,
- "14": 1545382400.0,
- "15": 1545382400.0,
- "16": 1545382400.0,
- "17": 1545382400.0,
- "18": 1545382400.0,
- "19": 1545382400.0,
- "20": 1545382400.0,
- "21": 1545382400.0,
- "22": 1545382400.0,
- "23": 1545382400.0,
- "24": 1545382400.0,
- "25": 1545382400.0,
- "26": 1545382400.0,
- "27": 1545382400.0,
- "28": 1545382400.0,
- "29": 1545382400.0,
- "30": 1545382400.0,
- "31": 1545382400.0,
- "32": 1545382400.0,
- "33": 1545382400.0,
- "34": 1545382400.0,
- "35": 1545382400.0,
- "36": 1545382400.0,
- "37": 1545382400.0,
- "38": 1545382400.0,
- "39": 1545382400.0,
- "40": 1545382400.0,
- "41": 1545382400.0,
- "42": 1545382400.0,
- "43": 1545382400.0,
- "44": 1545382400.0,
- "45": 1545382400.0,
- "46": 1545382400.0,
- "47": 1545382400.0,
- "48": 1545382400.0,
- "49": 1545382400.0,
- "50": 1545382400.0
+ "1": 1478339584.0,
+ "2": 1543416320.0,
+ "3": 1543416320.0,
+ "4": 1543416320.0,
+ "5": 1543416320.0,
+ "6": 1543416320.0,
+ "7": 1543416320.0,
+ "8": 1543416320.0,
+ "9": 1543416320.0,
+ "10": 1543416320.0,
+ "11": 1543416320.0,
+ "12": 1543416320.0,
+ "13": 1543416320.0,
+ "14": 1543416320.0,
+ "15": 1543416320.0,
+ "16": 1543416320.0,
+ "17": 1543416320.0,
+ "18": 1543416320.0,
+ "19": 1543416320.0,
+ "20": 1543416320.0,
+ "21": 1543416320.0,
+ "22": 1543416320.0,
+ "23": 1543416320.0,
+ "24": 1543416320.0,
+ "25": 1543416320.0,
+ "26": 1543416320.0,
+ "27": 1543416320.0,
+ "28": 1543416320.0,
+ "29": 1543416320.0,
+ "30": 1543416320.0,
+ "31": 1543416320.0,
+ "32": 1543416320.0,
+ "33": 1543416320.0,
+ "34": 1543416320.0,
+ "35": 1543416320.0,
+ "36": 1543416320.0,
+ "37": 1543416320.0,
+ "38": 1543416320.0,
+ "39": 1543416320.0,
+ "40": 1543416320.0,
+ "41": 1543416320.0,
+ "42": 1543416320.0,
+ "43": 1543416320.0,
+ "44": 1543416320.0,
+ "45": 1543416320.0,
+ "46": 1543416320.0,
+ "47": 1543416320.0,
+ "48": 1543416320.0,
+ "49": 1543416320.0,
+ "50": 1543416320.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 9.29646,
- "2": 0.25495,
- "3": 0.23221,
- "4": 0.21344,
- "5": 0.21407,
- "6": 0.2135,
- "7": 0.2133,
- "8": 0.2143,
- "9": 0.2448,
- "10": 0.21516,
- "11": 0.21366,
- "12": 0.21308,
- "13": 0.21405,
- "14": 0.21663,
- "15": 0.21321,
- "16": 0.21331,
- "17": 0.21649,
- "18": 0.21423,
- "19": 0.21617,
- "20": 0.21504,
- "21": 0.21521,
- "22": 0.21474,
- "23": 0.21516,
- "24": 0.21334,
- "25": 0.21673,
- "26": 0.2145,
- "27": 0.21534,
- "28": 0.21454,
- "29": 0.21458,
- "30": 0.21608,
- "31": 0.2147,
- "32": 0.21508,
- "33": 0.21429,
- "34": 0.21502,
- "35": 0.21469,
- "36": 0.21553,
- "37": 0.21385,
- "38": 0.21644,
- "39": 0.2164,
- "40": 0.21622,
- "41": 0.21355,
- "42": 0.21641,
- "43": 0.21488,
- "44": 0.21246,
- "45": 0.58026,
- "46": 0.2168,
- "47": 0.21774,
- "48": 0.21503,
- "49": 0.21695,
- "50": 0.21799
+ "1": "nan",
+ "2": 4.76898,
+ "3": 0.23934,
+ "4": 0.2191,
+ "5": 0.22015,
+ "6": 0.21913,
+ "7": 0.2188,
+ "8": 0.22012,
+ "9": 0.21947,
+ "10": 0.21894,
+ "11": 0.2183,
+ "12": 0.2186,
+ "13": 0.21805,
+ "14": 0.21858,
+ "15": 0.21845,
+ "16": 0.22043,
+ "17": 0.21927,
+ "18": 0.22394,
+ "19": 0.22229,
+ "20": 0.22468,
+ "21": 0.22392,
+ "22": 0.22242,
+ "23": 0.22012,
+ "24": 0.22225,
+ "25": 0.22003,
+ "26": 0.22266,
+ "27": 0.22105,
+ "28": 0.22104,
+ "29": 0.22075,
+ "30": 0.22138,
+ "31": 0.22175,
+ "32": 0.22088,
+ "33": 0.22244,
+ "34": 0.22093,
+ "35": 0.22045,
+ "36": 0.22145,
+ "37": 0.22131,
+ "38": 0.22115,
+ "39": 0.22173,
+ "40": 0.2204,
+ "41": 0.22134,
+ "42": 0.22517,
+ "43": 0.22707,
+ "44": 0.22068,
+ "45": 0.22325,
+ "46": 0.22253,
+ "47": 0.22158,
+ "48": 0.22542,
+ "49": 0.22258,
+ "50": 0.2217
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
index 944dfb0b489..11a72d75951 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_qk_layernorm_test_mode/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_qk_layernorm_test_mode/model_config.yaml
index 18a7195b436..9790a7f74ce 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_qk_layernorm_test_mode/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_qk_layernorm_test_mode/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -52,4 +52,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/golden_values_dev_dgx_h100.json
index 76ec80299fc..7a6bbdbf7b3 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.84163,
- "2": 10.85598,
- "3": 10.84413,
- "4": 10.84124,
- "5": 10.85872,
- "6": 10.86316,
- "7": 10.85184,
- "8": 10.84645,
- "9": 10.85647,
- "10": 10.81849,
- "11": 10.85923,
- "12": 10.84285,
- "13": 10.86432,
- "14": 10.85423,
- "15": 10.81015,
- "16": 10.81588,
- "17": 10.78949,
- "18": 10.79683,
- "19": 10.79073,
- "20": 10.70819,
- "21": 10.69322,
- "22": 10.58504,
- "23": 10.70217,
- "24": 10.60546,
- "25": 10.57102,
- "26": 10.61967,
- "27": 10.61501,
- "28": 10.56369,
- "29": 10.56725,
- "30": 10.39695,
- "31": 10.16591,
- "32": 10.4573,
- "33": 10.45199,
- "34": 10.2392,
- "35": 10.28351,
- "36": 10.24677,
- "37": 10.3427,
- "38": 10.20546,
- "39": 10.39187,
- "40": 10.09767,
- "41": 10.1526,
- "42": 10.21051,
- "43": 9.87726,
- "44": 9.98291,
- "45": 9.86165,
- "46": 9.83587,
- "47": 10.13369,
- "48": 9.87212,
- "49": 9.56121,
- "50": 9.91045,
- "51": 9.85839,
- "52": 9.7506,
- "53": 10.05817,
- "54": 9.96076,
- "55": 9.88738,
- "56": 9.6344,
- "57": 9.4967,
- "58": 9.83343,
- "59": 9.59391,
- "60": 9.51376,
- "61": 9.69928,
- "62": 9.98089,
- "63": 9.39065,
- "64": 9.77599,
- "65": 8.9571,
- "66": 9.70054,
- "67": 9.37,
- "68": 9.78529,
- "69": 9.78966,
- "70": 9.74676,
- "71": 9.61906,
- "72": 9.58963,
- "73": 9.49629,
- "74": 8.94963,
- "75": 9.42381,
- "76": 9.07799,
- "77": 10.07105,
- "78": 9.72632,
- "79": 9.37966,
- "80": 9.40721,
- "81": 9.48238,
- "82": 9.70152,
- "83": 9.30657,
- "84": 9.41464,
- "85": 9.61784,
- "86": 9.08212,
- "87": 9.59511,
- "88": 9.75008,
- "89": 9.60356,
- "90": 9.82256,
- "91": 9.33721,
- "92": 9.35861,
- "93": 9.07956,
- "94": 8.83268,
- "95": 9.51351,
- "96": 9.52947,
- "97": 9.31813,
- "98": 9.67451,
- "99": 8.88607,
- "100": 9.40106
+ "1": 10.89723,
+ "2": 10.89039,
+ "3": 10.88307,
+ "4": 10.88936,
+ "5": 10.88232,
+ "6": 10.88662,
+ "7": 10.88761,
+ "8": 10.88217,
+ "9": 10.8849,
+ "10": 10.88111,
+ "11": 10.87928,
+ "12": 10.87753,
+ "13": 10.87277,
+ "14": 10.86825,
+ "15": 10.84625,
+ "16": 10.83788,
+ "17": 10.83727,
+ "18": 10.8269,
+ "19": 10.83394,
+ "20": 10.76068,
+ "21": 10.74169,
+ "22": 10.7253,
+ "23": 10.71963,
+ "24": 10.70186,
+ "25": 10.69466,
+ "26": 10.67749,
+ "27": 10.65204,
+ "28": 10.5744,
+ "29": 10.55037,
+ "30": 10.52252,
+ "31": 10.52588,
+ "32": 10.51364,
+ "33": 10.46604,
+ "34": 10.43791,
+ "35": 10.43778,
+ "36": 10.42157,
+ "37": 10.38929,
+ "38": 10.3894,
+ "39": 10.35675,
+ "40": 10.34752,
+ "41": 10.32295,
+ "42": 10.30553,
+ "43": 10.28355,
+ "44": 10.26326,
+ "45": 10.26616,
+ "46": 10.23683,
+ "47": 10.22148,
+ "48": 10.17275,
+ "49": 10.18154,
+ "50": 10.17793,
+ "51": 10.18019,
+ "52": 10.13373,
+ "53": 10.1315,
+ "54": 10.10642,
+ "55": 10.08299,
+ "56": 10.10639,
+ "57": 10.10204,
+ "58": 10.10815,
+ "59": 10.04453,
+ "60": 10.07109,
+ "61": 10.0256,
+ "62": 9.99846,
+ "63": 10.06819,
+ "64": 10.01962,
+ "65": 9.98075,
+ "66": 10.02428,
+ "67": 10.00273,
+ "68": 9.96779,
+ "69": 9.98681,
+ "70": 9.9667,
+ "71": 9.99226,
+ "72": 9.97695,
+ "73": 9.96099,
+ "74": 9.95635,
+ "75": 9.92491,
+ "76": 9.95107,
+ "77": 9.94714,
+ "78": 9.90073,
+ "79": 9.90294,
+ "80": 9.91743,
+ "81": 9.94303,
+ "82": 9.8817,
+ "83": 9.84051,
+ "84": 9.78041,
+ "85": 9.77117,
+ "86": 9.87348,
+ "87": 9.90186,
+ "88": 9.87565,
+ "89": 9.82672,
+ "90": 9.81116,
+ "91": 9.8207,
+ "92": 9.81383,
+ "93": 9.74934,
+ "94": 9.81784,
+ "95": 9.8082,
+ "96": 9.79459,
+ "97": 9.74389,
+ "98": 9.76749,
+ "99": 9.81863,
+ "100": 9.705
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1736.0,
- "2": 1692.0,
- "3": 1695.0,
- "4": 1761.0,
- "5": 1955.0,
- "6": 1791.0,
- "7": 1943.0,
- "8": 1681.0,
- "9": 1884.0,
- "10": 1441.0,
- "11": 1942.0,
- "12": 1786.0,
- "13": 1940.0,
- "14": 1862.0,
- "15": 1907.0,
- "16": 1947.0,
- "17": 1827.0,
- "18": 1907.0,
- "19": 1818.0,
- "20": 1700.0,
- "21": 1911.0,
- "22": 1720.0,
- "23": 1938.0,
- "24": 1707.0,
- "25": 1686.0,
- "26": 1792.0,
- "27": 1891.0,
- "28": 1976.0,
- "29": 1958.0,
- "30": 1941.0,
- "31": 1622.0,
- "32": 1970.0,
- "33": 2129.0,
- "34": 1830.0,
- "35": 1907.0,
- "36": 1892.0,
- "37": 2395.0,
- "38": 2161.0,
- "39": 2493.0,
- "40": 2224.0,
- "41": 2201.0,
- "42": 2175.0,
- "43": 1920.0,
- "44": 1955.0,
- "45": 1956.0,
- "46": 2166.0,
- "47": 2517.0,
- "48": 2272.0,
- "49": 2211.0,
- "50": 2232.0,
- "51": 2621.0,
- "52": 2597.0,
- "53": 2926.0,
- "54": 2633.0,
- "55": 2206.0,
- "56": 2627.0,
- "57": 2328.0,
- "58": 2886.0,
- "59": 2639.0,
- "60": 2157.0,
- "61": 2736.0,
- "62": 2544.0,
- "63": 2332.0,
- "64": 2948.0,
- "65": 2630.0,
- "66": 2931.0,
- "67": 2717.0,
- "68": 2643.0,
- "69": 2955.0,
- "70": 3040.0,
- "71": 2882.0,
- "72": 2390.0,
- "73": 2812.0,
- "74": 1844.0,
- "75": 2461.0,
- "76": 3067.0,
- "77": 3152.0,
- "78": 3018.0,
- "79": 3008.0,
- "80": 3104.0,
- "81": 3589.0,
- "82": 3218.0,
- "83": 2748.0,
- "84": 3217.0,
- "85": 3167.0,
- "86": 2876.0,
- "87": 3604.0,
- "88": 3017.0,
- "89": 3249.0,
- "90": 3069.0,
- "91": 2865.0,
- "92": 3074.0,
- "93": 2680.0,
- "94": 3392.0,
- "95": 3206.0,
- "96": 3401.0,
- "97": 3107.0,
- "98": 3624.0,
- "99": 3007.0,
- "100": 3111.0
+ "1": 1842.0,
+ "2": 1747.0,
+ "3": 1798.0,
+ "4": 1756.0,
+ "5": 1843.0,
+ "6": 1866.0,
+ "7": 1878.0,
+ "8": 1740.0,
+ "9": 1664.0,
+ "10": 1694.0,
+ "11": 1777.0,
+ "12": 1763.0,
+ "13": 1794.0,
+ "14": 1961.0,
+ "15": 1708.0,
+ "16": 1873.0,
+ "17": 1828.0,
+ "18": 1807.0,
+ "19": 1703.0,
+ "20": 1809.0,
+ "21": 1823.0,
+ "22": 1798.0,
+ "23": 1823.0,
+ "24": 1882.0,
+ "25": 1798.0,
+ "26": 1840.0,
+ "27": 1876.0,
+ "28": 1776.0,
+ "29": 2037.0,
+ "30": 1728.0,
+ "31": 1990.0,
+ "32": 1897.0,
+ "33": 1977.0,
+ "34": 2036.0,
+ "35": 2065.0,
+ "36": 2018.0,
+ "37": 2375.0,
+ "38": 2136.0,
+ "39": 2241.0,
+ "40": 2382.0,
+ "41": 2342.0,
+ "42": 2013.0,
+ "43": 2345.0,
+ "44": 2183.0,
+ "45": 2673.0,
+ "46": 2457.0,
+ "47": 2520.0,
+ "48": 2607.0,
+ "49": 2899.0,
+ "50": 2592.0,
+ "51": 2637.0,
+ "52": 2871.0,
+ "53": 2636.0,
+ "54": 2970.0,
+ "55": 2582.0,
+ "56": 2823.0,
+ "57": 2225.0,
+ "58": 3660.0,
+ "59": 2965.0,
+ "60": 2864.0,
+ "61": 2677.0,
+ "62": 3102.0,
+ "63": 3353.0,
+ "64": 3833.0,
+ "65": 2567.0,
+ "66": 3265.0,
+ "67": 3776.0,
+ "68": 3535.0,
+ "69": 3118.0,
+ "70": 3221.0,
+ "71": 3097.0,
+ "72": 3003.0,
+ "73": 3379.0,
+ "74": 3261.0,
+ "75": 3172.0,
+ "76": 3260.0,
+ "77": 3753.0,
+ "78": 3266.0,
+ "79": 3368.0,
+ "80": 3127.0,
+ "81": 3608.0,
+ "82": 2987.0,
+ "83": 3167.0,
+ "84": 2790.0,
+ "85": 2700.0,
+ "86": 3394.0,
+ "87": 2936.0,
+ "88": 2982.0,
+ "89": 3011.0,
+ "90": 3939.0,
+ "91": 3120.0,
+ "92": 2907.0,
+ "93": 3188.0,
+ "94": 3297.0,
+ "95": 3228.0,
+ "96": 3455.0,
+ "97": 3575.0,
+ "98": 3393.0,
+ "99": 3026.0,
+ "100": 3072.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 299204096.0,
- "2": 299204096.0,
- "3": 299204096.0,
- "4": 299204096.0,
- "5": 299204096.0,
- "6": 299204096.0,
- "7": 299204096.0,
- "8": 299204096.0,
- "9": 299204096.0,
- "10": 299204096.0,
- "11": 299204096.0,
- "12": 299204096.0,
- "13": 299204096.0,
- "14": 299204096.0,
- "15": 299204096.0,
- "16": 299204096.0,
- "17": 299204096.0,
- "18": 299204096.0,
- "19": 299204096.0,
- "20": 299204096.0,
- "21": 299204096.0,
- "22": 299204096.0,
- "23": 299204096.0,
- "24": 299204096.0,
- "25": 299204096.0,
- "26": 299204096.0,
- "27": 299204096.0,
- "28": 299204096.0,
- "29": 299204096.0,
- "30": 299204096.0,
- "31": 299204096.0,
- "32": 299204096.0,
- "33": 299204096.0,
- "34": 299204096.0,
- "35": 299204096.0,
- "36": 299204096.0,
- "37": 299204096.0,
- "38": 299204096.0,
- "39": 299204096.0,
- "40": 299204096.0,
- "41": 299204096.0,
- "42": 299204096.0,
- "43": 299204096.0,
- "44": 299204096.0,
- "45": 299204096.0,
- "46": 299204096.0,
- "47": 299204096.0,
- "48": 299204096.0,
- "49": 299204096.0,
- "50": 299204096.0,
- "51": 299204096.0,
- "52": 299204096.0,
- "53": 299204096.0,
- "54": 299204096.0,
- "55": 299204096.0,
- "56": 299204096.0,
- "57": 299204096.0,
- "58": 299204096.0,
- "59": 299204096.0,
- "60": 299204096.0,
- "61": 299204096.0,
- "62": 299204096.0,
- "63": 299204096.0,
- "64": 299204096.0,
- "65": 299204096.0,
- "66": 299204096.0,
- "67": 299204096.0,
- "68": 299204096.0,
- "69": 299204096.0,
- "70": 299204096.0,
- "71": 299204096.0,
- "72": 299204096.0,
- "73": 299204096.0,
- "74": 299204096.0,
- "75": 299204096.0,
- "76": 299204096.0,
- "77": 299204096.0,
- "78": 299204096.0,
- "79": 299204096.0,
- "80": 299204096.0,
- "81": 299204096.0,
- "82": 299204096.0,
- "83": 299204096.0,
- "84": 299204096.0,
- "85": 299204096.0,
- "86": 299204096.0,
- "87": 299204096.0,
- "88": 299204096.0,
- "89": 299204096.0,
- "90": 299204096.0,
- "91": 299204096.0,
- "92": 299204096.0,
- "93": 299204096.0,
- "94": 299204096.0,
- "95": 299204096.0,
- "96": 299204096.0,
- "97": 299204096.0,
- "98": 299204096.0,
- "99": 299204096.0,
- "100": 299204096.0
+ "1": 299728384.0,
+ "2": 299728384.0,
+ "3": 299728384.0,
+ "4": 299728384.0,
+ "5": 299728384.0,
+ "6": 299728384.0,
+ "7": 299728384.0,
+ "8": 299728384.0,
+ "9": 299728384.0,
+ "10": 299728384.0,
+ "11": 299728384.0,
+ "12": 299728384.0,
+ "13": 299728384.0,
+ "14": 299728384.0,
+ "15": 299728384.0,
+ "16": 299728384.0,
+ "17": 299728384.0,
+ "18": 299728384.0,
+ "19": 299728384.0,
+ "20": 299728384.0,
+ "21": 299728384.0,
+ "22": 299728384.0,
+ "23": 299728384.0,
+ "24": 299728384.0,
+ "25": 299728384.0,
+ "26": 299728384.0,
+ "27": 299728384.0,
+ "28": 299728384.0,
+ "29": 299728384.0,
+ "30": 299728384.0,
+ "31": 299728384.0,
+ "32": 299728384.0,
+ "33": 299728384.0,
+ "34": 299728384.0,
+ "35": 299728384.0,
+ "36": 299728384.0,
+ "37": 299728384.0,
+ "38": 299728384.0,
+ "39": 299728384.0,
+ "40": 299728384.0,
+ "41": 299728384.0,
+ "42": 299728384.0,
+ "43": 299728384.0,
+ "44": 299728384.0,
+ "45": 299728384.0,
+ "46": 299728384.0,
+ "47": 299728384.0,
+ "48": 299728384.0,
+ "49": 299728384.0,
+ "50": 299728384.0,
+ "51": 299728384.0,
+ "52": 299728384.0,
+ "53": 299728384.0,
+ "54": 299728384.0,
+ "55": 299728384.0,
+ "56": 299728384.0,
+ "57": 299728384.0,
+ "58": 299728384.0,
+ "59": 299728384.0,
+ "60": 299728384.0,
+ "61": 299728384.0,
+ "62": 299728384.0,
+ "63": 299728384.0,
+ "64": 299728384.0,
+ "65": 299728384.0,
+ "66": 299728384.0,
+ "67": 299728384.0,
+ "68": 299728384.0,
+ "69": 299728384.0,
+ "70": 299728384.0,
+ "71": 299728384.0,
+ "72": 299728384.0,
+ "73": 299728384.0,
+ "74": 299728384.0,
+ "75": 299728384.0,
+ "76": 299728384.0,
+ "77": 299728384.0,
+ "78": 299728384.0,
+ "79": 299728384.0,
+ "80": 299728384.0,
+ "81": 299728384.0,
+ "82": 299728384.0,
+ "83": 299728384.0,
+ "84": 299728384.0,
+ "85": 299728384.0,
+ "86": 299728384.0,
+ "87": 299728384.0,
+ "88": 299728384.0,
+ "89": 299728384.0,
+ "90": 299728384.0,
+ "91": 299728384.0,
+ "92": 299728384.0,
+ "93": 299728384.0,
+ "94": 299728384.0,
+ "95": 299728384.0,
+ "96": 299728384.0,
+ "97": 299728384.0,
+ "98": 299728384.0,
+ "99": 299728384.0,
+ "100": 299728384.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 977519616.0,
- "2": 1042465280.0,
- "3": 1042465280.0,
- "4": 1042465280.0,
- "5": 1042465280.0,
- "6": 1042465280.0,
- "7": 1042465280.0,
- "8": 1042465280.0,
- "9": 1042465280.0,
- "10": 1042465280.0,
- "11": 1042465280.0,
- "12": 1042465280.0,
- "13": 1042465280.0,
- "14": 1042465280.0,
- "15": 1042465280.0,
- "16": 1042465280.0,
- "17": 1042465280.0,
- "18": 1042465280.0,
- "19": 1042465280.0,
- "20": 1042465280.0,
- "21": 1042465280.0,
- "22": 1042465280.0,
- "23": 1042465280.0,
- "24": 1042465280.0,
- "25": 1042465280.0,
- "26": 1042465280.0,
- "27": 1042465280.0,
- "28": 1042465280.0,
- "29": 1042465280.0,
- "30": 1042465280.0,
- "31": 1042465280.0,
- "32": 1042465280.0,
- "33": 1042465280.0,
- "34": 1042465280.0,
- "35": 1042465280.0,
- "36": 1042465280.0,
- "37": 1042465280.0,
- "38": 1042465280.0,
- "39": 1042465280.0,
- "40": 1042465280.0,
- "41": 1042465280.0,
- "42": 1042465280.0,
- "43": 1042465280.0,
- "44": 1042465280.0,
- "45": 1042465280.0,
- "46": 1042465280.0,
- "47": 1042465280.0,
- "48": 1042465280.0,
- "49": 1042465280.0,
- "50": 1042465280.0,
- "51": 1042465280.0,
- "52": 1042465280.0,
- "53": 1042465280.0,
- "54": 1042465280.0,
- "55": 1042465280.0,
- "56": 1042465280.0,
- "57": 1042465280.0,
- "58": 1042465280.0,
- "59": 1042465280.0,
- "60": 1042465280.0,
- "61": 1042465280.0,
- "62": 1042465280.0,
- "63": 1042465280.0,
- "64": 1042465280.0,
- "65": 1042465280.0,
- "66": 1042465280.0,
- "67": 1042465280.0,
- "68": 1042465280.0,
- "69": 1042465280.0,
- "70": 1042465280.0,
- "71": 1042465280.0,
- "72": 1042465280.0,
- "73": 1042465280.0,
- "74": 1042465280.0,
- "75": 1042465280.0,
- "76": 1042465280.0,
- "77": 1042465280.0,
- "78": 1042465280.0,
- "79": 1042465280.0,
- "80": 1042465280.0,
- "81": 1042465280.0,
- "82": 1042465280.0,
- "83": 1042465280.0,
- "84": 1042465280.0,
- "85": 1042465280.0,
- "86": 1042465280.0,
- "87": 1042465280.0,
- "88": 1042465280.0,
- "89": 1042465280.0,
- "90": 1042465280.0,
- "91": 1042465280.0,
- "92": 1042465280.0,
- "93": 1042465280.0,
- "94": 1042465280.0,
- "95": 1042465280.0,
- "96": 1042465280.0,
- "97": 1042465280.0,
- "98": 1042465280.0,
- "99": 1042465280.0,
- "100": 1042465280.0
+ "1": 976471552.0,
+ "2": 1041940992.0,
+ "3": 1041940992.0,
+ "4": 1041940992.0,
+ "5": 1041940992.0,
+ "6": 1041940992.0,
+ "7": 1041940992.0,
+ "8": 1041940992.0,
+ "9": 1041940992.0,
+ "10": 1041940992.0,
+ "11": 1041940992.0,
+ "12": 1041940992.0,
+ "13": 1041940992.0,
+ "14": 1041940992.0,
+ "15": 1041940992.0,
+ "16": 1041940992.0,
+ "17": 1041940992.0,
+ "18": 1041940992.0,
+ "19": 1041940992.0,
+ "20": 1041940992.0,
+ "21": 1041940992.0,
+ "22": 1041940992.0,
+ "23": 1041940992.0,
+ "24": 1041940992.0,
+ "25": 1041940992.0,
+ "26": 1041940992.0,
+ "27": 1041940992.0,
+ "28": 1041940992.0,
+ "29": 1041940992.0,
+ "30": 1041940992.0,
+ "31": 1041940992.0,
+ "32": 1041940992.0,
+ "33": 1041940992.0,
+ "34": 1041940992.0,
+ "35": 1041940992.0,
+ "36": 1041940992.0,
+ "37": 1041940992.0,
+ "38": 1041940992.0,
+ "39": 1041940992.0,
+ "40": 1041940992.0,
+ "41": 1041940992.0,
+ "42": 1041940992.0,
+ "43": 1041940992.0,
+ "44": 1041940992.0,
+ "45": 1041940992.0,
+ "46": 1041940992.0,
+ "47": 1041940992.0,
+ "48": 1041940992.0,
+ "49": 1041940992.0,
+ "50": 1041940992.0,
+ "51": 1041940992.0,
+ "52": 1041940992.0,
+ "53": 1041940992.0,
+ "54": 1041940992.0,
+ "55": 1041940992.0,
+ "56": 1041940992.0,
+ "57": 1041940992.0,
+ "58": 1041940992.0,
+ "59": 1041940992.0,
+ "60": 1041940992.0,
+ "61": 1041940992.0,
+ "62": 1041940992.0,
+ "63": 1041940992.0,
+ "64": 1041940992.0,
+ "65": 1041940992.0,
+ "66": 1041940992.0,
+ "67": 1041940992.0,
+ "68": 1041940992.0,
+ "69": 1041940992.0,
+ "70": 1041940992.0,
+ "71": 1041940992.0,
+ "72": 1041940992.0,
+ "73": 1041940992.0,
+ "74": 1041940992.0,
+ "75": 1041940992.0,
+ "76": 1041940992.0,
+ "77": 1041940992.0,
+ "78": 1041940992.0,
+ "79": 1041940992.0,
+ "80": 1041940992.0,
+ "81": 1041940992.0,
+ "82": 1041940992.0,
+ "83": 1041940992.0,
+ "84": 1041940992.0,
+ "85": 1041940992.0,
+ "86": 1041940992.0,
+ "87": 1041940992.0,
+ "88": 1041940992.0,
+ "89": 1041940992.0,
+ "90": 1041940992.0,
+ "91": 1041940992.0,
+ "92": 1041940992.0,
+ "93": 1041940992.0,
+ "94": 1041940992.0,
+ "95": 1041940992.0,
+ "96": 1041940992.0,
+ "97": 1041940992.0,
+ "98": 1041940992.0,
+ "99": 1041940992.0,
+ "100": 1041940992.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 9.3573,
- "2": 0.22781,
- "3": 0.20223,
- "4": 0.18298,
- "5": 0.18347,
- "6": 0.18262,
- "7": 0.18305,
- "8": 0.18295,
- "9": 0.18205,
- "10": 0.18986,
- "11": 0.18455,
- "12": 0.18245,
- "13": 0.18257,
- "14": 0.18276,
- "15": 0.18245,
- "16": 0.18291,
- "17": 0.18246,
- "18": 0.18732,
- "19": 0.18256,
- "20": 0.17944,
- "21": 0.18071,
- "22": 0.17927,
- "23": 0.18026,
- "24": 0.17928,
- "25": 0.17797,
- "26": 0.17889,
- "27": 0.17809,
- "28": 0.17769,
- "29": 0.1779,
- "30": 0.17904,
- "31": 0.1865,
- "32": 0.17922,
- "33": 0.17866,
- "34": 0.17807,
- "35": 0.17828,
- "36": 0.17941,
- "37": 0.17744,
- "38": 0.17752,
- "39": 0.17793,
- "40": 0.17906,
- "41": 0.17769,
- "42": 0.17938,
- "43": 0.17822,
- "44": 0.17848,
- "45": 0.17846,
- "46": 0.17952,
- "47": 0.17854,
- "48": 0.17937,
- "49": 0.17929,
- "50": 0.17767,
- "51": 0.19143,
- "52": 0.18056,
- "53": 0.18054,
- "54": 0.18173,
- "55": 0.18101,
- "56": 0.18146,
- "57": 0.1796,
- "58": 0.18116,
- "59": 0.18351,
- "60": 0.17824,
- "61": 0.17784,
- "62": 0.17757,
- "63": 0.17868,
- "64": 0.17881,
- "65": 0.17844,
- "66": 0.1766,
- "67": 0.17725,
- "68": 0.17696,
- "69": 0.1769,
- "70": 0.17752,
- "71": 0.17684,
- "72": 0.17943,
- "73": 0.17816,
- "74": 0.1781,
- "75": 0.17671,
- "76": 0.17658,
- "77": 0.17778,
- "78": 0.1771,
- "79": 0.17667,
- "80": 0.17694,
- "81": 0.17739,
- "82": 0.18259,
- "83": 0.1806,
- "84": 0.18169,
- "85": 0.18154,
- "86": 0.1832,
- "87": 0.18284,
- "88": 0.18358,
- "89": 0.18203,
- "90": 0.18406,
- "91": 0.18296,
- "92": 0.18249,
- "93": 0.1823,
- "94": 0.1834,
- "95": 0.18246,
- "96": 0.19284,
- "97": 0.7432,
- "98": 0.20476,
- "99": 0.19058,
- "100": 0.18263
+ "1": "nan",
+ "2": 5.20605,
+ "3": 0.20611,
+ "4": 0.18648,
+ "5": 0.18627,
+ "6": 0.18872,
+ "7": 0.18567,
+ "8": 0.18585,
+ "9": 0.19081,
+ "10": 0.18684,
+ "11": 0.18516,
+ "12": 0.18879,
+ "13": 0.1861,
+ "14": 0.18567,
+ "15": 0.18843,
+ "16": 0.18703,
+ "17": 0.186,
+ "18": 0.1858,
+ "19": 0.1884,
+ "20": 0.18686,
+ "21": 0.18625,
+ "22": 0.18998,
+ "23": 0.18559,
+ "24": 0.18422,
+ "25": 0.18672,
+ "26": 0.1853,
+ "27": 0.18557,
+ "28": 0.18427,
+ "29": 0.18683,
+ "30": 0.18552,
+ "31": 0.18535,
+ "32": 0.18485,
+ "33": 0.186,
+ "34": 0.18628,
+ "35": 0.1859,
+ "36": 0.18535,
+ "37": 0.18475,
+ "38": 0.18565,
+ "39": 0.18685,
+ "40": 0.18583,
+ "41": 0.18474,
+ "42": 0.18622,
+ "43": 0.18465,
+ "44": 0.18679,
+ "45": 0.19029,
+ "46": 0.19188,
+ "47": 0.19044,
+ "48": 0.18695,
+ "49": 0.19,
+ "50": 0.19197,
+ "51": 0.22015,
+ "52": 0.18986,
+ "53": 0.18727,
+ "54": 0.187,
+ "55": 0.18961,
+ "56": 0.18536,
+ "57": 0.1853,
+ "58": 0.18738,
+ "59": 0.18591,
+ "60": 0.18627,
+ "61": 0.18687,
+ "62": 0.18454,
+ "63": 0.18763,
+ "64": 0.18494,
+ "65": 0.18575,
+ "66": 0.18737,
+ "67": 0.18569,
+ "68": 0.18546,
+ "69": 0.18527,
+ "70": 0.1862,
+ "71": 0.18622,
+ "72": 0.18742,
+ "73": 0.18602,
+ "74": 0.18532,
+ "75": 0.18742,
+ "76": 0.18476,
+ "77": 0.18581,
+ "78": 0.18606,
+ "79": 0.1857,
+ "80": 0.18512,
+ "81": 0.18458,
+ "82": 0.18566,
+ "83": 0.18492,
+ "84": 0.18692,
+ "85": 0.18591,
+ "86": 0.18579,
+ "87": 0.18512,
+ "88": 0.18657,
+ "89": 0.18519,
+ "90": 0.18562,
+ "91": 0.18718,
+ "92": 0.18625,
+ "93": 0.19482,
+ "94": 0.19707,
+ "95": 0.19368,
+ "96": 0.19232,
+ "97": 0.19346,
+ "98": 0.18566,
+ "99": 0.18668,
+ "100": 0.1863
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/model_config.yaml
index fe8e0f493d1..bd6f526c209 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_gb200.json
index e73d1df6682..382c1034159 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_gb200.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.83993,
- "2": 10.85182,
- "3": 10.84166,
- "4": 10.84441,
- "5": 10.85514,
- "6": 10.86428,
- "7": 10.85243,
- "8": 10.84464,
- "9": 10.84864,
- "10": 10.81333,
- "11": 10.85638,
- "12": 10.84233,
- "13": 10.86046,
- "14": 10.84976,
- "15": 10.81618,
- "16": 10.80886,
- "17": 10.78242,
- "18": 10.79155,
- "19": 10.79495,
- "20": 10.7055,
- "21": 10.6978,
- "22": 10.58349,
- "23": 10.69268,
- "24": 10.60558,
- "25": 10.56742,
- "26": 10.61456,
- "27": 10.6067,
- "28": 10.55905,
- "29": 10.56526,
- "30": 10.37918,
- "31": 10.16276,
- "32": 10.45543,
- "33": 10.45037,
- "34": 10.23993,
- "35": 10.27354,
- "36": 10.24224,
- "37": 10.34559,
- "38": 10.21738,
- "39": 10.39453,
- "40": 10.095,
- "41": 10.15093,
- "42": 10.21235,
- "43": 9.87982,
- "44": 9.97875,
- "45": 9.85588,
- "46": 9.83349,
- "47": 10.14101,
- "48": 9.86418,
- "49": 9.55509,
- "50": 9.91636,
- "51": 9.86104,
- "52": 9.75109,
- "53": 10.06631,
- "54": 9.95634,
- "55": 9.89354,
- "56": 9.637,
- "57": 9.49142,
- "58": 9.8341,
- "59": 9.5931,
- "60": 9.51379,
- "61": 9.69183,
- "62": 9.99162,
- "63": 9.39196,
- "64": 9.77455,
- "65": 8.96319,
- "66": 9.70663,
- "67": 9.3789,
- "68": 9.78328,
- "69": 9.79736,
- "70": 9.73753,
- "71": 9.62711,
- "72": 9.58907,
- "73": 9.50446,
- "74": 8.94975,
- "75": 9.4278,
- "76": 9.08764,
- "77": 10.06759,
- "78": 9.72141,
- "79": 9.3861,
- "80": 9.40495,
- "81": 9.48596,
- "82": 9.70195,
- "83": 9.31553,
- "84": 9.41806,
- "85": 9.61378,
- "86": 9.08145,
- "87": 9.59631,
- "88": 9.75008,
- "89": 9.60386,
- "90": 9.82838,
- "91": 9.33622,
- "92": 9.35764,
- "93": 9.08795,
- "94": 8.83437,
- "95": 9.53352,
- "96": 9.53315,
- "97": 9.31129,
- "98": 9.67176,
- "99": 8.89816,
- "100": 9.40969
+ "1": 10.89594,
+ "2": 10.88351,
+ "3": 10.88336,
+ "4": 10.87907,
+ "5": 10.88164,
+ "6": 10.87691,
+ "7": 10.88192,
+ "8": 10.87682,
+ "9": 10.88198,
+ "10": 10.87749,
+ "11": 10.87492,
+ "12": 10.87081,
+ "13": 10.86708,
+ "14": 10.8669,
+ "15": 10.8376,
+ "16": 10.83405,
+ "17": 10.84807,
+ "18": 10.81903,
+ "19": 10.84136,
+ "20": 10.75949,
+ "21": 10.74746,
+ "22": 10.73572,
+ "23": 10.72721,
+ "24": 10.69985,
+ "25": 10.69419,
+ "26": 10.6836,
+ "27": 10.65169,
+ "28": 10.58843,
+ "29": 10.55142,
+ "30": 10.52746,
+ "31": 10.52345,
+ "32": 10.51391,
+ "33": 10.47623,
+ "34": 10.45223,
+ "35": 10.45088,
+ "36": 10.42395,
+ "37": 10.39412,
+ "38": 10.39667,
+ "39": 10.36573,
+ "40": 10.35691,
+ "41": 10.33,
+ "42": 10.31048,
+ "43": 10.29193,
+ "44": 10.26507,
+ "45": 10.27365,
+ "46": 10.24093,
+ "47": 10.22524,
+ "48": 10.18099,
+ "49": 10.18013,
+ "50": 10.18078,
+ "51": 10.18221,
+ "52": 10.13888,
+ "53": 10.14366,
+ "54": 10.10725,
+ "55": 10.0832,
+ "56": 10.10807,
+ "57": 10.09845,
+ "58": 10.11324,
+ "59": 10.05957,
+ "60": 10.0799,
+ "61": 10.02735,
+ "62": 10.00023,
+ "63": 10.07511,
+ "64": 10.03034,
+ "65": 10.00784,
+ "66": 10.02969,
+ "67": 10.00779,
+ "68": 9.96951,
+ "69": 9.99308,
+ "70": 9.96956,
+ "71": 9.9989,
+ "72": 9.98007,
+ "73": 9.97028,
+ "74": 9.95555,
+ "75": 9.93139,
+ "76": 9.96387,
+ "77": 9.95309,
+ "78": 9.90668,
+ "79": 9.91325,
+ "80": 9.92641,
+ "81": 9.95382,
+ "82": 9.88688,
+ "83": 9.85304,
+ "84": 9.79421,
+ "85": 9.78737,
+ "86": 9.87887,
+ "87": 9.90277,
+ "88": 9.87661,
+ "89": 9.82315,
+ "90": 9.81096,
+ "91": 9.82772,
+ "92": 9.81635,
+ "93": 9.75384,
+ "94": 9.82169,
+ "95": 9.82081,
+ "96": 9.80275,
+ "97": 9.74809,
+ "98": 9.77581,
+ "99": 9.81931,
+ "100": 9.71181
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1621.0,
- "2": 1756.0,
- "3": 1698.0,
- "4": 1764.0,
- "5": 2045.0,
- "6": 1927.0,
- "7": 1901.0,
- "8": 1768.0,
- "9": 1823.0,
- "10": 1456.0,
- "11": 1884.0,
- "12": 1834.0,
- "13": 2003.0,
- "14": 1786.0,
- "15": 1879.0,
- "16": 1948.0,
- "17": 1849.0,
- "18": 1718.0,
- "19": 1870.0,
- "20": 1750.0,
- "21": 1977.0,
- "22": 1741.0,
- "23": 1946.0,
- "24": 1642.0,
- "25": 1636.0,
- "26": 1817.0,
- "27": 1926.0,
- "28": 1981.0,
- "29": 1993.0,
- "30": 1929.0,
- "31": 1630.0,
- "32": 1896.0,
- "33": 2115.0,
- "34": 1824.0,
- "35": 1960.0,
- "36": 1935.0,
- "37": 2410.0,
- "38": 2259.0,
- "39": 2428.0,
- "40": 2119.0,
- "41": 2278.0,
- "42": 2118.0,
- "43": 1992.0,
- "44": 2041.0,
- "45": 1992.0,
- "46": 2158.0,
- "47": 2416.0,
- "48": 2338.0,
- "49": 2315.0,
- "50": 2242.0,
- "51": 2431.0,
- "52": 2467.0,
- "53": 2794.0,
- "54": 2675.0,
- "55": 2313.0,
- "56": 2597.0,
- "57": 2278.0,
- "58": 2887.0,
- "59": 2701.0,
- "60": 2190.0,
- "61": 2764.0,
- "62": 2576.0,
- "63": 2405.0,
- "64": 2903.0,
- "65": 2516.0,
- "66": 2885.0,
- "67": 2700.0,
- "68": 2682.0,
- "69": 2987.0,
- "70": 3141.0,
- "71": 3055.0,
- "72": 2413.0,
- "73": 2864.0,
- "74": 1870.0,
- "75": 2450.0,
- "76": 3032.0,
- "77": 3230.0,
- "78": 3125.0,
- "79": 2982.0,
- "80": 3203.0,
- "81": 3657.0,
- "82": 3174.0,
- "83": 2818.0,
- "84": 3190.0,
- "85": 3166.0,
- "86": 2793.0,
- "87": 3635.0,
- "88": 3005.0,
- "89": 3373.0,
- "90": 3066.0,
- "91": 2857.0,
- "92": 3080.0,
- "93": 2533.0,
- "94": 3303.0,
- "95": 3270.0,
- "96": 3416.0,
- "97": 3085.0,
- "98": 3437.0,
- "99": 3243.0,
- "100": 3119.0
+ "1": 1792.0,
+ "2": 1785.0,
+ "3": 1867.0,
+ "4": 1822.0,
+ "5": 1837.0,
+ "6": 1806.0,
+ "7": 1844.0,
+ "8": 1795.0,
+ "9": 1896.0,
+ "10": 1778.0,
+ "11": 1709.0,
+ "12": 1782.0,
+ "13": 1809.0,
+ "14": 1898.0,
+ "15": 1706.0,
+ "16": 1758.0,
+ "17": 1867.0,
+ "18": 1830.0,
+ "19": 1720.0,
+ "20": 1724.0,
+ "21": 1761.0,
+ "22": 1746.0,
+ "23": 1864.0,
+ "24": 1888.0,
+ "25": 1707.0,
+ "26": 1842.0,
+ "27": 1792.0,
+ "28": 1792.0,
+ "29": 1950.0,
+ "30": 1848.0,
+ "31": 2041.0,
+ "32": 1959.0,
+ "33": 1943.0,
+ "34": 2094.0,
+ "35": 2135.0,
+ "36": 2018.0,
+ "37": 2296.0,
+ "38": 2140.0,
+ "39": 2249.0,
+ "40": 2280.0,
+ "41": 2362.0,
+ "42": 2056.0,
+ "43": 2418.0,
+ "44": 2119.0,
+ "45": 2796.0,
+ "46": 2520.0,
+ "47": 2562.0,
+ "48": 2677.0,
+ "49": 2860.0,
+ "50": 2620.0,
+ "51": 2532.0,
+ "52": 2899.0,
+ "53": 2684.0,
+ "54": 2981.0,
+ "55": 2639.0,
+ "56": 2753.0,
+ "57": 2285.0,
+ "58": 3848.0,
+ "59": 3051.0,
+ "60": 3147.0,
+ "61": 2758.0,
+ "62": 3373.0,
+ "63": 3312.0,
+ "64": 3586.0,
+ "65": 2768.0,
+ "66": 3157.0,
+ "67": 4091.0,
+ "68": 3446.0,
+ "69": 3151.0,
+ "70": 3308.0,
+ "71": 2984.0,
+ "72": 2990.0,
+ "73": 3473.0,
+ "74": 3414.0,
+ "75": 3153.0,
+ "76": 3272.0,
+ "77": 3730.0,
+ "78": 3333.0,
+ "79": 3289.0,
+ "80": 3158.0,
+ "81": 3542.0,
+ "82": 2869.0,
+ "83": 3032.0,
+ "84": 2873.0,
+ "85": 2714.0,
+ "86": 3109.0,
+ "87": 2897.0,
+ "88": 3151.0,
+ "89": 3074.0,
+ "90": 3957.0,
+ "91": 2949.0,
+ "92": 3038.0,
+ "93": 3232.0,
+ "94": 3092.0,
+ "95": 3332.0,
+ "96": 3500.0,
+ "97": 3538.0,
+ "98": 3559.0,
+ "99": 3254.0,
+ "100": 3444.0
}
},
"mem-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 999540224.0,
- "2": 1065140736.0,
- "3": 1065140736.0,
- "4": 1065140736.0,
- "5": 1065140736.0,
- "6": 1065140736.0,
- "7": 1065140736.0,
- "8": 1065140736.0,
- "9": 1065140736.0,
- "10": 1065140736.0,
- "11": 1065140736.0,
- "12": 1065140736.0,
- "13": 1065140736.0,
- "14": 1065140736.0,
- "15": 1065140736.0,
- "16": 1065140736.0,
- "17": 1065140736.0,
- "18": 1065140736.0,
- "19": 1065140736.0,
- "20": 1065140736.0,
- "21": 1065140736.0,
- "22": 1065140736.0,
- "23": 1065140736.0,
- "24": 1065140736.0,
- "25": 1065140736.0,
- "26": 1065140736.0,
- "27": 1065140736.0,
- "28": 1065140736.0,
- "29": 1065140736.0,
- "30": 1065140736.0,
- "31": 1065140736.0,
- "32": 1065140736.0,
- "33": 1065140736.0,
- "34": 1065140736.0,
- "35": 1065140736.0,
- "36": 1065140736.0,
- "37": 1065140736.0,
- "38": 1065140736.0,
- "39": 1065140736.0,
- "40": 1065140736.0,
- "41": 1065140736.0,
- "42": 1065140736.0,
- "43": 1065140736.0,
- "44": 1065140736.0,
- "45": 1065140736.0,
- "46": 1065140736.0,
- "47": 1065140736.0,
- "48": 1065140736.0,
- "49": 1065140736.0,
- "50": 1065140736.0,
- "51": 1065140736.0,
- "52": 1065140736.0,
- "53": 1065140736.0,
- "54": 1065140736.0,
- "55": 1065140736.0,
- "56": 1065140736.0,
- "57": 1065140736.0,
- "58": 1065140736.0,
- "59": 1065140736.0,
- "60": 1065140736.0,
- "61": 1065140736.0,
- "62": 1065140736.0,
- "63": 1065140736.0,
- "64": 1065140736.0,
- "65": 1065140736.0,
- "66": 1065140736.0,
- "67": 1065140736.0,
- "68": 1065140736.0,
- "69": 1065140736.0,
- "70": 1065140736.0,
- "71": 1065140736.0,
- "72": 1065140736.0,
- "73": 1065140736.0,
- "74": 1065140736.0,
- "75": 1065140736.0,
- "76": 1065140736.0,
- "77": 1065140736.0,
- "78": 1065140736.0,
- "79": 1065140736.0,
- "80": 1065140736.0,
- "81": 1065140736.0,
- "82": 1065140736.0,
- "83": 1065140736.0,
- "84": 1065140736.0,
- "85": 1065140736.0,
- "86": 1065140736.0,
- "87": 1065140736.0,
- "88": 1065140736.0,
- "89": 1065140736.0,
- "90": 1065140736.0,
- "91": 1065140736.0,
- "92": 1065140736.0,
- "93": 1065140736.0,
- "94": 1065140736.0,
- "95": 1065140736.0,
- "96": 1065140736.0,
- "97": 1065140736.0,
- "98": 1065140736.0,
- "99": 1065140736.0,
- "100": 1065140736.0
+ "1": 995345920.0,
+ "2": 1059371008.0,
+ "3": 1059371008.0,
+ "4": 1059371008.0,
+ "5": 1059371008.0,
+ "6": 1059371008.0,
+ "7": 1059371008.0,
+ "8": 1059371008.0,
+ "9": 1059371008.0,
+ "10": 1059371008.0,
+ "11": 1059371008.0,
+ "12": 1059371008.0,
+ "13": 1059371008.0,
+ "14": 1059371008.0,
+ "15": 1059371008.0,
+ "16": 1059371008.0,
+ "17": 1059371008.0,
+ "18": 1059371008.0,
+ "19": 1059371008.0,
+ "20": 1059371008.0,
+ "21": 1059371008.0,
+ "22": 1059371008.0,
+ "23": 1059371008.0,
+ "24": 1059371008.0,
+ "25": 1059371008.0,
+ "26": 1059371008.0,
+ "27": 1059371008.0,
+ "28": 1059371008.0,
+ "29": 1059371008.0,
+ "30": 1059371008.0,
+ "31": 1059371008.0,
+ "32": 1059371008.0,
+ "33": 1059371008.0,
+ "34": 1059371008.0,
+ "35": 1059371008.0,
+ "36": 1059371008.0,
+ "37": 1059371008.0,
+ "38": 1059371008.0,
+ "39": 1059371008.0,
+ "40": 1059371008.0,
+ "41": 1059371008.0,
+ "42": 1059371008.0,
+ "43": 1059371008.0,
+ "44": 1059371008.0,
+ "45": 1059371008.0,
+ "46": 1059371008.0,
+ "47": 1059371008.0,
+ "48": 1059371008.0,
+ "49": 1059371008.0,
+ "50": 1059371008.0,
+ "51": 1059371008.0,
+ "52": 1059371008.0,
+ "53": 1059371008.0,
+ "54": 1059371008.0,
+ "55": 1059371008.0,
+ "56": 1059371008.0,
+ "57": 1059371008.0,
+ "58": 1059371008.0,
+ "59": 1059371008.0,
+ "60": 1059371008.0,
+ "61": 1059371008.0,
+ "62": 1059371008.0,
+ "63": 1059371008.0,
+ "64": 1059371008.0,
+ "65": 1059371008.0,
+ "66": 1059371008.0,
+ "67": 1059371008.0,
+ "68": 1059371008.0,
+ "69": 1059371008.0,
+ "70": 1059371008.0,
+ "71": 1059371008.0,
+ "72": 1059371008.0,
+ "73": 1059371008.0,
+ "74": 1059371008.0,
+ "75": 1059371008.0,
+ "76": 1059371008.0,
+ "77": 1059371008.0,
+ "78": 1059371008.0,
+ "79": 1059371008.0,
+ "80": 1059371008.0,
+ "81": 1059371008.0,
+ "82": 1059371008.0,
+ "83": 1059371008.0,
+ "84": 1059371008.0,
+ "85": 1059371008.0,
+ "86": 1059371008.0,
+ "87": 1059371008.0,
+ "88": 1059371008.0,
+ "89": 1059371008.0,
+ "90": 1059371008.0,
+ "91": 1059371008.0,
+ "92": 1059371008.0,
+ "93": 1059371008.0,
+ "94": 1059371008.0,
+ "95": 1059371008.0,
+ "96": 1059371008.0,
+ "97": 1059371008.0,
+ "98": 1059371008.0,
+ "99": 1059371008.0,
+ "100": 1059371008.0
}
},
"iteration-time": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.50092,
- "3": 0.30393,
- "4": 0.296,
- "5": 0.29464,
- "6": 0.29386,
- "7": 0.29621,
- "8": 0.2946,
- "9": 0.29682,
- "10": 0.29745,
- "11": 0.3056,
- "12": 0.30475,
- "13": 0.30581,
- "14": 0.3052,
- "15": 0.31033,
- "16": 0.30534,
- "17": 0.30586,
- "18": 0.3053,
- "19": 0.30668,
- "20": 0.3062,
- "21": 0.31086,
- "22": 0.30673,
- "23": 0.30645,
- "24": 0.30648,
- "25": 0.30922,
- "26": 0.30442,
- "27": 0.30196,
- "28": 0.3042,
- "29": 0.30389,
- "30": 0.30468,
- "31": 0.30661,
- "32": 0.30468,
- "33": 0.30645,
- "34": 0.30588,
- "35": 0.3037,
- "36": 0.30433,
- "37": 0.30504,
- "38": 0.30676,
- "39": 0.30639,
- "40": 0.30854,
- "41": 0.31017,
- "42": 0.30559,
- "43": 0.30359,
- "44": 0.30728,
- "45": 0.30737,
- "46": 0.30728,
- "47": 0.30866,
- "48": 0.30981,
- "49": 0.3097,
- "50": 0.30633,
- "51": 0.31798,
- "52": 0.30466,
- "53": 0.30302,
- "54": 0.30516,
- "55": 0.30263,
- "56": 0.30315,
- "57": 0.30305,
- "58": 0.30451,
- "59": 0.30443,
- "60": 0.30525,
- "61": 0.30503,
- "62": 0.3063,
- "63": 0.30517,
- "64": 0.30552,
- "65": 0.30685,
- "66": 0.30584,
- "67": 0.31593,
- "68": 0.34589,
- "69": 0.30682,
- "70": 0.30582,
- "71": 0.30682,
- "72": 0.30578,
- "73": 0.30496,
- "74": 0.30689,
- "75": 0.30927,
- "76": 0.31024,
- "77": 0.3125,
- "78": 0.31093,
- "79": 0.31106,
- "80": 0.30717,
- "81": 0.30815,
- "82": 0.30914,
- "83": 0.30911,
- "84": 0.30335,
- "85": 0.29792,
- "86": 0.2997,
- "87": 0.3032,
- "88": 0.30139,
- "89": 0.30675,
- "90": 0.30412,
- "91": 0.30454,
- "92": 0.30497,
- "93": 0.30233,
- "94": 0.30714,
- "95": 0.30673,
- "96": 0.30193,
- "97": 0.30472,
- "98": 0.3103,
- "99": 0.30957,
- "100": 0.30828
+ "2": 4.8267,
+ "3": 0.34085,
+ "4": 0.32982,
+ "5": 0.32892,
+ "6": 0.33192,
+ "7": 0.32911,
+ "8": 0.33028,
+ "9": 0.33089,
+ "10": 0.33287,
+ "11": 0.33274,
+ "12": 0.33118,
+ "13": 0.33152,
+ "14": 0.33094,
+ "15": 0.33153,
+ "16": 0.33331,
+ "17": 0.33266,
+ "18": 0.33104,
+ "19": 0.33133,
+ "20": 0.33182,
+ "21": 0.33045,
+ "22": 0.33149,
+ "23": 0.33316,
+ "24": 0.3327,
+ "25": 0.33129,
+ "26": 0.33073,
+ "27": 0.33218,
+ "28": 0.33376,
+ "29": 0.33369,
+ "30": 0.33251,
+ "31": 0.33054,
+ "32": 0.33315,
+ "33": 0.32994,
+ "34": 0.3313,
+ "35": 0.32957,
+ "36": 0.33034,
+ "37": 0.32903,
+ "38": 0.33317,
+ "39": 0.3302,
+ "40": 0.33014,
+ "41": 0.3313,
+ "42": 0.44119,
+ "43": 0.43647,
+ "44": 0.40484,
+ "45": 0.37665,
+ "46": 0.32959,
+ "47": 0.33032,
+ "48": 0.3313,
+ "49": 0.3296,
+ "50": 0.33059,
+ "51": 0.48138,
+ "52": 0.33566,
+ "53": 0.33373,
+ "54": 0.32999,
+ "55": 0.3319,
+ "56": 0.3311,
+ "57": 0.3327,
+ "58": 0.33471,
+ "59": 0.33269,
+ "60": 0.33252,
+ "61": 0.3339,
+ "62": 0.33293,
+ "63": 0.33378,
+ "64": 0.33271,
+ "65": 0.33429,
+ "66": 0.33318,
+ "67": 0.33326,
+ "68": 0.33249,
+ "69": 0.3328,
+ "70": 0.33331,
+ "71": 0.33409,
+ "72": 0.33333,
+ "73": 0.33467,
+ "74": 0.33465,
+ "75": 0.33585,
+ "76": 0.33294,
+ "77": 0.3342,
+ "78": 0.3349,
+ "79": 0.33721,
+ "80": 0.33599,
+ "81": 0.33565,
+ "82": 0.33229,
+ "83": 0.33474,
+ "84": 0.33586,
+ "85": 0.33447,
+ "86": 0.33328,
+ "87": 0.33385,
+ "88": 0.33369,
+ "89": 0.3338,
+ "90": 0.33435,
+ "91": 0.33466,
+ "92": 0.3353,
+ "93": 0.3356,
+ "94": 0.33347,
+ "95": 0.33564,
+ "96": 0.3349,
+ "97": 0.33443,
+ "98": 0.33348,
+ "99": 0.33407,
+ "100": 0.33374
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_h100.json
index 2d2d349a867..bb55b89c3e1 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.84163,
- "2": 10.85598,
- "3": 10.84413,
- "4": 10.84124,
- "5": 10.85872,
- "6": 10.86316,
- "7": 10.85184,
- "8": 10.84645,
- "9": 10.85647,
- "10": 10.81849,
- "11": 10.85923,
- "12": 10.84285,
- "13": 10.86432,
- "14": 10.85423,
- "15": 10.81015,
- "16": 10.81588,
- "17": 10.78949,
- "18": 10.79683,
- "19": 10.79073,
- "20": 10.70819,
- "21": 10.69322,
- "22": 10.58504,
- "23": 10.70217,
- "24": 10.60546,
- "25": 10.57102,
- "26": 10.61967,
- "27": 10.61501,
- "28": 10.56369,
- "29": 10.56725,
- "30": 10.39695,
- "31": 10.16591,
- "32": 10.4573,
- "33": 10.45199,
- "34": 10.2392,
- "35": 10.28351,
- "36": 10.24677,
- "37": 10.3427,
- "38": 10.20546,
- "39": 10.39187,
- "40": 10.09767,
- "41": 10.1526,
- "42": 10.21051,
- "43": 9.87726,
- "44": 9.98291,
- "45": 9.86165,
- "46": 9.83587,
- "47": 10.13369,
- "48": 9.87212,
- "49": 9.56121,
- "50": 9.91045,
- "51": 9.85839,
- "52": 9.7506,
- "53": 10.05817,
- "54": 9.96076,
- "55": 9.88738,
- "56": 9.6344,
- "57": 9.4967,
- "58": 9.83343,
- "59": 9.59391,
- "60": 9.51376,
- "61": 9.69928,
- "62": 9.98089,
- "63": 9.39065,
- "64": 9.77599,
- "65": 8.9571,
- "66": 9.70054,
- "67": 9.37,
- "68": 9.78529,
- "69": 9.78966,
- "70": 9.74676,
- "71": 9.61906,
- "72": 9.58963,
- "73": 9.49629,
- "74": 8.94963,
- "75": 9.42381,
- "76": 9.07799,
- "77": 10.07105,
- "78": 9.72632,
- "79": 9.37966,
- "80": 9.40721,
- "81": 9.48238,
- "82": 9.70152,
- "83": 9.30657,
- "84": 9.41464,
- "85": 9.61784,
- "86": 9.08212,
- "87": 9.59511,
- "88": 9.75008,
- "89": 9.60356,
- "90": 9.82256,
- "91": 9.33721,
- "92": 9.35861,
- "93": 9.07956,
- "94": 8.83268,
- "95": 9.51351,
- "96": 9.52947,
- "97": 9.31813,
- "98": 9.67451,
- "99": 8.88607,
- "100": 9.40106
+ "1": 10.89723,
+ "2": 10.89039,
+ "3": 10.88307,
+ "4": 10.88936,
+ "5": 10.88232,
+ "6": 10.88662,
+ "7": 10.88761,
+ "8": 10.88217,
+ "9": 10.8849,
+ "10": 10.88111,
+ "11": 10.87928,
+ "12": 10.87753,
+ "13": 10.87277,
+ "14": 10.86825,
+ "15": 10.84625,
+ "16": 10.83788,
+ "17": 10.83727,
+ "18": 10.8269,
+ "19": 10.83394,
+ "20": 10.76068,
+ "21": 10.74169,
+ "22": 10.7253,
+ "23": 10.71963,
+ "24": 10.70186,
+ "25": 10.69466,
+ "26": 10.67749,
+ "27": 10.65204,
+ "28": 10.5744,
+ "29": 10.55037,
+ "30": 10.52252,
+ "31": 10.52588,
+ "32": 10.51364,
+ "33": 10.46604,
+ "34": 10.43791,
+ "35": 10.43778,
+ "36": 10.42157,
+ "37": 10.38929,
+ "38": 10.3894,
+ "39": 10.35675,
+ "40": 10.34752,
+ "41": 10.32295,
+ "42": 10.30553,
+ "43": 10.28355,
+ "44": 10.26326,
+ "45": 10.26616,
+ "46": 10.23683,
+ "47": 10.22148,
+ "48": 10.17275,
+ "49": 10.18154,
+ "50": 10.17793,
+ "51": 10.18019,
+ "52": 10.13373,
+ "53": 10.1315,
+ "54": 10.10642,
+ "55": 10.08299,
+ "56": 10.10639,
+ "57": 10.10204,
+ "58": 10.10815,
+ "59": 10.04453,
+ "60": 10.07109,
+ "61": 10.0256,
+ "62": 9.99846,
+ "63": 10.06819,
+ "64": 10.01962,
+ "65": 9.98075,
+ "66": 10.02428,
+ "67": 10.00273,
+ "68": 9.96779,
+ "69": 9.98681,
+ "70": 9.9667,
+ "71": 9.99226,
+ "72": 9.97695,
+ "73": 9.96099,
+ "74": 9.95635,
+ "75": 9.92491,
+ "76": 9.95107,
+ "77": 9.94714,
+ "78": 9.90073,
+ "79": 9.90294,
+ "80": 9.91743,
+ "81": 9.94303,
+ "82": 9.8817,
+ "83": 9.84051,
+ "84": 9.78041,
+ "85": 9.77117,
+ "86": 9.87348,
+ "87": 9.90186,
+ "88": 9.87565,
+ "89": 9.82672,
+ "90": 9.81116,
+ "91": 9.8207,
+ "92": 9.81383,
+ "93": 9.74934,
+ "94": 9.81784,
+ "95": 9.8082,
+ "96": 9.79459,
+ "97": 9.74389,
+ "98": 9.76749,
+ "99": 9.81863,
+ "100": 9.705
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1736.0,
- "2": 1692.0,
- "3": 1695.0,
- "4": 1761.0,
- "5": 1955.0,
- "6": 1791.0,
- "7": 1943.0,
- "8": 1681.0,
- "9": 1884.0,
- "10": 1441.0,
- "11": 1942.0,
- "12": 1786.0,
- "13": 1940.0,
- "14": 1862.0,
- "15": 1907.0,
- "16": 1947.0,
- "17": 1827.0,
- "18": 1907.0,
- "19": 1818.0,
- "20": 1700.0,
- "21": 1911.0,
- "22": 1720.0,
- "23": 1938.0,
- "24": 1707.0,
- "25": 1686.0,
- "26": 1792.0,
- "27": 1891.0,
- "28": 1976.0,
- "29": 1958.0,
- "30": 1941.0,
- "31": 1622.0,
- "32": 1970.0,
- "33": 2129.0,
- "34": 1830.0,
- "35": 1907.0,
- "36": 1892.0,
- "37": 2395.0,
- "38": 2161.0,
- "39": 2493.0,
- "40": 2224.0,
- "41": 2201.0,
- "42": 2175.0,
- "43": 1920.0,
- "44": 1955.0,
- "45": 1956.0,
- "46": 2166.0,
- "47": 2517.0,
- "48": 2272.0,
- "49": 2211.0,
- "50": 2232.0,
- "51": 2621.0,
- "52": 2597.0,
- "53": 2926.0,
- "54": 2633.0,
- "55": 2206.0,
- "56": 2627.0,
- "57": 2328.0,
- "58": 2886.0,
- "59": 2639.0,
- "60": 2157.0,
- "61": 2736.0,
- "62": 2544.0,
- "63": 2332.0,
- "64": 2948.0,
- "65": 2630.0,
- "66": 2931.0,
- "67": 2717.0,
- "68": 2643.0,
- "69": 2955.0,
- "70": 3040.0,
- "71": 2882.0,
- "72": 2390.0,
- "73": 2812.0,
- "74": 1844.0,
- "75": 2461.0,
- "76": 3067.0,
- "77": 3152.0,
- "78": 3018.0,
- "79": 3008.0,
- "80": 3104.0,
- "81": 3589.0,
- "82": 3218.0,
- "83": 2748.0,
- "84": 3217.0,
- "85": 3167.0,
- "86": 2876.0,
- "87": 3604.0,
- "88": 3017.0,
- "89": 3249.0,
- "90": 3069.0,
- "91": 2865.0,
- "92": 3074.0,
- "93": 2680.0,
- "94": 3392.0,
- "95": 3206.0,
- "96": 3401.0,
- "97": 3107.0,
- "98": 3624.0,
- "99": 3007.0,
- "100": 3111.0
+ "1": 1842.0,
+ "2": 1747.0,
+ "3": 1798.0,
+ "4": 1756.0,
+ "5": 1843.0,
+ "6": 1866.0,
+ "7": 1878.0,
+ "8": 1740.0,
+ "9": 1664.0,
+ "10": 1694.0,
+ "11": 1777.0,
+ "12": 1763.0,
+ "13": 1794.0,
+ "14": 1961.0,
+ "15": 1708.0,
+ "16": 1873.0,
+ "17": 1828.0,
+ "18": 1807.0,
+ "19": 1703.0,
+ "20": 1809.0,
+ "21": 1823.0,
+ "22": 1798.0,
+ "23": 1823.0,
+ "24": 1882.0,
+ "25": 1798.0,
+ "26": 1840.0,
+ "27": 1876.0,
+ "28": 1776.0,
+ "29": 2037.0,
+ "30": 1728.0,
+ "31": 1990.0,
+ "32": 1897.0,
+ "33": 1977.0,
+ "34": 2036.0,
+ "35": 2065.0,
+ "36": 2018.0,
+ "37": 2375.0,
+ "38": 2136.0,
+ "39": 2241.0,
+ "40": 2382.0,
+ "41": 2342.0,
+ "42": 2013.0,
+ "43": 2345.0,
+ "44": 2183.0,
+ "45": 2673.0,
+ "46": 2457.0,
+ "47": 2520.0,
+ "48": 2607.0,
+ "49": 2899.0,
+ "50": 2592.0,
+ "51": 2637.0,
+ "52": 2871.0,
+ "53": 2636.0,
+ "54": 2970.0,
+ "55": 2582.0,
+ "56": 2823.0,
+ "57": 2225.0,
+ "58": 3660.0,
+ "59": 2965.0,
+ "60": 2864.0,
+ "61": 2677.0,
+ "62": 3102.0,
+ "63": 3353.0,
+ "64": 3833.0,
+ "65": 2567.0,
+ "66": 3265.0,
+ "67": 3776.0,
+ "68": 3535.0,
+ "69": 3118.0,
+ "70": 3221.0,
+ "71": 3097.0,
+ "72": 3003.0,
+ "73": 3379.0,
+ "74": 3261.0,
+ "75": 3172.0,
+ "76": 3260.0,
+ "77": 3753.0,
+ "78": 3266.0,
+ "79": 3368.0,
+ "80": 3127.0,
+ "81": 3608.0,
+ "82": 2987.0,
+ "83": 3167.0,
+ "84": 2790.0,
+ "85": 2700.0,
+ "86": 3394.0,
+ "87": 2936.0,
+ "88": 2982.0,
+ "89": 3011.0,
+ "90": 3939.0,
+ "91": 3120.0,
+ "92": 2907.0,
+ "93": 3188.0,
+ "94": 3297.0,
+ "95": 3228.0,
+ "96": 3455.0,
+ "97": 3575.0,
+ "98": 3393.0,
+ "99": 3026.0,
+ "100": 3072.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 299204096.0,
- "2": 299204096.0,
- "3": 299204096.0,
- "4": 299204096.0,
- "5": 299204096.0,
- "6": 299204096.0,
- "7": 299204096.0,
- "8": 299204096.0,
- "9": 299204096.0,
- "10": 299204096.0,
- "11": 299204096.0,
- "12": 299204096.0,
- "13": 299204096.0,
- "14": 299204096.0,
- "15": 299204096.0,
- "16": 299204096.0,
- "17": 299204096.0,
- "18": 299204096.0,
- "19": 299204096.0,
- "20": 299204096.0,
- "21": 299204096.0,
- "22": 299204096.0,
- "23": 299204096.0,
- "24": 299204096.0,
- "25": 299204096.0,
- "26": 299204096.0,
- "27": 299204096.0,
- "28": 299204096.0,
- "29": 299204096.0,
- "30": 299204096.0,
- "31": 299204096.0,
- "32": 299204096.0,
- "33": 299204096.0,
- "34": 299204096.0,
- "35": 299204096.0,
- "36": 299204096.0,
- "37": 299204096.0,
- "38": 299204096.0,
- "39": 299204096.0,
- "40": 299204096.0,
- "41": 299204096.0,
- "42": 299204096.0,
- "43": 299204096.0,
- "44": 299204096.0,
- "45": 299204096.0,
- "46": 299204096.0,
- "47": 299204096.0,
- "48": 299204096.0,
- "49": 299204096.0,
- "50": 299204096.0,
- "51": 299204096.0,
- "52": 299204096.0,
- "53": 299204096.0,
- "54": 299204096.0,
- "55": 299204096.0,
- "56": 299204096.0,
- "57": 299204096.0,
- "58": 299204096.0,
- "59": 299204096.0,
- "60": 299204096.0,
- "61": 299204096.0,
- "62": 299204096.0,
- "63": 299204096.0,
- "64": 299204096.0,
- "65": 299204096.0,
- "66": 299204096.0,
- "67": 299204096.0,
- "68": 299204096.0,
- "69": 299204096.0,
- "70": 299204096.0,
- "71": 299204096.0,
- "72": 299204096.0,
- "73": 299204096.0,
- "74": 299204096.0,
- "75": 299204096.0,
- "76": 299204096.0,
- "77": 299204096.0,
- "78": 299204096.0,
- "79": 299204096.0,
- "80": 299204096.0,
- "81": 299204096.0,
- "82": 299204096.0,
- "83": 299204096.0,
- "84": 299204096.0,
- "85": 299204096.0,
- "86": 299204096.0,
- "87": 299204096.0,
- "88": 299204096.0,
- "89": 299204096.0,
- "90": 299204096.0,
- "91": 299204096.0,
- "92": 299204096.0,
- "93": 299204096.0,
- "94": 299204096.0,
- "95": 299204096.0,
- "96": 299204096.0,
- "97": 299204096.0,
- "98": 299204096.0,
- "99": 299204096.0,
- "100": 299204096.0
+ "1": 299728384.0,
+ "2": 299728384.0,
+ "3": 299728384.0,
+ "4": 299728384.0,
+ "5": 299728384.0,
+ "6": 299728384.0,
+ "7": 299728384.0,
+ "8": 299728384.0,
+ "9": 299728384.0,
+ "10": 299728384.0,
+ "11": 299728384.0,
+ "12": 299728384.0,
+ "13": 299728384.0,
+ "14": 299728384.0,
+ "15": 299728384.0,
+ "16": 299728384.0,
+ "17": 299728384.0,
+ "18": 299728384.0,
+ "19": 299728384.0,
+ "20": 299728384.0,
+ "21": 299728384.0,
+ "22": 299728384.0,
+ "23": 299728384.0,
+ "24": 299728384.0,
+ "25": 299728384.0,
+ "26": 299728384.0,
+ "27": 299728384.0,
+ "28": 299728384.0,
+ "29": 299728384.0,
+ "30": 299728384.0,
+ "31": 299728384.0,
+ "32": 299728384.0,
+ "33": 299728384.0,
+ "34": 299728384.0,
+ "35": 299728384.0,
+ "36": 299728384.0,
+ "37": 299728384.0,
+ "38": 299728384.0,
+ "39": 299728384.0,
+ "40": 299728384.0,
+ "41": 299728384.0,
+ "42": 299728384.0,
+ "43": 299728384.0,
+ "44": 299728384.0,
+ "45": 299728384.0,
+ "46": 299728384.0,
+ "47": 299728384.0,
+ "48": 299728384.0,
+ "49": 299728384.0,
+ "50": 299728384.0,
+ "51": 299728384.0,
+ "52": 299728384.0,
+ "53": 299728384.0,
+ "54": 299728384.0,
+ "55": 299728384.0,
+ "56": 299728384.0,
+ "57": 299728384.0,
+ "58": 299728384.0,
+ "59": 299728384.0,
+ "60": 299728384.0,
+ "61": 299728384.0,
+ "62": 299728384.0,
+ "63": 299728384.0,
+ "64": 299728384.0,
+ "65": 299728384.0,
+ "66": 299728384.0,
+ "67": 299728384.0,
+ "68": 299728384.0,
+ "69": 299728384.0,
+ "70": 299728384.0,
+ "71": 299728384.0,
+ "72": 299728384.0,
+ "73": 299728384.0,
+ "74": 299728384.0,
+ "75": 299728384.0,
+ "76": 299728384.0,
+ "77": 299728384.0,
+ "78": 299728384.0,
+ "79": 299728384.0,
+ "80": 299728384.0,
+ "81": 299728384.0,
+ "82": 299728384.0,
+ "83": 299728384.0,
+ "84": 299728384.0,
+ "85": 299728384.0,
+ "86": 299728384.0,
+ "87": 299728384.0,
+ "88": 299728384.0,
+ "89": 299728384.0,
+ "90": 299728384.0,
+ "91": 299728384.0,
+ "92": 299728384.0,
+ "93": 299728384.0,
+ "94": 299728384.0,
+ "95": 299728384.0,
+ "96": 299728384.0,
+ "97": 299728384.0,
+ "98": 299728384.0,
+ "99": 299728384.0,
+ "100": 299728384.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 977520128.0,
- "2": 1042465280.0,
- "3": 1042465280.0,
- "4": 1042465280.0,
- "5": 1042465280.0,
- "6": 1042465280.0,
- "7": 1042465280.0,
- "8": 1042465280.0,
- "9": 1042465280.0,
- "10": 1042465280.0,
- "11": 1042465280.0,
- "12": 1042465280.0,
- "13": 1042465280.0,
- "14": 1042465280.0,
- "15": 1042465280.0,
- "16": 1042465280.0,
- "17": 1042465280.0,
- "18": 1042465280.0,
- "19": 1042465280.0,
- "20": 1042465280.0,
- "21": 1042465280.0,
- "22": 1042465280.0,
- "23": 1042465280.0,
- "24": 1042465280.0,
- "25": 1042465280.0,
- "26": 1042465280.0,
- "27": 1042465280.0,
- "28": 1042465280.0,
- "29": 1042465280.0,
- "30": 1042465280.0,
- "31": 1042465280.0,
- "32": 1042465280.0,
- "33": 1042465280.0,
- "34": 1042465280.0,
- "35": 1042465280.0,
- "36": 1042465280.0,
- "37": 1042465280.0,
- "38": 1042465280.0,
- "39": 1042465280.0,
- "40": 1042465280.0,
- "41": 1042465280.0,
- "42": 1042465280.0,
- "43": 1042465280.0,
- "44": 1042465280.0,
- "45": 1042465280.0,
- "46": 1042465280.0,
- "47": 1042465280.0,
- "48": 1042465280.0,
- "49": 1042465280.0,
- "50": 1042465280.0,
- "51": 1042465280.0,
- "52": 1042465280.0,
- "53": 1042465280.0,
- "54": 1042465280.0,
- "55": 1042465280.0,
- "56": 1042465280.0,
- "57": 1042465280.0,
- "58": 1042465280.0,
- "59": 1042465280.0,
- "60": 1042465280.0,
- "61": 1042465280.0,
- "62": 1042465280.0,
- "63": 1042465280.0,
- "64": 1042465280.0,
- "65": 1042465280.0,
- "66": 1042465280.0,
- "67": 1042465280.0,
- "68": 1042465280.0,
- "69": 1042465280.0,
- "70": 1042465280.0,
- "71": 1042465280.0,
- "72": 1042465280.0,
- "73": 1042465280.0,
- "74": 1042465280.0,
- "75": 1042465280.0,
- "76": 1042465280.0,
- "77": 1042465280.0,
- "78": 1042465280.0,
- "79": 1042465280.0,
- "80": 1042465280.0,
- "81": 1042465280.0,
- "82": 1042465280.0,
- "83": 1042465280.0,
- "84": 1042465280.0,
- "85": 1042465280.0,
- "86": 1042465280.0,
- "87": 1042465280.0,
- "88": 1042465280.0,
- "89": 1042465280.0,
- "90": 1042465280.0,
- "91": 1042465280.0,
- "92": 1042465280.0,
- "93": 1042465280.0,
- "94": 1042465280.0,
- "95": 1042465280.0,
- "96": 1042465280.0,
- "97": 1042465280.0,
- "98": 1042465280.0,
- "99": 1042465280.0,
- "100": 1042465280.0
+ "1": 976471552.0,
+ "2": 1041940992.0,
+ "3": 1041940992.0,
+ "4": 1041940992.0,
+ "5": 1041940992.0,
+ "6": 1041940992.0,
+ "7": 1041940992.0,
+ "8": 1041940992.0,
+ "9": 1041940992.0,
+ "10": 1041940992.0,
+ "11": 1041940992.0,
+ "12": 1041940992.0,
+ "13": 1041940992.0,
+ "14": 1041940992.0,
+ "15": 1041940992.0,
+ "16": 1041940992.0,
+ "17": 1041940992.0,
+ "18": 1041940992.0,
+ "19": 1041940992.0,
+ "20": 1041940992.0,
+ "21": 1041940992.0,
+ "22": 1041940992.0,
+ "23": 1041940992.0,
+ "24": 1041940992.0,
+ "25": 1041940992.0,
+ "26": 1041940992.0,
+ "27": 1041940992.0,
+ "28": 1041940992.0,
+ "29": 1041940992.0,
+ "30": 1041940992.0,
+ "31": 1041940992.0,
+ "32": 1041940992.0,
+ "33": 1041940992.0,
+ "34": 1041940992.0,
+ "35": 1041940992.0,
+ "36": 1041940992.0,
+ "37": 1041940992.0,
+ "38": 1041940992.0,
+ "39": 1041940992.0,
+ "40": 1041940992.0,
+ "41": 1041940992.0,
+ "42": 1041940992.0,
+ "43": 1041940992.0,
+ "44": 1041940992.0,
+ "45": 1041940992.0,
+ "46": 1041940992.0,
+ "47": 1041940992.0,
+ "48": 1041940992.0,
+ "49": 1041940992.0,
+ "50": 1041940992.0,
+ "51": 1041940992.0,
+ "52": 1041940992.0,
+ "53": 1041940992.0,
+ "54": 1041940992.0,
+ "55": 1041940992.0,
+ "56": 1041940992.0,
+ "57": 1041940992.0,
+ "58": 1041940992.0,
+ "59": 1041940992.0,
+ "60": 1041940992.0,
+ "61": 1041940992.0,
+ "62": 1041940992.0,
+ "63": 1041940992.0,
+ "64": 1041940992.0,
+ "65": 1041940992.0,
+ "66": 1041940992.0,
+ "67": 1041940992.0,
+ "68": 1041940992.0,
+ "69": 1041940992.0,
+ "70": 1041940992.0,
+ "71": 1041940992.0,
+ "72": 1041940992.0,
+ "73": 1041940992.0,
+ "74": 1041940992.0,
+ "75": 1041940992.0,
+ "76": 1041940992.0,
+ "77": 1041940992.0,
+ "78": 1041940992.0,
+ "79": 1041940992.0,
+ "80": 1041940992.0,
+ "81": 1041940992.0,
+ "82": 1041940992.0,
+ "83": 1041940992.0,
+ "84": 1041940992.0,
+ "85": 1041940992.0,
+ "86": 1041940992.0,
+ "87": 1041940992.0,
+ "88": 1041940992.0,
+ "89": 1041940992.0,
+ "90": 1041940992.0,
+ "91": 1041940992.0,
+ "92": 1041940992.0,
+ "93": 1041940992.0,
+ "94": 1041940992.0,
+ "95": 1041940992.0,
+ "96": 1041940992.0,
+ "97": 1041940992.0,
+ "98": 1041940992.0,
+ "99": 1041940992.0,
+ "100": 1041940992.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 9.84544,
- "2": 0.22725,
- "3": 0.20768,
- "4": 0.18628,
- "5": 0.18333,
- "6": 0.18666,
- "7": 0.18629,
- "8": 0.18455,
- "9": 0.18539,
- "10": 0.18537,
- "11": 0.18771,
- "12": 0.18396,
- "13": 0.18789,
- "14": 0.18938,
- "15": 0.18649,
- "16": 0.18634,
- "17": 0.18623,
- "18": 0.18688,
- "19": 0.18602,
- "20": 0.18599,
- "21": 0.18725,
- "22": 0.19085,
- "23": 0.18959,
- "24": 0.19257,
- "25": 0.18881,
- "26": 0.18884,
- "27": 0.18993,
- "28": 0.1897,
- "29": 0.19097,
- "30": 0.1895,
- "31": 0.19115,
- "32": 0.18792,
- "33": 0.19346,
- "34": 0.19005,
- "35": 0.18315,
- "36": 0.18197,
- "37": 0.18748,
- "38": 0.18402,
- "39": 0.18451,
- "40": 0.1843,
- "41": 0.18427,
- "42": 0.18674,
- "43": 0.18376,
- "44": 0.18419,
- "45": 0.55191,
- "46": 0.18443,
- "47": 0.18303,
- "48": 0.18819,
- "49": 0.19592,
- "50": 0.1913,
- "51": 0.19759,
- "52": 0.19085,
- "53": 0.19262,
- "54": 0.19058,
- "55": 0.18897,
- "56": 0.1883,
- "57": 0.18757,
- "58": 0.18848,
- "59": 0.19004,
- "60": 0.18932,
- "61": 0.1889,
- "62": 0.18729,
- "63": 0.18757,
- "64": 0.18917,
- "65": 0.18796,
- "66": 0.1903,
- "67": 0.18985,
- "68": 0.18947,
- "69": 0.19134,
- "70": 0.19142,
- "71": 0.18328,
- "72": 0.18321,
- "73": 0.18529,
- "74": 0.18166,
- "75": 0.18265,
- "76": 0.18168,
- "77": 0.18263,
- "78": 0.18274,
- "79": 0.18238,
- "80": 0.18213,
- "81": 0.18186,
- "82": 0.1829,
- "83": 0.18266,
- "84": 0.18204,
- "85": 0.18191,
- "86": 0.18213,
- "87": 0.1812,
- "88": 0.18092,
- "89": 0.18123,
- "90": 0.22177,
- "91": 0.18593,
- "92": 0.18075,
- "93": 0.18389,
- "94": 0.18596,
- "95": 0.18215,
- "96": 0.18128,
- "97": 0.18129,
- "98": 0.18622,
- "99": 0.18532,
- "100": 0.18343
+ "1": "nan",
+ "2": 5.22136,
+ "3": 0.21036,
+ "4": 0.19136,
+ "5": 0.19257,
+ "6": 0.19454,
+ "7": 0.19743,
+ "8": 0.19647,
+ "9": 0.19646,
+ "10": 0.1984,
+ "11": 0.19638,
+ "12": 0.19654,
+ "13": 0.19584,
+ "14": 0.19577,
+ "15": 0.19632,
+ "16": 0.19557,
+ "17": 0.19601,
+ "18": 0.19585,
+ "19": 0.19712,
+ "20": 0.19608,
+ "21": 0.19595,
+ "22": 0.19576,
+ "23": 0.196,
+ "24": 0.1959,
+ "25": 0.19625,
+ "26": 0.1958,
+ "27": 0.19628,
+ "28": 0.19633,
+ "29": 0.19634,
+ "30": 0.19608,
+ "31": 0.19554,
+ "32": 0.19676,
+ "33": 0.19551,
+ "34": 0.19555,
+ "35": 0.19249,
+ "36": 0.19236,
+ "37": 0.19241,
+ "38": 0.19416,
+ "39": 0.19606,
+ "40": 0.1954,
+ "41": 0.19176,
+ "42": 0.19193,
+ "43": 0.19275,
+ "44": 0.19109,
+ "45": 0.1941,
+ "46": 0.19091,
+ "47": 0.19301,
+ "48": 0.19174,
+ "49": 0.19171,
+ "50": 0.1906,
+ "51": 0.22532,
+ "52": 0.19861,
+ "53": 0.19458,
+ "54": 0.19394,
+ "55": 0.1949,
+ "56": 0.19258,
+ "57": 0.19365,
+ "58": 0.19284,
+ "59": 0.19439,
+ "60": 0.19285,
+ "61": 0.19349,
+ "62": 0.19366,
+ "63": 0.1965,
+ "64": 0.19374,
+ "65": 0.19365,
+ "66": 0.19526,
+ "67": 0.19457,
+ "68": 0.1937,
+ "69": 0.19621,
+ "70": 0.19562,
+ "71": 0.19571,
+ "72": 0.19384,
+ "73": 0.19404,
+ "74": 0.19314,
+ "75": 0.19417,
+ "76": 0.19356,
+ "77": 0.19322,
+ "78": 0.19414,
+ "79": 0.19536,
+ "80": 0.19475,
+ "81": 0.19437,
+ "82": 0.19667,
+ "83": 0.19582,
+ "84": 0.19464,
+ "85": 0.19827,
+ "86": 0.19394,
+ "87": 0.19281,
+ "88": 0.1931,
+ "89": 0.19285,
+ "90": 0.1932,
+ "91": 0.19339,
+ "92": 0.19343,
+ "93": 0.19372,
+ "94": 0.19406,
+ "95": 0.19547,
+ "96": 0.19422,
+ "97": 0.31947,
+ "98": 0.1948,
+ "99": 0.19609,
+ "100": 0.19357
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
index 136c696ef2f..6cc70da70fc 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -53,4 +53,7 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --async-strategy: mcore
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode/golden_values_dev_dgx_h100.json
index 0b8045d999a..e04f9c744b2 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.84194,
- "2": 10.85713,
- "3": 10.84346,
- "4": 10.84202,
- "5": 10.85873,
- "6": 10.86412,
- "7": 10.851,
- "8": 10.84731,
- "9": 10.85736,
- "10": 10.81845,
- "11": 10.8595,
- "12": 10.84335,
- "13": 10.86446,
- "14": 10.85336,
- "15": 10.81222,
- "16": 10.81549,
- "17": 10.78956,
- "18": 10.79784,
- "19": 10.79279,
- "20": 10.71072,
- "21": 10.6971,
- "22": 10.58894,
- "23": 10.7072,
- "24": 10.60764,
- "25": 10.57461,
- "26": 10.6238,
- "27": 10.62036,
- "28": 10.567,
- "29": 10.57013,
- "30": 10.40091,
- "31": 10.17393,
- "32": 10.46119,
- "33": 10.45713,
- "34": 10.24672,
- "35": 10.28875,
- "36": 10.25284,
- "37": 10.3466,
- "38": 10.20914,
- "39": 10.39432,
- "40": 10.10167,
- "41": 10.159,
- "42": 10.21413,
- "43": 9.8848,
- "44": 9.98809,
- "45": 9.86955,
- "46": 9.84366,
- "47": 10.1377,
- "48": 9.87973,
- "49": 9.56916,
- "50": 9.91374,
- "51": 9.86379,
- "52": 9.75652,
- "53": 10.06157,
- "54": 9.96418,
- "55": 9.89204,
- "56": 9.63681,
- "57": 9.49807,
- "58": 9.83504,
- "59": 9.59701,
- "60": 9.51573,
- "61": 9.70155,
- "62": 9.97973,
- "63": 9.38914,
- "64": 9.77552,
- "65": 8.95939,
- "66": 9.6978,
- "67": 9.37174,
- "68": 9.78449,
- "69": 9.79058,
- "70": 9.74555,
- "71": 9.61867,
- "72": 9.58317,
- "73": 9.49175,
- "74": 8.939,
- "75": 9.41848,
- "76": 9.07237,
- "77": 10.06903,
- "78": 9.72443,
- "79": 9.3767,
- "80": 9.40261,
- "81": 9.47859,
- "82": 9.6984,
- "83": 9.30086,
- "84": 9.41299,
- "85": 9.61514,
- "86": 9.07881,
- "87": 9.59402,
- "88": 9.74658,
- "89": 9.60096,
- "90": 9.81999,
- "91": 9.32977,
- "92": 9.35625,
- "93": 9.07406,
- "94": 8.82774,
- "95": 9.51099,
- "96": 9.52501,
- "97": 9.3163,
- "98": 9.67278,
- "99": 8.88493,
- "100": 9.39984
+ "1": 10.89706,
+ "2": 10.88994,
+ "3": 10.8831,
+ "4": 10.88988,
+ "5": 10.88253,
+ "6": 10.8874,
+ "7": 10.88758,
+ "8": 10.88283,
+ "9": 10.88381,
+ "10": 10.88124,
+ "11": 10.8794,
+ "12": 10.87714,
+ "13": 10.87398,
+ "14": 10.86887,
+ "15": 10.84731,
+ "16": 10.83849,
+ "17": 10.83888,
+ "18": 10.8281,
+ "19": 10.83477,
+ "20": 10.76241,
+ "21": 10.74322,
+ "22": 10.72762,
+ "23": 10.72148,
+ "24": 10.70344,
+ "25": 10.69674,
+ "26": 10.6804,
+ "27": 10.65401,
+ "28": 10.57729,
+ "29": 10.55341,
+ "30": 10.52541,
+ "31": 10.52801,
+ "32": 10.51614,
+ "33": 10.47007,
+ "34": 10.44061,
+ "35": 10.44035,
+ "36": 10.42415,
+ "37": 10.39191,
+ "38": 10.39172,
+ "39": 10.35863,
+ "40": 10.34949,
+ "41": 10.32483,
+ "42": 10.30818,
+ "43": 10.28561,
+ "44": 10.26534,
+ "45": 10.26835,
+ "46": 10.23862,
+ "47": 10.22371,
+ "48": 10.17488,
+ "49": 10.18427,
+ "50": 10.17929,
+ "51": 10.18135,
+ "52": 10.13576,
+ "53": 10.13436,
+ "54": 10.10783,
+ "55": 10.0851,
+ "56": 10.10727,
+ "57": 10.1042,
+ "58": 10.11002,
+ "59": 10.04654,
+ "60": 10.0728,
+ "61": 10.02696,
+ "62": 10.0011,
+ "63": 10.0693,
+ "64": 10.02132,
+ "65": 9.98316,
+ "66": 10.02644,
+ "67": 10.00488,
+ "68": 9.96999,
+ "69": 9.98963,
+ "70": 9.96898,
+ "71": 9.99461,
+ "72": 9.98039,
+ "73": 9.96538,
+ "74": 9.95993,
+ "75": 9.92968,
+ "76": 9.95437,
+ "77": 9.95042,
+ "78": 9.90489,
+ "79": 9.9072,
+ "80": 9.92278,
+ "81": 9.94804,
+ "82": 9.88461,
+ "83": 9.84481,
+ "84": 9.78357,
+ "85": 9.77467,
+ "86": 9.87594,
+ "87": 9.90365,
+ "88": 9.87788,
+ "89": 9.82813,
+ "90": 9.81306,
+ "91": 9.82258,
+ "92": 9.81555,
+ "93": 9.75134,
+ "94": 9.82024,
+ "95": 9.80987,
+ "96": 9.79636,
+ "97": 9.74638,
+ "98": 9.76836,
+ "99": 9.82002,
+ "100": 9.70573
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1664.0,
- "2": 1707.0,
- "3": 1836.0,
- "4": 1861.0,
- "5": 2007.0,
- "6": 1868.0,
- "7": 1826.0,
- "8": 1697.0,
- "9": 1815.0,
- "10": 1469.0,
- "11": 1876.0,
- "12": 1879.0,
- "13": 1979.0,
- "14": 1902.0,
- "15": 1992.0,
- "16": 1988.0,
- "17": 1879.0,
- "18": 1802.0,
- "19": 1886.0,
- "20": 1767.0,
- "21": 1929.0,
- "22": 1714.0,
- "23": 2031.0,
- "24": 1685.0,
- "25": 1747.0,
- "26": 1811.0,
- "27": 1915.0,
- "28": 1929.0,
- "29": 2020.0,
- "30": 1936.0,
- "31": 1680.0,
- "32": 1878.0,
- "33": 2204.0,
- "34": 1888.0,
- "35": 1963.0,
- "36": 1928.0,
- "37": 2383.0,
- "38": 2177.0,
- "39": 2388.0,
- "40": 2274.0,
- "41": 2194.0,
- "42": 2167.0,
- "43": 1922.0,
- "44": 1978.0,
- "45": 2043.0,
- "46": 2112.0,
- "47": 2556.0,
- "48": 2251.0,
- "49": 2320.0,
- "50": 2278.0,
- "51": 2563.0,
- "52": 2431.0,
- "53": 2917.0,
- "54": 2655.0,
- "55": 2307.0,
- "56": 2605.0,
- "57": 2385.0,
- "58": 2952.0,
- "59": 2730.0,
- "60": 2287.0,
- "61": 2904.0,
- "62": 2601.0,
- "63": 2452.0,
- "64": 2810.0,
- "65": 2544.0,
- "66": 2914.0,
- "67": 2664.0,
- "68": 2709.0,
- "69": 2967.0,
- "70": 3049.0,
- "71": 2936.0,
- "72": 2410.0,
- "73": 2991.0,
- "74": 1882.0,
- "75": 2539.0,
- "76": 3060.0,
- "77": 3219.0,
- "78": 3023.0,
- "79": 3084.0,
- "80": 3101.0,
- "81": 3530.0,
- "82": 3298.0,
- "83": 2666.0,
- "84": 3154.0,
- "85": 3288.0,
- "86": 2827.0,
- "87": 3720.0,
- "88": 3168.0,
- "89": 3275.0,
- "90": 3168.0,
- "91": 2919.0,
- "92": 3071.0,
- "93": 2751.0,
- "94": 3412.0,
- "95": 3186.0,
- "96": 3429.0,
- "97": 3083.0,
- "98": 3477.0,
- "99": 3093.0,
- "100": 3212.0
+ "1": 1873.0,
+ "2": 1703.0,
+ "3": 1801.0,
+ "4": 1750.0,
+ "5": 1905.0,
+ "6": 1785.0,
+ "7": 2037.0,
+ "8": 1770.0,
+ "9": 1855.0,
+ "10": 1774.0,
+ "11": 1818.0,
+ "12": 1868.0,
+ "13": 1763.0,
+ "14": 1936.0,
+ "15": 1663.0,
+ "16": 1838.0,
+ "17": 1885.0,
+ "18": 1816.0,
+ "19": 1869.0,
+ "20": 1785.0,
+ "21": 1847.0,
+ "22": 1823.0,
+ "23": 1816.0,
+ "24": 1920.0,
+ "25": 1807.0,
+ "26": 1802.0,
+ "27": 1876.0,
+ "28": 1887.0,
+ "29": 1918.0,
+ "30": 1866.0,
+ "31": 1989.0,
+ "32": 2064.0,
+ "33": 1957.0,
+ "34": 2045.0,
+ "35": 2161.0,
+ "36": 2036.0,
+ "37": 2267.0,
+ "38": 2142.0,
+ "39": 2265.0,
+ "40": 2337.0,
+ "41": 2395.0,
+ "42": 2046.0,
+ "43": 2466.0,
+ "44": 2143.0,
+ "45": 2556.0,
+ "46": 2487.0,
+ "47": 2459.0,
+ "48": 2631.0,
+ "49": 2960.0,
+ "50": 2521.0,
+ "51": 2534.0,
+ "52": 2758.0,
+ "53": 2662.0,
+ "54": 2953.0,
+ "55": 2735.0,
+ "56": 2845.0,
+ "57": 2311.0,
+ "58": 3714.0,
+ "59": 3010.0,
+ "60": 2870.0,
+ "61": 2701.0,
+ "62": 3038.0,
+ "63": 3267.0,
+ "64": 3673.0,
+ "65": 2686.0,
+ "66": 3225.0,
+ "67": 3900.0,
+ "68": 3543.0,
+ "69": 3074.0,
+ "70": 3405.0,
+ "71": 3106.0,
+ "72": 2979.0,
+ "73": 3464.0,
+ "74": 3196.0,
+ "75": 3237.0,
+ "76": 3271.0,
+ "77": 3723.0,
+ "78": 3207.0,
+ "79": 3458.0,
+ "80": 3225.0,
+ "81": 3686.0,
+ "82": 2996.0,
+ "83": 3055.0,
+ "84": 2840.0,
+ "85": 2766.0,
+ "86": 3234.0,
+ "87": 2895.0,
+ "88": 3049.0,
+ "89": 3141.0,
+ "90": 3882.0,
+ "91": 2981.0,
+ "92": 2910.0,
+ "93": 3123.0,
+ "94": 3190.0,
+ "95": 3218.0,
+ "96": 3509.0,
+ "97": 3491.0,
+ "98": 3409.0,
+ "99": 3119.0,
+ "100": 3143.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 397748736.0,
- "2": 397748736.0,
- "3": 397748736.0,
- "4": 397748736.0,
- "5": 397748736.0,
- "6": 397748736.0,
- "7": 397748736.0,
- "8": 397748736.0,
- "9": 397748736.0,
- "10": 397748736.0,
- "11": 397748736.0,
- "12": 397748736.0,
- "13": 397748736.0,
- "14": 397748736.0,
- "15": 397748736.0,
- "16": 397748736.0,
- "17": 397748736.0,
- "18": 397748736.0,
- "19": 397748736.0,
- "20": 397748736.0,
- "21": 397748736.0,
- "22": 397748736.0,
- "23": 397748736.0,
- "24": 397748736.0,
- "25": 397748736.0,
- "26": 397748736.0,
- "27": 397748736.0,
- "28": 397748736.0,
- "29": 397748736.0,
- "30": 397748736.0,
- "31": 397748736.0,
- "32": 397748736.0,
- "33": 397748736.0,
- "34": 397748736.0,
- "35": 397748736.0,
- "36": 397748736.0,
- "37": 397748736.0,
- "38": 397748736.0,
- "39": 397748736.0,
- "40": 397748736.0,
- "41": 397748736.0,
- "42": 397748736.0,
- "43": 397748736.0,
- "44": 397748736.0,
- "45": 397748736.0,
- "46": 397748736.0,
- "47": 397748736.0,
- "48": 397748736.0,
- "49": 397748736.0,
- "50": 397748736.0,
- "51": 397748736.0,
- "52": 397748736.0,
- "53": 397748736.0,
- "54": 397748736.0,
- "55": 397748736.0,
- "56": 397748736.0,
- "57": 397748736.0,
- "58": 397748736.0,
- "59": 397748736.0,
- "60": 397748736.0,
- "61": 397748736.0,
- "62": 397748736.0,
- "63": 397748736.0,
- "64": 397748736.0,
- "65": 397748736.0,
- "66": 397748736.0,
- "67": 397748736.0,
- "68": 397748736.0,
- "69": 397748736.0,
- "70": 397748736.0,
- "71": 397748736.0,
- "72": 397748736.0,
- "73": 397748736.0,
- "74": 397748736.0,
- "75": 397748736.0,
- "76": 397748736.0,
- "77": 397748736.0,
- "78": 397748736.0,
- "79": 397748736.0,
- "80": 397748736.0,
- "81": 397748736.0,
- "82": 397748736.0,
- "83": 397748736.0,
- "84": 397748736.0,
- "85": 397748736.0,
- "86": 397748736.0,
- "87": 397748736.0,
- "88": 397748736.0,
- "89": 397748736.0,
- "90": 397748736.0,
- "91": 397748736.0,
- "92": 397748736.0,
- "93": 397748736.0,
- "94": 397748736.0,
- "95": 397748736.0,
- "96": 397748736.0,
- "97": 397748736.0,
- "98": 397748736.0,
- "99": 397748736.0,
- "100": 397748736.0
+ "1": 398273024.0,
+ "2": 398273024.0,
+ "3": 398273024.0,
+ "4": 398273024.0,
+ "5": 398273024.0,
+ "6": 398273024.0,
+ "7": 398273024.0,
+ "8": 398273024.0,
+ "9": 398273024.0,
+ "10": 398273024.0,
+ "11": 398273024.0,
+ "12": 398273024.0,
+ "13": 398273024.0,
+ "14": 398273024.0,
+ "15": 398273024.0,
+ "16": 398273024.0,
+ "17": 398273024.0,
+ "18": 398273024.0,
+ "19": 398273024.0,
+ "20": 398273024.0,
+ "21": 398273024.0,
+ "22": 398273024.0,
+ "23": 398273024.0,
+ "24": 398273024.0,
+ "25": 398273024.0,
+ "26": 398273024.0,
+ "27": 398273024.0,
+ "28": 398273024.0,
+ "29": 398273024.0,
+ "30": 398273024.0,
+ "31": 398273024.0,
+ "32": 398273024.0,
+ "33": 398273024.0,
+ "34": 398273024.0,
+ "35": 398273024.0,
+ "36": 398273024.0,
+ "37": 398273024.0,
+ "38": 398273024.0,
+ "39": 398273024.0,
+ "40": 398273024.0,
+ "41": 398273024.0,
+ "42": 398273024.0,
+ "43": 398273024.0,
+ "44": 398273024.0,
+ "45": 398273024.0,
+ "46": 398273024.0,
+ "47": 398273024.0,
+ "48": 398273024.0,
+ "49": 398273024.0,
+ "50": 398273024.0,
+ "51": 398273024.0,
+ "52": 398273024.0,
+ "53": 398273024.0,
+ "54": 398273024.0,
+ "55": 398273024.0,
+ "56": 398273024.0,
+ "57": 398273024.0,
+ "58": 398273024.0,
+ "59": 398273024.0,
+ "60": 398273024.0,
+ "61": 398273024.0,
+ "62": 398273024.0,
+ "63": 398273024.0,
+ "64": 398273024.0,
+ "65": 398273024.0,
+ "66": 398273024.0,
+ "67": 398273024.0,
+ "68": 398273024.0,
+ "69": 398273024.0,
+ "70": 398273024.0,
+ "71": 398273024.0,
+ "72": 398273024.0,
+ "73": 398273024.0,
+ "74": 398273024.0,
+ "75": 398273024.0,
+ "76": 398273024.0,
+ "77": 398273024.0,
+ "78": 398273024.0,
+ "79": 398273024.0,
+ "80": 398273024.0,
+ "81": 398273024.0,
+ "82": 398273024.0,
+ "83": 398273024.0,
+ "84": 398273024.0,
+ "85": 398273024.0,
+ "86": 398273024.0,
+ "87": 398273024.0,
+ "88": 398273024.0,
+ "89": 398273024.0,
+ "90": 398273024.0,
+ "91": 398273024.0,
+ "92": 398273024.0,
+ "93": 398273024.0,
+ "94": 398273024.0,
+ "95": 398273024.0,
+ "96": 398273024.0,
+ "97": 398273024.0,
+ "98": 398273024.0,
+ "99": 398273024.0,
+ "100": 398273024.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1044755968.0,
- "2": 1178234368.0,
- "3": 1178234368.0,
- "4": 1178234368.0,
- "5": 1178234368.0,
- "6": 1178234368.0,
- "7": 1178234368.0,
- "8": 1178234368.0,
- "9": 1178234368.0,
- "10": 1178234368.0,
- "11": 1178234368.0,
- "12": 1178234368.0,
- "13": 1178234368.0,
- "14": 1178234368.0,
- "15": 1178234368.0,
- "16": 1178234368.0,
- "17": 1178234368.0,
- "18": 1178234368.0,
- "19": 1178234368.0,
- "20": 1178234368.0,
- "21": 1178234368.0,
- "22": 1178234368.0,
- "23": 1178234368.0,
- "24": 1178234368.0,
- "25": 1178234368.0,
- "26": 1178234368.0,
- "27": 1178234368.0,
- "28": 1178234368.0,
- "29": 1178234368.0,
- "30": 1178234368.0,
- "31": 1178234368.0,
- "32": 1178234368.0,
- "33": 1178234368.0,
- "34": 1178234368.0,
- "35": 1178234368.0,
- "36": 1178234368.0,
- "37": 1178234368.0,
- "38": 1178234368.0,
- "39": 1178234368.0,
- "40": 1178234368.0,
- "41": 1178234368.0,
- "42": 1178234368.0,
- "43": 1178234368.0,
- "44": 1178234368.0,
- "45": 1178234368.0,
- "46": 1178234368.0,
- "47": 1178234368.0,
- "48": 1178234368.0,
- "49": 1178234368.0,
- "50": 1178234368.0,
- "51": 1178234368.0,
- "52": 1178234368.0,
- "53": 1178234368.0,
- "54": 1178234368.0,
- "55": 1178234368.0,
- "56": 1178234368.0,
- "57": 1178234368.0,
- "58": 1178234368.0,
- "59": 1178234368.0,
- "60": 1178234368.0,
- "61": 1178234368.0,
- "62": 1178234368.0,
- "63": 1178234368.0,
- "64": 1178234368.0,
- "65": 1178234368.0,
- "66": 1178234368.0,
- "67": 1178234368.0,
- "68": 1178234368.0,
- "69": 1178234368.0,
- "70": 1178234368.0,
- "71": 1178234368.0,
- "72": 1178234368.0,
- "73": 1178234368.0,
- "74": 1178234368.0,
- "75": 1178234368.0,
- "76": 1178234368.0,
- "77": 1178234368.0,
- "78": 1178234368.0,
- "79": 1178234368.0,
- "80": 1178234368.0,
- "81": 1178234368.0,
- "82": 1178234368.0,
- "83": 1178234368.0,
- "84": 1178234368.0,
- "85": 1178234368.0,
- "86": 1178234368.0,
- "87": 1178234368.0,
- "88": 1178234368.0,
- "89": 1178234368.0,
- "90": 1178234368.0,
- "91": 1178234368.0,
- "92": 1178234368.0,
- "93": 1178234368.0,
- "94": 1178234368.0,
- "95": 1178234368.0,
- "96": 1178234368.0,
- "97": 1178234368.0,
- "98": 1178234368.0,
- "99": 1178234368.0,
- "100": 1178234368.0
+ "1": 1043708416.0,
+ "2": 1177314304.0,
+ "3": 1177314304.0,
+ "4": 1177314304.0,
+ "5": 1177314304.0,
+ "6": 1177314304.0,
+ "7": 1177314304.0,
+ "8": 1177314304.0,
+ "9": 1177314304.0,
+ "10": 1177314304.0,
+ "11": 1177314304.0,
+ "12": 1177314304.0,
+ "13": 1177314304.0,
+ "14": 1177314304.0,
+ "15": 1177314304.0,
+ "16": 1177314304.0,
+ "17": 1177314304.0,
+ "18": 1177314304.0,
+ "19": 1177314304.0,
+ "20": 1177314304.0,
+ "21": 1177314304.0,
+ "22": 1177314304.0,
+ "23": 1177314304.0,
+ "24": 1177314304.0,
+ "25": 1177314304.0,
+ "26": 1177314304.0,
+ "27": 1177314304.0,
+ "28": 1177314304.0,
+ "29": 1177314304.0,
+ "30": 1177314304.0,
+ "31": 1177314304.0,
+ "32": 1177314304.0,
+ "33": 1177314304.0,
+ "34": 1177314304.0,
+ "35": 1177314304.0,
+ "36": 1177314304.0,
+ "37": 1177314304.0,
+ "38": 1177314304.0,
+ "39": 1177314304.0,
+ "40": 1177314304.0,
+ "41": 1177314304.0,
+ "42": 1177314304.0,
+ "43": 1177314304.0,
+ "44": 1177314304.0,
+ "45": 1177314304.0,
+ "46": 1177314304.0,
+ "47": 1177314304.0,
+ "48": 1177314304.0,
+ "49": 1177314304.0,
+ "50": 1177314304.0,
+ "51": 1177314304.0,
+ "52": 1177314304.0,
+ "53": 1177314304.0,
+ "54": 1177314304.0,
+ "55": 1177314304.0,
+ "56": 1177314304.0,
+ "57": 1177314304.0,
+ "58": 1177314304.0,
+ "59": 1177314304.0,
+ "60": 1177314304.0,
+ "61": 1177314304.0,
+ "62": 1177314304.0,
+ "63": 1177314304.0,
+ "64": 1177314304.0,
+ "65": 1177314304.0,
+ "66": 1177314304.0,
+ "67": 1177314304.0,
+ "68": 1177314304.0,
+ "69": 1177314304.0,
+ "70": 1177314304.0,
+ "71": 1177314304.0,
+ "72": 1177314304.0,
+ "73": 1177314304.0,
+ "74": 1177314304.0,
+ "75": 1177314304.0,
+ "76": 1177314304.0,
+ "77": 1177314304.0,
+ "78": 1177314304.0,
+ "79": 1177314304.0,
+ "80": 1177314304.0,
+ "81": 1177314304.0,
+ "82": 1177314304.0,
+ "83": 1177314304.0,
+ "84": 1177314304.0,
+ "85": 1177314304.0,
+ "86": 1177314304.0,
+ "87": 1177314304.0,
+ "88": 1177314304.0,
+ "89": 1177314304.0,
+ "90": 1177314304.0,
+ "91": 1177314304.0,
+ "92": 1177314304.0,
+ "93": 1177314304.0,
+ "94": 1177314304.0,
+ "95": 1177314304.0,
+ "96": 1177314304.0,
+ "97": 1177314304.0,
+ "98": 1177314304.0,
+ "99": 1177314304.0,
+ "100": 1177314304.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.36091,
- "2": 0.34885,
- "3": 0.28252,
- "4": 0.26078,
- "5": 0.25876,
- "6": 0.25718,
- "7": 0.26528,
- "8": 0.26311,
- "9": 0.26375,
- "10": 0.26354,
- "11": 0.26207,
- "12": 0.26033,
- "13": 0.26467,
- "14": 0.26281,
- "15": 0.26355,
- "16": 0.26138,
- "17": 0.2649,
- "18": 0.26631,
- "19": 0.26244,
- "20": 0.26263,
- "21": 0.26939,
- "22": 0.26538,
- "23": 0.26644,
- "24": 0.26284,
- "25": 0.26534,
- "26": 0.2629,
- "27": 0.2631,
- "28": 0.26216,
- "29": 0.26306,
- "30": 0.26559,
- "31": 0.26198,
- "32": 0.26229,
- "33": 0.26263,
- "34": 0.26154,
- "35": 0.26277,
- "36": 0.26291,
- "37": 0.26156,
- "38": 0.26052,
- "39": 0.26366,
- "40": 0.26065,
- "41": 0.26364,
- "42": 0.62325,
- "43": 0.26139,
- "44": 0.2631,
- "45": 0.26374,
- "46": 0.26054,
- "47": 0.26187,
- "48": 0.26188,
- "49": 0.25929,
- "50": 0.25984,
- "51": 0.26978,
- "52": 0.26013,
- "53": 0.26513,
- "54": 0.26111,
- "55": 0.26044,
- "56": 0.2624,
- "57": 0.26412,
- "58": 0.26108,
- "59": 0.26051,
- "60": 0.263,
- "61": 0.26363,
- "62": 0.27145,
- "63": 0.27074,
- "64": 0.26955,
- "65": 0.65636,
- "66": 0.26945,
- "67": 0.27333,
- "68": 0.27517,
- "69": 0.27206,
- "70": 0.27181,
- "71": 0.27216,
- "72": 0.9521,
- "73": 0.27086,
- "74": 0.27375,
- "75": 0.89877,
- "76": 0.27077,
- "77": 0.26534,
- "78": 0.2565,
- "79": 0.26961,
- "80": 0.26648,
- "81": 0.26175,
- "82": 0.26268,
- "83": 0.26668,
- "84": 0.26108,
- "85": 0.25906,
- "86": 0.25936,
- "87": 0.25961,
- "88": 0.25714,
- "89": 0.26171,
- "90": 0.26239,
- "91": 0.26137,
- "92": 0.25975,
- "93": 0.25965,
- "94": 0.2611,
- "95": 0.25793,
- "96": 0.26009,
- "97": 0.26077,
- "98": 0.25869,
- "99": 0.2601,
- "100": 0.25909
+ "1": "nan",
+ "2": 5.39088,
+ "3": 0.28263,
+ "4": 0.27043,
+ "5": 0.26899,
+ "6": 0.27001,
+ "7": 0.26823,
+ "8": 0.26864,
+ "9": 0.27106,
+ "10": 0.27192,
+ "11": 0.27032,
+ "12": 0.27145,
+ "13": 0.27181,
+ "14": 0.27234,
+ "15": 0.27168,
+ "16": 0.27299,
+ "17": 0.27097,
+ "18": 0.27305,
+ "19": 0.27069,
+ "20": 0.26799,
+ "21": 0.26928,
+ "22": 0.26816,
+ "23": 0.26743,
+ "24": 0.26738,
+ "25": 0.27134,
+ "26": 0.26815,
+ "27": 0.26816,
+ "28": 0.27302,
+ "29": 0.27273,
+ "30": 0.27046,
+ "31": 0.26912,
+ "32": 0.2736,
+ "33": 0.27172,
+ "34": 0.2729,
+ "35": 0.26959,
+ "36": 0.26722,
+ "37": 0.26798,
+ "38": 0.26955,
+ "39": 0.26682,
+ "40": 0.27129,
+ "41": 0.26994,
+ "42": 0.26731,
+ "43": 0.26883,
+ "44": 0.26677,
+ "45": 0.26563,
+ "46": 0.26683,
+ "47": 0.26766,
+ "48": 0.26618,
+ "49": 0.26777,
+ "50": 0.27024,
+ "51": 0.35354,
+ "52": 0.26754,
+ "53": 0.26414,
+ "54": 0.27097,
+ "55": 0.26722,
+ "56": 0.26448,
+ "57": 0.26508,
+ "58": 0.26694,
+ "59": 0.26574,
+ "60": 0.26484,
+ "61": 0.26551,
+ "62": 0.26688,
+ "63": 0.2722,
+ "64": 0.26609,
+ "65": 0.26554,
+ "66": 0.26655,
+ "67": 0.26506,
+ "68": 0.26565,
+ "69": 0.2664,
+ "70": 0.2652,
+ "71": 0.26717,
+ "72": 0.2668,
+ "73": 0.26405,
+ "74": 0.26548,
+ "75": 0.2648,
+ "76": 0.26345,
+ "77": 0.26476,
+ "78": 0.27292,
+ "79": 0.26449,
+ "80": 0.26566,
+ "81": 0.26497,
+ "82": 0.26741,
+ "83": 0.26862,
+ "84": 0.26323,
+ "85": 0.2674,
+ "86": 0.2654,
+ "87": 0.26412,
+ "88": 0.26674,
+ "89": 0.26631,
+ "90": 0.26614,
+ "91": 0.26485,
+ "92": 0.26414,
+ "93": 0.2653,
+ "94": 0.26347,
+ "95": 0.26487,
+ "96": 0.26609,
+ "97": 0.77479,
+ "98": 0.26714,
+ "99": 0.26494,
+ "100": 0.2694
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode/model_config.yaml
index 755e9ba49e9..addb335475f 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,6 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp2_frozen_resume_torch_dist_reshard_8x1xNone/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp2_frozen_resume_torch_dist_reshard_8x1xNone/model_config.yaml
index 4aa0b36a84b..2a4ce95cd6d 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp2_frozen_resume_torch_dist_reshard_8x1xNone/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp2_frozen_resume_torch_dist_reshard_8x1xNone/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -49,4 +49,5 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: frozen-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp2_resume_torch_dist_reshard_8x1xNone/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp2_resume_torch_dist_reshard_8x1xNone/model_config.yaml
index 620eeaeff46..881fe7ebe2d 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp2_resume_torch_dist_reshard_8x1xNone/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp2_resume_torch_dist_reshard_8x1xNone/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -50,4 +50,5 @@ MODEL_ARGS:
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
--log-memory-to-tensorboard: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp1_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp1_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
index b8a79c7a083..f28a2d05a5c 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp1_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp1_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,6 @@ MODEL_ARGS:
--bf16: true
--apply-query-key-layer-scaling: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp1_fsdp2_resume_torch_dist_te/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp1_fsdp2_resume_torch_dist_te/model_config.yaml
index 4febeeb3aca..c9eea9f5b0c 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp1_fsdp2_resume_torch_dist_te/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp1_fsdp2_resume_torch_dist_te/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -42,7 +42,6 @@ MODEL_ARGS:
--use-torch-fsdp2: true
--deterministic-mode: true
--no-gradient-accumulation-fusion: true
- --no-async-tensor-model-parallel-allreduce: true
--use-checkpoint-opt_param-scheduler: true
--use-mcore-models: true
--ckpt-format: torch_dist
@@ -52,4 +51,6 @@ MODEL_ARGS:
--fp16: true
--apply-query-key-layer-scaling: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
index 8793230c3c9..823312a21df 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -53,4 +53,6 @@ MODEL_ARGS:
--fp16: true
--apply-query-key-layer-scaling: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2/golden_values_dev_dgx_gb200.json
index 063c93b8168..f6f7f91fb03 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2/golden_values_dev_dgx_gb200.json
@@ -7,52 +7,52 @@
"1": 10.7702,
"2": 10.78031,
"3": 10.77782,
- "4": 10.73861,
- "5": 10.81197,
+ "4": 10.7386,
+ "5": 10.81196,
"6": 10.81962,
- "7": 10.79512,
- "8": 10.78158,
- "9": 10.79081,
- "10": 10.71741,
- "11": 10.85173,
- "12": 10.80653,
- "13": 10.82058,
- "14": 10.84404,
- "15": 10.74918,
- "16": 10.752,
- "17": 10.70902,
- "18": 10.752,
- "19": 10.74635,
+ "7": 10.79511,
+ "8": 10.7816,
+ "9": 10.79083,
+ "10": 10.71745,
+ "11": 10.85175,
+ "12": 10.80655,
+ "13": 10.82059,
+ "14": 10.84403,
+ "15": 10.74914,
+ "16": 10.75203,
+ "17": 10.70906,
+ "18": 10.75193,
+ "19": 10.74634,
"20": 10.63769,
- "21": 10.61672,
- "22": 10.44317,
- "23": 10.6675,
- "24": 10.50949,
- "25": 10.45557,
- "26": 10.53435,
- "27": 10.54753,
- "28": 10.51646,
+ "21": 10.61671,
+ "22": 10.4432,
+ "23": 10.66744,
+ "24": 10.50948,
+ "25": 10.45559,
+ "26": 10.53431,
+ "27": 10.54757,
+ "28": 10.51645,
"29": 10.55435,
- "30": 10.28785,
- "31": 10.00156,
+ "30": 10.28787,
+ "31": 10.00155,
"32": 10.40963,
- "33": 10.40243,
- "34": 10.13341,
- "35": 10.19694,
- "36": 10.14213,
- "37": 10.2869,
- "38": 10.10508,
- "39": 10.35217,
- "40": 10.00199,
- "41": 10.07363,
- "42": 10.1522,
- "43": 9.74558,
- "44": 9.8738,
- "45": 9.74764,
- "46": 9.74951,
+ "33": 10.40242,
+ "34": 10.13338,
+ "35": 10.19695,
+ "36": 10.1421,
+ "37": 10.28683,
+ "38": 10.10505,
+ "39": 10.35216,
+ "40": 10.00202,
+ "41": 10.07367,
+ "42": 10.15221,
+ "43": 9.74561,
+ "44": 9.87383,
+ "45": 9.74763,
+ "46": 9.74947,
"47": 10.09152,
- "48": 9.77892,
- "49": 9.44822,
+ "48": 9.77894,
+ "49": 9.44821,
"50": 9.84214
}
},
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1564.0,
- "2": 1610.0,
- "3": 1608.0,
- "4": 1854.0,
- "5": 1873.0,
- "6": 1812.0,
- "7": 1744.0,
- "8": 1614.0,
- "9": 1857.0,
- "10": 1358.0,
- "11": 1844.0,
- "12": 1788.0,
- "13": 1826.0,
- "14": 1801.0,
- "15": 1892.0,
- "16": 1892.0,
- "17": 1758.0,
- "18": 1714.0,
- "19": 1677.0,
- "20": 1582.0,
- "21": 1824.0,
- "22": 1579.0,
- "23": 1987.0,
- "24": 1533.0,
- "25": 1602.0,
- "26": 1651.0,
- "27": 1901.0,
- "28": 2044.0,
- "29": 1911.0,
- "30": 1823.0,
- "31": 1583.0,
- "32": 1926.0,
- "33": 2108.0,
- "34": 1914.0,
- "35": 2058.0,
- "36": 1946.0,
- "37": 2325.0,
- "38": 2268.0,
- "39": 2376.0,
- "40": 2208.0,
- "41": 2448.0,
- "42": 2209.0,
- "43": 1977.0,
- "44": 2049.0,
- "45": 2266.0,
- "46": 2481.0,
- "47": 2583.0,
- "48": 2450.0,
- "49": 2255.0,
- "50": 2453.0
+ "1": 1545.0,
+ "2": 1614.0,
+ "3": 1597.0,
+ "4": 1771.0,
+ "5": 1838.0,
+ "6": 1827.0,
+ "7": 1729.0,
+ "8": 1579.0,
+ "9": 1729.0,
+ "10": 1299.0,
+ "11": 1846.0,
+ "12": 1784.0,
+ "13": 1946.0,
+ "14": 1729.0,
+ "15": 1964.0,
+ "16": 1854.0,
+ "17": 1865.0,
+ "18": 1643.0,
+ "19": 1764.0,
+ "20": 1552.0,
+ "21": 1822.0,
+ "22": 1686.0,
+ "23": 2112.0,
+ "24": 1614.0,
+ "25": 1483.0,
+ "26": 1688.0,
+ "27": 1832.0,
+ "28": 1991.0,
+ "29": 2058.0,
+ "30": 1858.0,
+ "31": 1588.0,
+ "32": 2046.0,
+ "33": 2046.0,
+ "34": 1853.0,
+ "35": 1930.0,
+ "36": 1917.0,
+ "37": 2401.0,
+ "38": 2114.0,
+ "39": 2439.0,
+ "40": 2222.0,
+ "41": 2363.0,
+ "42": 2266.0,
+ "43": 1970.0,
+ "44": 2099.0,
+ "45": 2180.0,
+ "46": 2464.0,
+ "47": 2567.0,
+ "48": 2445.0,
+ "49": 2257.0,
+ "50": 2379.0
}
},
"mem-allocated-bytes": {
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 2.42188,
- "3": 0.13001,
- "4": 0.11595,
- "5": 0.1162,
- "6": 0.11616,
- "7": 0.11716,
- "8": 0.11796,
- "9": 0.1186,
- "10": 0.11774,
- "11": 0.11769,
- "12": 0.12298,
- "13": 0.11717,
- "14": 0.11738,
- "15": 0.11771,
- "16": 0.11772,
- "17": 0.11737,
- "18": 0.11751,
- "19": 0.11697,
- "20": 0.11822,
- "21": 0.11647,
- "22": 0.11724,
- "23": 0.11721,
- "24": 0.11677,
- "25": 0.1171,
- "26": 0.11775,
- "27": 0.11748,
- "28": 0.11705,
- "29": 0.11727,
- "30": 0.11693,
- "31": 0.11818,
- "32": 0.11738,
- "33": 0.11726,
- "34": 0.11675,
- "35": 0.11722,
- "36": 0.11753,
- "37": 0.11779,
- "38": 0.11683,
- "39": 0.11725,
- "40": 0.11779,
- "41": 0.11794,
- "42": 0.11724,
- "43": 0.11807,
- "44": 0.11744,
- "45": 0.12537,
- "46": 0.11752,
- "47": 0.11739,
- "48": 0.11765,
- "49": 0.11763,
- "50": 0.11812
+ "2": 5.51649,
+ "3": 0.13651,
+ "4": 0.12558,
+ "5": 0.12425,
+ "6": 0.12377,
+ "7": 0.12426,
+ "8": 0.12448,
+ "9": 0.12506,
+ "10": 0.1238,
+ "11": 0.12386,
+ "12": 0.12399,
+ "13": 0.12491,
+ "14": 0.12453,
+ "15": 0.12395,
+ "16": 0.12277,
+ "17": 0.12394,
+ "18": 0.12323,
+ "19": 0.12604,
+ "20": 0.12596,
+ "21": 0.12437,
+ "22": 0.12727,
+ "23": 0.12659,
+ "24": 0.12448,
+ "25": 0.12564,
+ "26": 0.12482,
+ "27": 0.12443,
+ "28": 0.12433,
+ "29": 0.12788,
+ "30": 0.12624,
+ "31": 0.12539,
+ "32": 0.12574,
+ "33": 0.12581,
+ "34": 0.1247,
+ "35": 0.12598,
+ "36": 0.12382,
+ "37": 0.1255,
+ "38": 0.12347,
+ "39": 0.12604,
+ "40": 0.1256,
+ "41": 0.1264,
+ "42": 0.12518,
+ "43": 0.12661,
+ "44": 0.12629,
+ "45": 0.12632,
+ "46": 0.12683,
+ "47": 0.12404,
+ "48": 0.12424,
+ "49": 0.12655,
+ "50": 0.12582
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2/model_config.yaml
index 4cc6e53b8c8..7ee44b85c81 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -49,4 +49,6 @@ MODEL_ARGS:
--bf16: true
--apply-query-key-layer-scaling: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2_fp16/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2_fp16/model_config.yaml
index f4c058fb0a0..8e68343c17e 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2_fp16/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2_fp16/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -48,4 +48,6 @@ MODEL_ARGS:
--fp16: true
--apply-query-key-layer-scaling: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2_resume_torch_dist/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2_resume_torch_dist/model_config.yaml
index e2a0f1f1f69..abe46384819 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2_resume_torch_dist/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp2_resume_torch_dist/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -49,4 +49,6 @@ MODEL_ARGS:
--fp16: true
--apply-query-key-layer-scaling: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp4/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp4/golden_values_dev_dgx_h100.json
index 4aa2800617e..b12d4c68133 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp4/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp4/golden_values_dev_dgx_h100.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.85678,
"2": 10.86405,
- "3": 10.86854,
- "4": 10.85128,
- "5": 10.88398,
+ "3": 10.86853,
+ "4": 10.8513,
+ "5": 10.884,
"6": 10.89024,
- "7": 10.86645,
- "8": 10.86924,
- "9": 10.87305,
- "10": 10.84079,
- "11": 10.87928,
- "12": 10.8729,
- "13": 10.8779,
- "14": 10.89011,
- "15": 10.82504,
+ "7": 10.86648,
+ "8": 10.86922,
+ "9": 10.87306,
+ "10": 10.84082,
+ "11": 10.87926,
+ "12": 10.87287,
+ "13": 10.87791,
+ "14": 10.89008,
+ "15": 10.82506,
"16": 10.82957,
- "17": 10.80875,
- "18": 10.81163,
- "19": 10.81545,
- "20": 10.71913,
- "21": 10.70404,
- "22": 10.56646,
- "23": 10.71861,
- "24": 10.60988,
- "25": 10.55482,
- "26": 10.60879,
- "27": 10.62303,
- "28": 10.56953,
- "29": 10.57966,
- "30": 10.35999,
- "31": 10.11305,
- "32": 10.46585,
- "33": 10.45153,
- "34": 10.20832,
- "35": 10.26936,
- "36": 10.21924,
- "37": 10.33851,
- "38": 10.18603,
- "39": 10.39977,
- "40": 10.08397,
- "41": 10.13423,
+ "17": 10.80876,
+ "18": 10.8116,
+ "19": 10.81547,
+ "20": 10.71915,
+ "21": 10.70406,
+ "22": 10.56647,
+ "23": 10.71857,
+ "24": 10.60989,
+ "25": 10.5548,
+ "26": 10.60875,
+ "27": 10.62301,
+ "28": 10.56951,
+ "29": 10.57968,
+ "30": 10.36,
+ "31": 10.1131,
+ "32": 10.46586,
+ "33": 10.45154,
+ "34": 10.20828,
+ "35": 10.26935,
+ "36": 10.21928,
+ "37": 10.3385,
+ "38": 10.18604,
+ "39": 10.39979,
+ "40": 10.08399,
+ "41": 10.13419,
"42": 10.20889,
"43": 9.82535,
- "44": 9.95909,
- "45": 9.82563,
- "46": 9.8062,
- "47": 10.135,
- "48": 9.84004,
- "49": 9.52485,
- "50": 9.90723
+ "44": 9.95908,
+ "45": 9.82559,
+ "46": 9.80623,
+ "47": 10.13498,
+ "48": 9.84003,
+ "49": 9.52483,
+ "50": 9.90726
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1654.0,
- "2": 1646.0,
- "3": 1565.0,
- "4": 1759.0,
- "5": 1860.0,
- "6": 1741.0,
- "7": 1752.0,
- "8": 1579.0,
- "9": 1849.0,
- "10": 1317.0,
- "11": 1901.0,
- "12": 1765.0,
- "13": 1910.0,
- "14": 1773.0,
- "15": 1864.0,
- "16": 1759.0,
- "17": 1794.0,
- "18": 1805.0,
- "19": 1846.0,
- "20": 1770.0,
- "21": 1963.0,
- "22": 1706.0,
- "23": 1983.0,
- "24": 1609.0,
- "25": 1593.0,
- "26": 1643.0,
- "27": 1696.0,
- "28": 1882.0,
- "29": 1946.0,
- "30": 1925.0,
- "31": 1574.0,
- "32": 1863.0,
- "33": 2024.0,
- "34": 1878.0,
- "35": 1941.0,
- "36": 1887.0,
- "37": 2294.0,
- "38": 2142.0,
- "39": 2288.0,
- "40": 2053.0,
- "41": 2189.0,
- "42": 2331.0,
- "43": 1933.0,
- "44": 2042.0,
- "45": 1956.0,
- "46": 2285.0,
- "47": 2470.0,
- "48": 2437.0,
- "49": 2238.0,
- "50": 2004.0
+ "1": 1608.0,
+ "2": 1684.0,
+ "3": 1618.0,
+ "4": 1745.0,
+ "5": 1847.0,
+ "6": 1758.0,
+ "7": 1838.0,
+ "8": 1585.0,
+ "9": 1818.0,
+ "10": 1346.0,
+ "11": 1847.0,
+ "12": 1686.0,
+ "13": 1839.0,
+ "14": 1742.0,
+ "15": 1780.0,
+ "16": 1799.0,
+ "17": 1771.0,
+ "18": 1747.0,
+ "19": 1831.0,
+ "20": 1749.0,
+ "21": 1887.0,
+ "22": 1714.0,
+ "23": 1971.0,
+ "24": 1688.0,
+ "25": 1579.0,
+ "26": 1640.0,
+ "27": 1836.0,
+ "28": 1879.0,
+ "29": 1995.0,
+ "30": 1862.0,
+ "31": 1500.0,
+ "32": 1775.0,
+ "33": 2088.0,
+ "34": 1804.0,
+ "35": 1967.0,
+ "36": 1862.0,
+ "37": 2231.0,
+ "38": 2082.0,
+ "39": 2290.0,
+ "40": 2123.0,
+ "41": 2206.0,
+ "42": 2166.0,
+ "43": 1985.0,
+ "44": 2111.0,
+ "45": 1937.0,
+ "46": 2111.0,
+ "47": 2429.0,
+ "48": 2301.0,
+ "49": 2278.0,
+ "50": 2040.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 777900032.0,
- "2": 777900032.0,
- "3": 777900032.0,
- "4": 777900032.0,
- "5": 777900032.0,
- "6": 777900032.0,
- "7": 777900032.0,
- "8": 777900032.0,
- "9": 777900032.0,
- "10": 777900032.0,
- "11": 777900032.0,
- "12": 777900032.0,
- "13": 777900032.0,
- "14": 777900032.0,
- "15": 777900032.0,
- "16": 777900032.0,
- "17": 777900032.0,
- "18": 777900032.0,
- "19": 777900032.0,
- "20": 777900032.0,
- "21": 777900032.0,
- "22": 777900032.0,
- "23": 777900032.0,
- "24": 777900032.0,
- "25": 777900032.0,
- "26": 777900032.0,
- "27": 777900032.0,
- "28": 777900032.0,
- "29": 777900032.0,
- "30": 777900032.0,
- "31": 777900032.0,
- "32": 777900032.0,
- "33": 777900032.0,
- "34": 777900032.0,
- "35": 777900032.0,
- "36": 777900032.0,
- "37": 777900032.0,
- "38": 777900032.0,
- "39": 777900032.0,
- "40": 777900032.0,
- "41": 777900032.0,
- "42": 777900032.0,
- "43": 777900032.0,
- "44": 777900032.0,
- "45": 777900032.0,
- "46": 777900032.0,
- "47": 777900032.0,
- "48": 777900032.0,
- "49": 777900032.0,
- "50": 777900032.0
+ "1": 778948608.0,
+ "2": 778948608.0,
+ "3": 778948608.0,
+ "4": 778948608.0,
+ "5": 778948608.0,
+ "6": 778948608.0,
+ "7": 778948608.0,
+ "8": 778948608.0,
+ "9": 778948608.0,
+ "10": 778948608.0,
+ "11": 778948608.0,
+ "12": 778948608.0,
+ "13": 778948608.0,
+ "14": 778948608.0,
+ "15": 778948608.0,
+ "16": 778948608.0,
+ "17": 778948608.0,
+ "18": 778948608.0,
+ "19": 778948608.0,
+ "20": 778948608.0,
+ "21": 778948608.0,
+ "22": 778948608.0,
+ "23": 778948608.0,
+ "24": 778948608.0,
+ "25": 778948608.0,
+ "26": 778948608.0,
+ "27": 778948608.0,
+ "28": 778948608.0,
+ "29": 778948608.0,
+ "30": 778948608.0,
+ "31": 778948608.0,
+ "32": 778948608.0,
+ "33": 778948608.0,
+ "34": 778948608.0,
+ "35": 778948608.0,
+ "36": 778948608.0,
+ "37": 778948608.0,
+ "38": 778948608.0,
+ "39": 778948608.0,
+ "40": 778948608.0,
+ "41": 778948608.0,
+ "42": 778948608.0,
+ "43": 778948608.0,
+ "44": 778948608.0,
+ "45": 778948608.0,
+ "46": 778948608.0,
+ "47": 778948608.0,
+ "48": 778948608.0,
+ "49": 778948608.0,
+ "50": 778948608.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 2463815680.0,
- "2": 2744478720.0,
- "3": 2744478720.0,
- "4": 2744478720.0,
- "5": 2744478720.0,
- "6": 2744478720.0,
- "7": 2744478720.0,
- "8": 2744478720.0,
- "9": 2744478720.0,
- "10": 2744478720.0,
- "11": 2744478720.0,
- "12": 2744478720.0,
- "13": 2744478720.0,
- "14": 2744478720.0,
- "15": 2744478720.0,
- "16": 2744478720.0,
- "17": 2744478720.0,
- "18": 2744478720.0,
- "19": 2744478720.0,
- "20": 2744478720.0,
- "21": 2744478720.0,
- "22": 2744478720.0,
- "23": 2744478720.0,
- "24": 2744478720.0,
- "25": 2744478720.0,
- "26": 2744478720.0,
- "27": 2744478720.0,
- "28": 2744478720.0,
- "29": 2744478720.0,
- "30": 2744478720.0,
- "31": 2744478720.0,
- "32": 2744478720.0,
- "33": 2744478720.0,
- "34": 2744478720.0,
- "35": 2744478720.0,
- "36": 2744478720.0,
- "37": 2744478720.0,
- "38": 2744478720.0,
- "39": 2744478720.0,
- "40": 2744478720.0,
- "41": 2744478720.0,
- "42": 2744478720.0,
- "43": 2744478720.0,
- "44": 2744478720.0,
- "45": 2744478720.0,
- "46": 2744478720.0,
- "47": 2744478720.0,
- "48": 2744478720.0,
- "49": 2744478720.0,
- "50": 2744478720.0
+ "1": 2462767104.0,
+ "2": 2746575872.0,
+ "3": 2746575872.0,
+ "4": 2746575872.0,
+ "5": 2746575872.0,
+ "6": 2746575872.0,
+ "7": 2746575872.0,
+ "8": 2746575872.0,
+ "9": 2746575872.0,
+ "10": 2746575872.0,
+ "11": 2746575872.0,
+ "12": 2746575872.0,
+ "13": 2746575872.0,
+ "14": 2746575872.0,
+ "15": 2746575872.0,
+ "16": 2746575872.0,
+ "17": 2746575872.0,
+ "18": 2746575872.0,
+ "19": 2746575872.0,
+ "20": 2746575872.0,
+ "21": 2746575872.0,
+ "22": 2746575872.0,
+ "23": 2746575872.0,
+ "24": 2746575872.0,
+ "25": 2746575872.0,
+ "26": 2746575872.0,
+ "27": 2746575872.0,
+ "28": 2746575872.0,
+ "29": 2746575872.0,
+ "30": 2746575872.0,
+ "31": 2746575872.0,
+ "32": 2746575872.0,
+ "33": 2746575872.0,
+ "34": 2746575872.0,
+ "35": 2746575872.0,
+ "36": 2746575872.0,
+ "37": 2746575872.0,
+ "38": 2746575872.0,
+ "39": 2746575872.0,
+ "40": 2746575872.0,
+ "41": 2746575872.0,
+ "42": 2746575872.0,
+ "43": 2746575872.0,
+ "44": 2746575872.0,
+ "45": 2746575872.0,
+ "46": 2746575872.0,
+ "47": 2746575872.0,
+ "48": 2746575872.0,
+ "49": 2746575872.0,
+ "50": 2746575872.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 7.818,
- "2": 0.14182,
- "3": 0.12081,
- "4": 0.09954,
- "5": 0.09861,
- "6": 0.10039,
- "7": 0.09846,
- "8": 0.09916,
- "9": 0.10232,
- "10": 0.10158,
- "11": 0.09888,
- "12": 0.09744,
- "13": 0.09991,
- "14": 0.09707,
- "15": 0.09748,
- "16": 0.09761,
- "17": 0.09792,
- "18": 0.09795,
- "19": 0.09792,
- "20": 0.09738,
- "21": 0.10014,
- "22": 0.09781,
- "23": 0.09834,
- "24": 0.09956,
- "25": 0.09768,
- "26": 0.09722,
- "27": 0.09836,
- "28": 0.09714,
- "29": 0.09695,
- "30": 0.09751,
- "31": 0.09809,
- "32": 0.09759,
- "33": 0.09764,
- "34": 0.09711,
- "35": 0.09791,
- "36": 0.09751,
- "37": 0.09778,
- "38": 0.09695,
- "39": 0.09907,
- "40": 0.09654,
- "41": 0.09746,
- "42": 0.09685,
- "43": 0.09736,
- "44": 0.09954,
- "45": 0.09768,
- "46": 0.09735,
- "47": 0.09905,
- "48": 0.09815,
- "49": 0.09684,
- "50": 0.09793
+ "1": "nan",
+ "2": 6.11405,
+ "3": 0.12147,
+ "4": 0.10402,
+ "5": 0.09689,
+ "6": 0.09621,
+ "7": 0.09601,
+ "8": 0.09683,
+ "9": 0.09798,
+ "10": 0.09721,
+ "11": 0.09607,
+ "12": 0.09586,
+ "13": 0.09658,
+ "14": 0.09806,
+ "15": 0.09742,
+ "16": 0.09593,
+ "17": 0.09607,
+ "18": 0.0965,
+ "19": 0.09647,
+ "20": 0.09635,
+ "21": 0.09631,
+ "22": 0.09646,
+ "23": 0.09779,
+ "24": 0.09709,
+ "25": 0.09739,
+ "26": 0.09629,
+ "27": 0.09658,
+ "28": 0.0966,
+ "29": 0.09641,
+ "30": 0.09681,
+ "31": 0.09671,
+ "32": 0.09692,
+ "33": 0.09737,
+ "34": 0.09716,
+ "35": 0.09694,
+ "36": 0.09598,
+ "37": 0.09623,
+ "38": 0.09571,
+ "39": 0.0999,
+ "40": 0.10205,
+ "41": 0.10186,
+ "42": 0.09612,
+ "43": 0.09678,
+ "44": 0.09781,
+ "45": 0.09572,
+ "46": 0.09747,
+ "47": 0.09574,
+ "48": 0.09602,
+ "49": 0.09759,
+ "50": 0.09631
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp4/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp4/model_config.yaml
index b9b786ee247..57db98fd87e 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp4/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp4/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -49,4 +49,6 @@ MODEL_ARGS:
--bf16: true
--apply-query-key-layer-scaling: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp4_resume_torch_dist/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp4_resume_torch_dist/model_config.yaml
index b4991e3621e..9bd0e50311d 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp4_resume_torch_dist/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp1_pp4_resume_torch_dist/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -50,4 +50,6 @@ MODEL_ARGS:
--fp16: true
--apply-query-key-layer-scaling: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp2_pp2_resume_torch_dist_uninstall_te/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp2_pp2_resume_torch_dist_uninstall_te/model_config.yaml
index cc6a76a97d9..a13ee609ace 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp2_pp2_resume_torch_dist_uninstall_te/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp2_pp2_resume_torch_dist_uninstall_te/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -51,4 +51,6 @@ MODEL_ARGS:
--fp16: true
--apply-query-key-layer-scaling: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp2_pp2_uninstall_te/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp2_pp2_uninstall_te/model_config.yaml
index 7601d0188ae..d44645ba5e3 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp2_pp2_uninstall_te/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp2_pp2_uninstall_te/model_config.yaml
@@ -25,9 +25,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -54,4 +54,6 @@ MODEL_ARGS:
--fp16: true
--apply-query-key-layer-scaling: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp4_pp1/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp4_pp1/model_config.yaml
index a365aae9089..abc889ac89e 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp4_pp1/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp4_pp1/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -48,4 +48,6 @@ MODEL_ARGS:
--bf16: true
--apply-query-key-layer-scaling: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp4_pp1_resume_torch/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp4_pp1_resume_torch/model_config.yaml
index c9473f99f96..e61ad0b9ea9 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp4_pp1_resume_torch/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp4_pp1_resume_torch/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp4_pp1_resume_torch_dist/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp4_pp1_resume_torch_dist/model_config.yaml
index 23b58cdc782..a46065b1121 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp4_pp1_resume_torch_dist/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp4_pp1_resume_torch_dist/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -49,4 +49,6 @@ MODEL_ARGS:
--fp16: true
--apply-query-key-layer-scaling: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/model_config.yaml
deleted file mode 100644
index f6892ae5c24..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/model_config.yaml
+++ /dev/null
@@ -1,58 +0,0 @@
-ENV_VARS:
- CUDA_DEVICE_MAX_CONNECTIONS: 32
- NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
- CUBLAS_WORKSPACE_CONFIG: :4096:8
- NCCL_ALGO: Ring
-MODEL_ARGS:
- --num-layers: 12
- --hidden-size: 512
- --num-attention-heads: 8
- --log-params-norm: true
- --log-num-zeros-in-grad: true
- --log-validation-ppl-to-tensorboard: true
- --log-timers-to-tensorboard: true
- --tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
- --seq-length: 1024
- --max-position-embeddings: 1024
- --train-iters: 2000
- --timing-log-level: 0
- --lr-decay-iters: 320000
- --save: ${CHECKPOINT_SAVE_PATH}
- --load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
- --split: 949,50,1
- --distributed-backend: nccl
- --lr: 0.00015
- --lr-decay-style: cosine
- --min-lr: 1.0e-5
- --weight-decay: 1e-2
- --clip-grad: 1.0
- --lr-warmup-fraction: .01
- --log-interval: 1
- --save-interval: 10000
- --eval-interval: 1000
- --eval-iters: 10
- --transformer-impl: transformer_engine
- --tensor-model-parallel-size: 2
- --pipeline-model-parallel-size: 2
- --sequence-parallel: true
- --tp-comm-overlap: true
- --tp-comm-overlap-cfg: tests/functional_tests/test_cases/gpt/gpt3_345m_weekly_dgx_b200_1N8G_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/tp_comm_overlap_cfg.yaml
- --deterministic-mode: true
- --no-gradient-accumulation-fusion: true
- --fp8-format: hybrid
- --fp8-recipe: tensorwise
- --fp8-param-gather: true
- --use-distributed-optimizer: true
- --attention-softmax-in-fp32: true
- --ckpt-format: torch_dist
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
- --data-cache-path: ${DATA_CACHE_PATH}
- --bf16: true
- --log-memory-to-tensorboard: true
- --normalization: RMSNorm
-TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp4_cp2_current_scaling_native_fp8_tp_fsdp/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp4_cp2_current_scaling_native_fp8_tp_fsdp/model_config.yaml
deleted file mode 100644
index 9c23cb7938f..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp4_cp2_current_scaling_native_fp8_tp_fsdp/model_config.yaml
+++ /dev/null
@@ -1,57 +0,0 @@
-ENV_VARS:
- CUDA_DEVICE_MAX_CONNECTIONS: 32
- NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
- CUBLAS_WORKSPACE_CONFIG: :4096:8
- NCCL_ALGO: Ring
-MODEL_ARGS:
- --num-layers: 12
- --hidden-size: 512
- --num-attention-heads: 8
- --log-params-norm: true
- --log-num-zeros-in-grad: true
- --log-validation-ppl-to-tensorboard: true
- --log-timers-to-tensorboard: true
- --tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
- --seq-length: 1024
- --max-position-embeddings: 1024
- --train-iters: 2000
- --timing-log-level: 0
- --lr-decay-iters: 320000
- --save: ${CHECKPOINT_SAVE_PATH}
- --load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
- --split: 949,50,1
- --distributed-backend: nccl
- --lr: 0.00015
- --lr-decay-style: cosine
- --min-lr: 1.0e-5
- --weight-decay: 1e-2
- --clip-grad: 1.0
- --lr-warmup-fraction: .01
- --log-interval: 1
- --save-interval: 10000
- --eval-interval: 1000
- --eval-iters: 10
- --transformer-impl: transformer_engine
- --tensor-model-parallel-size: 2
- --sequence-parallel: true
- --use-megatron-fsdp: true
- --data-parallel-sharding-strategy: optim_grads_params
- --no-gradient-accumulation-fusion: true
- --use-distributed-optimizer: true
- --deterministic-mode: true
- --fp8-format: hybrid
- --fp8-recipe: tensorwise
- --first-last-layers-bf16: true
- --fp8-param-gather: true
- --attention-softmax-in-fp32: true
- --ckpt-format: torch_dist
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
- --data-cache-path: ${DATA_CACHE_PATH}
- --bf16: true
- --log-memory-to-tensorboard: true
-TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml
deleted file mode 100644
index 4727007ffe2..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml
+++ /dev/null
@@ -1,57 +0,0 @@
-ENV_VARS:
- CUDA_DEVICE_MAX_CONNECTIONS: 32
- NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
- CUBLAS_WORKSPACE_CONFIG: :4096:8
- NCCL_ALGO: Ring
-MODEL_ARGS:
- --num-layers: 12
- --hidden-size: 512
- --num-attention-heads: 8
- --log-params-norm: true
- --log-num-zeros-in-grad: true
- --log-validation-ppl-to-tensorboard: true
- --log-timers-to-tensorboard: true
- --tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
- --seq-length: 1024
- --max-position-embeddings: 1024
- --train-iters: 2000
- --timing-log-level: 0
- --lr-decay-iters: 320000
- --save: ${CHECKPOINT_SAVE_PATH}
- --load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
- --split: 949,50,1
- --distributed-backend: nccl
- --lr: 0.00015
- --lr-decay-style: cosine
- --min-lr: 1.0e-5
- --weight-decay: 1e-2
- --clip-grad: 1.0
- --lr-warmup-fraction: .01
- --log-interval: 1
- --save-interval: 10000
- --eval-interval: 1000
- --eval-iters: 10
- --transformer-impl: transformer_engine
- --tensor-model-parallel-size: 4
- --context-parallel-size: 2
- --sequence-parallel: true
- --deterministic-mode: true
- --no-gradient-accumulation-fusion: true
- --fp8-format: hybrid
- --fp8-recipe: tensorwise
- --fp8-param-gather: true
- --use-distributed-optimizer: true
- --attention-softmax-in-fp32: true
- --ckpt-format: torch_dist
- --dist-ckpt-optim-fully-reshardable: true
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
- --data-cache-path: ${DATA_CACHE_PATH}
- --bf16: true
- --log-memory-to-tensorboard: true
- --normalization: RMSNorm
-TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp4_cp2_mxfp8_tp_sp_cp/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp4_cp2_mxfp8_tp_sp_cp/model_config.yaml
deleted file mode 100644
index bba1f1ad19e..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp4_cp2_mxfp8_tp_sp_cp/model_config.yaml
+++ /dev/null
@@ -1,54 +0,0 @@
-ENV_VARS:
- CUDA_DEVICE_MAX_CONNECTIONS: 32
- NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
- CUBLAS_WORKSPACE_CONFIG: :4096:8
- NCCL_ALGO: Ring
-MODEL_ARGS:
- --num-layers: 12
- --hidden-size: 512
- --num-attention-heads: 8
- --log-params-norm: true
- --log-num-zeros-in-grad: true
- --log-validation-ppl-to-tensorboard: true
- --log-timers-to-tensorboard: true
- --tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
- --seq-length: 1024
- --max-position-embeddings: 1024
- --train-iters: 2000
- --timing-log-level: 0
- --lr-decay-iters: 320000
- --save: ${CHECKPOINT_SAVE_PATH}
- --load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
- --split: 949,50,1
- --distributed-backend: nccl
- --lr: 0.00015
- --lr-decay-style: cosine
- --min-lr: 1.0e-5
- --weight-decay: 1e-2
- --clip-grad: 1.0
- --lr-warmup-fraction: .01
- --log-interval: 1
- --save-interval: 10000
- --eval-interval: 1000
- --eval-iters: 10
- --transformer-impl: transformer_engine
- --tensor-model-parallel-size: 4
- --context-parallel-size: 2
- --sequence-parallel: true
- --deterministic-mode: true
- --no-gradient-accumulation-fusion: true
- --fp8-format: hybrid
- --fp8-recipe: mxfp8
- --use-distributed-optimizer: true
- --attention-softmax-in-fp32: true
- --ckpt-format: torch_dist
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
- --data-cache-path: ${DATA_CACHE_PATH}
- --bf16: true
- --log-memory-to-tensorboard: true
-TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp4_cp2_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp4_cp2_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml
deleted file mode 100644
index 28f0a8a7b53..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp4_cp2_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml
+++ /dev/null
@@ -1,55 +0,0 @@
-ENV_VARS:
- CUDA_DEVICE_MAX_CONNECTIONS: 32
- CUBLAS_WORKSPACE_CONFIG: :4096:8
- NCCL_ALGO: Ring
-MODEL_ARGS:
- --num-layers: 12
- --hidden-size: 512
- --num-attention-heads: 8
- --log-params-norm: true
- --log-num-zeros-in-grad: true
- --log-validation-ppl-to-tensorboard: true
- --log-timers-to-tensorboard: true
- --tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
- --seq-length: 1024
- --max-position-embeddings: 1024
- --train-iters: 2000
- --timing-log-level: 0
- --lr-decay-iters: 320000
- --save: ${CHECKPOINT_SAVE_PATH}
- --load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
- --split: 949,50,1
- --distributed-backend: nccl
- --lr: 0.00015
- --lr-decay-style: cosine
- --min-lr: 1.0e-5
- --weight-decay: 1e-2
- --clip-grad: 1.0
- --lr-warmup-fraction: .01
- --log-interval: 1
- --save-interval: 10000
- --eval-interval: 1000
- --eval-iters: 10
- --transformer-impl: transformer_engine
- --tensor-model-parallel-size: 4
- --context-parallel-size: 2
- --sequence-parallel: true
- --deterministic-mode: true
- --no-gradient-accumulation-fusion: true
- --fp8-format: hybrid
- --fp8-amax-history-len: 1024
- --fp8-amax-compute-algo: max
- --fp8-param-gather: true
- --use-distributed-optimizer: true
- --attention-softmax-in-fp32: true
- --ckpt-format: torch_dist
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
- --data-cache-path: ${DATA_CACHE_PATH}
- --bf16: true
- --log-memory-to-tensorboard: true
-TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_nondet_tp1_pp1_fp8_no_model_parallel/golden_values_dev_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_nondet_tp1_pp1_fp8_no_model_parallel/golden_values_dev_dgx_a100.json
deleted file mode 100644
index c759ae47565..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_nondet_tp1_pp1_fp8_no_model_parallel/golden_values_dev_dgx_a100.json
+++ /dev/null
@@ -1,1223 +0,0 @@
-{
- "lm loss": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 10.8833,
- 10.90244,
- 10.88662,
- 10.83318,
- 10.6762,
- 10.64934,
- 10.43397,
- 10.15132,
- 9.93913,
- 9.84134,
- 9.5886,
- 9.85452,
- 9.88457,
- 9.62953,
- 9.78805,
- 9.51138,
- 9.45839,
- 9.64923,
- 9.38614,
- 9.33215,
- 9.24219,
- 9.14557,
- 9.17566,
- 8.99559,
- 9.18951,
- 9.06004,
- 9.15559,
- 9.16505,
- 9.29785,
- 8.9846,
- 8.92921,
- 9.04387,
- 9.04308,
- 8.65511,
- 8.71722,
- 8.75347,
- 8.68373,
- 8.73448,
- 8.65881,
- 8.76509,
- 8.66102,
- 8.85001,
- 8.83242,
- 8.49967,
- 8.3894,
- 8.43185,
- 8.49362,
- 8.38492,
- 8.43303,
- 8.58006,
- 8.36747,
- 8.19262,
- 8.22634,
- 8.22256,
- 8.26796,
- 7.91388,
- 8.09614,
- 7.89146,
- 8.2469,
- 8.23091,
- 8.00558,
- 7.96607,
- 7.91878,
- 7.74064,
- 7.74043,
- 7.64353,
- 7.51615,
- 7.90743,
- 7.69899,
- 7.45239,
- 7.74097,
- 7.76829,
- 7.54181,
- 7.29901,
- 7.45239,
- 7.33607,
- 7.46255,
- 7.22408,
- 7.63701,
- 7.27971,
- 7.35197,
- 7.21312,
- 7.21651,
- 7.42255,
- 7.17701,
- 7.28049,
- 7.00057,
- 7.00362,
- 7.0382,
- 7.13584,
- 6.82274,
- 6.98508,
- 7.08808,
- 7.00046,
- 6.87376,
- 6.75595,
- 6.99172,
- 7.05761,
- 6.70449,
- 6.5819,
- 6.72818,
- 6.74414,
- 6.73568,
- 6.74025,
- 6.65976,
- 6.4086,
- 6.64092,
- 6.621,
- 6.44769,
- 6.63067,
- 6.74419,
- 6.61028,
- 6.72574,
- 6.69594,
- 6.62546,
- 6.50829,
- 6.60018,
- 6.40775,
- 6.66564,
- 6.25029,
- 6.2517,
- 6.30277,
- 6.39006,
- 6.34934,
- 6.45014,
- 6.29146,
- 6.34189,
- 6.23672,
- 6.20135,
- 6.39859,
- 6.32501,
- 6.32243,
- 6.16493,
- 6.15827,
- 6.23907,
- 6.38353,
- 6.19887,
- 6.14407,
- 6.17562,
- 6.10888,
- 6.05387,
- 6.06583,
- 6.25304,
- 6.40434,
- 6.25162,
- 6.29199,
- 6.09114,
- 6.17247,
- 5.99466,
- 6.02134,
- 5.95061,
- 6.23865,
- 6.17959,
- 5.95837,
- 5.77693,
- 6.11779,
- 5.84072,
- 6.09813,
- 5.78476,
- 6.15517,
- 6.14253,
- 6.08389,
- 5.92776,
- 6.11285,
- 5.94312,
- 6.19361,
- 5.89575,
- 5.79177,
- 5.77658,
- 5.68463,
- 6.01517,
- 5.99439,
- 6.06379,
- 5.88864,
- 6.03938,
- 5.96752,
- 5.99173,
- 5.98642,
- 5.94693,
- 5.83816,
- 5.95021,
- 5.61696,
- 5.69931,
- 5.88617,
- 5.8418,
- 5.85952,
- 5.76089,
- 5.83643,
- 5.72472,
- 5.55795,
- 5.72279,
- 5.62456,
- 5.83384,
- 5.60371,
- 5.70964,
- 5.71305,
- 5.90077,
- 5.64296,
- 5.84721,
- 5.73799,
- 5.87065,
- 5.32845,
- 5.89503,
- 5.87432,
- 5.85262,
- 5.4122,
- 5.40753,
- 5.6225,
- 5.59374,
- 5.48037,
- 5.56952,
- 5.67164,
- 5.474,
- 5.74128,
- 5.50855,
- 5.59254,
- 5.62042,
- 5.6173,
- 5.50903,
- 5.61307,
- 5.6694,
- 5.68176,
- 5.58253,
- 5.66074,
- 5.37239,
- 5.67835,
- 5.62699,
- 5.41742,
- 5.58719,
- 5.62981,
- 5.55162,
- 5.33784,
- 5.53833,
- 5.48177,
- 5.48342,
- 5.37902,
- 5.55461,
- 5.60113,
- 5.38725,
- 5.52265,
- 5.48637,
- 5.32902,
- 5.50379,
- 5.40804,
- 5.44024,
- 5.31412,
- 5.06315,
- 5.47637,
- 5.56625,
- 5.71066,
- 5.41144,
- 5.59641,
- 5.6328,
- 5.23123,
- 5.27182,
- 5.39253,
- 5.39442,
- 5.32567,
- 5.49583,
- 5.18092,
- 5.2993,
- 5.24857,
- 5.37717,
- 5.25715,
- 5.44127,
- 5.53765,
- 5.3134,
- 5.43978,
- 5.33655,
- 5.07222,
- 5.31412,
- 5.25439,
- 5.30253,
- 5.10951,
- 5.27338,
- 5.26801,
- 5.47298,
- 5.15965,
- 5.26921,
- 5.20696,
- 5.35595,
- 4.98275,
- 4.91391,
- 5.32139,
- 5.38782,
- 5.22672,
- 5.31644,
- 5.10423,
- 5.15896,
- 5.26163,
- 5.06463,
- 5.26136,
- 5.07195,
- 5.33749,
- 5.24642,
- 5.14987,
- 5.23852,
- 5.03778,
- 5.31313,
- 5.04992,
- 5.02354,
- 5.14081,
- 5.10984,
- 5.26921,
- 5.14803,
- 5.27454,
- 5.09393,
- 5.09412,
- 5.24833,
- 5.31694,
- 5.25175,
- 5.18843,
- 5.14133,
- 5.28374,
- 4.94582,
- 5.20544,
- 5.08881,
- 5.30053,
- 5.17192,
- 5.18279,
- 5.11003,
- 4.98355,
- 4.99209,
- 5.21882,
- 5.30942,
- 5.09283,
- 5.05041,
- 4.91204,
- 5.11771,
- 5.1167,
- 4.92322,
- 5.33275,
- 5.01952,
- 5.10011,
- 5.15937,
- 5.00254,
- 5.05909,
- 5.06306,
- 4.98904,
- 5.07423,
- 5.15838,
- 4.97483,
- 5.17683,
- 4.92747,
- 4.91596,
- 5.06215,
- 4.99131,
- 4.90548,
- 4.76895,
- 4.93875,
- 5.1077,
- 5.01313,
- 5.01358,
- 5.32429,
- 4.95302,
- 4.99177,
- 5.03879,
- 4.79987,
- 4.73503,
- 4.9917,
- 5.03536,
- 4.87166,
- 4.9475,
- 5.03845,
- 5.01972,
- 4.80886,
- 4.88618,
- 4.89985,
- 4.82715,
- 4.74128,
- 5.00393,
- 4.74546,
- 5.20303,
- 4.77871,
- 4.98658,
- 4.73073,
- 4.78023,
- 4.81501,
- 4.64456,
- 4.65279,
- 4.83952,
- 4.80146,
- 4.79663,
- 4.91833,
- 4.87809,
- 4.91911,
- 4.76246,
- 4.87827,
- 4.72709,
- 4.90772,
- 4.95311,
- 4.86859,
- 4.70331,
- 4.77605,
- 4.89682,
- 4.70384,
- 4.8551,
- 4.68524,
- 4.68185,
- 4.64443
- ]
- },
- "num-zeros": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 86.0,
- 97.0,
- 77.0,
- 63.0,
- 77.0,
- 73.0,
- 117.0,
- 81.0,
- 106.0,
- 103.0,
- 117.0,
- 150.0,
- 140.0,
- 168.0,
- 169.0,
- 177.0,
- 200.0,
- 196.0,
- 202.0,
- 184.0,
- 166.0,
- 177.0,
- 199.0,
- 168.0,
- 193.0,
- 149.0,
- 175.0,
- 178.0,
- 154.0,
- 158.0,
- 159.0,
- 148.0,
- 142.0,
- 183.0,
- 168.0,
- 167.0,
- 171.0,
- 215.0,
- 165.0,
- 183.0,
- 195.0,
- 168.0,
- 143.0,
- 185.0,
- 201.0,
- 162.0,
- 190.0,
- 207.0,
- 174.0,
- 224.0,
- 217.0,
- 159.0,
- 191.0,
- 169.0,
- 196.0,
- 212.0,
- 174.0,
- 143.0,
- 219.0,
- 232.0,
- 180.0,
- 220.0,
- 234.0,
- 169.0,
- 214.0,
- 259.0,
- 218.0,
- 212.0,
- 232.0,
- 207.0,
- 251.0,
- 250.0,
- 161.0,
- 235.0,
- 207.0,
- 186.0,
- 261.0,
- 191.0,
- 267.0,
- 228.0,
- 253.0,
- 229.0,
- 221.0,
- 235.0,
- 216.0,
- 201.0,
- 207.0,
- 215.0,
- 210.0,
- 223.0,
- 178.0,
- 229.0,
- 241.0,
- 206.0,
- 211.0,
- 157.0,
- 218.0,
- 221.0,
- 199.0,
- 158.0,
- 167.0,
- 178.0,
- 168.0,
- 188.0,
- 165.0,
- 158.0,
- 158.0,
- 158.0,
- 137.0,
- 193.0,
- 185.0,
- 148.0,
- 165.0,
- 158.0,
- 174.0,
- 137.0,
- 167.0,
- 119.0,
- 185.0,
- 167.0,
- 162.0,
- 123.0,
- 145.0,
- 161.0,
- 113.0,
- 131.0,
- 94.0,
- 139.0,
- 133.0,
- 137.0,
- 170.0,
- 126.0,
- 144.0,
- 127.0,
- 120.0,
- 127.0,
- 152.0,
- 137.0,
- 133.0,
- 134.0,
- 162.0,
- 137.0,
- 95.0,
- 150.0,
- 133.0,
- 144.0,
- 147.0,
- 141.0,
- 136.0,
- 125.0,
- 103.0,
- 115.0,
- 97.0,
- 111.0,
- 111.0,
- 89.0,
- 110.0,
- 117.0,
- 107.0,
- 127.0,
- 110.0,
- 116.0,
- 116.0,
- 136.0,
- 103.0,
- 99.0,
- 111.0,
- 124.0,
- 105.0,
- 109.0,
- 103.0,
- 118.0,
- 109.0,
- 95.0,
- 118.0,
- 144.0,
- 93.0,
- 108.0,
- 100.0,
- 121.0,
- 108.0,
- 96.0,
- 106.0,
- 144.0,
- 125.0,
- 122.0,
- 93.0,
- 114.0,
- 101.0,
- 127.0,
- 107.0,
- 126.0,
- 102.0,
- 100.0,
- 98.0,
- 112.0,
- 103.0,
- 116.0,
- 134.0,
- 94.0,
- 126.0,
- 118.0,
- 118.0,
- 100.0,
- 123.0,
- 106.0,
- 105.0,
- 83.0,
- 111.0,
- 102.0,
- 108.0,
- 110.0,
- 100.0,
- 115.0,
- 103.0,
- 98.0,
- 107.0,
- 102.0,
- 99.0,
- 106.0,
- 130.0,
- 126.0,
- 127.0,
- 90.0,
- 98.0,
- 90.0,
- 117.0,
- 119.0,
- 100.0,
- 96.0,
- 121.0,
- 101.0,
- 99.0,
- 111.0,
- 105.0,
- 91.0,
- 103.0,
- 94.0,
- 110.0,
- 90.0,
- 110.0,
- 109.0,
- 95.0,
- 98.0,
- 100.0,
- 109.0,
- 98.0,
- 128.0,
- 109.0,
- 99.0,
- 103.0,
- 99.0,
- 114.0,
- 98.0,
- 110.0,
- 85.0,
- 97.0,
- 142.0,
- 90.0,
- 117.0,
- 83.0,
- 107.0,
- 104.0,
- 102.0,
- 105.0,
- 99.0,
- 104.0,
- 88.0,
- 101.0,
- 107.0,
- 108.0,
- 99.0,
- 104.0,
- 108.0,
- 105.0,
- 97.0,
- 101.0,
- 108.0,
- 110.0,
- 114.0,
- 116.0,
- 100.0,
- 108.0,
- 111.0,
- 134.0,
- 97.0,
- 109.0,
- 106.0,
- 114.0,
- 85.0,
- 117.0,
- 114.0,
- 103.0,
- 123.0,
- 95.0,
- 88.0,
- 89.0,
- 101.0,
- 120.0,
- 116.0,
- 127.0,
- 98.0,
- 130.0,
- 118.0,
- 103.0,
- 120.0,
- 93.0,
- 101.0,
- 125.0,
- 102.0,
- 110.0,
- 119.0,
- 101.0,
- 88.0,
- 127.0,
- 103.0,
- 120.0,
- 121.0,
- 112.0,
- 136.0,
- 126.0,
- 101.0,
- 111.0,
- 114.0,
- 103.0,
- 105.0,
- 109.0,
- 116.0,
- 111.0,
- 108.0,
- 109.0,
- 105.0,
- 117.0,
- 95.0,
- 112.0,
- 116.0,
- 118.0,
- 121.0,
- 109.0,
- 107.0,
- 97.0,
- 101.0,
- 110.0,
- 96.0,
- 88.0,
- 130.0,
- 104.0,
- 116.0,
- 141.0,
- 110.0,
- 126.0,
- 111.0,
- 120.0,
- 115.0,
- 132.0,
- 101.0,
- 132.0,
- 103.0,
- 87.0,
- 123.0,
- 101.0,
- 96.0,
- 101.0,
- 113.0,
- 107.0,
- 121.0,
- 116.0,
- 113.0,
- 95.0,
- 99.0,
- 104.0,
- 112.0,
- 90.0,
- 108.0,
- 103.0,
- 117.0,
- 106.0,
- 114.0,
- 126.0,
- 113.0,
- 90.0,
- 114.0,
- 113.0,
- 140.0,
- 112.0,
- 115.0,
- 125.0,
- 122.0,
- 122.0,
- 121.0,
- 108.0,
- 123.0,
- 98.0,
- 122.0,
- 112.0,
- 114.0,
- 136.0,
- 135.0,
- 124.0,
- 127.0
- ]
- },
- "iteration-time": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 9.33072,
- 0.37969,
- 0.3867,
- 0.39046,
- 0.71873,
- 0.38256,
- 0.37315,
- 0.37524,
- 0.36944,
- 0.37312,
- 0.37427,
- 0.37609,
- 0.37691,
- 0.37378,
- 0.3748,
- 0.37171,
- 0.37454,
- 0.37374,
- 0.36874,
- 0.3752,
- 0.3711,
- 0.37096,
- 0.37248,
- 0.36855,
- 0.37987,
- 0.38237,
- 0.37301,
- 0.37064,
- 0.37284,
- 0.37218,
- 0.36973,
- 0.36736,
- 0.36966,
- 0.37499,
- 0.37066,
- 0.37764,
- 0.37572,
- 0.37094,
- 0.37367,
- 0.37253,
- 0.37593,
- 0.37116,
- 0.3711,
- 0.37778,
- 0.37155,
- 0.37085,
- 0.36952,
- 0.37508,
- 0.37548,
- 0.38095,
- 0.37291,
- 0.37154,
- 0.37099,
- 0.36927,
- 0.3727,
- 0.37748,
- 0.37423,
- 0.38161,
- 0.37206,
- 0.37582,
- 0.3751,
- 0.37521,
- 0.37579,
- 0.3843,
- 0.38471,
- 0.39343,
- 0.38245,
- 0.37202,
- 0.37512,
- 0.37457,
- 0.3767,
- 0.3809,
- 0.37685,
- 0.37794,
- 0.37766,
- 0.37182,
- 0.37032,
- 0.36853,
- 0.37837,
- 0.38023,
- 0.37444,
- 0.37133,
- 0.37618,
- 0.37766,
- 0.37506,
- 0.37632,
- 0.3801,
- 0.37886,
- 0.37663,
- 0.36943,
- 0.36983,
- 0.3715,
- 0.36856,
- 0.36971,
- 0.37105,
- 0.36821,
- 0.36936,
- 0.37346,
- 0.41784,
- 0.37673,
- 0.37144,
- 0.37071,
- 0.37031,
- 0.37298,
- 0.37588,
- 0.3756,
- 0.37347,
- 0.38242,
- 0.37911,
- 0.54764,
- 0.37973,
- 0.38156,
- 0.39236,
- 0.37822,
- 0.3697,
- 0.37285,
- 0.38125,
- 0.38209,
- 0.37865,
- 0.38072,
- 0.38122,
- 0.37986,
- 0.38034,
- 0.37981,
- 0.38328,
- 0.37807,
- 0.38055,
- 0.3832,
- 0.36995,
- 0.38206,
- 0.38372,
- 0.38567,
- 0.3812,
- 0.38005,
- 0.38254,
- 0.38244,
- 0.38168,
- 0.38118,
- 0.38283,
- 0.38472,
- 0.3835,
- 0.38063,
- 0.38557,
- 0.3843,
- 0.38091,
- 0.38202,
- 0.38245,
- 0.38516,
- 0.37498,
- 0.3723,
- 0.37436,
- 0.37103,
- 0.3695,
- 0.37203,
- 0.37519,
- 0.54118,
- 0.37475,
- 0.37358,
- 0.37411,
- 0.37405,
- 0.37456,
- 0.3745,
- 0.37136,
- 0.37621,
- 0.37202,
- 0.373,
- 0.37397,
- 0.37221,
- 0.37845,
- 0.37294,
- 0.37833,
- 0.37992,
- 0.37911,
- 0.37803,
- 0.37925,
- 0.37985,
- 0.3727,
- 0.37901,
- 0.37373,
- 0.37542,
- 0.37778,
- 0.37402,
- 0.37537,
- 0.37345,
- 0.37323,
- 0.3796,
- 0.37226,
- 0.37563,
- 0.37458,
- 0.37784,
- 0.37195,
- 0.37503,
- 0.3753,
- 0.54991,
- 0.3707,
- 0.37072,
- 0.36734,
- 0.37155,
- 0.37337,
- 0.37254,
- 0.37077,
- 0.37423,
- 0.37483,
- 0.37004,
- 0.37069,
- 0.37081,
- 0.37165,
- 0.37034,
- 0.37015,
- 0.37095,
- 0.37197,
- 0.37337,
- 0.40008,
- 0.37329,
- 0.37851,
- 0.374,
- 0.37858,
- 0.37453,
- 0.37638,
- 0.37597,
- 0.37286,
- 0.38096,
- 0.37707,
- 0.37106,
- 0.37352,
- 0.37279,
- 0.37524,
- 0.37497,
- 0.41076,
- 0.36917,
- 0.37087,
- 0.37171,
- 0.37311,
- 0.37307,
- 0.36955,
- 0.36813,
- 0.36729,
- 0.38713,
- 0.37491,
- 0.37489,
- 0.37253,
- 0.37112,
- 0.37728,
- 0.36993,
- 0.37452,
- 0.37127,
- 0.37009,
- 0.37711,
- 0.37699,
- 0.37589,
- 0.37554,
- 0.37267,
- 0.3819,
- 0.37774,
- 0.37236,
- 0.3769,
- 0.37198,
- 0.37151,
- 0.36707,
- 0.37125,
- 0.37855,
- 0.37806,
- 0.37014,
- 0.37031,
- 0.37164,
- 0.37899,
- 0.37467,
- 0.37348,
- 0.38182,
- 0.37435,
- 0.3806,
- 0.37719,
- 0.37638,
- 0.37477,
- 0.37237,
- 0.37865,
- 0.3711,
- 0.37491,
- 0.37158,
- 0.37482,
- 0.3744,
- 0.37558,
- 0.37408,
- 0.3765,
- 0.37491,
- 0.37773,
- 0.37945,
- 0.37283,
- 0.37409,
- 0.57331,
- 0.37267,
- 0.37515,
- 0.37876,
- 0.37131,
- 0.36998,
- 0.36831,
- 0.37689,
- 0.37104,
- 0.37796,
- 0.3776,
- 0.37889,
- 0.3789,
- 0.38167,
- 0.37888,
- 0.37782,
- 0.38072,
- 0.37906,
- 0.39179,
- 0.37362,
- 0.37514,
- 0.37884,
- 0.3718,
- 0.3732,
- 0.37328,
- 0.37193,
- 0.37268,
- 0.37438,
- 0.37533,
- 0.37737,
- 0.3799,
- 0.37824,
- 0.37318,
- 0.37348,
- 0.38644,
- 0.37317,
- 0.37552,
- 0.37349,
- 0.37952,
- 0.37279,
- 0.37525,
- 0.37729,
- 0.37658,
- 0.38175,
- 0.37911,
- 0.38285,
- 0.37703,
- 0.37386,
- 0.37333,
- 0.37254,
- 0.38348,
- 0.38624,
- 0.38767,
- 0.37729,
- 0.37494,
- 0.3748,
- 0.37604,
- 0.37341,
- 0.37345,
- 0.37398,
- 0.37676,
- 0.37484,
- 0.37314,
- 0.37221,
- 0.37146,
- 0.37354,
- 0.37185,
- 0.37237,
- 0.37319,
- 0.37544,
- 0.37588,
- 0.37402,
- 0.38246,
- 0.377,
- 0.3754,
- 0.37227,
- 0.38037,
- 0.38689,
- 0.38215,
- 0.38483,
- 0.38456,
- 0.38612,
- 0.37346,
- 0.37238,
- 0.3736,
- 0.37485,
- 0.3753,
- 0.37849,
- 0.38602,
- 0.38352,
- 0.38006,
- 0.38036,
- 0.38583,
- 0.38083,
- 0.37255,
- 0.37355,
- 0.37625,
- 0.40762,
- 0.37445,
- 0.37449,
- 0.37462,
- 0.37751,
- 0.38402,
- 0.3824,
- 0.37623,
- 0.37718,
- 0.38762,
- 0.37136,
- 0.37556,
- 0.37615,
- 0.37207
- ]
- }
-}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_nondet_tp1_pp1_fp8_no_model_parallel/golden_values_lts_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_nondet_tp1_pp1_fp8_no_model_parallel/golden_values_lts_dgx_a100.json
deleted file mode 100644
index 18ec1c2a173..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_nondet_tp1_pp1_fp8_no_model_parallel/golden_values_lts_dgx_a100.json
+++ /dev/null
@@ -1,1223 +0,0 @@
-{
- "lm loss": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 10.88323,
- 10.90276,
- 10.88694,
- 10.83322,
- 10.67715,
- 10.64953,
- 10.43427,
- 10.15183,
- 9.93935,
- 9.84176,
- 9.5891,
- 9.85451,
- 9.88462,
- 9.6297,
- 9.78821,
- 9.51159,
- 9.45846,
- 9.64933,
- 9.3862,
- 9.3321,
- 9.24228,
- 9.14561,
- 9.17558,
- 8.99543,
- 9.18928,
- 9.05999,
- 9.15558,
- 9.16512,
- 9.29813,
- 8.98492,
- 8.92943,
- 9.04419,
- 9.04322,
- 8.65521,
- 8.71738,
- 8.75365,
- 8.68379,
- 8.73429,
- 8.65884,
- 8.76517,
- 8.66123,
- 8.85001,
- 8.83236,
- 8.4994,
- 8.38904,
- 8.43166,
- 8.49319,
- 8.38452,
- 8.43286,
- 8.57956,
- 8.36712,
- 8.19207,
- 8.22579,
- 8.22194,
- 8.26717,
- 7.91302,
- 8.0955,
- 7.89089,
- 8.24619,
- 8.23017,
- 8.00469,
- 7.96542,
- 7.91804,
- 7.73978,
- 7.73961,
- 7.64245,
- 7.51511,
- 7.90632,
- 7.69783,
- 7.45086,
- 7.73945,
- 7.76671,
- 7.54095,
- 7.29791,
- 7.45173,
- 7.33462,
- 7.4612,
- 7.22294,
- 7.63514,
- 7.27784,
- 7.35079,
- 7.21176,
- 7.21704,
- 7.42198,
- 7.1767,
- 7.28254,
- 7.00176,
- 7.0057,
- 7.04106,
- 7.14049,
- 6.82528,
- 6.98673,
- 7.08928,
- 7.00172,
- 6.87462,
- 6.75859,
- 6.99286,
- 7.05962,
- 6.70626,
- 6.58385,
- 6.72973,
- 6.74483,
- 6.73638,
- 6.74114,
- 6.66099,
- 6.40952,
- 6.64131,
- 6.62122,
- 6.44763,
- 6.63054,
- 6.74432,
- 6.60975,
- 6.72503,
- 6.69474,
- 6.6247,
- 6.50691,
- 6.59911,
- 6.4064,
- 6.66409,
- 6.24856,
- 6.2516,
- 6.3016,
- 6.38875,
- 6.34796,
- 6.44852,
- 6.28545,
- 6.33925,
- 6.23596,
- 6.20233,
- 6.39825,
- 6.32525,
- 6.32413,
- 6.16984,
- 6.16253,
- 6.24375,
- 6.3879,
- 6.20637,
- 6.15552,
- 6.18702,
- 6.12144,
- 6.06949,
- 6.07869,
- 6.26293,
- 6.41494,
- 6.26452,
- 6.30693,
- 6.10587,
- 6.18713,
- 6.01158,
- 6.03875,
- 5.96545,
- 6.25534,
- 6.19897,
- 5.97346,
- 5.79144,
- 6.13388,
- 5.85851,
- 6.11375,
- 5.79987,
- 6.16878,
- 6.15254,
- 6.09497,
- 5.93885,
- 6.1206,
- 5.94963,
- 6.20011,
- 5.901,
- 5.79876,
- 5.78176,
- 5.6937,
- 6.02012,
- 6.00074,
- 6.06782,
- 5.89184,
- 6.04281,
- 5.97078,
- 5.99763,
- 5.98979,
- 5.94805,
- 5.84122,
- 5.95124,
- 5.61843,
- 5.70225,
- 5.8906,
- 5.84333,
- 5.8628,
- 5.76133,
- 5.83588,
- 5.72872,
- 5.56229,
- 5.72027,
- 5.62406,
- 5.83386,
- 5.60151,
- 5.71159,
- 5.71751,
- 5.89971,
- 5.64532,
- 5.85138,
- 5.73855,
- 5.87273,
- 5.33013,
- 5.8957,
- 5.8746,
- 5.85218,
- 5.41494,
- 5.41026,
- 5.62571,
- 5.59371,
- 5.48334,
- 5.57165,
- 5.67238,
- 5.4744,
- 5.74362,
- 5.51126,
- 5.59605,
- 5.62107,
- 5.61572,
- 5.50856,
- 5.60876,
- 5.67058,
- 5.68967,
- 5.58943,
- 5.65884,
- 5.37283,
- 5.68049,
- 5.62588,
- 5.42149,
- 5.58882,
- 5.6294,
- 5.55294,
- 5.33966,
- 5.53728,
- 5.48414,
- 5.48307,
- 5.37506,
- 5.55721,
- 5.60131,
- 5.38633,
- 5.53162,
- 5.48787,
- 5.33174,
- 5.50407,
- 5.4065,
- 5.44014,
- 5.31531,
- 5.06354,
- 5.47634,
- 5.5663,
- 5.70998,
- 5.41495,
- 5.59526,
- 5.6328,
- 5.2319,
- 5.2739,
- 5.39497,
- 5.39608,
- 5.32487,
- 5.49737,
- 5.18209,
- 5.29492,
- 5.24643,
- 5.37552,
- 5.25606,
- 5.44308,
- 5.53741,
- 5.31228,
- 5.44067,
- 5.33998,
- 5.07194,
- 5.31518,
- 5.24712,
- 5.30351,
- 5.10936,
- 5.27335,
- 5.26643,
- 5.46934,
- 5.15835,
- 5.2678,
- 5.20457,
- 5.35651,
- 4.9827,
- 4.91355,
- 5.31913,
- 5.38813,
- 5.22706,
- 5.31863,
- 5.09862,
- 5.15647,
- 5.25815,
- 5.06521,
- 5.26139,
- 5.07559,
- 5.34225,
- 5.2435,
- 5.14354,
- 5.23796,
- 5.03841,
- 5.31227,
- 5.05047,
- 5.02308,
- 5.14022,
- 5.10954,
- 5.27005,
- 5.14834,
- 5.2764,
- 5.09643,
- 5.09616,
- 5.24991,
- 5.31987,
- 5.25189,
- 5.18613,
- 5.14096,
- 5.28633,
- 4.94797,
- 5.20474,
- 5.08641,
- 5.3005,
- 5.17427,
- 5.18273,
- 5.10837,
- 4.98264,
- 4.99144,
- 5.22303,
- 5.30945,
- 5.09288,
- 5.0515,
- 4.9141,
- 5.12157,
- 5.11768,
- 4.92193,
- 5.33538,
- 5.01865,
- 5.09977,
- 5.15945,
- 5.00134,
- 5.062,
- 5.06352,
- 4.98951,
- 5.07403,
- 5.15561,
- 4.97364,
- 5.17698,
- 4.92401,
- 4.91763,
- 5.06561,
- 4.98934,
- 4.90514,
- 4.77142,
- 4.93751,
- 5.10748,
- 5.01115,
- 5.01315,
- 5.32269,
- 4.95385,
- 4.98933,
- 5.03967,
- 4.80287,
- 4.73643,
- 4.99208,
- 5.03327,
- 4.86668,
- 4.9473,
- 5.03761,
- 5.01854,
- 4.81126,
- 4.88589,
- 4.89708,
- 4.82611,
- 4.73767,
- 5.00493,
- 4.74564,
- 5.20177,
- 4.77793,
- 4.98531,
- 4.72962,
- 4.77857,
- 4.81505,
- 4.64522,
- 4.64996,
- 4.83534,
- 4.80065,
- 4.79383,
- 4.91643,
- 4.87724,
- 4.9168,
- 4.7603,
- 4.87501,
- 4.72665,
- 4.90429,
- 4.95354,
- 4.86716,
- 4.70097,
- 4.77165,
- 4.89297,
- 4.70177,
- 4.85355,
- 4.68265,
- 4.68029,
- 4.64235
- ]
- },
- "num-zeros": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 77.0,
- 69.0,
- 83.0,
- 75.0,
- 87.0,
- 65.0,
- 107.0,
- 100.0,
- 110.0,
- 118.0,
- 128.0,
- 140.0,
- 140.0,
- 162.0,
- 158.0,
- 163.0,
- 148.0,
- 189.0,
- 182.0,
- 184.0,
- 191.0,
- 164.0,
- 191.0,
- 164.0,
- 211.0,
- 159.0,
- 188.0,
- 172.0,
- 153.0,
- 168.0,
- 138.0,
- 173.0,
- 164.0,
- 177.0,
- 160.0,
- 145.0,
- 170.0,
- 214.0,
- 177.0,
- 204.0,
- 172.0,
- 193.0,
- 183.0,
- 202.0,
- 179.0,
- 168.0,
- 190.0,
- 212.0,
- 194.0,
- 198.0,
- 193.0,
- 149.0,
- 204.0,
- 143.0,
- 158.0,
- 203.0,
- 173.0,
- 140.0,
- 230.0,
- 258.0,
- 215.0,
- 193.0,
- 220.0,
- 189.0,
- 186.0,
- 282.0,
- 204.0,
- 168.0,
- 197.0,
- 185.0,
- 249.0,
- 253.0,
- 197.0,
- 222.0,
- 213.0,
- 190.0,
- 240.0,
- 197.0,
- 291.0,
- 232.0,
- 198.0,
- 294.0,
- 223.0,
- 233.0,
- 193.0,
- 212.0,
- 198.0,
- 232.0,
- 226.0,
- 219.0,
- 227.0,
- 226.0,
- 240.0,
- 208.0,
- 186.0,
- 151.0,
- 200.0,
- 222.0,
- 199.0,
- 187.0,
- 193.0,
- 200.0,
- 158.0,
- 181.0,
- 167.0,
- 144.0,
- 177.0,
- 172.0,
- 156.0,
- 209.0,
- 196.0,
- 153.0,
- 160.0,
- 178.0,
- 164.0,
- 152.0,
- 154.0,
- 130.0,
- 182.0,
- 142.0,
- 158.0,
- 145.0,
- 157.0,
- 155.0,
- 140.0,
- 161.0,
- 141.0,
- 139.0,
- 112.0,
- 117.0,
- 146.0,
- 132.0,
- 123.0,
- 121.0,
- 152.0,
- 140.0,
- 145.0,
- 86.0,
- 111.0,
- 122.0,
- 94.0,
- 130.0,
- 133.0,
- 140.0,
- 154.0,
- 134.0,
- 113.0,
- 112.0,
- 127.0,
- 130.0,
- 104.0,
- 111.0,
- 102.0,
- 110.0,
- 143.0,
- 106.0,
- 94.0,
- 81.0,
- 83.0,
- 101.0,
- 119.0,
- 108.0,
- 133.0,
- 151.0,
- 119.0,
- 96.0,
- 105.0,
- 124.0,
- 137.0,
- 104.0,
- 103.0,
- 98.0,
- 97.0,
- 92.0,
- 120.0,
- 116.0,
- 115.0,
- 139.0,
- 118.0,
- 86.0,
- 120.0,
- 109.0,
- 121.0,
- 120.0,
- 92.0,
- 125.0,
- 121.0,
- 110.0,
- 74.0,
- 92.0,
- 107.0,
- 115.0,
- 116.0,
- 105.0,
- 83.0,
- 95.0,
- 112.0,
- 95.0,
- 110.0,
- 118.0,
- 97.0,
- 97.0,
- 112.0,
- 107.0,
- 118.0,
- 104.0,
- 114.0,
- 109.0,
- 118.0,
- 105.0,
- 125.0,
- 87.0,
- 102.0,
- 109.0,
- 110.0,
- 99.0,
- 90.0,
- 129.0,
- 123.0,
- 109.0,
- 117.0,
- 74.0,
- 90.0,
- 121.0,
- 92.0,
- 106.0,
- 96.0,
- 138.0,
- 104.0,
- 123.0,
- 101.0,
- 104.0,
- 105.0,
- 102.0,
- 99.0,
- 119.0,
- 101.0,
- 101.0,
- 102.0,
- 84.0,
- 97.0,
- 89.0,
- 104.0,
- 98.0,
- 92.0,
- 103.0,
- 106.0,
- 118.0,
- 113.0,
- 122.0,
- 121.0,
- 115.0,
- 119.0,
- 118.0,
- 103.0,
- 106.0,
- 113.0,
- 118.0,
- 115.0,
- 112.0,
- 115.0,
- 91.0,
- 107.0,
- 90.0,
- 95.0,
- 106.0,
- 91.0,
- 104.0,
- 106.0,
- 116.0,
- 82.0,
- 111.0,
- 104.0,
- 130.0,
- 112.0,
- 105.0,
- 93.0,
- 107.0,
- 98.0,
- 105.0,
- 86.0,
- 98.0,
- 105.0,
- 119.0,
- 112.0,
- 106.0,
- 116.0,
- 104.0,
- 124.0,
- 104.0,
- 114.0,
- 102.0,
- 98.0,
- 98.0,
- 107.0,
- 118.0,
- 107.0,
- 98.0,
- 102.0,
- 111.0,
- 126.0,
- 97.0,
- 118.0,
- 126.0,
- 112.0,
- 91.0,
- 93.0,
- 108.0,
- 124.0,
- 119.0,
- 98.0,
- 147.0,
- 96.0,
- 119.0,
- 109.0,
- 112.0,
- 119.0,
- 96.0,
- 105.0,
- 96.0,
- 122.0,
- 100.0,
- 107.0,
- 110.0,
- 121.0,
- 82.0,
- 105.0,
- 108.0,
- 98.0,
- 100.0,
- 111.0,
- 99.0,
- 121.0,
- 89.0,
- 129.0,
- 102.0,
- 92.0,
- 119.0,
- 106.0,
- 110.0,
- 116.0,
- 109.0,
- 100.0,
- 125.0,
- 88.0,
- 101.0,
- 104.0,
- 88.0,
- 109.0,
- 111.0,
- 99.0,
- 113.0,
- 111.0,
- 136.0,
- 111.0,
- 113.0,
- 135.0,
- 95.0,
- 94.0,
- 110.0,
- 121.0,
- 123.0,
- 134.0,
- 132.0,
- 118.0,
- 112.0,
- 98.0,
- 116.0,
- 100.0,
- 95.0,
- 103.0,
- 111.0,
- 100.0,
- 111.0,
- 112.0,
- 127.0,
- 108.0,
- 108.0,
- 104.0,
- 120.0,
- 123.0,
- 124.0,
- 133.0,
- 116.0,
- 130.0,
- 119.0,
- 115.0,
- 135.0,
- 119.0,
- 109.0,
- 114.0,
- 97.0,
- 120.0,
- 122.0,
- 107.0,
- 151.0,
- 131.0,
- 130.0,
- 133.0,
- 116.0
- ]
- },
- "iteration-time": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 23.49073,
- 0.45673,
- 0.49857,
- 0.45742,
- 0.45417,
- 0.45498,
- 0.45169,
- 0.44995,
- 0.44985,
- 0.46253,
- 0.44641,
- 0.45172,
- 0.44994,
- 0.44786,
- 0.44991,
- 0.46752,
- 0.44937,
- 0.44931,
- 0.45455,
- 0.45638,
- 0.44949,
- 0.44578,
- 0.45615,
- 0.45432,
- 0.45019,
- 0.45684,
- 0.45146,
- 0.45053,
- 0.44516,
- 0.44513,
- 0.44748,
- 0.44806,
- 0.70306,
- 0.44525,
- 0.45604,
- 0.45039,
- 0.44938,
- 0.44478,
- 0.45854,
- 0.44939,
- 0.4453,
- 0.4508,
- 0.44723,
- 0.44863,
- 0.4456,
- 0.44644,
- 0.45712,
- 0.45015,
- 0.44577,
- 0.44529,
- 0.44891,
- 0.45444,
- 0.45302,
- 0.44825,
- 0.44762,
- 0.45019,
- 0.44869,
- 0.57727,
- 0.4499,
- 0.45275,
- 0.46154,
- 0.44858,
- 0.44579,
- 0.45551,
- 0.45026,
- 0.44368,
- 0.44584,
- 0.44692,
- 0.44436,
- 0.44468,
- 0.46316,
- 0.44645,
- 0.44314,
- 0.4448,
- 0.4471,
- 0.45064,
- 0.44559,
- 0.44749,
- 0.45139,
- 0.4535,
- 0.58646,
- 0.44962,
- 0.44927,
- 0.46076,
- 0.44914,
- 0.4463,
- 0.44803,
- 0.45468,
- 0.44878,
- 0.45252,
- 0.45032,
- 0.45193,
- 0.44895,
- 0.44717,
- 0.45458,
- 0.45081,
- 0.44639,
- 0.45649,
- 0.44958,
- 0.44661,
- 0.44544,
- 0.45127,
- 0.45634,
- 0.44936,
- 0.44802,
- 0.45893,
- 0.70259,
- 0.58713,
- 0.4441,
- 0.44774,
- 0.44927,
- 0.45009,
- 0.45029,
- 0.44752,
- 0.45399,
- 0.44921,
- 0.45252,
- 0.44728,
- 0.45779,
- 0.45171,
- 0.44784,
- 0.45047,
- 0.44749,
- 0.45711,
- 0.45055,
- 0.44951,
- 0.4473,
- 0.44734,
- 0.58434,
- 0.45093,
- 0.44969,
- 0.56992,
- 0.44965,
- 0.45071,
- 0.44913,
- 0.44756,
- 0.44547,
- 0.44971,
- 0.45838,
- 0.4574,
- 0.45394,
- 0.45483,
- 0.4512,
- 0.44954,
- 0.4479,
- 0.44758,
- 0.44853,
- 0.45108,
- 0.44804,
- 0.44791,
- 0.44831,
- 0.45494,
- 0.44761,
- 0.44412,
- 0.44433,
- 0.44519,
- 0.45125,
- 0.447,
- 0.4492,
- 0.44787,
- 0.44944,
- 0.44622,
- 0.4476,
- 0.4447,
- 0.45124,
- 0.44854,
- 0.44716,
- 0.44676,
- 0.44755,
- 0.4655,
- 0.4487,
- 0.44985,
- 0.44982,
- 0.44694,
- 0.44611,
- 0.44694,
- 0.44286,
- 0.44458,
- 0.44491,
- 0.45147,
- 0.44613,
- 0.5801,
- 0.45263,
- 0.44887,
- 0.44979,
- 0.44625,
- 0.45051,
- 0.44896,
- 0.4423,
- 0.4475,
- 0.44896,
- 0.45016,
- 0.45298,
- 0.44594,
- 0.44685,
- 0.45698,
- 0.44779,
- 0.44749,
- 0.44739,
- 0.45153,
- 0.57538,
- 0.44826,
- 0.45017,
- 0.44753,
- 0.44927,
- 0.44831,
- 0.44866,
- 0.44895,
- 0.44796,
- 0.45036,
- 0.44825,
- 0.4478,
- 0.44693,
- 0.45241,
- 0.44821,
- 0.44687,
- 0.44895,
- 0.45248,
- 0.45022,
- 0.44649,
- 0.4508,
- 0.45026,
- 0.4497,
- 0.45016,
- 0.44784,
- 0.44722,
- 0.45425,
- 0.44892,
- 0.45033,
- 0.45322,
- 0.45187,
- 0.44969,
- 0.45852,
- 0.45233,
- 0.45326,
- 0.44695,
- 0.44901,
- 0.44797,
- 0.45123,
- 0.44468,
- 0.44681,
- 0.45333,
- 0.44879,
- 0.44331,
- 0.44989,
- 0.45159,
- 0.44991,
- 0.44774,
- 0.44604,
- 0.58441,
- 0.44958,
- 0.44496,
- 0.44421,
- 0.44393,
- 0.44478,
- 0.44417,
- 0.44427,
- 0.44729,
- 0.4465,
- 0.45195,
- 0.44517,
- 0.44747,
- 0.4465,
- 0.44691,
- 0.44759,
- 0.44365,
- 0.44855,
- 0.44391,
- 0.44652,
- 0.44474,
- 0.45265,
- 0.44285,
- 0.44348,
- 0.46714,
- 0.44438,
- 0.44968,
- 0.58646,
- 0.4456,
- 0.57565,
- 0.4451,
- 0.44392,
- 0.44762,
- 0.44584,
- 0.44731,
- 0.44368,
- 0.44143,
- 0.44348,
- 0.44286,
- 0.44866,
- 0.44303,
- 0.4467,
- 0.44242,
- 0.44594,
- 0.44457,
- 0.44212,
- 0.45173,
- 0.45314,
- 0.4537,
- 0.45345,
- 0.44645,
- 0.44564,
- 0.44791,
- 0.44538,
- 0.56436,
- 0.4463,
- 0.44361,
- 0.44583,
- 0.4472,
- 0.44565,
- 0.44765,
- 0.44352,
- 0.44439,
- 0.45014,
- 0.45393,
- 0.44761,
- 0.44365,
- 0.44194,
- 0.44055,
- 0.44391,
- 0.44516,
- 0.43991,
- 0.43973,
- 0.44667,
- 0.59303,
- 0.44362,
- 0.44564,
- 0.4467,
- 0.45244,
- 0.84618,
- 0.44873,
- 0.44536,
- 0.446,
- 0.4484,
- 0.45038,
- 0.44833,
- 0.45815,
- 0.44989,
- 0.45457,
- 0.45252,
- 0.45002,
- 0.45094,
- 0.44968,
- 0.45105,
- 0.44441,
- 0.4415,
- 0.44859,
- 0.43942,
- 0.44673,
- 0.60446,
- 0.44265,
- 0.44754,
- 0.45059,
- 0.4443,
- 0.57371,
- 0.45333,
- 0.44117,
- 0.44025,
- 0.44493,
- 0.44453,
- 0.44295,
- 0.44557,
- 0.4392,
- 0.44354,
- 0.45185,
- 0.44735,
- 0.4481,
- 0.45094,
- 0.44791,
- 0.45131,
- 0.44821,
- 0.44249,
- 0.44289,
- 0.44532,
- 0.58138,
- 0.44778,
- 0.44834,
- 0.44647,
- 0.44908,
- 0.71286,
- 0.44635,
- 0.44907,
- 0.44524,
- 0.44548,
- 0.44391,
- 0.44473,
- 0.4419,
- 0.44386,
- 0.44348,
- 0.44854,
- 0.44606,
- 0.4454,
- 0.44354,
- 0.44676,
- 0.44494,
- 0.44387,
- 0.44867,
- 0.44496,
- 0.44666,
- 0.44531,
- 0.44669
- ]
- }
-}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_nondet_tp1_pp1_fp8_no_model_parallel/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_nondet_tp1_pp1_fp8_no_model_parallel/model_config.yaml
deleted file mode 100644
index 0e70965cb2b..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_nondet_tp1_pp1_fp8_no_model_parallel/model_config.yaml
+++ /dev/null
@@ -1,51 +0,0 @@
-ENV_VARS:
- CUDA_DEVICE_MAX_CONNECTIONS: 1
- NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1
-MODEL_ARGS:
- --num-layers: 12
- --hidden-size: 512
- --num-attention-heads: 8
- --log-params-norm: true
- --log-num-zeros-in-grad: true
- --log-validation-ppl-to-tensorboard: true
- --log-timers-to-tensorboard: true
- --tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
- --seq-length: 1024
- --max-position-embeddings: 1024
- --train-iters: 2000
- --timing-log-level: 0
- --lr-decay-iters: 320000
- --save: ${CHECKPOINT_SAVE_PATH}
- --load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
- --split: 949,50,1
- --distributed-backend: nccl
- --lr: 0.00015
- --lr-decay-style: cosine
- --min-lr: 1.0e-5
- --weight-decay: 1e-2
- --clip-grad: 1.0
- --lr-warmup-fraction: .01
- --log-interval: 1
- --save-interval: 10000
- --eval-interval: 1000
- --eval-iters: 10
- --transformer-impl: transformer_engine
- --tensor-model-parallel-size: 1
- --pipeline-model-parallel-size: 1
- --no-gradient-accumulation-fusion: true
- --fp8-format: hybrid
- --fp8-amax-history-len: 1024
- --fp8-amax-compute-algo: max
- --attention-softmax-in-fp32: true
- --ckpt-format: torch_dist
- --dist-ckpt-optim-fully-reshardable: true
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
- --data-cache-path: ${DATA_CACHE_PATH}
- --bf16: true
- --log-memory-to-tensorboard: true
-TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_bf16_baseline/golden_values_dev_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_bf16_baseline/golden_values_dev_dgx_a100.json
deleted file mode 100644
index c7f6bc8588d..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_bf16_baseline/golden_values_dev_dgx_a100.json
+++ /dev/null
@@ -1,1220 +0,0 @@
-{
- "lm loss": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 10.89393,
- 10.90229,
- 10.90382,
- 10.89922,
- 10.90215,
- 10.87439,
- 10.80338,
- 10.63346,
- 10.44036,
- 10.2933,
- 10.02712,
- 10.16747,
- 10.13781,
- 9.86191,
- 9.97684,
- 9.67806,
- 9.59836,
- 9.7815,
- 9.50325,
- 9.44529,
- 9.35262,
- 9.25422,
- 9.27971,
- 9.09386,
- 9.28651,
- 9.15722,
- 9.24673,
- 9.26197,
- 9.39815,
- 9.08902,
- 9.03506,
- 9.14524,
- 9.15344,
- 8.76086,
- 8.82546,
- 8.85801,
- 8.78594,
- 8.83766,
- 8.76271,
- 8.8693,
- 8.76505,
- 8.95513,
- 8.94138,
- 8.60415,
- 8.49526,
- 8.5414,
- 8.6052,
- 8.49377,
- 8.54563,
- 8.69588,
- 8.4793,
- 8.31046,
- 8.3419,
- 8.3376,
- 8.38481,
- 8.03115,
- 8.21697,
- 8.01004,
- 8.36596,
- 8.3517,
- 8.12379,
- 8.08902,
- 8.03892,
- 7.85883,
- 7.86204,
- 7.76178,
- 7.63785,
- 8.03256,
- 7.82491,
- 7.57768,
- 7.87018,
- 7.89664,
- 7.66577,
- 7.41891,
- 7.57946,
- 7.45949,
- 7.58407,
- 7.3365,
- 7.75477,
- 7.39311,
- 7.46005,
- 7.326,
- 7.3226,
- 7.53323,
- 7.28431,
- 7.39059,
- 7.10454,
- 7.10309,
- 7.135,
- 7.23329,
- 6.91494,
- 7.07307,
- 7.1732,
- 7.08149,
- 6.95567,
- 6.83555,
- 7.07147,
- 7.13599,
- 6.77635,
- 6.65371,
- 6.79924,
- 6.81095,
- 6.80156,
- 6.80623,
- 6.72479,
- 6.46997,
- 6.70288,
- 6.67891,
- 6.50415,
- 6.69017,
- 6.80201,
- 6.66743,
- 6.78224,
- 6.74909,
- 6.68039,
- 6.55852,
- 6.65127,
- 6.45883,
- 6.71595,
- 6.30029,
- 6.29946,
- 6.35125,
- 6.43625,
- 6.39727,
- 6.50048,
- 6.33651,
- 6.38488,
- 6.28047,
- 6.24359,
- 6.44009,
- 6.36825,
- 6.36402,
- 6.2045,
- 6.19664,
- 6.27933,
- 6.42468,
- 6.24025,
- 6.18585,
- 6.21348,
- 6.14842,
- 6.09617,
- 6.1035,
- 6.28976,
- 6.44192,
- 6.28932,
- 6.33177,
- 6.12937,
- 6.2119,
- 6.03064,
- 6.05658,
- 5.98505,
- 6.27562,
- 6.21999,
- 5.99254,
- 5.81222,
- 6.1522,
- 5.87811,
- 6.13276,
- 5.81621,
- 6.18981,
- 6.17418,
- 6.11405,
- 5.95877,
- 6.13943,
- 5.96879,
- 6.22137,
- 5.92302,
- 5.81813,
- 5.80612,
- 5.71127,
- 6.04011,
- 6.02026,
- 6.09059,
- 5.91133,
- 6.0647,
- 5.9908,
- 6.01775,
- 6.01088,
- 5.97305,
- 5.86247,
- 5.97385,
- 5.63832,
- 5.72202,
- 5.91221,
- 5.86536,
- 5.88217,
- 5.78585,
- 5.85599,
- 5.74904,
- 5.58238,
- 5.74505,
- 5.64738,
- 5.8552,
- 5.62673,
- 5.73069,
- 5.73403,
- 5.92154,
- 5.66651,
- 5.86965,
- 5.76023,
- 5.89258,
- 5.35098,
- 5.9205,
- 5.89567,
- 5.87366,
- 5.43348,
- 5.42769,
- 5.64532,
- 5.61424,
- 5.50172,
- 5.5911,
- 5.69239,
- 5.49278,
- 5.76306,
- 5.53002,
- 5.61324,
- 5.64004,
- 5.63451,
- 5.52873,
- 5.63026,
- 5.68897,
- 5.69849,
- 5.60119,
- 5.67641,
- 5.3926,
- 5.69571,
- 5.64274,
- 5.43772,
- 5.59953,
- 5.64251,
- 5.56535,
- 5.35493,
- 5.55145,
- 5.49555,
- 5.49469,
- 5.38646,
- 5.5675,
- 5.61485,
- 5.39936,
- 5.53506,
- 5.49708,
- 5.34111,
- 5.51556,
- 5.42086,
- 5.4521,
- 5.32709,
- 5.07441,
- 5.48669,
- 5.57797,
- 5.72108,
- 5.42477,
- 5.60744,
- 5.64535,
- 5.24322,
- 5.28211,
- 5.40464,
- 5.40345,
- 5.33686,
- 5.51041,
- 5.19531,
- 5.30946,
- 5.26092,
- 5.38482,
- 5.26778,
- 5.45655,
- 5.54658,
- 5.32255,
- 5.44786,
- 5.34468,
- 5.0817,
- 5.3265,
- 5.26443,
- 5.31477,
- 5.1223,
- 5.28586,
- 5.27616,
- 5.48205,
- 5.16778,
- 5.27791,
- 5.21918,
- 5.37082,
- 4.99576,
- 4.92396,
- 5.33114,
- 5.40116,
- 5.23548,
- 5.32971,
- 5.1098,
- 5.16761,
- 5.27075,
- 5.07658,
- 5.27525,
- 5.09175,
- 5.35657,
- 5.25632,
- 5.16135,
- 5.24941,
- 5.05151,
- 5.32323,
- 5.06328,
- 5.03807,
- 5.15012,
- 5.12121,
- 5.2805,
- 5.1623,
- 5.28751,
- 5.10857,
- 5.107,
- 5.26185,
- 5.33273,
- 5.26325,
- 5.19866,
- 5.15283,
- 5.29684,
- 4.9578,
- 5.21696,
- 5.09944,
- 5.30924,
- 5.18412,
- 5.19534,
- 5.12112,
- 4.99133,
- 5.00084,
- 5.23319,
- 5.32054,
- 5.10638,
- 5.06456,
- 4.92573,
- 5.13168,
- 5.12607,
- 4.93273,
- 5.3413,
- 5.03043,
- 5.10934,
- 5.16974,
- 5.01126,
- 5.07104,
- 5.07587,
- 5.0034,
- 5.08619,
- 5.1671,
- 4.98476,
- 5.18902,
- 4.93793,
- 4.92414,
- 5.07774,
- 4.99851,
- 4.91554,
- 4.78269,
- 4.95064,
- 5.12237,
- 5.02596,
- 5.02298,
- 5.33707,
- 4.96446,
- 4.99962,
- 5.05063,
- 4.81016,
- 4.74605,
- 5.00281,
- 5.04573,
- 4.88142,
- 4.95871,
- 5.04942,
- 5.02997,
- 4.81942,
- 4.89951,
- 4.91098,
- 4.83717,
- 4.74869,
- 5.01582,
- 4.75783,
- 5.21702,
- 4.79022,
- 4.99791,
- 4.74194,
- 4.7912,
- 4.82664,
- 4.65524,
- 4.6621,
- 4.85014,
- 4.81175,
- 4.80742,
- 4.93171,
- 4.88928,
- 4.92931,
- 4.77459,
- 4.8876,
- 4.73984,
- 4.91676,
- 4.96546,
- 4.87897,
- 4.71224,
- 4.78675,
- 4.90579,
- 4.71528,
- 4.86716,
- 4.69307,
- 4.69138,
- 4.65331
- ]
- },
- "iteration-time": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 9.25578,
- 0.3326,
- 0.33822,
- 0.32857,
- 0.3426,
- 0.34934,
- 0.34164,
- 0.34303,
- 0.34646,
- 0.3405,
- 0.34386,
- 0.35065,
- 0.33857,
- 0.33893,
- 0.35587,
- 0.34445,
- 0.3386,
- 0.34381,
- 0.3394,
- 0.34322,
- 0.33866,
- 0.34045,
- 0.34327,
- 0.34138,
- 0.34855,
- 0.34967,
- 0.34407,
- 0.34762,
- 0.35319,
- 0.33655,
- 0.33613,
- 0.33455,
- 0.33412,
- 0.34143,
- 0.33898,
- 0.33485,
- 0.3759,
- 0.34214,
- 0.33791,
- 0.33356,
- 0.33752,
- 0.334,
- 0.33322,
- 0.33482,
- 0.33655,
- 0.33394,
- 0.33331,
- 0.3351,
- 0.3314,
- 0.33591,
- 0.33346,
- 0.33519,
- 0.33236,
- 0.33088,
- 0.33279,
- 0.3329,
- 0.3359,
- 0.33962,
- 0.33166,
- 0.3389,
- 0.33537,
- 0.33003,
- 0.33507,
- 0.33086,
- 0.33492,
- 0.3322,
- 0.33134,
- 0.33302,
- 0.3341,
- 0.33216,
- 0.33239,
- 0.33318,
- 0.33361,
- 0.33237,
- 0.33266,
- 0.33698,
- 0.33954,
- 0.33607,
- 0.33264,
- 0.33248,
- 0.33964,
- 0.33521,
- 0.33566,
- 0.33367,
- 0.33504,
- 0.33451,
- 0.33413,
- 0.33504,
- 0.33696,
- 0.3376,
- 0.33765,
- 0.33646,
- 0.3365,
- 0.33915,
- 0.33487,
- 0.33518,
- 0.33513,
- 0.33649,
- 0.33811,
- 0.33604,
- 0.33597,
- 0.33456,
- 0.33512,
- 0.33801,
- 0.33645,
- 0.337,
- 0.3365,
- 0.33969,
- 0.34136,
- 0.33618,
- 0.3333,
- 0.33291,
- 0.33287,
- 0.51594,
- 0.34363,
- 0.33638,
- 0.33456,
- 0.33793,
- 0.33855,
- 0.3359,
- 0.33867,
- 0.33647,
- 0.3352,
- 0.33624,
- 0.33617,
- 0.51401,
- 0.33827,
- 0.33714,
- 0.33569,
- 0.33609,
- 0.334,
- 0.33524,
- 0.33575,
- 0.33371,
- 0.33439,
- 0.34352,
- 0.33393,
- 0.33376,
- 0.33687,
- 0.3341,
- 0.33377,
- 0.33715,
- 0.33643,
- 0.33704,
- 0.34004,
- 0.33701,
- 0.34317,
- 0.34338,
- 0.33355,
- 0.34018,
- 0.33372,
- 0.33971,
- 0.33659,
- 0.33682,
- 0.34053,
- 0.34117,
- 0.33512,
- 0.33493,
- 0.3356,
- 0.33062,
- 0.33407,
- 0.33178,
- 0.33299,
- 0.33624,
- 0.33672,
- 0.33162,
- 0.33801,
- 0.50818,
- 0.33122,
- 0.33524,
- 0.33395,
- 0.33144,
- 0.33808,
- 0.33398,
- 0.33057,
- 0.33247,
- 0.33608,
- 0.33554,
- 0.33546,
- 0.33375,
- 0.3376,
- 0.34091,
- 0.3369,
- 0.33926,
- 0.33962,
- 0.33152,
- 0.327,
- 0.32552,
- 0.32939,
- 0.32366,
- 0.32998,
- 0.32721,
- 0.3246,
- 0.32935,
- 0.32592,
- 0.3266,
- 0.33091,
- 0.3258,
- 0.32938,
- 0.32694,
- 0.33356,
- 0.3274,
- 0.32466,
- 0.33347,
- 0.3323,
- 0.33117,
- 0.32588,
- 0.32403,
- 0.32795,
- 0.32369,
- 0.32203,
- 0.32301,
- 0.32286,
- 0.32055,
- 0.3398,
- 0.32238,
- 0.33633,
- 0.3256,
- 0.33198,
- 0.50333,
- 0.33007,
- 0.33025,
- 0.3307,
- 0.32366,
- 0.3305,
- 0.33215,
- 0.32605,
- 0.70345,
- 0.33425,
- 0.33421,
- 0.32842,
- 0.33332,
- 0.33075,
- 0.32626,
- 0.32712,
- 0.32341,
- 0.32308,
- 0.32473,
- 0.32353,
- 0.32932,
- 0.33035,
- 0.32401,
- 0.33502,
- 0.33327,
- 0.33395,
- 0.32981,
- 0.32419,
- 0.32325,
- 0.33309,
- 0.32184,
- 0.33265,
- 0.32364,
- 0.3237,
- 0.33155,
- 0.32372,
- 0.32382,
- 0.32291,
- 0.32388,
- 0.32158,
- 0.32223,
- 0.32498,
- 0.3253,
- 0.33429,
- 0.32815,
- 0.32815,
- 0.32262,
- 0.32595,
- 0.33413,
- 0.33488,
- 0.32392,
- 0.32413,
- 0.32569,
- 0.49049,
- 0.3248,
- 0.33109,
- 0.32587,
- 0.32642,
- 0.32518,
- 0.32592,
- 0.32421,
- 0.71015,
- 0.33488,
- 0.33222,
- 0.33776,
- 0.33626,
- 0.33446,
- 0.33173,
- 0.33291,
- 0.33359,
- 0.3356,
- 0.32588,
- 0.32604,
- 0.32374,
- 0.32432,
- 0.32517,
- 0.32336,
- 0.32242,
- 0.32382,
- 0.32447,
- 0.32621,
- 0.32442,
- 0.33073,
- 0.32577,
- 0.32967,
- 0.32407,
- 0.32569,
- 0.32784,
- 0.3461,
- 0.32392,
- 0.32392,
- 0.32443,
- 0.32222,
- 0.32412,
- 0.32365,
- 0.32223,
- 0.3256,
- 0.32161,
- 0.32484,
- 0.32165,
- 0.32169,
- 0.32734,
- 0.32352,
- 0.32425,
- 0.32547,
- 0.3233,
- 0.32457,
- 0.32423,
- 0.32358,
- 0.32516,
- 0.32609,
- 0.32614,
- 0.32573,
- 0.32359,
- 0.50412,
- 0.32385,
- 0.3249,
- 0.33249,
- 0.34813,
- 0.33455,
- 0.33984,
- 0.33686,
- 0.33544,
- 0.32686,
- 0.32733,
- 0.32357,
- 0.33073,
- 0.32781,
- 0.32687,
- 0.32707,
- 0.3227,
- 0.32312,
- 0.32367,
- 0.32418,
- 0.32795,
- 0.32217,
- 0.32661,
- 0.32769,
- 0.32438,
- 0.32866,
- 0.32324,
- 0.32266,
- 0.32478,
- 0.32267,
- 0.3259,
- 0.32629,
- 0.32532,
- 0.33247,
- 0.33203,
- 0.32868,
- 0.32809,
- 0.32677,
- 0.32893,
- 0.32629,
- 0.32723,
- 0.32658,
- 0.32474,
- 0.33155,
- 0.33378,
- 0.3288,
- 0.33409,
- 0.32907,
- 0.32732,
- 0.32661,
- 0.32706,
- 0.51517,
- 0.51886,
- 0.32875,
- 0.32613,
- 0.32755,
- 0.32594,
- 0.32591,
- 0.3275,
- 0.32658,
- 0.32598,
- 0.32571,
- 0.33078,
- 0.32567,
- 0.33064,
- 0.32718,
- 0.32881
- ]
- },
- "num-zeros": {
- "start_step": 0,
- "end_step": 1983,
- "step_interval": 5,
- "values": [
- 951.0,
- 1294.0,
- 1060.0,
- 971.0,
- 901.0,
- 1117.0,
- 1146.0,
- 1481.0,
- 1450.0,
- 1359.0,
- 1524.0,
- 1946.0,
- 2172.0,
- 1538.0,
- 2168.0,
- 1978.0,
- 1941.0,
- 2017.0,
- 2514.0,
- 1951.0,
- 2211.0,
- 2190.0,
- 2499.0,
- 3109.0,
- 2431.0,
- 2741.0,
- 2536.0,
- 2192.0,
- 2064.0,
- 2948.0,
- 2423.0,
- 3485.0,
- 2438.0,
- 2456.0,
- 2498.0,
- 3614.0,
- 2079.0,
- 2299.0,
- 2218.0,
- 2691.0,
- 3765.0,
- 2801.0,
- 2213.0,
- 2801.0,
- 2673.0,
- 2229.0,
- 2614.0,
- 2534.0,
- 2395.0,
- 3023.0,
- 3073.0,
- 2519.0,
- 2574.0,
- 2151.0,
- 2685.0,
- 3348.0,
- 2764.0,
- 2698.0,
- 2394.0,
- 3505.0,
- 2414.0,
- 2978.0,
- 2468.0,
- 2605.0,
- 2317.0,
- 3165.0,
- 2865.0,
- 2919.0,
- 2342.0,
- 2556.0,
- 2184.0,
- 2857.0,
- 2932.0,
- 2812.0,
- 3367.0,
- 2539.0,
- 2770.0,
- 2638.0,
- 3112.0,
- 2799.0,
- 2681.0,
- 2540.0,
- 3130.0,
- 2387.0,
- 2738.0,
- 2862.0,
- 2676.0,
- 2320.0,
- 2382.0,
- 2816.0,
- 2529.0,
- 3200.0,
- 2496.0,
- 2423.0,
- 2581.0,
- 2432.0,
- 2336.0,
- 1902.0,
- 2306.0,
- 2607.0,
- 2764.0,
- 2214.0,
- 2000.0,
- 2180.0,
- 1834.0,
- 2352.0,
- 2325.0,
- 2334.0,
- 2259.0,
- 2077.0,
- 2207.0,
- 2478.0,
- 2327.0,
- 2507.0,
- 2306.0,
- 2729.0,
- 2650.0,
- 2051.0,
- 2485.0,
- 1970.0,
- 2732.0,
- 2407.0,
- 2140.0,
- 2130.0,
- 2047.0,
- 2243.0,
- 1970.0,
- 2569.0,
- 2417.0,
- 2222.0,
- 2205.0,
- 2295.0,
- 2373.0,
- 2311.0,
- 1908.0,
- 2299.0,
- 2581.0,
- 2254.0,
- 2282.0,
- 1506.0,
- 2124.0,
- 2356.0,
- 2072.0,
- 2489.0,
- 2119.0,
- 1906.0,
- 2289.0,
- 1838.0,
- 2039.0,
- 2864.0,
- 2402.0,
- 2108.0,
- 1676.0,
- 1774.0,
- 2390.0,
- 1925.0,
- 2184.0,
- 1979.0,
- 2190.0,
- 2016.0,
- 1830.0,
- 2377.0,
- 1660.0,
- 2153.0,
- 2079.0,
- 1918.0,
- 2331.0,
- 2555.0,
- 1930.0,
- 1627.0,
- 1710.0,
- 1702.0,
- 1998.0,
- 2075.0,
- 1579.0,
- 1644.0,
- 1901.0,
- 2428.0,
- 2111.0,
- 2256.0,
- 2057.0,
- 2184.0,
- 2241.0,
- 2111.0,
- 2126.0,
- 2146.0,
- 1818.0,
- 2432.0,
- 1563.0,
- 1864.0,
- 1830.0,
- 1783.0,
- 1874.0,
- 1963.0,
- 1715.0,
- 2022.0,
- 2143.0,
- 2015.0,
- 1604.0,
- 2044.0,
- 1998.0,
- 2159.0,
- 2247.0,
- 2858.0,
- 2284.0,
- 2138.0,
- 2515.0,
- 2295.0,
- 2514.0,
- 1794.0,
- 2096.0,
- 2257.0,
- 2612.0,
- 2054.0,
- 2084.0,
- 2161.0,
- 2071.0,
- 1911.0,
- 1998.0,
- 2301.0,
- 2014.0,
- 2010.0,
- 1940.0,
- 2338.0,
- 2206.0,
- 2436.0,
- 2084.0,
- 2300.0,
- 1838.0,
- 2266.0,
- 2007.0,
- 2320.0,
- 1960.0,
- 2174.0,
- 2067.0,
- 1904.0,
- 2017.0,
- 1784.0,
- 1804.0,
- 2096.0,
- 2006.0,
- 2020.0,
- 1881.0,
- 2441.0,
- 2440.0,
- 2196.0,
- 1856.0,
- 2861.0,
- 2097.0,
- 2002.0,
- 1886.0,
- 1765.0,
- 2257.0,
- 2195.0,
- 1946.0,
- 1758.0,
- 2432.0,
- 1695.0,
- 2473.0,
- 1924.0,
- 1741.0,
- 1858.0,
- 2479.0,
- 2441.0,
- 2083.0,
- 2289.0,
- 2251.0,
- 1860.0,
- 1983.0,
- 1939.0,
- 2148.0,
- 2379.0,
- 2339.0,
- 2165.0,
- 2381.0,
- 2161.0,
- 1997.0,
- 1732.0,
- 1901.0,
- 1990.0,
- 2229.0,
- 2281.0,
- 2032.0,
- 2062.0,
- 2072.0,
- 2291.0,
- 2069.0,
- 1668.0,
- 1720.0,
- 2157.0,
- 2187.0,
- 2037.0,
- 2461.0,
- 2170.0,
- 2121.0,
- 2135.0,
- 1806.0,
- 2596.0,
- 2088.0,
- 2654.0,
- 1959.0,
- 1994.0,
- 1881.0,
- 1998.0,
- 2453.0,
- 1943.0,
- 2221.0,
- 2296.0,
- 1837.0,
- 1837.0,
- 2352.0,
- 2099.0,
- 2125.0,
- 2191.0,
- 2173.0,
- 1981.0,
- 2218.0,
- 1957.0,
- 2445.0,
- 2377.0,
- 2214.0,
- 2626.0,
- 2131.0,
- 2373.0,
- 2530.0,
- 2365.0,
- 2106.0,
- 1956.0,
- 2205.0,
- 2115.0,
- 2344.0,
- 2587.0,
- 2484.0,
- 2203.0,
- 2093.0,
- 2128.0,
- 2109.0,
- 2625.0,
- 2027.0,
- 2489.0,
- 2424.0,
- 2757.0,
- 2901.0,
- 2295.0,
- 2267.0,
- 2149.0,
- 2081.0,
- 2612.0,
- 2195.0,
- 2530.0,
- 1823.0,
- 2341.0,
- 2129.0,
- 2062.0,
- 2221.0,
- 2154.0,
- 2172.0,
- 2180.0,
- 2068.0,
- 2300.0,
- 2189.0,
- 2218.0,
- 2369.0,
- 2760.0,
- 2058.0,
- 2860.0,
- 2391.0,
- 2134.0,
- 2120.0,
- 2116.0,
- 2631.0,
- 2251.0,
- 2356.0,
- 2021.0,
- 2205.0,
- 2097.0,
- 2232.0,
- 2222.0,
- 2815.0,
- 2508.0,
- 2266.0,
- 2552.0,
- 2356.0,
- 2411.0,
- 2664.0,
- 2454.0,
- 2733.0,
- 2655.0,
- 2190.0,
- 2416.0,
- 2274.0,
- 2189.0,
- 2105.0,
- 2436.0,
- 2280.0,
- 2628.0,
- 2479.0
- ]
- }
-}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_bf16_baseline/golden_values_lts_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_bf16_baseline/golden_values_lts_dgx_a100.json
deleted file mode 100644
index fdeaa49aa1b..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_bf16_baseline/golden_values_lts_dgx_a100.json
+++ /dev/null
@@ -1 +0,0 @@
-{"forward-backward-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [17.4566, 0.37175, 0.37134, 0.37017, 0.37156, 0.37759, 0.37765, 0.37162, 0.3761, 0.37226, 0.53616, 0.37589, 0.37516, 0.37683, 0.37327, 0.37614, 0.37342, 0.3739, 0.37649, 0.37491, 0.38081, 0.37232, 0.37401, 0.37224, 0.37132, 0.38167, 0.37456, 0.37215, 0.36647, 0.37435, 0.38453, 0.36353, 0.36605, 0.36205, 0.36329, 0.36758, 0.36245, 0.36564, 0.3674, 0.38594, 0.36767, 0.36685, 0.36727, 0.36428, 0.3664, 0.36716, 0.36619, 0.36593, 0.36805, 0.36393, 0.3666, 0.36486, 0.36817, 0.36273, 0.36485, 0.36634, 0.36443, 0.3672, 0.36462, 0.36335, 0.35994, 0.36774, 0.36167, 0.36089, 0.36216, 0.36236, 0.36412, 0.36497, 0.3673, 0.36303, 0.36566, 0.36239, 0.36323, 0.36008, 0.46258, 0.36181, 0.3621, 0.36509, 0.36772, 0.36417, 0.36489, 0.36688, 0.3704, 0.36443, 0.36411, 0.36221, 0.36185, 0.36498, 0.36202, 0.36553, 0.36574, 0.36507, 0.37335, 0.36256, 0.3648, 0.36324, 0.36253, 0.36685, 0.3644, 0.36463, 0.36584, 0.36426, 0.36134, 0.36175, 0.45788, 0.36568, 0.36196, 0.38364, 0.36164, 0.36331, 0.36346, 0.3683, 0.36544, 0.36245, 0.37051, 0.37092, 0.36741, 0.3695, 0.3651, 0.37195, 0.36315, 0.36425, 0.36904, 0.36828, 0.3648, 0.36763, 0.36895, 0.37272, 0.3749, 0.36753, 0.36573, 0.36845, 0.36886, 0.37096, 0.47625, 0.36339, 0.36255, 0.36368, 0.44639, 0.51442, 0.3673, 0.36637, 0.36885, 0.37285, 0.36987, 0.36631, 0.36485, 0.36259, 0.36217, 0.364, 0.36364, 0.36588, 0.3619, 0.36604, 0.36798, 0.36772, 0.36665, 0.36769, 0.36628, 0.36592, 0.36831, 0.36583, 0.36842, 0.36695, 0.37069, 0.36526, 0.36421, 0.3661, 0.36543, 0.36845, 0.36581, 0.3674, 0.36575, 0.36568, 0.36949, 0.36761, 0.36684, 0.36852, 0.36408, 0.37073, 0.36602, 0.36769, 0.3609, 0.36264, 0.36736, 0.36549, 0.36517, 0.36003, 0.36081, 0.36006, 0.36167, 0.36361, 0.36172, 0.36296, 0.36716, 0.36645, 0.36705, 0.36621, 0.45574, 0.36247, 0.36105, 0.36408, 0.3621, 0.36088, 0.36271, 0.36349, 0.36811, 0.36958, 0.36968, 0.36582, 0.36294, 0.36436, 0.36894, 0.36266, 0.36585, 0.36633, 0.36462, 0.36885, 0.36711, 0.36754, 0.36317, 0.36285, 0.36581, 0.37564, 0.37346, 0.3622, 0.36404, 0.45901, 0.36362, 0.36726, 0.37058, 0.36812, 0.36666, 0.37189, 0.46883, 0.37275, 0.3719, 0.36704, 0.36448, 0.3629, 0.36582, 0.36225, 0.36061, 0.4845, 0.36483, 0.36652, 0.36811, 0.36819, 0.37464, 0.36516, 0.36721, 0.36426, 0.35999, 0.36267, 0.36286, 0.36833, 0.36584, 0.3632, 0.36415, 0.36569, 0.37494, 0.36226, 0.46516, 0.36495, 0.36254, 0.36943, 0.36585, 0.36664, 0.36827, 0.36557, 0.37484, 0.36946, 0.37108, 0.36825, 0.36775, 0.36137, 0.36521, 0.3697, 0.36415, 0.36338, 0.36383, 0.36505, 0.3677, 0.36976, 0.36576, 0.36964, 0.37212, 0.36584, 0.36475, 0.36537, 0.36914, 0.36892, 0.45897, 0.36567, 0.3641, 0.36657, 0.3698, 0.36867, 0.36599, 0.3679, 0.36742, 0.36813, 0.36659, 0.36737, 0.36653, 0.36785, 0.37243, 0.36895, 0.37086, 0.365, 0.36719, 0.37471, 0.36717, 0.3738, 0.37016, 0.37206, 0.3695, 0.36911, 0.36946, 0.36669, 0.36636, 0.3628, 0.3661, 0.36516, 0.36275, 0.3657, 0.3654, 0.36521, 0.3662, 0.4682, 0.36931, 0.3668, 0.37172, 0.37189, 0.36942, 0.37165, 0.37159, 0.37333, 0.37491, 0.37221, 0.36907, 0.37154, 0.37633, 0.36937, 0.36886, 0.36922, 0.36659, 0.36692, 0.36765, 0.36709, 0.3641, 0.36625, 0.36742, 0.36073, 0.36646, 0.36662, 0.36508, 0.37343, 0.36701, 0.3642, 0.36688, 0.36861, 0.36833, 0.36153, 0.36529, 0.36657, 0.36866, 0.37542, 0.36846, 0.36817, 0.36445, 0.36398, 0.36799, 0.36631, 0.3632, 0.36525, 0.36782, 0.36786, 0.37064, 0.36604, 0.36767, 0.36737, 0.36678, 0.36919, 0.36757, 0.36912, 0.36819, 0.46929, 0.37321, 0.37017, 0.4569, 0.36994, 0.37357, 0.36984, 0.57706, 0.37035, 0.37045, 0.36802, 0.36852, 0.36742]}, "forward-compute-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [9.27486, 0.20418, 0.20397, 0.20285, 0.20434, 0.20758, 0.20634, 0.20416, 0.20426, 0.20434, 0.3669, 0.20758, 0.20442, 0.20546, 0.20278, 0.20684, 0.20447, 0.20408, 0.20756, 0.20602, 0.20443, 0.20251, 0.20574, 0.20384, 0.2029, 0.21254, 0.21029, 0.20601, 0.20107, 0.20291, 0.20989, 0.19612, 0.20052, 0.19662, 0.19784, 0.20061, 0.19675, 0.19997, 0.20194, 0.22257, 0.2025, 0.20076, 0.2025, 0.20065, 0.20083, 0.19995, 0.19982, 0.20085, 0.20083, 0.19933, 0.20226, 0.20132, 0.203, 0.19623, 0.1999, 0.19978, 0.1976, 0.19962, 0.19949, 0.19977, 0.19439, 0.19749, 0.19772, 0.19546, 0.19711, 0.19707, 0.19839, 0.19731, 0.20084, 0.19819, 0.2011, 0.1983, 0.19858, 0.1937, 0.29471, 0.19528, 0.19534, 0.19901, 0.20146, 0.19982, 0.19907, 0.20086, 0.20405, 0.19915, 0.2005, 0.19581, 0.19278, 0.19863, 0.19822, 0.1993, 0.1988, 0.19998, 0.2005, 0.19725, 0.20091, 0.19918, 0.19836, 0.2016, 0.19765, 0.19811, 0.19903, 0.19646, 0.19645, 0.19682, 0.28975, 0.19888, 0.19522, 0.21159, 0.19644, 0.19881, 0.19777, 0.20279, 0.19972, 0.19755, 0.20374, 0.20397, 0.20052, 0.20409, 0.20046, 0.20573, 0.19813, 0.19893, 0.20396, 0.20108, 0.1991, 0.20018, 0.20247, 0.20606, 0.20496, 0.20146, 0.20113, 0.20109, 0.20373, 0.20131, 0.30688, 0.19978, 0.19719, 0.19856, 0.27425, 0.34575, 0.20073, 0.20027, 0.20292, 0.20753, 0.20162, 0.19901, 0.19974, 0.19616, 0.19556, 0.19818, 0.19745, 0.20023, 0.19768, 0.1993, 0.20152, 0.20191, 0.20046, 0.19952, 0.19909, 0.20067, 0.20206, 0.20028, 0.2009, 0.20109, 0.20231, 0.20057, 0.19849, 0.2014, 0.19862, 0.20162, 0.1995, 0.20168, 0.19859, 0.20023, 0.20137, 0.19954, 0.19893, 0.20032, 0.19926, 0.20288, 0.20082, 0.20203, 0.1964, 0.19744, 0.20075, 0.19839, 0.19941, 0.19592, 0.19584, 0.19507, 0.19602, 0.19868, 0.19785, 0.19642, 0.20146, 0.20135, 0.20162, 0.20061, 0.28565, 0.19898, 0.19699, 0.20018, 0.1975, 0.19765, 0.19836, 0.20012, 0.20347, 0.20455, 0.20461, 0.20103, 0.1993, 0.20097, 0.20324, 0.19779, 0.20128, 0.20136, 0.19977, 0.20189, 0.20216, 0.19869, 0.19833, 0.19963, 0.20166, 0.21162, 0.2062, 0.19807, 0.19895, 0.29325, 0.19845, 0.1994, 0.20325, 0.20285, 0.20049, 0.20554, 0.30108, 0.20617, 0.20644, 0.20131, 0.20084, 0.19867, 0.20111, 0.19928, 0.19687, 0.31861, 0.20096, 0.20262, 0.20309, 0.20325, 0.20819, 0.20113, 0.20301, 0.19969, 0.19603, 0.19693, 0.19763, 0.2004, 0.20179, 0.19742, 0.19937, 0.20128, 0.20616, 0.19831, 0.29924, 0.19973, 0.19859, 0.20413, 0.20138, 0.20285, 0.20388, 0.20206, 0.20671, 0.20471, 0.20646, 0.20241, 0.20408, 0.19861, 0.20125, 0.20732, 0.20159, 0.20035, 0.20096, 0.20012, 0.20294, 0.20424, 0.20101, 0.20564, 0.2044, 0.2008, 0.19955, 0.20264, 0.2049, 0.20446, 0.293, 0.20181, 0.20025, 0.20162, 0.20369, 0.20417, 0.20115, 0.20265, 0.20363, 0.2044, 0.20297, 0.20322, 0.20046, 0.20222, 0.20483, 0.20332, 0.20676, 0.19998, 0.2015, 0.2054, 0.20246, 0.20845, 0.20406, 0.20619, 0.20592, 0.20453, 0.20274, 0.20274, 0.20162, 0.20007, 0.20274, 0.20276, 0.19873, 0.20293, 0.20198, 0.20198, 0.20314, 0.30676, 0.20607, 0.2049, 0.20889, 0.20967, 0.2072, 0.20824, 0.20768, 0.20857, 0.20862, 0.20898, 0.20615, 0.20827, 0.21418, 0.20637, 0.20388, 0.2067, 0.20272, 0.20336, 0.20429, 0.20148, 0.20112, 0.20264, 0.20322, 0.19861, 0.20195, 0.20314, 0.1996, 0.20578, 0.2036, 0.20073, 0.20362, 0.20652, 0.20449, 0.19954, 0.20273, 0.203, 0.2032, 0.20757, 0.2034, 0.20482, 0.19991, 0.20078, 0.20474, 0.20356, 0.19886, 0.20118, 0.20177, 0.20291, 0.20253, 0.20141, 0.20341, 0.20352, 0.20319, 0.20478, 0.20413, 0.20568, 0.20319, 0.30235, 0.20813, 0.20681, 0.29099, 0.20567, 0.20759, 0.20528, 0.41177, 0.20714, 0.20416, 0.20342, 0.20429, 0.20393]}, "backward-compute-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.48483, 0.17652, 0.17828, 0.17737, 0.17731, 0.18012, 0.18059, 0.17933, 0.18228, 0.17963, 0.17741, 0.17905, 0.17875, 0.18023, 0.17598, 0.17735, 0.17563, 0.1774, 0.17814, 0.17775, 0.1797, 0.17589, 0.17512, 0.17493, 0.17423, 0.17574, 0.17442, 0.17392, 0.17429, 0.18376, 0.17762, 0.17577, 0.17608, 0.17519, 0.17371, 0.17562, 0.1743, 0.17634, 0.17747, 0.1794, 0.17639, 0.1769, 0.17749, 0.17644, 0.17597, 0.17611, 0.17772, 0.17605, 0.17799, 0.1756, 0.17762, 0.17478, 0.17987, 0.17366, 0.17669, 0.17775, 0.17802, 0.17908, 0.17514, 0.17554, 0.17388, 0.17483, 0.17431, 0.17275, 0.17497, 0.17541, 0.17514, 0.17686, 0.17728, 0.17469, 0.17508, 0.17519, 0.17517, 0.17377, 0.17594, 0.17621, 0.17553, 0.17702, 0.18, 0.17602, 0.17593, 0.17864, 0.17997, 0.1755, 0.17822, 0.17772, 0.17671, 0.17725, 0.1778, 0.17809, 0.17954, 0.17593, 0.17541, 0.17441, 0.17679, 0.17798, 0.17778, 0.17724, 0.17552, 0.17811, 0.18023, 0.17981, 0.17557, 0.17566, 0.17625, 0.17625, 0.17558, 0.19425, 0.1762, 0.17767, 0.17763, 0.18372, 0.17971, 0.17752, 0.18218, 0.18258, 0.18042, 0.18083, 0.17934, 0.18263, 0.17612, 0.17585, 0.18209, 0.17892, 0.17504, 0.18056, 0.18269, 0.18216, 0.18105, 0.18046, 0.17895, 0.18001, 0.18287, 0.18048, 0.18107, 0.1792, 0.177, 0.17595, 0.17833, 0.17997, 0.18026, 0.18064, 0.18103, 0.18122, 0.1807, 0.17741, 0.17696, 0.175, 0.17708, 0.17762, 0.17496, 0.17994, 0.17504, 0.17879, 0.18178, 0.1796, 0.18007, 0.18397, 0.18212, 0.18076, 0.18234, 0.18066, 0.18359, 0.18244, 0.18094, 0.18093, 0.17869, 0.18132, 0.18028, 0.18293, 0.17692, 0.181, 0.1778, 0.178, 0.18006, 0.18483, 0.18337, 0.18495, 0.18069, 0.18012, 0.18124, 0.18343, 0.17705, 0.17668, 0.17849, 0.18112, 0.17754, 0.1764, 0.17576, 0.17489, 0.17603, 0.17867, 0.17875, 0.17778, 0.17783, 0.18028, 0.18098, 0.18147, 0.18117, 0.17707, 0.17356, 0.17855, 0.17723, 0.175, 0.17556, 0.17674, 0.17749, 0.17698, 0.17866, 0.17541, 0.17473, 0.17725, 0.17976, 0.17814, 0.17815, 0.17912, 0.17571, 0.18059, 0.18163, 0.17964, 0.17657, 0.1773, 0.17872, 0.18756, 0.18502, 0.17691, 0.17601, 0.1773, 0.17751, 0.17745, 0.18072, 0.17998, 0.17849, 0.18172, 0.17785, 0.18296, 0.17966, 0.18029, 0.17622, 0.17684, 0.17683, 0.17525, 0.17514, 0.17546, 0.17768, 0.17616, 0.17827, 0.17873, 0.18236, 0.17864, 0.17902, 0.17866, 0.17537, 0.17824, 0.17634, 0.17765, 0.17745, 0.17691, 0.17855, 0.17773, 0.1776, 0.17553, 0.17612, 0.17682, 0.17445, 0.17573, 0.17792, 0.17697, 0.17758, 0.17799, 0.18179, 0.17862, 0.17828, 0.17902, 0.17716, 0.17378, 0.17466, 0.17969, 0.17531, 0.17449, 0.1762, 0.17533, 0.17786, 0.17799, 0.1739, 0.17695, 0.17997, 0.17727, 0.17594, 0.17599, 0.17877, 0.17835, 0.17768, 0.17619, 0.1761, 0.17947, 0.18082, 0.17999, 0.17973, 0.18161, 0.17878, 0.18107, 0.17669, 0.17787, 0.17714, 0.17987, 0.17952, 0.18139, 0.1814, 0.17879, 0.17819, 0.17967, 0.17842, 0.18204, 0.17981, 0.18039, 0.1779, 0.17786, 0.18096, 0.17907, 0.17853, 0.17539, 0.17682, 0.17666, 0.17653, 0.17793, 0.17688, 0.1782, 0.17909, 0.17471, 0.17743, 0.17531, 0.17878, 0.17697, 0.1762, 0.17958, 0.17827, 0.17938, 0.17923, 0.17797, 0.1763, 0.17776, 0.18097, 0.17754, 0.18018, 0.17934, 0.1806, 0.1751, 0.17845, 0.18106, 0.17667, 0.17809, 0.17911, 0.17624, 0.17874, 0.1795, 0.17661, 0.18214, 0.18117, 0.17941, 0.17482, 0.17595, 0.17616, 0.17509, 0.17725, 0.17932, 0.18085, 0.18292, 0.17986, 0.17974, 0.17799, 0.17756, 0.17851, 0.17744, 0.17724, 0.17992, 0.18197, 0.18128, 0.1816, 0.17718, 0.1781, 0.18028, 0.17962, 0.18211, 0.17904, 0.18027, 0.179, 0.1805, 0.18514, 0.18111, 0.17608, 0.18024, 0.1833, 0.1823, 0.1797, 0.17902, 0.18251, 0.18061, 0.17877, 0.17926]}, "batch-generator-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.60562, 0.0038, 0.00384, 0.00379, 0.00392, 0.00392, 0.00391, 0.00387, 0.00391, 0.00397, 0.00392, 0.00405, 0.00383, 0.00388, 0.00387, 0.0042, 0.00394, 0.00394, 0.00387, 0.00379, 0.00413, 0.00393, 0.00403, 0.00383, 0.00384, 0.004, 0.0044, 0.00355, 0.00419, 0.00392, 0.00399, 0.00394, 0.0037, 0.00364, 0.00369, 0.00383, 0.00379, 0.00369, 0.0038, 0.00364, 0.00377, 0.00393, 0.00365, 0.00367, 0.00383, 0.00366, 0.00382, 0.00371, 0.00355, 0.00439, 0.00359, 0.00368, 0.00365, 0.00383, 0.00363, 0.00374, 0.00373, 0.00378, 0.00373, 0.00352, 0.00362, 0.0036, 0.00343, 0.00349, 0.00382, 0.00374, 0.00356, 0.00374, 0.00365, 0.00391, 0.0037, 0.00375, 0.00369, 0.00366, 0.00397, 0.00372, 0.00358, 0.00365, 0.00406, 0.00355, 0.00339, 0.00398, 0.00424, 0.0036, 0.00363, 0.00389, 0.00371, 0.00377, 0.00362, 0.00383, 0.00373, 0.0037, 0.00388, 0.00356, 0.00358, 0.00363, 0.00387, 0.00375, 0.00383, 0.00372, 0.00369, 0.00374, 0.00411, 0.00364, 0.0039, 0.00376, 0.00383, 0.00364, 0.00379, 0.00378, 0.00364, 0.00365, 0.00392, 0.00347, 0.00361, 0.00377, 0.00359, 0.00364, 0.00383, 0.00375, 0.00368, 0.00367, 0.0041, 0.00379, 0.00359, 0.00366, 0.00379, 0.00376, 0.00387, 0.00368, 0.00361, 0.00375, 0.00401, 0.0038, 0.00393, 0.00377, 0.00358, 0.00402, 0.00479, 0.00399, 0.00374, 0.00392, 0.00379, 0.00391, 0.00355, 0.00378, 0.00356, 0.00362, 0.0036, 0.00351, 0.00348, 0.00422, 0.00355, 0.00359, 0.00351, 0.00373, 0.00362, 0.00377, 0.00378, 0.00386, 0.0037, 0.00367, 0.00361, 0.0038, 0.00392, 0.00338, 0.00354, 0.00357, 0.00375, 0.00369, 0.0038, 0.0036, 0.00386, 0.00388, 0.00354, 0.00367, 0.00381, 0.00354, 0.00366, 0.0038, 0.00367, 0.00378, 0.00363, 0.00368, 0.00358, 0.00359, 0.00373, 0.00355, 0.00402, 0.00361, 0.00364, 0.00369, 0.0035, 0.00356, 0.00387, 0.00375, 0.00381, 0.0038, 0.00396, 0.00375, 0.03419, 0.00346, 0.00373, 0.00413, 0.0035, 0.00359, 0.00362, 0.00344, 0.00367, 0.00349, 0.00362, 0.00369, 0.00353, 0.00388, 0.00372, 0.00358, 0.0036, 0.00347, 0.00344, 0.00368, 0.00381, 0.00355, 0.00366, 0.0035, 0.00362, 0.00372, 0.0037, 0.00382, 0.00365, 0.00381, 0.00385, 0.00362, 0.00358, 0.00369, 0.00374, 0.00368, 0.00355, 0.00377, 0.00348, 0.00351, 0.00355, 0.00339, 0.00354, 0.00335, 0.00357, 0.00367, 0.00363, 0.00377, 0.00357, 0.00363, 0.00374, 0.00361, 0.00358, 0.00354, 0.00336, 0.00361, 0.00371, 0.00365, 0.00354, 0.00394, 0.00379, 0.00378, 0.00379, 0.00401, 0.00398, 0.00384, 0.00395, 0.0042, 0.00424, 0.00421, 0.00426, 0.00442, 0.00415, 0.00404, 0.0043, 0.00406, 0.00434, 0.00442, 0.00416, 0.0043, 0.00409, 0.00403, 0.00412, 0.004, 0.00407, 0.00448, 0.00415, 0.00407, 0.0041, 0.0041, 0.00402, 0.00417, 0.00421, 0.00402, 0.00399, 0.00398, 0.00422, 0.00414, 0.00414, 0.00417, 0.00412, 0.004, 0.00405, 0.00393, 0.00399, 0.00391, 0.00392, 0.00387, 0.00417, 0.00413, 0.00408, 0.004, 0.00415, 0.00409, 0.00421, 0.00397, 0.00405, 0.00396, 0.00405, 0.00404, 0.00407, 0.00408, 0.00399, 0.004, 0.00392, 0.00412, 0.00432, 0.00438, 0.00426, 0.00415, 0.00429, 0.00422, 0.00401, 0.00419, 0.0041, 0.00398, 0.00406, 0.00453, 0.00398, 0.00413, 0.00404, 0.00406, 0.00404, 0.00404, 0.0041, 0.00409, 0.00402, 0.00399, 0.0041, 0.00413, 0.00436, 0.00417, 0.00418, 0.00424, 0.00423, 0.00429, 0.00425, 0.00417, 0.00427, 0.00432, 0.00421, 0.00425, 0.00421, 0.00433, 0.00423, 0.00439, 0.00428, 0.00423, 0.00424, 0.0041, 0.00423, 0.00424, 0.00433, 0.00424, 0.00436, 0.0043, 0.00407, 0.00429, 0.0041, 0.00429, 0.00431, 0.00428, 0.0043, 0.00425, 0.00416, 0.00427, 0.00405, 0.00443, 0.00417, 0.0042, 0.00449, 0.00406, 0.004, 0.00406, 0.0042, 0.00421, 0.00409, 0.00421, 0.00421, 0.00413]}, "layernorm-grads-all-reduce-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [3e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 2e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05]}, "embedding-grads-all-reduce-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [4e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 5e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05]}, "all-grads-sync-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [5.81083, 0.0018, 0.00179, 0.00169, 0.00153, 0.00181, 0.00157, 0.00183, 0.00159, 0.00178, 0.00159, 0.00178, 0.00153, 0.00181, 0.0016, 0.0018, 0.00158, 0.00176, 0.00155, 0.00182, 0.00162, 0.00179, 0.00159, 0.00178, 0.0016, 0.00183, 0.00159, 0.00181, 0.0016, 0.00181, 0.00161, 0.0018, 0.00156, 0.00165, 0.0016, 0.00177, 0.00157, 0.00177, 0.00159, 0.00175, 0.00158, 0.00178, 0.00159, 0.00182, 0.00158, 0.00177, 0.00158, 0.00177, 0.00159, 0.00179, 0.00155, 0.00183, 0.00158, 0.00178, 0.00156, 0.00181, 0.00154, 0.0018, 0.00154, 0.00178, 0.00159, 0.00181, 0.00157, 0.00181, 0.00155, 0.00183, 0.00159, 0.0018, 0.00155, 0.00179, 0.00158, 0.00181, 0.00159, 0.00179, 0.00153, 0.00178, 0.00157, 0.00178, 0.00156, 0.00176, 0.00156, 0.00179, 0.00157, 0.00182, 0.00152, 0.00181, 0.00152, 0.00183, 0.00157, 0.00179, 0.00159, 0.00187, 0.00159, 0.00182, 0.00156, 0.0018, 0.00161, 0.0018, 0.00157, 0.00176, 0.00159, 0.00179, 0.00157, 0.00182, 0.00158, 0.0018, 0.0016, 0.00182, 0.00159, 0.00172, 0.00157, 0.00179, 0.00154, 0.00166, 0.00158, 0.00176, 0.00159, 0.00184, 0.00156, 0.00179, 0.00157, 0.00174, 0.00157, 0.00173, 0.00157, 0.0018, 0.00159, 0.00181, 0.00156, 0.00183, 0.00157, 0.00181, 0.00158, 0.00179, 0.00157, 0.00184, 0.00158, 0.00174, 0.00163, 0.00175, 0.00158, 0.0018, 0.00152, 0.00183, 0.00158, 0.00174, 0.00159, 0.00179, 0.00155, 0.00182, 0.00157, 0.0018, 0.00159, 0.00183, 0.00156, 0.00181, 0.00158, 0.00176, 0.00158, 0.00176, 0.00156, 0.00178, 0.00158, 0.00181, 0.00153, 0.0018, 0.00155, 0.0018, 0.0016, 0.0019, 0.0016, 0.00175, 0.0016, 0.0018, 0.00153, 0.00178, 0.00158, 0.0018, 0.00156, 0.00172, 0.00159, 0.00182, 0.00157, 0.00175, 0.00157, 0.00173, 0.00156, 0.00186, 0.00158, 0.00178, 0.00158, 0.00188, 0.00159, 0.00181, 0.00153, 0.00175, 0.00155, 0.00181, 0.00156, 0.00181, 0.00177, 0.00157, 0.00162, 0.00165, 0.00173, 0.00157, 0.00173, 0.00165, 0.00167, 0.00151, 0.00172, 0.00167, 0.00174, 0.00157, 0.00168, 0.00168, 0.00174, 0.00157, 0.00175, 0.00166, 0.00174, 0.00154, 0.00174, 0.00167, 0.00171, 0.00159, 0.00174, 0.00165, 0.00173, 0.00159, 0.00174, 0.00162, 0.00175, 0.00157, 0.00174, 0.00167, 0.00172, 0.00156, 0.00174, 0.00164, 0.00175, 0.00154, 0.00161, 0.0016, 0.00174, 0.00156, 0.00179, 0.00167, 0.00167, 0.00155, 0.00175, 0.00167, 0.00173, 0.00158, 0.00176, 0.00166, 0.00173, 0.00157, 0.00173, 0.00161, 0.00176, 0.0016, 0.00168, 0.00162, 0.00174, 0.00158, 0.00174, 0.00167, 0.00174, 0.00158, 0.00168, 0.00161, 0.00175, 0.00159, 0.00173, 0.00168, 0.00175, 0.00158, 0.00174, 0.00163, 0.00176, 0.00153, 0.00175, 0.00168, 0.00168, 0.00153, 0.00172, 0.00165, 0.00175, 0.00159, 0.00174, 0.00164, 0.00176, 0.00153, 0.00171, 0.00162, 0.00173, 0.00156, 0.00174, 0.00165, 0.00168, 0.00158, 0.00174, 0.00167, 0.00176, 0.00158, 0.00175, 0.00167, 0.00174, 0.00158, 0.00168, 0.00166, 0.00173, 0.00157, 0.00176, 0.00161, 0.00173, 0.00159, 0.00178, 0.00165, 0.00174, 0.00156, 0.00167, 0.00163, 0.00165, 0.00158, 0.00173, 0.00162, 0.00176, 0.00157, 0.00173, 0.00166, 0.00173, 0.0016, 0.0018, 0.00165, 0.00172, 0.00159, 0.00168, 0.00165, 0.00175, 0.00154, 0.00171, 0.00164, 0.00169, 0.00153, 0.00175, 0.00166, 0.00175, 0.00159, 0.00176, 0.00164, 0.00172, 0.00159, 0.00169, 0.00166, 0.00173, 0.00153, 0.00167, 0.00164, 0.00172, 0.00159, 0.00167, 0.00168, 0.00175, 0.00157, 0.00173, 0.00167, 0.00172, 0.0016, 0.00173, 0.00166, 0.00175, 0.00153, 0.00174, 0.00163, 0.00172, 0.00157, 0.00167, 0.00165, 0.00171, 0.00159, 0.00175, 0.00166, 0.00166, 0.00158, 0.00166, 0.00164, 0.00167, 0.00157, 0.0017, 0.00168, 0.00169, 0.00158, 0.00176, 0.00168, 0.00172, 0.00157, 0.00173, 0.00167]}, "optimizer-copy-to-main-grad-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00181, 0.00152, 0.00153, 0.0015, 0.00157, 0.00156, 0.00152, 0.00157, 0.00162, 0.0015, 0.00152, 0.00155, 0.00152, 0.00155, 0.00155, 0.00161, 0.00151, 0.00151, 0.00196, 0.0015, 0.00161, 0.0015, 0.00162, 0.00161, 0.00157, 0.00151, 0.0015, 0.0015, 0.00156, 0.00153, 0.00171, 0.00252, 0.00165, 0.0018, 0.00159, 0.00153, 0.00157, 0.00159, 0.00159, 0.00157, 0.00156, 0.00163, 0.00152, 0.0015, 0.00163, 0.00153, 0.00149, 0.00156, 0.00156, 0.00152, 0.00157, 0.00152, 0.0016, 0.00159, 0.00155, 0.00157, 0.00157, 0.00156, 0.00151, 0.00156, 0.00152, 0.00151, 0.00157, 0.00157, 0.00163, 0.00153, 0.00158, 0.00155, 0.00149, 0.00161, 0.0015, 0.00156, 0.00151, 0.00162, 0.00158, 0.00148, 0.00156, 0.0015, 0.00157, 0.00151, 0.00155, 0.00155, 0.00161, 0.0027, 0.00157, 0.00156, 0.00156, 0.00151, 0.00156, 0.00149, 0.00158, 0.0015, 0.00152, 0.00156, 0.00155, 0.0024, 0.00156, 0.0016, 0.00156, 0.0015, 0.0016, 0.00155, 0.00151, 0.00154, 0.00158, 0.0015, 0.0015, 0.00155, 0.00156, 0.00155, 0.00157, 0.0015, 0.0015, 0.00155, 0.00157, 0.00155, 0.00157, 0.0015, 0.00157, 0.00155, 0.00155, 0.0015, 0.00164, 0.0016, 0.00151, 0.0015, 0.00165, 0.00151, 0.00157, 0.00157, 0.00158, 0.00154, 0.00157, 0.0016, 0.0016, 0.00149, 0.00154, 0.00156, 0.00333, 0.00159, 0.00153, 0.00149, 0.00149, 0.00166, 0.00165, 0.00158, 0.00149, 0.00155, 0.00152, 0.00155, 0.00156, 0.00152, 0.00155, 0.00156, 0.00164, 0.00155, 0.00156, 0.00152, 0.00166, 0.00153, 0.0015, 0.0015, 0.00155, 0.00156, 0.00158, 0.00149, 0.00165, 0.00155, 0.0015, 0.0015, 0.0015, 0.00154, 0.00155, 0.00165, 0.00156, 0.00155, 0.0015, 0.00148, 0.00154, 0.00156, 0.00156, 0.0015, 0.00148, 0.00157, 0.00152, 0.0015, 0.00149, 0.00157, 0.00149, 0.00149, 0.0015, 0.0028, 0.0015, 0.00151, 0.00157, 0.00155, 0.00148, 0.0015, 0.00169, 0.00149, 0.0015, 0.00159, 0.00155, 0.00149, 0.0015, 0.00148, 0.00149, 0.00154, 0.00155, 0.00149, 0.00147, 0.00149, 0.00156, 0.00148, 0.00146, 0.00151, 0.00152, 0.00147, 0.00147, 0.00147, 0.00155, 0.00147, 0.00148, 0.00144, 0.0015, 0.0015, 0.00159, 0.00156, 0.00149, 0.00151, 0.0016, 0.00149, 0.0015, 0.00154, 0.0015, 0.00147, 0.00147, 0.00154, 0.00156, 0.00153, 0.0015, 0.0015, 0.002, 0.00151, 0.00246, 0.0015, 0.00147, 0.00144, 0.00148, 0.00171, 0.00148, 0.0015, 0.00157, 0.00174, 0.00156, 0.00157, 0.00148, 0.00147, 0.00149, 0.00148, 0.0015, 0.00148, 0.00151, 0.00158, 0.00149, 0.00147, 0.00153, 0.00151, 0.00154, 0.00148, 0.00157, 0.00157, 0.00148, 0.0016, 0.00153, 0.00155, 0.00156, 0.00157, 0.00149, 0.00154, 0.00148, 0.00151, 0.00149, 0.00155, 0.00148, 0.00155, 0.00155, 0.0015, 0.00149, 0.0015, 0.00149, 0.00153, 0.00164, 0.0016, 0.0015, 0.00153, 0.00149, 0.00158, 0.00154, 0.00149, 0.00154, 0.00165, 0.00151, 0.00148, 0.00158, 0.00157, 0.00158, 0.0015, 0.00149, 0.00154, 0.00152, 0.00155, 0.00158, 0.00149, 0.00157, 0.0015, 0.00158, 0.00163, 0.00159, 0.00158, 0.00159, 0.00157, 0.00157, 0.0015, 0.00151, 0.00151, 0.00154, 0.00154, 0.00159, 0.00155, 0.00155, 0.00148, 0.00198, 0.00154, 0.00149, 0.00156, 0.00151, 0.00157, 0.00149, 0.00148, 0.00151, 0.00154, 0.00153, 0.00148, 0.00151, 0.00149, 0.0015, 0.00155, 0.00155, 0.00151, 0.00156, 0.00154, 0.0015, 0.0015, 0.00151, 0.00157, 0.00156, 0.00158, 0.0015, 0.00155, 0.00148, 0.00153, 0.00151, 0.0015, 0.0015, 0.00152, 0.00151, 0.00156, 0.00158, 0.00151, 0.0015, 0.00149, 0.00156, 0.00156, 0.00157, 0.0015, 0.00148, 0.00158, 0.00158, 0.00156, 0.00155, 0.00154, 0.00165, 0.00162, 0.00157, 0.00166, 0.0015, 0.00156, 0.00155, 0.00152, 0.00152, 0.00154, 0.0015, 0.00153, 0.0016, 0.0015, 0.00151, 0.00152, 0.00155, 0.00155]}, "optimizer-unscale-and-check-inf-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.60633, 0.00085, 0.00071, 0.0006, 0.00062, 0.0006, 0.00062, 0.00062, 0.00063, 0.00059, 0.00063, 0.00062, 0.00063, 0.00063, 0.00063, 0.00068, 0.00062, 0.00063, 0.00065, 0.00064, 0.00064, 0.0006, 0.00063, 0.00064, 0.00063, 0.00061, 0.00062, 0.00062, 0.00063, 0.00061, 0.0007, 0.00092, 0.00063, 0.00071, 0.00063, 0.00069, 0.00063, 0.00062, 0.00063, 0.00063, 0.00064, 0.0006, 0.00061, 0.00064, 0.00062, 0.00063, 0.00061, 0.00065, 0.00062, 0.00062, 0.0006, 0.00062, 0.00067, 0.00061, 0.00062, 0.00062, 0.00061, 0.00063, 0.00061, 0.00061, 0.0006, 0.00062, 0.00061, 0.00062, 0.00062, 0.00062, 0.00064, 0.00061, 0.00062, 0.00063, 0.00061, 0.00062, 0.00061, 0.00065, 0.00063, 0.0006, 0.0006, 0.0006, 0.00064, 0.00063, 0.00064, 0.0006, 0.00061, 0.00077, 0.00062, 0.00062, 0.00062, 0.00061, 0.00061, 0.00064, 0.00062, 0.0006, 0.00062, 0.00062, 0.00059, 0.00067, 0.00061, 0.00065, 0.0006, 0.00061, 0.00063, 0.00062, 0.00063, 0.00063, 0.00062, 0.0006, 0.00061, 0.00062, 0.00062, 0.0006, 0.00063, 0.00061, 0.0006, 0.0006, 0.00059, 0.00061, 0.0006, 0.00063, 0.00062, 0.00062, 0.00062, 0.00059, 0.00063, 0.0006, 0.00062, 0.00062, 0.00062, 0.00059, 0.00062, 0.00063, 0.0006, 0.00061, 0.0006, 0.00067, 0.00069, 0.00061, 0.00061, 0.00063, 0.00074, 0.0006, 0.00061, 0.00061, 0.00061, 0.00066, 0.00071, 0.00062, 0.00061, 0.0006, 0.00061, 0.00063, 0.0006, 0.00063, 0.00062, 0.00063, 0.00061, 0.00063, 0.00063, 0.00063, 0.00064, 0.00063, 0.00065, 0.00064, 0.00062, 0.00061, 0.00063, 0.00061, 0.00062, 0.00061, 0.00062, 0.00062, 0.00061, 0.00063, 0.00063, 0.00064, 0.00063, 0.00063, 0.00062, 0.00063, 0.00061, 0.00064, 0.00067, 0.0006, 0.00061, 0.00062, 0.00071, 0.00062, 0.00059, 0.00063, 0.00062, 0.0006, 0.00061, 0.00065, 0.00061, 0.00062, 0.00063, 0.00063, 0.00062, 0.00061, 0.00065, 0.00061, 0.00059, 0.0006, 0.00062, 0.0006, 0.00063, 0.00063, 0.0006, 0.00061, 0.00059, 0.00062, 0.00062, 0.0006, 0.00064, 0.00058, 0.00059, 0.00063, 0.00059, 0.0006, 0.00059, 0.00061, 0.00063, 0.00063, 0.0006, 0.0006, 0.00062, 0.0006, 0.00061, 0.00062, 0.00059, 0.00063, 0.0006, 0.00063, 0.0006, 0.00063, 0.00061, 0.00076, 0.0006, 0.00059, 0.00059, 0.00059, 0.0006, 0.00063, 0.00067, 0.00062, 0.00096, 0.00064, 0.00063, 0.00065, 0.00059, 0.00066, 0.00059, 0.0006, 0.00063, 0.00062, 0.00061, 0.00063, 0.00062, 0.00063, 0.00063, 0.00063, 0.0006, 0.00064, 0.00062, 0.00067, 0.00059, 0.00061, 0.00062, 0.00061, 0.00062, 0.0006, 0.0006, 0.00063, 0.00062, 0.00066, 0.00063, 0.00062, 0.00061, 0.00062, 0.00063, 0.00065, 0.00063, 0.00062, 0.00064, 0.00064, 0.00062, 0.00061, 0.00062, 0.00065, 0.00062, 0.00062, 0.00059, 0.00063, 0.00064, 0.0006, 0.00063, 0.00063, 0.00062, 0.00064, 0.00061, 0.00063, 0.00061, 0.0006, 0.00063, 0.00064, 0.00067, 0.00066, 0.00063, 0.00062, 0.00061, 0.00063, 0.00061, 0.00063, 0.00062, 0.00062, 0.00063, 0.00064, 0.00063, 0.00061, 0.00063, 0.00062, 0.00066, 0.00062, 0.00062, 0.00062, 0.00062, 0.00063, 0.00066, 0.00062, 0.00067, 0.00068, 0.00094, 0.00061, 0.00091, 0.00064, 0.00062, 0.00061, 0.00062, 0.00062, 0.00061, 0.00062, 0.00061, 0.00063, 0.00059, 0.00062, 0.00062, 0.00062, 0.00062, 0.00062, 0.00062, 0.00059, 0.00066, 0.00062, 0.00062, 0.0006, 0.00062, 0.00061, 0.00063, 0.00062, 0.00062, 0.00062, 0.00059, 0.0006, 0.00061, 0.0006, 0.00062, 0.00063, 0.00063, 0.00061, 0.00063, 0.00064, 0.00061, 0.00062, 0.00062, 0.00062, 0.00093, 0.00063, 0.00063, 0.00063, 0.00062, 0.00059, 0.00061, 0.00062, 0.00062, 0.00064, 0.00062, 0.00064, 0.00063, 0.00064, 0.00064, 0.00063, 0.00062, 0.00063, 0.00062, 0.00062, 0.00066, 0.00064, 0.00074, 0.00063, 0.00063, 0.00062]}, "optimizer-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.60837, 0.00254, 0.00241, 0.00228, 0.01048, 0.01037, 0.01037, 0.01043, 0.01058, 0.01048, 0.01043, 0.01043, 0.01041, 0.0104, 0.01041, 0.01065, 0.01035, 0.01034, 0.01163, 0.01037, 0.01065, 0.01028, 0.01071, 0.01072, 0.01046, 0.0103, 0.01034, 0.01036, 0.01049, 0.01035, 0.01149, 0.01326, 0.01057, 0.0123, 0.01043, 0.0108, 0.01045, 0.01043, 0.01054, 0.01044, 0.01042, 0.01047, 0.01038, 0.01036, 0.01051, 0.01045, 0.01031, 0.01066, 0.01039, 0.01038, 0.01045, 0.01039, 0.01082, 0.01041, 0.01037, 0.01039, 0.0104, 0.01052, 0.01036, 0.01042, 0.01043, 0.01041, 0.01041, 0.01038, 0.01048, 0.01055, 0.01067, 0.01037, 0.01034, 0.01046, 0.01031, 0.01091, 0.01032, 0.01102, 0.0105, 0.01027, 0.01037, 0.01029, 0.01047, 0.0104, 0.01046, 0.01038, 0.01047, 0.01178, 0.0104, 0.01074, 0.01048, 0.01035, 0.01038, 0.01049, 0.01045, 0.01029, 0.0104, 0.01038, 0.01035, 0.01254, 0.01037, 0.01078, 0.01036, 0.01033, 0.01045, 0.01036, 0.01034, 0.01037, 0.01041, 0.01036, 0.01033, 0.01079, 0.01038, 0.01041, 0.01023, 0.01009, 0.01031, 0.01035, 0.01038, 0.01037, 0.01044, 0.01035, 0.01041, 0.01038, 0.01021, 0.0103, 0.01049, 0.01051, 0.01036, 0.01032, 0.01054, 0.01033, 0.01041, 0.01043, 0.01041, 0.01037, 0.01014, 0.01109, 0.01092, 0.01032, 0.01033, 0.01042, 0.02222, 0.01043, 0.01036, 0.01031, 0.01034, 0.01109, 0.01102, 0.01041, 0.01027, 0.01035, 0.0103, 0.01041, 0.01036, 0.01039, 0.01035, 0.01041, 0.01048, 0.01069, 0.01042, 0.01035, 0.01064, 0.01041, 0.01045, 0.01034, 0.01039, 0.01039, 0.01043, 0.01033, 0.01133, 0.01034, 0.01033, 0.01034, 0.01031, 0.01035, 0.0104, 0.01052, 0.01043, 0.01047, 0.01036, 0.01029, 0.01035, 0.01042, 0.01057, 0.0103, 0.0103, 0.01039, 0.0109, 0.0103, 0.0103, 0.0105, 0.01036, 0.01034, 0.01033, 0.01214, 0.01032, 0.0103, 0.01039, 0.01085, 0.01031, 0.01031, 0.01064, 0.01141, 0.01028, 0.01048, 0.01035, 0.01021, 0.01033, 0.01032, 0.01023, 0.01127, 0.01075, 0.01024, 0.01023, 0.01023, 0.01033, 0.01036, 0.01017, 0.01034, 0.01026, 0.01036, 0.01019, 0.01026, 0.01033, 0.01163, 0.0102, 0.01023, 0.01031, 0.01033, 0.01042, 0.01049, 0.01036, 0.01032, 0.01053, 0.01033, 0.01034, 0.01037, 0.01037, 0.01078, 0.01026, 0.01052, 0.01028, 0.01028, 0.01025, 0.01028, 0.01147, 0.01035, 0.01173, 0.01035, 0.01038, 0.01027, 0.01027, 0.01065, 0.01023, 0.01027, 0.01043, 0.01054, 0.01038, 0.01054, 0.01028, 0.01026, 0.0103, 0.01038, 0.0104, 0.0103, 0.0104, 0.01114, 0.01027, 0.01028, 0.01042, 0.01027, 0.01037, 0.01028, 0.01061, 0.01066, 0.01034, 0.0108, 0.01035, 0.01037, 0.01038, 0.01034, 0.01138, 0.01141, 0.01027, 0.01041, 0.01039, 0.01039, 0.01031, 0.01042, 0.01036, 0.01077, 0.01045, 0.01035, 0.0105, 0.01039, 0.01057, 0.01041, 0.01033, 0.01039, 0.01029, 0.0106, 0.01032, 0.01029, 0.01034, 0.01044, 0.01035, 0.01034, 0.0111, 0.01066, 0.01041, 0.0103, 0.01025, 0.01038, 0.01037, 0.01064, 0.0105, 0.0103, 0.01048, 0.01051, 0.01052, 0.01041, 0.0104, 0.01041, 0.01044, 0.01036, 0.01043, 0.01038, 0.01034, 0.01033, 0.01126, 0.01037, 0.01044, 0.01078, 0.01116, 0.01162, 0.01139, 0.01058, 0.0105, 0.01061, 0.01053, 0.01057, 0.01058, 0.01058, 0.01057, 0.0106, 0.01051, 0.01054, 0.01067, 0.0109, 0.01057, 0.01057, 0.01057, 0.01051, 0.01063, 0.01186, 0.0105, 0.01054, 0.01053, 0.01061, 0.01062, 0.01089, 0.01057, 0.0106, 0.01047, 0.01071, 0.0105, 0.01049, 0.01052, 0.01054, 0.01057, 0.0106, 0.01078, 0.01062, 0.01067, 0.01052, 0.01059, 0.01061, 0.01212, 0.01052, 0.01054, 0.01063, 0.0106, 0.01057, 0.01098, 0.01059, 0.01077, 0.01074, 0.01076, 0.01115, 0.01053, 0.01121, 0.01063, 0.01056, 0.01057, 0.01061, 0.01059, 0.01061, 0.01076, 0.01059, 0.01075, 0.01057, 0.01058, 0.01057]}, "learning-rate": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05]}, "learning-rate vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05]}, "batch-size": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0]}, "batch-size vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0]}, "lm loss": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [10.89393, 10.90229, 10.90382, 10.89922, 10.90215, 10.87439, 10.80338, 10.63346, 10.44036, 10.2933, 10.02711, 10.16747, 10.13781, 9.86192, 9.97684, 9.67806, 9.59835, 9.78149, 9.50324, 9.44529, 9.35262, 9.25422, 9.27971, 9.09386, 9.28651, 9.15722, 9.24673, 9.26197, 9.39815, 9.08902, 9.03506, 9.14524, 9.15344, 8.76086, 8.82546, 8.85801, 8.78594, 8.83766, 8.7627, 8.8693, 8.76505, 8.95513, 8.94138, 8.60415, 8.49526, 8.5414, 8.6052, 8.49378, 8.54563, 8.69589, 8.47931, 8.31047, 8.34191, 8.33761, 8.38482, 8.03117, 8.21698, 8.01005, 8.36597, 8.35171, 8.1238, 8.08903, 8.03892, 7.85884, 7.86204, 7.76178, 7.63785, 8.03256, 7.82491, 7.57767, 7.87018, 7.89663, 7.66576, 7.41891, 7.57945, 7.45949, 7.58407, 7.3365, 7.75478, 7.39312, 7.46005, 7.32601, 7.32261, 7.53324, 7.28432, 7.3906, 7.10455, 7.1031, 7.135, 7.2333, 6.91495, 7.07308, 7.17321, 7.08148, 6.95568, 6.83552, 7.07146, 7.13597, 6.77633, 6.6537, 6.79923, 6.81094, 6.80156, 6.80623, 6.72479, 6.46997, 6.7029, 6.67891, 6.50414, 6.69017, 6.80201, 6.66742, 6.78223, 6.74908, 6.68039, 6.55851, 6.65127, 6.45882, 6.71595, 6.3003, 6.29947, 6.35127, 6.43626, 6.39728, 6.5005, 6.33652, 6.38489, 6.2805, 6.24364, 6.44007, 6.36837, 6.36408, 6.20465, 6.19665, 6.27951, 6.42484, 6.24039, 6.18602, 6.21368, 6.14857, 6.09651, 6.10359, 6.28963, 6.44182, 6.28988, 6.33247, 6.13546, 6.21108, 6.0349, 6.06273, 5.987, 6.28025, 6.22641, 5.99808, 5.81837, 6.16027, 5.88364, 6.139, 5.82189, 6.19536, 6.17777, 6.11785, 5.96408, 6.14649, 5.9753, 6.22609, 5.92665, 5.82529, 5.80636, 5.7182, 6.04353, 6.02584, 6.092, 5.9119, 6.06757, 5.99273, 6.02669, 6.01523, 5.97662, 5.86429, 5.97653, 5.6431, 5.7275, 5.9135, 5.8664, 5.88797, 5.78842, 5.86055, 5.75215, 5.58542, 5.74699, 5.6532, 5.85871, 5.63063, 5.7325, 5.73883, 5.92312, 5.66992, 5.87123, 5.76346, 5.89613, 5.35339, 5.91985, 5.89554, 5.87623, 5.43362, 5.42829, 5.64744, 5.61678, 5.5103, 5.59917, 5.6988, 5.49854, 5.77013, 5.53314, 5.61954, 5.64553, 5.64008, 5.53513, 5.63528, 5.69717, 5.71522, 5.60874, 5.6802, 5.39435, 5.70021, 5.64782, 5.44435, 5.60824, 5.65007, 5.57098, 5.36362, 5.55798, 5.50433, 5.50082, 5.39457, 5.57452, 5.62082, 5.40855, 5.54177, 5.50319, 5.34993, 5.52256, 5.42475, 5.457, 5.33418, 5.08125, 5.49351, 5.58285, 5.72877, 5.42977, 5.613, 5.64847, 5.2484, 5.28756, 5.41008, 5.40961, 5.34061, 5.51276, 5.19903, 5.31256, 5.26266, 5.3907, 5.27539, 5.46188, 5.55243, 5.32608, 5.4523, 5.34935, 5.085, 5.3281, 5.26395, 5.31744, 5.12555, 5.28677, 5.2827, 5.486, 5.17172, 5.28031, 5.22155, 5.37027, 4.99359, 4.92973, 5.33403, 5.3997, 5.23719, 5.33061, 5.11473, 5.1717, 5.27268, 5.07733, 5.2767, 5.0858, 5.35129, 5.2583, 5.16657, 5.25468, 5.05243, 5.32453, 5.06278, 5.03705, 5.15134, 5.12068, 5.28265, 5.15883, 5.28883, 5.10618, 5.10727, 5.2621, 5.33107, 5.26622, 5.20237, 5.15543, 5.29779, 4.95636, 5.21799, 5.10164, 5.30924, 5.18679, 5.19599, 5.12317, 4.99367, 5.00306, 5.23171, 5.32198, 5.10695, 5.0647, 4.92646, 5.13309, 5.12718, 4.93681, 5.34691, 5.03142, 5.11047, 5.16889, 5.01087, 5.07032, 5.07588, 5.00122, 5.08773, 5.16951, 4.98692, 5.18998, 4.93899, 4.92741, 5.07395, 5.00085, 4.91692, 4.78186, 4.94917, 5.12365, 5.02541, 5.02437, 5.33759, 4.96582, 5.00145, 5.05138, 4.81301, 4.74456, 5.00203, 5.04679, 4.88367, 4.95882, 5.05212, 5.03024, 4.82289, 4.89705, 4.91162, 4.83722, 4.75468, 5.01694, 4.75625, 5.21634, 4.78922, 4.99899, 4.74083, 4.79117, 4.82499, 4.65555, 4.66118, 4.84502, 4.812, 4.80818, 4.93087, 4.88819, 4.92996, 4.77146, 4.88927, 4.73848, 4.91779, 4.96467, 4.87947, 4.7104, 4.78793, 4.90438, 4.71479, 4.86815, 4.69617, 4.69095, 4.65249]}, "lm loss vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [10.89393, 10.90229, 10.90382, 10.89922, 10.90215, 10.87439, 10.80338, 10.63346, 10.44036, 10.2933, 10.02711, 10.16747, 10.13781, 9.86192, 9.97684, 9.67806, 9.59835, 9.78149, 9.50324, 9.44529, 9.35262, 9.25422, 9.27971, 9.09386, 9.28651, 9.15722, 9.24673, 9.26197, 9.39815, 9.08902, 9.03506, 9.14524, 9.15344, 8.76086, 8.82546, 8.85801, 8.78594, 8.83766, 8.7627, 8.8693, 8.76505, 8.95513, 8.94138, 8.60415, 8.49526, 8.5414, 8.6052, 8.49378, 8.54563, 8.69589, 8.47931, 8.31047, 8.34191, 8.33761, 8.38482, 8.03117, 8.21698, 8.01005, 8.36597, 8.35171, 8.1238, 8.08903, 8.03892, 7.85884, 7.86204, 7.76178, 7.63785, 8.03256, 7.82491, 7.57767, 7.87018, 7.89663, 7.66576, 7.41891, 7.57945, 7.45949, 7.58407, 7.3365, 7.75478, 7.39312, 7.46005, 7.32601, 7.32261, 7.53324, 7.28432, 7.3906, 7.10455, 7.1031, 7.135, 7.2333, 6.91495, 7.07308, 7.17321, 7.08148, 6.95568, 6.83552, 7.07146, 7.13597, 6.77633, 6.6537, 6.79923, 6.81094, 6.80156, 6.80623, 6.72479, 6.46997, 6.7029, 6.67891, 6.50414, 6.69017, 6.80201, 6.66742, 6.78223, 6.74908, 6.68039, 6.55851, 6.65127, 6.45882, 6.71595, 6.3003, 6.29947, 6.35127, 6.43626, 6.39728, 6.5005, 6.33652, 6.38489, 6.2805, 6.24364, 6.44007, 6.36837, 6.36408, 6.20465, 6.19665, 6.27951, 6.42484, 6.24039, 6.18602, 6.21368, 6.14857, 6.09651, 6.10359, 6.28963, 6.44182, 6.28988, 6.33247, 6.13546, 6.21108, 6.0349, 6.06273, 5.987, 6.28025, 6.22641, 5.99808, 5.81837, 6.16027, 5.88364, 6.139, 5.82189, 6.19536, 6.17777, 6.11785, 5.96408, 6.14649, 5.9753, 6.22609, 5.92665, 5.82529, 5.80636, 5.7182, 6.04353, 6.02584, 6.092, 5.9119, 6.06757, 5.99273, 6.02669, 6.01523, 5.97662, 5.86429, 5.97653, 5.6431, 5.7275, 5.9135, 5.8664, 5.88797, 5.78842, 5.86055, 5.75215, 5.58542, 5.74699, 5.6532, 5.85871, 5.63063, 5.7325, 5.73883, 5.92312, 5.66992, 5.87123, 5.76346, 5.89613, 5.35339, 5.91985, 5.89554, 5.87623, 5.43362, 5.42829, 5.64744, 5.61678, 5.5103, 5.59917, 5.6988, 5.49854, 5.77013, 5.53314, 5.61954, 5.64553, 5.64008, 5.53513, 5.63528, 5.69717, 5.71522, 5.60874, 5.6802, 5.39435, 5.70021, 5.64782, 5.44435, 5.60824, 5.65007, 5.57098, 5.36362, 5.55798, 5.50433, 5.50082, 5.39457, 5.57452, 5.62082, 5.40855, 5.54177, 5.50319, 5.34993, 5.52256, 5.42475, 5.457, 5.33418, 5.08125, 5.49351, 5.58285, 5.72877, 5.42977, 5.613, 5.64847, 5.2484, 5.28756, 5.41008, 5.40961, 5.34061, 5.51276, 5.19903, 5.31256, 5.26266, 5.3907, 5.27539, 5.46188, 5.55243, 5.32608, 5.4523, 5.34935, 5.085, 5.3281, 5.26395, 5.31744, 5.12555, 5.28677, 5.2827, 5.486, 5.17172, 5.28031, 5.22155, 5.37027, 4.99359, 4.92973, 5.33403, 5.3997, 5.23719, 5.33061, 5.11473, 5.1717, 5.27268, 5.07733, 5.2767, 5.0858, 5.35129, 5.2583, 5.16657, 5.25468, 5.05243, 5.32453, 5.06278, 5.03705, 5.15134, 5.12068, 5.28265, 5.15883, 5.28883, 5.10618, 5.10727, 5.2621, 5.33107, 5.26622, 5.20237, 5.15543, 5.29779, 4.95636, 5.21799, 5.10164, 5.30924, 5.18679, 5.19599, 5.12317, 4.99367, 5.00306, 5.23171, 5.32198, 5.10695, 5.0647, 4.92646, 5.13309, 5.12718, 4.93681, 5.34691, 5.03142, 5.11047, 5.16889, 5.01087, 5.07032, 5.07588, 5.00122, 5.08773, 5.16951, 4.98692, 5.18998, 4.93899, 4.92741, 5.07395, 5.00085, 4.91692, 4.78186, 4.94917, 5.12365, 5.02541, 5.02437, 5.33759, 4.96582, 5.00145, 5.05138, 4.81301, 4.74456, 5.00203, 5.04679, 4.88367, 4.95882, 5.05212, 5.03024, 4.82289, 4.89705, 4.91162, 4.83722, 4.75468, 5.01694, 4.75625, 5.21634, 4.78922, 4.99899, 4.74083, 4.79117, 4.82499, 4.65555, 4.66118, 4.84502, 4.812, 4.80818, 4.93087, 4.88819, 4.92996, 4.77146, 4.88927, 4.73848, 4.91779, 4.96467, 4.87947, 4.7104, 4.78793, 4.90438, 4.71479, 4.86815, 4.69617, 4.69095, 4.65249]}, "loss-scale": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [4294967296.0, 134217728.0, 4194304.0, 131072.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0]}, "loss-scale vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [4294967296.0, 134217728.0, 4194304.0, 131072.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 65536.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0, 131072.0]}, "params-norm": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [179.95639, 179.95639, 179.95639, 179.95639, 179.95639, 179.95639, 179.95636, 179.95616, 179.95595, 179.9552, 179.95465, 179.95432, 179.95352, 179.953, 179.95229, 179.95172, 179.95114, 179.95059, 179.95015, 179.94978, 179.94951, 179.94933, 179.94916, 179.94899, 179.94891, 179.94894, 179.94923, 179.95026, 179.95171, 179.9529, 179.95413, 179.95543, 179.95691, 179.95865, 179.96053, 179.96269, 179.96513, 179.96796, 179.97112, 179.97466, 179.97838, 179.98239, 179.98705, 179.9922, 179.99811, 180.00458, 180.01144, 180.0188, 180.0265, 180.0349, 180.04382, 180.05347, 180.06361, 180.07454, 180.0863, 180.09869, 180.1114, 180.12436, 180.13821, 180.15294, 180.16814, 180.18376, 180.20035, 180.21758, 180.23528, 180.25388, 180.27333, 180.2935, 180.31477, 180.33707, 180.36023, 180.38481, 180.4104, 180.43663, 180.46335, 180.49043, 180.51775, 180.54597, 180.57475, 180.60458, 180.63466, 180.66501, 180.69615, 180.72832, 180.76106, 180.79457, 180.82857, 180.86211, 180.89636, 180.93251, 180.97021, 181.00865, 181.04654, 181.08444, 181.12204, 181.1591, 181.19463, 181.22873, 181.26352, 181.29965, 181.33498, 181.36926, 181.40433, 181.44101, 181.47787, 181.51541, 181.55309, 181.58995, 181.62593, 181.66238, 181.69963, 181.73865, 181.77856, 181.819, 181.85893, 181.89955, 181.94034, 181.98015, 182.01802, 182.05594, 182.09499, 182.13466, 182.17516, 182.21599, 182.25551, 182.29494, 182.33302, 182.36942, 182.40552, 182.44077, 182.47746, 182.51506, 182.55521, 182.59557, 182.63631, 182.67693, 182.71771, 182.75752, 182.79524, 182.83229, 182.8694, 182.90648, 182.94411, 182.98082, 183.01617, 183.05077, 183.08421, 183.11528, 183.14688, 183.17844, 183.21207, 183.24745, 183.28352, 183.31885, 183.35526, 183.39171, 183.42731, 183.46333, 183.49973, 183.53497, 183.57001, 183.60588, 183.64211, 183.6795, 183.71835, 183.75874, 183.79941, 183.83905, 183.87886, 183.91798, 183.95557, 183.99252, 184.02957, 184.06734, 184.1066, 184.14734, 184.18813, 184.22699, 184.26306, 184.29767, 184.33336, 184.36948, 184.40587, 184.44305, 184.48088, 184.51953, 184.55611, 184.58971, 184.62381, 184.65984, 184.6958, 184.73257, 184.76843, 184.80443, 184.84024, 184.87787, 184.91624, 184.9561, 184.99586, 185.03816, 185.08003, 185.12041, 185.16002, 185.19998, 185.23941, 185.27916, 185.31915, 185.35942, 185.3989, 185.43639, 185.4734, 185.51125, 185.54845, 185.5865, 185.62511, 185.66444, 185.70372, 185.74438, 185.78564, 185.82716, 185.86717, 185.90334, 185.937, 185.97195, 186.00873, 186.04741, 186.0872, 186.12794, 186.16808, 186.20654, 186.24687, 186.28903, 186.3307, 186.3723, 186.4149, 186.45834, 186.50229, 186.54523, 186.58723, 186.62804, 186.66795, 186.70871, 186.75044, 186.79398, 186.83716, 186.88002, 186.92215, 186.96371, 187.00597, 187.04924, 187.09216, 187.13554, 187.17883, 187.22208, 187.26509, 187.30769, 187.34932, 187.39163, 187.43529, 187.47867, 187.52255, 187.5659, 187.6091, 187.65163, 187.6926, 187.7334, 187.77498, 187.81706, 187.85999, 187.90363, 187.94743, 187.99174, 188.03735, 188.08296, 188.12976, 188.17722, 188.22394, 188.27153, 188.31853, 188.3636, 188.40756, 188.45032, 188.49333, 188.53738, 188.58321, 188.62881, 188.67557, 188.722, 188.76859, 188.81543, 188.86082, 188.90515, 188.94725, 188.9901, 189.0343, 189.07765, 189.12099, 189.16522, 189.21011, 189.25642, 189.3047, 189.35202, 189.39963, 189.4478, 189.49484, 189.5425, 189.59079, 189.63968, 189.68971, 189.74034, 189.79134, 189.84206, 189.89209, 189.9409, 189.99072, 190.04274, 190.09349, 190.14539, 190.19702, 190.24873, 190.30104, 190.35287, 190.4046, 190.45503, 190.50591, 190.55637, 190.60674, 190.65721, 190.70746, 190.75826, 190.80876, 190.8571, 190.90599, 190.95639, 191.00842, 191.06157, 191.11446, 191.16747, 191.22209, 191.2751, 191.32907, 191.38292, 191.43571, 191.48894, 191.54314, 191.59731, 191.65059, 191.70349, 191.75555, 191.80978, 191.86417, 191.91782, 191.97105, 192.02429, 192.0775, 192.13084, 192.18512, 192.24028, 192.29506, 192.35016, 192.40651, 192.4622, 192.51747, 192.57224, 192.62952, 192.687, 192.74483, 192.80281, 192.86006, 192.91705, 192.97177, 193.02679, 193.08273, 193.13742, 193.1917, 193.24458, 193.29779, 193.35132, 193.40689, 193.46413, 193.52164, 193.57927, 193.63789, 193.69646, 193.75464, 193.81409, 193.87488, 193.93707, 193.99841, 194.05937, 194.11984, 194.17958, 194.23772, 194.29633, 194.35521, 194.41174, 194.46733, 194.52335, 194.58064, 194.6398]}, "params-norm vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [179.95639, 179.95639, 179.95639, 179.95639, 179.95639, 179.95639, 179.95636, 179.95616, 179.95595, 179.9552, 179.95465, 179.95432, 179.95352, 179.953, 179.95229, 179.95172, 179.95114, 179.95059, 179.95015, 179.94978, 179.94951, 179.94933, 179.94916, 179.94899, 179.94891, 179.94894, 179.94923, 179.95026, 179.95171, 179.9529, 179.95413, 179.95543, 179.95691, 179.95865, 179.96053, 179.96269, 179.96513, 179.96796, 179.97112, 179.97466, 179.97838, 179.98239, 179.98705, 179.9922, 179.99811, 180.00458, 180.01144, 180.0188, 180.0265, 180.0349, 180.04382, 180.05347, 180.06361, 180.07454, 180.0863, 180.09869, 180.1114, 180.12436, 180.13821, 180.15294, 180.16814, 180.18376, 180.20035, 180.21758, 180.23528, 180.25388, 180.27333, 180.2935, 180.31477, 180.33707, 180.36023, 180.38481, 180.4104, 180.43663, 180.46335, 180.49043, 180.51775, 180.54597, 180.57475, 180.60458, 180.63466, 180.66501, 180.69615, 180.72832, 180.76106, 180.79457, 180.82857, 180.86211, 180.89636, 180.93251, 180.97021, 181.00865, 181.04654, 181.08444, 181.12204, 181.1591, 181.19463, 181.22873, 181.26352, 181.29965, 181.33498, 181.36926, 181.40433, 181.44101, 181.47787, 181.51541, 181.55309, 181.58995, 181.62593, 181.66238, 181.69963, 181.73865, 181.77856, 181.819, 181.85893, 181.89955, 181.94034, 181.98015, 182.01802, 182.05594, 182.09499, 182.13466, 182.17516, 182.21599, 182.25551, 182.29494, 182.33302, 182.36942, 182.40552, 182.44077, 182.47746, 182.51506, 182.55521, 182.59557, 182.63631, 182.67693, 182.71771, 182.75752, 182.79524, 182.83229, 182.8694, 182.90648, 182.94411, 182.98082, 183.01617, 183.05077, 183.08421, 183.11528, 183.14688, 183.17844, 183.21207, 183.24745, 183.28352, 183.31885, 183.35526, 183.39171, 183.42731, 183.46333, 183.49973, 183.53497, 183.57001, 183.60588, 183.64211, 183.6795, 183.71835, 183.75874, 183.79941, 183.83905, 183.87886, 183.91798, 183.95557, 183.99252, 184.02957, 184.06734, 184.1066, 184.14734, 184.18813, 184.22699, 184.26306, 184.29767, 184.33336, 184.36948, 184.40587, 184.44305, 184.48088, 184.51953, 184.55611, 184.58971, 184.62381, 184.65984, 184.6958, 184.73257, 184.76843, 184.80443, 184.84024, 184.87787, 184.91624, 184.9561, 184.99586, 185.03816, 185.08003, 185.12041, 185.16002, 185.19998, 185.23941, 185.27916, 185.31915, 185.35942, 185.3989, 185.43639, 185.4734, 185.51125, 185.54845, 185.5865, 185.62511, 185.66444, 185.70372, 185.74438, 185.78564, 185.82716, 185.86717, 185.90334, 185.937, 185.97195, 186.00873, 186.04741, 186.0872, 186.12794, 186.16808, 186.20654, 186.24687, 186.28903, 186.3307, 186.3723, 186.4149, 186.45834, 186.50229, 186.54523, 186.58723, 186.62804, 186.66795, 186.70871, 186.75044, 186.79398, 186.83716, 186.88002, 186.92215, 186.96371, 187.00597, 187.04924, 187.09216, 187.13554, 187.17883, 187.22208, 187.26509, 187.30769, 187.34932, 187.39163, 187.43529, 187.47867, 187.52255, 187.5659, 187.6091, 187.65163, 187.6926, 187.7334, 187.77498, 187.81706, 187.85999, 187.90363, 187.94743, 187.99174, 188.03735, 188.08296, 188.12976, 188.17722, 188.22394, 188.27153, 188.31853, 188.3636, 188.40756, 188.45032, 188.49333, 188.53738, 188.58321, 188.62881, 188.67557, 188.722, 188.76859, 188.81543, 188.86082, 188.90515, 188.94725, 188.9901, 189.0343, 189.07765, 189.12099, 189.16522, 189.21011, 189.25642, 189.3047, 189.35202, 189.39963, 189.4478, 189.49484, 189.5425, 189.59079, 189.63968, 189.68971, 189.74034, 189.79134, 189.84206, 189.89209, 189.9409, 189.99072, 190.04274, 190.09349, 190.14539, 190.19702, 190.24873, 190.30104, 190.35287, 190.4046, 190.45503, 190.50591, 190.55637, 190.60674, 190.65721, 190.70746, 190.75826, 190.80876, 190.8571, 190.90599, 190.95639, 191.00842, 191.06157, 191.11446, 191.16747, 191.22209, 191.2751, 191.32907, 191.38292, 191.43571, 191.48894, 191.54314, 191.59731, 191.65059, 191.70349, 191.75555, 191.80978, 191.86417, 191.91782, 191.97105, 192.02429, 192.0775, 192.13084, 192.18512, 192.24028, 192.29506, 192.35016, 192.40651, 192.4622, 192.51747, 192.57224, 192.62952, 192.687, 192.74483, 192.80281, 192.86006, 192.91705, 192.97177, 193.02679, 193.08273, 193.13742, 193.1917, 193.24458, 193.29779, 193.35132, 193.40689, 193.46413, 193.52164, 193.57927, 193.63789, 193.69646, 193.75464, 193.81409, 193.87488, 193.93707, 193.99841, 194.05937, 194.11984, 194.17958, 194.23772, 194.29633, 194.35521, 194.41174, 194.46733, 194.52335, 194.58064, 194.6398]}, "iteration-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [18.07681, 0.38236, 0.3815, 0.38004, 0.39049, 0.39656, 0.39642, 0.39048, 0.39523, 0.39194, 0.5552, 0.3948, 0.39398, 0.39561, 0.39214, 0.39537, 0.39216, 0.39261, 0.39694, 0.39356, 0.4003, 0.39114, 0.39355, 0.3919, 0.39064, 0.40086, 0.39355, 0.39139, 0.38492, 0.3927, 0.40428, 0.38479, 0.38466, 0.38299, 0.38174, 0.38636, 0.38086, 0.38401, 0.38601, 0.40511, 0.38629, 0.38521, 0.3855, 0.38256, 0.38493, 0.38553, 0.38438, 0.38462, 0.38628, 0.38214, 0.38492, 0.38322, 0.38706, 0.38103, 0.38314, 0.38469, 0.38271, 0.38565, 0.38283, 0.38163, 0.37833, 0.38621, 0.37993, 0.37921, 0.38058, 0.38093, 0.38301, 0.38316, 0.38564, 0.38136, 0.38386, 0.38121, 0.38145, 0.37922, 0.48103, 0.37987, 0.38025, 0.38308, 0.38613, 0.38258, 0.38336, 0.38508, 0.3887, 0.38459, 0.38233, 0.38094, 0.38026, 0.38316, 0.3802, 0.38401, 0.38409, 0.38327, 0.39188, 0.38081, 0.38297, 0.38391, 0.38075, 0.38566, 0.38249, 0.38281, 0.38433, 0.38249, 0.37955, 0.38003, 0.47628, 0.38394, 0.38015, 0.40241, 0.37987, 0.38149, 0.38158, 0.38618, 0.38356, 0.38072, 0.3889, 0.38918, 0.38574, 0.38775, 0.38338, 0.39021, 0.38146, 0.38236, 0.38742, 0.3868, 0.38407, 0.38593, 0.38727, 0.39089, 0.39337, 0.38585, 0.38443, 0.38667, 0.3868, 0.39023, 0.49507, 0.38161, 0.38081, 0.38199, 0.48238, 0.53269, 0.38537, 0.38444, 0.38705, 0.39224, 0.38871, 0.3845, 0.38286, 0.38071, 0.38022, 0.38228, 0.38177, 0.38417, 0.3801, 0.38435, 0.38639, 0.38626, 0.38489, 0.38587, 0.38488, 0.38407, 0.3867, 0.38401, 0.3866, 0.38593, 0.38916, 0.3833, 0.38389, 0.3843, 0.38359, 0.38697, 0.38383, 0.38577, 0.38399, 0.38402, 0.38788, 0.3861, 0.38511, 0.38672, 0.38227, 0.38915, 0.38446, 0.3859, 0.37898, 0.381, 0.38613, 0.38362, 0.3831, 0.37854, 0.37897, 0.37818, 0.37983, 0.38369, 0.37982, 0.38105, 0.38549, 0.38522, 0.38518, 0.38435, 0.47441, 0.38233, 0.37927, 0.38248, 0.38035, 0.37886, 0.38094, 0.3816, 0.38623, 0.38907, 0.38824, 0.38363, 0.38085, 0.38241, 0.38688, 0.3809, 0.38401, 0.3846, 0.38278, 0.38686, 0.38509, 0.38569, 0.38138, 0.38221, 0.38366, 0.39376, 0.39173, 0.38031, 0.38231, 0.47746, 0.38191, 0.38528, 0.38919, 0.38627, 0.38485, 0.39016, 0.48709, 0.39134, 0.38991, 0.38575, 0.3826, 0.38101, 0.38387, 0.38025, 0.37997, 0.50302, 0.38436, 0.38473, 0.38639, 0.38633, 0.3928, 0.38343, 0.38522, 0.38229, 0.37817, 0.38096, 0.38116, 0.3867, 0.38377, 0.38146, 0.38226, 0.38398, 0.39339, 0.3803, 0.48334, 0.38398, 0.38072, 0.38756, 0.38406, 0.38475, 0.3865, 0.3837, 0.39344, 0.38796, 0.38926, 0.38703, 0.38603, 0.37954, 0.38341, 0.38785, 0.38335, 0.38263, 0.38197, 0.38334, 0.3861, 0.38808, 0.38389, 0.38779, 0.39044, 0.38432, 0.38303, 0.38348, 0.38756, 0.38699, 0.47757, 0.38391, 0.38223, 0.38479, 0.38831, 0.38749, 0.384, 0.3864, 0.38554, 0.38656, 0.38469, 0.38559, 0.38552, 0.38634, 0.39068, 0.38718, 0.38906, 0.38314, 0.38526, 0.39355, 0.38547, 0.3918, 0.38838, 0.39149, 0.38788, 0.38735, 0.38776, 0.38498, 0.3845, 0.3809, 0.38438, 0.38342, 0.38109, 0.38385, 0.3847, 0.38354, 0.38456, 0.48679, 0.38819, 0.38623, 0.3908, 0.39049, 0.38764, 0.39009, 0.3899, 0.39171, 0.39325, 0.39116, 0.38744, 0.38994, 0.3945, 0.38791, 0.3872, 0.3882, 0.38525, 0.38534, 0.38602, 0.38534, 0.38256, 0.38598, 0.38572, 0.37898, 0.38512, 0.38512, 0.38361, 0.39213, 0.38551, 0.38269, 0.38516, 0.38696, 0.38679, 0.37971, 0.38365, 0.38484, 0.38698, 0.39395, 0.38701, 0.38655, 0.38288, 0.38233, 0.38642, 0.38468, 0.38309, 0.38362, 0.38617, 0.3863, 0.38907, 0.38471, 0.38686, 0.38576, 0.3853, 0.38783, 0.3863, 0.38804, 0.38654, 0.48838, 0.39169, 0.38856, 0.47555, 0.38859, 0.39202, 0.38824, 0.59598, 0.38895, 0.38921, 0.38633, 0.38705, 0.38574]}, "optimizer-clip-main-grad-time": {"start_step": 0, "end_step": 1983, "step_interval": 5, "values": [0.02457, 0.00089, 0.00088, 0.00089, 0.00088, 0.00089, 0.00089, 0.00089, 0.0009, 0.00089, 0.00091, 0.00095, 0.00088, 0.0009, 0.00088, 0.00088, 0.00089, 0.0009, 0.0009, 0.00089, 0.0009, 0.00088, 0.00088, 0.00088, 0.00089, 0.00089, 0.00089, 0.00088, 0.00087, 0.00088, 0.00088, 0.00088, 0.00088, 0.00089, 0.00093, 0.00088, 0.00088, 0.0009, 0.00092, 0.00089, 0.00088, 0.00088, 0.00089, 0.00088, 0.00089, 0.00089, 0.00089, 0.00099, 0.00088, 0.00088, 0.00089, 0.00089, 0.00088, 0.00088, 0.00088, 0.00088, 0.00088, 0.0009, 0.00126, 0.00088, 0.00088, 0.00088, 0.00094, 0.00088, 0.00087, 0.00088, 0.00087, 0.00088, 0.00088, 0.0009, 0.00087, 0.00088, 0.00088, 0.00088, 0.00087, 0.00088, 0.00087, 0.00125, 0.00093, 0.0009, 0.00088, 0.00088, 0.00088, 0.00088, 0.00088, 0.00088, 0.00088, 0.00098, 0.00088, 0.00112, 0.00088, 0.00088, 0.00089, 0.00087, 0.00088, 0.00087, 0.00088, 0.00088, 0.00088, 0.00089, 0.0009, 0.00087, 0.00088, 0.00088, 0.00091, 0.00088, 0.00088, 0.00088, 0.00088, 0.00092, 0.00087, 0.00066, 0.00088, 0.00088, 0.0009, 0.00065, 0.00088, 0.00088, 0.00066, 0.00089, 0.00089, 0.00066, 0.00088, 0.001, 0.00088, 0.00088, 0.0009, 0.00066, 0.00066, 0.00088, 0.00067, 0.00089, 0.00089, 0.00067, 0.00088, 0.00089, 0.00087, 0.00087, 0.00095, 0.00088, 0.00087, 0.00088, 0.00087, 0.00089, 0.00089, 0.00088, 0.00089, 0.00089, 0.00088, 0.00089, 0.0009, 0.00087, 0.00087, 0.00089, 0.00088, 0.00087, 0.00087, 0.00087, 0.00087, 0.00088, 0.00088, 0.00089, 0.00088, 0.0009, 0.00089, 0.00087, 0.00087, 0.00087, 0.00089, 0.00089, 0.00094, 0.00088, 0.00087, 0.00087, 0.00088, 0.00088, 0.00087, 0.00087, 0.00088, 0.00088, 0.00088, 0.00087, 0.00087, 0.00087, 0.00087, 0.00088, 0.00088, 0.00087, 0.00087, 0.00098, 0.00088, 0.00091, 0.00087, 0.00087, 0.00089, 0.00088, 0.00088, 0.00088, 0.00091, 0.00087, 0.00088, 0.00107, 0.00095, 0.00088, 0.00087, 0.00088, 0.00094, 0.00093, 0.00087, 0.00089, 0.00087, 0.00088, 0.00087, 0.00089, 0.00087, 0.00087, 0.00087, 0.00087, 0.00088, 0.00089, 0.00087, 0.00087, 0.00088, 0.00089, 0.00087, 0.00087, 0.00094, 0.00088, 0.00087, 0.00089, 0.00093, 0.00088, 0.00087, 0.00087, 0.00088, 0.00088, 0.00088, 0.00088, 0.00095, 0.00087, 0.00087, 0.00087, 0.00087, 0.00087, 0.00108, 0.00087, 0.00089, 0.00089, 0.00089, 0.00088, 0.001, 0.00088, 0.00094, 0.00088, 0.00087, 0.00088, 0.00095, 0.0009, 0.00089, 0.00089, 0.00088, 0.00088, 0.00089, 0.00088, 0.0009, 0.00089, 0.00088, 0.00088, 0.00087, 0.00088, 0.00089, 0.00088, 0.00087, 0.00088, 0.00087, 0.00089, 0.00091, 0.00088, 0.00096, 0.00088, 0.00092, 0.00088, 0.00088, 0.00088, 0.00088, 0.00088, 0.00088, 0.00087, 0.00089, 0.00088, 0.00091, 0.00095, 0.00088, 0.00088, 0.00095, 0.0009, 0.00089, 0.00092, 0.00093, 0.00099, 0.00088, 0.0009, 0.00087, 0.00088, 0.00096, 0.00088, 0.00097, 0.00087, 0.00088, 0.00087, 0.00088, 0.00088, 0.00098, 0.00089, 0.00097, 0.00087, 0.00087, 0.00087, 0.00088, 0.00089, 0.00088, 0.00089, 0.00088, 0.00088, 0.00087, 0.00087, 0.00099, 0.00089, 0.00088, 0.00088, 0.00087, 0.00088, 0.00088, 0.00089, 0.00087, 0.00088, 0.00088, 0.0009, 0.00091, 0.00089, 0.00087, 0.00088, 0.00089, 0.00089, 0.00087, 0.00088, 0.00094, 0.00088, 0.00088, 0.00088, 0.00088, 0.00089, 0.00087, 0.00106, 0.0009, 0.00089, 0.00088, 0.00096, 0.00089, 0.00098, 0.00088, 0.00088, 0.00088, 0.00091, 0.00087, 0.00089, 0.00088, 0.00088, 0.00088, 0.00088, 0.00087, 0.00089, 0.00089, 0.00088, 0.00089, 0.00089, 0.00088, 0.00091, 0.00089, 0.00087, 0.0009, 0.00088, 0.00089, 0.00088, 0.00093, 0.00116, 0.00101, 0.00088, 0.00095, 0.00092, 0.00089, 0.00088, 0.00087, 0.00089, 0.00105, 0.0009, 0.00087]}, "optimizer-count-zeros-time": {"start_step": 0, "end_step": 1983, "step_interval": 5, "values": [0.01277, 0.00497, 0.00488, 0.00489, 0.00489, 0.00494, 0.00489, 0.0049, 0.00489, 0.00488, 0.00497, 0.00521, 0.0049, 0.00492, 0.00492, 0.0049, 0.00494, 0.00492, 0.00489, 0.00489, 0.00493, 0.0049, 0.00492, 0.0051, 0.00487, 0.00629, 0.005, 0.0049, 0.00492, 0.0049, 0.0049, 0.0049, 0.00488, 0.00492, 0.00535, 0.0049, 0.0049, 0.00494, 0.0049, 0.00494, 0.00489, 0.00489, 0.0049, 0.00491, 0.00492, 0.00491, 0.00599, 0.00523, 0.00489, 0.00489, 0.00491, 0.00491, 0.00491, 0.00494, 0.0049, 0.00489, 0.00491, 0.0049, 0.00491, 0.0049, 0.00491, 0.0049, 0.00525, 0.00492, 0.00493, 0.00489, 0.00489, 0.00492, 0.00491, 0.0049, 0.00491, 0.00491, 0.00492, 0.00489, 0.00489, 0.00493, 0.00493, 0.00498, 0.00519, 0.00491, 0.00491, 0.00492, 0.00498, 0.00492, 0.00494, 0.0049, 0.00489, 0.00567, 0.00489, 0.00491, 0.00491, 0.00524, 0.00489, 0.00491, 0.00489, 0.00504, 0.0056, 0.00501, 0.00491, 0.00493, 0.00492, 0.00491, 0.00491, 0.00491, 0.00489, 0.0049, 0.0049, 0.0049, 0.00492, 0.0049, 0.00491, 0.00491, 0.00602, 0.0049, 0.00494, 0.00489, 0.0049, 0.0049, 0.00491, 0.00492, 0.0049, 0.0049, 0.00491, 0.00598, 0.00492, 0.00491, 0.00489, 0.00494, 0.00491, 0.00491, 0.0049, 0.00494, 0.00492, 0.00544, 0.00488, 0.00491, 0.0049, 0.0049, 0.00503, 0.00491, 0.00491, 0.00491, 0.00493, 0.00494, 0.00493, 0.00492, 0.0049, 0.00492, 0.00488, 0.00489, 0.00515, 0.0049, 0.00498, 0.00492, 0.00493, 0.0049, 0.00491, 0.005, 0.00491, 0.00491, 0.00491, 0.00491, 0.00489, 0.00491, 0.0049, 0.0049, 0.00496, 0.00492, 0.00488, 0.00492, 0.00538, 0.00492, 0.00491, 0.00492, 0.00567, 0.00488, 0.00491, 0.00493, 0.00492, 0.00487, 0.00493, 0.0049, 0.00488, 0.00491, 0.00492, 0.0049, 0.00492, 0.0049, 0.0049, 0.00492, 0.0049, 0.0051, 0.0049, 0.00519, 0.00491, 0.00491, 0.00488, 0.00488, 0.00489, 0.00489, 0.00491, 0.00583, 0.0049, 0.0049, 0.00489, 0.00488, 0.0049, 0.00489, 0.00491, 0.00488, 0.0049, 0.00501, 0.00492, 0.00491, 0.0049, 0.0049, 0.0049, 0.00488, 0.0049, 0.00489, 0.00489, 0.0049, 0.00489, 0.00492, 0.00493, 0.00488, 0.0049, 0.00489, 0.0049, 0.00489, 0.00494, 0.00489, 0.00491, 0.00489, 0.00489, 0.0049, 0.00492, 0.00487, 0.00491, 0.00491, 0.00489, 0.00489, 0.00489, 0.00491, 0.00578, 0.0049, 0.00488, 0.00487, 0.00492, 0.0049, 0.00491, 0.00489, 0.00489, 0.00488, 0.0049, 0.00489, 0.00489, 0.00491, 0.00515, 0.00494, 0.0049, 0.00489, 0.00492, 0.00489, 0.00502, 0.00489, 0.00493, 0.00489, 0.00491, 0.00491, 0.00489, 0.0049, 0.00582, 0.00487, 0.00489, 0.0049, 0.00491, 0.00488, 0.00489, 0.00492, 0.00488, 0.00489, 0.00491, 0.00489, 0.00489, 0.0049, 0.00489, 0.00558, 0.00491, 0.0056, 0.00495, 0.00488, 0.00491, 0.00489, 0.00489, 0.00488, 0.0049, 0.0049, 0.00489, 0.00492, 0.00491, 0.0049, 0.00491, 0.00489, 0.0049, 0.00491, 0.00492, 0.00512, 0.00493, 0.00491, 0.00491, 0.0049, 0.00491, 0.00492, 0.00579, 0.00626, 0.00489, 0.00489, 0.0049, 0.00489, 0.00491, 0.00494, 0.00489, 0.00491, 0.0049, 0.0049, 0.00491, 0.00512, 0.0051, 0.00514, 0.00513, 0.00513, 0.00514, 0.00513, 0.00512, 0.00511, 0.00512, 0.00514, 0.0052, 0.00512, 0.00511, 0.00513, 0.00514, 0.00511, 0.00511, 0.00514, 0.00564, 0.00511, 0.00512, 0.00509, 0.00512, 0.00512, 0.00536, 0.00513, 0.00512, 0.00513, 0.00512, 0.00513, 0.00512, 0.00512, 0.00512, 0.00512, 0.00509, 0.00512, 0.00512, 0.00513, 0.00512, 0.00514, 0.00515, 0.00514, 0.00516, 0.00512, 0.00513, 0.00514, 0.00511, 0.00513, 0.00524, 0.00511, 0.00514, 0.00512, 0.00511, 0.00509, 0.00513, 0.00511, 0.00514, 0.00513, 0.00513, 0.00512, 0.0055, 0.0054, 0.00513, 0.0051, 0.0051, 0.00512, 0.00514, 0.00515, 0.00515]}, "optimizer-inner-step-time": {"start_step": 0, "end_step": 1983, "step_interval": 5, "values": [0.00686, 0.00099, 0.00098, 0.00098, 0.00098, 0.001, 0.00099, 0.00099, 0.00098, 0.00099, 0.00101, 0.00098, 0.00099, 0.00099, 0.00099, 0.001, 0.00099, 0.001, 0.001, 0.001, 0.001, 0.00098, 0.00097, 0.00099, 0.00098, 0.00124, 0.00098, 0.00098, 0.00098, 0.00098, 0.00098, 0.00101, 0.00101, 0.001, 0.001, 0.00098, 0.00099, 0.001, 0.00102, 0.00098, 0.00098, 0.00099, 0.00098, 0.00098, 0.00098, 0.00097, 0.001, 0.00102, 0.00097, 0.00098, 0.00099, 0.001, 0.00097, 0.00102, 0.00099, 0.00098, 0.00098, 0.00098, 0.001, 0.001, 0.001, 0.00098, 0.00098, 0.00098, 0.00098, 0.00098, 0.00097, 0.00097, 0.00099, 0.00098, 0.00098, 0.00098, 0.00104, 0.00097, 0.00098, 0.00099, 0.00098, 0.00117, 0.00101, 0.00101, 0.00099, 0.00097, 0.00098, 0.00097, 0.00099, 0.00098, 0.00098, 0.00101, 0.00099, 0.00098, 0.00098, 0.00098, 0.001, 0.00097, 0.00097, 0.00098, 0.001, 0.00097, 0.00097, 0.00098, 0.00099, 0.00098, 0.00098, 0.00098, 0.00098, 0.00097, 0.00097, 0.00098, 0.001, 0.00099, 0.00097, 0.00098, 0.001, 0.00098, 0.00098, 0.00099, 0.00098, 0.00098, 0.00099, 0.00099, 0.00099, 0.00097, 0.00097, 0.00099, 0.00098, 0.00097, 0.001, 0.00099, 0.00098, 0.00099, 0.001, 0.00097, 0.00099, 0.00102, 0.00099, 0.00098, 0.00097, 0.00099, 0.00099, 0.001, 0.00097, 0.00097, 0.00098, 0.00099, 0.001, 0.001, 0.00098, 0.001, 0.001, 0.00097, 0.00101, 0.00097, 0.00099, 0.00099, 0.00098, 0.001, 0.00099, 0.00098, 0.001, 0.00097, 0.00098, 0.001, 0.00099, 0.00099, 0.00099, 0.00098, 0.00098, 0.00097, 0.00098, 0.00099, 0.00098, 0.00099, 0.00097, 0.00098, 0.00103, 0.00097, 0.00097, 0.001, 0.00099, 0.00098, 0.00098, 0.00099, 0.00097, 0.00098, 0.00098, 0.00101, 0.001, 0.00099, 0.00098, 0.00098, 0.00097, 0.00102, 0.00098, 0.00097, 0.00098, 0.00097, 0.00097, 0.00099, 0.00102, 0.00096, 0.00099, 0.00097, 0.00096, 0.00097, 0.00097, 0.00099, 0.00096, 0.00097, 0.00097, 0.00097, 0.00096, 0.00097, 0.00098, 0.00097, 0.00098, 0.00156, 0.00097, 0.00096, 0.00097, 0.00096, 0.001, 0.00101, 0.00097, 0.00099, 0.00097, 0.00096, 0.00098, 0.00098, 0.00103, 0.00097, 0.00097, 0.00097, 0.00096, 0.00097, 0.00096, 0.00097, 0.00099, 0.00097, 0.00096, 0.00098, 0.00098, 0.00097, 0.00098, 0.00099, 0.00099, 0.00098, 0.00097, 0.00098, 0.00097, 0.00098, 0.00099, 0.001, 0.00099, 0.00098, 0.001, 0.00099, 0.00099, 0.00101, 0.00102, 0.00099, 0.00099, 0.00098, 0.00098, 0.00099, 0.00098, 0.00099, 0.00099, 0.00099, 0.001, 0.00099, 0.00099, 0.00098, 0.00101, 0.00099, 0.00099, 0.00099, 0.00097, 0.00099, 0.00099, 0.00098, 0.00098, 0.00104, 0.00098, 0.00099, 0.00099, 0.00098, 0.00099, 0.00099, 0.00099, 0.00099, 0.001, 0.00097, 0.00099, 0.00098, 0.00098, 0.001, 0.00099, 0.00099, 0.00098, 0.00099, 0.00098, 0.00097, 0.00098, 0.00099, 0.00099, 0.00099, 0.00098, 0.00104, 0.00099, 0.00098, 0.00099, 0.00099, 0.001, 0.00099, 0.00098, 0.001, 0.00099, 0.00096, 0.00098, 0.00099, 0.00099, 0.001, 0.00099, 0.00097, 0.00099, 0.00099, 0.00098, 0.00099, 0.00099, 0.00103, 0.00099, 0.00098, 0.00099, 0.00097, 0.00098, 0.00099, 0.00098, 0.00098, 0.00101, 0.00098, 0.00099, 0.00099, 0.00098, 0.00156, 0.00103, 0.00098, 0.001, 0.00098, 0.00099, 0.00098, 0.00098, 0.00099, 0.00098, 0.001, 0.001, 0.00098, 0.00102, 0.00098, 0.00098, 0.00099, 0.00098, 0.00098, 0.00099, 0.001, 0.00098, 0.00098, 0.00098, 0.00098, 0.00098, 0.00099, 0.00097, 0.00099, 0.00096, 0.00102, 0.00098, 0.00099, 0.001, 0.001, 0.001, 0.001, 0.00099, 0.001, 0.00099, 0.00099, 0.001, 0.001, 0.00104, 0.00099, 0.00099, 0.00099, 0.00099, 0.001, 0.00099, 0.00099, 0.001, 0.00099, 0.00099]}, "optimizer-copy-main-to-model-params-time": {"start_step": 0, "end_step": 1983, "step_interval": 5, "values": [0.00107, 0.00103, 0.00102, 0.00102, 0.00102, 0.00102, 0.00102, 0.00102, 0.00103, 0.00103, 0.00103, 0.00103, 0.00104, 0.00101, 0.00103, 0.00103, 0.00104, 0.00105, 0.00103, 0.00103, 0.00104, 0.00103, 0.00102, 0.00104, 0.00102, 0.00163, 0.00103, 0.00103, 0.00102, 0.00102, 0.00102, 0.00102, 0.00102, 0.00104, 0.00104, 0.00103, 0.00102, 0.00103, 0.00104, 0.00103, 0.00103, 0.00103, 0.00102, 0.00103, 0.00103, 0.00102, 0.00108, 0.00106, 0.00102, 0.00103, 0.00103, 0.00104, 0.00102, 0.00103, 0.00102, 0.00103, 0.00102, 0.00103, 0.00102, 0.00103, 0.00103, 0.00103, 0.00103, 0.00104, 0.00102, 0.00103, 0.00102, 0.00103, 0.00102, 0.00103, 0.00102, 0.00102, 0.00103, 0.00103, 0.00102, 0.00103, 0.00102, 0.00115, 0.00105, 0.00126, 0.00103, 0.00103, 0.00102, 0.00103, 0.00103, 0.00103, 0.00102, 0.00106, 0.00102, 0.00103, 0.00102, 0.00114, 0.00102, 0.00103, 0.00102, 0.00102, 0.00104, 0.00103, 0.00103, 0.00103, 0.00102, 0.00103, 0.00102, 0.00102, 0.00102, 0.00102, 0.00102, 0.00103, 0.00102, 0.00103, 0.00102, 0.00103, 0.00107, 0.00103, 0.00103, 0.00103, 0.00102, 0.00103, 0.00102, 0.00103, 0.00102, 0.00103, 0.00102, 0.00109, 0.00103, 0.00103, 0.00103, 0.00105, 0.00102, 0.00103, 0.00103, 0.00103, 0.00103, 0.00105, 0.00102, 0.00102, 0.00103, 0.00103, 0.00102, 0.00102, 0.00103, 0.00103, 0.00102, 0.00103, 0.00103, 0.00103, 0.00102, 0.00103, 0.00102, 0.00102, 0.00104, 0.00103, 0.00103, 0.00103, 0.00102, 0.00102, 0.00102, 0.00102, 0.00102, 0.00102, 0.00102, 0.00103, 0.00103, 0.00103, 0.00103, 0.00103, 0.00102, 0.00102, 0.00102, 0.00104, 0.00102, 0.00103, 0.00102, 0.00102, 0.00108, 0.00103, 0.00102, 0.00103, 0.00115, 0.00102, 0.00103, 0.00103, 0.00102, 0.00103, 0.00103, 0.00103, 0.00102, 0.00103, 0.00104, 0.00103, 0.00102, 0.00106, 0.00102, 0.00102, 0.00103, 0.00103, 0.00099, 0.001, 0.00103, 0.001, 0.001, 0.00105, 0.00101, 0.00099, 0.00099, 0.001, 0.00099, 0.001, 0.00099, 0.00099, 0.00099, 0.00099, 0.00099, 0.00099, 0.001, 0.00111, 0.001, 0.00099, 0.001, 0.00099, 0.00105, 0.00099, 0.00099, 0.001, 0.00099, 0.00099, 0.00099, 0.00099, 0.001, 0.001, 0.00099, 0.001, 0.00099, 0.00099, 0.00101, 0.00099, 0.00101, 0.001, 0.00099, 0.001, 0.00106, 0.001, 0.001, 0.001, 0.00104, 0.001, 0.001, 0.001, 0.00099, 0.00106, 0.001, 0.001, 0.001, 0.001, 0.00101, 0.00102, 0.00099, 0.00101, 0.00101, 0.001, 0.00099, 0.001, 0.00101, 0.00101, 0.00101, 0.00106, 0.001, 0.00101, 0.001, 0.00102, 0.001, 0.00101, 0.00106, 0.001, 0.001, 0.00101, 0.00099, 0.00101, 0.00101, 0.00101, 0.001, 0.00101, 0.00101, 0.00101, 0.001, 0.00101, 0.00101, 0.00105, 0.00101, 0.00103, 0.00101, 0.001, 0.001, 0.00101, 0.00107, 0.001, 0.00106, 0.001, 0.001, 0.001, 0.001, 0.001, 0.001, 0.001, 0.001, 0.00101, 0.00102, 0.00102, 0.001, 0.001, 0.00101, 0.001, 0.001, 0.001, 0.00106, 0.00107, 0.00099, 0.00107, 0.001, 0.00101, 0.001, 0.001, 0.001, 0.00101, 0.001, 0.001, 0.00101, 0.001, 0.00099, 0.001, 0.00101, 0.001, 0.001, 0.001, 0.001, 0.00107, 0.001, 0.001, 0.00101, 0.001, 0.001, 0.001, 0.00101, 0.00106, 0.00099, 0.00102, 0.00102, 0.001, 0.001, 0.001, 0.001, 0.00101, 0.00099, 0.00103, 0.00101, 0.001, 0.001, 0.001, 0.001, 0.001, 0.001, 0.00101, 0.00103, 0.00102, 0.001, 0.00101, 0.001, 0.001, 0.001, 0.001, 0.001, 0.001, 0.00099, 0.00102, 0.001, 0.001, 0.001, 0.00101, 0.00101, 0.001, 0.00099, 0.001, 0.00101, 0.00102, 0.00101, 0.00102, 0.00101, 0.00101, 0.001, 0.00101, 0.00101, 0.00101, 0.001, 0.001, 0.00101, 0.001, 0.001]}, "grad-norm": {"start_step": 0, "end_step": 1983, "step_interval": 5, "values": [11.77525, 12.26804, 11.19281, 14.50237, 14.014, 11.57186, 8.3922, 7.10897, 4.47266, 4.00434, 3.4, 2.71736, 2.45629, 2.30739, 2.29493, 2.25132, 2.01839, 2.41173, 2.01298, 2.00525, 2.18932, 1.91353, 1.88951, 2.28883, 2.07903, 1.8844, 1.87495, 2.08513, 2.01874, 2.01118, 2.0102, 1.89229, 1.99489, 1.65446, 2.02134, 1.98456, 2.13312, 2.05074, 1.91832, 1.88506, 1.86975, 1.90714, 2.10548, 1.83107, 1.85561, 1.89757, 1.77389, 1.83901, 1.60882, 1.67073, 1.57953, 1.73056, 1.77582, 1.85094, 1.58796, 1.69243, 2.01012, 1.72305, 1.68342, 1.77634, 1.52051, 1.58604, 1.75613, 1.50876, 1.38814, 1.4853, 1.45829, 1.51675, 1.54655, 1.47158, 1.51099, 1.4708, 1.47268, 1.47452, 1.44323, 1.32185, 1.33599, 1.35564, 1.29533, 1.27928, 1.44962, 1.33226, 1.18991, 1.39956, 1.21257, 1.16175, 1.05645, 1.15134, 1.32979, 1.15427, 1.22191, 1.18197, 1.5911, 1.3589, 1.27604, 1.13871, 1.30626, 1.67866, 1.52014, 1.03431, 1.05476, 1.3049, 1.25479, 1.22714, 1.69201, 1.08131, 1.00908, 1.10419, 1.08066, 1.12768, 1.24403, 0.87723, 0.92972, 1.02293, 1.07062, 0.98243, 1.24502, 1.2897, 0.94461, 1.09023, 1.04658, 0.90251, 1.12421, 1.65432, 1.09595, 1.17882, 1.36022, 0.96059, 0.98043, 1.05339, 0.96416, 1.13229, 1.12844, 0.93359, 1.82877, 1.40011, 1.43068, 1.3027, 1.089, 1.64716, 1.37833, 1.56985, 1.16612, 1.85125, 1.24379, 1.71309, 1.39309, 1.27937, 1.17708, 1.73543, 1.05896, 1.24373, 1.38937, 1.36918, 1.42323, 1.77943, 1.13157, 1.27948, 1.19267, 1.34154, 1.40098, 1.16252, 1.42404, 1.2011, 1.00676, 1.48416, 1.13391, 1.33486, 1.5395, 1.27609, 1.42471, 1.30575, 1.22047, 1.81347, 1.74187, 1.56562, 1.47675, 1.51655, 1.70821, 1.44154, 1.50096, 1.28826, 1.74901, 1.90029, 1.42234, 1.44455, 1.76719, 1.84971, 1.73982, 1.24814, 1.53885, 1.39306, 1.62267, 1.27091, 1.59048, 1.06674, 1.40639, 1.29128, 1.69617, 1.31246, 1.4525, 1.29959, 1.38347, 1.4963, 1.45118, 1.62261, 1.8211, 1.48622, 1.35396, 1.364, 1.22302, 1.21036, 1.59732, 1.16621, 1.43458, 1.39264, 1.50491, 1.74865, 1.69988, 1.54719, 1.66156, 1.38606, 1.43929, 1.37822, 1.30248, 1.79296, 1.45361, 1.24972, 1.59221, 1.3686, 1.22551, 1.4158, 1.49894, 1.55813, 1.52684, 1.44435, 2.05338, 1.36019, 1.34284, 1.20815, 1.7307, 1.50669, 2.1527, 1.33714, 1.40114, 1.51052, 1.35152, 1.43159, 1.42052, 1.44093, 1.62874, 1.70468, 1.84621, 1.36339, 1.49409, 1.99351, 1.25437, 1.69787, 1.77453, 1.53971, 1.98798, 1.46692, 1.21412, 1.35855, 1.61255, 1.37129, 1.69078, 1.53059, 1.31087, 1.87886, 1.31042, 1.42235, 1.38194, 1.39636, 1.83392, 1.47651, 1.46996, 1.64541, 1.53153, 1.47267, 1.75528, 1.44853, 1.39865, 1.75941, 1.63286, 1.32552, 1.6715, 2.26149, 1.61139, 1.35216, 1.34936, 1.25166, 1.69472, 1.58245, 1.4379, 1.43627, 1.60457, 1.82215, 1.39138, 1.38678, 1.55708, 1.41296, 1.29816, 1.46066, 1.39994, 1.45437, 1.25759, 1.34921, 1.47682, 1.55246, 1.48338, 1.2271, 1.36154, 1.44453, 1.47772, 1.43402, 1.21249, 1.8034, 1.50506, 1.3131, 1.37503, 1.35584, 1.41307, 1.45748, 1.26629, 1.31721, 1.47686, 1.80237, 1.55348, 1.5369, 1.32871, 1.35524, 1.76226, 1.27945, 1.40786, 1.56063, 1.18102, 1.26595, 1.41714, 1.27185, 1.59955, 1.53902, 1.50856, 1.38342, 1.3716, 1.52597, 1.55924, 1.33891, 1.44137, 1.66178, 1.44058, 1.53213, 1.34923, 1.54826, 1.51369, 1.26166, 1.22057, 1.64988, 1.4183, 1.45977, 1.27097, 1.31805, 1.24715, 1.52412, 1.48112, 1.51313, 1.58975, 1.42731, 1.32647, 1.44532, 1.53827, 1.72661, 1.53155, 1.57687, 1.2723, 1.26403, 1.36125, 1.36611, 1.46818, 1.38679, 1.58433, 1.49566, 1.44288, 1.37271, 1.45317, 1.36918, 1.35342, 1.27732, 1.37088, 1.29411, 1.25869, 1.46478, 1.43992, 1.66108, 1.34488, 1.17599, 1.3251]}, "grad-norm vs samples": {"start_step": 0, "end_step": 1983, "step_interval": 5, "values": [11.77525, 12.26804, 11.19281, 14.50237, 14.014, 11.57186, 8.3922, 7.10897, 4.47266, 4.00434, 3.4, 2.71736, 2.45629, 2.30739, 2.29493, 2.25132, 2.01839, 2.41173, 2.01298, 2.00525, 2.18932, 1.91353, 1.88951, 2.28883, 2.07903, 1.8844, 1.87495, 2.08513, 2.01874, 2.01118, 2.0102, 1.89229, 1.99489, 1.65446, 2.02134, 1.98456, 2.13312, 2.05074, 1.91832, 1.88506, 1.86975, 1.90714, 2.10548, 1.83107, 1.85561, 1.89757, 1.77389, 1.83901, 1.60882, 1.67073, 1.57953, 1.73056, 1.77582, 1.85094, 1.58796, 1.69243, 2.01012, 1.72305, 1.68342, 1.77634, 1.52051, 1.58604, 1.75613, 1.50876, 1.38814, 1.4853, 1.45829, 1.51675, 1.54655, 1.47158, 1.51099, 1.4708, 1.47268, 1.47452, 1.44323, 1.32185, 1.33599, 1.35564, 1.29533, 1.27928, 1.44962, 1.33226, 1.18991, 1.39956, 1.21257, 1.16175, 1.05645, 1.15134, 1.32979, 1.15427, 1.22191, 1.18197, 1.5911, 1.3589, 1.27604, 1.13871, 1.30626, 1.67866, 1.52014, 1.03431, 1.05476, 1.3049, 1.25479, 1.22714, 1.69201, 1.08131, 1.00908, 1.10419, 1.08066, 1.12768, 1.24403, 0.87723, 0.92972, 1.02293, 1.07062, 0.98243, 1.24502, 1.2897, 0.94461, 1.09023, 1.04658, 0.90251, 1.12421, 1.65432, 1.09595, 1.17882, 1.36022, 0.96059, 0.98043, 1.05339, 0.96416, 1.13229, 1.12844, 0.93359, 1.82877, 1.40011, 1.43068, 1.3027, 1.089, 1.64716, 1.37833, 1.56985, 1.16612, 1.85125, 1.24379, 1.71309, 1.39309, 1.27937, 1.17708, 1.73543, 1.05896, 1.24373, 1.38937, 1.36918, 1.42323, 1.77943, 1.13157, 1.27948, 1.19267, 1.34154, 1.40098, 1.16252, 1.42404, 1.2011, 1.00676, 1.48416, 1.13391, 1.33486, 1.5395, 1.27609, 1.42471, 1.30575, 1.22047, 1.81347, 1.74187, 1.56562, 1.47675, 1.51655, 1.70821, 1.44154, 1.50096, 1.28826, 1.74901, 1.90029, 1.42234, 1.44455, 1.76719, 1.84971, 1.73982, 1.24814, 1.53885, 1.39306, 1.62267, 1.27091, 1.59048, 1.06674, 1.40639, 1.29128, 1.69617, 1.31246, 1.4525, 1.29959, 1.38347, 1.4963, 1.45118, 1.62261, 1.8211, 1.48622, 1.35396, 1.364, 1.22302, 1.21036, 1.59732, 1.16621, 1.43458, 1.39264, 1.50491, 1.74865, 1.69988, 1.54719, 1.66156, 1.38606, 1.43929, 1.37822, 1.30248, 1.79296, 1.45361, 1.24972, 1.59221, 1.3686, 1.22551, 1.4158, 1.49894, 1.55813, 1.52684, 1.44435, 2.05338, 1.36019, 1.34284, 1.20815, 1.7307, 1.50669, 2.1527, 1.33714, 1.40114, 1.51052, 1.35152, 1.43159, 1.42052, 1.44093, 1.62874, 1.70468, 1.84621, 1.36339, 1.49409, 1.99351, 1.25437, 1.69787, 1.77453, 1.53971, 1.98798, 1.46692, 1.21412, 1.35855, 1.61255, 1.37129, 1.69078, 1.53059, 1.31087, 1.87886, 1.31042, 1.42235, 1.38194, 1.39636, 1.83392, 1.47651, 1.46996, 1.64541, 1.53153, 1.47267, 1.75528, 1.44853, 1.39865, 1.75941, 1.63286, 1.32552, 1.6715, 2.26149, 1.61139, 1.35216, 1.34936, 1.25166, 1.69472, 1.58245, 1.4379, 1.43627, 1.60457, 1.82215, 1.39138, 1.38678, 1.55708, 1.41296, 1.29816, 1.46066, 1.39994, 1.45437, 1.25759, 1.34921, 1.47682, 1.55246, 1.48338, 1.2271, 1.36154, 1.44453, 1.47772, 1.43402, 1.21249, 1.8034, 1.50506, 1.3131, 1.37503, 1.35584, 1.41307, 1.45748, 1.26629, 1.31721, 1.47686, 1.80237, 1.55348, 1.5369, 1.32871, 1.35524, 1.76226, 1.27945, 1.40786, 1.56063, 1.18102, 1.26595, 1.41714, 1.27185, 1.59955, 1.53902, 1.50856, 1.38342, 1.3716, 1.52597, 1.55924, 1.33891, 1.44137, 1.66178, 1.44058, 1.53213, 1.34923, 1.54826, 1.51369, 1.26166, 1.22057, 1.64988, 1.4183, 1.45977, 1.27097, 1.31805, 1.24715, 1.52412, 1.48112, 1.51313, 1.58975, 1.42731, 1.32647, 1.44532, 1.53827, 1.72661, 1.53155, 1.57687, 1.2723, 1.26403, 1.36125, 1.36611, 1.46818, 1.38679, 1.58433, 1.49566, 1.44288, 1.37271, 1.45317, 1.36918, 1.35342, 1.27732, 1.37088, 1.29411, 1.25869, 1.46478, 1.43992, 1.66108, 1.34488, 1.17599, 1.3251]}, "num-zeros": {"start_step": 0, "end_step": 1983, "step_interval": 5, "values": [951.0, 1294.0, 1060.0, 971.0, 901.0, 1117.0, 1205.0, 1364.0, 1468.0, 1319.0, 1539.0, 1911.0, 2180.0, 1576.0, 2216.0, 1925.0, 2038.0, 2028.0, 2476.0, 2015.0, 2201.0, 2215.0, 2438.0, 3135.0, 2444.0, 2806.0, 2540.0, 2188.0, 2052.0, 2885.0, 2408.0, 3553.0, 2417.0, 2497.0, 2486.0, 3667.0, 2116.0, 2243.0, 2127.0, 2649.0, 3818.0, 2985.0, 2311.0, 2810.0, 2580.0, 2214.0, 2672.0, 2502.0, 2376.0, 2941.0, 3128.0, 2507.0, 2600.0, 2152.0, 2790.0, 3240.0, 2769.0, 2720.0, 2392.0, 3522.0, 2236.0, 2883.0, 2397.0, 2586.0, 2219.0, 3154.0, 2799.0, 2803.0, 2345.0, 2563.0, 2171.0, 2874.0, 2837.0, 2656.0, 3389.0, 2526.0, 2817.0, 2625.0, 3000.0, 2814.0, 2754.0, 2414.0, 3081.0, 2380.0, 2876.0, 2737.0, 2780.0, 2271.0, 2333.0, 2839.0, 2519.0, 3210.0, 2404.0, 2291.0, 2433.0, 2383.0, 2435.0, 1919.0, 2351.0, 2585.0, 2779.0, 2221.0, 2014.0, 2114.0, 1881.0, 2304.0, 2397.0, 2309.0, 2239.0, 2116.0, 2239.0, 2377.0, 2323.0, 2496.0, 2298.0, 2773.0, 2696.0, 1952.0, 2435.0, 2042.0, 2813.0, 2452.0, 2068.0, 2032.0, 2127.0, 2176.0, 2056.0, 2569.0, 2495.0, 2156.0, 2202.0, 2372.0, 2368.0, 2313.0, 1956.0, 2287.0, 2471.0, 2251.0, 2132.0, 1626.0, 2076.0, 2288.0, 2009.0, 1987.0, 2433.0, 1651.0, 2033.0, 2061.0, 1927.0, 2837.0, 2589.0, 2063.0, 1738.0, 1964.0, 2334.0, 1899.0, 2516.0, 2136.0, 2214.0, 1965.0, 1875.0, 2415.0, 1921.0, 2352.0, 2174.0, 1887.0, 2165.0, 2616.0, 1911.0, 1825.0, 1959.0, 1908.0, 1822.0, 1574.0, 1545.0, 2160.0, 1942.0, 2081.0, 1733.0, 2008.0, 2010.0, 2212.0, 1875.0, 1390.0, 1972.0, 2540.0, 1825.0, 2152.0, 1632.0, 2232.0, 1792.0, 1887.0, 1971.0, 2046.0, 1779.0, 2139.0, 2024.0, 1999.0, 1614.0, 1985.0, 1902.0, 2128.0, 2445.0, 2671.0, 2214.0, 2029.0, 2081.0, 2209.0, 2226.0, 1957.0, 2210.0, 2419.0, 2685.0, 2294.0, 1932.0, 2118.0, 1963.0, 1818.0, 1841.0, 2149.0, 2110.0, 2155.0, 1868.0, 2220.0, 2120.0, 2379.0, 1886.0, 2361.0, 1763.0, 2055.0, 1972.0, 2155.0, 1934.0, 2167.0, 1959.0, 1882.0, 1705.0, 1826.0, 1964.0, 2224.0, 1818.0, 1883.0, 1743.0, 2488.0, 2393.0, 2103.0, 2005.0, 2728.0, 2142.0, 2054.0, 1951.0, 1819.0, 2038.0, 2170.0, 2265.0, 1808.0, 2431.0, 1807.0, 2184.0, 2053.0, 1687.0, 1931.0, 2549.0, 2587.0, 1986.0, 2273.0, 2103.0, 2063.0, 2204.0, 2021.0, 2110.0, 2428.0, 2484.0, 2060.0, 2244.0, 2025.0, 1999.0, 1965.0, 1906.0, 2137.0, 2024.0, 2234.0, 1998.0, 2022.0, 1943.0, 2254.0, 2008.0, 1619.0, 1850.0, 2446.0, 2316.0, 1952.0, 2008.0, 2201.0, 2018.0, 2191.0, 1856.0, 2363.0, 2138.0, 2632.0, 1897.0, 2331.0, 1915.0, 2017.0, 2347.0, 2073.0, 2221.0, 2341.0, 1910.0, 1944.0, 2197.0, 2136.0, 2140.0, 2057.0, 2254.0, 1992.0, 2377.0, 1829.0, 2323.0, 2256.0, 2248.0, 2664.0, 2091.0, 2351.0, 2363.0, 2417.0, 1953.0, 2010.0, 2111.0, 2082.0, 2141.0, 2449.0, 2394.0, 2165.0, 2019.0, 2307.0, 2446.0, 2932.0, 2123.0, 2428.0, 2294.0, 2499.0, 2597.0, 2391.0, 2142.0, 2085.0, 2112.0, 2498.0, 2172.0, 2546.0, 2086.0, 2278.0, 2000.0, 2060.0, 2222.0, 2327.0, 2377.0, 2181.0, 1943.0, 2370.0, 2170.0, 2277.0, 2360.0, 2822.0, 2306.0, 2709.0, 2210.0, 2127.0, 2321.0, 2202.0, 2780.0, 2249.0, 2312.0, 2033.0, 2114.0, 2287.0, 2292.0, 2301.0, 2735.0, 2674.0, 2246.0, 2584.0, 2280.0, 2624.0, 2634.0, 2653.0, 2502.0, 2748.0, 2256.0, 2492.0, 2276.0, 2217.0, 1995.0, 2408.0, 2306.0, 2584.0, 2373.0]}, "num-zeros vs samples": {"start_step": 0, "end_step": 1983, "step_interval": 5, "values": [951.0, 1294.0, 1060.0, 971.0, 901.0, 1117.0, 1205.0, 1364.0, 1468.0, 1319.0, 1539.0, 1911.0, 2180.0, 1576.0, 2216.0, 1925.0, 2038.0, 2028.0, 2476.0, 2015.0, 2201.0, 2215.0, 2438.0, 3135.0, 2444.0, 2806.0, 2540.0, 2188.0, 2052.0, 2885.0, 2408.0, 3553.0, 2417.0, 2497.0, 2486.0, 3667.0, 2116.0, 2243.0, 2127.0, 2649.0, 3818.0, 2985.0, 2311.0, 2810.0, 2580.0, 2214.0, 2672.0, 2502.0, 2376.0, 2941.0, 3128.0, 2507.0, 2600.0, 2152.0, 2790.0, 3240.0, 2769.0, 2720.0, 2392.0, 3522.0, 2236.0, 2883.0, 2397.0, 2586.0, 2219.0, 3154.0, 2799.0, 2803.0, 2345.0, 2563.0, 2171.0, 2874.0, 2837.0, 2656.0, 3389.0, 2526.0, 2817.0, 2625.0, 3000.0, 2814.0, 2754.0, 2414.0, 3081.0, 2380.0, 2876.0, 2737.0, 2780.0, 2271.0, 2333.0, 2839.0, 2519.0, 3210.0, 2404.0, 2291.0, 2433.0, 2383.0, 2435.0, 1919.0, 2351.0, 2585.0, 2779.0, 2221.0, 2014.0, 2114.0, 1881.0, 2304.0, 2397.0, 2309.0, 2239.0, 2116.0, 2239.0, 2377.0, 2323.0, 2496.0, 2298.0, 2773.0, 2696.0, 1952.0, 2435.0, 2042.0, 2813.0, 2452.0, 2068.0, 2032.0, 2127.0, 2176.0, 2056.0, 2569.0, 2495.0, 2156.0, 2202.0, 2372.0, 2368.0, 2313.0, 1956.0, 2287.0, 2471.0, 2251.0, 2132.0, 1626.0, 2076.0, 2288.0, 2009.0, 1987.0, 2433.0, 1651.0, 2033.0, 2061.0, 1927.0, 2837.0, 2589.0, 2063.0, 1738.0, 1964.0, 2334.0, 1899.0, 2516.0, 2136.0, 2214.0, 1965.0, 1875.0, 2415.0, 1921.0, 2352.0, 2174.0, 1887.0, 2165.0, 2616.0, 1911.0, 1825.0, 1959.0, 1908.0, 1822.0, 1574.0, 1545.0, 2160.0, 1942.0, 2081.0, 1733.0, 2008.0, 2010.0, 2212.0, 1875.0, 1390.0, 1972.0, 2540.0, 1825.0, 2152.0, 1632.0, 2232.0, 1792.0, 1887.0, 1971.0, 2046.0, 1779.0, 2139.0, 2024.0, 1999.0, 1614.0, 1985.0, 1902.0, 2128.0, 2445.0, 2671.0, 2214.0, 2029.0, 2081.0, 2209.0, 2226.0, 1957.0, 2210.0, 2419.0, 2685.0, 2294.0, 1932.0, 2118.0, 1963.0, 1818.0, 1841.0, 2149.0, 2110.0, 2155.0, 1868.0, 2220.0, 2120.0, 2379.0, 1886.0, 2361.0, 1763.0, 2055.0, 1972.0, 2155.0, 1934.0, 2167.0, 1959.0, 1882.0, 1705.0, 1826.0, 1964.0, 2224.0, 1818.0, 1883.0, 1743.0, 2488.0, 2393.0, 2103.0, 2005.0, 2728.0, 2142.0, 2054.0, 1951.0, 1819.0, 2038.0, 2170.0, 2265.0, 1808.0, 2431.0, 1807.0, 2184.0, 2053.0, 1687.0, 1931.0, 2549.0, 2587.0, 1986.0, 2273.0, 2103.0, 2063.0, 2204.0, 2021.0, 2110.0, 2428.0, 2484.0, 2060.0, 2244.0, 2025.0, 1999.0, 1965.0, 1906.0, 2137.0, 2024.0, 2234.0, 1998.0, 2022.0, 1943.0, 2254.0, 2008.0, 1619.0, 1850.0, 2446.0, 2316.0, 1952.0, 2008.0, 2201.0, 2018.0, 2191.0, 1856.0, 2363.0, 2138.0, 2632.0, 1897.0, 2331.0, 1915.0, 2017.0, 2347.0, 2073.0, 2221.0, 2341.0, 1910.0, 1944.0, 2197.0, 2136.0, 2140.0, 2057.0, 2254.0, 1992.0, 2377.0, 1829.0, 2323.0, 2256.0, 2248.0, 2664.0, 2091.0, 2351.0, 2363.0, 2417.0, 1953.0, 2010.0, 2111.0, 2082.0, 2141.0, 2449.0, 2394.0, 2165.0, 2019.0, 2307.0, 2446.0, 2932.0, 2123.0, 2428.0, 2294.0, 2499.0, 2597.0, 2391.0, 2142.0, 2085.0, 2112.0, 2498.0, 2172.0, 2546.0, 2086.0, 2278.0, 2000.0, 2060.0, 2222.0, 2327.0, 2377.0, 2181.0, 1943.0, 2370.0, 2170.0, 2277.0, 2360.0, 2822.0, 2306.0, 2709.0, 2210.0, 2127.0, 2321.0, 2202.0, 2780.0, 2249.0, 2312.0, 2033.0, 2114.0, 2287.0, 2292.0, 2301.0, 2735.0, 2674.0, 2246.0, 2584.0, 2280.0, 2624.0, 2634.0, 2653.0, 2502.0, 2748.0, 2256.0, 2492.0, 2276.0, 2217.0, 1995.0, 2408.0, 2306.0, 2584.0, 2373.0]}, "lm loss validation": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [5.62692]}, "lm loss validation vs samples": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [5.62692]}, "lm loss validation ppl": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [277.80627]}, "lm loss validation ppl vs samples": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [277.80627]}}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_bf16_baseline/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_bf16_baseline/model_config.yaml
deleted file mode 100644
index db5dea3ae6e..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_bf16_baseline/model_config.yaml
+++ /dev/null
@@ -1,50 +0,0 @@
-ENV_VARS:
- CUDA_DEVICE_MAX_CONNECTIONS: 1
- NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
- NCCL_ALGO: Ring
- CUBLAS_WORKSPACE_CONFIG: :4096:8
-MODEL_ARGS:
- --num-layers: 12
- --hidden-size: 512
- --num-attention-heads: 8
- --log-params-norm: true
- --log-num-zeros-in-grad: true
- --log-validation-ppl-to-tensorboard: true
- --log-timers-to-tensorboard: true
- --tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
- --seq-length: 1024
- --max-position-embeddings: 1024
- --train-iters: 2000
- --timing-log-level: 0
- --lr-decay-iters: 320000
- --save: ${CHECKPOINT_SAVE_PATH}
- --load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
- --split: 949,50,1
- --distributed-backend: nccl
- --lr: 0.00015
- --lr-decay-style: cosine
- --min-lr: 1.0e-5
- --weight-decay: 1e-2
- --clip-grad: 1.0
- --lr-warmup-fraction: .01
- --log-interval: 1
- --save-interval: 10000
- --eval-interval: 1000
- --eval-iters: 10
- --transformer-impl: local
- --tensor-model-parallel-size: 1
- --pipeline-model-parallel-size: 1
- --deterministic-mode: true
- --no-gradient-accumulation-fusion: true
- --ckpt-format: torch_dist
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
- --data-cache-path: ${DATA_CACHE_PATH}
- --fp16: true
- --apply-query-key-layer-scaling: true
- --log-memory-to-tensorboard: true
-TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_fp8_no_model_parallel/golden_values_dev_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_fp8_no_model_parallel/golden_values_dev_dgx_a100.json
deleted file mode 100644
index f2cc2651bba..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_fp8_no_model_parallel/golden_values_dev_dgx_a100.json
+++ /dev/null
@@ -1,1223 +0,0 @@
-{
- "lm loss": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 10.8833,
- 10.90234,
- 10.8867,
- 10.83313,
- 10.67611,
- 10.64923,
- 10.43399,
- 10.15135,
- 9.93913,
- 9.84138,
- 9.58862,
- 9.85447,
- 9.88459,
- 9.62945,
- 9.78806,
- 9.51139,
- 9.45835,
- 9.64919,
- 9.38616,
- 9.33214,
- 9.24217,
- 9.14552,
- 9.17556,
- 8.99549,
- 9.18942,
- 9.06,
- 9.15557,
- 9.16494,
- 9.29777,
- 8.98447,
- 8.9291,
- 9.0438,
- 9.04302,
- 8.65501,
- 8.71714,
- 8.75345,
- 8.68366,
- 8.73437,
- 8.65884,
- 8.76497,
- 8.66083,
- 8.84974,
- 8.83206,
- 8.49923,
- 8.38904,
- 8.43157,
- 8.49322,
- 8.38452,
- 8.43264,
- 8.57965,
- 8.36711,
- 8.19222,
- 8.22606,
- 8.22221,
- 8.26779,
- 7.91377,
- 8.09628,
- 7.89164,
- 8.2472,
- 8.23126,
- 8.00591,
- 7.9665,
- 7.91908,
- 7.74099,
- 7.7407,
- 7.64366,
- 7.51608,
- 7.90725,
- 7.6987,
- 7.45218,
- 7.74074,
- 7.76788,
- 7.54126,
- 7.29845,
- 7.45178,
- 7.3355,
- 7.46213,
- 7.22379,
- 7.63678,
- 7.27944,
- 7.35187,
- 7.21324,
- 7.21605,
- 7.42279,
- 7.17674,
- 7.28039,
- 7.00049,
- 7.00348,
- 7.0378,
- 7.13559,
- 6.8226,
- 6.98478,
- 7.08778,
- 7.00054,
- 6.87352,
- 6.7548,
- 6.98975,
- 7.05529,
- 6.70191,
- 6.57996,
- 6.72276,
- 6.73919,
- 6.73242,
- 6.73508,
- 6.65475,
- 6.40522,
- 6.63735,
- 6.61784,
- 6.44466,
- 6.62795,
- 6.74118,
- 6.60668,
- 6.72226,
- 6.69283,
- 6.62263,
- 6.50666,
- 6.59776,
- 6.40564,
- 6.66354,
- 6.24776,
- 6.2498,
- 6.30069,
- 6.38858,
- 6.34831,
- 6.45112,
- 6.29344,
- 6.33922,
- 6.23941,
- 6.20371,
- 6.40027,
- 6.32848,
- 6.32525,
- 6.17126,
- 6.1643,
- 6.2454,
- 6.39032,
- 6.20693,
- 6.15596,
- 6.18982,
- 6.12202,
- 6.07039,
- 6.07971,
- 6.26493,
- 6.41807,
- 6.26721,
- 6.30841,
- 6.10624,
- 6.18818,
- 6.01112,
- 6.03436,
- 5.96365,
- 6.25335,
- 6.19771,
- 5.97183,
- 5.78965,
- 6.12772,
- 5.85318,
- 6.10697,
- 5.79207,
- 6.16231,
- 6.14778,
- 6.08858,
- 5.93222,
- 6.11354,
- 5.94235,
- 6.19392,
- 5.89409,
- 5.79284,
- 5.77325,
- 5.68417,
- 6.01344,
- 5.99765,
- 6.06104,
- 5.88062,
- 6.03537,
- 5.96403,
- 5.99065,
- 5.98597,
- 5.9429,
- 5.83537,
- 5.94528,
- 5.61064,
- 5.69396,
- 5.88331,
- 5.83611,
- 5.8572,
- 5.75616,
- 5.8315,
- 5.72086,
- 5.55559,
- 5.71476,
- 5.62107,
- 5.82784,
- 5.59614,
- 5.70294,
- 5.70926,
- 5.89205,
- 5.63787,
- 5.84442,
- 5.73328,
- 5.86482,
- 5.32391,
- 5.88991,
- 5.86664,
- 5.84821,
- 5.40773,
- 5.40279,
- 5.6189,
- 5.58915,
- 5.47606,
- 5.56698,
- 5.66844,
- 5.46942,
- 5.73811,
- 5.50571,
- 5.58896,
- 5.61865,
- 5.61286,
- 5.50477,
- 5.60628,
- 5.66565,
- 5.69156,
- 5.58829,
- 5.65549,
- 5.3707,
- 5.67705,
- 5.62292,
- 5.41672,
- 5.5855,
- 5.62763,
- 5.55004,
- 5.33605,
- 5.5357,
- 5.48154,
- 5.47891,
- 5.37306,
- 5.55395,
- 5.59949,
- 5.38543,
- 5.52273,
- 5.48203,
- 5.3275,
- 5.50172,
- 5.40512,
- 5.4376,
- 5.31466,
- 5.06074,
- 5.47521,
- 5.56277,
- 5.70758,
- 5.41112,
- 5.59472,
- 5.62927,
- 5.23143,
- 5.26976,
- 5.39082,
- 5.38949,
- 5.32381,
- 5.49509,
- 5.18131,
- 5.29884,
- 5.24876,
- 5.37339,
- 5.25697,
- 5.44221,
- 5.53619,
- 5.30996,
- 5.43641,
- 5.33417,
- 5.06948,
- 5.3127,
- 5.25169,
- 5.30028,
- 5.10715,
- 5.2724,
- 5.26524,
- 5.46862,
- 5.15665,
- 5.26598,
- 5.20649,
- 5.35982,
- 4.98371,
- 4.91206,
- 5.31959,
- 5.38874,
- 5.22559,
- 5.31589,
- 5.1,
- 5.15578,
- 5.25723,
- 5.065,
- 5.26354,
- 5.07334,
- 5.33639,
- 5.24541,
- 5.15041,
- 5.24112,
- 5.03819,
- 5.31,
- 5.0477,
- 5.02146,
- 5.13877,
- 5.10876,
- 5.26714,
- 5.14932,
- 5.27649,
- 5.0965,
- 5.09542,
- 5.24706,
- 5.31762,
- 5.25262,
- 5.18876,
- 5.13842,
- 5.28319,
- 4.94386,
- 5.20599,
- 5.08696,
- 5.29641,
- 5.1744,
- 5.18255,
- 5.10891,
- 4.98033,
- 4.99108,
- 5.21829,
- 5.31066,
- 5.09636,
- 5.05054,
- 4.91569,
- 5.12013,
- 5.11714,
- 4.92205,
- 5.33319,
- 5.02061,
- 5.09671,
- 5.15803,
- 4.99994,
- 5.0584,
- 5.06511,
- 4.98874,
- 5.0743,
- 5.15696,
- 4.97546,
- 5.17775,
- 4.92623,
- 4.91526,
- 5.06578,
- 4.98937,
- 4.90649,
- 4.77326,
- 4.94086,
- 5.1121,
- 5.01488,
- 5.01357,
- 5.32596,
- 4.95425,
- 4.99115,
- 5.0419,
- 4.80405,
- 4.73491,
- 4.9946,
- 5.03423,
- 4.87011,
- 4.94783,
- 5.04177,
- 5.02083,
- 4.81039,
- 4.88762,
- 4.90025,
- 4.8257,
- 4.74307,
- 5.00644,
- 4.74731,
- 5.20296,
- 4.78234,
- 4.98845,
- 4.73187,
- 4.78111,
- 4.81624,
- 4.64753,
- 4.65382,
- 4.83884,
- 4.80187,
- 4.79782,
- 4.91858,
- 4.87993,
- 4.92242,
- 4.7636,
- 4.87789,
- 4.73001,
- 4.90747,
- 4.95247,
- 4.87195,
- 4.70431,
- 4.77676,
- 4.89474,
- 4.70621,
- 4.85602,
- 4.68499,
- 4.68274,
- 4.64493
- ]
- },
- "num-zeros": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 86.0,
- 65.0,
- 73.0,
- 73.0,
- 63.0,
- 79.0,
- 89.0,
- 101.0,
- 111.0,
- 114.0,
- 120.0,
- 130.0,
- 146.0,
- 151.0,
- 186.0,
- 176.0,
- 158.0,
- 185.0,
- 193.0,
- 154.0,
- 152.0,
- 162.0,
- 215.0,
- 192.0,
- 212.0,
- 153.0,
- 177.0,
- 162.0,
- 152.0,
- 166.0,
- 157.0,
- 177.0,
- 124.0,
- 172.0,
- 160.0,
- 155.0,
- 166.0,
- 189.0,
- 180.0,
- 206.0,
- 200.0,
- 165.0,
- 175.0,
- 186.0,
- 176.0,
- 183.0,
- 210.0,
- 187.0,
- 205.0,
- 245.0,
- 226.0,
- 175.0,
- 186.0,
- 163.0,
- 175.0,
- 207.0,
- 167.0,
- 137.0,
- 265.0,
- 259.0,
- 187.0,
- 185.0,
- 194.0,
- 173.0,
- 204.0,
- 254.0,
- 212.0,
- 218.0,
- 212.0,
- 228.0,
- 242.0,
- 261.0,
- 198.0,
- 226.0,
- 204.0,
- 204.0,
- 257.0,
- 207.0,
- 273.0,
- 231.0,
- 237.0,
- 222.0,
- 180.0,
- 234.0,
- 254.0,
- 226.0,
- 221.0,
- 194.0,
- 233.0,
- 188.0,
- 190.0,
- 215.0,
- 234.0,
- 212.0,
- 214.0,
- 162.0,
- 213.0,
- 214.0,
- 173.0,
- 130.0,
- 192.0,
- 183.0,
- 184.0,
- 150.0,
- 162.0,
- 148.0,
- 167.0,
- 133.0,
- 145.0,
- 190.0,
- 173.0,
- 194.0,
- 181.0,
- 174.0,
- 141.0,
- 129.0,
- 160.0,
- 131.0,
- 201.0,
- 153.0,
- 148.0,
- 141.0,
- 134.0,
- 155.0,
- 121.0,
- 99.0,
- 131.0,
- 121.0,
- 132.0,
- 144.0,
- 144.0,
- 137.0,
- 154.0,
- 113.0,
- 129.0,
- 130.0,
- 162.0,
- 109.0,
- 92.0,
- 124.0,
- 112.0,
- 117.0,
- 122.0,
- 96.0,
- 121.0,
- 120.0,
- 109.0,
- 130.0,
- 122.0,
- 141.0,
- 133.0,
- 105.0,
- 103.0,
- 131.0,
- 107.0,
- 120.0,
- 122.0,
- 101.0,
- 119.0,
- 124.0,
- 131.0,
- 116.0,
- 117.0,
- 150.0,
- 121.0,
- 112.0,
- 124.0,
- 96.0,
- 127.0,
- 103.0,
- 92.0,
- 105.0,
- 103.0,
- 124.0,
- 119.0,
- 108.0,
- 82.0,
- 110.0,
- 93.0,
- 105.0,
- 124.0,
- 126.0,
- 115.0,
- 125.0,
- 93.0,
- 99.0,
- 96.0,
- 103.0,
- 86.0,
- 86.0,
- 130.0,
- 97.0,
- 121.0,
- 114.0,
- 113.0,
- 112.0,
- 100.0,
- 106.0,
- 113.0,
- 105.0,
- 106.0,
- 105.0,
- 110.0,
- 135.0,
- 116.0,
- 90.0,
- 95.0,
- 88.0,
- 131.0,
- 113.0,
- 116.0,
- 101.0,
- 109.0,
- 119.0,
- 87.0,
- 91.0,
- 107.0,
- 103.0,
- 99.0,
- 94.0,
- 116.0,
- 58.0,
- 90.0,
- 95.0,
- 106.0,
- 98.0,
- 120.0,
- 113.0,
- 106.0,
- 90.0,
- 122.0,
- 98.0,
- 92.0,
- 119.0,
- 122.0,
- 120.0,
- 110.0,
- 111.0,
- 106.0,
- 95.0,
- 120.0,
- 119.0,
- 115.0,
- 119.0,
- 106.0,
- 95.0,
- 108.0,
- 119.0,
- 116.0,
- 102.0,
- 121.0,
- 103.0,
- 124.0,
- 116.0,
- 99.0,
- 77.0,
- 107.0,
- 98.0,
- 81.0,
- 108.0,
- 106.0,
- 88.0,
- 122.0,
- 86.0,
- 89.0,
- 98.0,
- 114.0,
- 109.0,
- 122.0,
- 119.0,
- 110.0,
- 115.0,
- 91.0,
- 133.0,
- 114.0,
- 106.0,
- 114.0,
- 115.0,
- 122.0,
- 127.0,
- 91.0,
- 85.0,
- 101.0,
- 89.0,
- 97.0,
- 106.0,
- 120.0,
- 85.0,
- 98.0,
- 94.0,
- 109.0,
- 98.0,
- 106.0,
- 119.0,
- 97.0,
- 80.0,
- 95.0,
- 103.0,
- 107.0,
- 102.0,
- 134.0,
- 107.0,
- 117.0,
- 123.0,
- 102.0,
- 105.0,
- 97.0,
- 108.0,
- 134.0,
- 113.0,
- 93.0,
- 118.0,
- 101.0,
- 94.0,
- 123.0,
- 109.0,
- 104.0,
- 120.0,
- 109.0,
- 136.0,
- 102.0,
- 98.0,
- 77.0,
- 105.0,
- 120.0,
- 94.0,
- 106.0,
- 109.0,
- 89.0,
- 103.0,
- 137.0,
- 111.0,
- 96.0,
- 125.0,
- 138.0,
- 99.0,
- 142.0,
- 107.0,
- 107.0,
- 95.0,
- 124.0,
- 117.0,
- 142.0,
- 123.0,
- 124.0,
- 97.0,
- 110.0,
- 91.0,
- 131.0,
- 115.0,
- 106.0,
- 102.0,
- 120.0,
- 114.0,
- 117.0,
- 102.0,
- 116.0,
- 126.0,
- 105.0,
- 100.0,
- 107.0,
- 114.0,
- 118.0,
- 101.0,
- 109.0,
- 112.0,
- 99.0,
- 97.0,
- 114.0,
- 107.0,
- 127.0,
- 119.0,
- 121.0,
- 107.0,
- 120.0,
- 119.0,
- 102.0,
- 110.0,
- 116.0,
- 107.0,
- 117.0,
- 117.0,
- 121.0,
- 130.0,
- 128.0,
- 102.0,
- 126.0,
- 115.0,
- 114.0,
- 119.0,
- 128.0,
- 112.0,
- 98.0,
- 141.0,
- 109.0,
- 103.0,
- 106.0,
- 114.0,
- 122.0,
- 121.0
- ]
- },
- "iteration-time": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 13.16275,
- 0.4518,
- 0.44557,
- 0.45576,
- 0.45722,
- 0.44122,
- 0.44896,
- 0.44797,
- 0.45127,
- 0.44355,
- 0.44203,
- 0.44107,
- 0.44753,
- 0.44562,
- 0.44125,
- 0.44515,
- 0.67142,
- 0.44532,
- 0.46026,
- 0.44572,
- 0.44982,
- 0.44886,
- 0.44864,
- 0.44877,
- 0.44648,
- 0.4424,
- 0.44248,
- 0.44394,
- 0.44792,
- 0.44757,
- 0.45034,
- 0.44906,
- 0.458,
- 0.4431,
- 0.44402,
- 0.44226,
- 0.44968,
- 0.44244,
- 0.43928,
- 0.45458,
- 0.44414,
- 0.44266,
- 0.44257,
- 0.44323,
- 0.44374,
- 0.44748,
- 0.44303,
- 0.4441,
- 0.44285,
- 0.44733,
- 0.44378,
- 0.44354,
- 0.4399,
- 0.44097,
- 0.44394,
- 0.4429,
- 0.44266,
- 0.44164,
- 0.44233,
- 0.44097,
- 0.43971,
- 0.6223,
- 0.44021,
- 0.43751,
- 0.44529,
- 0.43738,
- 0.43829,
- 0.4386,
- 0.43992,
- 0.43998,
- 0.43889,
- 0.43767,
- 0.43834,
- 0.43759,
- 0.43777,
- 0.43857,
- 0.43711,
- 0.43941,
- 0.43784,
- 0.44083,
- 0.43811,
- 0.43937,
- 0.44198,
- 0.44123,
- 0.44152,
- 0.44023,
- 0.44153,
- 0.44214,
- 0.4395,
- 0.44473,
- 0.44356,
- 0.44158,
- 0.44242,
- 0.4424,
- 0.4404,
- 0.44416,
- 0.44469,
- 0.44324,
- 0.44225,
- 0.43921,
- 0.44046,
- 0.61905,
- 0.4415,
- 0.44022,
- 0.44161,
- 0.44571,
- 0.44336,
- 0.44323,
- 0.4464,
- 0.45359,
- 0.44064,
- 0.44296,
- 0.44293,
- 0.44022,
- 0.44093,
- 0.44096,
- 0.44293,
- 0.44476,
- 0.44293,
- 0.44493,
- 0.44441,
- 0.44481,
- 0.44206,
- 0.44245,
- 0.44282,
- 0.44194,
- 0.4442,
- 0.44265,
- 0.44176,
- 0.44137,
- 0.44235,
- 0.4394,
- 0.43896,
- 0.44163,
- 0.44138,
- 0.44107,
- 0.44214,
- 0.44424,
- 0.44448,
- 0.44264,
- 0.4416,
- 0.44032,
- 0.43985,
- 0.43852,
- 0.4412,
- 0.43765,
- 0.43824,
- 0.43891,
- 0.44181,
- 0.43809,
- 0.78158,
- 0.62586,
- 0.44007,
- 0.44167,
- 0.44119,
- 0.44323,
- 0.44293,
- 0.44258,
- 0.44257,
- 0.44383,
- 0.44055,
- 0.44274,
- 0.44198,
- 0.44248,
- 0.44257,
- 0.44076,
- 0.44018,
- 0.44336,
- 0.44473,
- 0.44424,
- 0.4397,
- 0.44067,
- 0.44098,
- 0.43695,
- 0.43881,
- 0.43582,
- 0.43518,
- 0.43505,
- 0.43754,
- 0.43588,
- 0.43662,
- 0.43699,
- 0.43687,
- 0.43919,
- 0.43661,
- 0.43689,
- 0.43479,
- 0.43653,
- 0.43585,
- 0.43678,
- 0.43698,
- 0.43872,
- 0.43736,
- 0.43695,
- 0.43692,
- 0.6126,
- 0.43542,
- 0.60845,
- 0.43535,
- 0.43582,
- 0.44167,
- 0.44049,
- 0.44041,
- 0.43948,
- 0.43837,
- 0.4451,
- 0.44758,
- 0.43922,
- 0.43796,
- 0.43914,
- 0.43744,
- 0.43686,
- 0.43836,
- 0.43649,
- 0.43807,
- 0.43912,
- 0.43758,
- 0.43832,
- 0.43758,
- 0.43794,
- 0.43713,
- 0.436,
- 0.43768,
- 0.47048,
- 0.43956,
- 0.4375,
- 0.43873,
- 0.4394,
- 0.43764,
- 0.43801,
- 0.44127,
- 0.44216,
- 0.4391,
- 0.43815,
- 0.43822,
- 0.43702,
- 0.43794,
- 0.61667,
- 0.44311,
- 0.43731,
- 0.43777,
- 0.43921,
- 0.43875,
- 0.44131,
- 0.44003,
- 0.4415,
- 0.43932,
- 0.43866,
- 0.43727,
- 0.43777,
- 0.43796,
- 0.43822,
- 0.44556,
- 0.44349,
- 0.4382,
- 0.44057,
- 0.44268,
- 0.4425,
- 0.43738,
- 0.43736,
- 0.43793,
- 0.43862,
- 0.43893,
- 0.43846,
- 0.43905,
- 0.43842,
- 0.43863,
- 0.43678,
- 0.43877,
- 0.43998,
- 0.43905,
- 0.43837,
- 0.44205,
- 0.43732,
- 0.43694,
- 0.43718,
- 0.43541,
- 0.44457,
- 0.469,
- 0.44256,
- 0.44183,
- 0.44406,
- 0.44573,
- 0.44202,
- 0.44479,
- 0.43977,
- 0.45002,
- 0.45362,
- 0.45377,
- 0.45436,
- 0.44253,
- 0.44457,
- 0.45383,
- 0.45596,
- 0.45261,
- 0.4516,
- 0.45161,
- 0.45303,
- 0.43464,
- 0.43652,
- 0.44758,
- 0.44901,
- 0.44729,
- 0.45325,
- 0.44638,
- 0.43862,
- 0.4353,
- 0.44012,
- 0.44375,
- 0.44691,
- 0.44508,
- 0.44783,
- 0.44662,
- 0.45161,
- 0.43977,
- 0.43968,
- 0.4409,
- 0.44272,
- 0.44165,
- 0.4453,
- 0.4461,
- 0.44635,
- 0.44321,
- 0.43877,
- 0.44548,
- 0.44124,
- 0.44386,
- 0.44185,
- 0.43882,
- 0.43874,
- 0.61671,
- 0.44295,
- 0.4451,
- 0.43869,
- 0.44223,
- 0.43833,
- 0.44469,
- 0.44476,
- 0.44294,
- 0.44362,
- 0.4417,
- 0.44045,
- 0.44113,
- 0.44174,
- 0.4438,
- 0.44235,
- 0.44348,
- 0.44315,
- 0.44249,
- 0.43979,
- 0.43901,
- 0.43734,
- 0.43836,
- 0.43776,
- 0.44259,
- 0.43817,
- 0.4403,
- 0.43646,
- 0.43628,
- 0.43735,
- 0.43576,
- 0.43537,
- 0.43519,
- 0.43657,
- 0.4395,
- 0.44075,
- 0.4379,
- 0.43864,
- 0.43931,
- 0.43933,
- 0.43914,
- 0.43998,
- 0.60863,
- 0.44024,
- 0.44234,
- 0.61444,
- 0.4406,
- 0.44103,
- 0.44089,
- 0.43894,
- 0.43643,
- 0.43311,
- 0.43426,
- 0.43504,
- 0.43528,
- 0.43329,
- 0.43387,
- 0.43408,
- 0.43608,
- 0.43761,
- 0.43604,
- 0.43664,
- 0.44061,
- 0.43728,
- 0.4362,
- 0.43852,
- 0.4395,
- 0.44056,
- 0.43729,
- 0.4387
- ]
- }
-}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_fp8_no_model_parallel/golden_values_lts_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_fp8_no_model_parallel/golden_values_lts_dgx_a100.json
deleted file mode 100644
index 01ae9fa603f..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp1_fp8_no_model_parallel/golden_values_lts_dgx_a100.json
+++ /dev/null
@@ -1,1223 +0,0 @@
-{
- "lm loss": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 10.88328,
- 10.90257,
- 10.88663,
- 10.83293,
- 10.67628,
- 10.64935,
- 10.43401,
- 10.15135,
- 9.93919,
- 9.84145,
- 9.5886,
- 9.85443,
- 9.88471,
- 9.6295,
- 9.78811,
- 9.51135,
- 9.45833,
- 9.64922,
- 9.3861,
- 9.33215,
- 9.24219,
- 9.14551,
- 9.17554,
- 8.99539,
- 9.18938,
- 9.05997,
- 9.15548,
- 9.16492,
- 9.29764,
- 8.98435,
- 8.92898,
- 9.04372,
- 9.04285,
- 8.65475,
- 8.71696,
- 8.75327,
- 8.68353,
- 8.73425,
- 8.65866,
- 8.7648,
- 8.66088,
- 8.84978,
- 8.83233,
- 8.49954,
- 8.38931,
- 8.43182,
- 8.49351,
- 8.38471,
- 8.43278,
- 8.57978,
- 8.36719,
- 8.19226,
- 8.22606,
- 8.22217,
- 8.26751,
- 7.91344,
- 8.09563,
- 7.89094,
- 8.24624,
- 8.23026,
- 8.00472,
- 7.96522,
- 7.91788,
- 7.7397,
- 7.73956,
- 7.64272,
- 7.5154,
- 7.90678,
- 7.6983,
- 7.45188,
- 7.7404,
- 7.76772,
- 7.54129,
- 7.29853,
- 7.45244,
- 7.33556,
- 7.46205,
- 7.2239,
- 7.63657,
- 7.27934,
- 7.35205,
- 7.21344,
- 7.2184,
- 7.42314,
- 7.17762,
- 7.28364,
- 7.00217,
- 7.00609,
- 7.04135,
- 7.14062,
- 6.82539,
- 6.98709,
- 7.08964,
- 7.00127,
- 6.87463,
- 6.75505,
- 6.98955,
- 7.05522,
- 6.70122,
- 6.57704,
- 6.7241,
- 6.73883,
- 6.73084,
- 6.73626,
- 6.65691,
- 6.40601,
- 6.6385,
- 6.61945,
- 6.44599,
- 6.62978,
- 6.7427,
- 6.60925,
- 6.72472,
- 6.69413,
- 6.62417,
- 6.50597,
- 6.59855,
- 6.40573,
- 6.66284,
- 6.24739,
- 6.24997,
- 6.30097,
- 6.388,
- 6.34802,
- 6.45034,
- 6.28816,
- 6.33919,
- 6.23671,
- 6.20179,
- 6.39922,
- 6.32737,
- 6.32553,
- 6.17013,
- 6.16365,
- 6.24434,
- 6.39029,
- 6.20574,
- 6.15527,
- 6.18471,
- 6.1222,
- 6.07029,
- 6.07979,
- 6.26575,
- 6.41726,
- 6.26706,
- 6.30954,
- 6.10595,
- 6.18734,
- 6.00692,
- 6.03492,
- 5.96423,
- 6.2551,
- 6.19408,
- 5.97048,
- 5.78933,
- 6.12844,
- 5.85507,
- 6.10685,
- 5.79224,
- 6.16384,
- 6.15379,
- 6.09028,
- 5.93344,
- 6.11618,
- 5.94755,
- 6.19909,
- 5.89849,
- 5.79479,
- 5.78215,
- 5.68723,
- 6.01666,
- 5.99873,
- 6.06846,
- 5.89225,
- 6.04309,
- 5.97331,
- 5.99586,
- 5.98785,
- 5.9482,
- 5.83937,
- 5.9539,
- 5.61502,
- 5.699,
- 5.88897,
- 5.84054,
- 5.86112,
- 5.75936,
- 5.8375,
- 5.72064,
- 5.55646,
- 5.71958,
- 5.62394,
- 5.82954,
- 5.59832,
- 5.70553,
- 5.71488,
- 5.89528,
- 5.63976,
- 5.84631,
- 5.73496,
- 5.86743,
- 5.32607,
- 5.8903,
- 5.86889,
- 5.85006,
- 5.40738,
- 5.40549,
- 5.61986,
- 5.59188,
- 5.48192,
- 5.57349,
- 5.66996,
- 5.47178,
- 5.74017,
- 5.5091,
- 5.5953,
- 5.62066,
- 5.61598,
- 5.50824,
- 5.60964,
- 5.66876,
- 5.67788,
- 5.58421,
- 5.65722,
- 5.37016,
- 5.67677,
- 5.62454,
- 5.41705,
- 5.58431,
- 5.62542,
- 5.551,
- 5.33804,
- 5.5352,
- 5.48161,
- 5.4792,
- 5.37255,
- 5.55166,
- 5.59953,
- 5.38742,
- 5.52882,
- 5.48399,
- 5.32717,
- 5.50198,
- 5.40392,
- 5.43702,
- 5.3136,
- 5.06117,
- 5.47389,
- 5.56557,
- 5.70853,
- 5.41216,
- 5.59341,
- 5.63164,
- 5.23055,
- 5.27033,
- 5.38841,
- 5.39231,
- 5.32637,
- 5.49634,
- 5.17964,
- 5.29868,
- 5.24799,
- 5.37548,
- 5.25701,
- 5.44548,
- 5.5335,
- 5.31052,
- 5.43683,
- 5.3353,
- 5.07101,
- 5.31399,
- 5.25159,
- 5.30391,
- 5.10938,
- 5.27301,
- 5.26584,
- 5.47183,
- 5.15833,
- 5.26797,
- 5.2042,
- 5.35548,
- 4.98018,
- 4.91368,
- 5.31818,
- 5.38695,
- 5.2229,
- 5.31671,
- 5.10441,
- 5.157,
- 5.26026,
- 5.0625,
- 5.25998,
- 5.07253,
- 5.3394,
- 5.24357,
- 5.1487,
- 5.23894,
- 5.03446,
- 5.31002,
- 5.04729,
- 5.02048,
- 5.13726,
- 5.10974,
- 5.26597,
- 5.14767,
- 5.27512,
- 5.09179,
- 5.09166,
- 5.24809,
- 5.31963,
- 5.24883,
- 5.18566,
- 5.13848,
- 5.28494,
- 4.94428,
- 5.20203,
- 5.08707,
- 5.2953,
- 5.17219,
- 5.18368,
- 5.10813,
- 4.97968,
- 4.98627,
- 5.21879,
- 5.30748,
- 5.09449,
- 5.05013,
- 4.90918,
- 5.1167,
- 5.11153,
- 4.92276,
- 5.33502,
- 5.01879,
- 5.09746,
- 5.15679,
- 5.00133,
- 5.05827,
- 5.0642,
- 4.99125,
- 5.07529,
- 5.15683,
- 4.97325,
- 5.18006,
- 4.92846,
- 4.91522,
- 5.06502,
- 4.98714,
- 4.90587,
- 4.76968,
- 4.93606,
- 5.10905,
- 5.01253,
- 5.01189,
- 5.32285,
- 4.95232,
- 4.98602,
- 5.03643,
- 4.79932,
- 4.73082,
- 4.98974,
- 5.03227,
- 4.869,
- 4.94652,
- 5.03569,
- 5.01991,
- 4.80827,
- 4.8843,
- 4.90063,
- 4.82504,
- 4.74012,
- 5.00614,
- 4.74848,
- 5.20476,
- 4.78042,
- 4.98499,
- 4.73025,
- 4.7785,
- 4.81295,
- 4.64494,
- 4.65243,
- 4.83669,
- 4.8024,
- 4.79669,
- 4.91921,
- 4.87673,
- 4.91715,
- 4.76372,
- 4.87698,
- 4.72822,
- 4.90557,
- 4.95497,
- 4.8678,
- 4.70245,
- 4.77753,
- 4.89528,
- 4.70375,
- 4.8549,
- 4.68367,
- 4.68022,
- 4.64383
- ]
- },
- "num-zeros": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 73.0,
- 74.0,
- 89.0,
- 69.0,
- 80.0,
- 81.0,
- 114.0,
- 120.0,
- 136.0,
- 153.0,
- 132.0,
- 143.0,
- 138.0,
- 166.0,
- 183.0,
- 152.0,
- 149.0,
- 170.0,
- 167.0,
- 164.0,
- 173.0,
- 182.0,
- 184.0,
- 196.0,
- 177.0,
- 176.0,
- 223.0,
- 188.0,
- 191.0,
- 163.0,
- 168.0,
- 143.0,
- 156.0,
- 162.0,
- 162.0,
- 141.0,
- 176.0,
- 203.0,
- 169.0,
- 205.0,
- 142.0,
- 165.0,
- 143.0,
- 172.0,
- 177.0,
- 173.0,
- 201.0,
- 208.0,
- 179.0,
- 206.0,
- 233.0,
- 183.0,
- 204.0,
- 136.0,
- 161.0,
- 206.0,
- 173.0,
- 168.0,
- 219.0,
- 264.0,
- 191.0,
- 180.0,
- 185.0,
- 177.0,
- 187.0,
- 250.0,
- 225.0,
- 175.0,
- 235.0,
- 183.0,
- 228.0,
- 253.0,
- 184.0,
- 214.0,
- 206.0,
- 216.0,
- 273.0,
- 223.0,
- 279.0,
- 243.0,
- 277.0,
- 232.0,
- 223.0,
- 213.0,
- 232.0,
- 183.0,
- 193.0,
- 226.0,
- 226.0,
- 198.0,
- 212.0,
- 211.0,
- 229.0,
- 210.0,
- 220.0,
- 188.0,
- 216.0,
- 189.0,
- 182.0,
- 190.0,
- 153.0,
- 170.0,
- 180.0,
- 173.0,
- 139.0,
- 137.0,
- 158.0,
- 153.0,
- 131.0,
- 185.0,
- 187.0,
- 148.0,
- 178.0,
- 153.0,
- 149.0,
- 126.0,
- 169.0,
- 112.0,
- 166.0,
- 167.0,
- 188.0,
- 146.0,
- 137.0,
- 138.0,
- 126.0,
- 118.0,
- 127.0,
- 139.0,
- 133.0,
- 142.0,
- 143.0,
- 105.0,
- 131.0,
- 128.0,
- 154.0,
- 108.0,
- 163.0,
- 113.0,
- 113.0,
- 103.0,
- 110.0,
- 113.0,
- 98.0,
- 122.0,
- 156.0,
- 119.0,
- 129.0,
- 148.0,
- 133.0,
- 119.0,
- 97.0,
- 97.0,
- 129.0,
- 129.0,
- 120.0,
- 101.0,
- 108.0,
- 146.0,
- 113.0,
- 136.0,
- 90.0,
- 121.0,
- 130.0,
- 125.0,
- 87.0,
- 103.0,
- 105.0,
- 130.0,
- 102.0,
- 122.0,
- 139.0,
- 106.0,
- 108.0,
- 96.0,
- 132.0,
- 98.0,
- 115.0,
- 135.0,
- 116.0,
- 119.0,
- 102.0,
- 126.0,
- 146.0,
- 111.0,
- 127.0,
- 135.0,
- 126.0,
- 106.0,
- 114.0,
- 118.0,
- 113.0,
- 87.0,
- 126.0,
- 87.0,
- 113.0,
- 84.0,
- 126.0,
- 131.0,
- 121.0,
- 93.0,
- 121.0,
- 116.0,
- 112.0,
- 102.0,
- 112.0,
- 111.0,
- 107.0,
- 80.0,
- 114.0,
- 100.0,
- 111.0,
- 99.0,
- 112.0,
- 127.0,
- 109.0,
- 83.0,
- 108.0,
- 118.0,
- 109.0,
- 102.0,
- 104.0,
- 140.0,
- 108.0,
- 115.0,
- 110.0,
- 112.0,
- 112.0,
- 130.0,
- 89.0,
- 113.0,
- 129.0,
- 91.0,
- 92.0,
- 95.0,
- 99.0,
- 97.0,
- 105.0,
- 93.0,
- 126.0,
- 78.0,
- 105.0,
- 115.0,
- 98.0,
- 104.0,
- 111.0,
- 95.0,
- 110.0,
- 109.0,
- 107.0,
- 123.0,
- 111.0,
- 95.0,
- 130.0,
- 110.0,
- 107.0,
- 96.0,
- 96.0,
- 116.0,
- 101.0,
- 116.0,
- 94.0,
- 91.0,
- 126.0,
- 97.0,
- 96.0,
- 111.0,
- 131.0,
- 104.0,
- 112.0,
- 123.0,
- 108.0,
- 109.0,
- 96.0,
- 113.0,
- 116.0,
- 124.0,
- 91.0,
- 106.0,
- 108.0,
- 105.0,
- 97.0,
- 96.0,
- 96.0,
- 112.0,
- 115.0,
- 107.0,
- 120.0,
- 74.0,
- 108.0,
- 100.0,
- 98.0,
- 87.0,
- 115.0,
- 92.0,
- 94.0,
- 111.0,
- 109.0,
- 110.0,
- 111.0,
- 106.0,
- 133.0,
- 101.0,
- 110.0,
- 121.0,
- 98.0,
- 121.0,
- 89.0,
- 106.0,
- 111.0,
- 112.0,
- 116.0,
- 121.0,
- 92.0,
- 103.0,
- 115.0,
- 102.0,
- 102.0,
- 112.0,
- 109.0,
- 104.0,
- 131.0,
- 103.0,
- 99.0,
- 88.0,
- 131.0,
- 107.0,
- 105.0,
- 116.0,
- 111.0,
- 107.0,
- 109.0,
- 92.0,
- 114.0,
- 110.0,
- 131.0,
- 89.0,
- 131.0,
- 121.0,
- 107.0,
- 113.0,
- 122.0,
- 127.0,
- 118.0,
- 113.0,
- 123.0,
- 99.0,
- 113.0,
- 95.0,
- 139.0,
- 133.0,
- 115.0,
- 103.0,
- 135.0,
- 140.0,
- 121.0,
- 116.0,
- 112.0,
- 108.0,
- 109.0,
- 120.0,
- 121.0,
- 139.0,
- 131.0,
- 121.0,
- 99.0,
- 129.0,
- 128.0,
- 112.0,
- 111.0,
- 113.0,
- 108.0,
- 125.0,
- 104.0,
- 125.0,
- 108.0,
- 132.0,
- 113.0,
- 131.0,
- 106.0,
- 105.0,
- 127.0,
- 103.0,
- 112.0,
- 126.0,
- 138.0,
- 122.0,
- 93.0,
- 117.0,
- 108.0,
- 131.0,
- 109.0,
- 119.0,
- 116.0,
- 112.0,
- 117.0,
- 110.0,
- 113.0,
- 117.0,
- 123.0,
- 129.0
- ]
- },
- "iteration-time": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 23.51382,
- 0.5342,
- 0.5223,
- 0.51941,
- 0.51915,
- 0.51817,
- 0.52172,
- 0.52039,
- 1.09879,
- 0.51789,
- 0.53759,
- 0.51941,
- 0.51486,
- 0.52394,
- 0.55868,
- 0.51619,
- 0.51826,
- 0.52072,
- 0.5198,
- 0.52296,
- 0.51791,
- 0.5175,
- 0.51812,
- 0.5142,
- 0.5147,
- 0.51385,
- 0.51787,
- 0.51458,
- 0.51261,
- 0.5194,
- 0.52308,
- 0.52639,
- 0.51319,
- 0.51711,
- 0.51849,
- 0.52014,
- 0.51598,
- 0.51621,
- 0.51753,
- 0.51941,
- 0.52438,
- 0.51841,
- 0.52176,
- 0.51987,
- 0.53153,
- 0.51724,
- 0.51824,
- 0.51341,
- 0.51785,
- 0.51604,
- 0.51734,
- 0.51496,
- 0.51307,
- 0.5287,
- 0.51388,
- 0.52769,
- 0.52046,
- 0.64635,
- 0.51593,
- 0.51775,
- 0.52198,
- 0.51714,
- 0.52393,
- 0.54984,
- 0.53386,
- 0.52318,
- 0.53634,
- 0.51966,
- 0.51953,
- 0.52496,
- 0.52002,
- 0.52185,
- 0.52079,
- 0.51802,
- 0.51931,
- 0.52004,
- 0.52012,
- 0.52253,
- 0.56766,
- 0.52277,
- 0.51891,
- 0.52244,
- 0.77939,
- 0.52675,
- 0.52298,
- 0.52169,
- 0.54141,
- 0.51931,
- 0.52167,
- 0.52006,
- 0.52623,
- 0.52106,
- 0.52152,
- 0.51996,
- 0.52123,
- 0.52206,
- 0.52184,
- 0.5221,
- 0.52339,
- 0.5196,
- 0.52264,
- 0.56193,
- 0.51873,
- 0.51733,
- 0.52052,
- 0.52492,
- 0.51965,
- 0.9034,
- 0.52445,
- 0.52113,
- 0.52863,
- 0.52107,
- 0.53136,
- 0.53476,
- 0.52098,
- 0.51906,
- 0.52323,
- 0.52001,
- 0.52096,
- 0.51763,
- 0.52786,
- 0.51903,
- 0.51973,
- 0.51829,
- 0.52265,
- 0.53926,
- 0.52064,
- 0.52148,
- 0.51749,
- 0.52273,
- 0.5196,
- 0.64915,
- 0.52709,
- 0.52382,
- 0.52177,
- 0.52138,
- 0.51704,
- 0.52011,
- 0.5235,
- 0.52066,
- 0.5224,
- 0.5223,
- 0.52268,
- 0.5202,
- 0.52043,
- 0.52099,
- 0.51814,
- 0.51833,
- 0.52443,
- 0.51872,
- 0.5226,
- 0.51996,
- 0.5247,
- 0.52329,
- 0.52019,
- 0.5266,
- 0.52223,
- 0.51963,
- 0.52204,
- 0.52169,
- 0.51858,
- 0.52132,
- 0.52141,
- 0.52373,
- 0.52127,
- 0.51793,
- 0.53003,
- 0.51861,
- 0.5225,
- 0.52182,
- 0.51846,
- 0.52272,
- 0.51992,
- 0.5237,
- 0.51685,
- 0.5209,
- 0.51901,
- 0.51631,
- 0.52358,
- 0.51629,
- 0.51963,
- 0.52068,
- 0.52867,
- 0.77752,
- 0.51921,
- 0.52025,
- 0.52279,
- 0.51598,
- 0.51949,
- 0.5185,
- 0.51599,
- 0.51831,
- 0.51714,
- 0.52096,
- 0.51531,
- 0.51772,
- 0.52075,
- 0.51527,
- 0.52285,
- 0.51419,
- 0.50962,
- 0.52299,
- 0.51823,
- 0.5203,
- 0.52057,
- 0.6447,
- 0.52388,
- 0.52098,
- 0.51617,
- 0.52062,
- 0.51981,
- 0.51981,
- 0.52216,
- 0.51694,
- 0.52074,
- 0.51891,
- 0.51763,
- 0.52161,
- 0.51535,
- 0.51916,
- 0.51601,
- 0.51886,
- 0.52694,
- 0.51739,
- 0.52451,
- 0.51812,
- 0.51682,
- 0.51817,
- 0.51679,
- 0.51488,
- 0.51481,
- 0.64785,
- 0.51418,
- 0.51997,
- 0.5195,
- 0.51253,
- 0.55243,
- 0.5133,
- 0.51914,
- 0.51872,
- 0.5117,
- 0.52929,
- 0.51388,
- 0.51762,
- 0.51507,
- 0.51904,
- 0.51979,
- 0.53219,
- 0.51427,
- 0.51907,
- 0.52006,
- 0.52028,
- 0.5158,
- 0.51359,
- 0.51582,
- 0.51882,
- 0.77271,
- 0.51317,
- 0.51263,
- 0.5189,
- 0.51467,
- 0.52205,
- 0.51684,
- 0.51957,
- 0.51527,
- 0.52485,
- 0.5329,
- 0.51602,
- 0.52031,
- 0.52254,
- 0.52213,
- 0.51582,
- 0.52159,
- 0.5168,
- 0.51972,
- 0.51313,
- 0.51875,
- 0.52647,
- 0.5295,
- 0.51793,
- 0.52266,
- 0.51713,
- 0.51426,
- 0.51708,
- 0.51628,
- 0.51718,
- 0.51698,
- 0.51493,
- 0.51322,
- 0.51916,
- 0.52679,
- 0.52173,
- 0.52442,
- 0.52011,
- 0.52081,
- 0.52103,
- 0.51937,
- 0.51853,
- 0.51432,
- 0.51971,
- 0.51314,
- 0.5217,
- 0.51693,
- 0.52016,
- 0.51948,
- 0.52146,
- 0.6434,
- 0.51345,
- 0.51714,
- 0.52033,
- 0.52025,
- 0.52005,
- 0.52095,
- 0.5176,
- 0.51568,
- 0.52952,
- 0.51954,
- 0.5179,
- 0.51824,
- 0.51634,
- 0.51696,
- 0.52052,
- 0.51605,
- 0.51911,
- 0.5166,
- 0.51723,
- 0.51968,
- 0.51804,
- 0.51805,
- 0.51944,
- 0.65632,
- 0.51506,
- 0.51541,
- 0.52912,
- 0.51706,
- 0.51487,
- 0.51405,
- 0.51718,
- 0.52008,
- 0.51812,
- 0.5149,
- 0.51969,
- 0.51459,
- 0.51746,
- 0.51199,
- 0.51806,
- 0.51521,
- 0.51985,
- 0.52113,
- 0.5151,
- 0.52832,
- 0.51726,
- 0.51874,
- 0.52492,
- 0.52264,
- 0.52255,
- 0.52119,
- 0.52146,
- 0.52374,
- 0.52585,
- 0.52001,
- 0.52957,
- 0.52158,
- 0.52306,
- 0.53198,
- 0.51875,
- 0.52172,
- 0.52141,
- 0.52506,
- 0.52701,
- 0.52335,
- 0.52579,
- 0.52561,
- 0.52567,
- 0.52299,
- 0.52173,
- 0.52358,
- 0.52268,
- 0.5225,
- 0.53389,
- 0.79026,
- 0.52767,
- 0.52103,
- 0.53508,
- 0.52025,
- 0.51955,
- 0.52579,
- 0.52352,
- 0.51858,
- 0.51765,
- 0.52118,
- 0.52567,
- 0.52257,
- 0.52435,
- 0.51912,
- 0.538,
- 0.52183,
- 0.52136,
- 0.51694,
- 0.51741
- ]
- }
-}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp2_fp8_pp/golden_values_dev_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp2_fp8_pp/golden_values_dev_dgx_a100.json
deleted file mode 100644
index 0b03b850b4b..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp2_fp8_pp/golden_values_dev_dgx_a100.json
+++ /dev/null
@@ -1 +0,0 @@
-{"forward-backward-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [21.16929, 0.69842, 0.69865, 0.68092, 0.68114, 0.68076, 0.68553, 0.6784, 0.70132, 0.68656, 0.68867, 0.69143, 0.69023, 0.68774, 0.70094, 0.68596, 0.68549, 0.6811, 0.68151, 0.67743, 0.6818, 0.67512, 0.68645, 0.67903, 0.68158, 0.68543, 0.68715, 0.68897, 0.70747, 0.68759, 0.68732, 0.68723, 0.69033, 0.68094, 0.68856, 0.6856, 0.69221, 0.68087, 0.69125, 0.68605, 0.69475, 0.68504, 0.6893, 0.69096, 0.69541, 0.70004, 0.69576, 0.69211, 0.70539, 0.69068, 0.68902, 0.69335, 0.68369, 0.68436, 0.68239, 0.68834, 0.6958, 0.68962, 0.68485, 0.69578, 0.6843, 0.68984, 0.69245, 0.68747, 0.68675, 0.69129, 0.68873, 0.68069, 0.69138, 0.69036, 0.68756, 0.68003, 0.68118, 0.68219, 0.68967, 0.68462, 0.68795, 0.68699, 0.6881, 0.6895, 0.6908, 0.68981, 0.68371, 0.68631, 0.68376, 0.81573, 0.69039, 0.69127, 0.69453, 0.69743, 0.69357, 0.68918, 0.68915, 0.68957, 0.69407, 0.68945, 0.69186, 0.68603, 0.68977, 0.70044, 0.69469, 0.69533, 0.69415, 0.69884, 0.69538, 0.69372, 0.69623, 0.69454, 0.6948, 0.69135, 0.69206, 0.68673, 0.68936, 0.68303, 0.68538, 0.68582, 0.69851, 0.70083, 0.69592, 0.69452, 0.69303, 0.69071, 0.70246, 0.6973, 0.69795, 0.69114, 0.69795, 0.69698, 0.69429, 0.69158, 0.69376, 0.69794, 0.69244, 0.69205, 0.69394, 0.69551, 0.69657, 0.69487, 0.69462, 0.69874, 0.69622, 0.69596, 0.69702, 0.69605, 0.69381, 0.68895, 0.69096, 0.69099, 0.69224, 0.68822, 0.69238, 0.68894, 0.69956, 0.69462, 0.69596, 0.69826, 0.69791, 0.69829, 0.69528, 0.69581, 0.69246, 0.69712, 0.69164, 0.69373, 0.69112, 0.69522, 0.68973, 0.69375, 0.69191, 0.69554, 0.69908, 0.69725, 0.69744, 0.69566, 0.69832, 0.69791, 0.69806, 0.69817, 0.69569, 0.69697, 0.69849, 0.69511, 0.69491, 0.69873, 0.69972, 0.70371, 0.69973, 0.70041, 0.69955, 0.69404, 0.69642, 0.69525, 0.70125, 0.69189, 0.70768, 0.71527, 0.70077, 0.69532, 0.6961, 0.7031, 0.67909, 0.68793, 0.70461, 0.69523, 0.69673, 0.70017, 0.69796, 0.69461, 0.70307, 0.69829, 0.69545, 0.69288, 0.75214, 0.70015, 0.70134, 0.69495, 0.70155, 0.70094, 0.69651, 0.69772, 0.69954, 0.69592, 0.6977, 0.69059, 0.69677, 0.69829, 0.69779, 0.69192, 0.69617, 0.69978, 0.68964, 0.69432, 0.69761, 0.69629, 0.69975, 0.69141, 0.69977, 0.69704, 0.70403, 0.68958, 0.69117, 0.68705, 0.69675, 0.68817, 0.69828, 0.69189, 0.69446, 0.6924, 0.69063, 0.691, 0.69163, 0.69402, 0.69605, 0.69383, 0.69327, 0.69636, 0.69175, 0.69468, 0.69281, 0.70044, 0.70067, 0.7016, 0.69557, 0.69614, 0.69761, 0.69793, 0.69322, 0.69689, 0.70043, 0.69446, 0.69543, 0.69346, 0.69441, 0.68931, 0.69592, 0.6914, 0.6929, 0.69539, 0.69954, 0.69999, 0.69447, 0.69508, 0.69638, 0.69699, 0.69614, 0.69655, 0.6957, 0.69348, 0.698, 0.70136, 0.69861, 0.69224, 0.69369, 0.69763, 0.69759, 0.69166, 0.69413, 0.69071, 0.69463, 0.69072, 0.69754, 0.69663, 0.69249, 0.69603, 0.80113, 0.69556, 0.69325, 0.69439, 0.69712, 0.69274, 0.69473, 0.68837, 0.69493, 0.69602, 0.69314, 0.69884, 0.70264, 0.70625, 0.69696, 0.69541, 0.69344, 0.70656, 0.69704, 0.69417, 0.70121, 0.69558, 0.7002, 0.815, 0.69817, 0.69499, 0.70038, 0.70281, 0.70226, 0.69884, 0.69724, 0.69581, 0.69287, 0.69618, 0.71318, 0.69943, 0.70407, 0.69607, 0.69718, 0.68881, 0.69211, 0.69118, 0.69873, 0.69888, 0.70284, 0.6967, 0.70012, 0.69679, 0.69994, 0.69768, 0.7015, 0.70388, 0.69342, 0.69641, 0.70208, 0.6909, 0.69959, 0.69723, 0.69969, 0.70232, 0.69828, 0.697, 0.69714, 0.69676, 0.69506, 0.69683, 0.69519, 0.68973, 0.70075, 0.69457, 0.69842, 0.69584, 0.69872, 0.69358, 0.69875, 0.69346, 0.70004, 0.69971, 0.70151, 0.70016, 0.70414, 0.70754, 0.70082, 0.69723, 0.70207, 0.70466, 0.70276, 0.69824, 0.70085, 0.70049, 0.70134, 0.70037, 0.705, 0.70761, 0.70114, 0.69824]}, "forward-compute-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [4.92979, 0.36862, 0.36896, 0.35994, 0.3634, 0.36131, 0.37528, 0.36745, 0.39414, 0.37596, 0.3798, 0.38001, 0.38263, 0.37794, 0.39251, 0.3769, 0.37612, 0.3675, 0.37072, 0.36701, 0.37163, 0.36679, 0.37704, 0.36833, 0.37308, 0.37264, 0.37893, 0.37759, 0.39953, 0.37377, 0.37903, 0.37511, 0.37891, 0.37243, 0.38146, 0.37534, 0.38244, 0.37164, 0.38228, 0.37646, 0.38605, 0.37539, 0.38035, 0.38244, 0.38642, 0.3893, 0.38511, 0.3827, 0.39156, 0.3782, 0.37799, 0.38401, 0.37401, 0.37169, 0.37072, 0.37641, 0.38295, 0.38051, 0.37444, 0.38482, 0.37469, 0.38129, 0.38054, 0.37571, 0.37578, 0.37992, 0.37782, 0.37386, 0.3813, 0.38374, 0.3775, 0.37428, 0.37254, 0.37234, 0.37719, 0.37627, 0.37853, 0.37526, 0.38087, 0.38099, 0.38071, 0.38191, 0.37329, 0.3773, 0.3734, 0.5018, 0.38253, 0.38164, 0.38606, 0.38733, 0.38592, 0.38071, 0.37964, 0.37907, 0.38532, 0.37904, 0.38222, 0.37656, 0.38031, 0.38646, 0.38574, 0.38602, 0.37899, 0.38893, 0.38764, 0.38446, 0.38488, 0.38659, 0.38646, 0.38256, 0.38198, 0.37894, 0.38195, 0.37524, 0.37462, 0.37752, 0.38757, 0.39104, 0.38931, 0.38235, 0.38351, 0.38268, 0.39375, 0.3868, 0.38798, 0.38182, 0.39008, 0.38803, 0.38668, 0.38465, 0.38639, 0.38737, 0.38331, 0.37911, 0.38492, 0.38652, 0.38697, 0.38654, 0.38596, 0.39074, 0.38492, 0.38717, 0.38731, 0.38942, 0.386, 0.38148, 0.38444, 0.38374, 0.38416, 0.37792, 0.37748, 0.37957, 0.39104, 0.38581, 0.38566, 0.38678, 0.38966, 0.38882, 0.38683, 0.38264, 0.38507, 0.38712, 0.38306, 0.38289, 0.38103, 0.38363, 0.37743, 0.37875, 0.37956, 0.38316, 0.3891, 0.38796, 0.38596, 0.38565, 0.38554, 0.38556, 0.38505, 0.38092, 0.38387, 0.38393, 0.38859, 0.37887, 0.38497, 0.38623, 0.39043, 0.39246, 0.38914, 0.38962, 0.38901, 0.38336, 0.38644, 0.38387, 0.38958, 0.38133, 0.39066, 0.39461, 0.39129, 0.38237, 0.3862, 0.39181, 0.37212, 0.37912, 0.39389, 0.384, 0.38439, 0.38586, 0.38505, 0.38157, 0.38622, 0.38765, 0.38617, 0.38274, 0.44388, 0.39087, 0.3907, 0.38612, 0.38867, 0.39114, 0.38539, 0.38934, 0.38921, 0.38784, 0.38206, 0.38157, 0.38685, 0.39031, 0.38789, 0.38326, 0.38644, 0.38897, 0.38075, 0.3856, 0.38903, 0.3866, 0.38941, 0.37995, 0.38647, 0.388, 0.3933, 0.38074, 0.38111, 0.37964, 0.38635, 0.37942, 0.38546, 0.38117, 0.38291, 0.38281, 0.38246, 0.38276, 0.38171, 0.382, 0.3865, 0.37957, 0.3856, 0.38543, 0.38204, 0.38551, 0.38485, 0.39262, 0.39183, 0.38966, 0.38778, 0.38805, 0.3857, 0.3903, 0.38332, 0.38621, 0.38966, 0.38839, 0.3794, 0.38725, 0.38481, 0.38106, 0.38522, 0.3806, 0.38384, 0.38521, 0.38656, 0.39255, 0.38382, 0.38686, 0.38703, 0.38844, 0.38459, 0.38745, 0.38311, 0.38465, 0.38785, 0.39146, 0.38846, 0.38178, 0.38121, 0.38932, 0.38613, 0.38272, 0.38328, 0.38309, 0.38433, 0.38086, 0.38574, 0.38715, 0.38325, 0.38613, 0.4565, 0.38631, 0.38538, 0.38553, 0.38639, 0.38282, 0.38384, 0.37918, 0.38658, 0.38666, 0.38487, 0.39121, 0.3908, 0.39786, 0.3849, 0.38844, 0.38522, 0.394, 0.38769, 0.38524, 0.39367, 0.38775, 0.39338, 0.50382, 0.39159, 0.38743, 0.39102, 0.39523, 0.39356, 0.39205, 0.38578, 0.38801, 0.38304, 0.38678, 0.3987, 0.39171, 0.39597, 0.38708, 0.3908, 0.38146, 0.38222, 0.38202, 0.39012, 0.39068, 0.39269, 0.38682, 0.39099, 0.38924, 0.39219, 0.38971, 0.39066, 0.39542, 0.38474, 0.38829, 0.39181, 0.38288, 0.38918, 0.3886, 0.39087, 0.39457, 0.3877, 0.3877, 0.38997, 0.39047, 0.38458, 0.38887, 0.3875, 0.38266, 0.38907, 0.38748, 0.38772, 0.387, 0.38822, 0.38247, 0.39155, 0.38528, 0.39151, 0.39019, 0.39332, 0.39078, 0.3911, 0.39847, 0.3899, 0.39043, 0.39299, 0.39763, 0.39582, 0.39107, 0.39252, 0.39507, 0.39717, 0.3953, 0.40187, 0.40236, 0.39559, 0.39145]}, "backward-compute-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.8012, 0.29387, 0.2986, 0.28406, 0.28522, 0.28969, 0.29061, 0.28796, 0.29063, 0.28667, 0.29358, 0.29506, 0.2922, 0.2852, 0.28989, 0.28483, 0.28642, 0.28342, 0.28232, 0.28136, 0.28422, 0.28036, 0.28492, 0.28314, 0.281, 0.28245, 0.28442, 0.28445, 0.28814, 0.28551, 0.2857, 0.28486, 0.28705, 0.28407, 0.28536, 0.28489, 0.28989, 0.28255, 0.28845, 0.28647, 0.28944, 0.28337, 0.28838, 0.28849, 0.2897, 0.29269, 0.28788, 0.28852, 0.29394, 0.28953, 0.28786, 0.28768, 0.28428, 0.28563, 0.28458, 0.28775, 0.29324, 0.28892, 0.28616, 0.29034, 0.28456, 0.28682, 0.28841, 0.28729, 0.28425, 0.28778, 0.28741, 0.2839, 0.28832, 0.28804, 0.2861, 0.28333, 0.28362, 0.28274, 0.28476, 0.28495, 0.28365, 0.28409, 0.28405, 0.28625, 0.28429, 0.28647, 0.28314, 0.28367, 0.28409, 0.28622, 0.28505, 0.28438, 0.28134, 0.28462, 0.28536, 0.28398, 0.28654, 0.2869, 0.28809, 0.28601, 0.28761, 0.28425, 0.28676, 0.2862, 0.28997, 0.28934, 0.28731, 0.29342, 0.28795, 0.28707, 0.2867, 0.28661, 0.28811, 0.28616, 0.28592, 0.28428, 0.28508, 0.28396, 0.28659, 0.28265, 0.28697, 0.2894, 0.28687, 0.28772, 0.28913, 0.28621, 0.29195, 0.28847, 0.29125, 0.28862, 0.29011, 0.29025, 0.28931, 0.28814, 0.28955, 0.2908, 0.28871, 0.28801, 0.28793, 0.28964, 0.29306, 0.29007, 0.28963, 0.29251, 0.29069, 0.29194, 0.28984, 0.29084, 0.28995, 0.28615, 0.28778, 0.28795, 0.2882, 0.28737, 0.2876, 0.28691, 0.29135, 0.28807, 0.28993, 0.29202, 0.29116, 0.29034, 0.28863, 0.29346, 0.29111, 0.29416, 0.29263, 0.293, 0.29317, 0.2931, 0.28845, 0.288, 0.28664, 0.28885, 0.29051, 0.28976, 0.28937, 0.29252, 0.29727, 0.29583, 0.29602, 0.29658, 0.2931, 0.29603, 0.29621, 0.29395, 0.29259, 0.29542, 0.29412, 0.29939, 0.29634, 0.2902, 0.29267, 0.28896, 0.2887, 0.28951, 0.29196, 0.29075, 0.29727, 0.30019, 0.29535, 0.2896, 0.28882, 0.29318, 0.28687, 0.28581, 0.29387, 0.28979, 0.28852, 0.29025, 0.28988, 0.28996, 0.2906, 0.29127, 0.29091, 0.29027, 0.34386, 0.29092, 0.29145, 0.28886, 0.29332, 0.29127, 0.29064, 0.29054, 0.29117, 0.28886, 0.28689, 0.28524, 0.29113, 0.29077, 0.28956, 0.28788, 0.28875, 0.29066, 0.28696, 0.28828, 0.28986, 0.28975, 0.29179, 0.28765, 0.29054, 0.29018, 0.29236, 0.28513, 0.28796, 0.28625, 0.28988, 0.28486, 0.2901, 0.28715, 0.28807, 0.29103, 0.28636, 0.28731, 0.28709, 0.2878, 0.28863, 0.28922, 0.28858, 0.28861, 0.28721, 0.28911, 0.28891, 0.29009, 0.29181, 0.29183, 0.2921, 0.28906, 0.29246, 0.29132, 0.28922, 0.29183, 0.29154, 0.29016, 0.29033, 0.29069, 0.28941, 0.28627, 0.28999, 0.28617, 0.28792, 0.2909, 0.29099, 0.29284, 0.29202, 0.28998, 0.29186, 0.29297, 0.29177, 0.2896, 0.29112, 0.28824, 0.29124, 0.29518, 0.29288, 0.28876, 0.29026, 0.29318, 0.2932, 0.2894, 0.28931, 0.28848, 0.28934, 0.28881, 0.29144, 0.28798, 0.28986, 0.29212, 0.28958, 0.2898, 0.28969, 0.2893, 0.29213, 0.29, 0.29098, 0.29085, 0.29077, 0.29035, 0.29027, 0.29142, 0.29441, 0.29571, 0.29203, 0.29018, 0.29127, 0.29433, 0.29091, 0.28877, 0.29354, 0.29063, 0.29084, 0.29118, 0.29114, 0.29201, 0.29191, 0.29316, 0.29428, 0.29139, 0.29115, 0.29268, 0.28887, 0.29386, 0.29765, 0.29295, 0.29535, 0.29245, 0.29159, 0.28784, 0.29096, 0.28864, 0.2923, 0.29471, 0.29453, 0.2914, 0.29447, 0.29151, 0.29226, 0.29155, 0.29343, 0.29271, 0.28917, 0.29026, 0.2943, 0.28854, 0.29114, 0.29123, 0.2918, 0.29223, 0.29626, 0.29746, 0.29042, 0.29175, 0.29069, 0.29, 0.2892, 0.28808, 0.29535, 0.28977, 0.29205, 0.29056, 0.29189, 0.2899, 0.28981, 0.2895, 0.2929, 0.29123, 0.29288, 0.29252, 0.29518, 0.29616, 0.29356, 0.29361, 0.29532, 0.29564, 0.29465, 0.29223, 0.29483, 0.29279, 0.29075, 0.29144, 0.29105, 0.29375, 0.28857, 0.288]}, "batch-generator-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.30565, 0.00631, 0.0066, 0.00601, 0.00609, 0.00586, 0.00613, 0.00583, 0.00602, 0.00583, 0.00598, 0.00604, 0.00582, 0.00568, 0.00583, 0.0058, 0.00563, 0.00578, 0.00557, 0.0058, 0.00592, 0.00586, 0.0058, 0.00562, 0.00562, 0.00571, 0.00557, 0.00573, 0.00596, 0.00583, 0.00566, 0.00601, 0.00607, 0.00572, 0.00607, 0.00595, 0.00598, 0.00592, 0.00585, 0.00609, 0.00585, 0.0059, 0.00582, 0.00578, 0.00588, 0.00604, 0.00563, 0.00593, 0.00592, 0.00559, 0.00549, 0.00584, 0.00593, 0.00559, 0.00713, 0.00734, 0.00689, 0.00723, 0.00685, 0.00763, 0.00701, 0.00722, 0.0072, 0.00755, 0.00717, 0.00727, 0.00721, 0.00707, 0.00703, 0.00729, 0.00703, 0.00682, 0.00659, 0.00573, 0.00594, 0.00596, 0.00621, 0.00602, 0.00602, 0.00599, 0.00597, 0.00616, 0.0059, 0.00598, 0.00575, 0.00606, 0.00592, 0.00596, 0.00602, 0.00605, 0.00587, 0.00585, 0.00596, 0.00675, 0.00617, 0.0062, 0.00592, 0.00581, 0.00613, 0.00611, 0.00624, 0.00629, 0.00603, 0.00622, 0.00608, 0.00595, 0.00632, 0.00599, 0.00611, 0.00597, 0.00588, 0.00587, 0.0057, 0.00574, 0.00589, 0.00569, 0.00565, 0.00566, 0.0061, 0.00592, 0.00603, 0.00553, 0.00587, 0.00577, 0.00567, 0.00584, 0.00581, 0.00607, 0.00583, 0.00565, 0.00581, 0.0058, 0.00582, 0.00595, 0.0057, 0.00596, 0.00605, 0.00582, 0.00559, 0.00575, 0.00572, 0.00562, 0.00565, 0.00583, 0.00603, 0.00568, 0.00564, 0.00603, 0.00593, 0.0059, 0.00581, 0.0055, 0.00598, 0.00604, 0.00607, 0.00585, 0.00585, 0.00603, 0.00588, 0.00599, 0.00567, 0.00593, 0.00614, 0.0058, 0.00592, 0.00575, 0.00581, 0.00624, 0.00582, 0.00616, 0.00572, 0.00591, 0.0061, 0.00614, 0.00597, 0.00606, 0.00588, 0.00578, 0.00631, 0.00589, 0.00584, 0.00574, 0.00613, 0.00566, 0.0061, 0.00599, 0.0059, 0.00589, 0.00595, 0.00596, 0.00595, 0.00595, 0.00613, 0.00585, 0.00569, 0.00609, 0.00603, 0.00615, 0.00617, 0.00606, 0.06212, 0.00708, 0.00731, 0.00708, 0.00688, 0.0068, 0.00715, 0.00694, 0.00689, 0.00682, 0.00592, 0.00599, 0.00671, 0.00709, 0.00695, 0.00727, 0.00736, 0.00727, 0.00737, 0.00678, 0.00708, 0.00694, 0.00721, 0.00727, 0.00742, 0.00681, 0.00707, 0.00694, 0.00708, 0.00695, 0.00706, 0.00698, 0.00707, 0.0067, 0.00718, 0.00733, 0.00718, 0.00687, 0.00725, 0.00712, 0.00718, 0.00685, 0.00603, 0.00744, 0.00676, 0.00683, 0.00724, 0.00706, 0.00733, 0.00734, 0.00681, 0.00744, 0.00713, 0.00687, 0.00667, 0.00687, 0.00723, 0.00685, 0.00677, 0.00724, 0.00676, 0.00673, 0.0071, 0.00721, 0.00713, 0.00707, 0.00719, 0.00656, 0.00681, 0.0069, 0.00711, 0.00704, 0.00728, 0.00686, 0.00705, 0.00647, 0.00678, 0.00724, 0.00671, 0.00729, 0.00729, 0.00693, 0.00727, 0.00705, 0.0073, 0.0069, 0.00703, 0.00703, 0.00673, 0.00641, 0.00649, 0.0059, 0.00591, 0.00589, 0.00611, 0.00602, 0.00581, 0.00591, 0.006, 0.00615, 0.00591, 0.00611, 0.00606, 0.00605, 0.00645, 0.00595, 0.00594, 0.00596, 0.006, 0.00598, 0.00594, 0.00601, 0.00655, 0.00617, 0.00603, 0.0059, 0.00628, 0.00583, 0.00608, 0.00585, 0.00604, 0.00603, 0.00594, 0.00582, 0.00576, 0.00596, 0.00605, 0.00641, 0.00601, 0.00602, 0.0061, 0.00618, 0.00595, 0.00602, 0.00597, 0.00581, 0.00598, 0.00598, 0.00614, 0.00599, 0.00582, 0.00612, 0.00597, 0.00575, 0.00572, 0.00623, 0.00601, 0.00597, 0.00619, 0.00626, 0.00606, 0.00592, 0.00607, 0.00584, 0.00593, 0.00602, 0.00617, 0.00621, 0.00612, 0.00602, 0.00597, 0.00594, 0.00615, 0.00599, 0.00604, 0.00617, 0.00631, 0.00558, 0.00552, 0.0057, 0.00568, 0.00594, 0.00614, 0.00588, 0.006, 0.00605, 0.00607, 0.00624, 0.00636, 0.00582, 0.00604, 0.00595, 0.0061, 0.00615, 0.00599, 0.00599, 0.00621, 0.00604, 0.00599, 0.00599, 0.00589, 0.00621, 0.00584, 0.00586, 0.00593, 0.00614, 0.00623, 0.00591, 0.00632, 0.00604]}, "forward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [4.95821, 0.02363, 0.0227, 0.02332, 0.02256, 0.02319, 0.0228, 0.02261, 0.0228, 0.02242, 0.02284, 0.02259, 0.02245, 0.02309, 0.02332, 0.02185, 0.02227, 0.02241, 0.02251, 0.02246, 0.02257, 0.02259, 0.02212, 0.02254, 0.02299, 0.02339, 0.02258, 0.02339, 0.02279, 0.02234, 0.0221, 0.02333, 0.02239, 0.02203, 0.02184, 0.02211, 0.02224, 0.022, 0.0223, 0.02282, 0.02196, 0.02285, 0.02194, 0.02233, 0.02238, 0.0221, 0.02287, 0.02259, 0.02353, 0.02258, 0.02174, 0.02244, 0.02248, 0.02249, 0.02286, 0.02274, 0.02231, 0.02301, 0.02252, 0.02226, 0.02309, 0.0226, 0.02248, 0.02257, 0.02247, 0.02239, 0.02245, 0.02239, 0.02245, 0.02226, 0.02251, 0.02235, 0.02229, 0.02229, 0.02224, 0.02218, 0.02269, 0.02222, 0.02297, 0.0233, 0.02355, 0.02353, 0.02351, 0.02353, 0.0231, 0.02266, 0.02205, 0.02248, 0.02239, 0.02243, 0.02337, 0.02243, 0.02265, 0.02251, 0.0227, 0.02251, 0.02262, 0.0223, 0.02239, 0.02302, 0.02253, 0.0224, 0.02341, 0.02267, 0.02201, 0.02288, 0.02223, 0.02234, 0.02247, 0.02274, 0.0227, 0.02223, 0.02278, 0.02249, 0.02233, 0.02353, 0.02284, 0.02293, 0.02146, 0.02395, 0.02287, 0.02228, 0.02286, 0.02372, 0.02285, 0.02195, 0.02251, 0.02292, 0.02278, 0.02298, 0.02247, 0.02293, 0.02269, 0.02272, 0.02289, 0.0229, 0.0226, 0.02277, 0.02291, 0.02243, 0.02298, 0.02242, 0.02233, 0.02273, 0.0224, 0.02231, 0.02213, 0.02282, 0.02271, 0.02257, 0.02245, 0.02266, 0.02226, 0.02234, 0.02242, 0.02287, 0.02231, 0.02272, 0.02271, 0.02261, 0.02279, 0.02239, 0.02238, 0.02237, 0.02245, 0.02246, 0.023, 0.02279, 0.02277, 0.02299, 0.02326, 0.0223, 0.02341, 0.02259, 0.02308, 0.02252, 0.02308, 0.02263, 0.02343, 0.02234, 0.02287, 0.02253, 0.02261, 0.02291, 0.02258, 0.02266, 0.02272, 0.02323, 0.02251, 0.02228, 0.0226, 0.02245, 0.02282, 0.02319, 0.02275, 0.02246, 0.02327, 0.02259, 0.02253, 0.0224, 0.01758, 0.02244, 0.02255, 0.02222, 0.02295, 0.02246, 0.02236, 0.02202, 0.02348, 0.02237, 0.02232, 0.02231, 0.02262, 0.02284, 0.02278, 0.02292, 0.02249, 0.02264, 0.02288, 0.02264, 0.02232, 0.02331, 0.02235, 0.02266, 0.02272, 0.02229, 0.02285, 0.02276, 0.02283, 0.02355, 0.02243, 0.02224, 0.02272, 0.02285, 0.02224, 0.02355, 0.02275, 0.02246, 0.02254, 0.02335, 0.02272, 0.02208, 0.02249, 0.02229, 0.02237, 0.02251, 0.0228, 0.02259, 0.02238, 0.02269, 0.02278, 0.02234, 0.02262, 0.02237, 0.02265, 0.02234, 0.0239, 0.02204, 0.02217, 0.02222, 0.02262, 0.02231, 0.02208, 0.02252, 0.02267, 0.02293, 0.02253, 0.02228, 0.02237, 0.02246, 0.02294, 0.02246, 0.02182, 0.0225, 0.02229, 0.02265, 0.02222, 0.02222, 0.02264, 0.02241, 0.02246, 0.02208, 0.02243, 0.0227, 0.02237, 0.02231, 0.02228, 0.02312, 0.02228, 0.02236, 0.02245, 0.02239, 0.02316, 0.02216, 0.02227, 0.02241, 0.0226, 0.02206, 0.02266, 0.0223, 0.02225, 0.02286, 0.0223, 0.02201, 0.02235, 0.02378, 0.02224, 0.02326, 0.02229, 0.02293, 0.02211, 0.02198, 0.02233, 0.0224, 0.02212, 0.02248, 0.02253, 0.02253, 0.02258, 0.02203, 0.02237, 0.02274, 0.0222, 0.02237, 0.02238, 0.02242, 0.02229, 0.02263, 0.02196, 0.02243, 0.02239, 0.02243, 0.02221, 0.02264, 0.02264, 0.02249, 0.02235, 0.0226, 0.02289, 0.02232, 0.0227, 0.02252, 0.02225, 0.02254, 0.02223, 0.02268, 0.02244, 0.02292, 0.02284, 0.02271, 0.02275, 0.02258, 0.02303, 0.02263, 0.02297, 0.02275, 0.0227, 0.023, 0.02298, 0.02297, 0.02199, 0.02326, 0.02298, 0.02263, 0.02262, 0.02296, 0.02268, 0.0225, 0.02268, 0.02273, 0.02239, 0.02231, 0.02302, 0.02284, 0.02258, 0.02376, 0.02298, 0.02258, 0.02269, 0.02282, 0.02248, 0.02296, 0.02259, 0.02303, 0.02252, 0.02322, 0.02265, 0.0226, 0.02282, 0.0227, 0.02325, 0.02263, 0.02282, 0.02297, 0.02259, 0.02313, 0.02262, 0.02287, 0.02288, 0.02356]}, "forward-send-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [3.00337, 0.00014, 0.00014, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00015, 0.00015, 0.00014, 0.00015, 0.00014, 0.00014, 0.00016, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00015, 0.00015, 0.00014, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00015, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00013, 0.00014, 0.00014, 0.00014, 0.00014, 0.00013, 0.00017, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00015, 0.00013, 0.00014, 0.00014, 0.00013, 0.00014, 0.00015, 0.00014, 0.00013, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00013, 0.00013, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00013, 0.00014, 0.00014, 0.00014, 0.00013, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00015, 0.00014, 0.00013, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00013, 0.00014, 0.00013, 0.00013, 0.00014, 0.00014, 0.00014, 0.00013, 0.00015, 0.00013, 0.00014, 0.00014, 0.00015, 0.00014, 0.00013, 0.00014, 0.00015, 0.00014, 0.00016, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00013, 0.00014, 0.00014, 0.00013, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00013, 0.00013, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00013, 0.00014, 0.00014, 0.00014, 0.00013, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00013, 0.00015, 0.00015, 0.00014, 0.00016, 0.00013, 0.00016, 0.00014, 0.00015, 0.00013, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00016, 0.00014, 0.00014, 0.00015, 0.00014, 0.00015, 0.00014, 0.00018, 0.00014, 0.00015, 0.00014, 0.00015, 0.00015, 0.00014, 0.00014, 0.00015, 0.00013, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00015, 0.00015, 0.00015, 0.00014, 0.00014, 0.00015, 0.00014, 0.00015, 0.00014, 0.00015, 0.00014, 0.00014, 0.00017, 0.00014, 0.00014, 0.00014, 0.00015, 0.00015, 0.00015, 0.00014, 0.00017, 0.00014, 0.00015, 0.00014, 0.00014, 0.00013, 0.00015, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00015, 0.00014, 0.00018, 0.00014, 0.00015, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00015, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00013, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00015, 0.00014, 0.00015, 0.00013, 0.00014, 0.00015, 0.00016, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00015, 0.00013, 0.00014, 0.00014, 0.00014, 0.00013, 0.00014, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00015, 0.00014, 0.00015, 0.00014, 0.00015, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00015, 0.00015, 0.00014, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00015, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00015, 0.00014, 0.00015, 0.00014, 0.00015, 0.00014, 0.00014, 0.00015, 0.00015, 0.00015, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00016, 0.00014, 0.00014, 0.00015, 0.00015, 0.00015, 0.00015, 0.00014, 0.00014, 0.00014, 0.00015, 0.00015, 0.00014, 0.00016, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00015, 0.00015, 0.00014, 0.00015, 0.00014, 0.00015, 0.00014, 0.00014, 0.00015, 0.00014, 0.00014, 0.00015, 0.00015, 0.00014, 0.00016, 0.00014, 0.00015, 0.00015, 0.00015]}, "backward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.02248, 0.02331, 0.02263, 0.02336, 0.02223, 0.02299, 0.02211, 0.02247, 0.0226, 0.02292, 0.02307, 0.02276, 0.02341, 0.02329, 0.02311, 0.02274, 0.02235, 0.0235, 0.02241, 0.02254, 0.0226, 0.02238, 0.02202, 0.02262, 0.02257, 0.02202, 0.02244, 0.02212, 0.02257, 0.02222, 0.02301, 0.02231, 0.02146, 0.02328, 0.0228, 0.02276, 0.02277, 0.02305, 0.02315, 0.02206, 0.02273, 0.02196, 0.02292, 0.0229, 0.02318, 0.02404, 0.02342, 0.02372, 0.024, 0.02283, 0.02293, 0.02329, 0.02241, 0.02288, 0.02249, 0.02209, 0.0225, 0.02317, 0.02289, 0.02337, 0.02275, 0.02241, 0.02374, 0.02164, 0.02208, 0.02228, 0.02281, 0.02282, 0.02272, 0.0226, 0.0227, 0.02228, 0.02281, 0.02266, 0.02389, 0.02245, 0.02241, 0.02233, 0.02295, 0.02231, 0.0221, 0.02223, 0.0226, 0.02234, 0.02195, 0.02202, 0.02245, 0.0226, 0.02275, 0.02248, 0.0222, 0.02241, 0.02244, 0.02231, 0.02257, 0.02222, 0.02266, 0.02423, 0.02272, 0.02227, 0.02299, 0.02249, 0.0224, 0.02471, 0.02315, 0.02261, 0.02228, 0.02296, 0.02277, 0.02251, 0.02275, 0.02249, 0.02349, 0.022, 0.02327, 0.0234, 0.02263, 0.02233, 0.02301, 0.02227, 0.02246, 0.02257, 0.02278, 0.02253, 0.02246, 0.02297, 0.02258, 0.02373, 0.02268, 0.02299, 0.02323, 0.02295, 0.02269, 0.02271, 0.02329, 0.02248, 0.02289, 0.02291, 0.02254, 0.02282, 0.02401, 0.02262, 0.02444, 0.02261, 0.0226, 0.02263, 0.02259, 0.02307, 0.02224, 0.02211, 0.02289, 0.02273, 0.02385, 0.02337, 0.02258, 0.02316, 0.02269, 0.02287, 0.02301, 0.0225, 0.02248, 0.02339, 0.02296, 0.02226, 0.02308, 0.02301, 0.02193, 0.02223, 0.02389, 0.02273, 0.02314, 0.0224, 0.02271, 0.02292, 0.0234, 0.02311, 0.02278, 0.02281, 0.02287, 0.02271, 0.02258, 0.02224, 0.02289, 0.02216, 0.02306, 0.02215, 0.02293, 0.02325, 0.02272, 0.02257, 0.02265, 0.02257, 0.02237, 0.02338, 0.02396, 0.02264, 0.02255, 0.02263, 0.02261, 0.02319, 0.02273, 0.0227, 0.02359, 0.02237, 0.02352, 0.02453, 0.02244, 0.02254, 0.02341, 0.02295, 0.02318, 0.02233, 0.02248, 0.02304, 0.02424, 0.02304, 0.02275, 0.02374, 0.02258, 0.02316, 0.02275, 0.02259, 0.02278, 0.02276, 0.02303, 0.02314, 0.02359, 0.02289, 0.02295, 0.02301, 0.02271, 0.02295, 0.02286, 0.02295, 0.02288, 0.02247, 0.02599, 0.02329, 0.02375, 0.02231, 0.0227, 0.0222, 0.02287, 0.02291, 0.02232, 0.02287, 0.02269, 0.0222, 0.02306, 0.02281, 0.0228, 0.02143, 0.02285, 0.02337, 0.02236, 0.02228, 0.02243, 0.02313, 0.02393, 0.02356, 0.02319, 0.02319, 0.02354, 0.02282, 0.02254, 0.02335, 0.02225, 0.02305, 0.0231, 0.02313, 0.02277, 0.02351, 0.02342, 0.02326, 0.02253, 0.02222, 0.02252, 0.02264, 0.02318, 0.02321, 0.02292, 0.02334, 0.02285, 0.02282, 0.02307, 0.02259, 0.02166, 0.02265, 0.02214, 0.02373, 0.02309, 0.0232, 0.02261, 0.02274, 0.02256, 0.02221, 0.02164, 0.02324, 0.02299, 0.02313, 0.02404, 0.02301, 0.02264, 0.02252, 0.02325, 0.02343, 0.02291, 0.02247, 0.0231, 0.02252, 0.02239, 0.02337, 0.02232, 0.02332, 0.02306, 0.02293, 0.02287, 0.02295, 0.02297, 0.02351, 0.02268, 0.02263, 0.02425, 0.02263, 0.02361, 0.023, 0.02223, 0.02273, 0.02318, 0.02333, 0.0232, 0.02407, 0.02312, 0.0227, 0.02288, 0.02285, 0.02227, 0.0233, 0.02303, 0.02288, 0.0233, 0.0231, 0.02299, 0.02245, 0.02284, 0.02224, 0.02277, 0.02352, 0.02304, 0.02289, 0.02369, 0.02293, 0.02308, 0.02248, 0.02362, 0.02358, 0.02328, 0.02302, 0.0234, 0.02273, 0.02296, 0.02329, 0.0228, 0.0234, 0.02231, 0.02262, 0.02265, 0.02299, 0.02199, 0.02303, 0.02291, 0.02278, 0.02341, 0.0232, 0.02291, 0.02339, 0.02355, 0.02363, 0.02324, 0.02236, 0.023, 0.02327, 0.02343, 0.02262, 0.02317, 0.02371, 0.02282, 0.02307, 0.0239, 0.02366, 0.02297, 0.02286, 0.02285, 0.0232, 0.02342, 0.02385, 0.02348, 0.02254, 0.02321, 0.02256]}, "backward-send-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00014, 0.00018, 0.00017, 0.00019, 0.00013, 0.00016, 0.00015, 0.00015, 0.00015, 0.00015, 0.00015, 0.00016, 0.00015, 0.00015, 0.00015, 0.00015, 0.00015, 0.00017, 0.00015, 0.00016, 0.00015, 0.00015, 0.00017, 0.00015, 0.00015, 0.00015, 0.00016, 0.00015, 0.00015, 0.00015, 0.00015, 0.00016, 0.00015, 0.00015, 0.00015, 0.00015, 0.00015, 0.00015, 0.00015, 0.00016, 0.00015, 0.00015, 0.00015, 0.00016, 0.00015, 0.00017, 0.00016, 0.00015, 0.00015, 0.00016, 0.00014, 0.00015, 0.00015, 0.00015, 0.00015, 0.00015, 0.00019, 0.00015, 0.00015, 0.00017, 0.00015, 0.00015, 0.00015, 0.00015, 0.00015, 0.00015, 0.00014, 0.00012, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00016, 0.00017, 0.00016, 0.00012, 0.00016, 0.00012, 0.00012, 0.00013, 0.00013, 0.00016, 0.00013, 0.00013, 0.00014, 0.00013, 0.00012, 0.00012, 0.00012, 0.00014, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00017, 0.00014, 0.00017, 0.00013, 0.00013, 0.00013, 0.00019, 0.00014, 0.00014, 0.00013, 0.00018, 0.00013, 0.00014, 0.00013, 0.00016, 0.00015, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00014, 0.00015, 0.00013, 0.00014, 0.00012, 0.00013, 0.00013, 0.00012, 0.00016, 0.00017, 0.00013, 0.00014, 0.00013, 0.00015, 0.00013, 0.00013, 0.00015, 0.00016, 0.00013, 0.00013, 0.00013, 0.00015, 0.00013, 0.00016, 0.00015, 0.00013, 0.00013, 0.00013, 0.00013, 0.00017, 0.00015, 0.00017, 0.00014, 0.00013, 0.00014, 0.00013, 0.00014, 0.00013, 0.00013, 0.00015, 0.00014, 0.00013, 0.00015, 0.00014, 0.00012, 0.00014, 0.00013, 0.00016, 0.00015, 0.00015, 0.00013, 0.00013, 0.00013, 0.00013, 0.00016, 0.00012, 0.00013, 0.00015, 0.00013, 0.00015, 0.00014, 0.00016, 0.00013, 0.00013, 0.00015, 0.00016, 0.00012, 0.00016, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00019, 0.00013, 0.00013, 0.00014, 0.00015, 0.00014, 0.00013, 0.00014, 0.00016, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00015, 0.00013, 0.00013, 0.00014, 0.00014, 0.00013, 0.00016, 0.00013, 0.00018, 0.00012, 0.00014, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00014, 0.00016, 0.00012, 0.00012, 0.00013, 0.00014, 0.00012, 0.00018, 0.00013, 0.00013, 0.00013, 0.00014, 0.00015, 0.00014, 0.00014, 0.00012, 0.00013, 0.00013, 0.00014, 0.00014, 0.00015, 0.00015, 0.00013, 0.00013, 0.00014, 0.00013, 0.00012, 0.00013, 0.00012, 0.00015, 0.00013, 0.00013, 0.00014, 0.00015, 0.00012, 0.00013, 0.00013, 0.00014, 0.00013, 0.00012, 0.00017, 0.00013, 0.00015, 0.00017, 0.00013, 0.00014, 0.00016, 0.00012, 0.00014, 0.00013, 0.00014, 0.00013, 0.00015, 0.00015, 0.00016, 0.00017, 0.00013, 0.00018, 0.00014, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00014, 0.00016, 0.00014, 0.00013, 0.00014, 0.00012, 0.00013, 0.00012, 0.00016, 0.00012, 0.00015, 0.00013, 0.00013, 0.00013, 0.00012, 0.00016, 0.00017, 0.00013, 0.00013, 0.00013, 0.00014, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00014, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00014, 0.00015, 0.00013, 0.00013, 0.00013, 0.00017, 0.00014, 0.00014, 0.00016, 0.00013, 0.00015, 0.00014, 0.00017, 0.00016, 0.00014, 0.00014, 0.00013, 0.00015, 0.00012, 0.00013, 0.00012, 0.00013, 0.00014, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00012, 0.00013, 0.00015, 0.00014, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00014, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00014, 0.00013, 0.00013, 0.00013, 0.00012, 0.00015, 0.00016, 0.00013, 0.00013, 0.00014, 0.00014, 0.00017, 0.00012, 0.00015, 0.00016, 0.00016, 0.00013, 0.00015, 0.00014, 0.00013, 0.00013, 0.00012, 0.00012, 0.00017, 0.00013, 0.00013, 0.00012, 0.00012]}, "forward-send-backward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [7.29163, 0.07663, 0.08035, 0.06332, 0.06621, 0.06965, 0.06672, 0.06872, 0.07455, 0.0683, 0.06975, 0.07264, 0.07308, 0.06869, 0.0749, 0.06785, 0.06696, 0.07011, 0.07008, 0.06771, 0.06763, 0.06853, 0.06929, 0.06793, 0.0646, 0.06794, 0.06582, 0.06618, 0.07898, 0.06585, 0.0677, 0.06681, 0.07017, 0.06602, 0.06883, 0.06722, 0.06997, 0.06853, 0.07057, 0.06872, 0.06884, 0.06699, 0.06869, 0.07012, 0.06782, 0.06999, 0.06845, 0.06563, 0.07187, 0.06575, 0.06637, 0.06468, 0.06438, 0.06646, 0.06395, 0.06524, 0.08025, 0.06764, 0.06976, 0.06968, 0.06431, 0.06784, 0.06839, 0.06965, 0.06878, 0.06848, 0.06691, 0.06998, 0.07092, 0.06857, 0.0693, 0.06815, 0.07095, 0.07046, 0.07279, 0.07009, 0.07045, 0.07242, 0.06971, 0.06878, 0.0711, 0.06854, 0.0703, 0.07136, 0.07206, 0.19699, 0.06856, 0.07017, 0.0772, 0.07413, 0.06965, 0.06662, 0.06863, 0.07002, 0.06852, 0.06895, 0.06723, 0.06766, 0.06739, 0.07615, 0.06865, 0.0659, 0.07051, 0.0678, 0.06754, 0.06717, 0.07145, 0.07015, 0.06808, 0.06744, 0.06521, 0.06518, 0.06265, 0.06299, 0.06279, 0.06454, 0.07004, 0.06844, 0.06842, 0.06744, 0.06305, 0.06615, 0.07084, 0.06889, 0.06934, 0.0652, 0.07021, 0.0665, 0.06497, 0.06458, 0.06483, 0.0654, 0.0651, 0.06488, 0.06369, 0.06434, 0.06672, 0.06482, 0.06827, 0.06829, 0.0643, 0.06825, 0.06762, 0.06752, 0.06536, 0.06267, 0.06412, 0.06238, 0.0644, 0.06315, 0.06427, 0.06278, 0.06772, 0.06453, 0.06547, 0.06433, 0.06477, 0.06262, 0.06246, 0.0656, 0.06412, 0.06447, 0.06356, 0.06614, 0.0655, 0.06558, 0.06542, 0.06499, 0.06312, 0.06403, 0.06715, 0.06427, 0.06479, 0.06361, 0.06722, 0.06583, 0.06476, 0.06651, 0.06877, 0.06755, 0.06567, 0.06624, 0.06526, 0.06717, 0.06755, 0.06946, 0.06655, 0.06526, 0.06418, 0.06359, 0.06533, 0.06548, 0.06698, 0.06537, 0.06464, 0.07565, 0.06673, 0.06462, 0.06523, 0.06525, 0.05829, 0.06037, 0.06399, 0.06429, 0.06234, 0.06138, 0.06591, 0.06529, 0.06565, 0.06508, 0.0686, 0.06838, 0.12228, 0.06666, 0.06636, 0.0641, 0.06601, 0.06468, 0.06395, 0.06568, 0.06779, 0.06425, 0.06928, 0.06612, 0.06928, 0.0652, 0.06359, 0.06153, 0.06449, 0.06439, 0.06432, 0.06445, 0.06351, 0.06481, 0.06503, 0.06334, 0.0646, 0.06418, 0.06493, 0.06414, 0.06257, 0.06426, 0.06752, 0.06251, 0.06434, 0.06117, 0.06509, 0.06177, 0.06484, 0.06385, 0.06538, 0.06711, 0.0659, 0.06606, 0.06549, 0.06518, 0.06537, 0.06313, 0.0654, 0.0676, 0.06603, 0.06663, 0.06705, 0.06676, 0.0651, 0.0677, 0.06421, 0.06506, 0.06513, 0.06577, 0.06915, 0.06804, 0.06617, 0.06569, 0.06722, 0.06636, 0.06674, 0.06574, 0.06698, 0.06664, 0.06663, 0.06459, 0.06384, 0.06515, 0.06699, 0.06757, 0.06645, 0.06668, 0.0657, 0.06812, 0.06673, 0.06651, 0.06468, 0.06953, 0.06688, 0.06585, 0.06531, 0.06508, 0.06559, 0.06487, 0.0647, 0.06539, 0.06861, 0.06738, 0.06026, 0.06597, 0.06493, 0.06467, 0.06738, 0.06641, 0.06506, 0.0673, 0.06795, 0.06714, 0.06848, 0.06828, 0.07103, 0.0742, 0.06691, 0.06638, 0.06521, 0.06791, 0.06493, 0.06647, 0.06851, 0.06674, 0.06949, 0.18067, 0.06896, 0.0653, 0.06795, 0.06966, 0.06981, 0.0677, 0.06607, 0.06924, 0.06499, 0.06831, 0.06832, 0.06949, 0.07135, 0.06537, 0.07037, 0.06461, 0.06603, 0.06572, 0.06904, 0.06866, 0.06911, 0.06296, 0.0684, 0.06727, 0.06737, 0.069, 0.06738, 0.07025, 0.06407, 0.06509, 0.06963, 0.06441, 0.07069, 0.07222, 0.07463, 0.07367, 0.07032, 0.07129, 0.07156, 0.07253, 0.06858, 0.06926, 0.06916, 0.06788, 0.06771, 0.06859, 0.06745, 0.07278, 0.06943, 0.06671, 0.0691, 0.06585, 0.06975, 0.07019, 0.07413, 0.0711, 0.07228, 0.07684, 0.07091, 0.0736, 0.07134, 0.07497, 0.07213, 0.06976, 0.07166, 0.0746, 0.0763, 0.06965, 0.07059, 0.07384, 0.07021, 0.07072]}, "backward-send-forward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.84189, 0.0034, 0.00335, 0.0028, 0.00275, 0.0029, 0.00298, 0.00297, 0.00304, 0.00306, 0.00309, 0.00308, 0.00301, 0.00302, 0.00299, 0.00294, 0.003, 0.00307, 0.0031, 0.00304, 0.00303, 0.00294, 0.00305, 0.00298, 0.00301, 0.00306, 0.0029, 0.00302, 0.00303, 0.0031, 0.00306, 0.00304, 0.00303, 0.00301, 0.00294, 0.00305, 0.00312, 0.00303, 0.00301, 0.00328, 0.00302, 0.00288, 0.00306, 0.00304, 0.00304, 0.00303, 0.00299, 0.00297, 0.003, 0.00305, 0.00302, 0.00306, 0.00303, 0.00307, 0.00305, 0.00294, 0.00385, 0.00305, 0.00293, 0.00307, 0.00295, 0.003, 0.00297, 0.00308, 0.00305, 0.00303, 0.00302, 0.00254, 0.00275, 0.00284, 0.00252, 0.00253, 0.00257, 0.00262, 0.00255, 0.00266, 0.00264, 0.0026, 0.00255, 0.00265, 0.00267, 0.00266, 0.00269, 0.0026, 0.00263, 0.00301, 0.00264, 0.00265, 0.00269, 0.00261, 0.00267, 0.00257, 0.00268, 0.0027, 0.00261, 0.00268, 0.00261, 0.00264, 0.00255, 0.00261, 0.00281, 0.00269, 0.00271, 0.00271, 0.00264, 0.00265, 0.00268, 0.0026, 0.00262, 0.00283, 0.00271, 0.00272, 0.00266, 0.00257, 0.00253, 0.00256, 0.00276, 0.00272, 0.00264, 0.00283, 0.00271, 0.00262, 0.00269, 0.00277, 0.00266, 0.0026, 0.00277, 0.00282, 0.00271, 0.00264, 0.00273, 0.00268, 0.00264, 0.00266, 0.0027, 0.00274, 0.00274, 0.0027, 0.00271, 0.00273, 0.00279, 0.0027, 0.00276, 0.00265, 0.0028, 0.00278, 0.00273, 0.00287, 0.00273, 0.00277, 0.00273, 0.00265, 0.00272, 0.00267, 0.00277, 0.00265, 0.00267, 0.0027, 0.00268, 0.00269, 0.00264, 0.00278, 0.00271, 0.00267, 0.00258, 0.00265, 0.00262, 0.00273, 0.00273, 0.00285, 0.00277, 0.00264, 0.00285, 0.00276, 0.00269, 0.00275, 0.00339, 0.00271, 0.00288, 0.00276, 0.00282, 0.00266, 0.00281, 0.00268, 0.00277, 0.00269, 0.00271, 0.0028, 0.00273, 0.00293, 0.00264, 0.00265, 0.00285, 0.0026, 0.00269, 0.00287, 0.00272, 0.00278, 0.0028, 0.00271, 0.00259, 0.00259, 0.00273, 0.00266, 0.0027, 0.00278, 0.00275, 0.0029, 0.00268, 0.00277, 0.0027, 0.00273, 0.00744, 0.00272, 0.00261, 0.00274, 0.00281, 0.00282, 0.00277, 0.00264, 0.00277, 0.00268, 0.00266, 0.00256, 0.00267, 0.00276, 0.00287, 0.00271, 0.00271, 0.00265, 0.00268, 0.00304, 0.00294, 0.00305, 0.0029, 0.00293, 0.00278, 0.00294, 0.00291, 0.00285, 0.00291, 0.00286, 0.00284, 0.00295, 0.0029, 0.0029, 0.00287, 0.00287, 0.0029, 0.00282, 0.00289, 0.0028, 0.0029, 0.00288, 0.0028, 0.00266, 0.0026, 0.00273, 0.00266, 0.00275, 0.00276, 0.00275, 0.00283, 0.0027, 0.00268, 0.00279, 0.00265, 0.00277, 0.00279, 0.00278, 0.00276, 0.00273, 0.00266, 0.00264, 0.00265, 0.00264, 0.00268, 0.00279, 0.00284, 0.00276, 0.00269, 0.00277, 0.00277, 0.00268, 0.00268, 0.00266, 0.00263, 0.00274, 0.0026, 0.00268, 0.00269, 0.00259, 0.00258, 0.00283, 0.00267, 0.00256, 0.00279, 0.0026, 0.00276, 0.00258, 0.00269, 0.00264, 0.00266, 0.00272, 0.10829, 0.00271, 0.00273, 0.00261, 0.00278, 0.00265, 0.00268, 0.00259, 0.00272, 0.00286, 0.00273, 0.00271, 0.00286, 0.00269, 0.00267, 0.0027, 0.00281, 0.0027, 0.00267, 0.00273, 0.0027, 0.00257, 0.0026, 0.00298, 0.0026, 0.00269, 0.00264, 0.00279, 0.00281, 0.00269, 0.0031, 0.0027, 0.0027, 0.00273, 0.0028, 0.00277, 0.00279, 0.00274, 0.00279, 0.00256, 0.00277, 0.00273, 0.00275, 0.00268, 0.00277, 0.00282, 0.0028, 0.00268, 0.00285, 0.00263, 0.00275, 0.00272, 0.0027, 0.00272, 0.00269, 0.00263, 0.00272, 0.00262, 0.00268, 0.0027, 0.00275, 0.0027, 0.00256, 0.00261, 0.00265, 0.00271, 0.00266, 0.00266, 0.00275, 0.00281, 0.00274, 0.00263, 0.00267, 0.00277, 0.00271, 0.00263, 0.00267, 0.00269, 0.00285, 0.00267, 0.00275, 0.00276, 0.00277, 0.0026, 0.00277, 0.0027, 0.00279, 0.00284, 0.00284, 0.0028, 0.00331, 0.00286, 0.0027, 0.00271, 0.00257, 0.00255]}, "layernorm-grads-all-reduce-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [4e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 2e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 3e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 2e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05]}, "embedding-grads-all-reduce-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00071, 0.00045, 0.00045, 0.00046, 0.00046, 0.00046, 0.00046, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00047, 0.00045, 0.00045, 0.00046, 0.00045, 0.00046, 0.00049, 0.00046, 0.00046, 0.00046, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00046, 0.00046, 0.00045, 0.00046, 0.00046, 0.00047, 0.00045, 0.00046, 0.00046, 0.00045, 0.00047, 0.00045, 0.00046, 0.00046, 0.00046, 0.00048, 0.00046, 0.00048, 0.00045, 0.00046, 0.00048, 0.00045, 0.00045, 0.00046, 0.00046, 0.00046, 0.00047, 0.00048, 0.00046, 0.00045, 0.00045, 0.00046, 0.00046, 0.00045, 0.00046, 0.00046, 0.00047, 0.00044, 0.00044, 0.00045, 0.00045, 0.00047, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00047, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00047, 0.00045, 0.00045, 0.00049, 0.00045, 0.00046, 0.00044, 0.00046, 0.00047, 0.00045, 0.00045, 0.00045, 0.00046, 0.00047, 0.00045, 0.00044, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00081, 0.00046, 0.00046, 0.00047, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00048, 0.00047, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00046, 0.00045, 0.00045, 0.00047, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00044, 0.00045, 0.00045, 0.00047, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00046, 0.00047, 0.00046, 0.00047, 0.00045, 0.00046, 0.00044, 0.00045, 0.00045, 0.00045, 0.00044, 0.00046, 0.00045, 0.00046, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00046, 0.00046, 0.00045, 0.00045, 0.00047, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00047, 0.00049, 0.00047, 0.00045, 0.00045, 0.00049, 0.00045, 0.00049, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00049, 0.00046, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00048, 0.00045, 0.00046, 0.00046, 0.00044, 0.00045, 0.00045, 0.00044, 0.00046, 0.00045, 0.00044, 0.00048, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00047, 0.00046, 0.00048, 0.00045, 0.00045, 0.00045, 0.00047, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00051, 0.00049, 0.00045, 0.00046, 0.00047, 0.00046, 0.00045, 0.00045, 0.00045, 0.00046, 0.00047, 0.00049, 0.0005, 0.00046, 0.00045, 0.00047, 0.00046, 0.00045, 0.00045, 0.00049, 0.00045, 0.00049, 0.00045, 0.00045, 0.00046, 0.00045, 0.0005, 0.00045, 0.00046, 0.00044, 0.00045, 0.00049, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00049, 0.00046, 0.00048, 0.00047, 0.00045, 0.00045, 0.00046, 0.00048, 0.00045, 0.00046, 0.00045, 0.00045, 0.00047, 0.00046, 0.00048, 0.00048, 0.00048, 0.00048, 0.00045, 0.00045, 0.00048, 0.00047, 0.00045, 0.00048, 0.00045, 0.00046, 0.00045, 0.00046, 0.00045, 0.00047, 0.00047, 0.00046, 0.00046, 0.00047, 0.00047, 0.00044, 0.00045, 0.00045, 0.00048, 0.00048, 0.00048, 0.00045, 0.00045, 0.00046, 0.00045, 0.00048, 0.00048, 0.00045, 0.00045, 0.00045, 0.00046, 0.00044, 0.00045, 0.00045, 0.00049, 0.00045, 0.00045, 0.00045, 0.00045, 0.00048, 0.00045, 0.00046, 0.00049, 0.00046, 0.00046, 0.00044, 0.00048, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00048, 0.00047, 0.00049, 0.00045, 0.00045, 0.00053, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00044, 0.00049, 0.00045, 0.00044, 0.00048, 0.00045, 0.00045, 0.00045, 0.00045]}, "all-grads-sync-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [6.13385, 0.00147, 0.00148, 0.00147, 0.00149, 0.00151, 0.00148, 0.00148, 0.00147, 0.00149, 0.00149, 0.00147, 0.00149, 0.00149, 0.00147, 0.00147, 0.00147, 0.00149, 0.00149, 0.00148, 0.00148, 0.00148, 0.0015, 0.0015, 0.00147, 0.00148, 0.00149, 0.00148, 0.00148, 0.00148, 0.00147, 0.00148, 0.00149, 0.00149, 0.00148, 0.00148, 0.00149, 0.00147, 0.00148, 0.00148, 0.00147, 0.00147, 0.00148, 0.00149, 0.00148, 0.00148, 0.00149, 0.00148, 0.00149, 0.00148, 0.00147, 0.00147, 0.00147, 0.00148, 0.00148, 0.00149, 0.00148, 0.00149, 0.00147, 0.00147, 0.00149, 0.00148, 0.00148, 0.00149, 0.0015, 0.00148, 0.00148, 0.00149, 0.00148, 0.00149, 0.00148, 0.00147, 0.00149, 0.00149, 0.00148, 0.00146, 0.00147, 0.00148, 0.00147, 0.00148, 0.00149, 0.00147, 0.00146, 0.00148, 0.00148, 0.00147, 0.00149, 0.00148, 0.00149, 0.0015, 0.00148, 0.00147, 0.00147, 0.00147, 0.00149, 0.00149, 0.00148, 0.00148, 0.00148, 0.00148, 0.00147, 0.00149, 0.00148, 0.00148, 0.00149, 0.00148, 0.00147, 0.00147, 0.00147, 0.00148, 0.00149, 0.00147, 0.00148, 0.00148, 0.00147, 0.00149, 0.00147, 0.00147, 0.00149, 0.00149, 0.00146, 0.00149, 0.00147, 0.00149, 0.00149, 0.00148, 0.00147, 0.00148, 0.00148, 0.00148, 0.00149, 0.00148, 0.00147, 0.00149, 0.00151, 0.00147, 0.00148, 0.00147, 0.00148, 0.00148, 0.00147, 0.00147, 0.0015, 0.00149, 0.00148, 0.00147, 0.00148, 0.00147, 0.00148, 0.00148, 0.00147, 0.0015, 0.00147, 0.00147, 0.00147, 0.00148, 0.0015, 0.00148, 0.00148, 0.00147, 0.00148, 0.00149, 0.00149, 0.00148, 0.00148, 0.00148, 0.00148, 0.00147, 0.00149, 0.00149, 0.00149, 0.00147, 0.00147, 0.00148, 0.00147, 0.00147, 0.00147, 0.00148, 0.00146, 0.00148, 0.00147, 0.00149, 0.00147, 0.00149, 0.00149, 0.00147, 0.00147, 0.00148, 0.00147, 0.00148, 0.00148, 0.00148, 0.00148, 0.00149, 0.00147, 0.00149, 0.00148, 0.00148, 0.00148, 0.00149, 0.0015, 0.00148, 0.00149, 0.00148, 0.00149, 0.00149, 0.00149, 0.00148, 0.00148, 0.00149, 0.00149, 0.0015, 0.00149, 0.00148, 0.00149, 0.00149, 0.00149, 0.00151, 0.00148, 0.0015, 0.00148, 0.00149, 0.00149, 0.00149, 0.00149, 0.00149, 0.00149, 0.0015, 0.0015, 0.0015, 0.00149, 0.0015, 0.00149, 0.00149, 0.00147, 0.00148, 0.00149, 0.0015, 0.0015, 0.00149, 0.00147, 0.00149, 0.0015, 0.00149, 0.00148, 0.00148, 0.00148, 0.00148, 0.00148, 0.0015, 0.0015, 0.0015, 0.00148, 0.00149, 0.00149, 0.00149, 0.00149, 0.00148, 0.0015, 0.00149, 0.00148, 0.00151, 0.00149, 0.00148, 0.00149, 0.00147, 0.00147, 0.00154, 0.00149, 0.00147, 0.00148, 0.0015, 0.00149, 0.00152, 0.00148, 0.00148, 0.00148, 0.00148, 0.00149, 0.00148, 0.00151, 0.00147, 0.00148, 0.00151, 0.0015, 0.00149, 0.00147, 0.00148, 0.00149, 0.00149, 0.00151, 0.00148, 0.00149, 0.00149, 0.00149, 0.00147, 0.00148, 0.00148, 0.00147, 0.00148, 0.00148, 0.00149, 0.00148, 0.00148, 0.00149, 0.00148, 0.00148, 0.00152, 0.00149, 0.0015, 0.00148, 0.00148, 0.00147, 0.00148, 0.00149, 0.00149, 0.00147, 0.00149, 0.00151, 0.00147, 0.00148, 0.00148, 0.00149, 0.00147, 0.0015, 0.00149, 0.00149, 0.00148, 0.00149, 0.00149, 0.00149, 0.00149, 0.00148, 0.00148, 0.00148, 0.00148, 0.00149, 0.00149, 0.00149, 0.00149, 0.00148, 0.00149, 0.00149, 0.00149, 0.00148, 0.0015, 0.00148, 0.00151, 0.00148, 0.00151, 0.00147, 0.00147, 0.00149, 0.00148, 0.00148, 0.00148, 0.00148, 0.00147, 0.00149, 0.00149, 0.00149, 0.00148, 0.00149, 0.0015, 0.00148, 0.00148, 0.00149, 0.00148, 0.00148, 0.00149, 0.00148, 0.00149, 0.0015, 0.00147, 0.00149, 0.00148, 0.00149, 0.00149, 0.00148, 0.00147, 0.00149, 0.0015, 0.0015, 0.00149, 0.00148, 0.00147, 0.00149, 0.00147, 0.0015, 0.00149, 0.00149, 0.00149, 0.0015, 0.00148, 0.00149, 0.00149, 0.0015, 0.00148, 0.00148, 0.00148]}, "optimizer-copy-to-main-grad-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00022, 0.00015, 0.00014, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00014, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00014, 0.00014, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00014, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00014, 0.00013, 0.00012, 0.00014, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00014, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00014, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00014, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00014, 0.00012, 0.00012, 0.00013, 0.00012, 0.00014, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00015, 0.00014, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00015, 0.00013, 0.00014, 0.00014, 0.00012, 0.00014, 0.00013, 0.00012, 0.00012, 0.00014, 0.00013, 0.00012, 0.00013, 0.00012, 0.00014, 0.00012, 0.00014, 0.00014, 0.00012, 0.00012, 0.00014, 0.00013, 0.00014, 0.00012, 0.00014, 0.00014, 0.00014, 0.00013, 0.00014, 0.00012, 0.00013, 0.00014, 0.00012, 0.00014, 0.00013, 0.00014, 0.00012, 0.00014, 0.00012, 0.00014, 0.00012, 0.00012, 0.00012, 0.00012, 0.00014, 0.00012, 0.00013, 0.00014, 0.00012, 0.00014, 0.00012, 0.00013, 0.00013, 0.00014, 0.00012, 0.00015, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00014, 0.00014, 0.00014, 0.00014, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00015, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00012, 0.00013, 0.00013, 0.00015, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00014, 0.00014, 0.00013, 0.00014, 0.00012, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00015, 0.00014, 0.00014, 0.00013, 0.00012, 0.00014, 0.00013, 0.00013, 0.00013, 0.00014, 0.00015, 0.00015, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00015, 0.00014, 0.00015, 0.00013, 0.00013, 0.00014, 0.00013, 0.00014, 0.00014, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00017, 0.00014, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00014, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012]}, "optimizer-clip-main-grad-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.11156, 0.00067, 0.00064, 0.00065, 0.00062, 0.00063, 0.00062, 0.00063, 0.00062, 0.00062, 0.00062, 0.00062, 0.00062, 0.00062, 0.00067, 0.00062, 0.00063, 0.00063, 0.00063, 0.00063, 0.00062, 0.00062, 0.00061, 0.00062, 0.00062, 0.00062, 0.00064, 0.00064, 0.00064, 0.00063, 0.00062, 0.00062, 0.00063, 0.00062, 0.00062, 0.00066, 0.00062, 0.00062, 0.00063, 0.00063, 0.00063, 0.00062, 0.00062, 0.00062, 0.00062, 0.00065, 0.00062, 0.00064, 0.00066, 0.00062, 0.00062, 0.00063, 0.00062, 0.00062, 0.00065, 0.00065, 0.00064, 0.00063, 0.00062, 0.00064, 0.00063, 0.00062, 0.00067, 0.00062, 0.00062, 0.00063, 0.00062, 0.00062, 0.00063, 0.00064, 0.00062, 0.00062, 0.00062, 0.00064, 0.00062, 0.00062, 0.00063, 0.00062, 0.00062, 0.00062, 0.00064, 0.00063, 0.00064, 0.00063, 0.00066, 0.00062, 0.00062, 0.00062, 0.00061, 0.00062, 0.00062, 0.00062, 0.00062, 0.00062, 0.00062, 0.00061, 0.00062, 0.00071, 0.00046, 0.00069, 0.00062, 0.00068, 0.00062, 0.00062, 0.00045, 0.00062, 0.00062, 0.00063, 0.00062, 0.00062, 0.0005, 0.00048, 0.00062, 0.00062, 0.00062, 0.00062, 0.00048, 0.00062, 0.00062, 0.00064, 0.00047, 0.00062, 0.00066, 0.00062, 0.00062, 0.00062, 0.00062, 0.00064, 0.00064, 0.00062, 0.00046, 0.00062, 0.00062, 0.00062, 0.00065, 0.00062, 0.00062, 0.00062, 0.00062, 0.00062, 0.00062, 0.00062, 0.00062, 0.00062, 0.00067, 0.00064, 0.00061, 0.00063, 0.00064, 0.00061, 0.00064, 0.00062, 0.00062, 0.00062, 0.00047, 0.00062, 0.00062, 0.00062, 0.00062, 0.00064, 0.00061, 0.00064, 0.00064, 0.00062, 0.00063, 0.00064, 0.00067, 0.00064, 0.00062, 0.00064, 0.00063, 0.00062, 0.00064, 0.00063, 0.00062, 0.00065, 0.00064, 0.00064, 0.00064, 0.00063, 0.00064, 0.00063, 0.00065, 0.00062, 0.00063, 0.00062, 0.00065, 0.00062, 0.00061, 0.00063, 0.00061, 0.00062, 0.00066, 0.00062, 0.00065, 0.00062, 0.00061, 0.00063, 0.00063, 0.00062, 0.00069, 0.00066, 0.00066, 0.00067, 0.00067, 0.00071, 0.00067, 0.00067, 0.00065, 0.00065, 0.00067, 0.00067, 0.00066, 0.00066, 0.00067, 0.00067, 0.00071, 0.00066, 0.00066, 0.00066, 0.00066, 0.00067, 0.00067, 0.00067, 0.00066, 0.00068, 0.00066, 0.00067, 0.00065, 0.00066, 0.00066, 0.00065, 0.00069, 0.00067, 0.00066, 0.00066, 0.00068, 0.00065, 0.00064, 0.00065, 0.00067, 0.00065, 0.00066, 0.00066, 0.00067, 0.00066, 0.00066, 0.00067, 0.00066, 0.00067, 0.00067, 0.00067, 0.00066, 0.00067, 0.00067, 0.00067, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00073, 0.00069, 0.00066, 0.00065, 0.00064, 0.00067, 0.00066, 0.00067, 0.00066, 0.00073, 0.00067, 0.00067, 0.00067, 0.00067, 0.00066, 0.00068, 0.00065, 0.00065, 0.00067, 0.00067, 0.00066, 0.00066, 0.00067, 0.00067, 0.00067, 0.00067, 0.00066, 0.00064, 0.00066, 0.00067, 0.00064, 0.00064, 0.00064, 0.00064, 0.00064, 0.00064, 0.00064, 0.00063, 0.00064, 0.00064, 0.00064, 0.00064, 0.00066, 0.00065, 0.00064, 0.00064, 0.00064, 0.00064, 0.00063, 0.00064, 0.00064, 0.00065, 0.00065, 0.00064, 0.00073, 0.00064, 0.00063, 0.00064, 0.00063, 0.00064, 0.00064, 0.00064, 0.00064, 0.00064, 0.00064, 0.00066, 0.00065, 0.00064, 0.00063, 0.00063, 0.00064, 0.00065, 0.00065, 0.00065, 0.00065, 0.00063, 0.00064, 0.00063, 0.00063, 0.00064, 0.00064, 0.00065, 0.00064, 0.00063, 0.00063, 0.00065, 0.00063, 0.00064, 0.00063, 0.00064, 0.00063, 0.00066, 0.00063, 0.00065, 0.00064, 0.00063, 0.00064, 0.00063, 0.00064, 0.00064, 0.00064, 0.00066, 0.00066, 0.00065, 0.00064, 0.00066, 0.00065, 0.00065, 0.00065, 0.00066, 0.00065, 0.00065, 0.00065, 0.00066, 0.00066, 0.00064, 0.00063, 0.00065, 0.00065, 0.00066, 0.00064, 0.00066, 0.00065, 0.00066, 0.00067, 0.00066, 0.00066, 0.00065, 0.00066, 0.00065, 0.00068, 0.00066, 0.00066, 0.00065, 0.00063, 0.00064, 0.00063, 0.00063, 0.00064]}, "optimizer-count-zeros-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00352, 0.00261, 0.00262, 0.00279, 0.00266, 0.00279, 0.00264, 0.00264, 0.00265, 0.00263, 0.00263, 0.00263, 0.00266, 0.00265, 0.00265, 0.00266, 0.00262, 0.00265, 0.00264, 0.00267, 0.00262, 0.00264, 0.00263, 0.00264, 0.00265, 0.00263, 0.00264, 0.00266, 0.00265, 0.00262, 0.00263, 0.00265, 0.00266, 0.00263, 0.00264, 0.00264, 0.00264, 0.00264, 0.00264, 0.00265, 0.00265, 0.00264, 0.00265, 0.00266, 0.00264, 0.00316, 0.00266, 0.00263, 0.00279, 0.0027, 0.00263, 0.00263, 0.00267, 0.00263, 0.00264, 0.00264, 0.00265, 0.00262, 0.00265, 0.00265, 0.00264, 0.00266, 0.00277, 0.00265, 0.00266, 0.00266, 0.00265, 0.00265, 0.00264, 0.00266, 0.00267, 0.00263, 0.00263, 0.00266, 0.00265, 0.00263, 0.00263, 0.00265, 0.00263, 0.00265, 0.00293, 0.00263, 0.00273, 0.00264, 0.00285, 0.00263, 0.00265, 0.00265, 0.00265, 0.00263, 0.00264, 0.00265, 0.00264, 0.00263, 0.00263, 0.00265, 0.00262, 0.00298, 0.00265, 0.0031, 0.00263, 0.00312, 0.00264, 0.00267, 0.00263, 0.00296, 0.00265, 0.00262, 0.00266, 0.00263, 0.00298, 0.00266, 0.00265, 0.00263, 0.00276, 0.00265, 0.00266, 0.00264, 0.00264, 0.00266, 0.00264, 0.00265, 0.00268, 0.00265, 0.00264, 0.00264, 0.00263, 0.00266, 0.00264, 0.00265, 0.00264, 0.00264, 0.00263, 0.00262, 0.00284, 0.00263, 0.00263, 0.00265, 0.00265, 0.00264, 0.00263, 0.00263, 0.00264, 0.00265, 0.00298, 0.00264, 0.00263, 0.00266, 0.00264, 0.00265, 0.00264, 0.00264, 0.00267, 0.00264, 0.00265, 0.00262, 0.00264, 0.00271, 0.00266, 0.00266, 0.00265, 0.00266, 0.00267, 0.00268, 0.00263, 0.00265, 0.00282, 0.00266, 0.0027, 0.00265, 0.00266, 0.00265, 0.00264, 0.00267, 0.00269, 0.00278, 0.00264, 0.00268, 0.00264, 0.00265, 0.00265, 0.00267, 0.00267, 0.00265, 0.00265, 0.00265, 0.00267, 0.00265, 0.00266, 0.00264, 0.00265, 0.00263, 0.00265, 0.00265, 0.00267, 0.00267, 0.00263, 0.00264, 0.00264, 0.00265, 0.00262, 0.00264, 0.00266, 0.00263, 0.00267, 0.00264, 0.00264, 0.00264, 0.00266, 0.00265, 0.00266, 0.00264, 0.00264, 0.00267, 0.00265, 0.00262, 0.00266, 0.00265, 0.00267, 0.00266, 0.00267, 0.00295, 0.00267, 0.00268, 0.00263, 0.00265, 0.00265, 0.00263, 0.00266, 0.00299, 0.00264, 0.00267, 0.00262, 0.00269, 0.00265, 0.00264, 0.00265, 0.00263, 0.00265, 0.00265, 0.00286, 0.00266, 0.00266, 0.00264, 0.00264, 0.00265, 0.00264, 0.00266, 0.00266, 0.00267, 0.00264, 0.00265, 0.00265, 0.00265, 0.00266, 0.00264, 0.00268, 0.00264, 0.00262, 0.00267, 0.00263, 0.00312, 0.00265, 0.00265, 0.00264, 0.00263, 0.00265, 0.00265, 0.00264, 0.00266, 0.00268, 0.00264, 0.00266, 0.00263, 0.00267, 0.00265, 0.00263, 0.00266, 0.0027, 0.00266, 0.00263, 0.00264, 0.00276, 0.00265, 0.00266, 0.00264, 0.00264, 0.00264, 0.00302, 0.00265, 0.00265, 0.00269, 0.00264, 0.00263, 0.00266, 0.00264, 0.00267, 0.00263, 0.00264, 0.00265, 0.00266, 0.00264, 0.00265, 0.00265, 0.00265, 0.00267, 0.00261, 0.00262, 0.00266, 0.00263, 0.00265, 0.00266, 0.00265, 0.00262, 0.00266, 0.00267, 0.00262, 0.00266, 0.00265, 0.00264, 0.00263, 0.00265, 0.00263, 0.00268, 0.00282, 0.00266, 0.00264, 0.00264, 0.00262, 0.00266, 0.00265, 0.00266, 0.00264, 0.00276, 0.00264, 0.00264, 0.00265, 0.00263, 0.00265, 0.00265, 0.00266, 0.00265, 0.00265, 0.00264, 0.00262, 0.00264, 0.00264, 0.00265, 0.00265, 0.00266, 0.00267, 0.00266, 0.00268, 0.00265, 0.00275, 0.00263, 0.00275, 0.00263, 0.00265, 0.00264, 0.00265, 0.00264, 0.00265, 0.00264, 0.00266, 0.00269, 0.00266, 0.00264, 0.00263, 0.00266, 0.00267, 0.00266, 0.00266, 0.00268, 0.00267, 0.00265, 0.00265, 0.00266, 0.00265, 0.00265, 0.00263, 0.00266, 0.00264, 0.00268, 0.00266, 0.00263, 0.00268, 0.00265, 0.00265, 0.00278, 0.0027, 0.00264, 0.00264, 0.00263, 0.00265, 0.00266, 0.00265, 0.00269, 0.00264, 0.00265]}, "optimizer-inner-step-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.0024, 0.00067, 0.00066, 0.00066, 0.00066, 0.00067, 0.00067, 0.00067, 0.00066, 0.00066, 0.00066, 0.00067, 0.00066, 0.00067, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00067, 0.00066, 0.00066, 0.00066, 0.00065, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00065, 0.00066, 0.00066, 0.00065, 0.00066, 0.00067, 0.00066, 0.00067, 0.00065, 0.00065, 0.00066, 0.0007, 0.00065, 0.00066, 0.00065, 0.00066, 0.00066, 0.00067, 0.00066, 0.00067, 0.00067, 0.00066, 0.00065, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00065, 0.00065, 0.00066, 0.00066, 0.00065, 0.00065, 0.00065, 0.00066, 0.00065, 0.00065, 0.00065, 0.00065, 0.00066, 0.00065, 0.00067, 0.00065, 0.00066, 0.00065, 0.00066, 0.00066, 0.00065, 0.00066, 0.00066, 0.00065, 0.00065, 0.00065, 0.00065, 0.00065, 0.00065, 0.00065, 0.00065, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00065, 0.00065, 0.00065, 0.00067, 0.00065, 0.00065, 0.00065, 0.00065, 0.00066, 0.00065, 0.00066, 0.00065, 0.00066, 0.00065, 0.00066, 0.00066, 0.00065, 0.00065, 0.00065, 0.00065, 0.00065, 0.00065, 0.00065, 0.00065, 0.00065, 0.00069, 0.00066, 0.00065, 0.00065, 0.00065, 0.00065, 0.00066, 0.00066, 0.00065, 0.00066, 0.00065, 0.00065, 0.00066, 0.00065, 0.00065, 0.00065, 0.00066, 0.00067, 0.00066, 0.00065, 0.00066, 0.00066, 0.00065, 0.00066, 0.00065, 0.00066, 0.00065, 0.00065, 0.00065, 0.00065, 0.00068, 0.00065, 0.00066, 0.00065, 0.00066, 0.00066, 0.00066, 0.00065, 0.00065, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00065, 0.00065, 0.00066, 0.00065, 0.00066, 0.00065, 0.00065, 0.00066, 0.00065, 0.00066, 0.00066, 0.00065, 0.00065, 0.00066, 0.00065, 0.00065, 0.00066, 0.00065, 0.00065, 0.00066, 0.00066, 0.00066, 0.00065, 0.00066, 0.00066, 0.00065, 0.00065, 0.00067, 0.00066, 0.00069, 0.00068, 0.00069, 0.00069, 0.00068, 0.0007, 0.00069, 0.00069, 0.00067, 0.00067, 0.00068, 0.00068, 0.00068, 0.00068, 0.00069, 0.00068, 0.00069, 0.00068, 0.00068, 0.00069, 0.00091, 0.00068, 0.00068, 0.00069, 0.00068, 0.00068, 0.00068, 0.00068, 0.00068, 0.00069, 0.00068, 0.00068, 0.00068, 0.00067, 0.00068, 0.00067, 0.00068, 0.00067, 0.00067, 0.00068, 0.00071, 0.00068, 0.00068, 0.00068, 0.00068, 0.00069, 0.00068, 0.00067, 0.00068, 0.00067, 0.0007, 0.00069, 0.00067, 0.00069, 0.00068, 0.00068, 0.00068, 0.00068, 0.00068, 0.00068, 0.00067, 0.00069, 0.00068, 0.00068, 0.00068, 0.00067, 0.00068, 0.00068, 0.00067, 0.00068, 0.00068, 0.00068, 0.00068, 0.00068, 0.00067, 0.00068, 0.00067, 0.00067, 0.00068, 0.00067, 0.00068, 0.00068, 0.00069, 0.00068, 0.00069, 0.00068, 0.00068, 0.00068, 0.00067, 0.00068, 0.00068, 0.00067, 0.00067, 0.00066, 0.00067, 0.00067, 0.00067, 0.00067, 0.00067, 0.00066, 0.00066, 0.00068, 0.00067, 0.00067, 0.00067, 0.00066, 0.00067, 0.00066, 0.00067, 0.00067, 0.00067, 0.00066, 0.00067, 0.00067, 0.00067, 0.00066, 0.00067, 0.00067, 0.00067, 0.00068, 0.00067, 0.00067, 0.00066, 0.00067, 0.00066, 0.00067, 0.00067, 0.00067, 0.00067, 0.00067, 0.00066, 0.00066, 0.00067, 0.00068, 0.00066, 0.00067, 0.00067, 0.00067, 0.00068, 0.00067, 0.00067, 0.00067, 0.00067, 0.00067, 0.00068, 0.00067, 0.00068, 0.00067, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00066, 0.00067, 0.00067, 0.00067, 0.00067, 0.00066, 0.00066, 0.00067, 0.00067, 0.00067, 0.00067, 0.00068, 0.00066, 0.00067, 0.00067, 0.00067, 0.00067, 0.00067, 0.00067, 0.00067, 0.00067, 0.00067, 0.00067, 0.00066, 0.00066, 0.00067, 0.00067, 0.00068, 0.00067, 0.00067, 0.00068, 0.00068, 0.00067, 0.00067, 0.00067, 0.00067, 0.00067, 0.00068, 0.00067, 0.00069, 0.00067, 0.00067, 0.00066, 0.00067, 0.00066, 0.00067, 0.00066]}, "optimizer-copy-main-to-model-params-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.0006, 0.00055, 0.00055, 0.00054, 0.00054, 0.00053, 0.00053, 0.00053, 0.00054, 0.00052, 0.00052, 0.00052, 0.00052, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00052, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00052, 0.00052, 0.00053, 0.00061, 0.00052, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00052, 0.00053, 0.00053, 0.00053, 0.00054, 0.00052, 0.00053, 0.00053, 0.00053, 0.00052, 0.00052, 0.00053, 0.00052, 0.00052, 0.00052, 0.00053, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00053, 0.00053, 0.00053, 0.00052, 0.00052, 0.00053, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00053, 0.00052, 0.00053, 0.00052, 0.00053, 0.00052, 0.00052, 0.00052, 0.00053, 0.00053, 0.00052, 0.00052, 0.00053, 0.00056, 0.00053, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00053, 0.00052, 0.00052, 0.00052, 0.00052, 0.00053, 0.00053, 0.00053, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00053, 0.00052, 0.00055, 0.00052, 0.00052, 0.00052, 0.00053, 0.00053, 0.00052, 0.00053, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00053, 0.00053, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00053, 0.00053, 0.00052, 0.00052, 0.00052, 0.00053, 0.00053, 0.00052, 0.00052, 0.00052, 0.00052, 0.00052, 0.00053, 0.00052, 0.00054, 0.00053, 0.00052, 0.00052, 0.00052, 0.00053, 0.00052, 0.00052, 0.00052, 0.00053, 0.00052, 0.00053, 0.00052, 0.00052, 0.00053, 0.00052, 0.00052, 0.00053, 0.00052, 0.00052, 0.00053, 0.00053, 0.00052, 0.00052, 0.00052, 0.00053, 0.00054, 0.00054, 0.00054, 0.00053, 0.00054, 0.00053, 0.00054, 0.00053, 0.00054, 0.00053, 0.00054, 0.00053, 0.00053, 0.00054, 0.00053, 0.00053, 0.00053, 0.00053, 0.00054, 0.00054, 0.00054, 0.00054, 0.00054, 0.00053, 0.00054, 0.00054, 0.00054, 0.00053, 0.00056, 0.00054, 0.00054, 0.00053, 0.00054, 0.00054, 0.00053, 0.00054, 0.00053, 0.00054, 0.00054, 0.00054, 0.00053, 0.00053, 0.00053, 0.00053, 0.00054, 0.00053, 0.00054, 0.00053, 0.00053, 0.00053, 0.00055, 0.00053, 0.00054, 0.00054, 0.00054, 0.00053, 0.00054, 0.00054, 0.00054, 0.00053, 0.00055, 0.00053, 0.00054, 0.00053, 0.00053, 0.00054, 0.00055, 0.00053, 0.00055, 0.00054, 0.00054, 0.00054, 0.00054, 0.00053, 0.00054, 0.00054, 0.00054, 0.00054, 0.00055, 0.00054, 0.00054, 0.00054, 0.00054, 0.00054, 0.00054, 0.00054, 0.00054, 0.00053, 0.00054, 0.00054, 0.00054, 0.00053, 0.00054, 0.00053, 0.00054, 0.00053, 0.00054, 0.00053, 0.00053, 0.00053, 0.00054, 0.00053, 0.00053, 0.00053, 0.00054, 0.00053, 0.00053, 0.00053, 0.00053, 0.00054, 0.00053, 0.00053, 0.00053, 0.00053, 0.00054, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00054, 0.00054, 0.00053, 0.00053, 0.00054, 0.00054, 0.00054, 0.00054, 0.00054, 0.00054, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00054, 0.00054, 0.00053, 0.00053, 0.00055, 0.00053, 0.00054, 0.00053, 0.00054, 0.00054, 0.00053, 0.00053, 0.00054, 0.00053, 0.00053, 0.00054, 0.00054, 0.00054, 0.00054, 0.00053, 0.00053, 0.00054, 0.00053, 0.00053, 0.00054, 0.00054, 0.00054, 0.00054, 0.00054, 0.00053, 0.00054, 0.00053, 0.00054, 0.00053, 0.00054, 0.00053, 0.00053, 0.00054, 0.00054, 0.00053, 0.00054, 0.00054, 0.00054, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00053, 0.00054, 0.00053, 0.00054, 0.00053, 0.00053, 0.00054, 0.00053, 0.00054, 0.00053, 0.00053, 0.00053, 0.00053, 0.00054, 0.0006]}, "optimizer-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.12049, 0.00501, 0.00496, 0.00513, 0.00494, 0.00512, 0.00493, 0.00495, 0.00494, 0.00491, 0.00493, 0.00491, 0.00494, 0.00492, 0.00498, 0.00492, 0.0049, 0.00495, 0.00492, 0.00497, 0.00492, 0.00491, 0.00492, 0.00492, 0.00492, 0.00491, 0.00496, 0.00498, 0.00494, 0.00491, 0.0049, 0.00492, 0.00494, 0.00492, 0.00491, 0.00497, 0.00492, 0.00491, 0.00492, 0.00493, 0.00493, 0.00491, 0.00492, 0.00494, 0.00492, 0.00556, 0.00493, 0.00491, 0.00512, 0.00512, 0.00492, 0.00493, 0.00494, 0.0049, 0.00494, 0.00495, 0.00496, 0.00491, 0.00491, 0.00496, 0.00492, 0.00493, 0.00512, 0.00493, 0.00493, 0.00494, 0.00491, 0.0049, 0.00491, 0.00496, 0.00492, 0.0049, 0.00489, 0.00495, 0.00491, 0.00488, 0.00493, 0.00491, 0.0049, 0.0049, 0.00526, 0.00491, 0.00503, 0.0049, 0.00519, 0.00488, 0.00492, 0.00491, 0.0049, 0.00491, 0.00489, 0.00491, 0.0049, 0.00487, 0.00489, 0.0049, 0.00489, 0.00539, 0.00473, 0.00548, 0.00489, 0.00551, 0.0049, 0.00493, 0.00471, 0.00529, 0.00491, 0.0049, 0.00491, 0.00489, 0.00522, 0.00479, 0.00492, 0.00492, 0.00503, 0.0049, 0.0048, 0.0049, 0.00492, 0.00494, 0.00475, 0.0049, 0.00498, 0.0049, 0.0049, 0.00489, 0.0049, 0.00536, 0.00494, 0.00492, 0.00474, 0.00491, 0.0049, 0.00491, 0.00516, 0.00489, 0.00491, 0.0049, 0.00492, 0.00493, 0.00506, 0.00489, 0.00489, 0.00491, 0.00534, 0.00497, 0.00488, 0.00496, 0.00493, 0.00489, 0.00494, 0.0049, 0.00493, 0.00492, 0.00478, 0.00489, 0.0049, 0.00501, 0.00493, 0.00496, 0.0049, 0.00496, 0.00496, 0.00496, 0.00492, 0.00494, 0.00516, 0.00496, 0.00497, 0.00495, 0.00494, 0.00494, 0.00493, 0.00496, 0.00494, 0.0051, 0.00495, 0.00495, 0.00493, 0.00492, 0.00495, 0.00493, 0.00498, 0.00491, 0.00494, 0.00492, 0.00496, 0.00491, 0.00491, 0.00493, 0.00492, 0.0049, 0.005, 0.00491, 0.00498, 0.00494, 0.00489, 0.00494, 0.00496, 0.00491, 0.00501, 0.00504, 0.00502, 0.00501, 0.00506, 0.00508, 0.00502, 0.00501, 0.00497, 0.00496, 0.005, 0.005, 0.00498, 0.00504, 0.00502, 0.00497, 0.00511, 0.00499, 0.00502, 0.00502, 0.00535, 0.00532, 0.00503, 0.00507, 0.005, 0.00501, 0.005, 0.00499, 0.00499, 0.00538, 0.00498, 0.00502, 0.00499, 0.00505, 0.00503, 0.00497, 0.00504, 0.00493, 0.00495, 0.00499, 0.00529, 0.00499, 0.00499, 0.00502, 0.00499, 0.00504, 0.00497, 0.00502, 0.005, 0.00501, 0.00503, 0.00504, 0.00496, 0.00502, 0.00502, 0.00501, 0.00503, 0.005, 0.00501, 0.00502, 0.00495, 0.00563, 0.00504, 0.005, 0.00496, 0.00494, 0.00501, 0.005, 0.00499, 0.0054, 0.00512, 0.00507, 0.00502, 0.005, 0.00501, 0.005, 0.00499, 0.00498, 0.00504, 0.00503, 0.00499, 0.00501, 0.00511, 0.00502, 0.00506, 0.00502, 0.00501, 0.00499, 0.00535, 0.00498, 0.00501, 0.00499, 0.00494, 0.00493, 0.00496, 0.00494, 0.00496, 0.00495, 0.00495, 0.00494, 0.00498, 0.00495, 0.00498, 0.00498, 0.00495, 0.005, 0.00492, 0.00493, 0.00494, 0.00492, 0.00498, 0.00494, 0.00496, 0.00495, 0.00497, 0.00506, 0.00494, 0.00497, 0.00498, 0.00495, 0.00494, 0.00495, 0.00497, 0.005, 0.00512, 0.00495, 0.00495, 0.00497, 0.00493, 0.00495, 0.00494, 0.00498, 0.00495, 0.00509, 0.005, 0.00498, 0.00493, 0.00494, 0.00496, 0.00495, 0.00497, 0.00495, 0.00495, 0.00496, 0.00491, 0.00494, 0.00498, 0.00494, 0.00494, 0.00495, 0.00496, 0.00495, 0.00501, 0.00495, 0.00508, 0.00493, 0.00505, 0.00493, 0.00494, 0.00495, 0.00495, 0.00496, 0.00501, 0.00497, 0.00499, 0.00499, 0.00499, 0.00495, 0.00494, 0.00498, 0.00498, 0.00498, 0.00497, 0.00499, 0.00499, 0.00497, 0.00494, 0.00495, 0.00497, 0.00497, 0.00496, 0.00496, 0.00496, 0.00501, 0.00501, 0.00497, 0.00503, 0.00498, 0.00498, 0.0051, 0.00507, 0.005, 0.00498, 0.00497, 0.00499, 0.00495, 0.00494, 0.00496, 0.00495, 0.00502]}, "learning-rate": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05]}, "learning-rate vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05]}, "batch-size": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0]}, "batch-size vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0]}, "lm loss": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [[10.85923, 10.87023, 10.85489, 10.80333, 10.64103, 10.62632, 10.41615, 10.12834, 9.92596, 9.82486, 9.56936, 9.84047, 9.86936, 9.61428, 9.77592, 9.5009, 9.45233, 9.6411, 9.38016, 9.32632, 9.23845, 9.14183, 9.1729, 8.99275, 9.18807, 9.05765, 9.15474, 9.16451, 9.29849, 8.98678, 8.93052, 9.04732, 9.04618, 8.65655, 8.71669, 8.75537, 8.68517, 8.73662, 8.66118, 8.76495, 8.66219, 8.84922, 8.83085, 8.49818, 8.38745, 8.42836, 8.49044, 8.382, 8.43016, 8.57741, 8.36339, 8.18962, 8.224, 8.21853, 8.26289, 7.90907, 8.08969, 7.88743, 8.2399, 8.22485, 7.99855, 7.957, 7.912, 7.73262, 7.73338, 7.63664, 7.50898, 7.901, 7.6936, 7.44837, 7.7358, 7.76377, 7.53817, 7.29824, 7.45144, 7.33385, 7.46316, 7.22539, 7.63728, 7.27958, 7.35368, 7.21218, 7.21575, 7.42215, 7.17602, 7.28245, 7.00192, 7.00469, 7.03971, 7.13978, 6.82475, 6.98931, 7.09285, 7.00639, 6.88033, 6.76325, 7.00029, 7.06554, 6.71236, 6.58726, 6.73592, 6.74949, 6.73975, 6.74439, 6.66212, 6.41149, 6.64232, 6.62291, 6.45022, 6.63291, 6.74866, 6.61138, 6.72821, 6.69582, 6.62652, 6.51079, 6.60173, 6.40695, 6.6651, 6.24958, 6.25428, 6.30228, 6.39091, 6.35025, 6.45293, 6.29142, 6.33874, 6.23767, 6.20065, 6.39857, 6.32269, 6.3228, 6.16182, 6.15926, 6.23776, 6.38332, 6.19803, 6.14428, 6.17698, 6.10887, 6.05395, 6.06419, 6.25281, 6.40183, 6.25099, 6.29064, 6.08998, 6.17295, 5.99435, 6.02412, 5.94638, 6.23762, 6.18173, 5.95605, 5.77457, 6.11905, 5.84106, 6.09466, 5.7815, 6.15165, 6.14387, 6.09099, 5.92349, 6.11093, 5.94011, 6.18702, 5.88743, 5.79255, 5.77583, 5.68777, 6.00996, 5.99442, 6.0609, 5.8856, 6.03674, 5.964, 5.98984, 5.98577, 5.9438, 5.83404, 5.94515, 5.61197, 5.6964, 5.88652, 5.84113, 5.86014, 5.75727, 5.83814, 5.72107, 5.55799, 5.71863, 5.62698, 5.83073, 5.60536, 5.70755, 5.71315, 5.89651, 5.64286, 5.84706, 5.73871, 5.86823, 5.33053, 5.89671, 5.87127, 5.8562, 5.41227, 5.41025, 5.62486, 5.59271, 5.48387, 5.57354, 5.66953, 5.47502, 5.7438, 5.50731, 5.58968, 5.62227, 5.62105, 5.51021, 5.62193, 5.67201, 5.68247, 5.58859, 5.6615, 5.3736, 5.68112, 5.62447, 5.42761, 5.5852, 5.6344, 5.55235, 5.34483, 5.53696, 5.49184, 5.48457, 5.3781, 5.55465, 5.60886, 5.3922, 5.52851, 5.48934, 5.33658, 5.50741, 5.41226, 5.44624, 5.32132, 5.07087, 5.48264, 5.57109, 5.71529, 5.41689, 5.60753, 5.64089, 5.23456, 5.27636, 5.39623, 5.3984, 5.32972, 5.50051, 5.18915, 5.30774, 5.24961, 5.37609, 5.26117, 5.44966, 5.54003, 5.31448, 5.43684, 5.34004, 5.075, 5.31082, 5.25819, 5.30818, 5.1128, 5.27999, 5.26894, 5.47687, 5.16136, 5.27097, 5.21148, 5.36261, 4.98578, 4.92082, 5.32826, 5.39137, 5.22964, 5.3205, 5.1092, 5.15998, 5.26261, 5.0687, 5.26609, 5.07169, 5.34746, 5.24844, 5.14867, 5.24307, 5.04394, 5.31787, 5.05565, 5.02645, 5.14371, 5.11318, 5.27013, 5.15185, 5.27763, 5.09398, 5.09405, 5.24967, 5.32347, 5.2541, 5.19013, 5.1415, 5.28894, 4.94852, 5.20826, 5.09061, 5.30126, 5.17763, 5.1897, 5.11234, 4.9815, 4.98813, 5.22155, 5.30993, 5.09181, 5.05592, 4.91299, 5.13291, 5.11559, 4.92722, 5.33997, 5.0226, 5.10555, 5.1622, 5.00033, 5.06477, 5.07102, 5.00003, 5.08189, 5.1633, 4.97774, 5.18186, 4.9303, 4.92454, 5.06873, 4.99463, 4.91058, 4.77791, 4.94546, 5.12001, 5.01893, 5.02431, 5.33063, 4.96009, 4.99615, 5.04752, 4.80947, 4.73743, 4.99719, 5.03939, 4.87605, 4.95494, 5.04514, 5.02158, 4.81826, 4.89331, 4.90558, 4.82858, 4.7439, 5.01644, 4.75404, 5.21573, 4.787, 4.99317, 4.74039, 4.7886, 4.82294, 4.65004, 4.65685, 4.84811, 4.80756, 4.80216, 4.92915, 4.88364, 4.93397, 4.76931, 4.88652, 4.73528, 4.91493, 4.95747, 4.87675, 4.70743, 4.789, 4.8982, 4.71336, 4.86672, 4.69407, 4.69651, 4.64994]]}, "lm loss vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [10.85966, 10.87073, 10.85528, 10.80344, 10.64111, 10.62649, 10.41586, 10.12808, 9.92567, 9.82477, 9.56932, 9.84031, 9.86916, 9.61422, 9.77599, 9.50086, 9.45226, 9.6411, 9.38013, 9.32634, 9.2385, 9.14186, 9.17287, 8.9927, 9.18814, 9.05768, 9.15476, 9.16458, 9.29864, 8.98678, 8.93067, 9.0473, 9.04611, 8.65648, 8.71651, 8.75511, 8.6848, 8.73632, 8.66102, 8.76482, 8.66202, 8.84911, 8.83074, 8.49813, 8.38745, 8.42847, 8.49038, 8.38199, 8.43014, 8.57752, 8.36366, 8.18998, 8.22416, 8.21877, 8.26315, 7.90938, 8.09005, 7.88773, 8.24, 8.22485, 7.99867, 7.95704, 7.91177, 7.73255, 7.73299, 7.63614, 7.50837, 7.90027, 7.69288, 7.44749, 7.73489, 7.76278, 7.53675, 7.29662, 7.44913, 7.33262, 7.46188, 7.22442, 7.63668, 7.27892, 7.3525, 7.21173, 7.21816, 7.422, 7.17639, 7.28501, 7.00259, 7.00597, 7.03995, 7.14192, 6.82608, 6.98941, 7.09192, 7.00491, 6.87719, 6.75925, 6.994, 7.05741, 6.70391, 6.57997, 6.72686, 6.74254, 6.73498, 6.73924, 6.65693, 6.40819, 6.63945, 6.61998, 6.44777, 6.63026, 6.7458, 6.60872, 6.72566, 6.6941, 6.62478, 6.5113, 6.60016, 6.40683, 6.66647, 6.25038, 6.25487, 6.30344, 6.39244, 6.35319, 6.45279, 6.29501, 6.34432, 6.24122, 6.20479, 6.40226, 6.3298, 6.33253, 6.17365, 6.1703, 6.25122, 6.39707, 6.21313, 6.16095, 6.19193, 6.12904, 6.07716, 6.08434, 6.27156, 6.42116, 6.27092, 6.31502, 6.1099, 6.19051, 6.01202, 6.04186, 5.96572, 6.2566, 6.1994, 5.97238, 5.79066, 6.13517, 5.8567, 6.11381, 5.79621, 6.16806, 6.15725, 6.09481, 5.94172, 6.12313, 5.95406, 6.20205, 5.90266, 5.80426, 5.78673, 5.69691, 6.02057, 6.00205, 6.07073, 5.89354, 6.04415, 5.97229, 5.99763, 5.99201, 5.9504, 5.83989, 5.95152, 5.61741, 5.70128, 5.88995, 5.84414, 5.86222, 5.76021, 5.83835, 5.72362, 5.56328, 5.72206, 5.62699, 5.83296, 5.60473, 5.71241, 5.71399, 5.89863, 5.64481, 5.85045, 5.74116, 5.86786, 5.33069, 5.89739, 5.87147, 5.85621, 5.41402, 5.40885, 5.6244, 5.5909, 5.48288, 5.57328, 5.66993, 5.47325, 5.74532, 5.50733, 5.58951, 5.62335, 5.61873, 5.50712, 5.61686, 5.67259, 5.68325, 5.58652, 5.65724, 5.37154, 5.68206, 5.62545, 5.42293, 5.5898, 5.63487, 5.55215, 5.34318, 5.53918, 5.48775, 5.48384, 5.38046, 5.5524, 5.6054, 5.39011, 5.52269, 5.48564, 5.33339, 5.50751, 5.41235, 5.44463, 5.32284, 5.07354, 5.47834, 5.57158, 5.71691, 5.41899, 5.60533, 5.64283, 5.2342, 5.27417, 5.39872, 5.39954, 5.33267, 5.50546, 5.18598, 5.3031, 5.25146, 5.37886, 5.25856, 5.45542, 5.53656, 5.3141, 5.4389, 5.34171, 5.07715, 5.31356, 5.26151, 5.30932, 5.1132, 5.27888, 5.26913, 5.47802, 5.16411, 5.27179, 5.21046, 5.36047, 4.98558, 4.92161, 5.33001, 5.39104, 5.23106, 5.32226, 5.1108, 5.16307, 5.26011, 5.06878, 5.26621, 5.0712, 5.34447, 5.24947, 5.15197, 5.24511, 5.04213, 5.3173, 5.05677, 5.03031, 5.14366, 5.11315, 5.27152, 5.15384, 5.27818, 5.09471, 5.09718, 5.25022, 5.32221, 5.25368, 5.19177, 5.14141, 5.29041, 4.95105, 5.2074, 5.08987, 5.30215, 5.17471, 5.18799, 5.1137, 4.98327, 4.99184, 5.2222, 5.31185, 5.09737, 5.05507, 4.91447, 5.12386, 5.11467, 4.92535, 5.33586, 5.02667, 5.10506, 5.16491, 5.00221, 5.06296, 5.06915, 4.9949, 5.07922, 5.16029, 4.97927, 5.18201, 4.92792, 4.92204, 5.06399, 4.99471, 4.90735, 4.77765, 4.94535, 5.11795, 5.01969, 5.02225, 5.33057, 4.96058, 4.9931, 5.0457, 4.81181, 4.74328, 4.99687, 5.0383, 4.87423, 4.95276, 5.04325, 5.02264, 4.81956, 4.89599, 4.90754, 4.8294, 4.74438, 5.01179, 4.75262, 5.2095, 4.78557, 4.99344, 4.73813, 4.78739, 4.82401, 4.64885, 4.65631, 4.84474, 4.80822, 4.80327, 4.92878, 4.88473, 4.93264, 4.7706, 4.88531, 4.73767, 4.91524, 4.95719, 4.87814, 4.70608, 4.7878, 4.89822, 4.71172, 4.87123, 4.69258, 4.69633, 4.64631]}, "loss-scale": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]}, "loss-scale vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]}, "grad-norm": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [13.56517, 13.52183, 13.82389, 12.68199, 12.11513, 9.42628, 6.78009, 6.96682, 6.03524, 4.63457, 4.1513, 2.87067, 2.35463, 2.3279, 2.02459, 2.22441, 2.16108, 1.87618, 2.21105, 2.06296, 2.12729, 2.152, 2.00687, 2.2248, 1.98285, 2.1147, 1.92124, 1.92395, 1.94527, 2.15653, 2.0865, 1.94545, 1.87214, 2.15774, 2.14492, 2.10813, 1.99702, 1.84398, 1.93326, 1.73194, 2.15655, 1.83365, 1.74796, 1.87637, 1.87935, 1.82812, 1.70882, 1.75031, 1.75541, 1.56033, 1.72362, 1.80715, 1.77318, 1.81611, 1.66844, 1.80559, 1.7625, 1.84598, 1.62632, 1.48661, 1.64786, 1.45473, 1.77763, 1.80854, 1.64942, 1.65627, 1.70353, 1.60171, 1.44031, 1.72339, 1.43433, 1.37767, 1.68581, 1.37671, 1.40648, 1.61691, 1.50881, 1.38382, 1.44532, 1.27357, 1.36667, 1.33118, 1.30365, 1.39513, 1.39043, 1.4631, 1.55974, 1.45774, 1.22995, 1.11972, 1.09726, 1.20059, 1.10224, 1.31175, 1.01034, 1.30362, 1.38885, 1.05046, 0.94787, 1.76252, 1.11012, 1.2148, 1.71468, 1.62278, 0.95552, 1.16789, 1.17655, 1.03922, 1.21282, 1.1032, 0.98669, 0.95678, 1.1193, 1.05737, 1.01498, 1.16799, 0.97578, 1.42941, 1.13594, 1.05985, 0.9398, 1.10182, 1.02064, 1.3517, 1.44708, 2.04415, 1.69036, 1.40806, 1.38738, 1.3424, 0.99552, 1.67778, 1.38915, 1.16703, 1.21285, 1.27027, 1.08112, 1.56529, 1.11243, 1.55047, 1.88478, 1.49661, 1.24747, 1.30858, 1.0413, 1.79193, 1.1894, 1.10832, 1.14553, 1.37473, 1.12916, 1.19043, 1.55147, 1.14787, 0.9831, 1.97748, 1.30968, 1.75548, 1.42903, 1.47772, 1.63806, 1.08487, 1.3989, 1.02365, 1.24838, 1.43469, 1.42662, 1.30881, 1.20964, 1.49347, 1.21919, 1.05332, 1.18399, 1.38555, 1.13727, 1.36432, 1.2528, 1.17022, 1.32348, 1.07935, 1.19539, 1.48684, 1.19029, 1.2198, 1.81559, 1.52452, 1.79334, 1.66013, 1.20616, 1.67532, 1.19437, 1.28, 1.33364, 1.69679, 1.53842, 1.37202, 1.34387, 1.37081, 1.28649, 1.5618, 1.03326, 1.39685, 1.27238, 1.20598, 1.32922, 1.41054, 1.32813, 1.46075, 1.18533, 1.18314, 1.37783, 1.39264, 1.2322, 1.35301, 1.51994, 1.29479, 1.54145, 1.57876, 1.23038, 1.67935, 1.59903, 1.7688, 1.38891, 1.39714, 1.41056, 1.56263, 1.84649, 1.31226, 2.25632, 1.5966, 1.20159, 1.49708, 1.73963, 1.47932, 1.74434, 1.84578, 1.28148, 1.58712, 1.57826, 1.14575, 1.37743, 1.14726, 1.36495, 1.54092, 1.1998, 1.83908, 1.60608, 1.22735, 1.39352, 1.48052, 1.44922, 1.5986, 1.86828, 1.2133, 1.28534, 1.44591, 1.40707, 1.6217, 1.68123, 1.16996, 1.40545, 1.79994, 1.32408, 1.35454, 1.82216, 1.50619, 1.25331, 1.36593, 1.33067, 1.20379, 1.1715, 1.34612, 1.23828, 1.2249, 1.23199, 1.50931, 1.24187, 1.31666, 1.33544, 1.15247, 1.35164, 1.31814, 1.51121, 1.22179, 1.26518, 1.48248, 1.47105, 2.08081, 1.48841, 1.53234, 1.46321, 1.4755, 1.16048, 1.44268, 1.5642, 1.52523, 1.38495, 1.80119, 1.63483, 1.41261, 1.60553, 1.28802, 1.15347, 1.54912, 1.53753, 1.36296, 1.66631, 1.63888, 1.24348, 1.42956, 1.32686, 1.487, 1.7063, 1.383, 1.67566, 1.4665, 1.41433, 1.44807, 1.36307, 1.13744, 1.63129, 1.56395, 1.59787, 1.49857, 1.45091, 1.60777, 1.36633, 1.34096, 1.63579, 1.34741, 1.48819, 1.66258, 1.532, 1.46235, 1.36272, 1.36735, 1.33239, 1.3176, 1.2966, 1.56971, 1.31551, 1.50053, 1.27598, 1.29926, 1.5045, 1.39074, 1.41138, 1.40198, 1.46432, 1.38696, 1.52639, 1.55526, 1.4432, 1.27923, 1.48503, 1.17404, 1.20825, 1.60545, 1.81024, 1.35059, 1.28697, 1.50174, 1.46699, 1.33784, 1.08159, 1.61115, 1.46019, 1.37898, 1.35614, 1.65157, 1.46597, 1.60688, 1.72399, 1.30124, 1.44364, 1.32297, 1.13212, 1.45342, 1.38164, 1.21948, 1.26404, 1.33477, 1.30704, 1.51357, 1.26848, 1.55252, 1.33368, 1.41811, 1.47778, 1.31706, 1.20105, 1.48475, 1.28543, 1.46568, 1.42638, 1.25259, 1.60254, 1.36812, 1.3586, 1.15672]}, "grad-norm vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [13.56517, 13.52183, 13.82389, 12.68199, 12.11513, 9.42628, 6.78009, 6.96682, 6.03524, 4.63457, 4.1513, 2.87067, 2.35463, 2.3279, 2.02459, 2.22441, 2.16108, 1.87618, 2.21105, 2.06296, 2.12729, 2.152, 2.00687, 2.2248, 1.98285, 2.1147, 1.92124, 1.92395, 1.94527, 2.15653, 2.0865, 1.94545, 1.87214, 2.15774, 2.14492, 2.10813, 1.99702, 1.84398, 1.93326, 1.73194, 2.15655, 1.83365, 1.74796, 1.87637, 1.87935, 1.82812, 1.70882, 1.75031, 1.75541, 1.56033, 1.72362, 1.80715, 1.77318, 1.81611, 1.66844, 1.80559, 1.7625, 1.84598, 1.62632, 1.48661, 1.64786, 1.45473, 1.77763, 1.80854, 1.64942, 1.65627, 1.70353, 1.60171, 1.44031, 1.72339, 1.43433, 1.37767, 1.68581, 1.37671, 1.40648, 1.61691, 1.50881, 1.38382, 1.44532, 1.27357, 1.36667, 1.33118, 1.30365, 1.39513, 1.39043, 1.4631, 1.55974, 1.45774, 1.22995, 1.11972, 1.09726, 1.20059, 1.10224, 1.31175, 1.01034, 1.30362, 1.38885, 1.05046, 0.94787, 1.76252, 1.11012, 1.2148, 1.71468, 1.62278, 0.95552, 1.16789, 1.17655, 1.03922, 1.21282, 1.1032, 0.98669, 0.95678, 1.1193, 1.05737, 1.01498, 1.16799, 0.97578, 1.42941, 1.13594, 1.05985, 0.9398, 1.10182, 1.02064, 1.3517, 1.44708, 2.04415, 1.69036, 1.40806, 1.38738, 1.3424, 0.99552, 1.67778, 1.38915, 1.16703, 1.21285, 1.27027, 1.08112, 1.56529, 1.11243, 1.55047, 1.88478, 1.49661, 1.24747, 1.30858, 1.0413, 1.79193, 1.1894, 1.10832, 1.14553, 1.37473, 1.12916, 1.19043, 1.55147, 1.14787, 0.9831, 1.97748, 1.30968, 1.75548, 1.42903, 1.47772, 1.63806, 1.08487, 1.3989, 1.02365, 1.24838, 1.43469, 1.42662, 1.30881, 1.20964, 1.49347, 1.21919, 1.05332, 1.18399, 1.38555, 1.13727, 1.36432, 1.2528, 1.17022, 1.32348, 1.07935, 1.19539, 1.48684, 1.19029, 1.2198, 1.81559, 1.52452, 1.79334, 1.66013, 1.20616, 1.67532, 1.19437, 1.28, 1.33364, 1.69679, 1.53842, 1.37202, 1.34387, 1.37081, 1.28649, 1.5618, 1.03326, 1.39685, 1.27238, 1.20598, 1.32922, 1.41054, 1.32813, 1.46075, 1.18533, 1.18314, 1.37783, 1.39264, 1.2322, 1.35301, 1.51994, 1.29479, 1.54145, 1.57876, 1.23038, 1.67935, 1.59903, 1.7688, 1.38891, 1.39714, 1.41056, 1.56263, 1.84649, 1.31226, 2.25632, 1.5966, 1.20159, 1.49708, 1.73963, 1.47932, 1.74434, 1.84578, 1.28148, 1.58712, 1.57826, 1.14575, 1.37743, 1.14726, 1.36495, 1.54092, 1.1998, 1.83908, 1.60608, 1.22735, 1.39352, 1.48052, 1.44922, 1.5986, 1.86828, 1.2133, 1.28534, 1.44591, 1.40707, 1.6217, 1.68123, 1.16996, 1.40545, 1.79994, 1.32408, 1.35454, 1.82216, 1.50619, 1.25331, 1.36593, 1.33067, 1.20379, 1.1715, 1.34612, 1.23828, 1.2249, 1.23199, 1.50931, 1.24187, 1.31666, 1.33544, 1.15247, 1.35164, 1.31814, 1.51121, 1.22179, 1.26518, 1.48248, 1.47105, 2.08081, 1.48841, 1.53234, 1.46321, 1.4755, 1.16048, 1.44268, 1.5642, 1.52523, 1.38495, 1.80119, 1.63483, 1.41261, 1.60553, 1.28802, 1.15347, 1.54912, 1.53753, 1.36296, 1.66631, 1.63888, 1.24348, 1.42956, 1.32686, 1.487, 1.7063, 1.383, 1.67566, 1.4665, 1.41433, 1.44807, 1.36307, 1.13744, 1.63129, 1.56395, 1.59787, 1.49857, 1.45091, 1.60777, 1.36633, 1.34096, 1.63579, 1.34741, 1.48819, 1.66258, 1.532, 1.46235, 1.36272, 1.36735, 1.33239, 1.3176, 1.2966, 1.56971, 1.31551, 1.50053, 1.27598, 1.29926, 1.5045, 1.39074, 1.41138, 1.40198, 1.46432, 1.38696, 1.52639, 1.55526, 1.4432, 1.27923, 1.48503, 1.17404, 1.20825, 1.60545, 1.81024, 1.35059, 1.28697, 1.50174, 1.46699, 1.33784, 1.08159, 1.61115, 1.46019, 1.37898, 1.35614, 1.65157, 1.46597, 1.60688, 1.72399, 1.30124, 1.44364, 1.32297, 1.13212, 1.45342, 1.38164, 1.21948, 1.26404, 1.33477, 1.30704, 1.51357, 1.26848, 1.55252, 1.33368, 1.41811, 1.47778, 1.31706, 1.20105, 1.48475, 1.28543, 1.46568, 1.42638, 1.25259, 1.60254, 1.36812, 1.3586, 1.15672]}, "num-zeros": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [74.0, 66.0, 60.0, 92.0, 66.0, 92.0, 104.0, 103.0, 99.0, 124.0, 96.0, 151.0, 118.0, 149.0, 190.0, 162.0, 160.0, 183.0, 169.0, 192.0, 161.0, 189.0, 179.0, 160.0, 174.0, 142.0, 205.0, 175.0, 151.0, 152.0, 142.0, 147.0, 141.0, 142.0, 153.0, 136.0, 181.0, 223.0, 189.0, 182.0, 152.0, 185.0, 170.0, 146.0, 191.0, 178.0, 181.0, 178.0, 160.0, 186.0, 204.0, 171.0, 210.0, 153.0, 169.0, 174.0, 161.0, 146.0, 229.0, 200.0, 195.0, 216.0, 178.0, 172.0, 197.0, 240.0, 211.0, 188.0, 228.0, 200.0, 244.0, 216.0, 163.0, 226.0, 205.0, 191.0, 215.0, 207.0, 254.0, 225.0, 236.0, 238.0, 186.0, 234.0, 202.0, 180.0, 135.0, 203.0, 183.0, 215.0, 205.0, 204.0, 203.0, 187.0, 194.0, 186.0, 185.0, 219.0, 179.0, 145.0, 184.0, 155.0, 171.0, 147.0, 159.0, 163.0, 177.0, 151.0, 151.0, 172.0, 174.0, 157.0, 166.0, 160.0, 159.0, 151.0, 143.0, 110.0, 167.0, 149.0, 151.0, 159.0, 141.0, 148.0, 104.0, 139.0, 124.0, 166.0, 147.0, 125.0, 156.0, 132.0, 147.0, 126.0, 157.0, 137.0, 135.0, 138.0, 110.0, 132.0, 133.0, 116.0, 115.0, 137.0, 146.0, 122.0, 133.0, 106.0, 126.0, 112.0, 103.0, 105.0, 98.0, 117.0, 119.0, 86.0, 108.0, 103.0, 128.0, 124.0, 98.0, 72.0, 119.0, 116.0, 106.0, 130.0, 126.0, 109.0, 117.0, 85.0, 115.0, 117.0, 127.0, 111.0, 98.0, 108.0, 119.0, 136.0, 118.0, 114.0, 128.0, 109.0, 118.0, 119.0, 91.0, 95.0, 91.0, 89.0, 94.0, 121.0, 117.0, 94.0, 114.0, 94.0, 136.0, 89.0, 83.0, 92.0, 125.0, 92.0, 119.0, 119.0, 134.0, 107.0, 102.0, 134.0, 88.0, 101.0, 89.0, 121.0, 104.0, 104.0, 98.0, 118.0, 108.0, 111.0, 118.0, 87.0, 105.0, 92.0, 126.0, 108.0, 95.0, 82.0, 92.0, 106.0, 100.0, 84.0, 99.0, 116.0, 109.0, 87.0, 103.0, 95.0, 85.0, 111.0, 111.0, 112.0, 110.0, 94.0, 126.0, 94.0, 110.0, 126.0, 104.0, 97.0, 108.0, 104.0, 106.0, 121.0, 125.0, 75.0, 101.0, 113.0, 106.0, 118.0, 96.0, 112.0, 114.0, 109.0, 89.0, 93.0, 120.0, 89.0, 89.0, 82.0, 106.0, 124.0, 118.0, 106.0, 114.0, 121.0, 115.0, 82.0, 98.0, 105.0, 120.0, 115.0, 114.0, 118.0, 89.0, 116.0, 104.0, 112.0, 125.0, 100.0, 129.0, 95.0, 108.0, 85.0, 112.0, 104.0, 124.0, 119.0, 90.0, 85.0, 115.0, 97.0, 104.0, 117.0, 124.0, 98.0, 108.0, 106.0, 87.0, 96.0, 104.0, 125.0, 117.0, 108.0, 103.0, 96.0, 78.0, 115.0, 114.0, 84.0, 111.0, 108.0, 121.0, 112.0, 108.0, 87.0, 99.0, 110.0, 110.0, 138.0, 93.0, 101.0, 89.0, 122.0, 98.0, 96.0, 123.0, 106.0, 125.0, 139.0, 121.0, 124.0, 89.0, 124.0, 107.0, 108.0, 102.0, 106.0, 122.0, 97.0, 120.0, 102.0, 92.0, 123.0, 96.0, 108.0, 113.0, 123.0, 122.0, 121.0, 103.0, 128.0, 111.0, 106.0, 122.0, 104.0, 92.0, 94.0, 124.0, 118.0, 120.0, 125.0, 123.0, 112.0, 101.0, 94.0, 96.0, 111.0, 99.0, 104.0, 111.0, 108.0, 112.0, 127.0, 108.0, 122.0, 133.0, 112.0, 104.0, 93.0, 114.0, 111.0, 139.0, 117.0, 117.0, 103.0, 129.0, 120.0, 118.0, 113.0, 116.0, 109.0, 129.0, 121.0]}, "num-zeros vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [78.0, 71.0, 69.0, 77.0, 83.0, 93.0, 106.0, 92.0, 92.0, 132.0, 100.0, 151.0, 124.0, 174.0, 156.0, 150.0, 169.0, 195.0, 167.0, 147.0, 152.0, 152.0, 200.0, 189.0, 169.0, 153.0, 197.0, 164.0, 147.0, 172.0, 144.0, 157.0, 169.0, 165.0, 146.0, 179.0, 172.0, 212.0, 186.0, 196.0, 171.0, 138.0, 152.0, 197.0, 156.0, 167.0, 212.0, 178.0, 187.0, 180.0, 190.0, 159.0, 176.0, 163.0, 179.0, 191.0, 150.0, 150.0, 227.0, 225.0, 197.0, 184.0, 184.0, 199.0, 214.0, 235.0, 186.0, 197.0, 214.0, 222.0, 193.0, 241.0, 159.0, 264.0, 193.0, 187.0, 201.0, 208.0, 227.0, 223.0, 225.0, 212.0, 231.0, 219.0, 202.0, 196.0, 178.0, 182.0, 185.0, 210.0, 201.0, 198.0, 213.0, 214.0, 205.0, 161.0, 183.0, 193.0, 198.0, 178.0, 190.0, 166.0, 137.0, 154.0, 183.0, 150.0, 165.0, 166.0, 127.0, 174.0, 160.0, 171.0, 188.0, 172.0, 159.0, 152.0, 151.0, 127.0, 137.0, 145.0, 172.0, 135.0, 151.0, 158.0, 141.0, 113.0, 114.0, 93.0, 113.0, 128.0, 148.0, 125.0, 114.0, 127.0, 121.0, 117.0, 146.0, 116.0, 148.0, 137.0, 108.0, 114.0, 129.0, 141.0, 130.0, 107.0, 113.0, 126.0, 130.0, 102.0, 127.0, 110.0, 108.0, 109.0, 112.0, 65.0, 98.0, 84.0, 105.0, 108.0, 95.0, 135.0, 103.0, 123.0, 101.0, 102.0, 101.0, 117.0, 109.0, 106.0, 123.0, 114.0, 102.0, 88.0, 131.0, 104.0, 116.0, 108.0, 142.0, 118.0, 121.0, 115.0, 118.0, 115.0, 106.0, 119.0, 105.0, 84.0, 106.0, 91.0, 120.0, 114.0, 140.0, 96.0, 85.0, 100.0, 114.0, 103.0, 153.0, 88.0, 120.0, 96.0, 122.0, 111.0, 89.0, 107.0, 111.0, 97.0, 128.0, 103.0, 123.0, 90.0, 94.0, 82.0, 100.0, 109.0, 112.0, 104.0, 119.0, 90.0, 77.0, 114.0, 82.0, 103.0, 104.0, 104.0, 97.0, 127.0, 67.0, 99.0, 126.0, 90.0, 84.0, 109.0, 94.0, 97.0, 107.0, 113.0, 127.0, 100.0, 115.0, 102.0, 96.0, 116.0, 125.0, 102.0, 91.0, 126.0, 114.0, 101.0, 113.0, 110.0, 96.0, 126.0, 121.0, 99.0, 104.0, 108.0, 86.0, 143.0, 120.0, 83.0, 115.0, 92.0, 73.0, 113.0, 117.0, 111.0, 93.0, 106.0, 131.0, 93.0, 121.0, 109.0, 108.0, 115.0, 117.0, 116.0, 105.0, 110.0, 103.0, 112.0, 85.0, 118.0, 126.0, 119.0, 120.0, 104.0, 112.0, 111.0, 108.0, 107.0, 126.0, 123.0, 100.0, 81.0, 101.0, 106.0, 93.0, 109.0, 104.0, 131.0, 134.0, 98.0, 105.0, 129.0, 83.0, 87.0, 128.0, 116.0, 114.0, 111.0, 94.0, 114.0, 91.0, 97.0, 93.0, 116.0, 135.0, 122.0, 111.0, 126.0, 107.0, 107.0, 101.0, 82.0, 120.0, 142.0, 124.0, 120.0, 124.0, 122.0, 97.0, 96.0, 107.0, 102.0, 123.0, 115.0, 126.0, 116.0, 122.0, 115.0, 107.0, 111.0, 95.0, 93.0, 113.0, 117.0, 101.0, 110.0, 126.0, 113.0, 112.0, 127.0, 138.0, 118.0, 133.0, 94.0, 105.0, 119.0, 121.0, 122.0, 102.0, 98.0, 119.0, 103.0, 108.0, 134.0, 116.0, 107.0, 105.0, 99.0, 99.0, 117.0, 106.0, 133.0, 108.0, 110.0, 99.0, 140.0, 107.0, 104.0, 114.0, 112.0, 117.0, 106.0, 105.0, 92.0, 111.0, 99.0, 124.0, 101.0, 102.0, 144.0, 129.0, 122.0, 110.0, 116.0, 123.0, 136.0]}, "params-norm": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [179.94354, 179.94354, 179.94354, 179.94353, 179.94351, 179.94351, 179.9435, 179.94337, 179.94319, 179.94301, 179.94168, 179.94092, 179.94034, 179.9382, 179.93718, 179.93637, 179.93611, 179.93633, 179.93683, 179.93695, 179.93684, 179.93649, 179.9361, 179.93663, 179.93771, 179.93913, 179.94032, 179.94113, 179.94214, 179.94365, 179.94586, 179.94824, 179.95052, 179.95296, 179.95572, 179.95921, 179.96291, 179.96681, 179.97093, 179.97545, 179.98062, 179.98616, 179.99197, 179.99846, 180.00552, 180.01314, 180.02119, 180.03004, 180.0396, 180.05011, 180.06131, 180.07315, 180.08542, 180.0985, 180.11215, 180.12645, 180.14087, 180.15598, 180.17198, 180.18895, 180.20711, 180.22621, 180.24666, 180.26831, 180.28981, 180.31268, 180.33565, 180.35945, 180.38472, 180.41133, 180.43765, 180.46451, 180.49187, 180.51939, 180.54758, 180.57634, 180.60477, 180.63396, 180.66389, 180.69472, 180.72603, 180.7572, 180.78957, 180.823, 180.85631, 180.88991, 180.92371, 180.95706, 180.99092, 181.02626, 181.06326, 181.10162, 181.1391, 181.17641, 181.21402, 181.25211, 181.28955, 181.32634, 181.36447, 181.40189, 181.4381, 181.47331, 181.50807, 181.54071, 181.57346, 181.60866, 181.64577, 181.68417, 181.72168, 181.75914, 181.79767, 181.83748, 181.87747, 181.91742, 181.95695, 181.99832, 182.03812, 182.07738, 182.11449, 182.15204, 182.19035, 182.22978, 182.2695, 182.31001, 182.34891, 182.38696, 182.42218, 182.45525, 182.48941, 182.52226, 182.55621, 182.58896, 182.62086, 182.65288, 182.68657, 182.72272, 182.76212, 182.80115, 182.83951, 182.87524, 182.90919, 182.94313, 182.97842, 183.01477, 183.0529, 183.09117, 183.127, 183.16306, 183.20122, 183.24178, 183.28111, 183.32036, 183.35971, 183.3998, 183.43983, 183.47787, 183.51186, 183.54558, 183.57816, 183.6123, 183.64774, 183.68333, 183.72012, 183.75874, 183.79793, 183.83867, 183.87993, 183.92157, 183.96465, 184.00539, 184.04436, 184.0843, 184.12569, 184.16653, 184.20705, 184.24741, 184.28691, 184.32756, 184.36906, 184.41148, 184.45378, 184.4951, 184.53712, 184.57993, 184.62045, 184.65775, 184.69293, 184.72659, 184.76007, 184.79503, 184.83018, 184.86899, 184.90979, 184.95056, 184.99091, 185.03053, 185.07204, 185.11502, 185.15868, 185.20329, 185.24709, 185.29115, 185.33409, 185.37717, 185.4185, 185.45804, 185.49718, 185.53632, 185.57599, 185.61728, 185.65776, 185.69963, 185.74083, 185.78281, 185.82603, 185.86871, 185.91023, 185.94936, 185.98782, 186.0262, 186.06454, 186.10416, 186.14491, 186.1852, 186.2245, 186.26433, 186.30334, 186.34256, 186.38142, 186.41753, 186.45586, 186.49515, 186.5363, 186.57649, 186.61508, 186.65221, 186.6895, 186.72816, 186.76711, 186.80779, 186.84801, 186.88885, 186.93158, 186.97491, 187.01726, 187.06096, 187.10196, 187.14183, 187.18462, 187.22882, 187.27315, 187.31848, 187.36339, 187.40767, 187.45337, 187.49886, 187.54268, 187.58609, 187.62961, 187.67044, 187.71268, 187.75528, 187.79819, 187.84183, 187.88416, 187.92462, 187.96719, 188.0098, 188.0549, 188.10202, 188.14798, 188.19414, 188.23969, 188.28632, 188.33499, 188.38423, 188.43146, 188.47794, 188.52431, 188.57013, 188.61865, 188.66565, 188.71187, 188.75861, 188.80621, 188.85393, 188.90173, 188.94839, 188.99448, 189.04036, 189.08531, 189.13077, 189.17767, 189.22517, 189.27315, 189.32074, 189.36909, 189.41704, 189.46393, 189.5119, 189.5609, 189.61021, 189.66124, 189.71246, 189.76324, 189.81259, 189.86185, 189.91013, 189.96013, 190.0108, 190.061, 190.11232, 190.1635, 190.21367, 190.2627, 190.31346, 190.36389, 190.41492, 190.46727, 190.51939, 190.57338, 190.62749, 190.68044, 190.73311, 190.78491, 190.83577, 190.8877, 190.93848, 190.98965, 191.04053, 191.09221, 191.1438, 191.19595, 191.24683, 191.29836, 191.35121, 191.40576, 191.45865, 191.51144, 191.56329, 191.61534, 191.66661, 191.71944, 191.77365, 191.82733, 191.88013, 191.93358, 191.98837, 192.04231, 192.09724, 192.15228, 192.20715, 192.26242, 192.32021, 192.37662, 192.4319, 192.48772, 192.54413, 192.59987, 192.65529, 192.71152, 192.76802, 192.82562, 192.88312, 192.94026, 192.99599, 193.05467, 193.11278, 193.17015, 193.22783, 193.28326, 193.33839, 193.39395, 193.44897, 193.50545, 193.563, 193.61928, 193.67555, 193.73364, 193.79195, 193.85016, 193.90939, 193.96805, 194.02667, 194.08534, 194.14226, 194.20026, 194.25986, 194.32065, 194.38155, 194.44293, 194.50323, 194.56407, 194.62587, 194.68752, 194.74759, 194.80595, 194.86389, 194.92307, 194.98349]}, "params-norm vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [179.94354, 179.94354, 179.94354, 179.94353, 179.94351, 179.94351, 179.9435, 179.94337, 179.94319, 179.94301, 179.94168, 179.94092, 179.94034, 179.9382, 179.93718, 179.93637, 179.93611, 179.93633, 179.93683, 179.93695, 179.93684, 179.93649, 179.9361, 179.93663, 179.93771, 179.93913, 179.94032, 179.94113, 179.94214, 179.94365, 179.94586, 179.94824, 179.95052, 179.95296, 179.95572, 179.95921, 179.96291, 179.96681, 179.97093, 179.97545, 179.98062, 179.98616, 179.99197, 179.99846, 180.00552, 180.01314, 180.02119, 180.03004, 180.0396, 180.05011, 180.06131, 180.07315, 180.08542, 180.0985, 180.11215, 180.12645, 180.14087, 180.15598, 180.17198, 180.18895, 180.20711, 180.22621, 180.24666, 180.26831, 180.28981, 180.31268, 180.33565, 180.35945, 180.38472, 180.41133, 180.43765, 180.46451, 180.49187, 180.51939, 180.54758, 180.57634, 180.60477, 180.63396, 180.66389, 180.69472, 180.72603, 180.7572, 180.78957, 180.823, 180.85631, 180.88991, 180.92371, 180.95706, 180.99092, 181.02626, 181.06326, 181.10162, 181.1391, 181.17641, 181.21402, 181.25211, 181.28955, 181.32634, 181.36447, 181.40189, 181.4381, 181.47331, 181.50807, 181.54071, 181.57346, 181.60866, 181.64577, 181.68417, 181.72168, 181.75914, 181.79767, 181.83748, 181.87747, 181.91742, 181.95695, 181.99832, 182.03812, 182.07738, 182.11449, 182.15204, 182.19035, 182.22978, 182.2695, 182.31001, 182.34891, 182.38696, 182.42218, 182.45525, 182.48941, 182.52226, 182.55621, 182.58896, 182.62086, 182.65288, 182.68657, 182.72272, 182.76212, 182.80115, 182.83951, 182.87524, 182.90919, 182.94313, 182.97842, 183.01477, 183.0529, 183.09117, 183.127, 183.16306, 183.20122, 183.24178, 183.28111, 183.32036, 183.35971, 183.3998, 183.43983, 183.47787, 183.51186, 183.54558, 183.57816, 183.6123, 183.64774, 183.68333, 183.72012, 183.75874, 183.79793, 183.83867, 183.87993, 183.92157, 183.96465, 184.00539, 184.04436, 184.0843, 184.12569, 184.16653, 184.20705, 184.24741, 184.28691, 184.32756, 184.36906, 184.41148, 184.45378, 184.4951, 184.53712, 184.57993, 184.62045, 184.65775, 184.69293, 184.72659, 184.76007, 184.79503, 184.83018, 184.86899, 184.90979, 184.95056, 184.99091, 185.03053, 185.07204, 185.11502, 185.15868, 185.20329, 185.24709, 185.29115, 185.33409, 185.37717, 185.4185, 185.45804, 185.49718, 185.53632, 185.57599, 185.61728, 185.65776, 185.69963, 185.74083, 185.78281, 185.82603, 185.86871, 185.91023, 185.94936, 185.98782, 186.0262, 186.06454, 186.10416, 186.14491, 186.1852, 186.2245, 186.26433, 186.30334, 186.34256, 186.38142, 186.41753, 186.45586, 186.49515, 186.5363, 186.57649, 186.61508, 186.65221, 186.6895, 186.72816, 186.76711, 186.80779, 186.84801, 186.88885, 186.93158, 186.97491, 187.01726, 187.06096, 187.10196, 187.14183, 187.18462, 187.22882, 187.27315, 187.31848, 187.36339, 187.40767, 187.45337, 187.49886, 187.54268, 187.58609, 187.62961, 187.67044, 187.71268, 187.75528, 187.79819, 187.84183, 187.88416, 187.92462, 187.96719, 188.0098, 188.0549, 188.10202, 188.14798, 188.19414, 188.23969, 188.28632, 188.33499, 188.38423, 188.43146, 188.47794, 188.52431, 188.57013, 188.61865, 188.66565, 188.71187, 188.75861, 188.80621, 188.85393, 188.90173, 188.94839, 188.99448, 189.04036, 189.08531, 189.13077, 189.17767, 189.22517, 189.27315, 189.32074, 189.36909, 189.41704, 189.46393, 189.5119, 189.5609, 189.61021, 189.66124, 189.71246, 189.76324, 189.81259, 189.86185, 189.91013, 189.96013, 190.0108, 190.061, 190.11232, 190.1635, 190.21367, 190.2627, 190.31346, 190.36389, 190.41492, 190.46727, 190.51939, 190.57338, 190.62749, 190.68044, 190.73311, 190.78491, 190.83577, 190.8877, 190.93848, 190.98965, 191.04053, 191.09221, 191.1438, 191.19595, 191.24683, 191.29836, 191.35121, 191.40576, 191.45865, 191.51144, 191.56329, 191.61534, 191.66661, 191.71944, 191.77365, 191.82733, 191.88013, 191.93358, 191.98837, 192.04231, 192.09724, 192.15228, 192.20715, 192.26242, 192.32021, 192.37662, 192.4319, 192.48772, 192.54413, 192.59987, 192.65529, 192.71152, 192.76802, 192.82562, 192.88312, 192.94026, 192.99599, 193.05467, 193.11278, 193.17015, 193.22783, 193.28326, 193.33839, 193.39395, 193.44897, 193.50545, 193.563, 193.61928, 193.67555, 193.73364, 193.79195, 193.85016, 193.90939, 193.96805, 194.02667, 194.08534, 194.14226, 194.20026, 194.25986, 194.32065, 194.38155, 194.44293, 194.50323, 194.56407, 194.62587, 194.68752, 194.74759, 194.80595, 194.86389, 194.92307, 194.98349]}, "iteration-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [23.29918, 0.71187, 0.71207, 0.69449, 0.69446, 0.69443, 0.6988, 0.69196, 0.7146, 0.69983, 0.70196, 0.70471, 0.70358, 0.70105, 0.71451, 0.69917, 0.69866, 0.69442, 0.6948, 0.69086, 0.69495, 0.68836, 0.69965, 0.69226, 0.69484, 0.69875, 0.70073, 0.70246, 0.72083, 0.7009, 0.70048, 0.7008, 0.70366, 0.69412, 0.70178, 0.69908, 0.70543, 0.69424, 0.70464, 0.69955, 0.70803, 0.69841, 0.70257, 0.70418, 0.70875, 0.715, 0.70906, 0.70541, 0.71931, 0.7041, 0.70223, 0.70658, 0.69701, 0.69756, 0.69594, 0.70155, 0.70926, 0.70288, 0.6981, 0.70914, 0.69799, 0.70314, 0.70633, 0.70075, 0.70007, 0.70459, 0.70195, 0.69392, 0.7045, 0.70374, 0.70075, 0.69331, 0.69436, 0.6955, 0.70291, 0.69782, 0.70126, 0.70025, 0.70132, 0.7027, 0.70476, 0.70307, 0.69742, 0.69952, 0.69723, 0.8289, 0.70367, 0.7045, 0.70784, 0.71072, 0.70676, 0.70275, 0.70232, 0.70275, 0.70734, 0.70267, 0.70508, 0.70045, 0.70283, 0.71431, 0.708, 0.70934, 0.70749, 0.71204, 0.70839, 0.70834, 0.70947, 0.70787, 0.70812, 0.70457, 0.70563, 0.69994, 0.70262, 0.69627, 0.69863, 0.69913, 0.71178, 0.71423, 0.70926, 0.70785, 0.70607, 0.70391, 0.71582, 0.71055, 0.71123, 0.70438, 0.71121, 0.71074, 0.70765, 0.70483, 0.70686, 0.71125, 0.70564, 0.70533, 0.7078, 0.70873, 0.70986, 0.70805, 0.70797, 0.71206, 0.70956, 0.70912, 0.71021, 0.70934, 0.70819, 0.70233, 0.70414, 0.70448, 0.70564, 0.7015, 0.70586, 0.70217, 0.7129, 0.70787, 0.7092, 0.71158, 0.7112, 0.71167, 0.70869, 0.70914, 0.70573, 0.7106, 0.70502, 0.70709, 0.70454, 0.70862, 0.70342, 0.70716, 0.70517, 0.70888, 0.71242, 0.71066, 0.71063, 0.70907, 0.71159, 0.71233, 0.7117, 0.7115, 0.70892, 0.71015, 0.71212, 0.70842, 0.70856, 0.71199, 0.71305, 0.71701, 0.71312, 0.71367, 0.71284, 0.70741, 0.70964, 0.70851, 0.71466, 0.70509, 0.72116, 0.72852, 0.71403, 0.70864, 0.70955, 0.7163, 0.6926, 0.70139, 0.71844, 0.70855, 0.71025, 0.71363, 0.7113, 0.7081, 0.71651, 0.71161, 0.7088, 0.70621, 0.76558, 0.71366, 0.71465, 0.70832, 0.71501, 0.71439, 0.70996, 0.71112, 0.71318, 0.71005, 0.71114, 0.70462, 0.71021, 0.71174, 0.71118, 0.70552, 0.70941, 0.71352, 0.70296, 0.7077, 0.71087, 0.70967, 0.71319, 0.70487, 0.71314, 0.71027, 0.71726, 0.70291, 0.70583, 0.70043, 0.71003, 0.70162, 0.71159, 0.70538, 0.70772, 0.7058, 0.70393, 0.70436, 0.70523, 0.7076, 0.70951, 0.7073, 0.70677, 0.70977, 0.70523, 0.70814, 0.70619, 0.71387, 0.71394, 0.71664, 0.709, 0.70954, 0.71091, 0.71119, 0.7066, 0.71015, 0.71379, 0.70807, 0.7089, 0.70687, 0.70782, 0.70284, 0.7093, 0.70472, 0.70627, 0.70878, 0.7131, 0.71354, 0.70817, 0.7085, 0.70989, 0.7104, 0.70981, 0.70998, 0.70926, 0.70687, 0.71184, 0.7147, 0.71202, 0.70554, 0.70696, 0.71095, 0.7109, 0.70487, 0.7074, 0.70395, 0.70783, 0.70406, 0.71161, 0.70987, 0.70579, 0.70936, 0.81441, 0.70896, 0.70653, 0.70759, 0.71046, 0.70652, 0.70807, 0.70162, 0.70833, 0.70934, 0.70659, 0.71222, 0.71582, 0.71966, 0.71029, 0.70866, 0.70674, 0.71991, 0.7103, 0.70757, 0.71472, 0.70914, 0.71354, 0.8287, 0.71145, 0.70825, 0.71369, 0.71612, 0.71567, 0.71261, 0.71066, 0.70918, 0.70607, 0.70956, 0.72641, 0.7127, 0.71743, 0.70933, 0.71054, 0.70211, 0.7054, 0.70442, 0.712, 0.71222, 0.71615, 0.71003, 0.71338, 0.71009, 0.71334, 0.71107, 0.71501, 0.71714, 0.70686, 0.70974, 0.71546, 0.70423, 0.71293, 0.71055, 0.71309, 0.71563, 0.71163, 0.71034, 0.71044, 0.71, 0.70833, 0.71033, 0.70852, 0.7031, 0.71412, 0.70792, 0.71185, 0.70919, 0.7121, 0.70689, 0.71208, 0.70677, 0.7134, 0.71312, 0.71483, 0.71357, 0.71752, 0.7209, 0.71431, 0.71061, 0.71548, 0.7187, 0.71617, 0.71164, 0.71417, 0.71386, 0.71464, 0.71363, 0.71829, 0.72097, 0.71465, 0.7123]}, "lm loss validation": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [5.60433]}, "lm loss validation vs samples": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [5.60433]}, "lm loss validation ppl": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [271.59912]}, "lm loss validation ppl vs samples": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [271.59912]}}
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp2_fp8_pp/golden_values_lts_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp2_fp8_pp/golden_values_lts_dgx_a100.json
deleted file mode 100644
index 0a4099a0f99..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp2_fp8_pp/golden_values_lts_dgx_a100.json
+++ /dev/null
@@ -1,1223 +0,0 @@
-{
- "lm loss": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 10.85943,
- 10.87053,
- 10.8552,
- 10.80356,
- 10.64125,
- 10.62658,
- 10.41609,
- 10.12827,
- 9.92585,
- 9.82486,
- 9.56933,
- 9.84044,
- 9.86925,
- 9.61422,
- 9.77596,
- 9.50084,
- 9.45229,
- 9.6411,
- 9.38015,
- 9.32643,
- 9.23852,
- 9.14191,
- 9.17285,
- 8.9927,
- 9.18814,
- 9.05775,
- 9.15479,
- 9.16462,
- 9.29869,
- 8.98698,
- 8.93083,
- 9.04739,
- 9.04626,
- 8.65646,
- 8.71654,
- 8.75519,
- 8.68493,
- 8.73641,
- 8.66113,
- 8.76487,
- 8.66214,
- 8.84933,
- 8.83099,
- 8.49833,
- 8.38764,
- 8.42872,
- 8.49081,
- 8.38216,
- 8.4304,
- 8.57772,
- 8.3637,
- 8.19009,
- 8.2243,
- 8.21889,
- 8.26311,
- 7.90921,
- 8.08965,
- 7.88749,
- 8.23972,
- 8.2245,
- 7.99829,
- 7.95654,
- 7.91147,
- 7.73211,
- 7.73278,
- 7.63576,
- 7.50815,
- 7.89999,
- 7.69271,
- 7.44759,
- 7.73518,
- 7.76308,
- 7.53726,
- 7.29755,
- 7.45042,
- 7.3335,
- 7.46271,
- 7.225,
- 7.63686,
- 7.2791,
- 7.35262,
- 7.21194,
- 7.21749,
- 7.42206,
- 7.17637,
- 7.28451,
- 7.00229,
- 7.00565,
- 7.03947,
- 7.14154,
- 6.82546,
- 6.98874,
- 7.09158,
- 7.00468,
- 6.87701,
- 6.76252,
- 6.99607,
- 7.06246,
- 6.7093,
- 6.58432,
- 6.73413,
- 6.74992,
- 6.73916,
- 6.74503,
- 6.66397,
- 6.41283,
- 6.64356,
- 6.62408,
- 6.4507,
- 6.63348,
- 6.74925,
- 6.61194,
- 6.72888,
- 6.69712,
- 6.62816,
- 6.51254,
- 6.60259,
- 6.40806,
- 6.66632,
- 6.2507,
- 6.25539,
- 6.30384,
- 6.39197,
- 6.35089,
- 6.45101,
- 6.2955,
- 6.34162,
- 6.23953,
- 6.2031,
- 6.40112,
- 6.32791,
- 6.32743,
- 6.16712,
- 6.16395,
- 6.24217,
- 6.38851,
- 6.20408,
- 6.15194,
- 6.18454,
- 6.1209,
- 6.06687,
- 6.07678,
- 6.26378,
- 6.41474,
- 6.26293,
- 6.30777,
- 6.10302,
- 6.18498,
- 6.00557,
- 6.03665,
- 5.96024,
- 6.2507,
- 6.19188,
- 5.96584,
- 5.78516,
- 6.12539,
- 5.85253,
- 6.10869,
- 5.78882,
- 6.16044,
- 6.14583,
- 6.08775,
- 5.93339,
- 6.11557,
- 5.94544,
- 6.19493,
- 5.89494,
- 5.79561,
- 5.77741,
- 5.68874,
- 6.0135,
- 5.99903,
- 6.06725,
- 5.8872,
- 6.03788,
- 5.96513,
- 5.99395,
- 5.98839,
- 5.94543,
- 5.83698,
- 5.94898,
- 5.61313,
- 5.69872,
- 5.88749,
- 5.84072,
- 5.8593,
- 5.76366,
- 5.83328,
- 5.72126,
- 5.55865,
- 5.71778,
- 5.62379,
- 5.82983,
- 5.60127,
- 5.70628,
- 5.71074,
- 5.89526,
- 5.64025,
- 5.84484,
- 5.73462,
- 5.86678,
- 5.32703,
- 5.89388,
- 5.86988,
- 5.85354,
- 5.41104,
- 5.40723,
- 5.62371,
- 5.58859,
- 5.48045,
- 5.57103,
- 5.66878,
- 5.47266,
- 5.74241,
- 5.50355,
- 5.58657,
- 5.6171,
- 5.6132,
- 5.50529,
- 5.61047,
- 5.6702,
- 5.67709,
- 5.58565,
- 5.65642,
- 5.36862,
- 5.67635,
- 5.62256,
- 5.42287,
- 5.57977,
- 5.62805,
- 5.54907,
- 5.33789,
- 5.53276,
- 5.47933,
- 5.47544,
- 5.3732,
- 5.54994,
- 5.60231,
- 5.38211,
- 5.51886,
- 5.48037,
- 5.32973,
- 5.50123,
- 5.40609,
- 5.44142,
- 5.31615,
- 5.06636,
- 5.47338,
- 5.56525,
- 5.70949,
- 5.41185,
- 5.59801,
- 5.63224,
- 5.22911,
- 5.26901,
- 5.38983,
- 5.39245,
- 5.32727,
- 5.49282,
- 5.18151,
- 5.30008,
- 5.24082,
- 5.37393,
- 5.25404,
- 5.443,
- 5.53676,
- 5.31112,
- 5.43487,
- 5.33659,
- 5.07047,
- 5.30683,
- 5.25186,
- 5.30466,
- 5.11066,
- 5.27622,
- 5.26326,
- 5.47457,
- 5.15806,
- 5.26885,
- 5.20826,
- 5.35837,
- 4.98081,
- 4.9145,
- 5.32227,
- 5.38824,
- 5.22777,
- 5.3152,
- 5.10173,
- 5.1612,
- 5.2585,
- 5.06606,
- 5.26362,
- 5.06839,
- 5.34424,
- 5.24663,
- 5.15173,
- 5.24493,
- 5.0382,
- 5.31517,
- 5.05402,
- 5.02588,
- 5.1416,
- 5.11464,
- 5.26976,
- 5.1508,
- 5.2759,
- 5.09641,
- 5.09478,
- 5.24899,
- 5.32187,
- 5.25358,
- 5.18918,
- 5.14007,
- 5.28993,
- 4.94923,
- 5.20665,
- 5.09082,
- 5.30279,
- 5.17751,
- 5.1877,
- 5.11038,
- 4.97967,
- 4.98954,
- 5.21943,
- 5.31096,
- 5.09497,
- 5.05772,
- 4.91641,
- 5.12945,
- 5.11765,
- 4.92879,
- 5.34097,
- 5.02317,
- 5.10375,
- 5.1625,
- 5.00244,
- 5.06493,
- 5.07017,
- 4.9971,
- 5.07986,
- 5.162,
- 4.9804,
- 5.18135,
- 4.9301,
- 4.92184,
- 5.06864,
- 4.99078,
- 4.90547,
- 4.77408,
- 4.94473,
- 5.11756,
- 5.01899,
- 5.02253,
- 5.33217,
- 4.96101,
- 4.99441,
- 5.04553,
- 4.80626,
- 4.7391,
- 4.99364,
- 5.03728,
- 4.87194,
- 4.95067,
- 5.04413,
- 5.02255,
- 4.81787,
- 4.89308,
- 4.90769,
- 4.82921,
- 4.7438,
- 5.01691,
- 4.75193,
- 5.21153,
- 4.78624,
- 4.99548,
- 4.73862,
- 4.78812,
- 4.81836,
- 4.64864,
- 4.65649,
- 4.84617,
- 4.80992,
- 4.80425,
- 4.92585,
- 4.88618,
- 4.93246,
- 4.76987,
- 4.88471,
- 4.73751,
- 4.91636,
- 4.95806,
- 4.87967,
- 4.70744,
- 4.78973,
- 4.89998,
- 4.71284,
- 4.87002,
- 4.69686,
- 4.69721,
- 4.648
- ]
- },
- "num-zeros": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 61.0,
- 66.0,
- 86.0,
- 64.0,
- 68.0,
- 81.0,
- 100.0,
- 92.0,
- 106.0,
- 131.0,
- 123.0,
- 149.0,
- 140.0,
- 182.0,
- 180.0,
- 159.0,
- 169.0,
- 200.0,
- 163.0,
- 164.0,
- 168.0,
- 177.0,
- 167.0,
- 183.0,
- 190.0,
- 162.0,
- 188.0,
- 162.0,
- 143.0,
- 160.0,
- 156.0,
- 192.0,
- 152.0,
- 179.0,
- 141.0,
- 176.0,
- 168.0,
- 202.0,
- 176.0,
- 202.0,
- 157.0,
- 168.0,
- 183.0,
- 180.0,
- 177.0,
- 205.0,
- 201.0,
- 158.0,
- 189.0,
- 219.0,
- 217.0,
- 173.0,
- 211.0,
- 145.0,
- 197.0,
- 176.0,
- 160.0,
- 154.0,
- 207.0,
- 234.0,
- 196.0,
- 193.0,
- 167.0,
- 160.0,
- 196.0,
- 207.0,
- 190.0,
- 186.0,
- 186.0,
- 185.0,
- 225.0,
- 236.0,
- 162.0,
- 247.0,
- 175.0,
- 184.0,
- 230.0,
- 220.0,
- 230.0,
- 201.0,
- 226.0,
- 212.0,
- 204.0,
- 260.0,
- 192.0,
- 186.0,
- 160.0,
- 202.0,
- 184.0,
- 209.0,
- 187.0,
- 214.0,
- 225.0,
- 203.0,
- 185.0,
- 171.0,
- 178.0,
- 193.0,
- 222.0,
- 182.0,
- 155.0,
- 154.0,
- 159.0,
- 141.0,
- 167.0,
- 143.0,
- 154.0,
- 181.0,
- 142.0,
- 149.0,
- 169.0,
- 177.0,
- 185.0,
- 167.0,
- 161.0,
- 143.0,
- 148.0,
- 138.0,
- 177.0,
- 141.0,
- 152.0,
- 132.0,
- 145.0,
- 144.0,
- 115.0,
- 111.0,
- 100.0,
- 130.0,
- 120.0,
- 124.0,
- 154.0,
- 121.0,
- 140.0,
- 122.0,
- 121.0,
- 116.0,
- 138.0,
- 116.0,
- 115.0,
- 109.0,
- 106.0,
- 84.0,
- 120.0,
- 118.0,
- 127.0,
- 108.0,
- 106.0,
- 135.0,
- 101.0,
- 96.0,
- 120.0,
- 123.0,
- 88.0,
- 134.0,
- 143.0,
- 109.0,
- 116.0,
- 102.0,
- 104.0,
- 118.0,
- 116.0,
- 125.0,
- 104.0,
- 122.0,
- 111.0,
- 95.0,
- 111.0,
- 101.0,
- 125.0,
- 103.0,
- 112.0,
- 121.0,
- 103.0,
- 90.0,
- 147.0,
- 120.0,
- 110.0,
- 114.0,
- 89.0,
- 111.0,
- 111.0,
- 101.0,
- 108.0,
- 123.0,
- 75.0,
- 100.0,
- 85.0,
- 125.0,
- 95.0,
- 114.0,
- 109.0,
- 99.0,
- 102.0,
- 95.0,
- 108.0,
- 99.0,
- 102.0,
- 76.0,
- 102.0,
- 112.0,
- 95.0,
- 71.0,
- 104.0,
- 124.0,
- 103.0,
- 106.0,
- 106.0,
- 85.0,
- 132.0,
- 112.0,
- 106.0,
- 100.0,
- 94.0,
- 126.0,
- 105.0,
- 102.0,
- 112.0,
- 126.0,
- 127.0,
- 83.0,
- 73.0,
- 102.0,
- 84.0,
- 99.0,
- 121.0,
- 106.0,
- 112.0,
- 101.0,
- 89.0,
- 117.0,
- 109.0,
- 92.0,
- 117.0,
- 111.0,
- 111.0,
- 111.0,
- 102.0,
- 92.0,
- 120.0,
- 102.0,
- 99.0,
- 98.0,
- 105.0,
- 101.0,
- 108.0,
- 87.0,
- 86.0,
- 114.0,
- 115.0,
- 112.0,
- 101.0,
- 126.0,
- 108.0,
- 110.0,
- 105.0,
- 87.0,
- 117.0,
- 90.0,
- 126.0,
- 107.0,
- 103.0,
- 109.0,
- 111.0,
- 85.0,
- 105.0,
- 103.0,
- 113.0,
- 97.0,
- 119.0,
- 117.0,
- 138.0,
- 133.0,
- 110.0,
- 105.0,
- 115.0,
- 103.0,
- 86.0,
- 132.0,
- 102.0,
- 119.0,
- 93.0,
- 99.0,
- 100.0,
- 110.0,
- 116.0,
- 87.0,
- 116.0,
- 81.0,
- 114.0,
- 103.0,
- 103.0,
- 103.0,
- 111.0,
- 92.0,
- 88.0,
- 95.0,
- 92.0,
- 103.0,
- 98.0,
- 97.0,
- 110.0,
- 129.0,
- 110.0,
- 99.0,
- 118.0,
- 111.0,
- 88.0,
- 101.0,
- 138.0,
- 104.0,
- 102.0,
- 114.0,
- 88.0,
- 116.0,
- 108.0,
- 101.0,
- 104.0,
- 108.0,
- 104.0,
- 104.0,
- 129.0,
- 121.0,
- 89.0,
- 104.0,
- 98.0,
- 100.0,
- 118.0,
- 103.0,
- 98.0,
- 90.0,
- 90.0,
- 100.0,
- 106.0,
- 111.0,
- 116.0,
- 102.0,
- 117.0,
- 130.0,
- 131.0,
- 108.0,
- 110.0,
- 129.0,
- 116.0,
- 112.0,
- 95.0,
- 98.0,
- 107.0,
- 97.0,
- 114.0,
- 119.0,
- 94.0,
- 95.0,
- 113.0,
- 114.0,
- 116.0,
- 102.0,
- 126.0,
- 119.0,
- 103.0,
- 116.0,
- 110.0,
- 124.0,
- 132.0,
- 117.0,
- 110.0,
- 115.0,
- 116.0,
- 91.0,
- 105.0,
- 126.0,
- 77.0,
- 107.0,
- 100.0,
- 119.0,
- 116.0,
- 137.0,
- 86.0,
- 132.0,
- 102.0,
- 108.0,
- 119.0,
- 106.0,
- 135.0,
- 117.0,
- 98.0,
- 111.0,
- 138.0,
- 120.0,
- 103.0,
- 102.0,
- 133.0,
- 102.0,
- 139.0,
- 112.0,
- 108.0,
- 104.0,
- 106.0,
- 110.0,
- 125.0,
- 106.0
- ]
- },
- "iteration-time": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 22.12982,
- 0.58407,
- 0.59544,
- 0.57636,
- 0.5766,
- 0.58301,
- 0.57644,
- 0.58681,
- 0.58148,
- 0.57124,
- 0.56572,
- 0.58109,
- 0.56543,
- 0.5649,
- 0.56341,
- 0.56668,
- 0.56923,
- 0.57023,
- 0.57002,
- 0.57163,
- 0.5698,
- 0.57588,
- 0.57051,
- 0.56835,
- 0.57262,
- 0.57082,
- 0.5649,
- 0.57266,
- 0.57393,
- 0.58758,
- 0.56761,
- 0.57161,
- 0.57422,
- 0.57961,
- 0.57363,
- 0.59229,
- 0.56483,
- 0.57134,
- 0.56808,
- 0.5692,
- 0.56593,
- 0.5711,
- 0.56922,
- 0.5683,
- 0.56701,
- 0.57467,
- 0.58127,
- 0.56473,
- 0.56993,
- 0.57385,
- 0.57146,
- 0.57652,
- 0.57352,
- 0.56785,
- 0.5726,
- 0.57374,
- 0.56621,
- 0.56991,
- 0.57008,
- 0.57409,
- 0.5744,
- 0.57432,
- 0.57083,
- 0.57352,
- 0.57249,
- 0.57474,
- 0.57472,
- 0.58684,
- 0.5799,
- 0.57096,
- 0.57292,
- 0.56708,
- 0.5663,
- 0.56501,
- 0.56504,
- 0.56721,
- 0.56683,
- 0.56252,
- 0.77946,
- 0.56722,
- 0.56653,
- 0.57422,
- 0.57071,
- 0.56657,
- 0.56506,
- 0.56584,
- 0.56691,
- 0.56745,
- 0.57057,
- 0.56428,
- 0.56687,
- 0.57132,
- 0.56594,
- 0.56782,
- 0.56891,
- 0.56753,
- 0.56906,
- 0.56673,
- 0.88584,
- 0.56888,
- 0.57701,
- 0.57547,
- 0.56962,
- 0.5688,
- 0.57167,
- 0.57702,
- 0.57411,
- 0.57094,
- 0.57176,
- 0.56854,
- 0.56903,
- 0.56946,
- 0.56935,
- 0.56407,
- 0.56657,
- 0.57094,
- 0.56615,
- 0.57381,
- 0.56941,
- 0.57691,
- 0.57244,
- 0.57915,
- 0.57743,
- 0.57646,
- 0.56386,
- 0.56966,
- 0.56538,
- 0.56642,
- 0.56814,
- 0.56657,
- 0.57645,
- 0.57776,
- 0.57771,
- 0.57127,
- 0.57046,
- 0.56543,
- 0.56914,
- 0.57383,
- 0.59003,
- 0.57928,
- 0.57644,
- 0.56492,
- 0.57059,
- 0.56832,
- 0.57254,
- 0.57276,
- 0.56747,
- 0.57186,
- 0.571,
- 0.56967,
- 0.56653,
- 0.57611,
- 0.57206,
- 0.57268,
- 0.57845,
- 0.56889,
- 0.56949,
- 0.58288,
- 0.57504,
- 0.57406,
- 0.57109,
- 0.58614,
- 0.56961,
- 0.56989,
- 0.57728,
- 0.57191,
- 0.56862,
- 0.57399,
- 0.56928,
- 0.57292,
- 0.57047,
- 0.57538,
- 0.5753,
- 0.57291,
- 0.57288,
- 0.58911,
- 0.57434,
- 0.57201,
- 0.57334,
- 0.57987,
- 0.5698,
- 0.57996,
- 0.57766,
- 0.57099,
- 0.57237,
- 0.57303,
- 0.67546,
- 0.56788,
- 0.56501,
- 0.57103,
- 0.56997,
- 0.56764,
- 0.57336,
- 0.56641,
- 0.5662,
- 0.60418,
- 0.56859,
- 0.57566,
- 0.56885,
- 0.58381,
- 0.56215,
- 0.57305,
- 0.58455,
- 0.57298,
- 0.56641,
- 0.56918,
- 0.57446,
- 0.57409,
- 0.57287,
- 0.57556,
- 0.569,
- 0.58387,
- 0.56755,
- 0.57091,
- 0.57385,
- 0.57298,
- 0.57161,
- 0.57035,
- 0.56803,
- 0.5801,
- 0.57192,
- 0.57401,
- 0.57126,
- 0.57158,
- 0.56959,
- 0.57293,
- 0.5672,
- 0.57462,
- 0.57167,
- 0.57014,
- 0.57475,
- 0.57603,
- 0.5714,
- 0.62444,
- 0.57036,
- 0.56999,
- 0.57522,
- 0.5716,
- 0.58197,
- 0.5765,
- 0.56999,
- 0.58429,
- 0.56856,
- 0.58173,
- 0.57178,
- 0.56779,
- 0.56947,
- 0.57295,
- 0.56857,
- 0.56829,
- 0.57295,
- 0.57504,
- 0.57254,
- 0.5675,
- 0.56824,
- 0.56877,
- 0.57088,
- 0.58067,
- 0.57834,
- 0.58238,
- 0.57541,
- 0.57865,
- 0.5778,
- 0.57228,
- 0.57535,
- 0.57627,
- 0.56977,
- 0.57269,
- 0.57535,
- 0.5772,
- 0.5831,
- 0.56943,
- 0.57879,
- 0.57353,
- 0.57324,
- 0.57476,
- 0.57759,
- 0.57151,
- 0.57047,
- 0.56246,
- 0.56374,
- 0.57046,
- 0.56893,
- 0.57193,
- 0.5791,
- 0.58222,
- 0.5705,
- 0.57925,
- 0.58343,
- 0.58822,
- 0.57432,
- 0.57436,
- 0.57976,
- 0.57785,
- 0.57198,
- 0.57174,
- 0.56859,
- 0.56547,
- 0.57031,
- 0.56948,
- 0.57002,
- 0.57584,
- 0.57149,
- 0.581,
- 0.57702,
- 0.58343,
- 0.57227,
- 0.57291,
- 0.57608,
- 0.57163,
- 0.5767,
- 0.56671,
- 0.5697,
- 0.5685,
- 0.56652,
- 0.57017,
- 0.56761,
- 0.57061,
- 0.56876,
- 0.56891,
- 0.59662,
- 0.59338,
- 0.59138,
- 0.57587,
- 0.59007,
- 0.5826,
- 0.5951,
- 0.58781,
- 0.58277,
- 0.58392,
- 0.58454,
- 0.58183,
- 0.58321,
- 0.58162,
- 0.58178,
- 0.58315,
- 0.58576,
- 0.58984,
- 0.58447,
- 0.58384,
- 0.58444,
- 0.57882,
- 0.58178,
- 0.58201,
- 0.58621,
- 0.58435,
- 0.58728,
- 0.58479,
- 0.58194,
- 0.58203,
- 0.58472,
- 0.58349,
- 0.58442,
- 0.5844,
- 0.59043,
- 0.58246,
- 0.57817,
- 0.59224,
- 0.58333,
- 0.58317,
- 0.58198,
- 0.57783,
- 0.58072,
- 0.57983,
- 0.57676,
- 0.57121,
- 0.57894,
- 0.57207,
- 0.57802,
- 0.5724,
- 0.57705,
- 0.57431,
- 0.57357,
- 0.56963,
- 0.57063,
- 0.57408,
- 0.57724,
- 0.57667,
- 0.57465,
- 0.57229,
- 0.57231,
- 0.57426,
- 0.57414,
- 0.57398,
- 0.57718,
- 0.57464,
- 0.57416,
- 0.57254,
- 0.5724,
- 0.58836,
- 0.57475,
- 0.57042,
- 0.57821,
- 0.58139,
- 0.57394,
- 0.57683,
- 0.57436,
- 0.57166,
- 0.57692,
- 0.57586
- ]
- }
-}
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp2_fp8_pp/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp2_fp8_pp/model_config.yaml
deleted file mode 100644
index 51a2f6cfc7c..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp1_pp2_fp8_pp/model_config.yaml
+++ /dev/null
@@ -1,55 +0,0 @@
-ENV_VARS:
- CUDA_DEVICE_MAX_CONNECTIONS: 1
- NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
- NVTE_FUSED_ATTN: 0
- NCCL_ALGO: Ring
- CUBLAS_WORKSPACE_CONFIG: :4096:8
-MODEL_ARGS:
- --num-layers: 12
- --hidden-size: 512
- --num-attention-heads: 8
- --log-params-norm: true
- --log-num-zeros-in-grad: true
- --log-validation-ppl-to-tensorboard: true
- --log-timers-to-tensorboard: true
- --tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
- --seq-length: 1024
- --max-position-embeddings: 1024
- --train-iters: 2000
- --timing-log-level: 0
- --lr-decay-iters: 320000
- --save: ${CHECKPOINT_SAVE_PATH}
- --load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
- --split: 949,50,1
- --distributed-backend: nccl
- --lr: 0.00015
- --lr-decay-style: cosine
- --min-lr: 1.0e-5
- --weight-decay: 1e-2
- --clip-grad: 1.0
- --lr-warmup-fraction: .01
- --log-interval: 1
- --save-interval: 10000
- --eval-interval: 1000
- --eval-iters: 10
- --transformer-impl: transformer_engine
- --tensor-model-parallel-size: 1
- --pipeline-model-parallel-size: 2
- --deterministic-mode: true
- --no-gradient-accumulation-fusion: true
- --fp8-format: hybrid
- --fp8-amax-history-len: 1024
- --fp8-amax-compute-algo: max
- --attention-softmax-in-fp32: true
- --ckpt-format: torch_dist
- --dist-ckpt-optim-fully-reshardable: true
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
- --data-cache-path: ${DATA_CACHE_PATH}
- --bf16: true
- --log-memory-to-tensorboard: true
-TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/golden_values_dev_dgx_gb200.json
deleted file mode 100644
index 9e26dfeeb6e..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/golden_values_dev_dgx_gb200.json
+++ /dev/null
@@ -1 +0,0 @@
-{}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/tp_comm_overlap_cfg.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/tp_comm_overlap_cfg.yaml
deleted file mode 100644
index 9aec16146aa..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/tp_comm_overlap_cfg.yaml
+++ /dev/null
@@ -1,45 +0,0 @@
-qkv_fprop:
- method: ring_exchange
- aggregate: 1
-
-proj_fprop:
- method: pipeline
- num_sm: 4
- num_splits: 4
- set_sm_margin: 0
-
-fc1_fprop:
- method: ring_exchange
- aggregate: 0
-
-fc2_fprop:
- method: ring_exchange
- aggregate: 0
-
-fc2_dgrad:
- method: ring_exchange
- aggregate: 0
-
-fc1_dgrad:
- method: bulk
- num_sm: 2
- set_sm_margin: 0
-
-fc1_wgrad:
- method: bulk
- num_sm: 2
- set_sm_margin: 0
-
-proj_dgrad:
- method: ring_exchange
- aggregate: 1
-
-qkv_dgrad:
- method: bulk
- num_sm: 2
- set_sm_margin: 0
-
-qkv_wgrad:
- method: bulk
- num_sm: 2
- set_sm_margin: 0
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp/golden_values_dev_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp/golden_values_dev_dgx_a100.json
deleted file mode 100644
index 178565f517f..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp/golden_values_dev_dgx_a100.json
+++ /dev/null
@@ -1,1223 +0,0 @@
-{
- "lm loss": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 10.84434,
- 10.87343,
- 10.85057,
- 10.81084,
- 10.64478,
- 10.63856,
- 10.42829,
- 10.13529,
- 9.9354,
- 9.83536,
- 9.58562,
- 9.84798,
- 9.88582,
- 9.63128,
- 9.79015,
- 9.51139,
- 9.45969,
- 9.65541,
- 9.38989,
- 9.33926,
- 9.24938,
- 9.15128,
- 9.18196,
- 9.0045,
- 9.19833,
- 9.06658,
- 9.16104,
- 9.16968,
- 9.30055,
- 8.98918,
- 8.92952,
- 9.05033,
- 9.04653,
- 8.66027,
- 8.72522,
- 8.75656,
- 8.69485,
- 8.74326,
- 8.66685,
- 8.7728,
- 8.67074,
- 8.86153,
- 8.8433,
- 8.50914,
- 8.39911,
- 8.43859,
- 8.49596,
- 8.39384,
- 8.44083,
- 8.59281,
- 8.37629,
- 8.2001,
- 8.23362,
- 8.23015,
- 8.27548,
- 7.92086,
- 8.10003,
- 7.89799,
- 8.25216,
- 8.23462,
- 8.01021,
- 7.97597,
- 7.9264,
- 7.74459,
- 7.748,
- 7.65018,
- 7.52046,
- 7.91112,
- 7.70254,
- 7.456,
- 7.74697,
- 7.77483,
- 7.54415,
- 7.3027,
- 7.45591,
- 7.34318,
- 7.46577,
- 7.22819,
- 7.63648,
- 7.28207,
- 7.34835,
- 7.21309,
- 7.21075,
- 7.41924,
- 7.17318,
- 7.28141,
- 6.99426,
- 7.00286,
- 7.03961,
- 7.13676,
- 6.822,
- 6.9855,
- 7.08945,
- 6.99871,
- 6.87487,
- 6.75719,
- 6.99117,
- 7.06005,
- 6.70456,
- 6.58452,
- 6.72787,
- 6.74473,
- 6.73373,
- 6.7382,
- 6.6584,
- 6.40648,
- 6.63688,
- 6.61955,
- 6.44576,
- 6.62788,
- 6.74244,
- 6.61006,
- 6.72544,
- 6.69264,
- 6.62569,
- 6.50572,
- 6.59635,
- 6.40504,
- 6.66311,
- 6.24639,
- 6.25134,
- 6.30293,
- 6.39011,
- 6.3472,
- 6.45168,
- 6.29229,
- 6.33985,
- 6.23688,
- 6.20384,
- 6.40017,
- 6.32742,
- 6.32422,
- 6.16691,
- 6.16021,
- 6.24067,
- 6.38468,
- 6.20364,
- 6.15286,
- 6.18196,
- 6.11784,
- 6.06616,
- 6.07804,
- 6.26273,
- 6.41356,
- 6.26419,
- 6.30289,
- 6.10616,
- 6.18152,
- 6.00825,
- 6.03597,
- 5.96121,
- 6.25362,
- 6.19475,
- 5.97105,
- 5.78892,
- 6.1312,
- 5.85287,
- 6.10817,
- 5.79121,
- 6.16545,
- 6.14698,
- 6.08542,
- 5.92808,
- 6.11875,
- 5.94753,
- 6.19922,
- 5.89541,
- 5.79008,
- 5.78091,
- 5.68691,
- 6.01341,
- 6.00102,
- 6.06828,
- 5.89084,
- 6.04196,
- 5.96792,
- 5.99841,
- 5.99525,
- 5.95169,
- 5.84243,
- 5.95132,
- 5.61796,
- 5.70314,
- 5.88856,
- 5.84026,
- 5.86305,
- 5.76304,
- 5.83656,
- 5.72719,
- 5.56214,
- 5.72112,
- 5.62344,
- 5.83074,
- 5.60385,
- 5.7076,
- 5.70851,
- 5.89941,
- 5.64331,
- 5.84777,
- 5.74091,
- 5.86663,
- 5.32913,
- 5.89635,
- 5.87437,
- 5.85388,
- 5.41178,
- 5.40838,
- 5.62884,
- 5.59534,
- 5.48296,
- 5.57705,
- 5.67454,
- 5.47707,
- 5.74309,
- 5.50833,
- 5.59207,
- 5.62207,
- 5.61979,
- 5.51213,
- 5.61257,
- 5.67073,
- 5.67911,
- 5.58501,
- 5.66043,
- 5.37203,
- 5.67588,
- 5.62767,
- 5.42011,
- 5.58178,
- 5.62963,
- 5.55361,
- 5.3406,
- 5.53513,
- 5.48634,
- 5.48134,
- 5.38001,
- 5.55335,
- 5.60291,
- 5.3855,
- 5.51982,
- 5.4869,
- 5.33392,
- 5.50985,
- 5.4109,
- 5.44586,
- 5.31905,
- 5.06585,
- 5.47792,
- 5.56891,
- 5.71472,
- 5.4116,
- 5.6004,
- 5.63428,
- 5.23158,
- 5.26784,
- 5.39219,
- 5.39546,
- 5.32677,
- 5.49847,
- 5.18449,
- 5.2968,
- 5.24785,
- 5.37475,
- 5.25356,
- 5.4427,
- 5.53544,
- 5.30755,
- 5.43162,
- 5.34057,
- 5.07742,
- 5.3105,
- 5.2513,
- 5.30299,
- 5.10864,
- 5.27348,
- 5.26261,
- 5.47314,
- 5.15993,
- 5.26482,
- 5.20655,
- 5.3524,
- 4.98067,
- 4.91136,
- 5.32265,
- 5.39056,
- 5.22683,
- 5.32037,
- 5.10162,
- 5.16075,
- 5.26068,
- 5.07477,
- 5.2665,
- 5.06803,
- 5.34087,
- 5.24754,
- 5.14536,
- 5.2427,
- 5.03942,
- 5.31639,
- 5.05259,
- 5.028,
- 5.13985,
- 5.10959,
- 5.2711,
- 5.15231,
- 5.27332,
- 5.09281,
- 5.09413,
- 5.24576,
- 5.32664,
- 5.25301,
- 5.19004,
- 5.14196,
- 5.29006,
- 4.9529,
- 5.20696,
- 5.09518,
- 5.30439,
- 5.17088,
- 5.18705,
- 5.11541,
- 4.98195,
- 4.99339,
- 5.2219,
- 5.30712,
- 5.09994,
- 5.05467,
- 4.91696,
- 5.12387,
- 5.1162,
- 4.92675,
- 5.33512,
- 5.02297,
- 5.09855,
- 5.1647,
- 5.00177,
- 5.06604,
- 5.06519,
- 4.9938,
- 5.07915,
- 5.16172,
- 4.97704,
- 5.18061,
- 4.92631,
- 4.92011,
- 5.06494,
- 4.98947,
- 4.90622,
- 4.7743,
- 4.94211,
- 5.11143,
- 5.01084,
- 5.0159,
- 5.3267,
- 4.95652,
- 4.98832,
- 5.04364,
- 4.80948,
- 4.72945,
- 4.99165,
- 5.0429,
- 4.87065,
- 4.95272,
- 5.04422,
- 5.02216,
- 4.81261,
- 4.89101,
- 4.90203,
- 4.82648,
- 4.73442,
- 5.00558,
- 4.75484,
- 5.20509,
- 4.78834,
- 4.99179,
- 4.73272,
- 4.78083,
- 4.81532,
- 4.64586,
- 4.65217,
- 4.83878,
- 4.8041,
- 4.79376,
- 4.91789,
- 4.88008,
- 4.92551,
- 4.76829,
- 4.87736,
- 4.72836,
- 4.9114,
- 4.95389,
- 4.87038,
- 4.70453,
- 4.77938,
- 4.89906,
- 4.70579,
- 4.85315,
- 4.68969,
- 4.68533,
- 4.6408
- ]
- },
- "num-zeros": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 65.0,
- 72.0,
- 81.0,
- 76.0,
- 70.0,
- 86.0,
- 92.0,
- 100.0,
- 95.0,
- 121.0,
- 118.0,
- 150.0,
- 126.0,
- 174.0,
- 178.0,
- 176.0,
- 175.0,
- 175.0,
- 160.0,
- 180.0,
- 172.0,
- 163.0,
- 172.0,
- 175.0,
- 186.0,
- 162.0,
- 218.0,
- 187.0,
- 173.0,
- 157.0,
- 155.0,
- 146.0,
- 159.0,
- 193.0,
- 130.0,
- 155.0,
- 129.0,
- 199.0,
- 160.0,
- 180.0,
- 150.0,
- 169.0,
- 170.0,
- 198.0,
- 157.0,
- 171.0,
- 158.0,
- 193.0,
- 206.0,
- 230.0,
- 179.0,
- 203.0,
- 193.0,
- 154.0,
- 162.0,
- 189.0,
- 160.0,
- 154.0,
- 194.0,
- 223.0,
- 184.0,
- 182.0,
- 174.0,
- 151.0,
- 198.0,
- 237.0,
- 186.0,
- 168.0,
- 179.0,
- 178.0,
- 237.0,
- 233.0,
- 164.0,
- 208.0,
- 216.0,
- 192.0,
- 228.0,
- 205.0,
- 225.0,
- 214.0,
- 206.0,
- 237.0,
- 234.0,
- 263.0,
- 225.0,
- 192.0,
- 197.0,
- 207.0,
- 156.0,
- 211.0,
- 177.0,
- 199.0,
- 215.0,
- 208.0,
- 212.0,
- 170.0,
- 214.0,
- 204.0,
- 209.0,
- 186.0,
- 187.0,
- 180.0,
- 166.0,
- 145.0,
- 154.0,
- 169.0,
- 145.0,
- 162.0,
- 152.0,
- 192.0,
- 162.0,
- 175.0,
- 167.0,
- 161.0,
- 136.0,
- 135.0,
- 140.0,
- 121.0,
- 164.0,
- 128.0,
- 137.0,
- 114.0,
- 120.0,
- 142.0,
- 116.0,
- 128.0,
- 97.0,
- 132.0,
- 132.0,
- 105.0,
- 157.0,
- 143.0,
- 145.0,
- 130.0,
- 135.0,
- 126.0,
- 122.0,
- 102.0,
- 137.0,
- 107.0,
- 127.0,
- 87.0,
- 99.0,
- 136.0,
- 96.0,
- 119.0,
- 96.0,
- 121.0,
- 127.0,
- 141.0,
- 120.0,
- 132.0,
- 97.0,
- 117.0,
- 97.0,
- 102.0,
- 118.0,
- 127.0,
- 104.0,
- 100.0,
- 128.0,
- 104.0,
- 107.0,
- 103.0,
- 110.0,
- 97.0,
- 108.0,
- 126.0,
- 102.0,
- 126.0,
- 127.0,
- 100.0,
- 108.0,
- 111.0,
- 106.0,
- 112.0,
- 94.0,
- 105.0,
- 116.0,
- 106.0,
- 96.0,
- 114.0,
- 116.0,
- 149.0,
- 120.0,
- 102.0,
- 111.0,
- 117.0,
- 94.0,
- 103.0,
- 114.0,
- 101.0,
- 112.0,
- 110.0,
- 112.0,
- 87.0,
- 116.0,
- 95.0,
- 119.0,
- 116.0,
- 116.0,
- 93.0,
- 103.0,
- 99.0,
- 93.0,
- 115.0,
- 115.0,
- 92.0,
- 99.0,
- 125.0,
- 114.0,
- 102.0,
- 102.0,
- 100.0,
- 115.0,
- 107.0,
- 118.0,
- 113.0,
- 109.0,
- 110.0,
- 97.0,
- 103.0,
- 96.0,
- 99.0,
- 115.0,
- 118.0,
- 105.0,
- 117.0,
- 104.0,
- 105.0,
- 113.0,
- 97.0,
- 97.0,
- 114.0,
- 97.0,
- 99.0,
- 96.0,
- 98.0,
- 94.0,
- 126.0,
- 101.0,
- 98.0,
- 99.0,
- 79.0,
- 99.0,
- 80.0,
- 105.0,
- 104.0,
- 106.0,
- 107.0,
- 123.0,
- 109.0,
- 104.0,
- 122.0,
- 122.0,
- 107.0,
- 102.0,
- 103.0,
- 92.0,
- 111.0,
- 112.0,
- 102.0,
- 127.0,
- 96.0,
- 112.0,
- 106.0,
- 104.0,
- 90.0,
- 86.0,
- 96.0,
- 112.0,
- 115.0,
- 100.0,
- 128.0,
- 109.0,
- 107.0,
- 109.0,
- 101.0,
- 99.0,
- 95.0,
- 99.0,
- 127.0,
- 102.0,
- 118.0,
- 107.0,
- 94.0,
- 130.0,
- 89.0,
- 101.0,
- 103.0,
- 81.0,
- 92.0,
- 105.0,
- 102.0,
- 95.0,
- 99.0,
- 122.0,
- 110.0,
- 97.0,
- 107.0,
- 114.0,
- 105.0,
- 125.0,
- 91.0,
- 111.0,
- 108.0,
- 85.0,
- 105.0,
- 118.0,
- 113.0,
- 100.0,
- 101.0,
- 120.0,
- 98.0,
- 98.0,
- 92.0,
- 93.0,
- 107.0,
- 119.0,
- 132.0,
- 132.0,
- 100.0,
- 120.0,
- 112.0,
- 114.0,
- 92.0,
- 88.0,
- 104.0,
- 120.0,
- 125.0,
- 106.0,
- 99.0,
- 125.0,
- 106.0,
- 94.0,
- 138.0,
- 104.0,
- 106.0,
- 111.0,
- 95.0,
- 109.0,
- 116.0,
- 108.0,
- 114.0,
- 110.0,
- 106.0,
- 123.0,
- 102.0,
- 134.0,
- 125.0,
- 112.0,
- 102.0,
- 119.0,
- 111.0,
- 102.0,
- 120.0,
- 110.0,
- 102.0,
- 124.0,
- 106.0,
- 115.0,
- 112.0,
- 100.0,
- 127.0,
- 123.0,
- 112.0,
- 118.0,
- 113.0,
- 112.0,
- 92.0,
- 111.0,
- 112.0,
- 85.0,
- 87.0,
- 132.0,
- 118.0,
- 100.0,
- 99.0,
- 87.0,
- 114.0,
- 108.0,
- 131.0,
- 120.0,
- 127.0,
- 113.0,
- 111.0,
- 102.0,
- 126.0,
- 117.0,
- 132.0,
- 103.0,
- 120.0,
- 114.0,
- 120.0,
- 101.0,
- 107.0,
- 106.0,
- 124.0,
- 137.0,
- 117.0
- ]
- },
- "iteration-time": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 18.53864,
- 0.95588,
- 0.98728,
- 0.9479,
- 0.9533,
- 0.94063,
- 0.94265,
- 0.94346,
- 0.94,
- 0.94193,
- 0.94448,
- 0.94,
- 0.94178,
- 0.95318,
- 0.94344,
- 0.94282,
- 0.93703,
- 0.9594,
- 0.93761,
- 0.93676,
- 0.94059,
- 0.94063,
- 0.94496,
- 0.93892,
- 0.9449,
- 0.95488,
- 0.94465,
- 0.95353,
- 0.94176,
- 0.95336,
- 0.95058,
- 0.98447,
- 0.94686,
- 0.98878,
- 0.95268,
- 0.94258,
- 0.94399,
- 0.93889,
- 0.94158,
- 0.94559,
- 0.97363,
- 0.95633,
- 0.95485,
- 0.96508,
- 0.94859,
- 0.94248,
- 0.94135,
- 0.93696,
- 0.946,
- 0.93538,
- 0.94544,
- 0.9507,
- 0.94314,
- 0.94298,
- 0.93954,
- 0.93721,
- 0.94889,
- 0.93927,
- 0.93203,
- 0.93941,
- 0.94011,
- 0.94392,
- 0.94659,
- 0.94179,
- 0.94991,
- 0.94921,
- 0.94542,
- 0.94419,
- 0.95155,
- 0.94371,
- 0.95683,
- 0.93985,
- 0.94159,
- 0.95114,
- 0.94329,
- 0.93652,
- 0.94172,
- 0.94478,
- 0.94508,
- 0.9586,
- 0.94289,
- 0.94346,
- 0.9572,
- 0.94962,
- 0.95027,
- 0.94705,
- 0.94819,
- 0.94109,
- 0.94809,
- 0.95085,
- 0.95144,
- 0.94471,
- 0.94746,
- 0.96865,
- 0.96892,
- 0.94386,
- 0.96563,
- 0.9431,
- 0.94067,
- 0.94592,
- 0.95403,
- 0.96047,
- 0.95154,
- 0.94462,
- 0.94607,
- 0.95516,
- 0.94081,
- 0.95113,
- 0.93236,
- 0.94367,
- 0.94485,
- 0.94482,
- 0.94763,
- 0.95326,
- 0.9491,
- 0.94093,
- 0.94773,
- 0.95426,
- 0.96206,
- 0.94813,
- 0.97033,
- 0.94237,
- 0.94199,
- 0.94838,
- 0.95178,
- 0.94135,
- 0.94579,
- 0.93951,
- 0.94911,
- 0.95218,
- 0.94178,
- 0.94851,
- 0.9509,
- 0.94999,
- 0.9493,
- 0.94828,
- 0.94978,
- 0.94476,
- 0.94705,
- 0.95521,
- 0.95104,
- 0.94511,
- 0.94837,
- 0.94912,
- 0.94671,
- 0.9459,
- 0.94956,
- 0.95319,
- 0.95821,
- 0.9485,
- 0.95174,
- 0.94765,
- 0.96003,
- 0.94582,
- 0.95184,
- 0.95612,
- 0.95158,
- 0.98107,
- 0.94641,
- 0.95282,
- 0.95172,
- 0.9491,
- 0.94978,
- 0.94789,
- 0.94792,
- 0.94025,
- 0.93956,
- 0.93183,
- 0.93056,
- 0.93823,
- 0.93333,
- 0.96058,
- 0.93797,
- 0.93793,
- 0.94018,
- 0.93813,
- 0.93817,
- 0.95695,
- 0.93824,
- 0.94699,
- 0.94388,
- 0.94587,
- 0.95454,
- 0.94299,
- 0.94677,
- 0.9404,
- 0.93396,
- 0.9321,
- 0.93528,
- 0.94403,
- 0.9477,
- 0.94225,
- 0.94179,
- 0.93868,
- 0.95141,
- 0.94067,
- 0.94856,
- 0.94009,
- 0.9422,
- 0.94504,
- 0.94152,
- 0.96476,
- 0.94531,
- 0.94649,
- 0.94942,
- 0.94029,
- 1.0097,
- 0.94409,
- 0.95112,
- 0.94884,
- 0.95061,
- 0.95583,
- 0.95095,
- 0.95022,
- 0.95212,
- 0.94448,
- 0.94873,
- 0.95662,
- 0.96522,
- 0.94569,
- 0.94838,
- 0.94514,
- 0.94892,
- 0.95044,
- 0.96233,
- 0.95231,
- 0.94812,
- 0.94006,
- 0.94158,
- 0.943,
- 0.94399,
- 0.94347,
- 0.95689,
- 0.95405,
- 0.95444,
- 0.94624,
- 0.93701,
- 0.94525,
- 0.94239,
- 0.94211,
- 0.94566,
- 0.9479,
- 0.94417,
- 0.94624,
- 0.94886,
- 0.96213,
- 0.94232,
- 0.94635,
- 0.94811,
- 0.94497,
- 0.94019,
- 0.93701,
- 0.94403,
- 0.93885,
- 0.94132,
- 0.94052,
- 0.93236,
- 0.95086,
- 0.9407,
- 0.94154,
- 0.9449,
- 0.94425,
- 0.94813,
- 0.94489,
- 0.94435,
- 0.94217,
- 0.94314,
- 0.93934,
- 0.95872,
- 0.94958,
- 0.94957,
- 0.95599,
- 0.95388,
- 0.95606,
- 0.94371,
- 0.94632,
- 0.94553,
- 0.95892,
- 0.953,
- 0.94963,
- 0.94155,
- 0.95559,
- 0.94947,
- 0.94817,
- 0.95593,
- 0.95566,
- 0.94408,
- 0.95495,
- 0.949,
- 0.95776,
- 0.95699,
- 0.95315,
- 0.95048,
- 0.95401,
- 0.96139,
- 0.97114,
- 0.94534,
- 0.94445,
- 0.94874,
- 0.94385,
- 0.95005,
- 0.95314,
- 0.95076,
- 0.94059,
- 0.95293,
- 0.95445,
- 0.95102,
- 0.9472,
- 0.93973,
- 0.94443,
- 0.9388,
- 0.94286,
- 0.94317,
- 0.94195,
- 0.9419,
- 0.94506,
- 0.95338,
- 0.94558,
- 0.94449,
- 0.94354,
- 0.93761,
- 0.95019,
- 0.93809,
- 0.94284,
- 0.94196,
- 0.93931,
- 0.93559,
- 0.94288,
- 0.93906,
- 0.93847,
- 0.93964,
- 0.93919,
- 0.94356,
- 0.95154,
- 0.9405,
- 0.94607,
- 0.94801,
- 0.94918,
- 0.9443,
- 0.97237,
- 0.94775,
- 0.94762,
- 0.94701,
- 0.94383,
- 0.95085,
- 0.95617,
- 0.95529,
- 0.95966,
- 0.95961,
- 0.96501,
- 0.95501,
- 0.94915,
- 0.94926,
- 0.94879,
- 0.95826,
- 0.95473,
- 0.95968,
- 0.94356,
- 0.96027,
- 0.95401,
- 0.94791,
- 0.95295,
- 0.947,
- 0.95173,
- 0.94958,
- 0.94613,
- 0.94941,
- 0.94801,
- 0.9486,
- 0.96463,
- 0.94302,
- 0.95219,
- 0.9442,
- 0.94287,
- 0.93815,
- 0.93529,
- 0.93952,
- 0.94162,
- 0.93707,
- 0.93837,
- 0.94009,
- 0.94154,
- 0.94407,
- 0.94597,
- 0.94076,
- 0.93482,
- 0.93691,
- 0.94139,
- 0.94406,
- 0.94631,
- 0.93728,
- 0.92955,
- 0.94906,
- 0.94489,
- 0.94899,
- 0.94887,
- 0.94665,
- 0.94811,
- 0.93798,
- 0.94313
- ]
- }
-}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp/golden_values_lts_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp/golden_values_lts_dgx_a100.json
deleted file mode 100644
index f822a205e19..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp/golden_values_lts_dgx_a100.json
+++ /dev/null
@@ -1,1223 +0,0 @@
-{
- "lm loss": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 10.84435,
- 10.87318,
- 10.85036,
- 10.81075,
- 10.64476,
- 10.63865,
- 10.4284,
- 10.13527,
- 9.9354,
- 9.83535,
- 9.58564,
- 9.84799,
- 9.88584,
- 9.63126,
- 9.79019,
- 9.51136,
- 9.45967,
- 9.65536,
- 9.38991,
- 9.3393,
- 9.24938,
- 9.15121,
- 9.1819,
- 9.00438,
- 9.19827,
- 9.06667,
- 9.1611,
- 9.16974,
- 9.30047,
- 8.98931,
- 8.9295,
- 9.05025,
- 9.04643,
- 8.66023,
- 8.72503,
- 8.75641,
- 8.69453,
- 8.74311,
- 8.66664,
- 8.77265,
- 8.67046,
- 8.86117,
- 8.84289,
- 8.50887,
- 8.39866,
- 8.43817,
- 8.49539,
- 8.39331,
- 8.44014,
- 8.59211,
- 8.37558,
- 8.19954,
- 8.23308,
- 8.22973,
- 8.27486,
- 7.9203,
- 8.09935,
- 7.89759,
- 8.25172,
- 8.23421,
- 8.00968,
- 7.97527,
- 7.92604,
- 7.74403,
- 7.74728,
- 7.64954,
- 7.51978,
- 7.9104,
- 7.70203,
- 7.45557,
- 7.74663,
- 7.7747,
- 7.54395,
- 7.30276,
- 7.45598,
- 7.34312,
- 7.46591,
- 7.22838,
- 7.63706,
- 7.28267,
- 7.34901,
- 7.21386,
- 7.21177,
- 7.41978,
- 7.17382,
- 7.2822,
- 6.99443,
- 7.00278,
- 7.03963,
- 7.13669,
- 6.82176,
- 6.98519,
- 7.08886,
- 6.99826,
- 6.87461,
- 6.75718,
- 6.99116,
- 7.06112,
- 6.70481,
- 6.58484,
- 6.72791,
- 6.74611,
- 6.73451,
- 6.73883,
- 6.6589,
- 6.40659,
- 6.63739,
- 6.6201,
- 6.44607,
- 6.62819,
- 6.74266,
- 6.6102,
- 6.72607,
- 6.69279,
- 6.6261,
- 6.50591,
- 6.59661,
- 6.40511,
- 6.66302,
- 6.24641,
- 6.25042,
- 6.30258,
- 6.38946,
- 6.34694,
- 6.45156,
- 6.2927,
- 6.33962,
- 6.23686,
- 6.20391,
- 6.39902,
- 6.32867,
- 6.32319,
- 6.16976,
- 6.16361,
- 6.24291,
- 6.38627,
- 6.2076,
- 6.15571,
- 6.1854,
- 6.12408,
- 6.07117,
- 6.07793,
- 6.26449,
- 6.41645,
- 6.26318,
- 6.30431,
- 6.10357,
- 6.18374,
- 6.00783,
- 6.03849,
- 5.96044,
- 6.26013,
- 6.19494,
- 5.97729,
- 5.79578,
- 6.1331,
- 5.85925,
- 6.11082,
- 5.79246,
- 6.16831,
- 6.14892,
- 6.08853,
- 5.92954,
- 6.11667,
- 5.94404,
- 6.19642,
- 5.89309,
- 5.78869,
- 5.77689,
- 5.68542,
- 6.01319,
- 5.99761,
- 6.06692,
- 5.88893,
- 6.04105,
- 5.96721,
- 5.99332,
- 5.99407,
- 5.95322,
- 5.84284,
- 5.95079,
- 5.62035,
- 5.70822,
- 5.89257,
- 5.84404,
- 5.86509,
- 5.76428,
- 5.83817,
- 5.72742,
- 5.56185,
- 5.72363,
- 5.62165,
- 5.83076,
- 5.60152,
- 5.70824,
- 5.70544,
- 5.90203,
- 5.64105,
- 5.84826,
- 5.73964,
- 5.86591,
- 5.32604,
- 5.89223,
- 5.87356,
- 5.85147,
- 5.41,
- 5.41144,
- 5.62864,
- 5.59674,
- 5.48661,
- 5.57868,
- 5.67447,
- 5.47953,
- 5.74541,
- 5.51107,
- 5.59383,
- 5.62438,
- 5.62002,
- 5.52107,
- 5.61786,
- 5.67207,
- 5.6824,
- 5.58833,
- 5.66064,
- 5.37433,
- 5.6798,
- 5.63448,
- 5.42498,
- 5.58338,
- 5.63097,
- 5.55613,
- 5.34386,
- 5.53696,
- 5.48795,
- 5.48091,
- 5.37734,
- 5.55326,
- 5.60019,
- 5.38949,
- 5.5279,
- 5.48792,
- 5.33294,
- 5.50621,
- 5.40686,
- 5.44259,
- 5.31539,
- 5.06376,
- 5.47807,
- 5.5693,
- 5.71381,
- 5.41187,
- 5.59881,
- 5.63378,
- 5.2309,
- 5.26996,
- 5.39128,
- 5.39766,
- 5.32837,
- 5.49524,
- 5.18234,
- 5.29608,
- 5.24551,
- 5.37455,
- 5.25382,
- 5.44198,
- 5.53542,
- 5.30722,
- 5.4305,
- 5.33574,
- 5.07255,
- 5.30787,
- 5.24998,
- 5.30133,
- 5.11033,
- 5.27279,
- 5.26164,
- 5.47438,
- 5.15836,
- 5.26302,
- 5.20727,
- 5.35287,
- 4.97954,
- 4.90839,
- 5.32324,
- 5.38545,
- 5.22544,
- 5.31832,
- 5.1045,
- 5.16052,
- 5.26033,
- 5.06436,
- 5.26,
- 5.06647,
- 5.33914,
- 5.24433,
- 5.14664,
- 5.24337,
- 5.03905,
- 5.31384,
- 5.05093,
- 5.02403,
- 5.13908,
- 5.11049,
- 5.27154,
- 5.14863,
- 5.27243,
- 5.09211,
- 5.09214,
- 5.24408,
- 5.32506,
- 5.25134,
- 5.19195,
- 5.14156,
- 5.28838,
- 4.95217,
- 5.20555,
- 5.09208,
- 5.30144,
- 5.17197,
- 5.18544,
- 5.11186,
- 4.98156,
- 4.99246,
- 5.22268,
- 5.31003,
- 5.09805,
- 5.05635,
- 4.91749,
- 5.12083,
- 5.11431,
- 4.92685,
- 5.33318,
- 5.02149,
- 5.09798,
- 5.16452,
- 5.003,
- 5.06512,
- 5.06538,
- 4.99155,
- 5.08009,
- 5.16075,
- 4.97693,
- 5.18415,
- 4.92412,
- 4.9196,
- 5.06212,
- 4.99168,
- 4.90728,
- 4.77422,
- 4.94399,
- 5.11441,
- 5.01167,
- 5.01683,
- 5.32789,
- 4.95546,
- 4.99161,
- 5.0459,
- 4.81109,
- 4.7342,
- 4.99359,
- 5.04093,
- 4.87128,
- 4.95515,
- 5.04762,
- 5.02569,
- 4.81796,
- 4.8971,
- 4.90335,
- 4.82861,
- 4.73834,
- 5.00766,
- 4.75352,
- 5.20734,
- 4.79121,
- 4.99076,
- 4.73247,
- 4.782,
- 4.81736,
- 4.64772,
- 4.65226,
- 4.84032,
- 4.80478,
- 4.79458,
- 4.91773,
- 4.88236,
- 4.92733,
- 4.77215,
- 4.87882,
- 4.7305,
- 4.91488,
- 4.95406,
- 4.8724,
- 4.70482,
- 4.77933,
- 4.89858,
- 4.70781,
- 4.85495,
- 4.69185,
- 4.69004,
- 4.64291
- ]
- },
- "num-zeros": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 78.0,
- 81.0,
- 63.0,
- 62.0,
- 74.0,
- 67.0,
- 96.0,
- 102.0,
- 121.0,
- 141.0,
- 102.0,
- 133.0,
- 149.0,
- 150.0,
- 194.0,
- 155.0,
- 151.0,
- 191.0,
- 179.0,
- 169.0,
- 155.0,
- 187.0,
- 186.0,
- 195.0,
- 184.0,
- 160.0,
- 216.0,
- 201.0,
- 146.0,
- 147.0,
- 163.0,
- 147.0,
- 125.0,
- 170.0,
- 114.0,
- 185.0,
- 171.0,
- 195.0,
- 182.0,
- 185.0,
- 149.0,
- 175.0,
- 173.0,
- 175.0,
- 187.0,
- 170.0,
- 188.0,
- 173.0,
- 156.0,
- 216.0,
- 201.0,
- 172.0,
- 211.0,
- 171.0,
- 173.0,
- 194.0,
- 163.0,
- 159.0,
- 226.0,
- 243.0,
- 167.0,
- 158.0,
- 197.0,
- 183.0,
- 197.0,
- 250.0,
- 222.0,
- 204.0,
- 183.0,
- 188.0,
- 225.0,
- 262.0,
- 197.0,
- 237.0,
- 209.0,
- 240.0,
- 237.0,
- 241.0,
- 253.0,
- 210.0,
- 218.0,
- 226.0,
- 196.0,
- 229.0,
- 204.0,
- 174.0,
- 185.0,
- 196.0,
- 174.0,
- 186.0,
- 198.0,
- 183.0,
- 213.0,
- 204.0,
- 212.0,
- 154.0,
- 195.0,
- 191.0,
- 168.0,
- 162.0,
- 155.0,
- 186.0,
- 170.0,
- 178.0,
- 133.0,
- 154.0,
- 161.0,
- 158.0,
- 155.0,
- 189.0,
- 176.0,
- 160.0,
- 148.0,
- 161.0,
- 147.0,
- 141.0,
- 142.0,
- 102.0,
- 160.0,
- 139.0,
- 160.0,
- 120.0,
- 120.0,
- 148.0,
- 144.0,
- 95.0,
- 100.0,
- 137.0,
- 114.0,
- 139.0,
- 133.0,
- 138.0,
- 134.0,
- 113.0,
- 125.0,
- 130.0,
- 111.0,
- 128.0,
- 114.0,
- 115.0,
- 115.0,
- 110.0,
- 112.0,
- 129.0,
- 124.0,
- 125.0,
- 123.0,
- 125.0,
- 121.0,
- 115.0,
- 129.0,
- 109.0,
- 119.0,
- 123.0,
- 106.0,
- 113.0,
- 115.0,
- 137.0,
- 131.0,
- 135.0,
- 128.0,
- 118.0,
- 123.0,
- 97.0,
- 115.0,
- 123.0,
- 112.0,
- 105.0,
- 115.0,
- 120.0,
- 112.0,
- 91.0,
- 89.0,
- 96.0,
- 121.0,
- 127.0,
- 106.0,
- 114.0,
- 115.0,
- 111.0,
- 99.0,
- 103.0,
- 94.0,
- 146.0,
- 102.0,
- 113.0,
- 104.0,
- 114.0,
- 117.0,
- 116.0,
- 111.0,
- 135.0,
- 117.0,
- 126.0,
- 98.0,
- 102.0,
- 99.0,
- 100.0,
- 101.0,
- 106.0,
- 125.0,
- 92.0,
- 121.0,
- 123.0,
- 106.0,
- 115.0,
- 88.0,
- 95.0,
- 123.0,
- 98.0,
- 99.0,
- 81.0,
- 95.0,
- 118.0,
- 90.0,
- 102.0,
- 109.0,
- 91.0,
- 106.0,
- 92.0,
- 114.0,
- 105.0,
- 91.0,
- 97.0,
- 107.0,
- 95.0,
- 97.0,
- 100.0,
- 97.0,
- 117.0,
- 119.0,
- 104.0,
- 85.0,
- 113.0,
- 115.0,
- 118.0,
- 94.0,
- 103.0,
- 112.0,
- 94.0,
- 89.0,
- 111.0,
- 119.0,
- 114.0,
- 111.0,
- 104.0,
- 121.0,
- 122.0,
- 123.0,
- 106.0,
- 109.0,
- 106.0,
- 115.0,
- 118.0,
- 124.0,
- 91.0,
- 98.0,
- 110.0,
- 106.0,
- 104.0,
- 104.0,
- 100.0,
- 96.0,
- 87.0,
- 104.0,
- 115.0,
- 99.0,
- 114.0,
- 126.0,
- 108.0,
- 128.0,
- 110.0,
- 109.0,
- 115.0,
- 103.0,
- 127.0,
- 86.0,
- 107.0,
- 98.0,
- 107.0,
- 110.0,
- 118.0,
- 88.0,
- 109.0,
- 113.0,
- 90.0,
- 92.0,
- 100.0,
- 110.0,
- 103.0,
- 104.0,
- 119.0,
- 98.0,
- 121.0,
- 113.0,
- 121.0,
- 97.0,
- 109.0,
- 87.0,
- 120.0,
- 136.0,
- 123.0,
- 100.0,
- 96.0,
- 111.0,
- 116.0,
- 97.0,
- 108.0,
- 134.0,
- 93.0,
- 102.0,
- 93.0,
- 101.0,
- 126.0,
- 102.0,
- 100.0,
- 96.0,
- 123.0,
- 111.0,
- 123.0,
- 89.0,
- 106.0,
- 118.0,
- 125.0,
- 99.0,
- 121.0,
- 92.0,
- 109.0,
- 123.0,
- 126.0,
- 96.0,
- 124.0,
- 135.0,
- 94.0,
- 107.0,
- 117.0,
- 114.0,
- 95.0,
- 123.0,
- 103.0,
- 119.0,
- 124.0,
- 115.0,
- 115.0,
- 115.0,
- 101.0,
- 115.0,
- 88.0,
- 106.0,
- 105.0,
- 122.0,
- 125.0,
- 131.0,
- 112.0,
- 130.0,
- 117.0,
- 102.0,
- 94.0,
- 129.0,
- 115.0,
- 130.0,
- 92.0,
- 126.0,
- 105.0,
- 125.0,
- 107.0,
- 93.0,
- 137.0,
- 113.0,
- 93.0,
- 104.0,
- 106.0,
- 89.0,
- 126.0,
- 97.0,
- 92.0,
- 122.0,
- 105.0,
- 107.0,
- 121.0,
- 111.0,
- 122.0,
- 118.0,
- 137.0,
- 130.0,
- 124.0,
- 119.0,
- 98.0,
- 117.0,
- 92.0,
- 101.0,
- 119.0,
- 112.0,
- 128.0,
- 104.0,
- 125.0,
- 94.0,
- 105.0,
- 97.0,
- 121.0
- ]
- },
- "iteration-time": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 23.34406,
- 1.17039,
- 1.173,
- 1.16494,
- 1.16483,
- 1.16575,
- 1.16204,
- 1.15812,
- 1.15652,
- 1.16643,
- 1.16078,
- 1.15939,
- 1.17115,
- 1.16564,
- 1.17859,
- 1.17606,
- 1.17037,
- 1.19888,
- 1.16983,
- 1.16754,
- 1.16743,
- 1.17055,
- 1.18175,
- 1.16888,
- 1.17043,
- 1.17177,
- 1.17337,
- 1.15677,
- 1.1758,
- 1.17204,
- 1.16365,
- 1.17047,
- 1.16702,
- 1.16606,
- 1.16955,
- 1.31288,
- 1.17263,
- 1.16582,
- 1.17041,
- 1.16844,
- 1.17019,
- 1.1644,
- 1.16909,
- 1.17402,
- 1.16538,
- 1.16778,
- 1.17243,
- 1.17766,
- 1.16747,
- 1.17131,
- 1.16449,
- 1.1653,
- 1.16464,
- 1.15861,
- 1.16313,
- 1.16527,
- 1.17028,
- 1.15912,
- 1.17959,
- 1.1734,
- 1.16816,
- 1.16551,
- 1.16725,
- 1.16506,
- 1.16617,
- 1.16308,
- 1.1618,
- 1.16946,
- 1.16336,
- 1.16426,
- 1.17998,
- 1.16623,
- 1.17535,
- 1.16411,
- 1.16492,
- 1.17299,
- 1.1715,
- 1.17869,
- 1.1699,
- 1.16461,
- 1.16863,
- 1.16382,
- 1.17047,
- 1.16995,
- 1.1666,
- 1.16418,
- 1.16868,
- 1.16579,
- 1.15628,
- 1.16798,
- 1.17082,
- 1.17331,
- 1.17053,
- 1.17126,
- 1.17403,
- 1.16881,
- 1.16136,
- 1.16745,
- 1.16624,
- 1.16489,
- 1.18239,
- 1.17464,
- 1.1711,
- 1.17745,
- 1.17608,
- 1.18067,
- 1.18708,
- 1.18901,
- 1.18633,
- 1.18603,
- 1.1786,
- 1.19418,
- 1.17856,
- 1.18123,
- 1.1837,
- 1.18369,
- 1.18422,
- 1.18768,
- 1.19076,
- 1.1812,
- 1.19114,
- 1.18605,
- 1.14129,
- 1.1575,
- 1.14066,
- 1.17639,
- 1.18425,
- 1.17001,
- 1.19176,
- 1.19108,
- 1.1768,
- 1.18485,
- 1.20499,
- 1.19189,
- 1.18064,
- 1.17787,
- 1.19195,
- 1.19927,
- 1.23073,
- 1.18677,
- 1.19046,
- 1.18187,
- 1.18937,
- 1.21167,
- 1.18566,
- 1.16935,
- 1.1701,
- 1.17709,
- 1.19274,
- 1.17738,
- 1.17826,
- 1.1664,
- 1.17572,
- 1.16895,
- 1.16753,
- 1.17343,
- 1.16903,
- 1.16971,
- 1.16984,
- 1.1811,
- 1.18941,
- 1.17477,
- 1.1806,
- 1.18288,
- 1.1785,
- 1.17701,
- 1.17703,
- 1.17515,
- 1.18327,
- 1.17311,
- 1.1815,
- 1.17316,
- 1.17856,
- 1.17628,
- 1.17449,
- 1.17852,
- 1.17782,
- 1.17168,
- 1.17438,
- 1.17469,
- 1.17762,
- 1.17228,
- 1.17742,
- 1.17533,
- 1.18953,
- 1.18268,
- 1.18624,
- 1.18127,
- 1.20293,
- 1.18602,
- 1.16879,
- 1.17376,
- 1.17027,
- 1.17957,
- 1.17958,
- 1.16575,
- 1.15516,
- 1.16934,
- 1.16302,
- 1.15534,
- 1.1531,
- 1.15489,
- 1.15748,
- 1.1576,
- 1.15839,
- 1.16766,
- 1.15465,
- 1.15694,
- 1.18582,
- 1.16999,
- 1.1796,
- 1.16425,
- 1.17182,
- 1.15726,
- 1.1736,
- 1.17724,
- 1.17386,
- 1.17529,
- 1.17695,
- 1.17936,
- 1.18069,
- 1.19431,
- 1.18189,
- 1.18116,
- 1.19235,
- 1.17797,
- 1.18177,
- 1.18354,
- 1.18555,
- 1.18237,
- 1.17595,
- 1.17961,
- 1.17756,
- 1.18234,
- 1.18358,
- 1.19028,
- 1.18217,
- 1.18209,
- 1.17902,
- 1.18184,
- 1.18224,
- 1.19588,
- 1.17959,
- 1.18437,
- 1.18271,
- 1.18035,
- 1.18619,
- 1.18573,
- 1.18876,
- 1.18917,
- 1.18496,
- 1.18739,
- 1.19656,
- 1.1969,
- 1.19473,
- 1.19324,
- 1.19377,
- 1.18283,
- 1.18739,
- 1.18158,
- 1.16288,
- 1.16683,
- 1.16152,
- 1.16074,
- 1.1663,
- 1.16591,
- 1.17901,
- 1.16145,
- 1.17191,
- 1.17179,
- 1.16773,
- 1.17832,
- 1.1581,
- 1.16003,
- 1.15189,
- 1.15472,
- 1.16209,
- 1.16107,
- 1.1599,
- 1.16155,
- 1.16286,
- 1.17,
- 1.16147,
- 1.15785,
- 1.16164,
- 1.15976,
- 1.15927,
- 1.57688,
- 1.17603,
- 1.17314,
- 1.19224,
- 1.17822,
- 1.1882,
- 1.176,
- 1.17781,
- 1.17984,
- 1.17471,
- 1.17492,
- 1.18073,
- 1.17692,
- 1.17325,
- 1.1761,
- 1.17727,
- 1.17111,
- 1.17951,
- 1.17441,
- 1.1568,
- 1.17807,
- 1.17874,
- 1.17104,
- 1.2905,
- 1.17805,
- 1.17121,
- 1.17166,
- 1.17232,
- 1.17459,
- 1.17913,
- 1.1708,
- 1.17391,
- 1.17531,
- 1.17594,
- 1.15935,
- 1.18042,
- 1.19,
- 1.17793,
- 1.17594,
- 1.17602,
- 1.17535,
- 1.17812,
- 1.17362,
- 1.17173,
- 1.17584,
- 1.17377,
- 1.17806,
- 1.17619,
- 1.17216,
- 1.18278,
- 1.18527,
- 1.17597,
- 1.18145,
- 1.17917,
- 1.18892,
- 1.17329,
- 1.17202,
- 1.17508,
- 1.17162,
- 1.17129,
- 1.17396,
- 1.1761,
- 1.17031,
- 1.17211,
- 1.17692,
- 1.17391,
- 1.17361,
- 1.17899,
- 1.1729,
- 1.18055,
- 1.17626,
- 1.18141,
- 1.17443,
- 1.18144,
- 1.17746,
- 1.17164,
- 1.17448,
- 1.17469,
- 1.17222,
- 1.16882,
- 1.17741,
- 1.1801,
- 1.17277,
- 1.17196,
- 1.17407,
- 1.17266,
- 1.18371,
- 1.16781,
- 1.17137,
- 1.18646,
- 1.17403,
- 1.17343,
- 1.18012,
- 1.19053,
- 1.18436,
- 1.18323,
- 1.18326,
- 1.19376,
- 1.18423,
- 1.18445,
- 1.18876,
- 1.18424,
- 1.18265,
- 1.18961,
- 1.18624,
- 1.18422,
- 1.19539,
- 1.18601,
- 1.18424,
- 1.18663,
- 1.19269,
- 1.18535,
- 1.18709
- ]
- }
-}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp/model_config.yaml
deleted file mode 100644
index 8d85af9b7d4..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp/model_config.yaml
+++ /dev/null
@@ -1,54 +0,0 @@
-ENV_VARS:
- CUDA_DEVICE_MAX_CONNECTIONS: 1
- NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
- NVTE_FUSED_ATTN: 0
- NCCL_ALGO: Ring
- CUBLAS_WORKSPACE_CONFIG: :4096:8
-MODEL_ARGS:
- --num-layers: 12
- --hidden-size: 512
- --num-attention-heads: 8
- --log-params-norm: true
- --log-num-zeros-in-grad: true
- --log-validation-ppl-to-tensorboard: true
- --log-timers-to-tensorboard: true
- --tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
- --seq-length: 1024
- --max-position-embeddings: 1024
- --train-iters: 2000
- --timing-log-level: 0
- --lr-decay-iters: 320000
- --save: ${CHECKPOINT_SAVE_PATH}
- --load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
- --split: 949,50,1
- --distributed-backend: nccl
- --lr: 0.00015
- --lr-decay-style: cosine
- --min-lr: 1.0e-5
- --weight-decay: 1e-2
- --clip-grad: 1.0
- --lr-warmup-fraction: .01
- --log-interval: 1
- --save-interval: 10000
- --eval-interval: 1000
- --eval-iters: 10
- --transformer-impl: transformer_engine
- --tensor-model-parallel-size: 2
- --pipeline-model-parallel-size: 2
- --deterministic-mode: true
- --no-gradient-accumulation-fusion: true
- --fp8-format: hybrid
- --fp8-amax-history-len: 1024
- --fp8-amax-compute-algo: max
- --attention-softmax-in-fp32: true
- --ckpt-format: torch_dist
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
- --data-cache-path: ${DATA_CACHE_PATH}
- --bf16: true
- --log-memory-to-tensorboard: true
-TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp_sp/golden_values_dev_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp_sp/golden_values_dev_dgx_a100.json
deleted file mode 100644
index 0c3d0a67e66..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp_sp/golden_values_dev_dgx_a100.json
+++ /dev/null
@@ -1 +0,0 @@
-{"forward-backward-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [21.65799, 1.57316, 1.56036, 1.56197, 1.56002, 1.57036, 1.57498, 1.57179, 1.57223, 1.56447, 1.57065, 1.57253, 1.56833, 1.57388, 1.58074, 1.57741, 1.58388, 1.58795, 1.5903, 1.58075, 1.57656, 1.58312, 1.57306, 1.57348, 1.58999, 1.57118, 1.56942, 1.57642, 1.58455, 1.57798, 1.57753, 1.5848, 1.57952, 1.57466, 1.5634, 1.5759, 1.57055, 1.56518, 1.64863, 1.56915, 1.57234, 1.57176, 1.59307, 1.58513, 1.59397, 1.59455, 1.58862, 1.58627, 1.57781, 1.5836, 1.59175, 1.58787, 1.58531, 1.56743, 1.56768, 1.57061, 1.57416, 1.56759, 1.5696, 1.57589, 1.57313, 1.571, 1.58684, 1.58081, 1.58172, 1.57572, 1.58332, 1.58369, 1.5742, 1.58521, 1.57857, 1.57985, 1.59598, 1.58564, 1.58954, 1.58921, 1.58516, 1.58693, 1.58278, 1.58855, 1.58036, 1.58425, 1.57404, 1.56846, 1.57061, 1.57471, 1.57444, 1.57552, 1.58566, 1.59602, 1.57809, 1.59795, 1.58523, 1.58552, 1.58948, 1.5857, 1.58918, 1.58406, 1.58274, 1.58292, 1.5878, 1.57929, 1.57852, 1.57229, 1.58645, 1.58337, 1.57647, 1.56993, 1.57461, 1.57583, 1.57981, 1.58228, 1.58026, 1.58041, 1.57147, 1.57774, 1.57198, 1.56711, 1.56216, 1.57948, 1.57013, 1.5652, 1.57538, 1.59385, 1.58672, 1.57603, 1.57508, 1.58044, 1.56643, 1.57319, 1.56412, 1.56703, 1.57342, 1.57169, 1.58538, 1.57905, 1.57735, 1.5713, 1.56908, 1.56945, 1.57129, 1.5672, 1.57775, 1.58937, 1.59019, 1.5751, 1.58049, 1.58855, 1.58446, 1.59003, 1.58787, 1.58871, 1.59524, 1.59317, 1.59223, 1.59165, 1.58901, 1.59193, 1.5866, 1.59184, 1.59323, 1.59575, 1.58596, 1.59591, 1.58463, 1.58779, 1.59392, 1.59398, 1.59893, 1.5974, 1.59446, 1.58691, 1.58241, 1.58352, 1.59639, 1.58013, 1.59181, 1.58597, 1.58425, 1.58787, 1.58445, 1.58197, 1.58869, 1.5852, 1.58751, 1.5889, 1.58458, 1.57701, 1.58666, 1.584, 1.57776, 1.58858, 1.58222, 1.58721, 1.60018, 1.59115, 1.59271, 1.58842, 1.59023, 1.58933, 1.57882, 1.59135, 1.5868, 1.57554, 1.58258, 1.58243, 1.58389, 1.58426, 1.5849, 1.58819, 1.58199, 1.58031, 1.58504, 1.58277, 1.5863, 1.57949, 1.58628, 1.58781, 1.58443, 1.57924, 1.58531, 1.59139, 1.58724, 1.58582, 1.59165, 1.58221, 1.58782, 1.59196, 1.58549, 1.58279, 1.59669, 1.58729, 1.58776, 1.58434, 1.58643, 1.57486, 1.58484, 1.57875, 1.58178, 1.58296, 1.57564, 1.57269, 1.73935, 1.63419, 1.58507, 1.59194, 1.5809, 1.60067, 1.59666, 1.59408, 1.59512, 1.68832, 1.59093, 1.57923, 1.58167, 1.5802, 1.58149, 1.59105, 1.58674, 1.59021, 1.59488, 1.60007, 1.59231, 1.59296, 1.59159, 1.588, 1.58471, 1.58515, 1.58686, 1.58415, 1.58593, 1.58185, 1.58805, 1.59063, 1.58623, 1.58868, 1.5863, 1.58712, 1.58387, 1.58919, 1.58738, 1.58618, 1.58901, 1.58673, 1.5896, 1.59327, 1.58995, 1.59034, 1.59043, 1.58508, 1.58835, 1.59575, 1.59028, 1.58788, 1.59495, 1.59031, 1.58998, 1.58896, 1.59037, 1.58923, 1.59259, 1.59082, 1.59843, 1.59394, 1.59716, 1.58592, 1.58443, 1.59841, 1.58588, 1.59009, 1.58471, 1.58793, 1.59585, 1.58806, 1.59097, 1.59974, 1.58594, 1.59971, 1.5913, 1.5727, 1.57474, 1.58074, 1.57644, 1.58641, 1.58808, 1.58075, 1.5907, 1.58838, 1.58642, 1.58856, 1.58469, 1.58982, 1.59264, 1.59172, 1.58848, 1.59119, 1.59145, 1.58124, 1.60003, 1.58841, 1.59199, 1.58955, 1.59024, 1.58713, 1.58159, 1.58812, 1.58697, 1.59477, 1.58735, 1.68808, 1.60409, 1.59368, 1.68921, 1.59656, 1.59503, 1.59737, 1.5981, 1.6072, 1.60584, 1.60205, 1.60339, 1.59005, 1.59398, 1.59059, 1.5983, 1.59588, 1.58451, 1.59372, 1.59209, 1.58828, 1.59305, 1.59272, 1.59217, 1.59417, 1.59371, 1.60293, 1.6081, 1.59666, 1.59861, 1.59979, 1.59362, 1.60255, 1.60302, 1.60884, 1.60587, 1.5947, 1.59209, 1.60211, 1.60023, 1.60283, 1.60565, 1.6008, 1.5957, 1.60008, 1.59899, 1.59865, 1.59781, 1.59196, 1.59478, 1.59227]}, "forward-compute-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [6.22042, 0.7887, 0.79083, 0.78962, 0.78756, 0.78885, 0.8016, 0.80118, 0.79635, 0.79549, 0.79171, 0.803, 0.8016, 0.79277, 0.79347, 0.80205, 0.80724, 0.8102, 0.80595, 0.79227, 0.78683, 0.79736, 0.79666, 0.79876, 0.80245, 0.79592, 0.79874, 0.79753, 0.81164, 0.79672, 0.79701, 0.80746, 0.80543, 0.79696, 0.79511, 0.79932, 0.79557, 0.79429, 0.84751, 0.79126, 0.79445, 0.79427, 0.81209, 0.80591, 0.79877, 0.8166, 0.8125, 0.80956, 0.80732, 0.79604, 0.80371, 0.80021, 0.79673, 0.78625, 0.79742, 0.79855, 0.79833, 0.79792, 0.79392, 0.79627, 0.78993, 0.80003, 0.78776, 0.80568, 0.77968, 0.7912, 0.79925, 0.79922, 0.79071, 0.79884, 0.78877, 0.79858, 0.81252, 0.8067, 0.79219, 0.81833, 0.81779, 0.80094, 0.80137, 0.81945, 0.80719, 0.79232, 0.79516, 0.80871, 0.80104, 0.79685, 0.80162, 0.80637, 0.80248, 0.80857, 0.81037, 0.80869, 0.7965, 0.80743, 0.8098, 0.80128, 0.80589, 0.80206, 0.80032, 0.80015, 0.79522, 0.79329, 0.80165, 0.80384, 0.80062, 0.79949, 0.80381, 0.78559, 0.80393, 0.80321, 0.80107, 0.79216, 0.79542, 0.79246, 0.80303, 0.8106, 0.79065, 0.79761, 0.79846, 0.80131, 0.80281, 0.79732, 0.7963, 0.81465, 0.81139, 0.79778, 0.80117, 0.79101, 0.78623, 0.79644, 0.7976, 0.79653, 0.79953, 0.79765, 0.80015, 0.81095, 0.80579, 0.7998, 0.7917, 0.79794, 0.79775, 0.79275, 0.80199, 0.81948, 0.81204, 0.79625, 0.79973, 0.79652, 0.80445, 0.80534, 0.80518, 0.79884, 0.81423, 0.80952, 0.81247, 0.80766, 0.80443, 0.81182, 0.80591, 0.81339, 0.80677, 0.79581, 0.79801, 0.81209, 0.7963, 0.79413, 0.8031, 0.80814, 0.80927, 0.81215, 0.81255, 0.79604, 0.80852, 0.80814, 0.81295, 0.80402, 0.81318, 0.8097, 0.80155, 0.81294, 0.81295, 0.80384, 0.81085, 0.80809, 0.81049, 0.81462, 0.81121, 0.80114, 0.81317, 0.8073, 0.80801, 0.81335, 0.81351, 0.81644, 0.8235, 0.8092, 0.81494, 0.80197, 0.80738, 0.80524, 0.80729, 0.81006, 0.81098, 0.8058, 0.81736, 0.81018, 0.81686, 0.81077, 0.81584, 0.81737, 0.81149, 0.81076, 0.81213, 0.8138, 0.81013, 0.80497, 0.82135, 0.81652, 0.81154, 0.81448, 0.81949, 0.81162, 0.81162, 0.80853, 0.81191, 0.81703, 0.8125, 0.80932, 0.80851, 0.79798, 0.81183, 0.80938, 0.80838, 0.81083, 0.81336, 0.81205, 0.81618, 0.80587, 0.81362, 0.81042, 0.80604, 0.80513, 0.95515, 0.83951, 0.81274, 0.80912, 0.80158, 0.81243, 0.81495, 0.81427, 0.81731, 0.90437, 0.812, 0.81127, 0.80335, 0.80701, 0.81174, 0.81789, 0.8062, 0.81818, 0.81364, 0.82457, 0.81861, 0.81831, 0.81451, 0.81624, 0.819, 0.81664, 0.81149, 0.81897, 0.82098, 0.80639, 0.82356, 0.81998, 0.82291, 0.8172, 0.81813, 0.82015, 0.82009, 0.8243, 0.82188, 0.82103, 0.81895, 0.8227, 0.81898, 0.81687, 0.82231, 0.82276, 0.82281, 0.81752, 0.81589, 0.81308, 0.81283, 0.8171, 0.82039, 0.81907, 0.81497, 0.81934, 0.81714, 0.8101, 0.8135, 0.81914, 0.82468, 0.81829, 0.82195, 0.81334, 0.81505, 0.83, 0.82284, 0.82566, 0.82499, 0.82531, 0.81828, 0.81665, 0.82509, 0.82012, 0.82215, 0.82179, 0.81542, 0.80285, 0.81044, 0.80469, 0.8102, 0.8158, 0.81485, 0.82051, 0.80883, 0.82724, 0.81536, 0.8108, 0.81338, 0.81843, 0.81932, 0.81808, 0.81079, 0.81136, 0.82409, 0.81369, 0.81194, 0.81256, 0.81683, 0.81111, 0.8172, 0.80945, 0.80932, 0.8134, 0.81086, 0.81202, 0.81131, 0.86018, 0.81312, 0.81026, 0.91292, 0.81781, 0.81732, 0.82904, 0.82523, 0.83411, 0.83407, 0.83166, 0.82856, 0.81239, 0.81494, 0.82555, 0.83157, 0.82113, 0.80701, 0.81497, 0.8215, 0.80867, 0.81134, 0.82362, 0.81971, 0.808, 0.80408, 0.81663, 0.82201, 0.81271, 0.82346, 0.82415, 0.81743, 0.8063, 0.80216, 0.80964, 0.8105, 0.8118, 0.81122, 0.81369, 0.81864, 0.82566, 0.81149, 0.80986, 0.81981, 0.81964, 0.82004, 0.80608, 0.81446, 0.81929, 0.8075, 0.80881]}, "backward-compute-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [4.62942, 0.75097, 0.74, 0.74537, 0.74999, 0.75094, 0.74822, 0.74322, 0.74143, 0.74188, 0.75087, 0.75511, 0.75059, 0.75125, 0.75555, 0.7505, 0.76577, 0.75929, 0.75813, 0.75798, 0.75777, 0.75449, 0.75219, 0.76004, 0.76606, 0.74726, 0.75154, 0.75719, 0.75304, 0.75913, 0.75194, 0.76105, 0.75155, 0.75361, 0.75194, 0.74863, 0.75344, 0.75699, 0.76125, 0.76168, 0.75845, 0.75545, 0.76173, 0.76702, 0.76538, 0.76769, 0.75666, 0.75657, 0.75518, 0.75767, 0.75791, 0.75998, 0.76253, 0.75636, 0.75269, 0.75165, 0.75005, 0.74953, 0.7487, 0.76173, 0.75616, 0.75523, 0.77089, 0.75678, 0.76, 0.7504, 0.7563, 0.75155, 0.75497, 0.74943, 0.75435, 0.75485, 0.76133, 0.75829, 0.75424, 0.74885, 0.75032, 0.76341, 0.76306, 0.75225, 0.74967, 0.75803, 0.74607, 0.74997, 0.75189, 0.75522, 0.75126, 0.75345, 0.75402, 0.76221, 0.75573, 0.75879, 0.7447, 0.75592, 0.75875, 0.76088, 0.76149, 0.75471, 0.75716, 0.7483, 0.75544, 0.7486, 0.75419, 0.75681, 0.75858, 0.76287, 0.75413, 0.75433, 0.75404, 0.75102, 0.75167, 0.75697, 0.75394, 0.75963, 0.75308, 0.75609, 0.74811, 0.74816, 0.74646, 0.74523, 0.74868, 0.74707, 0.74934, 0.7508, 0.76531, 0.76133, 0.75869, 0.75454, 0.74851, 0.74933, 0.74654, 0.74315, 0.74234, 0.74764, 0.75289, 0.7578, 0.75618, 0.75315, 0.75232, 0.75728, 0.75011, 0.75412, 0.75242, 0.74889, 0.75119, 0.75527, 0.75085, 0.7583, 0.76477, 0.75215, 0.75071, 0.76072, 0.75986, 0.76825, 0.75337, 0.75661, 0.75384, 0.76056, 0.76054, 0.76494, 0.7674, 0.76549, 0.75611, 0.76183, 0.75053, 0.75482, 0.75715, 0.76983, 0.77042, 0.76028, 0.77021, 0.75151, 0.75914, 0.75118, 0.76133, 0.75325, 0.76558, 0.75951, 0.76119, 0.75926, 0.75073, 0.75384, 0.75883, 0.7634, 0.76168, 0.76652, 0.75731, 0.75344, 0.76068, 0.75369, 0.75137, 0.75963, 0.7697, 0.751, 0.77098, 0.75284, 0.75939, 0.75995, 0.75928, 0.75802, 0.75677, 0.76065, 0.75638, 0.75119, 0.76038, 0.75423, 0.75553, 0.75918, 0.75995, 0.75408, 0.76136, 0.74612, 0.75854, 0.75865, 0.7593, 0.75419, 0.75151, 0.75761, 0.76577, 0.75463, 0.74788, 0.75358, 0.76279, 0.76172, 0.76321, 0.75292, 0.75124, 0.75794, 0.76269, 0.76049, 0.75669, 0.7573, 0.75738, 0.75375, 0.76126, 0.75621, 0.75055, 0.75297, 0.75603, 0.75099, 0.75101, 0.74554, 0.83246, 0.7545, 0.75293, 0.75203, 0.75391, 0.7554, 0.75839, 0.75728, 0.76242, 0.75203, 0.75857, 0.7516, 0.75317, 0.75327, 0.75445, 0.7579, 0.753, 0.753, 0.75219, 0.75665, 0.75118, 0.75048, 0.74602, 0.74682, 0.75041, 0.74864, 0.75542, 0.74976, 0.74748, 0.75186, 0.75401, 0.75027, 0.74959, 0.75363, 0.74766, 0.75374, 0.751, 0.75381, 0.75069, 0.74504, 0.75077, 0.75083, 0.75402, 0.74825, 0.75092, 0.75145, 0.75314, 0.75502, 0.74951, 0.7579, 0.75347, 0.7511, 0.75538, 0.75696, 0.7579, 0.75511, 0.75693, 0.75306, 0.74836, 0.7533, 0.75717, 0.76271, 0.75482, 0.75341, 0.74896, 0.75096, 0.74632, 0.75083, 0.74516, 0.74075, 0.75065, 0.75718, 0.75375, 0.7557, 0.7462, 0.75504, 0.75655, 0.74982, 0.75081, 0.74949, 0.74808, 0.75239, 0.75544, 0.74273, 0.75537, 0.75449, 0.75109, 0.7469, 0.7528, 0.75193, 0.75171, 0.75366, 0.75959, 0.74847, 0.75215, 0.75052, 0.76098, 0.75632, 0.75747, 0.74845, 0.74437, 0.75406, 0.75357, 0.75105, 0.75484, 0.75765, 0.75917, 0.7582, 0.75622, 0.75762, 0.74952, 0.75592, 0.75778, 0.74829, 0.75888, 0.75085, 0.75064, 0.74667, 0.751, 0.75208, 0.75768, 0.74883, 0.75857, 0.7487, 0.75962, 0.76274, 0.75413, 0.75644, 0.75008, 0.75022, 0.75465, 0.76027, 0.75685, 0.7526, 0.7567, 0.75515, 0.75552, 0.75496, 0.75875, 0.76104, 0.77511, 0.77406, 0.768, 0.7781, 0.77247, 0.78055, 0.77825, 0.76677, 0.78188, 0.77415, 0.77114, 0.77225, 0.77049, 0.77717, 0.77115, 0.76807, 0.77259, 0.77472]}, "batch-generator-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.20334, 0.0143, 0.01667, 0.01326, 0.01295, 0.01293, 0.01334, 0.01436, 0.01318, 0.01437, 0.01301, 0.01378, 0.01472, 0.01468, 0.01314, 0.01281, 0.01302, 0.01378, 0.01285, 0.01444, 0.01432, 0.01486, 0.01305, 0.01348, 0.01674, 0.01301, 0.01444, 0.01426, 0.01437, 0.01321, 0.01305, 0.01316, 0.01395, 0.01333, 0.01301, 0.01363, 0.01284, 0.01423, 0.01642, 0.01753, 0.01691, 0.01476, 0.01495, 0.01652, 0.01707, 0.02019, 0.01642, 0.01534, 0.01555, 0.01455, 0.01613, 0.01682, 0.01611, 0.01302, 0.01316, 0.01386, 0.0152, 0.01835, 0.01342, 0.01579, 0.01295, 0.01372, 0.01717, 0.0153, 0.01567, 0.01348, 0.01623, 0.0153, 0.01466, 0.01622, 0.01222, 0.01602, 0.02111, 0.01556, 0.01731, 0.01708, 0.01773, 0.0175, 0.01682, 0.0175, 0.01625, 0.0172, 0.01748, 0.02121, 0.01676, 0.01653, 0.01683, 0.01767, 0.01788, 0.01764, 0.01715, 0.02209, 0.01681, 0.01797, 0.01754, 0.01797, 0.01781, 0.01828, 0.0179, 0.01691, 0.01823, 0.0176, 0.01724, 0.0166, 0.01718, 0.01732, 0.0149, 0.01363, 0.01477, 0.01454, 0.01309, 0.01297, 0.01408, 0.0145, 0.01297, 0.01965, 0.01506, 0.01303, 0.01404, 0.01373, 0.01435, 0.01442, 0.01449, 0.01568, 0.01599, 0.01299, 0.01288, 0.01478, 0.01302, 0.01354, 0.01604, 0.01518, 0.01493, 0.01391, 0.01308, 0.01275, 0.01267, 0.01483, 0.0133, 0.01279, 0.01339, 0.01261, 0.01553, 0.01269, 0.0125, 0.01256, 0.01329, 0.0129, 0.01284, 0.01681, 0.01599, 0.01537, 0.0153, 0.01362, 0.01518, 0.01566, 0.01486, 0.01485, 0.01522, 0.01745, 0.01558, 0.01496, 0.01484, 0.01693, 0.01487, 0.01546, 0.02093, 0.01683, 0.01724, 0.01738, 0.01648, 0.01861, 0.01776, 0.01745, 0.01724, 0.01583, 0.02118, 0.01682, 0.01836, 0.02112, 0.01766, 0.0169, 0.01696, 0.01695, 0.01754, 0.01652, 0.0184, 0.0173, 0.01627, 0.01667, 0.01742, 0.01775, 0.01745, 0.01643, 0.01709, 0.01696, 0.01761, 0.01648, 0.01725, 0.01672, 0.21908, 0.01675, 0.01611, 0.01752, 0.01616, 0.01728, 0.01777, 0.0171, 0.01749, 0.01847, 0.01858, 0.01789, 0.01723, 0.01628, 0.01773, 0.01691, 0.01878, 0.01787, 0.0209, 0.01796, 0.01741, 0.01777, 0.01829, 0.01892, 0.01729, 0.01774, 0.01727, 0.02061, 0.01571, 0.01771, 0.01838, 0.01772, 0.0174, 0.01766, 0.01725, 0.01763, 0.01752, 0.01709, 0.01817, 0.02143, 0.0161, 0.01751, 0.09405, 0.06723, 0.01758, 0.01661, 0.02181, 0.02167, 0.01822, 0.01785, 0.01747, 0.01708, 0.01826, 0.01765, 0.01811, 0.01727, 0.01812, 0.01807, 0.01812, 0.01919, 0.01774, 0.01749, 0.01737, 0.01751, 0.01714, 0.02283, 0.01759, 0.01975, 0.02057, 0.01799, 0.01752, 0.01739, 0.01757, 0.01773, 0.01789, 0.01729, 0.01642, 0.01712, 0.0176, 0.01717, 0.01691, 0.01727, 0.01589, 0.01789, 0.0174, 0.0174, 0.01722, 0.01761, 0.01802, 0.0174, 0.02069, 0.0171, 0.01719, 0.01766, 0.01768, 0.01677, 0.01705, 0.01777, 0.01669, 0.02073, 0.01723, 0.01707, 0.01707, 0.01723, 0.01751, 0.01953, 0.0174, 0.0167, 0.01749, 0.01753, 0.01974, 0.01695, 0.01888, 0.01805, 0.01809, 0.01779, 0.0192, 0.01732, 0.01965, 0.01793, 0.01875, 0.01855, 0.01915, 0.01839, 0.01868, 0.01864, 0.01893, 0.01823, 0.01908, 0.01892, 0.01884, 0.01914, 0.02012, 0.01861, 0.02283, 0.01928, 0.01945, 0.01841, 0.01795, 0.01816, 0.0187, 0.01867, 0.01891, 0.02308, 0.0188, 0.01869, 0.01974, 0.02014, 0.02234, 0.0193, 0.01762, 0.01819, 0.0184, 0.01952, 0.01974, 0.01869, 0.0205, 0.018, 0.0183, 0.01719, 0.01915, 0.01879, 0.0194, 0.01781, 0.01856, 0.01773, 0.01734, 0.01914, 0.0169, 0.019, 0.01792, 0.01743, 0.02488, 0.01724, 0.01703, 0.01755, 0.01784, 0.01774, 0.01824, 0.01859, 0.02236, 0.01639, 0.0181, 0.01772, 0.01786, 0.01787, 0.01629, 0.01663, 0.01687, 0.01734, 0.01643, 0.0175, 0.0166, 0.01686, 0.0162, 0.01662, 0.02025, 0.01762, 0.01683, 0.01837]}, "forward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [5.65416, 0.02537, 0.02635, 0.02461, 0.02504, 0.02484, 0.02542, 0.02517, 0.02613, 0.02496, 0.02499, 0.02526, 0.02517, 0.02669, 0.02527, 0.02523, 0.02555, 0.02514, 0.02531, 0.02544, 0.02502, 0.02866, 0.02534, 0.02519, 0.02546, 0.02642, 0.02449, 0.02505, 0.02448, 0.02468, 0.02481, 0.02534, 0.02569, 0.02662, 0.02525, 0.02575, 0.02553, 0.02468, 0.02518, 0.02486, 0.02617, 0.0262, 0.02498, 0.02481, 0.02556, 0.02544, 0.02525, 0.02507, 0.02521, 0.02526, 0.02607, 0.02518, 0.02513, 0.02559, 0.02488, 0.02586, 0.02585, 0.02611, 0.02926, 0.02566, 0.02649, 0.02556, 0.02541, 0.02684, 0.0255, 0.02555, 0.0255, 0.0255, 0.02545, 0.02694, 0.02533, 0.02962, 0.02527, 0.02528, 0.02579, 0.02515, 0.02509, 0.02553, 0.02514, 0.02532, 0.02535, 0.02565, 0.02505, 0.02564, 0.02529, 0.02581, 0.02662, 0.02629, 0.02709, 0.02508, 0.0255, 0.02567, 0.02579, 0.0251, 0.02471, 0.02553, 0.02567, 0.02524, 0.02526, 0.02542, 0.02549, 0.02485, 0.0254, 0.02557, 0.02563, 0.02532, 0.02527, 0.02538, 0.02679, 0.02564, 0.02917, 0.02565, 0.02736, 0.02515, 0.02504, 0.02493, 0.02534, 0.0255, 0.02468, 0.02576, 0.02535, 0.02502, 0.02542, 0.02937, 0.02618, 0.02564, 0.02552, 0.02493, 0.02464, 0.02534, 0.02541, 0.02506, 0.02906, 0.02585, 0.02551, 0.02458, 0.02524, 0.0254, 0.02487, 0.02705, 0.02476, 0.02422, 0.02846, 0.02862, 0.02919, 0.02491, 0.02528, 0.0255, 0.02536, 0.02481, 0.02663, 0.02537, 0.02529, 0.02555, 0.02495, 0.02532, 0.02892, 0.02477, 0.02508, 0.0255, 0.02505, 0.0255, 0.02603, 0.02601, 0.02543, 0.0257, 0.02514, 0.02658, 0.02696, 0.02519, 0.02558, 0.02777, 0.027, 0.02528, 0.02566, 0.02491, 0.02592, 0.02533, 0.02595, 0.0256, 0.02521, 0.02524, 0.02528, 0.02552, 0.02639, 0.02554, 0.02548, 0.02553, 0.02553, 0.02546, 0.02481, 0.02518, 0.02516, 0.02541, 0.02568, 0.02495, 0.02523, 0.02848, 0.02556, 0.02499, 0.022, 0.02884, 0.02809, 0.02537, 0.02485, 0.02541, 0.0241, 0.02529, 0.02531, 0.02522, 0.02532, 0.02491, 0.02523, 0.02501, 0.02691, 0.02738, 0.02935, 0.02585, 0.02542, 0.02516, 0.02571, 0.03013, 0.02563, 0.02483, 0.0253, 0.02509, 0.02525, 0.0255, 0.02513, 0.02517, 0.02489, 0.02524, 0.02485, 0.02507, 0.02536, 0.02583, 0.02534, 0.02509, 0.0251, 0.02531, 0.02518, 0.02475, 0.02917, 0.02567, 0.02587, 0.02568, 0.02609, 0.02628, 0.02622, 0.02564, 0.02497, 0.02578, 0.02549, 0.02526, 0.02494, 0.02571, 0.02582, 0.02631, 0.02647, 0.02581, 0.02643, 0.02664, 0.0263, 0.02556, 0.025, 0.02535, 0.02517, 0.02527, 0.0252, 0.02486, 0.02861, 0.02534, 0.02604, 0.02568, 0.02564, 0.02728, 0.02552, 0.02578, 0.02551, 0.02575, 0.02545, 0.02536, 0.02514, 0.02619, 0.02548, 0.02549, 0.02561, 0.02555, 0.02574, 0.02616, 0.02572, 0.02599, 0.02561, 0.02503, 0.02535, 0.02684, 0.02548, 0.02545, 0.02557, 0.02504, 0.02542, 0.0261, 0.02567, 0.02546, 0.0255, 0.02529, 0.02633, 0.03021, 0.0287, 0.0293, 0.0291, 0.03051, 0.03077, 0.02941, 0.03025, 0.02889, 0.02504, 0.02563, 0.02509, 0.02514, 0.02874, 0.02525, 0.02524, 0.02529, 0.02567, 0.02595, 0.02539, 0.02551, 0.02571, 0.02607, 0.02531, 0.02862, 0.02572, 0.02526, 0.02664, 0.02609, 0.02882, 0.02605, 0.02621, 0.02593, 0.02588, 0.02619, 0.02534, 0.02604, 0.02557, 0.02616, 0.02561, 0.02542, 0.02469, 0.02539, 0.02533, 0.02624, 0.02525, 0.02545, 0.02533, 0.02553, 0.02573, 0.02577, 0.0253, 0.02529, 0.02629, 0.02636, 0.02548, 0.02577, 0.0255, 0.02611, 0.02473, 0.02582, 0.02551, 0.02567, 0.0253, 0.02519, 0.0256, 0.02642, 0.02489, 0.02549, 0.02566, 0.0257, 0.02523, 0.02566, 0.02708, 0.02568, 0.025, 0.02826, 0.02772, 0.02446, 0.02415, 0.0242, 0.02452, 0.02402, 0.02491, 0.02511, 0.02443, 0.0247, 0.02457, 0.02433, 0.02427, 0.02485, 0.02473, 0.02411]}, "forward-send-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.82565, 0.00013, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00019, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00014, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00015, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00014, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00018, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00014, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00014, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00014, 0.00014, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012]}, "backward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.02047, 0.0283, 0.02457, 0.02402, 0.02376, 0.02455, 0.02368, 0.02489, 0.03547, 0.02397, 0.02483, 0.02383, 0.02354, 0.02677, 0.02403, 0.02404, 0.02385, 0.02413, 0.02382, 0.02401, 0.02447, 0.02418, 0.02565, 0.02458, 0.02399, 0.02426, 0.02371, 0.02373, 0.02497, 0.02531, 0.02428, 0.02424, 0.02812, 0.02847, 0.02391, 0.0276, 0.02414, 0.02342, 0.02403, 0.0241, 0.02246, 0.0239, 0.02373, 0.02354, 0.024, 0.02551, 0.02523, 0.02434, 0.02333, 0.02695, 0.02802, 0.03335, 0.024, 0.02415, 0.02428, 0.0235, 0.02721, 0.02385, 0.02396, 0.02372, 0.02372, 0.02589, 0.02448, 0.02657, 0.02807, 0.02364, 0.02407, 0.02393, 0.02278, 0.02609, 0.02324, 0.02406, 0.02392, 0.02575, 0.02435, 0.02335, 0.02423, 0.02688, 0.02482, 0.02464, 0.0283, 0.02798, 0.02454, 0.02403, 0.02385, 0.02375, 0.024, 0.02436, 0.02658, 0.02418, 0.02444, 0.02438, 0.02772, 0.02445, 0.02469, 0.02482, 0.025, 0.0236, 0.02423, 0.02583, 0.02383, 0.02532, 0.02443, 0.02397, 0.02832, 0.02453, 0.02425, 0.02386, 0.02401, 0.02329, 0.02374, 0.02459, 0.02345, 0.02812, 0.02257, 0.02428, 0.03159, 0.02496, 0.02394, 0.02407, 0.02348, 0.02404, 0.0242, 0.02606, 0.02405, 0.02413, 0.02672, 0.02751, 0.02579, 0.02343, 0.02459, 0.02392, 0.02467, 0.02321, 0.02966, 0.02406, 0.02342, 0.02901, 0.02438, 0.02338, 0.02418, 0.02428, 0.02389, 0.02408, 0.02451, 0.02382, 0.02778, 0.02307, 0.02734, 0.02437, 0.02405, 0.02422, 0.02458, 0.02387, 0.02398, 0.02622, 0.0253, 0.02883, 0.02608, 0.02311, 0.02341, 0.0239, 0.02486, 0.02775, 0.02913, 0.02946, 0.03162, 0.03164, 0.03243, 0.02904, 0.03427, 0.02606, 0.02427, 0.02426, 0.02481, 0.02533, 0.02412, 0.02331, 0.02327, 0.02433, 0.02456, 0.02446, 0.02307, 0.02419, 0.02354, 0.02436, 0.02445, 0.02378, 0.02468, 0.02434, 0.02455, 0.02741, 0.02293, 0.02633, 0.02903, 0.02671, 0.02326, 0.0238, 0.02369, 0.02323, 0.02472, 0.02363, 0.02637, 0.02415, 0.0239, 0.02407, 0.02419, 0.0237, 0.02387, 0.02419, 0.02417, 0.02427, 0.02439, 0.02456, 0.02399, 0.02419, 0.0259, 0.02715, 0.02432, 0.02384, 0.02406, 0.02463, 0.02389, 0.02404, 0.02528, 0.02496, 0.0241, 0.02492, 0.02586, 0.02752, 0.02936, 0.02831, 0.02641, 0.02748, 0.02535, 0.0236, 0.02441, 0.02391, 0.02402, 0.02375, 0.02392, 0.02658, 0.02281, 0.02404, 0.02443, 0.02393, 0.02425, 0.02565, 0.02492, 0.02922, 0.02822, 0.02695, 0.02827, 0.02425, 0.02791, 0.02429, 0.02507, 0.02421, 0.02448, 0.02504, 0.02444, 0.02428, 0.02484, 0.02431, 0.0247, 0.02476, 0.02429, 0.02826, 0.02806, 0.02466, 0.02444, 0.02446, 0.02398, 0.0246, 0.02694, 0.02743, 0.02754, 0.02821, 0.02752, 0.02768, 0.02846, 0.02827, 0.02821, 0.02757, 0.02781, 0.03032, 0.0282, 0.02767, 0.02766, 0.02791, 0.02891, 0.02728, 0.02724, 0.02826, 0.02818, 0.0275, 0.02704, 0.02768, 0.02881, 0.02841, 0.02812, 0.02758, 0.02852, 0.02732, 0.02863, 0.0247, 0.02488, 0.02405, 0.02493, 0.02485, 0.025, 0.02485, 0.0248, 0.02492, 0.02512, 0.02464, 0.02467, 0.02816, 0.02752, 0.02469, 0.02368, 0.02464, 0.02438, 0.02448, 0.02474, 0.0246, 0.0247, 0.02471, 0.02492, 0.02452, 0.02459, 0.02436, 0.02461, 0.02714, 0.02468, 0.02624, 0.02941, 0.02449, 0.02703, 0.02762, 0.0284, 0.02681, 0.02872, 0.02442, 0.02456, 0.02406, 0.02457, 0.02358, 0.02347, 0.02871, 0.03113, 0.02849, 0.02643, 0.02442, 0.02499, 0.02477, 0.02568, 0.02464, 0.02487, 0.02408, 0.0248, 0.0262, 0.02523, 0.02571, 0.02565, 0.02504, 0.02409, 0.02564, 0.02393, 0.02423, 0.02644, 0.0241, 0.02354, 0.02445, 0.02479, 0.02481, 0.02499, 0.02444, 0.02433, 0.02438, 0.02439, 0.02468, 0.02426, 0.02465, 0.02263, 0.02673, 0.0262, 0.02622, 0.02641, 0.0272, 0.02655, 0.02722, 0.02659, 0.02705, 0.02744, 0.02687, 0.02797, 0.02579, 0.0241, 0.02442]}, "backward-send-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00019, 0.00019, 0.00016, 0.0002, 0.00018, 0.00018, 0.00016, 0.00018, 0.00022, 0.00017, 0.00018, 0.00017, 0.00018, 0.00016, 0.00017, 0.00017, 0.00018, 0.00017, 0.00016, 0.00016, 0.00019, 0.00019, 0.00016, 0.00018, 0.00016, 0.00016, 0.00018, 0.00016, 0.00017, 0.00018, 0.00016, 0.00019, 0.00018, 0.00016, 0.00019, 0.00016, 0.00016, 0.00018, 0.00018, 0.00018, 0.0002, 0.00018, 0.00018, 0.00017, 0.00016, 0.00018, 0.00017, 0.00017, 0.00018, 0.00021, 0.00019, 0.00018, 0.00019, 0.00018, 0.00016, 0.00017, 0.00018, 0.00018, 0.00018, 0.00016, 0.00018, 0.00018, 0.00016, 0.00018, 0.0002, 0.00018, 0.00018, 0.00017, 0.00018, 0.00016, 0.00016, 0.00019, 0.00021, 0.00017, 0.00016, 0.00016, 0.00018, 0.00018, 0.00019, 0.00018, 0.00019, 0.00019, 0.00019, 0.00016, 0.00018, 0.00021, 0.00017, 0.00018, 0.00017, 0.00018, 0.00016, 0.00016, 0.00018, 0.00017, 0.00016, 0.00018, 0.00036, 0.00016, 0.00022, 0.00016, 0.00016, 0.00019, 0.00019, 0.00018, 0.00017, 0.00016, 0.00018, 0.00016, 0.00018, 0.00018, 0.00016, 0.00017, 0.00018, 0.00019, 0.00018, 0.00018, 0.00032, 0.00018, 0.00018, 0.00016, 0.00021, 0.00018, 0.00017, 0.00018, 0.00018, 0.00018, 0.00016, 0.00018, 0.00017, 0.00018, 0.00018, 0.00018, 0.00021, 0.00016, 0.00019, 0.00019, 0.00018, 0.00017, 0.00017, 0.00018, 0.00018, 0.00018, 0.00018, 0.00019, 0.00018, 0.00016, 0.00018, 0.00019, 0.00018, 0.00021, 0.00016, 0.00017, 0.00016, 0.00016, 0.00017, 0.0002, 0.00018, 0.00017, 0.00018, 0.00017, 0.00018, 0.00019, 0.00018, 0.00018, 0.00018, 0.00019, 0.00027, 0.00031, 0.00017, 0.00017, 0.00016, 0.00016, 0.00019, 0.00018, 0.00016, 0.00018, 0.00018, 0.0002, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00016, 0.00018, 0.00018, 0.00019, 0.00018, 0.00019, 0.00018, 0.0002, 0.0002, 0.00018, 0.00017, 0.00019, 0.00018, 0.00018, 0.00018, 0.00016, 0.00019, 0.00016, 0.00018, 0.00018, 0.00018, 0.00016, 0.00018, 0.00019, 0.00018, 0.00018, 0.00016, 0.00018, 0.00018, 0.00018, 0.00017, 0.00017, 0.00019, 0.00019, 0.00018, 0.00019, 0.00018, 0.00016, 0.00018, 0.00018, 0.00016, 0.0002, 0.00016, 0.00018, 0.00016, 0.00017, 0.00018, 0.00018, 0.00019, 0.00018, 0.00016, 0.00018, 0.00019, 0.00018, 0.00016, 0.00016, 0.00018, 0.00016, 0.00016, 0.00016, 0.00019, 0.00018, 0.00018, 0.00018, 0.00018, 0.00019, 0.00019, 0.00016, 0.00018, 0.00017, 0.00019, 0.00037, 0.00017, 0.00017, 0.00019, 0.00018, 0.00018, 0.00016, 0.00018, 0.00018, 0.00018, 0.00019, 0.0002, 0.00016, 0.00018, 0.00029, 0.00019, 0.0002, 0.00018, 0.00016, 0.00016, 0.00016, 0.00016, 0.00018, 0.00018, 0.00018, 0.00017, 0.00037, 0.00019, 0.00019, 0.00018, 0.00018, 0.00018, 0.0002, 0.00016, 0.00018, 0.00029, 0.00017, 0.00024, 0.00016, 0.00019, 0.00016, 0.00017, 0.00035, 0.00036, 0.00017, 0.00016, 0.0002, 0.00034, 0.0002, 0.00016, 0.00017, 0.0002, 0.00016, 0.00018, 0.00018, 0.00016, 0.00017, 0.00017, 0.00016, 0.00017, 0.00018, 0.00018, 0.00018, 0.00025, 0.00018, 0.00016, 0.00016, 0.00016, 0.00017, 0.00017, 0.00018, 0.00016, 0.00017, 0.00018, 0.00017, 0.00016, 0.00018, 0.00018, 0.00017, 0.00018, 0.00018, 0.00018, 0.00017, 0.00016, 0.00016, 0.00019, 0.00017, 0.00018, 0.00019, 0.00016, 0.00018, 0.00018, 0.00019, 0.00016, 0.00016, 0.00019, 0.00017, 0.00019, 0.00017, 0.00017, 0.00018, 0.00017, 0.00016, 0.00018, 0.00016, 0.00018, 0.00019, 0.00016, 0.00017, 0.00019, 0.00016, 0.00017, 0.00016, 0.00016, 0.00019, 0.00017, 0.00018, 0.00018, 0.00018, 0.00016, 0.00019, 0.00018, 0.00017, 0.00018, 0.00016, 0.00018, 0.0002, 0.00017, 0.00016, 0.00017, 0.00017, 0.00018, 0.00018, 0.00018, 0.00016, 0.00019, 0.00018, 0.00018, 0.00018, 0.00019, 0.00016, 0.00016, 0.00017, 0.00018, 0.00018, 0.00016]}, "forward-send-backward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [7.72045, 0.09004, 0.10467, 0.09849, 0.09238, 0.09943, 0.10332, 0.10911, 0.10563, 0.10498, 0.10272, 0.10382, 0.10192, 0.10289, 0.10891, 0.10722, 0.1057, 0.11565, 0.11445, 0.10746, 0.11354, 0.10514, 0.10376, 0.08937, 0.09262, 0.08764, 0.08288, 0.09035, 0.09702, 0.09008, 0.09616, 0.09645, 0.09564, 0.08936, 0.08325, 0.08878, 0.08887, 0.08097, 0.16157, 0.08262, 0.08896, 0.09145, 0.09803, 0.08184, 0.09702, 0.0971, 0.09683, 0.09764, 0.08935, 0.0971, 0.10578, 0.09846, 0.10251, 0.08742, 0.08778, 0.08971, 0.09353, 0.08897, 0.09, 0.08803, 0.08686, 0.08756, 0.09058, 0.08647, 0.08759, 0.09747, 0.10439, 0.10521, 0.09647, 0.10904, 0.09397, 0.09736, 0.10653, 0.0936, 0.10631, 0.1059, 0.10256, 0.09952, 0.09927, 0.10519, 0.10149, 0.09551, 0.10221, 0.10051, 0.09736, 0.09577, 0.0979, 0.09361, 0.09726, 0.10742, 0.0922, 0.10792, 0.10335, 0.10219, 0.1015, 0.09685, 0.09726, 0.10184, 0.09792, 0.10191, 0.1005, 0.10051, 0.09742, 0.09427, 0.09441, 0.08885, 0.09704, 0.09172, 0.09714, 0.09629, 0.10183, 0.09676, 0.09562, 0.09133, 0.09003, 0.10068, 0.09125, 0.0941, 0.09629, 0.10409, 0.09294, 0.09359, 0.10104, 0.10583, 0.09162, 0.08569, 0.08813, 0.093, 0.08756, 0.10008, 0.09688, 0.1054, 0.10747, 0.10112, 0.10023, 0.10296, 0.09747, 0.0945, 0.09503, 0.09075, 0.10094, 0.09821, 0.10359, 0.11126, 0.11094, 0.10686, 0.10472, 0.10387, 0.09679, 0.10627, 0.11005, 0.10858, 0.10916, 0.10819, 0.11254, 0.11227, 0.1067, 0.10979, 0.10635, 0.10862, 0.11093, 0.10588, 0.1078, 0.11054, 0.10333, 0.10314, 0.11111, 0.10133, 0.10064, 0.10338, 0.09919, 0.10252, 0.10368, 0.10692, 0.11169, 0.10373, 0.1082, 0.11025, 0.09905, 0.10905, 0.11343, 0.10499, 0.10807, 0.10315, 0.09841, 0.10583, 0.10804, 0.09746, 0.10771, 0.10609, 0.10625, 0.1058, 0.10401, 0.10832, 0.10595, 0.10705, 0.11742, 0.10139, 0.10969, 0.09952, 0.10696, 0.11066, 0.10165, 0.10114, 0.10538, 0.10594, 0.11402, 0.10492, 0.10645, 0.11173, 0.10848, 0.11309, 0.10714, 0.10786, 0.10722, 0.10193, 0.11309, 0.0997, 0.10535, 0.10927, 0.11186, 0.11523, 0.10176, 0.11174, 0.10738, 0.10339, 0.10818, 0.10428, 0.10357, 0.102, 0.11031, 0.10504, 0.10603, 0.10464, 0.10777, 0.10003, 0.11154, 0.10215, 0.10884, 0.1135, 0.10294, 0.10521, 0.18146, 0.15513, 0.10795, 0.10192, 0.09492, 0.1123, 0.11068, 0.10753, 0.10062, 0.20176, 0.10053, 0.10546, 0.10178, 0.10047, 0.10162, 0.10317, 0.10396, 0.10664, 0.11601, 0.12091, 0.11596, 0.11321, 0.11757, 0.11585, 0.1102, 0.10582, 0.10902, 0.11204, 0.11498, 0.11048, 0.11561, 0.12266, 0.11204, 0.10563, 0.11232, 0.10806, 0.10523, 0.11245, 0.10857, 0.10998, 0.10637, 0.11004, 0.10832, 0.1137, 0.11249, 0.1137, 0.11325, 0.10714, 0.10913, 0.11342, 0.10767, 0.11168, 0.1127, 0.10979, 0.10867, 0.10899, 0.11074, 0.10988, 0.11196, 0.11045, 0.10625, 0.10876, 0.11621, 0.10786, 0.11166, 0.1137, 0.1159, 0.12034, 0.12688, 0.13086, 0.12051, 0.11583, 0.12425, 0.12785, 0.11994, 0.1156, 0.11305, 0.1064, 0.11037, 0.11458, 0.10783, 0.11267, 0.11832, 0.11674, 0.12221, 0.11896, 0.11355, 0.12228, 0.11929, 0.11934, 0.11071, 0.11311, 0.12323, 0.11815, 0.1124, 0.10574, 0.10714, 0.11404, 0.1155, 0.11749, 0.11507, 0.11217, 0.11336, 0.11724, 0.11529, 0.11873, 0.11413, 0.11342, 0.11662, 0.11253, 0.21031, 0.1153, 0.11949, 0.12203, 0.12384, 0.12782, 0.12363, 0.12548, 0.12785, 0.11974, 0.12339, 0.11698, 0.1138, 0.11801, 0.11508, 0.12193, 0.1161, 0.11722, 0.11675, 0.12016, 0.12149, 0.12239, 0.12005, 0.12773, 0.12921, 0.11853, 0.11824, 0.12298, 0.11989, 0.12376, 0.12606, 0.12268, 0.12167, 0.11886, 0.10748, 0.11973, 0.11767, 0.12515, 0.11708, 0.11935, 0.12016, 0.12159, 0.11803, 0.11151, 0.11606, 0.11651, 0.12057, 0.10879]}, "backward-send-forward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [3.17241, 0.01112, 0.01172, 0.00869, 0.00901, 0.01001, 0.01115, 0.00794, 0.00798, 0.0109, 0.01029, 0.01093, 0.01077, 0.01317, 0.01259, 0.00838, 0.01022, 0.00884, 0.01678, 0.0152, 0.00915, 0.00886, 0.00872, 0.00978, 0.01165, 0.00864, 0.01118, 0.01286, 0.00996, 0.0125, 0.01039, 0.01705, 0.00824, 0.00886, 0.00817, 0.00863, 0.0105, 0.00871, 0.08171, 0.01193, 0.01314, 0.01206, 0.01407, 0.01071, 0.01251, 0.01179, 0.01146, 0.00929, 0.01052, 0.01215, 0.0084, 0.00818, 0.00939, 0.0111, 0.00825, 0.01008, 0.01023, 0.00961, 0.0079, 0.01198, 0.0144, 0.00802, 0.01242, 0.00847, 0.01011, 0.00724, 0.00808, 0.0078, 0.00899, 0.00896, 0.00949, 0.00922, 0.01098, 0.01, 0.01342, 0.00965, 0.00844, 0.01778, 0.01504, 0.00876, 0.01126, 0.01156, 0.00994, 0.00745, 0.01045, 0.01139, 0.01102, 0.01004, 0.01044, 0.01421, 0.01363, 0.0147, 0.01748, 0.01497, 0.01481, 0.01661, 0.00933, 0.01088, 0.01211, 0.01187, 0.0114, 0.01087, 0.00985, 0.01082, 0.01058, 0.01129, 0.00882, 0.01084, 0.00902, 0.0079, 0.01036, 0.01589, 0.01561, 0.01591, 0.00899, 0.01108, 0.00841, 0.01003, 0.00851, 0.00882, 0.00846, 0.00785, 0.01152, 0.00747, 0.01326, 0.01202, 0.01211, 0.01078, 0.00952, 0.00873, 0.00881, 0.00874, 0.00915, 0.00875, 0.01297, 0.01552, 0.0151, 0.01016, 0.00992, 0.01251, 0.01115, 0.01149, 0.00982, 0.01462, 0.01529, 0.0145, 0.01056, 0.01488, 0.01365, 0.01448, 0.00917, 0.0134, 0.01205, 0.01572, 0.0126, 0.01488, 0.01305, 0.01335, 0.0138, 0.0164, 0.01209, 0.01237, 0.01442, 0.01402, 0.01277, 0.01318, 0.01188, 0.0129, 0.01144, 0.01322, 0.01297, 0.0121, 0.01209, 0.01029, 0.01079, 0.01249, 0.01233, 0.0121, 0.01022, 0.0128, 0.01174, 0.01218, 0.01303, 0.01323, 0.01318, 0.01287, 0.00961, 0.01202, 0.0124, 0.00992, 0.00876, 0.00935, 0.01319, 0.01636, 0.01632, 0.01494, 0.01298, 0.01614, 0.01406, 0.01537, 0.01153, 0.01115, 0.01271, 0.0107, 0.01222, 0.01248, 0.01198, 0.01383, 0.01146, 0.01187, 0.01068, 0.01125, 0.00998, 0.01224, 0.01454, 0.01162, 0.00956, 0.01122, 0.0154, 0.01199, 0.01342, 0.01294, 0.01456, 0.01293, 0.01589, 0.01161, 0.01349, 0.01587, 0.0161, 0.01506, 0.01604, 0.01245, 0.01415, 0.01038, 0.01375, 0.01225, 0.01179, 0.01138, 0.01149, 0.0114, 0.01157, 0.01201, 0.09678, 0.06875, 0.01665, 0.01943, 0.01672, 0.01779, 0.01975, 0.01513, 0.01188, 0.01383, 0.01055, 0.01209, 0.01624, 0.01171, 0.01034, 0.00943, 0.0124, 0.01104, 0.01002, 0.00883, 0.01064, 0.01032, 0.00949, 0.01005, 0.01087, 0.01209, 0.01055, 0.00979, 0.00997, 0.01044, 0.01106, 0.01088, 0.01076, 0.01045, 0.01152, 0.01085, 0.0105, 0.01114, 0.01146, 0.01082, 0.01229, 0.01175, 0.01162, 0.01101, 0.01116, 0.01256, 0.01128, 0.01152, 0.0107, 0.00988, 0.0095, 0.01009, 0.01045, 0.01003, 0.00992, 0.01213, 0.01087, 0.01368, 0.00953, 0.01064, 0.01243, 0.01214, 0.01155, 0.01008, 0.00976, 0.01033, 0.00912, 0.0081, 0.00967, 0.01116, 0.00911, 0.00921, 0.00997, 0.01136, 0.01025, 0.01241, 0.01273, 0.01327, 0.01109, 0.01279, 0.01226, 0.0121, 0.01061, 0.01401, 0.0134, 0.01432, 0.01133, 0.01394, 0.01414, 0.01459, 0.01155, 0.01481, 0.01262, 0.01169, 0.01079, 0.01328, 0.01375, 0.01229, 0.01428, 0.01132, 0.0128, 0.01126, 0.01216, 0.01314, 0.01251, 0.01231, 0.01489, 0.10504, 0.01146, 0.01181, 0.10182, 0.00974, 0.01066, 0.01245, 0.01188, 0.01268, 0.01247, 0.01243, 0.0136, 0.0116, 0.01212, 0.01459, 0.01641, 0.0161, 0.01189, 0.01301, 0.01594, 0.01101, 0.01209, 0.0146, 0.01388, 0.01439, 0.01206, 0.01364, 0.01212, 0.01313, 0.01581, 0.01511, 0.01362, 0.01411, 0.0139, 0.01423, 0.01307, 0.01509, 0.01644, 0.01567, 0.01653, 0.01601, 0.0161, 0.01324, 0.01587, 0.01735, 0.01691, 0.01574, 0.01699, 0.01222, 0.01273, 0.0119]}, "layernorm-grads-all-reduce-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00124, 0.00087, 0.00088, 0.00087, 0.00086, 0.00085, 0.00085, 0.00085, 0.00098, 0.00088, 0.00087, 0.00087, 0.00087, 0.00088, 0.00085, 0.00085, 0.00086, 0.00082, 0.00084, 0.00083, 0.00103, 0.00352, 0.00085, 0.00084, 0.00084, 0.00089, 0.00086, 0.00084, 0.00085, 0.00084, 0.00085, 0.00087, 0.00085, 0.00085, 0.00086, 0.00086, 0.00084, 0.00086, 0.00086, 0.00085, 0.00087, 0.00086, 0.00085, 0.00087, 0.00084, 0.00086, 0.00085, 0.00084, 0.00167, 0.00083, 0.00086, 0.00111, 0.00108, 0.00101, 0.00084, 0.00085, 0.00085, 0.00086, 0.00084, 0.00084, 0.00086, 0.00083, 0.00083, 0.00083, 0.00111, 0.0009, 0.00086, 0.00088, 0.00086, 0.00084, 0.00086, 0.00084, 0.00091, 0.00085, 0.00084, 0.00087, 0.00083, 0.00083, 0.00241, 0.00085, 0.00086, 0.00109, 0.00086, 0.00085, 0.00085, 0.00084, 0.00084, 0.00084, 0.00084, 0.00092, 0.00087, 0.00083, 0.00087, 0.00532, 0.00083, 0.00085, 0.00101, 0.00113, 0.0011, 0.00089, 0.00088, 0.00086, 0.00113, 0.00084, 0.00122, 0.00087, 0.00086, 0.00085, 0.00086, 0.00088, 0.00085, 0.00088, 0.0031, 0.00085, 0.00087, 0.00085, 0.001, 0.00116, 0.00088, 0.00088, 0.00086, 0.00085, 0.00085, 0.00084, 0.00426, 0.00086, 0.00086, 0.00116, 0.00089, 0.00087, 0.00087, 0.00085, 0.00085, 0.00084, 0.00087, 0.00084, 0.00084, 0.0009, 0.00108, 0.00085, 0.00085, 0.00086, 0.00086, 0.00088, 0.00084, 0.00085, 0.00084, 0.00104, 0.00087, 0.00104, 0.00084, 0.00083, 0.00084, 0.00086, 0.00086, 0.00087, 0.00084, 0.00083, 0.00086, 0.00218, 0.00084, 0.004, 0.00086, 0.00087, 0.00087, 0.00105, 0.00103, 0.00103, 0.00107, 0.00089, 0.00107, 0.00114, 0.00113, 0.00085, 0.00107, 0.00086, 0.00089, 0.00088, 0.00089, 0.00086, 0.00085, 0.00085, 0.00086, 0.00088, 0.00087, 0.00085, 0.00086, 0.00087, 0.00085, 0.00085, 0.00087, 0.00089, 0.00085, 0.00088, 0.00087, 0.00086, 0.00241, 0.00085, 0.00084, 0.00087, 0.00099, 0.001, 0.00108, 0.00085, 0.00084, 0.00086, 0.00085, 0.00088, 0.00085, 0.00085, 0.00084, 0.00086, 0.00088, 0.00084, 0.00085, 0.00087, 0.00087, 0.00087, 0.00111, 0.00086, 0.00085, 0.00086, 0.00086, 0.00084, 0.00083, 0.00084, 0.00083, 0.00088, 0.00084, 0.00085, 0.0011, 0.0011, 0.00116, 0.00089, 0.00115, 0.00087, 0.00378, 0.00087, 0.00085, 0.00085, 0.0009, 0.00086, 0.00089, 0.00086, 0.00085, 0.00085, 0.00084, 0.00087, 0.00086, 0.00086, 0.00104, 0.00088, 0.00085, 0.00115, 0.00106, 0.00088, 0.00086, 0.00106, 0.00086, 0.00087, 0.00086, 0.0026, 0.00449, 0.00471, 0.00277, 0.00087, 0.00088, 0.00085, 0.00107, 0.0011, 0.00118, 0.00086, 0.00089, 0.00084, 0.00084, 0.00084, 0.00085, 0.00087, 0.00108, 0.0011, 0.00098, 0.00109, 0.00111, 0.0011, 0.0011, 0.0011, 0.0011, 0.00111, 0.00111, 0.00107, 0.0011, 0.00103, 0.00103, 0.00111, 0.00112, 0.00109, 0.00106, 0.00108, 0.00103, 0.00103, 0.00111, 0.00102, 0.00112, 0.00112, 0.00111, 0.00112, 0.00109, 0.00329, 0.00093, 0.00085, 0.00089, 0.00085, 0.00089, 0.00087, 0.00086, 0.00536, 0.0011, 0.00111, 0.00111, 0.00116, 0.00086, 0.00084, 0.00087, 0.0009, 0.00085, 0.00084, 0.00087, 0.00086, 0.00087, 0.00086, 0.00084, 0.00085, 0.00088, 0.00086, 0.00086, 0.00417, 0.00088, 0.00121, 0.00085, 0.00085, 0.00085, 0.00085, 0.00095, 0.00116, 0.00086, 0.00086, 0.00086, 0.00499, 0.00318, 0.00107, 0.00371, 0.00087, 0.00089, 0.00087, 0.00086, 0.00085, 0.00084, 0.00084, 0.00086, 0.00083, 0.00088, 0.00085, 0.00085, 0.00087, 0.00085, 0.00087, 0.00086, 0.00086, 0.00087, 0.00085, 0.00084, 0.00085, 0.00085, 0.00086, 0.00086, 0.00085, 0.00084, 0.00088, 0.00086, 0.00085, 0.00086, 0.00085, 0.0009, 0.00095, 0.00448, 0.00088, 0.00088, 0.00089, 0.00089, 0.00086, 0.00087, 0.00087, 0.0009, 0.00086, 0.00086, 0.00088, 0.00087, 0.00088, 0.0009, 0.00101]}, "embedding-grads-all-reduce-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00038, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00031, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00031, 0.00032, 0.00032, 0.00034, 0.00032, 0.00032, 0.00031, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00034, 0.00031, 0.00031, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00033, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00034, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00031, 0.00032, 0.00034, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00033, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00031, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00034, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00031, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00031, 0.00033, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00031, 0.00033, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00033, 0.00033, 0.00031, 0.00032, 0.00031, 0.00031, 0.00032, 0.00031, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00031, 0.00032, 0.00032, 0.00033, 0.00032, 0.00034, 0.00032, 0.00032, 0.00031, 0.00032, 0.00034, 0.00034, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032]}, "all-grads-sync-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.1656, 0.00059, 0.0006, 0.0006, 0.00059, 0.00062, 0.0006, 0.00059, 0.00058, 0.0006, 0.00059, 0.00058, 0.00059, 0.00059, 0.0006, 0.00058, 0.00059, 0.00058, 0.00059, 0.00058, 0.00059, 0.00065, 0.00064, 0.00063, 0.00059, 0.00059, 0.0006, 0.00058, 0.00059, 0.00059, 0.00059, 0.00059, 0.00061, 0.0006, 0.00058, 0.00064, 0.00058, 0.00058, 0.0006, 0.0006, 0.00058, 0.00058, 0.00059, 0.00059, 0.00063, 0.00059, 0.00058, 0.00058, 0.00059, 0.00059, 0.00059, 0.00059, 0.00063, 0.00059, 0.00059, 0.00059, 0.00058, 0.00059, 0.00064, 0.00058, 0.0006, 0.00059, 0.00059, 0.00059, 0.0006, 0.00059, 0.00059, 0.00059, 0.00059, 0.00058, 0.0006, 0.00058, 0.0006, 0.00059, 0.0006, 0.0006, 0.00057, 0.00059, 0.0006, 0.00058, 0.00059, 0.00059, 0.00064, 0.00058, 0.00059, 0.00063, 0.00059, 0.00058, 0.00059, 0.0006, 0.00059, 0.00058, 0.00058, 0.00058, 0.00059, 0.00058, 0.00058, 0.00059, 0.00058, 0.00059, 0.00059, 0.00059, 0.00059, 0.00063, 0.00059, 0.00059, 0.00059, 0.00057, 0.00058, 0.00059, 0.00058, 0.00062, 0.00059, 0.00059, 0.00059, 0.00058, 0.00059, 0.0006, 0.00058, 0.0006, 0.00058, 0.00062, 0.00059, 0.00063, 0.0006, 0.00059, 0.00059, 0.0006, 0.00059, 0.00059, 0.00059, 0.00058, 0.00063, 0.00058, 0.00059, 0.00058, 0.00058, 0.00058, 0.0006, 0.00058, 0.00058, 0.00059, 0.00059, 0.00059, 0.00058, 0.00057, 0.00059, 0.00058, 0.00059, 0.00059, 0.00058, 0.00058, 0.0006, 0.00058, 0.00058, 0.00059, 0.00058, 0.00059, 0.00059, 0.00058, 0.00058, 0.00059, 0.00059, 0.0006, 0.00063, 0.00059, 0.00059, 0.00058, 0.00059, 0.00062, 0.00062, 0.00058, 0.00059, 0.00059, 0.00058, 0.00059, 0.00059, 0.00058, 0.00059, 0.00058, 0.00058, 0.00059, 0.00059, 0.00059, 0.00058, 0.00059, 0.00059, 0.00058, 0.00059, 0.00059, 0.00058, 0.00059, 0.00059, 0.00058, 0.00058, 0.00059, 0.00074, 0.00059, 0.00059, 0.00059, 0.0006, 0.0006, 0.0006, 0.00058, 0.00058, 0.00058, 0.00059, 0.00059, 0.00059, 0.00059, 0.00058, 0.00059, 0.00059, 0.00059, 0.00058, 0.00059, 0.00058, 0.00064, 0.00059, 0.00063, 0.00059, 0.00059, 0.0006, 0.00058, 0.00058, 0.00058, 0.00058, 0.00059, 0.00058, 0.0006, 0.0006, 0.00059, 0.00058, 0.00058, 0.00057, 0.00058, 0.0006, 0.00059, 0.00058, 0.00058, 0.00059, 0.0006, 0.00058, 0.00058, 0.00059, 0.00058, 0.00057, 0.00058, 0.00059, 0.00058, 0.00059, 0.00059, 0.00059, 0.00058, 0.00058, 0.00058, 0.00059, 0.0006, 0.00058, 0.00065, 0.00059, 0.00062, 0.00058, 0.00057, 0.00061, 0.00059, 0.00059, 0.00058, 0.0006, 0.00063, 0.00059, 0.00058, 0.00059, 0.00058, 0.00062, 0.00059, 0.00058, 0.00059, 0.00058, 0.00058, 0.00058, 0.00058, 0.0006, 0.0006, 0.00059, 0.00058, 0.00059, 0.0006, 0.00058, 0.00059, 0.00059, 0.00059, 0.00059, 0.00058, 0.00058, 0.00058, 0.00059, 0.00064, 0.00058, 0.00057, 0.00059, 0.00058, 0.00059, 0.00057, 0.00058, 0.00058, 0.00058, 0.00059, 0.00059, 0.00064, 0.00058, 0.00058, 0.00059, 0.00058, 0.00058, 0.00058, 0.00057, 0.00059, 0.00058, 0.00059, 0.00058, 0.00058, 0.00058, 0.00059, 0.00059, 0.00058, 0.00058, 0.00058, 0.00059, 0.00059, 0.00059, 0.00059, 0.00059, 0.00058, 0.00058, 0.00063, 0.00058, 0.00063, 0.00059, 0.0006, 0.00057, 0.00058, 0.00059, 0.00059, 0.00059, 0.00059, 0.00062, 0.00062, 0.00058, 0.00057, 0.00058, 0.00059, 0.00059, 0.00059, 0.00059, 0.00059, 0.00059, 0.00059, 0.00059, 0.00059, 0.00059, 0.00059, 0.00059, 0.0006, 0.0006, 0.00058, 0.00058, 0.00059, 0.00063, 0.00057, 0.00059, 0.00063, 0.00059, 0.00059, 0.00059, 0.00058, 0.00058, 0.00058, 0.00059, 0.00059, 0.00063, 0.00059, 0.00059, 0.00059, 0.00059, 0.0006, 0.00059, 0.00059, 0.00059, 0.00059, 0.00059, 0.00059, 0.00058, 0.00059, 0.00058, 0.00059, 0.00059, 0.00058, 0.00059, 0.00059]}, "optimizer-copy-to-main-grad-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00016, 0.00012, 0.00011, 0.00011, 0.00011, 0.00011, 0.00012, 0.00011, 0.00012, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.0001, 0.00011, 0.00011, 0.0001, 0.00011, 0.0001, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.00011, 0.00011, 0.00012, 0.00011, 0.0001, 0.0001, 0.00011, 0.00011, 0.00012, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.0001, 0.0001, 0.00011, 0.0001, 0.00011, 0.0001, 0.00011, 0.00011, 0.0001, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.0001, 0.00011, 0.00011, 0.0001, 0.0001, 0.00011, 0.00011, 0.00011, 0.0001, 0.00012, 0.00011, 0.00011, 0.00012, 0.0001, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.0001, 0.00012, 0.00011, 0.00012, 0.0001, 0.0001, 0.00011, 0.00011, 0.0001, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.00011, 0.00011, 0.0001, 0.0001, 0.00012, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.0001, 0.0001, 0.00011, 0.0001, 0.00011, 0.0001, 0.0001, 0.00011, 0.00011, 0.00011, 0.00011, 0.0001, 0.0001, 0.00011, 0.00011, 0.00011, 0.0001, 0.0001, 0.0001, 0.00011, 0.00011, 0.00011, 0.0001, 0.00012, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.00011, 0.0001, 0.0001, 0.00012, 0.00012, 0.00011, 0.0001, 0.0001, 0.0001, 0.00011, 0.00011, 0.0001, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00012, 0.00011, 0.0001, 0.0001, 0.00011, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.00011, 0.0001, 0.0001, 0.0001, 0.00011, 0.00011, 0.00012, 0.00011, 0.0001, 0.0001, 0.00011, 0.0001, 0.00011, 0.00011, 0.00011, 0.00011, 0.00012, 0.0001, 0.00011, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.0001, 0.0001, 0.0001, 0.0001, 0.00011, 0.0001, 0.0001, 0.0001, 0.0001, 0.00011, 0.00011, 0.0001, 0.00011, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.0001, 0.00011, 0.0001, 0.0001, 0.0001, 0.0001, 0.00011, 0.0001, 0.0001, 0.0001, 0.0001, 0.00011, 0.0001, 0.0001, 0.0001, 0.0001, 0.00011, 0.00011, 0.0001, 0.0001, 0.0001, 0.0001, 0.00011, 0.00011, 0.0001, 0.00011, 0.0001, 0.0001, 0.0001, 0.00011, 0.00011, 0.00011, 0.0001, 0.0001, 0.00011, 0.00011, 0.00011, 0.00012, 0.0001, 0.00011, 0.00011, 0.00011, 0.0001, 0.0001, 0.00011, 0.00011, 0.00011, 0.0001, 0.0001, 0.0001, 0.0001, 0.00011, 0.0001, 0.0001, 0.0001, 0.0001, 0.00011, 0.0001, 0.00012, 0.00011, 0.00011, 0.00011, 0.0001, 0.0001, 0.0001, 0.0001, 0.00011, 0.0001, 0.00012, 0.0001, 0.00011, 0.00011, 0.0001, 0.0001, 0.0001, 0.0001, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.0001, 0.00011, 0.00011, 0.00019, 0.0001, 0.0001, 0.00011, 0.00011, 0.00011, 0.0001, 0.0001, 0.00011, 0.0001, 0.0001, 0.0001, 0.00011, 0.00011, 0.0001, 0.0001, 0.0001, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.0001, 0.0001, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.0001, 0.00012, 0.00011, 0.0001, 0.0001, 0.0001, 0.00011, 0.00011, 0.0001, 0.0001, 0.0001, 0.00011, 0.00011, 0.0001, 0.0001, 0.0001, 0.00011, 0.00011, 0.0001, 0.0001, 0.0001, 0.00011, 0.00011, 0.0001, 0.0001, 0.0001, 0.0001, 0.00011, 0.0001, 0.00011, 0.0001, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.00011, 0.0001, 0.00011, 0.00011, 0.00011]}, "optimizer-clip-main-grad-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [4.25848, 0.00058, 0.00058, 0.00057, 0.00057, 0.00058, 0.00058, 0.00057, 0.00057, 0.00058, 0.00057, 0.00057, 0.00056, 0.00056, 0.00057, 0.00056, 0.00059, 0.00056, 0.00056, 0.00055, 0.00056, 0.00057, 0.00056, 0.00056, 0.00056, 0.00057, 0.00056, 0.00056, 0.00057, 0.00056, 0.00057, 0.00056, 0.00055, 0.00055, 0.00057, 0.00057, 0.00058, 0.00055, 0.00056, 0.00056, 0.00056, 0.00055, 0.00057, 0.00056, 0.00056, 0.00056, 0.00058, 0.00056, 0.00055, 0.00056, 0.00056, 0.00057, 0.00057, 0.0006, 0.00056, 0.00056, 0.00055, 0.00056, 0.00056, 0.00055, 0.00056, 0.00055, 0.00055, 0.00056, 0.00057, 0.00057, 0.00056, 0.00056, 0.00056, 0.00057, 0.00059, 0.00056, 0.00058, 0.00056, 0.00056, 0.00057, 0.00055, 0.00055, 0.00056, 0.00056, 0.00056, 0.00071, 0.00056, 0.00056, 0.00057, 0.00057, 0.00055, 0.00056, 0.00055, 0.0006, 0.00055, 0.00056, 0.00055, 0.00055, 0.00057, 0.00055, 0.00055, 0.00057, 0.00046, 0.00057, 0.00057, 0.00057, 0.00056, 0.00055, 0.00071, 0.00056, 0.00056, 0.00057, 0.00057, 0.00047, 0.00056, 0.00048, 0.00046, 0.00056, 0.00057, 0.00055, 0.00055, 0.00056, 0.00055, 0.00057, 0.00056, 0.00056, 0.00056, 0.00056, 0.00046, 0.00056, 0.00055, 0.00055, 0.00056, 0.00058, 0.00045, 0.00056, 0.00057, 0.00055, 0.00057, 0.00055, 0.00055, 0.00055, 0.00056, 0.00056, 0.00055, 0.00055, 0.00057, 0.00046, 0.00046, 0.00056, 0.00056, 0.00056, 0.00056, 0.00056, 0.00055, 0.00055, 0.00055, 0.00055, 0.00056, 0.00057, 0.00055, 0.00055, 0.00057, 0.00057, 0.00064, 0.00056, 0.00056, 0.00057, 0.00057, 0.00055, 0.00056, 0.00055, 0.00055, 0.00056, 0.00056, 0.00057, 0.00056, 0.00056, 0.00056, 0.00056, 0.00055, 0.00055, 0.00056, 0.00055, 0.00055, 0.00056, 0.00057, 0.00056, 0.00057, 0.00056, 0.00055, 0.00058, 0.00056, 0.00056, 0.00056, 0.00056, 0.00055, 0.00056, 0.00056, 0.00055, 0.00056, 0.00056, 0.00055, 0.00057, 0.00077, 0.00056, 0.00056, 0.00057, 0.00056, 0.00057, 0.00056, 0.00057, 0.00056, 0.00056, 0.00055, 0.00056, 0.00058, 0.00055, 0.00056, 0.00055, 0.00056, 0.00056, 0.00056, 0.00056, 0.00056, 0.00056, 0.00056, 0.00056, 0.00056, 0.00055, 0.00055, 0.00054, 0.00055, 0.00055, 0.00056, 0.00062, 0.00058, 0.00055, 0.00056, 0.00056, 0.00056, 0.00056, 0.00055, 0.00055, 0.00055, 0.00056, 0.00056, 0.00055, 0.00057, 0.00057, 0.00056, 0.00055, 0.00055, 0.00055, 0.00055, 0.00058, 0.00055, 0.00056, 0.00056, 0.00056, 0.00055, 0.00056, 0.00055, 0.00056, 0.00056, 0.00057, 0.00057, 0.00061, 0.00057, 0.00057, 0.00056, 0.00057, 0.00055, 0.00056, 0.00056, 0.00056, 0.00058, 0.00056, 0.00057, 0.00055, 0.0006, 0.00056, 0.00057, 0.00056, 0.00055, 0.00056, 0.00056, 0.00057, 0.00057, 0.00055, 0.00056, 0.00056, 0.0006, 0.00063, 0.00057, 0.00056, 0.00056, 0.00057, 0.00058, 0.00056, 0.00059, 0.00057, 0.00056, 0.00055, 0.00056, 0.00064, 0.00056, 0.00056, 0.00057, 0.00056, 0.00056, 0.00057, 0.00068, 0.00056, 0.00056, 0.00056, 0.00058, 0.00056, 0.00059, 0.00056, 0.00055, 0.00057, 0.00057, 0.00055, 0.00057, 0.00056, 0.00057, 0.00057, 0.00056, 0.00056, 0.00055, 0.00057, 0.00057, 0.00055, 0.00056, 0.00057, 0.00056, 0.00056, 0.00056, 0.00057, 0.00056, 0.00056, 0.00058, 0.00056, 0.00055, 0.00056, 0.00056, 0.00057, 0.00056, 0.00056, 0.00076, 0.00058, 0.00057, 0.00057, 0.00056, 0.00056, 0.00057, 0.00056, 0.00056, 0.00056, 0.00057, 0.00057, 0.00056, 0.00055, 0.00055, 0.00057, 0.00056, 0.00056, 0.00056, 0.00055, 0.00056, 0.00057, 0.00056, 0.00055, 0.00061, 0.00056, 0.00055, 0.00056, 0.00055, 0.00056, 0.00056, 0.00055, 0.00057, 0.00055, 0.00055, 0.00056, 0.00057, 0.00056, 0.00057, 0.00056, 0.00056, 0.00056, 0.00057, 0.00057, 0.00057, 0.00057, 0.00057, 0.00057, 0.00057, 0.00056, 0.00056, 0.00056, 0.00056, 0.00056]}, "optimizer-count-zeros-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00381, 0.00273, 0.0027, 0.0027, 0.00273, 0.00271, 0.00267, 0.00283, 0.00274, 0.00269, 0.0027, 0.00269, 0.00272, 0.00273, 0.0027, 0.0027, 0.00269, 0.00268, 0.0027, 0.0027, 0.00273, 0.00272, 0.00268, 0.0027, 0.00278, 0.00278, 0.00271, 0.00269, 0.00268, 0.0027, 0.00271, 0.00271, 0.00269, 0.00273, 0.00271, 0.0027, 0.00267, 0.00269, 0.0027, 0.00271, 0.00271, 0.00269, 0.00269, 0.00267, 0.00269, 0.00269, 0.00269, 0.0027, 0.0027, 0.00271, 0.00271, 0.00288, 0.00277, 0.00297, 0.0027, 0.00269, 0.00268, 0.00269, 0.00268, 0.00269, 0.00269, 0.0027, 0.00268, 0.0027, 0.00272, 0.00269, 0.0027, 0.00271, 0.00273, 0.0027, 0.00284, 0.0027, 0.00271, 0.00282, 0.0027, 0.00268, 0.00268, 0.00268, 0.0027, 0.0027, 0.00272, 0.00496, 0.0027, 0.00268, 0.00269, 0.00269, 0.00271, 0.00269, 0.00271, 0.00292, 0.0027, 0.00269, 0.00269, 0.00268, 0.00269, 0.00271, 0.00271, 0.00275, 0.00271, 0.00271, 0.00268, 0.00271, 0.00291, 0.00269, 0.00286, 0.00271, 0.00269, 0.00269, 0.00271, 0.00269, 0.0027, 0.00272, 0.00269, 0.00267, 0.00268, 0.00269, 0.00272, 0.00269, 0.00272, 0.0027, 0.00268, 0.00268, 0.00269, 0.0027, 0.00269, 0.0027, 0.00272, 0.0027, 0.00271, 0.00269, 0.00273, 0.0027, 0.0027, 0.0027, 0.00268, 0.00269, 0.0027, 0.00272, 0.00271, 0.00271, 0.00269, 0.0027, 0.00267, 0.00271, 0.00269, 0.00268, 0.00268, 0.0027, 0.00269, 0.00269, 0.00267, 0.0027, 0.00268, 0.00269, 0.0027, 0.0027, 0.00269, 0.00269, 0.00268, 0.00269, 0.00269, 0.00269, 0.00269, 0.00281, 0.0028, 0.00273, 0.00272, 0.00273, 0.00273, 0.00274, 0.00271, 0.00272, 0.0027, 0.00271, 0.0027, 0.00271, 0.00273, 0.00271, 0.00269, 0.00271, 0.00272, 0.00272, 0.00272, 0.0027, 0.00269, 0.00281, 0.00272, 0.00282, 0.00271, 0.0027, 0.00269, 0.00272, 0.00273, 0.00271, 0.00269, 0.0027, 0.0027, 0.00269, 0.00271, 0.00271, 0.00282, 0.00271, 0.00269, 0.00271, 0.0027, 0.00313, 0.0027, 0.00269, 0.00271, 0.00271, 0.0027, 0.0027, 0.00271, 0.00269, 0.00278, 0.00269, 0.00272, 0.00278, 0.00271, 0.0027, 0.00269, 0.00271, 0.0027, 0.0027, 0.0027, 0.00269, 0.00271, 0.00271, 0.00269, 0.00272, 0.00271, 0.00296, 0.00271, 0.00271, 0.0027, 0.00271, 0.00271, 0.00275, 0.00269, 0.00267, 0.00271, 0.00274, 0.00267, 0.00271, 0.0027, 0.00273, 0.00272, 0.00271, 0.00271, 0.00273, 0.00272, 0.0027, 0.00274, 0.00273, 0.0027, 0.00272, 0.00271, 0.0027, 0.00271, 0.00265, 0.00264, 0.00264, 0.00273, 0.00262, 0.00291, 0.00266, 0.00273, 0.00265, 0.00265, 0.00263, 0.00265, 0.00264, 0.00274, 0.00272, 0.00262, 0.00274, 0.00265, 0.00273, 0.00264, 0.00274, 0.00264, 0.00274, 0.0028, 0.00265, 0.00263, 0.00263, 0.00272, 0.00271, 0.00276, 0.00267, 0.00265, 0.00262, 0.00272, 0.00277, 0.00264, 0.00269, 0.00264, 0.00264, 0.00272, 0.00271, 0.00294, 0.00388, 0.00268, 0.00273, 0.00273, 0.00265, 0.00357, 0.00265, 0.00304, 0.00272, 0.00261, 0.00268, 0.0027, 0.00266, 0.00267, 0.00264, 0.00278, 0.00274, 0.00267, 0.00269, 0.00268, 0.0027, 0.00269, 0.0027, 0.00269, 0.0027, 0.00271, 0.00269, 0.00267, 0.0027, 0.00268, 0.0027, 0.00272, 0.00271, 0.0027, 0.00272, 0.00272, 0.00274, 0.00269, 0.00313, 0.00269, 0.00269, 0.00269, 0.00271, 0.00271, 0.00273, 0.00283, 0.0027, 0.00269, 0.00278, 0.00276, 0.00271, 0.00271, 0.0027, 0.0027, 0.00271, 0.00272, 0.00271, 0.00272, 0.00271, 0.00271, 0.00268, 0.00273, 0.00271, 0.00269, 0.0027, 0.00273, 0.00275, 0.00269, 0.00273, 0.00271, 0.00271, 0.0027, 0.00272, 0.00269, 0.00269, 0.00272, 0.00274, 0.00271, 0.00272, 0.00272, 0.0027, 0.0027, 0.00272, 0.0027, 0.00271, 0.00271, 0.00273, 0.00271, 0.00268, 0.0027, 0.00271, 0.00273, 0.00272, 0.0027, 0.00269, 0.00272, 0.00272, 0.0027, 0.00271]}, "optimizer-inner-step-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.0026, 0.00046, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00044, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00044, 0.00045, 0.00044, 0.00045, 0.00045, 0.00044, 0.00051, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00044, 0.00046, 0.00045, 0.00046, 0.00044, 0.00045, 0.00045, 0.00044, 0.00045, 0.00044, 0.00044, 0.00044, 0.00045, 0.00046, 0.00045, 0.00044, 0.00044, 0.00044, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00046, 0.00044, 0.00045, 0.00046, 0.00045, 0.00045, 0.00048, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00046, 0.00045, 0.00044, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00047, 0.00044, 0.00057, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.0005, 0.00044, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00047, 0.00045, 0.00045, 0.00045, 0.00046, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00049, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00044, 0.00045, 0.00045, 0.00059, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00051, 0.00046, 0.00046, 0.00046, 0.00045, 0.00045, 0.00046, 0.00046, 0.00061, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00046, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00044, 0.00044, 0.00045, 0.00045, 0.00044, 0.00046, 0.00054, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00044, 0.00045, 0.00044, 0.00044, 0.00044, 0.00045, 0.00044, 0.00044, 0.00045, 0.00044, 0.00045, 0.00044, 0.00044, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00046, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00046, 0.00045, 0.00046, 0.00055, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00044, 0.00045, 0.00046, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00044, 0.00044, 0.00045, 0.00046, 0.00045, 0.00044, 0.00076, 0.00046, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00046, 0.00045, 0.00046, 0.00045, 0.00045, 0.00046, 0.00044, 0.00045, 0.00048, 0.00045, 0.00045, 0.00048, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00052, 0.0005, 0.00056, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00044, 0.00045, 0.00046, 0.00044, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00044, 0.00045, 0.00045, 0.00046, 0.00045, 0.00046, 0.00046, 0.00046, 0.00045, 0.00045, 0.00046, 0.00046, 0.00046, 0.00046, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045]}, "optimizer-copy-main-to-model-params-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00055, 0.00049, 0.0005, 0.0005, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.0005, 0.0005, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.0005, 0.00049, 0.00051, 0.00049, 0.00049, 0.00049, 0.00066, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00048, 0.00049, 0.0005, 0.00049, 0.00049, 0.00068, 0.00049, 0.0005, 0.00049, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.0005, 0.00049, 0.0005, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00067, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00063, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00048, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00068, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00076, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00051, 0.00049, 0.0005, 0.00049, 0.0005, 0.00049, 0.00052, 0.00049, 0.00066, 0.0005, 0.00049, 0.00049, 0.00049, 0.0005, 0.00049, 0.00049, 0.0005, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.0005, 0.0005, 0.0005, 0.00049, 0.00049, 0.0005, 0.0005, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00051, 0.0005, 0.0005, 0.00072, 0.00049, 0.0005, 0.00049, 0.00049, 0.0005, 0.00049, 0.0005, 0.00049, 0.00049, 0.0005, 0.00049, 0.00049, 0.0005, 0.0005, 0.00049, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.00049, 0.0005, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.0005, 0.00052, 0.0005, 0.00049, 0.00049, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00066, 0.0005, 0.00049, 0.0005, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00052, 0.0005, 0.00049, 0.0005, 0.00049, 0.00049, 0.0005, 0.00049, 0.00049, 0.00053, 0.00049, 0.00052, 0.00049, 0.00049, 0.00049, 0.00076, 0.00049, 0.0005, 0.00049, 0.0005, 0.00049, 0.00064, 0.0005, 0.00051, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.0005, 0.00066, 0.00049, 0.00051, 0.00063, 0.0005, 0.0005, 0.0005, 0.00049, 0.00049, 0.0005, 0.0005, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.0005, 0.0005, 0.0005, 0.00051, 0.00049, 0.00049, 0.0005, 0.0005, 0.0005, 0.00053, 0.0005, 0.00073, 0.00072, 0.00072, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.0005, 0.00051, 0.00051, 0.0005, 0.00049, 0.0005, 0.0005, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.0005, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.00049, 0.0005, 0.00049, 0.00049, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.0005, 0.00051, 0.0005, 0.0005, 0.0005, 0.00049, 0.0005]}, "optimizer-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [4.26785, 0.00472, 0.00469, 0.00468, 0.0047, 0.00469, 0.00466, 0.00479, 0.00473, 0.00465, 0.00467, 0.00466, 0.00467, 0.00467, 0.00464, 0.00466, 0.00468, 0.00461, 0.00465, 0.00464, 0.00469, 0.00469, 0.00464, 0.00465, 0.00473, 0.00473, 0.00467, 0.00463, 0.00464, 0.00465, 0.00468, 0.00467, 0.00464, 0.00516, 0.00466, 0.00468, 0.00465, 0.00465, 0.00465, 0.00469, 0.00466, 0.00464, 0.00465, 0.00462, 0.00463, 0.00466, 0.00466, 0.00464, 0.00465, 0.00466, 0.00468, 0.00483, 0.00473, 0.005, 0.00465, 0.00465, 0.00463, 0.00466, 0.00463, 0.00463, 0.00465, 0.00465, 0.00461, 0.00465, 0.00467, 0.00467, 0.00464, 0.00464, 0.00468, 0.00465, 0.00483, 0.00466, 0.0047, 0.00478, 0.00466, 0.00466, 0.00461, 0.00462, 0.00467, 0.00465, 0.00469, 0.00749, 0.00467, 0.00465, 0.00466, 0.00466, 0.00465, 0.00465, 0.00465, 0.00495, 0.00465, 0.00465, 0.00463, 0.00463, 0.00466, 0.00467, 0.00464, 0.00472, 0.00456, 0.00469, 0.00464, 0.00466, 0.0049, 0.00463, 0.00555, 0.00466, 0.00464, 0.00464, 0.00466, 0.00456, 0.00466, 0.0046, 0.00453, 0.00464, 0.00465, 0.00461, 0.00466, 0.00495, 0.00466, 0.00467, 0.00463, 0.00461, 0.00463, 0.00465, 0.00458, 0.00465, 0.00467, 0.00464, 0.00466, 0.00467, 0.00456, 0.00464, 0.00465, 0.00464, 0.00465, 0.00462, 0.00462, 0.00464, 0.00466, 0.00465, 0.00464, 0.00465, 0.00463, 0.00456, 0.00455, 0.00464, 0.00462, 0.00466, 0.00464, 0.00466, 0.00461, 0.00462, 0.00463, 0.00464, 0.00468, 0.00465, 0.00462, 0.00463, 0.00466, 0.00465, 0.00472, 0.00464, 0.00465, 0.00477, 0.00511, 0.00469, 0.00467, 0.00467, 0.00468, 0.00471, 0.00465, 0.00468, 0.00465, 0.00522, 0.00464, 0.00465, 0.00466, 0.00465, 0.00464, 0.00465, 0.00465, 0.00466, 0.00467, 0.00466, 0.00464, 0.00475, 0.00467, 0.0048, 0.00468, 0.00466, 0.00466, 0.00467, 0.00478, 0.00466, 0.00469, 0.00465, 0.00466, 0.00465, 0.00499, 0.0047, 0.00568, 0.00465, 0.00465, 0.00466, 0.00466, 0.00541, 0.00464, 0.00465, 0.00465, 0.00465, 0.00463, 0.00465, 0.00469, 0.00464, 0.00473, 0.00463, 0.00466, 0.00474, 0.00466, 0.00465, 0.00464, 0.00467, 0.00464, 0.00466, 0.00464, 0.00462, 0.00464, 0.00466, 0.00463, 0.00467, 0.00467, 0.00542, 0.00468, 0.00466, 0.00465, 0.00465, 0.00467, 0.0047, 0.00463, 0.00461, 0.00466, 0.00468, 0.00464, 0.00466, 0.00467, 0.00468, 0.00467, 0.00465, 0.00467, 0.00468, 0.00465, 0.00469, 0.00468, 0.00468, 0.00464, 0.00466, 0.00467, 0.00464, 0.00464, 0.00461, 0.00462, 0.00463, 0.0047, 0.00464, 0.00489, 0.00464, 0.00469, 0.0046, 0.00459, 0.00459, 0.0046, 0.00459, 0.00472, 0.00501, 0.00458, 0.00468, 0.00465, 0.00469, 0.00461, 0.00469, 0.00458, 0.0047, 0.00478, 0.0046, 0.00464, 0.00461, 0.00468, 0.00468, 0.00476, 0.00469, 0.00461, 0.00457, 0.00469, 0.00472, 0.00468, 0.00464, 0.00467, 0.00461, 0.00467, 0.00463, 0.00558, 0.00601, 0.00464, 0.0047, 0.0047, 0.00459, 0.00574, 0.00463, 0.00519, 0.00467, 0.00462, 0.00464, 0.00469, 0.00461, 0.00476, 0.00462, 0.00501, 0.00471, 0.00465, 0.0049, 0.00465, 0.00465, 0.00465, 0.00465, 0.00462, 0.00466, 0.00466, 0.00465, 0.00463, 0.00464, 0.00464, 0.00465, 0.00468, 0.00466, 0.00465, 0.00469, 0.00468, 0.0047, 0.00466, 0.00514, 0.00464, 0.00465, 0.00469, 0.00468, 0.00511, 0.00511, 0.00571, 0.00469, 0.00467, 0.00473, 0.00471, 0.00465, 0.00469, 0.00466, 0.00464, 0.00465, 0.00468, 0.00467, 0.00468, 0.00465, 0.00464, 0.00464, 0.00468, 0.00467, 0.00464, 0.00464, 0.00467, 0.00472, 0.00466, 0.00466, 0.00473, 0.00466, 0.00465, 0.00468, 0.00463, 0.00465, 0.00465, 0.00469, 0.00467, 0.00465, 0.00469, 0.00464, 0.00467, 0.00468, 0.00468, 0.00467, 0.00468, 0.00469, 0.00467, 0.00465, 0.00466, 0.00468, 0.0047, 0.0047, 0.00469, 0.00467, 0.00475, 0.00469, 0.00466, 0.00467]}, "learning-rate": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05]}, "learning-rate vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05]}, "batch-size": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0]}, "batch-size vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0]}, "lm loss": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [10.8433, 10.87237, 10.85095, 10.81043, 10.6448, 10.63777, 10.42844, 10.13521, 9.93305, 9.83545, 9.58571, 9.84725, 9.88565, 9.63113, 9.78975, 9.51098, 9.46049, 9.65567, 9.38995, 9.33878, 9.24969, 9.1513, 9.18163, 9.00531, 9.19823, 9.06713, 9.1611, 9.17005, 9.3017, 8.9895, 8.93016, 9.05038, 9.04655, 8.66038, 8.72409, 8.75638, 8.69407, 8.74224, 8.66588, 8.77332, 8.66981, 8.86037, 8.84252, 8.50864, 8.39881, 8.43745, 8.49708, 8.39264, 8.44075, 8.59292, 8.37673, 8.20006, 8.23344, 8.22992, 8.27498, 7.92069, 8.10023, 7.89834, 8.25194, 8.23411, 8.01021, 7.97604, 7.92659, 7.7431, 7.74693, 7.65012, 7.52119, 7.91055, 7.70207, 7.45595, 7.74651, 7.77427, 7.54475, 7.30211, 7.45561, 7.34181, 7.46593, 7.22843, 7.63637, 7.28176, 7.3489, 7.21432, 7.21203, 7.41989, 7.17357, 7.28165, 6.99531, 7.00302, 7.03928, 7.13515, 6.82262, 6.98384, 7.08844, 6.99761, 6.87404, 6.75706, 6.99011, 7.05967, 6.70357, 6.58305, 6.72733, 6.74414, 6.73255, 6.73774, 6.65784, 6.40634, 6.63614, 6.61858, 6.44649, 6.62891, 6.74367, 6.61188, 6.72737, 6.69765, 6.62758, 6.50905, 6.60081, 6.41086, 6.6679, 6.25211, 6.25445, 6.3058, 6.39337, 6.35086, 6.45124, 6.29329, 6.34001, 6.23796, 6.20375, 6.39631, 6.32396, 6.32157, 6.16598, 6.16128, 6.23961, 6.38624, 6.20441, 6.15484, 6.18327, 6.11856, 6.0643, 6.07587, 6.25885, 6.40985, 6.25773, 6.29364, 6.09777, 6.17617, 6.00018, 6.02579, 5.95395, 6.25004, 6.1835, 5.9641, 5.78086, 6.1243, 5.84676, 6.10204, 5.78497, 6.16105, 6.14236, 6.08122, 5.92779, 6.11353, 5.94712, 6.19855, 5.89495, 5.79053, 5.78161, 5.68895, 6.01539, 6.00005, 6.07273, 5.88766, 6.04042, 5.96921, 5.9968, 5.99511, 5.95382, 5.84206, 5.94819, 5.61857, 5.70118, 5.88914, 5.84134, 5.85987, 5.76315, 5.83815, 5.72167, 5.55909, 5.7186, 5.61929, 5.82758, 5.59625, 5.7042, 5.70308, 5.89746, 5.6397, 5.8423, 5.73483, 5.86656, 5.3246, 5.89117, 5.87078, 5.84956, 5.41021, 5.40477, 5.62248, 5.59081, 5.47867, 5.57199, 5.67087, 5.47386, 5.73778, 5.50719, 5.5907, 5.61801, 5.61375, 5.51366, 5.61481, 5.66685, 5.6779, 5.58491, 5.65921, 5.37261, 5.67583, 5.62837, 5.42192, 5.58097, 5.62665, 5.55611, 5.34326, 5.53554, 5.48465, 5.48233, 5.38246, 5.55371, 5.59988, 5.3888, 5.51915, 5.48693, 5.33624, 5.50426, 5.40732, 5.44588, 5.31986, 5.06542, 5.47702, 5.5691, 5.71712, 5.4168, 5.60428, 5.63765, 5.23416, 5.27033, 5.39354, 5.39714, 5.32901, 5.4987, 5.18235, 5.2957, 5.24436, 5.37457, 5.2529, 5.44104, 5.53543, 5.31003, 5.43328, 5.33746, 5.0731, 5.3098, 5.25225, 5.30292, 5.11018, 5.27443, 5.26715, 5.47556, 5.15707, 5.26288, 5.20645, 5.35219, 4.98181, 4.9111, 5.32523, 5.39056, 5.22715, 5.31629, 5.10465, 5.16067, 5.26308, 5.06303, 5.26135, 5.06321, 5.3436, 5.24949, 5.14663, 5.23912, 5.03809, 5.31464, 5.05119, 5.02764, 5.1413, 5.10928, 5.27105, 5.15582, 5.27468, 5.09195, 5.0903, 5.24747, 5.32385, 5.25035, 5.18939, 5.14008, 5.28936, 4.94914, 5.20395, 5.09147, 5.29734, 5.1695, 5.18774, 5.11232, 4.98053, 4.98857, 5.21914, 5.31229, 5.09605, 5.05198, 4.91409, 5.12399, 5.11458, 4.92544, 5.3328, 5.02108, 5.09621, 5.16445, 5.00235, 5.06211, 5.06284, 4.99345, 5.07584, 5.16228, 4.97677, 5.17728, 4.92784, 4.918, 5.06063, 4.99291, 4.90737, 4.77256, 4.94113, 5.11089, 5.01099, 5.01211, 5.32888, 4.95413, 4.98755, 5.04195, 4.80724, 4.73022, 4.99215, 5.04011, 4.87028, 4.95205, 5.04766, 5.02175, 4.81256, 4.89346, 4.90447, 4.8296, 4.73532, 5.01127, 4.74826, 5.20326, 4.78795, 4.98997, 4.73269, 4.78049, 4.81697, 4.6476, 4.65082, 4.84007, 4.80171, 4.79196, 4.91846, 4.88285, 4.91969, 4.76846, 4.87797, 4.72424, 4.9076, 4.94932, 4.86605, 4.70549, 4.77921, 4.89662, 4.7052, 4.86264, 4.69237, 4.69072, 4.64046]}, "lm loss vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [10.84303, 10.87155, 10.85032, 10.81087, 10.64537, 10.63943, 10.42704, 10.13551, 9.93496, 9.83494, 9.58592, 9.84757, 9.88552, 9.63097, 9.79022, 9.51147, 9.4606, 9.65582, 9.39007, 9.33886, 9.24978, 9.152, 9.18226, 9.00447, 9.19856, 9.06681, 9.16059, 9.16939, 9.30049, 8.98819, 8.92948, 9.0507, 9.0463, 8.66041, 8.72526, 8.75716, 8.69559, 8.74303, 8.66681, 8.77472, 8.67057, 8.8619, 8.84447, 8.50989, 8.39988, 8.43941, 8.49864, 8.39575, 8.4422, 8.59464, 8.37842, 8.20138, 8.236, 8.2319, 8.27672, 7.92273, 8.10152, 7.8984, 8.25217, 8.23541, 8.01089, 7.97596, 7.92706, 7.74403, 7.7485, 7.65015, 7.52079, 7.9112, 7.70347, 7.45605, 7.74759, 7.77568, 7.54533, 7.30357, 7.45723, 7.3426, 7.46645, 7.22831, 7.63649, 7.28211, 7.34866, 7.21221, 7.21132, 7.41795, 7.17177, 7.28168, 6.99581, 7.004, 7.04074, 7.1367, 6.82354, 6.98508, 7.08921, 6.99769, 6.87461, 6.75657, 6.99031, 7.05959, 6.70411, 6.5827, 6.72604, 6.74348, 6.73218, 6.73708, 6.65685, 6.4055, 6.63559, 6.61892, 6.44639, 6.62609, 6.74333, 6.61179, 6.7261, 6.69431, 6.62741, 6.50922, 6.59901, 6.40739, 6.6657, 6.24852, 6.25199, 6.30265, 6.39086, 6.34866, 6.4484, 6.29117, 6.33917, 6.23682, 6.20019, 6.39713, 6.32382, 6.32063, 6.16132, 6.15692, 6.23736, 6.38207, 6.20216, 6.14927, 6.18286, 6.11574, 6.06273, 6.07513, 6.25658, 6.40785, 6.25681, 6.2924, 6.09673, 6.17564, 6.00002, 6.02568, 5.95394, 6.24995, 6.18499, 5.96441, 5.78379, 6.12452, 5.8475, 6.10173, 5.78491, 6.16542, 6.14406, 6.08134, 5.92727, 6.11254, 5.94363, 6.20077, 5.89399, 5.7901, 5.78128, 5.68813, 6.01482, 5.99528, 6.06741, 5.89085, 6.03981, 5.96811, 5.99655, 5.98984, 5.94628, 5.83848, 5.9481, 5.61614, 5.7002, 5.88656, 5.83806, 5.86311, 5.75859, 5.83316, 5.72072, 5.55659, 5.71965, 5.61978, 5.82718, 5.59717, 5.70318, 5.70327, 5.89853, 5.63883, 5.84367, 5.73571, 5.86365, 5.32462, 5.89684, 5.87059, 5.85018, 5.40966, 5.40521, 5.6244, 5.59463, 5.48385, 5.57514, 5.67111, 5.47486, 5.74063, 5.50617, 5.58954, 5.62055, 5.61722, 5.51063, 5.6138, 5.67042, 5.67814, 5.58421, 5.65728, 5.36779, 5.67697, 5.62608, 5.41953, 5.57893, 5.62664, 5.55034, 5.33858, 5.53624, 5.48821, 5.48891, 5.37489, 5.5499, 5.60024, 5.39139, 5.51868, 5.4935, 5.33216, 5.50746, 5.41318, 5.44698, 5.31869, 5.06634, 5.48126, 5.57099, 5.71639, 5.41515, 5.60293, 5.63581, 5.23321, 5.27358, 5.3934, 5.40049, 5.32861, 5.49563, 5.18115, 5.29818, 5.24632, 5.377, 5.25164, 5.44247, 5.53356, 5.31175, 5.43649, 5.33683, 5.07482, 5.31199, 5.25123, 5.30045, 5.10952, 5.27365, 5.26615, 5.4733, 5.15569, 5.2676, 5.21227, 5.35586, 4.98451, 4.91017, 5.32431, 5.38997, 5.22667, 5.3209, 5.10232, 5.16141, 5.26239, 5.0658, 5.26091, 5.06389, 5.34895, 5.24827, 5.1463, 5.24113, 5.03942, 5.31795, 5.05285, 5.02784, 5.14139, 5.11164, 5.27303, 5.15115, 5.2757, 5.09401, 5.09338, 5.24504, 5.32369, 5.25347, 5.19226, 5.14165, 5.29079, 4.95338, 5.20578, 5.09105, 5.30122, 5.17357, 5.19235, 5.11365, 4.98113, 4.9916, 5.22149, 5.30937, 5.10092, 5.0529, 4.91086, 5.12305, 5.11531, 4.92812, 5.3389, 5.02814, 5.10063, 5.16722, 5.00342, 5.0656, 5.06853, 5.0, 5.08165, 5.16456, 4.98252, 5.1839, 4.93148, 4.92569, 5.06682, 4.99595, 4.90624, 4.77517, 4.94606, 5.11508, 5.01539, 5.01397, 5.3327, 4.96029, 4.9915, 5.04439, 4.80654, 4.73199, 4.99639, 5.04237, 4.8734, 4.95425, 5.04678, 5.02392, 4.81994, 4.89463, 4.90711, 4.83288, 4.74257, 5.01934, 4.75352, 5.20696, 4.79359, 4.99212, 4.73894, 4.7885, 4.82299, 4.65617, 4.65522, 4.84524, 4.81217, 4.79792, 4.92038, 4.88607, 4.92565, 4.7712, 4.88216, 4.73528, 4.92078, 4.96145, 4.87447, 4.71317, 4.78702, 4.90462, 4.71624, 4.86657, 4.69712, 4.69196, 4.64876]}, "loss-scale": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]}, "loss-scale vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]}, "grad-norm": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [13.92196, 13.29306, 13.8377, 12.64037, 11.97375, 9.45262, 6.78823, 6.89004, 5.94557, 4.54615, 4.13637, 2.82375, 2.38927, 2.34389, 2.05973, 2.22596, 2.14457, 1.88597, 2.17986, 2.06069, 2.12423, 2.1677, 2.0115, 2.21442, 1.98307, 2.0966, 1.90389, 1.86829, 1.92477, 2.13027, 2.09469, 2.11211, 1.95723, 2.18758, 2.38519, 2.04808, 2.04244, 1.85027, 1.9837, 1.78603, 2.12943, 1.83753, 1.73653, 1.84787, 1.96175, 1.78052, 1.76095, 1.7401, 1.76961, 1.54057, 1.76088, 1.7938, 1.76365, 1.83855, 1.58517, 1.79545, 1.7158, 1.81815, 1.53518, 1.48648, 1.68949, 1.4562, 1.8648, 1.85145, 1.61928, 1.6745, 1.65487, 1.55646, 1.47797, 1.6989, 1.43883, 1.43836, 1.46011, 1.39711, 1.37457, 1.48663, 1.40785, 1.35385, 1.34051, 1.27757, 1.35283, 1.29709, 1.2816, 1.30185, 1.24092, 1.29738, 1.41961, 1.34489, 1.44199, 1.06928, 1.09491, 1.16108, 1.14396, 1.33634, 1.03654, 1.30756, 1.08982, 1.27845, 0.98191, 1.37412, 1.30793, 1.21672, 1.05131, 1.25909, 1.09643, 1.13996, 1.20961, 1.09191, 1.24074, 0.97878, 1.18535, 0.97714, 0.95456, 1.10186, 1.24389, 1.07847, 1.01822, 1.2519, 1.18392, 1.42087, 1.00253, 1.23223, 1.05494, 1.02956, 0.95692, 1.27887, 1.54081, 1.2168, 1.18019, 1.34805, 0.93443, 1.06987, 1.00938, 1.19729, 1.32572, 1.18029, 1.39724, 1.01719, 1.76109, 1.21222, 1.26256, 1.31969, 1.1555, 0.93801, 0.99546, 1.01521, 1.36553, 1.55577, 1.11391, 1.2491, 1.45721, 1.65042, 1.60593, 1.30243, 1.29342, 2.04924, 1.3376, 1.21234, 1.37945, 1.79037, 1.23389, 1.08215, 1.31811, 1.12901, 1.35786, 1.8341, 1.46143, 1.31586, 1.39491, 1.24546, 1.26969, 1.25412, 1.27022, 1.43967, 1.14847, 1.3362, 1.91114, 1.35642, 1.06973, 1.20518, 1.11732, 1.73877, 1.36915, 1.34679, 1.25766, 1.64809, 1.37397, 1.17279, 1.169, 1.49772, 1.11509, 1.29145, 1.479, 1.60514, 1.12787, 1.20465, 1.52478, 1.37769, 1.40825, 1.40433, 1.19434, 1.52129, 1.49087, 1.60752, 1.51416, 1.37753, 1.49097, 1.59106, 1.33146, 1.56964, 1.54958, 1.2024, 1.29844, 1.28184, 1.63096, 1.29563, 1.41842, 1.57651, 1.29669, 1.23902, 1.51872, 1.34276, 1.28172, 1.67239, 1.39643, 1.57361, 1.69097, 1.37206, 1.81716, 1.3501, 1.2879, 1.45938, 1.9477, 1.77504, 2.56828, 1.55284, 1.34454, 1.21685, 1.65336, 1.29693, 2.2136, 1.28644, 1.78502, 1.52285, 1.47963, 1.65183, 1.23421, 1.41797, 1.5183, 1.31219, 1.29375, 1.3932, 1.5544, 1.2678, 1.61107, 1.43809, 1.9371, 1.64335, 1.38939, 1.24473, 1.15131, 1.26598, 1.37433, 1.20588, 1.22283, 1.31678, 1.40086, 1.53213, 1.35367, 1.43407, 1.41639, 1.25063, 1.37444, 1.20928, 1.40445, 1.48011, 1.49606, 1.43456, 1.4511, 1.51505, 1.49329, 1.32736, 1.34283, 1.56947, 1.3986, 1.38533, 1.4325, 1.36846, 1.40113, 1.40195, 1.41944, 1.73207, 1.35246, 1.98477, 1.75001, 1.59412, 1.33312, 1.55175, 1.45641, 1.40103, 1.32697, 1.19674, 1.19056, 1.56111, 1.64, 1.52329, 1.62982, 1.42489, 1.1143, 1.42326, 1.36052, 1.20749, 1.49372, 1.38211, 1.6856, 1.48198, 1.34985, 1.48241, 1.24509, 1.40355, 1.44024, 1.31152, 1.30253, 1.59307, 1.35212, 1.78683, 1.61562, 1.61575, 1.46207, 1.29047, 1.55842, 1.39097, 1.35377, 1.50655, 1.67836, 1.37929, 1.32311, 1.35305, 1.77455, 1.48895, 1.40827, 1.23883, 1.35995, 1.46576, 1.39021, 1.55027, 1.27874, 1.53316, 1.30645, 1.32818, 1.41856, 1.40297, 1.19176, 1.73797, 1.28462, 1.46556, 1.31822, 1.27157, 1.29905, 1.43641, 1.37732, 1.32041, 1.45048, 1.30403, 1.12439, 1.41266, 1.49642, 1.41634, 1.48283, 1.73467, 1.90209, 1.41005, 1.66166, 1.51488, 1.35734, 1.47652, 1.40564, 1.6499, 1.41346, 1.24965, 1.34929, 1.35141, 1.18107, 1.30851, 1.17223, 1.29341, 1.38306, 1.247, 1.29013, 1.70946, 1.36584, 1.4061, 1.82813, 1.27073, 1.45088, 1.55944, 1.5925, 1.64727, 1.42815, 1.19955]}, "grad-norm vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [13.92196, 13.29306, 13.8377, 12.64037, 11.97375, 9.45262, 6.78823, 6.89004, 5.94557, 4.54615, 4.13637, 2.82375, 2.38927, 2.34389, 2.05973, 2.22596, 2.14457, 1.88597, 2.17986, 2.06069, 2.12423, 2.1677, 2.0115, 2.21442, 1.98307, 2.0966, 1.90389, 1.86829, 1.92477, 2.13027, 2.09469, 2.11211, 1.95723, 2.18758, 2.38519, 2.04808, 2.04244, 1.85027, 1.9837, 1.78603, 2.12943, 1.83753, 1.73653, 1.84787, 1.96175, 1.78052, 1.76095, 1.7401, 1.76961, 1.54057, 1.76088, 1.7938, 1.76365, 1.83855, 1.58517, 1.79545, 1.7158, 1.81815, 1.53518, 1.48648, 1.68949, 1.4562, 1.8648, 1.85145, 1.61928, 1.6745, 1.65487, 1.55646, 1.47797, 1.6989, 1.43883, 1.43836, 1.46011, 1.39711, 1.37457, 1.48663, 1.40785, 1.35385, 1.34051, 1.27757, 1.35283, 1.29709, 1.2816, 1.30185, 1.24092, 1.29738, 1.41961, 1.34489, 1.44199, 1.06928, 1.09491, 1.16108, 1.14396, 1.33634, 1.03654, 1.30756, 1.08982, 1.27845, 0.98191, 1.37412, 1.30793, 1.21672, 1.05131, 1.25909, 1.09643, 1.13996, 1.20961, 1.09191, 1.24074, 0.97878, 1.18535, 0.97714, 0.95456, 1.10186, 1.24389, 1.07847, 1.01822, 1.2519, 1.18392, 1.42087, 1.00253, 1.23223, 1.05494, 1.02956, 0.95692, 1.27887, 1.54081, 1.2168, 1.18019, 1.34805, 0.93443, 1.06987, 1.00938, 1.19729, 1.32572, 1.18029, 1.39724, 1.01719, 1.76109, 1.21222, 1.26256, 1.31969, 1.1555, 0.93801, 0.99546, 1.01521, 1.36553, 1.55577, 1.11391, 1.2491, 1.45721, 1.65042, 1.60593, 1.30243, 1.29342, 2.04924, 1.3376, 1.21234, 1.37945, 1.79037, 1.23389, 1.08215, 1.31811, 1.12901, 1.35786, 1.8341, 1.46143, 1.31586, 1.39491, 1.24546, 1.26969, 1.25412, 1.27022, 1.43967, 1.14847, 1.3362, 1.91114, 1.35642, 1.06973, 1.20518, 1.11732, 1.73877, 1.36915, 1.34679, 1.25766, 1.64809, 1.37397, 1.17279, 1.169, 1.49772, 1.11509, 1.29145, 1.479, 1.60514, 1.12787, 1.20465, 1.52478, 1.37769, 1.40825, 1.40433, 1.19434, 1.52129, 1.49087, 1.60752, 1.51416, 1.37753, 1.49097, 1.59106, 1.33146, 1.56964, 1.54958, 1.2024, 1.29844, 1.28184, 1.63096, 1.29563, 1.41842, 1.57651, 1.29669, 1.23902, 1.51872, 1.34276, 1.28172, 1.67239, 1.39643, 1.57361, 1.69097, 1.37206, 1.81716, 1.3501, 1.2879, 1.45938, 1.9477, 1.77504, 2.56828, 1.55284, 1.34454, 1.21685, 1.65336, 1.29693, 2.2136, 1.28644, 1.78502, 1.52285, 1.47963, 1.65183, 1.23421, 1.41797, 1.5183, 1.31219, 1.29375, 1.3932, 1.5544, 1.2678, 1.61107, 1.43809, 1.9371, 1.64335, 1.38939, 1.24473, 1.15131, 1.26598, 1.37433, 1.20588, 1.22283, 1.31678, 1.40086, 1.53213, 1.35367, 1.43407, 1.41639, 1.25063, 1.37444, 1.20928, 1.40445, 1.48011, 1.49606, 1.43456, 1.4511, 1.51505, 1.49329, 1.32736, 1.34283, 1.56947, 1.3986, 1.38533, 1.4325, 1.36846, 1.40113, 1.40195, 1.41944, 1.73207, 1.35246, 1.98477, 1.75001, 1.59412, 1.33312, 1.55175, 1.45641, 1.40103, 1.32697, 1.19674, 1.19056, 1.56111, 1.64, 1.52329, 1.62982, 1.42489, 1.1143, 1.42326, 1.36052, 1.20749, 1.49372, 1.38211, 1.6856, 1.48198, 1.34985, 1.48241, 1.24509, 1.40355, 1.44024, 1.31152, 1.30253, 1.59307, 1.35212, 1.78683, 1.61562, 1.61575, 1.46207, 1.29047, 1.55842, 1.39097, 1.35377, 1.50655, 1.67836, 1.37929, 1.32311, 1.35305, 1.77455, 1.48895, 1.40827, 1.23883, 1.35995, 1.46576, 1.39021, 1.55027, 1.27874, 1.53316, 1.30645, 1.32818, 1.41856, 1.40297, 1.19176, 1.73797, 1.28462, 1.46556, 1.31822, 1.27157, 1.29905, 1.43641, 1.37732, 1.32041, 1.45048, 1.30403, 1.12439, 1.41266, 1.49642, 1.41634, 1.48283, 1.73467, 1.90209, 1.41005, 1.66166, 1.51488, 1.35734, 1.47652, 1.40564, 1.6499, 1.41346, 1.24965, 1.34929, 1.35141, 1.18107, 1.30851, 1.17223, 1.29341, 1.38306, 1.247, 1.29013, 1.70946, 1.36584, 1.4061, 1.82813, 1.27073, 1.45088, 1.55944, 1.5925, 1.64727, 1.42815, 1.19955]}, "num-zeros": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [74.0, 72.0, 73.0, 74.0, 73.0, 90.0, 126.0, 114.0, 113.0, 140.0, 116.0, 153.0, 141.0, 172.0, 170.0, 168.0, 175.0, 182.0, 140.0, 176.0, 137.0, 166.0, 172.0, 196.0, 193.0, 159.0, 182.0, 170.0, 180.0, 179.0, 141.0, 166.0, 148.0, 198.0, 144.0, 177.0, 155.0, 219.0, 170.0, 192.0, 162.0, 168.0, 146.0, 172.0, 183.0, 182.0, 165.0, 172.0, 179.0, 209.0, 199.0, 157.0, 189.0, 149.0, 190.0, 189.0, 146.0, 172.0, 220.0, 227.0, 191.0, 197.0, 178.0, 159.0, 180.0, 222.0, 178.0, 168.0, 208.0, 190.0, 237.0, 231.0, 183.0, 220.0, 201.0, 186.0, 220.0, 207.0, 221.0, 220.0, 231.0, 238.0, 207.0, 247.0, 221.0, 200.0, 178.0, 203.0, 198.0, 192.0, 200.0, 178.0, 214.0, 214.0, 255.0, 154.0, 214.0, 180.0, 179.0, 196.0, 182.0, 176.0, 151.0, 176.0, 164.0, 147.0, 165.0, 147.0, 127.0, 163.0, 192.0, 165.0, 146.0, 151.0, 131.0, 165.0, 166.0, 110.0, 158.0, 148.0, 129.0, 137.0, 142.0, 143.0, 162.0, 144.0, 125.0, 159.0, 141.0, 123.0, 161.0, 126.0, 116.0, 116.0, 131.0, 88.0, 135.0, 126.0, 119.0, 156.0, 112.0, 129.0, 126.0, 142.0, 130.0, 141.0, 134.0, 134.0, 133.0, 101.0, 78.0, 104.0, 100.0, 130.0, 115.0, 82.0, 108.0, 97.0, 80.0, 99.0, 134.0, 98.0, 85.0, 116.0, 84.0, 97.0, 107.0, 114.0, 119.0, 111.0, 105.0, 109.0, 88.0, 96.0, 119.0, 133.0, 101.0, 108.0, 135.0, 135.0, 111.0, 146.0, 131.0, 113.0, 107.0, 132.0, 109.0, 110.0, 96.0, 93.0, 137.0, 103.0, 118.0, 111.0, 112.0, 120.0, 92.0, 111.0, 111.0, 93.0, 86.0, 105.0, 114.0, 114.0, 105.0, 119.0, 114.0, 111.0, 98.0, 123.0, 123.0, 100.0, 120.0, 124.0, 73.0, 91.0, 106.0, 110.0, 80.0, 93.0, 105.0, 111.0, 101.0, 113.0, 94.0, 116.0, 90.0, 120.0, 75.0, 106.0, 95.0, 82.0, 98.0, 117.0, 100.0, 101.0, 107.0, 103.0, 98.0, 111.0, 102.0, 90.0, 108.0, 106.0, 117.0, 98.0, 89.0, 113.0, 116.0, 91.0, 124.0, 108.0, 106.0, 108.0, 102.0, 109.0, 112.0, 113.0, 97.0, 107.0, 98.0, 104.0, 135.0, 105.0, 108.0, 115.0, 116.0, 79.0, 102.0, 112.0, 132.0, 107.0, 103.0, 102.0, 107.0, 90.0, 101.0, 116.0, 106.0, 120.0, 120.0, 109.0, 116.0, 97.0, 111.0, 106.0, 104.0, 122.0, 86.0, 95.0, 129.0, 88.0, 129.0, 126.0, 96.0, 104.0, 115.0, 91.0, 100.0, 104.0, 115.0, 111.0, 101.0, 117.0, 89.0, 97.0, 107.0, 95.0, 113.0, 92.0, 106.0, 120.0, 111.0, 109.0, 112.0, 128.0, 110.0, 111.0, 125.0, 132.0, 106.0, 103.0, 111.0, 109.0, 115.0, 117.0, 110.0, 110.0, 85.0, 104.0, 119.0, 101.0, 104.0, 111.0, 106.0, 107.0, 104.0, 124.0, 101.0, 119.0, 134.0, 120.0, 134.0, 116.0, 122.0, 98.0, 95.0, 101.0, 116.0, 127.0, 107.0, 105.0, 117.0, 92.0, 131.0, 110.0, 135.0, 121.0, 117.0, 124.0, 90.0, 113.0, 109.0, 103.0, 143.0, 98.0, 94.0, 93.0, 101.0, 104.0, 113.0, 111.0, 90.0, 103.0, 94.0, 102.0, 99.0, 109.0, 124.0, 123.0, 124.0, 118.0, 116.0, 112.0, 121.0, 127.0, 130.0, 101.0, 111.0, 124.0, 106.0, 131.0, 122.0, 126.0, 124.0, 110.0, 108.0, 81.0, 97.0, 132.0]}, "num-zeros vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [65.0, 80.0, 81.0, 75.0, 72.0, 103.0, 108.0, 112.0, 107.0, 122.0, 99.0, 159.0, 148.0, 150.0, 167.0, 157.0, 165.0, 144.0, 182.0, 187.0, 180.0, 162.0, 181.0, 129.0, 189.0, 148.0, 195.0, 190.0, 137.0, 181.0, 151.0, 155.0, 152.0, 166.0, 152.0, 170.0, 160.0, 209.0, 168.0, 214.0, 166.0, 181.0, 190.0, 185.0, 161.0, 162.0, 169.0, 187.0, 184.0, 239.0, 225.0, 187.0, 190.0, 131.0, 187.0, 182.0, 159.0, 161.0, 248.0, 226.0, 201.0, 211.0, 174.0, 164.0, 168.0, 225.0, 202.0, 174.0, 223.0, 202.0, 243.0, 235.0, 180.0, 239.0, 219.0, 205.0, 210.0, 192.0, 216.0, 207.0, 209.0, 245.0, 217.0, 227.0, 212.0, 207.0, 191.0, 173.0, 196.0, 193.0, 194.0, 186.0, 203.0, 189.0, 210.0, 160.0, 204.0, 187.0, 189.0, 159.0, 168.0, 209.0, 181.0, 159.0, 173.0, 153.0, 175.0, 152.0, 147.0, 174.0, 180.0, 153.0, 176.0, 146.0, 165.0, 154.0, 147.0, 106.0, 147.0, 133.0, 174.0, 148.0, 152.0, 143.0, 173.0, 127.0, 116.0, 130.0, 127.0, 123.0, 143.0, 142.0, 146.0, 123.0, 131.0, 124.0, 138.0, 139.0, 109.0, 107.0, 130.0, 103.0, 121.0, 157.0, 131.0, 148.0, 139.0, 96.0, 120.0, 101.0, 96.0, 102.0, 102.0, 122.0, 105.0, 84.0, 114.0, 117.0, 95.0, 90.0, 106.0, 137.0, 136.0, 131.0, 122.0, 95.0, 111.0, 99.0, 117.0, 119.0, 129.0, 111.0, 104.0, 112.0, 108.0, 102.0, 88.0, 97.0, 120.0, 121.0, 124.0, 96.0, 126.0, 134.0, 122.0, 98.0, 97.0, 115.0, 102.0, 102.0, 128.0, 120.0, 104.0, 104.0, 97.0, 112.0, 104.0, 96.0, 117.0, 97.0, 136.0, 100.0, 92.0, 104.0, 95.0, 111.0, 97.0, 87.0, 108.0, 128.0, 94.0, 111.0, 106.0, 122.0, 99.0, 94.0, 110.0, 104.0, 116.0, 119.0, 114.0, 112.0, 104.0, 104.0, 108.0, 88.0, 105.0, 114.0, 103.0, 105.0, 96.0, 98.0, 92.0, 92.0, 91.0, 102.0, 119.0, 106.0, 86.0, 104.0, 60.0, 110.0, 92.0, 91.0, 80.0, 91.0, 114.0, 106.0, 80.0, 119.0, 117.0, 112.0, 114.0, 98.0, 102.0, 109.0, 101.0, 100.0, 102.0, 126.0, 124.0, 99.0, 112.0, 110.0, 129.0, 111.0, 99.0, 119.0, 101.0, 82.0, 110.0, 84.0, 95.0, 104.0, 96.0, 107.0, 83.0, 114.0, 105.0, 93.0, 104.0, 108.0, 94.0, 99.0, 104.0, 101.0, 88.0, 112.0, 101.0, 101.0, 108.0, 119.0, 118.0, 103.0, 100.0, 107.0, 94.0, 104.0, 118.0, 111.0, 115.0, 100.0, 114.0, 90.0, 110.0, 107.0, 90.0, 91.0, 145.0, 113.0, 112.0, 120.0, 101.0, 98.0, 97.0, 96.0, 109.0, 100.0, 115.0, 120.0, 120.0, 121.0, 128.0, 103.0, 94.0, 104.0, 110.0, 89.0, 102.0, 106.0, 113.0, 117.0, 113.0, 115.0, 93.0, 114.0, 119.0, 132.0, 82.0, 112.0, 105.0, 96.0, 124.0, 107.0, 108.0, 104.0, 145.0, 119.0, 124.0, 115.0, 116.0, 94.0, 130.0, 98.0, 115.0, 117.0, 120.0, 122.0, 122.0, 110.0, 108.0, 87.0, 117.0, 102.0, 123.0, 108.0, 123.0, 107.0, 99.0, 127.0, 94.0, 107.0, 72.0, 102.0, 86.0, 91.0, 94.0, 116.0, 106.0, 120.0, 127.0, 115.0, 124.0, 126.0, 129.0, 117.0, 112.0, 120.0, 119.0, 126.0, 111.0, 119.0, 91.0, 102.0, 95.0, 118.0, 111.0, 99.0, 122.0, 125.0]}, "params-norm": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [180.0215, 180.0215, 180.0215, 180.0215, 180.02148, 180.02147, 180.02145, 180.02135, 180.02115, 180.02094, 180.01958, 180.01897, 180.01846, 180.01624, 180.01547, 180.01477, 180.01425, 180.01407, 180.0143, 180.01436, 180.01428, 180.01392, 180.01378, 180.01396, 180.01488, 180.0161, 180.0174, 180.01826, 180.0192, 180.0206, 180.02252, 180.02473, 180.0269, 180.02902, 180.03149, 180.0345, 180.03801, 180.04181, 180.04555, 180.04977, 180.05466, 180.05992, 180.06546, 180.0715, 180.07837, 180.08574, 180.09346, 180.10184, 180.11102, 180.12125, 180.13229, 180.14392, 180.15607, 180.16887, 180.1824, 180.19664, 180.21123, 180.22636, 180.24251, 180.25967, 180.27742, 180.29587, 180.31598, 180.33707, 180.3582, 180.3808, 180.40411, 180.42862, 180.45422, 180.48024, 180.50642, 180.53325, 180.56082, 180.58878, 180.61742, 180.64685, 180.67635, 180.70671, 180.73753, 180.76909, 180.80096, 180.83255, 180.86522, 180.89883, 180.93253, 180.96713, 181.00252, 181.03773, 181.07297, 181.10829, 181.14496, 181.18279, 181.22028, 181.25752, 181.29439, 181.32959, 181.36458, 181.40088, 181.43741, 181.47369, 181.50917, 181.54332, 181.57774, 181.61334, 181.64902, 181.68596, 181.7242, 181.7617, 181.79843, 181.83513, 181.87192, 181.90961, 181.94727, 181.9857, 182.02441, 182.06326, 182.1035, 182.14424, 182.18398, 182.22302, 182.26132, 182.30066, 182.33942, 182.37904, 182.41917, 182.45876, 182.49632, 182.53271, 182.56963, 182.60735, 182.64554, 182.68359, 182.72183, 182.75928, 182.79482, 182.83173, 182.86961, 182.90521, 182.94044, 182.97412, 183.00899, 183.04352, 183.0809, 183.12045, 183.16031, 183.20035, 183.24016, 183.27913, 183.31721, 183.35562, 183.39336, 183.42928, 183.46495, 183.50055, 183.53683, 183.57225, 183.60655, 183.64061, 183.67566, 183.71036, 183.74536, 183.78122, 183.81776, 183.85562, 183.89389, 183.93182, 183.96855, 184.00623, 184.04614, 184.08539, 184.12434, 184.16336, 184.20358, 184.2431, 184.28152, 184.32024, 184.3553, 184.3905, 184.42917, 184.4704, 184.51273, 184.55392, 184.59485, 184.63615, 184.67656, 184.71397, 184.74928, 184.78352, 184.82126, 184.86098, 184.90076, 184.94235, 184.98337, 185.02277, 185.0623, 185.10294, 185.14499, 185.18594, 185.22719, 185.26956, 185.31255, 185.35408, 185.39359, 185.43069, 185.46863, 185.50841, 185.54842, 185.5876, 185.62738, 185.66747, 185.7076, 185.74796, 185.78799, 185.82808, 185.86952, 185.91144, 185.95245, 185.99278, 186.03255, 186.07283, 186.11411, 186.15575, 186.19742, 186.2375, 186.27637, 186.31621, 186.35637, 186.39667, 186.43544, 186.4731, 186.51167, 186.55107, 186.5916, 186.63014, 186.66568, 186.69972, 186.73563, 186.77632, 186.81931, 186.86119, 186.89891, 186.93753, 186.97639, 187.01602, 187.0556, 187.0981, 187.14053, 187.1834, 187.22716, 187.27185, 187.31763, 187.36372, 187.4113, 187.45898, 187.506, 187.55214, 187.59671, 187.64069, 187.68445, 187.73042, 187.77773, 187.82211, 187.86797, 187.91481, 187.96231, 188.00858, 188.05304, 188.09511, 188.13795, 188.1804, 188.22424, 188.27013, 188.31894, 188.36742, 188.41576, 188.4644, 188.51416, 188.56253, 188.60983, 188.65424, 188.69913, 188.7431, 188.78632, 188.83072, 188.87659, 188.92245, 188.96892, 189.01532, 189.06158, 189.10831, 189.15527, 189.20079, 189.2475, 189.29361, 189.33777, 189.38203, 189.42827, 189.47591, 189.52328, 189.57204, 189.62096, 189.6709, 189.72188, 189.77139, 189.81842, 189.8649, 189.91235, 189.95949, 190.0078, 190.05704, 190.10622, 190.15698, 190.20724, 190.25786, 190.30705, 190.35727, 190.40851, 190.45973, 190.51111, 190.56392, 190.61598, 190.66782, 190.7196, 190.77359, 190.82573, 190.87747, 190.92769, 190.97775, 191.02827, 191.07834, 191.12999, 191.17932, 191.22862, 191.27965, 191.33025, 191.38222, 191.433, 191.48625, 191.53882, 191.59085, 191.64409, 191.698, 191.7515, 191.8065, 191.86282, 191.91794, 191.97198, 192.02602, 192.07971, 192.1337, 192.18675, 192.24236, 192.29745, 192.35396, 192.40863, 192.46198, 192.51579, 192.57161, 192.62778, 192.68323, 192.73868, 192.79523, 192.85144, 192.9077, 192.96512, 193.02281, 193.07899, 193.13582, 193.19206, 193.24911, 193.30396, 193.35805, 193.41168, 193.46552, 193.52077, 193.57597, 193.63229, 193.68961, 193.74706, 193.80554, 193.86365, 193.92087, 193.97789, 194.03809, 194.09793, 194.15579, 194.21254, 194.27122, 194.33063, 194.39035, 194.44989, 194.51079, 194.56964, 194.62762, 194.68622, 194.74329, 194.79973, 194.85442, 194.91043, 194.96838]}, "params-norm vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [180.0215, 180.0215, 180.0215, 180.0215, 180.02148, 180.02147, 180.02145, 180.02135, 180.02115, 180.02094, 180.01958, 180.01897, 180.01846, 180.01624, 180.01547, 180.01477, 180.01425, 180.01407, 180.0143, 180.01436, 180.01428, 180.01392, 180.01378, 180.01396, 180.01488, 180.0161, 180.0174, 180.01826, 180.0192, 180.0206, 180.02252, 180.02473, 180.0269, 180.02902, 180.03149, 180.0345, 180.03801, 180.04181, 180.04555, 180.04977, 180.05466, 180.05992, 180.06546, 180.0715, 180.07837, 180.08574, 180.09346, 180.10184, 180.11102, 180.12125, 180.13229, 180.14392, 180.15607, 180.16887, 180.1824, 180.19664, 180.21123, 180.22636, 180.24251, 180.25967, 180.27742, 180.29587, 180.31598, 180.33707, 180.3582, 180.3808, 180.40411, 180.42862, 180.45422, 180.48024, 180.50642, 180.53325, 180.56082, 180.58878, 180.61742, 180.64685, 180.67635, 180.70671, 180.73753, 180.76909, 180.80096, 180.83255, 180.86522, 180.89883, 180.93253, 180.96713, 181.00252, 181.03773, 181.07297, 181.10829, 181.14496, 181.18279, 181.22028, 181.25752, 181.29439, 181.32959, 181.36458, 181.40088, 181.43741, 181.47369, 181.50917, 181.54332, 181.57774, 181.61334, 181.64902, 181.68596, 181.7242, 181.7617, 181.79843, 181.83513, 181.87192, 181.90961, 181.94727, 181.9857, 182.02441, 182.06326, 182.1035, 182.14424, 182.18398, 182.22302, 182.26132, 182.30066, 182.33942, 182.37904, 182.41917, 182.45876, 182.49632, 182.53271, 182.56963, 182.60735, 182.64554, 182.68359, 182.72183, 182.75928, 182.79482, 182.83173, 182.86961, 182.90521, 182.94044, 182.97412, 183.00899, 183.04352, 183.0809, 183.12045, 183.16031, 183.20035, 183.24016, 183.27913, 183.31721, 183.35562, 183.39336, 183.42928, 183.46495, 183.50055, 183.53683, 183.57225, 183.60655, 183.64061, 183.67566, 183.71036, 183.74536, 183.78122, 183.81776, 183.85562, 183.89389, 183.93182, 183.96855, 184.00623, 184.04614, 184.08539, 184.12434, 184.16336, 184.20358, 184.2431, 184.28152, 184.32024, 184.3553, 184.3905, 184.42917, 184.4704, 184.51273, 184.55392, 184.59485, 184.63615, 184.67656, 184.71397, 184.74928, 184.78352, 184.82126, 184.86098, 184.90076, 184.94235, 184.98337, 185.02277, 185.0623, 185.10294, 185.14499, 185.18594, 185.22719, 185.26956, 185.31255, 185.35408, 185.39359, 185.43069, 185.46863, 185.50841, 185.54842, 185.5876, 185.62738, 185.66747, 185.7076, 185.74796, 185.78799, 185.82808, 185.86952, 185.91144, 185.95245, 185.99278, 186.03255, 186.07283, 186.11411, 186.15575, 186.19742, 186.2375, 186.27637, 186.31621, 186.35637, 186.39667, 186.43544, 186.4731, 186.51167, 186.55107, 186.5916, 186.63014, 186.66568, 186.69972, 186.73563, 186.77632, 186.81931, 186.86119, 186.89891, 186.93753, 186.97639, 187.01602, 187.0556, 187.0981, 187.14053, 187.1834, 187.22716, 187.27185, 187.31763, 187.36372, 187.4113, 187.45898, 187.506, 187.55214, 187.59671, 187.64069, 187.68445, 187.73042, 187.77773, 187.82211, 187.86797, 187.91481, 187.96231, 188.00858, 188.05304, 188.09511, 188.13795, 188.1804, 188.22424, 188.27013, 188.31894, 188.36742, 188.41576, 188.4644, 188.51416, 188.56253, 188.60983, 188.65424, 188.69913, 188.7431, 188.78632, 188.83072, 188.87659, 188.92245, 188.96892, 189.01532, 189.06158, 189.10831, 189.15527, 189.20079, 189.2475, 189.29361, 189.33777, 189.38203, 189.42827, 189.47591, 189.52328, 189.57204, 189.62096, 189.6709, 189.72188, 189.77139, 189.81842, 189.8649, 189.91235, 189.95949, 190.0078, 190.05704, 190.10622, 190.15698, 190.20724, 190.25786, 190.30705, 190.35727, 190.40851, 190.45973, 190.51111, 190.56392, 190.61598, 190.66782, 190.7196, 190.77359, 190.82573, 190.87747, 190.92769, 190.97775, 191.02827, 191.07834, 191.12999, 191.17932, 191.22862, 191.27965, 191.33025, 191.38222, 191.433, 191.48625, 191.53882, 191.59085, 191.64409, 191.698, 191.7515, 191.8065, 191.86282, 191.91794, 191.97198, 192.02602, 192.07971, 192.1337, 192.18675, 192.24236, 192.29745, 192.35396, 192.40863, 192.46198, 192.51579, 192.57161, 192.62778, 192.68323, 192.73868, 192.79523, 192.85144, 192.9077, 192.96512, 193.02281, 193.07899, 193.13582, 193.19206, 193.24911, 193.30396, 193.35805, 193.41168, 193.46552, 193.52077, 193.57597, 193.63229, 193.68961, 193.74706, 193.80554, 193.86365, 193.92087, 193.97789, 194.03809, 194.09793, 194.15579, 194.21254, 194.27122, 194.33063, 194.39035, 194.44989, 194.51079, 194.56964, 194.62762, 194.68622, 194.74329, 194.79973, 194.85442, 194.91043, 194.96838]}, "iteration-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [25.9357, 1.58651, 1.57374, 1.5753, 1.57369, 1.58365, 1.58825, 1.58527, 1.58564, 1.5777, 1.58419, 1.58585, 1.58154, 1.58741, 1.59392, 1.59071, 1.59711, 1.6014, 1.60351, 1.59396, 1.5899, 1.59645, 1.58704, 1.58712, 1.60341, 1.58462, 1.5838, 1.58964, 1.5977, 1.5914, 1.59087, 1.59805, 1.5927, 1.59042, 1.57661, 1.58906, 1.58372, 1.5783, 1.662, 1.58247, 1.58561, 1.58497, 1.60619, 1.59828, 1.60708, 1.60788, 1.6018, 1.59949, 1.59104, 1.5968, 1.60548, 1.60125, 1.59943, 1.58135, 1.58089, 1.58389, 1.58725, 1.58116, 1.58404, 1.58902, 1.58673, 1.58415, 1.60076, 1.59392, 1.59498, 1.58949, 1.59688, 1.59686, 1.58746, 1.59881, 1.5919, 1.59305, 1.60935, 1.59895, 1.60324, 1.60238, 1.59829, 1.60008, 1.59605, 1.60176, 1.59396, 1.60186, 1.58731, 1.58171, 1.58397, 1.58802, 1.58792, 1.5888, 1.5989, 1.60961, 1.59174, 1.61116, 1.59839, 1.5987, 1.60266, 1.59894, 1.60234, 1.59759, 1.59588, 1.59656, 1.60095, 1.59247, 1.59334, 1.58581, 1.60076, 1.5966, 1.58958, 1.58303, 1.58777, 1.58897, 1.59327, 1.59617, 1.59379, 1.59354, 1.58468, 1.59116, 1.58522, 1.58052, 1.57531, 1.59285, 1.58327, 1.57928, 1.58856, 1.60734, 1.60047, 1.58954, 1.5887, 1.59365, 1.57967, 1.58675, 1.57718, 1.58018, 1.58698, 1.58486, 1.59903, 1.5922, 1.59084, 1.58453, 1.58231, 1.58267, 1.58483, 1.58037, 1.5909, 1.60252, 1.60356, 1.58876, 1.59367, 1.60171, 1.59771, 1.6032, 1.60106, 1.60184, 1.60827, 1.60637, 1.60548, 1.60525, 1.60212, 1.60506, 1.59982, 1.60509, 1.60647, 1.60886, 1.60014, 1.60931, 1.59824, 1.60157, 1.60774, 1.60732, 1.61218, 1.61074, 1.60769, 1.60031, 1.59568, 1.59819, 1.6096, 1.59367, 1.60494, 1.59917, 1.59747, 1.60124, 1.59771, 1.59534, 1.60201, 1.59851, 1.60069, 1.60225, 1.59775, 1.59041, 1.60108, 1.59759, 1.59096, 1.60191, 1.5962, 1.60086, 1.61379, 1.60436, 1.60606, 1.60163, 1.60378, 1.60305, 1.59492, 1.60456, 1.60034, 1.58872, 1.59577, 1.59654, 1.59711, 1.59749, 1.59808, 1.60144, 1.59512, 1.59382, 1.59822, 1.59585, 1.59994, 1.59286, 1.59958, 1.60154, 1.59764, 1.59284, 1.59867, 1.6049, 1.6004, 1.59909, 1.60488, 1.59532, 1.60133, 1.60538, 1.5991, 1.59608, 1.60992, 1.60101, 1.60144, 1.59775, 1.59962, 1.58809, 1.59851, 1.59204, 1.59492, 1.59647, 1.58928, 1.58595, 1.7535, 1.6478, 1.59827, 1.60514, 1.59426, 1.61414, 1.60982, 1.60735, 1.60866, 1.70147, 1.60416, 1.59248, 1.59525, 1.59344, 1.59499, 1.60459, 1.6003, 1.60341, 1.60801, 1.61343, 1.60596, 1.60611, 1.60542, 1.60121, 1.59801, 1.59823, 1.59998, 1.59829, 1.59898, 1.59531, 1.60142, 1.60403, 1.59966, 1.60202, 1.59979, 1.60042, 1.59732, 1.60245, 1.60091, 1.5998, 1.60238, 1.59984, 1.60274, 1.60666, 1.60321, 1.6036, 1.6041, 1.59868, 1.6015, 1.60892, 1.60377, 1.60116, 1.60829, 1.60355, 1.60349, 1.60256, 1.60399, 1.60265, 1.60684, 1.60536, 1.61211, 1.60719, 1.6104, 1.59911, 1.59879, 1.61165, 1.60015, 1.6048, 1.59789, 1.60116, 1.60929, 1.60128, 1.60444, 1.6133, 1.59942, 1.6132, 1.60448, 1.58597, 1.58802, 1.59401, 1.58972, 1.59965, 1.60201, 1.59413, 1.60397, 1.60165, 1.59963, 1.60178, 1.59826, 1.60301, 1.6063, 1.60499, 1.6023, 1.60467, 1.6048, 1.59497, 1.61355, 1.60237, 1.60516, 1.60289, 1.60404, 1.60076, 1.59623, 1.60269, 1.60248, 1.60802, 1.60059, 1.70142, 1.61751, 1.60679, 1.7026, 1.60996, 1.6083, 1.61064, 1.61183, 1.62052, 1.61909, 1.61534, 1.61668, 1.6033, 1.60768, 1.60386, 1.61143, 1.60918, 1.59776, 1.60709, 1.60535, 1.60161, 1.60666, 1.60582, 1.60545, 1.6075, 1.60733, 1.61657, 1.62133, 1.60999, 1.61188, 1.61305, 1.6069, 1.61671, 1.61762, 1.62212, 1.61922, 1.6081, 1.60551, 1.61555, 1.61354, 1.61632, 1.61937, 1.6141, 1.60911, 1.614, 1.61245, 1.61194, 1.6115, 1.60534, 1.60841, 1.60561]}, "lm loss validation": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [5.60068]}, "lm loss validation vs samples": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [5.60068]}, "lm loss validation ppl": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [270.6116]}, "lm loss validation ppl vs samples": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [270.6116]}}
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp_sp/golden_values_lts_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp_sp/golden_values_lts_dgx_a100.json
deleted file mode 100644
index a35e26a051e..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp_sp/golden_values_lts_dgx_a100.json
+++ /dev/null
@@ -1,1223 +0,0 @@
-{
- "lm loss": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 10.8433,
- 10.87216,
- 10.85097,
- 10.81057,
- 10.64498,
- 10.63797,
- 10.42832,
- 10.1351,
- 9.93295,
- 9.83546,
- 9.58578,
- 9.84727,
- 9.88557,
- 9.63112,
- 9.78975,
- 9.51097,
- 9.46053,
- 9.65561,
- 9.38985,
- 9.33875,
- 9.24965,
- 9.15115,
- 9.18159,
- 9.0052,
- 9.19808,
- 9.06695,
- 9.16091,
- 9.1698,
- 9.30148,
- 8.98938,
- 8.93015,
- 9.05033,
- 9.04671,
- 8.6605,
- 8.72421,
- 8.7564,
- 8.69398,
- 8.74219,
- 8.66582,
- 8.77332,
- 8.66956,
- 8.86027,
- 8.84233,
- 8.50836,
- 8.39846,
- 8.43707,
- 8.49655,
- 8.3923,
- 8.44026,
- 8.59249,
- 8.37646,
- 8.19976,
- 8.23307,
- 8.22963,
- 8.27479,
- 7.92058,
- 8.10004,
- 7.89816,
- 8.25172,
- 8.23393,
- 8.00992,
- 7.97561,
- 7.92646,
- 7.74305,
- 7.74692,
- 7.65003,
- 7.52118,
- 7.9107,
- 7.70218,
- 7.45619,
- 7.74663,
- 7.77434,
- 7.54472,
- 7.30219,
- 7.45562,
- 7.34225,
- 7.4663,
- 7.22885,
- 7.63694,
- 7.28225,
- 7.34927,
- 7.21438,
- 7.2123,
- 7.41995,
- 7.17344,
- 7.28172,
- 6.99562,
- 7.00344,
- 7.03963,
- 7.13579,
- 6.82325,
- 6.98445,
- 7.08899,
- 6.9983,
- 6.87452,
- 6.75788,
- 6.99066,
- 7.06067,
- 6.7043,
- 6.58385,
- 6.72775,
- 6.74509,
- 6.73344,
- 6.73876,
- 6.65841,
- 6.40697,
- 6.63707,
- 6.61924,
- 6.44764,
- 6.62983,
- 6.74426,
- 6.61288,
- 6.7285,
- 6.69814,
- 6.62789,
- 6.5095,
- 6.60077,
- 6.4111,
- 6.66805,
- 6.25121,
- 6.25386,
- 6.30497,
- 6.39297,
- 6.35015,
- 6.45052,
- 6.29239,
- 6.33772,
- 6.23653,
- 6.20335,
- 6.39766,
- 6.32931,
- 6.32402,
- 6.16665,
- 6.16073,
- 6.24498,
- 6.39081,
- 6.20983,
- 6.15811,
- 6.18613,
- 6.12077,
- 6.06707,
- 6.07875,
- 6.2603,
- 6.41272,
- 6.26029,
- 6.29743,
- 6.10372,
- 6.17934,
- 6.00337,
- 6.03327,
- 5.95626,
- 6.25001,
- 6.18658,
- 5.96576,
- 5.78222,
- 6.12481,
- 5.84972,
- 6.10096,
- 5.7787,
- 6.1571,
- 6.13811,
- 6.07667,
- 5.91993,
- 6.1058,
- 5.93861,
- 6.19054,
- 5.8876,
- 5.78366,
- 5.77474,
- 5.67724,
- 6.01276,
- 5.99316,
- 6.06932,
- 5.88025,
- 6.03632,
- 5.96629,
- 5.99202,
- 5.99008,
- 5.94835,
- 5.83833,
- 5.94727,
- 5.61592,
- 5.69919,
- 5.88738,
- 5.8384,
- 5.85844,
- 5.76008,
- 5.83456,
- 5.72247,
- 5.5562,
- 5.71973,
- 5.61737,
- 5.82798,
- 5.59515,
- 5.70364,
- 5.70223,
- 5.89583,
- 5.63733,
- 5.84261,
- 5.73575,
- 5.86229,
- 5.32317,
- 5.89115,
- 5.86999,
- 5.84671,
- 5.40951,
- 5.40436,
- 5.6212,
- 5.59155,
- 5.48065,
- 5.57597,
- 5.66742,
- 5.47404,
- 5.73806,
- 5.50481,
- 5.58667,
- 5.6193,
- 5.6155,
- 5.5126,
- 5.61325,
- 5.66966,
- 5.68001,
- 5.58356,
- 5.66216,
- 5.37338,
- 5.6761,
- 5.6246,
- 5.42226,
- 5.58018,
- 5.62977,
- 5.55311,
- 5.34344,
- 5.53626,
- 5.48679,
- 5.4797,
- 5.37801,
- 5.55102,
- 5.59981,
- 5.38386,
- 5.52082,
- 5.48425,
- 5.32963,
- 5.501,
- 5.40703,
- 5.44227,
- 5.31599,
- 5.06438,
- 5.47765,
- 5.56882,
- 5.71613,
- 5.41382,
- 5.60171,
- 5.63397,
- 5.22909,
- 5.27054,
- 5.39242,
- 5.39593,
- 5.32649,
- 5.49503,
- 5.17951,
- 5.29869,
- 5.24187,
- 5.37352,
- 5.24905,
- 5.43951,
- 5.53349,
- 5.30617,
- 5.43051,
- 5.33592,
- 5.07569,
- 5.30806,
- 5.2527,
- 5.30192,
- 5.11002,
- 5.27549,
- 5.26604,
- 5.46869,
- 5.15386,
- 5.26145,
- 5.2071,
- 5.35322,
- 4.98154,
- 4.91142,
- 5.32291,
- 5.3909,
- 5.22591,
- 5.31717,
- 5.10092,
- 5.15923,
- 5.26361,
- 5.06622,
- 5.26522,
- 5.06572,
- 5.3425,
- 5.24739,
- 5.14577,
- 5.24209,
- 5.03756,
- 5.31387,
- 5.0503,
- 5.02538,
- 5.14018,
- 5.11039,
- 5.26931,
- 5.15823,
- 5.2748,
- 5.0928,
- 5.09208,
- 5.24848,
- 5.32417,
- 5.25092,
- 5.18929,
- 5.14216,
- 5.2897,
- 4.95024,
- 5.20765,
- 5.09114,
- 5.29977,
- 5.17091,
- 5.18545,
- 5.11166,
- 4.98284,
- 4.99251,
- 5.22042,
- 5.31276,
- 5.09889,
- 5.05435,
- 4.91545,
- 5.12121,
- 5.11554,
- 4.92359,
- 5.33454,
- 5.025,
- 5.09862,
- 5.16274,
- 4.99956,
- 5.06415,
- 5.0649,
- 4.99341,
- 5.07472,
- 5.16265,
- 4.97826,
- 5.17995,
- 4.93075,
- 4.91859,
- 5.05945,
- 4.99392,
- 4.90857,
- 4.77498,
- 4.9436,
- 5.11445,
- 5.01364,
- 5.01518,
- 5.33019,
- 4.95707,
- 4.99153,
- 5.04396,
- 4.80742,
- 4.73198,
- 4.99256,
- 5.03894,
- 4.87089,
- 4.95255,
- 5.04391,
- 5.02208,
- 4.81371,
- 4.89476,
- 4.9065,
- 4.82799,
- 4.73929,
- 5.01075,
- 4.7501,
- 5.20377,
- 4.78747,
- 4.99112,
- 4.73231,
- 4.78664,
- 4.81588,
- 4.64822,
- 4.65182,
- 4.84317,
- 4.80235,
- 4.79212,
- 4.9188,
- 4.88263,
- 4.92355,
- 4.76776,
- 4.87695,
- 4.72503,
- 4.91002,
- 4.95134,
- 4.86752,
- 4.70681,
- 4.78211,
- 4.89966,
- 4.70737,
- 4.86201,
- 4.69452,
- 4.6934,
- 4.64409
- ]
- },
- "num-zeros": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 57.0,
- 81.0,
- 77.0,
- 72.0,
- 77.0,
- 93.0,
- 94.0,
- 98.0,
- 109.0,
- 154.0,
- 104.0,
- 177.0,
- 128.0,
- 161.0,
- 184.0,
- 170.0,
- 167.0,
- 179.0,
- 151.0,
- 171.0,
- 159.0,
- 186.0,
- 173.0,
- 161.0,
- 188.0,
- 172.0,
- 192.0,
- 179.0,
- 144.0,
- 149.0,
- 153.0,
- 147.0,
- 168.0,
- 183.0,
- 148.0,
- 162.0,
- 157.0,
- 193.0,
- 185.0,
- 184.0,
- 162.0,
- 177.0,
- 152.0,
- 214.0,
- 178.0,
- 182.0,
- 188.0,
- 183.0,
- 180.0,
- 187.0,
- 216.0,
- 175.0,
- 191.0,
- 164.0,
- 169.0,
- 200.0,
- 171.0,
- 149.0,
- 212.0,
- 229.0,
- 188.0,
- 202.0,
- 188.0,
- 176.0,
- 202.0,
- 241.0,
- 202.0,
- 187.0,
- 194.0,
- 222.0,
- 204.0,
- 213.0,
- 180.0,
- 231.0,
- 210.0,
- 195.0,
- 193.0,
- 225.0,
- 216.0,
- 195.0,
- 224.0,
- 249.0,
- 209.0,
- 252.0,
- 223.0,
- 206.0,
- 162.0,
- 215.0,
- 184.0,
- 212.0,
- 207.0,
- 190.0,
- 244.0,
- 172.0,
- 198.0,
- 164.0,
- 218.0,
- 212.0,
- 154.0,
- 162.0,
- 186.0,
- 168.0,
- 173.0,
- 164.0,
- 165.0,
- 153.0,
- 177.0,
- 171.0,
- 130.0,
- 172.0,
- 184.0,
- 164.0,
- 151.0,
- 156.0,
- 137.0,
- 134.0,
- 151.0,
- 106.0,
- 165.0,
- 132.0,
- 127.0,
- 171.0,
- 105.0,
- 159.0,
- 149.0,
- 137.0,
- 140.0,
- 144.0,
- 111.0,
- 112.0,
- 105.0,
- 125.0,
- 136.0,
- 118.0,
- 107.0,
- 119.0,
- 118.0,
- 116.0,
- 126.0,
- 134.0,
- 138.0,
- 128.0,
- 128.0,
- 112.0,
- 122.0,
- 142.0,
- 107.0,
- 141.0,
- 142.0,
- 89.0,
- 119.0,
- 100.0,
- 105.0,
- 105.0,
- 143.0,
- 100.0,
- 95.0,
- 110.0,
- 136.0,
- 126.0,
- 121.0,
- 106.0,
- 128.0,
- 96.0,
- 103.0,
- 94.0,
- 112.0,
- 118.0,
- 110.0,
- 104.0,
- 103.0,
- 90.0,
- 86.0,
- 118.0,
- 124.0,
- 88.0,
- 122.0,
- 100.0,
- 158.0,
- 114.0,
- 129.0,
- 117.0,
- 108.0,
- 94.0,
- 122.0,
- 107.0,
- 83.0,
- 124.0,
- 108.0,
- 96.0,
- 99.0,
- 119.0,
- 93.0,
- 91.0,
- 103.0,
- 99.0,
- 80.0,
- 84.0,
- 112.0,
- 117.0,
- 119.0,
- 100.0,
- 91.0,
- 139.0,
- 125.0,
- 111.0,
- 118.0,
- 86.0,
- 114.0,
- 132.0,
- 95.0,
- 133.0,
- 104.0,
- 102.0,
- 92.0,
- 111.0,
- 99.0,
- 106.0,
- 75.0,
- 102.0,
- 99.0,
- 82.0,
- 103.0,
- 102.0,
- 100.0,
- 129.0,
- 103.0,
- 121.0,
- 110.0,
- 110.0,
- 111.0,
- 101.0,
- 98.0,
- 94.0,
- 99.0,
- 121.0,
- 90.0,
- 106.0,
- 107.0,
- 98.0,
- 103.0,
- 103.0,
- 106.0,
- 114.0,
- 106.0,
- 112.0,
- 91.0,
- 96.0,
- 100.0,
- 103.0,
- 110.0,
- 122.0,
- 97.0,
- 125.0,
- 97.0,
- 93.0,
- 94.0,
- 99.0,
- 95.0,
- 92.0,
- 99.0,
- 105.0,
- 108.0,
- 112.0,
- 119.0,
- 80.0,
- 123.0,
- 103.0,
- 98.0,
- 92.0,
- 110.0,
- 116.0,
- 97.0,
- 91.0,
- 113.0,
- 95.0,
- 116.0,
- 103.0,
- 116.0,
- 121.0,
- 108.0,
- 105.0,
- 120.0,
- 107.0,
- 90.0,
- 81.0,
- 108.0,
- 106.0,
- 112.0,
- 102.0,
- 104.0,
- 81.0,
- 118.0,
- 104.0,
- 97.0,
- 102.0,
- 90.0,
- 103.0,
- 98.0,
- 115.0,
- 140.0,
- 103.0,
- 121.0,
- 98.0,
- 92.0,
- 103.0,
- 94.0,
- 94.0,
- 125.0,
- 95.0,
- 110.0,
- 138.0,
- 122.0,
- 108.0,
- 115.0,
- 101.0,
- 86.0,
- 116.0,
- 119.0,
- 115.0,
- 109.0,
- 116.0,
- 90.0,
- 96.0,
- 105.0,
- 114.0,
- 102.0,
- 105.0,
- 139.0,
- 104.0,
- 92.0,
- 111.0,
- 113.0,
- 100.0,
- 115.0,
- 94.0,
- 108.0,
- 120.0,
- 100.0,
- 115.0,
- 106.0,
- 98.0,
- 96.0,
- 117.0,
- 105.0,
- 104.0,
- 105.0,
- 95.0,
- 126.0,
- 138.0,
- 116.0,
- 94.0,
- 134.0,
- 96.0,
- 120.0,
- 113.0,
- 139.0,
- 118.0,
- 118.0,
- 137.0,
- 111.0,
- 120.0,
- 114.0,
- 115.0,
- 141.0,
- 119.0,
- 110.0,
- 104.0,
- 111.0,
- 87.0,
- 97.0,
- 117.0,
- 126.0,
- 135.0,
- 119.0,
- 115.0,
- 87.0,
- 112.0,
- 111.0,
- 119.0,
- 101.0,
- 108.0,
- 132.0,
- 135.0,
- 122.0,
- 131.0,
- 93.0,
- 135.0,
- 131.0,
- 123.0,
- 121.0,
- 102.0,
- 108.0,
- 104.0,
- 106.0,
- 122.0,
- 91.0,
- 94.0,
- 129.0,
- 116.0
- ]
- },
- "iteration-time": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 24.02205,
- 1.24506,
- 1.24858,
- 1.24333,
- 1.25283,
- 1.25037,
- 1.25421,
- 1.2463,
- 1.2501,
- 1.26513,
- 1.24828,
- 1.26203,
- 1.26152,
- 1.25844,
- 1.24358,
- 1.24649,
- 1.24037,
- 1.26933,
- 1.24565,
- 1.24581,
- 1.25219,
- 1.26148,
- 1.25382,
- 1.28389,
- 1.25754,
- 1.2668,
- 1.25991,
- 1.26913,
- 1.25979,
- 1.27196,
- 1.26206,
- 1.27391,
- 1.2598,
- 1.2609,
- 1.26823,
- 1.41237,
- 1.25989,
- 1.27952,
- 1.26096,
- 1.2672,
- 1.2739,
- 1.26104,
- 1.26514,
- 1.26304,
- 1.26101,
- 1.26808,
- 1.28355,
- 1.25498,
- 1.25385,
- 1.26471,
- 1.26743,
- 1.27834,
- 1.25081,
- 1.24998,
- 1.273,
- 1.25459,
- 1.28314,
- 1.25536,
- 1.27322,
- 1.25723,
- 1.25258,
- 1.2737,
- 1.25174,
- 1.25458,
- 1.25465,
- 1.26423,
- 1.25884,
- 1.25794,
- 1.29369,
- 1.25823,
- 1.26468,
- 1.25525,
- 1.28545,
- 1.25487,
- 1.25381,
- 1.26521,
- 1.26327,
- 1.25623,
- 1.26167,
- 1.28421,
- 1.25744,
- 1.23929,
- 1.25396,
- 1.25408,
- 1.26624,
- 1.26554,
- 1.25271,
- 1.26468,
- 1.27195,
- 1.27503,
- 1.2657,
- 1.2661,
- 1.27456,
- 1.26939,
- 1.26586,
- 1.28144,
- 1.26291,
- 1.26343,
- 1.27277,
- 1.26516,
- 1.25715,
- 1.25949,
- 1.26476,
- 1.27715,
- 1.263,
- 1.27197,
- 1.2799,
- 1.26544,
- 1.26319,
- 1.26268,
- 1.27214,
- 1.26451,
- 1.26377,
- 1.26014,
- 1.27229,
- 1.25668,
- 1.26217,
- 1.27766,
- 1.25964,
- 1.26318,
- 1.26686,
- 1.27178,
- 1.28624,
- 1.26331,
- 1.27682,
- 1.4189,
- 1.28511,
- 1.272,
- 1.26632,
- 1.27543,
- 1.28147,
- 1.27518,
- 1.28733,
- 1.28232,
- 1.27614,
- 1.27792,
- 1.27502,
- 1.2703,
- 1.269,
- 1.26508,
- 1.27296,
- 1.26464,
- 1.27352,
- 1.25925,
- 1.27647,
- 1.27531,
- 1.262,
- 1.27258,
- 1.26864,
- 1.26393,
- 1.27468,
- 1.2704,
- 1.2669,
- 1.27408,
- 1.26653,
- 1.25934,
- 1.27085,
- 1.26066,
- 1.26381,
- 1.27106,
- 1.26813,
- 1.27425,
- 1.2675,
- 1.26972,
- 1.27219,
- 1.2599,
- 1.25343,
- 1.26631,
- 1.26613,
- 1.26456,
- 1.26363,
- 1.24696,
- 1.24735,
- 1.23999,
- 1.24278,
- 1.24375,
- 1.30135,
- 1.29599,
- 1.41849,
- 1.55305,
- 1.28657,
- 1.28352,
- 1.27354,
- 1.27715,
- 1.27402,
- 1.26602,
- 1.2595,
- 1.27111,
- 1.25739,
- 1.26466,
- 1.26356,
- 1.27812,
- 1.27551,
- 1.25594,
- 1.26434,
- 1.26429,
- 1.26587,
- 1.26167,
- 1.25603,
- 1.26467,
- 1.25248,
- 1.28015,
- 1.25039,
- 1.26242,
- 1.25191,
- 1.25406,
- 1.28967,
- 1.25465,
- 1.25278,
- 1.24787,
- 1.28566,
- 1.24579,
- 1.23833,
- 1.25526,
- 1.24804,
- 1.25288,
- 1.25311,
- 1.27069,
- 1.2692,
- 1.26358,
- 1.26482,
- 1.26587,
- 1.25692,
- 1.24695,
- 1.2519,
- 1.25969,
- 1.25174,
- 1.25841,
- 1.26427,
- 1.2659,
- 1.24632,
- 1.2552,
- 1.24879,
- 1.26097,
- 1.25377,
- 1.25145,
- 1.2607,
- 1.25105,
- 1.26351,
- 1.2637,
- 1.26492,
- 1.26318,
- 1.25456,
- 1.25979,
- 1.25791,
- 1.26316,
- 1.25826,
- 1.25874,
- 1.25298,
- 1.2801,
- 1.25579,
- 1.26876,
- 1.2587,
- 1.24948,
- 1.2555,
- 1.25745,
- 1.26029,
- 1.25145,
- 1.26455,
- 1.25779,
- 1.25424,
- 1.25778,
- 1.2666,
- 1.26833,
- 1.25606,
- 1.25517,
- 1.24487,
- 1.26487,
- 1.26401,
- 1.25739,
- 1.25258,
- 1.25456,
- 1.26282,
- 1.2624,
- 1.25291,
- 1.24606,
- 1.24381,
- 1.2644,
- 1.26256,
- 1.24699,
- 1.25568,
- 1.26046,
- 1.26178,
- 1.24752,
- 1.24631,
- 1.25387,
- 1.25042,
- 1.25335,
- 1.24857,
- 1.2779,
- 1.25834,
- 1.26516,
- 1.26356,
- 1.25971,
- 1.24704,
- 1.24808,
- 1.25221,
- 1.25458,
- 1.24918,
- 1.24796,
- 1.25898,
- 1.25776,
- 1.24651,
- 1.25908,
- 1.25272,
- 1.24913,
- 1.25911,
- 1.25475,
- 1.25986,
- 1.25067,
- 1.26015,
- 1.25973,
- 1.26456,
- 1.24812,
- 1.26296,
- 1.26051,
- 1.25975,
- 1.25669,
- 1.25402,
- 1.2504,
- 1.24884,
- 1.25361,
- 1.25258,
- 1.24646,
- 1.25477,
- 1.26152,
- 1.25586,
- 1.24538,
- 1.24197,
- 1.24636,
- 1.26242,
- 1.24754,
- 1.25326,
- 1.25781,
- 1.25382,
- 1.25739,
- 1.25142,
- 1.25264,
- 1.26736,
- 1.25905,
- 1.25007,
- 1.25292,
- 1.25509,
- 1.25421,
- 1.25501,
- 1.26274,
- 1.25472,
- 1.24705,
- 1.2509,
- 1.24897,
- 1.25724,
- 1.26927,
- 1.2435,
- 1.24864,
- 1.25188,
- 1.26436,
- 1.25981,
- 1.253,
- 1.27425,
- 1.25967,
- 1.25959,
- 1.25327,
- 1.27673,
- 1.25991,
- 1.26104,
- 1.27188,
- 1.26418,
- 1.26076,
- 1.26686,
- 1.26275,
- 1.25723,
- 1.25852,
- 1.26733,
- 1.26316,
- 1.25518,
- 1.25632,
- 1.26586,
- 1.26115,
- 1.25001,
- 1.25691,
- 1.26643,
- 1.26538,
- 1.26127,
- 1.2626,
- 1.25793,
- 1.26064,
- 1.24679,
- 1.26877,
- 1.26311,
- 1.26057,
- 1.26505,
- 1.26031,
- 1.25609,
- 1.25635,
- 1.27454,
- 1.2607,
- 1.25592,
- 1.26731,
- 1.26013,
- 1.25184
- ]
- }
-}
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp_sp/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp_sp/model_config.yaml
deleted file mode 100644
index 54237309642..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_fp8_tp_pp_sp/model_config.yaml
+++ /dev/null
@@ -1,55 +0,0 @@
-ENV_VARS:
- CUDA_DEVICE_MAX_CONNECTIONS: 1
- NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
- NVTE_FUSED_ATTN: 0
- NCCL_ALGO: Ring
- CUBLAS_WORKSPACE_CONFIG: :4096:8
-MODEL_ARGS:
- --num-layers: 12
- --hidden-size: 512
- --num-attention-heads: 8
- --log-params-norm: true
- --log-num-zeros-in-grad: true
- --log-validation-ppl-to-tensorboard: true
- --log-timers-to-tensorboard: true
- --tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
- --seq-length: 1024
- --max-position-embeddings: 1024
- --train-iters: 2000
- --timing-log-level: 0
- --lr-decay-iters: 320000
- --save: ${CHECKPOINT_SAVE_PATH}
- --load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
- --split: 949,50,1
- --distributed-backend: nccl
- --lr: 0.00015
- --lr-decay-style: cosine
- --min-lr: 1.0e-5
- --weight-decay: 1e-2
- --clip-grad: 1.0
- --lr-warmup-fraction: .01
- --log-interval: 1
- --save-interval: 10000
- --eval-interval: 1000
- --eval-iters: 10
- --transformer-impl: transformer_engine
- --tensor-model-parallel-size: 2
- --pipeline-model-parallel-size: 2
- --sequence-parallel: true
- --deterministic-mode: true
- --no-gradient-accumulation-fusion: true
- --fp8-format: hybrid
- --fp8-amax-history-len: 1024
- --fp8-amax-compute-algo: max
- --attention-softmax-in-fp32: true
- --ckpt-format: torch_dist
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
- --data-cache-path: ${DATA_CACHE_PATH}
- --bf16: true
- --log-memory-to-tensorboard: true
-TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_native_fp8_tp_pp_sp/golden_values_dev_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_native_fp8_tp_pp_sp/golden_values_dev_dgx_a100.json
deleted file mode 100644
index 392f14d9abc..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_native_fp8_tp_pp_sp/golden_values_dev_dgx_a100.json
+++ /dev/null
@@ -1 +0,0 @@
-{"forward-backward-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [21.99065, 1.57716, 1.57152, 1.59321, 1.5863, 1.57478, 1.58558, 1.58758, 1.58479, 1.57895, 1.58493, 1.59324, 1.58233, 1.56795, 1.58055, 1.56688, 1.5678, 1.56791, 1.63783, 1.57453, 1.56285, 1.57599, 1.58307, 1.58147, 1.58884, 1.58487, 1.58844, 1.57964, 1.57929, 1.58802, 1.58073, 1.58122, 1.58449, 1.66952, 1.599, 1.60139, 1.74695, 1.60068, 1.58989, 1.59432, 1.59007, 1.58621, 1.5794, 1.58271, 1.58306, 1.59677, 1.59288, 1.58275, 1.67584, 1.59164, 1.60367, 1.5998, 1.60414, 1.59758, 1.58882, 1.60425, 1.59006, 1.58295, 1.58588, 1.60115, 1.59394, 1.6001, 1.59693, 1.5944, 1.59722, 1.60347, 1.59248, 1.67877, 1.59416, 1.59784, 1.61277, 1.59908, 1.59639, 1.5935, 1.59862, 1.61381, 1.60093, 1.59916, 1.59139, 1.59544, 1.60373, 1.59931, 1.59729, 1.58924, 1.59278, 1.60393, 1.59751, 1.59588, 1.597, 1.5921, 1.60557, 1.5915, 1.59296, 1.59099, 1.58952, 1.59785, 1.59236, 1.59138, 1.58196, 1.68409, 1.59552, 1.60388, 1.59454, 1.58942, 1.58688, 1.59613, 1.60092, 1.59976, 1.59462, 1.60601, 1.59966, 1.59879, 1.59803, 1.59743, 1.60087, 1.60123, 1.60561, 1.59721, 1.60002, 1.59717, 1.60267, 1.60202, 1.58969, 1.5937, 1.59501, 1.59729, 1.6055, 1.59373, 1.59552, 1.59903, 1.60628, 1.59959, 1.60033, 1.59523, 1.59534, 1.59886, 1.59989, 1.59127, 1.60846, 1.60265, 1.6054, 1.59487, 1.59192, 1.58491, 1.59173, 1.59624, 1.60184, 1.59635, 1.60701, 1.59973, 1.59592, 1.58783, 1.59596, 1.59257, 1.60207, 1.59766, 1.59014, 1.59147, 1.58958, 1.58849, 1.59599, 1.59796, 1.59187, 1.59629, 1.59167, 1.59103, 1.58381, 1.59206, 1.58888, 1.5904, 1.58555, 1.59114, 1.58539, 1.58566, 1.5894, 1.58315, 1.57556, 1.5798, 1.57936, 1.59144, 1.59188, 1.58985, 1.58744, 1.57959, 1.57707, 1.58114, 1.57447, 1.58757, 1.58393, 1.5814, 1.58214, 1.56869, 1.59904, 1.58832, 1.58446, 1.5886, 1.5964, 1.59995, 1.58984, 1.58458, 1.57848, 1.58262, 1.58372, 1.58511, 1.57472, 1.58482, 1.57884, 1.57655, 1.57371, 1.56768, 1.58436, 1.57434, 1.58546, 1.57895, 1.58824, 1.58943, 1.58534, 1.58931, 1.58768, 1.67183, 1.5994, 1.59551, 1.58731, 1.58941, 1.59427, 1.59768, 1.58889, 1.5907, 1.58959, 1.58719, 1.59215, 1.5863, 1.59281, 1.59155, 1.58447, 1.58437, 1.5847, 1.58696, 1.59622, 1.58517, 1.59019, 1.60434, 1.59968, 1.5969, 1.59751, 1.59456, 1.6066, 1.59805, 1.59315, 1.59835, 1.60342, 1.62288, 1.59735, 1.59455, 1.59386, 1.5899, 1.60537, 1.58935, 1.59479, 1.5931, 1.59564, 1.61221, 1.59658, 1.59741, 1.60139, 1.59726, 1.60686, 1.59462, 1.59958, 1.59653, 1.59254, 1.60457, 1.59551, 1.59428, 1.60093, 1.5944, 1.60142, 1.59772, 1.58999, 1.59811, 1.59342, 1.59459, 1.59229, 1.59446, 1.59758, 1.59514, 1.59376, 1.60015, 1.59289, 1.60569, 1.59243, 1.59995, 1.60277, 1.58962, 1.59704, 1.59408, 1.58742, 1.59956, 1.5946, 1.59711, 1.59521, 1.60094, 1.60537, 1.59472, 1.60512, 1.59709, 1.59942, 1.60326, 1.59747, 1.59643, 1.60252, 1.59668, 1.5978, 1.59291, 1.60286, 1.59494, 1.60307, 1.6023, 1.61125, 1.60608, 1.60499, 1.60013, 1.60294, 1.59839, 1.59445, 1.59771, 1.59912, 1.59625, 1.60071, 1.592, 1.59986, 1.59715, 1.59092, 1.5888, 1.58483, 1.58369, 1.58578, 1.58892, 1.58607, 1.57772, 1.58567, 1.58058, 1.57579, 1.58081, 1.57885, 1.57944, 1.5775, 1.57886, 1.58441, 1.64955, 1.57793, 1.57628, 1.57996, 1.60901, 1.5979, 1.59148, 1.58504, 1.58873, 1.61471, 1.61412, 1.59947, 1.59781, 1.59535, 1.61042, 1.60213, 1.59684, 1.59637, 1.59781, 1.60971, 1.59714, 1.58835, 1.59658, 1.5958, 1.5924, 1.59655, 1.59597, 1.60519, 1.60003, 1.61195, 1.61366, 1.6023, 1.60659, 1.59405, 1.60115, 1.6049, 1.6052, 1.60253, 1.59948, 1.5816, 1.59621, 1.58755, 1.59445, 1.59719, 1.59069, 1.60911, 1.59481, 1.59684, 1.60214, 1.59905, 1.60381]}, "forward-compute-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [6.16126, 0.78048, 0.77638, 0.78285, 0.77945, 0.7768, 0.78398, 0.78215, 0.7833, 0.77542, 0.78468, 0.78711, 0.78251, 0.76662, 0.76894, 0.76826, 0.77171, 0.76847, 0.83221, 0.7706, 0.76442, 0.77548, 0.77966, 0.76518, 0.7854, 0.7799, 0.77136, 0.76634, 0.78834, 0.77019, 0.78986, 0.77045, 0.78652, 0.87018, 0.80011, 0.7944, 0.94182, 0.79666, 0.78564, 0.78708, 0.78355, 0.78735, 0.78535, 0.79227, 0.79173, 0.79116, 0.79578, 0.78576, 0.88058, 0.78541, 0.7905, 0.80177, 0.80159, 0.79536, 0.78436, 0.80424, 0.79113, 0.78133, 0.79513, 0.79725, 0.78505, 0.80445, 0.7974, 0.80505, 0.80566, 0.79011, 0.78303, 0.8828, 0.7992, 0.80046, 0.79496, 0.80104, 0.80208, 0.78598, 0.79918, 0.79817, 0.80692, 0.79948, 0.79832, 0.80065, 0.79953, 0.80613, 0.80349, 0.79995, 0.80406, 0.8022, 0.80453, 0.80228, 0.8056, 0.79734, 0.80242, 0.78707, 0.79319, 0.80876, 0.78925, 0.79762, 0.79177, 0.81095, 0.78559, 0.87702, 0.80826, 0.80874, 0.79998, 0.78873, 0.79623, 0.80044, 0.7965, 0.80088, 0.80451, 0.80617, 0.80803, 0.80736, 0.80357, 0.80072, 0.80574, 0.80861, 0.80081, 0.80256, 0.8016, 0.80416, 0.80062, 0.79705, 0.79613, 0.7934, 0.79423, 0.79439, 0.79639, 0.79437, 0.80375, 0.79641, 0.8075, 0.79693, 0.80388, 0.79802, 0.79685, 0.80158, 0.79875, 0.79886, 0.80926, 0.81104, 0.80752, 0.80381, 0.79608, 0.7893, 0.78982, 0.79582, 0.79985, 0.79486, 0.8058, 0.79802, 0.79424, 0.79685, 0.79506, 0.79473, 0.79858, 0.79203, 0.79193, 0.79375, 0.79263, 0.78662, 0.78983, 0.79242, 0.78834, 0.78866, 0.78847, 0.79475, 0.78474, 0.78928, 0.78727, 0.7942, 0.78678, 0.78404, 0.7855, 0.78669, 0.7807, 0.79077, 0.78107, 0.78201, 0.78183, 0.80216, 0.79952, 0.79773, 0.7904, 0.78485, 0.7784, 0.78943, 0.78644, 0.78928, 0.79161, 0.79481, 0.79068, 0.78383, 0.79727, 0.78767, 0.79378, 0.79855, 0.79573, 0.79906, 0.79796, 0.78811, 0.77833, 0.78832, 0.79352, 0.78682, 0.78545, 0.78929, 0.78422, 0.78978, 0.78901, 0.78354, 0.78883, 0.78807, 0.79656, 0.79382, 0.79009, 0.79261, 0.79204, 0.79399, 0.79138, 0.87044, 0.79415, 0.78856, 0.7904, 0.7891, 0.78842, 0.79047, 0.78866, 0.78816, 0.78669, 0.78557, 0.78863, 0.79242, 0.79337, 0.78575, 0.78866, 0.78509, 0.78346, 0.78462, 0.78704, 0.78025, 0.78234, 0.78547, 0.78832, 0.78406, 0.79176, 0.78752, 0.79148, 0.7926, 0.78905, 0.79623, 0.79876, 0.80189, 0.79329, 0.78938, 0.78571, 0.79206, 0.79022, 0.78916, 0.79198, 0.78965, 0.78841, 0.79706, 0.79681, 0.79422, 0.79582, 0.7978, 0.7929, 0.79692, 0.79951, 0.79613, 0.78441, 0.78081, 0.78582, 0.78913, 0.79294, 0.7902, 0.78677, 0.79445, 0.79001, 0.79247, 0.78884, 0.78757, 0.79082, 0.79372, 0.79339, 0.79117, 0.79464, 0.79238, 0.78456, 0.80253, 0.7832, 0.79582, 0.78585, 0.78817, 0.7996, 0.80334, 0.80038, 0.78266, 0.79835, 0.80583, 0.7884, 0.803, 0.7964, 0.7803, 0.80771, 0.78154, 0.78737, 0.78425, 0.79511, 0.79935, 0.79899, 0.80031, 0.79737, 0.7882, 0.78726, 0.80196, 0.78826, 0.79069, 0.79987, 0.80053, 0.79658, 0.80868, 0.78979, 0.79176, 0.80466, 0.79718, 0.80577, 0.78989, 0.78977, 0.79845, 0.80176, 0.79513, 0.79765, 0.78377, 0.78605, 0.7817, 0.78486, 0.78251, 0.782, 0.77773, 0.78515, 0.78532, 0.7826, 0.78594, 0.7847, 0.78814, 0.78399, 0.78924, 0.78495, 0.85297, 0.78501, 0.78455, 0.78521, 0.79499, 0.78326, 0.78572, 0.78491, 0.78588, 0.79342, 0.79911, 0.79939, 0.79997, 0.78403, 0.79216, 0.80483, 0.79356, 0.79564, 0.79104, 0.79195, 0.79461, 0.79321, 0.78786, 0.79505, 0.78766, 0.78873, 0.7989, 0.79328, 0.79827, 0.79828, 0.79999, 0.80446, 0.80505, 0.79428, 0.80603, 0.80135, 0.79708, 0.78828, 0.78401, 0.78511, 0.79061, 0.7807, 0.78293, 0.7859, 0.78918, 0.79204, 0.7906, 0.79616, 0.79381, 0.7949, 0.79715]}, "backward-compute-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [4.59311, 0.76076, 0.76217, 0.75984, 0.7615, 0.75659, 0.76053, 0.7532, 0.76274, 0.76117, 0.76101, 0.76233, 0.76144, 0.75668, 0.76922, 0.75609, 0.75913, 0.76116, 0.76025, 0.76541, 0.75884, 0.75825, 0.75703, 0.766, 0.76226, 0.76154, 0.76489, 0.76817, 0.75764, 0.76666, 0.76075, 0.75889, 0.75671, 0.76413, 0.76441, 0.76109, 0.75862, 0.76306, 0.74826, 0.75641, 0.74619, 0.74555, 0.74425, 0.74896, 0.74343, 0.75132, 0.74633, 0.74611, 0.74624, 0.74486, 0.75681, 0.756, 0.75967, 0.7522, 0.74699, 0.75759, 0.75126, 0.74675, 0.75177, 0.75405, 0.7585, 0.75155, 0.75405, 0.75102, 0.75148, 0.75893, 0.74911, 0.74587, 0.75218, 0.74921, 0.76638, 0.74462, 0.7501, 0.7496, 0.74661, 0.7608, 0.75236, 0.74756, 0.74835, 0.74741, 0.75597, 0.74513, 0.75335, 0.74569, 0.74992, 0.75987, 0.73959, 0.74426, 0.7594, 0.74595, 0.75601, 0.74294, 0.74297, 0.75107, 0.74798, 0.75807, 0.74348, 0.75472, 0.74211, 0.7499, 0.7459, 0.75376, 0.74383, 0.74411, 0.74537, 0.74321, 0.75045, 0.74449, 0.75823, 0.74876, 0.74922, 0.75592, 0.75588, 0.75204, 0.74904, 0.74934, 0.76179, 0.74708, 0.74898, 0.7495, 0.749, 0.75109, 0.75134, 0.74604, 0.74742, 0.74319, 0.75078, 0.74752, 0.75245, 0.74673, 0.75517, 0.75235, 0.74881, 0.74945, 0.75053, 0.74903, 0.75641, 0.74336, 0.76521, 0.75829, 0.75724, 0.75492, 0.7561, 0.75292, 0.74603, 0.75381, 0.74787, 0.75257, 0.76831, 0.74923, 0.75133, 0.74595, 0.75539, 0.74856, 0.75247, 0.75168, 0.74839, 0.75531, 0.74901, 0.75107, 0.75151, 0.75163, 0.75496, 0.75207, 0.75274, 0.75371, 0.75218, 0.75324, 0.75429, 0.74775, 0.75082, 0.74975, 0.75003, 0.74514, 0.74798, 0.7422, 0.74955, 0.74687, 0.74432, 0.76318, 0.76862, 0.75695, 0.75138, 0.74947, 0.74824, 0.74949, 0.74673, 0.76097, 0.75456, 0.75612, 0.74619, 0.74667, 0.75557, 0.75602, 0.74867, 0.74532, 0.75908, 0.75984, 0.75566, 0.75544, 0.74912, 0.74344, 0.74466, 0.743, 0.74211, 0.75391, 0.74844, 0.74322, 0.7419, 0.7391, 0.75107, 0.74688, 0.74472, 0.74867, 0.74188, 0.75312, 0.75735, 0.75298, 0.75011, 0.83767, 0.75688, 0.7468, 0.75125, 0.75873, 0.75439, 0.76222, 0.74909, 0.75114, 0.74996, 0.74891, 0.75631, 0.75529, 0.75222, 0.74576, 0.74916, 0.74348, 0.7422, 0.74917, 0.74763, 0.74945, 0.74253, 0.75781, 0.74585, 0.75081, 0.75209, 0.75165, 0.7532, 0.75146, 0.75199, 0.75085, 0.75606, 0.76797, 0.74123, 0.75583, 0.7498, 0.74976, 0.76018, 0.74891, 0.74315, 0.74567, 0.74733, 0.76326, 0.74371, 0.74843, 0.74397, 0.74563, 0.76375, 0.74742, 0.7484, 0.75035, 0.74757, 0.75381, 0.7431, 0.74767, 0.74383, 0.74076, 0.75278, 0.75322, 0.74717, 0.74642, 0.74435, 0.74553, 0.75415, 0.75172, 0.74406, 0.74946, 0.74845, 0.7471, 0.74058, 0.74992, 0.74948, 0.74994, 0.75938, 0.75195, 0.75199, 0.75277, 0.74398, 0.75468, 0.74625, 0.74009, 0.75462, 0.74436, 0.75709, 0.75842, 0.75583, 0.75652, 0.75955, 0.75822, 0.74976, 0.74693, 0.7489, 0.7484, 0.74876, 0.75623, 0.75485, 0.75131, 0.75086, 0.75519, 0.7563, 0.75201, 0.74461, 0.75083, 0.75104, 0.7491, 0.74353, 0.74963, 0.74824, 0.75106, 0.75407, 0.74618, 0.7523, 0.75149, 0.74913, 0.74663, 0.74746, 0.7482, 0.74592, 0.74512, 0.75269, 0.74881, 0.75383, 0.74575, 0.74092, 0.74646, 0.74972, 0.75151, 0.74727, 0.74596, 0.75029, 0.74634, 0.74441, 0.75077, 0.76193, 0.7811, 0.76201, 0.76484, 0.77016, 0.76471, 0.76985, 0.76565, 0.75567, 0.76091, 0.76601, 0.7782, 0.76131, 0.75676, 0.76458, 0.76377, 0.77738, 0.75801, 0.75902, 0.762, 0.75749, 0.75518, 0.75814, 0.7671, 0.76157, 0.76399, 0.77689, 0.76899, 0.76062, 0.76435, 0.76315, 0.75948, 0.77408, 0.75612, 0.76269, 0.75559, 0.76227, 0.77122, 0.76094, 0.76349, 0.7582, 0.75871, 0.77745, 0.76055, 0.76243, 0.76016, 0.76322, 0.76742]}, "batch-generator-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.19292, 0.01741, 0.01488, 0.01641, 0.01712, 0.01701, 0.01724, 0.01612, 0.01735, 0.01689, 0.01449, 0.01795, 0.01495, 0.01541, 0.01502, 0.01516, 0.01428, 0.01451, 0.01769, 0.01847, 0.0169, 0.01788, 0.01813, 0.01751, 0.01774, 0.01679, 0.01619, 0.01655, 0.01654, 0.01696, 0.0174, 0.0185, 0.01671, 0.01581, 0.01697, 0.01627, 0.02111, 0.01585, 0.0176, 0.01783, 0.01799, 0.01548, 0.01578, 0.01602, 0.01539, 0.01659, 0.01748, 0.01708, 0.01454, 0.01909, 0.01622, 0.01722, 0.01943, 0.01822, 0.01639, 0.01887, 0.0157, 0.01802, 0.01601, 0.01682, 0.01679, 0.01666, 0.01696, 0.01447, 0.01725, 0.01735, 0.01643, 0.01884, 0.01609, 0.0185, 0.0184, 0.01703, 0.01561, 0.01899, 0.01693, 0.01673, 0.01557, 0.02037, 0.01648, 0.02182, 0.01581, 0.01883, 0.01486, 0.01422, 0.01602, 0.0206, 0.01692, 0.01644, 0.01443, 0.0164, 0.01772, 0.01699, 0.01792, 0.01841, 0.01616, 0.01914, 0.01786, 0.01399, 0.01385, 0.01298, 0.01984, 0.01393, 0.01641, 0.01237, 0.01672, 0.01523, 0.01481, 0.01312, 0.01514, 0.0141, 0.01688, 0.01659, 0.01531, 0.01306, 0.01415, 0.01307, 0.01504, 0.01566, 0.01521, 0.01304, 0.0151, 0.01337, 0.01578, 0.01428, 0.01733, 0.01324, 0.01568, 0.01651, 0.01314, 0.01407, 0.01374, 0.01429, 0.01421, 0.01802, 0.01439, 0.01347, 0.01541, 0.01301, 0.01489, 0.01769, 0.01406, 0.01394, 0.01544, 0.01425, 0.01399, 0.01414, 0.01541, 0.01538, 0.01478, 0.01476, 0.01498, 0.01626, 0.01614, 0.01516, 0.0146, 0.02163, 0.01496, 0.01399, 0.0156, 0.01517, 0.01657, 0.01525, 0.02091, 0.01583, 0.01574, 0.01726, 0.01555, 0.01523, 0.01459, 0.01318, 0.01563, 0.01531, 0.01592, 0.01602, 0.01375, 0.01616, 0.01854, 0.0199, 0.01523, 0.01384, 0.01396, 0.01413, 0.01587, 0.01384, 0.01554, 0.01277, 0.0125, 0.01321, 0.01511, 0.01439, 0.01651, 0.01382, 0.01689, 0.01614, 0.01571, 0.01361, 0.01704, 0.01534, 0.01385, 0.01423, 0.20705, 0.01218, 0.01233, 0.01727, 0.01275, 0.01244, 0.01327, 0.01272, 0.01371, 0.01665, 0.01392, 0.01222, 0.01222, 0.01188, 0.01265, 0.01482, 0.01632, 0.01649, 0.01702, 0.10117, 0.01844, 0.01611, 0.01574, 0.01967, 0.01779, 0.0181, 0.01873, 0.01598, 0.01615, 0.0136, 0.01405, 0.0131, 0.01348, 0.01358, 0.01592, 0.01254, 0.01772, 0.01503, 0.01408, 0.01322, 0.01435, 0.0158, 0.01713, 0.01512, 0.01582, 0.01578, 0.01584, 0.01532, 0.01652, 0.01516, 0.01295, 0.01398, 0.01359, 0.01339, 0.01358, 0.01304, 0.01422, 0.01314, 0.01282, 0.01422, 0.01411, 0.01529, 0.01575, 0.01454, 0.01377, 0.01423, 0.0158, 0.0128, 0.01659, 0.0174, 0.01592, 0.01617, 0.01462, 0.01415, 0.01495, 0.01263, 0.01928, 0.01701, 0.01799, 0.01302, 0.01537, 0.01683, 0.01358, 0.01378, 0.01553, 0.01478, 0.01516, 0.01864, 0.01487, 0.0145, 0.01315, 0.0163, 0.01453, 0.01978, 0.01808, 0.01337, 0.01516, 0.01483, 0.0141, 0.01325, 0.01391, 0.01431, 0.01452, 0.01452, 0.01284, 0.01318, 0.01339, 0.01336, 0.01442, 0.01234, 0.01424, 0.01284, 0.01762, 0.01661, 0.01281, 0.01962, 0.01329, 0.01356, 0.01369, 0.01291, 0.01345, 0.01577, 0.01307, 0.01371, 0.01245, 0.0144, 0.01266, 0.01493, 0.01942, 0.01384, 0.01403, 0.01338, 0.01325, 0.01563, 0.0138, 0.01307, 0.01453, 0.0157, 0.01517, 0.01449, 0.01345, 0.01482, 0.01389, 0.01533, 0.01504, 0.01529, 0.01484, 0.01361, 0.01578, 0.01436, 0.01584, 0.01282, 0.01395, 0.01777, 0.01465, 0.01446, 0.01422, 0.01426, 0.01624, 0.01786, 0.01661, 0.01321, 0.01562, 0.016, 0.0161, 0.01445, 0.01562, 0.01697, 0.01694, 0.01328, 0.01308, 0.01623, 0.01535, 0.01156, 0.01359, 0.01294, 0.01787, 0.01354, 0.01547, 0.01746, 0.01479, 0.01512, 0.0137, 0.01697, 0.01836, 0.0165, 0.01597, 0.01426, 0.01481, 0.01758, 0.01613, 0.01995, 0.01744, 0.01619, 0.02014, 0.01917, 0.01834, 0.02092, 0.0156, 0.01825]}, "forward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [5.93081, 0.02344, 0.02331, 0.02309, 0.02318, 0.02288, 0.02295, 0.02315, 0.02278, 0.02311, 0.02303, 0.02319, 0.02297, 0.02355, 0.0232, 0.02307, 0.02294, 0.02279, 0.02348, 0.02322, 0.02312, 0.02338, 0.02754, 0.02903, 0.02328, 0.02314, 0.02339, 0.02314, 0.02316, 0.02611, 0.02298, 0.02317, 0.02368, 0.02303, 0.02318, 0.0236, 0.02624, 0.02329, 0.02423, 0.02403, 0.02326, 0.02356, 0.02358, 0.02322, 0.02307, 0.02339, 0.02352, 0.02314, 0.02321, 0.02319, 0.02427, 0.02732, 0.02447, 0.02413, 0.02414, 0.02384, 0.02448, 0.02435, 0.0243, 0.02437, 0.02392, 0.02395, 0.02424, 0.0244, 0.02386, 0.02399, 0.02583, 0.02402, 0.02381, 0.02363, 0.02384, 0.02415, 0.02408, 0.02332, 0.02351, 0.02417, 0.02341, 0.02374, 0.0239, 0.02359, 0.02348, 0.02367, 0.02309, 0.02341, 0.02304, 0.02341, 0.02349, 0.02339, 0.02324, 0.02343, 0.02447, 0.02397, 0.02425, 0.02336, 0.02357, 0.02378, 0.02358, 0.02333, 0.02324, 0.02381, 0.02363, 0.02361, 0.02379, 0.023, 0.02331, 0.02406, 0.02303, 0.02381, 0.02338, 0.0233, 0.02375, 0.02361, 0.02338, 0.0254, 0.02366, 0.02346, 0.02319, 0.0231, 0.02322, 0.02336, 0.02359, 0.02301, 0.0232, 0.0231, 0.02325, 0.02535, 0.02543, 0.0249, 0.0258, 0.02421, 0.02631, 0.02569, 0.02546, 0.02523, 0.02374, 0.02369, 0.02287, 0.02328, 0.02335, 0.02342, 0.02348, 0.02584, 0.02846, 0.02333, 0.02325, 0.02317, 0.02344, 0.02362, 0.02449, 0.02398, 0.02331, 0.02313, 0.02338, 0.02374, 0.02377, 0.02343, 0.02294, 0.02316, 0.02278, 0.02313, 0.02341, 0.02344, 0.02325, 0.02347, 0.02341, 0.02425, 0.0234, 0.0236, 0.02348, 0.02328, 0.02322, 0.02797, 0.02349, 0.02368, 0.02483, 0.02541, 0.02365, 0.02349, 0.02286, 0.02337, 0.02361, 0.02351, 0.02501, 0.02329, 0.02303, 0.02332, 0.02369, 0.02402, 0.02326, 0.02743, 0.02371, 0.02333, 0.02452, 0.02852, 0.02423, 0.02431, 0.02363, 0.02347, 0.0234, 0.02355, 0.0171, 0.02364, 0.02374, 0.02365, 0.02307, 0.02279, 0.02328, 0.02362, 0.0233, 0.02395, 0.02325, 0.02349, 0.0286, 0.02347, 0.02365, 0.02351, 0.02314, 0.02283, 0.02321, 0.02365, 0.02339, 0.02363, 0.02445, 0.0234, 0.023, 0.02306, 0.02312, 0.0258, 0.02371, 0.02351, 0.02414, 0.02516, 0.02398, 0.02387, 0.02789, 0.02332, 0.02291, 0.02319, 0.02382, 0.02362, 0.02352, 0.0236, 0.02482, 0.02336, 0.02343, 0.02386, 0.02373, 0.02332, 0.02345, 0.02366, 0.02371, 0.02383, 0.02391, 0.02309, 0.02396, 0.0237, 0.02358, 0.02332, 0.02354, 0.0237, 0.02431, 0.02339, 0.02333, 0.02358, 0.02566, 0.02353, 0.02329, 0.02355, 0.02334, 0.02388, 0.02322, 0.02748, 0.02759, 0.02327, 0.02777, 0.02798, 0.0238, 0.02318, 0.02324, 0.02335, 0.02358, 0.02398, 0.02384, 0.02417, 0.02338, 0.02373, 0.02324, 0.02322, 0.02308, 0.02335, 0.02824, 0.02882, 0.02297, 0.02325, 0.02282, 0.02322, 0.02355, 0.02322, 0.02216, 0.02334, 0.02367, 0.02317, 0.0235, 0.02347, 0.02352, 0.02303, 0.02358, 0.02344, 0.02281, 0.02283, 0.02317, 0.02298, 0.02317, 0.02316, 0.02391, 0.02343, 0.02303, 0.02332, 0.02335, 0.02338, 0.02344, 0.0231, 0.02322, 0.02326, 0.02319, 0.02352, 0.02355, 0.02458, 0.02323, 0.02296, 0.02379, 0.02609, 0.02363, 0.02342, 0.02402, 0.02329, 0.02315, 0.02333, 0.02366, 0.02341, 0.02336, 0.02367, 0.02372, 0.02313, 0.02316, 0.02322, 0.0229, 0.02346, 0.02318, 0.02345, 0.0231, 0.02329, 0.0234, 0.02416, 0.02352, 0.0233, 0.02333, 0.02358, 0.02304, 0.0234, 0.02373, 0.02367, 0.02364, 0.02394, 0.02331, 0.02361, 0.02549, 0.02611, 0.02307, 0.02307, 0.02339, 0.02305, 0.02337, 0.02343, 0.02331, 0.02306, 0.02371, 0.02326, 0.02401, 0.02338, 0.02329, 0.02355, 0.02339, 0.02318, 0.02379, 0.02372, 0.02332, 0.02367, 0.02321, 0.02384, 0.0232, 0.02419, 0.02337, 0.02355, 0.0235, 0.02303, 0.02314, 0.02384, 0.02385, 0.02327]}, "forward-send-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.86591, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00011, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00014, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00015, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00011, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00014, 0.00012, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00014, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00012, 0.00014, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00016, 0.00013, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.0001, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00019, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00014, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00014, 0.00012, 0.00012, 0.00013, 0.00013, 0.00021, 0.00017, 0.00013, 0.00016, 0.00019, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00015, 0.00017, 0.00012, 0.00012, 0.00013, 0.00012, 0.00011, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00011, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00013, 0.00014, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00014, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00012, 0.00012, 0.00016, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00012]}, "backward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.02324, 0.02473, 0.02485, 0.0257, 0.02421, 0.02511, 0.02424, 0.02512, 0.02482, 0.02484, 0.02503, 0.02501, 0.02497, 0.02408, 0.02453, 0.02476, 0.02472, 0.0245, 0.02469, 0.0238, 0.02472, 0.02383, 0.02443, 0.02414, 0.02458, 0.02427, 0.02418, 0.02518, 0.02515, 0.02471, 0.02487, 0.02507, 0.0252, 0.04234, 0.02563, 0.02482, 0.02527, 0.0252, 0.02511, 0.02616, 0.02552, 0.02553, 0.02507, 0.0247, 0.02488, 0.02838, 0.02802, 0.0284, 0.02834, 0.02994, 0.02821, 0.02845, 0.02966, 0.02456, 0.02638, 0.02786, 0.02477, 0.02529, 0.02816, 0.0278, 0.024, 0.02485, 0.02472, 0.02443, 0.02679, 0.02889, 0.02923, 0.02446, 0.02467, 0.02491, 0.02448, 0.02524, 0.0247, 0.02381, 0.02482, 0.02267, 0.02554, 0.02506, 0.02479, 0.02511, 0.02493, 0.02473, 0.02445, 0.02465, 0.02466, 0.02435, 0.02438, 0.02454, 0.02703, 0.02859, 0.02838, 0.02463, 0.02457, 0.02449, 0.02484, 0.02427, 0.02489, 0.02919, 0.02783, 0.02446, 0.02864, 0.02839, 0.02885, 0.02916, 0.02535, 0.02922, 0.02859, 0.02867, 0.02674, 0.02913, 0.02404, 0.02357, 0.02473, 0.02426, 0.0237, 0.02368, 0.02461, 0.02449, 0.02432, 0.02416, 0.02668, 0.0259, 0.02394, 0.02449, 0.0245, 0.02639, 0.02567, 0.02428, 0.02416, 0.0239, 0.0246, 0.0245, 0.02396, 0.02903, 0.02872, 0.02891, 0.0242, 0.0248, 0.02619, 0.02586, 0.02476, 0.02646, 0.02366, 0.02382, 0.02621, 0.02353, 0.02399, 0.02459, 0.02528, 0.02408, 0.0246, 0.02424, 0.028, 0.02928, 0.02952, 0.02881, 0.02431, 0.02457, 0.02417, 0.02444, 0.02498, 0.02401, 0.02303, 0.02437, 0.02609, 0.02618, 0.0244, 0.02636, 0.02449, 0.02888, 0.0291, 0.02963, 0.02433, 0.02789, 0.03263, 0.03258, 0.02856, 0.02595, 0.02508, 0.02561, 0.02568, 0.02893, 0.02364, 0.02454, 0.02431, 0.02431, 0.02435, 0.02361, 0.02447, 0.02415, 0.02557, 0.02442, 0.02388, 0.02473, 0.02836, 0.02932, 0.02902, 0.02464, 0.02588, 0.02525, 0.02855, 0.02485, 0.03232, 0.02798, 0.02376, 0.02448, 0.02369, 0.02397, 0.02417, 0.02554, 0.02412, 0.02385, 0.02386, 0.02939, 0.02461, 0.02396, 0.02522, 0.02468, 0.02408, 0.02344, 0.02381, 0.02444, 0.02442, 0.02457, 0.02446, 0.02491, 0.02474, 0.02468, 0.02463, 0.02469, 0.02618, 0.02458, 0.0243, 0.02465, 0.02436, 0.0246, 0.02381, 0.02431, 0.02492, 0.02438, 0.0239, 0.02778, 0.03263, 0.03015, 0.02489, 0.02497, 0.02827, 0.02851, 0.02831, 0.02923, 0.02893, 0.02474, 0.02501, 0.02434, 0.02523, 0.02437, 0.02557, 0.02446, 0.02462, 0.02479, 0.02496, 0.02454, 0.02469, 0.02509, 0.02486, 0.02485, 0.02426, 0.02434, 0.025, 0.02506, 0.02464, 0.02457, 0.02548, 0.0244, 0.025, 0.02478, 0.0246, 0.025, 0.02481, 0.02465, 0.02469, 0.02502, 0.02443, 0.02451, 0.025, 0.02468, 0.02437, 0.02501, 0.02475, 0.02536, 0.02455, 0.02462, 0.02512, 0.02448, 0.0247, 0.02447, 0.02432, 0.02473, 0.02472, 0.02439, 0.02441, 0.02485, 0.02461, 0.02454, 0.02434, 0.02462, 0.02469, 0.02464, 0.02438, 0.02452, 0.02463, 0.02444, 0.02442, 0.02471, 0.02629, 0.02488, 0.02491, 0.02465, 0.02437, 0.02469, 0.02484, 0.02511, 0.02481, 0.02578, 0.02498, 0.02521, 0.02506, 0.02571, 0.02539, 0.02521, 0.02412, 0.0257, 0.02473, 0.02452, 0.02527, 0.0256, 0.02517, 0.02489, 0.0251, 0.02453, 0.02495, 0.02483, 0.02495, 0.02445, 0.02472, 0.02508, 0.02487, 0.02471, 0.02495, 0.02544, 0.02447, 0.025, 0.02531, 0.02509, 0.02923, 0.02837, 0.02804, 0.02863, 0.03514, 0.02454, 0.02525, 0.02518, 0.02502, 0.02481, 0.02521, 0.02523, 0.02482, 0.02487, 0.02487, 0.02585, 0.02467, 0.02474, 0.02498, 0.02461, 0.02536, 0.02543, 0.02452, 0.02512, 0.02501, 0.02421, 0.02508, 0.02507, 0.02588, 0.02699, 0.02457, 0.02568, 0.0256, 0.02542, 0.02475, 0.02461, 0.02444, 0.0296, 0.02899, 0.02863, 0.02732, 0.02767, 0.02899, 0.02482, 0.02467, 0.02404]}, "backward-send-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00015, 0.00019, 0.00016, 0.00019, 0.00017, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00017, 0.00023, 0.00016, 0.00017, 0.00017, 0.00018, 0.00019, 0.00018, 0.00018, 0.00017, 0.00018, 0.00016, 0.00018, 0.00016, 0.00018, 0.00016, 0.00016, 0.00018, 0.00016, 0.00017, 0.00016, 0.00016, 0.00018, 0.00016, 0.00016, 0.00016, 0.00016, 0.00016, 0.00016, 0.00016, 0.00017, 0.00018, 0.0002, 0.00016, 0.00019, 0.00017, 0.00021, 0.00016, 0.00018, 0.00019, 0.00016, 0.00017, 0.00017, 0.00018, 0.0002, 0.00016, 0.00016, 0.00016, 0.00016, 0.00019, 0.00017, 0.00017, 0.00018, 0.00019, 0.00017, 0.00019, 0.00016, 0.00017, 0.00018, 0.00017, 0.00019, 0.00016, 0.00018, 0.00018, 0.00018, 0.00016, 0.00018, 0.00018, 0.00017, 0.00016, 0.00018, 0.00016, 0.00018, 0.00017, 0.00018, 0.00018, 0.00018, 0.00019, 0.00018, 0.00017, 0.00018, 0.00018, 0.00016, 0.00018, 0.00016, 0.00024, 0.00017, 0.00018, 0.00016, 0.00016, 0.00019, 0.00019, 0.00018, 0.00026, 0.00017, 0.00018, 0.00018, 0.00019, 0.00018, 0.00018, 0.00018, 0.00016, 0.00018, 0.00018, 0.00017, 0.00018, 0.00017, 0.00016, 0.00016, 0.00018, 0.00019, 0.00017, 0.00018, 0.00018, 0.00018, 0.00018, 0.00017, 0.00018, 0.00019, 0.00018, 0.00018, 0.00018, 0.00021, 0.00016, 0.00018, 0.00019, 0.00022, 0.00017, 0.00016, 0.00017, 0.00017, 0.00018, 0.00018, 0.00018, 0.00018, 0.00016, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00016, 0.00019, 0.00018, 0.00018, 0.00017, 0.00019, 0.00017, 0.00019, 0.00018, 0.00018, 0.00016, 0.00017, 0.00016, 0.00016, 0.00018, 0.00017, 0.00016, 0.00029, 0.00017, 0.00019, 0.0002, 0.00016, 0.00019, 0.00032, 0.00019, 0.00016, 0.00018, 0.00018, 0.00016, 0.00018, 0.00023, 0.00018, 0.00018, 0.00018, 0.00017, 0.00019, 0.00018, 0.00016, 0.00018, 0.00017, 0.00018, 0.00018, 0.00018, 0.00019, 0.00018, 0.0002, 0.00016, 0.0002, 0.00018, 0.00018, 0.00018, 0.00016, 0.00018, 0.00016, 0.00016, 0.00018, 0.00016, 0.00017, 0.00019, 0.00018, 0.00016, 0.00019, 0.00022, 0.00016, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00017, 0.00016, 0.00019, 0.00018, 0.00018, 0.00016, 0.00018, 0.00019, 0.00016, 0.00018, 0.00016, 0.00017, 0.00017, 0.00026, 0.00016, 0.00016, 0.00019, 0.00018, 0.00016, 0.00018, 0.00018, 0.00016, 0.00018, 0.00016, 0.00016, 0.00018, 0.00018, 0.00018, 0.00016, 0.00017, 0.00017, 0.00016, 0.00019, 0.00018, 0.00017, 0.00016, 0.00018, 0.00016, 0.00016, 0.00016, 0.00018, 0.00016, 0.00019, 0.00019, 0.00017, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00019, 0.00016, 0.00018, 0.00018, 0.00022, 0.00016, 0.00018, 0.00018, 0.00017, 0.00016, 0.00018, 0.00018, 0.00016, 0.00018, 0.00018, 0.00016, 0.00018, 0.00018, 0.00016, 0.00016, 0.00018, 0.00018, 0.00018, 0.00018, 0.00016, 0.00019, 0.00016, 0.00018, 0.00017, 0.00017, 0.00016, 0.00018, 0.00018, 0.00018, 0.00019, 0.00018, 0.00016, 0.00016, 0.00017, 0.00016, 0.00018, 0.00016, 0.00017, 0.00019, 0.00017, 0.00018, 0.00019, 0.00019, 0.00018, 0.00016, 0.00016, 0.00017, 0.00018, 0.00016, 0.00019, 0.00016, 0.00016, 0.00016, 0.00016, 0.00016, 0.00019, 0.00016, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00018, 0.00016, 0.00016, 0.0002, 0.00018, 0.00018, 0.00019, 0.00019, 0.00018, 0.00018, 0.0003, 0.00016, 0.00018, 0.00018, 0.00016, 0.00019, 0.00018, 0.00019, 0.00016, 0.00016, 0.00016, 0.00018, 0.00019, 0.00018, 0.00018, 0.00017, 0.00018, 0.00019, 0.00017, 0.00016, 0.00018, 0.00017, 0.00018, 0.00018, 0.00017, 0.00018, 0.00016, 0.00016, 0.00018, 0.00019, 0.00017, 0.00018, 0.00018, 0.00017, 0.00016, 0.00035, 0.00022, 0.00019, 0.00018, 0.00018, 0.00017, 0.00016, 0.00017]}, "forward-send-backward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [7.52895, 0.10767, 0.10288, 0.12221, 0.10839, 0.10916, 0.11683, 0.11949, 0.11244, 0.10662, 0.11634, 0.12145, 0.11448, 0.10239, 0.10115, 0.10144, 0.10622, 0.1006, 0.1586, 0.10078, 0.09436, 0.10994, 0.11246, 0.10473, 0.11165, 0.11062, 0.10864, 0.10698, 0.11094, 0.1123, 0.11651, 0.11274, 0.11336, 0.17984, 0.1238, 0.12939, 0.27709, 0.1391, 0.13093, 0.12511, 0.13066, 0.1225, 0.11928, 0.11852, 0.12105, 0.1235, 0.12183, 0.11095, 0.20461, 0.11574, 0.12325, 0.12774, 0.1342, 0.12396, 0.11854, 0.1264, 0.11539, 0.11273, 0.1179, 0.13162, 0.11525, 0.13348, 0.13, 0.12472, 0.13424, 0.1156, 0.11969, 0.21123, 0.12519, 0.12897, 0.136, 0.13444, 0.12965, 0.12283, 0.13807, 0.13035, 0.12784, 0.13095, 0.12328, 0.12278, 0.1242, 0.13846, 0.1251, 0.11622, 0.12258, 0.12174, 0.12831, 0.12841, 0.12632, 0.11745, 0.12732, 0.12029, 0.13155, 0.12567, 0.11834, 0.12549, 0.12416, 0.12349, 0.11452, 0.20614, 0.12415, 0.11944, 0.12148, 0.11366, 0.12373, 0.12834, 0.11722, 0.11892, 0.11557, 0.12715, 0.12886, 0.12057, 0.12682, 0.12601, 0.13364, 0.12815, 0.12626, 0.1317, 0.12917, 0.12301, 0.12818, 0.12239, 0.12231, 0.12391, 0.12264, 0.1209, 0.12986, 0.12429, 0.11971, 0.12228, 0.12907, 0.12399, 0.12889, 0.11751, 0.11734, 0.11985, 0.12419, 0.11939, 0.12896, 0.13183, 0.13356, 0.12001, 0.12131, 0.11604, 0.11794, 0.12429, 0.1355, 0.12631, 0.13817, 0.12757, 0.12565, 0.12479, 0.12459, 0.11863, 0.12603, 0.11965, 0.11957, 0.11941, 0.12277, 0.12152, 0.13238, 0.12899, 0.12039, 0.12936, 0.12185, 0.12027, 0.11834, 0.12565, 0.12003, 0.12064, 0.11734, 0.11796, 0.11982, 0.11829, 0.11018, 0.11427, 0.10291, 0.11078, 0.11775, 0.12251, 0.11736, 0.12288, 0.11757, 0.10965, 0.1101, 0.1111, 0.10524, 0.11035, 0.1194, 0.10687, 0.1104, 0.1029, 0.11414, 0.11835, 0.11073, 0.10671, 0.11471, 0.11713, 0.11142, 0.11427, 0.10551, 0.11576, 0.10811, 0.12352, 0.11089, 0.10827, 0.11418, 0.11243, 0.11291, 0.10774, 0.10575, 0.10895, 0.11133, 0.10168, 0.11589, 0.11188, 0.11403, 0.12083, 0.12527, 0.20209, 0.12301, 0.12835, 0.1167, 0.12035, 0.12158, 0.11749, 0.11785, 0.11663, 0.11859, 0.11189, 0.11229, 0.11518, 0.1205, 0.11283, 0.11679, 0.11705, 0.11627, 0.12181, 0.12372, 0.12191, 0.12006, 0.1168, 0.12252, 0.11718, 0.12814, 0.12688, 0.12696, 0.12607, 0.12079, 0.13508, 0.13166, 0.13101, 0.12769, 0.12321, 0.12875, 0.12726, 0.12271, 0.12496, 0.13106, 0.12712, 0.12831, 0.11758, 0.13314, 0.13148, 0.13269, 0.13383, 0.1235, 0.1316, 0.14168, 0.13684, 0.12388, 0.11908, 0.12703, 0.12329, 0.12975, 0.12484, 0.11743, 0.13142, 0.12276, 0.12584, 0.12278, 0.12351, 0.12006, 0.1275, 0.12997, 0.12275, 0.12374, 0.1258, 0.12674, 0.1382, 0.11985, 0.12902, 0.11699, 0.12694, 0.12671, 0.12528, 0.12577, 0.12335, 0.12793, 0.12913, 0.12309, 0.13132, 0.12457, 0.12253, 0.11803, 0.11645, 0.12181, 0.12507, 0.12528, 0.12214, 0.12812, 0.12471, 0.11918, 0.12456, 0.12769, 0.12304, 0.12153, 0.11907, 0.13148, 0.13103, 0.13068, 0.13318, 0.12552, 0.12933, 0.13261, 0.12839, 0.13023, 0.12205, 0.12863, 0.12765, 0.12548, 0.12592, 0.12495, 0.12574, 0.12193, 0.12065, 0.12433, 0.12257, 0.11243, 0.11188, 0.11552, 0.11773, 0.11637, 0.1131, 0.11535, 0.11323, 0.11728, 0.11383, 0.11656, 0.18458, 0.11533, 0.1158, 0.11306, 0.12884, 0.12649, 0.12032, 0.11208, 0.11803, 0.13436, 0.14069, 0.12596, 0.12808, 0.12036, 0.127, 0.12774, 0.12746, 0.13166, 0.1288, 0.11946, 0.12914, 0.12045, 0.1215, 0.117, 0.11498, 0.11583, 0.11774, 0.12264, 0.12134, 0.12257, 0.12649, 0.1233, 0.12733, 0.11514, 0.12185, 0.12051, 0.13736, 0.13171, 0.13031, 0.11491, 0.11951, 0.10565, 0.11503, 0.1165, 0.11394, 0.11312, 0.11865, 0.11953, 0.12351, 0.12231, 0.12042]}, "backward-send-forward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [3.33774, 0.00722, 0.00727, 0.01025, 0.00728, 0.00714, 0.00814, 0.00897, 0.00966, 0.00746, 0.00801, 0.00911, 0.00716, 0.01132, 0.00906, 0.00969, 0.00832, 0.01171, 0.00765, 0.00889, 0.00886, 0.01056, 0.00822, 0.01186, 0.00789, 0.00921, 0.01483, 0.01149, 0.00732, 0.00899, 0.00802, 0.00967, 0.01211, 0.00836, 0.00778, 0.0097, 0.00744, 0.00738, 0.00799, 0.00783, 0.00895, 0.00733, 0.00808, 0.00821, 0.00953, 0.00947, 0.00803, 0.00716, 0.0083, 0.01092, 0.01169, 0.01197, 0.01099, 0.0139, 0.01319, 0.01223, 0.00743, 0.01124, 0.01269, 0.01365, 0.01106, 0.01186, 0.01247, 0.01377, 0.01372, 0.00895, 0.00817, 0.0122, 0.00886, 0.01409, 0.01218, 0.0116, 0.01184, 0.01054, 0.0083, 0.01112, 0.01398, 0.01443, 0.01304, 0.01159, 0.01508, 0.01227, 0.01243, 0.00996, 0.01336, 0.0103, 0.0121, 0.00939, 0.01351, 0.0109, 0.0119, 0.00743, 0.01152, 0.01082, 0.0077, 0.013, 0.00863, 0.01128, 0.00747, 0.10318, 0.00737, 0.01277, 0.0074, 0.00766, 0.00929, 0.00731, 0.00777, 0.00773, 0.01305, 0.01203, 0.01277, 0.01218, 0.01038, 0.01189, 0.01149, 0.01182, 0.01209, 0.0087, 0.01115, 0.0143, 0.01389, 0.01471, 0.01226, 0.01046, 0.01269, 0.01445, 0.0131, 0.01159, 0.01285, 0.01374, 0.01248, 0.01373, 0.01412, 0.01487, 0.01463, 0.0142, 0.01491, 0.01425, 0.01332, 0.01294, 0.01394, 0.01396, 0.01223, 0.01179, 0.01522, 0.01396, 0.01383, 0.01262, 0.0137, 0.01453, 0.01605, 0.01203, 0.01365, 0.01102, 0.01296, 0.01149, 0.01352, 0.0141, 0.01337, 0.01015, 0.01142, 0.01244, 0.01056, 0.01302, 0.0136, 0.01251, 0.014, 0.01398, 0.01294, 0.01334, 0.01177, 0.01235, 0.01091, 0.01036, 0.01476, 0.01084, 0.01117, 0.01139, 0.01169, 0.01222, 0.01155, 0.0115, 0.01538, 0.01662, 0.01196, 0.01265, 0.01353, 0.0155, 0.01451, 0.01302, 0.01135, 0.01115, 0.01301, 0.01401, 0.01239, 0.01337, 0.0134, 0.01449, 0.01454, 0.01499, 0.02199, 0.01511, 0.01449, 0.01437, 0.01499, 0.01473, 0.01696, 0.01373, 0.01165, 0.01224, 0.01255, 0.01026, 0.01816, 0.01732, 0.01392, 0.01205, 0.01326, 0.012, 0.0125, 0.09407, 0.01373, 0.01234, 0.01352, 0.01298, 0.01393, 0.01293, 0.01272, 0.01269, 0.00988, 0.01398, 0.01371, 0.01512, 0.00926, 0.01203, 0.00886, 0.01072, 0.01094, 0.01129, 0.01236, 0.01167, 0.01127, 0.0134, 0.01164, 0.01227, 0.01086, 0.01128, 0.01424, 0.01338, 0.01286, 0.01139, 0.0124, 0.01253, 0.01306, 0.0104, 0.01044, 0.00925, 0.01349, 0.0106, 0.01304, 0.013, 0.01652, 0.01247, 0.01259, 0.01119, 0.01241, 0.01609, 0.01301, 0.01673, 0.01245, 0.01358, 0.01293, 0.01395, 0.01222, 0.01281, 0.01194, 0.01332, 0.01097, 0.01369, 0.01398, 0.0117, 0.01357, 0.0128, 0.01277, 0.01159, 0.01226, 0.01271, 0.0131, 0.01357, 0.0123, 0.01025, 0.01114, 0.01335, 0.01274, 0.00948, 0.01342, 0.01348, 0.01171, 0.01274, 0.01313, 0.01262, 0.01167, 0.00993, 0.01158, 0.0107, 0.01309, 0.01347, 0.015, 0.01426, 0.01127, 0.01224, 0.0128, 0.01251, 0.01492, 0.01369, 0.01553, 0.01256, 0.01398, 0.01419, 0.01663, 0.01442, 0.01314, 0.01126, 0.01132, 0.01161, 0.01215, 0.01208, 0.01721, 0.01103, 0.01311, 0.00802, 0.01029, 0.01351, 0.00888, 0.01039, 0.00882, 0.00933, 0.00881, 0.00926, 0.01082, 0.01021, 0.00961, 0.01001, 0.00836, 0.00918, 0.01044, 0.01016, 0.00966, 0.00991, 0.01218, 0.07892, 0.00899, 0.01009, 0.01201, 0.00867, 0.01068, 0.01049, 0.01158, 0.01334, 0.0109, 0.01304, 0.00961, 0.01538, 0.01469, 0.01646, 0.00905, 0.01059, 0.01386, 0.01332, 0.01461, 0.01223, 0.01253, 0.0166, 0.01015, 0.01471, 0.01602, 0.01097, 0.01225, 0.01068, 0.01085, 0.01135, 0.00802, 0.00878, 0.01148, 0.01009, 0.00941, 0.00919, 0.01177, 0.00968, 0.01046, 0.00955, 0.01107, 0.00923, 0.00916, 0.00864, 0.01069, 0.01075, 0.00939, 0.01202, 0.00876, 0.01073]}, "layernorm-grads-all-reduce-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.0012, 0.00075, 0.00074, 0.00352, 0.00166, 0.00076, 0.00077, 0.00076, 0.00319, 0.00077, 0.00076, 0.00445, 0.00077, 0.00075, 0.00153, 0.00077, 0.00076, 0.00076, 0.00076, 0.00077, 0.00076, 0.00075, 0.00076, 0.00075, 0.00077, 0.00075, 0.00077, 0.00075, 0.00077, 0.00077, 0.00075, 0.00076, 0.00076, 0.00076, 0.00076, 0.00076, 0.00077, 0.00076, 0.00076, 0.00077, 0.00078, 0.00076, 0.00077, 0.00076, 0.00076, 0.00429, 0.00076, 0.00076, 0.00076, 0.00078, 0.00077, 0.00077, 0.00078, 0.00077, 0.0008, 0.00079, 0.00079, 0.00077, 0.00078, 0.00078, 0.00079, 0.00519, 0.00079, 0.00078, 0.00077, 0.00078, 0.00079, 0.00079, 0.00079, 0.00077, 0.00079, 0.00079, 0.00079, 0.00078, 0.00078, 0.00078, 0.00077, 0.00079, 0.00079, 0.00079, 0.00078, 0.00078, 0.00077, 0.00078, 0.00077, 0.00078, 0.00078, 0.00078, 0.00078, 0.00078, 0.00083, 0.00306, 0.00078, 0.00076, 0.00078, 0.00078, 0.00078, 0.00077, 0.00078, 0.0008, 0.00079, 0.00079, 0.00077, 0.00079, 0.00078, 0.00078, 0.00081, 0.00335, 0.00078, 0.00079, 0.0008, 0.00078, 0.00079, 0.00079, 0.00078, 0.00077, 0.00079, 0.00078, 0.00079, 0.0008, 0.00078, 0.00078, 0.00078, 0.00078, 0.00077, 0.00078, 0.00078, 0.00078, 0.00078, 0.00077, 0.00077, 0.00078, 0.00078, 0.00077, 0.00077, 0.00078, 0.00077, 0.00078, 0.00077, 0.00078, 0.00077, 0.00077, 0.00077, 0.00079, 0.00078, 0.00078, 0.00077, 0.00078, 0.00078, 0.00078, 0.00078, 0.00077, 0.00077, 0.00078, 0.00078, 0.00079, 0.00086, 0.00079, 0.00078, 0.00079, 0.00078, 0.00078, 0.00078, 0.00078, 0.00078, 0.00078, 0.00078, 0.00077, 0.0008, 0.0008, 0.00079, 0.00078, 0.00079, 0.00078, 0.00078, 0.00082, 0.00081, 0.00083, 0.00078, 0.00077, 0.00079, 0.00082, 0.0008, 0.00077, 0.00076, 0.00077, 0.00078, 0.00077, 0.00078, 0.00077, 0.00077, 0.00077, 0.00078, 0.00077, 0.00077, 0.00077, 0.00077, 0.00078, 0.00078, 0.00077, 0.00077, 0.00078, 0.00082, 0.00083, 0.00078, 0.00078, 0.00078, 0.00077, 0.00078, 0.00078, 0.00077, 0.00078, 0.00079, 0.00078, 0.00452, 0.00077, 0.00078, 0.00077, 0.00077, 0.0008, 0.00078, 0.00079, 0.00079, 0.00078, 0.00223, 0.00078, 0.00077, 0.00077, 0.00079, 0.00078, 0.00078, 0.00078, 0.00295, 0.00077, 0.00077, 0.00077, 0.00077, 0.00077, 0.00076, 0.00077, 0.0042, 0.00081, 0.00079, 0.00087, 0.00078, 0.00078, 0.00078, 0.00078, 0.00076, 0.00078, 0.0008, 0.00076, 0.00079, 0.00077, 0.00078, 0.00077, 0.00077, 0.00077, 0.00078, 0.00078, 0.00077, 0.00078, 0.00077, 0.00077, 0.00078, 0.00077, 0.00078, 0.00078, 0.00078, 0.00078, 0.00076, 0.00076, 0.00077, 0.00077, 0.00077, 0.00077, 0.00078, 0.00079, 0.00085, 0.00078, 0.00078, 0.00077, 0.00079, 0.00079, 0.00079, 0.00077, 0.00078, 0.00078, 0.00078, 0.00078, 0.00078, 0.00077, 0.00078, 0.00078, 0.00077, 0.00078, 0.00077, 0.00077, 0.00077, 0.00079, 0.00079, 0.00078, 0.00078, 0.00077, 0.00078, 0.00078, 0.00078, 0.00079, 0.00077, 0.00078, 0.00078, 0.00077, 0.00077, 0.00078, 0.00077, 0.00077, 0.00079, 0.00079, 0.00077, 0.00077, 0.00078, 0.00077, 0.00078, 0.00078, 0.00078, 0.00079, 0.00078, 0.00078, 0.00077, 0.00078, 0.00078, 0.00077, 0.00079, 0.00078, 0.00077, 0.00079, 0.00078, 0.00078, 0.00077, 0.00077, 0.0008, 0.00078, 0.00078, 0.00079, 0.00077, 0.00079, 0.00077, 0.00077, 0.00077, 0.00079, 0.00078, 0.00078, 0.00078, 0.00083, 0.0009, 0.00079, 0.00082, 0.0008, 0.0008, 0.00078, 0.00077, 0.00077, 0.00078, 0.00078, 0.00079, 0.00078, 0.00077, 0.00078, 0.00078, 0.00078, 0.00078, 0.00078, 0.00077, 0.0008, 0.00079, 0.00078, 0.00078, 0.00078, 0.00078, 0.00077, 0.00078, 0.00078, 0.00078, 0.00078, 0.00078, 0.00078, 0.00077, 0.00084, 0.00077, 0.00077, 0.00077, 0.0008, 0.00078, 0.00078, 0.00077, 0.00078, 0.00153, 0.00078, 0.00078, 0.00076]}, "embedding-grads-all-reduce-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00036, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00033, 0.00032, 0.00032, 0.00031, 0.00032, 0.00034, 0.00032, 0.00031, 0.00037, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00034, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00034, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00033, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00033, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00031, 0.00032, 0.00032, 0.00032, 0.00031, 0.00034, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00031, 0.00032, 0.00031, 0.00031, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00034, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00034, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00031, 0.00032, 0.00031, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00031, 0.00032, 0.00031, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00031, 0.00031, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00033, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00032, 0.00032, 0.00031, 0.00032, 0.00031, 0.00031, 0.00031, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00032, 0.00031]}, "all-grads-sync-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.22391, 0.00071, 0.00073, 0.0009, 0.00073, 0.00075, 0.00074, 0.00093, 0.00097, 0.00072, 0.00071, 0.00084, 0.00088, 0.00075, 0.00086, 0.00072, 0.00072, 0.00071, 0.00072, 0.00073, 0.00072, 0.00072, 0.00073, 0.00073, 0.00072, 0.00072, 0.00072, 0.00072, 0.00071, 0.0007, 0.00072, 0.00071, 0.00072, 0.00072, 0.00071, 0.00071, 0.00074, 0.00072, 0.00074, 0.00073, 0.00073, 0.00075, 0.00074, 0.00072, 0.00072, 0.00073, 0.0009, 0.00081, 0.00071, 0.00073, 0.00073, 0.00071, 0.00074, 0.00084, 0.00072, 0.00072, 0.00083, 0.00072, 0.00073, 0.00072, 0.0009, 0.00072, 0.00072, 0.00072, 0.00074, 0.00072, 0.00073, 0.00073, 0.00073, 0.00072, 0.00074, 0.00075, 0.00072, 0.00073, 0.00073, 0.00072, 0.00073, 0.00074, 0.00073, 0.00072, 0.00073, 0.00074, 0.00073, 0.00074, 0.00073, 0.00073, 0.00073, 0.00072, 0.00072, 0.00071, 0.00074, 0.00093, 0.00074, 0.00072, 0.00072, 0.00072, 0.00072, 0.00069, 0.00084, 0.00071, 0.00073, 0.00073, 0.0008, 0.00086, 0.00098, 0.00092, 0.00099, 0.00087, 0.00096, 0.00093, 0.00073, 0.00074, 0.00072, 0.00072, 0.00072, 0.00074, 0.00072, 0.00072, 0.00072, 0.00073, 0.00073, 0.00073, 0.00072, 0.00073, 0.00072, 0.00073, 0.00073, 0.00072, 0.00073, 0.00077, 0.00075, 0.00074, 0.00087, 0.00072, 0.00073, 0.00072, 0.00073, 0.00082, 0.00081, 0.00074, 0.00074, 0.00073, 0.00072, 0.00072, 0.00074, 0.00073, 0.00071, 0.00075, 0.00076, 0.00072, 0.00085, 0.00072, 0.00073, 0.00072, 0.00074, 0.00082, 0.00097, 0.00073, 0.00072, 0.00072, 0.00073, 0.00073, 0.00073, 0.00072, 0.00072, 0.00073, 0.00073, 0.00073, 0.00077, 0.00072, 0.00073, 0.00086, 0.00087, 0.00073, 0.00093, 0.00084, 0.00097, 0.00089, 0.00074, 0.00074, 0.00087, 0.00093, 0.00087, 0.00073, 0.00072, 0.00074, 0.00072, 0.00074, 0.00074, 0.00074, 0.00073, 0.00072, 0.00093, 0.00074, 0.00073, 0.00075, 0.00085, 0.00073, 0.00072, 0.00072, 0.00073, 0.00092, 0.00074, 0.00088, 0.00073, 0.00074, 0.00073, 0.00073, 0.00072, 0.00072, 0.00075, 0.00073, 0.00072, 0.00081, 0.00073, 0.00073, 0.00071, 0.00072, 0.00071, 0.00071, 0.00072, 0.00074, 0.00072, 0.00073, 0.00093, 0.00072, 0.00074, 0.00072, 0.00073, 0.00071, 0.00074, 0.00074, 0.00087, 0.00086, 0.00072, 0.00072, 0.00074, 0.00072, 0.00074, 0.00072, 0.00079, 0.00095, 0.00083, 0.00071, 0.00093, 0.00088, 0.00072, 0.00072, 0.00073, 0.00071, 0.00075, 0.00091, 0.00072, 0.00071, 0.00072, 0.00073, 0.0007, 0.00072, 0.00074, 0.00072, 0.00074, 0.00073, 0.00075, 0.00073, 0.00073, 0.00072, 0.00073, 0.00073, 0.00071, 0.00074, 0.00072, 0.00071, 0.00071, 0.00073, 0.00072, 0.00073, 0.00073, 0.00071, 0.00074, 0.00072, 0.00073, 0.00073, 0.0007, 0.00072, 0.00072, 0.00072, 0.00073, 0.00074, 0.00072, 0.00074, 0.00073, 0.00073, 0.00074, 0.0007, 0.00072, 0.00072, 0.00073, 0.00074, 0.00071, 0.00073, 0.00072, 0.00071, 0.00073, 0.00071, 0.00073, 0.00072, 0.00074, 0.00071, 0.00073, 0.00071, 0.00073, 0.00073, 0.00071, 0.0007, 0.00072, 0.00072, 0.00073, 0.00072, 0.00071, 0.00072, 0.00073, 0.00074, 0.00071, 0.00074, 0.00071, 0.00073, 0.00072, 0.00073, 0.00073, 0.00071, 0.00073, 0.00072, 0.00073, 0.00074, 0.00074, 0.00071, 0.00072, 0.00072, 0.00074, 0.00072, 0.00073, 0.00072, 0.00074, 0.00072, 0.00073, 0.00073, 0.00073, 0.00073, 0.00074, 0.00074, 0.00075, 0.00072, 0.00073, 0.00097, 0.00103, 0.00091, 0.00097, 0.00092, 0.00088, 0.00072, 0.00071, 0.00073, 0.00074, 0.00073, 0.00075, 0.0007, 0.00072, 0.00072, 0.00072, 0.00071, 0.00073, 0.00072, 0.00074, 0.00072, 0.00073, 0.00074, 0.00073, 0.00074, 0.00073, 0.00072, 0.00073, 0.00074, 0.00074, 0.00072, 0.00075, 0.0007, 0.00072, 0.00076, 0.00073, 0.00072, 0.00072, 0.00094, 0.00082, 0.00087, 0.00071, 0.00071, 0.00096, 0.00083, 0.00089, 0.00089]}, "params-all-gather-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00024, 0.00025, 0.00024, 0.00043, 0.00027, 0.00024, 0.00024, 0.00024, 0.00035, 0.00024, 0.00024, 0.0004, 0.00025, 0.00024, 0.0003, 0.00025, 0.00024, 0.00024, 0.00024, 0.00025, 0.00024, 0.00025, 0.00024, 0.00024, 0.00024, 0.00024, 0.00024, 0.00024, 0.00024, 0.00024, 0.00024, 0.00024, 0.00025, 0.00025, 0.00026, 0.00024, 0.00024, 0.00025, 0.00024, 0.00025, 0.00025, 0.00024, 0.00024, 0.00024, 0.00024, 0.0003, 0.00025, 0.00025, 0.00025, 0.00025, 0.00042, 0.00025, 0.00027, 0.00025, 0.00048, 0.00025, 0.00026, 0.00025, 0.00025, 0.00026, 0.00026, 0.00056, 0.00026, 0.00043, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00026, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00024, 0.00025, 0.00025, 0.00024, 0.00025, 0.00025, 0.00033, 0.00025, 0.00026, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00026, 0.00025, 0.00025, 0.00025, 0.00025, 0.00028, 0.00043, 0.00026, 0.00034, 0.0003, 0.00025, 0.0003, 0.00024, 0.00025, 0.00026, 0.00026, 0.00024, 0.00025, 0.00024, 0.00025, 0.00025, 0.00024, 0.00025, 0.00025, 0.00024, 0.00026, 0.00024, 0.00024, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00025, 0.00024, 0.00024, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00024, 0.00025, 0.00026, 0.00024, 0.00024, 0.00025, 0.00028, 0.00025, 0.00025, 0.00025, 0.00025, 0.00028, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00026, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00027, 0.00025, 0.00025, 0.00026, 0.00026, 0.00027, 0.00025, 0.00026, 0.00025, 0.00026, 0.00046, 0.00025, 0.00025, 0.00025, 0.00025, 0.00045, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00025, 0.00024, 0.00027, 0.00025, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00025, 0.00025, 0.00024, 0.00024, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00026, 0.00024, 0.00025, 0.00025, 0.00025, 0.00025, 0.00024, 0.00024, 0.00043, 0.00024, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00025, 0.00025, 0.00025, 0.00032, 0.0005, 0.00025, 0.00024, 0.0005, 0.00038, 0.00024, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00042, 0.00025, 0.0004, 0.00026, 0.00025, 0.00025, 0.00025, 0.00025, 0.00026, 0.00025, 0.00027, 0.00025, 0.00026, 0.00025, 0.00026, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00026, 0.00025, 0.00026, 0.00026, 0.00025, 0.00025, 0.00025, 0.00025, 0.00024, 0.00024, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00039, 0.00029, 0.00026, 0.00025, 0.00025, 0.00033, 0.00025, 0.00025, 0.00026, 0.00026, 0.00027, 0.00033, 0.00025, 0.00024, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00026, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00024, 0.00024, 0.00025, 0.00025, 0.00024, 0.00024, 0.00024, 0.00026, 0.00024, 0.00025, 0.00025, 0.00025, 0.00025, 0.00024, 0.00024, 0.00024, 0.00024, 0.00025, 0.00025, 0.00044, 0.00044, 0.00046, 0.00041, 0.00047, 0.00026, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00025, 0.00026, 0.00024, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00025, 0.00024, 0.00025, 0.00025, 0.00026, 0.00025, 0.00026, 0.00025, 0.00025, 0.00026, 0.00025, 0.00025, 0.00024, 0.00043, 0.00026, 0.00053, 0.00025, 0.00026, 0.00025, 0.00028, 0.00042, 0.00025, 0.00025]}, "optimizer-copy-to-main-grad-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00041, 0.00039, 0.00039, 0.00041, 0.00042, 0.0004, 0.00041, 0.0004, 0.0004, 0.0004, 0.0004, 0.00054, 0.0004, 0.0004, 0.00056, 0.00042, 0.0004, 0.00041, 0.00041, 0.00041, 0.00041, 0.00041, 0.00042, 0.0004, 0.0004, 0.00041, 0.00041, 0.00041, 0.0004, 0.00041, 0.0004, 0.00041, 0.00041, 0.00041, 0.0004, 0.00041, 0.00042, 0.00041, 0.00042, 0.00041, 0.00042, 0.00042, 0.0004, 0.00041, 0.00042, 0.00042, 0.0004, 0.00041, 0.00043, 0.00041, 0.00042, 0.00041, 0.00042, 0.00042, 0.00043, 0.00042, 0.00042, 0.00042, 0.00043, 0.00042, 0.00041, 0.00041, 0.00042, 0.00042, 0.00043, 0.00042, 0.00043, 0.00042, 0.00042, 0.00041, 0.00042, 0.00042, 0.00042, 0.00041, 0.00042, 0.00042, 0.00042, 0.00043, 0.00043, 0.00043, 0.0004, 0.00041, 0.00043, 0.00042, 0.00042, 0.00043, 0.00042, 0.00043, 0.00042, 0.00042, 0.00048, 0.00042, 0.00041, 0.00042, 0.00042, 0.00042, 0.00043, 0.00044, 0.00042, 0.00042, 0.00041, 0.0004, 0.00041, 0.00041, 0.00041, 0.00041, 0.00041, 0.00042, 0.00042, 0.00038, 0.0004, 0.00043, 0.00041, 0.00043, 0.00041, 0.0004, 0.0004, 0.0004, 0.00041, 0.00042, 0.00041, 0.00042, 0.00041, 0.00041, 0.0004, 0.00041, 0.00041, 0.00041, 0.00041, 0.00041, 0.00041, 0.00041, 0.00041, 0.00043, 0.00043, 0.00041, 0.00041, 0.00041, 0.00041, 0.00042, 0.00038, 0.0004, 0.00039, 0.00041, 0.00042, 0.00043, 0.00038, 0.00038, 0.0004, 0.00042, 0.0004, 0.0004, 0.0004, 0.00041, 0.00041, 0.0004, 0.00045, 0.00041, 0.00041, 0.0004, 0.00043, 0.00042, 0.00042, 0.00042, 0.00041, 0.00042, 0.00042, 0.00042, 0.00041, 0.00041, 0.00041, 0.00041, 0.0004, 0.00041, 0.00041, 0.0004, 0.00041, 0.00041, 0.0004, 0.00041, 0.00041, 0.0004, 0.00041, 0.00041, 0.0004, 0.00041, 0.00042, 0.00041, 0.0004, 0.00041, 0.00042, 0.00041, 0.00041, 0.0004, 0.00041, 0.0004, 0.00041, 0.00043, 0.0004, 0.00042, 0.00042, 0.00043, 0.00041, 0.00041, 0.00041, 0.00041, 0.00041, 0.00043, 0.00042, 0.00041, 0.00038, 0.00042, 0.00041, 0.00041, 0.00041, 0.00042, 0.00041, 0.00041, 0.00042, 0.00041, 0.0004, 0.00041, 0.00041, 0.00041, 0.00041, 0.00041, 0.00041, 0.00042, 0.00043, 0.00042, 0.00041, 0.00041, 0.00041, 0.00041, 0.00041, 0.00043, 0.00041, 0.0004, 0.00042, 0.00041, 0.00041, 0.00042, 0.00041, 0.00041, 0.00042, 0.00042, 0.0004, 0.00041, 0.00041, 0.00041, 0.00046, 0.00043, 0.00043, 0.00042, 0.00042, 0.00042, 0.00042, 0.00043, 0.00042, 0.00041, 0.00043, 0.00043, 0.00039, 0.00043, 0.00042, 0.00042, 0.00043, 0.00042, 0.00042, 0.00042, 0.00043, 0.0004, 0.00042, 0.0004, 0.00043, 0.00041, 0.00042, 0.00042, 0.00043, 0.00041, 0.00041, 0.00041, 0.00042, 0.00042, 0.00042, 0.00041, 0.00043, 0.00042, 0.0004, 0.00043, 0.00041, 0.00042, 0.00041, 0.00041, 0.00043, 0.00042, 0.00042, 0.00043, 0.00042, 0.00042, 0.00041, 0.00041, 0.00041, 0.00041, 0.00041, 0.00042, 0.00041, 0.00042, 0.00042, 0.00042, 0.00042, 0.00042, 0.00042, 0.00043, 0.00041, 0.00042, 0.00042, 0.00043, 0.00044, 0.00043, 0.00041, 0.00041, 0.00042, 0.00042, 0.00041, 0.00043, 0.00041, 0.00042, 0.00041, 0.00042, 0.00041, 0.00039, 0.00041, 0.00042, 0.00042, 0.00041, 0.00042, 0.00042, 0.00042, 0.00042, 0.00041, 0.00041, 0.00042, 0.00041, 0.00041, 0.00043, 0.00042, 0.00042, 0.00042, 0.00041, 0.00041, 0.00042, 0.00043, 0.00041, 0.00041, 0.00041, 0.00042, 0.00043, 0.00042, 0.00042, 0.00044, 0.00043, 0.00042, 0.00041, 0.00042, 0.00041, 0.00043, 0.00041, 0.00044, 0.0004, 0.00042, 0.00042, 0.00041, 0.00042, 0.00042, 0.00043, 0.00042, 0.00041, 0.00041, 0.00041, 0.00042, 0.00041, 0.00041, 0.00042, 0.00041, 0.0004, 0.00041, 0.00041, 0.00041, 0.00042, 0.00041, 0.0004, 0.00052, 0.00042, 0.00042, 0.00042, 0.0004, 0.00042, 0.00041, 0.00041]}, "optimizer-clip-main-grad-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.02442, 0.00047, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00044, 0.00046, 0.00045, 0.00046, 0.00069, 0.00045, 0.00045, 0.00046, 0.00046, 0.00045, 0.00045, 0.00046, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00044, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.0005, 0.00046, 0.00045, 0.00044, 0.00047, 0.00046, 0.00045, 0.00053, 0.00046, 0.00045, 0.00045, 0.00045, 0.00046, 0.00044, 0.00045, 0.00046, 0.00045, 0.00047, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00044, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00052, 0.00045, 0.00047, 0.00046, 0.00039, 0.00045, 0.00045, 0.00046, 0.00046, 0.00046, 0.00045, 0.00045, 0.00045, 0.00044, 0.0004, 0.00046, 0.00044, 0.0004, 0.00046, 0.00044, 0.0004, 0.0004, 0.0004, 0.00041, 0.00047, 0.00046, 0.0004, 0.00046, 0.00045, 0.00045, 0.00039, 0.00045, 0.00047, 0.00045, 0.0004, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00047, 0.00044, 0.00045, 0.00044, 0.00045, 0.00045, 0.00046, 0.00044, 0.00044, 0.00045, 0.00045, 0.00045, 0.00047, 0.00045, 0.00046, 0.00045, 0.00045, 0.00049, 0.00046, 0.00046, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00046, 0.00046, 0.00047, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00047, 0.00048, 0.00047, 0.00046, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00047, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00047, 0.00045, 0.00046, 0.00046, 0.00045, 0.00045, 0.00046, 0.00046, 0.00058, 0.00047, 0.00044, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00046, 0.00045, 0.00054, 0.00045, 0.00045, 0.00045, 0.00046, 0.00044, 0.00045, 0.00047, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00051, 0.00046, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00044, 0.00046, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00045, 0.00048, 0.00045, 0.00045, 0.00044, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00046, 0.00048, 0.00044, 0.00044, 0.00045, 0.00045, 0.00044, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00047, 0.00045, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00045, 0.00046, 0.00046, 0.00046, 0.00046, 0.00045, 0.00046, 0.00047, 0.00045, 0.00057, 0.00046, 0.00045, 0.00045, 0.00045, 0.00046, 0.00047, 0.00047, 0.00045, 0.00046, 0.00045, 0.00045, 0.00044, 0.00046, 0.00046, 0.00045, 0.00045, 0.00047, 0.00047, 0.00045, 0.00045, 0.00045, 0.00046, 0.00045, 0.00046, 0.00045, 0.00046, 0.00045, 0.00045, 0.00046, 0.00045, 0.00046, 0.00045, 0.00045, 0.00046, 0.00059, 0.00045, 0.00047, 0.00045, 0.00046, 0.00045, 0.00045, 0.00045]}, "optimizer-count-zeros-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00264, 0.00186, 0.00189, 0.00186, 0.00191, 0.00186, 0.00187, 0.00189, 0.0019, 0.00189, 0.00189, 0.002, 0.00187, 0.00201, 0.0019, 0.00186, 0.00187, 0.00185, 0.00187, 0.00187, 0.00186, 0.00186, 0.00187, 0.00186, 0.00187, 0.00189, 0.00189, 0.00185, 0.00188, 0.00186, 0.00187, 0.00188, 0.00188, 0.00186, 0.00188, 0.00187, 0.00189, 0.00185, 0.00189, 0.00189, 0.00187, 0.00186, 0.00186, 0.00189, 0.00188, 0.00186, 0.00186, 0.0019, 0.00186, 0.00187, 0.00188, 0.00186, 0.00213, 0.00189, 0.00185, 0.00186, 0.00188, 0.00189, 0.00186, 0.00185, 0.00187, 0.00186, 0.00186, 0.00186, 0.00186, 0.00186, 0.00185, 0.00186, 0.00187, 0.00186, 0.00186, 0.00189, 0.00188, 0.0019, 0.00189, 0.00187, 0.00187, 0.00188, 0.00186, 0.00187, 0.00187, 0.00188, 0.00186, 0.00186, 0.00186, 0.00185, 0.00186, 0.00186, 0.00187, 0.00186, 0.00217, 0.0019, 0.00195, 0.00188, 0.00187, 0.00188, 0.00188, 0.00186, 0.00188, 0.00186, 0.00188, 0.00188, 0.00186, 0.00187, 0.00188, 0.00185, 0.00208, 0.00187, 0.00187, 0.00186, 0.00185, 0.00185, 0.00188, 0.00185, 0.00186, 0.00186, 0.00186, 0.00186, 0.00186, 0.00186, 0.00187, 0.00185, 0.00185, 0.00188, 0.00186, 0.00185, 0.00188, 0.00186, 0.00186, 0.00184, 0.00187, 0.00186, 0.00189, 0.00186, 0.00185, 0.0019, 0.00187, 0.00186, 0.00186, 0.00186, 0.00186, 0.00186, 0.00189, 0.00187, 0.0019, 0.00186, 0.00186, 0.00187, 0.00188, 0.00185, 0.00186, 0.00186, 0.00189, 0.00186, 0.00187, 0.00187, 0.00203, 0.00186, 0.00186, 0.00188, 0.00187, 0.00186, 0.00188, 0.00184, 0.00185, 0.00186, 0.00187, 0.00185, 0.00186, 0.00187, 0.00188, 0.00198, 0.00198, 0.00186, 0.00185, 0.00187, 0.00188, 0.00186, 0.00188, 0.00185, 0.00185, 0.00187, 0.00187, 0.00186, 0.00185, 0.00185, 0.00187, 0.00186, 0.00186, 0.00187, 0.00187, 0.00185, 0.00187, 0.00187, 0.00186, 0.00185, 0.00186, 0.00187, 0.00188, 0.00191, 0.00186, 0.00188, 0.00188, 0.00187, 0.00188, 0.00187, 0.00188, 0.00186, 0.00187, 0.0019, 0.00187, 0.00187, 0.00186, 0.00187, 0.00187, 0.00186, 0.0019, 0.00188, 0.00187, 0.0019, 0.0019, 0.00191, 0.00191, 0.00186, 0.00187, 0.00188, 0.00187, 0.00186, 0.00188, 0.00188, 0.00189, 0.00189, 0.00188, 0.00188, 0.00189, 0.00189, 0.00189, 0.00186, 0.00191, 0.00189, 0.00187, 0.00186, 0.0019, 0.00188, 0.00188, 0.00187, 0.00188, 0.0019, 0.00189, 0.0019, 0.00219, 0.00189, 0.0019, 0.00187, 0.00188, 0.00187, 0.00187, 0.00188, 0.00188, 0.00187, 0.00186, 0.00189, 0.00188, 0.00188, 0.00188, 0.00188, 0.00188, 0.00189, 0.00188, 0.00216, 0.00188, 0.00189, 0.00188, 0.00189, 0.00189, 0.00189, 0.00187, 0.00187, 0.00188, 0.00188, 0.00199, 0.00187, 0.00201, 0.00189, 0.00187, 0.00191, 0.00189, 0.00187, 0.00188, 0.00188, 0.00189, 0.00246, 0.00272, 0.00189, 0.00189, 0.00189, 0.00288, 0.00189, 0.00187, 0.00189, 0.00189, 0.0019, 0.0019, 0.00188, 0.0019, 0.0019, 0.00191, 0.0019, 0.0019, 0.0019, 0.00191, 0.00191, 0.00189, 0.00189, 0.0019, 0.0019, 0.00189, 0.00188, 0.00188, 0.0019, 0.00197, 0.00187, 0.00189, 0.00188, 0.00189, 0.00187, 0.0019, 0.00187, 0.00189, 0.00188, 0.00189, 0.00188, 0.00187, 0.00187, 0.00188, 0.0019, 0.00187, 0.00188, 0.00188, 0.00188, 0.00191, 0.00216, 0.00186, 0.00188, 0.00189, 0.00189, 0.00187, 0.00189, 0.0019, 0.00187, 0.00189, 0.00187, 0.00199, 0.00189, 0.00188, 0.00187, 0.00187, 0.00188, 0.00189, 0.00188, 0.00188, 0.00188, 0.00188, 0.00187, 0.00188, 0.00188, 0.00188, 0.00189, 0.00188, 0.00188, 0.0019, 0.00187, 0.00189, 0.00189, 0.00188, 0.00189, 0.00188, 0.00188, 0.00188, 0.00189, 0.00186, 0.00189, 0.00187, 0.00189, 0.0019, 0.0019, 0.00194, 0.00189, 0.00187, 0.00187, 0.00189, 0.00189, 0.002, 0.00187, 0.00187, 0.00189, 0.00187, 0.00188, 0.00189, 0.00195]}, "optimizer-inner-step-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00219, 0.00036, 0.00035, 0.00037, 0.00037, 0.00039, 0.00038, 0.00037, 0.00037, 0.00038, 0.00037, 0.0004, 0.00038, 0.00038, 0.00047, 0.00037, 0.00038, 0.00038, 0.00037, 0.00037, 0.00037, 0.00039, 0.00038, 0.00037, 0.00039, 0.00037, 0.00038, 0.00038, 0.00037, 0.00037, 0.00037, 0.00038, 0.00038, 0.00038, 0.00037, 0.00037, 0.00038, 0.00039, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00037, 0.00038, 0.00037, 0.00038, 0.00037, 0.00039, 0.00039, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00037, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00038, 0.00037, 0.00038, 0.0004, 0.00039, 0.0004, 0.00038, 0.00039, 0.00039, 0.00039, 0.00039, 0.00038, 0.00038, 0.00037, 0.00038, 0.00039, 0.00039, 0.00038, 0.00038, 0.00039, 0.00039, 0.00038, 0.00038, 0.00044, 0.00039, 0.0004, 0.00039, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00039, 0.00039, 0.00038, 0.00038, 0.00039, 0.00038, 0.00039, 0.00038, 0.00038, 0.00039, 0.00037, 0.00039, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00039, 0.0004, 0.00038, 0.00038, 0.00039, 0.00039, 0.0004, 0.00039, 0.00038, 0.00038, 0.00039, 0.00039, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00039, 0.00039, 0.00038, 0.00039, 0.00039, 0.00037, 0.00039, 0.00037, 0.00038, 0.00041, 0.00037, 0.00037, 0.00038, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00038, 0.00039, 0.00038, 0.0004, 0.00038, 0.00038, 0.00038, 0.00039, 0.00039, 0.00038, 0.00038, 0.0004, 0.00038, 0.0004, 0.00038, 0.00038, 0.0004, 0.00038, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00039, 0.00038, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00038, 0.0004, 0.00038, 0.00038, 0.00038, 0.00038, 0.00037, 0.00038, 0.00039, 0.00038, 0.00038, 0.00039, 0.00039, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00039, 0.00038, 0.00038, 0.00039, 0.00039, 0.00038, 0.00039, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00037, 0.00038, 0.00038, 0.00037, 0.00038, 0.00038, 0.00037, 0.00037, 0.00039, 0.00038, 0.00038, 0.00037, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00038, 0.00038, 0.00039, 0.00039, 0.00038, 0.00037, 0.00037, 0.00038, 0.00038, 0.00043, 0.00037, 0.00038, 0.00038, 0.00037, 0.00038, 0.00038, 0.00038, 0.00038, 0.00037, 0.00038, 0.00037, 0.00037, 0.00038, 0.00037, 0.00039, 0.00037, 0.00037, 0.00039, 0.00038, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00038, 0.0004, 0.0004, 0.00038, 0.00039, 0.00038, 0.00038, 0.00039, 0.0004, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00038, 0.00038, 0.00039, 0.00039, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00038, 0.00039, 0.00038, 0.00037, 0.00038, 0.00039, 0.00039, 0.00038, 0.00037, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00039, 0.00039, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00037, 0.00039, 0.00038, 0.00038, 0.00038, 0.00039, 0.0004, 0.00039, 0.00038, 0.00038, 0.00041, 0.0004, 0.00039, 0.00038, 0.0004, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00039, 0.00039, 0.00039, 0.00038, 0.00038, 0.00039, 0.00038, 0.00038, 0.0004, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00038, 0.00038, 0.00039, 0.00038, 0.00038, 0.00038, 0.00039, 0.00039, 0.00038, 0.00039, 0.00038, 0.00039, 0.00038, 0.00039, 0.00038, 0.00041, 0.00039, 0.00039, 0.00041, 0.00038, 0.00038, 0.00052, 0.00038, 0.00039, 0.00038, 0.00038, 0.00038, 0.00038, 0.00038]}, "optimizer-copy-main-to-model-params-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00097, 0.00085, 0.00083, 0.00104, 0.00084, 0.00083, 0.00084, 0.00085, 0.00085, 0.00084, 0.00083, 0.00085, 0.00083, 0.00085, 0.00178, 0.00084, 0.00084, 0.00084, 0.00084, 0.00084, 0.00085, 0.00085, 0.00085, 0.00083, 0.00082, 0.00083, 0.00084, 0.00083, 0.00084, 0.00084, 0.00083, 0.00084, 0.00083, 0.00086, 0.00085, 0.00085, 0.00084, 0.00084, 0.00085, 0.00085, 0.00084, 0.00084, 0.00084, 0.00085, 0.00085, 0.00084, 0.00085, 0.00085, 0.00084, 0.00085, 0.00118, 0.00086, 0.00087, 0.00086, 0.00108, 0.00085, 0.00085, 0.00084, 0.00084, 0.00083, 0.00085, 0.00109, 0.00084, 0.00083, 0.00084, 0.00086, 0.00085, 0.00086, 0.00085, 0.00085, 0.00085, 0.00086, 0.00085, 0.00084, 0.00087, 0.00085, 0.00087, 0.00084, 0.00086, 0.00085, 0.00085, 0.00084, 0.00085, 0.00084, 0.00085, 0.00084, 0.00085, 0.00087, 0.00085, 0.00087, 0.00096, 0.00085, 0.00085, 0.00086, 0.00084, 0.00085, 0.00086, 0.00083, 0.00085, 0.00084, 0.00083, 0.00084, 0.00084, 0.00083, 0.00084, 0.00084, 0.00085, 0.00083, 0.00083, 0.00083, 0.00083, 0.00084, 0.00083, 0.00084, 0.00083, 0.00083, 0.00085, 0.00084, 0.00083, 0.00084, 0.00083, 0.00084, 0.00083, 0.00084, 0.00084, 0.00083, 0.00084, 0.00086, 0.00084, 0.00083, 0.00084, 0.00084, 0.00083, 0.00085, 0.00085, 0.00084, 0.00083, 0.00086, 0.00086, 0.00084, 0.00085, 0.00083, 0.00084, 0.00084, 0.00083, 0.00084, 0.00083, 0.00083, 0.00083, 0.00084, 0.00085, 0.00085, 0.00083, 0.00084, 0.00083, 0.00083, 0.00094, 0.00084, 0.00084, 0.00083, 0.00084, 0.00084, 0.00083, 0.00085, 0.00084, 0.00084, 0.00083, 0.00085, 0.00084, 0.00084, 0.00083, 0.00085, 0.00084, 0.00083, 0.00085, 0.00083, 0.00083, 0.00085, 0.00083, 0.00084, 0.00098, 0.00085, 0.00084, 0.00085, 0.00083, 0.00083, 0.00084, 0.00085, 0.00085, 0.00085, 0.00084, 0.00084, 0.00084, 0.00084, 0.00083, 0.00085, 0.00085, 0.00084, 0.00087, 0.00084, 0.00083, 0.00084, 0.00085, 0.00084, 0.00084, 0.00084, 0.00085, 0.00086, 0.00086, 0.00083, 0.00083, 0.00083, 0.00085, 0.00084, 0.00085, 0.00084, 0.00084, 0.00085, 0.00084, 0.00084, 0.00084, 0.00084, 0.00083, 0.00082, 0.00084, 0.00109, 0.00084, 0.00084, 0.00084, 0.00084, 0.00084, 0.00083, 0.00083, 0.00085, 0.00085, 0.00084, 0.00084, 0.00085, 0.00084, 0.00085, 0.00083, 0.00085, 0.00084, 0.00084, 0.00084, 0.00085, 0.00085, 0.00085, 0.00084, 0.00083, 0.00093, 0.00084, 0.00083, 0.00085, 0.00084, 0.00084, 0.00084, 0.00084, 0.00084, 0.00085, 0.00085, 0.00083, 0.00084, 0.00084, 0.00083, 0.00085, 0.00086, 0.00085, 0.00083, 0.00085, 0.00085, 0.00084, 0.00085, 0.00084, 0.00084, 0.00085, 0.00085, 0.00085, 0.00084, 0.00085, 0.00083, 0.00084, 0.00083, 0.00084, 0.00085, 0.00083, 0.00084, 0.00086, 0.00086, 0.00085, 0.00084, 0.00102, 0.00089, 0.00085, 0.00084, 0.00084, 0.00084, 0.00084, 0.00085, 0.00085, 0.00084, 0.00086, 0.00096, 0.00083, 0.00085, 0.00084, 0.00084, 0.00085, 0.00084, 0.00084, 0.00083, 0.00085, 0.00084, 0.00085, 0.00085, 0.00083, 0.00084, 0.00085, 0.00085, 0.00084, 0.00086, 0.00084, 0.00084, 0.00083, 0.00095, 0.00084, 0.00084, 0.00086, 0.00085, 0.00084, 0.00085, 0.00084, 0.00084, 0.00086, 0.00085, 0.00085, 0.00085, 0.00084, 0.00083, 0.00087, 0.00084, 0.00093, 0.00085, 0.00084, 0.00084, 0.00085, 0.00083, 0.00083, 0.00084, 0.00083, 0.00085, 0.00086, 0.00084, 0.00113, 0.00084, 0.00083, 0.00084, 0.00103, 0.00085, 0.00084, 0.00087, 0.00084, 0.00084, 0.00084, 0.00083, 0.00084, 0.00086, 0.00084, 0.00084, 0.00082, 0.00085, 0.00085, 0.00083, 0.00084, 0.00084, 0.00084, 0.00084, 0.00085, 0.00084, 0.00084, 0.00082, 0.00085, 0.00084, 0.00083, 0.00084, 0.00085, 0.00094, 0.00085, 0.00085, 0.00086, 0.00116, 0.00084, 0.00137, 0.00084, 0.00083, 0.00084, 0.00084, 0.00104, 0.00085, 0.00083]}, "optimizer-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.03257, 0.00561, 0.00555, 0.00673, 0.00567, 0.00562, 0.00561, 0.00563, 0.00577, 0.00565, 0.00561, 0.00611, 0.00562, 0.00577, 0.00929, 0.00564, 0.00561, 0.00562, 0.0056, 0.00562, 0.0056, 0.00563, 0.00563, 0.00561, 0.00559, 0.00561, 0.00563, 0.00561, 0.00562, 0.00557, 0.0056, 0.00562, 0.00562, 0.00563, 0.00562, 0.00562, 0.00568, 0.00562, 0.00565, 0.00566, 0.00566, 0.00565, 0.0056, 0.00567, 0.00567, 0.00569, 0.00566, 0.00568, 0.00565, 0.00563, 0.00698, 0.00565, 0.00598, 0.0057, 0.00701, 0.00568, 0.00567, 0.00565, 0.00567, 0.00568, 0.00563, 0.00767, 0.00563, 0.00608, 0.00566, 0.00565, 0.00568, 0.00565, 0.00565, 0.00567, 0.00566, 0.00571, 0.00568, 0.00567, 0.00567, 0.00565, 0.00569, 0.00575, 0.00565, 0.00565, 0.00562, 0.00577, 0.00568, 0.00567, 0.00563, 0.00564, 0.00565, 0.0057, 0.00565, 0.00567, 0.00638, 0.00578, 0.00578, 0.00572, 0.0056, 0.00567, 0.00571, 0.00565, 0.00565, 0.00567, 0.00563, 0.00563, 0.00563, 0.00563, 0.00562, 0.00635, 0.00583, 0.00568, 0.00584, 0.00555, 0.00577, 0.00559, 0.0056, 0.00558, 0.00584, 0.00561, 0.00557, 0.00564, 0.00562, 0.00566, 0.00555, 0.00562, 0.00565, 0.00566, 0.00559, 0.0056, 0.00561, 0.00566, 0.00564, 0.00561, 0.00563, 0.00564, 0.00564, 0.00565, 0.00564, 0.00568, 0.00564, 0.00565, 0.00566, 0.00568, 0.00554, 0.00562, 0.00556, 0.00562, 0.0057, 0.00565, 0.00583, 0.00554, 0.00562, 0.00561, 0.00564, 0.00571, 0.00563, 0.00563, 0.00565, 0.0056, 0.00607, 0.00565, 0.00564, 0.00564, 0.00565, 0.00565, 0.00563, 0.00564, 0.00563, 0.00566, 0.00564, 0.00565, 0.00565, 0.00567, 0.00565, 0.00576, 0.00575, 0.00563, 0.00566, 0.00658, 0.00565, 0.00564, 0.00568, 0.00562, 0.00663, 0.00565, 0.00564, 0.00564, 0.00562, 0.00563, 0.00568, 0.00566, 0.00565, 0.00564, 0.00565, 0.00563, 0.00565, 0.00561, 0.00564, 0.00563, 0.00562, 0.00564, 0.00568, 0.00568, 0.00567, 0.00567, 0.00569, 0.00566, 0.0056, 0.00564, 0.00567, 0.00567, 0.00586, 0.00568, 0.00555, 0.00567, 0.00562, 0.00558, 0.00585, 0.00563, 0.00566, 0.00565, 0.00565, 0.00566, 0.00559, 0.00566, 0.00566, 0.00561, 0.00573, 0.00721, 0.00562, 0.00564, 0.00593, 0.00595, 0.00563, 0.00564, 0.00566, 0.00567, 0.00565, 0.00569, 0.00564, 0.00566, 0.00568, 0.00566, 0.00578, 0.00588, 0.0064, 0.00571, 0.00566, 0.00564, 0.00565, 0.00567, 0.00566, 0.00564, 0.00643, 0.00566, 0.00567, 0.00564, 0.00601, 0.00563, 0.00566, 0.00566, 0.00566, 0.00563, 0.00566, 0.00565, 0.00557, 0.00567, 0.00564, 0.00566, 0.00565, 0.00566, 0.00564, 0.00596, 0.00567, 0.00562, 0.00565, 0.00566, 0.00564, 0.00564, 0.00569, 0.00568, 0.00569, 0.00569, 0.00575, 0.00567, 0.00583, 0.00568, 0.00566, 0.00566, 0.00567, 0.00566, 0.00567, 0.00566, 0.00564, 0.00689, 0.00665, 0.00563, 0.00566, 0.00566, 0.00685, 0.00566, 0.00565, 0.00567, 0.00567, 0.00574, 0.00611, 0.00563, 0.00565, 0.00569, 0.00568, 0.00568, 0.00568, 0.0057, 0.00566, 0.00569, 0.00567, 0.0057, 0.00566, 0.00569, 0.00564, 0.00565, 0.00568, 0.00569, 0.00571, 0.00564, 0.00566, 0.00565, 0.0058, 0.00566, 0.00565, 0.00564, 0.00566, 0.00566, 0.00567, 0.00556, 0.00565, 0.00568, 0.00564, 0.00567, 0.00566, 0.00566, 0.00566, 0.00566, 0.00565, 0.00622, 0.00564, 0.00563, 0.00565, 0.0058, 0.00565, 0.00563, 0.00567, 0.00564, 0.00566, 0.00569, 0.00579, 0.0071, 0.00625, 0.00661, 0.00596, 0.00708, 0.00571, 0.00566, 0.00572, 0.0057, 0.00565, 0.00566, 0.00568, 0.00566, 0.00569, 0.00565, 0.00568, 0.00558, 0.00572, 0.00566, 0.00564, 0.00571, 0.00569, 0.00569, 0.00567, 0.00567, 0.00564, 0.00569, 0.00563, 0.0057, 0.00565, 0.00567, 0.00569, 0.00565, 0.00602, 0.00567, 0.00566, 0.00568, 0.00691, 0.00568, 0.00824, 0.00567, 0.00569, 0.00565, 0.00566, 0.00689, 0.00567, 0.00569]}, "learning-rate": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05]}, "learning-rate vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05]}, "batch-size": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0]}, "batch-size vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0]}, "lm loss": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [10.8433, 10.85765, 10.84779, 10.84476, 10.76311, 10.77117, 10.67823, 10.52752, 10.37993, 10.29638, 9.93195, 10.03509, 10.0426, 9.75307, 9.86889, 9.5734, 9.50903, 9.70491, 9.4312, 9.37508, 9.28309, 9.18169, 9.20577, 9.02386, 9.21628, 9.08364, 9.17244, 9.18282, 9.31596, 9.0048, 8.94512, 9.05935, 9.05717, 8.66601, 8.72832, 8.75869, 8.69275, 8.74055, 8.6626, 8.76871, 8.66379, 8.85229, 8.8339, 8.49642, 8.38634, 8.42672, 8.48466, 8.37859, 8.42664, 8.57856, 8.36195, 8.18567, 8.21753, 8.21329, 8.25896, 7.90534, 8.08583, 7.88164, 8.23415, 8.21584, 7.99096, 7.95558, 7.90491, 7.72205, 7.72605, 7.6289, 7.49968, 7.88829, 7.68144, 7.43346, 7.72641, 7.75429, 7.52412, 7.28309, 7.43578, 7.32461, 7.44873, 7.21189, 7.61912, 7.26534, 7.33401, 7.19818, 7.19879, 7.40517, 7.15831, 7.26654, 6.98097, 6.98873, 7.02577, 7.12311, 6.80994, 6.9713, 7.07655, 6.98656, 6.86237, 6.74308, 6.97741, 7.04512, 6.6892, 6.56911, 6.70842, 6.72744, 6.71821, 6.72252, 6.6415, 6.39227, 6.62344, 6.6066, 6.43533, 6.61754, 6.73372, 6.60246, 6.71828, 6.68928, 6.61913, 6.50141, 6.59197, 6.4038, 6.66146, 6.24279, 6.24693, 6.29915, 6.38884, 6.34615, 6.44807, 6.28858, 6.33623, 6.2327, 6.19805, 6.39278, 6.32018, 6.31748, 6.15883, 6.15355, 6.23186, 6.37861, 6.19447, 6.14485, 6.1733, 6.10804, 6.05466, 6.06414, 6.24514, 6.3995, 6.24908, 6.28746, 6.08812, 6.16815, 5.99306, 6.01895, 5.94959, 6.24347, 6.17773, 5.95991, 5.77827, 6.11616, 5.84215, 6.09747, 5.77523, 6.15215, 6.13478, 6.07243, 5.91679, 6.10325, 5.93318, 6.18522, 5.88104, 5.77729, 5.77183, 5.67085, 6.00059, 5.98318, 6.05535, 5.87842, 6.02672, 5.95703, 5.98143, 5.97599, 5.93931, 5.83179, 5.9381, 5.60666, 5.69093, 5.87661, 5.83166, 5.85725, 5.75469, 5.82709, 5.71508, 5.55284, 5.71442, 5.61457, 5.82158, 5.59478, 5.70073, 5.70005, 5.89549, 5.63767, 5.84273, 5.73351, 5.86251, 5.3238, 5.89106, 5.86774, 5.84522, 5.40975, 5.40264, 5.62175, 5.59059, 5.47771, 5.57089, 5.66784, 5.47115, 5.73871, 5.50633, 5.58597, 5.61567, 5.61569, 5.50604, 5.61122, 5.66663, 5.67443, 5.58163, 5.65574, 5.36724, 5.67456, 5.62197, 5.42234, 5.57798, 5.62266, 5.55291, 5.34573, 5.5345, 5.48019, 5.47665, 5.38005, 5.54985, 5.60007, 5.38622, 5.51749, 5.48316, 5.33148, 5.49982, 5.40449, 5.44324, 5.31566, 5.06363, 5.47841, 5.5691, 5.71408, 5.41548, 5.60635, 5.63525, 5.23472, 5.27189, 5.39367, 5.39769, 5.3288, 5.49398, 5.18196, 5.29891, 5.24595, 5.37805, 5.25379, 5.4444, 5.53625, 5.3118, 5.43692, 5.33895, 5.07945, 5.31174, 5.25433, 5.30498, 5.11513, 5.27718, 5.26206, 5.47608, 5.15887, 5.26425, 5.21348, 5.35846, 4.9858, 4.91634, 5.32535, 5.39184, 5.23322, 5.32273, 5.10676, 5.16478, 5.26314, 5.06733, 5.26641, 5.06795, 5.34712, 5.25384, 5.15068, 5.24204, 5.04041, 5.31825, 5.05553, 5.03059, 5.14352, 5.1141, 5.27551, 5.15912, 5.27903, 5.09426, 5.09379, 5.24785, 5.32857, 5.2547, 5.19567, 5.14313, 5.29062, 4.95221, 5.21032, 5.09608, 5.30523, 5.17392, 5.19286, 5.11816, 4.98511, 4.99538, 5.22333, 5.31529, 5.10038, 5.05941, 4.91674, 5.12756, 5.12029, 4.93474, 5.3446, 5.02767, 5.10269, 5.16837, 5.00565, 5.06744, 5.07125, 4.99847, 5.08296, 5.16749, 4.98067, 5.18306, 4.93375, 4.92594, 5.0664, 4.99659, 4.90949, 4.77712, 4.94745, 5.12054, 5.0185, 5.01985, 5.33344, 4.9602, 4.99514, 5.05213, 4.81431, 4.73906, 4.99924, 5.04442, 4.87459, 4.95901, 5.0525, 5.02541, 4.81849, 4.89819, 4.91224, 4.83311, 4.74468, 5.01583, 4.7552, 5.21058, 4.79037, 4.99637, 4.74215, 4.78879, 4.82079, 4.65284, 4.65944, 4.84537, 4.80978, 4.80376, 4.92422, 4.88911, 4.93392, 4.77435, 4.88266, 4.73357, 4.91568, 4.96037, 4.87459, 4.7064, 4.78699, 4.90799, 4.71496, 4.87497, 4.70188, 4.70185, 4.64815]}, "lm loss vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [10.84303, 10.86032, 10.84988, 10.84755, 10.76639, 10.77411, 10.67857, 10.53004, 10.38397, 10.29666, 9.92036, 10.03609, 10.04286, 9.75368, 9.87024, 9.57458, 9.50956, 9.70645, 9.43156, 9.37511, 9.284, 9.18283, 9.20684, 9.02346, 9.21677, 9.08417, 9.17277, 9.18323, 9.31569, 9.00474, 8.94547, 9.06044, 9.05792, 8.66708, 8.73014, 8.76017, 8.69512, 8.74237, 8.66438, 8.77103, 8.66577, 8.85394, 8.83642, 8.49824, 8.38764, 8.42876, 8.48638, 8.38112, 8.42721, 8.57916, 8.36213, 8.18555, 8.21868, 8.21376, 8.25912, 7.90597, 8.08558, 7.88018, 8.23297, 8.21565, 7.99013, 7.95413, 7.90374, 7.72213, 7.72557, 7.62784, 7.49843, 7.88783, 7.68211, 7.43256, 7.72606, 7.75519, 7.5254, 7.28466, 7.43748, 7.32478, 7.44941, 7.21198, 7.61949, 7.26498, 7.33394, 7.19595, 7.19608, 7.40347, 7.15606, 7.26585, 6.98127, 6.98967, 7.02701, 7.12404, 6.81114, 6.9732, 7.07844, 6.98715, 6.86379, 6.74535, 6.97969, 7.04992, 6.69473, 6.57332, 6.71755, 6.73627, 6.72482, 6.72951, 6.64965, 6.39869, 6.62934, 6.6128, 6.44062, 6.62092, 6.73782, 6.60642, 6.72099, 6.69098, 6.62325, 6.50501, 6.59411, 6.40344, 6.66286, 6.24475, 6.24827, 6.29959, 6.38833, 6.34649, 6.44604, 6.28662, 6.33306, 6.23143, 6.1945, 6.39075, 6.31833, 6.31606, 6.15661, 6.15059, 6.23078, 6.37677, 6.19418, 6.14556, 6.174, 6.10964, 6.05825, 6.06794, 6.25281, 6.40554, 6.25551, 6.29757, 6.09544, 6.1725, 6.00218, 6.02712, 5.95524, 6.25067, 6.1861, 5.96596, 5.78395, 6.12333, 5.84793, 6.10088, 5.78605, 6.16305, 6.14324, 6.08193, 5.9272, 6.11128, 5.94147, 6.19288, 5.88909, 5.78652, 5.77759, 5.68182, 6.00901, 5.99171, 6.064, 5.887, 6.03556, 5.96156, 5.98678, 5.98309, 5.94332, 5.83241, 5.94309, 5.60951, 5.69435, 5.88169, 5.83567, 5.85447, 5.75902, 5.83004, 5.71739, 5.55081, 5.71567, 5.61507, 5.82158, 5.59427, 5.70169, 5.70024, 5.89399, 5.63586, 5.84189, 5.73395, 5.86128, 5.31906, 5.89065, 5.8668, 5.84568, 5.40705, 5.40162, 5.61805, 5.58944, 5.47887, 5.57169, 5.66894, 5.46961, 5.737, 5.50292, 5.58399, 5.61697, 5.61602, 5.50714, 5.6077, 5.6651, 5.67541, 5.58049, 5.65548, 5.36443, 5.67256, 5.62445, 5.41886, 5.57712, 5.62171, 5.55213, 5.34421, 5.53498, 5.48095, 5.4778, 5.37859, 5.55337, 5.60077, 5.38946, 5.5161, 5.4845, 5.3308, 5.503, 5.40661, 5.44202, 5.3156, 5.06608, 5.47488, 5.56633, 5.71203, 5.41237, 5.602, 5.6336, 5.23514, 5.26957, 5.38908, 5.39646, 5.32832, 5.49536, 5.18302, 5.2973, 5.24699, 5.3738, 5.2533, 5.4419, 5.53407, 5.31248, 5.43315, 5.33688, 5.07446, 5.3117, 5.25312, 5.30184, 5.11129, 5.27552, 5.26324, 5.47224, 5.15822, 5.26777, 5.21213, 5.35617, 4.98409, 4.9122, 5.32204, 5.39135, 5.22909, 5.3223, 5.10207, 5.16342, 5.26324, 5.06816, 5.26642, 5.06638, 5.34472, 5.24739, 5.15433, 5.24748, 5.04399, 5.32024, 5.05488, 5.02871, 5.1457, 5.11299, 5.27264, 5.15675, 5.28106, 5.09695, 5.09458, 5.25141, 5.32789, 5.25804, 5.19731, 5.14154, 5.29133, 4.95279, 5.2099, 5.09154, 5.30528, 5.17547, 5.19246, 5.11436, 4.986, 4.99619, 5.22741, 5.31255, 5.10417, 5.06172, 4.91443, 5.12691, 5.1217, 4.93205, 5.34318, 5.02802, 5.10574, 5.17142, 5.00778, 5.07028, 5.0728, 4.99912, 5.08403, 5.16803, 4.98253, 5.18553, 4.93609, 4.93034, 5.06451, 5.00328, 4.9143, 4.78254, 4.9515, 5.1248, 5.02128, 5.01937, 5.34246, 4.96515, 4.99654, 5.05289, 4.816, 4.74072, 4.99878, 5.04752, 4.87941, 4.96151, 5.05319, 5.02704, 4.8254, 4.8992, 4.91046, 4.83957, 4.74493, 5.01861, 4.76013, 5.21014, 4.79858, 5.00113, 4.74548, 4.79219, 4.82659, 4.65777, 4.66208, 4.84897, 4.81474, 4.80913, 4.92799, 4.89236, 4.93339, 4.77993, 4.89168, 4.7432, 4.92229, 4.96619, 4.88011, 4.71273, 4.7931, 4.91139, 4.72229, 4.87421, 4.70468, 4.69956, 4.65227]}, "loss-scale": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]}, "loss-scale vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]}, "grad-norm": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [13.92196, 13.64105, 14.19575, 13.10329, 13.56093, 11.06924, 10.32704, 12.58903, 11.89406, 9.6749, 7.04626, 4.0336, 3.15187, 2.82418, 2.35804, 2.43442, 2.16004, 1.97461, 2.14035, 2.12249, 2.20138, 2.2657, 2.05671, 2.22896, 1.95829, 2.02503, 1.88632, 1.84693, 1.87101, 2.18322, 2.10962, 1.97689, 1.94956, 2.15482, 2.33059, 2.0713, 2.06596, 1.83468, 1.98146, 1.78906, 2.08095, 1.74031, 1.73584, 1.83223, 1.93635, 1.78517, 1.74533, 1.74989, 1.72773, 1.51419, 1.74951, 1.76214, 1.76755, 1.83739, 1.54724, 1.80208, 1.67454, 1.80868, 1.51645, 1.42949, 1.65422, 1.43167, 1.74384, 1.82674, 1.56795, 1.61973, 1.62231, 1.51322, 1.4269, 1.55439, 1.3649, 1.40671, 1.47679, 1.40979, 1.35488, 1.43798, 1.41114, 1.34745, 1.32431, 1.23395, 1.36576, 1.22914, 1.25372, 1.35028, 1.23455, 1.29297, 1.37717, 1.26373, 1.37004, 1.08995, 1.10379, 1.10875, 1.15108, 1.26523, 0.89985, 1.39001, 1.10735, 1.30884, 1.00577, 1.31705, 1.15922, 1.16049, 1.08293, 1.30514, 0.98385, 1.11074, 1.1592, 0.9745, 1.26156, 1.13226, 0.98984, 0.97441, 0.96023, 0.94898, 1.04337, 1.04095, 0.96044, 1.19634, 1.26146, 1.4137, 0.97849, 1.01274, 1.06643, 1.01496, 0.94459, 1.13752, 1.02579, 1.05074, 1.22247, 1.26548, 1.04774, 1.44863, 1.15549, 1.15597, 1.19734, 1.2287, 1.25743, 1.88802, 1.76897, 1.48112, 1.4651, 1.39709, 1.38654, 1.09404, 1.62425, 1.69258, 1.31425, 1.11912, 1.16099, 1.18343, 1.29282, 1.58176, 1.59702, 1.35711, 1.25116, 1.93028, 1.26411, 1.16234, 1.73045, 1.37516, 1.21056, 1.1698, 1.36362, 1.31019, 1.41174, 1.1141, 1.35444, 1.27655, 1.56101, 1.26438, 1.09582, 1.27416, 1.41508, 1.54422, 1.36323, 1.24407, 1.29014, 1.18935, 1.13176, 1.03122, 1.33001, 1.37077, 1.14753, 1.11258, 1.66325, 1.11887, 1.76805, 1.40233, 1.37783, 1.50291, 1.27142, 1.30216, 1.29887, 1.46138, 1.55382, 1.23876, 1.8076, 1.40113, 1.63396, 1.55057, 1.08699, 1.24471, 1.22211, 1.14251, 1.26485, 1.45246, 1.55789, 1.71804, 1.37054, 1.61527, 1.57346, 1.43675, 1.26103, 1.17063, 1.56904, 1.17977, 1.4408, 1.72049, 1.50941, 1.30391, 1.34373, 1.32377, 1.27909, 1.56247, 1.31671, 1.38601, 1.61151, 1.49478, 1.75857, 1.27914, 1.31454, 2.08285, 1.65152, 1.54337, 1.46369, 1.68505, 1.74708, 1.34813, 1.53151, 1.36655, 1.5068, 1.33926, 1.42092, 1.39573, 1.3088, 1.90711, 1.46652, 1.29613, 1.44842, 1.30354, 1.28453, 1.49548, 1.47812, 1.39914, 1.32083, 1.19715, 1.79989, 1.43253, 1.35222, 1.42532, 1.23793, 1.41904, 1.21814, 1.25683, 1.2335, 1.46238, 1.48727, 1.4808, 1.33354, 1.33662, 1.26457, 1.31807, 1.46217, 1.35853, 1.55295, 1.20988, 1.50233, 1.51611, 1.48328, 1.32591, 1.35903, 1.25739, 1.45462, 1.40772, 1.52784, 1.49325, 1.48176, 1.41498, 1.37099, 1.4565, 1.35995, 1.85538, 1.22436, 1.50223, 1.62834, 2.02006, 1.60123, 1.72187, 1.44841, 1.22003, 1.2907, 1.31733, 1.13053, 1.33575, 1.57284, 1.47894, 1.41277, 1.40064, 1.30099, 1.35607, 1.52515, 1.48522, 1.31187, 1.24496, 1.36995, 1.60389, 1.24009, 1.55027, 1.2329, 1.34795, 1.32343, 1.38946, 1.27338, 1.46297, 1.50613, 1.56272, 1.67908, 1.41893, 1.40655, 1.34016, 1.79612, 1.52344, 1.31538, 1.82889, 1.5317, 1.18989, 1.44241, 1.33335, 1.49631, 1.45109, 1.41567, 1.28181, 1.28831, 1.39113, 1.42151, 1.1475, 1.49249, 1.42727, 1.4635, 1.13088, 1.41, 1.30719, 1.30003, 1.92172, 1.44667, 1.42061, 1.31137, 1.5365, 1.46596, 1.30019, 1.53226, 1.21709, 1.36071, 1.47588, 1.10067, 1.46261, 1.69979, 1.33386, 1.3067, 1.50275, 1.48945, 1.4021, 1.56615, 1.59437, 1.41693, 1.52987, 1.27517, 1.55287, 1.38137, 1.28009, 1.33198, 1.29291, 1.40497, 1.25603, 1.18811, 1.37138, 1.43758, 1.46419, 1.4718, 1.35085, 1.22463, 1.2576, 1.44724, 1.32087, 1.61352, 1.4648, 1.47154, 1.80709, 1.41366, 1.12723]}, "grad-norm vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [13.92196, 13.64105, 14.19575, 13.10329, 13.56093, 11.06924, 10.32704, 12.58903, 11.89406, 9.6749, 7.04626, 4.0336, 3.15187, 2.82418, 2.35804, 2.43442, 2.16004, 1.97461, 2.14035, 2.12249, 2.20138, 2.2657, 2.05671, 2.22896, 1.95829, 2.02503, 1.88632, 1.84693, 1.87101, 2.18322, 2.10962, 1.97689, 1.94956, 2.15482, 2.33059, 2.0713, 2.06596, 1.83468, 1.98146, 1.78906, 2.08095, 1.74031, 1.73584, 1.83223, 1.93635, 1.78517, 1.74533, 1.74989, 1.72773, 1.51419, 1.74951, 1.76214, 1.76755, 1.83739, 1.54724, 1.80208, 1.67454, 1.80868, 1.51645, 1.42949, 1.65422, 1.43167, 1.74384, 1.82674, 1.56795, 1.61973, 1.62231, 1.51322, 1.4269, 1.55439, 1.3649, 1.40671, 1.47679, 1.40979, 1.35488, 1.43798, 1.41114, 1.34745, 1.32431, 1.23395, 1.36576, 1.22914, 1.25372, 1.35028, 1.23455, 1.29297, 1.37717, 1.26373, 1.37004, 1.08995, 1.10379, 1.10875, 1.15108, 1.26523, 0.89985, 1.39001, 1.10735, 1.30884, 1.00577, 1.31705, 1.15922, 1.16049, 1.08293, 1.30514, 0.98385, 1.11074, 1.1592, 0.9745, 1.26156, 1.13226, 0.98984, 0.97441, 0.96023, 0.94898, 1.04337, 1.04095, 0.96044, 1.19634, 1.26146, 1.4137, 0.97849, 1.01274, 1.06643, 1.01496, 0.94459, 1.13752, 1.02579, 1.05074, 1.22247, 1.26548, 1.04774, 1.44863, 1.15549, 1.15597, 1.19734, 1.2287, 1.25743, 1.88802, 1.76897, 1.48112, 1.4651, 1.39709, 1.38654, 1.09404, 1.62425, 1.69258, 1.31425, 1.11912, 1.16099, 1.18343, 1.29282, 1.58176, 1.59702, 1.35711, 1.25116, 1.93028, 1.26411, 1.16234, 1.73045, 1.37516, 1.21056, 1.1698, 1.36362, 1.31019, 1.41174, 1.1141, 1.35444, 1.27655, 1.56101, 1.26438, 1.09582, 1.27416, 1.41508, 1.54422, 1.36323, 1.24407, 1.29014, 1.18935, 1.13176, 1.03122, 1.33001, 1.37077, 1.14753, 1.11258, 1.66325, 1.11887, 1.76805, 1.40233, 1.37783, 1.50291, 1.27142, 1.30216, 1.29887, 1.46138, 1.55382, 1.23876, 1.8076, 1.40113, 1.63396, 1.55057, 1.08699, 1.24471, 1.22211, 1.14251, 1.26485, 1.45246, 1.55789, 1.71804, 1.37054, 1.61527, 1.57346, 1.43675, 1.26103, 1.17063, 1.56904, 1.17977, 1.4408, 1.72049, 1.50941, 1.30391, 1.34373, 1.32377, 1.27909, 1.56247, 1.31671, 1.38601, 1.61151, 1.49478, 1.75857, 1.27914, 1.31454, 2.08285, 1.65152, 1.54337, 1.46369, 1.68505, 1.74708, 1.34813, 1.53151, 1.36655, 1.5068, 1.33926, 1.42092, 1.39573, 1.3088, 1.90711, 1.46652, 1.29613, 1.44842, 1.30354, 1.28453, 1.49548, 1.47812, 1.39914, 1.32083, 1.19715, 1.79989, 1.43253, 1.35222, 1.42532, 1.23793, 1.41904, 1.21814, 1.25683, 1.2335, 1.46238, 1.48727, 1.4808, 1.33354, 1.33662, 1.26457, 1.31807, 1.46217, 1.35853, 1.55295, 1.20988, 1.50233, 1.51611, 1.48328, 1.32591, 1.35903, 1.25739, 1.45462, 1.40772, 1.52784, 1.49325, 1.48176, 1.41498, 1.37099, 1.4565, 1.35995, 1.85538, 1.22436, 1.50223, 1.62834, 2.02006, 1.60123, 1.72187, 1.44841, 1.22003, 1.2907, 1.31733, 1.13053, 1.33575, 1.57284, 1.47894, 1.41277, 1.40064, 1.30099, 1.35607, 1.52515, 1.48522, 1.31187, 1.24496, 1.36995, 1.60389, 1.24009, 1.55027, 1.2329, 1.34795, 1.32343, 1.38946, 1.27338, 1.46297, 1.50613, 1.56272, 1.67908, 1.41893, 1.40655, 1.34016, 1.79612, 1.52344, 1.31538, 1.82889, 1.5317, 1.18989, 1.44241, 1.33335, 1.49631, 1.45109, 1.41567, 1.28181, 1.28831, 1.39113, 1.42151, 1.1475, 1.49249, 1.42727, 1.4635, 1.13088, 1.41, 1.30719, 1.30003, 1.92172, 1.44667, 1.42061, 1.31137, 1.5365, 1.46596, 1.30019, 1.53226, 1.21709, 1.36071, 1.47588, 1.10067, 1.46261, 1.69979, 1.33386, 1.3067, 1.50275, 1.48945, 1.4021, 1.56615, 1.59437, 1.41693, 1.52987, 1.27517, 1.55287, 1.38137, 1.28009, 1.33198, 1.29291, 1.40497, 1.25603, 1.18811, 1.37138, 1.43758, 1.46419, 1.4718, 1.35085, 1.22463, 1.2576, 1.44724, 1.32087, 1.61352, 1.4648, 1.47154, 1.80709, 1.41366, 1.12723]}, "num-zeros": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [74.0, 72.0, 69.0, 56.0, 80.0, 91.0, 67.0, 82.0, 93.0, 105.0, 110.0, 142.0, 141.0, 159.0, 161.0, 143.0, 169.0, 195.0, 170.0, 186.0, 163.0, 157.0, 166.0, 142.0, 194.0, 179.0, 181.0, 188.0, 153.0, 168.0, 155.0, 140.0, 149.0, 178.0, 131.0, 158.0, 174.0, 213.0, 189.0, 168.0, 175.0, 162.0, 144.0, 163.0, 204.0, 186.0, 182.0, 175.0, 171.0, 240.0, 213.0, 187.0, 193.0, 135.0, 188.0, 193.0, 180.0, 152.0, 257.0, 211.0, 178.0, 190.0, 194.0, 197.0, 192.0, 244.0, 203.0, 170.0, 219.0, 176.0, 233.0, 241.0, 188.0, 245.0, 213.0, 197.0, 209.0, 194.0, 234.0, 208.0, 231.0, 214.0, 225.0, 229.0, 216.0, 159.0, 178.0, 183.0, 178.0, 197.0, 209.0, 187.0, 229.0, 177.0, 234.0, 198.0, 226.0, 238.0, 175.0, 169.0, 196.0, 165.0, 145.0, 159.0, 168.0, 161.0, 159.0, 160.0, 138.0, 155.0, 179.0, 147.0, 156.0, 157.0, 140.0, 140.0, 147.0, 114.0, 135.0, 143.0, 137.0, 115.0, 128.0, 145.0, 145.0, 120.0, 101.0, 156.0, 137.0, 136.0, 128.0, 132.0, 120.0, 117.0, 168.0, 126.0, 140.0, 114.0, 115.0, 139.0, 112.0, 107.0, 119.0, 143.0, 113.0, 120.0, 146.0, 116.0, 122.0, 116.0, 105.0, 89.0, 128.0, 113.0, 99.0, 112.0, 117.0, 122.0, 132.0, 130.0, 130.0, 112.0, 113.0, 115.0, 105.0, 120.0, 108.0, 108.0, 90.0, 123.0, 120.0, 126.0, 95.0, 94.0, 119.0, 111.0, 108.0, 116.0, 91.0, 102.0, 101.0, 82.0, 111.0, 156.0, 116.0, 105.0, 98.0, 113.0, 120.0, 93.0, 112.0, 106.0, 103.0, 112.0, 89.0, 108.0, 104.0, 87.0, 113.0, 100.0, 106.0, 104.0, 119.0, 142.0, 123.0, 114.0, 110.0, 88.0, 117.0, 119.0, 96.0, 132.0, 102.0, 97.0, 99.0, 89.0, 110.0, 116.0, 100.0, 111.0, 130.0, 118.0, 93.0, 99.0, 102.0, 106.0, 120.0, 105.0, 109.0, 118.0, 81.0, 66.0, 75.0, 103.0, 113.0, 96.0, 95.0, 103.0, 97.0, 97.0, 108.0, 91.0, 93.0, 115.0, 108.0, 101.0, 97.0, 96.0, 120.0, 87.0, 103.0, 104.0, 101.0, 88.0, 100.0, 101.0, 97.0, 119.0, 99.0, 141.0, 110.0, 117.0, 103.0, 111.0, 118.0, 88.0, 110.0, 111.0, 109.0, 85.0, 113.0, 82.0, 97.0, 94.0, 116.0, 112.0, 122.0, 94.0, 146.0, 103.0, 102.0, 99.0, 100.0, 93.0, 120.0, 81.0, 91.0, 95.0, 120.0, 91.0, 129.0, 93.0, 113.0, 118.0, 71.0, 111.0, 102.0, 117.0, 123.0, 109.0, 114.0, 104.0, 118.0, 109.0, 104.0, 96.0, 96.0, 89.0, 121.0, 108.0, 94.0, 130.0, 109.0, 119.0, 129.0, 115.0, 96.0, 119.0, 107.0, 104.0, 111.0, 102.0, 98.0, 105.0, 116.0, 106.0, 118.0, 110.0, 115.0, 90.0, 115.0, 81.0, 118.0, 114.0, 93.0, 99.0, 105.0, 115.0, 112.0, 92.0, 128.0, 117.0, 131.0, 119.0, 115.0, 106.0, 132.0, 103.0, 97.0, 132.0, 108.0, 127.0, 125.0, 115.0, 130.0, 103.0, 105.0, 113.0, 113.0, 96.0, 116.0, 127.0, 120.0, 96.0, 132.0, 95.0, 110.0, 99.0, 101.0, 107.0, 108.0, 99.0, 117.0, 118.0, 117.0, 129.0, 109.0, 96.0, 106.0, 106.0, 116.0, 130.0, 121.0, 124.0, 126.0, 142.0, 127.0, 139.0, 123.0, 127.0, 119.0, 133.0, 107.0, 94.0, 78.0, 114.0, 122.0, 103.0, 104.0, 140.0]}, "num-zeros vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [65.0, 71.0, 74.0, 78.0, 68.0, 65.0, 79.0, 104.0, 95.0, 118.0, 116.0, 161.0, 141.0, 148.0, 182.0, 146.0, 164.0, 199.0, 174.0, 205.0, 166.0, 167.0, 186.0, 158.0, 195.0, 179.0, 188.0, 208.0, 187.0, 145.0, 145.0, 146.0, 156.0, 175.0, 132.0, 180.0, 177.0, 205.0, 172.0, 159.0, 158.0, 175.0, 153.0, 203.0, 196.0, 170.0, 185.0, 179.0, 140.0, 227.0, 198.0, 165.0, 172.0, 149.0, 199.0, 213.0, 179.0, 157.0, 255.0, 240.0, 186.0, 191.0, 164.0, 186.0, 208.0, 229.0, 213.0, 198.0, 198.0, 178.0, 246.0, 222.0, 177.0, 236.0, 193.0, 215.0, 226.0, 205.0, 251.0, 226.0, 224.0, 245.0, 219.0, 205.0, 198.0, 190.0, 171.0, 191.0, 171.0, 187.0, 182.0, 207.0, 233.0, 201.0, 220.0, 152.0, 216.0, 194.0, 175.0, 157.0, 165.0, 188.0, 163.0, 163.0, 160.0, 155.0, 160.0, 167.0, 144.0, 190.0, 194.0, 143.0, 153.0, 175.0, 158.0, 147.0, 166.0, 115.0, 142.0, 141.0, 117.0, 131.0, 132.0, 130.0, 164.0, 131.0, 136.0, 129.0, 150.0, 146.0, 133.0, 96.0, 139.0, 119.0, 108.0, 124.0, 109.0, 114.0, 113.0, 123.0, 125.0, 129.0, 99.0, 159.0, 109.0, 115.0, 127.0, 128.0, 101.0, 122.0, 118.0, 113.0, 110.0, 107.0, 112.0, 89.0, 107.0, 118.0, 89.0, 101.0, 127.0, 125.0, 111.0, 110.0, 121.0, 125.0, 111.0, 123.0, 109.0, 116.0, 118.0, 107.0, 87.0, 105.0, 121.0, 111.0, 127.0, 128.0, 116.0, 128.0, 116.0, 112.0, 135.0, 122.0, 106.0, 97.0, 100.0, 121.0, 94.0, 117.0, 124.0, 93.0, 116.0, 99.0, 114.0, 107.0, 96.0, 105.0, 102.0, 84.0, 138.0, 100.0, 100.0, 115.0, 133.0, 101.0, 99.0, 105.0, 116.0, 109.0, 100.0, 109.0, 120.0, 131.0, 107.0, 110.0, 111.0, 98.0, 118.0, 97.0, 122.0, 115.0, 121.0, 114.0, 91.0, 86.0, 116.0, 85.0, 79.0, 99.0, 97.0, 89.0, 103.0, 78.0, 108.0, 107.0, 78.0, 101.0, 99.0, 96.0, 119.0, 87.0, 98.0, 113.0, 112.0, 101.0, 78.0, 125.0, 101.0, 102.0, 137.0, 85.0, 97.0, 96.0, 119.0, 119.0, 93.0, 84.0, 94.0, 91.0, 132.0, 108.0, 113.0, 98.0, 127.0, 102.0, 88.0, 93.0, 124.0, 102.0, 99.0, 97.0, 99.0, 85.0, 103.0, 94.0, 108.0, 116.0, 103.0, 114.0, 105.0, 123.0, 122.0, 94.0, 104.0, 101.0, 103.0, 109.0, 115.0, 117.0, 125.0, 81.0, 115.0, 112.0, 116.0, 100.0, 108.0, 105.0, 97.0, 101.0, 105.0, 98.0, 124.0, 98.0, 101.0, 103.0, 123.0, 124.0, 122.0, 115.0, 102.0, 115.0, 116.0, 122.0, 111.0, 88.0, 99.0, 95.0, 112.0, 122.0, 131.0, 110.0, 112.0, 96.0, 108.0, 100.0, 103.0, 106.0, 119.0, 104.0, 102.0, 97.0, 125.0, 93.0, 117.0, 133.0, 112.0, 137.0, 110.0, 104.0, 120.0, 115.0, 111.0, 118.0, 113.0, 100.0, 125.0, 108.0, 109.0, 122.0, 99.0, 128.0, 105.0, 112.0, 122.0, 112.0, 114.0, 109.0, 108.0, 111.0, 113.0, 114.0, 105.0, 101.0, 110.0, 104.0, 112.0, 114.0, 109.0, 92.0, 111.0, 102.0, 91.0, 119.0, 111.0, 95.0, 107.0, 97.0, 115.0, 108.0, 124.0, 118.0, 123.0, 119.0, 122.0, 112.0, 106.0, 101.0, 93.0, 116.0, 123.0, 112.0, 120.0, 87.0, 102.0, 116.0, 113.0, 118.0, 135.0, 110.0]}, "params-norm": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [180.02467, 180.43758, 180.43758, 180.43758, 180.43777, 180.43793, 180.43831, 180.43793, 180.43802, 180.43829, 180.42958, 180.42429, 180.42331, 180.41566, 180.40419, 180.39563, 180.38898, 180.38272, 180.37767, 180.36685, 180.3624, 180.35443, 180.34731, 180.34276, 180.34059, 180.33824, 180.33578, 180.33348, 180.33122, 180.32733, 180.32587, 180.33919, 180.33737, 180.34363, 180.342, 180.33069, 180.32681, 180.32585, 180.32092, 180.32108, 180.32138, 180.32286, 180.32475, 180.31996, 180.32315, 180.32661, 180.32353, 180.32631, 180.3324, 180.33495, 180.34297, 180.3517, 180.3622, 180.37233, 180.38312, 180.39899, 180.40675, 180.41966, 180.43352, 180.4433, 180.45685, 180.47159, 180.48726, 180.50627, 180.52652, 180.54735, 180.56612, 180.58878, 180.60876, 180.63284, 180.65726, 180.68228, 180.70798, 180.73802, 180.77544, 180.79449, 180.82513, 180.85388, 180.88318, 180.90952, 180.93996, 180.9704, 181.00171, 181.03206, 181.06531, 181.1013, 181.13477, 181.15898, 181.19191, 181.22948, 181.26605, 181.30205, 181.33176, 181.36722, 181.40222, 181.43898, 181.4686, 181.50232, 181.53323, 181.56693, 181.60017, 181.63365, 181.66275, 181.69737, 181.73155, 181.76347, 181.8042, 181.83623, 181.86909, 181.90247, 181.93695, 181.96951, 182.00578, 182.04301, 182.07603, 182.11412, 182.15521, 182.18857, 182.22928, 182.26672, 182.3042, 182.34148, 182.37926, 182.41901, 182.45923, 182.49518, 182.53793, 182.57965, 182.61847, 182.65536, 182.6929, 182.72876, 182.76958, 182.80853, 182.85202, 182.88937, 182.92555, 182.96187, 182.99063, 183.02582, 183.05833, 183.08974, 183.12651, 183.16095, 183.19424, 183.233, 183.26149, 183.29265, 183.32909, 183.36882, 183.40269, 183.43456, 183.47014, 183.51022, 183.54683, 183.57953, 183.61252, 183.64738, 183.68155, 183.71558, 183.75716, 183.79567, 183.83615, 183.87654, 183.9173, 183.9584, 184.00073, 184.04141, 184.08711, 184.12192, 184.16089, 184.19904, 184.23912, 184.27597, 184.31317, 184.35162, 184.39233, 184.43021, 184.46562, 184.50061, 184.54076, 184.5798, 184.62137, 184.66426, 184.70601, 184.74544, 184.7812, 184.8163, 184.85382, 184.89362, 184.9332, 184.9715, 185.00937, 185.05093, 185.09132, 185.12502, 185.16487, 185.20316, 185.24188, 185.27464, 185.31422, 185.35551, 185.3972, 185.43919, 185.47906, 185.52074, 185.56161, 185.60054, 185.64554, 185.68713, 185.72649, 185.76546, 185.80576, 185.84767, 185.89198, 185.9361, 185.98022, 186.01895, 186.05711, 186.10294, 186.13905, 186.17926, 186.22005, 186.25861, 186.29631, 186.33633, 186.37819, 186.41498, 186.452, 186.48996, 186.52638, 186.56227, 186.59106, 186.62415, 186.66559, 186.70592, 186.74504, 186.78651, 186.83006, 186.87518, 186.91788, 186.96049, 187.00543, 187.05008, 187.09511, 187.13741, 187.17758, 187.21588, 187.25984, 187.30086, 187.34575, 187.39095, 187.43542, 187.4792, 187.51852, 187.56268, 187.60396, 187.64711, 187.68872, 187.73135, 187.77692, 187.81973, 187.86543, 187.91296, 187.96025, 188.00529, 188.04802, 188.0909, 188.13518, 188.18434, 188.22716, 188.27409, 188.32169, 188.36803, 188.41319, 188.45816, 188.50641, 188.54868, 188.59381, 188.6367, 188.68343, 188.72693, 188.77374, 188.8172, 188.86154, 188.90767, 188.95059, 188.99326, 189.04083, 189.08832, 189.13934, 189.1855, 189.2296, 189.27489, 189.32558, 189.36694, 189.41133, 189.45744, 189.50322, 189.54796, 189.59531, 189.6389, 189.68634, 189.73462, 189.78769, 189.83501, 189.88196, 189.92941, 189.97726, 190.02953, 190.08095, 190.13335, 190.18449, 190.23326, 190.28383, 190.33415, 190.38512, 190.43832, 190.49026, 190.5453, 190.59666, 190.65088, 190.70216, 190.75441, 190.80804, 190.85649, 190.90819, 190.957, 191.00778, 191.05713, 191.10803, 191.15628, 191.20445, 191.25539, 191.30585, 191.35631, 191.40929, 191.46144, 191.5153, 191.5732, 191.6273, 191.67821, 191.73494, 191.79005, 191.84462, 191.89845, 191.95538, 192.01093, 192.06554, 192.1189, 192.17081, 192.2244, 192.2774, 192.33224, 192.38445, 192.44177, 192.49707, 192.55254, 192.60464, 192.65576, 192.70808, 192.76437, 192.82317, 192.88344, 192.93953, 192.99843, 193.05219, 193.1062, 193.16641, 193.22375, 193.28175, 193.3349, 193.39145, 193.44878, 193.50717, 193.55751, 193.61333, 193.66898, 193.72675, 193.79041, 193.84534, 193.90236, 193.96567, 194.0249, 194.08501, 194.14468, 194.2052, 194.2684, 194.32666, 194.38776, 194.44768, 194.50999, 194.57324, 194.63622, 194.69333, 194.74876, 194.80455, 194.86299, 194.92128, 194.97459]}, "params-norm vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [180.02467, 180.43758, 180.43758, 180.43758, 180.43777, 180.43793, 180.43831, 180.43793, 180.43802, 180.43829, 180.42958, 180.42429, 180.42331, 180.41566, 180.40419, 180.39563, 180.38898, 180.38272, 180.37767, 180.36685, 180.3624, 180.35443, 180.34731, 180.34276, 180.34059, 180.33824, 180.33578, 180.33348, 180.33122, 180.32733, 180.32587, 180.33919, 180.33737, 180.34363, 180.342, 180.33069, 180.32681, 180.32585, 180.32092, 180.32108, 180.32138, 180.32286, 180.32475, 180.31996, 180.32315, 180.32661, 180.32353, 180.32631, 180.3324, 180.33495, 180.34297, 180.3517, 180.3622, 180.37233, 180.38312, 180.39899, 180.40675, 180.41966, 180.43352, 180.4433, 180.45685, 180.47159, 180.48726, 180.50627, 180.52652, 180.54735, 180.56612, 180.58878, 180.60876, 180.63284, 180.65726, 180.68228, 180.70798, 180.73802, 180.77544, 180.79449, 180.82513, 180.85388, 180.88318, 180.90952, 180.93996, 180.9704, 181.00171, 181.03206, 181.06531, 181.1013, 181.13477, 181.15898, 181.19191, 181.22948, 181.26605, 181.30205, 181.33176, 181.36722, 181.40222, 181.43898, 181.4686, 181.50232, 181.53323, 181.56693, 181.60017, 181.63365, 181.66275, 181.69737, 181.73155, 181.76347, 181.8042, 181.83623, 181.86909, 181.90247, 181.93695, 181.96951, 182.00578, 182.04301, 182.07603, 182.11412, 182.15521, 182.18857, 182.22928, 182.26672, 182.3042, 182.34148, 182.37926, 182.41901, 182.45923, 182.49518, 182.53793, 182.57965, 182.61847, 182.65536, 182.6929, 182.72876, 182.76958, 182.80853, 182.85202, 182.88937, 182.92555, 182.96187, 182.99063, 183.02582, 183.05833, 183.08974, 183.12651, 183.16095, 183.19424, 183.233, 183.26149, 183.29265, 183.32909, 183.36882, 183.40269, 183.43456, 183.47014, 183.51022, 183.54683, 183.57953, 183.61252, 183.64738, 183.68155, 183.71558, 183.75716, 183.79567, 183.83615, 183.87654, 183.9173, 183.9584, 184.00073, 184.04141, 184.08711, 184.12192, 184.16089, 184.19904, 184.23912, 184.27597, 184.31317, 184.35162, 184.39233, 184.43021, 184.46562, 184.50061, 184.54076, 184.5798, 184.62137, 184.66426, 184.70601, 184.74544, 184.7812, 184.8163, 184.85382, 184.89362, 184.9332, 184.9715, 185.00937, 185.05093, 185.09132, 185.12502, 185.16487, 185.20316, 185.24188, 185.27464, 185.31422, 185.35551, 185.3972, 185.43919, 185.47906, 185.52074, 185.56161, 185.60054, 185.64554, 185.68713, 185.72649, 185.76546, 185.80576, 185.84767, 185.89198, 185.9361, 185.98022, 186.01895, 186.05711, 186.10294, 186.13905, 186.17926, 186.22005, 186.25861, 186.29631, 186.33633, 186.37819, 186.41498, 186.452, 186.48996, 186.52638, 186.56227, 186.59106, 186.62415, 186.66559, 186.70592, 186.74504, 186.78651, 186.83006, 186.87518, 186.91788, 186.96049, 187.00543, 187.05008, 187.09511, 187.13741, 187.17758, 187.21588, 187.25984, 187.30086, 187.34575, 187.39095, 187.43542, 187.4792, 187.51852, 187.56268, 187.60396, 187.64711, 187.68872, 187.73135, 187.77692, 187.81973, 187.86543, 187.91296, 187.96025, 188.00529, 188.04802, 188.0909, 188.13518, 188.18434, 188.22716, 188.27409, 188.32169, 188.36803, 188.41319, 188.45816, 188.50641, 188.54868, 188.59381, 188.6367, 188.68343, 188.72693, 188.77374, 188.8172, 188.86154, 188.90767, 188.95059, 188.99326, 189.04083, 189.08832, 189.13934, 189.1855, 189.2296, 189.27489, 189.32558, 189.36694, 189.41133, 189.45744, 189.50322, 189.54796, 189.59531, 189.6389, 189.68634, 189.73462, 189.78769, 189.83501, 189.88196, 189.92941, 189.97726, 190.02953, 190.08095, 190.13335, 190.18449, 190.23326, 190.28383, 190.33415, 190.38512, 190.43832, 190.49026, 190.5453, 190.59666, 190.65088, 190.70216, 190.75441, 190.80804, 190.85649, 190.90819, 190.957, 191.00778, 191.05713, 191.10803, 191.15628, 191.20445, 191.25539, 191.30585, 191.35631, 191.40929, 191.46144, 191.5153, 191.5732, 191.6273, 191.67821, 191.73494, 191.79005, 191.84462, 191.89845, 191.95538, 192.01093, 192.06554, 192.1189, 192.17081, 192.2244, 192.2774, 192.33224, 192.38445, 192.44177, 192.49707, 192.55254, 192.60464, 192.65576, 192.70808, 192.76437, 192.82317, 192.88344, 192.93953, 192.99843, 193.05219, 193.1062, 193.16641, 193.22375, 193.28175, 193.3349, 193.39145, 193.44878, 193.50717, 193.55751, 193.61333, 193.66898, 193.72675, 193.79041, 193.84534, 193.90236, 193.96567, 194.0249, 194.08501, 194.14468, 194.2052, 194.2684, 194.32666, 194.38776, 194.44768, 194.50999, 194.57324, 194.63622, 194.69333, 194.74876, 194.80455, 194.86299, 194.92128, 194.97459]}, "iteration-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [26.15537, 1.59225, 1.58677, 1.61174, 1.60131, 1.58979, 1.6009, 1.60255, 1.59989, 1.59397, 1.59991, 1.60879, 1.59752, 1.58326, 1.60593, 1.58196, 1.58281, 1.58285, 1.65512, 1.58951, 1.57778, 1.59099, 1.59905, 1.5964, 1.60421, 1.59987, 1.60383, 1.59456, 1.59474, 1.60292, 1.59587, 1.59615, 1.59953, 1.68491, 1.61405, 1.61646, 1.76204, 1.6157, 1.60582, 1.60949, 1.60517, 1.60169, 1.5944, 1.59771, 1.59812, 1.61186, 1.60798, 1.59786, 1.69134, 1.607, 1.62116, 1.61495, 1.61958, 1.61282, 1.60615, 1.61947, 1.6053, 1.59812, 1.60103, 1.61637, 1.60915, 1.61703, 1.61268, 1.61077, 1.61236, 1.61876, 1.60773, 1.69396, 1.60939, 1.61301, 1.62827, 1.61429, 1.61159, 1.60859, 1.61405, 1.62895, 1.61614, 1.61446, 1.60675, 1.61067, 1.61896, 1.61461, 1.61244, 1.60436, 1.6079, 1.619, 1.61303, 1.61117, 1.61223, 1.60766, 1.62186, 1.60682, 1.60832, 1.60625, 1.60469, 1.61342, 1.60768, 1.60669, 1.59722, 1.69938, 1.61072, 1.61909, 1.61007, 1.6046, 1.60277, 1.61264, 1.61634, 1.61492, 1.61043, 1.62152, 1.61505, 1.61393, 1.61336, 1.61268, 1.61629, 1.61635, 1.62076, 1.61243, 1.61515, 1.61244, 1.61769, 1.61729, 1.60493, 1.60897, 1.61012, 1.61259, 1.6206, 1.60935, 1.61072, 1.61412, 1.62132, 1.61512, 1.61556, 1.61045, 1.6109, 1.61406, 1.61499, 1.60648, 1.62368, 1.61793, 1.62077, 1.61115, 1.607, 1.60097, 1.60715, 1.61148, 1.61713, 1.61144, 1.62249, 1.61481, 1.61115, 1.6037, 1.61119, 1.60767, 1.6172, 1.61279, 1.60574, 1.60707, 1.60482, 1.60401, 1.61113, 1.61346, 1.60704, 1.61142, 1.60677, 1.60612, 1.59885, 1.60751, 1.60394, 1.60565, 1.60074, 1.60646, 1.60139, 1.60114, 1.60502, 1.59931, 1.59106, 1.59528, 1.59562, 1.60655, 1.61019, 1.60604, 1.60255, 1.59481, 1.59218, 1.59628, 1.58975, 1.60275, 1.59914, 1.59723, 1.59728, 1.58386, 1.61425, 1.60353, 1.60061, 1.60375, 1.61192, 1.61512, 1.60494, 1.59982, 1.59392, 1.59773, 1.59899, 1.60034, 1.59034, 1.59986, 1.59404, 1.59171, 1.58924, 1.58292, 1.59951, 1.58972, 1.60076, 1.59525, 1.60354, 1.60474, 1.6007, 1.60461, 1.60303, 1.68738, 1.61462, 1.6112, 1.60314, 1.60468, 1.60954, 1.61515, 1.60446, 1.60607, 1.60574, 1.60376, 1.60767, 1.60168, 1.60809, 1.60685, 1.59979, 1.59981, 1.59996, 1.60233, 1.61191, 1.60192, 1.60578, 1.61979, 1.6159, 1.61226, 1.6128, 1.60991, 1.62187, 1.61382, 1.60853, 1.61365, 1.6207, 1.63823, 1.61317, 1.60999, 1.6096, 1.6053, 1.62098, 1.60515, 1.61012, 1.60877, 1.61097, 1.62766, 1.61189, 1.61276, 1.61683, 1.61267, 1.62231, 1.61022, 1.61488, 1.61227, 1.60799, 1.61989, 1.61118, 1.60947, 1.61635, 1.60971, 1.61707, 1.61308, 1.60535, 1.61359, 1.60892, 1.61075, 1.60793, 1.60987, 1.61295, 1.61056, 1.60924, 1.61593, 1.60828, 1.62137, 1.60777, 1.6163, 1.61976, 1.60496, 1.61232, 1.60943, 1.60387, 1.61497, 1.60986, 1.61254, 1.61053, 1.61641, 1.62112, 1.60996, 1.62043, 1.61238, 1.61482, 1.61865, 1.61289, 1.61175, 1.61784, 1.61203, 1.6132, 1.60843, 1.61847, 1.61033, 1.6185, 1.61766, 1.6264, 1.62151, 1.62048, 1.61539, 1.61807, 1.61346, 1.60979, 1.61291, 1.61433, 1.61137, 1.616, 1.60714, 1.6154, 1.61351, 1.60767, 1.60384, 1.60001, 1.59921, 1.60103, 1.60417, 1.60117, 1.59284, 1.60079, 1.59673, 1.59125, 1.59593, 1.59394, 1.59478, 1.59263, 1.59408, 1.59955, 1.66468, 1.59302, 1.59156, 1.59525, 1.62673, 1.61448, 1.60772, 1.60098, 1.6066, 1.62998, 1.62933, 1.6147, 1.61299, 1.61044, 1.62556, 1.61734, 1.61197, 1.61149, 1.61287, 1.62523, 1.61258, 1.60355, 1.6117, 1.61092, 1.60763, 1.61177, 1.61161, 1.6207, 1.61553, 1.62712, 1.62883, 1.6176, 1.62185, 1.60923, 1.61676, 1.62142, 1.62074, 1.61866, 1.61459, 1.59668, 1.61134, 1.60642, 1.60975, 1.61506, 1.60601, 1.62434, 1.61024, 1.61231, 1.61973, 1.61419, 1.61888]}, "lm loss validation": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [5.5974]}, "lm loss validation vs samples": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [5.5974]}, "lm loss validation ppl": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [269.72311]}, "lm loss validation ppl vs samples": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [269.72311]}}
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_native_fp8_tp_pp_sp/golden_values_lts_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_native_fp8_tp_pp_sp/golden_values_lts_dgx_a100.json
deleted file mode 100644
index f451bade907..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_native_fp8_tp_pp_sp/golden_values_lts_dgx_a100.json
+++ /dev/null
@@ -1,1223 +0,0 @@
-{
- "lm loss": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 10.8433,
- 10.86044,
- 10.85061,
- 10.84734,
- 10.76548,
- 10.77301,
- 10.67965,
- 10.52932,
- 10.38131,
- 10.2974,
- 9.93358,
- 10.03588,
- 10.0431,
- 9.75389,
- 9.86963,
- 9.57405,
- 9.5096,
- 9.70629,
- 9.43192,
- 9.37522,
- 9.284,
- 9.1822,
- 9.20626,
- 9.02414,
- 9.21657,
- 9.08442,
- 9.17322,
- 9.18366,
- 9.31703,
- 9.00597,
- 8.94641,
- 9.06062,
- 9.05821,
- 8.66725,
- 8.7293,
- 8.75948,
- 8.69311,
- 8.74107,
- 8.66315,
- 8.7692,
- 8.66419,
- 8.85248,
- 8.83414,
- 8.49646,
- 8.38634,
- 8.42674,
- 8.48452,
- 8.37818,
- 8.42615,
- 8.57789,
- 8.36141,
- 8.18501,
- 8.21689,
- 8.21279,
- 8.25813,
- 7.90478,
- 8.08492,
- 7.88061,
- 8.2332,
- 8.21498,
- 7.98981,
- 7.95442,
- 7.90402,
- 7.72141,
- 7.72532,
- 7.62803,
- 7.49905,
- 7.88742,
- 7.68058,
- 7.43268,
- 7.72562,
- 7.75354,
- 7.52404,
- 7.283,
- 7.43599,
- 7.32465,
- 7.44892,
- 7.21194,
- 7.61927,
- 7.26538,
- 7.33426,
- 7.19855,
- 7.19861,
- 7.40556,
- 7.15878,
- 7.26703,
- 6.98161,
- 6.98947,
- 7.02642,
- 7.12381,
- 6.81041,
- 6.97196,
- 7.07748,
- 6.98749,
- 6.86311,
- 6.74439,
- 6.97854,
- 7.04679,
- 6.69093,
- 6.57072,
- 6.71136,
- 6.73236,
- 6.71979,
- 6.7272,
- 6.64643,
- 6.39789,
- 6.62843,
- 6.6105,
- 6.43797,
- 6.61969,
- 6.73555,
- 6.60277,
- 6.71805,
- 6.68657,
- 6.6186,
- 6.49971,
- 6.59035,
- 6.4017,
- 6.65875,
- 6.24131,
- 6.24596,
- 6.29903,
- 6.3883,
- 6.34534,
- 6.44873,
- 6.29075,
- 6.33714,
- 6.23406,
- 6.2,
- 6.39474,
- 6.32229,
- 6.3185,
- 6.15978,
- 6.1549,
- 6.23433,
- 6.38093,
- 6.19594,
- 6.14735,
- 6.17407,
- 6.10894,
- 6.05539,
- 6.06758,
- 6.24744,
- 6.40151,
- 6.24847,
- 6.28705,
- 6.08923,
- 6.16761,
- 5.99264,
- 6.01994,
- 5.94543,
- 6.23683,
- 6.17643,
- 5.95473,
- 5.77213,
- 6.11864,
- 5.84026,
- 6.09588,
- 5.77668,
- 6.15345,
- 6.13462,
- 6.07869,
- 5.91897,
- 6.10742,
- 5.93962,
- 6.19145,
- 5.88782,
- 5.78511,
- 5.77656,
- 5.68132,
- 6.00891,
- 5.98944,
- 6.06282,
- 5.88285,
- 6.03259,
- 5.962,
- 5.98778,
- 5.9836,
- 5.94381,
- 5.82984,
- 5.93888,
- 5.60808,
- 5.69371,
- 5.87962,
- 5.83333,
- 5.85729,
- 5.75536,
- 5.82874,
- 5.71799,
- 5.55439,
- 5.71537,
- 5.61547,
- 5.82285,
- 5.59518,
- 5.70178,
- 5.70193,
- 5.89973,
- 5.64349,
- 5.84024,
- 5.7335,
- 5.86261,
- 5.32628,
- 5.8955,
- 5.87228,
- 5.85021,
- 5.41476,
- 5.40861,
- 5.62304,
- 5.59442,
- 5.48225,
- 5.575,
- 5.67376,
- 5.47435,
- 5.74214,
- 5.50969,
- 5.58812,
- 5.62033,
- 5.62505,
- 5.51148,
- 5.61484,
- 5.66881,
- 5.67915,
- 5.58549,
- 5.66219,
- 5.3723,
- 5.68302,
- 5.62277,
- 5.42565,
- 5.58011,
- 5.62513,
- 5.55422,
- 5.33956,
- 5.53529,
- 5.48344,
- 5.47864,
- 5.38058,
- 5.55141,
- 5.60161,
- 5.38117,
- 5.51959,
- 5.48208,
- 5.32799,
- 5.5011,
- 5.40461,
- 5.44282,
- 5.31546,
- 5.06338,
- 5.47685,
- 5.56844,
- 5.71304,
- 5.41518,
- 5.60351,
- 5.6332,
- 5.23378,
- 5.2708,
- 5.39252,
- 5.39433,
- 5.32688,
- 5.49317,
- 5.17959,
- 5.29648,
- 5.24403,
- 5.37611,
- 5.25199,
- 5.44219,
- 5.53486,
- 5.30852,
- 5.43435,
- 5.33672,
- 5.07326,
- 5.30935,
- 5.25295,
- 5.30193,
- 5.1137,
- 5.2765,
- 5.26065,
- 5.4709,
- 5.15537,
- 5.26079,
- 5.21266,
- 5.35725,
- 4.98376,
- 4.91218,
- 5.32196,
- 5.39014,
- 5.22652,
- 5.31696,
- 5.10431,
- 5.16315,
- 5.26294,
- 5.06551,
- 5.26331,
- 5.065,
- 5.34523,
- 5.24779,
- 5.14999,
- 5.23909,
- 5.03872,
- 5.31514,
- 5.05221,
- 5.0306,
- 5.1433,
- 5.11124,
- 5.27385,
- 5.15503,
- 5.27616,
- 5.09274,
- 5.09304,
- 5.24611,
- 5.3273,
- 5.25057,
- 5.19665,
- 5.14298,
- 5.28995,
- 4.95043,
- 5.21059,
- 5.09648,
- 5.3046,
- 5.17404,
- 5.18934,
- 5.11588,
- 4.9846,
- 4.99496,
- 5.2241,
- 5.31583,
- 5.10197,
- 5.05823,
- 4.91741,
- 5.12453,
- 5.11774,
- 4.93535,
- 5.34519,
- 5.02909,
- 5.10301,
- 5.16644,
- 5.00345,
- 5.0682,
- 5.07218,
- 4.998,
- 5.08202,
- 5.1646,
- 4.9791,
- 5.18399,
- 4.93201,
- 4.92304,
- 5.06461,
- 4.99669,
- 4.91342,
- 4.77777,
- 4.94601,
- 5.1212,
- 5.01688,
- 5.02069,
- 5.33321,
- 4.96044,
- 4.99679,
- 5.05127,
- 4.81294,
- 4.73819,
- 4.99932,
- 5.04478,
- 4.87544,
- 4.96009,
- 5.05348,
- 5.02688,
- 4.81746,
- 4.8976,
- 4.91081,
- 4.83628,
- 4.7431,
- 5.01539,
- 4.75603,
- 5.21485,
- 4.78994,
- 4.99325,
- 4.73922,
- 4.78654,
- 4.81871,
- 4.65038,
- 4.65649,
- 4.84773,
- 4.80858,
- 4.80152,
- 4.92483,
- 4.88939,
- 4.93094,
- 4.77431,
- 4.88226,
- 4.73507,
- 4.91472,
- 4.95863,
- 4.87414,
- 4.70518,
- 4.78362,
- 4.90312,
- 4.71195,
- 4.86873,
- 4.69654,
- 4.69772,
- 4.64816
- ]
- },
- "num-zeros": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 57.0,
- 74.0,
- 67.0,
- 65.0,
- 85.0,
- 70.0,
- 66.0,
- 105.0,
- 87.0,
- 112.0,
- 112.0,
- 159.0,
- 132.0,
- 158.0,
- 146.0,
- 138.0,
- 187.0,
- 176.0,
- 186.0,
- 203.0,
- 162.0,
- 136.0,
- 174.0,
- 164.0,
- 210.0,
- 165.0,
- 187.0,
- 193.0,
- 177.0,
- 161.0,
- 157.0,
- 191.0,
- 160.0,
- 188.0,
- 128.0,
- 177.0,
- 157.0,
- 199.0,
- 163.0,
- 171.0,
- 152.0,
- 172.0,
- 179.0,
- 153.0,
- 165.0,
- 172.0,
- 169.0,
- 214.0,
- 170.0,
- 202.0,
- 205.0,
- 185.0,
- 192.0,
- 154.0,
- 196.0,
- 180.0,
- 181.0,
- 160.0,
- 253.0,
- 233.0,
- 194.0,
- 215.0,
- 189.0,
- 176.0,
- 209.0,
- 253.0,
- 183.0,
- 190.0,
- 214.0,
- 201.0,
- 234.0,
- 238.0,
- 198.0,
- 225.0,
- 197.0,
- 205.0,
- 233.0,
- 208.0,
- 283.0,
- 232.0,
- 231.0,
- 237.0,
- 195.0,
- 234.0,
- 241.0,
- 191.0,
- 176.0,
- 191.0,
- 168.0,
- 204.0,
- 199.0,
- 194.0,
- 218.0,
- 214.0,
- 225.0,
- 174.0,
- 208.0,
- 204.0,
- 177.0,
- 144.0,
- 155.0,
- 141.0,
- 187.0,
- 152.0,
- 168.0,
- 122.0,
- 136.0,
- 172.0,
- 124.0,
- 193.0,
- 174.0,
- 134.0,
- 193.0,
- 158.0,
- 124.0,
- 171.0,
- 159.0,
- 113.0,
- 144.0,
- 157.0,
- 125.0,
- 146.0,
- 107.0,
- 136.0,
- 114.0,
- 108.0,
- 134.0,
- 128.0,
- 117.0,
- 126.0,
- 134.0,
- 122.0,
- 131.0,
- 124.0,
- 138.0,
- 107.0,
- 145.0,
- 103.0,
- 97.0,
- 120.0,
- 134.0,
- 127.0,
- 136.0,
- 147.0,
- 132.0,
- 116.0,
- 114.0,
- 134.0,
- 118.0,
- 118.0,
- 97.0,
- 132.0,
- 115.0,
- 135.0,
- 114.0,
- 87.0,
- 87.0,
- 122.0,
- 100.0,
- 102.0,
- 133.0,
- 121.0,
- 124.0,
- 112.0,
- 100.0,
- 115.0,
- 107.0,
- 109.0,
- 92.0,
- 99.0,
- 123.0,
- 123.0,
- 94.0,
- 111.0,
- 129.0,
- 106.0,
- 103.0,
- 121.0,
- 114.0,
- 128.0,
- 132.0,
- 98.0,
- 102.0,
- 116.0,
- 112.0,
- 98.0,
- 84.0,
- 120.0,
- 99.0,
- 92.0,
- 119.0,
- 109.0,
- 129.0,
- 115.0,
- 123.0,
- 76.0,
- 74.0,
- 77.0,
- 99.0,
- 108.0,
- 126.0,
- 102.0,
- 91.0,
- 107.0,
- 112.0,
- 107.0,
- 100.0,
- 93.0,
- 108.0,
- 106.0,
- 93.0,
- 96.0,
- 107.0,
- 110.0,
- 90.0,
- 117.0,
- 107.0,
- 102.0,
- 111.0,
- 102.0,
- 98.0,
- 99.0,
- 108.0,
- 96.0,
- 90.0,
- 95.0,
- 101.0,
- 114.0,
- 113.0,
- 111.0,
- 88.0,
- 90.0,
- 104.0,
- 93.0,
- 101.0,
- 94.0,
- 90.0,
- 101.0,
- 116.0,
- 99.0,
- 99.0,
- 121.0,
- 98.0,
- 127.0,
- 120.0,
- 111.0,
- 85.0,
- 106.0,
- 110.0,
- 129.0,
- 109.0,
- 98.0,
- 127.0,
- 89.0,
- 116.0,
- 107.0,
- 115.0,
- 114.0,
- 129.0,
- 120.0,
- 99.0,
- 117.0,
- 102.0,
- 111.0,
- 114.0,
- 91.0,
- 120.0,
- 101.0,
- 114.0,
- 105.0,
- 117.0,
- 100.0,
- 107.0,
- 96.0,
- 98.0,
- 98.0,
- 105.0,
- 102.0,
- 117.0,
- 92.0,
- 101.0,
- 99.0,
- 105.0,
- 128.0,
- 91.0,
- 96.0,
- 105.0,
- 109.0,
- 110.0,
- 101.0,
- 99.0,
- 95.0,
- 111.0,
- 109.0,
- 94.0,
- 89.0,
- 117.0,
- 102.0,
- 104.0,
- 120.0,
- 109.0,
- 89.0,
- 114.0,
- 115.0,
- 101.0,
- 87.0,
- 75.0,
- 119.0,
- 116.0,
- 122.0,
- 94.0,
- 114.0,
- 86.0,
- 120.0,
- 110.0,
- 116.0,
- 106.0,
- 134.0,
- 100.0,
- 129.0,
- 116.0,
- 100.0,
- 107.0,
- 107.0,
- 131.0,
- 109.0,
- 103.0,
- 110.0,
- 112.0,
- 123.0,
- 84.0,
- 99.0,
- 99.0,
- 116.0,
- 107.0,
- 118.0,
- 104.0,
- 137.0,
- 105.0,
- 101.0,
- 123.0,
- 119.0,
- 118.0,
- 123.0,
- 100.0,
- 110.0,
- 126.0,
- 116.0,
- 108.0,
- 102.0,
- 114.0,
- 112.0,
- 114.0,
- 101.0,
- 124.0,
- 96.0,
- 139.0,
- 120.0,
- 109.0,
- 119.0,
- 115.0,
- 105.0,
- 111.0,
- 96.0,
- 121.0,
- 119.0,
- 87.0,
- 95.0,
- 94.0,
- 104.0,
- 124.0,
- 124.0,
- 90.0,
- 106.0,
- 102.0,
- 114.0,
- 108.0,
- 106.0,
- 124.0,
- 110.0,
- 122.0,
- 118.0,
- 151.0,
- 122.0,
- 90.0,
- 116.0,
- 114.0,
- 114.0,
- 108.0,
- 132.0,
- 124.0,
- 97.0,
- 109.0,
- 111.0,
- 104.0,
- 114.0,
- 107.0,
- 111.0,
- 124.0,
- 123.0
- ]
- },
- "iteration-time": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 25.403,
- 1.36901,
- 1.32789,
- 1.35574,
- 1.34115,
- 1.3441,
- 1.34468,
- 1.33177,
- 1.31979,
- 1.35178,
- 1.32886,
- 1.33111,
- 1.34487,
- 1.3273,
- 1.34186,
- 1.34676,
- 1.32736,
- 1.33277,
- 1.34223,
- 1.3278,
- 1.33346,
- 1.33096,
- 1.35516,
- 1.33304,
- 1.34537,
- 1.32876,
- 1.33649,
- 1.33633,
- 1.32353,
- 1.31875,
- 1.3419,
- 1.32045,
- 1.31879,
- 1.33556,
- 1.32183,
- 1.33539,
- 1.33467,
- 1.31998,
- 1.34002,
- 1.32021,
- 1.31828,
- 1.34009,
- 1.32231,
- 1.32892,
- 1.34004,
- 1.34102,
- 1.33151,
- 1.34109,
- 1.34054,
- 1.32736,
- 1.33084,
- 1.33943,
- 1.33163,
- 1.34679,
- 1.3493,
- 1.34079,
- 1.34467,
- 1.36311,
- 1.36072,
- 1.33909,
- 1.35483,
- 1.34492,
- 1.3287,
- 1.34086,
- 1.34508,
- 1.3343,
- 1.33604,
- 1.34284,
- 1.32854,
- 1.33619,
- 1.34638,
- 1.32885,
- 1.34151,
- 1.3311,
- 1.32446,
- 1.33974,
- 1.33736,
- 1.34269,
- 1.34906,
- 1.34377,
- 1.33473,
- 1.343,
- 1.34132,
- 1.33943,
- 1.341,
- 1.33716,
- 1.32547,
- 1.3371,
- 1.33437,
- 1.32555,
- 1.33543,
- 1.33621,
- 1.3215,
- 1.33266,
- 1.31534,
- 1.32595,
- 1.32734,
- 1.32015,
- 1.32492,
- 1.31855,
- 1.33359,
- 1.66786,
- 1.31743,
- 1.32696,
- 1.33579,
- 1.32251,
- 1.33627,
- 1.32576,
- 1.32653,
- 1.34276,
- 1.31981,
- 1.33486,
- 1.32873,
- 1.32028,
- 1.32507,
- 1.32211,
- 1.32709,
- 1.33106,
- 1.3183,
- 1.33122,
- 1.31664,
- 1.33108,
- 1.34366,
- 1.31693,
- 1.32452,
- 1.32835,
- 1.31419,
- 1.32546,
- 1.31977,
- 1.3262,
- 1.33176,
- 1.31601,
- 1.33275,
- 1.32058,
- 1.32678,
- 1.32324,
- 1.317,
- 1.3437,
- 1.31867,
- 1.32231,
- 1.32286,
- 1.3207,
- 1.33345,
- 1.3182,
- 1.3252,
- 1.33531,
- 1.32194,
- 1.33212,
- 1.32008,
- 1.33452,
- 1.32165,
- 1.31727,
- 1.33005,
- 1.31945,
- 1.32647,
- 1.32811,
- 1.31652,
- 1.33327,
- 1.32326,
- 1.3281,
- 1.32732,
- 1.31953,
- 1.33364,
- 1.33098,
- 1.45235,
- 1.32995,
- 1.3361,
- 1.32739,
- 1.33322,
- 1.33125,
- 1.32348,
- 1.33073,
- 1.32539,
- 1.3246,
- 1.32195,
- 1.31924,
- 1.32845,
- 1.32487,
- 1.32061,
- 1.31966,
- 1.31579,
- 1.3277,
- 1.32271,
- 1.32605,
- 1.32261,
- 1.32156,
- 1.32647,
- 1.31813,
- 1.3288,
- 1.32253,
- 1.3231,
- 1.32536,
- 1.31897,
- 1.32751,
- 1.32578,
- 1.32909,
- 1.33532,
- 1.33326,
- 1.33105,
- 1.32709,
- 1.33676,
- 1.33904,
- 1.3295,
- 1.32664,
- 1.35848,
- 1.32898,
- 1.33485,
- 1.33037,
- 1.32875,
- 1.33465,
- 1.33401,
- 1.33837,
- 1.3293,
- 1.33445,
- 1.34421,
- 1.32972,
- 1.33724,
- 1.34139,
- 1.33243,
- 1.33291,
- 1.33723,
- 1.33388,
- 1.32865,
- 1.33127,
- 1.33318,
- 1.33165,
- 1.34222,
- 1.33634,
- 1.3365,
- 1.33796,
- 1.34048,
- 1.32719,
- 1.33315,
- 1.33195,
- 1.32817,
- 1.3339,
- 1.32838,
- 1.33821,
- 1.3587,
- 1.34806,
- 1.35603,
- 1.33734,
- 1.32992,
- 1.33619,
- 1.33521,
- 1.33764,
- 1.33246,
- 1.33105,
- 1.332,
- 1.33518,
- 1.33735,
- 1.32633,
- 1.33962,
- 1.33025,
- 1.33331,
- 1.332,
- 1.33835,
- 1.32945,
- 1.33547,
- 1.3322,
- 1.32881,
- 1.33281,
- 1.3315,
- 1.33043,
- 1.32953,
- 1.3237,
- 1.3313,
- 1.32987,
- 1.32727,
- 1.33098,
- 1.3258,
- 1.32451,
- 1.33015,
- 1.32723,
- 1.32992,
- 1.32266,
- 1.31868,
- 1.32973,
- 1.32567,
- 1.32905,
- 1.3309,
- 1.33101,
- 1.33208,
- 1.3296,
- 1.32644,
- 1.33636,
- 1.33075,
- 1.32271,
- 1.33314,
- 1.32512,
- 1.32355,
- 1.32919,
- 1.32649,
- 1.33633,
- 1.32914,
- 1.32897,
- 1.33177,
- 1.32609,
- 1.32965,
- 1.33361,
- 1.32785,
- 1.33132,
- 1.33811,
- 1.32252,
- 1.33111,
- 1.3308,
- 1.32999,
- 1.32903,
- 1.32462,
- 1.32932,
- 1.33299,
- 1.32873,
- 1.33539,
- 1.33319,
- 1.32521,
- 1.33441,
- 1.33404,
- 1.33913,
- 1.3349,
- 1.33111,
- 1.3365,
- 1.33511,
- 1.32963,
- 1.33379,
- 1.33388,
- 1.32718,
- 1.33768,
- 1.32834,
- 1.32755,
- 1.33517,
- 1.32821,
- 1.32989,
- 1.32599,
- 1.32244,
- 1.33073,
- 1.32566,
- 1.32905,
- 1.32964,
- 1.32515,
- 1.32781,
- 1.32553,
- 1.33138,
- 1.33053,
- 1.32261,
- 1.33906,
- 1.32748,
- 1.31974,
- 1.33166,
- 1.32414,
- 1.3312,
- 1.32577,
- 1.32043,
- 1.33388,
- 1.32097,
- 1.32899,
- 1.32974,
- 1.32268,
- 1.32709,
- 1.32536,
- 1.32531,
- 1.32299,
- 1.32853,
- 1.32355,
- 1.3324,
- 1.3289,
- 1.32327,
- 1.32737,
- 1.45318,
- 1.32088,
- 1.32958,
- 1.32066,
- 1.32821,
- 1.32819,
- 1.32165,
- 1.33189,
- 1.32339,
- 1.33049,
- 1.32136,
- 1.32188,
- 1.32441,
- 1.32573,
- 1.3288,
- 1.32306,
- 1.32552,
- 1.32893,
- 1.31947,
- 1.32236,
- 1.31683,
- 1.33123,
- 1.32665,
- 1.31857,
- 1.32751,
- 1.32303,
- 1.33184,
- 1.32535,
- 1.32112,
- 1.32827,
- 1.3264,
- 1.32321,
- 1.3315
- ]
- }
-}
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_cp2_current_scaling_native_fp8_tp_fsdp/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_cp2_current_scaling_native_fp8_tp_fsdp/model_config.yaml
deleted file mode 100644
index ac5ff6cfbbf..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_cp2_current_scaling_native_fp8_tp_fsdp/model_config.yaml
+++ /dev/null
@@ -1,56 +0,0 @@
-ENV_VARS:
- NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
- CUBLAS_WORKSPACE_CONFIG: :4096:8
- NCCL_ALGO: Ring
-MODEL_ARGS:
- --num-layers: 12
- --hidden-size: 512
- --num-attention-heads: 8
- --log-params-norm: true
- --log-num-zeros-in-grad: true
- --log-validation-ppl-to-tensorboard: true
- --log-timers-to-tensorboard: true
- --tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
- --seq-length: 1024
- --max-position-embeddings: 1024
- --train-iters: 2000
- --timing-log-level: 0
- --lr-decay-iters: 320000
- --save: ${CHECKPOINT_SAVE_PATH}
- --load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
- --split: 949,50,1
- --distributed-backend: nccl
- --lr: 0.00015
- --lr-decay-style: cosine
- --min-lr: 1.0e-5
- --weight-decay: 1e-2
- --clip-grad: 1.0
- --lr-warmup-fraction: .01
- --log-interval: 1
- --save-interval: 10000
- --eval-interval: 1000
- --eval-iters: 10
- --transformer-impl: transformer_engine
- --tensor-model-parallel-size: 2
- --sequence-parallel: true
- --use-megatron-fsdp: true
- --data-parallel-sharding-strategy: optim_grads_params
- --no-gradient-accumulation-fusion: true
- --use-distributed-optimizer: true
- --deterministic-mode: true
- --fp8-format: hybrid
- --fp8-recipe: tensorwise
- --first-last-layers-bf16: true
- --fp8-param-gather: true
- --attention-softmax-in-fp32: true
- --ckpt-format: torch_dist
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
- --data-cache-path: ${DATA_CACHE_PATH}
- --bf16: true
- --log-memory-to-tensorboard: true
-TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/golden_values_dev_dgx_h100.json
deleted file mode 100644
index 0322fc3db23..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/golden_values_dev_dgx_h100.json
+++ /dev/null
@@ -1,2042 +0,0 @@
-{
- "lm loss": {
- "start_step": 1,
- "end_step": 2000,
- "step_interval": 5,
- "values": {
- "1": 10.84749,
- "5": 10.85081,
- "10": 10.84524,
- "15": 10.77884,
- "20": 10.6881,
- "25": 10.55833,
- "30": 10.26993,
- "35": 10.23479,
- "40": 10.06687,
- "45": 10.00575,
- "50": 9.94171,
- "55": 9.73746,
- "60": 9.64473,
- "65": 9.59425,
- "70": 9.42245,
- "75": 9.23951,
- "80": 9.35863,
- "85": 9.39876,
- "90": 9.41238,
- "95": 9.41795,
- "100": 9.35646,
- "105": 9.37485,
- "110": 9.36517,
- "115": 9.32133,
- "120": 9.08081,
- "125": 9.21649,
- "130": 9.13834,
- "135": 8.88684,
- "140": 8.87065,
- "145": 9.10362,
- "150": 9.00925,
- "155": 9.13594,
- "160": 8.89055,
- "165": 8.92906,
- "170": 9.04618,
- "175": 8.70851,
- "180": 8.74263,
- "185": 8.88301,
- "190": 8.53632,
- "195": 8.59356,
- "200": 8.73544,
- "205": 8.22364,
- "210": 8.58241,
- "215": 8.56524,
- "220": 8.54641,
- "225": 8.57187,
- "230": 8.50022,
- "235": 8.56346,
- "240": 8.67349,
- "245": 8.59931,
- "250": 8.426,
- "255": 8.29932,
- "260": 8.24366,
- "265": 8.3358,
- "270": 8.26862,
- "275": 8.01255,
- "280": 7.75947,
- "285": 8.07407,
- "290": 8.13207,
- "295": 7.96997,
- "300": 7.85604,
- "305": 8.00054,
- "310": 7.97828,
- "315": 7.83127,
- "320": 7.56903,
- "325": 7.79981,
- "330": 7.75554,
- "335": 7.70069,
- "340": 7.75965,
- "345": 7.61082,
- "350": 7.65708,
- "355": 7.76554,
- "360": 7.63029,
- "365": 7.62108,
- "370": 7.46556,
- "375": 7.53138,
- "380": 7.38617,
- "385": 7.39583,
- "390": 7.2421,
- "395": 7.43998,
- "400": 7.29827,
- "405": 7.26279,
- "410": 7.31687,
- "415": 7.33266,
- "420": 7.41099,
- "425": 7.31666,
- "430": 7.2054,
- "435": 7.01973,
- "440": 7.06234,
- "445": 6.9961,
- "450": 7.01705,
- "455": 7.03277,
- "460": 7.0626,
- "465": 7.02948,
- "470": 7.04327,
- "475": 7.16618,
- "480": 7.02531,
- "485": 6.92781,
- "490": 6.8209,
- "495": 6.62969,
- "500": 6.65035,
- "505": 6.61731,
- "510": 6.81805,
- "515": 6.44492,
- "520": 6.85683,
- "525": 6.67294,
- "530": 6.62303,
- "535": 6.74878,
- "540": 6.58322,
- "545": 6.81205,
- "550": 6.52648,
- "555": 6.45816,
- "560": 6.58161,
- "565": 6.51314,
- "570": 6.36198,
- "575": 6.64116,
- "580": 6.56287,
- "585": 6.4212,
- "590": 6.28563,
- "595": 6.35455,
- "600": 6.45248,
- "605": 6.48765,
- "610": 6.56602,
- "615": 6.38966,
- "620": 6.40242,
- "625": 6.27332,
- "630": 6.3586,
- "635": 6.5392,
- "640": 6.2867,
- "645": 6.23834,
- "650": 6.47594,
- "655": 6.51792,
- "660": 6.3853,
- "665": 6.39609,
- "670": 6.24146,
- "675": 6.37809,
- "680": 6.28733,
- "685": 6.29245,
- "690": 6.17169,
- "695": 6.32173,
- "700": 6.03933,
- "705": 6.12077,
- "710": 6.15294,
- "715": 6.11042,
- "720": 6.26214,
- "725": 6.02958,
- "730": 6.17619,
- "735": 6.26799,
- "740": 5.87343,
- "745": 6.14569,
- "750": 6.16714,
- "755": 6.12884,
- "760": 6.02442,
- "765": 5.92762,
- "770": 6.29723,
- "775": 6.02459,
- "780": 6.13604,
- "785": 6.09419,
- "790": 6.26483,
- "795": 6.00813,
- "800": 6.01796,
- "805": 5.82886,
- "810": 6.01825,
- "815": 5.81178,
- "820": 5.73191,
- "825": 6.18892,
- "830": 5.94216,
- "835": 6.00374,
- "840": 6.18371,
- "845": 6.09056,
- "850": 5.8476,
- "855": 5.81787,
- "860": 6.09834,
- "865": 5.93858,
- "870": 5.97087,
- "875": 5.91393,
- "880": 6.01308,
- "885": 5.74796,
- "890": 6.01857,
- "895": 5.84407,
- "900": 5.89689,
- "905": 5.91376,
- "910": 5.86582,
- "915": 5.80263,
- "920": 5.89505,
- "925": 5.62505,
- "930": 5.96642,
- "935": 5.6487,
- "940": 5.97083,
- "945": 5.70845,
- "950": 5.85477,
- "955": 5.53724,
- "960": 5.81112,
- "965": 5.64842,
- "970": 5.65632,
- "975": 5.86723,
- "980": 5.77396,
- "985": 5.59898,
- "990": 5.63478,
- "995": 5.74199,
- "1000": 5.6962,
- "1005": 5.54461,
- "1010": 5.79118,
- "1015": 5.56834,
- "1020": 5.62304,
- "1025": 5.57975,
- "1030": 5.69635,
- "1035": 5.72231,
- "1040": 5.57739,
- "1045": 5.97512,
- "1050": 5.70675,
- "1055": 5.60655,
- "1060": 5.83033,
- "1065": 5.55987,
- "1070": 5.81993,
- "1075": 5.71321,
- "1080": 5.64958,
- "1085": 5.41532,
- "1090": 5.53871,
- "1095": 5.58492,
- "1100": 5.65102,
- "1105": 5.56268,
- "1110": 5.67203,
- "1115": 5.60848,
- "1120": 5.64692,
- "1125": 5.69169,
- "1130": 5.55311,
- "1135": 5.52897,
- "1140": 5.27207,
- "1145": 5.37618,
- "1150": 5.4141,
- "1155": 5.44506,
- "1160": 5.80125,
- "1165": 5.54075,
- "1170": 5.59527,
- "1175": 5.35133,
- "1180": 5.47571,
- "1185": 5.29965,
- "1190": 5.41143,
- "1195": 5.33639,
- "1200": 5.34033,
- "1205": 5.50042,
- "1210": 5.51215,
- "1215": 5.24713,
- "1220": 5.39384,
- "1225": 5.44086,
- "1230": 5.42119,
- "1235": 5.37829,
- "1240": 5.47427,
- "1245": 5.20567,
- "1250": 5.34909,
- "1255": 5.52528,
- "1260": 5.51962,
- "1265": 5.31872,
- "1270": 5.41018,
- "1275": 5.3933,
- "1280": 5.46776,
- "1285": 5.51742,
- "1290": 5.50986,
- "1295": 5.35812,
- "1300": 5.22217,
- "1305": 5.38402,
- "1310": 5.27754,
- "1315": 5.435,
- "1320": 5.43167,
- "1325": 5.41291,
- "1330": 5.40921,
- "1335": 5.38172,
- "1340": 5.42547,
- "1345": 5.35567,
- "1350": 5.04466,
- "1355": 5.12887,
- "1360": 5.1896,
- "1365": 5.03287,
- "1370": 5.20984,
- "1375": 5.47242,
- "1380": 5.27727,
- "1385": 5.40473,
- "1390": 5.26593,
- "1395": 5.34002,
- "1400": 5.27887,
- "1405": 5.12732,
- "1410": 5.38425,
- "1415": 5.45576,
- "1420": 5.45383,
- "1425": 5.60296,
- "1430": 5.34534,
- "1435": 5.21796,
- "1440": 5.35971,
- "1445": 5.08794,
- "1450": 5.35815,
- "1455": 5.14353,
- "1460": 5.25109,
- "1465": 5.28156,
- "1470": 5.13594,
- "1475": 5.20096,
- "1480": 5.1961,
- "1485": 5.30703,
- "1490": 5.05939,
- "1495": 5.14983,
- "1500": 5.11001,
- "1505": 5.02317,
- "1510": 5.03851,
- "1515": 5.37773,
- "1520": 5.15292,
- "1525": 5.22772,
- "1530": 5.2579,
- "1535": 5.17975,
- "1540": 5.13071,
- "1545": 5.13342,
- "1550": 4.98919,
- "1555": 5.12823,
- "1560": 5.12762,
- "1565": 5.10538,
- "1570": 5.18884,
- "1575": 5.24784,
- "1580": 5.14204,
- "1585": 5.13642,
- "1590": 5.11594,
- "1595": 5.1264,
- "1600": 4.88618,
- "1605": 5.0508,
- "1610": 5.06379,
- "1615": 5.09607,
- "1620": 5.15822,
- "1625": 5.13799,
- "1630": 5.08496,
- "1635": 5.0703,
- "1640": 5.02665,
- "1645": 4.98204,
- "1650": 5.07428,
- "1655": 5.18265,
- "1660": 4.82984,
- "1665": 5.12817,
- "1670": 5.13224,
- "1675": 5.06732,
- "1680": 5.05715,
- "1685": 5.15345,
- "1690": 5.0168,
- "1695": 4.93734,
- "1700": 4.96669,
- "1705": 4.98208,
- "1710": 5.05319,
- "1715": 4.94092,
- "1720": 5.04375,
- "1725": 5.2119,
- "1730": 4.93312,
- "1735": 5.2325,
- "1740": 4.85912,
- "1745": 4.84987,
- "1750": 5.0352,
- "1755": 4.91834,
- "1760": 4.85635,
- "1765": 5.17218,
- "1770": 5.00228,
- "1775": 4.97681,
- "1780": 4.85735,
- "1785": 5.19605,
- "1790": 4.82904,
- "1795": 4.99714,
- "1800": 4.93673,
- "1805": 4.9314,
- "1810": 5.12436,
- "1815": 4.89503,
- "1820": 4.72127,
- "1825": 5.09292,
- "1830": 4.94711,
- "1835": 4.98656,
- "1840": 4.94051,
- "1845": 5.0387,
- "1850": 4.90184,
- "1855": 4.84977,
- "1860": 4.8965,
- "1865": 4.89538,
- "1870": 4.97599,
- "1875": 4.88655,
- "1880": 4.92871,
- "1885": 4.95185,
- "1890": 4.74453,
- "1895": 4.82703,
- "1900": 4.94482,
- "1905": 4.5821,
- "1910": 4.8227,
- "1915": 4.90039,
- "1920": 4.80439,
- "1925": 5.09175,
- "1930": 4.9522,
- "1935": 5.15011,
- "1940": 4.86547,
- "1945": 4.7273,
- "1950": 4.83284,
- "1955": 4.8885,
- "1960": "nan",
- "1965": "nan",
- "1970": "nan",
- "1975": "nan",
- "1980": "nan",
- "1985": "nan",
- "1990": "nan",
- "1995": "nan",
- "2000": "nan"
- }
- },
- "num-zeros": {
- "start_step": 1,
- "end_step": 2000,
- "step_interval": 5,
- "values": {
- "1": 29.0,
- "5": 26.0,
- "10": 26.0,
- "15": 24.0,
- "20": 40.0,
- "25": 50.0,
- "30": 39.0,
- "35": 52.0,
- "40": 51.0,
- "45": 59.0,
- "50": 45.0,
- "55": 62.0,
- "60": 58.0,
- "65": 75.0,
- "70": 61.0,
- "75": 61.0,
- "80": 61.0,
- "85": 60.0,
- "90": 57.0,
- "95": 55.0,
- "100": 65.0,
- "105": 80.0,
- "110": 75.0,
- "115": 73.0,
- "120": 78.0,
- "125": 76.0,
- "130": 67.0,
- "135": 70.0,
- "140": 65.0,
- "145": 88.0,
- "150": 58.0,
- "155": 62.0,
- "160": 67.0,
- "165": 62.0,
- "170": 56.0,
- "175": 67.0,
- "180": 63.0,
- "185": 81.0,
- "190": 57.0,
- "195": 59.0,
- "200": 58.0,
- "205": 60.0,
- "210": 58.0,
- "215": 65.0,
- "220": 83.0,
- "225": 60.0,
- "230": 70.0,
- "235": 74.0,
- "240": 75.0,
- "245": 75.0,
- "250": 81.0,
- "255": 87.0,
- "260": 71.0,
- "265": 67.0,
- "270": 90.0,
- "275": 72.0,
- "280": 62.0,
- "285": 60.0,
- "290": 52.0,
- "295": 88.0,
- "300": 80.0,
- "305": 43.0,
- "310": 92.0,
- "315": 69.0,
- "320": 88.0,
- "325": 52.0,
- "330": 80.0,
- "335": 80.0,
- "340": 79.0,
- "345": 93.0,
- "350": 89.0,
- "355": 84.0,
- "360": 90.0,
- "365": 100.0,
- "370": 111.0,
- "375": 88.0,
- "380": 79.0,
- "385": 65.0,
- "390": 96.0,
- "395": 100.0,
- "400": 92.0,
- "405": 87.0,
- "410": 94.0,
- "415": 93.0,
- "420": 96.0,
- "425": 91.0,
- "430": 85.0,
- "435": 72.0,
- "440": 65.0,
- "445": 84.0,
- "450": 88.0,
- "455": 109.0,
- "460": 92.0,
- "465": 77.0,
- "470": 93.0,
- "475": 90.0,
- "480": 64.0,
- "485": 65.0,
- "490": 81.0,
- "495": 72.0,
- "500": 55.0,
- "505": 85.0,
- "510": 64.0,
- "515": 79.0,
- "520": 71.0,
- "525": 90.0,
- "530": 72.0,
- "535": 52.0,
- "540": 71.0,
- "545": 58.0,
- "550": 48.0,
- "555": 65.0,
- "560": 65.0,
- "565": 61.0,
- "570": 51.0,
- "575": 67.0,
- "580": 59.0,
- "585": 55.0,
- "590": 49.0,
- "595": 47.0,
- "600": 49.0,
- "605": 43.0,
- "610": 71.0,
- "615": 52.0,
- "620": 48.0,
- "625": 51.0,
- "630": 52.0,
- "635": 41.0,
- "640": 56.0,
- "645": 57.0,
- "650": 72.0,
- "655": 64.0,
- "660": 43.0,
- "665": 62.0,
- "670": 47.0,
- "675": 52.0,
- "680": 50.0,
- "685": 42.0,
- "690": 43.0,
- "695": 53.0,
- "700": 35.0,
- "705": 39.0,
- "710": 38.0,
- "715": 36.0,
- "720": 32.0,
- "725": 48.0,
- "730": 41.0,
- "735": 51.0,
- "740": 42.0,
- "745": 41.0,
- "750": 30.0,
- "755": 35.0,
- "760": 40.0,
- "765": 42.0,
- "770": 31.0,
- "775": 47.0,
- "780": 45.0,
- "785": 38.0,
- "790": 36.0,
- "795": 38.0,
- "800": 27.0,
- "805": 35.0,
- "810": 40.0,
- "815": 56.0,
- "820": 37.0,
- "825": 43.0,
- "830": 39.0,
- "835": 38.0,
- "840": 48.0,
- "845": 49.0,
- "850": 40.0,
- "855": 40.0,
- "860": 42.0,
- "865": 43.0,
- "870": 35.0,
- "875": 38.0,
- "880": 37.0,
- "885": 32.0,
- "890": 34.0,
- "895": 32.0,
- "900": 41.0,
- "905": 26.0,
- "910": 38.0,
- "915": 39.0,
- "920": 38.0,
- "925": 35.0,
- "930": 28.0,
- "935": 37.0,
- "940": 34.0,
- "945": 31.0,
- "950": 34.0,
- "955": 34.0,
- "960": 33.0,
- "965": 42.0,
- "970": 45.0,
- "975": 53.0,
- "980": 42.0,
- "985": 37.0,
- "990": 38.0,
- "995": 36.0,
- "1000": 30.0,
- "1005": 28.0,
- "1010": 30.0,
- "1015": 27.0,
- "1020": 34.0,
- "1025": 38.0,
- "1030": 30.0,
- "1035": 30.0,
- "1040": 32.0,
- "1045": 31.0,
- "1050": 37.0,
- "1055": 49.0,
- "1060": 23.0,
- "1065": 45.0,
- "1070": 35.0,
- "1075": 40.0,
- "1080": 44.0,
- "1085": 28.0,
- "1090": 36.0,
- "1095": 33.0,
- "1100": 27.0,
- "1105": 36.0,
- "1110": 37.0,
- "1115": 32.0,
- "1120": 35.0,
- "1125": 34.0,
- "1130": 35.0,
- "1135": 39.0,
- "1140": 34.0,
- "1145": 36.0,
- "1150": 30.0,
- "1155": 35.0,
- "1160": 39.0,
- "1165": 30.0,
- "1170": 43.0,
- "1175": 38.0,
- "1180": 32.0,
- "1185": 39.0,
- "1190": 36.0,
- "1195": 40.0,
- "1200": 39.0,
- "1205": 36.0,
- "1210": 36.0,
- "1215": 43.0,
- "1220": 36.0,
- "1225": 35.0,
- "1230": 26.0,
- "1235": 29.0,
- "1240": 40.0,
- "1245": 35.0,
- "1250": 31.0,
- "1255": 33.0,
- "1260": 30.0,
- "1265": 25.0,
- "1270": 39.0,
- "1275": 23.0,
- "1280": 50.0,
- "1285": 43.0,
- "1290": 36.0,
- "1295": 41.0,
- "1300": 28.0,
- "1305": 32.0,
- "1310": 26.0,
- "1315": 35.0,
- "1320": 37.0,
- "1325": 33.0,
- "1330": 31.0,
- "1335": 41.0,
- "1340": 35.0,
- "1345": 32.0,
- "1350": 38.0,
- "1355": 30.0,
- "1360": 35.0,
- "1365": 35.0,
- "1370": 41.0,
- "1375": 29.0,
- "1380": 40.0,
- "1385": 40.0,
- "1390": 32.0,
- "1395": 26.0,
- "1400": 37.0,
- "1405": 45.0,
- "1410": 51.0,
- "1415": 28.0,
- "1420": 27.0,
- "1425": 34.0,
- "1430": 37.0,
- "1435": 27.0,
- "1440": 37.0,
- "1445": 29.0,
- "1450": 37.0,
- "1455": 20.0,
- "1460": 43.0,
- "1465": 35.0,
- "1470": 32.0,
- "1475": 35.0,
- "1480": 29.0,
- "1485": 36.0,
- "1490": 35.0,
- "1495": 41.0,
- "1500": 31.0,
- "1505": 35.0,
- "1510": 29.0,
- "1515": 39.0,
- "1520": 38.0,
- "1525": 39.0,
- "1530": 23.0,
- "1535": 26.0,
- "1540": 35.0,
- "1545": 40.0,
- "1550": 38.0,
- "1555": 29.0,
- "1560": 46.0,
- "1565": 37.0,
- "1570": 36.0,
- "1575": 36.0,
- "1580": 32.0,
- "1585": 35.0,
- "1590": 44.0,
- "1595": 42.0,
- "1600": 45.0,
- "1605": 39.0,
- "1610": 30.0,
- "1615": 39.0,
- "1620": 25.0,
- "1625": 35.0,
- "1630": 35.0,
- "1635": 45.0,
- "1640": 34.0,
- "1645": 39.0,
- "1650": 39.0,
- "1655": 27.0,
- "1660": 41.0,
- "1665": 34.0,
- "1670": 37.0,
- "1675": 43.0,
- "1680": 44.0,
- "1685": 35.0,
- "1690": 41.0,
- "1695": 49.0,
- "1700": 38.0,
- "1705": 41.0,
- "1710": 30.0,
- "1715": 42.0,
- "1720": 40.0,
- "1725": 42.0,
- "1730": 37.0,
- "1735": 49.0,
- "1740": 36.0,
- "1745": 41.0,
- "1750": 40.0,
- "1755": 47.0,
- "1760": 40.0,
- "1765": 48.0,
- "1770": 39.0,
- "1775": 44.0,
- "1780": 37.0,
- "1785": 45.0,
- "1790": 47.0,
- "1795": 37.0,
- "1800": 43.0,
- "1805": 41.0,
- "1810": 29.0,
- "1815": 30.0,
- "1820": 32.0,
- "1825": 43.0,
- "1830": 39.0,
- "1835": 42.0,
- "1840": 40.0,
- "1845": 39.0,
- "1850": 36.0,
- "1855": 38.0,
- "1860": 35.0,
- "1865": 40.0,
- "1870": 35.0,
- "1875": 57.0,
- "1880": 39.0,
- "1885": 38.0,
- "1890": 37.0,
- "1895": 34.0,
- "1900": 47.0,
- "1905": 37.0,
- "1910": 40.0,
- "1915": 43.0,
- "1920": 38.0,
- "1925": 37.0,
- "1930": 53.0,
- "1935": 33.0,
- "1940": 49.0,
- "1945": 40.0,
- "1950": 38.0,
- "1955": 49.0,
- "1960": "nan",
- "1965": "nan",
- "1970": "nan",
- "1975": "nan",
- "1980": "nan",
- "1985": "nan",
- "1990": "nan",
- "1995": "nan",
- "2000": "nan"
- }
- },
- "mem-allocated-bytes": {
- "start_step": 1,
- "end_step": 2000,
- "step_interval": 5,
- "values": {
- "1": 300579328.0,
- "5": 300579328.0,
- "10": 300579328.0,
- "15": 300579328.0,
- "20": 300579328.0,
- "25": 300579328.0,
- "30": 300579328.0,
- "35": 300579328.0,
- "40": 300579328.0,
- "45": 300579328.0,
- "50": 300579328.0,
- "55": 300579328.0,
- "60": 300579328.0,
- "65": 300579328.0,
- "70": 300579328.0,
- "75": 300579328.0,
- "80": 300579328.0,
- "85": 300579328.0,
- "90": 300579328.0,
- "95": 300579328.0,
- "100": 300579328.0,
- "105": 300579328.0,
- "110": 300579328.0,
- "115": 300579328.0,
- "120": 300579328.0,
- "125": 300579328.0,
- "130": 300579328.0,
- "135": 300579328.0,
- "140": 300579328.0,
- "145": 300579328.0,
- "150": 300579328.0,
- "155": 300579328.0,
- "160": 300579328.0,
- "165": 300579328.0,
- "170": 300579328.0,
- "175": 300579328.0,
- "180": 300579328.0,
- "185": 300579328.0,
- "190": 300579328.0,
- "195": 300579328.0,
- "200": 300579328.0,
- "205": 300579328.0,
- "210": 300579328.0,
- "215": 300579328.0,
- "220": 300579328.0,
- "225": 300579328.0,
- "230": 300579328.0,
- "235": 300579328.0,
- "240": 300579328.0,
- "245": 300579328.0,
- "250": 300579328.0,
- "255": 300579328.0,
- "260": 300579328.0,
- "265": 300579328.0,
- "270": 300579328.0,
- "275": 300579328.0,
- "280": 300579328.0,
- "285": 300579328.0,
- "290": 300579328.0,
- "295": 300579328.0,
- "300": 300579328.0,
- "305": 300579328.0,
- "310": 300579328.0,
- "315": 300579328.0,
- "320": 300579328.0,
- "325": 300579328.0,
- "330": 300579328.0,
- "335": 300579328.0,
- "340": 300579328.0,
- "345": 300579328.0,
- "350": 300579328.0,
- "355": 300579328.0,
- "360": 300579328.0,
- "365": 300579328.0,
- "370": 300579328.0,
- "375": 300579328.0,
- "380": 300579328.0,
- "385": 300579328.0,
- "390": 300579328.0,
- "395": 300579328.0,
- "400": 300579328.0,
- "405": 300579328.0,
- "410": 300579328.0,
- "415": 300579328.0,
- "420": 300579328.0,
- "425": 300579328.0,
- "430": 300579328.0,
- "435": 300579328.0,
- "440": 300579328.0,
- "445": 300579328.0,
- "450": 300579328.0,
- "455": 300579328.0,
- "460": 300579328.0,
- "465": 300579328.0,
- "470": 300579328.0,
- "475": 300579328.0,
- "480": 300579328.0,
- "485": 300579328.0,
- "490": 300579328.0,
- "495": 300579328.0,
- "500": 300579328.0,
- "505": 300579328.0,
- "510": 300579328.0,
- "515": 300579328.0,
- "520": 300579328.0,
- "525": 300579328.0,
- "530": 300579328.0,
- "535": 300579328.0,
- "540": 300579328.0,
- "545": 300579328.0,
- "550": 300579328.0,
- "555": 300579328.0,
- "560": 300579328.0,
- "565": 300579328.0,
- "570": 300579328.0,
- "575": 300579328.0,
- "580": 300579328.0,
- "585": 300579328.0,
- "590": 300579328.0,
- "595": 300579328.0,
- "600": 300579328.0,
- "605": 300579328.0,
- "610": 300579328.0,
- "615": 300579328.0,
- "620": 300579328.0,
- "625": 300579328.0,
- "630": 300579328.0,
- "635": 300579328.0,
- "640": 300579328.0,
- "645": 300579328.0,
- "650": 300579328.0,
- "655": 300579328.0,
- "660": 300579328.0,
- "665": 300579328.0,
- "670": 300579328.0,
- "675": 300579328.0,
- "680": 300579328.0,
- "685": 300579328.0,
- "690": 300579328.0,
- "695": 300579328.0,
- "700": 300579328.0,
- "705": 300579328.0,
- "710": 300579328.0,
- "715": 300579328.0,
- "720": 300579328.0,
- "725": 300579328.0,
- "730": 300579328.0,
- "735": 300579328.0,
- "740": 300579328.0,
- "745": 300579328.0,
- "750": 300579328.0,
- "755": 300579328.0,
- "760": 300579328.0,
- "765": 300579328.0,
- "770": 300579328.0,
- "775": 300579328.0,
- "780": 300579328.0,
- "785": 300579328.0,
- "790": 300579328.0,
- "795": 300579328.0,
- "800": 300579328.0,
- "805": 300579328.0,
- "810": 300579328.0,
- "815": 300579328.0,
- "820": 300579328.0,
- "825": 300579328.0,
- "830": 300579328.0,
- "835": 300579328.0,
- "840": 300579328.0,
- "845": 300579328.0,
- "850": 300579328.0,
- "855": 300579328.0,
- "860": 300579328.0,
- "865": 300579328.0,
- "870": 300579328.0,
- "875": 300579328.0,
- "880": 300579328.0,
- "885": 300579328.0,
- "890": 300579328.0,
- "895": 300579328.0,
- "900": 300579328.0,
- "905": 300579328.0,
- "910": 300579328.0,
- "915": 300579328.0,
- "920": 300579328.0,
- "925": 300579328.0,
- "930": 300579328.0,
- "935": 300579328.0,
- "940": 300579328.0,
- "945": 300579328.0,
- "950": 300579328.0,
- "955": 300579328.0,
- "960": 300579328.0,
- "965": 300579328.0,
- "970": 300579328.0,
- "975": 300579328.0,
- "980": 300579328.0,
- "985": 300579328.0,
- "990": 300579328.0,
- "995": 300579328.0,
- "1000": 300579328.0,
- "1005": 300579328.0,
- "1010": 300579328.0,
- "1015": 300579328.0,
- "1020": 300579328.0,
- "1025": 300579328.0,
- "1030": 300579328.0,
- "1035": 300579328.0,
- "1040": 300579328.0,
- "1045": 300579328.0,
- "1050": 300579328.0,
- "1055": 300579328.0,
- "1060": 300579328.0,
- "1065": 300579328.0,
- "1070": 300579328.0,
- "1075": 300579328.0,
- "1080": 300579328.0,
- "1085": 300579328.0,
- "1090": 300579328.0,
- "1095": 300579328.0,
- "1100": 300579328.0,
- "1105": 300579328.0,
- "1110": 300579328.0,
- "1115": 300579328.0,
- "1120": 300579328.0,
- "1125": 300579328.0,
- "1130": 300579328.0,
- "1135": 300579328.0,
- "1140": 300579328.0,
- "1145": 300579328.0,
- "1150": 300579328.0,
- "1155": 300579328.0,
- "1160": 300579328.0,
- "1165": 300579328.0,
- "1170": 300579328.0,
- "1175": 300579328.0,
- "1180": 300579328.0,
- "1185": 300579328.0,
- "1190": 300579328.0,
- "1195": 300579328.0,
- "1200": 300579328.0,
- "1205": 300579328.0,
- "1210": 300579328.0,
- "1215": 300579328.0,
- "1220": 300579328.0,
- "1225": 300579328.0,
- "1230": 300579328.0,
- "1235": 300579328.0,
- "1240": 300579328.0,
- "1245": 300579328.0,
- "1250": 300579328.0,
- "1255": 300579328.0,
- "1260": 300579328.0,
- "1265": 300579328.0,
- "1270": 300579328.0,
- "1275": 300579328.0,
- "1280": 300579328.0,
- "1285": 300579328.0,
- "1290": 300579328.0,
- "1295": 300579328.0,
- "1300": 300579328.0,
- "1305": 300579328.0,
- "1310": 300579328.0,
- "1315": 300579328.0,
- "1320": 300579328.0,
- "1325": 300579328.0,
- "1330": 300579328.0,
- "1335": 300579328.0,
- "1340": 300579328.0,
- "1345": 300579328.0,
- "1350": 300579328.0,
- "1355": 300579328.0,
- "1360": 300579328.0,
- "1365": 300579328.0,
- "1370": 300579328.0,
- "1375": 300579328.0,
- "1380": 300579328.0,
- "1385": 300579328.0,
- "1390": 300579328.0,
- "1395": 300579328.0,
- "1400": 300579328.0,
- "1405": 300579328.0,
- "1410": 300579328.0,
- "1415": 300579328.0,
- "1420": 300579328.0,
- "1425": 300579328.0,
- "1430": 300579328.0,
- "1435": 300579328.0,
- "1440": 300579328.0,
- "1445": 300579328.0,
- "1450": 300579328.0,
- "1455": 300579328.0,
- "1460": 300579328.0,
- "1465": 300579328.0,
- "1470": 300579328.0,
- "1475": 300579328.0,
- "1480": 300579328.0,
- "1485": 300579328.0,
- "1490": 300579328.0,
- "1495": 300579328.0,
- "1500": 300579328.0,
- "1505": 300579328.0,
- "1510": 300579328.0,
- "1515": 300579328.0,
- "1520": 300579328.0,
- "1525": 300579328.0,
- "1530": 300579328.0,
- "1535": 300579328.0,
- "1540": 300579328.0,
- "1545": 300579328.0,
- "1550": 300579328.0,
- "1555": 300579328.0,
- "1560": 300579328.0,
- "1565": 300579328.0,
- "1570": 300579328.0,
- "1575": 300579328.0,
- "1580": 300579328.0,
- "1585": 300579328.0,
- "1590": 300579328.0,
- "1595": 300579328.0,
- "1600": 300579328.0,
- "1605": 300579328.0,
- "1610": 300579328.0,
- "1615": 300579328.0,
- "1620": 300579328.0,
- "1625": 300579328.0,
- "1630": 300579328.0,
- "1635": 300579328.0,
- "1640": 300579328.0,
- "1645": 300579328.0,
- "1650": 300579328.0,
- "1655": 300579328.0,
- "1660": 300579328.0,
- "1665": 300579328.0,
- "1670": 300579328.0,
- "1675": 300579328.0,
- "1680": 300579328.0,
- "1685": 300579328.0,
- "1690": 300579328.0,
- "1695": 300579328.0,
- "1700": 300579328.0,
- "1705": 300579328.0,
- "1710": 300579328.0,
- "1715": 300579328.0,
- "1720": 300579328.0,
- "1725": 300579328.0,
- "1730": 300579328.0,
- "1735": 300579328.0,
- "1740": 300579328.0,
- "1745": 300579328.0,
- "1750": 300579328.0,
- "1755": 300579328.0,
- "1760": 300579328.0,
- "1765": 300579328.0,
- "1770": 300579328.0,
- "1775": 300579328.0,
- "1780": 300579328.0,
- "1785": 300579328.0,
- "1790": 300579328.0,
- "1795": 300579328.0,
- "1800": 300579328.0,
- "1805": 300579328.0,
- "1810": 300579328.0,
- "1815": 300579328.0,
- "1820": 300579328.0,
- "1825": 300579328.0,
- "1830": 300579328.0,
- "1835": 300579328.0,
- "1840": 300579328.0,
- "1845": 300579328.0,
- "1850": 300579328.0,
- "1855": 300579328.0,
- "1860": 300579328.0,
- "1865": 300579328.0,
- "1870": 300579328.0,
- "1875": 300579328.0,
- "1880": 300579328.0,
- "1885": 300579328.0,
- "1890": 300579328.0,
- "1895": 300579328.0,
- "1900": 300579328.0,
- "1905": 300579328.0,
- "1910": 300579328.0,
- "1915": 300579328.0,
- "1920": 300579328.0,
- "1925": 300579328.0,
- "1930": 300579328.0,
- "1935": 300579328.0,
- "1940": 300579328.0,
- "1945": 300579328.0,
- "1950": 300579328.0,
- "1955": 300579328.0,
- "1960": "nan",
- "1965": "nan",
- "1970": "nan",
- "1975": "nan",
- "1980": "nan",
- "1985": "nan",
- "1990": "nan",
- "1995": "nan",
- "2000": "nan"
- }
- },
- "mem-max-allocated-bytes": {
- "start_step": 1,
- "end_step": 2000,
- "step_interval": 5,
- "values": {
- "1": 363738624.0,
- "5": 429681152.0,
- "10": 429681152.0,
- "15": 429681152.0,
- "20": 429681152.0,
- "25": 429681152.0,
- "30": 429681152.0,
- "35": 429681152.0,
- "40": 429681152.0,
- "45": 429681152.0,
- "50": 429681152.0,
- "55": 429681152.0,
- "60": 429681152.0,
- "65": 429681152.0,
- "70": 429681152.0,
- "75": 429681152.0,
- "80": 429681152.0,
- "85": 429681152.0,
- "90": 429681152.0,
- "95": 429681152.0,
- "100": 429681152.0,
- "105": 429681152.0,
- "110": 429681152.0,
- "115": 429681152.0,
- "120": 429681152.0,
- "125": 429681152.0,
- "130": 429681152.0,
- "135": 429681152.0,
- "140": 429681152.0,
- "145": 429681152.0,
- "150": 429681152.0,
- "155": 429681152.0,
- "160": 429681152.0,
- "165": 429681152.0,
- "170": 429681152.0,
- "175": 429681152.0,
- "180": 429681152.0,
- "185": 429681152.0,
- "190": 429681152.0,
- "195": 429681152.0,
- "200": 429681152.0,
- "205": 429681152.0,
- "210": 429681152.0,
- "215": 429681152.0,
- "220": 429681152.0,
- "225": 429681152.0,
- "230": 429681152.0,
- "235": 429681152.0,
- "240": 429681152.0,
- "245": 429681152.0,
- "250": 429681152.0,
- "255": 429681152.0,
- "260": 429681152.0,
- "265": 429681152.0,
- "270": 429681152.0,
- "275": 429681152.0,
- "280": 429681152.0,
- "285": 429681152.0,
- "290": 429681152.0,
- "295": 429681152.0,
- "300": 429681152.0,
- "305": 429681152.0,
- "310": 429681152.0,
- "315": 429681152.0,
- "320": 429681152.0,
- "325": 429681152.0,
- "330": 429681152.0,
- "335": 429681152.0,
- "340": 429681152.0,
- "345": 429681152.0,
- "350": 429681152.0,
- "355": 429681152.0,
- "360": 429681152.0,
- "365": 429681152.0,
- "370": 429681152.0,
- "375": 429681152.0,
- "380": 429681152.0,
- "385": 429681152.0,
- "390": 429681152.0,
- "395": 429681152.0,
- "400": 429681152.0,
- "405": 429681152.0,
- "410": 429681152.0,
- "415": 429681152.0,
- "420": 429681152.0,
- "425": 429681152.0,
- "430": 429681152.0,
- "435": 429681152.0,
- "440": 429681152.0,
- "445": 429681152.0,
- "450": 429681152.0,
- "455": 429681152.0,
- "460": 429681152.0,
- "465": 429681152.0,
- "470": 429681152.0,
- "475": 429681152.0,
- "480": 429681152.0,
- "485": 429681152.0,
- "490": 429681152.0,
- "495": 429681152.0,
- "500": 429681152.0,
- "505": 429681152.0,
- "510": 429681152.0,
- "515": 429681152.0,
- "520": 429681152.0,
- "525": 429681152.0,
- "530": 429681152.0,
- "535": 429681152.0,
- "540": 429681152.0,
- "545": 429681152.0,
- "550": 429681152.0,
- "555": 429681152.0,
- "560": 429681152.0,
- "565": 429681152.0,
- "570": 429681152.0,
- "575": 429681152.0,
- "580": 429681152.0,
- "585": 429681152.0,
- "590": 429681152.0,
- "595": 429681152.0,
- "600": 429681152.0,
- "605": 429681152.0,
- "610": 429681152.0,
- "615": 429681152.0,
- "620": 429681152.0,
- "625": 429681152.0,
- "630": 429681152.0,
- "635": 429681152.0,
- "640": 429681152.0,
- "645": 429681152.0,
- "650": 429681152.0,
- "655": 429681152.0,
- "660": 429681152.0,
- "665": 429681152.0,
- "670": 429681152.0,
- "675": 429681152.0,
- "680": 429681152.0,
- "685": 429681152.0,
- "690": 429681152.0,
- "695": 429681152.0,
- "700": 429681152.0,
- "705": 429681152.0,
- "710": 429681152.0,
- "715": 429681152.0,
- "720": 429681152.0,
- "725": 429681152.0,
- "730": 429681152.0,
- "735": 429681152.0,
- "740": 429681152.0,
- "745": 429681152.0,
- "750": 429681152.0,
- "755": 429681152.0,
- "760": 429681152.0,
- "765": 429681152.0,
- "770": 429681152.0,
- "775": 429681152.0,
- "780": 429681152.0,
- "785": 429681152.0,
- "790": 429681152.0,
- "795": 429681152.0,
- "800": 429681152.0,
- "805": 429681152.0,
- "810": 429681152.0,
- "815": 429681152.0,
- "820": 429681152.0,
- "825": 429681152.0,
- "830": 429681152.0,
- "835": 429681152.0,
- "840": 429681152.0,
- "845": 429681152.0,
- "850": 429681152.0,
- "855": 429681152.0,
- "860": 429681152.0,
- "865": 429681152.0,
- "870": 429681152.0,
- "875": 429681152.0,
- "880": 429681152.0,
- "885": 429681152.0,
- "890": 429681152.0,
- "895": 429681152.0,
- "900": 429681152.0,
- "905": 429681152.0,
- "910": 429681152.0,
- "915": 429681152.0,
- "920": 429681152.0,
- "925": 429681152.0,
- "930": 429681152.0,
- "935": 429681152.0,
- "940": 429681152.0,
- "945": 429681152.0,
- "950": 429681152.0,
- "955": 429681152.0,
- "960": 429681152.0,
- "965": 429681152.0,
- "970": 429681152.0,
- "975": 429681152.0,
- "980": 429681152.0,
- "985": 429681152.0,
- "990": 429681152.0,
- "995": 429681152.0,
- "1000": 429681152.0,
- "1005": 429681152.0,
- "1010": 429681152.0,
- "1015": 429681152.0,
- "1020": 429681152.0,
- "1025": 429681152.0,
- "1030": 429681152.0,
- "1035": 429681152.0,
- "1040": 429681152.0,
- "1045": 429681152.0,
- "1050": 429681152.0,
- "1055": 429681152.0,
- "1060": 429681152.0,
- "1065": 429681152.0,
- "1070": 429681152.0,
- "1075": 429681152.0,
- "1080": 429681152.0,
- "1085": 429681152.0,
- "1090": 429681152.0,
- "1095": 429681152.0,
- "1100": 429681152.0,
- "1105": 429681152.0,
- "1110": 429681152.0,
- "1115": 429681152.0,
- "1120": 429681152.0,
- "1125": 429681152.0,
- "1130": 429681152.0,
- "1135": 429681152.0,
- "1140": 429681152.0,
- "1145": 429681152.0,
- "1150": 429681152.0,
- "1155": 429681152.0,
- "1160": 429681152.0,
- "1165": 429681152.0,
- "1170": 429681152.0,
- "1175": 429681152.0,
- "1180": 429681152.0,
- "1185": 429681152.0,
- "1190": 429681152.0,
- "1195": 429681152.0,
- "1200": 429681152.0,
- "1205": 429681152.0,
- "1210": 429681152.0,
- "1215": 429681152.0,
- "1220": 429681152.0,
- "1225": 429681152.0,
- "1230": 429681152.0,
- "1235": 429681152.0,
- "1240": 429681152.0,
- "1245": 429681152.0,
- "1250": 429681152.0,
- "1255": 429681152.0,
- "1260": 429681152.0,
- "1265": 429681152.0,
- "1270": 429681152.0,
- "1275": 429681152.0,
- "1280": 429681152.0,
- "1285": 429681152.0,
- "1290": 429681152.0,
- "1295": 429681152.0,
- "1300": 429681152.0,
- "1305": 429681152.0,
- "1310": 429681152.0,
- "1315": 429681152.0,
- "1320": 429681152.0,
- "1325": 429681152.0,
- "1330": 429681152.0,
- "1335": 429681152.0,
- "1340": 429681152.0,
- "1345": 429681152.0,
- "1350": 429681152.0,
- "1355": 429681152.0,
- "1360": 429681152.0,
- "1365": 429681152.0,
- "1370": 429681152.0,
- "1375": 429681152.0,
- "1380": 429681152.0,
- "1385": 429681152.0,
- "1390": 429681152.0,
- "1395": 429681152.0,
- "1400": 429681152.0,
- "1405": 429681152.0,
- "1410": 429681152.0,
- "1415": 429681152.0,
- "1420": 429681152.0,
- "1425": 429681152.0,
- "1430": 429681152.0,
- "1435": 429681152.0,
- "1440": 429681152.0,
- "1445": 429681152.0,
- "1450": 429681152.0,
- "1455": 429681152.0,
- "1460": 429681152.0,
- "1465": 429681152.0,
- "1470": 429681152.0,
- "1475": 429681152.0,
- "1480": 429681152.0,
- "1485": 429681152.0,
- "1490": 429681152.0,
- "1495": 429681152.0,
- "1500": 429681152.0,
- "1505": 429681152.0,
- "1510": 429681152.0,
- "1515": 429681152.0,
- "1520": 429681152.0,
- "1525": 429681152.0,
- "1530": 429681152.0,
- "1535": 429681152.0,
- "1540": 429681152.0,
- "1545": 429681152.0,
- "1550": 429681152.0,
- "1555": 429681152.0,
- "1560": 429681152.0,
- "1565": 429681152.0,
- "1570": 429681152.0,
- "1575": 429681152.0,
- "1580": 429681152.0,
- "1585": 429681152.0,
- "1590": 429681152.0,
- "1595": 429681152.0,
- "1600": 429681152.0,
- "1605": 429681152.0,
- "1610": 429681152.0,
- "1615": 429681152.0,
- "1620": 429681152.0,
- "1625": 429681152.0,
- "1630": 429681152.0,
- "1635": 429681152.0,
- "1640": 429681152.0,
- "1645": 429681152.0,
- "1650": 429681152.0,
- "1655": 429681152.0,
- "1660": 429681152.0,
- "1665": 429681152.0,
- "1670": 429681152.0,
- "1675": 429681152.0,
- "1680": 429681152.0,
- "1685": 429681152.0,
- "1690": 429681152.0,
- "1695": 429681152.0,
- "1700": 429681152.0,
- "1705": 429681152.0,
- "1710": 429681152.0,
- "1715": 429681152.0,
- "1720": 429681152.0,
- "1725": 429681152.0,
- "1730": 429681152.0,
- "1735": 429681152.0,
- "1740": 429681152.0,
- "1745": 429681152.0,
- "1750": 429681152.0,
- "1755": 429681152.0,
- "1760": 429681152.0,
- "1765": 429681152.0,
- "1770": 429681152.0,
- "1775": 429681152.0,
- "1780": 429681152.0,
- "1785": 429681152.0,
- "1790": 429681152.0,
- "1795": 429681152.0,
- "1800": 429681152.0,
- "1805": 429681152.0,
- "1810": 429681152.0,
- "1815": 429681152.0,
- "1820": 429681152.0,
- "1825": 429681152.0,
- "1830": 429681152.0,
- "1835": 429681152.0,
- "1840": 429681152.0,
- "1845": 429681152.0,
- "1850": 429681152.0,
- "1855": 429681152.0,
- "1860": 429681152.0,
- "1865": 429681152.0,
- "1870": 429681152.0,
- "1875": 429681152.0,
- "1880": 429681152.0,
- "1885": 429681152.0,
- "1890": 429681152.0,
- "1895": 429681152.0,
- "1900": 429681152.0,
- "1905": 429681152.0,
- "1910": 429681152.0,
- "1915": 429681152.0,
- "1920": 429681152.0,
- "1925": 429681152.0,
- "1930": 429681152.0,
- "1935": 429681152.0,
- "1940": 429681152.0,
- "1945": 429681152.0,
- "1950": 429681152.0,
- "1955": 429681152.0,
- "1960": "nan",
- "1965": "nan",
- "1970": "nan",
- "1975": "nan",
- "1980": "nan",
- "1985": "nan",
- "1990": "nan",
- "1995": "nan",
- "2000": "nan"
- }
- },
- "iteration-time": {
- "start_step": 1,
- "end_step": 2000,
- "step_interval": 5,
- "values": {
- "1": 21.12507,
- "5": 7.49331,
- "10": 7.00576,
- "15": 7.04629,
- "20": 7.07208,
- "25": 7.00993,
- "30": 7.03565,
- "35": 7.21659,
- "40": 6.99376,
- "45": 6.99799,
- "50": 6.93059,
- "55": 7.01743,
- "60": 6.94161,
- "65": 7.61789,
- "70": 6.9494,
- "75": 6.97805,
- "80": 6.93652,
- "85": 6.93081,
- "90": 6.90049,
- "95": 7.39441,
- "100": 6.86289,
- "105": 6.93607,
- "110": 7.07378,
- "115": 6.93173,
- "120": 6.89483,
- "125": 7.44574,
- "130": 6.89552,
- "135": 6.91809,
- "140": 6.90906,
- "145": 6.93663,
- "150": 6.93002,
- "155": 6.91321,
- "160": 6.91771,
- "165": 6.92693,
- "170": 6.90617,
- "175": 6.93168,
- "180": 6.97556,
- "185": 6.98943,
- "190": 7.19046,
- "195": 6.9481,
- "200": 6.97387,
- "205": 6.88536,
- "210": 6.98827,
- "215": 7.19745,
- "220": 7.00849,
- "225": 6.98424,
- "230": 6.98606,
- "235": 6.93942,
- "240": 6.96165,
- "245": 6.94648,
- "250": 6.95126,
- "255": 6.8774,
- "260": 6.90637,
- "265": 6.86094,
- "270": 7.11734,
- "275": 6.87241,
- "280": 6.97305,
- "285": 6.86227,
- "290": 6.99272,
- "295": 7.02359,
- "300": 7.11055,
- "305": 6.84613,
- "310": 6.96629,
- "315": 6.8745,
- "320": 6.94893,
- "325": 6.82912,
- "330": 6.91518,
- "335": 6.86857,
- "340": 6.90418,
- "345": 6.92639,
- "350": 6.85386,
- "355": 6.89745,
- "360": 6.87211,
- "365": 6.8853,
- "370": 6.80679,
- "375": 7.17635,
- "380": 6.82143,
- "385": 6.88471,
- "390": 6.86587,
- "395": 6.9137,
- "400": 7.36499,
- "405": 6.89839,
- "410": 6.89008,
- "415": 6.84923,
- "420": 7.09663,
- "425": 6.85147,
- "430": 6.88198,
- "435": 6.99751,
- "440": 6.88975,
- "445": 6.79365,
- "450": 7.50603,
- "455": 7.10337,
- "460": 6.87946,
- "465": 6.89046,
- "470": 7.04055,
- "475": 6.85188,
- "480": 7.65045,
- "485": 6.86313,
- "490": 6.78386,
- "495": 6.88579,
- "500": 6.82451,
- "505": 7.07694,
- "510": 7.32635,
- "515": 6.77647,
- "520": 6.83445,
- "525": 6.71773,
- "530": 6.82847,
- "535": 7.02455,
- "540": 6.83842,
- "545": 6.83077,
- "550": 6.7782,
- "555": 6.81546,
- "560": 6.71832,
- "565": 6.82061,
- "570": 6.74754,
- "575": 6.79436,
- "580": 6.78477,
- "585": 6.71156,
- "590": 6.83705,
- "595": 6.77825,
- "600": 6.85249,
- "605": 6.8063,
- "610": 6.83814,
- "615": 6.85186,
- "620": 6.77621,
- "625": 7.0539,
- "630": 6.79749,
- "635": 6.84041,
- "640": 6.81829,
- "645": 6.81622,
- "650": 6.8255,
- "655": 6.92056,
- "660": 6.81133,
- "665": 6.96346,
- "670": 6.85443,
- "675": 6.77635,
- "680": 6.81276,
- "685": 6.80553,
- "690": 6.74335,
- "695": 6.80023,
- "700": 6.73332,
- "705": 6.80337,
- "710": 6.7912,
- "715": 6.7503,
- "720": 7.07012,
- "725": 6.73797,
- "730": 6.80782,
- "735": 6.80341,
- "740": 6.80517,
- "745": 6.79608,
- "750": 6.72513,
- "755": 6.8012,
- "760": 6.73488,
- "765": 6.80073,
- "770": 7.01145,
- "775": 6.95817,
- "780": 6.8135,
- "785": 6.72833,
- "790": 6.80079,
- "795": 6.77938,
- "800": 6.97825,
- "805": 6.9712,
- "810": 6.72623,
- "815": 6.83338,
- "820": 6.78526,
- "825": 6.81302,
- "830": 6.82367,
- "835": 6.96099,
- "840": 6.81708,
- "845": 6.75672,
- "850": 6.86365,
- "855": 6.80902,
- "860": 6.83987,
- "865": 7.02383,
- "870": 6.78086,
- "875": 6.81173,
- "880": 6.9514,
- "885": 6.8015,
- "890": 6.79609,
- "895": 7.32916,
- "900": 6.80647,
- "905": 6.74718,
- "910": 6.78621,
- "915": 6.72967,
- "920": 6.80054,
- "925": 6.80304,
- "930": 6.75417,
- "935": 6.80538,
- "940": 6.76214,
- "945": 6.81332,
- "950": 6.83729,
- "955": 7.289,
- "960": 6.83001,
- "965": 6.76958,
- "970": 6.79873,
- "975": 6.7652,
- "980": 7.01177,
- "985": 6.82918,
- "990": 6.80054,
- "995": 6.81069,
- "1000": 6.96679,
- "1005": 6.79881,
- "1010": 6.8125,
- "1015": 6.78973,
- "1020": 6.82065,
- "1025": 6.8089,
- "1030": 6.81805,
- "1035": 6.82075,
- "1040": 6.79677,
- "1045": 6.81588,
- "1050": 6.76341,
- "1055": 6.78995,
- "1060": 6.98459,
- "1065": 7.04911,
- "1070": 7.07525,
- "1075": 6.77976,
- "1080": 6.81789,
- "1085": 6.76916,
- "1090": 6.84872,
- "1095": 6.82596,
- "1100": 6.82415,
- "1105": 7.08346,
- "1110": 6.77184,
- "1115": 6.82554,
- "1120": 6.74734,
- "1125": 6.78971,
- "1130": 6.97935,
- "1135": 6.78482,
- "1140": 7.2055,
- "1145": 6.96662,
- "1150": 6.80129,
- "1155": 6.74065,
- "1160": 6.7419,
- "1165": 6.77394,
- "1170": 6.69931,
- "1175": 6.76576,
- "1180": 6.74597,
- "1185": 6.71705,
- "1190": 6.93295,
- "1195": 6.69454,
- "1200": 6.76199,
- "1205": 6.75755,
- "1210": 6.7016,
- "1215": 6.75554,
- "1220": 6.8741,
- "1225": 6.75159,
- "1230": 6.77132,
- "1235": 6.69805,
- "1240": 6.77757,
- "1245": 6.78197,
- "1250": 6.94834,
- "1255": 6.76297,
- "1260": 6.7016,
- "1265": 6.75885,
- "1270": 6.72882,
- "1275": 6.75951,
- "1280": 6.94551,
- "1285": 6.69878,
- "1290": 6.75151,
- "1295": 6.72995,
- "1300": 6.77554,
- "1305": 6.76755,
- "1310": 6.72034,
- "1315": 6.78531,
- "1320": 6.74309,
- "1325": 6.77947,
- "1330": 6.79799,
- "1335": 6.70873,
- "1340": 6.76639,
- "1345": 6.7227,
- "1350": 6.76281,
- "1355": 6.78672,
- "1360": 6.71198,
- "1365": 6.77713,
- "1370": 7.03803,
- "1375": 6.79368,
- "1380": 6.78317,
- "1385": 6.70868,
- "1390": 6.79601,
- "1395": 6.76727,
- "1400": 6.74844,
- "1405": 7.01822,
- "1410": 6.7047,
- "1415": 6.7536,
- "1420": 6.73226,
- "1425": 6.74642,
- "1430": 7.06262,
- "1435": 6.71011,
- "1440": 6.76017,
- "1445": 6.73964,
- "1450": 6.75625,
- "1455": 6.75609,
- "1460": 7.06935,
- "1465": 6.7634,
- "1470": 6.76298,
- "1475": 6.72342,
- "1480": 6.75422,
- "1485": 7.11098,
- "1490": 7.06357,
- "1495": 6.96678,
- "1500": 6.75799,
- "1505": 6.76385,
- "1510": 6.70609,
- "1515": 6.93745,
- "1520": 7.06079,
- "1525": 6.73736,
- "1530": 6.76012,
- "1535": 6.70887,
- "1540": 6.71979,
- "1545": 6.94391,
- "1550": 7.01154,
- "1555": 6.73786,
- "1560": 6.73334,
- "1565": 6.9755,
- "1570": 6.73626,
- "1575": 6.86097,
- "1580": 6.73961,
- "1585": 6.75518,
- "1590": 6.69811,
- "1595": 6.72418,
- "1600": 6.70251,
- "1605": 6.74605,
- "1610": 6.73628,
- "1615": 6.69455,
- "1620": 6.75237,
- "1625": 6.7303,
- "1630": 6.73721,
- "1635": 6.9236,
- "1640": 6.70974,
- "1645": 6.95271,
- "1650": 6.74647,
- "1655": 6.71129,
- "1660": 6.73558,
- "1665": 6.69023,
- "1670": 6.73159,
- "1675": 6.80183,
- "1680": 6.77675,
- "1685": 6.79809,
- "1690": 6.76503,
- "1695": 6.80935,
- "1700": 6.79727,
- "1705": 6.77845,
- "1710": 6.80131,
- "1715": 6.77718,
- "1720": 6.7988,
- "1725": 6.99283,
- "1730": 7.01483,
- "1735": 6.80757,
- "1740": 6.79298,
- "1745": 6.79097,
- "1750": 6.78918,
- "1755": 6.80891,
- "1760": 6.78222,
- "1765": 6.8139,
- "1770": 6.8315,
- "1775": 6.77841,
- "1780": 6.82671,
- "1785": 6.78358,
- "1790": 6.81896,
- "1795": 6.80082,
- "1800": 6.80744,
- "1805": 6.83906,
- "1810": 7.01643,
- "1815": 6.81382,
- "1820": 6.77985,
- "1825": 6.82288,
- "1830": 6.8047,
- "1835": 6.81015,
- "1840": 6.80067,
- "1845": 6.9625,
- "1850": 6.81223,
- "1855": 6.76164,
- "1860": 6.83943,
- "1865": 6.81648,
- "1870": 6.80706,
- "1875": 6.98557,
- "1880": 6.77758,
- "1885": 7.04482,
- "1890": 7.01977,
- "1895": 6.82284,
- "1900": 6.83398,
- "1905": 7.00112,
- "1910": 7.02281,
- "1915": 6.78654,
- "1920": 6.8245,
- "1925": 6.78352,
- "1930": 6.82028,
- "1935": 7.04035,
- "1940": 6.84142,
- "1945": 6.82102,
- "1950": 6.78053,
- "1955": 6.80661,
- "1960": "nan",
- "1965": "nan",
- "1970": "nan",
- "1975": "nan",
- "1980": "nan",
- "1985": "nan",
- "1990": "nan",
- "1995": "nan",
- "2000": "nan"
- }
- }
-}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_cp2_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_cp2_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml
deleted file mode 100644
index ddb34ad850b..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_cp2_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml
+++ /dev/null
@@ -1,61 +0,0 @@
-ENV_VARS:
- CUDA_DEVICE_MAX_CONNECTIONS: 1
- NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
- NON_DETERMINSTIC_RESULTS: 1
- CUBLAS_WORKSPACE_CONFIG: :4096:8
- NCCL_ALGO: Ring
-MODEL_ARGS:
- --num-layers: 12
- --hidden-size: 512
- --num-attention-heads: 8
- --log-params-norm: true
- --log-num-zeros-in-grad: true
- --log-validation-ppl-to-tensorboard: true
- --log-timers-to-tensorboard: true
- --tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
- --seq-length: 1024
- --max-position-embeddings: 1024
- --train-iters: 2000
- --timing-log-level: 0
- --lr-decay-iters: 320000
- --save: ${CHECKPOINT_SAVE_PATH}
- --load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
- --split: 949,50,1
- --distributed-backend: nccl
- --lr: 0.00015
- --lr-decay-style: cosine
- --min-lr: 1.0e-5
- --weight-decay: 1e-2
- --clip-grad: 1.0
- --lr-warmup-fraction: .01
- --log-interval: 1
- --save-interval: 10000
- --eval-interval: 1000
- --eval-iters: 10
- --transformer-impl: transformer_engine
- --tensor-model-parallel-size: 4
- --context-parallel-size: 2
- --sequence-parallel: true
- --deterministic-mode: true
- --no-gradient-accumulation-fusion: true
- --fp8-format: hybrid
- --fp8-amax-history-len: 1024
- --fp8-amax-compute-algo: max
- --fp8-param-gather: true
- --use-distributed-optimizer: true
- --attention-softmax-in-fp32: true
- --ckpt-format: torch_dist
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
- --data-cache-path: ${DATA_CACHE_PATH}
- --bf16: true
- --log-memory-to-tensorboard: true
-TEST_TYPE: regular
-METRICS:
- - iteration-time
- - lm loss
- - "mem-allocated-bytes"
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_pp2_fp8_tp_pp/golden_values_dev_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_pp2_fp8_tp_pp/golden_values_dev_dgx_a100.json
deleted file mode 100644
index 410ce0432c2..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_pp2_fp8_tp_pp/golden_values_dev_dgx_a100.json
+++ /dev/null
@@ -1 +0,0 @@
-{"forward-backward-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [15.91085, 1.83696, 1.80977, 1.80614, 1.80726, 1.80478, 1.79131, 1.78726, 1.78783, 1.78922, 1.77727, 1.77268, 1.79506, 1.77591, 1.78579, 1.73441, 1.73281, 1.71725, 1.7452, 1.79112, 1.71713, 1.71391, 1.71555, 1.70838, 1.71577, 1.71204, 1.70777, 1.86607, 1.72441, 1.72591, 1.70281, 1.70759, 1.71359, 1.70764, 2.0202, 1.70824, 1.71156, 1.72055, 1.71634, 1.72374, 1.93145, 1.71296, 1.96517, 1.70426, 1.71396, 1.71072, 1.72478, 1.71329, 1.70891, 1.70824, 1.71032, 1.71153, 1.70874, 1.71511, 1.71205, 1.70972, 1.73233, 1.72187, 1.71536, 1.71399, 1.7368, 1.71495, 1.71292, 1.73073, 1.72036, 1.71789, 1.70771, 1.72211, 1.71455, 1.74019, 1.7122, 1.7112, 1.71796, 1.71199, 1.73553, 1.71529, 1.73592, 1.71594, 1.71027, 1.71673, 1.70741, 1.73431, 1.72286, 1.72962, 1.70988, 1.71949, 1.71223, 1.71075, 1.71048, 1.70371, 1.7433, 1.70766, 1.71592, 1.7109, 1.71432, 1.71488, 1.71199, 1.71265, 1.71789, 1.71226, 1.70924, 1.71394, 1.71992, 1.71838, 1.72476, 1.72213, 1.72334, 1.7156, 1.71199, 1.71831, 1.72554, 1.72452, 1.90237, 1.71646, 1.72407, 1.72142, 1.70768, 1.71577, 1.72074, 1.72296, 1.72108, 1.71421, 1.71615, 1.71327, 1.71352, 1.71744, 1.71843, 1.72, 1.71691, 1.71452, 1.72623, 1.71137, 1.72452, 1.72814, 1.71396, 1.71438, 1.71782, 1.71212, 1.71277, 1.71122, 1.70761, 1.70626, 1.7082, 1.72674, 1.72145, 1.72692, 1.71902, 1.71694, 1.71626, 1.72313, 1.73762, 1.71092, 1.72399, 1.71397, 1.71661, 1.72078, 1.72314, 1.72762, 1.72185, 1.73771, 1.74159, 1.71527, 1.87793, 1.71543, 1.73315, 1.71045, 1.73711, 1.86628, 1.73295, 1.73053, 1.72785, 1.7325, 1.72782, 1.7401, 1.73445, 1.7301, 1.71283, 1.725, 1.72956, 1.71122, 1.71346, 1.7259, 1.71636, 1.71639, 1.72224, 1.71405, 1.71888, 1.72167, 1.74466, 1.72145, 1.72256, 1.71785, 1.73237, 1.71755, 1.73361, 1.87342, 1.72273, 1.71588, 1.71152, 1.70929, 1.73331, 1.98295, 1.73263, 1.72317, 1.72815, 1.72399, 1.72154, 1.72787, 1.71935, 1.70989, 1.73251, 1.72929, 1.72421, 1.72359, 1.74518, 1.72365, 1.73636, 1.72601, 1.73111, 1.73181, 1.73839, 1.71392, 1.71397, 1.72263, 1.72065, 1.74302, 1.73401, 1.73779, 1.72222, 1.72737, 1.73283, 1.72085, 1.72936, 1.72362, 1.7256, 1.74208, 1.72115, 1.71544, 1.72076, 1.72955, 1.72763, 1.72611, 1.74549, 1.7277, 1.73079, 1.73834, 1.73241, 1.73023, 1.73279, 1.73489, 1.71967, 1.72319, 1.71603, 1.72084, 1.72097, 1.72216, 1.71813, 1.72503, 1.72355, 1.72027, 1.72502, 1.7275, 1.72949, 1.74652, 1.73389, 1.73062, 1.74625, 1.7301, 1.73085, 1.74929, 1.7465, 1.73308, 1.73309, 1.75066, 1.72428, 1.71878, 1.73281, 1.73721, 1.73632, 1.74495, 1.74192, 1.89678, 1.75791, 1.74287, 1.74488, 1.74174, 1.74912, 1.73966, 1.73073, 1.74247, 1.73943, 1.73241, 1.73387, 1.7354, 1.73672, 1.72734, 1.74088, 1.73541, 1.73319, 1.72887, 1.7347, 1.72386, 1.74493, 1.75477, 1.7379, 1.73869, 1.72879, 1.75842, 1.86561, 1.73231, 1.73067, 1.71481, 1.72675, 1.72519, 1.72542, 1.72161, 1.74312, 1.7586, 1.73301, 1.73628, 1.73147, 1.73535, 1.72166, 1.7426, 1.73831, 1.74172, 1.73201, 1.72598, 1.73468, 1.72978, 1.74594, 1.72837, 1.72974, 1.72696, 1.72749, 1.71986, 1.72418, 1.74451, 1.73976, 1.72418, 1.73033, 1.72318, 1.72358, 1.72234, 1.73501, 1.74727, 1.73672, 1.73396, 1.72119, 1.73312, 1.73844, 1.73203, 1.72536, 1.72736, 1.72921, 1.72902, 1.72597, 1.729, 1.72536, 1.72794, 1.72241, 1.72447, 1.76392, 1.72969, 1.73799, 1.73613, 1.7343, 1.7378, 1.72936, 1.72889, 1.72255, 1.72257, 1.73736, 1.72374, 1.71941, 1.7165, 1.7345, 1.71725, 1.73605, 1.72722, 1.72686, 1.72866, 1.72684, 1.72293, 1.71739, 1.74362, 1.73332, 1.73303, 1.7425, 1.72774, 1.73892, 1.7353, 1.72182, 1.72797, 1.72439, 1.72746, 1.71428, 1.72893, 1.74479, 1.7415]}, "forward-compute-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [5.27974, 0.92476, 0.908, 0.90497, 0.89824, 0.90235, 0.89809, 0.8966, 0.90222, 0.89841, 0.89773, 0.89794, 0.91032, 0.90164, 0.90366, 0.8798, 0.85978, 0.85635, 0.86574, 0.9276, 0.86588, 0.86939, 0.86315, 0.85811, 0.86334, 0.87008, 0.86455, 1.01313, 0.86663, 0.86218, 0.85967, 0.8605, 0.86165, 0.86323, 1.14708, 0.85574, 0.8703, 0.86767, 0.86908, 0.86187, 1.07789, 0.86528, 1.12189, 0.85841, 0.86562, 0.86087, 0.86678, 0.85857, 0.85849, 0.85587, 0.86075, 0.85955, 0.86255, 0.86382, 0.86096, 0.86082, 0.88491, 0.86714, 0.86145, 0.86598, 0.86262, 0.86216, 0.8598, 0.86822, 0.86517, 0.8615, 0.85542, 0.86554, 0.85967, 0.88114, 0.87415, 0.87113, 0.87435, 0.87365, 0.88072, 0.87208, 0.88427, 0.87465, 0.87417, 0.87413, 0.86979, 0.87976, 0.87638, 0.88143, 0.87323, 0.88024, 0.87253, 0.87241, 0.87326, 0.87005, 0.87768, 0.8722, 0.87722, 0.87083, 0.87413, 0.87638, 0.87373, 0.87466, 0.87538, 0.8739, 0.87128, 0.87652, 0.87684, 0.87492, 0.87492, 0.87841, 0.88201, 0.87239, 0.87229, 0.8727, 0.8745, 0.87675, 1.03042, 0.87759, 0.87849, 0.87833, 0.87258, 0.87289, 0.87691, 0.87708, 0.87829, 0.87145, 0.87654, 0.87384, 0.87603, 0.87778, 0.87475, 0.88107, 0.88273, 0.8755, 0.88983, 0.87658, 0.88826, 0.88529, 0.87022, 0.86963, 0.87267, 0.86283, 0.86251, 0.86344, 0.86249, 0.85909, 0.86139, 0.87196, 0.86979, 0.88568, 0.87822, 0.87581, 0.87502, 0.88115, 0.88601, 0.8723, 0.8784, 0.87265, 0.86503, 0.86948, 0.87822, 0.88652, 0.88499, 0.88414, 0.88617, 0.87527, 1.00974, 0.87737, 0.87871, 0.87676, 0.88065, 1.0214, 0.88389, 0.88101, 0.87608, 0.88023, 0.88084, 0.88801, 0.87903, 0.87909, 0.87263, 0.87795, 0.87985, 0.87246, 0.87553, 0.87596, 0.87479, 0.87985, 0.88479, 0.87485, 0.87367, 0.87478, 0.88854, 0.86956, 0.87644, 0.87245, 0.88081, 0.87041, 0.88619, 1.02913, 0.88217, 0.87685, 0.87585, 0.87573, 0.87689, 1.15391, 0.88585, 0.87942, 0.88207, 0.87985, 0.87296, 0.87708, 0.87636, 0.87093, 0.8781, 0.87653, 0.87856, 0.87024, 0.88302, 0.87709, 0.88516, 0.88086, 0.881, 0.87553, 0.87679, 0.8639, 0.86032, 0.86351, 0.86184, 0.8859, 0.87955, 0.88593, 0.87819, 0.87667, 0.88472, 0.88141, 0.8836, 0.87845, 0.87966, 0.88392, 0.87781, 0.87099, 0.86132, 0.87548, 0.86865, 0.86776, 0.87463, 0.86901, 0.86998, 0.87005, 0.86783, 0.87008, 0.86883, 0.87182, 0.86786, 0.86944, 0.86712, 0.86634, 0.86996, 0.86649, 0.8693, 0.87065, 0.8695, 0.86742, 0.87595, 0.8798, 0.88174, 0.89356, 0.88888, 0.88392, 0.89001, 0.87835, 0.87956, 0.89109, 0.89368, 0.88418, 0.88296, 0.89126, 0.8815, 0.8757, 0.8795, 0.87994, 0.88066, 0.88371, 0.88006, 1.03877, 0.88852, 0.88485, 0.87943, 0.87942, 0.87742, 0.87816, 0.87364, 0.88536, 0.87926, 0.87207, 0.8692, 0.87981, 0.88494, 0.87843, 0.8858, 0.87785, 0.87487, 0.88061, 0.88278, 0.87623, 0.88861, 0.89711, 0.88263, 0.88098, 0.87228, 0.89083, 0.98169, 0.88718, 0.88541, 0.87728, 0.88271, 0.88471, 0.88101, 0.88129, 0.88509, 0.88811, 0.88892, 0.88848, 0.88806, 0.89311, 0.88677, 0.8931, 0.89243, 0.88674, 0.88201, 0.87923, 0.88648, 0.88669, 0.89113, 0.88862, 0.88512, 0.87385, 0.87365, 0.86762, 0.87279, 0.88084, 0.88115, 0.87063, 0.87302, 0.87228, 0.86979, 0.86968, 0.87774, 0.88151, 0.87809, 0.8777, 0.86883, 0.88423, 0.87251, 0.87362, 0.87846, 0.88901, 0.88901, 0.8903, 0.87767, 0.89278, 0.86871, 0.87407, 0.87211, 0.87185, 0.90188, 0.87839, 0.88045, 0.87551, 0.89016, 0.8888, 0.86903, 0.87126, 0.8686, 0.86688, 0.87951, 0.87084, 0.86641, 0.86045, 0.8685, 0.86338, 0.86591, 0.86874, 0.868, 0.86988, 0.86257, 0.86558, 0.86056, 0.86937, 0.86676, 0.87491, 0.87899, 0.86954, 0.87024, 0.87, 0.86476, 0.86347, 0.85924, 0.85839, 0.86084, 0.86428, 0.88494, 0.87888]}, "backward-compute-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.61138, 0.91507, 0.89466, 0.88764, 0.89351, 0.89127, 0.88566, 0.87739, 0.88475, 0.87298, 0.87085, 0.86968, 0.88216, 0.86716, 0.87363, 0.85479, 0.85473, 0.84913, 0.86094, 0.87134, 0.86851, 0.85568, 0.85368, 0.85232, 0.85432, 0.85092, 0.85061, 0.85479, 0.86242, 0.851, 0.85047, 0.85033, 0.85533, 0.85011, 0.85424, 0.85133, 0.85164, 0.86008, 0.84896, 0.85436, 0.85009, 0.85188, 0.84516, 0.85105, 0.84911, 0.85064, 0.85614, 0.85265, 0.85002, 0.85142, 0.85037, 0.85287, 0.84938, 0.84888, 0.85015, 0.84923, 0.85977, 0.8521, 0.85054, 0.85151, 0.85739, 0.8511, 0.85362, 0.86199, 0.85183, 0.84953, 0.84846, 0.85565, 0.8496, 0.86463, 0.84836, 0.846, 0.85149, 0.84996, 0.85524, 0.84993, 0.8621, 0.85083, 0.84627, 0.85239, 0.8468, 0.8558, 0.84961, 0.85553, 0.84238, 0.84755, 0.84118, 0.84308, 0.84064, 0.84121, 0.85217, 0.8417, 0.84514, 0.84333, 0.84864, 0.84592, 0.84643, 0.84487, 0.84697, 0.84689, 0.83238, 0.83815, 0.83582, 0.83558, 0.83878, 0.83583, 0.83366, 0.83299, 0.82963, 0.83401, 0.83512, 0.83867, 0.83585, 0.83291, 0.83492, 0.83421, 0.84142, 0.84662, 0.84889, 0.85184, 0.84665, 0.8493, 0.84818, 0.84392, 0.84382, 0.84606, 0.8466, 0.84836, 0.84785, 0.84999, 0.85142, 0.8476, 0.85095, 0.85574, 0.84838, 0.847, 0.85306, 0.84791, 0.84815, 0.84686, 0.84802, 0.84713, 0.84782, 0.8531, 0.84956, 0.84682, 0.8464, 0.85106, 0.8472, 0.84937, 0.86219, 0.84664, 0.85264, 0.84814, 0.85019, 0.85177, 0.85338, 0.84996, 0.84687, 0.86036, 0.86255, 0.84671, 0.84887, 0.84805, 0.85477, 0.84768, 0.86104, 0.85398, 0.84826, 0.84665, 0.84898, 0.85671, 0.85008, 0.85696, 0.855, 0.85115, 0.84581, 0.84531, 0.84777, 0.84786, 0.84844, 0.85929, 0.85028, 0.84593, 0.849, 0.84756, 0.84563, 0.84857, 0.85391, 0.84403, 0.85011, 0.84902, 0.84817, 0.8481, 0.84844, 0.84708, 0.84912, 0.84604, 0.84568, 0.84703, 0.84534, 0.85124, 0.8503, 0.84787, 0.8503, 0.84714, 0.84668, 0.8519, 0.85239, 0.84751, 0.85275, 0.85144, 0.84903, 0.84828, 0.85916, 0.84911, 0.84955, 0.84809, 0.85284, 0.85372, 0.85631, 0.85106, 0.84883, 0.85006, 0.8477, 0.84935, 0.85021, 0.85287, 0.84833, 0.84624, 0.84973, 0.85093, 0.85471, 0.85216, 0.85474, 0.86191, 0.85037, 0.85043, 0.85103, 0.85148, 0.85167, 0.85098, 0.85903, 0.85338, 0.85377, 0.85441, 0.85201, 0.85598, 0.85913, 0.85803, 0.8503, 0.85407, 0.85119, 0.85447, 0.85366, 0.8536, 0.85294, 0.85701, 0.85682, 0.8527, 0.85842, 0.85561, 0.85812, 0.86642, 0.85747, 0.85565, 0.86347, 0.84916, 0.84782, 0.86157, 0.85875, 0.85274, 0.85028, 0.85395, 0.8445, 0.84001, 0.83727, 0.8368, 0.84377, 0.84634, 0.85181, 0.8478, 0.85205, 0.84972, 0.85065, 0.85247, 0.84924, 0.84691, 0.84351, 0.84507, 0.84331, 0.84422, 0.84688, 0.84837, 0.84275, 0.83973, 0.8522, 0.846, 0.85116, 0.84637, 0.84391, 0.84359, 0.84426, 0.847, 0.84179, 0.84541, 0.84492, 0.85567, 0.88277, 0.84968, 0.84944, 0.84404, 0.85146, 0.84423, 0.84822, 0.84524, 0.84831, 0.85871, 0.84654, 0.84634, 0.84712, 0.85481, 0.84775, 0.85028, 0.84986, 0.85249, 0.85171, 0.84634, 0.85273, 0.84939, 0.85902, 0.85057, 0.85222, 0.8497, 0.85191, 0.84756, 0.85156, 0.86199, 0.85865, 0.85158, 0.85267, 0.85066, 0.8517, 0.853, 0.85486, 0.86228, 0.85677, 0.85444, 0.85096, 0.85419, 0.85697, 0.85415, 0.85344, 0.85057, 0.84957, 0.84846, 0.84903, 0.84876, 0.84807, 0.84926, 0.84798, 0.85028, 0.85864, 0.8555, 0.8584, 0.85401, 0.84649, 0.85263, 0.85661, 0.85475, 0.84958, 0.85258, 0.85845, 0.85462, 0.85336, 0.85504, 0.85019, 0.84394, 0.85064, 0.84532, 0.84911, 0.85298, 0.84658, 0.84921, 0.84856, 0.87125, 0.85999, 0.84821, 0.85567, 0.85311, 0.86131, 0.85589, 0.84993, 0.85075, 0.84962, 0.84874, 0.84913, 0.85332, 0.86182, 0.85561]}, "batch-generator-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [1.02426, 0.0329, 0.0249, 0.02644, 0.02588, 0.02655, 0.02669, 0.02578, 0.02382, 0.02208, 0.02137, 0.02091, 0.01978, 0.02148, 0.02156, 0.0211, 0.02062, 0.02039, 0.02049, 0.0216, 0.02173, 0.02121, 0.02058, 0.02072, 0.02029, 0.02074, 0.02026, 0.17277, 0.01978, 0.0205, 0.02, 0.0202, 0.02001, 0.0185, 0.02028, 0.01906, 0.02004, 0.01937, 0.02164, 0.01966, 0.01949, 0.02064, 0.27843, 0.02027, 0.02079, 0.02007, 0.01977, 0.01949, 0.01893, 0.02078, 0.02045, 0.01979, 0.02078, 0.0205, 0.02023, 0.02091, 0.02261, 0.02036, 0.02051, 0.01994, 0.02008, 0.01958, 0.02054, 0.02091, 0.02053, 0.02042, 0.02017, 0.02037, 0.02082, 0.02099, 0.02042, 0.0209, 0.0207, 0.02036, 0.02064, 0.02077, 0.02098, 0.02083, 0.02084, 0.02069, 0.02003, 0.02087, 0.02046, 0.02092, 0.0201, 0.02189, 0.02047, 0.02029, 0.02055, 0.02031, 0.02114, 0.02003, 0.02033, 0.0207, 0.02055, 0.02085, 0.02027, 0.02088, 0.02063, 0.02045, 0.01999, 0.02066, 0.02033, 0.02044, 0.02032, 0.02121, 0.02115, 0.0204, 0.02093, 0.02073, 0.02048, 0.02103, 0.02114, 0.02127, 0.02082, 0.02119, 0.02069, 0.02086, 0.021, 0.02104, 0.021, 0.02118, 0.02064, 0.02074, 0.02083, 0.02064, 0.02014, 0.02081, 0.0214, 0.02087, 0.02187, 0.02104, 0.02099, 0.02106, 0.0207, 0.02045, 0.0205, 0.0203, 0.02004, 0.01976, 0.02022, 0.02004, 0.02057, 0.0202, 0.02204, 0.02111, 0.02051, 0.02232, 0.02195, 0.02312, 0.0222, 0.02389, 0.02129, 0.02166, 0.02053, 0.02095, 0.02174, 0.02142, 0.02168, 0.02155, 0.02118, 0.0207, 0.02069, 0.02117, 0.02071, 0.02083, 0.02099, 0.16059, 0.02106, 0.02084, 0.02111, 0.02063, 0.02119, 0.02117, 0.02114, 0.02137, 0.02133, 0.02108, 0.02113, 0.02064, 0.02093, 0.02089, 0.02093, 0.02088, 0.0212, 0.02076, 0.02081, 0.02066, 0.02172, 0.02061, 0.02058, 0.0208, 0.02102, 0.02094, 0.02218, 0.17295, 0.02113, 0.02058, 0.02117, 0.02128, 0.35969, 0.02151, 0.0211, 0.0214, 0.0213, 0.02116, 0.02106, 0.02126, 0.02105, 0.02081, 0.02104, 0.02082, 0.02149, 0.02084, 0.02237, 0.0206, 0.02146, 0.02086, 0.02125, 0.02153, 0.02053, 0.02032, 0.02063, 0.01992, 0.02014, 0.04303, 0.02057, 0.02442, 0.02111, 0.02072, 0.0212, 0.02117, 0.02148, 0.02068, 0.02128, 0.02163, 0.02197, 0.02078, 0.02058, 0.02049, 0.01993, 0.01985, 0.02088, 0.02023, 0.02054, 0.02038, 0.02089, 0.02059, 0.0208, 0.02029, 0.02026, 0.02019, 0.02086, 0.02058, 0.02054, 0.02004, 0.02027, 0.02022, 0.02082, 0.01997, 0.02084, 0.02159, 0.02117, 0.02177, 0.02086, 0.02147, 0.02159, 0.02065, 0.02156, 0.02107, 0.02158, 0.02138, 0.02092, 0.02115, 0.02086, 0.02094, 0.02044, 0.02172, 0.02171, 0.02117, 0.02108, 0.18362, 0.0212, 0.02138, 0.021, 0.02133, 0.02101, 0.02222, 0.02173, 0.0209, 0.02105, 0.02026, 0.0203, 0.02138, 0.02138, 0.02124, 0.02189, 0.02133, 0.02099, 0.02092, 0.02135, 0.02105, 0.02186, 0.02137, 0.02079, 0.02122, 0.02095, 0.02196, 0.02475, 0.02099, 0.02097, 0.02135, 0.02151, 0.02119, 0.02172, 0.02161, 0.02281, 0.02135, 0.02147, 0.0214, 0.02095, 0.02134, 0.02077, 0.02105, 0.0211, 0.02123, 0.0206, 0.02066, 0.02073, 0.02048, 0.02256, 0.02159, 0.02174, 0.02167, 0.01909, 0.01984, 0.02252, 0.02096, 0.02085, 0.02038, 0.02062, 0.02065, 0.02019, 0.02166, 0.02036, 0.0205, 0.02063, 0.02107, 0.02006, 0.02268, 0.0204, 0.02079, 0.02162, 0.02206, 0.02151, 0.0224, 0.02095, 0.0223, 0.02048, 0.02019, 0.0206, 0.02065, 0.02061, 0.02138, 0.02213, 0.02136, 0.02138, 0.02185, 0.02053, 0.02168, 0.02001, 0.01992, 0.02119, 0.02112, 0.02044, 0.02033, 0.01944, 0.02022, 0.02026, 0.01989, 0.02043, 0.02022, 0.02011, 0.02051, 0.02071, 0.02048, 0.02137, 0.01947, 0.02084, 0.02018, 0.02001, 0.01966, 0.02054, 0.01911, 0.02098, 0.02074, 0.02055, 0.01954, 0.01982, 0.0206]}, "forward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [5.29414, 0.01849, 0.01577, 0.01544, 0.01522, 0.01549, 0.01476, 0.01521, 0.01608, 0.01508, 0.01504, 0.01467, 0.01464, 0.01476, 0.01466, 0.01509, 0.01494, 0.01537, 0.01531, 0.01765, 0.01498, 0.01516, 0.01457, 0.01469, 0.01511, 0.01501, 0.01494, 0.0147, 0.0156, 0.01512, 0.01511, 0.01426, 0.01524, 0.01471, 0.01434, 0.01491, 0.01566, 0.01521, 0.01533, 0.01484, 0.01527, 0.0153, 0.01526, 0.01553, 0.01555, 0.01538, 0.01472, 0.01524, 0.01475, 0.01538, 0.0153, 0.01496, 0.01466, 0.01512, 0.01513, 0.01511, 0.01523, 0.01544, 0.01485, 0.01531, 0.01527, 0.01482, 0.01527, 0.01519, 0.01517, 0.01471, 0.01509, 0.01499, 0.01497, 0.0154, 0.01547, 0.01551, 0.01547, 0.01555, 0.01567, 0.01541, 0.01498, 0.01537, 0.01548, 0.01538, 0.01521, 0.01559, 0.01561, 0.01542, 0.01555, 0.01516, 0.01527, 0.01559, 0.01571, 0.01493, 0.01562, 0.01543, 0.01556, 0.01595, 0.01527, 0.01566, 0.01555, 0.01584, 0.0154, 0.01559, 0.01531, 0.01552, 0.01518, 0.01571, 0.01557, 0.01509, 0.0155, 0.01537, 0.01557, 0.0152, 0.01562, 0.01552, 0.01529, 0.01531, 0.01548, 0.01557, 0.01566, 0.01499, 0.01536, 0.01527, 0.0156, 0.01512, 0.01572, 0.01519, 0.01522, 0.0157, 0.01561, 0.01538, 0.01509, 0.01534, 0.01576, 0.01545, 0.01514, 0.01562, 0.01553, 0.01521, 0.01538, 0.01501, 0.01537, 0.01551, 0.01535, 0.01536, 0.01524, 0.01517, 0.0157, 0.01547, 0.01543, 0.0156, 0.01547, 0.01558, 0.01588, 0.01571, 0.01546, 0.01569, 0.01524, 0.01546, 0.01566, 0.01568, 0.01551, 0.0156, 0.01559, 0.0155, 0.01584, 0.01556, 0.01555, 0.01575, 0.01529, 0.01572, 0.0157, 0.01568, 0.01574, 0.01542, 0.01566, 0.01559, 0.01534, 0.01573, 0.01588, 0.0155, 0.01579, 0.01539, 0.01542, 0.01531, 0.0158, 0.01569, 0.0151, 0.01551, 0.01572, 0.01564, 0.01563, 0.01609, 0.0154, 0.01577, 0.01532, 0.01548, 0.01678, 0.01554, 0.01577, 0.0156, 0.01568, 0.01547, 0.01622, 0.01714, 0.01578, 0.01563, 0.01565, 0.01575, 0.01556, 0.01595, 0.01585, 0.01567, 0.01544, 0.01582, 0.01566, 0.01555, 0.01581, 0.01577, 0.01599, 0.0157, 0.01603, 0.01561, 0.01546, 0.01538, 0.01567, 0.01545, 0.01552, 0.01534, 0.01588, 0.01606, 0.01568, 0.01534, 0.01574, 0.01544, 0.01571, 0.01529, 0.01571, 0.01562, 0.01526, 0.01584, 0.01522, 0.01679, 0.01548, 0.01505, 0.01526, 0.01537, 0.01522, 0.01522, 0.01525, 0.0154, 0.01561, 0.01545, 0.01503, 0.01522, 0.01538, 0.01527, 0.0152, 0.01511, 0.01518, 0.01546, 0.01556, 0.0152, 0.01516, 0.01588, 0.0154, 0.01555, 0.01555, 0.01589, 0.01585, 0.01516, 0.01578, 0.01698, 0.01562, 0.01567, 0.01565, 0.01574, 0.01528, 0.01532, 0.01576, 0.01576, 0.01531, 0.01581, 0.01562, 0.01551, 0.0159, 0.01558, 0.01542, 0.01561, 0.01565, 0.01562, 0.01551, 0.01603, 0.01561, 0.01503, 0.01544, 0.01568, 0.01534, 0.01553, 0.01577, 0.01562, 0.01594, 0.01576, 0.01582, 0.01594, 0.01574, 0.01565, 0.01587, 0.01573, 0.01524, 0.01564, 0.01568, 0.01568, 0.01566, 0.01557, 0.01563, 0.01592, 0.01578, 0.0153, 0.01557, 0.0156, 0.0154, 0.01546, 0.01545, 0.01593, 0.01593, 0.0158, 0.01595, 0.01603, 0.01577, 0.0157, 0.01574, 0.0156, 0.01565, 0.01558, 0.0162, 0.01532, 0.01522, 0.01536, 0.01552, 0.01528, 0.01549, 0.01528, 0.01513, 0.01546, 0.01554, 0.01541, 0.01597, 0.01543, 0.01541, 0.0159, 0.01547, 0.01591, 0.01544, 0.01537, 0.01558, 0.01589, 0.01598, 0.01593, 0.01562, 0.0157, 0.01529, 0.01534, 0.01537, 0.01535, 0.01515, 0.01552, 0.01585, 0.01569, 0.01598, 0.01579, 0.01528, 0.01539, 0.01527, 0.01514, 0.01524, 0.01536, 0.01545, 0.01555, 0.01509, 0.01486, 0.01553, 0.01523, 0.01539, 0.01546, 0.01501, 0.01559, 0.01528, 0.01527, 0.01524, 0.0155, 0.01552, 0.01555, 0.01532, 0.01541, 0.01518, 0.01514, 0.01527, 0.01493, 0.01513, 0.01525, 0.01553, 0.01567]}, "forward-send-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [1.14944, 0.00014, 0.00011, 0.0001, 0.0001, 0.00011, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 9e-05, 0.0001, 0.00012, 0.0002, 9e-05, 9e-05, 0.0001, 0.0001, 9e-05, 0.0001, 0.0001, 0.00012, 9e-05, 9e-05, 9e-05, 0.0001, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 0.0001, 9e-05, 0.0001, 9e-05, 9e-05, 0.0001, 9e-05, 0.0001, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 0.00011, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 0.0001, 0.0001, 0.0001, 9e-05, 9e-05, 0.00013, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 0.00011, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 0.0001, 9e-05, 0.0001, 9e-05, 9e-05, 0.0001, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 0.0001, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 0.0001, 0.0001, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 0.0001, 9e-05, 0.00012, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 0.00011, 0.0001, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 0.0001, 9e-05, 9e-05, 0.0001, 9e-05, 0.0001, 9e-05, 0.0001, 9e-05, 0.0001, 9e-05, 0.0001, 9e-05, 9e-05, 0.00011, 0.0001, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 0.00013, 0.00013, 0.0001, 0.0001, 0.0001, 9e-05, 0.0001, 0.0001, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 0.0001, 9e-05, 0.0001, 0.0001, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 0.00012, 9e-05, 9e-05, 0.0001, 0.0001, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 0.00011, 9e-05, 0.0001, 0.00012, 0.0001, 9e-05, 0.0001, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 0.00012, 0.0001, 9e-05, 0.0001, 0.0001, 9e-05, 0.0001, 0.0001, 9e-05, 9e-05, 0.0001, 0.00012, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 9e-05, 0.0001, 0.0001, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 0.00011, 0.00012, 0.0001, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 0.0001, 0.00011, 9e-05, 0.0001, 0.0001, 9e-05, 0.0001, 9e-05, 0.0001, 9e-05, 0.0001, 0.0001, 9e-05, 9e-05, 9e-05, 0.0001, 0.0001, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 0.0001, 0.0001, 0.0001, 0.0001, 9e-05, 9e-05, 9e-05, 0.0001, 0.0001, 0.0001, 9e-05, 0.0001, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 0.00011, 0.0001, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 0.0001, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 9e-05, 0.0001, 0.0001, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 0.0001, 9e-05, 0.0001, 9e-05, 9e-05, 9e-05, 0.0001, 9e-05, 0.0001, 9e-05]}, "backward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.01399, 0.01323, 0.01439, 0.0141, 0.01413, 0.01316, 0.01446, 0.01359, 0.01366, 0.01383, 0.01394, 0.01362, 0.01371, 0.01299, 0.01397, 0.01328, 0.01357, 0.01322, 0.01348, 0.01277, 0.01312, 0.01319, 0.0134, 0.01284, 0.01369, 0.01309, 0.01303, 0.01297, 0.01395, 0.01345, 0.01305, 0.01344, 0.01332, 0.01275, 0.01286, 0.01353, 0.01281, 0.01271, 0.01323, 0.013, 0.01321, 0.01335, 0.01302, 0.01378, 0.01302, 0.01312, 0.01355, 0.01324, 0.01352, 0.01346, 0.01354, 0.01315, 0.01335, 0.01339, 0.01286, 0.01344, 0.01341, 0.01332, 0.01334, 0.01323, 0.01361, 0.01324, 0.01322, 0.01341, 0.01309, 0.01364, 0.01336, 0.01332, 0.01332, 0.0132, 0.01335, 0.01494, 0.01374, 0.01376, 0.01329, 0.01354, 0.01368, 0.01359, 0.01303, 0.0133, 0.01343, 0.01318, 0.0134, 0.0135, 0.01381, 0.01334, 0.01337, 0.01297, 0.01348, 0.01291, 0.01378, 0.01345, 0.01356, 0.01329, 0.01335, 0.01339, 0.01368, 0.01358, 0.01315, 0.01306, 0.01384, 0.0132, 0.01277, 0.0133, 0.01348, 0.01354, 0.01436, 0.01344, 0.01333, 0.01358, 0.01527, 0.01401, 0.01361, 0.0139, 0.01355, 0.01399, 0.0136, 0.01366, 0.01353, 0.01394, 0.01369, 0.01388, 0.01336, 0.01347, 0.01367, 0.01369, 0.01346, 0.01339, 0.01351, 0.01392, 0.01357, 0.01364, 0.01352, 0.01382, 0.01325, 0.01389, 0.01309, 0.01636, 0.01335, 0.01361, 0.01365, 0.01329, 0.01346, 0.01332, 0.01388, 0.01361, 0.01349, 0.01347, 0.01328, 0.01355, 0.01391, 0.0134, 0.01392, 0.01339, 0.01382, 0.01352, 0.0146, 0.01318, 0.01344, 0.01356, 0.0138, 0.01316, 0.01329, 0.01336, 0.01409, 0.01342, 0.01364, 0.01379, 0.01317, 0.0132, 0.01351, 0.01355, 0.0137, 0.01391, 0.01363, 0.01329, 0.01345, 0.01328, 0.01343, 0.0132, 0.01389, 0.01328, 0.01323, 0.0136, 0.01364, 0.0141, 0.01319, 0.01314, 0.01355, 0.01362, 0.01341, 0.01311, 0.01366, 0.01354, 0.01397, 0.01382, 0.01338, 0.01322, 0.01367, 0.01319, 0.01345, 0.01366, 0.01346, 0.0135, 0.01345, 0.01345, 0.01296, 0.0137, 0.01356, 0.01338, 0.01337, 0.01338, 0.01343, 0.01367, 0.01374, 0.0135, 0.01383, 0.0135, 0.0135, 0.0135, 0.01322, 0.01373, 0.01326, 0.01327, 0.01321, 0.01329, 0.01369, 0.01393, 0.01472, 0.01343, 0.01339, 0.01351, 0.0134, 0.01376, 0.01357, 0.01341, 0.01321, 0.01361, 0.01355, 0.0134, 0.01357, 0.01352, 0.01323, 0.01333, 0.01309, 0.01279, 0.01341, 0.01356, 0.01367, 0.01351, 0.01365, 0.01348, 0.01363, 0.01354, 0.01364, 0.01325, 0.0135, 0.01298, 0.01355, 0.01376, 0.01358, 0.0134, 0.01318, 0.01328, 0.01339, 0.01375, 0.01335, 0.01335, 0.01341, 0.01326, 0.01339, 0.01334, 0.0133, 0.01334, 0.01346, 0.01314, 0.01386, 0.01417, 0.0138, 0.01369, 0.01375, 0.0131, 0.01349, 0.01438, 0.01391, 0.01419, 0.01455, 0.01387, 0.01391, 0.01388, 0.01384, 0.01394, 0.01408, 0.01389, 0.01334, 0.01368, 0.01364, 0.01318, 0.01409, 0.01369, 0.01307, 0.01309, 0.01442, 0.01442, 0.01387, 0.01355, 0.01369, 0.01515, 0.01375, 0.0131, 0.01295, 0.01347, 0.01348, 0.01339, 0.01344, 0.01348, 0.01449, 0.0139, 0.01418, 0.0137, 0.01365, 0.01373, 0.01341, 0.01337, 0.01401, 0.01387, 0.01364, 0.01394, 0.01386, 0.0136, 0.01327, 0.01354, 0.01365, 0.01346, 0.01357, 0.01323, 0.01345, 0.01362, 0.01421, 0.01349, 0.01356, 0.0133, 0.01342, 0.01393, 0.01294, 0.01345, 0.01332, 0.01347, 0.0134, 0.01344, 0.01464, 0.01384, 0.01344, 0.01378, 0.01261, 0.01312, 0.01323, 0.01366, 0.01307, 0.01329, 0.01305, 0.01339, 0.01326, 0.01354, 0.013, 0.01336, 0.01331, 0.01319, 0.01341, 0.01357, 0.01368, 0.01314, 0.01403, 0.0134, 0.01315, 0.01334, 0.01337, 0.01337, 0.01355, 0.01319, 0.01341, 0.01355, 0.01312, 0.01328, 0.01334, 0.01325, 0.01313, 0.01385, 0.0136, 0.01308, 0.01305, 0.01317, 0.0135, 0.01349, 0.01334, 0.01329, 0.01268, 0.01343, 0.01322, 0.01354]}, "backward-send-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00014, 0.00017, 0.00017, 0.00018, 0.00014, 0.00014, 0.00017, 0.00013, 0.00017, 0.00014, 0.00013, 0.00017, 0.00017, 0.00017, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00016, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00014, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00014, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00014, 0.00014, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00015, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00011, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00015, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00015, 0.00014, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00012, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00014, 0.00013, 0.00012, 0.00013, 0.00014, 0.00014, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00015, 0.00012, 0.00014, 0.00012, 0.00013, 0.00012, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00014, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013, 0.00013, 0.00014, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00013, 0.00013]}, "forward-send-backward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [6.38697, 0.04018, 0.05114, 0.05601, 0.05873, 0.05195, 0.04987, 0.05386, 0.0467, 0.06235, 0.05096, 0.05, 0.04356, 0.05077, 0.05412, 0.04405, 0.06755, 0.06516, 0.07663, 0.0433, 0.03979, 0.03715, 0.05255, 0.04816, 0.05197, 0.04384, 0.04425, 0.04907, 0.04283, 0.05974, 0.04362, 0.04976, 0.05271, 0.04377, 0.35111, 0.05242, 0.04081, 0.04836, 0.0552, 0.06056, 0.06082, 0.04572, 0.0485, 0.04555, 0.05074, 0.05021, 0.05488, 0.05383, 0.05437, 0.05459, 0.05261, 0.05295, 0.04898, 0.05179, 0.05377, 0.05217, 0.04713, 0.05227, 0.05549, 0.04959, 0.06902, 0.05336, 0.05215, 0.05649, 0.05608, 0.05937, 0.05649, 0.05375, 0.05632, 0.04937, 0.05043, 0.0527, 0.04686, 0.04528, 0.05122, 0.05016, 0.04472, 0.04442, 0.05164, 0.0466, 0.05055, 0.06029, 0.05474, 0.04835, 0.05161, 0.04652, 0.05275, 0.05027, 0.04993, 0.04972, 0.05958, 0.04592, 0.05065, 0.05336, 0.04616, 0.04607, 0.04493, 0.05229, 0.05286, 0.04993, 0.05639, 0.05282, 0.06146, 0.06286, 0.06387, 0.06047, 0.06233, 0.05922, 0.05856, 0.06096, 0.06608, 0.05802, 0.24394, 0.0543, 0.06111, 0.05823, 0.0515, 0.04933, 0.0552, 0.0466, 0.04993, 0.05055, 0.05602, 0.05161, 0.05172, 0.05064, 0.05203, 0.04687, 0.04181, 0.04201, 0.04335, 0.04237, 0.0379, 0.04024, 0.04624, 0.04904, 0.04284, 0.04865, 0.05318, 0.05688, 0.05379, 0.05465, 0.05463, 0.05795, 0.05672, 0.05633, 0.05259, 0.04848, 0.05166, 0.04998, 0.04771, 0.0491, 0.05044, 0.05014, 0.05551, 0.05319, 0.04673, 0.04602, 0.04842, 0.04265, 0.05122, 0.05095, 0.21106, 0.04994, 0.05747, 0.04375, 0.04899, 0.04385, 0.05122, 0.05645, 0.05822, 0.04817, 0.04906, 0.04682, 0.05428, 0.04907, 0.04982, 0.0557, 0.05776, 0.04846, 0.04442, 0.04182, 0.04942, 0.05261, 0.04575, 0.04697, 0.05955, 0.05463, 0.05978, 0.06309, 0.05621, 0.05425, 0.06256, 0.0578, 0.05102, 0.05338, 0.04999, 0.0479, 0.04606, 0.04367, 0.06008, 0.02804, 0.04771, 0.04548, 0.04455, 0.04154, 0.05402, 0.04873, 0.04935, 0.05024, 0.05543, 0.05585, 0.05276, 0.05753, 0.0581, 0.05616, 0.05672, 0.05125, 0.05363, 0.05413, 0.05549, 0.05512, 0.05756, 0.05931, 0.06033, 0.05832, 0.05802, 0.04943, 0.05106, 0.05706, 0.05065, 0.04361, 0.04691, 0.04829, 0.04424, 0.04914, 0.04665, 0.04713, 0.05329, 0.04757, 0.05485, 0.05316, 0.05854, 0.05352, 0.05543, 0.06179, 0.0553, 0.05379, 0.05248, 0.05376, 0.0502, 0.04979, 0.04897, 0.0512, 0.04778, 0.05176, 0.04751, 0.04764, 0.04922, 0.04979, 0.0426, 0.04577, 0.04617, 0.04402, 0.0434, 0.04604, 0.04551, 0.0488, 0.04843, 0.04906, 0.04756, 0.04709, 0.05359, 0.05485, 0.04989, 0.05155, 0.06944, 0.07321, 0.06088, 0.06389, 0.06638, 0.06567, 0.06076, 0.06339, 0.06625, 0.06534, 0.06787, 0.06199, 0.07012, 0.0655, 0.07256, 0.06984, 0.0689, 0.0634, 0.06663, 0.06266, 0.05694, 0.06832, 0.0594, 0.05576, 0.06391, 0.0573, 0.06422, 0.06444, 0.06765, 0.06433, 0.0655, 0.06109, 0.05275, 0.05136, 0.04868, 0.04719, 0.04868, 0.05021, 0.04823, 0.04759, 0.05882, 0.07525, 0.04803, 0.05204, 0.04726, 0.03991, 0.03848, 0.05475, 0.04907, 0.0624, 0.05486, 0.05835, 0.05204, 0.04832, 0.04886, 0.05172, 0.04399, 0.05413, 0.05631, 0.05744, 0.0523, 0.05914, 0.05482, 0.05773, 0.06129, 0.05258, 0.05842, 0.05233, 0.05639, 0.05902, 0.05897, 0.05693, 0.05299, 0.04834, 0.06334, 0.05971, 0.05273, 0.04536, 0.04564, 0.04144, 0.04847, 0.04042, 0.05862, 0.05768, 0.05357, 0.05353, 0.05478, 0.04817, 0.05044, 0.05169, 0.04269, 0.0443, 0.05639, 0.05494, 0.05594, 0.0527, 0.05179, 0.05078, 0.04955, 0.05161, 0.05872, 0.05658, 0.06249, 0.05896, 0.05678, 0.05506, 0.06666, 0.05614, 0.05873, 0.05324, 0.05836, 0.05877, 0.05866, 0.05716, 0.05964, 0.05831, 0.05562, 0.06136, 0.0624, 0.06832, 0.05467, 0.06074, 0.05704, 0.0582]}, "backward-send-forward-recv-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.27056, 0.05321, 0.02395, 0.02619, 0.02521, 0.02973, 0.02321, 0.02069, 0.02424, 0.02149, 0.01901, 0.02414, 0.03676, 0.02004, 0.02545, 0.02745, 0.01146, 0.01461, 0.01172, 0.06025, 0.02102, 0.02101, 0.01696, 0.01774, 0.01439, 0.02087, 0.01731, 0.16985, 0.01985, 0.01352, 0.01806, 0.01439, 0.0155, 0.01762, 0.01896, 0.01564, 0.02044, 0.02368, 0.01254, 0.01416, 0.23016, 0.01705, 0.27563, 0.01513, 0.01365, 0.01311, 0.01215, 0.01362, 0.01278, 0.01189, 0.01265, 0.01249, 0.0124, 0.01431, 0.01481, 0.01327, 0.01483, 0.01286, 0.01206, 0.01463, 0.01244, 0.01308, 0.01213, 0.01465, 0.01167, 0.01178, 0.01236, 0.01343, 0.01221, 0.01484, 0.01308, 0.01209, 0.0156, 0.01428, 0.01766, 0.01399, 0.01873, 0.01523, 0.01199, 0.01338, 0.01288, 0.0137, 0.01206, 0.01417, 0.01277, 0.01565, 0.01233, 0.01353, 0.0135, 0.01412, 0.01278, 0.01451, 0.01335, 0.01435, 0.01508, 0.01772, 0.01478, 0.01215, 0.01264, 0.01466, 0.01141, 0.01721, 0.01181, 0.01205, 0.01134, 0.01213, 0.01384, 0.0119, 0.01272, 0.01118, 0.01148, 0.01115, 0.01419, 0.01292, 0.01139, 0.01213, 0.01238, 0.01461, 0.01173, 0.01384, 0.01255, 0.01365, 0.01207, 0.01199, 0.01186, 0.0117, 0.01268, 0.01254, 0.0135, 0.01597, 0.02046, 0.01378, 0.01954, 0.01809, 0.014, 0.01212, 0.01496, 0.01378, 0.01273, 0.01214, 0.01143, 0.01276, 0.01125, 0.01212, 0.01108, 0.01241, 0.01148, 0.015, 0.01253, 0.01635, 0.02591, 0.01277, 0.0127, 0.01269, 0.01116, 0.01436, 0.01275, 0.0185, 0.01871, 0.01525, 0.01294, 0.01183, 0.01366, 0.01207, 0.01489, 0.01357, 0.01333, 0.15823, 0.01342, 0.01265, 0.01186, 0.01437, 0.01406, 0.0141, 0.01168, 0.01348, 0.0129, 0.01227, 0.01286, 0.01352, 0.01405, 0.01486, 0.01468, 0.01211, 0.01803, 0.0155, 0.01203, 0.013, 0.01327, 0.01162, 0.01277, 0.01431, 0.01404, 0.01375, 0.01696, 0.1659, 0.01775, 0.01902, 0.01424, 0.01614, 0.01287, 0.27201, 0.01543, 0.01337, 0.0157, 0.01845, 0.0134, 0.01417, 0.01659, 0.01271, 0.01198, 0.01225, 0.01357, 0.01181, 0.01216, 0.01226, 0.0134, 0.01493, 0.01616, 0.0124, 0.01139, 0.01234, 0.01342, 0.01268, 0.01167, 0.03678, 0.01167, 0.01517, 0.01192, 0.01182, 0.01281, 0.01455, 0.01415, 0.01241, 0.01418, 0.01332, 0.01403, 0.01506, 0.01131, 0.01827, 0.01234, 0.01284, 0.01296, 0.01215, 0.01151, 0.01261, 0.01275, 0.01282, 0.01199, 0.01391, 0.01197, 0.01214, 0.01113, 0.0127, 0.0122, 0.01149, 0.01163, 0.01365, 0.01859, 0.0172, 0.02036, 0.01842, 0.01887, 0.01782, 0.02133, 0.01801, 0.02215, 0.0172, 0.01796, 0.01826, 0.0219, 0.01935, 0.01681, 0.02619, 0.01735, 0.01281, 0.01144, 0.01152, 0.01711, 0.01687, 0.01612, 0.17976, 0.01531, 0.01219, 0.01569, 0.01642, 0.01536, 0.01137, 0.01144, 0.01318, 0.01122, 0.01129, 0.01132, 0.01149, 0.01153, 0.012, 0.0132, 0.01167, 0.01221, 0.01237, 0.01275, 0.01213, 0.01162, 0.01554, 0.01173, 0.01183, 0.01215, 0.01526, 0.08468, 0.01333, 0.01392, 0.01562, 0.01788, 0.0139, 0.01552, 0.01452, 0.01693, 0.01196, 0.01296, 0.01374, 0.01278, 0.01554, 0.01542, 0.01382, 0.01269, 0.01278, 0.01287, 0.01238, 0.01247, 0.01279, 0.01266, 0.0131, 0.01537, 0.01288, 0.0124, 0.0116, 0.01273, 0.01235, 0.01342, 0.01194, 0.01178, 0.01223, 0.01223, 0.01244, 0.01219, 0.01296, 0.01226, 0.01173, 0.01464, 0.01332, 0.01237, 0.01163, 0.01322, 0.01488, 0.01492, 0.01997, 0.01383, 0.01982, 0.01175, 0.01194, 0.01173, 0.014, 0.03556, 0.0162, 0.01538, 0.01361, 0.01715, 0.01531, 0.01491, 0.01261, 0.01202, 0.012, 0.01376, 0.01233, 0.01674, 0.01779, 0.01167, 0.01245, 0.01226, 0.01145, 0.0123, 0.01193, 0.01141, 0.01315, 0.01148, 0.02204, 0.0162, 0.01338, 0.01211, 0.01177, 0.01745, 0.01798, 0.01299, 0.01124, 0.01163, 0.01154, 0.01183, 0.01135, 0.01151, 0.01162]}, "layernorm-grads-all-reduce-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2e-05, 2e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 3e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 3e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 4e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05]}, "embedding-grads-all-reduce-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00041, 0.00029, 0.00021, 0.00021, 0.00021, 0.00021, 0.00022, 0.00023, 0.00021, 0.00021, 0.00021, 0.00022, 0.0002, 0.00021, 0.00022, 0.00024, 0.00021, 0.00021, 0.00021, 0.00021, 0.00021, 0.00019, 0.0002, 0.0002, 0.00021, 0.00021, 0.0002, 0.0002, 0.0002, 0.00021, 0.0002, 0.0002, 0.0002, 0.0002, 0.00021, 0.0002, 0.00021, 0.0002, 0.0002, 0.0002, 0.00019, 0.0002, 0.00022, 0.0002, 0.0002, 0.00021, 0.0002, 0.00021, 0.0002, 0.00019, 0.0002, 0.0002, 0.00019, 0.00019, 0.00019, 0.00022, 0.0002, 0.0002, 0.0002, 0.00019, 0.0002, 0.00019, 0.0002, 0.0002, 0.00021, 0.0002, 0.0002, 0.00019, 0.00019, 0.00019, 0.00019, 0.0002, 0.00019, 0.00019, 0.00019, 0.00022, 0.0002, 0.0002, 0.00019, 0.0002, 0.00019, 0.00019, 0.00021, 0.0002, 0.0002, 0.0002, 0.00019, 0.0002, 0.00019, 0.00019, 0.00019, 0.00019, 0.00021, 0.0002, 0.00019, 0.00019, 0.0002, 0.00019, 0.0002, 0.00019, 0.00019, 0.00021, 0.00021, 0.0002, 0.00021, 0.0002, 0.00019, 0.0002, 0.00021, 0.00019, 0.00019, 0.00019, 0.0002, 0.00019, 0.0002, 0.0002, 0.0002, 0.00019, 0.00019, 0.00019, 0.0002, 0.0002, 0.00019, 0.00019, 0.0002, 0.00019, 0.0002, 0.0002, 0.00019, 0.0002, 0.0002, 0.00021, 0.0002, 0.0002, 0.0002, 0.00019, 0.0002, 0.00019, 0.0002, 0.00021, 0.0002, 0.00019, 0.00021, 0.00019, 0.0002, 0.00021, 0.00019, 0.0002, 0.00019, 0.00019, 0.00019, 0.00019, 0.00021, 0.00021, 0.0002, 0.0002, 0.00021, 0.00019, 0.0002, 0.00019, 0.0002, 0.00019, 0.0002, 0.00019, 0.0002, 0.00021, 0.00022, 0.00019, 0.0002, 0.0002, 0.00019, 0.00019, 0.0002, 0.00019, 0.0002, 0.00019, 0.0002, 0.00019, 0.00019, 0.0002, 0.0002, 0.00019, 0.00019, 0.0002, 0.0002, 0.0002, 0.00019, 0.00019, 0.00019, 0.00019, 0.0002, 0.0002, 0.00019, 0.00019, 0.0002, 0.00019, 0.00021, 0.00019, 0.0002, 0.00019, 0.00021, 0.00022, 0.00022, 0.00019, 0.0002, 0.0002, 0.0002, 0.0002, 0.00023, 0.00019, 0.00019, 0.00019, 0.00019, 0.0002, 0.00019, 0.00021, 0.00019, 0.00023, 0.00021, 0.00021, 0.0002, 0.00019, 0.0002, 0.0002, 0.00019, 0.0002, 0.00019, 0.0002, 0.00021, 0.00019, 0.00019, 0.00019, 0.0002, 0.0002, 0.00019, 0.00019, 0.00019, 0.0002, 0.0002, 0.00022, 0.0002, 0.0002, 0.00021, 0.00021, 0.0002, 0.00019, 0.0002, 0.0002, 0.00019, 0.0002, 0.00019, 0.00019, 0.00019, 0.00019, 0.00019, 0.0002, 0.0002, 0.00019, 0.0002, 0.0002, 0.00019, 0.00019, 0.00019, 0.0002, 0.00021, 0.0002, 0.0002, 0.00019, 0.0002, 0.0002, 0.0002, 0.0002, 0.00019, 0.00019, 0.00019, 0.00019, 0.00019, 0.00022, 0.0002, 0.00019, 0.00019, 0.00019, 0.00019, 0.0002, 0.00019, 0.0002, 0.00019, 0.0002, 0.00019, 0.0002, 0.00019, 0.00019, 0.0002, 0.00019, 0.00019, 0.00019, 0.00019, 0.00021, 0.00019, 0.00019, 0.00019, 0.00019, 0.0002, 0.0002, 0.00019, 0.00019, 0.0002, 0.0002, 0.0002, 0.00019, 0.00019, 0.00019, 0.00019, 0.0002, 0.00021, 0.0002, 0.0002, 0.0002, 0.00019, 0.00019, 0.0002, 0.0002, 0.00019, 0.00019, 0.0002, 0.00019, 0.0002, 0.00019, 0.0002, 0.00019, 0.0002, 0.0002, 0.00019, 0.0002, 0.00021, 0.0002, 0.0002, 0.0002, 0.00019, 0.00019, 0.0002, 0.0002, 0.00019, 0.00019, 0.00019, 0.0002, 0.00019, 0.0002, 0.0002, 0.0002, 0.0002, 0.00019, 0.0002, 0.00019, 0.00021, 0.00019, 0.00019, 0.00019, 0.00019, 0.0002, 0.0002, 0.00019, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.00019, 0.00019, 0.00019, 0.0002, 0.00019, 0.0002, 0.0002, 0.00019, 0.00019, 0.0002, 0.00019, 0.00019, 0.0002, 0.00019, 0.00019, 0.0002, 0.00021, 0.00022, 0.0002, 0.00019, 0.00019, 0.0002, 0.0002, 0.00019, 0.0002, 0.00019, 0.00019, 0.0002, 0.00019, 0.00019, 0.00019]}, "all-grads-sync-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.63844, 0.00028, 0.00029, 0.00026, 0.00022, 0.00026, 0.00024, 0.00022, 0.00025, 0.00027, 0.00021, 0.0002, 0.00022, 0.0002, 0.00025, 0.00031, 0.0002, 0.00021, 0.0002, 0.00019, 0.00017, 0.00019, 0.00019, 0.00019, 0.00018, 0.00021, 0.0002, 0.0002, 0.0002, 0.00019, 0.00019, 0.0002, 0.00021, 0.00019, 0.00017, 0.00019, 0.00018, 0.0002, 0.00018, 0.00018, 0.00018, 0.0002, 0.00018, 0.0002, 0.00018, 0.00018, 0.0002, 0.0002, 0.00019, 0.00017, 0.0002, 0.0002, 0.00017, 0.00021, 0.00017, 0.00017, 0.00017, 0.00017, 0.00017, 0.00017, 0.00019, 0.00017, 0.00019, 0.00021, 0.00019, 0.00018, 0.00019, 0.00017, 0.00018, 0.0002, 0.00017, 0.00018, 0.00018, 0.00018, 0.00018, 0.0002, 0.00017, 0.00019, 0.00018, 0.00018, 0.00017, 0.00017, 0.00018, 0.00019, 0.00018, 0.00019, 0.00018, 0.00019, 0.00017, 0.00017, 0.00017, 0.00017, 0.00018, 0.00018, 0.00019, 0.00022, 0.0002, 0.00019, 0.00019, 0.00017, 0.00018, 0.00018, 0.00019, 0.00018, 0.00017, 0.00018, 0.00018, 0.00017, 0.00017, 0.00017, 0.00018, 0.00017, 0.00018, 0.00017, 0.00017, 0.0002, 0.00017, 0.00017, 0.00018, 0.00017, 0.00017, 0.00018, 0.00018, 0.00017, 0.00017, 0.00017, 0.00017, 0.0002, 0.00019, 0.0002, 0.0002, 0.00018, 0.00018, 0.00019, 0.00018, 0.00018, 0.0002, 0.00018, 0.00019, 0.00017, 0.00017, 0.00018, 0.00018, 0.00018, 0.0002, 0.00022, 0.00018, 0.00023, 0.00019, 0.00018, 0.00019, 0.00017, 0.00018, 0.0002, 0.00017, 0.00017, 0.00019, 0.00018, 0.00019, 0.00018, 0.00021, 0.00017, 0.0002, 0.00019, 0.00017, 0.00018, 0.00019, 0.00018, 0.00019, 0.00019, 0.00019, 0.00018, 0.00019, 0.00018, 0.0002, 0.00018, 0.00018, 0.00017, 0.00017, 0.00018, 0.00017, 0.00017, 0.00018, 0.00018, 0.00019, 0.00019, 0.00018, 0.00018, 0.00017, 0.00019, 0.00019, 0.00018, 0.00017, 0.00017, 0.00019, 0.00018, 0.00017, 0.00019, 0.00017, 0.00017, 0.00023, 0.00027, 0.00024, 0.00017, 0.00019, 0.0002, 0.00018, 0.00019, 0.00026, 0.0002, 0.00018, 0.00017, 0.00018, 0.00018, 0.00018, 0.00018, 0.0002, 0.00019, 0.00022, 0.00019, 0.00018, 0.00018, 0.00017, 0.00018, 0.00018, 0.00021, 0.00017, 0.00022, 0.00019, 0.00019, 0.00019, 0.00019, 0.00019, 0.00019, 0.00017, 0.00018, 0.00018, 0.00021, 0.00018, 0.00023, 0.0002, 0.00017, 0.00018, 0.0002, 0.00017, 0.00021, 0.00018, 0.0002, 0.00017, 0.00019, 0.00018, 0.00017, 0.00017, 0.0002, 0.00017, 0.00018, 0.00019, 0.00018, 0.00019, 0.00019, 0.00017, 0.00018, 0.00017, 0.00018, 0.0002, 0.00018, 0.00019, 0.00017, 0.00019, 0.00019, 0.00019, 0.00019, 0.00018, 0.00019, 0.00019, 0.00018, 0.00018, 0.00019, 0.00018, 0.00019, 0.00019, 0.00019, 0.00017, 0.00025, 0.00017, 0.00022, 0.00017, 0.00017, 0.00018, 0.00018, 0.00017, 0.00017, 0.0002, 0.00019, 0.00018, 0.00017, 0.00019, 0.00018, 0.00019, 0.00018, 0.00018, 0.00019, 0.00017, 0.00019, 0.00018, 0.00017, 0.00022, 0.00021, 0.00019, 0.00019, 0.00018, 0.00018, 0.00018, 0.00017, 0.00017, 0.00018, 0.0002, 0.00017, 0.00019, 0.00018, 0.0002, 0.00018, 0.00019, 0.00018, 0.00019, 0.00018, 0.00017, 0.00019, 0.00018, 0.00017, 0.00019, 0.00017, 0.00019, 0.00017, 0.00019, 0.00019, 0.00021, 0.00019, 0.00019, 0.00017, 0.00018, 0.00018, 0.00018, 0.00017, 0.00017, 0.00018, 0.00018, 0.0002, 0.00017, 0.00018, 0.0002, 0.00019, 0.00018, 0.00018, 0.00019, 0.00019, 0.00018, 0.00017, 0.00018, 0.00018, 0.00017, 0.00017, 0.00017, 0.00021, 0.00018, 0.00019, 0.00018, 0.00017, 0.0002, 0.00017, 0.00017, 0.00017, 0.00018, 0.00017, 0.00021, 0.00018, 0.0002, 0.00018, 0.00018, 0.00017, 0.00018, 0.00019, 0.00019, 0.00017, 0.00018, 0.00019, 0.00019, 0.00019, 0.00017, 0.00018, 0.00017, 0.00019, 0.0002, 0.0002, 0.00017, 0.00018, 0.00017, 0.00018, 0.0002, 0.00018]}, "optimizer-copy-to-main-grad-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00015, 0.00018, 0.00012, 0.00013, 0.00013, 0.00013, 0.00013, 0.00012, 0.00012, 0.00012, 0.00012, 0.00011, 0.00013, 0.00012, 0.00012, 0.00014, 8e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 8e-05, 7e-05, 7e-05, 8e-05, 8e-05, 8e-05, 7e-05, 8e-05, 9e-05, 8e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 8e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 0.00011, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 9e-05, 7e-05, 7e-05, 8e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 9e-05, 8e-05, 8e-05, 8e-05, 8e-05, 7e-05, 7e-05, 0.0001, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 0.0001, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 0.00011, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 0.0001, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05]}, "optimizer-clip-main-grad-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.48907, 0.00115, 0.00068, 0.00069, 0.00072, 0.00069, 0.00069, 0.00074, 0.0007, 0.0007, 0.00067, 0.00069, 0.00067, 0.00066, 0.00068, 0.00075, 0.00065, 0.00068, 0.00068, 0.00068, 0.00064, 0.00065, 0.00065, 0.00066, 0.00065, 0.00072, 0.00063, 0.00064, 0.00083, 0.00065, 0.00066, 0.00065, 0.00064, 0.00066, 0.00067, 0.00068, 0.00066, 0.00065, 0.00065, 0.00066, 0.00063, 0.00064, 0.00063, 0.00064, 0.00065, 0.00065, 0.00064, 0.00064, 0.00064, 0.00065, 0.00064, 0.00063, 0.00063, 0.00063, 0.00064, 0.00066, 0.00064, 0.00065, 0.00064, 0.00065, 0.00063, 0.00064, 0.00065, 0.00068, 0.00063, 0.00065, 0.00066, 0.00064, 0.00064, 0.00064, 0.00065, 0.00063, 0.00063, 0.00065, 0.00064, 0.00063, 0.00067, 0.00066, 0.00065, 0.00065, 0.00064, 0.00063, 0.00064, 0.00064, 0.00063, 0.00065, 0.00066, 0.00063, 0.00064, 0.00064, 0.00066, 0.00064, 0.00064, 0.00064, 0.00058, 0.00065, 0.00061, 0.00064, 0.00072, 0.00064, 0.00065, 0.00067, 0.00064, 0.00067, 0.00064, 0.00064, 0.00065, 0.00064, 0.00064, 0.00062, 0.00059, 0.0006, 0.00065, 0.00058, 0.00065, 0.00066, 0.00065, 0.00064, 0.00058, 0.00064, 0.00064, 0.00064, 0.00064, 0.00065, 0.00062, 0.00065, 0.00063, 0.00064, 0.00063, 0.00065, 0.00066, 0.00064, 0.00065, 0.00064, 0.00063, 0.00064, 0.00061, 0.00064, 0.00064, 0.00065, 0.00064, 0.00066, 0.00064, 0.00064, 0.00058, 0.00064, 0.00067, 0.00063, 0.00065, 0.00064, 0.00064, 0.00064, 0.00063, 0.00064, 0.00063, 0.00064, 0.00066, 0.00065, 0.00066, 0.00068, 0.00067, 0.00064, 0.00066, 0.00068, 0.00063, 0.00065, 0.00065, 0.00067, 0.00066, 0.00064, 0.00065, 0.00064, 0.00067, 0.00064, 0.00067, 0.00064, 0.00064, 0.00063, 0.00072, 0.00063, 0.00065, 0.00064, 0.00065, 0.00065, 0.00068, 0.00065, 0.00063, 0.00063, 0.00065, 0.00064, 0.00064, 0.00064, 0.00065, 0.00066, 0.00065, 0.00063, 0.00064, 0.00063, 0.00063, 0.00069, 0.00067, 0.00069, 0.00066, 0.00063, 0.00068, 0.00065, 0.00064, 0.00065, 0.00066, 0.00065, 0.00072, 0.00064, 0.00065, 0.00063, 0.00064, 0.00066, 0.00064, 0.00067, 0.00065, 0.00065, 0.00066, 0.00064, 0.00067, 0.00068, 0.00067, 0.00064, 0.00064, 0.00067, 0.00068, 0.00066, 0.00074, 0.00065, 0.00064, 0.00064, 0.00071, 0.00071, 0.00065, 0.00064, 0.00064, 0.00106, 0.00065, 0.00064, 0.00068, 0.00065, 0.00065, 0.00064, 0.00065, 0.00063, 0.00063, 0.00066, 0.00064, 0.00065, 0.00065, 0.00064, 0.00064, 0.00065, 0.00065, 0.00063, 0.0007, 0.00064, 0.00063, 0.00064, 0.00064, 0.00064, 0.00066, 0.00072, 0.00063, 0.00064, 0.00064, 0.00065, 0.00065, 0.00066, 0.00071, 0.00064, 0.00063, 0.00063, 0.00066, 0.00065, 0.00063, 0.00064, 0.00064, 0.00064, 0.00065, 0.00076, 0.00064, 0.00065, 0.00074, 0.00063, 0.00065, 0.00065, 0.00073, 0.00064, 0.00065, 0.00064, 0.00064, 0.00063, 0.00065, 0.00066, 0.00065, 0.00063, 0.00066, 0.00064, 0.00064, 0.00067, 0.00064, 0.00066, 0.00071, 0.0007, 0.00066, 0.00066, 0.00073, 0.00063, 0.00063, 0.00064, 0.00063, 0.00064, 0.00068, 0.00066, 0.00064, 0.00066, 0.00064, 0.00063, 0.00064, 0.00066, 0.00066, 0.00066, 0.00063, 0.0007, 0.00067, 0.00064, 0.00066, 0.00064, 0.00067, 0.00065, 0.00063, 0.00064, 0.00064, 0.00064, 0.00066, 0.00063, 0.00069, 0.00063, 0.00065, 0.00063, 0.00064, 0.00065, 0.00064, 0.00067, 0.00064, 0.00069, 0.00071, 0.00067, 0.00066, 0.00065, 0.00066, 0.00065, 0.00065, 0.00065, 0.00066, 0.00067, 0.00063, 0.00064, 0.00065, 0.00065, 0.00065, 0.00063, 0.00067, 0.00064, 0.00071, 0.00064, 0.00065, 0.00067, 0.00067, 0.00066, 0.00066, 0.00065, 0.00067, 0.00068, 0.00066, 0.00065, 0.00065, 0.00064, 0.00065, 0.00065, 0.00065, 0.00065, 0.0007, 0.00066, 0.00066, 0.00064, 0.00064, 0.00063, 0.00067, 0.00067, 0.00065, 0.00064, 0.00064, 0.00064, 0.00065, 0.00064]}, "optimizer-count-zeros-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00739, 0.00364, 0.00226, 0.00226, 0.00225, 0.00225, 0.00225, 0.0024, 0.00234, 0.00234, 0.00233, 0.00234, 0.00231, 0.0023, 0.00231, 0.00234, 0.00233, 0.00237, 0.00239, 0.00246, 0.00232, 0.00237, 0.00239, 0.00235, 0.00232, 0.00235, 0.00232, 0.00238, 0.00232, 0.00237, 0.00233, 0.00234, 0.00233, 0.00239, 0.00233, 0.00235, 0.00239, 0.00238, 0.00239, 0.00239, 0.00233, 0.00235, 0.00235, 0.00234, 0.00237, 0.0024, 0.00232, 0.00236, 0.00236, 0.00237, 0.00234, 0.00231, 0.00232, 0.00231, 0.00238, 0.00236, 0.00238, 0.00234, 0.00236, 0.00234, 0.00232, 0.00232, 0.00235, 0.0024, 0.00231, 0.00231, 0.00237, 0.00233, 0.00233, 0.00233, 0.00232, 0.00233, 0.00238, 0.00243, 0.00242, 0.00232, 0.00237, 0.00232, 0.00231, 0.00237, 0.00234, 0.00233, 0.00248, 0.00235, 0.0025, 0.00238, 0.00234, 0.00234, 0.00236, 0.00235, 0.00232, 0.00247, 0.00246, 0.00233, 0.00234, 0.00239, 0.00246, 0.00239, 0.0026, 0.00244, 0.00235, 0.00241, 0.00241, 0.00238, 0.00238, 0.00241, 0.00236, 0.00236, 0.00236, 0.00235, 0.00233, 0.00234, 0.00235, 0.00239, 0.00234, 0.00232, 0.00237, 0.00233, 0.00239, 0.0024, 0.00236, 0.00237, 0.00236, 0.00233, 0.00236, 0.00236, 0.00244, 0.00234, 0.00235, 0.00236, 0.00237, 0.0024, 0.00233, 0.00236, 0.00234, 0.00233, 0.00238, 0.00232, 0.00233, 0.00238, 0.00231, 0.00238, 0.00233, 0.00233, 0.00232, 0.00234, 0.00236, 0.00233, 0.00235, 0.00233, 0.00234, 0.00236, 0.00235, 0.00232, 0.00234, 0.00235, 0.00233, 0.00234, 0.00235, 0.00248, 0.00234, 0.00237, 0.00237, 0.00237, 0.00233, 0.00239, 0.00236, 0.00233, 0.00237, 0.00234, 0.00245, 0.00234, 0.00232, 0.00244, 0.00234, 0.00254, 0.00233, 0.00233, 0.00235, 0.00234, 0.00233, 0.00235, 0.00236, 0.00234, 0.00234, 0.00239, 0.00238, 0.00237, 0.00234, 0.00241, 0.00234, 0.00238, 0.00233, 0.00236, 0.00238, 0.00235, 0.00238, 0.00234, 0.00233, 0.00235, 0.00242, 0.00239, 0.00232, 0.00243, 0.00238, 0.00234, 0.00234, 0.00246, 0.00239, 0.00235, 0.00234, 0.00243, 0.00233, 0.00234, 0.00235, 0.00234, 0.00236, 0.00234, 0.00238, 0.00239, 0.00241, 0.00234, 0.00236, 0.00236, 0.00233, 0.00232, 0.00236, 0.00242, 0.00234, 0.00238, 0.0024, 0.00244, 0.00235, 0.00235, 0.00239, 0.0024, 0.00245, 0.00233, 0.00233, 0.00288, 0.0025, 0.00237, 0.00237, 0.00233, 0.00234, 0.00238, 0.00237, 0.00238, 0.00237, 0.00235, 0.00238, 0.00238, 0.00236, 0.00238, 0.00237, 0.00235, 0.00235, 0.00239, 0.00237, 0.00236, 0.00234, 0.00235, 0.00245, 0.00237, 0.00238, 0.00235, 0.00235, 0.00248, 0.00234, 0.00236, 0.0024, 0.00232, 0.00256, 0.00232, 0.00233, 0.00239, 0.0024, 0.00246, 0.00243, 0.00233, 0.00238, 0.00238, 0.00234, 0.00244, 0.00238, 0.00234, 0.00234, 0.00246, 0.00238, 0.00234, 0.00239, 0.00234, 0.00239, 0.00238, 0.00236, 0.00234, 0.00236, 0.00248, 0.00239, 0.00236, 0.00241, 0.00236, 0.00235, 0.00245, 0.00239, 0.00237, 0.00258, 0.00238, 0.00238, 0.0024, 0.00237, 0.00241, 0.00235, 0.00237, 0.00239, 0.00237, 0.00245, 0.00239, 0.00234, 0.00233, 0.00239, 0.00235, 0.00235, 0.00245, 0.00235, 0.00235, 0.00243, 0.00258, 0.00239, 0.0024, 0.00241, 0.00246, 0.00255, 0.00232, 0.00234, 0.00233, 0.00234, 0.00257, 0.00234, 0.00238, 0.0024, 0.00234, 0.00236, 0.00234, 0.00238, 0.00236, 0.00235, 0.00236, 0.00238, 0.00258, 0.00237, 0.00245, 0.00235, 0.00238, 0.0024, 0.0024, 0.00242, 0.0024, 0.00239, 0.00252, 0.00236, 0.00236, 0.00233, 0.00239, 0.00238, 0.00234, 0.00241, 0.00237, 0.00257, 0.00233, 0.00237, 0.00239, 0.00238, 0.00236, 0.00235, 0.00232, 0.00234, 0.00237, 0.00233, 0.00232, 0.00236, 0.00234, 0.00238, 0.00252, 0.00239, 0.00237, 0.00257, 0.00236, 0.00238, 0.00238, 0.00237, 0.00234, 0.00236, 0.00237, 0.00236, 0.0023, 0.00234, 0.00238, 0.00241, 0.00236]}, "optimizer-inner-step-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00678, 0.00052, 0.00045, 0.00046, 0.00048, 0.00047, 0.00048, 0.00039, 0.00046, 0.00046, 0.00045, 0.00037, 0.00038, 0.00041, 0.00046, 0.00047, 0.00038, 0.00039, 0.00034, 0.00031, 0.00032, 0.0003, 0.00033, 0.00036, 0.00032, 0.00032, 0.00037, 0.00036, 0.00036, 0.00036, 0.0003, 0.00032, 0.00038, 0.0003, 0.00032, 0.00032, 0.00031, 0.00033, 0.00032, 0.00032, 0.00036, 0.00032, 0.00032, 0.00031, 0.00031, 0.00031, 0.00031, 0.00034, 0.00035, 0.0003, 0.00033, 0.00033, 0.00029, 0.00038, 0.0003, 0.0003, 0.00031, 0.0003, 0.0003, 0.0003, 0.00031, 0.0003, 0.00033, 0.00031, 0.00032, 0.00032, 0.00037, 0.0003, 0.00031, 0.00034, 0.0003, 0.00033, 0.00032, 0.00032, 0.00031, 0.00038, 0.0003, 0.0003, 0.0003, 0.00031, 0.0003, 0.0003, 0.00031, 0.0003, 0.0003, 0.0003, 0.0003, 0.00032, 0.00032, 0.0003, 0.0003, 0.0003, 0.00032, 0.00032, 0.00036, 0.00038, 0.00032, 0.0003, 0.00032, 0.0003, 0.0003, 0.0003, 0.00034, 0.00031, 0.0003, 0.0003, 0.00032, 0.00031, 0.0003, 0.0003, 0.0003, 0.0003, 0.00032, 0.0003, 0.0003, 0.00033, 0.0003, 0.0003, 0.00031, 0.0003, 0.00029, 0.00032, 0.0003, 0.00031, 0.00031, 0.0003, 0.0003, 0.0003, 0.0003, 0.00031, 0.00031, 0.0003, 0.0003, 0.00032, 0.00037, 0.00031, 0.0003, 0.0003, 0.0003, 0.0003, 0.0003, 0.0003, 0.00033, 0.00035, 0.0003, 0.00037, 0.00035, 0.00036, 0.00038, 0.0003, 0.00032, 0.00031, 0.00031, 0.00033, 0.0003, 0.0003, 0.00034, 0.0003, 0.0003, 0.00031, 0.00037, 0.0003, 0.00036, 0.0003, 0.0003, 0.00031, 0.00032, 0.00031, 0.00032, 0.0003, 0.00033, 0.00031, 0.0003, 0.0003, 0.00031, 0.0003, 0.00031, 0.0003, 0.00031, 0.00035, 0.0003, 0.0003, 0.0003, 0.0003, 0.00031, 0.00031, 0.00031, 0.0003, 0.0003, 0.00036, 0.00029, 0.0003, 0.0003, 0.00031, 0.0003, 0.0003, 0.0003, 0.00031, 0.0003, 0.00029, 0.00037, 0.00044, 0.00044, 0.00032, 0.00031, 0.00039, 0.0003, 0.0003, 0.00041, 0.00037, 0.00032, 0.00032, 0.00032, 0.00032, 0.0003, 0.00031, 0.00033, 0.00032, 0.00038, 0.00033, 0.00037, 0.00033, 0.0003, 0.00031, 0.0003, 0.00038, 0.00031, 0.00039, 0.00032, 0.0003, 0.00032, 0.0003, 0.0003, 0.00038, 0.0003, 0.00034, 0.0003, 0.00038, 0.0003, 0.0012, 0.00034, 0.00031, 0.00033, 0.00031, 0.0003, 0.00037, 0.0003, 0.00037, 0.00032, 0.00032, 0.0003, 0.00032, 0.00029, 0.00037, 0.0003, 0.0003, 0.00029, 0.00031, 0.0003, 0.0003, 0.0003, 0.0003, 0.0003, 0.00031, 0.0003, 0.00031, 0.00031, 0.00031, 0.00031, 0.00035, 0.00031, 0.0003, 0.00032, 0.0003, 0.0003, 0.0003, 0.00031, 0.00032, 0.0003, 0.00032, 0.00031, 0.0003, 0.00031, 0.00037, 0.0003, 0.00034, 0.00029, 0.0003, 0.00032, 0.0003, 0.00031, 0.00032, 0.00031, 0.00031, 0.00036, 0.00031, 0.00032, 0.00031, 0.0003, 0.0003, 0.00032, 0.00033, 0.00032, 0.00031, 0.00029, 0.0003, 0.00034, 0.00037, 0.0003, 0.00036, 0.00031, 0.00031, 0.00031, 0.00031, 0.00031, 0.0003, 0.00032, 0.0003, 0.0003, 0.00034, 0.00031, 0.0003, 0.00036, 0.0003, 0.0003, 0.0003, 0.0003, 0.00036, 0.00031, 0.0003, 0.00034, 0.0003, 0.00034, 0.0003, 0.0003, 0.00033, 0.00037, 0.00032, 0.0003, 0.0003, 0.00031, 0.00031, 0.0003, 0.00029, 0.00031, 0.0003, 0.00031, 0.0003, 0.00031, 0.00037, 0.00033, 0.00032, 0.0003, 0.00031, 0.00032, 0.00032, 0.0003, 0.00031, 0.0003, 0.00032, 0.0003, 0.0003, 0.00037, 0.00035, 0.00029, 0.0003, 0.00032, 0.00029, 0.00033, 0.00031, 0.00029, 0.0003, 0.0003, 0.0003, 0.00031, 0.0003, 0.0003, 0.0003, 0.00035, 0.0003, 0.00029, 0.0003, 0.0003, 0.0003, 0.0003, 0.00032, 0.00031, 0.0003, 0.0003, 0.00036, 0.0003, 0.00032, 0.00031, 0.0003, 0.0003, 0.0003, 0.00033, 0.00031, 0.00038, 0.0003]}, "optimizer-copy-main-to-model-params-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.00059, 0.00049, 0.00045, 0.00045, 0.00045, 0.00045, 0.00044, 0.00044, 0.00046, 0.00045, 0.00045, 0.00044, 0.00045, 0.00044, 0.00045, 0.00045, 0.00043, 0.00044, 0.00043, 0.00043, 0.00044, 0.00042, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00042, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00042, 0.00042, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00044, 0.00043, 0.00042, 0.00043, 0.00042, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00042, 0.00042, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00042, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00042, 0.00043, 0.00042, 0.00042, 0.00042, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00044, 0.00042, 0.00043, 0.00043, 0.00042, 0.00043, 0.00042, 0.00043, 0.00043, 0.00042, 0.00043, 0.00042, 0.00043, 0.00043, 0.00044, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00042, 0.00042, 0.00044, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00044, 0.00042, 0.00043, 0.00043, 0.00043, 0.00042, 0.00044, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00042, 0.00042, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00042, 0.00043, 0.00042, 0.00043, 0.00043, 0.00042, 0.00043, 0.00042, 0.00042, 0.00042, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00052, 0.00043, 0.00042, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00042, 0.00042, 0.00043, 0.00042, 0.00042, 0.00042, 0.00043, 0.00043, 0.00042, 0.00043, 0.00042, 0.00042, 0.00042, 0.00043, 0.00043, 0.00042, 0.00044, 0.00045, 0.00043, 0.00042, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00042, 0.00042, 0.00043, 0.00043, 0.00043, 0.00042, 0.00046, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00042, 0.00042, 0.00042, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00042, 0.00043, 0.00042, 0.00043, 0.00043, 0.00048, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00042, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00042, 0.00042, 0.00042, 0.00043, 0.00043, 0.00042, 0.00042, 0.00042, 0.00042, 0.00042, 0.00043, 0.00044, 0.00043, 0.00042, 0.00043, 0.00043, 0.00042, 0.00042, 0.00043, 0.00043, 0.00043, 0.00043, 0.00042, 0.00043, 0.00043, 0.00042]}, "optimizer-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [2.50593, 0.00645, 0.00427, 0.00434, 0.00434, 0.0043, 0.00432, 0.00444, 0.00439, 0.0044, 0.00434, 0.00427, 0.00425, 0.00426, 0.00442, 0.00448, 0.00419, 0.00429, 0.00423, 0.00425, 0.00416, 0.00412, 0.00418, 0.00417, 0.00409, 0.00421, 0.00414, 0.00419, 0.00431, 0.0042, 0.00408, 0.00411, 0.00415, 0.00413, 0.00412, 0.00416, 0.00416, 0.00416, 0.00417, 0.00419, 0.00414, 0.00413, 0.00409, 0.00409, 0.00412, 0.00415, 0.00407, 0.00415, 0.00416, 0.00412, 0.0041, 0.00406, 0.00403, 0.00412, 0.00413, 0.00417, 0.00417, 0.00409, 0.00412, 0.00417, 0.00406, 0.00407, 0.00415, 0.00419, 0.00405, 0.00409, 0.00421, 0.00406, 0.00407, 0.0041, 0.00406, 0.0041, 0.00412, 0.0042, 0.00419, 0.00414, 0.00414, 0.0041, 0.00406, 0.00412, 0.00407, 0.00406, 0.00424, 0.00407, 0.00423, 0.00412, 0.00409, 0.0041, 0.00411, 0.0041, 0.00408, 0.00421, 0.00422, 0.00409, 0.00409, 0.00422, 0.00421, 0.00413, 0.00446, 0.00417, 0.00409, 0.0042, 0.00418, 0.00418, 0.00412, 0.00414, 0.00413, 0.0041, 0.0041, 0.00407, 0.00401, 0.00404, 0.00412, 0.00408, 0.00408, 0.00413, 0.00411, 0.00407, 0.00407, 0.00414, 0.00409, 0.00414, 0.0041, 0.00407, 0.00408, 0.0041, 0.00416, 0.00409, 0.00407, 0.0041, 0.00413, 0.00414, 0.00407, 0.00412, 0.00416, 0.00407, 0.00414, 0.00406, 0.00407, 0.00413, 0.00403, 0.00415, 0.00408, 0.00412, 0.00399, 0.00417, 0.0042, 0.00415, 0.0042, 0.00406, 0.00409, 0.0041, 0.00408, 0.00412, 0.0041, 0.00407, 0.00416, 0.00409, 0.0041, 0.00427, 0.00419, 0.0041, 0.00421, 0.00414, 0.00406, 0.00415, 0.00416, 0.00409, 0.00414, 0.00406, 0.00423, 0.00409, 0.00408, 0.00417, 0.00411, 0.00428, 0.00409, 0.00406, 0.00419, 0.00416, 0.0041, 0.00408, 0.00412, 0.00408, 0.00412, 0.0042, 0.0041, 0.0041, 0.00414, 0.00422, 0.00407, 0.00411, 0.00406, 0.00412, 0.00418, 0.00407, 0.0041, 0.00406, 0.00405, 0.00412, 0.00426, 0.00434, 0.00425, 0.00418, 0.00419, 0.00422, 0.00407, 0.0042, 0.00431, 0.00415, 0.00418, 0.00418, 0.00411, 0.00411, 0.00409, 0.00408, 0.00414, 0.00411, 0.00421, 0.00417, 0.00427, 0.0041, 0.00413, 0.00415, 0.00408, 0.00414, 0.0042, 0.00427, 0.00415, 0.00412, 0.00426, 0.00423, 0.00408, 0.00419, 0.00426, 0.00425, 0.00419, 0.00413, 0.00408, 0.00694, 0.00429, 0.00417, 0.00421, 0.00406, 0.00411, 0.0042, 0.00411, 0.00417, 0.00415, 0.00412, 0.0041, 0.00413, 0.00409, 0.00419, 0.0041, 0.00411, 0.00408, 0.00413, 0.00421, 0.0041, 0.00407, 0.00412, 0.00418, 0.0041, 0.00413, 0.00417, 0.0041, 0.00421, 0.00406, 0.0042, 0.00416, 0.00407, 0.00444, 0.00408, 0.00405, 0.00411, 0.00416, 0.00426, 0.00414, 0.00408, 0.00413, 0.00411, 0.00407, 0.00448, 0.00412, 0.00412, 0.00417, 0.00418, 0.00415, 0.00409, 0.00422, 0.00409, 0.00416, 0.00411, 0.00417, 0.00406, 0.00415, 0.00424, 0.00422, 0.00408, 0.00418, 0.00411, 0.00412, 0.00422, 0.00418, 0.00413, 0.00447, 0.00427, 0.00415, 0.00422, 0.00421, 0.00414, 0.00408, 0.00411, 0.00412, 0.00411, 0.00427, 0.00415, 0.00407, 0.00416, 0.00414, 0.00407, 0.00416, 0.0042, 0.00408, 0.00409, 0.00417, 0.00445, 0.00415, 0.00413, 0.00421, 0.00419, 0.00438, 0.00405, 0.00408, 0.00411, 0.00421, 0.00434, 0.0041, 0.00411, 0.00423, 0.00408, 0.00411, 0.00406, 0.00411, 0.00412, 0.0041, 0.00412, 0.00411, 0.00445, 0.00424, 0.00425, 0.00412, 0.00412, 0.00418, 0.00417, 0.00417, 0.00415, 0.00414, 0.0043, 0.00409, 0.00408, 0.00415, 0.00419, 0.0041, 0.00406, 0.0042, 0.00408, 0.00448, 0.00406, 0.0041, 0.00416, 0.00416, 0.00411, 0.00411, 0.00407, 0.00411, 0.00414, 0.00416, 0.00405, 0.0041, 0.0041, 0.00414, 0.00427, 0.00414, 0.00414, 0.0044, 0.00412, 0.00417, 0.00419, 0.0041, 0.00408, 0.00416, 0.00414, 0.0041, 0.00402, 0.00411, 0.00411, 0.00421, 0.00412]}, "learning-rate": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05]}, "learning-rate vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 1e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 2e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 3e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 4e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 5e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 6e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 7e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 8e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05, 9e-05]}, "batch-size": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0]}, "batch-size vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0, 128.0]}, "lm loss": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [10.89913, 10.90768, 10.89258, 10.83558, 10.68347, 10.65957, 10.44874, 10.16298, 9.95823, 9.85931, 9.60267, 9.85448, 9.88896, 9.63283, 9.79416, 9.51077, 9.46452, 9.65474, 9.39303, 9.33891, 9.24974, 9.15413, 9.1799, 9.00652, 9.19898, 9.06462, 9.16252, 9.16628, 9.30046, 8.98957, 8.93846, 9.05768, 9.05239, 8.66384, 8.72654, 8.76695, 8.70049, 8.7485, 8.67207, 8.78319, 8.67816, 8.86784, 8.84942, 8.51529, 8.40635, 8.45078, 8.50987, 8.40639, 8.45206, 8.60248, 8.38482, 8.21373, 8.24279, 8.2386, 8.28505, 7.93108, 8.10687, 7.90564, 8.25924, 8.23983, 8.01396, 7.97887, 7.93189, 7.74875, 7.74952, 7.65295, 7.52397, 7.91334, 7.70468, 7.4615, 7.7454, 7.77328, 7.54365, 7.30492, 7.45798, 7.34465, 7.46796, 7.22991, 7.64058, 7.27994, 7.34996, 7.21151, 7.21093, 7.42121, 7.17404, 7.28056, 6.99816, 7.00187, 7.03663, 7.13195, 6.82349, 6.98827, 7.0878, 6.99784, 6.87313, 6.75507, 6.98467, 7.05698, 6.69967, 6.57871, 6.71928, 6.73563, 6.72919, 6.73392, 6.64984, 6.40377, 6.63158, 6.61637, 6.44045, 6.62208, 6.73713, 6.60229, 6.7201, 6.6855, 6.61682, 6.50401, 6.59317, 6.39881, 6.65822, 6.24152, 6.2452, 6.29731, 6.3828, 6.34021, 6.44085, 6.28383, 6.329, 6.22922, 6.19228, 6.38636, 6.31695, 6.31001, 6.15226, 6.14734, 6.22668, 6.37438, 6.18797, 6.13621, 6.16902, 6.10406, 6.04744, 6.06108, 6.24255, 6.39422, 6.2458, 6.284, 6.08157, 6.16415, 5.99061, 6.02156, 5.94437, 6.2389, 6.17376, 5.95486, 5.77921, 6.11867, 5.84238, 6.09465, 5.78691, 6.15643, 6.14146, 6.08403, 5.92734, 6.11211, 5.9414, 6.1909, 5.88926, 5.79076, 5.77594, 5.68012, 6.00691, 5.98869, 6.0616, 5.88167, 6.03501, 5.96091, 5.98667, 5.98233, 5.94294, 5.83159, 5.94469, 5.61383, 5.69739, 5.88208, 5.83783, 5.85647, 5.75359, 5.8293, 5.71663, 5.54972, 5.71476, 5.61805, 5.82148, 5.59645, 5.7046, 5.70388, 5.89118, 5.63818, 5.84407, 5.73403, 5.86464, 5.32399, 5.89231, 5.86685, 5.84835, 5.41039, 5.39989, 5.62175, 5.59208, 5.47993, 5.57198, 5.6706, 5.47017, 5.74137, 5.50537, 5.58997, 5.61705, 5.61569, 5.50878, 5.61368, 5.67021, 5.6796, 5.58462, 5.65767, 5.36943, 5.67868, 5.62273, 5.41823, 5.57655, 5.62803, 5.55076, 5.34162, 5.53284, 5.48499, 5.48067, 5.37314, 5.5522, 5.60377, 5.3855, 5.51883, 5.48805, 5.33305, 5.50438, 5.40837, 5.44646, 5.31737, 5.06747, 5.48486, 5.5727, 5.71602, 5.41542, 5.6005, 5.63654, 5.23257, 5.2731, 5.39321, 5.39531, 5.33164, 5.49936, 5.18243, 5.29899, 5.24416, 5.37687, 5.25765, 5.44188, 5.54176, 5.31448, 5.43676, 5.33643, 5.07327, 5.31163, 5.25792, 5.30629, 5.11098, 5.27254, 5.26504, 5.47787, 5.16706, 5.26752, 5.21469, 5.35574, 4.99013, 4.91368, 5.33262, 5.39207, 5.2358, 5.31677, 5.10593, 5.16606, 5.26629, 5.0692, 5.2713, 5.07218, 5.34842, 5.2468, 5.14931, 5.24288, 5.04098, 5.31807, 5.05081, 5.02892, 5.14027, 5.11638, 5.26992, 5.14976, 5.27441, 5.08839, 5.0939, 5.24735, 5.32718, 5.25749, 5.19305, 5.14479, 5.29137, 4.95079, 5.20634, 5.09379, 5.30222, 5.17249, 5.19061, 5.1184, 4.98363, 4.98895, 5.22344, 5.3082, 5.0995, 5.05248, 4.918, 5.12558, 5.12077, 4.93023, 5.33931, 5.02066, 5.1036, 5.16752, 5.0013, 5.06232, 5.06982, 4.99551, 5.07864, 5.16478, 4.98139, 5.18171, 4.93094, 4.92837, 5.06899, 5.00137, 4.9149, 4.77784, 4.94461, 5.11809, 5.01598, 5.02127, 5.33033, 4.95783, 4.9952, 5.05204, 4.80991, 4.7377, 4.99918, 5.04469, 4.87951, 4.95537, 5.04608, 5.02474, 4.82217, 4.89846, 4.90951, 4.83736, 4.75068, 5.01543, 4.75048, 5.21264, 4.79165, 5.00346, 4.74267, 4.79351, 4.82094, 4.65323, 4.66147, 4.84627, 4.81058, 4.81182, 4.92434, 4.88712, 4.93733, 4.7758, 4.88555, 4.74111, 4.923, 4.96049, 4.87815, 4.71239, 4.79301, 4.90162, 4.71655, 4.8736, 4.69974, 4.70298, 4.65388]}, "lm loss vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [10.89913, 10.90768, 10.89258, 10.83558, 10.68347, 10.65957, 10.44874, 10.16298, 9.95823, 9.85931, 9.60267, 9.85448, 9.88896, 9.63283, 9.79416, 9.51077, 9.46452, 9.65474, 9.39303, 9.33891, 9.24974, 9.15413, 9.1799, 9.00652, 9.19898, 9.06462, 9.16252, 9.16628, 9.30046, 8.98957, 8.93846, 9.05768, 9.05239, 8.66384, 8.72654, 8.76695, 8.70049, 8.7485, 8.67207, 8.78319, 8.67816, 8.86784, 8.84942, 8.51529, 8.40635, 8.45078, 8.50987, 8.40639, 8.45206, 8.60248, 8.38482, 8.21373, 8.24279, 8.2386, 8.28505, 7.93108, 8.10687, 7.90564, 8.25924, 8.23983, 8.01396, 7.97887, 7.93189, 7.74875, 7.74952, 7.65295, 7.52397, 7.91334, 7.70468, 7.4615, 7.7454, 7.77328, 7.54365, 7.30492, 7.45798, 7.34465, 7.46796, 7.22991, 7.64058, 7.27994, 7.34996, 7.21151, 7.21093, 7.42121, 7.17404, 7.28056, 6.99816, 7.00187, 7.03663, 7.13195, 6.82349, 6.98827, 7.0878, 6.99784, 6.87313, 6.75507, 6.98467, 7.05698, 6.69967, 6.57871, 6.71928, 6.73563, 6.72919, 6.73392, 6.64984, 6.40377, 6.63158, 6.61637, 6.44045, 6.62208, 6.73713, 6.60229, 6.7201, 6.6855, 6.61682, 6.50401, 6.59317, 6.39881, 6.65822, 6.24152, 6.2452, 6.29731, 6.3828, 6.34021, 6.44085, 6.28383, 6.329, 6.22922, 6.19228, 6.38636, 6.31695, 6.31001, 6.15226, 6.14734, 6.22668, 6.37438, 6.18797, 6.13621, 6.16902, 6.10406, 6.04744, 6.06108, 6.24255, 6.39422, 6.2458, 6.284, 6.08157, 6.16415, 5.99061, 6.02156, 5.94437, 6.2389, 6.17376, 5.95486, 5.77921, 6.11867, 5.84238, 6.09465, 5.78691, 6.15643, 6.14146, 6.08403, 5.92734, 6.11211, 5.9414, 6.1909, 5.88926, 5.79076, 5.77594, 5.68012, 6.00691, 5.98869, 6.0616, 5.88167, 6.03501, 5.96091, 5.98667, 5.98233, 5.94294, 5.83159, 5.94469, 5.61383, 5.69739, 5.88208, 5.83783, 5.85647, 5.75359, 5.8293, 5.71663, 5.54972, 5.71476, 5.61805, 5.82148, 5.59645, 5.7046, 5.70388, 5.89118, 5.63818, 5.84407, 5.73403, 5.86464, 5.32399, 5.89231, 5.86685, 5.84835, 5.41039, 5.39989, 5.62175, 5.59208, 5.47993, 5.57198, 5.6706, 5.47017, 5.74137, 5.50537, 5.58997, 5.61705, 5.61569, 5.50878, 5.61368, 5.67021, 5.6796, 5.58462, 5.65767, 5.36943, 5.67868, 5.62273, 5.41823, 5.57655, 5.62803, 5.55076, 5.34162, 5.53284, 5.48499, 5.48067, 5.37314, 5.5522, 5.60377, 5.3855, 5.51883, 5.48805, 5.33305, 5.50438, 5.40837, 5.44646, 5.31737, 5.06747, 5.48486, 5.5727, 5.71602, 5.41542, 5.6005, 5.63654, 5.23257, 5.2731, 5.39321, 5.39531, 5.33164, 5.49936, 5.18243, 5.29899, 5.24416, 5.37687, 5.25765, 5.44188, 5.54176, 5.31448, 5.43676, 5.33643, 5.07327, 5.31163, 5.25792, 5.30629, 5.11098, 5.27254, 5.26504, 5.47787, 5.16706, 5.26752, 5.21469, 5.35574, 4.99013, 4.91368, 5.33262, 5.39207, 5.2358, 5.31677, 5.10593, 5.16606, 5.26629, 5.0692, 5.2713, 5.07218, 5.34842, 5.2468, 5.14931, 5.24288, 5.04098, 5.31807, 5.05081, 5.02892, 5.14027, 5.11638, 5.26992, 5.14976, 5.27441, 5.08839, 5.0939, 5.24735, 5.32718, 5.25749, 5.19305, 5.14479, 5.29137, 4.95079, 5.20634, 5.09379, 5.30222, 5.17249, 5.19061, 5.1184, 4.98363, 4.98895, 5.22344, 5.3082, 5.0995, 5.05248, 4.918, 5.12558, 5.12077, 4.93023, 5.33931, 5.02066, 5.1036, 5.16752, 5.0013, 5.06232, 5.06982, 4.99551, 5.07864, 5.16478, 4.98139, 5.18171, 4.93094, 4.92837, 5.06899, 5.00137, 4.9149, 4.77784, 4.94461, 5.11809, 5.01598, 5.02127, 5.33033, 4.95783, 4.9952, 5.05204, 4.80991, 4.7377, 4.99918, 5.04469, 4.87951, 4.95537, 5.04608, 5.02474, 4.82217, 4.89846, 4.90951, 4.83736, 4.75068, 5.01543, 4.75048, 5.21264, 4.79165, 5.00346, 4.74267, 4.79351, 4.82094, 4.65323, 4.66147, 4.84627, 4.81058, 4.81182, 4.92434, 4.88712, 4.93733, 4.7758, 4.88555, 4.74111, 4.923, 4.96049, 4.87815, 4.71239, 4.79301, 4.90162, 4.71655, 4.8736, 4.69974, 4.70298, 4.65388]}, "loss-scale": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]}, "loss-scale vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]}, "grad-norm": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [13.85078, 13.18214, 13.66323, 12.70284, 12.09224, 9.52286, 6.94629, 7.0906, 6.10744, 4.68805, 4.27923, 2.88041, 2.44505, 2.38119, 2.05617, 2.21829, 2.16794, 1.88908, 2.22196, 2.07722, 2.13294, 2.16643, 2.0255, 2.23892, 2.00255, 2.1468, 1.909, 1.8914, 1.93899, 2.06927, 2.17429, 2.25885, 1.90288, 2.34707, 2.36934, 2.15239, 2.14878, 1.8334, 2.04013, 1.74856, 2.34179, 1.94848, 1.82059, 1.87135, 1.95474, 1.80759, 1.72382, 1.76832, 1.75386, 1.54852, 1.75847, 1.74505, 1.74315, 1.934, 1.66976, 1.9002, 1.75945, 1.83439, 1.52145, 1.48453, 1.63689, 1.50053, 1.80874, 1.84804, 1.61011, 1.60696, 1.63765, 1.60516, 1.41707, 1.61014, 1.35755, 1.37838, 1.75329, 1.40606, 1.36529, 1.42107, 1.35362, 1.41859, 1.30889, 1.28207, 1.37053, 1.22728, 1.40288, 1.1887, 1.18077, 1.33758, 1.55936, 1.2681, 1.19394, 1.06216, 1.15629, 1.24879, 1.03956, 1.0728, 0.9879, 1.25738, 0.99242, 1.34839, 1.08186, 1.49339, 1.31629, 1.35559, 1.2587, 1.34653, 1.04512, 1.10012, 1.07721, 1.16603, 1.07931, 0.88403, 0.84804, 0.94924, 1.03703, 0.90657, 1.20063, 1.09118, 1.06536, 1.39946, 0.8902, 1.01025, 1.05199, 1.12692, 1.02282, 1.04798, 0.99926, 1.14919, 1.12248, 1.1294, 1.23794, 1.14553, 1.27834, 1.25691, 1.10116, 1.03642, 1.22267, 1.29353, 0.91452, 1.30692, 1.02293, 1.14184, 1.09354, 1.18831, 1.29696, 1.0865, 0.89821, 1.46743, 1.18241, 1.38811, 1.25228, 1.68626, 1.50945, 1.7486, 1.2923, 1.51275, 1.79877, 1.64168, 1.14298, 1.38519, 1.89605, 1.27538, 1.55708, 1.30069, 1.23935, 1.2033, 1.29827, 1.39671, 1.50108, 1.37699, 1.52549, 1.26383, 1.08138, 1.02929, 1.51851, 1.73981, 1.47699, 1.30343, 1.45672, 1.1571, 1.24108, 1.19017, 1.29612, 1.28332, 1.44554, 1.49398, 1.43029, 1.21083, 1.34161, 1.47224, 1.18337, 1.47947, 1.49535, 1.63101, 1.50036, 1.71739, 1.57237, 1.71104, 1.86198, 1.56646, 1.53736, 1.65331, 1.13651, 1.40126, 1.26581, 1.10028, 1.30712, 1.66779, 1.20489, 1.68026, 1.34067, 1.67876, 1.47506, 1.93206, 1.53418, 1.5662, 1.60998, 1.34624, 1.25258, 1.61379, 1.30832, 1.24696, 1.55499, 1.22777, 1.57723, 1.49173, 1.3016, 1.57934, 1.39858, 1.57422, 1.34451, 1.29559, 1.33579, 2.0102, 1.44742, 1.72844, 1.51969, 1.20546, 1.53729, 1.33621, 1.1701, 1.46057, 1.78343, 1.34591, 1.6587, 1.59379, 1.44379, 1.69606, 1.62714, 1.72274, 1.60404, 1.43431, 1.37981, 1.28771, 1.48844, 1.09986, 1.24011, 1.77308, 1.37109, 1.44084, 1.62755, 1.28204, 1.25748, 1.25812, 1.60866, 1.49243, 1.23832, 1.90719, 1.96886, 1.6413, 1.40509, 1.32485, 1.31804, 1.49446, 1.30898, 1.52892, 1.21795, 1.47551, 1.41365, 1.55899, 1.46352, 1.36026, 1.34636, 1.42092, 1.22943, 1.51525, 1.19331, 1.59104, 1.14424, 1.31382, 1.31199, 1.42941, 1.47566, 1.79962, 1.42412, 1.64474, 1.53875, 1.35465, 1.50623, 1.41632, 1.36482, 1.25797, 1.36103, 1.33178, 1.38348, 1.47978, 1.39511, 1.29437, 1.4757, 1.19421, 1.18546, 1.42844, 1.50609, 1.35696, 1.58833, 1.53065, 1.63698, 1.17447, 1.57793, 1.45478, 1.13184, 1.3261, 1.84689, 1.52489, 1.22527, 1.53044, 1.29203, 1.46694, 1.36199, 1.51584, 1.40091, 1.51617, 1.33582, 1.69525, 1.16884, 1.82555, 1.35697, 1.35667, 1.38749, 1.31708, 1.56013, 1.5132, 1.32821, 1.20186, 1.37821, 1.32133, 1.39205, 1.39727, 1.49988, 1.87947, 1.25359, 1.24718, 1.54782, 1.28909, 1.75041, 1.46697, 1.32256, 1.37807, 1.36994, 1.28797, 1.46521, 1.30013, 1.51012, 1.36092, 1.38127, 1.39802, 1.28909, 1.34502, 1.47884, 1.76573, 1.3497, 1.73593, 1.33648, 1.41529, 1.83787, 1.62399, 1.4996, 1.37458, 1.49071, 1.25683, 1.19485, 1.34065, 1.25479, 1.3334, 1.50067, 1.24673, 1.17753, 1.37781, 1.42086, 1.42823, 1.19943, 1.37703, 1.25162, 1.32745, 1.4936, 1.40017, 1.39067, 1.43856, 1.40189, 1.30942, 1.16753, 1.27377]}, "grad-norm vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [13.85078, 13.18214, 13.66323, 12.70284, 12.09224, 9.52286, 6.94629, 7.0906, 6.10744, 4.68805, 4.27923, 2.88041, 2.44505, 2.38119, 2.05617, 2.21829, 2.16794, 1.88908, 2.22196, 2.07722, 2.13294, 2.16643, 2.0255, 2.23892, 2.00255, 2.1468, 1.909, 1.8914, 1.93899, 2.06927, 2.17429, 2.25885, 1.90288, 2.34707, 2.36934, 2.15239, 2.14878, 1.8334, 2.04013, 1.74856, 2.34179, 1.94848, 1.82059, 1.87135, 1.95474, 1.80759, 1.72382, 1.76832, 1.75386, 1.54852, 1.75847, 1.74505, 1.74315, 1.934, 1.66976, 1.9002, 1.75945, 1.83439, 1.52145, 1.48453, 1.63689, 1.50053, 1.80874, 1.84804, 1.61011, 1.60696, 1.63765, 1.60516, 1.41707, 1.61014, 1.35755, 1.37838, 1.75329, 1.40606, 1.36529, 1.42107, 1.35362, 1.41859, 1.30889, 1.28207, 1.37053, 1.22728, 1.40288, 1.1887, 1.18077, 1.33758, 1.55936, 1.2681, 1.19394, 1.06216, 1.15629, 1.24879, 1.03956, 1.0728, 0.9879, 1.25738, 0.99242, 1.34839, 1.08186, 1.49339, 1.31629, 1.35559, 1.2587, 1.34653, 1.04512, 1.10012, 1.07721, 1.16603, 1.07931, 0.88403, 0.84804, 0.94924, 1.03703, 0.90657, 1.20063, 1.09118, 1.06536, 1.39946, 0.8902, 1.01025, 1.05199, 1.12692, 1.02282, 1.04798, 0.99926, 1.14919, 1.12248, 1.1294, 1.23794, 1.14553, 1.27834, 1.25691, 1.10116, 1.03642, 1.22267, 1.29353, 0.91452, 1.30692, 1.02293, 1.14184, 1.09354, 1.18831, 1.29696, 1.0865, 0.89821, 1.46743, 1.18241, 1.38811, 1.25228, 1.68626, 1.50945, 1.7486, 1.2923, 1.51275, 1.79877, 1.64168, 1.14298, 1.38519, 1.89605, 1.27538, 1.55708, 1.30069, 1.23935, 1.2033, 1.29827, 1.39671, 1.50108, 1.37699, 1.52549, 1.26383, 1.08138, 1.02929, 1.51851, 1.73981, 1.47699, 1.30343, 1.45672, 1.1571, 1.24108, 1.19017, 1.29612, 1.28332, 1.44554, 1.49398, 1.43029, 1.21083, 1.34161, 1.47224, 1.18337, 1.47947, 1.49535, 1.63101, 1.50036, 1.71739, 1.57237, 1.71104, 1.86198, 1.56646, 1.53736, 1.65331, 1.13651, 1.40126, 1.26581, 1.10028, 1.30712, 1.66779, 1.20489, 1.68026, 1.34067, 1.67876, 1.47506, 1.93206, 1.53418, 1.5662, 1.60998, 1.34624, 1.25258, 1.61379, 1.30832, 1.24696, 1.55499, 1.22777, 1.57723, 1.49173, 1.3016, 1.57934, 1.39858, 1.57422, 1.34451, 1.29559, 1.33579, 2.0102, 1.44742, 1.72844, 1.51969, 1.20546, 1.53729, 1.33621, 1.1701, 1.46057, 1.78343, 1.34591, 1.6587, 1.59379, 1.44379, 1.69606, 1.62714, 1.72274, 1.60404, 1.43431, 1.37981, 1.28771, 1.48844, 1.09986, 1.24011, 1.77308, 1.37109, 1.44084, 1.62755, 1.28204, 1.25748, 1.25812, 1.60866, 1.49243, 1.23832, 1.90719, 1.96886, 1.6413, 1.40509, 1.32485, 1.31804, 1.49446, 1.30898, 1.52892, 1.21795, 1.47551, 1.41365, 1.55899, 1.46352, 1.36026, 1.34636, 1.42092, 1.22943, 1.51525, 1.19331, 1.59104, 1.14424, 1.31382, 1.31199, 1.42941, 1.47566, 1.79962, 1.42412, 1.64474, 1.53875, 1.35465, 1.50623, 1.41632, 1.36482, 1.25797, 1.36103, 1.33178, 1.38348, 1.47978, 1.39511, 1.29437, 1.4757, 1.19421, 1.18546, 1.42844, 1.50609, 1.35696, 1.58833, 1.53065, 1.63698, 1.17447, 1.57793, 1.45478, 1.13184, 1.3261, 1.84689, 1.52489, 1.22527, 1.53044, 1.29203, 1.46694, 1.36199, 1.51584, 1.40091, 1.51617, 1.33582, 1.69525, 1.16884, 1.82555, 1.35697, 1.35667, 1.38749, 1.31708, 1.56013, 1.5132, 1.32821, 1.20186, 1.37821, 1.32133, 1.39205, 1.39727, 1.49988, 1.87947, 1.25359, 1.24718, 1.54782, 1.28909, 1.75041, 1.46697, 1.32256, 1.37807, 1.36994, 1.28797, 1.46521, 1.30013, 1.51012, 1.36092, 1.38127, 1.39802, 1.28909, 1.34502, 1.47884, 1.76573, 1.3497, 1.73593, 1.33648, 1.41529, 1.83787, 1.62399, 1.4996, 1.37458, 1.49071, 1.25683, 1.19485, 1.34065, 1.25479, 1.3334, 1.50067, 1.24673, 1.17753, 1.37781, 1.42086, 1.42823, 1.19943, 1.37703, 1.25162, 1.32745, 1.4936, 1.40017, 1.39067, 1.43856, 1.40189, 1.30942, 1.16753, 1.27377]}, "num-zeros": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [74.0, 63.0, 75.0, 78.0, 66.0, 90.0, 123.0, 103.0, 125.0, 133.0, 115.0, 161.0, 126.0, 146.0, 188.0, 178.0, 161.0, 181.0, 158.0, 160.0, 164.0, 167.0, 201.0, 161.0, 165.0, 159.0, 177.0, 141.0, 137.0, 180.0, 158.0, 140.0, 154.0, 154.0, 128.0, 132.0, 126.0, 203.0, 172.0, 163.0, 139.0, 144.0, 168.0, 169.0, 172.0, 167.0, 175.0, 195.0, 154.0, 215.0, 202.0, 199.0, 185.0, 162.0, 187.0, 189.0, 169.0, 140.0, 203.0, 208.0, 199.0, 194.0, 180.0, 184.0, 178.0, 211.0, 195.0, 201.0, 211.0, 180.0, 206.0, 227.0, 163.0, 239.0, 206.0, 210.0, 244.0, 196.0, 247.0, 207.0, 223.0, 213.0, 203.0, 229.0, 216.0, 202.0, 160.0, 210.0, 186.0, 218.0, 186.0, 201.0, 220.0, 207.0, 212.0, 180.0, 201.0, 187.0, 177.0, 160.0, 153.0, 145.0, 159.0, 150.0, 138.0, 154.0, 133.0, 163.0, 130.0, 189.0, 177.0, 148.0, 170.0, 144.0, 134.0, 126.0, 158.0, 112.0, 178.0, 157.0, 137.0, 123.0, 147.0, 119.0, 152.0, 157.0, 131.0, 137.0, 146.0, 141.0, 142.0, 111.0, 116.0, 112.0, 113.0, 126.0, 175.0, 112.0, 111.0, 132.0, 117.0, 107.0, 131.0, 130.0, 146.0, 123.0, 110.0, 111.0, 111.0, 98.0, 111.0, 97.0, 115.0, 88.0, 83.0, 81.0, 98.0, 103.0, 94.0, 107.0, 113.0, 103.0, 103.0, 132.0, 104.0, 89.0, 86.0, 105.0, 124.0, 136.0, 110.0, 139.0, 91.0, 85.0, 114.0, 105.0, 119.0, 138.0, 109.0, 121.0, 111.0, 112.0, 102.0, 120.0, 104.0, 116.0, 109.0, 101.0, 100.0, 108.0, 114.0, 103.0, 107.0, 94.0, 95.0, 97.0, 65.0, 102.0, 102.0, 88.0, 135.0, 111.0, 103.0, 104.0, 92.0, 100.0, 157.0, 66.0, 111.0, 106.0, 113.0, 110.0, 106.0, 103.0, 96.0, 98.0, 116.0, 107.0, 108.0, 102.0, 87.0, 115.0, 106.0, 92.0, 105.0, 113.0, 108.0, 116.0, 107.0, 102.0, 88.0, 71.0, 97.0, 90.0, 107.0, 99.0, 86.0, 104.0, 116.0, 100.0, 104.0, 99.0, 97.0, 88.0, 105.0, 86.0, 93.0, 106.0, 117.0, 96.0, 92.0, 118.0, 113.0, 139.0, 121.0, 72.0, 111.0, 102.0, 112.0, 113.0, 114.0, 117.0, 98.0, 111.0, 135.0, 82.0, 84.0, 79.0, 101.0, 109.0, 103.0, 119.0, 99.0, 86.0, 122.0, 101.0, 99.0, 100.0, 120.0, 120.0, 106.0, 95.0, 125.0, 106.0, 109.0, 70.0, 117.0, 115.0, 103.0, 92.0, 117.0, 78.0, 112.0, 103.0, 130.0, 117.0, 104.0, 112.0, 123.0, 116.0, 126.0, 104.0, 121.0, 133.0, 100.0, 115.0, 110.0, 116.0, 125.0, 93.0, 119.0, 120.0, 110.0, 89.0, 88.0, 113.0, 112.0, 97.0, 110.0, 112.0, 94.0, 105.0, 109.0, 116.0, 110.0, 117.0, 117.0, 82.0, 108.0, 87.0, 119.0, 93.0, 114.0, 93.0, 127.0, 105.0, 96.0, 110.0, 113.0, 87.0, 128.0, 105.0, 96.0, 107.0, 100.0, 106.0, 108.0, 89.0, 109.0, 108.0, 109.0, 112.0, 112.0, 110.0, 116.0, 103.0, 116.0, 110.0, 103.0, 118.0, 114.0, 130.0, 111.0, 119.0, 107.0, 130.0, 112.0, 107.0, 101.0, 99.0, 113.0, 107.0, 103.0, 107.0, 112.0, 97.0, 98.0, 118.0, 119.0, 121.0, 121.0, 122.0, 113.0, 130.0, 112.0, 113.0, 116.0, 108.0, 135.0, 118.0, 126.0, 132.0, 97.0, 101.0, 100.0, 125.0, 103.0, 122.0, 136.0, 126.0]}, "num-zeros vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [74.0, 63.0, 75.0, 78.0, 66.0, 90.0, 123.0, 103.0, 125.0, 133.0, 115.0, 161.0, 126.0, 146.0, 188.0, 178.0, 161.0, 181.0, 158.0, 160.0, 164.0, 167.0, 201.0, 161.0, 165.0, 159.0, 177.0, 141.0, 137.0, 180.0, 158.0, 140.0, 154.0, 154.0, 128.0, 132.0, 126.0, 203.0, 172.0, 163.0, 139.0, 144.0, 168.0, 169.0, 172.0, 167.0, 175.0, 195.0, 154.0, 215.0, 202.0, 199.0, 185.0, 162.0, 187.0, 189.0, 169.0, 140.0, 203.0, 208.0, 199.0, 194.0, 180.0, 184.0, 178.0, 211.0, 195.0, 201.0, 211.0, 180.0, 206.0, 227.0, 163.0, 239.0, 206.0, 210.0, 244.0, 196.0, 247.0, 207.0, 223.0, 213.0, 203.0, 229.0, 216.0, 202.0, 160.0, 210.0, 186.0, 218.0, 186.0, 201.0, 220.0, 207.0, 212.0, 180.0, 201.0, 187.0, 177.0, 160.0, 153.0, 145.0, 159.0, 150.0, 138.0, 154.0, 133.0, 163.0, 130.0, 189.0, 177.0, 148.0, 170.0, 144.0, 134.0, 126.0, 158.0, 112.0, 178.0, 157.0, 137.0, 123.0, 147.0, 119.0, 152.0, 157.0, 131.0, 137.0, 146.0, 141.0, 142.0, 111.0, 116.0, 112.0, 113.0, 126.0, 175.0, 112.0, 111.0, 132.0, 117.0, 107.0, 131.0, 130.0, 146.0, 123.0, 110.0, 111.0, 111.0, 98.0, 111.0, 97.0, 115.0, 88.0, 83.0, 81.0, 98.0, 103.0, 94.0, 107.0, 113.0, 103.0, 103.0, 132.0, 104.0, 89.0, 86.0, 105.0, 124.0, 136.0, 110.0, 139.0, 91.0, 85.0, 114.0, 105.0, 119.0, 138.0, 109.0, 121.0, 111.0, 112.0, 102.0, 120.0, 104.0, 116.0, 109.0, 101.0, 100.0, 108.0, 114.0, 103.0, 107.0, 94.0, 95.0, 97.0, 65.0, 102.0, 102.0, 88.0, 135.0, 111.0, 103.0, 104.0, 92.0, 100.0, 157.0, 66.0, 111.0, 106.0, 113.0, 110.0, 106.0, 103.0, 96.0, 98.0, 116.0, 107.0, 108.0, 102.0, 87.0, 115.0, 106.0, 92.0, 105.0, 113.0, 108.0, 116.0, 107.0, 102.0, 88.0, 71.0, 97.0, 90.0, 107.0, 99.0, 86.0, 104.0, 116.0, 100.0, 104.0, 99.0, 97.0, 88.0, 105.0, 86.0, 93.0, 106.0, 117.0, 96.0, 92.0, 118.0, 113.0, 139.0, 121.0, 72.0, 111.0, 102.0, 112.0, 113.0, 114.0, 117.0, 98.0, 111.0, 135.0, 82.0, 84.0, 79.0, 101.0, 109.0, 103.0, 119.0, 99.0, 86.0, 122.0, 101.0, 99.0, 100.0, 120.0, 120.0, 106.0, 95.0, 125.0, 106.0, 109.0, 70.0, 117.0, 115.0, 103.0, 92.0, 117.0, 78.0, 112.0, 103.0, 130.0, 117.0, 104.0, 112.0, 123.0, 116.0, 126.0, 104.0, 121.0, 133.0, 100.0, 115.0, 110.0, 116.0, 125.0, 93.0, 119.0, 120.0, 110.0, 89.0, 88.0, 113.0, 112.0, 97.0, 110.0, 112.0, 94.0, 105.0, 109.0, 116.0, 110.0, 117.0, 117.0, 82.0, 108.0, 87.0, 119.0, 93.0, 114.0, 93.0, 127.0, 105.0, 96.0, 110.0, 113.0, 87.0, 128.0, 105.0, 96.0, 107.0, 100.0, 106.0, 108.0, 89.0, 109.0, 108.0, 109.0, 112.0, 112.0, 110.0, 116.0, 103.0, 116.0, 110.0, 103.0, 118.0, 114.0, 130.0, 111.0, 119.0, 107.0, 130.0, 112.0, 107.0, 101.0, 99.0, 113.0, 107.0, 103.0, 107.0, 112.0, 97.0, 98.0, 118.0, 119.0, 121.0, 121.0, 122.0, 113.0, 130.0, 112.0, 113.0, 116.0, 108.0, 135.0, 118.0, 126.0, 132.0, 97.0, 101.0, 100.0, 125.0, 103.0, 122.0, 136.0, 126.0]}, "params-norm": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [180.15739, 180.15739, 180.15739, 180.15739, 180.15739, 180.15739, 180.15736, 180.15726, 180.15707, 180.15691, 180.15549, 180.15459, 180.15424, 180.15189, 180.15099, 180.15024, 180.14986, 180.14993, 180.15019, 180.1503, 180.15027, 180.14986, 180.14977, 180.15002, 180.15099, 180.15236, 180.15358, 180.15434, 180.1554, 180.15681, 180.15871, 180.16106, 180.16335, 180.1655, 180.16797, 180.1711, 180.1745, 180.1783, 180.18207, 180.18634, 180.19115, 180.19635, 180.20181, 180.20787, 180.21454, 180.22186, 180.22972, 180.23808, 180.2473, 180.25745, 180.26848, 180.2802, 180.29237, 180.30516, 180.31874, 180.33293, 180.34735, 180.36238, 180.37834, 180.39542, 180.4135, 180.43236, 180.45271, 180.47404, 180.49562, 180.51866, 180.54253, 180.56715, 180.5934, 180.61932, 180.64636, 180.67368, 180.70193, 180.73018, 180.75891, 180.78816, 180.81766, 180.8484, 180.87955, 180.91142, 180.94348, 180.97565, 181.00879, 181.04236, 181.07651, 181.11137, 181.14594, 181.18066, 181.21619, 181.25278, 181.29031, 181.32835, 181.36548, 181.40294, 181.44122, 181.48024, 181.5182, 181.55528, 181.59256, 181.63011, 181.66725, 181.70305, 181.73674, 181.77116, 181.80685, 181.84525, 181.88437, 181.92274, 181.95988, 181.99857, 182.03806, 182.07884, 182.12015, 182.16119, 182.20111, 182.24168, 182.28267, 182.32266, 182.36147, 182.40109, 182.44116, 182.48097, 182.51984, 182.56007, 182.60045, 182.64178, 182.68237, 182.72194, 182.76109, 182.80022, 182.83957, 182.87726, 182.91669, 182.95601, 182.99387, 183.03162, 183.07095, 183.10947, 183.14935, 183.18875, 183.22766, 183.26535, 183.30247, 183.34052, 183.37903, 183.41861, 183.45737, 183.49628, 183.53458, 183.57204, 183.6071, 183.63815, 183.66853, 183.6991, 183.73117, 183.76399, 183.79651, 183.82997, 183.86507, 183.89973, 183.93646, 183.9742, 184.01169, 184.0497, 184.08951, 184.13031, 184.17166, 184.21358, 184.25455, 184.2946, 184.3347, 184.37413, 184.41353, 184.45135, 184.4884, 184.52621, 184.5629, 184.60046, 184.63802, 184.67714, 184.71693, 184.75653, 184.79752, 184.83904, 184.88031, 184.92084, 184.96179, 185.00244, 185.04277, 185.08441, 185.12462, 185.16237, 185.19899, 185.23643, 185.27388, 185.31174, 185.35019, 185.38876, 185.4269, 185.46609, 185.50525, 185.54359, 185.58316, 185.62428, 185.66612, 185.70808, 185.7489, 185.789, 185.82991, 185.8699, 185.90993, 185.94986, 185.98807, 186.0255, 186.06456, 186.10458, 186.14545, 186.18518, 186.22546, 186.26527, 186.30615, 186.34776, 186.3895, 186.43056, 186.47195, 186.51314, 186.55176, 186.59093, 186.62968, 186.66743, 186.70425, 186.74065, 186.77608, 186.81223, 186.84959, 186.88846, 186.92926, 186.97034, 187.01245, 187.05669, 187.09961, 187.14209, 187.18475, 187.22701, 187.26978, 187.31277, 187.3539, 187.39343, 187.43114, 187.47012, 187.51071, 187.55231, 187.59656, 187.64023, 187.68506, 187.73169, 187.77757, 187.82271, 187.86697, 187.91153, 187.95866, 188.00621, 188.05377, 188.09944, 188.14352, 188.18582, 188.22591, 188.26578, 188.30733, 188.35069, 188.39435, 188.43915, 188.48364, 188.52684, 188.57294, 188.61974, 188.66663, 188.71498, 188.76122, 188.80577, 188.85143, 188.89684, 188.9418, 188.98785, 189.03465, 189.08012, 189.12587, 189.1741, 189.22166, 189.26874, 189.31548, 189.3632, 189.40987, 189.45602, 189.50279, 189.54955, 189.59624, 189.64444, 189.69376, 189.74446, 189.79739, 189.85051, 189.90123, 189.95108, 189.99809, 190.04387, 190.09178, 190.14143, 190.19429, 190.24828, 190.30048, 190.35289, 190.40466, 190.45512, 190.50417, 190.55513, 190.60683, 190.66037, 190.71399, 190.76956, 190.82303, 190.87448, 190.92685, 190.97981, 191.03252, 191.08475, 191.13594, 191.18895, 191.2408, 191.29123, 191.34271, 191.39406, 191.44528, 191.4977, 191.55157, 191.6071, 191.66283, 191.71693, 191.77141, 191.82414, 191.87782, 191.93262, 191.98686, 192.04332, 192.10043, 192.15675, 192.21115, 192.26575, 192.31818, 192.37268, 192.42906, 192.48456, 192.53935, 192.59442, 192.64954, 192.70572, 192.7632, 192.82033, 192.87624, 192.93234, 192.98929, 193.04488, 193.10385, 193.16135, 193.21951, 193.27705, 193.33467, 193.39278, 193.44942, 193.50473, 193.5598, 193.61542, 193.672, 193.72774, 193.78313, 193.83984, 193.89583, 193.95193, 194.00967, 194.06923, 194.12787, 194.18706, 194.24593, 194.30592, 194.36789, 194.43033, 194.49274, 194.55455, 194.61639, 194.6769, 194.73872, 194.79979, 194.85854, 194.91742, 194.97757, 195.037, 195.09503, 195.15454, 195.21541, 195.27866]}, "params-norm vs samples": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [180.15739, 180.15739, 180.15739, 180.15739, 180.15739, 180.15739, 180.15736, 180.15726, 180.15707, 180.15691, 180.15549, 180.15459, 180.15424, 180.15189, 180.15099, 180.15024, 180.14986, 180.14993, 180.15019, 180.1503, 180.15027, 180.14986, 180.14977, 180.15002, 180.15099, 180.15236, 180.15358, 180.15434, 180.1554, 180.15681, 180.15871, 180.16106, 180.16335, 180.1655, 180.16797, 180.1711, 180.1745, 180.1783, 180.18207, 180.18634, 180.19115, 180.19635, 180.20181, 180.20787, 180.21454, 180.22186, 180.22972, 180.23808, 180.2473, 180.25745, 180.26848, 180.2802, 180.29237, 180.30516, 180.31874, 180.33293, 180.34735, 180.36238, 180.37834, 180.39542, 180.4135, 180.43236, 180.45271, 180.47404, 180.49562, 180.51866, 180.54253, 180.56715, 180.5934, 180.61932, 180.64636, 180.67368, 180.70193, 180.73018, 180.75891, 180.78816, 180.81766, 180.8484, 180.87955, 180.91142, 180.94348, 180.97565, 181.00879, 181.04236, 181.07651, 181.11137, 181.14594, 181.18066, 181.21619, 181.25278, 181.29031, 181.32835, 181.36548, 181.40294, 181.44122, 181.48024, 181.5182, 181.55528, 181.59256, 181.63011, 181.66725, 181.70305, 181.73674, 181.77116, 181.80685, 181.84525, 181.88437, 181.92274, 181.95988, 181.99857, 182.03806, 182.07884, 182.12015, 182.16119, 182.20111, 182.24168, 182.28267, 182.32266, 182.36147, 182.40109, 182.44116, 182.48097, 182.51984, 182.56007, 182.60045, 182.64178, 182.68237, 182.72194, 182.76109, 182.80022, 182.83957, 182.87726, 182.91669, 182.95601, 182.99387, 183.03162, 183.07095, 183.10947, 183.14935, 183.18875, 183.22766, 183.26535, 183.30247, 183.34052, 183.37903, 183.41861, 183.45737, 183.49628, 183.53458, 183.57204, 183.6071, 183.63815, 183.66853, 183.6991, 183.73117, 183.76399, 183.79651, 183.82997, 183.86507, 183.89973, 183.93646, 183.9742, 184.01169, 184.0497, 184.08951, 184.13031, 184.17166, 184.21358, 184.25455, 184.2946, 184.3347, 184.37413, 184.41353, 184.45135, 184.4884, 184.52621, 184.5629, 184.60046, 184.63802, 184.67714, 184.71693, 184.75653, 184.79752, 184.83904, 184.88031, 184.92084, 184.96179, 185.00244, 185.04277, 185.08441, 185.12462, 185.16237, 185.19899, 185.23643, 185.27388, 185.31174, 185.35019, 185.38876, 185.4269, 185.46609, 185.50525, 185.54359, 185.58316, 185.62428, 185.66612, 185.70808, 185.7489, 185.789, 185.82991, 185.8699, 185.90993, 185.94986, 185.98807, 186.0255, 186.06456, 186.10458, 186.14545, 186.18518, 186.22546, 186.26527, 186.30615, 186.34776, 186.3895, 186.43056, 186.47195, 186.51314, 186.55176, 186.59093, 186.62968, 186.66743, 186.70425, 186.74065, 186.77608, 186.81223, 186.84959, 186.88846, 186.92926, 186.97034, 187.01245, 187.05669, 187.09961, 187.14209, 187.18475, 187.22701, 187.26978, 187.31277, 187.3539, 187.39343, 187.43114, 187.47012, 187.51071, 187.55231, 187.59656, 187.64023, 187.68506, 187.73169, 187.77757, 187.82271, 187.86697, 187.91153, 187.95866, 188.00621, 188.05377, 188.09944, 188.14352, 188.18582, 188.22591, 188.26578, 188.30733, 188.35069, 188.39435, 188.43915, 188.48364, 188.52684, 188.57294, 188.61974, 188.66663, 188.71498, 188.76122, 188.80577, 188.85143, 188.89684, 188.9418, 188.98785, 189.03465, 189.08012, 189.12587, 189.1741, 189.22166, 189.26874, 189.31548, 189.3632, 189.40987, 189.45602, 189.50279, 189.54955, 189.59624, 189.64444, 189.69376, 189.74446, 189.79739, 189.85051, 189.90123, 189.95108, 189.99809, 190.04387, 190.09178, 190.14143, 190.19429, 190.24828, 190.30048, 190.35289, 190.40466, 190.45512, 190.50417, 190.55513, 190.60683, 190.66037, 190.71399, 190.76956, 190.82303, 190.87448, 190.92685, 190.97981, 191.03252, 191.08475, 191.13594, 191.18895, 191.2408, 191.29123, 191.34271, 191.39406, 191.44528, 191.4977, 191.55157, 191.6071, 191.66283, 191.71693, 191.77141, 191.82414, 191.87782, 191.93262, 191.98686, 192.04332, 192.10043, 192.15675, 192.21115, 192.26575, 192.31818, 192.37268, 192.42906, 192.48456, 192.53935, 192.59442, 192.64954, 192.70572, 192.7632, 192.82033, 192.87624, 192.93234, 192.98929, 193.04488, 193.10385, 193.16135, 193.21951, 193.27705, 193.33467, 193.39278, 193.44942, 193.50473, 193.5598, 193.61542, 193.672, 193.72774, 193.78313, 193.83984, 193.89583, 193.95193, 194.00967, 194.06923, 194.12787, 194.18706, 194.24593, 194.30592, 194.36789, 194.43033, 194.49274, 194.55455, 194.61639, 194.6769, 194.73872, 194.79979, 194.85854, 194.91742, 194.97757, 195.037, 195.09503, 195.15454, 195.21541, 195.27866]}, "iteration-time": {"start_step": 0, "end_step": 2000, "step_interval": 5, "values": [18.43353, 1.85226, 1.82214, 1.81825, 1.81981, 1.81719, 1.80366, 1.79948, 1.80048, 1.80169, 1.79, 1.78536, 1.80752, 1.78849, 1.79821, 1.74679, 1.74509, 1.72989, 1.75731, 1.80341, 1.7289, 1.72572, 1.7272, 1.71985, 1.72747, 1.72364, 1.71951, 1.8777, 1.73639, 1.73795, 1.71459, 1.71943, 1.72545, 1.71939, 2.03183, 1.72026, 1.72349, 1.73232, 1.72789, 1.73545, 1.94328, 1.72485, 1.97676, 1.71579, 1.72565, 1.72237, 1.73622, 1.72503, 1.72039, 1.71998, 1.72197, 1.72316, 1.72014, 1.72689, 1.72369, 1.72159, 1.74413, 1.73342, 1.7271, 1.72579, 1.74825, 1.72663, 1.72485, 1.74263, 1.73176, 1.7296, 1.71978, 1.73377, 1.72626, 1.75192, 1.72393, 1.72309, 1.72964, 1.72395, 1.7473, 1.72705, 1.74772, 1.72764, 1.72202, 1.72828, 1.71969, 1.74565, 1.73482, 1.74135, 1.72177, 1.73127, 1.724, 1.72244, 1.72226, 1.71529, 1.755, 1.71933, 1.72772, 1.72262, 1.72597, 1.72686, 1.7236, 1.72442, 1.73027, 1.72391, 1.72094, 1.72559, 1.73171, 1.73024, 1.73631, 1.73367, 1.73511, 1.72708, 1.72366, 1.7301, 1.73714, 1.73615, 1.91407, 1.72837, 1.73579, 1.73322, 1.71949, 1.72744, 1.73239, 1.73482, 1.7329, 1.72598, 1.7277, 1.72467, 1.72523, 1.72913, 1.72999, 1.73172, 1.72856, 1.72623, 1.73798, 1.72309, 1.7363, 1.74003, 1.72587, 1.72602, 1.72968, 1.72373, 1.72448, 1.72287, 1.71933, 1.71796, 1.71986, 1.73837, 1.73303, 1.73863, 1.73086, 1.72881, 1.72797, 1.73476, 1.74944, 1.72264, 1.73569, 1.72592, 1.72795, 1.73241, 1.73495, 1.73937, 1.73359, 1.74977, 1.75337, 1.72708, 1.89046, 1.72715, 1.74486, 1.722, 1.74896, 1.87803, 1.7446, 1.74223, 1.73969, 1.74413, 1.73943, 1.7519, 1.74639, 1.74251, 1.7245, 1.73672, 1.74147, 1.72322, 1.72526, 1.73758, 1.72812, 1.72801, 1.73395, 1.72585, 1.73031, 1.73342, 1.75634, 1.73337, 1.73418, 1.72951, 1.74401, 1.72931, 1.74541, 1.88514, 1.73449, 1.72763, 1.72313, 1.72098, 1.74526, 1.99525, 1.74443, 1.73494, 1.74003, 1.73573, 1.73333, 1.73953, 1.73127, 1.72163, 1.74426, 1.7409, 1.73597, 1.73513, 1.75695, 1.7354, 1.74814, 1.73746, 1.74335, 1.74366, 1.75028, 1.72559, 1.72574, 1.73452, 1.73232, 1.75479, 1.74589, 1.74991, 1.73419, 1.73913, 1.74467, 1.73278, 1.74103, 1.73526, 1.73749, 1.75397, 1.73296, 1.72731, 1.73248, 1.74505, 1.73965, 1.73801, 1.75714, 1.73939, 1.74253, 1.75025, 1.74395, 1.74206, 1.74458, 1.74656, 1.73134, 1.73471, 1.72781, 1.73288, 1.73243, 1.73364, 1.72983, 1.73679, 1.73534, 1.73197, 1.73653, 1.73921, 1.74103, 1.75819, 1.74546, 1.74243, 1.75797, 1.74168, 1.7422, 1.76138, 1.75808, 1.74491, 1.74537, 1.76205, 1.73577, 1.73037, 1.74437, 1.74913, 1.74798, 1.75661, 1.75383, 1.90843, 1.7694, 1.75494, 1.75637, 1.75355, 1.76083, 1.75152, 1.74229, 1.75401, 1.75135, 1.74417, 1.74565, 1.74718, 1.74854, 1.73901, 1.75268, 1.74731, 1.7452, 1.74059, 1.74651, 1.73562, 1.75669, 1.76629, 1.74961, 1.75024, 1.74137, 1.77053, 1.87714, 1.74436, 1.74255, 1.72662, 1.73832, 1.737, 1.73698, 1.73333, 1.75518, 1.77044, 1.74474, 1.74812, 1.74327, 1.7469, 1.73316, 1.75446, 1.74993, 1.75346, 1.74378, 1.73818, 1.74649, 1.74128, 1.75797, 1.73996, 1.74171, 1.73869, 1.73927, 1.73142, 1.73581, 1.75653, 1.75153, 1.73564, 1.74222, 1.73463, 1.73507, 1.73406, 1.74675, 1.75913, 1.74844, 1.74564, 1.7327, 1.74501, 1.75062, 1.74412, 1.73709, 1.73903, 1.74097, 1.74102, 1.73777, 1.74052, 1.73715, 1.73979, 1.73371, 1.73625, 1.77593, 1.74164, 1.74978, 1.74778, 1.74612, 1.7494, 1.74188, 1.74065, 1.73429, 1.73414, 1.74917, 1.73548, 1.73116, 1.7282, 1.74624, 1.72906, 1.74788, 1.73862, 1.73861, 1.74043, 1.7383, 1.73476, 1.72896, 1.75519, 1.7453, 1.7446, 1.75416, 1.73981, 1.75039, 1.74694, 1.73365, 1.73974, 1.73608, 1.73902, 1.72608, 1.74038, 1.75637, 1.75328]}, "lm loss validation": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [5.59759]}, "lm loss validation vs samples": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [5.59759]}, "lm loss validation ppl": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [269.77509]}, "lm loss validation ppl vs samples": {"start_step": 0, "end_step": 4, "step_interval": 5, "values": [269.77509]}}
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_pp2_fp8_tp_pp/golden_values_lts_dgx_a100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_pp2_fp8_tp_pp/golden_values_lts_dgx_a100.json
deleted file mode 100644
index 0463c4d01dc..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_pp2_fp8_tp_pp/golden_values_lts_dgx_a100.json
+++ /dev/null
@@ -1,1223 +0,0 @@
-{
- "lm loss": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 10.89904,
- 10.90777,
- 10.89232,
- 10.83544,
- 10.6834,
- 10.65974,
- 10.44873,
- 10.16308,
- 9.95831,
- 9.85932,
- 9.60254,
- 9.85446,
- 9.88893,
- 9.63287,
- 9.79405,
- 9.51078,
- 9.46463,
- 9.65471,
- 9.39306,
- 9.33895,
- 9.24972,
- 9.15413,
- 9.17988,
- 9.0065,
- 9.19899,
- 9.06474,
- 9.16249,
- 9.16631,
- 9.30043,
- 8.98957,
- 8.93842,
- 9.05744,
- 9.05222,
- 8.66356,
- 8.72626,
- 8.7667,
- 8.70006,
- 8.74817,
- 8.67179,
- 8.78274,
- 8.67795,
- 8.86767,
- 8.84929,
- 8.51536,
- 8.40624,
- 8.45093,
- 8.51004,
- 8.40653,
- 8.45216,
- 8.6026,
- 8.38502,
- 8.21394,
- 8.24297,
- 8.23879,
- 8.28518,
- 7.93123,
- 8.10705,
- 7.90575,
- 8.25948,
- 8.24016,
- 8.01415,
- 7.97894,
- 7.93174,
- 7.74864,
- 7.74918,
- 7.65293,
- 7.52384,
- 7.91349,
- 7.70509,
- 7.46214,
- 7.74596,
- 7.77384,
- 7.5447,
- 7.30561,
- 7.45871,
- 7.34545,
- 7.46856,
- 7.23017,
- 7.64088,
- 7.27983,
- 7.34981,
- 7.21134,
- 7.21081,
- 7.42102,
- 7.17384,
- 7.28052,
- 6.99786,
- 7.00152,
- 7.03624,
- 7.13136,
- 6.82298,
- 6.98762,
- 7.08699,
- 6.99714,
- 6.87231,
- 6.75444,
- 6.98392,
- 7.05773,
- 6.69999,
- 6.57801,
- 6.72248,
- 6.73865,
- 6.73005,
- 6.73698,
- 6.65374,
- 6.40729,
- 6.6365,
- 6.61972,
- 6.44423,
- 6.62637,
- 6.74067,
- 6.60551,
- 6.72345,
- 6.68935,
- 6.62052,
- 6.50773,
- 6.59703,
- 6.40181,
- 6.66219,
- 6.24576,
- 6.24815,
- 6.29992,
- 6.38652,
- 6.34284,
- 6.44395,
- 6.2868,
- 6.33137,
- 6.23064,
- 6.19419,
- 6.38932,
- 6.31955,
- 6.31115,
- 6.15595,
- 6.14904,
- 6.23012,
- 6.37609,
- 6.19108,
- 6.14016,
- 6.17443,
- 6.108,
- 6.05677,
- 6.07051,
- 6.2515,
- 6.40359,
- 6.25653,
- 6.30179,
- 6.09464,
- 6.1786,
- 6.00393,
- 6.03024,
- 5.95456,
- 6.25097,
- 6.18949,
- 5.96652,
- 5.78509,
- 6.12471,
- 5.85239,
- 6.09954,
- 5.78907,
- 6.1634,
- 6.14662,
- 6.08899,
- 5.93324,
- 6.11629,
- 5.94863,
- 6.19744,
- 5.89699,
- 5.79464,
- 5.78508,
- 5.6887,
- 6.01484,
- 5.99513,
- 6.06793,
- 5.88964,
- 6.04218,
- 5.96664,
- 5.9946,
- 5.98873,
- 5.94909,
- 5.83777,
- 5.94965,
- 5.62073,
- 5.70203,
- 5.88937,
- 5.84442,
- 5.86415,
- 5.75977,
- 5.83426,
- 5.72464,
- 5.56351,
- 5.71986,
- 5.62642,
- 5.83426,
- 5.60742,
- 5.71258,
- 5.70976,
- 5.8987,
- 5.64295,
- 5.85277,
- 5.73889,
- 5.87053,
- 5.32966,
- 5.89533,
- 5.87205,
- 5.85426,
- 5.41037,
- 5.40663,
- 5.62114,
- 5.59572,
- 5.48482,
- 5.57586,
- 5.67197,
- 5.4726,
- 5.74298,
- 5.50672,
- 5.5935,
- 5.61776,
- 5.6179,
- 5.51203,
- 5.61413,
- 5.67291,
- 5.68327,
- 5.58724,
- 5.66009,
- 5.37678,
- 5.68099,
- 5.62359,
- 5.42053,
- 5.57867,
- 5.62946,
- 5.54954,
- 5.33822,
- 5.53445,
- 5.48149,
- 5.47842,
- 5.37511,
- 5.5464,
- 5.60351,
- 5.38706,
- 5.51715,
- 5.48729,
- 5.33094,
- 5.50178,
- 5.40732,
- 5.44712,
- 5.31548,
- 5.06617,
- 5.47969,
- 5.56831,
- 5.7133,
- 5.41401,
- 5.59841,
- 5.63558,
- 5.2322,
- 5.27319,
- 5.38792,
- 5.39306,
- 5.32904,
- 5.49509,
- 5.17834,
- 5.29764,
- 5.24393,
- 5.37614,
- 5.25456,
- 5.44258,
- 5.54017,
- 5.31017,
- 5.43225,
- 5.33341,
- 5.07298,
- 5.31187,
- 5.2557,
- 5.30514,
- 5.10844,
- 5.27459,
- 5.26496,
- 5.47616,
- 5.16669,
- 5.26555,
- 5.21176,
- 5.355,
- 4.98377,
- 4.91178,
- 5.33096,
- 5.38935,
- 5.23414,
- 5.31329,
- 5.10388,
- 5.16417,
- 5.26356,
- 5.06801,
- 5.27045,
- 5.07377,
- 5.34602,
- 5.24563,
- 5.15001,
- 5.24094,
- 5.04069,
- 5.31488,
- 5.04958,
- 5.02979,
- 5.13788,
- 5.11434,
- 5.26734,
- 5.14852,
- 5.27369,
- 5.08851,
- 5.09324,
- 5.24624,
- 5.32324,
- 5.25443,
- 5.19052,
- 5.14435,
- 5.29055,
- 4.94885,
- 5.20441,
- 5.0907,
- 5.29874,
- 5.17267,
- 5.18858,
- 5.11677,
- 4.98159,
- 4.99122,
- 5.22123,
- 5.30764,
- 5.10222,
- 5.0544,
- 4.91358,
- 5.12177,
- 5.11614,
- 4.92915,
- 5.33612,
- 5.01913,
- 5.10051,
- 5.16573,
- 4.99929,
- 5.06049,
- 5.06814,
- 4.99437,
- 5.07642,
- 5.16464,
- 4.98109,
- 5.1825,
- 4.92945,
- 4.92916,
- 5.06868,
- 4.99902,
- 4.90979,
- 4.77687,
- 4.94499,
- 5.11671,
- 5.01541,
- 5.02126,
- 5.32954,
- 4.95713,
- 4.99895,
- 5.05055,
- 4.81011,
- 4.73872,
- 5.00091,
- 5.04398,
- 4.87805,
- 4.95233,
- 5.04347,
- 5.02539,
- 4.82104,
- 4.90025,
- 4.90912,
- 4.83747,
- 4.75039,
- 5.01482,
- 4.74829,
- 5.21037,
- 4.79047,
- 5.00245,
- 4.74175,
- 4.79189,
- 4.82107,
- 4.65381,
- 4.66051,
- 4.84616,
- 4.81073,
- 4.8078,
- 4.92405,
- 4.88723,
- 4.93597,
- 4.77468,
- 4.88361,
- 4.74125,
- 4.92209,
- 4.96252,
- 4.87874,
- 4.71289,
- 4.79114,
- 4.90017,
- 4.7175,
- 4.87202,
- 4.69846,
- 4.70626,
- 4.65256
- ]
- },
- "num-zeros": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 58.0,
- 87.0,
- 81.0,
- 84.0,
- 84.0,
- 90.0,
- 104.0,
- 124.0,
- 102.0,
- 132.0,
- 129.0,
- 152.0,
- 143.0,
- 181.0,
- 202.0,
- 161.0,
- 161.0,
- 177.0,
- 184.0,
- 189.0,
- 151.0,
- 167.0,
- 183.0,
- 182.0,
- 186.0,
- 154.0,
- 178.0,
- 163.0,
- 167.0,
- 148.0,
- 145.0,
- 138.0,
- 187.0,
- 168.0,
- 140.0,
- 142.0,
- 167.0,
- 204.0,
- 169.0,
- 203.0,
- 148.0,
- 155.0,
- 141.0,
- 200.0,
- 190.0,
- 169.0,
- 187.0,
- 196.0,
- 175.0,
- 229.0,
- 207.0,
- 188.0,
- 199.0,
- 157.0,
- 186.0,
- 178.0,
- 154.0,
- 138.0,
- 248.0,
- 232.0,
- 174.0,
- 186.0,
- 188.0,
- 193.0,
- 201.0,
- 239.0,
- 207.0,
- 166.0,
- 208.0,
- 203.0,
- 208.0,
- 254.0,
- 168.0,
- 251.0,
- 210.0,
- 201.0,
- 239.0,
- 211.0,
- 241.0,
- 211.0,
- 204.0,
- 215.0,
- 193.0,
- 225.0,
- 213.0,
- 184.0,
- 182.0,
- 191.0,
- 206.0,
- 206.0,
- 188.0,
- 218.0,
- 214.0,
- 205.0,
- 203.0,
- 166.0,
- 206.0,
- 174.0,
- 195.0,
- 174.0,
- 140.0,
- 154.0,
- 176.0,
- 165.0,
- 129.0,
- 148.0,
- 168.0,
- 157.0,
- 137.0,
- 180.0,
- 175.0,
- 163.0,
- 175.0,
- 145.0,
- 138.0,
- 134.0,
- 159.0,
- 128.0,
- 173.0,
- 161.0,
- 151.0,
- 113.0,
- 133.0,
- 129.0,
- 177.0,
- 125.0,
- 153.0,
- 137.0,
- 120.0,
- 142.0,
- 148.0,
- 143.0,
- 100.0,
- 113.0,
- 106.0,
- 124.0,
- 129.0,
- 93.0,
- 119.0,
- 125.0,
- 107.0,
- 107.0,
- 141.0,
- 141.0,
- 122.0,
- 91.0,
- 142.0,
- 120.0,
- 101.0,
- 141.0,
- 130.0,
- 112.0,
- 107.0,
- 110.0,
- 132.0,
- 105.0,
- 102.0,
- 116.0,
- 115.0,
- 122.0,
- 96.0,
- 122.0,
- 87.0,
- 104.0,
- 112.0,
- 91.0,
- 110.0,
- 107.0,
- 101.0,
- 103.0,
- 107.0,
- 117.0,
- 83.0,
- 102.0,
- 105.0,
- 133.0,
- 96.0,
- 115.0,
- 93.0,
- 128.0,
- 129.0,
- 113.0,
- 112.0,
- 104.0,
- 104.0,
- 90.0,
- 85.0,
- 92.0,
- 96.0,
- 79.0,
- 140.0,
- 112.0,
- 103.0,
- 85.0,
- 96.0,
- 103.0,
- 104.0,
- 90.0,
- 109.0,
- 115.0,
- 113.0,
- 82.0,
- 123.0,
- 128.0,
- 86.0,
- 113.0,
- 103.0,
- 100.0,
- 129.0,
- 90.0,
- 96.0,
- 92.0,
- 106.0,
- 106.0,
- 113.0,
- 127.0,
- 112.0,
- 118.0,
- 96.0,
- 106.0,
- 114.0,
- 93.0,
- 85.0,
- 74.0,
- 105.0,
- 113.0,
- 97.0,
- 113.0,
- 107.0,
- 97.0,
- 109.0,
- 87.0,
- 89.0,
- 108.0,
- 106.0,
- 87.0,
- 120.0,
- 115.0,
- 109.0,
- 111.0,
- 100.0,
- 114.0,
- 102.0,
- 106.0,
- 94.0,
- 106.0,
- 77.0,
- 124.0,
- 112.0,
- 102.0,
- 104.0,
- 111.0,
- 109.0,
- 125.0,
- 114.0,
- 109.0,
- 120.0,
- 120.0,
- 103.0,
- 107.0,
- 86.0,
- 111.0,
- 95.0,
- 102.0,
- 108.0,
- 78.0,
- 100.0,
- 90.0,
- 107.0,
- 101.0,
- 104.0,
- 119.0,
- 100.0,
- 113.0,
- 110.0,
- 113.0,
- 90.0,
- 101.0,
- 107.0,
- 106.0,
- 111.0,
- 88.0,
- 125.0,
- 93.0,
- 106.0,
- 103.0,
- 116.0,
- 127.0,
- 100.0,
- 84.0,
- 102.0,
- 97.0,
- 97.0,
- 94.0,
- 120.0,
- 109.0,
- 110.0,
- 98.0,
- 97.0,
- 113.0,
- 108.0,
- 106.0,
- 143.0,
- 104.0,
- 111.0,
- 106.0,
- 103.0,
- 99.0,
- 110.0,
- 106.0,
- 130.0,
- 121.0,
- 112.0,
- 103.0,
- 101.0,
- 97.0,
- 115.0,
- 127.0,
- 117.0,
- 116.0,
- 109.0,
- 101.0,
- 129.0,
- 101.0,
- 99.0,
- 112.0,
- 91.0,
- 113.0,
- 104.0,
- 122.0,
- 91.0,
- 120.0,
- 124.0,
- 89.0,
- 106.0,
- 106.0,
- 119.0,
- 101.0,
- 98.0,
- 102.0,
- 129.0,
- 107.0,
- 116.0,
- 126.0,
- 127.0,
- 112.0,
- 86.0,
- 106.0,
- 136.0,
- 135.0,
- 107.0,
- 93.0,
- 102.0,
- 118.0,
- 117.0,
- 104.0,
- 123.0,
- 99.0,
- 114.0,
- 92.0,
- 128.0,
- 92.0,
- 107.0,
- 92.0,
- 124.0,
- 106.0,
- 101.0,
- 112.0,
- 106.0,
- 99.0,
- 107.0,
- 110.0,
- 97.0,
- 108.0,
- 117.0,
- 119.0,
- 102.0,
- 116.0,
- 116.0,
- 118.0,
- 108.0,
- 130.0,
- 116.0,
- 118.0,
- 122.0,
- 105.0,
- 104.0,
- 126.0,
- 123.0,
- 118.0,
- 124.0,
- 126.0,
- 97.0,
- 123.0,
- 133.0,
- 101.0,
- 117.0,
- 114.0,
- 120.0,
- 139.0
- ]
- },
- "iteration-time": {
- "start_step": 0,
- "end_step": 2000,
- "step_interval": 5,
- "values": [
- 26.69562,
- 2.22579,
- 2.19499,
- 2.18927,
- 2.20347,
- 2.20486,
- 2.19616,
- 2.18953,
- 2.19243,
- 2.19162,
- 2.19709,
- 2.20446,
- 2.21739,
- 2.14781,
- 2.14676,
- 2.13968,
- 2.16989,
- 2.16276,
- 2.16801,
- 2.16925,
- 2.14717,
- 2.14458,
- 2.54878,
- 2.14922,
- 2.2881,
- 2.14125,
- 2.13827,
- 2.14419,
- 2.13768,
- 2.14618,
- 2.29108,
- 2.17599,
- 2.15672,
- 2.15376,
- 2.15175,
- 2.15365,
- 2.14489,
- 2.16446,
- 2.14511,
- 2.14607,
- 2.14463,
- 2.14073,
- 2.15082,
- 2.14429,
- 2.1629,
- 2.14914,
- 2.14671,
- 2.14152,
- 2.1578,
- 2.15036,
- 2.18156,
- 2.14947,
- 2.15499,
- 2.15448,
- 2.14498,
- 2.17022,
- 2.15074,
- 2.15315,
- 2.14479,
- 2.14643,
- 2.1539,
- 2.17161,
- 2.15621,
- 2.14956,
- 2.18535,
- 2.17453,
- 2.19533,
- 2.18873,
- 2.17428,
- 2.17286,
- 2.16489,
- 2.17738,
- 2.1729,
- 2.16198,
- 2.15566,
- 2.16685,
- 2.17114,
- 2.17505,
- 2.16943,
- 2.18665,
- 2.18086,
- 2.17335,
- 2.16894,
- 2.17859,
- 2.17143,
- 2.16927,
- 2.17751,
- 2.16672,
- 2.18668,
- 2.16427,
- 2.15535,
- 2.16126,
- 2.16744,
- 2.15529,
- 2.1683,
- 2.14738,
- 2.16013,
- 2.15296,
- 2.14264,
- 2.14233,
- 2.1445,
- 2.17158,
- 2.14916,
- 2.14433,
- 2.1608,
- 2.15794,
- 2.14246,
- 2.15069,
- 2.15369,
- 2.14475,
- 2.1647,
- 2.1604,
- 2.18225,
- 2.15673,
- 2.14813,
- 2.14564,
- 2.16483,
- 2.1564,
- 2.15075,
- 2.30566,
- 2.14216,
- 2.14965,
- 2.15397,
- 2.15357,
- 2.15392,
- 2.15154,
- 2.14714,
- 2.15537,
- 2.15606,
- 2.15318,
- 2.39222,
- 2.15518,
- 2.14998,
- 2.16426,
- 2.15347,
- 2.14496,
- 2.14627,
- 2.14836,
- 2.17996,
- 2.16333,
- 2.16367,
- 2.14627,
- 2.14971,
- 2.14499,
- 2.14774,
- 2.14902,
- 2.14984,
- 2.17596,
- 2.15014,
- 2.15114,
- 2.17123,
- 2.15357,
- 2.14945,
- 2.14978,
- 2.14929,
- 2.143,
- 2.15155,
- 2.16019,
- 2.17298,
- 2.16063,
- 2.15144,
- 2.16011,
- 2.14807,
- 2.14632,
- 2.15697,
- 2.15198,
- 2.1584,
- 2.15233,
- 2.16268,
- 2.1648,
- 2.1546,
- 2.14525,
- 2.14593,
- 2.14622,
- 2.14391,
- 2.15344,
- 2.16086,
- 2.15831,
- 2.15122,
- 2.14385,
- 2.15243,
- 2.13958,
- 2.14961,
- 2.16846,
- 2.1672,
- 2.15294,
- 2.1424,
- 2.14522,
- 2.19892,
- 2.17537,
- 2.16817,
- 2.1508,
- 2.15436,
- 2.15954,
- 2.15932,
- 2.15852,
- 2.15398,
- 2.13928,
- 2.13132,
- 2.16325,
- 2.14825,
- 2.16326,
- 2.17018,
- 2.16749,
- 2.17147,
- 2.16062,
- 2.16772,
- 2.1526,
- 2.15889,
- 2.16306,
- 2.17467,
- 2.15558,
- 2.16352,
- 2.1856,
- 2.19806,
- 2.2298,
- 2.20851,
- 2.17979,
- 2.17878,
- 2.17373,
- 2.17104,
- 2.18177,
- 2.15319,
- 2.15977,
- 2.16469,
- 2.16464,
- 2.1571,
- 2.15656,
- 2.16189,
- 2.16054,
- 2.16321,
- 2.14799,
- 2.1629,
- 2.14171,
- 2.1408,
- 2.14258,
- 2.14713,
- 2.17553,
- 2.17828,
- 2.15109,
- 2.14335,
- 2.14927,
- 2.1447,
- 2.15428,
- 2.14328,
- 2.14617,
- 2.14817,
- 2.14913,
- 2.1404,
- 2.15508,
- 2.13322,
- 2.1406,
- 2.14928,
- 2.13653,
- 2.14713,
- 2.13506,
- 2.27029,
- 2.15052,
- 2.14911,
- 2.14541,
- 2.16559,
- 2.16935,
- 2.15521,
- 2.13934,
- 2.16298,
- 2.16669,
- 2.1549,
- 2.13974,
- 2.14288,
- 2.13777,
- 2.14539,
- 2.13368,
- 2.14607,
- 2.14212,
- 2.15813,
- 2.14424,
- 2.20917,
- 2.15467,
- 2.15789,
- 2.13681,
- 2.142,
- 2.13498,
- 2.15345,
- 2.14681,
- 2.13383,
- 2.14469,
- 2.13318,
- 2.16468,
- 2.16004,
- 2.14196,
- 2.1427,
- 2.68517,
- 2.1476,
- 2.14172,
- 2.14451,
- 2.1428,
- 2.14565,
- 2.1421,
- 2.14395,
- 2.14997,
- 2.14164,
- 2.13444,
- 2.1407,
- 2.1462,
- 2.16449,
- 2.15818,
- 2.16163,
- 2.1363,
- 2.15192,
- 2.14322,
- 2.14276,
- 2.14054,
- 2.1415,
- 2.15422,
- 2.14653,
- 2.14785,
- 2.15357,
- 2.2487,
- 2.14206,
- 2.16734,
- 2.15219,
- 2.14305,
- 2.1461,
- 2.14578,
- 2.14928,
- 2.14065,
- 2.14592,
- 2.16086,
- 2.16724,
- 2.16219,
- 2.15334,
- 2.14984,
- 2.15032,
- 2.14921,
- 2.14531,
- 2.13826,
- 2.13748,
- 2.14995,
- 2.14539,
- 2.1389,
- 2.16049,
- 2.18618,
- 2.17643,
- 2.16597,
- 2.15903,
- 2.16816,
- 2.16298,
- 2.1688,
- 2.17148,
- 2.16559,
- 2.15895,
- 2.15812,
- 2.1641,
- 2.17292,
- 2.18083,
- 2.31263,
- 2.16745,
- 2.14954,
- 2.15456,
- 2.16475,
- 2.16778,
- 2.17943,
- 2.16494,
- 2.17602,
- 2.15629,
- 2.15465,
- 2.17417,
- 2.15746,
- 2.1614,
- 2.15894,
- 2.172,
- 2.19984,
- 2.16888,
- 2.16555,
- 2.17016,
- 2.16439,
- 2.18253,
- 2.18012,
- 2.16923,
- 2.1657,
- 2.16063,
- 2.14964,
- 2.14503,
- 2.15339,
- 2.15052,
- 2.14668,
- 2.13928,
- 2.16527,
- 2.17177,
- 2.1525,
- 2.15968,
- 2.16198,
- 2.16082,
- 2.17578,
- 2.1759,
- 2.14695,
- 2.15109,
- 2.15254,
- 2.15433,
- 2.17792
- ]
- }
-}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_pp2_fp8_tp_pp/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_pp2_fp8_tp_pp/model_config.yaml
deleted file mode 100644
index cf9f6b6ceb8..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_pp2_fp8_tp_pp/model_config.yaml
+++ /dev/null
@@ -1,54 +0,0 @@
-ENV_VARS:
- CUDA_DEVICE_MAX_CONNECTIONS: 1
- NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
- NVTE_FUSED_ATTN: 0
- NCCL_ALGO: Ring
- CUBLAS_WORKSPACE_CONFIG: :4096:8
-MODEL_ARGS:
- --num-layers: 12
- --hidden-size: 512
- --num-attention-heads: 8
- --log-params-norm: true
- --log-num-zeros-in-grad: true
- --log-validation-ppl-to-tensorboard: true
- --log-timers-to-tensorboard: true
- --tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
- --seq-length: 1024
- --max-position-embeddings: 1024
- --train-iters: 2000
- --timing-log-level: 0
- --lr-decay-iters: 320000
- --save: ${CHECKPOINT_SAVE_PATH}
- --load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
- --split: 949,50,1
- --distributed-backend: nccl
- --lr: 0.00015
- --lr-decay-style: cosine
- --min-lr: 1.0e-5
- --weight-decay: 1e-2
- --clip-grad: 1.0
- --lr-warmup-fraction: .01
- --log-interval: 1
- --save-interval: 10000
- --eval-interval: 1000
- --eval-iters: 10
- --transformer-impl: transformer_engine
- --tensor-model-parallel-size: 4
- --pipeline-model-parallel-size: 2
- --deterministic-mode: true
- --no-gradient-accumulation-fusion: true
- --fp8-format: hybrid
- --fp8-amax-history-len: 1024
- --fp8-amax-compute-algo: max
- --attention-softmax-in-fp32: true
- --ckpt-format: torch_dist
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
- --data-cache-path: ${DATA_CACHE_PATH}
- --bf16: true
- --log-memory-to-tensorboard: true
-TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/golden_values_dev_dgx_gb200.json
new file mode 100644
index 00000000000..ac1685e7437
--- /dev/null
+++ b/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/golden_values_dev_dgx_gb200.json
@@ -0,0 +1,10037 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 10.87366,
+ "2": 10.87917,
+ "3": 10.87494,
+ "4": 10.89357,
+ "5": 10.88004,
+ "6": 10.8742,
+ "7": 10.88062,
+ "8": 10.87499,
+ "9": 10.87295,
+ "10": 10.8673,
+ "11": 10.86555,
+ "12": 10.85015,
+ "13": 10.84359,
+ "14": 10.86463,
+ "15": 10.79543,
+ "16": 10.80844,
+ "17": 10.78667,
+ "18": 10.80756,
+ "19": 10.73111,
+ "20": 10.69618,
+ "21": 10.64834,
+ "22": 10.65104,
+ "23": 10.65157,
+ "24": 10.53914,
+ "25": 10.5528,
+ "26": 10.6232,
+ "27": 10.4345,
+ "28": 10.45665,
+ "29": 10.33033,
+ "30": 10.23666,
+ "31": 10.41313,
+ "32": 10.32912,
+ "33": 10.18953,
+ "34": 10.13714,
+ "35": 10.20572,
+ "36": 10.12451,
+ "37": 10.06,
+ "38": 10.00185,
+ "39": 10.01614,
+ "40": 10.04989,
+ "41": 9.9184,
+ "42": 9.92404,
+ "43": 9.83274,
+ "44": 9.97156,
+ "45": 9.98768,
+ "46": 9.82043,
+ "47": 9.9879,
+ "48": 9.79679,
+ "49": 9.93752,
+ "50": 9.93301,
+ "51": 9.57123,
+ "52": 9.78857,
+ "53": 9.61142,
+ "54": 9.88387,
+ "55": 9.72352,
+ "56": 9.84085,
+ "57": 9.84872,
+ "58": 9.86831,
+ "59": 9.53824,
+ "60": 9.63758,
+ "61": 9.87125,
+ "62": 9.75081,
+ "63": 9.67595,
+ "64": 9.82411,
+ "65": 9.59331,
+ "66": 9.62336,
+ "67": 9.73806,
+ "68": 9.60514,
+ "69": 9.29207,
+ "70": 9.41654,
+ "71": 9.78455,
+ "72": 9.71131,
+ "73": 9.61944,
+ "74": 9.44952,
+ "75": 9.23662,
+ "76": 9.51289,
+ "77": 9.57492,
+ "78": 9.56265,
+ "79": 9.31059,
+ "80": 9.35587,
+ "81": 9.46127,
+ "82": 9.55204,
+ "83": 9.53564,
+ "84": 9.35056,
+ "85": 9.40399,
+ "86": 9.65423,
+ "87": 9.23369,
+ "88": 9.49123,
+ "89": 9.22155,
+ "90": 9.41362,
+ "91": 9.39038,
+ "92": 9.37386,
+ "93": 9.35889,
+ "94": 9.51462,
+ "95": 9.42101,
+ "96": 9.33139,
+ "97": 9.2057,
+ "98": 9.49169,
+ "99": 9.29215,
+ "100": 9.35587,
+ "101": 9.24916,
+ "102": 9.24274,
+ "103": 9.07499,
+ "104": 9.167,
+ "105": 9.38153,
+ "106": 9.1528,
+ "107": 9.17022,
+ "108": 9.31151,
+ "109": 9.2892,
+ "110": 9.36552,
+ "111": 9.17649,
+ "112": 9.23484,
+ "113": 9.34677,
+ "114": 9.35195,
+ "115": 9.32715,
+ "116": 9.00134,
+ "117": 9.06272,
+ "118": 9.0661,
+ "119": 9.22262,
+ "120": 9.08427,
+ "121": 9.1967,
+ "122": 9.1377,
+ "123": 9.25714,
+ "124": 9.4538,
+ "125": 9.21282,
+ "126": 9.05617,
+ "127": 9.01707,
+ "128": 9.21395,
+ "129": 8.98223,
+ "130": 9.13951,
+ "131": 9.15587,
+ "132": 9.03259,
+ "133": 8.85537,
+ "134": 9.18631,
+ "135": 8.88197,
+ "136": 9.16455,
+ "137": 9.15593,
+ "138": 9.23207,
+ "139": 9.09438,
+ "140": 8.86495,
+ "141": 9.29818,
+ "142": 9.19802,
+ "143": 9.11031,
+ "144": 9.24085,
+ "145": 9.09498,
+ "146": 8.9806,
+ "147": 8.98659,
+ "148": 9.12289,
+ "149": 9.06089,
+ "150": 9.01563,
+ "151": 8.9177,
+ "152": 8.86757,
+ "153": 9.0586,
+ "154": 9.17496,
+ "155": 9.13498,
+ "156": 9.04401,
+ "157": 9.14668,
+ "158": 9.04417,
+ "159": 9.02829,
+ "160": 8.89249,
+ "161": 9.04657,
+ "162": 8.88992,
+ "163": 8.84359,
+ "164": 8.96687,
+ "165": 8.92736,
+ "166": 8.65654,
+ "167": 8.82859,
+ "168": 8.81621,
+ "169": 8.65852,
+ "170": 9.04153,
+ "171": 8.72392,
+ "172": 8.82151,
+ "173": 8.90764,
+ "174": 8.84703,
+ "175": 8.70995,
+ "176": 8.7632,
+ "177": 8.76235,
+ "178": 8.72527,
+ "179": 8.6426,
+ "180": 8.7332,
+ "181": 8.69473,
+ "182": 8.7222,
+ "183": 9.08064,
+ "184": 8.60837,
+ "185": 8.88407,
+ "186": 8.74154,
+ "187": 8.57625,
+ "188": 8.68079,
+ "189": 8.86579,
+ "190": 8.53438,
+ "191": 8.66566,
+ "192": 8.61335,
+ "193": 8.5767,
+ "194": 8.74972,
+ "195": 8.59833,
+ "196": 8.774,
+ "197": 8.74212,
+ "198": 8.62977,
+ "199": 8.77171,
+ "200": 8.73807,
+ "201": 8.67003,
+ "202": 8.54545,
+ "203": 8.53465,
+ "204": 8.71392,
+ "205": 8.21841,
+ "206": 8.85888,
+ "207": 8.67942,
+ "208": 8.70994,
+ "209": 8.75638,
+ "210": 8.57506,
+ "211": 8.84437,
+ "212": 8.49539,
+ "213": 8.57276,
+ "214": 8.51365,
+ "215": 8.56521,
+ "216": 8.50803,
+ "217": 8.53389,
+ "218": 8.53352,
+ "219": 8.64604,
+ "220": 8.54994,
+ "221": 8.40633,
+ "222": 8.51085,
+ "223": 8.44023,
+ "224": 8.52985,
+ "225": 8.56362,
+ "226": 8.43825,
+ "227": 8.67679,
+ "228": 8.38921,
+ "229": 8.45602,
+ "230": 8.50398,
+ "231": 8.49516,
+ "232": 8.49899,
+ "233": 8.4976,
+ "234": 8.63768,
+ "235": 8.5551,
+ "236": 8.39738,
+ "237": 8.48762,
+ "238": 8.30734,
+ "239": 8.56505,
+ "240": 8.67214,
+ "241": 8.44288,
+ "242": 8.46856,
+ "243": 8.51865,
+ "244": 8.36586,
+ "245": 8.59062,
+ "246": 8.59449,
+ "247": 8.43282,
+ "248": 8.51566,
+ "249": 8.52485,
+ "250": 8.42534,
+ "251": 8.37829,
+ "252": 8.54269,
+ "253": 8.31192,
+ "254": 8.34964,
+ "255": 8.29234,
+ "256": 8.20758,
+ "257": 8.39042,
+ "258": 8.45478,
+ "259": 8.23326,
+ "260": 8.24434,
+ "261": 8.23517,
+ "262": 8.34908,
+ "263": 8.30414,
+ "264": 8.19109,
+ "265": 8.33342,
+ "266": 8.23125,
+ "267": 7.89597,
+ "268": 8.37645,
+ "269": 8.40258,
+ "270": 8.26161,
+ "271": 8.27592,
+ "272": 8.31514,
+ "273": 8.13254,
+ "274": 8.09354,
+ "275": 8.01037,
+ "276": 7.92637,
+ "277": 8.23574,
+ "278": 8.04401,
+ "279": 7.95837,
+ "280": 7.7508,
+ "281": 8.09898,
+ "282": 8.14301,
+ "283": 8.15453,
+ "284": 8.10464,
+ "285": 8.06548,
+ "286": 7.89906,
+ "287": 7.98632,
+ "288": 8.2473,
+ "289": 8.16945,
+ "290": 8.11919,
+ "291": 8.24829,
+ "292": 8.07475,
+ "293": 8.11498,
+ "294": 7.97506,
+ "295": 7.96907,
+ "296": 8.23743,
+ "297": 7.79223,
+ "298": 8.03941,
+ "299": 7.93089,
+ "300": 7.85462,
+ "301": 8.0111,
+ "302": 7.94466,
+ "303": 7.98562,
+ "304": 7.95013,
+ "305": 7.98846,
+ "306": 7.96584,
+ "307": 7.98885,
+ "308": 7.98642,
+ "309": 8.00159,
+ "310": 7.97346,
+ "311": 7.92226,
+ "312": 7.88321,
+ "313": 7.82163,
+ "314": 7.82328,
+ "315": 7.81717,
+ "316": 7.74293,
+ "317": 7.92363,
+ "318": 7.97659,
+ "319": 7.81895,
+ "320": 7.57214,
+ "321": 7.74719,
+ "322": 7.82484,
+ "323": 7.77261,
+ "324": 7.90222,
+ "325": 7.79149,
+ "326": 7.64992,
+ "327": 7.85616,
+ "328": 7.78452,
+ "329": 7.88612,
+ "330": 7.74399,
+ "331": 7.52265,
+ "332": 7.80467,
+ "333": 7.83915,
+ "334": 7.67628,
+ "335": 7.69265,
+ "336": 7.91484,
+ "337": 7.65229,
+ "338": 7.89213,
+ "339": 7.72278,
+ "340": 7.75433,
+ "341": 7.70108,
+ "342": 7.81633,
+ "343": 7.61206,
+ "344": 7.58,
+ "345": 7.60593,
+ "346": 7.45677,
+ "347": 7.55777,
+ "348": 7.68192,
+ "349": 7.57847,
+ "350": 7.6538,
+ "351": 7.74144,
+ "352": 7.70327,
+ "353": 7.49927,
+ "354": 7.74223,
+ "355": 7.76148,
+ "356": 7.77553,
+ "357": 7.81155,
+ "358": 7.59453,
+ "359": 7.54617,
+ "360": 7.62513,
+ "361": 7.54098,
+ "362": 7.76197,
+ "363": 7.58857,
+ "364": 7.5658,
+ "365": 7.6216,
+ "366": 7.29915,
+ "367": 7.54771,
+ "368": 7.43252,
+ "369": 7.34361,
+ "370": 7.45878,
+ "371": 7.45811,
+ "372": 7.64361,
+ "373": 7.52143,
+ "374": 7.43546,
+ "375": 7.52578,
+ "376": 7.33721,
+ "377": 7.23786,
+ "378": 7.53307,
+ "379": 7.48777,
+ "380": 7.38455,
+ "381": 7.46424,
+ "382": 7.28845,
+ "383": 7.26821,
+ "384": 7.40512,
+ "385": 7.38667,
+ "386": 7.22461,
+ "387": 7.41883,
+ "388": 7.2772,
+ "389": 7.43701,
+ "390": 7.2352,
+ "391": 7.64004,
+ "392": 7.33472,
+ "393": 7.41329,
+ "394": 7.47448,
+ "395": 7.43318,
+ "396": 7.28045,
+ "397": 7.22679,
+ "398": 7.41972,
+ "399": 7.15231,
+ "400": 7.29797,
+ "401": 7.35119,
+ "402": 7.39125,
+ "403": 7.27906,
+ "404": 7.29821,
+ "405": 7.26315,
+ "406": 7.21516,
+ "407": 7.3584,
+ "408": 7.18033,
+ "409": 7.16866,
+ "410": 7.31124,
+ "411": 7.21645,
+ "412": 7.19831,
+ "413": 7.23421,
+ "414": 6.91237,
+ "415": 7.32768,
+ "416": 7.42238,
+ "417": 7.02151,
+ "418": 7.26919,
+ "419": 7.0376,
+ "420": 7.40855,
+ "421": 7.17064,
+ "422": 7.23689,
+ "423": 7.09688,
+ "424": 7.24526,
+ "425": 7.31263,
+ "426": 7.2949,
+ "427": 7.13033,
+ "428": 7.09544,
+ "429": 6.8804,
+ "430": 7.20014,
+ "431": 7.00178,
+ "432": 7.23191,
+ "433": 6.97259,
+ "434": 6.95831,
+ "435": 7.01715,
+ "436": 7.00886,
+ "437": 6.98948,
+ "438": 7.00141,
+ "439": 6.9364,
+ "440": 7.05737,
+ "441": 7.04133,
+ "442": 7.09722,
+ "443": 7.09138,
+ "444": 6.70981,
+ "445": 6.99517,
+ "446": 7.14181,
+ "447": 7.12793,
+ "448": 6.98231,
+ "449": 7.04588,
+ "450": 7.01289,
+ "451": 6.82986,
+ "452": 6.9101,
+ "453": 7.01874,
+ "454": 6.97055,
+ "455": 7.03675,
+ "456": 6.99358,
+ "457": 6.96959,
+ "458": 6.91162,
+ "459": 6.69208,
+ "460": 7.06149,
+ "461": 7.09495,
+ "462": 6.876,
+ "463": 7.05431,
+ "464": 6.64886,
+ "465": 7.02941,
+ "466": 7.00579,
+ "467": 7.00059,
+ "468": 6.95229,
+ "469": 6.82693,
+ "470": 7.04402,
+ "471": 6.87986,
+ "472": 6.95724,
+ "473": 6.82253,
+ "474": 6.97817,
+ "475": 7.16298,
+ "476": 6.76684,
+ "477": 6.89969,
+ "478": 6.91175,
+ "479": 6.70422,
+ "480": 7.0314,
+ "481": 6.99323,
+ "482": 6.73485,
+ "483": 6.77932,
+ "484": 6.75374,
+ "485": 6.93119,
+ "486": 7.06935,
+ "487": 6.6359,
+ "488": 6.89199,
+ "489": 6.76911,
+ "490": 6.82327,
+ "491": 6.70985,
+ "492": 6.69649,
+ "493": 6.76345,
+ "494": 6.68067,
+ "495": 6.63183,
+ "496": 6.58778,
+ "497": 6.83964,
+ "498": 6.64917,
+ "499": 6.85366,
+ "500": 6.65742,
+ "501": 6.73102,
+ "502": 6.83786,
+ "503": 6.71362,
+ "504": 6.61464,
+ "505": 6.61945,
+ "506": 6.74915,
+ "507": 6.86574,
+ "508": 6.8536,
+ "509": 6.64692,
+ "510": 6.82287,
+ "511": 6.73846,
+ "512": 6.73317,
+ "513": 6.65736,
+ "514": 6.70904,
+ "515": 6.45041,
+ "516": 6.74168,
+ "517": 6.70407,
+ "518": 6.53414,
+ "519": 6.63154,
+ "520": 6.86036,
+ "521": 6.6645,
+ "522": 6.70453,
+ "523": 6.74217,
+ "524": 6.7298,
+ "525": 6.67249,
+ "526": 6.41138,
+ "527": 6.7991,
+ "528": 6.65955,
+ "529": 6.6257,
+ "530": 6.62422,
+ "531": 6.63964,
+ "532": 6.62858,
+ "533": 6.76259,
+ "534": 6.61103,
+ "535": 6.74564,
+ "536": 6.62515,
+ "537": 6.64039,
+ "538": 6.52812,
+ "539": 6.54667,
+ "540": 6.58731,
+ "541": 6.45012,
+ "542": 6.66539,
+ "543": 6.67634,
+ "544": 6.67323,
+ "545": 6.80667,
+ "546": 6.63019,
+ "547": 6.40998,
+ "548": 6.7205,
+ "549": 6.69639,
+ "550": 6.52736,
+ "551": 6.75073,
+ "552": 6.63554,
+ "553": 6.48308,
+ "554": 6.62978,
+ "555": 6.45702,
+ "556": 6.61123,
+ "557": 6.63064,
+ "558": 6.38351,
+ "559": 6.36383,
+ "560": 6.57729,
+ "561": 6.72271,
+ "562": 6.63454,
+ "563": 6.7334,
+ "564": 6.34591,
+ "565": 6.51363,
+ "566": 6.69542,
+ "567": 6.55578,
+ "568": 6.5046,
+ "569": 6.44488,
+ "570": 6.35887,
+ "571": 6.62648,
+ "572": 6.31359,
+ "573": 6.57921,
+ "574": 6.47141,
+ "575": 6.63881,
+ "576": 6.51184,
+ "577": 6.52234,
+ "578": 6.47703,
+ "579": 6.46275,
+ "580": 6.56019,
+ "581": 6.60005,
+ "582": 6.47117,
+ "583": 6.50899,
+ "584": 6.518,
+ "585": 6.42107,
+ "586": 6.40372,
+ "587": 6.45289,
+ "588": 6.56365,
+ "589": 6.6171,
+ "590": 6.28832,
+ "591": 6.66701,
+ "592": 6.25538,
+ "593": 6.47198,
+ "594": 6.37444,
+ "595": 6.35476,
+ "596": 6.25125,
+ "597": 6.18471,
+ "598": 6.45392,
+ "599": 6.39473,
+ "600": 6.4516,
+ "601": 6.25271,
+ "602": 6.5232,
+ "603": 6.51529,
+ "604": 6.37744,
+ "605": 6.48824,
+ "606": 6.30741,
+ "607": 6.52263,
+ "608": 6.66416,
+ "609": 6.17126,
+ "610": 6.5652,
+ "611": 6.39564,
+ "612": 6.5696,
+ "613": 6.41103,
+ "614": 6.19666,
+ "615": 6.39227,
+ "616": 6.35466,
+ "617": 6.37444,
+ "618": 6.44411,
+ "619": 6.13342,
+ "620": 6.40948,
+ "621": 6.45819,
+ "622": 6.39255,
+ "623": 6.57786,
+ "624": 6.36135,
+ "625": 6.27921,
+ "626": 6.30121,
+ "627": 6.4409,
+ "628": 6.2511,
+ "629": 6.58617,
+ "630": 6.36355,
+ "631": 6.34386,
+ "632": 6.30615,
+ "633": 6.25949,
+ "634": 6.31003,
+ "635": 6.54298,
+ "636": 6.24305,
+ "637": 6.6361,
+ "638": 6.02255,
+ "639": 6.27843,
+ "640": 6.29487,
+ "641": 6.20972,
+ "642": 6.28691,
+ "643": 6.46184,
+ "644": 6.25998,
+ "645": 6.24925,
+ "646": 6.4091,
+ "647": 6.33724,
+ "648": 6.35572,
+ "649": 6.34866,
+ "650": 6.48504,
+ "651": 6.32999,
+ "652": 6.2549,
+ "653": 6.3836,
+ "654": 6.45033,
+ "655": 6.5287,
+ "656": 6.32829,
+ "657": 6.43046,
+ "658": 6.2472,
+ "659": 6.15841,
+ "660": 6.39991,
+ "661": 6.16851,
+ "662": 6.28087,
+ "663": 6.37394,
+ "664": 6.33473,
+ "665": 6.39796,
+ "666": 6.16918,
+ "667": 6.20229,
+ "668": 6.23823,
+ "669": 6.22075,
+ "670": 6.24978,
+ "671": 6.2449,
+ "672": 6.4915,
+ "673": 6.34115,
+ "674": 6.29796,
+ "675": 6.38583,
+ "676": 6.3935,
+ "677": 6.31128,
+ "678": 6.28415,
+ "679": 6.24171,
+ "680": 6.29057,
+ "681": 6.20638,
+ "682": 6.09079,
+ "683": 6.27805,
+ "684": 6.32936,
+ "685": 6.30422,
+ "686": 6.14994,
+ "687": 6.28813,
+ "688": 6.20688,
+ "689": 6.61907,
+ "690": 6.1833,
+ "691": 6.18039,
+ "692": 6.27603,
+ "693": 6.14397,
+ "694": 6.23409,
+ "695": 6.3254,
+ "696": 6.11929,
+ "697": 6.15504,
+ "698": 6.23217,
+ "699": 6.46,
+ "700": 6.04944,
+ "701": 6.06231,
+ "702": 6.2477,
+ "703": 6.18536,
+ "704": 6.21502,
+ "705": 6.12813,
+ "706": 6.07627,
+ "707": 6.25464,
+ "708": 6.31229,
+ "709": 6.00945,
+ "710": 6.16066,
+ "711": 6.25946,
+ "712": 6.1767,
+ "713": 5.89714,
+ "714": 6.10906,
+ "715": 6.11283,
+ "716": 6.41225,
+ "717": 6.18996,
+ "718": 6.23857,
+ "719": 6.27495,
+ "720": 6.26467,
+ "721": 6.25895,
+ "722": 6.23045,
+ "723": 6.07642,
+ "724": 6.22163,
+ "725": 6.03553,
+ "726": 6.29278,
+ "727": 6.01355,
+ "728": 6.04231,
+ "729": 6.08528,
+ "730": 6.17955,
+ "731": 6.09694,
+ "732": 6.07671,
+ "733": 6.12148,
+ "734": 6.37932,
+ "735": 6.27404,
+ "736": 6.17684,
+ "737": 6.36191,
+ "738": 6.12713,
+ "739": 6.14064,
+ "740": 5.87683,
+ "741": 5.99895,
+ "742": 5.9802,
+ "743": 6.166,
+ "744": 6.02464,
+ "745": 6.14584,
+ "746": 6.02816,
+ "747": 6.09297,
+ "748": 6.23045,
+ "749": 5.93067,
+ "750": 6.16137,
+ "751": 5.95501,
+ "752": 6.01701,
+ "753": 6.02102,
+ "754": 6.28656,
+ "755": 6.12756,
+ "756": 6.24534,
+ "757": 6.01421,
+ "758": 6.19481,
+ "759": 6.228,
+ "760": 6.02212,
+ "761": 6.18616,
+ "762": 6.21825,
+ "763": 6.02955,
+ "764": 5.96304,
+ "765": 5.92794,
+ "766": 5.96451,
+ "767": 5.81062,
+ "768": 6.18351,
+ "769": 6.26815,
+ "770": 6.2949,
+ "771": 5.78883,
+ "772": 6.02652,
+ "773": 6.17803,
+ "774": 5.87992,
+ "775": 6.02251,
+ "776": 6.12312,
+ "777": 5.87055,
+ "778": 6.05512,
+ "779": 5.8652,
+ "780": 6.13125,
+ "781": 5.84904,
+ "782": 6.03507,
+ "783": 5.94974,
+ "784": 5.91562,
+ "785": 6.09086,
+ "786": 6.10224,
+ "787": 5.65052,
+ "788": 5.98902,
+ "789": 6.21372,
+ "790": 6.26182,
+ "791": 5.78643,
+ "792": 5.99308,
+ "793": 6.17873,
+ "794": 6.02381,
+ "795": 6.0009,
+ "796": 6.16815,
+ "797": 6.05307,
+ "798": 6.05663,
+ "799": 6.11432,
+ "800": 6.01903,
+ "801": 6.15067,
+ "802": 5.98207,
+ "803": 6.15292,
+ "804": 6.01022,
+ "805": 5.82965,
+ "806": 6.09236,
+ "807": 6.04583,
+ "808": 5.92863,
+ "809": 5.77625,
+ "810": 6.01696,
+ "811": 5.92794,
+ "812": 5.90736,
+ "813": 5.96248,
+ "814": 6.03002,
+ "815": 5.81091,
+ "816": 6.11984,
+ "817": 5.93352,
+ "818": 6.06723,
+ "819": 6.01557,
+ "820": 5.73308,
+ "821": 5.94831,
+ "822": 6.20282,
+ "823": 5.82909,
+ "824": 5.98262,
+ "825": 6.19037,
+ "826": 6.19845,
+ "827": 6.05575,
+ "828": 6.06712,
+ "829": 5.8901,
+ "830": 5.93562,
+ "831": 5.89391,
+ "832": 5.96546,
+ "833": 6.06602,
+ "834": 5.99706,
+ "835": 6.00082,
+ "836": 5.79923,
+ "837": 6.11349,
+ "838": 5.86588,
+ "839": 5.8334,
+ "840": 6.18812,
+ "841": 5.78477,
+ "842": 5.89293,
+ "843": 5.95055,
+ "844": 6.01129,
+ "845": 6.0967,
+ "846": 5.69142,
+ "847": 5.75814,
+ "848": 5.96826,
+ "849": 6.09509,
+ "850": 5.85002,
+ "851": 6.01777,
+ "852": 5.75629,
+ "853": 5.99646,
+ "854": 6.02059,
+ "855": 5.82073,
+ "856": 6.00406,
+ "857": 6.00932,
+ "858": 6.06128,
+ "859": 5.95771,
+ "860": 6.10304,
+ "861": 6.06921,
+ "862": 6.009,
+ "863": 5.83657,
+ "864": 5.84414,
+ "865": 5.93978,
+ "866": 5.89258,
+ "867": 5.87809,
+ "868": 6.06592,
+ "869": 6.08771,
+ "870": 5.97318,
+ "871": 6.05153,
+ "872": 5.90559,
+ "873": 5.84901,
+ "874": 6.0331,
+ "875": 5.9168,
+ "876": 5.97181,
+ "877": 5.93555,
+ "878": 6.10049,
+ "879": 5.76881,
+ "880": 6.01695,
+ "881": 5.99451,
+ "882": 5.91611,
+ "883": 5.68026,
+ "884": 5.96702,
+ "885": 5.75121,
+ "886": 5.9986,
+ "887": 5.90913,
+ "888": 5.84411,
+ "889": 6.01633,
+ "890": 6.0262,
+ "891": 5.95527,
+ "892": 5.71056,
+ "893": 6.09676,
+ "894": 5.73524,
+ "895": 5.84694,
+ "896": 5.84154,
+ "897": 5.85949,
+ "898": 5.93308,
+ "899": 5.93695,
+ "900": 5.90022,
+ "901": 5.95513,
+ "902": 5.83952,
+ "903": 6.0587,
+ "904": 5.93343,
+ "905": 5.9082,
+ "906": 5.62302,
+ "907": 5.91578,
+ "908": 5.74145,
+ "909": 5.99412,
+ "910": 5.86986,
+ "911": 5.70717,
+ "912": 5.70987,
+ "913": 5.76748,
+ "914": 5.84075,
+ "915": 5.80821,
+ "916": 5.89408,
+ "917": 5.87426,
+ "918": 5.82952,
+ "919": 5.81168,
+ "920": 5.8993,
+ "921": 5.84794,
+ "922": 5.62795,
+ "923": 6.04302,
+ "924": 5.61237,
+ "925": 5.62736,
+ "926": 5.86319,
+ "927": 5.96738,
+ "928": 5.84354,
+ "929": 5.83087,
+ "930": 5.96535,
+ "931": 5.76597,
+ "932": 5.59891,
+ "933": 5.63957,
+ "934": 5.8205,
+ "935": 5.64545,
+ "936": 5.84111,
+ "937": 5.96883,
+ "938": 5.59656,
+ "939": 5.79189,
+ "940": 5.97017,
+ "941": 5.73702,
+ "942": 5.83765,
+ "943": 5.87247,
+ "944": 5.96218,
+ "945": 5.71079,
+ "946": 5.5637,
+ "947": 5.75043,
+ "948": 5.79983,
+ "949": 5.83499,
+ "950": 5.84923,
+ "951": 5.73498,
+ "952": 5.70548,
+ "953": 5.69015,
+ "954": 5.73749,
+ "955": 5.53655,
+ "956": 5.63017,
+ "957": 5.84852,
+ "958": 5.80511,
+ "959": 5.58144,
+ "960": 5.80828,
+ "961": 5.8399,
+ "962": 5.77225,
+ "963": 5.77503,
+ "964": 5.71236,
+ "965": 5.64535,
+ "966": 5.60606,
+ "967": 5.73397,
+ "968": 5.74907,
+ "969": 5.83006,
+ "970": 5.6489,
+ "971": 5.71288,
+ "972": 5.85777,
+ "973": 5.67714,
+ "974": 5.72173,
+ "975": 5.86986,
+ "976": 5.71206,
+ "977": 5.78066,
+ "978": 5.69449,
+ "979": 5.60025,
+ "980": 5.7722,
+ "981": 5.90369,
+ "982": 5.47962,
+ "983": 5.62216,
+ "984": 5.55359,
+ "985": 5.60004,
+ "986": 5.64813,
+ "987": 5.57505,
+ "988": 5.7177,
+ "989": 5.7022,
+ "990": 5.6332,
+ "991": 5.85364,
+ "992": 5.78444,
+ "993": 5.87798,
+ "994": 5.70543,
+ "995": 5.74369,
+ "996": 5.74633,
+ "997": 5.82124,
+ "998": 5.84086,
+ "999": 5.83814,
+ "1000": 5.69402,
+ "1001": 5.87418,
+ "1002": 5.76533,
+ "1003": 5.64881,
+ "1004": 5.8054,
+ "1005": 5.53793,
+ "1006": 5.334,
+ "1007": 5.77509,
+ "1008": 5.80209,
+ "1009": 5.65964,
+ "1010": 5.78786,
+ "1011": 5.9054,
+ "1012": 5.62874,
+ "1013": 5.62568,
+ "1014": 5.68589,
+ "1015": 5.57084,
+ "1016": 5.87727,
+ "1017": 5.8371,
+ "1018": 5.62997,
+ "1019": 5.73948,
+ "1020": 5.62532,
+ "1021": 5.86004,
+ "1022": 5.50514,
+ "1023": 5.66005,
+ "1024": 5.7559,
+ "1025": 5.57673,
+ "1026": 5.4182,
+ "1027": 5.60256,
+ "1028": 5.69511,
+ "1029": 5.69179,
+ "1030": 5.68816,
+ "1031": 5.40936,
+ "1032": 5.7907,
+ "1033": 5.5856,
+ "1034": 5.61919,
+ "1035": 5.72414,
+ "1036": 5.63275,
+ "1037": 5.37065,
+ "1038": 5.67066,
+ "1039": 5.65023,
+ "1040": 5.57799,
+ "1041": 5.59973,
+ "1042": 5.8181,
+ "1043": 5.56835,
+ "1044": 5.46977,
+ "1045": 5.97445,
+ "1046": 5.49021,
+ "1047": 5.39414,
+ "1048": 5.50449,
+ "1049": 5.68011,
+ "1050": 5.70162,
+ "1051": 5.58173,
+ "1052": 5.68957,
+ "1053": 5.63562,
+ "1054": 5.46169,
+ "1055": 5.60615,
+ "1056": 5.67902,
+ "1057": 5.76771,
+ "1058": 5.57232,
+ "1059": 5.7561,
+ "1060": 5.82572,
+ "1061": 5.47841,
+ "1062": 5.65589,
+ "1063": 5.50438,
+ "1064": 5.59374,
+ "1065": 5.5614,
+ "1066": 5.74963,
+ "1067": 5.67567,
+ "1068": 5.44388,
+ "1069": 5.60782,
+ "1070": 5.80997,
+ "1071": 5.51056,
+ "1072": 5.62289,
+ "1073": 5.6244,
+ "1074": 5.53323,
+ "1075": 5.70747,
+ "1076": 5.59922,
+ "1077": 5.71036,
+ "1078": 5.56598,
+ "1079": 5.62418,
+ "1080": 5.64709,
+ "1081": 5.62679,
+ "1082": 5.50781,
+ "1083": 5.64144,
+ "1084": 5.55511,
+ "1085": 5.41018,
+ "1086": 5.6229,
+ "1087": 5.44723,
+ "1088": 5.5159,
+ "1089": 5.76847,
+ "1090": 5.53231,
+ "1091": 5.51723,
+ "1092": 5.41343,
+ "1093": 5.70706,
+ "1094": 5.57366,
+ "1095": 5.58192,
+ "1096": 5.61858,
+ "1097": 5.64905,
+ "1098": 5.6497,
+ "1099": 5.51923,
+ "1100": 5.64343,
+ "1101": 5.67207,
+ "1102": 5.5348,
+ "1103": 5.54936,
+ "1104": 5.53445,
+ "1105": 5.54972,
+ "1106": 5.68285,
+ "1107": 5.67968,
+ "1108": 5.78702,
+ "1109": 5.53781,
+ "1110": 5.66658,
+ "1111": 5.5853,
+ "1112": 5.58696,
+ "1113": 5.6248,
+ "1114": 5.61665,
+ "1115": 5.60394,
+ "1116": 5.66931,
+ "1117": 5.65294,
+ "1118": 5.6513,
+ "1119": 5.71341,
+ "1120": 5.63125,
+ "1121": 5.37808,
+ "1122": 5.23783,
+ "1123": 5.47492,
+ "1124": 5.65309,
+ "1125": 5.67755,
+ "1126": 5.68394,
+ "1127": 5.57225,
+ "1128": 5.62874,
+ "1129": 5.29763,
+ "1130": 5.54207,
+ "1131": 5.62711,
+ "1132": 5.72601,
+ "1133": 5.51579,
+ "1134": 5.55996,
+ "1135": 5.51866,
+ "1136": 5.42473,
+ "1137": 5.4603,
+ "1138": 5.57216,
+ "1139": 5.42222,
+ "1140": 5.2653,
+ "1141": 5.58406,
+ "1142": 5.65283,
+ "1143": 5.39219,
+ "1144": 5.38668,
+ "1145": 5.36858,
+ "1146": 5.63453,
+ "1147": 5.49212,
+ "1148": 5.50407,
+ "1149": 5.52472,
+ "1150": 5.40059,
+ "1151": 5.5633,
+ "1152": 5.4077,
+ "1153": 5.44953,
+ "1154": 5.49407,
+ "1155": 5.43372,
+ "1156": 5.34105,
+ "1157": 5.65908,
+ "1158": 5.39474,
+ "1159": 5.33222,
+ "1160": 5.79735,
+ "1161": 5.53982,
+ "1162": 5.46,
+ "1163": 5.52543,
+ "1164": 5.38165,
+ "1165": 5.52483,
+ "1166": 5.4861,
+ "1167": 5.35734,
+ "1168": 5.49417,
+ "1169": 5.40168,
+ "1170": 5.58332,
+ "1171": 5.48072,
+ "1172": 5.63759,
+ "1173": 5.62365,
+ "1174": 5.50814,
+ "1175": 5.34243,
+ "1176": 5.38347,
+ "1177": 5.55244,
+ "1178": 5.47077,
+ "1179": 5.49471,
+ "1180": 5.46282,
+ "1181": 5.556,
+ "1182": 5.5984,
+ "1183": 5.76846,
+ "1184": 5.54697,
+ "1185": 5.29486,
+ "1186": 5.61075,
+ "1187": 5.55601,
+ "1188": 5.51435,
+ "1189": 5.39259,
+ "1190": 5.40303,
+ "1191": 5.38902,
+ "1192": 5.49771,
+ "1193": 5.45866,
+ "1194": 5.45071,
+ "1195": 5.32527,
+ "1196": 5.5226,
+ "1197": 5.47727,
+ "1198": 5.52522,
+ "1199": 5.38425,
+ "1200": 5.32625,
+ "1201": 5.49117,
+ "1202": 5.43526,
+ "1203": 5.49346,
+ "1204": 5.40879,
+ "1205": 5.48869,
+ "1206": 5.3355,
+ "1207": 5.58732,
+ "1208": 5.42463,
+ "1209": 5.29424,
+ "1210": 5.50158,
+ "1211": 5.50886,
+ "1212": 5.59678,
+ "1213": 5.418,
+ "1214": 5.50521,
+ "1215": 5.23695,
+ "1216": 5.40992,
+ "1217": 5.38788,
+ "1218": 5.45377,
+ "1219": 5.48218,
+ "1220": 5.38537,
+ "1221": 5.44824,
+ "1222": 5.31298,
+ "1223": 5.4764,
+ "1224": 5.41777,
+ "1225": 5.43316,
+ "1226": 5.32921,
+ "1227": 5.47979,
+ "1228": 5.73278,
+ "1229": 5.32505,
+ "1230": 5.40274,
+ "1231": 5.06713,
+ "1232": 5.78083,
+ "1233": 5.28378,
+ "1234": 5.24192,
+ "1235": 5.36643,
+ "1236": 5.48261,
+ "1237": 5.2079,
+ "1238": 5.41066,
+ "1239": 5.40853,
+ "1240": 5.46343,
+ "1241": 5.56733,
+ "1242": 5.45588,
+ "1243": 5.43018,
+ "1244": 5.50964,
+ "1245": 5.19262,
+ "1246": 5.71677,
+ "1247": 5.42453,
+ "1248": 5.29758,
+ "1249": 5.39919,
+ "1250": 5.33901,
+ "1251": 5.42088,
+ "1252": 5.57236,
+ "1253": 5.48811,
+ "1254": 5.30492,
+ "1255": 5.51282,
+ "1256": 5.6017,
+ "1257": 5.42126,
+ "1258": 5.56333,
+ "1259": 5.47966,
+ "1260": 5.50662,
+ "1261": 5.63771,
+ "1262": 5.39318,
+ "1263": 5.32584,
+ "1264": 5.50065,
+ "1265": 5.30533,
+ "1266": 5.23377,
+ "1267": 5.3718,
+ "1268": 5.39137,
+ "1269": 5.14769,
+ "1270": 5.40135,
+ "1271": 5.27211,
+ "1272": 5.5236,
+ "1273": 5.29448,
+ "1274": 5.345,
+ "1275": 5.3815,
+ "1276": 5.39647,
+ "1277": 5.45663,
+ "1278": 5.35334,
+ "1279": 5.43796,
+ "1280": 5.45772,
+ "1281": 5.3974,
+ "1282": 5.38022,
+ "1283": 5.42039,
+ "1284": 5.34663,
+ "1285": 5.49928,
+ "1286": 5.33212,
+ "1287": 5.58195,
+ "1288": 5.26302,
+ "1289": 5.42667,
+ "1290": 5.50073,
+ "1291": 5.50009,
+ "1292": 5.44594,
+ "1293": 5.40963,
+ "1294": 5.49454,
+ "1295": 5.34476,
+ "1296": 5.1855,
+ "1297": 5.16492,
+ "1298": 5.11631,
+ "1299": 5.30294,
+ "1300": 5.20553,
+ "1301": 5.29907,
+ "1302": 5.27097,
+ "1303": 5.35459,
+ "1304": 5.43511,
+ "1305": 5.36558,
+ "1306": 5.25007,
+ "1307": 5.18929,
+ "1308": 5.26544,
+ "1309": 5.40479,
+ "1310": 5.26439,
+ "1311": 5.37844,
+ "1312": 5.35656,
+ "1313": 5.30133,
+ "1314": 5.28969,
+ "1315": 5.41714,
+ "1316": 5.25975,
+ "1317": 5.27409,
+ "1318": 5.21355,
+ "1319": 5.34189,
+ "1320": 5.41529,
+ "1321": 5.44329,
+ "1322": 5.46414,
+ "1323": 5.37164,
+ "1324": 5.25174,
+ "1325": 5.40942,
+ "1326": 5.54178,
+ "1327": 5.39267,
+ "1328": 5.21706,
+ "1329": 5.41338,
+ "1330": 5.40074,
+ "1331": 5.31537,
+ "1332": 5.30719,
+ "1333": 5.37234,
+ "1334": 5.44449,
+ "1335": 5.3694,
+ "1336": 5.43918,
+ "1337": 5.46929,
+ "1338": 5.30204,
+ "1339": 5.13644,
+ "1340": 5.42018,
+ "1341": 5.34688,
+ "1342": 5.35642,
+ "1343": 5.47961,
+ "1344": 5.3761,
+ "1345": 5.34482,
+ "1346": 5.07959,
+ "1347": 5.38337,
+ "1348": 5.49009,
+ "1349": 5.40638,
+ "1350": 5.02353,
+ "1351": 5.31354,
+ "1352": 5.15867,
+ "1353": 5.34566,
+ "1354": 5.36104,
+ "1355": 5.11187,
+ "1356": 5.2573,
+ "1357": 5.29489,
+ "1358": 5.15831,
+ "1359": 5.11053,
+ "1360": 5.17879,
+ "1361": 5.30462,
+ "1362": 5.07076,
+ "1363": 5.29541,
+ "1364": 5.40068,
+ "1365": 5.02082,
+ "1366": 5.12121,
+ "1367": 5.33127,
+ "1368": 5.1807,
+ "1369": 5.22632,
+ "1370": 5.19693,
+ "1371": 5.2783,
+ "1372": 5.26174,
+ "1373": 5.28837,
+ "1374": 5.28073,
+ "1375": 5.45953,
+ "1376": 5.26993,
+ "1377": 5.27043,
+ "1378": 5.31466,
+ "1379": 5.22691,
+ "1380": 5.25909,
+ "1381": 5.48226,
+ "1382": 5.08687,
+ "1383": 5.37543,
+ "1384": 5.35954,
+ "1385": 5.39176,
+ "1386": 5.16586,
+ "1387": 5.16194,
+ "1388": 5.27291,
+ "1389": 5.30103,
+ "1390": 5.2539,
+ "1391": 5.27257,
+ "1392": 5.37635,
+ "1393": 5.37995,
+ "1394": 5.39995,
+ "1395": 5.31822,
+ "1396": 5.21434,
+ "1397": 5.28255,
+ "1398": 5.36906,
+ "1399": 5.35874,
+ "1400": 5.26408,
+ "1401": 5.36023,
+ "1402": 5.42192,
+ "1403": 5.20099,
+ "1404": 5.28133,
+ "1405": 5.11758,
+ "1406": 4.99072,
+ "1407": 5.40067,
+ "1408": 5.1894,
+ "1409": 5.39769,
+ "1410": 5.37602,
+ "1411": 4.91625,
+ "1412": 5.35423,
+ "1413": 5.4132,
+ "1414": 5.2191,
+ "1415": 5.43804,
+ "1416": 5.32427,
+ "1417": 5.39034,
+ "1418": 5.30273,
+ "1419": 5.31784,
+ "1420": 5.4374,
+ "1421": 5.39669,
+ "1422": 5.41845,
+ "1423": 4.99463,
+ "1424": 5.32366,
+ "1425": 5.58064,
+ "1426": 5.22663,
+ "1427": 5.31887,
+ "1428": 5.33335,
+ "1429": 5.07656,
+ "1430": 5.33259,
+ "1431": 5.3211,
+ "1432": 5.33924,
+ "1433": 5.18817,
+ "1434": 5.15901,
+ "1435": 5.19818,
+ "1436": 5.10673,
+ "1437": 5.23248,
+ "1438": 5.31333,
+ "1439": 5.34976,
+ "1440": 5.35009,
+ "1441": 5.1727,
+ "1442": 5.22176,
+ "1443": 5.20955,
+ "1444": 5.13601,
+ "1445": 5.07486,
+ "1446": 5.26539,
+ "1447": 5.26079,
+ "1448": 5.29069,
+ "1449": 5.2476,
+ "1450": 5.34043,
+ "1451": 5.06718,
+ "1452": 5.27088,
+ "1453": 5.16379,
+ "1454": 5.00911,
+ "1455": 5.12305,
+ "1456": 5.27022,
+ "1457": 5.18141,
+ "1458": 5.00413,
+ "1459": 5.21669,
+ "1460": 5.23064,
+ "1461": 5.07378,
+ "1462": 4.97145,
+ "1463": 5.1419,
+ "1464": 5.21707,
+ "1465": 5.27398,
+ "1466": 5.34957,
+ "1467": 5.34204,
+ "1468": 5.2172,
+ "1469": 5.04609,
+ "1470": 5.12218,
+ "1471": 5.25019,
+ "1472": 5.11625,
+ "1473": 5.10478,
+ "1474": 5.21201,
+ "1475": 5.1905,
+ "1476": 5.14959,
+ "1477": 5.25855,
+ "1478": 5.29622,
+ "1479": 5.00591,
+ "1480": 5.18019,
+ "1481": 5.24574,
+ "1482": 5.34899,
+ "1483": 5.26704,
+ "1484": 4.92527,
+ "1485": 5.28848,
+ "1486": 5.04422,
+ "1487": 4.88652,
+ "1488": 5.17721,
+ "1489": 5.1003,
+ "1490": 5.04103,
+ "1491": 5.31899,
+ "1492": 5.21967,
+ "1493": 4.93998,
+ "1494": 5.10845,
+ "1495": 5.13162,
+ "1496": 5.05318,
+ "1497": 5.36199,
+ "1498": 5.30651,
+ "1499": 5.13637,
+ "1500": 5.08942,
+ "1501": 5.03395,
+ "1502": 5.14879,
+ "1503": 5.42794,
+ "1504": 5.32157,
+ "1505": 4.99954,
+ "1506": 5.14396,
+ "1507": 5.16087,
+ "1508": 5.16227,
+ "1509": 5.31186,
+ "1510": 5.01477,
+ "1511": 5.11308,
+ "1512": 4.97622,
+ "1513": 5.16776,
+ "1514": 5.33042,
+ "1515": 5.36454,
+ "1516": 5.27296,
+ "1517": 5.22177,
+ "1518": 5.03252,
+ "1519": 5.29496,
+ "1520": 5.14407,
+ "1521": 5.15297,
+ "1522": 5.32967,
+ "1523": 5.24054,
+ "1524": 5.06788,
+ "1525": 5.20529,
+ "1526": 5.277,
+ "1527": 5.25166,
+ "1528": 5.23654,
+ "1529": 5.18018,
+ "1530": 5.23429,
+ "1531": 5.09197,
+ "1532": 5.15325,
+ "1533": 5.0532,
+ "1534": 5.21651,
+ "1535": 5.16071,
+ "1536": 5.09387,
+ "1537": 5.02036,
+ "1538": 4.90979,
+ "1539": 5.23348,
+ "1540": 5.10515,
+ "1541": 5.25084,
+ "1542": 5.23362,
+ "1543": 5.04888,
+ "1544": 5.07097,
+ "1545": 5.11103,
+ "1546": 5.32057,
+ "1547": 5.10258,
+ "1548": 5.22951,
+ "1549": 5.22917,
+ "1550": 4.97153,
+ "1551": 5.25348,
+ "1552": 5.0173,
+ "1553": 5.14292,
+ "1554": 5.10336,
+ "1555": 5.10505,
+ "1556": 5.19641,
+ "1557": 5.08413,
+ "1558": 5.22735,
+ "1559": 4.996,
+ "1560": 5.10765,
+ "1561": 5.13997,
+ "1562": 5.17742,
+ "1563": 5.23861,
+ "1564": 5.25897,
+ "1565": 5.08666,
+ "1566": 5.29242,
+ "1567": 5.03841,
+ "1568": 5.07974,
+ "1569": 5.19107,
+ "1570": 5.1668,
+ "1571": 4.95043,
+ "1572": 5.03803,
+ "1573": 5.02328,
+ "1574": 4.99131,
+ "1575": 5.23091,
+ "1576": 5.21325,
+ "1577": 5.12617,
+ "1578": 5.35983,
+ "1579": 4.93831,
+ "1580": 5.11904,
+ "1581": 5.09122,
+ "1582": 5.27534,
+ "1583": 5.04106,
+ "1584": 5.05402,
+ "1585": 5.11778,
+ "1586": 5.30025,
+ "1587": 5.1323,
+ "1588": 5.21885,
+ "1589": 4.83208,
+ "1590": 5.0969,
+ "1591": 5.17902,
+ "1592": 5.13907,
+ "1593": 5.22864,
+ "1594": 5.11502,
+ "1595": 5.10583,
+ "1596": 5.19048,
+ "1597": 5.11256,
+ "1598": 5.15547,
+ "1599": 5.1795,
+ "1600": 4.87112,
+ "1601": 5.11454,
+ "1602": 5.22681,
+ "1603": 5.19152,
+ "1604": 5.05075,
+ "1605": 5.03088,
+ "1606": 4.98497,
+ "1607": 5.06776,
+ "1608": 4.97889,
+ "1609": 5.07133,
+ "1610": 5.04289,
+ "1611": 4.99385,
+ "1612": 4.75135,
+ "1613": 5.03149,
+ "1614": 4.87997,
+ "1615": 5.07009,
+ "1616": 5.22999,
+ "1617": 5.06371,
+ "1618": 4.98519,
+ "1619": 5.18111,
+ "1620": 5.14617,
+ "1621": 5.30906,
+ "1622": 5.05531,
+ "1623": 5.1397,
+ "1624": 5.13322,
+ "1625": 5.11798,
+ "1626": 5.10466,
+ "1627": 5.10955,
+ "1628": 5.06495,
+ "1629": 4.92858,
+ "1630": 5.06407,
+ "1631": 5.0577,
+ "1632": 5.09896,
+ "1633": 4.96869,
+ "1634": 4.92311,
+ "1635": 5.05114,
+ "1636": 4.91539,
+ "1637": 5.23361,
+ "1638": 5.15965,
+ "1639": 4.98018,
+ "1640": 5.007,
+ "1641": 5.12904,
+ "1642": 5.08291,
+ "1643": 5.05623,
+ "1644": 5.11789,
+ "1645": 4.97282,
+ "1646": 5.12696,
+ "1647": 5.02919,
+ "1648": 5.18803,
+ "1649": 4.91991,
+ "1650": 5.06172,
+ "1651": 4.92829,
+ "1652": 5.20578,
+ "1653": 5.1568,
+ "1654": 5.1307,
+ "1655": 5.15642,
+ "1656": 5.34015,
+ "1657": 5.21244,
+ "1658": 5.04251,
+ "1659": 4.93088,
+ "1660": 4.81358,
+ "1661": 5.03256,
+ "1662": 5.14774,
+ "1663": 5.15485,
+ "1664": 4.97976,
+ "1665": 5.11491,
+ "1666": 5.10649,
+ "1667": 4.85296,
+ "1668": 5.11422,
+ "1669": 5.07686,
+ "1670": 5.11226,
+ "1671": 5.1695,
+ "1672": 4.77368,
+ "1673": 5.03412,
+ "1674": 4.91696,
+ "1675": 5.04774,
+ "1676": 5.00239,
+ "1677": 4.80224,
+ "1678": 5.02287,
+ "1679": 4.88514,
+ "1680": 5.03694,
+ "1681": 5.06499,
+ "1682": 5.02882,
+ "1683": 4.90765,
+ "1684": 5.06061,
+ "1685": 5.13117,
+ "1686": 5.07174,
+ "1687": 4.97764,
+ "1688": 5.15958,
+ "1689": 5.14521,
+ "1690": 4.9886,
+ "1691": 4.992,
+ "1692": 4.91515,
+ "1693": 5.018,
+ "1694": 4.94207,
+ "1695": 4.91633,
+ "1696": 5.08332,
+ "1697": 5.0476,
+ "1698": 4.94702,
+ "1699": 5.00384,
+ "1700": 4.94729,
+ "1701": 5.16183,
+ "1702": 5.07762,
+ "1703": 5.17188,
+ "1704": 5.14145,
+ "1705": 4.9611,
+ "1706": 4.98274,
+ "1707": 4.78645,
+ "1708": 5.03284,
+ "1709": 5.22751,
+ "1710": 5.03095,
+ "1711": 5.18352,
+ "1712": 5.19133,
+ "1713": 5.03509,
+ "1714": 5.05108,
+ "1715": 4.91546,
+ "1716": 4.92973,
+ "1717": 4.85883,
+ "1718": 5.02243,
+ "1719": 5.12377,
+ "1720": 5.02049,
+ "1721": 4.92512,
+ "1722": 5.05608,
+ "1723": 4.92914,
+ "1724": 5.03696,
+ "1725": 5.19017,
+ "1726": 5.07043,
+ "1727": 4.90664,
+ "1728": 5.01729,
+ "1729": 5.04566,
+ "1730": 4.91218,
+ "1731": 4.99771,
+ "1732": 4.91218,
+ "1733": 5.13137,
+ "1734": 4.82925,
+ "1735": 5.21179,
+ "1736": 4.91543,
+ "1737": 4.8628,
+ "1738": 4.97742,
+ "1739": 5.1639,
+ "1740": 4.84108,
+ "1741": 4.77691,
+ "1742": 4.91236,
+ "1743": 5.08564,
+ "1744": 4.97908,
+ "1745": 4.82472,
+ "1746": 4.94923,
+ "1747": 4.86641,
+ "1748": 5.05983,
+ "1749": 4.86161,
+ "1750": 5.01553,
+ "1751": 5.11509,
+ "1752": 4.90477,
+ "1753": 5.09018,
+ "1754": 5.06127,
+ "1755": 4.89975,
+ "1756": 5.01935,
+ "1757": 5.13966,
+ "1758": 4.87018,
+ "1759": 4.94176,
+ "1760": 4.832,
+ "1761": 5.01928,
+ "1762": 4.81517,
+ "1763": 4.7696,
+ "1764": 4.9321,
+ "1765": 5.14559,
+ "1766": 5.34067,
+ "1767": 5.2312,
+ "1768": 4.94529,
+ "1769": 5.00506,
+ "1770": 4.97682,
+ "1771": 4.96597,
+ "1772": 4.97908,
+ "1773": 4.97385,
+ "1774": 4.86722,
+ "1775": 4.95647,
+ "1776": 4.99722,
+ "1777": 4.9422,
+ "1778": 4.99538,
+ "1779": 5.08626,
+ "1780": 4.83609,
+ "1781": 5.05171,
+ "1782": 4.99925,
+ "1783": 5.02024,
+ "1784": 4.93385,
+ "1785": 5.16708,
+ "1786": 4.81332,
+ "1787": 4.97396,
+ "1788": 4.83,
+ "1789": 4.89336,
+ "1790": 4.80383,
+ "1791": 4.74376,
+ "1792": 4.88482,
+ "1793": 5.11054,
+ "1794": 4.98076,
+ "1795": 4.96577,
+ "1796": 4.99995,
+ "1797": 4.78788,
+ "1798": 4.76795,
+ "1799": 5.01672,
+ "1800": 4.91047,
+ "1801": 5.0488,
+ "1802": 4.82415,
+ "1803": 4.95177,
+ "1804": 4.88581,
+ "1805": 4.90389,
+ "1806": 4.87854,
+ "1807": 4.93106,
+ "1808": 4.92639,
+ "1809": 5.14626,
+ "1810": 5.09768,
+ "1811": 4.95353,
+ "1812": 4.79806,
+ "1813": 5.0943,
+ "1814": 4.77678,
+ "1815": 4.86153,
+ "1816": 5.04893,
+ "1817": 4.7919,
+ "1818": 4.80647,
+ "1819": 5.02586,
+ "1820": 4.68782,
+ "1821": 5.03019,
+ "1822": 4.66638,
+ "1823": 4.87419,
+ "1824": 4.79122,
+ "1825": 5.06616,
+ "1826": 4.82316,
+ "1827": 4.79802,
+ "1828": 4.94896,
+ "1829": 5.11651,
+ "1830": 4.92129,
+ "1831": 4.89904,
+ "1832": 4.83208,
+ "1833": 4.78422,
+ "1834": 4.9471,
+ "1835": 4.96349,
+ "1836": 4.90569,
+ "1837": 4.6695,
+ "1838": 4.80745,
+ "1839": 4.89842,
+ "1840": 4.90548,
+ "1841": 4.83249,
+ "1842": 4.95446,
+ "1843": 4.70867,
+ "1844": 4.61514,
+ "1845": 5.00458,
+ "1846": 4.74794,
+ "1847": 4.86801,
+ "1848": 4.90501,
+ "1849": 4.84528,
+ "1850": 4.86973,
+ "1851": 5.01871,
+ "1852": 4.97791,
+ "1853": 4.82871,
+ "1854": 4.86715,
+ "1855": 4.82042,
+ "1856": 4.75206,
+ "1857": 4.96745,
+ "1858": 4.96793,
+ "1859": 4.75215,
+ "1860": 4.86807,
+ "1861": 5.21265,
+ "1862": 4.61926,
+ "1863": 4.83862,
+ "1864": 4.75705,
+ "1865": 4.86523,
+ "1866": 4.79424,
+ "1867": 5.00684,
+ "1868": 4.71545,
+ "1869": 4.76159,
+ "1870": 4.93898,
+ "1871": 5.00036,
+ "1872": 4.68923,
+ "1873": 4.7012,
+ "1874": 4.84675,
+ "1875": 4.8576,
+ "1876": 4.74064,
+ "1877": 4.80806,
+ "1878": 4.81131,
+ "1879": 4.82227,
+ "1880": 4.89321,
+ "1881": 4.78686,
+ "1882": 4.79314,
+ "1883": 4.7817,
+ "1884": 4.97519,
+ "1885": 4.92045,
+ "1886": 4.82384,
+ "1887": 4.81486,
+ "1888": 4.9743,
+ "1889": 4.962,
+ "1890": 4.70832,
+ "1891": 4.65898,
+ "1892": 4.84845,
+ "1893": 4.65039,
+ "1894": 4.90659,
+ "1895": 4.79513,
+ "1896": 4.66,
+ "1897": 4.79205,
+ "1898": 4.91945,
+ "1899": 4.77715,
+ "1900": 4.91894,
+ "1901": 4.84633,
+ "1902": 4.78563,
+ "1903": 4.75799,
+ "1904": 4.65185,
+ "1905": 4.54312,
+ "1906": 4.81032,
+ "1907": 4.90271,
+ "1908": 5.02926,
+ "1909": 4.89131,
+ "1910": 4.79275,
+ "1911": 4.80972,
+ "1912": 4.65355,
+ "1913": 4.94801,
+ "1914": 4.88619,
+ "1915": 4.86016,
+ "1916": 4.92649,
+ "1917": 4.85623,
+ "1918": 4.87995,
+ "1919": 5.0042,
+ "1920": 4.76638,
+ "1921": 4.88975,
+ "1922": 4.81654,
+ "1923": 4.75564,
+ "1924": 4.82656,
+ "1925": 5.05829,
+ "1926": 4.93789,
+ "1927": 4.93326,
+ "1928": 4.93063,
+ "1929": 4.92975,
+ "1930": 4.91117,
+ "1931": 4.77644,
+ "1932": 4.86143,
+ "1933": 4.83825,
+ "1934": 4.84629,
+ "1935": 5.11694,
+ "1936": 4.88798,
+ "1937": 4.87891,
+ "1938": 4.8018,
+ "1939": 4.7078,
+ "1940": 4.82624,
+ "1941": 4.74187,
+ "1942": 4.88262,
+ "1943": 4.74268,
+ "1944": 4.74596,
+ "1945": 4.69162,
+ "1946": 4.92118,
+ "1947": 4.87238,
+ "1948": 4.6008,
+ "1949": 4.89656,
+ "1950": 4.79525,
+ "1951": 4.96278,
+ "1952": 4.73693,
+ "1953": 4.79727,
+ "1954": 4.73869,
+ "1955": 4.84303,
+ "1956": 4.88032,
+ "1957": 4.73464,
+ "1958": 4.70405,
+ "1959": 4.76808,
+ "1960": 4.76914,
+ "1961": 4.71316,
+ "1962": 4.83738,
+ "1963": 4.82535,
+ "1964": 4.85105,
+ "1965": 4.87562,
+ "1966": 4.79067,
+ "1967": 4.59379,
+ "1968": 4.82901,
+ "1969": 4.59402,
+ "1970": 4.57789,
+ "1971": 4.90835,
+ "1972": 4.89673,
+ "1973": 4.55698,
+ "1974": 4.82599,
+ "1975": 4.83122,
+ "1976": 4.71779,
+ "1977": 4.58336,
+ "1978": 5.01205,
+ "1979": 4.67028,
+ "1980": 4.75241,
+ "1981": 4.86344,
+ "1982": 4.72579,
+ "1983": 4.88901,
+ "1984": 4.64561,
+ "1985": 4.7882,
+ "1986": 4.69595,
+ "1987": 4.81318,
+ "1988": 4.8847,
+ "1989": 4.63457,
+ "1990": 4.79224,
+ "1991": 4.70247,
+ "1992": 4.7964,
+ "1993": 4.74467,
+ "1994": 4.85533,
+ "1995": 4.55714,
+ "1996": 4.65583,
+ "1997": 4.81273,
+ "1998": 4.68,
+ "1999": 4.73101,
+ "2000": 4.62846
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 62.0,
+ "2": 72.0,
+ "3": 65.0,
+ "4": 88.0,
+ "5": 66.0,
+ "6": 78.0,
+ "7": 84.0,
+ "8": 70.0,
+ "9": 84.0,
+ "10": 66.0,
+ "11": 54.0,
+ "12": 73.0,
+ "13": 77.0,
+ "14": 93.0,
+ "15": 63.0,
+ "16": 61.0,
+ "17": 69.0,
+ "18": 83.0,
+ "19": 58.0,
+ "20": 85.0,
+ "21": 79.0,
+ "22": 70.0,
+ "23": 80.0,
+ "24": 79.0,
+ "25": 81.0,
+ "26": 96.0,
+ "27": 80.0,
+ "28": 104.0,
+ "29": 115.0,
+ "30": 75.0,
+ "31": 125.0,
+ "32": 113.0,
+ "33": 108.0,
+ "34": 117.0,
+ "35": 97.0,
+ "36": 99.0,
+ "37": 126.0,
+ "38": 131.0,
+ "39": 115.0,
+ "40": 126.0,
+ "41": 114.0,
+ "42": 114.0,
+ "43": 132.0,
+ "44": 108.0,
+ "45": 145.0,
+ "46": 125.0,
+ "47": 134.0,
+ "48": 144.0,
+ "49": 123.0,
+ "50": 173.0,
+ "51": 119.0,
+ "52": 116.0,
+ "53": 114.0,
+ "54": 153.0,
+ "55": 128.0,
+ "56": 161.0,
+ "57": 129.0,
+ "58": 164.0,
+ "59": 131.0,
+ "60": 163.0,
+ "61": 141.0,
+ "62": 147.0,
+ "63": 176.0,
+ "64": 153.0,
+ "65": 152.0,
+ "66": 178.0,
+ "67": 165.0,
+ "68": 144.0,
+ "69": 143.0,
+ "70": 140.0,
+ "71": 175.0,
+ "72": 169.0,
+ "73": 152.0,
+ "74": 134.0,
+ "75": 159.0,
+ "76": 163.0,
+ "77": 183.0,
+ "78": 182.0,
+ "79": 147.0,
+ "80": 177.0,
+ "81": 180.0,
+ "82": 151.0,
+ "83": 155.0,
+ "84": 161.0,
+ "85": 166.0,
+ "86": 176.0,
+ "87": 176.0,
+ "88": 159.0,
+ "89": 164.0,
+ "90": 152.0,
+ "91": 194.0,
+ "92": 180.0,
+ "93": 170.0,
+ "94": 133.0,
+ "95": 154.0,
+ "96": 183.0,
+ "97": 191.0,
+ "98": 185.0,
+ "99": 170.0,
+ "100": 173.0,
+ "101": 191.0,
+ "102": 174.0,
+ "103": 164.0,
+ "104": 177.0,
+ "105": 201.0,
+ "106": 162.0,
+ "107": 163.0,
+ "108": 201.0,
+ "109": 149.0,
+ "110": 186.0,
+ "111": 181.0,
+ "112": 180.0,
+ "113": 167.0,
+ "114": 172.0,
+ "115": 169.0,
+ "116": 138.0,
+ "117": 169.0,
+ "118": 155.0,
+ "119": 184.0,
+ "120": 150.0,
+ "121": 170.0,
+ "122": 153.0,
+ "123": 173.0,
+ "124": 194.0,
+ "125": 196.0,
+ "126": 163.0,
+ "127": 198.0,
+ "128": 158.0,
+ "129": 178.0,
+ "130": 186.0,
+ "131": 197.0,
+ "132": 170.0,
+ "133": 167.0,
+ "134": 166.0,
+ "135": 157.0,
+ "136": 184.0,
+ "137": 152.0,
+ "138": 151.0,
+ "139": 154.0,
+ "140": 168.0,
+ "141": 161.0,
+ "142": 155.0,
+ "143": 197.0,
+ "144": 179.0,
+ "145": 161.0,
+ "146": 185.0,
+ "147": 134.0,
+ "148": 166.0,
+ "149": 182.0,
+ "150": 139.0,
+ "151": 179.0,
+ "152": 125.0,
+ "153": 144.0,
+ "154": 191.0,
+ "155": 149.0,
+ "156": 147.0,
+ "157": 151.0,
+ "158": 160.0,
+ "159": 159.0,
+ "160": 174.0,
+ "161": 122.0,
+ "162": 187.0,
+ "163": 148.0,
+ "164": 186.0,
+ "165": 159.0,
+ "166": 190.0,
+ "167": 144.0,
+ "168": 163.0,
+ "169": 180.0,
+ "170": 170.0,
+ "171": 154.0,
+ "172": 162.0,
+ "173": 162.0,
+ "174": 179.0,
+ "175": 162.0,
+ "176": 157.0,
+ "177": 173.0,
+ "178": 157.0,
+ "179": 175.0,
+ "180": 152.0,
+ "181": 147.0,
+ "182": 182.0,
+ "183": 188.0,
+ "184": 181.0,
+ "185": 179.0,
+ "186": 204.0,
+ "187": 176.0,
+ "188": 178.0,
+ "189": 191.0,
+ "190": 148.0,
+ "191": 177.0,
+ "192": 174.0,
+ "193": 195.0,
+ "194": 178.0,
+ "195": 161.0,
+ "196": 205.0,
+ "197": 173.0,
+ "198": 155.0,
+ "199": 162.0,
+ "200": 176.0,
+ "201": 160.0,
+ "202": 162.0,
+ "203": 168.0,
+ "204": 150.0,
+ "205": 147.0,
+ "206": 170.0,
+ "207": 159.0,
+ "208": 140.0,
+ "209": 177.0,
+ "210": 164.0,
+ "211": 168.0,
+ "212": 177.0,
+ "213": 154.0,
+ "214": 166.0,
+ "215": 149.0,
+ "216": 170.0,
+ "217": 188.0,
+ "218": 189.0,
+ "219": 168.0,
+ "220": 170.0,
+ "221": 176.0,
+ "222": 154.0,
+ "223": 200.0,
+ "224": 184.0,
+ "225": 175.0,
+ "226": 174.0,
+ "227": 194.0,
+ "228": 172.0,
+ "229": 182.0,
+ "230": 172.0,
+ "231": 188.0,
+ "232": 186.0,
+ "233": 197.0,
+ "234": 216.0,
+ "235": 196.0,
+ "236": 158.0,
+ "237": 178.0,
+ "238": 153.0,
+ "239": 161.0,
+ "240": 193.0,
+ "241": 157.0,
+ "242": 189.0,
+ "243": 183.0,
+ "244": 184.0,
+ "245": 177.0,
+ "246": 236.0,
+ "247": 196.0,
+ "248": 186.0,
+ "249": 204.0,
+ "250": 198.0,
+ "251": 195.0,
+ "252": 218.0,
+ "253": 147.0,
+ "254": 207.0,
+ "255": 234.0,
+ "256": 173.0,
+ "257": 193.0,
+ "258": 225.0,
+ "259": 169.0,
+ "260": 167.0,
+ "261": 241.0,
+ "262": 176.0,
+ "263": 191.0,
+ "264": 203.0,
+ "265": 174.0,
+ "266": 169.0,
+ "267": 133.0,
+ "268": 164.0,
+ "269": 205.0,
+ "270": 232.0,
+ "271": 171.0,
+ "272": 195.0,
+ "273": 165.0,
+ "274": 180.0,
+ "275": 155.0,
+ "276": 176.0,
+ "277": 169.0,
+ "278": 196.0,
+ "279": 195.0,
+ "280": 159.0,
+ "281": 142.0,
+ "282": 188.0,
+ "283": 194.0,
+ "284": 190.0,
+ "285": 153.0,
+ "286": 153.0,
+ "287": 220.0,
+ "288": 174.0,
+ "289": 206.0,
+ "290": 187.0,
+ "291": 231.0,
+ "292": 168.0,
+ "293": 219.0,
+ "294": 189.0,
+ "295": 223.0,
+ "296": 233.0,
+ "297": 189.0,
+ "298": 137.0,
+ "299": 214.0,
+ "300": 174.0,
+ "301": 173.0,
+ "302": 208.0,
+ "303": 170.0,
+ "304": 208.0,
+ "305": 148.0,
+ "306": 197.0,
+ "307": 204.0,
+ "308": 173.0,
+ "309": 188.0,
+ "310": 208.0,
+ "311": 179.0,
+ "312": 169.0,
+ "313": 202.0,
+ "314": 222.0,
+ "315": 173.0,
+ "316": 163.0,
+ "317": 162.0,
+ "318": 193.0,
+ "319": 214.0,
+ "320": 214.0,
+ "321": 191.0,
+ "322": 210.0,
+ "323": 147.0,
+ "324": 208.0,
+ "325": 181.0,
+ "326": 218.0,
+ "327": 223.0,
+ "328": 224.0,
+ "329": 201.0,
+ "330": 195.0,
+ "331": 186.0,
+ "332": 197.0,
+ "333": 222.0,
+ "334": 201.0,
+ "335": 204.0,
+ "336": 185.0,
+ "337": 172.0,
+ "338": 194.0,
+ "339": 193.0,
+ "340": 179.0,
+ "341": 198.0,
+ "342": 139.0,
+ "343": 209.0,
+ "344": 229.0,
+ "345": 205.0,
+ "346": 221.0,
+ "347": 217.0,
+ "348": 221.0,
+ "349": 188.0,
+ "350": 219.0,
+ "351": 237.0,
+ "352": 218.0,
+ "353": 234.0,
+ "354": 176.0,
+ "355": 242.0,
+ "356": 214.0,
+ "357": 226.0,
+ "358": 179.0,
+ "359": 226.0,
+ "360": 214.0,
+ "361": 184.0,
+ "362": 274.0,
+ "363": 219.0,
+ "364": 246.0,
+ "365": 208.0,
+ "366": 209.0,
+ "367": 200.0,
+ "368": 148.0,
+ "369": 187.0,
+ "370": 221.0,
+ "371": 195.0,
+ "372": 235.0,
+ "373": 226.0,
+ "374": 181.0,
+ "375": 203.0,
+ "376": 213.0,
+ "377": 179.0,
+ "378": 228.0,
+ "379": 213.0,
+ "380": 194.0,
+ "381": 220.0,
+ "382": 216.0,
+ "383": 233.0,
+ "384": 279.0,
+ "385": 186.0,
+ "386": 212.0,
+ "387": 207.0,
+ "388": 217.0,
+ "389": 194.0,
+ "390": 239.0,
+ "391": 225.0,
+ "392": 209.0,
+ "393": 244.0,
+ "394": 221.0,
+ "395": 234.0,
+ "396": 237.0,
+ "397": 189.0,
+ "398": 191.0,
+ "399": 190.0,
+ "400": 191.0,
+ "401": 236.0,
+ "402": 193.0,
+ "403": 251.0,
+ "404": 213.0,
+ "405": 168.0,
+ "406": 238.0,
+ "407": 220.0,
+ "408": 247.0,
+ "409": 207.0,
+ "410": 211.0,
+ "411": 186.0,
+ "412": 188.0,
+ "413": 181.0,
+ "414": 222.0,
+ "415": 218.0,
+ "416": 214.0,
+ "417": 209.0,
+ "418": 196.0,
+ "419": 185.0,
+ "420": 214.0,
+ "421": 218.0,
+ "422": 173.0,
+ "423": 215.0,
+ "424": 217.0,
+ "425": 218.0,
+ "426": 184.0,
+ "427": 218.0,
+ "428": 171.0,
+ "429": 203.0,
+ "430": 241.0,
+ "431": 158.0,
+ "432": 233.0,
+ "433": 206.0,
+ "434": 179.0,
+ "435": 172.0,
+ "436": 207.0,
+ "437": 178.0,
+ "438": 188.0,
+ "439": 181.0,
+ "440": 186.0,
+ "441": 175.0,
+ "442": 225.0,
+ "443": 238.0,
+ "444": 141.0,
+ "445": 195.0,
+ "446": 209.0,
+ "447": 217.0,
+ "448": 206.0,
+ "449": 253.0,
+ "450": 168.0,
+ "451": 197.0,
+ "452": 176.0,
+ "453": 161.0,
+ "454": 220.0,
+ "455": 238.0,
+ "456": 233.0,
+ "457": 217.0,
+ "458": 182.0,
+ "459": 197.0,
+ "460": 198.0,
+ "461": 226.0,
+ "462": 187.0,
+ "463": 203.0,
+ "464": 188.0,
+ "465": 171.0,
+ "466": 229.0,
+ "467": 215.0,
+ "468": 181.0,
+ "469": 208.0,
+ "470": 215.0,
+ "471": 199.0,
+ "472": 201.0,
+ "473": 202.0,
+ "474": 186.0,
+ "475": 187.0,
+ "476": 164.0,
+ "477": 155.0,
+ "478": 179.0,
+ "479": 205.0,
+ "480": 140.0,
+ "481": 188.0,
+ "482": 223.0,
+ "483": 167.0,
+ "484": 168.0,
+ "485": 204.0,
+ "486": 190.0,
+ "487": 165.0,
+ "488": 203.0,
+ "489": 161.0,
+ "490": 208.0,
+ "491": 168.0,
+ "492": 162.0,
+ "493": 182.0,
+ "494": 177.0,
+ "495": 174.0,
+ "496": 195.0,
+ "497": 160.0,
+ "498": 197.0,
+ "499": 167.0,
+ "500": 163.0,
+ "501": 154.0,
+ "502": 193.0,
+ "503": 173.0,
+ "504": 165.0,
+ "505": 200.0,
+ "506": 171.0,
+ "507": 197.0,
+ "508": 154.0,
+ "509": 169.0,
+ "510": 179.0,
+ "511": 160.0,
+ "512": 188.0,
+ "513": 134.0,
+ "514": 156.0,
+ "515": 164.0,
+ "516": 146.0,
+ "517": 196.0,
+ "518": 174.0,
+ "519": 143.0,
+ "520": 181.0,
+ "521": 141.0,
+ "522": 175.0,
+ "523": 171.0,
+ "524": 174.0,
+ "525": 175.0,
+ "526": 150.0,
+ "527": 180.0,
+ "528": 188.0,
+ "529": 177.0,
+ "530": 174.0,
+ "531": 167.0,
+ "532": 172.0,
+ "533": 174.0,
+ "534": 160.0,
+ "535": 138.0,
+ "536": 178.0,
+ "537": 150.0,
+ "538": 164.0,
+ "539": 137.0,
+ "540": 178.0,
+ "541": 131.0,
+ "542": 188.0,
+ "543": 170.0,
+ "544": 140.0,
+ "545": 183.0,
+ "546": 151.0,
+ "547": 168.0,
+ "548": 188.0,
+ "549": 153.0,
+ "550": 174.0,
+ "551": 183.0,
+ "552": 173.0,
+ "553": 164.0,
+ "554": 156.0,
+ "555": 176.0,
+ "556": 163.0,
+ "557": 163.0,
+ "558": 198.0,
+ "559": 155.0,
+ "560": 179.0,
+ "561": 204.0,
+ "562": 154.0,
+ "563": 180.0,
+ "564": 184.0,
+ "565": 115.0,
+ "566": 176.0,
+ "567": 168.0,
+ "568": 158.0,
+ "569": 135.0,
+ "570": 148.0,
+ "571": 139.0,
+ "572": 148.0,
+ "573": 163.0,
+ "574": 182.0,
+ "575": 164.0,
+ "576": 162.0,
+ "577": 152.0,
+ "578": 175.0,
+ "579": 155.0,
+ "580": 126.0,
+ "581": 170.0,
+ "582": 156.0,
+ "583": 132.0,
+ "584": 149.0,
+ "585": 141.0,
+ "586": 102.0,
+ "587": 143.0,
+ "588": 119.0,
+ "589": 135.0,
+ "590": 123.0,
+ "591": 153.0,
+ "592": 134.0,
+ "593": 136.0,
+ "594": 161.0,
+ "595": 135.0,
+ "596": 151.0,
+ "597": 119.0,
+ "598": 137.0,
+ "599": 162.0,
+ "600": 157.0,
+ "601": 146.0,
+ "602": 143.0,
+ "603": 120.0,
+ "604": 166.0,
+ "605": 163.0,
+ "606": 128.0,
+ "607": 146.0,
+ "608": 152.0,
+ "609": 160.0,
+ "610": 118.0,
+ "611": 130.0,
+ "612": 163.0,
+ "613": 167.0,
+ "614": 142.0,
+ "615": 149.0,
+ "616": 127.0,
+ "617": 114.0,
+ "618": 148.0,
+ "619": 105.0,
+ "620": 157.0,
+ "621": 97.0,
+ "622": 152.0,
+ "623": 128.0,
+ "624": 120.0,
+ "625": 110.0,
+ "626": 135.0,
+ "627": 139.0,
+ "628": 111.0,
+ "629": 141.0,
+ "630": 134.0,
+ "631": 130.0,
+ "632": 123.0,
+ "633": 108.0,
+ "634": 124.0,
+ "635": 134.0,
+ "636": 135.0,
+ "637": 135.0,
+ "638": 105.0,
+ "639": 172.0,
+ "640": 150.0,
+ "641": 144.0,
+ "642": 160.0,
+ "643": 134.0,
+ "644": 148.0,
+ "645": 139.0,
+ "646": 135.0,
+ "647": 143.0,
+ "648": 152.0,
+ "649": 163.0,
+ "650": 143.0,
+ "651": 147.0,
+ "652": 145.0,
+ "653": 155.0,
+ "654": 163.0,
+ "655": 154.0,
+ "656": 138.0,
+ "657": 145.0,
+ "658": 129.0,
+ "659": 125.0,
+ "660": 137.0,
+ "661": 140.0,
+ "662": 153.0,
+ "663": 136.0,
+ "664": 125.0,
+ "665": 151.0,
+ "666": 117.0,
+ "667": 114.0,
+ "668": 141.0,
+ "669": 111.0,
+ "670": 133.0,
+ "671": 143.0,
+ "672": 157.0,
+ "673": 121.0,
+ "674": 111.0,
+ "675": 131.0,
+ "676": 149.0,
+ "677": 107.0,
+ "678": 148.0,
+ "679": 102.0,
+ "680": 167.0,
+ "681": 136.0,
+ "682": 116.0,
+ "683": 141.0,
+ "684": 139.0,
+ "685": 151.0,
+ "686": 146.0,
+ "687": 141.0,
+ "688": 129.0,
+ "689": 159.0,
+ "690": 103.0,
+ "691": 142.0,
+ "692": 118.0,
+ "693": 124.0,
+ "694": 142.0,
+ "695": 120.0,
+ "696": 127.0,
+ "697": 129.0,
+ "698": 132.0,
+ "699": 133.0,
+ "700": 133.0,
+ "701": 129.0,
+ "702": 91.0,
+ "703": 101.0,
+ "704": 134.0,
+ "705": 130.0,
+ "706": 139.0,
+ "707": 120.0,
+ "708": 125.0,
+ "709": 133.0,
+ "710": 127.0,
+ "711": 110.0,
+ "712": 135.0,
+ "713": 95.0,
+ "714": 126.0,
+ "715": 107.0,
+ "716": 165.0,
+ "717": 96.0,
+ "718": 127.0,
+ "719": 105.0,
+ "720": 108.0,
+ "721": 135.0,
+ "722": 105.0,
+ "723": 105.0,
+ "724": 112.0,
+ "725": 133.0,
+ "726": 121.0,
+ "727": 135.0,
+ "728": 127.0,
+ "729": 111.0,
+ "730": 105.0,
+ "731": 125.0,
+ "732": 130.0,
+ "733": 99.0,
+ "734": 151.0,
+ "735": 131.0,
+ "736": 154.0,
+ "737": 127.0,
+ "738": 158.0,
+ "739": 117.0,
+ "740": 133.0,
+ "741": 101.0,
+ "742": 142.0,
+ "743": 115.0,
+ "744": 114.0,
+ "745": 121.0,
+ "746": 111.0,
+ "747": 125.0,
+ "748": 121.0,
+ "749": 113.0,
+ "750": 101.0,
+ "751": 89.0,
+ "752": 106.0,
+ "753": 109.0,
+ "754": 129.0,
+ "755": 102.0,
+ "756": 99.0,
+ "757": 90.0,
+ "758": 105.0,
+ "759": 88.0,
+ "760": 96.0,
+ "761": 106.0,
+ "762": 134.0,
+ "763": 112.0,
+ "764": 99.0,
+ "765": 110.0,
+ "766": 119.0,
+ "767": 105.0,
+ "768": 98.0,
+ "769": 120.0,
+ "770": 108.0,
+ "771": 97.0,
+ "772": 124.0,
+ "773": 107.0,
+ "774": 132.0,
+ "775": 101.0,
+ "776": 109.0,
+ "777": 109.0,
+ "778": 123.0,
+ "779": 117.0,
+ "780": 101.0,
+ "781": 111.0,
+ "782": 80.0,
+ "783": 84.0,
+ "784": 105.0,
+ "785": 113.0,
+ "786": 105.0,
+ "787": 91.0,
+ "788": 122.0,
+ "789": 92.0,
+ "790": 105.0,
+ "791": 97.0,
+ "792": 103.0,
+ "793": 121.0,
+ "794": 117.0,
+ "795": 116.0,
+ "796": 85.0,
+ "797": 119.0,
+ "798": 93.0,
+ "799": 89.0,
+ "800": 87.0,
+ "801": 103.0,
+ "802": 107.0,
+ "803": 102.0,
+ "804": 90.0,
+ "805": 106.0,
+ "806": 91.0,
+ "807": 101.0,
+ "808": 110.0,
+ "809": 103.0,
+ "810": 93.0,
+ "811": 114.0,
+ "812": 118.0,
+ "813": 126.0,
+ "814": 97.0,
+ "815": 100.0,
+ "816": 135.0,
+ "817": 115.0,
+ "818": 113.0,
+ "819": 122.0,
+ "820": 112.0,
+ "821": 105.0,
+ "822": 132.0,
+ "823": 120.0,
+ "824": 114.0,
+ "825": 95.0,
+ "826": 117.0,
+ "827": 133.0,
+ "828": 105.0,
+ "829": 121.0,
+ "830": 121.0,
+ "831": 114.0,
+ "832": 128.0,
+ "833": 126.0,
+ "834": 111.0,
+ "835": 132.0,
+ "836": 108.0,
+ "837": 104.0,
+ "838": 119.0,
+ "839": 106.0,
+ "840": 125.0,
+ "841": 130.0,
+ "842": 110.0,
+ "843": 102.0,
+ "844": 104.0,
+ "845": 117.0,
+ "846": 122.0,
+ "847": 107.0,
+ "848": 109.0,
+ "849": 123.0,
+ "850": 108.0,
+ "851": 136.0,
+ "852": 78.0,
+ "853": 95.0,
+ "854": 96.0,
+ "855": 107.0,
+ "856": 103.0,
+ "857": 113.0,
+ "858": 85.0,
+ "859": 88.0,
+ "860": 96.0,
+ "861": 121.0,
+ "862": 109.0,
+ "863": 114.0,
+ "864": 78.0,
+ "865": 118.0,
+ "866": 96.0,
+ "867": 109.0,
+ "868": 97.0,
+ "869": 107.0,
+ "870": 84.0,
+ "871": 112.0,
+ "872": 98.0,
+ "873": 95.0,
+ "874": 80.0,
+ "875": 94.0,
+ "876": 131.0,
+ "877": 94.0,
+ "878": 110.0,
+ "879": 111.0,
+ "880": 106.0,
+ "881": 123.0,
+ "882": 107.0,
+ "883": 134.0,
+ "884": 103.0,
+ "885": 99.0,
+ "886": 94.0,
+ "887": 122.0,
+ "888": 100.0,
+ "889": 107.0,
+ "890": 105.0,
+ "891": 114.0,
+ "892": 106.0,
+ "893": 125.0,
+ "894": 111.0,
+ "895": 88.0,
+ "896": 112.0,
+ "897": 110.0,
+ "898": 111.0,
+ "899": 110.0,
+ "900": 125.0,
+ "901": 95.0,
+ "902": 122.0,
+ "903": 113.0,
+ "904": 106.0,
+ "905": 106.0,
+ "906": 89.0,
+ "907": 117.0,
+ "908": 114.0,
+ "909": 110.0,
+ "910": 138.0,
+ "911": 116.0,
+ "912": 125.0,
+ "913": 107.0,
+ "914": 116.0,
+ "915": 109.0,
+ "916": 109.0,
+ "917": 99.0,
+ "918": 114.0,
+ "919": 126.0,
+ "920": 101.0,
+ "921": 96.0,
+ "922": 93.0,
+ "923": 106.0,
+ "924": 89.0,
+ "925": 87.0,
+ "926": 123.0,
+ "927": 107.0,
+ "928": 107.0,
+ "929": 91.0,
+ "930": 82.0,
+ "931": 97.0,
+ "932": 83.0,
+ "933": 101.0,
+ "934": 79.0,
+ "935": 114.0,
+ "936": 100.0,
+ "937": 109.0,
+ "938": 99.0,
+ "939": 113.0,
+ "940": 111.0,
+ "941": 110.0,
+ "942": 79.0,
+ "943": 113.0,
+ "944": 120.0,
+ "945": 98.0,
+ "946": 108.0,
+ "947": 97.0,
+ "948": 99.0,
+ "949": 94.0,
+ "950": 101.0,
+ "951": 72.0,
+ "952": 108.0,
+ "953": 125.0,
+ "954": 111.0,
+ "955": 95.0,
+ "956": 89.0,
+ "957": 105.0,
+ "958": 84.0,
+ "959": 103.0,
+ "960": 125.0,
+ "961": 97.0,
+ "962": 108.0,
+ "963": 89.0,
+ "964": 99.0,
+ "965": 114.0,
+ "966": 106.0,
+ "967": 108.0,
+ "968": 99.0,
+ "969": 115.0,
+ "970": 114.0,
+ "971": 113.0,
+ "972": 145.0,
+ "973": 92.0,
+ "974": 107.0,
+ "975": 111.0,
+ "976": 114.0,
+ "977": 110.0,
+ "978": 112.0,
+ "979": 109.0,
+ "980": 92.0,
+ "981": 81.0,
+ "982": 109.0,
+ "983": 87.0,
+ "984": 109.0,
+ "985": 109.0,
+ "986": 123.0,
+ "987": 102.0,
+ "988": 129.0,
+ "989": 124.0,
+ "990": 119.0,
+ "991": 116.0,
+ "992": 122.0,
+ "993": 114.0,
+ "994": 90.0,
+ "995": 114.0,
+ "996": 113.0,
+ "997": 127.0,
+ "998": 133.0,
+ "999": 114.0,
+ "1000": 100.0,
+ "1001": 104.0,
+ "1002": 110.0,
+ "1003": 108.0,
+ "1004": 128.0,
+ "1005": 96.0,
+ "1006": 103.0,
+ "1007": 108.0,
+ "1008": 94.0,
+ "1009": 115.0,
+ "1010": 114.0,
+ "1011": 115.0,
+ "1012": 103.0,
+ "1013": 135.0,
+ "1014": 112.0,
+ "1015": 101.0,
+ "1016": 120.0,
+ "1017": 100.0,
+ "1018": 107.0,
+ "1019": 110.0,
+ "1020": 106.0,
+ "1021": 88.0,
+ "1022": 106.0,
+ "1023": 105.0,
+ "1024": 87.0,
+ "1025": 112.0,
+ "1026": 105.0,
+ "1027": 118.0,
+ "1028": 97.0,
+ "1029": 81.0,
+ "1030": 95.0,
+ "1031": 103.0,
+ "1032": 106.0,
+ "1033": 107.0,
+ "1034": 115.0,
+ "1035": 100.0,
+ "1036": 106.0,
+ "1037": 130.0,
+ "1038": 116.0,
+ "1039": 114.0,
+ "1040": 107.0,
+ "1041": 117.0,
+ "1042": 101.0,
+ "1043": 111.0,
+ "1044": 127.0,
+ "1045": 106.0,
+ "1046": 99.0,
+ "1047": 97.0,
+ "1048": 99.0,
+ "1049": 110.0,
+ "1050": 97.0,
+ "1051": 142.0,
+ "1052": 106.0,
+ "1053": 84.0,
+ "1054": 100.0,
+ "1055": 107.0,
+ "1056": 100.0,
+ "1057": 110.0,
+ "1058": 123.0,
+ "1059": 103.0,
+ "1060": 98.0,
+ "1061": 123.0,
+ "1062": 114.0,
+ "1063": 118.0,
+ "1064": 108.0,
+ "1065": 117.0,
+ "1066": 111.0,
+ "1067": 101.0,
+ "1068": 102.0,
+ "1069": 107.0,
+ "1070": 128.0,
+ "1071": 109.0,
+ "1072": 107.0,
+ "1073": 119.0,
+ "1074": 119.0,
+ "1075": 120.0,
+ "1076": 110.0,
+ "1077": 114.0,
+ "1078": 108.0,
+ "1079": 97.0,
+ "1080": 98.0,
+ "1081": 102.0,
+ "1082": 108.0,
+ "1083": 122.0,
+ "1084": 105.0,
+ "1085": 110.0,
+ "1086": 134.0,
+ "1087": 97.0,
+ "1088": 107.0,
+ "1089": 120.0,
+ "1090": 96.0,
+ "1091": 99.0,
+ "1092": 111.0,
+ "1093": 105.0,
+ "1094": 117.0,
+ "1095": 103.0,
+ "1096": 103.0,
+ "1097": 95.0,
+ "1098": 111.0,
+ "1099": 112.0,
+ "1100": 119.0,
+ "1101": 119.0,
+ "1102": 98.0,
+ "1103": 112.0,
+ "1104": 123.0,
+ "1105": 132.0,
+ "1106": 106.0,
+ "1107": 77.0,
+ "1108": 120.0,
+ "1109": 112.0,
+ "1110": 81.0,
+ "1111": 108.0,
+ "1112": 102.0,
+ "1113": 118.0,
+ "1114": 86.0,
+ "1115": 92.0,
+ "1116": 89.0,
+ "1117": 84.0,
+ "1118": 94.0,
+ "1119": 80.0,
+ "1120": 109.0,
+ "1121": 118.0,
+ "1122": 102.0,
+ "1123": 87.0,
+ "1124": 99.0,
+ "1125": 118.0,
+ "1126": 89.0,
+ "1127": 85.0,
+ "1128": 74.0,
+ "1129": 84.0,
+ "1130": 105.0,
+ "1131": 115.0,
+ "1132": 110.0,
+ "1133": 80.0,
+ "1134": 101.0,
+ "1135": 101.0,
+ "1136": 108.0,
+ "1137": 105.0,
+ "1138": 92.0,
+ "1139": 97.0,
+ "1140": 82.0,
+ "1141": 93.0,
+ "1142": 67.0,
+ "1143": 66.0,
+ "1144": 104.0,
+ "1145": 106.0,
+ "1146": 93.0,
+ "1147": 88.0,
+ "1148": 92.0,
+ "1149": 106.0,
+ "1150": 94.0,
+ "1151": 91.0,
+ "1152": 122.0,
+ "1153": 79.0,
+ "1154": 108.0,
+ "1155": 99.0,
+ "1156": 93.0,
+ "1157": 107.0,
+ "1158": 90.0,
+ "1159": 121.0,
+ "1160": 78.0,
+ "1161": 85.0,
+ "1162": 102.0,
+ "1163": 125.0,
+ "1164": 86.0,
+ "1165": 98.0,
+ "1166": 112.0,
+ "1167": 99.0,
+ "1168": 96.0,
+ "1169": 124.0,
+ "1170": 100.0,
+ "1171": 107.0,
+ "1172": 102.0,
+ "1173": 90.0,
+ "1174": 96.0,
+ "1175": 121.0,
+ "1176": 97.0,
+ "1177": 101.0,
+ "1178": 127.0,
+ "1179": 86.0,
+ "1180": 100.0,
+ "1181": 98.0,
+ "1182": 100.0,
+ "1183": 102.0,
+ "1184": 99.0,
+ "1185": 118.0,
+ "1186": 86.0,
+ "1187": 102.0,
+ "1188": 99.0,
+ "1189": 125.0,
+ "1190": 128.0,
+ "1191": 109.0,
+ "1192": 110.0,
+ "1193": 111.0,
+ "1194": 96.0,
+ "1195": 126.0,
+ "1196": 103.0,
+ "1197": 103.0,
+ "1198": 113.0,
+ "1199": 78.0,
+ "1200": 121.0,
+ "1201": 86.0,
+ "1202": 103.0,
+ "1203": 109.0,
+ "1204": 104.0,
+ "1205": 112.0,
+ "1206": 119.0,
+ "1207": 116.0,
+ "1208": 109.0,
+ "1209": 113.0,
+ "1210": 112.0,
+ "1211": 108.0,
+ "1212": 98.0,
+ "1213": 102.0,
+ "1214": 94.0,
+ "1215": 119.0,
+ "1216": 115.0,
+ "1217": 91.0,
+ "1218": 113.0,
+ "1219": 106.0,
+ "1220": 99.0,
+ "1221": 119.0,
+ "1222": 113.0,
+ "1223": 108.0,
+ "1224": 97.0,
+ "1225": 86.0,
+ "1226": 89.0,
+ "1227": 108.0,
+ "1228": 83.0,
+ "1229": 82.0,
+ "1230": 108.0,
+ "1231": 68.0,
+ "1232": 111.0,
+ "1233": 89.0,
+ "1234": 89.0,
+ "1235": 103.0,
+ "1236": 114.0,
+ "1237": 116.0,
+ "1238": 88.0,
+ "1239": 114.0,
+ "1240": 118.0,
+ "1241": 126.0,
+ "1242": 86.0,
+ "1243": 112.0,
+ "1244": 117.0,
+ "1245": 109.0,
+ "1246": 119.0,
+ "1247": 116.0,
+ "1248": 117.0,
+ "1249": 100.0,
+ "1250": 126.0,
+ "1251": 75.0,
+ "1252": 101.0,
+ "1253": 108.0,
+ "1254": 116.0,
+ "1255": 94.0,
+ "1256": 105.0,
+ "1257": 124.0,
+ "1258": 129.0,
+ "1259": 105.0,
+ "1260": 96.0,
+ "1261": 117.0,
+ "1262": 117.0,
+ "1263": 110.0,
+ "1264": 125.0,
+ "1265": 104.0,
+ "1266": 103.0,
+ "1267": 93.0,
+ "1268": 103.0,
+ "1269": 96.0,
+ "1270": 98.0,
+ "1271": 87.0,
+ "1272": 97.0,
+ "1273": 79.0,
+ "1274": 95.0,
+ "1275": 107.0,
+ "1276": 88.0,
+ "1277": 107.0,
+ "1278": 86.0,
+ "1279": 85.0,
+ "1280": 103.0,
+ "1281": 103.0,
+ "1282": 89.0,
+ "1283": 109.0,
+ "1284": 122.0,
+ "1285": 94.0,
+ "1286": 89.0,
+ "1287": 122.0,
+ "1288": 103.0,
+ "1289": 107.0,
+ "1290": 132.0,
+ "1291": 89.0,
+ "1292": 124.0,
+ "1293": 137.0,
+ "1294": 113.0,
+ "1295": 88.0,
+ "1296": 117.0,
+ "1297": 100.0,
+ "1298": 113.0,
+ "1299": 93.0,
+ "1300": 116.0,
+ "1301": 125.0,
+ "1302": 133.0,
+ "1303": 102.0,
+ "1304": 97.0,
+ "1305": 105.0,
+ "1306": 90.0,
+ "1307": 101.0,
+ "1308": 123.0,
+ "1309": 77.0,
+ "1310": 102.0,
+ "1311": 115.0,
+ "1312": 116.0,
+ "1313": 78.0,
+ "1314": 101.0,
+ "1315": 121.0,
+ "1316": 103.0,
+ "1317": 97.0,
+ "1318": 98.0,
+ "1319": 105.0,
+ "1320": 90.0,
+ "1321": 127.0,
+ "1322": 66.0,
+ "1323": 92.0,
+ "1324": 113.0,
+ "1325": 99.0,
+ "1326": 98.0,
+ "1327": 97.0,
+ "1328": 105.0,
+ "1329": 108.0,
+ "1330": 92.0,
+ "1331": 113.0,
+ "1332": 81.0,
+ "1333": 114.0,
+ "1334": 122.0,
+ "1335": 109.0,
+ "1336": 96.0,
+ "1337": 98.0,
+ "1338": 102.0,
+ "1339": 95.0,
+ "1340": 98.0,
+ "1341": 93.0,
+ "1342": 100.0,
+ "1343": 95.0,
+ "1344": 106.0,
+ "1345": 110.0,
+ "1346": 97.0,
+ "1347": 102.0,
+ "1348": 98.0,
+ "1349": 124.0,
+ "1350": 122.0,
+ "1351": 109.0,
+ "1352": 103.0,
+ "1353": 113.0,
+ "1354": 94.0,
+ "1355": 124.0,
+ "1356": 112.0,
+ "1357": 87.0,
+ "1358": 117.0,
+ "1359": 82.0,
+ "1360": 104.0,
+ "1361": 114.0,
+ "1362": 86.0,
+ "1363": 114.0,
+ "1364": 106.0,
+ "1365": 107.0,
+ "1366": 100.0,
+ "1367": 120.0,
+ "1368": 103.0,
+ "1369": 119.0,
+ "1370": 98.0,
+ "1371": 108.0,
+ "1372": 78.0,
+ "1373": 118.0,
+ "1374": 118.0,
+ "1375": 112.0,
+ "1376": 106.0,
+ "1377": 89.0,
+ "1378": 108.0,
+ "1379": 125.0,
+ "1380": 93.0,
+ "1381": 99.0,
+ "1382": 110.0,
+ "1383": 117.0,
+ "1384": 96.0,
+ "1385": 110.0,
+ "1386": 117.0,
+ "1387": 110.0,
+ "1388": 105.0,
+ "1389": 96.0,
+ "1390": 105.0,
+ "1391": 111.0,
+ "1392": 89.0,
+ "1393": 106.0,
+ "1394": 107.0,
+ "1395": 78.0,
+ "1396": 121.0,
+ "1397": 91.0,
+ "1398": 115.0,
+ "1399": 110.0,
+ "1400": 99.0,
+ "1401": 109.0,
+ "1402": 108.0,
+ "1403": 99.0,
+ "1404": 122.0,
+ "1405": 129.0,
+ "1406": 93.0,
+ "1407": 119.0,
+ "1408": 89.0,
+ "1409": 110.0,
+ "1410": 90.0,
+ "1411": 112.0,
+ "1412": 98.0,
+ "1413": 107.0,
+ "1414": 106.0,
+ "1415": 109.0,
+ "1416": 127.0,
+ "1417": 105.0,
+ "1418": 92.0,
+ "1419": 111.0,
+ "1420": 118.0,
+ "1421": 121.0,
+ "1422": 110.0,
+ "1423": 100.0,
+ "1424": 109.0,
+ "1425": 127.0,
+ "1426": 91.0,
+ "1427": 112.0,
+ "1428": 96.0,
+ "1429": 105.0,
+ "1430": 100.0,
+ "1431": 113.0,
+ "1432": 105.0,
+ "1433": 106.0,
+ "1434": 118.0,
+ "1435": 102.0,
+ "1436": 90.0,
+ "1437": 111.0,
+ "1438": 97.0,
+ "1439": 104.0,
+ "1440": 84.0,
+ "1441": 107.0,
+ "1442": 103.0,
+ "1443": 100.0,
+ "1444": 85.0,
+ "1445": 103.0,
+ "1446": 110.0,
+ "1447": 120.0,
+ "1448": 98.0,
+ "1449": 114.0,
+ "1450": 111.0,
+ "1451": 107.0,
+ "1452": 133.0,
+ "1453": 116.0,
+ "1454": 102.0,
+ "1455": 109.0,
+ "1456": 123.0,
+ "1457": 124.0,
+ "1458": 108.0,
+ "1459": 107.0,
+ "1460": 96.0,
+ "1461": 115.0,
+ "1462": 90.0,
+ "1463": 115.0,
+ "1464": 86.0,
+ "1465": 88.0,
+ "1466": 117.0,
+ "1467": 84.0,
+ "1468": 107.0,
+ "1469": 86.0,
+ "1470": 124.0,
+ "1471": 109.0,
+ "1472": 87.0,
+ "1473": 137.0,
+ "1474": 126.0,
+ "1475": 93.0,
+ "1476": 101.0,
+ "1477": 106.0,
+ "1478": 116.0,
+ "1479": 125.0,
+ "1480": 103.0,
+ "1481": 108.0,
+ "1482": 102.0,
+ "1483": 96.0,
+ "1484": 109.0,
+ "1485": 96.0,
+ "1486": 106.0,
+ "1487": 95.0,
+ "1488": 116.0,
+ "1489": 110.0,
+ "1490": 110.0,
+ "1491": 101.0,
+ "1492": 114.0,
+ "1493": 121.0,
+ "1494": 130.0,
+ "1495": 147.0,
+ "1496": 105.0,
+ "1497": 103.0,
+ "1498": 94.0,
+ "1499": 95.0,
+ "1500": 120.0,
+ "1501": 108.0,
+ "1502": 118.0,
+ "1503": 96.0,
+ "1504": 87.0,
+ "1505": 122.0,
+ "1506": 103.0,
+ "1507": 86.0,
+ "1508": 107.0,
+ "1509": 111.0,
+ "1510": 124.0,
+ "1511": 115.0,
+ "1512": 100.0,
+ "1513": 113.0,
+ "1514": 119.0,
+ "1515": 113.0,
+ "1516": 104.0,
+ "1517": 111.0,
+ "1518": 85.0,
+ "1519": 100.0,
+ "1520": 86.0,
+ "1521": 126.0,
+ "1522": 117.0,
+ "1523": 104.0,
+ "1524": 91.0,
+ "1525": 118.0,
+ "1526": 97.0,
+ "1527": 113.0,
+ "1528": 121.0,
+ "1529": 95.0,
+ "1530": 119.0,
+ "1531": 106.0,
+ "1532": 119.0,
+ "1533": 118.0,
+ "1534": 109.0,
+ "1535": 125.0,
+ "1536": 100.0,
+ "1537": 111.0,
+ "1538": 88.0,
+ "1539": 112.0,
+ "1540": 125.0,
+ "1541": 105.0,
+ "1542": 113.0,
+ "1543": 115.0,
+ "1544": 84.0,
+ "1545": 95.0,
+ "1546": 94.0,
+ "1547": 96.0,
+ "1548": 94.0,
+ "1549": 116.0,
+ "1550": 89.0,
+ "1551": 110.0,
+ "1552": 94.0,
+ "1553": 117.0,
+ "1554": 103.0,
+ "1555": 113.0,
+ "1556": 98.0,
+ "1557": 105.0,
+ "1558": 118.0,
+ "1559": 107.0,
+ "1560": 101.0,
+ "1561": 111.0,
+ "1562": 94.0,
+ "1563": 107.0,
+ "1564": 129.0,
+ "1565": 108.0,
+ "1566": 96.0,
+ "1567": 119.0,
+ "1568": 113.0,
+ "1569": 102.0,
+ "1570": 104.0,
+ "1571": 101.0,
+ "1572": 101.0,
+ "1573": 119.0,
+ "1574": 100.0,
+ "1575": 99.0,
+ "1576": 99.0,
+ "1577": 94.0,
+ "1578": 104.0,
+ "1579": 111.0,
+ "1580": 107.0,
+ "1581": 136.0,
+ "1582": 128.0,
+ "1583": 115.0,
+ "1584": 109.0,
+ "1585": 117.0,
+ "1586": 129.0,
+ "1587": 102.0,
+ "1588": 83.0,
+ "1589": 106.0,
+ "1590": 127.0,
+ "1591": 102.0,
+ "1592": 113.0,
+ "1593": 109.0,
+ "1594": 104.0,
+ "1595": 107.0,
+ "1596": 130.0,
+ "1597": 117.0,
+ "1598": 124.0,
+ "1599": 101.0,
+ "1600": 106.0,
+ "1601": 113.0,
+ "1602": 96.0,
+ "1603": 119.0,
+ "1604": 94.0,
+ "1605": 121.0,
+ "1606": 96.0,
+ "1607": 104.0,
+ "1608": 97.0,
+ "1609": 113.0,
+ "1610": 94.0,
+ "1611": 92.0,
+ "1612": 113.0,
+ "1613": 119.0,
+ "1614": 112.0,
+ "1615": 108.0,
+ "1616": 101.0,
+ "1617": 109.0,
+ "1618": 107.0,
+ "1619": 104.0,
+ "1620": 100.0,
+ "1621": 110.0,
+ "1622": 111.0,
+ "1623": 121.0,
+ "1624": 113.0,
+ "1625": 119.0,
+ "1626": 111.0,
+ "1627": 94.0,
+ "1628": 111.0,
+ "1629": 86.0,
+ "1630": 113.0,
+ "1631": 106.0,
+ "1632": 115.0,
+ "1633": 100.0,
+ "1634": 112.0,
+ "1635": 99.0,
+ "1636": 106.0,
+ "1637": 110.0,
+ "1638": 105.0,
+ "1639": 114.0,
+ "1640": 108.0,
+ "1641": 114.0,
+ "1642": 104.0,
+ "1643": 92.0,
+ "1644": 106.0,
+ "1645": 102.0,
+ "1646": 122.0,
+ "1647": 103.0,
+ "1648": 134.0,
+ "1649": 91.0,
+ "1650": 113.0,
+ "1651": 110.0,
+ "1652": 119.0,
+ "1653": 113.0,
+ "1654": 96.0,
+ "1655": 89.0,
+ "1656": 95.0,
+ "1657": 97.0,
+ "1658": 86.0,
+ "1659": 104.0,
+ "1660": 84.0,
+ "1661": 98.0,
+ "1662": 92.0,
+ "1663": 102.0,
+ "1664": 119.0,
+ "1665": 93.0,
+ "1666": 98.0,
+ "1667": 102.0,
+ "1668": 120.0,
+ "1669": 103.0,
+ "1670": 111.0,
+ "1671": 107.0,
+ "1672": 97.0,
+ "1673": 96.0,
+ "1674": 102.0,
+ "1675": 82.0,
+ "1676": 113.0,
+ "1677": 103.0,
+ "1678": 93.0,
+ "1679": 99.0,
+ "1680": 106.0,
+ "1681": 98.0,
+ "1682": 109.0,
+ "1683": 96.0,
+ "1684": 137.0,
+ "1685": 116.0,
+ "1686": 103.0,
+ "1687": 93.0,
+ "1688": 116.0,
+ "1689": 105.0,
+ "1690": 102.0,
+ "1691": 86.0,
+ "1692": 110.0,
+ "1693": 108.0,
+ "1694": 98.0,
+ "1695": 89.0,
+ "1696": 100.0,
+ "1697": 108.0,
+ "1698": 107.0,
+ "1699": 107.0,
+ "1700": 91.0,
+ "1701": 124.0,
+ "1702": 102.0,
+ "1703": 107.0,
+ "1704": 86.0,
+ "1705": 101.0,
+ "1706": 115.0,
+ "1707": 87.0,
+ "1708": 121.0,
+ "1709": 119.0,
+ "1710": 117.0,
+ "1711": 146.0,
+ "1712": 105.0,
+ "1713": 98.0,
+ "1714": 84.0,
+ "1715": 124.0,
+ "1716": 112.0,
+ "1717": 123.0,
+ "1718": 129.0,
+ "1719": 90.0,
+ "1720": 110.0,
+ "1721": 122.0,
+ "1722": 110.0,
+ "1723": 100.0,
+ "1724": 125.0,
+ "1725": 112.0,
+ "1726": 101.0,
+ "1727": 112.0,
+ "1728": 110.0,
+ "1729": 97.0,
+ "1730": 105.0,
+ "1731": 95.0,
+ "1732": 110.0,
+ "1733": 100.0,
+ "1734": 101.0,
+ "1735": 101.0,
+ "1736": 93.0,
+ "1737": 96.0,
+ "1738": 115.0,
+ "1739": 110.0,
+ "1740": 97.0,
+ "1741": 123.0,
+ "1742": 108.0,
+ "1743": 98.0,
+ "1744": 111.0,
+ "1745": 97.0,
+ "1746": 110.0,
+ "1747": 117.0,
+ "1748": 115.0,
+ "1749": 103.0,
+ "1750": 125.0,
+ "1751": 114.0,
+ "1752": 109.0,
+ "1753": 99.0,
+ "1754": 93.0,
+ "1755": 107.0,
+ "1756": 121.0,
+ "1757": 138.0,
+ "1758": 97.0,
+ "1759": 120.0,
+ "1760": 106.0,
+ "1761": 108.0,
+ "1762": 93.0,
+ "1763": 93.0,
+ "1764": 133.0,
+ "1765": 101.0,
+ "1766": 93.0,
+ "1767": 112.0,
+ "1768": 105.0,
+ "1769": 108.0,
+ "1770": 104.0,
+ "1771": 120.0,
+ "1772": 123.0,
+ "1773": 102.0,
+ "1774": 108.0,
+ "1775": 116.0,
+ "1776": 94.0,
+ "1777": 133.0,
+ "1778": 119.0,
+ "1779": 103.0,
+ "1780": 121.0,
+ "1781": 103.0,
+ "1782": 100.0,
+ "1783": 100.0,
+ "1784": 110.0,
+ "1785": 109.0,
+ "1786": 110.0,
+ "1787": 97.0,
+ "1788": 105.0,
+ "1789": 117.0,
+ "1790": 105.0,
+ "1791": 90.0,
+ "1792": 117.0,
+ "1793": 110.0,
+ "1794": 121.0,
+ "1795": 123.0,
+ "1796": 111.0,
+ "1797": 130.0,
+ "1798": 99.0,
+ "1799": 117.0,
+ "1800": 116.0,
+ "1801": 114.0,
+ "1802": 101.0,
+ "1803": 114.0,
+ "1804": 110.0,
+ "1805": 119.0,
+ "1806": 114.0,
+ "1807": 101.0,
+ "1808": 99.0,
+ "1809": 129.0,
+ "1810": 115.0,
+ "1811": 113.0,
+ "1812": 119.0,
+ "1813": 107.0,
+ "1814": 137.0,
+ "1815": 105.0,
+ "1816": 123.0,
+ "1817": 96.0,
+ "1818": 121.0,
+ "1819": 110.0,
+ "1820": 104.0,
+ "1821": 101.0,
+ "1822": 89.0,
+ "1823": 128.0,
+ "1824": 115.0,
+ "1825": 102.0,
+ "1826": 111.0,
+ "1827": 94.0,
+ "1828": 117.0,
+ "1829": 99.0,
+ "1830": 118.0,
+ "1831": 100.0,
+ "1832": 127.0,
+ "1833": 95.0,
+ "1834": 117.0,
+ "1835": 116.0,
+ "1836": 107.0,
+ "1837": 128.0,
+ "1838": 109.0,
+ "1839": 112.0,
+ "1840": 107.0,
+ "1841": 105.0,
+ "1842": 90.0,
+ "1843": 116.0,
+ "1844": 111.0,
+ "1845": 101.0,
+ "1846": 113.0,
+ "1847": 107.0,
+ "1848": 123.0,
+ "1849": 92.0,
+ "1850": 111.0,
+ "1851": 105.0,
+ "1852": 115.0,
+ "1853": 90.0,
+ "1854": 103.0,
+ "1855": 114.0,
+ "1856": 143.0,
+ "1857": 110.0,
+ "1858": 90.0,
+ "1859": 97.0,
+ "1860": 100.0,
+ "1861": 95.0,
+ "1862": 99.0,
+ "1863": 116.0,
+ "1864": 107.0,
+ "1865": 112.0,
+ "1866": 99.0,
+ "1867": 103.0,
+ "1868": 125.0,
+ "1869": 101.0,
+ "1870": 109.0,
+ "1871": 106.0,
+ "1872": 116.0,
+ "1873": 102.0,
+ "1874": 105.0,
+ "1875": 101.0,
+ "1876": 123.0,
+ "1877": 110.0,
+ "1878": 123.0,
+ "1879": 110.0,
+ "1880": 113.0,
+ "1881": 118.0,
+ "1882": 100.0,
+ "1883": 95.0,
+ "1884": 133.0,
+ "1885": 124.0,
+ "1886": 109.0,
+ "1887": 117.0,
+ "1888": 119.0,
+ "1889": 92.0,
+ "1890": 102.0,
+ "1891": 111.0,
+ "1892": 106.0,
+ "1893": 102.0,
+ "1894": 112.0,
+ "1895": 104.0,
+ "1896": 122.0,
+ "1897": 120.0,
+ "1898": 87.0,
+ "1899": 116.0,
+ "1900": 124.0,
+ "1901": 109.0,
+ "1902": 93.0,
+ "1903": 105.0,
+ "1904": 116.0,
+ "1905": 94.0,
+ "1906": 111.0,
+ "1907": 129.0,
+ "1908": 110.0,
+ "1909": 117.0,
+ "1910": 104.0,
+ "1911": 130.0,
+ "1912": 104.0,
+ "1913": 121.0,
+ "1914": 98.0,
+ "1915": 105.0,
+ "1916": 116.0,
+ "1917": 123.0,
+ "1918": 100.0,
+ "1919": 135.0,
+ "1920": 98.0,
+ "1921": 112.0,
+ "1922": 100.0,
+ "1923": 122.0,
+ "1924": 140.0,
+ "1925": 118.0,
+ "1926": 115.0,
+ "1927": 112.0,
+ "1928": 124.0,
+ "1929": 127.0,
+ "1930": 118.0,
+ "1931": 103.0,
+ "1932": 129.0,
+ "1933": 119.0,
+ "1934": 111.0,
+ "1935": 115.0,
+ "1936": 125.0,
+ "1937": 93.0,
+ "1938": 100.0,
+ "1939": 128.0,
+ "1940": 107.0,
+ "1941": 109.0,
+ "1942": 105.0,
+ "1943": 88.0,
+ "1944": 112.0,
+ "1945": 129.0,
+ "1946": 111.0,
+ "1947": 112.0,
+ "1948": 119.0,
+ "1949": 100.0,
+ "1950": 122.0,
+ "1951": 111.0,
+ "1952": 121.0,
+ "1953": 107.0,
+ "1954": 125.0,
+ "1955": 115.0,
+ "1956": 100.0,
+ "1957": 108.0,
+ "1958": 120.0,
+ "1959": 124.0,
+ "1960": 87.0,
+ "1961": 114.0,
+ "1962": 103.0,
+ "1963": 110.0,
+ "1964": 112.0,
+ "1965": 124.0,
+ "1966": 142.0,
+ "1967": 142.0,
+ "1968": 115.0,
+ "1969": 112.0,
+ "1970": 110.0,
+ "1971": 107.0,
+ "1972": 107.0,
+ "1973": 101.0,
+ "1974": 91.0,
+ "1975": 88.0,
+ "1976": 122.0,
+ "1977": 107.0,
+ "1978": 108.0,
+ "1979": 116.0,
+ "1980": 115.0,
+ "1981": 136.0,
+ "1982": 125.0,
+ "1983": 114.0,
+ "1984": 96.0,
+ "1985": 109.0,
+ "1986": 138.0,
+ "1987": 102.0,
+ "1988": 148.0,
+ "1989": 116.0,
+ "1990": 105.0,
+ "1991": 121.0,
+ "1992": 136.0,
+ "1993": 121.0,
+ "1994": 105.0,
+ "1995": 112.0,
+ "1996": 131.0,
+ "1997": 111.0,
+ "1998": 103.0,
+ "1999": 102.0,
+ "2000": 130.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 465988096.0,
+ "2": 465988096.0,
+ "3": 465988096.0,
+ "4": 465988096.0,
+ "5": 465988096.0,
+ "6": 465988096.0,
+ "7": 465988096.0,
+ "8": 465988096.0,
+ "9": 465988096.0,
+ "10": 465988096.0,
+ "11": 465988096.0,
+ "12": 465988096.0,
+ "13": 465988096.0,
+ "14": 465988096.0,
+ "15": 465988096.0,
+ "16": 465988096.0,
+ "17": 465988096.0,
+ "18": 465988096.0,
+ "19": 465988096.0,
+ "20": 465988096.0,
+ "21": 465988096.0,
+ "22": 465988096.0,
+ "23": 465988096.0,
+ "24": 465988096.0,
+ "25": 465988096.0,
+ "26": 465988096.0,
+ "27": 465988096.0,
+ "28": 465988096.0,
+ "29": 465988096.0,
+ "30": 465988096.0,
+ "31": 465988096.0,
+ "32": 465988096.0,
+ "33": 465988096.0,
+ "34": 465988096.0,
+ "35": 465988096.0,
+ "36": 465988096.0,
+ "37": 465988096.0,
+ "38": 465988096.0,
+ "39": 465988096.0,
+ "40": 465988096.0,
+ "41": 465988096.0,
+ "42": 465988096.0,
+ "43": 465988096.0,
+ "44": 465988096.0,
+ "45": 465988096.0,
+ "46": 465988096.0,
+ "47": 465988096.0,
+ "48": 465988096.0,
+ "49": 465988096.0,
+ "50": 465988096.0,
+ "51": 465988096.0,
+ "52": 465988096.0,
+ "53": 465988096.0,
+ "54": 465988096.0,
+ "55": 465988096.0,
+ "56": 465988096.0,
+ "57": 465988096.0,
+ "58": 465988096.0,
+ "59": 465988096.0,
+ "60": 465988096.0,
+ "61": 465988096.0,
+ "62": 465988096.0,
+ "63": 465988096.0,
+ "64": 465988096.0,
+ "65": 465988096.0,
+ "66": 465988096.0,
+ "67": 465988096.0,
+ "68": 465988096.0,
+ "69": 465988096.0,
+ "70": 465988096.0,
+ "71": 465988096.0,
+ "72": 465988096.0,
+ "73": 465988096.0,
+ "74": 465988096.0,
+ "75": 465988096.0,
+ "76": 465988096.0,
+ "77": 465988096.0,
+ "78": 465988096.0,
+ "79": 465988096.0,
+ "80": 465988096.0,
+ "81": 465988096.0,
+ "82": 465988096.0,
+ "83": 465988096.0,
+ "84": 465988096.0,
+ "85": 465988096.0,
+ "86": 465988096.0,
+ "87": 465988096.0,
+ "88": 465988096.0,
+ "89": 465988096.0,
+ "90": 465988096.0,
+ "91": 465988096.0,
+ "92": 465988096.0,
+ "93": 465988096.0,
+ "94": 465988096.0,
+ "95": 465988096.0,
+ "96": 465988096.0,
+ "97": 465988096.0,
+ "98": 465988096.0,
+ "99": 465988096.0,
+ "100": 465988096.0,
+ "101": 465988096.0,
+ "102": 465988096.0,
+ "103": 465988096.0,
+ "104": 465988096.0,
+ "105": 465988096.0,
+ "106": 465988096.0,
+ "107": 465988096.0,
+ "108": 465988096.0,
+ "109": 465988096.0,
+ "110": 465988096.0,
+ "111": 465988096.0,
+ "112": 465988096.0,
+ "113": 465988096.0,
+ "114": 465988096.0,
+ "115": 465988096.0,
+ "116": 465988096.0,
+ "117": 465988096.0,
+ "118": 465988096.0,
+ "119": 465988096.0,
+ "120": 465988096.0,
+ "121": 465988096.0,
+ "122": 465988096.0,
+ "123": 465988096.0,
+ "124": 465988096.0,
+ "125": 465988096.0,
+ "126": 465988096.0,
+ "127": 465988096.0,
+ "128": 465988096.0,
+ "129": 465988096.0,
+ "130": 465988096.0,
+ "131": 465988096.0,
+ "132": 465988096.0,
+ "133": 465988096.0,
+ "134": 465988096.0,
+ "135": 465988096.0,
+ "136": 465988096.0,
+ "137": 465988096.0,
+ "138": 465988096.0,
+ "139": 465988096.0,
+ "140": 465988096.0,
+ "141": 465988096.0,
+ "142": 465988096.0,
+ "143": 465988096.0,
+ "144": 465988096.0,
+ "145": 465988096.0,
+ "146": 465988096.0,
+ "147": 465988096.0,
+ "148": 465988096.0,
+ "149": 465988096.0,
+ "150": 465988096.0,
+ "151": 465988096.0,
+ "152": 465988096.0,
+ "153": 465988096.0,
+ "154": 465988096.0,
+ "155": 465988096.0,
+ "156": 465988096.0,
+ "157": 465988096.0,
+ "158": 465988096.0,
+ "159": 465988096.0,
+ "160": 465988096.0,
+ "161": 465988096.0,
+ "162": 465988096.0,
+ "163": 465988096.0,
+ "164": 465988096.0,
+ "165": 465988096.0,
+ "166": 465988096.0,
+ "167": 465988096.0,
+ "168": 465988096.0,
+ "169": 465988096.0,
+ "170": 465988096.0,
+ "171": 465988096.0,
+ "172": 465988096.0,
+ "173": 465988096.0,
+ "174": 465988096.0,
+ "175": 465988096.0,
+ "176": 465988096.0,
+ "177": 465988096.0,
+ "178": 465988096.0,
+ "179": 465988096.0,
+ "180": 465988096.0,
+ "181": 465988096.0,
+ "182": 465988096.0,
+ "183": 465988096.0,
+ "184": 465988096.0,
+ "185": 465988096.0,
+ "186": 465988096.0,
+ "187": 465988096.0,
+ "188": 465988096.0,
+ "189": 465988096.0,
+ "190": 465988096.0,
+ "191": 465988096.0,
+ "192": 465988096.0,
+ "193": 465988096.0,
+ "194": 465988096.0,
+ "195": 465988096.0,
+ "196": 465988096.0,
+ "197": 465988096.0,
+ "198": 465988096.0,
+ "199": 465988096.0,
+ "200": 465988096.0,
+ "201": 465988096.0,
+ "202": 465988096.0,
+ "203": 465988096.0,
+ "204": 465988096.0,
+ "205": 465988096.0,
+ "206": 465988096.0,
+ "207": 465988096.0,
+ "208": 465988096.0,
+ "209": 465988096.0,
+ "210": 465988096.0,
+ "211": 465988096.0,
+ "212": 465988096.0,
+ "213": 465988096.0,
+ "214": 465988096.0,
+ "215": 465988096.0,
+ "216": 465988096.0,
+ "217": 465988096.0,
+ "218": 465988096.0,
+ "219": 465988096.0,
+ "220": 465988096.0,
+ "221": 465988096.0,
+ "222": 465988096.0,
+ "223": 465988096.0,
+ "224": 465988096.0,
+ "225": 465988096.0,
+ "226": 465988096.0,
+ "227": 465988096.0,
+ "228": 465988096.0,
+ "229": 465988096.0,
+ "230": 465988096.0,
+ "231": 465988096.0,
+ "232": 465988096.0,
+ "233": 465988096.0,
+ "234": 465988096.0,
+ "235": 465988096.0,
+ "236": 465988096.0,
+ "237": 465988096.0,
+ "238": 465988096.0,
+ "239": 465988096.0,
+ "240": 465988096.0,
+ "241": 465988096.0,
+ "242": 465988096.0,
+ "243": 465988096.0,
+ "244": 465988096.0,
+ "245": 465988096.0,
+ "246": 465988096.0,
+ "247": 465988096.0,
+ "248": 465988096.0,
+ "249": 465988096.0,
+ "250": 465988096.0,
+ "251": 465988096.0,
+ "252": 465988096.0,
+ "253": 465988096.0,
+ "254": 465988096.0,
+ "255": 465988096.0,
+ "256": 465988096.0,
+ "257": 465988096.0,
+ "258": 465988096.0,
+ "259": 465988096.0,
+ "260": 465988096.0,
+ "261": 465988096.0,
+ "262": 465988096.0,
+ "263": 465988096.0,
+ "264": 465988096.0,
+ "265": 465988096.0,
+ "266": 465988096.0,
+ "267": 465988096.0,
+ "268": 465988096.0,
+ "269": 465988096.0,
+ "270": 465988096.0,
+ "271": 465988096.0,
+ "272": 465988096.0,
+ "273": 465988096.0,
+ "274": 465988096.0,
+ "275": 465988096.0,
+ "276": 465988096.0,
+ "277": 465988096.0,
+ "278": 465988096.0,
+ "279": 465988096.0,
+ "280": 465988096.0,
+ "281": 465988096.0,
+ "282": 465988096.0,
+ "283": 465988096.0,
+ "284": 465988096.0,
+ "285": 465988096.0,
+ "286": 465988096.0,
+ "287": 465988096.0,
+ "288": 465988096.0,
+ "289": 465988096.0,
+ "290": 465988096.0,
+ "291": 465988096.0,
+ "292": 465988096.0,
+ "293": 465988096.0,
+ "294": 465988096.0,
+ "295": 465988096.0,
+ "296": 465988096.0,
+ "297": 465988096.0,
+ "298": 465988096.0,
+ "299": 465988096.0,
+ "300": 465988096.0,
+ "301": 465988096.0,
+ "302": 465988096.0,
+ "303": 465988096.0,
+ "304": 465988096.0,
+ "305": 465988096.0,
+ "306": 465988096.0,
+ "307": 465988096.0,
+ "308": 465988096.0,
+ "309": 465988096.0,
+ "310": 465988096.0,
+ "311": 465988096.0,
+ "312": 465988096.0,
+ "313": 465988096.0,
+ "314": 465988096.0,
+ "315": 465988096.0,
+ "316": 465988096.0,
+ "317": 465988096.0,
+ "318": 465988096.0,
+ "319": 465988096.0,
+ "320": 465988096.0,
+ "321": 465988096.0,
+ "322": 465988096.0,
+ "323": 465988096.0,
+ "324": 465988096.0,
+ "325": 465988096.0,
+ "326": 465988096.0,
+ "327": 465988096.0,
+ "328": 465988096.0,
+ "329": 465988096.0,
+ "330": 465988096.0,
+ "331": 465988096.0,
+ "332": 465988096.0,
+ "333": 465988096.0,
+ "334": 465988096.0,
+ "335": 465988096.0,
+ "336": 465988096.0,
+ "337": 465988096.0,
+ "338": 465988096.0,
+ "339": 465988096.0,
+ "340": 465988096.0,
+ "341": 465988096.0,
+ "342": 465988096.0,
+ "343": 465988096.0,
+ "344": 465988096.0,
+ "345": 465988096.0,
+ "346": 465988096.0,
+ "347": 465988096.0,
+ "348": 465988096.0,
+ "349": 465988096.0,
+ "350": 465988096.0,
+ "351": 465988096.0,
+ "352": 465988096.0,
+ "353": 465988096.0,
+ "354": 465988096.0,
+ "355": 465988096.0,
+ "356": 465988096.0,
+ "357": 465988096.0,
+ "358": 465988096.0,
+ "359": 465988096.0,
+ "360": 465988096.0,
+ "361": 465988096.0,
+ "362": 465988096.0,
+ "363": 465988096.0,
+ "364": 465988096.0,
+ "365": 465988096.0,
+ "366": 465988096.0,
+ "367": 465988096.0,
+ "368": 465988096.0,
+ "369": 465988096.0,
+ "370": 465988096.0,
+ "371": 465988096.0,
+ "372": 465988096.0,
+ "373": 465988096.0,
+ "374": 465988096.0,
+ "375": 465988096.0,
+ "376": 465988096.0,
+ "377": 465988096.0,
+ "378": 465988096.0,
+ "379": 465988096.0,
+ "380": 465988096.0,
+ "381": 465988096.0,
+ "382": 465988096.0,
+ "383": 465988096.0,
+ "384": 465988096.0,
+ "385": 465988096.0,
+ "386": 465988096.0,
+ "387": 465988096.0,
+ "388": 465988096.0,
+ "389": 465988096.0,
+ "390": 465988096.0,
+ "391": 465988096.0,
+ "392": 465988096.0,
+ "393": 465988096.0,
+ "394": 465988096.0,
+ "395": 465988096.0,
+ "396": 465988096.0,
+ "397": 465988096.0,
+ "398": 465988096.0,
+ "399": 465988096.0,
+ "400": 465988096.0,
+ "401": 465988096.0,
+ "402": 465988096.0,
+ "403": 465988096.0,
+ "404": 465988096.0,
+ "405": 465988096.0,
+ "406": 465988096.0,
+ "407": 465988096.0,
+ "408": 465988096.0,
+ "409": 465988096.0,
+ "410": 465988096.0,
+ "411": 465988096.0,
+ "412": 465988096.0,
+ "413": 465988096.0,
+ "414": 465988096.0,
+ "415": 465988096.0,
+ "416": 465988096.0,
+ "417": 465988096.0,
+ "418": 465988096.0,
+ "419": 465988096.0,
+ "420": 465988096.0,
+ "421": 465988096.0,
+ "422": 465988096.0,
+ "423": 465988096.0,
+ "424": 465988096.0,
+ "425": 465988096.0,
+ "426": 465988096.0,
+ "427": 465988096.0,
+ "428": 465988096.0,
+ "429": 465988096.0,
+ "430": 465988096.0,
+ "431": 465988096.0,
+ "432": 465988096.0,
+ "433": 465988096.0,
+ "434": 465988096.0,
+ "435": 465988096.0,
+ "436": 465988096.0,
+ "437": 465988096.0,
+ "438": 465988096.0,
+ "439": 465988096.0,
+ "440": 465988096.0,
+ "441": 465988096.0,
+ "442": 465988096.0,
+ "443": 465988096.0,
+ "444": 465988096.0,
+ "445": 465988096.0,
+ "446": 465988096.0,
+ "447": 465988096.0,
+ "448": 465988096.0,
+ "449": 465988096.0,
+ "450": 465988096.0,
+ "451": 465988096.0,
+ "452": 465988096.0,
+ "453": 465988096.0,
+ "454": 465988096.0,
+ "455": 465988096.0,
+ "456": 465988096.0,
+ "457": 465988096.0,
+ "458": 465988096.0,
+ "459": 465988096.0,
+ "460": 465988096.0,
+ "461": 465988096.0,
+ "462": 465988096.0,
+ "463": 465988096.0,
+ "464": 465988096.0,
+ "465": 465988096.0,
+ "466": 465988096.0,
+ "467": 465988096.0,
+ "468": 465988096.0,
+ "469": 465988096.0,
+ "470": 465988096.0,
+ "471": 465988096.0,
+ "472": 465988096.0,
+ "473": 465988096.0,
+ "474": 465988096.0,
+ "475": 465988096.0,
+ "476": 465988096.0,
+ "477": 465988096.0,
+ "478": 465988096.0,
+ "479": 465988096.0,
+ "480": 465988096.0,
+ "481": 465988096.0,
+ "482": 465988096.0,
+ "483": 465988096.0,
+ "484": 465988096.0,
+ "485": 465988096.0,
+ "486": 465988096.0,
+ "487": 465988096.0,
+ "488": 465988096.0,
+ "489": 465988096.0,
+ "490": 465988096.0,
+ "491": 465988096.0,
+ "492": 465988096.0,
+ "493": 465988096.0,
+ "494": 465988096.0,
+ "495": 465988096.0,
+ "496": 465988096.0,
+ "497": 465988096.0,
+ "498": 465988096.0,
+ "499": 465988096.0,
+ "500": 465988096.0,
+ "501": 465988096.0,
+ "502": 465988096.0,
+ "503": 465988096.0,
+ "504": 465988096.0,
+ "505": 465988096.0,
+ "506": 465988096.0,
+ "507": 465988096.0,
+ "508": 465988096.0,
+ "509": 465988096.0,
+ "510": 465988096.0,
+ "511": 465988096.0,
+ "512": 465988096.0,
+ "513": 465988096.0,
+ "514": 465988096.0,
+ "515": 465988096.0,
+ "516": 465988096.0,
+ "517": 465988096.0,
+ "518": 465988096.0,
+ "519": 465988096.0,
+ "520": 465988096.0,
+ "521": 465988096.0,
+ "522": 465988096.0,
+ "523": 465988096.0,
+ "524": 465988096.0,
+ "525": 465988096.0,
+ "526": 465988096.0,
+ "527": 465988096.0,
+ "528": 465988096.0,
+ "529": 465988096.0,
+ "530": 465988096.0,
+ "531": 465988096.0,
+ "532": 465988096.0,
+ "533": 465988096.0,
+ "534": 465988096.0,
+ "535": 465988096.0,
+ "536": 465988096.0,
+ "537": 465988096.0,
+ "538": 465988096.0,
+ "539": 465988096.0,
+ "540": 465988096.0,
+ "541": 465988096.0,
+ "542": 465988096.0,
+ "543": 465988096.0,
+ "544": 465988096.0,
+ "545": 465988096.0,
+ "546": 465988096.0,
+ "547": 465988096.0,
+ "548": 465988096.0,
+ "549": 465988096.0,
+ "550": 465988096.0,
+ "551": 465988096.0,
+ "552": 465988096.0,
+ "553": 465988096.0,
+ "554": 465988096.0,
+ "555": 465988096.0,
+ "556": 465988096.0,
+ "557": 465988096.0,
+ "558": 465988096.0,
+ "559": 465988096.0,
+ "560": 465988096.0,
+ "561": 465988096.0,
+ "562": 465988096.0,
+ "563": 465988096.0,
+ "564": 465988096.0,
+ "565": 465988096.0,
+ "566": 465988096.0,
+ "567": 465988096.0,
+ "568": 465988096.0,
+ "569": 465988096.0,
+ "570": 465988096.0,
+ "571": 465988096.0,
+ "572": 465988096.0,
+ "573": 465988096.0,
+ "574": 465988096.0,
+ "575": 465988096.0,
+ "576": 465988096.0,
+ "577": 465988096.0,
+ "578": 465988096.0,
+ "579": 465988096.0,
+ "580": 465988096.0,
+ "581": 465988096.0,
+ "582": 465988096.0,
+ "583": 465988096.0,
+ "584": 465988096.0,
+ "585": 465988096.0,
+ "586": 465988096.0,
+ "587": 465988096.0,
+ "588": 465988096.0,
+ "589": 465988096.0,
+ "590": 465988096.0,
+ "591": 465988096.0,
+ "592": 465988096.0,
+ "593": 465988096.0,
+ "594": 465988096.0,
+ "595": 465988096.0,
+ "596": 465988096.0,
+ "597": 465988096.0,
+ "598": 465988096.0,
+ "599": 465988096.0,
+ "600": 465988096.0,
+ "601": 465988096.0,
+ "602": 465988096.0,
+ "603": 465988096.0,
+ "604": 465988096.0,
+ "605": 465988096.0,
+ "606": 465988096.0,
+ "607": 465988096.0,
+ "608": 465988096.0,
+ "609": 465988096.0,
+ "610": 465988096.0,
+ "611": 465988096.0,
+ "612": 465988096.0,
+ "613": 465988096.0,
+ "614": 465988096.0,
+ "615": 465988096.0,
+ "616": 465988096.0,
+ "617": 465988096.0,
+ "618": 465988096.0,
+ "619": 465988096.0,
+ "620": 465988096.0,
+ "621": 465988096.0,
+ "622": 465988096.0,
+ "623": 465988096.0,
+ "624": 465988096.0,
+ "625": 465988096.0,
+ "626": 465988096.0,
+ "627": 465988096.0,
+ "628": 465988096.0,
+ "629": 465988096.0,
+ "630": 465988096.0,
+ "631": 465988096.0,
+ "632": 465988096.0,
+ "633": 465988096.0,
+ "634": 465988096.0,
+ "635": 465988096.0,
+ "636": 465988096.0,
+ "637": 465988096.0,
+ "638": 465988096.0,
+ "639": 465988096.0,
+ "640": 465988096.0,
+ "641": 465988096.0,
+ "642": 465988096.0,
+ "643": 465988096.0,
+ "644": 465988096.0,
+ "645": 465988096.0,
+ "646": 465988096.0,
+ "647": 465988096.0,
+ "648": 465988096.0,
+ "649": 465988096.0,
+ "650": 465988096.0,
+ "651": 465988096.0,
+ "652": 465988096.0,
+ "653": 465988096.0,
+ "654": 465988096.0,
+ "655": 465988096.0,
+ "656": 465988096.0,
+ "657": 465988096.0,
+ "658": 465988096.0,
+ "659": 465988096.0,
+ "660": 465988096.0,
+ "661": 465988096.0,
+ "662": 465988096.0,
+ "663": 465988096.0,
+ "664": 465988096.0,
+ "665": 465988096.0,
+ "666": 465988096.0,
+ "667": 465988096.0,
+ "668": 465988096.0,
+ "669": 465988096.0,
+ "670": 465988096.0,
+ "671": 465988096.0,
+ "672": 465988096.0,
+ "673": 465988096.0,
+ "674": 465988096.0,
+ "675": 465988096.0,
+ "676": 465988096.0,
+ "677": 465988096.0,
+ "678": 465988096.0,
+ "679": 465988096.0,
+ "680": 465988096.0,
+ "681": 465988096.0,
+ "682": 465988096.0,
+ "683": 465988096.0,
+ "684": 465988096.0,
+ "685": 465988096.0,
+ "686": 465988096.0,
+ "687": 465988096.0,
+ "688": 465988096.0,
+ "689": 465988096.0,
+ "690": 465988096.0,
+ "691": 465988096.0,
+ "692": 465988096.0,
+ "693": 465988096.0,
+ "694": 465988096.0,
+ "695": 465988096.0,
+ "696": 465988096.0,
+ "697": 465988096.0,
+ "698": 465988096.0,
+ "699": 465988096.0,
+ "700": 465988096.0,
+ "701": 465988096.0,
+ "702": 465988096.0,
+ "703": 465988096.0,
+ "704": 465988096.0,
+ "705": 465988096.0,
+ "706": 465988096.0,
+ "707": 465988096.0,
+ "708": 465988096.0,
+ "709": 465988096.0,
+ "710": 465988096.0,
+ "711": 465988096.0,
+ "712": 465988096.0,
+ "713": 465988096.0,
+ "714": 465988096.0,
+ "715": 465988096.0,
+ "716": 465988096.0,
+ "717": 465988096.0,
+ "718": 465988096.0,
+ "719": 465988096.0,
+ "720": 465988096.0,
+ "721": 465988096.0,
+ "722": 465988096.0,
+ "723": 465988096.0,
+ "724": 465988096.0,
+ "725": 465988096.0,
+ "726": 465988096.0,
+ "727": 465988096.0,
+ "728": 465988096.0,
+ "729": 465988096.0,
+ "730": 465988096.0,
+ "731": 465988096.0,
+ "732": 465988096.0,
+ "733": 465988096.0,
+ "734": 465988096.0,
+ "735": 465988096.0,
+ "736": 465988096.0,
+ "737": 465988096.0,
+ "738": 465988096.0,
+ "739": 465988096.0,
+ "740": 465988096.0,
+ "741": 465988096.0,
+ "742": 465988096.0,
+ "743": 465988096.0,
+ "744": 465988096.0,
+ "745": 465988096.0,
+ "746": 465988096.0,
+ "747": 465988096.0,
+ "748": 465988096.0,
+ "749": 465988096.0,
+ "750": 465988096.0,
+ "751": 465988096.0,
+ "752": 465988096.0,
+ "753": 465988096.0,
+ "754": 465988096.0,
+ "755": 465988096.0,
+ "756": 465988096.0,
+ "757": 465988096.0,
+ "758": 465988096.0,
+ "759": 465988096.0,
+ "760": 465988096.0,
+ "761": 465988096.0,
+ "762": 465988096.0,
+ "763": 465988096.0,
+ "764": 465988096.0,
+ "765": 465988096.0,
+ "766": 465988096.0,
+ "767": 465988096.0,
+ "768": 465988096.0,
+ "769": 465988096.0,
+ "770": 465988096.0,
+ "771": 465988096.0,
+ "772": 465988096.0,
+ "773": 465988096.0,
+ "774": 465988096.0,
+ "775": 465988096.0,
+ "776": 465988096.0,
+ "777": 465988096.0,
+ "778": 465988096.0,
+ "779": 465988096.0,
+ "780": 465988096.0,
+ "781": 465988096.0,
+ "782": 465988096.0,
+ "783": 465988096.0,
+ "784": 465988096.0,
+ "785": 465988096.0,
+ "786": 465988096.0,
+ "787": 465988096.0,
+ "788": 465988096.0,
+ "789": 465988096.0,
+ "790": 465988096.0,
+ "791": 465988096.0,
+ "792": 465988096.0,
+ "793": 465988096.0,
+ "794": 465988096.0,
+ "795": 465988096.0,
+ "796": 465988096.0,
+ "797": 465988096.0,
+ "798": 465988096.0,
+ "799": 465988096.0,
+ "800": 465988096.0,
+ "801": 465988096.0,
+ "802": 465988096.0,
+ "803": 465988096.0,
+ "804": 465988096.0,
+ "805": 465988096.0,
+ "806": 465988096.0,
+ "807": 465988096.0,
+ "808": 465988096.0,
+ "809": 465988096.0,
+ "810": 465988096.0,
+ "811": 465988096.0,
+ "812": 465988096.0,
+ "813": 465988096.0,
+ "814": 465988096.0,
+ "815": 465988096.0,
+ "816": 465988096.0,
+ "817": 465988096.0,
+ "818": 465988096.0,
+ "819": 465988096.0,
+ "820": 465988096.0,
+ "821": 465988096.0,
+ "822": 465988096.0,
+ "823": 465988096.0,
+ "824": 465988096.0,
+ "825": 465988096.0,
+ "826": 465988096.0,
+ "827": 465988096.0,
+ "828": 465988096.0,
+ "829": 465988096.0,
+ "830": 465988096.0,
+ "831": 465988096.0,
+ "832": 465988096.0,
+ "833": 465988096.0,
+ "834": 465988096.0,
+ "835": 465988096.0,
+ "836": 465988096.0,
+ "837": 465988096.0,
+ "838": 465988096.0,
+ "839": 465988096.0,
+ "840": 465988096.0,
+ "841": 465988096.0,
+ "842": 465988096.0,
+ "843": 465988096.0,
+ "844": 465988096.0,
+ "845": 465988096.0,
+ "846": 465988096.0,
+ "847": 465988096.0,
+ "848": 465988096.0,
+ "849": 465988096.0,
+ "850": 465988096.0,
+ "851": 465988096.0,
+ "852": 465988096.0,
+ "853": 465988096.0,
+ "854": 465988096.0,
+ "855": 465988096.0,
+ "856": 465988096.0,
+ "857": 465988096.0,
+ "858": 465988096.0,
+ "859": 465988096.0,
+ "860": 465988096.0,
+ "861": 465988096.0,
+ "862": 465988096.0,
+ "863": 465988096.0,
+ "864": 465988096.0,
+ "865": 465988096.0,
+ "866": 465988096.0,
+ "867": 465988096.0,
+ "868": 465988096.0,
+ "869": 465988096.0,
+ "870": 465988096.0,
+ "871": 465988096.0,
+ "872": 465988096.0,
+ "873": 465988096.0,
+ "874": 465988096.0,
+ "875": 465988096.0,
+ "876": 465988096.0,
+ "877": 465988096.0,
+ "878": 465988096.0,
+ "879": 465988096.0,
+ "880": 465988096.0,
+ "881": 465988096.0,
+ "882": 465988096.0,
+ "883": 465988096.0,
+ "884": 465988096.0,
+ "885": 465988096.0,
+ "886": 465988096.0,
+ "887": 465988096.0,
+ "888": 465988096.0,
+ "889": 465988096.0,
+ "890": 465988096.0,
+ "891": 465988096.0,
+ "892": 465988096.0,
+ "893": 465988096.0,
+ "894": 465988096.0,
+ "895": 465988096.0,
+ "896": 465988096.0,
+ "897": 465988096.0,
+ "898": 465988096.0,
+ "899": 465988096.0,
+ "900": 465988096.0,
+ "901": 465988096.0,
+ "902": 465988096.0,
+ "903": 465988096.0,
+ "904": 465988096.0,
+ "905": 465988096.0,
+ "906": 465988096.0,
+ "907": 465988096.0,
+ "908": 465988096.0,
+ "909": 465988096.0,
+ "910": 465988096.0,
+ "911": 465988096.0,
+ "912": 465988096.0,
+ "913": 465988096.0,
+ "914": 465988096.0,
+ "915": 465988096.0,
+ "916": 465988096.0,
+ "917": 465988096.0,
+ "918": 465988096.0,
+ "919": 465988096.0,
+ "920": 465988096.0,
+ "921": 465988096.0,
+ "922": 465988096.0,
+ "923": 465988096.0,
+ "924": 465988096.0,
+ "925": 465988096.0,
+ "926": 465988096.0,
+ "927": 465988096.0,
+ "928": 465988096.0,
+ "929": 465988096.0,
+ "930": 465988096.0,
+ "931": 465988096.0,
+ "932": 465988096.0,
+ "933": 465988096.0,
+ "934": 465988096.0,
+ "935": 465988096.0,
+ "936": 465988096.0,
+ "937": 465988096.0,
+ "938": 465988096.0,
+ "939": 465988096.0,
+ "940": 465988096.0,
+ "941": 465988096.0,
+ "942": 465988096.0,
+ "943": 465988096.0,
+ "944": 465988096.0,
+ "945": 465988096.0,
+ "946": 465988096.0,
+ "947": 465988096.0,
+ "948": 465988096.0,
+ "949": 465988096.0,
+ "950": 465988096.0,
+ "951": 465988096.0,
+ "952": 465988096.0,
+ "953": 465988096.0,
+ "954": 465988096.0,
+ "955": 465988096.0,
+ "956": 465988096.0,
+ "957": 465988096.0,
+ "958": 465988096.0,
+ "959": 465988096.0,
+ "960": 465988096.0,
+ "961": 465988096.0,
+ "962": 465988096.0,
+ "963": 465988096.0,
+ "964": 465988096.0,
+ "965": 465988096.0,
+ "966": 465988096.0,
+ "967": 465988096.0,
+ "968": 465988096.0,
+ "969": 465988096.0,
+ "970": 465988096.0,
+ "971": 465988096.0,
+ "972": 465988096.0,
+ "973": 465988096.0,
+ "974": 465988096.0,
+ "975": 465988096.0,
+ "976": 465988096.0,
+ "977": 465988096.0,
+ "978": 465988096.0,
+ "979": 465988096.0,
+ "980": 465988096.0,
+ "981": 465988096.0,
+ "982": 465988096.0,
+ "983": 465988096.0,
+ "984": 465988096.0,
+ "985": 465988096.0,
+ "986": 465988096.0,
+ "987": 465988096.0,
+ "988": 465988096.0,
+ "989": 465988096.0,
+ "990": 465988096.0,
+ "991": 465988096.0,
+ "992": 465988096.0,
+ "993": 465988096.0,
+ "994": 465988096.0,
+ "995": 465988096.0,
+ "996": 465988096.0,
+ "997": 465988096.0,
+ "998": 465988096.0,
+ "999": 465988096.0,
+ "1000": 465988096.0,
+ "1001": 465988096.0,
+ "1002": 465988096.0,
+ "1003": 465988096.0,
+ "1004": 465988096.0,
+ "1005": 465988096.0,
+ "1006": 465988096.0,
+ "1007": 465988096.0,
+ "1008": 465988096.0,
+ "1009": 465988096.0,
+ "1010": 465988096.0,
+ "1011": 465988096.0,
+ "1012": 465988096.0,
+ "1013": 465988096.0,
+ "1014": 465988096.0,
+ "1015": 465988096.0,
+ "1016": 465988096.0,
+ "1017": 465988096.0,
+ "1018": 465988096.0,
+ "1019": 465988096.0,
+ "1020": 465988096.0,
+ "1021": 465988096.0,
+ "1022": 465988096.0,
+ "1023": 465988096.0,
+ "1024": 465988096.0,
+ "1025": 465988096.0,
+ "1026": 465988096.0,
+ "1027": 465988096.0,
+ "1028": 465988096.0,
+ "1029": 465988096.0,
+ "1030": 465988096.0,
+ "1031": 465988096.0,
+ "1032": 465988096.0,
+ "1033": 465988096.0,
+ "1034": 465988096.0,
+ "1035": 465988096.0,
+ "1036": 465988096.0,
+ "1037": 465988096.0,
+ "1038": 465988096.0,
+ "1039": 465988096.0,
+ "1040": 465988096.0,
+ "1041": 465988096.0,
+ "1042": 465988096.0,
+ "1043": 465988096.0,
+ "1044": 465988096.0,
+ "1045": 465988096.0,
+ "1046": 465988096.0,
+ "1047": 465988096.0,
+ "1048": 465988096.0,
+ "1049": 465988096.0,
+ "1050": 465988096.0,
+ "1051": 465988096.0,
+ "1052": 465988096.0,
+ "1053": 465988096.0,
+ "1054": 465988096.0,
+ "1055": 465988096.0,
+ "1056": 465988096.0,
+ "1057": 465988096.0,
+ "1058": 465988096.0,
+ "1059": 465988096.0,
+ "1060": 465988096.0,
+ "1061": 465988096.0,
+ "1062": 465988096.0,
+ "1063": 465988096.0,
+ "1064": 465988096.0,
+ "1065": 465988096.0,
+ "1066": 465988096.0,
+ "1067": 465988096.0,
+ "1068": 465988096.0,
+ "1069": 465988096.0,
+ "1070": 465988096.0,
+ "1071": 465988096.0,
+ "1072": 465988096.0,
+ "1073": 465988096.0,
+ "1074": 465988096.0,
+ "1075": 465988096.0,
+ "1076": 465988096.0,
+ "1077": 465988096.0,
+ "1078": 465988096.0,
+ "1079": 465988096.0,
+ "1080": 465988096.0,
+ "1081": 465988096.0,
+ "1082": 465988096.0,
+ "1083": 465988096.0,
+ "1084": 465988096.0,
+ "1085": 465988096.0,
+ "1086": 465988096.0,
+ "1087": 465988096.0,
+ "1088": 465988096.0,
+ "1089": 465988096.0,
+ "1090": 465988096.0,
+ "1091": 465988096.0,
+ "1092": 465988096.0,
+ "1093": 465988096.0,
+ "1094": 465988096.0,
+ "1095": 465988096.0,
+ "1096": 465988096.0,
+ "1097": 465988096.0,
+ "1098": 465988096.0,
+ "1099": 465988096.0,
+ "1100": 465988096.0,
+ "1101": 465988096.0,
+ "1102": 465988096.0,
+ "1103": 465988096.0,
+ "1104": 465988096.0,
+ "1105": 465988096.0,
+ "1106": 465988096.0,
+ "1107": 465988096.0,
+ "1108": 465988096.0,
+ "1109": 465988096.0,
+ "1110": 465988096.0,
+ "1111": 465988096.0,
+ "1112": 465988096.0,
+ "1113": 465988096.0,
+ "1114": 465988096.0,
+ "1115": 465988096.0,
+ "1116": 465988096.0,
+ "1117": 465988096.0,
+ "1118": 465988096.0,
+ "1119": 465988096.0,
+ "1120": 465988096.0,
+ "1121": 465988096.0,
+ "1122": 465988096.0,
+ "1123": 465988096.0,
+ "1124": 465988096.0,
+ "1125": 465988096.0,
+ "1126": 465988096.0,
+ "1127": 465988096.0,
+ "1128": 465988096.0,
+ "1129": 465988096.0,
+ "1130": 465988096.0,
+ "1131": 465988096.0,
+ "1132": 465988096.0,
+ "1133": 465988096.0,
+ "1134": 465988096.0,
+ "1135": 465988096.0,
+ "1136": 465988096.0,
+ "1137": 465988096.0,
+ "1138": 465988096.0,
+ "1139": 465988096.0,
+ "1140": 465988096.0,
+ "1141": 465988096.0,
+ "1142": 465988096.0,
+ "1143": 465988096.0,
+ "1144": 465988096.0,
+ "1145": 465988096.0,
+ "1146": 465988096.0,
+ "1147": 465988096.0,
+ "1148": 465988096.0,
+ "1149": 465988096.0,
+ "1150": 465988096.0,
+ "1151": 465988096.0,
+ "1152": 465988096.0,
+ "1153": 465988096.0,
+ "1154": 465988096.0,
+ "1155": 465988096.0,
+ "1156": 465988096.0,
+ "1157": 465988096.0,
+ "1158": 465988096.0,
+ "1159": 465988096.0,
+ "1160": 465988096.0,
+ "1161": 465988096.0,
+ "1162": 465988096.0,
+ "1163": 465988096.0,
+ "1164": 465988096.0,
+ "1165": 465988096.0,
+ "1166": 465988096.0,
+ "1167": 465988096.0,
+ "1168": 465988096.0,
+ "1169": 465988096.0,
+ "1170": 465988096.0,
+ "1171": 465988096.0,
+ "1172": 465988096.0,
+ "1173": 465988096.0,
+ "1174": 465988096.0,
+ "1175": 465988096.0,
+ "1176": 465988096.0,
+ "1177": 465988096.0,
+ "1178": 465988096.0,
+ "1179": 465988096.0,
+ "1180": 465988096.0,
+ "1181": 465988096.0,
+ "1182": 465988096.0,
+ "1183": 465988096.0,
+ "1184": 465988096.0,
+ "1185": 465988096.0,
+ "1186": 465988096.0,
+ "1187": 465988096.0,
+ "1188": 465988096.0,
+ "1189": 465988096.0,
+ "1190": 465988096.0,
+ "1191": 465988096.0,
+ "1192": 465988096.0,
+ "1193": 465988096.0,
+ "1194": 465988096.0,
+ "1195": 465988096.0,
+ "1196": 465988096.0,
+ "1197": 465988096.0,
+ "1198": 465988096.0,
+ "1199": 465988096.0,
+ "1200": 465988096.0,
+ "1201": 465988096.0,
+ "1202": 465988096.0,
+ "1203": 465988096.0,
+ "1204": 465988096.0,
+ "1205": 465988096.0,
+ "1206": 465988096.0,
+ "1207": 465988096.0,
+ "1208": 465988096.0,
+ "1209": 465988096.0,
+ "1210": 465988096.0,
+ "1211": 465988096.0,
+ "1212": 465988096.0,
+ "1213": 465988096.0,
+ "1214": 465988096.0,
+ "1215": 465988096.0,
+ "1216": 465988096.0,
+ "1217": 465988096.0,
+ "1218": 465988096.0,
+ "1219": 465988096.0,
+ "1220": 465988096.0,
+ "1221": 465988096.0,
+ "1222": 465988096.0,
+ "1223": 465988096.0,
+ "1224": 465988096.0,
+ "1225": 465988096.0,
+ "1226": 465988096.0,
+ "1227": 465988096.0,
+ "1228": 465988096.0,
+ "1229": 465988096.0,
+ "1230": 465988096.0,
+ "1231": 465988096.0,
+ "1232": 465988096.0,
+ "1233": 465988096.0,
+ "1234": 465988096.0,
+ "1235": 465988096.0,
+ "1236": 465988096.0,
+ "1237": 465988096.0,
+ "1238": 465988096.0,
+ "1239": 465988096.0,
+ "1240": 465988096.0,
+ "1241": 465988096.0,
+ "1242": 465988096.0,
+ "1243": 465988096.0,
+ "1244": 465988096.0,
+ "1245": 465988096.0,
+ "1246": 465988096.0,
+ "1247": 465988096.0,
+ "1248": 465988096.0,
+ "1249": 465988096.0,
+ "1250": 465988096.0,
+ "1251": 465988096.0,
+ "1252": 465988096.0,
+ "1253": 465988096.0,
+ "1254": 465988096.0,
+ "1255": 465988096.0,
+ "1256": 465988096.0,
+ "1257": 465988096.0,
+ "1258": 465988096.0,
+ "1259": 465988096.0,
+ "1260": 465988096.0,
+ "1261": 465988096.0,
+ "1262": 465988096.0,
+ "1263": 465988096.0,
+ "1264": 465988096.0,
+ "1265": 465988096.0,
+ "1266": 465988096.0,
+ "1267": 465988096.0,
+ "1268": 465988096.0,
+ "1269": 465988096.0,
+ "1270": 465988096.0,
+ "1271": 465988096.0,
+ "1272": 465988096.0,
+ "1273": 465988096.0,
+ "1274": 465988096.0,
+ "1275": 465988096.0,
+ "1276": 465988096.0,
+ "1277": 465988096.0,
+ "1278": 465988096.0,
+ "1279": 465988096.0,
+ "1280": 465988096.0,
+ "1281": 465988096.0,
+ "1282": 465988096.0,
+ "1283": 465988096.0,
+ "1284": 465988096.0,
+ "1285": 465988096.0,
+ "1286": 465988096.0,
+ "1287": 465988096.0,
+ "1288": 465988096.0,
+ "1289": 465988096.0,
+ "1290": 465988096.0,
+ "1291": 465988096.0,
+ "1292": 465988096.0,
+ "1293": 465988096.0,
+ "1294": 465988096.0,
+ "1295": 465988096.0,
+ "1296": 465988096.0,
+ "1297": 465988096.0,
+ "1298": 465988096.0,
+ "1299": 465988096.0,
+ "1300": 465988096.0,
+ "1301": 465988096.0,
+ "1302": 465988096.0,
+ "1303": 465988096.0,
+ "1304": 465988096.0,
+ "1305": 465988096.0,
+ "1306": 465988096.0,
+ "1307": 465988096.0,
+ "1308": 465988096.0,
+ "1309": 465988096.0,
+ "1310": 465988096.0,
+ "1311": 465988096.0,
+ "1312": 465988096.0,
+ "1313": 465988096.0,
+ "1314": 465988096.0,
+ "1315": 465988096.0,
+ "1316": 465988096.0,
+ "1317": 465988096.0,
+ "1318": 465988096.0,
+ "1319": 465988096.0,
+ "1320": 465988096.0,
+ "1321": 465988096.0,
+ "1322": 465988096.0,
+ "1323": 465988096.0,
+ "1324": 465988096.0,
+ "1325": 465988096.0,
+ "1326": 465988096.0,
+ "1327": 465988096.0,
+ "1328": 465988096.0,
+ "1329": 465988096.0,
+ "1330": 465988096.0,
+ "1331": 465988096.0,
+ "1332": 465988096.0,
+ "1333": 465988096.0,
+ "1334": 465988096.0,
+ "1335": 465988096.0,
+ "1336": 465988096.0,
+ "1337": 465988096.0,
+ "1338": 465988096.0,
+ "1339": 465988096.0,
+ "1340": 465988096.0,
+ "1341": 465988096.0,
+ "1342": 465988096.0,
+ "1343": 465988096.0,
+ "1344": 465988096.0,
+ "1345": 465988096.0,
+ "1346": 465988096.0,
+ "1347": 465988096.0,
+ "1348": 465988096.0,
+ "1349": 465988096.0,
+ "1350": 465988096.0,
+ "1351": 465988096.0,
+ "1352": 465988096.0,
+ "1353": 465988096.0,
+ "1354": 465988096.0,
+ "1355": 465988096.0,
+ "1356": 465988096.0,
+ "1357": 465988096.0,
+ "1358": 465988096.0,
+ "1359": 465988096.0,
+ "1360": 465988096.0,
+ "1361": 465988096.0,
+ "1362": 465988096.0,
+ "1363": 465988096.0,
+ "1364": 465988096.0,
+ "1365": 465988096.0,
+ "1366": 465988096.0,
+ "1367": 465988096.0,
+ "1368": 465988096.0,
+ "1369": 465988096.0,
+ "1370": 465988096.0,
+ "1371": 465988096.0,
+ "1372": 465988096.0,
+ "1373": 465988096.0,
+ "1374": 465988096.0,
+ "1375": 465988096.0,
+ "1376": 465988096.0,
+ "1377": 465988096.0,
+ "1378": 465988096.0,
+ "1379": 465988096.0,
+ "1380": 465988096.0,
+ "1381": 465988096.0,
+ "1382": 465988096.0,
+ "1383": 465988096.0,
+ "1384": 465988096.0,
+ "1385": 465988096.0,
+ "1386": 465988096.0,
+ "1387": 465988096.0,
+ "1388": 465988096.0,
+ "1389": 465988096.0,
+ "1390": 465988096.0,
+ "1391": 465988096.0,
+ "1392": 465988096.0,
+ "1393": 465988096.0,
+ "1394": 465988096.0,
+ "1395": 465988096.0,
+ "1396": 465988096.0,
+ "1397": 465988096.0,
+ "1398": 465988096.0,
+ "1399": 465988096.0,
+ "1400": 465988096.0,
+ "1401": 465988096.0,
+ "1402": 465988096.0,
+ "1403": 465988096.0,
+ "1404": 465988096.0,
+ "1405": 465988096.0,
+ "1406": 465988096.0,
+ "1407": 465988096.0,
+ "1408": 465988096.0,
+ "1409": 465988096.0,
+ "1410": 465988096.0,
+ "1411": 465988096.0,
+ "1412": 465988096.0,
+ "1413": 465988096.0,
+ "1414": 465988096.0,
+ "1415": 465988096.0,
+ "1416": 465988096.0,
+ "1417": 465988096.0,
+ "1418": 465988096.0,
+ "1419": 465988096.0,
+ "1420": 465988096.0,
+ "1421": 465988096.0,
+ "1422": 465988096.0,
+ "1423": 465988096.0,
+ "1424": 465988096.0,
+ "1425": 465988096.0,
+ "1426": 465988096.0,
+ "1427": 465988096.0,
+ "1428": 465988096.0,
+ "1429": 465988096.0,
+ "1430": 465988096.0,
+ "1431": 465988096.0,
+ "1432": 465988096.0,
+ "1433": 465988096.0,
+ "1434": 465988096.0,
+ "1435": 465988096.0,
+ "1436": 465988096.0,
+ "1437": 465988096.0,
+ "1438": 465988096.0,
+ "1439": 465988096.0,
+ "1440": 465988096.0,
+ "1441": 465988096.0,
+ "1442": 465988096.0,
+ "1443": 465988096.0,
+ "1444": 465988096.0,
+ "1445": 465988096.0,
+ "1446": 465988096.0,
+ "1447": 465988096.0,
+ "1448": 465988096.0,
+ "1449": 465988096.0,
+ "1450": 465988096.0,
+ "1451": 465988096.0,
+ "1452": 465988096.0,
+ "1453": 465988096.0,
+ "1454": 465988096.0,
+ "1455": 465988096.0,
+ "1456": 465988096.0,
+ "1457": 465988096.0,
+ "1458": 465988096.0,
+ "1459": 465988096.0,
+ "1460": 465988096.0,
+ "1461": 465988096.0,
+ "1462": 465988096.0,
+ "1463": 465988096.0,
+ "1464": 465988096.0,
+ "1465": 465988096.0,
+ "1466": 465988096.0,
+ "1467": 465988096.0,
+ "1468": 465988096.0,
+ "1469": 465988096.0,
+ "1470": 465988096.0,
+ "1471": 465988096.0,
+ "1472": 465988096.0,
+ "1473": 465988096.0,
+ "1474": 465988096.0,
+ "1475": 465988096.0,
+ "1476": 465988096.0,
+ "1477": 465988096.0,
+ "1478": 465988096.0,
+ "1479": 465988096.0,
+ "1480": 465988096.0,
+ "1481": 465988096.0,
+ "1482": 465988096.0,
+ "1483": 465988096.0,
+ "1484": 465988096.0,
+ "1485": 465988096.0,
+ "1486": 465988096.0,
+ "1487": 465988096.0,
+ "1488": 465988096.0,
+ "1489": 465988096.0,
+ "1490": 465988096.0,
+ "1491": 465988096.0,
+ "1492": 465988096.0,
+ "1493": 465988096.0,
+ "1494": 465988096.0,
+ "1495": 465988096.0,
+ "1496": 465988096.0,
+ "1497": 465988096.0,
+ "1498": 465988096.0,
+ "1499": 465988096.0,
+ "1500": 465988096.0,
+ "1501": 465988096.0,
+ "1502": 465988096.0,
+ "1503": 465988096.0,
+ "1504": 465988096.0,
+ "1505": 465988096.0,
+ "1506": 465988096.0,
+ "1507": 465988096.0,
+ "1508": 465988096.0,
+ "1509": 465988096.0,
+ "1510": 465988096.0,
+ "1511": 465988096.0,
+ "1512": 465988096.0,
+ "1513": 465988096.0,
+ "1514": 465988096.0,
+ "1515": 465988096.0,
+ "1516": 465988096.0,
+ "1517": 465988096.0,
+ "1518": 465988096.0,
+ "1519": 465988096.0,
+ "1520": 465988096.0,
+ "1521": 465988096.0,
+ "1522": 465988096.0,
+ "1523": 465988096.0,
+ "1524": 465988096.0,
+ "1525": 465988096.0,
+ "1526": 465988096.0,
+ "1527": 465988096.0,
+ "1528": 465988096.0,
+ "1529": 465988096.0,
+ "1530": 465988096.0,
+ "1531": 465988096.0,
+ "1532": 465988096.0,
+ "1533": 465988096.0,
+ "1534": 465988096.0,
+ "1535": 465988096.0,
+ "1536": 465988096.0,
+ "1537": 465988096.0,
+ "1538": 465988096.0,
+ "1539": 465988096.0,
+ "1540": 465988096.0,
+ "1541": 465988096.0,
+ "1542": 465988096.0,
+ "1543": 465988096.0,
+ "1544": 465988096.0,
+ "1545": 465988096.0,
+ "1546": 465988096.0,
+ "1547": 465988096.0,
+ "1548": 465988096.0,
+ "1549": 465988096.0,
+ "1550": 465988096.0,
+ "1551": 465988096.0,
+ "1552": 465988096.0,
+ "1553": 465988096.0,
+ "1554": 465988096.0,
+ "1555": 465988096.0,
+ "1556": 465988096.0,
+ "1557": 465988096.0,
+ "1558": 465988096.0,
+ "1559": 465988096.0,
+ "1560": 465988096.0,
+ "1561": 465988096.0,
+ "1562": 465988096.0,
+ "1563": 465988096.0,
+ "1564": 465988096.0,
+ "1565": 465988096.0,
+ "1566": 465988096.0,
+ "1567": 465988096.0,
+ "1568": 465988096.0,
+ "1569": 465988096.0,
+ "1570": 465988096.0,
+ "1571": 465988096.0,
+ "1572": 465988096.0,
+ "1573": 465988096.0,
+ "1574": 465988096.0,
+ "1575": 465988096.0,
+ "1576": 465988096.0,
+ "1577": 465988096.0,
+ "1578": 465988096.0,
+ "1579": 465988096.0,
+ "1580": 465988096.0,
+ "1581": 465988096.0,
+ "1582": 465988096.0,
+ "1583": 465988096.0,
+ "1584": 465988096.0,
+ "1585": 465988096.0,
+ "1586": 465988096.0,
+ "1587": 465988096.0,
+ "1588": 465988096.0,
+ "1589": 465988096.0,
+ "1590": 465988096.0,
+ "1591": 465988096.0,
+ "1592": 465988096.0,
+ "1593": 465988096.0,
+ "1594": 465988096.0,
+ "1595": 465988096.0,
+ "1596": 465988096.0,
+ "1597": 465988096.0,
+ "1598": 465988096.0,
+ "1599": 465988096.0,
+ "1600": 465988096.0,
+ "1601": 465988096.0,
+ "1602": 465988096.0,
+ "1603": 465988096.0,
+ "1604": 465988096.0,
+ "1605": 465988096.0,
+ "1606": 465988096.0,
+ "1607": 465988096.0,
+ "1608": 465988096.0,
+ "1609": 465988096.0,
+ "1610": 465988096.0,
+ "1611": 465988096.0,
+ "1612": 465988096.0,
+ "1613": 465988096.0,
+ "1614": 465988096.0,
+ "1615": 465988096.0,
+ "1616": 465988096.0,
+ "1617": 465988096.0,
+ "1618": 465988096.0,
+ "1619": 465988096.0,
+ "1620": 465988096.0,
+ "1621": 465988096.0,
+ "1622": 465988096.0,
+ "1623": 465988096.0,
+ "1624": 465988096.0,
+ "1625": 465988096.0,
+ "1626": 465988096.0,
+ "1627": 465988096.0,
+ "1628": 465988096.0,
+ "1629": 465988096.0,
+ "1630": 465988096.0,
+ "1631": 465988096.0,
+ "1632": 465988096.0,
+ "1633": 465988096.0,
+ "1634": 465988096.0,
+ "1635": 465988096.0,
+ "1636": 465988096.0,
+ "1637": 465988096.0,
+ "1638": 465988096.0,
+ "1639": 465988096.0,
+ "1640": 465988096.0,
+ "1641": 465988096.0,
+ "1642": 465988096.0,
+ "1643": 465988096.0,
+ "1644": 465988096.0,
+ "1645": 465988096.0,
+ "1646": 465988096.0,
+ "1647": 465988096.0,
+ "1648": 465988096.0,
+ "1649": 465988096.0,
+ "1650": 465988096.0,
+ "1651": 465988096.0,
+ "1652": 465988096.0,
+ "1653": 465988096.0,
+ "1654": 465988096.0,
+ "1655": 465988096.0,
+ "1656": 465988096.0,
+ "1657": 465988096.0,
+ "1658": 465988096.0,
+ "1659": 465988096.0,
+ "1660": 465988096.0,
+ "1661": 465988096.0,
+ "1662": 465988096.0,
+ "1663": 465988096.0,
+ "1664": 465988096.0,
+ "1665": 465988096.0,
+ "1666": 465988096.0,
+ "1667": 465988096.0,
+ "1668": 465988096.0,
+ "1669": 465988096.0,
+ "1670": 465988096.0,
+ "1671": 465988096.0,
+ "1672": 465988096.0,
+ "1673": 465988096.0,
+ "1674": 465988096.0,
+ "1675": 465988096.0,
+ "1676": 465988096.0,
+ "1677": 465988096.0,
+ "1678": 465988096.0,
+ "1679": 465988096.0,
+ "1680": 465988096.0,
+ "1681": 465988096.0,
+ "1682": 465988096.0,
+ "1683": 465988096.0,
+ "1684": 465988096.0,
+ "1685": 465988096.0,
+ "1686": 465988096.0,
+ "1687": 465988096.0,
+ "1688": 465988096.0,
+ "1689": 465988096.0,
+ "1690": 465988096.0,
+ "1691": 465988096.0,
+ "1692": 465988096.0,
+ "1693": 465988096.0,
+ "1694": 465988096.0,
+ "1695": 465988096.0,
+ "1696": 465988096.0,
+ "1697": 465988096.0,
+ "1698": 465988096.0,
+ "1699": 465988096.0,
+ "1700": 465988096.0,
+ "1701": 465988096.0,
+ "1702": 465988096.0,
+ "1703": 465988096.0,
+ "1704": 465988096.0,
+ "1705": 465988096.0,
+ "1706": 465988096.0,
+ "1707": 465988096.0,
+ "1708": 465988096.0,
+ "1709": 465988096.0,
+ "1710": 465988096.0,
+ "1711": 465988096.0,
+ "1712": 465988096.0,
+ "1713": 465988096.0,
+ "1714": 465988096.0,
+ "1715": 465988096.0,
+ "1716": 465988096.0,
+ "1717": 465988096.0,
+ "1718": 465988096.0,
+ "1719": 465988096.0,
+ "1720": 465988096.0,
+ "1721": 465988096.0,
+ "1722": 465988096.0,
+ "1723": 465988096.0,
+ "1724": 465988096.0,
+ "1725": 465988096.0,
+ "1726": 465988096.0,
+ "1727": 465988096.0,
+ "1728": 465988096.0,
+ "1729": 465988096.0,
+ "1730": 465988096.0,
+ "1731": 465988096.0,
+ "1732": 465988096.0,
+ "1733": 465988096.0,
+ "1734": 465988096.0,
+ "1735": 465988096.0,
+ "1736": 465988096.0,
+ "1737": 465988096.0,
+ "1738": 465988096.0,
+ "1739": 465988096.0,
+ "1740": 465988096.0,
+ "1741": 465988096.0,
+ "1742": 465988096.0,
+ "1743": 465988096.0,
+ "1744": 465988096.0,
+ "1745": 465988096.0,
+ "1746": 465988096.0,
+ "1747": 465988096.0,
+ "1748": 465988096.0,
+ "1749": 465988096.0,
+ "1750": 465988096.0,
+ "1751": 465988096.0,
+ "1752": 465988096.0,
+ "1753": 465988096.0,
+ "1754": 465988096.0,
+ "1755": 465988096.0,
+ "1756": 465988096.0,
+ "1757": 465988096.0,
+ "1758": 465988096.0,
+ "1759": 465988096.0,
+ "1760": 465988096.0,
+ "1761": 465988096.0,
+ "1762": 465988096.0,
+ "1763": 465988096.0,
+ "1764": 465988096.0,
+ "1765": 465988096.0,
+ "1766": 465988096.0,
+ "1767": 465988096.0,
+ "1768": 465988096.0,
+ "1769": 465988096.0,
+ "1770": 465988096.0,
+ "1771": 465988096.0,
+ "1772": 465988096.0,
+ "1773": 465988096.0,
+ "1774": 465988096.0,
+ "1775": 465988096.0,
+ "1776": 465988096.0,
+ "1777": 465988096.0,
+ "1778": 465988096.0,
+ "1779": 465988096.0,
+ "1780": 465988096.0,
+ "1781": 465988096.0,
+ "1782": 465988096.0,
+ "1783": 465988096.0,
+ "1784": 465988096.0,
+ "1785": 465988096.0,
+ "1786": 465988096.0,
+ "1787": 465988096.0,
+ "1788": 465988096.0,
+ "1789": 465988096.0,
+ "1790": 465988096.0,
+ "1791": 465988096.0,
+ "1792": 465988096.0,
+ "1793": 465988096.0,
+ "1794": 465988096.0,
+ "1795": 465988096.0,
+ "1796": 465988096.0,
+ "1797": 465988096.0,
+ "1798": 465988096.0,
+ "1799": 465988096.0,
+ "1800": 465988096.0,
+ "1801": 465988096.0,
+ "1802": 465988096.0,
+ "1803": 465988096.0,
+ "1804": 465988096.0,
+ "1805": 465988096.0,
+ "1806": 465988096.0,
+ "1807": 465988096.0,
+ "1808": 465988096.0,
+ "1809": 465988096.0,
+ "1810": 465988096.0,
+ "1811": 465988096.0,
+ "1812": 465988096.0,
+ "1813": 465988096.0,
+ "1814": 465988096.0,
+ "1815": 465988096.0,
+ "1816": 465988096.0,
+ "1817": 465988096.0,
+ "1818": 465988096.0,
+ "1819": 465988096.0,
+ "1820": 465988096.0,
+ "1821": 465988096.0,
+ "1822": 465988096.0,
+ "1823": 465988096.0,
+ "1824": 465988096.0,
+ "1825": 465988096.0,
+ "1826": 465988096.0,
+ "1827": 465988096.0,
+ "1828": 465988096.0,
+ "1829": 465988096.0,
+ "1830": 465988096.0,
+ "1831": 465988096.0,
+ "1832": 465988096.0,
+ "1833": 465988096.0,
+ "1834": 465988096.0,
+ "1835": 465988096.0,
+ "1836": 465988096.0,
+ "1837": 465988096.0,
+ "1838": 465988096.0,
+ "1839": 465988096.0,
+ "1840": 465988096.0,
+ "1841": 465988096.0,
+ "1842": 465988096.0,
+ "1843": 465988096.0,
+ "1844": 465988096.0,
+ "1845": 465988096.0,
+ "1846": 465988096.0,
+ "1847": 465988096.0,
+ "1848": 465988096.0,
+ "1849": 465988096.0,
+ "1850": 465988096.0,
+ "1851": 465988096.0,
+ "1852": 465988096.0,
+ "1853": 465988096.0,
+ "1854": 465988096.0,
+ "1855": 465988096.0,
+ "1856": 465988096.0,
+ "1857": 465988096.0,
+ "1858": 465988096.0,
+ "1859": 465988096.0,
+ "1860": 465988096.0,
+ "1861": 465988096.0,
+ "1862": 465988096.0,
+ "1863": 465988096.0,
+ "1864": 465988096.0,
+ "1865": 465988096.0,
+ "1866": 465988096.0,
+ "1867": 465988096.0,
+ "1868": 465988096.0,
+ "1869": 465988096.0,
+ "1870": 465988096.0,
+ "1871": 465988096.0,
+ "1872": 465988096.0,
+ "1873": 465988096.0,
+ "1874": 465988096.0,
+ "1875": 465988096.0,
+ "1876": 465988096.0,
+ "1877": 465988096.0,
+ "1878": 465988096.0,
+ "1879": 465988096.0,
+ "1880": 465988096.0,
+ "1881": 465988096.0,
+ "1882": 465988096.0,
+ "1883": 465988096.0,
+ "1884": 465988096.0,
+ "1885": 465988096.0,
+ "1886": 465988096.0,
+ "1887": 465988096.0,
+ "1888": 465988096.0,
+ "1889": 465988096.0,
+ "1890": 465988096.0,
+ "1891": 465988096.0,
+ "1892": 465988096.0,
+ "1893": 465988096.0,
+ "1894": 465988096.0,
+ "1895": 465988096.0,
+ "1896": 465988096.0,
+ "1897": 465988096.0,
+ "1898": 465988096.0,
+ "1899": 465988096.0,
+ "1900": 465988096.0,
+ "1901": 465988096.0,
+ "1902": 465988096.0,
+ "1903": 465988096.0,
+ "1904": 465988096.0,
+ "1905": 465988096.0,
+ "1906": 465988096.0,
+ "1907": 465988096.0,
+ "1908": 465988096.0,
+ "1909": 465988096.0,
+ "1910": 465988096.0,
+ "1911": 465988096.0,
+ "1912": 465988096.0,
+ "1913": 465988096.0,
+ "1914": 465988096.0,
+ "1915": 465988096.0,
+ "1916": 465988096.0,
+ "1917": 465988096.0,
+ "1918": 465988096.0,
+ "1919": 465988096.0,
+ "1920": 465988096.0,
+ "1921": 465988096.0,
+ "1922": 465988096.0,
+ "1923": 465988096.0,
+ "1924": 465988096.0,
+ "1925": 465988096.0,
+ "1926": 465988096.0,
+ "1927": 465988096.0,
+ "1928": 465988096.0,
+ "1929": 465988096.0,
+ "1930": 465988096.0,
+ "1931": 465988096.0,
+ "1932": 465988096.0,
+ "1933": 465988096.0,
+ "1934": 465988096.0,
+ "1935": 465988096.0,
+ "1936": 465988096.0,
+ "1937": 465988096.0,
+ "1938": 465988096.0,
+ "1939": 465988096.0,
+ "1940": 465988096.0,
+ "1941": 465988096.0,
+ "1942": 465988096.0,
+ "1943": 465988096.0,
+ "1944": 465988096.0,
+ "1945": 465988096.0,
+ "1946": 465988096.0,
+ "1947": 465988096.0,
+ "1948": 465988096.0,
+ "1949": 465988096.0,
+ "1950": 465988096.0,
+ "1951": 465988096.0,
+ "1952": 465988096.0,
+ "1953": 465988096.0,
+ "1954": 465988096.0,
+ "1955": 465988096.0,
+ "1956": 465988096.0,
+ "1957": 465988096.0,
+ "1958": 465988096.0,
+ "1959": 465988096.0,
+ "1960": 465988096.0,
+ "1961": 465988096.0,
+ "1962": 465988096.0,
+ "1963": 465988096.0,
+ "1964": 465988096.0,
+ "1965": 465988096.0,
+ "1966": 465988096.0,
+ "1967": 465988096.0,
+ "1968": 465988096.0,
+ "1969": 465988096.0,
+ "1970": 465988096.0,
+ "1971": 465988096.0,
+ "1972": 465988096.0,
+ "1973": 465988096.0,
+ "1974": 465988096.0,
+ "1975": 465988096.0,
+ "1976": 465988096.0,
+ "1977": 465988096.0,
+ "1978": 465988096.0,
+ "1979": 465988096.0,
+ "1980": 465988096.0,
+ "1981": 465988096.0,
+ "1982": 465988096.0,
+ "1983": 465988096.0,
+ "1984": 465988096.0,
+ "1985": 465988096.0,
+ "1986": 465988096.0,
+ "1987": 465988096.0,
+ "1988": 465988096.0,
+ "1989": 465988096.0,
+ "1990": 465988096.0,
+ "1991": 465988096.0,
+ "1992": 465988096.0,
+ "1993": 465988096.0,
+ "1994": 465988096.0,
+ "1995": 465988096.0,
+ "1996": 465988096.0,
+ "1997": 465988096.0,
+ "1998": 465988096.0,
+ "1999": 465988096.0,
+ "2000": 465988096.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 785829376.0,
+ "2": 876622336.0,
+ "3": 876622336.0,
+ "4": 876622336.0,
+ "5": 876622336.0,
+ "6": 876622336.0,
+ "7": 876622336.0,
+ "8": 876622336.0,
+ "9": 876622336.0,
+ "10": 876622336.0,
+ "11": 876622336.0,
+ "12": 876622336.0,
+ "13": 876622336.0,
+ "14": 876622336.0,
+ "15": 876622336.0,
+ "16": 876622336.0,
+ "17": 876622336.0,
+ "18": 876622336.0,
+ "19": 876622336.0,
+ "20": 876622336.0,
+ "21": 876622336.0,
+ "22": 876622336.0,
+ "23": 876622336.0,
+ "24": 876622336.0,
+ "25": 876622336.0,
+ "26": 876622336.0,
+ "27": 876622336.0,
+ "28": 876622336.0,
+ "29": 876622336.0,
+ "30": 876622336.0,
+ "31": 876622336.0,
+ "32": 876622336.0,
+ "33": 876622336.0,
+ "34": 876622336.0,
+ "35": 876622336.0,
+ "36": 876622336.0,
+ "37": 876622336.0,
+ "38": 876622336.0,
+ "39": 876622336.0,
+ "40": 876622336.0,
+ "41": 876622336.0,
+ "42": 876622336.0,
+ "43": 876622336.0,
+ "44": 876622336.0,
+ "45": 876622336.0,
+ "46": 876622336.0,
+ "47": 876622336.0,
+ "48": 876622336.0,
+ "49": 876622336.0,
+ "50": 876622336.0,
+ "51": 876622336.0,
+ "52": 876622336.0,
+ "53": 876622336.0,
+ "54": 876622336.0,
+ "55": 876622336.0,
+ "56": 876622336.0,
+ "57": 876622336.0,
+ "58": 876622336.0,
+ "59": 876622336.0,
+ "60": 876622336.0,
+ "61": 876622336.0,
+ "62": 876622336.0,
+ "63": 876622336.0,
+ "64": 876622336.0,
+ "65": 876622336.0,
+ "66": 876622336.0,
+ "67": 876622336.0,
+ "68": 876622336.0,
+ "69": 876622336.0,
+ "70": 876622336.0,
+ "71": 876622336.0,
+ "72": 876622336.0,
+ "73": 876622336.0,
+ "74": 876622336.0,
+ "75": 876622336.0,
+ "76": 876622336.0,
+ "77": 876622336.0,
+ "78": 876622336.0,
+ "79": 876622336.0,
+ "80": 876622336.0,
+ "81": 876622336.0,
+ "82": 876622336.0,
+ "83": 876622336.0,
+ "84": 876622336.0,
+ "85": 876622336.0,
+ "86": 876622336.0,
+ "87": 876622336.0,
+ "88": 876622336.0,
+ "89": 876622336.0,
+ "90": 876622336.0,
+ "91": 876622336.0,
+ "92": 876622336.0,
+ "93": 876622336.0,
+ "94": 876622336.0,
+ "95": 876622336.0,
+ "96": 876622336.0,
+ "97": 876622336.0,
+ "98": 876622336.0,
+ "99": 876622336.0,
+ "100": 876622336.0,
+ "101": 876622336.0,
+ "102": 876622336.0,
+ "103": 876622336.0,
+ "104": 876622336.0,
+ "105": 876622336.0,
+ "106": 876622336.0,
+ "107": 876622336.0,
+ "108": 876622336.0,
+ "109": 876622336.0,
+ "110": 876622336.0,
+ "111": 876622336.0,
+ "112": 876622336.0,
+ "113": 876622336.0,
+ "114": 876622336.0,
+ "115": 876622336.0,
+ "116": 876622336.0,
+ "117": 876622336.0,
+ "118": 876622336.0,
+ "119": 876622336.0,
+ "120": 876622336.0,
+ "121": 876622336.0,
+ "122": 876622336.0,
+ "123": 876622336.0,
+ "124": 876622336.0,
+ "125": 876622336.0,
+ "126": 876622336.0,
+ "127": 876622336.0,
+ "128": 876622336.0,
+ "129": 876622336.0,
+ "130": 876622336.0,
+ "131": 876622336.0,
+ "132": 876622336.0,
+ "133": 876622336.0,
+ "134": 876622336.0,
+ "135": 876622336.0,
+ "136": 876622336.0,
+ "137": 876622336.0,
+ "138": 876622336.0,
+ "139": 876622336.0,
+ "140": 876622336.0,
+ "141": 876622336.0,
+ "142": 876622336.0,
+ "143": 876622336.0,
+ "144": 876622336.0,
+ "145": 876622336.0,
+ "146": 876622336.0,
+ "147": 876622336.0,
+ "148": 876622336.0,
+ "149": 876622336.0,
+ "150": 876622336.0,
+ "151": 876622336.0,
+ "152": 876622336.0,
+ "153": 876622336.0,
+ "154": 876622336.0,
+ "155": 876622336.0,
+ "156": 876622336.0,
+ "157": 876622336.0,
+ "158": 876622336.0,
+ "159": 876622336.0,
+ "160": 876622336.0,
+ "161": 876622336.0,
+ "162": 876622336.0,
+ "163": 876622336.0,
+ "164": 876622336.0,
+ "165": 876622336.0,
+ "166": 876622336.0,
+ "167": 876622336.0,
+ "168": 876622336.0,
+ "169": 876622336.0,
+ "170": 876622336.0,
+ "171": 876622336.0,
+ "172": 876622336.0,
+ "173": 876622336.0,
+ "174": 876622336.0,
+ "175": 876622336.0,
+ "176": 876622336.0,
+ "177": 876622336.0,
+ "178": 876622336.0,
+ "179": 876622336.0,
+ "180": 876622336.0,
+ "181": 876622336.0,
+ "182": 876622336.0,
+ "183": 876622336.0,
+ "184": 876622336.0,
+ "185": 876622336.0,
+ "186": 876622336.0,
+ "187": 876622336.0,
+ "188": 876622336.0,
+ "189": 876622336.0,
+ "190": 876622336.0,
+ "191": 876622336.0,
+ "192": 876622336.0,
+ "193": 876622336.0,
+ "194": 876622336.0,
+ "195": 876622336.0,
+ "196": 876622336.0,
+ "197": 876622336.0,
+ "198": 876622336.0,
+ "199": 876622336.0,
+ "200": 876622336.0,
+ "201": 876622336.0,
+ "202": 876622336.0,
+ "203": 876622336.0,
+ "204": 876622336.0,
+ "205": 876622336.0,
+ "206": 876622336.0,
+ "207": 876622336.0,
+ "208": 876622336.0,
+ "209": 876622336.0,
+ "210": 876622336.0,
+ "211": 876622336.0,
+ "212": 876622336.0,
+ "213": 876622336.0,
+ "214": 876622336.0,
+ "215": 876622336.0,
+ "216": 876622336.0,
+ "217": 876622336.0,
+ "218": 876622336.0,
+ "219": 876622336.0,
+ "220": 876622336.0,
+ "221": 876622336.0,
+ "222": 876622336.0,
+ "223": 876622336.0,
+ "224": 876622336.0,
+ "225": 876622336.0,
+ "226": 876622336.0,
+ "227": 876622336.0,
+ "228": 876622336.0,
+ "229": 876622336.0,
+ "230": 876622336.0,
+ "231": 876622336.0,
+ "232": 876622336.0,
+ "233": 876622336.0,
+ "234": 876622336.0,
+ "235": 876622336.0,
+ "236": 876622336.0,
+ "237": 876622336.0,
+ "238": 876622336.0,
+ "239": 876622336.0,
+ "240": 876622336.0,
+ "241": 876622336.0,
+ "242": 876622336.0,
+ "243": 876622336.0,
+ "244": 876622336.0,
+ "245": 876622336.0,
+ "246": 876622336.0,
+ "247": 876622336.0,
+ "248": 876622336.0,
+ "249": 876622336.0,
+ "250": 876622336.0,
+ "251": 876622336.0,
+ "252": 876622336.0,
+ "253": 876622336.0,
+ "254": 876622336.0,
+ "255": 876622336.0,
+ "256": 876622336.0,
+ "257": 876622336.0,
+ "258": 876622336.0,
+ "259": 876622336.0,
+ "260": 876622336.0,
+ "261": 876622336.0,
+ "262": 876622336.0,
+ "263": 876622336.0,
+ "264": 876622336.0,
+ "265": 876622336.0,
+ "266": 876622336.0,
+ "267": 876622336.0,
+ "268": 876622336.0,
+ "269": 876622336.0,
+ "270": 876622336.0,
+ "271": 876622336.0,
+ "272": 876622336.0,
+ "273": 876622336.0,
+ "274": 876622336.0,
+ "275": 876622336.0,
+ "276": 876622336.0,
+ "277": 876622336.0,
+ "278": 876622336.0,
+ "279": 876622336.0,
+ "280": 876622336.0,
+ "281": 876622336.0,
+ "282": 876622336.0,
+ "283": 876622336.0,
+ "284": 876622336.0,
+ "285": 876622336.0,
+ "286": 876622336.0,
+ "287": 876622336.0,
+ "288": 876622336.0,
+ "289": 876622336.0,
+ "290": 876622336.0,
+ "291": 876622336.0,
+ "292": 876622336.0,
+ "293": 876622336.0,
+ "294": 876622336.0,
+ "295": 876622336.0,
+ "296": 876622336.0,
+ "297": 876622336.0,
+ "298": 876622336.0,
+ "299": 876622336.0,
+ "300": 876622336.0,
+ "301": 876622336.0,
+ "302": 876622336.0,
+ "303": 876622336.0,
+ "304": 876622336.0,
+ "305": 876622336.0,
+ "306": 876622336.0,
+ "307": 876622336.0,
+ "308": 876622336.0,
+ "309": 876622336.0,
+ "310": 876622336.0,
+ "311": 876622336.0,
+ "312": 876622336.0,
+ "313": 876622336.0,
+ "314": 876622336.0,
+ "315": 876622336.0,
+ "316": 876622336.0,
+ "317": 876622336.0,
+ "318": 876622336.0,
+ "319": 876622336.0,
+ "320": 876622336.0,
+ "321": 876622336.0,
+ "322": 876622336.0,
+ "323": 876622336.0,
+ "324": 876622336.0,
+ "325": 876622336.0,
+ "326": 876622336.0,
+ "327": 876622336.0,
+ "328": 876622336.0,
+ "329": 876622336.0,
+ "330": 876622336.0,
+ "331": 876622336.0,
+ "332": 876622336.0,
+ "333": 876622336.0,
+ "334": 876622336.0,
+ "335": 876622336.0,
+ "336": 876622336.0,
+ "337": 876622336.0,
+ "338": 876622336.0,
+ "339": 876622336.0,
+ "340": 876622336.0,
+ "341": 876622336.0,
+ "342": 876622336.0,
+ "343": 876622336.0,
+ "344": 876622336.0,
+ "345": 876622336.0,
+ "346": 876622336.0,
+ "347": 876622336.0,
+ "348": 876622336.0,
+ "349": 876622336.0,
+ "350": 876622336.0,
+ "351": 876622336.0,
+ "352": 876622336.0,
+ "353": 876622336.0,
+ "354": 876622336.0,
+ "355": 876622336.0,
+ "356": 876622336.0,
+ "357": 876622336.0,
+ "358": 876622336.0,
+ "359": 876622336.0,
+ "360": 876622336.0,
+ "361": 876622336.0,
+ "362": 876622336.0,
+ "363": 876622336.0,
+ "364": 876622336.0,
+ "365": 876622336.0,
+ "366": 876622336.0,
+ "367": 876622336.0,
+ "368": 876622336.0,
+ "369": 876622336.0,
+ "370": 876622336.0,
+ "371": 876622336.0,
+ "372": 876622336.0,
+ "373": 876622336.0,
+ "374": 876622336.0,
+ "375": 876622336.0,
+ "376": 876622336.0,
+ "377": 876622336.0,
+ "378": 876622336.0,
+ "379": 876622336.0,
+ "380": 876622336.0,
+ "381": 876622336.0,
+ "382": 876622336.0,
+ "383": 876622336.0,
+ "384": 876622336.0,
+ "385": 876622336.0,
+ "386": 876622336.0,
+ "387": 876622336.0,
+ "388": 876622336.0,
+ "389": 876622336.0,
+ "390": 876622336.0,
+ "391": 876622336.0,
+ "392": 876622336.0,
+ "393": 876622336.0,
+ "394": 876622336.0,
+ "395": 876622336.0,
+ "396": 876622336.0,
+ "397": 876622336.0,
+ "398": 876622336.0,
+ "399": 876622336.0,
+ "400": 876622336.0,
+ "401": 876622336.0,
+ "402": 876622336.0,
+ "403": 876622336.0,
+ "404": 876622336.0,
+ "405": 876622336.0,
+ "406": 876622336.0,
+ "407": 876622336.0,
+ "408": 876622336.0,
+ "409": 876622336.0,
+ "410": 876622336.0,
+ "411": 876622336.0,
+ "412": 876622336.0,
+ "413": 876622336.0,
+ "414": 876622336.0,
+ "415": 876622336.0,
+ "416": 876622336.0,
+ "417": 876622336.0,
+ "418": 876622336.0,
+ "419": 876622336.0,
+ "420": 876622336.0,
+ "421": 876622336.0,
+ "422": 876622336.0,
+ "423": 876622336.0,
+ "424": 876622336.0,
+ "425": 876622336.0,
+ "426": 876622336.0,
+ "427": 876622336.0,
+ "428": 876622336.0,
+ "429": 876622336.0,
+ "430": 876622336.0,
+ "431": 876622336.0,
+ "432": 876622336.0,
+ "433": 876622336.0,
+ "434": 876622336.0,
+ "435": 876622336.0,
+ "436": 876622336.0,
+ "437": 876622336.0,
+ "438": 876622336.0,
+ "439": 876622336.0,
+ "440": 876622336.0,
+ "441": 876622336.0,
+ "442": 876622336.0,
+ "443": 876622336.0,
+ "444": 876622336.0,
+ "445": 876622336.0,
+ "446": 876622336.0,
+ "447": 876622336.0,
+ "448": 876622336.0,
+ "449": 876622336.0,
+ "450": 876622336.0,
+ "451": 876622336.0,
+ "452": 876622336.0,
+ "453": 876622336.0,
+ "454": 876622336.0,
+ "455": 876622336.0,
+ "456": 876622336.0,
+ "457": 876622336.0,
+ "458": 876622336.0,
+ "459": 876622336.0,
+ "460": 876622336.0,
+ "461": 876622336.0,
+ "462": 876622336.0,
+ "463": 876622336.0,
+ "464": 876622336.0,
+ "465": 876622336.0,
+ "466": 876622336.0,
+ "467": 876622336.0,
+ "468": 876622336.0,
+ "469": 876622336.0,
+ "470": 876622336.0,
+ "471": 876622336.0,
+ "472": 876622336.0,
+ "473": 876622336.0,
+ "474": 876622336.0,
+ "475": 876622336.0,
+ "476": 876622336.0,
+ "477": 876622336.0,
+ "478": 876622336.0,
+ "479": 876622336.0,
+ "480": 876622336.0,
+ "481": 876622336.0,
+ "482": 876622336.0,
+ "483": 876622336.0,
+ "484": 876622336.0,
+ "485": 876622336.0,
+ "486": 876622336.0,
+ "487": 876622336.0,
+ "488": 876622336.0,
+ "489": 876622336.0,
+ "490": 876622336.0,
+ "491": 876622336.0,
+ "492": 876622336.0,
+ "493": 876622336.0,
+ "494": 876622336.0,
+ "495": 876622336.0,
+ "496": 876622336.0,
+ "497": 876622336.0,
+ "498": 876622336.0,
+ "499": 876622336.0,
+ "500": 876622336.0,
+ "501": 876622336.0,
+ "502": 876622336.0,
+ "503": 876622336.0,
+ "504": 876622336.0,
+ "505": 876622336.0,
+ "506": 876622336.0,
+ "507": 876622336.0,
+ "508": 876622336.0,
+ "509": 876622336.0,
+ "510": 876622336.0,
+ "511": 876622336.0,
+ "512": 876622336.0,
+ "513": 876622336.0,
+ "514": 876622336.0,
+ "515": 876622336.0,
+ "516": 876622336.0,
+ "517": 876622336.0,
+ "518": 876622336.0,
+ "519": 876622336.0,
+ "520": 876622336.0,
+ "521": 876622336.0,
+ "522": 876622336.0,
+ "523": 876622336.0,
+ "524": 876622336.0,
+ "525": 876622336.0,
+ "526": 876622336.0,
+ "527": 876622336.0,
+ "528": 876622336.0,
+ "529": 876622336.0,
+ "530": 876622336.0,
+ "531": 876622336.0,
+ "532": 876622336.0,
+ "533": 876622336.0,
+ "534": 876622336.0,
+ "535": 876622336.0,
+ "536": 876622336.0,
+ "537": 876622336.0,
+ "538": 876622336.0,
+ "539": 876622336.0,
+ "540": 876622336.0,
+ "541": 876622336.0,
+ "542": 876622336.0,
+ "543": 876622336.0,
+ "544": 876622336.0,
+ "545": 876622336.0,
+ "546": 876622336.0,
+ "547": 876622336.0,
+ "548": 876622336.0,
+ "549": 876622336.0,
+ "550": 876622336.0,
+ "551": 876622336.0,
+ "552": 876622336.0,
+ "553": 876622336.0,
+ "554": 876622336.0,
+ "555": 876622336.0,
+ "556": 876622336.0,
+ "557": 876622336.0,
+ "558": 876622336.0,
+ "559": 876622336.0,
+ "560": 876622336.0,
+ "561": 876622336.0,
+ "562": 876622336.0,
+ "563": 876622336.0,
+ "564": 876622336.0,
+ "565": 876622336.0,
+ "566": 876622336.0,
+ "567": 876622336.0,
+ "568": 876622336.0,
+ "569": 876622336.0,
+ "570": 876622336.0,
+ "571": 876622336.0,
+ "572": 876622336.0,
+ "573": 876622336.0,
+ "574": 876622336.0,
+ "575": 876622336.0,
+ "576": 876622336.0,
+ "577": 876622336.0,
+ "578": 876622336.0,
+ "579": 876622336.0,
+ "580": 876622336.0,
+ "581": 876622336.0,
+ "582": 876622336.0,
+ "583": 876622336.0,
+ "584": 876622336.0,
+ "585": 876622336.0,
+ "586": 876622336.0,
+ "587": 876622336.0,
+ "588": 876622336.0,
+ "589": 876622336.0,
+ "590": 876622336.0,
+ "591": 876622336.0,
+ "592": 876622336.0,
+ "593": 876622336.0,
+ "594": 876622336.0,
+ "595": 876622336.0,
+ "596": 876622336.0,
+ "597": 876622336.0,
+ "598": 876622336.0,
+ "599": 876622336.0,
+ "600": 876622336.0,
+ "601": 876622336.0,
+ "602": 876622336.0,
+ "603": 876622336.0,
+ "604": 876622336.0,
+ "605": 876622336.0,
+ "606": 876622336.0,
+ "607": 876622336.0,
+ "608": 876622336.0,
+ "609": 876622336.0,
+ "610": 876622336.0,
+ "611": 876622336.0,
+ "612": 876622336.0,
+ "613": 876622336.0,
+ "614": 876622336.0,
+ "615": 876622336.0,
+ "616": 876622336.0,
+ "617": 876622336.0,
+ "618": 876622336.0,
+ "619": 876622336.0,
+ "620": 876622336.0,
+ "621": 876622336.0,
+ "622": 876622336.0,
+ "623": 876622336.0,
+ "624": 876622336.0,
+ "625": 876622336.0,
+ "626": 876622336.0,
+ "627": 876622336.0,
+ "628": 876622336.0,
+ "629": 876622336.0,
+ "630": 876622336.0,
+ "631": 876622336.0,
+ "632": 876622336.0,
+ "633": 876622336.0,
+ "634": 876622336.0,
+ "635": 876622336.0,
+ "636": 876622336.0,
+ "637": 876622336.0,
+ "638": 876622336.0,
+ "639": 876622336.0,
+ "640": 876622336.0,
+ "641": 876622336.0,
+ "642": 876622336.0,
+ "643": 876622336.0,
+ "644": 876622336.0,
+ "645": 876622336.0,
+ "646": 876622336.0,
+ "647": 876622336.0,
+ "648": 876622336.0,
+ "649": 876622336.0,
+ "650": 876622336.0,
+ "651": 876622336.0,
+ "652": 876622336.0,
+ "653": 876622336.0,
+ "654": 876622336.0,
+ "655": 876622336.0,
+ "656": 876622336.0,
+ "657": 876622336.0,
+ "658": 876622336.0,
+ "659": 876622336.0,
+ "660": 876622336.0,
+ "661": 876622336.0,
+ "662": 876622336.0,
+ "663": 876622336.0,
+ "664": 876622336.0,
+ "665": 876622336.0,
+ "666": 876622336.0,
+ "667": 876622336.0,
+ "668": 876622336.0,
+ "669": 876622336.0,
+ "670": 876622336.0,
+ "671": 876622336.0,
+ "672": 876622336.0,
+ "673": 876622336.0,
+ "674": 876622336.0,
+ "675": 876622336.0,
+ "676": 876622336.0,
+ "677": 876622336.0,
+ "678": 876622336.0,
+ "679": 876622336.0,
+ "680": 876622336.0,
+ "681": 876622336.0,
+ "682": 876622336.0,
+ "683": 876622336.0,
+ "684": 876622336.0,
+ "685": 876622336.0,
+ "686": 876622336.0,
+ "687": 876622336.0,
+ "688": 876622336.0,
+ "689": 876622336.0,
+ "690": 876622336.0,
+ "691": 876622336.0,
+ "692": 876622336.0,
+ "693": 876622336.0,
+ "694": 876622336.0,
+ "695": 876622336.0,
+ "696": 876622336.0,
+ "697": 876622336.0,
+ "698": 876622336.0,
+ "699": 876622336.0,
+ "700": 876622336.0,
+ "701": 876622336.0,
+ "702": 876622336.0,
+ "703": 876622336.0,
+ "704": 876622336.0,
+ "705": 876622336.0,
+ "706": 876622336.0,
+ "707": 876622336.0,
+ "708": 876622336.0,
+ "709": 876622336.0,
+ "710": 876622336.0,
+ "711": 876622336.0,
+ "712": 876622336.0,
+ "713": 876622336.0,
+ "714": 876622336.0,
+ "715": 876622336.0,
+ "716": 876622336.0,
+ "717": 876622336.0,
+ "718": 876622336.0,
+ "719": 876622336.0,
+ "720": 876622336.0,
+ "721": 876622336.0,
+ "722": 876622336.0,
+ "723": 876622336.0,
+ "724": 876622336.0,
+ "725": 876622336.0,
+ "726": 876622336.0,
+ "727": 876622336.0,
+ "728": 876622336.0,
+ "729": 876622336.0,
+ "730": 876622336.0,
+ "731": 876622336.0,
+ "732": 876622336.0,
+ "733": 876622336.0,
+ "734": 876622336.0,
+ "735": 876622336.0,
+ "736": 876622336.0,
+ "737": 876622336.0,
+ "738": 876622336.0,
+ "739": 876622336.0,
+ "740": 876622336.0,
+ "741": 876622336.0,
+ "742": 876622336.0,
+ "743": 876622336.0,
+ "744": 876622336.0,
+ "745": 876622336.0,
+ "746": 876622336.0,
+ "747": 876622336.0,
+ "748": 876622336.0,
+ "749": 876622336.0,
+ "750": 876622336.0,
+ "751": 876622336.0,
+ "752": 876622336.0,
+ "753": 876622336.0,
+ "754": 876622336.0,
+ "755": 876622336.0,
+ "756": 876622336.0,
+ "757": 876622336.0,
+ "758": 876622336.0,
+ "759": 876622336.0,
+ "760": 876622336.0,
+ "761": 876622336.0,
+ "762": 876622336.0,
+ "763": 876622336.0,
+ "764": 876622336.0,
+ "765": 876622336.0,
+ "766": 876622336.0,
+ "767": 876622336.0,
+ "768": 876622336.0,
+ "769": 876622336.0,
+ "770": 876622336.0,
+ "771": 876622336.0,
+ "772": 876622336.0,
+ "773": 876622336.0,
+ "774": 876622336.0,
+ "775": 876622336.0,
+ "776": 876622336.0,
+ "777": 876622336.0,
+ "778": 876622336.0,
+ "779": 876622336.0,
+ "780": 876622336.0,
+ "781": 876622336.0,
+ "782": 876622336.0,
+ "783": 876622336.0,
+ "784": 876622336.0,
+ "785": 876622336.0,
+ "786": 876622336.0,
+ "787": 876622336.0,
+ "788": 876622336.0,
+ "789": 876622336.0,
+ "790": 876622336.0,
+ "791": 876622336.0,
+ "792": 876622336.0,
+ "793": 876622336.0,
+ "794": 876622336.0,
+ "795": 876622336.0,
+ "796": 876622336.0,
+ "797": 876622336.0,
+ "798": 876622336.0,
+ "799": 876622336.0,
+ "800": 876622336.0,
+ "801": 876622336.0,
+ "802": 876622336.0,
+ "803": 876622336.0,
+ "804": 876622336.0,
+ "805": 876622336.0,
+ "806": 876622336.0,
+ "807": 876622336.0,
+ "808": 876622336.0,
+ "809": 876622336.0,
+ "810": 876622336.0,
+ "811": 876622336.0,
+ "812": 876622336.0,
+ "813": 876622336.0,
+ "814": 876622336.0,
+ "815": 876622336.0,
+ "816": 876622336.0,
+ "817": 876622336.0,
+ "818": 876622336.0,
+ "819": 876622336.0,
+ "820": 876622336.0,
+ "821": 876622336.0,
+ "822": 876622336.0,
+ "823": 876622336.0,
+ "824": 876622336.0,
+ "825": 876622336.0,
+ "826": 876622336.0,
+ "827": 876622336.0,
+ "828": 876622336.0,
+ "829": 876622336.0,
+ "830": 876622336.0,
+ "831": 876622336.0,
+ "832": 876622336.0,
+ "833": 876622336.0,
+ "834": 876622336.0,
+ "835": 876622336.0,
+ "836": 876622336.0,
+ "837": 876622336.0,
+ "838": 876622336.0,
+ "839": 876622336.0,
+ "840": 876622336.0,
+ "841": 876622336.0,
+ "842": 876622336.0,
+ "843": 876622336.0,
+ "844": 876622336.0,
+ "845": 876622336.0,
+ "846": 876622336.0,
+ "847": 876622336.0,
+ "848": 876622336.0,
+ "849": 876622336.0,
+ "850": 876622336.0,
+ "851": 876622336.0,
+ "852": 876622336.0,
+ "853": 876622336.0,
+ "854": 876622336.0,
+ "855": 876622336.0,
+ "856": 876622336.0,
+ "857": 876622336.0,
+ "858": 876622336.0,
+ "859": 876622336.0,
+ "860": 876622336.0,
+ "861": 876622336.0,
+ "862": 876622336.0,
+ "863": 876622336.0,
+ "864": 876622336.0,
+ "865": 876622336.0,
+ "866": 876622336.0,
+ "867": 876622336.0,
+ "868": 876622336.0,
+ "869": 876622336.0,
+ "870": 876622336.0,
+ "871": 876622336.0,
+ "872": 876622336.0,
+ "873": 876622336.0,
+ "874": 876622336.0,
+ "875": 876622336.0,
+ "876": 876622336.0,
+ "877": 876622336.0,
+ "878": 876622336.0,
+ "879": 876622336.0,
+ "880": 876622336.0,
+ "881": 876622336.0,
+ "882": 876622336.0,
+ "883": 876622336.0,
+ "884": 876622336.0,
+ "885": 876622336.0,
+ "886": 876622336.0,
+ "887": 876622336.0,
+ "888": 876622336.0,
+ "889": 876622336.0,
+ "890": 876622336.0,
+ "891": 876622336.0,
+ "892": 876622336.0,
+ "893": 876622336.0,
+ "894": 876622336.0,
+ "895": 876622336.0,
+ "896": 876622336.0,
+ "897": 876622336.0,
+ "898": 876622336.0,
+ "899": 876622336.0,
+ "900": 876622336.0,
+ "901": 876622336.0,
+ "902": 876622336.0,
+ "903": 876622336.0,
+ "904": 876622336.0,
+ "905": 876622336.0,
+ "906": 876622336.0,
+ "907": 876622336.0,
+ "908": 876622336.0,
+ "909": 876622336.0,
+ "910": 876622336.0,
+ "911": 876622336.0,
+ "912": 876622336.0,
+ "913": 876622336.0,
+ "914": 876622336.0,
+ "915": 876622336.0,
+ "916": 876622336.0,
+ "917": 876622336.0,
+ "918": 876622336.0,
+ "919": 876622336.0,
+ "920": 876622336.0,
+ "921": 876622336.0,
+ "922": 876622336.0,
+ "923": 876622336.0,
+ "924": 876622336.0,
+ "925": 876622336.0,
+ "926": 876622336.0,
+ "927": 876622336.0,
+ "928": 876622336.0,
+ "929": 876622336.0,
+ "930": 876622336.0,
+ "931": 876622336.0,
+ "932": 876622336.0,
+ "933": 876622336.0,
+ "934": 876622336.0,
+ "935": 876622336.0,
+ "936": 876622336.0,
+ "937": 876622336.0,
+ "938": 876622336.0,
+ "939": 876622336.0,
+ "940": 876622336.0,
+ "941": 876622336.0,
+ "942": 876622336.0,
+ "943": 876622336.0,
+ "944": 876622336.0,
+ "945": 876622336.0,
+ "946": 876622336.0,
+ "947": 876622336.0,
+ "948": 876622336.0,
+ "949": 876622336.0,
+ "950": 876622336.0,
+ "951": 876622336.0,
+ "952": 876622336.0,
+ "953": 876622336.0,
+ "954": 876622336.0,
+ "955": 876622336.0,
+ "956": 876622336.0,
+ "957": 876622336.0,
+ "958": 876622336.0,
+ "959": 876622336.0,
+ "960": 876622336.0,
+ "961": 876622336.0,
+ "962": 876622336.0,
+ "963": 876622336.0,
+ "964": 876622336.0,
+ "965": 876622336.0,
+ "966": 876622336.0,
+ "967": 876622336.0,
+ "968": 876622336.0,
+ "969": 876622336.0,
+ "970": 876622336.0,
+ "971": 876622336.0,
+ "972": 876622336.0,
+ "973": 876622336.0,
+ "974": 876622336.0,
+ "975": 876622336.0,
+ "976": 876622336.0,
+ "977": 876622336.0,
+ "978": 876622336.0,
+ "979": 876622336.0,
+ "980": 876622336.0,
+ "981": 876622336.0,
+ "982": 876622336.0,
+ "983": 876622336.0,
+ "984": 876622336.0,
+ "985": 876622336.0,
+ "986": 876622336.0,
+ "987": 876622336.0,
+ "988": 876622336.0,
+ "989": 876622336.0,
+ "990": 876622336.0,
+ "991": 876622336.0,
+ "992": 876622336.0,
+ "993": 876622336.0,
+ "994": 876622336.0,
+ "995": 876622336.0,
+ "996": 876622336.0,
+ "997": 876622336.0,
+ "998": 876622336.0,
+ "999": 876622336.0,
+ "1000": 876622336.0,
+ "1001": 876622336.0,
+ "1002": 876622336.0,
+ "1003": 876622336.0,
+ "1004": 876622336.0,
+ "1005": 876622336.0,
+ "1006": 876622336.0,
+ "1007": 876622336.0,
+ "1008": 876622336.0,
+ "1009": 876622336.0,
+ "1010": 876622336.0,
+ "1011": 876622336.0,
+ "1012": 876622336.0,
+ "1013": 876622336.0,
+ "1014": 876622336.0,
+ "1015": 876622336.0,
+ "1016": 876622336.0,
+ "1017": 876622336.0,
+ "1018": 876622336.0,
+ "1019": 876622336.0,
+ "1020": 876622336.0,
+ "1021": 876622336.0,
+ "1022": 876622336.0,
+ "1023": 876622336.0,
+ "1024": 876622336.0,
+ "1025": 876622336.0,
+ "1026": 876622336.0,
+ "1027": 876622336.0,
+ "1028": 876622336.0,
+ "1029": 876622336.0,
+ "1030": 876622336.0,
+ "1031": 876622336.0,
+ "1032": 876622336.0,
+ "1033": 876622336.0,
+ "1034": 876622336.0,
+ "1035": 876622336.0,
+ "1036": 876622336.0,
+ "1037": 876622336.0,
+ "1038": 876622336.0,
+ "1039": 876622336.0,
+ "1040": 876622336.0,
+ "1041": 876622336.0,
+ "1042": 876622336.0,
+ "1043": 876622336.0,
+ "1044": 876622336.0,
+ "1045": 876622336.0,
+ "1046": 876622336.0,
+ "1047": 876622336.0,
+ "1048": 876622336.0,
+ "1049": 876622336.0,
+ "1050": 876622336.0,
+ "1051": 876622336.0,
+ "1052": 876622336.0,
+ "1053": 876622336.0,
+ "1054": 876622336.0,
+ "1055": 876622336.0,
+ "1056": 876622336.0,
+ "1057": 876622336.0,
+ "1058": 876622336.0,
+ "1059": 876622336.0,
+ "1060": 876622336.0,
+ "1061": 876622336.0,
+ "1062": 876622336.0,
+ "1063": 876622336.0,
+ "1064": 876622336.0,
+ "1065": 876622336.0,
+ "1066": 876622336.0,
+ "1067": 876622336.0,
+ "1068": 876622336.0,
+ "1069": 876622336.0,
+ "1070": 876622336.0,
+ "1071": 876622336.0,
+ "1072": 876622336.0,
+ "1073": 876622336.0,
+ "1074": 876622336.0,
+ "1075": 876622336.0,
+ "1076": 876622336.0,
+ "1077": 876622336.0,
+ "1078": 876622336.0,
+ "1079": 876622336.0,
+ "1080": 876622336.0,
+ "1081": 876622336.0,
+ "1082": 876622336.0,
+ "1083": 876622336.0,
+ "1084": 876622336.0,
+ "1085": 876622336.0,
+ "1086": 876622336.0,
+ "1087": 876622336.0,
+ "1088": 876622336.0,
+ "1089": 876622336.0,
+ "1090": 876622336.0,
+ "1091": 876622336.0,
+ "1092": 876622336.0,
+ "1093": 876622336.0,
+ "1094": 876622336.0,
+ "1095": 876622336.0,
+ "1096": 876622336.0,
+ "1097": 876622336.0,
+ "1098": 876622336.0,
+ "1099": 876622336.0,
+ "1100": 876622336.0,
+ "1101": 876622336.0,
+ "1102": 876622336.0,
+ "1103": 876622336.0,
+ "1104": 876622336.0,
+ "1105": 876622336.0,
+ "1106": 876622336.0,
+ "1107": 876622336.0,
+ "1108": 876622336.0,
+ "1109": 876622336.0,
+ "1110": 876622336.0,
+ "1111": 876622336.0,
+ "1112": 876622336.0,
+ "1113": 876622336.0,
+ "1114": 876622336.0,
+ "1115": 876622336.0,
+ "1116": 876622336.0,
+ "1117": 876622336.0,
+ "1118": 876622336.0,
+ "1119": 876622336.0,
+ "1120": 876622336.0,
+ "1121": 876622336.0,
+ "1122": 876622336.0,
+ "1123": 876622336.0,
+ "1124": 876622336.0,
+ "1125": 876622336.0,
+ "1126": 876622336.0,
+ "1127": 876622336.0,
+ "1128": 876622336.0,
+ "1129": 876622336.0,
+ "1130": 876622336.0,
+ "1131": 876622336.0,
+ "1132": 876622336.0,
+ "1133": 876622336.0,
+ "1134": 876622336.0,
+ "1135": 876622336.0,
+ "1136": 876622336.0,
+ "1137": 876622336.0,
+ "1138": 876622336.0,
+ "1139": 876622336.0,
+ "1140": 876622336.0,
+ "1141": 876622336.0,
+ "1142": 876622336.0,
+ "1143": 876622336.0,
+ "1144": 876622336.0,
+ "1145": 876622336.0,
+ "1146": 876622336.0,
+ "1147": 876622336.0,
+ "1148": 876622336.0,
+ "1149": 876622336.0,
+ "1150": 876622336.0,
+ "1151": 876622336.0,
+ "1152": 876622336.0,
+ "1153": 876622336.0,
+ "1154": 876622336.0,
+ "1155": 876622336.0,
+ "1156": 876622336.0,
+ "1157": 876622336.0,
+ "1158": 876622336.0,
+ "1159": 876622336.0,
+ "1160": 876622336.0,
+ "1161": 876622336.0,
+ "1162": 876622336.0,
+ "1163": 876622336.0,
+ "1164": 876622336.0,
+ "1165": 876622336.0,
+ "1166": 876622336.0,
+ "1167": 876622336.0,
+ "1168": 876622336.0,
+ "1169": 876622336.0,
+ "1170": 876622336.0,
+ "1171": 876622336.0,
+ "1172": 876622336.0,
+ "1173": 876622336.0,
+ "1174": 876622336.0,
+ "1175": 876622336.0,
+ "1176": 876622336.0,
+ "1177": 876622336.0,
+ "1178": 876622336.0,
+ "1179": 876622336.0,
+ "1180": 876622336.0,
+ "1181": 876622336.0,
+ "1182": 876622336.0,
+ "1183": 876622336.0,
+ "1184": 876622336.0,
+ "1185": 876622336.0,
+ "1186": 876622336.0,
+ "1187": 876622336.0,
+ "1188": 876622336.0,
+ "1189": 876622336.0,
+ "1190": 876622336.0,
+ "1191": 876622336.0,
+ "1192": 876622336.0,
+ "1193": 876622336.0,
+ "1194": 876622336.0,
+ "1195": 876622336.0,
+ "1196": 876622336.0,
+ "1197": 876622336.0,
+ "1198": 876622336.0,
+ "1199": 876622336.0,
+ "1200": 876622336.0,
+ "1201": 876622336.0,
+ "1202": 876622336.0,
+ "1203": 876622336.0,
+ "1204": 876622336.0,
+ "1205": 876622336.0,
+ "1206": 876622336.0,
+ "1207": 876622336.0,
+ "1208": 876622336.0,
+ "1209": 876622336.0,
+ "1210": 876622336.0,
+ "1211": 876622336.0,
+ "1212": 876622336.0,
+ "1213": 876622336.0,
+ "1214": 876622336.0,
+ "1215": 876622336.0,
+ "1216": 876622336.0,
+ "1217": 876622336.0,
+ "1218": 876622336.0,
+ "1219": 876622336.0,
+ "1220": 876622336.0,
+ "1221": 876622336.0,
+ "1222": 876622336.0,
+ "1223": 876622336.0,
+ "1224": 876622336.0,
+ "1225": 876622336.0,
+ "1226": 876622336.0,
+ "1227": 876622336.0,
+ "1228": 876622336.0,
+ "1229": 876622336.0,
+ "1230": 876622336.0,
+ "1231": 876622336.0,
+ "1232": 876622336.0,
+ "1233": 876622336.0,
+ "1234": 876622336.0,
+ "1235": 876622336.0,
+ "1236": 876622336.0,
+ "1237": 876622336.0,
+ "1238": 876622336.0,
+ "1239": 876622336.0,
+ "1240": 876622336.0,
+ "1241": 876622336.0,
+ "1242": 876622336.0,
+ "1243": 876622336.0,
+ "1244": 876622336.0,
+ "1245": 876622336.0,
+ "1246": 876622336.0,
+ "1247": 876622336.0,
+ "1248": 876622336.0,
+ "1249": 876622336.0,
+ "1250": 876622336.0,
+ "1251": 876622336.0,
+ "1252": 876622336.0,
+ "1253": 876622336.0,
+ "1254": 876622336.0,
+ "1255": 876622336.0,
+ "1256": 876622336.0,
+ "1257": 876622336.0,
+ "1258": 876622336.0,
+ "1259": 876622336.0,
+ "1260": 876622336.0,
+ "1261": 876622336.0,
+ "1262": 876622336.0,
+ "1263": 876622336.0,
+ "1264": 876622336.0,
+ "1265": 876622336.0,
+ "1266": 876622336.0,
+ "1267": 876622336.0,
+ "1268": 876622336.0,
+ "1269": 876622336.0,
+ "1270": 876622336.0,
+ "1271": 876622336.0,
+ "1272": 876622336.0,
+ "1273": 876622336.0,
+ "1274": 876622336.0,
+ "1275": 876622336.0,
+ "1276": 876622336.0,
+ "1277": 876622336.0,
+ "1278": 876622336.0,
+ "1279": 876622336.0,
+ "1280": 876622336.0,
+ "1281": 876622336.0,
+ "1282": 876622336.0,
+ "1283": 876622336.0,
+ "1284": 876622336.0,
+ "1285": 876622336.0,
+ "1286": 876622336.0,
+ "1287": 876622336.0,
+ "1288": 876622336.0,
+ "1289": 876622336.0,
+ "1290": 876622336.0,
+ "1291": 876622336.0,
+ "1292": 876622336.0,
+ "1293": 876622336.0,
+ "1294": 876622336.0,
+ "1295": 876622336.0,
+ "1296": 876622336.0,
+ "1297": 876622336.0,
+ "1298": 876622336.0,
+ "1299": 876622336.0,
+ "1300": 876622336.0,
+ "1301": 876622336.0,
+ "1302": 876622336.0,
+ "1303": 876622336.0,
+ "1304": 876622336.0,
+ "1305": 876622336.0,
+ "1306": 876622336.0,
+ "1307": 876622336.0,
+ "1308": 876622336.0,
+ "1309": 876622336.0,
+ "1310": 876622336.0,
+ "1311": 876622336.0,
+ "1312": 876622336.0,
+ "1313": 876622336.0,
+ "1314": 876622336.0,
+ "1315": 876622336.0,
+ "1316": 876622336.0,
+ "1317": 876622336.0,
+ "1318": 876622336.0,
+ "1319": 876622336.0,
+ "1320": 876622336.0,
+ "1321": 876622336.0,
+ "1322": 876622336.0,
+ "1323": 876622336.0,
+ "1324": 876622336.0,
+ "1325": 876622336.0,
+ "1326": 876622336.0,
+ "1327": 876622336.0,
+ "1328": 876622336.0,
+ "1329": 876622336.0,
+ "1330": 876622336.0,
+ "1331": 876622336.0,
+ "1332": 876622336.0,
+ "1333": 876622336.0,
+ "1334": 876622336.0,
+ "1335": 876622336.0,
+ "1336": 876622336.0,
+ "1337": 876622336.0,
+ "1338": 876622336.0,
+ "1339": 876622336.0,
+ "1340": 876622336.0,
+ "1341": 876622336.0,
+ "1342": 876622336.0,
+ "1343": 876622336.0,
+ "1344": 876622336.0,
+ "1345": 876622336.0,
+ "1346": 876622336.0,
+ "1347": 876622336.0,
+ "1348": 876622336.0,
+ "1349": 876622336.0,
+ "1350": 876622336.0,
+ "1351": 876622336.0,
+ "1352": 876622336.0,
+ "1353": 876622336.0,
+ "1354": 876622336.0,
+ "1355": 876622336.0,
+ "1356": 876622336.0,
+ "1357": 876622336.0,
+ "1358": 876622336.0,
+ "1359": 876622336.0,
+ "1360": 876622336.0,
+ "1361": 876622336.0,
+ "1362": 876622336.0,
+ "1363": 876622336.0,
+ "1364": 876622336.0,
+ "1365": 876622336.0,
+ "1366": 876622336.0,
+ "1367": 876622336.0,
+ "1368": 876622336.0,
+ "1369": 876622336.0,
+ "1370": 876622336.0,
+ "1371": 876622336.0,
+ "1372": 876622336.0,
+ "1373": 876622336.0,
+ "1374": 876622336.0,
+ "1375": 876622336.0,
+ "1376": 876622336.0,
+ "1377": 876622336.0,
+ "1378": 876622336.0,
+ "1379": 876622336.0,
+ "1380": 876622336.0,
+ "1381": 876622336.0,
+ "1382": 876622336.0,
+ "1383": 876622336.0,
+ "1384": 876622336.0,
+ "1385": 876622336.0,
+ "1386": 876622336.0,
+ "1387": 876622336.0,
+ "1388": 876622336.0,
+ "1389": 876622336.0,
+ "1390": 876622336.0,
+ "1391": 876622336.0,
+ "1392": 876622336.0,
+ "1393": 876622336.0,
+ "1394": 876622336.0,
+ "1395": 876622336.0,
+ "1396": 876622336.0,
+ "1397": 876622336.0,
+ "1398": 876622336.0,
+ "1399": 876622336.0,
+ "1400": 876622336.0,
+ "1401": 876622336.0,
+ "1402": 876622336.0,
+ "1403": 876622336.0,
+ "1404": 876622336.0,
+ "1405": 876622336.0,
+ "1406": 876622336.0,
+ "1407": 876622336.0,
+ "1408": 876622336.0,
+ "1409": 876622336.0,
+ "1410": 876622336.0,
+ "1411": 876622336.0,
+ "1412": 876622336.0,
+ "1413": 876622336.0,
+ "1414": 876622336.0,
+ "1415": 876622336.0,
+ "1416": 876622336.0,
+ "1417": 876622336.0,
+ "1418": 876622336.0,
+ "1419": 876622336.0,
+ "1420": 876622336.0,
+ "1421": 876622336.0,
+ "1422": 876622336.0,
+ "1423": 876622336.0,
+ "1424": 876622336.0,
+ "1425": 876622336.0,
+ "1426": 876622336.0,
+ "1427": 876622336.0,
+ "1428": 876622336.0,
+ "1429": 876622336.0,
+ "1430": 876622336.0,
+ "1431": 876622336.0,
+ "1432": 876622336.0,
+ "1433": 876622336.0,
+ "1434": 876622336.0,
+ "1435": 876622336.0,
+ "1436": 876622336.0,
+ "1437": 876622336.0,
+ "1438": 876622336.0,
+ "1439": 876622336.0,
+ "1440": 876622336.0,
+ "1441": 876622336.0,
+ "1442": 876622336.0,
+ "1443": 876622336.0,
+ "1444": 876622336.0,
+ "1445": 876622336.0,
+ "1446": 876622336.0,
+ "1447": 876622336.0,
+ "1448": 876622336.0,
+ "1449": 876622336.0,
+ "1450": 876622336.0,
+ "1451": 876622336.0,
+ "1452": 876622336.0,
+ "1453": 876622336.0,
+ "1454": 876622336.0,
+ "1455": 876622336.0,
+ "1456": 876622336.0,
+ "1457": 876622336.0,
+ "1458": 876622336.0,
+ "1459": 876622336.0,
+ "1460": 876622336.0,
+ "1461": 876622336.0,
+ "1462": 876622336.0,
+ "1463": 876622336.0,
+ "1464": 876622336.0,
+ "1465": 876622336.0,
+ "1466": 876622336.0,
+ "1467": 876622336.0,
+ "1468": 876622336.0,
+ "1469": 876622336.0,
+ "1470": 876622336.0,
+ "1471": 876622336.0,
+ "1472": 876622336.0,
+ "1473": 876622336.0,
+ "1474": 876622336.0,
+ "1475": 876622336.0,
+ "1476": 876622336.0,
+ "1477": 876622336.0,
+ "1478": 876622336.0,
+ "1479": 876622336.0,
+ "1480": 876622336.0,
+ "1481": 876622336.0,
+ "1482": 876622336.0,
+ "1483": 876622336.0,
+ "1484": 876622336.0,
+ "1485": 876622336.0,
+ "1486": 876622336.0,
+ "1487": 876622336.0,
+ "1488": 876622336.0,
+ "1489": 876622336.0,
+ "1490": 876622336.0,
+ "1491": 876622336.0,
+ "1492": 876622336.0,
+ "1493": 876622336.0,
+ "1494": 876622336.0,
+ "1495": 876622336.0,
+ "1496": 876622336.0,
+ "1497": 876622336.0,
+ "1498": 876622336.0,
+ "1499": 876622336.0,
+ "1500": 876622336.0,
+ "1501": 876622336.0,
+ "1502": 876622336.0,
+ "1503": 876622336.0,
+ "1504": 876622336.0,
+ "1505": 876622336.0,
+ "1506": 876622336.0,
+ "1507": 876622336.0,
+ "1508": 876622336.0,
+ "1509": 876622336.0,
+ "1510": 876622336.0,
+ "1511": 876622336.0,
+ "1512": 876622336.0,
+ "1513": 876622336.0,
+ "1514": 876622336.0,
+ "1515": 876622336.0,
+ "1516": 876622336.0,
+ "1517": 876622336.0,
+ "1518": 876622336.0,
+ "1519": 876622336.0,
+ "1520": 876622336.0,
+ "1521": 876622336.0,
+ "1522": 876622336.0,
+ "1523": 876622336.0,
+ "1524": 876622336.0,
+ "1525": 876622336.0,
+ "1526": 876622336.0,
+ "1527": 876622336.0,
+ "1528": 876622336.0,
+ "1529": 876622336.0,
+ "1530": 876622336.0,
+ "1531": 876622336.0,
+ "1532": 876622336.0,
+ "1533": 876622336.0,
+ "1534": 876622336.0,
+ "1535": 876622336.0,
+ "1536": 876622336.0,
+ "1537": 876622336.0,
+ "1538": 876622336.0,
+ "1539": 876622336.0,
+ "1540": 876622336.0,
+ "1541": 876622336.0,
+ "1542": 876622336.0,
+ "1543": 876622336.0,
+ "1544": 876622336.0,
+ "1545": 876622336.0,
+ "1546": 876622336.0,
+ "1547": 876622336.0,
+ "1548": 876622336.0,
+ "1549": 876622336.0,
+ "1550": 876622336.0,
+ "1551": 876622336.0,
+ "1552": 876622336.0,
+ "1553": 876622336.0,
+ "1554": 876622336.0,
+ "1555": 876622336.0,
+ "1556": 876622336.0,
+ "1557": 876622336.0,
+ "1558": 876622336.0,
+ "1559": 876622336.0,
+ "1560": 876622336.0,
+ "1561": 876622336.0,
+ "1562": 876622336.0,
+ "1563": 876622336.0,
+ "1564": 876622336.0,
+ "1565": 876622336.0,
+ "1566": 876622336.0,
+ "1567": 876622336.0,
+ "1568": 876622336.0,
+ "1569": 876622336.0,
+ "1570": 876622336.0,
+ "1571": 876622336.0,
+ "1572": 876622336.0,
+ "1573": 876622336.0,
+ "1574": 876622336.0,
+ "1575": 876622336.0,
+ "1576": 876622336.0,
+ "1577": 876622336.0,
+ "1578": 876622336.0,
+ "1579": 876622336.0,
+ "1580": 876622336.0,
+ "1581": 876622336.0,
+ "1582": 876622336.0,
+ "1583": 876622336.0,
+ "1584": 876622336.0,
+ "1585": 876622336.0,
+ "1586": 876622336.0,
+ "1587": 876622336.0,
+ "1588": 876622336.0,
+ "1589": 876622336.0,
+ "1590": 876622336.0,
+ "1591": 876622336.0,
+ "1592": 876622336.0,
+ "1593": 876622336.0,
+ "1594": 876622336.0,
+ "1595": 876622336.0,
+ "1596": 876622336.0,
+ "1597": 876622336.0,
+ "1598": 876622336.0,
+ "1599": 876622336.0,
+ "1600": 876622336.0,
+ "1601": 876622336.0,
+ "1602": 876622336.0,
+ "1603": 876622336.0,
+ "1604": 876622336.0,
+ "1605": 876622336.0,
+ "1606": 876622336.0,
+ "1607": 876622336.0,
+ "1608": 876622336.0,
+ "1609": 876622336.0,
+ "1610": 876622336.0,
+ "1611": 876622336.0,
+ "1612": 876622336.0,
+ "1613": 876622336.0,
+ "1614": 876622336.0,
+ "1615": 876622336.0,
+ "1616": 876622336.0,
+ "1617": 876622336.0,
+ "1618": 876622336.0,
+ "1619": 876622336.0,
+ "1620": 876622336.0,
+ "1621": 876622336.0,
+ "1622": 876622336.0,
+ "1623": 876622336.0,
+ "1624": 876622336.0,
+ "1625": 876622336.0,
+ "1626": 876622336.0,
+ "1627": 876622336.0,
+ "1628": 876622336.0,
+ "1629": 876622336.0,
+ "1630": 876622336.0,
+ "1631": 876622336.0,
+ "1632": 876622336.0,
+ "1633": 876622336.0,
+ "1634": 876622336.0,
+ "1635": 876622336.0,
+ "1636": 876622336.0,
+ "1637": 876622336.0,
+ "1638": 876622336.0,
+ "1639": 876622336.0,
+ "1640": 876622336.0,
+ "1641": 876622336.0,
+ "1642": 876622336.0,
+ "1643": 876622336.0,
+ "1644": 876622336.0,
+ "1645": 876622336.0,
+ "1646": 876622336.0,
+ "1647": 876622336.0,
+ "1648": 876622336.0,
+ "1649": 876622336.0,
+ "1650": 876622336.0,
+ "1651": 876622336.0,
+ "1652": 876622336.0,
+ "1653": 876622336.0,
+ "1654": 876622336.0,
+ "1655": 876622336.0,
+ "1656": 876622336.0,
+ "1657": 876622336.0,
+ "1658": 876622336.0,
+ "1659": 876622336.0,
+ "1660": 876622336.0,
+ "1661": 876622336.0,
+ "1662": 876622336.0,
+ "1663": 876622336.0,
+ "1664": 876622336.0,
+ "1665": 876622336.0,
+ "1666": 876622336.0,
+ "1667": 876622336.0,
+ "1668": 876622336.0,
+ "1669": 876622336.0,
+ "1670": 876622336.0,
+ "1671": 876622336.0,
+ "1672": 876622336.0,
+ "1673": 876622336.0,
+ "1674": 876622336.0,
+ "1675": 876622336.0,
+ "1676": 876622336.0,
+ "1677": 876622336.0,
+ "1678": 876622336.0,
+ "1679": 876622336.0,
+ "1680": 876622336.0,
+ "1681": 876622336.0,
+ "1682": 876622336.0,
+ "1683": 876622336.0,
+ "1684": 876622336.0,
+ "1685": 876622336.0,
+ "1686": 876622336.0,
+ "1687": 876622336.0,
+ "1688": 876622336.0,
+ "1689": 876622336.0,
+ "1690": 876622336.0,
+ "1691": 876622336.0,
+ "1692": 876622336.0,
+ "1693": 876622336.0,
+ "1694": 876622336.0,
+ "1695": 876622336.0,
+ "1696": 876622336.0,
+ "1697": 876622336.0,
+ "1698": 876622336.0,
+ "1699": 876622336.0,
+ "1700": 876622336.0,
+ "1701": 876622336.0,
+ "1702": 876622336.0,
+ "1703": 876622336.0,
+ "1704": 876622336.0,
+ "1705": 876622336.0,
+ "1706": 876622336.0,
+ "1707": 876622336.0,
+ "1708": 876622336.0,
+ "1709": 876622336.0,
+ "1710": 876622336.0,
+ "1711": 876622336.0,
+ "1712": 876622336.0,
+ "1713": 876622336.0,
+ "1714": 876622336.0,
+ "1715": 876622336.0,
+ "1716": 876622336.0,
+ "1717": 876622336.0,
+ "1718": 876622336.0,
+ "1719": 876622336.0,
+ "1720": 876622336.0,
+ "1721": 876622336.0,
+ "1722": 876622336.0,
+ "1723": 876622336.0,
+ "1724": 876622336.0,
+ "1725": 876622336.0,
+ "1726": 876622336.0,
+ "1727": 876622336.0,
+ "1728": 876622336.0,
+ "1729": 876622336.0,
+ "1730": 876622336.0,
+ "1731": 876622336.0,
+ "1732": 876622336.0,
+ "1733": 876622336.0,
+ "1734": 876622336.0,
+ "1735": 876622336.0,
+ "1736": 876622336.0,
+ "1737": 876622336.0,
+ "1738": 876622336.0,
+ "1739": 876622336.0,
+ "1740": 876622336.0,
+ "1741": 876622336.0,
+ "1742": 876622336.0,
+ "1743": 876622336.0,
+ "1744": 876622336.0,
+ "1745": 876622336.0,
+ "1746": 876622336.0,
+ "1747": 876622336.0,
+ "1748": 876622336.0,
+ "1749": 876622336.0,
+ "1750": 876622336.0,
+ "1751": 876622336.0,
+ "1752": 876622336.0,
+ "1753": 876622336.0,
+ "1754": 876622336.0,
+ "1755": 876622336.0,
+ "1756": 876622336.0,
+ "1757": 876622336.0,
+ "1758": 876622336.0,
+ "1759": 876622336.0,
+ "1760": 876622336.0,
+ "1761": 876622336.0,
+ "1762": 876622336.0,
+ "1763": 876622336.0,
+ "1764": 876622336.0,
+ "1765": 876622336.0,
+ "1766": 876622336.0,
+ "1767": 876622336.0,
+ "1768": 876622336.0,
+ "1769": 876622336.0,
+ "1770": 876622336.0,
+ "1771": 876622336.0,
+ "1772": 876622336.0,
+ "1773": 876622336.0,
+ "1774": 876622336.0,
+ "1775": 876622336.0,
+ "1776": 876622336.0,
+ "1777": 876622336.0,
+ "1778": 876622336.0,
+ "1779": 876622336.0,
+ "1780": 876622336.0,
+ "1781": 876622336.0,
+ "1782": 876622336.0,
+ "1783": 876622336.0,
+ "1784": 876622336.0,
+ "1785": 876622336.0,
+ "1786": 876622336.0,
+ "1787": 876622336.0,
+ "1788": 876622336.0,
+ "1789": 876622336.0,
+ "1790": 876622336.0,
+ "1791": 876622336.0,
+ "1792": 876622336.0,
+ "1793": 876622336.0,
+ "1794": 876622336.0,
+ "1795": 876622336.0,
+ "1796": 876622336.0,
+ "1797": 876622336.0,
+ "1798": 876622336.0,
+ "1799": 876622336.0,
+ "1800": 876622336.0,
+ "1801": 876622336.0,
+ "1802": 876622336.0,
+ "1803": 876622336.0,
+ "1804": 876622336.0,
+ "1805": 876622336.0,
+ "1806": 876622336.0,
+ "1807": 876622336.0,
+ "1808": 876622336.0,
+ "1809": 876622336.0,
+ "1810": 876622336.0,
+ "1811": 876622336.0,
+ "1812": 876622336.0,
+ "1813": 876622336.0,
+ "1814": 876622336.0,
+ "1815": 876622336.0,
+ "1816": 876622336.0,
+ "1817": 876622336.0,
+ "1818": 876622336.0,
+ "1819": 876622336.0,
+ "1820": 876622336.0,
+ "1821": 876622336.0,
+ "1822": 876622336.0,
+ "1823": 876622336.0,
+ "1824": 876622336.0,
+ "1825": 876622336.0,
+ "1826": 876622336.0,
+ "1827": 876622336.0,
+ "1828": 876622336.0,
+ "1829": 876622336.0,
+ "1830": 876622336.0,
+ "1831": 876622336.0,
+ "1832": 876622336.0,
+ "1833": 876622336.0,
+ "1834": 876622336.0,
+ "1835": 876622336.0,
+ "1836": 876622336.0,
+ "1837": 876622336.0,
+ "1838": 876622336.0,
+ "1839": 876622336.0,
+ "1840": 876622336.0,
+ "1841": 876622336.0,
+ "1842": 876622336.0,
+ "1843": 876622336.0,
+ "1844": 876622336.0,
+ "1845": 876622336.0,
+ "1846": 876622336.0,
+ "1847": 876622336.0,
+ "1848": 876622336.0,
+ "1849": 876622336.0,
+ "1850": 876622336.0,
+ "1851": 876622336.0,
+ "1852": 876622336.0,
+ "1853": 876622336.0,
+ "1854": 876622336.0,
+ "1855": 876622336.0,
+ "1856": 876622336.0,
+ "1857": 876622336.0,
+ "1858": 876622336.0,
+ "1859": 876622336.0,
+ "1860": 876622336.0,
+ "1861": 876622336.0,
+ "1862": 876622336.0,
+ "1863": 876622336.0,
+ "1864": 876622336.0,
+ "1865": 876622336.0,
+ "1866": 876622336.0,
+ "1867": 876622336.0,
+ "1868": 876622336.0,
+ "1869": 876622336.0,
+ "1870": 876622336.0,
+ "1871": 876622336.0,
+ "1872": 876622336.0,
+ "1873": 876622336.0,
+ "1874": 876622336.0,
+ "1875": 876622336.0,
+ "1876": 876622336.0,
+ "1877": 876622336.0,
+ "1878": 876622336.0,
+ "1879": 876622336.0,
+ "1880": 876622336.0,
+ "1881": 876622336.0,
+ "1882": 876622336.0,
+ "1883": 876622336.0,
+ "1884": 876622336.0,
+ "1885": 876622336.0,
+ "1886": 876622336.0,
+ "1887": 876622336.0,
+ "1888": 876622336.0,
+ "1889": 876622336.0,
+ "1890": 876622336.0,
+ "1891": 876622336.0,
+ "1892": 876622336.0,
+ "1893": 876622336.0,
+ "1894": 876622336.0,
+ "1895": 876622336.0,
+ "1896": 876622336.0,
+ "1897": 876622336.0,
+ "1898": 876622336.0,
+ "1899": 876622336.0,
+ "1900": 876622336.0,
+ "1901": 876622336.0,
+ "1902": 876622336.0,
+ "1903": 876622336.0,
+ "1904": 876622336.0,
+ "1905": 876622336.0,
+ "1906": 876622336.0,
+ "1907": 876622336.0,
+ "1908": 876622336.0,
+ "1909": 876622336.0,
+ "1910": 876622336.0,
+ "1911": 876622336.0,
+ "1912": 876622336.0,
+ "1913": 876622336.0,
+ "1914": 876622336.0,
+ "1915": 876622336.0,
+ "1916": 876622336.0,
+ "1917": 876622336.0,
+ "1918": 876622336.0,
+ "1919": 876622336.0,
+ "1920": 876622336.0,
+ "1921": 876622336.0,
+ "1922": 876622336.0,
+ "1923": 876622336.0,
+ "1924": 876622336.0,
+ "1925": 876622336.0,
+ "1926": 876622336.0,
+ "1927": 876622336.0,
+ "1928": 876622336.0,
+ "1929": 876622336.0,
+ "1930": 876622336.0,
+ "1931": 876622336.0,
+ "1932": 876622336.0,
+ "1933": 876622336.0,
+ "1934": 876622336.0,
+ "1935": 876622336.0,
+ "1936": 876622336.0,
+ "1937": 876622336.0,
+ "1938": 876622336.0,
+ "1939": 876622336.0,
+ "1940": 876622336.0,
+ "1941": 876622336.0,
+ "1942": 876622336.0,
+ "1943": 876622336.0,
+ "1944": 876622336.0,
+ "1945": 876622336.0,
+ "1946": 876622336.0,
+ "1947": 876622336.0,
+ "1948": 876622336.0,
+ "1949": 876622336.0,
+ "1950": 876622336.0,
+ "1951": 876622336.0,
+ "1952": 876622336.0,
+ "1953": 876622336.0,
+ "1954": 876622336.0,
+ "1955": 876622336.0,
+ "1956": 876622336.0,
+ "1957": 876622336.0,
+ "1958": 876622336.0,
+ "1959": 876622336.0,
+ "1960": 876622336.0,
+ "1961": 876622336.0,
+ "1962": 876622336.0,
+ "1963": 876622336.0,
+ "1964": 876622336.0,
+ "1965": 876622336.0,
+ "1966": 876622336.0,
+ "1967": 876622336.0,
+ "1968": 876622336.0,
+ "1969": 876622336.0,
+ "1970": 876622336.0,
+ "1971": 876622336.0,
+ "1972": 876622336.0,
+ "1973": 876622336.0,
+ "1974": 876622336.0,
+ "1975": 876622336.0,
+ "1976": 876622336.0,
+ "1977": 876622336.0,
+ "1978": 876622336.0,
+ "1979": 876622336.0,
+ "1980": 876622336.0,
+ "1981": 876622336.0,
+ "1982": 876622336.0,
+ "1983": 876622336.0,
+ "1984": 876622336.0,
+ "1985": 876622336.0,
+ "1986": 876622336.0,
+ "1987": 876622336.0,
+ "1988": 876622336.0,
+ "1989": 876622336.0,
+ "1990": 876622336.0,
+ "1991": 876622336.0,
+ "1992": 876622336.0,
+ "1993": 876622336.0,
+ "1994": 876622336.0,
+ "1995": 876622336.0,
+ "1996": 876622336.0,
+ "1997": 876622336.0,
+ "1998": 876622336.0,
+ "1999": 876622336.0,
+ "2000": 876622336.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": "nan",
+ "2": 8.15973,
+ "3": 2.08191,
+ "4": 1.67828,
+ "5": 1.68639,
+ "6": 1.68502,
+ "7": 1.68153,
+ "8": 1.6776,
+ "9": 1.68865,
+ "10": 1.67894,
+ "11": 1.68452,
+ "12": 1.66936,
+ "13": 1.65646,
+ "14": 1.67561,
+ "15": 1.67192,
+ "16": 1.66749,
+ "17": 1.67473,
+ "18": 1.67751,
+ "19": 1.66652,
+ "20": 1.66111,
+ "21": 1.65933,
+ "22": 1.65768,
+ "23": 1.675,
+ "24": 1.67224,
+ "25": 1.67544,
+ "26": 1.66932,
+ "27": 1.67267,
+ "28": 1.66982,
+ "29": 1.67061,
+ "30": 1.69659,
+ "31": 1.68096,
+ "32": 1.6904,
+ "33": 1.68957,
+ "34": 1.68855,
+ "35": 1.69132,
+ "36": 1.69312,
+ "37": 1.69157,
+ "38": 1.68917,
+ "39": 1.68787,
+ "40": 1.69154,
+ "41": 1.69713,
+ "42": 1.69233,
+ "43": 1.69541,
+ "44": 1.69777,
+ "45": 1.69659,
+ "46": 1.6965,
+ "47": 1.6876,
+ "48": 1.68896,
+ "49": 1.68974,
+ "50": 1.6964,
+ "51": 1.69315,
+ "52": 1.73346,
+ "53": 1.7656,
+ "54": 2.28164,
+ "55": 1.91098,
+ "56": 1.67884,
+ "57": 2.02722,
+ "58": 1.76497,
+ "59": 1.6976,
+ "60": 1.75847,
+ "61": 1.68291,
+ "62": 1.68559,
+ "63": 1.68715,
+ "64": 1.68368,
+ "65": 1.68721,
+ "66": 2.43657,
+ "67": 9.59505,
+ "68": 12.8338,
+ "69": 3.62666,
+ "70": 1.83474,
+ "71": 1.67199,
+ "72": 1.67624,
+ "73": 3.24318,
+ "74": 1.6722,
+ "75": 1.68456,
+ "76": 1.69495,
+ "77": 1.78399,
+ "78": 1.6837,
+ "79": 1.66998,
+ "80": 2.17442,
+ "81": 1.72995,
+ "82": 1.67237,
+ "83": 1.68686,
+ "84": 1.69507,
+ "85": 1.68119,
+ "86": 1.67453,
+ "87": 1.67165,
+ "88": 1.68103,
+ "89": 1.67383,
+ "90": 1.67772,
+ "91": 1.68543,
+ "92": 1.67026,
+ "93": 1.68437,
+ "94": 1.70866,
+ "95": 1.68596,
+ "96": 1.68671,
+ "97": 1.68294,
+ "98": 1.68436,
+ "99": 1.67835,
+ "100": 1.68423,
+ "101": 1.67549,
+ "102": 1.67955,
+ "103": 1.6871,
+ "104": 1.68486,
+ "105": 1.67931,
+ "106": 1.68477,
+ "107": 1.68154,
+ "108": 1.70968,
+ "109": 1.69612,
+ "110": 1.69803,
+ "111": 1.70635,
+ "112": 1.68963,
+ "113": 1.68215,
+ "114": 1.67565,
+ "115": 1.67955,
+ "116": 1.67705,
+ "117": 1.67704,
+ "118": 1.66249,
+ "119": 1.68136,
+ "120": 1.68216,
+ "121": 1.65868,
+ "122": 1.65479,
+ "123": 1.64536,
+ "124": 1.68843,
+ "125": 1.66602,
+ "126": 1.66375,
+ "127": 1.65583,
+ "128": 1.67199,
+ "129": 1.67043,
+ "130": 1.66142,
+ "131": 1.65988,
+ "132": 1.66509,
+ "133": 1.66296,
+ "134": 1.66701,
+ "135": 1.66275,
+ "136": 1.65971,
+ "137": 1.67076,
+ "138": 1.66912,
+ "139": 1.67725,
+ "140": 1.67157,
+ "141": 1.66197,
+ "142": 1.66836,
+ "143": 1.66596,
+ "144": 1.66816,
+ "145": 1.65626,
+ "146": 1.6643,
+ "147": 1.66213,
+ "148": 1.65448,
+ "149": 1.65925,
+ "150": 1.68693,
+ "151": 1.66695,
+ "152": 1.6593,
+ "153": 1.64953,
+ "154": 1.685,
+ "155": 1.65662,
+ "156": 1.66745,
+ "157": 1.66342,
+ "158": 1.6745,
+ "159": 1.67475,
+ "160": 1.67587,
+ "161": 1.67459,
+ "162": 1.66965,
+ "163": 1.67635,
+ "164": 1.67577,
+ "165": 1.70054,
+ "166": 1.6765,
+ "167": 1.69023,
+ "168": 1.6828,
+ "169": 1.68217,
+ "170": 1.68344,
+ "171": 1.67789,
+ "172": 1.68916,
+ "173": 1.68356,
+ "174": 1.68932,
+ "175": 1.68734,
+ "176": 1.68633,
+ "177": 1.68708,
+ "178": 1.68385,
+ "179": 1.69194,
+ "180": 1.69213,
+ "181": 1.67941,
+ "182": 1.69345,
+ "183": 1.69765,
+ "184": 1.68228,
+ "185": 1.68775,
+ "186": 1.6914,
+ "187": 1.68482,
+ "188": 1.66288,
+ "189": 1.65736,
+ "190": 1.68141,
+ "191": 1.66484,
+ "192": 1.66865,
+ "193": 1.66429,
+ "194": 1.6974,
+ "195": 1.66792,
+ "196": 1.67775,
+ "197": 1.67631,
+ "198": 1.68824,
+ "199": 1.68154,
+ "200": 1.69356,
+ "201": 1.77027,
+ "202": 1.67388,
+ "203": 1.66554,
+ "204": 1.66753,
+ "205": 1.67386,
+ "206": 1.68106,
+ "207": 1.67661,
+ "208": 1.6776,
+ "209": 1.67686,
+ "210": 1.67766,
+ "211": 1.67833,
+ "212": 1.67922,
+ "213": 1.6773,
+ "214": 1.67735,
+ "215": 1.67664,
+ "216": 1.66653,
+ "217": 1.66772,
+ "218": 1.68155,
+ "219": 1.66501,
+ "220": 1.67378,
+ "221": 1.67079,
+ "222": 1.67791,
+ "223": 1.67601,
+ "224": 1.66539,
+ "225": 1.66981,
+ "226": 1.66491,
+ "227": 1.66698,
+ "228": 1.66192,
+ "229": 1.66036,
+ "230": 1.66691,
+ "231": 1.65832,
+ "232": 1.66767,
+ "233": 1.66414,
+ "234": 1.67389,
+ "235": 1.66835,
+ "236": 1.67448,
+ "237": 1.66312,
+ "238": 1.68801,
+ "239": 1.69467,
+ "240": 1.66703,
+ "241": 1.66552,
+ "242": 1.66422,
+ "243": 1.66627,
+ "244": 1.66985,
+ "245": 1.66801,
+ "246": 1.67359,
+ "247": 1.6692,
+ "248": 1.66749,
+ "249": 1.66388,
+ "250": 1.66655,
+ "251": 1.65971,
+ "252": 1.66594,
+ "253": 1.65865,
+ "254": 1.67247,
+ "255": 1.66616,
+ "256": 1.66735,
+ "257": 1.66326,
+ "258": 1.6681,
+ "259": 1.88457,
+ "260": 1.69114,
+ "261": 1.67138,
+ "262": 1.67021,
+ "263": 1.66116,
+ "264": 1.66989,
+ "265": 1.67733,
+ "266": 1.6664,
+ "267": 1.67194,
+ "268": 1.6678,
+ "269": 1.67004,
+ "270": 1.67386,
+ "271": 1.68389,
+ "272": 1.69024,
+ "273": 1.69418,
+ "274": 1.68099,
+ "275": 1.67153,
+ "276": 1.71533,
+ "277": 1.67414,
+ "278": 1.6626,
+ "279": 1.683,
+ "280": 1.67664,
+ "281": 1.66144,
+ "282": 1.6748,
+ "283": 1.67529,
+ "284": 1.66848,
+ "285": 1.67047,
+ "286": 1.67775,
+ "287": 1.67196,
+ "288": 1.65355,
+ "289": 1.6762,
+ "290": 1.67759,
+ "291": 1.65203,
+ "292": 1.65238,
+ "293": 1.67667,
+ "294": 1.65679,
+ "295": 1.65718,
+ "296": 1.74105,
+ "297": 1.65052,
+ "298": 1.65724,
+ "299": 1.65632,
+ "300": 1.65908,
+ "301": 1.66844,
+ "302": 1.65584,
+ "303": 1.65356,
+ "304": 1.6524,
+ "305": 1.66169,
+ "306": 1.67561,
+ "307": 1.68263,
+ "308": 1.65996,
+ "309": 1.6771,
+ "310": 1.65349,
+ "311": 1.68214,
+ "312": 1.65459,
+ "313": 1.65354,
+ "314": 1.64848,
+ "315": 1.65347,
+ "316": 1.65388,
+ "317": 1.65602,
+ "318": 1.65919,
+ "319": 1.65555,
+ "320": 1.65418,
+ "321": 1.65148,
+ "322": 1.65514,
+ "323": 1.65527,
+ "324": 1.64448,
+ "325": 1.64267,
+ "326": 1.65647,
+ "327": 1.66631,
+ "328": 1.65747,
+ "329": 1.65294,
+ "330": 1.64394,
+ "331": 1.65256,
+ "332": 1.65034,
+ "333": 1.65255,
+ "334": 1.63901,
+ "335": 1.67835,
+ "336": 1.68004,
+ "337": 1.64678,
+ "338": 1.6571,
+ "339": 1.65298,
+ "340": 1.65384,
+ "341": 1.64929,
+ "342": 1.64755,
+ "343": 1.65518,
+ "344": 1.66341,
+ "345": 1.66929,
+ "346": 1.6538,
+ "347": 1.65514,
+ "348": 1.65464,
+ "349": 1.6636,
+ "350": 1.65525,
+ "351": 1.6565,
+ "352": 1.65617,
+ "353": 1.65634,
+ "354": 1.65424,
+ "355": 1.65561,
+ "356": 1.65733,
+ "357": 1.65915,
+ "358": 1.66671,
+ "359": 1.66508,
+ "360": 1.66793,
+ "361": 1.66498,
+ "362": 1.66255,
+ "363": 1.66396,
+ "364": 1.6729,
+ "365": 1.658,
+ "366": 1.65755,
+ "367": 1.65257,
+ "368": 1.66051,
+ "369": 1.66582,
+ "370": 1.66146,
+ "371": 1.65593,
+ "372": 1.65653,
+ "373": 1.66342,
+ "374": 1.66992,
+ "375": 1.67332,
+ "376": 1.6693,
+ "377": 1.6615,
+ "378": 1.6625,
+ "379": 1.66044,
+ "380": 1.66123,
+ "381": 1.65893,
+ "382": 1.65701,
+ "383": 1.65345,
+ "384": 1.6512,
+ "385": 1.66331,
+ "386": 1.65082,
+ "387": 1.65165,
+ "388": 1.65117,
+ "389": 1.65433,
+ "390": 1.66417,
+ "391": 1.65474,
+ "392": 1.65172,
+ "393": 1.65499,
+ "394": 1.65893,
+ "395": 1.65582,
+ "396": 1.65628,
+ "397": 1.66416,
+ "398": 1.65202,
+ "399": 1.65278,
+ "400": 1.65194,
+ "401": 1.65398,
+ "402": 1.65714,
+ "403": 1.64989,
+ "404": 1.6563,
+ "405": 1.64448,
+ "406": 1.70094,
+ "407": 1.74785,
+ "408": 1.79969,
+ "409": 1.76832,
+ "410": 1.65868,
+ "411": 1.66821,
+ "412": 1.65062,
+ "413": 1.66882,
+ "414": 1.69244,
+ "415": 1.6588,
+ "416": 1.65284,
+ "417": 1.65702,
+ "418": 1.64385,
+ "419": 1.65363,
+ "420": 1.65402,
+ "421": 1.65607,
+ "422": 1.66348,
+ "423": 1.65788,
+ "424": 1.65234,
+ "425": 1.65726,
+ "426": 1.65789,
+ "427": 1.657,
+ "428": 1.64877,
+ "429": 1.65707,
+ "430": 1.65112,
+ "431": 1.65615,
+ "432": 1.662,
+ "433": 1.67881,
+ "434": 1.66468,
+ "435": 1.65946,
+ "436": 1.65648,
+ "437": 1.6623,
+ "438": 1.65765,
+ "439": 1.66914,
+ "440": 1.67008,
+ "441": 1.66189,
+ "442": 1.66014,
+ "443": 1.66681,
+ "444": 1.66109,
+ "445": 1.65618,
+ "446": 1.65134,
+ "447": 1.65699,
+ "448": 1.65843,
+ "449": 1.65367,
+ "450": 1.65605,
+ "451": 1.66547,
+ "452": 1.66841,
+ "453": 1.66022,
+ "454": 1.66067,
+ "455": 1.65955,
+ "456": 1.6661,
+ "457": 1.70499,
+ "458": 1.66834,
+ "459": 1.65698,
+ "460": 1.66059,
+ "461": 1.66048,
+ "462": 1.65708,
+ "463": 1.65926,
+ "464": 1.66001,
+ "465": 1.65662,
+ "466": 1.65735,
+ "467": 1.65031,
+ "468": 1.65216,
+ "469": 1.674,
+ "470": 1.6599,
+ "471": 1.6665,
+ "472": 1.66536,
+ "473": 1.66375,
+ "474": 1.65729,
+ "475": 1.65607,
+ "476": 1.67064,
+ "477": 2.06412,
+ "478": 1.66231,
+ "479": 1.6621,
+ "480": 1.66468,
+ "481": 1.6631,
+ "482": 1.67821,
+ "483": 1.66476,
+ "484": 1.65056,
+ "485": 1.6551,
+ "486": 1.65719,
+ "487": 1.71327,
+ "488": 1.6654,
+ "489": 1.66539,
+ "490": 1.6825,
+ "491": 1.96746,
+ "492": 1.65805,
+ "493": 1.65845,
+ "494": 1.66839,
+ "495": 1.66425,
+ "496": 1.66702,
+ "497": 1.66712,
+ "498": 1.66236,
+ "499": 1.66544,
+ "500": 1.66186,
+ "501": 1.66746,
+ "502": 1.70003,
+ "503": 1.68601,
+ "504": 1.66865,
+ "505": 1.69055,
+ "506": 1.66472,
+ "507": 1.66943,
+ "508": 1.67175,
+ "509": 1.661,
+ "510": 1.65918,
+ "511": 1.66158,
+ "512": 1.66404,
+ "513": 1.66571,
+ "514": 1.66037,
+ "515": 1.66881,
+ "516": 1.6581,
+ "517": 2.05273,
+ "518": 1.66695,
+ "519": 1.66928,
+ "520": 1.66052,
+ "521": 1.65807,
+ "522": 1.6726,
+ "523": 1.69055,
+ "524": 1.67417,
+ "525": 1.66854,
+ "526": 1.66702,
+ "527": 1.66432,
+ "528": 1.67837,
+ "529": 1.69108,
+ "530": 1.66243,
+ "531": 1.66133,
+ "532": 1.65867,
+ "533": 1.66438,
+ "534": 1.66007,
+ "535": 1.67106,
+ "536": 1.64225,
+ "537": 1.64491,
+ "538": 1.6443,
+ "539": 1.65291,
+ "540": 1.65684,
+ "541": 1.67062,
+ "542": 1.66116,
+ "543": 1.66137,
+ "544": 1.66159,
+ "545": 1.66375,
+ "546": 1.66946,
+ "547": 1.67578,
+ "548": 1.67439,
+ "549": 1.664,
+ "550": 1.66877,
+ "551": 1.66701,
+ "552": 1.66183,
+ "553": 1.67216,
+ "554": 1.66831,
+ "555": 1.668,
+ "556": 1.66498,
+ "557": 1.6688,
+ "558": 1.66726,
+ "559": 1.68522,
+ "560": 1.67658,
+ "561": 1.71499,
+ "562": 1.67938,
+ "563": 1.68178,
+ "564": 1.67698,
+ "565": 1.67902,
+ "566": 1.67734,
+ "567": 1.67806,
+ "568": 1.67649,
+ "569": 1.68405,
+ "570": 1.67401,
+ "571": 1.67889,
+ "572": 1.67764,
+ "573": 1.68206,
+ "574": 1.67661,
+ "575": 2.0777,
+ "576": 1.68518,
+ "577": 1.68992,
+ "578": 1.6757,
+ "579": 1.68152,
+ "580": 1.68871,
+ "581": 1.68712,
+ "582": 1.67826,
+ "583": 1.68491,
+ "584": 1.68578,
+ "585": 1.68366,
+ "586": 1.67477,
+ "587": 1.67643,
+ "588": 1.6808,
+ "589": 1.67878,
+ "590": 1.69068,
+ "591": 1.66849,
+ "592": 1.68021,
+ "593": 1.67861,
+ "594": 1.68303,
+ "595": 1.68643,
+ "596": 1.68189,
+ "597": 1.6806,
+ "598": 1.67299,
+ "599": 1.68227,
+ "600": 1.69314,
+ "601": 1.68691,
+ "602": 1.68261,
+ "603": 1.6811,
+ "604": 1.67597,
+ "605": 1.70403,
+ "606": 1.67627,
+ "607": 1.6756,
+ "608": 1.67588,
+ "609": 1.68407,
+ "610": 1.68629,
+ "611": 1.6712,
+ "612": 1.69374,
+ "613": 1.69147,
+ "614": 1.68133,
+ "615": 1.68345,
+ "616": 1.6896,
+ "617": 1.68463,
+ "618": 1.6742,
+ "619": 1.67147,
+ "620": 1.6778,
+ "621": 1.67392,
+ "622": 1.6641,
+ "623": 1.65926,
+ "624": 1.66462,
+ "625": 1.66692,
+ "626": 1.65249,
+ "627": 1.66205,
+ "628": 1.66151,
+ "629": 1.66022,
+ "630": 1.67349,
+ "631": 1.66731,
+ "632": 1.66869,
+ "633": 1.66017,
+ "634": 1.66162,
+ "635": 1.65163,
+ "636": 1.67265,
+ "637": 1.66471,
+ "638": 1.65815,
+ "639": 1.66253,
+ "640": 1.66732,
+ "641": 1.66454,
+ "642": 1.65871,
+ "643": 1.66158,
+ "644": 1.65727,
+ "645": 1.67246,
+ "646": 1.65643,
+ "647": 1.65828,
+ "648": 1.67537,
+ "649": 1.65788,
+ "650": 1.65271,
+ "651": 1.6666,
+ "652": 1.66001,
+ "653": 1.6631,
+ "654": 1.65868,
+ "655": 1.66221,
+ "656": 1.65724,
+ "657": 1.65981,
+ "658": 1.66027,
+ "659": 1.66216,
+ "660": 1.66383,
+ "661": 1.66293,
+ "662": 1.66366,
+ "663": 1.66718,
+ "664": 1.65309,
+ "665": 1.66202,
+ "666": 1.669,
+ "667": 1.67234,
+ "668": 1.64903,
+ "669": 1.66573,
+ "670": 1.65878,
+ "671": 1.67009,
+ "672": 1.65589,
+ "673": 1.65157,
+ "674": 1.67236,
+ "675": 1.65364,
+ "676": 1.6664,
+ "677": 1.65348,
+ "678": 1.65249,
+ "679": 1.64996,
+ "680": 1.65259,
+ "681": 1.6559,
+ "682": 1.66026,
+ "683": 1.66154,
+ "684": 1.66651,
+ "685": 1.67141,
+ "686": 1.66069,
+ "687": 1.6617,
+ "688": 1.66302,
+ "689": 1.66431,
+ "690": 1.67049,
+ "691": 1.66551,
+ "692": 1.65952,
+ "693": 1.65933,
+ "694": 1.66146,
+ "695": 1.6683,
+ "696": 1.66495,
+ "697": 1.66168,
+ "698": 1.66691,
+ "699": 1.66908,
+ "700": 1.66973,
+ "701": 1.66751,
+ "702": 1.6729,
+ "703": 1.67398,
+ "704": 1.66427,
+ "705": 1.66618,
+ "706": 1.66382,
+ "707": 1.66007,
+ "708": 1.67343,
+ "709": 1.66832,
+ "710": 1.66621,
+ "711": 1.67123,
+ "712": 1.67002,
+ "713": 1.70431,
+ "714": 1.67656,
+ "715": 1.67479,
+ "716": 1.67325,
+ "717": 1.67234,
+ "718": 1.68373,
+ "719": 1.68247,
+ "720": 1.68466,
+ "721": 1.69537,
+ "722": 1.68021,
+ "723": 1.68088,
+ "724": 1.64835,
+ "725": 1.67454,
+ "726": 1.67322,
+ "727": 1.67583,
+ "728": 1.6662,
+ "729": 1.68668,
+ "730": 1.68133,
+ "731": 1.67647,
+ "732": 1.68216,
+ "733": 1.68607,
+ "734": 1.69048,
+ "735": 1.68349,
+ "736": 1.68381,
+ "737": 1.67292,
+ "738": 1.66802,
+ "739": 1.67667,
+ "740": 1.6675,
+ "741": 1.67255,
+ "742": 1.6634,
+ "743": 1.65614,
+ "744": 1.67607,
+ "745": 1.66387,
+ "746": 1.66508,
+ "747": 1.66142,
+ "748": 1.65751,
+ "749": 1.65439,
+ "750": 1.66554,
+ "751": 1.65829,
+ "752": 1.66185,
+ "753": 1.67574,
+ "754": 1.67037,
+ "755": 1.68164,
+ "756": 1.67675,
+ "757": 1.68421,
+ "758": 1.67211,
+ "759": 1.67314,
+ "760": 1.67804,
+ "761": 1.68494,
+ "762": 1.67554,
+ "763": 1.66966,
+ "764": 1.67786,
+ "765": 1.67234,
+ "766": 1.67548,
+ "767": 1.67379,
+ "768": 1.66338,
+ "769": 1.65725,
+ "770": 1.65838,
+ "771": 1.69975,
+ "772": 1.68894,
+ "773": 1.67298,
+ "774": 1.66917,
+ "775": 1.67149,
+ "776": 1.66059,
+ "777": 1.66827,
+ "778": 1.66158,
+ "779": 1.66458,
+ "780": 1.67482,
+ "781": 1.66875,
+ "782": 1.6617,
+ "783": 1.66676,
+ "784": 1.67501,
+ "785": 1.67307,
+ "786": 1.67458,
+ "787": 1.67176,
+ "788": 1.66588,
+ "789": 1.73488,
+ "790": 1.67459,
+ "791": 1.67558,
+ "792": 1.6811,
+ "793": 1.67985,
+ "794": 1.67466,
+ "795": 1.67017,
+ "796": 1.68351,
+ "797": 1.68604,
+ "798": 1.6866,
+ "799": 1.67723,
+ "800": 1.70113,
+ "801": 1.67186,
+ "802": 1.65766,
+ "803": 1.68074,
+ "804": 1.68879,
+ "805": 1.68862,
+ "806": 1.68776,
+ "807": 1.68531,
+ "808": 1.6947,
+ "809": 1.68399,
+ "810": 1.69225,
+ "811": 1.69698,
+ "812": 1.68933,
+ "813": 1.68258,
+ "814": 1.685,
+ "815": 1.68536,
+ "816": 1.68313,
+ "817": 1.69305,
+ "818": 1.67452,
+ "819": 1.67792,
+ "820": 1.68594,
+ "821": 1.67842,
+ "822": 1.67198,
+ "823": 1.68128,
+ "824": 1.66627,
+ "825": 1.67852,
+ "826": 1.66593,
+ "827": 1.68474,
+ "828": 1.68503,
+ "829": 1.67553,
+ "830": 1.67334,
+ "831": 1.67121,
+ "832": 1.66846,
+ "833": 1.68649,
+ "834": 1.67049,
+ "835": 1.66976,
+ "836": 1.67409,
+ "837": 1.67184,
+ "838": 1.67272,
+ "839": 1.6704,
+ "840": 1.67203,
+ "841": 1.67262,
+ "842": 1.69392,
+ "843": 1.67646,
+ "844": 1.67224,
+ "845": 1.67677,
+ "846": 1.68374,
+ "847": 1.67953,
+ "848": 1.67433,
+ "849": 1.67289,
+ "850": 1.67019,
+ "851": 1.67607,
+ "852": 1.67445,
+ "853": 1.67253,
+ "854": 1.66588,
+ "855": 1.65948,
+ "856": 1.66108,
+ "857": 1.66339,
+ "858": 1.6618,
+ "859": 1.66206,
+ "860": 1.66373,
+ "861": 1.66838,
+ "862": 1.66554,
+ "863": 1.68108,
+ "864": 1.70406,
+ "865": 1.67187,
+ "866": 1.65897,
+ "867": 1.65992,
+ "868": 1.66854,
+ "869": 1.66168,
+ "870": 1.66367,
+ "871": 1.66886,
+ "872": 1.66319,
+ "873": 1.66839,
+ "874": 1.66977,
+ "875": 1.66615,
+ "876": 1.66553,
+ "877": 1.65766,
+ "878": 1.66596,
+ "879": 1.66414,
+ "880": 1.67152,
+ "881": 1.66996,
+ "882": 1.65807,
+ "883": 1.66451,
+ "884": 1.66807,
+ "885": 1.67105,
+ "886": 1.65852,
+ "887": 1.66682,
+ "888": 1.66686,
+ "889": 1.66925,
+ "890": 1.65721,
+ "891": 1.65268,
+ "892": 1.6582,
+ "893": 1.67573,
+ "894": 1.67907,
+ "895": 1.68801,
+ "896": 1.67708,
+ "897": 1.67243,
+ "898": 1.68166,
+ "899": 1.67597,
+ "900": 1.68752,
+ "901": 1.68085,
+ "902": 1.6708,
+ "903": 1.66987,
+ "904": 1.68093,
+ "905": 1.67679,
+ "906": 1.68457,
+ "907": 1.68883,
+ "908": 1.68548,
+ "909": 1.68698,
+ "910": 1.67594,
+ "911": 1.68156,
+ "912": 1.68866,
+ "913": 1.68265,
+ "914": 1.68821,
+ "915": 1.68128,
+ "916": 1.67355,
+ "917": 1.70903,
+ "918": 1.68779,
+ "919": 1.67941,
+ "920": 1.691,
+ "921": 1.68974,
+ "922": 1.68641,
+ "923": 1.68801,
+ "924": 1.68673,
+ "925": 1.69406,
+ "926": 1.6851,
+ "927": 1.68456,
+ "928": 1.68821,
+ "929": 1.67223,
+ "930": 1.68276,
+ "931": 1.67235,
+ "932": 1.67063,
+ "933": 1.66444,
+ "934": 1.67133,
+ "935": 1.68216,
+ "936": 1.68375,
+ "937": 1.67348,
+ "938": 1.68749,
+ "939": 1.7238,
+ "940": 1.69206,
+ "941": 1.68943,
+ "942": 1.69023,
+ "943": 1.68901,
+ "944": 1.68916,
+ "945": 1.69043,
+ "946": 1.68901,
+ "947": 1.70685,
+ "948": 1.69038,
+ "949": 1.69238,
+ "950": 1.69651,
+ "951": 1.68187,
+ "952": 1.69212,
+ "953": 1.69019,
+ "954": 1.69154,
+ "955": 1.68314,
+ "956": 1.69188,
+ "957": 1.68913,
+ "958": 1.687,
+ "959": 1.68936,
+ "960": 1.69969,
+ "961": 1.68448,
+ "962": 1.68638,
+ "963": 1.68613,
+ "964": 1.68898,
+ "965": 1.6837,
+ "966": 1.68386,
+ "967": 1.68657,
+ "968": 1.67965,
+ "969": 1.68951,
+ "970": 1.68497,
+ "971": 1.69574,
+ "972": 1.69006,
+ "973": 1.68732,
+ "974": 1.67503,
+ "975": 1.66451,
+ "976": 1.67807,
+ "977": 1.66935,
+ "978": 1.67192,
+ "979": 1.68033,
+ "980": 1.67529,
+ "981": 1.67115,
+ "982": 1.67082,
+ "983": 1.66979,
+ "984": 1.66114,
+ "985": 1.65283,
+ "986": 1.6658,
+ "987": 1.66778,
+ "988": 1.65696,
+ "989": 1.67278,
+ "990": 1.65672,
+ "991": 1.67991,
+ "992": 1.67594,
+ "993": 1.68565,
+ "994": 1.68296,
+ "995": 1.68711,
+ "996": 1.68931,
+ "997": 1.6808,
+ "998": 1.68412,
+ "999": 1.68911,
+ "1000": 1.69059,
+ "1001": 1.67329,
+ "1002": 1.68037,
+ "1003": 1.66565,
+ "1004": 1.68096,
+ "1005": 1.67604,
+ "1006": 1.63714,
+ "1007": 1.65161,
+ "1008": 1.66515,
+ "1009": 1.69278,
+ "1010": 1.68183,
+ "1011": 1.68684,
+ "1012": 1.67633,
+ "1013": 1.67887,
+ "1014": 1.68059,
+ "1015": 1.6961,
+ "1016": 1.69958,
+ "1017": 1.67937,
+ "1018": 1.66427,
+ "1019": 1.6838,
+ "1020": 1.67037,
+ "1021": 1.66959,
+ "1022": 1.68289,
+ "1023": 1.67886,
+ "1024": 1.68436,
+ "1025": 1.67014,
+ "1026": 1.66918,
+ "1027": 1.67754,
+ "1028": 1.66669,
+ "1029": 1.66255,
+ "1030": 1.66152,
+ "1031": 1.68652,
+ "1032": 1.67831,
+ "1033": 1.66376,
+ "1034": 1.67477,
+ "1035": 1.65563,
+ "1036": 1.68612,
+ "1037": 1.67832,
+ "1038": 1.67899,
+ "1039": 1.67561,
+ "1040": 1.65907,
+ "1041": 1.66772,
+ "1042": 1.66424,
+ "1043": 1.66918,
+ "1044": 1.66594,
+ "1045": 1.67661,
+ "1046": 1.67556,
+ "1047": 1.67132,
+ "1048": 1.67336,
+ "1049": 1.6751,
+ "1050": 1.66904,
+ "1051": 1.67174,
+ "1052": 1.67131,
+ "1053": 1.67225,
+ "1054": 1.67899,
+ "1055": 1.68179,
+ "1056": 1.68271,
+ "1057": 1.67216,
+ "1058": 1.66542,
+ "1059": 1.67399,
+ "1060": 1.67486,
+ "1061": 1.66147,
+ "1062": 1.6655,
+ "1063": 1.67114,
+ "1064": 1.67063,
+ "1065": 1.66658,
+ "1066": 1.66383,
+ "1067": 1.66965,
+ "1068": 1.66995,
+ "1069": 1.67189,
+ "1070": 1.67241,
+ "1071": 1.66707,
+ "1072": 1.66563,
+ "1073": 1.66802,
+ "1074": 1.66793,
+ "1075": 1.67205,
+ "1076": 1.67002,
+ "1077": 1.67186,
+ "1078": 1.66905,
+ "1079": 1.66899,
+ "1080": 1.6687,
+ "1081": 1.67113,
+ "1082": 1.6709,
+ "1083": 1.69655,
+ "1084": 1.66949,
+ "1085": 1.67021,
+ "1086": 1.6777,
+ "1087": 1.6726,
+ "1088": 1.68449,
+ "1089": 1.65859,
+ "1090": 1.6723,
+ "1091": 1.6587,
+ "1092": 1.67301,
+ "1093": 1.66636,
+ "1094": 1.65698,
+ "1095": 1.67112,
+ "1096": 1.65719,
+ "1097": 1.65382,
+ "1098": 1.66007,
+ "1099": 1.66073,
+ "1100": 1.65742,
+ "1101": 1.65956,
+ "1102": 1.66432,
+ "1103": 1.66356,
+ "1104": 1.66065,
+ "1105": 1.66633,
+ "1106": 1.66001,
+ "1107": 1.66098,
+ "1108": 1.66374,
+ "1109": 1.65969,
+ "1110": 1.6655,
+ "1111": 1.66305,
+ "1112": 1.66314,
+ "1113": 1.66446,
+ "1114": 1.66361,
+ "1115": 1.66219,
+ "1116": 1.66633,
+ "1117": 1.66727,
+ "1118": 1.66405,
+ "1119": 1.66884,
+ "1120": 1.66933,
+ "1121": 1.66823,
+ "1122": 1.67108,
+ "1123": 1.65942,
+ "1124": 1.67026,
+ "1125": 1.66804,
+ "1126": 1.66582,
+ "1127": 1.67314,
+ "1128": 1.66048,
+ "1129": 1.6649,
+ "1130": 1.6622,
+ "1131": 1.66952,
+ "1132": 1.66301,
+ "1133": 1.6625,
+ "1134": 1.66196,
+ "1135": 1.66389,
+ "1136": 1.66092,
+ "1137": 1.6661,
+ "1138": 1.66709,
+ "1139": 1.66129,
+ "1140": 1.66049,
+ "1141": 1.65438,
+ "1142": 1.66341,
+ "1143": 1.6575,
+ "1144": 1.66522,
+ "1145": 1.67469,
+ "1146": 1.65551,
+ "1147": 1.65691,
+ "1148": 1.66411,
+ "1149": 1.66252,
+ "1150": 1.67936,
+ "1151": 1.67901,
+ "1152": 1.67608,
+ "1153": 1.68069,
+ "1154": 1.683,
+ "1155": 1.68007,
+ "1156": 1.67153,
+ "1157": 1.67684,
+ "1158": 1.67692,
+ "1159": 1.6769,
+ "1160": 1.67717,
+ "1161": 1.67424,
+ "1162": 1.68452,
+ "1163": 1.67428,
+ "1164": 1.66878,
+ "1165": 1.68096,
+ "1166": 1.68239,
+ "1167": 1.68129,
+ "1168": 1.66883,
+ "1169": 1.67562,
+ "1170": 1.68246,
+ "1171": 1.68124,
+ "1172": 1.67485,
+ "1173": 1.68025,
+ "1174": 1.68613,
+ "1175": 1.67748,
+ "1176": 1.67696,
+ "1177": 1.67656,
+ "1178": 1.70884,
+ "1179": 1.67871,
+ "1180": 1.67135,
+ "1181": 1.66808,
+ "1182": 1.67186,
+ "1183": 1.66717,
+ "1184": 1.675,
+ "1185": 1.68092,
+ "1186": 1.68152,
+ "1187": 1.6804,
+ "1188": 1.66984,
+ "1189": 1.66388,
+ "1190": 1.66654,
+ "1191": 1.66691,
+ "1192": 1.67197,
+ "1193": 1.66518,
+ "1194": 1.6618,
+ "1195": 1.66114,
+ "1196": 1.66685,
+ "1197": 1.67053,
+ "1198": 1.66867,
+ "1199": 1.67506,
+ "1200": 1.66234,
+ "1201": 1.65999,
+ "1202": 1.66013,
+ "1203": 1.66581,
+ "1204": 1.66379,
+ "1205": 1.66378,
+ "1206": 1.65613,
+ "1207": 2.0144,
+ "1208": 1.66017,
+ "1209": 1.67357,
+ "1210": 1.66786,
+ "1211": 1.66254,
+ "1212": 1.67143,
+ "1213": 1.67589,
+ "1214": 1.66843,
+ "1215": 1.67158,
+ "1216": 1.68238,
+ "1217": 1.93752,
+ "1218": 1.67213,
+ "1219": 1.67718,
+ "1220": 1.66327,
+ "1221": 1.66691,
+ "1222": 1.67218,
+ "1223": 1.66736,
+ "1224": 1.6697,
+ "1225": 1.66522,
+ "1226": 1.66986,
+ "1227": 1.67132,
+ "1228": 1.6502,
+ "1229": 1.66488,
+ "1230": 1.67411,
+ "1231": 1.67465,
+ "1232": 1.67276,
+ "1233": 1.66284,
+ "1234": 1.67192,
+ "1235": 1.66858,
+ "1236": 1.6786,
+ "1237": 1.67619,
+ "1238": 1.6697,
+ "1239": 1.67168,
+ "1240": 1.67857,
+ "1241": 1.68057,
+ "1242": 1.68296,
+ "1243": 1.68293,
+ "1244": 1.67508,
+ "1245": 1.68038,
+ "1246": 1.68236,
+ "1247": 1.68675,
+ "1248": 1.67777,
+ "1249": 1.68082,
+ "1250": 1.67455,
+ "1251": 1.68172,
+ "1252": 1.70275,
+ "1253": 1.67225,
+ "1254": 1.66336,
+ "1255": 1.67891,
+ "1256": 1.68118,
+ "1257": 1.67656,
+ "1258": 1.67761,
+ "1259": 1.66753,
+ "1260": 1.67705,
+ "1261": 1.67978,
+ "1262": 1.68455,
+ "1263": 1.67997,
+ "1264": 1.6614,
+ "1265": 1.66613,
+ "1266": 1.67334,
+ "1267": 1.6664,
+ "1268": 1.65418,
+ "1269": 1.65813,
+ "1270": 1.67577,
+ "1271": 1.68002,
+ "1272": 1.67564,
+ "1273": 1.67418,
+ "1274": 1.67734,
+ "1275": 1.66991,
+ "1276": 1.67677,
+ "1277": 1.67327,
+ "1278": 1.66763,
+ "1279": 1.70782,
+ "1280": 1.65816,
+ "1281": 1.6681,
+ "1282": 1.67083,
+ "1283": 1.67735,
+ "1284": 1.67818,
+ "1285": 1.66252,
+ "1286": 1.67109,
+ "1287": 1.66818,
+ "1288": 1.67397,
+ "1289": 1.70275,
+ "1290": 1.67323,
+ "1291": 1.66925,
+ "1292": 1.67013,
+ "1293": 1.67219,
+ "1294": 1.6707,
+ "1295": 1.66291,
+ "1296": 1.67061,
+ "1297": 1.6661,
+ "1298": 1.67508,
+ "1299": 1.67383,
+ "1300": 1.67855,
+ "1301": 1.66793,
+ "1302": 1.66891,
+ "1303": 1.66824,
+ "1304": 1.66844,
+ "1305": 1.66834,
+ "1306": 1.67841,
+ "1307": 1.67241,
+ "1308": 1.67273,
+ "1309": 1.67051,
+ "1310": 1.67323,
+ "1311": 1.67215,
+ "1312": 1.67998,
+ "1313": 1.67913,
+ "1314": 1.66354,
+ "1315": 1.66151,
+ "1316": 1.65667,
+ "1317": 1.66285,
+ "1318": 1.67023,
+ "1319": 1.67304,
+ "1320": 1.67377,
+ "1321": 1.68023,
+ "1322": 1.68169,
+ "1323": 1.6638,
+ "1324": 1.66886,
+ "1325": 1.66964,
+ "1326": 1.66896,
+ "1327": 1.6732,
+ "1328": 1.67057,
+ "1329": 1.67122,
+ "1330": 1.67218,
+ "1331": 1.66726,
+ "1332": 1.6703,
+ "1333": 1.67547,
+ "1334": 1.67656,
+ "1335": 1.68177,
+ "1336": 1.67346,
+ "1337": 1.66507,
+ "1338": 1.66151,
+ "1339": 1.67012,
+ "1340": 1.67486,
+ "1341": 1.67665,
+ "1342": 1.67034,
+ "1343": 1.66906,
+ "1344": 1.66058,
+ "1345": 1.6725,
+ "1346": 1.67657,
+ "1347": 1.66223,
+ "1348": 1.6717,
+ "1349": 1.6698,
+ "1350": 1.67128,
+ "1351": 1.66234,
+ "1352": 1.66706,
+ "1353": 1.68181,
+ "1354": 1.66644,
+ "1355": 1.67379,
+ "1356": 1.67069,
+ "1357": 1.67291,
+ "1358": 1.67407,
+ "1359": 1.66811,
+ "1360": 1.66826,
+ "1361": 1.66579,
+ "1362": 1.66586,
+ "1363": 1.65984,
+ "1364": 1.66441,
+ "1365": 1.66483,
+ "1366": 1.64867,
+ "1367": 1.6595,
+ "1368": 1.67572,
+ "1369": 1.67074,
+ "1370": 1.67434,
+ "1371": 1.67847,
+ "1372": 1.6702,
+ "1373": 1.66993,
+ "1374": 1.66027,
+ "1375": 1.66704,
+ "1376": 1.66545,
+ "1377": 1.66729,
+ "1378": 1.66908,
+ "1379": 1.66178,
+ "1380": 1.65841,
+ "1381": 1.66812,
+ "1382": 1.66488,
+ "1383": 1.67112,
+ "1384": 1.68015,
+ "1385": 1.67961,
+ "1386": 1.67456,
+ "1387": 1.68029,
+ "1388": 1.67644,
+ "1389": 1.68095,
+ "1390": 1.68059,
+ "1391": 1.72961,
+ "1392": 1.71194,
+ "1393": 1.68764,
+ "1394": 1.68216,
+ "1395": 1.68497,
+ "1396": 1.66683,
+ "1397": 1.68176,
+ "1398": 1.68215,
+ "1399": 1.67436,
+ "1400": 1.66888,
+ "1401": 1.67242,
+ "1402": 1.65693,
+ "1403": 1.6764,
+ "1404": 1.66871,
+ "1405": 1.67625,
+ "1406": 1.66777,
+ "1407": 1.67683,
+ "1408": 1.6815,
+ "1409": 1.67801,
+ "1410": 1.67556,
+ "1411": 1.67652,
+ "1412": 1.6782,
+ "1413": 1.66969,
+ "1414": 1.67552,
+ "1415": 1.67366,
+ "1416": 1.67276,
+ "1417": 1.67412,
+ "1418": 1.68263,
+ "1419": 1.67562,
+ "1420": 1.67342,
+ "1421": 1.67866,
+ "1422": 1.66876,
+ "1423": 1.67018,
+ "1424": 1.68186,
+ "1425": 1.6866,
+ "1426": 1.70191,
+ "1427": 1.67833,
+ "1428": 1.69173,
+ "1429": 1.68678,
+ "1430": 1.6921,
+ "1431": 1.6879,
+ "1432": 1.6855,
+ "1433": 1.68284,
+ "1434": 1.66877,
+ "1435": 1.67579,
+ "1436": 1.68983,
+ "1437": 1.67187,
+ "1438": 1.68329,
+ "1439": 1.69205,
+ "1440": 1.67922,
+ "1441": 1.68166,
+ "1442": 1.69682,
+ "1443": 1.68988,
+ "1444": 2.01957,
+ "1445": 1.68686,
+ "1446": 1.68912,
+ "1447": 1.68837,
+ "1448": 1.706,
+ "1449": 1.67685,
+ "1450": 1.67146,
+ "1451": 1.67423,
+ "1452": 1.67834,
+ "1453": 1.69856,
+ "1454": 1.67979,
+ "1455": 1.67939,
+ "1456": 1.6797,
+ "1457": 1.69803,
+ "1458": 1.68295,
+ "1459": 1.681,
+ "1460": 1.68519,
+ "1461": 1.67413,
+ "1462": 1.67158,
+ "1463": 1.67529,
+ "1464": 1.6887,
+ "1465": 1.68065,
+ "1466": 1.66708,
+ "1467": 1.68199,
+ "1468": 1.68572,
+ "1469": 1.68658,
+ "1470": 1.68501,
+ "1471": 1.67676,
+ "1472": 1.66949,
+ "1473": 1.67044,
+ "1474": 1.68003,
+ "1475": 1.68454,
+ "1476": 1.68851,
+ "1477": 1.68441,
+ "1478": 1.68385,
+ "1479": 1.68346,
+ "1480": 1.68391,
+ "1481": 1.68722,
+ "1482": 1.67579,
+ "1483": 1.6856,
+ "1484": 1.6816,
+ "1485": 1.69364,
+ "1486": 1.67997,
+ "1487": 1.68547,
+ "1488": 1.6896,
+ "1489": 1.68189,
+ "1490": 1.68829,
+ "1491": 1.68858,
+ "1492": 1.68329,
+ "1493": 1.68677,
+ "1494": 1.68391,
+ "1495": 1.67982,
+ "1496": 1.68246,
+ "1497": 1.68725,
+ "1498": 1.68335,
+ "1499": 1.69265,
+ "1500": 1.6873,
+ "1501": 1.67816,
+ "1502": 1.69388,
+ "1503": 1.68874,
+ "1504": 1.67933,
+ "1505": 1.67416,
+ "1506": 1.67693,
+ "1507": 1.68022,
+ "1508": 1.68193,
+ "1509": 1.67083,
+ "1510": 1.67684,
+ "1511": 1.67347,
+ "1512": 1.67444,
+ "1513": 1.67844,
+ "1514": 1.6928,
+ "1515": 2.04601,
+ "1516": 1.68548,
+ "1517": 1.68113,
+ "1518": 1.67347,
+ "1519": 1.67537,
+ "1520": 1.68799,
+ "1521": 1.68817,
+ "1522": 1.6805,
+ "1523": 1.68408,
+ "1524": 1.68164,
+ "1525": 1.67889,
+ "1526": 1.67647,
+ "1527": 1.67783,
+ "1528": 1.65863,
+ "1529": 1.65014,
+ "1530": 1.6634,
+ "1531": 1.65667,
+ "1532": 1.64986,
+ "1533": 1.66257,
+ "1534": 1.69059,
+ "1535": 1.64338,
+ "1536": 1.64802,
+ "1537": 1.69142,
+ "1538": 1.66714,
+ "1539": 1.66571,
+ "1540": 1.68264,
+ "1541": 1.68079,
+ "1542": 1.67733,
+ "1543": 1.68508,
+ "1544": 1.68295,
+ "1545": 1.69014,
+ "1546": 1.68579,
+ "1547": 1.68315,
+ "1548": 1.68162,
+ "1549": 1.66951,
+ "1550": 1.67403,
+ "1551": 1.67829,
+ "1552": 1.67069,
+ "1553": 1.66737,
+ "1554": 1.66653,
+ "1555": 1.66778,
+ "1556": 1.66985,
+ "1557": 1.67123,
+ "1558": 1.65434,
+ "1559": 1.67259,
+ "1560": 1.66766,
+ "1561": 1.66868,
+ "1562": 1.67355,
+ "1563": 1.66406,
+ "1564": 1.6687,
+ "1565": 1.66951,
+ "1566": 1.67126,
+ "1567": 1.67182,
+ "1568": 1.67566,
+ "1569": 1.66313,
+ "1570": 1.6672,
+ "1571": 1.66881,
+ "1572": 1.66497,
+ "1573": 1.6691,
+ "1574": 1.67036,
+ "1575": 1.66945,
+ "1576": 1.67514,
+ "1577": 1.66786,
+ "1578": 1.66636,
+ "1579": 1.67313,
+ "1580": 1.6672,
+ "1581": 1.66815,
+ "1582": 1.67342,
+ "1583": 1.67434,
+ "1584": 1.68006,
+ "1585": 1.67225,
+ "1586": 1.66457,
+ "1587": 1.67075,
+ "1588": 1.67702,
+ "1589": 1.67325,
+ "1590": 1.66328,
+ "1591": 1.68268,
+ "1592": 1.67759,
+ "1593": 1.67632,
+ "1594": 1.66945,
+ "1595": 1.67408,
+ "1596": 1.668,
+ "1597": 1.67833,
+ "1598": 1.66851,
+ "1599": 1.66618,
+ "1600": 1.66334,
+ "1601": 1.6672,
+ "1602": 1.66443,
+ "1603": 1.66414,
+ "1604": 1.67506,
+ "1605": 1.66508,
+ "1606": 1.66172,
+ "1607": 1.70205,
+ "1608": 1.68273,
+ "1609": 1.68571,
+ "1610": 1.66672,
+ "1611": 1.6725,
+ "1612": 1.67288,
+ "1613": 1.66903,
+ "1614": 1.66543,
+ "1615": 1.65616,
+ "1616": 1.66599,
+ "1617": 1.66445,
+ "1618": 1.66741,
+ "1619": 1.67282,
+ "1620": 1.66564,
+ "1621": 1.66979,
+ "1622": 1.67316,
+ "1623": 1.70561,
+ "1624": 1.65444,
+ "1625": 1.66748,
+ "1626": 1.66589,
+ "1627": 1.68626,
+ "1628": 1.67481,
+ "1629": 1.66675,
+ "1630": 1.665,
+ "1631": 1.66499,
+ "1632": 1.66938,
+ "1633": 1.70528,
+ "1634": 1.68477,
+ "1635": 1.67508,
+ "1636": 1.6662,
+ "1637": 1.66655,
+ "1638": 1.66538,
+ "1639": 1.66556,
+ "1640": 1.66259,
+ "1641": 1.66386,
+ "1642": 1.66038,
+ "1643": 1.6698,
+ "1644": 1.66701,
+ "1645": 1.67538,
+ "1646": 1.67125,
+ "1647": 1.65938,
+ "1648": 1.66482,
+ "1649": 1.67374,
+ "1650": 1.66418,
+ "1651": 1.66486,
+ "1652": 1.66451,
+ "1653": 1.66553,
+ "1654": 1.66894,
+ "1655": 1.67177,
+ "1656": 1.6698,
+ "1657": 1.66775,
+ "1658": 1.67207,
+ "1659": 1.67418,
+ "1660": 1.66976,
+ "1661": 1.66932,
+ "1662": 1.66951,
+ "1663": 1.67432,
+ "1664": 1.69804,
+ "1665": 1.65614,
+ "1666": 1.66101,
+ "1667": 1.66935,
+ "1668": 1.66999,
+ "1669": 1.67251,
+ "1670": 1.66861,
+ "1671": 1.66838,
+ "1672": 1.68362,
+ "1673": 1.67611,
+ "1674": 1.68187,
+ "1675": 1.66942,
+ "1676": 1.67313,
+ "1677": 1.6745,
+ "1678": 1.66068,
+ "1679": 1.66653,
+ "1680": 1.66488,
+ "1681": 1.69635,
+ "1682": 1.67758,
+ "1683": 1.66878,
+ "1684": 1.66176,
+ "1685": 1.66033,
+ "1686": 1.6668,
+ "1687": 1.66999,
+ "1688": 1.67193,
+ "1689": 1.66356,
+ "1690": 1.665,
+ "1691": 1.6668,
+ "1692": 1.66859,
+ "1693": 1.66755,
+ "1694": 1.67361,
+ "1695": 1.67173,
+ "1696": 1.66681,
+ "1697": 1.66953,
+ "1698": 1.67545,
+ "1699": 1.67505,
+ "1700": 1.68215,
+ "1701": 1.67195,
+ "1702": 1.68931,
+ "1703": 1.67657,
+ "1704": 1.6692,
+ "1705": 1.66117,
+ "1706": 1.66997,
+ "1707": 1.66651,
+ "1708": 1.69071,
+ "1709": 1.69161,
+ "1710": 1.70228,
+ "1711": 1.69129,
+ "1712": 1.69679,
+ "1713": 1.70151,
+ "1714": 1.6904,
+ "1715": 1.69352,
+ "1716": 1.69972,
+ "1717": 1.70504,
+ "1718": 1.70116,
+ "1719": 1.69128,
+ "1720": 1.69033,
+ "1721": 1.70057,
+ "1722": 1.69583,
+ "1723": 1.69125,
+ "1724": 1.69034,
+ "1725": 1.69701,
+ "1726": 1.69367,
+ "1727": 1.68985,
+ "1728": 1.69137,
+ "1729": 1.69034,
+ "1730": 1.68627,
+ "1731": 1.69182,
+ "1732": 1.68666,
+ "1733": 1.69093,
+ "1734": 1.70135,
+ "1735": 1.7008,
+ "1736": 1.6968,
+ "1737": 1.6914,
+ "1738": 1.65543,
+ "1739": 1.66836,
+ "1740": 1.67357,
+ "1741": 1.6671,
+ "1742": 1.66359,
+ "1743": 1.66419,
+ "1744": 1.66805,
+ "1745": 1.66315,
+ "1746": 1.66476,
+ "1747": 1.65638,
+ "1748": 1.6686,
+ "1749": 1.67166,
+ "1750": 1.67273,
+ "1751": 1.67314,
+ "1752": 1.67432,
+ "1753": 1.67468,
+ "1754": 1.67716,
+ "1755": 1.66204,
+ "1756": 1.68613,
+ "1757": 1.67363,
+ "1758": 1.65889,
+ "1759": 1.65396,
+ "1760": 1.64763,
+ "1761": 1.65609,
+ "1762": 1.6596,
+ "1763": 1.6604,
+ "1764": 1.66728,
+ "1765": 1.67694,
+ "1766": 1.66303,
+ "1767": 1.66934,
+ "1768": 1.66864,
+ "1769": 2.40884,
+ "1770": 1.67104,
+ "1771": 1.83552,
+ "1772": 1.6667,
+ "1773": 1.66707,
+ "1774": 1.66906,
+ "1775": 1.66875,
+ "1776": 1.66476,
+ "1777": 1.67642,
+ "1778": 1.66363,
+ "1779": 1.89354,
+ "1780": 1.67021,
+ "1781": 1.67,
+ "1782": 1.6655,
+ "1783": 1.67588,
+ "1784": 1.66342,
+ "1785": 1.6658,
+ "1786": 1.66747,
+ "1787": 1.67577,
+ "1788": 1.69475,
+ "1789": 1.6715,
+ "1790": 1.67521,
+ "1791": 1.67204,
+ "1792": 1.67072,
+ "1793": 1.66782,
+ "1794": 1.66988,
+ "1795": 1.67116,
+ "1796": 1.6604,
+ "1797": 1.66546,
+ "1798": 1.66315,
+ "1799": 1.66874,
+ "1800": 1.67935,
+ "1801": 1.67592,
+ "1802": 1.6741,
+ "1803": 1.67832,
+ "1804": 1.67892,
+ "1805": 1.66926,
+ "1806": 1.66126,
+ "1807": 1.64187,
+ "1808": 1.64824,
+ "1809": 1.65132,
+ "1810": 1.64729,
+ "1811": 1.64567,
+ "1812": 1.6489,
+ "1813": 1.65217,
+ "1814": 1.65252,
+ "1815": 1.65227,
+ "1816": 1.64334,
+ "1817": 1.64742,
+ "1818": 1.67114,
+ "1819": 1.66848,
+ "1820": 1.67529,
+ "1821": 1.6802,
+ "1822": 1.67945,
+ "1823": 1.67185,
+ "1824": 1.6815,
+ "1825": 1.67411,
+ "1826": 1.67014,
+ "1827": 1.68186,
+ "1828": 1.67636,
+ "1829": 1.66784,
+ "1830": 1.6758,
+ "1831": 1.67235,
+ "1832": 1.67027,
+ "1833": 1.66056,
+ "1834": 1.67415,
+ "1835": 1.67323,
+ "1836": 1.68285,
+ "1837": 1.68142,
+ "1838": 1.67312,
+ "1839": 1.65643,
+ "1840": 1.65856,
+ "1841": 1.66345,
+ "1842": 1.66996,
+ "1843": 1.67516,
+ "1844": 1.68144,
+ "1845": 1.67107,
+ "1846": 1.69384,
+ "1847": 1.70009,
+ "1848": 1.68413,
+ "1849": 1.6772,
+ "1850": 1.6857,
+ "1851": 1.68846,
+ "1852": 1.67466,
+ "1853": 1.67741,
+ "1854": 1.66746,
+ "1855": 1.64931,
+ "1856": 1.63911,
+ "1857": 1.65241,
+ "1858": 1.66714,
+ "1859": 1.66601,
+ "1860": 1.67194,
+ "1861": 1.67196,
+ "1862": 1.66863,
+ "1863": 1.67747,
+ "1864": 1.67022,
+ "1865": 1.67496,
+ "1866": 1.66164,
+ "1867": 1.66717,
+ "1868": 1.67927,
+ "1869": 1.66994,
+ "1870": 1.67746,
+ "1871": 1.66638,
+ "1872": 1.66454,
+ "1873": 1.67252,
+ "1874": 1.66761,
+ "1875": 1.67057,
+ "1876": 1.67388,
+ "1877": 1.71218,
+ "1878": 1.66698,
+ "1879": 1.66441,
+ "1880": 1.66266,
+ "1881": 1.66381,
+ "1882": 1.669,
+ "1883": 1.66914,
+ "1884": 1.67276,
+ "1885": 1.67361,
+ "1886": 1.64481,
+ "1887": 1.65931,
+ "1888": 1.66352,
+ "1889": 1.66337,
+ "1890": 1.6683,
+ "1891": 1.66351,
+ "1892": 1.65669,
+ "1893": 1.64217,
+ "1894": 1.66872,
+ "1895": 1.68262,
+ "1896": 1.6616,
+ "1897": 1.64828,
+ "1898": 1.63784,
+ "1899": 1.6625,
+ "1900": 1.65942,
+ "1901": 1.63623,
+ "1902": 1.68296,
+ "1903": 1.66024,
+ "1904": 1.66268,
+ "1905": 1.63688,
+ "1906": 1.64583,
+ "1907": 1.63816,
+ "1908": 1.64619,
+ "1909": 1.66117,
+ "1910": 1.63797,
+ "1911": 1.63839,
+ "1912": 1.63868,
+ "1913": 1.6546,
+ "1914": 1.64703,
+ "1915": 1.64279,
+ "1916": 1.66491,
+ "1917": 1.65153,
+ "1918": 1.64185,
+ "1919": 1.63702,
+ "1920": 1.6406,
+ "1921": 1.64608,
+ "1922": 1.66261,
+ "1923": 1.63942,
+ "1924": 1.6434,
+ "1925": 1.63938,
+ "1926": 1.63948,
+ "1927": 1.63764,
+ "1928": 1.63944,
+ "1929": 1.66754,
+ "1930": 1.6439,
+ "1931": 1.64959,
+ "1932": 1.67446,
+ "1933": 1.66232,
+ "1934": 1.66056,
+ "1935": 1.65011,
+ "1936": 1.64755,
+ "1937": 1.64757,
+ "1938": 1.6481,
+ "1939": 1.6599,
+ "1940": 1.66496,
+ "1941": 1.66147,
+ "1942": 1.66511,
+ "1943": 1.67533,
+ "1944": 1.66355,
+ "1945": 1.66358,
+ "1946": 1.66286,
+ "1947": 1.66621,
+ "1948": 1.66905,
+ "1949": 1.66871,
+ "1950": 1.65994,
+ "1951": 1.66567,
+ "1952": 1.67398,
+ "1953": 1.65554,
+ "1954": 1.6545,
+ "1955": 1.66818,
+ "1956": 1.68605,
+ "1957": 1.66488,
+ "1958": 1.65588,
+ "1959": 1.65594,
+ "1960": 1.65547,
+ "1961": 1.65398,
+ "1962": 1.67472,
+ "1963": 1.65271,
+ "1964": 1.65744,
+ "1965": 1.6546,
+ "1966": 2.02204,
+ "1967": 1.66548,
+ "1968": 1.65884,
+ "1969": 1.67389,
+ "1970": 1.67625,
+ "1971": 1.65899,
+ "1972": 1.94232,
+ "1973": 1.65267,
+ "1974": 1.65362,
+ "1975": 1.65834,
+ "1976": 1.65911,
+ "1977": 1.68703,
+ "1978": 1.65093,
+ "1979": 1.6662,
+ "1980": 1.64536,
+ "1981": 1.65504,
+ "1982": 1.65809,
+ "1983": 1.65059,
+ "1984": 1.6527,
+ "1985": 1.65844,
+ "1986": 1.66446,
+ "1987": 1.65026,
+ "1988": 1.65161,
+ "1989": 1.64463,
+ "1990": 1.64823,
+ "1991": 1.65639,
+ "1992": 1.66257,
+ "1993": 1.66599,
+ "1994": 1.65856,
+ "1995": 1.66102,
+ "1996": 1.65723,
+ "1997": 1.65469,
+ "1998": 1.65504,
+ "1999": 1.64133,
+ "2000": 1.65714
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/golden_values_dev_dgx_h100.json
new file mode 100644
index 00000000000..15d82b3a27f
--- /dev/null
+++ b/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/golden_values_dev_dgx_h100.json
@@ -0,0 +1,10037 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 10.85948,
+ "2": 10.88007,
+ "3": 10.8773,
+ "4": 10.89988,
+ "5": 10.88692,
+ "6": 10.8733,
+ "7": 10.88239,
+ "8": 10.87252,
+ "9": 10.87297,
+ "10": 10.87516,
+ "11": 10.85211,
+ "12": 10.84396,
+ "13": 10.8421,
+ "14": 10.86485,
+ "15": 10.78662,
+ "16": 10.81068,
+ "17": 10.77431,
+ "18": 10.81254,
+ "19": 10.72203,
+ "20": 10.69605,
+ "21": 10.64269,
+ "22": 10.64967,
+ "23": 10.65281,
+ "24": 10.5423,
+ "25": 10.55491,
+ "26": 10.6381,
+ "27": 10.44116,
+ "28": 10.46896,
+ "29": 10.3498,
+ "30": 10.24644,
+ "31": 10.42625,
+ "32": 10.3379,
+ "33": 10.19568,
+ "34": 10.1406,
+ "35": 10.2218,
+ "36": 10.13207,
+ "37": 10.07533,
+ "38": 10.01531,
+ "39": 10.02986,
+ "40": 10.05753,
+ "41": 9.9322,
+ "42": 9.93972,
+ "43": 9.8498,
+ "44": 9.97914,
+ "45": 9.99939,
+ "46": 9.83294,
+ "47": 9.99699,
+ "48": 9.80966,
+ "49": 9.94881,
+ "50": 9.94531,
+ "51": 9.58199,
+ "52": 9.7933,
+ "53": 9.62551,
+ "54": 9.88691,
+ "55": 9.73471,
+ "56": 9.84494,
+ "57": 9.85704,
+ "58": 9.87626,
+ "59": 9.54218,
+ "60": 9.64485,
+ "61": 9.88334,
+ "62": 9.75928,
+ "63": 9.68099,
+ "64": 9.82465,
+ "65": 9.59479,
+ "66": 9.62864,
+ "67": 9.74006,
+ "68": 9.60208,
+ "69": 9.29221,
+ "70": 9.42136,
+ "71": 9.78758,
+ "72": 9.71247,
+ "73": 9.61814,
+ "74": 9.4478,
+ "75": 9.23896,
+ "76": 9.50832,
+ "77": 9.57954,
+ "78": 9.56051,
+ "79": 9.30813,
+ "80": 9.35769,
+ "81": 9.45808,
+ "82": 9.55364,
+ "83": 9.53411,
+ "84": 9.35437,
+ "85": 9.39928,
+ "86": 9.6528,
+ "87": 9.23448,
+ "88": 9.48756,
+ "89": 9.22211,
+ "90": 9.41071,
+ "91": 9.38757,
+ "92": 9.3768,
+ "93": 9.36025,
+ "94": 9.51507,
+ "95": 9.42128,
+ "96": 9.33612,
+ "97": 9.20399,
+ "98": 9.49541,
+ "99": 9.29292,
+ "100": 9.35912,
+ "101": 9.2476,
+ "102": 9.24679,
+ "103": 9.07742,
+ "104": 9.16671,
+ "105": 9.37863,
+ "106": 9.1496,
+ "107": 9.17568,
+ "108": 9.31603,
+ "109": 9.29106,
+ "110": 9.36424,
+ "111": 9.17991,
+ "112": 9.23474,
+ "113": 9.35293,
+ "114": 9.35265,
+ "115": 9.32671,
+ "116": 9.00219,
+ "117": 9.06474,
+ "118": 9.06638,
+ "119": 9.2242,
+ "120": 9.08483,
+ "121": 9.19674,
+ "122": 9.14162,
+ "123": 9.25928,
+ "124": 9.45512,
+ "125": 9.21516,
+ "126": 9.06415,
+ "127": 9.01811,
+ "128": 9.22123,
+ "129": 8.98189,
+ "130": 9.13969,
+ "131": 9.15859,
+ "132": 9.03558,
+ "133": 8.85979,
+ "134": 9.18543,
+ "135": 8.88999,
+ "136": 9.16798,
+ "137": 9.15781,
+ "138": 9.23512,
+ "139": 9.09194,
+ "140": 8.87222,
+ "141": 9.29901,
+ "142": 9.19959,
+ "143": 9.11685,
+ "144": 9.24318,
+ "145": 9.10455,
+ "146": 8.98719,
+ "147": 8.9862,
+ "148": 9.13259,
+ "149": 9.06337,
+ "150": 9.01509,
+ "151": 8.92795,
+ "152": 8.87388,
+ "153": 9.06337,
+ "154": 9.17912,
+ "155": 9.13392,
+ "156": 9.0489,
+ "157": 9.15064,
+ "158": 9.04948,
+ "159": 9.03258,
+ "160": 8.88984,
+ "161": 9.04543,
+ "162": 8.89571,
+ "163": 8.84266,
+ "164": 8.97529,
+ "165": 8.93125,
+ "166": 8.65954,
+ "167": 8.8323,
+ "168": 8.81937,
+ "169": 8.65641,
+ "170": 9.04618,
+ "171": 8.72266,
+ "172": 8.82145,
+ "173": 8.91154,
+ "174": 8.84741,
+ "175": 8.70609,
+ "176": 8.75429,
+ "177": 8.76244,
+ "178": 8.72001,
+ "179": 8.64031,
+ "180": 8.74045,
+ "181": 8.69389,
+ "182": 8.72185,
+ "183": 9.08355,
+ "184": 8.60874,
+ "185": 8.88344,
+ "186": 8.74189,
+ "187": 8.56935,
+ "188": 8.6797,
+ "189": 8.8646,
+ "190": 8.53524,
+ "191": 8.66619,
+ "192": 8.61256,
+ "193": 8.57463,
+ "194": 8.75194,
+ "195": 8.59285,
+ "196": 8.77396,
+ "197": 8.74227,
+ "198": 8.62719,
+ "199": 8.77451,
+ "200": 8.73793,
+ "201": 8.66978,
+ "202": 8.54592,
+ "203": 8.54182,
+ "204": 8.71304,
+ "205": 8.22281,
+ "206": 8.86043,
+ "207": 8.68158,
+ "208": 8.709,
+ "209": 8.75317,
+ "210": 8.57805,
+ "211": 8.84267,
+ "212": 8.49122,
+ "213": 8.57336,
+ "214": 8.51209,
+ "215": 8.56461,
+ "216": 8.50864,
+ "217": 8.53189,
+ "218": 8.53011,
+ "219": 8.64378,
+ "220": 8.54749,
+ "221": 8.40004,
+ "222": 8.50526,
+ "223": 8.43786,
+ "224": 8.53023,
+ "225": 8.57095,
+ "226": 8.43947,
+ "227": 8.67919,
+ "228": 8.38477,
+ "229": 8.45055,
+ "230": 8.49718,
+ "231": 8.49834,
+ "232": 8.49783,
+ "233": 8.49531,
+ "234": 8.63793,
+ "235": 8.55868,
+ "236": 8.3946,
+ "237": 8.4882,
+ "238": 8.30509,
+ "239": 8.56188,
+ "240": 8.66938,
+ "241": 8.44122,
+ "242": 8.47197,
+ "243": 8.51761,
+ "244": 8.36804,
+ "245": 8.59255,
+ "246": 8.59476,
+ "247": 8.44,
+ "248": 8.51259,
+ "249": 8.52021,
+ "250": 8.4217,
+ "251": 8.37733,
+ "252": 8.54374,
+ "253": 8.31443,
+ "254": 8.35081,
+ "255": 8.29,
+ "256": 8.20238,
+ "257": 8.39382,
+ "258": 8.45365,
+ "259": 8.23685,
+ "260": 8.2435,
+ "261": 8.236,
+ "262": 8.34915,
+ "263": 8.3066,
+ "264": 8.18814,
+ "265": 8.33493,
+ "266": 8.23353,
+ "267": 7.89907,
+ "268": 8.3805,
+ "269": 8.40456,
+ "270": 8.26253,
+ "271": 8.27887,
+ "272": 8.32102,
+ "273": 8.13749,
+ "274": 8.09662,
+ "275": 8.01379,
+ "276": 7.92608,
+ "277": 8.24029,
+ "278": 8.05019,
+ "279": 7.96697,
+ "280": 7.75642,
+ "281": 8.10712,
+ "282": 8.15055,
+ "283": 8.15615,
+ "284": 8.10358,
+ "285": 8.07243,
+ "286": 7.90478,
+ "287": 7.99634,
+ "288": 8.24864,
+ "289": 8.17587,
+ "290": 8.13085,
+ "291": 8.25773,
+ "292": 8.08139,
+ "293": 8.12056,
+ "294": 7.98179,
+ "295": 7.97112,
+ "296": 8.2413,
+ "297": 7.79644,
+ "298": 8.04861,
+ "299": 7.94264,
+ "300": 7.85758,
+ "301": 8.01661,
+ "302": 7.95104,
+ "303": 7.9962,
+ "304": 7.96409,
+ "305": 8.00293,
+ "306": 7.98315,
+ "307": 7.99383,
+ "308": 8.00508,
+ "309": 8.01372,
+ "310": 7.9783,
+ "311": 7.93222,
+ "312": 7.89451,
+ "313": 7.84061,
+ "314": 7.8301,
+ "315": 7.83374,
+ "316": 7.75124,
+ "317": 7.934,
+ "318": 7.98859,
+ "319": 7.83359,
+ "320": 7.57903,
+ "321": 7.75419,
+ "322": 7.83778,
+ "323": 7.77686,
+ "324": 7.91622,
+ "325": 7.80544,
+ "326": 7.65628,
+ "327": 7.86983,
+ "328": 7.79366,
+ "329": 7.89122,
+ "330": 7.75826,
+ "331": 7.52864,
+ "332": 7.81924,
+ "333": 7.84347,
+ "334": 7.68346,
+ "335": 7.69945,
+ "336": 7.919,
+ "337": 7.65337,
+ "338": 7.90247,
+ "339": 7.73039,
+ "340": 7.76025,
+ "341": 7.70868,
+ "342": 7.82807,
+ "343": 7.6179,
+ "344": 7.58796,
+ "345": 7.61568,
+ "346": 7.46383,
+ "347": 7.56081,
+ "348": 7.68697,
+ "349": 7.58307,
+ "350": 7.65728,
+ "351": 7.75374,
+ "352": 7.71081,
+ "353": 7.5044,
+ "354": 7.74899,
+ "355": 7.76969,
+ "356": 7.78245,
+ "357": 7.81824,
+ "358": 7.59994,
+ "359": 7.55157,
+ "360": 7.63175,
+ "361": 7.55267,
+ "362": 7.76832,
+ "363": 7.59445,
+ "364": 7.57895,
+ "365": 7.62804,
+ "366": 7.31065,
+ "367": 7.55863,
+ "368": 7.44523,
+ "369": 7.35464,
+ "370": 7.46939,
+ "371": 7.46571,
+ "372": 7.65443,
+ "373": 7.52954,
+ "374": 7.44811,
+ "375": 7.53597,
+ "376": 7.35231,
+ "377": 7.24281,
+ "378": 7.54269,
+ "379": 7.4988,
+ "380": 7.38821,
+ "381": 7.47528,
+ "382": 7.29913,
+ "383": 7.28453,
+ "384": 7.41211,
+ "385": 7.39774,
+ "386": 7.23601,
+ "387": 7.42494,
+ "388": 7.28443,
+ "389": 7.44366,
+ "390": 7.24535,
+ "391": 7.64755,
+ "392": 7.34206,
+ "393": 7.42423,
+ "394": 7.48195,
+ "395": 7.44424,
+ "396": 7.29198,
+ "397": 7.23334,
+ "398": 7.42461,
+ "399": 7.16128,
+ "400": 7.30116,
+ "401": 7.35799,
+ "402": 7.39784,
+ "403": 7.28769,
+ "404": 7.308,
+ "405": 7.27119,
+ "406": 7.22461,
+ "407": 7.36653,
+ "408": 7.18842,
+ "409": 7.17305,
+ "410": 7.31976,
+ "411": 7.22198,
+ "412": 7.20564,
+ "413": 7.24019,
+ "414": 6.91734,
+ "415": 7.33407,
+ "416": 7.43119,
+ "417": 7.02983,
+ "418": 7.28206,
+ "419": 7.04277,
+ "420": 7.41855,
+ "421": 7.18447,
+ "422": 7.23999,
+ "423": 7.09792,
+ "424": 7.2458,
+ "425": 7.32193,
+ "426": 7.29365,
+ "427": 7.13596,
+ "428": 7.09614,
+ "429": 6.87987,
+ "430": 7.20713,
+ "431": 7.00663,
+ "432": 7.23775,
+ "433": 6.98012,
+ "434": 6.96144,
+ "435": 7.02229,
+ "436": 7.01495,
+ "437": 6.99234,
+ "438": 7.00526,
+ "439": 6.94251,
+ "440": 7.0638,
+ "441": 7.04978,
+ "442": 7.10221,
+ "443": 7.09432,
+ "444": 6.712,
+ "445": 6.99857,
+ "446": 7.14646,
+ "447": 7.12769,
+ "448": 6.98658,
+ "449": 7.05126,
+ "450": 7.01798,
+ "451": 6.83292,
+ "452": 6.91611,
+ "453": 7.02097,
+ "454": 6.97065,
+ "455": 7.03192,
+ "456": 6.99487,
+ "457": 6.97329,
+ "458": 6.90707,
+ "459": 6.69522,
+ "460": 7.06795,
+ "461": 7.09894,
+ "462": 6.87148,
+ "463": 7.05568,
+ "464": 6.64953,
+ "465": 7.02912,
+ "466": 7.00636,
+ "467": 6.99959,
+ "468": 6.95236,
+ "469": 6.82955,
+ "470": 7.04657,
+ "471": 6.88352,
+ "472": 6.96142,
+ "473": 6.82404,
+ "474": 6.97251,
+ "475": 7.16932,
+ "476": 6.7637,
+ "477": 6.89782,
+ "478": 6.91155,
+ "479": 6.70395,
+ "480": 7.03041,
+ "481": 6.99764,
+ "482": 6.73613,
+ "483": 6.78473,
+ "484": 6.75431,
+ "485": 6.93166,
+ "486": 7.06831,
+ "487": 6.63671,
+ "488": 6.88745,
+ "489": 6.77117,
+ "490": 6.82677,
+ "491": 6.71133,
+ "492": 6.6988,
+ "493": 6.77167,
+ "494": 6.67634,
+ "495": 6.63755,
+ "496": 6.58978,
+ "497": 6.84604,
+ "498": 6.65297,
+ "499": 6.85957,
+ "500": 6.6584,
+ "501": 6.73572,
+ "502": 6.84555,
+ "503": 6.71144,
+ "504": 6.62126,
+ "505": 6.62322,
+ "506": 6.75317,
+ "507": 6.869,
+ "508": 6.86276,
+ "509": 6.65564,
+ "510": 6.82875,
+ "511": 6.74172,
+ "512": 6.74141,
+ "513": 6.66084,
+ "514": 6.71423,
+ "515": 6.4517,
+ "516": 6.74532,
+ "517": 6.71184,
+ "518": 6.53969,
+ "519": 6.63702,
+ "520": 6.86146,
+ "521": 6.667,
+ "522": 6.71043,
+ "523": 6.74849,
+ "524": 6.7349,
+ "525": 6.67827,
+ "526": 6.41606,
+ "527": 6.80077,
+ "528": 6.66142,
+ "529": 6.6337,
+ "530": 6.62724,
+ "531": 6.6452,
+ "532": 6.6347,
+ "533": 6.76564,
+ "534": 6.61371,
+ "535": 6.74957,
+ "536": 6.62739,
+ "537": 6.64034,
+ "538": 6.5321,
+ "539": 6.55769,
+ "540": 6.58886,
+ "541": 6.45401,
+ "542": 6.67286,
+ "543": 6.68196,
+ "544": 6.67848,
+ "545": 6.81469,
+ "546": 6.63553,
+ "547": 6.4191,
+ "548": 6.72493,
+ "549": 6.70086,
+ "550": 6.53083,
+ "551": 6.74883,
+ "552": 6.64136,
+ "553": 6.48594,
+ "554": 6.63541,
+ "555": 6.46442,
+ "556": 6.61944,
+ "557": 6.63664,
+ "558": 6.38891,
+ "559": 6.37489,
+ "560": 6.58434,
+ "561": 6.73468,
+ "562": 6.63703,
+ "563": 6.74502,
+ "564": 6.35319,
+ "565": 6.51569,
+ "566": 6.70305,
+ "567": 6.56987,
+ "568": 6.51155,
+ "569": 6.45611,
+ "570": 6.36763,
+ "571": 6.63653,
+ "572": 6.31388,
+ "573": 6.58623,
+ "574": 6.47594,
+ "575": 6.64959,
+ "576": 6.51693,
+ "577": 6.53112,
+ "578": 6.48481,
+ "579": 6.46691,
+ "580": 6.56709,
+ "581": 6.6083,
+ "582": 6.48347,
+ "583": 6.5157,
+ "584": 6.52848,
+ "585": 6.42708,
+ "586": 6.41716,
+ "587": 6.46035,
+ "588": 6.56815,
+ "589": 6.62487,
+ "590": 6.29113,
+ "591": 6.67478,
+ "592": 6.26886,
+ "593": 6.47655,
+ "594": 6.38568,
+ "595": 6.36293,
+ "596": 6.25932,
+ "597": 6.1884,
+ "598": 6.45541,
+ "599": 6.39807,
+ "600": 6.45423,
+ "601": 6.25953,
+ "602": 6.53096,
+ "603": 6.52057,
+ "604": 6.38646,
+ "605": 6.49711,
+ "606": 6.31261,
+ "607": 6.53384,
+ "608": 6.67197,
+ "609": 6.17481,
+ "610": 6.56956,
+ "611": 6.40004,
+ "612": 6.57667,
+ "613": 6.42307,
+ "614": 6.20375,
+ "615": 6.398,
+ "616": 6.35676,
+ "617": 6.37693,
+ "618": 6.44763,
+ "619": 6.13898,
+ "620": 6.40871,
+ "621": 6.45998,
+ "622": 6.39819,
+ "623": 6.57964,
+ "624": 6.35856,
+ "625": 6.28211,
+ "626": 6.30409,
+ "627": 6.44312,
+ "628": 6.25372,
+ "629": 6.58443,
+ "630": 6.36294,
+ "631": 6.34614,
+ "632": 6.30884,
+ "633": 6.25522,
+ "634": 6.30719,
+ "635": 6.5442,
+ "636": 6.24636,
+ "637": 6.63554,
+ "638": 6.01882,
+ "639": 6.2786,
+ "640": 6.29373,
+ "641": 6.20967,
+ "642": 6.28371,
+ "643": 6.45848,
+ "644": 6.25743,
+ "645": 6.24932,
+ "646": 6.4057,
+ "647": 6.33671,
+ "648": 6.35427,
+ "649": 6.34664,
+ "650": 6.48227,
+ "651": 6.33122,
+ "652": 6.25394,
+ "653": 6.38324,
+ "654": 6.4497,
+ "655": 6.52609,
+ "656": 6.32666,
+ "657": 6.43099,
+ "658": 6.24269,
+ "659": 6.15398,
+ "660": 6.39368,
+ "661": 6.1704,
+ "662": 6.27352,
+ "663": 6.37267,
+ "664": 6.33404,
+ "665": 6.40482,
+ "666": 6.16406,
+ "667": 6.19723,
+ "668": 6.23664,
+ "669": 6.21684,
+ "670": 6.24597,
+ "671": 6.24401,
+ "672": 6.48974,
+ "673": 6.34063,
+ "674": 6.29538,
+ "675": 6.38388,
+ "676": 6.39052,
+ "677": 6.30477,
+ "678": 6.27807,
+ "679": 6.23818,
+ "680": 6.2938,
+ "681": 6.20549,
+ "682": 6.08606,
+ "683": 6.27423,
+ "684": 6.32891,
+ "685": 6.30212,
+ "686": 6.15327,
+ "687": 6.28535,
+ "688": 6.20749,
+ "689": 6.62071,
+ "690": 6.17829,
+ "691": 6.18212,
+ "692": 6.27367,
+ "693": 6.14378,
+ "694": 6.23491,
+ "695": 6.32549,
+ "696": 6.11856,
+ "697": 6.15439,
+ "698": 6.23134,
+ "699": 6.46047,
+ "700": 6.04372,
+ "701": 6.06264,
+ "702": 6.2496,
+ "703": 6.18391,
+ "704": 6.21563,
+ "705": 6.12795,
+ "706": 6.07346,
+ "707": 6.25053,
+ "708": 6.31298,
+ "709": 6.00679,
+ "710": 6.15874,
+ "711": 6.25221,
+ "712": 6.17533,
+ "713": 5.89135,
+ "714": 6.10119,
+ "715": 6.10943,
+ "716": 6.4069,
+ "717": 6.18502,
+ "718": 6.23043,
+ "719": 6.26711,
+ "720": 6.25776,
+ "721": 6.25487,
+ "722": 6.22785,
+ "723": 6.07334,
+ "724": 6.22203,
+ "725": 6.03301,
+ "726": 6.29284,
+ "727": 6.00984,
+ "728": 6.03804,
+ "729": 6.08458,
+ "730": 6.17939,
+ "731": 6.09287,
+ "732": 6.07938,
+ "733": 6.1137,
+ "734": 6.37581,
+ "735": 6.2692,
+ "736": 6.1752,
+ "737": 6.35947,
+ "738": 6.12622,
+ "739": 6.13818,
+ "740": 5.87091,
+ "741": 5.99972,
+ "742": 5.97578,
+ "743": 6.1672,
+ "744": 6.02207,
+ "745": 6.14538,
+ "746": 6.02551,
+ "747": 6.08679,
+ "748": 6.22666,
+ "749": 5.93206,
+ "750": 6.16074,
+ "751": 5.95024,
+ "752": 6.0107,
+ "753": 6.01922,
+ "754": 6.28009,
+ "755": 6.12445,
+ "756": 6.24565,
+ "757": 6.01105,
+ "758": 6.1948,
+ "759": 6.21984,
+ "760": 6.01491,
+ "761": 6.1868,
+ "762": 6.21833,
+ "763": 6.02947,
+ "764": 5.95765,
+ "765": 5.92695,
+ "766": 5.96361,
+ "767": 5.80688,
+ "768": 6.18123,
+ "769": 6.26863,
+ "770": 6.28842,
+ "771": 5.78115,
+ "772": 6.02609,
+ "773": 6.17947,
+ "774": 5.87846,
+ "775": 6.02381,
+ "776": 6.12187,
+ "777": 5.87787,
+ "778": 6.05223,
+ "779": 5.86809,
+ "780": 6.13369,
+ "781": 5.85112,
+ "782": 6.04299,
+ "783": 5.95288,
+ "784": 5.91364,
+ "785": 6.09391,
+ "786": 6.10635,
+ "787": 5.65794,
+ "788": 5.99592,
+ "789": 6.21379,
+ "790": 6.2644,
+ "791": 5.78636,
+ "792": 5.99412,
+ "793": 6.17893,
+ "794": 6.02201,
+ "795": 5.99958,
+ "796": 6.16262,
+ "797": 6.0481,
+ "798": 6.05546,
+ "799": 6.11168,
+ "800": 6.01937,
+ "801": 6.14472,
+ "802": 5.98356,
+ "803": 6.15215,
+ "804": 6.0105,
+ "805": 5.83322,
+ "806": 6.08769,
+ "807": 6.04861,
+ "808": 5.92776,
+ "809": 5.77724,
+ "810": 6.02092,
+ "811": 5.92972,
+ "812": 5.91224,
+ "813": 5.96855,
+ "814": 6.03504,
+ "815": 5.81362,
+ "816": 6.11608,
+ "817": 5.93717,
+ "818": 6.06438,
+ "819": 6.011,
+ "820": 5.7306,
+ "821": 5.95027,
+ "822": 6.20433,
+ "823": 5.83304,
+ "824": 5.98482,
+ "825": 6.18765,
+ "826": 6.20235,
+ "827": 6.05879,
+ "828": 6.07114,
+ "829": 5.89376,
+ "830": 5.94059,
+ "831": 5.89761,
+ "832": 5.97327,
+ "833": 6.06553,
+ "834": 5.99826,
+ "835": 6.00743,
+ "836": 5.79586,
+ "837": 6.11313,
+ "838": 5.8706,
+ "839": 5.83816,
+ "840": 6.18766,
+ "841": 5.78507,
+ "842": 5.89233,
+ "843": 5.95289,
+ "844": 6.01245,
+ "845": 6.09241,
+ "846": 5.69208,
+ "847": 5.75889,
+ "848": 5.97131,
+ "849": 6.09692,
+ "850": 5.85205,
+ "851": 6.0206,
+ "852": 5.75603,
+ "853": 5.99468,
+ "854": 6.02229,
+ "855": 5.81834,
+ "856": 5.99972,
+ "857": 6.00412,
+ "858": 6.05561,
+ "859": 5.95477,
+ "860": 6.09992,
+ "861": 6.07144,
+ "862": 6.0053,
+ "863": 5.84074,
+ "864": 5.84476,
+ "865": 5.94211,
+ "866": 5.89614,
+ "867": 5.88027,
+ "868": 6.06438,
+ "869": 6.08525,
+ "870": 5.97278,
+ "871": 6.04488,
+ "872": 5.89678,
+ "873": 5.84437,
+ "874": 6.02841,
+ "875": 5.91615,
+ "876": 5.96972,
+ "877": 5.93292,
+ "878": 6.1001,
+ "879": 5.769,
+ "880": 6.01517,
+ "881": 5.99862,
+ "882": 5.91077,
+ "883": 5.67901,
+ "884": 5.96638,
+ "885": 5.7468,
+ "886": 5.99351,
+ "887": 5.91236,
+ "888": 5.84192,
+ "889": 6.01314,
+ "890": 6.02565,
+ "891": 5.95397,
+ "892": 5.71109,
+ "893": 6.09289,
+ "894": 5.73098,
+ "895": 5.84206,
+ "896": 5.84177,
+ "897": 5.85634,
+ "898": 5.92388,
+ "899": 5.9347,
+ "900": 5.89844,
+ "901": 5.94993,
+ "902": 5.83299,
+ "903": 6.05413,
+ "904": 5.93254,
+ "905": 5.90599,
+ "906": 5.61902,
+ "907": 5.91279,
+ "908": 5.73751,
+ "909": 5.9907,
+ "910": 5.86442,
+ "911": 5.70387,
+ "912": 5.7073,
+ "913": 5.7659,
+ "914": 5.83727,
+ "915": 5.80028,
+ "916": 5.88933,
+ "917": 5.86912,
+ "918": 5.82213,
+ "919": 5.81683,
+ "920": 5.89347,
+ "921": 5.84348,
+ "922": 5.62426,
+ "923": 6.03778,
+ "924": 5.60598,
+ "925": 5.62449,
+ "926": 5.8623,
+ "927": 5.96235,
+ "928": 5.84195,
+ "929": 5.82945,
+ "930": 5.96069,
+ "931": 5.76943,
+ "932": 5.59694,
+ "933": 5.63728,
+ "934": 5.81088,
+ "935": 5.63741,
+ "936": 5.83934,
+ "937": 5.96779,
+ "938": 5.59103,
+ "939": 5.78864,
+ "940": 5.96535,
+ "941": 5.72353,
+ "942": 5.83395,
+ "943": 5.8659,
+ "944": 5.95482,
+ "945": 5.70238,
+ "946": 5.55873,
+ "947": 5.74985,
+ "948": 5.7959,
+ "949": 5.82605,
+ "950": 5.8477,
+ "951": 5.72221,
+ "952": 5.69514,
+ "953": 5.68157,
+ "954": 5.72491,
+ "955": 5.53105,
+ "956": 5.62301,
+ "957": 5.84233,
+ "958": 5.7962,
+ "959": 5.57377,
+ "960": 5.80295,
+ "961": 5.8298,
+ "962": 5.7699,
+ "963": 5.76784,
+ "964": 5.70943,
+ "965": 5.64174,
+ "966": 5.59888,
+ "967": 5.72524,
+ "968": 5.74136,
+ "969": 5.82293,
+ "970": 5.64566,
+ "971": 5.70805,
+ "972": 5.85288,
+ "973": 5.66898,
+ "974": 5.71996,
+ "975": 5.86373,
+ "976": 5.70737,
+ "977": 5.76974,
+ "978": 5.68735,
+ "979": 5.58624,
+ "980": 5.76018,
+ "981": 5.89674,
+ "982": 5.47111,
+ "983": 5.61931,
+ "984": 5.54594,
+ "985": 5.59021,
+ "986": 5.64167,
+ "987": 5.5693,
+ "988": 5.71039,
+ "989": 5.69431,
+ "990": 5.6232,
+ "991": 5.85084,
+ "992": 5.78045,
+ "993": 5.87116,
+ "994": 5.69843,
+ "995": 5.7334,
+ "996": 5.73864,
+ "997": 5.81546,
+ "998": 5.83741,
+ "999": 5.83537,
+ "1000": 5.6845,
+ "1001": 5.86795,
+ "1002": 5.76117,
+ "1003": 5.64372,
+ "1004": 5.79976,
+ "1005": 5.538,
+ "1006": 5.32722,
+ "1007": 5.76884,
+ "1008": 5.79227,
+ "1009": 5.65325,
+ "1010": 5.78045,
+ "1011": 5.89483,
+ "1012": 5.62309,
+ "1013": 5.6158,
+ "1014": 5.68137,
+ "1015": 5.55973,
+ "1016": 5.87255,
+ "1017": 5.83329,
+ "1018": 5.6201,
+ "1019": 5.73433,
+ "1020": 5.61752,
+ "1021": 5.8498,
+ "1022": 5.49981,
+ "1023": 5.65186,
+ "1024": 5.74475,
+ "1025": 5.56944,
+ "1026": 5.41537,
+ "1027": 5.60586,
+ "1028": 5.69383,
+ "1029": 5.68685,
+ "1030": 5.68694,
+ "1031": 5.40628,
+ "1032": 5.78759,
+ "1033": 5.57993,
+ "1034": 5.62106,
+ "1035": 5.71117,
+ "1036": 5.62822,
+ "1037": 5.36807,
+ "1038": 5.66472,
+ "1039": 5.64357,
+ "1040": 5.57082,
+ "1041": 5.59646,
+ "1042": 5.81311,
+ "1043": 5.56443,
+ "1044": 5.46776,
+ "1045": 5.96456,
+ "1046": 5.48653,
+ "1047": 5.38978,
+ "1048": 5.50119,
+ "1049": 5.67228,
+ "1050": 5.7005,
+ "1051": 5.57872,
+ "1052": 5.68194,
+ "1053": 5.62766,
+ "1054": 5.45944,
+ "1055": 5.6032,
+ "1056": 5.67718,
+ "1057": 5.76121,
+ "1058": 5.56841,
+ "1059": 5.75267,
+ "1060": 5.82295,
+ "1061": 5.47954,
+ "1062": 5.64843,
+ "1063": 5.50349,
+ "1064": 5.59246,
+ "1065": 5.55633,
+ "1066": 5.7461,
+ "1067": 5.67584,
+ "1068": 5.44215,
+ "1069": 5.60824,
+ "1070": 5.81201,
+ "1071": 5.51383,
+ "1072": 5.62108,
+ "1073": 5.62271,
+ "1074": 5.52595,
+ "1075": 5.70839,
+ "1076": 5.59571,
+ "1077": 5.71283,
+ "1078": 5.56369,
+ "1079": 5.61982,
+ "1080": 5.64179,
+ "1081": 5.62377,
+ "1082": 5.49685,
+ "1083": 5.64328,
+ "1084": 5.55474,
+ "1085": 5.40877,
+ "1086": 5.62154,
+ "1087": 5.44541,
+ "1088": 5.51461,
+ "1089": 5.7727,
+ "1090": 5.53261,
+ "1091": 5.51605,
+ "1092": 5.41104,
+ "1093": 5.70327,
+ "1094": 5.57795,
+ "1095": 5.57638,
+ "1096": 5.61726,
+ "1097": 5.64734,
+ "1098": 5.65581,
+ "1099": 5.51973,
+ "1100": 5.64222,
+ "1101": 5.67227,
+ "1102": 5.54094,
+ "1103": 5.54959,
+ "1104": 5.53788,
+ "1105": 5.54659,
+ "1106": 5.68272,
+ "1107": 5.67967,
+ "1108": 5.78477,
+ "1109": 5.53712,
+ "1110": 5.66245,
+ "1111": 5.58675,
+ "1112": 5.58155,
+ "1113": 5.62417,
+ "1114": 5.61498,
+ "1115": 5.60148,
+ "1116": 5.66148,
+ "1117": 5.64827,
+ "1118": 5.65424,
+ "1119": 5.71038,
+ "1120": 5.63068,
+ "1121": 5.37745,
+ "1122": 5.23109,
+ "1123": 5.47696,
+ "1124": 5.65203,
+ "1125": 5.68155,
+ "1126": 5.68213,
+ "1127": 5.56996,
+ "1128": 5.62014,
+ "1129": 5.29781,
+ "1130": 5.54305,
+ "1131": 5.63518,
+ "1132": 5.72438,
+ "1133": 5.52164,
+ "1134": 5.55992,
+ "1135": 5.52418,
+ "1136": 5.42712,
+ "1137": 5.46013,
+ "1138": 5.5693,
+ "1139": 5.41494,
+ "1140": 5.26257,
+ "1141": 5.58415,
+ "1142": 5.63855,
+ "1143": 5.38368,
+ "1144": 5.38069,
+ "1145": 5.36318,
+ "1146": 5.63256,
+ "1147": 5.48872,
+ "1148": 5.50555,
+ "1149": 5.5263,
+ "1150": 5.40115,
+ "1151": 5.55821,
+ "1152": 5.41854,
+ "1153": 5.44995,
+ "1154": 5.50035,
+ "1155": 5.43489,
+ "1156": 5.35184,
+ "1157": 5.6619,
+ "1158": 5.39835,
+ "1159": 5.33812,
+ "1160": 5.79646,
+ "1161": 5.54232,
+ "1162": 5.45982,
+ "1163": 5.52607,
+ "1164": 5.38584,
+ "1165": 5.52786,
+ "1166": 5.48869,
+ "1167": 5.36129,
+ "1168": 5.49368,
+ "1169": 5.39585,
+ "1170": 5.58536,
+ "1171": 5.48449,
+ "1172": 5.63603,
+ "1173": 5.6223,
+ "1174": 5.50777,
+ "1175": 5.34588,
+ "1176": 5.38809,
+ "1177": 5.55347,
+ "1178": 5.46915,
+ "1179": 5.4944,
+ "1180": 5.45901,
+ "1181": 5.56419,
+ "1182": 5.5995,
+ "1183": 5.77414,
+ "1184": 5.55102,
+ "1185": 5.29208,
+ "1186": 5.60837,
+ "1187": 5.55411,
+ "1188": 5.51626,
+ "1189": 5.39218,
+ "1190": 5.40626,
+ "1191": 5.39237,
+ "1192": 5.50217,
+ "1193": 5.46288,
+ "1194": 5.45664,
+ "1195": 5.32823,
+ "1196": 5.52355,
+ "1197": 5.48014,
+ "1198": 5.52878,
+ "1199": 5.38696,
+ "1200": 5.33013,
+ "1201": 5.49038,
+ "1202": 5.43576,
+ "1203": 5.49496,
+ "1204": 5.40633,
+ "1205": 5.48536,
+ "1206": 5.33463,
+ "1207": 5.58352,
+ "1208": 5.43053,
+ "1209": 5.29424,
+ "1210": 5.50516,
+ "1211": 5.51313,
+ "1212": 5.59727,
+ "1213": 5.41973,
+ "1214": 5.50663,
+ "1215": 5.23771,
+ "1216": 5.40925,
+ "1217": 5.38364,
+ "1218": 5.45226,
+ "1219": 5.48163,
+ "1220": 5.38686,
+ "1221": 5.44951,
+ "1222": 5.3116,
+ "1223": 5.47832,
+ "1224": 5.42058,
+ "1225": 5.43618,
+ "1226": 5.32341,
+ "1227": 5.47695,
+ "1228": 5.72214,
+ "1229": 5.324,
+ "1230": 5.40406,
+ "1231": 5.06939,
+ "1232": 5.78868,
+ "1233": 5.2883,
+ "1234": 5.24622,
+ "1235": 5.36926,
+ "1236": 5.48432,
+ "1237": 5.21097,
+ "1238": 5.41741,
+ "1239": 5.40828,
+ "1240": 5.46843,
+ "1241": 5.57658,
+ "1242": 5.45558,
+ "1243": 5.43152,
+ "1244": 5.51994,
+ "1245": 5.19112,
+ "1246": 5.71919,
+ "1247": 5.43135,
+ "1248": 5.29921,
+ "1249": 5.39928,
+ "1250": 5.33958,
+ "1251": 5.41917,
+ "1252": 5.57109,
+ "1253": 5.48956,
+ "1254": 5.30529,
+ "1255": 5.5124,
+ "1256": 5.60777,
+ "1257": 5.42366,
+ "1258": 5.56291,
+ "1259": 5.48045,
+ "1260": 5.50796,
+ "1261": 5.63874,
+ "1262": 5.39345,
+ "1263": 5.32966,
+ "1264": 5.50463,
+ "1265": 5.30185,
+ "1266": 5.23892,
+ "1267": 5.3711,
+ "1268": 5.38989,
+ "1269": 5.15229,
+ "1270": 5.40221,
+ "1271": 5.2735,
+ "1272": 5.52015,
+ "1273": 5.29718,
+ "1274": 5.34594,
+ "1275": 5.38424,
+ "1276": 5.39608,
+ "1277": 5.46194,
+ "1278": 5.3475,
+ "1279": 5.43857,
+ "1280": 5.46121,
+ "1281": 5.40317,
+ "1282": 5.37932,
+ "1283": 5.42376,
+ "1284": 5.34774,
+ "1285": 5.50219,
+ "1286": 5.33438,
+ "1287": 5.58555,
+ "1288": 5.26424,
+ "1289": 5.42448,
+ "1290": 5.5007,
+ "1291": 5.50357,
+ "1292": 5.44618,
+ "1293": 5.41732,
+ "1294": 5.49591,
+ "1295": 5.34456,
+ "1296": 5.18825,
+ "1297": 5.17076,
+ "1298": 5.11821,
+ "1299": 5.30436,
+ "1300": 5.20918,
+ "1301": 5.30218,
+ "1302": 5.27523,
+ "1303": 5.35417,
+ "1304": 5.43049,
+ "1305": 5.36674,
+ "1306": 5.24817,
+ "1307": 5.18802,
+ "1308": 5.26777,
+ "1309": 5.4033,
+ "1310": 5.26147,
+ "1311": 5.38115,
+ "1312": 5.35285,
+ "1313": 5.29614,
+ "1314": 5.29055,
+ "1315": 5.41831,
+ "1316": 5.25781,
+ "1317": 5.27772,
+ "1318": 5.21802,
+ "1319": 5.34306,
+ "1320": 5.41777,
+ "1321": 5.44818,
+ "1322": 5.46316,
+ "1323": 5.36943,
+ "1324": 5.25559,
+ "1325": 5.40541,
+ "1326": 5.53615,
+ "1327": 5.39107,
+ "1328": 5.21608,
+ "1329": 5.41699,
+ "1330": 5.39742,
+ "1331": 5.31396,
+ "1332": 5.31337,
+ "1333": 5.37117,
+ "1334": 5.44593,
+ "1335": 5.36997,
+ "1336": 5.4385,
+ "1337": 5.47692,
+ "1338": 5.30176,
+ "1339": 5.13882,
+ "1340": 5.41377,
+ "1341": 5.34396,
+ "1342": 5.35991,
+ "1343": 5.47805,
+ "1344": 5.37657,
+ "1345": 5.34166,
+ "1346": 5.08003,
+ "1347": 5.38727,
+ "1348": 5.48991,
+ "1349": 5.40963,
+ "1350": 5.02917,
+ "1351": 5.31457,
+ "1352": 5.1566,
+ "1353": 5.34359,
+ "1354": 5.35736,
+ "1355": 5.10956,
+ "1356": 5.25718,
+ "1357": 5.29131,
+ "1358": 5.15619,
+ "1359": 5.10988,
+ "1360": 5.1739,
+ "1361": 5.30553,
+ "1362": 5.06361,
+ "1363": 5.29217,
+ "1364": 5.40071,
+ "1365": 5.01789,
+ "1366": 5.11439,
+ "1367": 5.32916,
+ "1368": 5.18171,
+ "1369": 5.22722,
+ "1370": 5.19463,
+ "1371": 5.28089,
+ "1372": 5.26381,
+ "1373": 5.28249,
+ "1374": 5.28204,
+ "1375": 5.45976,
+ "1376": 5.27068,
+ "1377": 5.26443,
+ "1378": 5.30961,
+ "1379": 5.22668,
+ "1380": 5.26083,
+ "1381": 5.477,
+ "1382": 5.08889,
+ "1383": 5.37541,
+ "1384": 5.35901,
+ "1385": 5.38849,
+ "1386": 5.16396,
+ "1387": 5.1583,
+ "1388": 5.28236,
+ "1389": 5.30286,
+ "1390": 5.25898,
+ "1391": 5.2694,
+ "1392": 5.37607,
+ "1393": 5.3831,
+ "1394": 5.40675,
+ "1395": 5.3281,
+ "1396": 5.21546,
+ "1397": 5.28236,
+ "1398": 5.37289,
+ "1399": 5.36068,
+ "1400": 5.26563,
+ "1401": 5.35908,
+ "1402": 5.42753,
+ "1403": 5.19491,
+ "1404": 5.27992,
+ "1405": 5.11536,
+ "1406": 4.9848,
+ "1407": 5.40083,
+ "1408": 5.19524,
+ "1409": 5.39404,
+ "1410": 5.36929,
+ "1411": 4.91721,
+ "1412": 5.35162,
+ "1413": 5.41372,
+ "1414": 5.21586,
+ "1415": 5.44157,
+ "1416": 5.32647,
+ "1417": 5.38826,
+ "1418": 5.30164,
+ "1419": 5.31481,
+ "1420": 5.43963,
+ "1421": 5.39318,
+ "1422": 5.41729,
+ "1423": 5.00631,
+ "1424": 5.3302,
+ "1425": 5.5858,
+ "1426": 5.23227,
+ "1427": 5.32035,
+ "1428": 5.33332,
+ "1429": 5.07663,
+ "1430": 5.33073,
+ "1431": 5.32483,
+ "1432": 5.33985,
+ "1433": 5.18838,
+ "1434": 5.16572,
+ "1435": 5.1958,
+ "1436": 5.10746,
+ "1437": 5.22734,
+ "1438": 5.31673,
+ "1439": 5.34868,
+ "1440": 5.35006,
+ "1441": 5.1648,
+ "1442": 5.21923,
+ "1443": 5.20787,
+ "1444": 5.13473,
+ "1445": 5.07937,
+ "1446": 5.26893,
+ "1447": 5.25812,
+ "1448": 5.29023,
+ "1449": 5.25124,
+ "1450": 5.34053,
+ "1451": 5.06818,
+ "1452": 5.26893,
+ "1453": 5.16402,
+ "1454": 5.01373,
+ "1455": 5.11873,
+ "1456": 5.27184,
+ "1457": 5.18567,
+ "1458": 5.00628,
+ "1459": 5.22219,
+ "1460": 5.23462,
+ "1461": 5.07075,
+ "1462": 4.9725,
+ "1463": 5.14484,
+ "1464": 5.21091,
+ "1465": 5.26885,
+ "1466": 5.34664,
+ "1467": 5.33439,
+ "1468": 5.2218,
+ "1469": 5.04444,
+ "1470": 5.11996,
+ "1471": 5.25367,
+ "1472": 5.12317,
+ "1473": 5.10366,
+ "1474": 5.21756,
+ "1475": 5.1893,
+ "1476": 5.15421,
+ "1477": 5.26198,
+ "1478": 5.30526,
+ "1479": 5.01174,
+ "1480": 5.18193,
+ "1481": 5.24634,
+ "1482": 5.3494,
+ "1483": 5.26702,
+ "1484": 4.92595,
+ "1485": 5.28899,
+ "1486": 5.04472,
+ "1487": 4.88488,
+ "1488": 5.18121,
+ "1489": 5.10266,
+ "1490": 5.04414,
+ "1491": 5.31893,
+ "1492": 5.22307,
+ "1493": 4.9418,
+ "1494": 5.10858,
+ "1495": 5.13457,
+ "1496": 5.05704,
+ "1497": 5.36465,
+ "1498": 5.30879,
+ "1499": 5.1366,
+ "1500": 5.09707,
+ "1501": 5.03438,
+ "1502": 5.15323,
+ "1503": 5.43116,
+ "1504": 5.32042,
+ "1505": 5.0025,
+ "1506": 5.14503,
+ "1507": 5.16031,
+ "1508": 5.16674,
+ "1509": 5.31267,
+ "1510": 5.01977,
+ "1511": 5.11765,
+ "1512": 4.98424,
+ "1513": 5.17204,
+ "1514": 5.34407,
+ "1515": 5.36401,
+ "1516": 5.27517,
+ "1517": 5.22532,
+ "1518": 5.02571,
+ "1519": 5.29909,
+ "1520": 5.14102,
+ "1521": 5.15936,
+ "1522": 5.32861,
+ "1523": 5.24322,
+ "1524": 5.07017,
+ "1525": 5.20586,
+ "1526": 5.27789,
+ "1527": 5.25771,
+ "1528": 5.23392,
+ "1529": 5.18407,
+ "1530": 5.24452,
+ "1531": 5.09721,
+ "1532": 5.15198,
+ "1533": 5.04999,
+ "1534": 5.21313,
+ "1535": 5.16084,
+ "1536": 5.09571,
+ "1537": 5.02584,
+ "1538": 4.91251,
+ "1539": 5.23704,
+ "1540": 5.11463,
+ "1541": 5.25234,
+ "1542": 5.23487,
+ "1543": 5.05103,
+ "1544": 5.0757,
+ "1545": 5.11687,
+ "1546": 5.32884,
+ "1547": 5.11126,
+ "1548": 5.23297,
+ "1549": 5.23554,
+ "1550": 4.9755,
+ "1551": 5.25626,
+ "1552": 5.0283,
+ "1553": 5.15188,
+ "1554": 5.11353,
+ "1555": 5.10966,
+ "1556": 5.19905,
+ "1557": 5.08642,
+ "1558": 5.23026,
+ "1559": 5.0028,
+ "1560": 5.11721,
+ "1561": 5.14477,
+ "1562": 5.18004,
+ "1563": 5.24484,
+ "1564": 5.26046,
+ "1565": 5.08621,
+ "1566": 5.29215,
+ "1567": 5.04297,
+ "1568": 5.0924,
+ "1569": 5.19825,
+ "1570": 5.17322,
+ "1571": 4.95354,
+ "1572": 5.04047,
+ "1573": 5.02833,
+ "1574": 4.99904,
+ "1575": 5.22914,
+ "1576": 5.20911,
+ "1577": 5.12435,
+ "1578": 5.36333,
+ "1579": 4.94147,
+ "1580": 5.12063,
+ "1581": 5.09806,
+ "1582": 5.28476,
+ "1583": 5.04673,
+ "1584": 5.05573,
+ "1585": 5.11934,
+ "1586": 5.30257,
+ "1587": 5.13431,
+ "1588": 5.21774,
+ "1589": 4.83758,
+ "1590": 5.09536,
+ "1591": 5.17304,
+ "1592": 5.13765,
+ "1593": 5.23293,
+ "1594": 5.11477,
+ "1595": 5.10474,
+ "1596": 5.18963,
+ "1597": 5.11455,
+ "1598": 5.15871,
+ "1599": 5.18832,
+ "1600": 4.867,
+ "1601": 5.11657,
+ "1602": 5.2318,
+ "1603": 5.19675,
+ "1604": 5.05141,
+ "1605": 5.02778,
+ "1606": 4.98264,
+ "1607": 5.07068,
+ "1608": 4.97472,
+ "1609": 5.06996,
+ "1610": 5.04447,
+ "1611": 4.99592,
+ "1612": 4.7496,
+ "1613": 5.03141,
+ "1614": 4.87859,
+ "1615": 5.07068,
+ "1616": 5.23091,
+ "1617": 5.06147,
+ "1618": 4.98903,
+ "1619": 5.18283,
+ "1620": 5.14442,
+ "1621": 5.31495,
+ "1622": 5.05896,
+ "1623": 5.14262,
+ "1624": 5.13392,
+ "1625": 5.11876,
+ "1626": 5.10167,
+ "1627": 5.10618,
+ "1628": 5.06249,
+ "1629": 4.9286,
+ "1630": 5.06663,
+ "1631": 5.05909,
+ "1632": 5.10185,
+ "1633": 4.97171,
+ "1634": 4.9212,
+ "1635": 5.05593,
+ "1636": 4.92086,
+ "1637": 5.23772,
+ "1638": 5.15614,
+ "1639": 4.9737,
+ "1640": 5.00627,
+ "1641": 5.12839,
+ "1642": 5.08116,
+ "1643": 5.04657,
+ "1644": 5.1167,
+ "1645": 4.96304,
+ "1646": 5.11677,
+ "1647": 5.03097,
+ "1648": 5.19302,
+ "1649": 4.92182,
+ "1650": 5.05753,
+ "1651": 4.92902,
+ "1652": 5.21174,
+ "1653": 5.15746,
+ "1654": 5.12656,
+ "1655": 5.15854,
+ "1656": 5.34227,
+ "1657": 5.20581,
+ "1658": 5.04089,
+ "1659": 4.92312,
+ "1660": 4.81039,
+ "1661": 5.0292,
+ "1662": 5.14199,
+ "1663": 5.15695,
+ "1664": 4.98242,
+ "1665": 5.11276,
+ "1666": 5.10599,
+ "1667": 4.84893,
+ "1668": 5.11561,
+ "1669": 5.0708,
+ "1670": 5.11196,
+ "1671": 5.16931,
+ "1672": 4.7741,
+ "1673": 5.03666,
+ "1674": 4.91624,
+ "1675": 5.04978,
+ "1676": 5.00507,
+ "1677": 4.80013,
+ "1678": 5.02504,
+ "1679": 4.89425,
+ "1680": 5.03784,
+ "1681": 5.0667,
+ "1682": 5.0307,
+ "1683": 4.90464,
+ "1684": 5.06247,
+ "1685": 5.13247,
+ "1686": 5.07232,
+ "1687": 4.9758,
+ "1688": 5.16521,
+ "1689": 5.1414,
+ "1690": 4.99677,
+ "1691": 4.99802,
+ "1692": 4.91922,
+ "1693": 5.0164,
+ "1694": 4.9446,
+ "1695": 4.91466,
+ "1696": 5.08193,
+ "1697": 5.04317,
+ "1698": 4.94882,
+ "1699": 5.00079,
+ "1700": 4.95005,
+ "1701": 5.16402,
+ "1702": 5.07614,
+ "1703": 5.1673,
+ "1704": 5.1408,
+ "1705": 4.96176,
+ "1706": 4.98206,
+ "1707": 4.79094,
+ "1708": 5.03566,
+ "1709": 5.23003,
+ "1710": 5.03135,
+ "1711": 5.1887,
+ "1712": 5.19475,
+ "1713": 5.03591,
+ "1714": 5.04526,
+ "1715": 4.91288,
+ "1716": 4.93193,
+ "1717": 4.85989,
+ "1718": 5.02315,
+ "1719": 5.12112,
+ "1720": 5.01984,
+ "1721": 4.92376,
+ "1722": 5.05403,
+ "1723": 4.93307,
+ "1724": 5.03681,
+ "1725": 5.19181,
+ "1726": 5.06518,
+ "1727": 4.90806,
+ "1728": 5.01889,
+ "1729": 5.0423,
+ "1730": 4.90041,
+ "1731": 4.99721,
+ "1732": 4.91602,
+ "1733": 5.13062,
+ "1734": 4.82913,
+ "1735": 5.20931,
+ "1736": 4.91465,
+ "1737": 4.85955,
+ "1738": 4.97659,
+ "1739": 5.16419,
+ "1740": 4.83418,
+ "1741": 4.77762,
+ "1742": 4.90672,
+ "1743": 5.08287,
+ "1744": 4.97814,
+ "1745": 4.82137,
+ "1746": 4.94608,
+ "1747": 4.8657,
+ "1748": 5.05939,
+ "1749": 4.86501,
+ "1750": 5.0105,
+ "1751": 5.11911,
+ "1752": 4.89833,
+ "1753": 5.08961,
+ "1754": 5.05622,
+ "1755": 4.89491,
+ "1756": 5.01907,
+ "1757": 5.14179,
+ "1758": 4.86585,
+ "1759": 4.94022,
+ "1760": 4.83142,
+ "1761": 5.01761,
+ "1762": 4.81176,
+ "1763": 4.77099,
+ "1764": 4.93585,
+ "1765": 5.14759,
+ "1766": 5.3369,
+ "1767": 5.22207,
+ "1768": 4.94936,
+ "1769": 5.00459,
+ "1770": 4.98043,
+ "1771": 4.96248,
+ "1772": 4.9802,
+ "1773": 4.96974,
+ "1774": 4.86656,
+ "1775": 4.95242,
+ "1776": 4.99511,
+ "1777": 4.94129,
+ "1778": 4.99174,
+ "1779": 5.08269,
+ "1780": 4.8318,
+ "1781": 5.05042,
+ "1782": 4.99417,
+ "1783": 5.00979,
+ "1784": 4.93038,
+ "1785": 5.16439,
+ "1786": 4.80989,
+ "1787": 4.96703,
+ "1788": 4.82414,
+ "1789": 4.89042,
+ "1790": 4.79635,
+ "1791": 4.73565,
+ "1792": 4.87572,
+ "1793": 5.10438,
+ "1794": 4.9781,
+ "1795": 4.96531,
+ "1796": 4.99575,
+ "1797": 4.78854,
+ "1798": 4.76673,
+ "1799": 5.01738,
+ "1800": 4.90966,
+ "1801": 5.04285,
+ "1802": 4.82461,
+ "1803": 4.94659,
+ "1804": 4.88402,
+ "1805": 4.90136,
+ "1806": 4.87147,
+ "1807": 4.92546,
+ "1808": 4.9262,
+ "1809": 5.14262,
+ "1810": 5.09505,
+ "1811": 4.95642,
+ "1812": 4.79788,
+ "1813": 5.0927,
+ "1814": 4.7753,
+ "1815": 4.85681,
+ "1816": 5.04864,
+ "1817": 4.78825,
+ "1818": 4.80029,
+ "1819": 5.02251,
+ "1820": 4.6829,
+ "1821": 5.02182,
+ "1822": 4.66083,
+ "1823": 4.86319,
+ "1824": 4.78714,
+ "1825": 5.0651,
+ "1826": 4.82004,
+ "1827": 4.79022,
+ "1828": 4.94795,
+ "1829": 5.10968,
+ "1830": 4.91251,
+ "1831": 4.89951,
+ "1832": 4.82961,
+ "1833": 4.78182,
+ "1834": 4.94773,
+ "1835": 4.95556,
+ "1836": 4.90778,
+ "1837": 4.67083,
+ "1838": 4.80671,
+ "1839": 4.89775,
+ "1840": 4.90381,
+ "1841": 4.82827,
+ "1842": 4.94657,
+ "1843": 4.70068,
+ "1844": 4.61412,
+ "1845": 4.99426,
+ "1846": 4.74152,
+ "1847": 4.85946,
+ "1848": 4.8938,
+ "1849": 4.847,
+ "1850": 4.8644,
+ "1851": 5.01616,
+ "1852": 4.9726,
+ "1853": 4.82815,
+ "1854": 4.86342,
+ "1855": 4.81897,
+ "1856": 4.75181,
+ "1857": 4.95994,
+ "1858": 4.96508,
+ "1859": 4.75056,
+ "1860": 4.86695,
+ "1861": 5.20669,
+ "1862": 4.61732,
+ "1863": 4.83224,
+ "1864": 4.75102,
+ "1865": 4.86452,
+ "1866": 4.79554,
+ "1867": 4.99825,
+ "1868": 4.713,
+ "1869": 4.7563,
+ "1870": 4.9315,
+ "1871": 4.99173,
+ "1872": 4.68178,
+ "1873": 4.69558,
+ "1874": 4.8497,
+ "1875": 4.85971,
+ "1876": 4.73882,
+ "1877": 4.80199,
+ "1878": 4.81156,
+ "1879": 4.81908,
+ "1880": 4.88978,
+ "1881": 4.78966,
+ "1882": 4.79307,
+ "1883": 4.77931,
+ "1884": 4.96984,
+ "1885": 4.91752,
+ "1886": 4.82246,
+ "1887": 4.81335,
+ "1888": 4.97338,
+ "1889": 4.95969,
+ "1890": 4.7038,
+ "1891": 4.6521,
+ "1892": 4.84567,
+ "1893": 4.64592,
+ "1894": 4.90149,
+ "1895": 4.78501,
+ "1896": 4.65725,
+ "1897": 4.78841,
+ "1898": 4.91203,
+ "1899": 4.77289,
+ "1900": 4.91497,
+ "1901": 4.84233,
+ "1902": 4.78012,
+ "1903": 4.75465,
+ "1904": 4.65154,
+ "1905": 4.54185,
+ "1906": 4.81121,
+ "1907": 4.90146,
+ "1908": 5.03036,
+ "1909": 4.8855,
+ "1910": 4.79188,
+ "1911": 4.80444,
+ "1912": 4.64704,
+ "1913": 4.94248,
+ "1914": 4.87799,
+ "1915": 4.85815,
+ "1916": 4.9233,
+ "1917": 4.85188,
+ "1918": 4.86821,
+ "1919": 4.99365,
+ "1920": 4.76403,
+ "1921": 4.88453,
+ "1922": 4.81322,
+ "1923": 4.75877,
+ "1924": 4.82358,
+ "1925": 5.05569,
+ "1926": 4.92765,
+ "1927": 4.92371,
+ "1928": 4.92547,
+ "1929": 4.92345,
+ "1930": 4.90814,
+ "1931": 4.77438,
+ "1932": 4.85786,
+ "1933": 4.83291,
+ "1934": 4.83985,
+ "1935": 5.10437,
+ "1936": 4.87842,
+ "1937": 4.87322,
+ "1938": 4.79495,
+ "1939": 4.71108,
+ "1940": 4.82371,
+ "1941": 4.73172,
+ "1942": 4.86949,
+ "1943": 4.73592,
+ "1944": 4.74162,
+ "1945": 4.68206,
+ "1946": 4.91299,
+ "1947": 4.8668,
+ "1948": 4.5951,
+ "1949": 4.89995,
+ "1950": 4.78675,
+ "1951": 4.95875,
+ "1952": 4.7339,
+ "1953": 4.79464,
+ "1954": 4.72809,
+ "1955": 4.84101,
+ "1956": 4.87704,
+ "1957": 4.73057,
+ "1958": 4.6974,
+ "1959": 4.75832,
+ "1960": 4.76529,
+ "1961": 4.71243,
+ "1962": 4.83569,
+ "1963": 4.81982,
+ "1964": 4.84725,
+ "1965": 4.87387,
+ "1966": 4.78553,
+ "1967": 4.58932,
+ "1968": 4.82785,
+ "1969": 4.5892,
+ "1970": 4.56608,
+ "1971": 4.90472,
+ "1972": 4.89801,
+ "1973": 4.54388,
+ "1974": 4.82108,
+ "1975": 4.82856,
+ "1976": 4.71102,
+ "1977": 4.57803,
+ "1978": 5.00505,
+ "1979": 4.66216,
+ "1980": 4.74215,
+ "1981": 4.86046,
+ "1982": 4.71964,
+ "1983": 4.87905,
+ "1984": 4.64353,
+ "1985": 4.78005,
+ "1986": 4.70111,
+ "1987": 4.81152,
+ "1988": 4.88506,
+ "1989": 4.63082,
+ "1990": 4.7934,
+ "1991": 4.69341,
+ "1992": 4.78836,
+ "1993": 4.73771,
+ "1994": 4.8448,
+ "1995": 4.55362,
+ "1996": 4.64967,
+ "1997": 4.80142,
+ "1998": 4.66988,
+ "1999": 4.72199,
+ "2000": 4.61573
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 75.0,
+ "2": 89.0,
+ "3": 87.0,
+ "4": 61.0,
+ "5": 63.0,
+ "6": 76.0,
+ "7": 87.0,
+ "8": 65.0,
+ "9": 67.0,
+ "10": 90.0,
+ "11": 67.0,
+ "12": 86.0,
+ "13": 82.0,
+ "14": 70.0,
+ "15": 80.0,
+ "16": 79.0,
+ "17": 82.0,
+ "18": 84.0,
+ "19": 58.0,
+ "20": 95.0,
+ "21": 76.0,
+ "22": 100.0,
+ "23": 78.0,
+ "24": 68.0,
+ "25": 99.0,
+ "26": 92.0,
+ "27": 84.0,
+ "28": 101.0,
+ "29": 96.0,
+ "30": 90.0,
+ "31": 103.0,
+ "32": 89.0,
+ "33": 103.0,
+ "34": 120.0,
+ "35": 110.0,
+ "36": 122.0,
+ "37": 116.0,
+ "38": 123.0,
+ "39": 110.0,
+ "40": 112.0,
+ "41": 103.0,
+ "42": 112.0,
+ "43": 133.0,
+ "44": 125.0,
+ "45": 117.0,
+ "46": 136.0,
+ "47": 135.0,
+ "48": 100.0,
+ "49": 132.0,
+ "50": 155.0,
+ "51": 140.0,
+ "52": 114.0,
+ "53": 133.0,
+ "54": 129.0,
+ "55": 133.0,
+ "56": 165.0,
+ "57": 134.0,
+ "58": 128.0,
+ "59": 148.0,
+ "60": 134.0,
+ "61": 142.0,
+ "62": 159.0,
+ "63": 175.0,
+ "64": 161.0,
+ "65": 136.0,
+ "66": 194.0,
+ "67": 178.0,
+ "68": 163.0,
+ "69": 143.0,
+ "70": 133.0,
+ "71": 190.0,
+ "72": 166.0,
+ "73": 158.0,
+ "74": 169.0,
+ "75": 156.0,
+ "76": 171.0,
+ "77": 182.0,
+ "78": 200.0,
+ "79": 125.0,
+ "80": 170.0,
+ "81": 165.0,
+ "82": 154.0,
+ "83": 167.0,
+ "84": 155.0,
+ "85": 164.0,
+ "86": 162.0,
+ "87": 188.0,
+ "88": 173.0,
+ "89": 170.0,
+ "90": 187.0,
+ "91": 189.0,
+ "92": 189.0,
+ "93": 155.0,
+ "94": 169.0,
+ "95": 161.0,
+ "96": 189.0,
+ "97": 186.0,
+ "98": 194.0,
+ "99": 166.0,
+ "100": 180.0,
+ "101": 179.0,
+ "102": 148.0,
+ "103": 182.0,
+ "104": 182.0,
+ "105": 197.0,
+ "106": 160.0,
+ "107": 178.0,
+ "108": 209.0,
+ "109": 168.0,
+ "110": 223.0,
+ "111": 203.0,
+ "112": 165.0,
+ "113": 143.0,
+ "114": 161.0,
+ "115": 195.0,
+ "116": 164.0,
+ "117": 190.0,
+ "118": 158.0,
+ "119": 193.0,
+ "120": 176.0,
+ "121": 172.0,
+ "122": 152.0,
+ "123": 171.0,
+ "124": 200.0,
+ "125": 185.0,
+ "126": 158.0,
+ "127": 187.0,
+ "128": 201.0,
+ "129": 172.0,
+ "130": 180.0,
+ "131": 189.0,
+ "132": 175.0,
+ "133": 176.0,
+ "134": 178.0,
+ "135": 161.0,
+ "136": 157.0,
+ "137": 153.0,
+ "138": 157.0,
+ "139": 170.0,
+ "140": 178.0,
+ "141": 160.0,
+ "142": 144.0,
+ "143": 196.0,
+ "144": 165.0,
+ "145": 156.0,
+ "146": 148.0,
+ "147": 119.0,
+ "148": 162.0,
+ "149": 142.0,
+ "150": 107.0,
+ "151": 173.0,
+ "152": 127.0,
+ "153": 125.0,
+ "154": 153.0,
+ "155": 187.0,
+ "156": 170.0,
+ "157": 145.0,
+ "158": 137.0,
+ "159": 182.0,
+ "160": 164.0,
+ "161": 142.0,
+ "162": 177.0,
+ "163": 129.0,
+ "164": 192.0,
+ "165": 173.0,
+ "166": 186.0,
+ "167": 128.0,
+ "168": 157.0,
+ "169": 195.0,
+ "170": 152.0,
+ "171": 150.0,
+ "172": 173.0,
+ "173": 171.0,
+ "174": 188.0,
+ "175": 187.0,
+ "176": 166.0,
+ "177": 162.0,
+ "178": 159.0,
+ "179": 166.0,
+ "180": 140.0,
+ "181": 164.0,
+ "182": 183.0,
+ "183": 153.0,
+ "184": 166.0,
+ "185": 175.0,
+ "186": 196.0,
+ "187": 183.0,
+ "188": 165.0,
+ "189": 169.0,
+ "190": 166.0,
+ "191": 177.0,
+ "192": 166.0,
+ "193": 178.0,
+ "194": 183.0,
+ "195": 165.0,
+ "196": 187.0,
+ "197": 164.0,
+ "198": 155.0,
+ "199": 139.0,
+ "200": 160.0,
+ "201": 170.0,
+ "202": 198.0,
+ "203": 133.0,
+ "204": 176.0,
+ "205": 125.0,
+ "206": 160.0,
+ "207": 174.0,
+ "208": 131.0,
+ "209": 217.0,
+ "210": 162.0,
+ "211": 169.0,
+ "212": 182.0,
+ "213": 170.0,
+ "214": 169.0,
+ "215": 150.0,
+ "216": 154.0,
+ "217": 169.0,
+ "218": 189.0,
+ "219": 178.0,
+ "220": 185.0,
+ "221": 177.0,
+ "222": 172.0,
+ "223": 164.0,
+ "224": 177.0,
+ "225": 184.0,
+ "226": 156.0,
+ "227": 181.0,
+ "228": 164.0,
+ "229": 209.0,
+ "230": 146.0,
+ "231": 150.0,
+ "232": 149.0,
+ "233": 198.0,
+ "234": 186.0,
+ "235": 199.0,
+ "236": 198.0,
+ "237": 213.0,
+ "238": 186.0,
+ "239": 130.0,
+ "240": 197.0,
+ "241": 167.0,
+ "242": 188.0,
+ "243": 213.0,
+ "244": 180.0,
+ "245": 213.0,
+ "246": 230.0,
+ "247": 214.0,
+ "248": 165.0,
+ "249": 204.0,
+ "250": 201.0,
+ "251": 195.0,
+ "252": 184.0,
+ "253": 162.0,
+ "254": 162.0,
+ "255": 211.0,
+ "256": 157.0,
+ "257": 199.0,
+ "258": 207.0,
+ "259": 156.0,
+ "260": 208.0,
+ "261": 182.0,
+ "262": 186.0,
+ "263": 199.0,
+ "264": 181.0,
+ "265": 159.0,
+ "266": 158.0,
+ "267": 149.0,
+ "268": 173.0,
+ "269": 193.0,
+ "270": 212.0,
+ "271": 199.0,
+ "272": 219.0,
+ "273": 170.0,
+ "274": 171.0,
+ "275": 154.0,
+ "276": 187.0,
+ "277": 182.0,
+ "278": 195.0,
+ "279": 201.0,
+ "280": 175.0,
+ "281": 190.0,
+ "282": 169.0,
+ "283": 198.0,
+ "284": 173.0,
+ "285": 163.0,
+ "286": 140.0,
+ "287": 200.0,
+ "288": 194.0,
+ "289": 217.0,
+ "290": 150.0,
+ "291": 227.0,
+ "292": 186.0,
+ "293": 199.0,
+ "294": 179.0,
+ "295": 232.0,
+ "296": 230.0,
+ "297": 181.0,
+ "298": 129.0,
+ "299": 163.0,
+ "300": 180.0,
+ "301": 174.0,
+ "302": 207.0,
+ "303": 158.0,
+ "304": 159.0,
+ "305": 134.0,
+ "306": 183.0,
+ "307": 170.0,
+ "308": 206.0,
+ "309": 203.0,
+ "310": 200.0,
+ "311": 195.0,
+ "312": 176.0,
+ "313": 192.0,
+ "314": 206.0,
+ "315": 180.0,
+ "316": 175.0,
+ "317": 178.0,
+ "318": 239.0,
+ "319": 196.0,
+ "320": 229.0,
+ "321": 186.0,
+ "322": 193.0,
+ "323": 196.0,
+ "324": 218.0,
+ "325": 178.0,
+ "326": 220.0,
+ "327": 228.0,
+ "328": 217.0,
+ "329": 201.0,
+ "330": 212.0,
+ "331": 202.0,
+ "332": 212.0,
+ "333": 212.0,
+ "334": 182.0,
+ "335": 194.0,
+ "336": 186.0,
+ "337": 199.0,
+ "338": 212.0,
+ "339": 197.0,
+ "340": 185.0,
+ "341": 189.0,
+ "342": 204.0,
+ "343": 256.0,
+ "344": 232.0,
+ "345": 209.0,
+ "346": 184.0,
+ "347": 176.0,
+ "348": 213.0,
+ "349": 188.0,
+ "350": 221.0,
+ "351": 215.0,
+ "352": 185.0,
+ "353": 229.0,
+ "354": 189.0,
+ "355": 197.0,
+ "356": 246.0,
+ "357": 180.0,
+ "358": 208.0,
+ "359": 229.0,
+ "360": 206.0,
+ "361": 182.0,
+ "362": 235.0,
+ "363": 205.0,
+ "364": 190.0,
+ "365": 179.0,
+ "366": 233.0,
+ "367": 219.0,
+ "368": 191.0,
+ "369": 171.0,
+ "370": 230.0,
+ "371": 216.0,
+ "372": 221.0,
+ "373": 237.0,
+ "374": 175.0,
+ "375": 226.0,
+ "376": 180.0,
+ "377": 193.0,
+ "378": 212.0,
+ "379": 229.0,
+ "380": 196.0,
+ "381": 235.0,
+ "382": 233.0,
+ "383": 202.0,
+ "384": 262.0,
+ "385": 171.0,
+ "386": 215.0,
+ "387": 205.0,
+ "388": 189.0,
+ "389": 227.0,
+ "390": 201.0,
+ "391": 255.0,
+ "392": 181.0,
+ "393": 210.0,
+ "394": 220.0,
+ "395": 250.0,
+ "396": 212.0,
+ "397": 206.0,
+ "398": 214.0,
+ "399": 219.0,
+ "400": 211.0,
+ "401": 227.0,
+ "402": 221.0,
+ "403": 222.0,
+ "404": 204.0,
+ "405": 199.0,
+ "406": 199.0,
+ "407": 203.0,
+ "408": 239.0,
+ "409": 191.0,
+ "410": 217.0,
+ "411": 209.0,
+ "412": 189.0,
+ "413": 159.0,
+ "414": 211.0,
+ "415": 234.0,
+ "416": 238.0,
+ "417": 205.0,
+ "418": 241.0,
+ "419": 173.0,
+ "420": 208.0,
+ "421": 210.0,
+ "422": 212.0,
+ "423": 223.0,
+ "424": 223.0,
+ "425": 238.0,
+ "426": 195.0,
+ "427": 258.0,
+ "428": 194.0,
+ "429": 211.0,
+ "430": 253.0,
+ "431": 207.0,
+ "432": 181.0,
+ "433": 202.0,
+ "434": 178.0,
+ "435": 199.0,
+ "436": 200.0,
+ "437": 188.0,
+ "438": 172.0,
+ "439": 175.0,
+ "440": 213.0,
+ "441": 170.0,
+ "442": 263.0,
+ "443": 225.0,
+ "444": 175.0,
+ "445": 193.0,
+ "446": 206.0,
+ "447": 240.0,
+ "448": 240.0,
+ "449": 230.0,
+ "450": 202.0,
+ "451": 190.0,
+ "452": 165.0,
+ "453": 191.0,
+ "454": 241.0,
+ "455": 248.0,
+ "456": 181.0,
+ "457": 196.0,
+ "458": 171.0,
+ "459": 158.0,
+ "460": 216.0,
+ "461": 218.0,
+ "462": 172.0,
+ "463": 190.0,
+ "464": 186.0,
+ "465": 177.0,
+ "466": 201.0,
+ "467": 188.0,
+ "468": 186.0,
+ "469": 217.0,
+ "470": 198.0,
+ "471": 241.0,
+ "472": 183.0,
+ "473": 211.0,
+ "474": 170.0,
+ "475": 211.0,
+ "476": 180.0,
+ "477": 180.0,
+ "478": 185.0,
+ "479": 172.0,
+ "480": 137.0,
+ "481": 189.0,
+ "482": 165.0,
+ "483": 198.0,
+ "484": 170.0,
+ "485": 191.0,
+ "486": 186.0,
+ "487": 148.0,
+ "488": 217.0,
+ "489": 191.0,
+ "490": 169.0,
+ "491": 162.0,
+ "492": 158.0,
+ "493": 209.0,
+ "494": 174.0,
+ "495": 156.0,
+ "496": 177.0,
+ "497": 144.0,
+ "498": 166.0,
+ "499": 174.0,
+ "500": 155.0,
+ "501": 165.0,
+ "502": 178.0,
+ "503": 181.0,
+ "504": 161.0,
+ "505": 157.0,
+ "506": 190.0,
+ "507": 176.0,
+ "508": 176.0,
+ "509": 193.0,
+ "510": 146.0,
+ "511": 167.0,
+ "512": 166.0,
+ "513": 193.0,
+ "514": 159.0,
+ "515": 162.0,
+ "516": 141.0,
+ "517": 173.0,
+ "518": 150.0,
+ "519": 168.0,
+ "520": 174.0,
+ "521": 128.0,
+ "522": 146.0,
+ "523": 151.0,
+ "524": 155.0,
+ "525": 161.0,
+ "526": 132.0,
+ "527": 212.0,
+ "528": 183.0,
+ "529": 174.0,
+ "530": 177.0,
+ "531": 162.0,
+ "532": 176.0,
+ "533": 198.0,
+ "534": 189.0,
+ "535": 151.0,
+ "536": 171.0,
+ "537": 131.0,
+ "538": 152.0,
+ "539": 108.0,
+ "540": 170.0,
+ "541": 139.0,
+ "542": 165.0,
+ "543": 145.0,
+ "544": 173.0,
+ "545": 160.0,
+ "546": 155.0,
+ "547": 179.0,
+ "548": 168.0,
+ "549": 161.0,
+ "550": 129.0,
+ "551": 172.0,
+ "552": 158.0,
+ "553": 190.0,
+ "554": 161.0,
+ "555": 156.0,
+ "556": 177.0,
+ "557": 158.0,
+ "558": 169.0,
+ "559": 140.0,
+ "560": 168.0,
+ "561": 169.0,
+ "562": 168.0,
+ "563": 137.0,
+ "564": 173.0,
+ "565": 151.0,
+ "566": 140.0,
+ "567": 152.0,
+ "568": 163.0,
+ "569": 146.0,
+ "570": 135.0,
+ "571": 156.0,
+ "572": 158.0,
+ "573": 156.0,
+ "574": 189.0,
+ "575": 149.0,
+ "576": 159.0,
+ "577": 150.0,
+ "578": 165.0,
+ "579": 162.0,
+ "580": 174.0,
+ "581": 174.0,
+ "582": 135.0,
+ "583": 137.0,
+ "584": 130.0,
+ "585": 136.0,
+ "586": 121.0,
+ "587": 140.0,
+ "588": 145.0,
+ "589": 119.0,
+ "590": 133.0,
+ "591": 154.0,
+ "592": 127.0,
+ "593": 155.0,
+ "594": 130.0,
+ "595": 123.0,
+ "596": 130.0,
+ "597": 119.0,
+ "598": 149.0,
+ "599": 155.0,
+ "600": 152.0,
+ "601": 147.0,
+ "602": 137.0,
+ "603": 158.0,
+ "604": 157.0,
+ "605": 135.0,
+ "606": 129.0,
+ "607": 142.0,
+ "608": 157.0,
+ "609": 146.0,
+ "610": 138.0,
+ "611": 149.0,
+ "612": 163.0,
+ "613": 142.0,
+ "614": 136.0,
+ "615": 134.0,
+ "616": 156.0,
+ "617": 113.0,
+ "618": 139.0,
+ "619": 151.0,
+ "620": 153.0,
+ "621": 137.0,
+ "622": 138.0,
+ "623": 139.0,
+ "624": 122.0,
+ "625": 120.0,
+ "626": 119.0,
+ "627": 125.0,
+ "628": 124.0,
+ "629": 157.0,
+ "630": 122.0,
+ "631": 152.0,
+ "632": 123.0,
+ "633": 161.0,
+ "634": 129.0,
+ "635": 132.0,
+ "636": 133.0,
+ "637": 136.0,
+ "638": 125.0,
+ "639": 152.0,
+ "640": 131.0,
+ "641": 120.0,
+ "642": 129.0,
+ "643": 156.0,
+ "644": 129.0,
+ "645": 143.0,
+ "646": 113.0,
+ "647": 132.0,
+ "648": 148.0,
+ "649": 166.0,
+ "650": 164.0,
+ "651": 136.0,
+ "652": 128.0,
+ "653": 153.0,
+ "654": 131.0,
+ "655": 143.0,
+ "656": 163.0,
+ "657": 151.0,
+ "658": 137.0,
+ "659": 117.0,
+ "660": 127.0,
+ "661": 132.0,
+ "662": 128.0,
+ "663": 109.0,
+ "664": 135.0,
+ "665": 127.0,
+ "666": 118.0,
+ "667": 115.0,
+ "668": 127.0,
+ "669": 136.0,
+ "670": 126.0,
+ "671": 117.0,
+ "672": 132.0,
+ "673": 126.0,
+ "674": 109.0,
+ "675": 132.0,
+ "676": 110.0,
+ "677": 141.0,
+ "678": 122.0,
+ "679": 92.0,
+ "680": 152.0,
+ "681": 115.0,
+ "682": 107.0,
+ "683": 156.0,
+ "684": 122.0,
+ "685": 120.0,
+ "686": 142.0,
+ "687": 141.0,
+ "688": 124.0,
+ "689": 119.0,
+ "690": 99.0,
+ "691": 147.0,
+ "692": 138.0,
+ "693": 143.0,
+ "694": 169.0,
+ "695": 119.0,
+ "696": 129.0,
+ "697": 128.0,
+ "698": 125.0,
+ "699": 131.0,
+ "700": 126.0,
+ "701": 135.0,
+ "702": 115.0,
+ "703": 101.0,
+ "704": 147.0,
+ "705": 126.0,
+ "706": 123.0,
+ "707": 140.0,
+ "708": 124.0,
+ "709": 111.0,
+ "710": 150.0,
+ "711": 146.0,
+ "712": 152.0,
+ "713": 117.0,
+ "714": 140.0,
+ "715": 115.0,
+ "716": 146.0,
+ "717": 133.0,
+ "718": 150.0,
+ "719": 122.0,
+ "720": 128.0,
+ "721": 122.0,
+ "722": 127.0,
+ "723": 128.0,
+ "724": 113.0,
+ "725": 129.0,
+ "726": 118.0,
+ "727": 94.0,
+ "728": 138.0,
+ "729": 118.0,
+ "730": 98.0,
+ "731": 107.0,
+ "732": 115.0,
+ "733": 122.0,
+ "734": 101.0,
+ "735": 111.0,
+ "736": 105.0,
+ "737": 108.0,
+ "738": 96.0,
+ "739": 111.0,
+ "740": 126.0,
+ "741": 104.0,
+ "742": 121.0,
+ "743": 107.0,
+ "744": 106.0,
+ "745": 130.0,
+ "746": 92.0,
+ "747": 144.0,
+ "748": 110.0,
+ "749": 96.0,
+ "750": 125.0,
+ "751": 117.0,
+ "752": 114.0,
+ "753": 110.0,
+ "754": 125.0,
+ "755": 124.0,
+ "756": 114.0,
+ "757": 103.0,
+ "758": 103.0,
+ "759": 123.0,
+ "760": 134.0,
+ "761": 90.0,
+ "762": 110.0,
+ "763": 102.0,
+ "764": 106.0,
+ "765": 122.0,
+ "766": 104.0,
+ "767": 128.0,
+ "768": 137.0,
+ "769": 121.0,
+ "770": 135.0,
+ "771": 119.0,
+ "772": 103.0,
+ "773": 111.0,
+ "774": 113.0,
+ "775": 84.0,
+ "776": 114.0,
+ "777": 110.0,
+ "778": 99.0,
+ "779": 88.0,
+ "780": 90.0,
+ "781": 102.0,
+ "782": 86.0,
+ "783": 84.0,
+ "784": 111.0,
+ "785": 92.0,
+ "786": 81.0,
+ "787": 83.0,
+ "788": 95.0,
+ "789": 104.0,
+ "790": 99.0,
+ "791": 120.0,
+ "792": 88.0,
+ "793": 131.0,
+ "794": 122.0,
+ "795": 108.0,
+ "796": 126.0,
+ "797": 115.0,
+ "798": 115.0,
+ "799": 102.0,
+ "800": 96.0,
+ "801": 121.0,
+ "802": 87.0,
+ "803": 117.0,
+ "804": 86.0,
+ "805": 93.0,
+ "806": 89.0,
+ "807": 122.0,
+ "808": 96.0,
+ "809": 109.0,
+ "810": 87.0,
+ "811": 106.0,
+ "812": 103.0,
+ "813": 88.0,
+ "814": 95.0,
+ "815": 99.0,
+ "816": 136.0,
+ "817": 112.0,
+ "818": 105.0,
+ "819": 106.0,
+ "820": 121.0,
+ "821": 118.0,
+ "822": 100.0,
+ "823": 92.0,
+ "824": 94.0,
+ "825": 110.0,
+ "826": 101.0,
+ "827": 125.0,
+ "828": 78.0,
+ "829": 101.0,
+ "830": 97.0,
+ "831": 138.0,
+ "832": 108.0,
+ "833": 119.0,
+ "834": 104.0,
+ "835": 91.0,
+ "836": 129.0,
+ "837": 129.0,
+ "838": 114.0,
+ "839": 109.0,
+ "840": 118.0,
+ "841": 105.0,
+ "842": 124.0,
+ "843": 90.0,
+ "844": 98.0,
+ "845": 147.0,
+ "846": 103.0,
+ "847": 134.0,
+ "848": 109.0,
+ "849": 114.0,
+ "850": 100.0,
+ "851": 114.0,
+ "852": 105.0,
+ "853": 97.0,
+ "854": 104.0,
+ "855": 102.0,
+ "856": 116.0,
+ "857": 135.0,
+ "858": 128.0,
+ "859": 115.0,
+ "860": 113.0,
+ "861": 103.0,
+ "862": 99.0,
+ "863": 97.0,
+ "864": 95.0,
+ "865": 109.0,
+ "866": 95.0,
+ "867": 112.0,
+ "868": 92.0,
+ "869": 115.0,
+ "870": 141.0,
+ "871": 128.0,
+ "872": 109.0,
+ "873": 109.0,
+ "874": 120.0,
+ "875": 87.0,
+ "876": 141.0,
+ "877": 107.0,
+ "878": 108.0,
+ "879": 111.0,
+ "880": 101.0,
+ "881": 102.0,
+ "882": 110.0,
+ "883": 99.0,
+ "884": 122.0,
+ "885": 124.0,
+ "886": 119.0,
+ "887": 107.0,
+ "888": 94.0,
+ "889": 122.0,
+ "890": 103.0,
+ "891": 112.0,
+ "892": 102.0,
+ "893": 104.0,
+ "894": 92.0,
+ "895": 96.0,
+ "896": 99.0,
+ "897": 90.0,
+ "898": 104.0,
+ "899": 117.0,
+ "900": 114.0,
+ "901": 97.0,
+ "902": 107.0,
+ "903": 136.0,
+ "904": 110.0,
+ "905": 98.0,
+ "906": 103.0,
+ "907": 107.0,
+ "908": 132.0,
+ "909": 119.0,
+ "910": 113.0,
+ "911": 123.0,
+ "912": 100.0,
+ "913": 136.0,
+ "914": 128.0,
+ "915": 116.0,
+ "916": 128.0,
+ "917": 123.0,
+ "918": 111.0,
+ "919": 92.0,
+ "920": 114.0,
+ "921": 107.0,
+ "922": 108.0,
+ "923": 120.0,
+ "924": 96.0,
+ "925": 79.0,
+ "926": 114.0,
+ "927": 91.0,
+ "928": 97.0,
+ "929": 83.0,
+ "930": 100.0,
+ "931": 86.0,
+ "932": 101.0,
+ "933": 113.0,
+ "934": 102.0,
+ "935": 96.0,
+ "936": 80.0,
+ "937": 112.0,
+ "938": 105.0,
+ "939": 102.0,
+ "940": 125.0,
+ "941": 100.0,
+ "942": 85.0,
+ "943": 107.0,
+ "944": 120.0,
+ "945": 122.0,
+ "946": 125.0,
+ "947": 114.0,
+ "948": 113.0,
+ "949": 118.0,
+ "950": 102.0,
+ "951": 143.0,
+ "952": 90.0,
+ "953": 115.0,
+ "954": 103.0,
+ "955": 107.0,
+ "956": 121.0,
+ "957": 99.0,
+ "958": 93.0,
+ "959": 117.0,
+ "960": 83.0,
+ "961": 142.0,
+ "962": 91.0,
+ "963": 106.0,
+ "964": 85.0,
+ "965": 110.0,
+ "966": 104.0,
+ "967": 141.0,
+ "968": 111.0,
+ "969": 112.0,
+ "970": 91.0,
+ "971": 122.0,
+ "972": 113.0,
+ "973": 90.0,
+ "974": 108.0,
+ "975": 110.0,
+ "976": 94.0,
+ "977": 124.0,
+ "978": 91.0,
+ "979": 121.0,
+ "980": 113.0,
+ "981": 102.0,
+ "982": 87.0,
+ "983": 97.0,
+ "984": 108.0,
+ "985": 124.0,
+ "986": 99.0,
+ "987": 93.0,
+ "988": 99.0,
+ "989": 102.0,
+ "990": 101.0,
+ "991": 111.0,
+ "992": 109.0,
+ "993": 85.0,
+ "994": 107.0,
+ "995": 124.0,
+ "996": 89.0,
+ "997": 117.0,
+ "998": 118.0,
+ "999": 116.0,
+ "1000": 117.0,
+ "1001": 98.0,
+ "1002": 112.0,
+ "1003": 96.0,
+ "1004": 106.0,
+ "1005": 76.0,
+ "1006": 112.0,
+ "1007": 110.0,
+ "1008": 115.0,
+ "1009": 110.0,
+ "1010": 111.0,
+ "1011": 125.0,
+ "1012": 106.0,
+ "1013": 104.0,
+ "1014": 97.0,
+ "1015": 98.0,
+ "1016": 112.0,
+ "1017": 92.0,
+ "1018": 109.0,
+ "1019": 92.0,
+ "1020": 113.0,
+ "1021": 88.0,
+ "1022": 112.0,
+ "1023": 109.0,
+ "1024": 122.0,
+ "1025": 116.0,
+ "1026": 100.0,
+ "1027": 92.0,
+ "1028": 93.0,
+ "1029": 91.0,
+ "1030": 91.0,
+ "1031": 112.0,
+ "1032": 100.0,
+ "1033": 91.0,
+ "1034": 110.0,
+ "1035": 99.0,
+ "1036": 82.0,
+ "1037": 105.0,
+ "1038": 100.0,
+ "1039": 120.0,
+ "1040": 90.0,
+ "1041": 109.0,
+ "1042": 102.0,
+ "1043": 91.0,
+ "1044": 130.0,
+ "1045": 106.0,
+ "1046": 111.0,
+ "1047": 113.0,
+ "1048": 86.0,
+ "1049": 130.0,
+ "1050": 96.0,
+ "1051": 125.0,
+ "1052": 100.0,
+ "1053": 82.0,
+ "1054": 84.0,
+ "1055": 138.0,
+ "1056": 91.0,
+ "1057": 96.0,
+ "1058": 101.0,
+ "1059": 72.0,
+ "1060": 92.0,
+ "1061": 79.0,
+ "1062": 117.0,
+ "1063": 88.0,
+ "1064": 95.0,
+ "1065": 131.0,
+ "1066": 112.0,
+ "1067": 102.0,
+ "1068": 106.0,
+ "1069": 91.0,
+ "1070": 111.0,
+ "1071": 101.0,
+ "1072": 105.0,
+ "1073": 101.0,
+ "1074": 117.0,
+ "1075": 97.0,
+ "1076": 113.0,
+ "1077": 103.0,
+ "1078": 105.0,
+ "1079": 100.0,
+ "1080": 104.0,
+ "1081": 105.0,
+ "1082": 111.0,
+ "1083": 96.0,
+ "1084": 88.0,
+ "1085": 105.0,
+ "1086": 121.0,
+ "1087": 99.0,
+ "1088": 104.0,
+ "1089": 86.0,
+ "1090": 110.0,
+ "1091": 103.0,
+ "1092": 99.0,
+ "1093": 89.0,
+ "1094": 122.0,
+ "1095": 91.0,
+ "1096": 114.0,
+ "1097": 118.0,
+ "1098": 89.0,
+ "1099": 119.0,
+ "1100": 113.0,
+ "1101": 99.0,
+ "1102": 95.0,
+ "1103": 100.0,
+ "1104": 101.0,
+ "1105": 104.0,
+ "1106": 98.0,
+ "1107": 93.0,
+ "1108": 95.0,
+ "1109": 107.0,
+ "1110": 105.0,
+ "1111": 106.0,
+ "1112": 95.0,
+ "1113": 119.0,
+ "1114": 93.0,
+ "1115": 107.0,
+ "1116": 92.0,
+ "1117": 128.0,
+ "1118": 106.0,
+ "1119": 119.0,
+ "1120": 100.0,
+ "1121": 79.0,
+ "1122": 106.0,
+ "1123": 86.0,
+ "1124": 106.0,
+ "1125": 90.0,
+ "1126": 102.0,
+ "1127": 102.0,
+ "1128": 105.0,
+ "1129": 101.0,
+ "1130": 111.0,
+ "1131": 106.0,
+ "1132": 111.0,
+ "1133": 84.0,
+ "1134": 109.0,
+ "1135": 86.0,
+ "1136": 86.0,
+ "1137": 103.0,
+ "1138": 104.0,
+ "1139": 129.0,
+ "1140": 105.0,
+ "1141": 90.0,
+ "1142": 118.0,
+ "1143": 121.0,
+ "1144": 106.0,
+ "1145": 106.0,
+ "1146": 99.0,
+ "1147": 97.0,
+ "1148": 116.0,
+ "1149": 116.0,
+ "1150": 79.0,
+ "1151": 96.0,
+ "1152": 89.0,
+ "1153": 94.0,
+ "1154": 88.0,
+ "1155": 97.0,
+ "1156": 73.0,
+ "1157": 108.0,
+ "1158": 99.0,
+ "1159": 98.0,
+ "1160": 96.0,
+ "1161": 101.0,
+ "1162": 90.0,
+ "1163": 110.0,
+ "1164": 116.0,
+ "1165": 95.0,
+ "1166": 100.0,
+ "1167": 91.0,
+ "1168": 111.0,
+ "1169": 114.0,
+ "1170": 108.0,
+ "1171": 99.0,
+ "1172": 112.0,
+ "1173": 87.0,
+ "1174": 96.0,
+ "1175": 110.0,
+ "1176": 86.0,
+ "1177": 103.0,
+ "1178": 108.0,
+ "1179": 103.0,
+ "1180": 96.0,
+ "1181": 75.0,
+ "1182": 123.0,
+ "1183": 94.0,
+ "1184": 97.0,
+ "1185": 95.0,
+ "1186": 87.0,
+ "1187": 107.0,
+ "1188": 120.0,
+ "1189": 106.0,
+ "1190": 113.0,
+ "1191": 93.0,
+ "1192": 86.0,
+ "1193": 106.0,
+ "1194": 90.0,
+ "1195": 107.0,
+ "1196": 98.0,
+ "1197": 111.0,
+ "1198": 102.0,
+ "1199": 92.0,
+ "1200": 111.0,
+ "1201": 110.0,
+ "1202": 113.0,
+ "1203": 105.0,
+ "1204": 110.0,
+ "1205": 106.0,
+ "1206": 116.0,
+ "1207": 103.0,
+ "1208": 112.0,
+ "1209": 77.0,
+ "1210": 106.0,
+ "1211": 109.0,
+ "1212": 117.0,
+ "1213": 109.0,
+ "1214": 104.0,
+ "1215": 81.0,
+ "1216": 110.0,
+ "1217": 100.0,
+ "1218": 116.0,
+ "1219": 117.0,
+ "1220": 109.0,
+ "1221": 99.0,
+ "1222": 95.0,
+ "1223": 111.0,
+ "1224": 96.0,
+ "1225": 115.0,
+ "1226": 91.0,
+ "1227": 108.0,
+ "1228": 108.0,
+ "1229": 107.0,
+ "1230": 90.0,
+ "1231": 93.0,
+ "1232": 88.0,
+ "1233": 99.0,
+ "1234": 105.0,
+ "1235": 94.0,
+ "1236": 106.0,
+ "1237": 110.0,
+ "1238": 97.0,
+ "1239": 106.0,
+ "1240": 97.0,
+ "1241": 118.0,
+ "1242": 105.0,
+ "1243": 91.0,
+ "1244": 100.0,
+ "1245": 131.0,
+ "1246": 101.0,
+ "1247": 107.0,
+ "1248": 111.0,
+ "1249": 125.0,
+ "1250": 116.0,
+ "1251": 125.0,
+ "1252": 112.0,
+ "1253": 80.0,
+ "1254": 109.0,
+ "1255": 116.0,
+ "1256": 101.0,
+ "1257": 110.0,
+ "1258": 96.0,
+ "1259": 99.0,
+ "1260": 96.0,
+ "1261": 104.0,
+ "1262": 124.0,
+ "1263": 89.0,
+ "1264": 122.0,
+ "1265": 78.0,
+ "1266": 115.0,
+ "1267": 104.0,
+ "1268": 108.0,
+ "1269": 117.0,
+ "1270": 109.0,
+ "1271": 109.0,
+ "1272": 120.0,
+ "1273": 97.0,
+ "1274": 111.0,
+ "1275": 105.0,
+ "1276": 116.0,
+ "1277": 104.0,
+ "1278": 114.0,
+ "1279": 86.0,
+ "1280": 111.0,
+ "1281": 118.0,
+ "1282": 88.0,
+ "1283": 94.0,
+ "1284": 117.0,
+ "1285": 83.0,
+ "1286": 115.0,
+ "1287": 99.0,
+ "1288": 95.0,
+ "1289": 95.0,
+ "1290": 90.0,
+ "1291": 98.0,
+ "1292": 98.0,
+ "1293": 82.0,
+ "1294": 108.0,
+ "1295": 80.0,
+ "1296": 81.0,
+ "1297": 126.0,
+ "1298": 85.0,
+ "1299": 88.0,
+ "1300": 109.0,
+ "1301": 115.0,
+ "1302": 91.0,
+ "1303": 104.0,
+ "1304": 109.0,
+ "1305": 119.0,
+ "1306": 134.0,
+ "1307": 81.0,
+ "1308": 115.0,
+ "1309": 128.0,
+ "1310": 97.0,
+ "1311": 108.0,
+ "1312": 110.0,
+ "1313": 106.0,
+ "1314": 103.0,
+ "1315": 113.0,
+ "1316": 105.0,
+ "1317": 113.0,
+ "1318": 103.0,
+ "1319": 117.0,
+ "1320": 111.0,
+ "1321": 116.0,
+ "1322": 78.0,
+ "1323": 100.0,
+ "1324": 94.0,
+ "1325": 120.0,
+ "1326": 95.0,
+ "1327": 104.0,
+ "1328": 83.0,
+ "1329": 93.0,
+ "1330": 109.0,
+ "1331": 100.0,
+ "1332": 109.0,
+ "1333": 112.0,
+ "1334": 102.0,
+ "1335": 100.0,
+ "1336": 99.0,
+ "1337": 113.0,
+ "1338": 103.0,
+ "1339": 85.0,
+ "1340": 119.0,
+ "1341": 107.0,
+ "1342": 112.0,
+ "1343": 109.0,
+ "1344": 84.0,
+ "1345": 114.0,
+ "1346": 99.0,
+ "1347": 89.0,
+ "1348": 106.0,
+ "1349": 97.0,
+ "1350": 84.0,
+ "1351": 111.0,
+ "1352": 108.0,
+ "1353": 105.0,
+ "1354": 110.0,
+ "1355": 116.0,
+ "1356": 122.0,
+ "1357": 96.0,
+ "1358": 114.0,
+ "1359": 107.0,
+ "1360": 94.0,
+ "1361": 110.0,
+ "1362": 111.0,
+ "1363": 117.0,
+ "1364": 109.0,
+ "1365": 111.0,
+ "1366": 101.0,
+ "1367": 99.0,
+ "1368": 107.0,
+ "1369": 102.0,
+ "1370": 124.0,
+ "1371": 95.0,
+ "1372": 100.0,
+ "1373": 119.0,
+ "1374": 104.0,
+ "1375": 101.0,
+ "1376": 117.0,
+ "1377": 120.0,
+ "1378": 109.0,
+ "1379": 91.0,
+ "1380": 100.0,
+ "1381": 99.0,
+ "1382": 108.0,
+ "1383": 121.0,
+ "1384": 118.0,
+ "1385": 105.0,
+ "1386": 106.0,
+ "1387": 113.0,
+ "1388": 90.0,
+ "1389": 100.0,
+ "1390": 111.0,
+ "1391": 110.0,
+ "1392": 83.0,
+ "1393": 94.0,
+ "1394": 89.0,
+ "1395": 106.0,
+ "1396": 95.0,
+ "1397": 93.0,
+ "1398": 98.0,
+ "1399": 105.0,
+ "1400": 105.0,
+ "1401": 92.0,
+ "1402": 89.0,
+ "1403": 134.0,
+ "1404": 101.0,
+ "1405": 117.0,
+ "1406": 123.0,
+ "1407": 116.0,
+ "1408": 91.0,
+ "1409": 110.0,
+ "1410": 116.0,
+ "1411": 103.0,
+ "1412": 99.0,
+ "1413": 107.0,
+ "1414": 106.0,
+ "1415": 109.0,
+ "1416": 105.0,
+ "1417": 93.0,
+ "1418": 108.0,
+ "1419": 97.0,
+ "1420": 119.0,
+ "1421": 115.0,
+ "1422": 105.0,
+ "1423": 81.0,
+ "1424": 107.0,
+ "1425": 126.0,
+ "1426": 100.0,
+ "1427": 82.0,
+ "1428": 109.0,
+ "1429": 104.0,
+ "1430": 104.0,
+ "1431": 120.0,
+ "1432": 97.0,
+ "1433": 114.0,
+ "1434": 93.0,
+ "1435": 130.0,
+ "1436": 104.0,
+ "1437": 132.0,
+ "1438": 86.0,
+ "1439": 102.0,
+ "1440": 90.0,
+ "1441": 117.0,
+ "1442": 108.0,
+ "1443": 105.0,
+ "1444": 108.0,
+ "1445": 93.0,
+ "1446": 100.0,
+ "1447": 98.0,
+ "1448": 112.0,
+ "1449": 110.0,
+ "1450": 108.0,
+ "1451": 105.0,
+ "1452": 93.0,
+ "1453": 115.0,
+ "1454": 100.0,
+ "1455": 98.0,
+ "1456": 76.0,
+ "1457": 113.0,
+ "1458": 99.0,
+ "1459": 123.0,
+ "1460": 87.0,
+ "1461": 101.0,
+ "1462": 85.0,
+ "1463": 113.0,
+ "1464": 114.0,
+ "1465": 106.0,
+ "1466": 119.0,
+ "1467": 118.0,
+ "1468": 90.0,
+ "1469": 102.0,
+ "1470": 117.0,
+ "1471": 117.0,
+ "1472": 102.0,
+ "1473": 110.0,
+ "1474": 95.0,
+ "1475": 108.0,
+ "1476": 102.0,
+ "1477": 129.0,
+ "1478": 87.0,
+ "1479": 114.0,
+ "1480": 71.0,
+ "1481": 104.0,
+ "1482": 106.0,
+ "1483": 92.0,
+ "1484": 103.0,
+ "1485": 120.0,
+ "1486": 106.0,
+ "1487": 110.0,
+ "1488": 113.0,
+ "1489": 93.0,
+ "1490": 116.0,
+ "1491": 110.0,
+ "1492": 93.0,
+ "1493": 100.0,
+ "1494": 132.0,
+ "1495": 101.0,
+ "1496": 118.0,
+ "1497": 108.0,
+ "1498": 117.0,
+ "1499": 91.0,
+ "1500": 111.0,
+ "1501": 119.0,
+ "1502": 128.0,
+ "1503": 94.0,
+ "1504": 105.0,
+ "1505": 106.0,
+ "1506": 118.0,
+ "1507": 100.0,
+ "1508": 95.0,
+ "1509": 126.0,
+ "1510": 95.0,
+ "1511": 123.0,
+ "1512": 95.0,
+ "1513": 109.0,
+ "1514": 79.0,
+ "1515": 112.0,
+ "1516": 121.0,
+ "1517": 112.0,
+ "1518": 93.0,
+ "1519": 95.0,
+ "1520": 113.0,
+ "1521": 98.0,
+ "1522": 106.0,
+ "1523": 110.0,
+ "1524": 62.0,
+ "1525": 97.0,
+ "1526": 102.0,
+ "1527": 83.0,
+ "1528": 120.0,
+ "1529": 99.0,
+ "1530": 110.0,
+ "1531": 106.0,
+ "1532": 101.0,
+ "1533": 123.0,
+ "1534": 113.0,
+ "1535": 108.0,
+ "1536": 100.0,
+ "1537": 101.0,
+ "1538": 115.0,
+ "1539": 109.0,
+ "1540": 112.0,
+ "1541": 117.0,
+ "1542": 99.0,
+ "1543": 104.0,
+ "1544": 100.0,
+ "1545": 105.0,
+ "1546": 103.0,
+ "1547": 93.0,
+ "1548": 92.0,
+ "1549": 105.0,
+ "1550": 111.0,
+ "1551": 113.0,
+ "1552": 112.0,
+ "1553": 119.0,
+ "1554": 113.0,
+ "1555": 100.0,
+ "1556": 123.0,
+ "1557": 105.0,
+ "1558": 114.0,
+ "1559": 99.0,
+ "1560": 97.0,
+ "1561": 115.0,
+ "1562": 114.0,
+ "1563": 105.0,
+ "1564": 107.0,
+ "1565": 113.0,
+ "1566": 110.0,
+ "1567": 107.0,
+ "1568": 103.0,
+ "1569": 105.0,
+ "1570": 117.0,
+ "1571": 87.0,
+ "1572": 102.0,
+ "1573": 133.0,
+ "1574": 85.0,
+ "1575": 108.0,
+ "1576": 89.0,
+ "1577": 102.0,
+ "1578": 98.0,
+ "1579": 115.0,
+ "1580": 98.0,
+ "1581": 103.0,
+ "1582": 97.0,
+ "1583": 111.0,
+ "1584": 97.0,
+ "1585": 116.0,
+ "1586": 118.0,
+ "1587": 126.0,
+ "1588": 113.0,
+ "1589": 106.0,
+ "1590": 119.0,
+ "1591": 122.0,
+ "1592": 116.0,
+ "1593": 112.0,
+ "1594": 122.0,
+ "1595": 124.0,
+ "1596": 103.0,
+ "1597": 104.0,
+ "1598": 115.0,
+ "1599": 109.0,
+ "1600": 100.0,
+ "1601": 131.0,
+ "1602": 94.0,
+ "1603": 102.0,
+ "1604": 108.0,
+ "1605": 113.0,
+ "1606": 110.0,
+ "1607": 108.0,
+ "1608": 119.0,
+ "1609": 115.0,
+ "1610": 113.0,
+ "1611": 107.0,
+ "1612": 103.0,
+ "1613": 111.0,
+ "1614": 108.0,
+ "1615": 126.0,
+ "1616": 99.0,
+ "1617": 92.0,
+ "1618": 95.0,
+ "1619": 112.0,
+ "1620": 120.0,
+ "1621": 94.0,
+ "1622": 117.0,
+ "1623": 106.0,
+ "1624": 134.0,
+ "1625": 105.0,
+ "1626": 117.0,
+ "1627": 125.0,
+ "1628": 126.0,
+ "1629": 124.0,
+ "1630": 94.0,
+ "1631": 93.0,
+ "1632": 107.0,
+ "1633": 111.0,
+ "1634": 124.0,
+ "1635": 115.0,
+ "1636": 110.0,
+ "1637": 100.0,
+ "1638": 91.0,
+ "1639": 115.0,
+ "1640": 101.0,
+ "1641": 108.0,
+ "1642": 101.0,
+ "1643": 117.0,
+ "1644": 121.0,
+ "1645": 109.0,
+ "1646": 107.0,
+ "1647": 123.0,
+ "1648": 107.0,
+ "1649": 81.0,
+ "1650": 115.0,
+ "1651": 102.0,
+ "1652": 110.0,
+ "1653": 100.0,
+ "1654": 128.0,
+ "1655": 104.0,
+ "1656": 110.0,
+ "1657": 100.0,
+ "1658": 110.0,
+ "1659": 117.0,
+ "1660": 119.0,
+ "1661": 114.0,
+ "1662": 92.0,
+ "1663": 115.0,
+ "1664": 108.0,
+ "1665": 114.0,
+ "1666": 97.0,
+ "1667": 106.0,
+ "1668": 110.0,
+ "1669": 102.0,
+ "1670": 118.0,
+ "1671": 104.0,
+ "1672": 126.0,
+ "1673": 105.0,
+ "1674": 123.0,
+ "1675": 94.0,
+ "1676": 126.0,
+ "1677": 97.0,
+ "1678": 100.0,
+ "1679": 95.0,
+ "1680": 107.0,
+ "1681": 124.0,
+ "1682": 108.0,
+ "1683": 125.0,
+ "1684": 119.0,
+ "1685": 125.0,
+ "1686": 119.0,
+ "1687": 99.0,
+ "1688": 119.0,
+ "1689": 105.0,
+ "1690": 100.0,
+ "1691": 120.0,
+ "1692": 93.0,
+ "1693": 111.0,
+ "1694": 104.0,
+ "1695": 133.0,
+ "1696": 101.0,
+ "1697": 118.0,
+ "1698": 139.0,
+ "1699": 120.0,
+ "1700": 114.0,
+ "1701": 108.0,
+ "1702": 103.0,
+ "1703": 100.0,
+ "1704": 97.0,
+ "1705": 104.0,
+ "1706": 107.0,
+ "1707": 88.0,
+ "1708": 121.0,
+ "1709": 89.0,
+ "1710": 130.0,
+ "1711": 96.0,
+ "1712": 113.0,
+ "1713": 112.0,
+ "1714": 115.0,
+ "1715": 133.0,
+ "1716": 85.0,
+ "1717": 119.0,
+ "1718": 131.0,
+ "1719": 104.0,
+ "1720": 138.0,
+ "1721": 105.0,
+ "1722": 114.0,
+ "1723": 85.0,
+ "1724": 120.0,
+ "1725": 103.0,
+ "1726": 108.0,
+ "1727": 117.0,
+ "1728": 100.0,
+ "1729": 119.0,
+ "1730": 123.0,
+ "1731": 109.0,
+ "1732": 123.0,
+ "1733": 105.0,
+ "1734": 95.0,
+ "1735": 122.0,
+ "1736": 106.0,
+ "1737": 98.0,
+ "1738": 105.0,
+ "1739": 117.0,
+ "1740": 112.0,
+ "1741": 126.0,
+ "1742": 131.0,
+ "1743": 111.0,
+ "1744": 116.0,
+ "1745": 141.0,
+ "1746": 118.0,
+ "1747": 107.0,
+ "1748": 99.0,
+ "1749": 108.0,
+ "1750": 97.0,
+ "1751": 109.0,
+ "1752": 101.0,
+ "1753": 101.0,
+ "1754": 120.0,
+ "1755": 122.0,
+ "1756": 111.0,
+ "1757": 127.0,
+ "1758": 118.0,
+ "1759": 114.0,
+ "1760": 112.0,
+ "1761": 126.0,
+ "1762": 117.0,
+ "1763": 124.0,
+ "1764": 116.0,
+ "1765": 117.0,
+ "1766": 98.0,
+ "1767": 125.0,
+ "1768": 99.0,
+ "1769": 85.0,
+ "1770": 101.0,
+ "1771": 125.0,
+ "1772": 120.0,
+ "1773": 118.0,
+ "1774": 117.0,
+ "1775": 85.0,
+ "1776": 118.0,
+ "1777": 105.0,
+ "1778": 114.0,
+ "1779": 107.0,
+ "1780": 106.0,
+ "1781": 123.0,
+ "1782": 125.0,
+ "1783": 124.0,
+ "1784": 100.0,
+ "1785": 121.0,
+ "1786": 112.0,
+ "1787": 109.0,
+ "1788": 106.0,
+ "1789": 101.0,
+ "1790": 115.0,
+ "1791": 122.0,
+ "1792": 91.0,
+ "1793": 120.0,
+ "1794": 135.0,
+ "1795": 113.0,
+ "1796": 126.0,
+ "1797": 101.0,
+ "1798": 88.0,
+ "1799": 98.0,
+ "1800": 100.0,
+ "1801": 108.0,
+ "1802": 107.0,
+ "1803": 113.0,
+ "1804": 103.0,
+ "1805": 136.0,
+ "1806": 129.0,
+ "1807": 107.0,
+ "1808": 83.0,
+ "1809": 110.0,
+ "1810": 113.0,
+ "1811": 112.0,
+ "1812": 110.0,
+ "1813": 97.0,
+ "1814": 147.0,
+ "1815": 103.0,
+ "1816": 115.0,
+ "1817": 117.0,
+ "1818": 131.0,
+ "1819": 110.0,
+ "1820": 114.0,
+ "1821": 129.0,
+ "1822": 126.0,
+ "1823": 102.0,
+ "1824": 78.0,
+ "1825": 106.0,
+ "1826": 116.0,
+ "1827": 114.0,
+ "1828": 99.0,
+ "1829": 116.0,
+ "1830": 122.0,
+ "1831": 94.0,
+ "1832": 109.0,
+ "1833": 118.0,
+ "1834": 120.0,
+ "1835": 121.0,
+ "1836": 112.0,
+ "1837": 117.0,
+ "1838": 117.0,
+ "1839": 102.0,
+ "1840": 132.0,
+ "1841": 108.0,
+ "1842": 112.0,
+ "1843": 114.0,
+ "1844": 102.0,
+ "1845": 116.0,
+ "1846": 119.0,
+ "1847": 117.0,
+ "1848": 105.0,
+ "1849": 88.0,
+ "1850": 103.0,
+ "1851": 106.0,
+ "1852": 117.0,
+ "1853": 112.0,
+ "1854": 107.0,
+ "1855": 83.0,
+ "1856": 139.0,
+ "1857": 102.0,
+ "1858": 104.0,
+ "1859": 109.0,
+ "1860": 95.0,
+ "1861": 113.0,
+ "1862": 100.0,
+ "1863": 105.0,
+ "1864": 103.0,
+ "1865": 112.0,
+ "1866": 106.0,
+ "1867": 111.0,
+ "1868": 111.0,
+ "1869": 105.0,
+ "1870": 97.0,
+ "1871": 111.0,
+ "1872": 99.0,
+ "1873": 108.0,
+ "1874": 118.0,
+ "1875": 118.0,
+ "1876": 104.0,
+ "1877": 126.0,
+ "1878": 106.0,
+ "1879": 96.0,
+ "1880": 106.0,
+ "1881": 97.0,
+ "1882": 117.0,
+ "1883": 109.0,
+ "1884": 124.0,
+ "1885": 106.0,
+ "1886": 109.0,
+ "1887": 136.0,
+ "1888": 128.0,
+ "1889": 118.0,
+ "1890": 115.0,
+ "1891": 115.0,
+ "1892": 123.0,
+ "1893": 112.0,
+ "1894": 129.0,
+ "1895": 114.0,
+ "1896": 107.0,
+ "1897": 109.0,
+ "1898": 101.0,
+ "1899": 115.0,
+ "1900": 108.0,
+ "1901": 124.0,
+ "1902": 106.0,
+ "1903": 121.0,
+ "1904": 108.0,
+ "1905": 99.0,
+ "1906": 116.0,
+ "1907": 109.0,
+ "1908": 104.0,
+ "1909": 115.0,
+ "1910": 92.0,
+ "1911": 129.0,
+ "1912": 118.0,
+ "1913": 119.0,
+ "1914": 114.0,
+ "1915": 105.0,
+ "1916": 147.0,
+ "1917": 128.0,
+ "1918": 113.0,
+ "1919": 121.0,
+ "1920": 100.0,
+ "1921": 116.0,
+ "1922": 104.0,
+ "1923": 95.0,
+ "1924": 117.0,
+ "1925": 123.0,
+ "1926": 117.0,
+ "1927": 100.0,
+ "1928": 113.0,
+ "1929": 118.0,
+ "1930": 106.0,
+ "1931": 107.0,
+ "1932": 114.0,
+ "1933": 104.0,
+ "1934": 107.0,
+ "1935": 86.0,
+ "1936": 118.0,
+ "1937": 112.0,
+ "1938": 89.0,
+ "1939": 116.0,
+ "1940": 110.0,
+ "1941": 111.0,
+ "1942": 121.0,
+ "1943": 121.0,
+ "1944": 116.0,
+ "1945": 92.0,
+ "1946": 113.0,
+ "1947": 122.0,
+ "1948": 119.0,
+ "1949": 123.0,
+ "1950": 114.0,
+ "1951": 104.0,
+ "1952": 121.0,
+ "1953": 103.0,
+ "1954": 124.0,
+ "1955": 124.0,
+ "1956": 106.0,
+ "1957": 103.0,
+ "1958": 120.0,
+ "1959": 107.0,
+ "1960": 115.0,
+ "1961": 108.0,
+ "1962": 105.0,
+ "1963": 127.0,
+ "1964": 112.0,
+ "1965": 89.0,
+ "1966": 109.0,
+ "1967": 106.0,
+ "1968": 109.0,
+ "1969": 106.0,
+ "1970": 106.0,
+ "1971": 111.0,
+ "1972": 92.0,
+ "1973": 94.0,
+ "1974": 131.0,
+ "1975": 97.0,
+ "1976": 119.0,
+ "1977": 99.0,
+ "1978": 131.0,
+ "1979": 118.0,
+ "1980": 122.0,
+ "1981": 106.0,
+ "1982": 129.0,
+ "1983": 119.0,
+ "1984": 108.0,
+ "1985": 137.0,
+ "1986": 106.0,
+ "1987": 113.0,
+ "1988": 104.0,
+ "1989": 93.0,
+ "1990": 120.0,
+ "1991": 110.0,
+ "1992": 105.0,
+ "1993": 126.0,
+ "1994": 128.0,
+ "1995": 106.0,
+ "1996": 135.0,
+ "1997": 127.0,
+ "1998": 111.0,
+ "1999": 111.0,
+ "2000": 101.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 476473856.0,
+ "2": 476473856.0,
+ "3": 476473856.0,
+ "4": 476473856.0,
+ "5": 476473856.0,
+ "6": 476473856.0,
+ "7": 476473856.0,
+ "8": 476473856.0,
+ "9": 476473856.0,
+ "10": 476473856.0,
+ "11": 476473856.0,
+ "12": 476473856.0,
+ "13": 476473856.0,
+ "14": 476473856.0,
+ "15": 476473856.0,
+ "16": 476473856.0,
+ "17": 476473856.0,
+ "18": 476473856.0,
+ "19": 476473856.0,
+ "20": 476473856.0,
+ "21": 476473856.0,
+ "22": 476473856.0,
+ "23": 476473856.0,
+ "24": 476473856.0,
+ "25": 476473856.0,
+ "26": 476473856.0,
+ "27": 476473856.0,
+ "28": 476473856.0,
+ "29": 476473856.0,
+ "30": 476473856.0,
+ "31": 476473856.0,
+ "32": 476473856.0,
+ "33": 476473856.0,
+ "34": 476473856.0,
+ "35": 476473856.0,
+ "36": 476473856.0,
+ "37": 476473856.0,
+ "38": 476473856.0,
+ "39": 476473856.0,
+ "40": 476473856.0,
+ "41": 476473856.0,
+ "42": 476473856.0,
+ "43": 476473856.0,
+ "44": 476473856.0,
+ "45": 476473856.0,
+ "46": 476473856.0,
+ "47": 476473856.0,
+ "48": 476473856.0,
+ "49": 476473856.0,
+ "50": 476473856.0,
+ "51": 476473856.0,
+ "52": 476473856.0,
+ "53": 476473856.0,
+ "54": 476473856.0,
+ "55": 476473856.0,
+ "56": 476473856.0,
+ "57": 476473856.0,
+ "58": 476473856.0,
+ "59": 476473856.0,
+ "60": 476473856.0,
+ "61": 476473856.0,
+ "62": 476473856.0,
+ "63": 476473856.0,
+ "64": 476473856.0,
+ "65": 476473856.0,
+ "66": 476473856.0,
+ "67": 476473856.0,
+ "68": 476473856.0,
+ "69": 476473856.0,
+ "70": 476473856.0,
+ "71": 476473856.0,
+ "72": 476473856.0,
+ "73": 476473856.0,
+ "74": 476473856.0,
+ "75": 476473856.0,
+ "76": 476473856.0,
+ "77": 476473856.0,
+ "78": 476473856.0,
+ "79": 476473856.0,
+ "80": 476473856.0,
+ "81": 476473856.0,
+ "82": 476473856.0,
+ "83": 476473856.0,
+ "84": 476473856.0,
+ "85": 476473856.0,
+ "86": 476473856.0,
+ "87": 476473856.0,
+ "88": 476473856.0,
+ "89": 476473856.0,
+ "90": 476473856.0,
+ "91": 476473856.0,
+ "92": 476473856.0,
+ "93": 476473856.0,
+ "94": 476473856.0,
+ "95": 476473856.0,
+ "96": 476473856.0,
+ "97": 476473856.0,
+ "98": 476473856.0,
+ "99": 476473856.0,
+ "100": 476473856.0,
+ "101": 476473856.0,
+ "102": 476473856.0,
+ "103": 476473856.0,
+ "104": 476473856.0,
+ "105": 476473856.0,
+ "106": 476473856.0,
+ "107": 476473856.0,
+ "108": 476473856.0,
+ "109": 476473856.0,
+ "110": 476473856.0,
+ "111": 476473856.0,
+ "112": 476473856.0,
+ "113": 476473856.0,
+ "114": 476473856.0,
+ "115": 476473856.0,
+ "116": 476473856.0,
+ "117": 476473856.0,
+ "118": 476473856.0,
+ "119": 476473856.0,
+ "120": 476473856.0,
+ "121": 476473856.0,
+ "122": 476473856.0,
+ "123": 476473856.0,
+ "124": 476473856.0,
+ "125": 476473856.0,
+ "126": 476473856.0,
+ "127": 476473856.0,
+ "128": 476473856.0,
+ "129": 476473856.0,
+ "130": 476473856.0,
+ "131": 476473856.0,
+ "132": 476473856.0,
+ "133": 476473856.0,
+ "134": 476473856.0,
+ "135": 476473856.0,
+ "136": 476473856.0,
+ "137": 476473856.0,
+ "138": 476473856.0,
+ "139": 476473856.0,
+ "140": 476473856.0,
+ "141": 476473856.0,
+ "142": 476473856.0,
+ "143": 476473856.0,
+ "144": 476473856.0,
+ "145": 476473856.0,
+ "146": 476473856.0,
+ "147": 476473856.0,
+ "148": 476473856.0,
+ "149": 476473856.0,
+ "150": 476473856.0,
+ "151": 476473856.0,
+ "152": 476473856.0,
+ "153": 476473856.0,
+ "154": 476473856.0,
+ "155": 476473856.0,
+ "156": 476473856.0,
+ "157": 476473856.0,
+ "158": 476473856.0,
+ "159": 476473856.0,
+ "160": 476473856.0,
+ "161": 476473856.0,
+ "162": 476473856.0,
+ "163": 476473856.0,
+ "164": 476473856.0,
+ "165": 476473856.0,
+ "166": 476473856.0,
+ "167": 476473856.0,
+ "168": 476473856.0,
+ "169": 476473856.0,
+ "170": 476473856.0,
+ "171": 476473856.0,
+ "172": 476473856.0,
+ "173": 476473856.0,
+ "174": 476473856.0,
+ "175": 476473856.0,
+ "176": 476473856.0,
+ "177": 476473856.0,
+ "178": 476473856.0,
+ "179": 476473856.0,
+ "180": 476473856.0,
+ "181": 476473856.0,
+ "182": 476473856.0,
+ "183": 476473856.0,
+ "184": 476473856.0,
+ "185": 476473856.0,
+ "186": 476473856.0,
+ "187": 476473856.0,
+ "188": 476473856.0,
+ "189": 476473856.0,
+ "190": 476473856.0,
+ "191": 476473856.0,
+ "192": 476473856.0,
+ "193": 476473856.0,
+ "194": 476473856.0,
+ "195": 476473856.0,
+ "196": 476473856.0,
+ "197": 476473856.0,
+ "198": 476473856.0,
+ "199": 476473856.0,
+ "200": 476473856.0,
+ "201": 476473856.0,
+ "202": 476473856.0,
+ "203": 476473856.0,
+ "204": 476473856.0,
+ "205": 476473856.0,
+ "206": 476473856.0,
+ "207": 476473856.0,
+ "208": 476473856.0,
+ "209": 476473856.0,
+ "210": 476473856.0,
+ "211": 476473856.0,
+ "212": 476473856.0,
+ "213": 476473856.0,
+ "214": 476473856.0,
+ "215": 476473856.0,
+ "216": 476473856.0,
+ "217": 476473856.0,
+ "218": 476473856.0,
+ "219": 476473856.0,
+ "220": 476473856.0,
+ "221": 476473856.0,
+ "222": 476473856.0,
+ "223": 476473856.0,
+ "224": 476473856.0,
+ "225": 476473856.0,
+ "226": 476473856.0,
+ "227": 476473856.0,
+ "228": 476473856.0,
+ "229": 476473856.0,
+ "230": 476473856.0,
+ "231": 476473856.0,
+ "232": 476473856.0,
+ "233": 476473856.0,
+ "234": 476473856.0,
+ "235": 476473856.0,
+ "236": 476473856.0,
+ "237": 476473856.0,
+ "238": 476473856.0,
+ "239": 476473856.0,
+ "240": 476473856.0,
+ "241": 476473856.0,
+ "242": 476473856.0,
+ "243": 476473856.0,
+ "244": 476473856.0,
+ "245": 476473856.0,
+ "246": 476473856.0,
+ "247": 476473856.0,
+ "248": 476473856.0,
+ "249": 476473856.0,
+ "250": 476473856.0,
+ "251": 476473856.0,
+ "252": 476473856.0,
+ "253": 476473856.0,
+ "254": 476473856.0,
+ "255": 476473856.0,
+ "256": 476473856.0,
+ "257": 476473856.0,
+ "258": 476473856.0,
+ "259": 476473856.0,
+ "260": 476473856.0,
+ "261": 476473856.0,
+ "262": 476473856.0,
+ "263": 476473856.0,
+ "264": 476473856.0,
+ "265": 476473856.0,
+ "266": 476473856.0,
+ "267": 476473856.0,
+ "268": 476473856.0,
+ "269": 476473856.0,
+ "270": 476473856.0,
+ "271": 476473856.0,
+ "272": 476473856.0,
+ "273": 476473856.0,
+ "274": 476473856.0,
+ "275": 476473856.0,
+ "276": 476473856.0,
+ "277": 476473856.0,
+ "278": 476473856.0,
+ "279": 476473856.0,
+ "280": 476473856.0,
+ "281": 476473856.0,
+ "282": 476473856.0,
+ "283": 476473856.0,
+ "284": 476473856.0,
+ "285": 476473856.0,
+ "286": 476473856.0,
+ "287": 476473856.0,
+ "288": 476473856.0,
+ "289": 476473856.0,
+ "290": 476473856.0,
+ "291": 476473856.0,
+ "292": 476473856.0,
+ "293": 476473856.0,
+ "294": 476473856.0,
+ "295": 476473856.0,
+ "296": 476473856.0,
+ "297": 476473856.0,
+ "298": 476473856.0,
+ "299": 476473856.0,
+ "300": 476473856.0,
+ "301": 476473856.0,
+ "302": 476473856.0,
+ "303": 476473856.0,
+ "304": 476473856.0,
+ "305": 476473856.0,
+ "306": 476473856.0,
+ "307": 476473856.0,
+ "308": 476473856.0,
+ "309": 476473856.0,
+ "310": 476473856.0,
+ "311": 476473856.0,
+ "312": 476473856.0,
+ "313": 476473856.0,
+ "314": 476473856.0,
+ "315": 476473856.0,
+ "316": 476473856.0,
+ "317": 476473856.0,
+ "318": 476473856.0,
+ "319": 476473856.0,
+ "320": 476473856.0,
+ "321": 476473856.0,
+ "322": 476473856.0,
+ "323": 476473856.0,
+ "324": 476473856.0,
+ "325": 476473856.0,
+ "326": 476473856.0,
+ "327": 476473856.0,
+ "328": 476473856.0,
+ "329": 476473856.0,
+ "330": 476473856.0,
+ "331": 476473856.0,
+ "332": 476473856.0,
+ "333": 476473856.0,
+ "334": 476473856.0,
+ "335": 476473856.0,
+ "336": 476473856.0,
+ "337": 476473856.0,
+ "338": 476473856.0,
+ "339": 476473856.0,
+ "340": 476473856.0,
+ "341": 476473856.0,
+ "342": 476473856.0,
+ "343": 476473856.0,
+ "344": 476473856.0,
+ "345": 476473856.0,
+ "346": 476473856.0,
+ "347": 476473856.0,
+ "348": 476473856.0,
+ "349": 476473856.0,
+ "350": 476473856.0,
+ "351": 476473856.0,
+ "352": 476473856.0,
+ "353": 476473856.0,
+ "354": 476473856.0,
+ "355": 476473856.0,
+ "356": 476473856.0,
+ "357": 476473856.0,
+ "358": 476473856.0,
+ "359": 476473856.0,
+ "360": 476473856.0,
+ "361": 476473856.0,
+ "362": 476473856.0,
+ "363": 476473856.0,
+ "364": 476473856.0,
+ "365": 476473856.0,
+ "366": 476473856.0,
+ "367": 476473856.0,
+ "368": 476473856.0,
+ "369": 476473856.0,
+ "370": 476473856.0,
+ "371": 476473856.0,
+ "372": 476473856.0,
+ "373": 476473856.0,
+ "374": 476473856.0,
+ "375": 476473856.0,
+ "376": 476473856.0,
+ "377": 476473856.0,
+ "378": 476473856.0,
+ "379": 476473856.0,
+ "380": 476473856.0,
+ "381": 476473856.0,
+ "382": 476473856.0,
+ "383": 476473856.0,
+ "384": 476473856.0,
+ "385": 476473856.0,
+ "386": 476473856.0,
+ "387": 476473856.0,
+ "388": 476473856.0,
+ "389": 476473856.0,
+ "390": 476473856.0,
+ "391": 476473856.0,
+ "392": 476473856.0,
+ "393": 476473856.0,
+ "394": 476473856.0,
+ "395": 476473856.0,
+ "396": 476473856.0,
+ "397": 476473856.0,
+ "398": 476473856.0,
+ "399": 476473856.0,
+ "400": 476473856.0,
+ "401": 476473856.0,
+ "402": 476473856.0,
+ "403": 476473856.0,
+ "404": 476473856.0,
+ "405": 476473856.0,
+ "406": 476473856.0,
+ "407": 476473856.0,
+ "408": 476473856.0,
+ "409": 476473856.0,
+ "410": 476473856.0,
+ "411": 476473856.0,
+ "412": 476473856.0,
+ "413": 476473856.0,
+ "414": 476473856.0,
+ "415": 476473856.0,
+ "416": 476473856.0,
+ "417": 476473856.0,
+ "418": 476473856.0,
+ "419": 476473856.0,
+ "420": 476473856.0,
+ "421": 476473856.0,
+ "422": 476473856.0,
+ "423": 476473856.0,
+ "424": 476473856.0,
+ "425": 476473856.0,
+ "426": 476473856.0,
+ "427": 476473856.0,
+ "428": 476473856.0,
+ "429": 476473856.0,
+ "430": 476473856.0,
+ "431": 476473856.0,
+ "432": 476473856.0,
+ "433": 476473856.0,
+ "434": 476473856.0,
+ "435": 476473856.0,
+ "436": 476473856.0,
+ "437": 476473856.0,
+ "438": 476473856.0,
+ "439": 476473856.0,
+ "440": 476473856.0,
+ "441": 476473856.0,
+ "442": 476473856.0,
+ "443": 476473856.0,
+ "444": 476473856.0,
+ "445": 476473856.0,
+ "446": 476473856.0,
+ "447": 476473856.0,
+ "448": 476473856.0,
+ "449": 476473856.0,
+ "450": 476473856.0,
+ "451": 476473856.0,
+ "452": 476473856.0,
+ "453": 476473856.0,
+ "454": 476473856.0,
+ "455": 476473856.0,
+ "456": 476473856.0,
+ "457": 476473856.0,
+ "458": 476473856.0,
+ "459": 476473856.0,
+ "460": 476473856.0,
+ "461": 476473856.0,
+ "462": 476473856.0,
+ "463": 476473856.0,
+ "464": 476473856.0,
+ "465": 476473856.0,
+ "466": 476473856.0,
+ "467": 476473856.0,
+ "468": 476473856.0,
+ "469": 476473856.0,
+ "470": 476473856.0,
+ "471": 476473856.0,
+ "472": 476473856.0,
+ "473": 476473856.0,
+ "474": 476473856.0,
+ "475": 476473856.0,
+ "476": 476473856.0,
+ "477": 476473856.0,
+ "478": 476473856.0,
+ "479": 476473856.0,
+ "480": 476473856.0,
+ "481": 476473856.0,
+ "482": 476473856.0,
+ "483": 476473856.0,
+ "484": 476473856.0,
+ "485": 476473856.0,
+ "486": 476473856.0,
+ "487": 476473856.0,
+ "488": 476473856.0,
+ "489": 476473856.0,
+ "490": 476473856.0,
+ "491": 476473856.0,
+ "492": 476473856.0,
+ "493": 476473856.0,
+ "494": 476473856.0,
+ "495": 476473856.0,
+ "496": 476473856.0,
+ "497": 476473856.0,
+ "498": 476473856.0,
+ "499": 476473856.0,
+ "500": 476473856.0,
+ "501": 476473856.0,
+ "502": 476473856.0,
+ "503": 476473856.0,
+ "504": 476473856.0,
+ "505": 476473856.0,
+ "506": 476473856.0,
+ "507": 476473856.0,
+ "508": 476473856.0,
+ "509": 476473856.0,
+ "510": 476473856.0,
+ "511": 476473856.0,
+ "512": 476473856.0,
+ "513": 476473856.0,
+ "514": 476473856.0,
+ "515": 476473856.0,
+ "516": 476473856.0,
+ "517": 476473856.0,
+ "518": 476473856.0,
+ "519": 476473856.0,
+ "520": 476473856.0,
+ "521": 476473856.0,
+ "522": 476473856.0,
+ "523": 476473856.0,
+ "524": 476473856.0,
+ "525": 476473856.0,
+ "526": 476473856.0,
+ "527": 476473856.0,
+ "528": 476473856.0,
+ "529": 476473856.0,
+ "530": 476473856.0,
+ "531": 476473856.0,
+ "532": 476473856.0,
+ "533": 476473856.0,
+ "534": 476473856.0,
+ "535": 476473856.0,
+ "536": 476473856.0,
+ "537": 476473856.0,
+ "538": 476473856.0,
+ "539": 476473856.0,
+ "540": 476473856.0,
+ "541": 476473856.0,
+ "542": 476473856.0,
+ "543": 476473856.0,
+ "544": 476473856.0,
+ "545": 476473856.0,
+ "546": 476473856.0,
+ "547": 476473856.0,
+ "548": 476473856.0,
+ "549": 476473856.0,
+ "550": 476473856.0,
+ "551": 476473856.0,
+ "552": 476473856.0,
+ "553": 476473856.0,
+ "554": 476473856.0,
+ "555": 476473856.0,
+ "556": 476473856.0,
+ "557": 476473856.0,
+ "558": 476473856.0,
+ "559": 476473856.0,
+ "560": 476473856.0,
+ "561": 476473856.0,
+ "562": 476473856.0,
+ "563": 476473856.0,
+ "564": 476473856.0,
+ "565": 476473856.0,
+ "566": 476473856.0,
+ "567": 476473856.0,
+ "568": 476473856.0,
+ "569": 476473856.0,
+ "570": 476473856.0,
+ "571": 476473856.0,
+ "572": 476473856.0,
+ "573": 476473856.0,
+ "574": 476473856.0,
+ "575": 476473856.0,
+ "576": 476473856.0,
+ "577": 476473856.0,
+ "578": 476473856.0,
+ "579": 476473856.0,
+ "580": 476473856.0,
+ "581": 476473856.0,
+ "582": 476473856.0,
+ "583": 476473856.0,
+ "584": 476473856.0,
+ "585": 476473856.0,
+ "586": 476473856.0,
+ "587": 476473856.0,
+ "588": 476473856.0,
+ "589": 476473856.0,
+ "590": 476473856.0,
+ "591": 476473856.0,
+ "592": 476473856.0,
+ "593": 476473856.0,
+ "594": 476473856.0,
+ "595": 476473856.0,
+ "596": 476473856.0,
+ "597": 476473856.0,
+ "598": 476473856.0,
+ "599": 476473856.0,
+ "600": 476473856.0,
+ "601": 476473856.0,
+ "602": 476473856.0,
+ "603": 476473856.0,
+ "604": 476473856.0,
+ "605": 476473856.0,
+ "606": 476473856.0,
+ "607": 476473856.0,
+ "608": 476473856.0,
+ "609": 476473856.0,
+ "610": 476473856.0,
+ "611": 476473856.0,
+ "612": 476473856.0,
+ "613": 476473856.0,
+ "614": 476473856.0,
+ "615": 476473856.0,
+ "616": 476473856.0,
+ "617": 476473856.0,
+ "618": 476473856.0,
+ "619": 476473856.0,
+ "620": 476473856.0,
+ "621": 476473856.0,
+ "622": 476473856.0,
+ "623": 476473856.0,
+ "624": 476473856.0,
+ "625": 476473856.0,
+ "626": 476473856.0,
+ "627": 476473856.0,
+ "628": 476473856.0,
+ "629": 476473856.0,
+ "630": 476473856.0,
+ "631": 476473856.0,
+ "632": 476473856.0,
+ "633": 476473856.0,
+ "634": 476473856.0,
+ "635": 476473856.0,
+ "636": 476473856.0,
+ "637": 476473856.0,
+ "638": 476473856.0,
+ "639": 476473856.0,
+ "640": 476473856.0,
+ "641": 476473856.0,
+ "642": 476473856.0,
+ "643": 476473856.0,
+ "644": 476473856.0,
+ "645": 476473856.0,
+ "646": 476473856.0,
+ "647": 476473856.0,
+ "648": 476473856.0,
+ "649": 476473856.0,
+ "650": 476473856.0,
+ "651": 476473856.0,
+ "652": 476473856.0,
+ "653": 476473856.0,
+ "654": 476473856.0,
+ "655": 476473856.0,
+ "656": 476473856.0,
+ "657": 476473856.0,
+ "658": 476473856.0,
+ "659": 476473856.0,
+ "660": 476473856.0,
+ "661": 476473856.0,
+ "662": 476473856.0,
+ "663": 476473856.0,
+ "664": 476473856.0,
+ "665": 476473856.0,
+ "666": 476473856.0,
+ "667": 476473856.0,
+ "668": 476473856.0,
+ "669": 476473856.0,
+ "670": 476473856.0,
+ "671": 476473856.0,
+ "672": 476473856.0,
+ "673": 476473856.0,
+ "674": 476473856.0,
+ "675": 476473856.0,
+ "676": 476473856.0,
+ "677": 476473856.0,
+ "678": 476473856.0,
+ "679": 476473856.0,
+ "680": 476473856.0,
+ "681": 476473856.0,
+ "682": 476473856.0,
+ "683": 476473856.0,
+ "684": 476473856.0,
+ "685": 476473856.0,
+ "686": 476473856.0,
+ "687": 476473856.0,
+ "688": 476473856.0,
+ "689": 476473856.0,
+ "690": 476473856.0,
+ "691": 476473856.0,
+ "692": 476473856.0,
+ "693": 476473856.0,
+ "694": 476473856.0,
+ "695": 476473856.0,
+ "696": 476473856.0,
+ "697": 476473856.0,
+ "698": 476473856.0,
+ "699": 476473856.0,
+ "700": 476473856.0,
+ "701": 476473856.0,
+ "702": 476473856.0,
+ "703": 476473856.0,
+ "704": 476473856.0,
+ "705": 476473856.0,
+ "706": 476473856.0,
+ "707": 476473856.0,
+ "708": 476473856.0,
+ "709": 476473856.0,
+ "710": 476473856.0,
+ "711": 476473856.0,
+ "712": 476473856.0,
+ "713": 476473856.0,
+ "714": 476473856.0,
+ "715": 476473856.0,
+ "716": 476473856.0,
+ "717": 476473856.0,
+ "718": 476473856.0,
+ "719": 476473856.0,
+ "720": 476473856.0,
+ "721": 476473856.0,
+ "722": 476473856.0,
+ "723": 476473856.0,
+ "724": 476473856.0,
+ "725": 476473856.0,
+ "726": 476473856.0,
+ "727": 476473856.0,
+ "728": 476473856.0,
+ "729": 476473856.0,
+ "730": 476473856.0,
+ "731": 476473856.0,
+ "732": 476473856.0,
+ "733": 476473856.0,
+ "734": 476473856.0,
+ "735": 476473856.0,
+ "736": 476473856.0,
+ "737": 476473856.0,
+ "738": 476473856.0,
+ "739": 476473856.0,
+ "740": 476473856.0,
+ "741": 476473856.0,
+ "742": 476473856.0,
+ "743": 476473856.0,
+ "744": 476473856.0,
+ "745": 476473856.0,
+ "746": 476473856.0,
+ "747": 476473856.0,
+ "748": 476473856.0,
+ "749": 476473856.0,
+ "750": 476473856.0,
+ "751": 476473856.0,
+ "752": 476473856.0,
+ "753": 476473856.0,
+ "754": 476473856.0,
+ "755": 476473856.0,
+ "756": 476473856.0,
+ "757": 476473856.0,
+ "758": 476473856.0,
+ "759": 476473856.0,
+ "760": 476473856.0,
+ "761": 476473856.0,
+ "762": 476473856.0,
+ "763": 476473856.0,
+ "764": 476473856.0,
+ "765": 476473856.0,
+ "766": 476473856.0,
+ "767": 476473856.0,
+ "768": 476473856.0,
+ "769": 476473856.0,
+ "770": 476473856.0,
+ "771": 476473856.0,
+ "772": 476473856.0,
+ "773": 476473856.0,
+ "774": 476473856.0,
+ "775": 476473856.0,
+ "776": 476473856.0,
+ "777": 476473856.0,
+ "778": 476473856.0,
+ "779": 476473856.0,
+ "780": 476473856.0,
+ "781": 476473856.0,
+ "782": 476473856.0,
+ "783": 476473856.0,
+ "784": 476473856.0,
+ "785": 476473856.0,
+ "786": 476473856.0,
+ "787": 476473856.0,
+ "788": 476473856.0,
+ "789": 476473856.0,
+ "790": 476473856.0,
+ "791": 476473856.0,
+ "792": 476473856.0,
+ "793": 476473856.0,
+ "794": 476473856.0,
+ "795": 476473856.0,
+ "796": 476473856.0,
+ "797": 476473856.0,
+ "798": 476473856.0,
+ "799": 476473856.0,
+ "800": 476473856.0,
+ "801": 476473856.0,
+ "802": 476473856.0,
+ "803": 476473856.0,
+ "804": 476473856.0,
+ "805": 476473856.0,
+ "806": 476473856.0,
+ "807": 476473856.0,
+ "808": 476473856.0,
+ "809": 476473856.0,
+ "810": 476473856.0,
+ "811": 476473856.0,
+ "812": 476473856.0,
+ "813": 476473856.0,
+ "814": 476473856.0,
+ "815": 476473856.0,
+ "816": 476473856.0,
+ "817": 476473856.0,
+ "818": 476473856.0,
+ "819": 476473856.0,
+ "820": 476473856.0,
+ "821": 476473856.0,
+ "822": 476473856.0,
+ "823": 476473856.0,
+ "824": 476473856.0,
+ "825": 476473856.0,
+ "826": 476473856.0,
+ "827": 476473856.0,
+ "828": 476473856.0,
+ "829": 476473856.0,
+ "830": 476473856.0,
+ "831": 476473856.0,
+ "832": 476473856.0,
+ "833": 476473856.0,
+ "834": 476473856.0,
+ "835": 476473856.0,
+ "836": 476473856.0,
+ "837": 476473856.0,
+ "838": 476473856.0,
+ "839": 476473856.0,
+ "840": 476473856.0,
+ "841": 476473856.0,
+ "842": 476473856.0,
+ "843": 476473856.0,
+ "844": 476473856.0,
+ "845": 476473856.0,
+ "846": 476473856.0,
+ "847": 476473856.0,
+ "848": 476473856.0,
+ "849": 476473856.0,
+ "850": 476473856.0,
+ "851": 476473856.0,
+ "852": 476473856.0,
+ "853": 476473856.0,
+ "854": 476473856.0,
+ "855": 476473856.0,
+ "856": 476473856.0,
+ "857": 476473856.0,
+ "858": 476473856.0,
+ "859": 476473856.0,
+ "860": 476473856.0,
+ "861": 476473856.0,
+ "862": 476473856.0,
+ "863": 476473856.0,
+ "864": 476473856.0,
+ "865": 476473856.0,
+ "866": 476473856.0,
+ "867": 476473856.0,
+ "868": 476473856.0,
+ "869": 476473856.0,
+ "870": 476473856.0,
+ "871": 476473856.0,
+ "872": 476473856.0,
+ "873": 476473856.0,
+ "874": 476473856.0,
+ "875": 476473856.0,
+ "876": 476473856.0,
+ "877": 476473856.0,
+ "878": 476473856.0,
+ "879": 476473856.0,
+ "880": 476473856.0,
+ "881": 476473856.0,
+ "882": 476473856.0,
+ "883": 476473856.0,
+ "884": 476473856.0,
+ "885": 476473856.0,
+ "886": 476473856.0,
+ "887": 476473856.0,
+ "888": 476473856.0,
+ "889": 476473856.0,
+ "890": 476473856.0,
+ "891": 476473856.0,
+ "892": 476473856.0,
+ "893": 476473856.0,
+ "894": 476473856.0,
+ "895": 476473856.0,
+ "896": 476473856.0,
+ "897": 476473856.0,
+ "898": 476473856.0,
+ "899": 476473856.0,
+ "900": 476473856.0,
+ "901": 476473856.0,
+ "902": 476473856.0,
+ "903": 476473856.0,
+ "904": 476473856.0,
+ "905": 476473856.0,
+ "906": 476473856.0,
+ "907": 476473856.0,
+ "908": 476473856.0,
+ "909": 476473856.0,
+ "910": 476473856.0,
+ "911": 476473856.0,
+ "912": 476473856.0,
+ "913": 476473856.0,
+ "914": 476473856.0,
+ "915": 476473856.0,
+ "916": 476473856.0,
+ "917": 476473856.0,
+ "918": 476473856.0,
+ "919": 476473856.0,
+ "920": 476473856.0,
+ "921": 476473856.0,
+ "922": 476473856.0,
+ "923": 476473856.0,
+ "924": 476473856.0,
+ "925": 476473856.0,
+ "926": 476473856.0,
+ "927": 476473856.0,
+ "928": 476473856.0,
+ "929": 476473856.0,
+ "930": 476473856.0,
+ "931": 476473856.0,
+ "932": 476473856.0,
+ "933": 476473856.0,
+ "934": 476473856.0,
+ "935": 476473856.0,
+ "936": 476473856.0,
+ "937": 476473856.0,
+ "938": 476473856.0,
+ "939": 476473856.0,
+ "940": 476473856.0,
+ "941": 476473856.0,
+ "942": 476473856.0,
+ "943": 476473856.0,
+ "944": 476473856.0,
+ "945": 476473856.0,
+ "946": 476473856.0,
+ "947": 476473856.0,
+ "948": 476473856.0,
+ "949": 476473856.0,
+ "950": 476473856.0,
+ "951": 476473856.0,
+ "952": 476473856.0,
+ "953": 476473856.0,
+ "954": 476473856.0,
+ "955": 476473856.0,
+ "956": 476473856.0,
+ "957": 476473856.0,
+ "958": 476473856.0,
+ "959": 476473856.0,
+ "960": 476473856.0,
+ "961": 476473856.0,
+ "962": 476473856.0,
+ "963": 476473856.0,
+ "964": 476473856.0,
+ "965": 476473856.0,
+ "966": 476473856.0,
+ "967": 476473856.0,
+ "968": 476473856.0,
+ "969": 476473856.0,
+ "970": 476473856.0,
+ "971": 476473856.0,
+ "972": 476473856.0,
+ "973": 476473856.0,
+ "974": 476473856.0,
+ "975": 476473856.0,
+ "976": 476473856.0,
+ "977": 476473856.0,
+ "978": 476473856.0,
+ "979": 476473856.0,
+ "980": 476473856.0,
+ "981": 476473856.0,
+ "982": 476473856.0,
+ "983": 476473856.0,
+ "984": 476473856.0,
+ "985": 476473856.0,
+ "986": 476473856.0,
+ "987": 476473856.0,
+ "988": 476473856.0,
+ "989": 476473856.0,
+ "990": 476473856.0,
+ "991": 476473856.0,
+ "992": 476473856.0,
+ "993": 476473856.0,
+ "994": 476473856.0,
+ "995": 476473856.0,
+ "996": 476473856.0,
+ "997": 476473856.0,
+ "998": 476473856.0,
+ "999": 476473856.0,
+ "1000": 476473856.0,
+ "1001": 476473856.0,
+ "1002": 476473856.0,
+ "1003": 476473856.0,
+ "1004": 476473856.0,
+ "1005": 476473856.0,
+ "1006": 476473856.0,
+ "1007": 476473856.0,
+ "1008": 476473856.0,
+ "1009": 476473856.0,
+ "1010": 476473856.0,
+ "1011": 476473856.0,
+ "1012": 476473856.0,
+ "1013": 476473856.0,
+ "1014": 476473856.0,
+ "1015": 476473856.0,
+ "1016": 476473856.0,
+ "1017": 476473856.0,
+ "1018": 476473856.0,
+ "1019": 476473856.0,
+ "1020": 476473856.0,
+ "1021": 476473856.0,
+ "1022": 476473856.0,
+ "1023": 476473856.0,
+ "1024": 476473856.0,
+ "1025": 476473856.0,
+ "1026": 476473856.0,
+ "1027": 476473856.0,
+ "1028": 476473856.0,
+ "1029": 476473856.0,
+ "1030": 476473856.0,
+ "1031": 476473856.0,
+ "1032": 476473856.0,
+ "1033": 476473856.0,
+ "1034": 476473856.0,
+ "1035": 476473856.0,
+ "1036": 476473856.0,
+ "1037": 476473856.0,
+ "1038": 476473856.0,
+ "1039": 476473856.0,
+ "1040": 476473856.0,
+ "1041": 476473856.0,
+ "1042": 476473856.0,
+ "1043": 476473856.0,
+ "1044": 476473856.0,
+ "1045": 476473856.0,
+ "1046": 476473856.0,
+ "1047": 476473856.0,
+ "1048": 476473856.0,
+ "1049": 476473856.0,
+ "1050": 476473856.0,
+ "1051": 476473856.0,
+ "1052": 476473856.0,
+ "1053": 476473856.0,
+ "1054": 476473856.0,
+ "1055": 476473856.0,
+ "1056": 476473856.0,
+ "1057": 476473856.0,
+ "1058": 476473856.0,
+ "1059": 476473856.0,
+ "1060": 476473856.0,
+ "1061": 476473856.0,
+ "1062": 476473856.0,
+ "1063": 476473856.0,
+ "1064": 476473856.0,
+ "1065": 476473856.0,
+ "1066": 476473856.0,
+ "1067": 476473856.0,
+ "1068": 476473856.0,
+ "1069": 476473856.0,
+ "1070": 476473856.0,
+ "1071": 476473856.0,
+ "1072": 476473856.0,
+ "1073": 476473856.0,
+ "1074": 476473856.0,
+ "1075": 476473856.0,
+ "1076": 476473856.0,
+ "1077": 476473856.0,
+ "1078": 476473856.0,
+ "1079": 476473856.0,
+ "1080": 476473856.0,
+ "1081": 476473856.0,
+ "1082": 476473856.0,
+ "1083": 476473856.0,
+ "1084": 476473856.0,
+ "1085": 476473856.0,
+ "1086": 476473856.0,
+ "1087": 476473856.0,
+ "1088": 476473856.0,
+ "1089": 476473856.0,
+ "1090": 476473856.0,
+ "1091": 476473856.0,
+ "1092": 476473856.0,
+ "1093": 476473856.0,
+ "1094": 476473856.0,
+ "1095": 476473856.0,
+ "1096": 476473856.0,
+ "1097": 476473856.0,
+ "1098": 476473856.0,
+ "1099": 476473856.0,
+ "1100": 476473856.0,
+ "1101": 476473856.0,
+ "1102": 476473856.0,
+ "1103": 476473856.0,
+ "1104": 476473856.0,
+ "1105": 476473856.0,
+ "1106": 476473856.0,
+ "1107": 476473856.0,
+ "1108": 476473856.0,
+ "1109": 476473856.0,
+ "1110": 476473856.0,
+ "1111": 476473856.0,
+ "1112": 476473856.0,
+ "1113": 476473856.0,
+ "1114": 476473856.0,
+ "1115": 476473856.0,
+ "1116": 476473856.0,
+ "1117": 476473856.0,
+ "1118": 476473856.0,
+ "1119": 476473856.0,
+ "1120": 476473856.0,
+ "1121": 476473856.0,
+ "1122": 476473856.0,
+ "1123": 476473856.0,
+ "1124": 476473856.0,
+ "1125": 476473856.0,
+ "1126": 476473856.0,
+ "1127": 476473856.0,
+ "1128": 476473856.0,
+ "1129": 476473856.0,
+ "1130": 476473856.0,
+ "1131": 476473856.0,
+ "1132": 476473856.0,
+ "1133": 476473856.0,
+ "1134": 476473856.0,
+ "1135": 476473856.0,
+ "1136": 476473856.0,
+ "1137": 476473856.0,
+ "1138": 476473856.0,
+ "1139": 476473856.0,
+ "1140": 476473856.0,
+ "1141": 476473856.0,
+ "1142": 476473856.0,
+ "1143": 476473856.0,
+ "1144": 476473856.0,
+ "1145": 476473856.0,
+ "1146": 476473856.0,
+ "1147": 476473856.0,
+ "1148": 476473856.0,
+ "1149": 476473856.0,
+ "1150": 476473856.0,
+ "1151": 476473856.0,
+ "1152": 476473856.0,
+ "1153": 476473856.0,
+ "1154": 476473856.0,
+ "1155": 476473856.0,
+ "1156": 476473856.0,
+ "1157": 476473856.0,
+ "1158": 476473856.0,
+ "1159": 476473856.0,
+ "1160": 476473856.0,
+ "1161": 476473856.0,
+ "1162": 476473856.0,
+ "1163": 476473856.0,
+ "1164": 476473856.0,
+ "1165": 476473856.0,
+ "1166": 476473856.0,
+ "1167": 476473856.0,
+ "1168": 476473856.0,
+ "1169": 476473856.0,
+ "1170": 476473856.0,
+ "1171": 476473856.0,
+ "1172": 476473856.0,
+ "1173": 476473856.0,
+ "1174": 476473856.0,
+ "1175": 476473856.0,
+ "1176": 476473856.0,
+ "1177": 476473856.0,
+ "1178": 476473856.0,
+ "1179": 476473856.0,
+ "1180": 476473856.0,
+ "1181": 476473856.0,
+ "1182": 476473856.0,
+ "1183": 476473856.0,
+ "1184": 476473856.0,
+ "1185": 476473856.0,
+ "1186": 476473856.0,
+ "1187": 476473856.0,
+ "1188": 476473856.0,
+ "1189": 476473856.0,
+ "1190": 476473856.0,
+ "1191": 476473856.0,
+ "1192": 476473856.0,
+ "1193": 476473856.0,
+ "1194": 476473856.0,
+ "1195": 476473856.0,
+ "1196": 476473856.0,
+ "1197": 476473856.0,
+ "1198": 476473856.0,
+ "1199": 476473856.0,
+ "1200": 476473856.0,
+ "1201": 476473856.0,
+ "1202": 476473856.0,
+ "1203": 476473856.0,
+ "1204": 476473856.0,
+ "1205": 476473856.0,
+ "1206": 476473856.0,
+ "1207": 476473856.0,
+ "1208": 476473856.0,
+ "1209": 476473856.0,
+ "1210": 476473856.0,
+ "1211": 476473856.0,
+ "1212": 476473856.0,
+ "1213": 476473856.0,
+ "1214": 476473856.0,
+ "1215": 476473856.0,
+ "1216": 476473856.0,
+ "1217": 476473856.0,
+ "1218": 476473856.0,
+ "1219": 476473856.0,
+ "1220": 476473856.0,
+ "1221": 476473856.0,
+ "1222": 476473856.0,
+ "1223": 476473856.0,
+ "1224": 476473856.0,
+ "1225": 476473856.0,
+ "1226": 476473856.0,
+ "1227": 476473856.0,
+ "1228": 476473856.0,
+ "1229": 476473856.0,
+ "1230": 476473856.0,
+ "1231": 476473856.0,
+ "1232": 476473856.0,
+ "1233": 476473856.0,
+ "1234": 476473856.0,
+ "1235": 476473856.0,
+ "1236": 476473856.0,
+ "1237": 476473856.0,
+ "1238": 476473856.0,
+ "1239": 476473856.0,
+ "1240": 476473856.0,
+ "1241": 476473856.0,
+ "1242": 476473856.0,
+ "1243": 476473856.0,
+ "1244": 476473856.0,
+ "1245": 476473856.0,
+ "1246": 476473856.0,
+ "1247": 476473856.0,
+ "1248": 476473856.0,
+ "1249": 476473856.0,
+ "1250": 476473856.0,
+ "1251": 476473856.0,
+ "1252": 476473856.0,
+ "1253": 476473856.0,
+ "1254": 476473856.0,
+ "1255": 476473856.0,
+ "1256": 476473856.0,
+ "1257": 476473856.0,
+ "1258": 476473856.0,
+ "1259": 476473856.0,
+ "1260": 476473856.0,
+ "1261": 476473856.0,
+ "1262": 476473856.0,
+ "1263": 476473856.0,
+ "1264": 476473856.0,
+ "1265": 476473856.0,
+ "1266": 476473856.0,
+ "1267": 476473856.0,
+ "1268": 476473856.0,
+ "1269": 476473856.0,
+ "1270": 476473856.0,
+ "1271": 476473856.0,
+ "1272": 476473856.0,
+ "1273": 476473856.0,
+ "1274": 476473856.0,
+ "1275": 476473856.0,
+ "1276": 476473856.0,
+ "1277": 476473856.0,
+ "1278": 476473856.0,
+ "1279": 476473856.0,
+ "1280": 476473856.0,
+ "1281": 476473856.0,
+ "1282": 476473856.0,
+ "1283": 476473856.0,
+ "1284": 476473856.0,
+ "1285": 476473856.0,
+ "1286": 476473856.0,
+ "1287": 476473856.0,
+ "1288": 476473856.0,
+ "1289": 476473856.0,
+ "1290": 476473856.0,
+ "1291": 476473856.0,
+ "1292": 476473856.0,
+ "1293": 476473856.0,
+ "1294": 476473856.0,
+ "1295": 476473856.0,
+ "1296": 476473856.0,
+ "1297": 476473856.0,
+ "1298": 476473856.0,
+ "1299": 476473856.0,
+ "1300": 476473856.0,
+ "1301": 476473856.0,
+ "1302": 476473856.0,
+ "1303": 476473856.0,
+ "1304": 476473856.0,
+ "1305": 476473856.0,
+ "1306": 476473856.0,
+ "1307": 476473856.0,
+ "1308": 476473856.0,
+ "1309": 476473856.0,
+ "1310": 476473856.0,
+ "1311": 476473856.0,
+ "1312": 476473856.0,
+ "1313": 476473856.0,
+ "1314": 476473856.0,
+ "1315": 476473856.0,
+ "1316": 476473856.0,
+ "1317": 476473856.0,
+ "1318": 476473856.0,
+ "1319": 476473856.0,
+ "1320": 476473856.0,
+ "1321": 476473856.0,
+ "1322": 476473856.0,
+ "1323": 476473856.0,
+ "1324": 476473856.0,
+ "1325": 476473856.0,
+ "1326": 476473856.0,
+ "1327": 476473856.0,
+ "1328": 476473856.0,
+ "1329": 476473856.0,
+ "1330": 476473856.0,
+ "1331": 476473856.0,
+ "1332": 476473856.0,
+ "1333": 476473856.0,
+ "1334": 476473856.0,
+ "1335": 476473856.0,
+ "1336": 476473856.0,
+ "1337": 476473856.0,
+ "1338": 476473856.0,
+ "1339": 476473856.0,
+ "1340": 476473856.0,
+ "1341": 476473856.0,
+ "1342": 476473856.0,
+ "1343": 476473856.0,
+ "1344": 476473856.0,
+ "1345": 476473856.0,
+ "1346": 476473856.0,
+ "1347": 476473856.0,
+ "1348": 476473856.0,
+ "1349": 476473856.0,
+ "1350": 476473856.0,
+ "1351": 476473856.0,
+ "1352": 476473856.0,
+ "1353": 476473856.0,
+ "1354": 476473856.0,
+ "1355": 476473856.0,
+ "1356": 476473856.0,
+ "1357": 476473856.0,
+ "1358": 476473856.0,
+ "1359": 476473856.0,
+ "1360": 476473856.0,
+ "1361": 476473856.0,
+ "1362": 476473856.0,
+ "1363": 476473856.0,
+ "1364": 476473856.0,
+ "1365": 476473856.0,
+ "1366": 476473856.0,
+ "1367": 476473856.0,
+ "1368": 476473856.0,
+ "1369": 476473856.0,
+ "1370": 476473856.0,
+ "1371": 476473856.0,
+ "1372": 476473856.0,
+ "1373": 476473856.0,
+ "1374": 476473856.0,
+ "1375": 476473856.0,
+ "1376": 476473856.0,
+ "1377": 476473856.0,
+ "1378": 476473856.0,
+ "1379": 476473856.0,
+ "1380": 476473856.0,
+ "1381": 476473856.0,
+ "1382": 476473856.0,
+ "1383": 476473856.0,
+ "1384": 476473856.0,
+ "1385": 476473856.0,
+ "1386": 476473856.0,
+ "1387": 476473856.0,
+ "1388": 476473856.0,
+ "1389": 476473856.0,
+ "1390": 476473856.0,
+ "1391": 476473856.0,
+ "1392": 476473856.0,
+ "1393": 476473856.0,
+ "1394": 476473856.0,
+ "1395": 476473856.0,
+ "1396": 476473856.0,
+ "1397": 476473856.0,
+ "1398": 476473856.0,
+ "1399": 476473856.0,
+ "1400": 476473856.0,
+ "1401": 476473856.0,
+ "1402": 476473856.0,
+ "1403": 476473856.0,
+ "1404": 476473856.0,
+ "1405": 476473856.0,
+ "1406": 476473856.0,
+ "1407": 476473856.0,
+ "1408": 476473856.0,
+ "1409": 476473856.0,
+ "1410": 476473856.0,
+ "1411": 476473856.0,
+ "1412": 476473856.0,
+ "1413": 476473856.0,
+ "1414": 476473856.0,
+ "1415": 476473856.0,
+ "1416": 476473856.0,
+ "1417": 476473856.0,
+ "1418": 476473856.0,
+ "1419": 476473856.0,
+ "1420": 476473856.0,
+ "1421": 476473856.0,
+ "1422": 476473856.0,
+ "1423": 476473856.0,
+ "1424": 476473856.0,
+ "1425": 476473856.0,
+ "1426": 476473856.0,
+ "1427": 476473856.0,
+ "1428": 476473856.0,
+ "1429": 476473856.0,
+ "1430": 476473856.0,
+ "1431": 476473856.0,
+ "1432": 476473856.0,
+ "1433": 476473856.0,
+ "1434": 476473856.0,
+ "1435": 476473856.0,
+ "1436": 476473856.0,
+ "1437": 476473856.0,
+ "1438": 476473856.0,
+ "1439": 476473856.0,
+ "1440": 476473856.0,
+ "1441": 476473856.0,
+ "1442": 476473856.0,
+ "1443": 476473856.0,
+ "1444": 476473856.0,
+ "1445": 476473856.0,
+ "1446": 476473856.0,
+ "1447": 476473856.0,
+ "1448": 476473856.0,
+ "1449": 476473856.0,
+ "1450": 476473856.0,
+ "1451": 476473856.0,
+ "1452": 476473856.0,
+ "1453": 476473856.0,
+ "1454": 476473856.0,
+ "1455": 476473856.0,
+ "1456": 476473856.0,
+ "1457": 476473856.0,
+ "1458": 476473856.0,
+ "1459": 476473856.0,
+ "1460": 476473856.0,
+ "1461": 476473856.0,
+ "1462": 476473856.0,
+ "1463": 476473856.0,
+ "1464": 476473856.0,
+ "1465": 476473856.0,
+ "1466": 476473856.0,
+ "1467": 476473856.0,
+ "1468": 476473856.0,
+ "1469": 476473856.0,
+ "1470": 476473856.0,
+ "1471": 476473856.0,
+ "1472": 476473856.0,
+ "1473": 476473856.0,
+ "1474": 476473856.0,
+ "1475": 476473856.0,
+ "1476": 476473856.0,
+ "1477": 476473856.0,
+ "1478": 476473856.0,
+ "1479": 476473856.0,
+ "1480": 476473856.0,
+ "1481": 476473856.0,
+ "1482": 476473856.0,
+ "1483": 476473856.0,
+ "1484": 476473856.0,
+ "1485": 476473856.0,
+ "1486": 476473856.0,
+ "1487": 476473856.0,
+ "1488": 476473856.0,
+ "1489": 476473856.0,
+ "1490": 476473856.0,
+ "1491": 476473856.0,
+ "1492": 476473856.0,
+ "1493": 476473856.0,
+ "1494": 476473856.0,
+ "1495": 476473856.0,
+ "1496": 476473856.0,
+ "1497": 476473856.0,
+ "1498": 476473856.0,
+ "1499": 476473856.0,
+ "1500": 476473856.0,
+ "1501": 476473856.0,
+ "1502": 476473856.0,
+ "1503": 476473856.0,
+ "1504": 476473856.0,
+ "1505": 476473856.0,
+ "1506": 476473856.0,
+ "1507": 476473856.0,
+ "1508": 476473856.0,
+ "1509": 476473856.0,
+ "1510": 476473856.0,
+ "1511": 476473856.0,
+ "1512": 476473856.0,
+ "1513": 476473856.0,
+ "1514": 476473856.0,
+ "1515": 476473856.0,
+ "1516": 476473856.0,
+ "1517": 476473856.0,
+ "1518": 476473856.0,
+ "1519": 476473856.0,
+ "1520": 476473856.0,
+ "1521": 476473856.0,
+ "1522": 476473856.0,
+ "1523": 476473856.0,
+ "1524": 476473856.0,
+ "1525": 476473856.0,
+ "1526": 476473856.0,
+ "1527": 476473856.0,
+ "1528": 476473856.0,
+ "1529": 476473856.0,
+ "1530": 476473856.0,
+ "1531": 476473856.0,
+ "1532": 476473856.0,
+ "1533": 476473856.0,
+ "1534": 476473856.0,
+ "1535": 476473856.0,
+ "1536": 476473856.0,
+ "1537": 476473856.0,
+ "1538": 476473856.0,
+ "1539": 476473856.0,
+ "1540": 476473856.0,
+ "1541": 476473856.0,
+ "1542": 476473856.0,
+ "1543": 476473856.0,
+ "1544": 476473856.0,
+ "1545": 476473856.0,
+ "1546": 476473856.0,
+ "1547": 476473856.0,
+ "1548": 476473856.0,
+ "1549": 476473856.0,
+ "1550": 476473856.0,
+ "1551": 476473856.0,
+ "1552": 476473856.0,
+ "1553": 476473856.0,
+ "1554": 476473856.0,
+ "1555": 476473856.0,
+ "1556": 476473856.0,
+ "1557": 476473856.0,
+ "1558": 476473856.0,
+ "1559": 476473856.0,
+ "1560": 476473856.0,
+ "1561": 476473856.0,
+ "1562": 476473856.0,
+ "1563": 476473856.0,
+ "1564": 476473856.0,
+ "1565": 476473856.0,
+ "1566": 476473856.0,
+ "1567": 476473856.0,
+ "1568": 476473856.0,
+ "1569": 476473856.0,
+ "1570": 476473856.0,
+ "1571": 476473856.0,
+ "1572": 476473856.0,
+ "1573": 476473856.0,
+ "1574": 476473856.0,
+ "1575": 476473856.0,
+ "1576": 476473856.0,
+ "1577": 476473856.0,
+ "1578": 476473856.0,
+ "1579": 476473856.0,
+ "1580": 476473856.0,
+ "1581": 476473856.0,
+ "1582": 476473856.0,
+ "1583": 476473856.0,
+ "1584": 476473856.0,
+ "1585": 476473856.0,
+ "1586": 476473856.0,
+ "1587": 476473856.0,
+ "1588": 476473856.0,
+ "1589": 476473856.0,
+ "1590": 476473856.0,
+ "1591": 476473856.0,
+ "1592": 476473856.0,
+ "1593": 476473856.0,
+ "1594": 476473856.0,
+ "1595": 476473856.0,
+ "1596": 476473856.0,
+ "1597": 476473856.0,
+ "1598": 476473856.0,
+ "1599": 476473856.0,
+ "1600": 476473856.0,
+ "1601": 476473856.0,
+ "1602": 476473856.0,
+ "1603": 476473856.0,
+ "1604": 476473856.0,
+ "1605": 476473856.0,
+ "1606": 476473856.0,
+ "1607": 476473856.0,
+ "1608": 476473856.0,
+ "1609": 476473856.0,
+ "1610": 476473856.0,
+ "1611": 476473856.0,
+ "1612": 476473856.0,
+ "1613": 476473856.0,
+ "1614": 476473856.0,
+ "1615": 476473856.0,
+ "1616": 476473856.0,
+ "1617": 476473856.0,
+ "1618": 476473856.0,
+ "1619": 476473856.0,
+ "1620": 476473856.0,
+ "1621": 476473856.0,
+ "1622": 476473856.0,
+ "1623": 476473856.0,
+ "1624": 476473856.0,
+ "1625": 476473856.0,
+ "1626": 476473856.0,
+ "1627": 476473856.0,
+ "1628": 476473856.0,
+ "1629": 476473856.0,
+ "1630": 476473856.0,
+ "1631": 476473856.0,
+ "1632": 476473856.0,
+ "1633": 476473856.0,
+ "1634": 476473856.0,
+ "1635": 476473856.0,
+ "1636": 476473856.0,
+ "1637": 476473856.0,
+ "1638": 476473856.0,
+ "1639": 476473856.0,
+ "1640": 476473856.0,
+ "1641": 476473856.0,
+ "1642": 476473856.0,
+ "1643": 476473856.0,
+ "1644": 476473856.0,
+ "1645": 476473856.0,
+ "1646": 476473856.0,
+ "1647": 476473856.0,
+ "1648": 476473856.0,
+ "1649": 476473856.0,
+ "1650": 476473856.0,
+ "1651": 476473856.0,
+ "1652": 476473856.0,
+ "1653": 476473856.0,
+ "1654": 476473856.0,
+ "1655": 476473856.0,
+ "1656": 476473856.0,
+ "1657": 476473856.0,
+ "1658": 476473856.0,
+ "1659": 476473856.0,
+ "1660": 476473856.0,
+ "1661": 476473856.0,
+ "1662": 476473856.0,
+ "1663": 476473856.0,
+ "1664": 476473856.0,
+ "1665": 476473856.0,
+ "1666": 476473856.0,
+ "1667": 476473856.0,
+ "1668": 476473856.0,
+ "1669": 476473856.0,
+ "1670": 476473856.0,
+ "1671": 476473856.0,
+ "1672": 476473856.0,
+ "1673": 476473856.0,
+ "1674": 476473856.0,
+ "1675": 476473856.0,
+ "1676": 476473856.0,
+ "1677": 476473856.0,
+ "1678": 476473856.0,
+ "1679": 476473856.0,
+ "1680": 476473856.0,
+ "1681": 476473856.0,
+ "1682": 476473856.0,
+ "1683": 476473856.0,
+ "1684": 476473856.0,
+ "1685": 476473856.0,
+ "1686": 476473856.0,
+ "1687": 476473856.0,
+ "1688": 476473856.0,
+ "1689": 476473856.0,
+ "1690": 476473856.0,
+ "1691": 476473856.0,
+ "1692": 476473856.0,
+ "1693": 476473856.0,
+ "1694": 476473856.0,
+ "1695": 476473856.0,
+ "1696": 476473856.0,
+ "1697": 476473856.0,
+ "1698": 476473856.0,
+ "1699": 476473856.0,
+ "1700": 476473856.0,
+ "1701": 476473856.0,
+ "1702": 476473856.0,
+ "1703": 476473856.0,
+ "1704": 476473856.0,
+ "1705": 476473856.0,
+ "1706": 476473856.0,
+ "1707": 476473856.0,
+ "1708": 476473856.0,
+ "1709": 476473856.0,
+ "1710": 476473856.0,
+ "1711": 476473856.0,
+ "1712": 476473856.0,
+ "1713": 476473856.0,
+ "1714": 476473856.0,
+ "1715": 476473856.0,
+ "1716": 476473856.0,
+ "1717": 476473856.0,
+ "1718": 476473856.0,
+ "1719": 476473856.0,
+ "1720": 476473856.0,
+ "1721": 476473856.0,
+ "1722": 476473856.0,
+ "1723": 476473856.0,
+ "1724": 476473856.0,
+ "1725": 476473856.0,
+ "1726": 476473856.0,
+ "1727": 476473856.0,
+ "1728": 476473856.0,
+ "1729": 476473856.0,
+ "1730": 476473856.0,
+ "1731": 476473856.0,
+ "1732": 476473856.0,
+ "1733": 476473856.0,
+ "1734": 476473856.0,
+ "1735": 476473856.0,
+ "1736": 476473856.0,
+ "1737": 476473856.0,
+ "1738": 476473856.0,
+ "1739": 476473856.0,
+ "1740": 476473856.0,
+ "1741": 476473856.0,
+ "1742": 476473856.0,
+ "1743": 476473856.0,
+ "1744": 476473856.0,
+ "1745": 476473856.0,
+ "1746": 476473856.0,
+ "1747": 476473856.0,
+ "1748": 476473856.0,
+ "1749": 476473856.0,
+ "1750": 476473856.0,
+ "1751": 476473856.0,
+ "1752": 476473856.0,
+ "1753": 476473856.0,
+ "1754": 476473856.0,
+ "1755": 476473856.0,
+ "1756": 476473856.0,
+ "1757": 476473856.0,
+ "1758": 476473856.0,
+ "1759": 476473856.0,
+ "1760": 476473856.0,
+ "1761": 476473856.0,
+ "1762": 476473856.0,
+ "1763": 476473856.0,
+ "1764": 476473856.0,
+ "1765": 476473856.0,
+ "1766": 476473856.0,
+ "1767": 476473856.0,
+ "1768": 476473856.0,
+ "1769": 476473856.0,
+ "1770": 476473856.0,
+ "1771": 476473856.0,
+ "1772": 476473856.0,
+ "1773": 476473856.0,
+ "1774": 476473856.0,
+ "1775": 476473856.0,
+ "1776": 476473856.0,
+ "1777": 476473856.0,
+ "1778": 476473856.0,
+ "1779": 476473856.0,
+ "1780": 476473856.0,
+ "1781": 476473856.0,
+ "1782": 476473856.0,
+ "1783": 476473856.0,
+ "1784": 476473856.0,
+ "1785": 476473856.0,
+ "1786": 476473856.0,
+ "1787": 476473856.0,
+ "1788": 476473856.0,
+ "1789": 476473856.0,
+ "1790": 476473856.0,
+ "1791": 476473856.0,
+ "1792": 476473856.0,
+ "1793": 476473856.0,
+ "1794": 476473856.0,
+ "1795": 476473856.0,
+ "1796": 476473856.0,
+ "1797": 476473856.0,
+ "1798": 476473856.0,
+ "1799": 476473856.0,
+ "1800": 476473856.0,
+ "1801": 476473856.0,
+ "1802": 476473856.0,
+ "1803": 476473856.0,
+ "1804": 476473856.0,
+ "1805": 476473856.0,
+ "1806": 476473856.0,
+ "1807": 476473856.0,
+ "1808": 476473856.0,
+ "1809": 476473856.0,
+ "1810": 476473856.0,
+ "1811": 476473856.0,
+ "1812": 476473856.0,
+ "1813": 476473856.0,
+ "1814": 476473856.0,
+ "1815": 476473856.0,
+ "1816": 476473856.0,
+ "1817": 476473856.0,
+ "1818": 476473856.0,
+ "1819": 476473856.0,
+ "1820": 476473856.0,
+ "1821": 476473856.0,
+ "1822": 476473856.0,
+ "1823": 476473856.0,
+ "1824": 476473856.0,
+ "1825": 476473856.0,
+ "1826": 476473856.0,
+ "1827": 476473856.0,
+ "1828": 476473856.0,
+ "1829": 476473856.0,
+ "1830": 476473856.0,
+ "1831": 476473856.0,
+ "1832": 476473856.0,
+ "1833": 476473856.0,
+ "1834": 476473856.0,
+ "1835": 476473856.0,
+ "1836": 476473856.0,
+ "1837": 476473856.0,
+ "1838": 476473856.0,
+ "1839": 476473856.0,
+ "1840": 476473856.0,
+ "1841": 476473856.0,
+ "1842": 476473856.0,
+ "1843": 476473856.0,
+ "1844": 476473856.0,
+ "1845": 476473856.0,
+ "1846": 476473856.0,
+ "1847": 476473856.0,
+ "1848": 476473856.0,
+ "1849": 476473856.0,
+ "1850": 476473856.0,
+ "1851": 476473856.0,
+ "1852": 476473856.0,
+ "1853": 476473856.0,
+ "1854": 476473856.0,
+ "1855": 476473856.0,
+ "1856": 476473856.0,
+ "1857": 476473856.0,
+ "1858": 476473856.0,
+ "1859": 476473856.0,
+ "1860": 476473856.0,
+ "1861": 476473856.0,
+ "1862": 476473856.0,
+ "1863": 476473856.0,
+ "1864": 476473856.0,
+ "1865": 476473856.0,
+ "1866": 476473856.0,
+ "1867": 476473856.0,
+ "1868": 476473856.0,
+ "1869": 476473856.0,
+ "1870": 476473856.0,
+ "1871": 476473856.0,
+ "1872": 476473856.0,
+ "1873": 476473856.0,
+ "1874": 476473856.0,
+ "1875": 476473856.0,
+ "1876": 476473856.0,
+ "1877": 476473856.0,
+ "1878": 476473856.0,
+ "1879": 476473856.0,
+ "1880": 476473856.0,
+ "1881": 476473856.0,
+ "1882": 476473856.0,
+ "1883": 476473856.0,
+ "1884": 476473856.0,
+ "1885": 476473856.0,
+ "1886": 476473856.0,
+ "1887": 476473856.0,
+ "1888": 476473856.0,
+ "1889": 476473856.0,
+ "1890": 476473856.0,
+ "1891": 476473856.0,
+ "1892": 476473856.0,
+ "1893": 476473856.0,
+ "1894": 476473856.0,
+ "1895": 476473856.0,
+ "1896": 476473856.0,
+ "1897": 476473856.0,
+ "1898": 476473856.0,
+ "1899": 476473856.0,
+ "1900": 476473856.0,
+ "1901": 476473856.0,
+ "1902": 476473856.0,
+ "1903": 476473856.0,
+ "1904": 476473856.0,
+ "1905": 476473856.0,
+ "1906": 476473856.0,
+ "1907": 476473856.0,
+ "1908": 476473856.0,
+ "1909": 476473856.0,
+ "1910": 476473856.0,
+ "1911": 476473856.0,
+ "1912": 476473856.0,
+ "1913": 476473856.0,
+ "1914": 476473856.0,
+ "1915": 476473856.0,
+ "1916": 476473856.0,
+ "1917": 476473856.0,
+ "1918": 476473856.0,
+ "1919": 476473856.0,
+ "1920": 476473856.0,
+ "1921": 476473856.0,
+ "1922": 476473856.0,
+ "1923": 476473856.0,
+ "1924": 476473856.0,
+ "1925": 476473856.0,
+ "1926": 476473856.0,
+ "1927": 476473856.0,
+ "1928": 476473856.0,
+ "1929": 476473856.0,
+ "1930": 476473856.0,
+ "1931": 476473856.0,
+ "1932": 476473856.0,
+ "1933": 476473856.0,
+ "1934": 476473856.0,
+ "1935": 476473856.0,
+ "1936": 476473856.0,
+ "1937": 476473856.0,
+ "1938": 476473856.0,
+ "1939": 476473856.0,
+ "1940": 476473856.0,
+ "1941": 476473856.0,
+ "1942": 476473856.0,
+ "1943": 476473856.0,
+ "1944": 476473856.0,
+ "1945": 476473856.0,
+ "1946": 476473856.0,
+ "1947": 476473856.0,
+ "1948": 476473856.0,
+ "1949": 476473856.0,
+ "1950": 476473856.0,
+ "1951": 476473856.0,
+ "1952": 476473856.0,
+ "1953": 476473856.0,
+ "1954": 476473856.0,
+ "1955": 476473856.0,
+ "1956": 476473856.0,
+ "1957": 476473856.0,
+ "1958": 476473856.0,
+ "1959": 476473856.0,
+ "1960": 476473856.0,
+ "1961": 476473856.0,
+ "1962": 476473856.0,
+ "1963": 476473856.0,
+ "1964": 476473856.0,
+ "1965": 476473856.0,
+ "1966": 476473856.0,
+ "1967": 476473856.0,
+ "1968": 476473856.0,
+ "1969": 476473856.0,
+ "1970": 476473856.0,
+ "1971": 476473856.0,
+ "1972": 476473856.0,
+ "1973": 476473856.0,
+ "1974": 476473856.0,
+ "1975": 476473856.0,
+ "1976": 476473856.0,
+ "1977": 476473856.0,
+ "1978": 476473856.0,
+ "1979": 476473856.0,
+ "1980": 476473856.0,
+ "1981": 476473856.0,
+ "1982": 476473856.0,
+ "1983": 476473856.0,
+ "1984": 476473856.0,
+ "1985": 476473856.0,
+ "1986": 476473856.0,
+ "1987": 476473856.0,
+ "1988": 476473856.0,
+ "1989": 476473856.0,
+ "1990": 476473856.0,
+ "1991": 476473856.0,
+ "1992": 476473856.0,
+ "1993": 476473856.0,
+ "1994": 476473856.0,
+ "1995": 476473856.0,
+ "1996": 476473856.0,
+ "1997": 476473856.0,
+ "1998": 476473856.0,
+ "1999": 476473856.0,
+ "2000": 476473856.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 794742272.0,
+ "2": 883962368.0,
+ "3": 883962368.0,
+ "4": 883962368.0,
+ "5": 883962368.0,
+ "6": 883962368.0,
+ "7": 883962368.0,
+ "8": 883962368.0,
+ "9": 883962368.0,
+ "10": 883962368.0,
+ "11": 883962368.0,
+ "12": 883962368.0,
+ "13": 883962368.0,
+ "14": 883962368.0,
+ "15": 883962368.0,
+ "16": 883962368.0,
+ "17": 883962368.0,
+ "18": 883962368.0,
+ "19": 883962368.0,
+ "20": 883962368.0,
+ "21": 883962368.0,
+ "22": 883962368.0,
+ "23": 883962368.0,
+ "24": 883962368.0,
+ "25": 883962368.0,
+ "26": 883962368.0,
+ "27": 883962368.0,
+ "28": 883962368.0,
+ "29": 883962368.0,
+ "30": 883962368.0,
+ "31": 883962368.0,
+ "32": 883962368.0,
+ "33": 883962368.0,
+ "34": 883962368.0,
+ "35": 883962368.0,
+ "36": 883962368.0,
+ "37": 883962368.0,
+ "38": 883962368.0,
+ "39": 883962368.0,
+ "40": 883962368.0,
+ "41": 883962368.0,
+ "42": 883962368.0,
+ "43": 883962368.0,
+ "44": 883962368.0,
+ "45": 883962368.0,
+ "46": 883962368.0,
+ "47": 883962368.0,
+ "48": 883962368.0,
+ "49": 883962368.0,
+ "50": 883962368.0,
+ "51": 883962368.0,
+ "52": 883962368.0,
+ "53": 883962368.0,
+ "54": 883962368.0,
+ "55": 883962368.0,
+ "56": 883962368.0,
+ "57": 883962368.0,
+ "58": 883962368.0,
+ "59": 883962368.0,
+ "60": 883962368.0,
+ "61": 883962368.0,
+ "62": 883962368.0,
+ "63": 883962368.0,
+ "64": 883962368.0,
+ "65": 883962368.0,
+ "66": 883962368.0,
+ "67": 883962368.0,
+ "68": 883962368.0,
+ "69": 883962368.0,
+ "70": 883962368.0,
+ "71": 883962368.0,
+ "72": 883962368.0,
+ "73": 883962368.0,
+ "74": 883962368.0,
+ "75": 883962368.0,
+ "76": 883962368.0,
+ "77": 883962368.0,
+ "78": 883962368.0,
+ "79": 883962368.0,
+ "80": 883962368.0,
+ "81": 883962368.0,
+ "82": 883962368.0,
+ "83": 883962368.0,
+ "84": 883962368.0,
+ "85": 883962368.0,
+ "86": 883962368.0,
+ "87": 883962368.0,
+ "88": 883962368.0,
+ "89": 883962368.0,
+ "90": 883962368.0,
+ "91": 883962368.0,
+ "92": 883962368.0,
+ "93": 883962368.0,
+ "94": 883962368.0,
+ "95": 883962368.0,
+ "96": 883962368.0,
+ "97": 883962368.0,
+ "98": 883962368.0,
+ "99": 883962368.0,
+ "100": 883962368.0,
+ "101": 883962368.0,
+ "102": 883962368.0,
+ "103": 883962368.0,
+ "104": 883962368.0,
+ "105": 883962368.0,
+ "106": 883962368.0,
+ "107": 883962368.0,
+ "108": 883962368.0,
+ "109": 883962368.0,
+ "110": 883962368.0,
+ "111": 883962368.0,
+ "112": 883962368.0,
+ "113": 883962368.0,
+ "114": 883962368.0,
+ "115": 883962368.0,
+ "116": 883962368.0,
+ "117": 883962368.0,
+ "118": 883962368.0,
+ "119": 883962368.0,
+ "120": 883962368.0,
+ "121": 883962368.0,
+ "122": 883962368.0,
+ "123": 883962368.0,
+ "124": 883962368.0,
+ "125": 883962368.0,
+ "126": 883962368.0,
+ "127": 883962368.0,
+ "128": 883962368.0,
+ "129": 883962368.0,
+ "130": 883962368.0,
+ "131": 883962368.0,
+ "132": 883962368.0,
+ "133": 883962368.0,
+ "134": 883962368.0,
+ "135": 883962368.0,
+ "136": 883962368.0,
+ "137": 883962368.0,
+ "138": 883962368.0,
+ "139": 883962368.0,
+ "140": 883962368.0,
+ "141": 883962368.0,
+ "142": 883962368.0,
+ "143": 883962368.0,
+ "144": 883962368.0,
+ "145": 883962368.0,
+ "146": 883962368.0,
+ "147": 883962368.0,
+ "148": 883962368.0,
+ "149": 883962368.0,
+ "150": 883962368.0,
+ "151": 883962368.0,
+ "152": 883962368.0,
+ "153": 883962368.0,
+ "154": 883962368.0,
+ "155": 883962368.0,
+ "156": 883962368.0,
+ "157": 883962368.0,
+ "158": 883962368.0,
+ "159": 883962368.0,
+ "160": 883962368.0,
+ "161": 883962368.0,
+ "162": 883962368.0,
+ "163": 883962368.0,
+ "164": 883962368.0,
+ "165": 883962368.0,
+ "166": 883962368.0,
+ "167": 883962368.0,
+ "168": 883962368.0,
+ "169": 883962368.0,
+ "170": 883962368.0,
+ "171": 883962368.0,
+ "172": 883962368.0,
+ "173": 883962368.0,
+ "174": 883962368.0,
+ "175": 883962368.0,
+ "176": 883962368.0,
+ "177": 883962368.0,
+ "178": 883962368.0,
+ "179": 883962368.0,
+ "180": 883962368.0,
+ "181": 883962368.0,
+ "182": 883962368.0,
+ "183": 883962368.0,
+ "184": 883962368.0,
+ "185": 883962368.0,
+ "186": 883962368.0,
+ "187": 883962368.0,
+ "188": 883962368.0,
+ "189": 883962368.0,
+ "190": 883962368.0,
+ "191": 883962368.0,
+ "192": 883962368.0,
+ "193": 883962368.0,
+ "194": 883962368.0,
+ "195": 883962368.0,
+ "196": 883962368.0,
+ "197": 883962368.0,
+ "198": 883962368.0,
+ "199": 883962368.0,
+ "200": 883962368.0,
+ "201": 883962368.0,
+ "202": 883962368.0,
+ "203": 883962368.0,
+ "204": 883962368.0,
+ "205": 883962368.0,
+ "206": 883962368.0,
+ "207": 883962368.0,
+ "208": 883962368.0,
+ "209": 883962368.0,
+ "210": 883962368.0,
+ "211": 883962368.0,
+ "212": 883962368.0,
+ "213": 883962368.0,
+ "214": 883962368.0,
+ "215": 883962368.0,
+ "216": 883962368.0,
+ "217": 883962368.0,
+ "218": 883962368.0,
+ "219": 883962368.0,
+ "220": 883962368.0,
+ "221": 883962368.0,
+ "222": 883962368.0,
+ "223": 883962368.0,
+ "224": 883962368.0,
+ "225": 883962368.0,
+ "226": 883962368.0,
+ "227": 883962368.0,
+ "228": 883962368.0,
+ "229": 883962368.0,
+ "230": 883962368.0,
+ "231": 883962368.0,
+ "232": 883962368.0,
+ "233": 883962368.0,
+ "234": 883962368.0,
+ "235": 883962368.0,
+ "236": 883962368.0,
+ "237": 883962368.0,
+ "238": 883962368.0,
+ "239": 883962368.0,
+ "240": 883962368.0,
+ "241": 883962368.0,
+ "242": 883962368.0,
+ "243": 883962368.0,
+ "244": 883962368.0,
+ "245": 883962368.0,
+ "246": 883962368.0,
+ "247": 883962368.0,
+ "248": 883962368.0,
+ "249": 883962368.0,
+ "250": 883962368.0,
+ "251": 883962368.0,
+ "252": 883962368.0,
+ "253": 883962368.0,
+ "254": 883962368.0,
+ "255": 883962368.0,
+ "256": 883962368.0,
+ "257": 883962368.0,
+ "258": 883962368.0,
+ "259": 883962368.0,
+ "260": 883962368.0,
+ "261": 883962368.0,
+ "262": 883962368.0,
+ "263": 883962368.0,
+ "264": 883962368.0,
+ "265": 883962368.0,
+ "266": 883962368.0,
+ "267": 883962368.0,
+ "268": 883962368.0,
+ "269": 883962368.0,
+ "270": 883962368.0,
+ "271": 883962368.0,
+ "272": 883962368.0,
+ "273": 883962368.0,
+ "274": 883962368.0,
+ "275": 883962368.0,
+ "276": 883962368.0,
+ "277": 883962368.0,
+ "278": 883962368.0,
+ "279": 883962368.0,
+ "280": 883962368.0,
+ "281": 883962368.0,
+ "282": 883962368.0,
+ "283": 883962368.0,
+ "284": 883962368.0,
+ "285": 883962368.0,
+ "286": 883962368.0,
+ "287": 883962368.0,
+ "288": 883962368.0,
+ "289": 883962368.0,
+ "290": 883962368.0,
+ "291": 883962368.0,
+ "292": 883962368.0,
+ "293": 883962368.0,
+ "294": 883962368.0,
+ "295": 883962368.0,
+ "296": 883962368.0,
+ "297": 883962368.0,
+ "298": 883962368.0,
+ "299": 883962368.0,
+ "300": 883962368.0,
+ "301": 883962368.0,
+ "302": 883962368.0,
+ "303": 883962368.0,
+ "304": 883962368.0,
+ "305": 883962368.0,
+ "306": 883962368.0,
+ "307": 883962368.0,
+ "308": 883962368.0,
+ "309": 883962368.0,
+ "310": 883962368.0,
+ "311": 883962368.0,
+ "312": 883962368.0,
+ "313": 883962368.0,
+ "314": 883962368.0,
+ "315": 883962368.0,
+ "316": 883962368.0,
+ "317": 883962368.0,
+ "318": 883962368.0,
+ "319": 883962368.0,
+ "320": 883962368.0,
+ "321": 883962368.0,
+ "322": 883962368.0,
+ "323": 883962368.0,
+ "324": 883962368.0,
+ "325": 883962368.0,
+ "326": 883962368.0,
+ "327": 883962368.0,
+ "328": 883962368.0,
+ "329": 883962368.0,
+ "330": 883962368.0,
+ "331": 883962368.0,
+ "332": 883962368.0,
+ "333": 883962368.0,
+ "334": 883962368.0,
+ "335": 883962368.0,
+ "336": 883962368.0,
+ "337": 883962368.0,
+ "338": 883962368.0,
+ "339": 883962368.0,
+ "340": 883962368.0,
+ "341": 883962368.0,
+ "342": 883962368.0,
+ "343": 883962368.0,
+ "344": 883962368.0,
+ "345": 883962368.0,
+ "346": 883962368.0,
+ "347": 883962368.0,
+ "348": 883962368.0,
+ "349": 883962368.0,
+ "350": 883962368.0,
+ "351": 883962368.0,
+ "352": 883962368.0,
+ "353": 883962368.0,
+ "354": 883962368.0,
+ "355": 883962368.0,
+ "356": 883962368.0,
+ "357": 883962368.0,
+ "358": 883962368.0,
+ "359": 883962368.0,
+ "360": 883962368.0,
+ "361": 883962368.0,
+ "362": 883962368.0,
+ "363": 883962368.0,
+ "364": 883962368.0,
+ "365": 883962368.0,
+ "366": 883962368.0,
+ "367": 883962368.0,
+ "368": 883962368.0,
+ "369": 883962368.0,
+ "370": 883962368.0,
+ "371": 883962368.0,
+ "372": 883962368.0,
+ "373": 883962368.0,
+ "374": 883962368.0,
+ "375": 883962368.0,
+ "376": 883962368.0,
+ "377": 883962368.0,
+ "378": 883962368.0,
+ "379": 883962368.0,
+ "380": 883962368.0,
+ "381": 883962368.0,
+ "382": 883962368.0,
+ "383": 883962368.0,
+ "384": 883962368.0,
+ "385": 883962368.0,
+ "386": 883962368.0,
+ "387": 883962368.0,
+ "388": 883962368.0,
+ "389": 883962368.0,
+ "390": 883962368.0,
+ "391": 883962368.0,
+ "392": 883962368.0,
+ "393": 883962368.0,
+ "394": 883962368.0,
+ "395": 883962368.0,
+ "396": 883962368.0,
+ "397": 883962368.0,
+ "398": 883962368.0,
+ "399": 883962368.0,
+ "400": 883962368.0,
+ "401": 883962368.0,
+ "402": 883962368.0,
+ "403": 883962368.0,
+ "404": 883962368.0,
+ "405": 883962368.0,
+ "406": 883962368.0,
+ "407": 883962368.0,
+ "408": 883962368.0,
+ "409": 883962368.0,
+ "410": 883962368.0,
+ "411": 883962368.0,
+ "412": 883962368.0,
+ "413": 883962368.0,
+ "414": 883962368.0,
+ "415": 883962368.0,
+ "416": 883962368.0,
+ "417": 883962368.0,
+ "418": 883962368.0,
+ "419": 883962368.0,
+ "420": 883962368.0,
+ "421": 883962368.0,
+ "422": 883962368.0,
+ "423": 883962368.0,
+ "424": 883962368.0,
+ "425": 883962368.0,
+ "426": 883962368.0,
+ "427": 883962368.0,
+ "428": 883962368.0,
+ "429": 883962368.0,
+ "430": 883962368.0,
+ "431": 883962368.0,
+ "432": 883962368.0,
+ "433": 883962368.0,
+ "434": 883962368.0,
+ "435": 883962368.0,
+ "436": 883962368.0,
+ "437": 883962368.0,
+ "438": 883962368.0,
+ "439": 883962368.0,
+ "440": 883962368.0,
+ "441": 883962368.0,
+ "442": 883962368.0,
+ "443": 883962368.0,
+ "444": 883962368.0,
+ "445": 883962368.0,
+ "446": 883962368.0,
+ "447": 883962368.0,
+ "448": 883962368.0,
+ "449": 883962368.0,
+ "450": 883962368.0,
+ "451": 883962368.0,
+ "452": 883962368.0,
+ "453": 883962368.0,
+ "454": 883962368.0,
+ "455": 883962368.0,
+ "456": 883962368.0,
+ "457": 883962368.0,
+ "458": 883962368.0,
+ "459": 883962368.0,
+ "460": 883962368.0,
+ "461": 883962368.0,
+ "462": 883962368.0,
+ "463": 883962368.0,
+ "464": 883962368.0,
+ "465": 883962368.0,
+ "466": 883962368.0,
+ "467": 883962368.0,
+ "468": 883962368.0,
+ "469": 883962368.0,
+ "470": 883962368.0,
+ "471": 883962368.0,
+ "472": 883962368.0,
+ "473": 883962368.0,
+ "474": 883962368.0,
+ "475": 883962368.0,
+ "476": 883962368.0,
+ "477": 883962368.0,
+ "478": 883962368.0,
+ "479": 883962368.0,
+ "480": 883962368.0,
+ "481": 883962368.0,
+ "482": 883962368.0,
+ "483": 883962368.0,
+ "484": 883962368.0,
+ "485": 883962368.0,
+ "486": 883962368.0,
+ "487": 883962368.0,
+ "488": 883962368.0,
+ "489": 883962368.0,
+ "490": 883962368.0,
+ "491": 883962368.0,
+ "492": 883962368.0,
+ "493": 883962368.0,
+ "494": 883962368.0,
+ "495": 883962368.0,
+ "496": 883962368.0,
+ "497": 883962368.0,
+ "498": 883962368.0,
+ "499": 883962368.0,
+ "500": 883962368.0,
+ "501": 883962368.0,
+ "502": 883962368.0,
+ "503": 883962368.0,
+ "504": 883962368.0,
+ "505": 883962368.0,
+ "506": 883962368.0,
+ "507": 883962368.0,
+ "508": 883962368.0,
+ "509": 883962368.0,
+ "510": 883962368.0,
+ "511": 883962368.0,
+ "512": 883962368.0,
+ "513": 883962368.0,
+ "514": 883962368.0,
+ "515": 883962368.0,
+ "516": 883962368.0,
+ "517": 883962368.0,
+ "518": 883962368.0,
+ "519": 883962368.0,
+ "520": 883962368.0,
+ "521": 883962368.0,
+ "522": 883962368.0,
+ "523": 883962368.0,
+ "524": 883962368.0,
+ "525": 883962368.0,
+ "526": 883962368.0,
+ "527": 883962368.0,
+ "528": 883962368.0,
+ "529": 883962368.0,
+ "530": 883962368.0,
+ "531": 883962368.0,
+ "532": 883962368.0,
+ "533": 883962368.0,
+ "534": 883962368.0,
+ "535": 883962368.0,
+ "536": 883962368.0,
+ "537": 883962368.0,
+ "538": 883962368.0,
+ "539": 883962368.0,
+ "540": 883962368.0,
+ "541": 883962368.0,
+ "542": 883962368.0,
+ "543": 883962368.0,
+ "544": 883962368.0,
+ "545": 883962368.0,
+ "546": 883962368.0,
+ "547": 883962368.0,
+ "548": 883962368.0,
+ "549": 883962368.0,
+ "550": 883962368.0,
+ "551": 883962368.0,
+ "552": 883962368.0,
+ "553": 883962368.0,
+ "554": 883962368.0,
+ "555": 883962368.0,
+ "556": 883962368.0,
+ "557": 883962368.0,
+ "558": 883962368.0,
+ "559": 883962368.0,
+ "560": 883962368.0,
+ "561": 883962368.0,
+ "562": 883962368.0,
+ "563": 883962368.0,
+ "564": 883962368.0,
+ "565": 883962368.0,
+ "566": 883962368.0,
+ "567": 883962368.0,
+ "568": 883962368.0,
+ "569": 883962368.0,
+ "570": 883962368.0,
+ "571": 883962368.0,
+ "572": 883962368.0,
+ "573": 883962368.0,
+ "574": 883962368.0,
+ "575": 883962368.0,
+ "576": 883962368.0,
+ "577": 883962368.0,
+ "578": 883962368.0,
+ "579": 883962368.0,
+ "580": 883962368.0,
+ "581": 883962368.0,
+ "582": 883962368.0,
+ "583": 883962368.0,
+ "584": 883962368.0,
+ "585": 883962368.0,
+ "586": 883962368.0,
+ "587": 883962368.0,
+ "588": 883962368.0,
+ "589": 883962368.0,
+ "590": 883962368.0,
+ "591": 883962368.0,
+ "592": 883962368.0,
+ "593": 883962368.0,
+ "594": 883962368.0,
+ "595": 883962368.0,
+ "596": 883962368.0,
+ "597": 883962368.0,
+ "598": 883962368.0,
+ "599": 883962368.0,
+ "600": 883962368.0,
+ "601": 883962368.0,
+ "602": 883962368.0,
+ "603": 883962368.0,
+ "604": 883962368.0,
+ "605": 883962368.0,
+ "606": 883962368.0,
+ "607": 883962368.0,
+ "608": 883962368.0,
+ "609": 883962368.0,
+ "610": 883962368.0,
+ "611": 883962368.0,
+ "612": 883962368.0,
+ "613": 883962368.0,
+ "614": 883962368.0,
+ "615": 883962368.0,
+ "616": 883962368.0,
+ "617": 883962368.0,
+ "618": 883962368.0,
+ "619": 883962368.0,
+ "620": 883962368.0,
+ "621": 883962368.0,
+ "622": 883962368.0,
+ "623": 883962368.0,
+ "624": 883962368.0,
+ "625": 883962368.0,
+ "626": 883962368.0,
+ "627": 883962368.0,
+ "628": 883962368.0,
+ "629": 883962368.0,
+ "630": 883962368.0,
+ "631": 883962368.0,
+ "632": 883962368.0,
+ "633": 883962368.0,
+ "634": 883962368.0,
+ "635": 883962368.0,
+ "636": 883962368.0,
+ "637": 883962368.0,
+ "638": 883962368.0,
+ "639": 883962368.0,
+ "640": 883962368.0,
+ "641": 883962368.0,
+ "642": 883962368.0,
+ "643": 883962368.0,
+ "644": 883962368.0,
+ "645": 883962368.0,
+ "646": 883962368.0,
+ "647": 883962368.0,
+ "648": 883962368.0,
+ "649": 883962368.0,
+ "650": 883962368.0,
+ "651": 883962368.0,
+ "652": 883962368.0,
+ "653": 883962368.0,
+ "654": 883962368.0,
+ "655": 883962368.0,
+ "656": 883962368.0,
+ "657": 883962368.0,
+ "658": 883962368.0,
+ "659": 883962368.0,
+ "660": 883962368.0,
+ "661": 883962368.0,
+ "662": 883962368.0,
+ "663": 883962368.0,
+ "664": 883962368.0,
+ "665": 883962368.0,
+ "666": 883962368.0,
+ "667": 883962368.0,
+ "668": 883962368.0,
+ "669": 883962368.0,
+ "670": 883962368.0,
+ "671": 883962368.0,
+ "672": 883962368.0,
+ "673": 883962368.0,
+ "674": 883962368.0,
+ "675": 883962368.0,
+ "676": 883962368.0,
+ "677": 883962368.0,
+ "678": 883962368.0,
+ "679": 883962368.0,
+ "680": 883962368.0,
+ "681": 883962368.0,
+ "682": 883962368.0,
+ "683": 883962368.0,
+ "684": 883962368.0,
+ "685": 883962368.0,
+ "686": 883962368.0,
+ "687": 883962368.0,
+ "688": 883962368.0,
+ "689": 883962368.0,
+ "690": 883962368.0,
+ "691": 883962368.0,
+ "692": 883962368.0,
+ "693": 883962368.0,
+ "694": 883962368.0,
+ "695": 883962368.0,
+ "696": 883962368.0,
+ "697": 883962368.0,
+ "698": 883962368.0,
+ "699": 883962368.0,
+ "700": 883962368.0,
+ "701": 883962368.0,
+ "702": 883962368.0,
+ "703": 883962368.0,
+ "704": 883962368.0,
+ "705": 883962368.0,
+ "706": 883962368.0,
+ "707": 883962368.0,
+ "708": 883962368.0,
+ "709": 883962368.0,
+ "710": 883962368.0,
+ "711": 883962368.0,
+ "712": 883962368.0,
+ "713": 883962368.0,
+ "714": 883962368.0,
+ "715": 883962368.0,
+ "716": 883962368.0,
+ "717": 883962368.0,
+ "718": 883962368.0,
+ "719": 883962368.0,
+ "720": 883962368.0,
+ "721": 883962368.0,
+ "722": 883962368.0,
+ "723": 883962368.0,
+ "724": 883962368.0,
+ "725": 883962368.0,
+ "726": 883962368.0,
+ "727": 883962368.0,
+ "728": 883962368.0,
+ "729": 883962368.0,
+ "730": 883962368.0,
+ "731": 883962368.0,
+ "732": 883962368.0,
+ "733": 883962368.0,
+ "734": 883962368.0,
+ "735": 883962368.0,
+ "736": 883962368.0,
+ "737": 883962368.0,
+ "738": 883962368.0,
+ "739": 883962368.0,
+ "740": 883962368.0,
+ "741": 883962368.0,
+ "742": 883962368.0,
+ "743": 883962368.0,
+ "744": 883962368.0,
+ "745": 883962368.0,
+ "746": 883962368.0,
+ "747": 883962368.0,
+ "748": 883962368.0,
+ "749": 883962368.0,
+ "750": 883962368.0,
+ "751": 883962368.0,
+ "752": 883962368.0,
+ "753": 883962368.0,
+ "754": 883962368.0,
+ "755": 883962368.0,
+ "756": 883962368.0,
+ "757": 883962368.0,
+ "758": 883962368.0,
+ "759": 883962368.0,
+ "760": 883962368.0,
+ "761": 883962368.0,
+ "762": 883962368.0,
+ "763": 883962368.0,
+ "764": 883962368.0,
+ "765": 883962368.0,
+ "766": 883962368.0,
+ "767": 883962368.0,
+ "768": 883962368.0,
+ "769": 883962368.0,
+ "770": 883962368.0,
+ "771": 883962368.0,
+ "772": 883962368.0,
+ "773": 883962368.0,
+ "774": 883962368.0,
+ "775": 883962368.0,
+ "776": 883962368.0,
+ "777": 883962368.0,
+ "778": 883962368.0,
+ "779": 883962368.0,
+ "780": 883962368.0,
+ "781": 883962368.0,
+ "782": 883962368.0,
+ "783": 883962368.0,
+ "784": 883962368.0,
+ "785": 883962368.0,
+ "786": 883962368.0,
+ "787": 883962368.0,
+ "788": 883962368.0,
+ "789": 883962368.0,
+ "790": 883962368.0,
+ "791": 883962368.0,
+ "792": 883962368.0,
+ "793": 883962368.0,
+ "794": 883962368.0,
+ "795": 883962368.0,
+ "796": 883962368.0,
+ "797": 883962368.0,
+ "798": 883962368.0,
+ "799": 883962368.0,
+ "800": 883962368.0,
+ "801": 883962368.0,
+ "802": 883962368.0,
+ "803": 883962368.0,
+ "804": 883962368.0,
+ "805": 883962368.0,
+ "806": 883962368.0,
+ "807": 883962368.0,
+ "808": 883962368.0,
+ "809": 883962368.0,
+ "810": 883962368.0,
+ "811": 883962368.0,
+ "812": 883962368.0,
+ "813": 883962368.0,
+ "814": 883962368.0,
+ "815": 883962368.0,
+ "816": 883962368.0,
+ "817": 883962368.0,
+ "818": 883962368.0,
+ "819": 883962368.0,
+ "820": 883962368.0,
+ "821": 883962368.0,
+ "822": 883962368.0,
+ "823": 883962368.0,
+ "824": 883962368.0,
+ "825": 883962368.0,
+ "826": 883962368.0,
+ "827": 883962368.0,
+ "828": 883962368.0,
+ "829": 883962368.0,
+ "830": 883962368.0,
+ "831": 883962368.0,
+ "832": 883962368.0,
+ "833": 883962368.0,
+ "834": 883962368.0,
+ "835": 883962368.0,
+ "836": 883962368.0,
+ "837": 883962368.0,
+ "838": 883962368.0,
+ "839": 883962368.0,
+ "840": 883962368.0,
+ "841": 883962368.0,
+ "842": 883962368.0,
+ "843": 883962368.0,
+ "844": 883962368.0,
+ "845": 883962368.0,
+ "846": 883962368.0,
+ "847": 883962368.0,
+ "848": 883962368.0,
+ "849": 883962368.0,
+ "850": 883962368.0,
+ "851": 883962368.0,
+ "852": 883962368.0,
+ "853": 883962368.0,
+ "854": 883962368.0,
+ "855": 883962368.0,
+ "856": 883962368.0,
+ "857": 883962368.0,
+ "858": 883962368.0,
+ "859": 883962368.0,
+ "860": 883962368.0,
+ "861": 883962368.0,
+ "862": 883962368.0,
+ "863": 883962368.0,
+ "864": 883962368.0,
+ "865": 883962368.0,
+ "866": 883962368.0,
+ "867": 883962368.0,
+ "868": 883962368.0,
+ "869": 883962368.0,
+ "870": 883962368.0,
+ "871": 883962368.0,
+ "872": 883962368.0,
+ "873": 883962368.0,
+ "874": 883962368.0,
+ "875": 883962368.0,
+ "876": 883962368.0,
+ "877": 883962368.0,
+ "878": 883962368.0,
+ "879": 883962368.0,
+ "880": 883962368.0,
+ "881": 883962368.0,
+ "882": 883962368.0,
+ "883": 883962368.0,
+ "884": 883962368.0,
+ "885": 883962368.0,
+ "886": 883962368.0,
+ "887": 883962368.0,
+ "888": 883962368.0,
+ "889": 883962368.0,
+ "890": 883962368.0,
+ "891": 883962368.0,
+ "892": 883962368.0,
+ "893": 883962368.0,
+ "894": 883962368.0,
+ "895": 883962368.0,
+ "896": 883962368.0,
+ "897": 883962368.0,
+ "898": 883962368.0,
+ "899": 883962368.0,
+ "900": 883962368.0,
+ "901": 883962368.0,
+ "902": 883962368.0,
+ "903": 883962368.0,
+ "904": 883962368.0,
+ "905": 883962368.0,
+ "906": 883962368.0,
+ "907": 883962368.0,
+ "908": 883962368.0,
+ "909": 883962368.0,
+ "910": 883962368.0,
+ "911": 883962368.0,
+ "912": 883962368.0,
+ "913": 883962368.0,
+ "914": 883962368.0,
+ "915": 883962368.0,
+ "916": 883962368.0,
+ "917": 883962368.0,
+ "918": 883962368.0,
+ "919": 883962368.0,
+ "920": 883962368.0,
+ "921": 883962368.0,
+ "922": 883962368.0,
+ "923": 883962368.0,
+ "924": 883962368.0,
+ "925": 883962368.0,
+ "926": 883962368.0,
+ "927": 883962368.0,
+ "928": 883962368.0,
+ "929": 883962368.0,
+ "930": 883962368.0,
+ "931": 883962368.0,
+ "932": 883962368.0,
+ "933": 883962368.0,
+ "934": 883962368.0,
+ "935": 883962368.0,
+ "936": 883962368.0,
+ "937": 883962368.0,
+ "938": 883962368.0,
+ "939": 883962368.0,
+ "940": 883962368.0,
+ "941": 883962368.0,
+ "942": 883962368.0,
+ "943": 883962368.0,
+ "944": 883962368.0,
+ "945": 883962368.0,
+ "946": 883962368.0,
+ "947": 883962368.0,
+ "948": 883962368.0,
+ "949": 883962368.0,
+ "950": 883962368.0,
+ "951": 883962368.0,
+ "952": 883962368.0,
+ "953": 883962368.0,
+ "954": 883962368.0,
+ "955": 883962368.0,
+ "956": 883962368.0,
+ "957": 883962368.0,
+ "958": 883962368.0,
+ "959": 883962368.0,
+ "960": 883962368.0,
+ "961": 883962368.0,
+ "962": 883962368.0,
+ "963": 883962368.0,
+ "964": 883962368.0,
+ "965": 883962368.0,
+ "966": 883962368.0,
+ "967": 883962368.0,
+ "968": 883962368.0,
+ "969": 883962368.0,
+ "970": 883962368.0,
+ "971": 883962368.0,
+ "972": 883962368.0,
+ "973": 883962368.0,
+ "974": 883962368.0,
+ "975": 883962368.0,
+ "976": 883962368.0,
+ "977": 883962368.0,
+ "978": 883962368.0,
+ "979": 883962368.0,
+ "980": 883962368.0,
+ "981": 883962368.0,
+ "982": 883962368.0,
+ "983": 883962368.0,
+ "984": 883962368.0,
+ "985": 883962368.0,
+ "986": 883962368.0,
+ "987": 883962368.0,
+ "988": 883962368.0,
+ "989": 883962368.0,
+ "990": 883962368.0,
+ "991": 883962368.0,
+ "992": 883962368.0,
+ "993": 883962368.0,
+ "994": 883962368.0,
+ "995": 883962368.0,
+ "996": 883962368.0,
+ "997": 883962368.0,
+ "998": 883962368.0,
+ "999": 883962368.0,
+ "1000": 883962368.0,
+ "1001": 883962368.0,
+ "1002": 883962368.0,
+ "1003": 883962368.0,
+ "1004": 883962368.0,
+ "1005": 883962368.0,
+ "1006": 883962368.0,
+ "1007": 883962368.0,
+ "1008": 883962368.0,
+ "1009": 883962368.0,
+ "1010": 883962368.0,
+ "1011": 883962368.0,
+ "1012": 883962368.0,
+ "1013": 883962368.0,
+ "1014": 883962368.0,
+ "1015": 883962368.0,
+ "1016": 883962368.0,
+ "1017": 883962368.0,
+ "1018": 883962368.0,
+ "1019": 883962368.0,
+ "1020": 883962368.0,
+ "1021": 883962368.0,
+ "1022": 883962368.0,
+ "1023": 883962368.0,
+ "1024": 883962368.0,
+ "1025": 883962368.0,
+ "1026": 883962368.0,
+ "1027": 883962368.0,
+ "1028": 883962368.0,
+ "1029": 883962368.0,
+ "1030": 883962368.0,
+ "1031": 883962368.0,
+ "1032": 883962368.0,
+ "1033": 883962368.0,
+ "1034": 883962368.0,
+ "1035": 883962368.0,
+ "1036": 883962368.0,
+ "1037": 883962368.0,
+ "1038": 883962368.0,
+ "1039": 883962368.0,
+ "1040": 883962368.0,
+ "1041": 883962368.0,
+ "1042": 883962368.0,
+ "1043": 883962368.0,
+ "1044": 883962368.0,
+ "1045": 883962368.0,
+ "1046": 883962368.0,
+ "1047": 883962368.0,
+ "1048": 883962368.0,
+ "1049": 883962368.0,
+ "1050": 883962368.0,
+ "1051": 883962368.0,
+ "1052": 883962368.0,
+ "1053": 883962368.0,
+ "1054": 883962368.0,
+ "1055": 883962368.0,
+ "1056": 883962368.0,
+ "1057": 883962368.0,
+ "1058": 883962368.0,
+ "1059": 883962368.0,
+ "1060": 883962368.0,
+ "1061": 883962368.0,
+ "1062": 883962368.0,
+ "1063": 883962368.0,
+ "1064": 883962368.0,
+ "1065": 883962368.0,
+ "1066": 883962368.0,
+ "1067": 883962368.0,
+ "1068": 883962368.0,
+ "1069": 883962368.0,
+ "1070": 883962368.0,
+ "1071": 883962368.0,
+ "1072": 883962368.0,
+ "1073": 883962368.0,
+ "1074": 883962368.0,
+ "1075": 883962368.0,
+ "1076": 883962368.0,
+ "1077": 883962368.0,
+ "1078": 883962368.0,
+ "1079": 883962368.0,
+ "1080": 883962368.0,
+ "1081": 883962368.0,
+ "1082": 883962368.0,
+ "1083": 883962368.0,
+ "1084": 883962368.0,
+ "1085": 883962368.0,
+ "1086": 883962368.0,
+ "1087": 883962368.0,
+ "1088": 883962368.0,
+ "1089": 883962368.0,
+ "1090": 883962368.0,
+ "1091": 883962368.0,
+ "1092": 883962368.0,
+ "1093": 883962368.0,
+ "1094": 883962368.0,
+ "1095": 883962368.0,
+ "1096": 883962368.0,
+ "1097": 883962368.0,
+ "1098": 883962368.0,
+ "1099": 883962368.0,
+ "1100": 883962368.0,
+ "1101": 883962368.0,
+ "1102": 883962368.0,
+ "1103": 883962368.0,
+ "1104": 883962368.0,
+ "1105": 883962368.0,
+ "1106": 883962368.0,
+ "1107": 883962368.0,
+ "1108": 883962368.0,
+ "1109": 883962368.0,
+ "1110": 883962368.0,
+ "1111": 883962368.0,
+ "1112": 883962368.0,
+ "1113": 883962368.0,
+ "1114": 883962368.0,
+ "1115": 883962368.0,
+ "1116": 883962368.0,
+ "1117": 883962368.0,
+ "1118": 883962368.0,
+ "1119": 883962368.0,
+ "1120": 883962368.0,
+ "1121": 883962368.0,
+ "1122": 883962368.0,
+ "1123": 883962368.0,
+ "1124": 883962368.0,
+ "1125": 883962368.0,
+ "1126": 883962368.0,
+ "1127": 883962368.0,
+ "1128": 883962368.0,
+ "1129": 883962368.0,
+ "1130": 883962368.0,
+ "1131": 883962368.0,
+ "1132": 883962368.0,
+ "1133": 883962368.0,
+ "1134": 883962368.0,
+ "1135": 883962368.0,
+ "1136": 883962368.0,
+ "1137": 883962368.0,
+ "1138": 883962368.0,
+ "1139": 883962368.0,
+ "1140": 883962368.0,
+ "1141": 883962368.0,
+ "1142": 883962368.0,
+ "1143": 883962368.0,
+ "1144": 883962368.0,
+ "1145": 883962368.0,
+ "1146": 883962368.0,
+ "1147": 883962368.0,
+ "1148": 883962368.0,
+ "1149": 883962368.0,
+ "1150": 883962368.0,
+ "1151": 883962368.0,
+ "1152": 883962368.0,
+ "1153": 883962368.0,
+ "1154": 883962368.0,
+ "1155": 883962368.0,
+ "1156": 883962368.0,
+ "1157": 883962368.0,
+ "1158": 883962368.0,
+ "1159": 883962368.0,
+ "1160": 883962368.0,
+ "1161": 883962368.0,
+ "1162": 883962368.0,
+ "1163": 883962368.0,
+ "1164": 883962368.0,
+ "1165": 883962368.0,
+ "1166": 883962368.0,
+ "1167": 883962368.0,
+ "1168": 883962368.0,
+ "1169": 883962368.0,
+ "1170": 883962368.0,
+ "1171": 883962368.0,
+ "1172": 883962368.0,
+ "1173": 883962368.0,
+ "1174": 883962368.0,
+ "1175": 883962368.0,
+ "1176": 883962368.0,
+ "1177": 883962368.0,
+ "1178": 883962368.0,
+ "1179": 883962368.0,
+ "1180": 883962368.0,
+ "1181": 883962368.0,
+ "1182": 883962368.0,
+ "1183": 883962368.0,
+ "1184": 883962368.0,
+ "1185": 883962368.0,
+ "1186": 883962368.0,
+ "1187": 883962368.0,
+ "1188": 883962368.0,
+ "1189": 883962368.0,
+ "1190": 883962368.0,
+ "1191": 883962368.0,
+ "1192": 883962368.0,
+ "1193": 883962368.0,
+ "1194": 883962368.0,
+ "1195": 883962368.0,
+ "1196": 883962368.0,
+ "1197": 883962368.0,
+ "1198": 883962368.0,
+ "1199": 883962368.0,
+ "1200": 883962368.0,
+ "1201": 883962368.0,
+ "1202": 883962368.0,
+ "1203": 883962368.0,
+ "1204": 883962368.0,
+ "1205": 883962368.0,
+ "1206": 883962368.0,
+ "1207": 883962368.0,
+ "1208": 883962368.0,
+ "1209": 883962368.0,
+ "1210": 883962368.0,
+ "1211": 883962368.0,
+ "1212": 883962368.0,
+ "1213": 883962368.0,
+ "1214": 883962368.0,
+ "1215": 883962368.0,
+ "1216": 883962368.0,
+ "1217": 883962368.0,
+ "1218": 883962368.0,
+ "1219": 883962368.0,
+ "1220": 883962368.0,
+ "1221": 883962368.0,
+ "1222": 883962368.0,
+ "1223": 883962368.0,
+ "1224": 883962368.0,
+ "1225": 883962368.0,
+ "1226": 883962368.0,
+ "1227": 883962368.0,
+ "1228": 883962368.0,
+ "1229": 883962368.0,
+ "1230": 883962368.0,
+ "1231": 883962368.0,
+ "1232": 883962368.0,
+ "1233": 883962368.0,
+ "1234": 883962368.0,
+ "1235": 883962368.0,
+ "1236": 883962368.0,
+ "1237": 883962368.0,
+ "1238": 883962368.0,
+ "1239": 883962368.0,
+ "1240": 883962368.0,
+ "1241": 883962368.0,
+ "1242": 883962368.0,
+ "1243": 883962368.0,
+ "1244": 883962368.0,
+ "1245": 883962368.0,
+ "1246": 883962368.0,
+ "1247": 883962368.0,
+ "1248": 883962368.0,
+ "1249": 883962368.0,
+ "1250": 883962368.0,
+ "1251": 883962368.0,
+ "1252": 883962368.0,
+ "1253": 883962368.0,
+ "1254": 883962368.0,
+ "1255": 883962368.0,
+ "1256": 883962368.0,
+ "1257": 883962368.0,
+ "1258": 883962368.0,
+ "1259": 883962368.0,
+ "1260": 883962368.0,
+ "1261": 883962368.0,
+ "1262": 883962368.0,
+ "1263": 883962368.0,
+ "1264": 883962368.0,
+ "1265": 883962368.0,
+ "1266": 883962368.0,
+ "1267": 883962368.0,
+ "1268": 883962368.0,
+ "1269": 883962368.0,
+ "1270": 883962368.0,
+ "1271": 883962368.0,
+ "1272": 883962368.0,
+ "1273": 883962368.0,
+ "1274": 883962368.0,
+ "1275": 883962368.0,
+ "1276": 883962368.0,
+ "1277": 883962368.0,
+ "1278": 883962368.0,
+ "1279": 883962368.0,
+ "1280": 883962368.0,
+ "1281": 883962368.0,
+ "1282": 883962368.0,
+ "1283": 883962368.0,
+ "1284": 883962368.0,
+ "1285": 883962368.0,
+ "1286": 883962368.0,
+ "1287": 883962368.0,
+ "1288": 883962368.0,
+ "1289": 883962368.0,
+ "1290": 883962368.0,
+ "1291": 883962368.0,
+ "1292": 883962368.0,
+ "1293": 883962368.0,
+ "1294": 883962368.0,
+ "1295": 883962368.0,
+ "1296": 883962368.0,
+ "1297": 883962368.0,
+ "1298": 883962368.0,
+ "1299": 883962368.0,
+ "1300": 883962368.0,
+ "1301": 883962368.0,
+ "1302": 883962368.0,
+ "1303": 883962368.0,
+ "1304": 883962368.0,
+ "1305": 883962368.0,
+ "1306": 883962368.0,
+ "1307": 883962368.0,
+ "1308": 883962368.0,
+ "1309": 883962368.0,
+ "1310": 883962368.0,
+ "1311": 883962368.0,
+ "1312": 883962368.0,
+ "1313": 883962368.0,
+ "1314": 883962368.0,
+ "1315": 883962368.0,
+ "1316": 883962368.0,
+ "1317": 883962368.0,
+ "1318": 883962368.0,
+ "1319": 883962368.0,
+ "1320": 883962368.0,
+ "1321": 883962368.0,
+ "1322": 883962368.0,
+ "1323": 883962368.0,
+ "1324": 883962368.0,
+ "1325": 883962368.0,
+ "1326": 883962368.0,
+ "1327": 883962368.0,
+ "1328": 883962368.0,
+ "1329": 883962368.0,
+ "1330": 883962368.0,
+ "1331": 883962368.0,
+ "1332": 883962368.0,
+ "1333": 883962368.0,
+ "1334": 883962368.0,
+ "1335": 883962368.0,
+ "1336": 883962368.0,
+ "1337": 883962368.0,
+ "1338": 883962368.0,
+ "1339": 883962368.0,
+ "1340": 883962368.0,
+ "1341": 883962368.0,
+ "1342": 883962368.0,
+ "1343": 883962368.0,
+ "1344": 883962368.0,
+ "1345": 883962368.0,
+ "1346": 883962368.0,
+ "1347": 883962368.0,
+ "1348": 883962368.0,
+ "1349": 883962368.0,
+ "1350": 883962368.0,
+ "1351": 883962368.0,
+ "1352": 883962368.0,
+ "1353": 883962368.0,
+ "1354": 883962368.0,
+ "1355": 883962368.0,
+ "1356": 883962368.0,
+ "1357": 883962368.0,
+ "1358": 883962368.0,
+ "1359": 883962368.0,
+ "1360": 883962368.0,
+ "1361": 883962368.0,
+ "1362": 883962368.0,
+ "1363": 883962368.0,
+ "1364": 883962368.0,
+ "1365": 883962368.0,
+ "1366": 883962368.0,
+ "1367": 883962368.0,
+ "1368": 883962368.0,
+ "1369": 883962368.0,
+ "1370": 883962368.0,
+ "1371": 883962368.0,
+ "1372": 883962368.0,
+ "1373": 883962368.0,
+ "1374": 883962368.0,
+ "1375": 883962368.0,
+ "1376": 883962368.0,
+ "1377": 883962368.0,
+ "1378": 883962368.0,
+ "1379": 883962368.0,
+ "1380": 883962368.0,
+ "1381": 883962368.0,
+ "1382": 883962368.0,
+ "1383": 883962368.0,
+ "1384": 883962368.0,
+ "1385": 883962368.0,
+ "1386": 883962368.0,
+ "1387": 883962368.0,
+ "1388": 883962368.0,
+ "1389": 883962368.0,
+ "1390": 883962368.0,
+ "1391": 883962368.0,
+ "1392": 883962368.0,
+ "1393": 883962368.0,
+ "1394": 883962368.0,
+ "1395": 883962368.0,
+ "1396": 883962368.0,
+ "1397": 883962368.0,
+ "1398": 883962368.0,
+ "1399": 883962368.0,
+ "1400": 883962368.0,
+ "1401": 883962368.0,
+ "1402": 883962368.0,
+ "1403": 883962368.0,
+ "1404": 883962368.0,
+ "1405": 883962368.0,
+ "1406": 883962368.0,
+ "1407": 883962368.0,
+ "1408": 883962368.0,
+ "1409": 883962368.0,
+ "1410": 883962368.0,
+ "1411": 883962368.0,
+ "1412": 883962368.0,
+ "1413": 883962368.0,
+ "1414": 883962368.0,
+ "1415": 883962368.0,
+ "1416": 883962368.0,
+ "1417": 883962368.0,
+ "1418": 883962368.0,
+ "1419": 883962368.0,
+ "1420": 883962368.0,
+ "1421": 883962368.0,
+ "1422": 883962368.0,
+ "1423": 883962368.0,
+ "1424": 883962368.0,
+ "1425": 883962368.0,
+ "1426": 883962368.0,
+ "1427": 883962368.0,
+ "1428": 883962368.0,
+ "1429": 883962368.0,
+ "1430": 883962368.0,
+ "1431": 883962368.0,
+ "1432": 883962368.0,
+ "1433": 883962368.0,
+ "1434": 883962368.0,
+ "1435": 883962368.0,
+ "1436": 883962368.0,
+ "1437": 883962368.0,
+ "1438": 883962368.0,
+ "1439": 883962368.0,
+ "1440": 883962368.0,
+ "1441": 883962368.0,
+ "1442": 883962368.0,
+ "1443": 883962368.0,
+ "1444": 883962368.0,
+ "1445": 883962368.0,
+ "1446": 883962368.0,
+ "1447": 883962368.0,
+ "1448": 883962368.0,
+ "1449": 883962368.0,
+ "1450": 883962368.0,
+ "1451": 883962368.0,
+ "1452": 883962368.0,
+ "1453": 883962368.0,
+ "1454": 883962368.0,
+ "1455": 883962368.0,
+ "1456": 883962368.0,
+ "1457": 883962368.0,
+ "1458": 883962368.0,
+ "1459": 883962368.0,
+ "1460": 883962368.0,
+ "1461": 883962368.0,
+ "1462": 883962368.0,
+ "1463": 883962368.0,
+ "1464": 883962368.0,
+ "1465": 883962368.0,
+ "1466": 883962368.0,
+ "1467": 883962368.0,
+ "1468": 883962368.0,
+ "1469": 883962368.0,
+ "1470": 883962368.0,
+ "1471": 883962368.0,
+ "1472": 883962368.0,
+ "1473": 883962368.0,
+ "1474": 883962368.0,
+ "1475": 883962368.0,
+ "1476": 883962368.0,
+ "1477": 883962368.0,
+ "1478": 883962368.0,
+ "1479": 883962368.0,
+ "1480": 883962368.0,
+ "1481": 883962368.0,
+ "1482": 883962368.0,
+ "1483": 883962368.0,
+ "1484": 883962368.0,
+ "1485": 883962368.0,
+ "1486": 883962368.0,
+ "1487": 883962368.0,
+ "1488": 883962368.0,
+ "1489": 883962368.0,
+ "1490": 883962368.0,
+ "1491": 883962368.0,
+ "1492": 883962368.0,
+ "1493": 883962368.0,
+ "1494": 883962368.0,
+ "1495": 883962368.0,
+ "1496": 883962368.0,
+ "1497": 883962368.0,
+ "1498": 883962368.0,
+ "1499": 883962368.0,
+ "1500": 883962368.0,
+ "1501": 883962368.0,
+ "1502": 883962368.0,
+ "1503": 883962368.0,
+ "1504": 883962368.0,
+ "1505": 883962368.0,
+ "1506": 883962368.0,
+ "1507": 883962368.0,
+ "1508": 883962368.0,
+ "1509": 883962368.0,
+ "1510": 883962368.0,
+ "1511": 883962368.0,
+ "1512": 883962368.0,
+ "1513": 883962368.0,
+ "1514": 883962368.0,
+ "1515": 883962368.0,
+ "1516": 883962368.0,
+ "1517": 883962368.0,
+ "1518": 883962368.0,
+ "1519": 883962368.0,
+ "1520": 883962368.0,
+ "1521": 883962368.0,
+ "1522": 883962368.0,
+ "1523": 883962368.0,
+ "1524": 883962368.0,
+ "1525": 883962368.0,
+ "1526": 883962368.0,
+ "1527": 883962368.0,
+ "1528": 883962368.0,
+ "1529": 883962368.0,
+ "1530": 883962368.0,
+ "1531": 883962368.0,
+ "1532": 883962368.0,
+ "1533": 883962368.0,
+ "1534": 883962368.0,
+ "1535": 883962368.0,
+ "1536": 883962368.0,
+ "1537": 883962368.0,
+ "1538": 883962368.0,
+ "1539": 883962368.0,
+ "1540": 883962368.0,
+ "1541": 883962368.0,
+ "1542": 883962368.0,
+ "1543": 883962368.0,
+ "1544": 883962368.0,
+ "1545": 883962368.0,
+ "1546": 883962368.0,
+ "1547": 883962368.0,
+ "1548": 883962368.0,
+ "1549": 883962368.0,
+ "1550": 883962368.0,
+ "1551": 883962368.0,
+ "1552": 883962368.0,
+ "1553": 883962368.0,
+ "1554": 883962368.0,
+ "1555": 883962368.0,
+ "1556": 883962368.0,
+ "1557": 883962368.0,
+ "1558": 883962368.0,
+ "1559": 883962368.0,
+ "1560": 883962368.0,
+ "1561": 883962368.0,
+ "1562": 883962368.0,
+ "1563": 883962368.0,
+ "1564": 883962368.0,
+ "1565": 883962368.0,
+ "1566": 883962368.0,
+ "1567": 883962368.0,
+ "1568": 883962368.0,
+ "1569": 883962368.0,
+ "1570": 883962368.0,
+ "1571": 883962368.0,
+ "1572": 883962368.0,
+ "1573": 883962368.0,
+ "1574": 883962368.0,
+ "1575": 883962368.0,
+ "1576": 883962368.0,
+ "1577": 883962368.0,
+ "1578": 883962368.0,
+ "1579": 883962368.0,
+ "1580": 883962368.0,
+ "1581": 883962368.0,
+ "1582": 883962368.0,
+ "1583": 883962368.0,
+ "1584": 883962368.0,
+ "1585": 883962368.0,
+ "1586": 883962368.0,
+ "1587": 883962368.0,
+ "1588": 883962368.0,
+ "1589": 883962368.0,
+ "1590": 883962368.0,
+ "1591": 883962368.0,
+ "1592": 883962368.0,
+ "1593": 883962368.0,
+ "1594": 883962368.0,
+ "1595": 883962368.0,
+ "1596": 883962368.0,
+ "1597": 883962368.0,
+ "1598": 883962368.0,
+ "1599": 883962368.0,
+ "1600": 883962368.0,
+ "1601": 883962368.0,
+ "1602": 883962368.0,
+ "1603": 883962368.0,
+ "1604": 883962368.0,
+ "1605": 883962368.0,
+ "1606": 883962368.0,
+ "1607": 883962368.0,
+ "1608": 883962368.0,
+ "1609": 883962368.0,
+ "1610": 883962368.0,
+ "1611": 883962368.0,
+ "1612": 883962368.0,
+ "1613": 883962368.0,
+ "1614": 883962368.0,
+ "1615": 883962368.0,
+ "1616": 883962368.0,
+ "1617": 883962368.0,
+ "1618": 883962368.0,
+ "1619": 883962368.0,
+ "1620": 883962368.0,
+ "1621": 883962368.0,
+ "1622": 883962368.0,
+ "1623": 883962368.0,
+ "1624": 883962368.0,
+ "1625": 883962368.0,
+ "1626": 883962368.0,
+ "1627": 883962368.0,
+ "1628": 883962368.0,
+ "1629": 883962368.0,
+ "1630": 883962368.0,
+ "1631": 883962368.0,
+ "1632": 883962368.0,
+ "1633": 883962368.0,
+ "1634": 883962368.0,
+ "1635": 883962368.0,
+ "1636": 883962368.0,
+ "1637": 883962368.0,
+ "1638": 883962368.0,
+ "1639": 883962368.0,
+ "1640": 883962368.0,
+ "1641": 883962368.0,
+ "1642": 883962368.0,
+ "1643": 883962368.0,
+ "1644": 883962368.0,
+ "1645": 883962368.0,
+ "1646": 883962368.0,
+ "1647": 883962368.0,
+ "1648": 883962368.0,
+ "1649": 883962368.0,
+ "1650": 883962368.0,
+ "1651": 883962368.0,
+ "1652": 883962368.0,
+ "1653": 883962368.0,
+ "1654": 883962368.0,
+ "1655": 883962368.0,
+ "1656": 883962368.0,
+ "1657": 883962368.0,
+ "1658": 883962368.0,
+ "1659": 883962368.0,
+ "1660": 883962368.0,
+ "1661": 883962368.0,
+ "1662": 883962368.0,
+ "1663": 883962368.0,
+ "1664": 883962368.0,
+ "1665": 883962368.0,
+ "1666": 883962368.0,
+ "1667": 883962368.0,
+ "1668": 883962368.0,
+ "1669": 883962368.0,
+ "1670": 883962368.0,
+ "1671": 883962368.0,
+ "1672": 883962368.0,
+ "1673": 883962368.0,
+ "1674": 883962368.0,
+ "1675": 883962368.0,
+ "1676": 883962368.0,
+ "1677": 883962368.0,
+ "1678": 883962368.0,
+ "1679": 883962368.0,
+ "1680": 883962368.0,
+ "1681": 883962368.0,
+ "1682": 883962368.0,
+ "1683": 883962368.0,
+ "1684": 883962368.0,
+ "1685": 883962368.0,
+ "1686": 883962368.0,
+ "1687": 883962368.0,
+ "1688": 883962368.0,
+ "1689": 883962368.0,
+ "1690": 883962368.0,
+ "1691": 883962368.0,
+ "1692": 883962368.0,
+ "1693": 883962368.0,
+ "1694": 883962368.0,
+ "1695": 883962368.0,
+ "1696": 883962368.0,
+ "1697": 883962368.0,
+ "1698": 883962368.0,
+ "1699": 883962368.0,
+ "1700": 883962368.0,
+ "1701": 883962368.0,
+ "1702": 883962368.0,
+ "1703": 883962368.0,
+ "1704": 883962368.0,
+ "1705": 883962368.0,
+ "1706": 883962368.0,
+ "1707": 883962368.0,
+ "1708": 883962368.0,
+ "1709": 883962368.0,
+ "1710": 883962368.0,
+ "1711": 883962368.0,
+ "1712": 883962368.0,
+ "1713": 883962368.0,
+ "1714": 883962368.0,
+ "1715": 883962368.0,
+ "1716": 883962368.0,
+ "1717": 883962368.0,
+ "1718": 883962368.0,
+ "1719": 883962368.0,
+ "1720": 883962368.0,
+ "1721": 883962368.0,
+ "1722": 883962368.0,
+ "1723": 883962368.0,
+ "1724": 883962368.0,
+ "1725": 883962368.0,
+ "1726": 883962368.0,
+ "1727": 883962368.0,
+ "1728": 883962368.0,
+ "1729": 883962368.0,
+ "1730": 883962368.0,
+ "1731": 883962368.0,
+ "1732": 883962368.0,
+ "1733": 883962368.0,
+ "1734": 883962368.0,
+ "1735": 883962368.0,
+ "1736": 883962368.0,
+ "1737": 883962368.0,
+ "1738": 883962368.0,
+ "1739": 883962368.0,
+ "1740": 883962368.0,
+ "1741": 883962368.0,
+ "1742": 883962368.0,
+ "1743": 883962368.0,
+ "1744": 883962368.0,
+ "1745": 883962368.0,
+ "1746": 883962368.0,
+ "1747": 883962368.0,
+ "1748": 883962368.0,
+ "1749": 883962368.0,
+ "1750": 883962368.0,
+ "1751": 883962368.0,
+ "1752": 883962368.0,
+ "1753": 883962368.0,
+ "1754": 883962368.0,
+ "1755": 883962368.0,
+ "1756": 883962368.0,
+ "1757": 883962368.0,
+ "1758": 883962368.0,
+ "1759": 883962368.0,
+ "1760": 883962368.0,
+ "1761": 883962368.0,
+ "1762": 883962368.0,
+ "1763": 883962368.0,
+ "1764": 883962368.0,
+ "1765": 883962368.0,
+ "1766": 883962368.0,
+ "1767": 883962368.0,
+ "1768": 883962368.0,
+ "1769": 883962368.0,
+ "1770": 883962368.0,
+ "1771": 883962368.0,
+ "1772": 883962368.0,
+ "1773": 883962368.0,
+ "1774": 883962368.0,
+ "1775": 883962368.0,
+ "1776": 883962368.0,
+ "1777": 883962368.0,
+ "1778": 883962368.0,
+ "1779": 883962368.0,
+ "1780": 883962368.0,
+ "1781": 883962368.0,
+ "1782": 883962368.0,
+ "1783": 883962368.0,
+ "1784": 883962368.0,
+ "1785": 883962368.0,
+ "1786": 883962368.0,
+ "1787": 883962368.0,
+ "1788": 883962368.0,
+ "1789": 883962368.0,
+ "1790": 883962368.0,
+ "1791": 883962368.0,
+ "1792": 883962368.0,
+ "1793": 883962368.0,
+ "1794": 883962368.0,
+ "1795": 883962368.0,
+ "1796": 883962368.0,
+ "1797": 883962368.0,
+ "1798": 883962368.0,
+ "1799": 883962368.0,
+ "1800": 883962368.0,
+ "1801": 883962368.0,
+ "1802": 883962368.0,
+ "1803": 883962368.0,
+ "1804": 883962368.0,
+ "1805": 883962368.0,
+ "1806": 883962368.0,
+ "1807": 883962368.0,
+ "1808": 883962368.0,
+ "1809": 883962368.0,
+ "1810": 883962368.0,
+ "1811": 883962368.0,
+ "1812": 883962368.0,
+ "1813": 883962368.0,
+ "1814": 883962368.0,
+ "1815": 883962368.0,
+ "1816": 883962368.0,
+ "1817": 883962368.0,
+ "1818": 883962368.0,
+ "1819": 883962368.0,
+ "1820": 883962368.0,
+ "1821": 883962368.0,
+ "1822": 883962368.0,
+ "1823": 883962368.0,
+ "1824": 883962368.0,
+ "1825": 883962368.0,
+ "1826": 883962368.0,
+ "1827": 883962368.0,
+ "1828": 883962368.0,
+ "1829": 883962368.0,
+ "1830": 883962368.0,
+ "1831": 883962368.0,
+ "1832": 883962368.0,
+ "1833": 883962368.0,
+ "1834": 883962368.0,
+ "1835": 883962368.0,
+ "1836": 883962368.0,
+ "1837": 883962368.0,
+ "1838": 883962368.0,
+ "1839": 883962368.0,
+ "1840": 883962368.0,
+ "1841": 883962368.0,
+ "1842": 883962368.0,
+ "1843": 883962368.0,
+ "1844": 883962368.0,
+ "1845": 883962368.0,
+ "1846": 883962368.0,
+ "1847": 883962368.0,
+ "1848": 883962368.0,
+ "1849": 883962368.0,
+ "1850": 883962368.0,
+ "1851": 883962368.0,
+ "1852": 883962368.0,
+ "1853": 883962368.0,
+ "1854": 883962368.0,
+ "1855": 883962368.0,
+ "1856": 883962368.0,
+ "1857": 883962368.0,
+ "1858": 883962368.0,
+ "1859": 883962368.0,
+ "1860": 883962368.0,
+ "1861": 883962368.0,
+ "1862": 883962368.0,
+ "1863": 883962368.0,
+ "1864": 883962368.0,
+ "1865": 883962368.0,
+ "1866": 883962368.0,
+ "1867": 883962368.0,
+ "1868": 883962368.0,
+ "1869": 883962368.0,
+ "1870": 883962368.0,
+ "1871": 883962368.0,
+ "1872": 883962368.0,
+ "1873": 883962368.0,
+ "1874": 883962368.0,
+ "1875": 883962368.0,
+ "1876": 883962368.0,
+ "1877": 883962368.0,
+ "1878": 883962368.0,
+ "1879": 883962368.0,
+ "1880": 883962368.0,
+ "1881": 883962368.0,
+ "1882": 883962368.0,
+ "1883": 883962368.0,
+ "1884": 883962368.0,
+ "1885": 883962368.0,
+ "1886": 883962368.0,
+ "1887": 883962368.0,
+ "1888": 883962368.0,
+ "1889": 883962368.0,
+ "1890": 883962368.0,
+ "1891": 883962368.0,
+ "1892": 883962368.0,
+ "1893": 883962368.0,
+ "1894": 883962368.0,
+ "1895": 883962368.0,
+ "1896": 883962368.0,
+ "1897": 883962368.0,
+ "1898": 883962368.0,
+ "1899": 883962368.0,
+ "1900": 883962368.0,
+ "1901": 883962368.0,
+ "1902": 883962368.0,
+ "1903": 883962368.0,
+ "1904": 883962368.0,
+ "1905": 883962368.0,
+ "1906": 883962368.0,
+ "1907": 883962368.0,
+ "1908": 883962368.0,
+ "1909": 883962368.0,
+ "1910": 883962368.0,
+ "1911": 883962368.0,
+ "1912": 883962368.0,
+ "1913": 883962368.0,
+ "1914": 883962368.0,
+ "1915": 883962368.0,
+ "1916": 883962368.0,
+ "1917": 883962368.0,
+ "1918": 883962368.0,
+ "1919": 883962368.0,
+ "1920": 883962368.0,
+ "1921": 883962368.0,
+ "1922": 883962368.0,
+ "1923": 883962368.0,
+ "1924": 883962368.0,
+ "1925": 883962368.0,
+ "1926": 883962368.0,
+ "1927": 883962368.0,
+ "1928": 883962368.0,
+ "1929": 883962368.0,
+ "1930": 883962368.0,
+ "1931": 883962368.0,
+ "1932": 883962368.0,
+ "1933": 883962368.0,
+ "1934": 883962368.0,
+ "1935": 883962368.0,
+ "1936": 883962368.0,
+ "1937": 883962368.0,
+ "1938": 883962368.0,
+ "1939": 883962368.0,
+ "1940": 883962368.0,
+ "1941": 883962368.0,
+ "1942": 883962368.0,
+ "1943": 883962368.0,
+ "1944": 883962368.0,
+ "1945": 883962368.0,
+ "1946": 883962368.0,
+ "1947": 883962368.0,
+ "1948": 883962368.0,
+ "1949": 883962368.0,
+ "1950": 883962368.0,
+ "1951": 883962368.0,
+ "1952": 883962368.0,
+ "1953": 883962368.0,
+ "1954": 883962368.0,
+ "1955": 883962368.0,
+ "1956": 883962368.0,
+ "1957": 883962368.0,
+ "1958": 883962368.0,
+ "1959": 883962368.0,
+ "1960": 883962368.0,
+ "1961": 883962368.0,
+ "1962": 883962368.0,
+ "1963": 883962368.0,
+ "1964": 883962368.0,
+ "1965": 883962368.0,
+ "1966": 883962368.0,
+ "1967": 883962368.0,
+ "1968": 883962368.0,
+ "1969": 883962368.0,
+ "1970": 883962368.0,
+ "1971": 883962368.0,
+ "1972": 883962368.0,
+ "1973": 883962368.0,
+ "1974": 883962368.0,
+ "1975": 883962368.0,
+ "1976": 883962368.0,
+ "1977": 883962368.0,
+ "1978": 883962368.0,
+ "1979": 883962368.0,
+ "1980": 883962368.0,
+ "1981": 883962368.0,
+ "1982": 883962368.0,
+ "1983": 883962368.0,
+ "1984": 883962368.0,
+ "1985": 883962368.0,
+ "1986": 883962368.0,
+ "1987": 883962368.0,
+ "1988": 883962368.0,
+ "1989": 883962368.0,
+ "1990": 883962368.0,
+ "1991": 883962368.0,
+ "1992": 883962368.0,
+ "1993": 883962368.0,
+ "1994": 883962368.0,
+ "1995": 883962368.0,
+ "1996": 883962368.0,
+ "1997": 883962368.0,
+ "1998": 883962368.0,
+ "1999": 883962368.0,
+ "2000": 883962368.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": "nan",
+ "2": 8.39699,
+ "3": 1.1382,
+ "4": 1.12255,
+ "5": 1.12208,
+ "6": 1.10922,
+ "7": 1.11008,
+ "8": 1.10646,
+ "9": 1.108,
+ "10": 1.10922,
+ "11": 1.10784,
+ "12": 1.11534,
+ "13": 1.10988,
+ "14": 1.10664,
+ "15": 1.10866,
+ "16": 1.11154,
+ "17": 1.10943,
+ "18": 1.10759,
+ "19": 1.10753,
+ "20": 1.10621,
+ "21": 1.10365,
+ "22": 1.10673,
+ "23": 1.10686,
+ "24": 1.10448,
+ "25": 1.10701,
+ "26": 1.10299,
+ "27": 1.10388,
+ "28": 1.10446,
+ "29": 1.1061,
+ "30": 1.10464,
+ "31": 1.11188,
+ "32": 1.10511,
+ "33": 1.10661,
+ "34": 1.10479,
+ "35": 1.10281,
+ "36": 1.10425,
+ "37": 1.10588,
+ "38": 1.10735,
+ "39": 1.10888,
+ "40": 1.10617,
+ "41": 1.1059,
+ "42": 1.10487,
+ "43": 1.1055,
+ "44": 1.14544,
+ "45": 1.10625,
+ "46": 1.10531,
+ "47": 1.10963,
+ "48": 1.10663,
+ "49": 1.10526,
+ "50": 1.10528,
+ "51": 1.10502,
+ "52": 1.10632,
+ "53": 1.10683,
+ "54": 1.10888,
+ "55": 1.10778,
+ "56": 1.10773,
+ "57": 1.10687,
+ "58": 1.10499,
+ "59": 1.107,
+ "60": 1.11042,
+ "61": 1.10512,
+ "62": 1.10794,
+ "63": 1.1101,
+ "64": 1.11145,
+ "65": 1.11141,
+ "66": 1.11465,
+ "67": 1.23492,
+ "68": 1.13316,
+ "69": 1.12674,
+ "70": 1.12367,
+ "71": 1.1217,
+ "72": 1.13038,
+ "73": 1.1217,
+ "74": 1.12271,
+ "75": 1.12235,
+ "76": 1.12603,
+ "77": 1.12251,
+ "78": 1.12022,
+ "79": 1.1198,
+ "80": 1.12289,
+ "81": 1.12278,
+ "82": 1.12128,
+ "83": 1.1173,
+ "84": 1.11708,
+ "85": 1.11708,
+ "86": 1.11778,
+ "87": 1.11713,
+ "88": 1.11672,
+ "89": 1.11723,
+ "90": 1.11582,
+ "91": 1.11232,
+ "92": 1.11325,
+ "93": 1.1171,
+ "94": 1.12105,
+ "95": 1.12487,
+ "96": 1.12385,
+ "97": 1.12216,
+ "98": 1.12562,
+ "99": 1.12228,
+ "100": 1.11951,
+ "101": 1.11966,
+ "102": 1.11997,
+ "103": 1.12536,
+ "104": 1.1206,
+ "105": 1.12171,
+ "106": 1.12178,
+ "107": 1.1259,
+ "108": 1.11702,
+ "109": 1.1167,
+ "110": 1.12657,
+ "111": 1.13041,
+ "112": 1.12935,
+ "113": 1.12785,
+ "114": 1.12903,
+ "115": 1.12854,
+ "116": 1.12995,
+ "117": 1.12832,
+ "118": 1.12725,
+ "119": 1.12637,
+ "120": 1.13148,
+ "121": 1.12911,
+ "122": 1.13395,
+ "123": 1.39768,
+ "124": 1.1618,
+ "125": 1.12534,
+ "126": 1.12286,
+ "127": 1.11922,
+ "128": 1.12234,
+ "129": 1.12035,
+ "130": 1.12235,
+ "131": 1.12329,
+ "132": 1.12229,
+ "133": 1.12287,
+ "134": 1.1185,
+ "135": 1.11326,
+ "136": 1.11493,
+ "137": 1.11638,
+ "138": 1.11299,
+ "139": 1.113,
+ "140": 1.11551,
+ "141": 1.11602,
+ "142": 1.11339,
+ "143": 1.11503,
+ "144": 1.11612,
+ "145": 1.11232,
+ "146": 1.115,
+ "147": 1.1247,
+ "148": 1.12674,
+ "149": 1.12632,
+ "150": 1.12484,
+ "151": 1.12469,
+ "152": 1.12588,
+ "153": 1.13535,
+ "154": 1.11657,
+ "155": 1.11756,
+ "156": 1.12125,
+ "157": 1.12418,
+ "158": 1.12183,
+ "159": 1.12239,
+ "160": 1.12327,
+ "161": 1.12838,
+ "162": 1.12811,
+ "163": 1.13376,
+ "164": 1.13214,
+ "165": 1.13217,
+ "166": 1.12749,
+ "167": 1.1229,
+ "168": 1.12573,
+ "169": 1.13953,
+ "170": 1.13349,
+ "171": 1.12594,
+ "172": 1.12603,
+ "173": 1.11778,
+ "174": 1.11723,
+ "175": 1.11453,
+ "176": 1.12001,
+ "177": 1.11842,
+ "178": 1.11982,
+ "179": 1.13595,
+ "180": 1.15391,
+ "181": 1.13454,
+ "182": 1.11992,
+ "183": 1.12282,
+ "184": 1.12292,
+ "185": 1.12485,
+ "186": 1.12582,
+ "187": 1.12479,
+ "188": 1.12387,
+ "189": 1.12512,
+ "190": 1.12093,
+ "191": 1.12206,
+ "192": 1.11897,
+ "193": 1.12018,
+ "194": 1.11559,
+ "195": 1.11325,
+ "196": 1.11455,
+ "197": 1.11777,
+ "198": 1.11397,
+ "199": 1.11289,
+ "200": 1.11788,
+ "201": 1.11897,
+ "202": 1.12116,
+ "203": 1.11851,
+ "204": 1.11722,
+ "205": 1.13831,
+ "206": 1.1202,
+ "207": 1.17473,
+ "208": 1.11937,
+ "209": 1.11748,
+ "210": 1.11545,
+ "211": 1.12006,
+ "212": 1.11761,
+ "213": 1.11997,
+ "214": 1.1198,
+ "215": 1.11782,
+ "216": 1.11747,
+ "217": 1.12108,
+ "218": 1.11463,
+ "219": 1.11746,
+ "220": 1.12089,
+ "221": 1.11938,
+ "222": 1.12046,
+ "223": 1.11955,
+ "224": 1.12167,
+ "225": 1.11841,
+ "226": 1.11525,
+ "227": 1.11654,
+ "228": 1.11412,
+ "229": 1.1127,
+ "230": 1.12191,
+ "231": 1.11687,
+ "232": 1.11768,
+ "233": 1.12674,
+ "234": 1.13461,
+ "235": 1.15639,
+ "236": 1.12004,
+ "237": 1.11648,
+ "238": 1.11984,
+ "239": 1.11827,
+ "240": 1.11772,
+ "241": 1.11872,
+ "242": 1.122,
+ "243": 1.13201,
+ "244": 1.12901,
+ "245": 1.12332,
+ "246": 1.12076,
+ "247": 1.12526,
+ "248": 1.11214,
+ "249": 1.11231,
+ "250": 1.1128,
+ "251": 1.1165,
+ "252": 1.11669,
+ "253": 1.11409,
+ "254": 1.11568,
+ "255": 1.11867,
+ "256": 1.11146,
+ "257": 1.11221,
+ "258": 1.10957,
+ "259": 1.11113,
+ "260": 1.11042,
+ "261": 1.1105,
+ "262": 1.1096,
+ "263": 1.11327,
+ "264": 1.11333,
+ "265": 1.11245,
+ "266": 1.11123,
+ "267": 1.11165,
+ "268": 1.11253,
+ "269": 1.11016,
+ "270": 1.11647,
+ "271": 1.11282,
+ "272": 1.11094,
+ "273": 1.11039,
+ "274": 1.11235,
+ "275": 1.11002,
+ "276": 1.11102,
+ "277": 1.11166,
+ "278": 1.11307,
+ "279": 1.11204,
+ "280": 1.11377,
+ "281": 1.1134,
+ "282": 1.11128,
+ "283": 1.11053,
+ "284": 1.10936,
+ "285": 1.1162,
+ "286": 1.11877,
+ "287": 1.11396,
+ "288": 1.12933,
+ "289": 1.14265,
+ "290": 1.12905,
+ "291": 1.13027,
+ "292": 1.12875,
+ "293": 1.1324,
+ "294": 1.1538,
+ "295": 1.19504,
+ "296": 1.12892,
+ "297": 1.13165,
+ "298": 1.12916,
+ "299": 1.13031,
+ "300": 1.13023,
+ "301": 1.13388,
+ "302": 1.13182,
+ "303": 1.1294,
+ "304": 1.13172,
+ "305": 1.12845,
+ "306": 1.13007,
+ "307": 1.12933,
+ "308": 1.12938,
+ "309": 1.13069,
+ "310": 1.12987,
+ "311": 1.12901,
+ "312": 1.12835,
+ "313": 1.12928,
+ "314": 1.13089,
+ "315": 1.12882,
+ "316": 1.12617,
+ "317": 1.11752,
+ "318": 1.11664,
+ "319": 1.11428,
+ "320": 1.11531,
+ "321": 1.11284,
+ "322": 1.11637,
+ "323": 1.11259,
+ "324": 1.12179,
+ "325": 1.12065,
+ "326": 1.11864,
+ "327": 1.1159,
+ "328": 1.54926,
+ "329": 1.11204,
+ "330": 1.11218,
+ "331": 1.11948,
+ "332": 1.11948,
+ "333": 1.13307,
+ "334": 1.12467,
+ "335": 1.12137,
+ "336": 1.12246,
+ "337": 1.11139,
+ "338": 1.10779,
+ "339": 1.10833,
+ "340": 1.10947,
+ "341": 1.11134,
+ "342": 1.11053,
+ "343": 1.11143,
+ "344": 1.10814,
+ "345": 1.10803,
+ "346": 1.1091,
+ "347": 1.10853,
+ "348": 1.11037,
+ "349": 1.10927,
+ "350": 1.11013,
+ "351": 1.11397,
+ "352": 1.109,
+ "353": 1.10803,
+ "354": 1.11168,
+ "355": 1.11303,
+ "356": 1.10947,
+ "357": 1.11683,
+ "358": 1.12412,
+ "359": 1.12278,
+ "360": 1.12222,
+ "361": 1.22383,
+ "362": 1.17055,
+ "363": 1.11455,
+ "364": 1.11253,
+ "365": 1.11603,
+ "366": 1.11275,
+ "367": 1.11328,
+ "368": 1.11569,
+ "369": 1.11236,
+ "370": 1.11379,
+ "371": 1.11716,
+ "372": 1.11809,
+ "373": 1.11535,
+ "374": 1.11454,
+ "375": 1.11187,
+ "376": 1.11193,
+ "377": 1.1154,
+ "378": 1.12015,
+ "379": 1.1136,
+ "380": 1.11453,
+ "381": 1.11731,
+ "382": 1.12438,
+ "383": 1.12364,
+ "384": 1.12448,
+ "385": 1.12394,
+ "386": 1.12373,
+ "387": 1.11853,
+ "388": 1.1242,
+ "389": 1.12377,
+ "390": 1.12305,
+ "391": 1.12457,
+ "392": 1.80762,
+ "393": 1.12938,
+ "394": 1.11756,
+ "395": 1.12415,
+ "396": 1.12355,
+ "397": 1.12529,
+ "398": 1.12367,
+ "399": 1.12763,
+ "400": 1.16356,
+ "401": 1.19442,
+ "402": 1.12341,
+ "403": 1.12384,
+ "404": 1.12782,
+ "405": 1.12491,
+ "406": 1.73426,
+ "407": 1.13164,
+ "408": 1.13455,
+ "409": 1.13031,
+ "410": 1.13214,
+ "411": 1.13269,
+ "412": 1.13609,
+ "413": 1.13204,
+ "414": 1.13331,
+ "415": 1.13335,
+ "416": 1.12864,
+ "417": 1.1278,
+ "418": 1.12322,
+ "419": 1.12158,
+ "420": 1.12698,
+ "421": 1.12223,
+ "422": 1.12294,
+ "423": 1.1224,
+ "424": 1.12383,
+ "425": 1.12239,
+ "426": 1.12346,
+ "427": 1.12575,
+ "428": 1.12305,
+ "429": 1.1238,
+ "430": 1.12306,
+ "431": 1.1266,
+ "432": 1.12362,
+ "433": 1.13271,
+ "434": 1.13049,
+ "435": 1.12483,
+ "436": 1.12537,
+ "437": 1.12487,
+ "438": 1.12339,
+ "439": 1.12387,
+ "440": 1.12244,
+ "441": 1.12306,
+ "442": 1.12818,
+ "443": 1.12374,
+ "444": 1.1232,
+ "445": 1.12334,
+ "446": 1.12326,
+ "447": 1.1227,
+ "448": 1.12369,
+ "449": 1.12463,
+ "450": 1.1241,
+ "451": 1.12366,
+ "452": 1.12411,
+ "453": 1.12282,
+ "454": 1.1227,
+ "455": 1.12431,
+ "456": 1.12443,
+ "457": 1.12257,
+ "458": 1.12569,
+ "459": 1.12294,
+ "460": 1.12501,
+ "461": 1.12355,
+ "462": 1.12337,
+ "463": 1.12639,
+ "464": 1.1228,
+ "465": 1.12272,
+ "466": 1.12452,
+ "467": 1.12384,
+ "468": 1.12265,
+ "469": 1.12286,
+ "470": 1.1222,
+ "471": 1.12267,
+ "472": 1.12734,
+ "473": 1.12215,
+ "474": 1.1225,
+ "475": 1.1231,
+ "476": 1.12341,
+ "477": 1.21577,
+ "478": 1.12503,
+ "479": 1.12429,
+ "480": 1.12318,
+ "481": 1.12309,
+ "482": 1.12248,
+ "483": 1.12483,
+ "484": 1.12163,
+ "485": 1.12557,
+ "486": 1.1224,
+ "487": 1.12436,
+ "488": 1.12415,
+ "489": 1.12284,
+ "490": 1.12316,
+ "491": 1.124,
+ "492": 1.12213,
+ "493": 1.12107,
+ "494": 1.12112,
+ "495": 1.12191,
+ "496": 1.12272,
+ "497": 1.12153,
+ "498": 1.12444,
+ "499": 1.12301,
+ "500": 1.12204,
+ "501": 1.12128,
+ "502": 1.12292,
+ "503": 1.12195,
+ "504": 1.12259,
+ "505": 1.12721,
+ "506": 1.12255,
+ "507": 1.12362,
+ "508": 1.12295,
+ "509": 1.12243,
+ "510": 1.12677,
+ "511": 1.12437,
+ "512": 1.12641,
+ "513": 1.12267,
+ "514": 1.12517,
+ "515": 1.12279,
+ "516": 1.12541,
+ "517": 1.12259,
+ "518": 1.12294,
+ "519": 1.12137,
+ "520": 1.12171,
+ "521": 1.12109,
+ "522": 1.12266,
+ "523": 1.1233,
+ "524": 1.12174,
+ "525": 1.12312,
+ "526": 1.12412,
+ "527": 1.12392,
+ "528": 1.12243,
+ "529": 1.12359,
+ "530": 1.12209,
+ "531": 1.12201,
+ "532": 1.12218,
+ "533": 1.12191,
+ "534": 1.12255,
+ "535": 1.12431,
+ "536": 1.12334,
+ "537": 1.12249,
+ "538": 1.17486,
+ "539": 1.36492,
+ "540": 1.13793,
+ "541": 1.14426,
+ "542": 1.14364,
+ "543": 1.14967,
+ "544": 1.14002,
+ "545": 1.13525,
+ "546": 1.13226,
+ "547": 1.14027,
+ "548": 1.49954,
+ "549": 1.13624,
+ "550": 1.14457,
+ "551": 1.15053,
+ "552": 1.14794,
+ "553": 1.14603,
+ "554": 1.125,
+ "555": 1.12537,
+ "556": 1.12399,
+ "557": 1.12324,
+ "558": 1.1235,
+ "559": 1.12422,
+ "560": 1.12479,
+ "561": 1.12326,
+ "562": 1.1262,
+ "563": 1.12512,
+ "564": 1.12555,
+ "565": 1.12685,
+ "566": 1.12608,
+ "567": 1.12542,
+ "568": 1.12828,
+ "569": 1.1302,
+ "570": 1.1225,
+ "571": 1.12226,
+ "572": 1.14001,
+ "573": 1.14051,
+ "574": 1.13924,
+ "575": 1.13952,
+ "576": 1.13836,
+ "577": 1.1396,
+ "578": 1.13975,
+ "579": 1.1401,
+ "580": 1.13855,
+ "581": 1.14054,
+ "582": 1.13847,
+ "583": 1.13775,
+ "584": 1.12667,
+ "585": 1.13773,
+ "586": 1.13704,
+ "587": 1.12306,
+ "588": 1.12344,
+ "589": 1.12677,
+ "590": 1.12413,
+ "591": 1.12394,
+ "592": 1.12512,
+ "593": 1.12671,
+ "594": 1.12428,
+ "595": 1.16595,
+ "596": 1.12354,
+ "597": 1.12809,
+ "598": 1.12808,
+ "599": 1.12418,
+ "600": 1.12401,
+ "601": 1.12416,
+ "602": 1.12474,
+ "603": 1.12499,
+ "604": 1.13434,
+ "605": 1.12306,
+ "606": 1.12183,
+ "607": 1.12164,
+ "608": 1.12379,
+ "609": 1.12263,
+ "610": 1.12013,
+ "611": 1.15271,
+ "612": 1.15382,
+ "613": 1.15497,
+ "614": 1.16124,
+ "615": 1.15,
+ "616": 1.12366,
+ "617": 1.12389,
+ "618": 1.12555,
+ "619": 1.12725,
+ "620": 1.12682,
+ "621": 1.12477,
+ "622": 1.12663,
+ "623": 1.12567,
+ "624": 1.12886,
+ "625": 1.13147,
+ "626": 1.13189,
+ "627": 1.12759,
+ "628": 1.12342,
+ "629": 1.12468,
+ "630": 1.12833,
+ "631": 1.13233,
+ "632": 1.12988,
+ "633": 1.12749,
+ "634": 1.12446,
+ "635": 1.12586,
+ "636": 1.12823,
+ "637": 1.13061,
+ "638": 1.1308,
+ "639": 1.13859,
+ "640": 1.13417,
+ "641": 1.12972,
+ "642": 1.12805,
+ "643": 1.12438,
+ "644": 1.12818,
+ "645": 1.13257,
+ "646": 1.13371,
+ "647": 1.12622,
+ "648": 1.12534,
+ "649": 1.12994,
+ "650": 1.13081,
+ "651": 1.13239,
+ "652": 1.12442,
+ "653": 1.1252,
+ "654": 1.13009,
+ "655": 1.13207,
+ "656": 1.13001,
+ "657": 1.12489,
+ "658": 1.12608,
+ "659": 1.13153,
+ "660": 1.13108,
+ "661": 1.1306,
+ "662": 1.12945,
+ "663": 1.12752,
+ "664": 1.13169,
+ "665": 1.13232,
+ "666": 1.12713,
+ "667": 1.12601,
+ "668": 1.13111,
+ "669": 1.12364,
+ "670": 1.12339,
+ "671": 1.12478,
+ "672": 1.1241,
+ "673": 1.129,
+ "674": 1.12592,
+ "675": 1.12565,
+ "676": 1.12436,
+ "677": 1.12704,
+ "678": 1.12431,
+ "679": 1.12378,
+ "680": 1.12478,
+ "681": 1.12441,
+ "682": 1.12461,
+ "683": 1.12341,
+ "684": 1.12374,
+ "685": 1.1251,
+ "686": 1.12927,
+ "687": 1.1258,
+ "688": 1.1248,
+ "689": 1.12703,
+ "690": 1.1279,
+ "691": 1.12485,
+ "692": 1.12461,
+ "693": 1.12636,
+ "694": 1.12595,
+ "695": 1.13266,
+ "696": 1.1238,
+ "697": 1.12509,
+ "698": 1.12521,
+ "699": 1.12426,
+ "700": 1.12815,
+ "701": 1.12643,
+ "702": 1.12662,
+ "703": 1.12569,
+ "704": 1.12873,
+ "705": 1.12512,
+ "706": 1.1268,
+ "707": 1.12491,
+ "708": 1.12495,
+ "709": 1.12411,
+ "710": 1.12782,
+ "711": 1.12457,
+ "712": 1.12566,
+ "713": 1.47565,
+ "714": 1.24316,
+ "715": 1.21594,
+ "716": 1.12424,
+ "717": 1.12946,
+ "718": 1.12851,
+ "719": 1.12686,
+ "720": 1.12708,
+ "721": 1.12689,
+ "722": 1.12487,
+ "723": 1.12344,
+ "724": 1.12491,
+ "725": 1.12369,
+ "726": 1.12414,
+ "727": 1.13132,
+ "728": 1.12568,
+ "729": 1.12394,
+ "730": 1.12553,
+ "731": 1.12486,
+ "732": 1.12671,
+ "733": 1.12737,
+ "734": 1.12384,
+ "735": 1.12388,
+ "736": 1.12464,
+ "737": 1.12579,
+ "738": 1.12802,
+ "739": 1.1261,
+ "740": 1.12553,
+ "741": 1.12389,
+ "742": 1.1243,
+ "743": 1.12456,
+ "744": 1.12534,
+ "745": 1.1259,
+ "746": 1.12557,
+ "747": 1.12552,
+ "748": 1.12698,
+ "749": 1.12622,
+ "750": 1.12564,
+ "751": 1.12585,
+ "752": 1.1244,
+ "753": 1.13026,
+ "754": 1.13439,
+ "755": 1.12646,
+ "756": 1.12686,
+ "757": 1.12756,
+ "758": 1.13359,
+ "759": 1.13605,
+ "760": 1.13608,
+ "761": 1.13669,
+ "762": 1.13538,
+ "763": 1.13599,
+ "764": 1.13457,
+ "765": 1.13583,
+ "766": 1.13216,
+ "767": 1.13821,
+ "768": 1.1353,
+ "769": 1.14154,
+ "770": 1.14012,
+ "771": 1.137,
+ "772": 1.13571,
+ "773": 1.13506,
+ "774": 1.13358,
+ "775": 1.14088,
+ "776": 1.13622,
+ "777": 1.13135,
+ "778": 1.12949,
+ "779": 1.12746,
+ "780": 1.13168,
+ "781": 1.12931,
+ "782": 1.12547,
+ "783": 1.1248,
+ "784": 1.12598,
+ "785": 1.12565,
+ "786": 1.12591,
+ "787": 1.12699,
+ "788": 1.12607,
+ "789": 1.12717,
+ "790": 1.1269,
+ "791": 1.12667,
+ "792": 1.1254,
+ "793": 1.12603,
+ "794": 1.12664,
+ "795": 1.12566,
+ "796": 1.12588,
+ "797": 1.1242,
+ "798": 1.12609,
+ "799": 1.12896,
+ "800": 1.12575,
+ "801": 1.12419,
+ "802": 1.12996,
+ "803": 1.12695,
+ "804": 1.12393,
+ "805": 1.12583,
+ "806": 1.1247,
+ "807": 1.12368,
+ "808": 1.12354,
+ "809": 1.12405,
+ "810": 1.12504,
+ "811": 1.11875,
+ "812": 1.11844,
+ "813": 1.11954,
+ "814": 1.11943,
+ "815": 1.12406,
+ "816": 1.13151,
+ "817": 1.12521,
+ "818": 1.12544,
+ "819": 1.1253,
+ "820": 1.12602,
+ "821": 1.12584,
+ "822": 1.12577,
+ "823": 1.12767,
+ "824": 1.12483,
+ "825": 1.1244,
+ "826": 1.12551,
+ "827": 1.12619,
+ "828": 1.12423,
+ "829": 1.12586,
+ "830": 1.13035,
+ "831": 1.12645,
+ "832": 1.12849,
+ "833": 1.12728,
+ "834": 1.12659,
+ "835": 1.13042,
+ "836": 1.12683,
+ "837": 1.12542,
+ "838": 1.12602,
+ "839": 1.12611,
+ "840": 1.12631,
+ "841": 1.12743,
+ "842": 1.12229,
+ "843": 1.1242,
+ "844": 1.12342,
+ "845": 1.12392,
+ "846": 1.13277,
+ "847": 1.12466,
+ "848": 1.12232,
+ "849": 1.12362,
+ "850": 1.12483,
+ "851": 1.12346,
+ "852": 1.12379,
+ "853": 1.12349,
+ "854": 1.12226,
+ "855": 1.12386,
+ "856": 1.12664,
+ "857": 1.12314,
+ "858": 1.12651,
+ "859": 1.12422,
+ "860": 1.1276,
+ "861": 1.1269,
+ "862": 1.12847,
+ "863": 1.11852,
+ "864": 1.11815,
+ "865": 1.11914,
+ "866": 1.11877,
+ "867": 1.12038,
+ "868": 1.12172,
+ "869": 1.11895,
+ "870": 1.11927,
+ "871": 1.11857,
+ "872": 1.11754,
+ "873": 1.11737,
+ "874": 1.11863,
+ "875": 1.11838,
+ "876": 1.11847,
+ "877": 1.12013,
+ "878": 1.11901,
+ "879": 1.11877,
+ "880": 1.11947,
+ "881": 1.11981,
+ "882": 1.11867,
+ "883": 1.11979,
+ "884": 1.11786,
+ "885": 1.12963,
+ "886": 1.11688,
+ "887": 1.11724,
+ "888": 1.12017,
+ "889": 1.12211,
+ "890": 1.12175,
+ "891": 1.2287,
+ "892": 1.12211,
+ "893": 1.12174,
+ "894": 1.12144,
+ "895": 1.11994,
+ "896": 1.12826,
+ "897": 1.12648,
+ "898": 1.12487,
+ "899": 1.12503,
+ "900": 1.12501,
+ "901": 1.12531,
+ "902": 1.12619,
+ "903": 1.12487,
+ "904": 1.12499,
+ "905": 1.1279,
+ "906": 1.12549,
+ "907": 1.12757,
+ "908": 1.12677,
+ "909": 1.12825,
+ "910": 1.12776,
+ "911": 1.12719,
+ "912": 1.12728,
+ "913": 1.12763,
+ "914": 1.12621,
+ "915": 1.12424,
+ "916": 1.12589,
+ "917": 1.1245,
+ "918": 1.12336,
+ "919": 1.12351,
+ "920": 1.12453,
+ "921": 1.12819,
+ "922": 1.12456,
+ "923": 1.13948,
+ "924": 1.13114,
+ "925": 1.12372,
+ "926": 1.12377,
+ "927": 1.12288,
+ "928": 1.12303,
+ "929": 1.12468,
+ "930": 1.1232,
+ "931": 1.12887,
+ "932": 1.12946,
+ "933": 1.13101,
+ "934": 1.13017,
+ "935": 1.12973,
+ "936": 1.13256,
+ "937": 1.13009,
+ "938": 1.13026,
+ "939": 1.13304,
+ "940": 1.13207,
+ "941": 1.13152,
+ "942": 1.13101,
+ "943": 1.13415,
+ "944": 1.1323,
+ "945": 1.13212,
+ "946": 1.13224,
+ "947": 1.13138,
+ "948": 1.13044,
+ "949": 1.13109,
+ "950": 1.12939,
+ "951": 1.13072,
+ "952": 1.13049,
+ "953": 1.13159,
+ "954": 1.14164,
+ "955": 1.13128,
+ "956": 1.13142,
+ "957": 1.13138,
+ "958": 1.13144,
+ "959": 1.13115,
+ "960": 1.13216,
+ "961": 1.12556,
+ "962": 1.12433,
+ "963": 1.12515,
+ "964": 1.12481,
+ "965": 1.1248,
+ "966": 1.12506,
+ "967": 1.12791,
+ "968": 1.12561,
+ "969": 1.12458,
+ "970": 1.12579,
+ "971": 1.11693,
+ "972": 1.11839,
+ "973": 1.1178,
+ "974": 1.11785,
+ "975": 1.11753,
+ "976": 1.11755,
+ "977": 1.11881,
+ "978": 1.11844,
+ "979": 1.11811,
+ "980": 1.11836,
+ "981": 1.11859,
+ "982": 1.11723,
+ "983": 1.11847,
+ "984": 1.11818,
+ "985": 1.11825,
+ "986": 1.11887,
+ "987": 1.11808,
+ "988": 1.1178,
+ "989": 1.11837,
+ "990": 1.11685,
+ "991": 1.11833,
+ "992": 1.11736,
+ "993": 1.11811,
+ "994": 1.12052,
+ "995": 1.11911,
+ "996": 1.11762,
+ "997": 1.12093,
+ "998": 1.12158,
+ "999": 1.11825,
+ "1000": 1.12158,
+ "1001": 1.11543,
+ "1002": 1.11552,
+ "1003": 1.11488,
+ "1004": 1.11759,
+ "1005": 1.11913,
+ "1006": 1.22276,
+ "1007": 1.11739,
+ "1008": 1.1179,
+ "1009": 1.11484,
+ "1010": 1.11603,
+ "1011": 1.11657,
+ "1012": 1.11387,
+ "1013": 1.11413,
+ "1014": 1.49902,
+ "1015": 1.11796,
+ "1016": 1.11747,
+ "1017": 1.11913,
+ "1018": 1.11813,
+ "1019": 1.11822,
+ "1020": 1.11511,
+ "1021": 1.11626,
+ "1022": 1.11354,
+ "1023": 1.16316,
+ "1024": 1.12945,
+ "1025": 1.1283,
+ "1026": 1.12738,
+ "1027": 1.12782,
+ "1028": 1.58598,
+ "1029": 1.12764,
+ "1030": 1.12798,
+ "1031": 1.12751,
+ "1032": 1.12696,
+ "1033": 1.12726,
+ "1034": 1.12865,
+ "1035": 1.12857,
+ "1036": 1.12885,
+ "1037": 1.12856,
+ "1038": 1.1341,
+ "1039": 1.12874,
+ "1040": 1.12764,
+ "1041": 1.12975,
+ "1042": 1.13131,
+ "1043": 1.12709,
+ "1044": 1.12687,
+ "1045": 1.13286,
+ "1046": 1.12746,
+ "1047": 1.1303,
+ "1048": 1.13783,
+ "1049": 1.12872,
+ "1050": 1.12878,
+ "1051": 1.12761,
+ "1052": 1.12883,
+ "1053": 1.12852,
+ "1054": 1.1283,
+ "1055": 1.12829,
+ "1056": 1.12806,
+ "1057": 1.12853,
+ "1058": 1.13021,
+ "1059": 1.12924,
+ "1060": 1.12836,
+ "1061": 1.12997,
+ "1062": 1.12822,
+ "1063": 1.24928,
+ "1064": 1.12745,
+ "1065": 1.12774,
+ "1066": 1.13161,
+ "1067": 1.13039,
+ "1068": 1.1363,
+ "1069": 1.16118,
+ "1070": 1.1283,
+ "1071": 1.12921,
+ "1072": 1.13048,
+ "1073": 1.12563,
+ "1074": 1.13068,
+ "1075": 1.13019,
+ "1076": 1.12843,
+ "1077": 1.4828,
+ "1078": 1.13072,
+ "1079": 1.12823,
+ "1080": 1.123,
+ "1081": 1.11697,
+ "1082": 1.11865,
+ "1083": 1.11702,
+ "1084": 1.1176,
+ "1085": 1.11671,
+ "1086": 1.11506,
+ "1087": 1.12073,
+ "1088": 1.11688,
+ "1089": 1.1178,
+ "1090": 1.117,
+ "1091": 1.11805,
+ "1092": 1.11929,
+ "1093": 1.11803,
+ "1094": 1.12274,
+ "1095": 1.12347,
+ "1096": 1.12216,
+ "1097": 1.12076,
+ "1098": 1.1219,
+ "1099": 1.1246,
+ "1100": 1.1227,
+ "1101": 1.12014,
+ "1102": 1.11811,
+ "1103": 1.11822,
+ "1104": 1.12364,
+ "1105": 1.12151,
+ "1106": 1.11953,
+ "1107": 1.11953,
+ "1108": 1.11744,
+ "1109": 1.11689,
+ "1110": 1.11802,
+ "1111": 1.11855,
+ "1112": 1.11724,
+ "1113": 1.11884,
+ "1114": 1.12075,
+ "1115": 1.11736,
+ "1116": 1.11952,
+ "1117": 1.11955,
+ "1118": 1.11795,
+ "1119": 1.11918,
+ "1120": 1.1203,
+ "1121": 1.12348,
+ "1122": 1.11942,
+ "1123": 1.12019,
+ "1124": 1.11993,
+ "1125": 1.11437,
+ "1126": 1.11874,
+ "1127": 1.11601,
+ "1128": 1.11439,
+ "1129": 1.12419,
+ "1130": 1.1148,
+ "1131": 1.11445,
+ "1132": 1.11428,
+ "1133": 1.11293,
+ "1134": 1.1152,
+ "1135": 1.11704,
+ "1136": 1.11531,
+ "1137": 1.11456,
+ "1138": 1.11502,
+ "1139": 1.11324,
+ "1140": 1.11352,
+ "1141": 1.11614,
+ "1142": 1.11379,
+ "1143": 1.12002,
+ "1144": 1.11596,
+ "1145": 1.11534,
+ "1146": 1.11485,
+ "1147": 1.1161,
+ "1148": 1.11521,
+ "1149": 1.11365,
+ "1150": 1.11653,
+ "1151": 1.11412,
+ "1152": 1.11533,
+ "1153": 1.11832,
+ "1154": 1.12466,
+ "1155": 1.12689,
+ "1156": 1.12567,
+ "1157": 1.12431,
+ "1158": 1.12403,
+ "1159": 1.12345,
+ "1160": 1.12537,
+ "1161": 1.12368,
+ "1162": 1.12377,
+ "1163": 1.12399,
+ "1164": 1.12493,
+ "1165": 1.12898,
+ "1166": 1.1255,
+ "1167": 1.1272,
+ "1168": 1.12488,
+ "1169": 1.12309,
+ "1170": 1.12341,
+ "1171": 1.12552,
+ "1172": 1.12427,
+ "1173": 1.12389,
+ "1174": 1.16449,
+ "1175": 1.12577,
+ "1176": 1.11526,
+ "1177": 1.11474,
+ "1178": 1.50474,
+ "1179": 1.11398,
+ "1180": 1.12072,
+ "1181": 1.11679,
+ "1182": 1.11866,
+ "1183": 1.12408,
+ "1184": 1.12467,
+ "1185": 1.12597,
+ "1186": 1.12351,
+ "1187": 1.12374,
+ "1188": 1.12695,
+ "1189": 1.12909,
+ "1190": 1.12483,
+ "1191": 1.12329,
+ "1192": 1.12523,
+ "1193": 1.12491,
+ "1194": 1.12381,
+ "1195": 1.12428,
+ "1196": 1.12567,
+ "1197": 1.12824,
+ "1198": 1.12337,
+ "1199": 1.12284,
+ "1200": 1.12409,
+ "1201": 1.12534,
+ "1202": 1.12438,
+ "1203": 1.12611,
+ "1204": 1.12327,
+ "1205": 1.12472,
+ "1206": 1.12385,
+ "1207": 1.12583,
+ "1208": 1.11846,
+ "1209": 1.11798,
+ "1210": 1.119,
+ "1211": 1.11698,
+ "1212": 1.12123,
+ "1213": 1.11863,
+ "1214": 1.11729,
+ "1215": 1.11485,
+ "1216": 1.11673,
+ "1217": 1.11758,
+ "1218": 1.12525,
+ "1219": 1.11506,
+ "1220": 1.11521,
+ "1221": 1.11929,
+ "1222": 1.11993,
+ "1223": 1.11972,
+ "1224": 1.11895,
+ "1225": 1.11959,
+ "1226": 1.12028,
+ "1227": 1.122,
+ "1228": 1.11982,
+ "1229": 1.12118,
+ "1230": 1.122,
+ "1231": 1.12421,
+ "1232": 1.1205,
+ "1233": 1.12199,
+ "1234": 1.12485,
+ "1235": 1.12184,
+ "1236": 1.1289,
+ "1237": 1.12113,
+ "1238": 1.12376,
+ "1239": 1.12093,
+ "1240": 1.12155,
+ "1241": 1.12483,
+ "1242": 1.12752,
+ "1243": 1.13159,
+ "1244": 1.12816,
+ "1245": 1.21834,
+ "1246": 1.1813,
+ "1247": 1.13151,
+ "1248": 1.11397,
+ "1249": 1.1308,
+ "1250": 1.12799,
+ "1251": 1.12805,
+ "1252": 1.12429,
+ "1253": 1.12301,
+ "1254": 1.13415,
+ "1255": 1.12962,
+ "1256": 1.11676,
+ "1257": 1.11735,
+ "1258": 1.11835,
+ "1259": 1.13807,
+ "1260": 1.12594,
+ "1261": 1.12214,
+ "1262": 1.11922,
+ "1263": 1.12158,
+ "1264": 1.11864,
+ "1265": 1.11908,
+ "1266": 1.12078,
+ "1267": 1.11968,
+ "1268": 1.11873,
+ "1269": 1.11707,
+ "1270": 1.1192,
+ "1271": 1.11886,
+ "1272": 1.11938,
+ "1273": 1.11939,
+ "1274": 1.11921,
+ "1275": 1.11834,
+ "1276": 1.11745,
+ "1277": 1.11998,
+ "1278": 1.11892,
+ "1279": 1.12398,
+ "1280": 1.1211,
+ "1281": 1.1196,
+ "1282": 1.12537,
+ "1283": 1.122,
+ "1284": 1.11929,
+ "1285": 1.12015,
+ "1286": 1.11925,
+ "1287": 1.11955,
+ "1288": 1.12164,
+ "1289": 1.12123,
+ "1290": 1.12109,
+ "1291": 1.11905,
+ "1292": 1.1208,
+ "1293": 1.11896,
+ "1294": 1.12099,
+ "1295": 1.11823,
+ "1296": 1.11983,
+ "1297": 1.11933,
+ "1298": 1.11836,
+ "1299": 1.11991,
+ "1300": 1.11877,
+ "1301": 1.11933,
+ "1302": 1.11774,
+ "1303": 1.11756,
+ "1304": 1.11887,
+ "1305": 1.11861,
+ "1306": 1.12245,
+ "1307": 1.11839,
+ "1308": 1.11826,
+ "1309": 1.11807,
+ "1310": 1.11805,
+ "1311": 1.11666,
+ "1312": 1.11473,
+ "1313": 1.11617,
+ "1314": 1.47981,
+ "1315": 1.12298,
+ "1316": 1.12784,
+ "1317": 1.12798,
+ "1318": 1.12981,
+ "1319": 1.13033,
+ "1320": 1.12838,
+ "1321": 1.12859,
+ "1322": 1.11062,
+ "1323": 1.11187,
+ "1324": 1.11163,
+ "1325": 1.11171,
+ "1326": 1.11141,
+ "1327": 1.11218,
+ "1328": 1.10987,
+ "1329": 1.116,
+ "1330": 1.11666,
+ "1331": 1.11554,
+ "1332": 1.11731,
+ "1333": 1.11688,
+ "1334": 1.11824,
+ "1335": 1.11259,
+ "1336": 1.11119,
+ "1337": 1.11127,
+ "1338": 1.11055,
+ "1339": 1.11049,
+ "1340": 1.1122,
+ "1341": 1.11151,
+ "1342": 1.124,
+ "1343": 1.11778,
+ "1344": 1.11637,
+ "1345": 1.10841,
+ "1346": 1.10829,
+ "1347": 1.10857,
+ "1348": 1.10999,
+ "1349": 1.10823,
+ "1350": 1.10805,
+ "1351": 1.10868,
+ "1352": 1.1088,
+ "1353": 1.10931,
+ "1354": 1.11232,
+ "1355": 1.10831,
+ "1356": 1.10787,
+ "1357": 1.10926,
+ "1358": 1.10972,
+ "1359": 1.10838,
+ "1360": 1.10953,
+ "1361": 1.13332,
+ "1362": 1.2017,
+ "1363": 1.11474,
+ "1364": 1.11368,
+ "1365": 1.11268,
+ "1366": 1.11221,
+ "1367": 1.11382,
+ "1368": 1.11681,
+ "1369": 1.11399,
+ "1370": 1.11689,
+ "1371": 1.11245,
+ "1372": 1.11223,
+ "1373": 1.11196,
+ "1374": 1.11156,
+ "1375": 1.11128,
+ "1376": 1.11011,
+ "1377": 1.11123,
+ "1378": 1.1116,
+ "1379": 1.11114,
+ "1380": 1.10994,
+ "1381": 1.11223,
+ "1382": 1.1126,
+ "1383": 1.11121,
+ "1384": 1.11153,
+ "1385": 1.11232,
+ "1386": 1.11193,
+ "1387": 1.11231,
+ "1388": 1.11183,
+ "1389": 1.11204,
+ "1390": 1.1051,
+ "1391": 1.10408,
+ "1392": 1.10446,
+ "1393": 1.10415,
+ "1394": 1.10411,
+ "1395": 1.10366,
+ "1396": 1.10337,
+ "1397": 1.10558,
+ "1398": 1.10346,
+ "1399": 1.10279,
+ "1400": 1.10317,
+ "1401": 1.10292,
+ "1402": 1.1032,
+ "1403": 1.10242,
+ "1404": 1.10348,
+ "1405": 1.10381,
+ "1406": 1.10445,
+ "1407": 1.10564,
+ "1408": 1.106,
+ "1409": 1.10488,
+ "1410": 1.10645,
+ "1411": 1.10533,
+ "1412": 1.10409,
+ "1413": 1.10392,
+ "1414": 1.10497,
+ "1415": 1.10555,
+ "1416": 1.10621,
+ "1417": 1.11076,
+ "1418": 1.10549,
+ "1419": 1.10479,
+ "1420": 1.10431,
+ "1421": 1.10446,
+ "1422": 1.10503,
+ "1423": 1.10624,
+ "1424": 1.10511,
+ "1425": 1.1075,
+ "1426": 1.10364,
+ "1427": 1.10458,
+ "1428": 1.10422,
+ "1429": 1.10343,
+ "1430": 1.10337,
+ "1431": 1.10359,
+ "1432": 1.10643,
+ "1433": 1.10445,
+ "1434": 1.1034,
+ "1435": 1.10443,
+ "1436": 1.10365,
+ "1437": 1.1038,
+ "1438": 1.105,
+ "1439": 1.10378,
+ "1440": 1.103,
+ "1441": 1.10428,
+ "1442": 1.1043,
+ "1443": 1.10274,
+ "1444": 1.10493,
+ "1445": 1.10272,
+ "1446": 1.10415,
+ "1447": 1.10533,
+ "1448": 1.10532,
+ "1449": 1.10603,
+ "1450": 1.10528,
+ "1451": 1.1108,
+ "1452": 1.10712,
+ "1453": 1.11176,
+ "1454": 1.10767,
+ "1455": 1.10629,
+ "1456": 1.10533,
+ "1457": 1.10421,
+ "1458": 1.1332,
+ "1459": 1.21658,
+ "1460": 1.10493,
+ "1461": 1.10654,
+ "1462": 1.11023,
+ "1463": 1.10258,
+ "1464": 1.10387,
+ "1465": 1.10399,
+ "1466": 1.10437,
+ "1467": 1.10334,
+ "1468": 1.10675,
+ "1469": 1.10948,
+ "1470": 1.10955,
+ "1471": 1.11346,
+ "1472": 1.10819,
+ "1473": 1.10829,
+ "1474": 1.10914,
+ "1475": 1.10918,
+ "1476": 1.10795,
+ "1477": 1.10898,
+ "1478": 1.11214,
+ "1479": 1.10774,
+ "1480": 1.11043,
+ "1481": 1.10307,
+ "1482": 1.10536,
+ "1483": 1.11552,
+ "1484": 1.11417,
+ "1485": 1.10932,
+ "1486": 1.11036,
+ "1487": 1.1092,
+ "1488": 1.10924,
+ "1489": 1.11411,
+ "1490": 1.10779,
+ "1491": 1.10974,
+ "1492": 1.10995,
+ "1493": 1.10873,
+ "1494": 1.10935,
+ "1495": 1.10896,
+ "1496": 1.11013,
+ "1497": 1.10912,
+ "1498": 1.11301,
+ "1499": 1.10457,
+ "1500": 1.10901,
+ "1501": 1.10911,
+ "1502": 1.10881,
+ "1503": 1.10951,
+ "1504": 1.10736,
+ "1505": 1.10876,
+ "1506": 1.10844,
+ "1507": 1.11636,
+ "1508": 1.10659,
+ "1509": 1.10492,
+ "1510": 1.10782,
+ "1511": 1.108,
+ "1512": 1.10793,
+ "1513": 1.10956,
+ "1514": 1.10935,
+ "1515": 1.10841,
+ "1516": 1.11175,
+ "1517": 1.10921,
+ "1518": 1.10866,
+ "1519": 1.10931,
+ "1520": 1.10796,
+ "1521": 1.10944,
+ "1522": 1.11048,
+ "1523": 1.11091,
+ "1524": 1.10764,
+ "1525": 1.11474,
+ "1526": 1.11003,
+ "1527": 1.10928,
+ "1528": 1.11034,
+ "1529": 1.10796,
+ "1530": 1.10869,
+ "1531": 1.10751,
+ "1532": 1.10865,
+ "1533": 1.11202,
+ "1534": 1.14671,
+ "1535": 1.16695,
+ "1536": 1.12718,
+ "1537": 1.12693,
+ "1538": 1.12728,
+ "1539": 1.11804,
+ "1540": 1.11849,
+ "1541": 1.1274,
+ "1542": 1.12816,
+ "1543": 1.22613,
+ "1544": 1.12158,
+ "1545": 1.10429,
+ "1546": 1.12379,
+ "1547": 1.12968,
+ "1548": 1.1096,
+ "1549": 1.10587,
+ "1550": 1.10455,
+ "1551": 1.10299,
+ "1552": 1.1089,
+ "1553": 1.10744,
+ "1554": 1.14403,
+ "1555": 1.1431,
+ "1556": 1.12783,
+ "1557": 1.1264,
+ "1558": 1.12901,
+ "1559": 1.10792,
+ "1560": 1.10656,
+ "1561": 1.16643,
+ "1562": 1.12022,
+ "1563": 1.12183,
+ "1564": 1.13269,
+ "1565": 1.13215,
+ "1566": 1.13015,
+ "1567": 1.13414,
+ "1568": 1.131,
+ "1569": 1.13041,
+ "1570": 1.12991,
+ "1571": 1.13159,
+ "1572": 1.13072,
+ "1573": 1.1316,
+ "1574": 1.12946,
+ "1575": 1.13264,
+ "1576": 1.13366,
+ "1577": 1.1299,
+ "1578": 1.13155,
+ "1579": 1.13117,
+ "1580": 1.13046,
+ "1581": 1.13053,
+ "1582": 1.12914,
+ "1583": 1.13077,
+ "1584": 1.13644,
+ "1585": 1.13215,
+ "1586": 1.13061,
+ "1587": 1.13206,
+ "1588": 1.13435,
+ "1589": 1.13632,
+ "1590": 1.13214,
+ "1591": 1.13265,
+ "1592": 1.13379,
+ "1593": 1.13377,
+ "1594": 1.13281,
+ "1595": 1.13276,
+ "1596": 1.11771,
+ "1597": 1.10837,
+ "1598": 1.13389,
+ "1599": 1.13382,
+ "1600": 1.13496,
+ "1601": 1.13462,
+ "1602": 1.13489,
+ "1603": 1.14129,
+ "1604": 1.1358,
+ "1605": 1.13595,
+ "1606": 1.13449,
+ "1607": 1.13569,
+ "1608": 1.13934,
+ "1609": 1.13664,
+ "1610": 1.13373,
+ "1611": 1.13085,
+ "1612": 1.11465,
+ "1613": 1.10995,
+ "1614": 1.11101,
+ "1615": 1.11086,
+ "1616": 1.13891,
+ "1617": 1.13336,
+ "1618": 1.13162,
+ "1619": 1.12995,
+ "1620": 1.13159,
+ "1621": 1.13338,
+ "1622": 1.13197,
+ "1623": 1.1326,
+ "1624": 1.13098,
+ "1625": 1.13095,
+ "1626": 1.12468,
+ "1627": 1.10967,
+ "1628": 1.11006,
+ "1629": 1.11608,
+ "1630": 1.13644,
+ "1631": 1.12998,
+ "1632": 1.11768,
+ "1633": 1.13222,
+ "1634": 1.13446,
+ "1635": 1.12991,
+ "1636": 1.11019,
+ "1637": 1.11259,
+ "1638": 1.11426,
+ "1639": 1.11482,
+ "1640": 1.11292,
+ "1641": 1.11276,
+ "1642": 1.11291,
+ "1643": 1.11516,
+ "1644": 1.11196,
+ "1645": 1.11102,
+ "1646": 1.11163,
+ "1647": 1.11426,
+ "1648": 1.11767,
+ "1649": 1.11193,
+ "1650": 1.11139,
+ "1651": 1.11149,
+ "1652": 1.11227,
+ "1653": 1.11439,
+ "1654": 1.11182,
+ "1655": 1.11176,
+ "1656": 1.11317,
+ "1657": 1.11992,
+ "1658": 1.11084,
+ "1659": 1.11458,
+ "1660": 1.11488,
+ "1661": 1.115,
+ "1662": 1.11831,
+ "1663": 1.11442,
+ "1664": 1.13569,
+ "1665": 1.24038,
+ "1666": 1.11076,
+ "1667": 1.11604,
+ "1668": 1.10517,
+ "1669": 1.10484,
+ "1670": 1.10544,
+ "1671": 1.10543,
+ "1672": 1.11566,
+ "1673": 1.16001,
+ "1674": 1.11776,
+ "1675": 1.1059,
+ "1676": 1.10532,
+ "1677": 1.10452,
+ "1678": 1.10572,
+ "1679": 1.1059,
+ "1680": 1.10488,
+ "1681": 1.10543,
+ "1682": 1.10537,
+ "1683": 1.1051,
+ "1684": 1.10657,
+ "1685": 1.10774,
+ "1686": 1.11319,
+ "1687": 1.11278,
+ "1688": 1.11174,
+ "1689": 1.11297,
+ "1690": 1.11211,
+ "1691": 1.11085,
+ "1692": 1.11347,
+ "1693": 1.11016,
+ "1694": 1.11193,
+ "1695": 1.11256,
+ "1696": 1.11308,
+ "1697": 1.11925,
+ "1698": 1.11216,
+ "1699": 1.11173,
+ "1700": 1.11487,
+ "1701": 1.11248,
+ "1702": 1.11503,
+ "1703": 1.11482,
+ "1704": 1.1142,
+ "1705": 1.116,
+ "1706": 1.11402,
+ "1707": 1.11163,
+ "1708": 1.11175,
+ "1709": 1.11295,
+ "1710": 1.11465,
+ "1711": 1.11208,
+ "1712": 1.11123,
+ "1713": 1.11318,
+ "1714": 1.11453,
+ "1715": 1.11488,
+ "1716": 1.11381,
+ "1717": 1.11546,
+ "1718": 1.11076,
+ "1719": 1.49171,
+ "1720": 1.11754,
+ "1721": 1.11674,
+ "1722": 1.12454,
+ "1723": 1.11158,
+ "1724": 1.11439,
+ "1725": 1.11233,
+ "1726": 1.12051,
+ "1727": 1.11616,
+ "1728": 1.11407,
+ "1729": 1.10857,
+ "1730": 1.11119,
+ "1731": 1.11007,
+ "1732": 1.1088,
+ "1733": 1.10844,
+ "1734": 1.10904,
+ "1735": 1.11409,
+ "1736": 1.10938,
+ "1737": 1.11216,
+ "1738": 1.10977,
+ "1739": 1.10871,
+ "1740": 1.1132,
+ "1741": 1.11495,
+ "1742": 1.11623,
+ "1743": 1.11235,
+ "1744": 1.11535,
+ "1745": 1.11463,
+ "1746": 1.11576,
+ "1747": 1.11573,
+ "1748": 1.11479,
+ "1749": 1.11607,
+ "1750": 1.11553,
+ "1751": 1.11594,
+ "1752": 1.11325,
+ "1753": 1.11214,
+ "1754": 1.11012,
+ "1755": 1.11115,
+ "1756": 1.11086,
+ "1757": 1.11561,
+ "1758": 1.11564,
+ "1759": 1.11633,
+ "1760": 1.11649,
+ "1761": 1.11526,
+ "1762": 1.11536,
+ "1763": 1.11383,
+ "1764": 1.11651,
+ "1765": 1.1142,
+ "1766": 1.11351,
+ "1767": 1.11301,
+ "1768": 1.10954,
+ "1769": 1.10883,
+ "1770": 1.11067,
+ "1771": 1.10929,
+ "1772": 1.11503,
+ "1773": 1.11828,
+ "1774": 1.11608,
+ "1775": 1.11866,
+ "1776": 1.11621,
+ "1777": 1.11669,
+ "1778": 1.11678,
+ "1779": 1.11327,
+ "1780": 1.11748,
+ "1781": 1.11803,
+ "1782": 1.11655,
+ "1783": 1.45489,
+ "1784": 1.11768,
+ "1785": 1.11656,
+ "1786": 1.11758,
+ "1787": 1.11564,
+ "1788": 1.11626,
+ "1789": 1.11554,
+ "1790": 1.12063,
+ "1791": 1.11504,
+ "1792": 1.11803,
+ "1793": 1.11447,
+ "1794": 1.1069,
+ "1795": 1.10484,
+ "1796": 1.10504,
+ "1797": 1.10686,
+ "1798": 1.10907,
+ "1799": 1.11374,
+ "1800": 1.11065,
+ "1801": 1.11139,
+ "1802": 1.10982,
+ "1803": 1.10818,
+ "1804": 1.10899,
+ "1805": 1.10885,
+ "1806": 1.10913,
+ "1807": 1.1054,
+ "1808": 1.10623,
+ "1809": 1.10458,
+ "1810": 1.10506,
+ "1811": 1.105,
+ "1812": 1.10588,
+ "1813": 1.10494,
+ "1814": 1.1057,
+ "1815": 1.10482,
+ "1816": 1.10377,
+ "1817": 1.10504,
+ "1818": 1.1049,
+ "1819": 1.10539,
+ "1820": 1.10897,
+ "1821": 1.10682,
+ "1822": 1.1033,
+ "1823": 1.10369,
+ "1824": 1.10486,
+ "1825": 1.10481,
+ "1826": 1.1048,
+ "1827": 1.10583,
+ "1828": 1.11156,
+ "1829": 1.10972,
+ "1830": 1.10899,
+ "1831": 1.10922,
+ "1832": 1.10836,
+ "1833": 1.10893,
+ "1834": 1.11047,
+ "1835": 1.11172,
+ "1836": 1.1094,
+ "1837": 1.10907,
+ "1838": 1.10801,
+ "1839": 1.10998,
+ "1840": 1.10888,
+ "1841": 1.10795,
+ "1842": 1.10927,
+ "1843": 1.1078,
+ "1844": 1.13592,
+ "1845": 1.39052,
+ "1846": 1.10239,
+ "1847": 1.1027,
+ "1848": 1.1064,
+ "1849": 1.10353,
+ "1850": 1.10309,
+ "1851": 1.10479,
+ "1852": 1.10593,
+ "1853": 1.10573,
+ "1854": 1.10593,
+ "1855": 1.10396,
+ "1856": 1.10268,
+ "1857": 1.10372,
+ "1858": 1.10352,
+ "1859": 1.10612,
+ "1860": 1.10233,
+ "1861": 1.11184,
+ "1862": 1.11167,
+ "1863": 1.11154,
+ "1864": 1.11202,
+ "1865": 1.11022,
+ "1866": 1.1102,
+ "1867": 1.11128,
+ "1868": 1.11037,
+ "1869": 1.11121,
+ "1870": 1.11389,
+ "1871": 1.11715,
+ "1872": 1.11769,
+ "1873": 1.11697,
+ "1874": 1.11587,
+ "1875": 1.11483,
+ "1876": 1.11143,
+ "1877": 1.11413,
+ "1878": 1.11871,
+ "1879": 1.11657,
+ "1880": 1.11894,
+ "1881": 1.12135,
+ "1882": 1.11631,
+ "1883": 1.11088,
+ "1884": 1.11281,
+ "1885": 1.11731,
+ "1886": 1.11559,
+ "1887": 1.11611,
+ "1888": 1.11653,
+ "1889": 1.11716,
+ "1890": 1.12013,
+ "1891": 1.11811,
+ "1892": 1.11741,
+ "1893": 1.11523,
+ "1894": 1.11524,
+ "1895": 1.1154,
+ "1896": 1.11514,
+ "1897": 1.1145,
+ "1898": 1.11514,
+ "1899": 1.11282,
+ "1900": 1.11464,
+ "1901": 1.1145,
+ "1902": 1.15861,
+ "1903": 1.21694,
+ "1904": 1.25454,
+ "1905": 1.10498,
+ "1906": 1.1047,
+ "1907": 1.14943,
+ "1908": 1.11047,
+ "1909": 1.11265,
+ "1910": 1.10906,
+ "1911": 1.11034,
+ "1912": 1.10976,
+ "1913": 1.10938,
+ "1914": 1.10879,
+ "1915": 1.11236,
+ "1916": 1.11142,
+ "1917": 1.11156,
+ "1918": 1.10967,
+ "1919": 1.1098,
+ "1920": 1.12664,
+ "1921": 1.10649,
+ "1922": 1.1063,
+ "1923": 1.10317,
+ "1924": 1.10305,
+ "1925": 1.1393,
+ "1926": 1.10532,
+ "1927": 1.10909,
+ "1928": 1.10556,
+ "1929": 1.10877,
+ "1930": 1.11316,
+ "1931": 1.11505,
+ "1932": 1.11216,
+ "1933": 1.10925,
+ "1934": 1.11048,
+ "1935": 1.11028,
+ "1936": 1.11684,
+ "1937": 1.114,
+ "1938": 1.10949,
+ "1939": 1.1146,
+ "1940": 1.11266,
+ "1941": 1.11345,
+ "1942": 1.11359,
+ "1943": 1.11273,
+ "1944": 1.10729,
+ "1945": 1.10822,
+ "1946": 1.10623,
+ "1947": 1.10905,
+ "1948": 1.10845,
+ "1949": 1.11023,
+ "1950": 1.11572,
+ "1951": 1.11557,
+ "1952": 1.11833,
+ "1953": 1.11128,
+ "1954": 1.10985,
+ "1955": 1.11088,
+ "1956": 1.11315,
+ "1957": 1.11554,
+ "1958": 1.11584,
+ "1959": 1.11529,
+ "1960": 1.11295,
+ "1961": 1.1115,
+ "1962": 1.11333,
+ "1963": 1.1106,
+ "1964": 1.10942,
+ "1965": 1.11327,
+ "1966": 1.11222,
+ "1967": 1.11145,
+ "1968": 1.11089,
+ "1969": 1.11477,
+ "1970": 1.11909,
+ "1971": 1.11318,
+ "1972": 1.11239,
+ "1973": 1.11121,
+ "1974": 1.11433,
+ "1975": 1.11055,
+ "1976": 1.10928,
+ "1977": 1.10941,
+ "1978": 1.11038,
+ "1979": 1.11511,
+ "1980": 1.11527,
+ "1981": 1.11219,
+ "1982": 1.11242,
+ "1983": 1.5062,
+ "1984": 1.11152,
+ "1985": 1.11114,
+ "1986": 1.11153,
+ "1987": 1.10941,
+ "1988": 1.1071,
+ "1989": 1.10744,
+ "1990": 1.10717,
+ "1991": 1.10712,
+ "1992": 1.1065,
+ "1993": 1.11006,
+ "1994": 1.10964,
+ "1995": 1.1116,
+ "1996": 1.11045,
+ "1997": 1.11066,
+ "1998": 1.10971,
+ "1999": 1.1116,
+ "2000": 1.1086
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/model_config.yaml
similarity index 80%
rename from tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/model_config.yaml
rename to tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/model_config.yaml
index 5668a7575e2..e8673fbae20 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -42,7 +42,7 @@ MODEL_ARGS:
--pipeline-model-parallel-size: 2
--sequence-parallel: true
--tp-comm-overlap: true
- --tp-comm-overlap-cfg: tests/functional_tests/test_cases/gpt/gpt3_345m_weekly_dgx_h100_1N8G_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/tp_comm_overlap_cfg.yaml
+ --tp-comm-overlap-cfg: tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/tp_comm_overlap_cfg.yaml
--deterministic-mode: true
--no-gradient-accumulation-fusion: true
--fp8-format: hybrid
@@ -56,6 +56,8 @@ MODEL_ARGS:
--bf16: true
--log-memory-to-tensorboard: true
--normalization: RMSNorm
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
METRICS:
- iteration-time
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/tp_comm_overlap_cfg.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/tp_comm_overlap_cfg.yaml
similarity index 100%
rename from tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_b200_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/tp_comm_overlap_cfg.yaml
rename to tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap/tp_comm_overlap_cfg.yaml
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/golden_values_dev_dgx_gb200.json
new file mode 100644
index 00000000000..695e6c48261
--- /dev/null
+++ b/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/golden_values_dev_dgx_gb200.json
@@ -0,0 +1,10037 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 10.84507,
+ "2": 10.85699,
+ "3": 10.84975,
+ "4": 10.86701,
+ "5": 10.85126,
+ "6": 10.84883,
+ "7": 10.85322,
+ "8": 10.83971,
+ "9": 10.84287,
+ "10": 10.8434,
+ "11": 10.83435,
+ "12": 10.82803,
+ "13": 10.81382,
+ "14": 10.83459,
+ "15": 10.77416,
+ "16": 10.78411,
+ "17": 10.7605,
+ "18": 10.78234,
+ "19": 10.72403,
+ "20": 10.6871,
+ "21": 10.6351,
+ "22": 10.63649,
+ "23": 10.64523,
+ "24": 10.54519,
+ "25": 10.55663,
+ "26": 10.62535,
+ "27": 10.45905,
+ "28": 10.47146,
+ "29": 10.36042,
+ "30": 10.2687,
+ "31": 10.42691,
+ "32": 10.35208,
+ "33": 10.22028,
+ "34": 10.17304,
+ "35": 10.23256,
+ "36": 10.14756,
+ "37": 10.1046,
+ "38": 10.03487,
+ "39": 10.04669,
+ "40": 10.06959,
+ "41": 9.95005,
+ "42": 9.95268,
+ "43": 9.86492,
+ "44": 9.98919,
+ "45": 10.0105,
+ "46": 9.84603,
+ "47": 10.00002,
+ "48": 9.81368,
+ "49": 9.94459,
+ "50": 9.94535,
+ "51": 9.59288,
+ "52": 9.80287,
+ "53": 9.62897,
+ "54": 9.88755,
+ "55": 9.73558,
+ "56": 9.85149,
+ "57": 9.85798,
+ "58": 9.87372,
+ "59": 9.54863,
+ "60": 9.64311,
+ "61": 9.87788,
+ "62": 9.76092,
+ "63": 9.68521,
+ "64": 9.83364,
+ "65": 9.59657,
+ "66": 9.62927,
+ "67": 9.74362,
+ "68": 9.61175,
+ "69": 9.29535,
+ "70": 9.42383,
+ "71": 9.78915,
+ "72": 9.71525,
+ "73": 9.6193,
+ "74": 9.45251,
+ "75": 9.24457,
+ "76": 9.51338,
+ "77": 9.57807,
+ "78": 9.56537,
+ "79": 9.31209,
+ "80": 9.35629,
+ "81": 9.46411,
+ "82": 9.55355,
+ "83": 9.53714,
+ "84": 9.35583,
+ "85": 9.40819,
+ "86": 9.65641,
+ "87": 9.23811,
+ "88": 9.48999,
+ "89": 9.22521,
+ "90": 9.41778,
+ "91": 9.38833,
+ "92": 9.37677,
+ "93": 9.36162,
+ "94": 9.51625,
+ "95": 9.42618,
+ "96": 9.33443,
+ "97": 9.2064,
+ "98": 9.49385,
+ "99": 9.29861,
+ "100": 9.35611,
+ "101": 9.25142,
+ "102": 9.24864,
+ "103": 9.07949,
+ "104": 9.16722,
+ "105": 9.38406,
+ "106": 9.15586,
+ "107": 9.17482,
+ "108": 9.31156,
+ "109": 9.28833,
+ "110": 9.37029,
+ "111": 9.17914,
+ "112": 9.243,
+ "113": 9.35193,
+ "114": 9.35491,
+ "115": 9.32935,
+ "116": 9.00739,
+ "117": 9.0668,
+ "118": 9.0653,
+ "119": 9.22679,
+ "120": 9.0908,
+ "121": 9.19901,
+ "122": 9.14192,
+ "123": 9.262,
+ "124": 9.46096,
+ "125": 9.21638,
+ "126": 9.0579,
+ "127": 9.02103,
+ "128": 9.21987,
+ "129": 8.98876,
+ "130": 9.14377,
+ "131": 9.16171,
+ "132": 9.0394,
+ "133": 8.86312,
+ "134": 9.18886,
+ "135": 8.88429,
+ "136": 9.16666,
+ "137": 9.16452,
+ "138": 9.23501,
+ "139": 9.09498,
+ "140": 8.86871,
+ "141": 9.30203,
+ "142": 9.20019,
+ "143": 9.11502,
+ "144": 9.24802,
+ "145": 9.1032,
+ "146": 8.98631,
+ "147": 8.98634,
+ "148": 9.13565,
+ "149": 9.06815,
+ "150": 9.01758,
+ "151": 8.9301,
+ "152": 8.8732,
+ "153": 9.07072,
+ "154": 9.18518,
+ "155": 9.14254,
+ "156": 9.0546,
+ "157": 9.15808,
+ "158": 9.04969,
+ "159": 9.03818,
+ "160": 8.89317,
+ "161": 9.053,
+ "162": 8.89461,
+ "163": 8.84831,
+ "164": 8.97328,
+ "165": 8.93062,
+ "166": 8.66341,
+ "167": 8.83283,
+ "168": 8.82019,
+ "169": 8.66179,
+ "170": 9.04828,
+ "171": 8.72637,
+ "172": 8.82551,
+ "173": 8.91233,
+ "174": 8.85271,
+ "175": 8.70978,
+ "176": 8.7666,
+ "177": 8.76571,
+ "178": 8.73001,
+ "179": 8.64132,
+ "180": 8.74262,
+ "181": 8.69547,
+ "182": 8.72555,
+ "183": 9.0847,
+ "184": 8.61538,
+ "185": 8.8842,
+ "186": 8.74825,
+ "187": 8.58157,
+ "188": 8.68306,
+ "189": 8.86408,
+ "190": 8.53755,
+ "191": 8.66683,
+ "192": 8.61391,
+ "193": 8.57763,
+ "194": 8.75057,
+ "195": 8.60083,
+ "196": 8.77433,
+ "197": 8.74393,
+ "198": 8.63069,
+ "199": 8.77111,
+ "200": 8.74044,
+ "201": 8.67434,
+ "202": 8.54541,
+ "203": 8.54269,
+ "204": 8.7113,
+ "205": 8.22396,
+ "206": 8.85877,
+ "207": 8.68323,
+ "208": 8.71036,
+ "209": 8.75829,
+ "210": 8.57608,
+ "211": 8.84278,
+ "212": 8.49778,
+ "213": 8.57099,
+ "214": 8.51319,
+ "215": 8.5662,
+ "216": 8.51117,
+ "217": 8.5364,
+ "218": 8.53623,
+ "219": 8.64692,
+ "220": 8.54713,
+ "221": 8.4081,
+ "222": 8.51315,
+ "223": 8.43735,
+ "224": 8.53439,
+ "225": 8.56581,
+ "226": 8.44196,
+ "227": 8.6784,
+ "228": 8.38791,
+ "229": 8.4572,
+ "230": 8.50224,
+ "231": 8.49495,
+ "232": 8.50288,
+ "233": 8.49656,
+ "234": 8.64023,
+ "235": 8.56014,
+ "236": 8.39951,
+ "237": 8.49142,
+ "238": 8.30904,
+ "239": 8.56323,
+ "240": 8.67356,
+ "241": 8.44291,
+ "242": 8.47248,
+ "243": 8.5215,
+ "244": 8.36757,
+ "245": 8.59092,
+ "246": 8.59835,
+ "247": 8.43445,
+ "248": 8.5197,
+ "249": 8.52872,
+ "250": 8.4266,
+ "251": 8.37936,
+ "252": 8.54267,
+ "253": 8.31471,
+ "254": 8.35245,
+ "255": 8.29273,
+ "256": 8.21136,
+ "257": 8.39122,
+ "258": 8.45595,
+ "259": 8.23162,
+ "260": 8.24397,
+ "261": 8.23945,
+ "262": 8.34894,
+ "263": 8.30521,
+ "264": 8.19228,
+ "265": 8.33244,
+ "266": 8.23141,
+ "267": 7.90032,
+ "268": 8.38008,
+ "269": 8.40323,
+ "270": 8.26498,
+ "271": 8.27921,
+ "272": 8.31752,
+ "273": 8.13347,
+ "274": 8.09398,
+ "275": 8.00807,
+ "276": 7.92907,
+ "277": 8.23646,
+ "278": 8.04763,
+ "279": 7.96007,
+ "280": 7.75776,
+ "281": 8.10699,
+ "282": 8.14841,
+ "283": 8.15457,
+ "284": 8.10415,
+ "285": 8.07053,
+ "286": 7.90383,
+ "287": 7.98943,
+ "288": 8.24817,
+ "289": 8.17501,
+ "290": 8.1223,
+ "291": 8.25189,
+ "292": 8.07521,
+ "293": 8.11506,
+ "294": 7.97731,
+ "295": 7.96769,
+ "296": 8.24169,
+ "297": 7.7969,
+ "298": 8.04212,
+ "299": 7.93566,
+ "300": 7.85747,
+ "301": 8.01266,
+ "302": 7.94773,
+ "303": 7.98757,
+ "304": 7.95384,
+ "305": 7.99123,
+ "306": 7.96975,
+ "307": 7.98439,
+ "308": 7.98952,
+ "309": 8.00261,
+ "310": 7.9741,
+ "311": 7.92233,
+ "312": 7.88139,
+ "313": 7.82552,
+ "314": 7.82224,
+ "315": 7.82054,
+ "316": 7.74347,
+ "317": 7.92489,
+ "318": 7.97914,
+ "319": 7.81977,
+ "320": 7.56707,
+ "321": 7.74617,
+ "322": 7.82544,
+ "323": 7.77083,
+ "324": 7.90489,
+ "325": 7.79565,
+ "326": 7.65124,
+ "327": 7.8567,
+ "328": 7.78421,
+ "329": 7.88464,
+ "330": 7.74866,
+ "331": 7.52173,
+ "332": 7.80746,
+ "333": 7.838,
+ "334": 7.67725,
+ "335": 7.69335,
+ "336": 7.91332,
+ "337": 7.64789,
+ "338": 7.89278,
+ "339": 7.7212,
+ "340": 7.75731,
+ "341": 7.70315,
+ "342": 7.8161,
+ "343": 7.61381,
+ "344": 7.58534,
+ "345": 7.60754,
+ "346": 7.46052,
+ "347": 7.56011,
+ "348": 7.68393,
+ "349": 7.58501,
+ "350": 7.65854,
+ "351": 7.74321,
+ "352": 7.70531,
+ "353": 7.50165,
+ "354": 7.74434,
+ "355": 7.76102,
+ "356": 7.77464,
+ "357": 7.81613,
+ "358": 7.59955,
+ "359": 7.54732,
+ "360": 7.6314,
+ "361": 7.54843,
+ "362": 7.76465,
+ "363": 7.5929,
+ "364": 7.57195,
+ "365": 7.62502,
+ "366": 7.30557,
+ "367": 7.55867,
+ "368": 7.44385,
+ "369": 7.34988,
+ "370": 7.46804,
+ "371": 7.46431,
+ "372": 7.65359,
+ "373": 7.52685,
+ "374": 7.44061,
+ "375": 7.53289,
+ "376": 7.34299,
+ "377": 7.24336,
+ "378": 7.54528,
+ "379": 7.49661,
+ "380": 7.39067,
+ "381": 7.46867,
+ "382": 7.29535,
+ "383": 7.27398,
+ "384": 7.41255,
+ "385": 7.39337,
+ "386": 7.23152,
+ "387": 7.42484,
+ "388": 7.28162,
+ "389": 7.44265,
+ "390": 7.24327,
+ "391": 7.64907,
+ "392": 7.3401,
+ "393": 7.42184,
+ "394": 7.47939,
+ "395": 7.44191,
+ "396": 7.28578,
+ "397": 7.23336,
+ "398": 7.4265,
+ "399": 7.15727,
+ "400": 7.30619,
+ "401": 7.35905,
+ "402": 7.39728,
+ "403": 7.29008,
+ "404": 7.31125,
+ "405": 7.26334,
+ "406": 7.21996,
+ "407": 7.36806,
+ "408": 7.18798,
+ "409": 7.17331,
+ "410": 7.32184,
+ "411": 7.22122,
+ "412": 7.20299,
+ "413": 7.23656,
+ "414": 6.91774,
+ "415": 7.33472,
+ "416": 7.43088,
+ "417": 7.02907,
+ "418": 7.27588,
+ "419": 7.04458,
+ "420": 7.41887,
+ "421": 7.18152,
+ "422": 7.24459,
+ "423": 7.1037,
+ "424": 7.24998,
+ "425": 7.32072,
+ "426": 7.29763,
+ "427": 7.13788,
+ "428": 7.10045,
+ "429": 6.88375,
+ "430": 7.20894,
+ "431": 7.01072,
+ "432": 7.23575,
+ "433": 6.97967,
+ "434": 6.96401,
+ "435": 7.02641,
+ "436": 7.01803,
+ "437": 6.99615,
+ "438": 7.00674,
+ "439": 6.94339,
+ "440": 7.06101,
+ "441": 7.04737,
+ "442": 7.10589,
+ "443": 7.09695,
+ "444": 6.71175,
+ "445": 7.00065,
+ "446": 7.14552,
+ "447": 7.12971,
+ "448": 6.9856,
+ "449": 7.05334,
+ "450": 7.02012,
+ "451": 6.83772,
+ "452": 6.91613,
+ "453": 7.0213,
+ "454": 6.97595,
+ "455": 7.03993,
+ "456": 7.00071,
+ "457": 6.97551,
+ "458": 6.9123,
+ "459": 6.69684,
+ "460": 7.06633,
+ "461": 7.10051,
+ "462": 6.88144,
+ "463": 7.0559,
+ "464": 6.65472,
+ "465": 7.03293,
+ "466": 7.01073,
+ "467": 7.00752,
+ "468": 6.96172,
+ "469": 6.83116,
+ "470": 7.04868,
+ "471": 6.88542,
+ "472": 6.96472,
+ "473": 6.82818,
+ "474": 6.98287,
+ "475": 7.16867,
+ "476": 6.77327,
+ "477": 6.90067,
+ "478": 6.91502,
+ "479": 6.70676,
+ "480": 7.03063,
+ "481": 6.99971,
+ "482": 6.73569,
+ "483": 6.78315,
+ "484": 6.75492,
+ "485": 6.93621,
+ "486": 7.07111,
+ "487": 6.63768,
+ "488": 6.89293,
+ "489": 6.77157,
+ "490": 6.82528,
+ "491": 6.71061,
+ "492": 6.69809,
+ "493": 6.76585,
+ "494": 6.67775,
+ "495": 6.6322,
+ "496": 6.59151,
+ "497": 6.84192,
+ "498": 6.64965,
+ "499": 6.85452,
+ "500": 6.65834,
+ "501": 6.73348,
+ "502": 6.84418,
+ "503": 6.71163,
+ "504": 6.61615,
+ "505": 6.62231,
+ "506": 6.75166,
+ "507": 6.86759,
+ "508": 6.86139,
+ "509": 6.6519,
+ "510": 6.82297,
+ "511": 6.73716,
+ "512": 6.73642,
+ "513": 6.66009,
+ "514": 6.7118,
+ "515": 6.45435,
+ "516": 6.74728,
+ "517": 6.70863,
+ "518": 6.5392,
+ "519": 6.63556,
+ "520": 6.86602,
+ "521": 6.66769,
+ "522": 6.71157,
+ "523": 6.74373,
+ "524": 6.7333,
+ "525": 6.67663,
+ "526": 6.41569,
+ "527": 6.80443,
+ "528": 6.66495,
+ "529": 6.63518,
+ "530": 6.6301,
+ "531": 6.64411,
+ "532": 6.63459,
+ "533": 6.76495,
+ "534": 6.616,
+ "535": 6.74913,
+ "536": 6.6239,
+ "537": 6.6434,
+ "538": 6.52827,
+ "539": 6.55719,
+ "540": 6.58737,
+ "541": 6.45231,
+ "542": 6.67055,
+ "543": 6.67996,
+ "544": 6.68259,
+ "545": 6.81589,
+ "546": 6.63522,
+ "547": 6.41371,
+ "548": 6.72418,
+ "549": 6.70285,
+ "550": 6.53323,
+ "551": 6.75065,
+ "552": 6.64092,
+ "553": 6.48731,
+ "554": 6.63708,
+ "555": 6.46332,
+ "556": 6.61458,
+ "557": 6.63677,
+ "558": 6.38565,
+ "559": 6.3692,
+ "560": 6.58216,
+ "561": 6.73081,
+ "562": 6.64299,
+ "563": 6.73844,
+ "564": 6.35255,
+ "565": 6.51454,
+ "566": 6.70171,
+ "567": 6.56106,
+ "568": 6.50265,
+ "569": 6.45514,
+ "570": 6.36162,
+ "571": 6.63081,
+ "572": 6.31441,
+ "573": 6.58474,
+ "574": 6.47487,
+ "575": 6.64497,
+ "576": 6.5144,
+ "577": 6.52625,
+ "578": 6.48163,
+ "579": 6.46825,
+ "580": 6.56261,
+ "581": 6.60437,
+ "582": 6.4771,
+ "583": 6.51296,
+ "584": 6.52024,
+ "585": 6.42735,
+ "586": 6.4108,
+ "587": 6.45746,
+ "588": 6.56673,
+ "589": 6.621,
+ "590": 6.29254,
+ "591": 6.67434,
+ "592": 6.26521,
+ "593": 6.47366,
+ "594": 6.38354,
+ "595": 6.36115,
+ "596": 6.2568,
+ "597": 6.18986,
+ "598": 6.45701,
+ "599": 6.39679,
+ "600": 6.45722,
+ "601": 6.25834,
+ "602": 6.53121,
+ "603": 6.52132,
+ "604": 6.38584,
+ "605": 6.49567,
+ "606": 6.30898,
+ "607": 6.53302,
+ "608": 6.67073,
+ "609": 6.17489,
+ "610": 6.57129,
+ "611": 6.39847,
+ "612": 6.57574,
+ "613": 6.42027,
+ "614": 6.20379,
+ "615": 6.39524,
+ "616": 6.35574,
+ "617": 6.3731,
+ "618": 6.44662,
+ "619": 6.13758,
+ "620": 6.40912,
+ "621": 6.45464,
+ "622": 6.39595,
+ "623": 6.57783,
+ "624": 6.35632,
+ "625": 6.27818,
+ "626": 6.30281,
+ "627": 6.44415,
+ "628": 6.25054,
+ "629": 6.58652,
+ "630": 6.36782,
+ "631": 6.34368,
+ "632": 6.30974,
+ "633": 6.25462,
+ "634": 6.30949,
+ "635": 6.54358,
+ "636": 6.25015,
+ "637": 6.64776,
+ "638": 6.02409,
+ "639": 6.28701,
+ "640": 6.30064,
+ "641": 6.21691,
+ "642": 6.29144,
+ "643": 6.45879,
+ "644": 6.26727,
+ "645": 6.25042,
+ "646": 6.41348,
+ "647": 6.33909,
+ "648": 6.35909,
+ "649": 6.35352,
+ "650": 6.49097,
+ "651": 6.33277,
+ "652": 6.26536,
+ "653": 6.3912,
+ "654": 6.456,
+ "655": 6.53882,
+ "656": 6.33584,
+ "657": 6.43984,
+ "658": 6.25551,
+ "659": 6.17068,
+ "660": 6.40127,
+ "661": 6.17766,
+ "662": 6.28931,
+ "663": 6.3842,
+ "664": 6.3415,
+ "665": 6.40893,
+ "666": 6.17455,
+ "667": 6.20749,
+ "668": 6.24844,
+ "669": 6.22944,
+ "670": 6.25801,
+ "671": 6.25521,
+ "672": 6.50031,
+ "673": 6.35213,
+ "674": 6.30614,
+ "675": 6.39435,
+ "676": 6.40229,
+ "677": 6.32216,
+ "678": 6.29286,
+ "679": 6.25338,
+ "680": 6.30459,
+ "681": 6.21958,
+ "682": 6.10232,
+ "683": 6.29008,
+ "684": 6.33899,
+ "685": 6.31379,
+ "686": 6.16639,
+ "687": 6.29873,
+ "688": 6.21969,
+ "689": 6.62806,
+ "690": 6.19381,
+ "691": 6.19446,
+ "692": 6.28767,
+ "693": 6.1556,
+ "694": 6.24433,
+ "695": 6.33761,
+ "696": 6.13237,
+ "697": 6.16611,
+ "698": 6.24519,
+ "699": 6.47249,
+ "700": 6.05866,
+ "701": 6.07451,
+ "702": 6.25665,
+ "703": 6.19418,
+ "704": 6.22892,
+ "705": 6.13898,
+ "706": 6.08476,
+ "707": 6.26658,
+ "708": 6.3221,
+ "709": 6.01834,
+ "710": 6.17303,
+ "711": 6.27051,
+ "712": 6.18868,
+ "713": 5.90504,
+ "714": 6.11697,
+ "715": 6.12399,
+ "716": 6.42469,
+ "717": 6.20383,
+ "718": 6.25193,
+ "719": 6.28794,
+ "720": 6.27594,
+ "721": 6.27119,
+ "722": 6.23851,
+ "723": 6.08711,
+ "724": 6.23104,
+ "725": 6.04702,
+ "726": 6.30914,
+ "727": 6.02703,
+ "728": 6.05301,
+ "729": 6.09695,
+ "730": 6.18887,
+ "731": 6.10701,
+ "732": 6.08737,
+ "733": 6.12649,
+ "734": 6.39053,
+ "735": 6.28191,
+ "736": 6.18928,
+ "737": 6.37342,
+ "738": 6.13973,
+ "739": 6.15016,
+ "740": 5.88315,
+ "741": 6.00513,
+ "742": 5.99181,
+ "743": 6.17414,
+ "744": 6.03567,
+ "745": 6.15802,
+ "746": 6.0419,
+ "747": 6.10632,
+ "748": 6.24432,
+ "749": 5.94421,
+ "750": 6.17373,
+ "751": 5.95767,
+ "752": 6.026,
+ "753": 6.03135,
+ "754": 6.29103,
+ "755": 6.13841,
+ "756": 6.25617,
+ "757": 6.02158,
+ "758": 6.20382,
+ "759": 6.23017,
+ "760": 6.02604,
+ "761": 6.19415,
+ "762": 6.22404,
+ "763": 6.03863,
+ "764": 5.96939,
+ "765": 5.93767,
+ "766": 5.97584,
+ "767": 5.82231,
+ "768": 6.19009,
+ "769": 6.27367,
+ "770": 6.30132,
+ "771": 5.79869,
+ "772": 6.03737,
+ "773": 6.19078,
+ "774": 5.88358,
+ "775": 6.02841,
+ "776": 6.13048,
+ "777": 5.88195,
+ "778": 6.06249,
+ "779": 5.87381,
+ "780": 6.14261,
+ "781": 5.859,
+ "782": 6.04884,
+ "783": 5.95885,
+ "784": 5.92152,
+ "785": 6.09739,
+ "786": 6.11046,
+ "787": 5.6605,
+ "788": 5.99861,
+ "789": 6.21843,
+ "790": 6.27053,
+ "791": 5.79359,
+ "792": 5.99968,
+ "793": 6.18765,
+ "794": 6.02986,
+ "795": 6.01035,
+ "796": 6.16717,
+ "797": 6.05921,
+ "798": 6.05646,
+ "799": 6.11879,
+ "800": 6.01798,
+ "801": 6.15546,
+ "802": 5.98694,
+ "803": 6.15454,
+ "804": 6.00758,
+ "805": 5.83074,
+ "806": 6.09642,
+ "807": 6.04914,
+ "808": 5.93046,
+ "809": 5.78038,
+ "810": 6.0199,
+ "811": 5.93619,
+ "812": 5.90923,
+ "813": 5.96746,
+ "814": 6.03466,
+ "815": 5.81425,
+ "816": 6.12545,
+ "817": 5.93601,
+ "818": 6.06715,
+ "819": 6.02104,
+ "820": 5.73227,
+ "821": 5.95128,
+ "822": 6.20423,
+ "823": 5.83075,
+ "824": 5.98673,
+ "825": 6.18815,
+ "826": 6.2004,
+ "827": 6.0571,
+ "828": 6.06994,
+ "829": 5.89451,
+ "830": 5.94394,
+ "831": 5.89854,
+ "832": 5.97376,
+ "833": 6.06789,
+ "834": 5.99968,
+ "835": 6.00619,
+ "836": 5.79821,
+ "837": 6.11358,
+ "838": 5.86715,
+ "839": 5.83886,
+ "840": 6.18611,
+ "841": 5.7851,
+ "842": 5.89507,
+ "843": 5.94982,
+ "844": 6.00961,
+ "845": 6.09957,
+ "846": 5.69383,
+ "847": 5.76478,
+ "848": 5.97139,
+ "849": 6.09575,
+ "850": 5.85278,
+ "851": 6.02073,
+ "852": 5.7573,
+ "853": 6.00125,
+ "854": 6.02094,
+ "855": 5.82189,
+ "856": 6.00704,
+ "857": 6.01208,
+ "858": 6.059,
+ "859": 5.95654,
+ "860": 6.09984,
+ "861": 6.06831,
+ "862": 6.00962,
+ "863": 5.83649,
+ "864": 5.84784,
+ "865": 5.93982,
+ "866": 5.89705,
+ "867": 5.88176,
+ "868": 6.06517,
+ "869": 6.09141,
+ "870": 5.97274,
+ "871": 6.05209,
+ "872": 5.89972,
+ "873": 5.84768,
+ "874": 6.03228,
+ "875": 5.91488,
+ "876": 5.97274,
+ "877": 5.93362,
+ "878": 6.10245,
+ "879": 5.77093,
+ "880": 6.01633,
+ "881": 5.99777,
+ "882": 5.91238,
+ "883": 5.68111,
+ "884": 5.96785,
+ "885": 5.74984,
+ "886": 5.99765,
+ "887": 5.91328,
+ "888": 5.84323,
+ "889": 6.01404,
+ "890": 6.02955,
+ "891": 5.95614,
+ "892": 5.71371,
+ "893": 6.09511,
+ "894": 5.73652,
+ "895": 5.84389,
+ "896": 5.84291,
+ "897": 5.85879,
+ "898": 5.93548,
+ "899": 5.93775,
+ "900": 5.90256,
+ "901": 5.94991,
+ "902": 5.83762,
+ "903": 6.05446,
+ "904": 5.93246,
+ "905": 5.9108,
+ "906": 5.62282,
+ "907": 5.91573,
+ "908": 5.74075,
+ "909": 5.99191,
+ "910": 5.86962,
+ "911": 5.70761,
+ "912": 5.71302,
+ "913": 5.77258,
+ "914": 5.83917,
+ "915": 5.80801,
+ "916": 5.89633,
+ "917": 5.87155,
+ "918": 5.83068,
+ "919": 5.81576,
+ "920": 5.89468,
+ "921": 5.84836,
+ "922": 5.6268,
+ "923": 6.03941,
+ "924": 5.61448,
+ "925": 5.62814,
+ "926": 5.86784,
+ "927": 5.96783,
+ "928": 5.84293,
+ "929": 5.83071,
+ "930": 5.958,
+ "931": 5.76619,
+ "932": 5.5955,
+ "933": 5.64209,
+ "934": 5.81785,
+ "935": 5.64635,
+ "936": 5.84241,
+ "937": 5.97063,
+ "938": 5.59234,
+ "939": 5.79413,
+ "940": 5.97141,
+ "941": 5.72998,
+ "942": 5.83688,
+ "943": 5.8706,
+ "944": 5.96011,
+ "945": 5.70833,
+ "946": 5.56431,
+ "947": 5.75307,
+ "948": 5.79763,
+ "949": 5.82899,
+ "950": 5.84954,
+ "951": 5.72825,
+ "952": 5.70073,
+ "953": 5.68826,
+ "954": 5.73309,
+ "955": 5.53659,
+ "956": 5.62644,
+ "957": 5.84728,
+ "958": 5.80368,
+ "959": 5.58053,
+ "960": 5.80782,
+ "961": 5.84089,
+ "962": 5.76979,
+ "963": 5.77394,
+ "964": 5.71529,
+ "965": 5.64021,
+ "966": 5.60098,
+ "967": 5.73377,
+ "968": 5.74696,
+ "969": 5.83151,
+ "970": 5.6545,
+ "971": 5.71237,
+ "972": 5.85467,
+ "973": 5.67408,
+ "974": 5.72363,
+ "975": 5.87152,
+ "976": 5.71592,
+ "977": 5.77993,
+ "978": 5.69186,
+ "979": 5.59848,
+ "980": 5.76905,
+ "981": 5.90129,
+ "982": 5.48048,
+ "983": 5.62175,
+ "984": 5.55315,
+ "985": 5.59739,
+ "986": 5.64528,
+ "987": 5.57677,
+ "988": 5.71408,
+ "989": 5.69493,
+ "990": 5.62813,
+ "991": 5.84752,
+ "992": 5.78279,
+ "993": 5.8761,
+ "994": 5.70489,
+ "995": 5.74092,
+ "996": 5.74154,
+ "997": 5.8182,
+ "998": 5.83858,
+ "999": 5.83735,
+ "1000": 5.68906,
+ "1001": 5.87121,
+ "1002": 5.76456,
+ "1003": 5.64512,
+ "1004": 5.80746,
+ "1005": 5.53612,
+ "1006": 5.33047,
+ "1007": 5.774,
+ "1008": 5.80226,
+ "1009": 5.65823,
+ "1010": 5.78646,
+ "1011": 5.9004,
+ "1012": 5.62598,
+ "1013": 5.62239,
+ "1014": 5.68357,
+ "1015": 5.56531,
+ "1016": 5.87733,
+ "1017": 5.83781,
+ "1018": 5.62737,
+ "1019": 5.73341,
+ "1020": 5.62002,
+ "1021": 5.85758,
+ "1022": 5.50501,
+ "1023": 5.6575,
+ "1024": 5.75105,
+ "1025": 5.57372,
+ "1026": 5.41661,
+ "1027": 5.60767,
+ "1028": 5.69364,
+ "1029": 5.69065,
+ "1030": 5.68957,
+ "1031": 5.41103,
+ "1032": 5.79084,
+ "1033": 5.58319,
+ "1034": 5.62563,
+ "1035": 5.72246,
+ "1036": 5.63308,
+ "1037": 5.36744,
+ "1038": 5.67126,
+ "1039": 5.651,
+ "1040": 5.57372,
+ "1041": 5.59982,
+ "1042": 5.8145,
+ "1043": 5.56584,
+ "1044": 5.47023,
+ "1045": 5.9635,
+ "1046": 5.48741,
+ "1047": 5.3904,
+ "1048": 5.50346,
+ "1049": 5.6788,
+ "1050": 5.70095,
+ "1051": 5.58101,
+ "1052": 5.68875,
+ "1053": 5.63262,
+ "1054": 5.46043,
+ "1055": 5.60346,
+ "1056": 5.67617,
+ "1057": 5.76024,
+ "1058": 5.5724,
+ "1059": 5.75531,
+ "1060": 5.82787,
+ "1061": 5.47878,
+ "1062": 5.65549,
+ "1063": 5.5045,
+ "1064": 5.5927,
+ "1065": 5.55825,
+ "1066": 5.74638,
+ "1067": 5.67127,
+ "1068": 5.43926,
+ "1069": 5.60481,
+ "1070": 5.80837,
+ "1071": 5.50476,
+ "1072": 5.62204,
+ "1073": 5.62611,
+ "1074": 5.52697,
+ "1075": 5.70862,
+ "1076": 5.59905,
+ "1077": 5.71103,
+ "1078": 5.56612,
+ "1079": 5.62142,
+ "1080": 5.64752,
+ "1081": 5.62038,
+ "1082": 5.50393,
+ "1083": 5.64282,
+ "1084": 5.55164,
+ "1085": 5.40773,
+ "1086": 5.62139,
+ "1087": 5.44411,
+ "1088": 5.51932,
+ "1089": 5.77279,
+ "1090": 5.53271,
+ "1091": 5.52175,
+ "1092": 5.40907,
+ "1093": 5.70113,
+ "1094": 5.57459,
+ "1095": 5.58039,
+ "1096": 5.61295,
+ "1097": 5.64663,
+ "1098": 5.65167,
+ "1099": 5.51557,
+ "1100": 5.64118,
+ "1101": 5.67408,
+ "1102": 5.53961,
+ "1103": 5.54845,
+ "1104": 5.53812,
+ "1105": 5.55257,
+ "1106": 5.68309,
+ "1107": 5.68054,
+ "1108": 5.78434,
+ "1109": 5.53369,
+ "1110": 5.66712,
+ "1111": 5.58517,
+ "1112": 5.58393,
+ "1113": 5.62572,
+ "1114": 5.61393,
+ "1115": 5.60046,
+ "1116": 5.66171,
+ "1117": 5.65023,
+ "1118": 5.65104,
+ "1119": 5.70663,
+ "1120": 5.6261,
+ "1121": 5.37461,
+ "1122": 5.23338,
+ "1123": 5.47208,
+ "1124": 5.65224,
+ "1125": 5.67607,
+ "1126": 5.681,
+ "1127": 5.56352,
+ "1128": 5.62462,
+ "1129": 5.29926,
+ "1130": 5.5448,
+ "1131": 5.63168,
+ "1132": 5.72192,
+ "1133": 5.51813,
+ "1134": 5.55963,
+ "1135": 5.51558,
+ "1136": 5.42274,
+ "1137": 5.45955,
+ "1138": 5.5649,
+ "1139": 5.41546,
+ "1140": 5.25974,
+ "1141": 5.58414,
+ "1142": 5.64178,
+ "1143": 5.38541,
+ "1144": 5.38111,
+ "1145": 5.3661,
+ "1146": 5.63202,
+ "1147": 5.48956,
+ "1148": 5.50313,
+ "1149": 5.51985,
+ "1150": 5.3961,
+ "1151": 5.55449,
+ "1152": 5.41056,
+ "1153": 5.45054,
+ "1154": 5.49205,
+ "1155": 5.43405,
+ "1156": 5.3447,
+ "1157": 5.65808,
+ "1158": 5.39324,
+ "1159": 5.32889,
+ "1160": 5.78987,
+ "1161": 5.53712,
+ "1162": 5.45338,
+ "1163": 5.52309,
+ "1164": 5.37879,
+ "1165": 5.52409,
+ "1166": 5.48515,
+ "1167": 5.35881,
+ "1168": 5.48927,
+ "1169": 5.39923,
+ "1170": 5.58513,
+ "1171": 5.48423,
+ "1172": 5.63291,
+ "1173": 5.61809,
+ "1174": 5.50573,
+ "1175": 5.34078,
+ "1176": 5.38022,
+ "1177": 5.55042,
+ "1178": 5.46996,
+ "1179": 5.49308,
+ "1180": 5.46218,
+ "1181": 5.55454,
+ "1182": 5.59358,
+ "1183": 5.75853,
+ "1184": 5.54174,
+ "1185": 5.28866,
+ "1186": 5.60177,
+ "1187": 5.554,
+ "1188": 5.51332,
+ "1189": 5.38643,
+ "1190": 5.39905,
+ "1191": 5.38627,
+ "1192": 5.49703,
+ "1193": 5.46042,
+ "1194": 5.45566,
+ "1195": 5.32449,
+ "1196": 5.52096,
+ "1197": 5.47575,
+ "1198": 5.52317,
+ "1199": 5.3855,
+ "1200": 5.32742,
+ "1201": 5.49044,
+ "1202": 5.4315,
+ "1203": 5.49238,
+ "1204": 5.40921,
+ "1205": 5.48998,
+ "1206": 5.33838,
+ "1207": 5.57937,
+ "1208": 5.42211,
+ "1209": 5.29022,
+ "1210": 5.4983,
+ "1211": 5.50889,
+ "1212": 5.59156,
+ "1213": 5.41573,
+ "1214": 5.50278,
+ "1215": 5.23495,
+ "1216": 5.40683,
+ "1217": 5.38308,
+ "1218": 5.45531,
+ "1219": 5.47683,
+ "1220": 5.3853,
+ "1221": 5.44609,
+ "1222": 5.31266,
+ "1223": 5.47286,
+ "1224": 5.41866,
+ "1225": 5.43143,
+ "1226": 5.32801,
+ "1227": 5.48242,
+ "1228": 5.74857,
+ "1229": 5.33201,
+ "1230": 5.41324,
+ "1231": 5.07614,
+ "1232": 5.79088,
+ "1233": 5.28685,
+ "1234": 5.24646,
+ "1235": 5.37184,
+ "1236": 5.48222,
+ "1237": 5.20747,
+ "1238": 5.41301,
+ "1239": 5.41068,
+ "1240": 5.4627,
+ "1241": 5.5684,
+ "1242": 5.4516,
+ "1243": 5.43065,
+ "1244": 5.51464,
+ "1245": 5.19172,
+ "1246": 5.71661,
+ "1247": 5.42312,
+ "1248": 5.29995,
+ "1249": 5.39852,
+ "1250": 5.34054,
+ "1251": 5.42244,
+ "1252": 5.57066,
+ "1253": 5.4859,
+ "1254": 5.30673,
+ "1255": 5.51321,
+ "1256": 5.60481,
+ "1257": 5.42813,
+ "1258": 5.56189,
+ "1259": 5.48616,
+ "1260": 5.50582,
+ "1261": 5.64425,
+ "1262": 5.40191,
+ "1263": 5.32952,
+ "1264": 5.50729,
+ "1265": 5.30431,
+ "1266": 5.23505,
+ "1267": 5.37396,
+ "1268": 5.39035,
+ "1269": 5.15072,
+ "1270": 5.39878,
+ "1271": 5.27991,
+ "1272": 5.52085,
+ "1273": 5.29741,
+ "1274": 5.35086,
+ "1275": 5.38038,
+ "1276": 5.39664,
+ "1277": 5.46246,
+ "1278": 5.35217,
+ "1279": 5.4441,
+ "1280": 5.45873,
+ "1281": 5.40172,
+ "1282": 5.38508,
+ "1283": 5.42297,
+ "1284": 5.34373,
+ "1285": 5.50278,
+ "1286": 5.33246,
+ "1287": 5.58845,
+ "1288": 5.26426,
+ "1289": 5.42779,
+ "1290": 5.50068,
+ "1291": 5.49817,
+ "1292": 5.4488,
+ "1293": 5.41753,
+ "1294": 5.49483,
+ "1295": 5.34718,
+ "1296": 5.18599,
+ "1297": 5.16897,
+ "1298": 5.11885,
+ "1299": 5.30338,
+ "1300": 5.20675,
+ "1301": 5.30456,
+ "1302": 5.2763,
+ "1303": 5.35362,
+ "1304": 5.43841,
+ "1305": 5.36638,
+ "1306": 5.24943,
+ "1307": 5.19449,
+ "1308": 5.27283,
+ "1309": 5.41315,
+ "1310": 5.26284,
+ "1311": 5.3832,
+ "1312": 5.35833,
+ "1313": 5.29677,
+ "1314": 5.29502,
+ "1315": 5.42183,
+ "1316": 5.26273,
+ "1317": 5.27805,
+ "1318": 5.2182,
+ "1319": 5.34723,
+ "1320": 5.4142,
+ "1321": 5.4463,
+ "1322": 5.46171,
+ "1323": 5.36998,
+ "1324": 5.25128,
+ "1325": 5.4067,
+ "1326": 5.54091,
+ "1327": 5.39133,
+ "1328": 5.22259,
+ "1329": 5.41812,
+ "1330": 5.40133,
+ "1331": 5.31823,
+ "1332": 5.30575,
+ "1333": 5.36965,
+ "1334": 5.44656,
+ "1335": 5.36661,
+ "1336": 5.44503,
+ "1337": 5.47119,
+ "1338": 5.3085,
+ "1339": 5.13907,
+ "1340": 5.4227,
+ "1341": 5.34674,
+ "1342": 5.36284,
+ "1343": 5.48248,
+ "1344": 5.37874,
+ "1345": 5.34762,
+ "1346": 5.08052,
+ "1347": 5.38817,
+ "1348": 5.49142,
+ "1349": 5.41431,
+ "1350": 5.02674,
+ "1351": 5.31977,
+ "1352": 5.15804,
+ "1353": 5.34826,
+ "1354": 5.36561,
+ "1355": 5.1145,
+ "1356": 5.25656,
+ "1357": 5.2955,
+ "1358": 5.15794,
+ "1359": 5.11147,
+ "1360": 5.17661,
+ "1361": 5.30714,
+ "1362": 5.06592,
+ "1363": 5.29543,
+ "1364": 5.40278,
+ "1365": 5.02306,
+ "1366": 5.12233,
+ "1367": 5.33295,
+ "1368": 5.18603,
+ "1369": 5.22652,
+ "1370": 5.19724,
+ "1371": 5.2791,
+ "1372": 5.26033,
+ "1373": 5.28584,
+ "1374": 5.27776,
+ "1375": 5.45659,
+ "1376": 5.27299,
+ "1377": 5.26758,
+ "1378": 5.31311,
+ "1379": 5.22635,
+ "1380": 5.26193,
+ "1381": 5.48118,
+ "1382": 5.08967,
+ "1383": 5.3788,
+ "1384": 5.36505,
+ "1385": 5.39365,
+ "1386": 5.17153,
+ "1387": 5.16382,
+ "1388": 5.28083,
+ "1389": 5.30555,
+ "1390": 5.25407,
+ "1391": 5.27226,
+ "1392": 5.3739,
+ "1393": 5.38451,
+ "1394": 5.40135,
+ "1395": 5.32702,
+ "1396": 5.21232,
+ "1397": 5.28713,
+ "1398": 5.37103,
+ "1399": 5.36623,
+ "1400": 5.27074,
+ "1401": 5.35908,
+ "1402": 5.42851,
+ "1403": 5.20432,
+ "1404": 5.28451,
+ "1405": 5.12263,
+ "1406": 4.98801,
+ "1407": 5.39897,
+ "1408": 5.19336,
+ "1409": 5.40028,
+ "1410": 5.37671,
+ "1411": 4.91827,
+ "1412": 5.35523,
+ "1413": 5.4158,
+ "1414": 5.22351,
+ "1415": 5.44046,
+ "1416": 5.33188,
+ "1417": 5.39368,
+ "1418": 5.30633,
+ "1419": 5.32456,
+ "1420": 5.44579,
+ "1421": 5.40561,
+ "1422": 5.43134,
+ "1423": 5.00394,
+ "1424": 5.33553,
+ "1425": 5.5936,
+ "1426": 5.23608,
+ "1427": 5.3279,
+ "1428": 5.33925,
+ "1429": 5.0801,
+ "1430": 5.33434,
+ "1431": 5.32204,
+ "1432": 5.34259,
+ "1433": 5.1874,
+ "1434": 5.16086,
+ "1435": 5.19426,
+ "1436": 5.10722,
+ "1437": 5.22856,
+ "1438": 5.31974,
+ "1439": 5.34687,
+ "1440": 5.34743,
+ "1441": 5.16979,
+ "1442": 5.22323,
+ "1443": 5.21197,
+ "1444": 5.12899,
+ "1445": 5.07532,
+ "1446": 5.27007,
+ "1447": 5.26523,
+ "1448": 5.29192,
+ "1449": 5.2518,
+ "1450": 5.34935,
+ "1451": 5.07283,
+ "1452": 5.27619,
+ "1453": 5.17196,
+ "1454": 5.01515,
+ "1455": 5.12685,
+ "1456": 5.27398,
+ "1457": 5.1864,
+ "1458": 5.01072,
+ "1459": 5.21897,
+ "1460": 5.2377,
+ "1461": 5.08143,
+ "1462": 4.97934,
+ "1463": 5.14902,
+ "1464": 5.2161,
+ "1465": 5.27361,
+ "1466": 5.34832,
+ "1467": 5.34339,
+ "1468": 5.22444,
+ "1469": 5.04562,
+ "1470": 5.12601,
+ "1471": 5.24908,
+ "1472": 5.11656,
+ "1473": 5.10515,
+ "1474": 5.22191,
+ "1475": 5.19387,
+ "1476": 5.15794,
+ "1477": 5.26513,
+ "1478": 5.30415,
+ "1479": 5.01489,
+ "1480": 5.18581,
+ "1481": 5.25047,
+ "1482": 5.3558,
+ "1483": 5.26651,
+ "1484": 4.93164,
+ "1485": 5.29348,
+ "1486": 5.04626,
+ "1487": 4.88636,
+ "1488": 5.18295,
+ "1489": 5.10593,
+ "1490": 5.05119,
+ "1491": 5.32192,
+ "1492": 5.2293,
+ "1493": 4.94553,
+ "1494": 5.11499,
+ "1495": 5.14123,
+ "1496": 5.06404,
+ "1497": 5.37411,
+ "1498": 5.31566,
+ "1499": 5.14408,
+ "1500": 5.10189,
+ "1501": 5.03886,
+ "1502": 5.1545,
+ "1503": 5.43745,
+ "1504": 5.32486,
+ "1505": 5.0025,
+ "1506": 5.14606,
+ "1507": 5.16619,
+ "1508": 5.16472,
+ "1509": 5.31744,
+ "1510": "nan",
+ "1511": "nan",
+ "1512": "nan",
+ "1513": "nan",
+ "1514": "nan",
+ "1515": "nan",
+ "1516": "nan",
+ "1517": "nan",
+ "1518": "nan",
+ "1519": "nan",
+ "1520": "nan",
+ "1521": "nan",
+ "1522": "nan",
+ "1523": "nan",
+ "1524": "nan",
+ "1525": "nan",
+ "1526": "nan",
+ "1527": "nan",
+ "1528": "nan",
+ "1529": "nan",
+ "1530": "nan",
+ "1531": "nan",
+ "1532": "nan",
+ "1533": "nan",
+ "1534": "nan",
+ "1535": "nan",
+ "1536": "nan",
+ "1537": "nan",
+ "1538": "nan",
+ "1539": "nan",
+ "1540": "nan",
+ "1541": "nan",
+ "1542": "nan",
+ "1543": "nan",
+ "1544": "nan",
+ "1545": "nan",
+ "1546": "nan",
+ "1547": "nan",
+ "1548": "nan",
+ "1549": "nan",
+ "1550": "nan",
+ "1551": "nan",
+ "1552": "nan",
+ "1553": "nan",
+ "1554": "nan",
+ "1555": "nan",
+ "1556": "nan",
+ "1557": "nan",
+ "1558": "nan",
+ "1559": "nan",
+ "1560": "nan",
+ "1561": "nan",
+ "1562": "nan",
+ "1563": "nan",
+ "1564": "nan",
+ "1565": "nan",
+ "1566": "nan",
+ "1567": "nan",
+ "1568": "nan",
+ "1569": "nan",
+ "1570": "nan",
+ "1571": "nan",
+ "1572": "nan",
+ "1573": "nan",
+ "1574": "nan",
+ "1575": "nan",
+ "1576": "nan",
+ "1577": "nan",
+ "1578": "nan",
+ "1579": "nan",
+ "1580": "nan",
+ "1581": "nan",
+ "1582": "nan",
+ "1583": "nan",
+ "1584": "nan",
+ "1585": "nan",
+ "1586": "nan",
+ "1587": "nan",
+ "1588": "nan",
+ "1589": "nan",
+ "1590": "nan",
+ "1591": "nan",
+ "1592": "nan",
+ "1593": "nan",
+ "1594": "nan",
+ "1595": "nan",
+ "1596": "nan",
+ "1597": "nan",
+ "1598": "nan",
+ "1599": "nan",
+ "1600": "nan",
+ "1601": "nan",
+ "1602": "nan",
+ "1603": "nan",
+ "1604": "nan",
+ "1605": "nan",
+ "1606": "nan",
+ "1607": "nan",
+ "1608": "nan",
+ "1609": "nan",
+ "1610": "nan",
+ "1611": "nan",
+ "1612": "nan",
+ "1613": "nan",
+ "1614": "nan",
+ "1615": "nan",
+ "1616": "nan",
+ "1617": "nan",
+ "1618": "nan",
+ "1619": "nan",
+ "1620": "nan",
+ "1621": "nan",
+ "1622": "nan",
+ "1623": "nan",
+ "1624": "nan",
+ "1625": "nan",
+ "1626": "nan",
+ "1627": "nan",
+ "1628": "nan",
+ "1629": "nan",
+ "1630": "nan",
+ "1631": "nan",
+ "1632": "nan",
+ "1633": "nan",
+ "1634": "nan",
+ "1635": "nan",
+ "1636": "nan",
+ "1637": "nan",
+ "1638": "nan",
+ "1639": "nan",
+ "1640": "nan",
+ "1641": "nan",
+ "1642": "nan",
+ "1643": "nan",
+ "1644": "nan",
+ "1645": "nan",
+ "1646": "nan",
+ "1647": "nan",
+ "1648": "nan",
+ "1649": "nan",
+ "1650": "nan",
+ "1651": "nan",
+ "1652": "nan",
+ "1653": "nan",
+ "1654": "nan",
+ "1655": "nan",
+ "1656": "nan",
+ "1657": "nan",
+ "1658": "nan",
+ "1659": "nan",
+ "1660": "nan",
+ "1661": "nan",
+ "1662": "nan",
+ "1663": "nan",
+ "1664": "nan",
+ "1665": "nan",
+ "1666": "nan",
+ "1667": "nan",
+ "1668": "nan",
+ "1669": "nan",
+ "1670": "nan",
+ "1671": "nan",
+ "1672": "nan",
+ "1673": "nan",
+ "1674": "nan",
+ "1675": "nan",
+ "1676": "nan",
+ "1677": "nan",
+ "1678": "nan",
+ "1679": "nan",
+ "1680": "nan",
+ "1681": "nan",
+ "1682": "nan",
+ "1683": "nan",
+ "1684": "nan",
+ "1685": "nan",
+ "1686": "nan",
+ "1687": "nan",
+ "1688": "nan",
+ "1689": "nan",
+ "1690": "nan",
+ "1691": "nan",
+ "1692": "nan",
+ "1693": "nan",
+ "1694": "nan",
+ "1695": "nan",
+ "1696": "nan",
+ "1697": "nan",
+ "1698": "nan",
+ "1699": "nan",
+ "1700": "nan",
+ "1701": "nan",
+ "1702": "nan",
+ "1703": "nan",
+ "1704": "nan",
+ "1705": "nan",
+ "1706": "nan",
+ "1707": "nan",
+ "1708": "nan",
+ "1709": "nan",
+ "1710": "nan",
+ "1711": "nan",
+ "1712": "nan",
+ "1713": "nan",
+ "1714": "nan",
+ "1715": "nan",
+ "1716": "nan",
+ "1717": "nan",
+ "1718": "nan",
+ "1719": "nan",
+ "1720": "nan",
+ "1721": "nan",
+ "1722": "nan",
+ "1723": "nan",
+ "1724": "nan",
+ "1725": "nan",
+ "1726": "nan",
+ "1727": "nan",
+ "1728": "nan",
+ "1729": "nan",
+ "1730": "nan",
+ "1731": "nan",
+ "1732": "nan",
+ "1733": "nan",
+ "1734": "nan",
+ "1735": "nan",
+ "1736": "nan",
+ "1737": "nan",
+ "1738": "nan",
+ "1739": "nan",
+ "1740": "nan",
+ "1741": "nan",
+ "1742": "nan",
+ "1743": "nan",
+ "1744": "nan",
+ "1745": "nan",
+ "1746": "nan",
+ "1747": "nan",
+ "1748": "nan",
+ "1749": "nan",
+ "1750": "nan",
+ "1751": "nan",
+ "1752": "nan",
+ "1753": "nan",
+ "1754": "nan",
+ "1755": "nan",
+ "1756": "nan",
+ "1757": "nan",
+ "1758": "nan",
+ "1759": "nan",
+ "1760": "nan",
+ "1761": "nan",
+ "1762": "nan",
+ "1763": "nan",
+ "1764": "nan",
+ "1765": "nan",
+ "1766": "nan",
+ "1767": "nan",
+ "1768": "nan",
+ "1769": "nan",
+ "1770": "nan",
+ "1771": "nan",
+ "1772": "nan",
+ "1773": "nan",
+ "1774": "nan",
+ "1775": "nan",
+ "1776": "nan",
+ "1777": "nan",
+ "1778": "nan",
+ "1779": "nan",
+ "1780": "nan",
+ "1781": "nan",
+ "1782": "nan",
+ "1783": "nan",
+ "1784": "nan",
+ "1785": "nan",
+ "1786": "nan",
+ "1787": "nan",
+ "1788": "nan",
+ "1789": "nan",
+ "1790": "nan",
+ "1791": "nan",
+ "1792": "nan",
+ "1793": "nan",
+ "1794": "nan",
+ "1795": "nan",
+ "1796": "nan",
+ "1797": "nan",
+ "1798": "nan",
+ "1799": "nan",
+ "1800": "nan",
+ "1801": "nan",
+ "1802": "nan",
+ "1803": "nan",
+ "1804": "nan",
+ "1805": "nan",
+ "1806": "nan",
+ "1807": "nan",
+ "1808": "nan",
+ "1809": "nan",
+ "1810": "nan",
+ "1811": "nan",
+ "1812": "nan",
+ "1813": "nan",
+ "1814": "nan",
+ "1815": "nan",
+ "1816": "nan",
+ "1817": "nan",
+ "1818": "nan",
+ "1819": "nan",
+ "1820": "nan",
+ "1821": "nan",
+ "1822": "nan",
+ "1823": "nan",
+ "1824": "nan",
+ "1825": "nan",
+ "1826": "nan",
+ "1827": "nan",
+ "1828": "nan",
+ "1829": "nan",
+ "1830": "nan",
+ "1831": "nan",
+ "1832": "nan",
+ "1833": "nan",
+ "1834": "nan",
+ "1835": "nan",
+ "1836": "nan",
+ "1837": "nan",
+ "1838": "nan",
+ "1839": "nan",
+ "1840": "nan",
+ "1841": "nan",
+ "1842": "nan",
+ "1843": "nan",
+ "1844": "nan",
+ "1845": "nan",
+ "1846": "nan",
+ "1847": "nan",
+ "1848": "nan",
+ "1849": "nan",
+ "1850": "nan",
+ "1851": "nan",
+ "1852": "nan",
+ "1853": "nan",
+ "1854": "nan",
+ "1855": "nan",
+ "1856": "nan",
+ "1857": "nan",
+ "1858": "nan",
+ "1859": "nan",
+ "1860": "nan",
+ "1861": "nan",
+ "1862": "nan",
+ "1863": "nan",
+ "1864": "nan",
+ "1865": "nan",
+ "1866": "nan",
+ "1867": "nan",
+ "1868": "nan",
+ "1869": "nan",
+ "1870": "nan",
+ "1871": "nan",
+ "1872": "nan",
+ "1873": "nan",
+ "1874": "nan",
+ "1875": "nan",
+ "1876": "nan",
+ "1877": "nan",
+ "1878": "nan",
+ "1879": "nan",
+ "1880": "nan",
+ "1881": "nan",
+ "1882": "nan",
+ "1883": "nan",
+ "1884": "nan",
+ "1885": "nan",
+ "1886": "nan",
+ "1887": "nan",
+ "1888": "nan",
+ "1889": "nan",
+ "1890": "nan",
+ "1891": "nan",
+ "1892": "nan",
+ "1893": "nan",
+ "1894": "nan",
+ "1895": "nan",
+ "1896": "nan",
+ "1897": "nan",
+ "1898": "nan",
+ "1899": "nan",
+ "1900": "nan",
+ "1901": "nan",
+ "1902": "nan",
+ "1903": "nan",
+ "1904": "nan",
+ "1905": "nan",
+ "1906": "nan",
+ "1907": "nan",
+ "1908": "nan",
+ "1909": "nan",
+ "1910": "nan",
+ "1911": "nan",
+ "1912": "nan",
+ "1913": "nan",
+ "1914": "nan",
+ "1915": "nan",
+ "1916": "nan",
+ "1917": "nan",
+ "1918": "nan",
+ "1919": "nan",
+ "1920": "nan",
+ "1921": "nan",
+ "1922": "nan",
+ "1923": "nan",
+ "1924": "nan",
+ "1925": "nan",
+ "1926": "nan",
+ "1927": "nan",
+ "1928": "nan",
+ "1929": "nan",
+ "1930": "nan",
+ "1931": "nan",
+ "1932": "nan",
+ "1933": "nan",
+ "1934": "nan",
+ "1935": "nan",
+ "1936": "nan",
+ "1937": "nan",
+ "1938": "nan",
+ "1939": "nan",
+ "1940": "nan",
+ "1941": "nan",
+ "1942": "nan",
+ "1943": "nan",
+ "1944": "nan",
+ "1945": "nan",
+ "1946": "nan",
+ "1947": "nan",
+ "1948": "nan",
+ "1949": "nan",
+ "1950": "nan",
+ "1951": "nan",
+ "1952": "nan",
+ "1953": "nan",
+ "1954": "nan",
+ "1955": "nan",
+ "1956": "nan",
+ "1957": "nan",
+ "1958": "nan",
+ "1959": "nan",
+ "1960": "nan",
+ "1961": "nan",
+ "1962": "nan",
+ "1963": "nan",
+ "1964": "nan",
+ "1965": "nan",
+ "1966": "nan",
+ "1967": "nan",
+ "1968": "nan",
+ "1969": "nan",
+ "1970": "nan",
+ "1971": "nan",
+ "1972": "nan",
+ "1973": "nan",
+ "1974": "nan",
+ "1975": "nan",
+ "1976": "nan",
+ "1977": "nan",
+ "1978": "nan",
+ "1979": "nan",
+ "1980": "nan",
+ "1981": "nan",
+ "1982": "nan",
+ "1983": "nan",
+ "1984": "nan",
+ "1985": "nan",
+ "1986": "nan",
+ "1987": "nan",
+ "1988": "nan",
+ "1989": "nan",
+ "1990": "nan",
+ "1991": "nan",
+ "1992": "nan",
+ "1993": "nan",
+ "1994": "nan",
+ "1995": "nan",
+ "1996": "nan",
+ "1997": "nan",
+ "1998": "nan",
+ "1999": "nan",
+ "2000": "nan"
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 32.0,
+ "2": 25.0,
+ "3": 35.0,
+ "4": 29.0,
+ "5": 34.0,
+ "6": 28.0,
+ "7": 37.0,
+ "8": 29.0,
+ "9": 31.0,
+ "10": 40.0,
+ "11": 23.0,
+ "12": 28.0,
+ "13": 31.0,
+ "14": 38.0,
+ "15": 34.0,
+ "16": 34.0,
+ "17": 37.0,
+ "18": 32.0,
+ "19": 27.0,
+ "20": 33.0,
+ "21": 30.0,
+ "22": 41.0,
+ "23": 47.0,
+ "24": 34.0,
+ "25": 30.0,
+ "26": 38.0,
+ "27": 32.0,
+ "28": 42.0,
+ "29": 49.0,
+ "30": 50.0,
+ "31": 44.0,
+ "32": 43.0,
+ "33": 56.0,
+ "34": 42.0,
+ "35": 39.0,
+ "36": 42.0,
+ "37": 45.0,
+ "38": 44.0,
+ "39": 35.0,
+ "40": 57.0,
+ "41": 65.0,
+ "42": 67.0,
+ "43": 54.0,
+ "44": 46.0,
+ "45": 60.0,
+ "46": 44.0,
+ "47": 66.0,
+ "48": 49.0,
+ "49": 56.0,
+ "50": 60.0,
+ "51": 38.0,
+ "52": 44.0,
+ "53": 40.0,
+ "54": 54.0,
+ "55": 61.0,
+ "56": 73.0,
+ "57": 53.0,
+ "58": 73.0,
+ "59": 59.0,
+ "60": 59.0,
+ "61": 77.0,
+ "62": 52.0,
+ "63": 59.0,
+ "64": 72.0,
+ "65": 75.0,
+ "66": 56.0,
+ "67": 79.0,
+ "68": 63.0,
+ "69": 61.0,
+ "70": 55.0,
+ "71": 92.0,
+ "72": 67.0,
+ "73": 61.0,
+ "74": 62.0,
+ "75": 68.0,
+ "76": 69.0,
+ "77": 79.0,
+ "78": 71.0,
+ "79": 56.0,
+ "80": 58.0,
+ "81": 65.0,
+ "82": 62.0,
+ "83": 69.0,
+ "84": 64.0,
+ "85": 57.0,
+ "86": 73.0,
+ "87": 77.0,
+ "88": 59.0,
+ "89": 69.0,
+ "90": 70.0,
+ "91": 76.0,
+ "92": 71.0,
+ "93": 67.0,
+ "94": 66.0,
+ "95": 63.0,
+ "96": 69.0,
+ "97": 72.0,
+ "98": 58.0,
+ "99": 66.0,
+ "100": 63.0,
+ "101": 67.0,
+ "102": 67.0,
+ "103": 79.0,
+ "104": 70.0,
+ "105": 80.0,
+ "106": 71.0,
+ "107": 73.0,
+ "108": 89.0,
+ "109": 59.0,
+ "110": 97.0,
+ "111": 73.0,
+ "112": 65.0,
+ "113": 69.0,
+ "114": 70.0,
+ "115": 59.0,
+ "116": 75.0,
+ "117": 70.0,
+ "118": 61.0,
+ "119": 51.0,
+ "120": 73.0,
+ "121": 69.0,
+ "122": 63.0,
+ "123": 57.0,
+ "124": 71.0,
+ "125": 87.0,
+ "126": 50.0,
+ "127": 68.0,
+ "128": 77.0,
+ "129": 61.0,
+ "130": 77.0,
+ "131": 61.0,
+ "132": 80.0,
+ "133": 69.0,
+ "134": 66.0,
+ "135": 70.0,
+ "136": 90.0,
+ "137": 51.0,
+ "138": 75.0,
+ "139": 67.0,
+ "140": 70.0,
+ "141": 72.0,
+ "142": 64.0,
+ "143": 68.0,
+ "144": 65.0,
+ "145": 71.0,
+ "146": 59.0,
+ "147": 58.0,
+ "148": 69.0,
+ "149": 57.0,
+ "150": 65.0,
+ "151": 59.0,
+ "152": 52.0,
+ "153": 53.0,
+ "154": 63.0,
+ "155": 71.0,
+ "156": 54.0,
+ "157": 57.0,
+ "158": 51.0,
+ "159": 63.0,
+ "160": 62.0,
+ "161": 58.0,
+ "162": 78.0,
+ "163": 43.0,
+ "164": 60.0,
+ "165": 62.0,
+ "166": 72.0,
+ "167": 64.0,
+ "168": 62.0,
+ "169": 72.0,
+ "170": 48.0,
+ "171": 54.0,
+ "172": 73.0,
+ "173": 58.0,
+ "174": 70.0,
+ "175": 71.0,
+ "176": 48.0,
+ "177": 69.0,
+ "178": 43.0,
+ "179": 71.0,
+ "180": 60.0,
+ "181": 52.0,
+ "182": 74.0,
+ "183": 69.0,
+ "184": 64.0,
+ "185": 76.0,
+ "186": 83.0,
+ "187": 80.0,
+ "188": 76.0,
+ "189": 71.0,
+ "190": 45.0,
+ "191": 73.0,
+ "192": 71.0,
+ "193": 78.0,
+ "194": 69.0,
+ "195": 64.0,
+ "196": 70.0,
+ "197": 54.0,
+ "198": 55.0,
+ "199": 54.0,
+ "200": 68.0,
+ "201": 53.0,
+ "202": 68.0,
+ "203": 65.0,
+ "204": 56.0,
+ "205": 54.0,
+ "206": 68.0,
+ "207": 61.0,
+ "208": 47.0,
+ "209": 70.0,
+ "210": 55.0,
+ "211": 85.0,
+ "212": 77.0,
+ "213": 65.0,
+ "214": 60.0,
+ "215": 64.0,
+ "216": 67.0,
+ "217": 50.0,
+ "218": 64.0,
+ "219": 74.0,
+ "220": 60.0,
+ "221": 53.0,
+ "222": 55.0,
+ "223": 82.0,
+ "224": 58.0,
+ "225": 73.0,
+ "226": 72.0,
+ "227": 85.0,
+ "228": 59.0,
+ "229": 80.0,
+ "230": 59.0,
+ "231": 68.0,
+ "232": 73.0,
+ "233": 82.0,
+ "234": 74.0,
+ "235": 91.0,
+ "236": 57.0,
+ "237": 70.0,
+ "238": 60.0,
+ "239": 62.0,
+ "240": 79.0,
+ "241": 65.0,
+ "242": 69.0,
+ "243": 63.0,
+ "244": 87.0,
+ "245": 82.0,
+ "246": 84.0,
+ "247": 81.0,
+ "248": 63.0,
+ "249": 91.0,
+ "250": 65.0,
+ "251": 88.0,
+ "252": 84.0,
+ "253": 62.0,
+ "254": 86.0,
+ "255": 86.0,
+ "256": 82.0,
+ "257": 79.0,
+ "258": 106.0,
+ "259": 77.0,
+ "260": 81.0,
+ "261": 79.0,
+ "262": 89.0,
+ "263": 79.0,
+ "264": 72.0,
+ "265": 69.0,
+ "266": 71.0,
+ "267": 58.0,
+ "268": 71.0,
+ "269": 76.0,
+ "270": 87.0,
+ "271": 70.0,
+ "272": 89.0,
+ "273": 68.0,
+ "274": 77.0,
+ "275": 63.0,
+ "276": 65.0,
+ "277": 71.0,
+ "278": 65.0,
+ "279": 74.0,
+ "280": 64.0,
+ "281": 43.0,
+ "282": 73.0,
+ "283": 91.0,
+ "284": 71.0,
+ "285": 82.0,
+ "286": 64.0,
+ "287": 86.0,
+ "288": 65.0,
+ "289": 97.0,
+ "290": 68.0,
+ "291": 86.0,
+ "292": 84.0,
+ "293": 74.0,
+ "294": 86.0,
+ "295": 91.0,
+ "296": 97.0,
+ "297": 83.0,
+ "298": 57.0,
+ "299": 84.0,
+ "300": 77.0,
+ "301": 73.0,
+ "302": 97.0,
+ "303": 80.0,
+ "304": 78.0,
+ "305": 57.0,
+ "306": 88.0,
+ "307": 79.0,
+ "308": 101.0,
+ "309": 78.0,
+ "310": 78.0,
+ "311": 67.0,
+ "312": 58.0,
+ "313": 74.0,
+ "314": 80.0,
+ "315": 96.0,
+ "316": 85.0,
+ "317": 78.0,
+ "318": 104.0,
+ "319": 81.0,
+ "320": 77.0,
+ "321": 73.0,
+ "322": 82.0,
+ "323": 97.0,
+ "324": 86.0,
+ "325": 81.0,
+ "326": 99.0,
+ "327": 96.0,
+ "328": 93.0,
+ "329": 92.0,
+ "330": 96.0,
+ "331": 89.0,
+ "332": 77.0,
+ "333": 98.0,
+ "334": 81.0,
+ "335": 86.0,
+ "336": 72.0,
+ "337": 92.0,
+ "338": 87.0,
+ "339": 97.0,
+ "340": 78.0,
+ "341": 81.0,
+ "342": 58.0,
+ "343": 91.0,
+ "344": 100.0,
+ "345": 70.0,
+ "346": 82.0,
+ "347": 89.0,
+ "348": 99.0,
+ "349": 82.0,
+ "350": 82.0,
+ "351": 99.0,
+ "352": 105.0,
+ "353": 113.0,
+ "354": 88.0,
+ "355": 97.0,
+ "356": 88.0,
+ "357": 96.0,
+ "358": 81.0,
+ "359": 80.0,
+ "360": 100.0,
+ "361": 68.0,
+ "362": 119.0,
+ "363": 123.0,
+ "364": 110.0,
+ "365": 88.0,
+ "366": 109.0,
+ "367": 83.0,
+ "368": 90.0,
+ "369": 71.0,
+ "370": 99.0,
+ "371": 94.0,
+ "372": 99.0,
+ "373": 99.0,
+ "374": 68.0,
+ "375": 104.0,
+ "376": 88.0,
+ "377": 92.0,
+ "378": 103.0,
+ "379": 112.0,
+ "380": 92.0,
+ "381": 88.0,
+ "382": 82.0,
+ "383": 75.0,
+ "384": 106.0,
+ "385": 78.0,
+ "386": 82.0,
+ "387": 57.0,
+ "388": 80.0,
+ "389": 101.0,
+ "390": 97.0,
+ "391": 126.0,
+ "392": 77.0,
+ "393": 97.0,
+ "394": 94.0,
+ "395": 95.0,
+ "396": 85.0,
+ "397": 96.0,
+ "398": 87.0,
+ "399": 95.0,
+ "400": 95.0,
+ "401": 103.0,
+ "402": 97.0,
+ "403": 103.0,
+ "404": 106.0,
+ "405": 89.0,
+ "406": 101.0,
+ "407": 81.0,
+ "408": 99.0,
+ "409": 100.0,
+ "410": 101.0,
+ "411": 95.0,
+ "412": 76.0,
+ "413": 64.0,
+ "414": 75.0,
+ "415": 118.0,
+ "416": 88.0,
+ "417": 86.0,
+ "418": 79.0,
+ "419": 89.0,
+ "420": 93.0,
+ "421": 95.0,
+ "422": 74.0,
+ "423": 58.0,
+ "424": 111.0,
+ "425": 115.0,
+ "426": 85.0,
+ "427": 100.0,
+ "428": 93.0,
+ "429": 79.0,
+ "430": 97.0,
+ "431": 74.0,
+ "432": 90.0,
+ "433": 73.0,
+ "434": 74.0,
+ "435": 79.0,
+ "436": 85.0,
+ "437": 73.0,
+ "438": 68.0,
+ "439": 84.0,
+ "440": 95.0,
+ "441": 102.0,
+ "442": 102.0,
+ "443": 100.0,
+ "444": 67.0,
+ "445": 98.0,
+ "446": 95.0,
+ "447": 99.0,
+ "448": 88.0,
+ "449": 113.0,
+ "450": 66.0,
+ "451": 89.0,
+ "452": 86.0,
+ "453": 77.0,
+ "454": 99.0,
+ "455": 109.0,
+ "456": 82.0,
+ "457": 106.0,
+ "458": 76.0,
+ "459": 86.0,
+ "460": 88.0,
+ "461": 104.0,
+ "462": 69.0,
+ "463": 78.0,
+ "464": 78.0,
+ "465": 75.0,
+ "466": 98.0,
+ "467": 74.0,
+ "468": 75.0,
+ "469": 75.0,
+ "470": 77.0,
+ "471": 109.0,
+ "472": 68.0,
+ "473": 86.0,
+ "474": 82.0,
+ "475": 96.0,
+ "476": 61.0,
+ "477": 71.0,
+ "478": 85.0,
+ "479": 81.0,
+ "480": 53.0,
+ "481": 80.0,
+ "482": 83.0,
+ "483": 61.0,
+ "484": 75.0,
+ "485": 79.0,
+ "486": 83.0,
+ "487": 80.0,
+ "488": 99.0,
+ "489": 60.0,
+ "490": 60.0,
+ "491": 69.0,
+ "492": 62.0,
+ "493": 90.0,
+ "494": 52.0,
+ "495": 64.0,
+ "496": 64.0,
+ "497": 68.0,
+ "498": 67.0,
+ "499": 64.0,
+ "500": 76.0,
+ "501": 81.0,
+ "502": 69.0,
+ "503": 79.0,
+ "504": 62.0,
+ "505": 69.0,
+ "506": 56.0,
+ "507": 74.0,
+ "508": 63.0,
+ "509": 63.0,
+ "510": 68.0,
+ "511": 62.0,
+ "512": 57.0,
+ "513": 55.0,
+ "514": 62.0,
+ "515": 59.0,
+ "516": 59.0,
+ "517": 60.0,
+ "518": 62.0,
+ "519": 73.0,
+ "520": 66.0,
+ "521": 60.0,
+ "522": 74.0,
+ "523": 67.0,
+ "524": 65.0,
+ "525": 68.0,
+ "526": 46.0,
+ "527": 76.0,
+ "528": 73.0,
+ "529": 66.0,
+ "530": 67.0,
+ "531": 65.0,
+ "532": 77.0,
+ "533": 85.0,
+ "534": 75.0,
+ "535": 68.0,
+ "536": 79.0,
+ "537": 52.0,
+ "538": 53.0,
+ "539": 47.0,
+ "540": 74.0,
+ "541": 45.0,
+ "542": 84.0,
+ "543": 75.0,
+ "544": 59.0,
+ "545": 57.0,
+ "546": 92.0,
+ "547": 74.0,
+ "548": 88.0,
+ "549": 70.0,
+ "550": 55.0,
+ "551": 69.0,
+ "552": 84.0,
+ "553": 58.0,
+ "554": 70.0,
+ "555": 51.0,
+ "556": 58.0,
+ "557": 55.0,
+ "558": 57.0,
+ "559": 48.0,
+ "560": 71.0,
+ "561": 66.0,
+ "562": 60.0,
+ "563": 84.0,
+ "564": 51.0,
+ "565": 62.0,
+ "566": 60.0,
+ "567": 80.0,
+ "568": 66.0,
+ "569": 66.0,
+ "570": 47.0,
+ "571": 52.0,
+ "572": 52.0,
+ "573": 68.0,
+ "574": 74.0,
+ "575": 59.0,
+ "576": 61.0,
+ "577": 54.0,
+ "578": 64.0,
+ "579": 54.0,
+ "580": 49.0,
+ "581": 59.0,
+ "582": 60.0,
+ "583": 62.0,
+ "584": 67.0,
+ "585": 51.0,
+ "586": 39.0,
+ "587": 58.0,
+ "588": 48.0,
+ "589": 43.0,
+ "590": 58.0,
+ "591": 60.0,
+ "592": 44.0,
+ "593": 66.0,
+ "594": 63.0,
+ "595": 45.0,
+ "596": 49.0,
+ "597": 39.0,
+ "598": 54.0,
+ "599": 57.0,
+ "600": 61.0,
+ "601": 53.0,
+ "602": 50.0,
+ "603": 64.0,
+ "604": 68.0,
+ "605": 55.0,
+ "606": 56.0,
+ "607": 57.0,
+ "608": 57.0,
+ "609": 49.0,
+ "610": 53.0,
+ "611": 44.0,
+ "612": 73.0,
+ "613": 62.0,
+ "614": 47.0,
+ "615": 54.0,
+ "616": 64.0,
+ "617": 52.0,
+ "618": 71.0,
+ "619": 50.0,
+ "620": 55.0,
+ "621": 58.0,
+ "622": 47.0,
+ "623": 47.0,
+ "624": 42.0,
+ "625": 46.0,
+ "626": 65.0,
+ "627": 54.0,
+ "628": 50.0,
+ "629": 70.0,
+ "630": 56.0,
+ "631": 51.0,
+ "632": 47.0,
+ "633": 43.0,
+ "634": 43.0,
+ "635": 55.0,
+ "636": 33.0,
+ "637": 43.0,
+ "638": 42.0,
+ "639": 46.0,
+ "640": 45.0,
+ "641": 44.0,
+ "642": 45.0,
+ "643": 63.0,
+ "644": 35.0,
+ "645": 59.0,
+ "646": 45.0,
+ "647": 41.0,
+ "648": 56.0,
+ "649": 63.0,
+ "650": 61.0,
+ "651": 55.0,
+ "652": 35.0,
+ "653": 51.0,
+ "654": 38.0,
+ "655": 50.0,
+ "656": 49.0,
+ "657": 39.0,
+ "658": 50.0,
+ "659": 40.0,
+ "660": 48.0,
+ "661": 52.0,
+ "662": 53.0,
+ "663": 56.0,
+ "664": 54.0,
+ "665": 54.0,
+ "666": 42.0,
+ "667": 54.0,
+ "668": 37.0,
+ "669": 61.0,
+ "670": 44.0,
+ "671": 48.0,
+ "672": 76.0,
+ "673": 47.0,
+ "674": 51.0,
+ "675": 59.0,
+ "676": 56.0,
+ "677": 51.0,
+ "678": 41.0,
+ "679": 37.0,
+ "680": 50.0,
+ "681": 43.0,
+ "682": 42.0,
+ "683": 47.0,
+ "684": 37.0,
+ "685": 55.0,
+ "686": 58.0,
+ "687": 45.0,
+ "688": 43.0,
+ "689": 46.0,
+ "690": 50.0,
+ "691": 44.0,
+ "692": 51.0,
+ "693": 43.0,
+ "694": 59.0,
+ "695": 45.0,
+ "696": 48.0,
+ "697": 47.0,
+ "698": 42.0,
+ "699": 50.0,
+ "700": 58.0,
+ "701": 50.0,
+ "702": 39.0,
+ "703": 28.0,
+ "704": 39.0,
+ "705": 41.0,
+ "706": 50.0,
+ "707": 32.0,
+ "708": 31.0,
+ "709": 35.0,
+ "710": 55.0,
+ "711": 40.0,
+ "712": 36.0,
+ "713": 36.0,
+ "714": 46.0,
+ "715": 32.0,
+ "716": 38.0,
+ "717": 39.0,
+ "718": 37.0,
+ "719": 44.0,
+ "720": 29.0,
+ "721": 35.0,
+ "722": 38.0,
+ "723": 41.0,
+ "724": 45.0,
+ "725": 41.0,
+ "726": 44.0,
+ "727": 39.0,
+ "728": 32.0,
+ "729": 30.0,
+ "730": 41.0,
+ "731": 40.0,
+ "732": 48.0,
+ "733": 48.0,
+ "734": 49.0,
+ "735": 64.0,
+ "736": 42.0,
+ "737": 44.0,
+ "738": 58.0,
+ "739": 57.0,
+ "740": 47.0,
+ "741": 72.0,
+ "742": 50.0,
+ "743": 32.0,
+ "744": 48.0,
+ "745": 46.0,
+ "746": 38.0,
+ "747": 51.0,
+ "748": 37.0,
+ "749": 43.0,
+ "750": 42.0,
+ "751": 49.0,
+ "752": 47.0,
+ "753": 41.0,
+ "754": 54.0,
+ "755": 51.0,
+ "756": 44.0,
+ "757": 32.0,
+ "758": 38.0,
+ "759": 43.0,
+ "760": 52.0,
+ "761": 41.0,
+ "762": 55.0,
+ "763": 42.0,
+ "764": 39.0,
+ "765": 29.0,
+ "766": 36.0,
+ "767": 41.0,
+ "768": 33.0,
+ "769": 47.0,
+ "770": 38.0,
+ "771": 26.0,
+ "772": 45.0,
+ "773": 38.0,
+ "774": 43.0,
+ "775": 34.0,
+ "776": 41.0,
+ "777": 48.0,
+ "778": 37.0,
+ "779": 35.0,
+ "780": 42.0,
+ "781": 50.0,
+ "782": 30.0,
+ "783": 47.0,
+ "784": 45.0,
+ "785": 31.0,
+ "786": 38.0,
+ "787": 48.0,
+ "788": 37.0,
+ "789": 40.0,
+ "790": 44.0,
+ "791": 33.0,
+ "792": 36.0,
+ "793": 39.0,
+ "794": 32.0,
+ "795": 31.0,
+ "796": 51.0,
+ "797": 39.0,
+ "798": 40.0,
+ "799": 22.0,
+ "800": 38.0,
+ "801": 36.0,
+ "802": 26.0,
+ "803": 48.0,
+ "804": 41.0,
+ "805": 44.0,
+ "806": 40.0,
+ "807": 36.0,
+ "808": 31.0,
+ "809": 35.0,
+ "810": 37.0,
+ "811": 44.0,
+ "812": 30.0,
+ "813": 28.0,
+ "814": 45.0,
+ "815": 52.0,
+ "816": 40.0,
+ "817": 37.0,
+ "818": 36.0,
+ "819": 34.0,
+ "820": 45.0,
+ "821": 31.0,
+ "822": 53.0,
+ "823": 36.0,
+ "824": 31.0,
+ "825": 46.0,
+ "826": 46.0,
+ "827": 53.0,
+ "828": 36.0,
+ "829": 36.0,
+ "830": 41.0,
+ "831": 41.0,
+ "832": 42.0,
+ "833": 44.0,
+ "834": 29.0,
+ "835": 33.0,
+ "836": 41.0,
+ "837": 33.0,
+ "838": 58.0,
+ "839": 41.0,
+ "840": 38.0,
+ "841": 36.0,
+ "842": 38.0,
+ "843": 35.0,
+ "844": 31.0,
+ "845": 35.0,
+ "846": 44.0,
+ "847": 24.0,
+ "848": 31.0,
+ "849": 44.0,
+ "850": 28.0,
+ "851": 49.0,
+ "852": 24.0,
+ "853": 40.0,
+ "854": 38.0,
+ "855": 45.0,
+ "856": 38.0,
+ "857": 38.0,
+ "858": 39.0,
+ "859": 42.0,
+ "860": 38.0,
+ "861": 28.0,
+ "862": 39.0,
+ "863": 39.0,
+ "864": 33.0,
+ "865": 36.0,
+ "866": 31.0,
+ "867": 29.0,
+ "868": 45.0,
+ "869": 33.0,
+ "870": 40.0,
+ "871": 41.0,
+ "872": 40.0,
+ "873": 39.0,
+ "874": 48.0,
+ "875": 46.0,
+ "876": 43.0,
+ "877": 44.0,
+ "878": 38.0,
+ "879": 41.0,
+ "880": 34.0,
+ "881": 40.0,
+ "882": 39.0,
+ "883": 39.0,
+ "884": 36.0,
+ "885": 27.0,
+ "886": 31.0,
+ "887": 38.0,
+ "888": 35.0,
+ "889": 40.0,
+ "890": 37.0,
+ "891": 32.0,
+ "892": 39.0,
+ "893": 39.0,
+ "894": 45.0,
+ "895": 32.0,
+ "896": 38.0,
+ "897": 39.0,
+ "898": 40.0,
+ "899": 40.0,
+ "900": 44.0,
+ "901": 40.0,
+ "902": 35.0,
+ "903": 45.0,
+ "904": 42.0,
+ "905": 44.0,
+ "906": 52.0,
+ "907": 31.0,
+ "908": 56.0,
+ "909": 45.0,
+ "910": 29.0,
+ "911": 46.0,
+ "912": 39.0,
+ "913": 43.0,
+ "914": 43.0,
+ "915": 35.0,
+ "916": 44.0,
+ "917": 56.0,
+ "918": 41.0,
+ "919": 32.0,
+ "920": 40.0,
+ "921": 38.0,
+ "922": 39.0,
+ "923": 45.0,
+ "924": 31.0,
+ "925": 36.0,
+ "926": 39.0,
+ "927": 43.0,
+ "928": 34.0,
+ "929": 42.0,
+ "930": 44.0,
+ "931": 26.0,
+ "932": 40.0,
+ "933": 26.0,
+ "934": 33.0,
+ "935": 38.0,
+ "936": 35.0,
+ "937": 39.0,
+ "938": 41.0,
+ "939": 28.0,
+ "940": 46.0,
+ "941": 38.0,
+ "942": 39.0,
+ "943": 41.0,
+ "944": 33.0,
+ "945": 36.0,
+ "946": 38.0,
+ "947": 45.0,
+ "948": 41.0,
+ "949": 35.0,
+ "950": 39.0,
+ "951": 58.0,
+ "952": 36.0,
+ "953": 35.0,
+ "954": 44.0,
+ "955": 31.0,
+ "956": 36.0,
+ "957": 39.0,
+ "958": 27.0,
+ "959": 32.0,
+ "960": 34.0,
+ "961": 39.0,
+ "962": 35.0,
+ "963": 35.0,
+ "964": 35.0,
+ "965": 41.0,
+ "966": 36.0,
+ "967": 29.0,
+ "968": 35.0,
+ "969": 33.0,
+ "970": 32.0,
+ "971": 44.0,
+ "972": 35.0,
+ "973": 37.0,
+ "974": 42.0,
+ "975": 41.0,
+ "976": 39.0,
+ "977": 30.0,
+ "978": 35.0,
+ "979": 35.0,
+ "980": 31.0,
+ "981": 42.0,
+ "982": 31.0,
+ "983": 33.0,
+ "984": 30.0,
+ "985": 29.0,
+ "986": 31.0,
+ "987": 31.0,
+ "988": 31.0,
+ "989": 38.0,
+ "990": 40.0,
+ "991": 36.0,
+ "992": 36.0,
+ "993": 50.0,
+ "994": 34.0,
+ "995": 44.0,
+ "996": 38.0,
+ "997": 37.0,
+ "998": 43.0,
+ "999": 37.0,
+ "1000": 32.0,
+ "1001": 34.0,
+ "1002": 49.0,
+ "1003": 35.0,
+ "1004": 29.0,
+ "1005": 42.0,
+ "1006": 29.0,
+ "1007": 44.0,
+ "1008": 33.0,
+ "1009": 35.0,
+ "1010": 36.0,
+ "1011": 37.0,
+ "1012": 40.0,
+ "1013": 39.0,
+ "1014": 26.0,
+ "1015": 35.0,
+ "1016": 31.0,
+ "1017": 29.0,
+ "1018": 35.0,
+ "1019": 49.0,
+ "1020": 42.0,
+ "1021": 47.0,
+ "1022": 33.0,
+ "1023": 30.0,
+ "1024": 34.0,
+ "1025": 29.0,
+ "1026": 37.0,
+ "1027": 36.0,
+ "1028": 26.0,
+ "1029": 35.0,
+ "1030": 31.0,
+ "1031": 29.0,
+ "1032": 44.0,
+ "1033": 35.0,
+ "1034": 47.0,
+ "1035": 31.0,
+ "1036": 31.0,
+ "1037": 46.0,
+ "1038": 34.0,
+ "1039": 35.0,
+ "1040": 36.0,
+ "1041": 46.0,
+ "1042": 39.0,
+ "1043": 42.0,
+ "1044": 34.0,
+ "1045": 35.0,
+ "1046": 37.0,
+ "1047": 41.0,
+ "1048": 42.0,
+ "1049": 32.0,
+ "1050": 39.0,
+ "1051": 31.0,
+ "1052": 40.0,
+ "1053": 32.0,
+ "1054": 38.0,
+ "1055": 41.0,
+ "1056": 34.0,
+ "1057": 36.0,
+ "1058": 30.0,
+ "1059": 31.0,
+ "1060": 49.0,
+ "1061": 30.0,
+ "1062": 44.0,
+ "1063": 42.0,
+ "1064": 33.0,
+ "1065": 32.0,
+ "1066": 42.0,
+ "1067": 37.0,
+ "1068": 32.0,
+ "1069": 38.0,
+ "1070": 33.0,
+ "1071": 25.0,
+ "1072": 33.0,
+ "1073": 44.0,
+ "1074": 30.0,
+ "1075": 45.0,
+ "1076": 45.0,
+ "1077": 35.0,
+ "1078": 31.0,
+ "1079": 27.0,
+ "1080": 42.0,
+ "1081": 49.0,
+ "1082": 31.0,
+ "1083": 38.0,
+ "1084": 42.0,
+ "1085": 31.0,
+ "1086": 40.0,
+ "1087": 31.0,
+ "1088": 41.0,
+ "1089": 32.0,
+ "1090": 42.0,
+ "1091": 25.0,
+ "1092": 45.0,
+ "1093": 35.0,
+ "1094": 41.0,
+ "1095": 37.0,
+ "1096": 42.0,
+ "1097": 40.0,
+ "1098": 35.0,
+ "1099": 43.0,
+ "1100": 44.0,
+ "1101": 38.0,
+ "1102": 44.0,
+ "1103": 41.0,
+ "1104": 30.0,
+ "1105": 51.0,
+ "1106": 28.0,
+ "1107": 41.0,
+ "1108": 39.0,
+ "1109": 37.0,
+ "1110": 38.0,
+ "1111": 39.0,
+ "1112": 53.0,
+ "1113": 43.0,
+ "1114": 26.0,
+ "1115": 42.0,
+ "1116": 33.0,
+ "1117": 35.0,
+ "1118": 34.0,
+ "1119": 32.0,
+ "1120": 30.0,
+ "1121": 37.0,
+ "1122": 43.0,
+ "1123": 36.0,
+ "1124": 34.0,
+ "1125": 46.0,
+ "1126": 41.0,
+ "1127": 34.0,
+ "1128": 39.0,
+ "1129": 29.0,
+ "1130": 41.0,
+ "1131": 41.0,
+ "1132": 36.0,
+ "1133": 49.0,
+ "1134": 32.0,
+ "1135": 44.0,
+ "1136": 33.0,
+ "1137": 31.0,
+ "1138": 31.0,
+ "1139": 35.0,
+ "1140": 28.0,
+ "1141": 43.0,
+ "1142": 50.0,
+ "1143": 32.0,
+ "1144": 40.0,
+ "1145": 33.0,
+ "1146": 36.0,
+ "1147": 31.0,
+ "1148": 39.0,
+ "1149": 42.0,
+ "1150": 32.0,
+ "1151": 49.0,
+ "1152": 41.0,
+ "1153": 35.0,
+ "1154": 34.0,
+ "1155": 34.0,
+ "1156": 32.0,
+ "1157": 43.0,
+ "1158": 35.0,
+ "1159": 36.0,
+ "1160": 35.0,
+ "1161": 27.0,
+ "1162": 30.0,
+ "1163": 43.0,
+ "1164": 53.0,
+ "1165": 37.0,
+ "1166": 36.0,
+ "1167": 30.0,
+ "1168": 36.0,
+ "1169": 32.0,
+ "1170": 30.0,
+ "1171": 40.0,
+ "1172": 41.0,
+ "1173": 40.0,
+ "1174": 32.0,
+ "1175": 32.0,
+ "1176": 47.0,
+ "1177": 48.0,
+ "1178": 32.0,
+ "1179": 42.0,
+ "1180": 38.0,
+ "1181": 33.0,
+ "1182": 54.0,
+ "1183": 36.0,
+ "1184": 49.0,
+ "1185": 38.0,
+ "1186": 39.0,
+ "1187": 34.0,
+ "1188": 39.0,
+ "1189": 38.0,
+ "1190": 47.0,
+ "1191": 29.0,
+ "1192": 39.0,
+ "1193": 40.0,
+ "1194": 30.0,
+ "1195": 39.0,
+ "1196": 38.0,
+ "1197": 41.0,
+ "1198": 38.0,
+ "1199": 35.0,
+ "1200": 36.0,
+ "1201": 39.0,
+ "1202": 42.0,
+ "1203": 40.0,
+ "1204": 25.0,
+ "1205": 38.0,
+ "1206": 33.0,
+ "1207": 34.0,
+ "1208": 33.0,
+ "1209": 26.0,
+ "1210": 45.0,
+ "1211": 37.0,
+ "1212": 32.0,
+ "1213": 20.0,
+ "1214": 33.0,
+ "1215": 49.0,
+ "1216": 40.0,
+ "1217": 27.0,
+ "1218": 30.0,
+ "1219": 37.0,
+ "1220": 42.0,
+ "1221": 41.0,
+ "1222": 39.0,
+ "1223": 31.0,
+ "1224": 35.0,
+ "1225": 33.0,
+ "1226": 21.0,
+ "1227": 50.0,
+ "1228": 21.0,
+ "1229": 29.0,
+ "1230": 21.0,
+ "1231": 31.0,
+ "1232": 42.0,
+ "1233": 26.0,
+ "1234": 32.0,
+ "1235": 23.0,
+ "1236": 31.0,
+ "1237": 37.0,
+ "1238": 34.0,
+ "1239": 31.0,
+ "1240": 37.0,
+ "1241": 37.0,
+ "1242": 38.0,
+ "1243": 34.0,
+ "1244": 32.0,
+ "1245": 43.0,
+ "1246": 38.0,
+ "1247": 42.0,
+ "1248": 42.0,
+ "1249": 37.0,
+ "1250": 36.0,
+ "1251": 39.0,
+ "1252": 36.0,
+ "1253": 34.0,
+ "1254": 42.0,
+ "1255": 36.0,
+ "1256": 30.0,
+ "1257": 29.0,
+ "1258": 44.0,
+ "1259": 43.0,
+ "1260": 39.0,
+ "1261": 41.0,
+ "1262": 35.0,
+ "1263": 36.0,
+ "1264": 40.0,
+ "1265": 39.0,
+ "1266": 43.0,
+ "1267": 38.0,
+ "1268": 32.0,
+ "1269": 44.0,
+ "1270": 51.0,
+ "1271": 31.0,
+ "1272": 40.0,
+ "1273": 42.0,
+ "1274": 28.0,
+ "1275": 48.0,
+ "1276": 45.0,
+ "1277": 41.0,
+ "1278": 34.0,
+ "1279": 36.0,
+ "1280": 36.0,
+ "1281": 21.0,
+ "1282": 29.0,
+ "1283": 35.0,
+ "1284": 41.0,
+ "1285": 47.0,
+ "1286": 35.0,
+ "1287": 43.0,
+ "1288": 42.0,
+ "1289": 27.0,
+ "1290": 32.0,
+ "1291": 34.0,
+ "1292": 43.0,
+ "1293": 37.0,
+ "1294": 30.0,
+ "1295": 35.0,
+ "1296": 31.0,
+ "1297": 35.0,
+ "1298": 41.0,
+ "1299": 46.0,
+ "1300": 33.0,
+ "1301": 30.0,
+ "1302": 30.0,
+ "1303": 33.0,
+ "1304": 44.0,
+ "1305": 46.0,
+ "1306": 28.0,
+ "1307": 45.0,
+ "1308": 25.0,
+ "1309": 38.0,
+ "1310": 39.0,
+ "1311": 31.0,
+ "1312": 28.0,
+ "1313": 39.0,
+ "1314": 26.0,
+ "1315": 38.0,
+ "1316": 35.0,
+ "1317": 43.0,
+ "1318": 29.0,
+ "1319": 43.0,
+ "1320": 33.0,
+ "1321": 41.0,
+ "1322": 31.0,
+ "1323": 33.0,
+ "1324": 33.0,
+ "1325": 36.0,
+ "1326": 37.0,
+ "1327": 39.0,
+ "1328": 39.0,
+ "1329": 24.0,
+ "1330": 36.0,
+ "1331": 34.0,
+ "1332": 51.0,
+ "1333": 36.0,
+ "1334": 33.0,
+ "1335": 36.0,
+ "1336": 37.0,
+ "1337": 39.0,
+ "1338": 25.0,
+ "1339": 26.0,
+ "1340": 34.0,
+ "1341": 32.0,
+ "1342": 37.0,
+ "1343": 35.0,
+ "1344": 33.0,
+ "1345": 47.0,
+ "1346": 39.0,
+ "1347": 45.0,
+ "1348": 41.0,
+ "1349": 33.0,
+ "1350": 42.0,
+ "1351": 47.0,
+ "1352": 61.0,
+ "1353": 40.0,
+ "1354": 35.0,
+ "1355": 32.0,
+ "1356": 40.0,
+ "1357": 40.0,
+ "1358": 36.0,
+ "1359": 40.0,
+ "1360": 36.0,
+ "1361": 37.0,
+ "1362": 45.0,
+ "1363": 36.0,
+ "1364": 35.0,
+ "1365": 34.0,
+ "1366": 43.0,
+ "1367": 33.0,
+ "1368": 43.0,
+ "1369": 35.0,
+ "1370": 41.0,
+ "1371": 36.0,
+ "1372": 35.0,
+ "1373": 41.0,
+ "1374": 37.0,
+ "1375": 55.0,
+ "1376": 33.0,
+ "1377": 43.0,
+ "1378": 34.0,
+ "1379": 31.0,
+ "1380": 48.0,
+ "1381": 43.0,
+ "1382": 35.0,
+ "1383": 35.0,
+ "1384": 38.0,
+ "1385": 50.0,
+ "1386": 25.0,
+ "1387": 41.0,
+ "1388": 44.0,
+ "1389": 32.0,
+ "1390": 39.0,
+ "1391": 26.0,
+ "1392": 36.0,
+ "1393": 44.0,
+ "1394": 40.0,
+ "1395": 23.0,
+ "1396": 31.0,
+ "1397": 29.0,
+ "1398": 31.0,
+ "1399": 44.0,
+ "1400": 28.0,
+ "1401": 36.0,
+ "1402": 35.0,
+ "1403": 30.0,
+ "1404": 30.0,
+ "1405": 28.0,
+ "1406": 45.0,
+ "1407": 40.0,
+ "1408": 31.0,
+ "1409": 26.0,
+ "1410": 36.0,
+ "1411": 45.0,
+ "1412": 27.0,
+ "1413": 37.0,
+ "1414": 32.0,
+ "1415": 29.0,
+ "1416": 35.0,
+ "1417": 46.0,
+ "1418": 34.0,
+ "1419": 33.0,
+ "1420": 35.0,
+ "1421": 29.0,
+ "1422": 34.0,
+ "1423": 27.0,
+ "1424": 45.0,
+ "1425": 34.0,
+ "1426": 34.0,
+ "1427": 39.0,
+ "1428": 35.0,
+ "1429": 32.0,
+ "1430": 38.0,
+ "1431": 36.0,
+ "1432": 42.0,
+ "1433": 38.0,
+ "1434": 41.0,
+ "1435": 35.0,
+ "1436": 39.0,
+ "1437": 37.0,
+ "1438": 32.0,
+ "1439": 31.0,
+ "1440": 41.0,
+ "1441": 44.0,
+ "1442": 42.0,
+ "1443": 30.0,
+ "1444": 34.0,
+ "1445": 42.0,
+ "1446": 36.0,
+ "1447": 33.0,
+ "1448": 45.0,
+ "1449": 41.0,
+ "1450": 27.0,
+ "1451": 42.0,
+ "1452": 24.0,
+ "1453": 37.0,
+ "1454": 35.0,
+ "1455": 34.0,
+ "1456": 40.0,
+ "1457": 44.0,
+ "1458": 43.0,
+ "1459": 46.0,
+ "1460": 25.0,
+ "1461": 53.0,
+ "1462": 23.0,
+ "1463": 31.0,
+ "1464": 25.0,
+ "1465": 34.0,
+ "1466": 36.0,
+ "1467": 24.0,
+ "1468": 33.0,
+ "1469": 31.0,
+ "1470": 37.0,
+ "1471": 42.0,
+ "1472": 30.0,
+ "1473": 32.0,
+ "1474": 26.0,
+ "1475": 43.0,
+ "1476": 34.0,
+ "1477": 52.0,
+ "1478": 39.0,
+ "1479": 46.0,
+ "1480": 32.0,
+ "1481": 39.0,
+ "1482": 42.0,
+ "1483": 43.0,
+ "1484": 35.0,
+ "1485": 47.0,
+ "1486": 43.0,
+ "1487": 33.0,
+ "1488": 31.0,
+ "1489": 34.0,
+ "1490": 32.0,
+ "1491": 37.0,
+ "1492": 31.0,
+ "1493": 32.0,
+ "1494": 23.0,
+ "1495": 40.0,
+ "1496": 46.0,
+ "1497": 44.0,
+ "1498": 36.0,
+ "1499": 34.0,
+ "1500": 34.0,
+ "1501": 41.0,
+ "1502": 33.0,
+ "1503": 46.0,
+ "1504": 35.0,
+ "1505": 35.0,
+ "1506": 34.0,
+ "1507": 28.0,
+ "1508": 41.0,
+ "1509": 44.0,
+ "1510": "nan",
+ "1511": "nan",
+ "1512": "nan",
+ "1513": "nan",
+ "1514": "nan",
+ "1515": "nan",
+ "1516": "nan",
+ "1517": "nan",
+ "1518": "nan",
+ "1519": "nan",
+ "1520": "nan",
+ "1521": "nan",
+ "1522": "nan",
+ "1523": "nan",
+ "1524": "nan",
+ "1525": "nan",
+ "1526": "nan",
+ "1527": "nan",
+ "1528": "nan",
+ "1529": "nan",
+ "1530": "nan",
+ "1531": "nan",
+ "1532": "nan",
+ "1533": "nan",
+ "1534": "nan",
+ "1535": "nan",
+ "1536": "nan",
+ "1537": "nan",
+ "1538": "nan",
+ "1539": "nan",
+ "1540": "nan",
+ "1541": "nan",
+ "1542": "nan",
+ "1543": "nan",
+ "1544": "nan",
+ "1545": "nan",
+ "1546": "nan",
+ "1547": "nan",
+ "1548": "nan",
+ "1549": "nan",
+ "1550": "nan",
+ "1551": "nan",
+ "1552": "nan",
+ "1553": "nan",
+ "1554": "nan",
+ "1555": "nan",
+ "1556": "nan",
+ "1557": "nan",
+ "1558": "nan",
+ "1559": "nan",
+ "1560": "nan",
+ "1561": "nan",
+ "1562": "nan",
+ "1563": "nan",
+ "1564": "nan",
+ "1565": "nan",
+ "1566": "nan",
+ "1567": "nan",
+ "1568": "nan",
+ "1569": "nan",
+ "1570": "nan",
+ "1571": "nan",
+ "1572": "nan",
+ "1573": "nan",
+ "1574": "nan",
+ "1575": "nan",
+ "1576": "nan",
+ "1577": "nan",
+ "1578": "nan",
+ "1579": "nan",
+ "1580": "nan",
+ "1581": "nan",
+ "1582": "nan",
+ "1583": "nan",
+ "1584": "nan",
+ "1585": "nan",
+ "1586": "nan",
+ "1587": "nan",
+ "1588": "nan",
+ "1589": "nan",
+ "1590": "nan",
+ "1591": "nan",
+ "1592": "nan",
+ "1593": "nan",
+ "1594": "nan",
+ "1595": "nan",
+ "1596": "nan",
+ "1597": "nan",
+ "1598": "nan",
+ "1599": "nan",
+ "1600": "nan",
+ "1601": "nan",
+ "1602": "nan",
+ "1603": "nan",
+ "1604": "nan",
+ "1605": "nan",
+ "1606": "nan",
+ "1607": "nan",
+ "1608": "nan",
+ "1609": "nan",
+ "1610": "nan",
+ "1611": "nan",
+ "1612": "nan",
+ "1613": "nan",
+ "1614": "nan",
+ "1615": "nan",
+ "1616": "nan",
+ "1617": "nan",
+ "1618": "nan",
+ "1619": "nan",
+ "1620": "nan",
+ "1621": "nan",
+ "1622": "nan",
+ "1623": "nan",
+ "1624": "nan",
+ "1625": "nan",
+ "1626": "nan",
+ "1627": "nan",
+ "1628": "nan",
+ "1629": "nan",
+ "1630": "nan",
+ "1631": "nan",
+ "1632": "nan",
+ "1633": "nan",
+ "1634": "nan",
+ "1635": "nan",
+ "1636": "nan",
+ "1637": "nan",
+ "1638": "nan",
+ "1639": "nan",
+ "1640": "nan",
+ "1641": "nan",
+ "1642": "nan",
+ "1643": "nan",
+ "1644": "nan",
+ "1645": "nan",
+ "1646": "nan",
+ "1647": "nan",
+ "1648": "nan",
+ "1649": "nan",
+ "1650": "nan",
+ "1651": "nan",
+ "1652": "nan",
+ "1653": "nan",
+ "1654": "nan",
+ "1655": "nan",
+ "1656": "nan",
+ "1657": "nan",
+ "1658": "nan",
+ "1659": "nan",
+ "1660": "nan",
+ "1661": "nan",
+ "1662": "nan",
+ "1663": "nan",
+ "1664": "nan",
+ "1665": "nan",
+ "1666": "nan",
+ "1667": "nan",
+ "1668": "nan",
+ "1669": "nan",
+ "1670": "nan",
+ "1671": "nan",
+ "1672": "nan",
+ "1673": "nan",
+ "1674": "nan",
+ "1675": "nan",
+ "1676": "nan",
+ "1677": "nan",
+ "1678": "nan",
+ "1679": "nan",
+ "1680": "nan",
+ "1681": "nan",
+ "1682": "nan",
+ "1683": "nan",
+ "1684": "nan",
+ "1685": "nan",
+ "1686": "nan",
+ "1687": "nan",
+ "1688": "nan",
+ "1689": "nan",
+ "1690": "nan",
+ "1691": "nan",
+ "1692": "nan",
+ "1693": "nan",
+ "1694": "nan",
+ "1695": "nan",
+ "1696": "nan",
+ "1697": "nan",
+ "1698": "nan",
+ "1699": "nan",
+ "1700": "nan",
+ "1701": "nan",
+ "1702": "nan",
+ "1703": "nan",
+ "1704": "nan",
+ "1705": "nan",
+ "1706": "nan",
+ "1707": "nan",
+ "1708": "nan",
+ "1709": "nan",
+ "1710": "nan",
+ "1711": "nan",
+ "1712": "nan",
+ "1713": "nan",
+ "1714": "nan",
+ "1715": "nan",
+ "1716": "nan",
+ "1717": "nan",
+ "1718": "nan",
+ "1719": "nan",
+ "1720": "nan",
+ "1721": "nan",
+ "1722": "nan",
+ "1723": "nan",
+ "1724": "nan",
+ "1725": "nan",
+ "1726": "nan",
+ "1727": "nan",
+ "1728": "nan",
+ "1729": "nan",
+ "1730": "nan",
+ "1731": "nan",
+ "1732": "nan",
+ "1733": "nan",
+ "1734": "nan",
+ "1735": "nan",
+ "1736": "nan",
+ "1737": "nan",
+ "1738": "nan",
+ "1739": "nan",
+ "1740": "nan",
+ "1741": "nan",
+ "1742": "nan",
+ "1743": "nan",
+ "1744": "nan",
+ "1745": "nan",
+ "1746": "nan",
+ "1747": "nan",
+ "1748": "nan",
+ "1749": "nan",
+ "1750": "nan",
+ "1751": "nan",
+ "1752": "nan",
+ "1753": "nan",
+ "1754": "nan",
+ "1755": "nan",
+ "1756": "nan",
+ "1757": "nan",
+ "1758": "nan",
+ "1759": "nan",
+ "1760": "nan",
+ "1761": "nan",
+ "1762": "nan",
+ "1763": "nan",
+ "1764": "nan",
+ "1765": "nan",
+ "1766": "nan",
+ "1767": "nan",
+ "1768": "nan",
+ "1769": "nan",
+ "1770": "nan",
+ "1771": "nan",
+ "1772": "nan",
+ "1773": "nan",
+ "1774": "nan",
+ "1775": "nan",
+ "1776": "nan",
+ "1777": "nan",
+ "1778": "nan",
+ "1779": "nan",
+ "1780": "nan",
+ "1781": "nan",
+ "1782": "nan",
+ "1783": "nan",
+ "1784": "nan",
+ "1785": "nan",
+ "1786": "nan",
+ "1787": "nan",
+ "1788": "nan",
+ "1789": "nan",
+ "1790": "nan",
+ "1791": "nan",
+ "1792": "nan",
+ "1793": "nan",
+ "1794": "nan",
+ "1795": "nan",
+ "1796": "nan",
+ "1797": "nan",
+ "1798": "nan",
+ "1799": "nan",
+ "1800": "nan",
+ "1801": "nan",
+ "1802": "nan",
+ "1803": "nan",
+ "1804": "nan",
+ "1805": "nan",
+ "1806": "nan",
+ "1807": "nan",
+ "1808": "nan",
+ "1809": "nan",
+ "1810": "nan",
+ "1811": "nan",
+ "1812": "nan",
+ "1813": "nan",
+ "1814": "nan",
+ "1815": "nan",
+ "1816": "nan",
+ "1817": "nan",
+ "1818": "nan",
+ "1819": "nan",
+ "1820": "nan",
+ "1821": "nan",
+ "1822": "nan",
+ "1823": "nan",
+ "1824": "nan",
+ "1825": "nan",
+ "1826": "nan",
+ "1827": "nan",
+ "1828": "nan",
+ "1829": "nan",
+ "1830": "nan",
+ "1831": "nan",
+ "1832": "nan",
+ "1833": "nan",
+ "1834": "nan",
+ "1835": "nan",
+ "1836": "nan",
+ "1837": "nan",
+ "1838": "nan",
+ "1839": "nan",
+ "1840": "nan",
+ "1841": "nan",
+ "1842": "nan",
+ "1843": "nan",
+ "1844": "nan",
+ "1845": "nan",
+ "1846": "nan",
+ "1847": "nan",
+ "1848": "nan",
+ "1849": "nan",
+ "1850": "nan",
+ "1851": "nan",
+ "1852": "nan",
+ "1853": "nan",
+ "1854": "nan",
+ "1855": "nan",
+ "1856": "nan",
+ "1857": "nan",
+ "1858": "nan",
+ "1859": "nan",
+ "1860": "nan",
+ "1861": "nan",
+ "1862": "nan",
+ "1863": "nan",
+ "1864": "nan",
+ "1865": "nan",
+ "1866": "nan",
+ "1867": "nan",
+ "1868": "nan",
+ "1869": "nan",
+ "1870": "nan",
+ "1871": "nan",
+ "1872": "nan",
+ "1873": "nan",
+ "1874": "nan",
+ "1875": "nan",
+ "1876": "nan",
+ "1877": "nan",
+ "1878": "nan",
+ "1879": "nan",
+ "1880": "nan",
+ "1881": "nan",
+ "1882": "nan",
+ "1883": "nan",
+ "1884": "nan",
+ "1885": "nan",
+ "1886": "nan",
+ "1887": "nan",
+ "1888": "nan",
+ "1889": "nan",
+ "1890": "nan",
+ "1891": "nan",
+ "1892": "nan",
+ "1893": "nan",
+ "1894": "nan",
+ "1895": "nan",
+ "1896": "nan",
+ "1897": "nan",
+ "1898": "nan",
+ "1899": "nan",
+ "1900": "nan",
+ "1901": "nan",
+ "1902": "nan",
+ "1903": "nan",
+ "1904": "nan",
+ "1905": "nan",
+ "1906": "nan",
+ "1907": "nan",
+ "1908": "nan",
+ "1909": "nan",
+ "1910": "nan",
+ "1911": "nan",
+ "1912": "nan",
+ "1913": "nan",
+ "1914": "nan",
+ "1915": "nan",
+ "1916": "nan",
+ "1917": "nan",
+ "1918": "nan",
+ "1919": "nan",
+ "1920": "nan",
+ "1921": "nan",
+ "1922": "nan",
+ "1923": "nan",
+ "1924": "nan",
+ "1925": "nan",
+ "1926": "nan",
+ "1927": "nan",
+ "1928": "nan",
+ "1929": "nan",
+ "1930": "nan",
+ "1931": "nan",
+ "1932": "nan",
+ "1933": "nan",
+ "1934": "nan",
+ "1935": "nan",
+ "1936": "nan",
+ "1937": "nan",
+ "1938": "nan",
+ "1939": "nan",
+ "1940": "nan",
+ "1941": "nan",
+ "1942": "nan",
+ "1943": "nan",
+ "1944": "nan",
+ "1945": "nan",
+ "1946": "nan",
+ "1947": "nan",
+ "1948": "nan",
+ "1949": "nan",
+ "1950": "nan",
+ "1951": "nan",
+ "1952": "nan",
+ "1953": "nan",
+ "1954": "nan",
+ "1955": "nan",
+ "1956": "nan",
+ "1957": "nan",
+ "1958": "nan",
+ "1959": "nan",
+ "1960": "nan",
+ "1961": "nan",
+ "1962": "nan",
+ "1963": "nan",
+ "1964": "nan",
+ "1965": "nan",
+ "1966": "nan",
+ "1967": "nan",
+ "1968": "nan",
+ "1969": "nan",
+ "1970": "nan",
+ "1971": "nan",
+ "1972": "nan",
+ "1973": "nan",
+ "1974": "nan",
+ "1975": "nan",
+ "1976": "nan",
+ "1977": "nan",
+ "1978": "nan",
+ "1979": "nan",
+ "1980": "nan",
+ "1981": "nan",
+ "1982": "nan",
+ "1983": "nan",
+ "1984": "nan",
+ "1985": "nan",
+ "1986": "nan",
+ "1987": "nan",
+ "1988": "nan",
+ "1989": "nan",
+ "1990": "nan",
+ "1991": "nan",
+ "1992": "nan",
+ "1993": "nan",
+ "1994": "nan",
+ "1995": "nan",
+ "1996": "nan",
+ "1997": "nan",
+ "1998": "nan",
+ "1999": "nan",
+ "2000": "nan"
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 291725824.0,
+ "2": 291725824.0,
+ "3": 291725824.0,
+ "4": 291725824.0,
+ "5": 291725824.0,
+ "6": 291725824.0,
+ "7": 291725824.0,
+ "8": 291725824.0,
+ "9": 291725824.0,
+ "10": 291725824.0,
+ "11": 291725824.0,
+ "12": 291725824.0,
+ "13": 291725824.0,
+ "14": 291725824.0,
+ "15": 291725824.0,
+ "16": 291725824.0,
+ "17": 291725824.0,
+ "18": 291725824.0,
+ "19": 291725824.0,
+ "20": 291725824.0,
+ "21": 291725824.0,
+ "22": 291725824.0,
+ "23": 291725824.0,
+ "24": 291725824.0,
+ "25": 291725824.0,
+ "26": 291725824.0,
+ "27": 291725824.0,
+ "28": 291725824.0,
+ "29": 291725824.0,
+ "30": 291725824.0,
+ "31": 291725824.0,
+ "32": 291725824.0,
+ "33": 291725824.0,
+ "34": 291725824.0,
+ "35": 291725824.0,
+ "36": 291725824.0,
+ "37": 291725824.0,
+ "38": 291725824.0,
+ "39": 291725824.0,
+ "40": 291725824.0,
+ "41": 291725824.0,
+ "42": 291725824.0,
+ "43": 291725824.0,
+ "44": 291725824.0,
+ "45": 291725824.0,
+ "46": 291725824.0,
+ "47": 291725824.0,
+ "48": 291725824.0,
+ "49": 291725824.0,
+ "50": 291725824.0,
+ "51": 291725824.0,
+ "52": 291725824.0,
+ "53": 291725824.0,
+ "54": 291725824.0,
+ "55": 291725824.0,
+ "56": 291725824.0,
+ "57": 291725824.0,
+ "58": 291725824.0,
+ "59": 291725824.0,
+ "60": 291725824.0,
+ "61": 291725824.0,
+ "62": 291725824.0,
+ "63": 291725824.0,
+ "64": 291725824.0,
+ "65": 291725824.0,
+ "66": 291725824.0,
+ "67": 291725824.0,
+ "68": 291725824.0,
+ "69": 291725824.0,
+ "70": 291725824.0,
+ "71": 291725824.0,
+ "72": 291725824.0,
+ "73": 291725824.0,
+ "74": 291725824.0,
+ "75": 291725824.0,
+ "76": 291725824.0,
+ "77": 291725824.0,
+ "78": 291725824.0,
+ "79": 291725824.0,
+ "80": 291725824.0,
+ "81": 291725824.0,
+ "82": 291725824.0,
+ "83": 291725824.0,
+ "84": 291725824.0,
+ "85": 291725824.0,
+ "86": 291725824.0,
+ "87": 291725824.0,
+ "88": 291725824.0,
+ "89": 291725824.0,
+ "90": 291725824.0,
+ "91": 291725824.0,
+ "92": 291725824.0,
+ "93": 291725824.0,
+ "94": 291725824.0,
+ "95": 291725824.0,
+ "96": 291725824.0,
+ "97": 291725824.0,
+ "98": 291725824.0,
+ "99": 291725824.0,
+ "100": 291725824.0,
+ "101": 291725824.0,
+ "102": 291725824.0,
+ "103": 291725824.0,
+ "104": 291725824.0,
+ "105": 291725824.0,
+ "106": 291725824.0,
+ "107": 291725824.0,
+ "108": 291725824.0,
+ "109": 291725824.0,
+ "110": 291725824.0,
+ "111": 291725824.0,
+ "112": 291725824.0,
+ "113": 291725824.0,
+ "114": 291725824.0,
+ "115": 291725824.0,
+ "116": 291725824.0,
+ "117": 291725824.0,
+ "118": 291725824.0,
+ "119": 291725824.0,
+ "120": 291725824.0,
+ "121": 291725824.0,
+ "122": 291725824.0,
+ "123": 291725824.0,
+ "124": 291725824.0,
+ "125": 291725824.0,
+ "126": 291725824.0,
+ "127": 291725824.0,
+ "128": 291725824.0,
+ "129": 291725824.0,
+ "130": 291725824.0,
+ "131": 291725824.0,
+ "132": 291725824.0,
+ "133": 291725824.0,
+ "134": 291725824.0,
+ "135": 291725824.0,
+ "136": 291725824.0,
+ "137": 291725824.0,
+ "138": 291725824.0,
+ "139": 291725824.0,
+ "140": 291725824.0,
+ "141": 291725824.0,
+ "142": 291725824.0,
+ "143": 291725824.0,
+ "144": 291725824.0,
+ "145": 291725824.0,
+ "146": 291725824.0,
+ "147": 291725824.0,
+ "148": 291725824.0,
+ "149": 291725824.0,
+ "150": 291725824.0,
+ "151": 291725824.0,
+ "152": 291725824.0,
+ "153": 291725824.0,
+ "154": 291725824.0,
+ "155": 291725824.0,
+ "156": 291725824.0,
+ "157": 291725824.0,
+ "158": 291725824.0,
+ "159": 291725824.0,
+ "160": 291725824.0,
+ "161": 291725824.0,
+ "162": 291725824.0,
+ "163": 291725824.0,
+ "164": 291725824.0,
+ "165": 291725824.0,
+ "166": 291725824.0,
+ "167": 291725824.0,
+ "168": 291725824.0,
+ "169": 291725824.0,
+ "170": 291725824.0,
+ "171": 291725824.0,
+ "172": 291725824.0,
+ "173": 291725824.0,
+ "174": 291725824.0,
+ "175": 291725824.0,
+ "176": 291725824.0,
+ "177": 291725824.0,
+ "178": 291725824.0,
+ "179": 291725824.0,
+ "180": 291725824.0,
+ "181": 291725824.0,
+ "182": 291725824.0,
+ "183": 291725824.0,
+ "184": 291725824.0,
+ "185": 291725824.0,
+ "186": 291725824.0,
+ "187": 291725824.0,
+ "188": 291725824.0,
+ "189": 291725824.0,
+ "190": 291725824.0,
+ "191": 291725824.0,
+ "192": 291725824.0,
+ "193": 291725824.0,
+ "194": 291725824.0,
+ "195": 291725824.0,
+ "196": 291725824.0,
+ "197": 291725824.0,
+ "198": 291725824.0,
+ "199": 291725824.0,
+ "200": 291725824.0,
+ "201": 291725824.0,
+ "202": 291725824.0,
+ "203": 291725824.0,
+ "204": 291725824.0,
+ "205": 291725824.0,
+ "206": 291725824.0,
+ "207": 291725824.0,
+ "208": 291725824.0,
+ "209": 291725824.0,
+ "210": 291725824.0,
+ "211": 291725824.0,
+ "212": 291725824.0,
+ "213": 291725824.0,
+ "214": 291725824.0,
+ "215": 291725824.0,
+ "216": 291725824.0,
+ "217": 291725824.0,
+ "218": 291725824.0,
+ "219": 291725824.0,
+ "220": 291725824.0,
+ "221": 291725824.0,
+ "222": 291725824.0,
+ "223": 291725824.0,
+ "224": 291725824.0,
+ "225": 291725824.0,
+ "226": 291725824.0,
+ "227": 291725824.0,
+ "228": 291725824.0,
+ "229": 291725824.0,
+ "230": 291725824.0,
+ "231": 291725824.0,
+ "232": 291725824.0,
+ "233": 291725824.0,
+ "234": 291725824.0,
+ "235": 291725824.0,
+ "236": 291725824.0,
+ "237": 291725824.0,
+ "238": 291725824.0,
+ "239": 291725824.0,
+ "240": 291725824.0,
+ "241": 291725824.0,
+ "242": 291725824.0,
+ "243": 291725824.0,
+ "244": 291725824.0,
+ "245": 291725824.0,
+ "246": 291725824.0,
+ "247": 291725824.0,
+ "248": 291725824.0,
+ "249": 291725824.0,
+ "250": 291725824.0,
+ "251": 291725824.0,
+ "252": 291725824.0,
+ "253": 291725824.0,
+ "254": 291725824.0,
+ "255": 291725824.0,
+ "256": 291725824.0,
+ "257": 291725824.0,
+ "258": 291725824.0,
+ "259": 291725824.0,
+ "260": 291725824.0,
+ "261": 291725824.0,
+ "262": 291725824.0,
+ "263": 291725824.0,
+ "264": 291725824.0,
+ "265": 291725824.0,
+ "266": 291725824.0,
+ "267": 291725824.0,
+ "268": 291725824.0,
+ "269": 291725824.0,
+ "270": 291725824.0,
+ "271": 291725824.0,
+ "272": 291725824.0,
+ "273": 291725824.0,
+ "274": 291725824.0,
+ "275": 291725824.0,
+ "276": 291725824.0,
+ "277": 291725824.0,
+ "278": 291725824.0,
+ "279": 291725824.0,
+ "280": 291725824.0,
+ "281": 291725824.0,
+ "282": 291725824.0,
+ "283": 291725824.0,
+ "284": 291725824.0,
+ "285": 291725824.0,
+ "286": 291725824.0,
+ "287": 291725824.0,
+ "288": 291725824.0,
+ "289": 291725824.0,
+ "290": 291725824.0,
+ "291": 291725824.0,
+ "292": 291725824.0,
+ "293": 291725824.0,
+ "294": 291725824.0,
+ "295": 291725824.0,
+ "296": 291725824.0,
+ "297": 291725824.0,
+ "298": 291725824.0,
+ "299": 291725824.0,
+ "300": 291725824.0,
+ "301": 291725824.0,
+ "302": 291725824.0,
+ "303": 291725824.0,
+ "304": 291725824.0,
+ "305": 291725824.0,
+ "306": 291725824.0,
+ "307": 291725824.0,
+ "308": 291725824.0,
+ "309": 291725824.0,
+ "310": 291725824.0,
+ "311": 291725824.0,
+ "312": 291725824.0,
+ "313": 291725824.0,
+ "314": 291725824.0,
+ "315": 291725824.0,
+ "316": 291725824.0,
+ "317": 291725824.0,
+ "318": 291725824.0,
+ "319": 291725824.0,
+ "320": 291725824.0,
+ "321": 291725824.0,
+ "322": 291725824.0,
+ "323": 291725824.0,
+ "324": 291725824.0,
+ "325": 291725824.0,
+ "326": 291725824.0,
+ "327": 291725824.0,
+ "328": 291725824.0,
+ "329": 291725824.0,
+ "330": 291725824.0,
+ "331": 291725824.0,
+ "332": 291725824.0,
+ "333": 291725824.0,
+ "334": 291725824.0,
+ "335": 291725824.0,
+ "336": 291725824.0,
+ "337": 291725824.0,
+ "338": 291725824.0,
+ "339": 291725824.0,
+ "340": 291725824.0,
+ "341": 291725824.0,
+ "342": 291725824.0,
+ "343": 291725824.0,
+ "344": 291725824.0,
+ "345": 291725824.0,
+ "346": 291725824.0,
+ "347": 291725824.0,
+ "348": 291725824.0,
+ "349": 291725824.0,
+ "350": 291725824.0,
+ "351": 291725824.0,
+ "352": 291725824.0,
+ "353": 291725824.0,
+ "354": 291725824.0,
+ "355": 291725824.0,
+ "356": 291725824.0,
+ "357": 291725824.0,
+ "358": 291725824.0,
+ "359": 291725824.0,
+ "360": 291725824.0,
+ "361": 291725824.0,
+ "362": 291725824.0,
+ "363": 291725824.0,
+ "364": 291725824.0,
+ "365": 291725824.0,
+ "366": 291725824.0,
+ "367": 291725824.0,
+ "368": 291725824.0,
+ "369": 291725824.0,
+ "370": 291725824.0,
+ "371": 291725824.0,
+ "372": 291725824.0,
+ "373": 291725824.0,
+ "374": 291725824.0,
+ "375": 291725824.0,
+ "376": 291725824.0,
+ "377": 291725824.0,
+ "378": 291725824.0,
+ "379": 291725824.0,
+ "380": 291725824.0,
+ "381": 291725824.0,
+ "382": 291725824.0,
+ "383": 291725824.0,
+ "384": 291725824.0,
+ "385": 291725824.0,
+ "386": 291725824.0,
+ "387": 291725824.0,
+ "388": 291725824.0,
+ "389": 291725824.0,
+ "390": 291725824.0,
+ "391": 291725824.0,
+ "392": 291725824.0,
+ "393": 291725824.0,
+ "394": 291725824.0,
+ "395": 291725824.0,
+ "396": 291725824.0,
+ "397": 291725824.0,
+ "398": 291725824.0,
+ "399": 291725824.0,
+ "400": 291725824.0,
+ "401": 291725824.0,
+ "402": 291725824.0,
+ "403": 291725824.0,
+ "404": 291725824.0,
+ "405": 291725824.0,
+ "406": 291725824.0,
+ "407": 291725824.0,
+ "408": 291725824.0,
+ "409": 291725824.0,
+ "410": 291725824.0,
+ "411": 291725824.0,
+ "412": 291725824.0,
+ "413": 291725824.0,
+ "414": 291725824.0,
+ "415": 291725824.0,
+ "416": 291725824.0,
+ "417": 291725824.0,
+ "418": 291725824.0,
+ "419": 291725824.0,
+ "420": 291725824.0,
+ "421": 291725824.0,
+ "422": 291725824.0,
+ "423": 291725824.0,
+ "424": 291725824.0,
+ "425": 291725824.0,
+ "426": 291725824.0,
+ "427": 291725824.0,
+ "428": 291725824.0,
+ "429": 291725824.0,
+ "430": 291725824.0,
+ "431": 291725824.0,
+ "432": 291725824.0,
+ "433": 291725824.0,
+ "434": 291725824.0,
+ "435": 291725824.0,
+ "436": 291725824.0,
+ "437": 291725824.0,
+ "438": 291725824.0,
+ "439": 291725824.0,
+ "440": 291725824.0,
+ "441": 291725824.0,
+ "442": 291725824.0,
+ "443": 291725824.0,
+ "444": 291725824.0,
+ "445": 291725824.0,
+ "446": 291725824.0,
+ "447": 291725824.0,
+ "448": 291725824.0,
+ "449": 291725824.0,
+ "450": 291725824.0,
+ "451": 291725824.0,
+ "452": 291725824.0,
+ "453": 291725824.0,
+ "454": 291725824.0,
+ "455": 291725824.0,
+ "456": 291725824.0,
+ "457": 291725824.0,
+ "458": 291725824.0,
+ "459": 291725824.0,
+ "460": 291725824.0,
+ "461": 291725824.0,
+ "462": 291725824.0,
+ "463": 291725824.0,
+ "464": 291725824.0,
+ "465": 291725824.0,
+ "466": 291725824.0,
+ "467": 291725824.0,
+ "468": 291725824.0,
+ "469": 291725824.0,
+ "470": 291725824.0,
+ "471": 291725824.0,
+ "472": 291725824.0,
+ "473": 291725824.0,
+ "474": 291725824.0,
+ "475": 291725824.0,
+ "476": 291725824.0,
+ "477": 291725824.0,
+ "478": 291725824.0,
+ "479": 291725824.0,
+ "480": 291725824.0,
+ "481": 291725824.0,
+ "482": 291725824.0,
+ "483": 291725824.0,
+ "484": 291725824.0,
+ "485": 291725824.0,
+ "486": 291725824.0,
+ "487": 291725824.0,
+ "488": 291725824.0,
+ "489": 291725824.0,
+ "490": 291725824.0,
+ "491": 291725824.0,
+ "492": 291725824.0,
+ "493": 291725824.0,
+ "494": 291725824.0,
+ "495": 291725824.0,
+ "496": 291725824.0,
+ "497": 291725824.0,
+ "498": 291725824.0,
+ "499": 291725824.0,
+ "500": 291725824.0,
+ "501": 291725824.0,
+ "502": 291725824.0,
+ "503": 291725824.0,
+ "504": 291725824.0,
+ "505": 291725824.0,
+ "506": 291725824.0,
+ "507": 291725824.0,
+ "508": 291725824.0,
+ "509": 291725824.0,
+ "510": 291725824.0,
+ "511": 291725824.0,
+ "512": 291725824.0,
+ "513": 291725824.0,
+ "514": 291725824.0,
+ "515": 291725824.0,
+ "516": 291725824.0,
+ "517": 291725824.0,
+ "518": 291725824.0,
+ "519": 291725824.0,
+ "520": 291725824.0,
+ "521": 291725824.0,
+ "522": 291725824.0,
+ "523": 291725824.0,
+ "524": 291725824.0,
+ "525": 291725824.0,
+ "526": 291725824.0,
+ "527": 291725824.0,
+ "528": 291725824.0,
+ "529": 291725824.0,
+ "530": 291725824.0,
+ "531": 291725824.0,
+ "532": 291725824.0,
+ "533": 291725824.0,
+ "534": 291725824.0,
+ "535": 291725824.0,
+ "536": 291725824.0,
+ "537": 291725824.0,
+ "538": 291725824.0,
+ "539": 291725824.0,
+ "540": 291725824.0,
+ "541": 291725824.0,
+ "542": 291725824.0,
+ "543": 291725824.0,
+ "544": 291725824.0,
+ "545": 291725824.0,
+ "546": 291725824.0,
+ "547": 291725824.0,
+ "548": 291725824.0,
+ "549": 291725824.0,
+ "550": 291725824.0,
+ "551": 291725824.0,
+ "552": 291725824.0,
+ "553": 291725824.0,
+ "554": 291725824.0,
+ "555": 291725824.0,
+ "556": 291725824.0,
+ "557": 291725824.0,
+ "558": 291725824.0,
+ "559": 291725824.0,
+ "560": 291725824.0,
+ "561": 291725824.0,
+ "562": 291725824.0,
+ "563": 291725824.0,
+ "564": 291725824.0,
+ "565": 291725824.0,
+ "566": 291725824.0,
+ "567": 291725824.0,
+ "568": 291725824.0,
+ "569": 291725824.0,
+ "570": 291725824.0,
+ "571": 291725824.0,
+ "572": 291725824.0,
+ "573": 291725824.0,
+ "574": 291725824.0,
+ "575": 291725824.0,
+ "576": 291725824.0,
+ "577": 291725824.0,
+ "578": 291725824.0,
+ "579": 291725824.0,
+ "580": 291725824.0,
+ "581": 291725824.0,
+ "582": 291725824.0,
+ "583": 291725824.0,
+ "584": 291725824.0,
+ "585": 291725824.0,
+ "586": 291725824.0,
+ "587": 291725824.0,
+ "588": 291725824.0,
+ "589": 291725824.0,
+ "590": 291725824.0,
+ "591": 291725824.0,
+ "592": 291725824.0,
+ "593": 291725824.0,
+ "594": 291725824.0,
+ "595": 291725824.0,
+ "596": 291725824.0,
+ "597": 291725824.0,
+ "598": 291725824.0,
+ "599": 291725824.0,
+ "600": 291725824.0,
+ "601": 291725824.0,
+ "602": 291725824.0,
+ "603": 291725824.0,
+ "604": 291725824.0,
+ "605": 291725824.0,
+ "606": 291725824.0,
+ "607": 291725824.0,
+ "608": 291725824.0,
+ "609": 291725824.0,
+ "610": 291725824.0,
+ "611": 291725824.0,
+ "612": 291725824.0,
+ "613": 291725824.0,
+ "614": 291725824.0,
+ "615": 291725824.0,
+ "616": 291725824.0,
+ "617": 291725824.0,
+ "618": 291725824.0,
+ "619": 291725824.0,
+ "620": 291725824.0,
+ "621": 291725824.0,
+ "622": 291725824.0,
+ "623": 291725824.0,
+ "624": 291725824.0,
+ "625": 291725824.0,
+ "626": 291725824.0,
+ "627": 291725824.0,
+ "628": 291725824.0,
+ "629": 291725824.0,
+ "630": 291725824.0,
+ "631": 291725824.0,
+ "632": 291725824.0,
+ "633": 291725824.0,
+ "634": 291725824.0,
+ "635": 291725824.0,
+ "636": 291725824.0,
+ "637": 291725824.0,
+ "638": 291725824.0,
+ "639": 291725824.0,
+ "640": 291725824.0,
+ "641": 291725824.0,
+ "642": 291725824.0,
+ "643": 291725824.0,
+ "644": 291725824.0,
+ "645": 291725824.0,
+ "646": 291725824.0,
+ "647": 291725824.0,
+ "648": 291725824.0,
+ "649": 291725824.0,
+ "650": 291725824.0,
+ "651": 291725824.0,
+ "652": 291725824.0,
+ "653": 291725824.0,
+ "654": 291725824.0,
+ "655": 291725824.0,
+ "656": 291725824.0,
+ "657": 291725824.0,
+ "658": 291725824.0,
+ "659": 291725824.0,
+ "660": 291725824.0,
+ "661": 291725824.0,
+ "662": 291725824.0,
+ "663": 291725824.0,
+ "664": 291725824.0,
+ "665": 291725824.0,
+ "666": 291725824.0,
+ "667": 291725824.0,
+ "668": 291725824.0,
+ "669": 291725824.0,
+ "670": 291725824.0,
+ "671": 291725824.0,
+ "672": 291725824.0,
+ "673": 291725824.0,
+ "674": 291725824.0,
+ "675": 291725824.0,
+ "676": 291725824.0,
+ "677": 291725824.0,
+ "678": 291725824.0,
+ "679": 291725824.0,
+ "680": 291725824.0,
+ "681": 291725824.0,
+ "682": 291725824.0,
+ "683": 291725824.0,
+ "684": 291725824.0,
+ "685": 291725824.0,
+ "686": 291725824.0,
+ "687": 291725824.0,
+ "688": 291725824.0,
+ "689": 291725824.0,
+ "690": 291725824.0,
+ "691": 291725824.0,
+ "692": 291725824.0,
+ "693": 291725824.0,
+ "694": 291725824.0,
+ "695": 291725824.0,
+ "696": 291725824.0,
+ "697": 291725824.0,
+ "698": 291725824.0,
+ "699": 291725824.0,
+ "700": 291725824.0,
+ "701": 291725824.0,
+ "702": 291725824.0,
+ "703": 291725824.0,
+ "704": 291725824.0,
+ "705": 291725824.0,
+ "706": 291725824.0,
+ "707": 291725824.0,
+ "708": 291725824.0,
+ "709": 291725824.0,
+ "710": 291725824.0,
+ "711": 291725824.0,
+ "712": 291725824.0,
+ "713": 291725824.0,
+ "714": 291725824.0,
+ "715": 291725824.0,
+ "716": 291725824.0,
+ "717": 291725824.0,
+ "718": 291725824.0,
+ "719": 291725824.0,
+ "720": 291725824.0,
+ "721": 291725824.0,
+ "722": 291725824.0,
+ "723": 291725824.0,
+ "724": 291725824.0,
+ "725": 291725824.0,
+ "726": 291725824.0,
+ "727": 291725824.0,
+ "728": 291725824.0,
+ "729": 291725824.0,
+ "730": 291725824.0,
+ "731": 291725824.0,
+ "732": 291725824.0,
+ "733": 291725824.0,
+ "734": 291725824.0,
+ "735": 291725824.0,
+ "736": 291725824.0,
+ "737": 291725824.0,
+ "738": 291725824.0,
+ "739": 291725824.0,
+ "740": 291725824.0,
+ "741": 291725824.0,
+ "742": 291725824.0,
+ "743": 291725824.0,
+ "744": 291725824.0,
+ "745": 291725824.0,
+ "746": 291725824.0,
+ "747": 291725824.0,
+ "748": 291725824.0,
+ "749": 291725824.0,
+ "750": 291725824.0,
+ "751": 291725824.0,
+ "752": 291725824.0,
+ "753": 291725824.0,
+ "754": 291725824.0,
+ "755": 291725824.0,
+ "756": 291725824.0,
+ "757": 291725824.0,
+ "758": 291725824.0,
+ "759": 291725824.0,
+ "760": 291725824.0,
+ "761": 291725824.0,
+ "762": 291725824.0,
+ "763": 291725824.0,
+ "764": 291725824.0,
+ "765": 291725824.0,
+ "766": 291725824.0,
+ "767": 291725824.0,
+ "768": 291725824.0,
+ "769": 291725824.0,
+ "770": 291725824.0,
+ "771": 291725824.0,
+ "772": 291725824.0,
+ "773": 291725824.0,
+ "774": 291725824.0,
+ "775": 291725824.0,
+ "776": 291725824.0,
+ "777": 291725824.0,
+ "778": 291725824.0,
+ "779": 291725824.0,
+ "780": 291725824.0,
+ "781": 291725824.0,
+ "782": 291725824.0,
+ "783": 291725824.0,
+ "784": 291725824.0,
+ "785": 291725824.0,
+ "786": 291725824.0,
+ "787": 291725824.0,
+ "788": 291725824.0,
+ "789": 291725824.0,
+ "790": 291725824.0,
+ "791": 291725824.0,
+ "792": 291725824.0,
+ "793": 291725824.0,
+ "794": 291725824.0,
+ "795": 291725824.0,
+ "796": 291725824.0,
+ "797": 291725824.0,
+ "798": 291725824.0,
+ "799": 291725824.0,
+ "800": 291725824.0,
+ "801": 291725824.0,
+ "802": 291725824.0,
+ "803": 291725824.0,
+ "804": 291725824.0,
+ "805": 291725824.0,
+ "806": 291725824.0,
+ "807": 291725824.0,
+ "808": 291725824.0,
+ "809": 291725824.0,
+ "810": 291725824.0,
+ "811": 291725824.0,
+ "812": 291725824.0,
+ "813": 291725824.0,
+ "814": 291725824.0,
+ "815": 291725824.0,
+ "816": 291725824.0,
+ "817": 291725824.0,
+ "818": 291725824.0,
+ "819": 291725824.0,
+ "820": 291725824.0,
+ "821": 291725824.0,
+ "822": 291725824.0,
+ "823": 291725824.0,
+ "824": 291725824.0,
+ "825": 291725824.0,
+ "826": 291725824.0,
+ "827": 291725824.0,
+ "828": 291725824.0,
+ "829": 291725824.0,
+ "830": 291725824.0,
+ "831": 291725824.0,
+ "832": 291725824.0,
+ "833": 291725824.0,
+ "834": 291725824.0,
+ "835": 291725824.0,
+ "836": 291725824.0,
+ "837": 291725824.0,
+ "838": 291725824.0,
+ "839": 291725824.0,
+ "840": 291725824.0,
+ "841": 291725824.0,
+ "842": 291725824.0,
+ "843": 291725824.0,
+ "844": 291725824.0,
+ "845": 291725824.0,
+ "846": 291725824.0,
+ "847": 291725824.0,
+ "848": 291725824.0,
+ "849": 291725824.0,
+ "850": 291725824.0,
+ "851": 291725824.0,
+ "852": 291725824.0,
+ "853": 291725824.0,
+ "854": 291725824.0,
+ "855": 291725824.0,
+ "856": 291725824.0,
+ "857": 291725824.0,
+ "858": 291725824.0,
+ "859": 291725824.0,
+ "860": 291725824.0,
+ "861": 291725824.0,
+ "862": 291725824.0,
+ "863": 291725824.0,
+ "864": 291725824.0,
+ "865": 291725824.0,
+ "866": 291725824.0,
+ "867": 291725824.0,
+ "868": 291725824.0,
+ "869": 291725824.0,
+ "870": 291725824.0,
+ "871": 291725824.0,
+ "872": 291725824.0,
+ "873": 291725824.0,
+ "874": 291725824.0,
+ "875": 291725824.0,
+ "876": 291725824.0,
+ "877": 291725824.0,
+ "878": 291725824.0,
+ "879": 291725824.0,
+ "880": 291725824.0,
+ "881": 291725824.0,
+ "882": 291725824.0,
+ "883": 291725824.0,
+ "884": 291725824.0,
+ "885": 291725824.0,
+ "886": 291725824.0,
+ "887": 291725824.0,
+ "888": 291725824.0,
+ "889": 291725824.0,
+ "890": 291725824.0,
+ "891": 291725824.0,
+ "892": 291725824.0,
+ "893": 291725824.0,
+ "894": 291725824.0,
+ "895": 291725824.0,
+ "896": 291725824.0,
+ "897": 291725824.0,
+ "898": 291725824.0,
+ "899": 291725824.0,
+ "900": 291725824.0,
+ "901": 291725824.0,
+ "902": 291725824.0,
+ "903": 291725824.0,
+ "904": 291725824.0,
+ "905": 291725824.0,
+ "906": 291725824.0,
+ "907": 291725824.0,
+ "908": 291725824.0,
+ "909": 291725824.0,
+ "910": 291725824.0,
+ "911": 291725824.0,
+ "912": 291725824.0,
+ "913": 291725824.0,
+ "914": 291725824.0,
+ "915": 291725824.0,
+ "916": 291725824.0,
+ "917": 291725824.0,
+ "918": 291725824.0,
+ "919": 291725824.0,
+ "920": 291725824.0,
+ "921": 291725824.0,
+ "922": 291725824.0,
+ "923": 291725824.0,
+ "924": 291725824.0,
+ "925": 291725824.0,
+ "926": 291725824.0,
+ "927": 291725824.0,
+ "928": 291725824.0,
+ "929": 291725824.0,
+ "930": 291725824.0,
+ "931": 291725824.0,
+ "932": 291725824.0,
+ "933": 291725824.0,
+ "934": 291725824.0,
+ "935": 291725824.0,
+ "936": 291725824.0,
+ "937": 291725824.0,
+ "938": 291725824.0,
+ "939": 291725824.0,
+ "940": 291725824.0,
+ "941": 291725824.0,
+ "942": 291725824.0,
+ "943": 291725824.0,
+ "944": 291725824.0,
+ "945": 291725824.0,
+ "946": 291725824.0,
+ "947": 291725824.0,
+ "948": 291725824.0,
+ "949": 291725824.0,
+ "950": 291725824.0,
+ "951": 291725824.0,
+ "952": 291725824.0,
+ "953": 291725824.0,
+ "954": 291725824.0,
+ "955": 291725824.0,
+ "956": 291725824.0,
+ "957": 291725824.0,
+ "958": 291725824.0,
+ "959": 291725824.0,
+ "960": 291725824.0,
+ "961": 291725824.0,
+ "962": 291725824.0,
+ "963": 291725824.0,
+ "964": 291725824.0,
+ "965": 291725824.0,
+ "966": 291725824.0,
+ "967": 291725824.0,
+ "968": 291725824.0,
+ "969": 291725824.0,
+ "970": 291725824.0,
+ "971": 291725824.0,
+ "972": 291725824.0,
+ "973": 291725824.0,
+ "974": 291725824.0,
+ "975": 291725824.0,
+ "976": 291725824.0,
+ "977": 291725824.0,
+ "978": 291725824.0,
+ "979": 291725824.0,
+ "980": 291725824.0,
+ "981": 291725824.0,
+ "982": 291725824.0,
+ "983": 291725824.0,
+ "984": 291725824.0,
+ "985": 291725824.0,
+ "986": 291725824.0,
+ "987": 291725824.0,
+ "988": 291725824.0,
+ "989": 291725824.0,
+ "990": 291725824.0,
+ "991": 291725824.0,
+ "992": 291725824.0,
+ "993": 291725824.0,
+ "994": 291725824.0,
+ "995": 291725824.0,
+ "996": 291725824.0,
+ "997": 291725824.0,
+ "998": 291725824.0,
+ "999": 291725824.0,
+ "1000": 291725824.0,
+ "1001": 291726848.0,
+ "1002": 291726848.0,
+ "1003": 291726848.0,
+ "1004": 291726848.0,
+ "1005": 291726848.0,
+ "1006": 291726848.0,
+ "1007": 291726848.0,
+ "1008": 291726848.0,
+ "1009": 291726848.0,
+ "1010": 291726848.0,
+ "1011": 291726848.0,
+ "1012": 291726848.0,
+ "1013": 291726848.0,
+ "1014": 291726848.0,
+ "1015": 291726848.0,
+ "1016": 291726848.0,
+ "1017": 291726848.0,
+ "1018": 291726848.0,
+ "1019": 291726848.0,
+ "1020": 291726848.0,
+ "1021": 291726848.0,
+ "1022": 291726848.0,
+ "1023": 291726848.0,
+ "1024": 291726848.0,
+ "1025": 291726848.0,
+ "1026": 291726848.0,
+ "1027": 291726848.0,
+ "1028": 291726848.0,
+ "1029": 291726848.0,
+ "1030": 291726848.0,
+ "1031": 291726848.0,
+ "1032": 291726848.0,
+ "1033": 291726848.0,
+ "1034": 291726848.0,
+ "1035": 291726848.0,
+ "1036": 291726848.0,
+ "1037": 291726848.0,
+ "1038": 291726848.0,
+ "1039": 291726848.0,
+ "1040": 291726848.0,
+ "1041": 291726848.0,
+ "1042": 291726848.0,
+ "1043": 291726848.0,
+ "1044": 291726848.0,
+ "1045": 291726848.0,
+ "1046": 291726848.0,
+ "1047": 291726848.0,
+ "1048": 291726848.0,
+ "1049": 291726848.0,
+ "1050": 291726848.0,
+ "1051": 291726848.0,
+ "1052": 291726848.0,
+ "1053": 291726848.0,
+ "1054": 291726848.0,
+ "1055": 291726848.0,
+ "1056": 291726848.0,
+ "1057": 291726848.0,
+ "1058": 291726848.0,
+ "1059": 291726848.0,
+ "1060": 291726848.0,
+ "1061": 291726848.0,
+ "1062": 291726848.0,
+ "1063": 291726848.0,
+ "1064": 291726848.0,
+ "1065": 291726848.0,
+ "1066": 291726848.0,
+ "1067": 291726848.0,
+ "1068": 291726848.0,
+ "1069": 291726848.0,
+ "1070": 291726848.0,
+ "1071": 291726848.0,
+ "1072": 291726848.0,
+ "1073": 291726848.0,
+ "1074": 291726848.0,
+ "1075": 291726848.0,
+ "1076": 291726848.0,
+ "1077": 291726848.0,
+ "1078": 291726848.0,
+ "1079": 291726848.0,
+ "1080": 291726848.0,
+ "1081": 291726848.0,
+ "1082": 291726848.0,
+ "1083": 291726848.0,
+ "1084": 291726848.0,
+ "1085": 291726848.0,
+ "1086": 291726848.0,
+ "1087": 291726848.0,
+ "1088": 291726848.0,
+ "1089": 291726848.0,
+ "1090": 291726848.0,
+ "1091": 291726848.0,
+ "1092": 291726848.0,
+ "1093": 291726848.0,
+ "1094": 291726848.0,
+ "1095": 291726848.0,
+ "1096": 291726848.0,
+ "1097": 291726848.0,
+ "1098": 291726848.0,
+ "1099": 291726848.0,
+ "1100": 291726848.0,
+ "1101": 291726848.0,
+ "1102": 291726848.0,
+ "1103": 291726848.0,
+ "1104": 291726848.0,
+ "1105": 291726848.0,
+ "1106": 291726848.0,
+ "1107": 291726848.0,
+ "1108": 291726848.0,
+ "1109": 291726848.0,
+ "1110": 291726848.0,
+ "1111": 291726848.0,
+ "1112": 291726848.0,
+ "1113": 291726848.0,
+ "1114": 291726848.0,
+ "1115": 291726848.0,
+ "1116": 291726848.0,
+ "1117": 291726848.0,
+ "1118": 291726848.0,
+ "1119": 291726848.0,
+ "1120": 291726848.0,
+ "1121": 291726848.0,
+ "1122": 291726848.0,
+ "1123": 291726848.0,
+ "1124": 291726848.0,
+ "1125": 291726848.0,
+ "1126": 291726848.0,
+ "1127": 291726848.0,
+ "1128": 291726848.0,
+ "1129": 291726848.0,
+ "1130": 291726848.0,
+ "1131": 291726848.0,
+ "1132": 291726848.0,
+ "1133": 291726848.0,
+ "1134": 291726848.0,
+ "1135": 291726848.0,
+ "1136": 291726848.0,
+ "1137": 291726848.0,
+ "1138": 291726848.0,
+ "1139": 291726848.0,
+ "1140": 291726848.0,
+ "1141": 291726848.0,
+ "1142": 291726848.0,
+ "1143": 291726848.0,
+ "1144": 291726848.0,
+ "1145": 291726848.0,
+ "1146": 291726848.0,
+ "1147": 291726848.0,
+ "1148": 291726848.0,
+ "1149": 291726848.0,
+ "1150": 291726848.0,
+ "1151": 291726848.0,
+ "1152": 291726848.0,
+ "1153": 291726848.0,
+ "1154": 291726848.0,
+ "1155": 291726848.0,
+ "1156": 291726848.0,
+ "1157": 291726848.0,
+ "1158": 291726848.0,
+ "1159": 291726848.0,
+ "1160": 291726848.0,
+ "1161": 291726848.0,
+ "1162": 291726848.0,
+ "1163": 291726848.0,
+ "1164": 291726848.0,
+ "1165": 291726848.0,
+ "1166": 291726848.0,
+ "1167": 291726848.0,
+ "1168": 291726848.0,
+ "1169": 291726848.0,
+ "1170": 291726848.0,
+ "1171": 291726848.0,
+ "1172": 291726848.0,
+ "1173": 291726848.0,
+ "1174": 291726848.0,
+ "1175": 291726848.0,
+ "1176": 291726848.0,
+ "1177": 291726848.0,
+ "1178": 291726848.0,
+ "1179": 291726848.0,
+ "1180": 291726848.0,
+ "1181": 291726848.0,
+ "1182": 291726848.0,
+ "1183": 291726848.0,
+ "1184": 291726848.0,
+ "1185": 291726848.0,
+ "1186": 291726848.0,
+ "1187": 291726848.0,
+ "1188": 291726848.0,
+ "1189": 291726848.0,
+ "1190": 291726848.0,
+ "1191": 291726848.0,
+ "1192": 291726848.0,
+ "1193": 291726848.0,
+ "1194": 291726848.0,
+ "1195": 291726848.0,
+ "1196": 291726848.0,
+ "1197": 291726848.0,
+ "1198": 291726848.0,
+ "1199": 291726848.0,
+ "1200": 291726848.0,
+ "1201": 291726848.0,
+ "1202": 291726848.0,
+ "1203": 291726848.0,
+ "1204": 291726848.0,
+ "1205": 291726848.0,
+ "1206": 291726848.0,
+ "1207": 291726848.0,
+ "1208": 291726848.0,
+ "1209": 291726848.0,
+ "1210": 291726848.0,
+ "1211": 291726848.0,
+ "1212": 291726848.0,
+ "1213": 291726848.0,
+ "1214": 291726848.0,
+ "1215": 291726848.0,
+ "1216": 291726848.0,
+ "1217": 291726848.0,
+ "1218": 291726848.0,
+ "1219": 291726848.0,
+ "1220": 291726848.0,
+ "1221": 291726848.0,
+ "1222": 291726848.0,
+ "1223": 291726848.0,
+ "1224": 291726848.0,
+ "1225": 291726848.0,
+ "1226": 291726848.0,
+ "1227": 291726848.0,
+ "1228": 291726848.0,
+ "1229": 291726848.0,
+ "1230": 291726848.0,
+ "1231": 291726848.0,
+ "1232": 291726848.0,
+ "1233": 291726848.0,
+ "1234": 291726848.0,
+ "1235": 291726848.0,
+ "1236": 291726848.0,
+ "1237": 291726848.0,
+ "1238": 291726848.0,
+ "1239": 291726848.0,
+ "1240": 291726848.0,
+ "1241": 291726848.0,
+ "1242": 291726848.0,
+ "1243": 291726848.0,
+ "1244": 291726848.0,
+ "1245": 291726848.0,
+ "1246": 291726848.0,
+ "1247": 291726848.0,
+ "1248": 291726848.0,
+ "1249": 291726848.0,
+ "1250": 291726848.0,
+ "1251": 291726848.0,
+ "1252": 291726848.0,
+ "1253": 291726848.0,
+ "1254": 291726848.0,
+ "1255": 291726848.0,
+ "1256": 291726848.0,
+ "1257": 291726848.0,
+ "1258": 291726848.0,
+ "1259": 291726848.0,
+ "1260": 291726848.0,
+ "1261": 291726848.0,
+ "1262": 291726848.0,
+ "1263": 291726848.0,
+ "1264": 291726848.0,
+ "1265": 291726848.0,
+ "1266": 291726848.0,
+ "1267": 291726848.0,
+ "1268": 291726848.0,
+ "1269": 291726848.0,
+ "1270": 291726848.0,
+ "1271": 291726848.0,
+ "1272": 291726848.0,
+ "1273": 291726848.0,
+ "1274": 291726848.0,
+ "1275": 291726848.0,
+ "1276": 291726848.0,
+ "1277": 291726848.0,
+ "1278": 291726848.0,
+ "1279": 291726848.0,
+ "1280": 291726848.0,
+ "1281": 291726848.0,
+ "1282": 291726848.0,
+ "1283": 291726848.0,
+ "1284": 291726848.0,
+ "1285": 291726848.0,
+ "1286": 291726848.0,
+ "1287": 291726848.0,
+ "1288": 291726848.0,
+ "1289": 291726848.0,
+ "1290": 291726848.0,
+ "1291": 291726848.0,
+ "1292": 291726848.0,
+ "1293": 291726848.0,
+ "1294": 291726848.0,
+ "1295": 291726848.0,
+ "1296": 291726848.0,
+ "1297": 291726848.0,
+ "1298": 291726848.0,
+ "1299": 291726848.0,
+ "1300": 291726848.0,
+ "1301": 291726848.0,
+ "1302": 291726848.0,
+ "1303": 291726848.0,
+ "1304": 291726848.0,
+ "1305": 291726848.0,
+ "1306": 291726848.0,
+ "1307": 291726848.0,
+ "1308": 291726848.0,
+ "1309": 291726848.0,
+ "1310": 291726848.0,
+ "1311": 291726848.0,
+ "1312": 291726848.0,
+ "1313": 291726848.0,
+ "1314": 291726848.0,
+ "1315": 291726848.0,
+ "1316": 291726848.0,
+ "1317": 291726848.0,
+ "1318": 291726848.0,
+ "1319": 291726848.0,
+ "1320": 291726848.0,
+ "1321": 291726848.0,
+ "1322": 291726848.0,
+ "1323": 291726848.0,
+ "1324": 291726848.0,
+ "1325": 291726848.0,
+ "1326": 291726848.0,
+ "1327": 291726848.0,
+ "1328": 291726848.0,
+ "1329": 291726848.0,
+ "1330": 291726848.0,
+ "1331": 291726848.0,
+ "1332": 291726848.0,
+ "1333": 291726848.0,
+ "1334": 291726848.0,
+ "1335": 291726848.0,
+ "1336": 291726848.0,
+ "1337": 291726848.0,
+ "1338": 291726848.0,
+ "1339": 291726848.0,
+ "1340": 291726848.0,
+ "1341": 291726848.0,
+ "1342": 291726848.0,
+ "1343": 291726848.0,
+ "1344": 291726848.0,
+ "1345": 291726848.0,
+ "1346": 291726848.0,
+ "1347": 291726848.0,
+ "1348": 291726848.0,
+ "1349": 291726848.0,
+ "1350": 291726848.0,
+ "1351": 291726848.0,
+ "1352": 291726848.0,
+ "1353": 291726848.0,
+ "1354": 291726848.0,
+ "1355": 291726848.0,
+ "1356": 291726848.0,
+ "1357": 291726848.0,
+ "1358": 291726848.0,
+ "1359": 291726848.0,
+ "1360": 291726848.0,
+ "1361": 291726848.0,
+ "1362": 291726848.0,
+ "1363": 291726848.0,
+ "1364": 291726848.0,
+ "1365": 291726848.0,
+ "1366": 291726848.0,
+ "1367": 291726848.0,
+ "1368": 291726848.0,
+ "1369": 291726848.0,
+ "1370": 291726848.0,
+ "1371": 291726848.0,
+ "1372": 291726848.0,
+ "1373": 291726848.0,
+ "1374": 291726848.0,
+ "1375": 291726848.0,
+ "1376": 291726848.0,
+ "1377": 291726848.0,
+ "1378": 291726848.0,
+ "1379": 291726848.0,
+ "1380": 291726848.0,
+ "1381": 291726848.0,
+ "1382": 291726848.0,
+ "1383": 291726848.0,
+ "1384": 291726848.0,
+ "1385": 291726848.0,
+ "1386": 291726848.0,
+ "1387": 291726848.0,
+ "1388": 291726848.0,
+ "1389": 291726848.0,
+ "1390": 291726848.0,
+ "1391": 291726848.0,
+ "1392": 291726848.0,
+ "1393": 291726848.0,
+ "1394": 291726848.0,
+ "1395": 291726848.0,
+ "1396": 291726848.0,
+ "1397": 291726848.0,
+ "1398": 291726848.0,
+ "1399": 291726848.0,
+ "1400": 291726848.0,
+ "1401": 291726848.0,
+ "1402": 291726848.0,
+ "1403": 291726848.0,
+ "1404": 291726848.0,
+ "1405": 291726848.0,
+ "1406": 291726848.0,
+ "1407": 291726848.0,
+ "1408": 291726848.0,
+ "1409": 291726848.0,
+ "1410": 291726848.0,
+ "1411": 291726848.0,
+ "1412": 291726848.0,
+ "1413": 291726848.0,
+ "1414": 291726848.0,
+ "1415": 291726848.0,
+ "1416": 291726848.0,
+ "1417": 291726848.0,
+ "1418": 291726848.0,
+ "1419": 291726848.0,
+ "1420": 291726848.0,
+ "1421": 291726848.0,
+ "1422": 291726848.0,
+ "1423": 291726848.0,
+ "1424": 291726848.0,
+ "1425": 291726848.0,
+ "1426": 291726848.0,
+ "1427": 291726848.0,
+ "1428": 291726848.0,
+ "1429": 291726848.0,
+ "1430": 291726848.0,
+ "1431": 291726848.0,
+ "1432": 291726848.0,
+ "1433": 291726848.0,
+ "1434": 291726848.0,
+ "1435": 291726848.0,
+ "1436": 291726848.0,
+ "1437": 291726848.0,
+ "1438": 291726848.0,
+ "1439": 291726848.0,
+ "1440": 291726848.0,
+ "1441": 291726848.0,
+ "1442": 291726848.0,
+ "1443": 291726848.0,
+ "1444": 291726848.0,
+ "1445": 291726848.0,
+ "1446": 291726848.0,
+ "1447": 291726848.0,
+ "1448": 291726848.0,
+ "1449": 291726848.0,
+ "1450": 291726848.0,
+ "1451": 291726848.0,
+ "1452": 291726848.0,
+ "1453": 291726848.0,
+ "1454": 291726848.0,
+ "1455": 291726848.0,
+ "1456": 291726848.0,
+ "1457": 291726848.0,
+ "1458": 291726848.0,
+ "1459": 291726848.0,
+ "1460": 291726848.0,
+ "1461": 291726848.0,
+ "1462": 291726848.0,
+ "1463": 291726848.0,
+ "1464": 291726848.0,
+ "1465": 291726848.0,
+ "1466": 291726848.0,
+ "1467": 291726848.0,
+ "1468": 291726848.0,
+ "1469": 291726848.0,
+ "1470": 291726848.0,
+ "1471": 291726848.0,
+ "1472": 291726848.0,
+ "1473": 291726848.0,
+ "1474": 291726848.0,
+ "1475": 291726848.0,
+ "1476": 291726848.0,
+ "1477": 291726848.0,
+ "1478": 291726848.0,
+ "1479": 291726848.0,
+ "1480": 291726848.0,
+ "1481": 291726848.0,
+ "1482": 291726848.0,
+ "1483": 291726848.0,
+ "1484": 291726848.0,
+ "1485": 291726848.0,
+ "1486": 291726848.0,
+ "1487": 291726848.0,
+ "1488": 291726848.0,
+ "1489": 291726848.0,
+ "1490": 291726848.0,
+ "1491": 291726848.0,
+ "1492": 291726848.0,
+ "1493": 291726848.0,
+ "1494": 291726848.0,
+ "1495": 291726848.0,
+ "1496": 291726848.0,
+ "1497": 291726848.0,
+ "1498": 291726848.0,
+ "1499": 291726848.0,
+ "1500": 291726848.0,
+ "1501": 291726848.0,
+ "1502": 291726848.0,
+ "1503": 291726848.0,
+ "1504": 291726848.0,
+ "1505": 291726848.0,
+ "1506": 291726848.0,
+ "1507": 291726848.0,
+ "1508": 291726848.0,
+ "1509": 291726848.0,
+ "1510": "nan",
+ "1511": "nan",
+ "1512": "nan",
+ "1513": "nan",
+ "1514": "nan",
+ "1515": "nan",
+ "1516": "nan",
+ "1517": "nan",
+ "1518": "nan",
+ "1519": "nan",
+ "1520": "nan",
+ "1521": "nan",
+ "1522": "nan",
+ "1523": "nan",
+ "1524": "nan",
+ "1525": "nan",
+ "1526": "nan",
+ "1527": "nan",
+ "1528": "nan",
+ "1529": "nan",
+ "1530": "nan",
+ "1531": "nan",
+ "1532": "nan",
+ "1533": "nan",
+ "1534": "nan",
+ "1535": "nan",
+ "1536": "nan",
+ "1537": "nan",
+ "1538": "nan",
+ "1539": "nan",
+ "1540": "nan",
+ "1541": "nan",
+ "1542": "nan",
+ "1543": "nan",
+ "1544": "nan",
+ "1545": "nan",
+ "1546": "nan",
+ "1547": "nan",
+ "1548": "nan",
+ "1549": "nan",
+ "1550": "nan",
+ "1551": "nan",
+ "1552": "nan",
+ "1553": "nan",
+ "1554": "nan",
+ "1555": "nan",
+ "1556": "nan",
+ "1557": "nan",
+ "1558": "nan",
+ "1559": "nan",
+ "1560": "nan",
+ "1561": "nan",
+ "1562": "nan",
+ "1563": "nan",
+ "1564": "nan",
+ "1565": "nan",
+ "1566": "nan",
+ "1567": "nan",
+ "1568": "nan",
+ "1569": "nan",
+ "1570": "nan",
+ "1571": "nan",
+ "1572": "nan",
+ "1573": "nan",
+ "1574": "nan",
+ "1575": "nan",
+ "1576": "nan",
+ "1577": "nan",
+ "1578": "nan",
+ "1579": "nan",
+ "1580": "nan",
+ "1581": "nan",
+ "1582": "nan",
+ "1583": "nan",
+ "1584": "nan",
+ "1585": "nan",
+ "1586": "nan",
+ "1587": "nan",
+ "1588": "nan",
+ "1589": "nan",
+ "1590": "nan",
+ "1591": "nan",
+ "1592": "nan",
+ "1593": "nan",
+ "1594": "nan",
+ "1595": "nan",
+ "1596": "nan",
+ "1597": "nan",
+ "1598": "nan",
+ "1599": "nan",
+ "1600": "nan",
+ "1601": "nan",
+ "1602": "nan",
+ "1603": "nan",
+ "1604": "nan",
+ "1605": "nan",
+ "1606": "nan",
+ "1607": "nan",
+ "1608": "nan",
+ "1609": "nan",
+ "1610": "nan",
+ "1611": "nan",
+ "1612": "nan",
+ "1613": "nan",
+ "1614": "nan",
+ "1615": "nan",
+ "1616": "nan",
+ "1617": "nan",
+ "1618": "nan",
+ "1619": "nan",
+ "1620": "nan",
+ "1621": "nan",
+ "1622": "nan",
+ "1623": "nan",
+ "1624": "nan",
+ "1625": "nan",
+ "1626": "nan",
+ "1627": "nan",
+ "1628": "nan",
+ "1629": "nan",
+ "1630": "nan",
+ "1631": "nan",
+ "1632": "nan",
+ "1633": "nan",
+ "1634": "nan",
+ "1635": "nan",
+ "1636": "nan",
+ "1637": "nan",
+ "1638": "nan",
+ "1639": "nan",
+ "1640": "nan",
+ "1641": "nan",
+ "1642": "nan",
+ "1643": "nan",
+ "1644": "nan",
+ "1645": "nan",
+ "1646": "nan",
+ "1647": "nan",
+ "1648": "nan",
+ "1649": "nan",
+ "1650": "nan",
+ "1651": "nan",
+ "1652": "nan",
+ "1653": "nan",
+ "1654": "nan",
+ "1655": "nan",
+ "1656": "nan",
+ "1657": "nan",
+ "1658": "nan",
+ "1659": "nan",
+ "1660": "nan",
+ "1661": "nan",
+ "1662": "nan",
+ "1663": "nan",
+ "1664": "nan",
+ "1665": "nan",
+ "1666": "nan",
+ "1667": "nan",
+ "1668": "nan",
+ "1669": "nan",
+ "1670": "nan",
+ "1671": "nan",
+ "1672": "nan",
+ "1673": "nan",
+ "1674": "nan",
+ "1675": "nan",
+ "1676": "nan",
+ "1677": "nan",
+ "1678": "nan",
+ "1679": "nan",
+ "1680": "nan",
+ "1681": "nan",
+ "1682": "nan",
+ "1683": "nan",
+ "1684": "nan",
+ "1685": "nan",
+ "1686": "nan",
+ "1687": "nan",
+ "1688": "nan",
+ "1689": "nan",
+ "1690": "nan",
+ "1691": "nan",
+ "1692": "nan",
+ "1693": "nan",
+ "1694": "nan",
+ "1695": "nan",
+ "1696": "nan",
+ "1697": "nan",
+ "1698": "nan",
+ "1699": "nan",
+ "1700": "nan",
+ "1701": "nan",
+ "1702": "nan",
+ "1703": "nan",
+ "1704": "nan",
+ "1705": "nan",
+ "1706": "nan",
+ "1707": "nan",
+ "1708": "nan",
+ "1709": "nan",
+ "1710": "nan",
+ "1711": "nan",
+ "1712": "nan",
+ "1713": "nan",
+ "1714": "nan",
+ "1715": "nan",
+ "1716": "nan",
+ "1717": "nan",
+ "1718": "nan",
+ "1719": "nan",
+ "1720": "nan",
+ "1721": "nan",
+ "1722": "nan",
+ "1723": "nan",
+ "1724": "nan",
+ "1725": "nan",
+ "1726": "nan",
+ "1727": "nan",
+ "1728": "nan",
+ "1729": "nan",
+ "1730": "nan",
+ "1731": "nan",
+ "1732": "nan",
+ "1733": "nan",
+ "1734": "nan",
+ "1735": "nan",
+ "1736": "nan",
+ "1737": "nan",
+ "1738": "nan",
+ "1739": "nan",
+ "1740": "nan",
+ "1741": "nan",
+ "1742": "nan",
+ "1743": "nan",
+ "1744": "nan",
+ "1745": "nan",
+ "1746": "nan",
+ "1747": "nan",
+ "1748": "nan",
+ "1749": "nan",
+ "1750": "nan",
+ "1751": "nan",
+ "1752": "nan",
+ "1753": "nan",
+ "1754": "nan",
+ "1755": "nan",
+ "1756": "nan",
+ "1757": "nan",
+ "1758": "nan",
+ "1759": "nan",
+ "1760": "nan",
+ "1761": "nan",
+ "1762": "nan",
+ "1763": "nan",
+ "1764": "nan",
+ "1765": "nan",
+ "1766": "nan",
+ "1767": "nan",
+ "1768": "nan",
+ "1769": "nan",
+ "1770": "nan",
+ "1771": "nan",
+ "1772": "nan",
+ "1773": "nan",
+ "1774": "nan",
+ "1775": "nan",
+ "1776": "nan",
+ "1777": "nan",
+ "1778": "nan",
+ "1779": "nan",
+ "1780": "nan",
+ "1781": "nan",
+ "1782": "nan",
+ "1783": "nan",
+ "1784": "nan",
+ "1785": "nan",
+ "1786": "nan",
+ "1787": "nan",
+ "1788": "nan",
+ "1789": "nan",
+ "1790": "nan",
+ "1791": "nan",
+ "1792": "nan",
+ "1793": "nan",
+ "1794": "nan",
+ "1795": "nan",
+ "1796": "nan",
+ "1797": "nan",
+ "1798": "nan",
+ "1799": "nan",
+ "1800": "nan",
+ "1801": "nan",
+ "1802": "nan",
+ "1803": "nan",
+ "1804": "nan",
+ "1805": "nan",
+ "1806": "nan",
+ "1807": "nan",
+ "1808": "nan",
+ "1809": "nan",
+ "1810": "nan",
+ "1811": "nan",
+ "1812": "nan",
+ "1813": "nan",
+ "1814": "nan",
+ "1815": "nan",
+ "1816": "nan",
+ "1817": "nan",
+ "1818": "nan",
+ "1819": "nan",
+ "1820": "nan",
+ "1821": "nan",
+ "1822": "nan",
+ "1823": "nan",
+ "1824": "nan",
+ "1825": "nan",
+ "1826": "nan",
+ "1827": "nan",
+ "1828": "nan",
+ "1829": "nan",
+ "1830": "nan",
+ "1831": "nan",
+ "1832": "nan",
+ "1833": "nan",
+ "1834": "nan",
+ "1835": "nan",
+ "1836": "nan",
+ "1837": "nan",
+ "1838": "nan",
+ "1839": "nan",
+ "1840": "nan",
+ "1841": "nan",
+ "1842": "nan",
+ "1843": "nan",
+ "1844": "nan",
+ "1845": "nan",
+ "1846": "nan",
+ "1847": "nan",
+ "1848": "nan",
+ "1849": "nan",
+ "1850": "nan",
+ "1851": "nan",
+ "1852": "nan",
+ "1853": "nan",
+ "1854": "nan",
+ "1855": "nan",
+ "1856": "nan",
+ "1857": "nan",
+ "1858": "nan",
+ "1859": "nan",
+ "1860": "nan",
+ "1861": "nan",
+ "1862": "nan",
+ "1863": "nan",
+ "1864": "nan",
+ "1865": "nan",
+ "1866": "nan",
+ "1867": "nan",
+ "1868": "nan",
+ "1869": "nan",
+ "1870": "nan",
+ "1871": "nan",
+ "1872": "nan",
+ "1873": "nan",
+ "1874": "nan",
+ "1875": "nan",
+ "1876": "nan",
+ "1877": "nan",
+ "1878": "nan",
+ "1879": "nan",
+ "1880": "nan",
+ "1881": "nan",
+ "1882": "nan",
+ "1883": "nan",
+ "1884": "nan",
+ "1885": "nan",
+ "1886": "nan",
+ "1887": "nan",
+ "1888": "nan",
+ "1889": "nan",
+ "1890": "nan",
+ "1891": "nan",
+ "1892": "nan",
+ "1893": "nan",
+ "1894": "nan",
+ "1895": "nan",
+ "1896": "nan",
+ "1897": "nan",
+ "1898": "nan",
+ "1899": "nan",
+ "1900": "nan",
+ "1901": "nan",
+ "1902": "nan",
+ "1903": "nan",
+ "1904": "nan",
+ "1905": "nan",
+ "1906": "nan",
+ "1907": "nan",
+ "1908": "nan",
+ "1909": "nan",
+ "1910": "nan",
+ "1911": "nan",
+ "1912": "nan",
+ "1913": "nan",
+ "1914": "nan",
+ "1915": "nan",
+ "1916": "nan",
+ "1917": "nan",
+ "1918": "nan",
+ "1919": "nan",
+ "1920": "nan",
+ "1921": "nan",
+ "1922": "nan",
+ "1923": "nan",
+ "1924": "nan",
+ "1925": "nan",
+ "1926": "nan",
+ "1927": "nan",
+ "1928": "nan",
+ "1929": "nan",
+ "1930": "nan",
+ "1931": "nan",
+ "1932": "nan",
+ "1933": "nan",
+ "1934": "nan",
+ "1935": "nan",
+ "1936": "nan",
+ "1937": "nan",
+ "1938": "nan",
+ "1939": "nan",
+ "1940": "nan",
+ "1941": "nan",
+ "1942": "nan",
+ "1943": "nan",
+ "1944": "nan",
+ "1945": "nan",
+ "1946": "nan",
+ "1947": "nan",
+ "1948": "nan",
+ "1949": "nan",
+ "1950": "nan",
+ "1951": "nan",
+ "1952": "nan",
+ "1953": "nan",
+ "1954": "nan",
+ "1955": "nan",
+ "1956": "nan",
+ "1957": "nan",
+ "1958": "nan",
+ "1959": "nan",
+ "1960": "nan",
+ "1961": "nan",
+ "1962": "nan",
+ "1963": "nan",
+ "1964": "nan",
+ "1965": "nan",
+ "1966": "nan",
+ "1967": "nan",
+ "1968": "nan",
+ "1969": "nan",
+ "1970": "nan",
+ "1971": "nan",
+ "1972": "nan",
+ "1973": "nan",
+ "1974": "nan",
+ "1975": "nan",
+ "1976": "nan",
+ "1977": "nan",
+ "1978": "nan",
+ "1979": "nan",
+ "1980": "nan",
+ "1981": "nan",
+ "1982": "nan",
+ "1983": "nan",
+ "1984": "nan",
+ "1985": "nan",
+ "1986": "nan",
+ "1987": "nan",
+ "1988": "nan",
+ "1989": "nan",
+ "1990": "nan",
+ "1991": "nan",
+ "1992": "nan",
+ "1993": "nan",
+ "1994": "nan",
+ "1995": "nan",
+ "1996": "nan",
+ "1997": "nan",
+ "1998": "nan",
+ "1999": "nan",
+ "2000": "nan"
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 423685120.0,
+ "2": 423685120.0,
+ "3": 423685120.0,
+ "4": 423685120.0,
+ "5": 423685120.0,
+ "6": 423685120.0,
+ "7": 423685120.0,
+ "8": 423685120.0,
+ "9": 423685120.0,
+ "10": 423685120.0,
+ "11": 423685120.0,
+ "12": 423685120.0,
+ "13": 423685120.0,
+ "14": 423685120.0,
+ "15": 423685120.0,
+ "16": 423685120.0,
+ "17": 423685120.0,
+ "18": 423685120.0,
+ "19": 423685120.0,
+ "20": 423685120.0,
+ "21": 423685120.0,
+ "22": 423685120.0,
+ "23": 423685120.0,
+ "24": 423685120.0,
+ "25": 423685120.0,
+ "26": 423685120.0,
+ "27": 423685120.0,
+ "28": 423685120.0,
+ "29": 423685120.0,
+ "30": 423685120.0,
+ "31": 423685120.0,
+ "32": 423685120.0,
+ "33": 423685120.0,
+ "34": 423685120.0,
+ "35": 423685120.0,
+ "36": 423685120.0,
+ "37": 423685120.0,
+ "38": 423685120.0,
+ "39": 423685120.0,
+ "40": 423685120.0,
+ "41": 423685120.0,
+ "42": 423685120.0,
+ "43": 423685120.0,
+ "44": 423685120.0,
+ "45": 423685120.0,
+ "46": 423685120.0,
+ "47": 423685120.0,
+ "48": 423685120.0,
+ "49": 423685120.0,
+ "50": 423685120.0,
+ "51": 423685120.0,
+ "52": 423685120.0,
+ "53": 423685120.0,
+ "54": 423685120.0,
+ "55": 423685120.0,
+ "56": 423685120.0,
+ "57": 423685120.0,
+ "58": 423685120.0,
+ "59": 423685120.0,
+ "60": 423685120.0,
+ "61": 423685120.0,
+ "62": 423685120.0,
+ "63": 423685120.0,
+ "64": 423685120.0,
+ "65": 423685120.0,
+ "66": 423685120.0,
+ "67": 423685120.0,
+ "68": 423685120.0,
+ "69": 423685120.0,
+ "70": 423685120.0,
+ "71": 423685120.0,
+ "72": 423685120.0,
+ "73": 423685120.0,
+ "74": 423685120.0,
+ "75": 423685120.0,
+ "76": 423685120.0,
+ "77": 423685120.0,
+ "78": 423685120.0,
+ "79": 423685120.0,
+ "80": 423685120.0,
+ "81": 423685120.0,
+ "82": 423685120.0,
+ "83": 423685120.0,
+ "84": 423685120.0,
+ "85": 423685120.0,
+ "86": 423685120.0,
+ "87": 423685120.0,
+ "88": 423685120.0,
+ "89": 423685120.0,
+ "90": 423685120.0,
+ "91": 423685120.0,
+ "92": 423685120.0,
+ "93": 423685120.0,
+ "94": 423685120.0,
+ "95": 423685120.0,
+ "96": 423685120.0,
+ "97": 423685120.0,
+ "98": 423685120.0,
+ "99": 423685120.0,
+ "100": 423685120.0,
+ "101": 423685120.0,
+ "102": 423685120.0,
+ "103": 423685120.0,
+ "104": 423685120.0,
+ "105": 423685120.0,
+ "106": 423685120.0,
+ "107": 423685120.0,
+ "108": 423685120.0,
+ "109": 423685120.0,
+ "110": 423685120.0,
+ "111": 423685120.0,
+ "112": 423685120.0,
+ "113": 423685120.0,
+ "114": 423685120.0,
+ "115": 423685120.0,
+ "116": 423685120.0,
+ "117": 423685120.0,
+ "118": 423685120.0,
+ "119": 423685120.0,
+ "120": 423685120.0,
+ "121": 423685120.0,
+ "122": 423685120.0,
+ "123": 423685120.0,
+ "124": 423685120.0,
+ "125": 423685120.0,
+ "126": 423685120.0,
+ "127": 423685120.0,
+ "128": 423685120.0,
+ "129": 423685120.0,
+ "130": 423685120.0,
+ "131": 423685120.0,
+ "132": 423685120.0,
+ "133": 423685120.0,
+ "134": 423685120.0,
+ "135": 423685120.0,
+ "136": 423685120.0,
+ "137": 423685120.0,
+ "138": 423685120.0,
+ "139": 423685120.0,
+ "140": 423685120.0,
+ "141": 423685120.0,
+ "142": 423685120.0,
+ "143": 423685120.0,
+ "144": 423685120.0,
+ "145": 423685120.0,
+ "146": 423685120.0,
+ "147": 423685120.0,
+ "148": 423685120.0,
+ "149": 423685120.0,
+ "150": 423685120.0,
+ "151": 423685120.0,
+ "152": 423685120.0,
+ "153": 423685120.0,
+ "154": 423685120.0,
+ "155": 423685120.0,
+ "156": 423685120.0,
+ "157": 423685120.0,
+ "158": 423685120.0,
+ "159": 423685120.0,
+ "160": 423685120.0,
+ "161": 423685120.0,
+ "162": 423685120.0,
+ "163": 423685120.0,
+ "164": 423685120.0,
+ "165": 423685120.0,
+ "166": 423685120.0,
+ "167": 423685120.0,
+ "168": 423685120.0,
+ "169": 423685120.0,
+ "170": 423685120.0,
+ "171": 423685120.0,
+ "172": 423685120.0,
+ "173": 423685120.0,
+ "174": 423685120.0,
+ "175": 423685120.0,
+ "176": 423685120.0,
+ "177": 423685120.0,
+ "178": 423685120.0,
+ "179": 423685120.0,
+ "180": 423685120.0,
+ "181": 423685120.0,
+ "182": 423685120.0,
+ "183": 423685120.0,
+ "184": 423685120.0,
+ "185": 423685120.0,
+ "186": 423685120.0,
+ "187": 423685120.0,
+ "188": 423685120.0,
+ "189": 423685120.0,
+ "190": 423685120.0,
+ "191": 423685120.0,
+ "192": 423685120.0,
+ "193": 423685120.0,
+ "194": 423685120.0,
+ "195": 423685120.0,
+ "196": 423685120.0,
+ "197": 423685120.0,
+ "198": 423685120.0,
+ "199": 423685120.0,
+ "200": 423685120.0,
+ "201": 423685120.0,
+ "202": 423685120.0,
+ "203": 423685120.0,
+ "204": 423685120.0,
+ "205": 423685120.0,
+ "206": 423685120.0,
+ "207": 423685120.0,
+ "208": 423685120.0,
+ "209": 423685120.0,
+ "210": 423685120.0,
+ "211": 423685120.0,
+ "212": 423685120.0,
+ "213": 423685120.0,
+ "214": 423685120.0,
+ "215": 423685120.0,
+ "216": 423685120.0,
+ "217": 423685120.0,
+ "218": 423685120.0,
+ "219": 423685120.0,
+ "220": 423685120.0,
+ "221": 423685120.0,
+ "222": 423685120.0,
+ "223": 423685120.0,
+ "224": 423685120.0,
+ "225": 423685120.0,
+ "226": 423685120.0,
+ "227": 423685120.0,
+ "228": 423685120.0,
+ "229": 423685120.0,
+ "230": 423685120.0,
+ "231": 423685120.0,
+ "232": 423685120.0,
+ "233": 423685120.0,
+ "234": 423685120.0,
+ "235": 423685120.0,
+ "236": 423685120.0,
+ "237": 423685120.0,
+ "238": 423685120.0,
+ "239": 423685120.0,
+ "240": 423685120.0,
+ "241": 423685120.0,
+ "242": 423685120.0,
+ "243": 423685120.0,
+ "244": 423685120.0,
+ "245": 423685120.0,
+ "246": 423685120.0,
+ "247": 423685120.0,
+ "248": 423685120.0,
+ "249": 423685120.0,
+ "250": 423685120.0,
+ "251": 423685120.0,
+ "252": 423685120.0,
+ "253": 423685120.0,
+ "254": 423685120.0,
+ "255": 423685120.0,
+ "256": 423685120.0,
+ "257": 423685120.0,
+ "258": 423685120.0,
+ "259": 423685120.0,
+ "260": 423685120.0,
+ "261": 423685120.0,
+ "262": 423685120.0,
+ "263": 423685120.0,
+ "264": 423685120.0,
+ "265": 423685120.0,
+ "266": 423685120.0,
+ "267": 423685120.0,
+ "268": 423685120.0,
+ "269": 423685120.0,
+ "270": 423685120.0,
+ "271": 423685120.0,
+ "272": 423685120.0,
+ "273": 423685120.0,
+ "274": 423685120.0,
+ "275": 423685120.0,
+ "276": 423685120.0,
+ "277": 423685120.0,
+ "278": 423685120.0,
+ "279": 423685120.0,
+ "280": 423685120.0,
+ "281": 423685120.0,
+ "282": 423685120.0,
+ "283": 423685120.0,
+ "284": 423685120.0,
+ "285": 423685120.0,
+ "286": 423685120.0,
+ "287": 423685120.0,
+ "288": 423685120.0,
+ "289": 423685120.0,
+ "290": 423685120.0,
+ "291": 423685120.0,
+ "292": 423685120.0,
+ "293": 423685120.0,
+ "294": 423685120.0,
+ "295": 423685120.0,
+ "296": 423685120.0,
+ "297": 423685120.0,
+ "298": 423685120.0,
+ "299": 423685120.0,
+ "300": 423685120.0,
+ "301": 423685120.0,
+ "302": 423685120.0,
+ "303": 423685120.0,
+ "304": 423685120.0,
+ "305": 423685120.0,
+ "306": 423685120.0,
+ "307": 423685120.0,
+ "308": 423685120.0,
+ "309": 423685120.0,
+ "310": 423685120.0,
+ "311": 423685120.0,
+ "312": 423685120.0,
+ "313": 423685120.0,
+ "314": 423685120.0,
+ "315": 423685120.0,
+ "316": 423685120.0,
+ "317": 423685120.0,
+ "318": 423685120.0,
+ "319": 423685120.0,
+ "320": 423685120.0,
+ "321": 423685120.0,
+ "322": 423685120.0,
+ "323": 423685120.0,
+ "324": 423685120.0,
+ "325": 423685120.0,
+ "326": 423685120.0,
+ "327": 423685120.0,
+ "328": 423685120.0,
+ "329": 423685120.0,
+ "330": 423685120.0,
+ "331": 423685120.0,
+ "332": 423685120.0,
+ "333": 423685120.0,
+ "334": 423685120.0,
+ "335": 423685120.0,
+ "336": 423685120.0,
+ "337": 423685120.0,
+ "338": 423685120.0,
+ "339": 423685120.0,
+ "340": 423685120.0,
+ "341": 423685120.0,
+ "342": 423685120.0,
+ "343": 423685120.0,
+ "344": 423685120.0,
+ "345": 423685120.0,
+ "346": 423685120.0,
+ "347": 423685120.0,
+ "348": 423685120.0,
+ "349": 423685120.0,
+ "350": 423685120.0,
+ "351": 423685120.0,
+ "352": 423685120.0,
+ "353": 423685120.0,
+ "354": 423685120.0,
+ "355": 423685120.0,
+ "356": 423685120.0,
+ "357": 423685120.0,
+ "358": 423685120.0,
+ "359": 423685120.0,
+ "360": 423685120.0,
+ "361": 423685120.0,
+ "362": 423685120.0,
+ "363": 423685120.0,
+ "364": 423685120.0,
+ "365": 423685120.0,
+ "366": 423685120.0,
+ "367": 423685120.0,
+ "368": 423685120.0,
+ "369": 423685120.0,
+ "370": 423685120.0,
+ "371": 423685120.0,
+ "372": 423685120.0,
+ "373": 423685120.0,
+ "374": 423685120.0,
+ "375": 423685120.0,
+ "376": 423685120.0,
+ "377": 423685120.0,
+ "378": 423685120.0,
+ "379": 423685120.0,
+ "380": 423685120.0,
+ "381": 423685120.0,
+ "382": 423685120.0,
+ "383": 423685120.0,
+ "384": 423685120.0,
+ "385": 423685120.0,
+ "386": 423685120.0,
+ "387": 423685120.0,
+ "388": 423685120.0,
+ "389": 423685120.0,
+ "390": 423685120.0,
+ "391": 423685120.0,
+ "392": 423685120.0,
+ "393": 423685120.0,
+ "394": 423685120.0,
+ "395": 423685120.0,
+ "396": 423685120.0,
+ "397": 423685120.0,
+ "398": 423685120.0,
+ "399": 423685120.0,
+ "400": 423685120.0,
+ "401": 423685120.0,
+ "402": 423685120.0,
+ "403": 423685120.0,
+ "404": 423685120.0,
+ "405": 423685120.0,
+ "406": 423685120.0,
+ "407": 423685120.0,
+ "408": 423685120.0,
+ "409": 423685120.0,
+ "410": 423685120.0,
+ "411": 423685120.0,
+ "412": 423685120.0,
+ "413": 423685120.0,
+ "414": 423685120.0,
+ "415": 423685120.0,
+ "416": 423685120.0,
+ "417": 423685120.0,
+ "418": 423685120.0,
+ "419": 423685120.0,
+ "420": 423685120.0,
+ "421": 423685120.0,
+ "422": 423685120.0,
+ "423": 423685120.0,
+ "424": 423685120.0,
+ "425": 423685120.0,
+ "426": 423685120.0,
+ "427": 423685120.0,
+ "428": 423685120.0,
+ "429": 423685120.0,
+ "430": 423685120.0,
+ "431": 423685120.0,
+ "432": 423685120.0,
+ "433": 423685120.0,
+ "434": 423685120.0,
+ "435": 423685120.0,
+ "436": 423685120.0,
+ "437": 423685120.0,
+ "438": 423685120.0,
+ "439": 423685120.0,
+ "440": 423685120.0,
+ "441": 423685120.0,
+ "442": 423685120.0,
+ "443": 423685120.0,
+ "444": 423685120.0,
+ "445": 423685120.0,
+ "446": 423685120.0,
+ "447": 423685120.0,
+ "448": 423685120.0,
+ "449": 423685120.0,
+ "450": 423685120.0,
+ "451": 423685120.0,
+ "452": 423685120.0,
+ "453": 423685120.0,
+ "454": 423685120.0,
+ "455": 423685120.0,
+ "456": 423685120.0,
+ "457": 423685120.0,
+ "458": 423685120.0,
+ "459": 423685120.0,
+ "460": 423685120.0,
+ "461": 423685120.0,
+ "462": 423685120.0,
+ "463": 423685120.0,
+ "464": 423685120.0,
+ "465": 423685120.0,
+ "466": 423685120.0,
+ "467": 423685120.0,
+ "468": 423685120.0,
+ "469": 423685120.0,
+ "470": 423685120.0,
+ "471": 423685120.0,
+ "472": 423685120.0,
+ "473": 423685120.0,
+ "474": 423685120.0,
+ "475": 423685120.0,
+ "476": 423685120.0,
+ "477": 423685120.0,
+ "478": 423685120.0,
+ "479": 423685120.0,
+ "480": 423685120.0,
+ "481": 423685120.0,
+ "482": 423685120.0,
+ "483": 423685120.0,
+ "484": 423685120.0,
+ "485": 423685120.0,
+ "486": 423685120.0,
+ "487": 423685120.0,
+ "488": 423685120.0,
+ "489": 423685120.0,
+ "490": 423685120.0,
+ "491": 423685120.0,
+ "492": 423685120.0,
+ "493": 423685120.0,
+ "494": 423685120.0,
+ "495": 423685120.0,
+ "496": 423685120.0,
+ "497": 423685120.0,
+ "498": 423685120.0,
+ "499": 423685120.0,
+ "500": 423685120.0,
+ "501": 423685120.0,
+ "502": 423685120.0,
+ "503": 423685120.0,
+ "504": 423685120.0,
+ "505": 423685120.0,
+ "506": 423685120.0,
+ "507": 423685120.0,
+ "508": 423685120.0,
+ "509": 423685120.0,
+ "510": 423685120.0,
+ "511": 423685120.0,
+ "512": 423685120.0,
+ "513": 423685120.0,
+ "514": 423685120.0,
+ "515": 423685120.0,
+ "516": 423685120.0,
+ "517": 423685120.0,
+ "518": 423685120.0,
+ "519": 423685120.0,
+ "520": 423685120.0,
+ "521": 423685120.0,
+ "522": 423685120.0,
+ "523": 423685120.0,
+ "524": 423685120.0,
+ "525": 423685120.0,
+ "526": 423685120.0,
+ "527": 423685120.0,
+ "528": 423685120.0,
+ "529": 423685120.0,
+ "530": 423685120.0,
+ "531": 423685120.0,
+ "532": 423685120.0,
+ "533": 423685120.0,
+ "534": 423685120.0,
+ "535": 423685120.0,
+ "536": 423685120.0,
+ "537": 423685120.0,
+ "538": 423685120.0,
+ "539": 423685120.0,
+ "540": 423685120.0,
+ "541": 423685120.0,
+ "542": 423685120.0,
+ "543": 423685120.0,
+ "544": 423685120.0,
+ "545": 423685120.0,
+ "546": 423685120.0,
+ "547": 423685120.0,
+ "548": 423685120.0,
+ "549": 423685120.0,
+ "550": 423685120.0,
+ "551": 423685120.0,
+ "552": 423685120.0,
+ "553": 423685120.0,
+ "554": 423685120.0,
+ "555": 423685120.0,
+ "556": 423685120.0,
+ "557": 423685120.0,
+ "558": 423685120.0,
+ "559": 423685120.0,
+ "560": 423685120.0,
+ "561": 423685120.0,
+ "562": 423685120.0,
+ "563": 423685120.0,
+ "564": 423685120.0,
+ "565": 423685120.0,
+ "566": 423685120.0,
+ "567": 423685120.0,
+ "568": 423685120.0,
+ "569": 423685120.0,
+ "570": 423685120.0,
+ "571": 423685120.0,
+ "572": 423685120.0,
+ "573": 423685120.0,
+ "574": 423685120.0,
+ "575": 423685120.0,
+ "576": 423685120.0,
+ "577": 423685120.0,
+ "578": 423685120.0,
+ "579": 423685120.0,
+ "580": 423685120.0,
+ "581": 423685120.0,
+ "582": 423685120.0,
+ "583": 423685120.0,
+ "584": 423685120.0,
+ "585": 423685120.0,
+ "586": 423685120.0,
+ "587": 423685120.0,
+ "588": 423685120.0,
+ "589": 423685120.0,
+ "590": 423685120.0,
+ "591": 423685120.0,
+ "592": 423685120.0,
+ "593": 423685120.0,
+ "594": 423685120.0,
+ "595": 423685120.0,
+ "596": 423685120.0,
+ "597": 423685120.0,
+ "598": 423685120.0,
+ "599": 423685120.0,
+ "600": 423685120.0,
+ "601": 423685120.0,
+ "602": 423685120.0,
+ "603": 423685120.0,
+ "604": 423685120.0,
+ "605": 423685120.0,
+ "606": 423685120.0,
+ "607": 423685120.0,
+ "608": 423685120.0,
+ "609": 423685120.0,
+ "610": 423685120.0,
+ "611": 423685120.0,
+ "612": 423685120.0,
+ "613": 423685120.0,
+ "614": 423685120.0,
+ "615": 423685120.0,
+ "616": 423685120.0,
+ "617": 423685120.0,
+ "618": 423685120.0,
+ "619": 423685120.0,
+ "620": 423685120.0,
+ "621": 423685120.0,
+ "622": 423685120.0,
+ "623": 423685120.0,
+ "624": 423685120.0,
+ "625": 423685120.0,
+ "626": 423685120.0,
+ "627": 423685120.0,
+ "628": 423685120.0,
+ "629": 423685120.0,
+ "630": 423685120.0,
+ "631": 423685120.0,
+ "632": 423685120.0,
+ "633": 423685120.0,
+ "634": 423685120.0,
+ "635": 423685120.0,
+ "636": 423685120.0,
+ "637": 423685120.0,
+ "638": 423685120.0,
+ "639": 423685120.0,
+ "640": 423685120.0,
+ "641": 423685120.0,
+ "642": 423685120.0,
+ "643": 423685120.0,
+ "644": 423685120.0,
+ "645": 423685120.0,
+ "646": 423685120.0,
+ "647": 423685120.0,
+ "648": 423685120.0,
+ "649": 423685120.0,
+ "650": 423685120.0,
+ "651": 423685120.0,
+ "652": 423685120.0,
+ "653": 423685120.0,
+ "654": 423685120.0,
+ "655": 423685120.0,
+ "656": 423685120.0,
+ "657": 423685120.0,
+ "658": 423685120.0,
+ "659": 423685120.0,
+ "660": 423685120.0,
+ "661": 423685120.0,
+ "662": 423685120.0,
+ "663": 423685120.0,
+ "664": 423685120.0,
+ "665": 423685120.0,
+ "666": 423685120.0,
+ "667": 423685120.0,
+ "668": 423685120.0,
+ "669": 423685120.0,
+ "670": 423685120.0,
+ "671": 423685120.0,
+ "672": 423685120.0,
+ "673": 423685120.0,
+ "674": 423685120.0,
+ "675": 423685120.0,
+ "676": 423685120.0,
+ "677": 423685120.0,
+ "678": 423685120.0,
+ "679": 423685120.0,
+ "680": 423685120.0,
+ "681": 423685120.0,
+ "682": 423685120.0,
+ "683": 423685120.0,
+ "684": 423685120.0,
+ "685": 423685120.0,
+ "686": 423685120.0,
+ "687": 423685120.0,
+ "688": 423685120.0,
+ "689": 423685120.0,
+ "690": 423685120.0,
+ "691": 423685120.0,
+ "692": 423685120.0,
+ "693": 423685120.0,
+ "694": 423685120.0,
+ "695": 423685120.0,
+ "696": 423685120.0,
+ "697": 423685120.0,
+ "698": 423685120.0,
+ "699": 423685120.0,
+ "700": 423685120.0,
+ "701": 423685120.0,
+ "702": 423685120.0,
+ "703": 423685120.0,
+ "704": 423685120.0,
+ "705": 423685120.0,
+ "706": 423685120.0,
+ "707": 423685120.0,
+ "708": 423685120.0,
+ "709": 423685120.0,
+ "710": 423685120.0,
+ "711": 423685120.0,
+ "712": 423685120.0,
+ "713": 423685120.0,
+ "714": 423685120.0,
+ "715": 423685120.0,
+ "716": 423685120.0,
+ "717": 423685120.0,
+ "718": 423685120.0,
+ "719": 423685120.0,
+ "720": 423685120.0,
+ "721": 423685120.0,
+ "722": 423685120.0,
+ "723": 423685120.0,
+ "724": 423685120.0,
+ "725": 423685120.0,
+ "726": 423685120.0,
+ "727": 423685120.0,
+ "728": 423685120.0,
+ "729": 423685120.0,
+ "730": 423685120.0,
+ "731": 423685120.0,
+ "732": 423685120.0,
+ "733": 423685120.0,
+ "734": 423685120.0,
+ "735": 423685120.0,
+ "736": 423685120.0,
+ "737": 423685120.0,
+ "738": 423685120.0,
+ "739": 423685120.0,
+ "740": 423685120.0,
+ "741": 423685120.0,
+ "742": 423685120.0,
+ "743": 423685120.0,
+ "744": 423685120.0,
+ "745": 423685120.0,
+ "746": 423685120.0,
+ "747": 423685120.0,
+ "748": 423685120.0,
+ "749": 423685120.0,
+ "750": 423685120.0,
+ "751": 423685120.0,
+ "752": 423685120.0,
+ "753": 423685120.0,
+ "754": 423685120.0,
+ "755": 423685120.0,
+ "756": 423685120.0,
+ "757": 423685120.0,
+ "758": 423685120.0,
+ "759": 423685120.0,
+ "760": 423685120.0,
+ "761": 423685120.0,
+ "762": 423685120.0,
+ "763": 423685120.0,
+ "764": 423685120.0,
+ "765": 423685120.0,
+ "766": 423685120.0,
+ "767": 423685120.0,
+ "768": 423685120.0,
+ "769": 423685120.0,
+ "770": 423685120.0,
+ "771": 423685120.0,
+ "772": 423685120.0,
+ "773": 423685120.0,
+ "774": 423685120.0,
+ "775": 423685120.0,
+ "776": 423685120.0,
+ "777": 423685120.0,
+ "778": 423685120.0,
+ "779": 423685120.0,
+ "780": 423685120.0,
+ "781": 423685120.0,
+ "782": 423685120.0,
+ "783": 423685120.0,
+ "784": 423685120.0,
+ "785": 423685120.0,
+ "786": 423685120.0,
+ "787": 423685120.0,
+ "788": 423685120.0,
+ "789": 423685120.0,
+ "790": 423685120.0,
+ "791": 423685120.0,
+ "792": 423685120.0,
+ "793": 423685120.0,
+ "794": 423685120.0,
+ "795": 423685120.0,
+ "796": 423685120.0,
+ "797": 423685120.0,
+ "798": 423685120.0,
+ "799": 423685120.0,
+ "800": 423685120.0,
+ "801": 423685120.0,
+ "802": 423685120.0,
+ "803": 423685120.0,
+ "804": 423685120.0,
+ "805": 423685120.0,
+ "806": 423685120.0,
+ "807": 423685120.0,
+ "808": 423685120.0,
+ "809": 423685120.0,
+ "810": 423685120.0,
+ "811": 423685120.0,
+ "812": 423685120.0,
+ "813": 423685120.0,
+ "814": 423685120.0,
+ "815": 423685120.0,
+ "816": 423685120.0,
+ "817": 423685120.0,
+ "818": 423685120.0,
+ "819": 423685120.0,
+ "820": 423685120.0,
+ "821": 423685120.0,
+ "822": 423685120.0,
+ "823": 423685120.0,
+ "824": 423685120.0,
+ "825": 423685120.0,
+ "826": 423685120.0,
+ "827": 423685120.0,
+ "828": 423685120.0,
+ "829": 423685120.0,
+ "830": 423685120.0,
+ "831": 423685120.0,
+ "832": 423685120.0,
+ "833": 423685120.0,
+ "834": 423685120.0,
+ "835": 423685120.0,
+ "836": 423685120.0,
+ "837": 423685120.0,
+ "838": 423685120.0,
+ "839": 423685120.0,
+ "840": 423685120.0,
+ "841": 423685120.0,
+ "842": 423685120.0,
+ "843": 423685120.0,
+ "844": 423685120.0,
+ "845": 423685120.0,
+ "846": 423685120.0,
+ "847": 423685120.0,
+ "848": 423685120.0,
+ "849": 423685120.0,
+ "850": 423685120.0,
+ "851": 423685120.0,
+ "852": 423685120.0,
+ "853": 423685120.0,
+ "854": 423685120.0,
+ "855": 423685120.0,
+ "856": 423685120.0,
+ "857": 423685120.0,
+ "858": 423685120.0,
+ "859": 423685120.0,
+ "860": 423685120.0,
+ "861": 423685120.0,
+ "862": 423685120.0,
+ "863": 423685120.0,
+ "864": 423685120.0,
+ "865": 423685120.0,
+ "866": 423685120.0,
+ "867": 423685120.0,
+ "868": 423685120.0,
+ "869": 423685120.0,
+ "870": 423685120.0,
+ "871": 423685120.0,
+ "872": 423685120.0,
+ "873": 423685120.0,
+ "874": 423685120.0,
+ "875": 423685120.0,
+ "876": 423685120.0,
+ "877": 423685120.0,
+ "878": 423685120.0,
+ "879": 423685120.0,
+ "880": 423685120.0,
+ "881": 423685120.0,
+ "882": 423685120.0,
+ "883": 423685120.0,
+ "884": 423685120.0,
+ "885": 423685120.0,
+ "886": 423685120.0,
+ "887": 423685120.0,
+ "888": 423685120.0,
+ "889": 423685120.0,
+ "890": 423685120.0,
+ "891": 423685120.0,
+ "892": 423685120.0,
+ "893": 423685120.0,
+ "894": 423685120.0,
+ "895": 423685120.0,
+ "896": 423685120.0,
+ "897": 423685120.0,
+ "898": 423685120.0,
+ "899": 423685120.0,
+ "900": 423685120.0,
+ "901": 423685120.0,
+ "902": 423685120.0,
+ "903": 423685120.0,
+ "904": 423685120.0,
+ "905": 423685120.0,
+ "906": 423685120.0,
+ "907": 423685120.0,
+ "908": 423685120.0,
+ "909": 423685120.0,
+ "910": 423685120.0,
+ "911": 423685120.0,
+ "912": 423685120.0,
+ "913": 423685120.0,
+ "914": 423685120.0,
+ "915": 423685120.0,
+ "916": 423685120.0,
+ "917": 423685120.0,
+ "918": 423685120.0,
+ "919": 423685120.0,
+ "920": 423685120.0,
+ "921": 423685120.0,
+ "922": 423685120.0,
+ "923": 423685120.0,
+ "924": 423685120.0,
+ "925": 423685120.0,
+ "926": 423685120.0,
+ "927": 423685120.0,
+ "928": 423685120.0,
+ "929": 423685120.0,
+ "930": 423685120.0,
+ "931": 423685120.0,
+ "932": 423685120.0,
+ "933": 423685120.0,
+ "934": 423685120.0,
+ "935": 423685120.0,
+ "936": 423685120.0,
+ "937": 423685120.0,
+ "938": 423685120.0,
+ "939": 423685120.0,
+ "940": 423685120.0,
+ "941": 423685120.0,
+ "942": 423685120.0,
+ "943": 423685120.0,
+ "944": 423685120.0,
+ "945": 423685120.0,
+ "946": 423685120.0,
+ "947": 423685120.0,
+ "948": 423685120.0,
+ "949": 423685120.0,
+ "950": 423685120.0,
+ "951": 423685120.0,
+ "952": 423685120.0,
+ "953": 423685120.0,
+ "954": 423685120.0,
+ "955": 423685120.0,
+ "956": 423685120.0,
+ "957": 423685120.0,
+ "958": 423685120.0,
+ "959": 423685120.0,
+ "960": 423685120.0,
+ "961": 423685120.0,
+ "962": 423685120.0,
+ "963": 423685120.0,
+ "964": 423685120.0,
+ "965": 423685120.0,
+ "966": 423685120.0,
+ "967": 423685120.0,
+ "968": 423685120.0,
+ "969": 423685120.0,
+ "970": 423685120.0,
+ "971": 423685120.0,
+ "972": 423685120.0,
+ "973": 423685120.0,
+ "974": 423685120.0,
+ "975": 423685120.0,
+ "976": 423685120.0,
+ "977": 423685120.0,
+ "978": 423685120.0,
+ "979": 423685120.0,
+ "980": 423685120.0,
+ "981": 423685120.0,
+ "982": 423685120.0,
+ "983": 423685120.0,
+ "984": 423685120.0,
+ "985": 423685120.0,
+ "986": 423685120.0,
+ "987": 423685120.0,
+ "988": 423685120.0,
+ "989": 423685120.0,
+ "990": 423685120.0,
+ "991": 423685120.0,
+ "992": 423685120.0,
+ "993": 423685120.0,
+ "994": 423685120.0,
+ "995": 423685120.0,
+ "996": 423685120.0,
+ "997": 423685120.0,
+ "998": 423685120.0,
+ "999": 423685120.0,
+ "1000": 423685120.0,
+ "1001": 433339392.0,
+ "1002": 433339392.0,
+ "1003": 433339392.0,
+ "1004": 433339392.0,
+ "1005": 433339392.0,
+ "1006": 433339392.0,
+ "1007": 433339392.0,
+ "1008": 433339392.0,
+ "1009": 433339392.0,
+ "1010": 433339392.0,
+ "1011": 433339392.0,
+ "1012": 433339392.0,
+ "1013": 433339392.0,
+ "1014": 433339392.0,
+ "1015": 433339392.0,
+ "1016": 433339392.0,
+ "1017": 433339392.0,
+ "1018": 433339392.0,
+ "1019": 433339392.0,
+ "1020": 433339392.0,
+ "1021": 433339392.0,
+ "1022": 433339392.0,
+ "1023": 433339392.0,
+ "1024": 433339392.0,
+ "1025": 433339392.0,
+ "1026": 433339392.0,
+ "1027": 433339392.0,
+ "1028": 433339392.0,
+ "1029": 433339392.0,
+ "1030": 433339392.0,
+ "1031": 433339392.0,
+ "1032": 433339392.0,
+ "1033": 433339392.0,
+ "1034": 433339392.0,
+ "1035": 433339392.0,
+ "1036": 433339392.0,
+ "1037": 433339392.0,
+ "1038": 433339392.0,
+ "1039": 433339392.0,
+ "1040": 433339392.0,
+ "1041": 433339392.0,
+ "1042": 433339392.0,
+ "1043": 433339392.0,
+ "1044": 433339392.0,
+ "1045": 433339392.0,
+ "1046": 433339392.0,
+ "1047": 433339392.0,
+ "1048": 433339392.0,
+ "1049": 433339392.0,
+ "1050": 433339392.0,
+ "1051": 433339392.0,
+ "1052": 433339392.0,
+ "1053": 433339392.0,
+ "1054": 433339392.0,
+ "1055": 433339392.0,
+ "1056": 433339392.0,
+ "1057": 433339392.0,
+ "1058": 433339392.0,
+ "1059": 433339392.0,
+ "1060": 433339392.0,
+ "1061": 433339392.0,
+ "1062": 433339392.0,
+ "1063": 433339392.0,
+ "1064": 433339392.0,
+ "1065": 433339392.0,
+ "1066": 433339392.0,
+ "1067": 433339392.0,
+ "1068": 433339392.0,
+ "1069": 433339392.0,
+ "1070": 433339392.0,
+ "1071": 433339392.0,
+ "1072": 433339392.0,
+ "1073": 433339392.0,
+ "1074": 433339392.0,
+ "1075": 433339392.0,
+ "1076": 433339392.0,
+ "1077": 433339392.0,
+ "1078": 433339392.0,
+ "1079": 433339392.0,
+ "1080": 433339392.0,
+ "1081": 433339392.0,
+ "1082": 433339392.0,
+ "1083": 433339392.0,
+ "1084": 433339392.0,
+ "1085": 433339392.0,
+ "1086": 433339392.0,
+ "1087": 433339392.0,
+ "1088": 433339392.0,
+ "1089": 433339392.0,
+ "1090": 433339392.0,
+ "1091": 433339392.0,
+ "1092": 433339392.0,
+ "1093": 433339392.0,
+ "1094": 433339392.0,
+ "1095": 433339392.0,
+ "1096": 433339392.0,
+ "1097": 433339392.0,
+ "1098": 433339392.0,
+ "1099": 433339392.0,
+ "1100": 433339392.0,
+ "1101": 433339392.0,
+ "1102": 433339392.0,
+ "1103": 433339392.0,
+ "1104": 433339392.0,
+ "1105": 433339392.0,
+ "1106": 433339392.0,
+ "1107": 433339392.0,
+ "1108": 433339392.0,
+ "1109": 433339392.0,
+ "1110": 433339392.0,
+ "1111": 433339392.0,
+ "1112": 433339392.0,
+ "1113": 433339392.0,
+ "1114": 433339392.0,
+ "1115": 433339392.0,
+ "1116": 433339392.0,
+ "1117": 433339392.0,
+ "1118": 433339392.0,
+ "1119": 433339392.0,
+ "1120": 433339392.0,
+ "1121": 433339392.0,
+ "1122": 433339392.0,
+ "1123": 433339392.0,
+ "1124": 433339392.0,
+ "1125": 433339392.0,
+ "1126": 433339392.0,
+ "1127": 433339392.0,
+ "1128": 433339392.0,
+ "1129": 433339392.0,
+ "1130": 433339392.0,
+ "1131": 433339392.0,
+ "1132": 433339392.0,
+ "1133": 433339392.0,
+ "1134": 433339392.0,
+ "1135": 433339392.0,
+ "1136": 433339392.0,
+ "1137": 433339392.0,
+ "1138": 433339392.0,
+ "1139": 433339392.0,
+ "1140": 433339392.0,
+ "1141": 433339392.0,
+ "1142": 433339392.0,
+ "1143": 433339392.0,
+ "1144": 433339392.0,
+ "1145": 433339392.0,
+ "1146": 433339392.0,
+ "1147": 433339392.0,
+ "1148": 433339392.0,
+ "1149": 433339392.0,
+ "1150": 433339392.0,
+ "1151": 433339392.0,
+ "1152": 433339392.0,
+ "1153": 433339392.0,
+ "1154": 433339392.0,
+ "1155": 433339392.0,
+ "1156": 433339392.0,
+ "1157": 433339392.0,
+ "1158": 433339392.0,
+ "1159": 433339392.0,
+ "1160": 433339392.0,
+ "1161": 433339392.0,
+ "1162": 433339392.0,
+ "1163": 433339392.0,
+ "1164": 433339392.0,
+ "1165": 433339392.0,
+ "1166": 433339392.0,
+ "1167": 433339392.0,
+ "1168": 433339392.0,
+ "1169": 433339392.0,
+ "1170": 433339392.0,
+ "1171": 433339392.0,
+ "1172": 433339392.0,
+ "1173": 433339392.0,
+ "1174": 433339392.0,
+ "1175": 433339392.0,
+ "1176": 433339392.0,
+ "1177": 433339392.0,
+ "1178": 433339392.0,
+ "1179": 433339392.0,
+ "1180": 433339392.0,
+ "1181": 433339392.0,
+ "1182": 433339392.0,
+ "1183": 433339392.0,
+ "1184": 433339392.0,
+ "1185": 433339392.0,
+ "1186": 433339392.0,
+ "1187": 433339392.0,
+ "1188": 433339392.0,
+ "1189": 433339392.0,
+ "1190": 433339392.0,
+ "1191": 433339392.0,
+ "1192": 433339392.0,
+ "1193": 433339392.0,
+ "1194": 433339392.0,
+ "1195": 433339392.0,
+ "1196": 433339392.0,
+ "1197": 433339392.0,
+ "1198": 433339392.0,
+ "1199": 433339392.0,
+ "1200": 433339392.0,
+ "1201": 433339392.0,
+ "1202": 433339392.0,
+ "1203": 433339392.0,
+ "1204": 433339392.0,
+ "1205": 433339392.0,
+ "1206": 433339392.0,
+ "1207": 433339392.0,
+ "1208": 433339392.0,
+ "1209": 433339392.0,
+ "1210": 433339392.0,
+ "1211": 433339392.0,
+ "1212": 433339392.0,
+ "1213": 433339392.0,
+ "1214": 433339392.0,
+ "1215": 433339392.0,
+ "1216": 433339392.0,
+ "1217": 433339392.0,
+ "1218": 433339392.0,
+ "1219": 433339392.0,
+ "1220": 433339392.0,
+ "1221": 433339392.0,
+ "1222": 433339392.0,
+ "1223": 433339392.0,
+ "1224": 433339392.0,
+ "1225": 433339392.0,
+ "1226": 433339392.0,
+ "1227": 433339392.0,
+ "1228": 433339392.0,
+ "1229": 433339392.0,
+ "1230": 433339392.0,
+ "1231": 433339392.0,
+ "1232": 433339392.0,
+ "1233": 433339392.0,
+ "1234": 433339392.0,
+ "1235": 433339392.0,
+ "1236": 433339392.0,
+ "1237": 433339392.0,
+ "1238": 433339392.0,
+ "1239": 433339392.0,
+ "1240": 433339392.0,
+ "1241": 433339392.0,
+ "1242": 433339392.0,
+ "1243": 433339392.0,
+ "1244": 433339392.0,
+ "1245": 433339392.0,
+ "1246": 433339392.0,
+ "1247": 433339392.0,
+ "1248": 433339392.0,
+ "1249": 433339392.0,
+ "1250": 433339392.0,
+ "1251": 433339392.0,
+ "1252": 433339392.0,
+ "1253": 433339392.0,
+ "1254": 433339392.0,
+ "1255": 433339392.0,
+ "1256": 433339392.0,
+ "1257": 433339392.0,
+ "1258": 433339392.0,
+ "1259": 433339392.0,
+ "1260": 433339392.0,
+ "1261": 433339392.0,
+ "1262": 433339392.0,
+ "1263": 433339392.0,
+ "1264": 433339392.0,
+ "1265": 433339392.0,
+ "1266": 433339392.0,
+ "1267": 433339392.0,
+ "1268": 433339392.0,
+ "1269": 433339392.0,
+ "1270": 433339392.0,
+ "1271": 433339392.0,
+ "1272": 433339392.0,
+ "1273": 433339392.0,
+ "1274": 433339392.0,
+ "1275": 433339392.0,
+ "1276": 433339392.0,
+ "1277": 433339392.0,
+ "1278": 433339392.0,
+ "1279": 433339392.0,
+ "1280": 433339392.0,
+ "1281": 433339392.0,
+ "1282": 433339392.0,
+ "1283": 433339392.0,
+ "1284": 433339392.0,
+ "1285": 433339392.0,
+ "1286": 433339392.0,
+ "1287": 433339392.0,
+ "1288": 433339392.0,
+ "1289": 433339392.0,
+ "1290": 433339392.0,
+ "1291": 433339392.0,
+ "1292": 433339392.0,
+ "1293": 433339392.0,
+ "1294": 433339392.0,
+ "1295": 433339392.0,
+ "1296": 433339392.0,
+ "1297": 433339392.0,
+ "1298": 433339392.0,
+ "1299": 433339392.0,
+ "1300": 433339392.0,
+ "1301": 433339392.0,
+ "1302": 433339392.0,
+ "1303": 433339392.0,
+ "1304": 433339392.0,
+ "1305": 433339392.0,
+ "1306": 433339392.0,
+ "1307": 433339392.0,
+ "1308": 433339392.0,
+ "1309": 433339392.0,
+ "1310": 433339392.0,
+ "1311": 433339392.0,
+ "1312": 433339392.0,
+ "1313": 433339392.0,
+ "1314": 433339392.0,
+ "1315": 433339392.0,
+ "1316": 433339392.0,
+ "1317": 433339392.0,
+ "1318": 433339392.0,
+ "1319": 433339392.0,
+ "1320": 433339392.0,
+ "1321": 433339392.0,
+ "1322": 433339392.0,
+ "1323": 433339392.0,
+ "1324": 433339392.0,
+ "1325": 433339392.0,
+ "1326": 433339392.0,
+ "1327": 433339392.0,
+ "1328": 433339392.0,
+ "1329": 433339392.0,
+ "1330": 433339392.0,
+ "1331": 433339392.0,
+ "1332": 433339392.0,
+ "1333": 433339392.0,
+ "1334": 433339392.0,
+ "1335": 433339392.0,
+ "1336": 433339392.0,
+ "1337": 433339392.0,
+ "1338": 433339392.0,
+ "1339": 433339392.0,
+ "1340": 433339392.0,
+ "1341": 433339392.0,
+ "1342": 433339392.0,
+ "1343": 433339392.0,
+ "1344": 433339392.0,
+ "1345": 433339392.0,
+ "1346": 433339392.0,
+ "1347": 433339392.0,
+ "1348": 433339392.0,
+ "1349": 433339392.0,
+ "1350": 433339392.0,
+ "1351": 433339392.0,
+ "1352": 433339392.0,
+ "1353": 433339392.0,
+ "1354": 433339392.0,
+ "1355": 433339392.0,
+ "1356": 433339392.0,
+ "1357": 433339392.0,
+ "1358": 433339392.0,
+ "1359": 433339392.0,
+ "1360": 433339392.0,
+ "1361": 433339392.0,
+ "1362": 433339392.0,
+ "1363": 433339392.0,
+ "1364": 433339392.0,
+ "1365": 433339392.0,
+ "1366": 433339392.0,
+ "1367": 433339392.0,
+ "1368": 433339392.0,
+ "1369": 433339392.0,
+ "1370": 433339392.0,
+ "1371": 433339392.0,
+ "1372": 433339392.0,
+ "1373": 433339392.0,
+ "1374": 433339392.0,
+ "1375": 433339392.0,
+ "1376": 433339392.0,
+ "1377": 433339392.0,
+ "1378": 433339392.0,
+ "1379": 433339392.0,
+ "1380": 433339392.0,
+ "1381": 433339392.0,
+ "1382": 433339392.0,
+ "1383": 433339392.0,
+ "1384": 433339392.0,
+ "1385": 433339392.0,
+ "1386": 433339392.0,
+ "1387": 433339392.0,
+ "1388": 433339392.0,
+ "1389": 433339392.0,
+ "1390": 433339392.0,
+ "1391": 433339392.0,
+ "1392": 433339392.0,
+ "1393": 433339392.0,
+ "1394": 433339392.0,
+ "1395": 433339392.0,
+ "1396": 433339392.0,
+ "1397": 433339392.0,
+ "1398": 433339392.0,
+ "1399": 433339392.0,
+ "1400": 433339392.0,
+ "1401": 433339392.0,
+ "1402": 433339392.0,
+ "1403": 433339392.0,
+ "1404": 433339392.0,
+ "1405": 433339392.0,
+ "1406": 433339392.0,
+ "1407": 433339392.0,
+ "1408": 433339392.0,
+ "1409": 433339392.0,
+ "1410": 433339392.0,
+ "1411": 433339392.0,
+ "1412": 433339392.0,
+ "1413": 433339392.0,
+ "1414": 433339392.0,
+ "1415": 433339392.0,
+ "1416": 433339392.0,
+ "1417": 433339392.0,
+ "1418": 433339392.0,
+ "1419": 433339392.0,
+ "1420": 433339392.0,
+ "1421": 433339392.0,
+ "1422": 433339392.0,
+ "1423": 433339392.0,
+ "1424": 433339392.0,
+ "1425": 433339392.0,
+ "1426": 433339392.0,
+ "1427": 433339392.0,
+ "1428": 433339392.0,
+ "1429": 433339392.0,
+ "1430": 433339392.0,
+ "1431": 433339392.0,
+ "1432": 433339392.0,
+ "1433": 433339392.0,
+ "1434": 433339392.0,
+ "1435": 433339392.0,
+ "1436": 433339392.0,
+ "1437": 433339392.0,
+ "1438": 433339392.0,
+ "1439": 433339392.0,
+ "1440": 433339392.0,
+ "1441": 433339392.0,
+ "1442": 433339392.0,
+ "1443": 433339392.0,
+ "1444": 433339392.0,
+ "1445": 433339392.0,
+ "1446": 433339392.0,
+ "1447": 433339392.0,
+ "1448": 433339392.0,
+ "1449": 433339392.0,
+ "1450": 433339392.0,
+ "1451": 433339392.0,
+ "1452": 433339392.0,
+ "1453": 433339392.0,
+ "1454": 433339392.0,
+ "1455": 433339392.0,
+ "1456": 433339392.0,
+ "1457": 433339392.0,
+ "1458": 433339392.0,
+ "1459": 433339392.0,
+ "1460": 433339392.0,
+ "1461": 433339392.0,
+ "1462": 433339392.0,
+ "1463": 433339392.0,
+ "1464": 433339392.0,
+ "1465": 433339392.0,
+ "1466": 433339392.0,
+ "1467": 433339392.0,
+ "1468": 433339392.0,
+ "1469": 433339392.0,
+ "1470": 433339392.0,
+ "1471": 433339392.0,
+ "1472": 433339392.0,
+ "1473": 433339392.0,
+ "1474": 433339392.0,
+ "1475": 433339392.0,
+ "1476": 433339392.0,
+ "1477": 433339392.0,
+ "1478": 433339392.0,
+ "1479": 433339392.0,
+ "1480": 433339392.0,
+ "1481": 433339392.0,
+ "1482": 433339392.0,
+ "1483": 433339392.0,
+ "1484": 433339392.0,
+ "1485": 433339392.0,
+ "1486": 433339392.0,
+ "1487": 433339392.0,
+ "1488": 433339392.0,
+ "1489": 433339392.0,
+ "1490": 433339392.0,
+ "1491": 433339392.0,
+ "1492": 433339392.0,
+ "1493": 433339392.0,
+ "1494": 433339392.0,
+ "1495": 433339392.0,
+ "1496": 433339392.0,
+ "1497": 433339392.0,
+ "1498": 433339392.0,
+ "1499": 433339392.0,
+ "1500": 433339392.0,
+ "1501": 433339392.0,
+ "1502": 433339392.0,
+ "1503": 433339392.0,
+ "1504": 433339392.0,
+ "1505": 433339392.0,
+ "1506": 433339392.0,
+ "1507": 433339392.0,
+ "1508": 433339392.0,
+ "1509": 433339392.0,
+ "1510": "nan",
+ "1511": "nan",
+ "1512": "nan",
+ "1513": "nan",
+ "1514": "nan",
+ "1515": "nan",
+ "1516": "nan",
+ "1517": "nan",
+ "1518": "nan",
+ "1519": "nan",
+ "1520": "nan",
+ "1521": "nan",
+ "1522": "nan",
+ "1523": "nan",
+ "1524": "nan",
+ "1525": "nan",
+ "1526": "nan",
+ "1527": "nan",
+ "1528": "nan",
+ "1529": "nan",
+ "1530": "nan",
+ "1531": "nan",
+ "1532": "nan",
+ "1533": "nan",
+ "1534": "nan",
+ "1535": "nan",
+ "1536": "nan",
+ "1537": "nan",
+ "1538": "nan",
+ "1539": "nan",
+ "1540": "nan",
+ "1541": "nan",
+ "1542": "nan",
+ "1543": "nan",
+ "1544": "nan",
+ "1545": "nan",
+ "1546": "nan",
+ "1547": "nan",
+ "1548": "nan",
+ "1549": "nan",
+ "1550": "nan",
+ "1551": "nan",
+ "1552": "nan",
+ "1553": "nan",
+ "1554": "nan",
+ "1555": "nan",
+ "1556": "nan",
+ "1557": "nan",
+ "1558": "nan",
+ "1559": "nan",
+ "1560": "nan",
+ "1561": "nan",
+ "1562": "nan",
+ "1563": "nan",
+ "1564": "nan",
+ "1565": "nan",
+ "1566": "nan",
+ "1567": "nan",
+ "1568": "nan",
+ "1569": "nan",
+ "1570": "nan",
+ "1571": "nan",
+ "1572": "nan",
+ "1573": "nan",
+ "1574": "nan",
+ "1575": "nan",
+ "1576": "nan",
+ "1577": "nan",
+ "1578": "nan",
+ "1579": "nan",
+ "1580": "nan",
+ "1581": "nan",
+ "1582": "nan",
+ "1583": "nan",
+ "1584": "nan",
+ "1585": "nan",
+ "1586": "nan",
+ "1587": "nan",
+ "1588": "nan",
+ "1589": "nan",
+ "1590": "nan",
+ "1591": "nan",
+ "1592": "nan",
+ "1593": "nan",
+ "1594": "nan",
+ "1595": "nan",
+ "1596": "nan",
+ "1597": "nan",
+ "1598": "nan",
+ "1599": "nan",
+ "1600": "nan",
+ "1601": "nan",
+ "1602": "nan",
+ "1603": "nan",
+ "1604": "nan",
+ "1605": "nan",
+ "1606": "nan",
+ "1607": "nan",
+ "1608": "nan",
+ "1609": "nan",
+ "1610": "nan",
+ "1611": "nan",
+ "1612": "nan",
+ "1613": "nan",
+ "1614": "nan",
+ "1615": "nan",
+ "1616": "nan",
+ "1617": "nan",
+ "1618": "nan",
+ "1619": "nan",
+ "1620": "nan",
+ "1621": "nan",
+ "1622": "nan",
+ "1623": "nan",
+ "1624": "nan",
+ "1625": "nan",
+ "1626": "nan",
+ "1627": "nan",
+ "1628": "nan",
+ "1629": "nan",
+ "1630": "nan",
+ "1631": "nan",
+ "1632": "nan",
+ "1633": "nan",
+ "1634": "nan",
+ "1635": "nan",
+ "1636": "nan",
+ "1637": "nan",
+ "1638": "nan",
+ "1639": "nan",
+ "1640": "nan",
+ "1641": "nan",
+ "1642": "nan",
+ "1643": "nan",
+ "1644": "nan",
+ "1645": "nan",
+ "1646": "nan",
+ "1647": "nan",
+ "1648": "nan",
+ "1649": "nan",
+ "1650": "nan",
+ "1651": "nan",
+ "1652": "nan",
+ "1653": "nan",
+ "1654": "nan",
+ "1655": "nan",
+ "1656": "nan",
+ "1657": "nan",
+ "1658": "nan",
+ "1659": "nan",
+ "1660": "nan",
+ "1661": "nan",
+ "1662": "nan",
+ "1663": "nan",
+ "1664": "nan",
+ "1665": "nan",
+ "1666": "nan",
+ "1667": "nan",
+ "1668": "nan",
+ "1669": "nan",
+ "1670": "nan",
+ "1671": "nan",
+ "1672": "nan",
+ "1673": "nan",
+ "1674": "nan",
+ "1675": "nan",
+ "1676": "nan",
+ "1677": "nan",
+ "1678": "nan",
+ "1679": "nan",
+ "1680": "nan",
+ "1681": "nan",
+ "1682": "nan",
+ "1683": "nan",
+ "1684": "nan",
+ "1685": "nan",
+ "1686": "nan",
+ "1687": "nan",
+ "1688": "nan",
+ "1689": "nan",
+ "1690": "nan",
+ "1691": "nan",
+ "1692": "nan",
+ "1693": "nan",
+ "1694": "nan",
+ "1695": "nan",
+ "1696": "nan",
+ "1697": "nan",
+ "1698": "nan",
+ "1699": "nan",
+ "1700": "nan",
+ "1701": "nan",
+ "1702": "nan",
+ "1703": "nan",
+ "1704": "nan",
+ "1705": "nan",
+ "1706": "nan",
+ "1707": "nan",
+ "1708": "nan",
+ "1709": "nan",
+ "1710": "nan",
+ "1711": "nan",
+ "1712": "nan",
+ "1713": "nan",
+ "1714": "nan",
+ "1715": "nan",
+ "1716": "nan",
+ "1717": "nan",
+ "1718": "nan",
+ "1719": "nan",
+ "1720": "nan",
+ "1721": "nan",
+ "1722": "nan",
+ "1723": "nan",
+ "1724": "nan",
+ "1725": "nan",
+ "1726": "nan",
+ "1727": "nan",
+ "1728": "nan",
+ "1729": "nan",
+ "1730": "nan",
+ "1731": "nan",
+ "1732": "nan",
+ "1733": "nan",
+ "1734": "nan",
+ "1735": "nan",
+ "1736": "nan",
+ "1737": "nan",
+ "1738": "nan",
+ "1739": "nan",
+ "1740": "nan",
+ "1741": "nan",
+ "1742": "nan",
+ "1743": "nan",
+ "1744": "nan",
+ "1745": "nan",
+ "1746": "nan",
+ "1747": "nan",
+ "1748": "nan",
+ "1749": "nan",
+ "1750": "nan",
+ "1751": "nan",
+ "1752": "nan",
+ "1753": "nan",
+ "1754": "nan",
+ "1755": "nan",
+ "1756": "nan",
+ "1757": "nan",
+ "1758": "nan",
+ "1759": "nan",
+ "1760": "nan",
+ "1761": "nan",
+ "1762": "nan",
+ "1763": "nan",
+ "1764": "nan",
+ "1765": "nan",
+ "1766": "nan",
+ "1767": "nan",
+ "1768": "nan",
+ "1769": "nan",
+ "1770": "nan",
+ "1771": "nan",
+ "1772": "nan",
+ "1773": "nan",
+ "1774": "nan",
+ "1775": "nan",
+ "1776": "nan",
+ "1777": "nan",
+ "1778": "nan",
+ "1779": "nan",
+ "1780": "nan",
+ "1781": "nan",
+ "1782": "nan",
+ "1783": "nan",
+ "1784": "nan",
+ "1785": "nan",
+ "1786": "nan",
+ "1787": "nan",
+ "1788": "nan",
+ "1789": "nan",
+ "1790": "nan",
+ "1791": "nan",
+ "1792": "nan",
+ "1793": "nan",
+ "1794": "nan",
+ "1795": "nan",
+ "1796": "nan",
+ "1797": "nan",
+ "1798": "nan",
+ "1799": "nan",
+ "1800": "nan",
+ "1801": "nan",
+ "1802": "nan",
+ "1803": "nan",
+ "1804": "nan",
+ "1805": "nan",
+ "1806": "nan",
+ "1807": "nan",
+ "1808": "nan",
+ "1809": "nan",
+ "1810": "nan",
+ "1811": "nan",
+ "1812": "nan",
+ "1813": "nan",
+ "1814": "nan",
+ "1815": "nan",
+ "1816": "nan",
+ "1817": "nan",
+ "1818": "nan",
+ "1819": "nan",
+ "1820": "nan",
+ "1821": "nan",
+ "1822": "nan",
+ "1823": "nan",
+ "1824": "nan",
+ "1825": "nan",
+ "1826": "nan",
+ "1827": "nan",
+ "1828": "nan",
+ "1829": "nan",
+ "1830": "nan",
+ "1831": "nan",
+ "1832": "nan",
+ "1833": "nan",
+ "1834": "nan",
+ "1835": "nan",
+ "1836": "nan",
+ "1837": "nan",
+ "1838": "nan",
+ "1839": "nan",
+ "1840": "nan",
+ "1841": "nan",
+ "1842": "nan",
+ "1843": "nan",
+ "1844": "nan",
+ "1845": "nan",
+ "1846": "nan",
+ "1847": "nan",
+ "1848": "nan",
+ "1849": "nan",
+ "1850": "nan",
+ "1851": "nan",
+ "1852": "nan",
+ "1853": "nan",
+ "1854": "nan",
+ "1855": "nan",
+ "1856": "nan",
+ "1857": "nan",
+ "1858": "nan",
+ "1859": "nan",
+ "1860": "nan",
+ "1861": "nan",
+ "1862": "nan",
+ "1863": "nan",
+ "1864": "nan",
+ "1865": "nan",
+ "1866": "nan",
+ "1867": "nan",
+ "1868": "nan",
+ "1869": "nan",
+ "1870": "nan",
+ "1871": "nan",
+ "1872": "nan",
+ "1873": "nan",
+ "1874": "nan",
+ "1875": "nan",
+ "1876": "nan",
+ "1877": "nan",
+ "1878": "nan",
+ "1879": "nan",
+ "1880": "nan",
+ "1881": "nan",
+ "1882": "nan",
+ "1883": "nan",
+ "1884": "nan",
+ "1885": "nan",
+ "1886": "nan",
+ "1887": "nan",
+ "1888": "nan",
+ "1889": "nan",
+ "1890": "nan",
+ "1891": "nan",
+ "1892": "nan",
+ "1893": "nan",
+ "1894": "nan",
+ "1895": "nan",
+ "1896": "nan",
+ "1897": "nan",
+ "1898": "nan",
+ "1899": "nan",
+ "1900": "nan",
+ "1901": "nan",
+ "1902": "nan",
+ "1903": "nan",
+ "1904": "nan",
+ "1905": "nan",
+ "1906": "nan",
+ "1907": "nan",
+ "1908": "nan",
+ "1909": "nan",
+ "1910": "nan",
+ "1911": "nan",
+ "1912": "nan",
+ "1913": "nan",
+ "1914": "nan",
+ "1915": "nan",
+ "1916": "nan",
+ "1917": "nan",
+ "1918": "nan",
+ "1919": "nan",
+ "1920": "nan",
+ "1921": "nan",
+ "1922": "nan",
+ "1923": "nan",
+ "1924": "nan",
+ "1925": "nan",
+ "1926": "nan",
+ "1927": "nan",
+ "1928": "nan",
+ "1929": "nan",
+ "1930": "nan",
+ "1931": "nan",
+ "1932": "nan",
+ "1933": "nan",
+ "1934": "nan",
+ "1935": "nan",
+ "1936": "nan",
+ "1937": "nan",
+ "1938": "nan",
+ "1939": "nan",
+ "1940": "nan",
+ "1941": "nan",
+ "1942": "nan",
+ "1943": "nan",
+ "1944": "nan",
+ "1945": "nan",
+ "1946": "nan",
+ "1947": "nan",
+ "1948": "nan",
+ "1949": "nan",
+ "1950": "nan",
+ "1951": "nan",
+ "1952": "nan",
+ "1953": "nan",
+ "1954": "nan",
+ "1955": "nan",
+ "1956": "nan",
+ "1957": "nan",
+ "1958": "nan",
+ "1959": "nan",
+ "1960": "nan",
+ "1961": "nan",
+ "1962": "nan",
+ "1963": "nan",
+ "1964": "nan",
+ "1965": "nan",
+ "1966": "nan",
+ "1967": "nan",
+ "1968": "nan",
+ "1969": "nan",
+ "1970": "nan",
+ "1971": "nan",
+ "1972": "nan",
+ "1973": "nan",
+ "1974": "nan",
+ "1975": "nan",
+ "1976": "nan",
+ "1977": "nan",
+ "1978": "nan",
+ "1979": "nan",
+ "1980": "nan",
+ "1981": "nan",
+ "1982": "nan",
+ "1983": "nan",
+ "1984": "nan",
+ "1985": "nan",
+ "1986": "nan",
+ "1987": "nan",
+ "1988": "nan",
+ "1989": "nan",
+ "1990": "nan",
+ "1991": "nan",
+ "1992": "nan",
+ "1993": "nan",
+ "1994": "nan",
+ "1995": "nan",
+ "1996": "nan",
+ "1997": "nan",
+ "1998": "nan",
+ "1999": "nan",
+ "2000": "nan"
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": "nan",
+ "2": 14.88549,
+ "3": 8.69619,
+ "4": 8.66801,
+ "5": 8.78105,
+ "6": 8.73761,
+ "7": 12.08244,
+ "8": 32.8806,
+ "9": 8.6462,
+ "10": 8.727,
+ "11": 8.73122,
+ "12": 8.7108,
+ "13": 9.09621,
+ "14": 8.68028,
+ "15": 8.78269,
+ "16": 8.80345,
+ "17": 18.27981,
+ "18": 10.70386,
+ "19": 9.18625,
+ "20": 9.47099,
+ "21": 8.7281,
+ "22": 9.38963,
+ "23": 8.78318,
+ "24": 9.03161,
+ "25": 8.72393,
+ "26": 9.49375,
+ "27": 8.75329,
+ "28": 8.7366,
+ "29": 8.76766,
+ "30": 9.6418,
+ "31": 8.69838,
+ "32": 8.76715,
+ "33": 8.70235,
+ "34": 8.68159,
+ "35": 8.7132,
+ "36": 8.76068,
+ "37": 8.75889,
+ "38": 8.79932,
+ "39": 8.73769,
+ "40": 8.76277,
+ "41": 8.66693,
+ "42": 8.74183,
+ "43": 8.82198,
+ "44": 9.72527,
+ "45": 12.41225,
+ "46": 9.21887,
+ "47": 9.12756,
+ "48": 8.83203,
+ "49": 8.79146,
+ "50": 8.70349,
+ "51": 9.08158,
+ "52": 8.80028,
+ "53": 8.81568,
+ "54": 8.85055,
+ "55": 8.83132,
+ "56": 8.75926,
+ "57": 9.15178,
+ "58": 8.76244,
+ "59": 8.66057,
+ "60": 9.71543,
+ "61": 9.17281,
+ "62": 9.38316,
+ "63": 8.68694,
+ "64": 9.42159,
+ "65": 8.738,
+ "66": 9.37624,
+ "67": 9.07383,
+ "68": 8.7961,
+ "69": 8.76434,
+ "70": 8.63752,
+ "71": 8.79046,
+ "72": 8.80796,
+ "73": 8.68886,
+ "74": 8.69061,
+ "75": 8.80439,
+ "76": 8.75022,
+ "77": 8.67152,
+ "78": 8.81391,
+ "79": 8.76662,
+ "80": 8.74896,
+ "81": 8.79652,
+ "82": 8.82042,
+ "83": 8.72903,
+ "84": 8.75297,
+ "85": 8.75637,
+ "86": 8.79287,
+ "87": 8.75317,
+ "88": 8.73331,
+ "89": 8.67978,
+ "90": 9.07194,
+ "91": 8.65478,
+ "92": 8.63544,
+ "93": 8.65299,
+ "94": 8.72437,
+ "95": 8.74669,
+ "96": 9.10307,
+ "97": 8.76066,
+ "98": 8.72012,
+ "99": 8.69641,
+ "100": 8.76717,
+ "101": 8.78195,
+ "102": 8.87336,
+ "103": 8.76907,
+ "104": 8.74696,
+ "105": 9.01553,
+ "106": 8.75137,
+ "107": 9.49272,
+ "108": 9.39877,
+ "109": 8.69497,
+ "110": 8.68579,
+ "111": 9.74289,
+ "112": 8.76714,
+ "113": 8.76017,
+ "114": 8.74042,
+ "115": 8.72684,
+ "116": 8.71649,
+ "117": 8.75458,
+ "118": 8.72401,
+ "119": 8.69683,
+ "120": 8.72789,
+ "121": 8.7142,
+ "122": 8.69725,
+ "123": 8.79298,
+ "124": 8.6575,
+ "125": 8.83224,
+ "126": 8.73115,
+ "127": 8.70489,
+ "128": 8.68012,
+ "129": 9.05502,
+ "130": 8.7168,
+ "131": 8.7272,
+ "132": 8.69669,
+ "133": 8.68589,
+ "134": 8.71827,
+ "135": 9.08775,
+ "136": 8.76889,
+ "137": 8.82858,
+ "138": 8.76736,
+ "139": 8.76956,
+ "140": 8.71062,
+ "141": 8.72338,
+ "142": 8.74247,
+ "143": 8.8624,
+ "144": 8.73634,
+ "145": 8.7752,
+ "146": 8.71779,
+ "147": 8.79815,
+ "148": 8.7082,
+ "149": 8.75763,
+ "150": 9.01071,
+ "151": 9.44462,
+ "152": 9.39655,
+ "153": 8.73316,
+ "154": 9.47092,
+ "155": 8.67235,
+ "156": 9.03047,
+ "157": 8.76787,
+ "158": 8.81281,
+ "159": 8.7031,
+ "160": 8.67715,
+ "161": 8.73739,
+ "162": 8.6842,
+ "163": 8.69407,
+ "164": 8.70446,
+ "165": 8.70913,
+ "166": 8.69067,
+ "167": 8.70301,
+ "168": 9.14593,
+ "169": 8.68595,
+ "170": 8.74162,
+ "171": 8.73719,
+ "172": 8.74286,
+ "173": 9.12678,
+ "174": 8.79363,
+ "175": 8.71981,
+ "176": 8.78718,
+ "177": 8.68481,
+ "178": 8.7927,
+ "179": 8.76997,
+ "180": 8.71354,
+ "181": 8.75287,
+ "182": 8.74123,
+ "183": 8.7706,
+ "184": 8.76504,
+ "185": 8.76494,
+ "186": 8.85619,
+ "187": 8.73957,
+ "188": 8.76342,
+ "189": 8.80181,
+ "190": 8.73625,
+ "191": 8.74549,
+ "192": 8.67977,
+ "193": 8.78924,
+ "194": 9.40521,
+ "195": 8.69255,
+ "196": 9.88449,
+ "197": 8.74418,
+ "198": 9.46227,
+ "199": 9.5559,
+ "200": 9.04738,
+ "201": 8.69412,
+ "202": 9.38011,
+ "203": 8.72418,
+ "204": 8.7347,
+ "205": 8.74077,
+ "206": 9.09819,
+ "207": 8.78635,
+ "208": 9.21057,
+ "209": 8.80899,
+ "210": 8.65269,
+ "211": 8.76685,
+ "212": 9.14207,
+ "213": 8.79878,
+ "214": 8.73326,
+ "215": 8.8295,
+ "216": 8.80676,
+ "217": 8.80315,
+ "218": 8.70347,
+ "219": 8.76191,
+ "220": 8.72336,
+ "221": 8.71731,
+ "222": 8.75249,
+ "223": 8.76231,
+ "224": 8.76,
+ "225": 8.78652,
+ "226": 8.64969,
+ "227": 8.70418,
+ "228": 8.72381,
+ "229": 8.78579,
+ "230": 8.74498,
+ "231": 8.85067,
+ "232": 8.77311,
+ "233": 8.77484,
+ "234": 8.84687,
+ "235": 8.72585,
+ "236": 8.70977,
+ "237": 8.65329,
+ "238": 9.41598,
+ "239": 8.82582,
+ "240": 9.9878,
+ "241": 8.74723,
+ "242": 9.48102,
+ "243": 9.04241,
+ "244": 8.76248,
+ "245": 9.13526,
+ "246": 8.84711,
+ "247": 8.7422,
+ "248": 8.61513,
+ "249": 8.72169,
+ "250": 9.1229,
+ "251": 8.79349,
+ "252": 8.76048,
+ "253": 8.7348,
+ "254": 8.76947,
+ "255": 8.70801,
+ "256": 8.70811,
+ "257": 8.77398,
+ "258": 8.74489,
+ "259": 8.73014,
+ "260": 8.74668,
+ "261": 8.71698,
+ "262": 8.65416,
+ "263": 8.67506,
+ "264": 8.75065,
+ "265": 8.73082,
+ "266": 8.79117,
+ "267": 8.78508,
+ "268": 8.73238,
+ "269": 8.73988,
+ "270": 8.77775,
+ "271": 8.76288,
+ "272": 8.72176,
+ "273": 8.72147,
+ "274": 8.77144,
+ "275": 8.86565,
+ "276": 8.79047,
+ "277": 8.63855,
+ "278": 8.69218,
+ "279": 8.72487,
+ "280": 8.65668,
+ "281": 9.04921,
+ "282": 8.75252,
+ "283": 9.37935,
+ "284": 9.46182,
+ "285": 9.18486,
+ "286": 9.37885,
+ "287": 9.07468,
+ "288": 8.76877,
+ "289": 9.12677,
+ "290": 8.7037,
+ "291": 8.75648,
+ "292": 8.80483,
+ "293": 8.64773,
+ "294": 8.71276,
+ "295": 8.67453,
+ "296": 8.72931,
+ "297": 8.74577,
+ "298": 8.73721,
+ "299": 8.71034,
+ "300": 8.71186,
+ "301": 8.68824,
+ "302": 8.77784,
+ "303": 8.70538,
+ "304": 8.73717,
+ "305": 8.72477,
+ "306": 8.71504,
+ "307": 8.7187,
+ "308": 8.7292,
+ "309": 8.68313,
+ "310": 8.79177,
+ "311": 8.7182,
+ "312": 8.74976,
+ "313": 8.82914,
+ "314": 8.7962,
+ "315": 8.75212,
+ "316": 8.68214,
+ "317": 8.75444,
+ "318": 8.77477,
+ "319": 8.7101,
+ "320": 8.81165,
+ "321": 8.65105,
+ "322": 8.66282,
+ "323": 9.06816,
+ "324": 8.71185,
+ "325": 8.69644,
+ "326": 9.40494,
+ "327": 8.7692,
+ "328": 9.84994,
+ "329": 8.75208,
+ "330": 9.90441,
+ "331": 8.62934,
+ "332": 9.03137,
+ "333": 8.70977,
+ "334": 8.66308,
+ "335": 8.71852,
+ "336": 8.73723,
+ "337": 8.72037,
+ "338": 8.72195,
+ "339": 8.7195,
+ "340": 8.7654,
+ "341": 8.73604,
+ "342": 8.70549,
+ "343": 8.75748,
+ "344": 8.72961,
+ "345": 8.76384,
+ "346": 8.7471,
+ "347": 8.73551,
+ "348": 8.6797,
+ "349": 8.71632,
+ "350": 8.72063,
+ "351": 8.685,
+ "352": 8.66136,
+ "353": 8.67665,
+ "354": 8.75958,
+ "355": 8.78762,
+ "356": 8.75485,
+ "357": 8.73086,
+ "358": 8.73616,
+ "359": 8.70034,
+ "360": 8.65118,
+ "361": 9.16983,
+ "362": 8.70391,
+ "363": 8.73766,
+ "364": 8.73773,
+ "365": 8.77847,
+ "366": 9.13772,
+ "367": 8.79244,
+ "368": 8.74306,
+ "369": 8.73918,
+ "370": 9.44844,
+ "371": 8.80922,
+ "372": 9.47019,
+ "373": 8.72549,
+ "374": 9.86138,
+ "375": 8.8155,
+ "376": 9.03422,
+ "377": 8.73794,
+ "378": 8.76834,
+ "379": 8.75406,
+ "380": 8.71565,
+ "381": 8.71601,
+ "382": 8.78463,
+ "383": 8.72498,
+ "384": 8.64482,
+ "385": 8.73897,
+ "386": 8.76947,
+ "387": 8.69491,
+ "388": 8.71946,
+ "389": 8.67264,
+ "390": 8.72047,
+ "391": 8.69611,
+ "392": 8.70566,
+ "393": 8.75371,
+ "394": 8.70107,
+ "395": 8.63999,
+ "396": 8.68941,
+ "397": 8.7768,
+ "398": 8.70048,
+ "399": 8.74518,
+ "400": 9.13325,
+ "401": 8.74344,
+ "402": 8.73166,
+ "403": 8.73528,
+ "404": 8.77199,
+ "405": 9.07368,
+ "406": 8.79548,
+ "407": 8.78198,
+ "408": 8.67305,
+ "409": 8.68681,
+ "410": 8.80478,
+ "411": 8.71852,
+ "412": 8.69408,
+ "413": 8.9998,
+ "414": 9.1211,
+ "415": 8.69497,
+ "416": 9.41946,
+ "417": 9.37242,
+ "418": 8.679,
+ "419": 9.57457,
+ "420": 8.71469,
+ "421": 8.72421,
+ "422": 8.71029,
+ "423": 8.72568,
+ "424": 8.69517,
+ "425": 8.72445,
+ "426": 8.76382,
+ "427": 8.66648,
+ "428": 8.67174,
+ "429": 8.69344,
+ "430": 8.71356,
+ "431": 8.80395,
+ "432": 8.78996,
+ "433": 8.79165,
+ "434": 8.79523,
+ "435": 8.68625,
+ "436": 8.74457,
+ "437": 8.76331,
+ "438": 8.62005,
+ "439": 9.06072,
+ "440": 8.78858,
+ "441": 8.64696,
+ "442": 8.72387,
+ "443": 8.75162,
+ "444": 9.14886,
+ "445": 8.71549,
+ "446": 8.68625,
+ "447": 8.74574,
+ "448": 8.77772,
+ "449": 8.79313,
+ "450": 8.73673,
+ "451": 8.7178,
+ "452": 8.71247,
+ "453": 8.78217,
+ "454": 8.73617,
+ "455": 8.72952,
+ "456": 8.7091,
+ "457": 9.04349,
+ "458": 9.07594,
+ "459": 8.64047,
+ "460": 9.40508,
+ "461": 8.73038,
+ "462": 9.46243,
+ "463": 9.08261,
+ "464": 9.16602,
+ "465": 8.7227,
+ "466": 8.7119,
+ "467": 8.69926,
+ "468": 8.71426,
+ "469": 8.71272,
+ "470": 8.65752,
+ "471": 8.80615,
+ "472": 8.79403,
+ "473": 8.69897,
+ "474": 8.67836,
+ "475": 8.68815,
+ "476": 8.78653,
+ "477": 8.66525,
+ "478": 9.12035,
+ "479": 8.7082,
+ "480": 8.79716,
+ "481": 8.64246,
+ "482": 9.10773,
+ "483": 8.71388,
+ "484": 8.75467,
+ "485": 8.75154,
+ "486": 8.7168,
+ "487": 8.7055,
+ "488": 8.70954,
+ "489": 8.78456,
+ "490": 8.79879,
+ "491": 8.71699,
+ "492": 8.71531,
+ "493": 8.66329,
+ "494": 8.63512,
+ "495": 8.65429,
+ "496": 8.5959,
+ "497": 8.63649,
+ "498": 8.67196,
+ "499": 8.6491,
+ "500": 8.74877,
+ "501": 8.61722,
+ "502": 9.26151,
+ "503": 8.67856,
+ "504": 9.34605,
+ "505": 8.62985,
+ "506": 9.3671,
+ "507": 8.57077,
+ "508": 8.96765,
+ "509": 9.04964,
+ "510": 8.62963,
+ "511": 8.58466,
+ "512": 8.65811,
+ "513": 8.6293,
+ "514": 8.65266,
+ "515": 8.66263,
+ "516": 8.60834,
+ "517": 9.04176,
+ "518": 8.70988,
+ "519": 8.64792,
+ "520": 8.64307,
+ "521": 9.02062,
+ "522": 8.694,
+ "523": 8.59501,
+ "524": 8.61435,
+ "525": 8.62253,
+ "526": 8.64452,
+ "527": 8.6225,
+ "528": 8.60346,
+ "529": 8.63039,
+ "530": 8.68799,
+ "531": 8.6925,
+ "532": 8.57813,
+ "533": 8.59486,
+ "534": 8.61832,
+ "535": 8.63593,
+ "536": 8.63774,
+ "537": 8.60103,
+ "538": 8.68419,
+ "539": 8.64667,
+ "540": 8.78363,
+ "541": 8.70812,
+ "542": 8.63367,
+ "543": 8.64657,
+ "544": 8.69717,
+ "545": 8.64448,
+ "546": 9.27207,
+ "547": 8.57347,
+ "548": 9.37958,
+ "549": 9.35578,
+ "550": 8.61681,
+ "551": 8.6155,
+ "552": 8.95592,
+ "553": 8.99152,
+ "554": 8.67618,
+ "555": 8.99625,
+ "556": 8.65444,
+ "557": 8.605,
+ "558": 8.63068,
+ "559": 8.61579,
+ "560": 8.98974,
+ "561": 8.68635,
+ "562": 8.61988,
+ "563": 8.65447,
+ "564": 8.63114,
+ "565": 8.5829,
+ "566": 8.64826,
+ "567": 8.69937,
+ "568": 8.60336,
+ "569": 8.62606,
+ "570": 8.60418,
+ "571": 8.62152,
+ "572": 8.68734,
+ "573": 8.63081,
+ "574": 8.72659,
+ "575": 8.61083,
+ "576": 8.68291,
+ "577": 8.6201,
+ "578": 8.69007,
+ "579": 8.62035,
+ "580": 8.60754,
+ "581": 8.71443,
+ "582": 8.64443,
+ "583": 8.66623,
+ "584": 8.64661,
+ "585": 8.65959,
+ "586": 8.62758,
+ "587": 8.61636,
+ "588": 8.99387,
+ "589": 9.34389,
+ "590": 8.71844,
+ "591": 8.78004,
+ "592": 9.31374,
+ "593": 9.39266,
+ "594": 9.02311,
+ "595": 9.056,
+ "596": 8.61128,
+ "597": 8.69404,
+ "598": 9.42191,
+ "599": 8.59345,
+ "600": 8.69855,
+ "601": 8.64664,
+ "602": 8.71824,
+ "603": 8.6435,
+ "604": 8.62505,
+ "605": 8.69551,
+ "606": 8.65118,
+ "607": 8.67219,
+ "608": 8.65425,
+ "609": 8.65415,
+ "610": 8.7281,
+ "611": 8.77189,
+ "612": 8.65765,
+ "613": 8.73565,
+ "614": 8.58316,
+ "615": 8.67478,
+ "616": 8.63075,
+ "617": 8.6511,
+ "618": 8.64966,
+ "619": 8.60754,
+ "620": 8.73057,
+ "621": 8.69859,
+ "622": 8.63572,
+ "623": 8.59256,
+ "624": 8.67684,
+ "625": 8.56647,
+ "626": 8.63505,
+ "627": 8.73915,
+ "628": 8.80931,
+ "629": 8.68435,
+ "630": 8.6489,
+ "631": 8.68776,
+ "632": 8.67772,
+ "633": 9.37018,
+ "634": 8.9988,
+ "635": 8.64944,
+ "636": 9.44881,
+ "637": 9.04685,
+ "638": 9.34815,
+ "639": 9.02755,
+ "640": 8.65788,
+ "641": 8.67746,
+ "642": 8.66413,
+ "643": 9.08095,
+ "644": 8.8266,
+ "645": 8.73682,
+ "646": 8.68026,
+ "647": 8.6653,
+ "648": 8.66012,
+ "649": 8.60817,
+ "650": 8.67351,
+ "651": 8.66452,
+ "652": 8.67794,
+ "653": 8.68251,
+ "654": 8.632,
+ "655": 8.5978,
+ "656": 8.67779,
+ "657": 8.69782,
+ "658": 8.69906,
+ "659": 8.64594,
+ "660": 8.73193,
+ "661": 8.70841,
+ "662": 8.77714,
+ "663": 8.71545,
+ "664": 8.6613,
+ "665": 8.60668,
+ "666": 8.67343,
+ "667": 8.66917,
+ "668": 8.75537,
+ "669": 8.74514,
+ "670": 8.7515,
+ "671": 9.01298,
+ "672": 8.68197,
+ "673": 8.60742,
+ "674": 8.61973,
+ "675": 9.05548,
+ "676": 8.63785,
+ "677": 8.6286,
+ "678": 9.37466,
+ "679": 8.64298,
+ "680": 9.30988,
+ "681": 8.67029,
+ "682": 9.36467,
+ "683": 8.66742,
+ "684": 9.01526,
+ "685": 8.69696,
+ "686": 8.73079,
+ "687": 9.08864,
+ "688": 8.66501,
+ "689": 8.69537,
+ "690": 8.67311,
+ "691": 8.71784,
+ "692": 8.66719,
+ "693": 8.68528,
+ "694": 8.63498,
+ "695": 8.74407,
+ "696": 8.6462,
+ "697": 8.64191,
+ "698": 8.69758,
+ "699": 8.66808,
+ "700": 8.61981,
+ "701": 8.66233,
+ "702": 8.65233,
+ "703": 8.62956,
+ "704": 8.703,
+ "705": 8.72893,
+ "706": 8.64178,
+ "707": 8.63103,
+ "708": 8.73548,
+ "709": 8.62754,
+ "710": 9.0468,
+ "711": 8.68048,
+ "712": 8.69484,
+ "713": 8.70335,
+ "714": 9.02883,
+ "715": 8.72755,
+ "716": 8.69721,
+ "717": 8.70812,
+ "718": 8.58749,
+ "719": 8.64602,
+ "720": 8.73364,
+ "721": 9.03836,
+ "722": 8.9722,
+ "723": 8.621,
+ "724": 9.37014,
+ "725": 9.35284,
+ "726": 8.70415,
+ "727": 9.00024,
+ "728": 8.71289,
+ "729": 8.65823,
+ "730": 8.73525,
+ "731": 8.69741,
+ "732": 9.1813,
+ "733": 8.64549,
+ "734": 8.59251,
+ "735": 8.61474,
+ "736": 8.63169,
+ "737": 8.75229,
+ "738": 8.67546,
+ "739": 8.60501,
+ "740": 8.63473,
+ "741": 8.7326,
+ "742": 8.62172,
+ "743": 8.62241,
+ "744": 8.6739,
+ "745": 8.68032,
+ "746": 8.63196,
+ "747": 8.65075,
+ "748": 9.03098,
+ "749": 8.73549,
+ "750": 8.70485,
+ "751": 8.73349,
+ "752": 9.02257,
+ "753": 8.62159,
+ "754": 8.66043,
+ "755": 8.60841,
+ "756": 8.62195,
+ "757": 8.71287,
+ "758": 8.75282,
+ "759": 8.63498,
+ "760": 8.70967,
+ "761": 8.796,
+ "762": 8.70437,
+ "763": 8.75568,
+ "764": 8.72915,
+ "765": 9.40577,
+ "766": 8.71981,
+ "767": 8.73409,
+ "768": 9.44882,
+ "769": 9.41079,
+ "770": 8.79751,
+ "771": 9.14899,
+ "772": 8.67785,
+ "773": 8.75396,
+ "774": 8.77709,
+ "775": 8.70473,
+ "776": 9.19249,
+ "777": 8.7305,
+ "778": 8.75822,
+ "779": 8.77091,
+ "780": 8.76,
+ "781": 8.77456,
+ "782": 8.7803,
+ "783": 8.73846,
+ "784": 8.748,
+ "785": 8.75607,
+ "786": 8.73481,
+ "787": 8.7063,
+ "788": 9.43789,
+ "789": 8.91288,
+ "790": 14.00687,
+ "791": 9.82781,
+ "792": 9.62374,
+ "793": 8.67973,
+ "794": 8.79629,
+ "795": 8.72368,
+ "796": 8.70473,
+ "797": 8.80231,
+ "798": 8.74809,
+ "799": 8.76464,
+ "800": 8.75895,
+ "801": 8.81437,
+ "802": 8.75495,
+ "803": 8.72457,
+ "804": 8.80935,
+ "805": 8.7104,
+ "806": 8.81735,
+ "807": 8.73982,
+ "808": 8.76655,
+ "809": 9.06609,
+ "810": 9.07926,
+ "811": 8.73743,
+ "812": 9.45011,
+ "813": 8.66754,
+ "814": 9.45243,
+ "815": 9.08849,
+ "816": 8.82309,
+ "817": 8.82411,
+ "818": 8.71468,
+ "819": 8.73156,
+ "820": 8.72754,
+ "821": 8.74778,
+ "822": 9.17914,
+ "823": 8.72446,
+ "824": 8.69397,
+ "825": 8.71152,
+ "826": 9.11132,
+ "827": 8.77546,
+ "828": 23.02795,
+ "829": 8.66945,
+ "830": 9.05446,
+ "831": 8.71627,
+ "832": 8.71954,
+ "833": 8.856,
+ "834": 8.74815,
+ "835": 8.67559,
+ "836": 8.67782,
+ "837": 8.7734,
+ "838": 8.70783,
+ "839": 8.61356,
+ "840": 8.71719,
+ "841": 8.73248,
+ "842": 8.7375,
+ "843": 8.69141,
+ "844": 8.69396,
+ "845": 8.74811,
+ "846": 8.7214,
+ "847": 8.81413,
+ "848": 8.81658,
+ "849": 8.79139,
+ "850": 8.74494,
+ "851": 8.76732,
+ "852": 8.77662,
+ "853": 8.69557,
+ "854": 9.41599,
+ "855": 8.75986,
+ "856": 9.40143,
+ "857": 9.48045,
+ "858": 11.68606,
+ "859": 8.71784,
+ "860": 9.07464,
+ "861": 8.71965,
+ "862": 8.70267,
+ "863": 8.71153,
+ "864": 8.73922,
+ "865": 9.12807,
+ "866": 9.15873,
+ "867": 8.71422,
+ "868": 9.10976,
+ "869": 8.7164,
+ "870": 8.75771,
+ "871": 8.75878,
+ "872": 8.67066,
+ "873": 8.71565,
+ "874": 8.73556,
+ "875": 8.76271,
+ "876": 8.67747,
+ "877": 8.67412,
+ "878": 8.75906,
+ "879": 8.72171,
+ "880": 8.7225,
+ "881": 8.74911,
+ "882": 8.66603,
+ "883": 8.71464,
+ "884": 8.72819,
+ "885": 8.71137,
+ "886": 8.70094,
+ "887": 8.69329,
+ "888": 8.77051,
+ "889": 8.72487,
+ "890": 8.73949,
+ "891": 8.77397,
+ "892": 8.72114,
+ "893": 8.77457,
+ "894": 8.74838,
+ "895": 8.7878,
+ "896": 8.75945,
+ "897": 13.06208,
+ "898": 9.11271,
+ "899": 8.71636,
+ "900": 9.37293,
+ "901": 9.40692,
+ "902": 8.68401,
+ "903": 9.49787,
+ "904": 8.78897,
+ "905": 8.80933,
+ "906": 8.79748,
+ "907": 9.04922,
+ "908": 8.71228,
+ "909": 8.81469,
+ "910": 8.70314,
+ "911": 9.1742,
+ "912": 8.79736,
+ "913": 8.65648,
+ "914": 8.74655,
+ "915": 8.75667,
+ "916": 8.71287,
+ "917": 8.76377,
+ "918": 8.7052,
+ "919": 8.66134,
+ "920": 8.72234,
+ "921": 8.73464,
+ "922": 8.73288,
+ "923": 8.75132,
+ "924": 8.78799,
+ "925": 8.76102,
+ "926": 8.70375,
+ "927": 8.80134,
+ "928": 8.71047,
+ "929": 8.72515,
+ "930": 8.77098,
+ "931": 8.7294,
+ "932": 8.72705,
+ "933": 8.70807,
+ "934": 8.70091,
+ "935": 8.79857,
+ "936": 8.76281,
+ "937": 8.7707,
+ "938": 8.77032,
+ "939": 8.73553,
+ "940": 8.72925,
+ "941": 9.33685,
+ "942": 9.16084,
+ "943": 8.77203,
+ "944": 9.41564,
+ "945": 9.50391,
+ "946": 9.14325,
+ "947": 9.05498,
+ "948": 8.65965,
+ "949": 8.73284,
+ "950": 8.71846,
+ "951": 8.77804,
+ "952": 8.77827,
+ "953": 8.76973,
+ "954": 8.76718,
+ "955": 9.17485,
+ "956": 8.73324,
+ "957": 8.65849,
+ "958": 8.70976,
+ "959": 8.73773,
+ "960": 8.78704,
+ "961": 8.76449,
+ "962": 8.76027,
+ "963": 8.69988,
+ "964": 8.67395,
+ "965": 8.68068,
+ "966": 8.73321,
+ "967": 8.72686,
+ "968": 8.8456,
+ "969": 8.75559,
+ "970": 8.71739,
+ "971": 8.67937,
+ "972": 8.73234,
+ "973": 8.78945,
+ "974": 8.71847,
+ "975": 8.71195,
+ "976": 8.75926,
+ "977": 8.7403,
+ "978": 8.69865,
+ "979": 8.72338,
+ "980": 9.1481,
+ "981": 8.78649,
+ "982": 8.75909,
+ "983": 8.677,
+ "984": 9.02197,
+ "985": 9.09206,
+ "986": 9.03892,
+ "987": 9.45662,
+ "988": 8.73024,
+ "989": 9.47578,
+ "990": 8.75281,
+ "991": 9.07324,
+ "992": 8.71826,
+ "993": 8.76065,
+ "994": 8.72885,
+ "995": 8.74837,
+ "996": 8.81194,
+ "997": 8.73847,
+ "998": 8.81103,
+ "999": 8.70856,
+ "1000": 9.2363,
+ "1001": 8.65349,
+ "1002": 8.6274,
+ "1003": 8.66472,
+ "1004": 8.6063,
+ "1005": 8.57999,
+ "1006": 8.70977,
+ "1007": 8.68944,
+ "1008": 8.63439,
+ "1009": 8.69968,
+ "1010": 8.68977,
+ "1011": 8.68347,
+ "1012": 8.65412,
+ "1013": 8.69489,
+ "1014": 8.70307,
+ "1015": 8.67777,
+ "1016": 9.03626,
+ "1017": 8.66774,
+ "1018": 8.73762,
+ "1019": 8.67175,
+ "1020": 9.09973,
+ "1021": 8.7109,
+ "1022": 8.7441,
+ "1023": 8.63104,
+ "1024": 8.7457,
+ "1025": 8.71245,
+ "1026": 9.06964,
+ "1027": 9.04942,
+ "1028": 8.63727,
+ "1029": 9.42852,
+ "1030": 9.41904,
+ "1031": 8.72906,
+ "1032": 9.10192,
+ "1033": 8.65531,
+ "1034": 8.67804,
+ "1035": 8.59598,
+ "1036": 8.70216,
+ "1037": 8.66874,
+ "1038": 8.67826,
+ "1039": 8.6773,
+ "1040": 8.72395,
+ "1041": 8.7653,
+ "1042": 9.13671,
+ "1043": 8.72463,
+ "1044": 8.62945,
+ "1045": 8.63672,
+ "1046": 8.69161,
+ "1047": 8.67936,
+ "1048": 8.63832,
+ "1049": 8.62641,
+ "1050": 8.65247,
+ "1051": 8.67312,
+ "1052": 8.67074,
+ "1053": 8.67269,
+ "1054": 8.7167,
+ "1055": 9.07136,
+ "1056": 8.75404,
+ "1057": 8.79635,
+ "1058": 8.66065,
+ "1059": 9.00241,
+ "1060": 8.72392,
+ "1061": 8.69619,
+ "1062": 8.64221,
+ "1063": 8.71011,
+ "1064": 8.70586,
+ "1065": 8.76292,
+ "1066": 8.70348,
+ "1067": 8.73824,
+ "1068": 8.66646,
+ "1069": 8.66707,
+ "1070": 9.05212,
+ "1071": 8.96464,
+ "1072": 8.74601,
+ "1073": 9.65382,
+ "1074": 8.70984,
+ "1075": 9.41641,
+ "1076": 9.01298,
+ "1077": 8.65861,
+ "1078": 8.65997,
+ "1079": 8.71619,
+ "1080": 8.57038,
+ "1081": 8.68147,
+ "1082": 8.57839,
+ "1083": 8.6082,
+ "1084": 8.61794,
+ "1085": 8.69004,
+ "1086": 8.67152,
+ "1087": 9.05798,
+ "1088": 8.56698,
+ "1089": 8.56572,
+ "1090": 8.61215,
+ "1091": 8.60925,
+ "1092": 8.62104,
+ "1093": 8.59085,
+ "1094": 9.03055,
+ "1095": 8.61638,
+ "1096": 8.60284,
+ "1097": 8.57485,
+ "1098": 8.97972,
+ "1099": 8.60557,
+ "1100": 8.60757,
+ "1101": 8.73134,
+ "1102": 8.54469,
+ "1103": 8.68402,
+ "1104": 8.57159,
+ "1105": 8.58549,
+ "1106": 8.67626,
+ "1107": 8.63786,
+ "1108": 8.77457,
+ "1109": 8.56713,
+ "1110": 8.56171,
+ "1111": 8.5732,
+ "1112": 8.67396,
+ "1113": 8.64574,
+ "1114": 8.59687,
+ "1115": 9.31662,
+ "1116": 8.66672,
+ "1117": 9.35597,
+ "1118": 8.57452,
+ "1119": 9.35801,
+ "1120": 8.94737,
+ "1121": 8.54403,
+ "1122": 8.58378,
+ "1123": 8.58739,
+ "1124": 8.68184,
+ "1125": 8.75632,
+ "1126": 8.6956,
+ "1127": 8.62404,
+ "1128": 8.81864,
+ "1129": 8.71534,
+ "1130": 8.57507,
+ "1131": 8.59802,
+ "1132": 8.62997,
+ "1133": 9.46967,
+ "1134": 8.64179,
+ "1135": 8.67779,
+ "1136": 8.54953,
+ "1137": 8.92885,
+ "1138": 8.60947,
+ "1139": 8.61747,
+ "1140": 8.59953,
+ "1141": 8.61265,
+ "1142": 8.66399,
+ "1143": 8.62861,
+ "1144": 8.66667,
+ "1145": 8.57334,
+ "1146": 8.60118,
+ "1147": 8.71083,
+ "1148": 8.64544,
+ "1149": 8.61442,
+ "1150": 8.64404,
+ "1151": 8.6117,
+ "1152": 8.61644,
+ "1153": 8.69601,
+ "1154": 8.59898,
+ "1155": 8.69451,
+ "1156": 8.63898,
+ "1157": 8.68107,
+ "1158": 8.55012,
+ "1159": 8.66953,
+ "1160": 9.44297,
+ "1161": 8.56164,
+ "1162": 9.41466,
+ "1163": 8.70577,
+ "1164": 9.31461,
+ "1165": 8.99604,
+ "1166": 8.63768,
+ "1167": 8.72912,
+ "1168": 8.6454,
+ "1169": 8.63549,
+ "1170": 8.64905,
+ "1171": 8.62687,
+ "1172": 9.04353,
+ "1173": 8.73273,
+ "1174": 8.69148,
+ "1175": 8.71461,
+ "1176": 9.04215,
+ "1177": 8.65411,
+ "1178": 9.10494,
+ "1179": 8.65059,
+ "1180": 8.72812,
+ "1181": 8.72255,
+ "1182": 8.67256,
+ "1183": 8.65579,
+ "1184": 8.68197,
+ "1185": 8.62635,
+ "1186": 8.66514,
+ "1187": 8.72188,
+ "1188": 8.66422,
+ "1189": 8.6905,
+ "1190": 8.64205,
+ "1191": 8.60871,
+ "1192": 8.63715,
+ "1193": 8.73142,
+ "1194": 8.61181,
+ "1195": 8.68786,
+ "1196": 8.64243,
+ "1197": 8.58931,
+ "1198": 8.62203,
+ "1199": 8.64157,
+ "1200": 8.63638,
+ "1201": 8.75347,
+ "1202": 8.71183,
+ "1203": 8.65707,
+ "1204": 8.97777,
+ "1205": 8.73674,
+ "1206": 9.33265,
+ "1207": 8.63212,
+ "1208": 9.31054,
+ "1209": 8.63702,
+ "1210": 9.15288,
+ "1211": 8.9992,
+ "1212": 8.73626,
+ "1213": 8.64293,
+ "1214": 8.64471,
+ "1215": 9.06069,
+ "1216": 8.93838,
+ "1217": 8.63945,
+ "1218": 8.74316,
+ "1219": 8.54126,
+ "1220": 8.64695,
+ "1221": 8.59168,
+ "1222": 9.07953,
+ "1223": 8.5631,
+ "1224": 8.67476,
+ "1225": 8.6551,
+ "1226": 8.56607,
+ "1227": 8.57149,
+ "1228": 8.48757,
+ "1229": 8.50335,
+ "1230": 8.44934,
+ "1231": 8.47979,
+ "1232": 8.48407,
+ "1233": 8.55429,
+ "1234": 8.56136,
+ "1235": 8.52971,
+ "1236": 8.54584,
+ "1237": 8.5799,
+ "1238": 8.61346,
+ "1239": 8.48796,
+ "1240": 45.53997,
+ "1241": 8.54869,
+ "1242": 8.52849,
+ "1243": 8.53005,
+ "1244": 8.48551,
+ "1245": 8.46474,
+ "1246": 8.57429,
+ "1247": 8.58682,
+ "1248": 8.78196,
+ "1249": 8.45323,
+ "1250": 9.67871,
+ "1251": 8.58812,
+ "1252": 9.26108,
+ "1253": 8.56589,
+ "1254": 9.24786,
+ "1255": 8.49394,
+ "1256": 8.52648,
+ "1257": 8.55674,
+ "1258": 8.46414,
+ "1259": 8.57273,
+ "1260": 8.61566,
+ "1261": 8.5387,
+ "1262": 8.55462,
+ "1263": 8.57664,
+ "1264": 8.52539,
+ "1265": 8.50949,
+ "1266": 8.51008,
+ "1267": 9.00356,
+ "1268": 8.58163,
+ "1269": 8.5138,
+ "1270": 8.48516,
+ "1271": 8.58737,
+ "1272": 8.50186,
+ "1273": 8.51685,
+ "1274": 8.54829,
+ "1275": 8.49049,
+ "1276": 8.52732,
+ "1277": 8.51894,
+ "1278": 8.52168,
+ "1279": 8.47983,
+ "1280": 8.48546,
+ "1281": 8.55423,
+ "1282": 8.41939,
+ "1283": 8.47639,
+ "1284": 8.50079,
+ "1285": 8.52949,
+ "1286": 8.48181,
+ "1287": 8.44213,
+ "1288": 8.52926,
+ "1289": 8.91817,
+ "1290": 8.52575,
+ "1291": 8.52206,
+ "1292": 8.91427,
+ "1293": 8.96591,
+ "1294": 8.57698,
+ "1295": 9.25403,
+ "1296": 9.20786,
+ "1297": 8.51413,
+ "1298": 8.95889,
+ "1299": 8.46908,
+ "1300": 8.49647,
+ "1301": 8.58348,
+ "1302": 8.6086,
+ "1303": 8.53819,
+ "1304": 8.54762,
+ "1305": 8.61048,
+ "1306": 8.50371,
+ "1307": 8.49369,
+ "1308": 8.50746,
+ "1309": 8.53074,
+ "1310": 8.5048,
+ "1311": 8.47194,
+ "1312": 8.46505,
+ "1313": 8.96321,
+ "1314": 8.58143,
+ "1315": 8.58588,
+ "1316": 8.54531,
+ "1317": 8.4874,
+ "1318": 8.50696,
+ "1319": 8.53939,
+ "1320": 8.54917,
+ "1321": 8.5756,
+ "1322": 8.52418,
+ "1323": 8.64078,
+ "1324": 8.54867,
+ "1325": 8.45669,
+ "1326": 8.52265,
+ "1327": 8.54128,
+ "1328": 8.97809,
+ "1329": 8.55028,
+ "1330": 8.57594,
+ "1331": 8.56634,
+ "1332": 8.90035,
+ "1333": 8.61456,
+ "1334": 8.62985,
+ "1335": 8.58386,
+ "1336": 8.61078,
+ "1337": 8.95588,
+ "1338": 8.65442,
+ "1339": 8.62813,
+ "1340": 9.41129,
+ "1341": 9.36096,
+ "1342": 8.60078,
+ "1343": 9.0135,
+ "1344": 8.64317,
+ "1345": 8.68065,
+ "1346": 8.71924,
+ "1347": 8.63705,
+ "1348": 8.72923,
+ "1349": 8.65392,
+ "1350": 8.66091,
+ "1351": 8.69621,
+ "1352": 8.74704,
+ "1353": 8.71655,
+ "1354": 8.63865,
+ "1355": 8.64465,
+ "1356": 8.67502,
+ "1357": 8.66609,
+ "1358": 9.07816,
+ "1359": 8.70592,
+ "1360": 8.65471,
+ "1361": 8.71117,
+ "1362": 8.63538,
+ "1363": 8.71345,
+ "1364": 8.64941,
+ "1365": 8.72267,
+ "1366": 8.66687,
+ "1367": 9.00631,
+ "1368": 8.63572,
+ "1369": 8.64515,
+ "1370": 8.62435,
+ "1371": 9.03939,
+ "1372": 8.65275,
+ "1373": 8.65151,
+ "1374": 8.64948,
+ "1375": 8.68869,
+ "1376": 8.82985,
+ "1377": 21.217,
+ "1378": 40.95872,
+ "1379": 20.01612,
+ "1380": 8.75383,
+ "1381": 9.3481,
+ "1382": 8.70083,
+ "1383": 8.65443,
+ "1384": 9.4217,
+ "1385": 9.43006,
+ "1386": 8.77636,
+ "1387": 9.0669,
+ "1388": 8.77777,
+ "1389": 8.70965,
+ "1390": 8.67851,
+ "1391": 8.63778,
+ "1392": 8.68641,
+ "1393": 8.75037,
+ "1394": 8.65608,
+ "1395": 8.66416,
+ "1396": 8.71299,
+ "1397": 8.77798,
+ "1398": 8.69553,
+ "1399": 8.68763,
+ "1400": 8.63441,
+ "1401": 8.62684,
+ "1402": 8.68906,
+ "1403": 9.1983,
+ "1404": 8.73694,
+ "1405": 8.66699,
+ "1406": 9.33992,
+ "1407": 9.01837,
+ "1408": 8.66395,
+ "1409": 8.73526,
+ "1410": 9.10279,
+ "1411": 8.67293,
+ "1412": 8.67123,
+ "1413": 8.73739,
+ "1414": 8.70842,
+ "1415": 8.72992,
+ "1416": 9.58032,
+ "1417": 8.71712,
+ "1418": 8.73911,
+ "1419": 8.80202,
+ "1420": 8.66567,
+ "1421": 8.67863,
+ "1422": 8.79449,
+ "1423": 8.66177,
+ "1424": 8.71751,
+ "1425": 9.38244,
+ "1426": 8.70115,
+ "1427": 8.76302,
+ "1428": 9.43496,
+ "1429": 9.42172,
+ "1430": 8.67113,
+ "1431": 9.10854,
+ "1432": 8.74069,
+ "1433": 8.71114,
+ "1434": 8.736,
+ "1435": 8.77025,
+ "1436": 8.71234,
+ "1437": 8.79655,
+ "1438": 8.64765,
+ "1439": 8.7016,
+ "1440": 8.6769,
+ "1441": 8.78807,
+ "1442": 8.67873,
+ "1443": 8.72378,
+ "1444": 8.72471,
+ "1445": 9.06454,
+ "1446": 8.72754,
+ "1447": 8.72121,
+ "1448": 9.54509,
+ "1449": 8.69379,
+ "1450": 8.7303,
+ "1451": 8.68379,
+ "1452": 8.63982,
+ "1453": 8.80999,
+ "1454": 8.66709,
+ "1455": 8.7115,
+ "1456": 8.75449,
+ "1457": 8.73172,
+ "1458": 8.65046,
+ "1459": 8.62662,
+ "1460": 8.66976,
+ "1461": 8.65035,
+ "1462": 8.67483,
+ "1463": 8.67235,
+ "1464": 8.66259,
+ "1465": 8.71793,
+ "1466": 8.73276,
+ "1467": 8.72361,
+ "1468": 8.74631,
+ "1469": 9.07112,
+ "1470": 9.03492,
+ "1471": 8.67742,
+ "1472": 9.3953,
+ "1473": 9.47543,
+ "1474": 8.66103,
+ "1475": 9.04497,
+ "1476": 8.67482,
+ "1477": 8.66024,
+ "1478": 8.67807,
+ "1479": 8.7691,
+ "1480": 8.66112,
+ "1481": 8.69242,
+ "1482": 8.63093,
+ "1483": 8.93329,
+ "1484": 8.84864,
+ "1485": 9.07026,
+ "1486": 8.69765,
+ "1487": 8.68548,
+ "1488": 9.0605,
+ "1489": 8.65309,
+ "1490": 8.7069,
+ "1491": 8.67222,
+ "1492": 8.66236,
+ "1493": 9.10427,
+ "1494": 8.67805,
+ "1495": 8.6195,
+ "1496": 8.6614,
+ "1497": 8.69983,
+ "1498": 8.69539,
+ "1499": 8.71311,
+ "1500": 8.73107,
+ "1501": 8.6485,
+ "1502": 8.69147,
+ "1503": 8.61687,
+ "1504": 8.61433,
+ "1505": 8.611,
+ "1506": 8.67697,
+ "1507": 8.71282,
+ "1508": 8.70033,
+ "1509": 8.76975,
+ "1510": "nan",
+ "1511": "nan",
+ "1512": "nan",
+ "1513": "nan",
+ "1514": "nan",
+ "1515": "nan",
+ "1516": "nan",
+ "1517": "nan",
+ "1518": "nan",
+ "1519": "nan",
+ "1520": "nan",
+ "1521": "nan",
+ "1522": "nan",
+ "1523": "nan",
+ "1524": "nan",
+ "1525": "nan",
+ "1526": "nan",
+ "1527": "nan",
+ "1528": "nan",
+ "1529": "nan",
+ "1530": "nan",
+ "1531": "nan",
+ "1532": "nan",
+ "1533": "nan",
+ "1534": "nan",
+ "1535": "nan",
+ "1536": "nan",
+ "1537": "nan",
+ "1538": "nan",
+ "1539": "nan",
+ "1540": "nan",
+ "1541": "nan",
+ "1542": "nan",
+ "1543": "nan",
+ "1544": "nan",
+ "1545": "nan",
+ "1546": "nan",
+ "1547": "nan",
+ "1548": "nan",
+ "1549": "nan",
+ "1550": "nan",
+ "1551": "nan",
+ "1552": "nan",
+ "1553": "nan",
+ "1554": "nan",
+ "1555": "nan",
+ "1556": "nan",
+ "1557": "nan",
+ "1558": "nan",
+ "1559": "nan",
+ "1560": "nan",
+ "1561": "nan",
+ "1562": "nan",
+ "1563": "nan",
+ "1564": "nan",
+ "1565": "nan",
+ "1566": "nan",
+ "1567": "nan",
+ "1568": "nan",
+ "1569": "nan",
+ "1570": "nan",
+ "1571": "nan",
+ "1572": "nan",
+ "1573": "nan",
+ "1574": "nan",
+ "1575": "nan",
+ "1576": "nan",
+ "1577": "nan",
+ "1578": "nan",
+ "1579": "nan",
+ "1580": "nan",
+ "1581": "nan",
+ "1582": "nan",
+ "1583": "nan",
+ "1584": "nan",
+ "1585": "nan",
+ "1586": "nan",
+ "1587": "nan",
+ "1588": "nan",
+ "1589": "nan",
+ "1590": "nan",
+ "1591": "nan",
+ "1592": "nan",
+ "1593": "nan",
+ "1594": "nan",
+ "1595": "nan",
+ "1596": "nan",
+ "1597": "nan",
+ "1598": "nan",
+ "1599": "nan",
+ "1600": "nan",
+ "1601": "nan",
+ "1602": "nan",
+ "1603": "nan",
+ "1604": "nan",
+ "1605": "nan",
+ "1606": "nan",
+ "1607": "nan",
+ "1608": "nan",
+ "1609": "nan",
+ "1610": "nan",
+ "1611": "nan",
+ "1612": "nan",
+ "1613": "nan",
+ "1614": "nan",
+ "1615": "nan",
+ "1616": "nan",
+ "1617": "nan",
+ "1618": "nan",
+ "1619": "nan",
+ "1620": "nan",
+ "1621": "nan",
+ "1622": "nan",
+ "1623": "nan",
+ "1624": "nan",
+ "1625": "nan",
+ "1626": "nan",
+ "1627": "nan",
+ "1628": "nan",
+ "1629": "nan",
+ "1630": "nan",
+ "1631": "nan",
+ "1632": "nan",
+ "1633": "nan",
+ "1634": "nan",
+ "1635": "nan",
+ "1636": "nan",
+ "1637": "nan",
+ "1638": "nan",
+ "1639": "nan",
+ "1640": "nan",
+ "1641": "nan",
+ "1642": "nan",
+ "1643": "nan",
+ "1644": "nan",
+ "1645": "nan",
+ "1646": "nan",
+ "1647": "nan",
+ "1648": "nan",
+ "1649": "nan",
+ "1650": "nan",
+ "1651": "nan",
+ "1652": "nan",
+ "1653": "nan",
+ "1654": "nan",
+ "1655": "nan",
+ "1656": "nan",
+ "1657": "nan",
+ "1658": "nan",
+ "1659": "nan",
+ "1660": "nan",
+ "1661": "nan",
+ "1662": "nan",
+ "1663": "nan",
+ "1664": "nan",
+ "1665": "nan",
+ "1666": "nan",
+ "1667": "nan",
+ "1668": "nan",
+ "1669": "nan",
+ "1670": "nan",
+ "1671": "nan",
+ "1672": "nan",
+ "1673": "nan",
+ "1674": "nan",
+ "1675": "nan",
+ "1676": "nan",
+ "1677": "nan",
+ "1678": "nan",
+ "1679": "nan",
+ "1680": "nan",
+ "1681": "nan",
+ "1682": "nan",
+ "1683": "nan",
+ "1684": "nan",
+ "1685": "nan",
+ "1686": "nan",
+ "1687": "nan",
+ "1688": "nan",
+ "1689": "nan",
+ "1690": "nan",
+ "1691": "nan",
+ "1692": "nan",
+ "1693": "nan",
+ "1694": "nan",
+ "1695": "nan",
+ "1696": "nan",
+ "1697": "nan",
+ "1698": "nan",
+ "1699": "nan",
+ "1700": "nan",
+ "1701": "nan",
+ "1702": "nan",
+ "1703": "nan",
+ "1704": "nan",
+ "1705": "nan",
+ "1706": "nan",
+ "1707": "nan",
+ "1708": "nan",
+ "1709": "nan",
+ "1710": "nan",
+ "1711": "nan",
+ "1712": "nan",
+ "1713": "nan",
+ "1714": "nan",
+ "1715": "nan",
+ "1716": "nan",
+ "1717": "nan",
+ "1718": "nan",
+ "1719": "nan",
+ "1720": "nan",
+ "1721": "nan",
+ "1722": "nan",
+ "1723": "nan",
+ "1724": "nan",
+ "1725": "nan",
+ "1726": "nan",
+ "1727": "nan",
+ "1728": "nan",
+ "1729": "nan",
+ "1730": "nan",
+ "1731": "nan",
+ "1732": "nan",
+ "1733": "nan",
+ "1734": "nan",
+ "1735": "nan",
+ "1736": "nan",
+ "1737": "nan",
+ "1738": "nan",
+ "1739": "nan",
+ "1740": "nan",
+ "1741": "nan",
+ "1742": "nan",
+ "1743": "nan",
+ "1744": "nan",
+ "1745": "nan",
+ "1746": "nan",
+ "1747": "nan",
+ "1748": "nan",
+ "1749": "nan",
+ "1750": "nan",
+ "1751": "nan",
+ "1752": "nan",
+ "1753": "nan",
+ "1754": "nan",
+ "1755": "nan",
+ "1756": "nan",
+ "1757": "nan",
+ "1758": "nan",
+ "1759": "nan",
+ "1760": "nan",
+ "1761": "nan",
+ "1762": "nan",
+ "1763": "nan",
+ "1764": "nan",
+ "1765": "nan",
+ "1766": "nan",
+ "1767": "nan",
+ "1768": "nan",
+ "1769": "nan",
+ "1770": "nan",
+ "1771": "nan",
+ "1772": "nan",
+ "1773": "nan",
+ "1774": "nan",
+ "1775": "nan",
+ "1776": "nan",
+ "1777": "nan",
+ "1778": "nan",
+ "1779": "nan",
+ "1780": "nan",
+ "1781": "nan",
+ "1782": "nan",
+ "1783": "nan",
+ "1784": "nan",
+ "1785": "nan",
+ "1786": "nan",
+ "1787": "nan",
+ "1788": "nan",
+ "1789": "nan",
+ "1790": "nan",
+ "1791": "nan",
+ "1792": "nan",
+ "1793": "nan",
+ "1794": "nan",
+ "1795": "nan",
+ "1796": "nan",
+ "1797": "nan",
+ "1798": "nan",
+ "1799": "nan",
+ "1800": "nan",
+ "1801": "nan",
+ "1802": "nan",
+ "1803": "nan",
+ "1804": "nan",
+ "1805": "nan",
+ "1806": "nan",
+ "1807": "nan",
+ "1808": "nan",
+ "1809": "nan",
+ "1810": "nan",
+ "1811": "nan",
+ "1812": "nan",
+ "1813": "nan",
+ "1814": "nan",
+ "1815": "nan",
+ "1816": "nan",
+ "1817": "nan",
+ "1818": "nan",
+ "1819": "nan",
+ "1820": "nan",
+ "1821": "nan",
+ "1822": "nan",
+ "1823": "nan",
+ "1824": "nan",
+ "1825": "nan",
+ "1826": "nan",
+ "1827": "nan",
+ "1828": "nan",
+ "1829": "nan",
+ "1830": "nan",
+ "1831": "nan",
+ "1832": "nan",
+ "1833": "nan",
+ "1834": "nan",
+ "1835": "nan",
+ "1836": "nan",
+ "1837": "nan",
+ "1838": "nan",
+ "1839": "nan",
+ "1840": "nan",
+ "1841": "nan",
+ "1842": "nan",
+ "1843": "nan",
+ "1844": "nan",
+ "1845": "nan",
+ "1846": "nan",
+ "1847": "nan",
+ "1848": "nan",
+ "1849": "nan",
+ "1850": "nan",
+ "1851": "nan",
+ "1852": "nan",
+ "1853": "nan",
+ "1854": "nan",
+ "1855": "nan",
+ "1856": "nan",
+ "1857": "nan",
+ "1858": "nan",
+ "1859": "nan",
+ "1860": "nan",
+ "1861": "nan",
+ "1862": "nan",
+ "1863": "nan",
+ "1864": "nan",
+ "1865": "nan",
+ "1866": "nan",
+ "1867": "nan",
+ "1868": "nan",
+ "1869": "nan",
+ "1870": "nan",
+ "1871": "nan",
+ "1872": "nan",
+ "1873": "nan",
+ "1874": "nan",
+ "1875": "nan",
+ "1876": "nan",
+ "1877": "nan",
+ "1878": "nan",
+ "1879": "nan",
+ "1880": "nan",
+ "1881": "nan",
+ "1882": "nan",
+ "1883": "nan",
+ "1884": "nan",
+ "1885": "nan",
+ "1886": "nan",
+ "1887": "nan",
+ "1888": "nan",
+ "1889": "nan",
+ "1890": "nan",
+ "1891": "nan",
+ "1892": "nan",
+ "1893": "nan",
+ "1894": "nan",
+ "1895": "nan",
+ "1896": "nan",
+ "1897": "nan",
+ "1898": "nan",
+ "1899": "nan",
+ "1900": "nan",
+ "1901": "nan",
+ "1902": "nan",
+ "1903": "nan",
+ "1904": "nan",
+ "1905": "nan",
+ "1906": "nan",
+ "1907": "nan",
+ "1908": "nan",
+ "1909": "nan",
+ "1910": "nan",
+ "1911": "nan",
+ "1912": "nan",
+ "1913": "nan",
+ "1914": "nan",
+ "1915": "nan",
+ "1916": "nan",
+ "1917": "nan",
+ "1918": "nan",
+ "1919": "nan",
+ "1920": "nan",
+ "1921": "nan",
+ "1922": "nan",
+ "1923": "nan",
+ "1924": "nan",
+ "1925": "nan",
+ "1926": "nan",
+ "1927": "nan",
+ "1928": "nan",
+ "1929": "nan",
+ "1930": "nan",
+ "1931": "nan",
+ "1932": "nan",
+ "1933": "nan",
+ "1934": "nan",
+ "1935": "nan",
+ "1936": "nan",
+ "1937": "nan",
+ "1938": "nan",
+ "1939": "nan",
+ "1940": "nan",
+ "1941": "nan",
+ "1942": "nan",
+ "1943": "nan",
+ "1944": "nan",
+ "1945": "nan",
+ "1946": "nan",
+ "1947": "nan",
+ "1948": "nan",
+ "1949": "nan",
+ "1950": "nan",
+ "1951": "nan",
+ "1952": "nan",
+ "1953": "nan",
+ "1954": "nan",
+ "1955": "nan",
+ "1956": "nan",
+ "1957": "nan",
+ "1958": "nan",
+ "1959": "nan",
+ "1960": "nan",
+ "1961": "nan",
+ "1962": "nan",
+ "1963": "nan",
+ "1964": "nan",
+ "1965": "nan",
+ "1966": "nan",
+ "1967": "nan",
+ "1968": "nan",
+ "1969": "nan",
+ "1970": "nan",
+ "1971": "nan",
+ "1972": "nan",
+ "1973": "nan",
+ "1974": "nan",
+ "1975": "nan",
+ "1976": "nan",
+ "1977": "nan",
+ "1978": "nan",
+ "1979": "nan",
+ "1980": "nan",
+ "1981": "nan",
+ "1982": "nan",
+ "1983": "nan",
+ "1984": "nan",
+ "1985": "nan",
+ "1986": "nan",
+ "1987": "nan",
+ "1988": "nan",
+ "1989": "nan",
+ "1990": "nan",
+ "1991": "nan",
+ "1992": "nan",
+ "1993": "nan",
+ "1994": "nan",
+ "1995": "nan",
+ "1996": "nan",
+ "1997": "nan",
+ "1998": "nan",
+ "1999": "nan",
+ "2000": "nan"
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/golden_values_dev_dgx_h100.json
new file mode 100644
index 00000000000..c97e14dbfc3
--- /dev/null
+++ b/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/golden_values_dev_dgx_h100.json
@@ -0,0 +1,10037 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 10.84754,
+ "2": 10.85706,
+ "3": 10.85346,
+ "4": 10.86642,
+ "5": 10.8509,
+ "6": 10.85013,
+ "7": 10.85754,
+ "8": 10.84975,
+ "9": 10.84593,
+ "10": 10.84527,
+ "11": 10.8338,
+ "12": 10.83054,
+ "13": 10.81881,
+ "14": 10.83804,
+ "15": 10.77891,
+ "16": 10.78962,
+ "17": 10.75965,
+ "18": 10.78537,
+ "19": 10.72289,
+ "20": 10.68818,
+ "21": 10.64002,
+ "22": 10.64238,
+ "23": 10.6509,
+ "24": 10.54606,
+ "25": 10.55844,
+ "26": 10.62983,
+ "27": 10.46355,
+ "28": 10.46889,
+ "29": 10.36199,
+ "30": 10.27007,
+ "31": 10.43007,
+ "32": 10.35126,
+ "33": 10.22009,
+ "34": 10.16739,
+ "35": 10.23504,
+ "36": 10.15396,
+ "37": 10.1023,
+ "38": 10.03134,
+ "39": 10.04517,
+ "40": 10.06709,
+ "41": 9.94869,
+ "42": 9.95224,
+ "43": 9.86872,
+ "44": 9.98941,
+ "45": 10.00586,
+ "46": 9.84491,
+ "47": 9.99851,
+ "48": 9.81444,
+ "49": 9.947,
+ "50": 9.94181,
+ "51": 9.59199,
+ "52": 9.79223,
+ "53": 9.62799,
+ "54": 9.88315,
+ "55": 9.73756,
+ "56": 9.84798,
+ "57": 9.85927,
+ "58": 9.8755,
+ "59": 9.54418,
+ "60": 9.64479,
+ "61": 9.87803,
+ "62": 9.76216,
+ "63": 9.6784,
+ "64": 9.82571,
+ "65": 9.59432,
+ "66": 9.62687,
+ "67": 9.7398,
+ "68": 9.60313,
+ "69": 9.2937,
+ "70": 9.42248,
+ "71": 9.78791,
+ "72": 9.71123,
+ "73": 9.61675,
+ "74": 9.45091,
+ "75": 9.23957,
+ "76": 9.50589,
+ "77": 9.57621,
+ "78": 9.55777,
+ "79": 9.30756,
+ "80": 9.35869,
+ "81": 9.4592,
+ "82": 9.55375,
+ "83": 9.53111,
+ "84": 9.35257,
+ "85": 9.3988,
+ "86": 9.64921,
+ "87": 9.23342,
+ "88": 9.48714,
+ "89": 9.22202,
+ "90": 9.41243,
+ "91": 9.38793,
+ "92": 9.37698,
+ "93": 9.36102,
+ "94": 9.51327,
+ "95": 9.41784,
+ "96": 9.33418,
+ "97": 9.20421,
+ "98": 9.49008,
+ "99": 9.2914,
+ "100": 9.35645,
+ "101": 9.2456,
+ "102": 9.24017,
+ "103": 9.07528,
+ "104": 9.16588,
+ "105": 9.37479,
+ "106": 9.14966,
+ "107": 9.17657,
+ "108": 9.31282,
+ "109": 9.28763,
+ "110": 9.36509,
+ "111": 9.17373,
+ "112": 9.23044,
+ "113": 9.35223,
+ "114": 9.3554,
+ "115": 9.32136,
+ "116": 9.00106,
+ "117": 9.06171,
+ "118": 9.06268,
+ "119": 9.22291,
+ "120": 9.08068,
+ "121": 9.19234,
+ "122": 9.13794,
+ "123": 9.25678,
+ "124": 9.45522,
+ "125": 9.21641,
+ "126": 9.05915,
+ "127": 9.01207,
+ "128": 9.21829,
+ "129": 8.98129,
+ "130": 9.13829,
+ "131": 9.15409,
+ "132": 9.03273,
+ "133": 8.85988,
+ "134": 9.18204,
+ "135": 8.88672,
+ "136": 9.16265,
+ "137": 9.15709,
+ "138": 9.23006,
+ "139": 9.08595,
+ "140": 8.87048,
+ "141": 9.29546,
+ "142": 9.19631,
+ "143": 9.11753,
+ "144": 9.24096,
+ "145": 9.10357,
+ "146": 8.98108,
+ "147": 8.98558,
+ "148": 9.13402,
+ "149": 9.06619,
+ "150": 9.00921,
+ "151": 8.92898,
+ "152": 8.87609,
+ "153": 9.06533,
+ "154": 9.17839,
+ "155": 9.13583,
+ "156": 9.04881,
+ "157": 9.1503,
+ "158": 9.05136,
+ "159": 9.03468,
+ "160": 8.89043,
+ "161": 9.04615,
+ "162": 8.89425,
+ "163": 8.8424,
+ "164": 8.97349,
+ "165": 8.92897,
+ "166": 8.6552,
+ "167": 8.83285,
+ "168": 8.81941,
+ "169": 8.65807,
+ "170": 9.04596,
+ "171": 8.72028,
+ "172": 8.81963,
+ "173": 8.90925,
+ "174": 8.84887,
+ "175": 8.70835,
+ "176": 8.75934,
+ "177": 8.76455,
+ "178": 8.72272,
+ "179": 8.63936,
+ "180": 8.74251,
+ "181": 8.69327,
+ "182": 8.72271,
+ "183": 9.08564,
+ "184": 8.60876,
+ "185": 8.88295,
+ "186": 8.74224,
+ "187": 8.57442,
+ "188": 8.68313,
+ "189": 8.8659,
+ "190": 8.53625,
+ "191": 8.66488,
+ "192": 8.6109,
+ "193": 8.57624,
+ "194": 8.75144,
+ "195": 8.59347,
+ "196": 8.7762,
+ "197": 8.74361,
+ "198": 8.62949,
+ "199": 8.77197,
+ "200": 8.7354,
+ "201": 8.66993,
+ "202": 8.55059,
+ "203": 8.54088,
+ "204": 8.71277,
+ "205": 8.22363,
+ "206": 8.85955,
+ "207": 8.67933,
+ "208": 8.70842,
+ "209": 8.7525,
+ "210": 8.58237,
+ "211": 8.84234,
+ "212": 8.49209,
+ "213": 8.57333,
+ "214": 8.51315,
+ "215": 8.56535,
+ "216": 8.50675,
+ "217": 8.53395,
+ "218": 8.53723,
+ "219": 8.64438,
+ "220": 8.5465,
+ "221": 8.39859,
+ "222": 8.50673,
+ "223": 8.44238,
+ "224": 8.52995,
+ "225": 8.57216,
+ "226": 8.44345,
+ "227": 8.67991,
+ "228": 8.38737,
+ "229": 8.45581,
+ "230": 8.50039,
+ "231": 8.50407,
+ "232": 8.50063,
+ "233": 8.49591,
+ "234": 8.6421,
+ "235": 8.56387,
+ "236": 8.40037,
+ "237": 8.49308,
+ "238": 8.30801,
+ "239": 8.56384,
+ "240": 8.67383,
+ "241": 8.44917,
+ "242": 8.47455,
+ "243": 8.51951,
+ "244": 8.37232,
+ "245": 8.59973,
+ "246": 8.59815,
+ "247": 8.44266,
+ "248": 8.51344,
+ "249": 8.52597,
+ "250": 8.42651,
+ "251": 8.38142,
+ "252": 8.55005,
+ "253": 8.31947,
+ "254": 8.35654,
+ "255": 8.29987,
+ "256": 8.2121,
+ "257": 8.39828,
+ "258": 8.45531,
+ "259": 8.23588,
+ "260": 8.24426,
+ "261": 8.24073,
+ "262": 8.35303,
+ "263": 8.31072,
+ "264": 8.19489,
+ "265": 8.33622,
+ "266": 8.23771,
+ "267": 7.90582,
+ "268": 8.38336,
+ "269": 8.40918,
+ "270": 8.26929,
+ "271": 8.28371,
+ "272": 8.32402,
+ "273": 8.13714,
+ "274": 8.10283,
+ "275": 8.0132,
+ "276": 7.92971,
+ "277": 8.24143,
+ "278": 8.05164,
+ "279": 7.96914,
+ "280": 7.76013,
+ "281": 8.10619,
+ "282": 8.15265,
+ "283": 8.16272,
+ "284": 8.1064,
+ "285": 8.07469,
+ "286": 7.906,
+ "287": 7.99693,
+ "288": 8.24948,
+ "289": 8.17804,
+ "290": 8.13285,
+ "291": 8.26011,
+ "292": 8.0828,
+ "293": 8.12566,
+ "294": 7.98113,
+ "295": 7.97089,
+ "296": 8.24262,
+ "297": 7.79573,
+ "298": 8.04911,
+ "299": 7.94486,
+ "300": 7.8569,
+ "301": 8.016,
+ "302": 7.95139,
+ "303": 7.99654,
+ "304": 7.96627,
+ "305": 8.00134,
+ "306": 7.98109,
+ "307": 7.9932,
+ "308": 8.00231,
+ "309": 8.01269,
+ "310": 7.97898,
+ "311": 7.93228,
+ "312": 7.88895,
+ "313": 7.83659,
+ "314": 7.8274,
+ "315": 7.83219,
+ "316": 7.75184,
+ "317": 7.93246,
+ "318": 7.98384,
+ "319": 7.83253,
+ "320": 7.56992,
+ "321": 7.75227,
+ "322": 7.83579,
+ "323": 7.77383,
+ "324": 7.91439,
+ "325": 7.80056,
+ "326": 7.6562,
+ "327": 7.86833,
+ "328": 7.79009,
+ "329": 7.88417,
+ "330": 7.75625,
+ "331": 7.52662,
+ "332": 7.8177,
+ "333": 7.84534,
+ "334": 7.68437,
+ "335": 7.70162,
+ "336": 7.91701,
+ "337": 7.65282,
+ "338": 7.89975,
+ "339": 7.72673,
+ "340": 7.76054,
+ "341": 7.71013,
+ "342": 7.82307,
+ "343": 7.61739,
+ "344": 7.59137,
+ "345": 7.61167,
+ "346": 7.46559,
+ "347": 7.55832,
+ "348": 7.68517,
+ "349": 7.58755,
+ "350": 7.65807,
+ "351": 7.74914,
+ "352": 7.70625,
+ "353": 7.50382,
+ "354": 7.74516,
+ "355": 7.76647,
+ "356": 7.77691,
+ "357": 7.81732,
+ "358": 7.60038,
+ "359": 7.54973,
+ "360": 7.63098,
+ "361": 7.54609,
+ "362": 7.76643,
+ "363": 7.5898,
+ "364": 7.58189,
+ "365": 7.62203,
+ "366": 7.30813,
+ "367": 7.56249,
+ "368": 7.44695,
+ "369": 7.34885,
+ "370": 7.46648,
+ "371": 7.46346,
+ "372": 7.65533,
+ "373": 7.52795,
+ "374": 7.44493,
+ "375": 7.532,
+ "376": 7.3483,
+ "377": 7.24136,
+ "378": 7.54143,
+ "379": 7.49396,
+ "380": 7.3868,
+ "381": 7.47194,
+ "382": 7.29399,
+ "383": 7.27632,
+ "384": 7.41307,
+ "385": 7.39635,
+ "386": 7.23361,
+ "387": 7.42234,
+ "388": 7.28286,
+ "389": 7.43727,
+ "390": 7.24258,
+ "391": 7.64786,
+ "392": 7.33599,
+ "393": 7.42065,
+ "394": 7.47711,
+ "395": 7.4406,
+ "396": 7.28474,
+ "397": 7.23178,
+ "398": 7.42322,
+ "399": 7.15796,
+ "400": 7.29876,
+ "401": 7.35503,
+ "402": 7.39325,
+ "403": 7.28394,
+ "404": 7.30529,
+ "405": 7.26333,
+ "406": 7.22014,
+ "407": 7.36277,
+ "408": 7.1848,
+ "409": 7.16717,
+ "410": 7.31764,
+ "411": 7.21973,
+ "412": 7.20015,
+ "413": 7.23299,
+ "414": 6.91457,
+ "415": 7.33319,
+ "416": 7.42971,
+ "417": 7.02293,
+ "418": 7.27558,
+ "419": 7.04131,
+ "420": 7.41182,
+ "421": 7.18335,
+ "422": 7.23823,
+ "423": 7.09606,
+ "424": 7.24419,
+ "425": 7.31707,
+ "426": 7.29031,
+ "427": 7.13265,
+ "428": 7.09429,
+ "429": 6.88025,
+ "430": 7.20628,
+ "431": 7.00842,
+ "432": 7.23284,
+ "433": 6.97896,
+ "434": 6.96145,
+ "435": 7.0205,
+ "436": 7.01066,
+ "437": 6.99356,
+ "438": 7.00339,
+ "439": 6.94136,
+ "440": 7.06286,
+ "441": 7.04265,
+ "442": 7.10181,
+ "443": 7.09417,
+ "444": 6.70764,
+ "445": 6.99676,
+ "446": 7.14381,
+ "447": 7.12662,
+ "448": 6.98445,
+ "449": 7.05101,
+ "450": 7.01776,
+ "451": 6.8324,
+ "452": 6.91238,
+ "453": 7.01672,
+ "454": 6.96891,
+ "455": 7.03323,
+ "456": 6.9961,
+ "457": 6.96944,
+ "458": 6.90553,
+ "459": 6.6917,
+ "460": 7.06296,
+ "461": 7.09643,
+ "462": 6.87185,
+ "463": 7.05421,
+ "464": 6.65055,
+ "465": 7.03022,
+ "466": 7.00744,
+ "467": 6.99927,
+ "468": 6.95498,
+ "469": 6.82807,
+ "470": 7.04408,
+ "471": 6.88198,
+ "472": 6.96087,
+ "473": 6.82239,
+ "474": 6.97441,
+ "475": 7.16647,
+ "476": 6.76228,
+ "477": 6.89619,
+ "478": 6.90621,
+ "479": 6.70331,
+ "480": 7.02606,
+ "481": 6.99322,
+ "482": 6.72974,
+ "483": 6.78204,
+ "484": 6.75039,
+ "485": 6.92854,
+ "486": 7.06354,
+ "487": 6.63025,
+ "488": 6.88238,
+ "489": 6.7679,
+ "490": 6.82121,
+ "491": 6.7059,
+ "492": 6.6899,
+ "493": 6.76344,
+ "494": 6.66976,
+ "495": 6.63002,
+ "496": 6.5856,
+ "497": 6.83721,
+ "498": 6.64223,
+ "499": 6.85071,
+ "500": 6.65057,
+ "501": 6.72636,
+ "502": 6.83668,
+ "503": 6.7054,
+ "504": 6.61338,
+ "505": 6.61732,
+ "506": 6.74139,
+ "507": 6.86095,
+ "508": 6.85354,
+ "509": 6.64671,
+ "510": 6.81842,
+ "511": 6.73691,
+ "512": 6.7354,
+ "513": 6.6547,
+ "514": 6.70871,
+ "515": 6.44451,
+ "516": 6.74068,
+ "517": 6.70227,
+ "518": 6.53664,
+ "519": 6.63151,
+ "520": 6.85691,
+ "521": 6.66058,
+ "522": 6.70538,
+ "523": 6.73863,
+ "524": 6.73369,
+ "525": 6.6726,
+ "526": 6.41314,
+ "527": 6.79748,
+ "528": 6.65831,
+ "529": 6.6297,
+ "530": 6.62283,
+ "531": 6.64209,
+ "532": 6.63083,
+ "533": 6.75974,
+ "534": 6.6096,
+ "535": 6.74828,
+ "536": 6.62409,
+ "537": 6.6379,
+ "538": 6.53035,
+ "539": 6.551,
+ "540": 6.58294,
+ "541": 6.45006,
+ "542": 6.66825,
+ "543": 6.67905,
+ "544": 6.6771,
+ "545": 6.81205,
+ "546": 6.63014,
+ "547": 6.41745,
+ "548": 6.72216,
+ "549": 6.69625,
+ "550": 6.52706,
+ "551": 6.74817,
+ "552": 6.63798,
+ "553": 6.48202,
+ "554": 6.63369,
+ "555": 6.45812,
+ "556": 6.61414,
+ "557": 6.63036,
+ "558": 6.38269,
+ "559": 6.36739,
+ "560": 6.58191,
+ "561": 6.73012,
+ "562": 6.63521,
+ "563": 6.73836,
+ "564": 6.34768,
+ "565": 6.51332,
+ "566": 6.69695,
+ "567": 6.5642,
+ "568": 6.50693,
+ "569": 6.45039,
+ "570": 6.36195,
+ "571": 6.62815,
+ "572": 6.31039,
+ "573": 6.57846,
+ "574": 6.46877,
+ "575": 6.64165,
+ "576": 6.5137,
+ "577": 6.52264,
+ "578": 6.48199,
+ "579": 6.4633,
+ "580": 6.56273,
+ "581": 6.60301,
+ "582": 6.47585,
+ "583": 6.51075,
+ "584": 6.51736,
+ "585": 6.42115,
+ "586": 6.40904,
+ "587": 6.45665,
+ "588": 6.56321,
+ "589": 6.62188,
+ "590": 6.28641,
+ "591": 6.67193,
+ "592": 6.26131,
+ "593": 6.47171,
+ "594": 6.38247,
+ "595": 6.35519,
+ "596": 6.25807,
+ "597": 6.1866,
+ "598": 6.45017,
+ "599": 6.39194,
+ "600": 6.45325,
+ "601": 6.25785,
+ "602": 6.52602,
+ "603": 6.51619,
+ "604": 6.38184,
+ "605": 6.48808,
+ "606": 6.30848,
+ "607": 6.52869,
+ "608": 6.66732,
+ "609": 6.16837,
+ "610": 6.5667,
+ "611": 6.39452,
+ "612": 6.57485,
+ "613": 6.41957,
+ "614": 6.19785,
+ "615": 6.39012,
+ "616": 6.35249,
+ "617": 6.37088,
+ "618": 6.44422,
+ "619": 6.13333,
+ "620": 6.40259,
+ "621": 6.4539,
+ "622": 6.39342,
+ "623": 6.57621,
+ "624": 6.35306,
+ "625": 6.2742,
+ "626": 6.2981,
+ "627": 6.43664,
+ "628": 6.24956,
+ "629": 6.58253,
+ "630": 6.35958,
+ "631": 6.34363,
+ "632": 6.30368,
+ "633": 6.25067,
+ "634": 6.30401,
+ "635": 6.54141,
+ "636": 6.23936,
+ "637": 6.63232,
+ "638": 6.01713,
+ "639": 6.27606,
+ "640": 6.28889,
+ "641": 6.20587,
+ "642": 6.27993,
+ "643": 6.45349,
+ "644": 6.25397,
+ "645": 6.24189,
+ "646": 6.40063,
+ "647": 6.33277,
+ "648": 6.35079,
+ "649": 6.34213,
+ "650": 6.48012,
+ "651": 6.32808,
+ "652": 6.24952,
+ "653": 6.37763,
+ "654": 6.445,
+ "655": 6.52173,
+ "656": 6.32265,
+ "657": 6.42629,
+ "658": 6.23889,
+ "659": 6.15329,
+ "660": 6.3884,
+ "661": 6.1652,
+ "662": 6.27045,
+ "663": 6.36828,
+ "664": 6.32903,
+ "665": 6.3993,
+ "666": 6.15825,
+ "667": 6.19378,
+ "668": 6.23345,
+ "669": 6.21216,
+ "670": 6.245,
+ "671": 6.2375,
+ "672": 6.48145,
+ "673": 6.33748,
+ "674": 6.29174,
+ "675": 6.38139,
+ "676": 6.38832,
+ "677": 6.30233,
+ "678": 6.27507,
+ "679": 6.23582,
+ "680": 6.29027,
+ "681": 6.19951,
+ "682": 6.08765,
+ "683": 6.27149,
+ "684": 6.32413,
+ "685": 6.29661,
+ "686": 6.14975,
+ "687": 6.28457,
+ "688": 6.20904,
+ "689": 6.62056,
+ "690": 6.17684,
+ "691": 6.18112,
+ "692": 6.27757,
+ "693": 6.14708,
+ "694": 6.23815,
+ "695": 6.33044,
+ "696": 6.12004,
+ "697": 6.15828,
+ "698": 6.23233,
+ "699": 6.46352,
+ "700": 6.04695,
+ "701": 6.06623,
+ "702": 6.25097,
+ "703": 6.1846,
+ "704": 6.21911,
+ "705": 6.12825,
+ "706": 6.0786,
+ "707": 6.25466,
+ "708": 6.31924,
+ "709": 6.01114,
+ "710": 6.16838,
+ "711": 6.26378,
+ "712": 6.18394,
+ "713": 5.89995,
+ "714": 6.10823,
+ "715": 6.11653,
+ "716": 6.41497,
+ "717": 6.19445,
+ "718": 6.24143,
+ "719": 6.2761,
+ "720": 6.26698,
+ "721": 6.2661,
+ "722": 6.23601,
+ "723": 6.08316,
+ "724": 6.2345,
+ "725": 6.04158,
+ "726": 6.30991,
+ "727": 6.02042,
+ "728": 6.0501,
+ "729": 6.09482,
+ "730": 6.18854,
+ "731": 6.10516,
+ "732": 6.09285,
+ "733": 6.13031,
+ "734": 6.38839,
+ "735": 6.28027,
+ "736": 6.18382,
+ "737": 6.36925,
+ "738": 6.13689,
+ "739": 6.15232,
+ "740": 5.88594,
+ "741": 6.00792,
+ "742": 5.98957,
+ "743": 6.17823,
+ "744": 6.03424,
+ "745": 6.1567,
+ "746": 6.04059,
+ "747": 6.10409,
+ "748": 6.2356,
+ "749": 5.94136,
+ "750": 6.17177,
+ "751": 5.96385,
+ "752": 6.02466,
+ "753": 6.03424,
+ "754": 6.29215,
+ "755": 6.13882,
+ "756": 6.2561,
+ "757": 6.02521,
+ "758": 6.20756,
+ "759": 6.2311,
+ "760": 6.02939,
+ "761": 6.1971,
+ "762": 6.23012,
+ "763": 6.04078,
+ "764": 5.96952,
+ "765": 5.9354,
+ "766": 5.97461,
+ "767": 5.81971,
+ "768": 6.19095,
+ "769": 6.28344,
+ "770": 6.30103,
+ "771": 5.79528,
+ "772": 6.03625,
+ "773": 6.18968,
+ "774": 5.88869,
+ "775": 6.03264,
+ "776": 6.13193,
+ "777": 5.88453,
+ "778": 6.06083,
+ "779": 5.87453,
+ "780": 6.14178,
+ "781": 5.85802,
+ "782": 6.04847,
+ "783": 5.95877,
+ "784": 5.92494,
+ "785": 6.09827,
+ "786": 6.10716,
+ "787": 5.6638,
+ "788": 6.00456,
+ "789": 6.21604,
+ "790": 6.26951,
+ "791": 5.79834,
+ "792": 5.99663,
+ "793": 6.1873,
+ "794": 6.03582,
+ "795": 6.01137,
+ "796": 6.16906,
+ "797": 6.06259,
+ "798": 6.06325,
+ "799": 6.11752,
+ "800": 6.01826,
+ "801": 6.15331,
+ "802": 5.98645,
+ "803": 6.1567,
+ "804": 6.01249,
+ "805": 5.8321,
+ "806": 6.09351,
+ "807": 6.05258,
+ "808": 5.93147,
+ "809": 5.78121,
+ "810": 6.02158,
+ "811": 5.93862,
+ "812": 5.91374,
+ "813": 5.96876,
+ "814": 6.03185,
+ "815": 5.8161,
+ "816": 6.12139,
+ "817": 5.94133,
+ "818": 6.06895,
+ "819": 6.01386,
+ "820": 5.73342,
+ "821": 5.95149,
+ "822": 6.20138,
+ "823": 5.83323,
+ "824": 5.98835,
+ "825": 6.19416,
+ "826": 6.20337,
+ "827": 6.06215,
+ "828": 6.07486,
+ "829": 5.89302,
+ "830": 5.94537,
+ "831": 5.90329,
+ "832": 5.9734,
+ "833": 6.06972,
+ "834": 5.99953,
+ "835": 6.00636,
+ "836": 5.79968,
+ "837": 6.11124,
+ "838": 5.87059,
+ "839": 5.83784,
+ "840": 6.18682,
+ "841": 5.78673,
+ "842": 5.8942,
+ "843": 5.95186,
+ "844": 6.01236,
+ "845": 6.09325,
+ "846": 5.69279,
+ "847": 5.76088,
+ "848": 5.96822,
+ "849": 6.09659,
+ "850": 5.85008,
+ "851": 6.02064,
+ "852": 5.7549,
+ "853": 5.99249,
+ "854": 6.02591,
+ "855": 5.82332,
+ "856": 6.00559,
+ "857": 6.00876,
+ "858": 6.06126,
+ "859": 5.96263,
+ "860": 6.10143,
+ "861": 6.0713,
+ "862": 6.00729,
+ "863": 5.84156,
+ "864": 5.84538,
+ "865": 5.94134,
+ "866": 5.89421,
+ "867": 5.88242,
+ "868": 6.06938,
+ "869": 6.09142,
+ "870": 5.97442,
+ "871": 6.04661,
+ "872": 5.89967,
+ "873": 5.84923,
+ "874": 6.03076,
+ "875": 5.91557,
+ "876": 5.97185,
+ "877": 5.93152,
+ "878": 6.10746,
+ "879": 5.77077,
+ "880": 6.01742,
+ "881": 6.00036,
+ "882": 5.91401,
+ "883": 5.68283,
+ "884": 5.96906,
+ "885": 5.75256,
+ "886": 5.9967,
+ "887": 5.91851,
+ "888": 5.84352,
+ "889": 6.01651,
+ "890": 6.02217,
+ "891": 5.95309,
+ "892": 5.71323,
+ "893": 6.09352,
+ "894": 5.73208,
+ "895": 5.84382,
+ "896": 5.84487,
+ "897": 5.86047,
+ "898": 5.92753,
+ "899": 5.94241,
+ "900": 5.90203,
+ "901": 5.9558,
+ "902": 5.83692,
+ "903": 6.05398,
+ "904": 5.93616,
+ "905": 5.90785,
+ "906": 5.62466,
+ "907": 5.91429,
+ "908": 5.74173,
+ "909": 5.99357,
+ "910": 5.86501,
+ "911": 5.70537,
+ "912": 5.70735,
+ "913": 5.76886,
+ "914": 5.83068,
+ "915": 5.80098,
+ "916": 5.89241,
+ "917": 5.87216,
+ "918": 5.82511,
+ "919": 5.81479,
+ "920": 5.89928,
+ "921": 5.84951,
+ "922": 5.62953,
+ "923": 6.04284,
+ "924": 5.61664,
+ "925": 5.6284,
+ "926": 5.86602,
+ "927": 5.96465,
+ "928": 5.84558,
+ "929": 5.831,
+ "930": 5.96078,
+ "931": 5.7636,
+ "932": 5.59927,
+ "933": 5.64075,
+ "934": 5.81637,
+ "935": 5.6448,
+ "936": 5.83809,
+ "937": 5.9661,
+ "938": 5.59466,
+ "939": 5.7903,
+ "940": 5.9661,
+ "941": 5.7269,
+ "942": 5.83517,
+ "943": 5.87119,
+ "944": 5.95685,
+ "945": 5.70597,
+ "946": 5.56301,
+ "947": 5.75282,
+ "948": 5.79766,
+ "949": 5.83042,
+ "950": 5.85196,
+ "951": 5.72778,
+ "952": 5.69963,
+ "953": 5.68442,
+ "954": 5.73234,
+ "955": 5.5349,
+ "956": 5.63088,
+ "957": 5.84662,
+ "958": 5.80508,
+ "959": 5.5795,
+ "960": 5.8075,
+ "961": 5.83647,
+ "962": 5.77697,
+ "963": 5.7727,
+ "964": 5.71274,
+ "965": 5.64476,
+ "966": 5.60853,
+ "967": 5.7317,
+ "968": 5.74358,
+ "969": 5.83164,
+ "970": 5.65593,
+ "971": 5.71414,
+ "972": 5.85748,
+ "973": 5.67758,
+ "974": 5.72305,
+ "975": 5.86483,
+ "976": 5.71632,
+ "977": 5.77972,
+ "978": 5.69181,
+ "979": 5.59759,
+ "980": 5.77009,
+ "981": 5.90349,
+ "982": 5.47759,
+ "983": 5.62392,
+ "984": 5.55496,
+ "985": 5.59426,
+ "986": 5.6446,
+ "987": 5.57738,
+ "988": 5.71596,
+ "989": 5.69944,
+ "990": 5.63204,
+ "991": 5.85272,
+ "992": 5.77703,
+ "993": 5.87546,
+ "994": 5.70447,
+ "995": 5.73744,
+ "996": 5.74525,
+ "997": 5.82485,
+ "998": 5.84481,
+ "999": 5.83892,
+ "1000": 5.6907,
+ "1001": 5.87569,
+ "1002": 5.76592,
+ "1003": 5.64754,
+ "1004": 5.80524,
+ "1005": 5.53839,
+ "1006": 5.33588,
+ "1007": 5.77605,
+ "1008": 5.80078,
+ "1009": 5.66068,
+ "1010": 5.78916,
+ "1011": 5.90288,
+ "1012": 5.62617,
+ "1013": 5.62099,
+ "1014": 5.68464,
+ "1015": 5.56766,
+ "1016": 5.87782,
+ "1017": 5.83714,
+ "1018": 5.62895,
+ "1019": 5.73628,
+ "1020": 5.61812,
+ "1021": 5.85843,
+ "1022": 5.50148,
+ "1023": 5.65666,
+ "1024": 5.749,
+ "1025": 5.5786,
+ "1026": 5.41747,
+ "1027": 5.60617,
+ "1028": 5.69657,
+ "1029": 5.68979,
+ "1030": 5.69331,
+ "1031": 5.41272,
+ "1032": 5.78846,
+ "1033": 5.58042,
+ "1034": 5.62925,
+ "1035": 5.71915,
+ "1036": 5.62969,
+ "1037": 5.37061,
+ "1038": 5.66767,
+ "1039": 5.6504,
+ "1040": 5.58195,
+ "1041": 5.60135,
+ "1042": 5.83001,
+ "1043": 5.57307,
+ "1044": 5.47853,
+ "1045": 5.97393,
+ "1046": 5.49969,
+ "1047": 5.40206,
+ "1048": 5.50344,
+ "1049": 5.68341,
+ "1050": 5.7042,
+ "1051": 5.58457,
+ "1052": 5.68806,
+ "1053": 5.63698,
+ "1054": 5.46791,
+ "1055": 5.60489,
+ "1056": 5.68481,
+ "1057": 5.76512,
+ "1058": 5.57732,
+ "1059": 5.76013,
+ "1060": 5.83194,
+ "1061": 5.47992,
+ "1062": 5.66633,
+ "1063": 5.5129,
+ "1064": 5.60119,
+ "1065": 5.56186,
+ "1066": 5.75508,
+ "1067": 5.68066,
+ "1068": 5.44601,
+ "1069": 5.61739,
+ "1070": 5.81763,
+ "1071": 5.51677,
+ "1072": 5.62937,
+ "1073": 5.62873,
+ "1074": 5.5305,
+ "1075": 5.71786,
+ "1076": 5.6033,
+ "1077": 5.72295,
+ "1078": 5.5746,
+ "1079": 5.62918,
+ "1080": 5.65223,
+ "1081": 5.63546,
+ "1082": 5.51023,
+ "1083": 5.65326,
+ "1084": 5.55981,
+ "1085": 5.41844,
+ "1086": 5.63271,
+ "1087": 5.45047,
+ "1088": 5.52241,
+ "1089": 5.77754,
+ "1090": 5.53785,
+ "1091": 5.52475,
+ "1092": 5.42003,
+ "1093": 5.70636,
+ "1094": 5.58207,
+ "1095": 5.58808,
+ "1096": 5.62735,
+ "1097": 5.65611,
+ "1098": 5.66246,
+ "1099": 5.52514,
+ "1100": 5.65315,
+ "1101": 5.69158,
+ "1102": 5.54437,
+ "1103": 5.55823,
+ "1104": 5.54948,
+ "1105": 5.56259,
+ "1106": 5.69503,
+ "1107": 5.68901,
+ "1108": 5.7915,
+ "1109": 5.54791,
+ "1110": 5.67063,
+ "1111": 5.59722,
+ "1112": 5.59618,
+ "1113": 5.63703,
+ "1114": 5.62484,
+ "1115": 5.60963,
+ "1116": 5.67482,
+ "1117": 5.65511,
+ "1118": 5.66516,
+ "1119": 5.71938,
+ "1120": 5.64848,
+ "1121": 5.3869,
+ "1122": 5.24022,
+ "1123": 5.48578,
+ "1124": 5.66499,
+ "1125": 5.69357,
+ "1126": 5.691,
+ "1127": 5.57811,
+ "1128": 5.63072,
+ "1129": 5.3042,
+ "1130": 5.55179,
+ "1131": 5.63538,
+ "1132": 5.73405,
+ "1133": 5.52486,
+ "1134": 5.56154,
+ "1135": 5.52997,
+ "1136": 5.43031,
+ "1137": 5.47657,
+ "1138": 5.57446,
+ "1139": 5.4261,
+ "1140": 5.27187,
+ "1141": 5.59712,
+ "1142": 5.65179,
+ "1143": 5.39838,
+ "1144": 5.39758,
+ "1145": 5.37476,
+ "1146": 5.64253,
+ "1147": 5.49683,
+ "1148": 5.51407,
+ "1149": 5.53215,
+ "1150": 5.40621,
+ "1151": 5.56645,
+ "1152": 5.41723,
+ "1153": 5.45911,
+ "1154": 5.49961,
+ "1155": 5.44064,
+ "1156": 5.34938,
+ "1157": 5.66505,
+ "1158": 5.40107,
+ "1159": 5.33945,
+ "1160": 5.79858,
+ "1161": 5.54893,
+ "1162": 5.46395,
+ "1163": 5.53698,
+ "1164": 5.39266,
+ "1165": 5.53988,
+ "1166": 5.50107,
+ "1167": 5.369,
+ "1168": 5.50378,
+ "1169": 5.40913,
+ "1170": 5.59704,
+ "1171": 5.49144,
+ "1172": 5.64815,
+ "1173": 5.63097,
+ "1174": 5.51805,
+ "1175": 5.35096,
+ "1176": 5.38824,
+ "1177": 5.56196,
+ "1178": 5.47408,
+ "1179": 5.50159,
+ "1180": 5.47138,
+ "1181": 5.5607,
+ "1182": 5.60403,
+ "1183": 5.77432,
+ "1184": 5.55393,
+ "1185": 5.29819,
+ "1186": 5.61417,
+ "1187": 5.56234,
+ "1188": 5.52362,
+ "1189": 5.396,
+ "1190": 5.40968,
+ "1191": 5.39503,
+ "1192": 5.50482,
+ "1193": 5.47101,
+ "1194": 5.46109,
+ "1195": 5.33163,
+ "1196": 5.52828,
+ "1197": 5.48547,
+ "1198": 5.53238,
+ "1199": 5.39388,
+ "1200": 5.33533,
+ "1201": 5.49594,
+ "1202": 5.44331,
+ "1203": 5.50076,
+ "1204": 5.41726,
+ "1205": 5.49796,
+ "1206": 5.34438,
+ "1207": 5.593,
+ "1208": 5.42955,
+ "1209": 5.30177,
+ "1210": 5.5067,
+ "1211": 5.51947,
+ "1212": 5.60376,
+ "1213": 5.42751,
+ "1214": 5.51066,
+ "1215": 5.24567,
+ "1216": 5.42092,
+ "1217": 5.39233,
+ "1218": 5.45748,
+ "1219": 5.49082,
+ "1220": 5.39042,
+ "1221": 5.45966,
+ "1222": 5.31736,
+ "1223": 5.484,
+ "1224": 5.42184,
+ "1225": 5.43552,
+ "1226": 5.32782,
+ "1227": 5.48101,
+ "1228": 5.73667,
+ "1229": 5.32755,
+ "1230": 5.4164,
+ "1231": 5.08,
+ "1232": 5.79336,
+ "1233": 5.29455,
+ "1234": 5.24901,
+ "1235": 5.37658,
+ "1236": 5.48789,
+ "1237": 5.21279,
+ "1238": 5.41996,
+ "1239": 5.41148,
+ "1240": 5.4706,
+ "1241": 5.58064,
+ "1242": 5.45972,
+ "1243": 5.43956,
+ "1244": 5.52374,
+ "1245": 5.20054,
+ "1246": 5.72403,
+ "1247": 5.43472,
+ "1248": 5.30553,
+ "1249": 5.40658,
+ "1250": 5.34624,
+ "1251": 5.42674,
+ "1252": 5.57988,
+ "1253": 5.49422,
+ "1254": 5.31494,
+ "1255": 5.52069,
+ "1256": 5.6145,
+ "1257": 5.43002,
+ "1258": 5.56337,
+ "1259": 5.48612,
+ "1260": 5.51488,
+ "1261": 5.64614,
+ "1262": 5.40278,
+ "1263": 5.33399,
+ "1264": 5.51306,
+ "1265": 5.31273,
+ "1266": 5.24419,
+ "1267": 5.37621,
+ "1268": 5.3936,
+ "1269": 5.15905,
+ "1270": 5.40456,
+ "1271": 5.28646,
+ "1272": 5.52854,
+ "1273": 5.30305,
+ "1274": 5.35324,
+ "1275": 5.38788,
+ "1276": 5.40318,
+ "1277": 5.46821,
+ "1278": 5.36115,
+ "1279": 5.44454,
+ "1280": 5.46697,
+ "1281": 5.40799,
+ "1282": 5.39014,
+ "1283": 5.42903,
+ "1284": 5.34895,
+ "1285": 5.51455,
+ "1286": 5.33959,
+ "1287": 5.59257,
+ "1288": 5.27215,
+ "1289": 5.4352,
+ "1290": 5.50862,
+ "1291": 5.50876,
+ "1292": 5.45545,
+ "1293": 5.4245,
+ "1294": 5.50132,
+ "1295": 5.35729,
+ "1296": 5.19368,
+ "1297": 5.17933,
+ "1298": 5.12454,
+ "1299": 5.30931,
+ "1300": 5.21692,
+ "1301": 5.31192,
+ "1302": 5.28326,
+ "1303": 5.36305,
+ "1304": 5.4399,
+ "1305": 5.3762,
+ "1306": 5.26052,
+ "1307": 5.19572,
+ "1308": 5.27448,
+ "1309": 5.41237,
+ "1310": 5.26681,
+ "1311": 5.38549,
+ "1312": 5.36152,
+ "1313": 5.30098,
+ "1314": 5.29659,
+ "1315": 5.42657,
+ "1316": 5.26597,
+ "1317": 5.28466,
+ "1318": 5.22082,
+ "1319": 5.34765,
+ "1320": 5.42273,
+ "1321": 5.45484,
+ "1322": 5.46787,
+ "1323": 5.37867,
+ "1324": 5.25895,
+ "1325": 5.41311,
+ "1326": 5.54702,
+ "1327": 5.39603,
+ "1328": 5.22477,
+ "1329": 5.4262,
+ "1330": 5.40755,
+ "1331": 5.31986,
+ "1332": 5.31812,
+ "1333": 5.38065,
+ "1334": 5.45166,
+ "1335": 5.37545,
+ "1336": 5.44626,
+ "1337": 5.47777,
+ "1338": 5.31196,
+ "1339": 5.14259,
+ "1340": 5.42266,
+ "1341": 5.34523,
+ "1342": 5.35986,
+ "1343": 5.47822,
+ "1344": 5.38455,
+ "1345": 5.34841,
+ "1346": 5.08361,
+ "1347": 5.39313,
+ "1348": 5.49724,
+ "1349": 5.41756,
+ "1350": 5.03551,
+ "1351": 5.32151,
+ "1352": 5.16669,
+ "1353": 5.34933,
+ "1354": 5.36585,
+ "1355": 5.11872,
+ "1356": 5.26492,
+ "1357": 5.29438,
+ "1358": 5.16735,
+ "1359": 5.1177,
+ "1360": 5.17836,
+ "1361": 5.31395,
+ "1362": 5.07729,
+ "1363": 5.30208,
+ "1364": 5.40541,
+ "1365": 5.03041,
+ "1366": 5.12212,
+ "1367": 5.3431,
+ "1368": 5.19169,
+ "1369": 5.23413,
+ "1370": 5.20833,
+ "1371": 5.28887,
+ "1372": 5.26922,
+ "1373": 5.29207,
+ "1374": 5.2866,
+ "1375": 5.46594,
+ "1376": 5.27158,
+ "1377": 5.27468,
+ "1378": 5.31995,
+ "1379": 5.23318,
+ "1380": 5.26423,
+ "1381": 5.48623,
+ "1382": 5.09237,
+ "1383": 5.38149,
+ "1384": 5.36601,
+ "1385": 5.39587,
+ "1386": 5.17179,
+ "1387": 5.16778,
+ "1388": 5.28306,
+ "1389": 5.30867,
+ "1390": 5.26124,
+ "1391": 5.27205,
+ "1392": 5.37826,
+ "1393": 5.38761,
+ "1394": 5.40922,
+ "1395": 5.33065,
+ "1396": 5.22161,
+ "1397": 5.28545,
+ "1398": 5.37338,
+ "1399": 5.36406,
+ "1400": 5.272,
+ "1401": 5.36768,
+ "1402": 5.42906,
+ "1403": 5.2051,
+ "1404": 5.2881,
+ "1405": 5.12101,
+ "1406": 4.99395,
+ "1407": 5.40297,
+ "1408": 5.1996,
+ "1409": 5.39905,
+ "1410": 5.37542,
+ "1411": 4.92212,
+ "1412": 5.35983,
+ "1413": 5.42065,
+ "1414": 5.22194,
+ "1415": 5.44705,
+ "1416": 5.33479,
+ "1417": 5.40003,
+ "1418": 5.30372,
+ "1419": 5.32065,
+ "1420": 5.44511,
+ "1421": 5.40092,
+ "1422": 5.4268,
+ "1423": 5.00931,
+ "1424": 5.34022,
+ "1425": 5.59553,
+ "1426": 5.23718,
+ "1427": 5.32579,
+ "1428": 5.34165,
+ "1429": 5.08678,
+ "1430": 5.33487,
+ "1431": 5.33063,
+ "1432": 5.3492,
+ "1433": 5.19402,
+ "1434": 5.16864,
+ "1435": 5.21003,
+ "1436": 5.11103,
+ "1437": 5.23292,
+ "1438": 5.32199,
+ "1439": 5.35624,
+ "1440": 5.35036,
+ "1441": 5.176,
+ "1442": 5.22766,
+ "1443": 5.21302,
+ "1444": 5.1377,
+ "1445": 5.08426,
+ "1446": 5.27121,
+ "1447": 5.26171,
+ "1448": 5.29977,
+ "1449": 5.25382,
+ "1450": 5.34992,
+ "1451": 5.07793,
+ "1452": 5.27596,
+ "1453": 5.17685,
+ "1454": 5.0179,
+ "1455": 5.12978,
+ "1456": 5.2791,
+ "1457": 5.19237,
+ "1458": 5.01255,
+ "1459": 5.22709,
+ "1460": 5.24067,
+ "1461": 5.0814,
+ "1462": 4.97051,
+ "1463": 5.15304,
+ "1464": 5.21787,
+ "1465": 5.27321,
+ "1466": 5.35166,
+ "1467": 5.3427,
+ "1468": 5.22962,
+ "1469": 5.05147,
+ "1470": 5.12485,
+ "1471": 5.25512,
+ "1472": 5.12462,
+ "1473": 5.10977,
+ "1474": 5.22399,
+ "1475": 5.19193,
+ "1476": 5.1622,
+ "1477": 5.26842,
+ "1478": 5.31298,
+ "1479": 5.01426,
+ "1480": 5.18717,
+ "1481": 5.25419,
+ "1482": 5.35537,
+ "1483": 5.27414,
+ "1484": 4.93321,
+ "1485": 5.29759,
+ "1486": 5.05298,
+ "1487": 4.88898,
+ "1488": 5.18445,
+ "1489": 5.10557,
+ "1490": 5.04776,
+ "1491": 5.32238,
+ "1492": 5.22618,
+ "1493": 4.94537,
+ "1494": 5.11156,
+ "1495": 5.13704,
+ "1496": 5.06068,
+ "1497": 5.36803,
+ "1498": 5.30875,
+ "1499": 5.14382,
+ "1500": 5.09918,
+ "1501": 5.03819,
+ "1502": 5.1556,
+ "1503": 5.43756,
+ "1504": 5.3277,
+ "1505": 5.01077,
+ "1506": 5.14855,
+ "1507": 5.16599,
+ "1508": 5.17149,
+ "1509": 5.32003,
+ "1510": 5.02729,
+ "1511": 5.12232,
+ "1512": 4.98459,
+ "1513": 5.17351,
+ "1514": 5.33797,
+ "1515": 5.36456,
+ "1516": 5.27738,
+ "1517": 5.22885,
+ "1518": 5.03482,
+ "1519": 5.30173,
+ "1520": 5.1419,
+ "1521": 5.16165,
+ "1522": 5.33835,
+ "1523": 5.24514,
+ "1524": 5.07441,
+ "1525": 5.21523,
+ "1526": 5.28565,
+ "1527": 5.26125,
+ "1528": 5.24319,
+ "1529": 5.18673,
+ "1530": 5.24399,
+ "1531": 5.10319,
+ "1532": 5.16061,
+ "1533": 5.05209,
+ "1534": 5.22512,
+ "1535": 5.17011,
+ "1536": 5.10499,
+ "1537": 5.03437,
+ "1538": 4.92156,
+ "1539": 5.24017,
+ "1540": 5.11703,
+ "1541": 5.25771,
+ "1542": 5.24207,
+ "1543": 5.05683,
+ "1544": 5.08282,
+ "1545": 5.12142,
+ "1546": 5.32949,
+ "1547": 5.11078,
+ "1548": 5.23843,
+ "1549": 5.23514,
+ "1550": 4.97465,
+ "1551": 5.26341,
+ "1552": 5.02487,
+ "1553": 5.15396,
+ "1554": 5.11398,
+ "1555": 5.11659,
+ "1556": 5.2043,
+ "1557": 5.09464,
+ "1558": 5.23535,
+ "1559": 5.00696,
+ "1560": 5.11615,
+ "1561": 5.15114,
+ "1562": 5.18687,
+ "1563": 5.24967,
+ "1564": 5.26861,
+ "1565": 5.09285,
+ "1566": 5.29624,
+ "1567": 5.0476,
+ "1568": 5.09072,
+ "1569": 5.20294,
+ "1570": 5.17486,
+ "1571": 4.95548,
+ "1572": 5.04718,
+ "1573": 5.03159,
+ "1574": 4.99832,
+ "1575": 5.23515,
+ "1576": 5.21197,
+ "1577": 5.13219,
+ "1578": 5.36691,
+ "1579": 4.94577,
+ "1580": 5.13091,
+ "1581": 5.10498,
+ "1582": 5.28558,
+ "1583": 5.0508,
+ "1584": 5.06036,
+ "1585": 5.12376,
+ "1586": 5.30984,
+ "1587": 5.13645,
+ "1588": 5.22716,
+ "1589": 4.84242,
+ "1590": 5.10332,
+ "1591": 5.18406,
+ "1592": 5.14231,
+ "1593": 5.23937,
+ "1594": 5.11902,
+ "1595": 5.11146,
+ "1596": 5.19604,
+ "1597": 5.11829,
+ "1598": 5.16546,
+ "1599": 5.19379,
+ "1600": 4.87282,
+ "1601": 5.12554,
+ "1602": 5.2348,
+ "1603": 5.19975,
+ "1604": 5.06036,
+ "1605": 5.03819,
+ "1606": 4.99511,
+ "1607": 5.07386,
+ "1608": 4.98739,
+ "1609": 5.07952,
+ "1610": 5.05422,
+ "1611": 4.99815,
+ "1612": 4.76476,
+ "1613": 5.03911,
+ "1614": 4.88654,
+ "1615": 5.08306,
+ "1616": 5.23126,
+ "1617": 5.07139,
+ "1618": 4.99111,
+ "1619": 5.188,
+ "1620": 5.14704,
+ "1621": 5.31775,
+ "1622": 5.06806,
+ "1623": 5.1518,
+ "1624": 5.13655,
+ "1625": 5.12558,
+ "1626": 5.10658,
+ "1627": 5.11532,
+ "1628": 5.06477,
+ "1629": 4.93911,
+ "1630": 5.06725,
+ "1631": 5.06209,
+ "1632": 5.10762,
+ "1633": 4.97665,
+ "1634": 4.92599,
+ "1635": 5.05765,
+ "1636": 4.9234,
+ "1637": 5.23589,
+ "1638": 5.16223,
+ "1639": 4.98259,
+ "1640": 5.0088,
+ "1641": 5.12901,
+ "1642": 5.08651,
+ "1643": 5.05407,
+ "1644": 5.12349,
+ "1645": 4.96758,
+ "1646": 5.12589,
+ "1647": 5.03714,
+ "1648": 5.195,
+ "1649": 4.92688,
+ "1650": 5.06195,
+ "1651": 4.93667,
+ "1652": 5.2142,
+ "1653": 5.16765,
+ "1654": 5.13544,
+ "1655": 5.16694,
+ "1656": 5.34867,
+ "1657": 5.21355,
+ "1658": 5.04428,
+ "1659": 4.93467,
+ "1660": 4.81625,
+ "1661": 5.0324,
+ "1662": 5.14711,
+ "1663": 5.16386,
+ "1664": 4.9897,
+ "1665": 5.11329,
+ "1666": 5.11174,
+ "1667": 4.85634,
+ "1668": 5.11543,
+ "1669": 5.07792,
+ "1670": 5.11575,
+ "1671": 5.17325,
+ "1672": 4.78012,
+ "1673": 5.03951,
+ "1674": 4.92276,
+ "1675": 5.05352,
+ "1676": 5.0099,
+ "1677": 4.80545,
+ "1678": 5.03315,
+ "1679": 4.89183,
+ "1680": 5.04424,
+ "1681": 5.06702,
+ "1682": 5.04422,
+ "1683": 4.90886,
+ "1684": 5.0687,
+ "1685": 5.13695,
+ "1686": 5.08186,
+ "1687": 4.98422,
+ "1688": 5.1747,
+ "1689": 5.1566,
+ "1690": 4.99763,
+ "1691": 5.0037,
+ "1692": 4.91606,
+ "1693": 5.02553,
+ "1694": 4.95156,
+ "1695": 4.92004,
+ "1696": 5.09039,
+ "1697": 5.05745,
+ "1698": 4.957,
+ "1699": 5.01405,
+ "1700": 4.95053,
+ "1701": 5.17374,
+ "1702": 5.07909,
+ "1703": 5.17033,
+ "1704": 5.14606,
+ "1705": 4.96693,
+ "1706": 4.98798,
+ "1707": 4.78911,
+ "1708": 5.03515,
+ "1709": 5.23405,
+ "1710": 5.03491,
+ "1711": 5.19432,
+ "1712": 5.19844,
+ "1713": 5.03948,
+ "1714": 5.05126,
+ "1715": 4.92306,
+ "1716": 4.94219,
+ "1717": 4.87086,
+ "1718": 5.03153,
+ "1719": 5.13099,
+ "1720": 5.02792,
+ "1721": 4.94099,
+ "1722": 5.06904,
+ "1723": 4.94202,
+ "1724": 5.04228,
+ "1725": 5.19765,
+ "1726": 5.07047,
+ "1727": 4.91798,
+ "1728": 5.02467,
+ "1729": 5.0549,
+ "1730": 4.91477,
+ "1731": 5.00546,
+ "1732": 4.91948,
+ "1733": 5.13533,
+ "1734": 4.83617,
+ "1735": 5.21555,
+ "1736": 4.92223,
+ "1737": 4.86817,
+ "1738": 4.98217,
+ "1739": 5.16787,
+ "1740": 4.84165,
+ "1741": 4.78313,
+ "1742": 4.91274,
+ "1743": 5.09646,
+ "1744": 4.98717,
+ "1745": 4.83246,
+ "1746": 4.95384,
+ "1747": 4.87336,
+ "1748": 5.07231,
+ "1749": 4.86869,
+ "1750": 5.0149,
+ "1751": 5.12799,
+ "1752": 4.91171,
+ "1753": 5.09495,
+ "1754": 5.05988,
+ "1755": 4.89813,
+ "1756": 5.02432,
+ "1757": 5.14562,
+ "1758": 4.87278,
+ "1759": 4.95172,
+ "1760": 4.83598,
+ "1761": 5.02863,
+ "1762": 4.8233,
+ "1763": 4.77993,
+ "1764": 4.93801,
+ "1765": 5.15365,
+ "1766": 5.34417,
+ "1767": 5.23015,
+ "1768": 4.94996,
+ "1769": 5.00542,
+ "1770": 4.9869,
+ "1771": 4.96844,
+ "1772": 4.98718,
+ "1773": 4.97768,
+ "1774": 4.87708,
+ "1775": 4.95647,
+ "1776": 4.99905,
+ "1777": 4.95187,
+ "1778": 4.99752,
+ "1779": 5.08936,
+ "1780": 4.84259,
+ "1781": 5.05873,
+ "1782": 5.00269,
+ "1783": 5.02237,
+ "1784": 4.93924,
+ "1785": 5.17368,
+ "1786": 4.81539,
+ "1787": 4.97855,
+ "1788": 4.83119,
+ "1789": 4.88708,
+ "1790": 4.80557,
+ "1791": 4.74722,
+ "1792": 4.88279,
+ "1793": 5.11274,
+ "1794": 4.99248,
+ "1795": 4.97577,
+ "1796": 5.00708,
+ "1797": 4.79358,
+ "1798": 4.77536,
+ "1799": 5.02564,
+ "1800": 4.91625,
+ "1801": 5.0544,
+ "1802": 4.82988,
+ "1803": 4.95935,
+ "1804": 4.89046,
+ "1805": 4.9127,
+ "1806": 4.88786,
+ "1807": 4.9323,
+ "1808": 4.92909,
+ "1809": 5.15547,
+ "1810": 5.10201,
+ "1811": 4.96646,
+ "1812": 4.81285,
+ "1813": 5.10649,
+ "1814": 4.78734,
+ "1815": 4.8686,
+ "1816": 5.05382,
+ "1817": 4.79906,
+ "1818": 4.80993,
+ "1819": 5.03015,
+ "1820": 4.69424,
+ "1821": 5.02943,
+ "1822": 4.6676,
+ "1823": 4.87459,
+ "1824": 4.8006,
+ "1825": 5.07318,
+ "1826": 4.82421,
+ "1827": 4.80355,
+ "1828": 4.95872,
+ "1829": 5.11451,
+ "1830": 4.92372,
+ "1831": 4.90408,
+ "1832": 4.83944,
+ "1833": 4.79594,
+ "1834": 4.95127,
+ "1835": 4.9632,
+ "1836": 4.91644,
+ "1837": 4.67872,
+ "1838": 4.81311,
+ "1839": 4.90503,
+ "1840": 4.91458,
+ "1841": 4.84667,
+ "1842": 4.96111,
+ "1843": 4.71579,
+ "1844": 4.62263,
+ "1845": 5.01032,
+ "1846": 4.75861,
+ "1847": 4.86874,
+ "1848": 4.91168,
+ "1849": 4.8548,
+ "1850": 4.8728,
+ "1851": 5.01984,
+ "1852": 4.98368,
+ "1853": 4.8321,
+ "1854": 4.86982,
+ "1855": 4.82613,
+ "1856": 4.7585,
+ "1857": 4.9702,
+ "1858": 4.97243,
+ "1859": 4.75763,
+ "1860": 4.87015,
+ "1861": 5.2155,
+ "1862": 4.61983,
+ "1863": 4.84121,
+ "1864": 4.75684,
+ "1865": 4.8705,
+ "1866": 4.79949,
+ "1867": 5.0133,
+ "1868": 4.7256,
+ "1869": 4.76912,
+ "1870": 4.94695,
+ "1871": 5.00622,
+ "1872": 4.68944,
+ "1873": 4.70357,
+ "1874": 4.85396,
+ "1875": 4.85816,
+ "1876": 4.74656,
+ "1877": 4.81047,
+ "1878": 4.81835,
+ "1879": 4.82843,
+ "1880": 4.90275,
+ "1881": 4.80004,
+ "1882": 4.80368,
+ "1883": 4.79001,
+ "1884": 4.9819,
+ "1885": 4.92449,
+ "1886": 4.83007,
+ "1887": 4.82592,
+ "1888": 4.97472,
+ "1889": 4.9682,
+ "1890": 4.7149,
+ "1891": 4.66146,
+ "1892": 4.8556,
+ "1893": 4.65319,
+ "1894": 4.90672,
+ "1895": 4.79755,
+ "1896": 4.66672,
+ "1897": 4.80011,
+ "1898": 4.92523,
+ "1899": 4.78238,
+ "1900": 4.91944,
+ "1901": 4.8578,
+ "1902": 4.78983,
+ "1903": 4.76699,
+ "1904": 4.65919,
+ "1905": 4.55447,
+ "1906": 4.82111,
+ "1907": 4.9157,
+ "1908": 5.03656,
+ "1909": 4.89756,
+ "1910": 4.79118,
+ "1911": 4.81797,
+ "1912": 4.6593,
+ "1913": 4.95552,
+ "1914": 4.88944,
+ "1915": 4.87053,
+ "1916": 4.93455,
+ "1917": 4.86059,
+ "1918": 4.87835,
+ "1919": 5.00177,
+ "1920": 4.77276,
+ "1921": 4.8912,
+ "1922": 4.82075,
+ "1923": 4.76053,
+ "1924": 4.83336,
+ "1925": 5.06421,
+ "1926": 4.94366,
+ "1927": 4.93372,
+ "1928": 4.93008,
+ "1929": 4.93572,
+ "1930": 4.91814,
+ "1931": 4.77823,
+ "1932": 4.86729,
+ "1933": 4.83739,
+ "1934": 4.8456,
+ "1935": 5.11401,
+ "1936": 4.8913,
+ "1937": 4.88548,
+ "1938": 4.80819,
+ "1939": 4.71066,
+ "1940": 4.83263,
+ "1941": 4.74525,
+ "1942": 4.88182,
+ "1943": 4.7456,
+ "1944": 4.75657,
+ "1945": 4.69693,
+ "1946": 4.92806,
+ "1947": 4.88115,
+ "1948": 4.60616,
+ "1949": 4.90548,
+ "1950": 4.8013,
+ "1951": 4.96918,
+ "1952": 4.7445,
+ "1953": 4.79985,
+ "1954": 4.74563,
+ "1955": 4.85863,
+ "1956": 4.88335,
+ "1957": 4.74162,
+ "1958": 4.70669,
+ "1959": 4.76879,
+ "1960": 4.77554,
+ "1961": 4.71832,
+ "1962": 4.83722,
+ "1963": 4.83508,
+ "1964": 4.85654,
+ "1965": 4.88125,
+ "1966": 4.79815,
+ "1967": 4.60289,
+ "1968": 4.83537,
+ "1969": 4.59996,
+ "1970": 4.5871,
+ "1971": 4.91156,
+ "1972": 4.90367,
+ "1973": 4.56255,
+ "1974": 4.83355,
+ "1975": 4.83492,
+ "1976": 4.72093,
+ "1977": 4.58721,
+ "1978": 5.01189,
+ "1979": 4.67192,
+ "1980": 4.75644,
+ "1981": 4.87937,
+ "1982": 4.72883,
+ "1983": 4.89794,
+ "1984": 4.6513,
+ "1985": 4.78983,
+ "1986": 4.70555,
+ "1987": 4.81876,
+ "1988": 4.8931,
+ "1989": 4.64299,
+ "1990": 4.803,
+ "1991": 4.70826,
+ "1992": 4.80856,
+ "1993": 4.74462,
+ "1994": 4.86177,
+ "1995": 4.56479,
+ "1996": 4.66056,
+ "1997": 4.81611,
+ "1998": 4.68273,
+ "1999": 4.73484,
+ "2000": 4.63373
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 27.0,
+ "2": 29.0,
+ "3": 41.0,
+ "4": 26.0,
+ "5": 37.0,
+ "6": 28.0,
+ "7": 38.0,
+ "8": 33.0,
+ "9": 23.0,
+ "10": 21.0,
+ "11": 40.0,
+ "12": 20.0,
+ "13": 30.0,
+ "14": 36.0,
+ "15": 31.0,
+ "16": 29.0,
+ "17": 37.0,
+ "18": 38.0,
+ "19": 26.0,
+ "20": 41.0,
+ "21": 39.0,
+ "22": 30.0,
+ "23": 41.0,
+ "24": 40.0,
+ "25": 32.0,
+ "26": 35.0,
+ "27": 30.0,
+ "28": 45.0,
+ "29": 34.0,
+ "30": 42.0,
+ "31": 44.0,
+ "32": 44.0,
+ "33": 50.0,
+ "34": 41.0,
+ "35": 46.0,
+ "36": 49.0,
+ "37": 49.0,
+ "38": 51.0,
+ "39": 50.0,
+ "40": 71.0,
+ "41": 62.0,
+ "42": 41.0,
+ "43": 52.0,
+ "44": 56.0,
+ "45": 53.0,
+ "46": 46.0,
+ "47": 66.0,
+ "48": 56.0,
+ "49": 54.0,
+ "50": 73.0,
+ "51": 45.0,
+ "52": 40.0,
+ "53": 49.0,
+ "54": 67.0,
+ "55": 51.0,
+ "56": 64.0,
+ "57": 58.0,
+ "58": 70.0,
+ "59": 63.0,
+ "60": 60.0,
+ "61": 55.0,
+ "62": 54.0,
+ "63": 60.0,
+ "64": 63.0,
+ "65": 66.0,
+ "66": 61.0,
+ "67": 72.0,
+ "68": 71.0,
+ "69": 57.0,
+ "70": 68.0,
+ "71": 84.0,
+ "72": 57.0,
+ "73": 61.0,
+ "74": 72.0,
+ "75": 59.0,
+ "76": 64.0,
+ "77": 65.0,
+ "78": 81.0,
+ "79": 52.0,
+ "80": 75.0,
+ "81": 73.0,
+ "82": 57.0,
+ "83": 72.0,
+ "84": 61.0,
+ "85": 67.0,
+ "86": 88.0,
+ "87": 48.0,
+ "88": 104.0,
+ "89": 56.0,
+ "90": 58.0,
+ "91": 73.0,
+ "92": 86.0,
+ "93": 59.0,
+ "94": 51.0,
+ "95": 42.0,
+ "96": 69.0,
+ "97": 95.0,
+ "98": 85.0,
+ "99": 67.0,
+ "100": 70.0,
+ "101": 64.0,
+ "102": 52.0,
+ "103": 65.0,
+ "104": 63.0,
+ "105": 81.0,
+ "106": 61.0,
+ "107": 74.0,
+ "108": 56.0,
+ "109": 70.0,
+ "110": 90.0,
+ "111": 86.0,
+ "112": 70.0,
+ "113": 78.0,
+ "114": 62.0,
+ "115": 65.0,
+ "116": 63.0,
+ "117": 64.0,
+ "118": 64.0,
+ "119": 76.0,
+ "120": 52.0,
+ "121": 81.0,
+ "122": 52.0,
+ "123": 80.0,
+ "124": 81.0,
+ "125": 78.0,
+ "126": 51.0,
+ "127": 69.0,
+ "128": 73.0,
+ "129": 75.0,
+ "130": 66.0,
+ "131": 75.0,
+ "132": 80.0,
+ "133": 67.0,
+ "134": 63.0,
+ "135": 55.0,
+ "136": 73.0,
+ "137": 64.0,
+ "138": 66.0,
+ "139": 56.0,
+ "140": 68.0,
+ "141": 70.0,
+ "142": 60.0,
+ "143": 64.0,
+ "144": 69.0,
+ "145": 66.0,
+ "146": 62.0,
+ "147": 64.0,
+ "148": 72.0,
+ "149": 65.0,
+ "150": 60.0,
+ "151": 54.0,
+ "152": 42.0,
+ "153": 50.0,
+ "154": 64.0,
+ "155": 50.0,
+ "156": 71.0,
+ "157": 62.0,
+ "158": 52.0,
+ "159": 58.0,
+ "160": 46.0,
+ "161": 49.0,
+ "162": 65.0,
+ "163": 46.0,
+ "164": 66.0,
+ "165": 68.0,
+ "166": 62.0,
+ "167": 51.0,
+ "168": 54.0,
+ "169": 68.0,
+ "170": 64.0,
+ "171": 52.0,
+ "172": 65.0,
+ "173": 74.0,
+ "174": 69.0,
+ "175": 58.0,
+ "176": 63.0,
+ "177": 63.0,
+ "178": 65.0,
+ "179": 64.0,
+ "180": 51.0,
+ "181": 58.0,
+ "182": 45.0,
+ "183": 86.0,
+ "184": 74.0,
+ "185": 75.0,
+ "186": 87.0,
+ "187": 67.0,
+ "188": 60.0,
+ "189": 73.0,
+ "190": 52.0,
+ "191": 62.0,
+ "192": 55.0,
+ "193": 74.0,
+ "194": 63.0,
+ "195": 51.0,
+ "196": 57.0,
+ "197": 62.0,
+ "198": 43.0,
+ "199": 57.0,
+ "200": 67.0,
+ "201": 62.0,
+ "202": 78.0,
+ "203": 46.0,
+ "204": 41.0,
+ "205": 65.0,
+ "206": 62.0,
+ "207": 70.0,
+ "208": 59.0,
+ "209": 61.0,
+ "210": 59.0,
+ "211": 50.0,
+ "212": 66.0,
+ "213": 61.0,
+ "214": 57.0,
+ "215": 61.0,
+ "216": 68.0,
+ "217": 65.0,
+ "218": 69.0,
+ "219": 77.0,
+ "220": 64.0,
+ "221": 54.0,
+ "222": 51.0,
+ "223": 72.0,
+ "224": 76.0,
+ "225": 76.0,
+ "226": 66.0,
+ "227": 89.0,
+ "228": 55.0,
+ "229": 55.0,
+ "230": 55.0,
+ "231": 75.0,
+ "232": 61.0,
+ "233": 68.0,
+ "234": 74.0,
+ "235": 96.0,
+ "236": 85.0,
+ "237": 80.0,
+ "238": 62.0,
+ "239": 72.0,
+ "240": 64.0,
+ "241": 76.0,
+ "242": 82.0,
+ "243": 69.0,
+ "244": 49.0,
+ "245": 77.0,
+ "246": 101.0,
+ "247": 67.0,
+ "248": 73.0,
+ "249": 83.0,
+ "250": 80.0,
+ "251": 69.0,
+ "252": 66.0,
+ "253": 61.0,
+ "254": 76.0,
+ "255": 79.0,
+ "256": 85.0,
+ "257": 76.0,
+ "258": 87.0,
+ "259": 69.0,
+ "260": 76.0,
+ "261": 67.0,
+ "262": 77.0,
+ "263": 87.0,
+ "264": 79.0,
+ "265": 50.0,
+ "266": 60.0,
+ "267": 60.0,
+ "268": 67.0,
+ "269": 84.0,
+ "270": 85.0,
+ "271": 73.0,
+ "272": 88.0,
+ "273": 51.0,
+ "274": 70.0,
+ "275": 86.0,
+ "276": 68.0,
+ "277": 71.0,
+ "278": 76.0,
+ "279": 83.0,
+ "280": 58.0,
+ "281": 72.0,
+ "282": 70.0,
+ "283": 101.0,
+ "284": 77.0,
+ "285": 61.0,
+ "286": 61.0,
+ "287": 76.0,
+ "288": 82.0,
+ "289": 85.0,
+ "290": 66.0,
+ "291": 98.0,
+ "292": 94.0,
+ "293": 103.0,
+ "294": 77.0,
+ "295": 98.0,
+ "296": 103.0,
+ "297": 76.0,
+ "298": 74.0,
+ "299": 79.0,
+ "300": 73.0,
+ "301": 82.0,
+ "302": 95.0,
+ "303": 60.0,
+ "304": 89.0,
+ "305": 66.0,
+ "306": 63.0,
+ "307": 80.0,
+ "308": 81.0,
+ "309": 82.0,
+ "310": 75.0,
+ "311": 83.0,
+ "312": 76.0,
+ "313": 83.0,
+ "314": 80.0,
+ "315": 59.0,
+ "316": 72.0,
+ "317": 75.0,
+ "318": 84.0,
+ "319": 86.0,
+ "320": 61.0,
+ "321": 86.0,
+ "322": 88.0,
+ "323": 82.0,
+ "324": 86.0,
+ "325": 56.0,
+ "326": 84.0,
+ "327": 89.0,
+ "328": 104.0,
+ "329": 102.0,
+ "330": 110.0,
+ "331": 95.0,
+ "332": 79.0,
+ "333": 77.0,
+ "334": 65.0,
+ "335": 95.0,
+ "336": 75.0,
+ "337": 85.0,
+ "338": 88.0,
+ "339": 82.0,
+ "340": 95.0,
+ "341": 82.0,
+ "342": 66.0,
+ "343": 100.0,
+ "344": 68.0,
+ "345": 82.0,
+ "346": 98.0,
+ "347": 81.0,
+ "348": 105.0,
+ "349": 72.0,
+ "350": 87.0,
+ "351": 94.0,
+ "352": 93.0,
+ "353": 97.0,
+ "354": 95.0,
+ "355": 68.0,
+ "356": 105.0,
+ "357": 78.0,
+ "358": 77.0,
+ "359": 104.0,
+ "360": 103.0,
+ "361": 66.0,
+ "362": 114.0,
+ "363": 87.0,
+ "364": 84.0,
+ "365": 81.0,
+ "366": 84.0,
+ "367": 90.0,
+ "368": 78.0,
+ "369": 83.0,
+ "370": 84.0,
+ "371": 76.0,
+ "372": 81.0,
+ "373": 98.0,
+ "374": 67.0,
+ "375": 110.0,
+ "376": 83.0,
+ "377": 105.0,
+ "378": 99.0,
+ "379": 92.0,
+ "380": 83.0,
+ "381": 108.0,
+ "382": 84.0,
+ "383": 71.0,
+ "384": 113.0,
+ "385": 79.0,
+ "386": 73.0,
+ "387": 92.0,
+ "388": 84.0,
+ "389": 95.0,
+ "390": 97.0,
+ "391": 104.0,
+ "392": 93.0,
+ "393": 90.0,
+ "394": 99.0,
+ "395": 84.0,
+ "396": 90.0,
+ "397": 82.0,
+ "398": 71.0,
+ "399": 74.0,
+ "400": 92.0,
+ "401": 111.0,
+ "402": 94.0,
+ "403": 89.0,
+ "404": 108.0,
+ "405": 71.0,
+ "406": 89.0,
+ "407": 85.0,
+ "408": 100.0,
+ "409": 71.0,
+ "410": 96.0,
+ "411": 85.0,
+ "412": 70.0,
+ "413": 74.0,
+ "414": 86.0,
+ "415": 103.0,
+ "416": 120.0,
+ "417": 87.0,
+ "418": 96.0,
+ "419": 90.0,
+ "420": 80.0,
+ "421": 98.0,
+ "422": 81.0,
+ "423": 78.0,
+ "424": 82.0,
+ "425": 85.0,
+ "426": 59.0,
+ "427": 126.0,
+ "428": 90.0,
+ "429": 91.0,
+ "430": 94.0,
+ "431": 68.0,
+ "432": 86.0,
+ "433": 80.0,
+ "434": 79.0,
+ "435": 75.0,
+ "436": 83.0,
+ "437": 90.0,
+ "438": 71.0,
+ "439": 78.0,
+ "440": 88.0,
+ "441": 70.0,
+ "442": 98.0,
+ "443": 103.0,
+ "444": 72.0,
+ "445": 72.0,
+ "446": 87.0,
+ "447": 82.0,
+ "448": 103.0,
+ "449": 95.0,
+ "450": 72.0,
+ "451": 81.0,
+ "452": 73.0,
+ "453": 78.0,
+ "454": 116.0,
+ "455": 113.0,
+ "456": 90.0,
+ "457": 107.0,
+ "458": 76.0,
+ "459": 59.0,
+ "460": 92.0,
+ "461": 85.0,
+ "462": 66.0,
+ "463": 77.0,
+ "464": 81.0,
+ "465": 62.0,
+ "466": 73.0,
+ "467": 86.0,
+ "468": 90.0,
+ "469": 87.0,
+ "470": 76.0,
+ "471": 83.0,
+ "472": 88.0,
+ "473": 83.0,
+ "474": 70.0,
+ "475": 93.0,
+ "476": 74.0,
+ "477": 61.0,
+ "478": 75.0,
+ "479": 73.0,
+ "480": 67.0,
+ "481": 95.0,
+ "482": 75.0,
+ "483": 70.0,
+ "484": 71.0,
+ "485": 78.0,
+ "486": 79.0,
+ "487": 59.0,
+ "488": 93.0,
+ "489": 88.0,
+ "490": 76.0,
+ "491": 71.0,
+ "492": 64.0,
+ "493": 86.0,
+ "494": 68.0,
+ "495": 82.0,
+ "496": 58.0,
+ "497": 64.0,
+ "498": 70.0,
+ "499": 67.0,
+ "500": 65.0,
+ "501": 71.0,
+ "502": 79.0,
+ "503": 66.0,
+ "504": 72.0,
+ "505": 86.0,
+ "506": 60.0,
+ "507": 55.0,
+ "508": 61.0,
+ "509": 62.0,
+ "510": 66.0,
+ "511": 56.0,
+ "512": 63.0,
+ "513": 67.0,
+ "514": 69.0,
+ "515": 78.0,
+ "516": 56.0,
+ "517": 57.0,
+ "518": 64.0,
+ "519": 78.0,
+ "520": 70.0,
+ "521": 48.0,
+ "522": 74.0,
+ "523": 59.0,
+ "524": 68.0,
+ "525": 84.0,
+ "526": 66.0,
+ "527": 87.0,
+ "528": 54.0,
+ "529": 63.0,
+ "530": 65.0,
+ "531": 73.0,
+ "532": 73.0,
+ "533": 80.0,
+ "534": 78.0,
+ "535": 45.0,
+ "536": 61.0,
+ "537": 73.0,
+ "538": 45.0,
+ "539": 51.0,
+ "540": 69.0,
+ "541": 57.0,
+ "542": 73.0,
+ "543": 61.0,
+ "544": 65.0,
+ "545": 70.0,
+ "546": 66.0,
+ "547": 76.0,
+ "548": 69.0,
+ "549": 70.0,
+ "550": 48.0,
+ "551": 65.0,
+ "552": 72.0,
+ "553": 62.0,
+ "554": 69.0,
+ "555": 53.0,
+ "556": 67.0,
+ "557": 59.0,
+ "558": 74.0,
+ "559": 65.0,
+ "560": 86.0,
+ "561": 65.0,
+ "562": 65.0,
+ "563": 74.0,
+ "564": 64.0,
+ "565": 56.0,
+ "566": 68.0,
+ "567": 66.0,
+ "568": 45.0,
+ "569": 69.0,
+ "570": 53.0,
+ "571": 61.0,
+ "572": 49.0,
+ "573": 75.0,
+ "574": 74.0,
+ "575": 67.0,
+ "576": 49.0,
+ "577": 55.0,
+ "578": 40.0,
+ "579": 55.0,
+ "580": 64.0,
+ "581": 60.0,
+ "582": 67.0,
+ "583": 62.0,
+ "584": 47.0,
+ "585": 68.0,
+ "586": 40.0,
+ "587": 78.0,
+ "588": 71.0,
+ "589": 58.0,
+ "590": 42.0,
+ "591": 56.0,
+ "592": 51.0,
+ "593": 69.0,
+ "594": 44.0,
+ "595": 45.0,
+ "596": 63.0,
+ "597": 46.0,
+ "598": 50.0,
+ "599": 64.0,
+ "600": 50.0,
+ "601": 45.0,
+ "602": 54.0,
+ "603": 48.0,
+ "604": 60.0,
+ "605": 51.0,
+ "606": 53.0,
+ "607": 59.0,
+ "608": 41.0,
+ "609": 42.0,
+ "610": 56.0,
+ "611": 52.0,
+ "612": 69.0,
+ "613": 64.0,
+ "614": 47.0,
+ "615": 53.0,
+ "616": 41.0,
+ "617": 54.0,
+ "618": 67.0,
+ "619": 53.0,
+ "620": 79.0,
+ "621": 58.0,
+ "622": 38.0,
+ "623": 53.0,
+ "624": 50.0,
+ "625": 55.0,
+ "626": 52.0,
+ "627": 44.0,
+ "628": 44.0,
+ "629": 46.0,
+ "630": 48.0,
+ "631": 40.0,
+ "632": 59.0,
+ "633": 52.0,
+ "634": 54.0,
+ "635": 57.0,
+ "636": 57.0,
+ "637": 47.0,
+ "638": 48.0,
+ "639": 55.0,
+ "640": 44.0,
+ "641": 51.0,
+ "642": 50.0,
+ "643": 66.0,
+ "644": 35.0,
+ "645": 60.0,
+ "646": 44.0,
+ "647": 47.0,
+ "648": 56.0,
+ "649": 55.0,
+ "650": 41.0,
+ "651": 53.0,
+ "652": 54.0,
+ "653": 60.0,
+ "654": 48.0,
+ "655": 50.0,
+ "656": 52.0,
+ "657": 47.0,
+ "658": 37.0,
+ "659": 34.0,
+ "660": 46.0,
+ "661": 46.0,
+ "662": 45.0,
+ "663": 56.0,
+ "664": 59.0,
+ "665": 45.0,
+ "666": 46.0,
+ "667": 43.0,
+ "668": 53.0,
+ "669": 46.0,
+ "670": 55.0,
+ "671": 50.0,
+ "672": 55.0,
+ "673": 40.0,
+ "674": 56.0,
+ "675": 52.0,
+ "676": 36.0,
+ "677": 55.0,
+ "678": 38.0,
+ "679": 35.0,
+ "680": 57.0,
+ "681": 54.0,
+ "682": 30.0,
+ "683": 56.0,
+ "684": 51.0,
+ "685": 50.0,
+ "686": 39.0,
+ "687": 47.0,
+ "688": 40.0,
+ "689": 36.0,
+ "690": 36.0,
+ "691": 39.0,
+ "692": 44.0,
+ "693": 40.0,
+ "694": 50.0,
+ "695": 21.0,
+ "696": 42.0,
+ "697": 24.0,
+ "698": 35.0,
+ "699": 49.0,
+ "700": 36.0,
+ "701": 41.0,
+ "702": 35.0,
+ "703": 39.0,
+ "704": 33.0,
+ "705": 47.0,
+ "706": 40.0,
+ "707": 41.0,
+ "708": 38.0,
+ "709": 26.0,
+ "710": 37.0,
+ "711": 45.0,
+ "712": 54.0,
+ "713": 39.0,
+ "714": 36.0,
+ "715": 47.0,
+ "716": 53.0,
+ "717": 39.0,
+ "718": 47.0,
+ "719": 40.0,
+ "720": 37.0,
+ "721": 57.0,
+ "722": 47.0,
+ "723": 47.0,
+ "724": 35.0,
+ "725": 40.0,
+ "726": 37.0,
+ "727": 41.0,
+ "728": 39.0,
+ "729": 34.0,
+ "730": 41.0,
+ "731": 38.0,
+ "732": 34.0,
+ "733": 37.0,
+ "734": 47.0,
+ "735": 49.0,
+ "736": 55.0,
+ "737": 45.0,
+ "738": 38.0,
+ "739": 58.0,
+ "740": 46.0,
+ "741": 57.0,
+ "742": 60.0,
+ "743": 31.0,
+ "744": 54.0,
+ "745": 36.0,
+ "746": 48.0,
+ "747": 46.0,
+ "748": 53.0,
+ "749": 45.0,
+ "750": 33.0,
+ "751": 46.0,
+ "752": 35.0,
+ "753": 45.0,
+ "754": 42.0,
+ "755": 33.0,
+ "756": 40.0,
+ "757": 43.0,
+ "758": 32.0,
+ "759": 43.0,
+ "760": 48.0,
+ "761": 35.0,
+ "762": 38.0,
+ "763": 34.0,
+ "764": 38.0,
+ "765": 28.0,
+ "766": 38.0,
+ "767": 29.0,
+ "768": 36.0,
+ "769": 40.0,
+ "770": 39.0,
+ "771": 32.0,
+ "772": 38.0,
+ "773": 50.0,
+ "774": 39.0,
+ "775": 29.0,
+ "776": 36.0,
+ "777": 43.0,
+ "778": 40.0,
+ "779": 28.0,
+ "780": 36.0,
+ "781": 40.0,
+ "782": 39.0,
+ "783": 47.0,
+ "784": 41.0,
+ "785": 33.0,
+ "786": 43.0,
+ "787": 46.0,
+ "788": 44.0,
+ "789": 43.0,
+ "790": 49.0,
+ "791": 41.0,
+ "792": 44.0,
+ "793": 51.0,
+ "794": 47.0,
+ "795": 51.0,
+ "796": 48.0,
+ "797": 40.0,
+ "798": 42.0,
+ "799": 37.0,
+ "800": 42.0,
+ "801": 48.0,
+ "802": 30.0,
+ "803": 44.0,
+ "804": 37.0,
+ "805": 52.0,
+ "806": 53.0,
+ "807": 44.0,
+ "808": 31.0,
+ "809": 40.0,
+ "810": 31.0,
+ "811": 54.0,
+ "812": 42.0,
+ "813": 48.0,
+ "814": 24.0,
+ "815": 51.0,
+ "816": 42.0,
+ "817": 43.0,
+ "818": 28.0,
+ "819": 38.0,
+ "820": 38.0,
+ "821": 35.0,
+ "822": 38.0,
+ "823": 36.0,
+ "824": 44.0,
+ "825": 40.0,
+ "826": 42.0,
+ "827": 40.0,
+ "828": 36.0,
+ "829": 39.0,
+ "830": 34.0,
+ "831": 39.0,
+ "832": 30.0,
+ "833": 44.0,
+ "834": 36.0,
+ "835": 31.0,
+ "836": 37.0,
+ "837": 46.0,
+ "838": 38.0,
+ "839": 37.0,
+ "840": 51.0,
+ "841": 50.0,
+ "842": 34.0,
+ "843": 40.0,
+ "844": 37.0,
+ "845": 38.0,
+ "846": 33.0,
+ "847": 45.0,
+ "848": 34.0,
+ "849": 45.0,
+ "850": 31.0,
+ "851": 46.0,
+ "852": 28.0,
+ "853": 44.0,
+ "854": 20.0,
+ "855": 32.0,
+ "856": 24.0,
+ "857": 38.0,
+ "858": 31.0,
+ "859": 35.0,
+ "860": 47.0,
+ "861": 31.0,
+ "862": 34.0,
+ "863": 36.0,
+ "864": 34.0,
+ "865": 33.0,
+ "866": 27.0,
+ "867": 37.0,
+ "868": 42.0,
+ "869": 37.0,
+ "870": 42.0,
+ "871": 40.0,
+ "872": 29.0,
+ "873": 44.0,
+ "874": 42.0,
+ "875": 32.0,
+ "876": 33.0,
+ "877": 29.0,
+ "878": 41.0,
+ "879": 48.0,
+ "880": 37.0,
+ "881": 33.0,
+ "882": 36.0,
+ "883": 40.0,
+ "884": 42.0,
+ "885": 35.0,
+ "886": 37.0,
+ "887": 34.0,
+ "888": 28.0,
+ "889": 36.0,
+ "890": 44.0,
+ "891": 39.0,
+ "892": 27.0,
+ "893": 47.0,
+ "894": 38.0,
+ "895": 33.0,
+ "896": 27.0,
+ "897": 38.0,
+ "898": 38.0,
+ "899": 31.0,
+ "900": 45.0,
+ "901": 34.0,
+ "902": 43.0,
+ "903": 37.0,
+ "904": 35.0,
+ "905": 40.0,
+ "906": 25.0,
+ "907": 40.0,
+ "908": 41.0,
+ "909": 31.0,
+ "910": 33.0,
+ "911": 42.0,
+ "912": 32.0,
+ "913": 40.0,
+ "914": 43.0,
+ "915": 34.0,
+ "916": 56.0,
+ "917": 34.0,
+ "918": 35.0,
+ "919": 40.0,
+ "920": 26.0,
+ "921": 38.0,
+ "922": 31.0,
+ "923": 45.0,
+ "924": 31.0,
+ "925": 35.0,
+ "926": 44.0,
+ "927": 39.0,
+ "928": 49.0,
+ "929": 47.0,
+ "930": 35.0,
+ "931": 31.0,
+ "932": 39.0,
+ "933": 37.0,
+ "934": 21.0,
+ "935": 50.0,
+ "936": 38.0,
+ "937": 49.0,
+ "938": 39.0,
+ "939": 45.0,
+ "940": 34.0,
+ "941": 36.0,
+ "942": 43.0,
+ "943": 42.0,
+ "944": 65.0,
+ "945": 40.0,
+ "946": 42.0,
+ "947": 36.0,
+ "948": 46.0,
+ "949": 32.0,
+ "950": 43.0,
+ "951": 36.0,
+ "952": 42.0,
+ "953": 29.0,
+ "954": 35.0,
+ "955": 33.0,
+ "956": 35.0,
+ "957": 32.0,
+ "958": 36.0,
+ "959": 35.0,
+ "960": 32.0,
+ "961": 32.0,
+ "962": 27.0,
+ "963": 38.0,
+ "964": 50.0,
+ "965": 50.0,
+ "966": 42.0,
+ "967": 40.0,
+ "968": 44.0,
+ "969": 27.0,
+ "970": 25.0,
+ "971": 39.0,
+ "972": 48.0,
+ "973": 30.0,
+ "974": 25.0,
+ "975": 42.0,
+ "976": 38.0,
+ "977": 32.0,
+ "978": 27.0,
+ "979": 31.0,
+ "980": 41.0,
+ "981": 42.0,
+ "982": 38.0,
+ "983": 40.0,
+ "984": 35.0,
+ "985": 51.0,
+ "986": 22.0,
+ "987": 27.0,
+ "988": 36.0,
+ "989": 43.0,
+ "990": 50.0,
+ "991": 34.0,
+ "992": 31.0,
+ "993": 37.0,
+ "994": 33.0,
+ "995": 42.0,
+ "996": 47.0,
+ "997": 33.0,
+ "998": 42.0,
+ "999": 44.0,
+ "1000": 49.0,
+ "1001": 23.0,
+ "1002": 31.0,
+ "1003": 37.0,
+ "1004": 41.0,
+ "1005": 43.0,
+ "1006": 39.0,
+ "1007": 37.0,
+ "1008": 46.0,
+ "1009": 38.0,
+ "1010": 36.0,
+ "1011": 46.0,
+ "1012": 40.0,
+ "1013": 26.0,
+ "1014": 35.0,
+ "1015": 24.0,
+ "1016": 47.0,
+ "1017": 35.0,
+ "1018": 29.0,
+ "1019": 41.0,
+ "1020": 47.0,
+ "1021": 34.0,
+ "1022": 33.0,
+ "1023": 36.0,
+ "1024": 35.0,
+ "1025": 26.0,
+ "1026": 44.0,
+ "1027": 37.0,
+ "1028": 31.0,
+ "1029": 32.0,
+ "1030": 20.0,
+ "1031": 42.0,
+ "1032": 31.0,
+ "1033": 43.0,
+ "1034": 34.0,
+ "1035": 29.0,
+ "1036": 29.0,
+ "1037": 41.0,
+ "1038": 33.0,
+ "1039": 41.0,
+ "1040": 28.0,
+ "1041": 39.0,
+ "1042": 24.0,
+ "1043": 33.0,
+ "1044": 38.0,
+ "1045": 36.0,
+ "1046": 24.0,
+ "1047": 33.0,
+ "1048": 31.0,
+ "1049": 41.0,
+ "1050": 42.0,
+ "1051": 35.0,
+ "1052": 42.0,
+ "1053": 31.0,
+ "1054": 31.0,
+ "1055": 47.0,
+ "1056": 34.0,
+ "1057": 35.0,
+ "1058": 30.0,
+ "1059": 45.0,
+ "1060": 26.0,
+ "1061": 30.0,
+ "1062": 28.0,
+ "1063": 28.0,
+ "1064": 26.0,
+ "1065": 36.0,
+ "1066": 26.0,
+ "1067": 31.0,
+ "1068": 33.0,
+ "1069": 40.0,
+ "1070": 38.0,
+ "1071": 47.0,
+ "1072": 28.0,
+ "1073": 37.0,
+ "1074": 40.0,
+ "1075": 31.0,
+ "1076": 33.0,
+ "1077": 30.0,
+ "1078": 48.0,
+ "1079": 43.0,
+ "1080": 29.0,
+ "1081": 34.0,
+ "1082": 39.0,
+ "1083": 29.0,
+ "1084": 50.0,
+ "1085": 34.0,
+ "1086": 45.0,
+ "1087": 35.0,
+ "1088": 33.0,
+ "1089": 39.0,
+ "1090": 42.0,
+ "1091": 35.0,
+ "1092": 20.0,
+ "1093": 30.0,
+ "1094": 32.0,
+ "1095": 30.0,
+ "1096": 45.0,
+ "1097": 31.0,
+ "1098": 31.0,
+ "1099": 35.0,
+ "1100": 36.0,
+ "1101": 35.0,
+ "1102": 37.0,
+ "1103": 26.0,
+ "1104": 37.0,
+ "1105": 28.0,
+ "1106": 32.0,
+ "1107": 53.0,
+ "1108": 35.0,
+ "1109": 37.0,
+ "1110": 51.0,
+ "1111": 34.0,
+ "1112": 22.0,
+ "1113": 41.0,
+ "1114": 35.0,
+ "1115": 43.0,
+ "1116": 35.0,
+ "1117": 40.0,
+ "1118": 31.0,
+ "1119": 37.0,
+ "1120": 32.0,
+ "1121": 33.0,
+ "1122": 32.0,
+ "1123": 29.0,
+ "1124": 48.0,
+ "1125": 45.0,
+ "1126": 30.0,
+ "1127": 42.0,
+ "1128": 35.0,
+ "1129": 29.0,
+ "1130": 38.0,
+ "1131": 33.0,
+ "1132": 33.0,
+ "1133": 27.0,
+ "1134": 33.0,
+ "1135": 34.0,
+ "1136": 27.0,
+ "1137": 35.0,
+ "1138": 31.0,
+ "1139": 27.0,
+ "1140": 31.0,
+ "1141": 33.0,
+ "1142": 29.0,
+ "1143": 34.0,
+ "1144": 31.0,
+ "1145": 36.0,
+ "1146": 36.0,
+ "1147": 46.0,
+ "1148": 25.0,
+ "1149": 33.0,
+ "1150": 36.0,
+ "1151": 39.0,
+ "1152": 34.0,
+ "1153": 41.0,
+ "1154": 48.0,
+ "1155": 37.0,
+ "1156": 33.0,
+ "1157": 41.0,
+ "1158": 37.0,
+ "1159": 45.0,
+ "1160": 42.0,
+ "1161": 33.0,
+ "1162": 34.0,
+ "1163": 27.0,
+ "1164": 48.0,
+ "1165": 20.0,
+ "1166": 34.0,
+ "1167": 39.0,
+ "1168": 38.0,
+ "1169": 47.0,
+ "1170": 23.0,
+ "1171": 20.0,
+ "1172": 42.0,
+ "1173": 34.0,
+ "1174": 30.0,
+ "1175": 31.0,
+ "1176": 39.0,
+ "1177": 42.0,
+ "1178": 35.0,
+ "1179": 35.0,
+ "1180": 37.0,
+ "1181": 38.0,
+ "1182": 38.0,
+ "1183": 42.0,
+ "1184": 26.0,
+ "1185": 37.0,
+ "1186": 35.0,
+ "1187": 41.0,
+ "1188": 35.0,
+ "1189": 46.0,
+ "1190": 38.0,
+ "1191": 42.0,
+ "1192": 32.0,
+ "1193": 31.0,
+ "1194": 41.0,
+ "1195": 26.0,
+ "1196": 45.0,
+ "1197": 36.0,
+ "1198": 49.0,
+ "1199": 35.0,
+ "1200": 34.0,
+ "1201": 36.0,
+ "1202": 30.0,
+ "1203": 33.0,
+ "1204": 45.0,
+ "1205": 28.0,
+ "1206": 36.0,
+ "1207": 35.0,
+ "1208": 40.0,
+ "1209": 38.0,
+ "1210": 43.0,
+ "1211": 24.0,
+ "1212": 35.0,
+ "1213": 33.0,
+ "1214": 34.0,
+ "1215": 31.0,
+ "1216": 30.0,
+ "1217": 38.0,
+ "1218": 50.0,
+ "1219": 41.0,
+ "1220": 35.0,
+ "1221": 25.0,
+ "1222": 34.0,
+ "1223": 37.0,
+ "1224": 36.0,
+ "1225": 45.0,
+ "1226": 35.0,
+ "1227": 39.0,
+ "1228": 29.0,
+ "1229": 28.0,
+ "1230": 27.0,
+ "1231": 38.0,
+ "1232": 43.0,
+ "1233": 40.0,
+ "1234": 36.0,
+ "1235": 29.0,
+ "1236": 42.0,
+ "1237": 32.0,
+ "1238": 31.0,
+ "1239": 34.0,
+ "1240": 27.0,
+ "1241": 53.0,
+ "1242": 45.0,
+ "1243": 35.0,
+ "1244": 35.0,
+ "1245": 36.0,
+ "1246": 40.0,
+ "1247": 26.0,
+ "1248": 34.0,
+ "1249": 31.0,
+ "1250": 36.0,
+ "1251": 45.0,
+ "1252": 38.0,
+ "1253": 32.0,
+ "1254": 32.0,
+ "1255": 40.0,
+ "1256": 29.0,
+ "1257": 30.0,
+ "1258": 42.0,
+ "1259": 30.0,
+ "1260": 29.0,
+ "1261": 33.0,
+ "1262": 47.0,
+ "1263": 29.0,
+ "1264": 48.0,
+ "1265": 36.0,
+ "1266": 34.0,
+ "1267": 32.0,
+ "1268": 33.0,
+ "1269": 30.0,
+ "1270": 38.0,
+ "1271": 33.0,
+ "1272": 46.0,
+ "1273": 25.0,
+ "1274": 38.0,
+ "1275": 39.0,
+ "1276": 43.0,
+ "1277": 40.0,
+ "1278": 27.0,
+ "1279": 39.0,
+ "1280": 26.0,
+ "1281": 35.0,
+ "1282": 39.0,
+ "1283": 41.0,
+ "1284": 37.0,
+ "1285": 30.0,
+ "1286": 39.0,
+ "1287": 33.0,
+ "1288": 25.0,
+ "1289": 46.0,
+ "1290": 39.0,
+ "1291": 38.0,
+ "1292": 37.0,
+ "1293": 40.0,
+ "1294": 28.0,
+ "1295": 24.0,
+ "1296": 31.0,
+ "1297": 27.0,
+ "1298": 36.0,
+ "1299": 42.0,
+ "1300": 33.0,
+ "1301": 45.0,
+ "1302": 36.0,
+ "1303": 41.0,
+ "1304": 36.0,
+ "1305": 37.0,
+ "1306": 32.0,
+ "1307": 32.0,
+ "1308": 36.0,
+ "1309": 32.0,
+ "1310": 38.0,
+ "1311": 40.0,
+ "1312": 33.0,
+ "1313": 43.0,
+ "1314": 39.0,
+ "1315": 44.0,
+ "1316": 47.0,
+ "1317": 35.0,
+ "1318": 25.0,
+ "1319": 28.0,
+ "1320": 44.0,
+ "1321": 31.0,
+ "1322": 36.0,
+ "1323": 33.0,
+ "1324": 39.0,
+ "1325": 30.0,
+ "1326": 38.0,
+ "1327": 31.0,
+ "1328": 32.0,
+ "1329": 27.0,
+ "1330": 28.0,
+ "1331": 34.0,
+ "1332": 28.0,
+ "1333": 35.0,
+ "1334": 39.0,
+ "1335": 45.0,
+ "1336": 34.0,
+ "1337": 35.0,
+ "1338": 39.0,
+ "1339": 41.0,
+ "1340": 37.0,
+ "1341": 41.0,
+ "1342": 27.0,
+ "1343": 52.0,
+ "1344": 30.0,
+ "1345": 34.0,
+ "1346": 48.0,
+ "1347": 35.0,
+ "1348": 42.0,
+ "1349": 33.0,
+ "1350": 34.0,
+ "1351": 38.0,
+ "1352": 34.0,
+ "1353": 28.0,
+ "1354": 44.0,
+ "1355": 33.0,
+ "1356": 34.0,
+ "1357": 25.0,
+ "1358": 44.0,
+ "1359": 34.0,
+ "1360": 49.0,
+ "1361": 50.0,
+ "1362": 30.0,
+ "1363": 48.0,
+ "1364": 35.0,
+ "1365": 39.0,
+ "1366": 30.0,
+ "1367": 27.0,
+ "1368": 43.0,
+ "1369": 33.0,
+ "1370": 50.0,
+ "1371": 35.0,
+ "1372": 32.0,
+ "1373": 41.0,
+ "1374": 37.0,
+ "1375": 40.0,
+ "1376": 39.0,
+ "1377": 38.0,
+ "1378": 33.0,
+ "1379": 48.0,
+ "1380": 35.0,
+ "1381": 28.0,
+ "1382": 37.0,
+ "1383": 46.0,
+ "1384": 34.0,
+ "1385": 49.0,
+ "1386": 29.0,
+ "1387": 38.0,
+ "1388": 44.0,
+ "1389": 39.0,
+ "1390": 33.0,
+ "1391": 59.0,
+ "1392": 30.0,
+ "1393": 25.0,
+ "1394": 31.0,
+ "1395": 31.0,
+ "1396": 40.0,
+ "1397": 42.0,
+ "1398": 26.0,
+ "1399": 39.0,
+ "1400": 41.0,
+ "1401": 43.0,
+ "1402": 29.0,
+ "1403": 38.0,
+ "1404": 37.0,
+ "1405": 46.0,
+ "1406": 34.0,
+ "1407": 43.0,
+ "1408": 25.0,
+ "1409": 37.0,
+ "1410": 41.0,
+ "1411": 55.0,
+ "1412": 33.0,
+ "1413": 29.0,
+ "1414": 44.0,
+ "1415": 43.0,
+ "1416": 33.0,
+ "1417": 34.0,
+ "1418": 35.0,
+ "1419": 52.0,
+ "1420": 32.0,
+ "1421": 43.0,
+ "1422": 36.0,
+ "1423": 37.0,
+ "1424": 33.0,
+ "1425": 43.0,
+ "1426": 27.0,
+ "1427": 39.0,
+ "1428": 43.0,
+ "1429": 31.0,
+ "1430": 45.0,
+ "1431": 41.0,
+ "1432": 45.0,
+ "1433": 36.0,
+ "1434": 36.0,
+ "1435": 40.0,
+ "1436": 37.0,
+ "1437": 44.0,
+ "1438": 34.0,
+ "1439": 31.0,
+ "1440": 43.0,
+ "1441": 32.0,
+ "1442": 37.0,
+ "1443": 51.0,
+ "1444": 28.0,
+ "1445": 34.0,
+ "1446": 31.0,
+ "1447": 40.0,
+ "1448": 34.0,
+ "1449": 30.0,
+ "1450": 39.0,
+ "1451": 41.0,
+ "1452": 35.0,
+ "1453": 32.0,
+ "1454": 31.0,
+ "1455": 42.0,
+ "1456": 27.0,
+ "1457": 40.0,
+ "1458": 37.0,
+ "1459": 35.0,
+ "1460": 31.0,
+ "1461": 40.0,
+ "1462": 32.0,
+ "1463": 36.0,
+ "1464": 31.0,
+ "1465": 30.0,
+ "1466": 35.0,
+ "1467": 45.0,
+ "1468": 41.0,
+ "1469": 46.0,
+ "1470": 38.0,
+ "1471": 37.0,
+ "1472": 29.0,
+ "1473": 36.0,
+ "1474": 32.0,
+ "1475": 39.0,
+ "1476": 28.0,
+ "1477": 36.0,
+ "1478": 24.0,
+ "1479": 31.0,
+ "1480": 34.0,
+ "1481": 35.0,
+ "1482": 42.0,
+ "1483": 37.0,
+ "1484": 29.0,
+ "1485": 32.0,
+ "1486": 32.0,
+ "1487": 28.0,
+ "1488": 36.0,
+ "1489": 28.0,
+ "1490": 48.0,
+ "1491": 45.0,
+ "1492": 48.0,
+ "1493": 42.0,
+ "1494": 40.0,
+ "1495": 29.0,
+ "1496": 30.0,
+ "1497": 34.0,
+ "1498": 34.0,
+ "1499": 37.0,
+ "1500": 45.0,
+ "1501": 35.0,
+ "1502": 37.0,
+ "1503": 33.0,
+ "1504": 32.0,
+ "1505": 46.0,
+ "1506": 43.0,
+ "1507": 35.0,
+ "1508": 35.0,
+ "1509": 30.0,
+ "1510": 32.0,
+ "1511": 42.0,
+ "1512": 38.0,
+ "1513": 37.0,
+ "1514": 26.0,
+ "1515": 46.0,
+ "1516": 27.0,
+ "1517": 37.0,
+ "1518": 30.0,
+ "1519": 40.0,
+ "1520": 37.0,
+ "1521": 40.0,
+ "1522": 36.0,
+ "1523": 32.0,
+ "1524": 29.0,
+ "1525": 42.0,
+ "1526": 27.0,
+ "1527": 42.0,
+ "1528": 45.0,
+ "1529": 43.0,
+ "1530": 28.0,
+ "1531": 30.0,
+ "1532": 36.0,
+ "1533": 37.0,
+ "1534": 37.0,
+ "1535": 43.0,
+ "1536": 34.0,
+ "1537": 37.0,
+ "1538": 40.0,
+ "1539": 35.0,
+ "1540": 28.0,
+ "1541": 42.0,
+ "1542": 39.0,
+ "1543": 28.0,
+ "1544": 34.0,
+ "1545": 31.0,
+ "1546": 39.0,
+ "1547": 43.0,
+ "1548": 39.0,
+ "1549": 43.0,
+ "1550": 33.0,
+ "1551": 31.0,
+ "1552": 24.0,
+ "1553": 33.0,
+ "1554": 41.0,
+ "1555": 41.0,
+ "1556": 37.0,
+ "1557": 40.0,
+ "1558": 33.0,
+ "1559": 29.0,
+ "1560": 39.0,
+ "1561": 33.0,
+ "1562": 41.0,
+ "1563": 48.0,
+ "1564": 33.0,
+ "1565": 47.0,
+ "1566": 50.0,
+ "1567": 39.0,
+ "1568": 39.0,
+ "1569": 53.0,
+ "1570": 56.0,
+ "1571": 27.0,
+ "1572": 40.0,
+ "1573": 38.0,
+ "1574": 26.0,
+ "1575": 33.0,
+ "1576": 35.0,
+ "1577": 45.0,
+ "1578": 51.0,
+ "1579": 47.0,
+ "1580": 34.0,
+ "1581": 36.0,
+ "1582": 27.0,
+ "1583": 46.0,
+ "1584": 40.0,
+ "1585": 45.0,
+ "1586": 34.0,
+ "1587": 34.0,
+ "1588": 35.0,
+ "1589": 29.0,
+ "1590": 28.0,
+ "1591": 47.0,
+ "1592": 49.0,
+ "1593": 30.0,
+ "1594": 27.0,
+ "1595": 44.0,
+ "1596": 35.0,
+ "1597": 42.0,
+ "1598": 33.0,
+ "1599": 42.0,
+ "1600": 32.0,
+ "1601": 36.0,
+ "1602": 39.0,
+ "1603": 38.0,
+ "1604": 37.0,
+ "1605": 31.0,
+ "1606": 45.0,
+ "1607": 44.0,
+ "1608": 29.0,
+ "1609": 36.0,
+ "1610": 32.0,
+ "1611": 32.0,
+ "1612": 29.0,
+ "1613": 35.0,
+ "1614": 37.0,
+ "1615": 30.0,
+ "1616": 38.0,
+ "1617": 34.0,
+ "1618": 33.0,
+ "1619": 32.0,
+ "1620": 39.0,
+ "1621": 45.0,
+ "1622": 33.0,
+ "1623": 42.0,
+ "1624": 35.0,
+ "1625": 42.0,
+ "1626": 38.0,
+ "1627": 35.0,
+ "1628": 31.0,
+ "1629": 42.0,
+ "1630": 43.0,
+ "1631": 31.0,
+ "1632": 48.0,
+ "1633": 44.0,
+ "1634": 35.0,
+ "1635": 54.0,
+ "1636": 40.0,
+ "1637": 36.0,
+ "1638": 34.0,
+ "1639": 40.0,
+ "1640": 48.0,
+ "1641": 51.0,
+ "1642": 28.0,
+ "1643": 36.0,
+ "1644": 41.0,
+ "1645": 34.0,
+ "1646": 33.0,
+ "1647": 34.0,
+ "1648": 39.0,
+ "1649": 30.0,
+ "1650": 43.0,
+ "1651": 24.0,
+ "1652": 44.0,
+ "1653": 38.0,
+ "1654": 40.0,
+ "1655": 34.0,
+ "1656": 36.0,
+ "1657": 46.0,
+ "1658": 36.0,
+ "1659": 41.0,
+ "1660": 33.0,
+ "1661": 43.0,
+ "1662": 36.0,
+ "1663": 38.0,
+ "1664": 31.0,
+ "1665": 50.0,
+ "1666": 43.0,
+ "1667": 37.0,
+ "1668": 40.0,
+ "1669": 34.0,
+ "1670": 48.0,
+ "1671": 39.0,
+ "1672": 35.0,
+ "1673": 33.0,
+ "1674": 41.0,
+ "1675": 28.0,
+ "1676": 33.0,
+ "1677": 30.0,
+ "1678": 35.0,
+ "1679": 40.0,
+ "1680": 31.0,
+ "1681": 49.0,
+ "1682": 23.0,
+ "1683": 40.0,
+ "1684": 36.0,
+ "1685": 34.0,
+ "1686": 24.0,
+ "1687": 52.0,
+ "1688": 40.0,
+ "1689": 37.0,
+ "1690": 44.0,
+ "1691": 38.0,
+ "1692": 46.0,
+ "1693": 36.0,
+ "1694": 35.0,
+ "1695": 45.0,
+ "1696": 39.0,
+ "1697": 38.0,
+ "1698": 38.0,
+ "1699": 26.0,
+ "1700": 38.0,
+ "1701": 37.0,
+ "1702": 41.0,
+ "1703": 41.0,
+ "1704": 35.0,
+ "1705": 38.0,
+ "1706": 30.0,
+ "1707": 47.0,
+ "1708": 32.0,
+ "1709": 26.0,
+ "1710": 51.0,
+ "1711": 42.0,
+ "1712": 52.0,
+ "1713": 40.0,
+ "1714": 36.0,
+ "1715": 40.0,
+ "1716": 42.0,
+ "1717": 45.0,
+ "1718": 37.0,
+ "1719": 46.0,
+ "1720": 44.0,
+ "1721": 35.0,
+ "1722": 43.0,
+ "1723": 42.0,
+ "1724": 42.0,
+ "1725": 32.0,
+ "1726": 33.0,
+ "1727": 39.0,
+ "1728": 49.0,
+ "1729": 37.0,
+ "1730": 44.0,
+ "1731": 35.0,
+ "1732": 36.0,
+ "1733": 40.0,
+ "1734": 39.0,
+ "1735": 39.0,
+ "1736": 26.0,
+ "1737": 39.0,
+ "1738": 39.0,
+ "1739": 45.0,
+ "1740": 46.0,
+ "1741": 45.0,
+ "1742": 36.0,
+ "1743": 45.0,
+ "1744": 41.0,
+ "1745": 34.0,
+ "1746": 47.0,
+ "1747": 30.0,
+ "1748": 37.0,
+ "1749": 43.0,
+ "1750": 46.0,
+ "1751": 29.0,
+ "1752": 43.0,
+ "1753": 43.0,
+ "1754": 50.0,
+ "1755": 40.0,
+ "1756": 34.0,
+ "1757": 40.0,
+ "1758": 28.0,
+ "1759": 47.0,
+ "1760": 37.0,
+ "1761": 43.0,
+ "1762": 40.0,
+ "1763": 27.0,
+ "1764": 33.0,
+ "1765": 31.0,
+ "1766": 38.0,
+ "1767": 41.0,
+ "1768": 33.0,
+ "1769": 30.0,
+ "1770": 41.0,
+ "1771": 39.0,
+ "1772": 50.0,
+ "1773": 35.0,
+ "1774": 50.0,
+ "1775": 40.0,
+ "1776": 39.0,
+ "1777": 36.0,
+ "1778": 44.0,
+ "1779": 44.0,
+ "1780": 36.0,
+ "1781": 41.0,
+ "1782": 39.0,
+ "1783": 35.0,
+ "1784": 36.0,
+ "1785": 45.0,
+ "1786": 40.0,
+ "1787": 48.0,
+ "1788": 43.0,
+ "1789": 38.0,
+ "1790": 40.0,
+ "1791": 33.0,
+ "1792": 38.0,
+ "1793": 42.0,
+ "1794": 49.0,
+ "1795": 34.0,
+ "1796": 39.0,
+ "1797": 41.0,
+ "1798": 27.0,
+ "1799": 48.0,
+ "1800": 44.0,
+ "1801": 48.0,
+ "1802": 40.0,
+ "1803": 30.0,
+ "1804": 36.0,
+ "1805": 34.0,
+ "1806": 40.0,
+ "1807": 34.0,
+ "1808": 51.0,
+ "1809": 30.0,
+ "1810": 43.0,
+ "1811": 34.0,
+ "1812": 43.0,
+ "1813": 27.0,
+ "1814": 34.0,
+ "1815": 36.0,
+ "1816": 36.0,
+ "1817": 37.0,
+ "1818": 42.0,
+ "1819": 43.0,
+ "1820": 51.0,
+ "1821": 41.0,
+ "1822": 49.0,
+ "1823": 38.0,
+ "1824": 28.0,
+ "1825": 35.0,
+ "1826": 32.0,
+ "1827": 41.0,
+ "1828": 37.0,
+ "1829": 49.0,
+ "1830": 49.0,
+ "1831": 33.0,
+ "1832": 37.0,
+ "1833": 43.0,
+ "1834": 43.0,
+ "1835": 41.0,
+ "1836": 45.0,
+ "1837": 47.0,
+ "1838": 41.0,
+ "1839": 33.0,
+ "1840": 44.0,
+ "1841": 42.0,
+ "1842": 32.0,
+ "1843": 33.0,
+ "1844": 48.0,
+ "1845": 37.0,
+ "1846": 33.0,
+ "1847": 47.0,
+ "1848": 35.0,
+ "1849": 32.0,
+ "1850": 37.0,
+ "1851": 30.0,
+ "1852": 38.0,
+ "1853": 40.0,
+ "1854": 39.0,
+ "1855": 43.0,
+ "1856": 40.0,
+ "1857": 43.0,
+ "1858": 24.0,
+ "1859": 36.0,
+ "1860": 34.0,
+ "1861": 41.0,
+ "1862": 37.0,
+ "1863": 43.0,
+ "1864": 39.0,
+ "1865": 45.0,
+ "1866": 41.0,
+ "1867": 41.0,
+ "1868": 28.0,
+ "1869": 34.0,
+ "1870": 35.0,
+ "1871": 36.0,
+ "1872": 29.0,
+ "1873": 39.0,
+ "1874": 27.0,
+ "1875": 46.0,
+ "1876": 33.0,
+ "1877": 41.0,
+ "1878": 30.0,
+ "1879": 41.0,
+ "1880": 40.0,
+ "1881": 32.0,
+ "1882": 53.0,
+ "1883": 35.0,
+ "1884": 30.0,
+ "1885": 36.0,
+ "1886": 28.0,
+ "1887": 41.0,
+ "1888": 54.0,
+ "1889": 48.0,
+ "1890": 36.0,
+ "1891": 50.0,
+ "1892": 37.0,
+ "1893": 39.0,
+ "1894": 49.0,
+ "1895": 28.0,
+ "1896": 34.0,
+ "1897": 42.0,
+ "1898": 40.0,
+ "1899": 35.0,
+ "1900": 57.0,
+ "1901": 37.0,
+ "1902": 38.0,
+ "1903": 42.0,
+ "1904": 36.0,
+ "1905": 31.0,
+ "1906": 39.0,
+ "1907": 44.0,
+ "1908": 27.0,
+ "1909": 42.0,
+ "1910": 45.0,
+ "1911": 33.0,
+ "1912": 37.0,
+ "1913": 44.0,
+ "1914": 51.0,
+ "1915": 38.0,
+ "1916": 41.0,
+ "1917": 36.0,
+ "1918": 36.0,
+ "1919": 37.0,
+ "1920": 35.0,
+ "1921": 34.0,
+ "1922": 42.0,
+ "1923": 37.0,
+ "1924": 45.0,
+ "1925": 33.0,
+ "1926": 41.0,
+ "1927": 52.0,
+ "1928": 35.0,
+ "1929": 46.0,
+ "1930": 53.0,
+ "1931": 42.0,
+ "1932": 39.0,
+ "1933": 40.0,
+ "1934": 46.0,
+ "1935": 37.0,
+ "1936": 33.0,
+ "1937": 34.0,
+ "1938": 56.0,
+ "1939": 47.0,
+ "1940": 37.0,
+ "1941": 55.0,
+ "1942": 36.0,
+ "1943": 44.0,
+ "1944": 52.0,
+ "1945": 42.0,
+ "1946": 47.0,
+ "1947": 37.0,
+ "1948": 46.0,
+ "1949": 34.0,
+ "1950": 51.0,
+ "1951": 51.0,
+ "1952": 45.0,
+ "1953": 52.0,
+ "1954": 39.0,
+ "1955": 39.0,
+ "1956": 36.0,
+ "1957": 42.0,
+ "1958": 47.0,
+ "1959": 39.0,
+ "1960": 33.0,
+ "1961": 32.0,
+ "1962": 47.0,
+ "1963": 47.0,
+ "1964": 41.0,
+ "1965": 58.0,
+ "1966": 45.0,
+ "1967": 37.0,
+ "1968": 46.0,
+ "1969": 42.0,
+ "1970": 32.0,
+ "1971": 29.0,
+ "1972": 39.0,
+ "1973": 34.0,
+ "1974": 36.0,
+ "1975": 33.0,
+ "1976": 37.0,
+ "1977": 34.0,
+ "1978": 25.0,
+ "1979": 43.0,
+ "1980": 45.0,
+ "1981": 38.0,
+ "1982": 28.0,
+ "1983": 38.0,
+ "1984": 38.0,
+ "1985": 43.0,
+ "1986": 50.0,
+ "1987": 49.0,
+ "1988": 36.0,
+ "1989": 32.0,
+ "1990": 38.0,
+ "1991": 37.0,
+ "1992": 36.0,
+ "1993": 58.0,
+ "1994": 42.0,
+ "1995": 50.0,
+ "1996": 39.0,
+ "1997": 40.0,
+ "1998": 42.0,
+ "1999": 39.0,
+ "2000": 34.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 301164032.0,
+ "2": 301164032.0,
+ "3": 301164032.0,
+ "4": 301164032.0,
+ "5": 301164032.0,
+ "6": 301164032.0,
+ "7": 301164032.0,
+ "8": 301164032.0,
+ "9": 301164032.0,
+ "10": 301164032.0,
+ "11": 301164032.0,
+ "12": 301164032.0,
+ "13": 301164032.0,
+ "14": 301164032.0,
+ "15": 301164032.0,
+ "16": 301164032.0,
+ "17": 301164032.0,
+ "18": 301164032.0,
+ "19": 301164032.0,
+ "20": 301164032.0,
+ "21": 301164032.0,
+ "22": 301164032.0,
+ "23": 301164032.0,
+ "24": 301164032.0,
+ "25": 301164032.0,
+ "26": 301164032.0,
+ "27": 301164032.0,
+ "28": 301164032.0,
+ "29": 301164032.0,
+ "30": 301164032.0,
+ "31": 301164032.0,
+ "32": 301164032.0,
+ "33": 301164032.0,
+ "34": 301164032.0,
+ "35": 301164032.0,
+ "36": 301164032.0,
+ "37": 301164032.0,
+ "38": 301164032.0,
+ "39": 301164032.0,
+ "40": 301164032.0,
+ "41": 301164032.0,
+ "42": 301164032.0,
+ "43": 301164032.0,
+ "44": 301164032.0,
+ "45": 301164032.0,
+ "46": 301164032.0,
+ "47": 301164032.0,
+ "48": 301164032.0,
+ "49": 301164032.0,
+ "50": 301164032.0,
+ "51": 301164032.0,
+ "52": 301164032.0,
+ "53": 301164032.0,
+ "54": 301164032.0,
+ "55": 301164032.0,
+ "56": 301164032.0,
+ "57": 301164032.0,
+ "58": 301164032.0,
+ "59": 301164032.0,
+ "60": 301164032.0,
+ "61": 301164032.0,
+ "62": 301164032.0,
+ "63": 301164032.0,
+ "64": 301164032.0,
+ "65": 301164032.0,
+ "66": 301164032.0,
+ "67": 301164032.0,
+ "68": 301164032.0,
+ "69": 301164032.0,
+ "70": 301164032.0,
+ "71": 301164032.0,
+ "72": 301164032.0,
+ "73": 301164032.0,
+ "74": 301164032.0,
+ "75": 301164032.0,
+ "76": 301164032.0,
+ "77": 301164032.0,
+ "78": 301164032.0,
+ "79": 301164032.0,
+ "80": 301164032.0,
+ "81": 301164032.0,
+ "82": 301164032.0,
+ "83": 301164032.0,
+ "84": 301164032.0,
+ "85": 301164032.0,
+ "86": 301164032.0,
+ "87": 301164032.0,
+ "88": 301164032.0,
+ "89": 301164032.0,
+ "90": 301164032.0,
+ "91": 301164032.0,
+ "92": 301164032.0,
+ "93": 301164032.0,
+ "94": 301164032.0,
+ "95": 301164032.0,
+ "96": 301164032.0,
+ "97": 301164032.0,
+ "98": 301164032.0,
+ "99": 301164032.0,
+ "100": 301164032.0,
+ "101": 301164032.0,
+ "102": 301164032.0,
+ "103": 301164032.0,
+ "104": 301164032.0,
+ "105": 301164032.0,
+ "106": 301164032.0,
+ "107": 301164032.0,
+ "108": 301164032.0,
+ "109": 301164032.0,
+ "110": 301164032.0,
+ "111": 301164032.0,
+ "112": 301164032.0,
+ "113": 301164032.0,
+ "114": 301164032.0,
+ "115": 301164032.0,
+ "116": 301164032.0,
+ "117": 301164032.0,
+ "118": 301164032.0,
+ "119": 301164032.0,
+ "120": 301164032.0,
+ "121": 301164032.0,
+ "122": 301164032.0,
+ "123": 301164032.0,
+ "124": 301164032.0,
+ "125": 301164032.0,
+ "126": 301164032.0,
+ "127": 301164032.0,
+ "128": 301164032.0,
+ "129": 301164032.0,
+ "130": 301164032.0,
+ "131": 301164032.0,
+ "132": 301164032.0,
+ "133": 301164032.0,
+ "134": 301164032.0,
+ "135": 301164032.0,
+ "136": 301164032.0,
+ "137": 301164032.0,
+ "138": 301164032.0,
+ "139": 301164032.0,
+ "140": 301164032.0,
+ "141": 301164032.0,
+ "142": 301164032.0,
+ "143": 301164032.0,
+ "144": 301164032.0,
+ "145": 301164032.0,
+ "146": 301164032.0,
+ "147": 301164032.0,
+ "148": 301164032.0,
+ "149": 301164032.0,
+ "150": 301164032.0,
+ "151": 301164032.0,
+ "152": 301164032.0,
+ "153": 301164032.0,
+ "154": 301164032.0,
+ "155": 301164032.0,
+ "156": 301164032.0,
+ "157": 301164032.0,
+ "158": 301164032.0,
+ "159": 301164032.0,
+ "160": 301164032.0,
+ "161": 301164032.0,
+ "162": 301164032.0,
+ "163": 301164032.0,
+ "164": 301164032.0,
+ "165": 301164032.0,
+ "166": 301164032.0,
+ "167": 301164032.0,
+ "168": 301164032.0,
+ "169": 301164032.0,
+ "170": 301164032.0,
+ "171": 301164032.0,
+ "172": 301164032.0,
+ "173": 301164032.0,
+ "174": 301164032.0,
+ "175": 301164032.0,
+ "176": 301164032.0,
+ "177": 301164032.0,
+ "178": 301164032.0,
+ "179": 301164032.0,
+ "180": 301164032.0,
+ "181": 301164032.0,
+ "182": 301164032.0,
+ "183": 301164032.0,
+ "184": 301164032.0,
+ "185": 301164032.0,
+ "186": 301164032.0,
+ "187": 301164032.0,
+ "188": 301164032.0,
+ "189": 301164032.0,
+ "190": 301164032.0,
+ "191": 301164032.0,
+ "192": 301164032.0,
+ "193": 301164032.0,
+ "194": 301164032.0,
+ "195": 301164032.0,
+ "196": 301164032.0,
+ "197": 301164032.0,
+ "198": 301164032.0,
+ "199": 301164032.0,
+ "200": 301164032.0,
+ "201": 301164032.0,
+ "202": 301164032.0,
+ "203": 301164032.0,
+ "204": 301164032.0,
+ "205": 301164032.0,
+ "206": 301164032.0,
+ "207": 301164032.0,
+ "208": 301164032.0,
+ "209": 301164032.0,
+ "210": 301164032.0,
+ "211": 301164032.0,
+ "212": 301164032.0,
+ "213": 301164032.0,
+ "214": 301164032.0,
+ "215": 301164032.0,
+ "216": 301164032.0,
+ "217": 301164032.0,
+ "218": 301164032.0,
+ "219": 301164032.0,
+ "220": 301164032.0,
+ "221": 301164032.0,
+ "222": 301164032.0,
+ "223": 301164032.0,
+ "224": 301164032.0,
+ "225": 301164032.0,
+ "226": 301164032.0,
+ "227": 301164032.0,
+ "228": 301164032.0,
+ "229": 301164032.0,
+ "230": 301164032.0,
+ "231": 301164032.0,
+ "232": 301164032.0,
+ "233": 301164032.0,
+ "234": 301164032.0,
+ "235": 301164032.0,
+ "236": 301164032.0,
+ "237": 301164032.0,
+ "238": 301164032.0,
+ "239": 301164032.0,
+ "240": 301164032.0,
+ "241": 301164032.0,
+ "242": 301164032.0,
+ "243": 301164032.0,
+ "244": 301164032.0,
+ "245": 301164032.0,
+ "246": 301164032.0,
+ "247": 301164032.0,
+ "248": 301164032.0,
+ "249": 301164032.0,
+ "250": 301164032.0,
+ "251": 301164032.0,
+ "252": 301164032.0,
+ "253": 301164032.0,
+ "254": 301164032.0,
+ "255": 301164032.0,
+ "256": 301164032.0,
+ "257": 301164032.0,
+ "258": 301164032.0,
+ "259": 301164032.0,
+ "260": 301164032.0,
+ "261": 301164032.0,
+ "262": 301164032.0,
+ "263": 301164032.0,
+ "264": 301164032.0,
+ "265": 301164032.0,
+ "266": 301164032.0,
+ "267": 301164032.0,
+ "268": 301164032.0,
+ "269": 301164032.0,
+ "270": 301164032.0,
+ "271": 301164032.0,
+ "272": 301164032.0,
+ "273": 301164032.0,
+ "274": 301164032.0,
+ "275": 301164032.0,
+ "276": 301164032.0,
+ "277": 301164032.0,
+ "278": 301164032.0,
+ "279": 301164032.0,
+ "280": 301164032.0,
+ "281": 301164032.0,
+ "282": 301164032.0,
+ "283": 301164032.0,
+ "284": 301164032.0,
+ "285": 301164032.0,
+ "286": 301164032.0,
+ "287": 301164032.0,
+ "288": 301164032.0,
+ "289": 301164032.0,
+ "290": 301164032.0,
+ "291": 301164032.0,
+ "292": 301164032.0,
+ "293": 301164032.0,
+ "294": 301164032.0,
+ "295": 301164032.0,
+ "296": 301164032.0,
+ "297": 301164032.0,
+ "298": 301164032.0,
+ "299": 301164032.0,
+ "300": 301164032.0,
+ "301": 301164032.0,
+ "302": 301164032.0,
+ "303": 301164032.0,
+ "304": 301164032.0,
+ "305": 301164032.0,
+ "306": 301164032.0,
+ "307": 301164032.0,
+ "308": 301164032.0,
+ "309": 301164032.0,
+ "310": 301164032.0,
+ "311": 301164032.0,
+ "312": 301164032.0,
+ "313": 301164032.0,
+ "314": 301164032.0,
+ "315": 301164032.0,
+ "316": 301164032.0,
+ "317": 301164032.0,
+ "318": 301164032.0,
+ "319": 301164032.0,
+ "320": 301164032.0,
+ "321": 301164032.0,
+ "322": 301164032.0,
+ "323": 301164032.0,
+ "324": 301164032.0,
+ "325": 301164032.0,
+ "326": 301164032.0,
+ "327": 301164032.0,
+ "328": 301164032.0,
+ "329": 301164032.0,
+ "330": 301164032.0,
+ "331": 301164032.0,
+ "332": 301164032.0,
+ "333": 301164032.0,
+ "334": 301164032.0,
+ "335": 301164032.0,
+ "336": 301164032.0,
+ "337": 301164032.0,
+ "338": 301164032.0,
+ "339": 301164032.0,
+ "340": 301164032.0,
+ "341": 301164032.0,
+ "342": 301164032.0,
+ "343": 301164032.0,
+ "344": 301164032.0,
+ "345": 301164032.0,
+ "346": 301164032.0,
+ "347": 301164032.0,
+ "348": 301164032.0,
+ "349": 301164032.0,
+ "350": 301164032.0,
+ "351": 301164032.0,
+ "352": 301164032.0,
+ "353": 301164032.0,
+ "354": 301164032.0,
+ "355": 301164032.0,
+ "356": 301164032.0,
+ "357": 301164032.0,
+ "358": 301164032.0,
+ "359": 301164032.0,
+ "360": 301164032.0,
+ "361": 301164032.0,
+ "362": 301164032.0,
+ "363": 301164032.0,
+ "364": 301164032.0,
+ "365": 301164032.0,
+ "366": 301164032.0,
+ "367": 301164032.0,
+ "368": 301164032.0,
+ "369": 301164032.0,
+ "370": 301164032.0,
+ "371": 301164032.0,
+ "372": 301164032.0,
+ "373": 301164032.0,
+ "374": 301164032.0,
+ "375": 301164032.0,
+ "376": 301164032.0,
+ "377": 301164032.0,
+ "378": 301164032.0,
+ "379": 301164032.0,
+ "380": 301164032.0,
+ "381": 301164032.0,
+ "382": 301164032.0,
+ "383": 301164032.0,
+ "384": 301164032.0,
+ "385": 301164032.0,
+ "386": 301164032.0,
+ "387": 301164032.0,
+ "388": 301164032.0,
+ "389": 301164032.0,
+ "390": 301164032.0,
+ "391": 301164032.0,
+ "392": 301164032.0,
+ "393": 301164032.0,
+ "394": 301164032.0,
+ "395": 301164032.0,
+ "396": 301164032.0,
+ "397": 301164032.0,
+ "398": 301164032.0,
+ "399": 301164032.0,
+ "400": 301164032.0,
+ "401": 301164032.0,
+ "402": 301164032.0,
+ "403": 301164032.0,
+ "404": 301164032.0,
+ "405": 301164032.0,
+ "406": 301164032.0,
+ "407": 301164032.0,
+ "408": 301164032.0,
+ "409": 301164032.0,
+ "410": 301164032.0,
+ "411": 301164032.0,
+ "412": 301164032.0,
+ "413": 301164032.0,
+ "414": 301164032.0,
+ "415": 301164032.0,
+ "416": 301164032.0,
+ "417": 301164032.0,
+ "418": 301164032.0,
+ "419": 301164032.0,
+ "420": 301164032.0,
+ "421": 301164032.0,
+ "422": 301164032.0,
+ "423": 301164032.0,
+ "424": 301164032.0,
+ "425": 301164032.0,
+ "426": 301164032.0,
+ "427": 301164032.0,
+ "428": 301164032.0,
+ "429": 301164032.0,
+ "430": 301164032.0,
+ "431": 301164032.0,
+ "432": 301164032.0,
+ "433": 301164032.0,
+ "434": 301164032.0,
+ "435": 301164032.0,
+ "436": 301164032.0,
+ "437": 301164032.0,
+ "438": 301164032.0,
+ "439": 301164032.0,
+ "440": 301164032.0,
+ "441": 301164032.0,
+ "442": 301164032.0,
+ "443": 301164032.0,
+ "444": 301164032.0,
+ "445": 301164032.0,
+ "446": 301164032.0,
+ "447": 301164032.0,
+ "448": 301164032.0,
+ "449": 301164032.0,
+ "450": 301164032.0,
+ "451": 301164032.0,
+ "452": 301164032.0,
+ "453": 301164032.0,
+ "454": 301164032.0,
+ "455": 301164032.0,
+ "456": 301164032.0,
+ "457": 301164032.0,
+ "458": 301164032.0,
+ "459": 301164032.0,
+ "460": 301164032.0,
+ "461": 301164032.0,
+ "462": 301164032.0,
+ "463": 301164032.0,
+ "464": 301164032.0,
+ "465": 301164032.0,
+ "466": 301164032.0,
+ "467": 301164032.0,
+ "468": 301164032.0,
+ "469": 301164032.0,
+ "470": 301164032.0,
+ "471": 301164032.0,
+ "472": 301164032.0,
+ "473": 301164032.0,
+ "474": 301164032.0,
+ "475": 301164032.0,
+ "476": 301164032.0,
+ "477": 301164032.0,
+ "478": 301164032.0,
+ "479": 301164032.0,
+ "480": 301164032.0,
+ "481": 301164032.0,
+ "482": 301164032.0,
+ "483": 301164032.0,
+ "484": 301164032.0,
+ "485": 301164032.0,
+ "486": 301164032.0,
+ "487": 301164032.0,
+ "488": 301164032.0,
+ "489": 301164032.0,
+ "490": 301164032.0,
+ "491": 301164032.0,
+ "492": 301164032.0,
+ "493": 301164032.0,
+ "494": 301164032.0,
+ "495": 301164032.0,
+ "496": 301164032.0,
+ "497": 301164032.0,
+ "498": 301164032.0,
+ "499": 301164032.0,
+ "500": 301164032.0,
+ "501": 301164032.0,
+ "502": 301164032.0,
+ "503": 301164032.0,
+ "504": 301164032.0,
+ "505": 301164032.0,
+ "506": 301164032.0,
+ "507": 301164032.0,
+ "508": 301164032.0,
+ "509": 301164032.0,
+ "510": 301164032.0,
+ "511": 301164032.0,
+ "512": 301164032.0,
+ "513": 301164032.0,
+ "514": 301164032.0,
+ "515": 301164032.0,
+ "516": 301164032.0,
+ "517": 301164032.0,
+ "518": 301164032.0,
+ "519": 301164032.0,
+ "520": 301164032.0,
+ "521": 301164032.0,
+ "522": 301164032.0,
+ "523": 301164032.0,
+ "524": 301164032.0,
+ "525": 301164032.0,
+ "526": 301164032.0,
+ "527": 301164032.0,
+ "528": 301164032.0,
+ "529": 301164032.0,
+ "530": 301164032.0,
+ "531": 301164032.0,
+ "532": 301164032.0,
+ "533": 301164032.0,
+ "534": 301164032.0,
+ "535": 301164032.0,
+ "536": 301164032.0,
+ "537": 301164032.0,
+ "538": 301164032.0,
+ "539": 301164032.0,
+ "540": 301164032.0,
+ "541": 301164032.0,
+ "542": 301164032.0,
+ "543": 301164032.0,
+ "544": 301164032.0,
+ "545": 301164032.0,
+ "546": 301164032.0,
+ "547": 301164032.0,
+ "548": 301164032.0,
+ "549": 301164032.0,
+ "550": 301164032.0,
+ "551": 301164032.0,
+ "552": 301164032.0,
+ "553": 301164032.0,
+ "554": 301164032.0,
+ "555": 301164032.0,
+ "556": 301164032.0,
+ "557": 301164032.0,
+ "558": 301164032.0,
+ "559": 301164032.0,
+ "560": 301164032.0,
+ "561": 301164032.0,
+ "562": 301164032.0,
+ "563": 301164032.0,
+ "564": 301164032.0,
+ "565": 301164032.0,
+ "566": 301164032.0,
+ "567": 301164032.0,
+ "568": 301164032.0,
+ "569": 301164032.0,
+ "570": 301164032.0,
+ "571": 301164032.0,
+ "572": 301164032.0,
+ "573": 301164032.0,
+ "574": 301164032.0,
+ "575": 301164032.0,
+ "576": 301164032.0,
+ "577": 301164032.0,
+ "578": 301164032.0,
+ "579": 301164032.0,
+ "580": 301164032.0,
+ "581": 301164032.0,
+ "582": 301164032.0,
+ "583": 301164032.0,
+ "584": 301164032.0,
+ "585": 301164032.0,
+ "586": 301164032.0,
+ "587": 301164032.0,
+ "588": 301164032.0,
+ "589": 301164032.0,
+ "590": 301164032.0,
+ "591": 301164032.0,
+ "592": 301164032.0,
+ "593": 301164032.0,
+ "594": 301164032.0,
+ "595": 301164032.0,
+ "596": 301164032.0,
+ "597": 301164032.0,
+ "598": 301164032.0,
+ "599": 301164032.0,
+ "600": 301164032.0,
+ "601": 301164032.0,
+ "602": 301164032.0,
+ "603": 301164032.0,
+ "604": 301164032.0,
+ "605": 301164032.0,
+ "606": 301164032.0,
+ "607": 301164032.0,
+ "608": 301164032.0,
+ "609": 301164032.0,
+ "610": 301164032.0,
+ "611": 301164032.0,
+ "612": 301164032.0,
+ "613": 301164032.0,
+ "614": 301164032.0,
+ "615": 301164032.0,
+ "616": 301164032.0,
+ "617": 301164032.0,
+ "618": 301164032.0,
+ "619": 301164032.0,
+ "620": 301164032.0,
+ "621": 301164032.0,
+ "622": 301164032.0,
+ "623": 301164032.0,
+ "624": 301164032.0,
+ "625": 301164032.0,
+ "626": 301164032.0,
+ "627": 301164032.0,
+ "628": 301164032.0,
+ "629": 301164032.0,
+ "630": 301164032.0,
+ "631": 301164032.0,
+ "632": 301164032.0,
+ "633": 301164032.0,
+ "634": 301164032.0,
+ "635": 301164032.0,
+ "636": 301164032.0,
+ "637": 301164032.0,
+ "638": 301164032.0,
+ "639": 301164032.0,
+ "640": 301164032.0,
+ "641": 301164032.0,
+ "642": 301164032.0,
+ "643": 301164032.0,
+ "644": 301164032.0,
+ "645": 301164032.0,
+ "646": 301164032.0,
+ "647": 301164032.0,
+ "648": 301164032.0,
+ "649": 301164032.0,
+ "650": 301164032.0,
+ "651": 301164032.0,
+ "652": 301164032.0,
+ "653": 301164032.0,
+ "654": 301164032.0,
+ "655": 301164032.0,
+ "656": 301164032.0,
+ "657": 301164032.0,
+ "658": 301164032.0,
+ "659": 301164032.0,
+ "660": 301164032.0,
+ "661": 301164032.0,
+ "662": 301164032.0,
+ "663": 301164032.0,
+ "664": 301164032.0,
+ "665": 301164032.0,
+ "666": 301164032.0,
+ "667": 301164032.0,
+ "668": 301164032.0,
+ "669": 301164032.0,
+ "670": 301164032.0,
+ "671": 301164032.0,
+ "672": 301164032.0,
+ "673": 301164032.0,
+ "674": 301164032.0,
+ "675": 301164032.0,
+ "676": 301164032.0,
+ "677": 301164032.0,
+ "678": 301164032.0,
+ "679": 301164032.0,
+ "680": 301164032.0,
+ "681": 301164032.0,
+ "682": 301164032.0,
+ "683": 301164032.0,
+ "684": 301164032.0,
+ "685": 301164032.0,
+ "686": 301164032.0,
+ "687": 301164032.0,
+ "688": 301164032.0,
+ "689": 301164032.0,
+ "690": 301164032.0,
+ "691": 301164032.0,
+ "692": 301164032.0,
+ "693": 301164032.0,
+ "694": 301164032.0,
+ "695": 301164032.0,
+ "696": 301164032.0,
+ "697": 301164032.0,
+ "698": 301164032.0,
+ "699": 301164032.0,
+ "700": 301164032.0,
+ "701": 301164032.0,
+ "702": 301164032.0,
+ "703": 301164032.0,
+ "704": 301164032.0,
+ "705": 301164032.0,
+ "706": 301164032.0,
+ "707": 301164032.0,
+ "708": 301164032.0,
+ "709": 301164032.0,
+ "710": 301164032.0,
+ "711": 301164032.0,
+ "712": 301164032.0,
+ "713": 301164032.0,
+ "714": 301164032.0,
+ "715": 301164032.0,
+ "716": 301164032.0,
+ "717": 301164032.0,
+ "718": 301164032.0,
+ "719": 301164032.0,
+ "720": 301164032.0,
+ "721": 301164032.0,
+ "722": 301164032.0,
+ "723": 301164032.0,
+ "724": 301164032.0,
+ "725": 301164032.0,
+ "726": 301164032.0,
+ "727": 301164032.0,
+ "728": 301164032.0,
+ "729": 301164032.0,
+ "730": 301164032.0,
+ "731": 301164032.0,
+ "732": 301164032.0,
+ "733": 301164032.0,
+ "734": 301164032.0,
+ "735": 301164032.0,
+ "736": 301164032.0,
+ "737": 301164032.0,
+ "738": 301164032.0,
+ "739": 301164032.0,
+ "740": 301164032.0,
+ "741": 301164032.0,
+ "742": 301164032.0,
+ "743": 301164032.0,
+ "744": 301164032.0,
+ "745": 301164032.0,
+ "746": 301164032.0,
+ "747": 301164032.0,
+ "748": 301164032.0,
+ "749": 301164032.0,
+ "750": 301164032.0,
+ "751": 301164032.0,
+ "752": 301164032.0,
+ "753": 301164032.0,
+ "754": 301164032.0,
+ "755": 301164032.0,
+ "756": 301164032.0,
+ "757": 301164032.0,
+ "758": 301164032.0,
+ "759": 301164032.0,
+ "760": 301164032.0,
+ "761": 301164032.0,
+ "762": 301164032.0,
+ "763": 301164032.0,
+ "764": 301164032.0,
+ "765": 301164032.0,
+ "766": 301164032.0,
+ "767": 301164032.0,
+ "768": 301164032.0,
+ "769": 301164032.0,
+ "770": 301164032.0,
+ "771": 301164032.0,
+ "772": 301164032.0,
+ "773": 301164032.0,
+ "774": 301164032.0,
+ "775": 301164032.0,
+ "776": 301164032.0,
+ "777": 301164032.0,
+ "778": 301164032.0,
+ "779": 301164032.0,
+ "780": 301164032.0,
+ "781": 301164032.0,
+ "782": 301164032.0,
+ "783": 301164032.0,
+ "784": 301164032.0,
+ "785": 301164032.0,
+ "786": 301164032.0,
+ "787": 301164032.0,
+ "788": 301164032.0,
+ "789": 301164032.0,
+ "790": 301164032.0,
+ "791": 301164032.0,
+ "792": 301164032.0,
+ "793": 301164032.0,
+ "794": 301164032.0,
+ "795": 301164032.0,
+ "796": 301164032.0,
+ "797": 301164032.0,
+ "798": 301164032.0,
+ "799": 301164032.0,
+ "800": 301164032.0,
+ "801": 301164032.0,
+ "802": 301164032.0,
+ "803": 301164032.0,
+ "804": 301164032.0,
+ "805": 301164032.0,
+ "806": 301164032.0,
+ "807": 301164032.0,
+ "808": 301164032.0,
+ "809": 301164032.0,
+ "810": 301164032.0,
+ "811": 301164032.0,
+ "812": 301164032.0,
+ "813": 301164032.0,
+ "814": 301164032.0,
+ "815": 301164032.0,
+ "816": 301164032.0,
+ "817": 301164032.0,
+ "818": 301164032.0,
+ "819": 301164032.0,
+ "820": 301164032.0,
+ "821": 301164032.0,
+ "822": 301164032.0,
+ "823": 301164032.0,
+ "824": 301164032.0,
+ "825": 301164032.0,
+ "826": 301164032.0,
+ "827": 301164032.0,
+ "828": 301164032.0,
+ "829": 301164032.0,
+ "830": 301164032.0,
+ "831": 301164032.0,
+ "832": 301164032.0,
+ "833": 301164032.0,
+ "834": 301164032.0,
+ "835": 301164032.0,
+ "836": 301164032.0,
+ "837": 301164032.0,
+ "838": 301164032.0,
+ "839": 301164032.0,
+ "840": 301164032.0,
+ "841": 301164032.0,
+ "842": 301164032.0,
+ "843": 301164032.0,
+ "844": 301164032.0,
+ "845": 301164032.0,
+ "846": 301164032.0,
+ "847": 301164032.0,
+ "848": 301164032.0,
+ "849": 301164032.0,
+ "850": 301164032.0,
+ "851": 301164032.0,
+ "852": 301164032.0,
+ "853": 301164032.0,
+ "854": 301164032.0,
+ "855": 301164032.0,
+ "856": 301164032.0,
+ "857": 301164032.0,
+ "858": 301164032.0,
+ "859": 301164032.0,
+ "860": 301164032.0,
+ "861": 301164032.0,
+ "862": 301164032.0,
+ "863": 301164032.0,
+ "864": 301164032.0,
+ "865": 301164032.0,
+ "866": 301164032.0,
+ "867": 301164032.0,
+ "868": 301164032.0,
+ "869": 301164032.0,
+ "870": 301164032.0,
+ "871": 301164032.0,
+ "872": 301164032.0,
+ "873": 301164032.0,
+ "874": 301164032.0,
+ "875": 301164032.0,
+ "876": 301164032.0,
+ "877": 301164032.0,
+ "878": 301164032.0,
+ "879": 301164032.0,
+ "880": 301164032.0,
+ "881": 301164032.0,
+ "882": 301164032.0,
+ "883": 301164032.0,
+ "884": 301164032.0,
+ "885": 301164032.0,
+ "886": 301164032.0,
+ "887": 301164032.0,
+ "888": 301164032.0,
+ "889": 301164032.0,
+ "890": 301164032.0,
+ "891": 301164032.0,
+ "892": 301164032.0,
+ "893": 301164032.0,
+ "894": 301164032.0,
+ "895": 301164032.0,
+ "896": 301164032.0,
+ "897": 301164032.0,
+ "898": 301164032.0,
+ "899": 301164032.0,
+ "900": 301164032.0,
+ "901": 301164032.0,
+ "902": 301164032.0,
+ "903": 301164032.0,
+ "904": 301164032.0,
+ "905": 301164032.0,
+ "906": 301164032.0,
+ "907": 301164032.0,
+ "908": 301164032.0,
+ "909": 301164032.0,
+ "910": 301164032.0,
+ "911": 301164032.0,
+ "912": 301164032.0,
+ "913": 301164032.0,
+ "914": 301164032.0,
+ "915": 301164032.0,
+ "916": 301164032.0,
+ "917": 301164032.0,
+ "918": 301164032.0,
+ "919": 301164032.0,
+ "920": 301164032.0,
+ "921": 301164032.0,
+ "922": 301164032.0,
+ "923": 301164032.0,
+ "924": 301164032.0,
+ "925": 301164032.0,
+ "926": 301164032.0,
+ "927": 301164032.0,
+ "928": 301164032.0,
+ "929": 301164032.0,
+ "930": 301164032.0,
+ "931": 301164032.0,
+ "932": 301164032.0,
+ "933": 301164032.0,
+ "934": 301164032.0,
+ "935": 301164032.0,
+ "936": 301164032.0,
+ "937": 301164032.0,
+ "938": 301164032.0,
+ "939": 301164032.0,
+ "940": 301164032.0,
+ "941": 301164032.0,
+ "942": 301164032.0,
+ "943": 301164032.0,
+ "944": 301164032.0,
+ "945": 301164032.0,
+ "946": 301164032.0,
+ "947": 301164032.0,
+ "948": 301164032.0,
+ "949": 301164032.0,
+ "950": 301164032.0,
+ "951": 301164032.0,
+ "952": 301164032.0,
+ "953": 301164032.0,
+ "954": 301164032.0,
+ "955": 301164032.0,
+ "956": 301164032.0,
+ "957": 301164032.0,
+ "958": 301164032.0,
+ "959": 301164032.0,
+ "960": 301164032.0,
+ "961": 301164032.0,
+ "962": 301164032.0,
+ "963": 301164032.0,
+ "964": 301164032.0,
+ "965": 301164032.0,
+ "966": 301164032.0,
+ "967": 301164032.0,
+ "968": 301164032.0,
+ "969": 301164032.0,
+ "970": 301164032.0,
+ "971": 301164032.0,
+ "972": 301164032.0,
+ "973": 301164032.0,
+ "974": 301164032.0,
+ "975": 301164032.0,
+ "976": 301164032.0,
+ "977": 301164032.0,
+ "978": 301164032.0,
+ "979": 301164032.0,
+ "980": 301164032.0,
+ "981": 301164032.0,
+ "982": 301164032.0,
+ "983": 301164032.0,
+ "984": 301164032.0,
+ "985": 301164032.0,
+ "986": 301164032.0,
+ "987": 301164032.0,
+ "988": 301164032.0,
+ "989": 301164032.0,
+ "990": 301164032.0,
+ "991": 301164032.0,
+ "992": 301164032.0,
+ "993": 301164032.0,
+ "994": 301164032.0,
+ "995": 301164032.0,
+ "996": 301164032.0,
+ "997": 301164032.0,
+ "998": 301164032.0,
+ "999": 301164032.0,
+ "1000": 301164032.0,
+ "1001": 301164032.0,
+ "1002": 301164032.0,
+ "1003": 301164032.0,
+ "1004": 301164032.0,
+ "1005": 301164032.0,
+ "1006": 301164032.0,
+ "1007": 301164032.0,
+ "1008": 301164032.0,
+ "1009": 301164032.0,
+ "1010": 301164032.0,
+ "1011": 301164032.0,
+ "1012": 301164032.0,
+ "1013": 301164032.0,
+ "1014": 301164032.0,
+ "1015": 301164032.0,
+ "1016": 301164032.0,
+ "1017": 301164032.0,
+ "1018": 301164032.0,
+ "1019": 301164032.0,
+ "1020": 301164032.0,
+ "1021": 301164032.0,
+ "1022": 301164032.0,
+ "1023": 301164032.0,
+ "1024": 301164032.0,
+ "1025": 301164032.0,
+ "1026": 301164032.0,
+ "1027": 301164032.0,
+ "1028": 301164032.0,
+ "1029": 301164032.0,
+ "1030": 301164032.0,
+ "1031": 301164032.0,
+ "1032": 301164032.0,
+ "1033": 301164032.0,
+ "1034": 301164032.0,
+ "1035": 301164032.0,
+ "1036": 301164032.0,
+ "1037": 301164032.0,
+ "1038": 301164032.0,
+ "1039": 301164032.0,
+ "1040": 301164032.0,
+ "1041": 301164032.0,
+ "1042": 301164032.0,
+ "1043": 301164032.0,
+ "1044": 301164032.0,
+ "1045": 301164032.0,
+ "1046": 301164032.0,
+ "1047": 301164032.0,
+ "1048": 301164032.0,
+ "1049": 301164032.0,
+ "1050": 301164032.0,
+ "1051": 301164032.0,
+ "1052": 301164032.0,
+ "1053": 301164032.0,
+ "1054": 301164032.0,
+ "1055": 301164032.0,
+ "1056": 301164032.0,
+ "1057": 301164032.0,
+ "1058": 301164032.0,
+ "1059": 301164032.0,
+ "1060": 301164032.0,
+ "1061": 301164032.0,
+ "1062": 301164032.0,
+ "1063": 301164032.0,
+ "1064": 301164032.0,
+ "1065": 301164032.0,
+ "1066": 301164032.0,
+ "1067": 301164032.0,
+ "1068": 301164032.0,
+ "1069": 301164032.0,
+ "1070": 301164032.0,
+ "1071": 301164032.0,
+ "1072": 301164032.0,
+ "1073": 301164032.0,
+ "1074": 301164032.0,
+ "1075": 301164032.0,
+ "1076": 301164032.0,
+ "1077": 301164032.0,
+ "1078": 301164032.0,
+ "1079": 301164032.0,
+ "1080": 301164032.0,
+ "1081": 301164032.0,
+ "1082": 301164032.0,
+ "1083": 301164032.0,
+ "1084": 301164032.0,
+ "1085": 301164032.0,
+ "1086": 301164032.0,
+ "1087": 301164032.0,
+ "1088": 301164032.0,
+ "1089": 301164032.0,
+ "1090": 301164032.0,
+ "1091": 301164032.0,
+ "1092": 301164032.0,
+ "1093": 301164032.0,
+ "1094": 301164032.0,
+ "1095": 301164032.0,
+ "1096": 301164032.0,
+ "1097": 301164032.0,
+ "1098": 301164032.0,
+ "1099": 301164032.0,
+ "1100": 301164032.0,
+ "1101": 301164032.0,
+ "1102": 301164032.0,
+ "1103": 301164032.0,
+ "1104": 301164032.0,
+ "1105": 301164032.0,
+ "1106": 301164032.0,
+ "1107": 301164032.0,
+ "1108": 301164032.0,
+ "1109": 301164032.0,
+ "1110": 301164032.0,
+ "1111": 301164032.0,
+ "1112": 301164032.0,
+ "1113": 301164032.0,
+ "1114": 301164032.0,
+ "1115": 301164032.0,
+ "1116": 301164032.0,
+ "1117": 301164032.0,
+ "1118": 301164032.0,
+ "1119": 301164032.0,
+ "1120": 301164032.0,
+ "1121": 301164032.0,
+ "1122": 301164032.0,
+ "1123": 301164032.0,
+ "1124": 301164032.0,
+ "1125": 301164032.0,
+ "1126": 301164032.0,
+ "1127": 301164032.0,
+ "1128": 301164032.0,
+ "1129": 301164032.0,
+ "1130": 301164032.0,
+ "1131": 301164032.0,
+ "1132": 301164032.0,
+ "1133": 301164032.0,
+ "1134": 301164032.0,
+ "1135": 301164032.0,
+ "1136": 301164032.0,
+ "1137": 301164032.0,
+ "1138": 301164032.0,
+ "1139": 301164032.0,
+ "1140": 301164032.0,
+ "1141": 301164032.0,
+ "1142": 301164032.0,
+ "1143": 301164032.0,
+ "1144": 301164032.0,
+ "1145": 301164032.0,
+ "1146": 301164032.0,
+ "1147": 301164032.0,
+ "1148": 301164032.0,
+ "1149": 301164032.0,
+ "1150": 301164032.0,
+ "1151": 301164032.0,
+ "1152": 301164032.0,
+ "1153": 301164032.0,
+ "1154": 301164032.0,
+ "1155": 301164032.0,
+ "1156": 301164032.0,
+ "1157": 301164032.0,
+ "1158": 301164032.0,
+ "1159": 301164032.0,
+ "1160": 301164032.0,
+ "1161": 301164032.0,
+ "1162": 301164032.0,
+ "1163": 301164032.0,
+ "1164": 301164032.0,
+ "1165": 301164032.0,
+ "1166": 301164032.0,
+ "1167": 301164032.0,
+ "1168": 301164032.0,
+ "1169": 301164032.0,
+ "1170": 301164032.0,
+ "1171": 301164032.0,
+ "1172": 301164032.0,
+ "1173": 301164032.0,
+ "1174": 301164032.0,
+ "1175": 301164032.0,
+ "1176": 301164032.0,
+ "1177": 301164032.0,
+ "1178": 301164032.0,
+ "1179": 301164032.0,
+ "1180": 301164032.0,
+ "1181": 301164032.0,
+ "1182": 301164032.0,
+ "1183": 301164032.0,
+ "1184": 301164032.0,
+ "1185": 301164032.0,
+ "1186": 301164032.0,
+ "1187": 301164032.0,
+ "1188": 301164032.0,
+ "1189": 301164032.0,
+ "1190": 301164032.0,
+ "1191": 301164032.0,
+ "1192": 301164032.0,
+ "1193": 301164032.0,
+ "1194": 301164032.0,
+ "1195": 301164032.0,
+ "1196": 301164032.0,
+ "1197": 301164032.0,
+ "1198": 301164032.0,
+ "1199": 301164032.0,
+ "1200": 301164032.0,
+ "1201": 301164032.0,
+ "1202": 301164032.0,
+ "1203": 301164032.0,
+ "1204": 301164032.0,
+ "1205": 301164032.0,
+ "1206": 301164032.0,
+ "1207": 301164032.0,
+ "1208": 301164032.0,
+ "1209": 301164032.0,
+ "1210": 301164032.0,
+ "1211": 301164032.0,
+ "1212": 301164032.0,
+ "1213": 301164032.0,
+ "1214": 301164032.0,
+ "1215": 301164032.0,
+ "1216": 301164032.0,
+ "1217": 301164032.0,
+ "1218": 301164032.0,
+ "1219": 301164032.0,
+ "1220": 301164032.0,
+ "1221": 301164032.0,
+ "1222": 301164032.0,
+ "1223": 301164032.0,
+ "1224": 301164032.0,
+ "1225": 301164032.0,
+ "1226": 301164032.0,
+ "1227": 301164032.0,
+ "1228": 301164032.0,
+ "1229": 301164032.0,
+ "1230": 301164032.0,
+ "1231": 301164032.0,
+ "1232": 301164032.0,
+ "1233": 301164032.0,
+ "1234": 301164032.0,
+ "1235": 301164032.0,
+ "1236": 301164032.0,
+ "1237": 301164032.0,
+ "1238": 301164032.0,
+ "1239": 301164032.0,
+ "1240": 301164032.0,
+ "1241": 301164032.0,
+ "1242": 301164032.0,
+ "1243": 301164032.0,
+ "1244": 301164032.0,
+ "1245": 301164032.0,
+ "1246": 301164032.0,
+ "1247": 301164032.0,
+ "1248": 301164032.0,
+ "1249": 301164032.0,
+ "1250": 301164032.0,
+ "1251": 301164032.0,
+ "1252": 301164032.0,
+ "1253": 301164032.0,
+ "1254": 301164032.0,
+ "1255": 301164032.0,
+ "1256": 301164032.0,
+ "1257": 301164032.0,
+ "1258": 301164032.0,
+ "1259": 301164032.0,
+ "1260": 301164032.0,
+ "1261": 301164032.0,
+ "1262": 301164032.0,
+ "1263": 301164032.0,
+ "1264": 301164032.0,
+ "1265": 301164032.0,
+ "1266": 301164032.0,
+ "1267": 301164032.0,
+ "1268": 301164032.0,
+ "1269": 301164032.0,
+ "1270": 301164032.0,
+ "1271": 301164032.0,
+ "1272": 301164032.0,
+ "1273": 301164032.0,
+ "1274": 301164032.0,
+ "1275": 301164032.0,
+ "1276": 301164032.0,
+ "1277": 301164032.0,
+ "1278": 301164032.0,
+ "1279": 301164032.0,
+ "1280": 301164032.0,
+ "1281": 301164032.0,
+ "1282": 301164032.0,
+ "1283": 301164032.0,
+ "1284": 301164032.0,
+ "1285": 301164032.0,
+ "1286": 301164032.0,
+ "1287": 301164032.0,
+ "1288": 301164032.0,
+ "1289": 301164032.0,
+ "1290": 301164032.0,
+ "1291": 301164032.0,
+ "1292": 301164032.0,
+ "1293": 301164032.0,
+ "1294": 301164032.0,
+ "1295": 301164032.0,
+ "1296": 301164032.0,
+ "1297": 301164032.0,
+ "1298": 301164032.0,
+ "1299": 301164032.0,
+ "1300": 301164032.0,
+ "1301": 301164032.0,
+ "1302": 301164032.0,
+ "1303": 301164032.0,
+ "1304": 301164032.0,
+ "1305": 301164032.0,
+ "1306": 301164032.0,
+ "1307": 301164032.0,
+ "1308": 301164032.0,
+ "1309": 301164032.0,
+ "1310": 301164032.0,
+ "1311": 301164032.0,
+ "1312": 301164032.0,
+ "1313": 301164032.0,
+ "1314": 301164032.0,
+ "1315": 301164032.0,
+ "1316": 301164032.0,
+ "1317": 301164032.0,
+ "1318": 301164032.0,
+ "1319": 301164032.0,
+ "1320": 301164032.0,
+ "1321": 301164032.0,
+ "1322": 301164032.0,
+ "1323": 301164032.0,
+ "1324": 301164032.0,
+ "1325": 301164032.0,
+ "1326": 301164032.0,
+ "1327": 301164032.0,
+ "1328": 301164032.0,
+ "1329": 301164032.0,
+ "1330": 301164032.0,
+ "1331": 301164032.0,
+ "1332": 301164032.0,
+ "1333": 301164032.0,
+ "1334": 301164032.0,
+ "1335": 301164032.0,
+ "1336": 301164032.0,
+ "1337": 301164032.0,
+ "1338": 301164032.0,
+ "1339": 301164032.0,
+ "1340": 301164032.0,
+ "1341": 301164032.0,
+ "1342": 301164032.0,
+ "1343": 301164032.0,
+ "1344": 301164032.0,
+ "1345": 301164032.0,
+ "1346": 301164032.0,
+ "1347": 301164032.0,
+ "1348": 301164032.0,
+ "1349": 301164032.0,
+ "1350": 301164032.0,
+ "1351": 301164032.0,
+ "1352": 301164032.0,
+ "1353": 301164032.0,
+ "1354": 301164032.0,
+ "1355": 301164032.0,
+ "1356": 301164032.0,
+ "1357": 301164032.0,
+ "1358": 301164032.0,
+ "1359": 301164032.0,
+ "1360": 301164032.0,
+ "1361": 301164032.0,
+ "1362": 301164032.0,
+ "1363": 301164032.0,
+ "1364": 301164032.0,
+ "1365": 301164032.0,
+ "1366": 301164032.0,
+ "1367": 301164032.0,
+ "1368": 301164032.0,
+ "1369": 301164032.0,
+ "1370": 301164032.0,
+ "1371": 301164032.0,
+ "1372": 301164032.0,
+ "1373": 301164032.0,
+ "1374": 301164032.0,
+ "1375": 301164032.0,
+ "1376": 301164032.0,
+ "1377": 301164032.0,
+ "1378": 301164032.0,
+ "1379": 301164032.0,
+ "1380": 301164032.0,
+ "1381": 301164032.0,
+ "1382": 301164032.0,
+ "1383": 301164032.0,
+ "1384": 301164032.0,
+ "1385": 301164032.0,
+ "1386": 301164032.0,
+ "1387": 301164032.0,
+ "1388": 301164032.0,
+ "1389": 301164032.0,
+ "1390": 301164032.0,
+ "1391": 301164032.0,
+ "1392": 301164032.0,
+ "1393": 301164032.0,
+ "1394": 301164032.0,
+ "1395": 301164032.0,
+ "1396": 301164032.0,
+ "1397": 301164032.0,
+ "1398": 301164032.0,
+ "1399": 301164032.0,
+ "1400": 301164032.0,
+ "1401": 301164032.0,
+ "1402": 301164032.0,
+ "1403": 301164032.0,
+ "1404": 301164032.0,
+ "1405": 301164032.0,
+ "1406": 301164032.0,
+ "1407": 301164032.0,
+ "1408": 301164032.0,
+ "1409": 301164032.0,
+ "1410": 301164032.0,
+ "1411": 301164032.0,
+ "1412": 301164032.0,
+ "1413": 301164032.0,
+ "1414": 301164032.0,
+ "1415": 301164032.0,
+ "1416": 301164032.0,
+ "1417": 301164032.0,
+ "1418": 301164032.0,
+ "1419": 301164032.0,
+ "1420": 301164032.0,
+ "1421": 301164032.0,
+ "1422": 301164032.0,
+ "1423": 301164032.0,
+ "1424": 301164032.0,
+ "1425": 301164032.0,
+ "1426": 301164032.0,
+ "1427": 301164032.0,
+ "1428": 301164032.0,
+ "1429": 301164032.0,
+ "1430": 301164032.0,
+ "1431": 301164032.0,
+ "1432": 301164032.0,
+ "1433": 301164032.0,
+ "1434": 301164032.0,
+ "1435": 301164032.0,
+ "1436": 301164032.0,
+ "1437": 301164032.0,
+ "1438": 301164032.0,
+ "1439": 301164032.0,
+ "1440": 301164032.0,
+ "1441": 301164032.0,
+ "1442": 301164032.0,
+ "1443": 301164032.0,
+ "1444": 301164032.0,
+ "1445": 301164032.0,
+ "1446": 301164032.0,
+ "1447": 301164032.0,
+ "1448": 301164032.0,
+ "1449": 301164032.0,
+ "1450": 301164032.0,
+ "1451": 301164032.0,
+ "1452": 301164032.0,
+ "1453": 301164032.0,
+ "1454": 301164032.0,
+ "1455": 301164032.0,
+ "1456": 301164032.0,
+ "1457": 301164032.0,
+ "1458": 301164032.0,
+ "1459": 301164032.0,
+ "1460": 301164032.0,
+ "1461": 301164032.0,
+ "1462": 301164032.0,
+ "1463": 301164032.0,
+ "1464": 301164032.0,
+ "1465": 301164032.0,
+ "1466": 301164032.0,
+ "1467": 301164032.0,
+ "1468": 301164032.0,
+ "1469": 301164032.0,
+ "1470": 301164032.0,
+ "1471": 301164032.0,
+ "1472": 301164032.0,
+ "1473": 301164032.0,
+ "1474": 301164032.0,
+ "1475": 301164032.0,
+ "1476": 301164032.0,
+ "1477": 301164032.0,
+ "1478": 301164032.0,
+ "1479": 301164032.0,
+ "1480": 301164032.0,
+ "1481": 301164032.0,
+ "1482": 301164032.0,
+ "1483": 301164032.0,
+ "1484": 301164032.0,
+ "1485": 301164032.0,
+ "1486": 301164032.0,
+ "1487": 301164032.0,
+ "1488": 301164032.0,
+ "1489": 301164032.0,
+ "1490": 301164032.0,
+ "1491": 301164032.0,
+ "1492": 301164032.0,
+ "1493": 301164032.0,
+ "1494": 301164032.0,
+ "1495": 301164032.0,
+ "1496": 301164032.0,
+ "1497": 301164032.0,
+ "1498": 301164032.0,
+ "1499": 301164032.0,
+ "1500": 301164032.0,
+ "1501": 301164032.0,
+ "1502": 301164032.0,
+ "1503": 301164032.0,
+ "1504": 301164032.0,
+ "1505": 301164032.0,
+ "1506": 301164032.0,
+ "1507": 301164032.0,
+ "1508": 301164032.0,
+ "1509": 301164032.0,
+ "1510": 301164032.0,
+ "1511": 301164032.0,
+ "1512": 301164032.0,
+ "1513": 301164032.0,
+ "1514": 301164032.0,
+ "1515": 301164032.0,
+ "1516": 301164032.0,
+ "1517": 301164032.0,
+ "1518": 301164032.0,
+ "1519": 301164032.0,
+ "1520": 301164032.0,
+ "1521": 301164032.0,
+ "1522": 301164032.0,
+ "1523": 301164032.0,
+ "1524": 301164032.0,
+ "1525": 301164032.0,
+ "1526": 301164032.0,
+ "1527": 301164032.0,
+ "1528": 301164032.0,
+ "1529": 301164032.0,
+ "1530": 301164032.0,
+ "1531": 301164032.0,
+ "1532": 301164032.0,
+ "1533": 301164032.0,
+ "1534": 301164032.0,
+ "1535": 301164032.0,
+ "1536": 301164032.0,
+ "1537": 301164032.0,
+ "1538": 301164032.0,
+ "1539": 301164032.0,
+ "1540": 301164032.0,
+ "1541": 301164032.0,
+ "1542": 301164032.0,
+ "1543": 301164032.0,
+ "1544": 301164032.0,
+ "1545": 301164032.0,
+ "1546": 301164032.0,
+ "1547": 301164032.0,
+ "1548": 301164032.0,
+ "1549": 301164032.0,
+ "1550": 301164032.0,
+ "1551": 301164032.0,
+ "1552": 301164032.0,
+ "1553": 301164032.0,
+ "1554": 301164032.0,
+ "1555": 301164032.0,
+ "1556": 301164032.0,
+ "1557": 301164032.0,
+ "1558": 301164032.0,
+ "1559": 301164032.0,
+ "1560": 301164032.0,
+ "1561": 301164032.0,
+ "1562": 301164032.0,
+ "1563": 301164032.0,
+ "1564": 301164032.0,
+ "1565": 301164032.0,
+ "1566": 301164032.0,
+ "1567": 301164032.0,
+ "1568": 301164032.0,
+ "1569": 301164032.0,
+ "1570": 301164032.0,
+ "1571": 301164032.0,
+ "1572": 301164032.0,
+ "1573": 301164032.0,
+ "1574": 301164032.0,
+ "1575": 301164032.0,
+ "1576": 301164032.0,
+ "1577": 301164032.0,
+ "1578": 301164032.0,
+ "1579": 301164032.0,
+ "1580": 301164032.0,
+ "1581": 301164032.0,
+ "1582": 301164032.0,
+ "1583": 301164032.0,
+ "1584": 301164032.0,
+ "1585": 301164032.0,
+ "1586": 301164032.0,
+ "1587": 301164032.0,
+ "1588": 301164032.0,
+ "1589": 301164032.0,
+ "1590": 301164032.0,
+ "1591": 301164032.0,
+ "1592": 301164032.0,
+ "1593": 301164032.0,
+ "1594": 301164032.0,
+ "1595": 301164032.0,
+ "1596": 301164032.0,
+ "1597": 301164032.0,
+ "1598": 301164032.0,
+ "1599": 301164032.0,
+ "1600": 301164032.0,
+ "1601": 301164032.0,
+ "1602": 301164032.0,
+ "1603": 301164032.0,
+ "1604": 301164032.0,
+ "1605": 301164032.0,
+ "1606": 301164032.0,
+ "1607": 301164032.0,
+ "1608": 301164032.0,
+ "1609": 301164032.0,
+ "1610": 301164032.0,
+ "1611": 301164032.0,
+ "1612": 301164032.0,
+ "1613": 301164032.0,
+ "1614": 301164032.0,
+ "1615": 301164032.0,
+ "1616": 301164032.0,
+ "1617": 301164032.0,
+ "1618": 301164032.0,
+ "1619": 301164032.0,
+ "1620": 301164032.0,
+ "1621": 301164032.0,
+ "1622": 301164032.0,
+ "1623": 301164032.0,
+ "1624": 301164032.0,
+ "1625": 301164032.0,
+ "1626": 301164032.0,
+ "1627": 301164032.0,
+ "1628": 301164032.0,
+ "1629": 301164032.0,
+ "1630": 301164032.0,
+ "1631": 301164032.0,
+ "1632": 301164032.0,
+ "1633": 301164032.0,
+ "1634": 301164032.0,
+ "1635": 301164032.0,
+ "1636": 301164032.0,
+ "1637": 301164032.0,
+ "1638": 301164032.0,
+ "1639": 301164032.0,
+ "1640": 301164032.0,
+ "1641": 301164032.0,
+ "1642": 301164032.0,
+ "1643": 301164032.0,
+ "1644": 301164032.0,
+ "1645": 301164032.0,
+ "1646": 301164032.0,
+ "1647": 301164032.0,
+ "1648": 301164032.0,
+ "1649": 301164032.0,
+ "1650": 301164032.0,
+ "1651": 301164032.0,
+ "1652": 301164032.0,
+ "1653": 301164032.0,
+ "1654": 301164032.0,
+ "1655": 301164032.0,
+ "1656": 301164032.0,
+ "1657": 301164032.0,
+ "1658": 301164032.0,
+ "1659": 301164032.0,
+ "1660": 301164032.0,
+ "1661": 301164032.0,
+ "1662": 301164032.0,
+ "1663": 301164032.0,
+ "1664": 301164032.0,
+ "1665": 301164032.0,
+ "1666": 301164032.0,
+ "1667": 301164032.0,
+ "1668": 301164032.0,
+ "1669": 301164032.0,
+ "1670": 301164032.0,
+ "1671": 301164032.0,
+ "1672": 301164032.0,
+ "1673": 301164032.0,
+ "1674": 301164032.0,
+ "1675": 301164032.0,
+ "1676": 301164032.0,
+ "1677": 301164032.0,
+ "1678": 301164032.0,
+ "1679": 301164032.0,
+ "1680": 301164032.0,
+ "1681": 301164032.0,
+ "1682": 301164032.0,
+ "1683": 301164032.0,
+ "1684": 301164032.0,
+ "1685": 301164032.0,
+ "1686": 301164032.0,
+ "1687": 301164032.0,
+ "1688": 301164032.0,
+ "1689": 301164032.0,
+ "1690": 301164032.0,
+ "1691": 301164032.0,
+ "1692": 301164032.0,
+ "1693": 301164032.0,
+ "1694": 301164032.0,
+ "1695": 301164032.0,
+ "1696": 301164032.0,
+ "1697": 301164032.0,
+ "1698": 301164032.0,
+ "1699": 301164032.0,
+ "1700": 301164032.0,
+ "1701": 301164032.0,
+ "1702": 301164032.0,
+ "1703": 301164032.0,
+ "1704": 301164032.0,
+ "1705": 301164032.0,
+ "1706": 301164032.0,
+ "1707": 301164032.0,
+ "1708": 301164032.0,
+ "1709": 301164032.0,
+ "1710": 301164032.0,
+ "1711": 301164032.0,
+ "1712": 301164032.0,
+ "1713": 301164032.0,
+ "1714": 301164032.0,
+ "1715": 301164032.0,
+ "1716": 301164032.0,
+ "1717": 301164032.0,
+ "1718": 301164032.0,
+ "1719": 301164032.0,
+ "1720": 301164032.0,
+ "1721": 301164032.0,
+ "1722": 301164032.0,
+ "1723": 301164032.0,
+ "1724": 301164032.0,
+ "1725": 301164032.0,
+ "1726": 301164032.0,
+ "1727": 301164032.0,
+ "1728": 301164032.0,
+ "1729": 301164032.0,
+ "1730": 301164032.0,
+ "1731": 301164032.0,
+ "1732": 301164032.0,
+ "1733": 301164032.0,
+ "1734": 301164032.0,
+ "1735": 301164032.0,
+ "1736": 301164032.0,
+ "1737": 301164032.0,
+ "1738": 301164032.0,
+ "1739": 301164032.0,
+ "1740": 301164032.0,
+ "1741": 301164032.0,
+ "1742": 301164032.0,
+ "1743": 301164032.0,
+ "1744": 301164032.0,
+ "1745": 301164032.0,
+ "1746": 301164032.0,
+ "1747": 301164032.0,
+ "1748": 301164032.0,
+ "1749": 301164032.0,
+ "1750": 301164032.0,
+ "1751": 301164032.0,
+ "1752": 301164032.0,
+ "1753": 301164032.0,
+ "1754": 301164032.0,
+ "1755": 301164032.0,
+ "1756": 301164032.0,
+ "1757": 301164032.0,
+ "1758": 301164032.0,
+ "1759": 301164032.0,
+ "1760": 301164032.0,
+ "1761": 301164032.0,
+ "1762": 301164032.0,
+ "1763": 301164032.0,
+ "1764": 301164032.0,
+ "1765": 301164032.0,
+ "1766": 301164032.0,
+ "1767": 301164032.0,
+ "1768": 301164032.0,
+ "1769": 301164032.0,
+ "1770": 301164032.0,
+ "1771": 301164032.0,
+ "1772": 301164032.0,
+ "1773": 301164032.0,
+ "1774": 301164032.0,
+ "1775": 301164032.0,
+ "1776": 301164032.0,
+ "1777": 301164032.0,
+ "1778": 301164032.0,
+ "1779": 301164032.0,
+ "1780": 301164032.0,
+ "1781": 301164032.0,
+ "1782": 301164032.0,
+ "1783": 301164032.0,
+ "1784": 301164032.0,
+ "1785": 301164032.0,
+ "1786": 301164032.0,
+ "1787": 301164032.0,
+ "1788": 301164032.0,
+ "1789": 301164032.0,
+ "1790": 301164032.0,
+ "1791": 301164032.0,
+ "1792": 301164032.0,
+ "1793": 301164032.0,
+ "1794": 301164032.0,
+ "1795": 301164032.0,
+ "1796": 301164032.0,
+ "1797": 301164032.0,
+ "1798": 301164032.0,
+ "1799": 301164032.0,
+ "1800": 301164032.0,
+ "1801": 301164032.0,
+ "1802": 301164032.0,
+ "1803": 301164032.0,
+ "1804": 301164032.0,
+ "1805": 301164032.0,
+ "1806": 301164032.0,
+ "1807": 301164032.0,
+ "1808": 301164032.0,
+ "1809": 301164032.0,
+ "1810": 301164032.0,
+ "1811": 301164032.0,
+ "1812": 301164032.0,
+ "1813": 301164032.0,
+ "1814": 301164032.0,
+ "1815": 301164032.0,
+ "1816": 301164032.0,
+ "1817": 301164032.0,
+ "1818": 301164032.0,
+ "1819": 301164032.0,
+ "1820": 301164032.0,
+ "1821": 301164032.0,
+ "1822": 301164032.0,
+ "1823": 301164032.0,
+ "1824": 301164032.0,
+ "1825": 301164032.0,
+ "1826": 301164032.0,
+ "1827": 301164032.0,
+ "1828": 301164032.0,
+ "1829": 301164032.0,
+ "1830": 301164032.0,
+ "1831": 301164032.0,
+ "1832": 301164032.0,
+ "1833": 301164032.0,
+ "1834": 301164032.0,
+ "1835": 301164032.0,
+ "1836": 301164032.0,
+ "1837": 301164032.0,
+ "1838": 301164032.0,
+ "1839": 301164032.0,
+ "1840": 301164032.0,
+ "1841": 301164032.0,
+ "1842": 301164032.0,
+ "1843": 301164032.0,
+ "1844": 301164032.0,
+ "1845": 301164032.0,
+ "1846": 301164032.0,
+ "1847": 301164032.0,
+ "1848": 301164032.0,
+ "1849": 301164032.0,
+ "1850": 301164032.0,
+ "1851": 301164032.0,
+ "1852": 301164032.0,
+ "1853": 301164032.0,
+ "1854": 301164032.0,
+ "1855": 301164032.0,
+ "1856": 301164032.0,
+ "1857": 301164032.0,
+ "1858": 301164032.0,
+ "1859": 301164032.0,
+ "1860": 301164032.0,
+ "1861": 301164032.0,
+ "1862": 301164032.0,
+ "1863": 301164032.0,
+ "1864": 301164032.0,
+ "1865": 301164032.0,
+ "1866": 301164032.0,
+ "1867": 301164032.0,
+ "1868": 301164032.0,
+ "1869": 301164032.0,
+ "1870": 301164032.0,
+ "1871": 301164032.0,
+ "1872": 301164032.0,
+ "1873": 301164032.0,
+ "1874": 301164032.0,
+ "1875": 301164032.0,
+ "1876": 301164032.0,
+ "1877": 301164032.0,
+ "1878": 301164032.0,
+ "1879": 301164032.0,
+ "1880": 301164032.0,
+ "1881": 301164032.0,
+ "1882": 301164032.0,
+ "1883": 301164032.0,
+ "1884": 301164032.0,
+ "1885": 301164032.0,
+ "1886": 301164032.0,
+ "1887": 301164032.0,
+ "1888": 301164032.0,
+ "1889": 301164032.0,
+ "1890": 301164032.0,
+ "1891": 301164032.0,
+ "1892": 301164032.0,
+ "1893": 301164032.0,
+ "1894": 301164032.0,
+ "1895": 301164032.0,
+ "1896": 301164032.0,
+ "1897": 301164032.0,
+ "1898": 301164032.0,
+ "1899": 301164032.0,
+ "1900": 301164032.0,
+ "1901": 301164032.0,
+ "1902": 301164032.0,
+ "1903": 301164032.0,
+ "1904": 301164032.0,
+ "1905": 301164032.0,
+ "1906": 301164032.0,
+ "1907": 301164032.0,
+ "1908": 301164032.0,
+ "1909": 301164032.0,
+ "1910": 301164032.0,
+ "1911": 301164032.0,
+ "1912": 301164032.0,
+ "1913": 301164032.0,
+ "1914": 301164032.0,
+ "1915": 301164032.0,
+ "1916": 301164032.0,
+ "1917": 301164032.0,
+ "1918": 301164032.0,
+ "1919": 301164032.0,
+ "1920": 301164032.0,
+ "1921": 301164032.0,
+ "1922": 301164032.0,
+ "1923": 301164032.0,
+ "1924": 301164032.0,
+ "1925": 301164032.0,
+ "1926": 301164032.0,
+ "1927": 301164032.0,
+ "1928": 301164032.0,
+ "1929": 301164032.0,
+ "1930": 301164032.0,
+ "1931": 301164032.0,
+ "1932": 301164032.0,
+ "1933": 301164032.0,
+ "1934": 301164032.0,
+ "1935": 301164032.0,
+ "1936": 301164032.0,
+ "1937": 301164032.0,
+ "1938": 301164032.0,
+ "1939": 301164032.0,
+ "1940": 301164032.0,
+ "1941": 301164032.0,
+ "1942": 301164032.0,
+ "1943": 301164032.0,
+ "1944": 301164032.0,
+ "1945": 301164032.0,
+ "1946": 301164032.0,
+ "1947": 301164032.0,
+ "1948": 301164032.0,
+ "1949": 301164032.0,
+ "1950": 301164032.0,
+ "1951": 301164032.0,
+ "1952": 301164032.0,
+ "1953": 301164032.0,
+ "1954": 301164032.0,
+ "1955": 301164032.0,
+ "1956": 301164032.0,
+ "1957": 301164032.0,
+ "1958": 301164032.0,
+ "1959": 301164032.0,
+ "1960": 301164032.0,
+ "1961": 301164032.0,
+ "1962": 301164032.0,
+ "1963": 301164032.0,
+ "1964": 301164032.0,
+ "1965": 301164032.0,
+ "1966": 301164032.0,
+ "1967": 301164032.0,
+ "1968": 301164032.0,
+ "1969": 301164032.0,
+ "1970": 301164032.0,
+ "1971": 301164032.0,
+ "1972": 301164032.0,
+ "1973": 301164032.0,
+ "1974": 301164032.0,
+ "1975": 301164032.0,
+ "1976": 301164032.0,
+ "1977": 301164032.0,
+ "1978": 301164032.0,
+ "1979": 301164032.0,
+ "1980": 301164032.0,
+ "1981": 301164032.0,
+ "1982": 301164032.0,
+ "1983": 301164032.0,
+ "1984": 301164032.0,
+ "1985": 301164032.0,
+ "1986": 301164032.0,
+ "1987": 301164032.0,
+ "1988": 301164032.0,
+ "1989": 301164032.0,
+ "1990": 301164032.0,
+ "1991": 301164032.0,
+ "1992": 301164032.0,
+ "1993": 301164032.0,
+ "1994": 301164032.0,
+ "1995": 301164032.0,
+ "1996": 301164032.0,
+ "1997": 301164032.0,
+ "1998": 301164032.0,
+ "1999": 301164032.0,
+ "2000": 301164032.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": 359027200.0,
+ "2": 425553408.0,
+ "3": 425553408.0,
+ "4": 425553408.0,
+ "5": 425553408.0,
+ "6": 425553408.0,
+ "7": 425553408.0,
+ "8": 425553408.0,
+ "9": 425553408.0,
+ "10": 425553408.0,
+ "11": 425553408.0,
+ "12": 425553408.0,
+ "13": 425553408.0,
+ "14": 425553408.0,
+ "15": 425553408.0,
+ "16": 425553408.0,
+ "17": 425553408.0,
+ "18": 425553408.0,
+ "19": 425553408.0,
+ "20": 425553408.0,
+ "21": 425553408.0,
+ "22": 425553408.0,
+ "23": 425553408.0,
+ "24": 425553408.0,
+ "25": 425553408.0,
+ "26": 425553408.0,
+ "27": 425553408.0,
+ "28": 425553408.0,
+ "29": 425553408.0,
+ "30": 425553408.0,
+ "31": 425553408.0,
+ "32": 425553408.0,
+ "33": 425553408.0,
+ "34": 425553408.0,
+ "35": 425553408.0,
+ "36": 425553408.0,
+ "37": 425553408.0,
+ "38": 425553408.0,
+ "39": 425553408.0,
+ "40": 425553408.0,
+ "41": 425553408.0,
+ "42": 425553408.0,
+ "43": 425553408.0,
+ "44": 425553408.0,
+ "45": 425553408.0,
+ "46": 425553408.0,
+ "47": 425553408.0,
+ "48": 425553408.0,
+ "49": 425553408.0,
+ "50": 425553408.0,
+ "51": 425553408.0,
+ "52": 425553408.0,
+ "53": 425553408.0,
+ "54": 425553408.0,
+ "55": 425553408.0,
+ "56": 425553408.0,
+ "57": 425553408.0,
+ "58": 425553408.0,
+ "59": 425553408.0,
+ "60": 425553408.0,
+ "61": 425553408.0,
+ "62": 425553408.0,
+ "63": 425553408.0,
+ "64": 425553408.0,
+ "65": 425553408.0,
+ "66": 425553408.0,
+ "67": 425553408.0,
+ "68": 425553408.0,
+ "69": 425553408.0,
+ "70": 425553408.0,
+ "71": 425553408.0,
+ "72": 425553408.0,
+ "73": 425553408.0,
+ "74": 425553408.0,
+ "75": 425553408.0,
+ "76": 425553408.0,
+ "77": 425553408.0,
+ "78": 425553408.0,
+ "79": 425553408.0,
+ "80": 425553408.0,
+ "81": 425553408.0,
+ "82": 425553408.0,
+ "83": 425553408.0,
+ "84": 425553408.0,
+ "85": 425553408.0,
+ "86": 425553408.0,
+ "87": 425553408.0,
+ "88": 425553408.0,
+ "89": 425553408.0,
+ "90": 425553408.0,
+ "91": 425553408.0,
+ "92": 425553408.0,
+ "93": 425553408.0,
+ "94": 425553408.0,
+ "95": 425553408.0,
+ "96": 425553408.0,
+ "97": 425553408.0,
+ "98": 425553408.0,
+ "99": 425553408.0,
+ "100": 425553408.0,
+ "101": 425553408.0,
+ "102": 425553408.0,
+ "103": 425553408.0,
+ "104": 425553408.0,
+ "105": 425553408.0,
+ "106": 425553408.0,
+ "107": 425553408.0,
+ "108": 425553408.0,
+ "109": 425553408.0,
+ "110": 425553408.0,
+ "111": 425553408.0,
+ "112": 425553408.0,
+ "113": 425553408.0,
+ "114": 425553408.0,
+ "115": 425553408.0,
+ "116": 425553408.0,
+ "117": 425553408.0,
+ "118": 425553408.0,
+ "119": 425553408.0,
+ "120": 425553408.0,
+ "121": 425553408.0,
+ "122": 425553408.0,
+ "123": 425553408.0,
+ "124": 425553408.0,
+ "125": 425553408.0,
+ "126": 425553408.0,
+ "127": 425553408.0,
+ "128": 425553408.0,
+ "129": 425553408.0,
+ "130": 425553408.0,
+ "131": 425553408.0,
+ "132": 425553408.0,
+ "133": 425553408.0,
+ "134": 425553408.0,
+ "135": 425553408.0,
+ "136": 425553408.0,
+ "137": 425553408.0,
+ "138": 425553408.0,
+ "139": 425553408.0,
+ "140": 425553408.0,
+ "141": 425553408.0,
+ "142": 425553408.0,
+ "143": 425553408.0,
+ "144": 425553408.0,
+ "145": 425553408.0,
+ "146": 425553408.0,
+ "147": 425553408.0,
+ "148": 425553408.0,
+ "149": 425553408.0,
+ "150": 425553408.0,
+ "151": 425553408.0,
+ "152": 425553408.0,
+ "153": 425553408.0,
+ "154": 425553408.0,
+ "155": 425553408.0,
+ "156": 425553408.0,
+ "157": 425553408.0,
+ "158": 425553408.0,
+ "159": 425553408.0,
+ "160": 425553408.0,
+ "161": 425553408.0,
+ "162": 425553408.0,
+ "163": 425553408.0,
+ "164": 425553408.0,
+ "165": 425553408.0,
+ "166": 425553408.0,
+ "167": 425553408.0,
+ "168": 425553408.0,
+ "169": 425553408.0,
+ "170": 425553408.0,
+ "171": 425553408.0,
+ "172": 425553408.0,
+ "173": 425553408.0,
+ "174": 425553408.0,
+ "175": 425553408.0,
+ "176": 425553408.0,
+ "177": 425553408.0,
+ "178": 425553408.0,
+ "179": 425553408.0,
+ "180": 425553408.0,
+ "181": 425553408.0,
+ "182": 425553408.0,
+ "183": 425553408.0,
+ "184": 425553408.0,
+ "185": 425553408.0,
+ "186": 425553408.0,
+ "187": 425553408.0,
+ "188": 425553408.0,
+ "189": 425553408.0,
+ "190": 425553408.0,
+ "191": 425553408.0,
+ "192": 425553408.0,
+ "193": 425553408.0,
+ "194": 425553408.0,
+ "195": 425553408.0,
+ "196": 425553408.0,
+ "197": 425553408.0,
+ "198": 425553408.0,
+ "199": 425553408.0,
+ "200": 425553408.0,
+ "201": 425553408.0,
+ "202": 425553408.0,
+ "203": 425553408.0,
+ "204": 425553408.0,
+ "205": 425553408.0,
+ "206": 425553408.0,
+ "207": 425553408.0,
+ "208": 425553408.0,
+ "209": 425553408.0,
+ "210": 425553408.0,
+ "211": 425553408.0,
+ "212": 425553408.0,
+ "213": 425553408.0,
+ "214": 425553408.0,
+ "215": 425553408.0,
+ "216": 425553408.0,
+ "217": 425553408.0,
+ "218": 425553408.0,
+ "219": 425553408.0,
+ "220": 425553408.0,
+ "221": 425553408.0,
+ "222": 425553408.0,
+ "223": 425553408.0,
+ "224": 425553408.0,
+ "225": 425553408.0,
+ "226": 425553408.0,
+ "227": 425553408.0,
+ "228": 425553408.0,
+ "229": 425553408.0,
+ "230": 425553408.0,
+ "231": 425553408.0,
+ "232": 425553408.0,
+ "233": 425553408.0,
+ "234": 425553408.0,
+ "235": 425553408.0,
+ "236": 425553408.0,
+ "237": 425553408.0,
+ "238": 425553408.0,
+ "239": 425553408.0,
+ "240": 425553408.0,
+ "241": 425553408.0,
+ "242": 425553408.0,
+ "243": 425553408.0,
+ "244": 425553408.0,
+ "245": 425553408.0,
+ "246": 425553408.0,
+ "247": 425553408.0,
+ "248": 425553408.0,
+ "249": 425553408.0,
+ "250": 425553408.0,
+ "251": 425553408.0,
+ "252": 425553408.0,
+ "253": 425553408.0,
+ "254": 425553408.0,
+ "255": 425553408.0,
+ "256": 425553408.0,
+ "257": 425553408.0,
+ "258": 425553408.0,
+ "259": 425553408.0,
+ "260": 425553408.0,
+ "261": 425553408.0,
+ "262": 425553408.0,
+ "263": 425553408.0,
+ "264": 425553408.0,
+ "265": 425553408.0,
+ "266": 425553408.0,
+ "267": 425553408.0,
+ "268": 425553408.0,
+ "269": 425553408.0,
+ "270": 425553408.0,
+ "271": 425553408.0,
+ "272": 425553408.0,
+ "273": 425553408.0,
+ "274": 425553408.0,
+ "275": 425553408.0,
+ "276": 425553408.0,
+ "277": 425553408.0,
+ "278": 425553408.0,
+ "279": 425553408.0,
+ "280": 425553408.0,
+ "281": 425553408.0,
+ "282": 425553408.0,
+ "283": 425553408.0,
+ "284": 425553408.0,
+ "285": 425553408.0,
+ "286": 425553408.0,
+ "287": 425553408.0,
+ "288": 425553408.0,
+ "289": 425553408.0,
+ "290": 425553408.0,
+ "291": 425553408.0,
+ "292": 425553408.0,
+ "293": 425553408.0,
+ "294": 425553408.0,
+ "295": 425553408.0,
+ "296": 425553408.0,
+ "297": 425553408.0,
+ "298": 425553408.0,
+ "299": 425553408.0,
+ "300": 425553408.0,
+ "301": 425553408.0,
+ "302": 425553408.0,
+ "303": 425553408.0,
+ "304": 425553408.0,
+ "305": 425553408.0,
+ "306": 425553408.0,
+ "307": 425553408.0,
+ "308": 425553408.0,
+ "309": 425553408.0,
+ "310": 425553408.0,
+ "311": 425553408.0,
+ "312": 425553408.0,
+ "313": 425553408.0,
+ "314": 425553408.0,
+ "315": 425553408.0,
+ "316": 425553408.0,
+ "317": 425553408.0,
+ "318": 425553408.0,
+ "319": 425553408.0,
+ "320": 425553408.0,
+ "321": 425553408.0,
+ "322": 425553408.0,
+ "323": 425553408.0,
+ "324": 425553408.0,
+ "325": 425553408.0,
+ "326": 425553408.0,
+ "327": 425553408.0,
+ "328": 425553408.0,
+ "329": 425553408.0,
+ "330": 425553408.0,
+ "331": 425553408.0,
+ "332": 425553408.0,
+ "333": 425553408.0,
+ "334": 425553408.0,
+ "335": 425553408.0,
+ "336": 425553408.0,
+ "337": 425553408.0,
+ "338": 425553408.0,
+ "339": 425553408.0,
+ "340": 425553408.0,
+ "341": 425553408.0,
+ "342": 425553408.0,
+ "343": 425553408.0,
+ "344": 425553408.0,
+ "345": 425553408.0,
+ "346": 425553408.0,
+ "347": 425553408.0,
+ "348": 425553408.0,
+ "349": 425553408.0,
+ "350": 425553408.0,
+ "351": 425553408.0,
+ "352": 425553408.0,
+ "353": 425553408.0,
+ "354": 425553408.0,
+ "355": 425553408.0,
+ "356": 425553408.0,
+ "357": 425553408.0,
+ "358": 425553408.0,
+ "359": 425553408.0,
+ "360": 425553408.0,
+ "361": 425553408.0,
+ "362": 425553408.0,
+ "363": 425553408.0,
+ "364": 425553408.0,
+ "365": 425553408.0,
+ "366": 425553408.0,
+ "367": 425553408.0,
+ "368": 425553408.0,
+ "369": 425553408.0,
+ "370": 425553408.0,
+ "371": 425553408.0,
+ "372": 425553408.0,
+ "373": 425553408.0,
+ "374": 425553408.0,
+ "375": 425553408.0,
+ "376": 425553408.0,
+ "377": 425553408.0,
+ "378": 425553408.0,
+ "379": 425553408.0,
+ "380": 425553408.0,
+ "381": 425553408.0,
+ "382": 425553408.0,
+ "383": 425553408.0,
+ "384": 425553408.0,
+ "385": 425553408.0,
+ "386": 425553408.0,
+ "387": 425553408.0,
+ "388": 425553408.0,
+ "389": 425553408.0,
+ "390": 425553408.0,
+ "391": 425553408.0,
+ "392": 425553408.0,
+ "393": 425553408.0,
+ "394": 425553408.0,
+ "395": 425553408.0,
+ "396": 425553408.0,
+ "397": 425553408.0,
+ "398": 425553408.0,
+ "399": 425553408.0,
+ "400": 425553408.0,
+ "401": 425553408.0,
+ "402": 425553408.0,
+ "403": 425553408.0,
+ "404": 425553408.0,
+ "405": 425553408.0,
+ "406": 425553408.0,
+ "407": 425553408.0,
+ "408": 425553408.0,
+ "409": 425553408.0,
+ "410": 425553408.0,
+ "411": 425553408.0,
+ "412": 425553408.0,
+ "413": 425553408.0,
+ "414": 425553408.0,
+ "415": 425553408.0,
+ "416": 425553408.0,
+ "417": 425553408.0,
+ "418": 425553408.0,
+ "419": 425553408.0,
+ "420": 425553408.0,
+ "421": 425553408.0,
+ "422": 425553408.0,
+ "423": 425553408.0,
+ "424": 425553408.0,
+ "425": 425553408.0,
+ "426": 425553408.0,
+ "427": 425553408.0,
+ "428": 425553408.0,
+ "429": 425553408.0,
+ "430": 425553408.0,
+ "431": 425553408.0,
+ "432": 425553408.0,
+ "433": 425553408.0,
+ "434": 425553408.0,
+ "435": 425553408.0,
+ "436": 425553408.0,
+ "437": 425553408.0,
+ "438": 425553408.0,
+ "439": 425553408.0,
+ "440": 425553408.0,
+ "441": 425553408.0,
+ "442": 425553408.0,
+ "443": 425553408.0,
+ "444": 425553408.0,
+ "445": 425553408.0,
+ "446": 425553408.0,
+ "447": 425553408.0,
+ "448": 425553408.0,
+ "449": 425553408.0,
+ "450": 425553408.0,
+ "451": 425553408.0,
+ "452": 425553408.0,
+ "453": 425553408.0,
+ "454": 425553408.0,
+ "455": 425553408.0,
+ "456": 425553408.0,
+ "457": 425553408.0,
+ "458": 425553408.0,
+ "459": 425553408.0,
+ "460": 425553408.0,
+ "461": 425553408.0,
+ "462": 425553408.0,
+ "463": 425553408.0,
+ "464": 425553408.0,
+ "465": 425553408.0,
+ "466": 425553408.0,
+ "467": 425553408.0,
+ "468": 425553408.0,
+ "469": 425553408.0,
+ "470": 425553408.0,
+ "471": 425553408.0,
+ "472": 425553408.0,
+ "473": 425553408.0,
+ "474": 425553408.0,
+ "475": 425553408.0,
+ "476": 425553408.0,
+ "477": 425553408.0,
+ "478": 425553408.0,
+ "479": 425553408.0,
+ "480": 425553408.0,
+ "481": 425553408.0,
+ "482": 425553408.0,
+ "483": 425553408.0,
+ "484": 425553408.0,
+ "485": 425553408.0,
+ "486": 425553408.0,
+ "487": 425553408.0,
+ "488": 425553408.0,
+ "489": 425553408.0,
+ "490": 425553408.0,
+ "491": 425553408.0,
+ "492": 425553408.0,
+ "493": 425553408.0,
+ "494": 425553408.0,
+ "495": 425553408.0,
+ "496": 425553408.0,
+ "497": 425553408.0,
+ "498": 425553408.0,
+ "499": 425553408.0,
+ "500": 425553408.0,
+ "501": 425553408.0,
+ "502": 425553408.0,
+ "503": 425553408.0,
+ "504": 425553408.0,
+ "505": 425553408.0,
+ "506": 425553408.0,
+ "507": 425553408.0,
+ "508": 425553408.0,
+ "509": 425553408.0,
+ "510": 425553408.0,
+ "511": 425553408.0,
+ "512": 425553408.0,
+ "513": 425553408.0,
+ "514": 425553408.0,
+ "515": 425553408.0,
+ "516": 425553408.0,
+ "517": 425553408.0,
+ "518": 425553408.0,
+ "519": 425553408.0,
+ "520": 425553408.0,
+ "521": 425553408.0,
+ "522": 425553408.0,
+ "523": 425553408.0,
+ "524": 425553408.0,
+ "525": 425553408.0,
+ "526": 425553408.0,
+ "527": 425553408.0,
+ "528": 425553408.0,
+ "529": 425553408.0,
+ "530": 425553408.0,
+ "531": 425553408.0,
+ "532": 425553408.0,
+ "533": 425553408.0,
+ "534": 425553408.0,
+ "535": 425553408.0,
+ "536": 425553408.0,
+ "537": 425553408.0,
+ "538": 425553408.0,
+ "539": 425553408.0,
+ "540": 425553408.0,
+ "541": 425553408.0,
+ "542": 425553408.0,
+ "543": 425553408.0,
+ "544": 425553408.0,
+ "545": 425553408.0,
+ "546": 425553408.0,
+ "547": 425553408.0,
+ "548": 425553408.0,
+ "549": 425553408.0,
+ "550": 425553408.0,
+ "551": 425553408.0,
+ "552": 425553408.0,
+ "553": 425553408.0,
+ "554": 425553408.0,
+ "555": 425553408.0,
+ "556": 425553408.0,
+ "557": 425553408.0,
+ "558": 425553408.0,
+ "559": 425553408.0,
+ "560": 425553408.0,
+ "561": 425553408.0,
+ "562": 425553408.0,
+ "563": 425553408.0,
+ "564": 425553408.0,
+ "565": 425553408.0,
+ "566": 425553408.0,
+ "567": 425553408.0,
+ "568": 425553408.0,
+ "569": 425553408.0,
+ "570": 425553408.0,
+ "571": 425553408.0,
+ "572": 425553408.0,
+ "573": 425553408.0,
+ "574": 425553408.0,
+ "575": 425553408.0,
+ "576": 425553408.0,
+ "577": 425553408.0,
+ "578": 425553408.0,
+ "579": 425553408.0,
+ "580": 425553408.0,
+ "581": 425553408.0,
+ "582": 425553408.0,
+ "583": 425553408.0,
+ "584": 425553408.0,
+ "585": 425553408.0,
+ "586": 425553408.0,
+ "587": 425553408.0,
+ "588": 425553408.0,
+ "589": 425553408.0,
+ "590": 425553408.0,
+ "591": 425553408.0,
+ "592": 425553408.0,
+ "593": 425553408.0,
+ "594": 425553408.0,
+ "595": 425553408.0,
+ "596": 425553408.0,
+ "597": 425553408.0,
+ "598": 425553408.0,
+ "599": 425553408.0,
+ "600": 425553408.0,
+ "601": 425553408.0,
+ "602": 425553408.0,
+ "603": 425553408.0,
+ "604": 425553408.0,
+ "605": 425553408.0,
+ "606": 425553408.0,
+ "607": 425553408.0,
+ "608": 425553408.0,
+ "609": 425553408.0,
+ "610": 425553408.0,
+ "611": 425553408.0,
+ "612": 425553408.0,
+ "613": 425553408.0,
+ "614": 425553408.0,
+ "615": 425553408.0,
+ "616": 425553408.0,
+ "617": 425553408.0,
+ "618": 425553408.0,
+ "619": 425553408.0,
+ "620": 425553408.0,
+ "621": 425553408.0,
+ "622": 425553408.0,
+ "623": 425553408.0,
+ "624": 425553408.0,
+ "625": 425553408.0,
+ "626": 425553408.0,
+ "627": 425553408.0,
+ "628": 425553408.0,
+ "629": 425553408.0,
+ "630": 425553408.0,
+ "631": 425553408.0,
+ "632": 425553408.0,
+ "633": 425553408.0,
+ "634": 425553408.0,
+ "635": 425553408.0,
+ "636": 425553408.0,
+ "637": 425553408.0,
+ "638": 425553408.0,
+ "639": 425553408.0,
+ "640": 425553408.0,
+ "641": 425553408.0,
+ "642": 425553408.0,
+ "643": 425553408.0,
+ "644": 425553408.0,
+ "645": 425553408.0,
+ "646": 425553408.0,
+ "647": 425553408.0,
+ "648": 425553408.0,
+ "649": 425553408.0,
+ "650": 425553408.0,
+ "651": 425553408.0,
+ "652": 425553408.0,
+ "653": 425553408.0,
+ "654": 425553408.0,
+ "655": 425553408.0,
+ "656": 425553408.0,
+ "657": 425553408.0,
+ "658": 425553408.0,
+ "659": 425553408.0,
+ "660": 425553408.0,
+ "661": 425553408.0,
+ "662": 425553408.0,
+ "663": 425553408.0,
+ "664": 425553408.0,
+ "665": 425553408.0,
+ "666": 425553408.0,
+ "667": 425553408.0,
+ "668": 425553408.0,
+ "669": 425553408.0,
+ "670": 425553408.0,
+ "671": 425553408.0,
+ "672": 425553408.0,
+ "673": 425553408.0,
+ "674": 425553408.0,
+ "675": 425553408.0,
+ "676": 425553408.0,
+ "677": 425553408.0,
+ "678": 425553408.0,
+ "679": 425553408.0,
+ "680": 425553408.0,
+ "681": 425553408.0,
+ "682": 425553408.0,
+ "683": 425553408.0,
+ "684": 425553408.0,
+ "685": 425553408.0,
+ "686": 425553408.0,
+ "687": 425553408.0,
+ "688": 425553408.0,
+ "689": 425553408.0,
+ "690": 425553408.0,
+ "691": 425553408.0,
+ "692": 425553408.0,
+ "693": 425553408.0,
+ "694": 425553408.0,
+ "695": 425553408.0,
+ "696": 425553408.0,
+ "697": 425553408.0,
+ "698": 425553408.0,
+ "699": 425553408.0,
+ "700": 425553408.0,
+ "701": 425553408.0,
+ "702": 425553408.0,
+ "703": 425553408.0,
+ "704": 425553408.0,
+ "705": 425553408.0,
+ "706": 425553408.0,
+ "707": 425553408.0,
+ "708": 425553408.0,
+ "709": 425553408.0,
+ "710": 425553408.0,
+ "711": 425553408.0,
+ "712": 425553408.0,
+ "713": 425553408.0,
+ "714": 425553408.0,
+ "715": 425553408.0,
+ "716": 425553408.0,
+ "717": 425553408.0,
+ "718": 425553408.0,
+ "719": 425553408.0,
+ "720": 425553408.0,
+ "721": 425553408.0,
+ "722": 425553408.0,
+ "723": 425553408.0,
+ "724": 425553408.0,
+ "725": 425553408.0,
+ "726": 425553408.0,
+ "727": 425553408.0,
+ "728": 425553408.0,
+ "729": 425553408.0,
+ "730": 425553408.0,
+ "731": 425553408.0,
+ "732": 425553408.0,
+ "733": 425553408.0,
+ "734": 425553408.0,
+ "735": 425553408.0,
+ "736": 425553408.0,
+ "737": 425553408.0,
+ "738": 425553408.0,
+ "739": 425553408.0,
+ "740": 425553408.0,
+ "741": 425553408.0,
+ "742": 425553408.0,
+ "743": 425553408.0,
+ "744": 425553408.0,
+ "745": 425553408.0,
+ "746": 425553408.0,
+ "747": 425553408.0,
+ "748": 425553408.0,
+ "749": 425553408.0,
+ "750": 425553408.0,
+ "751": 425553408.0,
+ "752": 425553408.0,
+ "753": 425553408.0,
+ "754": 425553408.0,
+ "755": 425553408.0,
+ "756": 425553408.0,
+ "757": 425553408.0,
+ "758": 425553408.0,
+ "759": 425553408.0,
+ "760": 425553408.0,
+ "761": 425553408.0,
+ "762": 425553408.0,
+ "763": 425553408.0,
+ "764": 425553408.0,
+ "765": 425553408.0,
+ "766": 425553408.0,
+ "767": 425553408.0,
+ "768": 425553408.0,
+ "769": 425553408.0,
+ "770": 425553408.0,
+ "771": 425553408.0,
+ "772": 425553408.0,
+ "773": 425553408.0,
+ "774": 425553408.0,
+ "775": 425553408.0,
+ "776": 425553408.0,
+ "777": 425553408.0,
+ "778": 425553408.0,
+ "779": 425553408.0,
+ "780": 425553408.0,
+ "781": 425553408.0,
+ "782": 425553408.0,
+ "783": 425553408.0,
+ "784": 425553408.0,
+ "785": 425553408.0,
+ "786": 425553408.0,
+ "787": 425553408.0,
+ "788": 425553408.0,
+ "789": 425553408.0,
+ "790": 425553408.0,
+ "791": 425553408.0,
+ "792": 425553408.0,
+ "793": 425553408.0,
+ "794": 425553408.0,
+ "795": 425553408.0,
+ "796": 425553408.0,
+ "797": 425553408.0,
+ "798": 425553408.0,
+ "799": 425553408.0,
+ "800": 425553408.0,
+ "801": 425553408.0,
+ "802": 425553408.0,
+ "803": 425553408.0,
+ "804": 425553408.0,
+ "805": 425553408.0,
+ "806": 425553408.0,
+ "807": 425553408.0,
+ "808": 425553408.0,
+ "809": 425553408.0,
+ "810": 425553408.0,
+ "811": 425553408.0,
+ "812": 425553408.0,
+ "813": 425553408.0,
+ "814": 425553408.0,
+ "815": 425553408.0,
+ "816": 425553408.0,
+ "817": 425553408.0,
+ "818": 425553408.0,
+ "819": 425553408.0,
+ "820": 425553408.0,
+ "821": 425553408.0,
+ "822": 425553408.0,
+ "823": 425553408.0,
+ "824": 425553408.0,
+ "825": 425553408.0,
+ "826": 425553408.0,
+ "827": 425553408.0,
+ "828": 425553408.0,
+ "829": 425553408.0,
+ "830": 425553408.0,
+ "831": 425553408.0,
+ "832": 425553408.0,
+ "833": 425553408.0,
+ "834": 425553408.0,
+ "835": 425553408.0,
+ "836": 425553408.0,
+ "837": 425553408.0,
+ "838": 425553408.0,
+ "839": 425553408.0,
+ "840": 425553408.0,
+ "841": 425553408.0,
+ "842": 425553408.0,
+ "843": 425553408.0,
+ "844": 425553408.0,
+ "845": 425553408.0,
+ "846": 425553408.0,
+ "847": 425553408.0,
+ "848": 425553408.0,
+ "849": 425553408.0,
+ "850": 425553408.0,
+ "851": 425553408.0,
+ "852": 425553408.0,
+ "853": 425553408.0,
+ "854": 425553408.0,
+ "855": 425553408.0,
+ "856": 425553408.0,
+ "857": 425553408.0,
+ "858": 425553408.0,
+ "859": 425553408.0,
+ "860": 425553408.0,
+ "861": 425553408.0,
+ "862": 425553408.0,
+ "863": 425553408.0,
+ "864": 425553408.0,
+ "865": 425553408.0,
+ "866": 425553408.0,
+ "867": 425553408.0,
+ "868": 425553408.0,
+ "869": 425553408.0,
+ "870": 425553408.0,
+ "871": 425553408.0,
+ "872": 425553408.0,
+ "873": 425553408.0,
+ "874": 425553408.0,
+ "875": 425553408.0,
+ "876": 425553408.0,
+ "877": 425553408.0,
+ "878": 425553408.0,
+ "879": 425553408.0,
+ "880": 425553408.0,
+ "881": 425553408.0,
+ "882": 425553408.0,
+ "883": 425553408.0,
+ "884": 425553408.0,
+ "885": 425553408.0,
+ "886": 425553408.0,
+ "887": 425553408.0,
+ "888": 425553408.0,
+ "889": 425553408.0,
+ "890": 425553408.0,
+ "891": 425553408.0,
+ "892": 425553408.0,
+ "893": 425553408.0,
+ "894": 425553408.0,
+ "895": 425553408.0,
+ "896": 425553408.0,
+ "897": 425553408.0,
+ "898": 425553408.0,
+ "899": 425553408.0,
+ "900": 425553408.0,
+ "901": 425553408.0,
+ "902": 425553408.0,
+ "903": 425553408.0,
+ "904": 425553408.0,
+ "905": 425553408.0,
+ "906": 425553408.0,
+ "907": 425553408.0,
+ "908": 425553408.0,
+ "909": 425553408.0,
+ "910": 425553408.0,
+ "911": 425553408.0,
+ "912": 425553408.0,
+ "913": 425553408.0,
+ "914": 425553408.0,
+ "915": 425553408.0,
+ "916": 425553408.0,
+ "917": 425553408.0,
+ "918": 425553408.0,
+ "919": 425553408.0,
+ "920": 425553408.0,
+ "921": 425553408.0,
+ "922": 425553408.0,
+ "923": 425553408.0,
+ "924": 425553408.0,
+ "925": 425553408.0,
+ "926": 425553408.0,
+ "927": 425553408.0,
+ "928": 425553408.0,
+ "929": 425553408.0,
+ "930": 425553408.0,
+ "931": 425553408.0,
+ "932": 425553408.0,
+ "933": 425553408.0,
+ "934": 425553408.0,
+ "935": 425553408.0,
+ "936": 425553408.0,
+ "937": 425553408.0,
+ "938": 425553408.0,
+ "939": 425553408.0,
+ "940": 425553408.0,
+ "941": 425553408.0,
+ "942": 425553408.0,
+ "943": 425553408.0,
+ "944": 425553408.0,
+ "945": 425553408.0,
+ "946": 425553408.0,
+ "947": 425553408.0,
+ "948": 425553408.0,
+ "949": 425553408.0,
+ "950": 425553408.0,
+ "951": 425553408.0,
+ "952": 425553408.0,
+ "953": 425553408.0,
+ "954": 425553408.0,
+ "955": 425553408.0,
+ "956": 425553408.0,
+ "957": 425553408.0,
+ "958": 425553408.0,
+ "959": 425553408.0,
+ "960": 425553408.0,
+ "961": 425553408.0,
+ "962": 425553408.0,
+ "963": 425553408.0,
+ "964": 425553408.0,
+ "965": 425553408.0,
+ "966": 425553408.0,
+ "967": 425553408.0,
+ "968": 425553408.0,
+ "969": 425553408.0,
+ "970": 425553408.0,
+ "971": 425553408.0,
+ "972": 425553408.0,
+ "973": 425553408.0,
+ "974": 425553408.0,
+ "975": 425553408.0,
+ "976": 425553408.0,
+ "977": 425553408.0,
+ "978": 425553408.0,
+ "979": 425553408.0,
+ "980": 425553408.0,
+ "981": 425553408.0,
+ "982": 425553408.0,
+ "983": 425553408.0,
+ "984": 425553408.0,
+ "985": 425553408.0,
+ "986": 425553408.0,
+ "987": 425553408.0,
+ "988": 425553408.0,
+ "989": 425553408.0,
+ "990": 425553408.0,
+ "991": 425553408.0,
+ "992": 425553408.0,
+ "993": 425553408.0,
+ "994": 425553408.0,
+ "995": 425553408.0,
+ "996": 425553408.0,
+ "997": 425553408.0,
+ "998": 425553408.0,
+ "999": 425553408.0,
+ "1000": 425553408.0,
+ "1001": 425553408.0,
+ "1002": 425553408.0,
+ "1003": 425553408.0,
+ "1004": 425553408.0,
+ "1005": 425553408.0,
+ "1006": 425553408.0,
+ "1007": 425553408.0,
+ "1008": 425553408.0,
+ "1009": 425553408.0,
+ "1010": 425553408.0,
+ "1011": 425553408.0,
+ "1012": 425553408.0,
+ "1013": 425553408.0,
+ "1014": 425553408.0,
+ "1015": 425553408.0,
+ "1016": 425553408.0,
+ "1017": 425553408.0,
+ "1018": 425553408.0,
+ "1019": 425553408.0,
+ "1020": 425553408.0,
+ "1021": 425553408.0,
+ "1022": 425553408.0,
+ "1023": 425553408.0,
+ "1024": 425553408.0,
+ "1025": 425553408.0,
+ "1026": 425553408.0,
+ "1027": 425553408.0,
+ "1028": 425553408.0,
+ "1029": 425553408.0,
+ "1030": 425553408.0,
+ "1031": 425553408.0,
+ "1032": 425553408.0,
+ "1033": 425553408.0,
+ "1034": 425553408.0,
+ "1035": 425553408.0,
+ "1036": 425553408.0,
+ "1037": 425553408.0,
+ "1038": 425553408.0,
+ "1039": 425553408.0,
+ "1040": 425553408.0,
+ "1041": 425553408.0,
+ "1042": 425553408.0,
+ "1043": 425553408.0,
+ "1044": 425553408.0,
+ "1045": 425553408.0,
+ "1046": 425553408.0,
+ "1047": 425553408.0,
+ "1048": 425553408.0,
+ "1049": 425553408.0,
+ "1050": 425553408.0,
+ "1051": 425553408.0,
+ "1052": 425553408.0,
+ "1053": 425553408.0,
+ "1054": 425553408.0,
+ "1055": 425553408.0,
+ "1056": 425553408.0,
+ "1057": 425553408.0,
+ "1058": 425553408.0,
+ "1059": 425553408.0,
+ "1060": 425553408.0,
+ "1061": 425553408.0,
+ "1062": 425553408.0,
+ "1063": 425553408.0,
+ "1064": 425553408.0,
+ "1065": 425553408.0,
+ "1066": 425553408.0,
+ "1067": 425553408.0,
+ "1068": 425553408.0,
+ "1069": 425553408.0,
+ "1070": 425553408.0,
+ "1071": 425553408.0,
+ "1072": 425553408.0,
+ "1073": 425553408.0,
+ "1074": 425553408.0,
+ "1075": 425553408.0,
+ "1076": 425553408.0,
+ "1077": 425553408.0,
+ "1078": 425553408.0,
+ "1079": 425553408.0,
+ "1080": 425553408.0,
+ "1081": 425553408.0,
+ "1082": 425553408.0,
+ "1083": 425553408.0,
+ "1084": 425553408.0,
+ "1085": 425553408.0,
+ "1086": 425553408.0,
+ "1087": 425553408.0,
+ "1088": 425553408.0,
+ "1089": 425553408.0,
+ "1090": 425553408.0,
+ "1091": 425553408.0,
+ "1092": 425553408.0,
+ "1093": 425553408.0,
+ "1094": 425553408.0,
+ "1095": 425553408.0,
+ "1096": 425553408.0,
+ "1097": 425553408.0,
+ "1098": 425553408.0,
+ "1099": 425553408.0,
+ "1100": 425553408.0,
+ "1101": 425553408.0,
+ "1102": 425553408.0,
+ "1103": 425553408.0,
+ "1104": 425553408.0,
+ "1105": 425553408.0,
+ "1106": 425553408.0,
+ "1107": 425553408.0,
+ "1108": 425553408.0,
+ "1109": 425553408.0,
+ "1110": 425553408.0,
+ "1111": 425553408.0,
+ "1112": 425553408.0,
+ "1113": 425553408.0,
+ "1114": 425553408.0,
+ "1115": 425553408.0,
+ "1116": 425553408.0,
+ "1117": 425553408.0,
+ "1118": 425553408.0,
+ "1119": 425553408.0,
+ "1120": 425553408.0,
+ "1121": 425553408.0,
+ "1122": 425553408.0,
+ "1123": 425553408.0,
+ "1124": 425553408.0,
+ "1125": 425553408.0,
+ "1126": 425553408.0,
+ "1127": 425553408.0,
+ "1128": 425553408.0,
+ "1129": 425553408.0,
+ "1130": 425553408.0,
+ "1131": 425553408.0,
+ "1132": 425553408.0,
+ "1133": 425553408.0,
+ "1134": 425553408.0,
+ "1135": 425553408.0,
+ "1136": 425553408.0,
+ "1137": 425553408.0,
+ "1138": 425553408.0,
+ "1139": 425553408.0,
+ "1140": 425553408.0,
+ "1141": 425553408.0,
+ "1142": 425553408.0,
+ "1143": 425553408.0,
+ "1144": 425553408.0,
+ "1145": 425553408.0,
+ "1146": 425553408.0,
+ "1147": 425553408.0,
+ "1148": 425553408.0,
+ "1149": 425553408.0,
+ "1150": 425553408.0,
+ "1151": 425553408.0,
+ "1152": 425553408.0,
+ "1153": 425553408.0,
+ "1154": 425553408.0,
+ "1155": 425553408.0,
+ "1156": 425553408.0,
+ "1157": 425553408.0,
+ "1158": 425553408.0,
+ "1159": 425553408.0,
+ "1160": 425553408.0,
+ "1161": 425553408.0,
+ "1162": 425553408.0,
+ "1163": 425553408.0,
+ "1164": 425553408.0,
+ "1165": 425553408.0,
+ "1166": 425553408.0,
+ "1167": 425553408.0,
+ "1168": 425553408.0,
+ "1169": 425553408.0,
+ "1170": 425553408.0,
+ "1171": 425553408.0,
+ "1172": 425553408.0,
+ "1173": 425553408.0,
+ "1174": 425553408.0,
+ "1175": 425553408.0,
+ "1176": 425553408.0,
+ "1177": 425553408.0,
+ "1178": 425553408.0,
+ "1179": 425553408.0,
+ "1180": 425553408.0,
+ "1181": 425553408.0,
+ "1182": 425553408.0,
+ "1183": 425553408.0,
+ "1184": 425553408.0,
+ "1185": 425553408.0,
+ "1186": 425553408.0,
+ "1187": 425553408.0,
+ "1188": 425553408.0,
+ "1189": 425553408.0,
+ "1190": 425553408.0,
+ "1191": 425553408.0,
+ "1192": 425553408.0,
+ "1193": 425553408.0,
+ "1194": 425553408.0,
+ "1195": 425553408.0,
+ "1196": 425553408.0,
+ "1197": 425553408.0,
+ "1198": 425553408.0,
+ "1199": 425553408.0,
+ "1200": 425553408.0,
+ "1201": 425553408.0,
+ "1202": 425553408.0,
+ "1203": 425553408.0,
+ "1204": 425553408.0,
+ "1205": 425553408.0,
+ "1206": 425553408.0,
+ "1207": 425553408.0,
+ "1208": 425553408.0,
+ "1209": 425553408.0,
+ "1210": 425553408.0,
+ "1211": 425553408.0,
+ "1212": 425553408.0,
+ "1213": 425553408.0,
+ "1214": 425553408.0,
+ "1215": 425553408.0,
+ "1216": 425553408.0,
+ "1217": 425553408.0,
+ "1218": 425553408.0,
+ "1219": 425553408.0,
+ "1220": 425553408.0,
+ "1221": 425553408.0,
+ "1222": 425553408.0,
+ "1223": 425553408.0,
+ "1224": 425553408.0,
+ "1225": 425553408.0,
+ "1226": 425553408.0,
+ "1227": 425553408.0,
+ "1228": 425553408.0,
+ "1229": 425553408.0,
+ "1230": 425553408.0,
+ "1231": 425553408.0,
+ "1232": 425553408.0,
+ "1233": 425553408.0,
+ "1234": 425553408.0,
+ "1235": 425553408.0,
+ "1236": 425553408.0,
+ "1237": 425553408.0,
+ "1238": 425553408.0,
+ "1239": 425553408.0,
+ "1240": 425553408.0,
+ "1241": 425553408.0,
+ "1242": 425553408.0,
+ "1243": 425553408.0,
+ "1244": 425553408.0,
+ "1245": 425553408.0,
+ "1246": 425553408.0,
+ "1247": 425553408.0,
+ "1248": 425553408.0,
+ "1249": 425553408.0,
+ "1250": 425553408.0,
+ "1251": 425553408.0,
+ "1252": 425553408.0,
+ "1253": 425553408.0,
+ "1254": 425553408.0,
+ "1255": 425553408.0,
+ "1256": 425553408.0,
+ "1257": 425553408.0,
+ "1258": 425553408.0,
+ "1259": 425553408.0,
+ "1260": 425553408.0,
+ "1261": 425553408.0,
+ "1262": 425553408.0,
+ "1263": 425553408.0,
+ "1264": 425553408.0,
+ "1265": 425553408.0,
+ "1266": 425553408.0,
+ "1267": 425553408.0,
+ "1268": 425553408.0,
+ "1269": 425553408.0,
+ "1270": 425553408.0,
+ "1271": 425553408.0,
+ "1272": 425553408.0,
+ "1273": 425553408.0,
+ "1274": 425553408.0,
+ "1275": 425553408.0,
+ "1276": 425553408.0,
+ "1277": 425553408.0,
+ "1278": 425553408.0,
+ "1279": 425553408.0,
+ "1280": 425553408.0,
+ "1281": 425553408.0,
+ "1282": 425553408.0,
+ "1283": 425553408.0,
+ "1284": 425553408.0,
+ "1285": 425553408.0,
+ "1286": 425553408.0,
+ "1287": 425553408.0,
+ "1288": 425553408.0,
+ "1289": 425553408.0,
+ "1290": 425553408.0,
+ "1291": 425553408.0,
+ "1292": 425553408.0,
+ "1293": 425553408.0,
+ "1294": 425553408.0,
+ "1295": 425553408.0,
+ "1296": 425553408.0,
+ "1297": 425553408.0,
+ "1298": 425553408.0,
+ "1299": 425553408.0,
+ "1300": 425553408.0,
+ "1301": 425553408.0,
+ "1302": 425553408.0,
+ "1303": 425553408.0,
+ "1304": 425553408.0,
+ "1305": 425553408.0,
+ "1306": 425553408.0,
+ "1307": 425553408.0,
+ "1308": 425553408.0,
+ "1309": 425553408.0,
+ "1310": 425553408.0,
+ "1311": 425553408.0,
+ "1312": 425553408.0,
+ "1313": 425553408.0,
+ "1314": 425553408.0,
+ "1315": 425553408.0,
+ "1316": 425553408.0,
+ "1317": 425553408.0,
+ "1318": 425553408.0,
+ "1319": 425553408.0,
+ "1320": 425553408.0,
+ "1321": 425553408.0,
+ "1322": 425553408.0,
+ "1323": 425553408.0,
+ "1324": 425553408.0,
+ "1325": 425553408.0,
+ "1326": 425553408.0,
+ "1327": 425553408.0,
+ "1328": 425553408.0,
+ "1329": 425553408.0,
+ "1330": 425553408.0,
+ "1331": 425553408.0,
+ "1332": 425553408.0,
+ "1333": 425553408.0,
+ "1334": 425553408.0,
+ "1335": 425553408.0,
+ "1336": 425553408.0,
+ "1337": 425553408.0,
+ "1338": 425553408.0,
+ "1339": 425553408.0,
+ "1340": 425553408.0,
+ "1341": 425553408.0,
+ "1342": 425553408.0,
+ "1343": 425553408.0,
+ "1344": 425553408.0,
+ "1345": 425553408.0,
+ "1346": 425553408.0,
+ "1347": 425553408.0,
+ "1348": 425553408.0,
+ "1349": 425553408.0,
+ "1350": 425553408.0,
+ "1351": 425553408.0,
+ "1352": 425553408.0,
+ "1353": 425553408.0,
+ "1354": 425553408.0,
+ "1355": 425553408.0,
+ "1356": 425553408.0,
+ "1357": 425553408.0,
+ "1358": 425553408.0,
+ "1359": 425553408.0,
+ "1360": 425553408.0,
+ "1361": 425553408.0,
+ "1362": 425553408.0,
+ "1363": 425553408.0,
+ "1364": 425553408.0,
+ "1365": 425553408.0,
+ "1366": 425553408.0,
+ "1367": 425553408.0,
+ "1368": 425553408.0,
+ "1369": 425553408.0,
+ "1370": 425553408.0,
+ "1371": 425553408.0,
+ "1372": 425553408.0,
+ "1373": 425553408.0,
+ "1374": 425553408.0,
+ "1375": 425553408.0,
+ "1376": 425553408.0,
+ "1377": 425553408.0,
+ "1378": 425553408.0,
+ "1379": 425553408.0,
+ "1380": 425553408.0,
+ "1381": 425553408.0,
+ "1382": 425553408.0,
+ "1383": 425553408.0,
+ "1384": 425553408.0,
+ "1385": 425553408.0,
+ "1386": 425553408.0,
+ "1387": 425553408.0,
+ "1388": 425553408.0,
+ "1389": 425553408.0,
+ "1390": 425553408.0,
+ "1391": 425553408.0,
+ "1392": 425553408.0,
+ "1393": 425553408.0,
+ "1394": 425553408.0,
+ "1395": 425553408.0,
+ "1396": 425553408.0,
+ "1397": 425553408.0,
+ "1398": 425553408.0,
+ "1399": 425553408.0,
+ "1400": 425553408.0,
+ "1401": 425553408.0,
+ "1402": 425553408.0,
+ "1403": 425553408.0,
+ "1404": 425553408.0,
+ "1405": 425553408.0,
+ "1406": 425553408.0,
+ "1407": 425553408.0,
+ "1408": 425553408.0,
+ "1409": 425553408.0,
+ "1410": 425553408.0,
+ "1411": 425553408.0,
+ "1412": 425553408.0,
+ "1413": 425553408.0,
+ "1414": 425553408.0,
+ "1415": 425553408.0,
+ "1416": 425553408.0,
+ "1417": 425553408.0,
+ "1418": 425553408.0,
+ "1419": 425553408.0,
+ "1420": 425553408.0,
+ "1421": 425553408.0,
+ "1422": 425553408.0,
+ "1423": 425553408.0,
+ "1424": 425553408.0,
+ "1425": 425553408.0,
+ "1426": 425553408.0,
+ "1427": 425553408.0,
+ "1428": 425553408.0,
+ "1429": 425553408.0,
+ "1430": 425553408.0,
+ "1431": 425553408.0,
+ "1432": 425553408.0,
+ "1433": 425553408.0,
+ "1434": 425553408.0,
+ "1435": 425553408.0,
+ "1436": 425553408.0,
+ "1437": 425553408.0,
+ "1438": 425553408.0,
+ "1439": 425553408.0,
+ "1440": 425553408.0,
+ "1441": 425553408.0,
+ "1442": 425553408.0,
+ "1443": 425553408.0,
+ "1444": 425553408.0,
+ "1445": 425553408.0,
+ "1446": 425553408.0,
+ "1447": 425553408.0,
+ "1448": 425553408.0,
+ "1449": 425553408.0,
+ "1450": 425553408.0,
+ "1451": 425553408.0,
+ "1452": 425553408.0,
+ "1453": 425553408.0,
+ "1454": 425553408.0,
+ "1455": 425553408.0,
+ "1456": 425553408.0,
+ "1457": 425553408.0,
+ "1458": 425553408.0,
+ "1459": 425553408.0,
+ "1460": 425553408.0,
+ "1461": 425553408.0,
+ "1462": 425553408.0,
+ "1463": 425553408.0,
+ "1464": 425553408.0,
+ "1465": 425553408.0,
+ "1466": 425553408.0,
+ "1467": 425553408.0,
+ "1468": 425553408.0,
+ "1469": 425553408.0,
+ "1470": 425553408.0,
+ "1471": 425553408.0,
+ "1472": 425553408.0,
+ "1473": 425553408.0,
+ "1474": 425553408.0,
+ "1475": 425553408.0,
+ "1476": 425553408.0,
+ "1477": 425553408.0,
+ "1478": 425553408.0,
+ "1479": 425553408.0,
+ "1480": 425553408.0,
+ "1481": 425553408.0,
+ "1482": 425553408.0,
+ "1483": 425553408.0,
+ "1484": 425553408.0,
+ "1485": 425553408.0,
+ "1486": 425553408.0,
+ "1487": 425553408.0,
+ "1488": 425553408.0,
+ "1489": 425553408.0,
+ "1490": 425553408.0,
+ "1491": 425553408.0,
+ "1492": 425553408.0,
+ "1493": 425553408.0,
+ "1494": 425553408.0,
+ "1495": 425553408.0,
+ "1496": 425553408.0,
+ "1497": 425553408.0,
+ "1498": 425553408.0,
+ "1499": 425553408.0,
+ "1500": 425553408.0,
+ "1501": 425553408.0,
+ "1502": 425553408.0,
+ "1503": 425553408.0,
+ "1504": 425553408.0,
+ "1505": 425553408.0,
+ "1506": 425553408.0,
+ "1507": 425553408.0,
+ "1508": 425553408.0,
+ "1509": 425553408.0,
+ "1510": 425553408.0,
+ "1511": 425553408.0,
+ "1512": 425553408.0,
+ "1513": 425553408.0,
+ "1514": 425553408.0,
+ "1515": 425553408.0,
+ "1516": 425553408.0,
+ "1517": 425553408.0,
+ "1518": 425553408.0,
+ "1519": 425553408.0,
+ "1520": 425553408.0,
+ "1521": 425553408.0,
+ "1522": 425553408.0,
+ "1523": 425553408.0,
+ "1524": 425553408.0,
+ "1525": 425553408.0,
+ "1526": 425553408.0,
+ "1527": 425553408.0,
+ "1528": 425553408.0,
+ "1529": 425553408.0,
+ "1530": 425553408.0,
+ "1531": 425553408.0,
+ "1532": 425553408.0,
+ "1533": 425553408.0,
+ "1534": 425553408.0,
+ "1535": 425553408.0,
+ "1536": 425553408.0,
+ "1537": 425553408.0,
+ "1538": 425553408.0,
+ "1539": 425553408.0,
+ "1540": 425553408.0,
+ "1541": 425553408.0,
+ "1542": 425553408.0,
+ "1543": 425553408.0,
+ "1544": 425553408.0,
+ "1545": 425553408.0,
+ "1546": 425553408.0,
+ "1547": 425553408.0,
+ "1548": 425553408.0,
+ "1549": 425553408.0,
+ "1550": 425553408.0,
+ "1551": 425553408.0,
+ "1552": 425553408.0,
+ "1553": 425553408.0,
+ "1554": 425553408.0,
+ "1555": 425553408.0,
+ "1556": 425553408.0,
+ "1557": 425553408.0,
+ "1558": 425553408.0,
+ "1559": 425553408.0,
+ "1560": 425553408.0,
+ "1561": 425553408.0,
+ "1562": 425553408.0,
+ "1563": 425553408.0,
+ "1564": 425553408.0,
+ "1565": 425553408.0,
+ "1566": 425553408.0,
+ "1567": 425553408.0,
+ "1568": 425553408.0,
+ "1569": 425553408.0,
+ "1570": 425553408.0,
+ "1571": 425553408.0,
+ "1572": 425553408.0,
+ "1573": 425553408.0,
+ "1574": 425553408.0,
+ "1575": 425553408.0,
+ "1576": 425553408.0,
+ "1577": 425553408.0,
+ "1578": 425553408.0,
+ "1579": 425553408.0,
+ "1580": 425553408.0,
+ "1581": 425553408.0,
+ "1582": 425553408.0,
+ "1583": 425553408.0,
+ "1584": 425553408.0,
+ "1585": 425553408.0,
+ "1586": 425553408.0,
+ "1587": 425553408.0,
+ "1588": 425553408.0,
+ "1589": 425553408.0,
+ "1590": 425553408.0,
+ "1591": 425553408.0,
+ "1592": 425553408.0,
+ "1593": 425553408.0,
+ "1594": 425553408.0,
+ "1595": 425553408.0,
+ "1596": 425553408.0,
+ "1597": 425553408.0,
+ "1598": 425553408.0,
+ "1599": 425553408.0,
+ "1600": 425553408.0,
+ "1601": 425553408.0,
+ "1602": 425553408.0,
+ "1603": 425553408.0,
+ "1604": 425553408.0,
+ "1605": 425553408.0,
+ "1606": 425553408.0,
+ "1607": 425553408.0,
+ "1608": 425553408.0,
+ "1609": 425553408.0,
+ "1610": 425553408.0,
+ "1611": 425553408.0,
+ "1612": 425553408.0,
+ "1613": 425553408.0,
+ "1614": 425553408.0,
+ "1615": 425553408.0,
+ "1616": 425553408.0,
+ "1617": 425553408.0,
+ "1618": 425553408.0,
+ "1619": 425553408.0,
+ "1620": 425553408.0,
+ "1621": 425553408.0,
+ "1622": 425553408.0,
+ "1623": 425553408.0,
+ "1624": 425553408.0,
+ "1625": 425553408.0,
+ "1626": 425553408.0,
+ "1627": 425553408.0,
+ "1628": 425553408.0,
+ "1629": 425553408.0,
+ "1630": 425553408.0,
+ "1631": 425553408.0,
+ "1632": 425553408.0,
+ "1633": 425553408.0,
+ "1634": 425553408.0,
+ "1635": 425553408.0,
+ "1636": 425553408.0,
+ "1637": 425553408.0,
+ "1638": 425553408.0,
+ "1639": 425553408.0,
+ "1640": 425553408.0,
+ "1641": 425553408.0,
+ "1642": 425553408.0,
+ "1643": 425553408.0,
+ "1644": 425553408.0,
+ "1645": 425553408.0,
+ "1646": 425553408.0,
+ "1647": 425553408.0,
+ "1648": 425553408.0,
+ "1649": 425553408.0,
+ "1650": 425553408.0,
+ "1651": 425553408.0,
+ "1652": 425553408.0,
+ "1653": 425553408.0,
+ "1654": 425553408.0,
+ "1655": 425553408.0,
+ "1656": 425553408.0,
+ "1657": 425553408.0,
+ "1658": 425553408.0,
+ "1659": 425553408.0,
+ "1660": 425553408.0,
+ "1661": 425553408.0,
+ "1662": 425553408.0,
+ "1663": 425553408.0,
+ "1664": 425553408.0,
+ "1665": 425553408.0,
+ "1666": 425553408.0,
+ "1667": 425553408.0,
+ "1668": 425553408.0,
+ "1669": 425553408.0,
+ "1670": 425553408.0,
+ "1671": 425553408.0,
+ "1672": 425553408.0,
+ "1673": 425553408.0,
+ "1674": 425553408.0,
+ "1675": 425553408.0,
+ "1676": 425553408.0,
+ "1677": 425553408.0,
+ "1678": 425553408.0,
+ "1679": 425553408.0,
+ "1680": 425553408.0,
+ "1681": 425553408.0,
+ "1682": 425553408.0,
+ "1683": 425553408.0,
+ "1684": 425553408.0,
+ "1685": 425553408.0,
+ "1686": 425553408.0,
+ "1687": 425553408.0,
+ "1688": 425553408.0,
+ "1689": 425553408.0,
+ "1690": 425553408.0,
+ "1691": 425553408.0,
+ "1692": 425553408.0,
+ "1693": 425553408.0,
+ "1694": 425553408.0,
+ "1695": 425553408.0,
+ "1696": 425553408.0,
+ "1697": 425553408.0,
+ "1698": 425553408.0,
+ "1699": 425553408.0,
+ "1700": 425553408.0,
+ "1701": 425553408.0,
+ "1702": 425553408.0,
+ "1703": 425553408.0,
+ "1704": 425553408.0,
+ "1705": 425553408.0,
+ "1706": 425553408.0,
+ "1707": 425553408.0,
+ "1708": 425553408.0,
+ "1709": 425553408.0,
+ "1710": 425553408.0,
+ "1711": 425553408.0,
+ "1712": 425553408.0,
+ "1713": 425553408.0,
+ "1714": 425553408.0,
+ "1715": 425553408.0,
+ "1716": 425553408.0,
+ "1717": 425553408.0,
+ "1718": 425553408.0,
+ "1719": 425553408.0,
+ "1720": 425553408.0,
+ "1721": 425553408.0,
+ "1722": 425553408.0,
+ "1723": 425553408.0,
+ "1724": 425553408.0,
+ "1725": 425553408.0,
+ "1726": 425553408.0,
+ "1727": 425553408.0,
+ "1728": 425553408.0,
+ "1729": 425553408.0,
+ "1730": 425553408.0,
+ "1731": 425553408.0,
+ "1732": 425553408.0,
+ "1733": 425553408.0,
+ "1734": 425553408.0,
+ "1735": 425553408.0,
+ "1736": 425553408.0,
+ "1737": 425553408.0,
+ "1738": 425553408.0,
+ "1739": 425553408.0,
+ "1740": 425553408.0,
+ "1741": 425553408.0,
+ "1742": 425553408.0,
+ "1743": 425553408.0,
+ "1744": 425553408.0,
+ "1745": 425553408.0,
+ "1746": 425553408.0,
+ "1747": 425553408.0,
+ "1748": 425553408.0,
+ "1749": 425553408.0,
+ "1750": 425553408.0,
+ "1751": 425553408.0,
+ "1752": 425553408.0,
+ "1753": 425553408.0,
+ "1754": 425553408.0,
+ "1755": 425553408.0,
+ "1756": 425553408.0,
+ "1757": 425553408.0,
+ "1758": 425553408.0,
+ "1759": 425553408.0,
+ "1760": 425553408.0,
+ "1761": 425553408.0,
+ "1762": 425553408.0,
+ "1763": 425553408.0,
+ "1764": 425553408.0,
+ "1765": 425553408.0,
+ "1766": 425553408.0,
+ "1767": 425553408.0,
+ "1768": 425553408.0,
+ "1769": 425553408.0,
+ "1770": 425553408.0,
+ "1771": 425553408.0,
+ "1772": 425553408.0,
+ "1773": 425553408.0,
+ "1774": 425553408.0,
+ "1775": 425553408.0,
+ "1776": 425553408.0,
+ "1777": 425553408.0,
+ "1778": 425553408.0,
+ "1779": 425553408.0,
+ "1780": 425553408.0,
+ "1781": 425553408.0,
+ "1782": 425553408.0,
+ "1783": 425553408.0,
+ "1784": 425553408.0,
+ "1785": 425553408.0,
+ "1786": 425553408.0,
+ "1787": 425553408.0,
+ "1788": 425553408.0,
+ "1789": 425553408.0,
+ "1790": 425553408.0,
+ "1791": 425553408.0,
+ "1792": 425553408.0,
+ "1793": 425553408.0,
+ "1794": 425553408.0,
+ "1795": 425553408.0,
+ "1796": 425553408.0,
+ "1797": 425553408.0,
+ "1798": 425553408.0,
+ "1799": 425553408.0,
+ "1800": 425553408.0,
+ "1801": 425553408.0,
+ "1802": 425553408.0,
+ "1803": 425553408.0,
+ "1804": 425553408.0,
+ "1805": 425553408.0,
+ "1806": 425553408.0,
+ "1807": 425553408.0,
+ "1808": 425553408.0,
+ "1809": 425553408.0,
+ "1810": 425553408.0,
+ "1811": 425553408.0,
+ "1812": 425553408.0,
+ "1813": 425553408.0,
+ "1814": 425553408.0,
+ "1815": 425553408.0,
+ "1816": 425553408.0,
+ "1817": 425553408.0,
+ "1818": 425553408.0,
+ "1819": 425553408.0,
+ "1820": 425553408.0,
+ "1821": 425553408.0,
+ "1822": 425553408.0,
+ "1823": 425553408.0,
+ "1824": 425553408.0,
+ "1825": 425553408.0,
+ "1826": 425553408.0,
+ "1827": 425553408.0,
+ "1828": 425553408.0,
+ "1829": 425553408.0,
+ "1830": 425553408.0,
+ "1831": 425553408.0,
+ "1832": 425553408.0,
+ "1833": 425553408.0,
+ "1834": 425553408.0,
+ "1835": 425553408.0,
+ "1836": 425553408.0,
+ "1837": 425553408.0,
+ "1838": 425553408.0,
+ "1839": 425553408.0,
+ "1840": 425553408.0,
+ "1841": 425553408.0,
+ "1842": 425553408.0,
+ "1843": 425553408.0,
+ "1844": 425553408.0,
+ "1845": 425553408.0,
+ "1846": 425553408.0,
+ "1847": 425553408.0,
+ "1848": 425553408.0,
+ "1849": 425553408.0,
+ "1850": 425553408.0,
+ "1851": 425553408.0,
+ "1852": 425553408.0,
+ "1853": 425553408.0,
+ "1854": 425553408.0,
+ "1855": 425553408.0,
+ "1856": 425553408.0,
+ "1857": 425553408.0,
+ "1858": 425553408.0,
+ "1859": 425553408.0,
+ "1860": 425553408.0,
+ "1861": 425553408.0,
+ "1862": 425553408.0,
+ "1863": 425553408.0,
+ "1864": 425553408.0,
+ "1865": 425553408.0,
+ "1866": 425553408.0,
+ "1867": 425553408.0,
+ "1868": 425553408.0,
+ "1869": 425553408.0,
+ "1870": 425553408.0,
+ "1871": 425553408.0,
+ "1872": 425553408.0,
+ "1873": 425553408.0,
+ "1874": 425553408.0,
+ "1875": 425553408.0,
+ "1876": 425553408.0,
+ "1877": 425553408.0,
+ "1878": 425553408.0,
+ "1879": 425553408.0,
+ "1880": 425553408.0,
+ "1881": 425553408.0,
+ "1882": 425553408.0,
+ "1883": 425553408.0,
+ "1884": 425553408.0,
+ "1885": 425553408.0,
+ "1886": 425553408.0,
+ "1887": 425553408.0,
+ "1888": 425553408.0,
+ "1889": 425553408.0,
+ "1890": 425553408.0,
+ "1891": 425553408.0,
+ "1892": 425553408.0,
+ "1893": 425553408.0,
+ "1894": 425553408.0,
+ "1895": 425553408.0,
+ "1896": 425553408.0,
+ "1897": 425553408.0,
+ "1898": 425553408.0,
+ "1899": 425553408.0,
+ "1900": 425553408.0,
+ "1901": 425553408.0,
+ "1902": 425553408.0,
+ "1903": 425553408.0,
+ "1904": 425553408.0,
+ "1905": 425553408.0,
+ "1906": 425553408.0,
+ "1907": 425553408.0,
+ "1908": 425553408.0,
+ "1909": 425553408.0,
+ "1910": 425553408.0,
+ "1911": 425553408.0,
+ "1912": 425553408.0,
+ "1913": 425553408.0,
+ "1914": 425553408.0,
+ "1915": 425553408.0,
+ "1916": 425553408.0,
+ "1917": 425553408.0,
+ "1918": 425553408.0,
+ "1919": 425553408.0,
+ "1920": 425553408.0,
+ "1921": 425553408.0,
+ "1922": 425553408.0,
+ "1923": 425553408.0,
+ "1924": 425553408.0,
+ "1925": 425553408.0,
+ "1926": 425553408.0,
+ "1927": 425553408.0,
+ "1928": 425553408.0,
+ "1929": 425553408.0,
+ "1930": 425553408.0,
+ "1931": 425553408.0,
+ "1932": 425553408.0,
+ "1933": 425553408.0,
+ "1934": 425553408.0,
+ "1935": 425553408.0,
+ "1936": 425553408.0,
+ "1937": 425553408.0,
+ "1938": 425553408.0,
+ "1939": 425553408.0,
+ "1940": 425553408.0,
+ "1941": 425553408.0,
+ "1942": 425553408.0,
+ "1943": 425553408.0,
+ "1944": 425553408.0,
+ "1945": 425553408.0,
+ "1946": 425553408.0,
+ "1947": 425553408.0,
+ "1948": 425553408.0,
+ "1949": 425553408.0,
+ "1950": 425553408.0,
+ "1951": 425553408.0,
+ "1952": 425553408.0,
+ "1953": 425553408.0,
+ "1954": 425553408.0,
+ "1955": 425553408.0,
+ "1956": 425553408.0,
+ "1957": 425553408.0,
+ "1958": 425553408.0,
+ "1959": 425553408.0,
+ "1960": 425553408.0,
+ "1961": 425553408.0,
+ "1962": 425553408.0,
+ "1963": 425553408.0,
+ "1964": 425553408.0,
+ "1965": 425553408.0,
+ "1966": 425553408.0,
+ "1967": 425553408.0,
+ "1968": 425553408.0,
+ "1969": 425553408.0,
+ "1970": 425553408.0,
+ "1971": 425553408.0,
+ "1972": 425553408.0,
+ "1973": 425553408.0,
+ "1974": 425553408.0,
+ "1975": 425553408.0,
+ "1976": 425553408.0,
+ "1977": 425553408.0,
+ "1978": 425553408.0,
+ "1979": 425553408.0,
+ "1980": 425553408.0,
+ "1981": 425553408.0,
+ "1982": 425553408.0,
+ "1983": 425553408.0,
+ "1984": 425553408.0,
+ "1985": 425553408.0,
+ "1986": 425553408.0,
+ "1987": 425553408.0,
+ "1988": 425553408.0,
+ "1989": 425553408.0,
+ "1990": 425553408.0,
+ "1991": 425553408.0,
+ "1992": 425553408.0,
+ "1993": 425553408.0,
+ "1994": 425553408.0,
+ "1995": 425553408.0,
+ "1996": 425553408.0,
+ "1997": 425553408.0,
+ "1998": 425553408.0,
+ "1999": 425553408.0,
+ "2000": 425553408.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 2000,
+ "step_interval": 1,
+ "values": {
+ "1": "nan",
+ "2": 12.51233,
+ "3": 5.38107,
+ "4": 5.43611,
+ "5": 5.38807,
+ "6": 5.37701,
+ "7": 5.38851,
+ "8": 5.38392,
+ "9": 5.35687,
+ "10": 5.36761,
+ "11": 5.34916,
+ "12": 5.34653,
+ "13": 5.36137,
+ "14": 5.36685,
+ "15": 5.38801,
+ "16": 5.36404,
+ "17": 5.81968,
+ "18": 5.36053,
+ "19": 5.36537,
+ "20": 6.37766,
+ "21": 5.8377,
+ "22": 5.40982,
+ "23": 5.86089,
+ "24": 5.78165,
+ "25": 5.40321,
+ "26": 5.39633,
+ "27": 5.39632,
+ "28": 5.39006,
+ "29": 5.39554,
+ "30": 5.94685,
+ "31": 5.41178,
+ "32": 5.47463,
+ "33": 5.40806,
+ "34": 5.41449,
+ "35": 5.42168,
+ "36": 5.39859,
+ "37": 5.38063,
+ "38": 5.38003,
+ "39": 5.38433,
+ "40": 5.3937,
+ "41": 5.39531,
+ "42": 5.4117,
+ "43": 5.40242,
+ "44": 5.47234,
+ "45": 5.40276,
+ "46": 5.40821,
+ "47": 5.40766,
+ "48": 5.39421,
+ "49": 5.40115,
+ "50": 5.42687,
+ "51": 5.40249,
+ "52": 5.39937,
+ "53": 5.40816,
+ "54": 5.4026,
+ "55": 5.42729,
+ "56": 5.40094,
+ "57": 5.40721,
+ "58": 5.78779,
+ "59": 5.75954,
+ "60": 5.40515,
+ "61": 5.8182,
+ "62": 5.39655,
+ "63": 5.42419,
+ "64": 5.42326,
+ "65": 5.84261,
+ "66": 5.41224,
+ "67": 6.21149,
+ "68": 5.44034,
+ "69": 5.42972,
+ "70": 5.40235,
+ "71": 5.39535,
+ "72": 5.40167,
+ "73": 5.79976,
+ "74": 5.4014,
+ "75": 5.40734,
+ "76": 5.40112,
+ "77": 5.39989,
+ "78": 5.3931,
+ "79": 5.38962,
+ "80": 5.38777,
+ "81": 5.40786,
+ "82": 5.39614,
+ "83": 5.40201,
+ "84": 5.39241,
+ "85": 5.40534,
+ "86": 5.4024,
+ "87": 5.4108,
+ "88": 5.40083,
+ "89": 5.38999,
+ "90": 5.38609,
+ "91": 5.39052,
+ "92": 5.39505,
+ "93": 5.39344,
+ "94": 5.48237,
+ "95": 5.4004,
+ "96": 5.39585,
+ "97": 6.13105,
+ "98": 5.42889,
+ "99": 5.37775,
+ "100": 5.40156,
+ "101": 5.3645,
+ "102": 5.41952,
+ "103": 5.38317,
+ "104": 5.39745,
+ "105": 5.37326,
+ "106": 5.43639,
+ "107": 5.85242,
+ "108": 5.3758,
+ "109": 5.41003,
+ "110": 5.81833,
+ "111": 5.83699,
+ "112": 5.74985,
+ "113": 5.34368,
+ "114": 5.34003,
+ "115": 5.33966,
+ "116": 5.34042,
+ "117": 5.34264,
+ "118": 5.73072,
+ "119": 5.3424,
+ "120": 5.33546,
+ "121": 5.34142,
+ "122": 5.36445,
+ "123": 5.33407,
+ "124": 5.38864,
+ "125": 5.34486,
+ "126": 5.33153,
+ "127": 5.34204,
+ "128": 5.3357,
+ "129": 5.33416,
+ "130": 5.36781,
+ "131": 5.37783,
+ "132": 5.39624,
+ "133": 5.37493,
+ "134": 5.36433,
+ "135": 6.11344,
+ "136": 5.36697,
+ "137": 5.37509,
+ "138": 5.37024,
+ "139": 5.38236,
+ "140": 5.37056,
+ "141": 5.35792,
+ "142": 5.35188,
+ "143": 5.43331,
+ "144": 5.39253,
+ "145": 5.37174,
+ "146": 5.39412,
+ "147": 5.38641,
+ "148": 5.39539,
+ "149": 5.38535,
+ "150": 5.39954,
+ "151": 5.41714,
+ "152": 5.85479,
+ "153": 5.4315,
+ "154": 5.87226,
+ "155": 5.81695,
+ "156": 5.78855,
+ "157": 5.38966,
+ "158": 5.41632,
+ "159": 5.39249,
+ "160": 5.38991,
+ "161": 5.39621,
+ "162": 5.77598,
+ "163": 5.38513,
+ "164": 5.38581,
+ "165": 5.38102,
+ "166": 5.3775,
+ "167": 5.50289,
+ "168": 5.39478,
+ "169": 5.37797,
+ "170": 5.36418,
+ "171": 5.33185,
+ "172": 5.33366,
+ "173": 6.07498,
+ "174": 5.43872,
+ "175": 5.33404,
+ "176": 5.33202,
+ "177": 5.33685,
+ "178": 5.36002,
+ "179": 5.37133,
+ "180": 5.38628,
+ "181": 5.40107,
+ "182": 5.40375,
+ "183": 5.4013,
+ "184": 5.40561,
+ "185": 5.39697,
+ "186": 5.39707,
+ "187": 5.38658,
+ "188": 5.37678,
+ "189": 5.37173,
+ "190": 5.37715,
+ "191": 5.37252,
+ "192": 5.37411,
+ "193": 5.37278,
+ "194": 5.37334,
+ "195": 5.37926,
+ "196": 5.80441,
+ "197": 5.33739,
+ "198": 5.78603,
+ "199": 6.14362,
+ "200": 5.3361,
+ "201": 5.33762,
+ "202": 5.33064,
+ "203": 5.32723,
+ "204": 5.33647,
+ "205": 5.72723,
+ "206": 5.36248,
+ "207": 5.34027,
+ "208": 5.33864,
+ "209": 5.33631,
+ "210": 5.36447,
+ "211": 5.71615,
+ "212": 5.72402,
+ "213": 5.33951,
+ "214": 5.37293,
+ "215": 5.33691,
+ "216": 5.36962,
+ "217": 5.39059,
+ "218": 5.39931,
+ "219": 5.36717,
+ "220": 5.3561,
+ "221": 5.37254,
+ "222": 5.3825,
+ "223": 5.38876,
+ "224": 5.36105,
+ "225": 5.37372,
+ "226": 5.35209,
+ "227": 5.3522,
+ "228": 5.34677,
+ "229": 5.55011,
+ "230": 5.57165,
+ "231": 5.3597,
+ "232": 5.35399,
+ "233": 5.35783,
+ "234": 5.34835,
+ "235": 5.36752,
+ "236": 5.37222,
+ "237": 5.37846,
+ "238": 5.38365,
+ "239": 5.37731,
+ "240": 5.38315,
+ "241": 5.79819,
+ "242": 6.04949,
+ "243": 6.19066,
+ "244": 5.37991,
+ "245": 5.37521,
+ "246": 5.39061,
+ "247": 5.37547,
+ "248": 5.37602,
+ "249": 6.15588,
+ "250": 5.74229,
+ "251": 5.38515,
+ "252": 5.37893,
+ "253": 5.37827,
+ "254": 5.38335,
+ "255": 5.37063,
+ "256": 5.37671,
+ "257": 5.37757,
+ "258": 5.37494,
+ "259": 5.37392,
+ "260": 5.3754,
+ "261": 5.37818,
+ "262": 5.39814,
+ "263": 5.37607,
+ "264": 5.37141,
+ "265": 5.38276,
+ "266": 5.37264,
+ "267": 5.37192,
+ "268": 5.36758,
+ "269": 5.3743,
+ "270": 5.37887,
+ "271": 5.37943,
+ "272": 5.36742,
+ "273": 5.38948,
+ "274": 5.3786,
+ "275": 5.37719,
+ "276": 5.37608,
+ "277": 5.38853,
+ "278": 5.40298,
+ "279": 5.63569,
+ "280": 5.33433,
+ "281": 5.33397,
+ "282": 5.34111,
+ "283": 5.33914,
+ "284": 5.36332,
+ "285": 5.41708,
+ "286": 5.78367,
+ "287": 5.76089,
+ "288": 6.88674,
+ "289": 5.3455,
+ "290": 5.32874,
+ "291": 5.3303,
+ "292": 5.32963,
+ "293": 5.3383,
+ "294": 5.72508,
+ "295": 5.33527,
+ "296": 5.33248,
+ "297": 5.33363,
+ "298": 5.33576,
+ "299": 5.34013,
+ "300": 5.3387,
+ "301": 5.34509,
+ "302": 5.3572,
+ "303": 5.34198,
+ "304": 5.34368,
+ "305": 5.33758,
+ "306": 5.35381,
+ "307": 5.35103,
+ "308": 5.35277,
+ "309": 5.35496,
+ "310": 5.3708,
+ "311": 5.35318,
+ "312": 5.35528,
+ "313": 5.3419,
+ "314": 5.344,
+ "315": 5.33865,
+ "316": 5.33438,
+ "317": 5.33138,
+ "318": 5.33038,
+ "319": 5.33401,
+ "320": 5.33214,
+ "321": 5.33234,
+ "322": 5.33488,
+ "323": 5.33058,
+ "324": 5.3341,
+ "325": 5.33141,
+ "326": 6.06721,
+ "327": 5.34541,
+ "328": 5.60892,
+ "329": 5.34021,
+ "330": 5.77231,
+ "331": 5.3414,
+ "332": 6.14807,
+ "333": 5.33897,
+ "334": 5.33416,
+ "335": 5.34238,
+ "336": 5.33039,
+ "337": 5.3327,
+ "338": 5.72458,
+ "339": 5.31977,
+ "340": 5.4941,
+ "341": 5.34033,
+ "342": 5.32988,
+ "343": 5.34033,
+ "344": 5.33804,
+ "345": 5.3294,
+ "346": 5.4022,
+ "347": 5.32792,
+ "348": 5.32317,
+ "349": 5.32405,
+ "350": 5.39646,
+ "351": 5.34439,
+ "352": 5.34008,
+ "353": 5.31893,
+ "354": 5.31786,
+ "355": 5.3256,
+ "356": 5.32723,
+ "357": 5.33685,
+ "358": 5.34858,
+ "359": 5.34684,
+ "360": 5.3441,
+ "361": 5.34912,
+ "362": 5.34559,
+ "363": 5.34605,
+ "364": 5.34379,
+ "365": 6.32129,
+ "366": 5.35471,
+ "367": 5.35577,
+ "368": 5.33216,
+ "369": 5.33469,
+ "370": 5.33013,
+ "371": 5.33693,
+ "372": 5.34241,
+ "373": 5.3579,
+ "374": 5.78389,
+ "375": 5.87048,
+ "376": 5.74997,
+ "377": 5.50656,
+ "378": 5.34186,
+ "379": 5.39843,
+ "380": 5.33396,
+ "381": 5.71694,
+ "382": 5.33425,
+ "383": 5.35855,
+ "384": 5.33921,
+ "385": 5.33978,
+ "386": 5.33436,
+ "387": 5.33441,
+ "388": 5.33455,
+ "389": 5.33695,
+ "390": 5.49468,
+ "391": 5.41034,
+ "392": 5.3234,
+ "393": 5.34048,
+ "394": 5.33444,
+ "395": 5.34855,
+ "396": 5.34664,
+ "397": 5.49699,
+ "398": 5.33294,
+ "399": 5.33901,
+ "400": 5.34532,
+ "401": 5.33257,
+ "402": 5.33348,
+ "403": 6.08214,
+ "404": 5.34681,
+ "405": 5.33367,
+ "406": 5.3466,
+ "407": 5.33965,
+ "408": 5.35401,
+ "409": 5.34442,
+ "410": 5.35179,
+ "411": 5.34334,
+ "412": 5.34333,
+ "413": 5.34804,
+ "414": 5.43767,
+ "415": 5.35507,
+ "416": 5.34641,
+ "417": 5.36275,
+ "418": 5.78921,
+ "419": 6.23226,
+ "420": 5.38676,
+ "421": 5.76643,
+ "422": 5.34597,
+ "423": 5.34811,
+ "424": 5.35012,
+ "425": 5.73256,
+ "426": 5.34156,
+ "427": 5.51796,
+ "428": 5.34889,
+ "429": 5.33723,
+ "430": 5.34124,
+ "431": 5.34529,
+ "432": 5.36247,
+ "433": 5.34164,
+ "434": 5.34469,
+ "435": 5.34123,
+ "436": 5.33387,
+ "437": 5.33603,
+ "438": 5.3367,
+ "439": 5.36131,
+ "440": 5.47961,
+ "441": 6.09917,
+ "442": 5.3466,
+ "443": 5.34527,
+ "444": 5.34466,
+ "445": 5.34952,
+ "446": 5.3403,
+ "447": 5.34167,
+ "448": 5.32421,
+ "449": 5.31606,
+ "450": 5.31731,
+ "451": 5.32048,
+ "452": 5.49269,
+ "453": 5.32086,
+ "454": 5.31902,
+ "455": 5.32121,
+ "456": 5.32793,
+ "457": 5.32358,
+ "458": 5.31817,
+ "459": 5.31755,
+ "460": 5.32279,
+ "461": 5.3354,
+ "462": 5.33399,
+ "463": 5.77169,
+ "464": 6.12628,
+ "465": 5.33523,
+ "466": 5.74268,
+ "467": 5.33068,
+ "468": 5.34697,
+ "469": 5.34437,
+ "470": 5.73494,
+ "471": 5.35257,
+ "472": 5.35073,
+ "473": 5.35221,
+ "474": 5.35559,
+ "475": 5.35035,
+ "476": 5.42878,
+ "477": 5.48222,
+ "478": 5.34389,
+ "479": 5.70513,
+ "480": 5.72111,
+ "481": 5.34552,
+ "482": 5.3487,
+ "483": 5.36978,
+ "484": 5.36234,
+ "485": 5.35375,
+ "486": 5.48301,
+ "487": 5.35391,
+ "488": 5.34697,
+ "489": 5.37903,
+ "490": 5.35629,
+ "491": 5.34831,
+ "492": 5.34471,
+ "493": 5.34772,
+ "494": 5.34459,
+ "495": 5.35838,
+ "496": 5.35635,
+ "497": 5.34682,
+ "498": 5.34692,
+ "499": 5.35023,
+ "500": 5.34548,
+ "501": 5.51183,
+ "502": 5.35089,
+ "503": 5.35211,
+ "504": 5.34707,
+ "505": 5.35219,
+ "506": 5.36181,
+ "507": 5.79546,
+ "508": 6.20197,
+ "509": 5.34896,
+ "510": 5.34612,
+ "511": 5.77396,
+ "512": 5.35309,
+ "513": 5.6625,
+ "514": 5.73599,
+ "515": 5.34708,
+ "516": 5.34587,
+ "517": 5.35125,
+ "518": 6.0996,
+ "519": 5.35963,
+ "520": 5.35867,
+ "521": 5.34826,
+ "522": 5.35273,
+ "523": 5.34801,
+ "524": 5.34867,
+ "525": 5.34521,
+ "526": 5.72797,
+ "527": 5.36603,
+ "528": 5.3424,
+ "529": 5.37762,
+ "530": 5.369,
+ "531": 5.36479,
+ "532": 5.35948,
+ "533": 5.36895,
+ "534": 5.35862,
+ "535": 5.37263,
+ "536": 5.35295,
+ "537": 5.36265,
+ "538": 5.78781,
+ "539": 5.35649,
+ "540": 5.36643,
+ "541": 5.36034,
+ "542": 5.35991,
+ "543": 5.37638,
+ "544": 5.37304,
+ "545": 5.36929,
+ "546": 5.36549,
+ "547": 5.36338,
+ "548": 5.36492,
+ "549": 5.36619,
+ "550": 5.9649,
+ "551": 5.76763,
+ "552": 5.78393,
+ "553": 5.37654,
+ "554": 5.36445,
+ "555": 5.38151,
+ "556": 6.51617,
+ "557": 5.75735,
+ "558": 5.37011,
+ "559": 5.36744,
+ "560": 5.37187,
+ "561": 5.35057,
+ "562": 5.37891,
+ "563": 5.36392,
+ "564": 5.3634,
+ "565": 5.45226,
+ "566": 5.38155,
+ "567": 5.3663,
+ "568": 5.40547,
+ "569": 5.40773,
+ "570": 5.40001,
+ "571": 5.40903,
+ "572": 5.38409,
+ "573": 5.37536,
+ "574": 5.39536,
+ "575": 5.35898,
+ "576": 5.35297,
+ "577": 5.3839,
+ "578": 5.3652,
+ "579": 5.35647,
+ "580": 5.34752,
+ "581": 5.35613,
+ "582": 5.36038,
+ "583": 5.40974,
+ "584": 5.40495,
+ "585": 5.39854,
+ "586": 5.40462,
+ "587": 5.40357,
+ "588": 5.39856,
+ "589": 5.41048,
+ "590": 5.38665,
+ "591": 5.34345,
+ "592": 5.34427,
+ "593": 5.34558,
+ "594": 5.8081,
+ "595": 6.49451,
+ "596": 5.77015,
+ "597": 5.35341,
+ "598": 5.35174,
+ "599": 5.37431,
+ "600": 5.36075,
+ "601": 6.15732,
+ "602": 5.35607,
+ "603": 5.37281,
+ "604": 5.35139,
+ "605": 5.35853,
+ "606": 5.34426,
+ "607": 5.35373,
+ "608": 5.34058,
+ "609": 5.33834,
+ "610": 5.3404,
+ "611": 5.34798,
+ "612": 5.35021,
+ "613": 5.33502,
+ "614": 5.34625,
+ "615": 5.3583,
+ "616": 5.37228,
+ "617": 5.41101,
+ "618": 5.41383,
+ "619": 5.39271,
+ "620": 5.39775,
+ "621": 5.39213,
+ "622": 5.39735,
+ "623": 5.39722,
+ "624": 5.37946,
+ "625": 5.38684,
+ "626": 5.37925,
+ "627": 5.42899,
+ "628": 5.34661,
+ "629": 5.35505,
+ "630": 5.371,
+ "631": 5.34981,
+ "632": 5.32814,
+ "633": 6.0842,
+ "634": 5.3372,
+ "635": 5.33124,
+ "636": 5.32745,
+ "637": 5.33008,
+ "638": 5.33725,
+ "639": 5.79535,
+ "640": 6.14819,
+ "641": 5.32646,
+ "642": 5.31501,
+ "643": 5.31665,
+ "644": 5.31897,
+ "645": 5.36289,
+ "646": 6.13707,
+ "647": 5.34291,
+ "648": 5.34813,
+ "649": 5.34634,
+ "650": 5.34942,
+ "651": 5.35473,
+ "652": 5.89563,
+ "653": 5.34927,
+ "654": 5.33879,
+ "655": 5.48244,
+ "656": 5.36941,
+ "657": 5.30935,
+ "658": 5.31271,
+ "659": 5.31694,
+ "660": 5.36652,
+ "661": 5.42274,
+ "662": 5.37283,
+ "663": 5.43423,
+ "664": 5.51146,
+ "665": 5.35386,
+ "666": 5.33377,
+ "667": 5.37764,
+ "668": 5.33403,
+ "669": 5.31786,
+ "670": 5.29321,
+ "671": 6.03712,
+ "672": 5.32534,
+ "673": 5.40214,
+ "674": 5.32025,
+ "675": 5.37137,
+ "676": 5.31187,
+ "677": 5.37397,
+ "678": 5.29768,
+ "679": 5.4012,
+ "680": 5.32528,
+ "681": 5.32253,
+ "682": 5.34392,
+ "683": 5.782,
+ "684": 6.16161,
+ "685": 5.30044,
+ "686": 5.29792,
+ "687": 5.30218,
+ "688": 5.30306,
+ "689": 5.74508,
+ "690": 5.91711,
+ "691": 5.69824,
+ "692": 5.28574,
+ "693": 5.28957,
+ "694": 5.29302,
+ "695": 5.30966,
+ "696": 5.29267,
+ "697": 5.29052,
+ "698": 5.28318,
+ "699": 5.29361,
+ "700": 5.38544,
+ "701": 5.31053,
+ "702": 5.32897,
+ "703": 5.32575,
+ "704": 5.33129,
+ "705": 5.32774,
+ "706": 5.33162,
+ "707": 5.48516,
+ "708": 5.32406,
+ "709": 5.29254,
+ "710": 6.02598,
+ "711": 5.28641,
+ "712": 5.33116,
+ "713": 5.32796,
+ "714": 5.32461,
+ "715": 5.32476,
+ "716": 5.31821,
+ "717": 5.31536,
+ "718": 5.34579,
+ "719": 5.35191,
+ "720": 5.35289,
+ "721": 5.37098,
+ "722": 5.32867,
+ "723": 5.3347,
+ "724": 5.32066,
+ "725": 5.32697,
+ "726": 5.34108,
+ "727": 6.34508,
+ "728": 5.74583,
+ "729": 5.48244,
+ "730": 5.32477,
+ "731": 5.32485,
+ "732": 5.32886,
+ "733": 5.72462,
+ "734": 5.31464,
+ "735": 5.31034,
+ "736": 5.73728,
+ "737": 5.31542,
+ "738": 5.38439,
+ "739": 5.27885,
+ "740": 5.27795,
+ "741": 5.2916,
+ "742": 5.2759,
+ "743": 5.27727,
+ "744": 5.29955,
+ "745": 5.28199,
+ "746": 5.28228,
+ "747": 5.28789,
+ "748": 6.02633,
+ "749": 5.34043,
+ "750": 5.3264,
+ "751": 5.73766,
+ "752": 5.32926,
+ "753": 5.30722,
+ "754": 5.30757,
+ "755": 5.31831,
+ "756": 5.32974,
+ "757": 5.31845,
+ "758": 5.32411,
+ "759": 5.31625,
+ "760": 5.31681,
+ "761": 5.3183,
+ "762": 5.31648,
+ "763": 5.30981,
+ "764": 5.55259,
+ "765": 5.25535,
+ "766": 5.27565,
+ "767": 5.26705,
+ "768": 5.28016,
+ "769": 5.27071,
+ "770": 5.28296,
+ "771": 6.13184,
+ "772": 5.70534,
+ "773": 5.2926,
+ "774": 5.28731,
+ "775": 5.296,
+ "776": 5.4408,
+ "777": 5.69767,
+ "778": 5.27914,
+ "779": 5.27235,
+ "780": 5.68447,
+ "781": 5.27418,
+ "782": 5.27048,
+ "783": 5.32361,
+ "784": 5.28044,
+ "785": 5.27604,
+ "786": 5.64646,
+ "787": 5.63194,
+ "788": 5.27044,
+ "789": 5.27906,
+ "790": 5.27383,
+ "791": 5.27232,
+ "792": 5.29428,
+ "793": 5.38329,
+ "794": 5.2857,
+ "795": 5.32362,
+ "796": 5.32215,
+ "797": 5.3241,
+ "798": 5.35573,
+ "799": 5.33254,
+ "800": 5.35652,
+ "801": 5.39299,
+ "802": 5.43078,
+ "803": 5.34628,
+ "804": 5.25493,
+ "805": 5.25398,
+ "806": 5.27352,
+ "807": 5.2762,
+ "808": 5.3041,
+ "809": 5.36687,
+ "810": 5.3079,
+ "811": 5.30002,
+ "812": 5.31695,
+ "813": 5.29553,
+ "814": 5.27269,
+ "815": 5.71772,
+ "816": 5.68091,
+ "817": 5.68997,
+ "818": 5.28735,
+ "819": 5.30471,
+ "820": 5.32561,
+ "821": 5.32306,
+ "822": 5.69795,
+ "823": 5.31608,
+ "824": 5.30894,
+ "825": 6.05549,
+ "826": 5.769,
+ "827": 5.31655,
+ "828": 5.3198,
+ "829": 5.30835,
+ "830": 5.31755,
+ "831": 5.31694,
+ "832": 5.31179,
+ "833": 5.33168,
+ "834": 5.32747,
+ "835": 5.31904,
+ "836": 5.31156,
+ "837": 5.3884,
+ "838": 5.3319,
+ "839": 5.49219,
+ "840": 5.33136,
+ "841": 5.31572,
+ "842": 5.3113,
+ "843": 5.35522,
+ "844": 5.31216,
+ "845": 5.30992,
+ "846": 5.31175,
+ "847": 5.31233,
+ "848": 5.31337,
+ "849": 5.30813,
+ "850": 5.37903,
+ "851": 5.37775,
+ "852": 5.28608,
+ "853": 5.26759,
+ "854": 5.26919,
+ "855": 5.27688,
+ "856": 5.26877,
+ "857": 5.29459,
+ "858": 5.31871,
+ "859": 5.71837,
+ "860": 6.09876,
+ "861": 5.28603,
+ "862": 5.28356,
+ "863": 6.01845,
+ "864": 5.24729,
+ "865": 5.23847,
+ "866": 5.64143,
+ "867": 5.24566,
+ "868": 5.25826,
+ "869": 5.25483,
+ "870": 5.23873,
+ "871": 5.65553,
+ "872": 5.26077,
+ "873": 5.24346,
+ "874": 5.27185,
+ "875": 5.289,
+ "876": 5.27628,
+ "877": 5.28131,
+ "878": 5.27434,
+ "879": 5.27313,
+ "880": 5.27917,
+ "881": 5.27329,
+ "882": 5.2713,
+ "883": 5.30342,
+ "884": 5.278,
+ "885": 5.28935,
+ "886": 5.27132,
+ "887": 5.27406,
+ "888": 5.29067,
+ "889": 5.67867,
+ "890": 5.30222,
+ "891": 5.28117,
+ "892": 5.27264,
+ "893": 5.27026,
+ "894": 5.26628,
+ "895": 5.2707,
+ "896": 5.26852,
+ "897": 5.27778,
+ "898": 5.27199,
+ "899": 5.2705,
+ "900": 5.2758,
+ "901": 5.65957,
+ "902": 5.66533,
+ "903": 6.10272,
+ "904": 5.6872,
+ "905": 5.29189,
+ "906": 5.27302,
+ "907": 5.27888,
+ "908": 5.28785,
+ "909": 5.67075,
+ "910": 5.2692,
+ "911": 5.27804,
+ "912": 5.26932,
+ "913": 5.32037,
+ "914": 5.2907,
+ "915": 5.67661,
+ "916": 5.27822,
+ "917": 5.27133,
+ "918": 5.27158,
+ "919": 5.2855,
+ "920": 5.33919,
+ "921": 5.32416,
+ "922": 5.31991,
+ "923": 5.31157,
+ "924": 5.32854,
+ "925": 5.32597,
+ "926": 5.34538,
+ "927": 5.38461,
+ "928": 5.33663,
+ "929": 5.33634,
+ "930": 5.45891,
+ "931": 5.2838,
+ "932": 5.28979,
+ "933": 5.26878,
+ "934": 5.26903,
+ "935": 5.28081,
+ "936": 5.26783,
+ "937": 5.28029,
+ "938": 5.29174,
+ "939": 5.37262,
+ "940": 6.06009,
+ "941": 5.32149,
+ "942": 5.36404,
+ "943": 5.35386,
+ "944": 5.36294,
+ "945": 5.39936,
+ "946": 5.35583,
+ "947": 5.78788,
+ "948": 5.79358,
+ "949": 5.36998,
+ "950": 5.38724,
+ "951": 5.42196,
+ "952": 5.42234,
+ "953": 5.8107,
+ "954": 5.37562,
+ "955": 5.42644,
+ "956": 5.39781,
+ "957": 5.35757,
+ "958": 5.36012,
+ "959": 5.36567,
+ "960": 5.85524,
+ "961": 5.37046,
+ "962": 5.39711,
+ "963": 5.37466,
+ "964": 5.41606,
+ "965": 5.33482,
+ "966": 5.37039,
+ "967": 5.34305,
+ "968": 5.35774,
+ "969": 5.33217,
+ "970": 5.32769,
+ "971": 5.29904,
+ "972": 5.31012,
+ "973": 5.30732,
+ "974": 5.27923,
+ "975": 5.27092,
+ "976": 5.29513,
+ "977": 5.28619,
+ "978": 6.02361,
+ "979": 5.28727,
+ "980": 5.28154,
+ "981": 5.32664,
+ "982": 5.34646,
+ "983": 5.33816,
+ "984": 5.34355,
+ "985": 5.34033,
+ "986": 5.32644,
+ "987": 5.35418,
+ "988": 5.35929,
+ "989": 5.34667,
+ "990": 5.33014,
+ "991": 5.29138,
+ "992": 5.75975,
+ "993": 5.74154,
+ "994": 5.36109,
+ "995": 5.3277,
+ "996": 5.33507,
+ "997": 5.32096,
+ "998": 5.75145,
+ "999": 5.354,
+ "1000": 5.32879,
+ "1001": 5.31668,
+ "1002": 5.3219,
+ "1003": 5.30942,
+ "1004": 5.72719,
+ "1005": 5.31808,
+ "1006": 5.31223,
+ "1007": 5.30501,
+ "1008": 5.32307,
+ "1009": 5.32333,
+ "1010": 5.31079,
+ "1011": 5.30929,
+ "1012": 5.30624,
+ "1013": 5.3121,
+ "1014": 5.30732,
+ "1015": 5.67497,
+ "1016": 5.69558,
+ "1017": 5.37133,
+ "1018": 5.35522,
+ "1019": 5.38407,
+ "1020": 5.34824,
+ "1021": 5.3493,
+ "1022": 5.31587,
+ "1023": 5.30628,
+ "1024": 5.31294,
+ "1025": 5.3045,
+ "1026": 5.29517,
+ "1027": 5.29317,
+ "1028": 5.30048,
+ "1029": 5.28105,
+ "1030": 5.28956,
+ "1031": 5.2875,
+ "1032": 5.30106,
+ "1033": 5.31821,
+ "1034": 5.82633,
+ "1035": 5.31548,
+ "1036": 5.7315,
+ "1037": 5.30905,
+ "1038": 5.31056,
+ "1039": 5.31731,
+ "1040": 5.70729,
+ "1041": 5.31833,
+ "1042": 5.32952,
+ "1043": 5.33397,
+ "1044": 5.32532,
+ "1045": 5.3218,
+ "1046": 5.31101,
+ "1047": 5.31562,
+ "1048": 5.31437,
+ "1049": 5.3225,
+ "1050": 5.7538,
+ "1051": 5.34286,
+ "1052": 5.32942,
+ "1053": 5.32889,
+ "1054": 6.1015,
+ "1055": 5.33823,
+ "1056": 5.34941,
+ "1057": 5.33712,
+ "1058": 5.55947,
+ "1059": 5.46609,
+ "1060": 5.33647,
+ "1061": 5.34579,
+ "1062": 5.32635,
+ "1063": 5.32029,
+ "1064": 5.32026,
+ "1065": 5.3163,
+ "1066": 5.34306,
+ "1067": 5.33146,
+ "1068": 5.33599,
+ "1069": 5.32787,
+ "1070": 5.33393,
+ "1071": 5.3375,
+ "1072": 5.3262,
+ "1073": 5.33097,
+ "1074": 5.32163,
+ "1075": 5.33309,
+ "1076": 5.32456,
+ "1077": 5.33718,
+ "1078": 5.78221,
+ "1079": 5.34537,
+ "1080": 5.74794,
+ "1081": 5.43355,
+ "1082": 5.34106,
+ "1083": 5.34389,
+ "1084": 5.74511,
+ "1085": 5.33701,
+ "1086": 5.33895,
+ "1087": 5.34328,
+ "1088": 5.34623,
+ "1089": 5.32927,
+ "1090": 5.34024,
+ "1091": 5.34367,
+ "1092": 5.7121,
+ "1093": 5.72868,
+ "1094": 5.33284,
+ "1095": 5.75299,
+ "1096": 5.33186,
+ "1097": 5.34066,
+ "1098": 5.32983,
+ "1099": 5.35196,
+ "1100": 5.3262,
+ "1101": 5.33277,
+ "1102": 5.31588,
+ "1103": 5.31994,
+ "1104": 5.31968,
+ "1105": 5.33959,
+ "1106": 5.32015,
+ "1107": 5.35197,
+ "1108": 5.34968,
+ "1109": 5.33445,
+ "1110": 5.33927,
+ "1111": 5.32862,
+ "1112": 5.33759,
+ "1113": 5.33148,
+ "1114": 5.32956,
+ "1115": 5.33433,
+ "1116": 5.33322,
+ "1117": 5.32817,
+ "1118": 5.33024,
+ "1119": 5.33046,
+ "1120": 5.33215,
+ "1121": 5.33368,
+ "1122": 5.75561,
+ "1123": 5.34097,
+ "1124": 5.76722,
+ "1125": 5.41546,
+ "1126": 5.42455,
+ "1127": 5.33863,
+ "1128": 5.72422,
+ "1129": 5.32852,
+ "1130": 5.69348,
+ "1131": 5.7036,
+ "1132": 5.33196,
+ "1133": 5.33321,
+ "1134": 5.31794,
+ "1135": 5.3196,
+ "1136": 5.3162,
+ "1137": 5.31514,
+ "1138": 5.3285,
+ "1139": 5.35085,
+ "1140": 5.74792,
+ "1141": 5.3237,
+ "1142": 5.3328,
+ "1143": 5.33102,
+ "1144": 5.33892,
+ "1145": 5.42008,
+ "1146": 5.64498,
+ "1147": 5.33496,
+ "1148": 5.33112,
+ "1149": 5.32984,
+ "1150": 5.33803,
+ "1151": 5.32808,
+ "1152": 5.33621,
+ "1153": 5.33802,
+ "1154": 5.32898,
+ "1155": 5.3424,
+ "1156": 5.31472,
+ "1157": 5.3238,
+ "1158": 5.37997,
+ "1159": 5.3164,
+ "1160": 5.31404,
+ "1161": 5.31623,
+ "1162": 5.31688,
+ "1163": 5.32519,
+ "1164": 5.31184,
+ "1165": 5.3198,
+ "1166": 5.32621,
+ "1167": 5.74092,
+ "1168": 6.12408,
+ "1169": 5.34133,
+ "1170": 5.71258,
+ "1171": 5.35747,
+ "1172": 5.32703,
+ "1173": 5.72184,
+ "1174": 5.32937,
+ "1175": 5.32564,
+ "1176": 5.32154,
+ "1177": 5.32452,
+ "1178": 5.32455,
+ "1179": 5.33716,
+ "1180": 5.3272,
+ "1181": 5.31722,
+ "1182": 5.32079,
+ "1183": 5.3292,
+ "1184": 5.33214,
+ "1185": 5.74396,
+ "1186": 5.32941,
+ "1187": 5.3201,
+ "1188": 5.34677,
+ "1189": 5.32953,
+ "1190": 5.40309,
+ "1191": 5.34742,
+ "1192": 5.34653,
+ "1193": 5.33854,
+ "1194": 5.34086,
+ "1195": 5.345,
+ "1196": 5.61311,
+ "1197": 5.35147,
+ "1198": 5.33658,
+ "1199": 5.34236,
+ "1200": 5.35179,
+ "1201": 5.34681,
+ "1202": 5.34335,
+ "1203": 5.34654,
+ "1204": 5.34289,
+ "1205": 5.33302,
+ "1206": 5.3361,
+ "1207": 5.69343,
+ "1208": 5.86487,
+ "1209": 5.3282,
+ "1210": 5.33241,
+ "1211": 6.13189,
+ "1212": 5.77896,
+ "1213": 5.33995,
+ "1214": 5.34813,
+ "1215": 5.33189,
+ "1216": 5.33445,
+ "1217": 5.73009,
+ "1218": 5.33223,
+ "1219": 5.34944,
+ "1220": 5.74636,
+ "1221": 5.34006,
+ "1222": 5.34817,
+ "1223": 5.3407,
+ "1224": 5.34002,
+ "1225": 5.35079,
+ "1226": 5.3291,
+ "1227": 5.3313,
+ "1228": 5.34495,
+ "1229": 5.34191,
+ "1230": 5.75882,
+ "1231": 5.33837,
+ "1232": 5.37363,
+ "1233": 5.39423,
+ "1234": 5.35185,
+ "1235": 5.3417,
+ "1236": 5.50865,
+ "1237": 5.31858,
+ "1238": 5.32078,
+ "1239": 5.31829,
+ "1240": 5.31661,
+ "1241": 5.33811,
+ "1242": 5.35188,
+ "1243": 5.34521,
+ "1244": 5.34612,
+ "1245": 5.71549,
+ "1246": 5.34596,
+ "1247": 5.71559,
+ "1248": 5.33246,
+ "1249": 5.33291,
+ "1250": 5.32675,
+ "1251": 5.30613,
+ "1252": 5.30507,
+ "1253": 5.32277,
+ "1254": 5.3089,
+ "1255": 6.10557,
+ "1256": 5.78157,
+ "1257": 5.30867,
+ "1258": 5.30813,
+ "1259": 5.30613,
+ "1260": 5.30262,
+ "1261": 5.72185,
+ "1262": 5.31825,
+ "1263": 5.30241,
+ "1264": 5.30885,
+ "1265": 5.31067,
+ "1266": 5.29121,
+ "1267": 5.29382,
+ "1268": 5.28799,
+ "1269": 5.29079,
+ "1270": 5.43982,
+ "1271": 5.28748,
+ "1272": 5.30503,
+ "1273": 5.32073,
+ "1274": 5.31184,
+ "1275": 5.73391,
+ "1276": 5.32947,
+ "1277": 5.33639,
+ "1278": 5.33161,
+ "1279": 5.3208,
+ "1280": 5.31685,
+ "1281": 5.33068,
+ "1282": 5.32207,
+ "1283": 5.68222,
+ "1284": 5.31021,
+ "1285": 5.31238,
+ "1286": 5.70598,
+ "1287": 5.32122,
+ "1288": 5.31231,
+ "1289": 5.31663,
+ "1290": 5.30903,
+ "1291": 5.31534,
+ "1292": 5.31093,
+ "1293": 5.32517,
+ "1294": 5.33023,
+ "1295": 5.3319,
+ "1296": 5.32527,
+ "1297": 5.35032,
+ "1298": 5.31208,
+ "1299": 6.10759,
+ "1300": 5.72768,
+ "1301": 5.3171,
+ "1302": 5.3123,
+ "1303": 5.33391,
+ "1304": 5.32105,
+ "1305": 5.69687,
+ "1306": 5.30746,
+ "1307": 5.31356,
+ "1308": 5.31388,
+ "1309": 5.32425,
+ "1310": 5.31748,
+ "1311": 5.31406,
+ "1312": 5.32971,
+ "1313": 5.32388,
+ "1314": 5.32458,
+ "1315": 5.3312,
+ "1316": 5.32126,
+ "1317": 5.3286,
+ "1318": 5.3391,
+ "1319": 5.32818,
+ "1320": 5.74387,
+ "1321": 5.70364,
+ "1322": 5.33143,
+ "1323": 5.34082,
+ "1324": 5.71456,
+ "1325": 5.33684,
+ "1326": 5.34404,
+ "1327": 5.32921,
+ "1328": 5.33022,
+ "1329": 5.39487,
+ "1330": 5.41362,
+ "1331": 5.35261,
+ "1332": 5.37793,
+ "1333": 5.34351,
+ "1334": 5.31993,
+ "1335": 5.32358,
+ "1336": 5.32136,
+ "1337": 5.32644,
+ "1338": 5.3269,
+ "1339": 5.3128,
+ "1340": 5.32071,
+ "1341": 5.32386,
+ "1342": 5.32249,
+ "1343": 5.72607,
+ "1344": 5.75031,
+ "1345": 5.76567,
+ "1346": 5.32649,
+ "1347": 5.32528,
+ "1348": 5.328,
+ "1349": 5.72516,
+ "1350": 5.33393,
+ "1351": 5.32269,
+ "1352": 5.33053,
+ "1353": 5.32501,
+ "1354": 5.32683,
+ "1355": 5.32693,
+ "1356": 5.33978,
+ "1357": 5.38347,
+ "1358": 5.332,
+ "1359": 5.33579,
+ "1360": 5.71001,
+ "1361": 5.31291,
+ "1362": 5.69096,
+ "1363": 5.31963,
+ "1364": 5.32554,
+ "1365": 5.73097,
+ "1366": 5.35591,
+ "1367": 5.35179,
+ "1368": 5.33422,
+ "1369": 5.33799,
+ "1370": 5.36388,
+ "1371": 5.33198,
+ "1372": 5.33396,
+ "1373": 5.32397,
+ "1374": 5.32394,
+ "1375": 5.33489,
+ "1376": 5.32536,
+ "1377": 5.33064,
+ "1378": 5.32759,
+ "1379": 5.32929,
+ "1380": 5.31842,
+ "1381": 5.31482,
+ "1382": 5.31699,
+ "1383": 5.33047,
+ "1384": 5.32231,
+ "1385": 5.33191,
+ "1386": 5.31442,
+ "1387": 5.71388,
+ "1388": 5.71023,
+ "1389": 5.74708,
+ "1390": 5.33462,
+ "1391": 5.34504,
+ "1392": 5.34841,
+ "1393": 5.73998,
+ "1394": 5.34446,
+ "1395": 5.34359,
+ "1396": 5.34016,
+ "1397": 5.33309,
+ "1398": 5.71716,
+ "1399": 5.34944,
+ "1400": 5.34629,
+ "1401": 5.71739,
+ "1402": 5.33643,
+ "1403": 5.33413,
+ "1404": 5.32455,
+ "1405": 5.33171,
+ "1406": 5.33227,
+ "1407": 5.35613,
+ "1408": 5.31802,
+ "1409": 5.31986,
+ "1410": 5.37851,
+ "1411": 5.76431,
+ "1412": 5.33903,
+ "1413": 5.33545,
+ "1414": 5.33465,
+ "1415": 5.32792,
+ "1416": 5.33036,
+ "1417": 5.33276,
+ "1418": 5.32714,
+ "1419": 5.33448,
+ "1420": 5.33457,
+ "1421": 5.34708,
+ "1422": 5.33855,
+ "1423": 5.32931,
+ "1424": 5.33382,
+ "1425": 5.32951,
+ "1426": 5.32385,
+ "1427": 5.32822,
+ "1428": 5.33471,
+ "1429": 5.33559,
+ "1430": 5.34263,
+ "1431": 5.7869,
+ "1432": 5.74059,
+ "1433": 5.7612,
+ "1434": 5.35909,
+ "1435": 5.33806,
+ "1436": 5.71607,
+ "1437": 5.74329,
+ "1438": 5.34001,
+ "1439": 5.34896,
+ "1440": 5.71864,
+ "1441": 5.34688,
+ "1442": 5.33898,
+ "1443": 5.34614,
+ "1444": 5.39662,
+ "1445": 5.35605,
+ "1446": 5.33639,
+ "1447": 5.34408,
+ "1448": 5.33963,
+ "1449": 5.32594,
+ "1450": 5.33295,
+ "1451": 5.31906,
+ "1452": 5.32314,
+ "1453": 5.31887,
+ "1454": 5.32237,
+ "1455": 5.32481,
+ "1456": 5.76454,
+ "1457": 5.33827,
+ "1458": 5.33386,
+ "1459": 5.33391,
+ "1460": 5.32325,
+ "1461": 5.32921,
+ "1462": 5.3392,
+ "1463": 5.33031,
+ "1464": 5.32847,
+ "1465": 5.33118,
+ "1466": 5.31961,
+ "1467": 5.33284,
+ "1468": 5.30867,
+ "1469": 5.31357,
+ "1470": 5.30973,
+ "1471": 5.30994,
+ "1472": 5.3098,
+ "1473": 5.32113,
+ "1474": 5.31648,
+ "1475": 6.08142,
+ "1476": 5.7034,
+ "1477": 5.72775,
+ "1478": 5.70943,
+ "1479": 5.33441,
+ "1480": 5.32526,
+ "1481": 6.05326,
+ "1482": 5.30275,
+ "1483": 5.31176,
+ "1484": 5.32466,
+ "1485": 5.31392,
+ "1486": 5.3123,
+ "1487": 5.31161,
+ "1488": 5.31297,
+ "1489": 5.3067,
+ "1490": 5.30334,
+ "1491": 5.29821,
+ "1492": 5.2962,
+ "1493": 5.3,
+ "1494": 5.31315,
+ "1495": 5.30788,
+ "1496": 5.30836,
+ "1497": 5.30952,
+ "1498": 5.30965,
+ "1499": 5.31348,
+ "1500": 5.73016,
+ "1501": 5.3546,
+ "1502": 5.33645,
+ "1503": 5.34041,
+ "1504": 5.32539,
+ "1505": 5.32755,
+ "1506": 5.34669,
+ "1507": 5.34089,
+ "1508": 5.33383,
+ "1509": 5.32748,
+ "1510": 5.31882,
+ "1511": 5.30917,
+ "1512": 5.32769,
+ "1513": 5.32181,
+ "1514": 5.6844,
+ "1515": 5.27912,
+ "1516": 5.66513,
+ "1517": 5.28292,
+ "1518": 5.47709,
+ "1519": 5.29407,
+ "1520": 6.04225,
+ "1521": 5.29335,
+ "1522": 5.72528,
+ "1523": 5.30616,
+ "1524": 5.31221,
+ "1525": 5.29698,
+ "1526": 5.70816,
+ "1527": 5.28714,
+ "1528": 5.28684,
+ "1529": 5.29559,
+ "1530": 5.29893,
+ "1531": 5.28889,
+ "1532": 5.3332,
+ "1533": 5.35654,
+ "1534": 5.36318,
+ "1535": 5.36189,
+ "1536": 5.35577,
+ "1537": 5.3629,
+ "1538": 5.37265,
+ "1539": 5.35687,
+ "1540": 5.3614,
+ "1541": 5.40991,
+ "1542": 5.36079,
+ "1543": 5.65089,
+ "1544": 5.39987,
+ "1545": 5.81511,
+ "1546": 5.40068,
+ "1547": 5.40642,
+ "1548": 5.40226,
+ "1549": 5.39258,
+ "1550": 5.38339,
+ "1551": 5.38876,
+ "1552": 5.7626,
+ "1553": 5.39227,
+ "1554": 5.40924,
+ "1555": 5.77927,
+ "1556": 5.3822,
+ "1557": 5.32648,
+ "1558": 5.32817,
+ "1559": 5.34604,
+ "1560": 5.32694,
+ "1561": 5.32689,
+ "1562": 5.36973,
+ "1563": 5.39541,
+ "1564": 6.1489,
+ "1565": 5.39443,
+ "1566": 5.78026,
+ "1567": 5.40976,
+ "1568": 5.42587,
+ "1569": 5.40199,
+ "1570": 5.79743,
+ "1571": 5.38703,
+ "1572": 5.40307,
+ "1573": 5.39456,
+ "1574": 5.41239,
+ "1575": 5.39883,
+ "1576": 5.40043,
+ "1577": 5.39875,
+ "1578": 5.40271,
+ "1579": 5.42979,
+ "1580": 5.4374,
+ "1581": 5.47503,
+ "1582": 5.40504,
+ "1583": 5.39604,
+ "1584": 5.39904,
+ "1585": 5.40405,
+ "1586": 5.40053,
+ "1587": 5.39363,
+ "1588": 5.39556,
+ "1589": 5.41287,
+ "1590": 5.77665,
+ "1591": 5.83291,
+ "1592": 5.46299,
+ "1593": 5.41762,
+ "1594": 5.79002,
+ "1595": 5.35668,
+ "1596": 5.3396,
+ "1597": 5.39144,
+ "1598": 5.39053,
+ "1599": 5.3869,
+ "1600": 5.38734,
+ "1601": 5.39301,
+ "1602": 5.39421,
+ "1603": 5.3952,
+ "1604": 5.39075,
+ "1605": 5.40889,
+ "1606": 5.3589,
+ "1607": 5.32116,
+ "1608": 6.09371,
+ "1609": 5.3217,
+ "1610": 5.72528,
+ "1611": 5.36688,
+ "1612": 5.31029,
+ "1613": 5.31233,
+ "1614": 5.70037,
+ "1615": 5.31647,
+ "1616": 5.31344,
+ "1617": 5.34765,
+ "1618": 5.34244,
+ "1619": 5.34788,
+ "1620": 5.34364,
+ "1621": 5.3087,
+ "1622": 5.29672,
+ "1623": 5.30631,
+ "1624": 5.30553,
+ "1625": 5.33698,
+ "1626": 5.33085,
+ "1627": 5.32789,
+ "1628": 5.3318,
+ "1629": 5.80354,
+ "1630": 5.33756,
+ "1631": 5.33332,
+ "1632": 5.70304,
+ "1633": 5.32374,
+ "1634": 5.32292,
+ "1635": 5.3212,
+ "1636": 5.75734,
+ "1637": 5.31067,
+ "1638": 5.31672,
+ "1639": 5.31582,
+ "1640": 5.3195,
+ "1641": 5.32102,
+ "1642": 5.3112,
+ "1643": 5.31152,
+ "1644": 5.30558,
+ "1645": 5.30775,
+ "1646": 5.30316,
+ "1647": 5.31594,
+ "1648": 5.30037,
+ "1649": 5.29762,
+ "1650": 5.30006,
+ "1651": 5.30723,
+ "1652": 6.05892,
+ "1653": 5.30779,
+ "1654": 5.68651,
+ "1655": 5.34908,
+ "1656": 5.31256,
+ "1657": 5.29291,
+ "1658": 5.70165,
+ "1659": 5.30307,
+ "1660": 5.28992,
+ "1661": 5.29633,
+ "1662": 5.30646,
+ "1663": 5.31879,
+ "1664": 5.33381,
+ "1665": 5.32496,
+ "1666": 5.31856,
+ "1667": 5.70521,
+ "1668": 5.30181,
+ "1669": 5.29874,
+ "1670": 5.7235,
+ "1671": 5.29944,
+ "1672": 5.30249,
+ "1673": 5.30367,
+ "1674": 5.29908,
+ "1675": 5.30219,
+ "1676": 5.34355,
+ "1677": 5.30561,
+ "1678": 5.30855,
+ "1679": 5.36178,
+ "1680": 5.31065,
+ "1681": 5.79206,
+ "1682": 5.29488,
+ "1683": 5.3007,
+ "1684": 5.30763,
+ "1685": 5.30001,
+ "1686": 5.31631,
+ "1687": 5.42252,
+ "1688": 5.33735,
+ "1689": 5.33696,
+ "1690": 5.3381,
+ "1691": 5.34605,
+ "1692": 5.34603,
+ "1693": 5.39975,
+ "1694": 5.34639,
+ "1695": 5.35336,
+ "1696": 5.75098,
+ "1697": 5.71539,
+ "1698": 5.7238,
+ "1699": 5.32357,
+ "1700": 5.33058,
+ "1701": 5.30998,
+ "1702": 5.701,
+ "1703": 5.31279,
+ "1704": 5.57806,
+ "1705": 5.67749,
+ "1706": 5.2948,
+ "1707": 5.32771,
+ "1708": 5.33296,
+ "1709": 5.71996,
+ "1710": 5.30607,
+ "1711": 5.30161,
+ "1712": 5.30164,
+ "1713": 5.30372,
+ "1714": 5.30396,
+ "1715": 5.32897,
+ "1716": 5.33218,
+ "1717": 5.68333,
+ "1718": 5.31835,
+ "1719": 5.31582,
+ "1720": 5.31564,
+ "1721": 5.32622,
+ "1722": 5.32538,
+ "1723": 5.33255,
+ "1724": 5.32588,
+ "1725": 5.74636,
+ "1726": 5.31693,
+ "1727": 5.31856,
+ "1728": 5.31785,
+ "1729": 5.32057,
+ "1730": 5.31797,
+ "1731": 5.31387,
+ "1732": 5.33328,
+ "1733": 5.31843,
+ "1734": 5.31554,
+ "1735": 5.32121,
+ "1736": 5.32408,
+ "1737": 5.30994,
+ "1738": 5.30646,
+ "1739": 5.3091,
+ "1740": 5.69561,
+ "1741": 5.90431,
+ "1742": 5.6841,
+ "1743": 5.68011,
+ "1744": 5.30654,
+ "1745": 5.31624,
+ "1746": 5.70436,
+ "1747": 5.69439,
+ "1748": 5.31266,
+ "1749": 5.31,
+ "1750": 5.30653,
+ "1751": 5.30879,
+ "1752": 5.30002,
+ "1753": 5.30366,
+ "1754": 5.31212,
+ "1755": 5.30525,
+ "1756": 5.30876,
+ "1757": 5.32869,
+ "1758": 5.31222,
+ "1759": 5.30623,
+ "1760": 5.31388,
+ "1761": 5.31214,
+ "1762": 5.30959,
+ "1763": 5.28303,
+ "1764": 5.28618,
+ "1765": 5.3032,
+ "1766": 5.30142,
+ "1767": 5.39708,
+ "1768": 5.28533,
+ "1769": 5.27033,
+ "1770": 5.27029,
+ "1771": 5.68958,
+ "1772": 5.27224,
+ "1773": 5.27525,
+ "1774": 5.27394,
+ "1775": 5.27316,
+ "1776": 5.27344,
+ "1777": 5.2811,
+ "1778": 5.27711,
+ "1779": 5.39894,
+ "1780": 5.30824,
+ "1781": 5.30045,
+ "1782": 5.66739,
+ "1783": 5.29631,
+ "1784": 5.69121,
+ "1785": 5.65582,
+ "1786": 6.04592,
+ "1787": 5.29927,
+ "1788": 5.30042,
+ "1789": 5.32303,
+ "1790": 5.69832,
+ "1791": 5.29177,
+ "1792": 5.2819,
+ "1793": 5.27917,
+ "1794": 5.28689,
+ "1795": 5.30036,
+ "1796": 5.2878,
+ "1797": 5.28797,
+ "1798": 5.28201,
+ "1799": 5.27976,
+ "1800": 5.28056,
+ "1801": 5.27129,
+ "1802": 5.28043,
+ "1803": 5.27676,
+ "1804": 5.2802,
+ "1805": 5.27494,
+ "1806": 5.29316,
+ "1807": 5.27278,
+ "1808": 5.28162,
+ "1809": 5.27072,
+ "1810": 5.27718,
+ "1811": 5.28225,
+ "1812": 5.29204,
+ "1813": 5.27878,
+ "1814": 5.28304,
+ "1815": 5.28019,
+ "1816": 5.70126,
+ "1817": 5.28192,
+ "1818": 5.27197,
+ "1819": 5.2775,
+ "1820": 5.65354,
+ "1821": 5.28314,
+ "1822": 5.3003,
+ "1823": 5.28291,
+ "1824": 5.66257,
+ "1825": 5.27526,
+ "1826": 5.27778,
+ "1827": 5.28045,
+ "1828": 5.68295,
+ "1829": 5.65198,
+ "1830": 5.67027,
+ "1831": 5.27599,
+ "1832": 5.31957,
+ "1833": 5.27956,
+ "1834": 5.7135,
+ "1835": 5.32169,
+ "1836": 5.32004,
+ "1837": 5.30878,
+ "1838": 5.31132,
+ "1839": 5.30715,
+ "1840": 5.32265,
+ "1841": 5.37081,
+ "1842": 5.33738,
+ "1843": 5.32598,
+ "1844": 5.32711,
+ "1845": 5.32441,
+ "1846": 5.32296,
+ "1847": 5.33141,
+ "1848": 5.32651,
+ "1849": 5.32256,
+ "1850": 5.32395,
+ "1851": 5.32835,
+ "1852": 5.32508,
+ "1853": 5.32404,
+ "1854": 5.34728,
+ "1855": 5.3401,
+ "1856": 5.31978,
+ "1857": 5.35013,
+ "1858": 5.70823,
+ "1859": 5.32803,
+ "1860": 5.32485,
+ "1861": 5.75438,
+ "1862": 5.33384,
+ "1863": 5.70446,
+ "1864": 5.30951,
+ "1865": 5.30391,
+ "1866": 5.30893,
+ "1867": 5.37124,
+ "1868": 5.32345,
+ "1869": 5.33757,
+ "1870": 5.33097,
+ "1871": 5.33587,
+ "1872": 5.33403,
+ "1873": 5.73122,
+ "1874": 5.69923,
+ "1875": 5.7144,
+ "1876": 5.33058,
+ "1877": 5.33228,
+ "1878": 5.34176,
+ "1879": 5.73575,
+ "1880": 5.30892,
+ "1881": 5.30528,
+ "1882": 5.29028,
+ "1883": 5.29587,
+ "1884": 5.29801,
+ "1885": 5.29538,
+ "1886": 5.27531,
+ "1887": 5.27152,
+ "1888": 5.28927,
+ "1889": 5.31821,
+ "1890": 5.32157,
+ "1891": 5.3352,
+ "1892": 5.45713,
+ "1893": 5.33925,
+ "1894": 5.3378,
+ "1895": 5.33719,
+ "1896": 5.71671,
+ "1897": 5.42117,
+ "1898": 5.3391,
+ "1899": 5.33491,
+ "1900": 5.34113,
+ "1901": 5.72791,
+ "1902": 5.43227,
+ "1903": 5.40732,
+ "1904": 5.67322,
+ "1905": 5.38508,
+ "1906": 5.74955,
+ "1907": 5.31355,
+ "1908": 5.36556,
+ "1909": 5.30959,
+ "1910": 5.31009,
+ "1911": 5.32046,
+ "1912": 5.31679,
+ "1913": 5.31813,
+ "1914": 5.37935,
+ "1915": 5.31518,
+ "1916": 5.3212,
+ "1917": 5.72567,
+ "1918": 5.67305,
+ "1919": 5.7326,
+ "1920": 5.31305,
+ "1921": 5.31439,
+ "1922": 5.31672,
+ "1923": 5.72894,
+ "1924": 5.32292,
+ "1925": 5.37521,
+ "1926": 5.3227,
+ "1927": 5.33898,
+ "1928": 5.411,
+ "1929": 5.35698,
+ "1930": 5.32646,
+ "1931": 5.31427,
+ "1932": 5.31665,
+ "1933": 5.31842,
+ "1934": 5.31868,
+ "1935": 5.67787,
+ "1936": 5.35873,
+ "1937": 5.34822,
+ "1938": 5.34749,
+ "1939": 5.34792,
+ "1940": 5.73239,
+ "1941": 5.35344,
+ "1942": 5.34736,
+ "1943": 5.34529,
+ "1944": 5.34634,
+ "1945": 5.35461,
+ "1946": 5.35907,
+ "1947": 5.36504,
+ "1948": 5.35061,
+ "1949": 5.34214,
+ "1950": 5.34188,
+ "1951": 5.76619,
+ "1952": 5.35525,
+ "1953": 5.36166,
+ "1954": 5.34642,
+ "1955": 5.34997,
+ "1956": 5.34383,
+ "1957": 5.34251,
+ "1958": 5.35039,
+ "1959": 5.32345,
+ "1960": 5.70915,
+ "1961": 5.32029,
+ "1962": 5.68241,
+ "1963": 5.65728,
+ "1964": 5.32803,
+ "1965": 5.36096,
+ "1966": 5.71921,
+ "1967": 5.32613,
+ "1968": 5.33584,
+ "1969": 5.3418,
+ "1970": 5.32655,
+ "1971": 5.32373,
+ "1972": 6.61849,
+ "1973": 5.69259,
+ "1974": 5.32121,
+ "1975": 5.34514,
+ "1976": 5.33539,
+ "1977": 5.33024,
+ "1978": 5.71369,
+ "1979": 5.32844,
+ "1980": 5.33303,
+ "1981": 5.30736,
+ "1982": 5.32579,
+ "1983": 5.31773,
+ "1984": 5.33124,
+ "1985": 5.32303,
+ "1986": 5.34222,
+ "1987": 5.35372,
+ "1988": 5.35724,
+ "1989": 5.36871,
+ "1990": 5.38439,
+ "1991": 5.37115,
+ "1992": 5.3544,
+ "1993": 5.34048,
+ "1994": 5.33628,
+ "1995": 5.327,
+ "1996": 5.75054,
+ "1997": 5.33898,
+ "1998": 5.33139,
+ "1999": 5.32919,
+ "2000": 5.33263
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml
similarity index 83%
rename from tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml
rename to tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml
index 3963a359ea9..2dccf144291 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt3_weekly_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -54,4 +54,6 @@ MODEL_ARGS:
--bf16: true
--log-memory-to-tensorboard: true
--normalization: RMSNorm
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp1_pp8_dp1_583m_logitsmatch_zmq/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp1_pp8_dp1_583m_logitsmatch_zmq/model_config.yaml
index bd34c11fc24..345fc250694 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp1_pp8_dp1_583m_logitsmatch_zmq/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp1_pp8_dp1_583m_logitsmatch_zmq/model_config.yaml
@@ -42,9 +42,6 @@ MODEL_ARGS:
--top_k: 1
--return-log-probs: true
--num-tokens-to-generate: 30
- --inference-dynamic-batching-max-requests-override: 8 # hardcode decode padding tokens to 7 for reproducibility
- --inference-dynamic-batching-buffer-guaranteed-fraction: 0
- --inference-dynamic-batching-buffer-overflow-factor: 0.2
--inference-dynamic-batching-buffer-size-gb: 20
--dist-ckpt-strictness: log_unexpected
--inference-ckpt-non-strict: true # To handle the extra_state errors
diff --git a/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp2_pp2_dp2_583m_logitsmatch_zmq/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp2_pp2_dp2_583m_logitsmatch_zmq/golden_values_dev_dgx_h100.json
index 55d6955055a..3af1d61504a 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp2_pp2_dp2_583m_logitsmatch_zmq/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp2_pp2_dp2_583m_logitsmatch_zmq/golden_values_dev_dgx_h100.json
@@ -1,158 +1,158 @@
{
"0": {
"input_prompt": "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies.",
- "generated_text": " And then you get to the end of the movie, and you realize that this is not New York at all. This is New York at the end",
+ "generated_text": " And that this is the place where you can be yourself, and be yourself in the most beautiful way. And that this is the place where you can",
"generated_tokens": [
3060,
- 2430,
- 1636,
- 2012,
- 1317,
- 1278,
- 2362,
- 1307,
+ 1455,
+ 1593,
+ 1395,
1278,
- 16070,
+ 3535,
+ 2478,
+ 1636,
+ 1710,
+ 1402,
+ 14019,
1044,
1321,
- 1636,
- 23067,
+ 1402,
+ 14019,
+ 1294,
+ 1278,
+ 2725,
+ 15568,
+ 3039,
+ 1046,
+ 3060,
1455,
1593,
1395,
- 1605,
- 3140,
- 5152,
- 1513,
- 1747,
- 1046,
- 2409,
- 1395,
- 3140,
- 5152,
- 1513,
1278,
- 2362
+ 3535,
+ 2478,
+ 1636,
+ 1710
],
- "latency": 44.73653959017247,
+ "latency": 56.18305364251137,
"logprobs": [
- -9.358970642089844,
- -2.7523813247680664,
- -4.628502368927002,
- -1.4058877229690552,
- -0.6050865054130554,
- -1.7354254722595215,
- -2.4828507900238037,
- -2.0520384311676025,
- -2.4089853763580322,
- -6.2649126052856445,
- -1.5644135475158691,
- -3.4096615314483643,
- -4.358163833618164,
- -3.866471767425537,
- -2.0575876235961914,
- -1.904883623123169,
- -3.7622976303100586,
- -6.835415363311768,
- -0.2829523980617523,
- -0.9827429056167603,
- -6.655940055847168,
- -7.188957214355469,
- -12.757233619689941,
- -2.1933951377868652,
- -3.808887481689453,
- -0.515199601650238,
- -4.323916912078857,
- -0.067625492811203,
- -0.09976530075073242,
- -3.228640556335449,
- -10.129311561584473,
- -1.1787357330322266,
- -5.97692346572876,
- -5.036575794219971,
- -3.8267176151275635,
- -2.6010468006134033,
- -3.366438865661621,
- -5.553505897521973,
- -1.6046268939971924,
- -5.442874908447266,
- -12.218503952026367,
- -12.597894668579102,
- -0.0976092740893364,
- -2.530579090118408,
- -1.4139617681503296,
- -2.8606526851654053,
- -1.1690009832382202,
- -0.0066696410067379475,
- -3.361189365386963,
- -13.191482543945312,
- -4.413737773895264,
- -2.639688491821289,
- -6.0114641189575195,
- -0.7672993540763855,
- -0.047326065599918365,
- -1.550362467765808,
- -1.137772798538208,
- -5.627618789672852,
- -0.40103790163993835,
- -4.908735275268555,
- -0.5704602599143982,
- -0.6625558733940125,
- -2.364135503768921,
- -13.609526634216309,
- -0.08865148574113846,
- -3.5251970291137695,
- -1.3791766166687012,
- -6.395696640014648,
- -0.588782787322998,
- -3.566770076751709,
- -0.8742034435272217,
- -1.5827170610427856,
- -5.3912353515625,
- -17.150842666625977,
- -6.6234588623046875,
- -0.885993242263794,
- -4.162992477416992,
- -1.1942744255065918,
- -2.281689405441284,
- -1.7708709239959717,
- -0.22030864655971527,
- -9.292593955993652,
- -0.1258234828710556,
- -7.346449851989746,
- -2.5470826625823975,
- -4.115433692932129,
- -3.5646262168884277,
- -1.9410749673843384,
- -2.3247878551483154,
- -1.523364543914795,
- -2.360647678375244,
- -1.708706021308899,
- -1.131014108657837,
- -2.944424867630005,
- -0.5273782014846802,
- -0.44912564754486084,
- -1.753378987312317,
- -0.8341047167778015,
- -0.4124295711517334,
- -0.9006240367889404,
- -1.4890273809432983,
- -0.4379286766052246,
- -1.6497018337249756,
- -0.5444425344467163,
- -1.2305881977081299,
- -1.164027214050293,
- -0.002498721005395055,
- -1.165798544883728,
- -0.007112303748726845,
- -0.718407154083252,
- -0.7442683577537537,
- -0.04299728572368622,
- -0.8688321113586426,
- -0.021008115261793137,
- -2.033963680267334,
- -1.2936673164367676,
- -0.78721684217453
+ -9.358942031860352,
+ -2.7132151126861572,
+ -4.606732368469238,
+ -1.4793059825897217,
+ -0.604263186454773,
+ -1.7374769449234009,
+ -2.485668897628784,
+ -2.1064839363098145,
+ -2.4603278636932373,
+ -6.253784656524658,
+ -1.4727367162704468,
+ -3.4053215980529785,
+ -4.36705207824707,
+ -3.8439993858337402,
+ -2.0021021366119385,
+ -1.8833506107330322,
+ -3.7835519313812256,
+ -6.891242980957031,
+ -0.28234225511550903,
+ -0.911859393119812,
+ -6.631955146789551,
+ -7.208620071411133,
+ -12.827497482299805,
+ -2.126032590866089,
+ -3.8147177696228027,
+ -0.5067541599273682,
+ -4.314828872680664,
+ -0.06301839649677277,
+ -0.10691610723733902,
+ -3.262773036956787,
+ -10.134418487548828,
+ -1.1751978397369385,
+ -6.014812469482422,
+ -5.020193576812744,
+ -3.8787002563476562,
+ -2.6112544536590576,
+ -3.366523027420044,
+ -5.561098098754883,
+ -1.622261643409729,
+ -5.453547477722168,
+ -12.218475341796875,
+ -12.583305358886719,
+ -0.09843693673610687,
+ -2.528902769088745,
+ -1.4074000120162964,
+ -2.8589088916778564,
+ -1.201108455657959,
+ -0.006660522893071175,
+ -3.3809216022491455,
+ -13.233884811401367,
+ -4.536578178405762,
+ -2.6009042263031006,
+ -6.00333309173584,
+ -0.7021315693855286,
+ -0.05108878016471863,
+ -1.5450650453567505,
+ -1.135668396949768,
+ -5.629673957824707,
+ -0.3985691964626312,
+ -4.960829257965088,
+ -0.5763677358627319,
+ -0.6644068360328674,
+ -2.4823904037475586,
+ -13.577603340148926,
+ -0.09277129173278809,
+ -3.5255157947540283,
+ -1.419959306716919,
+ -6.460269927978516,
+ -0.5874902009963989,
+ -3.569597005844116,
+ -0.8671818971633911,
+ -1.6106094121932983,
+ -5.3532209396362305,
+ -17.28781509399414,
+ -6.644278526306152,
+ -0.9023992419242859,
+ -4.112849235534668,
+ -1.1764109134674072,
+ -2.2248306274414062,
+ -1.7836185693740845,
+ -0.22097186744213104,
+ -9.294025421142578,
+ -0.13404981791973114,
+ -7.356222629547119,
+ -2.4790220260620117,
+ -4.13104248046875,
+ -3.6036787033081055,
+ -1.9332302808761597,
+ -2.328838348388672,
+ -2.2153103351593018,
+ -0.30630964040756226,
+ -1.4429736137390137,
+ -2.2041285037994385,
+ -1.1290805339813232,
+ -1.6353514194488525,
+ -1.4010272026062012,
+ -1.969598650932312,
+ -1.5041292905807495,
+ -0.8407909870147705,
+ -1.2370984554290771,
+ -1.9493464231491089,
+ -2.3773415088653564,
+ -1.624193787574768,
+ -0.9511104822158813,
+ -1.8184833526611328,
+ -2.394134521484375,
+ -1.2718405723571777,
+ -1.5075287818908691,
+ -0.5832712054252625,
+ -0.6682117581367493,
+ -0.5109573602676392,
+ -0.022901568561792374,
+ -0.483473539352417,
+ -0.2419981062412262,
+ -0.0857735350728035,
+ -0.13640964031219482,
+ -0.06942988932132721
]
}
}
diff --git a/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp2_pp2_dp2_583m_logitsmatch_zmq/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp2_pp2_dp2_583m_logitsmatch_zmq/model_config.yaml
index 13d409c5968..3b55b09e82e 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp2_pp2_dp2_583m_logitsmatch_zmq/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp2_pp2_dp2_583m_logitsmatch_zmq/model_config.yaml
@@ -23,6 +23,7 @@ MODEL_ARGS:
--distributed-backend: nccl
--log-interval: 1
--transformer-impl: inference_optimized
+ --inference-fuse-tp-communication: true
--sequence-parallel: true
--tensor-model-parallel-size: 2
--pipeline-model-parallel-size: 2
@@ -42,9 +43,6 @@ MODEL_ARGS:
--top_k: 1
--return-log-probs: true
--num-tokens-to-generate: 30
- --inference-dynamic-batching-max-requests-override: 8 # hardcode decode padding tokens to 7 for reproducibility
- --inference-dynamic-batching-buffer-guaranteed-fraction: 0
- --inference-dynamic-batching-buffer-overflow-factor: 0.2
--inference-dynamic-batching-buffer-size-gb: 20
--dist-ckpt-strictness: log_unexpected
--inference-ckpt-non-strict: true # To handle the extra_state errors
diff --git a/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp8_pp1_dp1_583m_logitsmatch_zmq/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp8_pp1_dp1_583m_logitsmatch_zmq/model_config.yaml
index 8d5779a5099..88a3e40a193 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp8_pp1_dp1_583m_logitsmatch_zmq/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt_dynamic_inference_tp8_pp1_dp1_583m_logitsmatch_zmq/model_config.yaml
@@ -51,9 +51,6 @@ MODEL_ARGS:
--incoming-requests-per-step: 32
--use-flashinfer-fused-rope: true
--inference-logging-step-interval: 1
- --cuda-graph-impl: local
- --inference-dynamic-batching-max-requests: 128
- --inference-dynamic-batching-num-cuda-graphs: 2
METRICS:
- "generated_tokens"
- "logprobs"
diff --git a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp2tp1_pp4pp2_dp8_583m_throughputtest/env_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_grpo_basic_function/env_config.yaml
similarity index 100%
rename from tests/functional_tests/test_cases/gpt/gpt_grpo_tp2tp1_pp4pp2_dp8_583m_throughputtest/env_config.yaml
rename to tests/functional_tests/test_cases/gpt/gpt_grpo_basic_function/env_config.yaml
diff --git a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp2tp1_pp4pp2_dp8_583m_throughputtest/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_grpo_basic_function/model_config.yaml
similarity index 58%
rename from tests/functional_tests/test_cases/gpt/gpt_grpo_tp2tp1_pp4pp2_dp8_583m_throughputtest/model_config.yaml
rename to tests/functional_tests/test_cases/gpt/gpt_grpo_basic_function/model_config.yaml
index b74417a898b..0143a39f017 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp2tp1_pp4pp2_dp8_583m_throughputtest/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt_grpo_basic_function/model_config.yaml
@@ -3,82 +3,97 @@ ENV_VARS:
NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
NCCL_ALGO: Ring
CUBLAS_WORKSPACE_CONFIG: :4096:8
+ SKIP_PYTEST: 1
TEST_TYPE: frozen-start
MODE: rl
MODEL_ARGS:
- --tiktoken-pattern: v2
- --use-mcore-models: true
- --tokenizer-type: TikTokenizer
- --tokenizer-model: ${CHECKPOINT_LOAD_PATH}/model/mcore_mistral/nemo_minitron-0.5b/v1/multiMixV8.gpt4o_nc_sd.500000.128k.vocab.json
- --load: ${CHECKPOINT_LOAD_PATH}/model/mcore_mistral/nemo_minitron-0.5b/v1/
- --auto-detect-ckpt-format: true
- --max-tokens-to-oom: 3600000
- --inference-max-seq-length: 1024
- --attention-backend: flash
- --mock-data: true
- --micro-batch-size: 1
- --no-load-optim: true
- --no-use-tokenizer-model-from-checkpoint-args: true
- --timing-log-level: 0
- --distributed-backend: nccl
- --log-interval: 1
- --log-progress: true
- --transformer-impl: transformer_engine
--tensor-model-parallel-size: 2
--pipeline-model-parallel-size: 4
+ --rl-inference-tensor-model-parallel-size: 1
+ --rl-inference-pipeline-model-parallel-size: 2
+ --refit-method: gloo
+ --inference-dynamic-batching-num-cuda-graphs: 1
+ --inference-dynamic-batching-unified-memory-level: 1
+ --inference-dynamic-batching-buffer-size-gb: 20
--ckpt-format: torch_dist
+ --seq-length: 1024
+ --inference-max-seq-length: 1024
+ --load: ${CHECKPOINT_LOAD_PATH}/model/qwen3-8b-dist
+ --untie-embeddings-and-output-weights: true
+ --num-layers: 36
+ --hidden-size: 4096
+ --ffn-hidden-size: 12288
+ --num-attention-heads: 32
+ --kv-channels: 128
+ --max-position-embeddings: 1024
+ --group-query-attention: true
+ --num-query-groups: 8
+ --normalization: RMSNorm
+ --norm-epsilon: 0.000001
+ --qk-layernorm: true
+ --position-embedding-type: rope
+ --rotary-percent: 1.0
+ --rotary-base: 1000000
+ --use-rotary-position-embeddings: true
+ --swiglu: true
+ --disable-bias-linear: true
+ --attention-dropout: 0.0
+ --hidden-dropout: 0.0
+ --no-masked-softmax-fusion: true
+ --attention-softmax-in-fp32: true
+ --tokenizer-type: HuggingFaceTokenizer
+ --tokenizer-model: ${CHECKPOINT_LOAD_PATH}/model/qwen3-8b-dist/tokenizer
+ --vocab-size: 151936
+ --make-vocab-size-divisible-by: 128
+ --optimizer: adam
+ --adam-beta1: 0.9
+ --adam-beta2: 0.999
+ --adam-eps: 0.00000001
+ --lr: 0.000001
+ --min-lr: 0.0000001
+ --lr-warmup-samples: 0
+ --clip-grad: 1.0
+ --weight-decay: 0.01
+ --use-mcore-models: true
--bf16: true
--log-memory-to-tensorboard: true
--log-num-zeros-in-grad: true
--log-validation-ppl-to-tensorboard: true
--log-timers-to-tensorboard: true
- --num-layers: 24
- --hidden-size: 1152
- --num-attention-heads: 16
- --max-position-embeddings: 1024
- --seq-length: 1024
--timing-log-option: minmax
--log-throughput: true
--no-create-attention-mask-in-dataloader: true
--straggler-minmax-count: 16
--tensorboard-log-interval: 1
+ --log-interval: 1
+ --log-progress: true
--empty-unused-memory-level: 2
- --langrl-inference-server-type: inplace_megatron
--seed: 42
--calculate-per-token-loss: true
--rl-use-sequence-packing: true
--rl-sequence-packing-algo: fifo
--rl-offload-optimizer-during-inference: true
--timing-log-level: 1
- --log-timers-to-tensorboard: true
--cuda-graph-impl: local
--micro-batch-size: 1
- --global-batch-size: 16
+ --global-batch-size: 4
--grpo-group-size: 2
- --grpo-prompts-per-step: 8
+ --grpo-prompts-per-step: 2
--grpo-iterations: 1
--grpo-clamp-eps-lower: 0.2
--grpo-clamp-eps-upper: 0.2
--grpo-kl-beta: 0.0
--grpo-entropy-term-weight: 0.0
- --langrl-env-config: tests/functional_tests/test_cases/gpt/gpt_grpo_tp1tp2_pp1_dp8_583m_throughputtest/env_config.yaml
+ --langrl-env-config: tests/functional_tests/test_cases/gpt/gpt_grpo_basic_function/env_config.yaml
--rl-partial-rollouts: true
- --lr: 0.000001
- --lr-warmup-samples: 0
- --clip-grad: 1.0
--use-checkpoint-args: true
--dist-ckpt-strictness: log_unexpected
--perform-rl-step: true
--train-samples: 48828125
- --exit-interval: 50
+ --exit-interval: 3
--tensorboard-dir: ${TENSORBOARD_PATH}
--save-interval: 1000000
--eval-interval: 1000000
--finetune: true
--inference-logging-step-interval: 1
- --rl-inference-tensor-model-parallel-size: 1
- --rl-inference-pipeline-model-parallel-size: 2
- --refit-method: gloo
-METRICS:
- - "mem-allocated-bytes"
- - "mem-max-allocated-bytes"
+METRICS: []
diff --git a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp1tp2_pp1_dp8_583m_throughputtest/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp1tp2_pp1_dp8_583m_throughputtest/model_config.yaml
index b12911358f0..4f9be214289 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp1tp2_pp1_dp8_583m_throughputtest/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp1tp2_pp1_dp8_583m_throughputtest/model_config.yaml
@@ -43,7 +43,6 @@ MODEL_ARGS:
--straggler-minmax-count: 16
--tensorboard-log-interval: 1
--empty-unused-memory-level: 2
- --langrl-inference-server-type: inplace_megatron
--seed: 42
--calculate-per-token-loss: true
--rl-use-sequence-packing: true
diff --git a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp1tp2_pp1_dp8_583m_throughputtest_github/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp1tp2_pp1_dp8_583m_throughputtest_github/model_config.yaml
index bff55aea7fe..c8fa19d0500 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp1tp2_pp1_dp8_583m_throughputtest_github/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp1tp2_pp1_dp8_583m_throughputtest_github/model_config.yaml
@@ -43,7 +43,6 @@ MODEL_ARGS:
--straggler-minmax-count: 16
--tensorboard-log-interval: 1
--empty-unused-memory-level: 2
- --langrl-inference-server-type: inplace_megatron
--seed: 42
--calculate-per-token-loss: true
--rl-use-sequence-packing: true
@@ -78,3 +77,13 @@ MODEL_ARGS:
--inference-logging-step-interval: 1
--rl-inference-tensor-model-parallel-size: 2
--refit-method: gloo
+
+METRICS:
+ - "iteration-time"
+ - "lm loss"
+ - "num-zeros"
+ - "mem-allocated-bytes"
+ - "mem-max-allocated-bytes"
+
+THROUGHPUT_TEST_PARAMS:
+ --start_step: 1
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp2tp1_pp4pp2_dp8_583m_throughputtest/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp2tp1_pp4pp2_dp8_583m_throughputtest/golden_values_dev_dgx_h100.json
deleted file mode 100644
index d985f671cab..00000000000
--- a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp2tp1_pp4pp2_dp8_583m_throughputtest/golden_values_dev_dgx_h100.json
+++ /dev/null
@@ -1,173 +0,0 @@
-{
- "mem-allocated-bytes": {
- "start_step": 1,
- "end_step": 50,
- "step_interval": 1,
- "values": {
- "1": 48967716864.0,
- "2": 48973631488.0,
- "3": 48974528512.0,
- "4": 48971538432.0,
- "5": 48974340096.0,
- "6": 48974143488.0,
- "7": 48977002496.0,
- "8": 48975851520.0,
- "9": 48974036992.0,
- "10": 48973709312.0,
- "11": 48973262848.0,
- "12": 48973705216.0,
- "13": 48973598720.0,
- "14": 48976703488.0,
- "15": 48975118336.0,
- "16": 48977072128.0,
- "17": 48976465920.0,
- "18": 48976470016.0,
- "19": 48976478208.0,
- "20": 48976654336.0,
- "21": 48976793600.0,
- "22": 48976052224.0,
- "23": 48976277504.0,
- "24": 48974708736.0,
- "25": 48973062144.0,
- "26": 48976236544.0,
- "27": 48975970304.0,
- "28": 48976711680.0,
- "29": 48975593472.0,
- "30": 48977321984.0,
- "31": 48977506304.0,
- "32": 48976646144.0,
- "33": 48976072704.0,
- "34": 48973631488.0,
- "35": 48976650240.0,
- "36": 48975650816.0,
- "37": 48974950400.0,
- "38": 48972750848.0,
- "39": 48976617472.0,
- "40": 48979308544.0,
- "41": 48978587648.0,
- "42": 48975626240.0,
- "43": 48975089664.0,
- "44": 48973688832.0,
- "45": 48975327232.0,
- "46": 48975159296.0,
- "47": 48975372288.0,
- "48": 48973856768.0,
- "49": 48973377536.0,
- "50": 48975568896.0
- }
- },
- "mem-max-allocated-bytes": {
- "start_step": 1,
- "end_step": 50,
- "step_interval": 1,
- "values": {
- "1": 49090379776.0,
- "2": 49937022976.0,
- "3": 49938366464.0,
- "4": 49938366464.0,
- "5": 49938366464.0,
- "6": 49938698240.0,
- "7": 49939156992.0,
- "8": 49939156992.0,
- "9": 49939156992.0,
- "10": 49939156992.0,
- "11": 49939156992.0,
- "12": 49939156992.0,
- "13": 49939156992.0,
- "14": 49940287488.0,
- "15": 49940287488.0,
- "16": 49940287488.0,
- "17": 49941729280.0,
- "18": 49941733376.0,
- "19": 49941741568.0,
- "20": 49941778432.0,
- "21": 49941778432.0,
- "22": 49941778432.0,
- "23": 49941778432.0,
- "24": 49941778432.0,
- "25": 49941778432.0,
- "26": 49941778432.0,
- "27": 49941934080.0,
- "28": 49941934080.0,
- "29": 49941934080.0,
- "30": 49941934080.0,
- "31": 49942675456.0,
- "32": 49942675456.0,
- "33": 49942675456.0,
- "34": 49942675456.0,
- "35": 49942675456.0,
- "36": 49942675456.0,
- "37": 49942675456.0,
- "38": 49942675456.0,
- "39": 49942675456.0,
- "40": 49944379392.0,
- "41": 49944379392.0,
- "42": 49944379392.0,
- "43": 49944379392.0,
- "44": 49944379392.0,
- "45": 49944379392.0,
- "46": 49944379392.0,
- "47": 49944379392.0,
- "48": 49944379392.0,
- "49": 49944379392.0,
- "50": 49944379392.0
- }
- },
- "iteration-time": {
- "start_step": 1,
- "end_step": 50,
- "step_interval": 1,
- "values": {
- "1": 63.07516,
- "2": 4.36236,
- "3": 3.83222,
- "4": 3.85784,
- "5": 3.74494,
- "6": 3.82661,
- "7": 4.05458,
- "8": 3.76622,
- "9": 3.90518,
- "10": 4.09283,
- "11": 3.96358,
- "12": 3.85778,
- "13": 3.84546,
- "14": 3.85497,
- "15": 4.35749,
- "16": 3.7861,
- "17": 3.8896,
- "18": 3.6267,
- "19": 3.76463,
- "20": 3.6953,
- "21": 3.63427,
- "22": 3.66652,
- "23": 3.60379,
- "24": 3.57701,
- "25": 3.57327,
- "26": 3.71371,
- "27": 3.69626,
- "28": 3.89285,
- "29": 3.62405,
- "30": 3.58297,
- "31": 3.56993,
- "32": 3.75257,
- "33": 3.72279,
- "34": 3.48095,
- "35": 3.60831,
- "36": 3.74971,
- "37": 3.72155,
- "38": 3.51054,
- "39": 3.64562,
- "40": 3.66038,
- "41": 3.86018,
- "42": 3.58341,
- "43": 3.82647,
- "44": 3.85728,
- "45": 3.62416,
- "46": 3.59141,
- "47": 3.74512,
- "48": 3.61762,
- "49": 3.57079,
- "50": 3.66209
- }
- }
-}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_cudagraphs_throughput/env_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_cudagraphs_throughput/env_config.yaml
new file mode 100644
index 00000000000..329246987bf
--- /dev/null
+++ b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_cudagraphs_throughput/env_config.yaml
@@ -0,0 +1,5 @@
+- agent_type: examples.rl.environments.countdown.countdown_agent.CountdownAgent
+ agent_args:
+ dataset_file: "/mnt/artifacts/rl_environments/Jiayi-Pan___countdown-tasks-3to4"
+ split: "train"
+ weight: 1.0
diff --git a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_cudagraphs_throughput/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_cudagraphs_throughput/golden_values_dev_dgx_h100.json
new file mode 100644
index 00000000000..52a17a100e8
--- /dev/null
+++ b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_cudagraphs_throughput/golden_values_dev_dgx_h100.json
@@ -0,0 +1,137 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 20,
+ "step_interval": 1,
+ "values": {
+ "1": 0.0,
+ "2": 0.0,
+ "3": 0.0,
+ "4": 0.0,
+ "5": 0.0,
+ "6": 0.0,
+ "7": 0.0,
+ "8": 0.0,
+ "9": 0.0,
+ "10": 0.0,
+ "11": 0.0,
+ "12": 0.0,
+ "13": 0.0,
+ "14": 0.0,
+ "15": 0.0,
+ "16": 0.0,
+ "17": 0.0,
+ "18": 0.0,
+ "19": 0.0,
+ "20": 0.0
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 20,
+ "step_interval": 1,
+ "values": {
+ "1": 8191783936.0,
+ "2": 8191783936.0,
+ "3": 8191783936.0,
+ "4": 8191783936.0,
+ "5": 8191783936.0,
+ "6": 8191783936.0,
+ "7": 8191783936.0,
+ "8": 8191783936.0,
+ "9": 8191783936.0,
+ "10": 8191783936.0,
+ "11": 8191783936.0,
+ "12": 8191783936.0,
+ "13": 8191783936.0,
+ "14": 8191783936.0,
+ "15": 8191783936.0,
+ "16": 8191783936.0,
+ "17": 8191783936.0,
+ "18": 8191783936.0,
+ "19": 8191783936.0,
+ "20": 8191783936.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 20,
+ "step_interval": 1,
+ "values": {
+ "1": 61233643520.0,
+ "2": 61300752384.0,
+ "3": 61300748288.0,
+ "4": 61300748288.0,
+ "5": 61300748288.0,
+ "6": 61300744192.0,
+ "7": 61300748288.0,
+ "8": 61300744192.0,
+ "9": 61300744192.0,
+ "10": 61300740096.0,
+ "11": 61300744192.0,
+ "12": 61300744192.0,
+ "13": 61300744192.0,
+ "14": 61300744192.0,
+ "15": 61300744192.0,
+ "16": 61300744192.0,
+ "17": 61300744192.0,
+ "18": 61300744192.0,
+ "19": 61300744192.0,
+ "20": 61300740096.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 20,
+ "step_interval": 1,
+ "values": {
+ "1": 61233647616.0,
+ "2": 62702190592.0,
+ "3": 62702190592.0,
+ "4": 62702190592.0,
+ "5": 62702190592.0,
+ "6": 62702190592.0,
+ "7": 62702190592.0,
+ "8": 62702190592.0,
+ "9": 62702190592.0,
+ "10": 62702190592.0,
+ "11": 62702190592.0,
+ "12": 62702190592.0,
+ "13": 62702190592.0,
+ "14": 62702190592.0,
+ "15": 62702190592.0,
+ "16": 62702190592.0,
+ "17": 62702190592.0,
+ "18": 62702190592.0,
+ "19": 62702190592.0,
+ "20": 62702190592.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 20,
+ "step_interval": 1,
+ "values": {
+ "1": "nan",
+ "2": 40.27668,
+ "3": 12.87449,
+ "4": 12.1227,
+ "5": 11.78624,
+ "6": 11.58806,
+ "7": 11.59093,
+ "8": 11.38792,
+ "9": 11.83294,
+ "10": 11.93856,
+ "11": 11.70984,
+ "12": 11.68329,
+ "13": 11.59815,
+ "14": 12.16894,
+ "15": 11.60602,
+ "16": 11.73938,
+ "17": 12.29221,
+ "18": 11.44932,
+ "19": 11.77467,
+ "20": 11.69175
+ }
+ }
+}
diff --git a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_cudagraphs_throughput/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_cudagraphs_throughput/model_config.yaml
new file mode 100644
index 00000000000..f3c0c4ecc5b
--- /dev/null
+++ b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_cudagraphs_throughput/model_config.yaml
@@ -0,0 +1,101 @@
+ENV_VARS:
+ CUDA_DEVICE_MAX_CONNECTIONS: 1
+ THROUGHPUT_START_STEP: 10
+ NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
+ NCCL_ALGO: Ring
+ CUBLAS_WORKSPACE_CONFIG: :4096:8
+ N_REPEAT: 1
+TEST_TYPE: frozen-start
+MODE: rl
+MODEL_ARGS:
+ --tensor-model-parallel-size: 4
+ --inference-dynamic-batching-num-cuda-graphs: 1
+ --inference-dynamic-batching-unified-memory-level: 1
+ --inference-dynamic-batching-buffer-size-gb: 20
+ --ckpt-format: torch_dist
+ --seq-length: 1024
+ --inference-max-seq-length: 1024
+ --load: ${CHECKPOINT_LOAD_PATH}/model/qwen3-8b-dist
+ --untie-embeddings-and-output-weights: true
+ --num-layers: 36
+ --hidden-size: 4096
+ --ffn-hidden-size: 12288
+ --num-attention-heads: 32
+ --kv-channels: 128
+ --max-position-embeddings: 1024
+ --group-query-attention: true
+ --num-query-groups: 8
+ --normalization: RMSNorm
+ --norm-epsilon: 0.000001
+ --qk-layernorm: true
+ --position-embedding-type: rope
+ --rotary-percent: 1.0
+ --rotary-base: 1000000
+ --use-rotary-position-embeddings: true
+ --swiglu: true
+ --disable-bias-linear: true
+ --attention-dropout: 0.0
+ --hidden-dropout: 0.0
+ --no-masked-softmax-fusion: true
+ --attention-softmax-in-fp32: true
+ --tokenizer-type: HuggingFaceTokenizer
+ --tokenizer-model: ${CHECKPOINT_LOAD_PATH}/model/qwen3-8b-dist/tokenizer
+ --vocab-size: 151936
+ --make-vocab-size-divisible-by: 128
+ --optimizer: adam
+ --adam-beta1: 0.9
+ --adam-beta2: 0.999
+ --adam-eps: 0.00000001
+ --lr: 0.000001
+ --min-lr: 0.0000001
+ --lr-warmup-samples: 0
+ --clip-grad: 1.0
+ --weight-decay: 0.01
+ --deterministic-mode: true
+ --use-mcore-models: true
+ --bf16: true
+ --log-memory-to-tensorboard: true
+ --log-num-zeros-in-grad: true
+ --log-validation-ppl-to-tensorboard: true
+ --log-timers-to-tensorboard: true
+ --timing-log-option: minmax
+ --log-throughput: true
+ --no-create-attention-mask-in-dataloader: true
+ --straggler-minmax-count: 16
+ --tensorboard-log-interval: 1
+ --log-interval: 1
+ --log-progress: true
+ --empty-unused-memory-level: 2
+ --seed: 42
+ --calculate-per-token-loss: true
+ --rl-use-sequence-packing: true
+ --rl-sequence-packing-algo: fifo
+ --rl-offload-optimizer-during-inference: true
+ --timing-log-level: 1
+ --cuda-graph-impl: local
+ --micro-batch-size: 1
+ --global-batch-size: 4
+ --grpo-group-size: 2
+ --grpo-prompts-per-step: 2
+ --grpo-iterations: 1
+ --grpo-clamp-eps-lower: 0.2
+ --grpo-clamp-eps-upper: 0.2
+ --grpo-kl-beta: 0.0
+ --grpo-entropy-term-weight: 0.0
+ --langrl-env-config: tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput/env_config.yaml
+ --rl-partial-rollouts: true
+ --use-checkpoint-args: true
+ --dist-ckpt-strictness: log_unexpected
+ --perform-rl-step: true
+ --train-samples: 48828125
+ --exit-interval: 20
+ --tensorboard-dir: ${TENSORBOARD_PATH}
+ --save-interval: 1000000
+ --eval-interval: 1000000
+ --finetune: true
+ --inference-logging-step-interval: 1
+ --rl-training-cuda-graphs: true
+METRICS:
+ - "mem-allocated-bytes"
+ - "mem-max-allocated-bytes"
+ - "iteration-time"
diff --git a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput/golden_values_dev_dgx_h100.json
index 42c13292446..3cfbff2350d 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput/golden_values_dev_dgx_h100.json
@@ -1,29 +1,83 @@
{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 20,
+ "step_interval": 1,
+ "values": {
+ "1": 0.0,
+ "2": 0.0,
+ "3": 0.0,
+ "4": 0.0,
+ "5": 0.0,
+ "6": 0.0,
+ "7": 0.0,
+ "8": 0.0,
+ "9": 0.0,
+ "10": 0.0,
+ "11": 0.0,
+ "12": 0.0,
+ "13": 0.0,
+ "14": 0.0,
+ "15": 0.0,
+ "16": 0.0,
+ "17": 0.0,
+ "18": 0.0,
+ "19": 0.0,
+ "20": 0.0
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 20,
+ "step_interval": 1,
+ "values": {
+ "1": 8191783936.0,
+ "2": 8191783936.0,
+ "3": 8191783936.0,
+ "4": 8191783936.0,
+ "5": 8191783936.0,
+ "6": 8191783936.0,
+ "7": 8191783936.0,
+ "8": 8191783936.0,
+ "9": 8191783936.0,
+ "10": 8191783936.0,
+ "11": 8191783936.0,
+ "12": 8191783936.0,
+ "13": 8191783936.0,
+ "14": 8191783936.0,
+ "15": 8191783936.0,
+ "16": 8191783936.0,
+ "17": 8191783936.0,
+ "18": 8191783936.0,
+ "19": 8191783936.0,
+ "20": 8191783936.0
+ }
+ },
"mem-allocated-bytes": {
"start_step": 1,
"end_step": 20,
"step_interval": 1,
"values": {
- "1": 60922068992.0,
- "2": 60922068992.0,
- "3": 60922331136.0,
- "4": 60922073088.0,
- "5": 60922331136.0,
- "6": 60922064896.0,
- "7": 60922331136.0,
- "8": 60922064896.0,
- "9": 60922322944.0,
- "10": 60922052608.0,
- "11": 60922056704.0,
- "12": 60922318848.0,
- "13": 60922056704.0,
- "14": 60922318848.0,
- "15": 60922056704.0,
- "16": 60922310656.0,
- "17": 60922052608.0,
- "18": 60922052608.0,
- "19": 60922048512.0,
- "20": 60922044416.0
+ "1": 60921389056.0,
+ "2": 60922458112.0,
+ "3": 60922458112.0,
+ "4": 60922458112.0,
+ "5": 60922441728.0,
+ "6": 60922445824.0,
+ "7": 60922437632.0,
+ "8": 60922437632.0,
+ "9": 60922429440.0,
+ "10": 60922429440.0,
+ "11": 60922429440.0,
+ "12": 60922425344.0,
+ "13": 60922425344.0,
+ "14": 60922425344.0,
+ "15": 60922421248.0,
+ "16": 60922425344.0,
+ "17": 60922421248.0,
+ "18": 60922421248.0,
+ "19": 60922421248.0,
+ "20": 60922421248.0
}
},
"mem-max-allocated-bytes": {
@@ -31,26 +85,26 @@
"end_step": 20,
"step_interval": 1,
"values": {
- "1": 60922073088.0,
- "2": 64156037120.0,
- "3": 64156037120.0,
- "4": 64156041216.0,
- "5": 64156041216.0,
- "6": 64156041216.0,
- "7": 64156041216.0,
- "8": 64156041216.0,
- "9": 64156041216.0,
- "10": 64156041216.0,
- "11": 64156041216.0,
- "12": 64156041216.0,
- "13": 64156041216.0,
- "14": 64156041216.0,
- "15": 64156041216.0,
- "16": 64156041216.0,
- "17": 64156041216.0,
- "18": 64156041216.0,
- "19": 64156041216.0,
- "20": 64156041216.0
+ "1": 60921389056.0,
+ "2": 64156164096.0,
+ "3": 64156164096.0,
+ "4": 64156164096.0,
+ "5": 64156164096.0,
+ "6": 64156164096.0,
+ "7": 64156164096.0,
+ "8": 64156164096.0,
+ "9": 64156164096.0,
+ "10": 64156164096.0,
+ "11": 64156164096.0,
+ "12": 64156164096.0,
+ "13": 64156164096.0,
+ "14": 64156164096.0,
+ "15": 64156164096.0,
+ "16": 64156164096.0,
+ "17": 64156164096.0,
+ "18": 64156164096.0,
+ "19": 64156164096.0,
+ "20": 64156164096.0
}
},
"iteration-time": {
@@ -59,25 +113,25 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 60.37194,
- "3": 13.25967,
- "4": 13.01461,
- "5": 14.04256,
- "6": 13.53259,
- "7": 13.3335,
- "8": 12.72344,
- "9": 13.64787,
- "10": 12.66485,
- "11": 13.15779,
- "12": 13.01275,
- "13": 12.72481,
- "14": 12.67697,
- "15": 12.7286,
- "16": 12.65032,
- "17": 12.86279,
- "18": 12.71745,
- "19": 13.4137,
- "20": 12.75566
+ "2": 37.71912,
+ "3": 14.70783,
+ "4": 12.33161,
+ "5": 13.75233,
+ "6": 12.28209,
+ "7": 13.06314,
+ "8": 12.63603,
+ "9": 12.2888,
+ "10": 13.30266,
+ "11": 12.10042,
+ "12": 12.228,
+ "13": 12.68127,
+ "14": 12.21672,
+ "15": 12.30011,
+ "16": 12.1762,
+ "17": 12.13298,
+ "18": 12.25718,
+ "19": 12.25184,
+ "20": 12.27994
}
}
}
diff --git a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput/model_config.yaml
index 3037e2e0803..80664dcdc59 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput/model_config.yaml
@@ -1,5 +1,6 @@
ENV_VARS:
CUDA_DEVICE_MAX_CONNECTIONS: 1
+ THROUGHPUT_START_STEP: 10
NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
NCCL_ALGO: Ring
CUBLAS_WORKSPACE_CONFIG: :4096:8
@@ -39,8 +40,6 @@ MODEL_ARGS:
--attention-softmax-in-fp32: true
--tokenizer-type: HuggingFaceTokenizer
--tokenizer-model: ${CHECKPOINT_LOAD_PATH}/model/qwen3-8b-dist/tokenizer
- --langrl-inference-server-type: inplace_megatron_chat
- --langrl-inference-server-conversation-template: ${CHECKPOINT_LOAD_PATH}/model/qwen3-8b-dist/tokenizer
--vocab-size: 151936
--make-vocab-size-divisible-by: 128
--optimizer: adam
@@ -99,5 +98,3 @@ METRICS:
- "mem-allocated-bytes"
- "mem-max-allocated-bytes"
- "iteration-time"
-THROUGHPUT_TEST_PARAMS:
- --start_step: 10
diff --git a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput_github/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput_github/golden_values_dev_dgx_h100.json
index e0bcb14f29b..d6397eb061c 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput_github/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput_github/golden_values_dev_dgx_h100.json
@@ -4,26 +4,26 @@
"end_step": 20,
"step_interval": 1,
"values": {
- "1": 60922077184.0,
- "2": 60922073088.0,
- "3": 60922068992.0,
- "4": 60922073088.0,
- "5": 60922331136.0,
- "6": 60922073088.0,
- "7": 60922335232.0,
- "8": 60922068992.0,
- "9": 60922073088.0,
- "10": 60922073088.0,
- "11": 60922077184.0,
- "12": 60922093568.0,
- "13": 60922351616.0,
- "14": 60922085376.0,
- "15": 60922085376.0,
- "16": 60922085376.0,
- "17": 60922089472.0,
- "18": 60922085376.0,
- "19": 60922085376.0,
- "20": 60922089472.0
+ "1": 60921806848.0,
+ "2": 60921806848.0,
+ "3": 60921794560.0,
+ "4": 60921790464.0,
+ "5": 60921782272.0,
+ "6": 60921774080.0,
+ "7": 60921769984.0,
+ "8": 60921765888.0,
+ "9": 60921749504.0,
+ "10": 60921331712.0,
+ "11": 60921331712.0,
+ "12": 60921335808.0,
+ "13": 60921335808.0,
+ "14": 60921339904.0,
+ "15": 60921339904.0,
+ "16": 60921339904.0,
+ "17": 60921344000.0,
+ "18": 60921344000.0,
+ "19": 60921344000.0,
+ "20": 60921344000.0
}
},
"mem-max-allocated-bytes": {
@@ -31,26 +31,26 @@
"end_step": 20,
"step_interval": 1,
"values": {
- "1": 60922081280.0,
- "2": 64156041216.0,
- "3": 64156041216.0,
- "4": 64156041216.0,
- "5": 64156041216.0,
- "6": 64156041216.0,
- "7": 64156041216.0,
- "8": 64156041216.0,
- "9": 64156041216.0,
- "10": 64156041216.0,
- "11": 64156045312.0,
- "12": 64156061696.0,
- "13": 64156061696.0,
- "14": 64156061696.0,
- "15": 64156061696.0,
- "16": 64156061696.0,
- "17": 64156061696.0,
- "18": 64156061696.0,
- "19": 64156061696.0,
- "20": 64156061696.0
+ "1": 60921810944.0,
+ "2": 64155770880.0,
+ "3": 64155770880.0,
+ "4": 64155770880.0,
+ "5": 64155770880.0,
+ "6": 64155770880.0,
+ "7": 64155770880.0,
+ "8": 64155770880.0,
+ "9": 64155770880.0,
+ "10": 64155770880.0,
+ "11": 64155770880.0,
+ "12": 64155770880.0,
+ "13": 64155770880.0,
+ "14": 64155770880.0,
+ "15": 64155770880.0,
+ "16": 64155770880.0,
+ "17": 64155770880.0,
+ "18": 64155770880.0,
+ "19": 64155770880.0,
+ "20": 64155770880.0
}
},
"iteration-time": {
@@ -59,25 +59,25 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 110.96005,
- "3": 49.76537,
- "4": 46.36583,
- "5": 46.63055,
- "6": 50.62966,
- "7": 46.52987,
- "8": 44.32473,
- "9": 46.39563,
- "10": 44.4327,
- "11": 43.93223,
- "12": 46.84642,
- "13": 43.45953,
- "14": 42.21466,
- "15": 42.70466,
- "16": 42.45673,
- "17": 43.68298,
- "18": 41.36069,
- "19": 42.64788,
- "20": 45.08387
+ "2": 85.15953,
+ "3": 36.37596,
+ "4": 34.46927,
+ "5": 35.70767,
+ "6": 36.40207,
+ "7": 36.7311,
+ "8": 36.84224,
+ "9": 36.35987,
+ "10": 37.55609,
+ "11": 35.2332,
+ "12": 34.93421,
+ "13": 34.76739,
+ "14": 34.73478,
+ "15": 34.23211,
+ "16": 35.30824,
+ "17": 35.10893,
+ "18": 35.80016,
+ "19": 37.51863,
+ "20": 36.1209
}
}
}
diff --git a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput_github/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput_github/model_config.yaml
index 456280fdb04..cc25f3ab90e 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput_github/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt_grpo_tp4_pp1_dp2_8b_throughput_github/model_config.yaml
@@ -1,5 +1,6 @@
ENV_VARS:
CUDA_DEVICE_MAX_CONNECTIONS: 1
+ THROUGHPUT_START_STEP: 10
NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
NCCL_ALGO: Ring
CUBLAS_WORKSPACE_CONFIG: :4096:8
@@ -39,8 +40,6 @@ MODEL_ARGS:
--attention-softmax-in-fp32: true
--tokenizer-type: HuggingFaceTokenizer
--tokenizer-model: ${CHECKPOINT_LOAD_PATH}/model/qwen3-8b-dist/tokenizer
- --langrl-inference-server-type: inplace_megatron_chat
- --langrl-inference-server-conversation-template: ${CHECKPOINT_LOAD_PATH}/model/qwen3-8b-dist/tokenizer
--vocab-size: 151936
--make-vocab-size-divisible-by: 128
--optimizer: adam
@@ -99,5 +98,3 @@ METRICS:
- "mem-allocated-bytes"
- "mem-max-allocated-bytes"
- "iteration-time"
-THROUGHPUT_TEST_PARAMS:
- --start_step: 10
diff --git a/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_16b_multiprompt_tokensmatch/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_16b_multiprompt_tokensmatch/model_config.yaml
index 6d63b0e4228..40b45024cb1 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_16b_multiprompt_tokensmatch/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_16b_multiprompt_tokensmatch/model_config.yaml
@@ -76,6 +76,7 @@ MODEL_ARGS:
--max-tokens-to-oom: 3600000
--inference-max-seq-length: 4096
--output-path: ${INFERENCE_OUTPUT_PATH}
+ --use-legacy-static-engine: true
--prompt-file: ./tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_16b_multiprompt_tokensmatch/test_prompts.jsonl
--incoming-requests-per-sec: -1 # all requests arrive up front.
--inference-logging-step-interval: 1
diff --git a/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_583m_cudagraphs/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_583m_cudagraphs/model_config.yaml
index 8f54fff0a2f..9a47281703a 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_583m_cudagraphs/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_583m_cudagraphs/model_config.yaml
@@ -48,6 +48,7 @@ MODEL_ARGS:
--inference-max-requests: 1
--dist-ckpt-strictness: log_unexpected
--output-path: ${INFERENCE_OUTPUT_PATH}
+ --use-legacy-static-engine: true
--prompts: "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies."
--incoming-requests-per-sec: -1 # all requests arrive up front.
--inference-logging-step-interval: 1
diff --git a/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs/model_config.yaml
index 1a1195baa2b..99bcc433ad1 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs/model_config.yaml
@@ -53,6 +53,7 @@ MODEL_ARGS:
--inference-max-requests: 1
--dist-ckpt-strictness: log_unexpected
--output-path: ${INFERENCE_OUTPUT_PATH}
+ --use-legacy-static-engine: true
--prompts: "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies."
--incoming-requests-per-sec: -1 # all requests arrive up front.
--inference-logging-step-interval: 1
diff --git a/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_583m_logitsmatch/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_583m_logitsmatch/model_config.yaml
index be00e4b3ce7..1c78b466b1e 100644
--- a/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_583m_logitsmatch/model_config.yaml
+++ b/tests/functional_tests/test_cases/gpt/gpt_static_inference_tp1_pp1_583m_logitsmatch/model_config.yaml
@@ -44,6 +44,7 @@ MODEL_ARGS:
--flash-decode: true
--dist-ckpt-strictness: log_unexpected
--output-path: ${INFERENCE_OUTPUT_PATH}
+ --use-legacy-static-engine: true
--prompts: "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies."
--incoming-requests-per-sec: -1 # all requests arrive up front.
METRICS:
diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m/golden_values_dev_dgx_h100.json
index 2a284c160aa..2ddc7c66678 100644
--- a/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m/golden_values_dev_dgx_h100.json
@@ -1,28 +1,166 @@
{
"0": {
"input_prompt": "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies.",
- "generated_text": " Then, when you're ready, go home and watch the movie again.",
+ "generated_text": " It is not. It is a place where you can get a job as a bartender, and where you can get a job as a writer",
"generated_tokens": [
- 6830,
- 1044,
- 2200,
+ 2157,
+ 1395,
+ 1605,
+ 1046,
+ 2157,
+ 1395,
+ 1261,
+ 3535,
+ 2478,
1636,
- 6185,
- 11831,
+ 1710,
+ 2012,
+ 1261,
+ 6854,
+ 1435,
+ 1261,
+ 1289,
+ 1490,
+ 3206,
1044,
- 1974,
- 4590,
1321,
- 9951,
- 1278,
- 16070,
- 2790,
- 1046,
- 2
+ 2478,
+ 1636,
+ 1710,
+ 2012,
+ 1261,
+ 6854,
+ 1435,
+ 1261,
+ 15353
],
- "latency": 22.701347589492798,
+ "latency": 1.906998872756958,
+ "ttft": 0.08748054504394531,
"cuda_graph_request_count_map": null,
- "step_count": 16,
+ "step_count": 30,
+ "top_n_logprobs": null,
+ "prompt_top_n_logprobs": null,
+ "prompt_logprobs": [
+ -9.498085021972656,
+ -3.787536859512329,
+ -3.0404648780822754,
+ -1.7445809841156006,
+ -0.29672086238861084,
+ -1.3661342859268188,
+ -2.3458175659179688,
+ -1.83931303024292,
+ -1.4894113540649414,
+ -6.440437316894531,
+ -0.8176816701889038,
+ -1.790361762046814,
+ -3.662419557571411,
+ -3.7036173343658447,
+ -1.6009434461593628,
+ -1.5501081943511963,
+ -2.846059799194336,
+ -6.732302665710449,
+ -0.06605878472328186,
+ -1.334327220916748,
+ -6.083745002746582,
+ -9.440131187438965,
+ -10.473882675170898,
+ -1.5964821577072144,
+ -4.702763557434082,
+ -0.7514524459838867,
+ -2.1461901664733887,
+ -0.012340382672846317,
+ -0.03605639934539795,
+ -3.0907557010650635,
+ -8.744739532470703,
+ -1.5410845279693604,
+ -5.84979772567749,
+ -3.0918972492218018,
+ -3.9814329147338867,
+ -3.78017520904541,
+ -2.5227086544036865,
+ -2.258594036102295,
+ -0.4719255566596985,
+ -1.0329649448394775,
+ -5.3284382820129395,
+ -8.25335693359375,
+ -0.015789249911904335,
+ -2.854100227355957,
+ -1.2236379384994507,
+ -3.905193328857422,
+ -0.9268187284469604,
+ -0.0030202509369701147,
+ -3.224249839782715,
+ -11.11172103881836,
+ -3.8121743202209473,
+ -2.3400487899780273,
+ -4.672845363616943,
+ -0.09729652851819992,
+ -0.06232408434152603,
+ -1.336004614830017,
+ -2.054157257080078,
+ -4.390933036804199,
+ -0.44248226284980774,
+ -3.9417736530303955,
+ -0.5888474583625793,
+ -0.26697415113449097,
+ -2.9271092414855957,
+ -13.515066146850586,
+ -0.10294333100318909,
+ -3.5007452964782715,
+ -0.8535972237586975,
+ -5.173652648925781,
+ -0.330394983291626,
+ -2.304553508758545,
+ -0.5418462753295898,
+ -1.300589919090271,
+ -4.9136152267456055,
+ -15.558022499084473,
+ -4.918652534484863,
+ -0.22206512093544006,
+ -6.589188575744629,
+ -0.9015690684318542,
+ -2.2228457927703857,
+ -1.8689247369766235,
+ -0.2006368339061737,
+ -5.918689727783203,
+ -0.006355076562613249,
+ -7.532094955444336,
+ -3.2708187103271484,
+ -3.743263006210327,
+ -2.011824131011963
+ ],
+ "generated_logprobs": [
+ -3.0331737995147705,
+ -1.2233026027679443,
+ -1.6082642078399658,
+ -0.9534461498260498,
+ -1.4570012092590332,
+ -0.29605966806411743,
+ -1.8120722770690918,
+ -2.9380805492401123,
+ -0.8330938220024109,
+ -1.776170253753662,
+ -1.1607933044433594,
+ -2.3149070739746094,
+ -2.0255284309387207,
+ -2.521214723587036,
+ -1.7217901945114136,
+ -0.1730123609304428,
+ -2.888899564743042,
+ -0.2459663301706314,
+ -0.0008181562297977507,
+ -1.4219350814819336,
+ -1.5678833723068237,
+ -2.491905927658081,
+ -0.8186907172203064,
+ -0.37120482325553894,
+ -2.210369110107422,
+ -1.2439723014831543,
+ -1.019776463508606,
+ -0.44152167439460754,
+ -0.20805038511753082,
+ -2.262563943862915
+ ],
"logprobs": [
-9.498085021972656,
-3.787536859512329,
@@ -36,97 +174,113 @@
-6.440437316894531,
-0.8176816701889038,
-1.790361762046814,
- -3.6521127223968506,
- -3.7014482021331787,
- -1.5858951807022095,
- -1.5492421388626099,
- -2.844204902648926,
- -6.694585800170898,
- -0.06552714854478836,
- -1.333437204360962,
- -6.077418327331543,
- -9.448220252990723,
- -10.46927261352539,
- -1.4987666606903076,
- -4.727880001068115,
- -0.7596290111541748,
- -2.152517795562744,
- -0.013758113607764244,
- -0.040566492825746536,
- -3.1010313034057617,
- -8.735280990600586,
- -1.5446771383285522,
- -5.841436862945557,
- -3.0970406532287598,
- -4.0269670486450195,
- -3.769413948059082,
- -2.466399669647217,
- -2.3482255935668945,
- -0.47234833240509033,
- -1.114174723625183,
- -5.310229778289795,
- -8.236719131469727,
- -0.015452657826244831,
- -2.854970932006836,
- -1.2198810577392578,
- -3.923705577850342,
- -0.9644856452941895,
- -0.0026721982285380363,
- -3.096668243408203,
- -11.110801696777344,
- -3.688267230987549,
- -2.3297765254974365,
- -4.670788764953613,
- -0.09854680299758911,
- -0.06234245002269745,
- -1.3255000114440918,
- -2.169330596923828,
- -4.490111827850342,
- -0.4412422776222229,
- -3.9356117248535156,
- -0.5775455832481384,
- -0.2409835010766983,
- -2.9197134971618652,
- -13.475022315979004,
- -0.10248012840747833,
- -3.5023770332336426,
- -0.8544933795928955,
- -5.194520473480225,
- -0.32954925298690796,
- -2.3026833534240723,
- -0.5346049070358276,
- -1.2862977981567383,
- -4.881562232971191,
- -15.555293083190918,
- -4.919404029846191,
- -0.22008435428142548,
- -6.644532680511475,
- -0.8938115239143372,
- -2.1304054260253906,
- -1.8866363763809204,
- -0.20106904208660126,
- -5.917205810546875,
- -0.0056310598738491535,
- -7.453446388244629,
- -3.1677205562591553,
- -3.706507682800293,
- -2.136584520339966,
- -2.9287283420562744,
- -1.4792609214782715,
- -2.4399306774139404,
- -1.2330785989761353,
- -1.9715899229049683,
- -1.9578948020935059,
- -0.23143476247787476,
- -2.052696466445923,
- -1.0413113832473755,
- -1.1709030866622925,
- -2.825991630554199,
- -1.6848523616790771,
- -2.2008259296417236,
- -1.5216114521026611,
- -1.2439141273498535,
- -1.412055253982544
+ -3.662419557571411,
+ -3.7036173343658447,
+ -1.6009434461593628,
+ -1.5501081943511963,
+ -2.846059799194336,
+ -6.732302665710449,
+ -0.06605878472328186,
+ -1.334327220916748,
+ -6.083745002746582,
+ -9.440131187438965,
+ -10.473882675170898,
+ -1.5964821577072144,
+ -4.702763557434082,
+ -0.7514524459838867,
+ -2.1461901664733887,
+ -0.012340382672846317,
+ -0.03605639934539795,
+ -3.0907557010650635,
+ -8.744739532470703,
+ -1.5410845279693604,
+ -5.84979772567749,
+ -3.0918972492218018,
+ -3.9814329147338867,
+ -3.78017520904541,
+ -2.5227086544036865,
+ -2.258594036102295,
+ -0.4719255566596985,
+ -1.0329649448394775,
+ -5.3284382820129395,
+ -8.25335693359375,
+ -0.015789249911904335,
+ -2.854100227355957,
+ -1.2236379384994507,
+ -3.905193328857422,
+ -0.9268187284469604,
+ -0.0030202509369701147,
+ -3.224249839782715,
+ -11.11172103881836,
+ -3.8121743202209473,
+ -2.3400487899780273,
+ -4.672845363616943,
+ -0.09729652851819992,
+ -0.06232408434152603,
+ -1.336004614830017,
+ -2.054157257080078,
+ -4.390933036804199,
+ -0.44248226284980774,
+ -3.9417736530303955,
+ -0.5888474583625793,
+ -0.26697415113449097,
+ -2.9271092414855957,
+ -13.515066146850586,
+ -0.10294333100318909,
+ -3.5007452964782715,
+ -0.8535972237586975,
+ -5.173652648925781,
+ -0.330394983291626,
+ -2.304553508758545,
+ -0.5418462753295898,
+ -1.300589919090271,
+ -4.9136152267456055,
+ -15.558022499084473,
+ -4.918652534484863,
+ -0.22206512093544006,
+ -6.589188575744629,
+ -0.9015690684318542,
+ -2.2228457927703857,
+ -1.8689247369766235,
+ -0.2006368339061737,
+ -5.918689727783203,
+ -0.006355076562613249,
+ -7.532094955444336,
+ -3.2708187103271484,
+ -3.743263006210327,
+ -2.011824131011963,
+ -3.0331737995147705,
+ -1.2233026027679443,
+ -1.6082642078399658,
+ -0.9534461498260498,
+ -1.4570012092590332,
+ -0.29605966806411743,
+ -1.8120722770690918,
+ -2.9380805492401123,
+ -0.8330938220024109,
+ -1.776170253753662,
+ -1.1607933044433594,
+ -2.3149070739746094,
+ -2.0255284309387207,
+ -2.521214723587036,
+ -1.7217901945114136,
+ -0.1730123609304428,
+ -2.888899564743042,
+ -0.2459663301706314,
+ -0.0008181562297977507,
+ -1.4219350814819336,
+ -1.5678833723068237,
+ -2.491905927658081,
+ -0.8186907172203064,
+ -0.37120482325553894,
+ -2.210369110107422,
+ -1.2439723014831543,
+ -1.019776463508606,
+ -0.44152167439460754,
+ -0.20805038511753082,
+ -2.262563943862915
]
- }
+ },
+ "mem-max-allocated-bytes": 48828583424,
+ "lifetime_prefill_token_count": 88
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m/model_config.yaml b/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m/model_config.yaml
index 0232bcb30bf..f5de6eaac72 100644
--- a/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m/model_config.yaml
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m/model_config.yaml
@@ -3,6 +3,8 @@ ENV_VARS:
NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
NCCL_ALGO: Ring
CUBLAS_WORKSPACE_CONFIG: :4096:8
+ TRITON_CACHE_AUTOTUNING: 0
+ MAMBA_DETERMINISTIC: 1
TEST_TYPE: frozen-start
MODE: inference
MODEL_ARGS:
@@ -22,19 +24,17 @@ MODEL_ARGS:
--pipeline-model-parallel-size: 1
--expert-model-parallel-size: 1
--use-mcore-models: true
- --is-hybrid-model: true
--model-provider: mamba
--init-method-std: 0.0198
--untie-embeddings-and-output-weights: true
--disable-bias-linear: true
--init-method-std: 0.014
--position-embedding-type: none
- --num-layers: 50
--hidden-size: 2048
--ffn-hidden-size: 11264
--num-attention-heads: 16
--kv-channels: 128
- --hybrid-override-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-
+ --hybrid-layer-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-
--spec: megatron.core.models.mamba.mamba_layer_specs mamba_stack_spec
--normalization: RMSNorm
--swiglu: true
@@ -68,6 +68,8 @@ MODEL_ARGS:
--incoming-requests-per-step: 32
--inference-repeat-n: 3
--no-record-throughput: true
+ --mamba-inference-conv-states-dtype: fp32
+ --mamba-inference-ssm-states-dtype: fp32
METRICS:
- "generated_tokens"
- "logprobs"
diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m_chunked_prefill/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m_chunked_prefill/golden_values_dev_dgx_h100.json
index 411bc8b74a6..928d335177a 100644
--- a/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m_chunked_prefill/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m_chunked_prefill/golden_values_dev_dgx_h100.json
@@ -34,7 +34,8 @@
1032,
1049
],
- "latency": 2.9149169921875,
+ "latency": 2.7100250720977783,
+ "ttft": 0.9113945960998535,
"cuda_graph_request_count_map": null,
"step_count": 40,
"top_n_logprobs": null,
@@ -56,2760 +57,2760 @@
-1.2136539220809937,
-0.4511846899986267,
-1.275371789932251,
- -6.52569055557251,
- -0.3268530070781708,
- -2.488239288330078,
- -1.1252245903015137,
- -0.004931548144668341,
- -1.1413307189941406,
- -2.4036614894866943,
- -0.593055784702301,
- -5.775687217712402,
- -0.7173333764076233,
- -6.7589006423950195,
- -4.472473621368408,
- -0.28561243414878845,
- -0.9266374111175537,
- -1.2420787811279297,
- -4.94831657409668,
- -0.4015401303768158,
- -2.405423879623413,
- -6.706996440887451,
- -2.3797435760498047,
- -6.879988193511963,
- -0.599727988243103,
- -4.6161346435546875,
- -0.016334740445017815,
- -1.4226453304290771,
- -4.064138412475586,
- -8.992555618286133,
- -0.7892558574676514,
- -2.565383195877075,
- -1.6011606454849243,
- -1.1192784309387207,
- -1.085118293762207,
- -1.452021598815918,
- -0.1256672590970993,
- -4.310093879699707,
- -0.039925139397382736,
- -0.09540079534053802,
- -4.4552788734436035,
- -2.6978704929351807,
- -0.3264457583427429,
- -0.9057141542434692,
- -0.2424505054950714,
- -0.2473771721124649,
- -0.04457908123731613,
- -2.5994861125946045,
- -0.5882505178451538,
- -2.4292445182800293,
- -0.1860235333442688,
- -2.6841845512390137,
- -5.8617939949035645,
- -1.7926914691925049,
- -0.6663980484008789,
- -0.029983440414071083,
- -1.0682772397994995,
- -0.0018566290382295847,
- -1.9571454524993896,
- -0.08927226811647415,
- -4.61471700668335,
- -0.002604546956717968,
- -0.2620302140712738,
- -0.006101197097450495,
- -7.435886859893799,
- -0.0376485139131546,
- -10.174129486083984,
- -0.9147175550460815,
- -4.526404857635498,
- -3.670576572418213,
- -4.566626071929932,
- -1.0199782848358154,
- -0.0006491222884505987,
- -0.14426420629024506,
- -0.03322957828640938,
- -0.0019640696700662374,
- -0.00022468426323030144,
- -0.0013444918440654874,
- -0.0011957883834838867,
- -0.007926556281745434,
- -0.011617152951657772,
- -0.0018109364900738,
- -0.00017581824795342982,
- -0.0018969652010127902,
- -6.282132380874828e-05,
- -0.0010078833438456059,
- -0.25652098655700684,
- -0.35659894347190857,
- -9.333651541965082e-05,
- -0.7947311401367188,
- -1.3594639301300049,
- -7.962863310240209e-05,
- -1.861167550086975,
- -0.5386030673980713,
- -0.00022075122979003936,
- -0.001347229932434857,
- -3.290122185717337e-05,
- -3.7342543601989746,
- -0.5175371170043945,
- -4.488879680633545,
- -0.007863753475248814,
- -0.08534510433673859,
- -0.0009170140838250518,
- -2.13382354559144e-05,
- -4.507952690124512,
- -0.5332688689231873,
- -0.004296358674764633,
- -2.062299427052494e-05,
- -5.2475104331970215,
- -0.020387964323163033,
- -0.1661914438009262,
- -0.0003081085451412946,
- -15.800027847290039,
- -8.108964920043945,
- -0.7285020351409912,
- -7.803549289703369,
- -5.010417938232422,
- -0.263860821723938,
- -4.3748852476710454e-05,
- -0.013306032866239548,
- -0.029512016102671623,
- -0.0036468682810664177,
- -0.00023231192608363926,
- -0.0002379134384682402,
- -0.0004920940846204758,
- -0.000873065204359591,
- -0.0029308719094842672,
- -0.0006667536217719316,
- -0.00013672371278516948,
- -0.0011686407960951328,
+ -6.502887725830078,
+ -0.3318161368370056,
+ -2.374776601791382,
+ -1.1216609477996826,
+ -0.004972116556018591,
+ -1.132584571838379,
+ -2.3807473182678223,
+ -0.6019684672355652,
+ -5.6695661544799805,
+ -0.7204529047012329,
+ -6.769031047821045,
+ -4.453886985778809,
+ -0.28745004534721375,
+ -0.9512360692024231,
+ -1.1893970966339111,
+ -4.947760581970215,
+ -0.36444246768951416,
+ -2.4082541465759277,
+ -6.61063814163208,
+ -2.503272294998169,
+ -6.87827205657959,
+ -0.6025495529174805,
+ -4.622146129608154,
+ -0.01804928109049797,
+ -1.4212173223495483,
+ -4.08338737487793,
+ -9.046302795410156,
+ -0.7913540601730347,
+ -2.5653929710388184,
+ -1.5746062994003296,
+ -1.1179513931274414,
+ -1.085464358329773,
+ -1.447007656097412,
+ -0.12754084169864655,
+ -4.321589469909668,
+ -0.03916267305612564,
+ -0.09776852279901505,
+ -4.340118885040283,
+ -2.6412477493286133,
+ -0.35567763447761536,
+ -0.9400036334991455,
+ -0.2579241991043091,
+ -0.2446412295103073,
+ -0.044584665447473526,
+ -2.601752996444702,
+ -0.5793261528015137,
+ -2.4348762035369873,
+ -0.1983177214860916,
+ -2.804823160171509,
+ -5.8624348640441895,
+ -1.890607237815857,
+ -0.6673221588134766,
+ -0.02870091050863266,
+ -1.0835812091827393,
+ -0.0018605556106194854,
+ -1.954398274421692,
+ -0.08902604877948761,
+ -4.561020851135254,
+ -0.002339128637686372,
+ -0.28425076603889465,
+ -0.006011382210999727,
+ -7.539747714996338,
+ -0.0332016721367836,
+ -10.147441864013672,
+ -0.9909310340881348,
+ -4.516211986541748,
+ -3.788264751434326,
+ -4.543238639831543,
+ -0.967991054058075,
+ -0.0006565084331668913,
+ -0.14520324766635895,
+ -0.032342080026865005,
+ -0.0019212374463677406,
+ -0.00022480344341602176,
+ -0.0013449679827317595,
+ -0.0011960265692323446,
+ -0.007967357523739338,
+ -0.011690796352922916,
+ -0.0019255208317190409,
+ -0.00019429226813372225,
+ -0.0017928492743521929,
+ -6.329813186312094e-05,
+ -0.0009005781612358987,
+ -0.25652363896369934,
+ -0.3569473922252655,
+ -8.11782301752828e-05,
+ -0.7947285771369934,
+ -1.3081212043762207,
+ -8.21318244561553e-05,
+ -1.8611676692962646,
+ -0.5729392766952515,
+ -0.0002153879904653877,
+ -0.0013609204906970263,
+ -3.2543604902457446e-05,
+ -3.7282280921936035,
+ -0.5535277128219604,
+ -4.489457607269287,
+ -0.008902146480977535,
+ -0.08532682061195374,
+ -0.0009254701435565948,
+ -2.098061486321967e-05,
+ -4.501499176025391,
+ -0.49491915106773376,
+ -0.004292679019272327,
+ -2.002696055569686e-05,
+ -5.250372409820557,
+ -0.02144506201148033,
+ -0.163783460855484,
+ -0.0003084660565946251,
+ -15.754971504211426,
+ -8.121952056884766,
+ -0.7163605093955994,
+ -7.821594715118408,
+ -5.016800880432129,
+ -0.26420897245407104,
+ -4.8040190449682996e-05,
+ -0.013264977373182774,
+ -0.0258948914706707,
+ -0.003469758667051792,
+ -0.0002307625545654446,
+ -0.00023052419419400394,
+ -0.0004808938247151673,
+ -0.000872946111485362,
+ -0.0031739119440317154,
+ -0.0007124984404072165,
+ -0.00013267113536130637,
+ -0.0011754278093576431,
+ -5.924526340095326e-05,
+ -0.000880449719261378,
+ -0.02788648009300232,
+ -0.06313233077526093,
+ -0.0001311216183239594,
+ -0.21269679069519043,
+ -0.6191936135292053,
+ -5.769562994828448e-05,
+ -1.637540340423584,
+ -1.6328482627868652,
+ -0.0003262225945945829,
+ -0.0016448076348751783,
+ -2.8967437174287625e-05,
+ -2.92718505859375,
+ -0.2820948660373688,
+ -6.417831897735596,
+ -3.146303176879883,
+ -0.7236800789833069,
+ -0.06164867803454399,
+ -0.0015048381173983216,
+ -2.825220326485578e-05,
+ -3.9370851516723633,
+ -0.09090036898851395,
+ -0.0035149003379046917,
+ -4.279521817807108e-05,
+ -4.558128833770752,
+ -0.010771229863166809,
+ -0.09514036774635315,
+ -0.00041416651220060885,
+ -4.436840534210205,
+ -2.224473714828491,
+ -3.2508153915405273,
+ -8.453686714172363,
+ -0.030126187950372696,
+ -2.3973264694213867,
+ -0.07391595095396042,
+ -0.5675545930862427,
+ -0.05637309327721596,
+ -0.21564748883247375,
+ -0.00012206286191940308,
+ -0.05818720534443855,
+ -0.12251465767621994,
+ -0.013046146370470524,
+ -0.00023183519078884274,
+ -0.0001517419150331989,
+ -0.0003564914222806692,
+ -0.0005465444410219789,
+ -0.0022260190453380346,
+ -0.0007184546557255089,
+ -9.011816291604191e-05,
+ -0.00105015910230577,
-4.625213477993384e-05,
- -0.0007901645149104297,
- -0.027857612818479538,
- -0.06313244253396988,
- -0.00013064485392533243,
- -0.2378876954317093,
- -0.6059458255767822,
- -5.757642793469131e-05,
- -1.5949885845184326,
- -1.6001688241958618,
- -0.00032574593205936253,
- -0.0016402851324528456,
+ -0.0005137792322784662,
+ -0.021607117727398872,
+ -0.023618897423148155,
+ -0.0001915509783430025,
+ -0.043488532304763794,
+ -2.8467631340026855,
+ -9.07141511561349e-05,
+ -0.9812690019607544,
+ -1.4124990701675415,
+ -0.00018153927521780133,
+ -0.0012271024752408266,
+ -1.6927575416048057e-05,
+ -1.0269899368286133,
+ -0.019673457369208336,
+ -0.045223724097013474,
+ -1.6328576803207397,
+ -0.00045599075383506715,
+ -0.000436091679148376,
-2.276871418871451e-05,
- -3.0335943698883057,
- -0.286937952041626,
- -6.517683982849121,
- -3.1465959548950195,
- -0.7292280793190002,
- -0.06161583960056305,
- -0.0014851979212835431,
- -2.777537883957848e-05,
- -3.946831226348877,
- -0.09084996581077576,
- -0.003532005939632654,
- -4.029192859889008e-05,
- -4.555190086364746,
- -0.011255813762545586,
- -0.10179147869348526,
- -0.0004140473320148885,
- -4.4321393966674805,
- -2.2296247482299805,
- -3.2771155834198,
- -8.323366165161133,
- -0.02779245562851429,
- -2.403028964996338,
- -0.07431145757436752,
- -0.5372196435928345,
- -0.05987980589270592,
- -0.20438668131828308,
- -0.00013136000779923052,
- -0.0572563000023365,
- -0.11035308241844177,
- -0.012903997674584389,
- -0.0002406545972917229,
- -0.0001517419150331989,
- -0.00036066226311959326,
- -0.0005477358354255557,
- -0.00229322025552392,
- -0.000697846058756113,
- -0.0001161031104857102,
- -0.001127441762946546,
- -3.814624506048858e-05,
- -0.0005136600811965764,
- -0.022026309743523598,
- -0.02361132949590683,
- -0.0002090712368953973,
- -0.04913746938109398,
- -2.7477238178253174,
- -9.202533692587167e-05,
- -0.9271803498268127,
- -1.3856279850006104,
- -0.0001754606782924384,
- -0.0012224590172991157,
- -1.7165990357170813e-05,
- -1.0239524841308594,
- -0.020712625235319138,
- -0.0451514832675457,
- -1.5345499515533447,
- -0.0004010588163509965,
- -0.0004401430196594447,
- -2.13382354559144e-05,
- -2.5878491401672363,
- -0.020529404282569885,
- -0.00043501926120370626,
- -2.682172998902388e-05,
- -0.3827762007713318,
- -0.00019298121333122253,
- -0.007158228196203709,
- -8.618460560683161e-05,
- -6.015654563903809,
- -4.037173271179199,
- -3.4229695796966553,
- -1.0183475017547607,
- -1.4963387250900269,
- -0.33330175280570984,
- -1.480197787284851,
- -2.0857536792755127,
- -2.225975513458252,
- -5.293066024780273,
- -0.43916723132133484,
- -0.00010048838157672435,
- -0.015328695066273212,
- -0.13567933440208435,
- -0.012453177943825722,
- -0.00017855956684798002,
- -0.00012778419477399439,
- -0.0002885640424210578,
- -0.0004291805380489677,
- -0.0008485292200930417,
- -0.0006668727728538215,
- -8.177422569133341e-05,
- -0.001060757553204894,
- -6.151010165922344e-05,
- -0.0005185451591387391,
- -0.028113562613725662,
- -0.03407377377152443,
- -0.0003861635341309011,
- -1.1215460300445557,
- -0.5561063885688782,
- -0.0001726001501083374,
- -2.5190887451171875,
- -0.6141397953033447,
+ -2.5508246421813965,
+ -0.018045533448457718,
+ -0.00042500998824834824,
+ -2.825220326485578e-05,
+ -0.3710917532444,
+ -0.00019667598826345056,
+ -0.006620378233492374,
+ -8.606540359323844e-05,
+ -6.00956916809082,
+ -4.060979843139648,
+ -3.422973155975342,
+ -1.019230604171753,
+ -1.5553637742996216,
+ -0.2993496060371399,
+ -1.4802078008651733,
+ -2.070681095123291,
+ -2.330273151397705,
+ -5.414191246032715,
+ -0.46552836894989014,
+ -8.868777513271198e-05,
+ -0.016082679852843285,
+ -0.15049032866954803,
+ -0.012319305911660194,
+ -0.00018225439998786896,
+ -0.00011705666838679463,
+ -0.00030763185350224376,
+ -0.0004362108593340963,
+ -0.0008486483711749315,
+ -0.0006685405969619751,
+ -0.00010477947944309562,
+ -0.0010546842822805047,
+ -5.376194530981593e-05,
+ -0.0005538121913559735,
+ -0.029470346868038177,
+ -0.030175579711794853,
+ -0.0003849719068966806,
+ -1.207515001296997,
+ -0.5080216526985168,
+ -0.00017128908075392246,
+ -2.5190865993499756,
+ -0.6522354483604431,
-0.0001227780303452164,
- -0.0012188870459794998,
- -1.6212332411669195e-05,
- -6.833529472351074,
- -6.0156097412109375,
- -0.03274226188659668,
- -0.014286145567893982,
- -0.0009454786195419729,
- -3.814624506048858e-05,
- -4.910149097442627,
- -0.009493326768279076,
- -0.001437702914699912,
+ -0.0012397230602800846,
+ -1.597391747054644e-05,
+ -6.705944538116455,
+ -6.01674222946167,
+ -0.032217077910900116,
+ -0.014274628832936287,
+ -0.0009184433147311211,
+ -3.8265450712060556e-05,
+ -5.022707939147949,
+ -0.0094671119004488,
+ -0.0014456784119829535,
+ -5.864924969500862e-05,
+ -0.37389981746673584,
+ -0.004271550104022026,
+ -0.06971022486686707,
+ -0.00017653337272349745,
+ -6.816598892211914,
+ -0.0015250729629769921,
+ -2.643510341644287,
+ -0.9823706150054932,
+ -1.4715224504470825,
+ -0.7379395365715027,
+ -1.4038898944854736,
+ -9.202533692587167e-05,
+ -0.04053020477294922,
+ -0.3290509581565857,
+ -0.02419864945113659,
+ -0.00028713393840007484,
+ -0.00013565097469836473,
+ -0.00026723151677288115,
+ -0.000376034586224705,
+ -0.0006663962849415839,
+ -0.0008694920688867569,
+ -9.881961887003854e-05,
+ -0.0012442474253475666,
+ -5.781483559985645e-05,
+ -0.0005551227368414402,
+ -0.013484349474310875,
+ -0.033842623233795166,
+ -0.00047017011092975736,
+ -0.28224796056747437,
+ -1.4657995700836182,
+ -0.00012790338951162994,
+ -1.4014155864715576,
+ -0.9573185443878174,
+ -0.0002760506176855415,
+ -0.0016822961624711752,
+ -2.3483953555114567e-05,
+ -5.273507595062256,
+ -2.626476764678955,
+ -0.1083991751074791,
+ -0.0007696765242144465,
+ -3.158996332786046e-05,
+ -5.912450790405273,
+ -0.05985802412033081,
+ -0.002310347044840455,
+ -2.7656173188006505e-05,
+ -0.5037983655929565,
+ -0.0008891443139873445,
+ -0.013665338978171349,
+ -8.546940807718784e-05,
+ -6.966069221496582,
+ -0.366066038608551,
+ -1.422469139099121,
+ -0.8349007964134216,
+ -0.468829870223999,
+ -2.5325279235839844,
+ -0.3129241466522217,
+ -7.700963973999023,
+ -0.35001978278160095,
+ -2.013795852661133,
+ -0.09784646332263947,
+ -7.86750388215296e-05,
+ -0.01667795516550541,
+ -0.18287953734397888,
+ -0.015566980466246605,
+ -0.0001851148990681395,
+ -7.593343616463244e-05,
+ -0.00014125302550382912,
+ -0.0002083561266772449,
+ -0.0004748170613311231,
+ -0.000512111175339669,
+ -4.815939246327616e-05,
+ -0.0011738798348233104,
+ -3.0517112463712692e-05,
+ -0.0003026265767402947,
+ -0.003111880039796233,
+ -0.009830750524997711,
+ -0.00022289653134066612,
+ -0.12992516160011292,
+ -1.1169023513793945,
+ -9.691245941212401e-05,
+ -1.1468124389648438,
+ -0.49202996492385864,
+ -7.009260298218578e-05,
+ -0.0014340127818286419,
+ -1.5735502529423684e-05,
+ -3.9838247299194336,
+ -0.02546604536473751,
+ -0.0001641377166379243,
+ -0.003960979636758566,
+ -0.0006631797295995057,
+ -2.9801878554280847e-05,
+ -0.15490414202213287,
+ -0.0008278042078018188,
+ -0.0008001701789908111,
+ -1.811964830267243e-05,
+ -0.21022747457027435,
+ -0.00018714107864070684,
+ -0.0019920284394174814,
+ -2.2053474822314456e-05,
+ -0.014113731682300568,
+ -1.4305104514278355e-06,
+ -0.011013787239789963,
+ -0.6143843531608582,
+ -2.3573765754699707,
+ -0.0103318365290761,
+ -0.00997463520616293,
+ -7.772143726469949e-05,
+ -0.010983250103890896,
+ -0.29723531007766724,
+ -0.022360580042004585,
+ -0.0001991788885788992,
+ -5.006664650863968e-05,
+ -0.00010573305189609528,
+ -0.00015805903240107,
+ -0.00035661060246638954,
+ -0.0004104725376237184,
+ -7.164221460698172e-05,
+ -0.0008939084364101291,
+ -3.135155202471651e-05,
+ -0.00031394799589179456,
+ -0.002726529957726598,
+ -0.008696303702890873,
+ -0.0003630455757956952,
+ -0.258771151304245,
+ -2.1416406631469727,
+ -0.0001802282058633864,
+ -1.8033326864242554,
+ -1.623347520828247,
+ -0.0002315968304174021,
+ -0.0006778326351195574,
+ -1.3470558769768104e-05,
+ -1.263992428779602,
+ -0.016645127907395363,
+ -2.042416572570801,
+ -1.4463900327682495,
+ -2.7770018577575684,
+ -0.5399353504180908,
+ -0.00030071981018409133,
+ -2.1576648578047752e-05,
+ -2.99349308013916,
+ -0.0013340153964236379,
+ -0.0007597897201776505,
+ -1.4662635294371285e-05,
+ -0.26145222783088684,
+ -0.00023266946664080024,
+ -0.010189891792833805,
+ -3.790783375734463e-05,
+ -2.1039741039276123,
+ -6.185567378997803,
+ -4.484244346618652,
+ -2.000312089920044,
+ -0.2879604697227478,
+ -0.05998960882425308,
+ -0.7370511889457703,
+ -0.01011425070464611,
+ -0.00029690624796785414,
+ -0.14329783618450165,
+ -0.0006635371246375144,
+ -0.00035398892941884696,
+ -0.00229904823936522,
+ -0.36686453223228455,
+ -0.09950104355812073,
+ -2.546339511871338,
+ -3.2053751945495605,
+ -0.11890234053134918,
+ -4.541770613286644e-05,
+ -0.015083776786923409,
+ -0.1699461191892624,
+ -0.008777014911174774,
+ -0.00011419598013162613,
+ -3.421248038648628e-05,
+ -6.115249561844394e-05,
+ -0.0001998939987970516,
+ -0.0003668589051812887,
+ -0.0003194298769813031,
+ -5.23315102327615e-05,
+ -0.0009507188224233687,
+ -2.3364747903542593e-05,
+ -0.00019131260341964662,
+ -0.0009372609201818705,
+ -0.004119957331568003,
+ -0.0002714027068577707,
+ -0.31108736991882324,
+ -0.4985818862915039,
+ -0.0001879753835964948,
+ -1.0743582248687744,
+ -0.28104329109191895,
+ -4.303362584323622e-05,
+ -0.0011857867939397693,
+ -1.9073304429184645e-05,
+ -1.1399143934249878,
+ -0.06796625256538391,
+ -0.0027990469243377447,
+ -8.654219709569588e-05,
+ -0.00018761781393550336,
+ -0.0004489606071729213,
+ -1.5735502529423684e-05,
+ -8.45077896118164,
+ -0.00031680811662226915,
+ -7.279982089996338,
+ -0.09349125623703003,
+ -0.004146312829107046,
+ -0.0008599635912105441,
+ -5.149708886165172e-05,
+ -0.7756458520889282,
+ -0.00035589560866355896,
+ -0.01518346555531025,
-5.876845170860179e-05,
- -0.3798050582408905,
- -0.003948037512600422,
- -0.07855644077062607,
- -0.00022420754248742014,
- -6.84205436706543,
- -0.0015236446633934975,
- -2.645585298538208,
- -0.9816564917564392,
- -1.3786735534667969,
- -0.7280330061912537,
- -1.4040117263793945,
- -9.035655966727063e-05,
- -0.033023953437805176,
- -0.3305729031562805,
- -0.027912795543670654,
- -0.0002892790944315493,
- -0.00012182447244413197,
- -0.00026901919045485556,
- -0.0004681444843299687,
- -0.0007345362100750208,
- -0.0008179179858416319,
- -0.00010549465514486656,
- -0.0013330630026757717,
- -5.7338023907504976e-05,
- -0.0005571481888182461,
- -0.013437421061098576,
- -0.033829718828201294,
- -0.0004694551753345877,
- -0.28239941596984863,
- -1.3776881694793701,
+ -3.550320625305176,
+ -0.037021368741989136,
+ -0.7441027164459229,
+ -1.400524377822876,
+ -0.6082935333251953,
+ -6.150390625,
+ -0.9888088703155518,
+ -1.3728853464126587,
+ -0.4236694276332855,
+ -4.5586042404174805,
+ -0.41667699813842773,
+ -0.022858109325170517,
+ -9.846202738117427e-05,
+ -0.009947134181857109,
+ -0.22649677097797394,
+ -0.020902235060930252,
-0.00014256415306590497,
- -1.4336698055267334,
- -0.9458242654800415,
- -0.0002739054325502366,
- -0.0015444743912667036,
- -2.169585604860913e-05,
- -5.267784118652344,
- -2.617713689804077,
- -0.1205064058303833,
- -0.000608854868914932,
- -2.47952248173533e-05,
- -6.116018772125244,
- -0.06051409989595413,
- -0.0021291938610374928,
- -2.777537883957848e-05,
- -0.5082104206085205,
- -0.0008528171456418931,
- -0.013313560746610165,
+ -9.405170567333698e-05,
+ -0.00010013079008786008,
+ -0.004698308650404215,
+ -0.00032884435495361686,
+ -0.0007784912013448775,
-9.381330892210826e-05,
- -6.970278739929199,
- -0.3628937304019928,
- -1.40151047706604,
- -0.8361061811447144,
- -0.4778183400630951,
- -2.494100570678711,
- -0.3126090466976166,
- -7.66262674331665,
- -0.3505229353904724,
- -2.1190404891967773,
- -0.08990062028169632,
- -8.201262971851975e-05,
- -0.01644204556941986,
- -0.1838725060224533,
- -0.015538694337010384,
- -0.00019107422849629074,
- -7.915183232398704e-05,
- -0.0001382732152706012,
- -0.0002119316632160917,
- -0.0004773192631546408,
- -0.0004781533498317003,
- -4.994744449504651e-05,
- -0.0011807858245447278,
- -3.0636318115284666e-05,
- -0.0003046525234822184,
- -0.0024103655014187098,
- -0.009829924441874027,
- -0.00022301571152638644,
- -0.12844854593276978,
- -1.1151821613311768,
- -9.512448741588742e-05,
- -1.1148451566696167,
- -0.45424169301986694,
- -7.128461584215984e-05,
- -0.001427346607670188,
- -1.2040065485052764e-05,
- -3.9783990383148193,
- -0.025781046599149704,
- -0.00015496007108595222,
- -0.003944831434637308,
- -0.000663894519675523,
- -3.015949550899677e-05,
- -0.15718017518520355,
- -0.0009197533945553005,
- -0.0007913556764833629,
- -1.8000440832111053e-05,
- -0.18712174892425537,
- -0.00016604475968051702,
- -0.0022110319696366787,
- -2.169585604860913e-05,
- -0.014111850410699844,
- -1.1920922133867862e-06,
- -0.00984656810760498,
- -0.5971966981887817,
- -2.393812894821167,
- -0.010224700905382633,
- -0.009953508153557777,
- -7.64102369430475e-05,
- -0.011833352968096733,
- -0.26886406540870667,
- -0.023419089615345,
- -0.00019762947340495884,
- -6.031808152329177e-05,
- -0.00010191874753218144,
- -0.00015889335190877318,
- -0.0003564914222806692,
- -0.0004101150552742183,
- -6.675497570540756e-05,
- -0.0009184433147311211,
- -3.158996332786046e-05,
- -0.00031442465842701495,
- -0.0027259355410933495,
- -0.008694176562130451,
- -0.00032658010604791343,
- -0.289438933134079,
- -2.1416351795196533,
- -0.00017987063620239496,
- -1.8434972763061523,
- -1.624247670173645,
- -0.00022980909852776676,
- -0.0006792622152715921,
- -1.0967194612021558e-05,
- -1.281017541885376,
- -0.01736496575176716,
- -1.955749750137329,
- -1.528749942779541,
- -2.776960611343384,
- -0.5374854803085327,
- -0.00029345019720494747,
- -2.539125671319198e-05,
- -3.0065665245056152,
- -0.0013523490633815527,
- -0.0007908792467787862,
- -1.4543427823809907e-05,
- -0.23400214314460754,
- -0.0002324311062693596,
- -0.010042970068752766,
- -4.088794958079234e-05,
- -2.1034951210021973,
- -6.140199184417725,
- -4.464273929595947,
- -1.9943883419036865,
- -0.2878473103046417,
- -0.05924016237258911,
- -0.7345774173736572,
- -0.011171765625476837,
- -0.0002982171718031168,
- -0.14330486953258514,
- -0.0007319155265577137,
- -0.0003812778159044683,
- -0.002302616136148572,
- -0.36087724566459656,
- -0.08833581954240799,
- -2.631582260131836,
- -3.1771137714385986,
- -0.11841163039207458,
- -4.482168878894299e-05,
- -0.014765388332307339,
- -0.17005765438079834,
- -0.010167589411139488,
- -0.00010823617776622996,
- -3.6477376852417365e-05,
- -5.936446541454643e-05,
- -0.00023493390472140163,
- -0.0003688847064040601,
- -0.000321336614433676,
- -4.756337511935271e-05,
- -0.000902007392141968,
- -2.9205850296420977e-05,
- -0.00024423000286333263,
- -0.000964533886872232,
- -0.00411722669377923,
- -0.0002711643755901605,
- -0.3081328868865967,
- -0.4985820949077606,
- -0.00018726025882642716,
- -1.1391643285751343,
- -0.27228832244873047,
- -4.2914423829643056e-05,
- -0.0012028133496642113,
- -1.9311717551317997e-05,
- -1.1735807657241821,
- -0.07005516439676285,
- -0.0024717275518924,
- -8.618460560683161e-05,
- -0.00016866691294126213,
- -0.00044764988706447184,
- -1.6093124941107817e-05,
- -8.586283683776855,
- -0.0002851079625543207,
- -7.490447998046875,
- -0.09369903802871704,
- -0.004145600367337465,
- -0.0008606782066635787,
- -4.827859811484814e-05,
- -0.7127438187599182,
- -0.0003618539194576442,
- -0.015226203016936779,
- -6.401333666872233e-05,
- -3.530060291290283,
- -0.040570154786109924,
- -0.7448150515556335,
- -1.4005241394042969,
- -0.5872946977615356,
- -6.073245048522949,
- -0.9850690364837646,
- -1.4459205865859985,
- -0.4346452057361603,
- -4.452149868011475,
- -0.3939701318740845,
- -0.02252959832549095,
- -9.440929716220126e-05,
- -0.012161390855908394,
- -0.25266116857528687,
- -0.021285664290189743,
- -0.00015770144818816334,
- -9.870042413240299e-05,
- -9.989239333663136e-05,
- -0.005311425309628248,
- -0.00032634177478030324,
- -0.0007045170641504228,
- -9.417090768693015e-05,
- -0.001260558608919382,
- -4.482168878894299e-05,
- -0.0003833036171272397,
- -0.0023484050761908293,
- -0.011129915714263916,
- -0.00040260792593471706,
- -0.1819346845149994,
- -1.1781600713729858,
- -0.00033241944038309157,
- -1.3525464534759521,
- -1.2726483345031738,
- -0.00018034738604910672,
- -0.0009054613183252513,
- -1.2040065485052764e-05,
- -1.7329559326171875,
- -0.009877022355794907,
- -0.030561018735170364,
- -0.9567705988883972,
- -0.0002079985715681687,
- -0.0003582789213396609,
+ -0.0012371037155389786,
+ -3.4927710657939315e-05,
+ -0.00037448544753715396,
+ -0.002363271312788129,
+ -0.011125907301902771,
+ -0.00035613393993116915,
+ -0.20121917128562927,
+ -1.1733814477920532,
+ -0.0003303935518488288,
+ -1.3426117897033691,
+ -1.271085262298584,
+ -0.00018285033002030104,
+ -0.0009109399979934096,
+ -1.168244216387393e-05,
+ -1.7527191638946533,
+ -0.009883751161396503,
+ -0.03264499828219414,
+ -1.0129823684692383,
+ -0.00021395778458099812,
+ -0.0002731903805397451,
-2.5510462364763953e-05,
- -1.3376575708389282,
- -0.043758541345596313,
- -0.0005255748401395977,
- -0.003921795636415482,
- -3.9934315282152966e-05,
- -0.013946342281997204,
- -0.001447345013730228,
- -0.09289155900478363,
- -0.00028975578607060015,
- -5.025714874267578,
- -5.600637435913086,
- -0.8190056681632996,
- -2.0997657775878906,
- -1.5471020936965942,
- -0.2830793261528015,
- -0.099715456366539,
- -0.00015341058315243572,
- -0.09538150578737259,
- -0.9440865516662598,
- -0.13964560627937317,
- -0.0003178806509822607,
- -0.00015531764074694365,
- -0.00016640232934150845,
- -0.00023398046323563904,
- -0.00039081089198589325,
- -0.0015487592900171876,
- -0.00010716341057559475,
- -0.0017987991450354457,
- -3.838465272565372e-05,
- -0.0006412595394067466,
- -0.00545145571231842,
- -0.02335585467517376,
- -0.0004077318590134382,
- -0.8720157146453857,
- -0.10373511165380478,
- -0.00014077626110520214,
- -0.5180479884147644,
- -0.17388182878494263,
- -0.00015746307326480746,
- -0.0043711354956030846,
- -2.9801878554280847e-05,
- -2.0693466663360596,
- -0.007648942526429892,
- -2.8729025871143676e-05,
- -0.0003301552205812186,
- -0.000542612629942596,
- -3.2543604902457446e-05,
- -0.27388375997543335,
- -0.00043752157944254577,
- -0.0005888396990485489,
- -1.7762025890988298e-05,
- -0.05423494055867195,
- -7.915183232398704e-05,
- -0.002435457892715931,
- -1.1205610462639015e-05,
- -0.01761529967188835,
- -7.152555099310121e-07,
- -0.005352570675313473,
- -0.1280955821275711,
- -2.3187625408172607,
- -0.009216856211423874,
- -0.008558499626815319,
- -0.0001072826053132303,
- -0.04680917039513588,
- -0.5660229325294495,
- -0.04951385408639908,
- -0.0002015625941567123,
+ -1.3228294849395752,
+ -0.04179994761943817,
+ -0.0005071069463156164,
+ -0.003949818201363087,
+ -4.2199197196168825e-05,
+ -0.013934234157204628,
+ -0.001443773857317865,
+ -0.09278758615255356,
+ -0.00022623363474849612,
+ -5.036643028259277,
+ -5.596894264221191,
+ -0.8243252635002136,
+ -2.068366050720215,
+ -1.5817551612854004,
+ -0.28322750329971313,
+ -0.10253381729125977,
+ -0.0001525762490928173,
+ -0.09576807916164398,
+ -0.9411507844924927,
+ -0.12982413172721863,
+ -0.00031871485407464206,
+ -0.0001546025014249608,
+ -0.00016318420239258558,
+ -0.00018702188390307128,
+ -0.00034338299883529544,
+ -0.0015476880362257361,
+ -0.00011431517486926168,
+ -0.0018185520311817527,
+ -3.8265450712060556e-05,
+ -0.0006036128615960479,
+ -0.005799253936856985,
+ -0.023355038836598396,
+ -0.00040618274942971766,
+ -0.8012296557426453,
+ -0.10373521596193314,
+ -0.00012778419477399439,
+ -0.4956749379634857,
+ -0.16527129709720612,
+ -0.0001530530134914443,
+ -0.004374577663838863,
+ -2.9205850296420977e-05,
+ -1.9665766954421997,
+ -0.00752566521987319,
+ -2.7417760065873154e-05,
+ -0.0003748429589904845,
+ -0.0005439232336357236,
+ -2.3245540432981215e-05,
+ -0.27362146973609924,
+ -0.00042989550274796784,
+ -0.0005857420619577169,
+ -1.764281842042692e-05,
+ -0.05366183817386627,
+ -6.425174069590867e-05,
+ -0.0021969962399452925,
+ -1.1324817933200393e-05,
+ -0.017568564042448997,
+ -8.344646857949556e-07,
+ -0.005509074777364731,
+ -0.1280696839094162,
+ -2.3235721588134766,
+ -0.008202792145311832,
+ -0.008413229137659073,
+ -0.00012110930401831865,
+ -0.042223334312438965,
+ -0.5657049417495728,
+ -0.05260366573929787,
+ -0.0002240883477497846,
-5.8410845667822286e-05,
- -9.440929716220126e-05,
- -0.00014828535495325923,
- -0.00037245964631438255,
- -0.0008362610242329538,
- -5.4596363042946905e-05,
- -0.0010970771545544267,
- -4.017272294731811e-05,
- -0.0004563482361845672,
- -0.0021864098962396383,
- -0.012597862631082535,
- -0.00036435641231946647,
- -0.07823580503463745,
- -1.1245288848876953,
+ -9.595887240720913e-05,
-0.0001472126314183697,
- -2.1236472129821777,
- -0.25363627076148987,
- -0.00011646069469861686,
- -0.0010031197452917695,
- -1.4662635294371285e-05,
- -11.853788375854492,
- -1.5205868482589722,
- -0.0017375147435814142,
- -0.00013374387344811112,
- -7.155948638916016,
- -3.82474422454834,
- -1.2793458700180054,
- -0.03748536482453346,
- -0.005961020477116108,
- -5.829164365422912e-05,
- -3.1456170082092285,
- -0.03318829461932182,
- -0.008591356687247753,
- -0.027652040123939514,
- -0.00012885693286079913,
- -1.5415722131729126,
- -0.979039192199707,
- -2.842726469039917,
- -9.05957317352295,
- -2.8234424591064453,
- -0.8373243808746338,
- -0.4019332230091095,
- -0.0004048719711136073,
- -0.03923225402832031,
- -0.4254666864871979,
- -0.027653662487864494,
- -0.0003177614707965404,
- -0.0001967951684491709,
- -0.00020883286197204143,
- -0.00025674383505247533,
- -0.0008311392739415169,
- -0.0012284121476113796,
+ -0.00037472377880476415,
+ -0.0007722970913164318,
+ -5.447716102935374e-05,
+ -0.0010968389688059688,
+ -3.266281055402942e-05,
+ -0.00045622908510267735,
+ -0.0020772088319063187,
+ -0.011120365932583809,
+ -0.00032586511224508286,
+ -0.07033942639827728,
+ -1.1209707260131836,
+ -0.00014518637908622622,
+ -2.0970425605773926,
+ -0.25196295976638794,
+ -0.00011765264935093,
+ -0.0009179668850265443,
+ -1.537788011773955e-05,
+ -11.904622077941895,
+ -1.5177723169326782,
+ -0.001602795091457665,
+ -0.00013302871957421303,
+ -7.185190200805664,
+ -3.704270601272583,
+ -1.288861632347107,
+ -0.03340679779648781,
+ -0.005939452908933163,
+ -4.494089080253616e-05,
+ -3.2633724212646484,
+ -0.033210549503564835,
+ -0.008218162693083286,
+ -0.02734771929681301,
+ -0.00012838016846217215,
+ -1.5429582595825195,
+ -1.0140559673309326,
+ -2.7621612548828125,
+ -9.184652328491211,
+ -2.8712878227233887,
+ -0.8348691463470459,
+ -0.4236343204975128,
+ -0.00042012447374872863,
+ -0.039101459085941315,
+ -0.42557674646377563,
+ -0.03131643310189247,
+ -0.0003195490571670234,
+ -0.00020704510097857565,
+ -0.00019476900342851877,
+ -0.000254241080256179,
+ -0.0008852138998918235,
+ -0.001123035908676684,
-0.00010787858627736568,
- -0.0024356956128031015,
- -6.258291978156194e-05,
- -0.00048565989709459245,
- -0.0021678535267710686,
- -0.012607751414179802,
- -0.00023588736075907946,
- -0.11036524921655655,
- -0.5750182867050171,
- -0.00017176583060063422,
- -1.9862632751464844,
- -1.2351702451705933,
- -0.00037520044133998454,
- -0.0013566347770392895,
- -2.5152843591058627e-05,
- -2.1086387634277344,
- -7.917232990264893,
- -0.05708145350217819,
- -0.06208256632089615,
- -0.000644237850792706,
- -8.308542601298541e-05,
- -5.1276655197143555,
- -0.16815905272960663,
- -0.0012461524456739426,
- -5.94836674281396e-05,
- -3.559391736984253,
- -5.411561965942383,
- -0.022293083369731903,
- -0.0005644158809445798,
- -0.017552750185132027,
- -0.00038842763751745224,
- -1.8479862213134766,
- -0.004095145035535097,
- -11.830594062805176,
- -0.4279360771179199,
- -3.7062158584594727,
- -2.9457836151123047,
- -1.9491567611694336,
- -0.06489256024360657,
- -0.00013660451804753393,
- -0.012157151475548744,
- -0.22074609994888306,
- -0.021073833107948303,
- -0.00021300431399140507,
- -0.00017593742813915014,
- -0.00023672162205912173,
+ -0.002168924082070589,
+ -5.566918844124302e-05,
+ -0.0005374894244596362,
+ -0.002329495269805193,
+ -0.014262876473367214,
+ -0.00026675479602999985,
+ -0.1103726178407669,
+ -0.6316244602203369,
+ -0.00015686711412854493,
+ -1.986260175704956,
+ -1.2387672662734985,
+ -0.00037448544753715396,
+ -0.0013405631761997938,
+ -2.5629668016335927e-05,
+ -2.157573699951172,
+ -7.917330741882324,
+ -0.056897230446338654,
+ -0.06321369111537933,
+ -0.0006852186052128673,
+ -8.248942322097719e-05,
+ -5.13627815246582,
+ -0.18123063445091248,
+ -0.0012298409128561616,
+ -5.936446541454643e-05,
+ -3.441296100616455,
+ -5.40785026550293,
+ -0.022266386076807976,
+ -0.0006132629350759089,
+ -0.017490316182374954,
+ -0.0003912875254172832,
+ -1.8065340518951416,
+ -0.004082322586327791,
+ -11.884273529052734,
+ -0.38534700870513916,
+ -3.640864849090576,
+ -2.807778835296631,
+ -1.9425075054168701,
+ -0.06503599882125854,
+ -0.00013207517622504383,
+ -0.012374058365821838,
+ -0.22333350777626038,
+ -0.02102574147284031,
+ -0.00023850933939684182,
+ -0.00015496007108595222,
+ -0.00024125049822032452,
-0.0003091811086051166,
- -0.0014552014181390405,
- -0.0013881819322705269,
- -0.00015245705435518175,
- -0.002331279218196869,
- -5.4238757002167404e-05,
- -0.000668659748043865,
- -0.002430463209748268,
- -0.016187194734811783,
- -0.0002441108226776123,
- -1.4263010025024414,
- -0.30179885029792786,
- -0.0001770101225702092,
- -0.5045080184936523,
- -0.07310019433498383,
+ -0.0015404274454340339,
+ -0.0015067425556480885,
+ -0.00014733182615600526,
+ -0.0023534002248197794,
+ -6.806619057897478e-05,
+ -0.0006721144891344011,
+ -0.0022511158604174852,
+ -0.0161884855479002,
+ -0.00022396916756406426,
+ -1.423966407775879,
+ -0.30094146728515625,
+ -0.0001429217227268964,
+ -0.527908444404602,
+ -0.07172392308712006,
-8.022463589441031e-05,
- -0.002168329432606697,
- -2.3841574147809297e-05,
- -1.7808306217193604,
- -0.02828705683350563,
- -6.115249561844394e-05,
- -0.0008904544520191848,
- -0.0005335576133802533,
- -3.957670196541585e-05,
- -0.03801318258047104,
- -0.0003077510336879641,
- -0.0005035324720665812,
- -2.169585604860913e-05,
- -0.02271897904574871,
- -3.1709168979432434e-05,
- -0.0018041539005935192,
- -1.8358061424805783e-05,
- -0.005899516865611076,
- -1.1920922133867862e-06,
- -0.002030455507338047,
- -0.27544423937797546,
- -1.1146715879440308,
- -0.012286689132452011,
- -0.004974251613020897,
- -6.389413465512916e-05,
- -0.010529793798923492,
- -0.2302529364824295,
- -0.015527778305113316,
- -0.00019524575327523053,
- -6.389413465512916e-05,
- -0.00013815402053296566,
- -0.00018165845540352166,
- -0.0005564333405345678,
- -0.000959531927946955,
- -6.151010165922344e-05,
- -0.001416394836269319,
- -5.531158240046352e-05,
- -0.00035363141796551645,
+ -0.001690983772277832,
+ -2.455681169521995e-05,
+ -1.7886372804641724,
+ -0.02579452283680439,
+ -6.294052582234144e-05,
+ -0.000840906286612153,
+ -0.0005334384622983634,
+ -4.0649541915627196e-05,
+ -0.03599982336163521,
+ -0.0002740246127359569,
+ -0.0005314130103215575,
+ -1.9788545614574105e-05,
+ -0.023407327011227608,
+ -3.182837463100441e-05,
+ -0.0016145778354257345,
+ -1.4185804502631072e-05,
+ -0.005731695331633091,
+ -9.536738616588991e-07,
+ -0.0020591262727975845,
+ -0.28849226236343384,
+ -1.1147009134292603,
+ -0.011014376766979694,
+ -0.004826680291444063,
+ -7.10462118149735e-05,
+ -0.011995912529528141,
+ -0.229905366897583,
+ -0.017444511875510216,
+ -0.00017188502533826977,
+ -8.189342770492658e-05,
+ -0.0001358893496217206,
+ -0.0001652104256208986,
+ -0.0005519058904610574,
+ -0.0009560781763866544,
+ -5.94836674281396e-05,
+ -0.0014123475411906838,
+ -4.589452510117553e-05,
+ -0.00034564718953333795,
-0.0010683787986636162,
- -0.012577733024954796,
+ -0.01257891021668911,
-0.00023934361524879932,
- -0.06311207264661789,
- -0.972044050693512,
- -0.00019929806876461953,
- -1.6224243640899658,
- -0.8333836197853088,
- -0.00016592556494288146,
- -0.0008984343148767948,
- -1.6927575416048057e-05,
- -0.8844207525253296,
- -0.023736946284770966,
- -4.01811408996582,
- -1.6215615272521973,
- -0.33087965846061707,
- -0.0035197706893086433,
- -0.00024148885859176517,
- -3.0874729418428615e-05,
- -3.097301721572876,
- -0.030017103999853134,
- -0.0006585336523130536,
- -1.9430925021879375e-05,
- -0.49424058198928833,
- -0.0001401803019689396,
- -0.00554167665541172,
+ -0.05590301379561424,
+ -0.9510307312011719,
+ -0.00019429226813372225,
+ -1.614063024520874,
+ -0.8332399129867554,
+ -0.0001472126314183697,
+ -0.0008833082392811775,
+ -1.680836794548668e-05,
+ -0.9284455180168152,
+ -0.02347230538725853,
+ -4.0024094581604,
+ -1.6216294765472412,
+ -0.36597940325737,
+ -0.003108671400696039,
+ -0.0002397011558059603,
+ -3.266281055402942e-05,
+ -2.997518539428711,
+ -0.03157311677932739,
+ -0.0006841464783065021,
-1.9073304429184645e-05,
- -0.5312279462814331,
- -5.748266220092773,
- -11.324613571166992,
- -1.1340491771697998,
- -0.16082678735256195,
- -0.8938052654266357,
- -3.726792335510254,
- -0.8781039714813232,
- -0.00017355366435367614,
- -0.009945128113031387,
- -0.18626560270786285,
- -0.013042616657912731,
- -0.00010859376925509423,
- -7.199982064776123e-05,
- -0.00010871296399272978,
- -0.00017796363681554794,
- -0.00034767304896377027,
- -0.0006170752458274364,
- -3.0636318115284666e-05,
- -0.001077071763575077,
- -4.076874756719917e-05,
- -0.00024029705673456192,
- -0.000982159748673439,
- -0.02636047638952732,
- -0.00021920185827184469,
- -0.632880687713623,
- -0.06617539376020432,
- -0.00016318420239258558,
- -0.4156720042228699,
- -0.034620899707078934,
- -5.6622808187967166e-05,
- -0.0011695933062583208,
- -1.597391747054644e-05,
- -10.639490127563477,
- -0.24528348445892334,
- -0.06833283603191376,
- -0.0033608165103942156,
- -0.02616957761347294,
- -0.00036054308293387294,
- -3.099393507000059e-05,
- -4.044595241546631,
- -2.188387393951416,
- -0.32720163464546204,
- -0.00974209699779749,
- -0.0011126763420179486,
- -3.302042750874534e-05,
- -0.19868847727775574,
- -7.56950321374461e-05,
- -0.005233398173004389,
+ -0.47262415289878845,
+ -0.00013982271775603294,
+ -0.005081594921648502,
+ -1.9192511899746023e-05,
+ -2.0542843341827393,
+ -5.7443647384643555,
+ -1.3145296573638916,
+ -1.1412832736968994,
+ -0.157257080078125,
+ -0.8434399366378784,
+ -3.784353733062744,
+ -0.8877203464508057,
+ -0.0001784403866622597,
+ -0.009449399076402187,
+ -0.16757570207118988,
+ -0.012964131310582161,
+ -0.00010609064338495955,
+ -7.164221460698172e-05,
+ -0.00012051333033014089,
+ -0.00016819016309455037,
+ -0.0003797286772169173,
+ -0.0006151691195555031,
+ -3.766942609217949e-05,
+ -0.001149470335803926,
+ -4.207999518257566e-05,
+ -0.00023898606013972312,
+ -0.0010438475292176008,
+ -0.026360591873526573,
+ -0.0002196785935666412,
+ -0.632854163646698,
+ -0.06455802172422409,
+ -0.00020728347590193152,
+ -0.37605050206184387,
+ -0.03835982084274292,
+ -5.304672595229931e-05,
+ -0.001157805323600769,
+ -1.5616295058862306e-05,
+ -10.66722297668457,
+ -0.23820555210113525,
+ -0.07690379023551941,
+ -0.0033806574065238237,
+ -0.02564801648259163,
+ -0.0004828002711292356,
+ -3.123234637314454e-05,
+ -4.0631422996521,
+ -2.188638687133789,
+ -0.3299391269683838,
+ -0.009875488467514515,
+ -0.0011928117601200938,
+ -2.9682672902708873e-05,
+ -0.22228410840034485,
+ -7.366862701019272e-05,
+ -0.005895250476896763,
-3.158996332786046e-05,
- -1.839617371559143,
- -0.17654305696487427,
- -0.7875567078590393,
- -2.1537787914276123,
- -0.3631034195423126,
- -0.9216613173484802,
- -2.0036990642547607,
- -0.09243497252464294,
- -0.00010740180005086586,
- -0.018314307555556297,
- -0.208140030503273,
- -0.01576320081949234,
- -0.00013136000779923052,
- -7.390703103737906e-05,
+ -1.8109334707260132,
+ -0.17614930868148804,
+ -0.7870113849639893,
+ -2.1647791862487793,
+ -0.40118229389190674,
+ -0.9958465695381165,
+ -2.018829584121704,
+ -0.09130532294511795,
-0.00011264643399044871,
- -0.00017045476124621928,
- -0.0005171154043637216,
- -0.0005422552349045873,
- -3.349725011503324e-05,
- -0.0013309201458469033,
- -4.255681051290594e-05,
- -0.00023767507809679955,
+ -0.017000189051032066,
+ -0.20741263031959534,
+ -0.015425659716129303,
+ -0.0001161031104857102,
+ -7.772143726469949e-05,
+ -0.00012587709352374077,
+ -0.00021896349790040404,
+ -0.0004892344586551189,
+ -0.0006121906917542219,
+ -3.516612196108326e-05,
+ -0.001400562352500856,
+ -4.1126360883936286e-05,
+ -0.00023803261865396053,
-0.001095648156479001,
- -0.14277544617652893,
+ -0.142775759100914,
-0.00021371940965764225,
- -0.00032217081752605736,
- -0.35286909341812134,
- -0.0002668739762157202,
- -1.7962173223495483,
- -0.07211553305387497,
- -7.974783511599526e-05,
- -0.000621959799900651,
- -1.2874520507466514e-05,
- -1.9048426151275635,
- -0.022713735699653625,
- -3.9457496313843876e-05,
- -0.0005820487276650965,
- -0.0002401778765488416,
- -3.325883881188929e-05,
- -0.02081700973212719,
- -0.00022492263815365732,
- -0.0003299168893136084,
- -2.038458114839159e-05,
- -0.008293120190501213,
- -1.7404405298293568e-05,
- -0.0012493670219555497,
- -1.4424220353248529e-05,
- -0.0041636452078819275,
+ -0.0004058252670802176,
+ -0.35273799300193787,
+ -0.0002616301644593477,
+ -1.8941857814788818,
+ -0.07712067663669586,
+ -7.354942499659956e-05,
+ -0.000623032043222338,
+ -1.2993727978027891e-05,
+ -1.8126565217971802,
+ -0.02175048366189003,
+ -3.8980677345534787e-05,
+ -0.0005631053354591131,
+ -0.0002426806604489684,
+ -3.099393507000059e-05,
+ -0.02033376693725586,
+ -0.000196556793525815,
+ -0.0003359945258125663,
+ -2.1815061700181104e-05,
+ -0.008229749277234077,
+ -1.7523612768854946e-05,
+ -0.0011329191038385034,
+ -1.4066597032069694e-05,
+ -0.004424781538546085,
-8.344646857949556e-07,
- -0.0020267677027732134,
- -0.13429519534111023,
- -1.9221405982971191,
- -0.0093602379783988,
- -0.005981876514852047,
+ -0.002038069535046816,
+ -0.15088149905204773,
+ -1.922121286392212,
+ -0.009393423795700073,
+ -0.006058305036276579,
-5.817244164063595e-05,
- -0.019257837906479836,
- -0.27827900648117065,
- -0.01921457052230835,
- -0.0001652104256208986,
- -8.546940807718784e-05,
- -0.0001510267611593008,
- -0.00016366096679121256,
- -0.0002616301644593477,
- -0.0005458295345306396,
- -3.480850500636734e-05,
- -0.0010807631770148873,
- -3.7431014789035544e-05,
- -0.0003626880934461951,
- -0.0010880271438509226,
- -0.6327179670333862,
- -0.0002374367177253589,
- -0.020488178357481956,
- -0.10384052991867065,
- -0.0001971527235582471,
- -0.16368740797042847,
- -0.026392173022031784,
- -0.00012170527770649642,
- -0.0025978884659707546,
- -1.9430925021879375e-05,
- -7.9701642990112305,
- -1.6003714799880981,
- -0.2391909956932068,
- -0.000502817565575242,
- -4.9232225137529895e-05,
- -4.135532855987549,
- -0.06158669665455818,
- -0.00044371772673912346,
- -3.755022044060752e-05,
- -0.18109248578548431,
- -0.00010883215873036534,
- -0.006367869209498167,
- -7.748303323751315e-05,
- -5.440160751342773,
- -5.081888198852539,
- -0.19470839202404022,
- -2.9904420375823975,
- -2.4235076904296875,
- -0.032352350652217865,
- -0.00044907975825481117,
- -0.04121795669198036,
- -0.43260514736175537,
- -0.04605478420853615,
- -0.00023982033599168062,
- -0.0003178806509822607,
- -0.00017188502533826977,
- -0.00022468426323030144,
- -0.0003400462737772614,
- -0.0010152667528018355,
- -0.00011729506513802335,
- -0.001335324952378869,
- -4.8874615458771586e-05,
- -0.001257463125512004,
- -0.004097400698810816,
- -0.0008996253600344062,
- -0.0002967870968859643,
- -0.15579743683338165,
- -1.3731565475463867,
- -0.00023183519078884274,
- -2.0089190006256104,
- -3.441042423248291,
- -0.0006145734223537147,
- -0.0012832987122237682,
- -1.9550132492440753e-05,
- -1.731110692024231,
- -0.027068600058555603,
- -2.8266828060150146,
- -0.35935577750205994,
- -0.023644626140594482,
- -0.0005504761938937008,
- -0.00017951308109331876,
- -2.396077979938127e-05,
- -2.3206820487976074,
- -0.003744971938431263,
- -0.000205018965061754,
- -2.288792165927589e-05,
- -0.08958229422569275,
- -6.592056161025539e-05,
- -0.0021721357479691505,
- -3.0397906812140718e-05,
- -4.5939412117004395,
- -8.534799575805664,
- -3.483549118041992,
- -1.681600570678711,
- -0.7201917767524719,
- -0.530266284942627,
- -0.7154921293258667,
- -2.835704803466797,
- -0.0004451475979294628,
- -0.02453603409230709,
- -0.31538400053977966,
- -0.0156102878972888,
- -0.00013124081306159496,
- -8.892617915989831e-05,
- -9.738924563862383e-05,
- -0.0011036264477297664,
- -0.00030357998912222683,
- -0.0010406322544440627,
- -6.0437283536884934e-05,
- -0.0014225849881768227,
- -3.671578815556131e-05,
- -0.00044705410255119205,
- -0.005232923664152622,
- -0.0001565095444675535,
- -0.0003033416287507862,
- -0.18575794994831085,
- -0.14061033725738525,
- -0.0002706876548472792,
- -0.5223819017410278,
- -0.035896092653274536,
- -5.4834770708112046e-05,
- -0.0012011463986709714,
- -1.6569954823353328e-05,
- -1.681032657623291,
- -0.011652856133878231,
- -1.6569954823353328e-05,
- -0.00047469791024923325,
- -0.000256982195423916,
- -3.361645576660521e-05,
- -0.01372707262635231,
- -0.00014852374442853034,
- -0.00046695294440723956,
- -2.288792165927589e-05,
- -0.0034659572411328554,
- -1.3708974620385561e-05,
- -0.0015382850542664528,
- -8.702239938429557e-06,
- -0.003346678102388978,
- -7.152555099310121e-07,
- -0.000867467257194221,
- -0.02539108693599701,
- -1.0509589910507202,
- -0.002976156771183014,
- -0.005069141276180744,
- -5.590759246842936e-05,
- -0.015196850523352623,
- -0.3093729317188263,
- -0.02090352028608322,
- -0.00013958434283267707,
- -6.460934673668817e-05,
- -8.296622399939224e-05,
- -0.0004457433824427426,
- -0.0005041282274760306,
- -0.0011976935202255845,
- -4.2914423829643056e-05,
- -0.0011085085570812225,
- -4.160317621426657e-05,
- -0.0005018643569201231,
- -0.004558410029858351,
- -9.476689592702314e-05,
- -0.00037269797758199275,
- -0.11347992718219757,
- -0.450020968914032,
- -0.0003301552205812186,
- -2.8804092407226562,
+ -0.017619749531149864,
+ -0.2778732478618622,
+ -0.020725470036268234,
+ -0.00014554394874721766,
+ -0.00010918975021922961,
+ -0.00012015574611723423,
+ -0.00015615197480656207,
+ -0.000262106885202229,
+ -0.0005429700831882656,
+ -3.3854863431770355e-05,
+ -0.0009723940747790039,
+ -4.3748852476710454e-05,
+ -0.0003530356043484062,
+ -0.0011967408936470747,
+ -0.6327288150787354,
+ -0.00023803261865396053,
+ -0.018105357885360718,
+ -0.09554284811019897,
+ -0.00019762947340495884,
+ -0.1636878103017807,
+ -0.029755044728517532,
+ -0.00012027494085486978,
+ -0.0026007420383393764,
+ -1.9192511899746023e-05,
+ -7.907725811004639,
+ -1.5921536684036255,
+ -0.26656997203826904,
+ -0.0005077026435174048,
+ -4.5060096454108134e-05,
+ -4.135128021240234,
+ -0.05467758700251579,
+ -0.000444551813416183,
+ -3.158996332786046e-05,
+ -0.18111994862556458,
+ -0.00011205045302631333,
+ -0.0063499826937913895,
+ -6.949660019017756e-05,
+ -5.444925308227539,
+ -5.080565452575684,
+ -0.19182506203651428,
+ -2.987712860107422,
+ -2.41878604888916,
+ -0.033490728586912155,
+ -0.0004508670826908201,
+ -0.03954065591096878,
+ -0.473906010389328,
+ -0.04054016247391701,
+ -0.00023624490131624043,
+ -0.0002554328821133822,
+ -0.00017534149810671806,
+ -0.00022349244682118297,
+ -0.00038914260221645236,
+ -0.0009974034037441015,
+ -0.00011860620725201443,
+ -0.0013256819220259786,
+ -5.245071224635467e-05,
+ -0.0010975534096360207,
+ -0.004627709276974201,
+ -0.0009116546134464443,
+ -0.0003274143091402948,
+ -0.1729803830385208,
+ -1.3756903409957886,
+ -0.00020394629973452538,
+ -2.0139496326446533,
+ -3.4418716430664062,
+ -0.0005534547381103039,
+ -0.0013410394312813878,
+ -2.002696055569686e-05,
+ -1.7302016019821167,
+ -0.02661030925810337,
+ -2.8714237213134766,
+ -0.3817090392112732,
+ -0.02706013061106205,
+ -0.0005696581210941076,
+ -0.00017414960893802345,
+ -2.2291887944447808e-05,
+ -2.393955707550049,
+ -0.0037057793233543634,
+ -0.00020561488054227084,
+ -2.3007127310847864e-05,
+ -0.08967646211385727,
+ -8.654219709569588e-05,
+ -0.001972516765818,
+ -2.7894584491150454e-05,
+ -4.549469947814941,
+ -8.546380996704102,
+ -3.5779974460601807,
+ -1.6664830446243286,
+ -0.7325427532196045,
+ -0.5301830768585205,
+ -0.7084848880767822,
+ -2.8345694541931152,
+ -0.0004906642716377974,
+ -0.02661263197660446,
+ -0.31519249081611633,
+ -0.015796994790434837,
+ -0.00013219437096267939,
+ -8.83301836438477e-05,
+ -9.619726915843785e-05,
+ -0.001105650793761015,
+ -0.0003044141922146082,
+ -0.0010319390567019582,
+ -5.745722592109814e-05,
+ -0.001517812255769968,
+ -3.683499380713329e-05,
+ -0.00039176418795250356,
+ -0.005421933718025684,
+ -0.00012611546844709665,
+ -0.0003044141922146082,
+ -0.18575865030288696,
+ -0.1348363161087036,
+ -0.00024780540843494236,
+ -0.5202672481536865,
+ -0.031899940222501755,
+ -5.507317473529838e-05,
+ -0.0010099078062921762,
+ -1.6212332411669195e-05,
+ -1.692710518836975,
+ -0.011726495809853077,
+ -1.9788545614574105e-05,
+ -0.0004633783537428826,
+ -0.00025054652360267937,
+ -3.135155202471651e-05,
+ -0.013559030368924141,
+ -0.00015043080202303827,
+ -0.0004663571889977902,
+ -2.1576648578047752e-05,
+ -0.0036806000862270594,
+ -1.7881233361549675e-05,
+ -0.001532214693725109,
+ -8.583032467868179e-06,
+ -0.0033115092664957047,
+ -5.960462772236497e-07,
+ -0.0009004590683616698,
+ -0.025490565225481987,
+ -0.9851224422454834,
+ -0.0029891119338572025,
+ -0.004928938578814268,
+ -5.817244164063595e-05,
+ -0.015181821770966053,
+ -0.3092004656791687,
+ -0.020951267331838608,
+ -0.00013302871957421303,
+ -7.033100700937212e-05,
+ -8.165503095369786e-05,
+ -0.00044526674901135266,
+ -0.00041130665340460837,
+ -0.0012225781101733446,
+ -4.2437604861333966e-05,
+ -0.0009884715545922518,
+ -3.957670196541585e-05,
+ -0.0005015069036744535,
+ -0.005099741276353598,
+ -7.521823135903105e-05,
+ -0.00032908268622122705,
+ -0.10078251361846924,
+ -0.4542389512062073,
+ -0.000291662581730634,
+ -2.8981728553771973,
-0.15156973898410797,
- -6.246371776796877e-05,
- -0.000683074293192476,
+ -6.329813186312094e-05,
+ -0.0006799769471399486,
-1.3947389561508317e-05,
- -2.0683939456939697,
- -0.02846144698560238,
- -0.04469490796327591,
- -1.889275074005127,
- -0.0001255195093108341,
- -0.00011228884250158444,
- -2.4914430468925275e-05,
- -7.980701446533203,
- -0.39261865615844727,
- -1.6454169750213623,
- -0.0018256916664540768,
- -0.0003761537664104253,
- -2.5987286790041253e-05,
- -0.27152737975120544,
- -3.8742269680369645e-05,
- -0.002314033918082714,
+ -1.9687650203704834,
+ -0.032300759106874466,
+ -0.04720155894756317,
+ -1.8380577564239502,
+ -0.00012540031457319856,
+ -8.880697714630514e-05,
+ -2.4437606043647975e-05,
+ -8.028385162353516,
+ -0.3949255347251892,
+ -1.6445553302764893,
+ -0.0018342590192332864,
+ -0.00035768310772255063,
+ -3.0040289857424796e-05,
+ -0.26507413387298584,
+ -3.576214658096433e-05,
+ -0.0023099901154637337,
-5.364274329622276e-05,
- -5.172288417816162,
- -0.007181781344115734,
- -0.8884671330451965,
- -0.20681926608085632,
- -1.529428243637085,
- -2.335056781768799,
- -0.02583100087940693,
- -1.8960939645767212,
- -0.257112592458725,
- -0.1720065474510193,
- -8.284702198579907e-05,
- -0.011070851236581802,
- -0.16333311796188354,
- -0.01678428426384926,
- -0.00010024998482549563,
- -4.911301948595792e-05,
- -6.41325386823155e-05,
- -0.0003518439189065248,
- -0.0003983181086368859,
- -0.0007211944903247058,
- -2.253030106658116e-05,
- -0.0009076051646843553,
- -2.884823152271565e-05,
- -0.00033682872890494764,
- -0.01127432007342577,
- -5.113947918289341e-05,
- -0.0003095386200584471,
- -0.162703275680542,
- -0.12824533879756927,
- -0.0002037079248111695,
- -0.5378345251083374,
- -0.013359789736568928,
- -4.625213477993384e-05,
- -0.0007819455349817872,
- -1.2993727978027891e-05,
- -1.4531102180480957,
- -0.9376159310340881,
- -0.02013481967151165,
- -3.182837463100441e-05,
- -0.00028391621890477836,
- -0.0002040654799202457,
+ -5.063076496124268,
+ -0.00801442563533783,
+ -0.8903906941413879,
+ -0.20995336771011353,
+ -1.5289034843444824,
+ -2.236264944076538,
+ -0.025192562490701675,
+ -1.8960940837860107,
+ -0.27068760991096497,
+ -0.17904573678970337,
+ -8.797258487902582e-05,
+ -0.010375725105404854,
+ -0.16252535581588745,
+ -0.015135912224650383,
+ -0.0001006075763143599,
+ -4.565611743601039e-05,
+ -6.782778655178845e-05,
+ -0.0003756771038752049,
+ -0.0004158347437623888,
+ -0.0007250064518302679,
+ -2.3483953555114567e-05,
+ -0.0009037939016707242,
+ -3.6954195820726454e-05,
+ -0.0003305127320345491,
+ -0.011275380849838257,
+ -5.006664650863968e-05,
+ -0.00030858523678034544,
+ -0.1449013352394104,
+ -0.14131033420562744,
+ -0.00020358874462544918,
+ -0.5081860423088074,
+ -0.013270506635308266,
+ -4.23184028477408e-05,
+ -0.000786591088399291,
+ -1.3708974620385561e-05,
+ -1.4630959033966064,
+ -0.9440104961395264,
+ -0.02034369483590126,
+ -2.5152843591058627e-05,
+ -0.00028463127091526985,
+ -0.00020847532141488045,
-1.6212332411669195e-05,
- -6.006290435791016,
- -0.23482508957386017,
- -0.0003094194398727268,
- -3.2066785934148356e-05,
- -0.2894707918167114,
- -0.00010334911348763853,
- -0.003178308717906475,
- -4.8397800128441304e-05,
- -3.3541419506073,
- -5.274465084075928,
- -2.3055055141448975,
- -1.0987294912338257,
- -0.019666209816932678,
- -0.00022790218645241112,
- -0.016233760863542557,
- -0.2816391885280609,
- -0.028503969311714172,
- -0.0001358893496217206,
- -0.00010394509445177391,
- -8.856858039507642e-05,
- -0.00036137725692242384,
- -0.00029452278977259994,
- -0.0008922410197556019,
- -2.539125671319198e-05,
- -0.0011102947173640132,
- -3.40932747349143e-05,
- -0.0004843492351938039,
- -0.006350101437419653,
- -5.9602869441732764e-05,
- -0.0002796259068418294,
- -0.3986394703388214,
- -0.10029242187738419,
+ -6.000129699707031,
+ -0.2337796688079834,
-0.00024196557933464646,
- -1.9691603183746338,
- -0.7402586936950684,
- -7.056941103655845e-05,
- -0.0003618539194576442,
- -1.0371154530730564e-05,
- -1.4170231819152832,
- -0.008172051049768925,
- -1.3708974620385561e-05,
- -0.00041607304592616856,
- -0.00014888131408952177,
- -2.6464111215318553e-05,
- -0.018121162429451942,
- -0.00010764019680209458,
- -0.0002335037279408425,
- -2.3007127310847864e-05,
- -0.002049014437943697,
- -1.0609570381348021e-05,
- -0.0011868583969771862,
- -7.867782187531702e-06,
- -0.0018794744974002242,
- -5.960462772236497e-07,
- -0.0007434703293256462,
- -0.02911354973912239,
- -1.7920753955841064,
- -0.0026135831139981747,
- -0.00308870617300272,
- -3.659658250398934e-05,
- -0.010810147039592266,
- -0.20098412036895752,
- -0.01644638366997242,
- -0.00013207517622504383,
- -6.854299135738984e-05,
- -7.152301259338856e-05,
- -0.00024720950750634074,
- -0.00033468366018496454,
- -0.0010001424234360456,
- -5.054346183896996e-05,
- -0.0009557208395563066,
- -3.981510963058099e-05,
- -0.0004465774691198021,
- -0.011578621342778206,
- -7.211902266135439e-05,
+ -3.2305197237292305e-05,
+ -0.34043580293655396,
+ -0.00010227633902104571,
+ -0.003176526166498661,
+ -4.124556289752945e-05,
+ -3.3743977546691895,
+ -5.277707099914551,
+ -2.324122428894043,
+ -1.1028555631637573,
+ -0.018945103511214256,
+ -0.00020013237372040749,
+ -0.01631820574402809,
+ -0.2806614339351654,
+ -0.02678731456398964,
+ -0.00013696208770852536,
+ -0.0001070442158379592,
+ -8.999896090244874e-05,
+ -0.00036113892565481365,
+ -0.00037353215157054365,
+ -0.0008961713756434619,
+ -2.6225699912174605e-05,
+ -0.0011189873330295086,
+ -3.349725011503324e-05,
+ -0.000409161759307608,
+ -0.005737621337175369,
+ -6.151010165922344e-05,
+ -0.00027509720530360937,
+ -0.4400036633014679,
+ -0.09854183346033096,
-0.0002416080387774855,
- -0.09539440274238586,
- -0.057392168790102005,
+ -1.9590201377868652,
+ -0.7413398027420044,
+ -6.961580220377073e-05,
+ -0.0003890234511345625,
+ -1.0132738680113107e-05,
+ -1.3245630264282227,
+ -0.007269597612321377,
+ -1.3470558769768104e-05,
+ -0.0003660247311927378,
+ -0.00014959646796341985,
+ -2.4914430468925275e-05,
+ -0.016247248277068138,
+ -0.00010752100206445903,
+ -0.00029118589009158313,
+ -2.3007127310847864e-05,
+ -0.0019515770254656672,
+ -1.0132738680113107e-05,
+ -0.0011845960980281234,
+ -7.510157047363464e-06,
+ -0.0018861376447603106,
+ -4.768370445162873e-07,
+ -0.0007256020326167345,
+ -0.03265838325023651,
+ -1.8349244594573975,
+ -0.002593251410871744,
+ -0.0031409950461238623,
+ -3.85038583772257e-05,
+ -0.0108514204621315,
+ -0.20090337097644806,
+ -0.014486847445368767,
+ -0.00010358751023886725,
+ -7.176141662057489e-05,
+ -7.199982064776123e-05,
+ -0.00023040501400828362,
+ -0.00031275625224225223,
+ -0.0008927173912525177,
+ -4.1126360883936286e-05,
+ -0.000972036796156317,
+ -4.124556289752945e-05,
+ -0.00044907975825481117,
+ -0.01303955726325512,
+ -7.521823135903105e-05,
+ -0.00024196557933464646,
+ -0.09538453817367554,
+ -0.05263691768050194,
-0.0002840353990904987,
- -0.21088920533657074,
- -0.0078902468085289,
- -8.606540359323844e-05,
- -0.0007384672062471509,
- -1.3589766240329482e-05,
- -0.8148440718650818,
- -0.025661379098892212,
- -2.113894462585449,
- -0.01820814050734043,
- -0.0010720703285187483,
- -0.0002908283786382526,
- -0.00011181206355104223,
+ -0.19917485117912292,
+ -0.007841872982680798,
+ -8.546940807718784e-05,
+ -0.0007601470570079982,
+ -1.3232143828645349e-05,
+ -0.8271036148071289,
+ -0.024859817698597908,
+ -2.175525665283203,
+ -0.018430527299642563,
+ -0.0011276798322796822,
+ -0.00028713393840007484,
+ -0.00011169286881340668,
-1.9550132492440753e-05,
- -1.9963352680206299,
- -0.011685965582728386,
- -0.00010299152199877426,
- -1.6093124941107817e-05,
- -0.3427979350090027,
- -0.00010358751023886725,
- -0.002419165801256895,
- -5.07818695041351e-05,
- -9.356146812438965,
- -2.63590145111084,
- -0.0489899143576622,
- -0.429649293422699,
- -2.441277027130127,
- -0.09116854518651962,
- -1.7202471494674683,
- -1.2776923179626465,
- -1.2828468084335327,
- -0.1033272072672844,
- -0.013413426466286182,
- -0.00016091958968900144,
- -0.006314327474683523,
- -0.1650361269712448,
- -0.009155434556305408,
- -8.630380034446716e-05,
- -6.007967749610543e-05,
- -6.210611172718927e-05,
- -0.00027497802511788905,
- -0.0005628670332953334,
- -0.0008046964649111032,
- -4.160317621426657e-05,
- -0.0009633429581299424,
- -2.9444261599564925e-05,
- -0.0003147821989841759,
- -0.003070523263886571,
- -3.969590397900902e-05,
- -0.00025340684805996716,
- -0.16765674948692322,
- -0.220333993434906,
- -0.00025281094713136554,
- -1.6686129570007324,
- -0.08651255071163177,
- -7.4741430580616e-05,
- -0.00032062159152701497,
- -9.536697689327411e-06,
- -8.607754707336426,
- -2.7989468574523926,
- -0.006830438040196896,
- -0.00042500998824834824,
- -4.410646579344757e-05,
- -2.2325727939605713,
- -0.09642884135246277,
- -0.0005049622268415987,
- -1.4662635294371285e-05,
- -3.892613172531128,
- -0.0008376903715543449,
- -0.004279621876776218,
- -5.745722592109814e-05,
- -2.696786642074585,
- -0.44925373792648315,
- -0.37875908613204956,
- -0.27114248275756836,
- -1.023728609085083,
- -4.712882995605469,
- -1.415423035621643,
- -2.8054561614990234,
- -0.4460236430168152,
- -0.0005779979983344674,
- -0.02468189038336277,
- -0.30965328216552734,
- -0.02052520029246807,
- -0.00012730741582345217,
- -9.619726915843785e-05,
- -8.749579137656838e-05,
- -0.000350175570929423,
- -0.0003150205302517861,
- -0.0007310817018151283,
- -3.0636318115284666e-05,
- -0.0011643542675301433,
+ -1.969758152961731,
+ -0.010251722298562527,
+ -0.0001045410826918669,
+ -1.2874520507466514e-05,
+ -0.3413291573524475,
+ -0.00010489867418073118,
+ -0.0024227334652096033,
+ -5.61460001335945e-05,
+ -9.366641998291016,
+ -2.6359500885009766,
+ -0.05483028292655945,
+ -0.4293377697467804,
+ -2.4410433769226074,
+ -0.09051975607872009,
+ -1.7625658512115479,
+ -1.2852275371551514,
+ -1.2704421281814575,
+ -0.09968416392803192,
+ -0.011455003172159195,
+ -0.0001662831346038729,
+ -0.006489157676696777,
+ -0.14835502207279205,
+ -0.010291839018464088,
+ -8.654219709569588e-05,
+ -5.9960475482512265e-05,
+ -6.282132380874828e-05,
+ -0.0002961912250611931,
+ -0.0004487222759053111,
+ -0.0008200620068237185,
-3.2305197237292305e-05,
- -0.00026913834153674543,
- -0.011463016271591187,
- -5.411955135059543e-05,
- -0.00023231192608363926,
- -0.1063343733549118,
- -0.037034809589385986,
- -0.0001248043408850208,
- -0.3663400411605835,
- -0.01425135973840952,
- -5.376194530981593e-05,
- -0.000933926145080477,
- -1.4305012882687151e-05,
- -1.5244930982589722,
- -0.008558854460716248,
- -1.8358061424805783e-05,
- -0.0002698534226510674,
- -0.00022075122979003936,
+ -0.0008834273321554065,
+ -2.9205850296420977e-05,
+ -0.00031537804170511663,
+ -0.0030801494140177965,
+ -4.0649541915627196e-05,
+ -0.0002461368858348578,
+ -0.16866101324558258,
+ -0.2007424682378769,
+ -0.0002817710628733039,
+ -1.7346234321594238,
+ -0.09638641029596329,
+ -7.486063259420916e-05,
+ -0.0003212174342479557,
+ -9.417489309271332e-06,
+ -8.656828880310059,
+ -2.7857470512390137,
+ -0.007480112835764885,
+ -0.0005174727994017303,
+ -4.4225667807040736e-05,
+ -2.242187976837158,
+ -0.09535398334264755,
+ -0.0005254556890577078,
+ -1.4662635294371285e-05,
+ -3.884834051132202,
+ -0.00082923355512321,
+ -0.004454570822417736,
+ -5.781483559985645e-05,
+ -2.578787326812744,
+ -0.4335319995880127,
+ -0.3486354649066925,
+ -0.30149775743484497,
+ -1.103930950164795,
+ -4.709151268005371,
+ -1.359910249710083,
+ -2.805453300476074,
+ -0.40311169624328613,
+ -0.0005312938592396677,
+ -0.023901192471385002,
+ -0.31066638231277466,
+ -0.02003527246415615,
+ -0.00013159839727450162,
+ -9.7508447652217e-05,
+ -9.297892393078655e-05,
+ -0.0003295593778602779,
+ -0.0003240775258745998,
+ -0.0007370378007180989,
+ -2.9801878554280847e-05,
+ -0.0011587579501792789,
+ -3.158996332786046e-05,
+ -0.00026603974401950836,
+ -0.011517226696014404,
+ -5.3165931603871286e-05,
+ -0.00021288513380568475,
+ -0.11887810379266739,
+ -0.03703492507338524,
+ -0.00013326710904948413,
+ -0.33607202768325806,
+ -0.014220921322703362,
+ -5.364274329622276e-05,
+ -0.000709401210770011,
+ -1.5735502529423684e-05,
+ -1.5350909233093262,
+ -0.008527413941919804,
+ -1.5020257706055418e-05,
+ -0.00027247529942542315,
+ -0.0002302858338225633,
-3.576214658096433e-05,
- -0.01590365171432495,
- -0.00012706902634818107,
- -0.0002901133266277611,
- -2.2649508537142538e-05,
- -0.0032194233499467373,
- -1.1920858014491387e-05,
- -0.0013312773080542684,
- -8.22540732769994e-06,
- -0.001732040662318468,
+ -0.015459935180842876,
+ -0.00011562632425921038,
+ -0.00030489088385365903,
+ -1.8596476365928538e-05,
+ -0.0034538398031145334,
+ -9.417489309271332e-06,
+ -0.0013297295663505793,
+ -8.583032467868179e-06,
+ -0.0016905077500268817,
-4.768370445162873e-07,
- -0.0007115454645827413,
- -0.11607333272695541,
- -5.158000946044922,
- -0.00630958890542388,
- -0.006455875933170319,
- -3.886147169396281e-05,
- -0.007113605737686157,
- -0.16176439821720123,
- -0.01025608740746975,
- -9.321732068201527e-05,
- -5.435795901576057e-05,
- -7.70062324590981e-05,
- -0.0002002515539061278,
- -0.0003270567976869643,
- -0.0011002921964973211,
- -3.93382906622719e-05,
- -0.0009735850035212934,
+ -0.0007995745982043445,
+ -0.10312259942293167,
+ -5.149272918701172,
+ -0.0055675203911960125,
+ -0.007235039956867695,
+ -3.8980677345534787e-05,
+ -0.007169353775680065,
+ -0.17916721105575562,
+ -0.011401025578379631,
+ -9.035655966727063e-05,
+ -5.447716102935374e-05,
+ -7.593343616463244e-05,
+ -0.0002094287920044735,
+ -0.00034814971149899065,
+ -0.001026103738695383,
+ -4.2676016164477915e-05,
+ -0.0008634176338091493,
-4.076874756719917e-05,
- -0.00036042393185198307,
- -0.011448992416262627,
- -0.00010787858627736568,
- -0.00022289653134066612,
- -0.12719827890396118,
- -0.16689445078372955,
- -0.00029869386344216764,
- -1.129071831703186,
- -0.46998509764671326,
- -0.0001429217227268964,
- -0.0004334702098276466,
- -1.823885577323381e-05,
- -7.808990478515625,
- -0.6958405375480652,
- -0.0011538759572431445,
- -0.00010084597306558862,
+ -0.0003505330823827535,
+ -0.01161043718457222,
+ -0.00011455356434453279,
+ -0.0002146728802472353,
+ -0.14297251403331757,
+ -0.1774844229221344,
+ -0.00026425207033753395,
+ -1.2013825178146362,
+ -0.43942946195602417,
+ -0.00014280252798926085,
+ -0.00040356122190132737,
+ -1.8358061424805783e-05,
+ -7.889094352722168,
+ -0.7018532752990723,
+ -0.001150541938841343,
+ -9.965400386136025e-05,
-2.1815061700181104e-05,
- -3.412889242172241,
- -0.0024302254896610975,
- -0.1256120651960373,
- -0.0001486429391661659,
- -2.932505594799295e-05,
- -0.016119161620736122,
- -2.1219027985353023e-05,
- -0.0014936492079868913,
- -6.794906312279636e-06,
- -4.649867057800293,
- -0.42487168312072754,
- -1.3419163227081299,
- -0.3015914857387543,
- -0.00015341058315243572,
- -0.0032649326603859663,
- -0.11564143747091293,
- -0.00739337969571352,
- -5.8887653722194955e-05,
- -6.615896563744172e-05,
- -5.972207145532593e-05,
- -0.00020644917094614357,
- -0.000301673193462193,
- -0.0003761537664104253,
- -2.6702524337451905e-05,
- -0.0008094609947875142,
+ -3.3287391662597656,
+ -0.0022034193389117718,
+ -0.12544699013233185,
+ -0.00012003655137959868,
+ -2.90866428258596e-05,
+ -0.016096051782369614,
+ -2.2291887944447808e-05,
+ -0.0011514945654198527,
+ -6.437280717364047e-06,
+ -4.649921417236328,
+ -0.39337143301963806,
+ -1.3441730737686157,
+ -0.2537326514720917,
+ -0.00013386306818574667,
+ -0.0037233568727970123,
+ -0.12940432131290436,
+ -0.00740450294688344,
+ -5.9602869441732764e-05,
+ -5.829164365422912e-05,
+ -6.031808152329177e-05,
+ -0.00021002470748499036,
+ -0.0003022690652869642,
+ -0.00037698791129514575,
+ -2.586808113846928e-05,
+ -0.0008114859228953719,
-3.2305197237292305e-05,
- -0.0002474478678777814,
- -0.018454870209097862,
- -7.73638384998776e-05,
- -0.00022837892174720764,
- -0.04869883507490158,
- -0.02372216247022152,
- -0.0002051381452474743,
- -0.15266406536102295,
- -0.0037327392492443323,
+ -0.0002454218047205359,
+ -0.020887058228254318,
+ -7.879423355916515e-05,
+ -0.000226472009671852,
+ -0.043111126869916916,
+ -0.02126489207148552,
+ -0.0002022777043748647,
+ -0.15768083930015564,
+ -0.0037531666457653046,
-7.557583012385294e-05,
- -0.0005665604257956147,
+ -0.0005638201837427914,
+ -1.6093124941107817e-05,
+ -2.1179733276367188,
+ -0.025822287425398827,
+ -1.996516227722168,
+ -2.5962047576904297,
+ -0.14906328916549683,
+ -0.006481340620666742,
+ -0.00016091958968900144,
+ -1.7523612768854946e-05,
+ -0.8894410729408264,
+ -0.000976800569333136,
+ -0.0003759154351428151,
+ -1.2993727978027891e-05,
+ -0.07106750458478928,
+ -5.4238757002167404e-05,
+ -0.0015886317705735564,
+ -9.65590606938349e-06,
+ -4.332467079162598,
+ -6.821710586547852,
+ -3.370997428894043,
+ -1.8514046669006348,
+ -0.0004586121649481356,
+ -0.046185240149497986,
+ -0.06824732571840286,
+ -0.005082899704575539,
+ -9.059495641849935e-05,
+ -6.675497570540756e-05,
+ -8.630380034446716e-05,
+ -0.00025293012731708586,
+ -0.0003567297535482794,
+ -0.000636255950666964,
+ -3.123234637314454e-05,
+ -0.0012261499650776386,
+ -3.1470757676288486e-05,
+ -0.0002165798214264214,
+ -0.014394257217645645,
+ -0.00020966715237591416,
+ -0.00016699827392585576,
+ -7.259582343976945e-05,
+ -0.3280075490474701,
+ -0.00015531764074694365,
+ -2.5446929931640625,
+ -0.16115832328796387,
+ -0.0001174142598756589,
+ -0.0006781900301575661,
+ -2.2172682292875834e-05,
+ -1.6608847379684448,
+ -0.011715774424374104,
+ -1.9192511899746023e-05,
+ -0.0003228858404327184,
+ -0.00020346954988781363,
+ -3.111314072157256e-05,
+ -0.01600196771323681,
+ -0.00012587709352374077,
+ -0.00044741155579686165,
+ -2.1457441107486375e-05,
+ -0.007055368740111589,
-1.4662635294371285e-05,
- -2.1065256595611572,
- -0.02570541389286518,
- -2.0099081993103027,
- -2.7118430137634277,
- -0.1484161764383316,
- -0.007964756339788437,
- -0.00016342257731594145,
- -1.597391747054644e-05,
- -0.8920754194259644,
- -0.0009690594743005931,
+ -0.0012477001873776317,
+ -1.2278481335670222e-05,
+ -0.005830660928040743,
+ -4.768370445162873e-07,
+ -0.0016891986597329378,
+ -0.13408850133419037,
+ -3.2587075233459473,
+ -0.0026285641361027956,
+ -0.005026086233556271,
+ -3.516612196108326e-05,
+ -0.012018292210996151,
+ -0.18297713994979858,
+ -0.01304532214999199,
+ -0.00011467275908216834,
+ -8.5588610090781e-05,
+ -8.511180931236595e-05,
+ -0.00046957432641647756,
+ -0.00026556302327662706,
+ -0.0013249675976112485,
+ -9.023735765367746e-05,
+ -0.001595415873453021,
+ -5.4834770708112046e-05,
+ -0.0007428746903315187,
+ -0.6937626004219055,
+ -0.0007443041540682316,
-0.00029023250681348145,
- -1.2993727978027891e-05,
- -0.07993864268064499,
- -5.400034933700226e-05,
- -0.00158791767898947,
- -1.0609570381348021e-05,
- -4.331461429595947,
- -6.81968355178833,
- -3.366002082824707,
- -1.850673794746399,
- -0.00040391870425082743,
- -0.04611193388700485,
- -0.06791424006223679,
- -0.004945189692080021,
- -9.107174992095679e-05,
- -7.557583012385294e-05,
- -6.747018051100895e-05,
- -0.00024399164249189198,
- -0.000321336614433676,
- -0.0006528153317049146,
+ -0.00800082553178072,
+ -0.03613343834877014,
+ -0.0001821352052502334,
+ -0.08064392954111099,
+ -0.005144097842276096,
+ -0.00015829740732442588,
+ -0.002411079127341509,
+ -2.7179348762729205e-05,
+ -1.7293376922607422,
+ -0.7512492537498474,
+ -0.019934315234422684,
+ -1.7762025890988298e-05,
+ -0.0002302858338225633,
+ -0.00020954797219019383,
+ -2.706014311115723e-05,
+ -3.64949893951416,
+ -0.2676098644733429,
+ -9.60780744208023e-05,
+ -6.007967749610543e-05,
+ -0.13092315196990967,
+ -2.2291887944447808e-05,
+ -0.0036994845140725374,
-3.2782016205601394e-05,
- -0.0012151960982009768,
- -3.957670196541585e-05,
- -0.0002205128694185987,
- -0.016214992851018906,
- -0.00019095504831057042,
- -0.0001456631434848532,
- -7.712543447269127e-05,
- -0.33043625950813293,
- -0.00017629499780014157,
- -2.590480089187622,
- -0.16181793808937073,
- -0.00011646069469861686,
- -0.0006735440110787749,
- -2.109982233378105e-05,
- -1.6486821174621582,
- -0.01151864044368267,
- -1.8954096958623268e-05,
- -0.0003233625029679388,
- -0.00020644917094614357,
+ -2.8737730979919434,
+ -1.1481257677078247,
+ -2.7093358039855957,
+ -5.347209930419922,
+ -0.8316395878791809,
+ -0.00016532962035853416,
+ -0.8304288983345032,
+ -5.002294540405273,
+ -0.12272598594427109,
+ -0.008114584721624851,
+ -0.00037853704998269677,
+ -0.022495213896036148,
+ -0.5160256624221802,
+ -0.04559532180428505,
+ -0.0002172949316445738,
+ -0.00020132421923335642,
+ -0.00014661667228210717,
+ -0.0006960591417737305,
+ -0.0006668727728538215,
+ -0.0014327033422887325,
+ -6.615896563744172e-05,
+ -0.002525119110941887,
+ -4.5060096454108134e-05,
+ -0.0006106419023126364,
+ -0.000756216119043529,
+ -0.004223950207233429,
+ -0.0004472924047149718,
+ -0.1524876207113266,
+ -0.07079865783452988,
+ -0.00015948931104503572,
+ -2.295717239379883,
+ -0.44691845774650574,
+ -0.0001967951684491709,
+ -0.0006033745594322681,
+ -1.6927575416048057e-05,
+ -2.321779727935791,
+ -0.02190409228205681,
+ -0.030846793204545975,
+ -1.1255987882614136,
+ -0.0002225389762315899,
+ -0.00011848701251437888,
-3.111314072157256e-05,
- -0.017416512593626976,
- -0.00012766500003635883,
- -0.0003415954706724733,
- -2.13382354559144e-05,
- -0.006446637213230133,
- -1.823885577323381e-05,
- -0.0012438902631402016,
- -1.1205610462639015e-05,
- -0.006591127719730139,
- -7.152555099310121e-07,
- -0.0017049076268449426,
- -0.13135236501693726,
- -3.228759288787842,
- -0.002643782878294587,
- -0.004842340014874935,
- -3.480850500636734e-05,
- -0.010503842495381832,
- -0.16338221728801727,
- -0.011769498698413372,
- -0.00011574551899684593,
- -9.727005090098828e-05,
- -8.582700684200972e-05,
- -0.0004538459761533886,
- -0.00020740265608765185,
- -0.001342587056569755,
+ -3.019529342651367,
+ -0.005704077892005444,
+ -0.0004024887748528272,
+ -1.7881233361549675e-05,
+ -0.04252830147743225,
+ -0.00015853578224778175,
+ -0.005454538390040398,
+ -3.2543604902457446e-05,
+ -4.886053085327148,
+ -3.2839345932006836,
+ -0.17567910254001617,
+ -8.478743553161621,
+ -0.2580631673336029,
+ -0.00016127715934999287,
+ -0.004872709512710571,
+ -0.18758729100227356,
+ -0.024999113753437996,
+ -0.00015233787416946143,
+ -0.00012659224739763886,
-8.964136941358447e-05,
- -0.0014018717920407653,
- -4.935142715112306e-05,
- -0.0006431656656786799,
- -0.5765135288238525,
- -0.0009291622554883361,
- -0.00027998341829515994,
- -0.008964410983026028,
- -0.03303813934326172,
- -0.00018451895448379219,
- -0.07687719166278839,
- -0.00454594986513257,
- -0.00018439977429807186,
- -0.0023830130230635405,
- -2.706014311115723e-05,
- -1.8103313446044922,
- -0.7522969245910645,
- -0.022507335990667343,
- -2.074220174108632e-05,
- -0.00026222606538794935,
- -0.00020740265608765185,
- -2.706014311115723e-05,
- -3.700786590576172,
- -0.26737019419670105,
- -9.357491217087954e-05,
- -6.031808152329177e-05,
- -0.13705354928970337,
- -2.407998726994265e-05,
- -0.003684044349938631,
- -3.2782016205601394e-05,
- -2.9476141929626465,
- -1.1526018381118774,
- -2.6757259368896484,
- -5.31315279006958,
- -0.7695194482803345,
- -0.00014876213390380144,
- -0.8328413963317871,
- -5.100983142852783,
- -0.1275785118341446,
- -0.008235306479036808,
- -0.00037281715776771307,
- -0.02394961006939411,
- -0.5179875493049622,
- -0.04619366303086281,
- -0.00021705655672121793,
- -0.00021765247220173478,
- -0.0001461399078834802,
- -0.0007413261337205768,
- -0.0006660388899035752,
- -0.0015581621555611491,
- -6.8662193370983e-05,
- -0.002233869396150112,
- -4.494089080253616e-05,
- -0.0006101653561927378,
- -0.0006289887824095786,
- -0.0033358661457896233,
- -0.00045074793160893023,
- -0.15180595219135284,
- -0.07985830307006836,
- -0.00015937011630740017,
- -2.2477855682373047,
- -0.4471043348312378,
- -0.0001734344696160406,
- -0.0006040894077159464,
+ -0.0002896366349887103,
+ -0.0004929280839860439,
+ -0.0007099968497641385,
+ -5.5549986427649856e-05,
+ -0.0008168459753505886,
+ -2.9444261599564925e-05,
+ -0.0001512651506345719,
+ -0.00014768941036891192,
+ -0.0022286358289420605,
+ -0.00021073981770314276,
+ -0.11390721797943115,
+ -0.059136006981134415,
+ -0.00010716341057559475,
+ -0.403459757566452,
+ -0.01966036483645439,
+ -5.900685573578812e-05,
+ -0.0006700892699882388,
-1.680836794548668e-05,
- -2.318458080291748,
- -0.01888836920261383,
- -0.029085876420140266,
- -1.1253407001495361,
- -0.00021741411183029413,
- -0.00012003655137959868,
- -2.8013790142722428e-05,
- -3.1507949829101562,
- -0.005721264518797398,
- -0.00040904260822571814,
- -1.7881233361549675e-05,
- -0.04304421693086624,
- -0.0001591317413840443,
- -0.005429995711892843,
- -3.242440288886428e-05,
- -4.896542549133301,
- -3.2877321243286133,
- -0.17550288140773773,
- -8.526089668273926,
- -0.2559642493724823,
- -0.00015770144818816334,
- -0.004955509677529335,
- -0.20714037120342255,
- -0.023553114384412766,
- -0.00015496007108595222,
- -0.0001134808044298552,
- -9.250213042832911e-05,
- -0.000288087350782007,
- -0.0004409771354403347,
- -0.0007110689766705036,
- -4.6132929128361866e-05,
- -0.0009153467253781855,
- -3.433168603805825e-05,
- -0.00015484087634831667,
- -0.0001292145170737058,
- -0.0022287548054009676,
- -0.0002269487304147333,
- -0.11395295709371567,
- -0.05913611873984337,
- -8.356221951544285e-05,
- -0.4039720594882965,
- -0.019538793712854385,
- -5.924526340095326e-05,
- -0.0007176207727752626,
- -1.7881233361549675e-05,
- -1.6992816925048828,
- -0.004352619871497154,
- -6.6756979322235566e-06,
- -0.00017093151109293103,
- -0.0001284993631998077,
+ -1.6097873449325562,
+ -0.004351789131760597,
+ -6.318072337307967e-06,
+ -0.00017438798386137933,
+ -0.00012814177898690104,
-3.3378044463461265e-05,
- -0.013412484899163246,
- -8.713819261174649e-05,
- -0.0004928089329041541,
- -2.288792165927589e-05,
- -0.0012643685331568122,
+ -0.011513691395521164,
+ -8.618460560683161e-05,
+ -0.0004935238393954933,
+ -2.3007127310847864e-05,
+ -0.0012561534531414509,
-1.3351351299206726e-05,
- -0.0019104102393612266,
- -8.940656698541716e-06,
- -0.0033124599140137434,
+ -0.002182603348046541,
+ -9.059865078597795e-06,
+ -0.0031579884234815836,
-4.768370445162873e-07,
- -0.0009848987683653831,
- -0.07256874442100525,
- -1.7665941715240479,
- -0.00281461956910789,
- -0.0027610058896243572,
- -2.9682672902708873e-05,
- -0.0075036585330963135,
- -0.16648568212985992,
- -0.014109030365943909,
- -9.63164638960734e-05,
+ -0.0009639384225010872,
+ -0.07261596620082855,
+ -1.7404682636260986,
+ -0.002495629247277975,
+ -0.0031125929672271013,
+ -3.71926071238704e-05,
+ -0.006931781768798828,
+ -0.14954042434692383,
+ -0.014099039137363434,
+ -9.881961887003854e-05,
-6.603976362384856e-05,
- -7.331102824537084e-05,
- -0.0003323002893012017,
- -0.00042083943844772875,
- -0.0010620674584060907,
- -2.8609820219571702e-05,
- -0.000990257947705686,
- -4.029192859889008e-05,
- -0.0001541257370263338,
- -0.0001658063702052459,
- -0.0010433712741360068,
- -0.0002379134384682402,
- -0.08282912522554398,
- -0.1620505303144455,
+ -8.761498611420393e-05,
+ -0.00033396866638213396,
+ -0.0003670972364488989,
+ -0.0010602811817079782,
+ -3.504691630951129e-05,
+ -0.000943096645642072,
+ -3.635817120084539e-05,
+ -0.00014602071314584464,
+ -0.00018845213344320655,
+ -0.0012003129813820124,
+ -0.00024256148026324809,
+ -0.05201944336295128,
+ -0.14071618020534515,
-0.0001578206429257989,
- -1.9873682260513306,
- -0.03700195625424385,
- -8.594620157964528e-05,
- -0.00035232058144174516,
+ -2.140291213989258,
+ -0.04168115183711052,
+ -8.535020606359467e-05,
+ -0.0003649522550404072,
+ -2.253030106658116e-05,
+ -1.0512301921844482,
+ -0.012040670961141586,
+ -1.8452234268188477,
+ -0.047459427267313004,
+ -0.002904722234234214,
+ -0.0002766464895103127,
+ -7.939023635117337e-05,
+ -2.6225699912174605e-05,
+ -4.23543119430542,
+ -0.00821508839726448,
+ -7.4741430580616e-05,
+ -6.305972783593461e-05,
+ -5.399444580078125,
+ -0.0015713736647740006,
+ -0.0018196230521425605,
-2.90866428258596e-05,
- -1.0645859241485596,
- -0.012771833688020706,
- -1.8788448572158813,
- -0.04745874181389809,
- -0.0029150634072721004,
- -0.0002858230145648122,
- -8.082063141046092e-05,
- -2.8729025871143676e-05,
- -4.2793378829956055,
- -0.008196880109608173,
- -9.822363062994555e-05,
- -4.9470632802695036e-05,
- -5.399019241333008,
- -0.0015862513100728393,
- -0.0018035589018836617,
- -2.9444261599564925e-05,
- -3.8089842796325684,
- -1.3950530290603638,
- -0.17507919669151306,
- -4.1786346435546875,
- -9.410017013549805,
- -0.00014709345123264939,
- -2.16685152053833,
- -0.5008745193481445,
- -0.013433892279863358,
- -0.00029976642690598965,
- -0.006172403693199158,
- -0.22438427805900574,
- -0.015963135287165642,
- -0.00010489867418073118,
+ -3.806305170059204,
+ -1.2674520015716553,
+ -0.17437320947647095,
+ -4.168057441711426,
+ -9.435733795166016,
+ -0.00014697425649501383,
+ -2.166922092437744,
+ -0.5080004930496216,
+ -0.01319874543696642,
+ -0.0003053675754927099,
+ -0.006557967979460955,
+ -0.201018825173378,
+ -0.015605359338223934,
+ -0.00010835537250386551,
-7.426462980220094e-05,
- -6.890059739816934e-05,
- -0.0002874914789572358,
- -0.0004033228906337172,
- -0.0006624649395234883,
- -3.802703940891661e-05,
- -0.001104817260056734,
- -2.8967437174287625e-05,
- -0.000125281119835563,
- -0.00011634149996098131,
- -0.0016071987338364124,
- -0.0001752223033690825,
- -0.04927569255232811,
- -0.03999283164739609,
- -8.427741704508662e-05,
- -0.11036300659179688,
- -0.0022922686766833067,
+ -7.10462118149735e-05,
+ -0.0002862997353076935,
+ -0.0004632591735571623,
+ -0.0006872437661513686,
+ -3.6238969187252223e-05,
+ -0.0012379371328279376,
+ -2.9444261599564925e-05,
+ -0.00011765264935093,
+ -0.00010358751023886725,
+ -0.002063171239569783,
+ -0.0001770101225702092,
+ -0.0564521886408329,
+ -0.04466686397790909,
+ -8.237022848334163e-05,
+ -0.10021531581878662,
+ -0.0023065409623086452,
-5.125868119648658e-05,
- -0.0007711059297434986,
- -1.6569954823353328e-05,
- -1.1996040344238281,
- -6.017496585845947,
- -3.3771719932556152,
- -0.0015197168104350567,
- -0.0001720042055239901,
- -8.05822346592322e-05,
- -1.9701510667800903,
- -0.015215284191071987,
- -0.00046957432641647756,
- -4.5536911784438416e-05,
- -0.3501690626144409,
- -6.508615479106084e-05,
- -0.013412720523774624,
- -0.0002317160106031224,
- -10.721491813659668,
- -0.001794158248230815,
- -5.900764465332031,
- -0.05698608234524727,
- -1.9666205644607544,
- -0.34450024366378784,
- -0.24932177364826202,
- -1.1890842914581299,
- -0.9316995143890381,
- -0.5700393915176392,
- -0.18522746860980988,
- -0.08411185443401337,
- -0.00032610344351269305,
- -0.016760369762778282,
- -0.310769647359848,
- -0.04111167788505554,
- -0.00015889335190877318,
- -0.00011395759065635502,
- -0.00010418349120300263,
- -0.0003389737685211003,
- -0.0006182666402310133,
- -0.001039679627865553,
- -6.770858453819528e-05,
- -0.001258891774341464,
- -5.876845170860179e-05,
- -0.0003499372396618128,
- -0.00027724236133508384,
- -0.0029526231810450554,
- -0.0003165697562508285,
- -0.25983527302742004,
- -0.031029406934976578,
- -0.00018880968855228275,
- -0.7229459881782532,
- -0.42579957842826843,
- -0.00011705666838679463,
- -0.00047195740626193583,
- -2.3364747903542593e-05,
- -0.9790778160095215,
- -0.0029993331991136074,
- -5.125986263010418e-06,
- -0.00018690270371735096,
- -0.00016091958968900144,
- -3.755022044060752e-05,
- -0.00900670699775219,
- -8.642300235806033e-05,
- -0.0004804172203876078,
- -3.838465272565372e-05,
- -0.0015756584471091628,
- -1.168244216387393e-05,
- -0.001709667849354446,
- -1.0013530300057027e-05,
- -0.0022142434027045965,
- -5.960462772236497e-07,
- -0.0006964165368117392,
- -0.05425402522087097,
- -1.5528278350830078,
- -0.002721655648201704,
- -0.003402280155569315,
- -3.6477376852417365e-05,
- -0.007222968153655529,
- -0.14785511791706085,
- -0.013813492842018604,
- -0.00012063252506777644,
- -9.738924563862383e-05,
- -9.881961887003854e-05,
- -0.00025900822947733104,
- -0.00028236693469807506,
- -0.0010882653295993805,
- -4.446407547220588e-05,
- -0.0008232779800891876,
- -4.7801782784517854e-05,
- -0.0001911934232339263,
- -0.00020382710499688983,
- -0.0037347583565860987,
- -0.00023493390472140163,
- -0.016995148733258247,
- -0.028428077697753906,
- -0.00015054999676067382,
- -0.05958176776766777,
- -0.0022499265614897013,
- -8.928377064876258e-05,
- -0.0007566926069557667,
- -2.038458114839159e-05,
- -6.74626350402832,
- -4.031385898590088,
- -0.010314728133380413,
- -0.0005830018781125546,
+ -0.0009729895391501486,
+ -1.3470558769768104e-05,
+ -1.2119941711425781,
+ -5.893743515014648,
+ -3.3627188205718994,
+ -0.001396633917465806,
+ -0.000169382052263245,
+ -8.248942322097719e-05,
+ -1.9596554040908813,
+ -0.016238568350672722,
+ -0.0004700509598478675,
+ -4.541770613286644e-05,
+ -0.34185099601745605,
+ -5.3165931603871286e-05,
+ -0.013339557684957981,
+ -0.00023147765023168176,
+ -10.721809387207031,
+ -0.0019459851318970323,
+ -5.87699031829834,
+ -0.057010404765605927,
+ -1.955953598022461,
+ -0.3542339503765106,
+ -0.28593409061431885,
+ -1.1921348571777344,
+ -0.9411425590515137,
+ -0.5700472593307495,
+ -0.18382300436496735,
+ -0.08142641186714172,
+ -0.000379132863599807,
+ -0.01954592578113079,
+ -0.3134835958480835,
+ -0.03992319107055664,
-0.00016175392374861985,
- -4.279521817807108e-05,
- -4.910806655883789,
- -0.3867932856082916,
- -0.00020466140995267779,
- -2.455681169521995e-05,
- -0.40993309020996094,
- -3.075552376685664e-05,
- -0.002136925933882594,
- -1.5258672647178173e-05,
- -1.4743690490722656,
- -0.466409295797348,
- -2.986236095428467,
- -0.5145793557167053,
- -0.3861558437347412,
- -0.00023648326168768108,
- -0.060666244477033615,
- -0.0004374024283606559,
- -0.0032959445379674435,
- -0.003968104254454374,
- -0.0018072477541863918,
- -4.768258077092469e-05,
- -0.9783220291137695,
- -1.0383716821670532,
- -0.6705473065376282,
- -2.172899007797241,
- -0.1931028664112091,
- -0.05653104931116104,
- -0.0004231034545227885,
- -0.009201028384268284,
- -0.20085793733596802,
- -0.015902360901236534,
- -0.00013207517622504383,
- -0.00011634149996098131,
- -9.154854342341423e-05,
- -0.0002989322238136083,
- -0.000276765669696033,
- -0.0008761619683355093,
- -5.4596363042946905e-05,
- -0.0012877037515863776,
- -5.245071224635467e-05,
- -0.00014399446081370115,
- -0.00014304091746453196,
- -0.002012848388403654,
- -0.00026043839170597494,
- -0.050352130085229874,
- -0.016213351860642433,
- -0.00014923889830242842,
- -1.3270337581634521,
- -0.017757130786776543,
- -8.725739462533966e-05,
- -0.0003123987407889217,
- -2.3364747903542593e-05,
- -1.770219087600708,
- -0.027282992377877235,
- -1.7292673587799072,
- -1.5430668592453003,
- -0.09708311408758163,
- -0.06372363120317459,
- -0.00020180096908006817,
+ -0.00011359999916749075,
+ -0.00010442188795423135,
+ -0.0003408804477658123,
+ -0.0006943913758732378,
+ -0.0010699268896132708,
+ -6.353653589030728e-05,
+ -0.0012460333527997136,
+ -5.98412734689191e-05,
+ -0.0003594706067815423,
+ -0.00027581225731410086,
+ -0.002966054016724229,
+ -0.00029202012228779495,
+ -0.28885897994041443,
+ -0.028338506817817688,
+ -0.00018892886873800308,
+ -0.7341126203536987,
+ -0.36993715167045593,
+ -0.0001081169830285944,
+ -0.0004390706308186054,
+ -2.109982233378105e-05,
+ -0.9581791162490845,
+ -0.0029332491103559732,
+ -5.245195097813848e-06,
+ -0.0001817776501411572,
+ -0.00012373158824630082,
+ -3.802703940891661e-05,
+ -0.008104533888399601,
+ -8.415821503149346e-05,
+ -0.0005129451747052372,
+ -3.71926071238704e-05,
+ -0.0015855372184887528,
+ -1.490105023549404e-05,
+ -0.0021829602774232626,
+ -9.894321920000948e-06,
+ -0.0021203910000622272,
+ -5.960462772236497e-07,
+ -0.0006930810050107539,
+ -0.05077950283885002,
+ -1.5622856616973877,
+ -0.0027032282669097185,
+ -0.003475817386060953,
+ -3.361645576660521e-05,
+ -0.007188409101217985,
+ -0.14947564899921417,
+ -0.012775599956512451,
+ -0.00011979816190432757,
+ -8.67805938469246e-05,
+ -0.00010597144137136638,
+ -0.0002585315378382802,
+ -0.0003066784702241421,
+ -0.0010879080509766936,
+ -3.6000557884108275e-05,
+ -0.0008429311565123498,
-4.756337511935271e-05,
- -6.762560844421387,
- -0.11426064372062683,
- -0.0006945105269551277,
- -5.745722592109814e-05,
- -0.23964034020900726,
- -7.080780778778717e-05,
- -0.0019281383138149977,
- -0.00011657988943625242,
- -1.6634957790374756,
- -3.133596420288086,
- -1.06369948387146,
- -0.20282019674777985,
- -0.440325528383255,
- -2.2919445037841797,
- -2.6773011684417725,
- -2.4511003494262695,
- -2.022627353668213,
- -0.7157211899757385,
- -0.00033623288618400693,
- -0.006556428037583828,
- -0.18528789281845093,
- -0.010350123979151249,
- -9.691245941212401e-05,
- -9.941560711013153e-05,
- -0.0001062098381225951,
- -0.0002244459028588608,
- -0.0003002431185450405,
- -0.0003911683743353933,
+ -0.00019810620869975537,
+ -0.0002044230350293219,
+ -0.003736421000212431,
+ -0.0002343380037928,
+ -0.016844887286424637,
+ -0.026026494801044464,
+ -0.0001515035255579278,
+ -0.057031236588954926,
+ -0.0020061859395354986,
+ -8.904537389753386e-05,
+ -0.0007616956136189401,
+ -2.0861407392658293e-05,
+ -6.719094753265381,
+ -4.047027587890625,
+ -0.009809266775846481,
+ -0.0005860995152033865,
+ -0.00017188502533826977,
+ -4.2676016164477915e-05,
+ -4.904641151428223,
+ -0.34968045353889465,
+ -0.00019298121333122253,
+ -2.4914430468925275e-05,
+ -0.38600707054138184,
-3.158996332786046e-05,
- -0.0008713977294974029,
- -4.875540980719961e-05,
- -9.083335316972807e-05,
- -0.00013422065239865333,
- -0.0032467530108988285,
- -0.0002611534437164664,
- -0.011103743687272072,
- -0.014522447250783443,
- -0.0001003691868390888,
- -0.04763209819793701,
- -0.0015930355293676257,
- -8.880697714630514e-05,
- -0.0006610354175791144,
- -2.062299427052494e-05,
- -1.4736919403076172,
+ -0.002126338891685009,
+ -1.3708974620385561e-05,
+ -1.464985966682434,
+ -0.47231751680374146,
+ -2.9898223876953125,
+ -0.5146303176879883,
+ -0.38906559348106384,
+ -0.00023552982020191848,
+ -0.06135307624936104,
+ -0.00047267231275327504,
+ -0.003303905250504613,
+ -0.0036028018221259117,
+ -0.001978822285309434,
+ -4.660974445869215e-05,
+ -0.9326105117797852,
+ -1.0439518690109253,
+ -0.6865833401679993,
+ -2.1702210903167725,
+ -0.1792261153459549,
+ -0.056661274284124374,
+ -0.00042489083716645837,
+ -0.009141850285232067,
+ -0.2000894993543625,
+ -0.015266942791640759,
+ -0.00013195598148740828,
+ -0.00011300401820335537,
+ -9.047575440490618e-05,
+ -0.0002873722987715155,
+ -0.0002636561985127628,
+ -0.0008784249657765031,
+ -5.400034933700226e-05,
+ -0.0012316268403083086,
+ -5.209310256759636e-05,
+ -0.00014161060971673578,
+ -0.00014304091746453196,
+ -0.0022716925013810396,
+ -0.00026854246971197426,
+ -0.056410498917102814,
+ -0.016143208369612694,
+ -0.0001646144810365513,
+ -1.4203705787658691,
+ -0.020097782835364342,
+ -0.00011193125828867778,
+ -0.00031132620642893016,
+ -2.312633478140924e-05,
+ -1.7828859090805054,
+ -0.0273251011967659,
+ -1.7993203401565552,
+ -1.5256458520889282,
+ -0.10319281369447708,
+ -0.06369376927614212,
+ -0.00021634146105498075,
+ -4.9470632802695036e-05,
+ -6.747133731842041,
+ -0.1273529976606369,
+ -0.0006823595031164587,
+ -5.98412734689191e-05,
+ -0.22017809748649597,
+ -7.033100700937212e-05,
+ -0.0018899451242759824,
+ -0.00011717586312443018,
+ -1.6576848030090332,
+ -3.164034605026245,
+ -1.107333779335022,
+ -0.195778489112854,
+ -0.4392439126968384,
+ -2.308759927749634,
+ -2.6903655529022217,
+ -2.4507877826690674,
+ -2.0068390369415283,
+ -0.7151349186897278,
+ -0.0003326578007545322,
+ -0.006546480115503073,
+ -0.18491095304489136,
+ -0.010410410352051258,
+ -9.60780744208023e-05,
+ -0.00010573305189609528,
+ -0.00010561384988250211,
+ -0.00022551853908225894,
+ -0.00029130507027730346,
+ -0.00038580605178140104,
+ -3.0397906812140718e-05,
+ -0.001023364719003439,
+ -4.851700214203447e-05,
+ -0.00011336160969221964,
+ -0.0001045410826918669,
+ -0.0032530506141483784,
+ -0.0002673506969586015,
+ -0.008917978964745998,
+ -0.012813379056751728,
+ -0.0001006075763143599,
+ -0.047270920127630234,
+ -0.0018081996822729707,
+ -7.64102369430475e-05,
+ -0.0006587718962691724,
+ -2.1934269170742482e-05,
+ -1.4748883247375488,
+ -0.0014756753807887435,
+ -6.318072337307967e-06,
+ -0.00015400654228869826,
+ -0.00024816294899210334,
+ -4.31528314948082e-05,
+ -0.006116599775850773,
+ -8.296622399939224e-05,
+ -0.000395815703086555,
+ -4.088794958079234e-05,
-0.0015160269103944302,
- -5.722029527532868e-06,
- -0.0001426833332516253,
- -0.00025138078490272164,
- -4.303362584323622e-05,
- -0.006412051152437925,
- -8.177422569133341e-05,
- -0.0003953390696551651,
- -4.51792984677013e-05,
- -0.0015100754098966718,
- -1.0847986231965479e-05,
- -0.0021766559220850468,
- -1.3112935448589269e-05,
- -0.0017056216020137072,
+ -1.0609570381348021e-05,
+ -0.002186052966862917,
+ -1.4066597032069694e-05,
+ -0.0016952680889517069,
-5.960462772236497e-07,
- -0.00045658653834834695,
- -0.03380563110113144,
- -1.6861530542373657,
- -0.0011235122801735997,
- -0.0027228444814682007,
- -3.2543604902457446e-05,
- -0.0028300732374191284,
- -0.04190889745950699,
- -0.006303310859948397,
- -0.00010799778101500124,
- -7.295342220459133e-05,
- -6.90197994117625e-05,
- -0.0002094287920044735,
- -0.00017915551143232733,
- -0.0007649118197150528,
- -3.3854863431770355e-05,
- -0.0009750141180120409,
- -5.185469490243122e-05,
- -0.0001230164198204875,
- -0.00015221867943182588,
- -0.00366337806917727,
- -0.00027378625236451626,
- -0.00873471051454544,
- -0.014125015586614609,
- -0.00013779645087197423,
- -0.2786974012851715,
- -0.0429004468023777,
- -0.00015221867943182588,
- -0.0005259322933852673,
- -2.0861407392658293e-05,
- -7.4979376792907715,
- -2.5812153816223145,
- -0.0006475735572166741,
- -0.00032395837479270995,
- -4.3987260141875595e-05,
- -0.38662397861480713,
- -0.07727815210819244,
- -0.0005353448214009404,
- -6.210611172718927e-05,
- -0.10053620487451553,
- -4.51792984677013e-05,
- -0.004477594513446093,
- -3.0397906812140718e-05,
- -8.758296012878418,
- -0.4402102530002594,
- -0.2472418248653412,
- -0.5627955794334412,
- -0.042171675711870193,
- -0.03491748869419098,
- -5.941390514373779,
- -0.004192491993308067,
- -0.11302625387907028,
- -0.5369495153427124,
- -0.0003328961320221424,
- -0.0049365307204425335,
- -0.057854458689689636,
- -0.007558793295174837,
- -8.916457591112703e-05,
- -9.047575440490618e-05,
- -8.141662692651153e-05,
- -0.0006507901125587523,
- -0.00019464982324279845,
- -0.0006775943911634386,
- -2.3364747903542593e-05,
- -0.0012484145117923617,
- -5.447716102935374e-05,
- -0.00016425691137555987,
- -0.00019727191829588264,
- -0.012608221732079983,
- -0.00020859450160060078,
- -0.014227267354726791,
- -0.00964115560054779,
- -0.00013350549852475524,
- -0.03465360403060913,
- -0.0008008848526515067,
- -0.00010239553375868127,
- -0.0007454953738488257,
- -2.0861407392658293e-05,
- -2.182055950164795,
- -0.030151404440402985,
- -2.2387242317199707,
- -4.8748321533203125,
- -0.07910432666540146,
- -0.0014863882679492235,
- -0.00028081765049137175,
- -6.55629628454335e-05,
- -3.332869052886963,
- -4.393488883972168,
- -0.1467350423336029,
- -0.0036104037426412106,
- -0.0003040566807612777,
- -0.00010895135346800089,
- -0.2704607844352722,
- -3.6477376852417365e-05,
- -0.002591705648228526,
- -2.9682672902708873e-05,
- -4.947231292724609,
- -3.2159130573272705,
- -0.8367561101913452,
- -0.5556290149688721,
- -0.0002233732520835474,
- -0.0060651772655546665,
- -0.05365833640098572,
- -0.0071886456571519375,
- -9.63164638960734e-05,
- -0.00010072677832795307,
- -9.858122211880982e-05,
- -0.0003960540343541652,
- -0.0006039702566340566,
- -0.0006522196927107871,
- -1.811964830267243e-05,
- -0.001042775809764862,
- -3.790783375734463e-05,
- -0.00011514954530866817,
- -0.0001652104256208986,
- -0.05494809150695801,
+ -0.0005108005134388804,
+ -0.03380759060382843,
+ -1.533830165863037,
+ -0.001141611486673355,
+ -0.0027541108429431915,
+ -2.5748875486897305e-05,
+ -0.0027864461299031973,
+ -0.04172094166278839,
+ -0.0063011785969138145,
+ -0.00010632903286023065,
+ -6.937739817658439e-05,
+ -8.725739462533966e-05,
+ -0.00020954797219019383,
+ -0.00019786784832831472,
+ -0.0008331642020493746,
+ -2.8132995794294402e-05,
+ -0.0008716359734535217,
+ -5.2569914259947836e-05,
+ -0.0001294529065489769,
-0.00014506718434859067,
- -0.00021050144277978688,
- -0.014802505262196064,
- -0.00017915551143232733,
- -1.7102066278457642,
- -0.02825750596821308,
- -0.00011300401820335537,
- -0.0003519630990922451,
- -3.075552376685664e-05,
- -0.554995596408844,
- -0.0013822296168655157,
- -4.6491513785440475e-06,
- -0.00014482879487331957,
- -0.00019810620869975537,
- -3.504691630951129e-05,
- -0.006834581959992647,
- -6.389413465512916e-05,
- -0.0004396664153318852,
- -4.60137271147687e-05,
- -0.0012897277483716607,
+ -0.00366587215103209,
+ -0.0002796259068418294,
+ -0.008858784101903439,
+ -0.014021812006831169,
+ -0.0001358893496217206,
+ -0.25631147623062134,
+ -0.04293253645300865,
+ -0.0001731960946926847,
+ -0.0004926898400299251,
+ -1.9788545614574105e-05,
+ -7.50344181060791,
+ -2.5921072959899902,
+ -0.00064876489341259,
+ -0.00038985759601928294,
+ -4.184158387943171e-05,
+ -0.34731483459472656,
+ -0.07739675790071487,
+ -0.000596107158344239,
+ -6.103329360485077e-05,
+ -0.09895050525665283,
+ -4.2914423829643056e-05,
+ -0.004460386000573635,
+ -3.325883881188929e-05,
+ -8.770029067993164,
+ -0.41098108887672424,
+ -0.2444802075624466,
+ -0.5627976655960083,
+ -0.03904035687446594,
+ -0.0310506708920002,
+ -5.941311359405518,
+ -0.004236295353621244,
+ -0.11169589310884476,
+ -0.5418288707733154,
+ -0.00033444532891735435,
+ -0.0041848947294056416,
+ -0.052842844277620316,
+ -0.007783204782754183,
+ -8.83301836438477e-05,
+ -9.250213042832911e-05,
+ -8.129743218887597e-05,
+ -0.0005932478234171867,
+ -0.0002153879904653877,
+ -0.000596107158344239,
+ -2.3007127310847864e-05,
+ -0.0011825718684121966,
+ -5.447716102935374e-05,
+ -0.0001679517881711945,
+ -0.00020585325546562672,
+ -0.014266755431890488,
+ -0.0002109781780745834,
+ -0.009814225137233734,
+ -0.009642690420150757,
+ -0.0001045410826918669,
+ -0.033111490309238434,
+ -0.0008017186191864312,
+ -0.00010358751023886725,
+ -0.0007252446957863867,
+ -2.098061486321967e-05,
+ -2.1582047939300537,
+ -0.02757851965725422,
+ -2.2456743717193604,
+ -4.887596607208252,
+ -0.08968256413936615,
+ -0.001696577062830329,
+ -0.00027247529942542315,
+ -6.544376083184034e-05,
+ -3.444185495376587,
+ -4.3934831619262695,
+ -0.14697670936584473,
+ -0.003605177393183112,
+ -0.00031001531169749796,
+ -0.00011073929636040702,
+ -0.2715941369533539,
+ -3.659658250398934e-05,
+ -0.0025723245926201344,
+ -2.9205850296420977e-05,
+ -4.967594146728516,
+ -3.217968702316284,
+ -0.8693752288818359,
+ -0.5556545257568359,
+ -0.00022301571152638644,
+ -0.006144561339169741,
+ -0.048825882375240326,
+ -0.006224412936717272,
+ -7.56950321374461e-05,
+ -0.0001003691868390888,
+ -9.786603914108127e-05,
+ -0.0003904534096363932,
+ -0.0006033745594322681,
+ -0.0006375664379447699,
+ -1.9073304429184645e-05,
+ -0.0009195152088068426,
+ -3.802703940891661e-05,
+ -0.00012087091454304755,
+ -0.0001551984460093081,
+ -0.062036968767642975,
+ -0.00012468514614738524,
+ -0.00015269544383045286,
+ -0.015339847654104233,
+ -0.00016258825780823827,
+ -1.7683131694793701,
+ -0.028275582939386368,
+ -0.00011324241495458409,
+ -0.0003477922291494906,
+ -3.0636318115284666e-05,
+ -0.5490861535072327,
+ -0.0013725870521739125,
+ -4.768360213347478e-06,
+ -0.00015186110977083445,
+ -0.00019929806876461953,
+ -4.2676016164477915e-05,
+ -0.006663128267973661,
+ -7.247662142617628e-05,
+ -0.0004752936656586826,
+ -4.708655978902243e-05,
+ -0.0012991330586373806,
-1.1920858014491387e-05,
- -0.001943962532095611,
+ -0.0019418209558352828,
-1.4424220353248529e-05,
- -0.0016702761640772223,
- -5.960462772236497e-07,
- -0.0005274811992421746,
- -0.043414343148469925,
- -1.5102243423461914,
- -0.0018298563081771135,
- -0.0035949621815234423,
- -6.842378934379667e-05,
- -0.008245711214840412,
- -0.08723266422748566,
- -0.00939271505922079,
- -0.00011419598013162613,
- -0.0001230164198204875,
- -9.464769391342998e-05,
- -0.0002865380665753037,
- -0.0005069877952337265,
- -0.001016934053041041,
- -3.2305197237292305e-05,
- -0.0009629856795072556,
- -4.827859811484814e-05,
- -0.00021717573690693825,
- -0.00032848684350028634,
- -0.012733934447169304,
- -0.000196556793525815,
- -0.0012980615720152855,
- -0.0077531603164970875,
- -0.00012385078298393637,
- -0.01761084794998169,
- -0.0013621109537780285,
- -0.00011848701251437888,
- -0.0013394916895776987,
- -2.407998726994265e-05,
- -4.505744934082031,
- -1.2715730667114258,
- -0.0005052005290053785,
- -0.00024971229140646756,
- -3.635817120084539e-05,
- -4.3336405754089355,
- -0.0815289318561554,
- -0.028655847534537315,
- -0.00010430268594063818,
- -7.343022298300639e-05,
- -0.158114492893219,
+ -0.0016643255949020386,
+ -7.152555099310121e-07,
+ -0.0005178302526473999,
+ -0.04354605823755264,
+ -1.4898042678833008,
+ -0.002016298472881317,
+ -0.0036044646985828876,
+ -6.4490144723095e-05,
+ -0.008170513436198235,
+ -0.08714307099580765,
+ -0.009718603454530239,
+ -0.00011979816190432757,
+ -0.00013505500101018697,
+ -9.16677454370074e-05,
+ -0.00028701478731818497,
+ -0.0005073452484793961,
+ -0.0010153858456760645,
+ -4.255681051290594e-05,
+ -0.0009608419495634735,
+ -4.4225667807040736e-05,
+ -0.00021181246847845614,
+ -0.0003281293320469558,
+ -0.012738406658172607,
+ -0.00021324267436284572,
+ -0.0012884180760011077,
+ -0.007752331905066967,
+ -9.953480184776708e-05,
+ -0.01761096529662609,
+ -0.001328896265476942,
+ -0.00011085849109804258,
+ -0.0013175864005461335,
+ -2.3603161025675945e-05,
+ -4.519628524780273,
+ -1.2025973796844482,
+ -0.0004720765573438257,
+ -0.00025996167096309364,
+ -3.4689302992774174e-05,
+ -4.313958168029785,
+ -0.0816640704870224,
+ -0.030143653973937035,
+ -0.00011872540198964998,
+ -5.721882189391181e-05,
+ -0.15774711966514587,
-1.764281842042692e-05,
- -0.003166425507515669,
- -5.960446742392378e-06,
- -4.626138687133789,
- -0.5413240194320679,
- -11.11661148071289,
- -6.66420316696167,
- -0.5860735177993774,
- -1.0599334239959717,
- -2.200112819671631,
- -0.4268365502357483,
- -0.027302712202072144,
- -0.15124760568141937,
- -0.12854908406734467,
- -3.041227102279663,
- -0.026920655742287636,
- -0.0003856868715956807,
- -0.004746242426335812,
- -0.07085907459259033,
- -0.008411810733377934,
- -0.00010823617776622996,
- -5.972207145532593e-05,
- -5.507317473529838e-05,
- -0.00023850933939684182,
- -0.0004319211875554174,
- -0.0008380476501770318,
+ -0.004071637522429228,
+ -5.8412379075889476e-06,
+ -4.6149797439575195,
+ -0.5420227646827698,
+ -11.060333251953125,
+ -6.664154052734375,
+ -0.5853514671325684,
+ -1.061955451965332,
+ -2.200925588607788,
+ -0.430128276348114,
+ -0.027363842353224754,
+ -0.15776371955871582,
+ -0.12697036564350128,
+ -3.04177188873291,
+ -0.02755602076649666,
+ -0.00038258862332440913,
+ -0.004620708059519529,
+ -0.07068970054388046,
+ -0.00945353228598833,
+ -0.0001070442158379592,
+ -5.590759246842936e-05,
+ -5.411955135059543e-05,
+ -0.0002401778765488416,
+ -0.0004325169720686972,
+ -0.0008405489497818053,
-1.823885577323381e-05,
- -0.0009161804337054491,
- -3.683499380713329e-05,
- -0.00010918975021922961,
- -0.00016044282529037446,
- -0.0005364171229302883,
- -0.0001248043408850208,
- -0.10185468196868896,
- -0.02194770984351635,
- -0.00011252723925281316,
- -0.6942679286003113,
- -0.21981695294380188,
- -6.496695277746767e-05,
- -0.00030393750057555735,
- -2.13382354559144e-05,
- -3.1545064449310303,
- -0.021652380004525185,
- -0.02087036333978176,
- -0.89057856798172,
- -9.619726915843785e-05,
- -8.129743218887597e-05,
- -2.5152843591058627e-05,
- -4.086198806762695,
- -1.0591976642608643,
- -0.0020325970835983753,
- -4.1483970562694594e-05,
- -0.596172571182251,
- -3.242440288886428e-05,
- -0.0019346822518855333,
- -1.6927575416048057e-05,
- -3.4360618591308594,
- -2.4312753677368164,
- -1.9711253643035889,
- -4.358899116516113,
- -10.540913581848145,
- -5.990867614746094,
- -0.266180157661438,
- -0.000266278104390949,
- -0.003696990432217717,
- -0.03691418468952179,
- -0.005084204487502575,
- -7.73638384998776e-05,
+ -0.0009380945703014731,
+ -2.8729025871143676e-05,
+ -8.475421054754406e-05,
+ -0.0001662831346038729,
+ -0.0007327494095079601,
+ -0.00012575789878610522,
+ -0.09050984680652618,
+ -0.02065739408135414,
+ -0.00011276562872808427,
+ -0.6518867015838623,
+ -0.2065374255180359,
+ -6.568216485902667e-05,
+ -0.0003073934931308031,
+ -2.098061486321967e-05,
+ -3.1568620204925537,
+ -0.022977761924266815,
+ -0.021739285439252853,
+ -0.8908287286758423,
+ -8.415821503149346e-05,
+ -8.165503095369786e-05,
+ -2.7179348762729205e-05,
+ -4.090707778930664,
+ -1.056520700454712,
+ -0.00203557126224041,
+ -4.160317621426657e-05,
+ -0.5973647236824036,
+ -3.325883881188929e-05,
+ -0.0019417019793763757,
+ -1.6331539882230572e-05,
+ -3.437778949737549,
+ -2.4342737197875977,
+ -1.9497886896133423,
+ -4.228105545043945,
+ -10.535257339477539,
+ -5.980109691619873,
+ -0.26758477091789246,
+ -0.0002343380037928,
+ -0.0030076527036726475,
+ -0.03653198853135109,
+ -0.005291385110467672,
+ -9.846202738117427e-05,
-5.9960475482512265e-05,
- -6.12716976320371e-05,
- -0.0001915509783430025,
- -0.0004040378553327173,
- -0.0004508670826908201,
- -2.2172682292875834e-05,
- -0.0010245556477457285,
+ -6.222531374078244e-05,
+ -0.00018439977429807186,
+ -0.00040236959466710687,
+ -0.0004789874074049294,
+ -2.1576648578047752e-05,
+ -0.0009196343016810715,
-3.862306402879767e-05,
- -7.652943895664066e-05,
- -0.00010585224663373083,
- -0.00034791138023138046,
- -0.0001134808044298552,
- -0.009721791371703148,
- -0.01306991372257471,
- -7.86750388215296e-05,
- -0.06928819417953491,
- -0.0019708510953933,
- -8.070142939686775e-05,
- -0.0006008726777508855,
- -1.9550132492440753e-05,
- -1.2050050497055054,
- -0.0022362482268363237,
- -4.887569048150908e-06,
- -0.00016652150952722877,
- -0.0001282609737245366,
- -3.3854863431770355e-05,
- -0.005613160319626331,
- -4.935142715112306e-05,
- -0.00040618274942971766,
- -3.814624506048858e-05,
- -0.0012768696760758758,
- -6.9141146923357155e-06,
- -0.0021407324820756912,
+ -7.64102369430475e-05,
+ -0.00010609064338495955,
+ -0.00032491172896698117,
+ -0.00012206286191940308,
+ -0.00859454832971096,
+ -0.011765611357986927,
+ -9.548207890475169e-05,
+ -0.07109015434980392,
+ -0.00221234024502337,
+ -8.129743218887597e-05,
+ -0.0006108802044764161,
+ -1.6331539882230572e-05,
+ -1.2050546407699585,
+ -0.0022394596599042416,
+ -4.529942543740617e-06,
+ -0.0001578206429257989,
+ -0.0001357701694360003,
+ -3.373566141817719e-05,
+ -0.005778631195425987,
+ -5.018585216021165e-05,
+ -0.000390215078368783,
+ -3.0397906812140718e-05,
+ -0.0014029431622475386,
+ -8.821448318485636e-06,
+ -0.001671466277912259,
-1.0251946150674485e-05,
- -0.001328301033936441,
+ -0.0013266343157738447,
-4.768370445162873e-07,
- -0.00039104922325350344,
- -0.03403102979063988,
- -2.371554374694824,
- -0.0011966219171881676,
- -0.0017084777355194092,
- -1.2397689715726301e-05,
- -0.0012181727215647697,
- -0.027773091569542885,
- -0.004225967917591333,
- -7.688703772146255e-05,
+ -0.0003953390696551651,
+ -0.031116193160414696,
+ -2.2898995876312256,
+ -0.0012049565557390451,
+ -0.0017442979151383042,
+ -1.2159273865108844e-05,
+ -0.0014422263484448195,
+ -0.027814019471406937,
+ -0.0041466690599918365,
+ -7.724463648628443e-05,
-10.750052452087402,
- -0.09749454259872437,
- -0.0398833304643631,
- -0.05019160360097885,
- -0.02639356628060341,
- -0.001116844010539353,
- -0.010394011624157429,
- -0.0002687808300834149,
- -0.0412154421210289,
- -0.17060238122940063,
- -0.44570907950401306,
- -0.001759529928676784,
- -0.8481433987617493,
- -3.9174411296844482,
- -0.0011847150744870305,
- -1.8217713832855225,
- -1.9833719730377197,
- -0.0033980030566453934,
- -0.022340646013617516,
- -0.0005044856225140393,
- -11.916642189025879,
- -2.2062525749206543,
- -0.011109520681202412,
- -0.0025012181140482426,
- -0.00047839165199548006,
- -10.590877532958984,
- -5.111791133880615,
- -0.8751921653747559,
- -0.19319908320903778,
- -0.04376664385199547,
- -0.019606946036219597,
- -0.00042000532266683877,
- -9.505635261535645,
- -0.07715455442667007,
- -0.005082899704575539,
- -0.04224858805537224,
- -0.03572046384215355,
- -0.0011238694423809648,
- -5.344630241394043,
- -3.876430034637451,
- -12.252359390258789,
- -4.9860382080078125,
- -2.668943405151367,
- -1.16416597366333,
- -2.514509677886963,
- -2.5190258026123047,
- -14.754651069641113,
- -5.655267715454102,
- -6.61380672454834,
- -4.71486234664917,
- -0.5776815414428711,
- -1.3986684083938599,
- -2.637193202972412,
- -1.1604831218719482,
- -1.4959537982940674,
- -0.004402587655931711,
- -0.5065803527832031,
- -3.3776161670684814,
- -0.7203826308250427,
- -0.02161656692624092,
- -0.819121241569519,
- -0.04418942704796791,
- -1.7282390594482422,
- -0.05629342794418335,
- -0.008580365218222141,
- -0.000747877755202353,
- -0.013715313747525215,
- -0.00015138434537220746,
- -0.006047403905540705,
- -0.024643857032060623,
- -0.05186835676431656,
- -0.0005345107638277113,
- -0.10883784294128418,
- -1.3612172603607178,
- -0.0003692421887535602,
- -1.357957363128662,
- -0.05831316113471985,
- -0.00040570611599832773,
- -0.0035074164625257254,
- -6.437094270950183e-05,
- -1.7280149459838867,
- -0.026309387758374214,
- -2.3754658699035645,
- -0.05959097668528557,
- -0.0019271865021437407,
- -0.0006563892820850015,
- -0.00038985759601928294,
- -0.00013529339048545808,
- -6.799666881561279,
- -0.4319588541984558,
- -0.0018134353449568152,
- -0.00010084597306558862,
- -3.564793109893799,
- -0.0016862234333530068,
- -0.007215393707156181,
- -0.00018916724366135895,
- -4.893386363983154,
- -0.7495713233947754,
- -0.04057759419083595,
- -0.16563259065151215,
- -3.7694530487060547,
- -0.7686876654624939,
- -0.02867751009762287,
- -3.4293549060821533,
- -1.9938279390335083,
- -3.87074613571167,
- -7.779223918914795,
- -0.11301646381616592,
- -0.0007675323868170381,
- -0.0353383906185627,
- -0.5969783663749695,
- -0.03809810429811478,
- -0.00048828122089616954,
- -0.024168511852622032,
- -0.0024346255231648684,
- -0.006569692399352789,
- -0.002209961414337158,
- -0.001069331425242126,
- -7.819823804311454e-05,
- -0.0029135181102901697,
- -4.60137271147687e-05,
- -0.0003582789213396609,
+ -0.10282254964113235,
+ -0.039773814380168915,
+ -0.05065406113862991,
+ -0.0271708145737648,
+ -0.0011447074357420206,
+ -0.00991420354694128,
+ -0.00027700403006747365,
+ -0.03729129210114479,
+ -0.16962914168834686,
+ -0.44594845175743103,
+ -0.0018586517544463277,
+ -0.8312537670135498,
+ -4.031896114349365,
+ -0.001257463125512004,
+ -1.821771502494812,
+ -1.9813495874404907,
+ -0.0036345156840980053,
+ -0.022193634882569313,
+ -0.00048232366680167615,
+ -11.87901496887207,
+ -2.2126622200012207,
+ -0.012645655311644077,
+ -0.0024358145892620087,
+ -0.0005171154043637216,
+ -10.64317512512207,
+ -5.122398376464844,
+ -0.8483930826187134,
+ -0.21630460023880005,
+ -0.04415018856525421,
+ -0.020105261355638504,
+ -0.0003734129713848233,
+ -9.731037139892578,
+ -0.0780535638332367,
+ -0.005767015740275383,
+ -0.042193274945020676,
+ -0.03950341418385506,
+ -0.0010259846458211541,
+ -5.368424892425537,
+ -3.8717446327209473,
+ -12.254188537597656,
+ -4.985353469848633,
+ -2.6314144134521484,
+ -1.1586227416992188,
+ -2.499390125274658,
+ -2.519989013671875,
+ -14.693689346313477,
+ -5.696416854858398,
+ -6.6406755447387695,
+ -4.719296455383301,
+ -0.5249666571617126,
+ -1.4039437770843506,
+ -2.6573104858398438,
+ -1.2030140161514282,
+ -1.6004329919815063,
+ -0.00434051314368844,
+ -0.5139809846878052,
+ -3.259213447570801,
+ -0.7186939716339111,
+ -0.02127527818083763,
+ -0.8464836478233337,
+ -0.040414463728666306,
+ -1.7282320261001587,
+ -0.05802331864833832,
+ -0.008705994114279747,
+ -0.0007096394547261298,
+ -0.013228391297161579,
+ -0.00014125302550382912,
+ -0.00532672181725502,
+ -0.02297193743288517,
+ -0.05059921368956566,
+ -0.0005256939912214875,
+ -0.1093946099281311,
+ -1.3967794179916382,
+ -0.0003780603874474764,
+ -1.3749324083328247,
+ -0.053297750651836395,
+ -0.00040451448876410723,
+ -0.003983658738434315,
+ -6.425174069590867e-05,
+ -1.725297451019287,
+ -0.02523615211248398,
+ -2.2629284858703613,
+ -0.052103862166404724,
+ -0.0019268295727670193,
+ -0.0006641327636316419,
+ -0.0003833036171272397,
+ -0.00013553177996072918,
+ -6.900384426116943,
+ -0.39039379358291626,
+ -0.001796538126654923,
+ -0.00010990492592100054,
+ -3.5404791831970215,
+ -0.001702646492049098,
+ -0.007246401626616716,
+ -0.0001892864383989945,
+ -4.79708194732666,
+ -0.8172286748886108,
+ -0.04497222974896431,
+ -0.16637395322322845,
+ -3.8862829208374023,
+ -0.7114430069923401,
+ -0.02864970825612545,
+ -3.3703126907348633,
+ -1.9768058061599731,
+ -3.8709847927093506,
+ -7.714934825897217,
+ -0.10974952578544617,
+ -0.0008415018673986197,
+ -0.040607355535030365,
+ -0.5969805121421814,
+ -0.03528913855552673,
+ -0.0004911408759653568,
+ -0.024118708446621895,
+ -0.0022870355751365423,
+ -0.006580706220120192,
+ -0.002045564353466034,
+ -0.0012041230220347643,
+ -7.509902934543788e-05,
+ -0.0027602927293628454,
+ -4.8993817472364753e-05,
+ -0.0004024887748528272,
-0.001116367639042437,
-0.002629396505653858,
- -0.0002420847595203668,
- -0.17575480043888092,
- -0.017076482996344566,
- -0.0001431601122021675,
- -0.10536163300275803,
- -0.00507151335477829,
- -0.00011181206355104223,
- -0.0018749530427157879,
- -2.3603161025675945e-05,
- -0.8358778953552246,
- -0.002124911407008767,
- -9.894321920000948e-06,
- -0.00019214690837543458,
- -0.0002456601650919765,
- -3.516612196108326e-05,
- -0.008302814327180386,
- -0.00010895135346800089,
- -0.0006008726777508855,
- -3.2543604902457446e-05,
- -0.006115178111940622,
- -2.1219027985353023e-05,
- -0.0036275077145546675,
+ -0.00023934361524879932,
+ -0.1934685856103897,
+ -0.017828447744250298,
+ -0.00013469743134919554,
+ -0.10506075620651245,
+ -0.003996481653302908,
+ -0.00011205045302631333,
+ -0.0016538526397198439,
+ -2.3841574147809297e-05,
+ -0.8225477337837219,
+ -0.002133119385689497,
+ -1.07287787614041e-05,
+ -0.00020001317898277193,
+ -0.0002526917669456452,
+ -3.349725011503324e-05,
+ -0.007122246082872152,
+ -9.917721035890281e-05,
+ -0.0005891970940865576,
+ -3.290122185717337e-05,
+ -0.006138637196272612,
-1.7165990357170813e-05,
- -0.003067908575758338,
- -9.536738616588991e-07,
- -0.0006908176001161337,
- -0.02611708454787731,
- -1.3316965103149414,
- -0.003817296586930752,
- -0.006795391906052828,
- -4.684815212385729e-05,
- -0.007690228521823883,
- -0.14891591668128967,
- -0.013032732531428337,
- -0.0002714027068577707,
- -0.011644137091934681,
- -0.00091856240760535,
- -0.0013096098555251956,
- -0.0007771808886900544,
- -0.0009541726321913302,
- -5.638440416078083e-05,
- -0.0014388932613655925,
- -5.018585216021165e-05,
- -0.00020930961181875318,
- -0.0006467396160587668,
- -0.0013236580416560173,
- -0.00019333878299221396,
- -0.05778864026069641,
- -0.023562893271446228,
- -0.0001699779968475923,
- -0.4867134690284729,
- -0.17518886923789978,
- -6.01988795096986e-05,
- -0.00056429672986269,
- -2.396077979938127e-05,
- -10.983257293701172,
- -3.4146568775177,
- -0.007948435842990875,
- -0.005365850869566202,
- -0.00041166413575410843,
- -6.0437283536884934e-05,
- -1.4208624362945557,
- -0.014981495216488838,
- -0.00011193125828867778,
- -2.95634672511369e-05,
- -0.3359139859676361,
- -6.425174069590867e-05,
- -0.0036992470268160105,
- -1.7523612768854946e-05,
- -1.6273220777511597,
- -12.038379669189453,
- -1.8510823249816895,
- -4.6685380935668945,
- -1.03892183303833,
- -3.5619592666625977,
- -3.119525194168091,
- -8.74183177947998,
- -0.1955474466085434,
- -0.00022349244682118297,
- -0.005337630398571491,
- -0.07253769785165787,
- -0.0067605809308588505,
- -0.00018821375851985067,
- -0.01270250789821148,
- -0.0005373702733777463,
- -0.0013699679402634501,
- -0.0009596510208211839,
- -0.0003953390696551651,
+ -0.003627151483669877,
-1.7165990357170813e-05,
- -0.0010408704401925206,
- -3.4450891689630225e-05,
- -0.00011038171214750037,
- -0.00048351517762057483,
- -0.0015029336791485548,
- -0.00013958434283267707,
- -0.027578983455896378,
+ -0.003021558281034231,
+ -9.536738616588991e-07,
+ -0.0008679436868987978,
+ -0.026252955198287964,
+ -1.3258216381072998,
+ -0.0039386567659676075,
+ -0.006899935193359852,
+ -4.768258077092469e-05,
+ -0.0075222342275083065,
+ -0.14876028895378113,
+ -0.013457768596708775,
+ -0.00027247529942542315,
+ -0.014900107868015766,
+ -0.0009192770230583847,
+ -0.001426989445462823,
+ -0.000993116176687181,
+ -0.0009552444680593908,
+ -5.411955135059543e-05,
+ -0.0014282988850027323,
+ -4.3748852476710454e-05,
+ -0.0002076410164590925,
+ -0.0006455483380705118,
+ -0.0016991952434182167,
+ -0.00020656836568377912,
+ -0.05130963772535324,
+ -0.025677645578980446,
+ -0.00017248096992261708,
+ -0.4693552255630493,
+ -0.15605029463768005,
+ -5.686121585313231e-05,
+ -0.0005718026659451425,
+ -2.2053474822314456e-05,
+ -10.931877136230469,
+ -3.32647442817688,
+ -0.007924545556306839,
+ -0.005380197893828154,
+ -0.0003916450368706137,
+ -6.067568756407127e-05,
+ -1.4296157360076904,
+ -0.014987719245254993,
+ -0.00010942813969450071,
+ -2.7656173188006505e-05,
+ -0.31083944439888,
+ -6.16293036728166e-05,
+ -0.003689032746478915,
+ -1.764281842042692e-05,
+ -1.643222451210022,
+ -12.110017776489258,
+ -1.827063798904419,
+ -4.666322231292725,
+ -1.1210403442382812,
+ -3.5616214275360107,
+ -3.1083669662475586,
+ -8.859210014343262,
+ -0.21794311702251434,
+ -0.00023409964342135936,
+ -0.006090770475566387,
+ -0.0732051208615303,
+ -0.006583192851394415,
+ -0.00019178935326635838,
+ -0.014276274479925632,
+ -0.000547378440387547,
+ -0.0012900849105790257,
+ -0.0010100268991664052,
+ -0.0003967689990531653,
+ -2.1576648578047752e-05,
+ -0.001176380319520831,
+ -3.349725011503324e-05,
+ -0.00011359999916749075,
+ -0.0004829194222111255,
+ -0.0013847296359017491,
+ -0.00014411364099942148,
+ -0.027071036398410797,
-0.02192368544638157,
-8.141662692651153e-05,
- -0.11562338471412659,
- -0.0031276855152100325,
- -6.5205356804654e-05,
- -0.0007344171172007918,
- -2.1457441107486375e-05,
- -1.4039907455444336,
- -0.8585066795349121,
- -0.12097951024770737,
- -4.9232225137529895e-05,
- -0.00045503751607611775,
- -0.0001479277852922678,
- -2.8967437174287625e-05,
- -3.316209316253662,
- -0.22754307091236115,
- -0.037047676742076874,
- -0.00010632903286023065,
- -5.602679812000133e-05,
- -0.10701240599155426,
- -2.1815061700181104e-05,
- -0.0025769618805497885,
- -2.932505594799295e-05,
- -2.9098081588745117,
- -0.23772671818733215,
- -2.5728368759155273,
- -1.0628935098648071,
- -0.569791853427887,
- -1.5512791872024536,
- -0.22174018621444702,
- -0.2053954154253006,
- -0.668795108795166,
- -0.00032574593205936253,
- -0.005275258328765631,
- -0.17121490836143494,
- -0.01520049013197422,
- -0.00027164106722921133,
- -0.018145864829421043,
- -0.0008275659638457,
- -0.0013598490040749311,
- -0.0007223857101053,
- -0.0005415403284132481,
- -3.075552376685664e-05,
- -0.0016680150292813778,
- -4.124556289752945e-05,
- -0.00020203932945150882,
- -0.0005315321614034474,
- -0.0016384999034926295,
- -0.000169382052263245,
- -0.01945134624838829,
- -0.018782030791044235,
- -0.0001429217227268964,
- -1.4800734519958496,
- -0.046756841242313385,
- -9.667406266089529e-05,
- -0.0005499995895661414,
- -1.728519782773219e-05,
- -0.6545608639717102,
- -0.0013740155845880508,
- -5.8412379075889476e-06,
- -0.00015496007108595222,
- -0.0001935771433636546,
- -2.8967437174287625e-05,
- -0.01043801661580801,
- -7.974783511599526e-05,
- -0.0005525015876628458,
- -3.683499380713329e-05,
- -0.002455436158925295,
- -1.2874520507466514e-05,
- -0.0022639615926891565,
- -1.4543427823809907e-05,
- -0.00250252615660429,
- -8.344646857949556e-07,
- -0.0006089740199968219,
- -0.023519812151789665,
- -1.6231462955474854,
- -0.0013103241799399257,
- -0.0044088782742619514,
- -3.433168603805825e-05,
- -0.0076819476671516895,
- -0.13205960392951965,
- -0.01295448187738657,
- -0.0002797450579237193,
- -0.01799413561820984,
- -0.0008688965463079512,
- -0.0026737437583506107,
- -0.0004418112221173942,
- -0.001303895260207355,
- -6.16293036728166e-05,
- -0.0018553201807662845,
- -4.815939246327616e-05,
- -0.00024875884992070496,
- -0.000916537712328136,
- -0.005030237603932619,
- -0.00015853578224778175,
- -0.00936696957796812,
- -0.016335444524884224,
- -9.619726915843785e-05,
- -0.12435520440340042,
- -0.002912804950028658,
- -0.00010346830822527409,
- -0.0007908792467787862,
- -1.7165990357170813e-05,
- -6.260087490081787,
- -4.018156051635742,
- -0.05045890435576439,
- -0.00021360022947192192,
- -4.815939246327616e-05,
- -2.2203869819641113,
- -0.047356534749269485,
- -8.83301836438477e-05,
- -5.781483559985645e-05,
- -0.11337775737047195,
- -3.3378044463461265e-05,
- -0.0019444384379312396,
- -1.645074735279195e-05,
- -1.7198790311813354,
- -3.5991759300231934,
- -2.5881307125091553,
- -4.4389872550964355,
- -0.39235079288482666,
- -0.9257609248161316,
- -2.4064109325408936,
- -2.256807804107666,
- -0.012957894243299961,
- -6.8662193370983e-05,
- -0.005379723850637674,
- -0.1424376517534256,
- -0.008812819607555866,
- -0.00019667598826345056,
+ -0.13038524985313416,
+ -0.0034964873921126127,
+ -6.425174069590867e-05,
+ -0.0007306052139028907,
+ -2.13382354559144e-05,
+ -1.4955275058746338,
+ -0.9337477684020996,
+ -0.133364737033844,
+ -4.9470632802695036e-05,
+ -0.0004514628672040999,
+ -0.00015209948469419032,
+ -2.8013790142722428e-05,
+ -3.301778554916382,
+ -0.20383507013320923,
+ -0.0419706255197525,
+ -0.00013016807497479022,
+ -5.507317473529838e-05,
+ -0.10738518834114075,
+ -3.0040289857424796e-05,
+ -0.0028358979616314173,
+ -2.90866428258596e-05,
+ -2.9424922466278076,
+ -0.2351037710905075,
+ -2.5789151191711426,
+ -1.1322591304779053,
+ -0.5736383199691772,
+ -1.5535979270935059,
+ -0.22191236913204193,
+ -0.18333813548088074,
+ -0.664542555809021,
+ -0.0003240775258745998,
+ -0.005283796228468418,
+ -0.17099550366401672,
+ -0.014068713411688805,
+ -0.0002734287118073553,
+ -0.018341930583119392,
+ -0.0008268513483926654,
+ -0.001468295231461525,
+ -0.0007066613179631531,
+ -0.0006038511055521667,
+ -2.8609820219571702e-05,
+ -0.0019271865021437407,
+ -4.136476854910143e-05,
+ -0.00020990552729927003,
+ -0.0005302215577103198,
+ -0.0016419512685388327,
+ -0.00018070495571009815,
+ -0.019507931545376778,
+ -0.018029844388365746,
+ -0.00013004888023715466,
+ -1.4371798038482666,
+ -0.052820682525634766,
+ -0.00011884459672728553,
+ -0.00045980370487086475,
+ -1.7046782886609435e-05,
+ -0.6581263542175293,
+ -0.0013815154088661075,
+ -6.079655122448457e-06,
+ -0.0001554368354845792,
+ -0.00015615197480656207,
+ -3.611976353568025e-05,
+ -0.009467465803027153,
+ -8.916457591112703e-05,
+ -0.0005535738891921937,
+ -3.671578815556131e-05,
+ -0.002144420053809881,
+ -1.3112935448589269e-05,
+ -0.0022645562421530485,
+ -1.4424220353248529e-05,
+ -0.0024701817892491817,
+ -9.536738616588991e-07,
+ -0.0006376855308189988,
+ -0.02182525023818016,
+ -1.5834624767303467,
+ -0.0013265152228996158,
+ -0.004699495155364275,
+ -3.3854863431770355e-05,
+ -0.0077219316735863686,
+ -0.14429691433906555,
+ -0.01174628920853138,
+ -0.000324673397699371,
+ -0.017962872982025146,
+ -0.0008827127167023718,
+ -0.00236874190159142,
+ -0.0003970073303207755,
+ -0.0013256819220259786,
+ -6.0794889577664435e-05,
+ -0.0016473069554194808,
+ -4.8636207793606445e-05,
+ -0.0002506657037883997,
+ -0.0009217780898325145,
+ -0.004512722138315439,
+ -0.00019762947340495884,
+ -0.013031673617661,
+ -0.01633591391146183,
+ -0.00010299152199877426,
+ -0.1220099925994873,
+ -0.0024951535742729902,
+ -0.00010263393050990999,
+ -0.0008403107640333474,
+ -1.6331539882230572e-05,
+ -6.363530158996582,
+ -4.033095359802246,
+ -0.04472124204039574,
+ -0.00020919041708111763,
+ -4.792098479811102e-05,
+ -2.2731337547302246,
+ -0.049864355474710464,
+ -8.940297266235575e-05,
+ -5.221230458118953e-05,
+ -0.11411462724208832,
+ -3.266281055402942e-05,
+ -0.0019317077239975333,
+ -2.098061486321967e-05,
+ -1.7188977003097534,
+ -3.65950345993042,
+ -2.684589147567749,
+ -4.440427780151367,
+ -0.4001847505569458,
+ -0.9439029693603516,
+ -2.4071009159088135,
+ -2.2622978687286377,
+ -0.014948965050280094,
+ -5.972207145532593e-05,
+ -0.00555791798979044,
+ -0.14306095242500305,
+ -0.007800119463354349,
+ -0.00019858295854646713,
-0.012973662465810776,
- -0.0005903884884901345,
- -0.0019209994934499264,
- -0.0014405598631128669,
- -0.0006889115320518613,
- -1.645074735279195e-05,
- -0.0011966219171881676,
- -3.40932747349143e-05,
- -9.548207890475169e-05,
+ -0.0007591941393911839,
+ -0.0020857739727944136,
+ -0.0014411549782380462,
+ -0.000690460205078125,
+ -2.169585604860913e-05,
+ -0.0012124576605856419,
+ -3.421248038648628e-05,
+ -9.42901024245657e-05,
-0.0005439232336357236,
- -0.004501329269260168,
- -0.00011920218821614981,
- -0.03018992207944393,
- -0.013410485349595547,
- -0.00011467275908216834,
- -0.6566694378852844,
- -0.36726248264312744,
- -2.8490614567999728e-05,
- -0.00023707917716819793,
- -1.3351351299206726e-05,
- -1.051271915435791,
- -0.01689915731549263,
- -3.0722033977508545,
- -0.2818227708339691,
- -3.957169771194458,
- -0.004226442892104387,
- -0.00017248096992261708,
- -3.9457496313843876e-05,
- -5.733857154846191,
- -0.26561957597732544,
- -0.00047779586748220026,
- -2.5748875486897305e-05,
- -0.07624048739671707,
- -6.0437283536884934e-05,
- -0.001644212519749999,
- -1.549708758830093e-05,
- -2.1518163681030273,
- -0.19709540903568268,
- -3.698873996734619,
- -10.724569320678711,
- -2.996880292892456,
- -3.1366219520568848,
- -0.02801341563463211,
- -0.17601795494556427,
- -0.0965375229716301,
- -0.00014578233822248876,
- -0.0020983838476240635,
- -0.054011568427085876,
- -0.003581777447834611,
- -0.00014304091746453196,
- -0.011484465561807156,
- -0.000708090839907527,
- -0.0012874656822532415,
- -0.0009416675311513245,
- -0.0005903884884901345,
- -2.13382354559144e-05,
- -0.0007848043460398912,
- -2.3841574147809297e-05,
- -7.4741430580616e-05,
- -0.0002946419408544898,
- -0.0024204738438129425,
- -0.00011503035057103261,
- -0.006832095794379711,
- -0.010126759298145771,
- -5.876845170860179e-05,
+ -0.004523758310824633,
+ -0.000120751719805412,
+ -0.043312862515449524,
+ -0.01450247410684824,
+ -0.00010632903286023065,
+ -0.718799352645874,
+ -0.3663807809352875,
+ -2.95634672511369e-05,
+ -0.0001879753835964948,
+ -1.6331539882230572e-05,
+ -1.0465071201324463,
+ -0.018635429441928864,
+ -3.112912178039551,
+ -0.2903432548046112,
+ -3.879873752593994,
+ -0.004381461534649134,
+ -0.00013815402053296566,
+ -4.2437604861333966e-05,
+ -5.737420082092285,
+ -0.2639721632003784,
+ -0.0006080209277570248,
+ -2.610649426060263e-05,
+ -0.08038974553346634,
+ -6.151010165922344e-05,
+ -0.0016420703614130616,
+ -1.490105023549404e-05,
+ -2.140990734100342,
+ -0.2073119729757309,
+ -3.695624828338623,
+ -10.747465133666992,
+ -3.023293972015381,
+ -3.056196451187134,
+ -0.03145737573504448,
+ -0.1770988404750824,
+ -0.0917704701423645,
+ -0.00013183678674977273,
+ -0.002531302161514759,
+ -0.04845479875802994,
+ -0.003440296743065119,
+ -0.0001445904199499637,
+ -0.011499903164803982,
+ -0.0007746794726699591,
+ -0.0012647256953641772,
+ -0.0009414292871952057,
+ -0.0005958688561804593,
+ -2.0503786799963564e-05,
+ -0.0007793250260874629,
+ -2.3483953555114567e-05,
+ -7.581423415103927e-05,
+ -0.00029559535323642194,
+ -0.0024205928202718496,
+ -0.00010692501382436603,
+ -0.007723824121057987,
+ -0.009906531311571598,
+ -5.864924969500862e-05,
-0.09275738149881363,
- -0.003692833473905921,
- -4.0411134250462055e-05,
- -0.0005497612874023616,
- -1.537788011773955e-05,
- -1.182621717453003,
- -0.0008486483711749315,
+ -0.003653876017779112,
+ -4.792098479811102e-05,
+ -0.0005359405186027288,
+ -1.549708758830093e-05,
+ -1.2015808820724487,
+ -0.0008397151832468808,
-4.0531076592742465e-06,
- -0.00010585224663373083,
- -0.00011646069469861686,
- -2.407998726994265e-05,
- -0.00471824174746871,
- -5.352353764465079e-05,
- -0.0003631647559814155,
- -3.135155202471651e-05,
- -0.0011143434094265103,
- -1.1205610462639015e-05,
- -0.002159646013751626,
- -1.4185804502631072e-05,
- -0.0011845960980281234,
+ -0.00011955977242905647,
+ -0.00011383838864276186,
+ -2.4318398573086597e-05,
+ -0.005200904794037342,
+ -5.709961988031864e-05,
+ -0.0003819928097072989,
+ -2.5033637939486653e-05,
+ -0.001191859133541584,
+ -1.0847986231965479e-05,
+ -0.0019281383138149977,
+ -1.4543427823809907e-05,
+ -0.001212338567711413,
-7.152555099310121e-07,
- -0.0002699726028367877,
+ -0.00030763185350224376,
-0.008802657015621662,
- -1.1517901420593262,
- -0.0017283515771850944,
- -0.002493488835170865,
- -1.5258672647178173e-05,
- -0.0018479428254067898,
- -0.040569812059402466,
- -0.0041178204119205475,
- -0.00017176583060063422,
+ -1.1513752937316895,
+ -0.001739894854836166,
+ -0.0026299909222871065,
+ -1.2278481335670222e-05,
+ -0.0019224273273721337,
+ -0.03679193556308746,
+ -0.004175397567451,
+ -0.00016950124700088054,
-0.015839355066418648,
- -0.0005023409612476826,
- -0.0007201223634183407,
- -0.0005905076395720243,
- -0.0007784912013448775,
- -2.3483953555114567e-05,
- -0.0008902162662707269,
- -2.6702524337451905e-05,
- -9.512448741588742e-05,
- -0.0004555141495075077,
- -0.014392376877367496,
- -9.619726915843785e-05,
- -0.0002324311062693596,
- -0.01029337290674448,
- -0.00015984688070602715,
- -1.1049474477767944,
- -0.04663100838661194,
- -8.21318244561553e-05,
- -0.0003543464408721775,
- -1.3947389561508317e-05,
- -7.615281581878662,
- -4.125001907348633,
- -0.19173777103424072,
- -0.0005029367166571319,
+ -0.0005919373361393809,
+ -0.0008360228384844959,
+ -0.0006061147432774305,
+ -0.0008789013954810798,
+ -2.396077979938127e-05,
+ -0.0009041512385010719,
+ -2.8967437174287625e-05,
+ -0.00010132275929208845,
+ -0.0004866131057497114,
+ -0.014365584589540958,
+ -9.655486064730212e-05,
+ -0.00021050144277978688,
+ -0.011357063427567482,
+ -0.00016223068814724684,
+ -1.124318242073059,
+ -0.046641137450933456,
+ -6.48477507638745e-05,
+ -0.00036769305006600916,
+ -1.3232143828645349e-05,
+ -7.617920875549316,
+ -4.132878303527832,
+ -0.18580418825149536,
+ -0.0005179494037292898,
-4.100715523236431e-05,
- -2.0808839797973633,
- -0.026673687621951103,
- -7.70062324590981e-05,
- -2.9682672902708873e-05,
- -0.12381786853075027,
- -2.098061486321967e-05,
- -0.0029344377107918262,
- -1.3589766240329482e-05,
- -6.027270793914795,
- -0.344284325838089,
- -0.47963422536849976,
- -1.262589454650879,
- -1.8010940551757812,
- -2.51932430267334,
- -1.5027334690093994,
- -0.06264369934797287,
- -1.8616759777069092,
- -2.732039213180542,
- -6.854299135738984e-05,
- -0.001887565478682518,
- -0.02442971244454384,
- -0.0030983323231339455,
- -0.00013374387344811112,
- -0.010926888324320316,
- -0.0006349454633891582,
- -0.0010619483655318618,
- -0.0007469248375855386,
- -0.00040987672400660813,
- -1.537788011773955e-05,
+ -2.0765302181243896,
+ -0.026648614555597305,
+ -6.532455881824717e-05,
+ -2.455681169521995e-05,
+ -0.13987594842910767,
+ -2.062299427052494e-05,
+ -0.0029500082600861788,
+ -1.3112935448589269e-05,
+ -6.043766975402832,
+ -0.34587275981903076,
+ -0.47646504640579224,
+ -1.2680540084838867,
+ -1.7469899654388428,
+ -2.445939064025879,
+ -1.511207103729248,
+ -0.06483010947704315,
+ -1.8050661087036133,
+ -2.7358272075653076,
+ -6.949660019017756e-05,
+ -0.001785947591997683,
+ -0.02307841368019581,
+ -0.003315905574709177,
+ -0.00013207517622504383,
+ -0.009649066254496574,
+ -0.0005932478234171867,
+ -0.001046229270286858,
+ -0.0009538153535686433,
+ -0.00037079135654494166,
+ -1.4424220353248529e-05,
-0.0008891443139873445,
-2.4676019165781327e-05,
- -7.080780778778717e-05,
- -0.00043299360550008714,
- -0.2814013361930847,
- -6.8662193370983e-05,
- -0.0011491130571812391,
- -0.007679700385779142,
- -9.440929716220126e-05,
- -0.026545187458395958,
- -0.002912091789767146,
- -7.045020902296528e-05,
- -0.001142087858170271,
- -1.4662635294371285e-05,
- -1.6412137746810913,
- -9.728646278381348,
- -0.026286397129297256,
- -0.0002475670480635017,
- -7.60526381782256e-05,
- -2.191868782043457,
- -0.01760944165289402,
- -0.0004247716860845685,
- -4.684815212385729e-05,
- -0.03103969246149063,
- -9.297892393078655e-05,
- -0.011422710493206978,
- -3.6954195820726454e-05,
- -4.347017288208008,
- -0.000610999355558306,
- -2.17897367477417,
- -2.866166353225708,
- -0.23518076539039612,
- -0.00036125810584053397,
- -0.01150013878941536,
- -1.8427702188491821,
- -0.22964701056480408,
- -0.011748881079256535,
- -0.00036352223833091557,
- -2.021958827972412,
- -0.008272194303572178,
- -1.7123057842254639,
- -9.325576782226562,
- -1.3440426588058472,
- -3.209916830062866,
- -0.053304191678762436,
- -5.205663681030273,
- -0.03287550434470177,
- -1.384042501449585,
- -7.2653326988220215,
- -3.6932270526885986,
- -6.713709354400635,
- -0.08502203971147537,
- -3.0402512550354004,
- -0.043377358466386795,
- -0.00908633042126894,
- -0.013433421961963177,
- -1.5646146535873413,
- -0.007355276495218277,
- -5.929056167602539,
- -11.379992485046387,
- -3.6368532180786133,
- -0.45781779289245605,
- -0.003176526166498661,
- -0.039530687034130096,
- -0.0007678897818550467,
- -0.0003935516288038343,
- -0.0005339150666259229,
- -3.964613437652588,
- -0.9957391023635864,
- -0.2830953896045685,
- -0.2610830068588257,
- -0.07025375217199326,
- -0.010492399334907532,
- -3.3028924465179443,
- -4.77099084854126
+ -7.116541382856667e-05,
+ -0.00043275527423247695,
+ -0.17996622622013092,
+ -7.116541382856667e-05,
+ -0.0011485177092254162,
+ -0.007210304494947195,
+ -8.940297266235575e-05,
+ -0.027487365528941154,
+ -0.002912804950028658,
+ -7.033100700937212e-05,
+ -0.001149232150055468,
+ -1.8954096958623268e-05,
+ -1.6481224298477173,
+ -9.633079528808594,
+ -0.02628662995994091,
+ -0.0002915434306487441,
+ -5.936446541454643e-05,
+ -2.157845973968506,
+ -0.017832664772868156,
+ -0.0004151197790633887,
+ -4.672895011026412e-05,
+ -0.03097185306251049,
+ -9.202533692587167e-05,
+ -0.01144097838550806,
+ -3.71926071238704e-05,
+ -4.355002403259277,
+ -0.0006113567505963147,
+ -2.2188010215759277,
+ -2.907867908477783,
+ -0.23523163795471191,
+ -0.00036507140612229705,
+ -0.011011547408998013,
+ -1.8425185680389404,
+ -0.22946366667747498,
+ -0.011686672456562519,
+ -0.0003597089380491525,
+ -2.111253023147583,
+ -0.009250637143850327,
+ -1.7095400094985962,
+ -9.170680046081543,
+ -1.3476110696792603,
+ -3.1840837001800537,
+ -0.054254475980997086,
+ -5.210089206695557,
+ -0.03295567259192467,
+ -1.4402692317962646,
+ -7.255331993103027,
+ -3.6931405067443848,
+ -6.715616226196289,
+ -0.08208508044481277,
+ -3.03913950920105,
+ -0.043953072279691696,
+ -0.008821917697787285,
+ -0.014872273430228233,
+ -1.6501425504684448,
+ -0.007400597911328077,
+ -5.930353164672852,
+ -11.405633926391602,
+ -3.518193483352661,
+ -0.4507780373096466,
+ -0.0035801143385469913,
+ -0.035687681287527084,
+ -0.000773488252889365,
+ -0.0003968881501350552,
+ -0.0005335576133802533,
+ -4.078104019165039,
+ -0.9954992532730103,
+ -0.31775549054145813,
+ -0.26048052310943604,
+ -0.07195602357387543,
+ -0.010182575322687626,
+ -3.3031930923461914,
+ -4.769801139831543
],
"generated_logprobs": [
- -0.22153465449810028,
- -0.0006783091812394559,
- -0.010667562484741211,
- -0.0019300420535728335,
- -0.00015901254664640874,
- -0.0025420039892196655,
- -0.0038043521344661713,
- -0.0017481058603152633,
- -0.47604793310165405,
- -0.0035732248798012733,
- -1.3063528537750244,
- -1.869868278503418,
- -0.03979752957820892,
- -1.2474843263626099,
- -0.1659490168094635,
- -0.006920535117387772,
- -1.4199819564819336,
- -2.666736364364624,
- -1.218197226524353,
- -0.017583556473255157,
- -0.5422223806381226,
- -0.0014107999159023166,
- -0.0003762729174923152,
- -0.0010189585154876113,
- -0.03820022940635681,
- -0.0013802058529108763,
- -0.1537325382232666,
- -0.0007863528444431722,
- -0.003772999858483672,
- -0.019278066232800484
+ -0.21995607018470764,
+ -0.0007650309125892818,
+ -0.010548313148319721,
+ -0.00197001826018095,
+ -0.00015722469834145159,
+ -0.0025895654689520597,
+ -0.003808746114373207,
+ -0.0017583399312570691,
+ -0.47401726245880127,
+ -0.003675017738714814,
+ -1.3417091369628906,
+ -1.8469452857971191,
+ -0.040168169885873795,
+ -1.2434231042861938,
+ -0.1829521209001541,
+ -0.007097271271049976,
+ -1.3962904214859009,
+ -2.523759603500366,
+ -1.2317531108856201,
+ -0.017718834802508354,
+ -0.5408012866973877,
+ -0.0015022194711491466,
+ -0.00041237910045310855,
+ -0.0009870424401015043,
+ -0.0377543605864048,
+ -0.0013729440979659557,
+ -0.14942514896392822,
+ -0.0007965967524796724,
+ -0.0037569671403616667,
+ -0.019313964992761612
],
"logprobs": [
-2.0310330390930176,
@@ -2828,2759 +2829,2760 @@
-1.2136539220809937,
-0.4511846899986267,
-1.275371789932251,
- -6.52569055557251,
- -0.3268530070781708,
- -2.488239288330078,
- -1.1252245903015137,
- -0.004931548144668341,
- -1.1413307189941406,
- -2.4036614894866943,
- -0.593055784702301,
- -5.775687217712402,
- -0.7173333764076233,
- -6.7589006423950195,
- -4.472473621368408,
- -0.28561243414878845,
- -0.9266374111175537,
- -1.2420787811279297,
- -4.94831657409668,
- -0.4015401303768158,
- -2.405423879623413,
- -6.706996440887451,
- -2.3797435760498047,
- -6.879988193511963,
- -0.599727988243103,
- -4.6161346435546875,
- -0.016334740445017815,
- -1.4226453304290771,
- -4.064138412475586,
- -8.992555618286133,
- -0.7892558574676514,
- -2.565383195877075,
- -1.6011606454849243,
- -1.1192784309387207,
- -1.085118293762207,
- -1.452021598815918,
- -0.1256672590970993,
- -4.310093879699707,
- -0.039925139397382736,
- -0.09540079534053802,
- -4.4552788734436035,
- -2.6978704929351807,
- -0.3264457583427429,
- -0.9057141542434692,
- -0.2424505054950714,
- -0.2473771721124649,
- -0.04457908123731613,
- -2.5994861125946045,
- -0.5882505178451538,
- -2.4292445182800293,
- -0.1860235333442688,
- -2.6841845512390137,
- -5.8617939949035645,
- -1.7926914691925049,
- -0.6663980484008789,
- -0.029983440414071083,
- -1.0682772397994995,
- -0.0018566290382295847,
- -1.9571454524993896,
- -0.08927226811647415,
- -4.61471700668335,
- -0.002604546956717968,
- -0.2620302140712738,
- -0.006101197097450495,
- -7.435886859893799,
- -0.0376485139131546,
- -10.174129486083984,
- -0.9147175550460815,
- -4.526404857635498,
- -3.670576572418213,
- -4.566626071929932,
- -1.0199782848358154,
- -0.0006491222884505987,
- -0.14426420629024506,
- -0.03322957828640938,
- -0.0019640696700662374,
- -0.00022468426323030144,
- -0.0013444918440654874,
- -0.0011957883834838867,
- -0.007926556281745434,
- -0.011617152951657772,
- -0.0018109364900738,
- -0.00017581824795342982,
- -0.0018969652010127902,
- -6.282132380874828e-05,
- -0.0010078833438456059,
- -0.25652098655700684,
- -0.35659894347190857,
- -9.333651541965082e-05,
- -0.7947311401367188,
- -1.3594639301300049,
- -7.962863310240209e-05,
- -1.861167550086975,
- -0.5386030673980713,
- -0.00022075122979003936,
- -0.001347229932434857,
- -3.290122185717337e-05,
- -3.7342543601989746,
- -0.5175371170043945,
- -4.488879680633545,
- -0.007863753475248814,
- -0.08534510433673859,
- -0.0009170140838250518,
- -2.13382354559144e-05,
- -4.507952690124512,
- -0.5332688689231873,
- -0.004296358674764633,
- -2.062299427052494e-05,
- -5.2475104331970215,
- -0.020387964323163033,
- -0.1661914438009262,
- -0.0003081085451412946,
- -15.800027847290039,
- -8.108964920043945,
- -0.7285020351409912,
- -7.803549289703369,
- -5.010417938232422,
- -0.263860821723938,
- -4.3748852476710454e-05,
- -0.013306032866239548,
- -0.029512016102671623,
- -0.0036468682810664177,
- -0.00023231192608363926,
- -0.0002379134384682402,
- -0.0004920940846204758,
- -0.000873065204359591,
- -0.0029308719094842672,
- -0.0006667536217719316,
- -0.00013672371278516948,
- -0.0011686407960951328,
+ -6.502887725830078,
+ -0.3318161368370056,
+ -2.374776601791382,
+ -1.1216609477996826,
+ -0.004972116556018591,
+ -1.132584571838379,
+ -2.3807473182678223,
+ -0.6019684672355652,
+ -5.6695661544799805,
+ -0.7204529047012329,
+ -6.769031047821045,
+ -4.453886985778809,
+ -0.28745004534721375,
+ -0.9512360692024231,
+ -1.1893970966339111,
+ -4.947760581970215,
+ -0.36444246768951416,
+ -2.4082541465759277,
+ -6.61063814163208,
+ -2.503272294998169,
+ -6.87827205657959,
+ -0.6025495529174805,
+ -4.622146129608154,
+ -0.01804928109049797,
+ -1.4212173223495483,
+ -4.08338737487793,
+ -9.046302795410156,
+ -0.7913540601730347,
+ -2.5653929710388184,
+ -1.5746062994003296,
+ -1.1179513931274414,
+ -1.085464358329773,
+ -1.447007656097412,
+ -0.12754084169864655,
+ -4.321589469909668,
+ -0.03916267305612564,
+ -0.09776852279901505,
+ -4.340118885040283,
+ -2.6412477493286133,
+ -0.35567763447761536,
+ -0.9400036334991455,
+ -0.2579241991043091,
+ -0.2446412295103073,
+ -0.044584665447473526,
+ -2.601752996444702,
+ -0.5793261528015137,
+ -2.4348762035369873,
+ -0.1983177214860916,
+ -2.804823160171509,
+ -5.8624348640441895,
+ -1.890607237815857,
+ -0.6673221588134766,
+ -0.02870091050863266,
+ -1.0835812091827393,
+ -0.0018605556106194854,
+ -1.954398274421692,
+ -0.08902604877948761,
+ -4.561020851135254,
+ -0.002339128637686372,
+ -0.28425076603889465,
+ -0.006011382210999727,
+ -7.539747714996338,
+ -0.0332016721367836,
+ -10.147441864013672,
+ -0.9909310340881348,
+ -4.516211986541748,
+ -3.788264751434326,
+ -4.543238639831543,
+ -0.967991054058075,
+ -0.0006565084331668913,
+ -0.14520324766635895,
+ -0.032342080026865005,
+ -0.0019212374463677406,
+ -0.00022480344341602176,
+ -0.0013449679827317595,
+ -0.0011960265692323446,
+ -0.007967357523739338,
+ -0.011690796352922916,
+ -0.0019255208317190409,
+ -0.00019429226813372225,
+ -0.0017928492743521929,
+ -6.329813186312094e-05,
+ -0.0009005781612358987,
+ -0.25652363896369934,
+ -0.3569473922252655,
+ -8.11782301752828e-05,
+ -0.7947285771369934,
+ -1.3081212043762207,
+ -8.21318244561553e-05,
+ -1.8611676692962646,
+ -0.5729392766952515,
+ -0.0002153879904653877,
+ -0.0013609204906970263,
+ -3.2543604902457446e-05,
+ -3.7282280921936035,
+ -0.5535277128219604,
+ -4.489457607269287,
+ -0.008902146480977535,
+ -0.08532682061195374,
+ -0.0009254701435565948,
+ -2.098061486321967e-05,
+ -4.501499176025391,
+ -0.49491915106773376,
+ -0.004292679019272327,
+ -2.002696055569686e-05,
+ -5.250372409820557,
+ -0.02144506201148033,
+ -0.163783460855484,
+ -0.0003084660565946251,
+ -15.754971504211426,
+ -8.121952056884766,
+ -0.7163605093955994,
+ -7.821594715118408,
+ -5.016800880432129,
+ -0.26420897245407104,
+ -4.8040190449682996e-05,
+ -0.013264977373182774,
+ -0.0258948914706707,
+ -0.003469758667051792,
+ -0.0002307625545654446,
+ -0.00023052419419400394,
+ -0.0004808938247151673,
+ -0.000872946111485362,
+ -0.0031739119440317154,
+ -0.0007124984404072165,
+ -0.00013267113536130637,
+ -0.0011754278093576431,
+ -5.924526340095326e-05,
+ -0.000880449719261378,
+ -0.02788648009300232,
+ -0.06313233077526093,
+ -0.0001311216183239594,
+ -0.21269679069519043,
+ -0.6191936135292053,
+ -5.769562994828448e-05,
+ -1.637540340423584,
+ -1.6328482627868652,
+ -0.0003262225945945829,
+ -0.0016448076348751783,
+ -2.8967437174287625e-05,
+ -2.92718505859375,
+ -0.2820948660373688,
+ -6.417831897735596,
+ -3.146303176879883,
+ -0.7236800789833069,
+ -0.06164867803454399,
+ -0.0015048381173983216,
+ -2.825220326485578e-05,
+ -3.9370851516723633,
+ -0.09090036898851395,
+ -0.0035149003379046917,
+ -4.279521817807108e-05,
+ -4.558128833770752,
+ -0.010771229863166809,
+ -0.09514036774635315,
+ -0.00041416651220060885,
+ -4.436840534210205,
+ -2.224473714828491,
+ -3.2508153915405273,
+ -8.453686714172363,
+ -0.030126187950372696,
+ -2.3973264694213867,
+ -0.07391595095396042,
+ -0.5675545930862427,
+ -0.05637309327721596,
+ -0.21564748883247375,
+ -0.00012206286191940308,
+ -0.05818720534443855,
+ -0.12251465767621994,
+ -0.013046146370470524,
+ -0.00023183519078884274,
+ -0.0001517419150331989,
+ -0.0003564914222806692,
+ -0.0005465444410219789,
+ -0.0022260190453380346,
+ -0.0007184546557255089,
+ -9.011816291604191e-05,
+ -0.00105015910230577,
-4.625213477993384e-05,
- -0.0007901645149104297,
- -0.027857612818479538,
- -0.06313244253396988,
- -0.00013064485392533243,
- -0.2378876954317093,
- -0.6059458255767822,
- -5.757642793469131e-05,
- -1.5949885845184326,
- -1.6001688241958618,
- -0.00032574593205936253,
- -0.0016402851324528456,
+ -0.0005137792322784662,
+ -0.021607117727398872,
+ -0.023618897423148155,
+ -0.0001915509783430025,
+ -0.043488532304763794,
+ -2.8467631340026855,
+ -9.07141511561349e-05,
+ -0.9812690019607544,
+ -1.4124990701675415,
+ -0.00018153927521780133,
+ -0.0012271024752408266,
+ -1.6927575416048057e-05,
+ -1.0269899368286133,
+ -0.019673457369208336,
+ -0.045223724097013474,
+ -1.6328576803207397,
+ -0.00045599075383506715,
+ -0.000436091679148376,
-2.276871418871451e-05,
- -3.0335943698883057,
- -0.286937952041626,
- -6.517683982849121,
- -3.1465959548950195,
- -0.7292280793190002,
- -0.06161583960056305,
- -0.0014851979212835431,
- -2.777537883957848e-05,
- -3.946831226348877,
- -0.09084996581077576,
- -0.003532005939632654,
- -4.029192859889008e-05,
- -4.555190086364746,
- -0.011255813762545586,
- -0.10179147869348526,
- -0.0004140473320148885,
- -4.4321393966674805,
- -2.2296247482299805,
- -3.2771155834198,
- -8.323366165161133,
- -0.02779245562851429,
- -2.403028964996338,
- -0.07431145757436752,
- -0.5372196435928345,
- -0.05987980589270592,
- -0.20438668131828308,
- -0.00013136000779923052,
- -0.0572563000023365,
- -0.11035308241844177,
- -0.012903997674584389,
- -0.0002406545972917229,
- -0.0001517419150331989,
- -0.00036066226311959326,
- -0.0005477358354255557,
- -0.00229322025552392,
- -0.000697846058756113,
- -0.0001161031104857102,
- -0.001127441762946546,
- -3.814624506048858e-05,
- -0.0005136600811965764,
- -0.022026309743523598,
- -0.02361132949590683,
- -0.0002090712368953973,
- -0.04913746938109398,
- -2.7477238178253174,
- -9.202533692587167e-05,
- -0.9271803498268127,
- -1.3856279850006104,
- -0.0001754606782924384,
- -0.0012224590172991157,
- -1.7165990357170813e-05,
- -1.0239524841308594,
- -0.020712625235319138,
- -0.0451514832675457,
- -1.5345499515533447,
- -0.0004010588163509965,
- -0.0004401430196594447,
- -2.13382354559144e-05,
- -2.5878491401672363,
- -0.020529404282569885,
- -0.00043501926120370626,
- -2.682172998902388e-05,
- -0.3827762007713318,
- -0.00019298121333122253,
- -0.007158228196203709,
- -8.618460560683161e-05,
- -6.015654563903809,
- -4.037173271179199,
- -3.4229695796966553,
- -1.0183475017547607,
- -1.4963387250900269,
- -0.33330175280570984,
- -1.480197787284851,
- -2.0857536792755127,
- -2.225975513458252,
- -5.293066024780273,
- -0.43916723132133484,
- -0.00010048838157672435,
- -0.015328695066273212,
- -0.13567933440208435,
- -0.012453177943825722,
- -0.00017855956684798002,
- -0.00012778419477399439,
- -0.0002885640424210578,
- -0.0004291805380489677,
- -0.0008485292200930417,
- -0.0006668727728538215,
- -8.177422569133341e-05,
- -0.001060757553204894,
- -6.151010165922344e-05,
- -0.0005185451591387391,
- -0.028113562613725662,
- -0.03407377377152443,
- -0.0003861635341309011,
- -1.1215460300445557,
- -0.5561063885688782,
- -0.0001726001501083374,
- -2.5190887451171875,
- -0.6141397953033447,
+ -2.5508246421813965,
+ -0.018045533448457718,
+ -0.00042500998824834824,
+ -2.825220326485578e-05,
+ -0.3710917532444,
+ -0.00019667598826345056,
+ -0.006620378233492374,
+ -8.606540359323844e-05,
+ -6.00956916809082,
+ -4.060979843139648,
+ -3.422973155975342,
+ -1.019230604171753,
+ -1.5553637742996216,
+ -0.2993496060371399,
+ -1.4802078008651733,
+ -2.070681095123291,
+ -2.330273151397705,
+ -5.414191246032715,
+ -0.46552836894989014,
+ -8.868777513271198e-05,
+ -0.016082679852843285,
+ -0.15049032866954803,
+ -0.012319305911660194,
+ -0.00018225439998786896,
+ -0.00011705666838679463,
+ -0.00030763185350224376,
+ -0.0004362108593340963,
+ -0.0008486483711749315,
+ -0.0006685405969619751,
+ -0.00010477947944309562,
+ -0.0010546842822805047,
+ -5.376194530981593e-05,
+ -0.0005538121913559735,
+ -0.029470346868038177,
+ -0.030175579711794853,
+ -0.0003849719068966806,
+ -1.207515001296997,
+ -0.5080216526985168,
+ -0.00017128908075392246,
+ -2.5190865993499756,
+ -0.6522354483604431,
-0.0001227780303452164,
- -0.0012188870459794998,
- -1.6212332411669195e-05,
- -6.833529472351074,
- -6.0156097412109375,
- -0.03274226188659668,
- -0.014286145567893982,
- -0.0009454786195419729,
- -3.814624506048858e-05,
- -4.910149097442627,
- -0.009493326768279076,
- -0.001437702914699912,
+ -0.0012397230602800846,
+ -1.597391747054644e-05,
+ -6.705944538116455,
+ -6.01674222946167,
+ -0.032217077910900116,
+ -0.014274628832936287,
+ -0.0009184433147311211,
+ -3.8265450712060556e-05,
+ -5.022707939147949,
+ -0.0094671119004488,
+ -0.0014456784119829535,
+ -5.864924969500862e-05,
+ -0.37389981746673584,
+ -0.004271550104022026,
+ -0.06971022486686707,
+ -0.00017653337272349745,
+ -6.816598892211914,
+ -0.0015250729629769921,
+ -2.643510341644287,
+ -0.9823706150054932,
+ -1.4715224504470825,
+ -0.7379395365715027,
+ -1.4038898944854736,
+ -9.202533692587167e-05,
+ -0.04053020477294922,
+ -0.3290509581565857,
+ -0.02419864945113659,
+ -0.00028713393840007484,
+ -0.00013565097469836473,
+ -0.00026723151677288115,
+ -0.000376034586224705,
+ -0.0006663962849415839,
+ -0.0008694920688867569,
+ -9.881961887003854e-05,
+ -0.0012442474253475666,
+ -5.781483559985645e-05,
+ -0.0005551227368414402,
+ -0.013484349474310875,
+ -0.033842623233795166,
+ -0.00047017011092975736,
+ -0.28224796056747437,
+ -1.4657995700836182,
+ -0.00012790338951162994,
+ -1.4014155864715576,
+ -0.9573185443878174,
+ -0.0002760506176855415,
+ -0.0016822961624711752,
+ -2.3483953555114567e-05,
+ -5.273507595062256,
+ -2.626476764678955,
+ -0.1083991751074791,
+ -0.0007696765242144465,
+ -3.158996332786046e-05,
+ -5.912450790405273,
+ -0.05985802412033081,
+ -0.002310347044840455,
+ -2.7656173188006505e-05,
+ -0.5037983655929565,
+ -0.0008891443139873445,
+ -0.013665338978171349,
+ -8.546940807718784e-05,
+ -6.966069221496582,
+ -0.366066038608551,
+ -1.422469139099121,
+ -0.8349007964134216,
+ -0.468829870223999,
+ -2.5325279235839844,
+ -0.3129241466522217,
+ -7.700963973999023,
+ -0.35001978278160095,
+ -2.013795852661133,
+ -0.09784646332263947,
+ -7.86750388215296e-05,
+ -0.01667795516550541,
+ -0.18287953734397888,
+ -0.015566980466246605,
+ -0.0001851148990681395,
+ -7.593343616463244e-05,
+ -0.00014125302550382912,
+ -0.0002083561266772449,
+ -0.0004748170613311231,
+ -0.000512111175339669,
+ -4.815939246327616e-05,
+ -0.0011738798348233104,
+ -3.0517112463712692e-05,
+ -0.0003026265767402947,
+ -0.003111880039796233,
+ -0.009830750524997711,
+ -0.00022289653134066612,
+ -0.12992516160011292,
+ -1.1169023513793945,
+ -9.691245941212401e-05,
+ -1.1468124389648438,
+ -0.49202996492385864,
+ -7.009260298218578e-05,
+ -0.0014340127818286419,
+ -1.5735502529423684e-05,
+ -3.9838247299194336,
+ -0.02546604536473751,
+ -0.0001641377166379243,
+ -0.003960979636758566,
+ -0.0006631797295995057,
+ -2.9801878554280847e-05,
+ -0.15490414202213287,
+ -0.0008278042078018188,
+ -0.0008001701789908111,
+ -1.811964830267243e-05,
+ -0.21022747457027435,
+ -0.00018714107864070684,
+ -0.0019920284394174814,
+ -2.2053474822314456e-05,
+ -0.014113731682300568,
+ -1.4305104514278355e-06,
+ -0.011013787239789963,
+ -0.6143843531608582,
+ -2.3573765754699707,
+ -0.0103318365290761,
+ -0.00997463520616293,
+ -7.772143726469949e-05,
+ -0.010983250103890896,
+ -0.29723531007766724,
+ -0.022360580042004585,
+ -0.0001991788885788992,
+ -5.006664650863968e-05,
+ -0.00010573305189609528,
+ -0.00015805903240107,
+ -0.00035661060246638954,
+ -0.0004104725376237184,
+ -7.164221460698172e-05,
+ -0.0008939084364101291,
+ -3.135155202471651e-05,
+ -0.00031394799589179456,
+ -0.002726529957726598,
+ -0.008696303702890873,
+ -0.0003630455757956952,
+ -0.258771151304245,
+ -2.1416406631469727,
+ -0.0001802282058633864,
+ -1.8033326864242554,
+ -1.623347520828247,
+ -0.0002315968304174021,
+ -0.0006778326351195574,
+ -1.3470558769768104e-05,
+ -1.263992428779602,
+ -0.016645127907395363,
+ -2.042416572570801,
+ -1.4463900327682495,
+ -2.7770018577575684,
+ -0.5399353504180908,
+ -0.00030071981018409133,
+ -2.1576648578047752e-05,
+ -2.99349308013916,
+ -0.0013340153964236379,
+ -0.0007597897201776505,
+ -1.4662635294371285e-05,
+ -0.26145222783088684,
+ -0.00023266946664080024,
+ -0.010189891792833805,
+ -3.790783375734463e-05,
+ -2.1039741039276123,
+ -6.185567378997803,
+ -4.484244346618652,
+ -2.000312089920044,
+ -0.2879604697227478,
+ -0.05998960882425308,
+ -0.7370511889457703,
+ -0.01011425070464611,
+ -0.00029690624796785414,
+ -0.14329783618450165,
+ -0.0006635371246375144,
+ -0.00035398892941884696,
+ -0.00229904823936522,
+ -0.36686453223228455,
+ -0.09950104355812073,
+ -2.546339511871338,
+ -3.2053751945495605,
+ -0.11890234053134918,
+ -4.541770613286644e-05,
+ -0.015083776786923409,
+ -0.1699461191892624,
+ -0.008777014911174774,
+ -0.00011419598013162613,
+ -3.421248038648628e-05,
+ -6.115249561844394e-05,
+ -0.0001998939987970516,
+ -0.0003668589051812887,
+ -0.0003194298769813031,
+ -5.23315102327615e-05,
+ -0.0009507188224233687,
+ -2.3364747903542593e-05,
+ -0.00019131260341964662,
+ -0.0009372609201818705,
+ -0.004119957331568003,
+ -0.0002714027068577707,
+ -0.31108736991882324,
+ -0.4985818862915039,
+ -0.0001879753835964948,
+ -1.0743582248687744,
+ -0.28104329109191895,
+ -4.303362584323622e-05,
+ -0.0011857867939397693,
+ -1.9073304429184645e-05,
+ -1.1399143934249878,
+ -0.06796625256538391,
+ -0.0027990469243377447,
+ -8.654219709569588e-05,
+ -0.00018761781393550336,
+ -0.0004489606071729213,
+ -1.5735502529423684e-05,
+ -8.45077896118164,
+ -0.00031680811662226915,
+ -7.279982089996338,
+ -0.09349125623703003,
+ -0.004146312829107046,
+ -0.0008599635912105441,
+ -5.149708886165172e-05,
+ -0.7756458520889282,
+ -0.00035589560866355896,
+ -0.01518346555531025,
-5.876845170860179e-05,
- -0.3798050582408905,
- -0.003948037512600422,
- -0.07855644077062607,
- -0.00022420754248742014,
- -6.84205436706543,
- -0.0015236446633934975,
- -2.645585298538208,
- -0.9816564917564392,
- -1.3786735534667969,
- -0.7280330061912537,
- -1.4040117263793945,
- -9.035655966727063e-05,
- -0.033023953437805176,
- -0.3305729031562805,
- -0.027912795543670654,
- -0.0002892790944315493,
- -0.00012182447244413197,
- -0.00026901919045485556,
- -0.0004681444843299687,
- -0.0007345362100750208,
- -0.0008179179858416319,
- -0.00010549465514486656,
- -0.0013330630026757717,
- -5.7338023907504976e-05,
- -0.0005571481888182461,
- -0.013437421061098576,
- -0.033829718828201294,
- -0.0004694551753345877,
- -0.28239941596984863,
- -1.3776881694793701,
+ -3.550320625305176,
+ -0.037021368741989136,
+ -0.7441027164459229,
+ -1.400524377822876,
+ -0.6082935333251953,
+ -6.150390625,
+ -0.9888088703155518,
+ -1.3728853464126587,
+ -0.4236694276332855,
+ -4.5586042404174805,
+ -0.41667699813842773,
+ -0.022858109325170517,
+ -9.846202738117427e-05,
+ -0.009947134181857109,
+ -0.22649677097797394,
+ -0.020902235060930252,
-0.00014256415306590497,
- -1.4336698055267334,
- -0.9458242654800415,
- -0.0002739054325502366,
- -0.0015444743912667036,
- -2.169585604860913e-05,
- -5.267784118652344,
- -2.617713689804077,
- -0.1205064058303833,
- -0.000608854868914932,
- -2.47952248173533e-05,
- -6.116018772125244,
- -0.06051409989595413,
- -0.0021291938610374928,
- -2.777537883957848e-05,
- -0.5082104206085205,
- -0.0008528171456418931,
- -0.013313560746610165,
+ -9.405170567333698e-05,
+ -0.00010013079008786008,
+ -0.004698308650404215,
+ -0.00032884435495361686,
+ -0.0007784912013448775,
-9.381330892210826e-05,
- -6.970278739929199,
- -0.3628937304019928,
- -1.40151047706604,
- -0.8361061811447144,
- -0.4778183400630951,
- -2.494100570678711,
- -0.3126090466976166,
- -7.66262674331665,
- -0.3505229353904724,
- -2.1190404891967773,
- -0.08990062028169632,
- -8.201262971851975e-05,
- -0.01644204556941986,
- -0.1838725060224533,
- -0.015538694337010384,
- -0.00019107422849629074,
- -7.915183232398704e-05,
- -0.0001382732152706012,
- -0.0002119316632160917,
- -0.0004773192631546408,
- -0.0004781533498317003,
- -4.994744449504651e-05,
- -0.0011807858245447278,
- -3.0636318115284666e-05,
- -0.0003046525234822184,
- -0.0024103655014187098,
- -0.009829924441874027,
- -0.00022301571152638644,
- -0.12844854593276978,
- -1.1151821613311768,
- -9.512448741588742e-05,
- -1.1148451566696167,
- -0.45424169301986694,
- -7.128461584215984e-05,
- -0.001427346607670188,
- -1.2040065485052764e-05,
- -3.9783990383148193,
- -0.025781046599149704,
- -0.00015496007108595222,
- -0.003944831434637308,
- -0.000663894519675523,
- -3.015949550899677e-05,
- -0.15718017518520355,
- -0.0009197533945553005,
- -0.0007913556764833629,
- -1.8000440832111053e-05,
- -0.18712174892425537,
- -0.00016604475968051702,
- -0.0022110319696366787,
- -2.169585604860913e-05,
- -0.014111850410699844,
- -1.1920922133867862e-06,
- -0.00984656810760498,
- -0.5971966981887817,
- -2.393812894821167,
- -0.010224700905382633,
- -0.009953508153557777,
- -7.64102369430475e-05,
- -0.011833352968096733,
- -0.26886406540870667,
- -0.023419089615345,
- -0.00019762947340495884,
- -6.031808152329177e-05,
- -0.00010191874753218144,
- -0.00015889335190877318,
- -0.0003564914222806692,
- -0.0004101150552742183,
- -6.675497570540756e-05,
- -0.0009184433147311211,
- -3.158996332786046e-05,
- -0.00031442465842701495,
- -0.0027259355410933495,
- -0.008694176562130451,
- -0.00032658010604791343,
- -0.289438933134079,
- -2.1416351795196533,
- -0.00017987063620239496,
- -1.8434972763061523,
- -1.624247670173645,
- -0.00022980909852776676,
- -0.0006792622152715921,
- -1.0967194612021558e-05,
- -1.281017541885376,
- -0.01736496575176716,
- -1.955749750137329,
- -1.528749942779541,
- -2.776960611343384,
- -0.5374854803085327,
- -0.00029345019720494747,
- -2.539125671319198e-05,
- -3.0065665245056152,
- -0.0013523490633815527,
- -0.0007908792467787862,
- -1.4543427823809907e-05,
- -0.23400214314460754,
- -0.0002324311062693596,
- -0.010042970068752766,
- -4.088794958079234e-05,
- -2.1034951210021973,
- -6.140199184417725,
- -4.464273929595947,
- -1.9943883419036865,
- -0.2878473103046417,
- -0.05924016237258911,
- -0.7345774173736572,
- -0.011171765625476837,
- -0.0002982171718031168,
- -0.14330486953258514,
- -0.0007319155265577137,
- -0.0003812778159044683,
- -0.002302616136148572,
- -0.36087724566459656,
- -0.08833581954240799,
- -2.631582260131836,
- -3.1771137714385986,
- -0.11841163039207458,
- -4.482168878894299e-05,
- -0.014765388332307339,
- -0.17005765438079834,
- -0.010167589411139488,
- -0.00010823617776622996,
- -3.6477376852417365e-05,
- -5.936446541454643e-05,
- -0.00023493390472140163,
- -0.0003688847064040601,
- -0.000321336614433676,
- -4.756337511935271e-05,
- -0.000902007392141968,
- -2.9205850296420977e-05,
- -0.00024423000286333263,
- -0.000964533886872232,
- -0.00411722669377923,
- -0.0002711643755901605,
- -0.3081328868865967,
- -0.4985820949077606,
- -0.00018726025882642716,
- -1.1391643285751343,
- -0.27228832244873047,
- -4.2914423829643056e-05,
- -0.0012028133496642113,
- -1.9311717551317997e-05,
- -1.1735807657241821,
- -0.07005516439676285,
- -0.0024717275518924,
- -8.618460560683161e-05,
- -0.00016866691294126213,
- -0.00044764988706447184,
- -1.6093124941107817e-05,
- -8.586283683776855,
- -0.0002851079625543207,
- -7.490447998046875,
- -0.09369903802871704,
- -0.004145600367337465,
- -0.0008606782066635787,
- -4.827859811484814e-05,
- -0.7127438187599182,
- -0.0003618539194576442,
- -0.015226203016936779,
- -6.401333666872233e-05,
- -3.530060291290283,
- -0.040570154786109924,
- -0.7448150515556335,
- -1.4005241394042969,
- -0.5872946977615356,
- -6.073245048522949,
- -0.9850690364837646,
- -1.4459205865859985,
- -0.4346452057361603,
- -4.452149868011475,
- -0.3939701318740845,
- -0.02252959832549095,
- -9.440929716220126e-05,
- -0.012161390855908394,
- -0.25266116857528687,
- -0.021285664290189743,
- -0.00015770144818816334,
- -9.870042413240299e-05,
- -9.989239333663136e-05,
- -0.005311425309628248,
- -0.00032634177478030324,
- -0.0007045170641504228,
- -9.417090768693015e-05,
- -0.001260558608919382,
- -4.482168878894299e-05,
- -0.0003833036171272397,
- -0.0023484050761908293,
- -0.011129915714263916,
- -0.00040260792593471706,
- -0.1819346845149994,
- -1.1781600713729858,
- -0.00033241944038309157,
- -1.3525464534759521,
- -1.2726483345031738,
- -0.00018034738604910672,
- -0.0009054613183252513,
- -1.2040065485052764e-05,
- -1.7329559326171875,
- -0.009877022355794907,
- -0.030561018735170364,
- -0.9567705988883972,
- -0.0002079985715681687,
- -0.0003582789213396609,
+ -0.0012371037155389786,
+ -3.4927710657939315e-05,
+ -0.00037448544753715396,
+ -0.002363271312788129,
+ -0.011125907301902771,
+ -0.00035613393993116915,
+ -0.20121917128562927,
+ -1.1733814477920532,
+ -0.0003303935518488288,
+ -1.3426117897033691,
+ -1.271085262298584,
+ -0.00018285033002030104,
+ -0.0009109399979934096,
+ -1.168244216387393e-05,
+ -1.7527191638946533,
+ -0.009883751161396503,
+ -0.03264499828219414,
+ -1.0129823684692383,
+ -0.00021395778458099812,
+ -0.0002731903805397451,
-2.5510462364763953e-05,
- -1.3376575708389282,
- -0.043758541345596313,
- -0.0005255748401395977,
- -0.003921795636415482,
- -3.9934315282152966e-05,
- -0.013946342281997204,
- -0.001447345013730228,
- -0.09289155900478363,
- -0.00028975578607060015,
- -5.025714874267578,
- -5.600637435913086,
- -0.8190056681632996,
- -2.0997657775878906,
- -1.5471020936965942,
- -0.2830793261528015,
- -0.099715456366539,
- -0.00015341058315243572,
- -0.09538150578737259,
- -0.9440865516662598,
- -0.13964560627937317,
- -0.0003178806509822607,
- -0.00015531764074694365,
- -0.00016640232934150845,
- -0.00023398046323563904,
- -0.00039081089198589325,
- -0.0015487592900171876,
- -0.00010716341057559475,
- -0.0017987991450354457,
- -3.838465272565372e-05,
- -0.0006412595394067466,
- -0.00545145571231842,
- -0.02335585467517376,
- -0.0004077318590134382,
- -0.8720157146453857,
- -0.10373511165380478,
- -0.00014077626110520214,
- -0.5180479884147644,
- -0.17388182878494263,
- -0.00015746307326480746,
- -0.0043711354956030846,
- -2.9801878554280847e-05,
- -2.0693466663360596,
- -0.007648942526429892,
- -2.8729025871143676e-05,
- -0.0003301552205812186,
- -0.000542612629942596,
- -3.2543604902457446e-05,
- -0.27388375997543335,
- -0.00043752157944254577,
- -0.0005888396990485489,
- -1.7762025890988298e-05,
- -0.05423494055867195,
- -7.915183232398704e-05,
- -0.002435457892715931,
- -1.1205610462639015e-05,
- -0.01761529967188835,
- -7.152555099310121e-07,
- -0.005352570675313473,
- -0.1280955821275711,
- -2.3187625408172607,
- -0.009216856211423874,
- -0.008558499626815319,
- -0.0001072826053132303,
- -0.04680917039513588,
- -0.5660229325294495,
- -0.04951385408639908,
- -0.0002015625941567123,
+ -1.3228294849395752,
+ -0.04179994761943817,
+ -0.0005071069463156164,
+ -0.003949818201363087,
+ -4.2199197196168825e-05,
+ -0.013934234157204628,
+ -0.001443773857317865,
+ -0.09278758615255356,
+ -0.00022623363474849612,
+ -5.036643028259277,
+ -5.596894264221191,
+ -0.8243252635002136,
+ -2.068366050720215,
+ -1.5817551612854004,
+ -0.28322750329971313,
+ -0.10253381729125977,
+ -0.0001525762490928173,
+ -0.09576807916164398,
+ -0.9411507844924927,
+ -0.12982413172721863,
+ -0.00031871485407464206,
+ -0.0001546025014249608,
+ -0.00016318420239258558,
+ -0.00018702188390307128,
+ -0.00034338299883529544,
+ -0.0015476880362257361,
+ -0.00011431517486926168,
+ -0.0018185520311817527,
+ -3.8265450712060556e-05,
+ -0.0006036128615960479,
+ -0.005799253936856985,
+ -0.023355038836598396,
+ -0.00040618274942971766,
+ -0.8012296557426453,
+ -0.10373521596193314,
+ -0.00012778419477399439,
+ -0.4956749379634857,
+ -0.16527129709720612,
+ -0.0001530530134914443,
+ -0.004374577663838863,
+ -2.9205850296420977e-05,
+ -1.9665766954421997,
+ -0.00752566521987319,
+ -2.7417760065873154e-05,
+ -0.0003748429589904845,
+ -0.0005439232336357236,
+ -2.3245540432981215e-05,
+ -0.27362146973609924,
+ -0.00042989550274796784,
+ -0.0005857420619577169,
+ -1.764281842042692e-05,
+ -0.05366183817386627,
+ -6.425174069590867e-05,
+ -0.0021969962399452925,
+ -1.1324817933200393e-05,
+ -0.017568564042448997,
+ -8.344646857949556e-07,
+ -0.005509074777364731,
+ -0.1280696839094162,
+ -2.3235721588134766,
+ -0.008202792145311832,
+ -0.008413229137659073,
+ -0.00012110930401831865,
+ -0.042223334312438965,
+ -0.5657049417495728,
+ -0.05260366573929787,
+ -0.0002240883477497846,
-5.8410845667822286e-05,
- -9.440929716220126e-05,
- -0.00014828535495325923,
- -0.00037245964631438255,
- -0.0008362610242329538,
- -5.4596363042946905e-05,
- -0.0010970771545544267,
- -4.017272294731811e-05,
- -0.0004563482361845672,
- -0.0021864098962396383,
- -0.012597862631082535,
- -0.00036435641231946647,
- -0.07823580503463745,
- -1.1245288848876953,
+ -9.595887240720913e-05,
-0.0001472126314183697,
- -2.1236472129821777,
- -0.25363627076148987,
- -0.00011646069469861686,
- -0.0010031197452917695,
- -1.4662635294371285e-05,
- -11.853788375854492,
- -1.5205868482589722,
- -0.0017375147435814142,
- -0.00013374387344811112,
- -7.155948638916016,
- -3.82474422454834,
- -1.2793458700180054,
- -0.03748536482453346,
- -0.005961020477116108,
- -5.829164365422912e-05,
- -3.1456170082092285,
- -0.03318829461932182,
- -0.008591356687247753,
- -0.027652040123939514,
- -0.00012885693286079913,
- -1.5415722131729126,
- -0.979039192199707,
- -2.842726469039917,
- -9.05957317352295,
- -2.8234424591064453,
- -0.8373243808746338,
- -0.4019332230091095,
- -0.0004048719711136073,
- -0.03923225402832031,
- -0.4254666864871979,
- -0.027653662487864494,
- -0.0003177614707965404,
- -0.0001967951684491709,
- -0.00020883286197204143,
- -0.00025674383505247533,
- -0.0008311392739415169,
- -0.0012284121476113796,
+ -0.00037472377880476415,
+ -0.0007722970913164318,
+ -5.447716102935374e-05,
+ -0.0010968389688059688,
+ -3.266281055402942e-05,
+ -0.00045622908510267735,
+ -0.0020772088319063187,
+ -0.011120365932583809,
+ -0.00032586511224508286,
+ -0.07033942639827728,
+ -1.1209707260131836,
+ -0.00014518637908622622,
+ -2.0970425605773926,
+ -0.25196295976638794,
+ -0.00011765264935093,
+ -0.0009179668850265443,
+ -1.537788011773955e-05,
+ -11.904622077941895,
+ -1.5177723169326782,
+ -0.001602795091457665,
+ -0.00013302871957421303,
+ -7.185190200805664,
+ -3.704270601272583,
+ -1.288861632347107,
+ -0.03340679779648781,
+ -0.005939452908933163,
+ -4.494089080253616e-05,
+ -3.2633724212646484,
+ -0.033210549503564835,
+ -0.008218162693083286,
+ -0.02734771929681301,
+ -0.00012838016846217215,
+ -1.5429582595825195,
+ -1.0140559673309326,
+ -2.7621612548828125,
+ -9.184652328491211,
+ -2.8712878227233887,
+ -0.8348691463470459,
+ -0.4236343204975128,
+ -0.00042012447374872863,
+ -0.039101459085941315,
+ -0.42557674646377563,
+ -0.03131643310189247,
+ -0.0003195490571670234,
+ -0.00020704510097857565,
+ -0.00019476900342851877,
+ -0.000254241080256179,
+ -0.0008852138998918235,
+ -0.001123035908676684,
-0.00010787858627736568,
- -0.0024356956128031015,
- -6.258291978156194e-05,
- -0.00048565989709459245,
- -0.0021678535267710686,
- -0.012607751414179802,
- -0.00023588736075907946,
- -0.11036524921655655,
- -0.5750182867050171,
- -0.00017176583060063422,
- -1.9862632751464844,
- -1.2351702451705933,
- -0.00037520044133998454,
- -0.0013566347770392895,
- -2.5152843591058627e-05,
- -2.1086387634277344,
- -7.917232990264893,
- -0.05708145350217819,
- -0.06208256632089615,
- -0.000644237850792706,
- -8.308542601298541e-05,
- -5.1276655197143555,
- -0.16815905272960663,
- -0.0012461524456739426,
- -5.94836674281396e-05,
- -3.559391736984253,
- -5.411561965942383,
- -0.022293083369731903,
- -0.0005644158809445798,
- -0.017552750185132027,
- -0.00038842763751745224,
- -1.8479862213134766,
- -0.004095145035535097,
- -11.830594062805176,
- -0.4279360771179199,
- -3.7062158584594727,
- -2.9457836151123047,
- -1.9491567611694336,
- -0.06489256024360657,
- -0.00013660451804753393,
- -0.012157151475548744,
- -0.22074609994888306,
- -0.021073833107948303,
- -0.00021300431399140507,
- -0.00017593742813915014,
- -0.00023672162205912173,
+ -0.002168924082070589,
+ -5.566918844124302e-05,
+ -0.0005374894244596362,
+ -0.002329495269805193,
+ -0.014262876473367214,
+ -0.00026675479602999985,
+ -0.1103726178407669,
+ -0.6316244602203369,
+ -0.00015686711412854493,
+ -1.986260175704956,
+ -1.2387672662734985,
+ -0.00037448544753715396,
+ -0.0013405631761997938,
+ -2.5629668016335927e-05,
+ -2.157573699951172,
+ -7.917330741882324,
+ -0.056897230446338654,
+ -0.06321369111537933,
+ -0.0006852186052128673,
+ -8.248942322097719e-05,
+ -5.13627815246582,
+ -0.18123063445091248,
+ -0.0012298409128561616,
+ -5.936446541454643e-05,
+ -3.441296100616455,
+ -5.40785026550293,
+ -0.022266386076807976,
+ -0.0006132629350759089,
+ -0.017490316182374954,
+ -0.0003912875254172832,
+ -1.8065340518951416,
+ -0.004082322586327791,
+ -11.884273529052734,
+ -0.38534700870513916,
+ -3.640864849090576,
+ -2.807778835296631,
+ -1.9425075054168701,
+ -0.06503599882125854,
+ -0.00013207517622504383,
+ -0.012374058365821838,
+ -0.22333350777626038,
+ -0.02102574147284031,
+ -0.00023850933939684182,
+ -0.00015496007108595222,
+ -0.00024125049822032452,
-0.0003091811086051166,
- -0.0014552014181390405,
- -0.0013881819322705269,
- -0.00015245705435518175,
- -0.002331279218196869,
- -5.4238757002167404e-05,
- -0.000668659748043865,
- -0.002430463209748268,
- -0.016187194734811783,
- -0.0002441108226776123,
- -1.4263010025024414,
- -0.30179885029792786,
- -0.0001770101225702092,
- -0.5045080184936523,
- -0.07310019433498383,
+ -0.0015404274454340339,
+ -0.0015067425556480885,
+ -0.00014733182615600526,
+ -0.0023534002248197794,
+ -6.806619057897478e-05,
+ -0.0006721144891344011,
+ -0.0022511158604174852,
+ -0.0161884855479002,
+ -0.00022396916756406426,
+ -1.423966407775879,
+ -0.30094146728515625,
+ -0.0001429217227268964,
+ -0.527908444404602,
+ -0.07172392308712006,
-8.022463589441031e-05,
- -0.002168329432606697,
- -2.3841574147809297e-05,
- -1.7808306217193604,
- -0.02828705683350563,
- -6.115249561844394e-05,
- -0.0008904544520191848,
- -0.0005335576133802533,
- -3.957670196541585e-05,
- -0.03801318258047104,
- -0.0003077510336879641,
- -0.0005035324720665812,
- -2.169585604860913e-05,
- -0.02271897904574871,
- -3.1709168979432434e-05,
- -0.0018041539005935192,
- -1.8358061424805783e-05,
- -0.005899516865611076,
- -1.1920922133867862e-06,
- -0.002030455507338047,
- -0.27544423937797546,
- -1.1146715879440308,
- -0.012286689132452011,
- -0.004974251613020897,
- -6.389413465512916e-05,
- -0.010529793798923492,
- -0.2302529364824295,
- -0.015527778305113316,
- -0.00019524575327523053,
- -6.389413465512916e-05,
- -0.00013815402053296566,
- -0.00018165845540352166,
- -0.0005564333405345678,
- -0.000959531927946955,
- -6.151010165922344e-05,
- -0.001416394836269319,
- -5.531158240046352e-05,
- -0.00035363141796551645,
+ -0.001690983772277832,
+ -2.455681169521995e-05,
+ -1.7886372804641724,
+ -0.02579452283680439,
+ -6.294052582234144e-05,
+ -0.000840906286612153,
+ -0.0005334384622983634,
+ -4.0649541915627196e-05,
+ -0.03599982336163521,
+ -0.0002740246127359569,
+ -0.0005314130103215575,
+ -1.9788545614574105e-05,
+ -0.023407327011227608,
+ -3.182837463100441e-05,
+ -0.0016145778354257345,
+ -1.4185804502631072e-05,
+ -0.005731695331633091,
+ -9.536738616588991e-07,
+ -0.0020591262727975845,
+ -0.28849226236343384,
+ -1.1147009134292603,
+ -0.011014376766979694,
+ -0.004826680291444063,
+ -7.10462118149735e-05,
+ -0.011995912529528141,
+ -0.229905366897583,
+ -0.017444511875510216,
+ -0.00017188502533826977,
+ -8.189342770492658e-05,
+ -0.0001358893496217206,
+ -0.0001652104256208986,
+ -0.0005519058904610574,
+ -0.0009560781763866544,
+ -5.94836674281396e-05,
+ -0.0014123475411906838,
+ -4.589452510117553e-05,
+ -0.00034564718953333795,
-0.0010683787986636162,
- -0.012577733024954796,
+ -0.01257891021668911,
-0.00023934361524879932,
- -0.06311207264661789,
- -0.972044050693512,
- -0.00019929806876461953,
- -1.6224243640899658,
- -0.8333836197853088,
- -0.00016592556494288146,
- -0.0008984343148767948,
- -1.6927575416048057e-05,
- -0.8844207525253296,
- -0.023736946284770966,
- -4.01811408996582,
- -1.6215615272521973,
- -0.33087965846061707,
- -0.0035197706893086433,
- -0.00024148885859176517,
- -3.0874729418428615e-05,
- -3.097301721572876,
- -0.030017103999853134,
- -0.0006585336523130536,
- -1.9430925021879375e-05,
- -0.49424058198928833,
- -0.0001401803019689396,
- -0.00554167665541172,
+ -0.05590301379561424,
+ -0.9510307312011719,
+ -0.00019429226813372225,
+ -1.614063024520874,
+ -0.8332399129867554,
+ -0.0001472126314183697,
+ -0.0008833082392811775,
+ -1.680836794548668e-05,
+ -0.9284455180168152,
+ -0.02347230538725853,
+ -4.0024094581604,
+ -1.6216294765472412,
+ -0.36597940325737,
+ -0.003108671400696039,
+ -0.0002397011558059603,
+ -3.266281055402942e-05,
+ -2.997518539428711,
+ -0.03157311677932739,
+ -0.0006841464783065021,
-1.9073304429184645e-05,
- -0.5312279462814331,
- -5.748266220092773,
- -11.324613571166992,
- -1.1340491771697998,
- -0.16082678735256195,
- -0.8938052654266357,
- -3.726792335510254,
- -0.8781039714813232,
- -0.00017355366435367614,
- -0.009945128113031387,
- -0.18626560270786285,
- -0.013042616657912731,
- -0.00010859376925509423,
- -7.199982064776123e-05,
- -0.00010871296399272978,
- -0.00017796363681554794,
- -0.00034767304896377027,
- -0.0006170752458274364,
- -3.0636318115284666e-05,
- -0.001077071763575077,
- -4.076874756719917e-05,
- -0.00024029705673456192,
- -0.000982159748673439,
- -0.02636047638952732,
- -0.00021920185827184469,
- -0.632880687713623,
- -0.06617539376020432,
- -0.00016318420239258558,
- -0.4156720042228699,
- -0.034620899707078934,
- -5.6622808187967166e-05,
- -0.0011695933062583208,
- -1.597391747054644e-05,
- -10.639490127563477,
- -0.24528348445892334,
- -0.06833283603191376,
- -0.0033608165103942156,
- -0.02616957761347294,
- -0.00036054308293387294,
- -3.099393507000059e-05,
- -4.044595241546631,
- -2.188387393951416,
- -0.32720163464546204,
- -0.00974209699779749,
- -0.0011126763420179486,
- -3.302042750874534e-05,
- -0.19868847727775574,
- -7.56950321374461e-05,
- -0.005233398173004389,
+ -0.47262415289878845,
+ -0.00013982271775603294,
+ -0.005081594921648502,
+ -1.9192511899746023e-05,
+ -2.0542843341827393,
+ -5.7443647384643555,
+ -1.3145296573638916,
+ -1.1412832736968994,
+ -0.157257080078125,
+ -0.8434399366378784,
+ -3.784353733062744,
+ -0.8877203464508057,
+ -0.0001784403866622597,
+ -0.009449399076402187,
+ -0.16757570207118988,
+ -0.012964131310582161,
+ -0.00010609064338495955,
+ -7.164221460698172e-05,
+ -0.00012051333033014089,
+ -0.00016819016309455037,
+ -0.0003797286772169173,
+ -0.0006151691195555031,
+ -3.766942609217949e-05,
+ -0.001149470335803926,
+ -4.207999518257566e-05,
+ -0.00023898606013972312,
+ -0.0010438475292176008,
+ -0.026360591873526573,
+ -0.0002196785935666412,
+ -0.632854163646698,
+ -0.06455802172422409,
+ -0.00020728347590193152,
+ -0.37605050206184387,
+ -0.03835982084274292,
+ -5.304672595229931e-05,
+ -0.001157805323600769,
+ -1.5616295058862306e-05,
+ -10.66722297668457,
+ -0.23820555210113525,
+ -0.07690379023551941,
+ -0.0033806574065238237,
+ -0.02564801648259163,
+ -0.0004828002711292356,
+ -3.123234637314454e-05,
+ -4.0631422996521,
+ -2.188638687133789,
+ -0.3299391269683838,
+ -0.009875488467514515,
+ -0.0011928117601200938,
+ -2.9682672902708873e-05,
+ -0.22228410840034485,
+ -7.366862701019272e-05,
+ -0.005895250476896763,
-3.158996332786046e-05,
- -1.839617371559143,
- -0.17654305696487427,
- -0.7875567078590393,
- -2.1537787914276123,
- -0.3631034195423126,
- -0.9216613173484802,
- -2.0036990642547607,
- -0.09243497252464294,
- -0.00010740180005086586,
- -0.018314307555556297,
- -0.208140030503273,
- -0.01576320081949234,
- -0.00013136000779923052,
- -7.390703103737906e-05,
+ -1.8109334707260132,
+ -0.17614930868148804,
+ -0.7870113849639893,
+ -2.1647791862487793,
+ -0.40118229389190674,
+ -0.9958465695381165,
+ -2.018829584121704,
+ -0.09130532294511795,
-0.00011264643399044871,
- -0.00017045476124621928,
- -0.0005171154043637216,
- -0.0005422552349045873,
- -3.349725011503324e-05,
- -0.0013309201458469033,
- -4.255681051290594e-05,
- -0.00023767507809679955,
+ -0.017000189051032066,
+ -0.20741263031959534,
+ -0.015425659716129303,
+ -0.0001161031104857102,
+ -7.772143726469949e-05,
+ -0.00012587709352374077,
+ -0.00021896349790040404,
+ -0.0004892344586551189,
+ -0.0006121906917542219,
+ -3.516612196108326e-05,
+ -0.001400562352500856,
+ -4.1126360883936286e-05,
+ -0.00023803261865396053,
-0.001095648156479001,
- -0.14277544617652893,
+ -0.142775759100914,
-0.00021371940965764225,
- -0.00032217081752605736,
- -0.35286909341812134,
- -0.0002668739762157202,
- -1.7962173223495483,
- -0.07211553305387497,
- -7.974783511599526e-05,
- -0.000621959799900651,
- -1.2874520507466514e-05,
- -1.9048426151275635,
- -0.022713735699653625,
- -3.9457496313843876e-05,
- -0.0005820487276650965,
- -0.0002401778765488416,
- -3.325883881188929e-05,
- -0.02081700973212719,
- -0.00022492263815365732,
- -0.0003299168893136084,
- -2.038458114839159e-05,
- -0.008293120190501213,
- -1.7404405298293568e-05,
- -0.0012493670219555497,
- -1.4424220353248529e-05,
- -0.0041636452078819275,
+ -0.0004058252670802176,
+ -0.35273799300193787,
+ -0.0002616301644593477,
+ -1.8941857814788818,
+ -0.07712067663669586,
+ -7.354942499659956e-05,
+ -0.000623032043222338,
+ -1.2993727978027891e-05,
+ -1.8126565217971802,
+ -0.02175048366189003,
+ -3.8980677345534787e-05,
+ -0.0005631053354591131,
+ -0.0002426806604489684,
+ -3.099393507000059e-05,
+ -0.02033376693725586,
+ -0.000196556793525815,
+ -0.0003359945258125663,
+ -2.1815061700181104e-05,
+ -0.008229749277234077,
+ -1.7523612768854946e-05,
+ -0.0011329191038385034,
+ -1.4066597032069694e-05,
+ -0.004424781538546085,
-8.344646857949556e-07,
- -0.0020267677027732134,
- -0.13429519534111023,
- -1.9221405982971191,
- -0.0093602379783988,
- -0.005981876514852047,
+ -0.002038069535046816,
+ -0.15088149905204773,
+ -1.922121286392212,
+ -0.009393423795700073,
+ -0.006058305036276579,
-5.817244164063595e-05,
- -0.019257837906479836,
- -0.27827900648117065,
- -0.01921457052230835,
- -0.0001652104256208986,
- -8.546940807718784e-05,
- -0.0001510267611593008,
- -0.00016366096679121256,
- -0.0002616301644593477,
- -0.0005458295345306396,
- -3.480850500636734e-05,
- -0.0010807631770148873,
- -3.7431014789035544e-05,
- -0.0003626880934461951,
- -0.0010880271438509226,
- -0.6327179670333862,
- -0.0002374367177253589,
- -0.020488178357481956,
- -0.10384052991867065,
- -0.0001971527235582471,
- -0.16368740797042847,
- -0.026392173022031784,
- -0.00012170527770649642,
- -0.0025978884659707546,
- -1.9430925021879375e-05,
- -7.9701642990112305,
- -1.6003714799880981,
- -0.2391909956932068,
- -0.000502817565575242,
- -4.9232225137529895e-05,
- -4.135532855987549,
- -0.06158669665455818,
- -0.00044371772673912346,
- -3.755022044060752e-05,
- -0.18109248578548431,
- -0.00010883215873036534,
- -0.006367869209498167,
- -7.748303323751315e-05,
- -5.440160751342773,
- -5.081888198852539,
- -0.19470839202404022,
- -2.9904420375823975,
- -2.4235076904296875,
- -0.032352350652217865,
- -0.00044907975825481117,
- -0.04121795669198036,
- -0.43260514736175537,
- -0.04605478420853615,
- -0.00023982033599168062,
- -0.0003178806509822607,
- -0.00017188502533826977,
- -0.00022468426323030144,
- -0.0003400462737772614,
- -0.0010152667528018355,
- -0.00011729506513802335,
- -0.001335324952378869,
- -4.8874615458771586e-05,
- -0.001257463125512004,
- -0.004097400698810816,
- -0.0008996253600344062,
- -0.0002967870968859643,
- -0.15579743683338165,
- -1.3731565475463867,
- -0.00023183519078884274,
- -2.0089190006256104,
- -3.441042423248291,
- -0.0006145734223537147,
- -0.0012832987122237682,
- -1.9550132492440753e-05,
- -1.731110692024231,
- -0.027068600058555603,
- -2.8266828060150146,
- -0.35935577750205994,
- -0.023644626140594482,
- -0.0005504761938937008,
- -0.00017951308109331876,
- -2.396077979938127e-05,
- -2.3206820487976074,
- -0.003744971938431263,
- -0.000205018965061754,
- -2.288792165927589e-05,
- -0.08958229422569275,
- -6.592056161025539e-05,
- -0.0021721357479691505,
- -3.0397906812140718e-05,
- -4.5939412117004395,
- -8.534799575805664,
- -3.483549118041992,
- -1.681600570678711,
- -0.7201917767524719,
- -0.530266284942627,
- -0.7154921293258667,
- -2.835704803466797,
- -0.0004451475979294628,
- -0.02453603409230709,
- -0.31538400053977966,
- -0.0156102878972888,
- -0.00013124081306159496,
- -8.892617915989831e-05,
- -9.738924563862383e-05,
- -0.0011036264477297664,
- -0.00030357998912222683,
- -0.0010406322544440627,
- -6.0437283536884934e-05,
- -0.0014225849881768227,
- -3.671578815556131e-05,
- -0.00044705410255119205,
- -0.005232923664152622,
- -0.0001565095444675535,
- -0.0003033416287507862,
- -0.18575794994831085,
- -0.14061033725738525,
- -0.0002706876548472792,
- -0.5223819017410278,
- -0.035896092653274536,
- -5.4834770708112046e-05,
- -0.0012011463986709714,
- -1.6569954823353328e-05,
- -1.681032657623291,
- -0.011652856133878231,
- -1.6569954823353328e-05,
- -0.00047469791024923325,
- -0.000256982195423916,
- -3.361645576660521e-05,
- -0.01372707262635231,
- -0.00014852374442853034,
- -0.00046695294440723956,
- -2.288792165927589e-05,
- -0.0034659572411328554,
- -1.3708974620385561e-05,
- -0.0015382850542664528,
- -8.702239938429557e-06,
- -0.003346678102388978,
- -7.152555099310121e-07,
- -0.000867467257194221,
- -0.02539108693599701,
- -1.0509589910507202,
- -0.002976156771183014,
- -0.005069141276180744,
- -5.590759246842936e-05,
- -0.015196850523352623,
- -0.3093729317188263,
- -0.02090352028608322,
- -0.00013958434283267707,
- -6.460934673668817e-05,
- -8.296622399939224e-05,
- -0.0004457433824427426,
- -0.0005041282274760306,
- -0.0011976935202255845,
- -4.2914423829643056e-05,
- -0.0011085085570812225,
- -4.160317621426657e-05,
- -0.0005018643569201231,
- -0.004558410029858351,
- -9.476689592702314e-05,
- -0.00037269797758199275,
- -0.11347992718219757,
- -0.450020968914032,
- -0.0003301552205812186,
- -2.8804092407226562,
+ -0.017619749531149864,
+ -0.2778732478618622,
+ -0.020725470036268234,
+ -0.00014554394874721766,
+ -0.00010918975021922961,
+ -0.00012015574611723423,
+ -0.00015615197480656207,
+ -0.000262106885202229,
+ -0.0005429700831882656,
+ -3.3854863431770355e-05,
+ -0.0009723940747790039,
+ -4.3748852476710454e-05,
+ -0.0003530356043484062,
+ -0.0011967408936470747,
+ -0.6327288150787354,
+ -0.00023803261865396053,
+ -0.018105357885360718,
+ -0.09554284811019897,
+ -0.00019762947340495884,
+ -0.1636878103017807,
+ -0.029755044728517532,
+ -0.00012027494085486978,
+ -0.0026007420383393764,
+ -1.9192511899746023e-05,
+ -7.907725811004639,
+ -1.5921536684036255,
+ -0.26656997203826904,
+ -0.0005077026435174048,
+ -4.5060096454108134e-05,
+ -4.135128021240234,
+ -0.05467758700251579,
+ -0.000444551813416183,
+ -3.158996332786046e-05,
+ -0.18111994862556458,
+ -0.00011205045302631333,
+ -0.0063499826937913895,
+ -6.949660019017756e-05,
+ -5.444925308227539,
+ -5.080565452575684,
+ -0.19182506203651428,
+ -2.987712860107422,
+ -2.41878604888916,
+ -0.033490728586912155,
+ -0.0004508670826908201,
+ -0.03954065591096878,
+ -0.473906010389328,
+ -0.04054016247391701,
+ -0.00023624490131624043,
+ -0.0002554328821133822,
+ -0.00017534149810671806,
+ -0.00022349244682118297,
+ -0.00038914260221645236,
+ -0.0009974034037441015,
+ -0.00011860620725201443,
+ -0.0013256819220259786,
+ -5.245071224635467e-05,
+ -0.0010975534096360207,
+ -0.004627709276974201,
+ -0.0009116546134464443,
+ -0.0003274143091402948,
+ -0.1729803830385208,
+ -1.3756903409957886,
+ -0.00020394629973452538,
+ -2.0139496326446533,
+ -3.4418716430664062,
+ -0.0005534547381103039,
+ -0.0013410394312813878,
+ -2.002696055569686e-05,
+ -1.7302016019821167,
+ -0.02661030925810337,
+ -2.8714237213134766,
+ -0.3817090392112732,
+ -0.02706013061106205,
+ -0.0005696581210941076,
+ -0.00017414960893802345,
+ -2.2291887944447808e-05,
+ -2.393955707550049,
+ -0.0037057793233543634,
+ -0.00020561488054227084,
+ -2.3007127310847864e-05,
+ -0.08967646211385727,
+ -8.654219709569588e-05,
+ -0.001972516765818,
+ -2.7894584491150454e-05,
+ -4.549469947814941,
+ -8.546380996704102,
+ -3.5779974460601807,
+ -1.6664830446243286,
+ -0.7325427532196045,
+ -0.5301830768585205,
+ -0.7084848880767822,
+ -2.8345694541931152,
+ -0.0004906642716377974,
+ -0.02661263197660446,
+ -0.31519249081611633,
+ -0.015796994790434837,
+ -0.00013219437096267939,
+ -8.83301836438477e-05,
+ -9.619726915843785e-05,
+ -0.001105650793761015,
+ -0.0003044141922146082,
+ -0.0010319390567019582,
+ -5.745722592109814e-05,
+ -0.001517812255769968,
+ -3.683499380713329e-05,
+ -0.00039176418795250356,
+ -0.005421933718025684,
+ -0.00012611546844709665,
+ -0.0003044141922146082,
+ -0.18575865030288696,
+ -0.1348363161087036,
+ -0.00024780540843494236,
+ -0.5202672481536865,
+ -0.031899940222501755,
+ -5.507317473529838e-05,
+ -0.0010099078062921762,
+ -1.6212332411669195e-05,
+ -1.692710518836975,
+ -0.011726495809853077,
+ -1.9788545614574105e-05,
+ -0.0004633783537428826,
+ -0.00025054652360267937,
+ -3.135155202471651e-05,
+ -0.013559030368924141,
+ -0.00015043080202303827,
+ -0.0004663571889977902,
+ -2.1576648578047752e-05,
+ -0.0036806000862270594,
+ -1.7881233361549675e-05,
+ -0.001532214693725109,
+ -8.583032467868179e-06,
+ -0.0033115092664957047,
+ -5.960462772236497e-07,
+ -0.0009004590683616698,
+ -0.025490565225481987,
+ -0.9851224422454834,
+ -0.0029891119338572025,
+ -0.004928938578814268,
+ -5.817244164063595e-05,
+ -0.015181821770966053,
+ -0.3092004656791687,
+ -0.020951267331838608,
+ -0.00013302871957421303,
+ -7.033100700937212e-05,
+ -8.165503095369786e-05,
+ -0.00044526674901135266,
+ -0.00041130665340460837,
+ -0.0012225781101733446,
+ -4.2437604861333966e-05,
+ -0.0009884715545922518,
+ -3.957670196541585e-05,
+ -0.0005015069036744535,
+ -0.005099741276353598,
+ -7.521823135903105e-05,
+ -0.00032908268622122705,
+ -0.10078251361846924,
+ -0.4542389512062073,
+ -0.000291662581730634,
+ -2.8981728553771973,
-0.15156973898410797,
- -6.246371776796877e-05,
- -0.000683074293192476,
+ -6.329813186312094e-05,
+ -0.0006799769471399486,
-1.3947389561508317e-05,
- -2.0683939456939697,
- -0.02846144698560238,
- -0.04469490796327591,
- -1.889275074005127,
- -0.0001255195093108341,
- -0.00011228884250158444,
- -2.4914430468925275e-05,
- -7.980701446533203,
- -0.39261865615844727,
- -1.6454169750213623,
- -0.0018256916664540768,
- -0.0003761537664104253,
- -2.5987286790041253e-05,
- -0.27152737975120544,
- -3.8742269680369645e-05,
- -0.002314033918082714,
+ -1.9687650203704834,
+ -0.032300759106874466,
+ -0.04720155894756317,
+ -1.8380577564239502,
+ -0.00012540031457319856,
+ -8.880697714630514e-05,
+ -2.4437606043647975e-05,
+ -8.028385162353516,
+ -0.3949255347251892,
+ -1.6445553302764893,
+ -0.0018342590192332864,
+ -0.00035768310772255063,
+ -3.0040289857424796e-05,
+ -0.26507413387298584,
+ -3.576214658096433e-05,
+ -0.0023099901154637337,
-5.364274329622276e-05,
- -5.172288417816162,
- -0.007181781344115734,
- -0.8884671330451965,
- -0.20681926608085632,
- -1.529428243637085,
- -2.335056781768799,
- -0.02583100087940693,
- -1.8960939645767212,
- -0.257112592458725,
- -0.1720065474510193,
- -8.284702198579907e-05,
- -0.011070851236581802,
- -0.16333311796188354,
- -0.01678428426384926,
- -0.00010024998482549563,
- -4.911301948595792e-05,
- -6.41325386823155e-05,
- -0.0003518439189065248,
- -0.0003983181086368859,
- -0.0007211944903247058,
- -2.253030106658116e-05,
- -0.0009076051646843553,
- -2.884823152271565e-05,
- -0.00033682872890494764,
- -0.01127432007342577,
- -5.113947918289341e-05,
- -0.0003095386200584471,
- -0.162703275680542,
- -0.12824533879756927,
- -0.0002037079248111695,
- -0.5378345251083374,
- -0.013359789736568928,
- -4.625213477993384e-05,
- -0.0007819455349817872,
- -1.2993727978027891e-05,
- -1.4531102180480957,
- -0.9376159310340881,
- -0.02013481967151165,
- -3.182837463100441e-05,
- -0.00028391621890477836,
- -0.0002040654799202457,
+ -5.063076496124268,
+ -0.00801442563533783,
+ -0.8903906941413879,
+ -0.20995336771011353,
+ -1.5289034843444824,
+ -2.236264944076538,
+ -0.025192562490701675,
+ -1.8960940837860107,
+ -0.27068760991096497,
+ -0.17904573678970337,
+ -8.797258487902582e-05,
+ -0.010375725105404854,
+ -0.16252535581588745,
+ -0.015135912224650383,
+ -0.0001006075763143599,
+ -4.565611743601039e-05,
+ -6.782778655178845e-05,
+ -0.0003756771038752049,
+ -0.0004158347437623888,
+ -0.0007250064518302679,
+ -2.3483953555114567e-05,
+ -0.0009037939016707242,
+ -3.6954195820726454e-05,
+ -0.0003305127320345491,
+ -0.011275380849838257,
+ -5.006664650863968e-05,
+ -0.00030858523678034544,
+ -0.1449013352394104,
+ -0.14131033420562744,
+ -0.00020358874462544918,
+ -0.5081860423088074,
+ -0.013270506635308266,
+ -4.23184028477408e-05,
+ -0.000786591088399291,
+ -1.3708974620385561e-05,
+ -1.4630959033966064,
+ -0.9440104961395264,
+ -0.02034369483590126,
+ -2.5152843591058627e-05,
+ -0.00028463127091526985,
+ -0.00020847532141488045,
-1.6212332411669195e-05,
- -6.006290435791016,
- -0.23482508957386017,
- -0.0003094194398727268,
- -3.2066785934148356e-05,
- -0.2894707918167114,
- -0.00010334911348763853,
- -0.003178308717906475,
- -4.8397800128441304e-05,
- -3.3541419506073,
- -5.274465084075928,
- -2.3055055141448975,
- -1.0987294912338257,
- -0.019666209816932678,
- -0.00022790218645241112,
- -0.016233760863542557,
- -0.2816391885280609,
- -0.028503969311714172,
- -0.0001358893496217206,
- -0.00010394509445177391,
- -8.856858039507642e-05,
- -0.00036137725692242384,
- -0.00029452278977259994,
- -0.0008922410197556019,
- -2.539125671319198e-05,
- -0.0011102947173640132,
- -3.40932747349143e-05,
- -0.0004843492351938039,
- -0.006350101437419653,
- -5.9602869441732764e-05,
- -0.0002796259068418294,
- -0.3986394703388214,
- -0.10029242187738419,
+ -6.000129699707031,
+ -0.2337796688079834,
-0.00024196557933464646,
- -1.9691603183746338,
- -0.7402586936950684,
- -7.056941103655845e-05,
- -0.0003618539194576442,
- -1.0371154530730564e-05,
- -1.4170231819152832,
- -0.008172051049768925,
- -1.3708974620385561e-05,
- -0.00041607304592616856,
- -0.00014888131408952177,
- -2.6464111215318553e-05,
- -0.018121162429451942,
- -0.00010764019680209458,
- -0.0002335037279408425,
- -2.3007127310847864e-05,
- -0.002049014437943697,
- -1.0609570381348021e-05,
- -0.0011868583969771862,
- -7.867782187531702e-06,
- -0.0018794744974002242,
- -5.960462772236497e-07,
- -0.0007434703293256462,
- -0.02911354973912239,
- -1.7920753955841064,
- -0.0026135831139981747,
- -0.00308870617300272,
- -3.659658250398934e-05,
- -0.010810147039592266,
- -0.20098412036895752,
- -0.01644638366997242,
- -0.00013207517622504383,
- -6.854299135738984e-05,
- -7.152301259338856e-05,
- -0.00024720950750634074,
- -0.00033468366018496454,
- -0.0010001424234360456,
- -5.054346183896996e-05,
- -0.0009557208395563066,
- -3.981510963058099e-05,
- -0.0004465774691198021,
- -0.011578621342778206,
- -7.211902266135439e-05,
+ -3.2305197237292305e-05,
+ -0.34043580293655396,
+ -0.00010227633902104571,
+ -0.003176526166498661,
+ -4.124556289752945e-05,
+ -3.3743977546691895,
+ -5.277707099914551,
+ -2.324122428894043,
+ -1.1028555631637573,
+ -0.018945103511214256,
+ -0.00020013237372040749,
+ -0.01631820574402809,
+ -0.2806614339351654,
+ -0.02678731456398964,
+ -0.00013696208770852536,
+ -0.0001070442158379592,
+ -8.999896090244874e-05,
+ -0.00036113892565481365,
+ -0.00037353215157054365,
+ -0.0008961713756434619,
+ -2.6225699912174605e-05,
+ -0.0011189873330295086,
+ -3.349725011503324e-05,
+ -0.000409161759307608,
+ -0.005737621337175369,
+ -6.151010165922344e-05,
+ -0.00027509720530360937,
+ -0.4400036633014679,
+ -0.09854183346033096,
-0.0002416080387774855,
- -0.09539440274238586,
- -0.057392168790102005,
+ -1.9590201377868652,
+ -0.7413398027420044,
+ -6.961580220377073e-05,
+ -0.0003890234511345625,
+ -1.0132738680113107e-05,
+ -1.3245630264282227,
+ -0.007269597612321377,
+ -1.3470558769768104e-05,
+ -0.0003660247311927378,
+ -0.00014959646796341985,
+ -2.4914430468925275e-05,
+ -0.016247248277068138,
+ -0.00010752100206445903,
+ -0.00029118589009158313,
+ -2.3007127310847864e-05,
+ -0.0019515770254656672,
+ -1.0132738680113107e-05,
+ -0.0011845960980281234,
+ -7.510157047363464e-06,
+ -0.0018861376447603106,
+ -4.768370445162873e-07,
+ -0.0007256020326167345,
+ -0.03265838325023651,
+ -1.8349244594573975,
+ -0.002593251410871744,
+ -0.0031409950461238623,
+ -3.85038583772257e-05,
+ -0.0108514204621315,
+ -0.20090337097644806,
+ -0.014486847445368767,
+ -0.00010358751023886725,
+ -7.176141662057489e-05,
+ -7.199982064776123e-05,
+ -0.00023040501400828362,
+ -0.00031275625224225223,
+ -0.0008927173912525177,
+ -4.1126360883936286e-05,
+ -0.000972036796156317,
+ -4.124556289752945e-05,
+ -0.00044907975825481117,
+ -0.01303955726325512,
+ -7.521823135903105e-05,
+ -0.00024196557933464646,
+ -0.09538453817367554,
+ -0.05263691768050194,
-0.0002840353990904987,
- -0.21088920533657074,
- -0.0078902468085289,
- -8.606540359323844e-05,
- -0.0007384672062471509,
- -1.3589766240329482e-05,
- -0.8148440718650818,
- -0.025661379098892212,
- -2.113894462585449,
- -0.01820814050734043,
- -0.0010720703285187483,
- -0.0002908283786382526,
- -0.00011181206355104223,
+ -0.19917485117912292,
+ -0.007841872982680798,
+ -8.546940807718784e-05,
+ -0.0007601470570079982,
+ -1.3232143828645349e-05,
+ -0.8271036148071289,
+ -0.024859817698597908,
+ -2.175525665283203,
+ -0.018430527299642563,
+ -0.0011276798322796822,
+ -0.00028713393840007484,
+ -0.00011169286881340668,
-1.9550132492440753e-05,
- -1.9963352680206299,
- -0.011685965582728386,
- -0.00010299152199877426,
- -1.6093124941107817e-05,
- -0.3427979350090027,
- -0.00010358751023886725,
- -0.002419165801256895,
- -5.07818695041351e-05,
- -9.356146812438965,
- -2.63590145111084,
- -0.0489899143576622,
- -0.429649293422699,
- -2.441277027130127,
- -0.09116854518651962,
- -1.7202471494674683,
- -1.2776923179626465,
- -1.2828468084335327,
- -0.1033272072672844,
- -0.013413426466286182,
- -0.00016091958968900144,
- -0.006314327474683523,
- -0.1650361269712448,
- -0.009155434556305408,
- -8.630380034446716e-05,
- -6.007967749610543e-05,
- -6.210611172718927e-05,
- -0.00027497802511788905,
- -0.0005628670332953334,
- -0.0008046964649111032,
- -4.160317621426657e-05,
- -0.0009633429581299424,
- -2.9444261599564925e-05,
- -0.0003147821989841759,
- -0.003070523263886571,
- -3.969590397900902e-05,
- -0.00025340684805996716,
- -0.16765674948692322,
- -0.220333993434906,
- -0.00025281094713136554,
- -1.6686129570007324,
- -0.08651255071163177,
- -7.4741430580616e-05,
- -0.00032062159152701497,
- -9.536697689327411e-06,
- -8.607754707336426,
- -2.7989468574523926,
- -0.006830438040196896,
- -0.00042500998824834824,
- -4.410646579344757e-05,
- -2.2325727939605713,
- -0.09642884135246277,
- -0.0005049622268415987,
- -1.4662635294371285e-05,
- -3.892613172531128,
- -0.0008376903715543449,
- -0.004279621876776218,
- -5.745722592109814e-05,
- -2.696786642074585,
- -0.44925373792648315,
- -0.37875908613204956,
- -0.27114248275756836,
- -1.023728609085083,
- -4.712882995605469,
- -1.415423035621643,
- -2.8054561614990234,
- -0.4460236430168152,
- -0.0005779979983344674,
- -0.02468189038336277,
- -0.30965328216552734,
- -0.02052520029246807,
- -0.00012730741582345217,
- -9.619726915843785e-05,
- -8.749579137656838e-05,
- -0.000350175570929423,
- -0.0003150205302517861,
- -0.0007310817018151283,
- -3.0636318115284666e-05,
- -0.0011643542675301433,
+ -1.969758152961731,
+ -0.010251722298562527,
+ -0.0001045410826918669,
+ -1.2874520507466514e-05,
+ -0.3413291573524475,
+ -0.00010489867418073118,
+ -0.0024227334652096033,
+ -5.61460001335945e-05,
+ -9.366641998291016,
+ -2.6359500885009766,
+ -0.05483028292655945,
+ -0.4293377697467804,
+ -2.4410433769226074,
+ -0.09051975607872009,
+ -1.7625658512115479,
+ -1.2852275371551514,
+ -1.2704421281814575,
+ -0.09968416392803192,
+ -0.011455003172159195,
+ -0.0001662831346038729,
+ -0.006489157676696777,
+ -0.14835502207279205,
+ -0.010291839018464088,
+ -8.654219709569588e-05,
+ -5.9960475482512265e-05,
+ -6.282132380874828e-05,
+ -0.0002961912250611931,
+ -0.0004487222759053111,
+ -0.0008200620068237185,
-3.2305197237292305e-05,
- -0.00026913834153674543,
- -0.011463016271591187,
- -5.411955135059543e-05,
- -0.00023231192608363926,
- -0.1063343733549118,
- -0.037034809589385986,
- -0.0001248043408850208,
- -0.3663400411605835,
- -0.01425135973840952,
- -5.376194530981593e-05,
- -0.000933926145080477,
- -1.4305012882687151e-05,
- -1.5244930982589722,
- -0.008558854460716248,
- -1.8358061424805783e-05,
- -0.0002698534226510674,
- -0.00022075122979003936,
+ -0.0008834273321554065,
+ -2.9205850296420977e-05,
+ -0.00031537804170511663,
+ -0.0030801494140177965,
+ -4.0649541915627196e-05,
+ -0.0002461368858348578,
+ -0.16866101324558258,
+ -0.2007424682378769,
+ -0.0002817710628733039,
+ -1.7346234321594238,
+ -0.09638641029596329,
+ -7.486063259420916e-05,
+ -0.0003212174342479557,
+ -9.417489309271332e-06,
+ -8.656828880310059,
+ -2.7857470512390137,
+ -0.007480112835764885,
+ -0.0005174727994017303,
+ -4.4225667807040736e-05,
+ -2.242187976837158,
+ -0.09535398334264755,
+ -0.0005254556890577078,
+ -1.4662635294371285e-05,
+ -3.884834051132202,
+ -0.00082923355512321,
+ -0.004454570822417736,
+ -5.781483559985645e-05,
+ -2.578787326812744,
+ -0.4335319995880127,
+ -0.3486354649066925,
+ -0.30149775743484497,
+ -1.103930950164795,
+ -4.709151268005371,
+ -1.359910249710083,
+ -2.805453300476074,
+ -0.40311169624328613,
+ -0.0005312938592396677,
+ -0.023901192471385002,
+ -0.31066638231277466,
+ -0.02003527246415615,
+ -0.00013159839727450162,
+ -9.7508447652217e-05,
+ -9.297892393078655e-05,
+ -0.0003295593778602779,
+ -0.0003240775258745998,
+ -0.0007370378007180989,
+ -2.9801878554280847e-05,
+ -0.0011587579501792789,
+ -3.158996332786046e-05,
+ -0.00026603974401950836,
+ -0.011517226696014404,
+ -5.3165931603871286e-05,
+ -0.00021288513380568475,
+ -0.11887810379266739,
+ -0.03703492507338524,
+ -0.00013326710904948413,
+ -0.33607202768325806,
+ -0.014220921322703362,
+ -5.364274329622276e-05,
+ -0.000709401210770011,
+ -1.5735502529423684e-05,
+ -1.5350909233093262,
+ -0.008527413941919804,
+ -1.5020257706055418e-05,
+ -0.00027247529942542315,
+ -0.0002302858338225633,
-3.576214658096433e-05,
- -0.01590365171432495,
- -0.00012706902634818107,
- -0.0002901133266277611,
- -2.2649508537142538e-05,
- -0.0032194233499467373,
- -1.1920858014491387e-05,
- -0.0013312773080542684,
- -8.22540732769994e-06,
- -0.001732040662318468,
+ -0.015459935180842876,
+ -0.00011562632425921038,
+ -0.00030489088385365903,
+ -1.8596476365928538e-05,
+ -0.0034538398031145334,
+ -9.417489309271332e-06,
+ -0.0013297295663505793,
+ -8.583032467868179e-06,
+ -0.0016905077500268817,
-4.768370445162873e-07,
- -0.0007115454645827413,
- -0.11607333272695541,
- -5.158000946044922,
- -0.00630958890542388,
- -0.006455875933170319,
- -3.886147169396281e-05,
- -0.007113605737686157,
- -0.16176439821720123,
- -0.01025608740746975,
- -9.321732068201527e-05,
- -5.435795901576057e-05,
- -7.70062324590981e-05,
- -0.0002002515539061278,
- -0.0003270567976869643,
- -0.0011002921964973211,
- -3.93382906622719e-05,
- -0.0009735850035212934,
+ -0.0007995745982043445,
+ -0.10312259942293167,
+ -5.149272918701172,
+ -0.0055675203911960125,
+ -0.007235039956867695,
+ -3.8980677345534787e-05,
+ -0.007169353775680065,
+ -0.17916721105575562,
+ -0.011401025578379631,
+ -9.035655966727063e-05,
+ -5.447716102935374e-05,
+ -7.593343616463244e-05,
+ -0.0002094287920044735,
+ -0.00034814971149899065,
+ -0.001026103738695383,
+ -4.2676016164477915e-05,
+ -0.0008634176338091493,
-4.076874756719917e-05,
- -0.00036042393185198307,
- -0.011448992416262627,
- -0.00010787858627736568,
- -0.00022289653134066612,
- -0.12719827890396118,
- -0.16689445078372955,
- -0.00029869386344216764,
- -1.129071831703186,
- -0.46998509764671326,
- -0.0001429217227268964,
- -0.0004334702098276466,
- -1.823885577323381e-05,
- -7.808990478515625,
- -0.6958405375480652,
- -0.0011538759572431445,
- -0.00010084597306558862,
+ -0.0003505330823827535,
+ -0.01161043718457222,
+ -0.00011455356434453279,
+ -0.0002146728802472353,
+ -0.14297251403331757,
+ -0.1774844229221344,
+ -0.00026425207033753395,
+ -1.2013825178146362,
+ -0.43942946195602417,
+ -0.00014280252798926085,
+ -0.00040356122190132737,
+ -1.8358061424805783e-05,
+ -7.889094352722168,
+ -0.7018532752990723,
+ -0.001150541938841343,
+ -9.965400386136025e-05,
-2.1815061700181104e-05,
- -3.412889242172241,
- -0.0024302254896610975,
- -0.1256120651960373,
- -0.0001486429391661659,
- -2.932505594799295e-05,
- -0.016119161620736122,
- -2.1219027985353023e-05,
- -0.0014936492079868913,
- -6.794906312279636e-06,
- -4.649867057800293,
- -0.42487168312072754,
- -1.3419163227081299,
- -0.3015914857387543,
- -0.00015341058315243572,
- -0.0032649326603859663,
- -0.11564143747091293,
- -0.00739337969571352,
- -5.8887653722194955e-05,
- -6.615896563744172e-05,
- -5.972207145532593e-05,
- -0.00020644917094614357,
- -0.000301673193462193,
- -0.0003761537664104253,
- -2.6702524337451905e-05,
- -0.0008094609947875142,
+ -3.3287391662597656,
+ -0.0022034193389117718,
+ -0.12544699013233185,
+ -0.00012003655137959868,
+ -2.90866428258596e-05,
+ -0.016096051782369614,
+ -2.2291887944447808e-05,
+ -0.0011514945654198527,
+ -6.437280717364047e-06,
+ -4.649921417236328,
+ -0.39337143301963806,
+ -1.3441730737686157,
+ -0.2537326514720917,
+ -0.00013386306818574667,
+ -0.0037233568727970123,
+ -0.12940432131290436,
+ -0.00740450294688344,
+ -5.9602869441732764e-05,
+ -5.829164365422912e-05,
+ -6.031808152329177e-05,
+ -0.00021002470748499036,
+ -0.0003022690652869642,
+ -0.00037698791129514575,
+ -2.586808113846928e-05,
+ -0.0008114859228953719,
-3.2305197237292305e-05,
- -0.0002474478678777814,
- -0.018454870209097862,
- -7.73638384998776e-05,
- -0.00022837892174720764,
- -0.04869883507490158,
- -0.02372216247022152,
- -0.0002051381452474743,
- -0.15266406536102295,
- -0.0037327392492443323,
+ -0.0002454218047205359,
+ -0.020887058228254318,
+ -7.879423355916515e-05,
+ -0.000226472009671852,
+ -0.043111126869916916,
+ -0.02126489207148552,
+ -0.0002022777043748647,
+ -0.15768083930015564,
+ -0.0037531666457653046,
-7.557583012385294e-05,
- -0.0005665604257956147,
+ -0.0005638201837427914,
+ -1.6093124941107817e-05,
+ -2.1179733276367188,
+ -0.025822287425398827,
+ -1.996516227722168,
+ -2.5962047576904297,
+ -0.14906328916549683,
+ -0.006481340620666742,
+ -0.00016091958968900144,
+ -1.7523612768854946e-05,
+ -0.8894410729408264,
+ -0.000976800569333136,
+ -0.0003759154351428151,
+ -1.2993727978027891e-05,
+ -0.07106750458478928,
+ -5.4238757002167404e-05,
+ -0.0015886317705735564,
+ -9.65590606938349e-06,
+ -4.332467079162598,
+ -6.821710586547852,
+ -3.370997428894043,
+ -1.8514046669006348,
+ -0.0004586121649481356,
+ -0.046185240149497986,
+ -0.06824732571840286,
+ -0.005082899704575539,
+ -9.059495641849935e-05,
+ -6.675497570540756e-05,
+ -8.630380034446716e-05,
+ -0.00025293012731708586,
+ -0.0003567297535482794,
+ -0.000636255950666964,
+ -3.123234637314454e-05,
+ -0.0012261499650776386,
+ -3.1470757676288486e-05,
+ -0.0002165798214264214,
+ -0.014394257217645645,
+ -0.00020966715237591416,
+ -0.00016699827392585576,
+ -7.259582343976945e-05,
+ -0.3280075490474701,
+ -0.00015531764074694365,
+ -2.5446929931640625,
+ -0.16115832328796387,
+ -0.0001174142598756589,
+ -0.0006781900301575661,
+ -2.2172682292875834e-05,
+ -1.6608847379684448,
+ -0.011715774424374104,
+ -1.9192511899746023e-05,
+ -0.0003228858404327184,
+ -0.00020346954988781363,
+ -3.111314072157256e-05,
+ -0.01600196771323681,
+ -0.00012587709352374077,
+ -0.00044741155579686165,
+ -2.1457441107486375e-05,
+ -0.007055368740111589,
-1.4662635294371285e-05,
- -2.1065256595611572,
- -0.02570541389286518,
- -2.0099081993103027,
- -2.7118430137634277,
- -0.1484161764383316,
- -0.007964756339788437,
- -0.00016342257731594145,
- -1.597391747054644e-05,
- -0.8920754194259644,
- -0.0009690594743005931,
+ -0.0012477001873776317,
+ -1.2278481335670222e-05,
+ -0.005830660928040743,
+ -4.768370445162873e-07,
+ -0.0016891986597329378,
+ -0.13408850133419037,
+ -3.2587075233459473,
+ -0.0026285641361027956,
+ -0.005026086233556271,
+ -3.516612196108326e-05,
+ -0.012018292210996151,
+ -0.18297713994979858,
+ -0.01304532214999199,
+ -0.00011467275908216834,
+ -8.5588610090781e-05,
+ -8.511180931236595e-05,
+ -0.00046957432641647756,
+ -0.00026556302327662706,
+ -0.0013249675976112485,
+ -9.023735765367746e-05,
+ -0.001595415873453021,
+ -5.4834770708112046e-05,
+ -0.0007428746903315187,
+ -0.6937626004219055,
+ -0.0007443041540682316,
-0.00029023250681348145,
- -1.2993727978027891e-05,
- -0.07993864268064499,
- -5.400034933700226e-05,
- -0.00158791767898947,
- -1.0609570381348021e-05,
- -4.331461429595947,
- -6.81968355178833,
- -3.366002082824707,
- -1.850673794746399,
- -0.00040391870425082743,
- -0.04611193388700485,
- -0.06791424006223679,
- -0.004945189692080021,
- -9.107174992095679e-05,
- -7.557583012385294e-05,
- -6.747018051100895e-05,
- -0.00024399164249189198,
- -0.000321336614433676,
- -0.0006528153317049146,
+ -0.00800082553178072,
+ -0.03613343834877014,
+ -0.0001821352052502334,
+ -0.08064392954111099,
+ -0.005144097842276096,
+ -0.00015829740732442588,
+ -0.002411079127341509,
+ -2.7179348762729205e-05,
+ -1.7293376922607422,
+ -0.7512492537498474,
+ -0.019934315234422684,
+ -1.7762025890988298e-05,
+ -0.0002302858338225633,
+ -0.00020954797219019383,
+ -2.706014311115723e-05,
+ -3.64949893951416,
+ -0.2676098644733429,
+ -9.60780744208023e-05,
+ -6.007967749610543e-05,
+ -0.13092315196990967,
+ -2.2291887944447808e-05,
+ -0.0036994845140725374,
-3.2782016205601394e-05,
- -0.0012151960982009768,
- -3.957670196541585e-05,
- -0.0002205128694185987,
- -0.016214992851018906,
- -0.00019095504831057042,
- -0.0001456631434848532,
- -7.712543447269127e-05,
- -0.33043625950813293,
- -0.00017629499780014157,
- -2.590480089187622,
- -0.16181793808937073,
- -0.00011646069469861686,
- -0.0006735440110787749,
- -2.109982233378105e-05,
- -1.6486821174621582,
- -0.01151864044368267,
- -1.8954096958623268e-05,
- -0.0003233625029679388,
- -0.00020644917094614357,
+ -2.8737730979919434,
+ -1.1481257677078247,
+ -2.7093358039855957,
+ -5.347209930419922,
+ -0.8316395878791809,
+ -0.00016532962035853416,
+ -0.8304288983345032,
+ -5.002294540405273,
+ -0.12272598594427109,
+ -0.008114584721624851,
+ -0.00037853704998269677,
+ -0.022495213896036148,
+ -0.5160256624221802,
+ -0.04559532180428505,
+ -0.0002172949316445738,
+ -0.00020132421923335642,
+ -0.00014661667228210717,
+ -0.0006960591417737305,
+ -0.0006668727728538215,
+ -0.0014327033422887325,
+ -6.615896563744172e-05,
+ -0.002525119110941887,
+ -4.5060096454108134e-05,
+ -0.0006106419023126364,
+ -0.000756216119043529,
+ -0.004223950207233429,
+ -0.0004472924047149718,
+ -0.1524876207113266,
+ -0.07079865783452988,
+ -0.00015948931104503572,
+ -2.295717239379883,
+ -0.44691845774650574,
+ -0.0001967951684491709,
+ -0.0006033745594322681,
+ -1.6927575416048057e-05,
+ -2.321779727935791,
+ -0.02190409228205681,
+ -0.030846793204545975,
+ -1.1255987882614136,
+ -0.0002225389762315899,
+ -0.00011848701251437888,
-3.111314072157256e-05,
- -0.017416512593626976,
- -0.00012766500003635883,
- -0.0003415954706724733,
- -2.13382354559144e-05,
- -0.006446637213230133,
- -1.823885577323381e-05,
- -0.0012438902631402016,
- -1.1205610462639015e-05,
- -0.006591127719730139,
- -7.152555099310121e-07,
- -0.0017049076268449426,
- -0.13135236501693726,
- -3.228759288787842,
- -0.002643782878294587,
- -0.004842340014874935,
- -3.480850500636734e-05,
- -0.010503842495381832,
- -0.16338221728801727,
- -0.011769498698413372,
- -0.00011574551899684593,
- -9.727005090098828e-05,
- -8.582700684200972e-05,
- -0.0004538459761533886,
- -0.00020740265608765185,
- -0.001342587056569755,
+ -3.019529342651367,
+ -0.005704077892005444,
+ -0.0004024887748528272,
+ -1.7881233361549675e-05,
+ -0.04252830147743225,
+ -0.00015853578224778175,
+ -0.005454538390040398,
+ -3.2543604902457446e-05,
+ -4.886053085327148,
+ -3.2839345932006836,
+ -0.17567910254001617,
+ -8.478743553161621,
+ -0.2580631673336029,
+ -0.00016127715934999287,
+ -0.004872709512710571,
+ -0.18758729100227356,
+ -0.024999113753437996,
+ -0.00015233787416946143,
+ -0.00012659224739763886,
-8.964136941358447e-05,
- -0.0014018717920407653,
- -4.935142715112306e-05,
- -0.0006431656656786799,
- -0.5765135288238525,
- -0.0009291622554883361,
- -0.00027998341829515994,
- -0.008964410983026028,
- -0.03303813934326172,
- -0.00018451895448379219,
- -0.07687719166278839,
- -0.00454594986513257,
- -0.00018439977429807186,
- -0.0023830130230635405,
- -2.706014311115723e-05,
- -1.8103313446044922,
- -0.7522969245910645,
- -0.022507335990667343,
- -2.074220174108632e-05,
- -0.00026222606538794935,
- -0.00020740265608765185,
- -2.706014311115723e-05,
- -3.700786590576172,
- -0.26737019419670105,
- -9.357491217087954e-05,
- -6.031808152329177e-05,
- -0.13705354928970337,
- -2.407998726994265e-05,
- -0.003684044349938631,
- -3.2782016205601394e-05,
- -2.9476141929626465,
- -1.1526018381118774,
- -2.6757259368896484,
- -5.31315279006958,
- -0.7695194482803345,
- -0.00014876213390380144,
- -0.8328413963317871,
- -5.100983142852783,
- -0.1275785118341446,
- -0.008235306479036808,
- -0.00037281715776771307,
- -0.02394961006939411,
- -0.5179875493049622,
- -0.04619366303086281,
- -0.00021705655672121793,
- -0.00021765247220173478,
- -0.0001461399078834802,
- -0.0007413261337205768,
- -0.0006660388899035752,
- -0.0015581621555611491,
- -6.8662193370983e-05,
- -0.002233869396150112,
- -4.494089080253616e-05,
- -0.0006101653561927378,
- -0.0006289887824095786,
- -0.0033358661457896233,
- -0.00045074793160893023,
- -0.15180595219135284,
- -0.07985830307006836,
- -0.00015937011630740017,
- -2.2477855682373047,
- -0.4471043348312378,
- -0.0001734344696160406,
- -0.0006040894077159464,
+ -0.0002896366349887103,
+ -0.0004929280839860439,
+ -0.0007099968497641385,
+ -5.5549986427649856e-05,
+ -0.0008168459753505886,
+ -2.9444261599564925e-05,
+ -0.0001512651506345719,
+ -0.00014768941036891192,
+ -0.0022286358289420605,
+ -0.00021073981770314276,
+ -0.11390721797943115,
+ -0.059136006981134415,
+ -0.00010716341057559475,
+ -0.403459757566452,
+ -0.01966036483645439,
+ -5.900685573578812e-05,
+ -0.0006700892699882388,
-1.680836794548668e-05,
- -2.318458080291748,
- -0.01888836920261383,
- -0.029085876420140266,
- -1.1253407001495361,
- -0.00021741411183029413,
- -0.00012003655137959868,
- -2.8013790142722428e-05,
- -3.1507949829101562,
- -0.005721264518797398,
- -0.00040904260822571814,
- -1.7881233361549675e-05,
- -0.04304421693086624,
- -0.0001591317413840443,
- -0.005429995711892843,
- -3.242440288886428e-05,
- -4.896542549133301,
- -3.2877321243286133,
- -0.17550288140773773,
- -8.526089668273926,
- -0.2559642493724823,
- -0.00015770144818816334,
- -0.004955509677529335,
- -0.20714037120342255,
- -0.023553114384412766,
- -0.00015496007108595222,
- -0.0001134808044298552,
- -9.250213042832911e-05,
- -0.000288087350782007,
- -0.0004409771354403347,
- -0.0007110689766705036,
- -4.6132929128361866e-05,
- -0.0009153467253781855,
- -3.433168603805825e-05,
- -0.00015484087634831667,
- -0.0001292145170737058,
- -0.0022287548054009676,
- -0.0002269487304147333,
- -0.11395295709371567,
- -0.05913611873984337,
- -8.356221951544285e-05,
- -0.4039720594882965,
- -0.019538793712854385,
- -5.924526340095326e-05,
- -0.0007176207727752626,
- -1.7881233361549675e-05,
- -1.6992816925048828,
- -0.004352619871497154,
- -6.6756979322235566e-06,
- -0.00017093151109293103,
- -0.0001284993631998077,
+ -1.6097873449325562,
+ -0.004351789131760597,
+ -6.318072337307967e-06,
+ -0.00017438798386137933,
+ -0.00012814177898690104,
-3.3378044463461265e-05,
- -0.013412484899163246,
- -8.713819261174649e-05,
- -0.0004928089329041541,
- -2.288792165927589e-05,
- -0.0012643685331568122,
+ -0.011513691395521164,
+ -8.618460560683161e-05,
+ -0.0004935238393954933,
+ -2.3007127310847864e-05,
+ -0.0012561534531414509,
-1.3351351299206726e-05,
- -0.0019104102393612266,
- -8.940656698541716e-06,
- -0.0033124599140137434,
+ -0.002182603348046541,
+ -9.059865078597795e-06,
+ -0.0031579884234815836,
-4.768370445162873e-07,
- -0.0009848987683653831,
- -0.07256874442100525,
- -1.7665941715240479,
- -0.00281461956910789,
- -0.0027610058896243572,
- -2.9682672902708873e-05,
- -0.0075036585330963135,
- -0.16648568212985992,
- -0.014109030365943909,
- -9.63164638960734e-05,
+ -0.0009639384225010872,
+ -0.07261596620082855,
+ -1.7404682636260986,
+ -0.002495629247277975,
+ -0.0031125929672271013,
+ -3.71926071238704e-05,
+ -0.006931781768798828,
+ -0.14954042434692383,
+ -0.014099039137363434,
+ -9.881961887003854e-05,
-6.603976362384856e-05,
- -7.331102824537084e-05,
- -0.0003323002893012017,
- -0.00042083943844772875,
- -0.0010620674584060907,
- -2.8609820219571702e-05,
- -0.000990257947705686,
- -4.029192859889008e-05,
- -0.0001541257370263338,
- -0.0001658063702052459,
- -0.0010433712741360068,
- -0.0002379134384682402,
- -0.08282912522554398,
- -0.1620505303144455,
+ -8.761498611420393e-05,
+ -0.00033396866638213396,
+ -0.0003670972364488989,
+ -0.0010602811817079782,
+ -3.504691630951129e-05,
+ -0.000943096645642072,
+ -3.635817120084539e-05,
+ -0.00014602071314584464,
+ -0.00018845213344320655,
+ -0.0012003129813820124,
+ -0.00024256148026324809,
+ -0.05201944336295128,
+ -0.14071618020534515,
-0.0001578206429257989,
- -1.9873682260513306,
- -0.03700195625424385,
- -8.594620157964528e-05,
- -0.00035232058144174516,
+ -2.140291213989258,
+ -0.04168115183711052,
+ -8.535020606359467e-05,
+ -0.0003649522550404072,
+ -2.253030106658116e-05,
+ -1.0512301921844482,
+ -0.012040670961141586,
+ -1.8452234268188477,
+ -0.047459427267313004,
+ -0.002904722234234214,
+ -0.0002766464895103127,
+ -7.939023635117337e-05,
+ -2.6225699912174605e-05,
+ -4.23543119430542,
+ -0.00821508839726448,
+ -7.4741430580616e-05,
+ -6.305972783593461e-05,
+ -5.399444580078125,
+ -0.0015713736647740006,
+ -0.0018196230521425605,
-2.90866428258596e-05,
- -1.0645859241485596,
- -0.012771833688020706,
- -1.8788448572158813,
- -0.04745874181389809,
- -0.0029150634072721004,
- -0.0002858230145648122,
- -8.082063141046092e-05,
- -2.8729025871143676e-05,
- -4.2793378829956055,
- -0.008196880109608173,
- -9.822363062994555e-05,
- -4.9470632802695036e-05,
- -5.399019241333008,
- -0.0015862513100728393,
- -0.0018035589018836617,
- -2.9444261599564925e-05,
- -3.8089842796325684,
- -1.3950530290603638,
- -0.17507919669151306,
- -4.1786346435546875,
- -9.410017013549805,
- -0.00014709345123264939,
- -2.16685152053833,
- -0.5008745193481445,
- -0.013433892279863358,
- -0.00029976642690598965,
- -0.006172403693199158,
- -0.22438427805900574,
- -0.015963135287165642,
- -0.00010489867418073118,
+ -3.806305170059204,
+ -1.2674520015716553,
+ -0.17437320947647095,
+ -4.168057441711426,
+ -9.435733795166016,
+ -0.00014697425649501383,
+ -2.166922092437744,
+ -0.5080004930496216,
+ -0.01319874543696642,
+ -0.0003053675754927099,
+ -0.006557967979460955,
+ -0.201018825173378,
+ -0.015605359338223934,
+ -0.00010835537250386551,
-7.426462980220094e-05,
- -6.890059739816934e-05,
- -0.0002874914789572358,
- -0.0004033228906337172,
- -0.0006624649395234883,
- -3.802703940891661e-05,
- -0.001104817260056734,
- -2.8967437174287625e-05,
- -0.000125281119835563,
- -0.00011634149996098131,
- -0.0016071987338364124,
- -0.0001752223033690825,
- -0.04927569255232811,
- -0.03999283164739609,
- -8.427741704508662e-05,
- -0.11036300659179688,
- -0.0022922686766833067,
- -5.125868119648658e-05,
- -0.0007711059297434986,
- -1.6569954823353328e-05,
- -1.1996040344238281,
- -6.017496585845947,
- -3.3771719932556152,
- -0.0015197168104350567,
- -0.0001720042055239901,
- -8.05822346592322e-05,
- -1.9701510667800903,
- -0.015215284191071987,
- -0.00046957432641647756,
- -4.5536911784438416e-05,
- -0.3501690626144409,
- -6.508615479106084e-05,
- -0.013412720523774624,
- -0.0002317160106031224,
- -10.721491813659668,
- -0.001794158248230815,
- -5.900764465332031,
- -0.05698608234524727,
- -1.9666205644607544,
- -0.34450024366378784,
- -0.24932177364826202,
- -1.1890842914581299,
- -0.9316995143890381,
- -0.5700393915176392,
- -0.18522746860980988,
- -0.08411185443401337,
- -0.00032610344351269305,
- -0.016760369762778282,
- -0.310769647359848,
- -0.04111167788505554,
- -0.00015889335190877318,
- -0.00011395759065635502,
- -0.00010418349120300263,
- -0.0003389737685211003,
- -0.0006182666402310133,
- -0.001039679627865553,
- -6.770858453819528e-05,
- -0.001258891774341464,
- -5.876845170860179e-05,
- -0.0003499372396618128,
- -0.00027724236133508384,
- -0.0029526231810450554,
- -0.0003165697562508285,
- -0.25983527302742004,
- -0.031029406934976578,
- -0.00018880968855228275,
- -0.7229459881782532,
- -0.42579957842826843,
- -0.00011705666838679463,
- -0.00047195740626193583,
- -2.3364747903542593e-05,
- -0.9790778160095215,
- -0.0029993331991136074,
- -5.125986263010418e-06,
- -0.00018690270371735096,
- -0.00016091958968900144,
- -3.755022044060752e-05,
- -0.00900670699775219,
- -8.642300235806033e-05,
- -0.0004804172203876078,
- -3.838465272565372e-05,
- -0.0015756584471091628,
- -1.168244216387393e-05,
- -0.001709667849354446,
- -1.0013530300057027e-05,
- -0.0022142434027045965,
- -5.960462772236497e-07,
- -0.0006964165368117392,
- -0.05425402522087097,
- -1.5528278350830078,
- -0.002721655648201704,
- -0.003402280155569315,
- -3.6477376852417365e-05,
- -0.007222968153655529,
- -0.14785511791706085,
- -0.013813492842018604,
- -0.00012063252506777644,
- -9.738924563862383e-05,
- -9.881961887003854e-05,
- -0.00025900822947733104,
- -0.00028236693469807506,
- -0.0010882653295993805,
- -4.446407547220588e-05,
- -0.0008232779800891876,
- -4.7801782784517854e-05,
- -0.0001911934232339263,
- -0.00020382710499688983,
- -0.0037347583565860987,
- -0.00023493390472140163,
- -0.016995148733258247,
- -0.028428077697753906,
- -0.00015054999676067382,
- -0.05958176776766777,
- -0.0022499265614897013,
- -8.928377064876258e-05,
- -0.0007566926069557667,
- -2.038458114839159e-05,
- -6.74626350402832,
- -4.031385898590088,
- -0.010314728133380413,
- -0.0005830018781125546,
+ -7.10462118149735e-05,
+ -0.0002862997353076935,
+ -0.0004632591735571623,
+ -0.0006872437661513686,
+ -3.6238969187252223e-05,
+ -0.0012379371328279376,
+ -2.9444261599564925e-05,
+ -0.00011765264935093,
+ -0.00010358751023886725,
+ -0.002063171239569783,
+ -0.0001770101225702092,
+ -0.0564521886408329,
+ -0.04466686397790909,
+ -8.237022848334163e-05,
+ -0.10021531581878662,
+ -0.0023065409623086452,
+ -5.125868119648658e-05,
+ -0.0009729895391501486,
+ -1.3470558769768104e-05,
+ -1.2119941711425781,
+ -5.893743515014648,
+ -3.3627188205718994,
+ -0.001396633917465806,
+ -0.000169382052263245,
+ -8.248942322097719e-05,
+ -1.9596554040908813,
+ -0.016238568350672722,
+ -0.0004700509598478675,
+ -4.541770613286644e-05,
+ -0.34185099601745605,
+ -5.3165931603871286e-05,
+ -0.013339557684957981,
+ -0.00023147765023168176,
+ -10.721809387207031,
+ -0.0019459851318970323,
+ -5.87699031829834,
+ -0.057010404765605927,
+ -1.955953598022461,
+ -0.3542339503765106,
+ -0.28593409061431885,
+ -1.1921348571777344,
+ -0.9411425590515137,
+ -0.5700472593307495,
+ -0.18382300436496735,
+ -0.08142641186714172,
+ -0.000379132863599807,
+ -0.01954592578113079,
+ -0.3134835958480835,
+ -0.03992319107055664,
-0.00016175392374861985,
- -4.279521817807108e-05,
- -4.910806655883789,
- -0.3867932856082916,
- -0.00020466140995267779,
- -2.455681169521995e-05,
- -0.40993309020996094,
- -3.075552376685664e-05,
- -0.002136925933882594,
- -1.5258672647178173e-05,
- -1.4743690490722656,
- -0.466409295797348,
- -2.986236095428467,
- -0.5145793557167053,
- -0.3861558437347412,
- -0.00023648326168768108,
- -0.060666244477033615,
- -0.0004374024283606559,
- -0.0032959445379674435,
- -0.003968104254454374,
- -0.0018072477541863918,
- -4.768258077092469e-05,
- -0.9783220291137695,
- -1.0383716821670532,
- -0.6705473065376282,
- -2.172899007797241,
- -0.1931028664112091,
- -0.05653104931116104,
- -0.0004231034545227885,
- -0.009201028384268284,
- -0.20085793733596802,
- -0.015902360901236534,
- -0.00013207517622504383,
- -0.00011634149996098131,
- -9.154854342341423e-05,
- -0.0002989322238136083,
- -0.000276765669696033,
- -0.0008761619683355093,
- -5.4596363042946905e-05,
- -0.0012877037515863776,
- -5.245071224635467e-05,
- -0.00014399446081370115,
- -0.00014304091746453196,
- -0.002012848388403654,
- -0.00026043839170597494,
- -0.050352130085229874,
- -0.016213351860642433,
- -0.00014923889830242842,
- -1.3270337581634521,
- -0.017757130786776543,
- -8.725739462533966e-05,
- -0.0003123987407889217,
- -2.3364747903542593e-05,
- -1.770219087600708,
- -0.027282992377877235,
- -1.7292673587799072,
- -1.5430668592453003,
- -0.09708311408758163,
- -0.06372363120317459,
- -0.00020180096908006817,
+ -0.00011359999916749075,
+ -0.00010442188795423135,
+ -0.0003408804477658123,
+ -0.0006943913758732378,
+ -0.0010699268896132708,
+ -6.353653589030728e-05,
+ -0.0012460333527997136,
+ -5.98412734689191e-05,
+ -0.0003594706067815423,
+ -0.00027581225731410086,
+ -0.002966054016724229,
+ -0.00029202012228779495,
+ -0.28885897994041443,
+ -0.028338506817817688,
+ -0.00018892886873800308,
+ -0.7341126203536987,
+ -0.36993715167045593,
+ -0.0001081169830285944,
+ -0.0004390706308186054,
+ -2.109982233378105e-05,
+ -0.9581791162490845,
+ -0.0029332491103559732,
+ -5.245195097813848e-06,
+ -0.0001817776501411572,
+ -0.00012373158824630082,
+ -3.802703940891661e-05,
+ -0.008104533888399601,
+ -8.415821503149346e-05,
+ -0.0005129451747052372,
+ -3.71926071238704e-05,
+ -0.0015855372184887528,
+ -1.490105023549404e-05,
+ -0.0021829602774232626,
+ -9.894321920000948e-06,
+ -0.0021203910000622272,
+ -5.960462772236497e-07,
+ -0.0006930810050107539,
+ -0.05077950283885002,
+ -1.5622856616973877,
+ -0.0027032282669097185,
+ -0.003475817386060953,
+ -3.361645576660521e-05,
+ -0.007188409101217985,
+ -0.14947564899921417,
+ -0.012775599956512451,
+ -0.00011979816190432757,
+ -8.67805938469246e-05,
+ -0.00010597144137136638,
+ -0.0002585315378382802,
+ -0.0003066784702241421,
+ -0.0010879080509766936,
+ -3.6000557884108275e-05,
+ -0.0008429311565123498,
-4.756337511935271e-05,
- -6.762560844421387,
- -0.11426064372062683,
- -0.0006945105269551277,
- -5.745722592109814e-05,
- -0.23964034020900726,
- -7.080780778778717e-05,
- -0.0019281383138149977,
- -0.00011657988943625242,
- -1.6634957790374756,
- -3.133596420288086,
- -1.06369948387146,
- -0.20282019674777985,
- -0.440325528383255,
- -2.2919445037841797,
- -2.6773011684417725,
- -2.4511003494262695,
- -2.022627353668213,
- -0.7157211899757385,
- -0.00033623288618400693,
- -0.006556428037583828,
- -0.18528789281845093,
- -0.010350123979151249,
- -9.691245941212401e-05,
- -9.941560711013153e-05,
- -0.0001062098381225951,
- -0.0002244459028588608,
- -0.0003002431185450405,
- -0.0003911683743353933,
+ -0.00019810620869975537,
+ -0.0002044230350293219,
+ -0.003736421000212431,
+ -0.0002343380037928,
+ -0.016844887286424637,
+ -0.026026494801044464,
+ -0.0001515035255579278,
+ -0.057031236588954926,
+ -0.0020061859395354986,
+ -8.904537389753386e-05,
+ -0.0007616956136189401,
+ -2.0861407392658293e-05,
+ -6.719094753265381,
+ -4.047027587890625,
+ -0.009809266775846481,
+ -0.0005860995152033865,
+ -0.00017188502533826977,
+ -4.2676016164477915e-05,
+ -4.904641151428223,
+ -0.34968045353889465,
+ -0.00019298121333122253,
+ -2.4914430468925275e-05,
+ -0.38600707054138184,
-3.158996332786046e-05,
- -0.0008713977294974029,
- -4.875540980719961e-05,
- -9.083335316972807e-05,
- -0.00013422065239865333,
- -0.0032467530108988285,
- -0.0002611534437164664,
- -0.011103743687272072,
- -0.014522447250783443,
- -0.0001003691868390888,
- -0.04763209819793701,
- -0.0015930355293676257,
- -8.880697714630514e-05,
- -0.0006610354175791144,
- -2.062299427052494e-05,
- -1.4736919403076172,
+ -0.002126338891685009,
+ -1.3708974620385561e-05,
+ -1.464985966682434,
+ -0.47231751680374146,
+ -2.9898223876953125,
+ -0.5146303176879883,
+ -0.38906559348106384,
+ -0.00023552982020191848,
+ -0.06135307624936104,
+ -0.00047267231275327504,
+ -0.003303905250504613,
+ -0.0036028018221259117,
+ -0.001978822285309434,
+ -4.660974445869215e-05,
+ -0.9326105117797852,
+ -1.0439518690109253,
+ -0.6865833401679993,
+ -2.1702210903167725,
+ -0.1792261153459549,
+ -0.056661274284124374,
+ -0.00042489083716645837,
+ -0.009141850285232067,
+ -0.2000894993543625,
+ -0.015266942791640759,
+ -0.00013195598148740828,
+ -0.00011300401820335537,
+ -9.047575440490618e-05,
+ -0.0002873722987715155,
+ -0.0002636561985127628,
+ -0.0008784249657765031,
+ -5.400034933700226e-05,
+ -0.0012316268403083086,
+ -5.209310256759636e-05,
+ -0.00014161060971673578,
+ -0.00014304091746453196,
+ -0.0022716925013810396,
+ -0.00026854246971197426,
+ -0.056410498917102814,
+ -0.016143208369612694,
+ -0.0001646144810365513,
+ -1.4203705787658691,
+ -0.020097782835364342,
+ -0.00011193125828867778,
+ -0.00031132620642893016,
+ -2.312633478140924e-05,
+ -1.7828859090805054,
+ -0.0273251011967659,
+ -1.7993203401565552,
+ -1.5256458520889282,
+ -0.10319281369447708,
+ -0.06369376927614212,
+ -0.00021634146105498075,
+ -4.9470632802695036e-05,
+ -6.747133731842041,
+ -0.1273529976606369,
+ -0.0006823595031164587,
+ -5.98412734689191e-05,
+ -0.22017809748649597,
+ -7.033100700937212e-05,
+ -0.0018899451242759824,
+ -0.00011717586312443018,
+ -1.6576848030090332,
+ -3.164034605026245,
+ -1.107333779335022,
+ -0.195778489112854,
+ -0.4392439126968384,
+ -2.308759927749634,
+ -2.6903655529022217,
+ -2.4507877826690674,
+ -2.0068390369415283,
+ -0.7151349186897278,
+ -0.0003326578007545322,
+ -0.006546480115503073,
+ -0.18491095304489136,
+ -0.010410410352051258,
+ -9.60780744208023e-05,
+ -0.00010573305189609528,
+ -0.00010561384988250211,
+ -0.00022551853908225894,
+ -0.00029130507027730346,
+ -0.00038580605178140104,
+ -3.0397906812140718e-05,
+ -0.001023364719003439,
+ -4.851700214203447e-05,
+ -0.00011336160969221964,
+ -0.0001045410826918669,
+ -0.0032530506141483784,
+ -0.0002673506969586015,
+ -0.008917978964745998,
+ -0.012813379056751728,
+ -0.0001006075763143599,
+ -0.047270920127630234,
+ -0.0018081996822729707,
+ -7.64102369430475e-05,
+ -0.0006587718962691724,
+ -2.1934269170742482e-05,
+ -1.4748883247375488,
+ -0.0014756753807887435,
+ -6.318072337307967e-06,
+ -0.00015400654228869826,
+ -0.00024816294899210334,
+ -4.31528314948082e-05,
+ -0.006116599775850773,
+ -8.296622399939224e-05,
+ -0.000395815703086555,
+ -4.088794958079234e-05,
-0.0015160269103944302,
- -5.722029527532868e-06,
- -0.0001426833332516253,
- -0.00025138078490272164,
- -4.303362584323622e-05,
- -0.006412051152437925,
- -8.177422569133341e-05,
- -0.0003953390696551651,
- -4.51792984677013e-05,
- -0.0015100754098966718,
- -1.0847986231965479e-05,
- -0.0021766559220850468,
- -1.3112935448589269e-05,
- -0.0017056216020137072,
+ -1.0609570381348021e-05,
+ -0.002186052966862917,
+ -1.4066597032069694e-05,
+ -0.0016952680889517069,
-5.960462772236497e-07,
- -0.00045658653834834695,
- -0.03380563110113144,
- -1.6861530542373657,
- -0.0011235122801735997,
- -0.0027228444814682007,
- -3.2543604902457446e-05,
- -0.0028300732374191284,
- -0.04190889745950699,
- -0.006303310859948397,
- -0.00010799778101500124,
- -7.295342220459133e-05,
- -6.90197994117625e-05,
- -0.0002094287920044735,
- -0.00017915551143232733,
- -0.0007649118197150528,
- -3.3854863431770355e-05,
- -0.0009750141180120409,
- -5.185469490243122e-05,
- -0.0001230164198204875,
- -0.00015221867943182588,
- -0.00366337806917727,
- -0.00027378625236451626,
- -0.00873471051454544,
- -0.014125015586614609,
- -0.00013779645087197423,
- -0.2786974012851715,
- -0.0429004468023777,
- -0.00015221867943182588,
- -0.0005259322933852673,
- -2.0861407392658293e-05,
- -7.4979376792907715,
- -2.5812153816223145,
- -0.0006475735572166741,
- -0.00032395837479270995,
- -4.3987260141875595e-05,
- -0.38662397861480713,
- -0.07727815210819244,
- -0.0005353448214009404,
- -6.210611172718927e-05,
- -0.10053620487451553,
- -4.51792984677013e-05,
- -0.004477594513446093,
- -3.0397906812140718e-05,
- -8.758296012878418,
- -0.4402102530002594,
- -0.2472418248653412,
- -0.5627955794334412,
- -0.042171675711870193,
- -0.03491748869419098,
- -5.941390514373779,
- -0.004192491993308067,
- -0.11302625387907028,
- -0.5369495153427124,
- -0.0003328961320221424,
- -0.0049365307204425335,
- -0.057854458689689636,
- -0.007558793295174837,
- -8.916457591112703e-05,
- -9.047575440490618e-05,
- -8.141662692651153e-05,
- -0.0006507901125587523,
- -0.00019464982324279845,
- -0.0006775943911634386,
- -2.3364747903542593e-05,
- -0.0012484145117923617,
- -5.447716102935374e-05,
- -0.00016425691137555987,
- -0.00019727191829588264,
- -0.012608221732079983,
- -0.00020859450160060078,
- -0.014227267354726791,
- -0.00964115560054779,
- -0.00013350549852475524,
- -0.03465360403060913,
- -0.0008008848526515067,
- -0.00010239553375868127,
- -0.0007454953738488257,
- -2.0861407392658293e-05,
- -2.182055950164795,
- -0.030151404440402985,
- -2.2387242317199707,
- -4.8748321533203125,
- -0.07910432666540146,
- -0.0014863882679492235,
- -0.00028081765049137175,
- -6.55629628454335e-05,
- -3.332869052886963,
- -4.393488883972168,
- -0.1467350423336029,
- -0.0036104037426412106,
- -0.0003040566807612777,
- -0.00010895135346800089,
- -0.2704607844352722,
- -3.6477376852417365e-05,
- -0.002591705648228526,
- -2.9682672902708873e-05,
- -4.947231292724609,
- -3.2159130573272705,
- -0.8367561101913452,
- -0.5556290149688721,
- -0.0002233732520835474,
- -0.0060651772655546665,
- -0.05365833640098572,
- -0.0071886456571519375,
- -9.63164638960734e-05,
- -0.00010072677832795307,
- -9.858122211880982e-05,
- -0.0003960540343541652,
- -0.0006039702566340566,
- -0.0006522196927107871,
- -1.811964830267243e-05,
- -0.001042775809764862,
- -3.790783375734463e-05,
- -0.00011514954530866817,
- -0.0001652104256208986,
- -0.05494809150695801,
+ -0.0005108005134388804,
+ -0.03380759060382843,
+ -1.533830165863037,
+ -0.001141611486673355,
+ -0.0027541108429431915,
+ -2.5748875486897305e-05,
+ -0.0027864461299031973,
+ -0.04172094166278839,
+ -0.0063011785969138145,
+ -0.00010632903286023065,
+ -6.937739817658439e-05,
+ -8.725739462533966e-05,
+ -0.00020954797219019383,
+ -0.00019786784832831472,
+ -0.0008331642020493746,
+ -2.8132995794294402e-05,
+ -0.0008716359734535217,
+ -5.2569914259947836e-05,
+ -0.0001294529065489769,
-0.00014506718434859067,
- -0.00021050144277978688,
- -0.014802505262196064,
- -0.00017915551143232733,
- -1.7102066278457642,
- -0.02825750596821308,
- -0.00011300401820335537,
- -0.0003519630990922451,
- -3.075552376685664e-05,
- -0.554995596408844,
- -0.0013822296168655157,
- -4.6491513785440475e-06,
- -0.00014482879487331957,
- -0.00019810620869975537,
- -3.504691630951129e-05,
- -0.006834581959992647,
- -6.389413465512916e-05,
- -0.0004396664153318852,
- -4.60137271147687e-05,
- -0.0012897277483716607,
+ -0.00366587215103209,
+ -0.0002796259068418294,
+ -0.008858784101903439,
+ -0.014021812006831169,
+ -0.0001358893496217206,
+ -0.25631147623062134,
+ -0.04293253645300865,
+ -0.0001731960946926847,
+ -0.0004926898400299251,
+ -1.9788545614574105e-05,
+ -7.50344181060791,
+ -2.5921072959899902,
+ -0.00064876489341259,
+ -0.00038985759601928294,
+ -4.184158387943171e-05,
+ -0.34731483459472656,
+ -0.07739675790071487,
+ -0.000596107158344239,
+ -6.103329360485077e-05,
+ -0.09895050525665283,
+ -4.2914423829643056e-05,
+ -0.004460386000573635,
+ -3.325883881188929e-05,
+ -8.770029067993164,
+ -0.41098108887672424,
+ -0.2444802075624466,
+ -0.5627976655960083,
+ -0.03904035687446594,
+ -0.0310506708920002,
+ -5.941311359405518,
+ -0.004236295353621244,
+ -0.11169589310884476,
+ -0.5418288707733154,
+ -0.00033444532891735435,
+ -0.0041848947294056416,
+ -0.052842844277620316,
+ -0.007783204782754183,
+ -8.83301836438477e-05,
+ -9.250213042832911e-05,
+ -8.129743218887597e-05,
+ -0.0005932478234171867,
+ -0.0002153879904653877,
+ -0.000596107158344239,
+ -2.3007127310847864e-05,
+ -0.0011825718684121966,
+ -5.447716102935374e-05,
+ -0.0001679517881711945,
+ -0.00020585325546562672,
+ -0.014266755431890488,
+ -0.0002109781780745834,
+ -0.009814225137233734,
+ -0.009642690420150757,
+ -0.0001045410826918669,
+ -0.033111490309238434,
+ -0.0008017186191864312,
+ -0.00010358751023886725,
+ -0.0007252446957863867,
+ -2.098061486321967e-05,
+ -2.1582047939300537,
+ -0.02757851965725422,
+ -2.2456743717193604,
+ -4.887596607208252,
+ -0.08968256413936615,
+ -0.001696577062830329,
+ -0.00027247529942542315,
+ -6.544376083184034e-05,
+ -3.444185495376587,
+ -4.3934831619262695,
+ -0.14697670936584473,
+ -0.003605177393183112,
+ -0.00031001531169749796,
+ -0.00011073929636040702,
+ -0.2715941369533539,
+ -3.659658250398934e-05,
+ -0.0025723245926201344,
+ -2.9205850296420977e-05,
+ -4.967594146728516,
+ -3.217968702316284,
+ -0.8693752288818359,
+ -0.5556545257568359,
+ -0.00022301571152638644,
+ -0.006144561339169741,
+ -0.048825882375240326,
+ -0.006224412936717272,
+ -7.56950321374461e-05,
+ -0.0001003691868390888,
+ -9.786603914108127e-05,
+ -0.0003904534096363932,
+ -0.0006033745594322681,
+ -0.0006375664379447699,
+ -1.9073304429184645e-05,
+ -0.0009195152088068426,
+ -3.802703940891661e-05,
+ -0.00012087091454304755,
+ -0.0001551984460093081,
+ -0.062036968767642975,
+ -0.00012468514614738524,
+ -0.00015269544383045286,
+ -0.015339847654104233,
+ -0.00016258825780823827,
+ -1.7683131694793701,
+ -0.028275582939386368,
+ -0.00011324241495458409,
+ -0.0003477922291494906,
+ -3.0636318115284666e-05,
+ -0.5490861535072327,
+ -0.0013725870521739125,
+ -4.768360213347478e-06,
+ -0.00015186110977083445,
+ -0.00019929806876461953,
+ -4.2676016164477915e-05,
+ -0.006663128267973661,
+ -7.247662142617628e-05,
+ -0.0004752936656586826,
+ -4.708655978902243e-05,
+ -0.0012991330586373806,
-1.1920858014491387e-05,
- -0.001943962532095611,
+ -0.0019418209558352828,
-1.4424220353248529e-05,
- -0.0016702761640772223,
- -5.960462772236497e-07,
- -0.0005274811992421746,
- -0.043414343148469925,
- -1.5102243423461914,
- -0.0018298563081771135,
- -0.0035949621815234423,
- -6.842378934379667e-05,
- -0.008245711214840412,
- -0.08723266422748566,
- -0.00939271505922079,
- -0.00011419598013162613,
- -0.0001230164198204875,
- -9.464769391342998e-05,
- -0.0002865380665753037,
- -0.0005069877952337265,
- -0.001016934053041041,
- -3.2305197237292305e-05,
- -0.0009629856795072556,
- -4.827859811484814e-05,
- -0.00021717573690693825,
- -0.00032848684350028634,
- -0.012733934447169304,
- -0.000196556793525815,
- -0.0012980615720152855,
- -0.0077531603164970875,
- -0.00012385078298393637,
- -0.01761084794998169,
- -0.0013621109537780285,
- -0.00011848701251437888,
- -0.0013394916895776987,
- -2.407998726994265e-05,
- -4.505744934082031,
- -1.2715730667114258,
- -0.0005052005290053785,
- -0.00024971229140646756,
- -3.635817120084539e-05,
- -4.3336405754089355,
- -0.0815289318561554,
- -0.028655847534537315,
- -0.00010430268594063818,
- -7.343022298300639e-05,
- -0.158114492893219,
+ -0.0016643255949020386,
+ -7.152555099310121e-07,
+ -0.0005178302526473999,
+ -0.04354605823755264,
+ -1.4898042678833008,
+ -0.002016298472881317,
+ -0.0036044646985828876,
+ -6.4490144723095e-05,
+ -0.008170513436198235,
+ -0.08714307099580765,
+ -0.009718603454530239,
+ -0.00011979816190432757,
+ -0.00013505500101018697,
+ -9.16677454370074e-05,
+ -0.00028701478731818497,
+ -0.0005073452484793961,
+ -0.0010153858456760645,
+ -4.255681051290594e-05,
+ -0.0009608419495634735,
+ -4.4225667807040736e-05,
+ -0.00021181246847845614,
+ -0.0003281293320469558,
+ -0.012738406658172607,
+ -0.00021324267436284572,
+ -0.0012884180760011077,
+ -0.007752331905066967,
+ -9.953480184776708e-05,
+ -0.01761096529662609,
+ -0.001328896265476942,
+ -0.00011085849109804258,
+ -0.0013175864005461335,
+ -2.3603161025675945e-05,
+ -4.519628524780273,
+ -1.2025973796844482,
+ -0.0004720765573438257,
+ -0.00025996167096309364,
+ -3.4689302992774174e-05,
+ -4.313958168029785,
+ -0.0816640704870224,
+ -0.030143653973937035,
+ -0.00011872540198964998,
+ -5.721882189391181e-05,
+ -0.15774711966514587,
-1.764281842042692e-05,
- -0.003166425507515669,
- -5.960446742392378e-06,
- -4.626138687133789,
- -0.5413240194320679,
- -11.11661148071289,
- -6.66420316696167,
- -0.5860735177993774,
- -1.0599334239959717,
- -2.200112819671631,
- -0.4268365502357483,
- -0.027302712202072144,
- -0.15124760568141937,
- -0.12854908406734467,
- -3.041227102279663,
- -0.026920655742287636,
- -0.0003856868715956807,
- -0.004746242426335812,
- -0.07085907459259033,
- -0.008411810733377934,
- -0.00010823617776622996,
- -5.972207145532593e-05,
- -5.507317473529838e-05,
- -0.00023850933939684182,
- -0.0004319211875554174,
- -0.0008380476501770318,
+ -0.004071637522429228,
+ -5.8412379075889476e-06,
+ -4.6149797439575195,
+ -0.5420227646827698,
+ -11.060333251953125,
+ -6.664154052734375,
+ -0.5853514671325684,
+ -1.061955451965332,
+ -2.200925588607788,
+ -0.430128276348114,
+ -0.027363842353224754,
+ -0.15776371955871582,
+ -0.12697036564350128,
+ -3.04177188873291,
+ -0.02755602076649666,
+ -0.00038258862332440913,
+ -0.004620708059519529,
+ -0.07068970054388046,
+ -0.00945353228598833,
+ -0.0001070442158379592,
+ -5.590759246842936e-05,
+ -5.411955135059543e-05,
+ -0.0002401778765488416,
+ -0.0004325169720686972,
+ -0.0008405489497818053,
-1.823885577323381e-05,
- -0.0009161804337054491,
- -3.683499380713329e-05,
- -0.00010918975021922961,
- -0.00016044282529037446,
- -0.0005364171229302883,
- -0.0001248043408850208,
- -0.10185468196868896,
- -0.02194770984351635,
- -0.00011252723925281316,
- -0.6942679286003113,
- -0.21981695294380188,
- -6.496695277746767e-05,
- -0.00030393750057555735,
- -2.13382354559144e-05,
- -3.1545064449310303,
- -0.021652380004525185,
- -0.02087036333978176,
- -0.89057856798172,
- -9.619726915843785e-05,
- -8.129743218887597e-05,
- -2.5152843591058627e-05,
- -4.086198806762695,
- -1.0591976642608643,
- -0.0020325970835983753,
- -4.1483970562694594e-05,
- -0.596172571182251,
- -3.242440288886428e-05,
- -0.0019346822518855333,
- -1.6927575416048057e-05,
- -3.4360618591308594,
- -2.4312753677368164,
- -1.9711253643035889,
- -4.358899116516113,
- -10.540913581848145,
- -5.990867614746094,
- -0.266180157661438,
- -0.000266278104390949,
- -0.003696990432217717,
- -0.03691418468952179,
- -0.005084204487502575,
- -7.73638384998776e-05,
+ -0.0009380945703014731,
+ -2.8729025871143676e-05,
+ -8.475421054754406e-05,
+ -0.0001662831346038729,
+ -0.0007327494095079601,
+ -0.00012575789878610522,
+ -0.09050984680652618,
+ -0.02065739408135414,
+ -0.00011276562872808427,
+ -0.6518867015838623,
+ -0.2065374255180359,
+ -6.568216485902667e-05,
+ -0.0003073934931308031,
+ -2.098061486321967e-05,
+ -3.1568620204925537,
+ -0.022977761924266815,
+ -0.021739285439252853,
+ -0.8908287286758423,
+ -8.415821503149346e-05,
+ -8.165503095369786e-05,
+ -2.7179348762729205e-05,
+ -4.090707778930664,
+ -1.056520700454712,
+ -0.00203557126224041,
+ -4.160317621426657e-05,
+ -0.5973647236824036,
+ -3.325883881188929e-05,
+ -0.0019417019793763757,
+ -1.6331539882230572e-05,
+ -3.437778949737549,
+ -2.4342737197875977,
+ -1.9497886896133423,
+ -4.228105545043945,
+ -10.535257339477539,
+ -5.980109691619873,
+ -0.26758477091789246,
+ -0.0002343380037928,
+ -0.0030076527036726475,
+ -0.03653198853135109,
+ -0.005291385110467672,
+ -9.846202738117427e-05,
-5.9960475482512265e-05,
- -6.12716976320371e-05,
- -0.0001915509783430025,
- -0.0004040378553327173,
- -0.0004508670826908201,
- -2.2172682292875834e-05,
- -0.0010245556477457285,
+ -6.222531374078244e-05,
+ -0.00018439977429807186,
+ -0.00040236959466710687,
+ -0.0004789874074049294,
+ -2.1576648578047752e-05,
+ -0.0009196343016810715,
-3.862306402879767e-05,
- -7.652943895664066e-05,
- -0.00010585224663373083,
- -0.00034791138023138046,
- -0.0001134808044298552,
- -0.009721791371703148,
- -0.01306991372257471,
- -7.86750388215296e-05,
- -0.06928819417953491,
- -0.0019708510953933,
- -8.070142939686775e-05,
- -0.0006008726777508855,
- -1.9550132492440753e-05,
- -1.2050050497055054,
- -0.0022362482268363237,
- -4.887569048150908e-06,
- -0.00016652150952722877,
- -0.0001282609737245366,
- -3.3854863431770355e-05,
- -0.005613160319626331,
- -4.935142715112306e-05,
- -0.00040618274942971766,
- -3.814624506048858e-05,
- -0.0012768696760758758,
- -6.9141146923357155e-06,
- -0.0021407324820756912,
+ -7.64102369430475e-05,
+ -0.00010609064338495955,
+ -0.00032491172896698117,
+ -0.00012206286191940308,
+ -0.00859454832971096,
+ -0.011765611357986927,
+ -9.548207890475169e-05,
+ -0.07109015434980392,
+ -0.00221234024502337,
+ -8.129743218887597e-05,
+ -0.0006108802044764161,
+ -1.6331539882230572e-05,
+ -1.2050546407699585,
+ -0.0022394596599042416,
+ -4.529942543740617e-06,
+ -0.0001578206429257989,
+ -0.0001357701694360003,
+ -3.373566141817719e-05,
+ -0.005778631195425987,
+ -5.018585216021165e-05,
+ -0.000390215078368783,
+ -3.0397906812140718e-05,
+ -0.0014029431622475386,
+ -8.821448318485636e-06,
+ -0.001671466277912259,
-1.0251946150674485e-05,
- -0.001328301033936441,
+ -0.0013266343157738447,
-4.768370445162873e-07,
- -0.00039104922325350344,
- -0.03403102979063988,
- -2.371554374694824,
- -0.0011966219171881676,
- -0.0017084777355194092,
- -1.2397689715726301e-05,
- -0.0012181727215647697,
- -0.027773091569542885,
- -0.004225967917591333,
- -7.688703772146255e-05,
+ -0.0003953390696551651,
+ -0.031116193160414696,
+ -2.2898995876312256,
+ -0.0012049565557390451,
+ -0.0017442979151383042,
+ -1.2159273865108844e-05,
+ -0.0014422263484448195,
+ -0.027814019471406937,
+ -0.0041466690599918365,
+ -7.724463648628443e-05,
-10.750052452087402,
- -0.09749454259872437,
- -0.0398833304643631,
- -0.05019160360097885,
- -0.02639356628060341,
- -0.001116844010539353,
- -0.010394011624157429,
- -0.0002687808300834149,
- -0.0412154421210289,
- -0.17060238122940063,
- -0.44570907950401306,
- -0.001759529928676784,
- -0.8481433987617493,
- -3.9174411296844482,
- -0.0011847150744870305,
- -1.8217713832855225,
- -1.9833719730377197,
- -0.0033980030566453934,
- -0.022340646013617516,
- -0.0005044856225140393,
- -11.916642189025879,
- -2.2062525749206543,
- -0.011109520681202412,
- -0.0025012181140482426,
- -0.00047839165199548006,
- -10.590877532958984,
- -5.111791133880615,
- -0.8751921653747559,
- -0.19319908320903778,
- -0.04376664385199547,
- -0.019606946036219597,
- -0.00042000532266683877,
- -9.505635261535645,
- -0.07715455442667007,
- -0.005082899704575539,
- -0.04224858805537224,
- -0.03572046384215355,
- -0.0011238694423809648,
- -5.344630241394043,
- -3.876430034637451,
- -12.252359390258789,
- -4.9860382080078125,
- -2.668943405151367,
- -1.16416597366333,
- -2.514509677886963,
- -2.5190258026123047,
- -14.754651069641113,
- -5.655267715454102,
- -6.61380672454834,
- -4.71486234664917,
- -0.5776815414428711,
- -1.3986684083938599,
- -2.637193202972412,
- -1.1604831218719482,
- -1.4959537982940674,
- -0.004402587655931711,
- -0.5065803527832031,
- -3.3776161670684814,
- -0.7203826308250427,
- -0.02161656692624092,
- -0.819121241569519,
- -0.04418942704796791,
- -1.7282390594482422,
- -0.05629342794418335,
- -0.008580365218222141,
- -0.000747877755202353,
- -0.013715313747525215,
- -0.00015138434537220746,
- -0.006047403905540705,
- -0.024643857032060623,
- -0.05186835676431656,
- -0.0005345107638277113,
- -0.10883784294128418,
- -1.3612172603607178,
- -0.0003692421887535602,
- -1.357957363128662,
- -0.05831316113471985,
- -0.00040570611599832773,
- -0.0035074164625257254,
- -6.437094270950183e-05,
- -1.7280149459838867,
- -0.026309387758374214,
- -2.3754658699035645,
- -0.05959097668528557,
- -0.0019271865021437407,
- -0.0006563892820850015,
- -0.00038985759601928294,
- -0.00013529339048545808,
- -6.799666881561279,
- -0.4319588541984558,
- -0.0018134353449568152,
- -0.00010084597306558862,
- -3.564793109893799,
- -0.0016862234333530068,
- -0.007215393707156181,
- -0.00018916724366135895,
- -4.893386363983154,
- -0.7495713233947754,
- -0.04057759419083595,
- -0.16563259065151215,
- -3.7694530487060547,
- -0.7686876654624939,
- -0.02867751009762287,
- -3.4293549060821533,
- -1.9938279390335083,
- -3.87074613571167,
- -7.779223918914795,
- -0.11301646381616592,
- -0.0007675323868170381,
- -0.0353383906185627,
- -0.5969783663749695,
- -0.03809810429811478,
- -0.00048828122089616954,
- -0.024168511852622032,
- -0.0024346255231648684,
- -0.006569692399352789,
- -0.002209961414337158,
- -0.001069331425242126,
- -7.819823804311454e-05,
- -0.0029135181102901697,
- -4.60137271147687e-05,
- -0.0003582789213396609,
+ -0.10282254964113235,
+ -0.039773814380168915,
+ -0.05065406113862991,
+ -0.0271708145737648,
+ -0.0011447074357420206,
+ -0.00991420354694128,
+ -0.00027700403006747365,
+ -0.03729129210114479,
+ -0.16962914168834686,
+ -0.44594845175743103,
+ -0.0018586517544463277,
+ -0.8312537670135498,
+ -4.031896114349365,
+ -0.001257463125512004,
+ -1.821771502494812,
+ -1.9813495874404907,
+ -0.0036345156840980053,
+ -0.022193634882569313,
+ -0.00048232366680167615,
+ -11.87901496887207,
+ -2.2126622200012207,
+ -0.012645655311644077,
+ -0.0024358145892620087,
+ -0.0005171154043637216,
+ -10.64317512512207,
+ -5.122398376464844,
+ -0.8483930826187134,
+ -0.21630460023880005,
+ -0.04415018856525421,
+ -0.020105261355638504,
+ -0.0003734129713848233,
+ -9.731037139892578,
+ -0.0780535638332367,
+ -0.005767015740275383,
+ -0.042193274945020676,
+ -0.03950341418385506,
+ -0.0010259846458211541,
+ -5.368424892425537,
+ -3.8717446327209473,
+ -12.254188537597656,
+ -4.985353469848633,
+ -2.6314144134521484,
+ -1.1586227416992188,
+ -2.499390125274658,
+ -2.519989013671875,
+ -14.693689346313477,
+ -5.696416854858398,
+ -6.6406755447387695,
+ -4.719296455383301,
+ -0.5249666571617126,
+ -1.4039437770843506,
+ -2.6573104858398438,
+ -1.2030140161514282,
+ -1.6004329919815063,
+ -0.00434051314368844,
+ -0.5139809846878052,
+ -3.259213447570801,
+ -0.7186939716339111,
+ -0.02127527818083763,
+ -0.8464836478233337,
+ -0.040414463728666306,
+ -1.7282320261001587,
+ -0.05802331864833832,
+ -0.008705994114279747,
+ -0.0007096394547261298,
+ -0.013228391297161579,
+ -0.00014125302550382912,
+ -0.00532672181725502,
+ -0.02297193743288517,
+ -0.05059921368956566,
+ -0.0005256939912214875,
+ -0.1093946099281311,
+ -1.3967794179916382,
+ -0.0003780603874474764,
+ -1.3749324083328247,
+ -0.053297750651836395,
+ -0.00040451448876410723,
+ -0.003983658738434315,
+ -6.425174069590867e-05,
+ -1.725297451019287,
+ -0.02523615211248398,
+ -2.2629284858703613,
+ -0.052103862166404724,
+ -0.0019268295727670193,
+ -0.0006641327636316419,
+ -0.0003833036171272397,
+ -0.00013553177996072918,
+ -6.900384426116943,
+ -0.39039379358291626,
+ -0.001796538126654923,
+ -0.00010990492592100054,
+ -3.5404791831970215,
+ -0.001702646492049098,
+ -0.007246401626616716,
+ -0.0001892864383989945,
+ -4.79708194732666,
+ -0.8172286748886108,
+ -0.04497222974896431,
+ -0.16637395322322845,
+ -3.8862829208374023,
+ -0.7114430069923401,
+ -0.02864970825612545,
+ -3.3703126907348633,
+ -1.9768058061599731,
+ -3.8709847927093506,
+ -7.714934825897217,
+ -0.10974952578544617,
+ -0.0008415018673986197,
+ -0.040607355535030365,
+ -0.5969805121421814,
+ -0.03528913855552673,
+ -0.0004911408759653568,
+ -0.024118708446621895,
+ -0.0022870355751365423,
+ -0.006580706220120192,
+ -0.002045564353466034,
+ -0.0012041230220347643,
+ -7.509902934543788e-05,
+ -0.0027602927293628454,
+ -4.8993817472364753e-05,
+ -0.0004024887748528272,
-0.001116367639042437,
-0.002629396505653858,
- -0.0002420847595203668,
- -0.17575480043888092,
- -0.017076482996344566,
- -0.0001431601122021675,
- -0.10536163300275803,
- -0.00507151335477829,
- -0.00011181206355104223,
- -0.0018749530427157879,
- -2.3603161025675945e-05,
- -0.8358778953552246,
- -0.002124911407008767,
- -9.894321920000948e-06,
- -0.00019214690837543458,
- -0.0002456601650919765,
- -3.516612196108326e-05,
- -0.008302814327180386,
- -0.00010895135346800089,
- -0.0006008726777508855,
- -3.2543604902457446e-05,
- -0.006115178111940622,
- -2.1219027985353023e-05,
- -0.0036275077145546675,
+ -0.00023934361524879932,
+ -0.1934685856103897,
+ -0.017828447744250298,
+ -0.00013469743134919554,
+ -0.10506075620651245,
+ -0.003996481653302908,
+ -0.00011205045302631333,
+ -0.0016538526397198439,
+ -2.3841574147809297e-05,
+ -0.8225477337837219,
+ -0.002133119385689497,
+ -1.07287787614041e-05,
+ -0.00020001317898277193,
+ -0.0002526917669456452,
+ -3.349725011503324e-05,
+ -0.007122246082872152,
+ -9.917721035890281e-05,
+ -0.0005891970940865576,
+ -3.290122185717337e-05,
+ -0.006138637196272612,
-1.7165990357170813e-05,
- -0.003067908575758338,
- -9.536738616588991e-07,
- -0.0006908176001161337,
- -0.02611708454787731,
- -1.3316965103149414,
- -0.003817296586930752,
- -0.006795391906052828,
- -4.684815212385729e-05,
- -0.007690228521823883,
- -0.14891591668128967,
- -0.013032732531428337,
- -0.0002714027068577707,
- -0.011644137091934681,
- -0.00091856240760535,
- -0.0013096098555251956,
- -0.0007771808886900544,
- -0.0009541726321913302,
- -5.638440416078083e-05,
- -0.0014388932613655925,
- -5.018585216021165e-05,
- -0.00020930961181875318,
- -0.0006467396160587668,
- -0.0013236580416560173,
- -0.00019333878299221396,
- -0.05778864026069641,
- -0.023562893271446228,
- -0.0001699779968475923,
- -0.4867134690284729,
- -0.17518886923789978,
- -6.01988795096986e-05,
- -0.00056429672986269,
- -2.396077979938127e-05,
- -10.983257293701172,
- -3.4146568775177,
- -0.007948435842990875,
- -0.005365850869566202,
- -0.00041166413575410843,
- -6.0437283536884934e-05,
- -1.4208624362945557,
- -0.014981495216488838,
- -0.00011193125828867778,
- -2.95634672511369e-05,
- -0.3359139859676361,
- -6.425174069590867e-05,
- -0.0036992470268160105,
- -1.7523612768854946e-05,
- -1.6273220777511597,
- -12.038379669189453,
- -1.8510823249816895,
- -4.6685380935668945,
- -1.03892183303833,
- -3.5619592666625977,
- -3.119525194168091,
- -8.74183177947998,
- -0.1955474466085434,
- -0.00022349244682118297,
- -0.005337630398571491,
- -0.07253769785165787,
- -0.0067605809308588505,
- -0.00018821375851985067,
- -0.01270250789821148,
- -0.0005373702733777463,
- -0.0013699679402634501,
- -0.0009596510208211839,
- -0.0003953390696551651,
+ -0.003627151483669877,
-1.7165990357170813e-05,
- -0.0010408704401925206,
- -3.4450891689630225e-05,
- -0.00011038171214750037,
- -0.00048351517762057483,
- -0.0015029336791485548,
- -0.00013958434283267707,
- -0.027578983455896378,
+ -0.003021558281034231,
+ -9.536738616588991e-07,
+ -0.0008679436868987978,
+ -0.026252955198287964,
+ -1.3258216381072998,
+ -0.0039386567659676075,
+ -0.006899935193359852,
+ -4.768258077092469e-05,
+ -0.0075222342275083065,
+ -0.14876028895378113,
+ -0.013457768596708775,
+ -0.00027247529942542315,
+ -0.014900107868015766,
+ -0.0009192770230583847,
+ -0.001426989445462823,
+ -0.000993116176687181,
+ -0.0009552444680593908,
+ -5.411955135059543e-05,
+ -0.0014282988850027323,
+ -4.3748852476710454e-05,
+ -0.0002076410164590925,
+ -0.0006455483380705118,
+ -0.0016991952434182167,
+ -0.00020656836568377912,
+ -0.05130963772535324,
+ -0.025677645578980446,
+ -0.00017248096992261708,
+ -0.4693552255630493,
+ -0.15605029463768005,
+ -5.686121585313231e-05,
+ -0.0005718026659451425,
+ -2.2053474822314456e-05,
+ -10.931877136230469,
+ -3.32647442817688,
+ -0.007924545556306839,
+ -0.005380197893828154,
+ -0.0003916450368706137,
+ -6.067568756407127e-05,
+ -1.4296157360076904,
+ -0.014987719245254993,
+ -0.00010942813969450071,
+ -2.7656173188006505e-05,
+ -0.31083944439888,
+ -6.16293036728166e-05,
+ -0.003689032746478915,
+ -1.764281842042692e-05,
+ -1.643222451210022,
+ -12.110017776489258,
+ -1.827063798904419,
+ -4.666322231292725,
+ -1.1210403442382812,
+ -3.5616214275360107,
+ -3.1083669662475586,
+ -8.859210014343262,
+ -0.21794311702251434,
+ -0.00023409964342135936,
+ -0.006090770475566387,
+ -0.0732051208615303,
+ -0.006583192851394415,
+ -0.00019178935326635838,
+ -0.014276274479925632,
+ -0.000547378440387547,
+ -0.0012900849105790257,
+ -0.0010100268991664052,
+ -0.0003967689990531653,
+ -2.1576648578047752e-05,
+ -0.001176380319520831,
+ -3.349725011503324e-05,
+ -0.00011359999916749075,
+ -0.0004829194222111255,
+ -0.0013847296359017491,
+ -0.00014411364099942148,
+ -0.027071036398410797,
-0.02192368544638157,
-8.141662692651153e-05,
- -0.11562338471412659,
- -0.0031276855152100325,
- -6.5205356804654e-05,
- -0.0007344171172007918,
- -2.1457441107486375e-05,
- -1.4039907455444336,
- -0.8585066795349121,
- -0.12097951024770737,
- -4.9232225137529895e-05,
- -0.00045503751607611775,
- -0.0001479277852922678,
- -2.8967437174287625e-05,
- -3.316209316253662,
- -0.22754307091236115,
- -0.037047676742076874,
- -0.00010632903286023065,
- -5.602679812000133e-05,
- -0.10701240599155426,
- -2.1815061700181104e-05,
- -0.0025769618805497885,
- -2.932505594799295e-05,
- -2.9098081588745117,
- -0.23772671818733215,
- -2.5728368759155273,
- -1.0628935098648071,
- -0.569791853427887,
- -1.5512791872024536,
- -0.22174018621444702,
- -0.2053954154253006,
- -0.668795108795166,
- -0.00032574593205936253,
- -0.005275258328765631,
- -0.17121490836143494,
- -0.01520049013197422,
- -0.00027164106722921133,
- -0.018145864829421043,
- -0.0008275659638457,
- -0.0013598490040749311,
- -0.0007223857101053,
- -0.0005415403284132481,
- -3.075552376685664e-05,
- -0.0016680150292813778,
- -4.124556289752945e-05,
- -0.00020203932945150882,
- -0.0005315321614034474,
- -0.0016384999034926295,
- -0.000169382052263245,
- -0.01945134624838829,
- -0.018782030791044235,
- -0.0001429217227268964,
- -1.4800734519958496,
- -0.046756841242313385,
- -9.667406266089529e-05,
- -0.0005499995895661414,
- -1.728519782773219e-05,
- -0.6545608639717102,
- -0.0013740155845880508,
- -5.8412379075889476e-06,
- -0.00015496007108595222,
- -0.0001935771433636546,
- -2.8967437174287625e-05,
- -0.01043801661580801,
- -7.974783511599526e-05,
- -0.0005525015876628458,
- -3.683499380713329e-05,
- -0.002455436158925295,
- -1.2874520507466514e-05,
- -0.0022639615926891565,
- -1.4543427823809907e-05,
- -0.00250252615660429,
- -8.344646857949556e-07,
- -0.0006089740199968219,
- -0.023519812151789665,
- -1.6231462955474854,
- -0.0013103241799399257,
- -0.0044088782742619514,
- -3.433168603805825e-05,
- -0.0076819476671516895,
- -0.13205960392951965,
- -0.01295448187738657,
- -0.0002797450579237193,
- -0.01799413561820984,
- -0.0008688965463079512,
- -0.0026737437583506107,
- -0.0004418112221173942,
- -0.001303895260207355,
- -6.16293036728166e-05,
- -0.0018553201807662845,
- -4.815939246327616e-05,
- -0.00024875884992070496,
- -0.000916537712328136,
- -0.005030237603932619,
- -0.00015853578224778175,
- -0.00936696957796812,
- -0.016335444524884224,
- -9.619726915843785e-05,
- -0.12435520440340042,
- -0.002912804950028658,
- -0.00010346830822527409,
- -0.0007908792467787862,
- -1.7165990357170813e-05,
- -6.260087490081787,
- -4.018156051635742,
- -0.05045890435576439,
- -0.00021360022947192192,
- -4.815939246327616e-05,
- -2.2203869819641113,
- -0.047356534749269485,
- -8.83301836438477e-05,
- -5.781483559985645e-05,
- -0.11337775737047195,
- -3.3378044463461265e-05,
- -0.0019444384379312396,
- -1.645074735279195e-05,
- -1.7198790311813354,
- -3.5991759300231934,
- -2.5881307125091553,
- -4.4389872550964355,
- -0.39235079288482666,
- -0.9257609248161316,
- -2.4064109325408936,
- -2.256807804107666,
- -0.012957894243299961,
- -6.8662193370983e-05,
- -0.005379723850637674,
- -0.1424376517534256,
- -0.008812819607555866,
- -0.00019667598826345056,
+ -0.13038524985313416,
+ -0.0034964873921126127,
+ -6.425174069590867e-05,
+ -0.0007306052139028907,
+ -2.13382354559144e-05,
+ -1.4955275058746338,
+ -0.9337477684020996,
+ -0.133364737033844,
+ -4.9470632802695036e-05,
+ -0.0004514628672040999,
+ -0.00015209948469419032,
+ -2.8013790142722428e-05,
+ -3.301778554916382,
+ -0.20383507013320923,
+ -0.0419706255197525,
+ -0.00013016807497479022,
+ -5.507317473529838e-05,
+ -0.10738518834114075,
+ -3.0040289857424796e-05,
+ -0.0028358979616314173,
+ -2.90866428258596e-05,
+ -2.9424922466278076,
+ -0.2351037710905075,
+ -2.5789151191711426,
+ -1.1322591304779053,
+ -0.5736383199691772,
+ -1.5535979270935059,
+ -0.22191236913204193,
+ -0.18333813548088074,
+ -0.664542555809021,
+ -0.0003240775258745998,
+ -0.005283796228468418,
+ -0.17099550366401672,
+ -0.014068713411688805,
+ -0.0002734287118073553,
+ -0.018341930583119392,
+ -0.0008268513483926654,
+ -0.001468295231461525,
+ -0.0007066613179631531,
+ -0.0006038511055521667,
+ -2.8609820219571702e-05,
+ -0.0019271865021437407,
+ -4.136476854910143e-05,
+ -0.00020990552729927003,
+ -0.0005302215577103198,
+ -0.0016419512685388327,
+ -0.00018070495571009815,
+ -0.019507931545376778,
+ -0.018029844388365746,
+ -0.00013004888023715466,
+ -1.4371798038482666,
+ -0.052820682525634766,
+ -0.00011884459672728553,
+ -0.00045980370487086475,
+ -1.7046782886609435e-05,
+ -0.6581263542175293,
+ -0.0013815154088661075,
+ -6.079655122448457e-06,
+ -0.0001554368354845792,
+ -0.00015615197480656207,
+ -3.611976353568025e-05,
+ -0.009467465803027153,
+ -8.916457591112703e-05,
+ -0.0005535738891921937,
+ -3.671578815556131e-05,
+ -0.002144420053809881,
+ -1.3112935448589269e-05,
+ -0.0022645562421530485,
+ -1.4424220353248529e-05,
+ -0.0024701817892491817,
+ -9.536738616588991e-07,
+ -0.0006376855308189988,
+ -0.02182525023818016,
+ -1.5834624767303467,
+ -0.0013265152228996158,
+ -0.004699495155364275,
+ -3.3854863431770355e-05,
+ -0.0077219316735863686,
+ -0.14429691433906555,
+ -0.01174628920853138,
+ -0.000324673397699371,
+ -0.017962872982025146,
+ -0.0008827127167023718,
+ -0.00236874190159142,
+ -0.0003970073303207755,
+ -0.0013256819220259786,
+ -6.0794889577664435e-05,
+ -0.0016473069554194808,
+ -4.8636207793606445e-05,
+ -0.0002506657037883997,
+ -0.0009217780898325145,
+ -0.004512722138315439,
+ -0.00019762947340495884,
+ -0.013031673617661,
+ -0.01633591391146183,
+ -0.00010299152199877426,
+ -0.1220099925994873,
+ -0.0024951535742729902,
+ -0.00010263393050990999,
+ -0.0008403107640333474,
+ -1.6331539882230572e-05,
+ -6.363530158996582,
+ -4.033095359802246,
+ -0.04472124204039574,
+ -0.00020919041708111763,
+ -4.792098479811102e-05,
+ -2.2731337547302246,
+ -0.049864355474710464,
+ -8.940297266235575e-05,
+ -5.221230458118953e-05,
+ -0.11411462724208832,
+ -3.266281055402942e-05,
+ -0.0019317077239975333,
+ -2.098061486321967e-05,
+ -1.7188977003097534,
+ -3.65950345993042,
+ -2.684589147567749,
+ -4.440427780151367,
+ -0.4001847505569458,
+ -0.9439029693603516,
+ -2.4071009159088135,
+ -2.2622978687286377,
+ -0.014948965050280094,
+ -5.972207145532593e-05,
+ -0.00555791798979044,
+ -0.14306095242500305,
+ -0.007800119463354349,
+ -0.00019858295854646713,
-0.012973662465810776,
- -0.0005903884884901345,
- -0.0019209994934499264,
- -0.0014405598631128669,
- -0.0006889115320518613,
- -1.645074735279195e-05,
- -0.0011966219171881676,
- -3.40932747349143e-05,
- -9.548207890475169e-05,
+ -0.0007591941393911839,
+ -0.0020857739727944136,
+ -0.0014411549782380462,
+ -0.000690460205078125,
+ -2.169585604860913e-05,
+ -0.0012124576605856419,
+ -3.421248038648628e-05,
+ -9.42901024245657e-05,
-0.0005439232336357236,
- -0.004501329269260168,
- -0.00011920218821614981,
- -0.03018992207944393,
- -0.013410485349595547,
- -0.00011467275908216834,
- -0.6566694378852844,
- -0.36726248264312744,
- -2.8490614567999728e-05,
- -0.00023707917716819793,
- -1.3351351299206726e-05,
- -1.051271915435791,
- -0.01689915731549263,
- -3.0722033977508545,
- -0.2818227708339691,
- -3.957169771194458,
- -0.004226442892104387,
- -0.00017248096992261708,
- -3.9457496313843876e-05,
- -5.733857154846191,
- -0.26561957597732544,
- -0.00047779586748220026,
- -2.5748875486897305e-05,
- -0.07624048739671707,
- -6.0437283536884934e-05,
- -0.001644212519749999,
- -1.549708758830093e-05,
- -2.1518163681030273,
- -0.19709540903568268,
- -3.698873996734619,
- -10.724569320678711,
- -2.996880292892456,
- -3.1366219520568848,
- -0.02801341563463211,
- -0.17601795494556427,
- -0.0965375229716301,
- -0.00014578233822248876,
- -0.0020983838476240635,
- -0.054011568427085876,
- -0.003581777447834611,
- -0.00014304091746453196,
- -0.011484465561807156,
- -0.000708090839907527,
- -0.0012874656822532415,
- -0.0009416675311513245,
- -0.0005903884884901345,
- -2.13382354559144e-05,
- -0.0007848043460398912,
- -2.3841574147809297e-05,
- -7.4741430580616e-05,
- -0.0002946419408544898,
- -0.0024204738438129425,
- -0.00011503035057103261,
- -0.006832095794379711,
- -0.010126759298145771,
- -5.876845170860179e-05,
+ -0.004523758310824633,
+ -0.000120751719805412,
+ -0.043312862515449524,
+ -0.01450247410684824,
+ -0.00010632903286023065,
+ -0.718799352645874,
+ -0.3663807809352875,
+ -2.95634672511369e-05,
+ -0.0001879753835964948,
+ -1.6331539882230572e-05,
+ -1.0465071201324463,
+ -0.018635429441928864,
+ -3.112912178039551,
+ -0.2903432548046112,
+ -3.879873752593994,
+ -0.004381461534649134,
+ -0.00013815402053296566,
+ -4.2437604861333966e-05,
+ -5.737420082092285,
+ -0.2639721632003784,
+ -0.0006080209277570248,
+ -2.610649426060263e-05,
+ -0.08038974553346634,
+ -6.151010165922344e-05,
+ -0.0016420703614130616,
+ -1.490105023549404e-05,
+ -2.140990734100342,
+ -0.2073119729757309,
+ -3.695624828338623,
+ -10.747465133666992,
+ -3.023293972015381,
+ -3.056196451187134,
+ -0.03145737573504448,
+ -0.1770988404750824,
+ -0.0917704701423645,
+ -0.00013183678674977273,
+ -0.002531302161514759,
+ -0.04845479875802994,
+ -0.003440296743065119,
+ -0.0001445904199499637,
+ -0.011499903164803982,
+ -0.0007746794726699591,
+ -0.0012647256953641772,
+ -0.0009414292871952057,
+ -0.0005958688561804593,
+ -2.0503786799963564e-05,
+ -0.0007793250260874629,
+ -2.3483953555114567e-05,
+ -7.581423415103927e-05,
+ -0.00029559535323642194,
+ -0.0024205928202718496,
+ -0.00010692501382436603,
+ -0.007723824121057987,
+ -0.009906531311571598,
+ -5.864924969500862e-05,
-0.09275738149881363,
- -0.003692833473905921,
- -4.0411134250462055e-05,
- -0.0005497612874023616,
- -1.537788011773955e-05,
- -1.182621717453003,
- -0.0008486483711749315,
+ -0.003653876017779112,
+ -4.792098479811102e-05,
+ -0.0005359405186027288,
+ -1.549708758830093e-05,
+ -1.2015808820724487,
+ -0.0008397151832468808,
-4.0531076592742465e-06,
- -0.00010585224663373083,
- -0.00011646069469861686,
- -2.407998726994265e-05,
- -0.00471824174746871,
- -5.352353764465079e-05,
- -0.0003631647559814155,
- -3.135155202471651e-05,
- -0.0011143434094265103,
- -1.1205610462639015e-05,
- -0.002159646013751626,
- -1.4185804502631072e-05,
- -0.0011845960980281234,
+ -0.00011955977242905647,
+ -0.00011383838864276186,
+ -2.4318398573086597e-05,
+ -0.005200904794037342,
+ -5.709961988031864e-05,
+ -0.0003819928097072989,
+ -2.5033637939486653e-05,
+ -0.001191859133541584,
+ -1.0847986231965479e-05,
+ -0.0019281383138149977,
+ -1.4543427823809907e-05,
+ -0.001212338567711413,
-7.152555099310121e-07,
- -0.0002699726028367877,
+ -0.00030763185350224376,
-0.008802657015621662,
- -1.1517901420593262,
- -0.0017283515771850944,
- -0.002493488835170865,
- -1.5258672647178173e-05,
- -0.0018479428254067898,
- -0.040569812059402466,
- -0.0041178204119205475,
- -0.00017176583060063422,
+ -1.1513752937316895,
+ -0.001739894854836166,
+ -0.0026299909222871065,
+ -1.2278481335670222e-05,
+ -0.0019224273273721337,
+ -0.03679193556308746,
+ -0.004175397567451,
+ -0.00016950124700088054,
-0.015839355066418648,
- -0.0005023409612476826,
- -0.0007201223634183407,
- -0.0005905076395720243,
- -0.0007784912013448775,
- -2.3483953555114567e-05,
- -0.0008902162662707269,
- -2.6702524337451905e-05,
- -9.512448741588742e-05,
- -0.0004555141495075077,
- -0.014392376877367496,
- -9.619726915843785e-05,
- -0.0002324311062693596,
- -0.01029337290674448,
- -0.00015984688070602715,
- -1.1049474477767944,
- -0.04663100838661194,
- -8.21318244561553e-05,
- -0.0003543464408721775,
- -1.3947389561508317e-05,
- -7.615281581878662,
- -4.125001907348633,
- -0.19173777103424072,
- -0.0005029367166571319,
+ -0.0005919373361393809,
+ -0.0008360228384844959,
+ -0.0006061147432774305,
+ -0.0008789013954810798,
+ -2.396077979938127e-05,
+ -0.0009041512385010719,
+ -2.8967437174287625e-05,
+ -0.00010132275929208845,
+ -0.0004866131057497114,
+ -0.014365584589540958,
+ -9.655486064730212e-05,
+ -0.00021050144277978688,
+ -0.011357063427567482,
+ -0.00016223068814724684,
+ -1.124318242073059,
+ -0.046641137450933456,
+ -6.48477507638745e-05,
+ -0.00036769305006600916,
+ -1.3232143828645349e-05,
+ -7.617920875549316,
+ -4.132878303527832,
+ -0.18580418825149536,
+ -0.0005179494037292898,
-4.100715523236431e-05,
- -2.0808839797973633,
- -0.026673687621951103,
- -7.70062324590981e-05,
- -2.9682672902708873e-05,
- -0.12381786853075027,
- -2.098061486321967e-05,
- -0.0029344377107918262,
- -1.3589766240329482e-05,
- -6.027270793914795,
- -0.344284325838089,
- -0.47963422536849976,
- -1.262589454650879,
- -1.8010940551757812,
- -2.51932430267334,
- -1.5027334690093994,
- -0.06264369934797287,
- -1.8616759777069092,
- -2.732039213180542,
- -6.854299135738984e-05,
- -0.001887565478682518,
- -0.02442971244454384,
- -0.0030983323231339455,
- -0.00013374387344811112,
- -0.010926888324320316,
- -0.0006349454633891582,
- -0.0010619483655318618,
- -0.0007469248375855386,
- -0.00040987672400660813,
- -1.537788011773955e-05,
+ -2.0765302181243896,
+ -0.026648614555597305,
+ -6.532455881824717e-05,
+ -2.455681169521995e-05,
+ -0.13987594842910767,
+ -2.062299427052494e-05,
+ -0.0029500082600861788,
+ -1.3112935448589269e-05,
+ -6.043766975402832,
+ -0.34587275981903076,
+ -0.47646504640579224,
+ -1.2680540084838867,
+ -1.7469899654388428,
+ -2.445939064025879,
+ -1.511207103729248,
+ -0.06483010947704315,
+ -1.8050661087036133,
+ -2.7358272075653076,
+ -6.949660019017756e-05,
+ -0.001785947591997683,
+ -0.02307841368019581,
+ -0.003315905574709177,
+ -0.00013207517622504383,
+ -0.009649066254496574,
+ -0.0005932478234171867,
+ -0.001046229270286858,
+ -0.0009538153535686433,
+ -0.00037079135654494166,
+ -1.4424220353248529e-05,
-0.0008891443139873445,
-2.4676019165781327e-05,
- -7.080780778778717e-05,
- -0.00043299360550008714,
- -0.2814013361930847,
- -6.8662193370983e-05,
- -0.0011491130571812391,
- -0.007679700385779142,
- -9.440929716220126e-05,
- -0.026545187458395958,
- -0.002912091789767146,
- -7.045020902296528e-05,
- -0.001142087858170271,
- -1.4662635294371285e-05,
- -1.6412137746810913,
- -9.728646278381348,
- -0.026286397129297256,
- -0.0002475670480635017,
- -7.60526381782256e-05,
- -2.191868782043457,
- -0.01760944165289402,
- -0.0004247716860845685,
- -4.684815212385729e-05,
- -0.03103969246149063,
- -9.297892393078655e-05,
- -0.011422710493206978,
- -3.6954195820726454e-05,
- -4.347017288208008,
- -0.000610999355558306,
- -2.17897367477417,
- -2.866166353225708,
- -0.23518076539039612,
- -0.00036125810584053397,
- -0.01150013878941536,
- -1.8427702188491821,
- -0.22964701056480408,
- -0.011748881079256535,
- -0.00036352223833091557,
- -2.021958827972412,
- -0.008272194303572178,
- -1.7123057842254639,
- -9.325576782226562,
- -1.3440426588058472,
- -3.209916830062866,
- -0.053304191678762436,
- -5.205663681030273,
- -0.03287550434470177,
- -1.384042501449585,
- -7.2653326988220215,
- -3.6932270526885986,
- -6.713709354400635,
- -0.08502203971147537,
- -3.0402512550354004,
- -0.043377358466386795,
- -0.00908633042126894,
- -0.013433421961963177,
- -1.5646146535873413,
- -0.007355276495218277,
- -5.929056167602539,
- -11.379992485046387,
- -3.6368532180786133,
- -0.45781779289245605,
- -0.003176526166498661,
- -0.039530687034130096,
- -0.0007678897818550467,
- -0.0003935516288038343,
- -0.0005339150666259229,
- -3.964613437652588,
- -0.9957391023635864,
- -0.2830953896045685,
- -0.2610830068588257,
- -0.07025375217199326,
- -0.010492399334907532,
- -3.3028924465179443,
- -4.77099084854126,
- -0.22153465449810028,
- -0.0006783091812394559,
- -0.010667562484741211,
- -0.0019300420535728335,
- -0.00015901254664640874,
- -0.0025420039892196655,
- -0.0038043521344661713,
- -0.0017481058603152633,
- -0.47604793310165405,
- -0.0035732248798012733,
- -1.3063528537750244,
- -1.869868278503418,
- -0.03979752957820892,
- -1.2474843263626099,
- -0.1659490168094635,
- -0.006920535117387772,
- -1.4199819564819336,
- -2.666736364364624,
- -1.218197226524353,
- -0.017583556473255157,
- -0.5422223806381226,
- -0.0014107999159023166,
- -0.0003762729174923152,
- -0.0010189585154876113,
- -0.03820022940635681,
- -0.0013802058529108763,
- -0.1537325382232666,
- -0.0007863528444431722,
- -0.003772999858483672,
- -0.019278066232800484
+ -7.116541382856667e-05,
+ -0.00043275527423247695,
+ -0.17996622622013092,
+ -7.116541382856667e-05,
+ -0.0011485177092254162,
+ -0.007210304494947195,
+ -8.940297266235575e-05,
+ -0.027487365528941154,
+ -0.002912804950028658,
+ -7.033100700937212e-05,
+ -0.001149232150055468,
+ -1.8954096958623268e-05,
+ -1.6481224298477173,
+ -9.633079528808594,
+ -0.02628662995994091,
+ -0.0002915434306487441,
+ -5.936446541454643e-05,
+ -2.157845973968506,
+ -0.017832664772868156,
+ -0.0004151197790633887,
+ -4.672895011026412e-05,
+ -0.03097185306251049,
+ -9.202533692587167e-05,
+ -0.01144097838550806,
+ -3.71926071238704e-05,
+ -4.355002403259277,
+ -0.0006113567505963147,
+ -2.2188010215759277,
+ -2.907867908477783,
+ -0.23523163795471191,
+ -0.00036507140612229705,
+ -0.011011547408998013,
+ -1.8425185680389404,
+ -0.22946366667747498,
+ -0.011686672456562519,
+ -0.0003597089380491525,
+ -2.111253023147583,
+ -0.009250637143850327,
+ -1.7095400094985962,
+ -9.170680046081543,
+ -1.3476110696792603,
+ -3.1840837001800537,
+ -0.054254475980997086,
+ -5.210089206695557,
+ -0.03295567259192467,
+ -1.4402692317962646,
+ -7.255331993103027,
+ -3.6931405067443848,
+ -6.715616226196289,
+ -0.08208508044481277,
+ -3.03913950920105,
+ -0.043953072279691696,
+ -0.008821917697787285,
+ -0.014872273430228233,
+ -1.6501425504684448,
+ -0.007400597911328077,
+ -5.930353164672852,
+ -11.405633926391602,
+ -3.518193483352661,
+ -0.4507780373096466,
+ -0.0035801143385469913,
+ -0.035687681287527084,
+ -0.000773488252889365,
+ -0.0003968881501350552,
+ -0.0005335576133802533,
+ -4.078104019165039,
+ -0.9954992532730103,
+ -0.31775549054145813,
+ -0.26048052310943604,
+ -0.07195602357387543,
+ -0.010182575322687626,
+ -3.3031930923461914,
+ -4.769801139831543,
+ -0.21995607018470764,
+ -0.0007650309125892818,
+ -0.010548313148319721,
+ -0.00197001826018095,
+ -0.00015722469834145159,
+ -0.0025895654689520597,
+ -0.003808746114373207,
+ -0.0017583399312570691,
+ -0.47401726245880127,
+ -0.003675017738714814,
+ -1.3417091369628906,
+ -1.8469452857971191,
+ -0.040168169885873795,
+ -1.2434231042861938,
+ -0.1829521209001541,
+ -0.007097271271049976,
+ -1.3962904214859009,
+ -2.523759603500366,
+ -1.2317531108856201,
+ -0.017718834802508354,
+ -0.5408012866973877,
+ -0.0015022194711491466,
+ -0.00041237910045310855,
+ -0.0009870424401015043,
+ -0.0377543605864048,
+ -0.0013729440979659557,
+ -0.14942514896392822,
+ -0.0007965967524796724,
+ -0.0037569671403616667,
+ -0.019313964992761612
]
},
- "mem-max-allocated-bytes": 23252846080
-}
+ "mem-max-allocated-bytes": 25127683072,
+ "lifetime_prefill_token_count": 2739
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m_chunked_prefill/model_config.yaml b/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m_chunked_prefill/model_config.yaml
index 7ff5911a877..b10698d521f 100644
--- a/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m_chunked_prefill/model_config.yaml
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m_chunked_prefill/model_config.yaml
@@ -3,6 +3,8 @@ ENV_VARS:
NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
NCCL_ALGO: Ring
CUBLAS_WORKSPACE_CONFIG: :4096:8
+ TRITON_CACHE_AUTOTUNING: 0
+ MAMBA_DETERMINISTIC: 1
TEST_TYPE: frozen-start
MODE: inference
MODEL_ARGS:
@@ -22,19 +24,17 @@ MODEL_ARGS:
--pipeline-model-parallel-size: 1
--expert-model-parallel-size: 1
--use-mcore-models: true
- --is-hybrid-model: true
--model-provider: mamba
--init-method-std: 0.0198
--untie-embeddings-and-output-weights: true
--disable-bias-linear: true
--init-method-std: 0.014
--position-embedding-type: none
- --num-layers: 50
--hidden-size: 2048
--ffn-hidden-size: 11264
--num-attention-heads: 16
--kv-channels: 128
- --hybrid-override-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-
+ --hybrid-layer-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-
--spec: megatron.core.models.mamba.mamba_layer_specs mamba_stack_spec
--normalization: RMSNorm
--swiglu: true
@@ -71,6 +71,8 @@ MODEL_ARGS:
--incoming-requests-per-step: 32
--inference-repeat-n: 3
--no-record-throughput: true
+ --mamba-inference-conv-states-dtype: fp32
+ --mamba-inference-ssm-states-dtype: fp32
METRICS:
- "generated_tokens"
- "logprobs"
diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp1_cp1_dgx_a100_1N8G/model_config.yaml b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp1_cp1_dgx_a100_1N8G/model_config.yaml
index 123e2e98a36..6d40098499d 100644
--- a/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp1_cp1_dgx_a100_1N8G/model_config.yaml
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp1_cp1_dgx_a100_1N8G/model_config.yaml
@@ -4,14 +4,12 @@ ENV_VARS:
NCCL_ALGO: Ring
CUBLAS_WORKSPACE_CONFIG: :4096:8
MODEL_ARGS:
- --num-layers: 44
--hidden-size: 1024
--num-attention-heads: 16
--group-query-attention: true
--num-query-groups: 8
- --hybrid-override-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M-
+ --hybrid-layer-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M-
--spec: "[megatron.core.models.mamba.mamba_layer_specs mamba_stack_spec]"
- --is-hybrid-model: true
--log-params-norm: true
--log-num-zeros-in-grad: true
--log-validation-ppl-to-tensorboard: true
@@ -26,9 +24,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -57,5 +55,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
- --legacy-tokenizer: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp2_vpp2_cp1_dgx_a100_1N8G/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp2_vpp2_cp1_dgx_a100_1N8G/golden_values_dev_dgx_h100.json
new file mode 100644
index 00000000000..b3308e04b06
--- /dev/null
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp2_vpp2_cp1_dgx_a100_1N8G/golden_values_dev_dgx_h100.json
@@ -0,0 +1,287 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 10.98453,
+ "2": 10.98656,
+ "3": 10.98558,
+ "4": 10.97462,
+ "5": 11.01243,
+ "6": 11.01486,
+ "7": 10.99829,
+ "8": 10.98374,
+ "9": 10.97958,
+ "10": 10.95257,
+ "11": 11.00151,
+ "12": 10.97264,
+ "13": 10.97068,
+ "14": 10.9819,
+ "15": 10.86751,
+ "16": 10.86056,
+ "17": 10.82417,
+ "18": 10.83853,
+ "19": 10.82792,
+ "20": 10.63567,
+ "21": 10.5832,
+ "22": 10.34766,
+ "23": 10.61001,
+ "24": 10.3489,
+ "25": 10.24413,
+ "26": 10.37199,
+ "27": 10.3839,
+ "28": 10.34912,
+ "29": 10.3595,
+ "30": 9.90123,
+ "31": 9.46177,
+ "32": 10.08687,
+ "33": 10.07688,
+ "34": 9.63497,
+ "35": 9.68183,
+ "36": 9.56636,
+ "37": 9.80399,
+ "38": 9.50995,
+ "39": 9.91757,
+ "40": 9.32825,
+ "41": 9.47987,
+ "42": 9.55419,
+ "43": 9.02825,
+ "44": 9.14665,
+ "45": 8.99067,
+ "46": 9.05279,
+ "47": 9.47035,
+ "48": 9.03541,
+ "49": 8.57937,
+ "50": 9.11692
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 20941.0,
+ "2": 21706.0,
+ "3": 21312.0,
+ "4": 21029.0,
+ "5": 23603.0,
+ "6": 23558.0,
+ "7": 23492.0,
+ "8": 21856.0,
+ "9": 23107.0,
+ "10": 19088.0,
+ "11": 24660.0,
+ "12": 23306.0,
+ "13": 24163.0,
+ "14": 24444.0,
+ "15": 23148.0,
+ "16": 23702.0,
+ "17": 22014.0,
+ "18": 22378.0,
+ "19": 23608.0,
+ "20": 21520.0,
+ "21": 22232.0,
+ "22": 18801.0,
+ "23": 24318.0,
+ "24": 19502.0,
+ "25": 19048.0,
+ "26": 20393.0,
+ "27": 21793.0,
+ "28": 22862.0,
+ "29": 22737.0,
+ "30": 19741.0,
+ "31": 16792.0,
+ "32": 21327.0,
+ "33": 22863.0,
+ "34": 21230.0,
+ "35": 21207.0,
+ "36": 20330.0,
+ "37": 22367.0,
+ "38": 22291.0,
+ "39": 22436.0,
+ "40": 23187.0,
+ "41": 24131.0,
+ "42": 23488.0,
+ "43": 21513.0,
+ "44": 21418.0,
+ "45": 21854.0,
+ "46": 22905.0,
+ "47": 24925.0,
+ "48": 24925.0,
+ "49": 25464.0,
+ "50": 27681.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 2178379776.0,
+ "2": 2178379776.0,
+ "3": 2178641920.0,
+ "4": 2179428352.0,
+ "5": 2178641920.0,
+ "6": 2178641920.0,
+ "7": 2178641920.0,
+ "8": 2178641920.0,
+ "9": 2178641920.0,
+ "10": 2178641920.0,
+ "11": 2178641920.0,
+ "12": 2178641920.0,
+ "13": 2178641920.0,
+ "14": 2178641920.0,
+ "15": 2178641920.0,
+ "16": 2178641920.0,
+ "17": 2178641920.0,
+ "18": 2178641920.0,
+ "19": 2178641920.0,
+ "20": 2178641920.0,
+ "21": 2178641920.0,
+ "22": 2178641920.0,
+ "23": 2178641920.0,
+ "24": 2178641920.0,
+ "25": 2178641920.0,
+ "26": 2178641920.0,
+ "27": 2178641920.0,
+ "28": 2178641920.0,
+ "29": 2178641920.0,
+ "30": 2178641920.0,
+ "31": 2178641920.0,
+ "32": 2178641920.0,
+ "33": 2178641920.0,
+ "34": 2178641920.0,
+ "35": 2178641920.0,
+ "36": 2178641920.0,
+ "37": 2178641920.0,
+ "38": 2178379776.0,
+ "39": 2178641920.0,
+ "40": 2178379776.0,
+ "41": 2178379776.0,
+ "42": 2178904064.0,
+ "43": 2178379776.0,
+ "44": 2178641920.0,
+ "45": 2178641920.0,
+ "46": 2178641920.0,
+ "47": 2178641920.0,
+ "48": 2178379776.0,
+ "49": 2178641920.0,
+ "50": 2178641920.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 7284575744.0,
+ "2": 7744184832.0,
+ "3": 7744184832.0,
+ "4": 7744709120.0,
+ "5": 7744709120.0,
+ "6": 7744709120.0,
+ "7": 7744709120.0,
+ "8": 7744709120.0,
+ "9": 7744709120.0,
+ "10": 7744709120.0,
+ "11": 7744709120.0,
+ "12": 7744709120.0,
+ "13": 7744709120.0,
+ "14": 7744709120.0,
+ "15": 7744709120.0,
+ "16": 7744709120.0,
+ "17": 7744709120.0,
+ "18": 7744709120.0,
+ "19": 7744709120.0,
+ "20": 7744709120.0,
+ "21": 7744709120.0,
+ "22": 7744709120.0,
+ "23": 7744709120.0,
+ "24": 7744709120.0,
+ "25": 7744709120.0,
+ "26": 7744709120.0,
+ "27": 7744709120.0,
+ "28": 7744709120.0,
+ "29": 7744709120.0,
+ "30": 7744709120.0,
+ "31": 7744709120.0,
+ "32": 7744709120.0,
+ "33": 7744709120.0,
+ "34": 7744709120.0,
+ "35": 7744709120.0,
+ "36": 7744709120.0,
+ "37": 7744709120.0,
+ "38": 7744709120.0,
+ "39": 7744709120.0,
+ "40": 7744709120.0,
+ "41": 7744709120.0,
+ "42": 7744709120.0,
+ "43": 7744709120.0,
+ "44": 7744709120.0,
+ "45": 7744709120.0,
+ "46": 7744709120.0,
+ "47": 7744709120.0,
+ "48": 7744709120.0,
+ "49": 7744709120.0,
+ "50": 7744709120.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": "nan",
+ "2": 68.99515,
+ "3": 0.16958,
+ "4": 0.16632,
+ "5": 0.167,
+ "6": 0.16564,
+ "7": 0.16619,
+ "8": 0.16588,
+ "9": 0.16662,
+ "10": 0.16785,
+ "11": 0.82413,
+ "12": 0.1671,
+ "13": 0.16722,
+ "14": 0.16724,
+ "15": 0.16551,
+ "16": 0.1671,
+ "17": 0.16656,
+ "18": 0.1668,
+ "19": 0.16522,
+ "20": 0.16556,
+ "21": 0.81885,
+ "22": 0.16567,
+ "23": 0.16748,
+ "24": 0.16601,
+ "25": 0.16584,
+ "26": 0.16611,
+ "27": 0.16667,
+ "28": 0.16529,
+ "29": 0.1659,
+ "30": 0.16604,
+ "31": 0.80768,
+ "32": 0.16703,
+ "33": 0.16588,
+ "34": 0.16788,
+ "35": 0.16511,
+ "36": 0.16508,
+ "37": 0.1652,
+ "38": 0.16527,
+ "39": 0.16626,
+ "40": 0.16583,
+ "41": 0.81579,
+ "42": 0.1665,
+ "43": 0.16683,
+ "44": 0.16836,
+ "45": 0.16702,
+ "46": 0.1654,
+ "47": 0.16533,
+ "48": 0.16527,
+ "49": 0.16499,
+ "50": 0.16589
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_native_fp8_tp_pp_sp/model_config.yaml b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp2_vpp2_cp1_dgx_a100_1N8G/model_config.yaml
similarity index 54%
rename from tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_native_fp8_tp_pp_sp/model_config.yaml
rename to tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp2_vpp2_cp1_dgx_a100_1N8G/model_config.yaml
index a5903e51b6c..51492f98c6e 100644
--- a/tests/functional_tests/test_cases/gpt/gpt3_weekly_dgx_h100_mcore_tp2_pp2_native_fp8_tp_pp_sp/model_config.yaml
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp2_vpp2_cp1_dgx_a100_1N8G/model_config.yaml
@@ -1,30 +1,32 @@
ENV_VARS:
CUDA_DEVICE_MAX_CONNECTIONS: 1
- NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
- NVTE_FUSED_ATTN: 0
+ NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1
NCCL_ALGO: Ring
CUBLAS_WORKSPACE_CONFIG: :4096:8
MODEL_ARGS:
- --num-layers: 12
- --hidden-size: 512
- --num-attention-heads: 8
+ --hidden-size: 1024
+ --num-attention-heads: 16
+ --group-query-attention: true
+ --num-query-groups: 8
+ --hybrid-layer-pattern: M-M-M-M*-M-|M-M-M*-M-M-|M-M*-M-M-M-|M*-M-M-M-M-
+ --spec: "[megatron.core.models.mamba.mamba_layer_specs mamba_stack_spec]"
--log-params-norm: true
--log-num-zeros-in-grad: true
--log-validation-ppl-to-tensorboard: true
--log-timers-to-tensorboard: true
--tensorboard-dir: ${TENSORBOARD_PATH}
- --micro-batch-size: 2
- --global-batch-size: 128
+ --micro-batch-size: 4
+ --global-batch-size: 32
--seq-length: 1024
--max-position-embeddings: 1024
- --train-iters: 2000
+ --train-iters: 50
--timing-log-level: 0
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -38,21 +40,22 @@ MODEL_ARGS:
--eval-interval: 1000
--eval-iters: 10
--transformer-impl: transformer_engine
- --tensor-model-parallel-size: 2
+ --tensor-model-parallel-size: 1
--pipeline-model-parallel-size: 2
- --sequence-parallel: true
- --deterministic-mode: true
- --no-gradient-accumulation-fusion: true
- --fp8-format: hybrid
- --fp8-amax-history-len: 1024
- --fp8-amax-compute-algo: max
- --fp8-param-gather: true
--use-distributed-optimizer: true
+ --overlap-grad-reduce: true
+ --overlap-param-gather: true
+ --check-weight-hash-across-dp-replicas-interval: 10
+ --ckpt-fully-parallel-load: true
+ --no-gradient-accumulation-fusion: true
--attention-softmax-in-fp32: true
+ --use-mcore-models: true
--ckpt-format: torch_dist
--dist-ckpt-optim-fully-reshardable: true
- --dist-ckpt-strictness: log_all # backward compatibility for TE changes
--data-cache-path: ${DATA_CACHE_PATH}
--bf16: true
+ --attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/golden_values_dev_dgx_a100.json b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/golden_values_dev_dgx_a100.json
index db5414bfb90..fe4e1b47237 100644
--- a/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/golden_values_dev_dgx_a100.json
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/golden_values_dev_dgx_a100.json
@@ -1 +1,287 @@
-{"lm loss": {"start_step": 1, "end_step": 50, "step_interval": 5, "values": {"1": 10.81478, "5": 10.8517, "10": 10.78749, "15": 10.79506, "20": 10.69119, "25": 10.52293, "30": 10.34604, "35": 10.26168, "40": 10.07199, "45": 9.8098, "50": 9.88336}}, "num-zeros": {"start_step": 1, "end_step": 50, "step_interval": 5, "values": {"1": 1549.0, "5": 1915.0, "10": 1391.0, "15": 1773.0, "20": 1615.0, "25": 1748.0, "30": 1877.0, "35": 1915.0, "40": 2111.0, "45": 2009.0, "50": 2347.0}}, "mem-allocated-bytes": {"start_step": 1, "end_step": 50, "step_interval": 5, "values": {"1": 522846720.0, "5": 522846720.0, "10": 522846720.0, "15": 522846720.0, "20": 522846720.0, "25": 522846720.0, "30": 522846720.0, "35": 522846720.0, "40": 522846720.0, "45": 522846720.0, "50": 522846720.0}}, "mem-max-allocated-bytes": {"start_step": 1, "end_step": 50, "step_interval": 5, "values": {"1": 3768846848.0, "5": 3912608256.0, "10": 3912608256.0, "15": 3912608256.0, "20": 3912608256.0, "25": 3912608256.0, "30": 3912608256.0, "35": 3912608256.0, "40": 3912608256.0, "45": 3912608256.0, "50": 3912608256.0}}, "iteration-time": {"start_step": 1, "end_step": 50, "step_interval": 5, "values": {"1": 14.36782, "5": 0.18832, "10": 0.16735, "15": 0.16595, "20": 0.16466, "25": 0.16564, "30": 0.16594, "35": 0.16362, "40": 0.16524, "45": 0.16382, "50": 0.16329}}}
\ No newline at end of file
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 11.00877,
+ "2": 11.01519,
+ "3": 11.00482,
+ "4": 10.99234,
+ "5": 11.01519,
+ "6": 11.03124,
+ "7": 11.00366,
+ "8": 11.00197,
+ "9": 11.0015,
+ "10": 10.98399,
+ "11": 10.99266,
+ "12": 10.9843,
+ "13": 10.98228,
+ "14": 10.99069,
+ "15": 10.87369,
+ "16": 10.86468,
+ "17": 10.83441,
+ "18": 10.84799,
+ "19": 10.82979,
+ "20": 10.65983,
+ "21": 10.60262,
+ "22": 10.37409,
+ "23": 10.61651,
+ "24": 10.36412,
+ "25": 10.25993,
+ "26": 10.37776,
+ "27": 10.38284,
+ "28": 10.35243,
+ "29": 10.36265,
+ "30": 9.90501,
+ "31": 9.48621,
+ "32": 10.08722,
+ "33": 10.07604,
+ "34": 9.64526,
+ "35": 9.69425,
+ "36": 9.57868,
+ "37": 9.80085,
+ "38": 9.52328,
+ "39": 9.92115,
+ "40": 9.33512,
+ "41": 9.49131,
+ "42": 9.56855,
+ "43": 9.03905,
+ "44": 9.15098,
+ "45": 8.99463,
+ "46": 9.06041,
+ "47": 9.48005,
+ "48": 9.03809,
+ "49": 8.58598,
+ "50": 9.11529
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 21267.0,
+ "2": 21884.0,
+ "3": 21326.0,
+ "4": 20950.0,
+ "5": 23623.0,
+ "6": 23932.0,
+ "7": 23495.0,
+ "8": 21753.0,
+ "9": 22899.0,
+ "10": 19373.0,
+ "11": 24898.0,
+ "12": 23215.0,
+ "13": 24409.0,
+ "14": 24364.0,
+ "15": 23531.0,
+ "16": 23774.0,
+ "17": 22229.0,
+ "18": 22401.0,
+ "19": 23408.0,
+ "20": 21373.0,
+ "21": 22326.0,
+ "22": 19058.0,
+ "23": 24204.0,
+ "24": 19277.0,
+ "25": 19016.0,
+ "26": 20631.0,
+ "27": 21847.0,
+ "28": 23190.0,
+ "29": 22742.0,
+ "30": 19683.0,
+ "31": 16624.0,
+ "32": 21448.0,
+ "33": 22649.0,
+ "34": 20897.0,
+ "35": 21541.0,
+ "36": 20787.0,
+ "37": 22503.0,
+ "38": 22392.0,
+ "39": 22121.0,
+ "40": 23558.0,
+ "41": 23430.0,
+ "42": 23131.0,
+ "43": 22389.0,
+ "44": 22413.0,
+ "45": 22360.0,
+ "46": 23710.0,
+ "47": 25110.0,
+ "48": 25559.0,
+ "49": 25440.0,
+ "50": 28269.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 1784293376.0,
+ "2": 1784293376.0,
+ "3": 1784293376.0,
+ "4": 1784293376.0,
+ "5": 1784293376.0,
+ "6": 1784293376.0,
+ "7": 1784293376.0,
+ "8": 1784293376.0,
+ "9": 1784293376.0,
+ "10": 1784293376.0,
+ "11": 1784293376.0,
+ "12": 1784293376.0,
+ "13": 1784293376.0,
+ "14": 1784293376.0,
+ "15": 1784293376.0,
+ "16": 1784293376.0,
+ "17": 1784293376.0,
+ "18": 1784293376.0,
+ "19": 1784293376.0,
+ "20": 1784293376.0,
+ "21": 1784293376.0,
+ "22": 1784293376.0,
+ "23": 1784293376.0,
+ "24": 1784293376.0,
+ "25": 1784293376.0,
+ "26": 1784293376.0,
+ "27": 1784293376.0,
+ "28": 1784293376.0,
+ "29": 1784293376.0,
+ "30": 1784293376.0,
+ "31": 1784293376.0,
+ "32": 1784293376.0,
+ "33": 1784293376.0,
+ "34": 1784293376.0,
+ "35": 1784293376.0,
+ "36": 1784293376.0,
+ "37": 1784293376.0,
+ "38": 1784293376.0,
+ "39": 1784293376.0,
+ "40": 1784293376.0,
+ "41": 1784293376.0,
+ "42": 1784293376.0,
+ "43": 1784293376.0,
+ "44": 1784293376.0,
+ "45": 1784293376.0,
+ "46": 1784293376.0,
+ "47": 1784293376.0,
+ "48": 1784293376.0,
+ "49": 1784293376.0,
+ "50": 1784293376.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 3927581184.0,
+ "2": 4483710976.0,
+ "3": 4483712000.0,
+ "4": 4483712000.0,
+ "5": 4483712000.0,
+ "6": 4483712000.0,
+ "7": 4483712000.0,
+ "8": 4483712000.0,
+ "9": 4483712000.0,
+ "10": 4483712000.0,
+ "11": 4483712000.0,
+ "12": 4483712000.0,
+ "13": 4483712000.0,
+ "14": 4483712000.0,
+ "15": 4483712000.0,
+ "16": 4483712000.0,
+ "17": 4483712000.0,
+ "18": 4483712000.0,
+ "19": 4483712000.0,
+ "20": 4483712000.0,
+ "21": 4483712000.0,
+ "22": 4483712000.0,
+ "23": 4483712000.0,
+ "24": 4483712000.0,
+ "25": 4483712000.0,
+ "26": 4483712000.0,
+ "27": 4483712000.0,
+ "28": 4483712000.0,
+ "29": 4483712000.0,
+ "30": 4483712000.0,
+ "31": 4483712000.0,
+ "32": 4483712000.0,
+ "33": 4483712000.0,
+ "34": 4483712000.0,
+ "35": 4483712000.0,
+ "36": 4483712000.0,
+ "37": 4483712000.0,
+ "38": 4483712000.0,
+ "39": 4483712000.0,
+ "40": 4483712000.0,
+ "41": 4483712000.0,
+ "42": 4483712000.0,
+ "43": 4483712000.0,
+ "44": 4483712000.0,
+ "45": 4483712000.0,
+ "46": 4483712000.0,
+ "47": 4483712000.0,
+ "48": 4483712000.0,
+ "49": 4483712000.0,
+ "50": 4483712000.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": "nan",
+ "2": 132.6674,
+ "3": 0.27848,
+ "4": 0.26954,
+ "5": 0.26637,
+ "6": 0.26739,
+ "7": 0.26532,
+ "8": 0.25707,
+ "9": 0.25629,
+ "10": 0.25181,
+ "11": 0.67139,
+ "12": 0.24953,
+ "13": 0.25118,
+ "14": 0.24964,
+ "15": 0.24974,
+ "16": 0.25107,
+ "17": 0.25047,
+ "18": 0.24929,
+ "19": 0.24953,
+ "20": 0.24912,
+ "21": 0.65954,
+ "22": 0.24963,
+ "23": 0.24904,
+ "24": 0.24833,
+ "25": 0.24817,
+ "26": 0.24791,
+ "27": 0.2476,
+ "28": 0.25156,
+ "29": 0.24992,
+ "30": 0.24744,
+ "31": 0.66249,
+ "32": 0.24825,
+ "33": 0.24942,
+ "34": 0.24992,
+ "35": 0.24883,
+ "36": 0.24938,
+ "37": 0.24961,
+ "38": 0.25008,
+ "39": 0.24859,
+ "40": 0.24809,
+ "41": 0.65959,
+ "42": 0.24801,
+ "43": 0.24803,
+ "44": 0.24795,
+ "45": 0.24849,
+ "46": 0.25118,
+ "47": 0.24896,
+ "48": 0.24909,
+ "49": 0.24926,
+ "50": 0.24903
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/golden_values_dev_dgx_h100.json
new file mode 100644
index 00000000000..fe4e1b47237
--- /dev/null
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/golden_values_dev_dgx_h100.json
@@ -0,0 +1,287 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 11.00877,
+ "2": 11.01519,
+ "3": 11.00482,
+ "4": 10.99234,
+ "5": 11.01519,
+ "6": 11.03124,
+ "7": 11.00366,
+ "8": 11.00197,
+ "9": 11.0015,
+ "10": 10.98399,
+ "11": 10.99266,
+ "12": 10.9843,
+ "13": 10.98228,
+ "14": 10.99069,
+ "15": 10.87369,
+ "16": 10.86468,
+ "17": 10.83441,
+ "18": 10.84799,
+ "19": 10.82979,
+ "20": 10.65983,
+ "21": 10.60262,
+ "22": 10.37409,
+ "23": 10.61651,
+ "24": 10.36412,
+ "25": 10.25993,
+ "26": 10.37776,
+ "27": 10.38284,
+ "28": 10.35243,
+ "29": 10.36265,
+ "30": 9.90501,
+ "31": 9.48621,
+ "32": 10.08722,
+ "33": 10.07604,
+ "34": 9.64526,
+ "35": 9.69425,
+ "36": 9.57868,
+ "37": 9.80085,
+ "38": 9.52328,
+ "39": 9.92115,
+ "40": 9.33512,
+ "41": 9.49131,
+ "42": 9.56855,
+ "43": 9.03905,
+ "44": 9.15098,
+ "45": 8.99463,
+ "46": 9.06041,
+ "47": 9.48005,
+ "48": 9.03809,
+ "49": 8.58598,
+ "50": 9.11529
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 21267.0,
+ "2": 21884.0,
+ "3": 21326.0,
+ "4": 20950.0,
+ "5": 23623.0,
+ "6": 23932.0,
+ "7": 23495.0,
+ "8": 21753.0,
+ "9": 22899.0,
+ "10": 19373.0,
+ "11": 24898.0,
+ "12": 23215.0,
+ "13": 24409.0,
+ "14": 24364.0,
+ "15": 23531.0,
+ "16": 23774.0,
+ "17": 22229.0,
+ "18": 22401.0,
+ "19": 23408.0,
+ "20": 21373.0,
+ "21": 22326.0,
+ "22": 19058.0,
+ "23": 24204.0,
+ "24": 19277.0,
+ "25": 19016.0,
+ "26": 20631.0,
+ "27": 21847.0,
+ "28": 23190.0,
+ "29": 22742.0,
+ "30": 19683.0,
+ "31": 16624.0,
+ "32": 21448.0,
+ "33": 22649.0,
+ "34": 20897.0,
+ "35": 21541.0,
+ "36": 20787.0,
+ "37": 22503.0,
+ "38": 22392.0,
+ "39": 22121.0,
+ "40": 23558.0,
+ "41": 23430.0,
+ "42": 23131.0,
+ "43": 22389.0,
+ "44": 22413.0,
+ "45": 22360.0,
+ "46": 23710.0,
+ "47": 25110.0,
+ "48": 25559.0,
+ "49": 25440.0,
+ "50": 28269.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 1784293376.0,
+ "2": 1784293376.0,
+ "3": 1784293376.0,
+ "4": 1784293376.0,
+ "5": 1784293376.0,
+ "6": 1784293376.0,
+ "7": 1784293376.0,
+ "8": 1784293376.0,
+ "9": 1784293376.0,
+ "10": 1784293376.0,
+ "11": 1784293376.0,
+ "12": 1784293376.0,
+ "13": 1784293376.0,
+ "14": 1784293376.0,
+ "15": 1784293376.0,
+ "16": 1784293376.0,
+ "17": 1784293376.0,
+ "18": 1784293376.0,
+ "19": 1784293376.0,
+ "20": 1784293376.0,
+ "21": 1784293376.0,
+ "22": 1784293376.0,
+ "23": 1784293376.0,
+ "24": 1784293376.0,
+ "25": 1784293376.0,
+ "26": 1784293376.0,
+ "27": 1784293376.0,
+ "28": 1784293376.0,
+ "29": 1784293376.0,
+ "30": 1784293376.0,
+ "31": 1784293376.0,
+ "32": 1784293376.0,
+ "33": 1784293376.0,
+ "34": 1784293376.0,
+ "35": 1784293376.0,
+ "36": 1784293376.0,
+ "37": 1784293376.0,
+ "38": 1784293376.0,
+ "39": 1784293376.0,
+ "40": 1784293376.0,
+ "41": 1784293376.0,
+ "42": 1784293376.0,
+ "43": 1784293376.0,
+ "44": 1784293376.0,
+ "45": 1784293376.0,
+ "46": 1784293376.0,
+ "47": 1784293376.0,
+ "48": 1784293376.0,
+ "49": 1784293376.0,
+ "50": 1784293376.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 3927581184.0,
+ "2": 4483710976.0,
+ "3": 4483712000.0,
+ "4": 4483712000.0,
+ "5": 4483712000.0,
+ "6": 4483712000.0,
+ "7": 4483712000.0,
+ "8": 4483712000.0,
+ "9": 4483712000.0,
+ "10": 4483712000.0,
+ "11": 4483712000.0,
+ "12": 4483712000.0,
+ "13": 4483712000.0,
+ "14": 4483712000.0,
+ "15": 4483712000.0,
+ "16": 4483712000.0,
+ "17": 4483712000.0,
+ "18": 4483712000.0,
+ "19": 4483712000.0,
+ "20": 4483712000.0,
+ "21": 4483712000.0,
+ "22": 4483712000.0,
+ "23": 4483712000.0,
+ "24": 4483712000.0,
+ "25": 4483712000.0,
+ "26": 4483712000.0,
+ "27": 4483712000.0,
+ "28": 4483712000.0,
+ "29": 4483712000.0,
+ "30": 4483712000.0,
+ "31": 4483712000.0,
+ "32": 4483712000.0,
+ "33": 4483712000.0,
+ "34": 4483712000.0,
+ "35": 4483712000.0,
+ "36": 4483712000.0,
+ "37": 4483712000.0,
+ "38": 4483712000.0,
+ "39": 4483712000.0,
+ "40": 4483712000.0,
+ "41": 4483712000.0,
+ "42": 4483712000.0,
+ "43": 4483712000.0,
+ "44": 4483712000.0,
+ "45": 4483712000.0,
+ "46": 4483712000.0,
+ "47": 4483712000.0,
+ "48": 4483712000.0,
+ "49": 4483712000.0,
+ "50": 4483712000.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": "nan",
+ "2": 132.6674,
+ "3": 0.27848,
+ "4": 0.26954,
+ "5": 0.26637,
+ "6": 0.26739,
+ "7": 0.26532,
+ "8": 0.25707,
+ "9": 0.25629,
+ "10": 0.25181,
+ "11": 0.67139,
+ "12": 0.24953,
+ "13": 0.25118,
+ "14": 0.24964,
+ "15": 0.24974,
+ "16": 0.25107,
+ "17": 0.25047,
+ "18": 0.24929,
+ "19": 0.24953,
+ "20": 0.24912,
+ "21": 0.65954,
+ "22": 0.24963,
+ "23": 0.24904,
+ "24": 0.24833,
+ "25": 0.24817,
+ "26": 0.24791,
+ "27": 0.2476,
+ "28": 0.25156,
+ "29": 0.24992,
+ "30": 0.24744,
+ "31": 0.66249,
+ "32": 0.24825,
+ "33": 0.24942,
+ "34": 0.24992,
+ "35": 0.24883,
+ "36": 0.24938,
+ "37": 0.24961,
+ "38": 0.25008,
+ "39": 0.24859,
+ "40": 0.24809,
+ "41": 0.65959,
+ "42": 0.24801,
+ "43": 0.24803,
+ "44": 0.24795,
+ "45": 0.24849,
+ "46": 0.25118,
+ "47": 0.24896,
+ "48": 0.24909,
+ "49": 0.24926,
+ "50": 0.24903
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/golden_values_lts_dgx_a100.json b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/golden_values_lts_dgx_a100.json
index dc393d0dffc..fe4e1b47237 100644
--- a/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/golden_values_lts_dgx_a100.json
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/golden_values_lts_dgx_a100.json
@@ -1 +1,287 @@
-{"lm loss": {"start_step": 1, "end_step": 50, "step_interval": 5, "values": {"1": 10.82005, "5": 10.85284, "10": 10.78455, "15": 10.7923, "20": 10.69213, "25": 10.5241, "30": 10.34556, "35": 10.26241, "40": 10.07237, "45": 9.811, "50": 9.88419}}, "num-zeros": {"start_step": 1, "end_step": 50, "step_interval": 5, "values": {"1": 1559.0, "5": 1840.0, "10": 1380.0, "15": 1850.0, "20": 1699.0, "25": 1614.0, "30": 1905.0, "35": 1933.0, "40": 2169.0, "45": 2101.0, "50": 2421.0}}, "mem-allocated-bytes": {"start_step": 1, "end_step": 50, "step_interval": 5, "values": {"1": 523004928.0, "5": 523004928.0, "10": 523004928.0, "15": 523004928.0, "20": 523004928.0, "25": 523004928.0, "30": 523004928.0, "35": 523004928.0, "40": 523004928.0, "45": 523004928.0, "50": 523004928.0}}, "mem-max-allocated-bytes": {"start_step": 1, "end_step": 50, "step_interval": 5, "values": {"1": 3768873984.0, "5": 3912766464.0, "10": 3912766464.0, "15": 3912766464.0, "20": 3912766464.0, "25": 3912766464.0, "30": 3912766464.0, "35": 3912766464.0, "40": 3912766464.0, "45": 3912766464.0, "50": 3912766464.0}}, "iteration-time": {"start_step": 1, "end_step": 50, "step_interval": 5, "values": {"1": 18.88705, "5": 0.16956, "10": 0.17448, "15": 0.16853, "20": 0.1715, "25": 0.17071, "30": 0.17343, "35": 0.17213, "40": 0.1719, "45": 0.17357, "50": 0.17228}}}
\ No newline at end of file
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 11.00877,
+ "2": 11.01519,
+ "3": 11.00482,
+ "4": 10.99234,
+ "5": 11.01519,
+ "6": 11.03124,
+ "7": 11.00366,
+ "8": 11.00197,
+ "9": 11.0015,
+ "10": 10.98399,
+ "11": 10.99266,
+ "12": 10.9843,
+ "13": 10.98228,
+ "14": 10.99069,
+ "15": 10.87369,
+ "16": 10.86468,
+ "17": 10.83441,
+ "18": 10.84799,
+ "19": 10.82979,
+ "20": 10.65983,
+ "21": 10.60262,
+ "22": 10.37409,
+ "23": 10.61651,
+ "24": 10.36412,
+ "25": 10.25993,
+ "26": 10.37776,
+ "27": 10.38284,
+ "28": 10.35243,
+ "29": 10.36265,
+ "30": 9.90501,
+ "31": 9.48621,
+ "32": 10.08722,
+ "33": 10.07604,
+ "34": 9.64526,
+ "35": 9.69425,
+ "36": 9.57868,
+ "37": 9.80085,
+ "38": 9.52328,
+ "39": 9.92115,
+ "40": 9.33512,
+ "41": 9.49131,
+ "42": 9.56855,
+ "43": 9.03905,
+ "44": 9.15098,
+ "45": 8.99463,
+ "46": 9.06041,
+ "47": 9.48005,
+ "48": 9.03809,
+ "49": 8.58598,
+ "50": 9.11529
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 21267.0,
+ "2": 21884.0,
+ "3": 21326.0,
+ "4": 20950.0,
+ "5": 23623.0,
+ "6": 23932.0,
+ "7": 23495.0,
+ "8": 21753.0,
+ "9": 22899.0,
+ "10": 19373.0,
+ "11": 24898.0,
+ "12": 23215.0,
+ "13": 24409.0,
+ "14": 24364.0,
+ "15": 23531.0,
+ "16": 23774.0,
+ "17": 22229.0,
+ "18": 22401.0,
+ "19": 23408.0,
+ "20": 21373.0,
+ "21": 22326.0,
+ "22": 19058.0,
+ "23": 24204.0,
+ "24": 19277.0,
+ "25": 19016.0,
+ "26": 20631.0,
+ "27": 21847.0,
+ "28": 23190.0,
+ "29": 22742.0,
+ "30": 19683.0,
+ "31": 16624.0,
+ "32": 21448.0,
+ "33": 22649.0,
+ "34": 20897.0,
+ "35": 21541.0,
+ "36": 20787.0,
+ "37": 22503.0,
+ "38": 22392.0,
+ "39": 22121.0,
+ "40": 23558.0,
+ "41": 23430.0,
+ "42": 23131.0,
+ "43": 22389.0,
+ "44": 22413.0,
+ "45": 22360.0,
+ "46": 23710.0,
+ "47": 25110.0,
+ "48": 25559.0,
+ "49": 25440.0,
+ "50": 28269.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 1784293376.0,
+ "2": 1784293376.0,
+ "3": 1784293376.0,
+ "4": 1784293376.0,
+ "5": 1784293376.0,
+ "6": 1784293376.0,
+ "7": 1784293376.0,
+ "8": 1784293376.0,
+ "9": 1784293376.0,
+ "10": 1784293376.0,
+ "11": 1784293376.0,
+ "12": 1784293376.0,
+ "13": 1784293376.0,
+ "14": 1784293376.0,
+ "15": 1784293376.0,
+ "16": 1784293376.0,
+ "17": 1784293376.0,
+ "18": 1784293376.0,
+ "19": 1784293376.0,
+ "20": 1784293376.0,
+ "21": 1784293376.0,
+ "22": 1784293376.0,
+ "23": 1784293376.0,
+ "24": 1784293376.0,
+ "25": 1784293376.0,
+ "26": 1784293376.0,
+ "27": 1784293376.0,
+ "28": 1784293376.0,
+ "29": 1784293376.0,
+ "30": 1784293376.0,
+ "31": 1784293376.0,
+ "32": 1784293376.0,
+ "33": 1784293376.0,
+ "34": 1784293376.0,
+ "35": 1784293376.0,
+ "36": 1784293376.0,
+ "37": 1784293376.0,
+ "38": 1784293376.0,
+ "39": 1784293376.0,
+ "40": 1784293376.0,
+ "41": 1784293376.0,
+ "42": 1784293376.0,
+ "43": 1784293376.0,
+ "44": 1784293376.0,
+ "45": 1784293376.0,
+ "46": 1784293376.0,
+ "47": 1784293376.0,
+ "48": 1784293376.0,
+ "49": 1784293376.0,
+ "50": 1784293376.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 3927581184.0,
+ "2": 4483710976.0,
+ "3": 4483712000.0,
+ "4": 4483712000.0,
+ "5": 4483712000.0,
+ "6": 4483712000.0,
+ "7": 4483712000.0,
+ "8": 4483712000.0,
+ "9": 4483712000.0,
+ "10": 4483712000.0,
+ "11": 4483712000.0,
+ "12": 4483712000.0,
+ "13": 4483712000.0,
+ "14": 4483712000.0,
+ "15": 4483712000.0,
+ "16": 4483712000.0,
+ "17": 4483712000.0,
+ "18": 4483712000.0,
+ "19": 4483712000.0,
+ "20": 4483712000.0,
+ "21": 4483712000.0,
+ "22": 4483712000.0,
+ "23": 4483712000.0,
+ "24": 4483712000.0,
+ "25": 4483712000.0,
+ "26": 4483712000.0,
+ "27": 4483712000.0,
+ "28": 4483712000.0,
+ "29": 4483712000.0,
+ "30": 4483712000.0,
+ "31": 4483712000.0,
+ "32": 4483712000.0,
+ "33": 4483712000.0,
+ "34": 4483712000.0,
+ "35": 4483712000.0,
+ "36": 4483712000.0,
+ "37": 4483712000.0,
+ "38": 4483712000.0,
+ "39": 4483712000.0,
+ "40": 4483712000.0,
+ "41": 4483712000.0,
+ "42": 4483712000.0,
+ "43": 4483712000.0,
+ "44": 4483712000.0,
+ "45": 4483712000.0,
+ "46": 4483712000.0,
+ "47": 4483712000.0,
+ "48": 4483712000.0,
+ "49": 4483712000.0,
+ "50": 4483712000.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": "nan",
+ "2": 132.6674,
+ "3": 0.27848,
+ "4": 0.26954,
+ "5": 0.26637,
+ "6": 0.26739,
+ "7": 0.26532,
+ "8": 0.25707,
+ "9": 0.25629,
+ "10": 0.25181,
+ "11": 0.67139,
+ "12": 0.24953,
+ "13": 0.25118,
+ "14": 0.24964,
+ "15": 0.24974,
+ "16": 0.25107,
+ "17": 0.25047,
+ "18": 0.24929,
+ "19": 0.24953,
+ "20": 0.24912,
+ "21": 0.65954,
+ "22": 0.24963,
+ "23": 0.24904,
+ "24": 0.24833,
+ "25": 0.24817,
+ "26": 0.24791,
+ "27": 0.2476,
+ "28": 0.25156,
+ "29": 0.24992,
+ "30": 0.24744,
+ "31": 0.66249,
+ "32": 0.24825,
+ "33": 0.24942,
+ "34": 0.24992,
+ "35": 0.24883,
+ "36": 0.24938,
+ "37": 0.24961,
+ "38": 0.25008,
+ "39": 0.24859,
+ "40": 0.24809,
+ "41": 0.65959,
+ "42": 0.24801,
+ "43": 0.24803,
+ "44": 0.24795,
+ "45": 0.24849,
+ "46": 0.25118,
+ "47": 0.24896,
+ "48": 0.24909,
+ "49": 0.24926,
+ "50": 0.24903
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/model_config.yaml b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/model_config.yaml
index 0983337becc..6eff846884a 100644
--- a/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/model_config.yaml
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G/model_config.yaml
@@ -4,14 +4,12 @@ ENV_VARS:
NCCL_ALGO: Ring
CUBLAS_WORKSPACE_CONFIG: :4096:8
MODEL_ARGS:
- --num-layers: 44
--hidden-size: 1024
--num-attention-heads: 16
--group-query-attention: true
--num-query-groups: 8
- --hybrid-override-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M-
+ --hybrid-layer-pattern: M-M-M-M*-M-|M-M-M*-M-M-|M-M*-M-M-M-|M*-M-M-M-M-
--spec: "[megatron.core.models.mamba.mamba_layer_specs mamba_stack_spec]"
- --is-hybrid-model: true
--log-params-norm: true
--log-num-zeros-in-grad: true
--log-validation-ppl-to-tensorboard: true
@@ -26,9 +24,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -58,4 +56,7 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --async-strategy: mcore
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp2_pp1_cp1_dgx_a100_1N8G/model_config.yaml b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp2_pp1_cp1_dgx_a100_1N8G/model_config.yaml
index 7f7aac5d78b..7e5ca833000 100644
--- a/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp2_pp1_cp1_dgx_a100_1N8G/model_config.yaml
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp2_pp1_cp1_dgx_a100_1N8G/model_config.yaml
@@ -4,14 +4,12 @@ ENV_VARS:
NCCL_ALGO: Ring
CUBLAS_WORKSPACE_CONFIG: :4096:8
MODEL_ARGS:
- --num-layers: 44
--hidden-size: 1024
--num-attention-heads: 16
--group-query-attention: true
--num-query-groups: 8
- --hybrid-override-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M-
+ --hybrid-layer-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M-
--spec: "[megatron.core.models.mamba.mamba_layer_specs mamba_stack_spec]"
- --is-hybrid-model: true
--log-params-norm: true
--log-num-zeros-in-grad: true
--log-validation-ppl-to-tensorboard: true
@@ -26,9 +24,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -57,4 +55,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: unfused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp2_pp1_cp4_dgx_a100_1N8G/model_config.yaml b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp2_pp1_cp4_dgx_a100_1N8G/model_config.yaml
index 93418f580fc..44b588ee140 100644
--- a/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp2_pp1_cp4_dgx_a100_1N8G/model_config.yaml
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_mr_mcore_te_tp2_pp1_cp4_dgx_a100_1N8G/model_config.yaml
@@ -4,14 +4,12 @@ ENV_VARS:
NCCL_ALGO: Ring
CUBLAS_WORKSPACE_CONFIG: :4096:8
MODEL_ARGS:
- --num-layers: 44
--hidden-size: 1024
--num-attention-heads: 16
--group-query-attention: true
--num-query-groups: 8
- --hybrid-override-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M-
+ --hybrid-layer-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M*-M-M-M-M-
--spec: "[megatron.core.models.mamba.mamba_layer_specs mamba_stack_spec]"
- --is-hybrid-model: true
--log-params-norm: true
--log-num-zeros-in-grad: true
--log-validation-ppl-to-tensorboard: true
@@ -26,9 +24,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -59,4 +57,6 @@ MODEL_ARGS:
--bf16: true
--attention-backend: fused
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_static_inference_tp1_pp1_2B_cudagraphs/model_config.yaml b/tests/functional_tests/test_cases/hybrid/hybrid_static_inference_tp1_pp1_2B_cudagraphs/model_config.yaml
index 5bc40afede4..26708b32a60 100644
--- a/tests/functional_tests/test_cases/hybrid/hybrid_static_inference_tp1_pp1_2B_cudagraphs/model_config.yaml
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_static_inference_tp1_pp1_2B_cudagraphs/model_config.yaml
@@ -22,19 +22,17 @@ MODEL_ARGS:
--pipeline-model-parallel-size: 1
--expert-model-parallel-size: 1
--use-mcore-models: true
- --is-hybrid-model: true
--model-provider: mamba
--init-method-std: 0.0198
--untie-embeddings-and-output-weights: true
--disable-bias-linear: true
--init-method-std: 0.014
--position-embedding-type: none
- --num-layers: 50
--hidden-size: 2048
--ffn-hidden-size: 11264
--num-attention-heads: 16
--kv-channels: 128
- --hybrid-override-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-
+ --hybrid-layer-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-
--spec: megatron.core.models.mamba.mamba_layer_specs mamba_stack_spec
--normalization: RMSNorm
--swiglu: true
diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_static_inference_tp1_pp1_2B_logitsmatch/model_config.yaml b/tests/functional_tests/test_cases/hybrid/hybrid_static_inference_tp1_pp1_2B_logitsmatch/model_config.yaml
index b5c3c409605..3964bcb8ecb 100644
--- a/tests/functional_tests/test_cases/hybrid/hybrid_static_inference_tp1_pp1_2B_logitsmatch/model_config.yaml
+++ b/tests/functional_tests/test_cases/hybrid/hybrid_static_inference_tp1_pp1_2B_logitsmatch/model_config.yaml
@@ -22,19 +22,17 @@ MODEL_ARGS:
--pipeline-model-parallel-size: 1
--expert-model-parallel-size: 1
--use-mcore-models: true
- --is-hybrid-model: true
--model-provider: mamba
--init-method-std: 0.0198
--untie-embeddings-and-output-weights: true
--disable-bias-linear: true
--init-method-std: 0.014
--position-embedding-type: none
- --num-layers: 50
--hidden-size: 2048
--ffn-hidden-size: 11264
--num-attention-heads: 16
--kv-channels: 128
- --hybrid-override-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-
+ --hybrid-layer-pattern: M-M-M-M*-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M*-M-M-M-M-M-
--spec: megatron.core.models.mamba.mamba_layer_specs mamba_stack_spec
--normalization: RMSNorm
--swiglu: true
diff --git a/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp1_dp8/model_config.yaml b/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp1_dp8/model_config.yaml
index 2daf74b89a7..e95856e7308 100644
--- a/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp1_dp8/model_config.yaml
+++ b/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp1_dp8/model_config.yaml
@@ -25,7 +25,6 @@ MODEL_ARGS:
--load: ${CHECKPOINT_LOAD_PATH}
--tokenizer-type: HuggingFaceTokenizer
--tokenizer-model: llava-hf/llava-1.5-7b-hf
- --legacy-tokenizer: true
--distributed-backend: nccl
--lr: 0.001
--lr-decay-style: cosine
diff --git a/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp1_dp8_seq_packing/golden_values_dev.json b/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp1_dp8_seq_packing/golden_values_dev.json
new file mode 100644
index 00000000000..00ba74777b4
--- /dev/null
+++ b/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp1_dp8_seq_packing/golden_values_dev.json
@@ -0,0 +1,142 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 5,
+ "values": {
+ "1": 7.75485,
+ "5": 5.94659,
+ "10": 4.99645,
+ "15": 4.45849,
+ "20": 4.15285,
+ "25": 3.95535,
+ "30": 3.48175,
+ "35": 3.54835,
+ "40": 3.39398,
+ "45": 3.41881,
+ "50": 3.38677,
+ "55": 3.37007,
+ "60": 3.26995,
+ "65": 3.38823,
+ "70": 3.37181,
+ "75": 3.3364,
+ "80": 3.27803,
+ "85": 3.33499,
+ "90": 3.30274,
+ "95": 3.20702,
+ "100": 3.35925
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 5,
+ "values": {
+ "1": 0.0,
+ "5": 1.0,
+ "10": 1.0,
+ "15": 0.0,
+ "20": 0.0,
+ "25": 0.0,
+ "30": 0.0,
+ "35": 0.0,
+ "40": 0.0,
+ "45": 0.0,
+ "50": 0.0,
+ "55": 0.0,
+ "60": 0.0,
+ "65": 0.0,
+ "70": 0.0,
+ "75": 0.0,
+ "80": 0.0,
+ "85": 0.0,
+ "90": 0.0,
+ "95": 2.0,
+ "100": 1.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 5,
+ "values": {
+ "1": 28620292096.0,
+ "5": 28620292096.0,
+ "10": 28620292096.0,
+ "15": 28620292096.0,
+ "20": 28620292096.0,
+ "25": 28620292096.0,
+ "30": 28620292096.0,
+ "35": 28620292096.0,
+ "40": 28620292096.0,
+ "45": 28620292096.0,
+ "50": 28620292096.0,
+ "55": 28620292096.0,
+ "60": 28620292096.0,
+ "65": 28620292096.0,
+ "70": 28620292096.0,
+ "75": 28620292096.0,
+ "80": 28620292096.0,
+ "85": 28620292096.0,
+ "90": 28620292096.0,
+ "95": 28620292096.0,
+ "100": 28620292096.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 5,
+ "values": {
+ "1": 71807533056.0,
+ "5": 71975370752.0,
+ "10": 71975370752.0,
+ "15": 71975370752.0,
+ "20": 71975370752.0,
+ "25": 71975370752.0,
+ "30": 71975370752.0,
+ "35": 71975370752.0,
+ "40": 71975370752.0,
+ "45": 71975370752.0,
+ "50": 71975370752.0,
+ "55": 71975370752.0,
+ "60": 71975370752.0,
+ "65": 71975370752.0,
+ "70": 71975370752.0,
+ "75": 71975370752.0,
+ "80": 71975370752.0,
+ "85": 71975370752.0,
+ "90": 71975370752.0,
+ "95": 71975370752.0,
+ "100": 71975370752.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 5,
+ "values": {
+ "1": 10.99852,
+ "5": 2.41794,
+ "10": 2.42468,
+ "15": 2.42206,
+ "20": 2.42207,
+ "25": 2.41474,
+ "30": 2.41753,
+ "35": 2.42371,
+ "40": 2.42615,
+ "45": 2.4149,
+ "50": 2.41111,
+ "55": 2.41683,
+ "60": 2.41605,
+ "65": 2.415,
+ "70": 2.41618,
+ "75": 2.41691,
+ "80": 2.41523,
+ "85": 2.41522,
+ "90": 2.41486,
+ "95": 2.41373,
+ "100": 2.41716
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp1_dp8_seq_packing/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp1_dp8_seq_packing/golden_values_dev_dgx_h100.json
new file mode 100644
index 00000000000..581ae01b6a3
--- /dev/null
+++ b/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp1_dp8_seq_packing/golden_values_dev_dgx_h100.json
@@ -0,0 +1,537 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 1,
+ "values": {
+ "1": 7.74249,
+ "2": 7.94009,
+ "3": 7.64718,
+ "4": 7.11692,
+ "5": 6.09843,
+ "6": 5.8091,
+ "7": 5.59573,
+ "8": 5.39367,
+ "9": 5.12575,
+ "10": 5.09298,
+ "11": 4.8586,
+ "12": 4.76892,
+ "13": 4.68905,
+ "14": 4.57558,
+ "15": 4.49718,
+ "16": 4.46151,
+ "17": 4.33643,
+ "18": 4.28784,
+ "19": 4.22689,
+ "20": 4.12088,
+ "21": 4.15883,
+ "22": 4.17278,
+ "23": 4.01549,
+ "24": 3.78218,
+ "25": 3.84438,
+ "26": 3.63675,
+ "27": 3.60623,
+ "28": 3.78552,
+ "29": 3.73417,
+ "30": 3.4335,
+ "31": 3.60412,
+ "32": 3.43946,
+ "33": 3.37099,
+ "34": 3.43735,
+ "35": 3.51764,
+ "36": 3.32475,
+ "37": 3.41112,
+ "38": 3.3566,
+ "39": 3.46701,
+ "40": 3.3994,
+ "41": 3.36013,
+ "42": 3.50815,
+ "43": 3.49041,
+ "44": 3.33243,
+ "45": 3.40793,
+ "46": 3.36562,
+ "47": 3.37961,
+ "48": 3.30497,
+ "49": 3.36786,
+ "50": 3.3894,
+ "51": 3.45346,
+ "52": 3.24821,
+ "53": 3.29915,
+ "54": 3.43316,
+ "55": 3.36887,
+ "56": 3.49063,
+ "57": 3.35037,
+ "58": 3.27986,
+ "59": 3.36634,
+ "60": 3.25821,
+ "61": 3.30581,
+ "62": 3.32866,
+ "63": 3.4314,
+ "64": 3.34268,
+ "65": 3.39912,
+ "66": 3.31737,
+ "67": 3.11254,
+ "68": 3.41444,
+ "69": 3.33339,
+ "70": 3.36655,
+ "71": 3.26962,
+ "72": 3.22295,
+ "73": 3.29222,
+ "74": 3.34033,
+ "75": 3.34085,
+ "76": 3.28966,
+ "77": 3.38243,
+ "78": 3.3331,
+ "79": 3.40523,
+ "80": 3.27625,
+ "81": 3.36784,
+ "82": 3.31528,
+ "83": 3.1391,
+ "84": 3.33272,
+ "85": 3.33275,
+ "86": 3.19784,
+ "87": 3.31889,
+ "88": 3.24109,
+ "89": 3.28076,
+ "90": 3.28061,
+ "91": 3.31327,
+ "92": 3.03808,
+ "93": 3.2688,
+ "94": 3.40631,
+ "95": 3.20194,
+ "96": 3.36856,
+ "97": 3.32299,
+ "98": 3.28825,
+ "99": 3.43853,
+ "100": 3.34632
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 1,
+ "values": {
+ "1": 54.0,
+ "2": 48.0,
+ "3": 61.0,
+ "4": 78.0,
+ "5": 101.0,
+ "6": 93.0,
+ "7": 106.0,
+ "8": 107.0,
+ "9": 94.0,
+ "10": 123.0,
+ "11": 117.0,
+ "12": 89.0,
+ "13": 187.0,
+ "14": 87.0,
+ "15": 112.0,
+ "16": 180.0,
+ "17": 90.0,
+ "18": 154.0,
+ "19": 93.0,
+ "20": 149.0,
+ "21": 100.0,
+ "22": 139.0,
+ "23": 143.0,
+ "24": 184.0,
+ "25": 166.0,
+ "26": 204.0,
+ "27": 220.0,
+ "28": 152.0,
+ "29": 124.0,
+ "30": 177.0,
+ "31": 162.0,
+ "32": 141.0,
+ "33": 116.0,
+ "34": 207.0,
+ "35": 184.0,
+ "36": 119.0,
+ "37": 239.0,
+ "38": 201.0,
+ "39": 223.0,
+ "40": 138.0,
+ "41": 205.0,
+ "42": 209.0,
+ "43": 196.0,
+ "44": 204.0,
+ "45": 211.0,
+ "46": 217.0,
+ "47": 170.0,
+ "48": 180.0,
+ "49": 185.0,
+ "50": 199.0,
+ "51": 248.0,
+ "52": 193.0,
+ "53": 205.0,
+ "54": 201.0,
+ "55": 214.0,
+ "56": 160.0,
+ "57": 230.0,
+ "58": 170.0,
+ "59": 246.0,
+ "60": 211.0,
+ "61": 200.0,
+ "62": 188.0,
+ "63": 188.0,
+ "64": 215.0,
+ "65": 174.0,
+ "66": 238.0,
+ "67": 197.0,
+ "68": 166.0,
+ "69": 209.0,
+ "70": 211.0,
+ "71": 212.0,
+ "72": 154.0,
+ "73": 214.0,
+ "74": 238.0,
+ "75": 139.0,
+ "76": 222.0,
+ "77": 196.0,
+ "78": 203.0,
+ "79": 185.0,
+ "80": 144.0,
+ "81": 186.0,
+ "82": 272.0,
+ "83": 141.0,
+ "84": 190.0,
+ "85": 231.0,
+ "86": 235.0,
+ "87": 226.0,
+ "88": 250.0,
+ "89": 229.0,
+ "90": 229.0,
+ "91": 212.0,
+ "92": 183.0,
+ "93": 193.0,
+ "94": 197.0,
+ "95": 208.0,
+ "96": 164.0,
+ "97": 219.0,
+ "98": 251.0,
+ "99": 204.0,
+ "100": 222.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 1,
+ "values": {
+ "1": 14569809920.0,
+ "2": 14569809920.0,
+ "3": 14569809920.0,
+ "4": 14569809920.0,
+ "5": 14569809920.0,
+ "6": 14569809920.0,
+ "7": 14569809920.0,
+ "8": 14569809920.0,
+ "9": 14569809920.0,
+ "10": 14569809920.0,
+ "11": 14569809920.0,
+ "12": 14569809920.0,
+ "13": 14569809920.0,
+ "14": 14569809920.0,
+ "15": 14569809920.0,
+ "16": 14569809920.0,
+ "17": 14569809920.0,
+ "18": 14569809920.0,
+ "19": 14569809920.0,
+ "20": 14569809920.0,
+ "21": 14569809920.0,
+ "22": 14569809920.0,
+ "23": 14569809920.0,
+ "24": 14569809920.0,
+ "25": 14569809920.0,
+ "26": 14569809920.0,
+ "27": 14569809920.0,
+ "28": 14569809920.0,
+ "29": 14569809920.0,
+ "30": 14569809920.0,
+ "31": 14569809920.0,
+ "32": 14569809920.0,
+ "33": 14569809920.0,
+ "34": 14569809920.0,
+ "35": 14569809920.0,
+ "36": 14569809920.0,
+ "37": 14569809920.0,
+ "38": 14569809920.0,
+ "39": 14569809920.0,
+ "40": 14569809920.0,
+ "41": 14569809920.0,
+ "42": 14569809920.0,
+ "43": 14569809920.0,
+ "44": 14569809920.0,
+ "45": 14569809920.0,
+ "46": 14569809920.0,
+ "47": 14569809920.0,
+ "48": 14569809920.0,
+ "49": 14569809920.0,
+ "50": 14569809920.0,
+ "51": 14569809920.0,
+ "52": 14569809920.0,
+ "53": 14569809920.0,
+ "54": 14569809920.0,
+ "55": 14569809920.0,
+ "56": 14569809920.0,
+ "57": 14569809920.0,
+ "58": 14569809920.0,
+ "59": 14569809920.0,
+ "60": 14569809920.0,
+ "61": 14569809920.0,
+ "62": 14569809920.0,
+ "63": 14569809920.0,
+ "64": 14569809920.0,
+ "65": 14569809920.0,
+ "66": 14569809920.0,
+ "67": 14569809920.0,
+ "68": 14569809920.0,
+ "69": 14569809920.0,
+ "70": 14569809920.0,
+ "71": 14569809920.0,
+ "72": 14569809920.0,
+ "73": 14569809920.0,
+ "74": 14569809920.0,
+ "75": 14569809920.0,
+ "76": 14569809920.0,
+ "77": 14569809920.0,
+ "78": 14569809920.0,
+ "79": 14569809920.0,
+ "80": 14569809920.0,
+ "81": 14569809920.0,
+ "82": 14569809920.0,
+ "83": 14569809920.0,
+ "84": 14569809920.0,
+ "85": 14569809920.0,
+ "86": 14569809920.0,
+ "87": 14569809920.0,
+ "88": 14569809920.0,
+ "89": 14569809920.0,
+ "90": 14569809920.0,
+ "91": 14569809920.0,
+ "92": 14569809920.0,
+ "93": 14569809920.0,
+ "94": 14569809920.0,
+ "95": 14569809920.0,
+ "96": 14569809920.0,
+ "97": 14569809920.0,
+ "98": 14569809920.0,
+ "99": 14569809920.0,
+ "100": 14569809920.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 1,
+ "values": {
+ "1": 42746941440.0,
+ "2": 42746941440.0,
+ "3": 42746941440.0,
+ "4": 42746941440.0,
+ "5": 42746941440.0,
+ "6": 42746941440.0,
+ "7": 42746941440.0,
+ "8": 42746941440.0,
+ "9": 42746941440.0,
+ "10": 42746941440.0,
+ "11": 42746941440.0,
+ "12": 42746941440.0,
+ "13": 42746941440.0,
+ "14": 42746941440.0,
+ "15": 42746941440.0,
+ "16": 42746941440.0,
+ "17": 42746941440.0,
+ "18": 42746941440.0,
+ "19": 42746941440.0,
+ "20": 42746941440.0,
+ "21": 42746941440.0,
+ "22": 42746941440.0,
+ "23": 42746941440.0,
+ "24": 42746941440.0,
+ "25": 42746941440.0,
+ "26": 42746941440.0,
+ "27": 42746941440.0,
+ "28": 42746941440.0,
+ "29": 42746941440.0,
+ "30": 42746941440.0,
+ "31": 42746941440.0,
+ "32": 42746941440.0,
+ "33": 42746941440.0,
+ "34": 42746941440.0,
+ "35": 42746941440.0,
+ "36": 42746941440.0,
+ "37": 42746941440.0,
+ "38": 42746941440.0,
+ "39": 42746941440.0,
+ "40": 42746941440.0,
+ "41": 42746941440.0,
+ "42": 42746941440.0,
+ "43": 42746941440.0,
+ "44": 42746941440.0,
+ "45": 42746941440.0,
+ "46": 42746941440.0,
+ "47": 42746941440.0,
+ "48": 42746941440.0,
+ "49": 42746941440.0,
+ "50": 42746941440.0,
+ "51": 42746941440.0,
+ "52": 42746941440.0,
+ "53": 42746941440.0,
+ "54": 42746941440.0,
+ "55": 42746941440.0,
+ "56": 42746941440.0,
+ "57": 42746941440.0,
+ "58": 42746941440.0,
+ "59": 42746941440.0,
+ "60": 42746941440.0,
+ "61": 42746941440.0,
+ "62": 42746941440.0,
+ "63": 42746941440.0,
+ "64": 42746941440.0,
+ "65": 42746941440.0,
+ "66": 42746941440.0,
+ "67": 42746941440.0,
+ "68": 42746941440.0,
+ "69": 42746941440.0,
+ "70": 42746941440.0,
+ "71": 42746941440.0,
+ "72": 42746941440.0,
+ "73": 42746941440.0,
+ "74": 42746941440.0,
+ "75": 42746941440.0,
+ "76": 42746941440.0,
+ "77": 42746941440.0,
+ "78": 42746941440.0,
+ "79": 42746941440.0,
+ "80": 42746941440.0,
+ "81": 42746941440.0,
+ "82": 42746941440.0,
+ "83": 42746941440.0,
+ "84": 42746941440.0,
+ "85": 42746941440.0,
+ "86": 42746941440.0,
+ "87": 42746941440.0,
+ "88": 42746941440.0,
+ "89": 42746941440.0,
+ "90": 42746941440.0,
+ "91": 42746941440.0,
+ "92": 42746941440.0,
+ "93": 42746941440.0,
+ "94": 42746941440.0,
+ "95": 42746941440.0,
+ "96": 42746941440.0,
+ "97": 42746941440.0,
+ "98": 42746941440.0,
+ "99": 42746941440.0,
+ "100": 42746941440.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 1,
+ "values": {
+ "1": "nan",
+ "2": 8.98639,
+ "3": 0.80706,
+ "4": 2.0638,
+ "5": 0.79491,
+ "6": 0.7966,
+ "7": 0.79483,
+ "8": 0.79553,
+ "9": 0.81403,
+ "10": 0.79718,
+ "11": 0.80016,
+ "12": 0.79539,
+ "13": 0.7967,
+ "14": 0.79987,
+ "15": 0.80035,
+ "16": 0.79987,
+ "17": 0.82044,
+ "18": 0.80944,
+ "19": 0.80928,
+ "20": 0.79528,
+ "21": 0.80122,
+ "22": 0.79976,
+ "23": 0.79745,
+ "24": 2.10616,
+ "25": 0.79751,
+ "26": 0.79873,
+ "27": 0.79946,
+ "28": 0.8023,
+ "29": 0.79961,
+ "30": 0.79282,
+ "31": 0.82105,
+ "32": 0.80784,
+ "33": 0.80167,
+ "34": 0.80073,
+ "35": 0.79976,
+ "36": 0.7993,
+ "37": 0.81283,
+ "38": 0.80045,
+ "39": 0.80483,
+ "40": 0.8068,
+ "41": 0.80341,
+ "42": 0.80016,
+ "43": 0.80048,
+ "44": 0.79981,
+ "45": 0.80211,
+ "46": 0.80338,
+ "47": 0.79867,
+ "48": 0.79792,
+ "49": 0.79931,
+ "50": 0.79708,
+ "51": 0.80047,
+ "52": 0.79801,
+ "53": 0.8006,
+ "54": 0.80133,
+ "55": 0.79841,
+ "56": 0.80268,
+ "57": 0.80039,
+ "58": 0.80135,
+ "59": 0.80056,
+ "60": 0.80066,
+ "61": 0.80341,
+ "62": 0.80109,
+ "63": 0.79917,
+ "64": 0.79837,
+ "65": 0.80031,
+ "66": 0.79717,
+ "67": 0.80083,
+ "68": 0.78552,
+ "69": 0.77807,
+ "70": 0.77627,
+ "71": 0.78267,
+ "72": 0.7868,
+ "73": 0.78746,
+ "74": 0.7807,
+ "75": 0.7784,
+ "76": 0.77602,
+ "77": 0.77691,
+ "78": 0.77546,
+ "79": 1.34782,
+ "80": 0.7911,
+ "81": 0.78578,
+ "82": 0.79255,
+ "83": 0.78788,
+ "84": 0.79186,
+ "85": 0.79888,
+ "86": 0.78979,
+ "87": 0.7889,
+ "88": 0.78942,
+ "89": 0.79241,
+ "90": 0.78815,
+ "91": 0.79133,
+ "92": 0.78838,
+ "93": 0.7942,
+ "94": 0.79473,
+ "95": 0.78941,
+ "96": 0.79533,
+ "97": 1.29177,
+ "98": 0.79482,
+ "99": 0.79287,
+ "100": 0.79342
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp1_dp8_seq_packing/model_config.yaml b/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp1_dp8_seq_packing/model_config.yaml
new file mode 100644
index 00000000000..2e86278fa67
--- /dev/null
+++ b/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp1_dp8_seq_packing/model_config.yaml
@@ -0,0 +1,65 @@
+ENV_VARS:
+ CUDA_DEVICE_MAX_CONNECTIONS: 1
+ NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
+ NCCL_ALGO: Ring
+ CUBLAS_WORKSPACE_CONFIG: :4096:8
+ ARTIFACTS_ROOT: /workspace/checkpoints
+MODEL_ARGS:
+ --num-layers: 32
+ --hidden-size: 4096
+ --attention-dropout: 0.0
+ --hidden-dropout: 0.0
+ --num-attention-heads: 32
+ --log-params-norm: true
+ --log-num-zeros-in-grad: true
+ --log-validation-ppl-to-tensorboard: true
+ --log-timers-to-tensorboard: true
+ --tensorboard-dir: ${TENSORBOARD_PATH}
+ --micro-batch-size: 1
+ --global-batch-size: 32
+ --total-seq-length: 3200
+ --packing-buffer-size: 4
+ --seq-length: 4096
+ --max-position-embeddings: 4096
+ --train-iters: 100
+ --timing-log-level: 2
+ --lr-decay-iters: 2200
+ --save: ${CHECKPOINT_SAVE_PATH}
+ --load: ${CHECKPOINT_LOAD_PATH}
+ --tokenizer-type: HuggingFaceTokenizer
+ --tokenizer-model: llava-hf/llava-1.5-7b-hf
+ --distributed-backend: nccl
+ --lr: 0.001
+ --lr-decay-style: cosine
+ --min-lr: 2.0e-5
+ --lr-warmup-iters: 150
+ --log-interval: 1
+ --save-interval: 200
+ --eval-interval: 20000
+ --eval-iters: 30
+ --tensor-model-parallel-size: 1
+ --pipeline-model-parallel-size: 1
+ --context-parallel-size: 1
+ --transformer-impl: transformer_engine
+ --bf16: true
+ --no-gradient-accumulation-fusion: true
+ --init-method-std: 0.006
+ --adam-beta1: 0.9
+ --adam-beta2: 0.95
+ --dataset-provider: llava_vlm
+ --model-provider: llava_vlm
+ --deterministic-mode: true
+ --log-memory-to-tensorboard: true
+ --dataloader-type: external
+ --data-path: ${DATA_PATH}/mixed/mcore_mimo_vlm/llava_pretrain_energon
+ --language-model-checkpoint: ${CHECKPOINT_LOAD_PATH}/model/vicuna_7b_pyt/dcp/mcore-v1.5_fp32/weights
+ --auto-detect-ckpt-format: true
+ --accumulate-allreduce-grads-in-fp32: true
+ --position-embedding-type: rope
+TEST_TYPE: frozen-start
+METRICS:
+ - "iteration-time"
+ - "lm loss"
+ - "num-zeros"
+ - "mem-allocated-bytes"
+ - "mem-max-allocated-bytes"
diff --git a/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp2_dp8/golden_values_dev.json b/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp2_dp8/golden_values_dev.json
new file mode 100644
index 00000000000..d60b063b7cc
--- /dev/null
+++ b/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp2_dp8/golden_values_dev.json
@@ -0,0 +1,142 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 5,
+ "values": {
+ "1": 7.75111,
+ "5": 6.01226,
+ "10": 4.88889,
+ "15": 4.21673,
+ "20": 3.90692,
+ "25": 3.66772,
+ "30": 3.35733,
+ "35": 3.55234,
+ "40": 3.46748,
+ "45": 3.52796,
+ "50": 3.41669,
+ "55": 3.44401,
+ "60": 3.22886,
+ "65": 3.28848,
+ "70": 3.18906,
+ "75": 3.32109,
+ "80": 3.25588,
+ "85": 3.24558,
+ "90": 3.40519,
+ "95": 3.38313,
+ "100": 3.34749
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 5,
+ "values": {
+ "1": 47.0,
+ "5": 62.0,
+ "10": 80.0,
+ "15": 99.0,
+ "20": 89.0,
+ "25": 91.0,
+ "30": 81.0,
+ "35": 207.0,
+ "40": 206.0,
+ "45": 161.0,
+ "50": 144.0,
+ "55": 133.0,
+ "60": 189.0,
+ "65": 161.0,
+ "70": 185.0,
+ "75": 134.0,
+ "80": 133.0,
+ "85": 123.0,
+ "90": 161.0,
+ "95": 143.0,
+ "100": 115.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 5,
+ "values": {
+ "1": 14567188480.0,
+ "5": 14567188480.0,
+ "10": 14567188480.0,
+ "15": 14567188480.0,
+ "20": 14567188480.0,
+ "25": 14567188480.0,
+ "30": 14567188480.0,
+ "35": 14567188480.0,
+ "40": 14567188480.0,
+ "45": 14567188480.0,
+ "50": 14567188480.0,
+ "55": 14567188480.0,
+ "60": 14567188480.0,
+ "65": 14567188480.0,
+ "70": 14567188480.0,
+ "75": 14567188480.0,
+ "80": 14567188480.0,
+ "85": 14567188480.0,
+ "90": 14567188480.0,
+ "95": 14567188480.0,
+ "100": 14567188480.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 5,
+ "values": {
+ "1": 42743271424.0,
+ "5": 42743271424.0,
+ "10": 42743271424.0,
+ "15": 42743271424.0,
+ "20": 42743271424.0,
+ "25": 42743271424.0,
+ "30": 42743271424.0,
+ "35": 42743271424.0,
+ "40": 42743271424.0,
+ "45": 42743271424.0,
+ "50": 42743271424.0,
+ "55": 42743271424.0,
+ "60": 42743271424.0,
+ "65": 42743271424.0,
+ "70": 42743271424.0,
+ "75": 42743271424.0,
+ "80": 42743271424.0,
+ "85": 42743271424.0,
+ "90": 42743271424.0,
+ "95": 42743271424.0,
+ "100": 42743271424.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 5,
+ "values": {
+ "1": 15.80375,
+ "5": 2.05081,
+ "10": 2.05499,
+ "15": 2.06073,
+ "20": 2.05953,
+ "25": 2.06099,
+ "30": 2.05902,
+ "35": 2.06621,
+ "40": 2.0653,
+ "45": 2.0594,
+ "50": 2.06063,
+ "55": 2.05845,
+ "60": 2.06124,
+ "65": 2.06345,
+ "70": 2.0598,
+ "75": 2.21539,
+ "80": 2.06184,
+ "85": 2.06379,
+ "90": 2.05956,
+ "95": 2.05292,
+ "100": 2.08198
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp2_dp8/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp2_dp8/golden_values_dev_dgx_h100.json
new file mode 100644
index 00000000000..a50cbc39e27
--- /dev/null
+++ b/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp2_dp8/golden_values_dev_dgx_h100.json
@@ -0,0 +1,537 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 1,
+ "values": {
+ "1": 7.75233,
+ "2": 7.77448,
+ "3": 7.70302,
+ "4": 6.63518,
+ "5": 6.01176,
+ "6": 5.77595,
+ "7": 5.48486,
+ "8": 5.36699,
+ "9": 5.1468,
+ "10": 4.88775,
+ "11": 4.66914,
+ "12": 4.53205,
+ "13": 4.37947,
+ "14": 4.35984,
+ "15": 4.21684,
+ "16": 4.05957,
+ "17": 4.34442,
+ "18": 4.14671,
+ "19": 3.99585,
+ "20": 3.89078,
+ "21": 3.75709,
+ "22": 3.77234,
+ "23": 3.52609,
+ "24": 3.91072,
+ "25": 3.65776,
+ "26": 3.63895,
+ "27": 3.60595,
+ "28": 3.45648,
+ "29": 3.73849,
+ "30": 3.30868,
+ "31": 3.4276,
+ "32": 3.62189,
+ "33": 3.70653,
+ "34": 3.40568,
+ "35": 3.55094,
+ "36": 3.45308,
+ "37": 3.50025,
+ "38": 3.36815,
+ "39": 3.48028,
+ "40": 3.45472,
+ "41": 3.31981,
+ "42": 3.51003,
+ "43": 3.4993,
+ "44": 3.56531,
+ "45": 3.51197,
+ "46": 3.35803,
+ "47": 3.47229,
+ "48": 3.41256,
+ "49": 3.34425,
+ "50": 3.43778,
+ "51": 3.33532,
+ "52": 3.25471,
+ "53": 3.41513,
+ "54": 3.29364,
+ "55": 3.4157,
+ "56": 3.53343,
+ "57": 3.40335,
+ "58": 3.3663,
+ "59": 3.41197,
+ "60": 3.2307,
+ "61": 3.33513,
+ "62": 3.24475,
+ "63": 3.37945,
+ "64": 3.43959,
+ "65": 3.29508,
+ "66": 3.27725,
+ "67": 3.29703,
+ "68": 3.31435,
+ "69": 3.16158,
+ "70": 3.20195,
+ "71": 3.192,
+ "72": 3.39756,
+ "73": 3.23318,
+ "74": 3.28838,
+ "75": 3.33027,
+ "76": 3.24706,
+ "77": 3.16249,
+ "78": 3.26993,
+ "79": 3.37629,
+ "80": 3.24618,
+ "81": 3.19252,
+ "82": 3.33982,
+ "83": 3.32294,
+ "84": 3.4132,
+ "85": 3.23335,
+ "86": 3.54504,
+ "87": 3.32214,
+ "88": 3.22758,
+ "89": 3.36284,
+ "90": 3.3956,
+ "91": 3.27325,
+ "92": 3.20216,
+ "93": 3.37324,
+ "94": 3.36097,
+ "95": 3.37528,
+ "96": 3.35498,
+ "97": 3.30705,
+ "98": 3.38745,
+ "99": 3.06838,
+ "100": 3.33881
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 1,
+ "values": {
+ "1": 32.0,
+ "2": 44.0,
+ "3": 36.0,
+ "4": 43.0,
+ "5": 56.0,
+ "6": 60.0,
+ "7": 65.0,
+ "8": 95.0,
+ "9": 66.0,
+ "10": 95.0,
+ "11": 95.0,
+ "12": 100.0,
+ "13": 122.0,
+ "14": 110.0,
+ "15": 122.0,
+ "16": 92.0,
+ "17": 42.0,
+ "18": 76.0,
+ "19": 90.0,
+ "20": 90.0,
+ "21": 109.0,
+ "22": 135.0,
+ "23": 129.0,
+ "24": 68.0,
+ "25": 138.0,
+ "26": 84.0,
+ "27": 132.0,
+ "28": 155.0,
+ "29": 190.0,
+ "30": 164.0,
+ "31": 159.0,
+ "32": 140.0,
+ "33": 145.0,
+ "34": 213.0,
+ "35": 160.0,
+ "36": 179.0,
+ "37": 170.0,
+ "38": 114.0,
+ "39": 213.0,
+ "40": 223.0,
+ "41": 159.0,
+ "42": 156.0,
+ "43": 183.0,
+ "44": 113.0,
+ "45": 169.0,
+ "46": 138.0,
+ "47": 132.0,
+ "48": 134.0,
+ "49": 149.0,
+ "50": 108.0,
+ "51": 127.0,
+ "52": 141.0,
+ "53": 127.0,
+ "54": 155.0,
+ "55": 162.0,
+ "56": 160.0,
+ "57": 187.0,
+ "58": 201.0,
+ "59": 145.0,
+ "60": 181.0,
+ "61": 164.0,
+ "62": 168.0,
+ "63": 150.0,
+ "64": 212.0,
+ "65": 140.0,
+ "66": 168.0,
+ "67": 168.0,
+ "68": 163.0,
+ "69": 165.0,
+ "70": 183.0,
+ "71": 167.0,
+ "72": 177.0,
+ "73": 176.0,
+ "74": 183.0,
+ "75": 64.0,
+ "76": 127.0,
+ "77": 190.0,
+ "78": 151.0,
+ "79": 176.0,
+ "80": 128.0,
+ "81": 162.0,
+ "82": 101.0,
+ "83": 106.0,
+ "84": 115.0,
+ "85": 156.0,
+ "86": 138.0,
+ "87": 148.0,
+ "88": 142.0,
+ "89": 118.0,
+ "90": 181.0,
+ "91": 128.0,
+ "92": 149.0,
+ "93": 149.0,
+ "94": 142.0,
+ "95": 159.0,
+ "96": 170.0,
+ "97": 141.0,
+ "98": 139.0,
+ "99": 135.0,
+ "100": 141.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 1,
+ "values": {
+ "1": 14568238080.0,
+ "2": 14568238080.0,
+ "3": 14568238080.0,
+ "4": 14568238080.0,
+ "5": 14568238080.0,
+ "6": 14568238080.0,
+ "7": 14568238080.0,
+ "8": 14568238080.0,
+ "9": 14568238080.0,
+ "10": 14568238080.0,
+ "11": 14568238080.0,
+ "12": 14568238080.0,
+ "13": 14568238080.0,
+ "14": 14568238080.0,
+ "15": 14568238080.0,
+ "16": 14568238080.0,
+ "17": 14568238080.0,
+ "18": 14568238080.0,
+ "19": 14568238080.0,
+ "20": 14568238080.0,
+ "21": 14568238080.0,
+ "22": 14568238080.0,
+ "23": 14568238080.0,
+ "24": 14568238080.0,
+ "25": 14568238080.0,
+ "26": 14568238080.0,
+ "27": 14568238080.0,
+ "28": 14568238080.0,
+ "29": 14568238080.0,
+ "30": 14568238080.0,
+ "31": 14568238080.0,
+ "32": 14568238080.0,
+ "33": 14568238080.0,
+ "34": 14568238080.0,
+ "35": 14568238080.0,
+ "36": 14568238080.0,
+ "37": 14568238080.0,
+ "38": 14568238080.0,
+ "39": 14568238080.0,
+ "40": 14568238080.0,
+ "41": 14568238080.0,
+ "42": 14568238080.0,
+ "43": 14568238080.0,
+ "44": 14568238080.0,
+ "45": 14568238080.0,
+ "46": 14568238080.0,
+ "47": 14568238080.0,
+ "48": 14568238080.0,
+ "49": 14568238080.0,
+ "50": 14568238080.0,
+ "51": 14568238080.0,
+ "52": 14568238080.0,
+ "53": 14568238080.0,
+ "54": 14568238080.0,
+ "55": 14568238080.0,
+ "56": 14568238080.0,
+ "57": 14568238080.0,
+ "58": 14568238080.0,
+ "59": 14568238080.0,
+ "60": 14568238080.0,
+ "61": 14568238080.0,
+ "62": 14568238080.0,
+ "63": 14568238080.0,
+ "64": 14568238080.0,
+ "65": 14568238080.0,
+ "66": 14568238080.0,
+ "67": 14568238080.0,
+ "68": 14568238080.0,
+ "69": 14568238080.0,
+ "70": 14568238080.0,
+ "71": 14568238080.0,
+ "72": 14568238080.0,
+ "73": 14568238080.0,
+ "74": 14568238080.0,
+ "75": 14568238080.0,
+ "76": 14568238080.0,
+ "77": 14568238080.0,
+ "78": 14568238080.0,
+ "79": 14568238080.0,
+ "80": 14568238080.0,
+ "81": 14568238080.0,
+ "82": 14568238080.0,
+ "83": 14568238080.0,
+ "84": 14568238080.0,
+ "85": 14568238080.0,
+ "86": 14568238080.0,
+ "87": 14568238080.0,
+ "88": 14568238080.0,
+ "89": 14568238080.0,
+ "90": 14568238080.0,
+ "91": 14568238080.0,
+ "92": 14568238080.0,
+ "93": 14568238080.0,
+ "94": 14568238080.0,
+ "95": 14568238080.0,
+ "96": 14568238080.0,
+ "97": 14568238080.0,
+ "98": 14568238080.0,
+ "99": 14568238080.0,
+ "100": 14568238080.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 1,
+ "values": {
+ "1": 42744320000.0,
+ "2": 42744320000.0,
+ "3": 42744320000.0,
+ "4": 42744320000.0,
+ "5": 42744320000.0,
+ "6": 42744320000.0,
+ "7": 42744320000.0,
+ "8": 42744320000.0,
+ "9": 42744320000.0,
+ "10": 42744320000.0,
+ "11": 42744320000.0,
+ "12": 42744320000.0,
+ "13": 42744320000.0,
+ "14": 42744320000.0,
+ "15": 42744320000.0,
+ "16": 42744320000.0,
+ "17": 42744320000.0,
+ "18": 42744320000.0,
+ "19": 42744320000.0,
+ "20": 42744320000.0,
+ "21": 42744320000.0,
+ "22": 42744320000.0,
+ "23": 42744320000.0,
+ "24": 42744320000.0,
+ "25": 42744320000.0,
+ "26": 42744320000.0,
+ "27": 42744320000.0,
+ "28": 42744320000.0,
+ "29": 42744320000.0,
+ "30": 42744320000.0,
+ "31": 42744320000.0,
+ "32": 42744320000.0,
+ "33": 42744320000.0,
+ "34": 42744320000.0,
+ "35": 42744320000.0,
+ "36": 42744320000.0,
+ "37": 42744320000.0,
+ "38": 42744320000.0,
+ "39": 42744320000.0,
+ "40": 42744320000.0,
+ "41": 42744320000.0,
+ "42": 42744320000.0,
+ "43": 42744320000.0,
+ "44": 42744320000.0,
+ "45": 42744320000.0,
+ "46": 42744320000.0,
+ "47": 42744320000.0,
+ "48": 42744320000.0,
+ "49": 42744320000.0,
+ "50": 42744320000.0,
+ "51": 42744320000.0,
+ "52": 42744320000.0,
+ "53": 42744320000.0,
+ "54": 42744320000.0,
+ "55": 42744320000.0,
+ "56": 42744320000.0,
+ "57": 42744320000.0,
+ "58": 42744320000.0,
+ "59": 42744320000.0,
+ "60": 42744320000.0,
+ "61": 42744320000.0,
+ "62": 42744320000.0,
+ "63": 42744320000.0,
+ "64": 42744320000.0,
+ "65": 42744320000.0,
+ "66": 42744320000.0,
+ "67": 42744320000.0,
+ "68": 42744320000.0,
+ "69": 42744320000.0,
+ "70": 42744320000.0,
+ "71": 42744320000.0,
+ "72": 42744320000.0,
+ "73": 42744320000.0,
+ "74": 42744320000.0,
+ "75": 42744320000.0,
+ "76": 42744320000.0,
+ "77": 42744320000.0,
+ "78": 42744320000.0,
+ "79": 42744320000.0,
+ "80": 42744320000.0,
+ "81": 42744320000.0,
+ "82": 42744320000.0,
+ "83": 42744320000.0,
+ "84": 42744320000.0,
+ "85": 42744320000.0,
+ "86": 42744320000.0,
+ "87": 42744320000.0,
+ "88": 42744320000.0,
+ "89": 42744320000.0,
+ "90": 42744320000.0,
+ "91": 42744320000.0,
+ "92": 42744320000.0,
+ "93": 42744320000.0,
+ "94": 42744320000.0,
+ "95": 42744320000.0,
+ "96": 42744320000.0,
+ "97": 42744320000.0,
+ "98": 42744320000.0,
+ "99": 42744320000.0,
+ "100": 42744320000.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 100,
+ "step_interval": 1,
+ "values": {
+ "1": "nan",
+ "2": 11.46532,
+ "3": 2.09636,
+ "4": 2.08818,
+ "5": 2.08783,
+ "6": 2.093,
+ "7": 2.08929,
+ "8": 2.09327,
+ "9": 2.09291,
+ "10": 2.09914,
+ "11": 2.09973,
+ "12": 2.09822,
+ "13": 2.09721,
+ "14": 2.09807,
+ "15": 2.09948,
+ "16": 2.1009,
+ "17": 2.10084,
+ "18": 2.10194,
+ "19": 2.10234,
+ "20": 2.09885,
+ "21": 2.09665,
+ "22": 2.10309,
+ "23": 2.10578,
+ "24": 2.11025,
+ "25": 2.10443,
+ "26": 2.10805,
+ "27": 2.11676,
+ "28": 2.11733,
+ "29": 2.10998,
+ "30": 2.11004,
+ "31": 2.11136,
+ "32": 2.12083,
+ "33": 2.11543,
+ "34": 2.11967,
+ "35": 2.11602,
+ "36": 2.12401,
+ "37": 2.124,
+ "38": 2.1177,
+ "39": 2.11907,
+ "40": 2.12522,
+ "41": 2.12057,
+ "42": 2.11877,
+ "43": 2.12069,
+ "44": 2.12008,
+ "45": 2.11616,
+ "46": 2.13383,
+ "47": 2.12006,
+ "48": 2.11476,
+ "49": 2.12011,
+ "50": 2.12235,
+ "51": 2.11602,
+ "52": 2.11337,
+ "53": 2.1179,
+ "54": 2.11571,
+ "55": 2.11051,
+ "56": 2.10876,
+ "57": 2.12063,
+ "58": 2.11844,
+ "59": 2.13051,
+ "60": 2.11821,
+ "61": 2.11788,
+ "62": 2.11607,
+ "63": 2.11733,
+ "64": 2.11732,
+ "65": 2.11657,
+ "66": 2.12,
+ "67": 2.11789,
+ "68": 2.11423,
+ "69": 2.11747,
+ "70": 2.12091,
+ "71": 2.11746,
+ "72": 2.10772,
+ "73": 2.11412,
+ "74": 2.11761,
+ "75": 2.11572,
+ "76": 2.11459,
+ "77": 2.11054,
+ "78": 2.11426,
+ "79": 2.11893,
+ "80": 2.11476,
+ "81": 2.11969,
+ "82": 2.11923,
+ "83": 2.12159,
+ "84": 2.11823,
+ "85": 2.11883,
+ "86": 2.12147,
+ "87": 2.1191,
+ "88": 2.12309,
+ "89": 2.11469,
+ "90": 2.11618,
+ "91": 2.11967,
+ "92": 2.11922,
+ "93": 2.12028,
+ "94": 2.111,
+ "95": 2.11508,
+ "96": 2.1155,
+ "97": 2.11627,
+ "98": 2.11348,
+ "99": 2.11605,
+ "100": 2.11807
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp2_dp8/model_config.yaml b/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp2_dp8/model_config.yaml
new file mode 100644
index 00000000000..37c55e4cd93
--- /dev/null
+++ b/tests/functional_tests/test_cases/mimo/mimo_vlm_pretrain_convergence_tp1_pp1_cp2_dp8/model_config.yaml
@@ -0,0 +1,65 @@
+ENV_VARS:
+ CUDA_DEVICE_MAX_CONNECTIONS: 1
+ NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
+ NCCL_ALGO: Ring
+ CUBLAS_WORKSPACE_CONFIG: :4096:8
+ ARTIFACTS_ROOT: /workspace/checkpoints
+MODEL_ARGS:
+ --num-layers: 32
+ --hidden-size: 4096
+ --attention-dropout: 0.0
+ --hidden-dropout: 0.0
+ --num-attention-heads: 32
+ --log-params-norm: true
+ --log-num-zeros-in-grad: true
+ --log-validation-ppl-to-tensorboard: true
+ --log-timers-to-tensorboard: true
+ --tensorboard-dir: ${TENSORBOARD_PATH}
+ --micro-batch-size: 8
+ --global-batch-size: 128
+ --total-seq-length: 2048
+ --seq-length: 4096
+ --max-position-embeddings: 4096
+ --train-iters: 100
+ --timing-log-level: 2
+ --lr-decay-iters: 2200
+ --save: ${CHECKPOINT_SAVE_PATH}
+ --load: ${CHECKPOINT_LOAD_PATH}
+ --tokenizer-type: HuggingFaceTokenizer
+ --tokenizer-model: llava-hf/llava-1.5-7b-hf
+ --distributed-backend: nccl
+ --lr: 0.001
+ --lr-decay-style: cosine
+ --min-lr: 2.0e-5
+ --lr-warmup-iters: 150
+ --log-interval: 1
+ --save-interval: 200
+ --eval-interval: 20000
+ --eval-iters: 30
+ --tensor-model-parallel-size: 1
+ --pipeline-model-parallel-size: 1
+ --context-parallel-size: 2
+ --transformer-impl: transformer_engine
+ --attention-softmax-in-fp32: true
+ --bf16: true
+ --no-gradient-accumulation-fusion: true
+ --init-method-std: 0.006
+ --adam-beta1: 0.9
+ --adam-beta2: 0.95
+ --dataset-provider: llava_vlm
+ --model-provider: llava_vlm
+ --deterministic-mode: true
+ --log-memory-to-tensorboard: true
+ --dataloader-type: external
+ --data-path: ${DATA_PATH}/mixed/mcore_mimo_vlm/llava_pretrain_energon
+ --language-model-checkpoint: ${CHECKPOINT_LOAD_PATH}/model/vicuna_7b_pyt/dcp/mcore-v1.5_fp32/weights
+ --auto-detect-ckpt-format: true
+ --accumulate-allreduce-grads-in-fp32: true
+ --position-embedding-type: rope
+TEST_TYPE: frozen-start
+METRICS:
+ - "iteration-time"
+ - "lm loss"
+ - "num-zeros"
+ - "mem-allocated-bytes"
+ - "mem-max-allocated-bytes"
diff --git a/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release/model_config.yaml b/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release/model_config.yaml
index 7bc14780fb3..e504bcb1320 100644
--- a/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release/model_config.yaml
+++ b/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release/model_config.yaml
@@ -19,7 +19,7 @@ MODEL_ARGS:
--distributed-timeout-minutes: 60
--tensor-model-parallel-size: 2
--pipeline-model-parallel-size: 4
- --pipeline-model-parallel-layout: Et*2\\|\\(tt\\|\\)*5t\\|tmL # Et*2|(tt|)*5t|tmL
+ --pipeline-model-parallel-layout: Et*2|(tt|)*5t|tmL
--expert-model-parallel-size: 16
--context-parallel-size: 1
--expert-tensor-parallel-size: 1
@@ -132,8 +132,8 @@ MODEL_ARGS:
# Add checkpointing args
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --save-interval: 500
- --save-retain-interval: 10000
+ --save-interval: 250
+ --save-retain-interval: 5000
--dist-ckpt-strictness: log_all
# Add initialization args
diff --git a/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release_sm/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release_sm/golden_values_dev_dgx_h100.json
new file mode 100644
index 00000000000..7128b0fe3b7
--- /dev/null
+++ b/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release_sm/golden_values_dev_dgx_h100.json
@@ -0,0 +1,11492 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 9535,
+ "step_interval": 5,
+ "values": {
+ "1": 13.8975,
+ "5": 13.89167,
+ "10": 13.85847,
+ "15": 13.84968,
+ "20": 13.74133,
+ "25": 13.71288,
+ "30": 13.39133,
+ "35": 13.3242,
+ "40": 13.23301,
+ "45": 13.11661,
+ "50": 12.53562,
+ "55": 12.34932,
+ "60": 12.16994,
+ "65": 12.0086,
+ "70": 11.82944,
+ "75": 11.55916,
+ "80": 11.30521,
+ "85": 11.11605,
+ "90": 10.95916,
+ "95": 10.79633,
+ "100": 10.5844,
+ "105": 10.45442,
+ "110": 10.23497,
+ "115": 10.02655,
+ "120": 9.87485,
+ "125": 9.73412,
+ "130": 9.64325,
+ "135": 9.57807,
+ "140": 9.34464,
+ "145": 9.33249,
+ "150": 9.17438,
+ "155": 9.10879,
+ "160": 9.02711,
+ "165": 8.91289,
+ "170": 8.8625,
+ "175": 8.82414,
+ "180": 8.68006,
+ "185": 8.71967,
+ "190": 8.59279,
+ "195": 8.59809,
+ "200": 8.48717,
+ "205": 8.39733,
+ "210": 8.35469,
+ "215": 8.40679,
+ "220": 8.27876,
+ "225": 8.29525,
+ "230": 8.27805,
+ "235": 8.20492,
+ "240": 8.15417,
+ "245": 8.13441,
+ "250": 8.06861,
+ "255": 8.0828,
+ "260": 7.97617,
+ "265": 7.96133,
+ "270": 7.91405,
+ "275": 7.9018,
+ "280": 7.89165,
+ "285": 7.9087,
+ "290": 7.8535,
+ "295": 7.83972,
+ "300": 7.73754,
+ "305": 7.73704,
+ "310": 7.70609,
+ "315": 7.70571,
+ "320": 7.70058,
+ "325": 7.62237,
+ "330": 7.61227,
+ "335": 7.59391,
+ "340": 7.63439,
+ "345": 7.5201,
+ "350": 7.51016,
+ "355": 7.43711,
+ "360": 7.52944,
+ "365": 7.45579,
+ "370": 7.4863,
+ "375": 7.42865,
+ "380": 7.40735,
+ "385": 7.40293,
+ "390": 7.42461,
+ "395": 7.36228,
+ "400": 7.30134,
+ "405": 7.31421,
+ "410": 7.30364,
+ "415": 7.28755,
+ "420": 7.28671,
+ "425": 7.25188,
+ "430": 7.20275,
+ "435": 7.21412,
+ "440": 7.18198,
+ "445": 7.16684,
+ "450": 7.12052,
+ "455": 7.14285,
+ "460": 7.10807,
+ "465": 7.10474,
+ "470": 7.07215,
+ "475": 7.09424,
+ "480": 6.9693,
+ "485": 7.02346,
+ "490": 6.9746,
+ "495": 6.94912,
+ "500": 6.89692,
+ "505": 6.93296,
+ "510": 6.91399,
+ "515": 6.87426,
+ "520": 6.8686,
+ "525": 6.86475,
+ "530": 6.87154,
+ "535": 6.85358,
+ "540": 6.77684,
+ "545": 6.80942,
+ "550": 6.83146,
+ "555": 6.86188,
+ "560": 6.80812,
+ "565": 6.74148,
+ "570": 6.76331,
+ "575": 6.77472,
+ "580": 6.69916,
+ "585": 6.71508,
+ "590": 6.6559,
+ "595": 6.64992,
+ "600": 6.67331,
+ "605": 6.66076,
+ "610": 6.63867,
+ "615": 6.68854,
+ "620": 6.60367,
+ "625": 6.57378,
+ "630": 6.5671,
+ "635": 6.60386,
+ "640": 6.5864,
+ "645": 6.58717,
+ "650": 6.62806,
+ "655": 6.61265,
+ "660": 6.52891,
+ "665": 6.51468,
+ "670": 6.46305,
+ "675": 6.56687,
+ "680": 6.53344,
+ "685": 6.48865,
+ "690": 6.46364,
+ "695": 6.42717,
+ "700": 6.4209,
+ "705": 6.43092,
+ "710": 6.4637,
+ "715": 6.46843,
+ "720": 6.34683,
+ "725": 6.40273,
+ "730": 6.38913,
+ "735": 6.40946,
+ "740": 6.3456,
+ "745": 6.31143,
+ "750": 6.36264,
+ "755": 6.29049,
+ "760": 6.29903,
+ "765": 6.30676,
+ "770": 6.31096,
+ "775": 6.29631,
+ "780": 6.26703,
+ "785": 6.2826,
+ "790": 6.25132,
+ "795": 6.23616,
+ "800": 6.22101,
+ "805": 6.28938,
+ "810": 6.15703,
+ "815": 6.1735,
+ "820": 6.18537,
+ "825": 6.20103,
+ "830": 6.20966,
+ "835": 6.16573,
+ "840": 6.13099,
+ "845": 6.17425,
+ "850": 6.12159,
+ "855": 6.12766,
+ "860": 6.11723,
+ "865": 6.12896,
+ "870": 6.0897,
+ "875": 6.13607,
+ "880": 6.0892,
+ "885": 6.08796,
+ "890": 6.15143,
+ "895": 6.0392,
+ "900": 6.05227,
+ "905": 6.07396,
+ "910": 6.04615,
+ "915": 6.02793,
+ "920": 6.02626,
+ "925": 6.00901,
+ "930": 6.03689,
+ "935": 6.03037,
+ "940": 5.96187,
+ "945": 6.00482,
+ "950": 6.02433,
+ "955": 5.97341,
+ "960": 5.97459,
+ "965": 5.88915,
+ "970": 5.93424,
+ "975": 5.9327,
+ "980": 5.92445,
+ "985": 5.92272,
+ "990": 5.96553,
+ "995": 5.87903,
+ "1000": 5.8977,
+ "1005": 5.85006,
+ "1010": 5.88321,
+ "1015": 5.9072,
+ "1020": 5.84136,
+ "1025": 5.82354,
+ "1030": 5.82972,
+ "1035": 5.90894,
+ "1040": 5.83168,
+ "1045": 5.81232,
+ "1050": 5.85696,
+ "1055": 5.82251,
+ "1060": 5.77739,
+ "1065": 5.76455,
+ "1070": 5.80673,
+ "1075": 5.80091,
+ "1080": 5.79023,
+ "1085": 5.7979,
+ "1090": 5.77574,
+ "1095": 5.79703,
+ "1100": 5.75342,
+ "1105": 5.7223,
+ "1110": 5.77445,
+ "1115": 5.7046,
+ "1120": 5.64871,
+ "1125": 5.66259,
+ "1130": 5.72696,
+ "1135": 5.676,
+ "1140": 5.66883,
+ "1145": 5.6632,
+ "1150": 5.68939,
+ "1155": 5.6516,
+ "1160": 5.64173,
+ "1165": 5.67963,
+ "1170": 5.66521,
+ "1175": 5.62362,
+ "1180": 5.62421,
+ "1185": 5.61858,
+ "1190": 5.61483,
+ "1195": 5.60402,
+ "1200": 5.55083,
+ "1205": 5.65542,
+ "1210": 5.51629,
+ "1215": 5.54258,
+ "1220": 5.6294,
+ "1225": 5.51298,
+ "1230": 5.5668,
+ "1235": 5.51947,
+ "1240": 5.55604,
+ "1245": 5.52935,
+ "1250": 5.50513,
+ "1255": 5.50104,
+ "1260": 5.50196,
+ "1265": 5.47535,
+ "1270": 5.43932,
+ "1275": 5.52,
+ "1280": 5.45537,
+ "1285": 5.47194,
+ "1290": 5.43979,
+ "1295": 5.45883,
+ "1300": 5.47762,
+ "1305": 5.44139,
+ "1310": 5.38545,
+ "1315": 5.44371,
+ "1320": 5.43067,
+ "1325": 5.35986,
+ "1330": 5.42163,
+ "1335": 5.39396,
+ "1340": 5.44624,
+ "1345": 5.4109,
+ "1350": 5.3722,
+ "1355": 5.36963,
+ "1360": 5.37409,
+ "1365": 5.38194,
+ "1370": 5.31816,
+ "1375": 5.3231,
+ "1380": 5.37252,
+ "1385": 5.33986,
+ "1390": 5.33682,
+ "1395": 5.34811,
+ "1400": 5.34153,
+ "1405": 5.33869,
+ "1410": 5.31715,
+ "1415": 5.27957,
+ "1420": 5.32088,
+ "1425": 5.29978,
+ "1430": 5.34862,
+ "1435": 5.25701,
+ "1440": 5.27836,
+ "1445": 5.32059,
+ "1450": 5.29379,
+ "1455": 5.30013,
+ "1460": 5.26683,
+ "1465": 5.27051,
+ "1470": 5.29912,
+ "1475": 5.27252,
+ "1480": 5.27036,
+ "1485": 5.23002,
+ "1490": 5.22902,
+ "1495": 5.24418,
+ "1500": 5.24992,
+ "1505": 5.20843,
+ "1510": 5.22305,
+ "1515": 5.17317,
+ "1520": 5.19742,
+ "1525": 5.15733,
+ "1530": 5.18106,
+ "1535": 5.17074,
+ "1540": 5.17525,
+ "1545": 5.2009,
+ "1550": 5.20146,
+ "1555": 5.18343,
+ "1560": 5.12627,
+ "1565": 5.16588,
+ "1570": 5.17667,
+ "1575": 5.13623,
+ "1580": 5.16381,
+ "1585": 5.14035,
+ "1590": 5.1271,
+ "1595": 5.09654,
+ "1600": 5.16978,
+ "1605": 5.10451,
+ "1610": 5.10635,
+ "1615": 5.10077,
+ "1620": 5.11513,
+ "1625": 5.11098,
+ "1630": 5.08852,
+ "1635": 5.14428,
+ "1640": 5.09735,
+ "1645": 5.09269,
+ "1650": 5.07766,
+ "1655": 5.06996,
+ "1660": 5.06182,
+ "1665": 5.05394,
+ "1670": 5.06872,
+ "1675": 5.06397,
+ "1680": 5.00959,
+ "1685": 5.01814,
+ "1690": 4.9985,
+ "1695": 4.99827,
+ "1700": 5.0338,
+ "1705": 5.02217,
+ "1710": 5.01124,
+ "1715": 4.97185,
+ "1720": 4.96629,
+ "1725": 5.0022,
+ "1730": 4.95025,
+ "1735": 5.02659,
+ "1740": 4.95321,
+ "1745": 4.98346,
+ "1750": 4.96823,
+ "1755": 4.96616,
+ "1760": 4.97274,
+ "1765": 4.93821,
+ "1770": 4.93972,
+ "1775": 4.93962,
+ "1780": 4.97007,
+ "1785": 4.91751,
+ "1790": 4.942,
+ "1795": 4.94274,
+ "1800": 4.88675,
+ "1805": 4.88457,
+ "1810": 4.89756,
+ "1815": 4.90369,
+ "1820": 4.88431,
+ "1825": 4.8955,
+ "1830": 4.88026,
+ "1835": 4.88401,
+ "1840": 4.87175,
+ "1845": 4.86912,
+ "1850": 4.83888,
+ "1855": 4.88848,
+ "1860": 4.84484,
+ "1865": 4.84997,
+ "1870": 4.83233,
+ "1875": 4.83426,
+ "1880": 4.89183,
+ "1885": 4.83787,
+ "1890": 4.83371,
+ "1895": 4.78145,
+ "1900": 4.81765,
+ "1905": 4.81006,
+ "1910": 4.83514,
+ "1915": 4.79318,
+ "1920": 4.77967,
+ "1925": 4.79417,
+ "1930": 4.76819,
+ "1935": 4.79695,
+ "1940": 4.76013,
+ "1945": 4.80214,
+ "1950": 4.84529,
+ "1955": 4.778,
+ "1960": 4.7708,
+ "1965": 4.72835,
+ "1970": 4.73013,
+ "1975": 4.79752,
+ "1980": 4.72855,
+ "1985": 4.74403,
+ "1990": 4.77936,
+ "1995": 4.74713,
+ "2000": 4.76361,
+ "2005": 4.80344,
+ "2010": 4.71635,
+ "2015": 4.70076,
+ "2020": 4.70399,
+ "2025": 4.75511,
+ "2030": 4.67973,
+ "2035": 4.70092,
+ "2040": 4.67663,
+ "2045": 4.75717,
+ "2050": 4.7419,
+ "2055": 4.70763,
+ "2060": 4.70919,
+ "2065": 4.66515,
+ "2070": 4.67651,
+ "2075": 4.69227,
+ "2080": 4.67088,
+ "2085": 4.69422,
+ "2090": 4.62279,
+ "2095": 4.6481,
+ "2100": 4.6131,
+ "2105": 4.64581,
+ "2110": 4.64453,
+ "2115": 4.64601,
+ "2120": 4.63515,
+ "2125": 4.60877,
+ "2130": 4.60969,
+ "2135": 4.62843,
+ "2140": 4.62196,
+ "2145": 4.57944,
+ "2150": 4.61227,
+ "2155": 4.58414,
+ "2160": 4.60611,
+ "2165": 4.58085,
+ "2170": 4.60891,
+ "2175": 4.60531,
+ "2180": 4.58788,
+ "2185": 4.6026,
+ "2190": 4.58157,
+ "2195": 4.55818,
+ "2200": 4.55265,
+ "2205": 4.56601,
+ "2210": 4.61227,
+ "2215": 4.64394,
+ "2220": 4.59827,
+ "2225": 4.57195,
+ "2230": 4.56316,
+ "2235": 4.61861,
+ "2240": 4.50972,
+ "2245": 4.51169,
+ "2250": 4.52372,
+ "2255": 4.54046,
+ "2260": 4.47935,
+ "2265": 4.55973,
+ "2270": 4.49073,
+ "2275": 4.55476,
+ "2280": 4.50753,
+ "2285": 4.52819,
+ "2290": 4.52519,
+ "2295": 4.52864,
+ "2300": 4.52467,
+ "2305": 4.49293,
+ "2310": 4.52805,
+ "2315": 4.46386,
+ "2320": 4.51273,
+ "2325": 4.49522,
+ "2330": 4.48524,
+ "2335": 4.47049,
+ "2340": 4.49244,
+ "2345": 4.52236,
+ "2350": 4.47069,
+ "2355": 4.46875,
+ "2360": 4.44212,
+ "2365": 4.44859,
+ "2370": 4.4383,
+ "2375": 4.44217,
+ "2380": 4.38903,
+ "2385": 4.43322,
+ "2390": 4.43233,
+ "2395": 4.46225,
+ "2400": 4.42602,
+ "2405": 4.39877,
+ "2410": 4.45823,
+ "2415": 4.42476,
+ "2420": 4.42791,
+ "2425": 4.39438,
+ "2430": 4.41764,
+ "2435": 4.4038,
+ "2440": 4.39592,
+ "2445": 4.41069,
+ "2450": 4.3809,
+ "2455": 4.41952,
+ "2460": 4.36836,
+ "2465": 4.41477,
+ "2470": 4.40167,
+ "2475": 4.41586,
+ "2480": 4.34142,
+ "2485": 4.37168,
+ "2490": 4.38035,
+ "2495": 4.36252,
+ "2500": 4.35892,
+ "2505": 4.37316,
+ "2510": 4.41893,
+ "2515": 4.40491,
+ "2520": 4.34378,
+ "2525": 4.36052,
+ "2530": 4.36747,
+ "2535": 4.36956,
+ "2540": 4.37323,
+ "2545": 4.37782,
+ "2550": 4.30205,
+ "2555": 4.37395,
+ "2560": 4.34982,
+ "2565": 4.31053,
+ "2570": 4.33219,
+ "2575": 4.31384,
+ "2580": 4.30806,
+ "2585": 4.29216,
+ "2590": 4.31584,
+ "2595": 4.28499,
+ "2600": 4.29537,
+ "2605": 4.31332,
+ "2610": 4.32267,
+ "2615": 4.27623,
+ "2620": 4.26552,
+ "2625": 4.30548,
+ "2630": 4.22993,
+ "2635": 4.30683,
+ "2640": 4.30151,
+ "2645": 4.26213,
+ "2650": 4.29285,
+ "2655": 4.26701,
+ "2660": 4.21405,
+ "2665": 4.30473,
+ "2670": 4.26475,
+ "2675": 4.22641,
+ "2680": 4.25752,
+ "2685": 4.25667,
+ "2690": 4.22845,
+ "2695": 4.28071,
+ "2700": 4.18659,
+ "2705": 4.23661,
+ "2710": 4.26274,
+ "2715": 4.23865,
+ "2720": 4.24394,
+ "2725": 4.22266,
+ "2730": 4.23541,
+ "2735": 4.22715,
+ "2740": 4.20077,
+ "2745": 4.18687,
+ "2750": 4.21523,
+ "2755": 4.22621,
+ "2760": 4.22884,
+ "2765": 4.18542,
+ "2770": 4.23514,
+ "2775": 4.17938,
+ "2780": 4.21333,
+ "2785": 4.19886,
+ "2790": 4.22476,
+ "2795": 4.18911,
+ "2800": 4.1203,
+ "2805": 4.17142,
+ "2810": 4.16464,
+ "2815": 4.15222,
+ "2820": 4.19927,
+ "2825": 4.19376,
+ "2830": 4.17106,
+ "2835": 4.17382,
+ "2840": 4.16262,
+ "2845": 4.14526,
+ "2850": 4.16441,
+ "2855": 4.12236,
+ "2860": 4.14813,
+ "2865": 4.1697,
+ "2870": 4.14382,
+ "2875": 4.16853,
+ "2880": 4.08836,
+ "2885": 4.13982,
+ "2890": 4.11914,
+ "2895": 4.1651,
+ "2900": 4.09608,
+ "2905": 4.11334,
+ "2910": 4.10565,
+ "2915": 4.1508,
+ "2920": 4.12523,
+ "2925": 4.10386,
+ "2930": 4.08884,
+ "2935": 4.08356,
+ "2940": 4.09931,
+ "2945": 4.06739,
+ "2950": 4.03919,
+ "2955": 4.04123,
+ "2960": 4.04746,
+ "2965": 4.06348,
+ "2970": 4.07787,
+ "2975": 4.09493,
+ "2980": 4.0333,
+ "2985": 4.07062,
+ "2990": 4.09082,
+ "2995": 4.03398,
+ "3000": 4.04586,
+ "3005": 4.032,
+ "3010": 4.06719,
+ "3015": 4.02906,
+ "3020": 4.0364,
+ "3025": 4.02944,
+ "3030": 4.05565,
+ "3035": 4.0487,
+ "3040": 4.05271,
+ "3045": 4.04602,
+ "3050": 4.02013,
+ "3055": 4.0067,
+ "3060": 3.9961,
+ "3065": 4.02694,
+ "3070": 4.03589,
+ "3075": 3.9773,
+ "3080": 4.01036,
+ "3085": 4.00212,
+ "3090": 4.01137,
+ "3095": 4.02827,
+ "3100": 4.02048,
+ "3105": 3.9977,
+ "3110": 3.99552,
+ "3115": 3.93397,
+ "3120": 4.00945,
+ "3125": 3.94072,
+ "3130": 3.97711,
+ "3135": 3.96692,
+ "3140": 3.95485,
+ "3145": 3.93219,
+ "3150": 3.97038,
+ "3155": 3.96377,
+ "3160": 3.96628,
+ "3165": 3.96498,
+ "3170": 3.9673,
+ "3175": 3.93327,
+ "3180": 3.93884,
+ "3185": 3.90494,
+ "3190": 3.92968,
+ "3195": 3.91283,
+ "3200": 3.89811,
+ "3205": 3.92139,
+ "3210": 3.89852,
+ "3215": 3.91394,
+ "3220": 3.89377,
+ "3225": 3.91384,
+ "3230": 3.89961,
+ "3235": 3.91452,
+ "3240": 3.89391,
+ "3245": 3.89442,
+ "3250": 3.8418,
+ "3255": 3.8944,
+ "3260": 3.87988,
+ "3265": 3.92755,
+ "3270": 3.91803,
+ "3275": 3.86798,
+ "3280": 3.88986,
+ "3285": 3.86927,
+ "3290": 3.8731,
+ "3295": 3.84129,
+ "3300": 3.85365,
+ "3305": 3.86119,
+ "3310": 3.86203,
+ "3315": 3.90228,
+ "3320": 3.85478,
+ "3325": 3.84798,
+ "3330": 3.82591,
+ "3335": 3.86865,
+ "3340": 3.82132,
+ "3345": 3.8351,
+ "3350": 3.85832,
+ "3355": 3.85197,
+ "3360": 3.83708,
+ "3365": 3.84518,
+ "3370": 3.82629,
+ "3375": 3.85151,
+ "3380": 3.80249,
+ "3385": 3.81821,
+ "3390": 3.78994,
+ "3395": 3.86952,
+ "3400": 3.8402,
+ "3405": 3.8619,
+ "3410": 3.7836,
+ "3415": 3.73133,
+ "3420": 3.80101,
+ "3425": 3.81992,
+ "3430": 3.85252,
+ "3435": 3.81288,
+ "3440": 3.83119,
+ "3445": 3.77629,
+ "3450": 3.79155,
+ "3455": 3.80486,
+ "3460": 3.78749,
+ "3465": 3.76272,
+ "3470": 3.77298,
+ "3475": 3.7774,
+ "3480": 3.78271,
+ "3485": 3.80768,
+ "3490": 3.77432,
+ "3495": 3.8115,
+ "3500": 3.77688,
+ "3505": 3.77969,
+ "3510": 3.75353,
+ "3515": 3.80611,
+ "3520": 3.80049,
+ "3525": 3.76629,
+ "3530": 3.75967,
+ "3535": 3.76485,
+ "3540": 3.82139,
+ "3545": 3.73337,
+ "3550": 3.79312,
+ "3555": 3.72516,
+ "3560": 3.78882,
+ "3565": 3.74665,
+ "3570": 3.74627,
+ "3575": 3.71993,
+ "3580": 3.7775,
+ "3585": 3.76732,
+ "3590": 3.69321,
+ "3595": 3.76328,
+ "3600": 3.71482,
+ "3605": 3.72951,
+ "3610": 3.71468,
+ "3615": 3.75009,
+ "3620": 3.78866,
+ "3625": 3.72186,
+ "3630": 3.76163,
+ "3635": 3.68734,
+ "3640": 3.71541,
+ "3645": 3.74638,
+ "3650": 3.69401,
+ "3655": 3.7213,
+ "3660": 3.73442,
+ "3665": 3.7507,
+ "3670": 3.71895,
+ "3675": 3.71386,
+ "3680": 3.72931,
+ "3685": 3.67968,
+ "3690": 3.70287,
+ "3695": 3.68936,
+ "3700": 3.70866,
+ "3705": 3.68087,
+ "3710": 3.68513,
+ "3715": 3.68733,
+ "3720": 3.67101,
+ "3725": 3.65148,
+ "3730": 3.65459,
+ "3735": 3.69022,
+ "3740": 3.67732,
+ "3745": 3.66367,
+ "3750": 3.68514,
+ "3755": 3.66734,
+ "3760": 3.6738,
+ "3765": 3.66185,
+ "3770": 3.65693,
+ "3775": 3.64159,
+ "3780": 3.62858,
+ "3785": 3.67844,
+ "3790": 3.60706,
+ "3795": 3.64934,
+ "3800": 3.63846,
+ "3805": 3.62349,
+ "3810": 3.60015,
+ "3815": 3.63729,
+ "3820": 3.64046,
+ "3825": 3.65761,
+ "3830": 3.64371,
+ "3835": 3.60485,
+ "3840": 3.68091,
+ "3845": 3.66236,
+ "3850": 3.60703,
+ "3855": 3.60582,
+ "3860": 3.65843,
+ "3865": 3.61083,
+ "3870": 3.67567,
+ "3875": 3.58961,
+ "3880": 3.58503,
+ "3885": 3.61272,
+ "3890": 3.61249,
+ "3895": 3.5623,
+ "3900": 3.62342,
+ "3905": 3.59543,
+ "3910": 3.58234,
+ "3915": 3.58631,
+ "3920": 3.57266,
+ "3925": 3.5759,
+ "3930": 3.5838,
+ "3935": 3.58179,
+ "3940": 3.57825,
+ "3945": 3.57456,
+ "3950": 3.62514,
+ "3955": 3.57771,
+ "3960": 3.61109,
+ "3965": 3.59244,
+ "3970": 3.56829,
+ "3975": 3.56913,
+ "3980": 3.53327,
+ "3985": 3.60867,
+ "3990": 3.58747,
+ "3995": 3.61187,
+ "4000": 3.56288,
+ "4005": 3.54301,
+ "4010": 3.58938,
+ "4015": 3.58972,
+ "4020": 3.58906,
+ "4025": 3.5762,
+ "4030": 3.6331,
+ "4035": 3.57694,
+ "4040": 3.59071,
+ "4045": 3.60202,
+ "4050": 3.57826,
+ "4055": 3.57723,
+ "4060": 3.59347,
+ "4065": 3.58788,
+ "4070": 3.52379,
+ "4075": 3.56191,
+ "4080": 3.53253,
+ "4085": 3.55245,
+ "4090": 3.54869,
+ "4095": 3.53394,
+ "4100": 3.55312,
+ "4105": 3.53978,
+ "4110": 3.51755,
+ "4115": 3.56904,
+ "4120": 3.49958,
+ "4125": 3.49831,
+ "4130": 3.55307,
+ "4135": 3.54811,
+ "4140": 3.49448,
+ "4145": 3.51271,
+ "4150": 3.55617,
+ "4155": 3.48874,
+ "4160": 3.54774,
+ "4165": 3.56571,
+ "4170": 3.50525,
+ "4175": 3.5049,
+ "4180": 3.49756,
+ "4185": 3.51422,
+ "4190": 3.50409,
+ "4195": 3.50336,
+ "4200": 3.49861,
+ "4205": 3.53159,
+ "4210": 3.51978,
+ "4215": 3.5262,
+ "4220": 3.53205,
+ "4225": 3.50827,
+ "4230": 3.50113,
+ "4235": 3.52401,
+ "4240": 3.49524,
+ "4245": 3.49848,
+ "4250": 3.49124,
+ "4255": 3.51277,
+ "4260": 3.46876,
+ "4265": 3.49339,
+ "4270": 3.50609,
+ "4275": 3.54078,
+ "4280": 3.49156,
+ "4285": 3.5151,
+ "4290": 3.47566,
+ "4295": 3.48827,
+ "4300": 3.52712,
+ "4305": 3.49195,
+ "4310": 3.51667,
+ "4315": 3.50859,
+ "4320": 3.50754,
+ "4325": 3.51879,
+ "4330": 3.46426,
+ "4335": 3.4932,
+ "4340": 3.50633,
+ "4345": 3.4365,
+ "4350": 3.44957,
+ "4355": 3.52857,
+ "4360": 3.4855,
+ "4365": 3.4735,
+ "4370": 3.47732,
+ "4375": 3.44186,
+ "4380": 3.44541,
+ "4385": 3.42823,
+ "4390": 3.49484,
+ "4395": 3.47888,
+ "4400": 3.46998,
+ "4405": 3.42036,
+ "4410": 3.48818,
+ "4415": 3.45002,
+ "4420": 3.4443,
+ "4425": 3.47619,
+ "4430": 3.44941,
+ "4435": 3.49107,
+ "4440": 3.48888,
+ "4445": 3.44299,
+ "4450": 3.40351,
+ "4455": 3.46759,
+ "4460": 3.43862,
+ "4465": 3.45298,
+ "4470": 3.4258,
+ "4475": 3.45833,
+ "4480": 3.44632,
+ "4485": 3.44109,
+ "4490": 3.43526,
+ "4495": 3.38849,
+ "4500": 3.45425,
+ "4505": 3.43671,
+ "4510": 3.44545,
+ "4515": 3.40771,
+ "4520": 3.43987,
+ "4525": 3.41031,
+ "4530": 3.44295,
+ "4535": 3.39856,
+ "4540": 3.42341,
+ "4545": 3.43744,
+ "4550": 3.47904,
+ "4555": 3.40214,
+ "4560": 3.42568,
+ "4565": 3.379,
+ "4570": 3.4192,
+ "4575": 3.41629,
+ "4580": 3.45382,
+ "4585": 3.42788,
+ "4590": 3.4238,
+ "4595": 3.39937,
+ "4600": 3.39884,
+ "4605": 3.42343,
+ "4610": 3.4148,
+ "4615": 3.45645,
+ "4620": 3.39673,
+ "4625": 3.42891,
+ "4630": 3.4135,
+ "4635": 3.39663,
+ "4640": 3.42848,
+ "4645": 3.42552,
+ "4650": 3.43444,
+ "4655": 3.41002,
+ "4660": 3.39935,
+ "4665": 3.4145,
+ "4670": 3.44578,
+ "4675": 3.4037,
+ "4680": 3.43065,
+ "4685": 3.43,
+ "4690": 3.40202,
+ "4695": 3.38442,
+ "4700": 3.37514,
+ "4705": 3.35259,
+ "4710": 3.40907,
+ "4715": 3.39357,
+ "4720": 3.38969,
+ "4725": 3.36116,
+ "4730": 3.39799,
+ "4735": 3.326,
+ "4740": 3.36887,
+ "4745": 3.40301,
+ "4750": 3.36289,
+ "4755": 3.39261,
+ "4760": 3.41436,
+ "4765": 3.36469,
+ "4770": 3.36706,
+ "4775": 3.36498,
+ "4780": 3.37436,
+ "4785": 3.37702,
+ "4790": 3.41296,
+ "4795": 3.39531,
+ "4800": 3.34727,
+ "4805": 3.41516,
+ "4810": 3.35413,
+ "4815": 3.39053,
+ "4820": 3.35003,
+ "4825": 3.40296,
+ "4830": 3.38625,
+ "4835": 3.36988,
+ "4840": 3.38523,
+ "4845": 3.3303,
+ "4850": 3.39488,
+ "4855": 3.39611,
+ "4860": 3.32935,
+ "4865": 3.36728,
+ "4870": 3.35392,
+ "4875": 3.3951,
+ "4880": 3.40327,
+ "4885": 3.3527,
+ "4890": 3.36417,
+ "4895": 3.3575,
+ "4900": 3.33425,
+ "4905": 3.33313,
+ "4910": 3.3333,
+ "4915": 3.37851,
+ "4920": 3.36029,
+ "4925": 3.3137,
+ "4930": 3.34402,
+ "4935": 3.32874,
+ "4940": 3.28941,
+ "4945": 3.36448,
+ "4950": 3.30022,
+ "4955": 3.40672,
+ "4960": 3.35022,
+ "4965": 3.3452,
+ "4970": 3.3364,
+ "4975": 3.34444,
+ "4980": 3.36905,
+ "4985": 3.35664,
+ "4990": 3.33738,
+ "4995": 3.38654,
+ "5000": 3.31162,
+ "5005": 3.36066,
+ "5010": 3.363,
+ "5015": 3.31433,
+ "5020": 3.28926,
+ "5025": 3.31836,
+ "5030": 3.32833,
+ "5035": 3.33198,
+ "5040": 3.3103,
+ "5045": 3.35142,
+ "5050": 3.30874,
+ "5055": 3.32788,
+ "5060": 3.2925,
+ "5065": 3.33721,
+ "5070": 3.33489,
+ "5075": 3.34756,
+ "5080": 3.32106,
+ "5085": 3.34571,
+ "5090": 3.32486,
+ "5095": 3.29393,
+ "5100": 3.32708,
+ "5105": 3.33097,
+ "5110": 3.33453,
+ "5115": 3.30856,
+ "5120": 3.3452,
+ "5125": 3.32066,
+ "5130": 3.32012,
+ "5135": 3.30247,
+ "5140": 3.31376,
+ "5145": 3.31273,
+ "5150": 3.32261,
+ "5155": 3.32102,
+ "5160": 3.31182,
+ "5165": 3.34786,
+ "5170": 3.23284,
+ "5175": 3.32663,
+ "5180": 3.28869,
+ "5185": 3.3108,
+ "5190": 3.33109,
+ "5195": 3.30887,
+ "5200": 3.30978,
+ "5205": 3.3476,
+ "5210": 3.28545,
+ "5215": 3.29094,
+ "5220": 3.28478,
+ "5225": 3.28969,
+ "5230": 3.32369,
+ "5235": 3.28299,
+ "5240": 3.27546,
+ "5245": 3.29768,
+ "5250": 3.30375,
+ "5255": 3.28812,
+ "5260": 3.31146,
+ "5265": 3.27116,
+ "5270": 3.25491,
+ "5275": 3.2583,
+ "5280": 3.28436,
+ "5285": 3.31072,
+ "5290": 3.2594,
+ "5295": 3.27782,
+ "5300": 3.28363,
+ "5305": 3.26809,
+ "5310": 3.32843,
+ "5315": 3.26074,
+ "5320": 3.30782,
+ "5325": 3.31251,
+ "5330": 3.27968,
+ "5335": 3.2902,
+ "5340": 3.23351,
+ "5345": 3.28679,
+ "5350": 3.28576,
+ "5355": 3.287,
+ "5360": 3.23599,
+ "5365": 3.2527,
+ "5370": 3.29068,
+ "5375": 3.27468,
+ "5380": 3.24546,
+ "5385": 3.28547,
+ "5390": 3.28476,
+ "5395": 3.20555,
+ "5400": 3.29998,
+ "5405": 3.21551,
+ "5410": 3.29499,
+ "5415": 3.22394,
+ "5420": 3.2595,
+ "5425": 3.23823,
+ "5430": 3.2497,
+ "5435": 3.28232,
+ "5440": 3.21524,
+ "5445": 3.24231,
+ "5450": 3.24927,
+ "5455": 3.2334,
+ "5460": 3.25409,
+ "5465": 3.30064,
+ "5470": 3.27224,
+ "5475": 3.20307,
+ "5480": 3.28275,
+ "5485": 3.24371,
+ "5490": 3.26867,
+ "5495": 3.2732,
+ "5500": 3.22779,
+ "5505": 3.24204,
+ "5510": 3.28644,
+ "5515": 3.27266,
+ "5520": 3.239,
+ "5525": 3.28826,
+ "5530": 3.2315,
+ "5535": 3.26318,
+ "5540": 3.25502,
+ "5545": 3.26398,
+ "5550": 3.2513,
+ "5555": 3.23126,
+ "5560": 3.22391,
+ "5565": 3.27083,
+ "5570": 3.23195,
+ "5575": 3.26705,
+ "5580": 3.2373,
+ "5585": 3.18703,
+ "5590": 3.24721,
+ "5595": 3.21104,
+ "5600": 3.25718,
+ "5605": 3.17638,
+ "5610": 3.26195,
+ "5615": 3.25742,
+ "5620": 3.2599,
+ "5625": 3.25401,
+ "5630": 3.2421,
+ "5635": 3.223,
+ "5640": 3.24438,
+ "5645": 3.2096,
+ "5650": 3.20899,
+ "5655": 3.20858,
+ "5660": 3.21268,
+ "5665": 3.21028,
+ "5670": 3.20335,
+ "5675": 3.22934,
+ "5680": 3.19951,
+ "5685": 3.20768,
+ "5690": 3.21018,
+ "5695": 3.24487,
+ "5700": 3.1981,
+ "5705": 3.18695,
+ "5710": 3.17944,
+ "5715": 3.28746,
+ "5720": 3.25186,
+ "5725": 3.19971,
+ "5730": 3.24374,
+ "5735": 3.22994,
+ "5740": 3.22768,
+ "5745": 3.20401,
+ "5750": 3.23838,
+ "5755": 3.23887,
+ "5760": 3.22955,
+ "5765": 3.22921,
+ "5770": 3.25324,
+ "5775": 3.19655,
+ "5780": 3.22013,
+ "5785": 3.22005,
+ "5790": 3.23055,
+ "5795": 3.2266,
+ "5800": 3.17013,
+ "5805": 3.18582,
+ "5810": 3.2267,
+ "5815": 3.20699,
+ "5820": 3.16302,
+ "5825": 3.20853,
+ "5830": 3.16597,
+ "5835": 3.17832,
+ "5840": 3.20996,
+ "5845": 3.21883,
+ "5850": 3.21991,
+ "5855": 3.15233,
+ "5860": 3.17324,
+ "5865": 3.20005,
+ "5870": 3.16831,
+ "5875": 3.20236,
+ "5880": 3.19534,
+ "5885": 3.1958,
+ "5890": 3.21842,
+ "5895": 3.2338,
+ "5900": 3.19108,
+ "5905": 3.22094,
+ "5910": 3.20431,
+ "5915": 3.17532,
+ "5920": 3.19367,
+ "5925": 3.16067,
+ "5930": 3.19291,
+ "5935": 3.1945,
+ "5940": 3.20786,
+ "5945": 3.21992,
+ "5950": 3.20297,
+ "5955": 3.16575,
+ "5960": 3.22662,
+ "5965": 3.17842,
+ "5970": 3.21935,
+ "5975": 3.18732,
+ "5980": 3.25613,
+ "5985": 3.14287,
+ "5990": 3.23907,
+ "5995": 3.1543,
+ "6000": 3.1753,
+ "6005": 3.1578,
+ "6010": 3.16391,
+ "6015": 3.16991,
+ "6020": 3.17233,
+ "6025": 3.20965,
+ "6030": 3.14706,
+ "6035": 3.20284,
+ "6040": 3.18195,
+ "6045": 3.19984,
+ "6050": 3.20196,
+ "6055": 3.17347,
+ "6060": 3.18853,
+ "6065": 3.21395,
+ "6070": 3.16686,
+ "6075": 3.13387,
+ "6080": 3.19195,
+ "6085": 3.15131,
+ "6090": 3.19,
+ "6095": 3.18766,
+ "6100": 3.14325,
+ "6105": 3.18999,
+ "6110": 3.13306,
+ "6115": 3.18026,
+ "6120": 3.17413,
+ "6125": 3.18011,
+ "6130": 3.16702,
+ "6135": 3.16837,
+ "6140": 3.16543,
+ "6145": 3.14325,
+ "6150": 3.1804,
+ "6155": 3.15287,
+ "6160": 3.13081,
+ "6165": 3.16081,
+ "6170": 3.14709,
+ "6175": 3.1468,
+ "6180": 3.14574,
+ "6185": 3.19045,
+ "6190": 3.15967,
+ "6195": 3.12872,
+ "6200": 3.15616,
+ "6205": 3.14627,
+ "6210": 3.102,
+ "6215": 3.15806,
+ "6220": 3.15681,
+ "6225": 3.17335,
+ "6230": 3.10912,
+ "6235": 3.14436,
+ "6240": 3.08826,
+ "6245": 3.18399,
+ "6250": 3.14572,
+ "6255": 3.15766,
+ "6260": 3.14205,
+ "6265": 3.15808,
+ "6270": 3.10086,
+ "6275": 3.12325,
+ "6280": 3.1353,
+ "6285": 3.11993,
+ "6290": 3.14704,
+ "6295": 3.15504,
+ "6300": 3.15689,
+ "6305": 3.21091,
+ "6310": 3.11366,
+ "6315": 3.11115,
+ "6320": 3.16078,
+ "6325": 3.10255,
+ "6330": 3.16283,
+ "6335": 3.15613,
+ "6340": 3.11055,
+ "6345": 3.1649,
+ "6350": 3.12018,
+ "6355": 3.11927,
+ "6360": 3.11364,
+ "6365": 3.14998,
+ "6370": 3.16162,
+ "6375": 3.13429,
+ "6380": 3.15113,
+ "6385": 3.17096,
+ "6390": 3.12715,
+ "6395": 3.10503,
+ "6400": 3.10877,
+ "6405": 3.18996,
+ "6410": 3.1755,
+ "6415": 3.12674,
+ "6420": 3.17286,
+ "6425": 3.17403,
+ "6430": 3.16661,
+ "6435": 3.12554,
+ "6440": 3.13952,
+ "6445": 3.15456,
+ "6450": 3.09231,
+ "6455": 3.08724,
+ "6460": 3.12993,
+ "6465": 3.16933,
+ "6470": 3.14139,
+ "6475": 3.13479,
+ "6480": 3.15011,
+ "6485": 3.1146,
+ "6490": 3.08101,
+ "6495": 3.16718,
+ "6500": 3.14446,
+ "6505": 3.0873,
+ "6510": 3.14803,
+ "6515": 3.16461,
+ "6520": 3.09102,
+ "6525": 3.15147,
+ "6530": 3.1121,
+ "6535": 3.12534,
+ "6540": 3.18188,
+ "6545": 3.11606,
+ "6550": 3.1127,
+ "6555": 3.10963,
+ "6560": 3.07262,
+ "6565": 3.0807,
+ "6570": 3.10704,
+ "6575": 3.05807,
+ "6580": 3.17389,
+ "6585": 3.10692,
+ "6590": 3.08633,
+ "6595": 3.10756,
+ "6600": 3.10518,
+ "6605": 3.08821,
+ "6610": 3.08539,
+ "6615": 3.13395,
+ "6620": 3.0779,
+ "6625": 3.09806,
+ "6630": 3.09476,
+ "6635": 3.1303,
+ "6640": 3.09175,
+ "6645": 3.11139,
+ "6650": 3.1397,
+ "6655": 3.07444,
+ "6660": 3.11388,
+ "6665": 3.1275,
+ "6670": 3.08259,
+ "6675": 3.10526,
+ "6680": 3.10699,
+ "6685": 3.14278,
+ "6690": 3.11769,
+ "6695": 3.12478,
+ "6700": 3.11326,
+ "6705": 3.09376,
+ "6710": 3.10953,
+ "6715": 3.06097,
+ "6720": 3.13549,
+ "6725": 3.12625,
+ "6730": 3.11089,
+ "6735": 3.11,
+ "6740": 3.11809,
+ "6745": 3.093,
+ "6750": 3.1116,
+ "6755": 3.07037,
+ "6760": 3.06754,
+ "6765": 3.08833,
+ "6770": 3.07215,
+ "6775": 3.10746,
+ "6780": 3.07523,
+ "6785": 3.08023,
+ "6790": 3.10551,
+ "6795": 3.07342,
+ "6800": 3.09848,
+ "6805": 3.08978,
+ "6810": 3.1088,
+ "6815": 3.04614,
+ "6820": 3.07698,
+ "6825": 3.10371,
+ "6830": 3.08741,
+ "6835": 3.06043,
+ "6840": 3.06761,
+ "6845": 3.11323,
+ "6850": 3.08237,
+ "6855": 3.11219,
+ "6860": 3.06318,
+ "6865": 3.1093,
+ "6870": 3.07519,
+ "6875": 3.07719,
+ "6880": 3.08835,
+ "6885": 3.05428,
+ "6890": 3.07515,
+ "6895": 3.05461,
+ "6900": 3.05868,
+ "6905": 3.07504,
+ "6910": 3.09361,
+ "6915": 3.1151,
+ "6920": 3.0674,
+ "6925": 3.08441,
+ "6930": 3.06737,
+ "6935": 3.02686,
+ "6940": 3.06983,
+ "6945": 3.05606,
+ "6950": 3.07979,
+ "6955": 3.06011,
+ "6960": 3.05591,
+ "6965": 3.09927,
+ "6970": 3.04042,
+ "6975": 3.10958,
+ "6980": 3.06872,
+ "6985": 3.06787,
+ "6990": 3.11395,
+ "6995": 3.09471,
+ "7000": 3.02889,
+ "7005": 3.10007,
+ "7010": 3.07883,
+ "7015": 3.07653,
+ "7020": 3.10062,
+ "7025": 3.08597,
+ "7030": 3.08983,
+ "7035": 3.04443,
+ "7040": 3.02192,
+ "7045": 3.08247,
+ "7050": 3.10079,
+ "7055": 3.03769,
+ "7060": 3.10062,
+ "7065": 3.11223,
+ "7070": 3.05974,
+ "7075": 3.06488,
+ "7080": 3.11517,
+ "7085": 3.03519,
+ "7090": 3.05714,
+ "7095": 3.04937,
+ "7100": 3.07015,
+ "7105": 3.0202,
+ "7110": 3.06131,
+ "7115": 3.03767,
+ "7120": 3.08113,
+ "7125": 3.03453,
+ "7130": 3.05187,
+ "7135": 3.05719,
+ "7140": 3.06287,
+ "7145": 3.07075,
+ "7150": 3.02392,
+ "7155": 3.086,
+ "7160": 3.00539,
+ "7165": 3.04352,
+ "7170": 3.07925,
+ "7175": 3.03705,
+ "7180": 3.07295,
+ "7185": 3.09312,
+ "7190": 3.05533,
+ "7195": 3.06188,
+ "7200": 3.0594,
+ "7205": 3.04368,
+ "7210": 3.08986,
+ "7215": 3.06859,
+ "7220": 3.08964,
+ "7225": 3.072,
+ "7230": 3.0749,
+ "7235": 3.05587,
+ "7240": 3.05115,
+ "7245": 3.07243,
+ "7250": 3.01397,
+ "7255": 3.0328,
+ "7260": 3.0704,
+ "7265": 3.00407,
+ "7270": 3.04195,
+ "7275": 3.04097,
+ "7280": 3.0429,
+ "7285": 3.05573,
+ "7290": 3.07459,
+ "7295": 3.06631,
+ "7300": 3.03099,
+ "7305": 3.02882,
+ "7310": 3.04953,
+ "7315": 3.07692,
+ "7320": 3.0596,
+ "7325": 3.06425,
+ "7330": 3.02645,
+ "7335": 3.02842,
+ "7340": 3.05923,
+ "7345": 3.01049,
+ "7350": 3.0623,
+ "7355": 3.04671,
+ "7360": 3.03836,
+ "7365": 3.03951,
+ "7370": 3.03321,
+ "7375": 2.99949,
+ "7380": 3.05932,
+ "7385": 3.0767,
+ "7390": 3.0644,
+ "7395": 3.02618,
+ "7400": 3.07617,
+ "7405": 3.04437,
+ "7410": 3.06207,
+ "7415": 3.05301,
+ "7420": 3.03308,
+ "7425": 3.08781,
+ "7430": 3.02975,
+ "7435": 3.02017,
+ "7440": 3.04037,
+ "7445": 3.01619,
+ "7450": 2.99541,
+ "7455": 3.0495,
+ "7460": 3.04339,
+ "7465": 3.05203,
+ "7470": 3.05691,
+ "7475": 3.06924,
+ "7480": 3.02876,
+ "7485": 2.98725,
+ "7490": 2.99068,
+ "7495": 2.99982,
+ "7500": 3.03051,
+ "7505": 3.00785,
+ "7510": 2.98042,
+ "7515": 3.02428,
+ "7520": 3.01861,
+ "7525": 2.98366,
+ "7530": 3.02761,
+ "7535": 3.04553,
+ "7540": 3.02651,
+ "7545": 3.06191,
+ "7550": 3.06608,
+ "7555": 3.00805,
+ "7560": 3.01384,
+ "7565": 3.00951,
+ "7570": 3.03712,
+ "7575": 2.98125,
+ "7580": 3.03158,
+ "7585": 3.0202,
+ "7590": 3.01599,
+ "7595": 3.07386,
+ "7600": 3.0299,
+ "7605": 3.02285,
+ "7610": 3.00589,
+ "7615": 2.99706,
+ "7620": 2.99273,
+ "7625": 3.03871,
+ "7630": 3.02131,
+ "7635": 3.01897,
+ "7640": 3.01844,
+ "7645": 3.05048,
+ "7650": 3.04473,
+ "7655": 3.09104,
+ "7660": 2.96372,
+ "7665": 3.03061,
+ "7670": 3.01563,
+ "7675": 3.00407,
+ "7680": 3.00186,
+ "7685": 3.07265,
+ "7690": 3.01302,
+ "7695": 2.99688,
+ "7700": 3.05321,
+ "7705": 3.01387,
+ "7710": 3.06054,
+ "7715": 2.99859,
+ "7720": 3.08374,
+ "7725": 2.98229,
+ "7730": 2.99949,
+ "7735": 3.02914,
+ "7740": 3.01394,
+ "7745": 3.0045,
+ "7750": 3.00934,
+ "7755": 3.02157,
+ "7760": 2.98666,
+ "7765": 3.00418,
+ "7770": 3.02859,
+ "7775": 2.98968,
+ "7780": 2.9814,
+ "7785": 3.01615,
+ "7790": 3.00013,
+ "7795": 3.02363,
+ "7800": 3.00936,
+ "7805": 3.01249,
+ "7810": 3.03324,
+ "7815": 3.00497,
+ "7820": 3.00336,
+ "7825": 3.03284,
+ "7830": 3.03186,
+ "7835": 2.9682,
+ "7840": 3.04428,
+ "7845": 2.97999,
+ "7850": 2.94029,
+ "7855": 2.98645,
+ "7860": 2.9839,
+ "7865": 3.03111,
+ "7870": 2.97095,
+ "7875": 2.98945,
+ "7880": 3.00468,
+ "7885": 2.99819,
+ "7890": 3.03999,
+ "7895": 3.03045,
+ "7900": 3.03185,
+ "7905": 3.00037,
+ "7910": 3.01124,
+ "7915": 3.026,
+ "7920": 3.01365,
+ "7925": 2.99795,
+ "7930": 3.02978,
+ "7935": 2.99122,
+ "7940": 3.03771,
+ "7945": 3.05052,
+ "7950": 2.96513,
+ "7955": 2.99009,
+ "7960": 2.97024,
+ "7965": 2.94666,
+ "7970": 2.96563,
+ "7975": 2.99602,
+ "7980": 3.01057,
+ "7985": 2.97891,
+ "7990": 2.97683,
+ "7995": 2.96275,
+ "8000": 3.02166,
+ "8005": 2.98117,
+ "8010": 2.97672,
+ "8015": 2.9667,
+ "8020": 2.97892,
+ "8025": 2.95633,
+ "8030": 2.97614,
+ "8035": 2.97496,
+ "8040": 2.95737,
+ "8045": 3.01744,
+ "8050": 3.01384,
+ "8055": 2.97526,
+ "8060": 3.00589,
+ "8065": 2.99089,
+ "8070": 2.96895,
+ "8075": 2.97876,
+ "8080": 3.01143,
+ "8085": 2.9677,
+ "8090": 2.97996,
+ "8095": 3.00373,
+ "8100": 2.95218,
+ "8105": 2.99475,
+ "8110": 2.98269,
+ "8115": 2.95866,
+ "8120": 2.97313,
+ "8125": 3.00103,
+ "8130": 2.97111,
+ "8135": 2.98888,
+ "8140": 2.96903,
+ "8145": 2.95976,
+ "8150": 2.98132,
+ "8155": 2.95234,
+ "8160": 2.99794,
+ "8165": 2.99262,
+ "8170": 2.95725,
+ "8175": 2.95481,
+ "8180": 3.01528,
+ "8185": 2.98608,
+ "8190": 3.02273,
+ "8195": 2.99712,
+ "8200": 2.96571,
+ "8205": 2.97697,
+ "8210": 2.979,
+ "8215": 2.99182,
+ "8220": 2.9707,
+ "8225": 2.96366,
+ "8230": 2.9957,
+ "8235": 3.00331,
+ "8240": 2.97471,
+ "8245": 2.97612,
+ "8250": 3.01059,
+ "8255": 2.96794,
+ "8260": 2.97389,
+ "8265": 2.95616,
+ "8270": 2.97751,
+ "8275": 2.96901,
+ "8280": 2.94216,
+ "8285": 2.98004,
+ "8290": 2.9684,
+ "8295": 2.95291,
+ "8300": 2.96575,
+ "8305": 2.97677,
+ "8310": 2.98023,
+ "8315": 2.95754,
+ "8320": 2.97907,
+ "8325": 2.92954,
+ "8330": 2.89904,
+ "8335": 2.96703,
+ "8340": 2.9933,
+ "8345": 2.94479,
+ "8350": 2.96094,
+ "8355": 2.98684,
+ "8360": 2.96756,
+ "8365": 2.98458,
+ "8370": 2.99261,
+ "8375": 2.9382,
+ "8380": 2.94228,
+ "8385": 2.97409,
+ "8390": 2.94634,
+ "8395": 2.97586,
+ "8400": 2.95907,
+ "8405": 2.97502,
+ "8410": 3.03151,
+ "8415": 2.93654,
+ "8420": 2.91908,
+ "8425": 2.97718,
+ "8430": 2.98066,
+ "8435": 2.9325,
+ "8440": 3.01244,
+ "8445": 2.99184,
+ "8450": 2.96659,
+ "8455": 2.97183,
+ "8460": 2.98115,
+ "8465": 2.92704,
+ "8470": 2.9487,
+ "8475": 2.99252,
+ "8480": 2.93092,
+ "8485": 2.94018,
+ "8490": 2.94984,
+ "8495": 2.93458,
+ "8500": 2.96965,
+ "8505": 2.92286,
+ "8510": 3.00415,
+ "8515": 2.94044,
+ "8520": 2.95865,
+ "8525": 2.88676,
+ "8530": 2.95964,
+ "8535": 2.97749,
+ "8540": 2.93209,
+ "8545": 2.95866,
+ "8550": 2.92463,
+ "8555": 2.99129,
+ "8560": 2.99564,
+ "8565": 2.95207,
+ "8570": 2.9479,
+ "8575": 2.93448,
+ "8580": 2.96844,
+ "8585": 2.97701,
+ "8590": 2.97648,
+ "8595": 2.97914,
+ "8600": 2.94846,
+ "8605": 2.94622,
+ "8610": 2.95526,
+ "8615": 2.96143,
+ "8620": 2.92468,
+ "8625": 2.94798,
+ "8630": 2.95321,
+ "8635": 2.94577,
+ "8640": 2.92557,
+ "8645": 2.98235,
+ "8650": 2.92259,
+ "8655": 2.96727,
+ "8660": 2.97138,
+ "8665": 2.95521,
+ "8670": 2.96783,
+ "8675": 2.94149,
+ "8680": 2.9362,
+ "8685": 2.94884,
+ "8690": 2.96616,
+ "8695": 2.97188,
+ "8700": 2.94906,
+ "8705": 2.91894,
+ "8710": 2.97181,
+ "8715": 2.91716,
+ "8720": 2.97451,
+ "8725": 2.95032,
+ "8730": 2.94362,
+ "8735": 2.97252,
+ "8740": 2.92827,
+ "8745": 2.9665,
+ "8750": 2.9671,
+ "8755": 2.93111,
+ "8760": 2.95003,
+ "8765": 2.91591,
+ "8770": 2.96909,
+ "8775": 2.9418,
+ "8780": 2.9288,
+ "8785": 2.94754,
+ "8790": 2.92872,
+ "8795": 2.96608,
+ "8800": 2.92801,
+ "8805": 2.90095,
+ "8810": 2.93193,
+ "8815": 2.93334,
+ "8820": 2.90591,
+ "8825": 2.9259,
+ "8830": 2.91358,
+ "8835": 2.89875,
+ "8840": 2.91579,
+ "8845": 2.92837,
+ "8850": 2.95819,
+ "8855": 2.92981,
+ "8860": 2.98933,
+ "8865": 2.93395,
+ "8870": 2.90896,
+ "8875": 2.92285,
+ "8880": 2.92995,
+ "8885": 2.92221,
+ "8890": 2.94209,
+ "8895": 2.92252,
+ "8900": 2.94589,
+ "8905": 2.93689,
+ "8910": 2.92024,
+ "8915": 2.90453,
+ "8920": 2.91089,
+ "8925": 2.97639,
+ "8930": 2.96353,
+ "8935": 2.97249,
+ "8940": 2.94921,
+ "8945": 2.94907,
+ "8950": 2.93317,
+ "8955": 2.91722,
+ "8960": 2.89897,
+ "8965": 2.92883,
+ "8970": 2.94197,
+ "8975": 2.90617,
+ "8980": 2.89982,
+ "8985": 2.92146,
+ "8990": 2.96637,
+ "8995": 2.94208,
+ "9000": 2.89552,
+ "9005": 2.93923,
+ "9010": 2.97958,
+ "9015": 2.90334,
+ "9020": 2.90386,
+ "9025": 2.92295,
+ "9030": 2.9452,
+ "9035": 2.85861,
+ "9040": 2.93589,
+ "9045": 2.92513,
+ "9050": 2.96167,
+ "9055": 2.88856,
+ "9060": 2.95652,
+ "9065": 2.98676,
+ "9070": 2.92813,
+ "9075": 2.94286,
+ "9080": 2.93443,
+ "9085": 2.94469,
+ "9090": 2.93672,
+ "9095": 2.89927,
+ "9100": 2.90143,
+ "9105": 2.8916,
+ "9110": 2.93366,
+ "9115": 2.93998,
+ "9120": 2.97531,
+ "9125": 2.92878,
+ "9130": 2.92359,
+ "9135": 2.94142,
+ "9140": 2.94708,
+ "9145": 2.89442,
+ "9150": 2.9236,
+ "9155": 2.93238,
+ "9160": 2.93723,
+ "9165": 2.92645,
+ "9170": 2.94957,
+ "9175": 2.88835,
+ "9180": 2.93472,
+ "9185": 2.89552,
+ "9190": 2.94987,
+ "9195": 2.91328,
+ "9200": 2.9332,
+ "9205": 2.88835,
+ "9210": 2.93329,
+ "9215": 2.87972,
+ "9220": 2.9037,
+ "9225": 2.93529,
+ "9230": 2.86673,
+ "9235": 2.87947,
+ "9240": 2.89574,
+ "9245": 2.88473,
+ "9250": 2.88192,
+ "9255": 2.91278,
+ "9260": 2.87918,
+ "9265": 2.92251,
+ "9270": 2.89761,
+ "9275": 2.91399,
+ "9280": 2.91944,
+ "9285": 2.91784,
+ "9290": 2.93193,
+ "9295": 2.92972,
+ "9300": 2.87994,
+ "9305": 2.90929,
+ "9310": 2.89815,
+ "9315": 2.86654,
+ "9320": 2.86155,
+ "9325": 2.90528,
+ "9330": 2.95595,
+ "9335": 2.87597,
+ "9340": 2.93933,
+ "9345": 2.94762,
+ "9350": 2.91386,
+ "9355": 2.87826,
+ "9360": 2.89811,
+ "9365": 2.88261,
+ "9370": 2.93514,
+ "9375": 2.91182,
+ "9380": 2.86528,
+ "9385": 2.91427,
+ "9390": 2.92426,
+ "9395": 2.9204,
+ "9400": 2.89667,
+ "9405": 2.89248,
+ "9410": 2.91861,
+ "9415": 2.91768,
+ "9420": 2.89307,
+ "9425": 2.90073,
+ "9430": 2.87891,
+ "9435": 2.90423,
+ "9440": 2.89586,
+ "9445": 2.88422,
+ "9450": 2.89372,
+ "9455": 2.89147,
+ "9460": 2.94343,
+ "9465": 2.94716,
+ "9470": 2.88659,
+ "9475": 2.94008,
+ "9480": 2.88994,
+ "9485": 2.87951,
+ "9490": 2.89764,
+ "9495": 2.92355,
+ "9500": 2.8961,
+ "9505": 2.86881,
+ "9510": 2.89367,
+ "9515": 2.90496,
+ "9520": 2.91178,
+ "9525": 2.89163,
+ "9530": 2.888,
+ "9535": 2.91326
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 9535,
+ "step_interval": 5,
+ "values": {
+ "1": 1021639936.0,
+ "5": 1024063360.0,
+ "10": 1014250240.0,
+ "15": 1024077312.0,
+ "20": 1022485888.0,
+ "25": 1041373312.0,
+ "30": 1028112960.0,
+ "35": 1035624704.0,
+ "40": 1026328576.0,
+ "45": 1022350272.0,
+ "50": 1030098304.0,
+ "55": 1028966144.0,
+ "60": 1036320640.0,
+ "65": 1034679104.0,
+ "70": 1029375232.0,
+ "75": 1028744768.0,
+ "80": 1047575296.0,
+ "85": 1029448192.0,
+ "90": 1020467520.0,
+ "95": 1028309632.0,
+ "100": 1040961536.0,
+ "105": 1039436032.0,
+ "110": 1026879360.0,
+ "115": 1052313216.0,
+ "120": 1018862976.0,
+ "125": 1045372672.0,
+ "130": 1034330240.0,
+ "135": 1016616192.0,
+ "140": 1038582016.0,
+ "145": 1020688128.0,
+ "150": 1039787264.0,
+ "155": 1032796672.0,
+ "160": 1020953088.0,
+ "165": 1032424448.0,
+ "170": 1017396864.0,
+ "175": 1033427264.0,
+ "180": 1036118848.0,
+ "185": 1030573824.0,
+ "190": 1035673728.0,
+ "195": 1034555136.0,
+ "200": 1040973696.0,
+ "205": 1048501120.0,
+ "210": 1054480896.0,
+ "215": 1025159680.0,
+ "220": 1044962432.0,
+ "225": 1038076032.0,
+ "230": 1026222464.0,
+ "235": 1051134912.0,
+ "240": 1029276160.0,
+ "245": 1031398080.0,
+ "250": 1027879680.0,
+ "255": 1016929344.0,
+ "260": 1045008768.0,
+ "265": 1021329984.0,
+ "270": 1030964096.0,
+ "275": 1036911360.0,
+ "280": 1031742976.0,
+ "285": 1015015424.0,
+ "290": 1018756672.0,
+ "295": 1017237632.0,
+ "300": 1034761344.0,
+ "305": 1032165824.0,
+ "310": 1035583360.0,
+ "315": 1012733952.0,
+ "320": 1008275072.0,
+ "325": 1042742272.0,
+ "330": 1042870656.0,
+ "335": 1033509312.0,
+ "340": 1014465024.0,
+ "345": 1042618624.0,
+ "350": 1031852800.0,
+ "355": 1050843904.0,
+ "360": 1030258240.0,
+ "365": 1034595200.0,
+ "370": 1019435136.0,
+ "375": 1022143680.0,
+ "380": 1021325184.0,
+ "385": 1025589248.0,
+ "390": 1023195072.0,
+ "395": 1019653120.0,
+ "400": 1033520384.0,
+ "405": 1023880256.0,
+ "410": 1017908480.0,
+ "415": 1024288128.0,
+ "420": 1020623744.0,
+ "425": 1025854336.0,
+ "430": 1033854208.0,
+ "435": 1028181888.0,
+ "440": 1022090240.0,
+ "445": 1036767936.0,
+ "450": 1024999296.0,
+ "455": 1013852096.0,
+ "460": 1022094528.0,
+ "465": 1041431872.0,
+ "470": 1029038016.0,
+ "475": 1010064896.0,
+ "480": 1047608064.0,
+ "485": 1029724544.0,
+ "490": 1044667584.0,
+ "495": 1025229440.0,
+ "500": 1037464320.0,
+ "505": 1032181376.0,
+ "510": 1042853312.0,
+ "515": 1026159616.0,
+ "520": 1013409920.0,
+ "525": 1035147392.0,
+ "530": 1016374912.0,
+ "535": 1040112768.0,
+ "540": 1035052160.0,
+ "545": 1032114048.0,
+ "550": 1018673664.0,
+ "555": 1008638592.0,
+ "560": 1011927680.0,
+ "565": 1041824128.0,
+ "570": 1034942336.0,
+ "575": 1010199040.0,
+ "580": 1032210048.0,
+ "585": 1041262592.0,
+ "590": 1038868416.0,
+ "595": 1035742912.0,
+ "600": 1023772800.0,
+ "605": 1032294592.0,
+ "610": 1037749248.0,
+ "615": 1005974912.0,
+ "620": 1040407552.0,
+ "625": 1045208960.0,
+ "630": 1034414592.0,
+ "635": 1028522176.0,
+ "640": 1022644672.0,
+ "645": 1035875456.0,
+ "650": 1009255552.0,
+ "655": 997757824.0,
+ "660": 1029710592.0,
+ "665": 1025532608.0,
+ "670": 1048812416.0,
+ "675": 1025202688.0,
+ "680": 1019340160.0,
+ "685": 1027832192.0,
+ "690": 1029230080.0,
+ "695": 1040024448.0,
+ "700": 1042031616.0,
+ "705": 1034383360.0,
+ "710": 1020442496.0,
+ "715": 1031472512.0,
+ "720": 1040274688.0,
+ "725": 1023280128.0,
+ "730": 1022792704.0,
+ "735": 1025085952.0,
+ "740": 1038382208.0,
+ "745": 1045204864.0,
+ "750": 1013181376.0,
+ "755": 1031643904.0,
+ "760": 1032783872.0,
+ "765": 1027136256.0,
+ "770": 1023967232.0,
+ "775": 1025895104.0,
+ "780": 1038166400.0,
+ "785": 1025486848.0,
+ "790": 1040811136.0,
+ "795": 1032531584.0,
+ "800": 1039592704.0,
+ "805": 1024317440.0,
+ "810": 1034724736.0,
+ "815": 1036000896.0,
+ "820": 1035671296.0,
+ "825": 1051375360.0,
+ "830": 1035406720.0,
+ "835": 1022547456.0,
+ "840": 1036876288.0,
+ "845": 1025700992.0,
+ "850": 1048529344.0,
+ "855": 1014985472.0,
+ "860": 1033098880.0,
+ "865": 1031542784.0,
+ "870": 1040902848.0,
+ "875": 1023937664.0,
+ "880": 1028396288.0,
+ "885": 1054407552.0,
+ "890": 1019536512.0,
+ "895": 1045189888.0,
+ "900": 1031772608.0,
+ "905": 1020971904.0,
+ "910": 1031385600.0,
+ "915": 1032926976.0,
+ "920": 1038460224.0,
+ "925": 1026755200.0,
+ "930": 1025378432.0,
+ "935": 1031126976.0,
+ "940": 1057933440.0,
+ "945": 1029822464.0,
+ "950": 1014412416.0,
+ "955": 1032174208.0,
+ "960": 1026152064.0,
+ "965": 1062678976.0,
+ "970": 1030095232.0,
+ "975": 1036904704.0,
+ "980": 1027049472.0,
+ "985": 1030676736.0,
+ "990": 1020676480.0,
+ "995": 1042301824.0,
+ "1000": 1036832768.0,
+ "1005": 1050207488.0,
+ "1010": 1023802624.0,
+ "1015": 1020540352.0,
+ "1020": 1042587648.0,
+ "1025": 1037943936.0,
+ "1030": 1049209728.0,
+ "1035": 1012483456.0,
+ "1040": 1023092352.0,
+ "1045": 1039521024.0,
+ "1050": 1026826368.0,
+ "1055": 1034861248.0,
+ "1060": 1046128896.0,
+ "1065": 1036804800.0,
+ "1070": 1019995392.0,
+ "1075": 1025342336.0,
+ "1080": 1014979648.0,
+ "1085": 1030008320.0,
+ "1090": 1029062272.0,
+ "1095": 1020310080.0,
+ "1100": 1039835264.0,
+ "1105": 1048599936.0,
+ "1110": 1020704832.0,
+ "1115": 1024782720.0,
+ "1120": 1061895808.0,
+ "1125": 1043311360.0,
+ "1130": 1031219584.0,
+ "1135": 1041360576.0,
+ "1140": 1021486464.0,
+ "1145": 1051696384.0,
+ "1150": 1035590976.0,
+ "1155": 1029590784.0,
+ "1160": 1042565312.0,
+ "1165": 1026811264.0,
+ "1170": 1018001152.0,
+ "1175": 1033684224.0,
+ "1180": 1035634688.0,
+ "1185": 1023929344.0,
+ "1190": 1033160320.0,
+ "1195": 1024229696.0,
+ "1200": 1039116800.0,
+ "1205": 1031741568.0,
+ "1210": 1053249984.0,
+ "1215": 1024616960.0,
+ "1220": 1009041152.0,
+ "1225": 1036679872.0,
+ "1230": 1041258624.0,
+ "1235": 1053975040.0,
+ "1240": 1030357376.0,
+ "1245": 1017685056.0,
+ "1250": 1022772992.0,
+ "1255": 1033438976.0,
+ "1260": 1034284288.0,
+ "1265": 1034003328.0,
+ "1270": 1037322752.0,
+ "1275": 1029346176.0,
+ "1280": 1046490048.0,
+ "1285": 1028284992.0,
+ "1290": 1036577728.0,
+ "1295": 1032421120.0,
+ "1300": 1033065088.0,
+ "1305": 1030026688.0,
+ "1310": 1051263488.0,
+ "1315": 1035372992.0,
+ "1320": 1028263808.0,
+ "1325": 1049972352.0,
+ "1330": 1030132992.0,
+ "1335": 1031165312.0,
+ "1340": 1012759360.0,
+ "1345": 1044640576.0,
+ "1350": 1034956800.0,
+ "1355": 1033623424.0,
+ "1360": 1036683648.0,
+ "1365": 1038590464.0,
+ "1370": 1039851072.0,
+ "1375": 1034118272.0,
+ "1380": 1022886016.0,
+ "1385": 1018084928.0,
+ "1390": 1049053376.0,
+ "1395": 1034870080.0,
+ "1400": 1034998656.0,
+ "1405": 1034129408.0,
+ "1410": 1036367168.0,
+ "1415": 1043576768.0,
+ "1420": 1026110016.0,
+ "1425": 1033320832.0,
+ "1430": 1012807040.0,
+ "1435": 1038395392.0,
+ "1440": 1020970432.0,
+ "1445": 1032460032.0,
+ "1450": 1014039360.0,
+ "1455": 1011673856.0,
+ "1460": 1043275392.0,
+ "1465": 1014361728.0,
+ "1470": 1020655616.0,
+ "1475": 1030231168.0,
+ "1480": 1029369280.0,
+ "1485": 1022997888.0,
+ "1490": 1026783360.0,
+ "1495": 1021816320.0,
+ "1500": 1027177728.0,
+ "1505": 1034882176.0,
+ "1510": 1014397312.0,
+ "1515": 1042137792.0,
+ "1520": 1025793984.0,
+ "1525": 1036336704.0,
+ "1530": 1039949056.0,
+ "1535": 1047639808.0,
+ "1540": 1043539328.0,
+ "1545": 1034044672.0,
+ "1550": 1016110400.0,
+ "1555": 1015573120.0,
+ "1560": 1055022848.0,
+ "1565": 1015593088.0,
+ "1570": 1018245120.0,
+ "1575": 1032516288.0,
+ "1580": 1012984064.0,
+ "1585": 1025326080.0,
+ "1590": 1034127936.0,
+ "1595": 1057394560.0,
+ "1600": 1026867072.0,
+ "1605": 1019994240.0,
+ "1610": 1031267712.0,
+ "1615": 1035274624.0,
+ "1620": 1018016128.0,
+ "1625": 1028272512.0,
+ "1630": 1027203904.0,
+ "1635": 1023799360.0,
+ "1640": 1034120256.0,
+ "1645": 1021814016.0,
+ "1650": 1015262912.0,
+ "1655": 1018281088.0,
+ "1660": 1047983232.0,
+ "1665": 1027060736.0,
+ "1670": 1048219456.0,
+ "1675": 1021104768.0,
+ "1680": 1043287488.0,
+ "1685": 1052719488.0,
+ "1690": 1026724736.0,
+ "1695": 1040385536.0,
+ "1700": 1018036608.0,
+ "1705": 1020479872.0,
+ "1710": 1021024640.0,
+ "1715": 1026931392.0,
+ "1720": 1028351168.0,
+ "1725": 1034363008.0,
+ "1730": 1013692288.0,
+ "1735": 1018430976.0,
+ "1740": 1057258240.0,
+ "1745": 1029260928.0,
+ "1750": 1024358016.0,
+ "1755": 1029969920.0,
+ "1760": 1022193920.0,
+ "1765": 1040478272.0,
+ "1770": 1029669248.0,
+ "1775": 1046197312.0,
+ "1780": 1021956480.0,
+ "1785": 1035109376.0,
+ "1790": 1028263680.0,
+ "1795": 1031022208.0,
+ "1800": 1028300736.0,
+ "1805": 1025668736.0,
+ "1810": 1021555520.0,
+ "1815": 1033438976.0,
+ "1820": 1034886592.0,
+ "1825": 1020206912.0,
+ "1830": 1013885248.0,
+ "1835": 1031381760.0,
+ "1840": 1040392000.0,
+ "1845": 1034828480.0,
+ "1850": 1014481344.0,
+ "1855": 1019419648.0,
+ "1860": 1019570688.0,
+ "1865": 1035942016.0,
+ "1870": 1026243520.0,
+ "1875": 1031525248.0,
+ "1880": 1011590400.0,
+ "1885": 1041065536.0,
+ "1890": 1035000768.0,
+ "1895": 1028958848.0,
+ "1900": 1033996928.0,
+ "1905": 1027123776.0,
+ "1910": 1029217920.0,
+ "1915": 1030493312.0,
+ "1920": 1042920128.0,
+ "1925": 1038420544.0,
+ "1930": 1019303744.0,
+ "1935": 1032535232.0,
+ "1940": 1027804544.0,
+ "1945": 1034204928.0,
+ "1950": 1006036608.0,
+ "1955": 1032578752.0,
+ "1960": 1015721984.0,
+ "1965": 1029088896.0,
+ "1970": 1021553792.0,
+ "1975": 1034047552.0,
+ "1980": 1029367616.0,
+ "1985": 1027785472.0,
+ "1990": 1020946688.0,
+ "1995": 1010423232.0,
+ "2000": 1039617536.0,
+ "2005": 1001485440.0,
+ "2010": 1020423168.0,
+ "2015": 1032034688.0,
+ "2020": 1036299008.0,
+ "2025": 1037171968.0,
+ "2030": 1029769280.0,
+ "2035": 1040333952.0,
+ "2040": 1030114176.0,
+ "2045": 1032700288.0,
+ "2050": 1008016000.0,
+ "2055": 1045724032.0,
+ "2060": 1028142336.0,
+ "2065": 1038800448.0,
+ "2070": 1045644800.0,
+ "2075": 1035237120.0,
+ "2080": 1022880512.0,
+ "2085": 1024816512.0,
+ "2090": 1034362944.0,
+ "2095": 1005220288.0,
+ "2100": 1034643968.0,
+ "2105": 1035582720.0,
+ "2110": 1030685504.0,
+ "2115": 1029798400.0,
+ "2120": 1018845184.0,
+ "2125": 1021863808.0,
+ "2130": 1026636416.0,
+ "2135": 1053279296.0,
+ "2140": 1017061248.0,
+ "2145": 1019633920.0,
+ "2150": 1037130112.0,
+ "2155": 1033302720.0,
+ "2160": 1049035648.0,
+ "2165": 1039681856.0,
+ "2170": 1020308352.0,
+ "2175": 1027338304.0,
+ "2180": 1041702528.0,
+ "2185": 1028895616.0,
+ "2190": 1029309824.0,
+ "2195": 1028945024.0,
+ "2200": 1039639424.0,
+ "2205": 1036972032.0,
+ "2210": 1031740672.0,
+ "2215": 1021404544.0,
+ "2220": 1020911232.0,
+ "2225": 1033403136.0,
+ "2230": 1014200448.0,
+ "2235": 1029396224.0,
+ "2240": 1029885568.0,
+ "2245": 1026006656.0,
+ "2250": 1046268672.0,
+ "2255": 1032952576.0,
+ "2260": 1047495424.0,
+ "2265": 1023720448.0,
+ "2270": 1022566656.0,
+ "2275": 1028536832.0,
+ "2280": 1034972800.0,
+ "2285": 1031820544.0,
+ "2290": 1038649344.0,
+ "2295": 1028815872.0,
+ "2300": 1034450944.0,
+ "2305": 1032314560.0,
+ "2310": 1013586688.0,
+ "2315": 1048182336.0,
+ "2320": 1035209984.0,
+ "2325": 1046965376.0,
+ "2330": 1014696128.0,
+ "2335": 1027382272.0,
+ "2340": 1036736320.0,
+ "2345": 1020186496.0,
+ "2350": 1031016832.0,
+ "2355": 1037474816.0,
+ "2360": 1032608256.0,
+ "2365": 1028043520.0,
+ "2370": 1021002752.0,
+ "2375": 1022912000.0,
+ "2380": 1048556032.0,
+ "2385": 1044141824.0,
+ "2390": 1021986688.0,
+ "2395": 1020595328.0,
+ "2400": 1026931456.0,
+ "2405": 1038386176.0,
+ "2410": 1045395200.0,
+ "2415": 1048455936.0,
+ "2420": 1032227136.0,
+ "2425": 1029562496.0,
+ "2430": 1030386368.0,
+ "2435": 1029217408.0,
+ "2440": 1029168384.0,
+ "2445": 1033132032.0,
+ "2450": 1038557824.0,
+ "2455": 1034721984.0,
+ "2460": 1039984768.0,
+ "2465": 1032501504.0,
+ "2470": 1024144384.0,
+ "2475": 1016539072.0,
+ "2480": 1023611776.0,
+ "2485": 1021032192.0,
+ "2490": 1035921024.0,
+ "2495": 1032967744.0,
+ "2500": 1028107264.0,
+ "2505": 1015385856.0,
+ "2510": 1030968064.0,
+ "2515": 1025699072.0,
+ "2520": 1033325504.0,
+ "2525": 1029692800.0,
+ "2530": 1023986688.0,
+ "2535": 1071070528.0,
+ "2540": 1024537408.0,
+ "2545": 1033797760.0,
+ "2550": 1029448640.0,
+ "2555": 1029182336.0,
+ "2560": 1018115712.0,
+ "2565": 1031598592.0,
+ "2570": 1022845952.0,
+ "2575": 1026503552.0,
+ "2580": 1038622336.0,
+ "2585": 1025900224.0,
+ "2590": 1026100096.0,
+ "2595": 1046622592.0,
+ "2600": 1031104256.0,
+ "2605": 1001910272.0,
+ "2610": 1028423232.0,
+ "2615": 1025563904.0,
+ "2620": 1038651328.0,
+ "2625": 1026996480.0,
+ "2630": 1036830912.0,
+ "2635": 1021197824.0,
+ "2640": 1021865984.0,
+ "2645": 1039153536.0,
+ "2650": 1025943552.0,
+ "2655": 1013255744.0,
+ "2660": 1032646016.0,
+ "2665": 1035218496.0,
+ "2670": 1036437824.0,
+ "2675": 1039297344.0,
+ "2680": 1041661568.0,
+ "2685": 1034565760.0,
+ "2690": 1058871552.0,
+ "2695": 1019880192.0,
+ "2700": 1062627072.0,
+ "2705": 1035376064.0,
+ "2710": 1019542272.0,
+ "2715": 1031885568.0,
+ "2720": 1016404736.0,
+ "2725": 1040594688.0,
+ "2730": 1019586496.0,
+ "2735": 1030889472.0,
+ "2740": 1029291520.0,
+ "2745": 1040686848.0,
+ "2750": 1023880320.0,
+ "2755": 1011866752.0,
+ "2760": 1027685248.0,
+ "2765": 1030882368.0,
+ "2770": 1033119360.0,
+ "2775": 1026330752.0,
+ "2780": 1033684096.0,
+ "2785": 1024588416.0,
+ "2790": 1033735168.0,
+ "2795": 1045948608.0,
+ "2800": 1040286656.0,
+ "2805": 1019944704.0,
+ "2810": 1031448704.0,
+ "2815": 1030932992.0,
+ "2820": 1037857024.0,
+ "2825": 1041684352.0,
+ "2830": 1030460288.0,
+ "2835": 1013508288.0,
+ "2840": 1031448064.0,
+ "2845": 1030129536.0,
+ "2850": 1026618752.0,
+ "2855": 1024703872.0,
+ "2860": 1031700160.0,
+ "2865": 1027428928.0,
+ "2870": 1026688384.0,
+ "2875": 1012776960.0,
+ "2880": 1038300480.0,
+ "2885": 1017902592.0,
+ "2890": 1044199424.0,
+ "2895": 1036459776.0,
+ "2900": 1030653632.0,
+ "2905": 1035958016.0,
+ "2910": 1038717952.0,
+ "2915": 1039384064.0,
+ "2920": 1034782336.0,
+ "2925": 1043268608.0,
+ "2930": 1038228672.0,
+ "2935": 1021220736.0,
+ "2940": 1042308992.0,
+ "2945": 1045232128.0,
+ "2950": 1047526272.0,
+ "2955": 1034173312.0,
+ "2960": 1020891008.0,
+ "2965": 1027305856.0,
+ "2970": 1038796288.0,
+ "2975": 1034007936.0,
+ "2980": 1049591296.0,
+ "2985": 1034846848.0,
+ "2990": 1026008704.0,
+ "2995": 1034918912.0,
+ "3000": 1039018240.0,
+ "3005": 1038157312.0,
+ "3010": 1010908160.0,
+ "3015": 1044975552.0,
+ "3020": 1034050688.0,
+ "3025": 1037764928.0,
+ "3030": 1027722112.0,
+ "3035": 1041820544.0,
+ "3040": 1035312384.0,
+ "3045": 1027256576.0,
+ "3050": 1029708544.0,
+ "3055": 1028029824.0,
+ "3060": 1049977984.0,
+ "3065": 1024067456.0,
+ "3070": 1011545344.0,
+ "3075": 1042847040.0,
+ "3080": 1036094144.0,
+ "3085": 1030388480.0,
+ "3090": 1035262208.0,
+ "3095": 1013802688.0,
+ "3100": 1030145152.0,
+ "3105": 1017608704.0,
+ "3110": 1033369984.0,
+ "3115": 1023737472.0,
+ "3120": 1024878336.0,
+ "3125": 1046537984.0,
+ "3130": 1024676480.0,
+ "3135": 1025723008.0,
+ "3140": 1043779328.0,
+ "3145": 1044374144.0,
+ "3150": 1016483072.0,
+ "3155": 1042488768.0,
+ "3160": 1026833600.0,
+ "3165": 1031199232.0,
+ "3170": 1024332544.0,
+ "3175": 1024369152.0,
+ "3180": 1018204544.0,
+ "3185": 1034353408.0,
+ "3190": 1019222272.0,
+ "3195": 1028425408.0,
+ "3200": 1036080384.0,
+ "3205": 1016075328.0,
+ "3210": 1034110080.0,
+ "3215": 1031350016.0,
+ "3220": 1040833152.0,
+ "3225": 1022835200.0,
+ "3230": 1033257984.0,
+ "3235": 1019974656.0,
+ "3240": 1038131072.0,
+ "3245": 1031643264.0,
+ "3250": 1022390656.0,
+ "3255": 1032876160.0,
+ "3260": 1037751040.0,
+ "3265": 1021623936.0,
+ "3270": 1031241984.0,
+ "3275": 1038461440.0,
+ "3280": 1023236480.0,
+ "3285": 1031615360.0,
+ "3290": 1045248128.0,
+ "3295": 1043178496.0,
+ "3300": 1035084800.0,
+ "3305": 1042661376.0,
+ "3310": 1058092928.0,
+ "3315": 1024281920.0,
+ "3320": 1046014912.0,
+ "3325": 1023180288.0,
+ "3330": 1048037120.0,
+ "3335": 1036691008.0,
+ "3340": 1042122880.0,
+ "3345": 1030898368.0,
+ "3350": 1020619968.0,
+ "3355": 1025960832.0,
+ "3360": 1030304832.0,
+ "3365": 1031171200.0,
+ "3370": 1036453952.0,
+ "3375": 1023473280.0,
+ "3380": 1032382464.0,
+ "3385": 1038081024.0,
+ "3390": 1052809728.0,
+ "3395": 1012090624.0,
+ "3400": 1019208704.0,
+ "3405": 1021781440.0,
+ "3410": 1028434432.0,
+ "3415": 1058221760.0,
+ "3420": 1033492800.0,
+ "3425": 1029580288.0,
+ "3430": 1021150336.0,
+ "3435": 1034992128.0,
+ "3440": 1017961216.0,
+ "3445": 1025538368.0,
+ "3450": 1032253248.0,
+ "3455": 1036260480.0,
+ "3460": 1052071872.0,
+ "3465": 1027113728.0,
+ "3470": 1043729408.0,
+ "3475": 1033264704.0,
+ "3480": 1026620160.0,
+ "3485": 1029215616.0,
+ "3490": 1041040640.0,
+ "3495": 1019252800.0,
+ "3500": 1032060288.0,
+ "3505": 1025752512.0,
+ "3510": 1044367488.0,
+ "3515": 1013818880.0,
+ "3520": 1021847168.0,
+ "3525": 1032175616.0,
+ "3530": 1029789120.0,
+ "3535": 1034569600.0,
+ "3540": 1017731712.0,
+ "3545": 1035659392.0,
+ "3550": 1024534528.0,
+ "3555": 1035866496.0,
+ "3560": 1029739008.0,
+ "3565": 1028900736.0,
+ "3570": 1046029632.0,
+ "3575": 1039185408.0,
+ "3580": 1010837632.0,
+ "3585": 1031738176.0,
+ "3590": 1041450496.0,
+ "3595": 1037637440.0,
+ "3600": 1032763712.0,
+ "3605": 1045822208.0,
+ "3610": 1039235456.0,
+ "3615": 1036870912.0,
+ "3620": 1026929600.0,
+ "3625": 1033931072.0,
+ "3630": 1017582784.0,
+ "3635": 1026629376.0,
+ "3640": 1039529472.0,
+ "3645": 1022657152.0,
+ "3650": 1036842048.0,
+ "3655": 1023990144.0,
+ "3660": 1014988160.0,
+ "3665": 1026118272.0,
+ "3670": 1041672064.0,
+ "3675": 1033250816.0,
+ "3680": 1015353280.0,
+ "3685": 1029123264.0,
+ "3690": 1026204736.0,
+ "3695": 1043799616.0,
+ "3700": 1028613632.0,
+ "3705": 1049485248.0,
+ "3710": 1027179904.0,
+ "3715": 1016134784.0,
+ "3720": 1040818688.0,
+ "3725": 1032763904.0,
+ "3730": 1030920960.0,
+ "3735": 1019009984.0,
+ "3740": 1023824384.0,
+ "3745": 1046288192.0,
+ "3750": 1034462016.0,
+ "3755": 1032088960.0,
+ "3760": 1019366528.0,
+ "3765": 1031917120.0,
+ "3770": 1026678400.0,
+ "3775": 1035708544.0,
+ "3780": 1030670656.0,
+ "3785": 1027208128.0,
+ "3790": 1019583488.0,
+ "3795": 1030305536.0,
+ "3800": 1035615488.0,
+ "3805": 1035422592.0,
+ "3810": 1033293888.0,
+ "3815": 1033988224.0,
+ "3820": 1041106176.0,
+ "3825": 1024534656.0,
+ "3830": 1037631040.0,
+ "3835": 1040347008.0,
+ "3840": 1023445696.0,
+ "3845": 1048467840.0,
+ "3850": 1052489856.0,
+ "3855": 1028907456.0,
+ "3860": 1019531200.0,
+ "3865": 1035487744.0,
+ "3870": 1028491520.0,
+ "3875": 1041165568.0,
+ "3880": 1048855040.0,
+ "3885": 1027724800.0,
+ "3890": 1027488000.0,
+ "3895": 1034190976.0,
+ "3900": 1027646336.0,
+ "3905": 1027976192.0,
+ "3910": 1041571968.0,
+ "3915": 1043996288.0,
+ "3920": 1041063616.0,
+ "3925": 1030835840.0,
+ "3930": 1027072896.0,
+ "3935": 1033782016.0,
+ "3940": 1042274560.0,
+ "3945": 1036247232.0,
+ "3950": 1021432448.0,
+ "3955": 1036303104.0,
+ "3960": 1024184704.0,
+ "3965": 1027065856.0,
+ "3970": 1015984768.0,
+ "3975": 1041422976.0,
+ "3980": 1032455232.0,
+ "3985": 1037681280.0,
+ "3990": 1038684032.0,
+ "3995": 1023654272.0,
+ "4000": 1054410496.0,
+ "4005": 1029983488.0,
+ "4010": 1025137920.0,
+ "4015": 1030978176.0,
+ "4020": 1018472512.0,
+ "4025": 1027123328.0,
+ "4030": 1010307328.0,
+ "4035": 1038641600.0,
+ "4040": 1022256768.0,
+ "4045": 1025037248.0,
+ "4050": 1032349568.0,
+ "4055": 1022420992.0,
+ "4060": 1024521280.0,
+ "4065": 1032870464.0,
+ "4070": 1027790272.0,
+ "4075": 1025596672.0,
+ "4080": 1029366528.0,
+ "4085": 1020822912.0,
+ "4090": 1033323136.0,
+ "4095": 1024142016.0,
+ "4100": 1040949824.0,
+ "4105": 1027266816.0,
+ "4110": 1038791296.0,
+ "4115": 1023496768.0,
+ "4120": 1038942976.0,
+ "4125": 1048275136.0,
+ "4130": 1021490240.0,
+ "4135": 1034570496.0,
+ "4140": 1034613568.0,
+ "4145": 1044446976.0,
+ "4150": 1000352896.0,
+ "4155": 1028364032.0,
+ "4160": 1024242176.0,
+ "4165": 1033688960.0,
+ "4170": 1018887936.0,
+ "4175": 1026492672.0,
+ "4180": 1045408128.0,
+ "4185": 1033632640.0,
+ "4190": 1029574656.0,
+ "4195": 1038777472.0,
+ "4200": 1025102592.0,
+ "4205": 1019074816.0,
+ "4210": 1029560192.0,
+ "4215": 1032269888.0,
+ "4220": 1026241024.0,
+ "4225": 1031926848.0,
+ "4230": 1030270336.0,
+ "4235": 1027603904.0,
+ "4240": 1031480512.0,
+ "4245": 1028765184.0,
+ "4250": 1026987648.0,
+ "4255": 1021238784.0,
+ "4260": 1042082560.0,
+ "4265": 1025409664.0,
+ "4270": 1030170816.0,
+ "4275": 1012473216.0,
+ "4280": 1044505088.0,
+ "4285": 1019897728.0,
+ "4290": 1033058624.0,
+ "4295": 1033595904.0,
+ "4300": 1031637952.0,
+ "4305": 1023848320.0,
+ "4310": 1021569216.0,
+ "4315": 1047221504.0,
+ "4320": 1026520704.0,
+ "4325": 1005865472.0,
+ "4330": 1037667200.0,
+ "4335": 1022006400.0,
+ "4340": 1029010048.0,
+ "4345": 1033474752.0,
+ "4350": 1036886272.0,
+ "4355": 1026808256.0,
+ "4360": 1022938752.0,
+ "4365": 1028779264.0,
+ "4370": 1029623808.0,
+ "4375": 1042197504.0,
+ "4380": 1016100160.0,
+ "4385": 1045551232.0,
+ "4390": 1026270976.0,
+ "4395": 1029797248.0,
+ "4400": 1047365376.0,
+ "4405": 1029297344.0,
+ "4410": 1033423744.0,
+ "4415": 1028299520.0,
+ "4420": 1028148672.0,
+ "4425": 1033574400.0,
+ "4430": 1031374464.0,
+ "4435": 1028572416.0,
+ "4440": 1033123072.0,
+ "4445": 1028292992.0,
+ "4450": 1052211520.0,
+ "4455": 1026286784.0,
+ "4460": 1034885888.0,
+ "4465": 1031725568.0,
+ "4470": 1035446400.0,
+ "4475": 1036971648.0,
+ "4480": 1025118464.0,
+ "4485": 1034105152.0,
+ "4490": 1024630656.0,
+ "4495": 1047974272.0,
+ "4500": 1024707904.0,
+ "4505": 1038849920.0,
+ "4510": 1043723776.0,
+ "4515": 1044277248.0,
+ "4520": 1036871296.0,
+ "4525": 1058072512.0,
+ "4530": 1030972928.0,
+ "4535": 1032592640.0,
+ "4540": 1036427008.0,
+ "4545": 1025726592.0,
+ "4550": 1021748416.0,
+ "4555": 1037546048.0,
+ "4560": 1020099904.0,
+ "4565": 1036055680.0,
+ "4570": 1020501632.0,
+ "4575": 1050411648.0,
+ "4580": 1010438528.0,
+ "4585": 1022960640.0,
+ "4590": 1039711232.0,
+ "4595": 1023274368.0,
+ "4600": 1042477952.0,
+ "4605": 1039747584.0,
+ "4610": 1046103680.0,
+ "4615": 1018000064.0,
+ "4620": 1044734784.0,
+ "4625": 1030479488.0,
+ "4630": 1027261056.0,
+ "4635": 1026995392.0,
+ "4640": 1034901184.0,
+ "4645": 1036421120.0,
+ "4650": 1033710592.0,
+ "4655": 1035460352.0,
+ "4660": 1035324544.0,
+ "4665": 1020265472.0,
+ "4670": 1020057216.0,
+ "4675": 1054848576.0,
+ "4680": 1024895616.0,
+ "4685": 1027821248.0,
+ "4690": 1034449216.0,
+ "4695": 1039151360.0,
+ "4700": 1038865152.0,
+ "4705": 1027655552.0,
+ "4710": 1020522240.0,
+ "4715": 1031827328.0,
+ "4720": 1030299648.0,
+ "4725": 1030298880.0,
+ "4730": 1044097024.0,
+ "4735": 1046132224.0,
+ "4740": 1036177664.0,
+ "4745": 1039044224.0,
+ "4750": 1031790720.0,
+ "4755": 1047724224.0,
+ "4760": 1026178432.0,
+ "4765": 1034694976.0,
+ "4770": 1036521408.0,
+ "4775": 1029374976.0,
+ "4780": 1028543488.0,
+ "4785": 1028414720.0,
+ "4790": 1019620480.0,
+ "4795": 1033059456.0,
+ "4800": 1051866560.0,
+ "4805": 1015413376.0,
+ "4810": 1029453440.0,
+ "4815": 1009571328.0,
+ "4820": 1041051520.0,
+ "4825": 1026708864.0,
+ "4830": 1020451968.0,
+ "4835": 1051307904.0,
+ "4840": 1019456704.0,
+ "4845": 1032314880.0,
+ "4850": 1036794048.0,
+ "4855": 1031053248.0,
+ "4860": 1033130880.0,
+ "4865": 1032063104.0,
+ "4870": 1049832960.0,
+ "4875": 1025110592.0,
+ "4880": 1048477568.0,
+ "4885": 1016854208.0,
+ "4890": 1037317632.0,
+ "4895": 1024323264.0,
+ "4900": 1043373568.0,
+ "4905": 1033396864.0,
+ "4910": 1032829888.0,
+ "4915": 1016889728.0,
+ "4920": 1022294400.0,
+ "4925": 1034965888.0,
+ "4930": 1034629888.0,
+ "4935": 1025884992.0,
+ "4940": 1048398976.0,
+ "4945": 1025249344.0,
+ "4950": 1024208832.0,
+ "4955": 1007487104.0,
+ "4960": 1040213312.0,
+ "4965": 1018775552.0,
+ "4970": 1014274560.0,
+ "4975": 1038025536.0,
+ "4980": 1020917376.0,
+ "4985": 1029045888.0,
+ "4990": 1028395840.0,
+ "4995": 1032018816.0,
+ "5000": 1039790976.0,
+ "5005": 1024351616.0,
+ "5010": 1029147264.0,
+ "5015": 1021808896.0,
+ "5020": 1023506688.0,
+ "5025": 1037603968.0,
+ "5030": 1041946880.0,
+ "5035": 1047129600.0,
+ "5040": 1060956736.0,
+ "5045": 1032108032.0,
+ "5050": 1029534144.0,
+ "5055": 1024552704.0,
+ "5060": 1035282880.0,
+ "5065": 1021205760.0,
+ "5070": 1035754496.0,
+ "5075": 1015772864.0,
+ "5080": 1027040576.0,
+ "5085": 1021792640.0,
+ "5090": 1034974016.0,
+ "5095": 1015499520.0,
+ "5100": 1032257728.0,
+ "5105": 1017981504.0,
+ "5110": 1019585728.0,
+ "5115": 1036064640.0,
+ "5120": 1032694272.0,
+ "5125": 1019075712.0,
+ "5130": 1033404672.0,
+ "5135": 1041203072.0,
+ "5140": 1026258816.0,
+ "5145": 1033705856.0,
+ "5150": 1022044928.0,
+ "5155": 1032265856.0,
+ "5160": 1039626240.0,
+ "5165": 1031575680.0,
+ "5170": 1035556800.0,
+ "5175": 1026115200.0,
+ "5180": 1030315776.0,
+ "5185": 1024494720.0,
+ "5190": 1019492672.0,
+ "5195": 1035626880.0,
+ "5200": 1016905472.0,
+ "5205": 1013435392.0,
+ "5210": 1049396608.0,
+ "5215": 1030832512.0,
+ "5220": 1025277440.0,
+ "5225": 1035240704.0,
+ "5230": 1025929728.0,
+ "5235": 1025119488.0,
+ "5240": 1046308096.0,
+ "5245": 1022741120.0,
+ "5250": 1027062400.0,
+ "5255": 1023887680.0,
+ "5260": 1033820160.0,
+ "5265": 1045733760.0,
+ "5270": 1052499712.0,
+ "5275": 1033019328.0,
+ "5280": 1030072064.0,
+ "5285": 1025212736.0,
+ "5290": 1026575168.0,
+ "5295": 1032652544.0,
+ "5300": 1024368192.0,
+ "5305": 1029634944.0,
+ "5310": 1033196992.0,
+ "5315": 1032990592.0,
+ "5320": 1019521280.0,
+ "5325": 1022718016.0,
+ "5330": 1021334848.0,
+ "5335": 1039275776.0,
+ "5340": 1037219712.0,
+ "5345": 1039187776.0,
+ "5350": 1023700864.0,
+ "5355": 1029935808.0,
+ "5360": 1047045760.0,
+ "5365": 1037425920.0,
+ "5370": 1024382592.0,
+ "5375": 1042069504.0,
+ "5380": 1020368768.0,
+ "5385": 1021766912.0,
+ "5390": 1035132736.0,
+ "5395": 1049655616.0,
+ "5400": 1026016064.0,
+ "5405": 1036452480.0,
+ "5410": 1027635648.0,
+ "5415": 1042285824.0,
+ "5420": 1039943040.0,
+ "5425": 1028380288.0,
+ "5430": 1043799808.0,
+ "5435": 1032653568.0,
+ "5440": 1033384960.0,
+ "5445": 1034143488.0,
+ "5450": 1025300864.0,
+ "5455": 1034080128.0,
+ "5460": 1026811840.0,
+ "5465": 1027399424.0,
+ "5470": 1028969536.0,
+ "5475": 1037233024.0,
+ "5480": 1023831040.0,
+ "5485": 1019186496.0,
+ "5490": 1030891392.0,
+ "5495": 1029400192.0,
+ "5500": 1032682240.0,
+ "5505": 1018274816.0,
+ "5510": 1023987840.0,
+ "5515": 1025155904.0,
+ "5520": 1039527488.0,
+ "5525": 1018024576.0,
+ "5530": 1037665152.0,
+ "5535": 1031599872.0,
+ "5540": 1027564864.0,
+ "5545": 1033211904.0,
+ "5550": 1032115904.0,
+ "5555": 1044802432.0,
+ "5560": 1028510720.0,
+ "5565": 1029686144.0,
+ "5570": 1042028480.0,
+ "5575": 1025378496.0,
+ "5580": 1023716416.0,
+ "5585": 1044094336.0,
+ "5590": 1041320000.0,
+ "5595": 1031549824.0,
+ "5600": 1023400448.0,
+ "5605": 1040116224.0,
+ "5610": 1034088000.0,
+ "5615": 1021042560.0,
+ "5620": 1031005184.0,
+ "5625": 1030188032.0,
+ "5630": 1023501952.0,
+ "5635": 1026683072.0,
+ "5640": 1034589056.0,
+ "5645": 1018654912.0,
+ "5650": 1052379008.0,
+ "5655": 1048933760.0,
+ "5660": 1050076800.0,
+ "5665": 1033957696.0,
+ "5670": 1033750208.0,
+ "5675": 1025393920.0,
+ "5680": 1039377856.0,
+ "5685": 1033056256.0,
+ "5690": 1031464320.0,
+ "5695": 1021945600.0,
+ "5700": 1038065664.0,
+ "5705": 1043684480.0,
+ "5710": 1057231744.0,
+ "5715": 1014462656.0,
+ "5720": 1021258752.0,
+ "5725": 1041821760.0,
+ "5730": 1039455104.0,
+ "5735": 1025129728.0,
+ "5740": 1026045568.0,
+ "5745": 1036990336.0,
+ "5750": 1044551680.0,
+ "5755": 1011860672.0,
+ "5760": 1028391296.0,
+ "5765": 1028244992.0,
+ "5770": 1021530752.0,
+ "5775": 1051210816.0,
+ "5780": 1034984128.0,
+ "5785": 1037513536.0,
+ "5790": 1016956928.0,
+ "5795": 1027873536.0,
+ "5800": 1029780864.0,
+ "5805": 1050694784.0,
+ "5810": 1018477824.0,
+ "5815": 1036124096.0,
+ "5820": 1048408832.0,
+ "5825": 1030977152.0,
+ "5830": 1031573056.0,
+ "5835": 1034045184.0,
+ "5840": 1039843776.0,
+ "5845": 1021746304.0,
+ "5850": 1029808512.0,
+ "5855": 1038790912.0,
+ "5860": 1031436352.0,
+ "5865": 1026397696.0,
+ "5870": 1029860480.0,
+ "5875": 1032841280.0,
+ "5880": 1032675392.0,
+ "5885": 1024576768.0,
+ "5890": 1026799424.0,
+ "5895": 1015796096.0,
+ "5900": 1049707648.0,
+ "5905": 1025653248.0,
+ "5910": 1019150720.0,
+ "5915": 1042739584.0,
+ "5920": 1028046528.0,
+ "5925": 1034016704.0,
+ "5930": 1030962304.0,
+ "5935": 1038102208.0,
+ "5940": 1019172608.0,
+ "5945": 1025130112.0,
+ "5950": 1035529344.0,
+ "5955": 1050437632.0,
+ "5960": 1024548736.0,
+ "5965": 1029924160.0,
+ "5970": 1016428160.0,
+ "5975": 1036682880.0,
+ "5980": 1024118272.0,
+ "5985": 1035386816.0,
+ "5990": 1010550528.0,
+ "5995": 1047019008.0,
+ "6000": 1021244608.0,
+ "6005": 1040460992.0,
+ "6010": 1025359104.0,
+ "6015": 1050180032.0,
+ "6020": 1039514304.0,
+ "6025": 1030254592.0,
+ "6030": 1025932096.0,
+ "6035": 1021745984.0,
+ "6040": 1034116864.0,
+ "6045": 1028282496.0,
+ "6050": 1020111872.0,
+ "6055": 1040395648.0,
+ "6060": 1026347456.0,
+ "6065": 1022198080.0,
+ "6070": 1040667648.0,
+ "6075": 1046037888.0,
+ "6080": 1038582720.0,
+ "6085": 1041484800.0,
+ "6090": 1037205696.0,
+ "6095": 1036283136.0,
+ "6100": 1030454592.0,
+ "6105": 1019216448.0,
+ "6110": 1035356928.0,
+ "6115": 1019452864.0,
+ "6120": 1032189504.0,
+ "6125": 1020922112.0,
+ "6130": 1012013504.0,
+ "6135": 1038733888.0,
+ "6140": 1041737536.0,
+ "6145": 1041917056.0,
+ "6150": 1018958528.0,
+ "6155": 1024649728.0,
+ "6160": 1047971840.0,
+ "6165": 1050409600.0,
+ "6170": 1032504512.0,
+ "6175": 1045793664.0,
+ "6180": 1040067328.0,
+ "6185": 1029710080.0,
+ "6190": 1023294592.0,
+ "6195": 1050897664.0,
+ "6200": 1035035712.0,
+ "6205": 1036275200.0,
+ "6210": 1039772800.0,
+ "6215": 1033199680.0,
+ "6220": 1026162048.0,
+ "6225": 1036741760.0,
+ "6230": 1025144704.0,
+ "6235": 1019353152.0,
+ "6240": 1057104896.0,
+ "6245": 1018414016.0,
+ "6250": 1035336576.0,
+ "6255": 1025380672.0,
+ "6260": 1034863552.0,
+ "6265": 1027703040.0,
+ "6270": 1042117248.0,
+ "6275": 1037659392.0,
+ "6280": 1018270976.0,
+ "6285": 1032641664.0,
+ "6290": 1038598016.0,
+ "6295": 1031804352.0,
+ "6300": 1034634432.0,
+ "6305": 1011067904.0,
+ "6310": 1039458304.0,
+ "6315": 1030053888.0,
+ "6320": 1030534016.0,
+ "6325": 1038642176.0,
+ "6330": 1033909376.0,
+ "6335": 1032297472.0,
+ "6340": 1033545408.0,
+ "6345": 1031035776.0,
+ "6350": 1037451392.0,
+ "6355": 1028075392.0,
+ "6360": 1043313536.0,
+ "6365": 1025223680.0,
+ "6370": 1033939776.0,
+ "6375": 1036038144.0,
+ "6380": 1029107904.0,
+ "6385": 1025395200.0,
+ "6390": 1025518336.0,
+ "6395": 1048612288.0,
+ "6400": 1040734656.0,
+ "6405": 1024247552.0,
+ "6410": 1017490048.0,
+ "6415": 1042827264.0,
+ "6420": 1025201344.0,
+ "6425": 1027165120.0,
+ "6430": 1040569472.0,
+ "6435": 1022908544.0,
+ "6440": 1047994496.0,
+ "6445": 1036089984.0,
+ "6450": 1048531712.0,
+ "6455": 1037272576.0,
+ "6460": 1036750080.0,
+ "6465": 1033652928.0,
+ "6470": 1018134912.0,
+ "6475": 1034691712.0,
+ "6480": 1028993408.0,
+ "6485": 1033258496.0,
+ "6490": 1035638208.0,
+ "6495": 1024469632.0,
+ "6500": 1020572352.0,
+ "6505": 1059327104.0,
+ "6510": 1020471936.0,
+ "6515": 1018687744.0,
+ "6520": 1051470464.0,
+ "6525": 1035544960.0,
+ "6530": 1027896960.0,
+ "6535": 1022722176.0,
+ "6540": 1023273344.0,
+ "6545": 1033173120.0,
+ "6550": 1029488384.0,
+ "6555": 1029574720.0,
+ "6560": 1056438656.0,
+ "6565": 1054295040.0,
+ "6570": 1032318720.0,
+ "6575": 1041207552.0,
+ "6580": 1028133760.0,
+ "6585": 1036505280.0,
+ "6590": 1042455936.0,
+ "6595": 1038569088.0,
+ "6600": 1031387712.0,
+ "6605": 1045715520.0,
+ "6610": 1034713216.0,
+ "6615": 1015576640.0,
+ "6620": 1039114944.0,
+ "6625": 1054654848.0,
+ "6630": 1043092224.0,
+ "6635": 1032226432.0,
+ "6640": 1016738944.0,
+ "6645": 1016178816.0,
+ "6650": 1034693056.0,
+ "6655": 1031753216.0,
+ "6660": 1041401472.0,
+ "6665": 1024658048.0,
+ "6670": 1023819008.0,
+ "6675": 1038305344.0,
+ "6680": 1025625088.0,
+ "6685": 1045393152.0,
+ "6690": 1046389760.0,
+ "6695": 1027753984.0,
+ "6700": 1033473920.0,
+ "6705": 1038856128.0,
+ "6710": 1047485696.0,
+ "6715": 1043229952.0,
+ "6720": 1022995456.0,
+ "6725": 1018910080.0,
+ "6730": 1027525504.0,
+ "6735": 1016938880.0,
+ "6740": 1027238400.0,
+ "6745": 1030263552.0,
+ "6750": 1006373504.0,
+ "6755": 1034764480.0,
+ "6760": 1040735040.0,
+ "6765": 1023827264.0,
+ "6770": 1036440960.0,
+ "6775": 1019627392.0,
+ "6780": 1043723904.0,
+ "6785": 1037410176.0,
+ "6790": 1029402752.0,
+ "6795": 1026349888.0,
+ "6800": 1036628480.0,
+ "6805": 1024580480.0,
+ "6810": 1042340224.0,
+ "6815": 1035273344.0,
+ "6820": 1022594560.0,
+ "6825": 1034794496.0,
+ "6830": 1029862336.0,
+ "6835": 1041608960.0,
+ "6840": 1042283584.0,
+ "6845": 1018953984.0,
+ "6850": 1032171776.0,
+ "6855": 1034434688.0,
+ "6860": 1042054400.0,
+ "6865": 1021813120.0,
+ "6870": 1037016768.0,
+ "6875": 1030379392.0,
+ "6880": 1029361152.0,
+ "6885": 1030435648.0,
+ "6890": 1039890496.0,
+ "6895": 1027267392.0,
+ "6900": 1035173632.0,
+ "6905": 1043974848.0,
+ "6910": 1019762816.0,
+ "6915": 1017476736.0,
+ "6920": 1017184256.0,
+ "6925": 1030651712.0,
+ "6930": 1036672256.0,
+ "6935": 1042835776.0,
+ "6940": 1040312576.0,
+ "6945": 1044198144.0,
+ "6950": 1040513600.0,
+ "6955": 1036112256.0,
+ "6960": 1036435520.0,
+ "6965": 1019161536.0,
+ "6970": 1034728960.0,
+ "6975": 1019135040.0,
+ "6980": 1028436160.0,
+ "6985": 1023240064.0,
+ "6990": 1026995200.0,
+ "6995": 1027546816.0,
+ "7000": 1058819776.0,
+ "7005": 1013737728.0,
+ "7010": 1028959168.0,
+ "7015": 1037288576.0,
+ "7020": 1011880640.0,
+ "7025": 1017312960.0,
+ "7030": 1028300800.0,
+ "7035": 1035956288.0,
+ "7040": 1042965760.0,
+ "7045": 1028185792.0,
+ "7050": 1017979392.0,
+ "7055": 1035089088.0,
+ "7060": 1051802368.0,
+ "7065": 1007665024.0,
+ "7070": 1035818752.0,
+ "7075": 1031038208.0,
+ "7080": 1026143360.0,
+ "7085": 1044906624.0,
+ "7090": 1046261632.0,
+ "7095": 1043760768.0,
+ "7100": 1035088256.0,
+ "7105": 1049144192.0,
+ "7110": 1010962688.0,
+ "7115": 1033869824.0,
+ "7120": 1031267456.0,
+ "7125": 1037497344.0,
+ "7130": 1024881600.0,
+ "7135": 1031992384.0,
+ "7140": 1019090560.0,
+ "7145": 1033081472.0,
+ "7150": 1037554368.0,
+ "7155": 1015729216.0,
+ "7160": 1024724736.0,
+ "7165": 1030895616.0,
+ "7170": 1037367808.0,
+ "7175": 1028816512.0,
+ "7180": 1037632768.0,
+ "7185": 1016172800.0,
+ "7190": 1019807680.0,
+ "7195": 1040915584.0,
+ "7200": 1041375360.0,
+ "7205": 1026538368.0,
+ "7210": 1022638848.0,
+ "7215": 1041890432.0,
+ "7220": 1017742592.0,
+ "7225": 1027296896.0,
+ "7230": 1030199680.0,
+ "7235": 1035726720.0,
+ "7240": 1037855424.0,
+ "7245": 1023971200.0,
+ "7250": 1044707328.0,
+ "7255": 1031900160.0,
+ "7260": 1030128384.0,
+ "7265": 1036887424.0,
+ "7270": 1050097280.0,
+ "7275": 1029225728.0,
+ "7280": 1020231040.0,
+ "7285": 1029841600.0,
+ "7290": 1017219648.0,
+ "7295": 1029139712.0,
+ "7300": 1031533696.0,
+ "7305": 1027297664.0,
+ "7310": 1029089664.0,
+ "7315": 1022782016.0,
+ "7320": 1036458368.0,
+ "7325": 1036851520.0,
+ "7330": 1021707264.0,
+ "7335": 1030715008.0,
+ "7340": 1039382912.0,
+ "7345": 1040177920.0,
+ "7350": 1034973696.0,
+ "7355": 1033655168.0,
+ "7360": 1031254784.0,
+ "7365": 1048742208.0,
+ "7370": 1027297920.0,
+ "7375": 1041855488.0,
+ "7380": 1016725504.0,
+ "7385": 1017578816.0,
+ "7390": 1017234432.0,
+ "7395": 1046793536.0,
+ "7400": 1048442368.0,
+ "7405": 1013393920.0,
+ "7410": 1017386112.0,
+ "7415": 1017816000.0,
+ "7420": 1028043456.0,
+ "7425": 1012840960.0,
+ "7430": 1034041728.0,
+ "7435": 1032530560.0,
+ "7440": 1002691904.0,
+ "7445": 1034450944.0,
+ "7450": 1039304320.0,
+ "7455": 1019026240.0,
+ "7460": 1014740736.0,
+ "7465": 1027204672.0,
+ "7470": 1030423296.0,
+ "7475": 1033791232.0,
+ "7480": 1043316864.0,
+ "7485": 1038215232.0,
+ "7490": 1049001088.0,
+ "7495": 1028982528.0,
+ "7500": 1027426688.0,
+ "7505": 1028696704.0,
+ "7510": 1048886144.0,
+ "7515": 1035648448.0,
+ "7520": 1017198848.0,
+ "7525": 1036573056.0,
+ "7530": 1029262272.0,
+ "7535": 1027190144.0,
+ "7540": 1028337792.0,
+ "7545": 1025986432.0,
+ "7550": 1023026304.0,
+ "7555": 1033025152.0,
+ "7560": 1031405632.0,
+ "7565": 1022710528.0,
+ "7570": 1037591296.0,
+ "7575": 1022603264.0,
+ "7580": 1018123584.0,
+ "7585": 1033054336.0,
+ "7590": 1010992448.0,
+ "7595": 1018259840.0,
+ "7600": 1049904320.0,
+ "7605": 1037360896.0,
+ "7610": 1040414976.0,
+ "7615": 1035248256.0,
+ "7620": 1024231040.0,
+ "7625": 1020316416.0,
+ "7630": 1034939264.0,
+ "7635": 1043224704.0,
+ "7640": 1033491200.0,
+ "7645": 1034444992.0,
+ "7650": 1039805056.0,
+ "7655": 1031239936.0,
+ "7660": 1056627392.0,
+ "7665": 1031076608.0,
+ "7670": 1033684992.0,
+ "7675": 1030681088.0,
+ "7680": 1035398208.0,
+ "7685": 1018662016.0,
+ "7690": 1031921152.0,
+ "7695": 1025858176.0,
+ "7700": 1017714816.0,
+ "7705": 1036530816.0,
+ "7710": 1029893632.0,
+ "7715": 1053231104.0,
+ "7720": 1019514944.0,
+ "7725": 1042193024.0,
+ "7730": 1035620736.0,
+ "7735": 1020726400.0,
+ "7740": 1045575936.0,
+ "7745": 1026932736.0,
+ "7750": 1048550400.0,
+ "7755": 1022539072.0,
+ "7760": 1049531904.0,
+ "7765": 1029370240.0,
+ "7770": 1018375040.0,
+ "7775": 1021364608.0,
+ "7780": 1039770368.0,
+ "7785": 1039914112.0,
+ "7790": 1030516800.0,
+ "7795": 1039353600.0,
+ "7800": 1028187904.0,
+ "7805": 1027635840.0,
+ "7810": 1020970112.0,
+ "7815": 1035878784.0,
+ "7820": 1017666048.0,
+ "7825": 1018067392.0,
+ "7830": 1035104256.0,
+ "7835": 1044507456.0,
+ "7840": 1027835776.0,
+ "7845": 1032101504.0,
+ "7850": 1034609280.0,
+ "7855": 1025464448.0,
+ "7860": 1059051264.0,
+ "7865": 1016626752.0,
+ "7870": 1033729408.0,
+ "7875": 1044186368.0,
+ "7880": 1029083392.0,
+ "7885": 1040308480.0,
+ "7890": 1029555200.0,
+ "7895": 1032946560.0,
+ "7900": 1021407744.0,
+ "7905": 1020955648.0,
+ "7910": 1008994560.0,
+ "7915": 1023120384.0,
+ "7920": 1023070848.0,
+ "7925": 1030094208.0,
+ "7930": 1020712192.0,
+ "7935": 1019443456.0,
+ "7940": 1017809472.0,
+ "7945": 1014447616.0,
+ "7950": 1026303232.0,
+ "7955": 1034518144.0,
+ "7960": 1056025984.0,
+ "7965": 1031048064.0,
+ "7970": 1030417664.0,
+ "7975": 1022190080.0,
+ "7980": 1034475072.0,
+ "7985": 1047303744.0,
+ "7990": 1032065920.0,
+ "7995": 1044264832.0,
+ "8000": 1028876288.0,
+ "8005": 1028045568.0,
+ "8010": 1050665344.0,
+ "8015": 1019759296.0,
+ "8020": 1043297408.0,
+ "8025": 1039018880.0,
+ "8030": 1030868864.0,
+ "8035": 1045304192.0,
+ "8040": 1026310976.0,
+ "8045": 1024969600.0,
+ "8050": 1018405248.0,
+ "8055": 1033736640.0,
+ "8060": 1012986240.0,
+ "8065": 1022017024.0,
+ "8070": 1034776000.0,
+ "8075": 1042760192.0,
+ "8080": 1027758656.0,
+ "8085": 1037204864.0,
+ "8090": 1007008000.0,
+ "8095": 1030374144.0,
+ "8100": 1030725568.0,
+ "8105": 1027794432.0,
+ "8110": 1031557120.0,
+ "8115": 1037685184.0,
+ "8120": 1037693056.0,
+ "8125": 1031097344.0,
+ "8130": 1028626560.0,
+ "8135": 1029680128.0,
+ "8140": 1049904640.0,
+ "8145": 1043463040.0,
+ "8150": 1040086592.0,
+ "8155": 1046780416.0,
+ "8160": 1010198912.0,
+ "8165": 1031657856.0,
+ "8170": 1024483840.0,
+ "8175": 1035019904.0,
+ "8180": 1024461504.0,
+ "8185": 1021960576.0,
+ "8190": 1037125888.0,
+ "8195": 1022368448.0,
+ "8200": 1035635840.0,
+ "8205": 1026482816.0,
+ "8210": 1023888064.0,
+ "8215": 1014277056.0,
+ "8220": 1026755712.0,
+ "8225": 1028540416.0,
+ "8230": 1027163456.0,
+ "8235": 1037913856.0,
+ "8240": 1025909632.0,
+ "8245": 1024676864.0,
+ "8250": 1041635456.0,
+ "8255": 1031909056.0,
+ "8260": 1032424320.0,
+ "8265": 1023164160.0,
+ "8270": 1040172928.0,
+ "8275": 1038051712.0,
+ "8280": 1041849728.0,
+ "8285": 1038804096.0,
+ "8290": 1024074624.0,
+ "8295": 1028404352.0,
+ "8300": 1039341312.0,
+ "8305": 1012105088.0,
+ "8310": 1021881792.0,
+ "8315": 1027307264.0,
+ "8320": 1021636480.0,
+ "8325": 1048572160.0,
+ "8330": 1041040768.0,
+ "8335": 1037964928.0,
+ "8340": 1033019456.0,
+ "8345": 1043863936.0,
+ "8350": 1037713792.0,
+ "8355": 1029686528.0,
+ "8360": 1040667456.0,
+ "8365": 1027449728.0,
+ "8370": 1037743104.0,
+ "8375": 1041986688.0,
+ "8380": 1037628480.0,
+ "8385": 1023436544.0,
+ "8390": 1026068032.0,
+ "8395": 1028913920.0,
+ "8400": 1046530240.0,
+ "8405": 1040179776.0,
+ "8410": 1034252672.0,
+ "8415": 1040258944.0,
+ "8420": 1054730496.0,
+ "8425": 1031515200.0,
+ "8430": 1030295424.0,
+ "8435": 1045706752.0,
+ "8440": 1026310912.0,
+ "8445": 1029027840.0,
+ "8450": 1034202880.0,
+ "8455": 1031794048.0,
+ "8460": 1016828352.0,
+ "8465": 1035162688.0,
+ "8470": 1035184768.0,
+ "8475": 1024713280.0,
+ "8480": 1035900800.0,
+ "8485": 1028948544.0,
+ "8490": 1023079104.0,
+ "8495": 1037392768.0,
+ "8500": 1025960448.0,
+ "8505": 1042724992.0,
+ "8510": 1028167680.0,
+ "8515": 1038101120.0,
+ "8520": 1023107072.0,
+ "8525": 1037987456.0,
+ "8530": 1027572416.0,
+ "8535": 1041656320.0,
+ "8540": 1033881600.0,
+ "8545": 1015116544.0,
+ "8550": 1040187456.0,
+ "8555": 1016340736.0,
+ "8560": 1019330304.0,
+ "8565": 1021410048.0,
+ "8570": 1032031616.0,
+ "8575": 1031880064.0,
+ "8580": 1016011520.0,
+ "8585": 1030017472.0,
+ "8590": 1031636800.0,
+ "8595": 1017776128.0,
+ "8600": 1002393152.0,
+ "8605": 1030238976.0,
+ "8610": 1017532544.0,
+ "8615": 1023988736.0,
+ "8620": 1047205440.0,
+ "8625": 1034232448.0,
+ "8630": 1030921088.0,
+ "8635": 1051992960.0,
+ "8640": 1041134272.0,
+ "8645": 1024870720.0,
+ "8650": 1025595520.0,
+ "8655": 1036905344.0,
+ "8660": 1031171072.0,
+ "8665": 1032905472.0,
+ "8670": 1037400576.0,
+ "8675": 1029157632.0,
+ "8680": 1031264640.0,
+ "8685": 1041198016.0,
+ "8690": 1035036032.0,
+ "8695": 1008507648.0,
+ "8700": 1027458880.0,
+ "8705": 1051505024.0,
+ "8710": 1041678016.0,
+ "8715": 1034152448.0,
+ "8720": 1017596928.0,
+ "8725": 1025187200.0,
+ "8730": 1036610432.0,
+ "8735": 1014828800.0,
+ "8740": 1036081792.0,
+ "8745": 1021252224.0,
+ "8750": 1027867200.0,
+ "8755": 1020742400.0,
+ "8760": 1036899456.0,
+ "8765": 1058672448.0,
+ "8770": 1020463104.0,
+ "8775": 1031773376.0,
+ "8780": 1030892160.0,
+ "8785": 1032118016.0,
+ "8790": 1041034048.0,
+ "8795": 1019524096.0,
+ "8800": 1038245504.0,
+ "8805": 1035106304.0,
+ "8810": 1043257600.0,
+ "8815": 1026491136.0,
+ "8820": 1027666688.0,
+ "8825": 1043464448.0,
+ "8830": 1027481344.0,
+ "8835": 1038813632.0,
+ "8840": 1034491392.0,
+ "8845": 1033909120.0,
+ "8850": 1030490688.0,
+ "8855": 1042525440.0,
+ "8860": 1013002624.0,
+ "8865": 1038368832.0,
+ "8870": 1025186688.0,
+ "8875": 1012981248.0,
+ "8880": 1028376448.0,
+ "8885": 1046460288.0,
+ "8890": 1038603776.0,
+ "8895": 1037909248.0,
+ "8900": 1027294976.0,
+ "8905": 1032792448.0,
+ "8910": 1029795648.0,
+ "8915": 1030003456.0,
+ "8920": 1030340416.0,
+ "8925": 1028569088.0,
+ "8930": 1031637888.0,
+ "8935": 1022950528.0,
+ "8940": 1019847872.0,
+ "8945": 1031909376.0,
+ "8950": 1039951360.0,
+ "8955": 1041902400.0,
+ "8960": 1026878464.0,
+ "8965": 1022084544.0,
+ "8970": 1029559168.0,
+ "8975": 1038934528.0,
+ "8980": 1033859904.0,
+ "8985": 1030649216.0,
+ "8990": 1025014720.0,
+ "8995": 1013963904.0,
+ "9000": 1035286144.0,
+ "9005": 1028649600.0,
+ "9010": 1011913472.0,
+ "9015": 1038911744.0,
+ "9020": 1030154304.0,
+ "9025": 1024685120.0,
+ "9030": 1025862016.0,
+ "9035": 1054308608.0,
+ "9040": 1027294208.0,
+ "9045": 1036583360.0,
+ "9050": 1020929792.0,
+ "9055": 1043213056.0,
+ "9060": 1023159040.0,
+ "9065": 1023387456.0,
+ "9070": 1039364096.0,
+ "9075": 1026728192.0,
+ "9080": 1018873472.0,
+ "9085": 1015439552.0,
+ "9090": 1043765248.0,
+ "9095": 1014020864.0,
+ "9100": 1031975808.0,
+ "9105": 1026514112.0,
+ "9110": 1029230208.0,
+ "9115": 1024866048.0,
+ "9120": 999986432.0,
+ "9125": 1032843200.0,
+ "9130": 1038534656.0,
+ "9135": 1031037056.0,
+ "9140": 1025501824.0,
+ "9145": 1030405184.0,
+ "9150": 1029416192.0,
+ "9155": 1038269056.0,
+ "9160": 1046044160.0,
+ "9165": 1017948416.0,
+ "9170": 1040955136.0,
+ "9175": 1031287424.0,
+ "9180": 1037830144.0,
+ "9185": 1040683776.0,
+ "9190": 1028986496.0,
+ "9195": 1034312192.0,
+ "9200": 1035551488.0,
+ "9205": 1029846656.0,
+ "9210": 1026536192.0,
+ "9215": 1030520832.0,
+ "9220": 1025732416.0,
+ "9225": 1048001216.0,
+ "9230": 1041601536.0,
+ "9235": 1027775232.0,
+ "9240": 1025245184.0,
+ "9245": 1036211712.0,
+ "9250": 1041192064.0,
+ "9255": 1020064256.0,
+ "9260": 1035338112.0,
+ "9265": 1023102464.0,
+ "9270": 1038332800.0,
+ "9275": 1036053184.0,
+ "9280": 1026541440.0,
+ "9285": 1014286080.0,
+ "9290": 1018866304.0,
+ "9295": 1026915072.0,
+ "9300": 1037086208.0,
+ "9305": 1045435904.0,
+ "9310": 1033243392.0,
+ "9315": 1039044096.0,
+ "9320": 1048495040.0,
+ "9325": 1023059968.0,
+ "9330": 1031724544.0,
+ "9335": 1035673472.0,
+ "9340": 1013718656.0,
+ "9345": 1022571264.0,
+ "9350": 1026585344.0,
+ "9355": 1034807168.0,
+ "9360": 1029839488.0,
+ "9365": 1019863104.0,
+ "9370": 1006903296.0,
+ "9375": 1036232768.0,
+ "9380": 1049012864.0,
+ "9385": 1015905216.0,
+ "9390": 1029208640.0,
+ "9395": 1008932480.0,
+ "9400": 1026894080.0,
+ "9405": 1027653248.0,
+ "9410": 1040913152.0,
+ "9415": 1035129472.0,
+ "9420": 1030792320.0,
+ "9425": 1027581696.0,
+ "9430": 1032727424.0,
+ "9435": 1031795712.0,
+ "9440": 1051730048.0,
+ "9445": 1019626368.0,
+ "9450": 1044504000.0,
+ "9455": 1035774464.0,
+ "9460": 1013828096.0,
+ "9465": 1023404032.0,
+ "9470": 1023576576.0,
+ "9475": 1039164480.0,
+ "9480": 1029596416.0,
+ "9485": 1032076224.0,
+ "9490": 1020995072.0,
+ "9495": 1021375872.0,
+ "9500": 1035594368.0,
+ "9505": 1034478144.0,
+ "9510": 1014286848.0,
+ "9515": 1031308992.0,
+ "9520": 1026564416.0,
+ "9525": 1035853696.0,
+ "9530": 1031624960.0,
+ "9535": 1025926400.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 9535,
+ "step_interval": 5,
+ "values": {
+ "1": 33395402752.0,
+ "5": 33395515392.0,
+ "10": 33395527680.0,
+ "15": 33395521536.0,
+ "20": 33395605504.0,
+ "25": 33395546112.0,
+ "30": 33395572736.0,
+ "35": 33395728384.0,
+ "40": 33395693568.0,
+ "45": 33395658752.0,
+ "50": 33395507200.0,
+ "55": 33395404800.0,
+ "60": 33395460096.0,
+ "65": 33395580928.0,
+ "70": 33395390464.0,
+ "75": 33395396608.0,
+ "80": 33395496960.0,
+ "85": 33395386368.0,
+ "90": 33395376128.0,
+ "95": 33395482624.0,
+ "100": 33395343360.0,
+ "105": 33395216384.0,
+ "110": 33395128320.0,
+ "115": 33394774016.0,
+ "120": 33395429376.0,
+ "125": 33394958336.0,
+ "130": 33395079168.0,
+ "135": 33395386368.0,
+ "140": 33395079168.0,
+ "145": 33395181568.0,
+ "150": 33395378176.0,
+ "155": 33395247104.0,
+ "160": 33395179520.0,
+ "165": 33395253248.0,
+ "170": 33395908608.0,
+ "175": 33395261440.0,
+ "180": 33395419136.0,
+ "185": 33395404800.0,
+ "190": 33395200000.0,
+ "195": 33395353600.0,
+ "200": 33395415040.0,
+ "205": 33395578880.0,
+ "210": 33395593216.0,
+ "215": 33395195904.0,
+ "220": 33394958336.0,
+ "225": 33395015680.0,
+ "230": 33395173376.0,
+ "235": 33395273728.0,
+ "240": 33395384320.0,
+ "245": 33395486720.0,
+ "250": 33395376128.0,
+ "255": 33395376128.0,
+ "260": 33394851840.0,
+ "265": 33395531776.0,
+ "270": 33395535872.0,
+ "275": 33395597312.0,
+ "280": 33395341312.0,
+ "285": 33396054016.0,
+ "290": 33395263488.0,
+ "295": 33396097024.0,
+ "300": 33395957760.0,
+ "305": 33396420608.0,
+ "310": 33395929088.0,
+ "315": 33395265536.0,
+ "320": 33395593216.0,
+ "325": 33395585024.0,
+ "330": 33396002816.0,
+ "335": 33396017152.0,
+ "340": 33396695040.0,
+ "345": 33396146176.0,
+ "350": 33395781632.0,
+ "355": 33396094976.0,
+ "360": 33395486720.0,
+ "365": 33395560448.0,
+ "370": 33396115456.0,
+ "375": 33395363840.0,
+ "380": 33395591168.0,
+ "385": 33395855360.0,
+ "390": 33395466240.0,
+ "395": 33395126272.0,
+ "400": 33395316736.0,
+ "405": 33395757056.0,
+ "410": 33395021824.0,
+ "415": 33395845120.0,
+ "420": 33395585024.0,
+ "425": 33395372032.0,
+ "430": 33395097600.0,
+ "435": 33395589120.0,
+ "440": 33395431424.0,
+ "445": 33395744768.0,
+ "450": 33395613696.0,
+ "455": 33395054592.0,
+ "460": 33395236864.0,
+ "465": 33394866176.0,
+ "470": 33395818496.0,
+ "475": 33394599936.0,
+ "480": 33395671040.0,
+ "485": 33395783680.0,
+ "490": 33395439616.0,
+ "495": 33395253248.0,
+ "500": 33396035584.0,
+ "505": 33395765248.0,
+ "510": 33396322304.0,
+ "515": 33395353600.0,
+ "520": 33394589696.0,
+ "525": 33395339264.0,
+ "530": 33395122176.0,
+ "535": 33395718144.0,
+ "540": 33395675136.0,
+ "545": 33395333120.0,
+ "550": 33395458048.0,
+ "555": 33395417088.0,
+ "560": 33396023296.0,
+ "565": 33395486720.0,
+ "570": 33395664896.0,
+ "575": 33395210240.0,
+ "580": 33394003968.0,
+ "585": 33395040256.0,
+ "590": 33395703808.0,
+ "595": 33395970048.0,
+ "600": 33395058688.0,
+ "605": 33395562496.0,
+ "610": 33394741248.0,
+ "615": 33394552832.0,
+ "620": 33395922944.0,
+ "625": 33394497536.0,
+ "630": 33395324928.0,
+ "635": 33394866176.0,
+ "640": 33394169856.0,
+ "645": 33396119552.0,
+ "650": 33395318784.0,
+ "655": 33395259392.0,
+ "660": 33396572160.0,
+ "665": 33395456000.0,
+ "670": 33396113408.0,
+ "675": 33395695616.0,
+ "680": 33394900992.0,
+ "685": 33394731008.0,
+ "690": 33395924992.0,
+ "695": 33395503104.0,
+ "700": 33394352128.0,
+ "705": 33394950144.0,
+ "710": 33396275200.0,
+ "715": 33395331072.0,
+ "720": 33395056640.0,
+ "725": 33395398656.0,
+ "730": 33396199424.0,
+ "735": 33395286016.0,
+ "740": 33395924992.0,
+ "745": 33396078592.0,
+ "750": 33395773440.0,
+ "755": 33396611072.0,
+ "760": 33394454528.0,
+ "765": 33396142080.0,
+ "770": 33394800640.0,
+ "775": 33395597312.0,
+ "780": 33396717568.0,
+ "785": 33395943424.0,
+ "790": 33396166656.0,
+ "795": 33395974144.0,
+ "800": 33395683328.0,
+ "805": 33395896320.0,
+ "810": 33396023296.0,
+ "815": 33395286016.0,
+ "820": 33395140608.0,
+ "825": 33395351552.0,
+ "830": 33395394560.0,
+ "835": 33395456000.0,
+ "840": 33395687424.0,
+ "845": 33396152320.0,
+ "850": 33394233344.0,
+ "855": 33395544064.0,
+ "860": 33396080640.0,
+ "865": 33395693568.0,
+ "870": 33396180992.0,
+ "875": 33395550208.0,
+ "880": 33395533824.0,
+ "885": 33395429376.0,
+ "890": 33396371456.0,
+ "895": 33396142080.0,
+ "900": 33394741248.0,
+ "905": 33395798016.0,
+ "910": 33395879936.0,
+ "915": 33396322304.0,
+ "920": 33395662848.0,
+ "925": 33395826688.0,
+ "930": 33395363840.0,
+ "935": 33395929088.0,
+ "940": 33394540544.0,
+ "945": 33394776064.0,
+ "950": 33395214336.0,
+ "955": 33395865600.0,
+ "960": 33395763200.0,
+ "965": 33395306496.0,
+ "970": 33396412416.0,
+ "975": 33394954240.0,
+ "980": 33395515392.0,
+ "985": 33395220480.0,
+ "990": 33395490816.0,
+ "995": 33394821120.0,
+ "1000": 33395884032.0,
+ "1005": 33393950720.0,
+ "1010": 33395472384.0,
+ "1015": 33394649088.0,
+ "1020": 33394819072.0,
+ "1025": 33394624512.0,
+ "1030": 33395294208.0,
+ "1035": 33396613120.0,
+ "1040": 33395867648.0,
+ "1045": 33396789248.0,
+ "1050": 33394933760.0,
+ "1055": 33395040256.0,
+ "1060": 33395468288.0,
+ "1065": 33395363840.0,
+ "1070": 33394827264.0,
+ "1075": 33396213760.0,
+ "1080": 33395099648.0,
+ "1085": 33394890752.0,
+ "1090": 33396123648.0,
+ "1095": 33395466240.0,
+ "1100": 33395687424.0,
+ "1105": 33396725760.0,
+ "1110": 33395640320.0,
+ "1115": 33395691520.0,
+ "1120": 33395849216.0,
+ "1125": 33395924992.0,
+ "1130": 33395736576.0,
+ "1135": 33396310016.0,
+ "1140": 33395077120.0,
+ "1145": 33395668992.0,
+ "1150": 33395800064.0,
+ "1155": 33395890176.0,
+ "1160": 33395798016.0,
+ "1165": 33395468288.0,
+ "1170": 33396238336.0,
+ "1175": 33395011584.0,
+ "1180": 33396387840.0,
+ "1185": 33396252672.0,
+ "1190": 33395304448.0,
+ "1195": 33396588544.0,
+ "1200": 33395458048.0,
+ "1205": 33396029440.0,
+ "1210": 33394216960.0,
+ "1215": 33396275200.0,
+ "1220": 33395810304.0,
+ "1225": 33395658752.0,
+ "1230": 33395664896.0,
+ "1235": 33395851264.0,
+ "1240": 33395359744.0,
+ "1245": 33396064256.0,
+ "1250": 33395298304.0,
+ "1255": 33394823168.0,
+ "1260": 33395224576.0,
+ "1265": 33395302400.0,
+ "1270": 33395081216.0,
+ "1275": 33395372032.0,
+ "1280": 33395542016.0,
+ "1285": 33395056640.0,
+ "1290": 33395447808.0,
+ "1295": 33395929088.0,
+ "1300": 33395269632.0,
+ "1305": 33396029440.0,
+ "1310": 33395261440.0,
+ "1315": 33395591168.0,
+ "1320": 33395693568.0,
+ "1325": 33395169280.0,
+ "1330": 33394907136.0,
+ "1335": 33395685376.0,
+ "1340": 33395656704.0,
+ "1345": 33395574784.0,
+ "1350": 33396553728.0,
+ "1355": 33395908608.0,
+ "1360": 33396219904.0,
+ "1365": 33395308544.0,
+ "1370": 33394860032.0,
+ "1375": 33396072448.0,
+ "1380": 33396183040.0,
+ "1385": 33396699136.0,
+ "1390": 33394757632.0,
+ "1395": 33395658752.0,
+ "1400": 33395902464.0,
+ "1405": 33395558400.0,
+ "1410": 33395634176.0,
+ "1415": 33394993152.0,
+ "1420": 33394470912.0,
+ "1425": 33395396608.0,
+ "1430": 33395134464.0,
+ "1435": 33396025344.0,
+ "1440": 33394659328.0,
+ "1445": 33394927616.0,
+ "1450": 33395822592.0,
+ "1455": 33395929088.0,
+ "1460": 33395083264.0,
+ "1465": 33394718720.0,
+ "1470": 33395939328.0,
+ "1475": 33395394560.0,
+ "1480": 33394522112.0,
+ "1485": 33395517440.0,
+ "1490": 33395820544.0,
+ "1495": 33395361792.0,
+ "1500": 33394806784.0,
+ "1505": 33394982912.0,
+ "1510": 33394761728.0,
+ "1515": 33395144704.0,
+ "1520": 33395265536.0,
+ "1525": 33396230144.0,
+ "1530": 33396076544.0,
+ "1535": 33395234816.0,
+ "1540": 33395283968.0,
+ "1545": 33396033536.0,
+ "1550": 33396101120.0,
+ "1555": 33395859456.0,
+ "1560": 33395994624.0,
+ "1565": 33395329024.0,
+ "1570": 33394655232.0,
+ "1575": 33395542016.0,
+ "1580": 33395924992.0,
+ "1585": 33395453952.0,
+ "1590": 33395023872.0,
+ "1595": 33395048448.0,
+ "1600": 33395449856.0,
+ "1605": 33395730432.0,
+ "1610": 33395320832.0,
+ "1615": 33394778112.0,
+ "1620": 33394540544.0,
+ "1625": 33396545536.0,
+ "1630": 33396574208.0,
+ "1635": 33395447808.0,
+ "1640": 33395140608.0,
+ "1645": 33394374656.0,
+ "1650": 33394849792.0,
+ "1655": 33395552256.0,
+ "1660": 33395212288.0,
+ "1665": 33394888704.0,
+ "1670": 33395673088.0,
+ "1675": 33395294208.0,
+ "1680": 33394941952.0,
+ "1685": 33395738624.0,
+ "1690": 33395570688.0,
+ "1695": 33396066304.0,
+ "1700": 33395605504.0,
+ "1705": 33395329024.0,
+ "1710": 33395814400.0,
+ "1715": 33395501056.0,
+ "1720": 33395044352.0,
+ "1725": 33395245056.0,
+ "1730": 33394323456.0,
+ "1735": 33395681280.0,
+ "1740": 33395081216.0,
+ "1745": 33396482048.0,
+ "1750": 33394794496.0,
+ "1755": 33395208192.0,
+ "1760": 33395957760.0,
+ "1765": 33395832832.0,
+ "1770": 33396322304.0,
+ "1775": 33396432896.0,
+ "1780": 33396434944.0,
+ "1785": 33395961856.0,
+ "1790": 33395769344.0,
+ "1795": 33395818496.0,
+ "1800": 33394749440.0,
+ "1805": 33395410944.0,
+ "1810": 33394993152.0,
+ "1815": 33395795968.0,
+ "1820": 33395476480.0,
+ "1825": 33396338688.0,
+ "1830": 33394608128.0,
+ "1835": 33396365312.0,
+ "1840": 33395699712.0,
+ "1845": 33395752960.0,
+ "1850": 33395732480.0,
+ "1855": 33395613696.0,
+ "1860": 33394896896.0,
+ "1865": 33395152896.0,
+ "1870": 33395744768.0,
+ "1875": 33393991680.0,
+ "1880": 33395050496.0,
+ "1885": 33395200000.0,
+ "1890": 33395163136.0,
+ "1895": 33395322880.0,
+ "1900": 33395159040.0,
+ "1905": 33395402752.0,
+ "1910": 33395052544.0,
+ "1915": 33395879936.0,
+ "1920": 33395777536.0,
+ "1925": 33394176000.0,
+ "1930": 33396285440.0,
+ "1935": 33395343360.0,
+ "1940": 33395179520.0,
+ "1945": 33395302400.0,
+ "1950": 33396041728.0,
+ "1955": 33395496960.0,
+ "1960": 33396062208.0,
+ "1965": 33395253248.0,
+ "1970": 33394757632.0,
+ "1975": 33395853312.0,
+ "1980": 33395343360.0,
+ "1985": 33394974720.0,
+ "1990": 33395290112.0,
+ "1995": 33394898944.0,
+ "2000": 33396389888.0,
+ "2005": 33395675136.0,
+ "2010": 33395322880.0,
+ "2015": 33395486720.0,
+ "2020": 33395589120.0,
+ "2025": 33395812352.0,
+ "2030": 33396699136.0,
+ "2035": 33396391936.0,
+ "2040": 33395372032.0,
+ "2045": 33395048448.0,
+ "2050": 33396772864.0,
+ "2055": 33395929088.0,
+ "2060": 33396539392.0,
+ "2065": 33395679232.0,
+ "2070": 33396205568.0,
+ "2075": 33395972096.0,
+ "2080": 33396045824.0,
+ "2085": 33395240960.0,
+ "2090": 33395843072.0,
+ "2095": 33395804160.0,
+ "2100": 33395105792.0,
+ "2105": 33395152896.0,
+ "2110": 33395703808.0,
+ "2115": 33394851840.0,
+ "2120": 33395175424.0,
+ "2125": 33395486720.0,
+ "2130": 33395310592.0,
+ "2135": 33395912704.0,
+ "2140": 33394911232.0,
+ "2145": 33395435520.0,
+ "2150": 33395630080.0,
+ "2155": 33395070976.0,
+ "2160": 33395945472.0,
+ "2165": 33395206144.0,
+ "2170": 33394958336.0,
+ "2175": 33394720768.0,
+ "2180": 33395378176.0,
+ "2185": 33394890752.0,
+ "2190": 33394997248.0,
+ "2195": 33395154944.0,
+ "2200": 33395073024.0,
+ "2205": 33395161088.0,
+ "2210": 33395918848.0,
+ "2215": 33396256768.0,
+ "2220": 33395484672.0,
+ "2225": 33395361792.0,
+ "2230": 33394507776.0,
+ "2235": 33395326976.0,
+ "2240": 33396080640.0,
+ "2245": 33395605504.0,
+ "2250": 33395421184.0,
+ "2255": 33394802688.0,
+ "2260": 33395535872.0,
+ "2265": 33396434944.0,
+ "2270": 33395175424.0,
+ "2275": 33395539968.0,
+ "2280": 33395752960.0,
+ "2285": 33395935232.0,
+ "2290": 33394761728.0,
+ "2295": 33395849216.0,
+ "2300": 33395243008.0,
+ "2305": 33395636224.0,
+ "2310": 33396064256.0,
+ "2315": 33395798016.0,
+ "2320": 33395781632.0,
+ "2325": 33395808256.0,
+ "2330": 33395916800.0,
+ "2335": 33395005440.0,
+ "2340": 33395470336.0,
+ "2345": 33396101120.0,
+ "2350": 33394384896.0,
+ "2355": 33395949568.0,
+ "2360": 33395417088.0,
+ "2365": 33394540544.0,
+ "2370": 33395458048.0,
+ "2375": 33396238336.0,
+ "2380": 33395310592.0,
+ "2385": 33395445760.0,
+ "2390": 33395279872.0,
+ "2395": 33394671616.0,
+ "2400": 33395791872.0,
+ "2405": 33395998720.0,
+ "2410": 33394227200.0,
+ "2415": 33395195904.0,
+ "2420": 33395890176.0,
+ "2425": 33395955712.0,
+ "2430": 33395529728.0,
+ "2435": 33395161088.0,
+ "2440": 33396076544.0,
+ "2445": 33395503104.0,
+ "2450": 33395447808.0,
+ "2455": 33396084736.0,
+ "2460": 33396029440.0,
+ "2465": 33394989056.0,
+ "2470": 33394913280.0,
+ "2475": 33395249152.0,
+ "2480": 33395904512.0,
+ "2485": 33394565120.0,
+ "2490": 33395142656.0,
+ "2495": 33395425280.0,
+ "2500": 33395214336.0,
+ "2505": 33395093504.0,
+ "2510": 33395806208.0,
+ "2515": 33396707328.0,
+ "2520": 33395468288.0,
+ "2525": 33395566592.0,
+ "2530": 33395386368.0,
+ "2535": 33395109888.0,
+ "2540": 33396832256.0,
+ "2545": 33396420608.0,
+ "2550": 33396393984.0,
+ "2555": 33395924992.0,
+ "2560": 33395429376.0,
+ "2565": 33396293632.0,
+ "2570": 33395865600.0,
+ "2575": 33395814400.0,
+ "2580": 33395970048.0,
+ "2585": 33396176896.0,
+ "2590": 33396121600.0,
+ "2595": 33395542016.0,
+ "2600": 33394659328.0,
+ "2605": 33395453952.0,
+ "2610": 33394900992.0,
+ "2615": 33396727808.0,
+ "2620": 33396473856.0,
+ "2625": 33396396032.0,
+ "2630": 33395283968.0,
+ "2635": 33396090880.0,
+ "2640": 33395810304.0,
+ "2645": 33395261440.0,
+ "2650": 33395445760.0,
+ "2655": 33394599936.0,
+ "2660": 33395601408.0,
+ "2665": 33395683328.0,
+ "2670": 33395724288.0,
+ "2675": 33395732480.0,
+ "2680": 33396563968.0,
+ "2685": 33395656704.0,
+ "2690": 33395900416.0,
+ "2695": 33395744768.0,
+ "2700": 33395175424.0,
+ "2705": 33394814976.0,
+ "2710": 33395947520.0,
+ "2715": 33395431424.0,
+ "2720": 33395412992.0,
+ "2725": 33395806208.0,
+ "2730": 33394782208.0,
+ "2735": 33396148224.0,
+ "2740": 33395623936.0,
+ "2745": 33396002816.0,
+ "2750": 33395097600.0,
+ "2755": 33396088832.0,
+ "2760": 33395191808.0,
+ "2765": 33395798016.0,
+ "2770": 33395955712.0,
+ "2775": 33394450432.0,
+ "2780": 33394989056.0,
+ "2785": 33394731008.0,
+ "2790": 33394905088.0,
+ "2795": 33395605504.0,
+ "2800": 33395365888.0,
+ "2805": 33394286592.0,
+ "2810": 33395294208.0,
+ "2815": 33395869696.0,
+ "2820": 33394571264.0,
+ "2825": 33395716096.0,
+ "2830": 33395851264.0,
+ "2835": 33395560448.0,
+ "2840": 33394870272.0,
+ "2845": 33395863552.0,
+ "2850": 33396277248.0,
+ "2855": 33395470336.0,
+ "2860": 33396207616.0,
+ "2865": 33395529728.0,
+ "2870": 33395511296.0,
+ "2875": 33395470336.0,
+ "2880": 33394880512.0,
+ "2885": 33395851264.0,
+ "2890": 33395867648.0,
+ "2895": 33395978240.0,
+ "2900": 33395634176.0,
+ "2905": 33395507200.0,
+ "2910": 33395060736.0,
+ "2915": 33395709952.0,
+ "2920": 33396555776.0,
+ "2925": 33396205568.0,
+ "2930": 33396400128.0,
+ "2935": 33396406272.0,
+ "2940": 33395720192.0,
+ "2945": 33395372032.0,
+ "2950": 33395388416.0,
+ "2955": 33395972096.0,
+ "2960": 33393958912.0,
+ "2965": 33395236864.0,
+ "2970": 33395871744.0,
+ "2975": 33395798016.0,
+ "2980": 33395634176.0,
+ "2985": 33395068928.0,
+ "2990": 33395392512.0,
+ "2995": 33395453952.0,
+ "3000": 33396019200.0,
+ "3005": 33395091456.0,
+ "3010": 33395761152.0,
+ "3015": 33395906560.0,
+ "3020": 33394081792.0,
+ "3025": 33395089408.0,
+ "3030": 33396156416.0,
+ "3035": 33395771392.0,
+ "3040": 33395773440.0,
+ "3045": 33396283392.0,
+ "3050": 33394638848.0,
+ "3055": 33395032064.0,
+ "3060": 33395724288.0,
+ "3065": 33394847744.0,
+ "3070": 33396170752.0,
+ "3075": 33395335168.0,
+ "3080": 33395419136.0,
+ "3085": 33394352128.0,
+ "3090": 33395245056.0,
+ "3095": 33395683328.0,
+ "3100": 33394864128.0,
+ "3105": 33394864128.0,
+ "3110": 33396176896.0,
+ "3115": 33395396608.0,
+ "3120": 33396310016.0,
+ "3125": 33395564544.0,
+ "3130": 33396137984.0,
+ "3135": 33395732480.0,
+ "3140": 33396058112.0,
+ "3145": 33394909184.0,
+ "3150": 33395138560.0,
+ "3155": 33394749440.0,
+ "3160": 33395744768.0,
+ "3165": 33395613696.0,
+ "3170": 33395720192.0,
+ "3175": 33395142656.0,
+ "3180": 33395417088.0,
+ "3185": 33395658752.0,
+ "3190": 33395941376.0,
+ "3195": 33396226048.0,
+ "3200": 33395623936.0,
+ "3205": 33394651136.0,
+ "3210": 33395019776.0,
+ "3215": 33396092928.0,
+ "3220": 33395544064.0,
+ "3225": 33395329024.0,
+ "3230": 33395238912.0,
+ "3235": 33395560448.0,
+ "3240": 33395482624.0,
+ "3245": 33395488768.0,
+ "3250": 33395591168.0,
+ "3255": 33395429376.0,
+ "3260": 33395167232.0,
+ "3265": 33396054016.0,
+ "3270": 33395380224.0,
+ "3275": 33395908608.0,
+ "3280": 33395410944.0,
+ "3285": 33396230144.0,
+ "3290": 33395441664.0,
+ "3295": 33395607552.0,
+ "3300": 33395265536.0,
+ "3305": 33394616320.0,
+ "3310": 33395572736.0,
+ "3315": 33395572736.0,
+ "3320": 33395195904.0,
+ "3325": 33394618368.0,
+ "3330": 33395712000.0,
+ "3335": 33395855360.0,
+ "3340": 33394923520.0,
+ "3345": 33395068928.0,
+ "3350": 33395224576.0,
+ "3355": 33395873792.0,
+ "3360": 33395474432.0,
+ "3365": 33395367936.0,
+ "3370": 33394448384.0,
+ "3375": 33395843072.0,
+ "3380": 33396170752.0,
+ "3385": 33396058112.0,
+ "3390": 33395265536.0,
+ "3395": 33395191808.0,
+ "3400": 33396103168.0,
+ "3405": 33395603456.0,
+ "3410": 33395675136.0,
+ "3415": 33395900416.0,
+ "3420": 33396273152.0,
+ "3425": 33395544064.0,
+ "3430": 33394872320.0,
+ "3435": 33395642368.0,
+ "3440": 33395087360.0,
+ "3445": 33395070976.0,
+ "3450": 33394905088.0,
+ "3455": 33395781632.0,
+ "3460": 33395142656.0,
+ "3465": 33394714624.0,
+ "3470": 33395015680.0,
+ "3475": 33395849216.0,
+ "3480": 33396101120.0,
+ "3485": 33395482624.0,
+ "3490": 33394782208.0,
+ "3495": 33395470336.0,
+ "3500": 33395015680.0,
+ "3505": 33395986432.0,
+ "3510": 33395488768.0,
+ "3515": 33394937856.0,
+ "3520": 33395443712.0,
+ "3525": 33395722240.0,
+ "3530": 33395834880.0,
+ "3535": 33394892800.0,
+ "3540": 33395830784.0,
+ "3545": 33395746816.0,
+ "3550": 33395167232.0,
+ "3555": 33395810304.0,
+ "3560": 33395918848.0,
+ "3565": 33395171328.0,
+ "3570": 33395085312.0,
+ "3575": 33395632128.0,
+ "3580": 33394905088.0,
+ "3585": 33394925568.0,
+ "3590": 33395329024.0,
+ "3595": 33395611648.0,
+ "3600": 33395724288.0,
+ "3605": 33395587072.0,
+ "3610": 33395763200.0,
+ "3615": 33395386368.0,
+ "3620": 33395523584.0,
+ "3625": 33395322880.0,
+ "3630": 33395011584.0,
+ "3635": 33395152896.0,
+ "3640": 33395703808.0,
+ "3645": 33395144704.0,
+ "3650": 33395718144.0,
+ "3655": 33395714048.0,
+ "3660": 33394851840.0,
+ "3665": 33395533824.0,
+ "3670": 33394728960.0,
+ "3675": 33395572736.0,
+ "3680": 33396137984.0,
+ "3685": 33394860032.0,
+ "3690": 33396174848.0,
+ "3695": 33394939904.0,
+ "3700": 33394860032.0,
+ "3705": 33394755584.0,
+ "3710": 33394995200.0,
+ "3715": 33395705856.0,
+ "3720": 33396154368.0,
+ "3725": 33396084736.0,
+ "3730": 33395814400.0,
+ "3735": 33395841024.0,
+ "3740": 33396320256.0,
+ "3745": 33396375552.0,
+ "3750": 33396361216.0,
+ "3755": 33395224576.0,
+ "3760": 33395238912.0,
+ "3765": 33395851264.0,
+ "3770": 33395271680.0,
+ "3775": 33395818496.0,
+ "3780": 33396480000.0,
+ "3785": 33395998720.0,
+ "3790": 33396072448.0,
+ "3795": 33395851264.0,
+ "3800": 33395273728.0,
+ "3805": 33396260864.0,
+ "3810": 33395474432.0,
+ "3815": 33395484672.0,
+ "3820": 33395621888.0,
+ "3825": 33395998720.0,
+ "3830": 33396242432.0,
+ "3835": 33395388416.0,
+ "3840": 33395068928.0,
+ "3845": 33395486720.0,
+ "3850": 33395503104.0,
+ "3855": 33395468288.0,
+ "3860": 33395990528.0,
+ "3865": 33395431424.0,
+ "3870": 33395757056.0,
+ "3875": 33394579456.0,
+ "3880": 33395382272.0,
+ "3885": 33395200000.0,
+ "3890": 33395757056.0,
+ "3895": 33394427904.0,
+ "3900": 33394860032.0,
+ "3905": 33394905088.0,
+ "3910": 33395533824.0,
+ "3915": 33394542592.0,
+ "3920": 33395480576.0,
+ "3925": 33395269632.0,
+ "3930": 33396264960.0,
+ "3935": 33395544064.0,
+ "3940": 33395843072.0,
+ "3945": 33396074496.0,
+ "3950": 33395384320.0,
+ "3955": 33395943424.0,
+ "3960": 33395154944.0,
+ "3965": 33394937856.0,
+ "3970": 33395789824.0,
+ "3975": 33395349504.0,
+ "3980": 33395998720.0,
+ "3985": 33395900416.0,
+ "3990": 33396150272.0,
+ "3995": 33395957760.0,
+ "4000": 33395738624.0,
+ "4005": 33395658752.0,
+ "4010": 33396068352.0,
+ "4015": 33394958336.0,
+ "4020": 33396760576.0,
+ "4025": 33396164608.0,
+ "4030": 33395654656.0,
+ "4035": 33396623360.0,
+ "4040": 33396187136.0,
+ "4045": 33395687424.0,
+ "4050": 33396234240.0,
+ "4055": 33395331072.0,
+ "4060": 33394808832.0,
+ "4065": 33396029440.0,
+ "4070": 33394737152.0,
+ "4075": 33395412992.0,
+ "4080": 33396398080.0,
+ "4085": 33394565120.0,
+ "4090": 33395798016.0,
+ "4095": 33395564544.0,
+ "4100": 33396475904.0,
+ "4105": 33395519488.0,
+ "4110": 33395855360.0,
+ "4115": 33396543488.0,
+ "4120": 33395343360.0,
+ "4125": 33395367936.0,
+ "4130": 33394243584.0,
+ "4135": 33396684800.0,
+ "4140": 33395742720.0,
+ "4145": 33395638272.0,
+ "4150": 33395578880.0,
+ "4155": 33395060736.0,
+ "4160": 33396121600.0,
+ "4165": 33395562496.0,
+ "4170": 33394765824.0,
+ "4175": 33395789824.0,
+ "4180": 33394554880.0,
+ "4185": 33394569216.0,
+ "4190": 33395197952.0,
+ "4195": 33395826688.0,
+ "4200": 33395968000.0,
+ "4205": 33395474432.0,
+ "4210": 33396142080.0,
+ "4215": 33393922048.0,
+ "4220": 33395208192.0,
+ "4225": 33394503680.0,
+ "4230": 33396019200.0,
+ "4235": 33395429376.0,
+ "4240": 33395793920.0,
+ "4245": 33395601408.0,
+ "4250": 33396244480.0,
+ "4255": 33395513344.0,
+ "4260": 33395236864.0,
+ "4265": 33394325504.0,
+ "4270": 33394362368.0,
+ "4275": 33394475008.0,
+ "4280": 33394905088.0,
+ "4285": 33395877888.0,
+ "4290": 33395314688.0,
+ "4295": 33395073024.0,
+ "4300": 33396133888.0,
+ "4305": 33393848320.0,
+ "4310": 33395384320.0,
+ "4315": 33396402176.0,
+ "4320": 33395855360.0,
+ "4325": 33395030016.0,
+ "4330": 33395689472.0,
+ "4335": 33394448384.0,
+ "4340": 33396033536.0,
+ "4345": 33395576832.0,
+ "4350": 33396154368.0,
+ "4355": 33395804160.0,
+ "4360": 33395527680.0,
+ "4365": 33395257344.0,
+ "4370": 33393983488.0,
+ "4375": 33395816448.0,
+ "4380": 33395296256.0,
+ "4385": 33395089408.0,
+ "4390": 33395062784.0,
+ "4395": 33394917376.0,
+ "4400": 33395089408.0,
+ "4405": 33395464192.0,
+ "4410": 33396168704.0,
+ "4415": 33395081216.0,
+ "4420": 33394526208.0,
+ "4425": 33395140608.0,
+ "4430": 33395531776.0,
+ "4435": 33394839552.0,
+ "4440": 33396187136.0,
+ "4445": 33395572736.0,
+ "4450": 33395433472.0,
+ "4455": 33394980864.0,
+ "4460": 33395613696.0,
+ "4465": 33395101696.0,
+ "4470": 33396142080.0,
+ "4475": 33395822592.0,
+ "4480": 33396629504.0,
+ "4485": 33395116032.0,
+ "4490": 33395337216.0,
+ "4495": 33395079168.0,
+ "4500": 33394888704.0,
+ "4505": 33394968576.0,
+ "4510": 33395593216.0,
+ "4515": 33395466240.0,
+ "4520": 33395757056.0,
+ "4525": 33396148224.0,
+ "4530": 33395654656.0,
+ "4535": 33395073024.0,
+ "4540": 33395175424.0,
+ "4545": 33395269632.0,
+ "4550": 33395347456.0,
+ "4555": 33394423808.0,
+ "4560": 33394081792.0,
+ "4565": 33395259392.0,
+ "4570": 33395365888.0,
+ "4575": 33395331072.0,
+ "4580": 33395245056.0,
+ "4585": 33395171328.0,
+ "4590": 33395650560.0,
+ "4595": 33395265536.0,
+ "4600": 33395632128.0,
+ "4605": 33395666944.0,
+ "4610": 33395019776.0,
+ "4615": 33395068928.0,
+ "4620": 33395070976.0,
+ "4625": 33396461568.0,
+ "4630": 33395511296.0,
+ "4635": 33394956288.0,
+ "4640": 33394956288.0,
+ "4645": 33395875840.0,
+ "4650": 33395429376.0,
+ "4655": 33396047872.0,
+ "4660": 33395050496.0,
+ "4665": 33396121600.0,
+ "4670": 33395838976.0,
+ "4675": 33394778112.0,
+ "4680": 33395718144.0,
+ "4685": 33395431424.0,
+ "4690": 33395202048.0,
+ "4695": 33395255296.0,
+ "4700": 33396043776.0,
+ "4705": 33396082688.0,
+ "4710": 33395173376.0,
+ "4715": 33394894848.0,
+ "4720": 33396078592.0,
+ "4725": 33395871744.0,
+ "4730": 33396334592.0,
+ "4735": 33395470336.0,
+ "4740": 33395707904.0,
+ "4745": 33395052544.0,
+ "4750": 33396652032.0,
+ "4755": 33395517440.0,
+ "4760": 33396049920.0,
+ "4765": 33395601408.0,
+ "4770": 33394974720.0,
+ "4775": 33395912704.0,
+ "4780": 33395044352.0,
+ "4785": 33394919424.0,
+ "4790": 33394890752.0,
+ "4795": 33395163136.0,
+ "4800": 33395933184.0,
+ "4805": 33396152320.0,
+ "4810": 33393549312.0,
+ "4815": 33395970048.0,
+ "4820": 33395869696.0,
+ "4825": 33394968576.0,
+ "4830": 33396133888.0,
+ "4835": 33394178048.0,
+ "4840": 33395951616.0,
+ "4845": 33395163136.0,
+ "4850": 33396244480.0,
+ "4855": 33394724864.0,
+ "4860": 33395306496.0,
+ "4865": 33395286016.0,
+ "4870": 33395767296.0,
+ "4875": 33395927040.0,
+ "4880": 33395587072.0,
+ "4885": 33396076544.0,
+ "4890": 33395189760.0,
+ "4895": 33395654656.0,
+ "4900": 33395277824.0,
+ "4905": 33395156992.0,
+ "4910": 33396398080.0,
+ "4915": 33396031488.0,
+ "4920": 33395066880.0,
+ "4925": 33396078592.0,
+ "4930": 33395324928.0,
+ "4935": 33395847168.0,
+ "4940": 33395914752.0,
+ "4945": 33395847168.0,
+ "4950": 33395361792.0,
+ "4955": 33395200000.0,
+ "4960": 33395673088.0,
+ "4965": 33395316736.0,
+ "4970": 33394917376.0,
+ "4975": 33394933760.0,
+ "4980": 33395103744.0,
+ "4985": 33394923520.0,
+ "4990": 33395394560.0,
+ "4995": 33395904512.0,
+ "5000": 33395652608.0,
+ "5005": 33395916800.0,
+ "5010": 33395095552.0,
+ "5015": 33395091456.0,
+ "5020": 33394810880.0,
+ "5025": 33395630080.0,
+ "5030": 33395607552.0,
+ "5035": 33395032064.0,
+ "5040": 33395159040.0,
+ "5045": 33396021248.0,
+ "5050": 33395007488.0,
+ "5055": 33395681280.0,
+ "5060": 33396629504.0,
+ "5065": 33395472384.0,
+ "5070": 33396178944.0,
+ "5075": 33395230720.0,
+ "5080": 33395949568.0,
+ "5085": 33395353600.0,
+ "5090": 33395447808.0,
+ "5095": 33395515392.0,
+ "5100": 33395081216.0,
+ "5105": 33396072448.0,
+ "5110": 33396523008.0,
+ "5115": 33395359744.0,
+ "5120": 33396129792.0,
+ "5125": 33394216960.0,
+ "5130": 33394831360.0,
+ "5135": 33395318784.0,
+ "5140": 33395851264.0,
+ "5145": 33394948096.0,
+ "5150": 33394978816.0,
+ "5155": 33395412992.0,
+ "5160": 33394853888.0,
+ "5165": 33395298304.0,
+ "5170": 33394987008.0,
+ "5175": 33396170752.0,
+ "5180": 33394384896.0,
+ "5185": 33394827264.0,
+ "5190": 33395464192.0,
+ "5195": 33395951616.0,
+ "5200": 33395935232.0,
+ "5205": 33395554304.0,
+ "5210": 33394898944.0,
+ "5215": 33395884032.0,
+ "5220": 33395372032.0,
+ "5225": 33395640320.0,
+ "5230": 33395369984.0,
+ "5235": 33395073024.0,
+ "5240": 33396224000.0,
+ "5245": 33396213760.0,
+ "5250": 33396381696.0,
+ "5255": 33395427328.0,
+ "5260": 33394536448.0,
+ "5265": 33396418560.0,
+ "5270": 33396617216.0,
+ "5275": 33395986432.0,
+ "5280": 33394300928.0,
+ "5285": 33396189184.0,
+ "5290": 33396555776.0,
+ "5295": 33395060736.0,
+ "5300": 33395918848.0,
+ "5305": 33396105216.0,
+ "5310": 33395662848.0,
+ "5315": 33396256768.0,
+ "5320": 33394784256.0,
+ "5325": 33396525056.0,
+ "5330": 33396027392.0,
+ "5335": 33396332544.0,
+ "5340": 33394808832.0,
+ "5345": 33396307968.0,
+ "5350": 33395216384.0,
+ "5355": 33396191232.0,
+ "5360": 33395175424.0,
+ "5365": 33396363264.0,
+ "5370": 33395437568.0,
+ "5375": 33395859456.0,
+ "5380": 33395257344.0,
+ "5385": 33394946048.0,
+ "5390": 33395259392.0,
+ "5395": 33395382272.0,
+ "5400": 33395480576.0,
+ "5405": 33395165184.0,
+ "5410": 33395324928.0,
+ "5415": 33395865600.0,
+ "5420": 33395982336.0,
+ "5425": 33396031488.0,
+ "5430": 33395748864.0,
+ "5435": 33395453952.0,
+ "5440": 33395343360.0,
+ "5445": 33395361792.0,
+ "5450": 33396076544.0,
+ "5455": 33395785728.0,
+ "5460": 33394663424.0,
+ "5465": 33394788352.0,
+ "5470": 33395451904.0,
+ "5475": 33395089408.0,
+ "5480": 33394778112.0,
+ "5485": 33395478528.0,
+ "5490": 33395396608.0,
+ "5495": 33395628032.0,
+ "5500": 33395638272.0,
+ "5505": 33395699712.0,
+ "5510": 33395537920.0,
+ "5515": 33395863552.0,
+ "5520": 33396207616.0,
+ "5525": 33394757632.0,
+ "5530": 33395460096.0,
+ "5535": 33395695616.0,
+ "5540": 33395232768.0,
+ "5545": 33395836928.0,
+ "5550": 33396103168.0,
+ "5555": 33395097600.0,
+ "5560": 33395621888.0,
+ "5565": 33395294208.0,
+ "5570": 33395277824.0,
+ "5575": 33395036160.0,
+ "5580": 33396408320.0,
+ "5585": 33395642368.0,
+ "5590": 33396281344.0,
+ "5595": 33394806784.0,
+ "5600": 33394468864.0,
+ "5605": 33395484672.0,
+ "5610": 33396715520.0,
+ "5615": 33395544064.0,
+ "5620": 33396111360.0,
+ "5625": 33396543488.0,
+ "5630": 33396414464.0,
+ "5635": 33395343360.0,
+ "5640": 33395468288.0,
+ "5645": 33395922944.0,
+ "5650": 33394708480.0,
+ "5655": 33396287488.0,
+ "5660": 33396529152.0,
+ "5665": 33395867648.0,
+ "5670": 33395701760.0,
+ "5675": 33396047872.0,
+ "5680": 33395933184.0,
+ "5685": 33395771392.0,
+ "5690": 33395124224.0,
+ "5695": 33396146176.0,
+ "5700": 33395922944.0,
+ "5705": 33396121600.0,
+ "5710": 33394804736.0,
+ "5715": 33395333120.0,
+ "5720": 33395564544.0,
+ "5725": 33395851264.0,
+ "5730": 33395615744.0,
+ "5735": 33394366464.0,
+ "5740": 33395341312.0,
+ "5745": 33394915328.0,
+ "5750": 33394794496.0,
+ "5755": 33394868224.0,
+ "5760": 33395724288.0,
+ "5765": 33396211712.0,
+ "5770": 33395810304.0,
+ "5775": 33395050496.0,
+ "5780": 33394499584.0,
+ "5785": 33395908608.0,
+ "5790": 33395675136.0,
+ "5795": 33395767296.0,
+ "5800": 33395120128.0,
+ "5805": 33395744768.0,
+ "5810": 33395265536.0,
+ "5815": 33395881984.0,
+ "5820": 33395515392.0,
+ "5825": 33395490816.0,
+ "5830": 33395548160.0,
+ "5835": 33395486720.0,
+ "5840": 33395662848.0,
+ "5845": 33394944000.0,
+ "5850": 33395552256.0,
+ "5855": 33395417088.0,
+ "5860": 33395083264.0,
+ "5865": 33395314688.0,
+ "5870": 33395398656.0,
+ "5875": 33395865600.0,
+ "5880": 33395888128.0,
+ "5885": 33395402752.0,
+ "5890": 33396037632.0,
+ "5895": 33395621888.0,
+ "5900": 33395611648.0,
+ "5905": 33396185088.0,
+ "5910": 33395337216.0,
+ "5915": 33395404800.0,
+ "5920": 33395740672.0,
+ "5925": 33395785728.0,
+ "5930": 33395922944.0,
+ "5935": 33395724288.0,
+ "5940": 33395578880.0,
+ "5945": 33395810304.0,
+ "5950": 33394429952.0,
+ "5955": 33395628032.0,
+ "5960": 33396637696.0,
+ "5965": 33396662272.0,
+ "5970": 33396490240.0,
+ "5975": 33394880512.0,
+ "5980": 33395167232.0,
+ "5985": 33395310592.0,
+ "5990": 33395550208.0,
+ "5995": 33395195904.0,
+ "6000": 33395525632.0,
+ "6005": 33395625984.0,
+ "6010": 33395984384.0,
+ "6015": 33394741248.0,
+ "6020": 33396230144.0,
+ "6025": 33395341312.0,
+ "6030": 33395521536.0,
+ "6035": 33395019776.0,
+ "6040": 33395716096.0,
+ "6045": 33396060160.0,
+ "6050": 33396088832.0,
+ "6055": 33394700288.0,
+ "6060": 33394323456.0,
+ "6065": 33395585024.0,
+ "6070": 33395382272.0,
+ "6075": 33394782208.0,
+ "6080": 33395148800.0,
+ "6085": 33396023296.0,
+ "6090": 33395437568.0,
+ "6095": 33395183616.0,
+ "6100": 33395503104.0,
+ "6105": 33395640320.0,
+ "6110": 33395777536.0,
+ "6115": 33396115456.0,
+ "6120": 33394851840.0,
+ "6125": 33394651136.0,
+ "6130": 33396842496.0,
+ "6135": 33395820544.0,
+ "6140": 33395722240.0,
+ "6145": 33396869120.0,
+ "6150": 33395666944.0,
+ "6155": 33395068928.0,
+ "6160": 33395460096.0,
+ "6165": 33395138560.0,
+ "6170": 33396004864.0,
+ "6175": 33395714048.0,
+ "6180": 33395894272.0,
+ "6185": 33396201472.0,
+ "6190": 33395412992.0,
+ "6195": 33396269056.0,
+ "6200": 33394548736.0,
+ "6205": 33396137984.0,
+ "6210": 33396131840.0,
+ "6215": 33396002816.0,
+ "6220": 33395705856.0,
+ "6225": 33395486720.0,
+ "6230": 33396215808.0,
+ "6235": 33395363840.0,
+ "6240": 33396871168.0,
+ "6245": 33395126272.0,
+ "6250": 33395040256.0,
+ "6255": 33396146176.0,
+ "6260": 33395218432.0,
+ "6265": 33395695616.0,
+ "6270": 33396277248.0,
+ "6275": 33394798592.0,
+ "6280": 33395314688.0,
+ "6285": 33396398080.0,
+ "6290": 33395875840.0,
+ "6295": 33394929664.0,
+ "6300": 33395712000.0,
+ "6305": 33396822016.0,
+ "6310": 33395580928.0,
+ "6315": 33395429376.0,
+ "6320": 33394395136.0,
+ "6325": 33395484672.0,
+ "6330": 33395746816.0,
+ "6335": 33395716096.0,
+ "6340": 33395353600.0,
+ "6345": 33395767296.0,
+ "6350": 33396498432.0,
+ "6355": 33394946048.0,
+ "6360": 33394741248.0,
+ "6365": 33395359744.0,
+ "6370": 33395302400.0,
+ "6375": 33396264960.0,
+ "6380": 33395961856.0,
+ "6385": 33393627136.0,
+ "6390": 33395048448.0,
+ "6395": 33395130368.0,
+ "6400": 33395316736.0,
+ "6405": 33396436992.0,
+ "6410": 33396314112.0,
+ "6415": 33395052544.0,
+ "6420": 33394374656.0,
+ "6425": 33396006912.0,
+ "6430": 33395904512.0,
+ "6435": 33395965952.0,
+ "6440": 33396695040.0,
+ "6445": 33395189760.0,
+ "6450": 33395828736.0,
+ "6455": 33396137984.0,
+ "6460": 33396066304.0,
+ "6465": 33396529152.0,
+ "6470": 33395243008.0,
+ "6475": 33395333120.0,
+ "6480": 33394827264.0,
+ "6485": 33396297728.0,
+ "6490": 33396062208.0,
+ "6495": 33396264960.0,
+ "6500": 33395453952.0,
+ "6505": 33395023872.0,
+ "6510": 33395746816.0,
+ "6515": 33395425280.0,
+ "6520": 33396080640.0,
+ "6525": 33395902464.0,
+ "6530": 33395492864.0,
+ "6535": 33395666944.0,
+ "6540": 33395265536.0,
+ "6545": 33395116032.0,
+ "6550": 33396344832.0,
+ "6555": 33395681280.0,
+ "6560": 33395761152.0,
+ "6565": 33394556928.0,
+ "6570": 33395040256.0,
+ "6575": 33395286016.0,
+ "6580": 33396498432.0,
+ "6585": 33395902464.0,
+ "6590": 33396051968.0,
+ "6595": 33395707904.0,
+ "6600": 33394544640.0,
+ "6605": 33395292160.0,
+ "6610": 33395660800.0,
+ "6615": 33395873792.0,
+ "6620": 33395435520.0,
+ "6625": 33395101696.0,
+ "6630": 33394870272.0,
+ "6635": 33394941952.0,
+ "6640": 33395591168.0,
+ "6645": 33394591744.0,
+ "6650": 33395691520.0,
+ "6655": 33393573888.0,
+ "6660": 33395505152.0,
+ "6665": 33394712576.0,
+ "6670": 33395582976.0,
+ "6675": 33395609600.0,
+ "6680": 33394933760.0,
+ "6685": 33395081216.0,
+ "6690": 33395230720.0,
+ "6695": 33396033536.0,
+ "6700": 33395703808.0,
+ "6705": 33394972672.0,
+ "6710": 33395341312.0,
+ "6715": 33394511872.0,
+ "6720": 33395949568.0,
+ "6725": 33396121600.0,
+ "6730": 33396510720.0,
+ "6735": 33396160512.0,
+ "6740": 33395030016.0,
+ "6745": 33396236288.0,
+ "6750": 33393977344.0,
+ "6755": 33394817024.0,
+ "6760": 33395220480.0,
+ "6765": 33395367936.0,
+ "6770": 33396346880.0,
+ "6775": 33396641792.0,
+ "6780": 33394884608.0,
+ "6785": 33396082688.0,
+ "6790": 33396203520.0,
+ "6795": 33395691520.0,
+ "6800": 33394817024.0,
+ "6805": 33395134464.0,
+ "6810": 33394999296.0,
+ "6815": 33395918848.0,
+ "6820": 33395060736.0,
+ "6825": 33395838976.0,
+ "6830": 33396011008.0,
+ "6835": 33395675136.0,
+ "6840": 33396371456.0,
+ "6845": 33396146176.0,
+ "6850": 33396015104.0,
+ "6855": 33395804160.0,
+ "6860": 33395785728.0,
+ "6865": 33395038208.0,
+ "6870": 33394618368.0,
+ "6875": 33395253248.0,
+ "6880": 33395752960.0,
+ "6885": 33395816448.0,
+ "6890": 33395212288.0,
+ "6895": 33395449856.0,
+ "6900": 33395978240.0,
+ "6905": 33395277824.0,
+ "6910": 33395562496.0,
+ "6915": 33395609600.0,
+ "6920": 33394567168.0,
+ "6925": 33394257920.0,
+ "6930": 33394905088.0,
+ "6935": 33394964480.0,
+ "6940": 33395521536.0,
+ "6945": 33395271680.0,
+ "6950": 33395449856.0,
+ "6955": 33394640896.0,
+ "6960": 33395337216.0,
+ "6965": 33395079168.0,
+ "6970": 33395630080.0,
+ "6975": 33395441664.0,
+ "6980": 33395705856.0,
+ "6985": 33394366464.0,
+ "6990": 33396469760.0,
+ "6995": 33394317312.0,
+ "7000": 33396135936.0,
+ "7005": 33396045824.0,
+ "7010": 33395331072.0,
+ "7015": 33395169280.0,
+ "7020": 33395855360.0,
+ "7025": 33396529152.0,
+ "7030": 33394626560.0,
+ "7035": 33396506624.0,
+ "7040": 33396248576.0,
+ "7045": 33395253248.0,
+ "7050": 33396072448.0,
+ "7055": 33394393088.0,
+ "7060": 33395503104.0,
+ "7065": 33394827264.0,
+ "7070": 33394554880.0,
+ "7075": 33396322304.0,
+ "7080": 33394405376.0,
+ "7085": 33395836928.0,
+ "7090": 33395798016.0,
+ "7095": 33394921472.0,
+ "7100": 33396432896.0,
+ "7105": 33395490816.0,
+ "7110": 33396232192.0,
+ "7115": 33395988480.0,
+ "7120": 33395834880.0,
+ "7125": 33395458048.0,
+ "7130": 33395632128.0,
+ "7135": 33395621888.0,
+ "7140": 33395132416.0,
+ "7145": 33395972096.0,
+ "7150": 33394835456.0,
+ "7155": 33395714048.0,
+ "7160": 33395992576.0,
+ "7165": 33396273152.0,
+ "7170": 33396342784.0,
+ "7175": 33394446336.0,
+ "7180": 33395210240.0,
+ "7185": 33396344832.0,
+ "7190": 33394903040.0,
+ "7195": 33395384320.0,
+ "7200": 33396099072.0,
+ "7205": 33396019200.0,
+ "7210": 33395949568.0,
+ "7215": 33394810880.0,
+ "7220": 33395261440.0,
+ "7225": 33395836928.0,
+ "7230": 33395781632.0,
+ "7235": 33394319360.0,
+ "7240": 33395474432.0,
+ "7245": 33395204096.0,
+ "7250": 33394667520.0,
+ "7255": 33396109312.0,
+ "7260": 33395761152.0,
+ "7265": 33395281920.0,
+ "7270": 33395916800.0,
+ "7275": 33395970048.0,
+ "7280": 33396422656.0,
+ "7285": 33395695616.0,
+ "7290": 33396178944.0,
+ "7295": 33395177472.0,
+ "7300": 33394466816.0,
+ "7305": 33394675712.0,
+ "7310": 33395568640.0,
+ "7315": 33395597312.0,
+ "7320": 33395562496.0,
+ "7325": 33396236288.0,
+ "7330": 33395712000.0,
+ "7335": 33395863552.0,
+ "7340": 33395347456.0,
+ "7345": 33395568640.0,
+ "7350": 33395478528.0,
+ "7355": 33396039680.0,
+ "7360": 33394339840.0,
+ "7365": 33396031488.0,
+ "7370": 33395253248.0,
+ "7375": 33395884032.0,
+ "7380": 33397174272.0,
+ "7385": 33394921472.0,
+ "7390": 33395859456.0,
+ "7395": 33394874368.0,
+ "7400": 33395695616.0,
+ "7405": 33394786304.0,
+ "7410": 33396512768.0,
+ "7415": 33395613696.0,
+ "7420": 33395445760.0,
+ "7425": 33395986432.0,
+ "7430": 33395310592.0,
+ "7435": 33395374080.0,
+ "7440": 33395587072.0,
+ "7445": 33396066304.0,
+ "7450": 33395394560.0,
+ "7455": 33395503104.0,
+ "7460": 33394909184.0,
+ "7465": 33395165184.0,
+ "7470": 33395939328.0,
+ "7475": 33396094976.0,
+ "7480": 33394589696.0,
+ "7485": 33396338688.0,
+ "7490": 33396109312.0,
+ "7495": 33395144704.0,
+ "7500": 33394837504.0,
+ "7505": 33395660800.0,
+ "7510": 33394714624.0,
+ "7515": 33395550208.0,
+ "7520": 33395390464.0,
+ "7525": 33395834880.0,
+ "7530": 33395286016.0,
+ "7535": 33394948096.0,
+ "7540": 33394571264.0,
+ "7545": 33394495488.0,
+ "7550": 33395605504.0,
+ "7555": 33395560448.0,
+ "7560": 33394651136.0,
+ "7565": 33394413568.0,
+ "7570": 33394921472.0,
+ "7575": 33394888704.0,
+ "7580": 33395181568.0,
+ "7585": 33395156992.0,
+ "7590": 33394245632.0,
+ "7595": 33395740672.0,
+ "7600": 33395417088.0,
+ "7605": 33396154368.0,
+ "7610": 33395054592.0,
+ "7615": 33395744768.0,
+ "7620": 33395318784.0,
+ "7625": 33395253248.0,
+ "7630": 33395836928.0,
+ "7635": 33396103168.0,
+ "7640": 33395238912.0,
+ "7645": 33396719616.0,
+ "7650": 33395992576.0,
+ "7655": 33396285440.0,
+ "7660": 33394870272.0,
+ "7665": 33394939904.0,
+ "7670": 33395386368.0,
+ "7675": 33395320832.0,
+ "7680": 33395376128.0,
+ "7685": 33395689472.0,
+ "7690": 33395046400.0,
+ "7695": 33395388416.0,
+ "7700": 33394606080.0,
+ "7705": 33396303872.0,
+ "7710": 33395994624.0,
+ "7715": 33395136512.0,
+ "7720": 33396066304.0,
+ "7725": 33395480576.0,
+ "7730": 33396072448.0,
+ "7735": 33395978240.0,
+ "7740": 33395294208.0,
+ "7745": 33395632128.0,
+ "7750": 33394978816.0,
+ "7755": 33395709952.0,
+ "7760": 33396500480.0,
+ "7765": 33394827264.0,
+ "7770": 33396291584.0,
+ "7775": 33395630080.0,
+ "7780": 33394708480.0,
+ "7785": 33395736576.0,
+ "7790": 33395376128.0,
+ "7795": 33395908608.0,
+ "7800": 33394956288.0,
+ "7805": 33396068352.0,
+ "7810": 33395384320.0,
+ "7815": 33395173376.0,
+ "7820": 33395992576.0,
+ "7825": 33394606080.0,
+ "7830": 33395273728.0,
+ "7835": 33396264960.0,
+ "7840": 33395523584.0,
+ "7845": 33395429376.0,
+ "7850": 33395019776.0,
+ "7855": 33395945472.0,
+ "7860": 33395875840.0,
+ "7865": 33396324352.0,
+ "7870": 33395849216.0,
+ "7875": 33396240384.0,
+ "7880": 33396412416.0,
+ "7885": 33395449856.0,
+ "7890": 33395214336.0,
+ "7895": 33395095552.0,
+ "7900": 33396226048.0,
+ "7905": 33395734528.0,
+ "7910": 33394855936.0,
+ "7915": 33395068928.0,
+ "7920": 33395015680.0,
+ "7925": 33396217856.0,
+ "7930": 33395824640.0,
+ "7935": 33395738624.0,
+ "7940": 33395994624.0,
+ "7945": 33395937280.0,
+ "7950": 33396191232.0,
+ "7955": 33395001344.0,
+ "7960": 33394946048.0,
+ "7965": 33396088832.0,
+ "7970": 33395367936.0,
+ "7975": 33396170752.0,
+ "7980": 33395435520.0,
+ "7985": 33394669568.0,
+ "7990": 33394855936.0,
+ "7995": 33395720192.0,
+ "8000": 33396033536.0,
+ "8005": 33395476480.0,
+ "8010": 33396101120.0,
+ "8015": 33395126272.0,
+ "8020": 33396041728.0,
+ "8025": 33396506624.0,
+ "8030": 33395703808.0,
+ "8035": 33394761728.0,
+ "8040": 33395771392.0,
+ "8045": 33395943424.0,
+ "8050": 33394558976.0,
+ "8055": 33394868224.0,
+ "8060": 33395679232.0,
+ "8065": 33396279296.0,
+ "8070": 33395312640.0,
+ "8075": 33395609600.0,
+ "8080": 33395861504.0,
+ "8085": 33395785728.0,
+ "8090": 33394925568.0,
+ "8095": 33395185664.0,
+ "8100": 33396553728.0,
+ "8105": 33396154368.0,
+ "8110": 33395089408.0,
+ "8115": 33395746816.0,
+ "8120": 33395691520.0,
+ "8125": 33395251200.0,
+ "8130": 33394792448.0,
+ "8135": 33395126272.0,
+ "8140": 33396240384.0,
+ "8145": 33394634752.0,
+ "8150": 33395130368.0,
+ "8155": 33394884608.0,
+ "8160": 33395437568.0,
+ "8165": 33395662848.0,
+ "8170": 33395429376.0,
+ "8175": 33396318208.0,
+ "8180": 33395236864.0,
+ "8185": 33395474432.0,
+ "8190": 33396125696.0,
+ "8195": 33395965952.0,
+ "8200": 33395744768.0,
+ "8205": 33396074496.0,
+ "8210": 33394458624.0,
+ "8215": 33394462720.0,
+ "8220": 33395970048.0,
+ "8225": 33396326400.0,
+ "8230": 33394274304.0,
+ "8235": 33396594688.0,
+ "8240": 33395671040.0,
+ "8245": 33396289536.0,
+ "8250": 33396211712.0,
+ "8255": 33396156416.0,
+ "8260": 33396473856.0,
+ "8265": 33396412416.0,
+ "8270": 33395810304.0,
+ "8275": 33394860032.0,
+ "8280": 33395724288.0,
+ "8285": 33395920896.0,
+ "8290": 33395658752.0,
+ "8295": 33396160512.0,
+ "8300": 33395494912.0,
+ "8305": 33396092928.0,
+ "8310": 33395587072.0,
+ "8315": 33396391936.0,
+ "8320": 33395750912.0,
+ "8325": 33395623936.0,
+ "8330": 33395793920.0,
+ "8335": 33394573312.0,
+ "8340": 33394636800.0,
+ "8345": 33395998720.0,
+ "8350": 33396353024.0,
+ "8355": 33395912704.0,
+ "8360": 33394700288.0,
+ "8365": 33395275776.0,
+ "8370": 33395775488.0,
+ "8375": 33395085312.0,
+ "8380": 33396084736.0,
+ "8385": 33395892224.0,
+ "8390": 33394769920.0,
+ "8395": 33395492864.0,
+ "8400": 33395736576.0,
+ "8405": 33395970048.0,
+ "8410": 33395668992.0,
+ "8415": 33395716096.0,
+ "8420": 33395914752.0,
+ "8425": 33395408896.0,
+ "8430": 33395599360.0,
+ "8435": 33395105792.0,
+ "8440": 33395935232.0,
+ "8445": 33395617792.0,
+ "8450": 33395351552.0,
+ "8455": 33394403328.0,
+ "8460": 33395200000.0,
+ "8465": 33396199424.0,
+ "8470": 33395793920.0,
+ "8475": 33395281920.0,
+ "8480": 33396459520.0,
+ "8485": 33395705856.0,
+ "8490": 33395085312.0,
+ "8495": 33395806208.0,
+ "8500": 33395412992.0,
+ "8505": 33395552256.0,
+ "8510": 33396047872.0,
+ "8515": 33396062208.0,
+ "8520": 33394290688.0,
+ "8525": 33394606080.0,
+ "8530": 33395343360.0,
+ "8535": 33395937280.0,
+ "8540": 33395546112.0,
+ "8545": 33395668992.0,
+ "8550": 33395697664.0,
+ "8555": 33396629504.0,
+ "8560": 33395818496.0,
+ "8565": 33395589120.0,
+ "8570": 33396006912.0,
+ "8575": 33394374656.0,
+ "8580": 33395752960.0,
+ "8585": 33395777536.0,
+ "8590": 33396209664.0,
+ "8595": 33396344832.0,
+ "8600": 33396459520.0,
+ "8605": 33395341312.0,
+ "8610": 33396097024.0,
+ "8615": 33395849216.0,
+ "8620": 33394026496.0,
+ "8625": 33396125696.0,
+ "8630": 33395988480.0,
+ "8635": 33394714624.0,
+ "8640": 33395316736.0,
+ "8645": 33395081216.0,
+ "8650": 33395603456.0,
+ "8655": 33395384320.0,
+ "8660": 33395691520.0,
+ "8665": 33396606976.0,
+ "8670": 33395113984.0,
+ "8675": 33395656704.0,
+ "8680": 33396199424.0,
+ "8685": 33395646464.0,
+ "8690": 33396707328.0,
+ "8695": 33395197952.0,
+ "8700": 33395406848.0,
+ "8705": 33395656704.0,
+ "8710": 33396207616.0,
+ "8715": 33396174848.0,
+ "8720": 33395412992.0,
+ "8725": 33395740672.0,
+ "8730": 33394982912.0,
+ "8735": 33395206144.0,
+ "8740": 33395798016.0,
+ "8745": 33395361792.0,
+ "8750": 33396058112.0,
+ "8755": 33395277824.0,
+ "8760": 33396346880.0,
+ "8765": 33394616320.0,
+ "8770": 33395949568.0,
+ "8775": 33394925568.0,
+ "8780": 33395445760.0,
+ "8785": 33394616320.0,
+ "8790": 33395290112.0,
+ "8795": 33396252672.0,
+ "8800": 33395539968.0,
+ "8805": 33395027968.0,
+ "8810": 33395200000.0,
+ "8815": 33395499008.0,
+ "8820": 33395773440.0,
+ "8825": 33394659328.0,
+ "8830": 33395376128.0,
+ "8835": 33394874368.0,
+ "8840": 33394585600.0,
+ "8845": 33395343360.0,
+ "8850": 33396006912.0,
+ "8855": 33396209664.0,
+ "8860": 33396281344.0,
+ "8865": 33395732480.0,
+ "8870": 33394741248.0,
+ "8875": 33394610176.0,
+ "8880": 33395507200.0,
+ "8885": 33394106368.0,
+ "8890": 33395163136.0,
+ "8895": 33395732480.0,
+ "8900": 33395367936.0,
+ "8905": 33396240384.0,
+ "8910": 33396146176.0,
+ "8915": 33395058688.0,
+ "8920": 33395443712.0,
+ "8925": 33395515392.0,
+ "8930": 33396094976.0,
+ "8935": 33395150848.0,
+ "8940": 33395843072.0,
+ "8945": 33396555776.0,
+ "8950": 33395744768.0,
+ "8955": 33395712000.0,
+ "8960": 33395898368.0,
+ "8965": 33394221056.0,
+ "8970": 33395707904.0,
+ "8975": 33395687424.0,
+ "8980": 33395148800.0,
+ "8985": 33396623360.0,
+ "8990": 33395306496.0,
+ "8995": 33396291584.0,
+ "9000": 33395961856.0,
+ "9005": 33395980288.0,
+ "9010": 33393659904.0,
+ "9015": 33395769344.0,
+ "9020": 33394864128.0,
+ "9025": 33395369984.0,
+ "9030": 33396068352.0,
+ "9035": 33395566592.0,
+ "9040": 33396004864.0,
+ "9045": 33395433472.0,
+ "9050": 33395804160.0,
+ "9055": 33394857984.0,
+ "9060": 33396101120.0,
+ "9065": 33395765248.0,
+ "9070": 33395011584.0,
+ "9075": 33395312640.0,
+ "9080": 33394810880.0,
+ "9085": 33395623936.0,
+ "9090": 33396371456.0,
+ "9095": 33394731008.0,
+ "9100": 33396471808.0,
+ "9105": 33395298304.0,
+ "9110": 33395195904.0,
+ "9115": 33395601408.0,
+ "9120": 33395576832.0,
+ "9125": 33395261440.0,
+ "9130": 33394196480.0,
+ "9135": 33395222528.0,
+ "9140": 33394616320.0,
+ "9145": 33395867648.0,
+ "9150": 33395738624.0,
+ "9155": 33395601408.0,
+ "9160": 33396221952.0,
+ "9165": 33395818496.0,
+ "9170": 33395929088.0,
+ "9175": 33395384320.0,
+ "9180": 33395900416.0,
+ "9185": 33395503104.0,
+ "9190": 33394233344.0,
+ "9195": 33394804736.0,
+ "9200": 33395763200.0,
+ "9205": 33395453952.0,
+ "9210": 33395838976.0,
+ "9215": 33395290112.0,
+ "9220": 33396350976.0,
+ "9225": 33395290112.0,
+ "9230": 33395550208.0,
+ "9235": 33395521536.0,
+ "9240": 33395595264.0,
+ "9245": 33396215808.0,
+ "9250": 33395871744.0,
+ "9255": 33395167232.0,
+ "9260": 33396826112.0,
+ "9265": 33394561024.0,
+ "9270": 33395693568.0,
+ "9275": 33395884032.0,
+ "9280": 33395415040.0,
+ "9285": 33395634176.0,
+ "9290": 33395888128.0,
+ "9295": 33395542016.0,
+ "9300": 33395224576.0,
+ "9305": 33395908608.0,
+ "9310": 33395011584.0,
+ "9315": 33393938432.0,
+ "9320": 33394929664.0,
+ "9325": 33395914752.0,
+ "9330": 33395929088.0,
+ "9335": 33395363840.0,
+ "9340": 33395255296.0,
+ "9345": 33395320832.0,
+ "9350": 33396076544.0,
+ "9355": 33395187712.0,
+ "9360": 33394657280.0,
+ "9365": 33395437568.0,
+ "9370": 33394124800.0,
+ "9375": 33395486720.0,
+ "9380": 33395619840.0,
+ "9385": 33396049920.0,
+ "9390": 33395843072.0,
+ "9395": 33395759104.0,
+ "9400": 33395015680.0,
+ "9405": 33395625984.0,
+ "9410": 33394966528.0,
+ "9415": 33395406848.0,
+ "9420": 33394974720.0,
+ "9425": 33395003392.0,
+ "9430": 33395335168.0,
+ "9435": 33394814976.0,
+ "9440": 33394679808.0,
+ "9445": 33396217856.0,
+ "9450": 33394835456.0,
+ "9455": 33394302976.0,
+ "9460": 33396176896.0,
+ "9465": 33395183616.0,
+ "9470": 33395994624.0,
+ "9475": 33396236288.0,
+ "9480": 33396156416.0,
+ "9485": 33396099072.0,
+ "9490": 33396015104.0,
+ "9495": 33394454528.0,
+ "9500": 33394573312.0,
+ "9505": 33396008960.0,
+ "9510": 33395230720.0,
+ "9515": 33395333120.0,
+ "9520": 33395853312.0,
+ "9525": 33395834880.0,
+ "9530": 33395890176.0,
+ "9535": 33395396608.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 9535,
+ "step_interval": 5,
+ "values": {
+ "1": 37005148160.0,
+ "5": 45108281344.0,
+ "10": 45281837056.0,
+ "15": 45281837056.0,
+ "20": 45348462592.0,
+ "25": 45384892416.0,
+ "30": 45384892416.0,
+ "35": 45384892416.0,
+ "40": 45384892416.0,
+ "45": 45384892416.0,
+ "50": 45384892416.0,
+ "55": 45384892416.0,
+ "60": 45384892416.0,
+ "65": 45384892416.0,
+ "70": 45384892416.0,
+ "75": 45384892416.0,
+ "80": 45384892416.0,
+ "85": 45384892416.0,
+ "90": 45384892416.0,
+ "95": 45384892416.0,
+ "100": 45384892416.0,
+ "105": 45384892416.0,
+ "110": 45393076224.0,
+ "115": 45414809600.0,
+ "120": 45474807808.0,
+ "125": 45522124800.0,
+ "130": 45522124800.0,
+ "135": 45531688960.0,
+ "140": 45531688960.0,
+ "145": 45531688960.0,
+ "150": 45531688960.0,
+ "155": 45531688960.0,
+ "160": 45531688960.0,
+ "165": 45531688960.0,
+ "170": 45531688960.0,
+ "175": 45531688960.0,
+ "180": 45531688960.0,
+ "185": 45531688960.0,
+ "190": 45531688960.0,
+ "195": 45531688960.0,
+ "200": 45623947264.0,
+ "205": 45728985088.0,
+ "210": 45728985088.0,
+ "215": 45728985088.0,
+ "220": 45728985088.0,
+ "225": 45728985088.0,
+ "230": 45728985088.0,
+ "235": 45827989504.0,
+ "240": 46031273984.0,
+ "245": 46066565120.0,
+ "250": 46066565120.0,
+ "255": 46066565120.0,
+ "260": 46119915520.0,
+ "265": 46130237440.0,
+ "270": 46130237440.0,
+ "275": 46154203136.0,
+ "280": 46426300416.0,
+ "285": 46426300416.0,
+ "290": 46426300416.0,
+ "295": 46426300416.0,
+ "300": 46426300416.0,
+ "305": 46426300416.0,
+ "310": 46426300416.0,
+ "315": 46426300416.0,
+ "320": 46426300416.0,
+ "325": 46426300416.0,
+ "330": 46426300416.0,
+ "335": 46426300416.0,
+ "340": 46426300416.0,
+ "345": 46541557760.0,
+ "350": 46541557760.0,
+ "355": 46541557760.0,
+ "360": 46541557760.0,
+ "365": 46541557760.0,
+ "370": 46541557760.0,
+ "375": 46541557760.0,
+ "380": 46541557760.0,
+ "385": 46541557760.0,
+ "390": 46541557760.0,
+ "395": 46541557760.0,
+ "400": 46541557760.0,
+ "405": 46541557760.0,
+ "410": 46541557760.0,
+ "415": 46541557760.0,
+ "420": 46541557760.0,
+ "425": 46541557760.0,
+ "430": 46541557760.0,
+ "435": 46541557760.0,
+ "440": 46541557760.0,
+ "445": 46541557760.0,
+ "450": 46541557760.0,
+ "455": 46541557760.0,
+ "460": 46541557760.0,
+ "465": 46541557760.0,
+ "470": 46541557760.0,
+ "475": 46541557760.0,
+ "480": 46541557760.0,
+ "485": 46541557760.0,
+ "490": 46657650688.0,
+ "495": 46657650688.0,
+ "500": 46657650688.0,
+ "505": 46657650688.0,
+ "510": 46657650688.0,
+ "515": 46657650688.0,
+ "520": 46657650688.0,
+ "525": 46657650688.0,
+ "530": 46657650688.0,
+ "535": 46657650688.0,
+ "540": 46657650688.0,
+ "545": 46657650688.0,
+ "550": 46657650688.0,
+ "555": 46657650688.0,
+ "560": 46657650688.0,
+ "565": 46657650688.0,
+ "570": 46657650688.0,
+ "575": 46657650688.0,
+ "580": 46657650688.0,
+ "585": 46657650688.0,
+ "590": 46657650688.0,
+ "595": 46657650688.0,
+ "600": 46657650688.0,
+ "605": 46657650688.0,
+ "610": 46657650688.0,
+ "615": 46657650688.0,
+ "620": 46657650688.0,
+ "625": 46657650688.0,
+ "630": 46657650688.0,
+ "635": 46657650688.0,
+ "640": 46657650688.0,
+ "645": 46657650688.0,
+ "650": 46657650688.0,
+ "655": 46657650688.0,
+ "660": 46657650688.0,
+ "665": 46657650688.0,
+ "670": 46657650688.0,
+ "675": 46657650688.0,
+ "680": 46657650688.0,
+ "685": 46657650688.0,
+ "690": 46657650688.0,
+ "695": 46657650688.0,
+ "700": 46657650688.0,
+ "705": 46657650688.0,
+ "710": 46657650688.0,
+ "715": 46657650688.0,
+ "720": 46657650688.0,
+ "725": 46657650688.0,
+ "730": 46657650688.0,
+ "735": 46657650688.0,
+ "740": 46657650688.0,
+ "745": 46657650688.0,
+ "750": 46657650688.0,
+ "755": 46657650688.0,
+ "760": 46657650688.0,
+ "765": 46657650688.0,
+ "770": 46657650688.0,
+ "775": 46657650688.0,
+ "780": 46657650688.0,
+ "785": 46657650688.0,
+ "790": 46657650688.0,
+ "795": 46657650688.0,
+ "800": 46657650688.0,
+ "805": 46657650688.0,
+ "810": 46657650688.0,
+ "815": 46657650688.0,
+ "820": 46657650688.0,
+ "825": 46657650688.0,
+ "830": 46657650688.0,
+ "835": 46657650688.0,
+ "840": 46657650688.0,
+ "845": 46657650688.0,
+ "850": 46657650688.0,
+ "855": 46657650688.0,
+ "860": 46657650688.0,
+ "865": 46657650688.0,
+ "870": 46657650688.0,
+ "875": 46657650688.0,
+ "880": 46657650688.0,
+ "885": 46657650688.0,
+ "890": 46657650688.0,
+ "895": 46657650688.0,
+ "900": 46657650688.0,
+ "905": 46657650688.0,
+ "910": 46657650688.0,
+ "915": 46657650688.0,
+ "920": 46657650688.0,
+ "925": 46657650688.0,
+ "930": 46657650688.0,
+ "935": 46657650688.0,
+ "940": 46657650688.0,
+ "945": 46657650688.0,
+ "950": 46657650688.0,
+ "955": 46657650688.0,
+ "960": 46657650688.0,
+ "965": 46657650688.0,
+ "970": 46657650688.0,
+ "975": 46657650688.0,
+ "980": 46657650688.0,
+ "985": 46657650688.0,
+ "990": 46657650688.0,
+ "995": 46657650688.0,
+ "1000": 46657650688.0,
+ "1005": 46657650688.0,
+ "1010": 46657650688.0,
+ "1015": 46657650688.0,
+ "1020": 46657650688.0,
+ "1025": 46657650688.0,
+ "1030": 46657650688.0,
+ "1035": 46657650688.0,
+ "1040": 46657650688.0,
+ "1045": 46657650688.0,
+ "1050": 46657650688.0,
+ "1055": 46657650688.0,
+ "1060": 46657650688.0,
+ "1065": 46657650688.0,
+ "1070": 46657650688.0,
+ "1075": 46657650688.0,
+ "1080": 46657650688.0,
+ "1085": 46657650688.0,
+ "1090": 46657650688.0,
+ "1095": 46657650688.0,
+ "1100": 46657650688.0,
+ "1105": 46657650688.0,
+ "1110": 46657650688.0,
+ "1115": 46657650688.0,
+ "1120": 46657650688.0,
+ "1125": 46657650688.0,
+ "1130": 46657650688.0,
+ "1135": 46657650688.0,
+ "1140": 46657650688.0,
+ "1145": 46657650688.0,
+ "1150": 46657650688.0,
+ "1155": 46657650688.0,
+ "1160": 46657650688.0,
+ "1165": 46657650688.0,
+ "1170": 46657650688.0,
+ "1175": 46657650688.0,
+ "1180": 46371139584.0,
+ "1185": 46371139584.0,
+ "1190": 46371139584.0,
+ "1195": 46371139584.0,
+ "1200": 46371139584.0,
+ "1205": 46371139584.0,
+ "1210": 46371139584.0,
+ "1215": 46371139584.0,
+ "1220": 46371139584.0,
+ "1225": 46371139584.0,
+ "1230": 46371139584.0,
+ "1235": 46371139584.0,
+ "1240": 46371139584.0,
+ "1245": 46371139584.0,
+ "1250": 46371139584.0,
+ "1255": 46371139584.0,
+ "1260": 46371139584.0,
+ "1265": 46371139584.0,
+ "1270": 46371139584.0,
+ "1275": 46371139584.0,
+ "1280": 46371139584.0,
+ "1285": 46371139584.0,
+ "1290": 46371139584.0,
+ "1295": 46371139584.0,
+ "1300": 46371139584.0,
+ "1305": 46371139584.0,
+ "1310": 46371139584.0,
+ "1315": 46371139584.0,
+ "1320": 46371139584.0,
+ "1325": 46371139584.0,
+ "1330": 46371139584.0,
+ "1335": 46371139584.0,
+ "1340": 46371139584.0,
+ "1345": 46371139584.0,
+ "1350": 46371139584.0,
+ "1355": 46371139584.0,
+ "1360": 46371139584.0,
+ "1365": 46371139584.0,
+ "1370": 46371139584.0,
+ "1375": 46371139584.0,
+ "1380": 46371139584.0,
+ "1385": 46371139584.0,
+ "1390": 46371139584.0,
+ "1395": 46371139584.0,
+ "1400": 46371139584.0,
+ "1405": 46371139584.0,
+ "1410": 46371139584.0,
+ "1415": 46371139584.0,
+ "1420": 46371139584.0,
+ "1425": 46371139584.0,
+ "1430": 46371139584.0,
+ "1435": 46371139584.0,
+ "1440": 46371139584.0,
+ "1445": 46371139584.0,
+ "1450": 46371139584.0,
+ "1455": 46371139584.0,
+ "1460": 46371139584.0,
+ "1465": 46371139584.0,
+ "1470": 46371139584.0,
+ "1475": 46371139584.0,
+ "1480": 46371139584.0,
+ "1485": 46371139584.0,
+ "1490": 46371139584.0,
+ "1495": 46371139584.0,
+ "1500": 46371139584.0,
+ "1505": 46371139584.0,
+ "1510": 46371139584.0,
+ "1515": 46371139584.0,
+ "1520": 46371139584.0,
+ "1525": 46371139584.0,
+ "1530": 46371139584.0,
+ "1535": 46371139584.0,
+ "1540": 46371139584.0,
+ "1545": 46371139584.0,
+ "1550": 46371139584.0,
+ "1555": 46371139584.0,
+ "1560": 46371139584.0,
+ "1565": 46371139584.0,
+ "1570": 46371139584.0,
+ "1575": 46371139584.0,
+ "1580": 46371139584.0,
+ "1585": 46371139584.0,
+ "1590": 46371139584.0,
+ "1595": 46371139584.0,
+ "1600": 46371139584.0,
+ "1605": 46371139584.0,
+ "1610": 46371139584.0,
+ "1615": 46371139584.0,
+ "1620": 46371139584.0,
+ "1625": 46371139584.0,
+ "1630": 46371139584.0,
+ "1635": 46371139584.0,
+ "1640": 46371139584.0,
+ "1645": 46371139584.0,
+ "1650": 46371139584.0,
+ "1655": 46371139584.0,
+ "1660": 46371139584.0,
+ "1665": 46371139584.0,
+ "1670": 46371139584.0,
+ "1675": 46371139584.0,
+ "1680": 46371139584.0,
+ "1685": 46371139584.0,
+ "1690": 46371139584.0,
+ "1695": 46371139584.0,
+ "1700": 46371139584.0,
+ "1705": 46371139584.0,
+ "1710": 46371139584.0,
+ "1715": 46371139584.0,
+ "1720": 46371139584.0,
+ "1725": 46371139584.0,
+ "1730": 46371139584.0,
+ "1735": 46371139584.0,
+ "1740": 46371139584.0,
+ "1745": 46371139584.0,
+ "1750": 46371139584.0,
+ "1755": 46371139584.0,
+ "1760": 46371139584.0,
+ "1765": 46371139584.0,
+ "1770": 46371139584.0,
+ "1775": 46371139584.0,
+ "1780": 46371139584.0,
+ "1785": 46371139584.0,
+ "1790": 46371139584.0,
+ "1795": 46371139584.0,
+ "1800": 46371139584.0,
+ "1805": 46371139584.0,
+ "1810": 46371139584.0,
+ "1815": 46371139584.0,
+ "1820": 46371139584.0,
+ "1825": 46371139584.0,
+ "1830": 46371139584.0,
+ "1835": 46371139584.0,
+ "1840": 46371139584.0,
+ "1845": 46371139584.0,
+ "1850": 46371139584.0,
+ "1855": 46371139584.0,
+ "1860": 46371139584.0,
+ "1865": 46371139584.0,
+ "1870": 46371139584.0,
+ "1875": 46371139584.0,
+ "1880": 46371139584.0,
+ "1885": 46371139584.0,
+ "1890": 46371139584.0,
+ "1895": 46371139584.0,
+ "1900": 46371139584.0,
+ "1905": 46371139584.0,
+ "1910": 46371139584.0,
+ "1915": 46371139584.0,
+ "1920": 46371139584.0,
+ "1925": 46371139584.0,
+ "1930": 46371139584.0,
+ "1935": 46371139584.0,
+ "1940": 46371139584.0,
+ "1945": 46371139584.0,
+ "1950": 46371139584.0,
+ "1955": 46371139584.0,
+ "1960": 46371139584.0,
+ "1965": 46371139584.0,
+ "1970": 46376312832.0,
+ "1975": 46376312832.0,
+ "1980": 46376312832.0,
+ "1985": 46376312832.0,
+ "1990": 46376312832.0,
+ "1995": 46376312832.0,
+ "2000": 46376312832.0,
+ "2005": 46376312832.0,
+ "2010": 46376312832.0,
+ "2015": 46376312832.0,
+ "2020": 46376312832.0,
+ "2025": 46376312832.0,
+ "2030": 46376312832.0,
+ "2035": 46376312832.0,
+ "2040": 46376312832.0,
+ "2045": 46376312832.0,
+ "2050": 46376312832.0,
+ "2055": 46376312832.0,
+ "2060": 46376312832.0,
+ "2065": 46376312832.0,
+ "2070": 46376312832.0,
+ "2075": 46376312832.0,
+ "2080": 46376312832.0,
+ "2085": 46376312832.0,
+ "2090": 46376312832.0,
+ "2095": 45943558144.0,
+ "2100": 45964951552.0,
+ "2105": 46044495872.0,
+ "2110": 46078029824.0,
+ "2115": 46078029824.0,
+ "2120": 46078029824.0,
+ "2125": 46078029824.0,
+ "2130": 46208413696.0,
+ "2135": 46208413696.0,
+ "2140": 46208413696.0,
+ "2145": 46208413696.0,
+ "2150": 46208413696.0,
+ "2155": 46208413696.0,
+ "2160": 46208413696.0,
+ "2165": 46208413696.0,
+ "2170": 46208413696.0,
+ "2175": 46208413696.0,
+ "2180": 46208413696.0,
+ "2185": 46208413696.0,
+ "2190": 46208413696.0,
+ "2195": 46208413696.0,
+ "2200": 46208413696.0,
+ "2205": 46208413696.0,
+ "2210": 46208413696.0,
+ "2215": 46208413696.0,
+ "2220": 46208413696.0,
+ "2225": 46208413696.0,
+ "2230": 46208413696.0,
+ "2235": 46208413696.0,
+ "2240": 46208413696.0,
+ "2245": 46208413696.0,
+ "2250": 46208413696.0,
+ "2255": 46208413696.0,
+ "2260": 46208413696.0,
+ "2265": 46208413696.0,
+ "2270": 46208413696.0,
+ "2275": 46208413696.0,
+ "2280": 46208413696.0,
+ "2285": 46208413696.0,
+ "2290": 46208413696.0,
+ "2295": 46208413696.0,
+ "2300": 46223462400.0,
+ "2305": 46223462400.0,
+ "2310": 46223462400.0,
+ "2315": 46223462400.0,
+ "2320": 46223462400.0,
+ "2325": 46223462400.0,
+ "2330": 46223462400.0,
+ "2335": 46223462400.0,
+ "2340": 46223462400.0,
+ "2345": 46223462400.0,
+ "2350": 46223462400.0,
+ "2355": 46223462400.0,
+ "2360": 46223462400.0,
+ "2365": 46223462400.0,
+ "2370": 46223462400.0,
+ "2375": 46223462400.0,
+ "2380": 46223462400.0,
+ "2385": 46223462400.0,
+ "2390": 46335651840.0,
+ "2395": 46335651840.0,
+ "2400": 46335651840.0,
+ "2405": 46335651840.0,
+ "2410": 46335651840.0,
+ "2415": 46335651840.0,
+ "2420": 46335651840.0,
+ "2425": 46335651840.0,
+ "2430": 46335651840.0,
+ "2435": 46335651840.0,
+ "2440": 46335651840.0,
+ "2445": 46335651840.0,
+ "2450": 46335651840.0,
+ "2455": 46335651840.0,
+ "2460": 46335651840.0,
+ "2465": 46335651840.0,
+ "2470": 46335651840.0,
+ "2475": 46335651840.0,
+ "2480": 46335651840.0,
+ "2485": 46335651840.0,
+ "2490": 46335651840.0,
+ "2495": 46335651840.0,
+ "2500": 46335651840.0,
+ "2505": 46335651840.0,
+ "2510": 46335651840.0,
+ "2515": 46335651840.0,
+ "2520": 46335651840.0,
+ "2525": 46335651840.0,
+ "2530": 46335651840.0,
+ "2535": 46335651840.0,
+ "2540": 46335651840.0,
+ "2545": 46335651840.0,
+ "2550": 46335651840.0,
+ "2555": 46335651840.0,
+ "2560": 46335651840.0,
+ "2565": 46335651840.0,
+ "2570": 46335651840.0,
+ "2575": 46335651840.0,
+ "2580": 46335651840.0,
+ "2585": 46335651840.0,
+ "2590": 46335651840.0,
+ "2595": 46335651840.0,
+ "2600": 46335651840.0,
+ "2605": 46335651840.0,
+ "2610": 46335651840.0,
+ "2615": 46335651840.0,
+ "2620": 46335651840.0,
+ "2625": 46335651840.0,
+ "2630": 46335651840.0,
+ "2635": 46335651840.0,
+ "2640": 46335651840.0,
+ "2645": 46335651840.0,
+ "2650": 46335651840.0,
+ "2655": 46335651840.0,
+ "2660": 46335651840.0,
+ "2665": 46335651840.0,
+ "2670": 46335651840.0,
+ "2675": 46335651840.0,
+ "2680": 46335651840.0,
+ "2685": 46335651840.0,
+ "2690": 46335651840.0,
+ "2695": 46335651840.0,
+ "2700": 46335651840.0,
+ "2705": 46335651840.0,
+ "2710": 46335651840.0,
+ "2715": 46335651840.0,
+ "2720": 46335651840.0,
+ "2725": 46335651840.0,
+ "2730": 46335651840.0,
+ "2735": 46335651840.0,
+ "2740": 46335651840.0,
+ "2745": 46335651840.0,
+ "2750": 46335651840.0,
+ "2755": 46335651840.0,
+ "2760": 46335651840.0,
+ "2765": 46335651840.0,
+ "2770": 46335651840.0,
+ "2775": 46335651840.0,
+ "2780": 46335651840.0,
+ "2785": 46335651840.0,
+ "2790": 46335651840.0,
+ "2795": 46335651840.0,
+ "2800": 46335651840.0,
+ "2805": 46335651840.0,
+ "2810": 46335651840.0,
+ "2815": 46335651840.0,
+ "2820": 46335651840.0,
+ "2825": 46335651840.0,
+ "2830": 46335651840.0,
+ "2835": 46335651840.0,
+ "2840": 46335651840.0,
+ "2845": 46335651840.0,
+ "2850": 46335651840.0,
+ "2855": 46335651840.0,
+ "2860": 46335651840.0,
+ "2865": 46335651840.0,
+ "2870": 46335651840.0,
+ "2875": 46335651840.0,
+ "2880": 46335651840.0,
+ "2885": 46335651840.0,
+ "2890": 46335651840.0,
+ "2895": 46335651840.0,
+ "2900": 46335651840.0,
+ "2905": 46335651840.0,
+ "2910": 46335651840.0,
+ "2915": 46335651840.0,
+ "2920": 46335651840.0,
+ "2925": 46335651840.0,
+ "2930": 46335651840.0,
+ "2935": 46335651840.0,
+ "2940": 46335651840.0,
+ "2945": 46335651840.0,
+ "2950": 46335651840.0,
+ "2955": 46335651840.0,
+ "2960": 46335651840.0,
+ "2965": 46335651840.0,
+ "2970": 46335651840.0,
+ "2975": 46335651840.0,
+ "2980": 46335651840.0,
+ "2985": 46335651840.0,
+ "2990": 46335651840.0,
+ "2995": 46335651840.0,
+ "3000": 46335651840.0,
+ "3005": 46335651840.0,
+ "3010": 46335651840.0,
+ "3015": 45624537088.0,
+ "3020": 46012612608.0,
+ "3025": 46012612608.0,
+ "3030": 46012612608.0,
+ "3035": 46012612608.0,
+ "3040": 46012612608.0,
+ "3045": 46012612608.0,
+ "3050": 46012612608.0,
+ "3055": 46012612608.0,
+ "3060": 46069665792.0,
+ "3065": 46069665792.0,
+ "3070": 46069665792.0,
+ "3075": 46069665792.0,
+ "3080": 46069665792.0,
+ "3085": 46069665792.0,
+ "3090": 46069665792.0,
+ "3095": 46069665792.0,
+ "3100": 46069665792.0,
+ "3105": 46131302400.0,
+ "3110": 46131302400.0,
+ "3115": 46131302400.0,
+ "3120": 46131302400.0,
+ "3125": 46131302400.0,
+ "3130": 46137425920.0,
+ "3135": 46137425920.0,
+ "3140": 46137425920.0,
+ "3145": 46137425920.0,
+ "3150": 46137425920.0,
+ "3155": 46137425920.0,
+ "3160": 46137425920.0,
+ "3165": 46137425920.0,
+ "3170": 46137425920.0,
+ "3175": 46137425920.0,
+ "3180": 46137425920.0,
+ "3185": 46137425920.0,
+ "3190": 46137425920.0,
+ "3195": 46137425920.0,
+ "3200": 46137425920.0,
+ "3205": 46137425920.0,
+ "3210": 46137425920.0,
+ "3215": 46137425920.0,
+ "3220": 46137425920.0,
+ "3225": 46137425920.0,
+ "3230": 46137425920.0,
+ "3235": 46137425920.0,
+ "3240": 46137425920.0,
+ "3245": 46137425920.0,
+ "3250": 46137425920.0,
+ "3255": 46137425920.0,
+ "3260": 46137425920.0,
+ "3265": 46137425920.0,
+ "3270": 46137425920.0,
+ "3275": 46137425920.0,
+ "3280": 46137425920.0,
+ "3285": 46137425920.0,
+ "3290": 46137425920.0,
+ "3295": 46137425920.0,
+ "3300": 46137425920.0,
+ "3305": 46137425920.0,
+ "3310": 46137425920.0,
+ "3315": 46137425920.0,
+ "3320": 46137425920.0,
+ "3325": 46137425920.0,
+ "3330": 46137425920.0,
+ "3335": 46137425920.0,
+ "3340": 46137425920.0,
+ "3345": 46137425920.0,
+ "3350": 46137425920.0,
+ "3355": 46137425920.0,
+ "3360": 46137425920.0,
+ "3365": 46137425920.0,
+ "3370": 46137425920.0,
+ "3375": 46137425920.0,
+ "3380": 46137425920.0,
+ "3385": 46137425920.0,
+ "3390": 46137425920.0,
+ "3395": 46137425920.0,
+ "3400": 46137425920.0,
+ "3405": 46137425920.0,
+ "3410": 46137425920.0,
+ "3415": 46137425920.0,
+ "3420": 46137425920.0,
+ "3425": 46137425920.0,
+ "3430": 46137425920.0,
+ "3435": 46137425920.0,
+ "3440": 46137425920.0,
+ "3445": 46138445824.0,
+ "3450": 46138445824.0,
+ "3455": 46138445824.0,
+ "3460": 46138445824.0,
+ "3465": 46138445824.0,
+ "3470": 46138445824.0,
+ "3475": 46138445824.0,
+ "3480": 46138445824.0,
+ "3485": 46171656192.0,
+ "3490": 46171656192.0,
+ "3495": 46171656192.0,
+ "3500": 46250127360.0,
+ "3505": 46250127360.0,
+ "3510": 46250127360.0,
+ "3515": 46250127360.0,
+ "3520": 46250127360.0,
+ "3525": 46250127360.0,
+ "3530": 46250127360.0,
+ "3535": 46250127360.0,
+ "3540": 46250127360.0,
+ "3545": 46250127360.0,
+ "3550": 46250127360.0,
+ "3555": 46250127360.0,
+ "3560": 46250127360.0,
+ "3565": 46250127360.0,
+ "3570": 46250127360.0,
+ "3575": 46250127360.0,
+ "3580": 46250127360.0,
+ "3585": 46250127360.0,
+ "3590": 46250127360.0,
+ "3595": 46250127360.0,
+ "3600": 46250127360.0,
+ "3605": 46250127360.0,
+ "3610": 46250127360.0,
+ "3615": 46250127360.0,
+ "3620": 46250127360.0,
+ "3625": 46250127360.0,
+ "3630": 46250127360.0,
+ "3635": 46250127360.0,
+ "3640": 46250127360.0,
+ "3645": 46250127360.0,
+ "3650": 46250127360.0,
+ "3655": 46250127360.0,
+ "3660": 46250127360.0,
+ "3665": 46250127360.0,
+ "3670": 46250127360.0,
+ "3675": 46250127360.0,
+ "3680": 46250127360.0,
+ "3685": 46250127360.0,
+ "3690": 46250127360.0,
+ "3695": 46250127360.0,
+ "3700": 46250127360.0,
+ "3705": 46250127360.0,
+ "3710": 46250127360.0,
+ "3715": 46250127360.0,
+ "3720": 46250127360.0,
+ "3725": 46250127360.0,
+ "3730": 46250127360.0,
+ "3735": 46250127360.0,
+ "3740": 46250127360.0,
+ "3745": 46250127360.0,
+ "3750": 46250127360.0,
+ "3755": 46250127360.0,
+ "3760": 46250127360.0,
+ "3765": 46250127360.0,
+ "3770": 46250127360.0,
+ "3775": 46250127360.0,
+ "3780": 46250127360.0,
+ "3785": 46250127360.0,
+ "3790": 46250127360.0,
+ "3795": 46250127360.0,
+ "3800": 46250127360.0,
+ "3805": 46250127360.0,
+ "3810": 46250127360.0,
+ "3815": 46250127360.0,
+ "3820": 46250127360.0,
+ "3825": 46250127360.0,
+ "3830": 46250127360.0,
+ "3835": 46250127360.0,
+ "3840": 46250127360.0,
+ "3845": 46250127360.0,
+ "3850": 46250127360.0,
+ "3855": 46250127360.0,
+ "3860": 46250127360.0,
+ "3865": 46250127360.0,
+ "3870": 46250127360.0,
+ "3875": 46250127360.0,
+ "3880": 46250127360.0,
+ "3885": 46250127360.0,
+ "3890": 46250127360.0,
+ "3895": 46250127360.0,
+ "3900": 46250127360.0,
+ "3905": 46250127360.0,
+ "3910": 46250127360.0,
+ "3915": 46250127360.0,
+ "3920": 46250127360.0,
+ "3925": 46250127360.0,
+ "3930": 46250127360.0,
+ "3935": 46250127360.0,
+ "3940": 46250127360.0,
+ "3945": 46250127360.0,
+ "3950": 46250127360.0,
+ "3955": 46184550400.0,
+ "3960": 46184550400.0,
+ "3965": 46184550400.0,
+ "3970": 46184550400.0,
+ "3975": 46184550400.0,
+ "3980": 46184550400.0,
+ "3985": 46184550400.0,
+ "3990": 46184550400.0,
+ "3995": 46184550400.0,
+ "4000": 46184550400.0,
+ "4005": 46184550400.0,
+ "4010": 46184550400.0,
+ "4015": 46184550400.0,
+ "4020": 46184550400.0,
+ "4025": 46184550400.0,
+ "4030": 46184550400.0,
+ "4035": 46184550400.0,
+ "4040": 46184550400.0,
+ "4045": 46184550400.0,
+ "4050": 46184550400.0,
+ "4055": 46184550400.0,
+ "4060": 46184550400.0,
+ "4065": 46184550400.0,
+ "4070": 46184550400.0,
+ "4075": 46184550400.0,
+ "4080": 46184550400.0,
+ "4085": 46184550400.0,
+ "4090": 46184550400.0,
+ "4095": 46184550400.0,
+ "4100": 46184550400.0,
+ "4105": 46184550400.0,
+ "4110": 46184550400.0,
+ "4115": 46184550400.0,
+ "4120": 46184550400.0,
+ "4125": 46184550400.0,
+ "4130": 46184550400.0,
+ "4135": 46184550400.0,
+ "4140": 46184550400.0,
+ "4145": 46184550400.0,
+ "4150": 46184550400.0,
+ "4155": 46184550400.0,
+ "4160": 46184550400.0,
+ "4165": 46184550400.0,
+ "4170": 46184550400.0,
+ "4175": 46184550400.0,
+ "4180": 46184550400.0,
+ "4185": 46184550400.0,
+ "4190": 46184550400.0,
+ "4195": 46184550400.0,
+ "4200": 46184550400.0,
+ "4205": 46184550400.0,
+ "4210": 46184550400.0,
+ "4215": 46184550400.0,
+ "4220": 46184550400.0,
+ "4225": 46184550400.0,
+ "4230": 46184550400.0,
+ "4235": 46184550400.0,
+ "4240": 46184550400.0,
+ "4245": 46184550400.0,
+ "4250": 46184550400.0,
+ "4255": 46184550400.0,
+ "4260": 46184550400.0,
+ "4265": 46184550400.0,
+ "4270": 46184550400.0,
+ "4275": 46184550400.0,
+ "4280": 46184550400.0,
+ "4285": 46184550400.0,
+ "4290": 46184550400.0,
+ "4295": 46184550400.0,
+ "4300": 46184550400.0,
+ "4305": 46184550400.0,
+ "4310": 46184550400.0,
+ "4315": 46184550400.0,
+ "4320": 46184550400.0,
+ "4325": 46184550400.0,
+ "4330": 46184550400.0,
+ "4335": 46184550400.0,
+ "4340": 46184550400.0,
+ "4345": 46184550400.0,
+ "4350": 46184550400.0,
+ "4355": 46184550400.0,
+ "4360": 46184550400.0,
+ "4365": 46184550400.0,
+ "4370": 46184550400.0,
+ "4375": 46184550400.0,
+ "4380": 46184550400.0,
+ "4385": 46184550400.0,
+ "4390": 46184550400.0,
+ "4395": 46184550400.0,
+ "4400": 46184550400.0,
+ "4405": 46184550400.0,
+ "4410": 46184550400.0,
+ "4415": 46184550400.0,
+ "4420": 46184550400.0,
+ "4425": 46184550400.0,
+ "4430": 46184550400.0,
+ "4435": 46184550400.0,
+ "4440": 46184550400.0,
+ "4445": 46184550400.0,
+ "4450": 46184550400.0,
+ "4455": 46184550400.0,
+ "4460": 46184550400.0,
+ "4465": 46184550400.0,
+ "4470": 46184550400.0,
+ "4475": 46184550400.0,
+ "4480": 46184550400.0,
+ "4485": 46184550400.0,
+ "4490": 46184550400.0,
+ "4495": 46184550400.0,
+ "4500": 46184550400.0,
+ "4505": 46184550400.0,
+ "4510": 46184550400.0,
+ "4515": 46184550400.0,
+ "4520": 46184550400.0,
+ "4525": 46184550400.0,
+ "4530": 46184550400.0,
+ "4535": 46184550400.0,
+ "4540": 46184550400.0,
+ "4545": 46184550400.0,
+ "4550": 46210957312.0,
+ "4555": 46210957312.0,
+ "4560": 46210957312.0,
+ "4565": 46210957312.0,
+ "4570": 46210957312.0,
+ "4575": 46210957312.0,
+ "4580": 46210957312.0,
+ "4585": 46210957312.0,
+ "4590": 46210957312.0,
+ "4595": 46210957312.0,
+ "4600": 46210957312.0,
+ "4605": 46210957312.0,
+ "4610": 46210957312.0,
+ "4615": 46302752768.0,
+ "4620": 46302752768.0,
+ "4625": 46302752768.0,
+ "4630": 46302752768.0,
+ "4635": 46302752768.0,
+ "4640": 46302752768.0,
+ "4645": 46302752768.0,
+ "4650": 46302752768.0,
+ "4655": 46302752768.0,
+ "4660": 46302752768.0,
+ "4665": 46302752768.0,
+ "4670": 46302752768.0,
+ "4675": 46302752768.0,
+ "4680": 46302752768.0,
+ "4685": 46302752768.0,
+ "4690": 46302752768.0,
+ "4695": 46302752768.0,
+ "4700": 46302752768.0,
+ "4705": 46302752768.0,
+ "4710": 46302752768.0,
+ "4715": 46302752768.0,
+ "4720": 46302752768.0,
+ "4725": 46302752768.0,
+ "4730": 46302752768.0,
+ "4735": 46302752768.0,
+ "4740": 46302752768.0,
+ "4745": 46302752768.0,
+ "4750": 46302752768.0,
+ "4755": 46302752768.0,
+ "4760": 46302752768.0,
+ "4765": 46302752768.0,
+ "4770": 46302752768.0,
+ "4775": 46302752768.0,
+ "4780": 46302752768.0,
+ "4785": 46302752768.0,
+ "4790": 46302752768.0,
+ "4795": 46302752768.0,
+ "4800": 46302752768.0,
+ "4805": 46302752768.0,
+ "4810": 46302752768.0,
+ "4815": 46302752768.0,
+ "4820": 46302752768.0,
+ "4825": 46302752768.0,
+ "4830": 46302752768.0,
+ "4835": 46302752768.0,
+ "4840": 46302752768.0,
+ "4845": 46302752768.0,
+ "4850": 46302752768.0,
+ "4855": 46302752768.0,
+ "4860": 46302752768.0,
+ "4865": 46302752768.0,
+ "4870": 46302752768.0,
+ "4875": 46302752768.0,
+ "4880": 46302752768.0,
+ "4885": 45956227072.0,
+ "4890": 45969969152.0,
+ "4895": 45973594112.0,
+ "4900": 45973594112.0,
+ "4905": 46015721472.0,
+ "4910": 46015721472.0,
+ "4915": 46015721472.0,
+ "4920": 46015721472.0,
+ "4925": 46260936704.0,
+ "4930": 46260936704.0,
+ "4935": 46260936704.0,
+ "4940": 46260936704.0,
+ "4945": 46260936704.0,
+ "4950": 46260936704.0,
+ "4955": 46260936704.0,
+ "4960": 46260936704.0,
+ "4965": 46260936704.0,
+ "4970": 46260936704.0,
+ "4975": 46260936704.0,
+ "4980": 46260936704.0,
+ "4985": 46260936704.0,
+ "4990": 46260936704.0,
+ "4995": 46260936704.0,
+ "5000": 46260936704.0,
+ "5005": 46260936704.0,
+ "5010": 46260936704.0,
+ "5015": 46260936704.0,
+ "5020": 46260936704.0,
+ "5025": 46260936704.0,
+ "5030": 46260936704.0,
+ "5035": 46260936704.0,
+ "5040": 46260936704.0,
+ "5045": 46260936704.0,
+ "5050": 46260936704.0,
+ "5055": 46260936704.0,
+ "5060": 46260936704.0,
+ "5065": 46260936704.0,
+ "5070": 46260936704.0,
+ "5075": 46260936704.0,
+ "5080": 46260936704.0,
+ "5085": 46260936704.0,
+ "5090": 46260936704.0,
+ "5095": 46260936704.0,
+ "5100": 46260936704.0,
+ "5105": 46260936704.0,
+ "5110": 46260936704.0,
+ "5115": 46260936704.0,
+ "5120": 46260936704.0,
+ "5125": 46260936704.0,
+ "5130": 46260936704.0,
+ "5135": 46260936704.0,
+ "5140": 46260936704.0,
+ "5145": 46260936704.0,
+ "5150": 46260936704.0,
+ "5155": 46260936704.0,
+ "5160": 46260936704.0,
+ "5165": 46260936704.0,
+ "5170": 46260936704.0,
+ "5175": 46260936704.0,
+ "5180": 46260936704.0,
+ "5185": 46260936704.0,
+ "5190": 46260936704.0,
+ "5195": 46260936704.0,
+ "5200": 46267232256.0,
+ "5205": 46267232256.0,
+ "5210": 46267232256.0,
+ "5215": 46267232256.0,
+ "5220": 46267232256.0,
+ "5225": 46267232256.0,
+ "5230": 46267232256.0,
+ "5235": 46267232256.0,
+ "5240": 46267232256.0,
+ "5245": 46267232256.0,
+ "5250": 46267232256.0,
+ "5255": 46267232256.0,
+ "5260": 46267232256.0,
+ "5265": 46267232256.0,
+ "5270": 46267232256.0,
+ "5275": 46267232256.0,
+ "5280": 46267232256.0,
+ "5285": 46267232256.0,
+ "5290": 46267232256.0,
+ "5295": 46292983808.0,
+ "5300": 46292983808.0,
+ "5305": 46292983808.0,
+ "5310": 46292983808.0,
+ "5315": 46292983808.0,
+ "5320": 46292983808.0,
+ "5325": 46292983808.0,
+ "5330": 46292983808.0,
+ "5335": 46292983808.0,
+ "5340": 46292983808.0,
+ "5345": 46292983808.0,
+ "5350": 46292983808.0,
+ "5355": 46292983808.0,
+ "5360": 46292983808.0,
+ "5365": 46292983808.0,
+ "5370": 46292983808.0,
+ "5375": 46292983808.0,
+ "5380": 46292983808.0,
+ "5385": 46292983808.0,
+ "5390": 46292983808.0,
+ "5395": 46292983808.0,
+ "5400": 46292983808.0,
+ "5405": 46292983808.0,
+ "5410": 46292983808.0,
+ "5415": 46292983808.0,
+ "5420": 46292983808.0,
+ "5425": 46292983808.0,
+ "5430": 46292983808.0,
+ "5435": 46292983808.0,
+ "5440": 46292983808.0,
+ "5445": 46292983808.0,
+ "5450": 46292983808.0,
+ "5455": 46292983808.0,
+ "5460": 46292983808.0,
+ "5465": 46292983808.0,
+ "5470": 46292983808.0,
+ "5475": 46292983808.0,
+ "5480": 46292983808.0,
+ "5485": 46292983808.0,
+ "5490": 46292983808.0,
+ "5495": 46292983808.0,
+ "5500": 46292983808.0,
+ "5505": 46292983808.0,
+ "5510": 46292983808.0,
+ "5515": 46292983808.0,
+ "5520": 46292983808.0,
+ "5525": 46292983808.0,
+ "5530": 46292983808.0,
+ "5535": 46292983808.0,
+ "5540": 46292983808.0,
+ "5545": 46292983808.0,
+ "5550": 46292983808.0,
+ "5555": 46292983808.0,
+ "5560": 46292983808.0,
+ "5565": 46292983808.0,
+ "5570": 46292983808.0,
+ "5575": 46292983808.0,
+ "5580": 46292983808.0,
+ "5585": 46292983808.0,
+ "5590": 46292983808.0,
+ "5595": 46292983808.0,
+ "5600": 46292983808.0,
+ "5605": 46292983808.0,
+ "5610": 46292983808.0,
+ "5615": 46292983808.0,
+ "5620": 46292983808.0,
+ "5625": 46292983808.0,
+ "5630": 46292983808.0,
+ "5635": 46292983808.0,
+ "5640": 46292983808.0,
+ "5645": 46292983808.0,
+ "5650": 46292983808.0,
+ "5655": 46292983808.0,
+ "5660": 46292983808.0,
+ "5665": 46292983808.0,
+ "5670": 46292983808.0,
+ "5675": 46292983808.0,
+ "5680": 46292983808.0,
+ "5685": 46292983808.0,
+ "5690": 46292983808.0,
+ "5695": 46292983808.0,
+ "5700": 46292983808.0,
+ "5705": 46292983808.0,
+ "5710": 46292983808.0,
+ "5715": 46292983808.0,
+ "5720": 46292983808.0,
+ "5725": 46292983808.0,
+ "5730": 46292983808.0,
+ "5735": 46292983808.0,
+ "5740": 46292983808.0,
+ "5745": 46292983808.0,
+ "5750": 46292983808.0,
+ "5755": 46292983808.0,
+ "5760": 46292983808.0,
+ "5765": 46292983808.0,
+ "5770": 46292983808.0,
+ "5775": 46292983808.0,
+ "5780": 46292983808.0,
+ "5785": 46292983808.0,
+ "5790": 46292983808.0,
+ "5795": 46292983808.0,
+ "5800": 46292983808.0,
+ "5805": 46292983808.0,
+ "5810": 46292983808.0,
+ "5815": 45880664064.0,
+ "5820": 45880664064.0,
+ "5825": 45880664064.0,
+ "5830": 46040887296.0,
+ "5835": 46040887296.0,
+ "5840": 46040887296.0,
+ "5845": 46040887296.0,
+ "5850": 46040887296.0,
+ "5855": 46040887296.0,
+ "5860": 46040887296.0,
+ "5865": 46040887296.0,
+ "5870": 46040887296.0,
+ "5875": 46040887296.0,
+ "5880": 46040887296.0,
+ "5885": 46040887296.0,
+ "5890": 46040887296.0,
+ "5895": 46040887296.0,
+ "5900": 46040887296.0,
+ "5905": 46040887296.0,
+ "5910": 46130806784.0,
+ "5915": 46130806784.0,
+ "5920": 46130806784.0,
+ "5925": 46130806784.0,
+ "5930": 46130806784.0,
+ "5935": 46130806784.0,
+ "5940": 46130806784.0,
+ "5945": 46130806784.0,
+ "5950": 46130806784.0,
+ "5955": 46130806784.0,
+ "5960": 46130806784.0,
+ "5965": 46130806784.0,
+ "5970": 46130806784.0,
+ "5975": 46130806784.0,
+ "5980": 46130806784.0,
+ "5985": 46130806784.0,
+ "5990": 46130806784.0,
+ "5995": 46130806784.0,
+ "6000": 46130806784.0,
+ "6005": 46130806784.0,
+ "6010": 46130806784.0,
+ "6015": 46130806784.0,
+ "6020": 46130806784.0,
+ "6025": 46130806784.0,
+ "6030": 46130806784.0,
+ "6035": 46130806784.0,
+ "6040": 46130806784.0,
+ "6045": 46130806784.0,
+ "6050": 46130806784.0,
+ "6055": 46130806784.0,
+ "6060": 46130806784.0,
+ "6065": 46130806784.0,
+ "6070": 46130806784.0,
+ "6075": 46130806784.0,
+ "6080": 46130806784.0,
+ "6085": 46130806784.0,
+ "6090": 46132178944.0,
+ "6095": 46132178944.0,
+ "6100": 46132178944.0,
+ "6105": 46132178944.0,
+ "6110": 46132178944.0,
+ "6115": 46132178944.0,
+ "6120": 46132178944.0,
+ "6125": 46132178944.0,
+ "6130": 46132178944.0,
+ "6135": 46132178944.0,
+ "6140": 46132178944.0,
+ "6145": 46132178944.0,
+ "6150": 46132178944.0,
+ "6155": 46132178944.0,
+ "6160": 46132178944.0,
+ "6165": 46132178944.0,
+ "6170": 46132178944.0,
+ "6175": 46132178944.0,
+ "6180": 46132178944.0,
+ "6185": 46132178944.0,
+ "6190": 46132178944.0,
+ "6195": 46132178944.0,
+ "6200": 46132178944.0,
+ "6205": 46132178944.0,
+ "6210": 46132178944.0,
+ "6215": 46132178944.0,
+ "6220": 46132178944.0,
+ "6225": 46132178944.0,
+ "6230": 46132178944.0,
+ "6235": 46132178944.0,
+ "6240": 46132178944.0,
+ "6245": 46132178944.0,
+ "6250": 46132178944.0,
+ "6255": 46132178944.0,
+ "6260": 46164094976.0,
+ "6265": 46164094976.0,
+ "6270": 46164094976.0,
+ "6275": 46164094976.0,
+ "6280": 46164094976.0,
+ "6285": 46164094976.0,
+ "6290": 46164094976.0,
+ "6295": 46164094976.0,
+ "6300": 46164094976.0,
+ "6305": 46164094976.0,
+ "6310": 46164094976.0,
+ "6315": 46164094976.0,
+ "6320": 46201479168.0,
+ "6325": 46201479168.0,
+ "6330": 46201479168.0,
+ "6335": 46201479168.0,
+ "6340": 46201479168.0,
+ "6345": 46201479168.0,
+ "6350": 46201479168.0,
+ "6355": 46201479168.0,
+ "6360": 46201479168.0,
+ "6365": 46201479168.0,
+ "6370": 46201479168.0,
+ "6375": 46201479168.0,
+ "6380": 46201479168.0,
+ "6385": 46201479168.0,
+ "6390": 46201479168.0,
+ "6395": 46201479168.0,
+ "6400": 46201479168.0,
+ "6405": 46201479168.0,
+ "6410": 46201479168.0,
+ "6415": 46201479168.0,
+ "6420": 46201479168.0,
+ "6425": 46201479168.0,
+ "6430": 46201479168.0,
+ "6435": 46201479168.0,
+ "6440": 46201479168.0,
+ "6445": 46201479168.0,
+ "6450": 46201479168.0,
+ "6455": 46201479168.0,
+ "6460": 46201479168.0,
+ "6465": 46201479168.0,
+ "6470": 46201479168.0,
+ "6475": 46201479168.0,
+ "6480": 46201479168.0,
+ "6485": 46201479168.0,
+ "6490": 46201479168.0,
+ "6495": 46201479168.0,
+ "6500": 46201479168.0,
+ "6505": 46201479168.0,
+ "6510": 46201479168.0,
+ "6515": 46201479168.0,
+ "6520": 46201479168.0,
+ "6525": 46201479168.0,
+ "6530": 46201479168.0,
+ "6535": 46201479168.0,
+ "6540": 46201479168.0,
+ "6545": 46201479168.0,
+ "6550": 46201479168.0,
+ "6555": 46201479168.0,
+ "6560": 46201479168.0,
+ "6565": 46201479168.0,
+ "6570": 46201479168.0,
+ "6575": 46201479168.0,
+ "6580": 46201479168.0,
+ "6585": 46201479168.0,
+ "6590": 46201479168.0,
+ "6595": 46201479168.0,
+ "6600": 46201479168.0,
+ "6605": 46201479168.0,
+ "6610": 46201479168.0,
+ "6615": 46201479168.0,
+ "6620": 46201479168.0,
+ "6625": 46201479168.0,
+ "6630": 46201479168.0,
+ "6635": 46201479168.0,
+ "6640": 46201479168.0,
+ "6645": 46201479168.0,
+ "6650": 46201479168.0,
+ "6655": 46201479168.0,
+ "6660": 46201479168.0,
+ "6665": 46201479168.0,
+ "6670": 46201479168.0,
+ "6675": 46201479168.0,
+ "6680": 46201479168.0,
+ "6685": 46201479168.0,
+ "6690": 46201479168.0,
+ "6695": 46201479168.0,
+ "6700": 46201479168.0,
+ "6705": 46201479168.0,
+ "6710": 46201479168.0,
+ "6715": 46201479168.0,
+ "6720": 46201479168.0,
+ "6725": 46201479168.0,
+ "6730": 46201479168.0,
+ "6735": 46201479168.0,
+ "6740": 46201479168.0,
+ "6745": 46201479168.0,
+ "6750": 46201479168.0,
+ "6755": 46201479168.0,
+ "6760": 46201479168.0,
+ "6765": 46066221056.0,
+ "6770": 46066221056.0,
+ "6775": 46066221056.0,
+ "6780": 46066221056.0,
+ "6785": 46066221056.0,
+ "6790": 46066221056.0,
+ "6795": 46066221056.0,
+ "6800": 46066221056.0,
+ "6805": 46066221056.0,
+ "6810": 46066221056.0,
+ "6815": 46066221056.0,
+ "6820": 46066221056.0,
+ "6825": 46066221056.0,
+ "6830": 46066221056.0,
+ "6835": 46066221056.0,
+ "6840": 46066221056.0,
+ "6845": 46066221056.0,
+ "6850": 46066221056.0,
+ "6855": 46066221056.0,
+ "6860": 46066221056.0,
+ "6865": 46066221056.0,
+ "6870": 46066221056.0,
+ "6875": 46066221056.0,
+ "6880": 46066221056.0,
+ "6885": 46066221056.0,
+ "6890": 46066221056.0,
+ "6895": 46066221056.0,
+ "6900": 46066221056.0,
+ "6905": 46066221056.0,
+ "6910": 46066221056.0,
+ "6915": 46066221056.0,
+ "6920": 46066221056.0,
+ "6925": 46066221056.0,
+ "6930": 46066221056.0,
+ "6935": 46066221056.0,
+ "6940": 46066221056.0,
+ "6945": 46066221056.0,
+ "6950": 46066221056.0,
+ "6955": 46066221056.0,
+ "6960": 46077157376.0,
+ "6965": 46077157376.0,
+ "6970": 46077157376.0,
+ "6975": 46077157376.0,
+ "6980": 46077157376.0,
+ "6985": 46077157376.0,
+ "6990": 46077157376.0,
+ "6995": 46077157376.0,
+ "7000": 46077157376.0,
+ "7005": 46077157376.0,
+ "7010": 46077157376.0,
+ "7015": 46077157376.0,
+ "7020": 46077157376.0,
+ "7025": 46077157376.0,
+ "7030": 46077157376.0,
+ "7035": 46077157376.0,
+ "7040": 46077157376.0,
+ "7045": 46077157376.0,
+ "7050": 46077157376.0,
+ "7055": 46077157376.0,
+ "7060": 46077157376.0,
+ "7065": 46077157376.0,
+ "7070": 46077157376.0,
+ "7075": 46077157376.0,
+ "7080": 46077157376.0,
+ "7085": 46077157376.0,
+ "7090": 46077157376.0,
+ "7095": 46077157376.0,
+ "7100": 46077157376.0,
+ "7105": 46077157376.0,
+ "7110": 46077157376.0,
+ "7115": 46077157376.0,
+ "7120": 46077157376.0,
+ "7125": 46077157376.0,
+ "7130": 46077157376.0,
+ "7135": 46077157376.0,
+ "7140": 46077157376.0,
+ "7145": 46077157376.0,
+ "7150": 46077157376.0,
+ "7155": 46077157376.0,
+ "7160": 46077157376.0,
+ "7165": 46077157376.0,
+ "7170": 46077157376.0,
+ "7175": 46077157376.0,
+ "7180": 46098948096.0,
+ "7185": 46098948096.0,
+ "7190": 46098948096.0,
+ "7195": 46098948096.0,
+ "7200": 46098948096.0,
+ "7205": 46098948096.0,
+ "7210": 46098948096.0,
+ "7215": 46098948096.0,
+ "7220": 46098948096.0,
+ "7225": 46098948096.0,
+ "7230": 46098948096.0,
+ "7235": 46098948096.0,
+ "7240": 46098948096.0,
+ "7245": 46098948096.0,
+ "7250": 46098948096.0,
+ "7255": 46098948096.0,
+ "7260": 46098948096.0,
+ "7265": 46098948096.0,
+ "7270": 46098948096.0,
+ "7275": 46098948096.0,
+ "7280": 46098948096.0,
+ "7285": 46098948096.0,
+ "7290": 46098948096.0,
+ "7295": 46098948096.0,
+ "7300": 46165655552.0,
+ "7305": 46165655552.0,
+ "7310": 46165655552.0,
+ "7315": 46165655552.0,
+ "7320": 46165655552.0,
+ "7325": 46165655552.0,
+ "7330": 46165655552.0,
+ "7335": 46165655552.0,
+ "7340": 46165655552.0,
+ "7345": 46165655552.0,
+ "7350": 46165655552.0,
+ "7355": 46165655552.0,
+ "7360": 46165655552.0,
+ "7365": 46165655552.0,
+ "7370": 46165655552.0,
+ "7375": 46165655552.0,
+ "7380": 46165655552.0,
+ "7385": 46165655552.0,
+ "7390": 46165655552.0,
+ "7395": 46165655552.0,
+ "7400": 46165655552.0,
+ "7405": 46165655552.0,
+ "7410": 46165655552.0,
+ "7415": 46165655552.0,
+ "7420": 46165655552.0,
+ "7425": 46165655552.0,
+ "7430": 46165655552.0,
+ "7435": 46165655552.0,
+ "7440": 46165655552.0,
+ "7445": 46165655552.0,
+ "7450": 46165655552.0,
+ "7455": 46165655552.0,
+ "7460": 46165655552.0,
+ "7465": 46165655552.0,
+ "7470": 46165655552.0,
+ "7475": 46165655552.0,
+ "7480": 46165655552.0,
+ "7485": 46165655552.0,
+ "7490": 46165655552.0,
+ "7495": 46165655552.0,
+ "7500": 46165655552.0,
+ "7505": 46165655552.0,
+ "7510": 46165655552.0,
+ "7515": 46165655552.0,
+ "7520": 46165655552.0,
+ "7525": 46165655552.0,
+ "7530": 46165655552.0,
+ "7535": 46165655552.0,
+ "7540": 46165655552.0,
+ "7545": 46165655552.0,
+ "7550": 46165655552.0,
+ "7555": 46165655552.0,
+ "7560": 46165655552.0,
+ "7565": 46165655552.0,
+ "7570": 46165655552.0,
+ "7575": 46165655552.0,
+ "7580": 46165655552.0,
+ "7585": 46165655552.0,
+ "7590": 46165655552.0,
+ "7595": 46165655552.0,
+ "7600": 46165655552.0,
+ "7605": 46165655552.0,
+ "7610": 46165655552.0,
+ "7615": 46165655552.0,
+ "7620": 46165655552.0,
+ "7625": 46165655552.0,
+ "7630": 46165655552.0,
+ "7635": 46165655552.0,
+ "7640": 46165655552.0,
+ "7645": 46165655552.0,
+ "7650": 46165655552.0,
+ "7655": 46165655552.0,
+ "7660": 46165655552.0,
+ "7665": 46165655552.0,
+ "7670": 46165655552.0,
+ "7675": 46165655552.0,
+ "7680": 46165655552.0,
+ "7685": 46165655552.0,
+ "7690": 45955534848.0,
+ "7695": 45955534848.0,
+ "7700": 45955534848.0,
+ "7705": 45955534848.0,
+ "7710": 45955534848.0,
+ "7715": 45955534848.0,
+ "7720": 45955534848.0,
+ "7725": 45969104896.0,
+ "7730": 45969104896.0,
+ "7735": 45969104896.0,
+ "7740": 45969104896.0,
+ "7745": 45969104896.0,
+ "7750": 45969104896.0,
+ "7755": 45969104896.0,
+ "7760": 46011006976.0,
+ "7765": 46011006976.0,
+ "7770": 46011006976.0,
+ "7775": 46017036288.0,
+ "7780": 46017036288.0,
+ "7785": 46017036288.0,
+ "7790": 46017036288.0,
+ "7795": 46017036288.0,
+ "7800": 46017036288.0,
+ "7805": 46017036288.0,
+ "7810": 46069456896.0,
+ "7815": 46069456896.0,
+ "7820": 46069456896.0,
+ "7825": 46069456896.0,
+ "7830": 46069456896.0,
+ "7835": 46069456896.0,
+ "7840": 46069456896.0,
+ "7845": 46069456896.0,
+ "7850": 46210023424.0,
+ "7855": 46210023424.0,
+ "7860": 46210023424.0,
+ "7865": 46210023424.0,
+ "7870": 46210023424.0,
+ "7875": 46210023424.0,
+ "7880": 46210023424.0,
+ "7885": 46210023424.0,
+ "7890": 46210023424.0,
+ "7895": 46210023424.0,
+ "7900": 46210023424.0,
+ "7905": 46210023424.0,
+ "7910": 46210023424.0,
+ "7915": 46210023424.0,
+ "7920": 46210023424.0,
+ "7925": 46210023424.0,
+ "7930": 46210023424.0,
+ "7935": 46210023424.0,
+ "7940": 46210023424.0,
+ "7945": 46210023424.0,
+ "7950": 46210023424.0,
+ "7955": 46210023424.0,
+ "7960": 46210023424.0,
+ "7965": 46210023424.0,
+ "7970": 46210023424.0,
+ "7975": 46210023424.0,
+ "7980": 46210023424.0,
+ "7985": 46210023424.0,
+ "7990": 46210023424.0,
+ "7995": 46210023424.0,
+ "8000": 46210023424.0,
+ "8005": 46210023424.0,
+ "8010": 46210023424.0,
+ "8015": 46210023424.0,
+ "8020": 46210023424.0,
+ "8025": 46210023424.0,
+ "8030": 46210023424.0,
+ "8035": 46258020352.0,
+ "8040": 46258020352.0,
+ "8045": 46258020352.0,
+ "8050": 46258020352.0,
+ "8055": 46258020352.0,
+ "8060": 46258020352.0,
+ "8065": 46258020352.0,
+ "8070": 46258020352.0,
+ "8075": 46258020352.0,
+ "8080": 46258020352.0,
+ "8085": 46258020352.0,
+ "8090": 46258020352.0,
+ "8095": 46258020352.0,
+ "8100": 46258020352.0,
+ "8105": 46258020352.0,
+ "8110": 46258020352.0,
+ "8115": 46258020352.0,
+ "8120": 46258020352.0,
+ "8125": 46258020352.0,
+ "8130": 46258020352.0,
+ "8135": 46258020352.0,
+ "8140": 46258020352.0,
+ "8145": 46258020352.0,
+ "8150": 46258020352.0,
+ "8155": 46258020352.0,
+ "8160": 46258020352.0,
+ "8165": 46258020352.0,
+ "8170": 46258020352.0,
+ "8175": 46258020352.0,
+ "8180": 46258020352.0,
+ "8185": 46258020352.0,
+ "8190": 46258020352.0,
+ "8195": 46258020352.0,
+ "8200": 46258020352.0,
+ "8205": 46258020352.0,
+ "8210": 46258020352.0,
+ "8215": 46258020352.0,
+ "8220": 46258020352.0,
+ "8225": 46258020352.0,
+ "8230": 46258020352.0,
+ "8235": 46258020352.0,
+ "8240": 46258020352.0,
+ "8245": 46258020352.0,
+ "8250": 46258020352.0,
+ "8255": 46258020352.0,
+ "8260": 46258020352.0,
+ "8265": 46258020352.0,
+ "8270": 46258020352.0,
+ "8275": 46258020352.0,
+ "8280": 46258020352.0,
+ "8285": 46258020352.0,
+ "8290": 46258020352.0,
+ "8295": 46258020352.0,
+ "8300": 46258020352.0,
+ "8305": 46258020352.0,
+ "8310": 46258020352.0,
+ "8315": 46258020352.0,
+ "8320": 46258020352.0,
+ "8325": 46258020352.0,
+ "8330": 46258020352.0,
+ "8335": 46258020352.0,
+ "8340": 46258020352.0,
+ "8345": 46258020352.0,
+ "8350": 46258020352.0,
+ "8355": 46258020352.0,
+ "8360": 46258020352.0,
+ "8365": 46258020352.0,
+ "8370": 46258020352.0,
+ "8375": 46258020352.0,
+ "8380": 46258020352.0,
+ "8385": 46258020352.0,
+ "8390": 46258020352.0,
+ "8395": 46258020352.0,
+ "8400": 46258020352.0,
+ "8405": 46258020352.0,
+ "8410": 46258020352.0,
+ "8415": 46258020352.0,
+ "8420": 46258020352.0,
+ "8425": 46258020352.0,
+ "8430": 46258020352.0,
+ "8435": 46258020352.0,
+ "8440": 46258020352.0,
+ "8445": 46258020352.0,
+ "8450": 46258020352.0,
+ "8455": 46258020352.0,
+ "8460": 46258020352.0,
+ "8465": 46258020352.0,
+ "8470": 46258020352.0,
+ "8475": 46258020352.0,
+ "8480": 46258020352.0,
+ "8485": 46258020352.0,
+ "8490": 46258020352.0,
+ "8495": 46258020352.0,
+ "8500": 46258020352.0,
+ "8505": 46258020352.0,
+ "8510": 46258020352.0,
+ "8515": 46258020352.0,
+ "8520": 46258020352.0,
+ "8525": 46258020352.0,
+ "8530": 46258020352.0,
+ "8535": 46258020352.0,
+ "8540": 46258020352.0,
+ "8545": 46258020352.0,
+ "8550": 46258020352.0,
+ "8555": 46258020352.0,
+ "8560": 46258020352.0,
+ "8565": 46258020352.0,
+ "8570": 46258020352.0,
+ "8575": 46258020352.0,
+ "8580": 46258020352.0,
+ "8585": 46258020352.0,
+ "8590": 46258020352.0,
+ "8595": 46258020352.0,
+ "8600": 46258020352.0,
+ "8605": 46258020352.0,
+ "8610": 46258020352.0,
+ "8615": 46258020352.0,
+ "8620": 46258020352.0,
+ "8625": 46258020352.0,
+ "8630": 45904703488.0,
+ "8635": 45904703488.0,
+ "8640": 45928800256.0,
+ "8645": 45928800256.0,
+ "8650": 45928800256.0,
+ "8655": 46102806528.0,
+ "8660": 46102806528.0,
+ "8665": 46102806528.0,
+ "8670": 46102806528.0,
+ "8675": 46102806528.0,
+ "8680": 46102806528.0,
+ "8685": 46102806528.0,
+ "8690": 46102806528.0,
+ "8695": 46102806528.0,
+ "8700": 46102806528.0,
+ "8705": 46102806528.0,
+ "8710": 46102806528.0,
+ "8715": 46102806528.0,
+ "8720": 46102806528.0,
+ "8725": 46102806528.0,
+ "8730": 46102806528.0,
+ "8735": 46102806528.0,
+ "8740": 46150823936.0,
+ "8745": 46150823936.0,
+ "8750": 46150823936.0,
+ "8755": 46150823936.0,
+ "8760": 46150823936.0,
+ "8765": 46150823936.0,
+ "8770": 46150823936.0,
+ "8775": 46150823936.0,
+ "8780": 46150823936.0,
+ "8785": 46150823936.0,
+ "8790": 46150823936.0,
+ "8795": 46150823936.0,
+ "8800": 46150823936.0,
+ "8805": 46150823936.0,
+ "8810": 46150823936.0,
+ "8815": 46150823936.0,
+ "8820": 46150823936.0,
+ "8825": 46150823936.0,
+ "8830": 46150823936.0,
+ "8835": 46150823936.0,
+ "8840": 46150823936.0,
+ "8845": 46150823936.0,
+ "8850": 46150823936.0,
+ "8855": 46229737472.0,
+ "8860": 46229737472.0,
+ "8865": 46229737472.0,
+ "8870": 46229737472.0,
+ "8875": 46229737472.0,
+ "8880": 46229737472.0,
+ "8885": 46229737472.0,
+ "8890": 46229737472.0,
+ "8895": 46229737472.0,
+ "8900": 46229737472.0,
+ "8905": 46229737472.0,
+ "8910": 46229737472.0,
+ "8915": 46229737472.0,
+ "8920": 46229737472.0,
+ "8925": 46229737472.0,
+ "8930": 46229737472.0,
+ "8935": 46229737472.0,
+ "8940": 46229737472.0,
+ "8945": 46229737472.0,
+ "8950": 46229737472.0,
+ "8955": 46229737472.0,
+ "8960": 46229737472.0,
+ "8965": 46229737472.0,
+ "8970": 46229737472.0,
+ "8975": 46229737472.0,
+ "8980": 46229737472.0,
+ "8985": 46229737472.0,
+ "8990": 46229737472.0,
+ "8995": 46229737472.0,
+ "9000": 46229737472.0,
+ "9005": 46229737472.0,
+ "9010": 46229737472.0,
+ "9015": 46229737472.0,
+ "9020": 46229737472.0,
+ "9025": 46229737472.0,
+ "9030": 46229737472.0,
+ "9035": 46229737472.0,
+ "9040": 46229737472.0,
+ "9045": 46229737472.0,
+ "9050": 46229737472.0,
+ "9055": 46229737472.0,
+ "9060": 46229737472.0,
+ "9065": 46229737472.0,
+ "9070": 46229737472.0,
+ "9075": 46229737472.0,
+ "9080": 46229737472.0,
+ "9085": 46229737472.0,
+ "9090": 46229737472.0,
+ "9095": 46229737472.0,
+ "9100": 46229737472.0,
+ "9105": 46229737472.0,
+ "9110": 46229737472.0,
+ "9115": 46229737472.0,
+ "9120": 46229737472.0,
+ "9125": 46229737472.0,
+ "9130": 46229737472.0,
+ "9135": 46229737472.0,
+ "9140": 46229737472.0,
+ "9145": 46229737472.0,
+ "9150": 46229737472.0,
+ "9155": 46229737472.0,
+ "9160": 46229737472.0,
+ "9165": 46229737472.0,
+ "9170": 46229737472.0,
+ "9175": 46229737472.0,
+ "9180": 46229737472.0,
+ "9185": 46229737472.0,
+ "9190": 46229737472.0,
+ "9195": 46229737472.0,
+ "9200": 46229737472.0,
+ "9205": 46229737472.0,
+ "9210": 46229737472.0,
+ "9215": 46229737472.0,
+ "9220": 46229737472.0,
+ "9225": 46229737472.0,
+ "9230": 46229737472.0,
+ "9235": 46229737472.0,
+ "9240": 46229737472.0,
+ "9245": 46229737472.0,
+ "9250": 46229737472.0,
+ "9255": 46229737472.0,
+ "9260": 46229737472.0,
+ "9265": 46229737472.0,
+ "9270": 46229737472.0,
+ "9275": 46229737472.0,
+ "9280": 46229737472.0,
+ "9285": 46229737472.0,
+ "9290": 46229737472.0,
+ "9295": 46229737472.0,
+ "9300": 46229737472.0,
+ "9305": 46229737472.0,
+ "9310": 46229737472.0,
+ "9315": 46229737472.0,
+ "9320": 46229737472.0,
+ "9325": 46229737472.0,
+ "9330": 46229737472.0,
+ "9335": 46229737472.0,
+ "9340": 46229737472.0,
+ "9345": 46229737472.0,
+ "9350": 46229737472.0,
+ "9355": 46229737472.0,
+ "9360": 46229737472.0,
+ "9365": 46229737472.0,
+ "9370": 46229737472.0,
+ "9375": 46229737472.0,
+ "9380": 46229737472.0,
+ "9385": 46229737472.0,
+ "9390": 46229737472.0,
+ "9395": 46229737472.0,
+ "9400": 46229737472.0,
+ "9405": 46229737472.0,
+ "9410": 46229737472.0,
+ "9415": 46229737472.0,
+ "9420": 46229737472.0,
+ "9425": 46229737472.0,
+ "9430": 46229737472.0,
+ "9435": 46229737472.0,
+ "9440": 46229737472.0,
+ "9445": 46229737472.0,
+ "9450": 46229737472.0,
+ "9455": 46229737472.0,
+ "9460": 46229737472.0,
+ "9465": 46229737472.0,
+ "9470": 46229737472.0,
+ "9475": 46229737472.0,
+ "9480": 46229737472.0,
+ "9485": 46229737472.0,
+ "9490": 46229737472.0,
+ "9495": 46229737472.0,
+ "9500": 46229737472.0,
+ "9505": 46229737472.0,
+ "9510": 46229737472.0,
+ "9515": 46229737472.0,
+ "9520": 46229737472.0,
+ "9525": 46229737472.0,
+ "9530": 46229737472.0,
+ "9535": 46229737472.0
+ }
+ },
+ "mtp_1 loss": {
+ "start_step": 1,
+ "end_step": 9535,
+ "step_interval": 5,
+ "values": {
+ "1": 13.88881,
+ "5": 13.88985,
+ "10": 13.88765,
+ "15": 13.88579,
+ "20": 13.88057,
+ "25": 13.87774,
+ "30": 13.85561,
+ "35": 13.84817,
+ "40": 13.84578,
+ "45": 13.82667,
+ "50": 13.74785,
+ "55": 13.72461,
+ "60": 13.70779,
+ "65": 13.67525,
+ "70": 13.63787,
+ "75": 13.43969,
+ "80": 13.35818,
+ "85": 13.28244,
+ "90": 13.18556,
+ "95": 13.04883,
+ "100": 12.90428,
+ "105": 12.74282,
+ "110": 12.47822,
+ "115": 12.2616,
+ "120": 12.03661,
+ "125": 11.86422,
+ "130": 11.7425,
+ "135": 11.57804,
+ "140": 11.34393,
+ "145": 11.26548,
+ "150": 11.11543,
+ "155": 11.01787,
+ "160": 10.87847,
+ "165": 10.74891,
+ "170": 10.6548,
+ "175": 10.59386,
+ "180": 10.43396,
+ "185": 10.42319,
+ "190": 10.27104,
+ "195": 10.25345,
+ "200": 10.12726,
+ "205": 9.9752,
+ "210": 9.94299,
+ "215": 9.92138,
+ "220": 9.78964,
+ "225": 9.77027,
+ "230": 9.73008,
+ "235": 9.64394,
+ "240": 9.57407,
+ "245": 9.50526,
+ "250": 9.43752,
+ "255": 9.37006,
+ "260": 9.29488,
+ "265": 9.23872,
+ "270": 9.15298,
+ "275": 9.12461,
+ "280": 9.10074,
+ "285": 9.09326,
+ "290": 9.00513,
+ "295": 8.94309,
+ "300": 8.83007,
+ "305": 8.80927,
+ "310": 8.75431,
+ "315": 8.73651,
+ "320": 8.70665,
+ "325": 8.60948,
+ "330": 8.58293,
+ "335": 8.5501,
+ "340": 8.54366,
+ "345": 8.4229,
+ "350": 8.40498,
+ "355": 8.29483,
+ "360": 8.37588,
+ "365": 8.27699,
+ "370": 8.26798,
+ "375": 8.20905,
+ "380": 8.16655,
+ "385": 8.15446,
+ "390": 8.13176,
+ "395": 8.08497,
+ "400": 8.00327,
+ "405": 8.00116,
+ "410": 7.99128,
+ "415": 7.93586,
+ "420": 7.91575,
+ "425": 7.87073,
+ "430": 7.80319,
+ "435": 7.81577,
+ "440": 7.76186,
+ "445": 7.74268,
+ "450": 7.67572,
+ "455": 7.70441,
+ "460": 7.65227,
+ "465": 7.63086,
+ "470": 7.5946,
+ "475": 7.60547,
+ "480": 7.4743,
+ "485": 7.51895,
+ "490": 7.47355,
+ "495": 7.45627,
+ "500": 7.39696,
+ "505": 7.40474,
+ "510": 7.37407,
+ "515": 7.34655,
+ "520": 7.34312,
+ "525": 7.31894,
+ "530": 7.32072,
+ "535": 7.29865,
+ "540": 7.21292,
+ "545": 7.23511,
+ "550": 7.27002,
+ "555": 7.29692,
+ "560": 7.23534,
+ "565": 7.15998,
+ "570": 7.1702,
+ "575": 7.18855,
+ "580": 7.11604,
+ "585": 7.12229,
+ "590": 7.06551,
+ "595": 7.04779,
+ "600": 7.07157,
+ "605": 7.06594,
+ "610": 7.02376,
+ "615": 7.08218,
+ "620": 6.98418,
+ "625": 6.9579,
+ "630": 6.96022,
+ "635": 6.98637,
+ "640": 6.96266,
+ "645": 6.95187,
+ "650": 6.99866,
+ "655": 6.99453,
+ "660": 6.89185,
+ "665": 6.87544,
+ "670": 6.84232,
+ "675": 6.93253,
+ "680": 6.89066,
+ "685": 6.8512,
+ "690": 6.82864,
+ "695": 6.79142,
+ "700": 6.78596,
+ "705": 6.78021,
+ "710": 6.8174,
+ "715": 6.82204,
+ "720": 6.70691,
+ "725": 6.76316,
+ "730": 6.75157,
+ "735": 6.75195,
+ "740": 6.69673,
+ "745": 6.671,
+ "750": 6.73055,
+ "755": 6.65211,
+ "760": 6.65962,
+ "765": 6.65467,
+ "770": 6.6763,
+ "775": 6.64878,
+ "780": 6.61806,
+ "785": 6.6361,
+ "790": 6.58626,
+ "795": 6.59124,
+ "800": 6.58047,
+ "805": 6.64583,
+ "810": 6.51238,
+ "815": 6.53227,
+ "820": 6.5419,
+ "825": 6.55019,
+ "830": 6.56083,
+ "835": 6.51749,
+ "840": 6.48209,
+ "845": 6.5362,
+ "850": 6.49072,
+ "855": 6.48642,
+ "860": 6.48183,
+ "865": 6.48916,
+ "870": 6.45279,
+ "875": 6.50126,
+ "880": 6.46483,
+ "885": 6.43159,
+ "890": 6.50552,
+ "895": 6.39068,
+ "900": 6.41561,
+ "905": 6.44068,
+ "910": 6.4035,
+ "915": 6.38728,
+ "920": 6.38471,
+ "925": 6.36897,
+ "930": 6.40586,
+ "935": 6.39539,
+ "940": 6.34001,
+ "945": 6.36765,
+ "950": 6.3934,
+ "955": 6.34654,
+ "960": 6.35174,
+ "965": 6.24978,
+ "970": 6.31969,
+ "975": 6.30887,
+ "980": 6.28537,
+ "985": 6.28891,
+ "990": 6.34209,
+ "995": 6.26021,
+ "1000": 6.28008,
+ "1005": 6.23034,
+ "1010": 6.2651,
+ "1015": 6.29088,
+ "1020": 6.20383,
+ "1025": 6.2114,
+ "1030": 6.20941,
+ "1035": 6.30082,
+ "1040": 6.22544,
+ "1045": 6.20031,
+ "1050": 6.23097,
+ "1055": 6.2175,
+ "1060": 6.16961,
+ "1065": 6.16208,
+ "1070": 6.19484,
+ "1075": 6.19348,
+ "1080": 6.19639,
+ "1085": 6.19943,
+ "1090": 6.18347,
+ "1095": 6.18861,
+ "1100": 6.14283,
+ "1105": 6.11743,
+ "1110": 6.18172,
+ "1115": 6.11487,
+ "1120": 6.05693,
+ "1125": 6.09096,
+ "1130": 6.147,
+ "1135": 6.10043,
+ "1140": 6.08498,
+ "1145": 6.07135,
+ "1150": 6.0971,
+ "1155": 6.06654,
+ "1160": 6.05067,
+ "1165": 6.1004,
+ "1170": 6.07856,
+ "1175": 6.05007,
+ "1180": 6.05198,
+ "1185": 6.04052,
+ "1190": 6.05309,
+ "1195": 6.03041,
+ "1200": 5.97654,
+ "1205": 6.0788,
+ "1210": 5.94163,
+ "1215": 5.98501,
+ "1220": 6.06348,
+ "1225": 5.9526,
+ "1230": 5.99809,
+ "1235": 5.95666,
+ "1240": 5.99505,
+ "1245": 5.9703,
+ "1250": 5.95099,
+ "1255": 5.94679,
+ "1260": 5.95051,
+ "1265": 5.93146,
+ "1270": 5.90566,
+ "1275": 5.96943,
+ "1280": 5.90459,
+ "1285": 5.92249,
+ "1290": 5.90451,
+ "1295": 5.91772,
+ "1300": 5.93323,
+ "1305": 5.89904,
+ "1310": 5.83941,
+ "1315": 5.89842,
+ "1320": 5.89752,
+ "1325": 5.82579,
+ "1330": 5.88262,
+ "1335": 5.84933,
+ "1340": 5.91914,
+ "1345": 5.86491,
+ "1350": 5.84254,
+ "1355": 5.84304,
+ "1360": 5.84983,
+ "1365": 5.84149,
+ "1370": 5.79606,
+ "1375": 5.80338,
+ "1380": 5.8591,
+ "1385": 5.81587,
+ "1390": 5.81275,
+ "1395": 5.82669,
+ "1400": 5.82409,
+ "1405": 5.82815,
+ "1410": 5.78563,
+ "1415": 5.76877,
+ "1420": 5.80637,
+ "1425": 5.79355,
+ "1430": 5.8324,
+ "1435": 5.74599,
+ "1440": 5.76252,
+ "1445": 5.80898,
+ "1450": 5.78979,
+ "1455": 5.79815,
+ "1460": 5.75894,
+ "1465": 5.76322,
+ "1470": 5.79824,
+ "1475": 5.77043,
+ "1480": 5.77696,
+ "1485": 5.72273,
+ "1490": 5.72167,
+ "1495": 5.74518,
+ "1500": 5.75551,
+ "1505": 5.72121,
+ "1510": 5.74463,
+ "1515": 5.67413,
+ "1520": 5.70226,
+ "1525": 5.67371,
+ "1530": 5.69445,
+ "1535": 5.68746,
+ "1540": 5.67176,
+ "1545": 5.71689,
+ "1550": 5.7232,
+ "1555": 5.71053,
+ "1560": 5.64939,
+ "1565": 5.7014,
+ "1570": 5.71027,
+ "1575": 5.65707,
+ "1580": 5.69605,
+ "1585": 5.66738,
+ "1590": 5.655,
+ "1595": 5.63646,
+ "1600": 5.70903,
+ "1605": 5.64335,
+ "1610": 5.64359,
+ "1615": 5.6313,
+ "1620": 5.65534,
+ "1625": 5.64707,
+ "1630": 5.63085,
+ "1635": 5.68076,
+ "1640": 5.62668,
+ "1645": 5.64857,
+ "1650": 5.63708,
+ "1655": 5.61941,
+ "1660": 5.61498,
+ "1665": 5.60284,
+ "1670": 5.61196,
+ "1675": 5.61874,
+ "1680": 5.56074,
+ "1685": 5.56911,
+ "1690": 5.54865,
+ "1695": 5.55394,
+ "1700": 5.59977,
+ "1705": 5.5754,
+ "1710": 5.58511,
+ "1715": 5.54483,
+ "1720": 5.52427,
+ "1725": 5.57339,
+ "1730": 5.53337,
+ "1735": 5.57946,
+ "1740": 5.52207,
+ "1745": 5.55945,
+ "1750": 5.53548,
+ "1755": 5.531,
+ "1760": 5.54869,
+ "1765": 5.51133,
+ "1770": 5.51982,
+ "1775": 5.52022,
+ "1780": 5.53604,
+ "1785": 5.48729,
+ "1790": 5.51977,
+ "1795": 5.52471,
+ "1800": 5.46849,
+ "1805": 5.46528,
+ "1810": 5.47702,
+ "1815": 5.48659,
+ "1820": 5.48103,
+ "1825": 5.48256,
+ "1830": 5.46513,
+ "1835": 5.46192,
+ "1840": 5.46148,
+ "1845": 5.45058,
+ "1850": 5.42745,
+ "1855": 5.48267,
+ "1860": 5.43048,
+ "1865": 5.44094,
+ "1870": 5.43357,
+ "1875": 5.42317,
+ "1880": 5.48531,
+ "1885": 5.44623,
+ "1890": 5.44272,
+ "1895": 5.38335,
+ "1900": 5.42282,
+ "1905": 5.41037,
+ "1910": 5.43617,
+ "1915": 5.39603,
+ "1920": 5.37396,
+ "1925": 5.40489,
+ "1930": 5.37663,
+ "1935": 5.39784,
+ "1940": 5.3731,
+ "1945": 5.41557,
+ "1950": 5.45885,
+ "1955": 5.38913,
+ "1960": 5.38985,
+ "1965": 5.33886,
+ "1970": 5.33861,
+ "1975": 5.39899,
+ "1980": 5.34976,
+ "1985": 5.36974,
+ "1990": 5.39683,
+ "1995": 5.37069,
+ "2000": 5.37855,
+ "2005": 5.42901,
+ "2010": 5.32903,
+ "2015": 5.31726,
+ "2020": 5.32763,
+ "2025": 5.37559,
+ "2030": 5.31233,
+ "2035": 5.32451,
+ "2040": 5.29036,
+ "2045": 5.38194,
+ "2050": 5.35495,
+ "2055": 5.32906,
+ "2060": 5.3288,
+ "2065": 5.29487,
+ "2070": 5.29576,
+ "2075": 5.32825,
+ "2080": 5.30108,
+ "2085": 5.32175,
+ "2090": 5.24852,
+ "2095": 5.29492,
+ "2100": 5.25539,
+ "2105": 5.28298,
+ "2110": 5.28075,
+ "2115": 5.27648,
+ "2120": 5.27804,
+ "2125": 5.24556,
+ "2130": 5.25265,
+ "2135": 5.25098,
+ "2140": 5.25976,
+ "2145": 5.22187,
+ "2150": 5.24403,
+ "2155": 5.22485,
+ "2160": 5.23975,
+ "2165": 5.22686,
+ "2170": 5.2611,
+ "2175": 5.2585,
+ "2180": 5.23878,
+ "2185": 5.24366,
+ "2190": 5.22645,
+ "2195": 5.20061,
+ "2200": 5.20157,
+ "2205": 5.20896,
+ "2210": 5.25673,
+ "2215": 5.29766,
+ "2220": 5.24008,
+ "2225": 5.21781,
+ "2230": 5.21614,
+ "2235": 5.25425,
+ "2240": 5.15757,
+ "2245": 5.15649,
+ "2250": 5.18182,
+ "2255": 5.19309,
+ "2260": 5.12987,
+ "2265": 5.20869,
+ "2270": 5.13894,
+ "2275": 5.19503,
+ "2280": 5.1657,
+ "2285": 5.18099,
+ "2290": 5.17496,
+ "2295": 5.17637,
+ "2300": 5.17296,
+ "2305": 5.15399,
+ "2310": 5.17858,
+ "2315": 5.12023,
+ "2320": 5.1675,
+ "2325": 5.14869,
+ "2330": 5.14587,
+ "2335": 5.12746,
+ "2340": 5.14442,
+ "2345": 5.18489,
+ "2350": 5.12968,
+ "2355": 5.11548,
+ "2360": 5.09777,
+ "2365": 5.11272,
+ "2370": 5.10463,
+ "2375": 5.10734,
+ "2380": 5.04983,
+ "2385": 5.09468,
+ "2390": 5.118,
+ "2395": 5.12494,
+ "2400": 5.07809,
+ "2405": 5.06257,
+ "2410": 5.11844,
+ "2415": 5.08989,
+ "2420": 5.10807,
+ "2425": 5.06069,
+ "2430": 5.08956,
+ "2435": 5.08769,
+ "2440": 5.07184,
+ "2445": 5.0834,
+ "2450": 5.04743,
+ "2455": 5.09162,
+ "2460": 5.04708,
+ "2465": 5.08715,
+ "2470": 5.07823,
+ "2475": 5.10777,
+ "2480": 5.03,
+ "2485": 5.05462,
+ "2490": 5.04864,
+ "2495": 5.03799,
+ "2500": 5.03538,
+ "2505": 5.05093,
+ "2510": 5.09103,
+ "2515": 5.08387,
+ "2520": 5.02231,
+ "2525": 5.0377,
+ "2530": 5.04602,
+ "2535": 5.0379,
+ "2540": 5.0443,
+ "2545": 5.04963,
+ "2550": 4.99279,
+ "2555": 5.06278,
+ "2560": 5.02935,
+ "2565": 5.00224,
+ "2570": 5.02538,
+ "2575": 4.98921,
+ "2580": 5.00238,
+ "2585": 4.98277,
+ "2590": 5.00299,
+ "2595": 4.95778,
+ "2600": 4.9913,
+ "2605": 5.01218,
+ "2610": 5.00594,
+ "2615": 4.97407,
+ "2620": 4.95268,
+ "2625": 4.98923,
+ "2630": 4.92271,
+ "2635": 5.00286,
+ "2640": 4.99771,
+ "2645": 4.95593,
+ "2650": 4.98035,
+ "2655": 4.96785,
+ "2660": 4.91302,
+ "2665": 5.01091,
+ "2670": 4.95374,
+ "2675": 4.91836,
+ "2680": 4.96057,
+ "2685": 4.95513,
+ "2690": 4.9203,
+ "2695": 4.99573,
+ "2700": 4.90485,
+ "2705": 4.91571,
+ "2710": 4.96842,
+ "2715": 4.93549,
+ "2720": 4.96604,
+ "2725": 4.91764,
+ "2730": 4.94323,
+ "2735": 4.93516,
+ "2740": 4.92349,
+ "2745": 4.89701,
+ "2750": 4.9369,
+ "2755": 4.94222,
+ "2760": 4.94227,
+ "2765": 4.91351,
+ "2770": 4.94731,
+ "2775": 4.90327,
+ "2780": 4.93471,
+ "2785": 4.91344,
+ "2790": 4.93715,
+ "2795": 4.90182,
+ "2800": 4.8421,
+ "2805": 4.89106,
+ "2810": 4.88195,
+ "2815": 4.89191,
+ "2820": 4.93196,
+ "2825": 4.92185,
+ "2830": 4.90026,
+ "2835": 4.90197,
+ "2840": 4.89911,
+ "2845": 4.8706,
+ "2850": 4.9029,
+ "2855": 4.84,
+ "2860": 4.88851,
+ "2865": 4.90078,
+ "2870": 4.88936,
+ "2875": 4.91166,
+ "2880": 4.82877,
+ "2885": 4.87034,
+ "2890": 4.84699,
+ "2895": 4.88851,
+ "2900": 4.84144,
+ "2905": 4.8499,
+ "2910": 4.84963,
+ "2915": 4.89435,
+ "2920": 4.8717,
+ "2925": 4.84301,
+ "2930": 4.83224,
+ "2935": 4.83438,
+ "2940": 4.83968,
+ "2945": 4.80255,
+ "2950": 4.79065,
+ "2955": 4.79249,
+ "2960": 4.81151,
+ "2965": 4.82408,
+ "2970": 4.81906,
+ "2975": 4.8368,
+ "2980": 4.78407,
+ "2985": 4.82925,
+ "2990": 4.84666,
+ "2995": 4.79571,
+ "3000": 4.80418,
+ "3005": 4.78875,
+ "3010": 4.81326,
+ "3015": 4.77863,
+ "3020": 4.79275,
+ "3025": 4.80781,
+ "3030": 4.81223,
+ "3035": 4.82187,
+ "3040": 4.8284,
+ "3045": 4.80344,
+ "3050": 4.78983,
+ "3055": 4.794,
+ "3060": 4.77496,
+ "3065": 4.80096,
+ "3070": 4.81115,
+ "3075": 4.7545,
+ "3080": 4.78662,
+ "3085": 4.77768,
+ "3090": 4.76239,
+ "3095": 4.81347,
+ "3100": 4.79886,
+ "3105": 4.78113,
+ "3110": 4.76179,
+ "3115": 4.71543,
+ "3120": 4.78154,
+ "3125": 4.74072,
+ "3130": 4.75366,
+ "3135": 4.75308,
+ "3140": 4.73163,
+ "3145": 4.7119,
+ "3150": 4.74836,
+ "3155": 4.77689,
+ "3160": 4.76443,
+ "3165": 4.75771,
+ "3170": 4.75051,
+ "3175": 4.7437,
+ "3180": 4.73244,
+ "3185": 4.70374,
+ "3190": 4.70935,
+ "3195": 4.7056,
+ "3200": 4.68382,
+ "3205": 4.72325,
+ "3210": 4.67841,
+ "3215": 4.70872,
+ "3220": 4.6777,
+ "3225": 4.71223,
+ "3230": 4.69872,
+ "3235": 4.73435,
+ "3240": 4.68748,
+ "3245": 4.69797,
+ "3250": 4.64064,
+ "3255": 4.6944,
+ "3260": 4.67179,
+ "3265": 4.72298,
+ "3270": 4.71137,
+ "3275": 4.65681,
+ "3280": 4.68769,
+ "3285": 4.69629,
+ "3290": 4.66826,
+ "3295": 4.66629,
+ "3300": 4.66577,
+ "3305": 4.67102,
+ "3310": 4.65951,
+ "3315": 4.70634,
+ "3320": 4.64754,
+ "3325": 4.65487,
+ "3330": 4.64085,
+ "3335": 4.65341,
+ "3340": 4.62737,
+ "3345": 4.63956,
+ "3350": 4.64918,
+ "3355": 4.66093,
+ "3360": 4.64847,
+ "3365": 4.66553,
+ "3370": 4.63962,
+ "3375": 4.67416,
+ "3380": 4.6174,
+ "3385": 4.62811,
+ "3390": 4.60223,
+ "3395": 4.69393,
+ "3400": 4.64146,
+ "3405": 4.66544,
+ "3410": 4.60434,
+ "3415": 4.55343,
+ "3420": 4.61484,
+ "3425": 4.6316,
+ "3430": 4.66856,
+ "3435": 4.63031,
+ "3440": 4.65098,
+ "3445": 4.60093,
+ "3450": 4.59823,
+ "3455": 4.62086,
+ "3460": 4.58117,
+ "3465": 4.58009,
+ "3470": 4.5935,
+ "3475": 4.59839,
+ "3480": 4.59444,
+ "3485": 4.62375,
+ "3490": 4.60538,
+ "3495": 4.63163,
+ "3500": 4.58956,
+ "3505": 4.59911,
+ "3510": 4.60211,
+ "3515": 4.6373,
+ "3520": 4.6223,
+ "3525": 4.57194,
+ "3530": 4.58447,
+ "3535": 4.58052,
+ "3540": 4.63526,
+ "3545": 4.56162,
+ "3550": 4.62022,
+ "3555": 4.55536,
+ "3560": 4.62345,
+ "3565": 4.55198,
+ "3570": 4.56825,
+ "3575": 4.5333,
+ "3580": 4.59998,
+ "3585": 4.57916,
+ "3590": 4.51763,
+ "3595": 4.58526,
+ "3600": 4.54955,
+ "3605": 4.53621,
+ "3610": 4.53932,
+ "3615": 4.57349,
+ "3620": 4.62077,
+ "3625": 4.54957,
+ "3630": 4.5983,
+ "3635": 4.5056,
+ "3640": 4.52392,
+ "3645": 4.56749,
+ "3650": 4.52842,
+ "3655": 4.5432,
+ "3660": 4.55423,
+ "3665": 4.58993,
+ "3670": 4.53733,
+ "3675": 4.55275,
+ "3680": 4.57307,
+ "3685": 4.49269,
+ "3690": 4.54436,
+ "3695": 4.49572,
+ "3700": 4.53188,
+ "3705": 4.50833,
+ "3710": 4.51831,
+ "3715": 4.52952,
+ "3720": 4.5027,
+ "3725": 4.47826,
+ "3730": 4.48705,
+ "3735": 4.50217,
+ "3740": 4.50255,
+ "3745": 4.48063,
+ "3750": 4.51539,
+ "3755": 4.4847,
+ "3760": 4.49698,
+ "3765": 4.47767,
+ "3770": 4.49033,
+ "3775": 4.46754,
+ "3780": 4.45607,
+ "3785": 4.50868,
+ "3790": 4.42239,
+ "3795": 4.48447,
+ "3800": 4.46265,
+ "3805": 4.45777,
+ "3810": 4.42446,
+ "3815": 4.47891,
+ "3820": 4.4711,
+ "3825": 4.4794,
+ "3830": 4.46542,
+ "3835": 4.42327,
+ "3840": 4.52101,
+ "3845": 4.47664,
+ "3850": 4.41912,
+ "3855": 4.46206,
+ "3860": 4.48067,
+ "3865": 4.44245,
+ "3870": 4.50319,
+ "3875": 4.40911,
+ "3880": 4.42327,
+ "3885": 4.44734,
+ "3890": 4.43689,
+ "3895": 4.3807,
+ "3900": 4.4344,
+ "3905": 4.4155,
+ "3910": 4.42761,
+ "3915": 4.41966,
+ "3920": 4.41063,
+ "3925": 4.39671,
+ "3930": 4.40115,
+ "3935": 4.41882,
+ "3940": 4.41339,
+ "3945": 4.3936,
+ "3950": 4.45727,
+ "3955": 4.39291,
+ "3960": 4.44111,
+ "3965": 4.44754,
+ "3970": 4.39262,
+ "3975": 4.40418,
+ "3980": 4.36848,
+ "3985": 4.40751,
+ "3990": 4.40215,
+ "3995": 4.4478,
+ "4000": 4.38178,
+ "4005": 4.37125,
+ "4010": 4.40956,
+ "4015": 4.3998,
+ "4020": 4.44034,
+ "4025": 4.39402,
+ "4030": 4.44956,
+ "4035": 4.40761,
+ "4040": 4.4329,
+ "4045": 4.41232,
+ "4050": 4.4072,
+ "4055": 4.41557,
+ "4060": 4.41167,
+ "4065": 4.41325,
+ "4070": 4.34763,
+ "4075": 4.37513,
+ "4080": 4.35445,
+ "4085": 4.39641,
+ "4090": 4.37589,
+ "4095": 4.36092,
+ "4100": 4.37416,
+ "4105": 4.35951,
+ "4110": 4.32439,
+ "4115": 4.39614,
+ "4120": 4.3138,
+ "4125": 4.30638,
+ "4130": 4.39045,
+ "4135": 4.37395,
+ "4140": 4.3161,
+ "4145": 4.32541,
+ "4150": 4.37326,
+ "4155": 4.29739,
+ "4160": 4.35319,
+ "4165": 4.38293,
+ "4170": 4.32453,
+ "4175": 4.33169,
+ "4180": 4.32695,
+ "4185": 4.31956,
+ "4190": 4.3132,
+ "4195": 4.3171,
+ "4200": 4.31427,
+ "4205": 4.37115,
+ "4210": 4.32891,
+ "4215": 4.3539,
+ "4220": 4.33495,
+ "4225": 4.32499,
+ "4230": 4.30745,
+ "4235": 4.34729,
+ "4240": 4.30622,
+ "4245": 4.31491,
+ "4250": 4.30376,
+ "4255": 4.31573,
+ "4260": 4.28865,
+ "4265": 4.30517,
+ "4270": 4.29854,
+ "4275": 4.35854,
+ "4280": 4.29063,
+ "4285": 4.33568,
+ "4290": 4.27663,
+ "4295": 4.30488,
+ "4300": 4.32731,
+ "4305": 4.29109,
+ "4310": 4.33283,
+ "4315": 4.31616,
+ "4320": 4.30655,
+ "4325": 4.32825,
+ "4330": 4.2627,
+ "4335": 4.29931,
+ "4340": 4.28763,
+ "4345": 4.24037,
+ "4350": 4.25745,
+ "4355": 4.3322,
+ "4360": 4.30674,
+ "4365": 4.3049,
+ "4370": 4.27583,
+ "4375": 4.24247,
+ "4380": 4.25457,
+ "4385": 4.23393,
+ "4390": 4.3111,
+ "4395": 4.27514,
+ "4400": 4.25937,
+ "4405": 4.23076,
+ "4410": 4.28222,
+ "4415": 4.26269,
+ "4420": 4.24912,
+ "4425": 4.29029,
+ "4430": 4.24486,
+ "4435": 4.28695,
+ "4440": 4.28667,
+ "4445": 4.24739,
+ "4450": 4.20548,
+ "4455": 4.26035,
+ "4460": 4.23919,
+ "4465": 4.25318,
+ "4470": 4.24137,
+ "4475": 4.26638,
+ "4480": 4.25173,
+ "4485": 4.23339,
+ "4490": 4.23711,
+ "4495": 4.18184,
+ "4500": 4.24985,
+ "4505": 4.22998,
+ "4510": 4.23633,
+ "4515": 4.1939,
+ "4520": 4.23042,
+ "4525": 4.19755,
+ "4530": 4.24097,
+ "4535": 4.20381,
+ "4540": 4.21114,
+ "4545": 4.23599,
+ "4550": 4.27385,
+ "4555": 4.20575,
+ "4560": 4.22099,
+ "4565": 4.15465,
+ "4570": 4.21337,
+ "4575": 4.19217,
+ "4580": 4.2546,
+ "4585": 4.22009,
+ "4590": 4.21305,
+ "4595": 4.17157,
+ "4600": 4.16796,
+ "4605": 4.20216,
+ "4610": 4.20042,
+ "4615": 4.24431,
+ "4620": 4.15933,
+ "4625": 4.19274,
+ "4630": 4.20291,
+ "4635": 4.18496,
+ "4640": 4.2096,
+ "4645": 4.20897,
+ "4650": 4.23074,
+ "4655": 4.19586,
+ "4660": 4.18451,
+ "4665": 4.19589,
+ "4670": 4.23762,
+ "4675": 4.17778,
+ "4680": 4.20992,
+ "4685": 4.19622,
+ "4690": 4.1732,
+ "4695": 4.18711,
+ "4700": 4.16528,
+ "4705": 4.1425,
+ "4710": 4.20645,
+ "4715": 4.18608,
+ "4720": 4.15512,
+ "4725": 4.11927,
+ "4730": 4.18207,
+ "4735": 4.1053,
+ "4740": 4.14418,
+ "4745": 4.18257,
+ "4750": 4.13485,
+ "4755": 4.1937,
+ "4760": 4.19947,
+ "4765": 4.15145,
+ "4770": 4.14734,
+ "4775": 4.14872,
+ "4780": 4.15307,
+ "4785": 4.13998,
+ "4790": 4.19182,
+ "4795": 4.17545,
+ "4800": 4.13507,
+ "4805": 4.17991,
+ "4810": 4.13748,
+ "4815": 4.17143,
+ "4820": 4.11892,
+ "4825": 4.16996,
+ "4830": 4.16996,
+ "4835": 4.14969,
+ "4840": 4.15388,
+ "4845": 4.11222,
+ "4850": 4.17218,
+ "4855": 4.17617,
+ "4860": 4.11456,
+ "4865": 4.13993,
+ "4870": 4.13265,
+ "4875": 4.17534,
+ "4880": 4.17303,
+ "4885": 4.13273,
+ "4890": 4.12446,
+ "4895": 4.12256,
+ "4900": 4.10032,
+ "4905": 4.09119,
+ "4910": 4.09185,
+ "4915": 4.14664,
+ "4920": 4.12247,
+ "4925": 4.08907,
+ "4930": 4.09769,
+ "4935": 4.12034,
+ "4940": 4.04939,
+ "4945": 4.1366,
+ "4950": 4.08034,
+ "4955": 4.15818,
+ "4960": 4.11687,
+ "4965": 4.11673,
+ "4970": 4.10007,
+ "4975": 4.11705,
+ "4980": 4.12196,
+ "4985": 4.12815,
+ "4990": 4.09022,
+ "4995": 4.13107,
+ "5000": 4.05531,
+ "5005": 4.11813,
+ "5010": 4.10781,
+ "5015": 4.07534,
+ "5020": 4.05535,
+ "5025": 4.06148,
+ "5030": 4.09765,
+ "5035": 4.0827,
+ "5040": 4.04215,
+ "5045": 4.11083,
+ "5050": 4.06495,
+ "5055": 4.09162,
+ "5060": 4.03415,
+ "5065": 4.09674,
+ "5070": 4.06975,
+ "5075": 4.12349,
+ "5080": 4.07784,
+ "5085": 4.09645,
+ "5090": 4.08017,
+ "5095": 4.04491,
+ "5100": 4.07947,
+ "5105": 4.08015,
+ "5110": 4.08831,
+ "5115": 4.07593,
+ "5120": 4.09636,
+ "5125": 4.06245,
+ "5130": 4.06407,
+ "5135": 4.05206,
+ "5140": 4.06962,
+ "5145": 4.06108,
+ "5150": 4.06862,
+ "5155": 4.07596,
+ "5160": 4.05235,
+ "5165": 4.10013,
+ "5170": 3.96788,
+ "5175": 4.07695,
+ "5180": 4.03881,
+ "5185": 4.0638,
+ "5190": 4.08353,
+ "5195": 4.04661,
+ "5200": 4.06653,
+ "5205": 4.10296,
+ "5210": 4.00996,
+ "5215": 4.02532,
+ "5220": 4.02466,
+ "5225": 4.02373,
+ "5230": 4.06513,
+ "5235": 4.03848,
+ "5240": 4.02367,
+ "5245": 4.04608,
+ "5250": 4.04817,
+ "5255": 4.03312,
+ "5260": 4.05144,
+ "5265": 4.01508,
+ "5270": 3.98166,
+ "5275": 4.00551,
+ "5280": 4.01593,
+ "5285": 4.04383,
+ "5290": 4.00223,
+ "5295": 4.00532,
+ "5300": 4.02481,
+ "5305": 4.01028,
+ "5310": 4.05071,
+ "5315": 3.99772,
+ "5320": 4.03794,
+ "5325": 4.06595,
+ "5330": 3.99906,
+ "5335": 4.02462,
+ "5340": 3.97433,
+ "5345": 4.01482,
+ "5350": 4.02431,
+ "5355": 4.01675,
+ "5360": 3.96665,
+ "5365": 3.98726,
+ "5370": 4.03145,
+ "5375": 3.99579,
+ "5380": 3.98887,
+ "5385": 4.00929,
+ "5390": 3.99917,
+ "5395": 3.93391,
+ "5400": 4.02282,
+ "5405": 3.94768,
+ "5410": 4.03102,
+ "5415": 3.95322,
+ "5420": 3.98021,
+ "5425": 3.97002,
+ "5430": 3.9775,
+ "5435": 4.01237,
+ "5440": 3.96244,
+ "5445": 3.97136,
+ "5450": 3.98096,
+ "5455": 3.96427,
+ "5460": 3.98147,
+ "5465": 4.03732,
+ "5470": 3.99591,
+ "5475": 3.92622,
+ "5480": 4.00008,
+ "5485": 3.96755,
+ "5490": 3.99268,
+ "5495": 3.99566,
+ "5500": 3.95755,
+ "5505": 3.97186,
+ "5510": 4.00504,
+ "5515": 3.98193,
+ "5520": 3.9583,
+ "5525": 4.01431,
+ "5530": 3.95958,
+ "5535": 3.99033,
+ "5540": 3.96319,
+ "5545": 3.97795,
+ "5550": 3.97044,
+ "5555": 3.93598,
+ "5560": 3.94377,
+ "5565": 3.98857,
+ "5570": 3.94455,
+ "5575": 3.97907,
+ "5580": 3.95135,
+ "5585": 3.89214,
+ "5590": 3.96731,
+ "5595": 3.92103,
+ "5600": 3.97134,
+ "5605": 3.87763,
+ "5610": 3.96824,
+ "5615": 3.96505,
+ "5620": 3.98017,
+ "5625": 3.95971,
+ "5630": 3.94811,
+ "5635": 3.93461,
+ "5640": 3.95001,
+ "5645": 3.91835,
+ "5650": 3.89104,
+ "5655": 3.92038,
+ "5660": 3.91091,
+ "5665": 3.92773,
+ "5670": 3.91586,
+ "5675": 3.94666,
+ "5680": 3.91111,
+ "5685": 3.92344,
+ "5690": 3.927,
+ "5695": 3.9542,
+ "5700": 3.88687,
+ "5705": 3.88957,
+ "5710": 3.87795,
+ "5715": 3.9953,
+ "5720": 3.94708,
+ "5725": 3.89504,
+ "5730": 3.95019,
+ "5735": 3.93047,
+ "5740": 3.9246,
+ "5745": 3.89933,
+ "5750": 3.92546,
+ "5755": 3.94874,
+ "5760": 3.92848,
+ "5765": 3.92286,
+ "5770": 3.95223,
+ "5775": 3.86986,
+ "5780": 3.91681,
+ "5785": 3.91926,
+ "5790": 3.92741,
+ "5795": 3.93109,
+ "5800": 3.87022,
+ "5805": 3.87879,
+ "5810": 3.92933,
+ "5815": 3.89365,
+ "5820": 3.84033,
+ "5825": 3.8924,
+ "5830": 3.85167,
+ "5835": 3.886,
+ "5840": 3.89743,
+ "5845": 3.91338,
+ "5850": 3.90827,
+ "5855": 3.84657,
+ "5860": 3.86912,
+ "5865": 3.89856,
+ "5870": 3.86095,
+ "5875": 3.89827,
+ "5880": 3.88257,
+ "5885": 3.90053,
+ "5890": 3.90452,
+ "5895": 3.92781,
+ "5900": 3.85969,
+ "5905": 3.92211,
+ "5910": 3.8891,
+ "5915": 3.85176,
+ "5920": 3.89016,
+ "5925": 3.82488,
+ "5930": 3.88484,
+ "5935": 3.8717,
+ "5940": 3.90094,
+ "5945": 3.90248,
+ "5950": 3.88587,
+ "5955": 3.84387,
+ "5960": 3.91223,
+ "5965": 3.85518,
+ "5970": 3.90545,
+ "5975": 3.87245,
+ "5980": 3.94673,
+ "5985": 3.8205,
+ "5990": 3.91455,
+ "5995": 3.82665,
+ "6000": 3.86266,
+ "6005": 3.82463,
+ "6010": 3.84735,
+ "6015": 3.82827,
+ "6020": 3.84539,
+ "6025": 3.88262,
+ "6030": 3.82549,
+ "6035": 3.87777,
+ "6040": 3.85606,
+ "6045": 3.88869,
+ "6050": 3.86306,
+ "6055": 3.84262,
+ "6060": 3.86622,
+ "6065": 3.89851,
+ "6070": 3.84909,
+ "6075": 3.7938,
+ "6080": 3.86708,
+ "6085": 3.82944,
+ "6090": 3.86095,
+ "6095": 3.86089,
+ "6100": 3.82666,
+ "6105": 3.87236,
+ "6110": 3.8074,
+ "6115": 3.88014,
+ "6120": 3.85095,
+ "6125": 3.85715,
+ "6130": 3.85028,
+ "6135": 3.8272,
+ "6140": 3.82173,
+ "6145": 3.81273,
+ "6150": 3.8594,
+ "6155": 3.83915,
+ "6160": 3.80364,
+ "6165": 3.82308,
+ "6170": 3.81476,
+ "6175": 3.8082,
+ "6180": 3.80736,
+ "6185": 3.84594,
+ "6190": 3.81603,
+ "6195": 3.78172,
+ "6200": 3.80964,
+ "6205": 3.81429,
+ "6210": 3.773,
+ "6215": 3.82875,
+ "6220": 3.82387,
+ "6225": 3.8272,
+ "6230": 3.77123,
+ "6235": 3.80769,
+ "6240": 3.73575,
+ "6245": 3.84648,
+ "6250": 3.81013,
+ "6255": 3.82091,
+ "6260": 3.79799,
+ "6265": 3.8294,
+ "6270": 3.75837,
+ "6275": 3.78098,
+ "6280": 3.80055,
+ "6285": 3.78104,
+ "6290": 3.80043,
+ "6295": 3.80115,
+ "6300": 3.81041,
+ "6305": 3.8833,
+ "6310": 3.76979,
+ "6315": 3.76414,
+ "6320": 3.81903,
+ "6325": 3.75489,
+ "6330": 3.8197,
+ "6335": 3.82108,
+ "6340": 3.768,
+ "6345": 3.82324,
+ "6350": 3.76701,
+ "6355": 3.77573,
+ "6360": 3.75553,
+ "6365": 3.81035,
+ "6370": 3.81018,
+ "6375": 3.78883,
+ "6380": 3.80581,
+ "6385": 3.82035,
+ "6390": 3.78287,
+ "6395": 3.75999,
+ "6400": 3.76146,
+ "6405": 3.84036,
+ "6410": 3.83491,
+ "6415": 3.76274,
+ "6420": 3.82519,
+ "6425": 3.82963,
+ "6430": 3.80865,
+ "6435": 3.7801,
+ "6440": 3.76393,
+ "6445": 3.80425,
+ "6450": 3.73756,
+ "6455": 3.75086,
+ "6460": 3.77243,
+ "6465": 3.81062,
+ "6470": 3.7883,
+ "6475": 3.78268,
+ "6480": 3.81461,
+ "6485": 3.76417,
+ "6490": 3.71273,
+ "6495": 3.81399,
+ "6500": 3.79956,
+ "6505": 3.7273,
+ "6510": 3.79958,
+ "6515": 3.81791,
+ "6520": 3.73229,
+ "6525": 3.80854,
+ "6530": 3.77133,
+ "6535": 3.76236,
+ "6540": 3.82765,
+ "6545": 3.76417,
+ "6550": 3.77124,
+ "6555": 3.7583,
+ "6560": 3.70927,
+ "6565": 3.70824,
+ "6570": 3.7493,
+ "6575": 3.69534,
+ "6580": 3.81282,
+ "6585": 3.7597,
+ "6590": 3.72355,
+ "6595": 3.74692,
+ "6600": 3.74109,
+ "6605": 3.71931,
+ "6610": 3.7276,
+ "6615": 3.75731,
+ "6620": 3.70964,
+ "6625": 3.72465,
+ "6630": 3.72012,
+ "6635": 3.7624,
+ "6640": 3.74061,
+ "6645": 3.75182,
+ "6650": 3.78125,
+ "6655": 3.70685,
+ "6660": 3.73622,
+ "6665": 3.75691,
+ "6670": 3.72099,
+ "6675": 3.74269,
+ "6680": 3.7359,
+ "6685": 3.76585,
+ "6690": 3.74349,
+ "6695": 3.75843,
+ "6700": 3.74544,
+ "6705": 3.72895,
+ "6710": 3.7317,
+ "6715": 3.69416,
+ "6720": 3.77743,
+ "6725": 3.75842,
+ "6730": 3.74238,
+ "6735": 3.73953,
+ "6740": 3.7387,
+ "6745": 3.72189,
+ "6750": 3.74314,
+ "6755": 3.696,
+ "6760": 3.68251,
+ "6765": 3.74589,
+ "6770": 3.69648,
+ "6775": 3.74806,
+ "6780": 3.70535,
+ "6785": 3.70952,
+ "6790": 3.73623,
+ "6795": 3.69966,
+ "6800": 3.71994,
+ "6805": 3.72339,
+ "6810": 3.7356,
+ "6815": 3.66043,
+ "6820": 3.7027,
+ "6825": 3.72689,
+ "6830": 3.70704,
+ "6835": 3.68791,
+ "6840": 3.67686,
+ "6845": 3.75106,
+ "6850": 3.70598,
+ "6855": 3.73709,
+ "6860": 3.67005,
+ "6865": 3.73577,
+ "6870": 3.69492,
+ "6875": 3.69782,
+ "6880": 3.70538,
+ "6885": 3.67965,
+ "6890": 3.69271,
+ "6895": 3.67845,
+ "6900": 3.68245,
+ "6905": 3.68847,
+ "6910": 3.73023,
+ "6915": 3.73538,
+ "6920": 3.68966,
+ "6925": 3.68951,
+ "6930": 3.68833,
+ "6935": 3.62184,
+ "6940": 3.69413,
+ "6945": 3.68268,
+ "6950": 3.68095,
+ "6955": 3.67817,
+ "6960": 3.67928,
+ "6965": 3.72271,
+ "6970": 3.64813,
+ "6975": 3.72907,
+ "6980": 3.68375,
+ "6985": 3.68875,
+ "6990": 3.73488,
+ "6995": 3.70594,
+ "7000": 3.64046,
+ "7005": 3.71807,
+ "7010": 3.69136,
+ "7015": 3.68094,
+ "7020": 3.72175,
+ "7025": 3.70915,
+ "7030": 3.70482,
+ "7035": 3.65934,
+ "7040": 3.61581,
+ "7045": 3.69636,
+ "7050": 3.72066,
+ "7055": 3.64831,
+ "7060": 3.69293,
+ "7065": 3.74269,
+ "7070": 3.6732,
+ "7075": 3.67722,
+ "7080": 3.71856,
+ "7085": 3.64106,
+ "7090": 3.66237,
+ "7095": 3.63972,
+ "7100": 3.68446,
+ "7105": 3.6207,
+ "7110": 3.68404,
+ "7115": 3.63824,
+ "7120": 3.68749,
+ "7125": 3.63589,
+ "7130": 3.65669,
+ "7135": 3.6615,
+ "7140": 3.66579,
+ "7145": 3.68276,
+ "7150": 3.62823,
+ "7155": 3.69379,
+ "7160": 3.62527,
+ "7165": 3.64414,
+ "7170": 3.68434,
+ "7175": 3.64953,
+ "7180": 3.67759,
+ "7185": 3.70786,
+ "7190": 3.66231,
+ "7195": 3.66963,
+ "7200": 3.67008,
+ "7205": 3.65917,
+ "7210": 3.69356,
+ "7215": 3.67296,
+ "7220": 3.69199,
+ "7225": 3.66305,
+ "7230": 3.68823,
+ "7235": 3.65006,
+ "7240": 3.6473,
+ "7245": 3.66429,
+ "7250": 3.6054,
+ "7255": 3.62604,
+ "7260": 3.68065,
+ "7265": 3.60471,
+ "7270": 3.63901,
+ "7275": 3.64676,
+ "7280": 3.62714,
+ "7285": 3.65202,
+ "7290": 3.67313,
+ "7295": 3.66123,
+ "7300": 3.62484,
+ "7305": 3.62802,
+ "7310": 3.66343,
+ "7315": 3.67704,
+ "7320": 3.65348,
+ "7325": 3.65775,
+ "7330": 3.62505,
+ "7335": 3.62748,
+ "7340": 3.64435,
+ "7345": 3.60654,
+ "7350": 3.65843,
+ "7355": 3.64272,
+ "7360": 3.6184,
+ "7365": 3.63674,
+ "7370": 3.61972,
+ "7375": 3.5937,
+ "7380": 3.64902,
+ "7385": 3.67206,
+ "7390": 3.66143,
+ "7395": 3.60809,
+ "7400": 3.65693,
+ "7405": 3.64922,
+ "7410": 3.66211,
+ "7415": 3.64572,
+ "7420": 3.63516,
+ "7425": 3.68583,
+ "7430": 3.63524,
+ "7435": 3.6164,
+ "7440": 3.62739,
+ "7445": 3.6112,
+ "7450": 3.57237,
+ "7455": 3.64972,
+ "7460": 3.63661,
+ "7465": 3.63216,
+ "7470": 3.63965,
+ "7475": 3.64299,
+ "7480": 3.61364,
+ "7485": 3.57549,
+ "7490": 3.57347,
+ "7495": 3.58771,
+ "7500": 3.6164,
+ "7505": 3.59654,
+ "7510": 3.55824,
+ "7515": 3.61644,
+ "7520": 3.61228,
+ "7525": 3.56686,
+ "7530": 3.61404,
+ "7535": 3.62712,
+ "7540": 3.60986,
+ "7545": 3.65016,
+ "7550": 3.65968,
+ "7555": 3.58547,
+ "7560": 3.60347,
+ "7565": 3.59807,
+ "7570": 3.60876,
+ "7575": 3.57394,
+ "7580": 3.62143,
+ "7585": 3.60274,
+ "7590": 3.60227,
+ "7595": 3.66186,
+ "7600": 3.60715,
+ "7605": 3.597,
+ "7610": 3.58443,
+ "7615": 3.58487,
+ "7620": 3.56883,
+ "7625": 3.62268,
+ "7630": 3.60553,
+ "7635": 3.59484,
+ "7640": 3.59423,
+ "7645": 3.62655,
+ "7650": 3.62773,
+ "7655": 3.66489,
+ "7660": 3.53077,
+ "7665": 3.60609,
+ "7670": 3.60031,
+ "7675": 3.58564,
+ "7680": 3.57882,
+ "7685": 3.64699,
+ "7690": 3.58913,
+ "7695": 3.57137,
+ "7700": 3.63613,
+ "7705": 3.58873,
+ "7710": 3.62165,
+ "7715": 3.57624,
+ "7720": 3.65709,
+ "7725": 3.55445,
+ "7730": 3.57547,
+ "7735": 3.61057,
+ "7740": 3.58596,
+ "7745": 3.58661,
+ "7750": 3.57434,
+ "7755": 3.59627,
+ "7760": 3.56271,
+ "7765": 3.58383,
+ "7770": 3.60113,
+ "7775": 3.57186,
+ "7780": 3.55682,
+ "7785": 3.57862,
+ "7790": 3.57134,
+ "7795": 3.5873,
+ "7800": 3.57772,
+ "7805": 3.58313,
+ "7810": 3.60852,
+ "7815": 3.58186,
+ "7820": 3.57863,
+ "7825": 3.61969,
+ "7830": 3.59286,
+ "7835": 3.52901,
+ "7840": 3.62093,
+ "7845": 3.55604,
+ "7850": 3.51327,
+ "7855": 3.56807,
+ "7860": 3.54745,
+ "7865": 3.60488,
+ "7870": 3.54225,
+ "7875": 3.55775,
+ "7880": 3.57273,
+ "7885": 3.56156,
+ "7890": 3.60755,
+ "7895": 3.59562,
+ "7900": 3.60847,
+ "7905": 3.56518,
+ "7910": 3.58419,
+ "7915": 3.58454,
+ "7920": 3.59109,
+ "7925": 3.56854,
+ "7930": 3.59965,
+ "7935": 3.56045,
+ "7940": 3.61238,
+ "7945": 3.62942,
+ "7950": 3.53793,
+ "7955": 3.54668,
+ "7960": 3.53277,
+ "7965": 3.51796,
+ "7970": 3.52553,
+ "7975": 3.56012,
+ "7980": 3.56745,
+ "7985": 3.54203,
+ "7990": 3.5494,
+ "7995": 3.51957,
+ "8000": 3.57766,
+ "8005": 3.54643,
+ "8010": 3.53991,
+ "8015": 3.53754,
+ "8020": 3.53136,
+ "8025": 3.5155,
+ "8030": 3.54141,
+ "8035": 3.53515,
+ "8040": 3.52216,
+ "8045": 3.57841,
+ "8050": 3.57873,
+ "8055": 3.55046,
+ "8060": 3.57271,
+ "8065": 3.55036,
+ "8070": 3.53764,
+ "8075": 3.52808,
+ "8080": 3.57588,
+ "8085": 3.53013,
+ "8090": 3.53483,
+ "8095": 3.56395,
+ "8100": 3.51568,
+ "8105": 3.54987,
+ "8110": 3.54694,
+ "8115": 3.51487,
+ "8120": 3.52624,
+ "8125": 3.56575,
+ "8130": 3.52781,
+ "8135": 3.54026,
+ "8140": 3.52068,
+ "8145": 3.50411,
+ "8150": 3.52474,
+ "8155": 3.51156,
+ "8160": 3.56201,
+ "8165": 3.54529,
+ "8170": 3.51253,
+ "8175": 3.50415,
+ "8180": 3.57386,
+ "8185": 3.54758,
+ "8190": 3.58531,
+ "8195": 3.55206,
+ "8200": 3.52363,
+ "8205": 3.53113,
+ "8210": 3.53685,
+ "8215": 3.55891,
+ "8220": 3.52029,
+ "8225": 3.51186,
+ "8230": 3.53685,
+ "8235": 3.557,
+ "8240": 3.53963,
+ "8245": 3.53856,
+ "8250": 3.56789,
+ "8255": 3.51811,
+ "8260": 3.52989,
+ "8265": 3.52093,
+ "8270": 3.53064,
+ "8275": 3.51863,
+ "8280": 3.50393,
+ "8285": 3.52643,
+ "8290": 3.5276,
+ "8295": 3.49709,
+ "8300": 3.51744,
+ "8305": 3.54184,
+ "8310": 3.53447,
+ "8315": 3.50429,
+ "8320": 3.53144,
+ "8325": 3.47882,
+ "8330": 3.44314,
+ "8335": 3.51422,
+ "8340": 3.54178,
+ "8345": 3.49858,
+ "8350": 3.51287,
+ "8355": 3.54337,
+ "8360": 3.51723,
+ "8365": 3.53843,
+ "8370": 3.53305,
+ "8375": 3.4851,
+ "8380": 3.48647,
+ "8385": 3.53215,
+ "8390": 3.49586,
+ "8395": 3.52855,
+ "8400": 3.49457,
+ "8405": 3.51692,
+ "8410": 3.57628,
+ "8415": 3.48276,
+ "8420": 3.45375,
+ "8425": 3.53458,
+ "8430": 3.5414,
+ "8435": 3.47645,
+ "8440": 3.55245,
+ "8445": 3.53756,
+ "8450": 3.50866,
+ "8455": 3.53143,
+ "8460": 3.53629,
+ "8465": 3.4719,
+ "8470": 3.49508,
+ "8475": 3.552,
+ "8480": 3.47573,
+ "8485": 3.49609,
+ "8490": 3.48508,
+ "8495": 3.48353,
+ "8500": 3.52895,
+ "8505": 3.46794,
+ "8510": 3.54119,
+ "8515": 3.48915,
+ "8520": 3.49416,
+ "8525": 3.42391,
+ "8530": 3.50299,
+ "8535": 3.52266,
+ "8540": 3.47726,
+ "8545": 3.50154,
+ "8550": 3.47025,
+ "8555": 3.53614,
+ "8560": 3.53625,
+ "8565": 3.48787,
+ "8570": 3.48929,
+ "8575": 3.46476,
+ "8580": 3.50918,
+ "8585": 3.52899,
+ "8590": 3.51669,
+ "8595": 3.52589,
+ "8600": 3.50297,
+ "8605": 3.49081,
+ "8610": 3.49587,
+ "8615": 3.49602,
+ "8620": 3.46451,
+ "8625": 3.48802,
+ "8630": 3.4955,
+ "8635": 3.47575,
+ "8640": 3.46286,
+ "8645": 3.52917,
+ "8650": 3.45905,
+ "8655": 3.50395,
+ "8660": 3.5137,
+ "8665": 3.48991,
+ "8670": 3.50603,
+ "8675": 3.47472,
+ "8680": 3.46823,
+ "8685": 3.47944,
+ "8690": 3.5141,
+ "8695": 3.51713,
+ "8700": 3.48511,
+ "8705": 3.45387,
+ "8710": 3.50156,
+ "8715": 3.453,
+ "8720": 3.52778,
+ "8725": 3.48997,
+ "8730": 3.48217,
+ "8735": 3.5106,
+ "8740": 3.46098,
+ "8745": 3.50189,
+ "8750": 3.50679,
+ "8755": 3.46697,
+ "8760": 3.48366,
+ "8765": 3.44123,
+ "8770": 3.51206,
+ "8775": 3.47401,
+ "8780": 3.45756,
+ "8785": 3.47595,
+ "8790": 3.45966,
+ "8795": 3.49641,
+ "8800": 3.46423,
+ "8805": 3.43485,
+ "8810": 3.45219,
+ "8815": 3.47515,
+ "8820": 3.43845,
+ "8825": 3.46995,
+ "8830": 3.44579,
+ "8835": 3.42261,
+ "8840": 3.43517,
+ "8845": 3.45929,
+ "8850": 3.48297,
+ "8855": 3.46708,
+ "8860": 3.53308,
+ "8865": 3.46845,
+ "8870": 3.44858,
+ "8875": 3.45526,
+ "8880": 3.4575,
+ "8885": 3.4507,
+ "8890": 3.47325,
+ "8895": 3.45281,
+ "8900": 3.48013,
+ "8905": 3.468,
+ "8910": 3.45481,
+ "8915": 3.44308,
+ "8920": 3.43466,
+ "8925": 3.50848,
+ "8930": 3.49192,
+ "8935": 3.50161,
+ "8940": 3.47621,
+ "8945": 3.48008,
+ "8950": 3.45786,
+ "8955": 3.44537,
+ "8960": 3.43814,
+ "8965": 3.45904,
+ "8970": 3.47355,
+ "8975": 3.42274,
+ "8980": 3.42271,
+ "8985": 3.4467,
+ "8990": 3.50086,
+ "8995": 3.47839,
+ "9000": 3.42075,
+ "9005": 3.46368,
+ "9010": 3.51653,
+ "9015": 3.41767,
+ "9020": 3.43917,
+ "9025": 3.44725,
+ "9030": 3.47032,
+ "9035": 3.38232,
+ "9040": 3.4547,
+ "9045": 3.45666,
+ "9050": 3.49185,
+ "9055": 3.40238,
+ "9060": 3.49504,
+ "9065": 3.51386,
+ "9070": 3.44729,
+ "9075": 3.47617,
+ "9080": 3.4696,
+ "9085": 3.47466,
+ "9090": 3.46715,
+ "9095": 3.42336,
+ "9100": 3.42475,
+ "9105": 3.41388,
+ "9110": 3.45791,
+ "9115": 3.46493,
+ "9120": 3.52004,
+ "9125": 3.45343,
+ "9130": 3.43885,
+ "9135": 3.46034,
+ "9140": 3.47807,
+ "9145": 3.4246,
+ "9150": 3.44332,
+ "9155": 3.45254,
+ "9160": 3.45094,
+ "9165": 3.45648,
+ "9170": 3.47722,
+ "9175": 3.41237,
+ "9180": 3.45617,
+ "9185": 3.40973,
+ "9190": 3.46952,
+ "9195": 3.43613,
+ "9200": 3.44488,
+ "9205": 3.42464,
+ "9210": 3.45571,
+ "9215": 3.39623,
+ "9220": 3.4234,
+ "9225": 3.44831,
+ "9230": 3.37468,
+ "9235": 3.39547,
+ "9240": 3.42207,
+ "9245": 3.40803,
+ "9250": 3.40876,
+ "9255": 3.42162,
+ "9260": 3.39809,
+ "9265": 3.44466,
+ "9270": 3.40777,
+ "9275": 3.42928,
+ "9280": 3.44486,
+ "9285": 3.43977,
+ "9290": 3.45706,
+ "9295": 3.44569,
+ "9300": 3.39554,
+ "9305": 3.42636,
+ "9310": 3.41638,
+ "9315": 3.38331,
+ "9320": 3.37904,
+ "9325": 3.42196,
+ "9330": 3.47898,
+ "9335": 3.38882,
+ "9340": 3.47118,
+ "9345": 3.46416,
+ "9350": 3.42772,
+ "9355": 3.395,
+ "9360": 3.41676,
+ "9365": 3.41276,
+ "9370": 3.4625,
+ "9375": 3.42652,
+ "9380": 3.36412,
+ "9385": 3.43603,
+ "9390": 3.44441,
+ "9395": 3.45524,
+ "9400": 3.41642,
+ "9405": 3.40046,
+ "9410": 3.43685,
+ "9415": 3.42527,
+ "9420": 3.4026,
+ "9425": 3.42185,
+ "9430": 3.39442,
+ "9435": 3.41556,
+ "9440": 3.40204,
+ "9445": 3.40027,
+ "9450": 3.39831,
+ "9455": 3.39977,
+ "9460": 3.46315,
+ "9465": 3.46257,
+ "9470": 3.40636,
+ "9475": 3.45452,
+ "9480": 3.40763,
+ "9485": 3.40119,
+ "9490": 3.41196,
+ "9495": 3.44307,
+ "9500": 3.40602,
+ "9505": 3.37732,
+ "9510": 3.41556,
+ "9515": 3.41214,
+ "9520": 3.43173,
+ "9525": 3.40131,
+ "9530": 3.40283,
+ "9535": 3.42176
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 9535,
+ "step_interval": 5,
+ "values": {
+ "1": "nan",
+ "5": 11.74638,
+ "10": 11.71606,
+ "15": 11.62124,
+ "20": 11.56773,
+ "25": 11.55348,
+ "30": 11.54401,
+ "35": 11.52274,
+ "40": 11.51838,
+ "45": 11.5073,
+ "50": 11.5514,
+ "55": 22.60625,
+ "60": 11.63006,
+ "65": 11.50767,
+ "70": 11.70413,
+ "75": 11.65014,
+ "80": 11.65778,
+ "85": 11.62282,
+ "90": 16.78078,
+ "95": 11.71377,
+ "100": 11.75533,
+ "105": 11.86712,
+ "110": 11.87563,
+ "115": 12.04117,
+ "120": 12.32924,
+ "125": 17.54915,
+ "130": 12.48015,
+ "135": 12.57353,
+ "140": 12.943,
+ "145": 12.90407,
+ "150": 13.23393,
+ "155": 13.34397,
+ "160": 13.44046,
+ "165": 13.43119,
+ "170": 13.44237,
+ "175": 13.39568,
+ "180": 13.41763,
+ "185": 13.24419,
+ "190": 13.34754,
+ "195": 18.17407,
+ "200": 13.0844,
+ "205": 13.01785,
+ "210": 13.11044,
+ "215": 12.89733,
+ "220": 17.40567,
+ "225": 12.75557,
+ "230": 12.71616,
+ "235": 12.84982,
+ "240": 12.62285,
+ "245": 12.59676,
+ "250": 12.55649,
+ "255": 17.8502,
+ "260": 12.54592,
+ "265": 12.56118,
+ "270": 18.07692,
+ "275": 12.43569,
+ "280": 12.34155,
+ "285": 12.31488,
+ "290": 12.19143,
+ "295": 12.17074,
+ "300": 12.17133,
+ "305": 12.13935,
+ "310": 12.14755,
+ "315": 12.0608,
+ "320": 23.04873,
+ "325": 12.23077,
+ "330": 17.35937,
+ "335": 12.12925,
+ "340": 12.05107,
+ "345": 12.0238,
+ "350": 12.62283,
+ "355": 12.03638,
+ "360": 11.99067,
+ "365": 12.09598,
+ "370": 11.96099,
+ "375": 11.927,
+ "380": 11.9891,
+ "385": 12.123,
+ "390": 11.9521,
+ "395": 11.91706,
+ "400": 11.93682,
+ "405": 11.89295,
+ "410": 11.88267,
+ "415": 11.88729,
+ "420": 12.03614,
+ "425": 11.95188,
+ "430": 23.63071,
+ "435": 11.85686,
+ "440": 11.85877,
+ "445": 12.07395,
+ "450": 11.83426,
+ "455": 11.90826,
+ "460": 11.91483,
+ "465": 11.97821,
+ "470": 17.33956,
+ "475": 11.97049,
+ "480": 11.9647,
+ "485": 12.03174,
+ "490": 12.10039,
+ "495": 12.17126,
+ "500": 16.24935,
+ "505": 12.21115,
+ "510": 12.24095,
+ "515": 17.39133,
+ "520": 12.3946,
+ "525": 17.91544,
+ "530": 17.22759,
+ "535": 12.43651,
+ "540": 12.37398,
+ "545": 23.61839,
+ "550": 12.35847,
+ "555": 12.45413,
+ "560": 28.61048,
+ "565": 12.37263,
+ "570": 12.39731,
+ "575": 12.38876,
+ "580": 17.46513,
+ "585": 12.29047,
+ "590": 12.32576,
+ "595": 12.35946,
+ "600": 12.33053,
+ "605": 12.33072,
+ "610": 12.27186,
+ "615": 12.28946,
+ "620": 12.29203,
+ "625": 12.28402,
+ "630": 23.57432,
+ "635": 12.36512,
+ "640": 12.41204,
+ "645": 12.27254,
+ "650": 17.46731,
+ "655": 12.24251,
+ "660": 12.2041,
+ "665": 14.99678,
+ "670": 37.15938,
+ "675": 12.1899,
+ "680": 12.14101,
+ "685": 12.13092,
+ "690": 12.17929,
+ "695": 17.49305,
+ "700": 12.08416,
+ "705": 12.09208,
+ "710": 12.09768,
+ "715": 22.09082,
+ "720": 12.00133,
+ "725": 12.01571,
+ "730": 12.05165,
+ "735": 12.09758,
+ "740": 12.10947,
+ "745": 12.18888,
+ "750": 12.05628,
+ "755": 12.0394,
+ "760": 12.33288,
+ "765": 12.03319,
+ "770": 12.0076,
+ "775": 12.06031,
+ "780": 12.00809,
+ "785": 12.04389,
+ "790": 11.99905,
+ "795": 12.03911,
+ "800": 12.01608,
+ "805": 12.18327,
+ "810": 12.06469,
+ "815": 12.22141,
+ "820": 12.05644,
+ "825": 11.97103,
+ "830": 12.08702,
+ "835": 12.02165,
+ "840": 11.99977,
+ "845": 12.01228,
+ "850": 11.99497,
+ "855": 12.02844,
+ "860": 12.1111,
+ "865": 12.0751,
+ "870": 12.07942,
+ "875": 12.02113,
+ "880": 12.03037,
+ "885": 12.01521,
+ "890": 12.15932,
+ "895": 12.02617,
+ "900": 12.03115,
+ "905": 12.03471,
+ "910": 12.13221,
+ "915": 12.02233,
+ "920": 11.99455,
+ "925": 11.98459,
+ "930": 11.99582,
+ "935": 12.06912,
+ "940": 12.0621,
+ "945": 12.0327,
+ "950": 12.06879,
+ "955": 12.13658,
+ "960": 12.01902,
+ "965": 12.02244,
+ "970": 11.99911,
+ "975": 12.04229,
+ "980": 12.06882,
+ "985": 12.02709,
+ "990": 12.04923,
+ "995": 12.0895,
+ "1000": 12.11027,
+ "1005": 12.14185,
+ "1010": 12.12271,
+ "1015": 12.08477,
+ "1020": 23.07498,
+ "1025": 11.99821,
+ "1030": 12.06531,
+ "1035": 12.07982,
+ "1040": 12.14349,
+ "1045": 12.22344,
+ "1050": 12.06058,
+ "1055": 12.03879,
+ "1060": 12.04391,
+ "1065": 12.06657,
+ "1070": 12.76684,
+ "1075": 12.0434,
+ "1080": 12.05745,
+ "1085": 12.06564,
+ "1090": 12.05614,
+ "1095": 12.19868,
+ "1100": 17.75852,
+ "1105": 12.1275,
+ "1110": 12.06993,
+ "1115": 12.20579,
+ "1120": 12.05312,
+ "1125": 12.15032,
+ "1130": 12.0462,
+ "1135": 12.02085,
+ "1140": 12.00791,
+ "1145": 12.08059,
+ "1150": 12.07142,
+ "1155": 12.1178,
+ "1160": 12.13759,
+ "1165": 12.07373,
+ "1170": 12.20232,
+ "1175": 18.11697,
+ "1180": 12.64294,
+ "1185": 17.14898,
+ "1190": 12.05917,
+ "1195": 12.06234,
+ "1200": 11.99488,
+ "1205": 12.06263,
+ "1210": 12.15264,
+ "1215": 12.07356,
+ "1220": 12.03342,
+ "1225": 17.03695,
+ "1230": 12.04442,
+ "1235": 18.62236,
+ "1240": 12.01527,
+ "1245": 12.02989,
+ "1250": 12.10979,
+ "1255": 17.30344,
+ "1260": 12.05118,
+ "1265": 12.0283,
+ "1270": 29.06426,
+ "1275": 17.16334,
+ "1280": 12.02121,
+ "1285": 12.07671,
+ "1290": 11.97426,
+ "1295": 12.00951,
+ "1300": 12.22851,
+ "1305": 12.10035,
+ "1310": 17.44605,
+ "1315": 12.06021,
+ "1320": 12.08926,
+ "1325": 12.15463,
+ "1330": 12.24405,
+ "1335": 17.29863,
+ "1340": 20.43665,
+ "1345": 12.08081,
+ "1350": 12.02942,
+ "1355": 12.02503,
+ "1360": 12.0494,
+ "1365": 12.08879,
+ "1370": 12.2251,
+ "1375": 12.22301,
+ "1380": 12.05152,
+ "1385": 12.19908,
+ "1390": 12.11187,
+ "1395": 12.01147,
+ "1400": 12.04695,
+ "1405": 12.44867,
+ "1410": 12.14616,
+ "1415": 17.2707,
+ "1420": 12.15493,
+ "1425": 12.00884,
+ "1430": 12.05706,
+ "1435": 12.04022,
+ "1440": 17.13469,
+ "1445": 17.65059,
+ "1450": 12.00352,
+ "1455": 12.06822,
+ "1460": 11.99435,
+ "1465": 12.05829,
+ "1470": 12.03461,
+ "1475": 12.06868,
+ "1480": 12.06945,
+ "1485": 12.09449,
+ "1490": 12.08656,
+ "1495": 12.02279,
+ "1500": 12.15846,
+ "1505": 12.08881,
+ "1510": 12.01878,
+ "1515": 12.27681,
+ "1520": 12.23192,
+ "1525": 12.15017,
+ "1530": 12.2215,
+ "1535": 12.12907,
+ "1540": 11.9902,
+ "1545": 12.05489,
+ "1550": 12.00438,
+ "1555": 12.20034,
+ "1560": 12.1241,
+ "1565": 12.06577,
+ "1570": 12.13423,
+ "1575": 16.61537,
+ "1580": 12.19736,
+ "1585": 12.02532,
+ "1590": 12.12436,
+ "1595": 12.03422,
+ "1600": 12.00977,
+ "1605": 23.27287,
+ "1610": 17.29962,
+ "1615": 12.1692,
+ "1620": 12.42377,
+ "1625": 12.0538,
+ "1630": 12.07494,
+ "1635": 12.02796,
+ "1640": 12.00745,
+ "1645": 12.02804,
+ "1650": 11.99807,
+ "1655": 11.99339,
+ "1660": 23.44933,
+ "1665": 12.06552,
+ "1670": 12.06848,
+ "1675": 12.10936,
+ "1680": 12.04835,
+ "1685": 12.08947,
+ "1690": 12.10247,
+ "1695": 12.10678,
+ "1700": 17.38173,
+ "1705": 12.16391,
+ "1710": 12.07851,
+ "1715": 12.1407,
+ "1720": 12.1354,
+ "1725": 18.12115,
+ "1730": 12.05046,
+ "1735": 12.17396,
+ "1740": 12.15993,
+ "1745": 12.04432,
+ "1750": 12.17224,
+ "1755": 12.19196,
+ "1760": 12.05145,
+ "1765": 12.07931,
+ "1770": 12.058,
+ "1775": 12.03299,
+ "1780": 18.20423,
+ "1785": 12.03975,
+ "1790": 23.28873,
+ "1795": 12.05397,
+ "1800": 26.2849,
+ "1805": 23.28416,
+ "1810": 12.11463,
+ "1815": 12.08972,
+ "1820": 12.11859,
+ "1825": 12.04479,
+ "1830": 17.79151,
+ "1835": 17.90729,
+ "1840": 16.71125,
+ "1845": 23.33302,
+ "1850": 11.98855,
+ "1855": 29.5234,
+ "1860": 12.01709,
+ "1865": 23.54199,
+ "1870": 12.06206,
+ "1875": 12.02188,
+ "1880": 12.02192,
+ "1885": 12.04029,
+ "1890": 12.04373,
+ "1895": 12.07161,
+ "1900": 12.05659,
+ "1905": 12.21966,
+ "1910": 12.08813,
+ "1915": 12.01839,
+ "1920": 12.15181,
+ "1925": 12.06929,
+ "1930": 12.14006,
+ "1935": 12.01516,
+ "1940": 12.04445,
+ "1945": 12.08309,
+ "1950": 12.06244,
+ "1955": 12.05535,
+ "1960": 12.07967,
+ "1965": 12.06703,
+ "1970": 12.02503,
+ "1975": 12.06942,
+ "1980": 12.0604,
+ "1985": 12.01242,
+ "1990": 12.06831,
+ "1995": 11.99046,
+ "2000": 11.9936,
+ "2005": 12.1382,
+ "2010": 12.19701,
+ "2015": 12.05887,
+ "2020": 12.18334,
+ "2025": 12.08822,
+ "2030": 12.08146,
+ "2035": 12.06152,
+ "2040": 12.06112,
+ "2045": 12.03412,
+ "2050": 12.35559,
+ "2055": 12.07058,
+ "2060": 12.13077,
+ "2065": 12.02359,
+ "2070": 13.19286,
+ "2075": 12.11002,
+ "2080": 12.01943,
+ "2085": 12.0163,
+ "2090": 12.04744,
+ "2095": 95.09589,
+ "2100": 12.04589,
+ "2105": 12.08725,
+ "2110": 12.15522,
+ "2115": 11.99685,
+ "2120": 12.16709,
+ "2125": 17.47291,
+ "2130": 12.05281,
+ "2135": 12.04345,
+ "2140": 11.9309,
+ "2145": 12.02606,
+ "2150": 17.15638,
+ "2155": 28.75098,
+ "2160": 17.59205,
+ "2165": 12.06507,
+ "2170": 12.29733,
+ "2175": 12.23099,
+ "2180": 19.03867,
+ "2185": 12.08809,
+ "2190": 23.03177,
+ "2195": 12.02863,
+ "2200": 11.94224,
+ "2205": 12.08643,
+ "2210": 12.12769,
+ "2215": 11.96427,
+ "2220": 12.00506,
+ "2225": 11.99582,
+ "2230": 12.03776,
+ "2235": 11.97453,
+ "2240": 14.51879,
+ "2245": 11.97311,
+ "2250": 11.93011,
+ "2255": 12.02107,
+ "2260": 11.96801,
+ "2265": 12.00687,
+ "2270": 12.05062,
+ "2275": 12.14592,
+ "2280": 12.13636,
+ "2285": 12.06855,
+ "2290": 23.60284,
+ "2295": 11.98908,
+ "2300": 12.02421,
+ "2305": 11.9875,
+ "2310": 17.3919,
+ "2315": 11.95327,
+ "2320": 12.01618,
+ "2325": 12.02501,
+ "2330": 12.06841,
+ "2335": 12.00242,
+ "2340": 12.0303,
+ "2345": 11.98285,
+ "2350": 12.04455,
+ "2355": 11.95835,
+ "2360": 12.88899,
+ "2365": 11.98795,
+ "2370": 11.9649,
+ "2375": 23.44269,
+ "2380": 11.97357,
+ "2385": 12.00041,
+ "2390": 17.15683,
+ "2395": 11.96991,
+ "2400": 13.18225,
+ "2405": 18.35013,
+ "2410": 11.97451,
+ "2415": 12.01153,
+ "2420": 12.06077,
+ "2425": 11.98192,
+ "2430": 12.05665,
+ "2435": 23.36961,
+ "2440": 11.95462,
+ "2445": 11.99896,
+ "2450": 17.22313,
+ "2455": 12.10425,
+ "2460": 12.01534,
+ "2465": 12.05333,
+ "2470": 17.45707,
+ "2475": 12.01908,
+ "2480": 12.033,
+ "2485": 12.06353,
+ "2490": 11.98396,
+ "2495": 12.06592,
+ "2500": 12.03379,
+ "2505": 12.09202,
+ "2510": 12.1015,
+ "2515": 12.05454,
+ "2520": 11.95691,
+ "2525": 12.01022,
+ "2530": 12.06131,
+ "2535": 11.9875,
+ "2540": 12.0041,
+ "2545": 12.12128,
+ "2550": 32.67685,
+ "2555": 11.96034,
+ "2560": 11.9532,
+ "2565": 12.08606,
+ "2570": 16.05777,
+ "2575": 12.10205,
+ "2580": 12.04635,
+ "2585": 12.06672,
+ "2590": 11.97281,
+ "2595": 11.99234,
+ "2600": 23.05185,
+ "2605": 23.77828,
+ "2610": 12.04912,
+ "2615": 11.97698,
+ "2620": 11.98236,
+ "2625": 25.76622,
+ "2630": 12.2,
+ "2635": 12.05067,
+ "2640": 12.023,
+ "2645": 11.99898,
+ "2650": 23.57645,
+ "2655": 23.44326,
+ "2660": 23.33605,
+ "2665": 12.02986,
+ "2670": 12.11965,
+ "2675": 12.02702,
+ "2680": 11.98546,
+ "2685": 11.93881,
+ "2690": 11.9254,
+ "2695": 11.98774,
+ "2700": 12.08115,
+ "2705": 11.95822,
+ "2710": 11.96368,
+ "2715": 11.96762,
+ "2720": 12.02647,
+ "2725": 17.35565,
+ "2730": 12.01369,
+ "2735": 12.15608,
+ "2740": 12.02781,
+ "2745": 23.45308,
+ "2750": 12.0498,
+ "2755": 12.05056,
+ "2760": 12.0163,
+ "2765": 12.02506,
+ "2770": 11.93324,
+ "2775": 12.1107,
+ "2780": 11.9914,
+ "2785": 12.13689,
+ "2790": 11.95404,
+ "2795": 11.98987,
+ "2800": 11.99019,
+ "2805": 11.96052,
+ "2810": 12.00685,
+ "2815": 16.50378,
+ "2820": 12.01594,
+ "2825": 12.05585,
+ "2830": 14.37862,
+ "2835": 12.04817,
+ "2840": 14.73095,
+ "2845": 11.97047,
+ "2850": 12.09647,
+ "2855": 12.00726,
+ "2860": 12.09073,
+ "2865": 12.09086,
+ "2870": 11.96541,
+ "2875": 12.02802,
+ "2880": 11.97354,
+ "2885": 11.96056,
+ "2890": 12.01153,
+ "2895": 12.01113,
+ "2900": 12.01944,
+ "2905": 11.99946,
+ "2910": 11.9884,
+ "2915": 11.96792,
+ "2920": 12.14114,
+ "2925": 11.98067,
+ "2930": 11.97537,
+ "2935": 12.01285,
+ "2940": 11.9711,
+ "2945": 22.9472,
+ "2950": 12.13862,
+ "2955": 12.03482,
+ "2960": 12.05963,
+ "2965": 12.04058,
+ "2970": 11.9934,
+ "2975": 11.96865,
+ "2980": 12.09711,
+ "2985": 12.0403,
+ "2990": 11.99527,
+ "2995": 12.10345,
+ "3000": 11.9611,
+ "3005": 13.3386,
+ "3010": 11.98235,
+ "3015": "nan",
+ "3020": 12.58235,
+ "3025": 12.20592,
+ "3030": 12.08549,
+ "3035": 12.10115,
+ "3040": 12.14728,
+ "3045": 12.10928,
+ "3050": 12.19699,
+ "3055": 17.23946,
+ "3060": 12.13456,
+ "3065": 12.12407,
+ "3070": 12.12357,
+ "3075": 12.16242,
+ "3080": 12.10064,
+ "3085": 12.22666,
+ "3090": 12.18353,
+ "3095": 12.23022,
+ "3100": 12.27682,
+ "3105": 22.99263,
+ "3110": 12.13826,
+ "3115": 12.08714,
+ "3120": 12.09904,
+ "3125": 12.09836,
+ "3130": 12.0845,
+ "3135": 12.201,
+ "3140": 12.06398,
+ "3145": 12.05356,
+ "3150": 12.1205,
+ "3155": 23.05849,
+ "3160": 12.03783,
+ "3165": 12.16775,
+ "3170": 12.04224,
+ "3175": 12.17474,
+ "3180": 12.18384,
+ "3185": 12.11922,
+ "3190": 12.06415,
+ "3195": 12.1121,
+ "3200": 12.09898,
+ "3205": 12.07707,
+ "3210": 12.05763,
+ "3215": 12.01883,
+ "3220": 12.03026,
+ "3225": 12.17437,
+ "3230": 12.09146,
+ "3235": 12.16726,
+ "3240": 12.05884,
+ "3245": 12.07819,
+ "3250": 12.05481,
+ "3255": 12.13484,
+ "3260": 12.15727,
+ "3265": 12.19678,
+ "3270": 12.10192,
+ "3275": 12.14894,
+ "3280": 12.11447,
+ "3285": 12.07337,
+ "3290": 12.07075,
+ "3295": 12.12702,
+ "3300": 12.14357,
+ "3305": 17.39759,
+ "3310": 12.06641,
+ "3315": 12.09472,
+ "3320": 12.00378,
+ "3325": 12.13886,
+ "3330": 12.0691,
+ "3335": 12.2254,
+ "3340": 12.16984,
+ "3345": 12.26761,
+ "3350": 12.06951,
+ "3355": 17.30878,
+ "3360": 12.15877,
+ "3365": 12.09183,
+ "3370": 12.08344,
+ "3375": 12.10231,
+ "3380": 12.89721,
+ "3385": 14.7563,
+ "3390": 22.61337,
+ "3395": 12.09215,
+ "3400": 12.11075,
+ "3405": 12.0693,
+ "3410": 12.12554,
+ "3415": 12.06971,
+ "3420": 12.07287,
+ "3425": 12.12354,
+ "3430": 12.068,
+ "3435": 12.1323,
+ "3440": 12.06464,
+ "3445": 12.08418,
+ "3450": 28.62706,
+ "3455": 17.61308,
+ "3460": 12.07768,
+ "3465": 12.09643,
+ "3470": 12.1082,
+ "3475": 12.11131,
+ "3480": 12.03839,
+ "3485": 12.08087,
+ "3490": 12.00804,
+ "3495": 12.02463,
+ "3500": 12.10328,
+ "3505": 17.39374,
+ "3510": 12.14839,
+ "3515": 12.163,
+ "3520": 12.21711,
+ "3525": 12.04581,
+ "3530": 17.22428,
+ "3535": 12.11635,
+ "3540": 12.21692,
+ "3545": 12.13159,
+ "3550": 12.14012,
+ "3555": 12.12668,
+ "3560": 12.1902,
+ "3565": 17.99411,
+ "3570": 12.19451,
+ "3575": 12.10627,
+ "3580": 12.16686,
+ "3585": 12.16652,
+ "3590": 12.21366,
+ "3595": 12.02659,
+ "3600": 12.17245,
+ "3605": 12.11645,
+ "3610": 12.07511,
+ "3615": 12.19227,
+ "3620": 12.22395,
+ "3625": 12.06806,
+ "3630": 23.59244,
+ "3635": 12.14732,
+ "3640": 23.59428,
+ "3645": 12.17137,
+ "3650": 12.11488,
+ "3655": 17.57202,
+ "3660": 17.74532,
+ "3665": 17.98503,
+ "3670": 12.11526,
+ "3675": 12.1552,
+ "3680": 12.97689,
+ "3685": 12.1476,
+ "3690": 12.05715,
+ "3695": 12.17997,
+ "3700": 12.1672,
+ "3705": 12.03688,
+ "3710": 12.17369,
+ "3715": 12.11124,
+ "3720": 12.07522,
+ "3725": 12.08224,
+ "3730": 12.06251,
+ "3735": 18.1624,
+ "3740": 12.12604,
+ "3745": 12.17757,
+ "3750": 12.04736,
+ "3755": 12.06959,
+ "3760": 12.08065,
+ "3765": 12.18289,
+ "3770": 12.11854,
+ "3775": 12.07638,
+ "3780": 12.09127,
+ "3785": 12.09061,
+ "3790": 12.06931,
+ "3795": 12.07522,
+ "3800": 12.13646,
+ "3805": 12.28348,
+ "3810": 12.07555,
+ "3815": 12.11865,
+ "3820": 12.06766,
+ "3825": 12.28478,
+ "3830": 12.04206,
+ "3835": 12.05676,
+ "3840": 12.07887,
+ "3845": 12.03212,
+ "3850": 12.085,
+ "3855": 12.15783,
+ "3860": 12.05512,
+ "3865": 12.05258,
+ "3870": 12.08362,
+ "3875": 12.08618,
+ "3880": 12.03844,
+ "3885": 37.52757,
+ "3890": 12.1258,
+ "3895": 12.0943,
+ "3900": 12.05877,
+ "3905": 17.62779,
+ "3910": 12.08432,
+ "3915": 17.85638,
+ "3920": 12.14083,
+ "3925": 12.03494,
+ "3930": 12.09509,
+ "3935": 12.12955,
+ "3940": 12.11995,
+ "3945": 12.16791,
+ "3950": 12.22713,
+ "3955": 95.17224,
+ "3960": 12.22983,
+ "3965": 17.52108,
+ "3970": 17.33154,
+ "3975": 17.54123,
+ "3980": 17.14334,
+ "3985": 11.9976,
+ "3990": 12.0153,
+ "3995": 12.09925,
+ "4000": 12.07977,
+ "4005": 12.00321,
+ "4010": 17.04773,
+ "4015": 11.98109,
+ "4020": 12.09792,
+ "4025": 12.04973,
+ "4030": 12.04457,
+ "4035": 12.1162,
+ "4040": 12.16359,
+ "4045": 12.00834,
+ "4050": 17.6049,
+ "4055": 11.99392,
+ "4060": 12.0448,
+ "4065": 12.03396,
+ "4070": 12.08206,
+ "4075": 12.0158,
+ "4080": 12.03954,
+ "4085": 17.50196,
+ "4090": 12.00975,
+ "4095": 17.37615,
+ "4100": 12.05309,
+ "4105": 12.04364,
+ "4110": 11.97406,
+ "4115": 12.04741,
+ "4120": 12.11877,
+ "4125": 12.22159,
+ "4130": 17.66157,
+ "4135": 11.95322,
+ "4140": 12.00094,
+ "4145": 11.95883,
+ "4150": 12.06727,
+ "4155": 23.62712,
+ "4160": 12.01819,
+ "4165": 11.97169,
+ "4170": 12.01678,
+ "4175": 12.02209,
+ "4180": 12.03807,
+ "4185": 12.013,
+ "4190": 12.15375,
+ "4195": 11.9993,
+ "4200": 11.98377,
+ "4205": 12.01139,
+ "4210": 11.99338,
+ "4215": 18.07083,
+ "4220": 12.02306,
+ "4225": 12.01529,
+ "4230": 12.03861,
+ "4235": 12.00829,
+ "4240": 12.06055,
+ "4245": 12.07495,
+ "4250": 11.99841,
+ "4255": 11.97342,
+ "4260": 12.05805,
+ "4265": 12.06922,
+ "4270": 11.98133,
+ "4275": 11.99509,
+ "4280": 12.05986,
+ "4285": 11.99501,
+ "4290": 12.02425,
+ "4295": 12.02733,
+ "4300": 11.98723,
+ "4305": 12.11624,
+ "4310": 12.02537,
+ "4315": 17.10705,
+ "4320": 12.02035,
+ "4325": 11.97783,
+ "4330": 12.094,
+ "4335": 11.98988,
+ "4340": 11.99466,
+ "4345": 23.41181,
+ "4350": 11.96174,
+ "4355": 12.02967,
+ "4360": 12.04675,
+ "4365": 23.12241,
+ "4370": 12.0384,
+ "4375": 12.09053,
+ "4380": 12.05709,
+ "4385": 12.03169,
+ "4390": 12.04134,
+ "4395": 12.05838,
+ "4400": 12.00388,
+ "4405": 12.05844,
+ "4410": 12.01094,
+ "4415": 12.06089,
+ "4420": 11.97612,
+ "4425": 12.04149,
+ "4430": 17.19564,
+ "4435": 12.04592,
+ "4440": 12.03234,
+ "4445": 12.12801,
+ "4450": 11.94304,
+ "4455": 12.03362,
+ "4460": 11.97396,
+ "4465": 11.97825,
+ "4470": 12.04454,
+ "4475": 18.32298,
+ "4480": 11.97364,
+ "4485": 12.02994,
+ "4490": 12.06615,
+ "4495": 12.21985,
+ "4500": 11.97793,
+ "4505": 11.9986,
+ "4510": 11.98557,
+ "4515": 11.98491,
+ "4520": 12.07122,
+ "4525": 11.99476,
+ "4530": 12.13952,
+ "4535": 11.97011,
+ "4540": 12.08498,
+ "4545": 12.04388,
+ "4550": 11.9723,
+ "4555": 12.0213,
+ "4560": 12.94766,
+ "4565": 11.99376,
+ "4570": 12.10579,
+ "4575": 12.05257,
+ "4580": 11.98313,
+ "4585": 11.97304,
+ "4590": 12.05286,
+ "4595": 11.99662,
+ "4600": 12.09469,
+ "4605": 11.90371,
+ "4610": 11.9998,
+ "4615": 12.04112,
+ "4620": 11.99694,
+ "4625": 11.99055,
+ "4630": 12.01299,
+ "4635": 12.04423,
+ "4640": 12.03498,
+ "4645": 11.99636,
+ "4650": 12.01949,
+ "4655": 12.00476,
+ "4660": 12.19024,
+ "4665": 17.0313,
+ "4670": 12.08213,
+ "4675": 11.97263,
+ "4680": 12.14883,
+ "4685": 11.98722,
+ "4690": 11.95142,
+ "4695": 12.01482,
+ "4700": 11.99674,
+ "4705": 17.88216,
+ "4710": 12.02872,
+ "4715": 11.99452,
+ "4720": 11.98561,
+ "4725": 12.00923,
+ "4730": 11.98903,
+ "4735": 12.02491,
+ "4740": 12.05416,
+ "4745": 12.12314,
+ "4750": 11.97721,
+ "4755": 12.02318,
+ "4760": 11.99888,
+ "4765": 13.15443,
+ "4770": 11.98807,
+ "4775": 12.13821,
+ "4780": 12.23761,
+ "4785": 12.08535,
+ "4790": 12.02194,
+ "4795": 11.99974,
+ "4800": 12.01058,
+ "4805": 18.20909,
+ "4810": 22.19593,
+ "4815": 12.08767,
+ "4820": 11.99066,
+ "4825": 17.3692,
+ "4830": 11.96592,
+ "4835": 12.02465,
+ "4840": 12.15079,
+ "4845": 12.02288,
+ "4850": 11.99574,
+ "4855": 12.24097,
+ "4860": 12.06507,
+ "4865": 11.99956,
+ "4870": 11.93253,
+ "4875": 11.99562,
+ "4880": 11.97549,
+ "4885": 97.9965,
+ "4890": 23.26524,
+ "4895": 12.12904,
+ "4900": 12.02502,
+ "4905": 17.62521,
+ "4910": 12.109,
+ "4915": 12.30553,
+ "4920": 12.06756,
+ "4925": 12.08948,
+ "4930": 12.07576,
+ "4935": 12.09863,
+ "4940": 12.04853,
+ "4945": 23.09601,
+ "4950": 12.04599,
+ "4955": 12.14546,
+ "4960": 17.46752,
+ "4965": 12.10032,
+ "4970": 12.26463,
+ "4975": 12.05574,
+ "4980": 12.07291,
+ "4985": 12.09658,
+ "4990": 12.03323,
+ "4995": 12.05226,
+ "5000": 12.01128,
+ "5005": 23.51317,
+ "5010": 12.0753,
+ "5015": 14.76915,
+ "5020": 12.06683,
+ "5025": 12.12074,
+ "5030": 23.14332,
+ "5035": 12.11224,
+ "5040": 12.06147,
+ "5045": 12.06442,
+ "5050": 12.0675,
+ "5055": 12.14391,
+ "5060": 12.05162,
+ "5065": 12.0748,
+ "5070": 12.20809,
+ "5075": 12.13585,
+ "5080": 12.05242,
+ "5085": 12.17991,
+ "5090": 12.08148,
+ "5095": 12.12409,
+ "5100": 12.06104,
+ "5105": 12.11429,
+ "5110": 12.08759,
+ "5115": 12.03789,
+ "5120": 12.01043,
+ "5125": 12.10851,
+ "5130": 12.08331,
+ "5135": 12.07856,
+ "5140": 14.66916,
+ "5145": 12.12814,
+ "5150": 12.07523,
+ "5155": 12.07056,
+ "5160": 12.13564,
+ "5165": 12.3091,
+ "5170": 12.07991,
+ "5175": 17.97351,
+ "5180": 12.10067,
+ "5185": 13.0804,
+ "5190": 12.17774,
+ "5195": 12.3154,
+ "5200": 14.74282,
+ "5205": 12.1941,
+ "5210": 17.69639,
+ "5215": 12.14389,
+ "5220": 12.06835,
+ "5225": 12.53378,
+ "5230": 12.2152,
+ "5235": 12.08742,
+ "5240": 12.07842,
+ "5245": 12.09607,
+ "5250": 12.1703,
+ "5255": 12.11081,
+ "5260": 12.11889,
+ "5265": 12.04442,
+ "5270": 12.17308,
+ "5275": 12.10658,
+ "5280": 12.22118,
+ "5285": 12.06912,
+ "5290": 12.0757,
+ "5295": 12.23511,
+ "5300": 12.06881,
+ "5305": 12.06207,
+ "5310": 17.07406,
+ "5315": 17.43253,
+ "5320": 12.07731,
+ "5325": 12.08219,
+ "5330": 23.62977,
+ "5335": 12.09526,
+ "5340": 23.51612,
+ "5345": 12.04351,
+ "5350": 12.05806,
+ "5355": 12.1857,
+ "5360": 23.17471,
+ "5365": 12.04562,
+ "5370": 12.09198,
+ "5375": 12.22562,
+ "5380": 12.10305,
+ "5385": 12.08997,
+ "5390": 12.07223,
+ "5395": 16.24195,
+ "5400": 12.07489,
+ "5405": 12.22464,
+ "5410": 12.08809,
+ "5415": 12.08136,
+ "5420": 12.02882,
+ "5425": 12.18002,
+ "5430": 12.044,
+ "5435": 12.11552,
+ "5440": 12.08242,
+ "5445": 12.08994,
+ "5450": 12.1474,
+ "5455": 12.06427,
+ "5460": 12.02972,
+ "5465": 12.08186,
+ "5470": 12.14383,
+ "5475": 12.12484,
+ "5480": 12.18364,
+ "5485": 12.16456,
+ "5490": 12.09591,
+ "5495": 12.18212,
+ "5500": 12.08371,
+ "5505": 12.14,
+ "5510": 12.26758,
+ "5515": 12.02029,
+ "5520": 17.61228,
+ "5525": 12.11171,
+ "5530": 12.12853,
+ "5535": 12.2157,
+ "5540": 12.0447,
+ "5545": 12.05963,
+ "5550": 12.05468,
+ "5555": 12.02895,
+ "5560": 17.37934,
+ "5565": 12.11338,
+ "5570": 12.16494,
+ "5575": 12.03769,
+ "5580": 12.06056,
+ "5585": 12.10872,
+ "5590": 12.08438,
+ "5595": 12.05229,
+ "5600": 12.16298,
+ "5605": 12.08212,
+ "5610": 12.05598,
+ "5615": 12.09948,
+ "5620": 12.11926,
+ "5625": 12.06238,
+ "5630": 12.05684,
+ "5635": 12.31443,
+ "5640": 12.11188,
+ "5645": 12.35225,
+ "5650": 12.10406,
+ "5655": 12.015,
+ "5660": 12.09226,
+ "5665": 12.06425,
+ "5670": 12.10121,
+ "5675": 12.29176,
+ "5680": 12.1078,
+ "5685": 13.53345,
+ "5690": 12.06912,
+ "5695": 12.08094,
+ "5700": 12.06344,
+ "5705": 12.107,
+ "5710": 12.07118,
+ "5715": 12.06914,
+ "5720": 12.35322,
+ "5725": 12.10712,
+ "5730": 12.10294,
+ "5735": 12.22135,
+ "5740": 12.09168,
+ "5745": 12.06211,
+ "5750": 12.21443,
+ "5755": 12.02672,
+ "5760": 12.01408,
+ "5765": 12.0102,
+ "5770": 12.08365,
+ "5775": 12.08856,
+ "5780": 12.13163,
+ "5785": 12.06507,
+ "5790": 12.11272,
+ "5795": 12.07722,
+ "5800": 12.19872,
+ "5805": 12.06657,
+ "5810": 12.06403,
+ "5815": 12.16713,
+ "5820": 12.09628,
+ "5825": 12.18153,
+ "5830": 12.08826,
+ "5835": 12.02849,
+ "5840": 12.09496,
+ "5845": 12.02339,
+ "5850": 11.99955,
+ "5855": 12.00958,
+ "5860": 11.97281,
+ "5865": 17.50362,
+ "5870": 12.03277,
+ "5875": 12.04543,
+ "5880": 11.9768,
+ "5885": 17.3909,
+ "5890": 11.99813,
+ "5895": 12.18926,
+ "5900": 12.17929,
+ "5905": 17.52427,
+ "5910": 12.00519,
+ "5915": 17.1447,
+ "5920": 12.04834,
+ "5925": 12.05998,
+ "5930": 11.95898,
+ "5935": 11.98068,
+ "5940": 12.01225,
+ "5945": 12.05702,
+ "5950": 12.08394,
+ "5955": 11.99837,
+ "5960": 11.9934,
+ "5965": 11.9824,
+ "5970": 12.10284,
+ "5975": 12.17956,
+ "5980": 11.99432,
+ "5985": 11.99979,
+ "5990": 12.01079,
+ "5995": 12.03263,
+ "6000": 12.03332,
+ "6005": 12.03696,
+ "6010": 12.04994,
+ "6015": 12.00743,
+ "6020": 12.08843,
+ "6025": 21.42463,
+ "6030": 12.01631,
+ "6035": 12.0691,
+ "6040": 15.71875,
+ "6045": 16.71782,
+ "6050": 12.0158,
+ "6055": 22.5923,
+ "6060": 12.20213,
+ "6065": 12.04217,
+ "6070": 12.01592,
+ "6075": 12.05697,
+ "6080": 11.97775,
+ "6085": 12.00781,
+ "6090": 17.05471,
+ "6095": 11.92033,
+ "6100": 11.92768,
+ "6105": 12.21475,
+ "6110": 11.99354,
+ "6115": 12.09345,
+ "6120": 12.01281,
+ "6125": 11.99329,
+ "6130": 17.20973,
+ "6135": 12.1147,
+ "6140": 11.98698,
+ "6145": 12.03683,
+ "6150": 12.04441,
+ "6155": 12.02309,
+ "6160": 12.05495,
+ "6165": 12.05904,
+ "6170": 12.01332,
+ "6175": 12.02835,
+ "6180": 17.10044,
+ "6185": 12.09096,
+ "6190": 11.95051,
+ "6195": 12.00672,
+ "6200": 11.98207,
+ "6205": 12.03448,
+ "6210": 12.01109,
+ "6215": 12.0027,
+ "6220": 12.13659,
+ "6225": 12.00279,
+ "6230": 11.99114,
+ "6235": 11.99282,
+ "6240": 12.03338,
+ "6245": 11.99759,
+ "6250": 11.99598,
+ "6255": 18.27753,
+ "6260": 12.06263,
+ "6265": 12.23238,
+ "6270": 11.97802,
+ "6275": 12.01392,
+ "6280": 12.033,
+ "6285": 12.01391,
+ "6290": 11.97945,
+ "6295": 12.02986,
+ "6300": 13.06701,
+ "6305": 17.3811,
+ "6310": 11.98771,
+ "6315": 11.99254,
+ "6320": 12.10197,
+ "6325": 12.05157,
+ "6330": 12.0068,
+ "6335": 11.97679,
+ "6340": 12.01657,
+ "6345": 11.96766,
+ "6350": 12.05383,
+ "6355": 11.98311,
+ "6360": 12.06229,
+ "6365": 12.02586,
+ "6370": 12.00605,
+ "6375": 11.96454,
+ "6380": 12.06076,
+ "6385": 11.97515,
+ "6390": 12.00881,
+ "6395": 11.97208,
+ "6400": 11.96584,
+ "6405": 12.04544,
+ "6410": 12.04062,
+ "6415": 12.02412,
+ "6420": 11.98424,
+ "6425": 11.96525,
+ "6430": 11.96687,
+ "6435": 12.06708,
+ "6440": 17.19076,
+ "6445": 12.00984,
+ "6450": 11.94954,
+ "6455": 11.99452,
+ "6460": 11.98821,
+ "6465": 12.11336,
+ "6470": 12.0364,
+ "6475": 12.04094,
+ "6480": 12.0047,
+ "6485": 12.05982,
+ "6490": 11.99203,
+ "6495": 12.01851,
+ "6500": 12.02161,
+ "6505": 12.03322,
+ "6510": 11.9811,
+ "6515": 12.04216,
+ "6520": 11.99511,
+ "6525": 11.97702,
+ "6530": 11.97816,
+ "6535": 12.01245,
+ "6540": 11.98988,
+ "6545": 11.942,
+ "6550": 12.03309,
+ "6555": 12.16992,
+ "6560": 17.63231,
+ "6565": 12.02255,
+ "6570": 22.98051,
+ "6575": 11.98114,
+ "6580": 12.02207,
+ "6585": 11.95684,
+ "6590": 12.04226,
+ "6595": 11.99421,
+ "6600": 12.16134,
+ "6605": 11.97944,
+ "6610": 12.47764,
+ "6615": 12.00128,
+ "6620": 11.97768,
+ "6625": 12.12111,
+ "6630": 12.18783,
+ "6635": 11.99138,
+ "6640": 12.04762,
+ "6645": 12.0556,
+ "6650": 12.0104,
+ "6655": 12.03904,
+ "6660": 11.96358,
+ "6665": 11.95736,
+ "6670": 12.10759,
+ "6675": 12.03241,
+ "6680": 11.97862,
+ "6685": 11.99355,
+ "6690": 12.00157,
+ "6695": 12.0302,
+ "6700": 12.20154,
+ "6705": 12.13039,
+ "6710": 11.98275,
+ "6715": 12.01147,
+ "6720": 12.14224,
+ "6725": 12.0212,
+ "6730": 12.02564,
+ "6735": 11.98602,
+ "6740": 12.01507,
+ "6745": 12.03564,
+ "6750": 12.07665,
+ "6755": 12.02168,
+ "6760": 17.08851,
+ "6765": 12.1808,
+ "6770": 12.12242,
+ "6775": 12.05767,
+ "6780": 12.09579,
+ "6785": 17.36247,
+ "6790": 12.02822,
+ "6795": 29.96229,
+ "6800": 12.12225,
+ "6805": 15.04356,
+ "6810": 12.09575,
+ "6815": 12.19592,
+ "6820": 12.21685,
+ "6825": 12.31355,
+ "6830": 23.69474,
+ "6835": 12.17961,
+ "6840": 12.09243,
+ "6845": 12.05092,
+ "6850": 21.59167,
+ "6855": 24.13664,
+ "6860": 12.07989,
+ "6865": 28.51883,
+ "6870": 23.34192,
+ "6875": 12.23007,
+ "6880": 17.63662,
+ "6885": 12.19979,
+ "6890": 17.2875,
+ "6895": 12.16339,
+ "6900": 12.08369,
+ "6905": 12.07396,
+ "6910": 12.38012,
+ "6915": 12.02875,
+ "6920": 30.08166,
+ "6925": 12.07527,
+ "6930": 12.25225,
+ "6935": 12.06874,
+ "6940": 12.1616,
+ "6945": 12.12744,
+ "6950": 12.03937,
+ "6955": 12.0909,
+ "6960": 12.00088,
+ "6965": 12.13573,
+ "6970": 12.18782,
+ "6975": 12.11845,
+ "6980": 12.09853,
+ "6985": 12.11678,
+ "6990": 12.84109,
+ "6995": 12.08497,
+ "7000": 12.08809,
+ "7005": 12.14268,
+ "7010": 12.18868,
+ "7015": 12.16387,
+ "7020": 11.99399,
+ "7025": 12.02388,
+ "7030": 12.06144,
+ "7035": 12.16788,
+ "7040": 17.21614,
+ "7045": 11.99206,
+ "7050": 12.02574,
+ "7055": 12.06351,
+ "7060": 12.02326,
+ "7065": 12.09171,
+ "7070": 12.13569,
+ "7075": 12.01969,
+ "7080": 12.00548,
+ "7085": 18.24275,
+ "7090": 12.09344,
+ "7095": 12.10415,
+ "7100": 12.13474,
+ "7105": 12.09197,
+ "7110": 12.04749,
+ "7115": 12.03161,
+ "7120": 12.09078,
+ "7125": 17.35902,
+ "7130": 12.08913,
+ "7135": 17.30093,
+ "7140": 12.03829,
+ "7145": 12.05846,
+ "7150": 12.04047,
+ "7155": 12.18838,
+ "7160": 12.37793,
+ "7165": 22.90281,
+ "7170": 12.03698,
+ "7175": 12.29352,
+ "7180": 17.16954,
+ "7185": 12.05427,
+ "7190": 12.04494,
+ "7195": 14.63359,
+ "7200": 12.0596,
+ "7205": 12.06373,
+ "7210": 14.80632,
+ "7215": 12.02815,
+ "7220": 12.23693,
+ "7225": 11.99781,
+ "7230": 12.0368,
+ "7235": 12.11481,
+ "7240": 12.02721,
+ "7245": 17.36042,
+ "7250": 12.04156,
+ "7255": 12.13446,
+ "7260": 12.06813,
+ "7265": 12.05706,
+ "7270": 12.06285,
+ "7275": 12.19289,
+ "7280": 12.08874,
+ "7285": 23.91367,
+ "7290": 12.14143,
+ "7295": 12.06991,
+ "7300": 12.05525,
+ "7305": 12.18755,
+ "7310": 12.08263,
+ "7315": 12.10073,
+ "7320": 12.05481,
+ "7325": 12.14848,
+ "7330": 12.05782,
+ "7335": 12.15126,
+ "7340": 12.00938,
+ "7345": 12.18239,
+ "7350": 12.0334,
+ "7355": 12.04088,
+ "7360": 12.03703,
+ "7365": 12.34969,
+ "7370": 12.08689,
+ "7375": 12.09819,
+ "7380": 17.45507,
+ "7385": 12.09357,
+ "7390": 12.03953,
+ "7395": 12.04638,
+ "7400": 12.05157,
+ "7405": 22.79089,
+ "7410": 12.05014,
+ "7415": 12.181,
+ "7420": 12.14072,
+ "7425": 12.05131,
+ "7430": 12.04625,
+ "7435": 12.02017,
+ "7440": 12.05606,
+ "7445": 12.04471,
+ "7450": 12.0761,
+ "7455": 12.106,
+ "7460": 12.21437,
+ "7465": 12.03096,
+ "7470": 12.05216,
+ "7475": 12.03752,
+ "7480": 12.05708,
+ "7485": 12.20132,
+ "7490": 12.06037,
+ "7495": 12.07627,
+ "7500": 12.06468,
+ "7505": 12.60591,
+ "7510": 12.3498,
+ "7515": 12.12736,
+ "7520": 12.04266,
+ "7525": 12.02252,
+ "7530": 12.1098,
+ "7535": 16.7527,
+ "7540": 23.36724,
+ "7545": 12.03414,
+ "7550": 12.0897,
+ "7555": 12.07102,
+ "7560": 12.14587,
+ "7565": 12.18434,
+ "7570": 12.31327,
+ "7575": 12.04723,
+ "7580": 12.08119,
+ "7585": 12.1798,
+ "7590": 12.09736,
+ "7595": 12.19338,
+ "7600": 12.0789,
+ "7605": 12.05718,
+ "7610": 12.37752,
+ "7615": 12.16375,
+ "7620": 12.09816,
+ "7625": 12.0866,
+ "7630": 12.05009,
+ "7635": 23.13086,
+ "7640": 12.01961,
+ "7645": 12.06554,
+ "7650": 12.04623,
+ "7655": 12.1454,
+ "7660": 12.05502,
+ "7665": 12.01544,
+ "7670": 12.08461,
+ "7675": 12.05123,
+ "7680": 12.13788,
+ "7685": 12.02765,
+ "7690": 12.3752,
+ "7695": 17.25783,
+ "7700": 17.30564,
+ "7705": 17.26243,
+ "7710": 12.16804,
+ "7715": 12.06507,
+ "7720": 12.44663,
+ "7725": 12.00612,
+ "7730": 12.05696,
+ "7735": 12.03341,
+ "7740": 12.05255,
+ "7745": 12.19668,
+ "7750": 12.0311,
+ "7755": 12.1959,
+ "7760": 12.34561,
+ "7765": 12.0433,
+ "7770": 12.04335,
+ "7775": 12.04626,
+ "7780": 12.04451,
+ "7785": 12.06191,
+ "7790": 12.04573,
+ "7795": 12.96681,
+ "7800": 12.15326,
+ "7805": 12.01198,
+ "7810": 12.0927,
+ "7815": 12.0677,
+ "7820": 17.31202,
+ "7825": 12.25985,
+ "7830": 14.92717,
+ "7835": 12.00867,
+ "7840": 12.18663,
+ "7845": 12.1089,
+ "7850": 12.13711,
+ "7855": 12.03681,
+ "7860": 12.05064,
+ "7865": 12.02452,
+ "7870": 12.09259,
+ "7875": 12.04967,
+ "7880": 12.05208,
+ "7885": 12.03451,
+ "7890": 23.48879,
+ "7895": 12.04823,
+ "7900": 12.12262,
+ "7905": 12.00179,
+ "7910": 12.1176,
+ "7915": 23.13016,
+ "7920": 12.02581,
+ "7925": 12.07102,
+ "7930": 17.05358,
+ "7935": 17.58841,
+ "7940": 12.06211,
+ "7945": 12.00683,
+ "7950": 17.1904,
+ "7955": 12.05676,
+ "7960": 12.03249,
+ "7965": 12.05017,
+ "7970": 12.06876,
+ "7975": 12.11739,
+ "7980": 12.24979,
+ "7985": 12.04217,
+ "7990": 12.08736,
+ "7995": 23.59298,
+ "8000": 17.13168,
+ "8005": 12.11449,
+ "8010": 12.07845,
+ "8015": 12.12722,
+ "8020": 12.03253,
+ "8025": 12.16462,
+ "8030": 19.20207,
+ "8035": 12.06669,
+ "8040": 12.27918,
+ "8045": 12.02057,
+ "8050": 12.01148,
+ "8055": 12.05251,
+ "8060": 12.04089,
+ "8065": 12.03428,
+ "8070": 12.11348,
+ "8075": 12.02102,
+ "8080": 12.05754,
+ "8085": 12.13052,
+ "8090": 12.11185,
+ "8095": 12.04883,
+ "8100": 12.04197,
+ "8105": 12.1116,
+ "8110": 12.1138,
+ "8115": 12.21439,
+ "8120": 12.12073,
+ "8125": 12.31612,
+ "8130": 12.02632,
+ "8135": 12.06499,
+ "8140": 12.01099,
+ "8145": 12.11364,
+ "8150": 12.05281,
+ "8155": 12.07404,
+ "8160": 12.05686,
+ "8165": 16.5834,
+ "8170": 14.75554,
+ "8175": 13.08679,
+ "8180": 12.02005,
+ "8185": 12.22639,
+ "8190": 12.02017,
+ "8195": 12.02664,
+ "8200": 12.05579,
+ "8205": 12.25489,
+ "8210": 12.09628,
+ "8215": 17.17466,
+ "8220": 12.07921,
+ "8225": 12.06167,
+ "8230": 28.92289,
+ "8235": 12.03033,
+ "8240": 12.14244,
+ "8245": 12.04458,
+ "8250": 11.99029,
+ "8255": 12.01024,
+ "8260": 12.03846,
+ "8265": 12.05126,
+ "8270": 12.0203,
+ "8275": 12.03348,
+ "8280": 12.07653,
+ "8285": 12.03568,
+ "8290": 12.09785,
+ "8295": 12.10526,
+ "8300": 12.0189,
+ "8305": 12.20012,
+ "8310": 12.10337,
+ "8315": 11.98774,
+ "8320": 12.06816,
+ "8325": 11.98596,
+ "8330": 12.10245,
+ "8335": 12.07995,
+ "8340": 12.02466,
+ "8345": 12.02496,
+ "8350": 12.13616,
+ "8355": 12.02438,
+ "8360": 12.15416,
+ "8365": 12.03624,
+ "8370": 12.03615,
+ "8375": 12.07507,
+ "8380": 15.65409,
+ "8385": 12.02395,
+ "8390": 12.05566,
+ "8395": 12.0894,
+ "8400": 12.14548,
+ "8405": 12.06457,
+ "8410": 17.69603,
+ "8415": 12.06352,
+ "8420": 11.99664,
+ "8425": 12.01922,
+ "8430": 12.03946,
+ "8435": 12.05745,
+ "8440": 11.97788,
+ "8445": 12.0783,
+ "8450": 12.07747,
+ "8455": 12.02284,
+ "8460": 12.11115,
+ "8465": 12.056,
+ "8470": 12.09155,
+ "8475": 12.0575,
+ "8480": 12.04801,
+ "8485": 17.92816,
+ "8490": 12.11083,
+ "8495": 11.99211,
+ "8500": 12.19551,
+ "8505": 12.09547,
+ "8510": 12.16992,
+ "8515": 12.03736,
+ "8520": 17.40644,
+ "8525": 12.02834,
+ "8530": 12.02607,
+ "8535": 12.13724,
+ "8540": 12.07937,
+ "8545": 12.04481,
+ "8550": 12.22286,
+ "8555": 12.0389,
+ "8560": 12.03043,
+ "8565": 12.01456,
+ "8570": 12.06495,
+ "8575": 12.0593,
+ "8580": 12.0565,
+ "8585": 12.07024,
+ "8590": 12.10454,
+ "8595": 12.10681,
+ "8600": 23.33078,
+ "8605": 12.0833,
+ "8610": 12.05853,
+ "8615": 12.13845,
+ "8620": 12.07564,
+ "8625": 12.0518,
+ "8630": 12.22501,
+ "8635": 17.70349,
+ "8640": 12.0043,
+ "8645": 12.01351,
+ "8650": 11.9908,
+ "8655": 11.947,
+ "8660": 17.25628,
+ "8665": 11.99525,
+ "8670": 36.50058,
+ "8675": 11.96641,
+ "8680": 11.98381,
+ "8685": 11.9218,
+ "8690": 11.97541,
+ "8695": 12.03319,
+ "8700": 11.96755,
+ "8705": 11.94245,
+ "8710": 12.09415,
+ "8715": 12.01533,
+ "8720": 12.19038,
+ "8725": 11.93475,
+ "8730": 11.99852,
+ "8735": 12.03763,
+ "8740": 11.97101,
+ "8745": 11.98139,
+ "8750": 11.94178,
+ "8755": 11.99606,
+ "8760": 11.95502,
+ "8765": 11.96191,
+ "8770": 11.95313,
+ "8775": 12.00885,
+ "8780": 17.56132,
+ "8785": 11.97991,
+ "8790": 11.9127,
+ "8795": 11.98754,
+ "8800": 17.15594,
+ "8805": 11.95331,
+ "8810": 12.07362,
+ "8815": 11.98162,
+ "8820": 12.12152,
+ "8825": 17.09778,
+ "8830": 11.96095,
+ "8835": 11.9517,
+ "8840": 11.93945,
+ "8845": 11.96732,
+ "8850": 11.97836,
+ "8855": 12.07682,
+ "8860": 12.01426,
+ "8865": 14.60983,
+ "8870": 12.25702,
+ "8875": 11.97609,
+ "8880": 12.0166,
+ "8885": 17.4162,
+ "8890": 12.00248,
+ "8895": 17.49548,
+ "8900": 12.02987,
+ "8905": 17.54209,
+ "8910": 37.03334,
+ "8915": 11.99512,
+ "8920": 11.95539,
+ "8925": 11.98665,
+ "8930": 12.04025,
+ "8935": 11.98089,
+ "8940": 12.00516,
+ "8945": 17.12887,
+ "8950": 11.99412,
+ "8955": 12.04596,
+ "8960": 12.07074,
+ "8965": 12.17298,
+ "8970": 12.08316,
+ "8975": 12.02459,
+ "8980": 11.98741,
+ "8985": 12.02188,
+ "8990": 11.9376,
+ "8995": 22.56186,
+ "9000": 11.96505,
+ "9005": 11.93682,
+ "9010": 22.87924,
+ "9015": 12.07902,
+ "9020": 11.97164,
+ "9025": 17.26785,
+ "9030": 11.95046,
+ "9035": 11.94659,
+ "9040": 11.96095,
+ "9045": 11.97762,
+ "9050": 17.16623,
+ "9055": 11.99601,
+ "9060": 12.23872,
+ "9065": 12.04999,
+ "9070": 11.9628,
+ "9075": 11.94753,
+ "9080": 11.97356,
+ "9085": 11.9595,
+ "9090": 12.54836,
+ "9095": 11.98133,
+ "9100": 11.94348,
+ "9105": 12.06208,
+ "9110": 11.97089,
+ "9115": 23.88639,
+ "9120": 11.99678,
+ "9125": 12.07445,
+ "9130": 12.02274,
+ "9135": 12.21247,
+ "9140": 12.04684,
+ "9145": 11.94849,
+ "9150": 11.97013,
+ "9155": 11.97153,
+ "9160": 11.91088,
+ "9165": 12.04907,
+ "9170": 11.97104,
+ "9175": 11.93072,
+ "9180": 11.97585,
+ "9185": 11.94208,
+ "9190": 11.93762,
+ "9195": 11.98367,
+ "9200": 11.95356,
+ "9205": 12.02104,
+ "9210": 11.94425,
+ "9215": 11.93833,
+ "9220": 12.05504,
+ "9225": 12.28704,
+ "9230": 11.96364,
+ "9235": 11.99507,
+ "9240": 11.99655,
+ "9245": 11.95781,
+ "9250": 11.9708,
+ "9255": 11.97461,
+ "9260": 12.00571,
+ "9265": 11.98747,
+ "9270": 12.12105,
+ "9275": 23.40051,
+ "9280": 11.95001,
+ "9285": 11.98055,
+ "9290": 11.94982,
+ "9295": 11.95502,
+ "9300": 12.14843,
+ "9305": 11.98163,
+ "9310": 13.32581,
+ "9315": 11.963,
+ "9320": 11.99484,
+ "9325": 16.49577,
+ "9330": 11.91038,
+ "9335": 11.99314,
+ "9340": 11.91403,
+ "9345": 11.99311,
+ "9350": 12.02235,
+ "9355": 11.97251,
+ "9360": 23.23406,
+ "9365": 12.01839,
+ "9370": 11.99082,
+ "9375": 11.99109,
+ "9380": 12.01016,
+ "9385": 11.99889,
+ "9390": 12.03772,
+ "9395": 11.97163,
+ "9400": 11.99777,
+ "9405": 11.99525,
+ "9410": 12.04326,
+ "9415": 11.99924,
+ "9420": 11.94885,
+ "9425": 11.97809,
+ "9430": 11.98696,
+ "9435": 11.96284,
+ "9440": 17.28831,
+ "9445": 11.97503,
+ "9450": 11.97866,
+ "9455": 11.95827,
+ "9460": 11.9416,
+ "9465": 11.98559,
+ "9470": 12.05668,
+ "9475": 12.05906,
+ "9480": 12.04568,
+ "9485": 16.40843,
+ "9490": 12.12434,
+ "9495": 11.97472,
+ "9500": 12.08529,
+ "9505": 12.04891,
+ "9510": 11.95289,
+ "9515": 11.94648,
+ "9520": 11.9444,
+ "9525": 12.00067,
+ "9530": 11.95863,
+ "9535": 11.9676
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release_sm/model_config.yaml b/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release_sm/model_config.yaml
index cc8f2b814c2..49cca71a596 100644
--- a/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release_sm/model_config.yaml
+++ b/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_release_sm/model_config.yaml
@@ -19,7 +19,7 @@ MODEL_ARGS:
--distributed-timeout-minutes: 60
--tensor-model-parallel-size: 2
--pipeline-model-parallel-size: 4
- --pipeline-model-parallel-layout: Et*2\\|\\(tt\\|\\)*5t\\|tmL # Et*2|(tt|)*5t|tmL
+ --pipeline-model-parallel-layout: Et*2|(tt|)*5t|tmL
--expert-model-parallel-size: 16
--context-parallel-size: 1
--expert-tensor-parallel-size: 1
@@ -132,8 +132,8 @@ MODEL_ARGS:
# Add checkpointing args
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --save-interval: 500
- --save-retain-interval: 10000
+ --save-interval: 250
+ --save-retain-interval: 5000
--dist-ckpt-strictness: log_all
# Add initialization args
diff --git a/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_gb_200_release/model_config.yaml b/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp2pp2emp16etp1cp1_gb_200_release/model_config.yaml
similarity index 93%
rename from tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_gb_200_release/model_config.yaml
rename to tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp2pp2emp16etp1cp1_gb_200_release/model_config.yaml
index ced409e5b1e..4452ad22987 100644
--- a/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_gb_200_release/model_config.yaml
+++ b/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp2pp2emp16etp1cp1_gb_200_release/model_config.yaml
@@ -13,16 +13,16 @@ ENV_VARS:
NON_DETERMINSTIC_RESULTS: 1
NVSHMEM_IB_ENABLE_IBGDA: 0
CUDA_DEVICE_MAX_CONNECTIONS: 1
- NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN: 16
+ NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN: 8
USE_MNNVL: 1
TEST_TYPE: "release"
MODEL_ARGS:
# Distributed args
--distributed-timeout-minutes: 60
--tensor-model-parallel-size: 2
- --pipeline-model-parallel-size: 4
- --pipeline-model-parallel-layout: Et*2\\|\\(tt\\|\\)*5t\\|tmL # Et*2|(tt|)*5t|tmL
- --expert-model-parallel-size: 16
+ --pipeline-model-parallel-size: 2
+ --pipeline-model-parallel-layout: "Et*7|t*7mL"
+ --expert-model-parallel-size: 8
--context-parallel-size: 1
--expert-tensor-parallel-size: 1
--use-distributed-optimizer: true
@@ -34,7 +34,7 @@ MODEL_ARGS:
--sequence-parallel: true
--use-flash-attn: true
--disable-bias-linear: true
- --micro-batch-size: 1
+ --micro-batch-size: 4
--global-batch-size: 512
--train-samples: 24414062
--exit-duration-in-mins: 220
@@ -134,8 +134,8 @@ MODEL_ARGS:
# Add checkpointing args
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --save-interval: 500
- --save-retain-interval: 10000
+ --save-interval: 250
+ --save-retain-interval: 5000
--dist-ckpt-strictness: log_all
# Add initialization args
diff --git a/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp2pp2emp16etp1cp1_gb_200_release_sm/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp2pp2emp16etp1cp1_gb_200_release_sm/golden_values_dev_dgx_gb200.json
new file mode 100644
index 00000000000..f1fe2600baf
--- /dev/null
+++ b/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp2pp2emp16etp1cp1_gb_200_release_sm/golden_values_dev_dgx_gb200.json
@@ -0,0 +1,11492 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 9535,
+ "step_interval": 5,
+ "values": {
+ "1": 13.90137,
+ "5": 13.90372,
+ "10": 13.87493,
+ "15": 13.86127,
+ "20": 13.76245,
+ "25": 13.72657,
+ "30": 13.42477,
+ "35": 13.35267,
+ "40": 13.2701,
+ "45": 13.15966,
+ "50": 12.56281,
+ "55": 12.38218,
+ "60": 12.19755,
+ "65": 12.03141,
+ "70": 11.8598,
+ "75": 11.55717,
+ "80": 11.35361,
+ "85": 11.13845,
+ "90": 10.99346,
+ "95": 10.83936,
+ "100": 10.62267,
+ "105": 10.49232,
+ "110": 10.28024,
+ "115": 10.06877,
+ "120": 9.91657,
+ "125": 9.7739,
+ "130": 9.68549,
+ "135": 9.60673,
+ "140": 9.37968,
+ "145": 9.37166,
+ "150": 9.21319,
+ "155": 9.13417,
+ "160": 9.05577,
+ "165": 8.9384,
+ "170": 8.88972,
+ "175": 8.84479,
+ "180": 8.70299,
+ "185": 8.74117,
+ "190": 8.62106,
+ "195": 8.6257,
+ "200": 8.50992,
+ "205": 8.41997,
+ "210": 8.38159,
+ "215": 8.43073,
+ "220": 8.30381,
+ "225": 8.31599,
+ "230": 8.29915,
+ "235": 8.21696,
+ "240": 8.17383,
+ "245": 8.15279,
+ "250": 8.09205,
+ "255": 8.10103,
+ "260": 7.99771,
+ "265": 7.97892,
+ "270": 7.93198,
+ "275": 7.9201,
+ "280": 7.90955,
+ "285": 7.926,
+ "290": 7.86689,
+ "295": 7.8574,
+ "300": 7.75179,
+ "305": 7.74637,
+ "310": 7.70996,
+ "315": 7.70202,
+ "320": 7.69073,
+ "325": 7.60867,
+ "330": 7.59095,
+ "335": 7.57078,
+ "340": 7.6124,
+ "345": 7.48272,
+ "350": 7.47903,
+ "355": 7.41296,
+ "360": 7.49829,
+ "365": 7.42684,
+ "370": 7.45503,
+ "375": 7.4001,
+ "380": 7.37778,
+ "385": 7.37052,
+ "390": 7.39433,
+ "395": 7.33554,
+ "400": 7.27326,
+ "405": 7.28356,
+ "410": 7.2746,
+ "415": 7.25926,
+ "420": 7.26048,
+ "425": 7.23108,
+ "430": 7.18837,
+ "435": 7.19758,
+ "440": 7.16962,
+ "445": 7.15083,
+ "450": 7.10821,
+ "455": 7.12543,
+ "460": 7.09749,
+ "465": 7.08656,
+ "470": 7.05848,
+ "475": 7.08213,
+ "480": 6.95787,
+ "485": 7.01061,
+ "490": 6.96987,
+ "495": 6.95326,
+ "500": 6.89432,
+ "505": 6.93202,
+ "510": 6.91267,
+ "515": 6.88028,
+ "520": 6.87422,
+ "525": 6.86726,
+ "530": 6.8739,
+ "535": 6.86162,
+ "540": 6.78195,
+ "545": 6.81605,
+ "550": 6.83358,
+ "555": 6.85906,
+ "560": 6.80601,
+ "565": 6.74155,
+ "570": 6.75731,
+ "575": 6.77515,
+ "580": 6.69339,
+ "585": 6.70836,
+ "590": 6.6529,
+ "595": 6.65016,
+ "600": 6.67691,
+ "605": 6.67077,
+ "610": 6.64276,
+ "615": 6.69006,
+ "620": 6.6126,
+ "625": 6.58356,
+ "630": 6.58014,
+ "635": 6.6137,
+ "640": 6.59842,
+ "645": 6.58442,
+ "650": 6.6319,
+ "655": 6.62339,
+ "660": 6.53565,
+ "665": 6.51871,
+ "670": 6.46644,
+ "675": 6.57188,
+ "680": 6.54813,
+ "685": 6.49667,
+ "690": 6.48079,
+ "695": 6.44004,
+ "700": 6.44077,
+ "705": 6.43808,
+ "710": 6.47217,
+ "715": 6.48221,
+ "720": 6.3619,
+ "725": 6.41446,
+ "730": 6.40304,
+ "735": 6.41978,
+ "740": 6.3585,
+ "745": 6.32344,
+ "750": 6.38272,
+ "755": 6.30559,
+ "760": 6.31487,
+ "765": 6.31842,
+ "770": 6.32395,
+ "775": 6.3117,
+ "780": 6.27909,
+ "785": 6.29453,
+ "790": 6.25916,
+ "795": 6.23903,
+ "800": 6.24073,
+ "805": 6.30465,
+ "810": 6.1748,
+ "815": 6.19424,
+ "820": 6.20679,
+ "825": 6.22529,
+ "830": 6.22813,
+ "835": 6.1816,
+ "840": 6.15272,
+ "845": 6.19471,
+ "850": 6.13931,
+ "855": 6.16414,
+ "860": 6.1429,
+ "865": 6.16645,
+ "870": 6.12808,
+ "875": 6.16223,
+ "880": 6.12188,
+ "885": 6.11017,
+ "890": 6.16856,
+ "895": 6.06075,
+ "900": 6.06927,
+ "905": 6.08578,
+ "910": 6.06356,
+ "915": 6.04564,
+ "920": 6.03883,
+ "925": 6.03113,
+ "930": 6.05763,
+ "935": 6.05358,
+ "940": 5.98767,
+ "945": 6.03153,
+ "950": 6.0508,
+ "955": 6.01323,
+ "960": 6.01332,
+ "965": 5.92481,
+ "970": 5.96415,
+ "975": 5.96275,
+ "980": 5.94303,
+ "985": 5.93704,
+ "990": 5.9842,
+ "995": 5.89646,
+ "1000": 5.91884,
+ "1005": 5.87096,
+ "1010": 5.90555,
+ "1015": 5.93435,
+ "1020": 5.84891,
+ "1025": 5.83642,
+ "1030": 5.84465,
+ "1035": 5.9408,
+ "1040": 5.8555,
+ "1045": 5.83493,
+ "1050": 5.87466,
+ "1055": 5.85467,
+ "1060": 5.80325,
+ "1065": 5.78991,
+ "1070": 5.82227,
+ "1075": 5.8278,
+ "1080": 5.81056,
+ "1085": 5.81898,
+ "1090": 5.78662,
+ "1095": 5.80508,
+ "1100": 5.77297,
+ "1105": 5.73856,
+ "1110": 5.80972,
+ "1115": 5.73375,
+ "1120": 5.67813,
+ "1125": 5.694,
+ "1130": 5.74772,
+ "1135": 5.69944,
+ "1140": 5.70346,
+ "1145": 5.68783,
+ "1150": 5.71594,
+ "1155": 5.67149,
+ "1160": 5.66355,
+ "1165": 5.70291,
+ "1170": 5.68566,
+ "1175": 5.64336,
+ "1180": 5.64691,
+ "1185": 5.64423,
+ "1190": 5.63038,
+ "1195": 5.61933,
+ "1200": 5.57333,
+ "1205": 5.67994,
+ "1210": 5.54602,
+ "1215": 5.57286,
+ "1220": 5.65301,
+ "1225": 5.54112,
+ "1230": 5.5906,
+ "1235": 5.54623,
+ "1240": 5.57679,
+ "1245": 5.55389,
+ "1250": 5.53673,
+ "1255": 5.52802,
+ "1260": 5.53033,
+ "1265": 5.50506,
+ "1270": 5.47121,
+ "1275": 5.55119,
+ "1280": 5.48351,
+ "1285": 5.49578,
+ "1290": 5.46593,
+ "1295": 5.49756,
+ "1300": 5.4908,
+ "1305": 5.45855,
+ "1310": 5.408,
+ "1315": 5.47893,
+ "1320": 5.47433,
+ "1325": 5.39507,
+ "1330": 5.44739,
+ "1335": 5.42457,
+ "1340": 5.47764,
+ "1345": 5.43071,
+ "1350": 5.39948,
+ "1355": 5.404,
+ "1360": 5.40898,
+ "1365": 5.41188,
+ "1370": 5.34703,
+ "1375": 5.36678,
+ "1380": 5.41495,
+ "1385": 5.36851,
+ "1390": 5.3643,
+ "1395": 5.3745,
+ "1400": 5.37267,
+ "1405": 5.34998,
+ "1410": 5.33451,
+ "1415": 5.29482,
+ "1420": 5.3412,
+ "1425": 5.33001,
+ "1430": 5.36652,
+ "1435": 5.28266,
+ "1440": 5.30995,
+ "1445": 5.33855,
+ "1450": 5.34363,
+ "1455": 5.34312,
+ "1460": 5.29863,
+ "1465": 5.29333,
+ "1470": 5.32613,
+ "1475": 5.30183,
+ "1480": 5.30943,
+ "1485": 5.25949,
+ "1490": 5.25161,
+ "1495": 5.27493,
+ "1500": 5.27035,
+ "1505": 5.23318,
+ "1510": 5.26159,
+ "1515": 5.18754,
+ "1520": 5.22128,
+ "1525": 5.17966,
+ "1530": 5.21407,
+ "1535": 5.19264,
+ "1540": 5.19387,
+ "1545": 5.22973,
+ "1550": 5.23495,
+ "1555": 5.22091,
+ "1560": 5.16393,
+ "1565": 5.19626,
+ "1570": 5.20183,
+ "1575": 5.15886,
+ "1580": 5.18994,
+ "1585": 5.17031,
+ "1590": 5.15178,
+ "1595": 5.12216,
+ "1600": 5.19866,
+ "1605": 5.14247,
+ "1610": 5.14031,
+ "1615": 5.12474,
+ "1620": 5.13993,
+ "1625": 5.13053,
+ "1630": 5.10579,
+ "1635": 5.16153,
+ "1640": 5.12478,
+ "1645": 5.12378,
+ "1650": 5.10405,
+ "1655": 5.09131,
+ "1660": 5.09538,
+ "1665": 5.08157,
+ "1670": 5.09659,
+ "1675": 5.08729,
+ "1680": 5.03678,
+ "1685": 5.04267,
+ "1690": 5.01898,
+ "1695": 5.02161,
+ "1700": 5.06252,
+ "1705": 5.04299,
+ "1710": 5.03416,
+ "1715": 5.00243,
+ "1720": 4.99343,
+ "1725": 5.0245,
+ "1730": 4.98024,
+ "1735": 5.05638,
+ "1740": 4.97717,
+ "1745": 4.99984,
+ "1750": 4.99075,
+ "1755": 5.00278,
+ "1760": 5.00583,
+ "1765": 4.95727,
+ "1770": 4.96699,
+ "1775": 4.96574,
+ "1780": 5.0002,
+ "1785": 4.94378,
+ "1790": 4.96979,
+ "1795": 4.96585,
+ "1800": 4.90851,
+ "1805": 4.91094,
+ "1810": 4.92761,
+ "1815": 4.92918,
+ "1820": 4.91049,
+ "1825": 4.91949,
+ "1830": 4.90284,
+ "1835": 4.89652,
+ "1840": 4.89271,
+ "1845": 4.89107,
+ "1850": 4.86048,
+ "1855": 4.91771,
+ "1860": 4.87121,
+ "1865": 4.87755,
+ "1870": 4.86127,
+ "1875": 4.8634,
+ "1880": 4.92077,
+ "1885": 4.8676,
+ "1890": 4.86072,
+ "1895": 4.80712,
+ "1900": 4.85143,
+ "1905": 4.83773,
+ "1910": 4.85403,
+ "1915": 4.81865,
+ "1920": 4.80757,
+ "1925": 4.82107,
+ "1930": 4.79955,
+ "1935": 4.82364,
+ "1940": 4.78175,
+ "1945": 4.83198,
+ "1950": 4.86607,
+ "1955": 4.80403,
+ "1960": 4.79552,
+ "1965": 4.74955,
+ "1970": 4.75363,
+ "1975": 4.81745,
+ "1980": 4.74999,
+ "1985": 4.76769,
+ "1990": 4.80916,
+ "1995": 4.77382,
+ "2000": 4.78496,
+ "2005": 4.82884,
+ "2010": 4.74486,
+ "2015": 4.72042,
+ "2020": 4.72995,
+ "2025": 4.78311,
+ "2030": 4.70716,
+ "2035": 4.72882,
+ "2040": 4.69887,
+ "2045": 4.78539,
+ "2050": 4.76107,
+ "2055": 4.73212,
+ "2060": 4.73501,
+ "2065": 4.69643,
+ "2070": 4.71098,
+ "2075": 4.72907,
+ "2080": 4.69636,
+ "2085": 4.72163,
+ "2090": 4.63695,
+ "2095": 4.67168,
+ "2100": 4.6437,
+ "2105": 4.67007,
+ "2110": 4.66872,
+ "2115": 4.67385,
+ "2120": 4.66823,
+ "2125": 4.63885,
+ "2130": 4.63932,
+ "2135": 4.64717,
+ "2140": 4.65791,
+ "2145": 4.60594,
+ "2150": 4.63571,
+ "2155": 4.60776,
+ "2160": 4.63951,
+ "2165": 4.61005,
+ "2170": 4.63355,
+ "2175": 4.62935,
+ "2180": 4.6183,
+ "2185": 4.62781,
+ "2190": 4.60223,
+ "2195": 4.58434,
+ "2200": 4.57583,
+ "2205": 4.59158,
+ "2210": 4.63259,
+ "2215": 4.66719,
+ "2220": 4.61838,
+ "2225": 4.60454,
+ "2230": 4.60052,
+ "2235": 4.64699,
+ "2240": 4.5389,
+ "2245": 4.5358,
+ "2250": 4.54769,
+ "2255": 4.57068,
+ "2260": 4.5079,
+ "2265": 4.58371,
+ "2270": 4.51942,
+ "2275": 4.57992,
+ "2280": 4.53485,
+ "2285": 4.55578,
+ "2290": 4.5521,
+ "2295": 4.55767,
+ "2300": 4.55649,
+ "2305": 4.5144,
+ "2310": 4.5602,
+ "2315": 4.49582,
+ "2320": 4.54432,
+ "2325": 4.52196,
+ "2330": 4.51179,
+ "2335": 4.50424,
+ "2340": 4.5078,
+ "2345": 4.55039,
+ "2350": 4.488,
+ "2355": 4.49708,
+ "2360": 4.46327,
+ "2365": 4.47306,
+ "2370": 4.46522,
+ "2375": 4.46926,
+ "2380": 4.41939,
+ "2385": 4.45717,
+ "2390": 4.45953,
+ "2395": 4.4885,
+ "2400": 4.44936,
+ "2405": 4.42913,
+ "2410": 4.48082,
+ "2415": 4.44601,
+ "2420": 4.44905,
+ "2425": 4.42209,
+ "2430": 4.44411,
+ "2435": 4.42794,
+ "2440": 4.41977,
+ "2445": 4.43825,
+ "2450": 4.40971,
+ "2455": 4.44087,
+ "2460": 4.38721,
+ "2465": 4.43587,
+ "2470": 4.42801,
+ "2475": 4.44621,
+ "2480": 4.36757,
+ "2485": 4.39991,
+ "2490": 4.40413,
+ "2495": 4.38698,
+ "2500": 4.38565,
+ "2505": 4.39788,
+ "2510": 4.4417,
+ "2515": 4.42702,
+ "2520": 4.37302,
+ "2525": 4.38271,
+ "2530": 4.38895,
+ "2535": 4.39339,
+ "2540": 4.39043,
+ "2545": 4.40084,
+ "2550": 4.32859,
+ "2555": 4.39368,
+ "2560": 4.3748,
+ "2565": 4.3265,
+ "2570": 4.35907,
+ "2575": 4.33551,
+ "2580": 4.32803,
+ "2585": 4.31203,
+ "2590": 4.33923,
+ "2595": 4.31022,
+ "2600": 4.32499,
+ "2605": 4.33839,
+ "2610": 4.34892,
+ "2615": 4.3062,
+ "2620": 4.29137,
+ "2625": 4.33041,
+ "2630": 4.24989,
+ "2635": 4.335,
+ "2640": 4.32539,
+ "2645": 4.28815,
+ "2650": 4.31108,
+ "2655": 4.29491,
+ "2660": 4.24347,
+ "2665": 4.32749,
+ "2670": 4.29209,
+ "2675": 4.25567,
+ "2680": 4.28692,
+ "2685": 4.27892,
+ "2690": 4.26203,
+ "2695": 4.30913,
+ "2700": 4.22083,
+ "2705": 4.25889,
+ "2710": 4.28047,
+ "2715": 4.26443,
+ "2720": 4.26735,
+ "2725": 4.24557,
+ "2730": 4.25998,
+ "2735": 4.24841,
+ "2740": 4.22573,
+ "2745": 4.2091,
+ "2750": 4.24197,
+ "2755": 4.24754,
+ "2760": 4.25496,
+ "2765": 4.20886,
+ "2770": 4.26027,
+ "2775": 4.20286,
+ "2780": 4.23219,
+ "2785": 4.2193,
+ "2790": 4.24126,
+ "2795": 4.21322,
+ "2800": 4.14368,
+ "2805": 4.19319,
+ "2810": 4.18833,
+ "2815": 4.18403,
+ "2820": 4.22677,
+ "2825": 4.22003,
+ "2830": 4.19596,
+ "2835": 4.19978,
+ "2840": 4.19183,
+ "2845": 4.16832,
+ "2850": 4.19143,
+ "2855": 4.14369,
+ "2860": 4.17442,
+ "2865": 4.19958,
+ "2870": 4.1678,
+ "2875": 4.19024,
+ "2880": 4.11752,
+ "2885": 4.16723,
+ "2890": 4.14669,
+ "2895": 4.17849,
+ "2900": 4.12196,
+ "2905": 4.13837,
+ "2910": 4.13738,
+ "2915": 4.17375,
+ "2920": 4.15804,
+ "2925": 4.13102,
+ "2930": 4.10617,
+ "2935": 4.11201,
+ "2940": 4.12777,
+ "2945": 4.09246,
+ "2950": 4.05759,
+ "2955": 4.06767,
+ "2960": 4.07572,
+ "2965": 4.08835,
+ "2970": 4.10165,
+ "2975": 4.12102,
+ "2980": 4.06059,
+ "2985": 4.09503,
+ "2990": 4.12166,
+ "2995": 4.05877,
+ "3000": 4.07318,
+ "3005": 4.05398,
+ "3010": 4.08755,
+ "3015": 4.0513,
+ "3020": 4.06118,
+ "3025": 4.05242,
+ "3030": 4.08085,
+ "3035": 4.06503,
+ "3040": 4.08001,
+ "3045": 4.07026,
+ "3050": 4.04743,
+ "3055": 4.03433,
+ "3060": 4.02275,
+ "3065": 4.05662,
+ "3070": 4.0636,
+ "3075": 4.00951,
+ "3080": 4.03438,
+ "3085": 4.03236,
+ "3090": 4.03723,
+ "3095": 4.05431,
+ "3100": 4.05041,
+ "3105": 4.02391,
+ "3110": 4.01947,
+ "3115": 3.95768,
+ "3120": 4.0342,
+ "3125": 3.97112,
+ "3130": 4.0001,
+ "3135": 3.99325,
+ "3140": 3.98033,
+ "3145": 3.96439,
+ "3150": 3.9963,
+ "3155": 3.98705,
+ "3160": 3.98877,
+ "3165": 3.98974,
+ "3170": 3.99147,
+ "3175": 3.95975,
+ "3180": 3.96406,
+ "3185": 3.93573,
+ "3190": 3.95726,
+ "3195": 3.94057,
+ "3200": 3.9207,
+ "3205": 3.9528,
+ "3210": 3.92243,
+ "3215": 3.93403,
+ "3220": 3.92071,
+ "3225": 3.94411,
+ "3230": 3.93022,
+ "3235": 3.93943,
+ "3240": 3.92479,
+ "3245": 3.91765,
+ "3250": 3.86859,
+ "3255": 3.92312,
+ "3260": 3.91158,
+ "3265": 3.9524,
+ "3270": 3.93407,
+ "3275": 3.89812,
+ "3280": 3.91152,
+ "3285": 3.89707,
+ "3290": 3.89983,
+ "3295": 3.8711,
+ "3300": 3.88217,
+ "3305": 3.89236,
+ "3310": 3.88573,
+ "3315": 3.92748,
+ "3320": 3.87918,
+ "3325": 3.87256,
+ "3330": 3.85248,
+ "3335": 3.89293,
+ "3340": 3.84817,
+ "3345": 3.85824,
+ "3350": 3.88459,
+ "3355": 3.87445,
+ "3360": 3.86097,
+ "3365": 3.86681,
+ "3370": 3.84847,
+ "3375": 3.88018,
+ "3380": 3.82118,
+ "3385": 3.84082,
+ "3390": 3.81511,
+ "3395": 3.8912,
+ "3400": 3.86212,
+ "3405": 3.88629,
+ "3410": 3.80424,
+ "3415": 3.75668,
+ "3420": 3.82723,
+ "3425": 3.84411,
+ "3430": 3.87458,
+ "3435": 3.83744,
+ "3440": 3.84611,
+ "3445": 3.80172,
+ "3450": 3.81378,
+ "3455": 3.82823,
+ "3460": 3.80948,
+ "3465": 3.78525,
+ "3470": 3.79721,
+ "3475": 3.80318,
+ "3480": 3.80364,
+ "3485": 3.82555,
+ "3490": 3.79477,
+ "3495": 3.8291,
+ "3500": 3.79494,
+ "3505": 3.80124,
+ "3510": 3.78117,
+ "3515": 3.83014,
+ "3520": 3.82292,
+ "3525": 3.78376,
+ "3530": 3.78278,
+ "3535": 3.78437,
+ "3540": 3.84503,
+ "3545": 3.75429,
+ "3550": 3.81018,
+ "3555": 3.74657,
+ "3560": 3.80773,
+ "3565": 3.7701,
+ "3570": 3.7634,
+ "3575": 3.73747,
+ "3580": 3.79607,
+ "3585": 3.78701,
+ "3590": 3.71139,
+ "3595": 3.78595,
+ "3600": 3.73738,
+ "3605": 3.74362,
+ "3610": 3.73366,
+ "3615": 3.76504,
+ "3620": 3.80578,
+ "3625": 3.7465,
+ "3630": 3.77319,
+ "3635": 3.70283,
+ "3640": 3.73456,
+ "3645": 3.76473,
+ "3650": 3.71918,
+ "3655": 3.74087,
+ "3660": 3.74633,
+ "3665": 3.77489,
+ "3670": 3.73241,
+ "3675": 3.72974,
+ "3680": 3.73984,
+ "3685": 3.69058,
+ "3690": 3.71736,
+ "3695": 3.70248,
+ "3700": 3.72576,
+ "3705": 3.69692,
+ "3710": 3.69648,
+ "3715": 3.70696,
+ "3720": 3.68439,
+ "3725": 3.67042,
+ "3730": 3.67356,
+ "3735": 3.71274,
+ "3740": 3.6932,
+ "3745": 3.67682,
+ "3750": 3.70016,
+ "3755": 3.68023,
+ "3760": 3.69391,
+ "3765": 3.67921,
+ "3770": 3.66894,
+ "3775": 3.65256,
+ "3780": 3.64316,
+ "3785": 3.69333,
+ "3790": 3.61687,
+ "3795": 3.66583,
+ "3800": 3.64926,
+ "3805": 3.6419,
+ "3810": 3.61152,
+ "3815": 3.64733,
+ "3820": 3.65644,
+ "3825": 3.67393,
+ "3830": 3.65544,
+ "3835": 3.61626,
+ "3840": 3.69627,
+ "3845": 3.68163,
+ "3850": 3.61837,
+ "3855": 3.62662,
+ "3860": 3.67377,
+ "3865": 3.62774,
+ "3870": 3.68905,
+ "3875": 3.60126,
+ "3880": 3.59771,
+ "3885": 3.62173,
+ "3890": 3.62793,
+ "3895": 3.57784,
+ "3900": 3.6351,
+ "3905": 3.60982,
+ "3910": 3.59626,
+ "3915": 3.60346,
+ "3920": 3.58896,
+ "3925": 3.58883,
+ "3930": 3.59402,
+ "3935": 3.59221,
+ "3940": 3.58898,
+ "3945": 3.58767,
+ "3950": 3.63454,
+ "3955": 3.59026,
+ "3960": 3.62572,
+ "3965": 3.6052,
+ "3970": 3.58143,
+ "3975": 3.58141,
+ "3980": 3.54577,
+ "3985": 3.62174,
+ "3990": 3.59804,
+ "3995": 3.62617,
+ "4000": 3.57656,
+ "4005": 3.55832,
+ "4010": 3.60238,
+ "4015": 3.59813,
+ "4020": 3.60309,
+ "4025": 3.5864,
+ "4030": 3.64627,
+ "4035": 3.58587,
+ "4040": 3.60396,
+ "4045": 3.61402,
+ "4050": 3.59101,
+ "4055": 3.59269,
+ "4060": 3.60425,
+ "4065": 3.60121,
+ "4070": 3.53392,
+ "4075": 3.57256,
+ "4080": 3.54554,
+ "4085": 3.56671,
+ "4090": 3.56241,
+ "4095": 3.54644,
+ "4100": 3.56914,
+ "4105": 3.55106,
+ "4110": 3.53219,
+ "4115": 3.58295,
+ "4120": 3.5136,
+ "4125": 3.50884,
+ "4130": 3.56535,
+ "4135": 3.5618,
+ "4140": 3.50752,
+ "4145": 3.52274,
+ "4150": 3.57049,
+ "4155": 3.50254,
+ "4160": 3.55895,
+ "4165": 3.5748,
+ "4170": 3.51715,
+ "4175": 3.5163,
+ "4180": 3.51465,
+ "4185": 3.52626,
+ "4190": 3.51787,
+ "4195": 3.51175,
+ "4200": 3.50947,
+ "4205": 3.54211,
+ "4210": 3.53299,
+ "4215": 3.53499,
+ "4220": 3.54228,
+ "4225": 3.5205,
+ "4230": 3.51255,
+ "4235": 3.53387,
+ "4240": 3.50182,
+ "4245": 3.50827,
+ "4250": 3.50171,
+ "4255": 3.5193,
+ "4260": 3.48149,
+ "4265": 3.5037,
+ "4270": 3.51717,
+ "4275": 3.55051,
+ "4280": 3.50079,
+ "4285": 3.51915,
+ "4290": 3.47897,
+ "4295": 3.49759,
+ "4300": 3.53834,
+ "4305": 3.50542,
+ "4310": 3.52444,
+ "4315": 3.51898,
+ "4320": 3.52076,
+ "4325": 3.53495,
+ "4330": 3.47417,
+ "4335": 3.50397,
+ "4340": 3.51396,
+ "4345": 3.4492,
+ "4350": 3.45758,
+ "4355": 3.53572,
+ "4360": 3.49536,
+ "4365": 3.48284,
+ "4370": 3.48554,
+ "4375": 3.45147,
+ "4380": 3.45518,
+ "4385": 3.43686,
+ "4390": 3.50694,
+ "4395": 3.48994,
+ "4400": 3.48586,
+ "4405": 3.42794,
+ "4410": 3.49753,
+ "4415": 3.46487,
+ "4420": 3.45549,
+ "4425": 3.4938,
+ "4430": 3.46086,
+ "4435": 3.5043,
+ "4440": 3.49809,
+ "4445": 3.45086,
+ "4450": 3.40806,
+ "4455": 3.47726,
+ "4460": 3.4483,
+ "4465": 3.46225,
+ "4470": 3.43686,
+ "4475": 3.47078,
+ "4480": 3.45685,
+ "4485": 3.44994,
+ "4490": 3.44689,
+ "4495": 3.39912,
+ "4500": 3.4671,
+ "4505": 3.44693,
+ "4510": 3.4537,
+ "4515": 3.41693,
+ "4520": 3.45165,
+ "4525": 3.41343,
+ "4530": 3.45282,
+ "4535": 3.40622,
+ "4540": 3.43139,
+ "4545": 3.44504,
+ "4550": 3.48612,
+ "4555": 3.41672,
+ "4560": 3.43375,
+ "4565": 3.38931,
+ "4570": 3.42559,
+ "4575": 3.4257,
+ "4580": 3.46573,
+ "4585": 3.43425,
+ "4590": 3.43382,
+ "4595": 3.40845,
+ "4600": 3.40597,
+ "4605": 3.42905,
+ "4610": 3.4249,
+ "4615": 3.46442,
+ "4620": 3.40511,
+ "4625": 3.43702,
+ "4630": 3.42316,
+ "4635": 3.40645,
+ "4640": 3.43948,
+ "4645": 3.42856,
+ "4650": 3.44086,
+ "4655": 3.41645,
+ "4660": 3.4078,
+ "4665": 3.42546,
+ "4670": 3.45514,
+ "4675": 3.41355,
+ "4680": 3.43936,
+ "4685": 3.44032,
+ "4690": 3.41455,
+ "4695": 3.39512,
+ "4700": 3.38692,
+ "4705": 3.36216,
+ "4710": 3.4197,
+ "4715": 3.40698,
+ "4720": 3.39827,
+ "4725": 3.37044,
+ "4730": 3.40384,
+ "4735": 3.33173,
+ "4740": 3.37375,
+ "4745": 3.41323,
+ "4750": 3.36757,
+ "4755": 3.40305,
+ "4760": 3.42353,
+ "4765": 3.36938,
+ "4770": 3.37644,
+ "4775": 3.37236,
+ "4780": 3.38001,
+ "4785": 3.38383,
+ "4790": 3.42356,
+ "4795": 3.40289,
+ "4800": 3.35511,
+ "4805": 3.4232,
+ "4810": 3.36059,
+ "4815": 3.39891,
+ "4820": 3.36115,
+ "4825": 3.41107,
+ "4830": 3.3919,
+ "4835": 3.37666,
+ "4840": 3.42516,
+ "4845": 3.33553,
+ "4850": 3.40422,
+ "4855": 3.40483,
+ "4860": 3.33477,
+ "4865": 3.37463,
+ "4870": 3.35817,
+ "4875": 3.40332,
+ "4880": 3.40756,
+ "4885": 3.36419,
+ "4890": 3.3704,
+ "4895": 3.36552,
+ "4900": 3.3391,
+ "4905": 3.34145,
+ "4910": 3.33707,
+ "4915": 3.38449,
+ "4920": 3.36807,
+ "4925": 3.32242,
+ "4930": 3.35216,
+ "4935": 3.34014,
+ "4940": 3.29837,
+ "4945": 3.37071,
+ "4950": 3.30739,
+ "4955": 3.41626,
+ "4960": 3.35684,
+ "4965": 3.35495,
+ "4970": 3.34283,
+ "4975": 3.35307,
+ "4980": 3.37837,
+ "4985": 3.3632,
+ "4990": 3.34411,
+ "4995": 3.38853,
+ "5000": 3.31614,
+ "5005": 3.36343,
+ "5010": 3.36749,
+ "5015": 3.32044,
+ "5020": 3.29609,
+ "5025": 3.32443,
+ "5030": 3.33684,
+ "5035": 3.34249,
+ "5040": 3.31506,
+ "5045": 3.35913,
+ "5050": 3.3173,
+ "5055": 3.33889,
+ "5060": 3.29837,
+ "5065": 3.34634,
+ "5070": 3.34096,
+ "5075": 3.35375,
+ "5080": 3.32805,
+ "5085": 3.35001,
+ "5090": 3.33208,
+ "5095": 3.30049,
+ "5100": 3.33016,
+ "5105": 3.33714,
+ "5110": 3.34187,
+ "5115": 3.31339,
+ "5120": 3.3534,
+ "5125": 3.32752,
+ "5130": 3.32822,
+ "5135": 3.31071,
+ "5140": 3.32054,
+ "5145": 3.31821,
+ "5150": 3.33084,
+ "5155": 3.32474,
+ "5160": 3.32178,
+ "5165": 3.35645,
+ "5170": 3.23776,
+ "5175": 3.32997,
+ "5180": 3.29167,
+ "5185": 3.31811,
+ "5190": 3.3401,
+ "5195": 3.31594,
+ "5200": 3.31988,
+ "5205": 3.35436,
+ "5210": 3.29051,
+ "5215": 3.29647,
+ "5220": 3.28836,
+ "5225": 3.2945,
+ "5230": 3.32979,
+ "5235": 3.2927,
+ "5240": 3.28191,
+ "5245": 3.30256,
+ "5250": 3.30886,
+ "5255": 3.29346,
+ "5260": 3.32191,
+ "5265": 3.28005,
+ "5270": 3.264,
+ "5275": 3.26563,
+ "5280": 3.28933,
+ "5285": 3.31759,
+ "5290": 3.26837,
+ "5295": 3.28588,
+ "5300": 3.28597,
+ "5305": 3.27172,
+ "5310": 3.33344,
+ "5315": 3.26371,
+ "5320": 3.31352,
+ "5325": 3.32076,
+ "5330": 3.28681,
+ "5335": 3.29845,
+ "5340": 3.23905,
+ "5345": 3.28681,
+ "5350": 3.29528,
+ "5355": 3.2925,
+ "5360": 3.24225,
+ "5365": 3.26425,
+ "5370": 3.29855,
+ "5375": 3.27842,
+ "5380": 3.25538,
+ "5385": 3.29124,
+ "5390": 3.28934,
+ "5395": 3.21094,
+ "5400": 3.30836,
+ "5405": 3.22361,
+ "5410": 3.30176,
+ "5415": 3.22732,
+ "5420": 3.26506,
+ "5425": 3.24726,
+ "5430": 3.25425,
+ "5435": 3.28921,
+ "5440": 3.21956,
+ "5445": 3.25051,
+ "5450": 3.25264,
+ "5455": 3.23791,
+ "5460": 3.26176,
+ "5465": 3.30533,
+ "5470": 3.27866,
+ "5475": 3.21005,
+ "5480": 3.29072,
+ "5485": 3.24936,
+ "5490": 3.27635,
+ "5495": 3.2797,
+ "5500": 3.23463,
+ "5505": 3.2467,
+ "5510": 3.29267,
+ "5515": 3.27643,
+ "5520": 3.24521,
+ "5525": 3.29217,
+ "5530": 3.2359,
+ "5535": 3.2708,
+ "5540": 3.26258,
+ "5545": 3.26894,
+ "5550": 3.25652,
+ "5555": 3.23592,
+ "5560": 3.2303,
+ "5565": 3.2764,
+ "5570": 3.23566,
+ "5575": 3.27265,
+ "5580": 3.24633,
+ "5585": 3.19244,
+ "5590": 3.25225,
+ "5595": 3.21984,
+ "5600": 3.26135,
+ "5605": 3.18226,
+ "5610": 3.26836,
+ "5615": 3.2602,
+ "5620": 3.26819,
+ "5625": 3.2575,
+ "5630": 3.24789,
+ "5635": 3.22501,
+ "5640": 3.2534,
+ "5645": 3.2148,
+ "5650": 3.21316,
+ "5655": 3.20652,
+ "5660": 3.21447,
+ "5665": 3.21786,
+ "5670": 3.20699,
+ "5675": 3.23483,
+ "5680": 3.20462,
+ "5685": 3.21202,
+ "5690": 3.21459,
+ "5695": 3.25062,
+ "5700": 3.20088,
+ "5705": 3.19331,
+ "5710": 3.18404,
+ "5715": 3.29312,
+ "5720": 3.25473,
+ "5725": 3.2033,
+ "5730": 3.24318,
+ "5735": 3.23452,
+ "5740": 3.23299,
+ "5745": 3.21416,
+ "5750": 3.238,
+ "5755": 3.24426,
+ "5760": 3.23106,
+ "5765": 3.23312,
+ "5770": 3.25814,
+ "5775": 3.20578,
+ "5780": 3.22492,
+ "5785": 3.22461,
+ "5790": 3.23614,
+ "5795": 3.22921,
+ "5800": 3.17268,
+ "5805": 3.18929,
+ "5810": 3.23085,
+ "5815": 3.20836,
+ "5820": 3.16927,
+ "5825": 3.21348,
+ "5830": 3.17197,
+ "5835": 3.18181,
+ "5840": 3.21352,
+ "5845": 3.22154,
+ "5850": 3.22412,
+ "5855": 3.15808,
+ "5860": 3.1749,
+ "5865": 3.20572,
+ "5870": 3.16974,
+ "5875": 3.20687,
+ "5880": 3.19823,
+ "5885": 3.20131,
+ "5890": 3.22484,
+ "5895": 3.23967,
+ "5900": 3.19433,
+ "5905": 3.22418,
+ "5910": 3.20808,
+ "5915": 3.18093,
+ "5920": 3.19895,
+ "5925": 3.16397,
+ "5930": 3.19617,
+ "5935": 3.19588,
+ "5940": 3.21243,
+ "5945": 3.22431,
+ "5950": 3.20776,
+ "5955": 3.16712,
+ "5960": 3.22956,
+ "5965": 3.18229,
+ "5970": 3.22265,
+ "5975": 3.19139,
+ "5980": 3.26149,
+ "5985": 3.14699,
+ "5990": 3.2404,
+ "5995": 3.15733,
+ "6000": 3.18137,
+ "6005": 3.16121,
+ "6010": 3.16817,
+ "6015": 3.17092,
+ "6020": 3.17845,
+ "6025": 3.21374,
+ "6030": 3.15388,
+ "6035": 3.20901,
+ "6040": 3.18574,
+ "6045": 3.20493,
+ "6050": 3.20413,
+ "6055": 3.17556,
+ "6060": 3.18956,
+ "6065": 3.2171,
+ "6070": 3.16993,
+ "6075": 3.13742,
+ "6080": 3.19605,
+ "6085": 3.15615,
+ "6090": 3.19301,
+ "6095": 3.1936,
+ "6100": 3.14612,
+ "6105": 3.1956,
+ "6110": 3.13654,
+ "6115": 3.18623,
+ "6120": 3.17766,
+ "6125": 3.18345,
+ "6130": 3.17135,
+ "6135": 3.16849,
+ "6140": 3.16824,
+ "6145": 3.14543,
+ "6150": 3.18653,
+ "6155": 3.15905,
+ "6160": 3.13491,
+ "6165": 3.16459,
+ "6170": 3.15526,
+ "6175": 3.15099,
+ "6180": 3.14925,
+ "6185": 3.19344,
+ "6190": 3.15942,
+ "6195": 3.1311,
+ "6200": 3.15781,
+ "6205": 3.1519,
+ "6210": 3.10432,
+ "6215": 3.15922,
+ "6220": 3.16046,
+ "6225": 3.17595,
+ "6230": 3.1123,
+ "6235": 3.14563,
+ "6240": 3.08744,
+ "6245": 3.18824,
+ "6250": 3.14759,
+ "6255": 3.16334,
+ "6260": 3.14581,
+ "6265": 3.16362,
+ "6270": 3.1056,
+ "6275": 3.12837,
+ "6280": 3.141,
+ "6285": 3.12097,
+ "6290": 3.14963,
+ "6295": 3.15585,
+ "6300": 3.16221,
+ "6305": 3.21664,
+ "6310": 3.11775,
+ "6315": 3.11514,
+ "6320": 3.1634,
+ "6325": 3.10881,
+ "6330": 3.16926,
+ "6335": 3.15865,
+ "6340": 3.11939,
+ "6345": 3.17109,
+ "6350": 3.12082,
+ "6355": 3.12257,
+ "6360": 3.11532,
+ "6365": 3.15591,
+ "6370": 3.16345,
+ "6375": 3.1383,
+ "6380": 3.15314,
+ "6385": 3.17551,
+ "6390": 3.13046,
+ "6395": 3.10684,
+ "6400": 3.1102,
+ "6405": 3.19048,
+ "6410": 3.17798,
+ "6415": 3.13085,
+ "6420": 3.17574,
+ "6425": 3.1764,
+ "6430": 3.17202,
+ "6435": 3.12873,
+ "6440": 3.14255,
+ "6445": 3.15689,
+ "6450": 3.09595,
+ "6455": 3.09246,
+ "6460": 3.1324,
+ "6465": 3.17048,
+ "6470": 3.14497,
+ "6475": 3.13684,
+ "6480": 3.15394,
+ "6485": 3.11743,
+ "6490": 3.08482,
+ "6495": 3.16911,
+ "6500": 3.14809,
+ "6505": 3.08951,
+ "6510": 3.1511,
+ "6515": 3.16505,
+ "6520": 3.09273,
+ "6525": 3.15232,
+ "6530": 3.11343,
+ "6535": 3.12729,
+ "6540": 3.18295,
+ "6545": 3.12009,
+ "6550": 3.11425,
+ "6555": 3.11079,
+ "6560": 3.07571,
+ "6565": 3.08127,
+ "6570": 3.1089,
+ "6575": 3.0599,
+ "6580": 3.17609,
+ "6585": 3.11189,
+ "6590": 3.0934,
+ "6595": 3.1113,
+ "6600": 3.10549,
+ "6605": 3.08898,
+ "6610": 3.0878,
+ "6615": 3.13761,
+ "6620": 3.07879,
+ "6625": 3.10091,
+ "6630": 3.09863,
+ "6635": 3.13243,
+ "6640": 3.09232,
+ "6645": 3.11531,
+ "6650": 3.14167,
+ "6655": 3.07687,
+ "6660": 3.11462,
+ "6665": 3.1329,
+ "6670": 3.08537,
+ "6675": 3.10744,
+ "6680": 3.10919,
+ "6685": 3.14517,
+ "6690": 3.11878,
+ "6695": 3.12452,
+ "6700": 3.11556,
+ "6705": 3.09577,
+ "6710": 3.11007,
+ "6715": 3.06081,
+ "6720": 3.13911,
+ "6725": 3.12829,
+ "6730": 3.11357,
+ "6735": 3.11423,
+ "6740": 3.11863,
+ "6745": 3.09659,
+ "6750": 3.11418,
+ "6755": 3.07428,
+ "6760": 3.06982,
+ "6765": 3.08681,
+ "6770": 3.07458,
+ "6775": 3.11154,
+ "6780": 3.07987,
+ "6785": 3.08138,
+ "6790": 3.10966,
+ "6795": 3.07314,
+ "6800": 3.09848,
+ "6805": 3.08997,
+ "6810": 3.11427,
+ "6815": 3.04801,
+ "6820": 3.07658,
+ "6825": 3.10619,
+ "6830": 3.09045,
+ "6835": 3.06402,
+ "6840": 3.06939,
+ "6845": 3.11685,
+ "6850": 3.08641,
+ "6855": 3.11434,
+ "6860": 3.06353,
+ "6865": 3.11204,
+ "6870": 3.07894,
+ "6875": 3.07872,
+ "6880": 3.08922,
+ "6885": 3.0572,
+ "6890": 3.08052,
+ "6895": 3.05839,
+ "6900": 3.0601,
+ "6905": 3.07714,
+ "6910": 3.09632,
+ "6915": 3.11802,
+ "6920": 3.06802,
+ "6925": 3.08735,
+ "6930": 3.07005,
+ "6935": 3.02833,
+ "6940": 3.07262,
+ "6945": 3.06124,
+ "6950": 3.0831,
+ "6955": 3.06193,
+ "6960": 3.05909,
+ "6965": 3.1035,
+ "6970": 3.0394,
+ "6975": 3.11091,
+ "6980": 3.07019,
+ "6985": 3.06922,
+ "6990": 3.11558,
+ "6995": 3.09406,
+ "7000": 3.02967,
+ "7005": 3.10225,
+ "7010": 3.08,
+ "7015": 3.07816,
+ "7020": 3.10327,
+ "7025": 3.08792,
+ "7030": 3.08999,
+ "7035": 3.04406,
+ "7040": 3.02356,
+ "7045": 3.08573,
+ "7050": 3.10112,
+ "7055": 3.039,
+ "7060": 3.10071,
+ "7065": 3.11572,
+ "7070": 3.06359,
+ "7075": 3.06606,
+ "7080": 3.11665,
+ "7085": 3.03623,
+ "7090": 3.06064,
+ "7095": 3.05019,
+ "7100": 3.07065,
+ "7105": 3.02165,
+ "7110": 3.06693,
+ "7115": 3.04035,
+ "7120": 3.08366,
+ "7125": 3.03743,
+ "7130": 3.05289,
+ "7135": 3.05821,
+ "7140": 3.06319,
+ "7145": 3.0729,
+ "7150": 3.02538,
+ "7155": 3.09094,
+ "7160": 3.0078,
+ "7165": 3.0453,
+ "7170": 3.07848,
+ "7175": 3.03754,
+ "7180": 3.0738,
+ "7185": 3.09475,
+ "7190": 3.0571,
+ "7195": 3.06383,
+ "7200": 3.06344,
+ "7205": 3.04596,
+ "7210": 3.09243,
+ "7215": 3.07069,
+ "7220": 3.09224,
+ "7225": 3.07409,
+ "7230": 3.07803,
+ "7235": 3.05034,
+ "7240": 3.05229,
+ "7245": 3.07394,
+ "7250": 3.01314,
+ "7255": 3.03466,
+ "7260": 3.07331,
+ "7265": 3.00517,
+ "7270": 3.04426,
+ "7275": 3.04304,
+ "7280": 3.04517,
+ "7285": 3.05569,
+ "7290": 3.07632,
+ "7295": 3.06917,
+ "7300": 3.02908,
+ "7305": 3.03397,
+ "7310": 3.0531,
+ "7315": 3.07812,
+ "7320": 3.06016,
+ "7325": 3.06585,
+ "7330": 3.02967,
+ "7335": 3.02978,
+ "7340": 3.06469,
+ "7345": 3.01179,
+ "7350": 3.06517,
+ "7355": 3.04873,
+ "7360": 3.03868,
+ "7365": 3.04315,
+ "7370": 3.03485,
+ "7375": 3.00292,
+ "7380": 3.06472,
+ "7385": 3.07903,
+ "7390": 3.07032,
+ "7395": 3.02671,
+ "7400": 3.0787,
+ "7405": 3.04869,
+ "7410": 3.06361,
+ "7415": 3.05577,
+ "7420": 3.03517,
+ "7425": 3.08965,
+ "7430": 3.03176,
+ "7435": 3.02161,
+ "7440": 3.04259,
+ "7445": 3.01582,
+ "7450": 2.99636,
+ "7455": 3.05223,
+ "7460": 3.0446,
+ "7465": 3.05375,
+ "7470": 3.05949,
+ "7475": 3.06783,
+ "7480": 3.03044,
+ "7485": 2.99246,
+ "7490": 2.99247,
+ "7495": 3.00196,
+ "7500": 3.02965,
+ "7505": 3.00861,
+ "7510": 2.98019,
+ "7515": 3.02686,
+ "7520": 3.01988,
+ "7525": 2.98612,
+ "7530": 3.03007,
+ "7535": 3.0481,
+ "7540": 3.0281,
+ "7545": 3.06291,
+ "7550": 3.06535,
+ "7555": 3.009,
+ "7560": 3.01694,
+ "7565": 3.01103,
+ "7570": 3.03831,
+ "7575": 2.98318,
+ "7580": 3.03374,
+ "7585": 3.02282,
+ "7590": 3.01672,
+ "7595": 3.07181,
+ "7600": 3.03331,
+ "7605": 3.02458,
+ "7610": 3.00719,
+ "7615": 2.99847,
+ "7620": 2.99426,
+ "7625": 3.04117,
+ "7630": 3.02354,
+ "7635": 3.0199,
+ "7640": 3.01964,
+ "7645": 3.0499,
+ "7650": 3.04817,
+ "7655": 3.09287,
+ "7660": 2.96636,
+ "7665": 3.03047,
+ "7670": 3.01583,
+ "7675": 3.00768,
+ "7680": 3.00243,
+ "7685": 3.07329,
+ "7690": 3.0155,
+ "7695": 3.00104,
+ "7700": 3.05393,
+ "7705": 3.01746,
+ "7710": 3.06264,
+ "7715": 2.99843,
+ "7720": 3.08416,
+ "7725": 2.98362,
+ "7730": 2.99995,
+ "7735": 3.0291,
+ "7740": 3.01406,
+ "7745": 3.00596,
+ "7750": 3.01129,
+ "7755": 3.02351,
+ "7760": 2.98892,
+ "7765": 3.00502,
+ "7770": 3.02804,
+ "7775": 2.99122,
+ "7780": 2.98046,
+ "7785": 3.01694,
+ "7790": 3.00331,
+ "7795": 3.0285,
+ "7800": 3.01244,
+ "7805": 3.01432,
+ "7810": 3.03469,
+ "7815": 3.00479,
+ "7820": 3.00227,
+ "7825": 3.03548,
+ "7830": 3.03553,
+ "7835": 2.9655,
+ "7840": 3.04748,
+ "7845": 2.9833,
+ "7850": 2.94233,
+ "7855": 2.98825,
+ "7860": 2.98524,
+ "7865": 3.03238,
+ "7870": 2.9725,
+ "7875": 2.99196,
+ "7880": 3.00767,
+ "7885": 2.99914,
+ "7890": 3.04152,
+ "7895": 3.02976,
+ "7900": 3.03023,
+ "7905": 3.0012,
+ "7910": 3.01179,
+ "7915": 3.02856,
+ "7920": 3.01682,
+ "7925": 3.00177,
+ "7930": 3.03266,
+ "7935": 2.99092,
+ "7940": 3.03879,
+ "7945": 3.05106,
+ "7950": 2.96915,
+ "7955": 2.99035,
+ "7960": 2.97087,
+ "7965": 2.94657,
+ "7970": 2.96554,
+ "7975": 2.99871,
+ "7980": 3.01016,
+ "7985": 2.97941,
+ "7990": 2.97792,
+ "7995": 2.96231,
+ "8000": 3.02507,
+ "8005": 2.98376,
+ "8010": 2.97697,
+ "8015": 2.96605,
+ "8020": 2.97995,
+ "8025": 2.95625,
+ "8030": 2.97748,
+ "8035": 2.9714,
+ "8040": 2.95953,
+ "8045": 3.0186,
+ "8050": 3.01687,
+ "8055": 2.97725,
+ "8060": 3.00707,
+ "8065": 2.9911,
+ "8070": 2.97027,
+ "8075": 2.98007,
+ "8080": 3.01331,
+ "8085": 2.97074,
+ "8090": 2.9838,
+ "8095": 3.00467,
+ "8100": 2.95388,
+ "8105": 2.99604,
+ "8110": 2.98317,
+ "8115": 2.95737,
+ "8120": 2.9752,
+ "8125": 3.00079,
+ "8130": 2.97282,
+ "8135": 2.98828,
+ "8140": 2.9695,
+ "8145": 2.95979,
+ "8150": 2.98145,
+ "8155": 2.95372,
+ "8160": 2.99757,
+ "8165": 2.99437,
+ "8170": 2.96042,
+ "8175": 2.95683,
+ "8180": 3.01502,
+ "8185": 2.98826,
+ "8190": 3.02529,
+ "8195": 2.99754,
+ "8200": 2.96688,
+ "8205": 2.97936,
+ "8210": 2.9809,
+ "8215": 2.99241,
+ "8220": 2.97241,
+ "8225": 2.96598,
+ "8230": 2.99563,
+ "8235": 3.00527,
+ "8240": 2.97818,
+ "8245": 2.9762,
+ "8250": 3.01067,
+ "8255": 2.97102,
+ "8260": 2.97507,
+ "8265": 2.95804,
+ "8270": 2.9784,
+ "8275": 2.96899,
+ "8280": 2.94228,
+ "8285": 2.98064,
+ "8290": 2.97026,
+ "8295": 2.95426,
+ "8300": 2.9682,
+ "8305": 2.97642,
+ "8310": 2.98258,
+ "8315": 2.95973,
+ "8320": 2.98172,
+ "8325": 2.93194,
+ "8330": 2.89982,
+ "8335": 2.96861,
+ "8340": 2.99368,
+ "8345": 2.9463,
+ "8350": 2.9629,
+ "8355": 2.98848,
+ "8360": 2.96962,
+ "8365": 2.98418,
+ "8370": 2.99189,
+ "8375": 2.94013,
+ "8380": 2.94318,
+ "8385": 2.97483,
+ "8390": 2.94848,
+ "8395": 2.97735,
+ "8400": 2.96117,
+ "8405": 2.97663,
+ "8410": 3.03037,
+ "8415": 2.93829,
+ "8420": 2.92011,
+ "8425": 2.97748,
+ "8430": 2.98116,
+ "8435": 2.93271,
+ "8440": 3.01548,
+ "8445": 2.992,
+ "8450": 2.96723,
+ "8455": 2.97253,
+ "8460": 2.98133,
+ "8465": 2.92647,
+ "8470": 2.94843,
+ "8475": 2.99249,
+ "8480": 2.9314,
+ "8485": 2.94069,
+ "8490": 2.95085,
+ "8495": 2.9363,
+ "8500": 2.97163,
+ "8505": 2.92387,
+ "8510": 3.00526,
+ "8515": 2.94325,
+ "8520": 2.96039,
+ "8525": 2.88607,
+ "8530": 2.96086,
+ "8535": 2.97834,
+ "8540": 2.93481,
+ "8545": 2.95961,
+ "8550": 2.92545,
+ "8555": 2.99273,
+ "8560": 2.99546,
+ "8565": 2.953,
+ "8570": 2.94943,
+ "8575": 2.93605,
+ "8580": 2.96987,
+ "8585": 2.97782,
+ "8590": 2.9771,
+ "8595": 2.97996,
+ "8600": 2.94831,
+ "8605": 2.94602,
+ "8610": 2.95656,
+ "8615": 2.96288,
+ "8620": 2.92625,
+ "8625": 2.95014,
+ "8630": 2.95439,
+ "8635": 2.94431,
+ "8640": 2.92761,
+ "8645": 2.98473,
+ "8650": 2.92169,
+ "8655": 2.96751,
+ "8660": 2.97388,
+ "8665": 2.9555,
+ "8670": 2.96791,
+ "8675": 2.94353,
+ "8680": 2.93497,
+ "8685": 2.94813,
+ "8690": 2.96647,
+ "8695": 2.97341,
+ "8700": 2.9509,
+ "8705": 2.91852,
+ "8710": 2.97184,
+ "8715": 2.91804,
+ "8720": 2.97489,
+ "8725": 2.95074,
+ "8730": 2.9448,
+ "8735": 2.97278,
+ "8740": 2.92895,
+ "8745": 2.96824,
+ "8750": 2.96889,
+ "8755": 2.93058,
+ "8760": 2.94885,
+ "8765": 2.91693,
+ "8770": 2.96877,
+ "8775": 2.94238,
+ "8780": 2.92915,
+ "8785": 2.94981,
+ "8790": 2.93043,
+ "8795": 2.96602,
+ "8800": 2.93001,
+ "8805": 2.90304,
+ "8810": 2.93015,
+ "8815": 2.93315,
+ "8820": 2.90758,
+ "8825": 2.92642,
+ "8830": 2.91336,
+ "8835": 2.90113,
+ "8840": 2.91621,
+ "8845": 2.92894,
+ "8850": 2.95875,
+ "8855": 2.93098,
+ "8860": 2.99044,
+ "8865": 2.93774,
+ "8870": 2.9099,
+ "8875": 2.92437,
+ "8880": 2.93017,
+ "8885": 2.91961,
+ "8890": 2.94184,
+ "8895": 2.92432,
+ "8900": 2.94771,
+ "8905": 2.9393,
+ "8910": 2.91975,
+ "8915": 2.90632,
+ "8920": 2.91034,
+ "8925": 2.9761,
+ "8930": 2.96403,
+ "8935": 2.97461,
+ "8940": 2.95199,
+ "8945": 2.95035,
+ "8950": 2.9346,
+ "8955": 2.91671,
+ "8960": 2.90076,
+ "8965": 2.93031,
+ "8970": 2.94483,
+ "8975": 2.90586,
+ "8980": 2.89932,
+ "8985": 2.9246,
+ "8990": 2.96958,
+ "8995": 2.93963,
+ "9000": 2.89736,
+ "9005": 2.94135,
+ "9010": 2.9809,
+ "9015": 2.9053,
+ "9020": 2.90642,
+ "9025": 2.92304,
+ "9030": 2.94764,
+ "9035": 2.85756,
+ "9040": 2.9367,
+ "9045": 2.9263,
+ "9050": 2.96216,
+ "9055": 2.89055,
+ "9060": 2.95864,
+ "9065": 2.98752,
+ "9070": 2.92761,
+ "9075": 2.94354,
+ "9080": 2.93462,
+ "9085": 2.94737,
+ "9090": 2.93915,
+ "9095": 2.90014,
+ "9100": 2.9001,
+ "9105": 2.89367,
+ "9110": 2.93413,
+ "9115": 2.93968,
+ "9120": 2.97759,
+ "9125": 2.91873,
+ "9130": 2.92351,
+ "9135": 2.94298,
+ "9140": 2.94725,
+ "9145": 2.89474,
+ "9150": 2.92554,
+ "9155": 2.93383,
+ "9160": 2.93851,
+ "9165": 2.92901,
+ "9170": 2.94939,
+ "9175": 2.88871,
+ "9180": 2.93499,
+ "9185": 2.89423,
+ "9190": 2.9527,
+ "9195": 2.9146,
+ "9200": 2.93323,
+ "9205": 2.89096,
+ "9210": 2.93657,
+ "9215": 2.88074,
+ "9220": 2.90692,
+ "9225": 2.93683,
+ "9230": 2.86887,
+ "9235": 2.87904,
+ "9240": 2.89693,
+ "9245": 2.88503,
+ "9250": 2.88372,
+ "9255": 2.91256,
+ "9260": 2.8793,
+ "9265": 2.92448,
+ "9270": 2.89737,
+ "9275": 2.91446,
+ "9280": 2.92215,
+ "9285": 2.91919,
+ "9290": 2.93378,
+ "9295": 2.93182,
+ "9300": 2.88059,
+ "9305": 2.91061,
+ "9310": 2.89957,
+ "9315": 2.86843,
+ "9320": 2.86168,
+ "9325": 2.90649,
+ "9330": 2.95636,
+ "9335": 2.87569,
+ "9340": 2.94078,
+ "9345": 2.94977,
+ "9350": 2.91622,
+ "9355": 2.87863,
+ "9360": 2.89752,
+ "9365": 2.88455,
+ "9370": 2.93833,
+ "9375": 2.9106,
+ "9380": 2.86805,
+ "9385": 2.91586,
+ "9390": 2.92351,
+ "9395": 2.9233,
+ "9400": 2.89877,
+ "9405": 2.89296,
+ "9410": 2.92,
+ "9415": 2.91947,
+ "9420": 2.89504,
+ "9425": 2.90262,
+ "9430": 2.87998,
+ "9435": 2.90515,
+ "9440": 2.89712,
+ "9445": 2.88542,
+ "9450": 2.89256,
+ "9455": 2.89282,
+ "9460": 2.94565,
+ "9465": 2.94946,
+ "9470": 2.88918,
+ "9475": 2.94191,
+ "9480": 2.89187,
+ "9485": 2.88007,
+ "9490": 2.89883,
+ "9495": 2.92532,
+ "9500": 2.89845,
+ "9505": 2.86913,
+ "9510": 2.89654,
+ "9515": 2.9049,
+ "9520": 2.91234,
+ "9525": 2.89455,
+ "9530": 2.89106,
+ "9535": 2.91288
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 9535,
+ "step_interval": 5,
+ "values": {
+ "1": 1021640448.0,
+ "5": 1024063872.0,
+ "10": 1014250560.0,
+ "15": 1024077504.0,
+ "20": 1022486144.0,
+ "25": 1041373568.0,
+ "30": 1028113472.0,
+ "35": 1035625536.0,
+ "40": 1026328576.0,
+ "45": 1022350336.0,
+ "50": 1030098688.0,
+ "55": 1028966592.0,
+ "60": 1036320832.0,
+ "65": 1034679424.0,
+ "70": 1029375104.0,
+ "75": 1028745280.0,
+ "80": 1047575616.0,
+ "85": 1029448576.0,
+ "90": 1020467968.0,
+ "95": 1028309504.0,
+ "100": 1040961728.0,
+ "105": 1039436416.0,
+ "110": 1026879360.0,
+ "115": 1052313216.0,
+ "120": 1018863424.0,
+ "125": 1045372160.0,
+ "130": 1034330624.0,
+ "135": 1016615808.0,
+ "140": 1038582144.0,
+ "145": 1020688768.0,
+ "150": 1039787392.0,
+ "155": 1032796800.0,
+ "160": 1020953280.0,
+ "165": 1032424960.0,
+ "170": 1017397248.0,
+ "175": 1033428160.0,
+ "180": 1036120320.0,
+ "185": 1030575104.0,
+ "190": 1035675328.0,
+ "195": 1034556416.0,
+ "200": 1040974720.0,
+ "205": 1048502400.0,
+ "210": 1054482560.0,
+ "215": 1025161792.0,
+ "220": 1044964800.0,
+ "225": 1038078912.0,
+ "230": 1026225152.0,
+ "235": 1051137536.0,
+ "240": 1029279104.0,
+ "245": 1031401472.0,
+ "250": 1027882624.0,
+ "255": 1016932224.0,
+ "260": 1045011584.0,
+ "265": 1021333376.0,
+ "270": 1030968128.0,
+ "275": 1036915200.0,
+ "280": 1031746432.0,
+ "285": 1015017984.0,
+ "290": 1018760320.0,
+ "295": 1017241344.0,
+ "300": 1034764416.0,
+ "305": 1032169152.0,
+ "310": 1035586816.0,
+ "315": 1012737408.0,
+ "320": 1008279040.0,
+ "325": 1042745536.0,
+ "330": 1042873472.0,
+ "335": 1033512704.0,
+ "340": 1014467584.0,
+ "345": 1042621184.0,
+ "350": 1031855488.0,
+ "355": 1050846848.0,
+ "360": 1030261888.0,
+ "365": 1034598272.0,
+ "370": 1019438080.0,
+ "375": 1022147072.0,
+ "380": 1021328064.0,
+ "385": 1025592832.0,
+ "390": 1023197760.0,
+ "395": 1019656320.0,
+ "400": 1033523584.0,
+ "405": 1023882944.0,
+ "410": 1017911424.0,
+ "415": 1024290368.0,
+ "420": 1020626944.0,
+ "425": 1025858176.0,
+ "430": 1033857792.0,
+ "435": 1028185280.0,
+ "440": 1022093696.0,
+ "445": 1036771648.0,
+ "450": 1025001600.0,
+ "455": 1013855488.0,
+ "460": 1022098176.0,
+ "465": 1041434880.0,
+ "470": 1029041920.0,
+ "475": 1010068736.0,
+ "480": 1047611200.0,
+ "485": 1029727936.0,
+ "490": 1044670848.0,
+ "495": 1025233664.0,
+ "500": 1037468352.0,
+ "505": 1032184704.0,
+ "510": 1042856576.0,
+ "515": 1026163776.0,
+ "520": 1013413696.0,
+ "525": 1035151488.0,
+ "530": 1016379648.0,
+ "535": 1040117248.0,
+ "540": 1035057536.0,
+ "545": 1032119488.0,
+ "550": 1018677824.0,
+ "555": 1008643072.0,
+ "560": 1011932800.0,
+ "565": 1041829504.0,
+ "570": 1034947328.0,
+ "575": 1010204480.0,
+ "580": 1032215296.0,
+ "585": 1041268096.0,
+ "590": 1038873664.0,
+ "595": 1035748480.0,
+ "600": 1023778688.0,
+ "605": 1032300288.0,
+ "610": 1037755136.0,
+ "615": 1005981056.0,
+ "620": 1040413568.0,
+ "625": 1045215488.0,
+ "630": 1034421440.0,
+ "635": 1028528768.0,
+ "640": 1022651392.0,
+ "645": 1035881856.0,
+ "650": 1009261696.0,
+ "655": 997764416.0,
+ "660": 1029716992.0,
+ "665": 1025539968.0,
+ "670": 1048818624.0,
+ "675": 1025209728.0,
+ "680": 1019345472.0,
+ "685": 1027839360.0,
+ "690": 1029237184.0,
+ "695": 1040031424.0,
+ "700": 1042039040.0,
+ "705": 1034390144.0,
+ "710": 1020449024.0,
+ "715": 1031479552.0,
+ "720": 1040282048.0,
+ "725": 1023287424.0,
+ "730": 1022800256.0,
+ "735": 1025093376.0,
+ "740": 1038389440.0,
+ "745": 1045213120.0,
+ "750": 1013188608.0,
+ "755": 1031651712.0,
+ "760": 1032791552.0,
+ "765": 1027143424.0,
+ "770": 1023975296.0,
+ "775": 1025902976.0,
+ "780": 1038173312.0,
+ "785": 1025494784.0,
+ "790": 1040818176.0,
+ "795": 1032539904.0,
+ "800": 1039600384.0,
+ "805": 1024324800.0,
+ "810": 1034732352.0,
+ "815": 1036008704.0,
+ "820": 1035678592.0,
+ "825": 1051382528.0,
+ "830": 1035414208.0,
+ "835": 1022555392.0,
+ "840": 1036883264.0,
+ "845": 1025708160.0,
+ "850": 1048536960.0,
+ "855": 1014993856.0,
+ "860": 1033106304.0,
+ "865": 1031549568.0,
+ "870": 1040909440.0,
+ "875": 1023946368.0,
+ "880": 1028403712.0,
+ "885": 1054413568.0,
+ "890": 1019544576.0,
+ "895": 1045197312.0,
+ "900": 1031780480.0,
+ "905": 1020978560.0,
+ "910": 1031393152.0,
+ "915": 1032934144.0,
+ "920": 1038467264.0,
+ "925": 1026762048.0,
+ "930": 1025385984.0,
+ "935": 1031132672.0,
+ "940": 1057939840.0,
+ "945": 1029829760.0,
+ "950": 1014419008.0,
+ "955": 1032181248.0,
+ "960": 1026159744.0,
+ "965": 1062685568.0,
+ "970": 1030102400.0,
+ "975": 1036911360.0,
+ "980": 1027056384.0,
+ "985": 1030683584.0,
+ "990": 1020683776.0,
+ "995": 1042308608.0,
+ "1000": 1036839168.0,
+ "1005": 1050213632.0,
+ "1010": 1023809536.0,
+ "1015": 1020547264.0,
+ "1020": 1042594304.0,
+ "1025": 1037950336.0,
+ "1030": 1049217536.0,
+ "1035": 1012491392.0,
+ "1040": 1023099840.0,
+ "1045": 1039528064.0,
+ "1050": 1026833280.0,
+ "1055": 1034867904.0,
+ "1060": 1046134848.0,
+ "1065": 1036810048.0,
+ "1070": 1020002432.0,
+ "1075": 1025349632.0,
+ "1080": 1014985856.0,
+ "1085": 1030015616.0,
+ "1090": 1029068992.0,
+ "1095": 1020317056.0,
+ "1100": 1039842688.0,
+ "1105": 1048607424.0,
+ "1110": 1020710592.0,
+ "1115": 1024790016.0,
+ "1120": 1061902720.0,
+ "1125": 1043318144.0,
+ "1130": 1031227392.0,
+ "1135": 1041368448.0,
+ "1140": 1021493440.0,
+ "1145": 1051703296.0,
+ "1150": 1035597504.0,
+ "1155": 1029598592.0,
+ "1160": 1042572032.0,
+ "1165": 1026817856.0,
+ "1170": 1018008704.0,
+ "1175": 1033691392.0,
+ "1180": 1035640192.0,
+ "1185": 1023936320.0,
+ "1190": 1033167232.0,
+ "1195": 1024236096.0,
+ "1200": 1039124096.0,
+ "1205": 1031747840.0,
+ "1210": 1053257728.0,
+ "1215": 1024622976.0,
+ "1220": 1009048832.0,
+ "1225": 1036685440.0,
+ "1230": 1041264768.0,
+ "1235": 1053980544.0,
+ "1240": 1030364032.0,
+ "1245": 1017691648.0,
+ "1250": 1022780928.0,
+ "1255": 1033445248.0,
+ "1260": 1034291200.0,
+ "1265": 1034011968.0,
+ "1270": 1037329920.0,
+ "1275": 1029353216.0,
+ "1280": 1046496320.0,
+ "1285": 1028291712.0,
+ "1290": 1036584192.0,
+ "1295": 1032427328.0,
+ "1300": 1033072192.0,
+ "1305": 1030033408.0,
+ "1310": 1051270528.0,
+ "1315": 1035378688.0,
+ "1320": 1028270976.0,
+ "1325": 1049979520.0,
+ "1330": 1030140992.0,
+ "1335": 1031172800.0,
+ "1340": 1012765504.0,
+ "1345": 1044646656.0,
+ "1350": 1034963840.0,
+ "1355": 1033629440.0,
+ "1360": 1036689536.0,
+ "1365": 1038597248.0,
+ "1370": 1039858624.0,
+ "1375": 1034124032.0,
+ "1380": 1022892672.0,
+ "1385": 1018090880.0,
+ "1390": 1049059456.0,
+ "1395": 1034876928.0,
+ "1400": 1035005184.0,
+ "1405": 1034137920.0,
+ "1410": 1036374528.0,
+ "1415": 1043582848.0,
+ "1420": 1026118400.0,
+ "1425": 1033327360.0,
+ "1430": 1012813824.0,
+ "1435": 1038400320.0,
+ "1440": 1020978176.0,
+ "1445": 1032467264.0,
+ "1450": 1014044928.0,
+ "1455": 1011679488.0,
+ "1460": 1043281536.0,
+ "1465": 1014368128.0,
+ "1470": 1020661120.0,
+ "1475": 1030237440.0,
+ "1480": 1029376128.0,
+ "1485": 1023004160.0,
+ "1490": 1026789440.0,
+ "1495": 1021820672.0,
+ "1500": 1027182400.0,
+ "1505": 1034888320.0,
+ "1510": 1014403968.0,
+ "1515": 1042142336.0,
+ "1520": 1025800064.0,
+ "1525": 1036343808.0,
+ "1530": 1039954048.0,
+ "1535": 1047645952.0,
+ "1540": 1043545792.0,
+ "1545": 1034049600.0,
+ "1550": 1016115712.0,
+ "1555": 1015578560.0,
+ "1560": 1055028480.0,
+ "1565": 1015599744.0,
+ "1570": 1018251008.0,
+ "1575": 1032521600.0,
+ "1580": 1012990912.0,
+ "1585": 1025333376.0,
+ "1590": 1034133824.0,
+ "1595": 1057399552.0,
+ "1600": 1026873984.0,
+ "1605": 1019999680.0,
+ "1610": 1031273984.0,
+ "1615": 1035281408.0,
+ "1620": 1018022592.0,
+ "1625": 1028277888.0,
+ "1630": 1027210368.0,
+ "1635": 1023805312.0,
+ "1640": 1034126656.0,
+ "1645": 1021819520.0,
+ "1650": 1015268608.0,
+ "1655": 1018286848.0,
+ "1660": 1047988544.0,
+ "1665": 1027067136.0,
+ "1670": 1048224192.0,
+ "1675": 1021109376.0,
+ "1680": 1043293696.0,
+ "1685": 1052725760.0,
+ "1690": 1026729216.0,
+ "1695": 1040390656.0,
+ "1700": 1018042112.0,
+ "1705": 1020486016.0,
+ "1710": 1021030528.0,
+ "1715": 1026937984.0,
+ "1720": 1028357504.0,
+ "1725": 1034368768.0,
+ "1730": 1013699200.0,
+ "1735": 1018435712.0,
+ "1740": 1057263744.0,
+ "1745": 1029266880.0,
+ "1750": 1024363904.0,
+ "1755": 1029975232.0,
+ "1760": 1022199808.0,
+ "1765": 1040483520.0,
+ "1770": 1029674368.0,
+ "1775": 1046202368.0,
+ "1780": 1021961472.0,
+ "1785": 1035115264.0,
+ "1790": 1028268928.0,
+ "1795": 1031028864.0,
+ "1800": 1028305536.0,
+ "1805": 1025675520.0,
+ "1810": 1021561152.0,
+ "1815": 1033444416.0,
+ "1820": 1034891840.0,
+ "1825": 1020212352.0,
+ "1830": 1013890816.0,
+ "1835": 1031388416.0,
+ "1840": 1040396288.0,
+ "1845": 1034833856.0,
+ "1850": 1014487168.0,
+ "1855": 1019425408.0,
+ "1860": 1019576512.0,
+ "1865": 1035946624.0,
+ "1870": 1026248576.0,
+ "1875": 1031529984.0,
+ "1880": 1011596416.0,
+ "1885": 1041070464.0,
+ "1890": 1035005888.0,
+ "1895": 1028963584.0,
+ "1900": 1034003904.0,
+ "1905": 1027130560.0,
+ "1910": 1029222528.0,
+ "1915": 1030497792.0,
+ "1920": 1042926272.0,
+ "1925": 1038424960.0,
+ "1930": 1019309376.0,
+ "1935": 1032540800.0,
+ "1940": 1027811072.0,
+ "1945": 1034210432.0,
+ "1950": 1006041664.0,
+ "1955": 1032584576.0,
+ "1960": 1015725312.0,
+ "1965": 1029093888.0,
+ "1970": 1021560256.0,
+ "1975": 1034051968.0,
+ "1980": 1029371520.0,
+ "1985": 1027789952.0,
+ "1990": 1020952192.0,
+ "1995": 1010428800.0,
+ "2000": 1039621760.0,
+ "2005": 1001491840.0,
+ "2010": 1020428928.0,
+ "2015": 1032040192.0,
+ "2020": 1036303744.0,
+ "2025": 1037177920.0,
+ "2030": 1029773952.0,
+ "2035": 1040340928.0,
+ "2040": 1030119296.0,
+ "2045": 1032704768.0,
+ "2050": 1008020416.0,
+ "2055": 1045728960.0,
+ "2060": 1028148096.0,
+ "2065": 1038804992.0,
+ "2070": 1045650432.0,
+ "2075": 1035241984.0,
+ "2080": 1022886464.0,
+ "2085": 1024820992.0,
+ "2090": 1034369536.0,
+ "2095": 1005225344.0,
+ "2100": 1034647424.0,
+ "2105": 1035588032.0,
+ "2110": 1030691072.0,
+ "2115": 1029803776.0,
+ "2120": 1018851392.0,
+ "2125": 1021868992.0,
+ "2130": 1026643456.0,
+ "2135": 1053284480.0,
+ "2140": 1017067264.0,
+ "2145": 1019638976.0,
+ "2150": 1037135360.0,
+ "2155": 1033308096.0,
+ "2160": 1049042304.0,
+ "2165": 1039687488.0,
+ "2170": 1020313344.0,
+ "2175": 1027344128.0,
+ "2180": 1041707136.0,
+ "2185": 1028900864.0,
+ "2190": 1029316224.0,
+ "2195": 1028950720.0,
+ "2200": 1039643776.0,
+ "2205": 1036977152.0,
+ "2210": 1031747200.0,
+ "2215": 1021410560.0,
+ "2220": 1020915968.0,
+ "2225": 1033408256.0,
+ "2230": 1014206336.0,
+ "2235": 1029401216.0,
+ "2240": 1029888000.0,
+ "2245": 1026011776.0,
+ "2250": 1046273664.0,
+ "2255": 1032956928.0,
+ "2260": 1047501120.0,
+ "2265": 1023724800.0,
+ "2270": 1022571648.0,
+ "2275": 1028542016.0,
+ "2280": 1034979840.0,
+ "2285": 1031823680.0,
+ "2290": 1038656128.0,
+ "2295": 1028821056.0,
+ "2300": 1034456000.0,
+ "2305": 1032319936.0,
+ "2310": 1013591168.0,
+ "2315": 1048186624.0,
+ "2320": 1035214336.0,
+ "2325": 1046969856.0,
+ "2330": 1014701056.0,
+ "2335": 1027388160.0,
+ "2340": 1036741376.0,
+ "2345": 1020192000.0,
+ "2350": 1031021376.0,
+ "2355": 1037479936.0,
+ "2360": 1032613248.0,
+ "2365": 1028047104.0,
+ "2370": 1021009280.0,
+ "2375": 1022915904.0,
+ "2380": 1048561920.0,
+ "2385": 1044144704.0,
+ "2390": 1021990784.0,
+ "2395": 1020600192.0,
+ "2400": 1026936896.0,
+ "2405": 1038390528.0,
+ "2410": 1045400576.0,
+ "2415": 1048461184.0,
+ "2420": 1032233408.0,
+ "2425": 1029567744.0,
+ "2430": 1030391872.0,
+ "2435": 1029223104.0,
+ "2440": 1029173824.0,
+ "2445": 1033137792.0,
+ "2450": 1038562560.0,
+ "2455": 1034727232.0,
+ "2460": 1039988992.0,
+ "2465": 1032506368.0,
+ "2470": 1024148608.0,
+ "2475": 1016545216.0,
+ "2480": 1023619072.0,
+ "2485": 1021036544.0,
+ "2490": 1035925120.0,
+ "2495": 1032973952.0,
+ "2500": 1028114688.0,
+ "2505": 1015390720.0,
+ "2510": 1030972928.0,
+ "2515": 1025704320.0,
+ "2520": 1033331712.0,
+ "2525": 1029697664.0,
+ "2530": 1023991808.0,
+ "2535": 1071074624.0,
+ "2540": 1024544128.0,
+ "2545": 1033802752.0,
+ "2550": 1029453696.0,
+ "2555": 1029188224.0,
+ "2560": 1018120704.0,
+ "2565": 1031604032.0,
+ "2570": 1022851264.0,
+ "2575": 1026507968.0,
+ "2580": 1038627584.0,
+ "2585": 1025905728.0,
+ "2590": 1026105600.0,
+ "2595": 1046626688.0,
+ "2600": 1031110144.0,
+ "2605": 1001915648.0,
+ "2610": 1028426624.0,
+ "2615": 1025569344.0,
+ "2620": 1038655040.0,
+ "2625": 1027002624.0,
+ "2630": 1036836480.0,
+ "2635": 1021202688.0,
+ "2640": 1021870848.0,
+ "2645": 1039158080.0,
+ "2650": 1025949824.0,
+ "2655": 1013262080.0,
+ "2660": 1032650752.0,
+ "2665": 1035222912.0,
+ "2670": 1036443840.0,
+ "2675": 1039301120.0,
+ "2680": 1041667456.0,
+ "2685": 1034570816.0,
+ "2690": 1058876672.0,
+ "2695": 1019885952.0,
+ "2700": 1062632576.0,
+ "2705": 1035381760.0,
+ "2710": 1019548288.0,
+ "2715": 1031890048.0,
+ "2720": 1016410368.0,
+ "2725": 1040598912.0,
+ "2730": 1019590656.0,
+ "2735": 1030895104.0,
+ "2740": 1029296512.0,
+ "2745": 1040693952.0,
+ "2750": 1023886336.0,
+ "2755": 1011872064.0,
+ "2760": 1027690880.0,
+ "2765": 1030887680.0,
+ "2770": 1033124224.0,
+ "2775": 1026335616.0,
+ "2780": 1033690880.0,
+ "2785": 1024594432.0,
+ "2790": 1033740160.0,
+ "2795": 1045954176.0,
+ "2800": 1040291712.0,
+ "2805": 1019952256.0,
+ "2810": 1031454208.0,
+ "2815": 1030936896.0,
+ "2820": 1037860992.0,
+ "2825": 1041687680.0,
+ "2830": 1030465920.0,
+ "2835": 1013512576.0,
+ "2840": 1031453056.0,
+ "2845": 1030136576.0,
+ "2850": 1026623232.0,
+ "2855": 1024709760.0,
+ "2860": 1031704896.0,
+ "2865": 1027433600.0,
+ "2870": 1026694528.0,
+ "2875": 1012782400.0,
+ "2880": 1038305280.0,
+ "2885": 1017906944.0,
+ "2890": 1044205440.0,
+ "2895": 1036465408.0,
+ "2900": 1030659008.0,
+ "2905": 1035963648.0,
+ "2910": 1038723456.0,
+ "2915": 1039389888.0,
+ "2920": 1034787584.0,
+ "2925": 1043274560.0,
+ "2930": 1038233920.0,
+ "2935": 1021227136.0,
+ "2940": 1042312896.0,
+ "2945": 1045238336.0,
+ "2950": 1047530816.0,
+ "2955": 1034179392.0,
+ "2960": 1020896640.0,
+ "2965": 1027311936.0,
+ "2970": 1038801152.0,
+ "2975": 1034012096.0,
+ "2980": 1049594752.0,
+ "2985": 1034852416.0,
+ "2990": 1026014080.0,
+ "2995": 1034924096.0,
+ "3000": 1039024512.0,
+ "3005": 1038163584.0,
+ "3010": 1010912768.0,
+ "3015": 1044980928.0,
+ "3020": 1034056064.0,
+ "3025": 1037768832.0,
+ "3030": 1027727872.0,
+ "3035": 1041826432.0,
+ "3040": 1035317376.0,
+ "3045": 1027261824.0,
+ "3050": 1029713280.0,
+ "3055": 1028036096.0,
+ "3060": 1049983104.0,
+ "3065": 1024072192.0,
+ "3070": 1011550656.0,
+ "3075": 1042852224.0,
+ "3080": 1036100352.0,
+ "3085": 1030394752.0,
+ "3090": 1035267456.0,
+ "3095": 1013807424.0,
+ "3100": 1030148992.0,
+ "3105": 1017614592.0,
+ "3110": 1033375232.0,
+ "3115": 1023743680.0,
+ "3120": 1024883520.0,
+ "3125": 1046542592.0,
+ "3130": 1024682368.0,
+ "3135": 1025729216.0,
+ "3140": 1043783424.0,
+ "3145": 1044378368.0,
+ "3150": 1016489088.0,
+ "3155": 1042492800.0,
+ "3160": 1026839616.0,
+ "3165": 1031204672.0,
+ "3170": 1024337536.0,
+ "3175": 1024374016.0,
+ "3180": 1018209344.0,
+ "3185": 1034359808.0,
+ "3190": 1019227328.0,
+ "3195": 1028430848.0,
+ "3200": 1036084736.0,
+ "3205": 1016080704.0,
+ "3210": 1034114048.0,
+ "3215": 1031354816.0,
+ "3220": 1040838272.0,
+ "3225": 1022841408.0,
+ "3230": 1033262208.0,
+ "3235": 1019981184.0,
+ "3240": 1038135232.0,
+ "3245": 1031649152.0,
+ "3250": 1022395520.0,
+ "3255": 1032881920.0,
+ "3260": 1037756160.0,
+ "3265": 1021628672.0,
+ "3270": 1031249024.0,
+ "3275": 1038466240.0,
+ "3280": 1023241984.0,
+ "3285": 1031622016.0,
+ "3290": 1045253504.0,
+ "3295": 1043183232.0,
+ "3300": 1035089664.0,
+ "3305": 1042666304.0,
+ "3310": 1058096256.0,
+ "3315": 1024287552.0,
+ "3320": 1046020608.0,
+ "3325": 1023185152.0,
+ "3330": 1048042432.0,
+ "3335": 1036695936.0,
+ "3340": 1042128384.0,
+ "3345": 1030903936.0,
+ "3350": 1020624192.0,
+ "3355": 1025965376.0,
+ "3360": 1030309760.0,
+ "3365": 1031176960.0,
+ "3370": 1036459136.0,
+ "3375": 1023479040.0,
+ "3380": 1032387776.0,
+ "3385": 1038085760.0,
+ "3390": 1052816512.0,
+ "3395": 1012095488.0,
+ "3400": 1019214912.0,
+ "3405": 1021785920.0,
+ "3410": 1028439296.0,
+ "3415": 1058226176.0,
+ "3420": 1033497472.0,
+ "3425": 1029585280.0,
+ "3430": 1021155712.0,
+ "3435": 1034998144.0,
+ "3440": 1017966208.0,
+ "3445": 1025542272.0,
+ "3450": 1032259584.0,
+ "3455": 1036265472.0,
+ "3460": 1052076544.0,
+ "3465": 1027119168.0,
+ "3470": 1043734400.0,
+ "3475": 1033269312.0,
+ "3480": 1026625664.0,
+ "3485": 1029219392.0,
+ "3490": 1041046976.0,
+ "3495": 1019256448.0,
+ "3500": 1032066112.0,
+ "3505": 1025757440.0,
+ "3510": 1044371584.0,
+ "3515": 1013822720.0,
+ "3520": 1021851392.0,
+ "3525": 1032180672.0,
+ "3530": 1029793920.0,
+ "3535": 1034573952.0,
+ "3540": 1017737344.0,
+ "3545": 1035664384.0,
+ "3550": 1024539200.0,
+ "3555": 1035870720.0,
+ "3560": 1029743040.0,
+ "3565": 1028904320.0,
+ "3570": 1046034304.0,
+ "3575": 1039189504.0,
+ "3580": 1010844096.0,
+ "3585": 1031742464.0,
+ "3590": 1041454592.0,
+ "3595": 1037642240.0,
+ "3600": 1032767744.0,
+ "3605": 1045827008.0,
+ "3610": 1039240576.0,
+ "3615": 1036875008.0,
+ "3620": 1026934400.0,
+ "3625": 1033936256.0,
+ "3630": 1017587072.0,
+ "3635": 1026634176.0,
+ "3640": 1039534720.0,
+ "3645": 1022662016.0,
+ "3650": 1036845888.0,
+ "3655": 1023994560.0,
+ "3660": 1014993792.0,
+ "3665": 1026122112.0,
+ "3670": 1041676544.0,
+ "3675": 1033256704.0,
+ "3680": 1015358016.0,
+ "3685": 1029128064.0,
+ "3690": 1026208704.0,
+ "3695": 1043804736.0,
+ "3700": 1028618624.0,
+ "3705": 1049489024.0,
+ "3710": 1027184256.0,
+ "3715": 1016139648.0,
+ "3720": 1040824128.0,
+ "3725": 1032767616.0,
+ "3730": 1030926080.0,
+ "3735": 1019014144.0,
+ "3740": 1023829376.0,
+ "3745": 1046292480.0,
+ "3750": 1034466816.0,
+ "3755": 1032094272.0,
+ "3760": 1019371264.0,
+ "3765": 1031922176.0,
+ "3770": 1026683136.0,
+ "3775": 1035713920.0,
+ "3780": 1030675584.0,
+ "3785": 1027213184.0,
+ "3790": 1019588224.0,
+ "3795": 1030310016.0,
+ "3800": 1035620480.0,
+ "3805": 1035427200.0,
+ "3810": 1033298752.0,
+ "3815": 1033993728.0,
+ "3820": 1041110976.0,
+ "3825": 1024538752.0,
+ "3830": 1037634496.0,
+ "3835": 1040351872.0,
+ "3840": 1023450688.0,
+ "3845": 1048471808.0,
+ "3850": 1052492800.0,
+ "3855": 1028912064.0,
+ "3860": 1019536768.0,
+ "3865": 1035491520.0,
+ "3870": 1028495552.0,
+ "3875": 1041168704.0,
+ "3880": 1048859136.0,
+ "3885": 1027729984.0,
+ "3890": 1027491904.0,
+ "3895": 1034195584.0,
+ "3900": 1027651072.0,
+ "3905": 1027980928.0,
+ "3910": 1041576832.0,
+ "3915": 1043999360.0,
+ "3920": 1041068096.0,
+ "3925": 1030840960.0,
+ "3930": 1027076544.0,
+ "3935": 1033786880.0,
+ "3940": 1042279744.0,
+ "3945": 1036251392.0,
+ "3950": 1021437376.0,
+ "3955": 1036308992.0,
+ "3960": 1024188672.0,
+ "3965": 1027070464.0,
+ "3970": 1015988672.0,
+ "3975": 1041426624.0,
+ "3980": 1032459008.0,
+ "3985": 1037685056.0,
+ "3990": 1038688384.0,
+ "3995": 1023658624.0,
+ "4000": 1054414336.0,
+ "4005": 1029987392.0,
+ "4010": 1025141888.0,
+ "4015": 1030982784.0,
+ "4020": 1018476288.0,
+ "4025": 1027127232.0,
+ "4030": 1010311680.0,
+ "4035": 1038645120.0,
+ "4040": 1022260800.0,
+ "4045": 1025041408.0,
+ "4050": 1032353216.0,
+ "4055": 1022425216.0,
+ "4060": 1024525824.0,
+ "4065": 1032875264.0,
+ "4070": 1027794688.0,
+ "4075": 1025600064.0,
+ "4080": 1029370944.0,
+ "4085": 1020827520.0,
+ "4090": 1033328384.0,
+ "4095": 1024146176.0,
+ "4100": 1040953152.0,
+ "4105": 1027269888.0,
+ "4110": 1038795648.0,
+ "4115": 1023500928.0,
+ "4120": 1038947456.0,
+ "4125": 1048278720.0,
+ "4130": 1021494016.0,
+ "4135": 1034574848.0,
+ "4140": 1034618176.0,
+ "4145": 1044451200.0,
+ "4150": 1000356736.0,
+ "4155": 1028367552.0,
+ "4160": 1024247232.0,
+ "4165": 1033692544.0,
+ "4170": 1018891648.0,
+ "4175": 1026497792.0,
+ "4180": 1045412288.0,
+ "4185": 1033634944.0,
+ "4190": 1029578752.0,
+ "4195": 1038781632.0,
+ "4200": 1025106944.0,
+ "4205": 1019079296.0,
+ "4210": 1029564416.0,
+ "4215": 1032273280.0,
+ "4220": 1026245376.0,
+ "4225": 1031929856.0,
+ "4230": 1030274496.0,
+ "4235": 1027607552.0,
+ "4240": 1031483840.0,
+ "4245": 1028769152.0,
+ "4250": 1026991104.0,
+ "4255": 1021243008.0,
+ "4260": 1042085888.0,
+ "4265": 1025413952.0,
+ "4270": 1030174208.0,
+ "4275": 1012475648.0,
+ "4280": 1044508416.0,
+ "4285": 1019902720.0,
+ "4290": 1033062144.0,
+ "4295": 1033600128.0,
+ "4300": 1031643008.0,
+ "4305": 1023852352.0,
+ "4310": 1021572416.0,
+ "4315": 1047226368.0,
+ "4320": 1026524416.0,
+ "4325": 1005869568.0,
+ "4330": 1037671488.0,
+ "4335": 1022010176.0,
+ "4340": 1029013632.0,
+ "4345": 1033477632.0,
+ "4350": 1036891136.0,
+ "4355": 1026812416.0,
+ "4360": 1022941952.0,
+ "4365": 1028783296.0,
+ "4370": 1029627776.0,
+ "4375": 1042200832.0,
+ "4380": 1016104320.0,
+ "4385": 1045554944.0,
+ "4390": 1026274816.0,
+ "4395": 1029800704.0,
+ "4400": 1047369216.0,
+ "4405": 1029300672.0,
+ "4410": 1033427328.0,
+ "4415": 1028302656.0,
+ "4420": 1028152832.0,
+ "4425": 1033579008.0,
+ "4430": 1031378240.0,
+ "4435": 1028575680.0,
+ "4440": 1033127808.0,
+ "4445": 1028295872.0,
+ "4450": 1052215552.0,
+ "4455": 1026288896.0,
+ "4460": 1034890752.0,
+ "4465": 1031728832.0,
+ "4470": 1035449792.0,
+ "4475": 1036975360.0,
+ "4480": 1025122176.0,
+ "4485": 1034108864.0,
+ "4490": 1024634112.0,
+ "4495": 1047977728.0,
+ "4500": 1024712448.0,
+ "4505": 1038853632.0,
+ "4510": 1043728128.0,
+ "4515": 1044281344.0,
+ "4520": 1036876864.0,
+ "4525": 1058077632.0,
+ "4530": 1030976320.0,
+ "4535": 1032597184.0,
+ "4540": 1036432128.0,
+ "4545": 1025729344.0,
+ "4550": 1021752448.0,
+ "4555": 1037548544.0,
+ "4560": 1020104768.0,
+ "4565": 1036058624.0,
+ "4570": 1020504704.0,
+ "4575": 1050414336.0,
+ "4580": 1010442816.0,
+ "4585": 1022964736.0,
+ "4590": 1039714176.0,
+ "4595": 1023279040.0,
+ "4600": 1042481280.0,
+ "4605": 1039750464.0,
+ "4610": 1046108416.0,
+ "4615": 1018004864.0,
+ "4620": 1044737280.0,
+ "4625": 1030483072.0,
+ "4630": 1027264320.0,
+ "4635": 1026999168.0,
+ "4640": 1034904640.0,
+ "4645": 1036424704.0,
+ "4650": 1033715840.0,
+ "4655": 1035464192.0,
+ "4660": 1035328128.0,
+ "4665": 1020268800.0,
+ "4670": 1020060928.0,
+ "4675": 1054851840.0,
+ "4680": 1024898304.0,
+ "4685": 1027824000.0,
+ "4690": 1034453376.0,
+ "4695": 1039154048.0,
+ "4700": 1038869504.0,
+ "4705": 1027658176.0,
+ "4710": 1020526592.0,
+ "4715": 1031831424.0,
+ "4720": 1030303616.0,
+ "4725": 1030303872.0,
+ "4730": 1044100544.0,
+ "4735": 1046136704.0,
+ "4740": 1036180736.0,
+ "4745": 1039047424.0,
+ "4750": 1031794176.0,
+ "4755": 1047727104.0,
+ "4760": 1026181120.0,
+ "4765": 1034699008.0,
+ "4770": 1036524672.0,
+ "4775": 1029378944.0,
+ "4780": 1028547584.0,
+ "4785": 1028418176.0,
+ "4790": 1019624576.0,
+ "4795": 1033063168.0,
+ "4800": 1051871744.0,
+ "4805": 1015416960.0,
+ "4810": 1029457280.0,
+ "4815": 1009575872.0,
+ "4820": 1041054016.0,
+ "4825": 1026712832.0,
+ "4830": 1020455168.0,
+ "4835": 1051312000.0,
+ "4840": 1019460736.0,
+ "4845": 1032318464.0,
+ "4850": 1036798080.0,
+ "4855": 1031056640.0,
+ "4860": 1033134848.0,
+ "4865": 1032067456.0,
+ "4870": 1049835648.0,
+ "4875": 1025114304.0,
+ "4880": 1048481024.0,
+ "4885": 1016857344.0,
+ "4890": 1037321472.0,
+ "4895": 1024327232.0,
+ "4900": 1043376640.0,
+ "4905": 1033401088.0,
+ "4910": 1032833472.0,
+ "4915": 1016893952.0,
+ "4920": 1022298368.0,
+ "4925": 1034969600.0,
+ "4930": 1034633344.0,
+ "4935": 1025889664.0,
+ "4940": 1048402624.0,
+ "4945": 1025252096.0,
+ "4950": 1024212544.0,
+ "4955": 1007489792.0,
+ "4960": 1040217728.0,
+ "4965": 1018778624.0,
+ "4970": 1014277632.0,
+ "4975": 1038029696.0,
+ "4980": 1020920832.0,
+ "4985": 1029050368.0,
+ "4990": 1028398336.0,
+ "4995": 1032023744.0,
+ "5000": 1039793792.0,
+ "5005": 1024355328.0,
+ "5010": 1029150976.0,
+ "5015": 1021811072.0,
+ "5020": 1023510400.0,
+ "5025": 1037607808.0,
+ "5030": 1041950976.0,
+ "5035": 1047133312.0,
+ "5040": 1060959808.0,
+ "5045": 1032111232.0,
+ "5050": 1029537536.0,
+ "5055": 1024555776.0,
+ "5060": 1035287104.0,
+ "5065": 1021208960.0,
+ "5070": 1035759104.0,
+ "5075": 1015775232.0,
+ "5080": 1027043712.0,
+ "5085": 1021795840.0,
+ "5090": 1034977664.0,
+ "5095": 1015504320.0,
+ "5100": 1032260608.0,
+ "5105": 1017984960.0,
+ "5110": 1019589120.0,
+ "5115": 1036067328.0,
+ "5120": 1032697728.0,
+ "5125": 1019079936.0,
+ "5130": 1033409408.0,
+ "5135": 1041206208.0,
+ "5140": 1026262144.0,
+ "5145": 1033710080.0,
+ "5150": 1022047616.0,
+ "5155": 1032268160.0,
+ "5160": 1039629696.0,
+ "5165": 1031578816.0,
+ "5170": 1035559040.0,
+ "5175": 1026119104.0,
+ "5180": 1030320128.0,
+ "5185": 1024499776.0,
+ "5190": 1019495424.0,
+ "5195": 1035629760.0,
+ "5200": 1016909440.0,
+ "5205": 1013438720.0,
+ "5210": 1049398208.0,
+ "5215": 1030835328.0,
+ "5220": 1025281152.0,
+ "5225": 1035243008.0,
+ "5230": 1025932928.0,
+ "5235": 1025122560.0,
+ "5240": 1046311680.0,
+ "5245": 1022743680.0,
+ "5250": 1027065472.0,
+ "5255": 1023890944.0,
+ "5260": 1033824384.0,
+ "5265": 1045736832.0,
+ "5270": 1052503936.0,
+ "5275": 1033021696.0,
+ "5280": 1030076288.0,
+ "5285": 1025215296.0,
+ "5290": 1026578816.0,
+ "5295": 1032656640.0,
+ "5300": 1024370816.0,
+ "5305": 1029639040.0,
+ "5310": 1033201280.0,
+ "5315": 1032992768.0,
+ "5320": 1019525056.0,
+ "5325": 1022721216.0,
+ "5330": 1021339136.0,
+ "5335": 1039279680.0,
+ "5340": 1037223424.0,
+ "5345": 1039191936.0,
+ "5350": 1023703680.0,
+ "5355": 1029940096.0,
+ "5360": 1047048320.0,
+ "5365": 1037428864.0,
+ "5370": 1024385792.0,
+ "5375": 1042074304.0,
+ "5380": 1020371712.0,
+ "5385": 1021769088.0,
+ "5390": 1035135552.0,
+ "5395": 1049658240.0,
+ "5400": 1026018176.0,
+ "5405": 1036455808.0,
+ "5410": 1027638784.0,
+ "5415": 1042289664.0,
+ "5420": 1039946368.0,
+ "5425": 1028384128.0,
+ "5430": 1043802688.0,
+ "5435": 1032657280.0,
+ "5440": 1033387072.0,
+ "5445": 1034147648.0,
+ "5450": 1025303168.0,
+ "5455": 1034083008.0,
+ "5460": 1026815872.0,
+ "5465": 1027403712.0,
+ "5470": 1028971648.0,
+ "5475": 1037236736.0,
+ "5480": 1023833728.0,
+ "5485": 1019189376.0,
+ "5490": 1030894656.0,
+ "5495": 1029403008.0,
+ "5500": 1032684800.0,
+ "5505": 1018277824.0,
+ "5510": 1023991040.0,
+ "5515": 1025159552.0,
+ "5520": 1039530240.0,
+ "5525": 1018027520.0,
+ "5530": 1037666176.0,
+ "5535": 1031602176.0,
+ "5540": 1027568000.0,
+ "5545": 1033214848.0,
+ "5550": 1032119040.0,
+ "5555": 1044806144.0,
+ "5560": 1028514432.0,
+ "5565": 1029689600.0,
+ "5570": 1042031872.0,
+ "5575": 1025381760.0,
+ "5580": 1023719488.0,
+ "5585": 1044097408.0,
+ "5590": 1041322880.0,
+ "5595": 1031553408.0,
+ "5600": 1023403456.0,
+ "5605": 1040119424.0,
+ "5610": 1034091072.0,
+ "5615": 1021045888.0,
+ "5620": 1031008000.0,
+ "5625": 1030192192.0,
+ "5630": 1023505152.0,
+ "5635": 1026687168.0,
+ "5640": 1034592640.0,
+ "5645": 1018658944.0,
+ "5650": 1052382336.0,
+ "5655": 1048935040.0,
+ "5660": 1050081152.0,
+ "5665": 1033961088.0,
+ "5670": 1033753856.0,
+ "5675": 1025395968.0,
+ "5680": 1039381888.0,
+ "5685": 1033059584.0,
+ "5690": 1031467136.0,
+ "5695": 1021949312.0,
+ "5700": 1038069056.0,
+ "5705": 1043687808.0,
+ "5710": 1057235712.0,
+ "5715": 1014466752.0,
+ "5720": 1021261440.0,
+ "5725": 1041824640.0,
+ "5730": 1039458304.0,
+ "5735": 1025131392.0,
+ "5740": 1026048896.0,
+ "5745": 1036993280.0,
+ "5750": 1044555648.0,
+ "5755": 1011864704.0,
+ "5760": 1028393344.0,
+ "5765": 1028249408.0,
+ "5770": 1021534336.0,
+ "5775": 1051213696.0,
+ "5780": 1034986624.0,
+ "5785": 1037516864.0,
+ "5790": 1016960512.0,
+ "5795": 1027876608.0,
+ "5800": 1029783616.0,
+ "5805": 1050698816.0,
+ "5810": 1018481024.0,
+ "5815": 1036127232.0,
+ "5820": 1048411136.0,
+ "5825": 1030981504.0,
+ "5830": 1031575936.0,
+ "5835": 1034048448.0,
+ "5840": 1039847168.0,
+ "5845": 1021750272.0,
+ "5850": 1029811712.0,
+ "5855": 1038792704.0,
+ "5860": 1031439424.0,
+ "5865": 1026402048.0,
+ "5870": 1029863040.0,
+ "5875": 1032845312.0,
+ "5880": 1032679296.0,
+ "5885": 1024579520.0,
+ "5890": 1026802176.0,
+ "5895": 1015799424.0,
+ "5900": 1049709952.0,
+ "5905": 1025655744.0,
+ "5910": 1019154624.0,
+ "5915": 1042742272.0,
+ "5920": 1028049920.0,
+ "5925": 1034020352.0,
+ "5930": 1030965696.0,
+ "5935": 1038106752.0,
+ "5940": 1019175616.0,
+ "5945": 1025132672.0,
+ "5950": 1035532928.0,
+ "5955": 1050441024.0,
+ "5960": 1024551552.0,
+ "5965": 1029927040.0,
+ "5970": 1016430720.0,
+ "5975": 1036686400.0,
+ "5980": 1024121472.0,
+ "5985": 1035390208.0,
+ "5990": 1010553600.0,
+ "5995": 1047021568.0,
+ "6000": 1021248384.0,
+ "6005": 1040463872.0,
+ "6010": 1025361984.0,
+ "6015": 1050182848.0,
+ "6020": 1039517824.0,
+ "6025": 1030257792.0,
+ "6030": 1025934656.0,
+ "6035": 1021749120.0,
+ "6040": 1034119936.0,
+ "6045": 1028285312.0,
+ "6050": 1020115072.0,
+ "6055": 1040399744.0,
+ "6060": 1026350976.0,
+ "6065": 1022200448.0,
+ "6070": 1040671616.0,
+ "6075": 1046041088.0,
+ "6080": 1038585984.0,
+ "6085": 1041488576.0,
+ "6090": 1037208832.0,
+ "6095": 1036286592.0,
+ "6100": 1030458880.0,
+ "6105": 1019219072.0,
+ "6110": 1035360128.0,
+ "6115": 1019455744.0,
+ "6120": 1032192256.0,
+ "6125": 1020925056.0,
+ "6130": 1012016768.0,
+ "6135": 1038737152.0,
+ "6140": 1041740672.0,
+ "6145": 1041920192.0,
+ "6150": 1018961792.0,
+ "6155": 1024653184.0,
+ "6160": 1047975104.0,
+ "6165": 1050412800.0,
+ "6170": 1032507200.0,
+ "6175": 1045797248.0,
+ "6180": 1040070912.0,
+ "6185": 1029712896.0,
+ "6190": 1023297664.0,
+ "6195": 1050900864.0,
+ "6200": 1035037760.0,
+ "6205": 1036279232.0,
+ "6210": 1039776000.0,
+ "6215": 1033204480.0,
+ "6220": 1026165376.0,
+ "6225": 1036744448.0,
+ "6230": 1025147392.0,
+ "6235": 1019356416.0,
+ "6240": 1057107456.0,
+ "6245": 1018415872.0,
+ "6250": 1035340416.0,
+ "6255": 1025384192.0,
+ "6260": 1034866240.0,
+ "6265": 1027706560.0,
+ "6270": 1042120256.0,
+ "6275": 1037661888.0,
+ "6280": 1018273280.0,
+ "6285": 1032644864.0,
+ "6290": 1038601600.0,
+ "6295": 1031807040.0,
+ "6300": 1034636672.0,
+ "6305": 1011070656.0,
+ "6310": 1039461952.0,
+ "6315": 1030057664.0,
+ "6320": 1030537728.0,
+ "6325": 1038645504.0,
+ "6330": 1033911808.0,
+ "6335": 1032300416.0,
+ "6340": 1033548608.0,
+ "6345": 1031039744.0,
+ "6350": 1037455104.0,
+ "6355": 1028079616.0,
+ "6360": 1043316928.0,
+ "6365": 1025227392.0,
+ "6370": 1033942336.0,
+ "6375": 1036041856.0,
+ "6380": 1029111104.0,
+ "6385": 1025398912.0,
+ "6390": 1025520704.0,
+ "6395": 1048614592.0,
+ "6400": 1040738432.0,
+ "6405": 1024251840.0,
+ "6410": 1017492864.0,
+ "6415": 1042830336.0,
+ "6420": 1025204736.0,
+ "6425": 1027168192.0,
+ "6430": 1040571904.0,
+ "6435": 1022912384.0,
+ "6440": 1047997952.0,
+ "6445": 1036093696.0,
+ "6450": 1048534400.0,
+ "6455": 1037275072.0,
+ "6460": 1036753152.0,
+ "6465": 1033656320.0,
+ "6470": 1018138112.0,
+ "6475": 1034694400.0,
+ "6480": 1028997632.0,
+ "6485": 1033262464.0,
+ "6490": 1035642240.0,
+ "6495": 1024473472.0,
+ "6500": 1020575872.0,
+ "6505": 1059330176.0,
+ "6510": 1020475264.0,
+ "6515": 1018691584.0,
+ "6520": 1051473280.0,
+ "6525": 1035547968.0,
+ "6530": 1027900032.0,
+ "6535": 1022724672.0,
+ "6540": 1023277888.0,
+ "6545": 1033176064.0,
+ "6550": 1029491328.0,
+ "6555": 1029578432.0,
+ "6560": 1056442176.0,
+ "6565": 1054298752.0,
+ "6570": 1032322048.0,
+ "6575": 1041211648.0,
+ "6580": 1028136960.0,
+ "6585": 1036508032.0,
+ "6590": 1042459904.0,
+ "6595": 1038571712.0,
+ "6600": 1031391296.0,
+ "6605": 1045719168.0,
+ "6610": 1034716800.0,
+ "6615": 1015579968.0,
+ "6620": 1039118080.0,
+ "6625": 1054657920.0,
+ "6630": 1043096128.0,
+ "6635": 1032229952.0,
+ "6640": 1016741888.0,
+ "6645": 1016182592.0,
+ "6650": 1034695168.0,
+ "6655": 1031756928.0,
+ "6660": 1041405312.0,
+ "6665": 1024660992.0,
+ "6670": 1023822848.0,
+ "6675": 1038308864.0,
+ "6680": 1025628416.0,
+ "6685": 1045397056.0,
+ "6690": 1046394240.0,
+ "6695": 1027757440.0,
+ "6700": 1033476352.0,
+ "6705": 1038859648.0,
+ "6710": 1047490048.0,
+ "6715": 1043232640.0,
+ "6720": 1022999552.0,
+ "6725": 1018912512.0,
+ "6730": 1027528192.0,
+ "6735": 1016941696.0,
+ "6740": 1027241472.0,
+ "6745": 1030266368.0,
+ "6750": 1006376832.0,
+ "6755": 1034768960.0,
+ "6760": 1040737408.0,
+ "6765": 1023830528.0,
+ "6770": 1036443776.0,
+ "6775": 1019630528.0,
+ "6780": 1043726528.0,
+ "6785": 1037413120.0,
+ "6790": 1029405824.0,
+ "6795": 1026352832.0,
+ "6800": 1036631232.0,
+ "6805": 1024583232.0,
+ "6810": 1042342528.0,
+ "6815": 1035277184.0,
+ "6820": 1022597888.0,
+ "6825": 1034797056.0,
+ "6830": 1029865152.0,
+ "6835": 1041612224.0,
+ "6840": 1042286336.0,
+ "6845": 1018956800.0,
+ "6850": 1032174336.0,
+ "6855": 1034436608.0,
+ "6860": 1042058368.0,
+ "6865": 1021815744.0,
+ "6870": 1037019264.0,
+ "6875": 1030381568.0,
+ "6880": 1029364032.0,
+ "6885": 1030439808.0,
+ "6890": 1039892992.0,
+ "6895": 1027270272.0,
+ "6900": 1035176960.0,
+ "6905": 1043977536.0,
+ "6910": 1019766784.0,
+ "6915": 1017480192.0,
+ "6920": 1017186752.0,
+ "6925": 1030654720.0,
+ "6930": 1036676288.0,
+ "6935": 1042839040.0,
+ "6940": 1040315648.0,
+ "6945": 1044199936.0,
+ "6950": 1040515968.0,
+ "6955": 1036115456.0,
+ "6960": 1036438848.0,
+ "6965": 1019164416.0,
+ "6970": 1034731776.0,
+ "6975": 1019137280.0,
+ "6980": 1028439296.0,
+ "6985": 1023243776.0,
+ "6990": 1026997504.0,
+ "6995": 1027550208.0,
+ "7000": 1058822400.0,
+ "7005": 1013740480.0,
+ "7010": 1028962432.0,
+ "7015": 1037291264.0,
+ "7020": 1011883328.0,
+ "7025": 1017316864.0,
+ "7030": 1028304896.0,
+ "7035": 1035958144.0,
+ "7040": 1042968448.0,
+ "7045": 1028188928.0,
+ "7050": 1017982336.0,
+ "7055": 1035091584.0,
+ "7060": 1051804928.0,
+ "7065": 1007668160.0,
+ "7070": 1035822080.0,
+ "7075": 1031042560.0,
+ "7080": 1026146816.0,
+ "7085": 1044909824.0,
+ "7090": 1046264320.0,
+ "7095": 1043763840.0,
+ "7100": 1035091456.0,
+ "7105": 1049147392.0,
+ "7110": 1010966208.0,
+ "7115": 1033872128.0,
+ "7120": 1031270592.0,
+ "7125": 1037499392.0,
+ "7130": 1024885376.0,
+ "7135": 1031993856.0,
+ "7140": 1019094464.0,
+ "7145": 1033084288.0,
+ "7150": 1037557568.0,
+ "7155": 1015732224.0,
+ "7160": 1024727680.0,
+ "7165": 1030898560.0,
+ "7170": 1037370112.0,
+ "7175": 1028819776.0,
+ "7180": 1037635840.0,
+ "7185": 1016175872.0,
+ "7190": 1019811072.0,
+ "7195": 1040918528.0,
+ "7200": 1041377728.0,
+ "7205": 1026541504.0,
+ "7210": 1022641920.0,
+ "7215": 1041893248.0,
+ "7220": 1017746176.0,
+ "7225": 1027299840.0,
+ "7230": 1030203008.0,
+ "7235": 1035730112.0,
+ "7240": 1037858048.0,
+ "7245": 1023974656.0,
+ "7250": 1044710912.0,
+ "7255": 1031903488.0,
+ "7260": 1030130816.0,
+ "7265": 1036889728.0,
+ "7270": 1050099904.0,
+ "7275": 1029229440.0,
+ "7280": 1020234304.0,
+ "7285": 1029844928.0,
+ "7290": 1017222400.0,
+ "7295": 1029142912.0,
+ "7300": 1031537280.0,
+ "7305": 1027301248.0,
+ "7310": 1029092864.0,
+ "7315": 1022784896.0,
+ "7320": 1036460864.0,
+ "7325": 1036854400.0,
+ "7330": 1021710336.0,
+ "7335": 1030718464.0,
+ "7340": 1039385664.0,
+ "7345": 1040181120.0,
+ "7350": 1034976704.0,
+ "7355": 1033658816.0,
+ "7360": 1031257664.0,
+ "7365": 1048744384.0,
+ "7370": 1027300992.0,
+ "7375": 1041858048.0,
+ "7380": 1016728064.0,
+ "7385": 1017581440.0,
+ "7390": 1017236992.0,
+ "7395": 1046796288.0,
+ "7400": 1048443648.0,
+ "7405": 1013396864.0,
+ "7410": 1017388608.0,
+ "7415": 1017817856.0,
+ "7420": 1028046080.0,
+ "7425": 1012842688.0,
+ "7430": 1034045056.0,
+ "7435": 1032532672.0,
+ "7440": 1002695872.0,
+ "7445": 1034454016.0,
+ "7450": 1039307264.0,
+ "7455": 1019029248.0,
+ "7460": 1014743808.0,
+ "7465": 1027207744.0,
+ "7470": 1030425792.0,
+ "7475": 1033794304.0,
+ "7480": 1043320192.0,
+ "7485": 1038217152.0,
+ "7490": 1049004736.0,
+ "7495": 1028985536.0,
+ "7500": 1027430016.0,
+ "7505": 1028697920.0,
+ "7510": 1048889280.0,
+ "7515": 1035650304.0,
+ "7520": 1017202624.0,
+ "7525": 1036575872.0,
+ "7530": 1029265152.0,
+ "7535": 1027193216.0,
+ "7540": 1028340992.0,
+ "7545": 1025988992.0,
+ "7550": 1023029632.0,
+ "7555": 1033028480.0,
+ "7560": 1031407936.0,
+ "7565": 1022713856.0,
+ "7570": 1037594688.0,
+ "7575": 1022606720.0,
+ "7580": 1018126528.0,
+ "7585": 1033056640.0,
+ "7590": 1010996096.0,
+ "7595": 1018264576.0,
+ "7600": 1049907200.0,
+ "7605": 1037364352.0,
+ "7610": 1040417920.0,
+ "7615": 1035250688.0,
+ "7620": 1024233600.0,
+ "7625": 1020319680.0,
+ "7630": 1034941952.0,
+ "7635": 1043227520.0,
+ "7640": 1033494528.0,
+ "7645": 1034447616.0,
+ "7650": 1039807360.0,
+ "7655": 1031242752.0,
+ "7660": 1056630720.0,
+ "7665": 1031079040.0,
+ "7670": 1033687936.0,
+ "7675": 1030684288.0,
+ "7680": 1035401216.0,
+ "7685": 1018665280.0,
+ "7690": 1031924096.0,
+ "7695": 1025860672.0,
+ "7700": 1017717888.0,
+ "7705": 1036534272.0,
+ "7710": 1029895424.0,
+ "7715": 1053233408.0,
+ "7720": 1019517504.0,
+ "7725": 1042195328.0,
+ "7730": 1035623424.0,
+ "7735": 1020729600.0,
+ "7740": 1045578560.0,
+ "7745": 1026935936.0,
+ "7750": 1048552448.0,
+ "7755": 1022542912.0,
+ "7760": 1049534464.0,
+ "7765": 1029373824.0,
+ "7770": 1018378304.0,
+ "7775": 1021367680.0,
+ "7780": 1039772800.0,
+ "7785": 1039916288.0,
+ "7790": 1030520000.0,
+ "7795": 1039356928.0,
+ "7800": 1028190720.0,
+ "7805": 1027638400.0,
+ "7810": 1020972544.0,
+ "7815": 1035881216.0,
+ "7820": 1017670784.0,
+ "7825": 1018070080.0,
+ "7830": 1035107008.0,
+ "7835": 1044510848.0,
+ "7840": 1027838976.0,
+ "7845": 1032104128.0,
+ "7850": 1034612608.0,
+ "7855": 1025467328.0,
+ "7860": 1059053952.0,
+ "7865": 1016628992.0,
+ "7870": 1033731712.0,
+ "7875": 1044189184.0,
+ "7880": 1029087488.0,
+ "7885": 1040310656.0,
+ "7890": 1029558592.0,
+ "7895": 1032948992.0,
+ "7900": 1021410816.0,
+ "7905": 1020958080.0,
+ "7910": 1008996608.0,
+ "7915": 1023122688.0,
+ "7920": 1023073792.0,
+ "7925": 1030096768.0,
+ "7930": 1020715648.0,
+ "7935": 1019446592.0,
+ "7940": 1017811072.0,
+ "7945": 1014450752.0,
+ "7950": 1026306624.0,
+ "7955": 1034521536.0,
+ "7960": 1056028352.0,
+ "7965": 1031051648.0,
+ "7970": 1030420864.0,
+ "7975": 1022193024.0,
+ "7980": 1034477824.0,
+ "7985": 1047306432.0,
+ "7990": 1032068736.0,
+ "7995": 1044268160.0,
+ "8000": 1028878912.0,
+ "8005": 1028047936.0,
+ "8010": 1050667968.0,
+ "8015": 1019761920.0,
+ "8020": 1043299712.0,
+ "8025": 1039021760.0,
+ "8030": 1030871040.0,
+ "8035": 1045307264.0,
+ "8040": 1026313984.0,
+ "8045": 1024972224.0,
+ "8050": 1018407360.0,
+ "8055": 1033739392.0,
+ "8060": 1012988800.0,
+ "8065": 1022020224.0,
+ "8070": 1034778240.0,
+ "8075": 1042762368.0,
+ "8080": 1027761920.0,
+ "8085": 1037208064.0,
+ "8090": 1007010816.0,
+ "8095": 1030376640.0,
+ "8100": 1030728832.0,
+ "8105": 1027796288.0,
+ "8110": 1031560064.0,
+ "8115": 1037688832.0,
+ "8120": 1037695104.0,
+ "8125": 1031100544.0,
+ "8130": 1028629760.0,
+ "8135": 1029682688.0,
+ "8140": 1049907072.0,
+ "8145": 1043465600.0,
+ "8150": 1040089280.0,
+ "8155": 1046782464.0,
+ "8160": 1010201728.0,
+ "8165": 1031659904.0,
+ "8170": 1024486592.0,
+ "8175": 1035022336.0,
+ "8180": 1024464384.0,
+ "8185": 1021963840.0,
+ "8190": 1037127552.0,
+ "8195": 1022371072.0,
+ "8200": 1035639168.0,
+ "8205": 1026485504.0,
+ "8210": 1023891328.0,
+ "8215": 1014279936.0,
+ "8220": 1026757440.0,
+ "8225": 1028542464.0,
+ "8230": 1027165952.0,
+ "8235": 1037916288.0,
+ "8240": 1025911168.0,
+ "8245": 1024679936.0,
+ "8250": 1041637056.0,
+ "8255": 1031911232.0,
+ "8260": 1032427264.0,
+ "8265": 1023167232.0,
+ "8270": 1040175360.0,
+ "8275": 1038053248.0,
+ "8280": 1041852032.0,
+ "8285": 1038806272.0,
+ "8290": 1024077440.0,
+ "8295": 1028406464.0,
+ "8300": 1039343680.0,
+ "8305": 1012107392.0,
+ "8310": 1021884864.0,
+ "8315": 1027309824.0,
+ "8320": 1021640064.0,
+ "8325": 1048574592.0,
+ "8330": 1041043200.0,
+ "8335": 1037967808.0,
+ "8340": 1033022080.0,
+ "8345": 1043867520.0,
+ "8350": 1037716224.0,
+ "8355": 1029688704.0,
+ "8360": 1040670336.0,
+ "8365": 1027452160.0,
+ "8370": 1037745792.0,
+ "8375": 1041989120.0,
+ "8380": 1037630912.0,
+ "8385": 1023438592.0,
+ "8390": 1026071104.0,
+ "8395": 1028915392.0,
+ "8400": 1046532288.0,
+ "8405": 1040182720.0,
+ "8410": 1034255360.0,
+ "8415": 1040261504.0,
+ "8420": 1054733056.0,
+ "8425": 1031518080.0,
+ "8430": 1030298752.0,
+ "8435": 1045708992.0,
+ "8440": 1026313856.0,
+ "8445": 1029030528.0,
+ "8450": 1034205504.0,
+ "8455": 1031796672.0,
+ "8460": 1016830848.0,
+ "8465": 1035165568.0,
+ "8470": 1035187072.0,
+ "8475": 1024716032.0,
+ "8480": 1035902784.0,
+ "8485": 1028950272.0,
+ "8490": 1023081856.0,
+ "8495": 1037395712.0,
+ "8500": 1025963136.0,
+ "8505": 1042727104.0,
+ "8510": 1028170240.0,
+ "8515": 1038103808.0,
+ "8520": 1023109248.0,
+ "8525": 1037989632.0,
+ "8530": 1027574656.0,
+ "8535": 1041659648.0,
+ "8540": 1033882880.0,
+ "8545": 1015118848.0,
+ "8550": 1040191232.0,
+ "8555": 1016343168.0,
+ "8560": 1019333248.0,
+ "8565": 1021412096.0,
+ "8570": 1032034944.0,
+ "8575": 1031883648.0,
+ "8580": 1016014528.0,
+ "8585": 1030020480.0,
+ "8590": 1031639552.0,
+ "8595": 1017778944.0,
+ "8600": 1002396288.0,
+ "8605": 1030241280.0,
+ "8610": 1017534464.0,
+ "8615": 1023991808.0,
+ "8620": 1047207680.0,
+ "8625": 1034234688.0,
+ "8630": 1030923328.0,
+ "8635": 1051995520.0,
+ "8640": 1041136256.0,
+ "8645": 1024873344.0,
+ "8650": 1025597952.0,
+ "8655": 1036907648.0,
+ "8660": 1031172672.0,
+ "8665": 1032907456.0,
+ "8670": 1037403200.0,
+ "8675": 1029158848.0,
+ "8680": 1031266752.0,
+ "8685": 1041200576.0,
+ "8690": 1035037696.0,
+ "8695": 1008511232.0,
+ "8700": 1027461120.0,
+ "8705": 1051507776.0,
+ "8710": 1041679936.0,
+ "8715": 1034154816.0,
+ "8720": 1017599040.0,
+ "8725": 1025190336.0,
+ "8730": 1036613376.0,
+ "8735": 1014831040.0,
+ "8740": 1036084480.0,
+ "8745": 1021254784.0,
+ "8750": 1027869696.0,
+ "8755": 1020744960.0,
+ "8760": 1036901248.0,
+ "8765": 1058675328.0,
+ "8770": 1020465280.0,
+ "8775": 1031775872.0,
+ "8780": 1030894400.0,
+ "8785": 1032120576.0,
+ "8790": 1041036608.0,
+ "8795": 1019526528.0,
+ "8800": 1038247680.0,
+ "8805": 1035108992.0,
+ "8810": 1043259392.0,
+ "8815": 1026493632.0,
+ "8820": 1027669376.0,
+ "8825": 1043466176.0,
+ "8830": 1027484288.0,
+ "8835": 1038814976.0,
+ "8840": 1034492864.0,
+ "8845": 1033911488.0,
+ "8850": 1030493824.0,
+ "8855": 1042527616.0,
+ "8860": 1013005184.0,
+ "8865": 1038370176.0,
+ "8870": 1025188992.0,
+ "8875": 1012983040.0,
+ "8880": 1028378560.0,
+ "8885": 1046463232.0,
+ "8890": 1038605632.0,
+ "8895": 1037912128.0,
+ "8900": 1027296640.0,
+ "8905": 1032794304.0,
+ "8910": 1029798272.0,
+ "8915": 1030006208.0,
+ "8920": 1030342080.0,
+ "8925": 1028572288.0,
+ "8930": 1031639936.0,
+ "8935": 1022953216.0,
+ "8940": 1019850048.0,
+ "8945": 1031911488.0,
+ "8950": 1039954496.0,
+ "8955": 1041904384.0,
+ "8960": 1026881344.0,
+ "8965": 1022086208.0,
+ "8970": 1029561024.0,
+ "8975": 1038936896.0,
+ "8980": 1033862144.0,
+ "8985": 1030651456.0,
+ "8990": 1025016512.0,
+ "8995": 1013965952.0,
+ "9000": 1035287808.0,
+ "9005": 1028651008.0,
+ "9010": 1011916288.0,
+ "9015": 1038914432.0,
+ "9020": 1030156736.0,
+ "9025": 1024687168.0,
+ "9030": 1025863808.0,
+ "9035": 1054311296.0,
+ "9040": 1027297728.0,
+ "9045": 1036585344.0,
+ "9050": 1020932032.0,
+ "9055": 1043215360.0,
+ "9060": 1023161408.0,
+ "9065": 1023389760.0,
+ "9070": 1039365888.0,
+ "9075": 1026731584.0,
+ "9080": 1018875008.0,
+ "9085": 1015441856.0,
+ "9090": 1043766848.0,
+ "9095": 1014022720.0,
+ "9100": 1031977856.0,
+ "9105": 1026516160.0,
+ "9110": 1029232384.0,
+ "9115": 1024868480.0,
+ "9120": 999988416.0,
+ "9125": 1032845312.0,
+ "9130": 1038535744.0,
+ "9135": 1031039872.0,
+ "9140": 1025504704.0,
+ "9145": 1030408576.0,
+ "9150": 1029418496.0,
+ "9155": 1038270656.0,
+ "9160": 1046046272.0,
+ "9165": 1017951680.0,
+ "9170": 1040958272.0,
+ "9175": 1031289984.0,
+ "9180": 1037832448.0,
+ "9185": 1040685824.0,
+ "9190": 1028988224.0,
+ "9195": 1034314880.0,
+ "9200": 1035554112.0,
+ "9205": 1029849408.0,
+ "9210": 1026537728.0,
+ "9215": 1030523648.0,
+ "9220": 1025734272.0,
+ "9225": 1048003072.0,
+ "9230": 1041604160.0,
+ "9235": 1027777536.0,
+ "9240": 1025247872.0,
+ "9245": 1036213824.0,
+ "9250": 1041194624.0,
+ "9255": 1020066816.0,
+ "9260": 1035340544.0,
+ "9265": 1023105472.0,
+ "9270": 1038334720.0,
+ "9275": 1036055936.0,
+ "9280": 1026543744.0,
+ "9285": 1014288256.0,
+ "9290": 1018869056.0,
+ "9295": 1026917120.0,
+ "9300": 1037088384.0,
+ "9305": 1045438144.0,
+ "9310": 1033245888.0,
+ "9315": 1039046272.0,
+ "9320": 1048497728.0,
+ "9325": 1023062016.0,
+ "9330": 1031726720.0,
+ "9335": 1035676096.0,
+ "9340": 1013721344.0,
+ "9345": 1022574208.0,
+ "9350": 1026588160.0,
+ "9355": 1034810048.0,
+ "9360": 1029842176.0,
+ "9365": 1019865280.0,
+ "9370": 1006905344.0,
+ "9375": 1036234816.0,
+ "9380": 1049014400.0,
+ "9385": 1015906624.0,
+ "9390": 1029210880.0,
+ "9395": 1008934976.0,
+ "9400": 1026896512.0,
+ "9405": 1027655808.0,
+ "9410": 1040914752.0,
+ "9415": 1035131264.0,
+ "9420": 1030795456.0,
+ "9425": 1027583744.0,
+ "9430": 1032730304.0,
+ "9435": 1031798848.0,
+ "9440": 1051732544.0,
+ "9445": 1019628672.0,
+ "9450": 1044507584.0,
+ "9455": 1035775808.0,
+ "9460": 1013830272.0,
+ "9465": 1023405504.0,
+ "9470": 1023578688.0,
+ "9475": 1039166144.0,
+ "9480": 1029598912.0,
+ "9485": 1032077440.0,
+ "9490": 1020996928.0,
+ "9495": 1021377728.0,
+ "9500": 1035596288.0,
+ "9505": 1034480384.0,
+ "9510": 1014289600.0,
+ "9515": 1031312000.0,
+ "9520": 1026566336.0,
+ "9525": 1035855680.0,
+ "9530": 1031627008.0,
+ "9535": 1025929216.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 9535,
+ "step_interval": 5,
+ "values": {
+ "1": 58533826560.0,
+ "5": 58533810176.0,
+ "10": 58533834752.0,
+ "15": 58533756928.0,
+ "20": 58533720064.0,
+ "25": 58533736448.0,
+ "30": 58533740544.0,
+ "35": 58533761024.0,
+ "40": 58533736448.0,
+ "45": 58533711872.0,
+ "50": 58533785600.0,
+ "55": 58533724160.0,
+ "60": 58533740544.0,
+ "65": 58533801984.0,
+ "70": 58533756928.0,
+ "75": 58533740544.0,
+ "80": 58533785600.0,
+ "85": 58533691392.0,
+ "90": 58533724160.0,
+ "95": 58533765120.0,
+ "100": 58533732352.0,
+ "105": 58533761024.0,
+ "110": 58533724160.0,
+ "115": 58533748736.0,
+ "120": 58533695488.0,
+ "125": 58533744640.0,
+ "130": 58533736448.0,
+ "135": 58533806080.0,
+ "140": 58533773312.0,
+ "145": 58533699584.0,
+ "150": 58533761024.0,
+ "155": 58533851136.0,
+ "160": 58533847040.0,
+ "165": 58533838848.0,
+ "170": 58533711872.0,
+ "175": 58533752832.0,
+ "180": 58533679104.0,
+ "185": 58533715968.0,
+ "190": 58533625856.0,
+ "195": 58533609472.0,
+ "200": 58533539840.0,
+ "205": 58533523456.0,
+ "210": 58533486592.0,
+ "215": 58533470208.0,
+ "220": 58533478400.0,
+ "225": 58533519360.0,
+ "230": 58533494784.0,
+ "235": 58533498880.0,
+ "240": 58533486592.0,
+ "245": 58533466112.0,
+ "250": 58533498880.0,
+ "255": 58533543936.0,
+ "260": 58533552128.0,
+ "265": 58533478400.0,
+ "270": 58533507072.0,
+ "275": 58533519360.0,
+ "280": 58533629952.0,
+ "285": 58533593088.0,
+ "290": 58533552128.0,
+ "295": 58533543936.0,
+ "300": 58533601280.0,
+ "305": 58533584896.0,
+ "310": 58533486592.0,
+ "315": 58533531648.0,
+ "320": 58533560320.0,
+ "325": 58533601280.0,
+ "330": 58533494784.0,
+ "335": 58533531648.0,
+ "340": 58533580800.0,
+ "345": 58533617664.0,
+ "350": 58533617664.0,
+ "355": 58533584896.0,
+ "360": 58533670912.0,
+ "365": 58533535744.0,
+ "370": 58533576704.0,
+ "375": 58533695488.0,
+ "380": 58533650432.0,
+ "385": 58533621760.0,
+ "390": 58533486592.0,
+ "395": 58533474304.0,
+ "400": 58533675008.0,
+ "405": 58533617664.0,
+ "410": 58533785600.0,
+ "415": 58533666816.0,
+ "420": 58533732352.0,
+ "425": 58533703680.0,
+ "430": 58533769216.0,
+ "435": 58533740544.0,
+ "440": 58533605376.0,
+ "445": 58533838848.0,
+ "450": 58533519360.0,
+ "455": 58533437440.0,
+ "460": 58533781504.0,
+ "465": 58533924864.0,
+ "470": 58533851136.0,
+ "475": 58533756928.0,
+ "480": 58533875712.0,
+ "485": 58533888000.0,
+ "490": 58533855232.0,
+ "495": 58533920768.0,
+ "500": 58533847040.0,
+ "505": 58533871616.0,
+ "510": 58533847040.0,
+ "515": 58533556224.0,
+ "520": 58533519360.0,
+ "525": 58534002688.0,
+ "530": 58533867520.0,
+ "535": 58534006784.0,
+ "540": 58533982208.0,
+ "545": 58533871616.0,
+ "550": 58534006784.0,
+ "555": 58533888000.0,
+ "560": 58534084608.0,
+ "565": 58534010880.0,
+ "570": 58533855232.0,
+ "575": 58533675008.0,
+ "580": 58533687296.0,
+ "585": 58533916672.0,
+ "590": 58533957632.0,
+ "595": 58533810176.0,
+ "600": 58533933056.0,
+ "605": 58533904384.0,
+ "610": 58533998592.0,
+ "615": 58533822464.0,
+ "620": 58533953536.0,
+ "625": 58533928960.0,
+ "630": 58533945344.0,
+ "635": 58533527552.0,
+ "640": 58533765120.0,
+ "645": 58533888000.0,
+ "650": 58533806080.0,
+ "655": 58533748736.0,
+ "660": 58533994496.0,
+ "665": 58533924864.0,
+ "670": 58533847040.0,
+ "675": 58533847040.0,
+ "680": 58533838848.0,
+ "685": 58533806080.0,
+ "690": 58533769216.0,
+ "695": 58533498880.0,
+ "700": 58533543936.0,
+ "705": 58533752832.0,
+ "710": 58533728256.0,
+ "715": 58533789696.0,
+ "720": 58533724160.0,
+ "725": 58533756928.0,
+ "730": 58533601280.0,
+ "735": 58533728256.0,
+ "740": 58533699584.0,
+ "745": 58533752832.0,
+ "750": 58533822464.0,
+ "755": 58533691392.0,
+ "760": 58533470208.0,
+ "765": 58533715968.0,
+ "770": 58533773312.0,
+ "775": 58533769216.0,
+ "780": 58533695488.0,
+ "785": 58533814272.0,
+ "790": 58533699584.0,
+ "795": 58533822464.0,
+ "800": 58533814272.0,
+ "805": 58533793792.0,
+ "810": 58533650432.0,
+ "815": 58533662720.0,
+ "820": 58533462016.0,
+ "825": 58533781504.0,
+ "830": 58533662720.0,
+ "835": 58533806080.0,
+ "840": 58533789696.0,
+ "845": 58533892096.0,
+ "850": 58533781504.0,
+ "855": 58533740544.0,
+ "860": 58533703680.0,
+ "865": 58533769216.0,
+ "870": 58533756928.0,
+ "875": 58533658624.0,
+ "880": 58533629952.0,
+ "885": 58533699584.0,
+ "890": 58533601280.0,
+ "895": 58533675008.0,
+ "900": 58533752832.0,
+ "905": 58533765120.0,
+ "910": 58533642240.0,
+ "915": 58533748736.0,
+ "920": 58533748736.0,
+ "925": 58533634048.0,
+ "930": 58533670912.0,
+ "935": 58533703680.0,
+ "940": 58533576704.0,
+ "945": 58533605376.0,
+ "950": 58533703680.0,
+ "955": 58533695488.0,
+ "960": 58533756928.0,
+ "965": 58533584896.0,
+ "970": 58533769216.0,
+ "975": 58533814272.0,
+ "980": 58533736448.0,
+ "985": 58533654528.0,
+ "990": 58533769216.0,
+ "995": 58533740544.0,
+ "1000": 58533781504.0,
+ "1005": 58533511168.0,
+ "1010": 58533744640.0,
+ "1015": 58533740544.0,
+ "1020": 58533822464.0,
+ "1025": 58533847040.0,
+ "1030": 58533654528.0,
+ "1035": 58533625856.0,
+ "1040": 58533711872.0,
+ "1045": 58533765120.0,
+ "1050": 58533744640.0,
+ "1055": 58533875712.0,
+ "1060": 58533638144.0,
+ "1065": 58533691392.0,
+ "1070": 58533634048.0,
+ "1075": 58533814272.0,
+ "1080": 58533838848.0,
+ "1085": 58533683200.0,
+ "1090": 58533777408.0,
+ "1095": 58533814272.0,
+ "1100": 58533650432.0,
+ "1105": 58533871616.0,
+ "1110": 58533724160.0,
+ "1115": 58533683200.0,
+ "1120": 58533568512.0,
+ "1125": 58533560320.0,
+ "1130": 58533715968.0,
+ "1135": 58533724160.0,
+ "1140": 58533777408.0,
+ "1145": 58533797888.0,
+ "1150": 58533707776.0,
+ "1155": 58533691392.0,
+ "1160": 58533752832.0,
+ "1165": 58533715968.0,
+ "1170": 58533724160.0,
+ "1175": 58533715968.0,
+ "1180": 58533650432.0,
+ "1185": 58533629952.0,
+ "1190": 58533822464.0,
+ "1195": 58533761024.0,
+ "1200": 58533699584.0,
+ "1205": 58533756928.0,
+ "1210": 58533818368.0,
+ "1215": 58533732352.0,
+ "1220": 58533752832.0,
+ "1225": 58533797888.0,
+ "1230": 58533736448.0,
+ "1235": 58533679104.0,
+ "1240": 58533777408.0,
+ "1245": 58533769216.0,
+ "1250": 58533601280.0,
+ "1255": 58533822464.0,
+ "1260": 58533830656.0,
+ "1265": 58533638144.0,
+ "1270": 58533789696.0,
+ "1275": 58533638144.0,
+ "1280": 58533675008.0,
+ "1285": 58533740544.0,
+ "1290": 58533769216.0,
+ "1295": 58533842944.0,
+ "1300": 58533703680.0,
+ "1305": 58533769216.0,
+ "1310": 58533613568.0,
+ "1315": 58533605376.0,
+ "1320": 58533732352.0,
+ "1325": 58533896192.0,
+ "1330": 58533765120.0,
+ "1335": 58533761024.0,
+ "1340": 58533847040.0,
+ "1345": 58533867520.0,
+ "1350": 58533847040.0,
+ "1355": 58533724160.0,
+ "1360": 58533994496.0,
+ "1365": 58533842944.0,
+ "1370": 58533511168.0,
+ "1375": 58533740544.0,
+ "1380": 58533683200.0,
+ "1385": 58533732352.0,
+ "1390": 58533642240.0,
+ "1395": 58533683200.0,
+ "1400": 58533814272.0,
+ "1405": 58533732352.0,
+ "1410": 58533703680.0,
+ "1415": 58533662720.0,
+ "1420": 58533675008.0,
+ "1425": 58533707776.0,
+ "1430": 58533552128.0,
+ "1435": 58533699584.0,
+ "1440": 58533715968.0,
+ "1445": 58533744640.0,
+ "1450": 58533675008.0,
+ "1455": 58533613568.0,
+ "1460": 58533580800.0,
+ "1465": 58533711872.0,
+ "1470": 58533715968.0,
+ "1475": 58533707776.0,
+ "1480": 58533699584.0,
+ "1485": 58533765120.0,
+ "1490": 58533601280.0,
+ "1495": 58533699584.0,
+ "1500": 58533670912.0,
+ "1505": 58533703680.0,
+ "1510": 58533732352.0,
+ "1515": 58533883904.0,
+ "1520": 58533789696.0,
+ "1525": 58533748736.0,
+ "1530": 58533896192.0,
+ "1535": 58533642240.0,
+ "1540": 58533695488.0,
+ "1545": 58533810176.0,
+ "1550": 58533662720.0,
+ "1555": 58533580800.0,
+ "1560": 58533892096.0,
+ "1565": 58533732352.0,
+ "1570": 58533740544.0,
+ "1575": 58533707776.0,
+ "1580": 58533761024.0,
+ "1585": 58533781504.0,
+ "1590": 58533789696.0,
+ "1595": 58533769216.0,
+ "1600": 58533801984.0,
+ "1605": 58533691392.0,
+ "1610": 58533732352.0,
+ "1615": 58533646336.0,
+ "1620": 58533691392.0,
+ "1625": 58533654528.0,
+ "1630": 58533736448.0,
+ "1635": 58533896192.0,
+ "1640": 58533736448.0,
+ "1645": 58533761024.0,
+ "1650": 58533703680.0,
+ "1655": 58533683200.0,
+ "1660": 58533662720.0,
+ "1665": 58533687296.0,
+ "1670": 58533683200.0,
+ "1675": 58533486592.0,
+ "1680": 58533732352.0,
+ "1685": 58533851136.0,
+ "1690": 58533724160.0,
+ "1695": 58533539840.0,
+ "1700": 58533740544.0,
+ "1705": 58533695488.0,
+ "1710": 58533683200.0,
+ "1715": 58533675008.0,
+ "1720": 58533691392.0,
+ "1725": 58533707776.0,
+ "1730": 58533756928.0,
+ "1735": 58533556224.0,
+ "1740": 58533707776.0,
+ "1745": 58533752832.0,
+ "1750": 58533740544.0,
+ "1755": 58533826560.0,
+ "1760": 58533715968.0,
+ "1765": 58533904384.0,
+ "1770": 58533949440.0,
+ "1775": 58533842944.0,
+ "1780": 58533703680.0,
+ "1785": 58533789696.0,
+ "1790": 58533998592.0,
+ "1795": 58533830656.0,
+ "1800": 58533621760.0,
+ "1805": 58533691392.0,
+ "1810": 58533756928.0,
+ "1815": 58533744640.0,
+ "1820": 58533785600.0,
+ "1825": 58533814272.0,
+ "1830": 58533769216.0,
+ "1835": 58533908480.0,
+ "1840": 58533675008.0,
+ "1845": 58533801984.0,
+ "1850": 58533941248.0,
+ "1855": 58533830656.0,
+ "1860": 58533699584.0,
+ "1865": 58533683200.0,
+ "1870": 58533720064.0,
+ "1875": 58533793792.0,
+ "1880": 58533892096.0,
+ "1885": 58533834752.0,
+ "1890": 58533740544.0,
+ "1895": 58533691392.0,
+ "1900": 58533724160.0,
+ "1905": 58533748736.0,
+ "1910": 58533744640.0,
+ "1915": 58533736448.0,
+ "1920": 58533679104.0,
+ "1925": 58533634048.0,
+ "1930": 58533695488.0,
+ "1935": 58533670912.0,
+ "1940": 58533761024.0,
+ "1945": 58533941248.0,
+ "1950": 58533634048.0,
+ "1955": 58533785600.0,
+ "1960": 58533765120.0,
+ "1965": 58533748736.0,
+ "1970": 58533675008.0,
+ "1975": 58533814272.0,
+ "1980": 58533658624.0,
+ "1985": 58533675008.0,
+ "1990": 58533801984.0,
+ "1995": 58533715968.0,
+ "2000": 58533646336.0,
+ "2005": 58533740544.0,
+ "2010": 58533650432.0,
+ "2015": 58533621760.0,
+ "2020": 58533785600.0,
+ "2025": 58533646336.0,
+ "2030": 58533687296.0,
+ "2035": 58533740544.0,
+ "2040": 58533658624.0,
+ "2045": 58533593088.0,
+ "2050": 58533715968.0,
+ "2055": 58533769216.0,
+ "2060": 58533715968.0,
+ "2065": 58533707776.0,
+ "2070": 58533793792.0,
+ "2075": 58533728256.0,
+ "2080": 58533765120.0,
+ "2085": 58533732352.0,
+ "2090": 58533765120.0,
+ "2095": 58533777408.0,
+ "2100": 58533679104.0,
+ "2105": 58533490688.0,
+ "2110": 58533740544.0,
+ "2115": 58533793792.0,
+ "2120": 58533756928.0,
+ "2125": 58533793792.0,
+ "2130": 58533822464.0,
+ "2135": 58533859328.0,
+ "2140": 58533781504.0,
+ "2145": 58533888000.0,
+ "2150": 58533859328.0,
+ "2155": 58533752832.0,
+ "2160": 58533773312.0,
+ "2165": 58533560320.0,
+ "2170": 58533740544.0,
+ "2175": 58533793792.0,
+ "2180": 58533703680.0,
+ "2185": 58533806080.0,
+ "2190": 58533842944.0,
+ "2195": 58533724160.0,
+ "2200": 58533703680.0,
+ "2205": 58533675008.0,
+ "2210": 58533769216.0,
+ "2215": 58533871616.0,
+ "2220": 58533801984.0,
+ "2225": 58533675008.0,
+ "2230": 58533556224.0,
+ "2235": 58533847040.0,
+ "2240": 58533814272.0,
+ "2245": 58533679104.0,
+ "2250": 58533748736.0,
+ "2255": 58533584896.0,
+ "2260": 58533724160.0,
+ "2265": 58533580800.0,
+ "2270": 58533806080.0,
+ "2275": 58533777408.0,
+ "2280": 58533601280.0,
+ "2285": 58533679104.0,
+ "2290": 58533666816.0,
+ "2295": 58533871616.0,
+ "2300": 58533847040.0,
+ "2305": 58533920768.0,
+ "2310": 58533879808.0,
+ "2315": 58533662720.0,
+ "2320": 58533797888.0,
+ "2325": 58533818368.0,
+ "2330": 58533765120.0,
+ "2335": 58533691392.0,
+ "2340": 58533724160.0,
+ "2345": 58533703680.0,
+ "2350": 58533613568.0,
+ "2355": 58533740544.0,
+ "2360": 58533855232.0,
+ "2365": 58533642240.0,
+ "2370": 58533695488.0,
+ "2375": 58533691392.0,
+ "2380": 58533715968.0,
+ "2385": 58533793792.0,
+ "2390": 58533818368.0,
+ "2395": 58533666816.0,
+ "2400": 58533683200.0,
+ "2405": 58533695488.0,
+ "2410": 58533748736.0,
+ "2415": 58533715968.0,
+ "2420": 58533851136.0,
+ "2425": 58533695488.0,
+ "2430": 58533789696.0,
+ "2435": 58533670912.0,
+ "2440": 58533683200.0,
+ "2445": 58533740544.0,
+ "2450": 58533785600.0,
+ "2455": 58533752832.0,
+ "2460": 58533769216.0,
+ "2465": 58533859328.0,
+ "2470": 58533621760.0,
+ "2475": 58533777408.0,
+ "2480": 58533728256.0,
+ "2485": 58533761024.0,
+ "2490": 58533695488.0,
+ "2495": 58533732352.0,
+ "2500": 58533744640.0,
+ "2505": 58533900288.0,
+ "2510": 58533687296.0,
+ "2515": 58533797888.0,
+ "2520": 58533756928.0,
+ "2525": 58533785600.0,
+ "2530": 58533605376.0,
+ "2535": 58533662720.0,
+ "2540": 58533847040.0,
+ "2545": 58533785600.0,
+ "2550": 58533855232.0,
+ "2555": 58533781504.0,
+ "2560": 58533765120.0,
+ "2565": 58533777408.0,
+ "2570": 58533707776.0,
+ "2575": 58533773312.0,
+ "2580": 58533830656.0,
+ "2585": 58533744640.0,
+ "2590": 58533588992.0,
+ "2595": 58533736448.0,
+ "2600": 58533924864.0,
+ "2605": 58533826560.0,
+ "2610": 58533781504.0,
+ "2615": 58533597184.0,
+ "2620": 58533707776.0,
+ "2625": 58533666816.0,
+ "2630": 58533756928.0,
+ "2635": 58533781504.0,
+ "2640": 58533756928.0,
+ "2645": 58533810176.0,
+ "2650": 58533675008.0,
+ "2655": 58533642240.0,
+ "2660": 58533756928.0,
+ "2665": 58533707776.0,
+ "2670": 58533715968.0,
+ "2675": 58533797888.0,
+ "2680": 58533691392.0,
+ "2685": 58533752832.0,
+ "2690": 58533789696.0,
+ "2695": 58533777408.0,
+ "2700": 58533769216.0,
+ "2705": 58533851136.0,
+ "2710": 58533769216.0,
+ "2715": 58533568512.0,
+ "2720": 58533642240.0,
+ "2725": 58533666816.0,
+ "2730": 58533695488.0,
+ "2735": 58533724160.0,
+ "2740": 58533720064.0,
+ "2745": 58533621760.0,
+ "2750": 58533851136.0,
+ "2755": 58533769216.0,
+ "2760": 58533781504.0,
+ "2765": 58533736448.0,
+ "2770": 58533670912.0,
+ "2775": 58533507072.0,
+ "2780": 58533715968.0,
+ "2785": 58533781504.0,
+ "2790": 58533679104.0,
+ "2795": 58533982208.0,
+ "2800": 58533830656.0,
+ "2805": 58533687296.0,
+ "2810": 58533855232.0,
+ "2815": 58533732352.0,
+ "2820": 58533830656.0,
+ "2825": 58533621760.0,
+ "2830": 58533740544.0,
+ "2835": 58533593088.0,
+ "2840": 58533662720.0,
+ "2845": 58533842944.0,
+ "2850": 58533838848.0,
+ "2855": 58533830656.0,
+ "2860": 58533756928.0,
+ "2865": 58533691392.0,
+ "2870": 58533740544.0,
+ "2875": 58533842944.0,
+ "2880": 58533707776.0,
+ "2885": 58533769216.0,
+ "2890": 58533756928.0,
+ "2895": 58533658624.0,
+ "2900": 58533617664.0,
+ "2905": 58533744640.0,
+ "2910": 58533728256.0,
+ "2915": 58533756928.0,
+ "2920": 58533720064.0,
+ "2925": 58533797888.0,
+ "2930": 58533761024.0,
+ "2935": 58533756928.0,
+ "2940": 58533756928.0,
+ "2945": 58533810176.0,
+ "2950": 58533695488.0,
+ "2955": 58533720064.0,
+ "2960": 58533560320.0,
+ "2965": 58533728256.0,
+ "2970": 58533851136.0,
+ "2975": 58533699584.0,
+ "2980": 58533736448.0,
+ "2985": 58533797888.0,
+ "2990": 58533838848.0,
+ "2995": 58533588992.0,
+ "3000": 58533675008.0,
+ "3005": 58533715968.0,
+ "3010": 58533793792.0,
+ "3015": 58533756928.0,
+ "3020": 58533634048.0,
+ "3025": 58533879808.0,
+ "3030": 58533797888.0,
+ "3035": 58533801984.0,
+ "3040": 58533785600.0,
+ "3045": 58533695488.0,
+ "3050": 58533621760.0,
+ "3055": 58533675008.0,
+ "3060": 58533675008.0,
+ "3065": 58533769216.0,
+ "3070": 58533740544.0,
+ "3075": 58533928960.0,
+ "3080": 58533584896.0,
+ "3085": 58533744640.0,
+ "3090": 58533740544.0,
+ "3095": 58533679104.0,
+ "3100": 58533638144.0,
+ "3105": 58533851136.0,
+ "3110": 58533801984.0,
+ "3115": 58533818368.0,
+ "3120": 58533650432.0,
+ "3125": 58533838848.0,
+ "3130": 58533761024.0,
+ "3135": 58533732352.0,
+ "3140": 58533711872.0,
+ "3145": 58533842944.0,
+ "3150": 58533650432.0,
+ "3155": 58533789696.0,
+ "3160": 58533851136.0,
+ "3165": 58533826560.0,
+ "3170": 58533650432.0,
+ "3175": 58533863424.0,
+ "3180": 58533793792.0,
+ "3185": 58533666816.0,
+ "3190": 58533724160.0,
+ "3195": 58533707776.0,
+ "3200": 58533699584.0,
+ "3205": 58533654528.0,
+ "3210": 58533683200.0,
+ "3215": 58533662720.0,
+ "3220": 58533875712.0,
+ "3225": 58533724160.0,
+ "3230": 58533879808.0,
+ "3235": 58533728256.0,
+ "3240": 58533642240.0,
+ "3245": 58533707776.0,
+ "3250": 58533625856.0,
+ "3255": 58533613568.0,
+ "3260": 58533670912.0,
+ "3265": 58533666816.0,
+ "3270": 58533666816.0,
+ "3275": 58533703680.0,
+ "3280": 58533654528.0,
+ "3285": 58533752832.0,
+ "3290": 58533777408.0,
+ "3295": 58533838848.0,
+ "3300": 58533703680.0,
+ "3305": 58533724160.0,
+ "3310": 58533666816.0,
+ "3315": 58533847040.0,
+ "3320": 58533781504.0,
+ "3325": 58533568512.0,
+ "3330": 58533724160.0,
+ "3335": 58533683200.0,
+ "3340": 58533654528.0,
+ "3345": 58533855232.0,
+ "3350": 58533871616.0,
+ "3355": 58533756928.0,
+ "3360": 58533748736.0,
+ "3365": 58533777408.0,
+ "3370": 58533744640.0,
+ "3375": 58533732352.0,
+ "3380": 58533691392.0,
+ "3385": 58533621760.0,
+ "3390": 58533675008.0,
+ "3395": 58533740544.0,
+ "3400": 58533654528.0,
+ "3405": 58533732352.0,
+ "3410": 58533744640.0,
+ "3415": 58533871616.0,
+ "3420": 58533687296.0,
+ "3425": 58533662720.0,
+ "3430": 58533830656.0,
+ "3435": 58533961728.0,
+ "3440": 58533625856.0,
+ "3445": 58533535744.0,
+ "3450": 58533572608.0,
+ "3455": 58533724160.0,
+ "3460": 58533908480.0,
+ "3465": 58533720064.0,
+ "3470": 58533761024.0,
+ "3475": 58533806080.0,
+ "3480": 58533732352.0,
+ "3485": 58533605376.0,
+ "3490": 58533646336.0,
+ "3495": 58533765120.0,
+ "3500": 58533707776.0,
+ "3505": 58533646336.0,
+ "3510": 58533670912.0,
+ "3515": 58533744640.0,
+ "3520": 58533707776.0,
+ "3525": 58533617664.0,
+ "3530": 58533826560.0,
+ "3535": 58533851136.0,
+ "3540": 58533781504.0,
+ "3545": 58533736448.0,
+ "3550": 58533830656.0,
+ "3555": 58533789696.0,
+ "3560": 58533703680.0,
+ "3565": 58533584896.0,
+ "3570": 58533691392.0,
+ "3575": 58533756928.0,
+ "3580": 58533773312.0,
+ "3585": 58533662720.0,
+ "3590": 58533609472.0,
+ "3595": 58533748736.0,
+ "3600": 58533625856.0,
+ "3605": 58533740544.0,
+ "3610": 58533797888.0,
+ "3615": 58533838848.0,
+ "3620": 58533699584.0,
+ "3625": 58533720064.0,
+ "3630": 58533642240.0,
+ "3635": 58533670912.0,
+ "3640": 58533789696.0,
+ "3645": 58533814272.0,
+ "3650": 58533670912.0,
+ "3655": 58533781504.0,
+ "3660": 58533785600.0,
+ "3665": 58533715968.0,
+ "3670": 58533769216.0,
+ "3675": 58533720064.0,
+ "3680": 58533793792.0,
+ "3685": 58533830656.0,
+ "3690": 58533625856.0,
+ "3695": 58533740544.0,
+ "3700": 58533683200.0,
+ "3705": 58533785600.0,
+ "3710": 58533666816.0,
+ "3715": 58533773312.0,
+ "3720": 58533875712.0,
+ "3725": 58533752832.0,
+ "3730": 58533801984.0,
+ "3735": 58533679104.0,
+ "3740": 58533761024.0,
+ "3745": 58533883904.0,
+ "3750": 58533908480.0,
+ "3755": 58533687296.0,
+ "3760": 58533797888.0,
+ "3765": 58533748736.0,
+ "3770": 58533679104.0,
+ "3775": 58533724160.0,
+ "3780": 58533756928.0,
+ "3785": 58533838848.0,
+ "3790": 58533732352.0,
+ "3795": 58533720064.0,
+ "3800": 58533879808.0,
+ "3805": 58533900288.0,
+ "3810": 58533621760.0,
+ "3815": 58533490688.0,
+ "3820": 58533765120.0,
+ "3825": 58533781504.0,
+ "3830": 58533842944.0,
+ "3835": 58533900288.0,
+ "3840": 58533859328.0,
+ "3845": 58533629952.0,
+ "3850": 58533699584.0,
+ "3855": 58533814272.0,
+ "3860": 58533761024.0,
+ "3865": 58533683200.0,
+ "3870": 58533773312.0,
+ "3875": 58533658624.0,
+ "3880": 58533695488.0,
+ "3885": 58533777408.0,
+ "3890": 58533875712.0,
+ "3895": 58533568512.0,
+ "3900": 58533707776.0,
+ "3905": 58533609472.0,
+ "3910": 58533691392.0,
+ "3915": 58533662720.0,
+ "3920": 58533818368.0,
+ "3925": 58533679104.0,
+ "3930": 58533728256.0,
+ "3935": 58533830656.0,
+ "3940": 58533662720.0,
+ "3945": 58533683200.0,
+ "3950": 58533724160.0,
+ "3955": 58533834752.0,
+ "3960": 58533736448.0,
+ "3965": 58533797888.0,
+ "3970": 58533728256.0,
+ "3975": 58533810176.0,
+ "3980": 58533773312.0,
+ "3985": 58533826560.0,
+ "3990": 58533883904.0,
+ "3995": 58533855232.0,
+ "4000": 58533650432.0,
+ "4005": 58533642240.0,
+ "4010": 58533691392.0,
+ "4015": 58533658624.0,
+ "4020": 58533797888.0,
+ "4025": 58533638144.0,
+ "4030": 58533744640.0,
+ "4035": 58533736448.0,
+ "4040": 58533834752.0,
+ "4045": 58533744640.0,
+ "4050": 58533715968.0,
+ "4055": 58533687296.0,
+ "4060": 58533519360.0,
+ "4065": 58533613568.0,
+ "4070": 58533711872.0,
+ "4075": 58533687296.0,
+ "4080": 58533761024.0,
+ "4085": 58533785600.0,
+ "4090": 58533662720.0,
+ "4095": 58533699584.0,
+ "4100": 58533675008.0,
+ "4105": 58533695488.0,
+ "4110": 58533662720.0,
+ "4115": 58533765120.0,
+ "4120": 58533691392.0,
+ "4125": 58533769216.0,
+ "4130": 58533748736.0,
+ "4135": 58533740544.0,
+ "4140": 58533830656.0,
+ "4145": 58533679104.0,
+ "4150": 58533814272.0,
+ "4155": 58533769216.0,
+ "4160": 58533871616.0,
+ "4165": 58533851136.0,
+ "4170": 58533748736.0,
+ "4175": 58533761024.0,
+ "4180": 58533568512.0,
+ "4185": 58533621760.0,
+ "4190": 58533883904.0,
+ "4195": 58533765120.0,
+ "4200": 58533691392.0,
+ "4205": 58533740544.0,
+ "4210": 58533785600.0,
+ "4215": 58533707776.0,
+ "4220": 58533715968.0,
+ "4225": 58533711872.0,
+ "4230": 58533707776.0,
+ "4235": 58533826560.0,
+ "4240": 58533715968.0,
+ "4245": 58533642240.0,
+ "4250": 58533879808.0,
+ "4255": 58533797888.0,
+ "4260": 58533748736.0,
+ "4265": 58533597184.0,
+ "4270": 58533646336.0,
+ "4275": 58533834752.0,
+ "4280": 58533879808.0,
+ "4285": 58533793792.0,
+ "4290": 58533691392.0,
+ "4295": 58533617664.0,
+ "4300": 58533736448.0,
+ "4305": 58533662720.0,
+ "4310": 58533855232.0,
+ "4315": 58533646336.0,
+ "4320": 58533752832.0,
+ "4325": 58533822464.0,
+ "4330": 58533814272.0,
+ "4335": 58533699584.0,
+ "4340": 58533793792.0,
+ "4345": 58533740544.0,
+ "4350": 58533740544.0,
+ "4355": 58533744640.0,
+ "4360": 58533883904.0,
+ "4365": 58533670912.0,
+ "4370": 58533691392.0,
+ "4375": 58533609472.0,
+ "4380": 58533879808.0,
+ "4385": 58533773312.0,
+ "4390": 58533695488.0,
+ "4395": 58533724160.0,
+ "4400": 58533855232.0,
+ "4405": 58533650432.0,
+ "4410": 58533748736.0,
+ "4415": 58533617664.0,
+ "4420": 58533777408.0,
+ "4425": 58533642240.0,
+ "4430": 58533847040.0,
+ "4435": 58533810176.0,
+ "4440": 58533826560.0,
+ "4445": 58533773312.0,
+ "4450": 58533666816.0,
+ "4455": 58533736448.0,
+ "4460": 58533797888.0,
+ "4465": 58533863424.0,
+ "4470": 58533769216.0,
+ "4475": 58533765120.0,
+ "4480": 58533834752.0,
+ "4485": 58533625856.0,
+ "4490": 58533752832.0,
+ "4495": 58533691392.0,
+ "4500": 58533638144.0,
+ "4505": 58533703680.0,
+ "4510": 58533670912.0,
+ "4515": 58533748736.0,
+ "4520": 58533756928.0,
+ "4525": 58533552128.0,
+ "4530": 58533801984.0,
+ "4535": 58533732352.0,
+ "4540": 58533654528.0,
+ "4545": 58533691392.0,
+ "4550": 58533761024.0,
+ "4555": 58533728256.0,
+ "4560": 58533699584.0,
+ "4565": 58533785600.0,
+ "4570": 58533670912.0,
+ "4575": 58533728256.0,
+ "4580": 58533777408.0,
+ "4585": 58533806080.0,
+ "4590": 58533765120.0,
+ "4595": 58533801984.0,
+ "4600": 58533855232.0,
+ "4605": 58533634048.0,
+ "4610": 58533457920.0,
+ "4615": 58533781504.0,
+ "4620": 58533912576.0,
+ "4625": 58533675008.0,
+ "4630": 58533761024.0,
+ "4635": 58533683200.0,
+ "4640": 58533752832.0,
+ "4645": 58533773312.0,
+ "4650": 58533695488.0,
+ "4655": 58533593088.0,
+ "4660": 58533851136.0,
+ "4665": 58533715968.0,
+ "4670": 58533535744.0,
+ "4675": 58533715968.0,
+ "4680": 58533642240.0,
+ "4685": 58533777408.0,
+ "4690": 58533879808.0,
+ "4695": 58533720064.0,
+ "4700": 58533666816.0,
+ "4705": 58533736448.0,
+ "4710": 58533924864.0,
+ "4715": 58533789696.0,
+ "4720": 58533691392.0,
+ "4725": 58533773312.0,
+ "4730": 58533707776.0,
+ "4735": 58533834752.0,
+ "4740": 58533687296.0,
+ "4745": 58533707776.0,
+ "4750": 58533818368.0,
+ "4755": 58533695488.0,
+ "4760": 58533720064.0,
+ "4765": 58533773312.0,
+ "4770": 58533707776.0,
+ "4775": 58533761024.0,
+ "4780": 58533810176.0,
+ "4785": 58533797888.0,
+ "4790": 58533728256.0,
+ "4795": 58533675008.0,
+ "4800": 58533732352.0,
+ "4805": 58533740544.0,
+ "4810": 58533646336.0,
+ "4815": 58533810176.0,
+ "4820": 58533855232.0,
+ "4825": 58533773312.0,
+ "4830": 58533806080.0,
+ "4835": 58533838848.0,
+ "4840": 58533740544.0,
+ "4845": 58533777408.0,
+ "4850": 58533818368.0,
+ "4855": 58533511168.0,
+ "4860": 58533629952.0,
+ "4865": 58533814272.0,
+ "4870": 58533765120.0,
+ "4875": 58533773312.0,
+ "4880": 58533675008.0,
+ "4885": 58533822464.0,
+ "4890": 58533748736.0,
+ "4895": 58533761024.0,
+ "4900": 58533654528.0,
+ "4905": 58533797888.0,
+ "4910": 58533797888.0,
+ "4915": 58533588992.0,
+ "4920": 58533761024.0,
+ "4925": 58533847040.0,
+ "4930": 58533629952.0,
+ "4935": 58533806080.0,
+ "4940": 58533781504.0,
+ "4945": 58533740544.0,
+ "4950": 58533834752.0,
+ "4955": 58533789696.0,
+ "4960": 58533834752.0,
+ "4965": 58533724160.0,
+ "4970": 58533756928.0,
+ "4975": 58533462016.0,
+ "4980": 58533703680.0,
+ "4985": 58533765120.0,
+ "4990": 58533875712.0,
+ "4995": 58533625856.0,
+ "5000": 58533879808.0,
+ "5005": 58533703680.0,
+ "5010": 58533670912.0,
+ "5015": 58533781504.0,
+ "5020": 58533814272.0,
+ "5025": 58533724160.0,
+ "5030": 58533715968.0,
+ "5035": 58533679104.0,
+ "5040": 58533847040.0,
+ "5045": 58533695488.0,
+ "5050": 58533756928.0,
+ "5055": 58533724160.0,
+ "5060": 58533666816.0,
+ "5065": 58533629952.0,
+ "5070": 58533662720.0,
+ "5075": 58533785600.0,
+ "5080": 58533715968.0,
+ "5085": 58533769216.0,
+ "5090": 58533703680.0,
+ "5095": 58533793792.0,
+ "5100": 58533703680.0,
+ "5105": 58533912576.0,
+ "5110": 58533797888.0,
+ "5115": 58533838848.0,
+ "5120": 58533617664.0,
+ "5125": 58533867520.0,
+ "5130": 58533818368.0,
+ "5135": 58533814272.0,
+ "5140": 58533679104.0,
+ "5145": 58533748736.0,
+ "5150": 58533732352.0,
+ "5155": 58533806080.0,
+ "5160": 58533609472.0,
+ "5165": 58533662720.0,
+ "5170": 58533683200.0,
+ "5175": 58533773312.0,
+ "5180": 58533761024.0,
+ "5185": 58533720064.0,
+ "5190": 58533777408.0,
+ "5195": 58533765120.0,
+ "5200": 58533748736.0,
+ "5205": 58533638144.0,
+ "5210": 58533814272.0,
+ "5215": 58533670912.0,
+ "5220": 58533675008.0,
+ "5225": 58533756928.0,
+ "5230": 58533744640.0,
+ "5235": 58533715968.0,
+ "5240": 58533699584.0,
+ "5245": 58533650432.0,
+ "5250": 58533584896.0,
+ "5255": 58533756928.0,
+ "5260": 58533810176.0,
+ "5265": 58533720064.0,
+ "5270": 58533728256.0,
+ "5275": 58533638144.0,
+ "5280": 58533560320.0,
+ "5285": 58533769216.0,
+ "5290": 58533830656.0,
+ "5295": 58533781504.0,
+ "5300": 58533613568.0,
+ "5305": 58533736448.0,
+ "5310": 58533883904.0,
+ "5315": 58533699584.0,
+ "5320": 58533720064.0,
+ "5325": 58533888000.0,
+ "5330": 58533761024.0,
+ "5335": 58533838848.0,
+ "5340": 58533593088.0,
+ "5345": 58533851136.0,
+ "5350": 58533695488.0,
+ "5355": 58533806080.0,
+ "5360": 58533748736.0,
+ "5365": 58533703680.0,
+ "5370": 58533695488.0,
+ "5375": 58533732352.0,
+ "5380": 58533781504.0,
+ "5385": 58533801984.0,
+ "5390": 58533629952.0,
+ "5395": 58533724160.0,
+ "5400": 58533715968.0,
+ "5405": 58533601280.0,
+ "5410": 58533810176.0,
+ "5415": 58533715968.0,
+ "5420": 58533736448.0,
+ "5425": 58533949440.0,
+ "5430": 58533687296.0,
+ "5435": 58533785600.0,
+ "5440": 58533703680.0,
+ "5445": 58533896192.0,
+ "5450": 58533662720.0,
+ "5455": 58533654528.0,
+ "5460": 58533761024.0,
+ "5465": 58533691392.0,
+ "5470": 58533871616.0,
+ "5475": 58533732352.0,
+ "5480": 58533593088.0,
+ "5485": 58533756928.0,
+ "5490": 58533769216.0,
+ "5495": 58533756928.0,
+ "5500": 58533732352.0,
+ "5505": 58533699584.0,
+ "5510": 58533806080.0,
+ "5515": 58533691392.0,
+ "5520": 58533703680.0,
+ "5525": 58533597184.0,
+ "5530": 58533834752.0,
+ "5535": 58533810176.0,
+ "5540": 58533724160.0,
+ "5545": 58533724160.0,
+ "5550": 58533683200.0,
+ "5555": 58533707776.0,
+ "5560": 58533695488.0,
+ "5565": 58533801984.0,
+ "5570": 58533642240.0,
+ "5575": 58533687296.0,
+ "5580": 58533855232.0,
+ "5585": 58533609472.0,
+ "5590": 58533765120.0,
+ "5595": 58533580800.0,
+ "5600": 58533695488.0,
+ "5605": 58533695488.0,
+ "5610": 58533855232.0,
+ "5615": 58533654528.0,
+ "5620": 58533695488.0,
+ "5625": 58533761024.0,
+ "5630": 58533752832.0,
+ "5635": 58533679104.0,
+ "5640": 58533736448.0,
+ "5645": 58533642240.0,
+ "5650": 58533732352.0,
+ "5655": 58533699584.0,
+ "5660": 58533691392.0,
+ "5665": 58533761024.0,
+ "5670": 58533748736.0,
+ "5675": 58533736448.0,
+ "5680": 58533806080.0,
+ "5685": 58533720064.0,
+ "5690": 58533650432.0,
+ "5695": 58533736448.0,
+ "5700": 58533830656.0,
+ "5705": 58533597184.0,
+ "5710": 58533675008.0,
+ "5715": 58533797888.0,
+ "5720": 58533654528.0,
+ "5725": 58533728256.0,
+ "5730": 58533777408.0,
+ "5735": 58533859328.0,
+ "5740": 58533670912.0,
+ "5745": 58533617664.0,
+ "5750": 58533793792.0,
+ "5755": 58533724160.0,
+ "5760": 58533744640.0,
+ "5765": 58533765120.0,
+ "5770": 58533482496.0,
+ "5775": 58533765120.0,
+ "5780": 58533711872.0,
+ "5785": 58533761024.0,
+ "5790": 58533675008.0,
+ "5795": 58533797888.0,
+ "5800": 58533707776.0,
+ "5805": 58533765120.0,
+ "5810": 58533695488.0,
+ "5815": 58533756928.0,
+ "5820": 58533777408.0,
+ "5825": 58533699584.0,
+ "5830": 58533548032.0,
+ "5835": 58533871616.0,
+ "5840": 58533695488.0,
+ "5845": 58533658624.0,
+ "5850": 58533761024.0,
+ "5855": 58533609472.0,
+ "5860": 58533756928.0,
+ "5865": 58533769216.0,
+ "5870": 58533748736.0,
+ "5875": 58533736448.0,
+ "5880": 58533699584.0,
+ "5885": 58533871616.0,
+ "5890": 58533601280.0,
+ "5895": 58533666816.0,
+ "5900": 58533789696.0,
+ "5905": 58533740544.0,
+ "5910": 58533576704.0,
+ "5915": 58533752832.0,
+ "5920": 58533851136.0,
+ "5925": 58533675008.0,
+ "5930": 58533675008.0,
+ "5935": 58533711872.0,
+ "5940": 58533670912.0,
+ "5945": 58533916672.0,
+ "5950": 58533609472.0,
+ "5955": 58533601280.0,
+ "5960": 58533736448.0,
+ "5965": 58533711872.0,
+ "5970": 58533703680.0,
+ "5975": 58533580800.0,
+ "5980": 58533851136.0,
+ "5985": 58533691392.0,
+ "5990": 58533748736.0,
+ "5995": 58533703680.0,
+ "6000": 58533806080.0,
+ "6005": 58533646336.0,
+ "6010": 58533793792.0,
+ "6015": 58533679104.0,
+ "6020": 58533761024.0,
+ "6025": 58533847040.0,
+ "6030": 58533691392.0,
+ "6035": 58533797888.0,
+ "6040": 58533793792.0,
+ "6045": 58533867520.0,
+ "6050": 58533871616.0,
+ "6055": 58533847040.0,
+ "6060": 58533584896.0,
+ "6065": 58533781504.0,
+ "6070": 58533752832.0,
+ "6075": 58533609472.0,
+ "6080": 58533773312.0,
+ "6085": 58533822464.0,
+ "6090": 58533871616.0,
+ "6095": 58533732352.0,
+ "6100": 58533785600.0,
+ "6105": 58533879808.0,
+ "6110": 58533863424.0,
+ "6115": 58533761024.0,
+ "6120": 58533826560.0,
+ "6125": 58533863424.0,
+ "6130": 58533806080.0,
+ "6135": 58533642240.0,
+ "6140": 58533773312.0,
+ "6145": 58533625856.0,
+ "6150": 58533675008.0,
+ "6155": 58533638144.0,
+ "6160": 58533814272.0,
+ "6165": 58533683200.0,
+ "6170": 58533756928.0,
+ "6175": 58533851136.0,
+ "6180": 58533801984.0,
+ "6185": 58533650432.0,
+ "6190": 58533830656.0,
+ "6195": 58533724160.0,
+ "6200": 58533707776.0,
+ "6205": 58533838848.0,
+ "6210": 58533740544.0,
+ "6215": 58533756928.0,
+ "6220": 58533670912.0,
+ "6225": 58533699584.0,
+ "6230": 58533638144.0,
+ "6235": 58533679104.0,
+ "6240": 58533822464.0,
+ "6245": 58533781504.0,
+ "6250": 58533900288.0,
+ "6255": 58533679104.0,
+ "6260": 58533572608.0,
+ "6265": 58533806080.0,
+ "6270": 58533789696.0,
+ "6275": 58533687296.0,
+ "6280": 58533711872.0,
+ "6285": 58533761024.0,
+ "6290": 58533707776.0,
+ "6295": 58533789696.0,
+ "6300": 58533658624.0,
+ "6305": 58533728256.0,
+ "6310": 58533756928.0,
+ "6315": 58533785600.0,
+ "6320": 58533601280.0,
+ "6325": 58533744640.0,
+ "6330": 58533773312.0,
+ "6335": 58533748736.0,
+ "6340": 58533695488.0,
+ "6345": 58533728256.0,
+ "6350": 58533699584.0,
+ "6355": 58533781504.0,
+ "6360": 58533646336.0,
+ "6365": 58533740544.0,
+ "6370": 58533732352.0,
+ "6375": 58533756928.0,
+ "6380": 58533740544.0,
+ "6385": 58533617664.0,
+ "6390": 58533765120.0,
+ "6395": 58533928960.0,
+ "6400": 58533687296.0,
+ "6405": 58533715968.0,
+ "6410": 58533695488.0,
+ "6415": 58533806080.0,
+ "6420": 58533789696.0,
+ "6425": 58533707776.0,
+ "6430": 58533752832.0,
+ "6435": 58533662720.0,
+ "6440": 58533691392.0,
+ "6445": 58533580800.0,
+ "6450": 58533974016.0,
+ "6455": 58533814272.0,
+ "6460": 58533838848.0,
+ "6465": 58533748736.0,
+ "6470": 58533601280.0,
+ "6475": 58533756928.0,
+ "6480": 58533646336.0,
+ "6485": 58533765120.0,
+ "6490": 58533826560.0,
+ "6495": 58533842944.0,
+ "6500": 58533810176.0,
+ "6505": 58533654528.0,
+ "6510": 58533683200.0,
+ "6515": 58533748736.0,
+ "6520": 58533720064.0,
+ "6525": 58533904384.0,
+ "6530": 58533826560.0,
+ "6535": 58533707776.0,
+ "6540": 58533593088.0,
+ "6545": 58533801984.0,
+ "6550": 58533670912.0,
+ "6555": 58533867520.0,
+ "6560": 58533650432.0,
+ "6565": 58533588992.0,
+ "6570": 58533707776.0,
+ "6575": 58533781504.0,
+ "6580": 58533765120.0,
+ "6585": 58533797888.0,
+ "6590": 58533761024.0,
+ "6595": 58533769216.0,
+ "6600": 58533756928.0,
+ "6605": 58533748736.0,
+ "6610": 58533572608.0,
+ "6615": 58533715968.0,
+ "6620": 58533777408.0,
+ "6625": 58533494784.0,
+ "6630": 58533666816.0,
+ "6635": 58533806080.0,
+ "6640": 58533642240.0,
+ "6645": 58533781504.0,
+ "6650": 58533711872.0,
+ "6655": 58533756928.0,
+ "6660": 58533634048.0,
+ "6665": 58533801984.0,
+ "6670": 58533818368.0,
+ "6675": 58533732352.0,
+ "6680": 58533695488.0,
+ "6685": 58533613568.0,
+ "6690": 58533732352.0,
+ "6695": 58533699584.0,
+ "6700": 58533707776.0,
+ "6705": 58533777408.0,
+ "6710": 58533597184.0,
+ "6715": 58533838848.0,
+ "6720": 58533797888.0,
+ "6725": 58533728256.0,
+ "6730": 58533781504.0,
+ "6735": 58533707776.0,
+ "6740": 58533638144.0,
+ "6745": 58533695488.0,
+ "6750": 58533666816.0,
+ "6755": 58533703680.0,
+ "6760": 58533707776.0,
+ "6765": 58533785600.0,
+ "6770": 58533756928.0,
+ "6775": 58533625856.0,
+ "6780": 58533838848.0,
+ "6785": 58533752832.0,
+ "6790": 58533715968.0,
+ "6795": 58533654528.0,
+ "6800": 58533826560.0,
+ "6805": 58533670912.0,
+ "6810": 58533597184.0,
+ "6815": 58533744640.0,
+ "6820": 58533666816.0,
+ "6825": 58533715968.0,
+ "6830": 58533703680.0,
+ "6835": 58533732352.0,
+ "6840": 58533666816.0,
+ "6845": 58533711872.0,
+ "6850": 58533740544.0,
+ "6855": 58533855232.0,
+ "6860": 58533801984.0,
+ "6865": 58533756928.0,
+ "6870": 58533613568.0,
+ "6875": 58533937152.0,
+ "6880": 58533675008.0,
+ "6885": 58533871616.0,
+ "6890": 58533732352.0,
+ "6895": 58533863424.0,
+ "6900": 58533855232.0,
+ "6905": 58533601280.0,
+ "6910": 58533822464.0,
+ "6915": 58533814272.0,
+ "6920": 58533703680.0,
+ "6925": 58533789696.0,
+ "6930": 58533662720.0,
+ "6935": 58533683200.0,
+ "6940": 58533670912.0,
+ "6945": 58533756928.0,
+ "6950": 58533711872.0,
+ "6955": 58533711872.0,
+ "6960": 58533777408.0,
+ "6965": 58533662720.0,
+ "6970": 58533720064.0,
+ "6975": 58533724160.0,
+ "6980": 58533859328.0,
+ "6985": 58533748736.0,
+ "6990": 58533711872.0,
+ "6995": 58533519360.0,
+ "7000": 58533613568.0,
+ "7005": 58533666816.0,
+ "7010": 58533654528.0,
+ "7015": 58533724160.0,
+ "7020": 58533699584.0,
+ "7025": 58533806080.0,
+ "7030": 58533715968.0,
+ "7035": 58533773312.0,
+ "7040": 58533597184.0,
+ "7045": 58533863424.0,
+ "7050": 58533613568.0,
+ "7055": 58533683200.0,
+ "7060": 58533818368.0,
+ "7065": 58533650432.0,
+ "7070": 58533765120.0,
+ "7075": 58533675008.0,
+ "7080": 58533679104.0,
+ "7085": 58533740544.0,
+ "7090": 58533756928.0,
+ "7095": 58533765120.0,
+ "7100": 58533699584.0,
+ "7105": 58533769216.0,
+ "7110": 58533740544.0,
+ "7115": 58533732352.0,
+ "7120": 58533756928.0,
+ "7125": 58533687296.0,
+ "7130": 58533736448.0,
+ "7135": 58533732352.0,
+ "7140": 58533662720.0,
+ "7145": 58533773312.0,
+ "7150": 58533720064.0,
+ "7155": 58533752832.0,
+ "7160": 58533785600.0,
+ "7165": 58533785600.0,
+ "7170": 58533744640.0,
+ "7175": 58533642240.0,
+ "7180": 58533732352.0,
+ "7185": 58533793792.0,
+ "7190": 58533765120.0,
+ "7195": 58533654528.0,
+ "7200": 58533654528.0,
+ "7205": 58533814272.0,
+ "7210": 58533748736.0,
+ "7215": 58533863424.0,
+ "7220": 58533769216.0,
+ "7225": 58533650432.0,
+ "7230": 58533756928.0,
+ "7235": 58533625856.0,
+ "7240": 58533740544.0,
+ "7245": 58533728256.0,
+ "7250": 58533670912.0,
+ "7255": 58533761024.0,
+ "7260": 58533642240.0,
+ "7265": 58533724160.0,
+ "7270": 58533699584.0,
+ "7275": 58533806080.0,
+ "7280": 58533777408.0,
+ "7285": 58533720064.0,
+ "7290": 58533789696.0,
+ "7295": 58533539840.0,
+ "7300": 58533658624.0,
+ "7305": 58533769216.0,
+ "7310": 58533769216.0,
+ "7315": 58533703680.0,
+ "7320": 58533724160.0,
+ "7325": 58533748736.0,
+ "7330": 58533711872.0,
+ "7335": 58533670912.0,
+ "7340": 58533916672.0,
+ "7345": 58533765120.0,
+ "7350": 58533658624.0,
+ "7355": 58533781504.0,
+ "7360": 58533560320.0,
+ "7365": 58533658624.0,
+ "7370": 58533642240.0,
+ "7375": 58533851136.0,
+ "7380": 58533703680.0,
+ "7385": 58533691392.0,
+ "7390": 58533687296.0,
+ "7395": 58533675008.0,
+ "7400": 58533879808.0,
+ "7405": 58533629952.0,
+ "7410": 58533744640.0,
+ "7415": 58533699584.0,
+ "7420": 58533601280.0,
+ "7425": 58533691392.0,
+ "7430": 58533810176.0,
+ "7435": 58533879808.0,
+ "7440": 58533679104.0,
+ "7445": 58533892096.0,
+ "7450": 58533720064.0,
+ "7455": 58533756928.0,
+ "7460": 58533777408.0,
+ "7465": 58533707776.0,
+ "7470": 58533740544.0,
+ "7475": 58533842944.0,
+ "7480": 58533650432.0,
+ "7485": 58533728256.0,
+ "7490": 58533756928.0,
+ "7495": 58533814272.0,
+ "7500": 58533756928.0,
+ "7505": 58533810176.0,
+ "7510": 58533732352.0,
+ "7515": 58533740544.0,
+ "7520": 58533715968.0,
+ "7525": 58533736448.0,
+ "7530": 58533715968.0,
+ "7535": 58533847040.0,
+ "7540": 58533679104.0,
+ "7545": 58533863424.0,
+ "7550": 58533646336.0,
+ "7555": 58533761024.0,
+ "7560": 58533638144.0,
+ "7565": 58533830656.0,
+ "7570": 58533761024.0,
+ "7575": 58533679104.0,
+ "7580": 58533867520.0,
+ "7585": 58533732352.0,
+ "7590": 58533801984.0,
+ "7595": 58533736448.0,
+ "7600": 58533715968.0,
+ "7605": 58533646336.0,
+ "7610": 58533707776.0,
+ "7615": 58533847040.0,
+ "7620": 58533728256.0,
+ "7625": 58533736448.0,
+ "7630": 58533826560.0,
+ "7635": 58533777408.0,
+ "7640": 58533773312.0,
+ "7645": 58533765120.0,
+ "7650": 58533814272.0,
+ "7655": 58533658624.0,
+ "7660": 58533806080.0,
+ "7665": 58533593088.0,
+ "7670": 58533736448.0,
+ "7675": 58533752832.0,
+ "7680": 58533756928.0,
+ "7685": 58533736448.0,
+ "7690": 58533765120.0,
+ "7695": 58533761024.0,
+ "7700": 58533859328.0,
+ "7705": 58533740544.0,
+ "7710": 58533756928.0,
+ "7715": 58533699584.0,
+ "7720": 58533642240.0,
+ "7725": 58533609472.0,
+ "7730": 58533797888.0,
+ "7735": 58533736448.0,
+ "7740": 58533740544.0,
+ "7745": 58533691392.0,
+ "7750": 58533748736.0,
+ "7755": 58533675008.0,
+ "7760": 58533814272.0,
+ "7765": 58533838848.0,
+ "7770": 58533847040.0,
+ "7775": 58533789696.0,
+ "7780": 58533605376.0,
+ "7785": 58533584896.0,
+ "7790": 58533781504.0,
+ "7795": 58533748736.0,
+ "7800": 58533761024.0,
+ "7805": 58533732352.0,
+ "7810": 58533715968.0,
+ "7815": 58533773312.0,
+ "7820": 58533761024.0,
+ "7825": 58533912576.0,
+ "7830": 58533740544.0,
+ "7835": 58533761024.0,
+ "7840": 58533740544.0,
+ "7845": 58533629952.0,
+ "7850": 58533752832.0,
+ "7855": 58533724160.0,
+ "7860": 58533822464.0,
+ "7865": 58533634048.0,
+ "7870": 58533777408.0,
+ "7875": 58533601280.0,
+ "7880": 58533646336.0,
+ "7885": 58533785600.0,
+ "7890": 58533699584.0,
+ "7895": 58533720064.0,
+ "7900": 58533736448.0,
+ "7905": 58533646336.0,
+ "7910": 58533453824.0,
+ "7915": 58533814272.0,
+ "7920": 58533744640.0,
+ "7925": 58533707776.0,
+ "7930": 58533761024.0,
+ "7935": 58533855232.0,
+ "7940": 58533752832.0,
+ "7945": 58533699584.0,
+ "7950": 58533707776.0,
+ "7955": 58533801984.0,
+ "7960": 58533842944.0,
+ "7965": 58533732352.0,
+ "7970": 58533613568.0,
+ "7975": 58533773312.0,
+ "7980": 58533888000.0,
+ "7985": 58533904384.0,
+ "7990": 58533724160.0,
+ "7995": 58533699584.0,
+ "8000": 58533752832.0,
+ "8005": 58533609472.0,
+ "8010": 58533863424.0,
+ "8015": 58533756928.0,
+ "8020": 58533847040.0,
+ "8025": 58533793792.0,
+ "8030": 58533560320.0,
+ "8035": 58533826560.0,
+ "8040": 58533756928.0,
+ "8045": 58533761024.0,
+ "8050": 58533773312.0,
+ "8055": 58533822464.0,
+ "8060": 58533625856.0,
+ "8065": 58533748736.0,
+ "8070": 58533695488.0,
+ "8075": 58533781504.0,
+ "8080": 58533695488.0,
+ "8085": 58533810176.0,
+ "8090": 58533691392.0,
+ "8095": 58533797888.0,
+ "8100": 58533740544.0,
+ "8105": 58533756928.0,
+ "8110": 58533683200.0,
+ "8115": 58533855232.0,
+ "8120": 58533806080.0,
+ "8125": 58533773312.0,
+ "8130": 58533720064.0,
+ "8135": 58533789696.0,
+ "8140": 58533736448.0,
+ "8145": 58533715968.0,
+ "8150": 58533793792.0,
+ "8155": 58533683200.0,
+ "8160": 58533810176.0,
+ "8165": 58533720064.0,
+ "8170": 58533670912.0,
+ "8175": 58533703680.0,
+ "8180": 58533617664.0,
+ "8185": 58533658624.0,
+ "8190": 58533752832.0,
+ "8195": 58533806080.0,
+ "8200": 58533707776.0,
+ "8205": 58533847040.0,
+ "8210": 58533765120.0,
+ "8215": 58533646336.0,
+ "8220": 58533769216.0,
+ "8225": 58533715968.0,
+ "8230": 58533703680.0,
+ "8235": 58533789696.0,
+ "8240": 58533888000.0,
+ "8245": 58533707776.0,
+ "8250": 58533752832.0,
+ "8255": 58533765120.0,
+ "8260": 58533797888.0,
+ "8265": 58533588992.0,
+ "8270": 58533675008.0,
+ "8275": 58533613568.0,
+ "8280": 58533683200.0,
+ "8285": 58533666816.0,
+ "8290": 58533728256.0,
+ "8295": 58533801984.0,
+ "8300": 58533773312.0,
+ "8305": 58533752832.0,
+ "8310": 58533769216.0,
+ "8315": 58533650432.0,
+ "8320": 58533752832.0,
+ "8325": 58533675008.0,
+ "8330": 58533756928.0,
+ "8335": 58533511168.0,
+ "8340": 58533703680.0,
+ "8345": 58533912576.0,
+ "8350": 58533822464.0,
+ "8355": 58533683200.0,
+ "8360": 58533781504.0,
+ "8365": 58533797888.0,
+ "8370": 58533842944.0,
+ "8375": 58533621760.0,
+ "8380": 58533814272.0,
+ "8385": 58533785600.0,
+ "8390": 58533720064.0,
+ "8395": 58533744640.0,
+ "8400": 58533756928.0,
+ "8405": 58533679104.0,
+ "8410": 58533752832.0,
+ "8415": 58533654528.0,
+ "8420": 58533634048.0,
+ "8425": 58533736448.0,
+ "8430": 58533773312.0,
+ "8435": 58533613568.0,
+ "8440": 58533675008.0,
+ "8445": 58533761024.0,
+ "8450": 58533715968.0,
+ "8455": 58533597184.0,
+ "8460": 58533625856.0,
+ "8465": 58533810176.0,
+ "8470": 58533724160.0,
+ "8475": 58533761024.0,
+ "8480": 58533814272.0,
+ "8485": 58533658624.0,
+ "8490": 58533822464.0,
+ "8495": 58533814272.0,
+ "8500": 58533756928.0,
+ "8505": 58533658624.0,
+ "8510": 58533814272.0,
+ "8515": 58533654528.0,
+ "8520": 58533548032.0,
+ "8525": 58533740544.0,
+ "8530": 58533748736.0,
+ "8535": 58533888000.0,
+ "8540": 58533720064.0,
+ "8545": 58533732352.0,
+ "8550": 58533781504.0,
+ "8555": 58533744640.0,
+ "8560": 58533867520.0,
+ "8565": 58533687296.0,
+ "8570": 58533683200.0,
+ "8575": 58533715968.0,
+ "8580": 58533552128.0,
+ "8585": 58533756928.0,
+ "8590": 58533789696.0,
+ "8595": 58533642240.0,
+ "8600": 58533724160.0,
+ "8605": 58533654528.0,
+ "8610": 58533761024.0,
+ "8615": 58533797888.0,
+ "8620": 58533679104.0,
+ "8625": 58533675008.0,
+ "8630": 58533617664.0,
+ "8635": 58533691392.0,
+ "8640": 58533556224.0,
+ "8645": 58533617664.0,
+ "8650": 58533830656.0,
+ "8655": 58533720064.0,
+ "8660": 58533687296.0,
+ "8665": 58533670912.0,
+ "8670": 58533834752.0,
+ "8675": 58533761024.0,
+ "8680": 58533847040.0,
+ "8685": 58533683200.0,
+ "8690": 58533724160.0,
+ "8695": 58533818368.0,
+ "8700": 58533617664.0,
+ "8705": 58533711872.0,
+ "8710": 58533711872.0,
+ "8715": 58533728256.0,
+ "8720": 58533756928.0,
+ "8725": 58533703680.0,
+ "8730": 58533711872.0,
+ "8735": 58533642240.0,
+ "8740": 58533724160.0,
+ "8745": 58533859328.0,
+ "8750": 58533675008.0,
+ "8755": 58533765120.0,
+ "8760": 58533539840.0,
+ "8765": 58533642240.0,
+ "8770": 58533650432.0,
+ "8775": 58533781504.0,
+ "8780": 58533773312.0,
+ "8785": 58533715968.0,
+ "8790": 58533695488.0,
+ "8795": 58533662720.0,
+ "8800": 58533695488.0,
+ "8805": 58533584896.0,
+ "8810": 58533756928.0,
+ "8815": 58533720064.0,
+ "8820": 58533666816.0,
+ "8825": 58533527552.0,
+ "8830": 58533650432.0,
+ "8835": 58533789696.0,
+ "8840": 58533699584.0,
+ "8845": 58533703680.0,
+ "8850": 58533736448.0,
+ "8855": 58533781504.0,
+ "8860": 58533539840.0,
+ "8865": 58533806080.0,
+ "8870": 58533801984.0,
+ "8875": 58533601280.0,
+ "8880": 58533761024.0,
+ "8885": 58533560320.0,
+ "8890": 58533715968.0,
+ "8895": 58533769216.0,
+ "8900": 58533806080.0,
+ "8905": 58533801984.0,
+ "8910": 58533642240.0,
+ "8915": 58533761024.0,
+ "8920": 58533695488.0,
+ "8925": 58533806080.0,
+ "8930": 58533699584.0,
+ "8935": 58533703680.0,
+ "8940": 58533724160.0,
+ "8945": 58533621760.0,
+ "8950": 58533670912.0,
+ "8955": 58533732352.0,
+ "8960": 58533761024.0,
+ "8965": 58533769216.0,
+ "8970": 58533810176.0,
+ "8975": 58533675008.0,
+ "8980": 58533847040.0,
+ "8985": 58533801984.0,
+ "8990": 58533646336.0,
+ "8995": 58533744640.0,
+ "9000": 58533847040.0,
+ "9005": 58533728256.0,
+ "9010": 58533617664.0,
+ "9015": 58533695488.0,
+ "9020": 58533756928.0,
+ "9025": 58533728256.0,
+ "9030": 58533863424.0,
+ "9035": 58533769216.0,
+ "9040": 58533777408.0,
+ "9045": 58533650432.0,
+ "9050": 58533646336.0,
+ "9055": 58533683200.0,
+ "9060": 58533781504.0,
+ "9065": 58533625856.0,
+ "9070": 58533658624.0,
+ "9075": 58533675008.0,
+ "9080": 58533658624.0,
+ "9085": 58533736448.0,
+ "9090": 58533703680.0,
+ "9095": 58533773312.0,
+ "9100": 58533707776.0,
+ "9105": 58533736448.0,
+ "9110": 58533699584.0,
+ "9115": 58533904384.0,
+ "9120": 58533908480.0,
+ "9125": 58533695488.0,
+ "9130": 58533552128.0,
+ "9135": 58533785600.0,
+ "9140": 58533793792.0,
+ "9145": 58533728256.0,
+ "9150": 58533777408.0,
+ "9155": 58533769216.0,
+ "9160": 58533654528.0,
+ "9165": 58533568512.0,
+ "9170": 58533814272.0,
+ "9175": 58533699584.0,
+ "9180": 58533666816.0,
+ "9185": 58533744640.0,
+ "9190": 58533642240.0,
+ "9195": 58533605376.0,
+ "9200": 58533658624.0,
+ "9205": 58533695488.0,
+ "9210": 58533732352.0,
+ "9215": 58533765120.0,
+ "9220": 58533724160.0,
+ "9225": 58533691392.0,
+ "9230": 58533748736.0,
+ "9235": 58533789696.0,
+ "9240": 58533605376.0,
+ "9245": 58533736448.0,
+ "9250": 58533683200.0,
+ "9255": 58533646336.0,
+ "9260": 58533752832.0,
+ "9265": 58533781504.0,
+ "9270": 58533638144.0,
+ "9275": 58533658624.0,
+ "9280": 58533806080.0,
+ "9285": 58533748736.0,
+ "9290": 58533785600.0,
+ "9295": 58533801984.0,
+ "9300": 58533715968.0,
+ "9305": 58533658624.0,
+ "9310": 58533675008.0,
+ "9315": 58533625856.0,
+ "9320": 58533814272.0,
+ "9325": 58533806080.0,
+ "9330": 58533711872.0,
+ "9335": 58533625856.0,
+ "9340": 58533707776.0,
+ "9345": 58533650432.0,
+ "9350": 58533740544.0,
+ "9355": 58533826560.0,
+ "9360": 58533842944.0,
+ "9365": 58533715968.0,
+ "9370": 58533842944.0,
+ "9375": 58533572608.0,
+ "9380": 58533781504.0,
+ "9385": 58533736448.0,
+ "9390": 58533851136.0,
+ "9395": 58533724160.0,
+ "9400": 58533642240.0,
+ "9405": 58533707776.0,
+ "9410": 58533765120.0,
+ "9415": 58533818368.0,
+ "9420": 58533736448.0,
+ "9425": 58533801984.0,
+ "9430": 58533822464.0,
+ "9435": 58533572608.0,
+ "9440": 58533744640.0,
+ "9445": 58533699584.0,
+ "9450": 58533699584.0,
+ "9455": 58533744640.0,
+ "9460": 58533773312.0,
+ "9465": 58533736448.0,
+ "9470": 58533875712.0,
+ "9475": 58533904384.0,
+ "9480": 58533703680.0,
+ "9485": 58533765120.0,
+ "9490": 58533658624.0,
+ "9495": 58533597184.0,
+ "9500": 58533568512.0,
+ "9505": 58533728256.0,
+ "9510": 58533801984.0,
+ "9515": 58533785600.0,
+ "9520": 58533683200.0,
+ "9525": 58533789696.0,
+ "9530": 58533683200.0,
+ "9535": 58533650432.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 9535,
+ "step_interval": 5,
+ "values": {
+ "1": 85005172736.0,
+ "5": 99296133120.0,
+ "10": 99296133120.0,
+ "15": 99296133120.0,
+ "20": 99296133120.0,
+ "25": 99296133120.0,
+ "30": 99296133120.0,
+ "35": 99296133120.0,
+ "40": 99296133120.0,
+ "45": 99296133120.0,
+ "50": 99296133120.0,
+ "55": 99296133120.0,
+ "60": 99296133120.0,
+ "65": 99296133120.0,
+ "70": 99296133120.0,
+ "75": 99296133120.0,
+ "80": 99296133120.0,
+ "85": 99296133120.0,
+ "90": 99296133120.0,
+ "95": 99296133120.0,
+ "100": 99296133120.0,
+ "105": 99296133120.0,
+ "110": 99296133120.0,
+ "115": 99296133120.0,
+ "120": 99296133120.0,
+ "125": 99296133120.0,
+ "130": 99296133120.0,
+ "135": 99296133120.0,
+ "140": 99296133120.0,
+ "145": 99296133120.0,
+ "150": 99296133120.0,
+ "155": 99296133120.0,
+ "160": 99296133120.0,
+ "165": 99296133120.0,
+ "170": 99296133120.0,
+ "175": 99296133120.0,
+ "180": 99296133120.0,
+ "185": 99296133120.0,
+ "190": 99296133120.0,
+ "195": 99296133120.0,
+ "200": 99296133120.0,
+ "205": 99296133120.0,
+ "210": 99296133120.0,
+ "215": 99296133120.0,
+ "220": 99296133120.0,
+ "225": 99296133120.0,
+ "230": 99296133120.0,
+ "235": 99296133120.0,
+ "240": 99296133120.0,
+ "245": 99296133120.0,
+ "250": 99296133120.0,
+ "255": 99296133120.0,
+ "260": 99296133120.0,
+ "265": 99296133120.0,
+ "270": 99296133120.0,
+ "275": 99296133120.0,
+ "280": 99296133120.0,
+ "285": 99296133120.0,
+ "290": 99296133120.0,
+ "295": 99296133120.0,
+ "300": 99296133120.0,
+ "305": 99296133120.0,
+ "310": 99296133120.0,
+ "315": 99296133120.0,
+ "320": 99296133120.0,
+ "325": 99296133120.0,
+ "330": 99296133120.0,
+ "335": 99296133120.0,
+ "340": 99296133120.0,
+ "345": 99296133120.0,
+ "350": 99296133120.0,
+ "355": 99296133120.0,
+ "360": 99296133120.0,
+ "365": 99296133120.0,
+ "370": 99296133120.0,
+ "375": 99296133120.0,
+ "380": 99296133120.0,
+ "385": 99296133120.0,
+ "390": 99296133120.0,
+ "395": 99296133120.0,
+ "400": 99296133120.0,
+ "405": 99296133120.0,
+ "410": 99296133120.0,
+ "415": 99296133120.0,
+ "420": 99296133120.0,
+ "425": 99296133120.0,
+ "430": 99296133120.0,
+ "435": 99296133120.0,
+ "440": 99296133120.0,
+ "445": 99319971840.0,
+ "450": 99319971840.0,
+ "455": 99319971840.0,
+ "460": 99319971840.0,
+ "465": 99470999552.0,
+ "470": 99470999552.0,
+ "475": 99577782272.0,
+ "480": 99856146432.0,
+ "485": 99856146432.0,
+ "490": 99975626752.0,
+ "495": 100075380736.0,
+ "500": 100205191168.0,
+ "505": 100205191168.0,
+ "510": 100205191168.0,
+ "515": 100205191168.0,
+ "520": 100205191168.0,
+ "525": 100313350144.0,
+ "530": 100313350144.0,
+ "535": 100313350144.0,
+ "540": 100313350144.0,
+ "545": 100313350144.0,
+ "550": 100432117760.0,
+ "555": 100432117760.0,
+ "560": 100814946304.0,
+ "565": 100814946304.0,
+ "570": 100814946304.0,
+ "575": 100814946304.0,
+ "580": 100814946304.0,
+ "585": 100814946304.0,
+ "590": 100814946304.0,
+ "595": 100814946304.0,
+ "600": 100814946304.0,
+ "605": 100814946304.0,
+ "610": 100814946304.0,
+ "615": 100814946304.0,
+ "620": 100814946304.0,
+ "625": 100814946304.0,
+ "630": 100814946304.0,
+ "635": 100814946304.0,
+ "640": 100814946304.0,
+ "645": 100814946304.0,
+ "650": 100814946304.0,
+ "655": 100814946304.0,
+ "660": 100814946304.0,
+ "665": 100814946304.0,
+ "670": 100814946304.0,
+ "675": 100814946304.0,
+ "680": 100814946304.0,
+ "685": 100814946304.0,
+ "690": 100814946304.0,
+ "695": 100814946304.0,
+ "700": 100814946304.0,
+ "705": 100814946304.0,
+ "710": 100814946304.0,
+ "715": 100814946304.0,
+ "720": 100814946304.0,
+ "725": 100814946304.0,
+ "730": 100814946304.0,
+ "735": 100814946304.0,
+ "740": 100814946304.0,
+ "745": 100814946304.0,
+ "750": 100814946304.0,
+ "755": 100814946304.0,
+ "760": 100814946304.0,
+ "765": 100814946304.0,
+ "770": 100814946304.0,
+ "775": 100814946304.0,
+ "780": 100814946304.0,
+ "785": 100814946304.0,
+ "790": 100814946304.0,
+ "795": 100814946304.0,
+ "800": 100814946304.0,
+ "805": 100814946304.0,
+ "810": 100814946304.0,
+ "815": 100814946304.0,
+ "820": 100814946304.0,
+ "825": 100814946304.0,
+ "830": 100814946304.0,
+ "835": 100814946304.0,
+ "840": 100814946304.0,
+ "845": 100814946304.0,
+ "850": 100814946304.0,
+ "855": 100814946304.0,
+ "860": 100814946304.0,
+ "865": 100814946304.0,
+ "870": 100814946304.0,
+ "875": 100814946304.0,
+ "880": 100814946304.0,
+ "885": 100814946304.0,
+ "890": 100814946304.0,
+ "895": 100814946304.0,
+ "900": 100814946304.0,
+ "905": 100814946304.0,
+ "910": 100814946304.0,
+ "915": 100814946304.0,
+ "920": 100814946304.0,
+ "925": 100814946304.0,
+ "930": 100814946304.0,
+ "935": 100814946304.0,
+ "940": 100814946304.0,
+ "945": 100814946304.0,
+ "950": 100814946304.0,
+ "955": 100814946304.0,
+ "960": 100814946304.0,
+ "965": 100814946304.0,
+ "970": 100814946304.0,
+ "975": 100814946304.0,
+ "980": 100814946304.0,
+ "985": 100814946304.0,
+ "990": 100814946304.0,
+ "995": 100814946304.0,
+ "1000": 100814946304.0,
+ "1005": 100814946304.0,
+ "1010": 100814946304.0,
+ "1015": 100814946304.0,
+ "1020": 100814946304.0,
+ "1025": 100814946304.0,
+ "1030": 100814946304.0,
+ "1035": 100814946304.0,
+ "1040": 100814946304.0,
+ "1045": 100814946304.0,
+ "1050": 100814946304.0,
+ "1055": 100814946304.0,
+ "1060": 100814946304.0,
+ "1065": 100814946304.0,
+ "1070": 100814946304.0,
+ "1075": 100814946304.0,
+ "1080": 100814946304.0,
+ "1085": 100814946304.0,
+ "1090": 100814946304.0,
+ "1095": 100814946304.0,
+ "1100": 100814946304.0,
+ "1105": 100814946304.0,
+ "1110": 100814946304.0,
+ "1115": 100814946304.0,
+ "1120": 100814946304.0,
+ "1125": 100814946304.0,
+ "1130": 100814946304.0,
+ "1135": 100814946304.0,
+ "1140": 100814946304.0,
+ "1145": 100814946304.0,
+ "1150": 100814946304.0,
+ "1155": 100814946304.0,
+ "1160": 100814946304.0,
+ "1165": 100814946304.0,
+ "1170": 100814946304.0,
+ "1175": 100814946304.0,
+ "1180": 100814946304.0,
+ "1185": 100814946304.0,
+ "1190": 100814946304.0,
+ "1195": 100814946304.0,
+ "1200": 100814946304.0,
+ "1205": 100814946304.0,
+ "1210": 100814946304.0,
+ "1215": 100814946304.0,
+ "1220": 100814946304.0,
+ "1225": 100814946304.0,
+ "1230": 100814946304.0,
+ "1235": 100814946304.0,
+ "1240": 100814946304.0,
+ "1245": 100814946304.0,
+ "1250": 100814946304.0,
+ "1255": 100814946304.0,
+ "1260": 100814946304.0,
+ "1265": 100814946304.0,
+ "1270": 100814946304.0,
+ "1275": 100814946304.0,
+ "1280": 100814946304.0,
+ "1285": 100814946304.0,
+ "1290": 100814946304.0,
+ "1295": 100814946304.0,
+ "1300": 100788985856.0,
+ "1305": 100788985856.0,
+ "1310": 100788985856.0,
+ "1315": 100788985856.0,
+ "1320": 100788985856.0,
+ "1325": 100788985856.0,
+ "1330": 100788985856.0,
+ "1335": 100788985856.0,
+ "1340": 100788985856.0,
+ "1345": 100788985856.0,
+ "1350": 100788985856.0,
+ "1355": 100788985856.0,
+ "1360": 100788985856.0,
+ "1365": 100788985856.0,
+ "1370": 100788985856.0,
+ "1375": 100788985856.0,
+ "1380": 100788985856.0,
+ "1385": 100788985856.0,
+ "1390": 100788985856.0,
+ "1395": 100788985856.0,
+ "1400": 100788985856.0,
+ "1405": 100788985856.0,
+ "1410": 100788985856.0,
+ "1415": 100788985856.0,
+ "1420": 100788985856.0,
+ "1425": 100788985856.0,
+ "1430": 100788985856.0,
+ "1435": 100788985856.0,
+ "1440": 100788985856.0,
+ "1445": 100788985856.0,
+ "1450": 100788985856.0,
+ "1455": 100788985856.0,
+ "1460": 100788985856.0,
+ "1465": 100788985856.0,
+ "1470": 100788985856.0,
+ "1475": 100788985856.0,
+ "1480": 100788985856.0,
+ "1485": 100788985856.0,
+ "1490": 100788985856.0,
+ "1495": 100788985856.0,
+ "1500": 100788985856.0,
+ "1505": 100788985856.0,
+ "1510": 100788985856.0,
+ "1515": 100788985856.0,
+ "1520": 100788985856.0,
+ "1525": 100788985856.0,
+ "1530": 100788985856.0,
+ "1535": 100788985856.0,
+ "1540": 100788985856.0,
+ "1545": 100788985856.0,
+ "1550": 100788985856.0,
+ "1555": 100788985856.0,
+ "1560": 100788985856.0,
+ "1565": 100788985856.0,
+ "1570": 100788985856.0,
+ "1575": 100788985856.0,
+ "1580": 100788985856.0,
+ "1585": 100788985856.0,
+ "1590": 100788985856.0,
+ "1595": 100788985856.0,
+ "1600": 100788985856.0,
+ "1605": 100788985856.0,
+ "1610": 100788985856.0,
+ "1615": 100788985856.0,
+ "1620": 100788985856.0,
+ "1625": 100788985856.0,
+ "1630": 100788985856.0,
+ "1635": 100788985856.0,
+ "1640": 100788985856.0,
+ "1645": 100788985856.0,
+ "1650": 100788985856.0,
+ "1655": 100788985856.0,
+ "1660": 99383525376.0,
+ "1665": 99634765824.0,
+ "1670": 99634765824.0,
+ "1675": 99634765824.0,
+ "1680": 99634765824.0,
+ "1685": 99634765824.0,
+ "1690": 99634765824.0,
+ "1695": 99634765824.0,
+ "1700": 99634765824.0,
+ "1705": 99789824000.0,
+ "1710": 99789824000.0,
+ "1715": 99789824000.0,
+ "1720": 99789824000.0,
+ "1725": 99789824000.0,
+ "1730": 99789824000.0,
+ "1735": 99789824000.0,
+ "1740": 99789824000.0,
+ "1745": 99789824000.0,
+ "1750": 99789824000.0,
+ "1755": 99789824000.0,
+ "1760": 99789824000.0,
+ "1765": 99789824000.0,
+ "1770": 99789824000.0,
+ "1775": 99833782272.0,
+ "1780": 99833782272.0,
+ "1785": 99833782272.0,
+ "1790": 99927007232.0,
+ "1795": 99927007232.0,
+ "1800": 99927007232.0,
+ "1805": 99927007232.0,
+ "1810": 99927007232.0,
+ "1815": 99927007232.0,
+ "1820": 99927007232.0,
+ "1825": 99927007232.0,
+ "1830": 99927007232.0,
+ "1835": 99927007232.0,
+ "1840": 99927007232.0,
+ "1845": 99927007232.0,
+ "1850": 99927007232.0,
+ "1855": 99927007232.0,
+ "1860": 99927007232.0,
+ "1865": 99927007232.0,
+ "1870": 99927007232.0,
+ "1875": 99927007232.0,
+ "1880": 99927007232.0,
+ "1885": 100007444480.0,
+ "1890": 100150525952.0,
+ "1895": 100150525952.0,
+ "1900": 100150525952.0,
+ "1905": 100150525952.0,
+ "1910": 100315799552.0,
+ "1915": 100315799552.0,
+ "1920": 100315799552.0,
+ "1925": 100315799552.0,
+ "1930": 100315799552.0,
+ "1935": 100315799552.0,
+ "1940": 100315799552.0,
+ "1945": 100315799552.0,
+ "1950": 100315799552.0,
+ "1955": 100315799552.0,
+ "1960": 100315799552.0,
+ "1965": 100315799552.0,
+ "1970": 100315799552.0,
+ "1975": 100315799552.0,
+ "1980": 100315799552.0,
+ "1985": 100315799552.0,
+ "1990": 100315799552.0,
+ "1995": 100315799552.0,
+ "2000": 100315799552.0,
+ "2005": 100315799552.0,
+ "2010": 100315799552.0,
+ "2015": 100315799552.0,
+ "2020": 100315799552.0,
+ "2025": 100315799552.0,
+ "2030": 100315799552.0,
+ "2035": 100315799552.0,
+ "2040": 100315799552.0,
+ "2045": 100315799552.0,
+ "2050": 100315799552.0,
+ "2055": 100315799552.0,
+ "2060": 100315799552.0,
+ "2065": 100315799552.0,
+ "2070": 100315799552.0,
+ "2075": 100315799552.0,
+ "2080": 100315799552.0,
+ "2085": 100315799552.0,
+ "2090": 100315799552.0,
+ "2095": 100315799552.0,
+ "2100": 100315799552.0,
+ "2105": 100315799552.0,
+ "2110": 100315799552.0,
+ "2115": 100315799552.0,
+ "2120": 100315799552.0,
+ "2125": 100315799552.0,
+ "2130": 100315799552.0,
+ "2135": 100315799552.0,
+ "2140": 100315799552.0,
+ "2145": 100315799552.0,
+ "2150": 100315799552.0,
+ "2155": 100315799552.0,
+ "2160": 100315799552.0,
+ "2165": 100315799552.0,
+ "2170": 100315799552.0,
+ "2175": 100315799552.0,
+ "2180": 100315799552.0,
+ "2185": 100315799552.0,
+ "2190": 100315799552.0,
+ "2195": 100315799552.0,
+ "2200": 100315799552.0,
+ "2205": 100315799552.0,
+ "2210": 100315799552.0,
+ "2215": 100315799552.0,
+ "2220": 100315799552.0,
+ "2225": 100315799552.0,
+ "2230": 100315799552.0,
+ "2235": 100315799552.0,
+ "2240": 100315799552.0,
+ "2245": 100315799552.0,
+ "2250": 100315799552.0,
+ "2255": 100315799552.0,
+ "2260": 100315799552.0,
+ "2265": 100315799552.0,
+ "2270": 100315799552.0,
+ "2275": 100315799552.0,
+ "2280": 100315799552.0,
+ "2285": 100315799552.0,
+ "2290": 100315799552.0,
+ "2295": 100315799552.0,
+ "2300": 100315799552.0,
+ "2305": 100315799552.0,
+ "2310": 100315799552.0,
+ "2315": 100315799552.0,
+ "2320": 100315799552.0,
+ "2325": 100315799552.0,
+ "2330": 100315799552.0,
+ "2335": 100315799552.0,
+ "2340": 100315799552.0,
+ "2345": 100315799552.0,
+ "2350": 100315799552.0,
+ "2355": 100315799552.0,
+ "2360": 100315799552.0,
+ "2365": 100315799552.0,
+ "2370": 100315799552.0,
+ "2375": 100315799552.0,
+ "2380": 100315799552.0,
+ "2385": 100315799552.0,
+ "2390": 100315799552.0,
+ "2395": 100315799552.0,
+ "2400": 100315799552.0,
+ "2405": 100315799552.0,
+ "2410": 100315799552.0,
+ "2415": 100315799552.0,
+ "2420": 100315799552.0,
+ "2425": 100315799552.0,
+ "2430": 100315799552.0,
+ "2435": 100315799552.0,
+ "2440": 100315799552.0,
+ "2445": 100315799552.0,
+ "2450": 100315799552.0,
+ "2455": 100315799552.0,
+ "2460": 100315799552.0,
+ "2465": 100315799552.0,
+ "2470": 100315799552.0,
+ "2475": 100315799552.0,
+ "2480": 100315799552.0,
+ "2485": 100315799552.0,
+ "2490": 100315799552.0,
+ "2495": 100315799552.0,
+ "2500": 100315799552.0,
+ "2505": 100315799552.0,
+ "2510": 100315799552.0,
+ "2515": 100315799552.0,
+ "2520": 100315799552.0,
+ "2525": 100315799552.0,
+ "2530": 100315799552.0,
+ "2535": 100315799552.0,
+ "2540": 100315799552.0,
+ "2545": 100315799552.0,
+ "2550": 100315799552.0,
+ "2555": 100315799552.0,
+ "2560": 100315799552.0,
+ "2565": 100315799552.0,
+ "2570": 100315799552.0,
+ "2575": 100315799552.0,
+ "2580": 100315799552.0,
+ "2585": 100315799552.0,
+ "2590": 100315799552.0,
+ "2595": 100315799552.0,
+ "2600": 100315799552.0,
+ "2605": 100315799552.0,
+ "2610": 100315799552.0,
+ "2615": 100315799552.0,
+ "2620": 100315799552.0,
+ "2625": 100315799552.0,
+ "2630": 100315799552.0,
+ "2635": 100315799552.0,
+ "2640": 100315799552.0,
+ "2645": 100315799552.0,
+ "2650": 100315799552.0,
+ "2655": 100315799552.0,
+ "2660": 100315799552.0,
+ "2665": 100315799552.0,
+ "2670": 100315799552.0,
+ "2675": 100315799552.0,
+ "2680": 100315799552.0,
+ "2685": 100315799552.0,
+ "2690": 100315799552.0,
+ "2695": 100315799552.0,
+ "2700": 100315799552.0,
+ "2705": 100315799552.0,
+ "2710": 100315799552.0,
+ "2715": 100315799552.0,
+ "2720": 100315799552.0,
+ "2725": 100315799552.0,
+ "2730": 100315799552.0,
+ "2735": 100315799552.0,
+ "2740": 100315799552.0,
+ "2745": 100315799552.0,
+ "2750": 100315799552.0,
+ "2755": 100315799552.0,
+ "2760": 100315799552.0,
+ "2765": 100315799552.0,
+ "2770": 100315799552.0,
+ "2775": 100315799552.0,
+ "2780": 100315799552.0,
+ "2785": 100315799552.0,
+ "2790": 100315799552.0,
+ "2795": 100315799552.0,
+ "2800": 100315799552.0,
+ "2805": 100315799552.0,
+ "2810": 100315799552.0,
+ "2815": 100315799552.0,
+ "2820": 100315799552.0,
+ "2825": 100315799552.0,
+ "2830": 100315799552.0,
+ "2835": 100315799552.0,
+ "2840": 100315799552.0,
+ "2845": 100315799552.0,
+ "2850": 100315799552.0,
+ "2855": 100315799552.0,
+ "2860": 100315799552.0,
+ "2865": 100315799552.0,
+ "2870": 100315799552.0,
+ "2875": 100315799552.0,
+ "2880": 100315799552.0,
+ "2885": 100315799552.0,
+ "2890": 100315799552.0,
+ "2895": 100315799552.0,
+ "2900": 100315799552.0,
+ "2905": 100315799552.0,
+ "2910": 100315799552.0,
+ "2915": 100315799552.0,
+ "2920": 100315799552.0,
+ "2925": 100315799552.0,
+ "2930": 100315799552.0,
+ "2935": 100315799552.0,
+ "2940": 100315799552.0,
+ "2945": 100315799552.0,
+ "2950": 100315799552.0,
+ "2955": 100315799552.0,
+ "2960": 100315799552.0,
+ "2965": 100315799552.0,
+ "2970": 100315799552.0,
+ "2975": 100315799552.0,
+ "2980": 100315799552.0,
+ "2985": 100315799552.0,
+ "2990": 100315799552.0,
+ "2995": 100315799552.0,
+ "3000": 100315799552.0,
+ "3005": 100315799552.0,
+ "3010": 100315799552.0,
+ "3015": 100315799552.0,
+ "3020": 100315799552.0,
+ "3025": 100315799552.0,
+ "3030": 100315799552.0,
+ "3035": 100315799552.0,
+ "3040": 100315799552.0,
+ "3045": 100315799552.0,
+ "3050": 100315799552.0,
+ "3055": 100315799552.0,
+ "3060": 100315799552.0,
+ "3065": 100315799552.0,
+ "3070": 100315799552.0,
+ "3075": 100315799552.0,
+ "3080": 100315799552.0,
+ "3085": 100315799552.0,
+ "3090": 100315799552.0,
+ "3095": 100315799552.0,
+ "3100": 100315799552.0,
+ "3105": 100315799552.0,
+ "3110": 100315799552.0,
+ "3115": 100315799552.0,
+ "3120": 100315799552.0,
+ "3125": 100315799552.0,
+ "3130": 100315799552.0,
+ "3135": 100315799552.0,
+ "3140": 100315799552.0,
+ "3145": 100315799552.0,
+ "3150": 100315799552.0,
+ "3155": 100315799552.0,
+ "3160": 100315799552.0,
+ "3165": 100315799552.0,
+ "3170": 100315799552.0,
+ "3175": 100315799552.0,
+ "3180": 99491127296.0,
+ "3185": 99491127296.0,
+ "3190": 99491127296.0,
+ "3195": 99491127296.0,
+ "3200": 99491127296.0,
+ "3205": 99491127296.0,
+ "3210": 99491127296.0,
+ "3215": 99491127296.0,
+ "3220": 99491127296.0,
+ "3225": 99491127296.0,
+ "3230": 99491127296.0,
+ "3235": 99491127296.0,
+ "3240": 99491127296.0,
+ "3245": 99627114496.0,
+ "3250": 99627114496.0,
+ "3255": 99627114496.0,
+ "3260": 99627114496.0,
+ "3265": 99627114496.0,
+ "3270": 99627114496.0,
+ "3275": 99627114496.0,
+ "3280": 99627114496.0,
+ "3285": 99627114496.0,
+ "3290": 99627114496.0,
+ "3295": 99640221696.0,
+ "3300": 99640221696.0,
+ "3305": 99640221696.0,
+ "3310": 99640221696.0,
+ "3315": 99640221696.0,
+ "3320": 99640221696.0,
+ "3325": 99640221696.0,
+ "3330": 99640221696.0,
+ "3335": 99640221696.0,
+ "3340": 99814555648.0,
+ "3345": 99865575424.0,
+ "3350": 99865575424.0,
+ "3355": 99865575424.0,
+ "3360": 99865575424.0,
+ "3365": 99865575424.0,
+ "3370": 99865575424.0,
+ "3375": 99865575424.0,
+ "3380": 99865575424.0,
+ "3385": 99865575424.0,
+ "3390": 99865575424.0,
+ "3395": 99865575424.0,
+ "3400": 99865575424.0,
+ "3405": 99865575424.0,
+ "3410": 99865575424.0,
+ "3415": 99865575424.0,
+ "3420": 99865575424.0,
+ "3425": 99865575424.0,
+ "3430": 99865575424.0,
+ "3435": 99865575424.0,
+ "3440": 99865575424.0,
+ "3445": 99865575424.0,
+ "3450": 99865575424.0,
+ "3455": 99865575424.0,
+ "3460": 99865575424.0,
+ "3465": 99865575424.0,
+ "3470": 99865575424.0,
+ "3475": 99865575424.0,
+ "3480": 99865575424.0,
+ "3485": 99865575424.0,
+ "3490": 99865575424.0,
+ "3495": 99865575424.0,
+ "3500": 99865575424.0,
+ "3505": 100085440512.0,
+ "3510": 100085440512.0,
+ "3515": 100085440512.0,
+ "3520": 100085440512.0,
+ "3525": 100085440512.0,
+ "3530": 100060995584.0,
+ "3535": 100060995584.0,
+ "3540": 100060995584.0,
+ "3545": 100060995584.0,
+ "3550": 100060995584.0,
+ "3555": 100060995584.0,
+ "3560": 100060995584.0,
+ "3565": 100060995584.0,
+ "3570": 100060995584.0,
+ "3575": 100060995584.0,
+ "3580": 100060995584.0,
+ "3585": 100060995584.0,
+ "3590": 100060995584.0,
+ "3595": 100060995584.0,
+ "3600": 100060995584.0,
+ "3605": 100060995584.0,
+ "3610": 100060995584.0,
+ "3615": 100060995584.0,
+ "3620": 100060995584.0,
+ "3625": 100060995584.0,
+ "3630": 100060995584.0,
+ "3635": 100060995584.0,
+ "3640": 100060995584.0,
+ "3645": 100060995584.0,
+ "3650": 100060995584.0,
+ "3655": 100060995584.0,
+ "3660": 100060995584.0,
+ "3665": 100060995584.0,
+ "3670": 100060995584.0,
+ "3675": 100060995584.0,
+ "3680": 100060995584.0,
+ "3685": 100060995584.0,
+ "3690": 100060995584.0,
+ "3695": 100060995584.0,
+ "3700": 100060995584.0,
+ "3705": 100060995584.0,
+ "3710": 100060995584.0,
+ "3715": 100060995584.0,
+ "3720": 100060995584.0,
+ "3725": 100060995584.0,
+ "3730": 100060995584.0,
+ "3735": 100060995584.0,
+ "3740": 100060995584.0,
+ "3745": 100060995584.0,
+ "3750": 100060995584.0,
+ "3755": 100060995584.0,
+ "3760": 100060995584.0,
+ "3765": 100060995584.0,
+ "3770": 100060995584.0,
+ "3775": 100060995584.0,
+ "3780": 100060995584.0,
+ "3785": 100060995584.0,
+ "3790": 100060995584.0,
+ "3795": 100060995584.0,
+ "3800": 100060995584.0,
+ "3805": 100060995584.0,
+ "3810": 100060995584.0,
+ "3815": 100060995584.0,
+ "3820": 100060995584.0,
+ "3825": 100060995584.0,
+ "3830": 100060995584.0,
+ "3835": 100060995584.0,
+ "3840": 100060995584.0,
+ "3845": 100060995584.0,
+ "3850": 100060995584.0,
+ "3855": 100060995584.0,
+ "3860": 100060995584.0,
+ "3865": 100060995584.0,
+ "3870": 100060995584.0,
+ "3875": 100060995584.0,
+ "3880": 100060995584.0,
+ "3885": 100060995584.0,
+ "3890": 100246036480.0,
+ "3895": 100246036480.0,
+ "3900": 100246036480.0,
+ "3905": 100246036480.0,
+ "3910": 100246036480.0,
+ "3915": 100246036480.0,
+ "3920": 100246036480.0,
+ "3925": 100246036480.0,
+ "3930": 100246036480.0,
+ "3935": 100246036480.0,
+ "3940": 100246036480.0,
+ "3945": 100246036480.0,
+ "3950": 100246036480.0,
+ "3955": 100246036480.0,
+ "3960": 100246036480.0,
+ "3965": 100246036480.0,
+ "3970": 100246036480.0,
+ "3975": 100246036480.0,
+ "3980": 100246036480.0,
+ "3985": 100246036480.0,
+ "3990": 100246036480.0,
+ "3995": 100304855040.0,
+ "4000": 100304855040.0,
+ "4005": 100304855040.0,
+ "4010": 100304855040.0,
+ "4015": 100304855040.0,
+ "4020": 100304855040.0,
+ "4025": 100304855040.0,
+ "4030": 100304855040.0,
+ "4035": 100304855040.0,
+ "4040": 100304855040.0,
+ "4045": 100304855040.0,
+ "4050": 100304855040.0,
+ "4055": 100304855040.0,
+ "4060": 100304855040.0,
+ "4065": 100304855040.0,
+ "4070": 100304855040.0,
+ "4075": 100304855040.0,
+ "4080": 100304855040.0,
+ "4085": 100304855040.0,
+ "4090": 100304855040.0,
+ "4095": 100304855040.0,
+ "4100": 100304855040.0,
+ "4105": 100304855040.0,
+ "4110": 100304855040.0,
+ "4115": 100304855040.0,
+ "4120": 100304855040.0,
+ "4125": 100304855040.0,
+ "4130": 100304855040.0,
+ "4135": 100304855040.0,
+ "4140": 100304855040.0,
+ "4145": 100304855040.0,
+ "4150": 100304855040.0,
+ "4155": 100304855040.0,
+ "4160": 100304855040.0,
+ "4165": 100304855040.0,
+ "4170": 100304855040.0,
+ "4175": 100304855040.0,
+ "4180": 100304855040.0,
+ "4185": 100304855040.0,
+ "4190": 100304855040.0,
+ "4195": 100304855040.0,
+ "4200": 100304855040.0,
+ "4205": 100304855040.0,
+ "4210": 100304855040.0,
+ "4215": 100304855040.0,
+ "4220": 100304855040.0,
+ "4225": 100304855040.0,
+ "4230": 100304855040.0,
+ "4235": 100304855040.0,
+ "4240": 100304855040.0,
+ "4245": 100304855040.0,
+ "4250": 100304855040.0,
+ "4255": 100304855040.0,
+ "4260": 100304855040.0,
+ "4265": 100304855040.0,
+ "4270": 100304855040.0,
+ "4275": 100304855040.0,
+ "4280": 100304855040.0,
+ "4285": 100304855040.0,
+ "4290": 100304855040.0,
+ "4295": 100304855040.0,
+ "4300": 100304855040.0,
+ "4305": 100304855040.0,
+ "4310": 100304855040.0,
+ "4315": 100304855040.0,
+ "4320": 100304855040.0,
+ "4325": 100304855040.0,
+ "4330": 100304855040.0,
+ "4335": 100304855040.0,
+ "4340": 100304855040.0,
+ "4345": 100304855040.0,
+ "4350": 100304855040.0,
+ "4355": 100304855040.0,
+ "4360": 100304855040.0,
+ "4365": 100304855040.0,
+ "4370": 100304855040.0,
+ "4375": 100304855040.0,
+ "4380": 100304855040.0,
+ "4385": 100304855040.0,
+ "4390": 100304855040.0,
+ "4395": 100304855040.0,
+ "4400": 100304855040.0,
+ "4405": 100304855040.0,
+ "4410": 100304855040.0,
+ "4415": 100304855040.0,
+ "4420": 100304855040.0,
+ "4425": 100304855040.0,
+ "4430": 100304855040.0,
+ "4435": 100304855040.0,
+ "4440": 100304855040.0,
+ "4445": 100304855040.0,
+ "4450": 100304855040.0,
+ "4455": 100304855040.0,
+ "4460": 100304855040.0,
+ "4465": 100304855040.0,
+ "4470": 100304855040.0,
+ "4475": 100304855040.0,
+ "4480": 100304855040.0,
+ "4485": 100304855040.0,
+ "4490": 100304855040.0,
+ "4495": 100304855040.0,
+ "4500": 100304855040.0,
+ "4505": 100304855040.0,
+ "4510": 100304855040.0,
+ "4515": 100304855040.0,
+ "4520": 100304855040.0,
+ "4525": 100304855040.0,
+ "4530": 100304855040.0,
+ "4535": 100304855040.0,
+ "4540": 100304855040.0,
+ "4545": 100304855040.0,
+ "4550": 100304855040.0,
+ "4555": 100304855040.0,
+ "4560": 100304855040.0,
+ "4565": 100304855040.0,
+ "4570": 100304855040.0,
+ "4575": 100304855040.0,
+ "4580": 100304855040.0,
+ "4585": 100304855040.0,
+ "4590": 100304855040.0,
+ "4595": 100304855040.0,
+ "4600": 100304855040.0,
+ "4605": 100304855040.0,
+ "4610": 100304855040.0,
+ "4615": 100304855040.0,
+ "4620": 100304855040.0,
+ "4625": 100304855040.0,
+ "4630": 100304855040.0,
+ "4635": 100304855040.0,
+ "4640": 100304855040.0,
+ "4645": 100304855040.0,
+ "4650": 100304855040.0,
+ "4655": 100304855040.0,
+ "4660": 100304855040.0,
+ "4665": 100304855040.0,
+ "4670": 100304855040.0,
+ "4675": 100304855040.0,
+ "4680": 100304855040.0,
+ "4685": 100304855040.0,
+ "4690": 100304855040.0,
+ "4695": 100304855040.0,
+ "4700": 100304855040.0,
+ "4705": 100304855040.0,
+ "4710": 100304855040.0,
+ "4715": 100304855040.0,
+ "4720": 100304855040.0,
+ "4725": 100304855040.0,
+ "4730": 100304855040.0,
+ "4735": 100304855040.0,
+ "4740": 100304855040.0,
+ "4745": 100304855040.0,
+ "4750": 100304855040.0,
+ "4755": 100304855040.0,
+ "4760": 100304855040.0,
+ "4765": 100304855040.0,
+ "4770": 100304855040.0,
+ "4775": 100304855040.0,
+ "4780": 100304855040.0,
+ "4785": 100304855040.0,
+ "4790": 100304855040.0,
+ "4795": 100304855040.0,
+ "4800": 100304855040.0,
+ "4805": 100304855040.0,
+ "4810": 100304855040.0,
+ "4815": 100304855040.0,
+ "4820": 100304855040.0,
+ "4825": 100304855040.0,
+ "4830": 100304855040.0,
+ "4835": 100304855040.0,
+ "4840": 100304855040.0,
+ "4845": 100304855040.0,
+ "4850": 100304855040.0,
+ "4855": 100304855040.0,
+ "4860": 100394885120.0,
+ "4865": 100394885120.0,
+ "4870": 100394885120.0,
+ "4875": 100394885120.0,
+ "4880": 100394885120.0,
+ "4885": 100394885120.0,
+ "4890": 100394885120.0,
+ "4895": 98654183424.0,
+ "4900": 99573145600.0,
+ "4905": 99573145600.0,
+ "4910": 99573145600.0,
+ "4915": 99573145600.0,
+ "4920": 99573145600.0,
+ "4925": 99573145600.0,
+ "4930": 99573145600.0,
+ "4935": 99573145600.0,
+ "4940": 99573145600.0,
+ "4945": 99573145600.0,
+ "4950": 99573145600.0,
+ "4955": 99573145600.0,
+ "4960": 99573145600.0,
+ "4965": 99573145600.0,
+ "4970": 99573145600.0,
+ "4975": 99596902400.0,
+ "4980": 99596902400.0,
+ "4985": 99596902400.0,
+ "4990": 99596902400.0,
+ "4995": 99596902400.0,
+ "5000": 99596902400.0,
+ "5005": 99596902400.0,
+ "5010": 99596902400.0,
+ "5015": 99596902400.0,
+ "5020": 100471119872.0,
+ "5025": 100471119872.0,
+ "5030": 100471119872.0,
+ "5035": 100471119872.0,
+ "5040": 100471119872.0,
+ "5045": 100471119872.0,
+ "5050": 100471119872.0,
+ "5055": 100471119872.0,
+ "5060": 100471119872.0,
+ "5065": 100471119872.0,
+ "5070": 100471119872.0,
+ "5075": 100471119872.0,
+ "5080": 100471119872.0,
+ "5085": 100471119872.0,
+ "5090": 100471119872.0,
+ "5095": 100471119872.0,
+ "5100": 100471119872.0,
+ "5105": 100471119872.0,
+ "5110": 100471119872.0,
+ "5115": 100471119872.0,
+ "5120": 100471119872.0,
+ "5125": 100471119872.0,
+ "5130": 100471119872.0,
+ "5135": 100471119872.0,
+ "5140": 100471119872.0,
+ "5145": 100471119872.0,
+ "5150": 100471119872.0,
+ "5155": 100471119872.0,
+ "5160": 100471119872.0,
+ "5165": 100471119872.0,
+ "5170": 100471119872.0,
+ "5175": 100471119872.0,
+ "5180": 100471119872.0,
+ "5185": 100471119872.0,
+ "5190": 100471119872.0,
+ "5195": 100471119872.0,
+ "5200": 100471119872.0,
+ "5205": 100471119872.0,
+ "5210": 100471119872.0,
+ "5215": 100471119872.0,
+ "5220": 100471119872.0,
+ "5225": 100471119872.0,
+ "5230": 100471119872.0,
+ "5235": 100471119872.0,
+ "5240": 100471119872.0,
+ "5245": 100471119872.0,
+ "5250": 100471119872.0,
+ "5255": 100471119872.0,
+ "5260": 100471119872.0,
+ "5265": 100471119872.0,
+ "5270": 100471119872.0,
+ "5275": 100471119872.0,
+ "5280": 100471119872.0,
+ "5285": 100471119872.0,
+ "5290": 100471119872.0,
+ "5295": 100471119872.0,
+ "5300": 100471119872.0,
+ "5305": 100471119872.0,
+ "5310": 100471119872.0,
+ "5315": 100471119872.0,
+ "5320": 100471119872.0,
+ "5325": 100471119872.0,
+ "5330": 100471119872.0,
+ "5335": 100471119872.0,
+ "5340": 100471119872.0,
+ "5345": 100471119872.0,
+ "5350": 100471119872.0,
+ "5355": 100471119872.0,
+ "5360": 100471119872.0,
+ "5365": 100471119872.0,
+ "5370": 100471119872.0,
+ "5375": 100471119872.0,
+ "5380": 100471119872.0,
+ "5385": 100471119872.0,
+ "5390": 100471119872.0,
+ "5395": 100471119872.0,
+ "5400": 100471119872.0,
+ "5405": 100471119872.0,
+ "5410": 100471119872.0,
+ "5415": 100471119872.0,
+ "5420": 100471119872.0,
+ "5425": 100471119872.0,
+ "5430": 100471119872.0,
+ "5435": 100471119872.0,
+ "5440": 100471119872.0,
+ "5445": 99961618432.0,
+ "5450": 99961618432.0,
+ "5455": 99961618432.0,
+ "5460": 99961618432.0,
+ "5465": 99961618432.0,
+ "5470": 99961618432.0,
+ "5475": 99961618432.0,
+ "5480": 99961618432.0,
+ "5485": 99961618432.0,
+ "5490": 99961618432.0,
+ "5495": 99961618432.0,
+ "5500": 99961618432.0,
+ "5505": 99961618432.0,
+ "5510": 99961618432.0,
+ "5515": 99961618432.0,
+ "5520": 99961618432.0,
+ "5525": 99961618432.0,
+ "5530": 99961618432.0,
+ "5535": 99961618432.0,
+ "5540": 99961618432.0,
+ "5545": 99961618432.0,
+ "5550": 99961618432.0,
+ "5555": 99961618432.0,
+ "5560": 99961618432.0,
+ "5565": 99961618432.0,
+ "5570": 99961618432.0,
+ "5575": 99961618432.0,
+ "5580": 99961618432.0,
+ "5585": 99961618432.0,
+ "5590": 99961618432.0,
+ "5595": 99961618432.0,
+ "5600": 99961618432.0,
+ "5605": 99961618432.0,
+ "5610": 99961618432.0,
+ "5615": 99961618432.0,
+ "5620": 100030078976.0,
+ "5625": 100030078976.0,
+ "5630": 100030078976.0,
+ "5635": 100030078976.0,
+ "5640": 100030078976.0,
+ "5645": 100030078976.0,
+ "5650": 100030078976.0,
+ "5655": 100030078976.0,
+ "5660": 100030078976.0,
+ "5665": 100030078976.0,
+ "5670": 100030078976.0,
+ "5675": 100030078976.0,
+ "5680": 100030078976.0,
+ "5685": 100030078976.0,
+ "5690": 100030078976.0,
+ "5695": 100030078976.0,
+ "5700": 100030078976.0,
+ "5705": 100030078976.0,
+ "5710": 100030078976.0,
+ "5715": 100030078976.0,
+ "5720": 100030078976.0,
+ "5725": 100030078976.0,
+ "5730": 100030078976.0,
+ "5735": 100030078976.0,
+ "5740": 100030078976.0,
+ "5745": 100030078976.0,
+ "5750": 100030078976.0,
+ "5755": 100030078976.0,
+ "5760": 100030078976.0,
+ "5765": 100030078976.0,
+ "5770": 100030078976.0,
+ "5775": 100030078976.0,
+ "5780": 100030078976.0,
+ "5785": 100030078976.0,
+ "5790": 100030078976.0,
+ "5795": 100030078976.0,
+ "5800": 100030078976.0,
+ "5805": 100030078976.0,
+ "5810": 100030078976.0,
+ "5815": 100030078976.0,
+ "5820": 100030078976.0,
+ "5825": 100030078976.0,
+ "5830": 100030078976.0,
+ "5835": 100030078976.0,
+ "5840": 100030078976.0,
+ "5845": 100030078976.0,
+ "5850": 100030078976.0,
+ "5855": 100030078976.0,
+ "5860": 100030078976.0,
+ "5865": 100030078976.0,
+ "5870": 100030078976.0,
+ "5875": 100030078976.0,
+ "5880": 100030078976.0,
+ "5885": 100030078976.0,
+ "5890": 100030078976.0,
+ "5895": 100030078976.0,
+ "5900": 100030078976.0,
+ "5905": 100030078976.0,
+ "5910": 100030078976.0,
+ "5915": 100030078976.0,
+ "5920": 100030078976.0,
+ "5925": 100030078976.0,
+ "5930": 100030078976.0,
+ "5935": 100030078976.0,
+ "5940": 100030078976.0,
+ "5945": 100030078976.0,
+ "5950": 100030078976.0,
+ "5955": 100030078976.0,
+ "5960": 100030078976.0,
+ "5965": 100030078976.0,
+ "5970": 100030078976.0,
+ "5975": 100030078976.0,
+ "5980": 100116144128.0,
+ "5985": 100116144128.0,
+ "5990": 100116144128.0,
+ "5995": 100116144128.0,
+ "6000": 100116144128.0,
+ "6005": 100116144128.0,
+ "6010": 100116144128.0,
+ "6015": 100116144128.0,
+ "6020": 100116144128.0,
+ "6025": 100116144128.0,
+ "6030": 100116144128.0,
+ "6035": 100116144128.0,
+ "6040": 100116144128.0,
+ "6045": 100116144128.0,
+ "6050": 100116144128.0,
+ "6055": 100116144128.0,
+ "6060": 100116144128.0,
+ "6065": 100116144128.0,
+ "6070": 100116144128.0,
+ "6075": 100116144128.0,
+ "6080": 100116144128.0,
+ "6085": 100116144128.0,
+ "6090": 100116144128.0,
+ "6095": 100116144128.0,
+ "6100": 100116144128.0,
+ "6105": 100116144128.0,
+ "6110": 100116144128.0,
+ "6115": 100116144128.0,
+ "6120": 100116144128.0,
+ "6125": 100116144128.0,
+ "6130": 100116144128.0,
+ "6135": 100116144128.0,
+ "6140": 100116144128.0,
+ "6145": 100116144128.0,
+ "6150": 100116144128.0,
+ "6155": 100116144128.0,
+ "6160": 100116144128.0,
+ "6165": 100116144128.0,
+ "6170": 100116144128.0,
+ "6175": 100116144128.0,
+ "6180": 100116144128.0,
+ "6185": 100116144128.0,
+ "6190": 100116144128.0,
+ "6195": 100116144128.0,
+ "6200": 100116144128.0,
+ "6205": 100116144128.0,
+ "6210": 100116144128.0,
+ "6215": 100116144128.0,
+ "6220": 100116144128.0,
+ "6225": 100116144128.0,
+ "6230": 100116144128.0,
+ "6235": 100116144128.0,
+ "6240": 100116144128.0,
+ "6245": 100116144128.0,
+ "6250": 100116144128.0,
+ "6255": 100116144128.0,
+ "6260": 100116144128.0,
+ "6265": 100116144128.0,
+ "6270": 100116144128.0,
+ "6275": 100116144128.0,
+ "6280": 100116144128.0,
+ "6285": 100116144128.0,
+ "6290": 100116144128.0,
+ "6295": 100116144128.0,
+ "6300": 100116144128.0,
+ "6305": 100116144128.0,
+ "6310": 100116144128.0,
+ "6315": 100116144128.0,
+ "6320": 100116144128.0,
+ "6325": 100116144128.0,
+ "6330": 100116144128.0,
+ "6335": 100116144128.0,
+ "6340": 100116144128.0,
+ "6345": 100116144128.0,
+ "6350": 100116144128.0,
+ "6355": 100116144128.0,
+ "6360": 100116144128.0,
+ "6365": 100116144128.0,
+ "6370": 100116144128.0,
+ "6375": 100116144128.0,
+ "6380": 100116144128.0,
+ "6385": 100116144128.0,
+ "6390": 100116144128.0,
+ "6395": 100161732608.0,
+ "6400": 100161732608.0,
+ "6405": 100161732608.0,
+ "6410": 100161732608.0,
+ "6415": 100161732608.0,
+ "6420": 100161732608.0,
+ "6425": 100161732608.0,
+ "6430": 100161732608.0,
+ "6435": 100161732608.0,
+ "6440": 100161732608.0,
+ "6445": 100161732608.0,
+ "6450": 100161732608.0,
+ "6455": 100161732608.0,
+ "6460": 100161732608.0,
+ "6465": 100161732608.0,
+ "6470": 100161732608.0,
+ "6475": 100161732608.0,
+ "6480": 100161732608.0,
+ "6485": 100161732608.0,
+ "6490": 100161732608.0,
+ "6495": 100161732608.0,
+ "6500": 100161732608.0,
+ "6505": 100161732608.0,
+ "6510": 100161732608.0,
+ "6515": 100161732608.0,
+ "6520": 100161732608.0,
+ "6525": 100161732608.0,
+ "6530": 100161732608.0,
+ "6535": 100161732608.0,
+ "6540": 100161732608.0,
+ "6545": 100161732608.0,
+ "6550": 100456341504.0,
+ "6555": 100456341504.0,
+ "6560": 100456341504.0,
+ "6565": 100456341504.0,
+ "6570": 100456341504.0,
+ "6575": 100456341504.0,
+ "6580": 100456341504.0,
+ "6585": 100456341504.0,
+ "6590": 100456341504.0,
+ "6595": 100456341504.0,
+ "6600": 100456341504.0,
+ "6605": 100456341504.0,
+ "6610": 100456341504.0,
+ "6615": 100456341504.0,
+ "6620": 100456341504.0,
+ "6625": 100456341504.0,
+ "6630": 100456341504.0,
+ "6635": 100456341504.0,
+ "6640": 100456341504.0,
+ "6645": 100456341504.0,
+ "6650": 100456341504.0,
+ "6655": 100456341504.0,
+ "6660": 100456341504.0,
+ "6665": 100456341504.0,
+ "6670": 100456341504.0,
+ "6675": 100456341504.0,
+ "6680": 100456341504.0,
+ "6685": 100456341504.0,
+ "6690": 100456341504.0,
+ "6695": 100456341504.0,
+ "6700": 100456341504.0,
+ "6705": 100456341504.0,
+ "6710": 100456341504.0,
+ "6715": 100456341504.0,
+ "6720": 100456341504.0,
+ "6725": 100456341504.0,
+ "6730": 100456341504.0,
+ "6735": 100456341504.0,
+ "6740": 100456341504.0,
+ "6745": 100456341504.0,
+ "6750": 100456341504.0,
+ "6755": 100456341504.0,
+ "6760": 100456341504.0,
+ "6765": 100456341504.0,
+ "6770": 100456341504.0,
+ "6775": 100456341504.0,
+ "6780": 100456341504.0,
+ "6785": 100456341504.0,
+ "6790": 100456341504.0,
+ "6795": 100456341504.0,
+ "6800": 100456341504.0,
+ "6805": 100456341504.0,
+ "6810": 100456341504.0,
+ "6815": 100456341504.0,
+ "6820": 100456341504.0,
+ "6825": 100456341504.0,
+ "6830": 100456341504.0,
+ "6835": 100456341504.0,
+ "6840": 100456341504.0,
+ "6845": 100456341504.0,
+ "6850": 100456341504.0,
+ "6855": 100456341504.0,
+ "6860": 100456341504.0,
+ "6865": 100456341504.0,
+ "6870": 100456341504.0,
+ "6875": 100456341504.0,
+ "6880": 100456341504.0,
+ "6885": 100456341504.0,
+ "6890": 100456341504.0,
+ "6895": 100456341504.0,
+ "6900": 100456341504.0,
+ "6905": 100456341504.0,
+ "6910": 100456341504.0,
+ "6915": 100456341504.0,
+ "6920": 100456341504.0,
+ "6925": 100456341504.0,
+ "6930": 100456341504.0,
+ "6935": 100456341504.0,
+ "6940": 100456341504.0,
+ "6945": 100456341504.0,
+ "6950": 100456341504.0,
+ "6955": 100456341504.0,
+ "6960": 100456341504.0,
+ "6965": 100456341504.0,
+ "6970": 100456341504.0,
+ "6975": 100456341504.0,
+ "6980": 100456341504.0,
+ "6985": 99321937920.0,
+ "6990": 99358261248.0,
+ "6995": 99669385216.0,
+ "7000": 99669385216.0,
+ "7005": 99772514304.0,
+ "7010": 99772514304.0,
+ "7015": 99772514304.0,
+ "7020": 99772514304.0,
+ "7025": 99772514304.0,
+ "7030": 99772514304.0,
+ "7035": 99772514304.0,
+ "7040": 99772514304.0,
+ "7045": 100171079680.0,
+ "7050": 100171079680.0,
+ "7055": 100171079680.0,
+ "7060": 100171079680.0,
+ "7065": 100171079680.0,
+ "7070": 100171079680.0,
+ "7075": 100171079680.0,
+ "7080": 100171079680.0,
+ "7085": 100171079680.0,
+ "7090": 100171079680.0,
+ "7095": 100171079680.0,
+ "7100": 100171079680.0,
+ "7105": 100171079680.0,
+ "7110": 100171079680.0,
+ "7115": 100171079680.0,
+ "7120": 100171079680.0,
+ "7125": 100171079680.0,
+ "7130": 100171079680.0,
+ "7135": 100171079680.0,
+ "7140": 100171079680.0,
+ "7145": 100171079680.0,
+ "7150": 100171079680.0,
+ "7155": 100171079680.0,
+ "7160": 100171079680.0,
+ "7165": 100171079680.0,
+ "7170": 100171079680.0,
+ "7175": 100171079680.0,
+ "7180": 100171079680.0,
+ "7185": 100171079680.0,
+ "7190": 100171079680.0,
+ "7195": 100171079680.0,
+ "7200": 100171079680.0,
+ "7205": 100171079680.0,
+ "7210": 100171079680.0,
+ "7215": 100171079680.0,
+ "7220": 100171079680.0,
+ "7225": 100171079680.0,
+ "7230": 100171079680.0,
+ "7235": 100171079680.0,
+ "7240": 100171079680.0,
+ "7245": 100171079680.0,
+ "7250": 100171079680.0,
+ "7255": 100171079680.0,
+ "7260": 100171079680.0,
+ "7265": 100171079680.0,
+ "7270": 100171079680.0,
+ "7275": 100171079680.0,
+ "7280": 100171079680.0,
+ "7285": 100171079680.0,
+ "7290": 100171079680.0,
+ "7295": 100171079680.0,
+ "7300": 100171079680.0,
+ "7305": 100171079680.0,
+ "7310": 100171079680.0,
+ "7315": 100171079680.0,
+ "7320": 100171079680.0,
+ "7325": 100171079680.0,
+ "7330": 100171079680.0,
+ "7335": 100171079680.0,
+ "7340": 100171079680.0,
+ "7345": 100171079680.0,
+ "7350": 100171079680.0,
+ "7355": 100171079680.0,
+ "7360": 100171079680.0,
+ "7365": 100171079680.0,
+ "7370": 100171079680.0,
+ "7375": 100171079680.0,
+ "7380": 100171079680.0,
+ "7385": 100171079680.0,
+ "7390": 100171079680.0,
+ "7395": 100171079680.0,
+ "7400": 100171079680.0,
+ "7405": 100171079680.0,
+ "7410": 100171079680.0,
+ "7415": 100171079680.0,
+ "7420": 100171079680.0,
+ "7425": 100171079680.0,
+ "7430": 100171079680.0,
+ "7435": 100171079680.0,
+ "7440": 100171079680.0,
+ "7445": 100171079680.0,
+ "7450": 100171079680.0,
+ "7455": 100171079680.0,
+ "7460": 100171079680.0,
+ "7465": 100171079680.0,
+ "7470": 100171079680.0,
+ "7475": 100171079680.0,
+ "7480": 100171079680.0,
+ "7485": 100171079680.0,
+ "7490": 100171079680.0,
+ "7495": 100171079680.0,
+ "7500": 100171079680.0,
+ "7505": 100171079680.0,
+ "7510": 100171079680.0,
+ "7515": 100171079680.0,
+ "7520": 100171079680.0,
+ "7525": 100171079680.0,
+ "7530": 100171079680.0,
+ "7535": 100171079680.0,
+ "7540": 100171079680.0,
+ "7545": 100171079680.0,
+ "7550": 100171079680.0,
+ "7555": 100171079680.0,
+ "7560": 100171079680.0,
+ "7565": 100171079680.0,
+ "7570": 100171079680.0,
+ "7575": 100171079680.0,
+ "7580": 100171079680.0,
+ "7585": 100171079680.0,
+ "7590": 100171079680.0,
+ "7595": 100171079680.0,
+ "7600": 100171079680.0,
+ "7605": 100171079680.0,
+ "7610": 100171079680.0,
+ "7615": 100171079680.0,
+ "7620": 100171079680.0,
+ "7625": 100171079680.0,
+ "7630": 100171079680.0,
+ "7635": 100171079680.0,
+ "7640": 100171079680.0,
+ "7645": 100171079680.0,
+ "7650": 100171079680.0,
+ "7655": 100171079680.0,
+ "7660": 100171079680.0,
+ "7665": 100171079680.0,
+ "7670": 100171079680.0,
+ "7675": 100171079680.0,
+ "7680": 100171079680.0,
+ "7685": 100171079680.0,
+ "7690": 100171079680.0,
+ "7695": 100171079680.0,
+ "7700": 100171079680.0,
+ "7705": 100171079680.0,
+ "7710": 100171079680.0,
+ "7715": 100171079680.0,
+ "7720": 100171079680.0,
+ "7725": 100171079680.0,
+ "7730": 100171079680.0,
+ "7735": 100171079680.0,
+ "7740": 100171079680.0,
+ "7745": 100171079680.0,
+ "7750": 100171079680.0,
+ "7755": 100171079680.0,
+ "7760": 100171079680.0,
+ "7765": 100171079680.0,
+ "7770": 100171079680.0,
+ "7775": 100171079680.0,
+ "7780": 100171079680.0,
+ "7785": 100171079680.0,
+ "7790": 100171079680.0,
+ "7795": 100171079680.0,
+ "7800": 100171079680.0,
+ "7805": 100171079680.0,
+ "7810": 100171079680.0,
+ "7815": 100171079680.0,
+ "7820": 100171079680.0,
+ "7825": 100171079680.0,
+ "7830": 100171079680.0,
+ "7835": 100171079680.0,
+ "7840": 100171079680.0,
+ "7845": 100171079680.0,
+ "7850": 100171079680.0,
+ "7855": 100171079680.0,
+ "7860": 100171079680.0,
+ "7865": 100171079680.0,
+ "7870": 100171079680.0,
+ "7875": 100171079680.0,
+ "7880": 100171079680.0,
+ "7885": 100171079680.0,
+ "7890": 100171079680.0,
+ "7895": 100171079680.0,
+ "7900": 100171079680.0,
+ "7905": 100171079680.0,
+ "7910": 100171079680.0,
+ "7915": 100171079680.0,
+ "7920": 100171079680.0,
+ "7925": 100171079680.0,
+ "7930": 100171079680.0,
+ "7935": 100171079680.0,
+ "7940": 100171079680.0,
+ "7945": 100171079680.0,
+ "7950": 100171079680.0,
+ "7955": 100171079680.0,
+ "7960": 100171079680.0,
+ "7965": 100171079680.0,
+ "7970": 100171079680.0,
+ "7975": 100171079680.0,
+ "7980": 100171079680.0,
+ "7985": 100171079680.0,
+ "7990": 100171079680.0,
+ "7995": 100171079680.0,
+ "8000": 100171079680.0,
+ "8005": 100171079680.0,
+ "8010": 100171079680.0,
+ "8015": 100171079680.0,
+ "8020": 100171079680.0,
+ "8025": 100171079680.0,
+ "8030": 100171079680.0,
+ "8035": 100171079680.0,
+ "8040": 100171079680.0,
+ "8045": 100171079680.0,
+ "8050": 100171079680.0,
+ "8055": 100171079680.0,
+ "8060": 100171079680.0,
+ "8065": 100171079680.0,
+ "8070": 100171079680.0,
+ "8075": 100171079680.0,
+ "8080": 100171079680.0,
+ "8085": 100171079680.0,
+ "8090": 100171079680.0,
+ "8095": 100171079680.0,
+ "8100": 100171079680.0,
+ "8105": 100171079680.0,
+ "8110": 100171079680.0,
+ "8115": 100171079680.0,
+ "8120": 100171079680.0,
+ "8125": 100171079680.0,
+ "8130": 100171079680.0,
+ "8135": 100171079680.0,
+ "8140": 100171079680.0,
+ "8145": 100171079680.0,
+ "8150": 100171079680.0,
+ "8155": 100171079680.0,
+ "8160": 100171079680.0,
+ "8165": 100171079680.0,
+ "8170": 100171079680.0,
+ "8175": 100171079680.0,
+ "8180": 100171079680.0,
+ "8185": 100171079680.0,
+ "8190": 100171079680.0,
+ "8195": 100171079680.0,
+ "8200": 100171079680.0,
+ "8205": 100171079680.0,
+ "8210": 100171079680.0,
+ "8215": 100171079680.0,
+ "8220": 100171079680.0,
+ "8225": 100171079680.0,
+ "8230": 100171079680.0,
+ "8235": 100171079680.0,
+ "8240": 100171079680.0,
+ "8245": 100171079680.0,
+ "8250": 100171079680.0,
+ "8255": 100171079680.0,
+ "8260": 100171079680.0,
+ "8265": 100171079680.0,
+ "8270": 100171079680.0,
+ "8275": 100171079680.0,
+ "8280": 100171079680.0,
+ "8285": 100171079680.0,
+ "8290": 100171079680.0,
+ "8295": 100171079680.0,
+ "8300": 100171079680.0,
+ "8305": 100171079680.0,
+ "8310": 100171079680.0,
+ "8315": 100171079680.0,
+ "8320": 100171079680.0,
+ "8325": 100171079680.0,
+ "8330": 100171079680.0,
+ "8335": 100171079680.0,
+ "8340": 100171079680.0,
+ "8345": 100171079680.0,
+ "8350": 99323297792.0,
+ "8355": 99323297792.0,
+ "8360": 99323297792.0,
+ "8365": 99513737216.0,
+ "8370": 99513737216.0,
+ "8375": 99513737216.0,
+ "8380": 99513737216.0,
+ "8385": 99694624768.0,
+ "8390": 99781484544.0,
+ "8395": 99781484544.0,
+ "8400": 99781484544.0,
+ "8405": 99781484544.0,
+ "8410": 99781484544.0,
+ "8415": 100083359744.0,
+ "8420": 100083359744.0,
+ "8425": 100083359744.0,
+ "8430": 100083359744.0,
+ "8435": 100083359744.0,
+ "8440": 100083359744.0,
+ "8445": 100083359744.0,
+ "8450": 100083359744.0,
+ "8455": 100083359744.0,
+ "8460": 100083359744.0,
+ "8465": 100083359744.0,
+ "8470": 100083359744.0,
+ "8475": 100083359744.0,
+ "8480": 100083359744.0,
+ "8485": 100083359744.0,
+ "8490": 100083359744.0,
+ "8495": 100083359744.0,
+ "8500": 100083359744.0,
+ "8505": 100083359744.0,
+ "8510": 100083359744.0,
+ "8515": 100083359744.0,
+ "8520": 100083359744.0,
+ "8525": 100083359744.0,
+ "8530": 100083359744.0,
+ "8535": 100083359744.0,
+ "8540": 100083359744.0,
+ "8545": 100083359744.0,
+ "8550": 100083359744.0,
+ "8555": 100083359744.0,
+ "8560": 100083359744.0,
+ "8565": 100083359744.0,
+ "8570": 100083359744.0,
+ "8575": 100083359744.0,
+ "8580": 100083359744.0,
+ "8585": 100083359744.0,
+ "8590": 100083359744.0,
+ "8595": 100083359744.0,
+ "8600": 100083359744.0,
+ "8605": 100083359744.0,
+ "8610": 100083359744.0,
+ "8615": 100083359744.0,
+ "8620": 100083359744.0,
+ "8625": 100083359744.0,
+ "8630": 100083359744.0,
+ "8635": 100083359744.0,
+ "8640": 100083359744.0,
+ "8645": 100083359744.0,
+ "8650": 99754590208.0,
+ "8655": 99754590208.0,
+ "8660": 99754590208.0,
+ "8665": 99754590208.0,
+ "8670": 99754590208.0,
+ "8675": 99754590208.0,
+ "8680": 99754590208.0,
+ "8685": 99754590208.0,
+ "8690": 99928203264.0,
+ "8695": 99928203264.0,
+ "8700": 99928203264.0,
+ "8705": 99809550336.0,
+ "8710": 99809550336.0,
+ "8715": 99809550336.0,
+ "8720": 99809550336.0,
+ "8725": 99809550336.0,
+ "8730": 99921281024.0,
+ "8735": 99921281024.0,
+ "8740": 99921281024.0,
+ "8745": 99921281024.0,
+ "8750": 99921281024.0,
+ "8755": 99921281024.0,
+ "8760": 99921281024.0,
+ "8765": 99921281024.0,
+ "8770": 99921281024.0,
+ "8775": 99921281024.0,
+ "8780": 99921281024.0,
+ "8785": 99921281024.0,
+ "8790": 99921281024.0,
+ "8795": 99921281024.0,
+ "8800": 99921281024.0,
+ "8805": 99921281024.0,
+ "8810": 99921281024.0,
+ "8815": 99921281024.0,
+ "8820": 99921281024.0,
+ "8825": 99921281024.0,
+ "8830": 99921281024.0,
+ "8835": 99921281024.0,
+ "8840": 100527816704.0,
+ "8845": 100527816704.0,
+ "8850": 100527816704.0,
+ "8855": 100527816704.0,
+ "8860": 100527816704.0,
+ "8865": 100527816704.0,
+ "8870": 100527816704.0,
+ "8875": 100527816704.0,
+ "8880": 100527816704.0,
+ "8885": 100527816704.0,
+ "8890": 100527816704.0,
+ "8895": 100527816704.0,
+ "8900": 100527816704.0,
+ "8905": 100527816704.0,
+ "8910": 100527816704.0,
+ "8915": 100527816704.0,
+ "8920": 100527816704.0,
+ "8925": 100527816704.0,
+ "8930": 100527816704.0,
+ "8935": 100527816704.0,
+ "8940": 100527816704.0,
+ "8945": 100527816704.0,
+ "8950": 100527816704.0,
+ "8955": 100527816704.0,
+ "8960": 100527816704.0,
+ "8965": 100527816704.0,
+ "8970": 100527816704.0,
+ "8975": 100527816704.0,
+ "8980": 100527816704.0,
+ "8985": 100527816704.0,
+ "8990": 100527816704.0,
+ "8995": 100527816704.0,
+ "9000": 100527816704.0,
+ "9005": 100527816704.0,
+ "9010": 100527816704.0,
+ "9015": 100527816704.0,
+ "9020": 100527816704.0,
+ "9025": 100527816704.0,
+ "9030": 100527816704.0,
+ "9035": 100527816704.0,
+ "9040": 100527816704.0,
+ "9045": 100527816704.0,
+ "9050": 100527816704.0,
+ "9055": 100527816704.0,
+ "9060": 100527816704.0,
+ "9065": 100527816704.0,
+ "9070": 100527816704.0,
+ "9075": 100527816704.0,
+ "9080": 100527816704.0,
+ "9085": 100527816704.0,
+ "9090": 100527816704.0,
+ "9095": 100527816704.0,
+ "9100": 100527816704.0,
+ "9105": 100527816704.0,
+ "9110": 100527816704.0,
+ "9115": 100527816704.0,
+ "9120": 100527816704.0,
+ "9125": 100527816704.0,
+ "9130": 100527816704.0,
+ "9135": 100527816704.0,
+ "9140": 100527816704.0,
+ "9145": 100527816704.0,
+ "9150": 100527816704.0,
+ "9155": 100527816704.0,
+ "9160": 100527816704.0,
+ "9165": 100527816704.0,
+ "9170": 100527816704.0,
+ "9175": 100527816704.0,
+ "9180": 100527816704.0,
+ "9185": 100527816704.0,
+ "9190": 100527816704.0,
+ "9195": 100527816704.0,
+ "9200": 100527816704.0,
+ "9205": 100527816704.0,
+ "9210": 100527816704.0,
+ "9215": 100527816704.0,
+ "9220": 100527816704.0,
+ "9225": 100527816704.0,
+ "9230": 100527816704.0,
+ "9235": 100527816704.0,
+ "9240": 100527816704.0,
+ "9245": 100527816704.0,
+ "9250": 100527816704.0,
+ "9255": 100527816704.0,
+ "9260": 100527816704.0,
+ "9265": 100527816704.0,
+ "9270": 100527816704.0,
+ "9275": 100527816704.0,
+ "9280": 100527816704.0,
+ "9285": 100527816704.0,
+ "9290": 100527816704.0,
+ "9295": 100527816704.0,
+ "9300": 100527816704.0,
+ "9305": 100527816704.0,
+ "9310": 100527816704.0,
+ "9315": 100527816704.0,
+ "9320": 100527816704.0,
+ "9325": 100527816704.0,
+ "9330": 100527816704.0,
+ "9335": 100527816704.0,
+ "9340": 100527816704.0,
+ "9345": 100527816704.0,
+ "9350": 100527816704.0,
+ "9355": 100527816704.0,
+ "9360": 100527816704.0,
+ "9365": 100527816704.0,
+ "9370": 100527816704.0,
+ "9375": 100527816704.0,
+ "9380": 100527816704.0,
+ "9385": 100527816704.0,
+ "9390": 100527816704.0,
+ "9395": 100527816704.0,
+ "9400": 100527816704.0,
+ "9405": 100527816704.0,
+ "9410": 100527816704.0,
+ "9415": 100527816704.0,
+ "9420": 100527816704.0,
+ "9425": 100527816704.0,
+ "9430": 100527816704.0,
+ "9435": 100527816704.0,
+ "9440": 100527816704.0,
+ "9445": 100527816704.0,
+ "9450": 100527816704.0,
+ "9455": 100527816704.0,
+ "9460": 100527816704.0,
+ "9465": 100527816704.0,
+ "9470": 100527816704.0,
+ "9475": 100527816704.0,
+ "9480": 100527816704.0,
+ "9485": 100527816704.0,
+ "9490": 100527816704.0,
+ "9495": 100527816704.0,
+ "9500": 100527816704.0,
+ "9505": 100527816704.0,
+ "9510": 100527816704.0,
+ "9515": 100527816704.0,
+ "9520": 100527816704.0,
+ "9525": 100527816704.0,
+ "9530": 100527816704.0,
+ "9535": 100527816704.0
+ }
+ },
+ "mtp_1 loss": {
+ "start_step": 1,
+ "end_step": 9535,
+ "step_interval": 5,
+ "values": {
+ "1": 13.88149,
+ "5": 13.88459,
+ "10": 13.87742,
+ "15": 13.8784,
+ "20": 13.87227,
+ "25": 13.86402,
+ "30": 13.84302,
+ "35": 13.83065,
+ "40": 13.82503,
+ "45": 13.81402,
+ "50": 13.72521,
+ "55": 13.70233,
+ "60": 13.68399,
+ "65": 13.64861,
+ "70": 13.60669,
+ "75": 13.41734,
+ "80": 13.33709,
+ "85": 13.25466,
+ "90": 13.15983,
+ "95": 13.01828,
+ "100": 12.86067,
+ "105": 12.68712,
+ "110": 12.38538,
+ "115": 12.16587,
+ "120": 11.95218,
+ "125": 11.7955,
+ "130": 11.67052,
+ "135": 11.49862,
+ "140": 11.28916,
+ "145": 11.21005,
+ "150": 11.05991,
+ "155": 10.97334,
+ "160": 10.83939,
+ "165": 10.70654,
+ "170": 10.62639,
+ "175": 10.57382,
+ "180": 10.41387,
+ "185": 10.40926,
+ "190": 10.26597,
+ "195": 10.24462,
+ "200": 10.11775,
+ "205": 9.96939,
+ "210": 9.93921,
+ "215": 9.9114,
+ "220": 9.78091,
+ "225": 9.76808,
+ "230": 9.71982,
+ "235": 9.63333,
+ "240": 9.55828,
+ "245": 9.49171,
+ "250": 9.42536,
+ "255": 9.36534,
+ "260": 9.28417,
+ "265": 9.22589,
+ "270": 9.14087,
+ "275": 9.11265,
+ "280": 9.08509,
+ "285": 9.07832,
+ "290": 8.99347,
+ "295": 8.93526,
+ "300": 8.81808,
+ "305": 8.78944,
+ "310": 8.73361,
+ "315": 8.69871,
+ "320": 8.66114,
+ "325": 8.56239,
+ "330": 8.52803,
+ "335": 8.49425,
+ "340": 8.4854,
+ "345": 8.35752,
+ "350": 8.33583,
+ "355": 8.23235,
+ "360": 8.31891,
+ "365": 8.2196,
+ "370": 8.21417,
+ "375": 8.16192,
+ "380": 8.11794,
+ "385": 8.10804,
+ "390": 8.09194,
+ "395": 8.04724,
+ "400": 7.965,
+ "405": 7.96243,
+ "410": 7.95876,
+ "415": 7.90318,
+ "420": 7.89109,
+ "425": 7.84851,
+ "430": 7.79096,
+ "435": 7.80616,
+ "440": 7.75443,
+ "445": 7.73467,
+ "450": 7.6681,
+ "455": 7.69771,
+ "460": 7.6505,
+ "465": 7.62273,
+ "470": 7.58376,
+ "475": 7.59827,
+ "480": 7.46473,
+ "485": 7.51047,
+ "490": 7.46627,
+ "495": 7.45073,
+ "500": 7.39442,
+ "505": 7.40479,
+ "510": 7.38007,
+ "515": 7.3519,
+ "520": 7.34619,
+ "525": 7.32261,
+ "530": 7.32434,
+ "535": 7.30553,
+ "540": 7.21927,
+ "545": 7.24174,
+ "550": 7.2747,
+ "555": 7.29438,
+ "560": 7.23151,
+ "565": 7.15099,
+ "570": 7.15993,
+ "575": 7.17825,
+ "580": 7.1014,
+ "585": 7.11017,
+ "590": 7.05487,
+ "595": 7.03688,
+ "600": 7.06638,
+ "605": 7.06021,
+ "610": 7.02161,
+ "615": 7.07863,
+ "620": 6.98539,
+ "625": 6.96034,
+ "630": 6.96112,
+ "635": 6.99078,
+ "640": 6.9701,
+ "645": 6.95391,
+ "650": 7.00459,
+ "655": 6.99643,
+ "660": 6.89365,
+ "665": 6.87445,
+ "670": 6.84074,
+ "675": 6.93316,
+ "680": 6.89433,
+ "685": 6.85455,
+ "690": 6.83221,
+ "695": 6.79654,
+ "700": 6.79167,
+ "705": 6.78676,
+ "710": 6.82673,
+ "715": 6.82633,
+ "720": 6.71185,
+ "725": 6.76682,
+ "730": 6.75699,
+ "735": 6.75626,
+ "740": 6.70096,
+ "745": 6.67594,
+ "750": 6.73453,
+ "755": 6.65368,
+ "760": 6.66197,
+ "765": 6.65857,
+ "770": 6.67683,
+ "775": 6.64734,
+ "780": 6.61756,
+ "785": 6.6334,
+ "790": 6.585,
+ "795": 6.58465,
+ "800": 6.57883,
+ "805": 6.64411,
+ "810": 6.50849,
+ "815": 6.5313,
+ "820": 6.54072,
+ "825": 6.54651,
+ "830": 6.56061,
+ "835": 6.51406,
+ "840": 6.47897,
+ "845": 6.53209,
+ "850": 6.48384,
+ "855": 6.48294,
+ "860": 6.47583,
+ "865": 6.48821,
+ "870": 6.45192,
+ "875": 6.50047,
+ "880": 6.46354,
+ "885": 6.42948,
+ "890": 6.50676,
+ "895": 6.38904,
+ "900": 6.41234,
+ "905": 6.43685,
+ "910": 6.39805,
+ "915": 6.37881,
+ "920": 6.37824,
+ "925": 6.36255,
+ "930": 6.39793,
+ "935": 6.38605,
+ "940": 6.33445,
+ "945": 6.36303,
+ "950": 6.38742,
+ "955": 6.34038,
+ "960": 6.34516,
+ "965": 6.25084,
+ "970": 6.31565,
+ "975": 6.30751,
+ "980": 6.2837,
+ "985": 6.28202,
+ "990": 6.33642,
+ "995": 6.25773,
+ "1000": 6.27661,
+ "1005": 6.22219,
+ "1010": 6.25739,
+ "1015": 6.28493,
+ "1020": 6.20363,
+ "1025": 6.20671,
+ "1030": 6.19982,
+ "1035": 6.29515,
+ "1040": 6.2192,
+ "1045": 6.19353,
+ "1050": 6.22372,
+ "1055": 6.21471,
+ "1060": 6.16254,
+ "1065": 6.15381,
+ "1070": 6.18798,
+ "1075": 6.1886,
+ "1080": 6.188,
+ "1085": 6.19061,
+ "1090": 6.17415,
+ "1095": 6.17606,
+ "1100": 6.13774,
+ "1105": 6.11074,
+ "1110": 6.18132,
+ "1115": 6.11087,
+ "1120": 6.05066,
+ "1125": 6.08513,
+ "1130": 6.13784,
+ "1135": 6.09319,
+ "1140": 6.07951,
+ "1145": 6.06524,
+ "1150": 6.08902,
+ "1155": 6.06291,
+ "1160": 6.04749,
+ "1165": 6.09583,
+ "1170": 6.0711,
+ "1175": 6.04225,
+ "1180": 6.04768,
+ "1185": 6.03836,
+ "1190": 6.04667,
+ "1195": 6.02603,
+ "1200": 5.97435,
+ "1205": 6.07343,
+ "1210": 5.93685,
+ "1215": 5.98192,
+ "1220": 6.05691,
+ "1225": 5.94834,
+ "1230": 5.99496,
+ "1235": 5.95561,
+ "1240": 5.98691,
+ "1245": 5.96808,
+ "1250": 5.9477,
+ "1255": 5.94386,
+ "1260": 5.94879,
+ "1265": 5.92747,
+ "1270": 5.90474,
+ "1275": 5.96341,
+ "1280": 5.90595,
+ "1285": 5.91542,
+ "1290": 5.90264,
+ "1295": 5.92158,
+ "1300": 5.92174,
+ "1305": 5.89438,
+ "1310": 5.8357,
+ "1315": 5.89674,
+ "1320": 5.89598,
+ "1325": 5.82605,
+ "1330": 5.88269,
+ "1335": 5.84953,
+ "1340": 5.91413,
+ "1345": 5.86192,
+ "1350": 5.84207,
+ "1355": 5.83967,
+ "1360": 5.85167,
+ "1365": 5.84067,
+ "1370": 5.79368,
+ "1375": 5.80572,
+ "1380": 5.86178,
+ "1385": 5.81254,
+ "1390": 5.80853,
+ "1395": 5.82398,
+ "1400": 5.82035,
+ "1405": 5.81422,
+ "1410": 5.78475,
+ "1415": 5.7629,
+ "1420": 5.79752,
+ "1425": 5.78747,
+ "1430": 5.82972,
+ "1435": 5.73914,
+ "1440": 5.75819,
+ "1445": 5.81014,
+ "1450": 5.79226,
+ "1455": 5.79745,
+ "1460": 5.75867,
+ "1465": 5.76097,
+ "1470": 5.79993,
+ "1475": 5.76984,
+ "1480": 5.77882,
+ "1485": 5.71927,
+ "1490": 5.72329,
+ "1495": 5.75024,
+ "1500": 5.7493,
+ "1505": 5.71869,
+ "1510": 5.74199,
+ "1515": 5.66791,
+ "1520": 5.69933,
+ "1525": 5.6673,
+ "1530": 5.69234,
+ "1535": 5.6792,
+ "1540": 5.6694,
+ "1545": 5.71701,
+ "1550": 5.72191,
+ "1555": 5.70884,
+ "1560": 5.65081,
+ "1565": 5.69355,
+ "1570": 5.70543,
+ "1575": 5.65203,
+ "1580": 5.69117,
+ "1585": 5.66843,
+ "1590": 5.64892,
+ "1595": 5.63421,
+ "1600": 5.70224,
+ "1605": 5.6422,
+ "1610": 5.64226,
+ "1615": 5.62793,
+ "1620": 5.64949,
+ "1625": 5.63891,
+ "1630": 5.62164,
+ "1635": 5.67389,
+ "1640": 5.62092,
+ "1645": 5.64491,
+ "1650": 5.62856,
+ "1655": 5.61295,
+ "1660": 5.60838,
+ "1665": 5.59821,
+ "1670": 5.61023,
+ "1675": 5.61215,
+ "1680": 5.56047,
+ "1685": 5.56618,
+ "1690": 5.54252,
+ "1695": 5.54887,
+ "1700": 5.59588,
+ "1705": 5.5702,
+ "1710": 5.57945,
+ "1715": 5.53737,
+ "1720": 5.51813,
+ "1725": 5.56141,
+ "1730": 5.52484,
+ "1735": 5.57815,
+ "1740": 5.51468,
+ "1745": 5.54899,
+ "1750": 5.52724,
+ "1755": 5.52637,
+ "1760": 5.54373,
+ "1765": 5.50469,
+ "1770": 5.51855,
+ "1775": 5.51603,
+ "1780": 5.53176,
+ "1785": 5.47532,
+ "1790": 5.51315,
+ "1795": 5.51923,
+ "1800": 5.46118,
+ "1805": 5.45765,
+ "1810": 5.47178,
+ "1815": 5.47795,
+ "1820": 5.47256,
+ "1825": 5.4756,
+ "1830": 5.45775,
+ "1835": 5.45408,
+ "1840": 5.45442,
+ "1845": 5.44321,
+ "1850": 5.42073,
+ "1855": 5.47843,
+ "1860": 5.42692,
+ "1865": 5.43694,
+ "1870": 5.4315,
+ "1875": 5.41903,
+ "1880": 5.47704,
+ "1885": 5.44137,
+ "1890": 5.43511,
+ "1895": 5.37795,
+ "1900": 5.41841,
+ "1905": 5.40142,
+ "1910": 5.43049,
+ "1915": 5.39075,
+ "1920": 5.36942,
+ "1925": 5.40235,
+ "1930": 5.37334,
+ "1935": 5.39474,
+ "1940": 5.36726,
+ "1945": 5.4161,
+ "1950": 5.4522,
+ "1955": 5.38528,
+ "1960": 5.38889,
+ "1965": 5.33268,
+ "1970": 5.33408,
+ "1975": 5.40018,
+ "1980": 5.34567,
+ "1985": 5.36976,
+ "1990": 5.39686,
+ "1995": 5.36823,
+ "2000": 5.3746,
+ "2005": 5.42505,
+ "2010": 5.332,
+ "2015": 5.31338,
+ "2020": 5.3265,
+ "2025": 5.36896,
+ "2030": 5.30616,
+ "2035": 5.31999,
+ "2040": 5.28872,
+ "2045": 5.38051,
+ "2050": 5.35063,
+ "2055": 5.32057,
+ "2060": 5.32079,
+ "2065": 5.2925,
+ "2070": 5.2944,
+ "2075": 5.328,
+ "2080": 5.29527,
+ "2085": 5.32226,
+ "2090": 5.23851,
+ "2095": 5.28928,
+ "2100": 5.25427,
+ "2105": 5.27757,
+ "2110": 5.27304,
+ "2115": 5.27472,
+ "2120": 5.27742,
+ "2125": 5.24645,
+ "2130": 5.2458,
+ "2135": 5.24801,
+ "2140": 5.26018,
+ "2145": 5.22058,
+ "2150": 5.24022,
+ "2155": 5.22143,
+ "2160": 5.23599,
+ "2165": 5.22899,
+ "2170": 5.25856,
+ "2175": 5.25622,
+ "2180": 5.23867,
+ "2185": 5.24065,
+ "2190": 5.22138,
+ "2195": 5.19739,
+ "2200": 5.19989,
+ "2205": 5.20827,
+ "2210": 5.24721,
+ "2215": 5.28825,
+ "2220": 5.23195,
+ "2225": 5.21848,
+ "2230": 5.21576,
+ "2235": 5.25586,
+ "2240": 5.15501,
+ "2245": 5.15323,
+ "2250": 5.17878,
+ "2255": 5.19188,
+ "2260": 5.13026,
+ "2265": 5.20954,
+ "2270": 5.13726,
+ "2275": 5.19209,
+ "2280": 5.16132,
+ "2285": 5.18239,
+ "2290": 5.17175,
+ "2295": 5.17812,
+ "2300": 5.17404,
+ "2305": 5.14721,
+ "2310": 5.18112,
+ "2315": 5.12057,
+ "2320": 5.16861,
+ "2325": 5.14917,
+ "2330": 5.14295,
+ "2335": 5.1262,
+ "2340": 5.13025,
+ "2345": 5.17938,
+ "2350": 5.12325,
+ "2355": 5.10897,
+ "2360": 5.09256,
+ "2365": 5.1142,
+ "2370": 5.10097,
+ "2375": 5.10699,
+ "2380": 5.04789,
+ "2385": 5.0911,
+ "2390": 5.11281,
+ "2395": 5.12291,
+ "2400": 5.0766,
+ "2405": 5.0566,
+ "2410": 5.10874,
+ "2415": 5.08173,
+ "2420": 5.10224,
+ "2425": 5.05822,
+ "2430": 5.08851,
+ "2435": 5.0781,
+ "2440": 5.06984,
+ "2445": 5.07864,
+ "2450": 5.04327,
+ "2455": 5.0856,
+ "2460": 5.04108,
+ "2465": 5.07697,
+ "2470": 5.06975,
+ "2475": 5.10432,
+ "2480": 5.02375,
+ "2485": 5.05297,
+ "2490": 5.04607,
+ "2495": 5.03808,
+ "2500": 5.02744,
+ "2505": 5.04137,
+ "2510": 5.08226,
+ "2515": 5.07635,
+ "2520": 5.01744,
+ "2525": 5.02816,
+ "2530": 5.04333,
+ "2535": 5.03372,
+ "2540": 5.03998,
+ "2545": 5.04375,
+ "2550": 4.98994,
+ "2555": 5.04992,
+ "2560": 5.02916,
+ "2565": 4.99102,
+ "2570": 5.01959,
+ "2575": 4.98066,
+ "2580": 4.99189,
+ "2585": 4.97007,
+ "2590": 4.99362,
+ "2595": 4.94993,
+ "2600": 4.98457,
+ "2605": 5.00741,
+ "2610": 5.00432,
+ "2615": 4.97428,
+ "2620": 4.94537,
+ "2625": 4.98718,
+ "2630": 4.91333,
+ "2635": 5.00021,
+ "2640": 4.99287,
+ "2645": 4.9562,
+ "2650": 4.97439,
+ "2655": 4.95916,
+ "2660": 4.90584,
+ "2665": 4.99721,
+ "2670": 4.94319,
+ "2675": 4.91464,
+ "2680": 4.95412,
+ "2685": 4.94386,
+ "2690": 4.91668,
+ "2695": 4.98637,
+ "2700": 4.8984,
+ "2705": 4.91063,
+ "2710": 4.95243,
+ "2715": 4.93182,
+ "2720": 4.95484,
+ "2725": 4.91252,
+ "2730": 4.93428,
+ "2735": 4.92464,
+ "2740": 4.91163,
+ "2745": 4.88359,
+ "2750": 4.92336,
+ "2755": 4.93451,
+ "2760": 4.93159,
+ "2765": 4.89634,
+ "2770": 4.93762,
+ "2775": 4.88912,
+ "2780": 4.92376,
+ "2785": 4.89844,
+ "2790": 4.92471,
+ "2795": 4.8906,
+ "2800": 4.83617,
+ "2805": 4.88212,
+ "2810": 4.86525,
+ "2815": 4.88564,
+ "2820": 4.92477,
+ "2825": 4.91377,
+ "2830": 4.88762,
+ "2835": 4.89809,
+ "2840": 4.88595,
+ "2845": 4.85377,
+ "2850": 4.89008,
+ "2855": 4.82772,
+ "2860": 4.8781,
+ "2865": 4.89415,
+ "2870": 4.87846,
+ "2875": 4.89788,
+ "2880": 4.81728,
+ "2885": 4.86036,
+ "2890": 4.83415,
+ "2895": 4.87218,
+ "2900": 4.83194,
+ "2905": 4.83652,
+ "2910": 4.83564,
+ "2915": 4.87955,
+ "2920": 4.86054,
+ "2925": 4.83379,
+ "2930": 4.81376,
+ "2935": 4.82293,
+ "2940": 4.82943,
+ "2945": 4.79285,
+ "2950": 4.775,
+ "2955": 4.77983,
+ "2960": 4.7945,
+ "2965": 4.8119,
+ "2970": 4.80269,
+ "2975": 4.83083,
+ "2980": 4.77395,
+ "2985": 4.81333,
+ "2990": 4.83327,
+ "2995": 4.7798,
+ "3000": 4.78377,
+ "3005": 4.77054,
+ "3010": 4.80336,
+ "3015": 4.7622,
+ "3020": 4.77986,
+ "3025": 4.79315,
+ "3030": 4.79771,
+ "3035": 4.79119,
+ "3040": 4.80692,
+ "3045": 4.796,
+ "3050": 4.77712,
+ "3055": 4.77365,
+ "3060": 4.7557,
+ "3065": 4.79329,
+ "3070": 4.79759,
+ "3075": 4.73645,
+ "3080": 4.76326,
+ "3085": 4.75928,
+ "3090": 4.75142,
+ "3095": 4.79265,
+ "3100": 4.78431,
+ "3105": 4.76228,
+ "3110": 4.74788,
+ "3115": 4.70101,
+ "3120": 4.76733,
+ "3125": 4.72129,
+ "3130": 4.73619,
+ "3135": 4.7364,
+ "3140": 4.71187,
+ "3145": 4.69842,
+ "3150": 4.73617,
+ "3155": 4.75873,
+ "3160": 4.74244,
+ "3165": 4.7392,
+ "3170": 4.73383,
+ "3175": 4.72227,
+ "3180": 4.71526,
+ "3185": 4.69034,
+ "3190": 4.6963,
+ "3195": 4.68667,
+ "3200": 4.66148,
+ "3205": 4.70638,
+ "3210": 4.66493,
+ "3215": 4.68875,
+ "3220": 4.65996,
+ "3225": 4.69161,
+ "3230": 4.67733,
+ "3235": 4.70789,
+ "3240": 4.66538,
+ "3245": 4.67658,
+ "3250": 4.61979,
+ "3255": 4.6757,
+ "3260": 4.65501,
+ "3265": 4.70163,
+ "3270": 4.68996,
+ "3275": 4.64111,
+ "3280": 4.65999,
+ "3285": 4.67829,
+ "3290": 4.65133,
+ "3295": 4.65232,
+ "3300": 4.64861,
+ "3305": 4.65149,
+ "3310": 4.64711,
+ "3315": 4.68966,
+ "3320": 4.62786,
+ "3325": 4.64047,
+ "3330": 4.61796,
+ "3335": 4.63784,
+ "3340": 4.60149,
+ "3345": 4.61917,
+ "3350": 4.63369,
+ "3355": 4.64479,
+ "3360": 4.62888,
+ "3365": 4.64316,
+ "3370": 4.61979,
+ "3375": 4.657,
+ "3380": 4.59521,
+ "3385": 4.60585,
+ "3390": 4.58064,
+ "3395": 4.67319,
+ "3400": 4.62047,
+ "3405": 4.65003,
+ "3410": 4.5833,
+ "3415": 4.5347,
+ "3420": 4.6017,
+ "3425": 4.6088,
+ "3430": 4.64756,
+ "3435": 4.61219,
+ "3440": 4.62976,
+ "3445": 4.57914,
+ "3450": 4.57663,
+ "3455": 4.60584,
+ "3460": 4.55949,
+ "3465": 4.55917,
+ "3470": 4.56753,
+ "3475": 4.57993,
+ "3480": 4.57299,
+ "3485": 4.59968,
+ "3490": 4.5795,
+ "3495": 4.61114,
+ "3500": 4.5721,
+ "3505": 4.57766,
+ "3510": 4.57951,
+ "3515": 4.62046,
+ "3520": 4.6034,
+ "3525": 4.55094,
+ "3530": 4.56803,
+ "3535": 4.55873,
+ "3540": 4.61722,
+ "3545": 4.53679,
+ "3550": 4.59753,
+ "3555": 4.52748,
+ "3560": 4.59751,
+ "3565": 4.53911,
+ "3570": 4.54116,
+ "3575": 4.51426,
+ "3580": 4.57639,
+ "3585": 4.56725,
+ "3590": 4.49219,
+ "3595": 4.56804,
+ "3600": 4.52599,
+ "3605": 4.51425,
+ "3610": 4.51376,
+ "3615": 4.54316,
+ "3620": 4.59194,
+ "3625": 4.52946,
+ "3630": 4.56676,
+ "3635": 4.48406,
+ "3640": 4.50421,
+ "3645": 4.54983,
+ "3650": 4.50196,
+ "3655": 4.52135,
+ "3660": 4.52808,
+ "3665": 4.56095,
+ "3670": 4.51415,
+ "3675": 4.52607,
+ "3680": 4.5435,
+ "3685": 4.46543,
+ "3690": 4.52062,
+ "3695": 4.47607,
+ "3700": 4.50777,
+ "3705": 4.48359,
+ "3710": 4.48846,
+ "3715": 4.50598,
+ "3720": 4.47607,
+ "3725": 4.45481,
+ "3730": 4.4601,
+ "3735": 4.48458,
+ "3740": 4.47289,
+ "3745": 4.45504,
+ "3750": 4.48879,
+ "3755": 4.46014,
+ "3760": 4.47506,
+ "3765": 4.44993,
+ "3770": 4.46205,
+ "3775": 4.43908,
+ "3780": 4.42966,
+ "3785": 4.48678,
+ "3790": 4.39654,
+ "3795": 4.46045,
+ "3800": 4.43674,
+ "3805": 4.44213,
+ "3810": 4.39976,
+ "3815": 4.4473,
+ "3820": 4.44276,
+ "3825": 4.45501,
+ "3830": 4.44273,
+ "3835": 4.40298,
+ "3840": 4.49328,
+ "3845": 4.45749,
+ "3850": 4.40007,
+ "3855": 4.43312,
+ "3860": 4.45561,
+ "3865": 4.42134,
+ "3870": 4.48195,
+ "3875": 4.38396,
+ "3880": 4.39491,
+ "3885": 4.41684,
+ "3890": 4.40954,
+ "3895": 4.35339,
+ "3900": 4.4063,
+ "3905": 4.38699,
+ "3910": 4.40407,
+ "3915": 4.39489,
+ "3920": 4.38651,
+ "3925": 4.37057,
+ "3930": 4.37398,
+ "3935": 4.39051,
+ "3940": 4.38497,
+ "3945": 4.37177,
+ "3950": 4.43238,
+ "3955": 4.36565,
+ "3960": 4.41432,
+ "3965": 4.42046,
+ "3970": 4.36551,
+ "3975": 4.37468,
+ "3980": 4.34326,
+ "3985": 4.38184,
+ "3990": 4.37963,
+ "3995": 4.42078,
+ "4000": 4.36024,
+ "4005": 4.34645,
+ "4010": 4.38286,
+ "4015": 4.37269,
+ "4020": 4.41279,
+ "4025": 4.36812,
+ "4030": 4.42395,
+ "4035": 4.38149,
+ "4040": 4.40781,
+ "4045": 4.38535,
+ "4050": 4.3795,
+ "4055": 4.39281,
+ "4060": 4.38352,
+ "4065": 4.38947,
+ "4070": 4.32369,
+ "4075": 4.34979,
+ "4080": 4.32923,
+ "4085": 4.36894,
+ "4090": 4.3462,
+ "4095": 4.32981,
+ "4100": 4.35298,
+ "4105": 4.3347,
+ "4110": 4.2994,
+ "4115": 4.37373,
+ "4120": 4.28732,
+ "4125": 4.27737,
+ "4130": 4.36254,
+ "4135": 4.34837,
+ "4140": 4.29126,
+ "4145": 4.30245,
+ "4150": 4.3457,
+ "4155": 4.27261,
+ "4160": 4.32995,
+ "4165": 4.35466,
+ "4170": 4.30413,
+ "4175": 4.30505,
+ "4180": 4.29956,
+ "4185": 4.29422,
+ "4190": 4.28592,
+ "4195": 4.28697,
+ "4200": 4.28797,
+ "4205": 4.33635,
+ "4210": 4.30661,
+ "4215": 4.32855,
+ "4220": 4.31236,
+ "4225": 4.29734,
+ "4230": 4.27864,
+ "4235": 4.32409,
+ "4240": 4.2759,
+ "4245": 4.28766,
+ "4250": 4.27499,
+ "4255": 4.28434,
+ "4260": 4.26186,
+ "4265": 4.27818,
+ "4270": 4.27634,
+ "4275": 4.33456,
+ "4280": 4.26537,
+ "4285": 4.29906,
+ "4290": 4.24487,
+ "4295": 4.27661,
+ "4300": 4.30294,
+ "4305": 4.26518,
+ "4310": 4.30662,
+ "4315": 4.28861,
+ "4320": 4.28476,
+ "4325": 4.29817,
+ "4330": 4.23675,
+ "4335": 4.27291,
+ "4340": 4.26865,
+ "4345": 4.21479,
+ "4350": 4.23512,
+ "4355": 4.30376,
+ "4360": 4.27825,
+ "4365": 4.27525,
+ "4370": 4.25084,
+ "4375": 4.21551,
+ "4380": 4.22684,
+ "4385": 4.20439,
+ "4390": 4.28187,
+ "4395": 4.24681,
+ "4400": 4.23724,
+ "4405": 4.20106,
+ "4410": 4.25487,
+ "4415": 4.24401,
+ "4420": 4.22273,
+ "4425": 4.26679,
+ "4430": 4.21675,
+ "4435": 4.26482,
+ "4440": 4.25723,
+ "4445": 4.21859,
+ "4450": 4.17676,
+ "4455": 4.23315,
+ "4460": 4.20922,
+ "4465": 4.23115,
+ "4470": 4.21206,
+ "4475": 4.23953,
+ "4480": 4.22361,
+ "4485": 4.20841,
+ "4490": 4.2096,
+ "4495": 4.15559,
+ "4500": 4.22915,
+ "4505": 4.20739,
+ "4510": 4.21121,
+ "4515": 4.16963,
+ "4520": 4.20106,
+ "4525": 4.16437,
+ "4530": 4.21718,
+ "4535": 4.17277,
+ "4540": 4.18397,
+ "4545": 4.20919,
+ "4550": 4.2448,
+ "4555": 4.18477,
+ "4560": 4.19275,
+ "4565": 4.12656,
+ "4570": 4.18372,
+ "4575": 4.16935,
+ "4580": 4.22466,
+ "4585": 4.18907,
+ "4590": 4.18389,
+ "4595": 4.14587,
+ "4600": 4.1407,
+ "4605": 4.1795,
+ "4610": 4.17449,
+ "4615": 4.21775,
+ "4620": 4.13467,
+ "4625": 4.16332,
+ "4630": 4.17539,
+ "4635": 4.15435,
+ "4640": 4.18919,
+ "4645": 4.18006,
+ "4650": 4.19936,
+ "4655": 4.16868,
+ "4660": 4.16002,
+ "4665": 4.16688,
+ "4670": 4.21001,
+ "4675": 4.15174,
+ "4680": 4.18475,
+ "4685": 4.17353,
+ "4690": 4.15051,
+ "4695": 4.15948,
+ "4700": 4.13735,
+ "4705": 4.11622,
+ "4710": 4.18087,
+ "4715": 4.16134,
+ "4720": 4.11941,
+ "4725": 4.09231,
+ "4730": 4.15058,
+ "4735": 4.07771,
+ "4740": 4.11592,
+ "4745": 4.15551,
+ "4750": 4.10969,
+ "4755": 4.16687,
+ "4760": 4.16809,
+ "4765": 4.11658,
+ "4770": 4.12778,
+ "4775": 4.12027,
+ "4780": 4.12633,
+ "4785": 4.10891,
+ "4790": 4.16532,
+ "4795": 4.14672,
+ "4800": 4.10754,
+ "4805": 4.15476,
+ "4810": 4.10939,
+ "4815": 4.14768,
+ "4820": 4.09231,
+ "4825": 4.14272,
+ "4830": 4.14261,
+ "4835": 4.1177,
+ "4840": 4.15277,
+ "4845": 4.08402,
+ "4850": 4.14394,
+ "4855": 4.14423,
+ "4860": 4.08178,
+ "4865": 4.11277,
+ "4870": 4.10254,
+ "4875": 4.14764,
+ "4880": 4.14249,
+ "4885": 4.10137,
+ "4890": 4.09605,
+ "4895": 4.0959,
+ "4900": 4.07116,
+ "4905": 4.06261,
+ "4910": 4.06331,
+ "4915": 4.11803,
+ "4920": 4.09413,
+ "4925": 4.05985,
+ "4930": 4.07424,
+ "4935": 4.09135,
+ "4940": 4.02534,
+ "4945": 4.10244,
+ "4950": 4.0455,
+ "4955": 4.12878,
+ "4960": 4.08806,
+ "4965": 4.09006,
+ "4970": 4.07403,
+ "4975": 4.09262,
+ "4980": 4.09725,
+ "4985": 4.09841,
+ "4990": 4.06183,
+ "4995": 4.09918,
+ "5000": 4.03214,
+ "5005": 4.08553,
+ "5010": 4.0788,
+ "5015": 4.04509,
+ "5020": 4.02894,
+ "5025": 4.03482,
+ "5030": 4.06552,
+ "5035": 4.05532,
+ "5040": 4.01952,
+ "5045": 4.08424,
+ "5050": 4.03521,
+ "5055": 4.06618,
+ "5060": 4.00432,
+ "5065": 4.06857,
+ "5070": 4.0433,
+ "5075": 4.09624,
+ "5080": 4.05027,
+ "5085": 4.06887,
+ "5090": 4.04905,
+ "5095": 4.01395,
+ "5100": 4.04967,
+ "5105": 4.05233,
+ "5110": 4.05931,
+ "5115": 4.0414,
+ "5120": 4.06967,
+ "5125": 4.03577,
+ "5130": 4.03483,
+ "5135": 4.02094,
+ "5140": 4.04231,
+ "5145": 4.03314,
+ "5150": 4.04344,
+ "5155": 4.0534,
+ "5160": 4.02967,
+ "5165": 4.07123,
+ "5170": 3.9389,
+ "5175": 4.04748,
+ "5180": 4.00618,
+ "5185": 4.03337,
+ "5190": 4.05508,
+ "5195": 4.02215,
+ "5200": 4.03921,
+ "5205": 4.07147,
+ "5210": 3.98414,
+ "5215": 3.99757,
+ "5220": 3.99577,
+ "5225": 3.99804,
+ "5230": 4.03847,
+ "5235": 4.00981,
+ "5240": 3.9979,
+ "5245": 4.01366,
+ "5250": 4.01843,
+ "5255": 4.00548,
+ "5260": 4.02342,
+ "5265": 3.99233,
+ "5270": 3.95683,
+ "5275": 3.97862,
+ "5280": 3.99164,
+ "5285": 4.01689,
+ "5290": 3.97747,
+ "5295": 3.9771,
+ "5300": 4.00016,
+ "5305": 3.97913,
+ "5310": 4.02214,
+ "5315": 3.97225,
+ "5320": 4.01253,
+ "5325": 4.03543,
+ "5330": 3.97415,
+ "5335": 3.99354,
+ "5340": 3.94345,
+ "5345": 3.9851,
+ "5350": 3.9992,
+ "5355": 3.99164,
+ "5360": 3.94133,
+ "5365": 3.96284,
+ "5370": 4.00116,
+ "5375": 3.96938,
+ "5380": 3.963,
+ "5385": 3.98081,
+ "5390": 3.97146,
+ "5395": 3.90554,
+ "5400": 3.99622,
+ "5405": 3.91829,
+ "5410": 4.00543,
+ "5415": 3.91874,
+ "5420": 3.95185,
+ "5425": 3.94555,
+ "5430": 3.94955,
+ "5435": 3.98061,
+ "5440": 3.92995,
+ "5445": 3.94184,
+ "5450": 3.95383,
+ "5455": 3.93579,
+ "5460": 3.95027,
+ "5465": 4.00936,
+ "5470": 3.97067,
+ "5475": 3.89788,
+ "5480": 3.97755,
+ "5485": 3.94379,
+ "5490": 3.96599,
+ "5495": 3.96804,
+ "5500": 3.92717,
+ "5505": 3.94186,
+ "5510": 3.97651,
+ "5515": 3.9537,
+ "5520": 3.93226,
+ "5525": 3.98634,
+ "5530": 3.93022,
+ "5535": 3.96437,
+ "5540": 3.93462,
+ "5545": 3.95406,
+ "5550": 3.94319,
+ "5555": 3.90819,
+ "5560": 3.91543,
+ "5565": 3.96007,
+ "5570": 3.91536,
+ "5575": 3.95021,
+ "5580": 3.92504,
+ "5585": 3.86536,
+ "5590": 3.93795,
+ "5595": 3.89768,
+ "5600": 3.94462,
+ "5605": 3.8512,
+ "5610": 3.94377,
+ "5615": 3.93762,
+ "5620": 3.94958,
+ "5625": 3.93243,
+ "5630": 3.92113,
+ "5635": 3.90565,
+ "5640": 3.92615,
+ "5645": 3.88875,
+ "5650": 3.86315,
+ "5655": 3.89109,
+ "5660": 3.88519,
+ "5665": 3.89765,
+ "5670": 3.88296,
+ "5675": 3.91884,
+ "5680": 3.88787,
+ "5685": 3.89873,
+ "5690": 3.89822,
+ "5695": 3.92451,
+ "5700": 3.8606,
+ "5705": 3.86255,
+ "5710": 3.84664,
+ "5715": 3.96895,
+ "5720": 3.91725,
+ "5725": 3.8693,
+ "5730": 3.92069,
+ "5735": 3.90056,
+ "5740": 3.89483,
+ "5745": 3.87681,
+ "5750": 3.89541,
+ "5755": 3.91484,
+ "5760": 3.90197,
+ "5765": 3.89494,
+ "5770": 3.92857,
+ "5775": 3.84786,
+ "5780": 3.88736,
+ "5785": 3.89066,
+ "5790": 3.90257,
+ "5795": 3.90574,
+ "5800": 3.84263,
+ "5805": 3.84579,
+ "5810": 3.90166,
+ "5815": 3.8634,
+ "5820": 3.8147,
+ "5825": 3.86523,
+ "5830": 3.82584,
+ "5835": 3.85893,
+ "5840": 3.86919,
+ "5845": 3.88532,
+ "5850": 3.88099,
+ "5855": 3.82093,
+ "5860": 3.83864,
+ "5865": 3.86844,
+ "5870": 3.83726,
+ "5875": 3.87227,
+ "5880": 3.85189,
+ "5885": 3.87279,
+ "5890": 3.87696,
+ "5895": 3.89694,
+ "5900": 3.83242,
+ "5905": 3.89414,
+ "5910": 3.86071,
+ "5915": 3.83071,
+ "5920": 3.86042,
+ "5925": 3.79873,
+ "5930": 3.85824,
+ "5935": 3.84311,
+ "5940": 3.87506,
+ "5945": 3.87656,
+ "5950": 3.85992,
+ "5955": 3.81155,
+ "5960": 3.88658,
+ "5965": 3.8306,
+ "5970": 3.87807,
+ "5975": 3.84371,
+ "5980": 3.92016,
+ "5985": 3.79693,
+ "5990": 3.88402,
+ "5995": 3.79859,
+ "6000": 3.83962,
+ "6005": 3.79486,
+ "6010": 3.82349,
+ "6015": 3.80423,
+ "6020": 3.8174,
+ "6025": 3.85525,
+ "6030": 3.80177,
+ "6035": 3.85269,
+ "6040": 3.82946,
+ "6045": 3.86164,
+ "6050": 3.83788,
+ "6055": 3.81734,
+ "6060": 3.83819,
+ "6065": 3.87244,
+ "6070": 3.82016,
+ "6075": 3.76801,
+ "6080": 3.83717,
+ "6085": 3.80225,
+ "6090": 3.83193,
+ "6095": 3.83606,
+ "6100": 3.79639,
+ "6105": 3.84811,
+ "6110": 3.7806,
+ "6115": 3.85331,
+ "6120": 3.82496,
+ "6125": 3.82958,
+ "6130": 3.82613,
+ "6135": 3.80162,
+ "6140": 3.79661,
+ "6145": 3.78575,
+ "6150": 3.83392,
+ "6155": 3.80949,
+ "6160": 3.77487,
+ "6165": 3.79719,
+ "6170": 3.79806,
+ "6175": 3.78178,
+ "6180": 3.78378,
+ "6185": 3.82201,
+ "6190": 3.78773,
+ "6195": 3.75708,
+ "6200": 3.7869,
+ "6205": 3.78813,
+ "6210": 3.74418,
+ "6215": 3.8011,
+ "6220": 3.79797,
+ "6225": 3.80018,
+ "6230": 3.74264,
+ "6235": 3.78241,
+ "6240": 3.71042,
+ "6245": 3.82146,
+ "6250": 3.78455,
+ "6255": 3.79631,
+ "6260": 3.77736,
+ "6265": 3.80229,
+ "6270": 3.73138,
+ "6275": 3.75647,
+ "6280": 3.77528,
+ "6285": 3.75539,
+ "6290": 3.77505,
+ "6295": 3.77516,
+ "6300": 3.78742,
+ "6305": 3.85774,
+ "6310": 3.74547,
+ "6315": 3.73733,
+ "6320": 3.79203,
+ "6325": 3.73129,
+ "6330": 3.79906,
+ "6335": 3.79191,
+ "6340": 3.74069,
+ "6345": 3.79986,
+ "6350": 3.74249,
+ "6355": 3.74952,
+ "6360": 3.73009,
+ "6365": 3.78734,
+ "6370": 3.78519,
+ "6375": 3.7601,
+ "6380": 3.77819,
+ "6385": 3.7957,
+ "6390": 3.75703,
+ "6395": 3.73316,
+ "6400": 3.7367,
+ "6405": 3.81687,
+ "6410": 3.80734,
+ "6415": 3.73801,
+ "6420": 3.79845,
+ "6425": 3.80523,
+ "6430": 3.78501,
+ "6435": 3.75264,
+ "6440": 3.7411,
+ "6445": 3.77775,
+ "6450": 3.71228,
+ "6455": 3.728,
+ "6460": 3.74876,
+ "6465": 3.78834,
+ "6470": 3.76403,
+ "6475": 3.75865,
+ "6480": 3.78647,
+ "6485": 3.73855,
+ "6490": 3.68749,
+ "6495": 3.79103,
+ "6500": 3.77664,
+ "6505": 3.70115,
+ "6510": 3.77389,
+ "6515": 3.79404,
+ "6520": 3.70573,
+ "6525": 3.77742,
+ "6530": 3.74188,
+ "6535": 3.73756,
+ "6540": 3.80312,
+ "6545": 3.73864,
+ "6550": 3.74471,
+ "6555": 3.72786,
+ "6560": 3.68579,
+ "6565": 3.68413,
+ "6570": 3.7224,
+ "6575": 3.67196,
+ "6580": 3.78612,
+ "6585": 3.73906,
+ "6590": 3.70459,
+ "6595": 3.72444,
+ "6600": 3.71209,
+ "6605": 3.69346,
+ "6610": 3.70414,
+ "6615": 3.73669,
+ "6620": 3.68756,
+ "6625": 3.70069,
+ "6630": 3.69833,
+ "6635": 3.73861,
+ "6640": 3.71398,
+ "6645": 3.72829,
+ "6650": 3.75571,
+ "6655": 3.67894,
+ "6660": 3.71096,
+ "6665": 3.73482,
+ "6670": 3.6944,
+ "6675": 3.71667,
+ "6680": 3.71302,
+ "6685": 3.74257,
+ "6690": 3.71844,
+ "6695": 3.73205,
+ "6700": 3.72197,
+ "6705": 3.70619,
+ "6710": 3.70406,
+ "6715": 3.66782,
+ "6720": 3.75504,
+ "6725": 3.72915,
+ "6730": 3.71835,
+ "6735": 3.71667,
+ "6740": 3.71534,
+ "6745": 3.69988,
+ "6750": 3.72043,
+ "6755": 3.67272,
+ "6760": 3.65936,
+ "6765": 3.71643,
+ "6770": 3.67255,
+ "6775": 3.72376,
+ "6780": 3.68245,
+ "6785": 3.68536,
+ "6790": 3.71384,
+ "6795": 3.67148,
+ "6800": 3.69396,
+ "6805": 3.69771,
+ "6810": 3.71265,
+ "6815": 3.63694,
+ "6820": 3.67686,
+ "6825": 3.70088,
+ "6830": 3.68539,
+ "6835": 3.6642,
+ "6840": 3.65647,
+ "6845": 3.72608,
+ "6850": 3.68406,
+ "6855": 3.71181,
+ "6860": 3.64768,
+ "6865": 3.71351,
+ "6870": 3.66999,
+ "6875": 3.67274,
+ "6880": 3.68049,
+ "6885": 3.65734,
+ "6890": 3.67264,
+ "6895": 3.65662,
+ "6900": 3.65928,
+ "6905": 3.66592,
+ "6910": 3.70476,
+ "6915": 3.7136,
+ "6920": 3.6661,
+ "6925": 3.66891,
+ "6930": 3.66285,
+ "6935": 3.60247,
+ "6940": 3.66945,
+ "6945": 3.65357,
+ "6950": 3.6589,
+ "6955": 3.65339,
+ "6960": 3.65603,
+ "6965": 3.69921,
+ "6970": 3.62428,
+ "6975": 3.70477,
+ "6980": 3.65994,
+ "6985": 3.66657,
+ "6990": 3.70972,
+ "6995": 3.68451,
+ "7000": 3.61695,
+ "7005": 3.69431,
+ "7010": 3.6693,
+ "7015": 3.65399,
+ "7020": 3.70045,
+ "7025": 3.6838,
+ "7030": 3.67768,
+ "7035": 3.63498,
+ "7040": 3.59358,
+ "7045": 3.67025,
+ "7050": 3.6966,
+ "7055": 3.62635,
+ "7060": 3.6705,
+ "7065": 3.71809,
+ "7070": 3.65183,
+ "7075": 3.65318,
+ "7080": 3.69492,
+ "7085": 3.61852,
+ "7090": 3.64168,
+ "7095": 3.61466,
+ "7100": 3.65947,
+ "7105": 3.59681,
+ "7110": 3.6625,
+ "7115": 3.61596,
+ "7120": 3.66477,
+ "7125": 3.61524,
+ "7130": 3.63053,
+ "7135": 3.63828,
+ "7140": 3.64391,
+ "7145": 3.65954,
+ "7150": 3.60584,
+ "7155": 3.6692,
+ "7160": 3.60068,
+ "7165": 3.62116,
+ "7170": 3.66015,
+ "7175": 3.62225,
+ "7180": 3.6522,
+ "7185": 3.68343,
+ "7190": 3.64063,
+ "7195": 3.6459,
+ "7200": 3.64761,
+ "7205": 3.63461,
+ "7210": 3.66809,
+ "7215": 3.64769,
+ "7220": 3.67114,
+ "7225": 3.64189,
+ "7230": 3.66268,
+ "7235": 3.62312,
+ "7240": 3.6252,
+ "7245": 3.6429,
+ "7250": 3.58208,
+ "7255": 3.60271,
+ "7260": 3.65787,
+ "7265": 3.5821,
+ "7270": 3.61807,
+ "7275": 3.62339,
+ "7280": 3.60523,
+ "7285": 3.62818,
+ "7290": 3.6513,
+ "7295": 3.64103,
+ "7300": 3.60063,
+ "7305": 3.60612,
+ "7310": 3.63678,
+ "7315": 3.65306,
+ "7320": 3.62759,
+ "7325": 3.63666,
+ "7330": 3.60198,
+ "7335": 3.6039,
+ "7340": 3.62339,
+ "7345": 3.5871,
+ "7350": 3.63882,
+ "7355": 3.6197,
+ "7360": 3.59785,
+ "7365": 3.61636,
+ "7370": 3.5988,
+ "7375": 3.57323,
+ "7380": 3.6272,
+ "7385": 3.64829,
+ "7390": 3.64001,
+ "7395": 3.58452,
+ "7400": 3.63813,
+ "7405": 3.62708,
+ "7410": 3.63913,
+ "7415": 3.62534,
+ "7420": 3.61324,
+ "7425": 3.6649,
+ "7430": 3.61126,
+ "7435": 3.59071,
+ "7440": 3.60931,
+ "7445": 3.58592,
+ "7450": 3.55157,
+ "7455": 3.62743,
+ "7460": 3.61303,
+ "7465": 3.61217,
+ "7470": 3.61701,
+ "7475": 3.62026,
+ "7480": 3.59402,
+ "7485": 3.55842,
+ "7490": 3.5522,
+ "7495": 3.56333,
+ "7500": 3.59588,
+ "7505": 3.57473,
+ "7510": 3.53641,
+ "7515": 3.59655,
+ "7520": 3.59099,
+ "7525": 3.54802,
+ "7530": 3.59253,
+ "7535": 3.60312,
+ "7540": 3.58688,
+ "7545": 3.62561,
+ "7550": 3.63546,
+ "7555": 3.56542,
+ "7560": 3.58033,
+ "7565": 3.57709,
+ "7570": 3.58608,
+ "7575": 3.55399,
+ "7580": 3.6041,
+ "7585": 3.58238,
+ "7590": 3.58009,
+ "7595": 3.63911,
+ "7600": 3.59017,
+ "7605": 3.57713,
+ "7610": 3.56536,
+ "7615": 3.56396,
+ "7620": 3.54902,
+ "7625": 3.60282,
+ "7630": 3.58504,
+ "7635": 3.57229,
+ "7640": 3.56646,
+ "7645": 3.60242,
+ "7650": 3.60482,
+ "7655": 3.64727,
+ "7660": 3.51017,
+ "7665": 3.58631,
+ "7670": 3.57871,
+ "7675": 3.56493,
+ "7680": 3.55839,
+ "7685": 3.6249,
+ "7690": 3.56788,
+ "7695": 3.55332,
+ "7700": 3.61199,
+ "7705": 3.56821,
+ "7710": 3.60454,
+ "7715": 3.55825,
+ "7720": 3.63785,
+ "7725": 3.53221,
+ "7730": 3.55594,
+ "7735": 3.5893,
+ "7740": 3.56288,
+ "7745": 3.56571,
+ "7750": 3.55591,
+ "7755": 3.56977,
+ "7760": 3.54431,
+ "7765": 3.56223,
+ "7770": 3.57886,
+ "7775": 3.55066,
+ "7780": 3.53511,
+ "7785": 3.55692,
+ "7790": 3.55343,
+ "7795": 3.57112,
+ "7800": 3.5566,
+ "7805": 3.56123,
+ "7810": 3.58814,
+ "7815": 3.55731,
+ "7820": 3.55707,
+ "7825": 3.59912,
+ "7830": 3.5771,
+ "7835": 3.50576,
+ "7840": 3.60205,
+ "7845": 3.53506,
+ "7850": 3.49175,
+ "7855": 3.54244,
+ "7860": 3.52685,
+ "7865": 3.58474,
+ "7870": 3.52213,
+ "7875": 3.53946,
+ "7880": 3.55231,
+ "7885": 3.54064,
+ "7890": 3.58902,
+ "7895": 3.57418,
+ "7900": 3.58584,
+ "7905": 3.54428,
+ "7910": 3.56312,
+ "7915": 3.56255,
+ "7920": 3.57002,
+ "7925": 3.55042,
+ "7930": 3.58165,
+ "7935": 3.53937,
+ "7940": 3.5918,
+ "7945": 3.60828,
+ "7950": 3.51984,
+ "7955": 3.52849,
+ "7960": 3.5133,
+ "7965": 3.49852,
+ "7970": 3.50421,
+ "7975": 3.53912,
+ "7980": 3.54924,
+ "7985": 3.52434,
+ "7990": 3.52539,
+ "7995": 3.50061,
+ "8000": 3.56192,
+ "8005": 3.52714,
+ "8010": 3.51753,
+ "8015": 3.51556,
+ "8020": 3.51141,
+ "8025": 3.49693,
+ "8030": 3.52126,
+ "8035": 3.51508,
+ "8040": 3.50332,
+ "8045": 3.55986,
+ "8050": 3.55864,
+ "8055": 3.52971,
+ "8060": 3.55293,
+ "8065": 3.52985,
+ "8070": 3.51808,
+ "8075": 3.50892,
+ "8080": 3.55628,
+ "8085": 3.51194,
+ "8090": 3.51775,
+ "8095": 3.54559,
+ "8100": 3.50068,
+ "8105": 3.53339,
+ "8110": 3.52868,
+ "8115": 3.49254,
+ "8120": 3.50868,
+ "8125": 3.54598,
+ "8130": 3.50866,
+ "8135": 3.52095,
+ "8140": 3.50367,
+ "8145": 3.48666,
+ "8150": 3.505,
+ "8155": 3.49287,
+ "8160": 3.54304,
+ "8165": 3.52358,
+ "8170": 3.49625,
+ "8175": 3.48722,
+ "8180": 3.55617,
+ "8185": 3.52955,
+ "8190": 3.56841,
+ "8195": 3.53371,
+ "8200": 3.50422,
+ "8205": 3.51275,
+ "8210": 3.51784,
+ "8215": 3.53903,
+ "8220": 3.50159,
+ "8225": 3.49459,
+ "8230": 3.51811,
+ "8235": 3.53831,
+ "8240": 3.52124,
+ "8245": 3.51967,
+ "8250": 3.54947,
+ "8255": 3.5007,
+ "8260": 3.51175,
+ "8265": 3.502,
+ "8270": 3.51303,
+ "8275": 3.50068,
+ "8280": 3.48435,
+ "8285": 3.50904,
+ "8290": 3.51291,
+ "8295": 3.48207,
+ "8300": 3.50171,
+ "8305": 3.51977,
+ "8310": 3.51438,
+ "8315": 3.4843,
+ "8320": 3.5136,
+ "8325": 3.46288,
+ "8330": 3.42582,
+ "8335": 3.49746,
+ "8340": 3.52198,
+ "8345": 3.48183,
+ "8350": 3.49584,
+ "8355": 3.52609,
+ "8360": 3.49915,
+ "8365": 3.5191,
+ "8370": 3.51367,
+ "8375": 3.46935,
+ "8380": 3.46864,
+ "8385": 3.51324,
+ "8390": 3.47824,
+ "8395": 3.50843,
+ "8400": 3.47721,
+ "8405": 3.50395,
+ "8410": 3.56128,
+ "8415": 3.4701,
+ "8420": 3.43597,
+ "8425": 3.51509,
+ "8430": 3.52045,
+ "8435": 3.45584,
+ "8440": 3.53794,
+ "8445": 3.52173,
+ "8450": 3.49372,
+ "8455": 3.51219,
+ "8460": 3.5162,
+ "8465": 3.45483,
+ "8470": 3.47421,
+ "8475": 3.53118,
+ "8480": 3.45835,
+ "8485": 3.47726,
+ "8490": 3.46648,
+ "8495": 3.4642,
+ "8500": 3.51131,
+ "8505": 3.44858,
+ "8510": 3.52424,
+ "8515": 3.47245,
+ "8520": 3.47737,
+ "8525": 3.40609,
+ "8530": 3.48639,
+ "8535": 3.50673,
+ "8540": 3.45843,
+ "8545": 3.48269,
+ "8550": 3.45363,
+ "8555": 3.5202,
+ "8560": 3.51844,
+ "8565": 3.47223,
+ "8570": 3.47333,
+ "8575": 3.44548,
+ "8580": 3.49139,
+ "8585": 3.5119,
+ "8590": 3.50092,
+ "8595": 3.5088,
+ "8600": 3.48749,
+ "8605": 3.47306,
+ "8610": 3.47934,
+ "8615": 3.47701,
+ "8620": 3.44726,
+ "8625": 3.47096,
+ "8630": 3.48255,
+ "8635": 3.45718,
+ "8640": 3.4445,
+ "8645": 3.51366,
+ "8650": 3.44299,
+ "8655": 3.48614,
+ "8660": 3.49462,
+ "8665": 3.47167,
+ "8670": 3.48885,
+ "8675": 3.46297,
+ "8680": 3.44949,
+ "8685": 3.46355,
+ "8690": 3.49623,
+ "8695": 3.49778,
+ "8700": 3.46989,
+ "8705": 3.43596,
+ "8710": 3.48712,
+ "8715": 3.43746,
+ "8720": 3.51148,
+ "8725": 3.47121,
+ "8730": 3.46605,
+ "8735": 3.4937,
+ "8740": 3.44483,
+ "8745": 3.48474,
+ "8750": 3.49043,
+ "8755": 3.45251,
+ "8760": 3.46838,
+ "8765": 3.42556,
+ "8770": 3.49265,
+ "8775": 3.4545,
+ "8780": 3.44091,
+ "8785": 3.46068,
+ "8790": 3.44079,
+ "8795": 3.48072,
+ "8800": 3.45057,
+ "8805": 3.41755,
+ "8810": 3.43437,
+ "8815": 3.45667,
+ "8820": 3.42235,
+ "8825": 3.4544,
+ "8830": 3.42832,
+ "8835": 3.40606,
+ "8840": 3.42352,
+ "8845": 3.44346,
+ "8850": 3.46501,
+ "8855": 3.44759,
+ "8860": 3.51957,
+ "8865": 3.45373,
+ "8870": 3.43267,
+ "8875": 3.4413,
+ "8880": 3.43702,
+ "8885": 3.4346,
+ "8890": 3.45793,
+ "8895": 3.43743,
+ "8900": 3.46416,
+ "8905": 3.45271,
+ "8910": 3.4366,
+ "8915": 3.42601,
+ "8920": 3.41867,
+ "8925": 3.49267,
+ "8930": 3.47384,
+ "8935": 3.48902,
+ "8940": 3.46114,
+ "8945": 3.46168,
+ "8950": 3.44204,
+ "8955": 3.4328,
+ "8960": 3.42249,
+ "8965": 3.44097,
+ "8970": 3.45801,
+ "8975": 3.4084,
+ "8980": 3.40537,
+ "8985": 3.43342,
+ "8990": 3.48401,
+ "8995": 3.4565,
+ "9000": 3.40657,
+ "9005": 3.44912,
+ "9010": 3.50087,
+ "9015": 3.40291,
+ "9020": 3.42274,
+ "9025": 3.43051,
+ "9030": 3.45633,
+ "9035": 3.36616,
+ "9040": 3.44385,
+ "9045": 3.43968,
+ "9050": 3.47722,
+ "9055": 3.3902,
+ "9060": 3.4798,
+ "9065": 3.49791,
+ "9070": 3.43128,
+ "9075": 3.45851,
+ "9080": 3.4553,
+ "9085": 3.45875,
+ "9090": 3.4505,
+ "9095": 3.40593,
+ "9100": 3.40925,
+ "9105": 3.39938,
+ "9110": 3.44282,
+ "9115": 3.45157,
+ "9120": 3.50457,
+ "9125": 3.4285,
+ "9130": 3.42433,
+ "9135": 3.44407,
+ "9140": 3.46108,
+ "9145": 3.40791,
+ "9150": 3.42855,
+ "9155": 3.4367,
+ "9160": 3.43684,
+ "9165": 3.44331,
+ "9170": 3.46153,
+ "9175": 3.39787,
+ "9180": 3.43851,
+ "9185": 3.39535,
+ "9190": 3.45675,
+ "9195": 3.41912,
+ "9200": 3.43243,
+ "9205": 3.40951,
+ "9210": 3.44383,
+ "9215": 3.38018,
+ "9220": 3.41078,
+ "9225": 3.43523,
+ "9230": 3.36271,
+ "9235": 3.37863,
+ "9240": 3.40699,
+ "9245": 3.39148,
+ "9250": 3.39863,
+ "9255": 3.40849,
+ "9260": 3.38169,
+ "9265": 3.42781,
+ "9270": 3.39293,
+ "9275": 3.41312,
+ "9280": 3.42972,
+ "9285": 3.4263,
+ "9290": 3.44415,
+ "9295": 3.43147,
+ "9300": 3.38278,
+ "9305": 3.41082,
+ "9310": 3.4002,
+ "9315": 3.36923,
+ "9320": 3.36364,
+ "9325": 3.40956,
+ "9330": 3.4626,
+ "9335": 3.37303,
+ "9340": 3.46017,
+ "9345": 3.44958,
+ "9350": 3.41499,
+ "9355": 3.38024,
+ "9360": 3.40184,
+ "9365": 3.39776,
+ "9370": 3.44973,
+ "9375": 3.41205,
+ "9380": 3.3536,
+ "9385": 3.42216,
+ "9390": 3.42841,
+ "9395": 3.43984,
+ "9400": 3.40291,
+ "9405": 3.38475,
+ "9410": 3.42336,
+ "9415": 3.41185,
+ "9420": 3.38873,
+ "9425": 3.40875,
+ "9430": 3.38022,
+ "9435": 3.40153,
+ "9440": 3.38713,
+ "9445": 3.3853,
+ "9450": 3.38099,
+ "9455": 3.38817,
+ "9460": 3.44943,
+ "9465": 3.45068,
+ "9470": 3.39331,
+ "9475": 3.44405,
+ "9480": 3.39379,
+ "9485": 3.38787,
+ "9490": 3.39955,
+ "9495": 3.42976,
+ "9500": 3.39511,
+ "9505": 3.3648,
+ "9510": 3.40319,
+ "9515": 3.39644,
+ "9520": 3.41784,
+ "9525": 3.38811,
+ "9530": 3.38753,
+ "9535": 3.40673
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 9535,
+ "step_interval": 5,
+ "values": {
+ "1": "nan",
+ "5": 7.63571,
+ "10": 7.60248,
+ "15": 7.60059,
+ "20": 7.8579,
+ "25": 7.56043,
+ "30": 7.58071,
+ "35": 7.55957,
+ "40": 7.60999,
+ "45": 7.5594,
+ "50": 7.57568,
+ "55": 113.41206,
+ "60": 7.54485,
+ "65": 7.55365,
+ "70": 7.57101,
+ "75": 7.59674,
+ "80": 7.67616,
+ "85": 7.59559,
+ "90": 7.59964,
+ "95": 7.61129,
+ "100": 7.72127,
+ "105": 7.75361,
+ "110": 7.70323,
+ "115": 7.75888,
+ "120": 7.82366,
+ "125": 7.83547,
+ "130": 7.92559,
+ "135": 7.94501,
+ "140": 7.95728,
+ "145": 7.99371,
+ "150": 8.1368,
+ "155": 8.14086,
+ "160": 8.19943,
+ "165": 8.29726,
+ "170": 8.32617,
+ "175": 8.28395,
+ "180": 8.39876,
+ "185": 27.81327,
+ "190": 8.29333,
+ "195": 8.23506,
+ "200": 8.20625,
+ "205": 8.28595,
+ "210": 8.18291,
+ "215": 8.14145,
+ "220": 8.26046,
+ "225": 8.31268,
+ "230": 8.12702,
+ "235": 8.11373,
+ "240": 8.11381,
+ "245": 8.09969,
+ "250": 8.10156,
+ "255": 8.13233,
+ "260": 8.08119,
+ "265": 8.1123,
+ "270": 12.54723,
+ "275": 7.97042,
+ "280": 8.04516,
+ "285": 7.96989,
+ "290": 7.95572,
+ "295": 7.97123,
+ "300": 7.96778,
+ "305": 8.05275,
+ "310": 7.94442,
+ "315": 7.93712,
+ "320": 7.91242,
+ "325": 22.89971,
+ "330": 7.88921,
+ "335": 7.85024,
+ "340": 7.83743,
+ "345": 7.86504,
+ "350": 7.80315,
+ "355": 7.81569,
+ "360": 7.79258,
+ "365": 14.97893,
+ "370": 7.74557,
+ "375": 7.71598,
+ "380": 7.71617,
+ "385": 7.72421,
+ "390": 7.76175,
+ "395": 7.72591,
+ "400": 7.70221,
+ "405": 7.71975,
+ "410": 7.68924,
+ "415": 7.65985,
+ "420": 7.67022,
+ "425": 7.70637,
+ "430": 8.12779,
+ "435": 7.67647,
+ "440": 7.65685,
+ "445": 7.73773,
+ "450": 7.64831,
+ "455": 7.63814,
+ "460": 7.68705,
+ "465": 8.1685,
+ "470": 7.65725,
+ "475": 7.76604,
+ "480": 7.66427,
+ "485": 11.91771,
+ "490": 7.64901,
+ "495": 7.66169,
+ "500": 7.67552,
+ "505": 7.69539,
+ "510": 14.23388,
+ "515": 7.6726,
+ "520": 7.74989,
+ "525": 7.67324,
+ "530": 7.65863,
+ "535": 7.65508,
+ "540": 7.68234,
+ "545": 7.70565,
+ "550": 22.49964,
+ "555": 7.68128,
+ "560": 7.65807,
+ "565": 7.70583,
+ "570": 7.7129,
+ "575": 12.98028,
+ "580": 7.67495,
+ "585": 7.70463,
+ "590": 7.66657,
+ "595": 7.68539,
+ "600": 7.69206,
+ "605": 7.67827,
+ "610": 7.64609,
+ "615": 9.76686,
+ "620": 7.67409,
+ "625": 7.6579,
+ "630": 7.74727,
+ "635": 7.66807,
+ "640": 7.67486,
+ "645": 7.66428,
+ "650": 7.63101,
+ "655": 10.67056,
+ "660": 7.64333,
+ "665": 7.63964,
+ "670": 7.65723,
+ "675": 7.64705,
+ "680": 11.76242,
+ "685": 10.29346,
+ "690": 7.68523,
+ "695": 7.65414,
+ "700": 7.61913,
+ "705": 7.65524,
+ "710": 7.60757,
+ "715": 7.64966,
+ "720": 7.59714,
+ "725": 10.87303,
+ "730": 7.60222,
+ "735": 7.64107,
+ "740": 7.63246,
+ "745": 20.09047,
+ "750": 7.63938,
+ "755": 7.93368,
+ "760": 7.63051,
+ "765": 7.64769,
+ "770": 7.61749,
+ "775": 7.66937,
+ "780": 7.64204,
+ "785": 7.62773,
+ "790": 7.63075,
+ "795": 7.63695,
+ "800": 7.62237,
+ "805": 7.62311,
+ "810": 7.63187,
+ "815": 12.24189,
+ "820": 7.70076,
+ "825": 7.65444,
+ "830": 7.65498,
+ "835": 7.64201,
+ "840": 7.64897,
+ "845": 7.63626,
+ "850": 15.67102,
+ "855": 7.61591,
+ "860": 7.63434,
+ "865": 7.64367,
+ "870": 7.67836,
+ "875": 7.62674,
+ "880": 7.62936,
+ "885": 7.62153,
+ "890": 7.6535,
+ "895": 7.69326,
+ "900": 7.6335,
+ "905": 7.66212,
+ "910": 7.68466,
+ "915": 7.66356,
+ "920": 7.63939,
+ "925": 7.61784,
+ "930": 7.62596,
+ "935": 7.65905,
+ "940": 7.67688,
+ "945": 7.68376,
+ "950": 7.66639,
+ "955": 7.6102,
+ "960": 7.63057,
+ "965": 7.69547,
+ "970": 7.63224,
+ "975": 7.62427,
+ "980": 7.64126,
+ "985": 7.63024,
+ "990": 7.65079,
+ "995": 7.81978,
+ "1000": 7.67793,
+ "1005": 7.79115,
+ "1010": 7.71162,
+ "1015": 7.66675,
+ "1020": 18.72203,
+ "1025": 7.6736,
+ "1030": 7.63985,
+ "1035": 7.88797,
+ "1040": 7.65013,
+ "1045": 7.74366,
+ "1050": 7.77211,
+ "1055": 7.58451,
+ "1060": 7.68763,
+ "1065": 7.68339,
+ "1070": 7.67615,
+ "1075": 7.65744,
+ "1080": 7.65653,
+ "1085": 7.67378,
+ "1090": 7.65738,
+ "1095": 7.67888,
+ "1100": 7.638,
+ "1105": 7.63902,
+ "1110": 7.68908,
+ "1115": 7.69012,
+ "1120": 7.658,
+ "1125": 7.6355,
+ "1130": 7.71095,
+ "1135": 7.66302,
+ "1140": 7.65349,
+ "1145": 19.42355,
+ "1150": 13.75872,
+ "1155": 7.62456,
+ "1160": 7.65058,
+ "1165": 7.65505,
+ "1170": 7.63813,
+ "1175": 7.68731,
+ "1180": 7.6526,
+ "1185": 7.67226,
+ "1190": 47.99075,
+ "1195": 7.65099,
+ "1200": 7.69431,
+ "1205": 7.69614,
+ "1210": 11.43591,
+ "1215": 7.6818,
+ "1220": 7.63789,
+ "1225": 7.67533,
+ "1230": 7.64747,
+ "1235": 7.65168,
+ "1240": 7.67319,
+ "1245": 19.5857,
+ "1250": 16.52076,
+ "1255": 7.78339,
+ "1260": 7.73213,
+ "1265": 7.66881,
+ "1270": 7.72911,
+ "1275": 7.63717,
+ "1280": 7.79793,
+ "1285": 7.65871,
+ "1290": 7.6902,
+ "1295": 7.67193,
+ "1300": 7.62774,
+ "1305": 7.65611,
+ "1310": 7.65536,
+ "1315": 7.65891,
+ "1320": 7.6713,
+ "1325": 7.66388,
+ "1330": 7.65778,
+ "1335": 7.65652,
+ "1340": 7.65999,
+ "1345": 7.63213,
+ "1350": 7.66039,
+ "1355": 7.66835,
+ "1360": 7.6783,
+ "1365": 7.66287,
+ "1370": 7.67746,
+ "1375": 7.69075,
+ "1380": 7.65712,
+ "1385": 7.65631,
+ "1390": 7.67187,
+ "1395": 7.63339,
+ "1400": 7.69185,
+ "1405": 7.64865,
+ "1410": 7.67646,
+ "1415": 7.68597,
+ "1420": 7.65193,
+ "1425": 7.67605,
+ "1430": 7.67305,
+ "1435": 7.65757,
+ "1440": 7.69733,
+ "1445": 7.66306,
+ "1450": 7.69468,
+ "1455": 7.66435,
+ "1460": 7.64866,
+ "1465": 7.71247,
+ "1470": 7.6455,
+ "1475": 7.65576,
+ "1480": 7.66266,
+ "1485": 7.6815,
+ "1490": 7.65979,
+ "1495": 7.674,
+ "1500": 7.66163,
+ "1505": 8.4307,
+ "1510": 7.65753,
+ "1515": 7.64484,
+ "1520": 7.66859,
+ "1525": 7.6599,
+ "1530": 7.68802,
+ "1535": 7.64915,
+ "1540": 7.66011,
+ "1545": 7.63956,
+ "1550": 7.65619,
+ "1555": 7.7467,
+ "1560": 7.68607,
+ "1565": 7.68614,
+ "1570": 7.66686,
+ "1575": 7.64766,
+ "1580": 7.63928,
+ "1585": 7.66306,
+ "1590": 7.68344,
+ "1595": 7.6779,
+ "1600": 7.66017,
+ "1605": 7.67771,
+ "1610": 7.67648,
+ "1615": 7.6636,
+ "1620": 10.02574,
+ "1625": 7.67478,
+ "1630": 7.6908,
+ "1635": 7.67477,
+ "1640": 7.64287,
+ "1645": 7.6608,
+ "1650": 7.66025,
+ "1655": 7.66112,
+ "1660": 73.64888,
+ "1665": 7.64598,
+ "1670": 21.00186,
+ "1675": 7.67517,
+ "1680": 7.68626,
+ "1685": 7.71692,
+ "1690": 7.66889,
+ "1695": 7.70903,
+ "1700": 7.67766,
+ "1705": 15.49722,
+ "1710": 7.67505,
+ "1715": 7.69821,
+ "1720": 7.6821,
+ "1725": 7.72001,
+ "1730": 7.66687,
+ "1735": 7.6862,
+ "1740": 7.65111,
+ "1745": 7.68892,
+ "1750": 7.67065,
+ "1755": 8.77332,
+ "1760": 7.70004,
+ "1765": 7.70035,
+ "1770": 7.6861,
+ "1775": 20.16274,
+ "1780": 7.65714,
+ "1785": 7.68878,
+ "1790": 7.66701,
+ "1795": 7.71136,
+ "1800": 7.67282,
+ "1805": 7.67363,
+ "1810": 7.71413,
+ "1815": 7.65806,
+ "1820": 7.70723,
+ "1825": 7.7037,
+ "1830": 7.68082,
+ "1835": 7.69413,
+ "1840": 7.68251,
+ "1845": 7.65467,
+ "1850": 7.65665,
+ "1855": 7.66234,
+ "1860": 7.71155,
+ "1865": 7.7125,
+ "1870": 7.68713,
+ "1875": 7.68269,
+ "1880": 7.63473,
+ "1885": 7.72083,
+ "1890": 7.67216,
+ "1895": 7.67823,
+ "1900": 7.70191,
+ "1905": 7.68951,
+ "1910": 7.63163,
+ "1915": 7.66052,
+ "1920": 7.63215,
+ "1925": 7.65734,
+ "1930": 7.62867,
+ "1935": 7.63111,
+ "1940": 7.63333,
+ "1945": 7.68277,
+ "1950": 7.69301,
+ "1955": 7.65643,
+ "1960": 7.64064,
+ "1965": 7.67833,
+ "1970": 7.6552,
+ "1975": 7.62405,
+ "1980": 7.6324,
+ "1985": 7.65736,
+ "1990": 7.64673,
+ "1995": 7.62618,
+ "2000": 7.65073,
+ "2005": 7.63687,
+ "2010": 7.69791,
+ "2015": 7.61614,
+ "2020": 7.66593,
+ "2025": 7.6307,
+ "2030": 11.59374,
+ "2035": 7.63294,
+ "2040": 7.61089,
+ "2045": 7.62554,
+ "2050": 7.65308,
+ "2055": 7.65026,
+ "2060": 7.63079,
+ "2065": 7.62294,
+ "2070": 26.60238,
+ "2075": 7.65314,
+ "2080": 7.63414,
+ "2085": 7.65491,
+ "2090": 7.65736,
+ "2095": 7.65061,
+ "2100": 19.73258,
+ "2105": 7.67651,
+ "2110": 7.65446,
+ "2115": 7.62848,
+ "2120": 7.64628,
+ "2125": 7.64508,
+ "2130": 7.69211,
+ "2135": 7.70666,
+ "2140": 7.6197,
+ "2145": 7.66385,
+ "2150": 7.67493,
+ "2155": 7.64216,
+ "2160": 7.62842,
+ "2165": 7.65879,
+ "2170": 7.62991,
+ "2175": 7.61843,
+ "2180": 7.65472,
+ "2185": 15.12689,
+ "2190": 7.62412,
+ "2195": 7.62587,
+ "2200": 7.62339,
+ "2205": 7.65002,
+ "2210": 7.64636,
+ "2215": 7.65213,
+ "2220": 17.73433,
+ "2225": 7.65981,
+ "2230": 7.70716,
+ "2235": 7.67016,
+ "2240": 7.66867,
+ "2245": 7.64808,
+ "2250": 7.65558,
+ "2255": 7.59272,
+ "2260": 7.80462,
+ "2265": 7.62316,
+ "2270": 7.6566,
+ "2275": 7.64638,
+ "2280": 7.63988,
+ "2285": 7.65123,
+ "2290": 7.64062,
+ "2295": 17.88613,
+ "2300": 7.65668,
+ "2305": 7.67342,
+ "2310": 7.66926,
+ "2315": 7.62473,
+ "2320": 7.64602,
+ "2325": 7.63334,
+ "2330": 7.65095,
+ "2335": 7.63677,
+ "2340": 7.66131,
+ "2345": 7.66693,
+ "2350": 7.66064,
+ "2355": 53.17307,
+ "2360": 7.64107,
+ "2365": 7.62645,
+ "2370": 7.64207,
+ "2375": 7.65615,
+ "2380": 7.63502,
+ "2385": 7.6348,
+ "2390": 7.62523,
+ "2395": 7.60746,
+ "2400": 7.6581,
+ "2405": 7.65348,
+ "2410": 7.63399,
+ "2415": 7.60524,
+ "2420": 7.64958,
+ "2425": 7.65158,
+ "2430": 7.64938,
+ "2435": 7.64148,
+ "2440": 7.63409,
+ "2445": 11.5194,
+ "2450": 7.62915,
+ "2455": 7.64594,
+ "2460": 7.70762,
+ "2465": 7.67521,
+ "2470": 7.63065,
+ "2475": 7.66102,
+ "2480": 7.61188,
+ "2485": 7.65817,
+ "2490": 7.66834,
+ "2495": 7.62545,
+ "2500": 7.66207,
+ "2505": 7.6935,
+ "2510": 7.61555,
+ "2515": 7.59989,
+ "2520": 7.63385,
+ "2525": 7.66489,
+ "2530": 7.61639,
+ "2535": 7.66299,
+ "2540": 7.69087,
+ "2545": 26.67924,
+ "2550": 7.69245,
+ "2555": 7.62259,
+ "2560": 7.69273,
+ "2565": 7.62099,
+ "2570": 7.66306,
+ "2575": 7.63754,
+ "2580": 7.65942,
+ "2585": 7.63446,
+ "2590": 22.6069,
+ "2595": 7.61337,
+ "2600": 7.65695,
+ "2605": 7.73253,
+ "2610": 11.40212,
+ "2615": 7.65704,
+ "2620": 7.63853,
+ "2625": 7.61891,
+ "2630": 7.64097,
+ "2635": 7.63231,
+ "2640": 7.62386,
+ "2645": 7.66079,
+ "2650": 7.67268,
+ "2655": 7.6449,
+ "2660": 7.65795,
+ "2665": 7.69452,
+ "2670": 7.67986,
+ "2675": 7.65479,
+ "2680": 7.64669,
+ "2685": 7.63843,
+ "2690": 7.68683,
+ "2695": 7.64869,
+ "2700": 7.66944,
+ "2705": 7.66006,
+ "2710": 7.68048,
+ "2715": 7.66861,
+ "2720": 7.65685,
+ "2725": 7.6388,
+ "2730": 7.70132,
+ "2735": 7.65686,
+ "2740": 7.66162,
+ "2745": 7.61126,
+ "2750": 7.65927,
+ "2755": 7.64398,
+ "2760": 7.71338,
+ "2765": 7.6939,
+ "2770": 7.61678,
+ "2775": 7.68293,
+ "2780": 7.66333,
+ "2785": 7.63331,
+ "2790": 7.65288,
+ "2795": 7.65726,
+ "2800": 7.67493,
+ "2805": 7.64697,
+ "2810": 7.64906,
+ "2815": 7.65438,
+ "2820": 7.65187,
+ "2825": 7.6568,
+ "2830": 7.67584,
+ "2835": 7.65594,
+ "2840": 7.69515,
+ "2845": 7.67534,
+ "2850": 7.71364,
+ "2855": 7.6355,
+ "2860": 7.63384,
+ "2865": 7.64765,
+ "2870": 7.63896,
+ "2875": 7.6718,
+ "2880": 7.6996,
+ "2885": 7.6501,
+ "2890": 7.64396,
+ "2895": 7.65189,
+ "2900": 7.66363,
+ "2905": 7.63918,
+ "2910": 7.6584,
+ "2915": 7.6285,
+ "2920": 7.66655,
+ "2925": 7.63673,
+ "2930": 7.68269,
+ "2935": 7.61018,
+ "2940": 7.63732,
+ "2945": 7.66545,
+ "2950": 7.64084,
+ "2955": 7.66845,
+ "2960": 7.69649,
+ "2965": 7.64383,
+ "2970": 7.69607,
+ "2975": 7.62653,
+ "2980": 7.68481,
+ "2985": 7.65846,
+ "2990": 7.63769,
+ "2995": 7.65151,
+ "3000": 7.67066,
+ "3005": 7.68127,
+ "3010": 7.72784,
+ "3015": 7.67541,
+ "3020": 8.84047,
+ "3025": 7.66716,
+ "3030": 7.65122,
+ "3035": 7.66783,
+ "3040": 7.6411,
+ "3045": 7.66704,
+ "3050": 7.6805,
+ "3055": 7.66545,
+ "3060": 7.64951,
+ "3065": 7.64776,
+ "3070": 7.64527,
+ "3075": 7.68545,
+ "3080": 7.65609,
+ "3085": 7.66405,
+ "3090": 7.69109,
+ "3095": 7.66122,
+ "3100": 7.64723,
+ "3105": 7.62562,
+ "3110": 7.66217,
+ "3115": 7.64795,
+ "3120": 7.67503,
+ "3125": 7.65153,
+ "3130": 7.70494,
+ "3135": 7.6376,
+ "3140": 7.67789,
+ "3145": 7.62435,
+ "3150": 7.63443,
+ "3155": 7.6778,
+ "3160": 7.66811,
+ "3165": 7.65503,
+ "3170": 7.66287,
+ "3175": 7.66545,
+ "3180": "nan",
+ "3185": 7.59787,
+ "3190": 7.63441,
+ "3195": 7.66096,
+ "3200": 7.67517,
+ "3205": 7.6861,
+ "3210": 7.68266,
+ "3215": 7.64226,
+ "3220": 7.65583,
+ "3225": 7.64587,
+ "3230": 7.63476,
+ "3235": 7.69196,
+ "3240": 7.65363,
+ "3245": 7.62974,
+ "3250": 7.67173,
+ "3255": 7.59755,
+ "3260": 7.67828,
+ "3265": 7.70542,
+ "3270": 7.64597,
+ "3275": 7.65586,
+ "3280": 7.65621,
+ "3285": 7.64418,
+ "3290": 7.71645,
+ "3295": 7.65419,
+ "3300": 7.61136,
+ "3305": 7.64845,
+ "3310": 7.63743,
+ "3315": 7.63886,
+ "3320": 7.62067,
+ "3325": 7.66155,
+ "3330": 7.59679,
+ "3335": 7.64736,
+ "3340": 7.62526,
+ "3345": 7.63484,
+ "3350": 7.66377,
+ "3355": 7.62927,
+ "3360": 7.6678,
+ "3365": 7.64533,
+ "3370": 7.66942,
+ "3375": 7.60516,
+ "3380": 7.64808,
+ "3385": 7.65067,
+ "3390": 7.69531,
+ "3395": 7.65991,
+ "3400": 7.72274,
+ "3405": 7.61003,
+ "3410": 7.62369,
+ "3415": 7.62553,
+ "3420": 7.65846,
+ "3425": 7.63998,
+ "3430": 7.66528,
+ "3435": 7.65907,
+ "3440": 7.61179,
+ "3445": 7.68532,
+ "3450": 7.64069,
+ "3455": 7.66349,
+ "3460": 7.68717,
+ "3465": 7.68708,
+ "3470": 7.63238,
+ "3475": 7.6568,
+ "3480": 7.67111,
+ "3485": 7.66481,
+ "3490": 7.62887,
+ "3495": 7.65557,
+ "3500": 7.62625,
+ "3505": 7.71501,
+ "3510": 7.64996,
+ "3515": 7.69502,
+ "3520": 7.66362,
+ "3525": 7.63296,
+ "3530": 7.65757,
+ "3535": 7.71042,
+ "3540": 7.68235,
+ "3545": 7.68551,
+ "3550": 7.66414,
+ "3555": 7.68569,
+ "3560": 7.66991,
+ "3565": 7.70531,
+ "3570": 7.65212,
+ "3575": 7.68573,
+ "3580": 7.69626,
+ "3585": 7.68089,
+ "3590": 7.6568,
+ "3595": 7.61543,
+ "3600": 7.72845,
+ "3605": 7.68661,
+ "3610": 7.69481,
+ "3615": 13.41554,
+ "3620": 7.6373,
+ "3625": 7.66675,
+ "3630": 7.72517,
+ "3635": 7.71089,
+ "3640": 7.71975,
+ "3645": 12.60288,
+ "3650": 7.6318,
+ "3655": 7.68076,
+ "3660": 7.67209,
+ "3665": 7.63448,
+ "3670": 7.69758,
+ "3675": 7.67857,
+ "3680": 7.6426,
+ "3685": 7.66942,
+ "3690": 7.68178,
+ "3695": 7.68228,
+ "3700": 22.65199,
+ "3705": 10.95201,
+ "3710": 8.97118,
+ "3715": 7.782,
+ "3720": 7.71021,
+ "3725": 7.67955,
+ "3730": 7.72281,
+ "3735": 7.64037,
+ "3740": 7.68045,
+ "3745": 7.69966,
+ "3750": 7.69781,
+ "3755": 7.65335,
+ "3760": 7.66658,
+ "3765": 7.64471,
+ "3770": 7.73208,
+ "3775": 7.65868,
+ "3780": 7.73451,
+ "3785": 7.68562,
+ "3790": 7.64919,
+ "3795": 7.6445,
+ "3800": 7.65915,
+ "3805": 7.70201,
+ "3810": 7.64818,
+ "3815": 7.67157,
+ "3820": 7.69746,
+ "3825": 7.70803,
+ "3830": 7.69473,
+ "3835": 7.70084,
+ "3840": 7.69734,
+ "3845": 7.66876,
+ "3850": 7.65939,
+ "3855": 7.65685,
+ "3860": 7.63863,
+ "3865": 7.67232,
+ "3870": 7.63944,
+ "3875": 7.67016,
+ "3880": 7.66891,
+ "3885": 10.00338,
+ "3890": 7.71469,
+ "3895": 7.67657,
+ "3900": 7.71214,
+ "3905": 7.66439,
+ "3910": 7.66758,
+ "3915": 7.67499,
+ "3920": 7.74667,
+ "3925": 7.68673,
+ "3930": 7.66643,
+ "3935": 7.63239,
+ "3940": 7.66664,
+ "3945": 7.67862,
+ "3950": 7.68625,
+ "3955": 7.72395,
+ "3960": 7.68159,
+ "3965": 7.70805,
+ "3970": 7.72156,
+ "3975": 7.67959,
+ "3980": 15.32084,
+ "3985": 7.67343,
+ "3990": 7.68087,
+ "3995": 7.62694,
+ "4000": 7.64998,
+ "4005": 7.66822,
+ "4010": 7.69406,
+ "4015": 7.63434,
+ "4020": 7.70801,
+ "4025": 7.63371,
+ "4030": 7.69563,
+ "4035": 7.68986,
+ "4040": 7.6771,
+ "4045": 17.18097,
+ "4050": 7.67402,
+ "4055": 7.64008,
+ "4060": 7.69082,
+ "4065": 7.6721,
+ "4070": 7.66297,
+ "4075": 7.68569,
+ "4080": 7.67875,
+ "4085": 7.66644,
+ "4090": 7.67484,
+ "4095": 7.68586,
+ "4100": 28.01752,
+ "4105": 7.70327,
+ "4110": 7.63359,
+ "4115": 7.68437,
+ "4120": 7.68105,
+ "4125": 7.65511,
+ "4130": 7.66122,
+ "4135": 7.66655,
+ "4140": 16.4225,
+ "4145": 7.67369,
+ "4150": 7.67181,
+ "4155": 7.68623,
+ "4160": 7.68068,
+ "4165": 7.65254,
+ "4170": 7.64474,
+ "4175": 7.67178,
+ "4180": 7.68414,
+ "4185": 7.6303,
+ "4190": 7.66892,
+ "4195": 7.65353,
+ "4200": 7.62909,
+ "4205": 7.67635,
+ "4210": 54.64207,
+ "4215": 7.68482,
+ "4220": 7.66425,
+ "4225": 7.63716,
+ "4230": 7.68579,
+ "4235": 14.46487,
+ "4240": 7.658,
+ "4245": 7.68702,
+ "4250": 7.65865,
+ "4255": 17.354,
+ "4260": 8.23622,
+ "4265": 7.778,
+ "4270": 7.70374,
+ "4275": 7.65029,
+ "4280": 7.66319,
+ "4285": 7.65756,
+ "4290": 8.12667,
+ "4295": 7.65898,
+ "4300": 7.6764,
+ "4305": 7.73041,
+ "4310": 7.69675,
+ "4315": 7.65924,
+ "4320": 7.63593,
+ "4325": 7.63601,
+ "4330": 7.66656,
+ "4335": 7.66401,
+ "4340": 7.66813,
+ "4345": 7.64673,
+ "4350": 7.66468,
+ "4355": 20.04902,
+ "4360": 7.66611,
+ "4365": 7.65353,
+ "4370": 7.62179,
+ "4375": 7.66305,
+ "4380": 7.68996,
+ "4385": 7.63037,
+ "4390": 7.62821,
+ "4395": 7.65873,
+ "4400": 7.66904,
+ "4405": 7.6291,
+ "4410": 7.6745,
+ "4415": 7.66251,
+ "4420": 7.67086,
+ "4425": 7.67821,
+ "4430": 7.6416,
+ "4435": 7.68272,
+ "4440": 7.7135,
+ "4445": 7.68296,
+ "4450": 7.65296,
+ "4455": 7.67754,
+ "4460": 7.6939,
+ "4465": 7.65829,
+ "4470": 7.67054,
+ "4475": 7.66775,
+ "4480": 7.64988,
+ "4485": 7.67972,
+ "4490": 7.68712,
+ "4495": 7.67531,
+ "4500": 7.65847,
+ "4505": 7.66778,
+ "4510": 8.98397,
+ "4515": 7.63669,
+ "4520": 7.7103,
+ "4525": 7.68201,
+ "4530": 7.67072,
+ "4535": 7.66094,
+ "4540": 7.64541,
+ "4545": 7.65274,
+ "4550": 7.68404,
+ "4555": 7.70643,
+ "4560": 7.74737,
+ "4565": 7.62903,
+ "4570": 7.64178,
+ "4575": 7.63429,
+ "4580": 7.69763,
+ "4585": 7.65485,
+ "4590": 7.63337,
+ "4595": 7.66418,
+ "4600": 7.65143,
+ "4605": 7.63451,
+ "4610": 7.67576,
+ "4615": 7.69455,
+ "4620": 7.71329,
+ "4625": 7.63101,
+ "4630": 7.65277,
+ "4635": 7.68349,
+ "4640": 7.71953,
+ "4645": 7.68531,
+ "4650": 7.66915,
+ "4655": 7.67393,
+ "4660": 7.63697,
+ "4665": 7.63584,
+ "4670": 7.66257,
+ "4675": 7.695,
+ "4680": 7.68932,
+ "4685": 7.70765,
+ "4690": 7.63272,
+ "4695": 7.67098,
+ "4700": 7.65506,
+ "4705": 7.64756,
+ "4710": 7.70568,
+ "4715": 7.70601,
+ "4720": 7.65997,
+ "4725": 7.70427,
+ "4730": 7.69099,
+ "4735": 14.04016,
+ "4740": 7.72002,
+ "4745": 7.69168,
+ "4750": 7.67512,
+ "4755": 7.74976,
+ "4760": 7.66477,
+ "4765": 7.67183,
+ "4770": 7.6847,
+ "4775": 7.67532,
+ "4780": 43.25243,
+ "4785": 7.63326,
+ "4790": 7.69096,
+ "4795": 7.65318,
+ "4800": 7.69974,
+ "4805": 7.68321,
+ "4810": 7.67585,
+ "4815": 7.68149,
+ "4820": 7.68224,
+ "4825": 31.30009,
+ "4830": 7.6613,
+ "4835": 7.67486,
+ "4840": 7.63749,
+ "4845": 7.69415,
+ "4850": 7.71385,
+ "4855": 7.69102,
+ "4860": 7.65262,
+ "4865": 7.68144,
+ "4870": 7.67987,
+ "4875": 7.68057,
+ "4880": 7.64254,
+ "4885": 27.56509,
+ "4890": 7.64661,
+ "4895": "nan",
+ "4900": 7.68134,
+ "4905": 7.70593,
+ "4910": 9.58632,
+ "4915": 7.73211,
+ "4920": 7.68668,
+ "4925": 7.71361,
+ "4930": 7.91312,
+ "4935": 7.72251,
+ "4940": 7.68644,
+ "4945": 16.00447,
+ "4950": 7.69013,
+ "4955": 7.69151,
+ "4960": 7.71817,
+ "4965": 7.71112,
+ "4970": 26.2412,
+ "4975": 7.68352,
+ "4980": 7.68756,
+ "4985": 7.67801,
+ "4990": 7.71499,
+ "4995": 7.67015,
+ "5000": 7.79993,
+ "5005": 7.67475,
+ "5010": 7.67978,
+ "5015": 7.67998,
+ "5020": 7.70695,
+ "5025": 31.24429,
+ "5030": 7.68808,
+ "5035": 7.69809,
+ "5040": 7.70278,
+ "5045": 7.68088,
+ "5050": 7.72886,
+ "5055": 7.69057,
+ "5060": 7.68262,
+ "5065": 7.71677,
+ "5070": 7.67219,
+ "5075": 7.75284,
+ "5080": 7.71272,
+ "5085": 7.66355,
+ "5090": 7.6767,
+ "5095": 7.69415,
+ "5100": 7.70634,
+ "5105": 7.71839,
+ "5110": 7.72789,
+ "5115": 22.05469,
+ "5120": 7.67899,
+ "5125": 7.6935,
+ "5130": 7.74779,
+ "5135": 7.70712,
+ "5140": 7.67206,
+ "5145": 17.70929,
+ "5150": 7.70704,
+ "5155": 7.68127,
+ "5160": 7.72235,
+ "5165": 7.68984,
+ "5170": 15.5891,
+ "5175": 7.69291,
+ "5180": 7.66184,
+ "5185": 7.71608,
+ "5190": 7.69001,
+ "5195": 7.70726,
+ "5200": 7.70094,
+ "5205": 7.73004,
+ "5210": 7.69843,
+ "5215": 7.71327,
+ "5220": 7.73885,
+ "5225": 7.67857,
+ "5230": 14.3485,
+ "5235": 7.71084,
+ "5240": 7.7139,
+ "5245": 7.7056,
+ "5250": 7.67536,
+ "5255": 7.65597,
+ "5260": 7.70827,
+ "5265": 7.67382,
+ "5270": 7.69029,
+ "5275": 7.68198,
+ "5280": 17.0703,
+ "5285": 7.73332,
+ "5290": 7.7135,
+ "5295": 22.67106,
+ "5300": 7.67717,
+ "5305": 27.73863,
+ "5310": 7.67759,
+ "5315": 7.67961,
+ "5320": 7.71378,
+ "5325": 7.67838,
+ "5330": 7.74258,
+ "5335": 7.69614,
+ "5340": 35.16928,
+ "5345": 7.69894,
+ "5350": 7.68856,
+ "5355": 7.7126,
+ "5360": 7.70859,
+ "5365": 7.68087,
+ "5370": 7.7596,
+ "5375": 7.68473,
+ "5380": 7.69817,
+ "5385": 7.71854,
+ "5390": 7.69112,
+ "5395": 7.63647,
+ "5400": 7.70599,
+ "5405": 7.70342,
+ "5410": 7.70778,
+ "5415": 7.69746,
+ "5420": 7.68884,
+ "5425": 7.69676,
+ "5430": 7.68817,
+ "5435": 7.67805,
+ "5440": 7.71194,
+ "5445": 7.7285,
+ "5450": 7.70133,
+ "5455": 7.67386,
+ "5460": 7.67064,
+ "5465": 7.66238,
+ "5470": 7.74211,
+ "5475": 7.67017,
+ "5480": 7.69776,
+ "5485": 7.72524,
+ "5490": 7.62662,
+ "5495": 7.67569,
+ "5500": 7.73237,
+ "5505": 7.90632,
+ "5510": 7.76154,
+ "5515": 7.64476,
+ "5520": 7.6757,
+ "5525": 7.68599,
+ "5530": 7.66966,
+ "5535": 9.18675,
+ "5540": 7.6801,
+ "5545": 7.67432,
+ "5550": 7.68584,
+ "5555": 7.70635,
+ "5560": 7.70152,
+ "5565": 7.71209,
+ "5570": 7.68857,
+ "5575": 9.43548,
+ "5580": 7.66548,
+ "5585": 7.70493,
+ "5590": 7.67944,
+ "5595": 7.67922,
+ "5600": 7.68347,
+ "5605": 7.66992,
+ "5610": 7.69588,
+ "5615": 13.57089,
+ "5620": 7.70507,
+ "5625": 7.67431,
+ "5630": 7.66669,
+ "5635": 7.67469,
+ "5640": 7.74966,
+ "5645": 7.6919,
+ "5650": 7.68959,
+ "5655": 7.71422,
+ "5660": 7.63607,
+ "5665": 9.77754,
+ "5670": 7.64023,
+ "5675": 7.69112,
+ "5680": 7.70735,
+ "5685": 7.69287,
+ "5690": 7.72413,
+ "5695": 7.73045,
+ "5700": 7.71151,
+ "5705": 7.69381,
+ "5710": 7.66626,
+ "5715": 12.30891,
+ "5720": 7.66336,
+ "5725": 7.68645,
+ "5730": 7.64707,
+ "5735": 7.70967,
+ "5740": 7.70411,
+ "5745": 7.6946,
+ "5750": 7.64485,
+ "5755": 7.70134,
+ "5760": 7.69069,
+ "5765": 7.67883,
+ "5770": 7.71137,
+ "5775": 7.63267,
+ "5780": 7.85877,
+ "5785": 7.72289,
+ "5790": 7.67204,
+ "5795": 7.66597,
+ "5800": 7.72562,
+ "5805": 7.65974,
+ "5810": 7.64619,
+ "5815": 7.67024,
+ "5820": 7.62553,
+ "5825": 7.67124,
+ "5830": 7.66624,
+ "5835": 7.68946,
+ "5840": 7.65345,
+ "5845": 7.65122,
+ "5850": 7.67502,
+ "5855": 7.65158,
+ "5860": 7.66193,
+ "5865": 7.6739,
+ "5870": 7.68753,
+ "5875": 7.70964,
+ "5880": 7.67763,
+ "5885": 7.66601,
+ "5890": 7.65221,
+ "5895": 7.65165,
+ "5900": 7.65244,
+ "5905": 7.73687,
+ "5910": 7.64741,
+ "5915": 7.6742,
+ "5920": 7.67382,
+ "5925": 7.67278,
+ "5930": 7.66046,
+ "5935": 7.67212,
+ "5940": 7.72681,
+ "5945": 7.65546,
+ "5950": 7.76112,
+ "5955": 7.68477,
+ "5960": 7.65907,
+ "5965": 7.69466,
+ "5970": 7.67805,
+ "5975": 7.69033,
+ "5980": 8.79548,
+ "5985": 8.73726,
+ "5990": 8.7589,
+ "5995": 8.76382,
+ "6000": 9.89093,
+ "6005": 8.74983,
+ "6010": 8.75158,
+ "6015": 8.65243,
+ "6020": 8.73512,
+ "6025": 8.79845,
+ "6030": 10.54209,
+ "6035": 8.72273,
+ "6040": 8.7371,
+ "6045": 8.73574,
+ "6050": 8.77403,
+ "6055": 8.71515,
+ "6060": 8.77555,
+ "6065": 8.77075,
+ "6070": 8.79184,
+ "6075": 8.85241,
+ "6080": 9.60877,
+ "6085": 8.76196,
+ "6090": 8.72087,
+ "6095": 8.76681,
+ "6100": 8.74255,
+ "6105": 8.71004,
+ "6110": 8.75566,
+ "6115": 8.80475,
+ "6120": 8.74674,
+ "6125": 8.75779,
+ "6130": 8.77423,
+ "6135": 8.7476,
+ "6140": 8.78927,
+ "6145": 8.77839,
+ "6150": 8.84674,
+ "6155": 8.83688,
+ "6160": 8.77236,
+ "6165": 8.77225,
+ "6170": 8.73577,
+ "6175": 8.76183,
+ "6180": 8.7204,
+ "6185": 8.72854,
+ "6190": 8.74246,
+ "6195": 8.73676,
+ "6200": 8.73612,
+ "6205": 8.73403,
+ "6210": 8.76218,
+ "6215": 8.70949,
+ "6220": 8.74637,
+ "6225": 8.74396,
+ "6230": 8.79046,
+ "6235": 8.73752,
+ "6240": 8.7773,
+ "6245": 8.9205,
+ "6250": 8.74323,
+ "6255": 8.69041,
+ "6260": 8.76066,
+ "6265": 8.80149,
+ "6270": 8.71374,
+ "6275": 8.73549,
+ "6280": 8.73678,
+ "6285": 8.75697,
+ "6290": 8.69536,
+ "6295": 8.73382,
+ "6300": 8.72091,
+ "6305": 8.69085,
+ "6310": 8.74178,
+ "6315": 8.76341,
+ "6320": 8.72425,
+ "6325": 8.75025,
+ "6330": 35.45488,
+ "6335": 8.73051,
+ "6340": 8.72973,
+ "6345": 8.68953,
+ "6350": 8.74064,
+ "6355": 8.70852,
+ "6360": 8.74355,
+ "6365": 8.73836,
+ "6370": 8.73532,
+ "6375": 8.74372,
+ "6380": 8.79124,
+ "6385": 8.76221,
+ "6390": 18.40093,
+ "6395": 15.69404,
+ "6400": 8.72229,
+ "6405": 8.76673,
+ "6410": 8.80782,
+ "6415": 8.71979,
+ "6420": 8.76622,
+ "6425": 8.74861,
+ "6430": 9.80378,
+ "6435": 8.75522,
+ "6440": 8.74079,
+ "6445": 8.69742,
+ "6450": 8.69718,
+ "6455": 8.79111,
+ "6460": 12.86974,
+ "6465": 8.77111,
+ "6470": 8.73458,
+ "6475": 8.81103,
+ "6480": 22.22738,
+ "6485": 8.78657,
+ "6490": 8.79684,
+ "6495": 8.78682,
+ "6500": 8.782,
+ "6505": 22.09958,
+ "6510": 8.68731,
+ "6515": 8.68647,
+ "6520": 8.73748,
+ "6525": 8.75405,
+ "6530": 45.88803,
+ "6535": 8.73079,
+ "6540": 8.70524,
+ "6545": 8.70441,
+ "6550": 8.80887,
+ "6555": 8.79102,
+ "6560": 8.65396,
+ "6565": 12.6516,
+ "6570": 8.75772,
+ "6575": 8.76683,
+ "6580": 8.78237,
+ "6585": 8.78726,
+ "6590": 8.751,
+ "6595": 8.77629,
+ "6600": 8.76154,
+ "6605": 8.72373,
+ "6610": 8.70413,
+ "6615": 8.69621,
+ "6620": 8.75383,
+ "6625": 8.69736,
+ "6630": 8.72602,
+ "6635": 8.70564,
+ "6640": 11.1172,
+ "6645": 8.70857,
+ "6650": 8.69853,
+ "6655": 8.77941,
+ "6660": 8.82578,
+ "6665": 8.65203,
+ "6670": 8.70479,
+ "6675": 15.3011,
+ "6680": 8.76023,
+ "6685": 10.36584,
+ "6690": 8.72089,
+ "6695": 8.74824,
+ "6700": 15.81454,
+ "6705": 8.75746,
+ "6710": 8.69047,
+ "6715": 8.72232,
+ "6720": 8.78934,
+ "6725": 8.75915,
+ "6730": 20.76388,
+ "6735": 8.73769,
+ "6740": 8.75228,
+ "6745": 8.73909,
+ "6750": 8.73291,
+ "6755": 8.75026,
+ "6760": 8.66201,
+ "6765": 8.67505,
+ "6770": 8.71159,
+ "6775": 13.4563,
+ "6780": 8.72792,
+ "6785": 8.70931,
+ "6790": 8.73335,
+ "6795": 8.74191,
+ "6800": 10.2767,
+ "6805": 8.74382,
+ "6810": 8.78968,
+ "6815": 13.19024,
+ "6820": 8.69398,
+ "6825": 8.7942,
+ "6830": 8.76662,
+ "6835": 8.73506,
+ "6840": 8.79239,
+ "6845": 8.74376,
+ "6850": 8.75351,
+ "6855": 8.73868,
+ "6860": 8.74296,
+ "6865": 8.74008,
+ "6870": 8.7827,
+ "6875": 8.77137,
+ "6880": 8.76586,
+ "6885": 8.76657,
+ "6890": 11.92249,
+ "6895": 8.74391,
+ "6900": 8.72334,
+ "6905": 8.76226,
+ "6910": 8.81266,
+ "6915": 8.75704,
+ "6920": 20.76976,
+ "6925": 8.73751,
+ "6930": 8.74888,
+ "6935": 8.79712,
+ "6940": 8.81733,
+ "6945": 8.77829,
+ "6950": 8.69087,
+ "6955": 8.71353,
+ "6960": 8.79561,
+ "6965": 8.70962,
+ "6970": 24.5121,
+ "6975": 8.7613,
+ "6980": 8.82408,
+ "6985": 7.7344,
+ "6990": 7.6852,
+ "6995": 7.7523,
+ "7000": 7.68245,
+ "7005": 7.72012,
+ "7010": 7.70011,
+ "7015": 7.71018,
+ "7020": 7.71631,
+ "7025": 7.70545,
+ "7030": 13.70124,
+ "7035": 7.69324,
+ "7040": 7.72656,
+ "7045": 7.71318,
+ "7050": 7.6813,
+ "7055": 7.68302,
+ "7060": 7.69883,
+ "7065": 7.74768,
+ "7070": 7.75916,
+ "7075": 7.7017,
+ "7080": 17.81365,
+ "7085": 7.72346,
+ "7090": 7.75201,
+ "7095": 7.72953,
+ "7100": 7.72177,
+ "7105": 7.72924,
+ "7110": 7.71318,
+ "7115": 7.68422,
+ "7120": 17.15495,
+ "7125": 7.74503,
+ "7130": 7.70276,
+ "7135": 7.72032,
+ "7140": 7.71692,
+ "7145": 7.73425,
+ "7150": 7.74598,
+ "7155": 7.68825,
+ "7160": 7.72755,
+ "7165": 7.71583,
+ "7170": 7.71756,
+ "7175": 7.71892,
+ "7180": 8.62138,
+ "7185": 7.70819,
+ "7190": 25.15296,
+ "7195": 7.73694,
+ "7200": 7.74203,
+ "7205": 7.73459,
+ "7210": 7.7248,
+ "7215": 7.69932,
+ "7220": 7.70514,
+ "7225": 7.6906,
+ "7230": 7.73859,
+ "7235": 7.7486,
+ "7240": 7.71488,
+ "7245": 35.31186,
+ "7250": 7.73689,
+ "7255": 7.78282,
+ "7260": 7.69963,
+ "7265": 7.72129,
+ "7270": 7.68814,
+ "7275": 7.70352,
+ "7280": 7.73157,
+ "7285": 7.70384,
+ "7290": 7.73351,
+ "7295": 7.80747,
+ "7300": 7.73428,
+ "7305": 7.67337,
+ "7310": 7.701,
+ "7315": 7.73637,
+ "7320": 7.68732,
+ "7325": 7.72061,
+ "7330": 7.71733,
+ "7335": 7.74247,
+ "7340": 7.7538,
+ "7345": 15.56451,
+ "7350": 7.74871,
+ "7355": 7.71074,
+ "7360": 7.74262,
+ "7365": 7.72847,
+ "7370": 7.70963,
+ "7375": 7.73152,
+ "7380": 7.68074,
+ "7385": 7.70151,
+ "7390": 7.69577,
+ "7395": 7.70635,
+ "7400": 7.68218,
+ "7405": 7.73887,
+ "7410": 7.69486,
+ "7415": 26.09414,
+ "7420": 7.73635,
+ "7425": 7.72707,
+ "7430": 7.7482,
+ "7435": 7.70744,
+ "7440": 7.71831,
+ "7445": 7.71964,
+ "7450": 7.71185,
+ "7455": 7.75286,
+ "7460": 7.70447,
+ "7465": 7.67153,
+ "7470": 7.70877,
+ "7475": 7.73397,
+ "7480": 12.91944,
+ "7485": 7.72823,
+ "7490": 7.69517,
+ "7495": 7.70582,
+ "7500": 7.72312,
+ "7505": 7.86106,
+ "7510": 7.72171,
+ "7515": 8.63603,
+ "7520": 7.73724,
+ "7525": 7.70479,
+ "7530": 7.72689,
+ "7535": 7.71714,
+ "7540": 7.73467,
+ "7545": 7.75475,
+ "7550": 7.71459,
+ "7555": 7.75871,
+ "7560": 7.69906,
+ "7565": 7.81678,
+ "7570": 7.72792,
+ "7575": 7.73853,
+ "7580": 19.05192,
+ "7585": 7.70928,
+ "7590": 7.74247,
+ "7595": 7.70334,
+ "7600": 7.73216,
+ "7605": 7.68997,
+ "7610": 7.74887,
+ "7615": 7.70017,
+ "7620": 7.712,
+ "7625": 20.69044,
+ "7630": 7.72508,
+ "7635": 7.74431,
+ "7640": 7.74614,
+ "7645": 7.73495,
+ "7650": 7.75134,
+ "7655": 7.73525,
+ "7660": 7.71382,
+ "7665": 7.71879,
+ "7670": 7.71069,
+ "7675": 7.7545,
+ "7680": 7.731,
+ "7685": 28.38769,
+ "7690": 7.70013,
+ "7695": 7.70512,
+ "7700": 7.72898,
+ "7705": 7.71175,
+ "7710": 7.74054,
+ "7715": 25.79383,
+ "7720": 7.65866,
+ "7725": 7.68378,
+ "7730": 7.7442,
+ "7735": 7.72939,
+ "7740": 20.52677,
+ "7745": 7.68463,
+ "7750": 7.74328,
+ "7755": 8.09776,
+ "7760": 7.77523,
+ "7765": 7.79577,
+ "7770": 7.72056,
+ "7775": 14.58435,
+ "7780": 7.71121,
+ "7785": 7.6897,
+ "7790": 25.81522,
+ "7795": 7.70367,
+ "7800": 7.74868,
+ "7805": 7.72241,
+ "7810": 16.44418,
+ "7815": 7.70712,
+ "7820": 7.74024,
+ "7825": 7.73102,
+ "7830": 7.70797,
+ "7835": 7.74084,
+ "7840": 7.77577,
+ "7845": 19.41641,
+ "7850": 7.72432,
+ "7855": 7.71354,
+ "7860": 13.02327,
+ "7865": 7.70382,
+ "7870": 12.16723,
+ "7875": 7.73482,
+ "7880": 7.73417,
+ "7885": 7.73531,
+ "7890": 7.71453,
+ "7895": 7.72886,
+ "7900": 7.7253,
+ "7905": 7.7319,
+ "7910": 7.74598,
+ "7915": 10.66316,
+ "7920": 7.67918,
+ "7925": 7.71462,
+ "7930": 7.76094,
+ "7935": 7.72558,
+ "7940": 7.71017,
+ "7945": 7.72874,
+ "7950": 7.741,
+ "7955": 7.72791,
+ "7960": 7.71644,
+ "7965": 7.72177,
+ "7970": 7.75399,
+ "7975": 7.7282,
+ "7980": 7.69633,
+ "7985": 12.71552,
+ "7990": 7.71698,
+ "7995": 7.71279,
+ "8000": 7.73903,
+ "8005": 7.73097,
+ "8010": 7.72597,
+ "8015": 7.72989,
+ "8020": 7.70347,
+ "8025": 7.73356,
+ "8030": 7.71594,
+ "8035": 7.69841,
+ "8040": 7.73756,
+ "8045": 16.95034,
+ "8050": 7.70179,
+ "8055": 7.70637,
+ "8060": 7.6874,
+ "8065": 7.69848,
+ "8070": 7.75451,
+ "8075": 7.72501,
+ "8080": 7.73083,
+ "8085": 7.76536,
+ "8090": 7.76086,
+ "8095": 7.68885,
+ "8100": 7.68824,
+ "8105": 7.68343,
+ "8110": 7.7345,
+ "8115": 7.73921,
+ "8120": 7.75155,
+ "8125": 7.71054,
+ "8130": 7.77458,
+ "8135": 12.12122,
+ "8140": 7.71921,
+ "8145": 7.75589,
+ "8150": 7.70204,
+ "8155": 7.70644,
+ "8160": 12.72723,
+ "8165": 7.72022,
+ "8170": 7.67897,
+ "8175": 7.73813,
+ "8180": 7.73972,
+ "8185": 7.69324,
+ "8190": 7.72452,
+ "8195": 7.68867,
+ "8200": 15.48012,
+ "8205": 7.70131,
+ "8210": 7.73762,
+ "8215": 7.72735,
+ "8220": 7.72877,
+ "8225": 65.90765,
+ "8230": 7.73431,
+ "8235": 7.70452,
+ "8240": 7.74843,
+ "8245": 7.71159,
+ "8250": 7.75873,
+ "8255": 18.46645,
+ "8260": 7.69312,
+ "8265": 7.69817,
+ "8270": 20.16962,
+ "8275": 7.69891,
+ "8280": 7.71422,
+ "8285": 7.68322,
+ "8290": 7.74574,
+ "8295": 7.72721,
+ "8300": 7.69463,
+ "8305": 17.94482,
+ "8310": 7.74095,
+ "8315": 7.68236,
+ "8320": 7.73209,
+ "8325": 7.68919,
+ "8330": 7.74896,
+ "8335": 27.31071,
+ "8340": 7.72784,
+ "8345": 7.70944,
+ "8350": 7.7479,
+ "8355": 7.76406,
+ "8360": 8.44401,
+ "8365": 7.72299,
+ "8370": 9.13732,
+ "8375": 7.75345,
+ "8380": 7.73354,
+ "8385": 7.75105,
+ "8390": 7.75877,
+ "8395": 7.73427,
+ "8400": 7.77439,
+ "8405": 7.75556,
+ "8410": 7.74766,
+ "8415": 7.74364,
+ "8420": 7.74118,
+ "8425": 7.71627,
+ "8430": 7.76763,
+ "8435": 7.87226,
+ "8440": 7.74166,
+ "8445": 7.79113,
+ "8450": 7.75387,
+ "8455": 7.7455,
+ "8460": 7.8341,
+ "8465": 7.75567,
+ "8470": 7.73886,
+ "8475": 7.75875,
+ "8480": 7.77714,
+ "8485": 7.73162,
+ "8490": 7.76034,
+ "8495": 7.70186,
+ "8500": 7.74867,
+ "8505": 7.72104,
+ "8510": 7.77237,
+ "8515": 7.76571,
+ "8520": 7.80207,
+ "8525": 7.83793,
+ "8530": 7.72948,
+ "8535": 7.74241,
+ "8540": 7.76759,
+ "8545": 7.74558,
+ "8550": 9.78662,
+ "8555": 7.78862,
+ "8560": 7.69314,
+ "8565": 7.75092,
+ "8570": 7.788,
+ "8575": 7.77628,
+ "8580": 7.77852,
+ "8585": 7.75473,
+ "8590": 7.72584,
+ "8595": 7.77413,
+ "8600": 7.74906,
+ "8605": 7.78572,
+ "8610": 7.72038,
+ "8615": 7.72149,
+ "8620": 7.74595,
+ "8625": 7.76997,
+ "8630": 7.78023,
+ "8635": 7.77384,
+ "8640": 7.81542,
+ "8645": 7.75832,
+ "8650": 7.45772,
+ "8655": 7.50184,
+ "8660": 7.4452,
+ "8665": 7.51943,
+ "8670": 7.46972,
+ "8675": 7.48924,
+ "8680": 7.46635,
+ "8685": 7.46267,
+ "8690": 7.51511,
+ "8695": 7.5173,
+ "8700": 7.49462,
+ "8705": 36.97242,
+ "8710": 7.56266,
+ "8715": 7.56513,
+ "8720": 7.58076,
+ "8725": 7.55647,
+ "8730": 7.57453,
+ "8735": 7.57805,
+ "8740": 7.58131,
+ "8745": 7.59663,
+ "8750": 7.56867,
+ "8755": 7.58379,
+ "8760": 34.0399,
+ "8765": 7.59564,
+ "8770": 7.61435,
+ "8775": 7.62113,
+ "8780": 7.63075,
+ "8785": 10.50408,
+ "8790": 7.60323,
+ "8795": 7.59093,
+ "8800": 7.62682,
+ "8805": 7.55253,
+ "8810": 7.59003,
+ "8815": 7.56221,
+ "8820": 7.61346,
+ "8825": 7.59609,
+ "8830": 8.90255,
+ "8835": 7.5662,
+ "8840": 7.55823,
+ "8845": 7.59601,
+ "8850": 32.18616,
+ "8855": 7.56676,
+ "8860": 7.62033,
+ "8865": 7.56054,
+ "8870": 7.58606,
+ "8875": 7.5728,
+ "8880": 7.58588,
+ "8885": 7.58024,
+ "8890": 7.58467,
+ "8895": 7.59696,
+ "8900": 7.57192,
+ "8905": 14.21695,
+ "8910": 7.60357,
+ "8915": 7.56492,
+ "8920": 7.60684,
+ "8925": 7.58744,
+ "8930": 7.607,
+ "8935": 7.54646,
+ "8940": 15.37046,
+ "8945": 7.56413,
+ "8950": 7.63383,
+ "8955": 7.57658,
+ "8960": 7.57496,
+ "8965": 7.56415,
+ "8970": 7.57714,
+ "8975": 7.54666,
+ "8980": 13.66879,
+ "8985": 7.58742,
+ "8990": 7.59615,
+ "8995": 7.54326,
+ "9000": 36.72772,
+ "9005": 7.58381,
+ "9010": 7.58351,
+ "9015": 7.60301,
+ "9020": 7.6092,
+ "9025": 7.59685,
+ "9030": 11.75952,
+ "9035": 7.54508,
+ "9040": 7.61937,
+ "9045": 7.56783,
+ "9050": 7.58833,
+ "9055": 7.56681,
+ "9060": 7.57701,
+ "9065": 7.57428,
+ "9070": 7.64229,
+ "9075": 7.56663,
+ "9080": 7.57114,
+ "9085": 7.58561,
+ "9090": 7.57738,
+ "9095": 7.58988,
+ "9100": 9.30224,
+ "9105": 7.56069,
+ "9110": 7.57408,
+ "9115": 7.58594,
+ "9120": 7.62045,
+ "9125": 7.56998,
+ "9130": 7.61472,
+ "9135": 7.58332,
+ "9140": 7.5713,
+ "9145": 7.57564,
+ "9150": 7.60248,
+ "9155": 7.56456,
+ "9160": 7.56255,
+ "9165": 7.62102,
+ "9170": 7.60054,
+ "9175": 7.56118,
+ "9180": 7.56444,
+ "9185": 7.5746,
+ "9190": 7.57369,
+ "9195": 34.35552,
+ "9200": 7.54977,
+ "9205": 7.59118,
+ "9210": 7.55959,
+ "9215": 7.57658,
+ "9220": 7.57651,
+ "9225": 7.57532,
+ "9230": 7.57772,
+ "9235": 7.55808,
+ "9240": 7.60423,
+ "9245": 7.56324,
+ "9250": 7.57204,
+ "9255": 7.6287,
+ "9260": 7.58491,
+ "9265": 7.5846,
+ "9270": 34.79556,
+ "9275": 7.57817,
+ "9280": 7.58667,
+ "9285": 7.56099,
+ "9290": 7.59242,
+ "9295": 7.57733,
+ "9300": 19.57602,
+ "9305": 7.56234,
+ "9310": 7.59926,
+ "9315": 7.55985,
+ "9320": 7.60214,
+ "9325": 7.62397,
+ "9330": 7.52905,
+ "9335": 7.61258,
+ "9340": 7.57207,
+ "9345": 17.83714,
+ "9350": 7.57875,
+ "9355": 7.59114,
+ "9360": 7.61886,
+ "9365": 7.55746,
+ "9370": 7.62008,
+ "9375": 7.57546,
+ "9380": 7.55667,
+ "9385": 7.56997,
+ "9390": 7.60437,
+ "9395": 7.56261,
+ "9400": 7.63204,
+ "9405": 13.70497,
+ "9410": 7.58881,
+ "9415": 7.56621,
+ "9420": 7.56237,
+ "9425": 7.58213,
+ "9430": 7.5846,
+ "9435": 7.57692,
+ "9440": 7.58347,
+ "9445": 7.58848,
+ "9450": 25.3813,
+ "9455": 7.63607,
+ "9460": 7.56592,
+ "9465": 7.59055,
+ "9470": 7.59247,
+ "9475": 7.6003,
+ "9480": 7.57424,
+ "9485": 7.59309,
+ "9490": 7.57347,
+ "9495": 7.6102,
+ "9500": 7.5708,
+ "9505": 7.92231,
+ "9510": 7.57917,
+ "9515": 7.57732,
+ "9520": 7.62003,
+ "9525": 7.54771,
+ "9530": 7.62317,
+ "9535": 7.59545
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_gb_200_release_sm/model_config.yaml b/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp2pp2emp16etp1cp1_gb_200_release_sm/model_config.yaml
similarity index 92%
rename from tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_gb_200_release_sm/model_config.yaml
rename to tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp2pp2emp16etp1cp1_gb_200_release_sm/model_config.yaml
index 1ad8597d932..36cd34bf752 100644
--- a/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp1pp4emp16etp1cp1_gb_200_release_sm/model_config.yaml
+++ b/tests/functional_tests/test_cases/mixtral/deepseekv3_proxy_flex_tp2pp2emp16etp1cp1_gb_200_release_sm/model_config.yaml
@@ -13,16 +13,16 @@ ENV_VARS:
NON_DETERMINSTIC_RESULTS: 1
NVSHMEM_IB_ENABLE_IBGDA: 0
CUDA_DEVICE_MAX_CONNECTIONS: 1
- NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN: 16
+ NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN: 8
USE_MNNVL: 1
TEST_TYPE: "release"
MODEL_ARGS:
# Distributed args
--distributed-timeout-minutes: 60
--tensor-model-parallel-size: 2
- --pipeline-model-parallel-size: 4
- --pipeline-model-parallel-layout: Et*2\\|\\(tt\\|\\)*5t\\|tmL # Et*2|(tt|)*5t|tmL
- --expert-model-parallel-size: 16
+ --pipeline-model-parallel-size: 2
+ --pipeline-model-parallel-layout: "Et*7|t*7mL"
+ --expert-model-parallel-size: 8
--context-parallel-size: 1
--expert-tensor-parallel-size: 1
--use-distributed-optimizer: true
@@ -34,7 +34,7 @@ MODEL_ARGS:
--sequence-parallel: true
--use-flash-attn: true
--disable-bias-linear: true
- --micro-batch-size: 1
+ --micro-batch-size: 4
--global-batch-size: 512
--train-samples: 24414062
--exit-duration-in-mins: 220
@@ -94,7 +94,7 @@ MODEL_ARGS:
--adam-beta2: 0.95
# Add MoE args
- --num-experts: 64 # local 4 + 1 shared, EP16
+ --num-experts: 64 # local 8 + 1 shared, EP8
--moe-layer-freq: ([0]*3+[1]*11)
--moe-ffn-hidden-size: 2048
--moe-shared-expert-intermediate-size: 2048
@@ -134,8 +134,8 @@ MODEL_ARGS:
# Add checkpointing args
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --save-interval: 500
- --save-retain-interval: 10000
+ --save-interval: 250
+ --save-retain-interval: 5000
--dist-ckpt-strictness: log_all
# Add initialization args
diff --git a/tests/functional_tests/test_cases/mixtral/mixtral_8x22b_tp2pp8ep8vpp1_release/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/mixtral/mixtral_8x22b_tp2pp8ep8vpp1_release/golden_values_dev_dgx_h100.json
index d87affad78d..65b1c44baec 100644
--- a/tests/functional_tests/test_cases/mixtral/mixtral_8x22b_tp2pp8ep8vpp1_release/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/mixtral/mixtral_8x22b_tp2pp8ep8vpp1_release/golden_values_dev_dgx_h100.json
@@ -1 +1,192 @@
-{"lm loss": {"start_step": 1, "end_step": 150, "step_interval": 5, "values": {"1": 1.73921, "5": 1.76233, "10": 1.71016, "15": 1.71693, "20": 1.70925, "25": 1.65675, "30": 1.70044, "35": 1.59874, "40": 1.61213, "45": 1.61385, "50": 1.5751, "55": 1.58708, "60": 1.58154, "65": 1.57132, "70": 1.55148, "75": 1.57899, "80": 1.56981, "85": 1.56007, "90": 1.57537, "95": 1.56237, "100": 1.535, "105": 1.52638, "110": 1.54277, "115": 1.52584, "120": 1.53289, "125": 1.53919, "130": 1.57159, "135": 1.52419, "140": 1.53203, "145": 1.50764, "150": 1.53393}}, "num-zeros": {"start_step": 1, "end_step": 150, "step_interval": 5, "values": {"1": 45157344.0, "5": 48253032.0, "10": 45832968.0, "15": 48763376.0, "20": 49512520.0, "25": 49548924.0, "30": 46404436.0, "35": 46939256.0, "40": 46914044.0, "45": 47307180.0, "50": 46042360.0, "55": 50101696.0, "60": 47000412.0, "65": 49045092.0, "70": 45219292.0, "75": 46238708.0, "80": 48393620.0, "85": 46355312.0, "90": 48178600.0, "95": 47343576.0, "100": 47841512.0, "105": 49604120.0, "110": 49585476.0, "115": 43063424.0, "120": 44880640.0, "125": 47404780.0, "130": 48068596.0, "135": 48338664.0, "140": 47951836.0, "145": 45426960.0, "150": 48922204.0}}, "mem-allocated-bytes": {"start_step": 1, "end_step": 150, "step_interval": 5, "values": {"1": 22504325120.0, "5": 22504327168.0, "10": 22504327168.0, "15": 22504327168.0, "20": 22504327168.0, "25": 22504327168.0, "30": 22504327168.0, "35": 22504327168.0, "40": 22504327168.0, "45": 22504327168.0, "50": 22504327168.0, "55": 22504327168.0, "60": 22504327168.0, "65": 22504327168.0, "70": 22504327168.0, "75": 22504327168.0, "80": 22504327168.0, "85": 22504327168.0, "90": 22504327168.0, "95": 22504327168.0, "100": 22504327168.0, "105": 22504327168.0, "110": 22504327168.0, "115": 22504327168.0, "120": 22504327168.0, "125": 22504327168.0, "130": 22504327168.0, "135": 22504327168.0, "140": 22504327168.0, "145": 22504327168.0, "150": 22504327168.0}}, "mem-max-allocated-bytes": {"start_step": 1, "end_step": 150, "step_interval": 5, "values": {"1": 26732187648.0, "5": 27730509824.0, "10": 27929333760.0, "15": 27929333760.0, "20": 27929333760.0, "25": 27929333760.0, "30": 27929333760.0, "35": 27929333760.0, "40": 28029356032.0, "45": 28029356032.0, "50": 28029356032.0, "55": 28029356032.0, "60": 28030414848.0, "65": 28030414848.0, "70": 28030414848.0, "75": 28030414848.0, "80": 28030414848.0, "85": 28030414848.0, "90": 28030414848.0, "95": 28030414848.0, "100": 28030414848.0, "105": 28030414848.0, "110": 28030414848.0, "115": 28030414848.0, "120": 28030414848.0, "125": 28030414848.0, "130": 28030414848.0, "135": 28030414848.0, "140": 28030414848.0, "145": 28030414848.0, "150": 28030414848.0}}, "iteration-time": {"start_step": 1, "end_step": 150, "step_interval": 5, "values": {"1": 141.97864, "5": 6.34884, "10": 7.2048, "15": 6.66177, "20": 6.76477, "25": 6.54753, "30": 6.66009, "35": 6.91619, "40": 7.11599, "45": 6.48324, "50": 6.31372, "55": 6.47363, "60": 6.48183, "65": 6.20812, "70": 7.3018, "75": 6.45172, "80": 6.35535, "85": 6.32769, "90": 6.5653, "95": 6.1168, "100": 5.8939, "105": 6.52854, "110": 6.7756, "115": 6.12091, "120": 6.33951, "125": 6.82648, "130": 6.30181, "135": 6.59449, "140": 6.49127, "145": 6.34704, "150": 6.13838}}}
\ No newline at end of file
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 150,
+ "step_interval": 5,
+ "values": {
+ "1": 1.73932,
+ "5": 1.7625,
+ "10": 1.71024,
+ "15": 1.71696,
+ "20": 1.7093,
+ "25": 1.65693,
+ "30": 1.70056,
+ "35": 1.59849,
+ "40": 1.61145,
+ "45": 1.60968,
+ "50": 1.57245,
+ "55": 1.58498,
+ "60": 1.58013,
+ "65": 1.57,
+ "70": 1.55073,
+ "75": 1.57846,
+ "80": 1.56971,
+ "85": 1.55965,
+ "90": 1.57538,
+ "95": 1.56214,
+ "100": 1.5347,
+ "105": 1.52615,
+ "110": 1.54244,
+ "115": 1.52564,
+ "120": 1.53258,
+ "125": 1.53894,
+ "130": 1.57133,
+ "135": 1.52407,
+ "140": 1.53167,
+ "145": 1.50728,
+ "150": 1.5337
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 150,
+ "step_interval": 5,
+ "values": {
+ "1": 45157168.0,
+ "5": 48252968.0,
+ "10": 45833172.0,
+ "15": 48763448.0,
+ "20": 49512632.0,
+ "25": 49549132.0,
+ "30": 46404516.0,
+ "35": 46939324.0,
+ "40": 46914204.0,
+ "45": 47307464.0,
+ "50": 46042896.0,
+ "55": 50101452.0,
+ "60": 47000316.0,
+ "65": 49044716.0,
+ "70": 45219092.0,
+ "75": 46238420.0,
+ "80": 48393404.0,
+ "85": 46354672.0,
+ "90": 48178604.0,
+ "95": 47343576.0,
+ "100": 47841268.0,
+ "105": 49604104.0,
+ "110": 49585320.0,
+ "115": 43063064.0,
+ "120": 44880512.0,
+ "125": 47404648.0,
+ "130": 48068848.0,
+ "135": 48339044.0,
+ "140": 47952152.0,
+ "145": 45427352.0,
+ "150": 48922408.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 150,
+ "step_interval": 5,
+ "values": {
+ "1": 23002929152.0,
+ "5": 23002929152.0,
+ "10": 23002929152.0,
+ "15": 23002929152.0,
+ "20": 23002929152.0,
+ "25": 23002929152.0,
+ "30": 23002929152.0,
+ "35": 23002929152.0,
+ "40": 23002929152.0,
+ "45": 23002929152.0,
+ "50": 23002929152.0,
+ "55": 23002929152.0,
+ "60": 23002929152.0,
+ "65": 23002929152.0,
+ "70": 23002929152.0,
+ "75": 23002929152.0,
+ "80": 23002929152.0,
+ "85": 23002929152.0,
+ "90": 23002929152.0,
+ "95": 23002929152.0,
+ "100": 23002929152.0,
+ "105": 23002929152.0,
+ "110": 23002929152.0,
+ "115": 23002929152.0,
+ "120": 23002929152.0,
+ "125": 23002929152.0,
+ "130": 23002929152.0,
+ "135": 23002929152.0,
+ "140": 23002929152.0,
+ "145": 23002929152.0,
+ "150": 23002929152.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 150,
+ "step_interval": 5,
+ "values": {
+ "1": 23002945536.0,
+ "5": 28229746688.0,
+ "10": 28431153152.0,
+ "15": 28431153152.0,
+ "20": 28431153152.0,
+ "25": 28431153152.0,
+ "30": 28431153152.0,
+ "35": 28431153152.0,
+ "40": 28526688256.0,
+ "45": 28526688256.0,
+ "50": 28534185984.0,
+ "55": 28534185984.0,
+ "60": 28534185984.0,
+ "65": 28534185984.0,
+ "70": 28534185984.0,
+ "75": 28534185984.0,
+ "80": 28534185984.0,
+ "85": 28534185984.0,
+ "90": 28534185984.0,
+ "95": 28534185984.0,
+ "100": 28534185984.0,
+ "105": 28534185984.0,
+ "110": 28534185984.0,
+ "115": 28534185984.0,
+ "120": 28534185984.0,
+ "125": 28534185984.0,
+ "130": 28534185984.0,
+ "135": 28534185984.0,
+ "140": 28534185984.0,
+ "145": 28534185984.0,
+ "150": 28534185984.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 150,
+ "step_interval": 5,
+ "values": {
+ "1": "nan",
+ "5": 9.33326,
+ "10": 7.45877,
+ "15": 6.55202,
+ "20": 7.14405,
+ "25": 6.74357,
+ "30": 7.73142,
+ "35": 7.1059,
+ "40": 7.15075,
+ "45": 6.68013,
+ "50": 6.27464,
+ "55": 6.4961,
+ "60": 6.20048,
+ "65": 7.0085,
+ "70": 6.96858,
+ "75": 6.99176,
+ "80": 6.97374,
+ "85": 7.59797,
+ "90": 5.99314,
+ "95": 7.31466,
+ "100": 6.1496,
+ "105": 6.1462,
+ "110": 6.52008,
+ "115": 6.14035,
+ "120": 6.48433,
+ "125": 5.7572,
+ "130": 8.06756,
+ "135": 7.32705,
+ "140": 6.87268,
+ "145": 7.35193,
+ "150": 7.70667
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/mixtral/mixtral_8x7b_tp1pp4ep8vpp8_release/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/mixtral/mixtral_8x7b_tp1pp4ep8vpp8_release/golden_values_dev_dgx_h100.json
index f7a6a80a2b7..8f120988635 100644
--- a/tests/functional_tests/test_cases/mixtral/mixtral_8x7b_tp1pp4ep8vpp8_release/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/mixtral/mixtral_8x7b_tp1pp4ep8vpp8_release/golden_values_dev_dgx_h100.json
@@ -1 +1,242 @@
-{"lm loss": {"start_step": 1, "end_step": 200, "step_interval": 5, "values": {"1": 1.80634, "5": 1.83171, "10": 1.77662, "15": 1.78754, "20": 1.79513, "25": 1.75231, "30": 1.80496, "35": 1.70559, "40": 1.72637, "45": 1.72361, "50": 1.68547, "55": 1.69587, "60": 1.6929, "65": 1.68666, "70": 1.65599, "75": 1.68397, "80": 1.67445, "85": 1.66157, "90": 1.68308, "95": 1.66497, "100": 1.64207, "105": 1.62509, "110": 1.63681, "115": 1.63436, "120": 1.63301, "125": 1.64265, "130": 1.66905, "135": 1.62605, "140": 1.64112, "145": 1.60827, "150": 1.62734, "155": 1.61971, "160": 1.6186, "165": 1.62252, "170": 1.6258, "175": 1.60163, "180": 1.60656, "185": 1.6715, "190": 1.61267, "195": 1.64063, "200": 1.62415}}, "num-zeros": {"start_step": 1, "end_step": 200, "step_interval": 5, "values": {"1": 30107084.0, "5": 32170564.0, "10": 30557572.0, "15": 32510632.0, "20": 33010312.0, "25": 33034532.0, "30": 30938196.0, "35": 31294880.0, "40": 31277928.0, "45": 31540264.0, "50": 30696956.0, "55": 33403104.0, "60": 31335408.0, "65": 32698254.0, "70": 30148036.0, "75": 30827970.0, "80": 32263964.0, "85": 30905140.0, "90": 32120752.0, "95": 31564360.0, "100": 31895836.0, "105": 33071156.0, "110": 33058724.0, "115": 28710640.0, "120": 29922562.0, "125": 31605040.0, "130": 32047268.0, "135": 32227450.0, "140": 31969408.0, "145": 30286728.0, "150": 32616604.0, "155": 33017540.0, "160": 31523356.0, "165": 33046308.0, "170": 31870992.0, "175": 31379856.0, "180": 30557072.0, "185": 31367220.0, "190": 32951800.0, "195": 31584284.0, "200": 30110532.0}}, "mem-allocated-bytes": {"start_step": 1, "end_step": 200, "step_interval": 5, "values": {"1": 19259990016.0, "5": 19259990016.0, "10": 19259990016.0, "15": 19259990016.0, "20": 19259990016.0, "25": 19259990016.0, "30": 19259990016.0, "35": 19259990016.0, "40": 19259990016.0, "45": 19259990016.0, "50": 19259990016.0, "55": 19259990016.0, "60": 19259990016.0, "65": 19259990016.0, "70": 19259990016.0, "75": 19259990016.0, "80": 19259990016.0, "85": 19259990016.0, "90": 19259990016.0, "95": 19259990016.0, "100": 19259990016.0, "105": 19259990016.0, "110": 19259990016.0, "115": 19259990016.0, "120": 19259990016.0, "125": 19259990016.0, "130": 19259990016.0, "135": 19259990016.0, "140": 19259990016.0, "145": 19259990016.0, "150": 19259990016.0, "155": 19259990016.0, "160": 19259990016.0, "165": 19259990016.0, "170": 19259990016.0, "175": 19259990016.0, "180": 19259990016.0, "185": 19259990016.0, "190": 19259990016.0, "195": 19259990016.0, "200": 19259990016.0}}, "mem-max-allocated-bytes": {"start_step": 1, "end_step": 200, "step_interval": 5, "values": {"1": 28930920448.0, "5": 34950066176.0, "10": 34952785920.0, "15": 35008057344.0, "20": 35030818816.0, "25": 35030818816.0, "30": 35043815424.0, "35": 35043815424.0, "40": 35043815424.0, "45": 35043815424.0, "50": 35043815424.0, "55": 35043815424.0, "60": 35043815424.0, "65": 35043815424.0, "70": 35043815424.0, "75": 35043815424.0, "80": 35043815424.0, "85": 35043815424.0, "90": 35043815424.0, "95": 35043815424.0, "100": 35043815424.0, "105": 35043815424.0, "110": 35043815424.0, "115": 35043815424.0, "120": 35043815424.0, "125": 35043815424.0, "130": 35043815424.0, "135": 35043815424.0, "140": 35043815424.0, "145": 35043815424.0, "150": 35043815424.0, "155": 35043815424.0, "160": 35043815424.0, "165": 35043815424.0, "170": 35043815424.0, "175": 35043815424.0, "180": 35043815424.0, "185": 35043815424.0, "190": 35043815424.0, "195": 35043815424.0, "200": 35043815424.0}}, "iteration-time": {"start_step": 1, "end_step": 200, "step_interval": 5, "values": {"1": 98.7801, "5": 4.04517, "10": 3.97416, "15": 3.43908, "20": 3.71676, "25": 3.60356, "30": 3.41843, "35": 3.41826, "40": 3.42692, "45": 3.42386, "50": 3.46431, "55": 3.73202, "60": 3.40706, "65": 3.55895, "70": 3.39731, "75": 3.39689, "80": 3.37554, "85": 3.40303, "90": 3.36983, "95": 3.36802, "100": 3.37332, "105": 3.37588, "110": 3.38044, "115": 3.37542, "120": 3.77014, "125": 3.6352, "130": 3.83078, "135": 3.35818, "140": 3.58233, "145": 3.64723, "150": 3.40026, "155": 3.35303, "160": 3.56001, "165": 3.55831, "170": 3.34244, "175": 3.35938, "180": 3.37525, "185": 3.35005, "190": 3.59691, "195": 3.34804, "200": 3.33378}}}
\ No newline at end of file
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 200,
+ "step_interval": 5,
+ "values": {
+ "1": 1.80632,
+ "5": 1.83184,
+ "10": 1.77661,
+ "15": 1.78762,
+ "20": 1.79521,
+ "25": 1.75237,
+ "30": 1.8052,
+ "35": 1.7058,
+ "40": 1.72645,
+ "45": 1.72384,
+ "50": 1.68568,
+ "55": 1.69619,
+ "60": 1.69328,
+ "65": 1.68708,
+ "70": 1.65636,
+ "75": 1.6844,
+ "80": 1.67457,
+ "85": 1.66171,
+ "90": 1.68318,
+ "95": 1.66504,
+ "100": 1.64206,
+ "105": 1.62512,
+ "110": 1.63696,
+ "115": 1.63439,
+ "120": 1.63309,
+ "125": 1.64257,
+ "130": 1.66907,
+ "135": 1.62597,
+ "140": 1.64125,
+ "145": 1.6082,
+ "150": 1.62734,
+ "155": 1.61977,
+ "160": 1.61879,
+ "165": 1.62264,
+ "170": 1.62567,
+ "175": 1.60156,
+ "180": 1.60662,
+ "185": 1.67142,
+ "190": 1.6127,
+ "195": 1.64064,
+ "200": 1.62423
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 200,
+ "step_interval": 5,
+ "values": {
+ "1": 30107410.0,
+ "5": 32170588.0,
+ "10": 30557604.0,
+ "15": 32510750.0,
+ "20": 33010342.0,
+ "25": 33034544.0,
+ "30": 30938648.0,
+ "35": 31294908.0,
+ "40": 31277976.0,
+ "45": 31540080.0,
+ "50": 30697116.0,
+ "55": 33402768.0,
+ "60": 31335808.0,
+ "65": 32698444.0,
+ "70": 30147920.0,
+ "75": 30827774.0,
+ "80": 32264272.0,
+ "85": 30905426.0,
+ "90": 32120732.0,
+ "95": 31564122.0,
+ "100": 31896178.0,
+ "105": 33071324.0,
+ "110": 33058808.0,
+ "115": 28710980.0,
+ "120": 29922406.0,
+ "125": 31604900.0,
+ "130": 32047562.0,
+ "135": 32227678.0,
+ "140": 31969464.0,
+ "145": 30286908.0,
+ "150": 32616544.0,
+ "155": 33017648.0,
+ "160": 31523086.0,
+ "165": 33046236.0,
+ "170": 31871092.0,
+ "175": 31379762.0,
+ "180": 30557114.0,
+ "185": 31367138.0,
+ "190": 32951924.0,
+ "195": 31584384.0,
+ "200": 30110236.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 200,
+ "step_interval": 5,
+ "values": {
+ "1": 19743387648.0,
+ "5": 19743387648.0,
+ "10": 19743387648.0,
+ "15": 19743387648.0,
+ "20": 19743387648.0,
+ "25": 19743387648.0,
+ "30": 19743387648.0,
+ "35": 19743387648.0,
+ "40": 19743387648.0,
+ "45": 19743387648.0,
+ "50": 19743387648.0,
+ "55": 19743387648.0,
+ "60": 19743387648.0,
+ "65": 19743387648.0,
+ "70": 19743387648.0,
+ "75": 19743387648.0,
+ "80": 19743387648.0,
+ "85": 19743387648.0,
+ "90": 19743387648.0,
+ "95": 19743387648.0,
+ "100": 19743387648.0,
+ "105": 19743387648.0,
+ "110": 19743387648.0,
+ "115": 19743387648.0,
+ "120": 19743387648.0,
+ "125": 19743387648.0,
+ "130": 19743387648.0,
+ "135": 19743387648.0,
+ "140": 19743387648.0,
+ "145": 19743387648.0,
+ "150": 19743387648.0,
+ "155": 19743387648.0,
+ "160": 19743387648.0,
+ "165": 19743387648.0,
+ "170": 19743387648.0,
+ "175": 19743387648.0,
+ "180": 19743387648.0,
+ "185": 19743387648.0,
+ "190": 19743387648.0,
+ "195": 19743387648.0,
+ "200": 19743387648.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 200,
+ "step_interval": 5,
+ "values": {
+ "1": 29394774016.0,
+ "5": 35404840960.0,
+ "10": 35417063424.0,
+ "15": 35463147520.0,
+ "20": 35497099264.0,
+ "25": 35497099264.0,
+ "30": 35508408320.0,
+ "35": 35508408320.0,
+ "40": 35508408320.0,
+ "45": 35508408320.0,
+ "50": 35508408320.0,
+ "55": 35508408320.0,
+ "60": 35508408320.0,
+ "65": 35508408320.0,
+ "70": 35508408320.0,
+ "75": 35508408320.0,
+ "80": 35508408320.0,
+ "85": 35508408320.0,
+ "90": 35508408320.0,
+ "95": 35508408320.0,
+ "100": 35508408320.0,
+ "105": 35508408320.0,
+ "110": 35508408320.0,
+ "115": 35508408320.0,
+ "120": 35508408320.0,
+ "125": 35508408320.0,
+ "130": 35508408320.0,
+ "135": 35508408320.0,
+ "140": 35508408320.0,
+ "145": 35508408320.0,
+ "150": 35508408320.0,
+ "155": 35508408320.0,
+ "160": 35508408320.0,
+ "165": 35508408320.0,
+ "170": 35508408320.0,
+ "175": 35508408320.0,
+ "180": 35508408320.0,
+ "185": 35508408320.0,
+ "190": 35508408320.0,
+ "195": 35508408320.0,
+ "200": 35508408320.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 200,
+ "step_interval": 5,
+ "values": {
+ "1": "nan",
+ "5": 3.47451,
+ "10": 3.31169,
+ "15": 3.4018,
+ "20": 3.32494,
+ "25": 3.99476,
+ "30": 3.31795,
+ "35": 3.30721,
+ "40": 3.2989,
+ "45": 3.29245,
+ "50": 3.73078,
+ "55": 4.2852,
+ "60": 3.2697,
+ "65": 3.27747,
+ "70": 3.49739,
+ "75": 3.28191,
+ "80": 3.26247,
+ "85": 3.27631,
+ "90": 3.2584,
+ "95": 3.26058,
+ "100": 3.26095,
+ "105": 3.2444,
+ "110": 3.24824,
+ "115": 3.25435,
+ "120": 3.23352,
+ "125": 3.2335,
+ "130": 3.22,
+ "135": 3.23003,
+ "140": 3.22953,
+ "145": 3.22956,
+ "150": 3.26259,
+ "155": 3.21536,
+ "160": 3.23325,
+ "165": 3.91941,
+ "170": 3.22964,
+ "175": 3.22433,
+ "180": 3.21506,
+ "185": 3.567,
+ "190": 3.59199,
+ "195": 3.22657,
+ "200": 3.56689
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2/golden_values_dev_dgx_gb200.json
new file mode 100644
index 00000000000..fe90db38692
--- /dev/null
+++ b/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2/golden_values_dev_dgx_gb200.json
@@ -0,0 +1,230 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 10.90136,
+ "2": 10.93243,
+ "3": 10.33241,
+ "4": 9.85909,
+ "5": 9.95151,
+ "6": 9.89199,
+ "7": 9.50952,
+ "8": 9.40114,
+ "9": 9.34537,
+ "10": 9.0064,
+ "11": 9.52172,
+ "12": 9.31336,
+ "13": 9.30973,
+ "14": 9.31828,
+ "15": 8.9847,
+ "16": 8.91893,
+ "17": 8.74289,
+ "18": 8.83017,
+ "19": 8.77961,
+ "20": 8.54805,
+ "21": 8.60956,
+ "22": 8.21981,
+ "23": 8.68472,
+ "24": 8.32456,
+ "25": 8.1534,
+ "26": 8.38441,
+ "27": 8.48989,
+ "28": 8.61321,
+ "29": 8.60404,
+ "30": 8.13429,
+ "31": 7.61439,
+ "32": 8.36535,
+ "33": 8.45189,
+ "34": 7.95724,
+ "35": 8.04342,
+ "36": 7.96872,
+ "37": 8.18014,
+ "38": 7.93696,
+ "39": 8.30058,
+ "40": 7.9189,
+ "41": 7.94527,
+ "42": 8.13206,
+ "43": 7.71448,
+ "44": 7.81544,
+ "45": 7.69727,
+ "46": 7.79162,
+ "47": 8.05796,
+ "48": 7.78927,
+ "49": 7.45437,
+ "50": 7.88942
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 362722816.0,
+ "2": 362724352.0,
+ "3": 362724352.0,
+ "4": 362724352.0,
+ "5": 362724352.0,
+ "6": 362724352.0,
+ "7": 362724352.0,
+ "8": 362724352.0,
+ "9": 362724352.0,
+ "10": 362724352.0,
+ "11": 362724352.0,
+ "12": 362724352.0,
+ "13": 362724352.0,
+ "14": 362724352.0,
+ "15": 362724352.0,
+ "16": 362724352.0,
+ "17": 362724352.0,
+ "18": 362724352.0,
+ "19": 362724352.0,
+ "20": 362724352.0,
+ "21": 362724352.0,
+ "22": 362724352.0,
+ "23": 362724352.0,
+ "24": 362724352.0,
+ "25": 362724352.0,
+ "26": 362724352.0,
+ "27": 362724352.0,
+ "28": 362724352.0,
+ "29": 362724352.0,
+ "30": 362724352.0,
+ "31": 362724352.0,
+ "32": 362724352.0,
+ "33": 362724352.0,
+ "34": 362724352.0,
+ "35": 362724352.0,
+ "36": 362724352.0,
+ "37": 362724352.0,
+ "38": 362724352.0,
+ "39": 362724352.0,
+ "40": 362724352.0,
+ "41": 362724352.0,
+ "42": 362724352.0,
+ "43": 362724352.0,
+ "44": 362724352.0,
+ "45": 362724352.0,
+ "46": 362724352.0,
+ "47": 362724352.0,
+ "48": 362724352.0,
+ "49": 362724352.0,
+ "50": 362724352.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 2253365760.0,
+ "2": 2310949888.0,
+ "3": 2310949888.0,
+ "4": 2310949888.0,
+ "5": 2310949888.0,
+ "6": 2310949888.0,
+ "7": 2310949888.0,
+ "8": 2310949888.0,
+ "9": 2310949888.0,
+ "10": 2310949888.0,
+ "11": 2310949888.0,
+ "12": 2310949888.0,
+ "13": 2310949888.0,
+ "14": 2310949888.0,
+ "15": 2310949888.0,
+ "16": 2310949888.0,
+ "17": 2310949888.0,
+ "18": 2310949888.0,
+ "19": 2310949888.0,
+ "20": 2310949888.0,
+ "21": 2310949888.0,
+ "22": 2310949888.0,
+ "23": 2310949888.0,
+ "24": 2310949888.0,
+ "25": 2310949888.0,
+ "26": 2310949888.0,
+ "27": 2310949888.0,
+ "28": 2310949888.0,
+ "29": 2310949888.0,
+ "30": 2310949888.0,
+ "31": 2310949888.0,
+ "32": 2310949888.0,
+ "33": 2310949888.0,
+ "34": 2310949888.0,
+ "35": 2310949888.0,
+ "36": 2310949888.0,
+ "37": 2310949888.0,
+ "38": 2310949888.0,
+ "39": 2310949888.0,
+ "40": 2310949888.0,
+ "41": 2310949888.0,
+ "42": 2310949888.0,
+ "43": 2310949888.0,
+ "44": 2310949888.0,
+ "45": 2310949888.0,
+ "46": 2310949888.0,
+ "47": 2310949888.0,
+ "48": 2310949888.0,
+ "49": 2310949888.0,
+ "50": 2310949888.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": "nan",
+ "2": 14.57973,
+ "3": 0.13942,
+ "4": 0.0953,
+ "5": 0.10021,
+ "6": 0.08408,
+ "7": 0.07956,
+ "8": 0.08502,
+ "9": 0.09057,
+ "10": 0.08151,
+ "11": 0.08813,
+ "12": 0.08231,
+ "13": 0.08682,
+ "14": 0.07844,
+ "15": 0.08209,
+ "16": 0.12736,
+ "17": 0.07926,
+ "18": 0.07611,
+ "19": 0.07869,
+ "20": 0.11322,
+ "21": 0.07601,
+ "22": 0.07903,
+ "23": 0.07908,
+ "24": 0.0752,
+ "25": 0.07767,
+ "26": 0.07343,
+ "27": 0.07559,
+ "28": 0.07816,
+ "29": 0.07916,
+ "30": 0.07536,
+ "31": 0.07278,
+ "32": 0.07391,
+ "33": 0.08204,
+ "34": 0.07679,
+ "35": 0.07553,
+ "36": 0.07363,
+ "37": 0.0811,
+ "38": 0.07582,
+ "39": 0.07516,
+ "40": 0.07486,
+ "41": 0.07675,
+ "42": 0.07277,
+ "43": 0.07554,
+ "44": 0.07781,
+ "45": 0.07609,
+ "46": 0.08749,
+ "47": 0.07566,
+ "48": 0.07713,
+ "49": 0.07571,
+ "50": 0.07904
+ }
+ }
+}
diff --git a/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2/model_config.yaml b/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2/model_config.yaml
new file mode 100644
index 00000000000..7b820614bf8
--- /dev/null
+++ b/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2/model_config.yaml
@@ -0,0 +1,145 @@
+# DeepSeek-style proxy: 2 layers — first dense, second MoE with expert parallelism 4.
+# Used for functional testing of dense+MoE hybrid and EP.
+ENV_VARS:
+ NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
+ PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
+ NCCL_NVLS_ENABLE: 0
+ NCCL_DEBUG: VERSION
+ NCCL_ALGO: Ring
+
+MODEL_ARGS:
+
+# Distributed args
+ --distributed-timeout-minutes: 60
+ --tensor-model-parallel-size: 1
+ --pipeline-model-parallel-size: 1
+ --expert-model-parallel-size: 2
+ --context-parallel-size: 1
+ --expert-tensor-parallel-size: 1
+ --use-distributed-optimizer: true
+ --overlap-grad-reduce: true
+ --overlap-param-gather: true
+ --attention-backend: unfused
+
+ --num-distributed-optimizer-instances: 1
+
+ # FSDP args
+ --use-megatron-fsdp: true
+ --data-parallel-sharding-strategy: "optim_grads_params"
+ --outer-dp-sharding-strategy: "no_shard"
+
+ --deterministic-mode: true
+ --ckpt-format: "fsdp_dtensor"
+ --no-gradient-accumulation-fusion: true
+
+# Precision
+ --fp8-format: "e4m3"
+ --fp8-recipe: "tensorwise"
+ --fp8-param-gather: true
+
+
+# Training args
+ --use-mcore-models: true
+ --sequence-parallel: true
+ --disable-bias-linear: true
+ --micro-batch-size: 4
+ --global-batch-size: 32
+ --train-iters: 50
+ --exit-duration-in-mins: 60
+ --no-check-for-nan-in-loss-and-grad: true
+ --no-rope-fusion: true
+ --manual-gc: true
+ --manual-gc-interval: 100
+ --transformer-impl: transformer_engine
+
+ # Data args
+ --seq-length: 1024
+ --data-cache-path: ${DATA_CACHE_PATH}
+ --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --split: 949,50,1
+
+
+ # Network: 2 layers — first dense, second MoE
+ --num-layers: 2
+ --hidden-size: 512
+ --ffn-hidden-size: 2048
+ --num-attention-heads: 8
+ --kv-channels: 128
+ --max-position-embeddings: 1024
+ --position-embedding-type: rope
+ --rotary-base: 10000
+ --make-vocab-size-divisible-by: 64
+ --normalization: RMSNorm
+ --norm-epsilon: 1e-6
+ --swiglu: true
+ --untie-embeddings-and-output-weights: true
+ --multi-latent-attention: true
+ # No MTP for this 2-layer proxy
+ # Add regularization args
+ --attention-dropout: 0.0
+ --hidden-dropout: 0.0
+ --clip-grad: 1.0
+ --weight-decay: 0.1
+ --qk-layernorm: true
+ # Learning rate args
+ --lr-warmup-fraction: .01
+ --lr: 0.00015
+ --min-lr: 1.0e-5
+ --lr-decay-style: cosine
+ --adam-beta1: 0.9
+ --adam-beta2: 0.95
+ # MoE args (DeepSeek-style): 1 dense layer, 1 MoE layer, ep=4
+ --num-experts: 8
+ --moe-layer-freq: ([0]*1+[1]*1)
+ --moe-ffn-hidden-size: 1024
+ --moe-shared-expert-intermediate-size: 1024
+ --moe-router-load-balancing-type: seq_aux_loss
+ --moe-router-topk: 4
+ --moe-token-dispatcher-type: alltoall
+ --moe-router-pre-softmax: true
+ --moe-grouped-gemm: true
+ --moe-aux-loss-coeff: 1e-4
+ --moe-router-group-topk: 2
+ --moe-router-num-groups: 4
+ --moe-router-topk-scaling-factor: 2.0
+ --moe-router-score-function: sigmoid
+ --moe-router-enable-expert-bias: true
+ --moe-router-bias-update-rate: 1e-3
+ --moe-router-dtype: fp32
+ --moe-permute-fusion: true
+ # MLA args (DeepSeek-style)
+ --q-lora-rank: 256
+ --kv-lora-rank: 128
+ --qk-head-dim: 128
+ --qk-pos-emb-head-dim: 64
+ --v-head-dim: 128
+ --rotary-scaling-factor: 1.0
+ --mscale: 1.0
+ --mscale-all-dim: 1.0
+ # Validation, checkpointing, logging
+ --eval-iters: 10
+ --eval-interval: 200
+ #--save: ${CHECKPOINT_SAVE_PATH}
+ #--load: ${CHECKPOINT_LOAD_PATH}
+ #--ckpt-format: fsdp_dtensor
+ --save-interval: 25
+ --init-method-std: 0.02
+ --log-timers-to-tensorboard: true
+ --log-memory-to-tensorboard: true
+ --log-num-zeros-in-grad: true
+ --log-params-norm: true
+ --log-validation-ppl-to-tensorboard: true
+ --log-throughput: true
+ --log-interval: 1
+ --logging-level: 10
+ --tensorboard-dir: ${TENSORBOARD_PATH}
+
+ --exit-interval: 50
+TEST_TYPE: regular
+METRICS:
+ - "iteration-time"
+ - "lm loss"
+ - "mem-allocated-bytes"
+ - "mem-max-allocated-bytes"
diff --git a/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2_1node/golden_values_dev_dgx_gb200.json
new file mode 100644
index 00000000000..b0d86902adb
--- /dev/null
+++ b/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2_1node/golden_values_dev_dgx_gb200.json
@@ -0,0 +1,287 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 10.90135,
+ "2": 10.93243,
+ "3": 10.64754,
+ "4": 10.41167,
+ "5": 10.40069,
+ "6": 10.36614,
+ "7": 10.11277,
+ "8": 9.90182,
+ "9": 9.96398,
+ "10": 9.51268,
+ "11": 10.16304,
+ "12": 9.86191,
+ "13": 9.87018,
+ "14": 9.89993,
+ "15": 9.50655,
+ "16": 9.45032,
+ "17": 9.26003,
+ "18": 9.30218,
+ "19": 9.20949,
+ "20": 8.9719,
+ "21": 9.00542,
+ "22": 8.60675,
+ "23": 9.09476,
+ "24": 8.68458,
+ "25": 8.50639,
+ "26": 8.73839,
+ "27": 8.82732,
+ "28": 8.95619,
+ "29": 8.94384,
+ "30": 8.49379,
+ "31": 7.96025,
+ "32": 8.67938,
+ "33": 8.74669,
+ "34": 8.26615,
+ "35": 8.35472,
+ "36": 8.27002,
+ "37": 8.50763,
+ "38": 8.26683,
+ "39": 8.63275,
+ "40": 8.22562,
+ "41": 8.27215,
+ "42": 8.42439,
+ "43": 8.00629,
+ "44": 8.11473,
+ "45": 7.99682,
+ "46": 8.08318,
+ "47": 8.38377,
+ "48": 8.09378,
+ "49": 7.72935,
+ "50": 8.19223
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 30095348.0,
+ "2": 30298064.0,
+ "3": 29950172.0,
+ "4": 30612092.0,
+ "5": 30081950.0,
+ "6": 30408494.0,
+ "7": 30139284.0,
+ "8": 30307504.0,
+ "9": 30212912.0,
+ "10": 30297206.0,
+ "11": 29846428.0,
+ "12": 29802964.0,
+ "13": 30294772.0,
+ "14": 29730990.0,
+ "15": 30193332.0,
+ "16": 30203586.0,
+ "17": 30183556.0,
+ "18": 29936824.0,
+ "19": 29974044.0,
+ "20": 30059084.0,
+ "21": 30108760.0,
+ "22": 30164962.0,
+ "23": 29892328.0,
+ "24": 30141572.0,
+ "25": 30189138.0,
+ "26": 29897096.0,
+ "27": 29812076.0,
+ "28": 29801112.0,
+ "29": 29881432.0,
+ "30": 29982158.0,
+ "31": 30335166.0,
+ "32": 29941260.0,
+ "33": 29913044.0,
+ "34": 30203188.0,
+ "35": 30161196.0,
+ "36": 29942784.0,
+ "37": 29846856.0,
+ "38": 30273732.0,
+ "39": 30172416.0,
+ "40": 30022848.0,
+ "41": 30018664.0,
+ "42": 30029684.0,
+ "43": 30361156.0,
+ "44": 30112332.0,
+ "45": 30026360.0,
+ "46": 30258968.0,
+ "47": 29989716.0,
+ "48": 30301304.0,
+ "49": 30085556.0,
+ "50": 30278752.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 544347136.0,
+ "2": 544348672.0,
+ "3": 544348672.0,
+ "4": 544348672.0,
+ "5": 544348672.0,
+ "6": 544348672.0,
+ "7": 544348672.0,
+ "8": 544348672.0,
+ "9": 544348672.0,
+ "10": 544348672.0,
+ "11": 544348672.0,
+ "12": 544348672.0,
+ "13": 544348672.0,
+ "14": 544348672.0,
+ "15": 544348672.0,
+ "16": 544348672.0,
+ "17": 544348672.0,
+ "18": 544348672.0,
+ "19": 544348672.0,
+ "20": 544348672.0,
+ "21": 544348672.0,
+ "22": 544348672.0,
+ "23": 544348672.0,
+ "24": 544348672.0,
+ "25": 544348672.0,
+ "26": 544348672.0,
+ "27": 544348672.0,
+ "28": 544348672.0,
+ "29": 544348672.0,
+ "30": 544348672.0,
+ "31": 544348672.0,
+ "32": 544348672.0,
+ "33": 544348672.0,
+ "34": 544348672.0,
+ "35": 544348672.0,
+ "36": 544348672.0,
+ "37": 544348672.0,
+ "38": 544348672.0,
+ "39": 544348672.0,
+ "40": 544348672.0,
+ "41": 544348672.0,
+ "42": 544348672.0,
+ "43": 544348672.0,
+ "44": 544348672.0,
+ "45": 544348672.0,
+ "46": 544348672.0,
+ "47": 544348672.0,
+ "48": 544348672.0,
+ "49": 544348672.0,
+ "50": 544348672.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": 2343199232.0,
+ "2": 2472750080.0,
+ "3": 2493632512.0,
+ "4": 2493632512.0,
+ "5": 2493632512.0,
+ "6": 2493632512.0,
+ "7": 2493632512.0,
+ "8": 2493632512.0,
+ "9": 2493632512.0,
+ "10": 2493632512.0,
+ "11": 2493632512.0,
+ "12": 2493632512.0,
+ "13": 2493632512.0,
+ "14": 2493632512.0,
+ "15": 2493632512.0,
+ "16": 2493632512.0,
+ "17": 2493632512.0,
+ "18": 2493632512.0,
+ "19": 2493632512.0,
+ "20": 2493632512.0,
+ "21": 2493632512.0,
+ "22": 2493632512.0,
+ "23": 2493632512.0,
+ "24": 2493632512.0,
+ "25": 2493632512.0,
+ "26": 2493632512.0,
+ "27": 2493632512.0,
+ "28": 2493632512.0,
+ "29": 2493632512.0,
+ "30": 2493632512.0,
+ "31": 2493632512.0,
+ "32": 2493632512.0,
+ "33": 2493632512.0,
+ "34": 2493632512.0,
+ "35": 2493632512.0,
+ "36": 2493632512.0,
+ "37": 2493632512.0,
+ "38": 2493632512.0,
+ "39": 2495503872.0,
+ "40": 2495503872.0,
+ "41": 2495503872.0,
+ "42": 2495503872.0,
+ "43": 2495503872.0,
+ "44": 2495503872.0,
+ "45": 2502984192.0,
+ "46": 2502984192.0,
+ "47": 2502984192.0,
+ "48": 2502984192.0,
+ "49": 2502984192.0,
+ "50": 2502984192.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 50,
+ "step_interval": 1,
+ "values": {
+ "1": "nan",
+ "2": 13.99138,
+ "3": 0.14064,
+ "4": 0.54109,
+ "5": 0.77559,
+ "6": 1.00585,
+ "7": 0.32296,
+ "8": 0.50468,
+ "9": 0.77725,
+ "10": 0.67454,
+ "11": 0.52958,
+ "12": 0.95906,
+ "13": 0.42026,
+ "14": 0.38992,
+ "15": 0.41075,
+ "16": 0.45549,
+ "17": 0.61248,
+ "18": 0.8595,
+ "19": 1.00506,
+ "20": 0.45817,
+ "21": 0.61284,
+ "22": 0.4656,
+ "23": 1.29112,
+ "24": 0.56122,
+ "25": 0.26919,
+ "26": 0.3938,
+ "27": 1.59488,
+ "28": 0.74547,
+ "29": 1.12729,
+ "30": 0.97505,
+ "31": 1.67021,
+ "32": 0.52422,
+ "33": 1.69778,
+ "34": 1.09099,
+ "35": 0.689,
+ "36": 1.51307,
+ "37": 1.042,
+ "38": 0.56121,
+ "39": 1.57156,
+ "40": 1.39225,
+ "41": 2.0502,
+ "42": 1.04844,
+ "43": 0.85072,
+ "44": 1.84513,
+ "45": 0.98738,
+ "46": 0.61715,
+ "47": 1.79248,
+ "48": 1.14581,
+ "49": 1.2607,
+ "50": 2.07325
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2_1node/model_config.yaml b/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2_1node/model_config.yaml
new file mode 100644
index 00000000000..a243bcf6b84
--- /dev/null
+++ b/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2_1node/model_config.yaml
@@ -0,0 +1,141 @@
+# DeepSeek-style proxy: 2 layers — first dense, second MoE with expert parallelism 4.
+# Used for functional testing of dense+MoE hybrid and EP.
+ENV_VARS:
+ NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
+ PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
+ NCCL_NVLS_ENABLE: 0
+ NCCL_DEBUG: VERSION
+ NCCL_ALGO: Ring
+
+MODEL_ARGS:
+ # Distributed args
+ --distributed-timeout-minutes: 60
+ --tensor-model-parallel-size: 1
+ --pipeline-model-parallel-size: 1
+ --expert-model-parallel-size: 2
+ --context-parallel-size: 1
+ --expert-tensor-parallel-size: 1
+ --use-distributed-optimizer: true
+ --overlap-grad-reduce: true
+ --overlap-param-gather: true
+ --attention-backend: unfused
+
+ --num-distributed-optimizer-instances: 1
+
+ # FSDP args
+ --use-megatron-fsdp: true
+ --data-parallel-sharding-strategy: "optim_grads_params"
+ --outer-dp-sharding-strategy: "no_shard"
+
+ --deterministic-mode: true
+ --ckpt-format: "fsdp_dtensor"
+ --no-gradient-accumulation-fusion: true
+
+ # Precision
+ --fp8-format: "e4m3"
+ --fp8-recipe: "tensorwise"
+ --fp8-param-gather: true
+
+ # Training args
+ --use-mcore-models: true
+ --sequence-parallel: true
+ --disable-bias-linear: true
+ --micro-batch-size: 4
+ --global-batch-size: 32
+ --train-iters: 50
+ --exit-duration-in-mins: 60
+ --no-check-for-nan-in-loss-and-grad: true
+ --no-rope-fusion: true
+ --manual-gc: true
+ --manual-gc-interval: 100
+ --transformer-impl: transformer_engine
+
+ # Data args
+ --seq-length: 1024
+ --data-cache-path: ${DATA_CACHE_PATH}
+ --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --split: 949,50,1
+
+ # Network: 2 layers — first dense, second MoE
+ --num-layers: 2
+ --hidden-size: 512
+ --ffn-hidden-size: 2048
+ --num-attention-heads: 8
+ --kv-channels: 128
+ --max-position-embeddings: 1024
+ --position-embedding-type: rope
+ --rotary-base: 10000
+ --make-vocab-size-divisible-by: 64
+ --normalization: RMSNorm
+ --norm-epsilon: 1e-6
+ --swiglu: true
+ --untie-embeddings-and-output-weights: true
+ --multi-latent-attention: true
+ # No MTP for this 2-layer proxy
+ # Add regularization args
+ --attention-dropout: 0.0
+ --hidden-dropout: 0.0
+ --clip-grad: 1.0
+ --weight-decay: 0.1
+ --qk-layernorm: true
+ # Learning rate args
+ --lr-warmup-fraction: .01
+ --lr: 0.00015
+ --min-lr: 1.0e-5
+ --lr-decay-style: cosine
+ --adam-beta1: 0.9
+ --adam-beta2: 0.95
+ # MoE args (DeepSeek-style): 1 dense layer, 1 MoE layer, ep=4
+ --num-experts: 8
+ --moe-layer-freq: ([0]*1+[1]*1)
+ --moe-ffn-hidden-size: 1024
+ --moe-shared-expert-intermediate-size: 1024
+ --moe-router-load-balancing-type: seq_aux_loss
+ --moe-router-topk: 4
+ --moe-token-dispatcher-type: alltoall
+ --moe-router-pre-softmax: true
+ --moe-grouped-gemm: true
+ --moe-aux-loss-coeff: 1e-4
+ --moe-router-group-topk: 2
+ --moe-router-num-groups: 4
+ --moe-router-topk-scaling-factor: 2.0
+ --moe-router-score-function: sigmoid
+ --moe-router-enable-expert-bias: true
+ --moe-router-bias-update-rate: 1e-3
+ --moe-router-dtype: fp32
+ --moe-permute-fusion: true
+ # MLA args (DeepSeek-style)
+ --q-lora-rank: 256
+ --kv-lora-rank: 128
+ --qk-head-dim: 128
+ --qk-pos-emb-head-dim: 64
+ --v-head-dim: 128
+ --rotary-scaling-factor: 1.0
+ --mscale: 1.0
+ --mscale-all-dim: 1.0
+ # Validation, checkpointing, logging
+ --eval-iters: 10
+ --eval-interval: 200
+ #--save: ${CHECKPOINT_SAVE_PATH}
+ #--load: ${CHECKPOINT_LOAD_PATH}
+ #--ckpt-format: fsdp_dtensor
+ --save-interval: 25
+ --init-method-std: 0.02
+ --log-timers-to-tensorboard: true
+ --log-memory-to-tensorboard: true
+ --log-num-zeros-in-grad: true
+ --log-params-norm: true
+ --log-validation-ppl-to-tensorboard: true
+ --log-throughput: true
+ --log-interval: 1
+ --logging-level: 10
+ --tensorboard-dir: ${TENSORBOARD_PATH}
+
+ --exit-interval: 50
+TEST_TYPE: regular
+METRICS:
+ - "lm loss"
+ - "mem-allocated-bytes"
+ - "mem-max-allocated-bytes"
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_cp2_pp2_ep2_te_4experts2parallel_nondeterministic/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_cp2_pp2_ep2_te_4experts2parallel_nondeterministic/model_config.yaml
index cdabc4b6225..b3e8a82de72 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_cp2_pp2_ep2_te_4experts2parallel_nondeterministic/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_cp2_pp2_ep2_te_4experts2parallel_nondeterministic/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -56,4 +56,6 @@ MODEL_ARGS:
--disable-bias-linear: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_cp2_pp2_ep2_te_4experts2parallel_nondeterministic_dp_last/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_cp2_pp2_ep2_te_4experts2parallel_nondeterministic_dp_last/model_config.yaml
index 731ff82d8d4..59887d4eec9 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_cp2_pp2_ep2_te_4experts2parallel_nondeterministic_dp_last/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_cp2_pp2_ep2_te_4experts2parallel_nondeterministic_dp_last/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -58,4 +58,6 @@ MODEL_ARGS:
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
--use-tp-pp-dp-mapping: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp1_pp2_resume_torch_dist_reshard_2x1x4_te_8experts2parallel_dist_optimizer/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp1_pp2_resume_torch_dist_reshard_2x1x4_te_8experts2parallel_dist_optimizer/model_config.yaml
index f7fd8b2963d..4b9f7b03ace 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp1_pp2_resume_torch_dist_reshard_2x1x4_te_8experts2parallel_dist_optimizer/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp1_pp2_resume_torch_dist_reshard_2x1x4_te_8experts2parallel_dist_optimizer/model_config.yaml
@@ -19,9 +19,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -56,4 +56,6 @@ MODEL_ARGS:
--disable-bias-linear: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_te_8experts2parallel_dist_optimizer/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_te_8experts2parallel_dist_optimizer/model_config.yaml
index 61b5c9339ba..c4bc4528090 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_te_8experts2parallel_dist_optimizer/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_te_8experts2parallel_dist_optimizer/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -60,4 +60,6 @@ MODEL_ARGS:
--disable-bias-linear: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: frozen-resume
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_te_8experts2parallel_groupedGEMM/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_te_8experts2parallel_groupedGEMM/model_config.yaml
index a3995df9627..bdef7c88323 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_te_8experts2parallel_groupedGEMM/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_te_8experts2parallel_groupedGEMM/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -56,4 +56,6 @@ MODEL_ARGS:
--bf16: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: frozen-resume
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer/golden_values_dev_dgx_gb200.json
index 3dd007cc9ec..4c4c8c0c0cf 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer/golden_values_dev_dgx_gb200.json
@@ -6,104 +6,104 @@
"values": {
"1": 10.7915,
"2": 10.8072,
- "3": 10.79214,
- "4": 10.77571,
- "5": 10.82495,
- "6": 10.83191,
- "7": 10.82037,
- "8": 10.81565,
- "9": 10.81976,
- "10": 10.7695,
- "11": 10.8454,
- "12": 10.83063,
- "13": 10.83651,
- "14": 10.85696,
- "15": 10.80631,
- "16": 10.78763,
- "17": 10.75856,
- "18": 10.79234,
- "19": 10.78331,
- "20": 10.73181,
- "21": 10.71017,
- "22": 10.57574,
- "23": 10.71599,
- "24": 10.62049,
- "25": 10.58266,
- "26": 10.61764,
- "27": 10.65105,
- "28": 10.63303,
- "29": 10.63022,
- "30": 10.44254,
- "31": 10.20049,
- "32": 10.52014,
- "33": 10.50814,
- "34": 10.29535,
- "35": 10.33643,
- "36": 10.30247,
- "37": 10.41766,
- "38": 10.28067,
- "39": 10.46149,
- "40": 10.18213,
- "41": 10.21349,
- "42": 10.28426,
- "43": 9.9557,
- "44": 10.05793,
- "45": 9.9574,
- "46": 9.93571,
- "47": 10.22719,
- "48": 9.96561,
- "49": 9.66581,
- "50": 10.00922,
- "51": 9.94826,
- "52": 9.84653,
- "53": 10.14876,
- "54": 10.03737,
- "55": 9.97454,
- "56": 9.71384,
- "57": 9.5955,
- "58": 9.92044,
- "59": 9.67604,
- "60": 9.61264,
- "61": 9.79194,
- "62": 10.05699,
- "63": 9.47838,
- "64": 9.84479,
- "65": 9.03861,
- "66": 9.78386,
- "67": 9.43595,
- "68": 9.85188,
- "69": 9.84445,
- "70": 9.79288,
- "71": 9.69163,
- "72": 9.64893,
- "73": 9.55502,
- "74": 9.04736,
- "75": 9.49186,
- "76": 9.17766,
- "77": 10.11289,
- "78": 9.7687,
- "79": 9.43966,
- "80": 9.45416,
- "81": 9.53142,
- "82": 9.7541,
- "83": 9.38201,
- "84": 9.46121,
- "85": 9.66928,
- "86": 9.13531,
- "87": 9.63413,
- "88": 9.8011,
- "89": 9.66658,
- "90": 9.86173,
- "91": 9.39963,
- "92": 9.41066,
- "93": 9.14665,
- "94": 8.8869,
- "95": 9.56959,
- "96": 9.57609,
- "97": 9.34309,
- "98": 9.72749,
- "99": 8.96222,
- "100": 9.44903
+ "3": 10.79211,
+ "4": 10.77572,
+ "5": 10.82476,
+ "6": 10.83254,
+ "7": 10.81992,
+ "8": 10.81584,
+ "9": 10.81974,
+ "10": 10.76945,
+ "11": 10.84429,
+ "12": 10.83026,
+ "13": 10.83727,
+ "14": 10.857,
+ "15": 10.80627,
+ "16": 10.78742,
+ "17": 10.75904,
+ "18": 10.79229,
+ "19": 10.78343,
+ "20": 10.73204,
+ "21": 10.70924,
+ "22": 10.57637,
+ "23": 10.7162,
+ "24": 10.62002,
+ "25": 10.58213,
+ "26": 10.61692,
+ "27": 10.65114,
+ "28": 10.63239,
+ "29": 10.63059,
+ "30": 10.44345,
+ "31": 10.19924,
+ "32": 10.5204,
+ "33": 10.50791,
+ "34": 10.29555,
+ "35": 10.33583,
+ "36": 10.30347,
+ "37": 10.41745,
+ "38": 10.28027,
+ "39": 10.46232,
+ "40": 10.18155,
+ "41": 10.21302,
+ "42": 10.28416,
+ "43": 9.95622,
+ "44": 10.0575,
+ "45": 9.95696,
+ "46": 9.93598,
+ "47": 10.22691,
+ "48": 9.96494,
+ "49": 9.66599,
+ "50": 10.00894,
+ "51": 9.9482,
+ "52": 9.84637,
+ "53": 10.14872,
+ "54": 10.0373,
+ "55": 9.97437,
+ "56": 9.71411,
+ "57": 9.5952,
+ "58": 9.92061,
+ "59": 9.67629,
+ "60": 9.61278,
+ "61": 9.79184,
+ "62": 10.05672,
+ "63": 9.478,
+ "64": 9.84502,
+ "65": 9.03853,
+ "66": 9.78391,
+ "67": 9.43621,
+ "68": 9.85197,
+ "69": 9.84485,
+ "70": 9.79321,
+ "71": 9.6918,
+ "72": 9.64914,
+ "73": 9.5548,
+ "74": 9.04782,
+ "75": 9.49197,
+ "76": 9.17823,
+ "77": 10.11286,
+ "78": 9.76804,
+ "79": 9.43992,
+ "80": 9.45387,
+ "81": 9.53151,
+ "82": 9.7542,
+ "83": 9.38211,
+ "84": 9.46106,
+ "85": 9.66973,
+ "86": 9.13544,
+ "87": 9.63433,
+ "88": 9.80069,
+ "89": 9.66687,
+ "90": 9.86192,
+ "91": 9.39952,
+ "92": 9.41067,
+ "93": 9.1467,
+ "94": 8.88715,
+ "95": 9.56966,
+ "96": 9.57637,
+ "97": 9.34292,
+ "98": 9.72745,
+ "99": 8.96159,
+ "100": 9.44932
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 31217.0,
- "2": 33106.0,
- "3": 33596.0,
- "4": 31946.0,
- "5": 36783.0,
- "6": 37252.0,
- "7": 35314.0,
- "8": 31970.0,
- "9": 34937.0,
- "10": 29900.0,
- "11": 38039.0,
- "12": 34886.0,
- "13": 37108.0,
- "14": 37755.0,
- "15": 35069.0,
- "16": 36687.0,
- "17": 34887.0,
- "18": 35219.0,
- "19": 35710.0,
- "20": 32682.0,
- "21": 33456.0,
- "22": 30216.0,
- "23": 37780.0,
- "24": 32298.0,
- "25": 30789.0,
- "26": 34549.0,
- "27": 35611.0,
- "28": 36806.0,
- "29": 37955.0,
- "30": 32950.0,
- "31": 30468.0,
- "32": 36291.0,
- "33": 37916.0,
- "34": 32820.0,
- "35": 34371.0,
- "36": 34957.0,
- "37": 38282.0,
- "38": 35878.0,
- "39": 38974.0,
- "40": 36048.0,
- "41": 35988.0,
- "42": 37320.0,
- "43": 33909.0,
- "44": 33889.0,
- "45": 35577.0,
- "46": 37076.0,
- "47": 40966.0,
- "48": 35327.0,
- "49": 34682.0,
- "50": 39871.0,
- "51": 36802.0,
- "52": 36445.0,
- "53": 41968.0,
- "54": 40797.0,
- "55": 36920.0,
- "56": 40345.0,
- "57": 36961.0,
- "58": 41622.0,
- "59": 37988.0,
- "60": 40534.0,
- "61": 40456.0,
- "62": 43543.0,
- "63": 37438.0,
- "64": 42659.0,
- "65": 39924.0,
- "66": 44122.0,
- "67": 40136.0,
- "68": 40005.0,
- "69": 41675.0,
- "70": 45011.0,
- "71": 40746.0,
- "72": 41647.0,
- "73": 44080.0,
- "74": 35412.0,
- "75": 39478.0,
- "76": 46254.0,
- "77": 44764.0,
- "78": 47985.0,
- "79": 48646.0,
- "80": 46686.0,
- "81": 50102.0,
- "82": 50188.0,
- "83": 44717.0,
- "84": 46114.0,
- "85": 49347.0,
- "86": 45770.0,
- "87": 49671.0,
- "88": 46449.0,
- "89": 49666.0,
- "90": 51087.0,
- "91": 45827.0,
- "92": 48163.0,
- "93": 46547.0,
- "94": 47562.0,
- "95": 48540.0,
- "96": 50182.0,
- "97": 46055.0,
- "98": 50271.0,
- "99": 48494.0,
- "100": 45373.0
+ "1": 30880.0,
+ "2": 33134.0,
+ "3": 34050.0,
+ "4": 31735.0,
+ "5": 37029.0,
+ "6": 37301.0,
+ "7": 35264.0,
+ "8": 31446.0,
+ "9": 34979.0,
+ "10": 29776.0,
+ "11": 37948.0,
+ "12": 35277.0,
+ "13": 36889.0,
+ "14": 37873.0,
+ "15": 35129.0,
+ "16": 36685.0,
+ "17": 35245.0,
+ "18": 35040.0,
+ "19": 35795.0,
+ "20": 32424.0,
+ "21": 33245.0,
+ "22": 30312.0,
+ "23": 37765.0,
+ "24": 32307.0,
+ "25": 30615.0,
+ "26": 35162.0,
+ "27": 35279.0,
+ "28": 36753.0,
+ "29": 38210.0,
+ "30": 32956.0,
+ "31": 30176.0,
+ "32": 36478.0,
+ "33": 38169.0,
+ "34": 32816.0,
+ "35": 34299.0,
+ "36": 35094.0,
+ "37": 38301.0,
+ "38": 35465.0,
+ "39": 38725.0,
+ "40": 36179.0,
+ "41": 36169.0,
+ "42": 37201.0,
+ "43": 33594.0,
+ "44": 34260.0,
+ "45": 35612.0,
+ "46": 36862.0,
+ "47": 41077.0,
+ "48": 35724.0,
+ "49": 35419.0,
+ "50": 39856.0,
+ "51": 36420.0,
+ "52": 36293.0,
+ "53": 41745.0,
+ "54": 40909.0,
+ "55": 36730.0,
+ "56": 40013.0,
+ "57": 36880.0,
+ "58": 41605.0,
+ "59": 37882.0,
+ "60": 40354.0,
+ "61": 40374.0,
+ "62": 43810.0,
+ "63": 37847.0,
+ "64": 42754.0,
+ "65": 40712.0,
+ "66": 43936.0,
+ "67": 39899.0,
+ "68": 39751.0,
+ "69": 41842.0,
+ "70": 44867.0,
+ "71": 40732.0,
+ "72": 41487.0,
+ "73": 44646.0,
+ "74": 35559.0,
+ "75": 39104.0,
+ "76": 46425.0,
+ "77": 44861.0,
+ "78": 48216.0,
+ "79": 48320.0,
+ "80": 47159.0,
+ "81": 50341.0,
+ "82": 49881.0,
+ "83": 44680.0,
+ "84": 46232.0,
+ "85": 50001.0,
+ "86": 45505.0,
+ "87": 49100.0,
+ "88": 45978.0,
+ "89": 49672.0,
+ "90": 50663.0,
+ "91": 45690.0,
+ "92": 48221.0,
+ "93": 46168.0,
+ "94": 47472.0,
+ "95": 48600.0,
+ "96": 50290.0,
+ "97": 46454.0,
+ "98": 50024.0,
+ "99": 48124.0,
+ "100": 45275.0
}
},
"mem-allocated-bytes": {
@@ -220,104 +220,104 @@
"values": {
"1": 892874752.0,
"2": 892866560.0,
- "3": 892869120.0,
+ "3": 892869632.0,
"4": 892876800.0,
- "5": 892869120.0,
+ "5": 892870656.0,
"6": 892870656.0,
- "7": 892874240.0,
+ "7": 892874752.0,
"8": 892868608.0,
"9": 892869632.0,
- "10": 892868608.0,
- "11": 892869632.0,
- "12": 892867072.0,
- "13": 892872192.0,
- "14": 892873216.0,
+ "10": 892869632.0,
+ "11": 892868608.0,
+ "12": 892867584.0,
+ "13": 892870656.0,
+ "14": 892872192.0,
"15": 892870656.0,
- "16": 892868608.0,
+ "16": 892867584.0,
"17": 892879360.0,
- "18": 892867072.0,
- "19": 892870656.0,
+ "18": 892867584.0,
+ "19": 892870144.0,
"20": 892867072.0,
"21": 892871168.0,
- "22": 892874752.0,
- "23": 892877824.0,
+ "22": 892875264.0,
+ "23": 892877312.0,
"24": 892869120.0,
- "25": 892877312.0,
- "26": 892873216.0,
- "27": 892865024.0,
- "28": 892870144.0,
- "29": 892869632.0,
- "30": 892871680.0,
+ "25": 892876288.0,
+ "26": 892875264.0,
+ "27": 892866048.0,
+ "28": 892869632.0,
+ "29": 892869120.0,
+ "30": 892870656.0,
"31": 892881920.0,
"32": 892874752.0,
- "33": 892870144.0,
- "34": 892872192.0,
- "35": 892874240.0,
- "36": 892869632.0,
+ "33": 892869632.0,
+ "34": 892871168.0,
+ "35": 892873728.0,
+ "36": 892870656.0,
"37": 892868096.0,
- "38": 892867072.0,
- "39": 892871168.0,
- "40": 892869120.0,
- "41": 892873728.0,
- "42": 892868608.0,
- "43": 892871168.0,
- "44": 892871680.0,
- "45": 892869632.0,
- "46": 892876800.0,
- "47": 892869632.0,
+ "38": 892869120.0,
+ "39": 892870144.0,
+ "40": 892868096.0,
+ "41": 892875776.0,
+ "42": 892868096.0,
+ "43": 892870144.0,
+ "44": 892872192.0,
+ "45": 892868608.0,
+ "46": 892877312.0,
+ "47": 892869120.0,
"48": 892875264.0,
- "49": 892872704.0,
- "50": 892869120.0,
- "51": 892872192.0,
+ "49": 892871168.0,
+ "50": 892868608.0,
+ "51": 892871168.0,
"52": 892875776.0,
- "53": 892868096.0,
- "54": 892872192.0,
- "55": 892867072.0,
- "56": 892865024.0,
- "57": 892876288.0,
- "58": 892869120.0,
- "59": 892871680.0,
- "60": 892869120.0,
+ "53": 892869120.0,
+ "54": 892873216.0,
+ "55": 892868096.0,
+ "56": 892866048.0,
+ "57": 892876800.0,
+ "58": 892868608.0,
+ "59": 892872704.0,
+ "60": 892868096.0,
"61": 892869120.0,
- "62": 892869632.0,
- "63": 892870656.0,
+ "62": 892870656.0,
+ "63": 892872192.0,
"64": 892865536.0,
- "65": 892872192.0,
- "66": 892864512.0,
+ "65": 892872704.0,
+ "66": 892864000.0,
"67": 892862464.0,
- "68": 892867584.0,
+ "68": 892866560.0,
"69": 892861952.0,
- "70": 892867072.0,
+ "70": 892866048.0,
"71": 892870656.0,
- "72": 892862464.0,
+ "72": 892861952.0,
"73": 892861440.0,
- "74": 892849664.0,
- "75": 892868096.0,
+ "74": 892850176.0,
+ "75": 892870144.0,
"76": 892869632.0,
- "77": 892868096.0,
- "78": 892859392.0,
- "79": 892865024.0,
- "80": 892855296.0,
- "81": 892856320.0,
- "82": 892860416.0,
- "83": 892869632.0,
- "84": 892852736.0,
- "85": 892871680.0,
- "86": 892861952.0,
+ "77": 892867584.0,
+ "78": 892860928.0,
+ "79": 892864512.0,
+ "80": 892853760.0,
+ "81": 892854784.0,
+ "82": 892860928.0,
+ "83": 892869120.0,
+ "84": 892854784.0,
+ "85": 892872192.0,
+ "86": 892862464.0,
"87": 892869120.0,
"88": 892869632.0,
- "89": 892859392.0,
- "90": 892867072.0,
- "91": 892865536.0,
+ "89": 892860416.0,
+ "90": 892868096.0,
+ "91": 892867584.0,
"92": 892865536.0,
"93": 892861440.0,
- "94": 892860928.0,
- "95": 892869120.0,
- "96": 892866560.0,
- "97": 892856320.0,
- "98": 892869120.0,
- "99": 892864512.0,
- "100": 892864000.0
+ "94": 892858880.0,
+ "95": 892869632.0,
+ "96": 892866048.0,
+ "97": 892855808.0,
+ "98": 892869632.0,
+ "99": 892861952.0,
+ "100": 892864512.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1948027904.0,
- "2": 2183897088.0,
- "3": 2184431104.0,
- "4": 2190589952.0,
- "5": 2190589952.0,
- "6": 2190589952.0,
- "7": 2190589952.0,
- "8": 2190589952.0,
- "9": 2190589952.0,
- "10": 2190589952.0,
- "11": 2190589952.0,
- "12": 2190589952.0,
- "13": 2190589952.0,
- "14": 2190589952.0,
- "15": 2190589952.0,
- "16": 2190589952.0,
- "17": 2194543104.0,
- "18": 2194543104.0,
- "19": 2194543104.0,
- "20": 2194543104.0,
- "21": 2194543104.0,
- "22": 2194543104.0,
- "23": 2194543104.0,
- "24": 2194543104.0,
- "25": 2194543104.0,
- "26": 2194543104.0,
- "27": 2194543104.0,
- "28": 2194543104.0,
- "29": 2194543104.0,
- "30": 2194543104.0,
- "31": 2195852288.0,
- "32": 2195852288.0,
- "33": 2195852288.0,
- "34": 2195852288.0,
- "35": 2195852288.0,
- "36": 2195852288.0,
- "37": 2195852288.0,
- "38": 2195852288.0,
- "39": 2195852288.0,
- "40": 2195852288.0,
- "41": 2195852288.0,
- "42": 2195852288.0,
- "43": 2195852288.0,
- "44": 2195852288.0,
- "45": 2195852288.0,
- "46": 2195852288.0,
- "47": 2195852288.0,
- "48": 2195852288.0,
- "49": 2195852288.0,
- "50": 2195852288.0,
- "51": 2195852288.0,
- "52": 2195852288.0,
- "53": 2195852288.0,
- "54": 2195852288.0,
- "55": 2195852288.0,
- "56": 2195852288.0,
- "57": 2195852288.0,
- "58": 2195852288.0,
- "59": 2195852288.0,
- "60": 2195852288.0,
- "61": 2195852288.0,
- "62": 2195852288.0,
- "63": 2195852288.0,
- "64": 2195852288.0,
- "65": 2195852288.0,
- "66": 2195852288.0,
- "67": 2195852288.0,
- "68": 2195852288.0,
- "69": 2195852288.0,
- "70": 2195852288.0,
- "71": 2195852288.0,
- "72": 2195852288.0,
- "73": 2195852288.0,
- "74": 2195852288.0,
- "75": 2195852288.0,
- "76": 2195852288.0,
- "77": 2195852288.0,
- "78": 2195852288.0,
- "79": 2195852288.0,
- "80": 2195852288.0,
- "81": 2195852288.0,
- "82": 2195852288.0,
- "83": 2195852288.0,
- "84": 2195852288.0,
- "85": 2195852288.0,
- "86": 2195852288.0,
- "87": 2195852288.0,
- "88": 2195852288.0,
- "89": 2195852288.0,
- "90": 2195852288.0,
- "91": 2195852288.0,
- "92": 2195852288.0,
- "93": 2195852288.0,
- "94": 2195852288.0,
- "95": 2195852288.0,
- "96": 2195852288.0,
- "97": 2195852288.0,
- "98": 2195852288.0,
- "99": 2195852288.0,
- "100": 2195852288.0
+ "1": 1946773504.0,
+ "2": 2182914048.0,
+ "3": 2184069120.0,
+ "4": 2188971008.0,
+ "5": 2188971008.0,
+ "6": 2188971008.0,
+ "7": 2188971008.0,
+ "8": 2188971008.0,
+ "9": 2188971008.0,
+ "10": 2188971008.0,
+ "11": 2188971008.0,
+ "12": 2188971008.0,
+ "13": 2188971008.0,
+ "14": 2188971008.0,
+ "15": 2188971008.0,
+ "16": 2188971008.0,
+ "17": 2191747584.0,
+ "18": 2191747584.0,
+ "19": 2191747584.0,
+ "20": 2191747584.0,
+ "21": 2191747584.0,
+ "22": 2191747584.0,
+ "23": 2191747584.0,
+ "24": 2191747584.0,
+ "25": 2191747584.0,
+ "26": 2191747584.0,
+ "27": 2191747584.0,
+ "28": 2191747584.0,
+ "29": 2191747584.0,
+ "30": 2191747584.0,
+ "31": 2193825792.0,
+ "32": 2193825792.0,
+ "33": 2193825792.0,
+ "34": 2193825792.0,
+ "35": 2193825792.0,
+ "36": 2193825792.0,
+ "37": 2193825792.0,
+ "38": 2193825792.0,
+ "39": 2193825792.0,
+ "40": 2193825792.0,
+ "41": 2193825792.0,
+ "42": 2193825792.0,
+ "43": 2193825792.0,
+ "44": 2193825792.0,
+ "45": 2193825792.0,
+ "46": 2193825792.0,
+ "47": 2193825792.0,
+ "48": 2193825792.0,
+ "49": 2193825792.0,
+ "50": 2193825792.0,
+ "51": 2193825792.0,
+ "52": 2193825792.0,
+ "53": 2193825792.0,
+ "54": 2193825792.0,
+ "55": 2193825792.0,
+ "56": 2193825792.0,
+ "57": 2194713600.0,
+ "58": 2194713600.0,
+ "59": 2194713600.0,
+ "60": 2194713600.0,
+ "61": 2194713600.0,
+ "62": 2194713600.0,
+ "63": 2194713600.0,
+ "64": 2194713600.0,
+ "65": 2194713600.0,
+ "66": 2194713600.0,
+ "67": 2194713600.0,
+ "68": 2194713600.0,
+ "69": 2194713600.0,
+ "70": 2194713600.0,
+ "71": 2194713600.0,
+ "72": 2194713600.0,
+ "73": 2194713600.0,
+ "74": 2194713600.0,
+ "75": 2194713600.0,
+ "76": 2194713600.0,
+ "77": 2194713600.0,
+ "78": 2194713600.0,
+ "79": 2194713600.0,
+ "80": 2194713600.0,
+ "81": 2194713600.0,
+ "82": 2194713600.0,
+ "83": 2194713600.0,
+ "84": 2194713600.0,
+ "85": 2194713600.0,
+ "86": 2194713600.0,
+ "87": 2194713600.0,
+ "88": 2194713600.0,
+ "89": 2194713600.0,
+ "90": 2194713600.0,
+ "91": 2194713600.0,
+ "92": 2194713600.0,
+ "93": 2194713600.0,
+ "94": 2194713600.0,
+ "95": 2194713600.0,
+ "96": 2194713600.0,
+ "97": 2194713600.0,
+ "98": 2194713600.0,
+ "99": 2194713600.0,
+ "100": 2194713600.0
}
},
"iteration-time": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 3.86481,
- "3": 0.3588,
- "4": 0.35276,
- "5": 0.33575,
- "6": 0.3344,
- "7": 0.3406,
- "8": 0.33551,
- "9": 0.33157,
- "10": 0.32814,
- "11": 0.32882,
- "12": 0.3298,
- "13": 0.32887,
- "14": 0.32898,
- "15": 0.33409,
- "16": 0.32679,
- "17": 0.34317,
- "18": 0.33153,
- "19": 0.32828,
- "20": 0.33077,
- "21": 0.32713,
- "22": 0.32603,
- "23": 0.32819,
- "24": 0.33158,
- "25": 0.32832,
- "26": 0.32593,
- "27": 0.33086,
- "28": 0.32481,
- "29": 0.32607,
- "30": 0.33032,
- "31": 0.33561,
- "32": 0.33149,
- "33": 0.32643,
- "34": 0.34262,
- "35": 0.32889,
- "36": 0.32749,
- "37": 0.32097,
- "38": 0.33036,
- "39": 0.69454,
- "40": 0.33723,
- "41": 0.3284,
- "42": 0.32735,
- "43": 0.33334,
- "44": 0.3333,
- "45": 0.33315,
- "46": 0.33505,
- "47": 0.32976,
- "48": 0.32918,
- "49": 0.34661,
- "50": 0.32681,
- "51": 0.3427,
- "52": 0.3299,
- "53": 0.32454,
- "54": 0.3251,
- "55": 0.32968,
- "56": 0.34696,
- "57": 0.33819,
- "58": 0.32649,
- "59": 0.3341,
- "60": 0.33324,
- "61": 0.33925,
- "62": 0.33532,
- "63": 0.34334,
- "64": 0.34963,
- "65": 0.38392,
- "66": 0.33805,
- "67": 0.3728,
- "68": 0.33745,
- "69": 0.33504,
- "70": 0.33581,
- "71": 0.35385,
- "72": 0.34934,
- "73": 0.34952,
- "74": 0.35756,
- "75": 0.35105,
- "76": 0.34933,
- "77": 0.33518,
- "78": 0.34556,
- "79": 0.34603,
- "80": 0.36355,
- "81": 0.34186,
- "82": 0.34271,
- "83": 0.39765,
- "84": 0.36927,
- "85": 0.33938,
- "86": 0.35142,
- "87": 0.34329,
- "88": 0.33135,
- "89": 0.34535,
- "90": 0.33856,
- "91": 0.3522,
- "92": 0.33934,
- "93": 0.38169,
- "94": 0.36358,
- "95": 0.33846,
- "96": 0.33554,
- "97": 0.34438,
- "98": 0.32586,
- "99": 0.43185,
- "100": 0.33974
+ "2": 6.67539,
+ "3": 0.407,
+ "4": 0.39073,
+ "5": 0.37543,
+ "6": 0.37537,
+ "7": 0.38454,
+ "8": 0.38441,
+ "9": 0.36994,
+ "10": 0.3708,
+ "11": 0.36361,
+ "12": 0.36417,
+ "13": 0.37453,
+ "14": 0.38004,
+ "15": 0.37984,
+ "16": 0.36878,
+ "17": 0.37576,
+ "18": 0.37765,
+ "19": 0.36829,
+ "20": 0.37498,
+ "21": 0.42033,
+ "22": 0.41011,
+ "23": 0.47059,
+ "24": 0.46929,
+ "25": 0.4628,
+ "26": 0.40125,
+ "27": 0.37004,
+ "28": 0.36512,
+ "29": 0.36666,
+ "30": 0.37229,
+ "31": 0.37489,
+ "32": 0.37701,
+ "33": 0.37213,
+ "34": 0.38959,
+ "35": 0.37309,
+ "36": 0.38121,
+ "37": 0.36506,
+ "38": 0.36892,
+ "39": 0.37227,
+ "40": 0.37401,
+ "41": 0.37309,
+ "42": 0.3734,
+ "43": 0.37679,
+ "44": 0.37378,
+ "45": 0.37577,
+ "46": 0.37224,
+ "47": 0.3656,
+ "48": 0.37878,
+ "49": 0.38841,
+ "50": 0.37117,
+ "51": 0.41362,
+ "52": 0.36969,
+ "53": 0.36731,
+ "54": 0.3675,
+ "55": 0.37619,
+ "56": 0.39642,
+ "57": 0.38389,
+ "58": 0.37131,
+ "59": 0.37528,
+ "60": 0.38109,
+ "61": 0.37573,
+ "62": 0.37706,
+ "63": 0.38387,
+ "64": 0.38217,
+ "65": 0.41628,
+ "66": 0.37827,
+ "67": 0.40668,
+ "68": 0.37316,
+ "69": 0.37861,
+ "70": 0.38107,
+ "71": 0.38377,
+ "72": 0.37354,
+ "73": 0.3807,
+ "74": 0.38445,
+ "75": 0.38628,
+ "76": 0.42439,
+ "77": 0.39532,
+ "78": 0.42914,
+ "79": 0.39185,
+ "80": 0.39213,
+ "81": 0.37941,
+ "82": 0.37259,
+ "83": 0.37258,
+ "84": 0.38481,
+ "85": 0.38022,
+ "86": 0.39917,
+ "87": 0.38057,
+ "88": 0.38203,
+ "89": 0.38656,
+ "90": 0.37227,
+ "91": 0.38274,
+ "92": 0.37855,
+ "93": 0.39076,
+ "94": 0.40202,
+ "95": 0.37695,
+ "96": 0.37707,
+ "97": 0.38216,
+ "98": 0.3776,
+ "99": 0.38682,
+ "100": 0.37803
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer/golden_values_dev_dgx_h100.json
index 64a0d3b0293..0d1db950951 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.78091,
- "2": 10.80272,
- "3": 10.8036,
- "4": 10.77566,
- "5": 10.83259,
- "6": 10.83704,
- "7": 10.79793,
- "8": 10.79364,
- "9": 10.808,
- "10": 10.76116,
- "11": 10.85297,
- "12": 10.84152,
- "13": 10.8247,
- "14": 10.85822,
- "15": 10.78238,
- "16": 10.77927,
- "17": 10.74878,
- "18": 10.7897,
- "19": 10.7749,
- "20": 10.71704,
- "21": 10.70811,
- "22": 10.54787,
- "23": 10.72978,
- "24": 10.60324,
- "25": 10.55979,
- "26": 10.61611,
- "27": 10.6446,
- "28": 10.62463,
- "29": 10.63492,
- "30": 10.42362,
- "31": 10.16499,
- "32": 10.51313,
- "33": 10.5094,
- "34": 10.2668,
- "35": 10.32318,
- "36": 10.28865,
- "37": 10.41114,
- "38": 10.26426,
- "39": 10.45,
- "40": 10.17473,
- "41": 10.20958,
- "42": 10.27824,
- "43": 9.91831,
- "44": 10.03131,
- "45": 9.91995,
- "46": 9.8862,
- "47": 10.19255,
- "48": 9.92803,
- "49": 9.61616,
- "50": 9.98532,
- "51": 9.90528,
- "52": 9.80364,
- "53": 10.12728,
- "54": 10.00036,
- "55": 9.9362,
- "56": 9.68506,
- "57": 9.55805,
- "58": 9.90514,
- "59": 9.63857,
- "60": 9.57451,
- "61": 9.76864,
- "62": 10.03802,
- "63": 9.44503,
- "64": 9.82796,
- "65": 9.00712,
- "66": 9.77422,
- "67": 9.41277,
- "68": 9.84111,
- "69": 9.82784,
- "70": 9.79011,
- "71": 9.66957,
- "72": 9.62799,
- "73": 9.5473,
- "74": 9.03663,
- "75": 9.49153,
- "76": 9.16783,
- "77": 10.10857,
- "78": 9.77081,
- "79": 9.4383,
- "80": 9.45436,
- "81": 9.52266,
- "82": 9.7424,
- "83": 9.37076,
- "84": 9.45377,
- "85": 9.65832,
- "86": 9.12522,
- "87": 9.62697,
- "88": 9.79619,
- "89": 9.66054,
- "90": 9.85081,
- "91": 9.39408,
- "92": 9.40744,
- "93": 9.13595,
- "94": 8.89048,
- "95": 9.563,
- "96": 9.5714,
- "97": 9.34318,
- "98": 9.73026,
- "99": 8.95002,
- "100": 9.4424
+ "1": 10.92624,
+ "2": 10.91761,
+ "3": 10.92333,
+ "4": 10.92688,
+ "5": 10.92998,
+ "6": 10.92516,
+ "7": 10.9275,
+ "8": 10.91969,
+ "9": 10.92241,
+ "10": 10.92325,
+ "11": 10.9076,
+ "12": 10.92327,
+ "13": 10.90646,
+ "14": 10.88873,
+ "15": 10.89667,
+ "16": 10.87425,
+ "17": 10.8811,
+ "18": 10.87935,
+ "19": 10.87685,
+ "20": 10.82302,
+ "21": 10.82145,
+ "22": 10.79818,
+ "23": 10.80844,
+ "24": 10.7706,
+ "25": 10.77826,
+ "26": 10.75612,
+ "27": 10.74801,
+ "28": 10.68583,
+ "29": 10.65833,
+ "30": 10.62792,
+ "31": 10.61133,
+ "32": 10.60564,
+ "33": 10.57859,
+ "34": 10.53528,
+ "35": 10.53778,
+ "36": 10.5287,
+ "37": 10.497,
+ "38": 10.49486,
+ "39": 10.46012,
+ "40": 10.4421,
+ "41": 10.42198,
+ "42": 10.41557,
+ "43": 10.39881,
+ "44": 10.36689,
+ "45": 10.37274,
+ "46": 10.33604,
+ "47": 10.31811,
+ "48": 10.28666,
+ "49": 10.27521,
+ "50": 10.27537,
+ "51": 10.26994,
+ "52": 10.22194,
+ "53": 10.22936,
+ "54": 10.19416,
+ "55": 10.17384,
+ "56": 10.18863,
+ "57": 10.17852,
+ "58": 10.18568,
+ "59": 10.13496,
+ "60": 10.15559,
+ "61": 10.10619,
+ "62": 10.07424,
+ "63": 10.13238,
+ "64": 10.09213,
+ "65": 10.06807,
+ "66": 10.09499,
+ "67": 10.07254,
+ "68": 10.03573,
+ "69": 10.05515,
+ "70": 10.03301,
+ "71": 10.0495,
+ "72": 10.04093,
+ "73": 10.03871,
+ "74": 10.02118,
+ "75": 9.98197,
+ "76": 10.00326,
+ "77": 10.01464,
+ "78": 9.96846,
+ "79": 9.96967,
+ "80": 9.98526,
+ "81": 10.01102,
+ "82": 9.94221,
+ "83": 9.91686,
+ "84": 9.85331,
+ "85": 9.83738,
+ "86": 9.94372,
+ "87": 9.95605,
+ "88": 9.93513,
+ "89": 9.87152,
+ "90": 9.87909,
+ "91": 9.89522,
+ "92": 9.86927,
+ "93": 9.8019,
+ "94": 9.88867,
+ "95": 9.86027,
+ "96": 9.84921,
+ "97": 9.79907,
+ "98": 9.83252,
+ "99": 9.8642,
+ "100": 9.7602
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 30994.0,
- "2": 32962.0,
- "3": 33026.0,
- "4": 30732.0,
- "5": 36042.0,
- "6": 37038.0,
- "7": 34481.0,
- "8": 31368.0,
- "9": 33980.0,
- "10": 29532.0,
- "11": 37852.0,
- "12": 34972.0,
- "13": 36994.0,
- "14": 37789.0,
- "15": 34058.0,
- "16": 36656.0,
- "17": 34700.0,
- "18": 34946.0,
- "19": 35228.0,
- "20": 32392.0,
- "21": 33247.0,
- "22": 30040.0,
- "23": 37891.0,
- "24": 32099.0,
- "25": 30921.0,
- "26": 34212.0,
- "27": 34975.0,
- "28": 36746.0,
- "29": 37759.0,
- "30": 32786.0,
- "31": 30423.0,
- "32": 35992.0,
- "33": 36915.0,
- "34": 32293.0,
- "35": 33654.0,
- "36": 34755.0,
- "37": 37859.0,
- "38": 36022.0,
- "39": 38343.0,
- "40": 35963.0,
- "41": 35882.0,
- "42": 36774.0,
- "43": 34186.0,
- "44": 33572.0,
- "45": 35574.0,
- "46": 37208.0,
- "47": 40154.0,
- "48": 36385.0,
- "49": 36259.0,
- "50": 38861.0,
- "51": 38061.0,
- "52": 37025.0,
- "53": 41802.0,
- "54": 41253.0,
- "55": 37654.0,
- "56": 41164.0,
- "57": 37682.0,
- "58": 41782.0,
- "59": 39444.0,
- "60": 40691.0,
- "61": 40876.0,
- "62": 43113.0,
- "63": 38389.0,
- "64": 43217.0,
- "65": 41689.0,
- "66": 45525.0,
- "67": 41717.0,
- "68": 40369.0,
- "69": 41287.0,
- "70": 45545.0,
- "71": 41651.0,
- "72": 41881.0,
- "73": 45139.0,
- "74": 35747.0,
- "75": 39155.0,
- "76": 44874.0,
- "77": 45442.0,
- "78": 46782.0,
- "79": 48776.0,
- "80": 47161.0,
- "81": 51277.0,
- "82": 49953.0,
- "83": 45334.0,
- "84": 46096.0,
- "85": 49238.0,
- "86": 46118.0,
- "87": 49880.0,
- "88": 47115.0,
- "89": 48583.0,
- "90": 49057.0,
- "91": 45950.0,
- "92": 47820.0,
- "93": 46437.0,
- "94": 47530.0,
- "95": 48000.0,
- "96": 50285.0,
- "97": 46225.0,
- "98": 49809.0,
- "99": 47890.0,
- "100": 44636.0
+ "1": 36362.0,
+ "2": 36609.0,
+ "3": 36699.0,
+ "4": 36555.0,
+ "5": 36453.0,
+ "6": 36089.0,
+ "7": 37058.0,
+ "8": 36230.0,
+ "9": 37158.0,
+ "10": 37488.0,
+ "11": 36217.0,
+ "12": 35290.0,
+ "13": 36912.0,
+ "14": 36742.0,
+ "15": 36189.0,
+ "16": 35910.0,
+ "17": 36022.0,
+ "18": 36611.0,
+ "19": 36847.0,
+ "20": 36461.0,
+ "21": 36265.0,
+ "22": 36869.0,
+ "23": 36415.0,
+ "24": 36472.0,
+ "25": 36080.0,
+ "26": 36569.0,
+ "27": 36365.0,
+ "28": 35354.0,
+ "29": 35730.0,
+ "30": 35276.0,
+ "31": 36070.0,
+ "32": 36388.0,
+ "33": 35414.0,
+ "34": 35466.0,
+ "35": 36090.0,
+ "36": 35199.0,
+ "37": 36091.0,
+ "38": 35899.0,
+ "39": 36310.0,
+ "40": 37711.0,
+ "41": 35849.0,
+ "42": 35453.0,
+ "43": 37033.0,
+ "44": 35693.0,
+ "45": 39252.0,
+ "46": 38243.0,
+ "47": 37802.0,
+ "48": 38167.0,
+ "49": 41834.0,
+ "50": 38589.0,
+ "51": 38480.0,
+ "52": 42085.0,
+ "53": 40546.0,
+ "54": 41309.0,
+ "55": 38912.0,
+ "56": 41066.0,
+ "57": 37232.0,
+ "58": 45800.0,
+ "59": 41680.0,
+ "60": 42510.0,
+ "61": 41454.0,
+ "62": 44592.0,
+ "63": 43438.0,
+ "64": 49908.0,
+ "65": 41819.0,
+ "66": 45362.0,
+ "67": 50227.0,
+ "68": 47070.0,
+ "69": 44957.0,
+ "70": 47249.0,
+ "71": 45554.0,
+ "72": 44893.0,
+ "73": 48127.0,
+ "74": 46038.0,
+ "75": 47097.0,
+ "76": 46732.0,
+ "77": 47586.0,
+ "78": 48952.0,
+ "79": 47179.0,
+ "80": 45314.0,
+ "81": 53362.0,
+ "82": 42269.0,
+ "83": 46322.0,
+ "84": 45606.0,
+ "85": 45258.0,
+ "86": 45280.0,
+ "87": 36969.0,
+ "88": 45865.0,
+ "89": 48808.0,
+ "90": 56355.0,
+ "91": 37880.0,
+ "92": 49390.0,
+ "93": 46985.0,
+ "94": 42703.0,
+ "95": 46050.0,
+ "96": 50108.0,
+ "97": 47616.0,
+ "98": 49989.0,
+ "99": 44641.0,
+ "100": 45440.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 892865536.0,
- "2": 892869632.0,
- "3": 892869632.0,
- "4": 892865536.0,
- "5": 892866048.0,
- "6": 892867584.0,
- "7": 892867072.0,
- "8": 892868608.0,
- "9": 892866560.0,
- "10": 892868608.0,
- "11": 892867072.0,
+ "1": 892871168.0,
+ "2": 892866048.0,
+ "3": 892867584.0,
+ "4": 892867584.0,
+ "5": 892868096.0,
+ "6": 892871168.0,
+ "7": 892869632.0,
+ "8": 892867072.0,
+ "9": 892868096.0,
+ "10": 892866560.0,
+ "11": 892868096.0,
"12": 892866560.0,
- "13": 892866560.0,
- "14": 892869120.0,
- "15": 892868608.0,
- "16": 892868096.0,
- "17": 892868608.0,
- "18": 892870656.0,
- "19": 892869120.0,
- "20": 892867584.0,
+ "13": 892869632.0,
+ "14": 892868608.0,
+ "15": 892866560.0,
+ "16": 892865024.0,
+ "17": 892866560.0,
+ "18": 892866560.0,
+ "19": 892871168.0,
+ "20": 892866048.0,
"21": 892867584.0,
- "22": 892864512.0,
- "23": 892865536.0,
- "24": 892868096.0,
- "25": 892864512.0,
- "26": 892867584.0,
- "27": 892868096.0,
- "28": 892866560.0,
- "29": 892867072.0,
- "30": 892864512.0,
- "31": 892863488.0,
- "32": 892862976.0,
- "33": 892867072.0,
- "34": 892866560.0,
- "35": 892866048.0,
- "36": 892869632.0,
- "37": 892868096.0,
- "38": 892867584.0,
- "39": 892867072.0,
- "40": 892868096.0,
- "41": 892866560.0,
- "42": 892868608.0,
- "43": 892862464.0,
- "44": 892864000.0,
- "45": 892866048.0,
- "46": 892865536.0,
- "47": 892866048.0,
- "48": 892862464.0,
- "49": 892864512.0,
- "50": 892868096.0,
- "51": 892861440.0,
- "52": 892859904.0,
- "53": 892862464.0,
- "54": 892862464.0,
- "55": 892863488.0,
- "56": 892866048.0,
- "57": 892858368.0,
- "58": 892860416.0,
- "59": 892859904.0,
- "60": 892860928.0,
- "61": 892869632.0,
- "62": 892866560.0,
- "63": 892862976.0,
- "64": 892864512.0,
- "65": 892852736.0,
- "66": 892867072.0,
- "67": 892862464.0,
- "68": 892869632.0,
+ "22": 892868096.0,
+ "23": 892869120.0,
+ "24": 892867072.0,
+ "25": 892868096.0,
+ "26": 892865024.0,
+ "27": 892866560.0,
+ "28": 892868608.0,
+ "29": 892871168.0,
+ "30": 892867072.0,
+ "31": 892868608.0,
+ "32": 892865024.0,
+ "33": 892868096.0,
+ "34": 892866048.0,
+ "35": 892865536.0,
+ "36": 892868096.0,
+ "37": 892867072.0,
+ "38": 892866560.0,
+ "39": 892866560.0,
+ "40": 892869632.0,
+ "41": 892869120.0,
+ "42": 892868096.0,
+ "43": 892866048.0,
+ "44": 892867584.0,
+ "45": 892868096.0,
+ "46": 892868096.0,
+ "47": 892869120.0,
+ "48": 892866048.0,
+ "49": 892865024.0,
+ "50": 892870656.0,
+ "51": 892866048.0,
+ "52": 892867584.0,
+ "53": 892867072.0,
+ "54": 892867072.0,
+ "55": 892867584.0,
+ "56": 892867584.0,
+ "57": 892868096.0,
+ "58": 892869632.0,
+ "59": 892866048.0,
+ "60": 892865536.0,
+ "61": 892866048.0,
+ "62": 892866048.0,
+ "63": 892865536.0,
+ "64": 892868096.0,
+ "65": 892866048.0,
+ "66": 892869120.0,
+ "67": 892864512.0,
+ "68": 892868096.0,
"69": 892865536.0,
- "70": 892867584.0,
- "71": 892869632.0,
- "72": 892861440.0,
- "73": 892869120.0,
- "74": 892859392.0,
- "75": 892868096.0,
- "76": 892867584.0,
- "77": 892868096.0,
- "78": 892864512.0,
- "79": 892865536.0,
- "80": 892865536.0,
- "81": 892867072.0,
- "82": 892865024.0,
- "83": 892861952.0,
+ "70": 892865536.0,
+ "71": 892867584.0,
+ "72": 892869632.0,
+ "73": 892866048.0,
+ "74": 892865536.0,
+ "75": 892868608.0,
+ "76": 892865024.0,
+ "77": 892862464.0,
+ "78": 892865024.0,
+ "79": 892866560.0,
+ "80": 892865024.0,
+ "81": 892861952.0,
+ "82": 892862976.0,
+ "83": 892867584.0,
"84": 892862464.0,
- "85": 892862976.0,
- "86": 892862464.0,
- "87": 892871168.0,
- "88": 892863488.0,
- "89": 892865536.0,
- "90": 892867072.0,
- "91": 892868096.0,
- "92": 892866560.0,
- "93": 892869632.0,
- "94": 892865536.0,
- "95": 892866048.0,
- "96": 892866048.0,
- "97": 892864000.0,
- "98": 892868608.0,
- "99": 892860928.0,
- "100": 892862976.0
+ "85": 892860416.0,
+ "86": 892853248.0,
+ "87": 892867072.0,
+ "88": 892867584.0,
+ "89": 892859392.0,
+ "90": 892858880.0,
+ "91": 892866048.0,
+ "92": 892854784.0,
+ "93": 892861440.0,
+ "94": 892854784.0,
+ "95": 892853248.0,
+ "96": 892868096.0,
+ "97": 892854272.0,
+ "98": 892846592.0,
+ "99": 892853760.0,
+ "100": 892852224.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1918568448.0,
- "2": 2157712384.0,
- "3": 2157712384.0,
- "4": 2157712384.0,
- "5": 2159109632.0,
- "6": 2159109632.0,
- "7": 2159109632.0,
- "8": 2159109632.0,
- "9": 2159109632.0,
- "10": 2159142912.0,
- "11": 2159142912.0,
- "12": 2159142912.0,
- "13": 2159142912.0,
- "14": 2159633920.0,
- "15": 2159633920.0,
- "16": 2159633920.0,
- "17": 2159633920.0,
- "18": 2159633920.0,
- "19": 2159633920.0,
- "20": 2159633920.0,
- "21": 2159633920.0,
- "22": 2159633920.0,
- "23": 2159633920.0,
- "24": 2159633920.0,
- "25": 2159633920.0,
- "26": 2159802368.0,
- "27": 2159802368.0,
- "28": 2159802368.0,
- "29": 2159802368.0,
- "30": 2159802368.0,
- "31": 2159802368.0,
- "32": 2159802368.0,
- "33": 2159802368.0,
- "34": 2159802368.0,
- "35": 2159802368.0,
- "36": 2159802368.0,
- "37": 2159802368.0,
- "38": 2159802368.0,
- "39": 2159802368.0,
- "40": 2159802368.0,
- "41": 2159802368.0,
- "42": 2159802368.0,
- "43": 2159802368.0,
- "44": 2159802368.0,
- "45": 2159802368.0,
- "46": 2159802368.0,
- "47": 2159802368.0,
- "48": 2159802368.0,
- "49": 2159802368.0,
- "50": 2159802368.0,
- "51": 2159802368.0,
- "52": 2159802368.0,
- "53": 2159802368.0,
- "54": 2159802368.0,
- "55": 2159802368.0,
- "56": 2159802368.0,
- "57": 2159802368.0,
- "58": 2159802368.0,
- "59": 2159802368.0,
- "60": 2159802368.0,
- "61": 2159802368.0,
- "62": 2159802368.0,
- "63": 2159802368.0,
- "64": 2159802368.0,
- "65": 2159802368.0,
- "66": 2159802368.0,
- "67": 2159802368.0,
- "68": 2159802368.0,
- "69": 2159802368.0,
- "70": 2159802368.0,
- "71": 2159802368.0,
- "72": 2159802368.0,
- "73": 2160337408.0,
- "74": 2160337408.0,
- "75": 2160337408.0,
- "76": 2160337408.0,
- "77": 2160337408.0,
- "78": 2160337408.0,
- "79": 2160337408.0,
- "80": 2160337408.0,
- "81": 2160337408.0,
- "82": 2160337408.0,
- "83": 2160337408.0,
- "84": 2161362944.0,
- "85": 2161362944.0,
- "86": 2161362944.0,
- "87": 2161362944.0,
- "88": 2161362944.0,
- "89": 2161362944.0,
- "90": 2161362944.0,
- "91": 2161362944.0,
- "92": 2161362944.0,
- "93": 2161362944.0,
- "94": 2161362944.0,
- "95": 2162391552.0,
- "96": 2162391552.0,
- "97": 2162391552.0,
- "98": 2162391552.0,
- "99": 2162391552.0,
- "100": 2162391552.0
+ "1": 1921131008.0,
+ "2": 2156760064.0,
+ "3": 2156760064.0,
+ "4": 2156760064.0,
+ "5": 2156760064.0,
+ "6": 2159165952.0,
+ "7": 2159165952.0,
+ "8": 2159165952.0,
+ "9": 2159165952.0,
+ "10": 2159165952.0,
+ "11": 2159165952.0,
+ "12": 2159165952.0,
+ "13": 2159165952.0,
+ "14": 2159165952.0,
+ "15": 2159165952.0,
+ "16": 2159165952.0,
+ "17": 2159165952.0,
+ "18": 2159165952.0,
+ "19": 2159165952.0,
+ "20": 2159165952.0,
+ "21": 2159165952.0,
+ "22": 2159165952.0,
+ "23": 2159165952.0,
+ "24": 2159165952.0,
+ "25": 2159165952.0,
+ "26": 2159165952.0,
+ "27": 2159165952.0,
+ "28": 2159398912.0,
+ "29": 2159398912.0,
+ "30": 2159398912.0,
+ "31": 2159398912.0,
+ "32": 2159398912.0,
+ "33": 2159398912.0,
+ "34": 2159398912.0,
+ "35": 2159398912.0,
+ "36": 2159415808.0,
+ "37": 2159415808.0,
+ "38": 2159415808.0,
+ "39": 2159415808.0,
+ "40": 2159415808.0,
+ "41": 2159415808.0,
+ "42": 2159415808.0,
+ "43": 2159415808.0,
+ "44": 2159415808.0,
+ "45": 2159415808.0,
+ "46": 2159415808.0,
+ "47": 2159415808.0,
+ "48": 2159415808.0,
+ "49": 2159415808.0,
+ "50": 2159415808.0,
+ "51": 2159415808.0,
+ "52": 2159415808.0,
+ "53": 2159415808.0,
+ "54": 2159415808.0,
+ "55": 2159415808.0,
+ "56": 2159415808.0,
+ "57": 2159415808.0,
+ "58": 2159415808.0,
+ "59": 2159415808.0,
+ "60": 2159415808.0,
+ "61": 2159415808.0,
+ "62": 2159415808.0,
+ "63": 2159415808.0,
+ "64": 2159415808.0,
+ "65": 2159415808.0,
+ "66": 2159415808.0,
+ "67": 2159415808.0,
+ "68": 2159415808.0,
+ "69": 2159415808.0,
+ "70": 2159415808.0,
+ "71": 2159415808.0,
+ "72": 2159415808.0,
+ "73": 2159415808.0,
+ "74": 2159415808.0,
+ "75": 2159415808.0,
+ "76": 2159415808.0,
+ "77": 2159415808.0,
+ "78": 2159415808.0,
+ "79": 2159415808.0,
+ "80": 2159415808.0,
+ "81": 2159415808.0,
+ "82": 2159415808.0,
+ "83": 2159415808.0,
+ "84": 2159415808.0,
+ "85": 2159415808.0,
+ "86": 2159415808.0,
+ "87": 2159415808.0,
+ "88": 2159415808.0,
+ "89": 2159415808.0,
+ "90": 2159415808.0,
+ "91": 2159415808.0,
+ "92": 2159415808.0,
+ "93": 2159415808.0,
+ "94": 2159415808.0,
+ "95": 2159415808.0,
+ "96": 2159415808.0,
+ "97": 2159415808.0,
+ "98": 2159415808.0,
+ "99": 2159415808.0,
+ "100": 2159415808.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 14.93722,
- "2": 0.29196,
- "3": 0.25566,
- "4": 0.22819,
- "5": 0.21657,
- "6": 0.22742,
- "7": 0.23255,
- "8": 0.21868,
- "9": 0.23203,
- "10": 0.22911,
- "11": 0.22371,
- "12": 0.22358,
- "13": 0.21762,
- "14": 0.2166,
- "15": 0.2341,
- "16": 0.21834,
- "17": 0.21429,
- "18": 0.21499,
- "19": 0.2158,
- "20": 0.21523,
- "21": 0.21654,
- "22": 0.21788,
- "23": 0.21597,
- "24": 0.20917,
- "25": 0.2076,
- "26": 0.20309,
- "27": 0.20463,
- "28": 0.57074,
- "29": 0.20266,
- "30": 0.21832,
- "31": 0.23121,
- "32": 0.2052,
- "33": 0.20847,
- "34": 0.22756,
- "35": 0.21093,
- "36": 0.20495,
- "37": 0.20762,
- "38": 0.20131,
- "39": 0.1991,
- "40": 0.20426,
- "41": 0.20518,
+ "1": "nan",
+ "2": 8.15624,
+ "3": 0.2661,
+ "4": 0.22945,
+ "5": 0.21557,
+ "6": 0.2408,
+ "7": 0.21635,
+ "8": 0.20491,
+ "9": 0.2055,
+ "10": 0.20512,
+ "11": 0.20284,
+ "12": 0.20371,
+ "13": 0.20418,
+ "14": 0.20829,
+ "15": 0.20776,
+ "16": 0.19883,
+ "17": 0.19911,
+ "18": 0.20069,
+ "19": 0.19976,
+ "20": 0.20149,
+ "21": 0.20278,
+ "22": 0.20127,
+ "23": 0.20069,
+ "24": 0.20033,
+ "25": 0.19965,
+ "26": 0.20296,
+ "27": 0.20508,
+ "28": 0.2014,
+ "29": 0.2044,
+ "30": 0.19907,
+ "31": 0.20565,
+ "32": 0.20101,
+ "33": 0.1996,
+ "34": 0.201,
+ "35": 0.20133,
+ "36": 0.19955,
+ "37": 0.20007,
+ "38": 0.19919,
+ "39": 0.20203,
+ "40": 0.19849,
+ "41": 0.19919,
"42": 0.20555,
- "43": 0.21112,
- "44": 0.20079,
- "45": 0.21854,
- "46": 0.22885,
- "47": 0.20366,
- "48": 0.21784,
- "49": 0.23722,
- "50": 0.20288,
- "51": 0.23225,
- "52": 0.23281,
- "53": 0.20606,
- "54": 0.21135,
- "55": 0.21897,
- "56": 0.25991,
- "57": 0.22845,
- "58": 0.21751,
- "59": 0.21469,
- "60": 0.21187,
- "61": 0.20946,
- "62": 0.21358,
- "63": 0.21765,
- "64": 0.20357,
- "65": 0.27698,
- "66": 0.2118,
- "67": 0.25518,
- "68": 0.22631,
- "69": 0.21209,
- "70": 0.2039,
- "71": 0.22504,
- "72": 0.22276,
- "73": 0.25179,
- "74": 0.22993,
- "75": 0.21538,
- "76": 0.23629,
- "77": 0.20835,
- "78": 0.21168,
- "79": 0.21631,
- "80": 0.21797,
- "81": 0.20362,
- "82": 0.20269,
- "83": 0.21014,
- "84": 0.21456,
- "85": 0.20971,
- "86": 0.22253,
- "87": 0.20037,
- "88": 0.20403,
- "89": 0.21541,
- "90": 0.21443,
- "91": 0.23258,
- "92": 0.21749,
- "93": 0.22377,
- "94": 0.23559,
- "95": 0.21351,
- "96": 0.20316,
- "97": 0.21349,
- "98": 0.20244,
- "99": 0.21023,
- "100": 0.20508
+ "43": 0.20165,
+ "44": 0.20028,
+ "45": 0.20065,
+ "46": 0.19793,
+ "47": 0.20111,
+ "48": 0.20528,
+ "49": 0.19992,
+ "50": 0.19903,
+ "51": 0.23211,
+ "52": 0.21679,
+ "53": 0.21392,
+ "54": 0.22103,
+ "55": 0.20901,
+ "56": 0.20135,
+ "57": 0.20055,
+ "58": 0.21136,
+ "59": 0.20376,
+ "60": 0.20394,
+ "61": 0.20358,
+ "62": 0.20489,
+ "63": 0.20522,
+ "64": 0.20085,
+ "65": 0.20105,
+ "66": 0.2007,
+ "67": 0.2013,
+ "68": 0.2027,
+ "69": 0.20016,
+ "70": 0.20063,
+ "71": 0.20993,
+ "72": 0.20128,
+ "73": 0.20208,
+ "74": 0.21714,
+ "75": 0.21499,
+ "76": 0.2105,
+ "77": 0.20889,
+ "78": 0.21225,
+ "79": 0.21534,
+ "80": 0.21713,
+ "81": 0.22213,
+ "82": 0.21058,
+ "83": 0.26098,
+ "84": 0.22386,
+ "85": 0.22371,
+ "86": 0.22214,
+ "87": 0.20121,
+ "88": 0.21264,
+ "89": 0.24222,
+ "90": 0.22288,
+ "91": 0.2238,
+ "92": 0.22792,
+ "93": 0.22505,
+ "94": 0.22473,
+ "95": 0.24128,
+ "96": 0.25457,
+ "97": 0.24471,
+ "98": 0.25068,
+ "99": 0.22442,
+ "100": 0.24497
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer/model_config.yaml
index 8672163186c..764c576645e 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -60,4 +60,6 @@ MODEL_ARGS:
--disable-bias-linear: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_groupedGEMM/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_groupedGEMM/model_config.yaml
index 9a827a4ee72..381039b4905 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_groupedGEMM/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_groupedGEMM/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -56,4 +56,6 @@ MODEL_ARGS:
--bf16: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances/golden_values_dev_dgx_gb200.json
index 31167be6de5..160ba7e60e2 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances/golden_values_dev_dgx_gb200.json
@@ -6,104 +6,104 @@
"values": {
"1": 10.7915,
"2": 10.8072,
- "3": 10.79214,
- "4": 10.77571,
- "5": 10.82495,
- "6": 10.83193,
- "7": 10.82077,
- "8": 10.81496,
- "9": 10.81973,
- "10": 10.7692,
- "11": 10.84519,
- "12": 10.83101,
- "13": 10.83652,
- "14": 10.85771,
- "15": 10.80581,
- "16": 10.78733,
- "17": 10.75844,
- "18": 10.79297,
- "19": 10.78295,
- "20": 10.73199,
- "21": 10.70953,
- "22": 10.57675,
- "23": 10.71651,
- "24": 10.61983,
- "25": 10.58207,
- "26": 10.61694,
- "27": 10.6509,
- "28": 10.63261,
- "29": 10.63024,
- "30": 10.4432,
- "31": 10.19983,
- "32": 10.52048,
- "33": 10.5079,
- "34": 10.29565,
- "35": 10.33536,
- "36": 10.30278,
- "37": 10.41788,
- "38": 10.28121,
- "39": 10.46185,
- "40": 10.18169,
- "41": 10.21391,
- "42": 10.28457,
- "43": 9.95538,
- "44": 10.05751,
- "45": 9.95713,
- "46": 9.93528,
- "47": 10.22675,
- "48": 9.96521,
- "49": 9.66603,
- "50": 10.009,
- "51": 9.94789,
- "52": 9.84665,
- "53": 10.14887,
- "54": 10.03772,
- "55": 9.97445,
- "56": 9.71378,
- "57": 9.59509,
- "58": 9.92081,
- "59": 9.67609,
- "60": 9.61253,
- "61": 9.79221,
- "62": 10.05653,
- "63": 9.47849,
- "64": 9.84455,
- "65": 9.03889,
- "66": 9.78399,
- "67": 9.43609,
- "68": 9.85203,
- "69": 9.84438,
- "70": 9.7933,
- "71": 9.69163,
- "72": 9.64909,
- "73": 9.55528,
- "74": 9.04743,
- "75": 9.49185,
- "76": 9.178,
- "77": 10.11275,
- "78": 9.76838,
- "79": 9.4398,
- "80": 9.45421,
- "81": 9.53191,
- "82": 9.75402,
- "83": 9.38186,
- "84": 9.46162,
- "85": 9.66959,
- "86": 9.1349,
- "87": 9.6343,
- "88": 9.80083,
- "89": 9.66682,
- "90": 9.86175,
- "91": 9.39987,
- "92": 9.41063,
- "93": 9.14654,
- "94": 8.88648,
- "95": 9.56986,
- "96": 9.57642,
- "97": 9.34305,
- "98": 9.72786,
- "99": 8.96203,
- "100": 9.44942
+ "3": 10.79211,
+ "4": 10.77572,
+ "5": 10.82476,
+ "6": 10.83254,
+ "7": 10.81992,
+ "8": 10.81584,
+ "9": 10.81974,
+ "10": 10.76945,
+ "11": 10.84429,
+ "12": 10.83036,
+ "13": 10.83709,
+ "14": 10.85718,
+ "15": 10.80556,
+ "16": 10.78705,
+ "17": 10.75861,
+ "18": 10.79283,
+ "19": 10.78296,
+ "20": 10.73173,
+ "21": 10.71002,
+ "22": 10.5767,
+ "23": 10.71556,
+ "24": 10.6198,
+ "25": 10.58215,
+ "26": 10.61717,
+ "27": 10.651,
+ "28": 10.63244,
+ "29": 10.63052,
+ "30": 10.44314,
+ "31": 10.20041,
+ "32": 10.52007,
+ "33": 10.50862,
+ "34": 10.29649,
+ "35": 10.33562,
+ "36": 10.3035,
+ "37": 10.41783,
+ "38": 10.28036,
+ "39": 10.46158,
+ "40": 10.18138,
+ "41": 10.21356,
+ "42": 10.28463,
+ "43": 9.95586,
+ "44": 10.05773,
+ "45": 9.95666,
+ "46": 9.93559,
+ "47": 10.22716,
+ "48": 9.96512,
+ "49": 9.66568,
+ "50": 10.00925,
+ "51": 9.94787,
+ "52": 9.84659,
+ "53": 10.14898,
+ "54": 10.03733,
+ "55": 9.97466,
+ "56": 9.71366,
+ "57": 9.59511,
+ "58": 9.92053,
+ "59": 9.67602,
+ "60": 9.61281,
+ "61": 9.79168,
+ "62": 10.05651,
+ "63": 9.47825,
+ "64": 9.84472,
+ "65": 9.03871,
+ "66": 9.78383,
+ "67": 9.43591,
+ "68": 9.85174,
+ "69": 9.84474,
+ "70": 9.79281,
+ "71": 9.69147,
+ "72": 9.64923,
+ "73": 9.55514,
+ "74": 9.04748,
+ "75": 9.49223,
+ "76": 9.17797,
+ "77": 10.11311,
+ "78": 9.76827,
+ "79": 9.44002,
+ "80": 9.4543,
+ "81": 9.5315,
+ "82": 9.7541,
+ "83": 9.3821,
+ "84": 9.46112,
+ "85": 9.67014,
+ "86": 9.13473,
+ "87": 9.63424,
+ "88": 9.80046,
+ "89": 9.66691,
+ "90": 9.86202,
+ "91": 9.39937,
+ "92": 9.41036,
+ "93": 9.14676,
+ "94": 8.88677,
+ "95": 9.5697,
+ "96": 9.57651,
+ "97": 9.34308,
+ "98": 9.72802,
+ "99": 8.96192,
+ "100": 9.44958
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 31217.0,
- "2": 33106.0,
- "3": 33596.0,
- "4": 31946.0,
- "5": 36783.0,
- "6": 37377.0,
- "7": 35362.0,
- "8": 31711.0,
- "9": 34749.0,
- "10": 29758.0,
- "11": 38348.0,
- "12": 35446.0,
- "13": 37087.0,
- "14": 37869.0,
- "15": 35242.0,
- "16": 36520.0,
- "17": 35190.0,
- "18": 35191.0,
- "19": 35614.0,
- "20": 32571.0,
- "21": 33220.0,
- "22": 30518.0,
- "23": 37619.0,
- "24": 32547.0,
- "25": 30591.0,
- "26": 34546.0,
- "27": 35275.0,
- "28": 36936.0,
- "29": 37531.0,
- "30": 33354.0,
- "31": 30754.0,
- "32": 36331.0,
- "33": 38273.0,
- "34": 32645.0,
- "35": 34237.0,
- "36": 35092.0,
- "37": 37931.0,
- "38": 35480.0,
- "39": 39175.0,
- "40": 36296.0,
- "41": 35902.0,
- "42": 37609.0,
- "43": 33748.0,
- "44": 34027.0,
- "45": 35215.0,
- "46": 37108.0,
- "47": 41056.0,
- "48": 35765.0,
- "49": 35087.0,
- "50": 39734.0,
- "51": 36712.0,
- "52": 36176.0,
- "53": 41774.0,
- "54": 40447.0,
- "55": 37071.0,
- "56": 39975.0,
- "57": 36828.0,
- "58": 41815.0,
- "59": 37962.0,
- "60": 40415.0,
- "61": 39921.0,
- "62": 43840.0,
- "63": 37890.0,
- "64": 42699.0,
- "65": 40347.0,
- "66": 44159.0,
- "67": 40057.0,
- "68": 39563.0,
- "69": 42246.0,
- "70": 44867.0,
- "71": 40910.0,
- "72": 40982.0,
- "73": 44363.0,
- "74": 35672.0,
- "75": 39602.0,
- "76": 46157.0,
- "77": 44919.0,
- "78": 48134.0,
- "79": 48666.0,
- "80": 46770.0,
- "81": 50144.0,
- "82": 49680.0,
- "83": 44991.0,
- "84": 45912.0,
- "85": 49371.0,
- "86": 45600.0,
- "87": 49292.0,
- "88": 46411.0,
- "89": 49710.0,
- "90": 51008.0,
- "91": 45796.0,
- "92": 47991.0,
- "93": 46847.0,
- "94": 47360.0,
- "95": 48680.0,
- "96": 50369.0,
- "97": 46162.0,
- "98": 49921.0,
- "99": 48235.0,
- "100": 45390.0
+ "1": 30880.0,
+ "2": 33134.0,
+ "3": 34050.0,
+ "4": 31735.0,
+ "5": 37029.0,
+ "6": 37301.0,
+ "7": 35264.0,
+ "8": 31446.0,
+ "9": 34979.0,
+ "10": 29776.0,
+ "11": 37948.0,
+ "12": 35317.0,
+ "13": 37121.0,
+ "14": 38011.0,
+ "15": 34963.0,
+ "16": 36503.0,
+ "17": 35632.0,
+ "18": 35207.0,
+ "19": 35782.0,
+ "20": 32724.0,
+ "21": 33425.0,
+ "22": 30572.0,
+ "23": 37790.0,
+ "24": 32343.0,
+ "25": 30529.0,
+ "26": 34671.0,
+ "27": 35017.0,
+ "28": 36377.0,
+ "29": 37929.0,
+ "30": 33302.0,
+ "31": 30382.0,
+ "32": 35917.0,
+ "33": 38367.0,
+ "34": 33007.0,
+ "35": 34400.0,
+ "36": 35079.0,
+ "37": 38363.0,
+ "38": 35530.0,
+ "39": 39156.0,
+ "40": 36183.0,
+ "41": 36060.0,
+ "42": 38064.0,
+ "43": 33730.0,
+ "44": 34062.0,
+ "45": 35453.0,
+ "46": 37450.0,
+ "47": 41140.0,
+ "48": 35440.0,
+ "49": 35339.0,
+ "50": 40009.0,
+ "51": 37067.0,
+ "52": 36328.0,
+ "53": 41921.0,
+ "54": 40571.0,
+ "55": 36973.0,
+ "56": 39943.0,
+ "57": 37141.0,
+ "58": 41862.0,
+ "59": 38103.0,
+ "60": 40594.0,
+ "61": 40414.0,
+ "62": 43834.0,
+ "63": 37595.0,
+ "64": 42651.0,
+ "65": 40408.0,
+ "66": 43557.0,
+ "67": 40116.0,
+ "68": 40091.0,
+ "69": 41858.0,
+ "70": 45155.0,
+ "71": 40651.0,
+ "72": 41736.0,
+ "73": 44373.0,
+ "74": 35726.0,
+ "75": 39266.0,
+ "76": 46415.0,
+ "77": 44954.0,
+ "78": 47867.0,
+ "79": 48322.0,
+ "80": 46736.0,
+ "81": 50266.0,
+ "82": 50166.0,
+ "83": 44518.0,
+ "84": 46086.0,
+ "85": 49137.0,
+ "86": 45604.0,
+ "87": 48987.0,
+ "88": 46500.0,
+ "89": 49453.0,
+ "90": 50338.0,
+ "91": 45534.0,
+ "92": 47623.0,
+ "93": 46662.0,
+ "94": 47357.0,
+ "95": 48641.0,
+ "96": 50298.0,
+ "97": 46694.0,
+ "98": 50054.0,
+ "99": 48186.0,
+ "100": 44915.0
}
},
"mem-allocated-bytes": {
@@ -220,104 +220,104 @@
"values": {
"1": 1254511616.0,
"2": 1254503424.0,
- "3": 1254505984.0,
+ "3": 1254506496.0,
"4": 1254513664.0,
- "5": 1254505984.0,
+ "5": 1254507520.0,
"6": 1254507520.0,
- "7": 1254511104.0,
+ "7": 1254511616.0,
"8": 1254505472.0,
- "9": 1254505984.0,
+ "9": 1254506496.0,
"10": 1254506496.0,
- "11": 1254507520.0,
- "12": 1254503936.0,
- "13": 1254509568.0,
- "14": 1254510080.0,
- "15": 1254506496.0,
+ "11": 1254505472.0,
+ "12": 1254504448.0,
+ "13": 1254507520.0,
+ "14": 1254509568.0,
+ "15": 1254508544.0,
"16": 1254505984.0,
"17": 1254516224.0,
- "18": 1254503424.0,
+ "18": 1254504448.0,
"19": 1254506496.0,
- "20": 1254504960.0,
- "21": 1254508032.0,
- "22": 1254510592.0,
- "23": 1254512640.0,
+ "20": 1254503936.0,
+ "21": 1254508544.0,
+ "22": 1254509056.0,
+ "23": 1254514176.0,
"24": 1254505472.0,
- "25": 1254513664.0,
- "26": 1254512128.0,
- "27": 1254501888.0,
- "28": 1254509056.0,
+ "25": 1254513152.0,
+ "26": 1254510592.0,
+ "27": 1254502912.0,
+ "28": 1254508032.0,
"29": 1254508032.0,
- "30": 1254509056.0,
- "31": 1254519296.0,
- "32": 1254512128.0,
+ "30": 1254508032.0,
+ "31": 1254518784.0,
+ "32": 1254512640.0,
"33": 1254507008.0,
- "34": 1254509056.0,
+ "34": 1254508544.0,
"35": 1254510080.0,
"36": 1254507008.0,
"37": 1254504448.0,
- "38": 1254505472.0,
- "39": 1254508032.0,
- "40": 1254505984.0,
- "41": 1254512128.0,
- "42": 1254504960.0,
- "43": 1254507008.0,
- "44": 1254508032.0,
- "45": 1254506496.0,
+ "38": 1254504448.0,
+ "39": 1254507008.0,
+ "40": 1254506496.0,
+ "41": 1254511616.0,
+ "42": 1254505472.0,
+ "43": 1254508544.0,
+ "44": 1254509056.0,
+ "45": 1254504960.0,
"46": 1254513664.0,
- "47": 1254507008.0,
+ "47": 1254505984.0,
"48": 1254511616.0,
- "49": 1254508032.0,
- "50": 1254506496.0,
- "51": 1254508032.0,
- "52": 1254513152.0,
- "53": 1254505984.0,
- "54": 1254508544.0,
- "55": 1254503936.0,
- "56": 1254502912.0,
- "57": 1254515200.0,
- "58": 1254503936.0,
+ "49": 1254509056.0,
+ "50": 1254505984.0,
+ "51": 1254509056.0,
+ "52": 1254512640.0,
+ "53": 1254506496.0,
+ "54": 1254509056.0,
+ "55": 1254504448.0,
+ "56": 1254503424.0,
+ "57": 1254514176.0,
+ "58": 1254504448.0,
"59": 1254508544.0,
- "60": 1254503936.0,
- "61": 1254507008.0,
- "62": 1254508032.0,
- "63": 1254507520.0,
+ "60": 1254505984.0,
+ "61": 1254505984.0,
+ "62": 1254507008.0,
+ "63": 1254508032.0,
"64": 1254502400.0,
- "65": 1254509568.0,
+ "65": 1254508544.0,
"66": 1254501376.0,
"67": 1254499328.0,
- "68": 1254503936.0,
- "69": 1254499328.0,
+ "68": 1254504960.0,
+ "69": 1254498816.0,
"70": 1254502912.0,
- "71": 1254507520.0,
+ "71": 1254508032.0,
"72": 1254499328.0,
- "73": 1254497280.0,
- "74": 1254486016.0,
+ "73": 1254498304.0,
+ "74": 1254485504.0,
"75": 1254504960.0,
- "76": 1254507008.0,
- "77": 1254504448.0,
+ "76": 1254506496.0,
+ "77": 1254503936.0,
"78": 1254496256.0,
- "79": 1254500864.0,
- "80": 1254491648.0,
- "81": 1254493696.0,
- "82": 1254497280.0,
+ "79": 1254501376.0,
+ "80": 1254492160.0,
+ "81": 1254492160.0,
+ "82": 1254497792.0,
"83": 1254505984.0,
- "84": 1254489600.0,
- "85": 1254505984.0,
- "86": 1254500352.0,
+ "84": 1254490112.0,
+ "85": 1254507520.0,
+ "86": 1254499328.0,
"87": 1254505472.0,
- "88": 1254506496.0,
- "89": 1254498304.0,
- "90": 1254504448.0,
- "91": 1254501888.0,
- "92": 1254501888.0,
+ "88": 1254505472.0,
+ "89": 1254497792.0,
+ "90": 1254503424.0,
+ "91": 1254503424.0,
+ "92": 1254503424.0,
"93": 1254499328.0,
- "94": 1254494720.0,
- "95": 1254504960.0,
- "96": 1254503424.0,
+ "94": 1254497280.0,
+ "95": 1254505984.0,
+ "96": 1254502912.0,
"97": 1254492672.0,
- "98": 1254505984.0,
- "99": 1254499328.0,
- "100": 1254501888.0
+ "98": 1254507008.0,
+ "99": 1254501376.0,
+ "100": 1254500864.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 2066381824.0,
- "2": 2543623168.0,
- "3": 2544637440.0,
- "4": 2550310912.0,
- "5": 2550310912.0,
- "6": 2550310912.0,
- "7": 2550310912.0,
- "8": 2550310912.0,
- "9": 2550310912.0,
- "10": 2550310912.0,
- "11": 2550310912.0,
- "12": 2550310912.0,
- "13": 2550310912.0,
- "14": 2550310912.0,
- "15": 2550310912.0,
- "16": 2550310912.0,
- "17": 2554848768.0,
- "18": 2554848768.0,
- "19": 2554848768.0,
- "20": 2554848768.0,
- "21": 2554848768.0,
- "22": 2554848768.0,
- "23": 2554848768.0,
- "24": 2554848768.0,
- "25": 2554848768.0,
- "26": 2554848768.0,
- "27": 2554848768.0,
- "28": 2554848768.0,
- "29": 2554848768.0,
- "30": 2554848768.0,
- "31": 2556047872.0,
- "32": 2556047872.0,
- "33": 2556047872.0,
- "34": 2556047872.0,
- "35": 2556047872.0,
- "36": 2556047872.0,
- "37": 2556047872.0,
- "38": 2556047872.0,
- "39": 2556047872.0,
- "40": 2556047872.0,
- "41": 2556047872.0,
- "42": 2556047872.0,
- "43": 2556047872.0,
- "44": 2556047872.0,
- "45": 2556047872.0,
- "46": 2556047872.0,
- "47": 2556047872.0,
- "48": 2556047872.0,
- "49": 2556047872.0,
- "50": 2556047872.0,
- "51": 2556047872.0,
- "52": 2556047872.0,
- "53": 2556047872.0,
- "54": 2556047872.0,
- "55": 2556047872.0,
- "56": 2556047872.0,
- "57": 2556047872.0,
- "58": 2556047872.0,
- "59": 2556047872.0,
- "60": 2556047872.0,
- "61": 2556047872.0,
- "62": 2556047872.0,
- "63": 2556047872.0,
- "64": 2556047872.0,
- "65": 2556047872.0,
- "66": 2556047872.0,
- "67": 2556047872.0,
- "68": 2556047872.0,
- "69": 2556047872.0,
- "70": 2556047872.0,
- "71": 2556047872.0,
- "72": 2556047872.0,
- "73": 2556047872.0,
- "74": 2556047872.0,
- "75": 2556047872.0,
- "76": 2556047872.0,
- "77": 2556047872.0,
- "78": 2556047872.0,
- "79": 2556047872.0,
- "80": 2556047872.0,
- "81": 2556047872.0,
- "82": 2556047872.0,
- "83": 2556047872.0,
- "84": 2556047872.0,
- "85": 2556047872.0,
- "86": 2556047872.0,
- "87": 2556047872.0,
- "88": 2556047872.0,
- "89": 2556047872.0,
- "90": 2556047872.0,
- "91": 2556047872.0,
- "92": 2556047872.0,
- "93": 2556047872.0,
- "94": 2556047872.0,
- "95": 2556047872.0,
- "96": 2556047872.0,
- "97": 2556047872.0,
- "98": 2556047872.0,
- "99": 2556047872.0,
- "100": 2556047872.0
+ "1": 2066350080.0,
+ "2": 2542998528.0,
+ "3": 2544289792.0,
+ "4": 2549362688.0,
+ "5": 2549362688.0,
+ "6": 2549362688.0,
+ "7": 2549362688.0,
+ "8": 2549362688.0,
+ "9": 2549362688.0,
+ "10": 2549362688.0,
+ "11": 2549362688.0,
+ "12": 2549362688.0,
+ "13": 2549362688.0,
+ "14": 2549362688.0,
+ "15": 2549362688.0,
+ "16": 2549362688.0,
+ "17": 2552193536.0,
+ "18": 2552193536.0,
+ "19": 2552193536.0,
+ "20": 2552193536.0,
+ "21": 2552193536.0,
+ "22": 2552193536.0,
+ "23": 2552193536.0,
+ "24": 2552193536.0,
+ "25": 2552193536.0,
+ "26": 2552193536.0,
+ "27": 2552193536.0,
+ "28": 2552193536.0,
+ "29": 2552193536.0,
+ "30": 2552193536.0,
+ "31": 2555549696.0,
+ "32": 2555549696.0,
+ "33": 2555549696.0,
+ "34": 2555549696.0,
+ "35": 2555549696.0,
+ "36": 2555549696.0,
+ "37": 2555549696.0,
+ "38": 2555549696.0,
+ "39": 2555549696.0,
+ "40": 2555549696.0,
+ "41": 2555549696.0,
+ "42": 2555549696.0,
+ "43": 2555549696.0,
+ "44": 2555549696.0,
+ "45": 2555549696.0,
+ "46": 2555549696.0,
+ "47": 2555549696.0,
+ "48": 2555549696.0,
+ "49": 2555549696.0,
+ "50": 2555549696.0,
+ "51": 2555549696.0,
+ "52": 2555549696.0,
+ "53": 2555549696.0,
+ "54": 2555549696.0,
+ "55": 2555549696.0,
+ "56": 2555549696.0,
+ "57": 2555549696.0,
+ "58": 2555549696.0,
+ "59": 2555549696.0,
+ "60": 2555549696.0,
+ "61": 2555549696.0,
+ "62": 2555549696.0,
+ "63": 2555549696.0,
+ "64": 2555549696.0,
+ "65": 2555549696.0,
+ "66": 2555549696.0,
+ "67": 2555549696.0,
+ "68": 2555549696.0,
+ "69": 2555549696.0,
+ "70": 2555549696.0,
+ "71": 2555549696.0,
+ "72": 2555549696.0,
+ "73": 2555549696.0,
+ "74": 2555549696.0,
+ "75": 2555549696.0,
+ "76": 2555549696.0,
+ "77": 2555549696.0,
+ "78": 2555549696.0,
+ "79": 2555549696.0,
+ "80": 2555549696.0,
+ "81": 2555549696.0,
+ "82": 2555549696.0,
+ "83": 2555549696.0,
+ "84": 2555549696.0,
+ "85": 2555549696.0,
+ "86": 2555549696.0,
+ "87": 2555549696.0,
+ "88": 2555549696.0,
+ "89": 2555549696.0,
+ "90": 2555549696.0,
+ "91": 2555549696.0,
+ "92": 2555549696.0,
+ "93": 2555549696.0,
+ "94": 2555549696.0,
+ "95": 2555549696.0,
+ "96": 2555549696.0,
+ "97": 2555549696.0,
+ "98": 2555549696.0,
+ "99": 2555549696.0,
+ "100": 2555549696.0
}
},
"iteration-time": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 4.80614,
- "3": 0.71249,
- "4": 0.40839,
- "5": 0.39358,
- "6": 0.3911,
- "7": 0.39032,
- "8": 0.38318,
- "9": 0.70382,
- "10": 0.34707,
- "11": 0.34403,
- "12": 0.34043,
- "13": 0.33959,
- "14": 0.33461,
- "15": 0.34767,
- "16": 0.33495,
- "17": 0.34839,
- "18": 0.33673,
- "19": 0.33335,
- "20": 0.33161,
- "21": 0.32643,
- "22": 0.33565,
- "23": 0.33625,
- "24": 0.33009,
- "25": 0.33065,
- "26": 0.33344,
- "27": 0.33552,
- "28": 0.33047,
- "29": 0.33011,
- "30": 0.33358,
- "31": 0.34631,
- "32": 0.33536,
- "33": 0.33271,
- "34": 0.33949,
- "35": 0.33073,
- "36": 0.32877,
- "37": 0.32806,
- "38": 0.33111,
- "39": 0.33408,
- "40": 0.33428,
- "41": 0.34927,
- "42": 1.47745,
- "43": 0.48012,
- "44": 0.33077,
- "45": 0.33262,
- "46": 0.34066,
- "47": 0.33152,
- "48": 0.33512,
- "49": 0.34429,
- "50": 0.33697,
- "51": 0.34656,
- "52": 0.337,
- "53": 0.33133,
- "54": 0.33172,
- "55": 0.33188,
- "56": 0.35163,
- "57": 0.34162,
- "58": 0.33258,
- "59": 0.7122,
- "60": 0.33979,
- "61": 0.33569,
- "62": 0.33523,
- "63": 0.33864,
- "64": 0.34776,
- "65": 0.37658,
- "66": 0.3377,
- "67": 0.36916,
- "68": 0.3452,
- "69": 0.33854,
- "70": 0.34023,
- "71": 0.3544,
- "72": 0.34395,
- "73": 0.3567,
- "74": 0.35025,
- "75": 0.35164,
- "76": 0.35012,
- "77": 0.3364,
- "78": 0.34491,
- "79": 0.34789,
- "80": 0.35388,
- "81": 0.34075,
- "82": 0.34743,
- "83": 0.34211,
- "84": 0.34722,
- "85": 0.33956,
- "86": 0.35402,
- "87": 0.34301,
- "88": 0.34056,
- "89": 0.35764,
- "90": 0.33476,
- "91": 0.3539,
- "92": 0.34448,
- "93": 0.34895,
- "94": 0.3624,
- "95": 0.34001,
- "96": 0.3382,
- "97": 0.35217,
- "98": 0.33252,
- "99": 0.34909,
- "100": 0.34966
+ "2": 8.3521,
+ "3": 0.41669,
+ "4": 0.41284,
+ "5": 0.39807,
+ "6": 0.39421,
+ "7": 0.40258,
+ "8": 0.39083,
+ "9": 0.38556,
+ "10": 0.39045,
+ "11": 0.38493,
+ "12": 0.38652,
+ "13": 0.38654,
+ "14": 0.38244,
+ "15": 0.39635,
+ "16": 0.38332,
+ "17": 0.38972,
+ "18": 0.38226,
+ "19": 0.37884,
+ "20": 0.38559,
+ "21": 0.41074,
+ "22": 0.88305,
+ "23": 0.38557,
+ "24": 0.38914,
+ "25": 0.38815,
+ "26": 0.38762,
+ "27": 0.38501,
+ "28": 0.38776,
+ "29": 0.38603,
+ "30": 0.38705,
+ "31": 0.38791,
+ "32": 0.3888,
+ "33": 0.38221,
+ "34": 0.40037,
+ "35": 0.387,
+ "36": 0.3968,
+ "37": 0.3961,
+ "38": 0.38866,
+ "39": 0.3965,
+ "40": 0.389,
+ "41": 0.38873,
+ "42": 0.3895,
+ "43": 0.38428,
+ "44": 0.38845,
+ "45": 0.38913,
+ "46": 0.38268,
+ "47": 0.38233,
+ "48": 0.38844,
+ "49": 0.3963,
+ "50": 0.38408,
+ "51": 0.42069,
+ "52": 0.38714,
+ "53": 0.38472,
+ "54": 0.38779,
+ "55": 0.38914,
+ "56": 0.40949,
+ "57": 0.39708,
+ "58": 0.39612,
+ "59": 0.3944,
+ "60": 0.38052,
+ "61": 0.39057,
+ "62": 0.38649,
+ "63": 0.4128,
+ "64": 0.38573,
+ "65": 0.41257,
+ "66": 0.39183,
+ "67": 0.42577,
+ "68": 0.4009,
+ "69": 0.38896,
+ "70": 0.39152,
+ "71": 0.40707,
+ "72": 0.40087,
+ "73": 0.41537,
+ "74": 0.43436,
+ "75": 0.45456,
+ "76": 0.41193,
+ "77": 0.38627,
+ "78": 0.39302,
+ "79": 0.40532,
+ "80": 0.39666,
+ "81": 0.39606,
+ "82": 0.39713,
+ "83": 0.39956,
+ "84": 0.39716,
+ "85": 0.40563,
+ "86": 0.40773,
+ "87": 0.39254,
+ "88": 0.402,
+ "89": 0.40411,
+ "90": 0.40046,
+ "91": 0.41284,
+ "92": 0.39708,
+ "93": 0.40273,
+ "94": 0.42216,
+ "95": 0.40147,
+ "96": 0.39007,
+ "97": 0.40448,
+ "98": 0.40171,
+ "99": 0.40175,
+ "100": 0.39847
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances/golden_values_dev_dgx_h100.json
index bf57cfecddc..b7b8787cd8f 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.78091,
- "2": 10.80272,
- "3": 10.8036,
- "4": 10.77566,
- "5": 10.83259,
- "6": 10.83704,
- "7": 10.79728,
- "8": 10.79467,
- "9": 10.80828,
- "10": 10.76154,
- "11": 10.85384,
- "12": 10.84189,
- "13": 10.82465,
- "14": 10.85824,
- "15": 10.78235,
- "16": 10.77923,
- "17": 10.7484,
- "18": 10.78919,
- "19": 10.77567,
- "20": 10.71707,
- "21": 10.70767,
- "22": 10.54782,
- "23": 10.72977,
- "24": 10.60346,
- "25": 10.55815,
- "26": 10.61659,
- "27": 10.6449,
- "28": 10.62536,
- "29": 10.6349,
- "30": 10.42303,
- "31": 10.16459,
- "32": 10.51284,
- "33": 10.50836,
- "34": 10.2667,
- "35": 10.32353,
- "36": 10.2895,
- "37": 10.41051,
- "38": 10.26406,
- "39": 10.44988,
- "40": 10.17537,
- "41": 10.20908,
- "42": 10.27843,
- "43": 9.91808,
- "44": 10.03128,
- "45": 9.92032,
- "46": 9.88579,
- "47": 10.19208,
- "48": 9.92758,
- "49": 9.61634,
- "50": 9.98512,
- "51": 9.90532,
- "52": 9.8039,
- "53": 10.12749,
- "54": 10.00016,
- "55": 9.93664,
- "56": 9.68581,
- "57": 9.55837,
- "58": 9.90508,
- "59": 9.63839,
- "60": 9.57464,
- "61": 9.76841,
- "62": 10.03826,
- "63": 9.44553,
- "64": 9.82755,
- "65": 9.00746,
- "66": 9.77476,
- "67": 9.41315,
- "68": 9.84101,
- "69": 9.8283,
- "70": 9.79049,
- "71": 9.66947,
- "72": 9.62799,
- "73": 9.54696,
- "74": 9.03684,
- "75": 9.49167,
- "76": 9.16779,
- "77": 10.1088,
- "78": 9.77072,
- "79": 9.43806,
- "80": 9.45438,
- "81": 9.5225,
- "82": 9.74228,
- "83": 9.36999,
- "84": 9.45397,
- "85": 9.65808,
- "86": 9.12501,
- "87": 9.62705,
- "88": 9.79641,
- "89": 9.66075,
- "90": 9.8512,
- "91": 9.39414,
- "92": 9.40741,
- "93": 9.13573,
- "94": 8.89066,
- "95": 9.56273,
- "96": 9.5712,
- "97": 9.34355,
- "98": 9.73013,
- "99": 8.95039,
- "100": 9.44212
+ "1": 10.92624,
+ "2": 10.91761,
+ "3": 10.92333,
+ "4": 10.92688,
+ "5": 10.92998,
+ "6": 10.92516,
+ "7": 10.9275,
+ "8": 10.91969,
+ "9": 10.92241,
+ "10": 10.92325,
+ "11": 10.90758,
+ "12": 10.92281,
+ "13": 10.90611,
+ "14": 10.88927,
+ "15": 10.89641,
+ "16": 10.87453,
+ "17": 10.88139,
+ "18": 10.8804,
+ "19": 10.87739,
+ "20": 10.82169,
+ "21": 10.82193,
+ "22": 10.79733,
+ "23": 10.80911,
+ "24": 10.77025,
+ "25": 10.77926,
+ "26": 10.75659,
+ "27": 10.74995,
+ "28": 10.68537,
+ "29": 10.65808,
+ "30": 10.62706,
+ "31": 10.61225,
+ "32": 10.60551,
+ "33": 10.57906,
+ "34": 10.53561,
+ "35": 10.53697,
+ "36": 10.52757,
+ "37": 10.4965,
+ "38": 10.49545,
+ "39": 10.46027,
+ "40": 10.4421,
+ "41": 10.42212,
+ "42": 10.41463,
+ "43": 10.39809,
+ "44": 10.36678,
+ "45": 10.37212,
+ "46": 10.33609,
+ "47": 10.31814,
+ "48": 10.2871,
+ "49": 10.27515,
+ "50": 10.27458,
+ "51": 10.26989,
+ "52": 10.22227,
+ "53": 10.22935,
+ "54": 10.19391,
+ "55": 10.17367,
+ "56": 10.1885,
+ "57": 10.17867,
+ "58": 10.18563,
+ "59": 10.13475,
+ "60": 10.15585,
+ "61": 10.1061,
+ "62": 10.07416,
+ "63": 10.13232,
+ "64": 10.09184,
+ "65": 10.06796,
+ "66": 10.09491,
+ "67": 10.07263,
+ "68": 10.03568,
+ "69": 10.05528,
+ "70": 10.03301,
+ "71": 10.04951,
+ "72": 10.04131,
+ "73": 10.03884,
+ "74": 10.0213,
+ "75": 9.98199,
+ "76": 10.00336,
+ "77": 10.01449,
+ "78": 9.96848,
+ "79": 9.96952,
+ "80": 9.9854,
+ "81": 10.01106,
+ "82": 9.94235,
+ "83": 9.91708,
+ "84": 9.85309,
+ "85": 9.83718,
+ "86": 9.94391,
+ "87": 9.95588,
+ "88": 9.93524,
+ "89": 9.87143,
+ "90": 9.87914,
+ "91": 9.89531,
+ "92": 9.86941,
+ "93": 9.80184,
+ "94": 9.88889,
+ "95": 9.86035,
+ "96": 9.84915,
+ "97": 9.79916,
+ "98": 9.83229,
+ "99": 9.86435,
+ "100": 9.76028
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 30994.0,
- "2": 32962.0,
- "3": 33026.0,
- "4": 30732.0,
- "5": 36042.0,
- "6": 36987.0,
- "7": 34490.0,
- "8": 31442.0,
- "9": 33931.0,
- "10": 29993.0,
- "11": 37681.0,
- "12": 34978.0,
- "13": 36675.0,
- "14": 37601.0,
- "15": 34369.0,
- "16": 36581.0,
- "17": 34615.0,
- "18": 34408.0,
- "19": 35362.0,
- "20": 32532.0,
- "21": 33181.0,
- "22": 30426.0,
- "23": 37807.0,
- "24": 32299.0,
- "25": 30879.0,
- "26": 33994.0,
- "27": 34721.0,
- "28": 36576.0,
- "29": 37196.0,
- "30": 32443.0,
- "31": 30177.0,
- "32": 35948.0,
- "33": 37549.0,
- "34": 32243.0,
- "35": 33961.0,
- "36": 34340.0,
- "37": 37853.0,
- "38": 35694.0,
- "39": 38797.0,
- "40": 36317.0,
- "41": 35380.0,
- "42": 36704.0,
- "43": 34045.0,
- "44": 33691.0,
- "45": 35877.0,
- "46": 36737.0,
- "47": 40148.0,
- "48": 36696.0,
- "49": 36203.0,
- "50": 38688.0,
- "51": 37791.0,
- "52": 37021.0,
- "53": 41944.0,
- "54": 40947.0,
- "55": 37727.0,
- "56": 40761.0,
- "57": 37481.0,
- "58": 41787.0,
- "59": 39365.0,
- "60": 40922.0,
- "61": 41100.0,
- "62": 43388.0,
- "63": 38269.0,
- "64": 43526.0,
- "65": 41821.0,
- "66": 44876.0,
- "67": 42497.0,
- "68": 39967.0,
- "69": 41255.0,
- "70": 45781.0,
- "71": 42348.0,
- "72": 42151.0,
- "73": 45043.0,
- "74": 35705.0,
- "75": 39397.0,
- "76": 45340.0,
- "77": 45670.0,
- "78": 46614.0,
- "79": 49159.0,
- "80": 47317.0,
- "81": 51048.0,
- "82": 49312.0,
- "83": 45257.0,
- "84": 45494.0,
- "85": 49366.0,
- "86": 45783.0,
- "87": 50223.0,
- "88": 47536.0,
- "89": 48826.0,
- "90": 49499.0,
- "91": 45726.0,
- "92": 47926.0,
- "93": 46433.0,
- "94": 47675.0,
- "95": 47504.0,
- "96": 50174.0,
- "97": 46465.0,
- "98": 49255.0,
- "99": 48053.0,
- "100": 44507.0
+ "1": 36362.0,
+ "2": 36609.0,
+ "3": 36699.0,
+ "4": 36555.0,
+ "5": 36453.0,
+ "6": 36089.0,
+ "7": 37058.0,
+ "8": 36230.0,
+ "9": 37158.0,
+ "10": 37488.0,
+ "11": 36342.0,
+ "12": 35261.0,
+ "13": 36789.0,
+ "14": 37057.0,
+ "15": 36340.0,
+ "16": 35948.0,
+ "17": 35839.0,
+ "18": 36187.0,
+ "19": 36474.0,
+ "20": 36432.0,
+ "21": 36003.0,
+ "22": 36478.0,
+ "23": 36667.0,
+ "24": 36803.0,
+ "25": 35489.0,
+ "26": 36681.0,
+ "27": 36435.0,
+ "28": 35477.0,
+ "29": 35626.0,
+ "30": 35422.0,
+ "31": 35906.0,
+ "32": 36408.0,
+ "33": 35212.0,
+ "34": 35448.0,
+ "35": 36258.0,
+ "36": 35159.0,
+ "37": 35748.0,
+ "38": 35210.0,
+ "39": 36697.0,
+ "40": 37410.0,
+ "41": 35947.0,
+ "42": 35734.0,
+ "43": 36629.0,
+ "44": 36434.0,
+ "45": 39424.0,
+ "46": 37606.0,
+ "47": 38380.0,
+ "48": 38425.0,
+ "49": 42025.0,
+ "50": 38729.0,
+ "51": 38953.0,
+ "52": 41533.0,
+ "53": 40334.0,
+ "54": 41563.0,
+ "55": 38730.0,
+ "56": 40622.0,
+ "57": 37188.0,
+ "58": 45588.0,
+ "59": 42468.0,
+ "60": 42712.0,
+ "61": 41432.0,
+ "62": 45084.0,
+ "63": 43313.0,
+ "64": 49910.0,
+ "65": 41404.0,
+ "66": 45308.0,
+ "67": 49836.0,
+ "68": 47343.0,
+ "69": 45252.0,
+ "70": 47321.0,
+ "71": 45979.0,
+ "72": 45381.0,
+ "73": 47947.0,
+ "74": 45948.0,
+ "75": 47248.0,
+ "76": 47233.0,
+ "77": 47359.0,
+ "78": 49473.0,
+ "79": 47402.0,
+ "80": 45358.0,
+ "81": 54002.0,
+ "82": 42501.0,
+ "83": 46352.0,
+ "84": 45583.0,
+ "85": 45669.0,
+ "86": 44994.0,
+ "87": 37014.0,
+ "88": 46077.0,
+ "89": 49463.0,
+ "90": 56966.0,
+ "91": 38031.0,
+ "92": 49758.0,
+ "93": 46854.0,
+ "94": 42811.0,
+ "95": 46033.0,
+ "96": 50077.0,
+ "97": 47335.0,
+ "98": 49463.0,
+ "99": 44429.0,
+ "100": 46003.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1254502400.0,
- "2": 1254506496.0,
- "3": 1254506496.0,
- "4": 1254502400.0,
- "5": 1254502912.0,
- "6": 1254504448.0,
- "7": 1254504960.0,
- "8": 1254504960.0,
- "9": 1254502912.0,
- "10": 1254504448.0,
- "11": 1254504960.0,
- "12": 1254503936.0,
- "13": 1254501888.0,
- "14": 1254506496.0,
- "15": 1254505472.0,
- "16": 1254504448.0,
- "17": 1254505472.0,
- "18": 1254503424.0,
- "19": 1254504960.0,
- "20": 1254504448.0,
- "21": 1254504448.0,
- "22": 1254502400.0,
- "23": 1254501888.0,
- "24": 1254504448.0,
- "25": 1254501376.0,
- "26": 1254503424.0,
- "27": 1254502912.0,
- "28": 1254503936.0,
- "29": 1254506496.0,
- "30": 1254501376.0,
- "31": 1254500352.0,
- "32": 1254501376.0,
- "33": 1254503936.0,
- "34": 1254503936.0,
- "35": 1254502912.0,
- "36": 1254506496.0,
- "37": 1254504448.0,
- "38": 1254504960.0,
- "39": 1254503936.0,
- "40": 1254503936.0,
- "41": 1254504448.0,
- "42": 1254503936.0,
- "43": 1254500864.0,
- "44": 1254502400.0,
- "45": 1254503424.0,
- "46": 1254501888.0,
- "47": 1254504960.0,
- "48": 1254500864.0,
- "49": 1254501376.0,
- "50": 1254503936.0,
- "51": 1254497792.0,
- "52": 1254497280.0,
- "53": 1254498816.0,
- "54": 1254499328.0,
- "55": 1254501376.0,
- "56": 1254502912.0,
- "57": 1254494208.0,
- "58": 1254499328.0,
- "59": 1254496256.0,
- "60": 1254497792.0,
- "61": 1254505984.0,
- "62": 1254504960.0,
- "63": 1254500352.0,
- "64": 1254499840.0,
- "65": 1254489600.0,
- "66": 1254503936.0,
- "67": 1254499328.0,
- "68": 1254507008.0,
- "69": 1254502400.0,
- "70": 1254503936.0,
- "71": 1254505984.0,
- "72": 1254497280.0,
- "73": 1254505472.0,
- "74": 1254496256.0,
- "75": 1254505472.0,
- "76": 1254504448.0,
- "77": 1254504960.0,
- "78": 1254501376.0,
- "79": 1254501888.0,
- "80": 1254500864.0,
- "81": 1254504448.0,
- "82": 1254501376.0,
- "83": 1254498816.0,
- "84": 1254498304.0,
- "85": 1254500352.0,
- "86": 1254499840.0,
- "87": 1254506496.0,
- "88": 1254500352.0,
- "89": 1254501888.0,
- "90": 1254503936.0,
- "91": 1254506496.0,
- "92": 1254503936.0,
- "93": 1254506496.0,
- "94": 1254501376.0,
- "95": 1254502912.0,
- "96": 1254502912.0,
- "97": 1254500352.0,
- "98": 1254508544.0,
- "99": 1254498304.0,
- "100": 1254500864.0
+ "1": 1254751744.0,
+ "2": 1254746624.0,
+ "3": 1254748160.0,
+ "4": 1254748160.0,
+ "5": 1254748672.0,
+ "6": 1254751744.0,
+ "7": 1254750208.0,
+ "8": 1254747648.0,
+ "9": 1254748672.0,
+ "10": 1254747136.0,
+ "11": 1254748672.0,
+ "12": 1254746624.0,
+ "13": 1254749696.0,
+ "14": 1254749184.0,
+ "15": 1254746112.0,
+ "16": 1254746112.0,
+ "17": 1254747136.0,
+ "18": 1254747648.0,
+ "19": 1254752256.0,
+ "20": 1254746624.0,
+ "21": 1254748672.0,
+ "22": 1254749184.0,
+ "23": 1254749184.0,
+ "24": 1254747648.0,
+ "25": 1254747136.0,
+ "26": 1254745600.0,
+ "27": 1254748672.0,
+ "28": 1254748160.0,
+ "29": 1254750208.0,
+ "30": 1254747136.0,
+ "31": 1254749184.0,
+ "32": 1254747136.0,
+ "33": 1254747648.0,
+ "34": 1254747136.0,
+ "35": 1254748160.0,
+ "36": 1254748160.0,
+ "37": 1254748160.0,
+ "38": 1254747136.0,
+ "39": 1254747648.0,
+ "40": 1254750720.0,
+ "41": 1254750208.0,
+ "42": 1254748672.0,
+ "43": 1254746624.0,
+ "44": 1254748160.0,
+ "45": 1254748672.0,
+ "46": 1254749184.0,
+ "47": 1254749184.0,
+ "48": 1254747136.0,
+ "49": 1254745600.0,
+ "50": 1254750208.0,
+ "51": 1254746112.0,
+ "52": 1254746624.0,
+ "53": 1254748672.0,
+ "54": 1254747136.0,
+ "55": 1254749184.0,
+ "56": 1254747648.0,
+ "57": 1254748672.0,
+ "58": 1254750208.0,
+ "59": 1254747136.0,
+ "60": 1254746112.0,
+ "61": 1254746624.0,
+ "62": 1254747648.0,
+ "63": 1254746112.0,
+ "64": 1254748672.0,
+ "65": 1254747648.0,
+ "66": 1254750208.0,
+ "67": 1254745600.0,
+ "68": 1254750208.0,
+ "69": 1254745088.0,
+ "70": 1254745600.0,
+ "71": 1254748672.0,
+ "72": 1254750208.0,
+ "73": 1254746624.0,
+ "74": 1254746624.0,
+ "75": 1254748160.0,
+ "76": 1254746624.0,
+ "77": 1254742528.0,
+ "78": 1254746112.0,
+ "79": 1254747648.0,
+ "80": 1254745600.0,
+ "81": 1254742528.0,
+ "82": 1254744064.0,
+ "83": 1254748672.0,
+ "84": 1254742016.0,
+ "85": 1254739968.0,
+ "86": 1254733312.0,
+ "87": 1254746624.0,
+ "88": 1254748672.0,
+ "89": 1254739456.0,
+ "90": 1254739968.0,
+ "91": 1254747136.0,
+ "92": 1254736384.0,
+ "93": 1254742016.0,
+ "94": 1254736896.0,
+ "95": 1254732800.0,
+ "96": 1254748672.0,
+ "97": 1254734848.0,
+ "98": 1254728192.0,
+ "99": 1254733824.0,
+ "100": 1254731264.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 2038519808.0,
- "2": 2520255488.0,
- "3": 2520255488.0,
- "4": 2520255488.0,
- "5": 2520552960.0,
- "6": 2520552960.0,
- "7": 2520552960.0,
- "8": 2520552960.0,
- "9": 2520552960.0,
- "10": 2520552960.0,
- "11": 2520552960.0,
- "12": 2520552960.0,
- "13": 2520552960.0,
- "14": 2520552960.0,
- "15": 2520552960.0,
- "16": 2520552960.0,
- "17": 2520552960.0,
- "18": 2520552960.0,
- "19": 2520552960.0,
- "20": 2520552960.0,
- "21": 2520552960.0,
- "22": 2520552960.0,
- "23": 2520552960.0,
- "24": 2520552960.0,
- "25": 2520552960.0,
- "26": 2520552960.0,
- "27": 2520552960.0,
- "28": 2520552960.0,
- "29": 2520552960.0,
- "30": 2520552960.0,
- "31": 2520552960.0,
- "32": 2520552960.0,
- "33": 2521159680.0,
- "34": 2521159680.0,
- "35": 2521159680.0,
- "36": 2521159680.0,
- "37": 2521159680.0,
- "38": 2521159680.0,
- "39": 2521159680.0,
- "40": 2521159680.0,
- "41": 2521159680.0,
- "42": 2521159680.0,
- "43": 2521159680.0,
- "44": 2521159680.0,
- "45": 2521159680.0,
- "46": 2521615360.0,
- "47": 2521615360.0,
- "48": 2521615360.0,
- "49": 2521615360.0,
- "50": 2521615360.0,
- "51": 2521615360.0,
- "52": 2521615360.0,
- "53": 2521615360.0,
- "54": 2521615360.0,
- "55": 2521615360.0,
- "56": 2521615360.0,
- "57": 2521615360.0,
- "58": 2521615360.0,
- "59": 2521615360.0,
- "60": 2521615360.0,
- "61": 2521615360.0,
- "62": 2521615360.0,
- "63": 2521615360.0,
- "64": 2521615360.0,
- "65": 2521615360.0,
- "66": 2521615360.0,
- "67": 2521615360.0,
- "68": 2521615360.0,
- "69": 2521615360.0,
- "70": 2521615360.0,
- "71": 2521615360.0,
- "72": 2521615360.0,
- "73": 2521615360.0,
- "74": 2521615360.0,
- "75": 2521615360.0,
- "76": 2521615360.0,
- "77": 2521615360.0,
- "78": 2521615360.0,
- "79": 2521615360.0,
- "80": 2521615360.0,
- "81": 2521615360.0,
- "82": 2521615360.0,
- "83": 2521615360.0,
- "84": 2521615360.0,
- "85": 2521615360.0,
- "86": 2521615360.0,
- "87": 2521615360.0,
- "88": 2521615360.0,
- "89": 2521615360.0,
- "90": 2521615360.0,
- "91": 2521615360.0,
- "92": 2521615360.0,
- "93": 2521615360.0,
- "94": 2521615360.0,
- "95": 2523076096.0,
- "96": 2523076096.0,
- "97": 2523076096.0,
- "98": 2523076096.0,
- "99": 2523076096.0,
- "100": 2523076096.0
+ "1": 2042493440.0,
+ "2": 2517739520.0,
+ "3": 2518202368.0,
+ "4": 2518202368.0,
+ "5": 2518202368.0,
+ "6": 2519861760.0,
+ "7": 2519861760.0,
+ "8": 2519861760.0,
+ "9": 2519861760.0,
+ "10": 2519861760.0,
+ "11": 2519861760.0,
+ "12": 2519861760.0,
+ "13": 2519861760.0,
+ "14": 2519861760.0,
+ "15": 2519861760.0,
+ "16": 2519861760.0,
+ "17": 2519861760.0,
+ "18": 2519861760.0,
+ "19": 2519861760.0,
+ "20": 2519861760.0,
+ "21": 2519861760.0,
+ "22": 2519861760.0,
+ "23": 2519861760.0,
+ "24": 2519861760.0,
+ "25": 2519861760.0,
+ "26": 2519861760.0,
+ "27": 2519861760.0,
+ "28": 2519861760.0,
+ "29": 2519861760.0,
+ "30": 2519861760.0,
+ "31": 2519861760.0,
+ "32": 2519861760.0,
+ "33": 2519861760.0,
+ "34": 2519861760.0,
+ "35": 2519861760.0,
+ "36": 2519861760.0,
+ "37": 2519861760.0,
+ "38": 2519861760.0,
+ "39": 2519861760.0,
+ "40": 2519861760.0,
+ "41": 2519861760.0,
+ "42": 2519861760.0,
+ "43": 2519861760.0,
+ "44": 2519861760.0,
+ "45": 2519861760.0,
+ "46": 2519861760.0,
+ "47": 2519861760.0,
+ "48": 2519861760.0,
+ "49": 2519861760.0,
+ "50": 2519861760.0,
+ "51": 2519861760.0,
+ "52": 2519861760.0,
+ "53": 2519861760.0,
+ "54": 2519861760.0,
+ "55": 2519861760.0,
+ "56": 2519861760.0,
+ "57": 2519861760.0,
+ "58": 2519861760.0,
+ "59": 2519861760.0,
+ "60": 2519861760.0,
+ "61": 2519861760.0,
+ "62": 2519861760.0,
+ "63": 2519861760.0,
+ "64": 2519861760.0,
+ "65": 2519861760.0,
+ "66": 2519861760.0,
+ "67": 2519861760.0,
+ "68": 2519861760.0,
+ "69": 2519861760.0,
+ "70": 2519861760.0,
+ "71": 2519861760.0,
+ "72": 2519861760.0,
+ "73": 2519861760.0,
+ "74": 2519861760.0,
+ "75": 2519861760.0,
+ "76": 2519861760.0,
+ "77": 2519861760.0,
+ "78": 2519861760.0,
+ "79": 2519861760.0,
+ "80": 2519861760.0,
+ "81": 2519861760.0,
+ "82": 2519861760.0,
+ "83": 2519861760.0,
+ "84": 2519861760.0,
+ "85": 2519861760.0,
+ "86": 2519861760.0,
+ "87": 2519861760.0,
+ "88": 2519861760.0,
+ "89": 2519861760.0,
+ "90": 2519861760.0,
+ "91": 2519861760.0,
+ "92": 2519861760.0,
+ "93": 2519861760.0,
+ "94": 2519861760.0,
+ "95": 2519861760.0,
+ "96": 2519861760.0,
+ "97": 2519861760.0,
+ "98": 2519861760.0,
+ "99": 2519861760.0,
+ "100": 2519861760.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 17.78784,
- "2": 0.2935,
- "3": 0.25416,
- "4": 0.28848,
- "5": 0.27342,
- "6": 0.21986,
- "7": 0.22775,
- "8": 0.21125,
- "9": 0.22242,
- "10": 0.20696,
- "11": 0.21121,
- "12": 0.20562,
- "13": 0.20918,
- "14": 0.20486,
- "15": 0.22312,
- "16": 0.20648,
- "17": 0.21741,
- "18": 0.20596,
- "19": 0.20449,
- "20": 0.20633,
- "21": 0.20648,
- "22": 0.20939,
- "23": 0.20613,
- "24": 0.2098,
- "25": 0.21077,
- "26": 0.20978,
- "27": 0.20622,
- "28": 0.20953,
- "29": 0.2052,
- "30": 0.20858,
- "31": 0.23751,
- "32": 0.20916,
- "33": 0.21528,
- "34": 0.22994,
- "35": 0.20666,
- "36": 0.56591,
- "37": 0.2088,
- "38": 0.20535,
- "39": 0.20334,
- "40": 0.21053,
- "41": 0.20731,
- "42": 0.21647,
- "43": 0.21279,
- "44": 0.20733,
- "45": 0.22499,
- "46": 0.22926,
- "47": 0.21023,
- "48": 0.21769,
- "49": 0.24399,
- "50": 0.21286,
- "51": 0.238,
- "52": 0.23293,
- "53": 0.20987,
- "54": 0.21516,
- "55": 0.22388,
- "56": 0.25985,
- "57": 0.22604,
- "58": 0.61513,
- "59": 0.22219,
- "60": 0.21734,
- "61": 0.90688,
- "62": 0.21705,
- "63": 0.23992,
- "64": 0.21828,
- "65": 0.27683,
- "66": 0.21653,
- "67": 0.27213,
- "68": 0.8349,
- "69": 0.21293,
- "70": 0.21051,
- "71": 0.22862,
- "72": 0.22498,
- "73": 0.24298,
- "74": 0.23094,
- "75": 0.22956,
- "76": 0.24583,
- "77": 0.21646,
- "78": 0.22364,
- "79": 0.22898,
- "80": 0.21878,
- "81": 0.21415,
- "82": 0.21267,
- "83": 0.22485,
- "84": 0.22454,
- "85": 0.21746,
- "86": 0.23031,
- "87": 0.21423,
- "88": 0.21226,
- "89": 0.2196,
- "90": 0.21327,
- "91": 0.23392,
- "92": 0.22086,
- "93": 0.23306,
- "94": 0.24169,
- "95": 0.22202,
- "96": 0.2155,
- "97": 0.22184,
- "98": 0.2139,
- "99": 0.21705,
- "100": 0.21654
+ "1": "nan",
+ "2": 9.13976,
+ "3": 0.25035,
+ "4": 0.23553,
+ "5": 0.21504,
+ "6": 0.22001,
+ "7": 0.2229,
+ "8": 0.21229,
+ "9": 0.21379,
+ "10": 0.21208,
+ "11": 0.2118,
+ "12": 0.21048,
+ "13": 0.2115,
+ "14": 0.21066,
+ "15": 0.21855,
+ "16": 0.21087,
+ "17": 0.20703,
+ "18": 0.20894,
+ "19": 0.20887,
+ "20": 0.2104,
+ "21": 0.21022,
+ "22": 0.21074,
+ "23": 0.20888,
+ "24": 0.20836,
+ "25": 0.21432,
+ "26": 0.21152,
+ "27": 0.21606,
+ "28": 0.20664,
+ "29": 0.20772,
+ "30": 0.20937,
+ "31": 0.21234,
+ "32": 0.20697,
+ "33": 0.20948,
+ "34": 0.20793,
+ "35": 0.20778,
+ "36": 0.20847,
+ "37": 0.23285,
+ "38": 0.20687,
+ "39": 0.21235,
+ "40": 0.20715,
+ "41": 0.20852,
+ "42": 0.20621,
+ "43": 0.20557,
+ "44": 0.20679,
+ "45": 0.20637,
+ "46": 0.20435,
+ "47": 0.20644,
+ "48": 0.21085,
+ "49": 0.20511,
+ "50": 0.21115,
+ "51": 0.25894,
+ "52": 0.20754,
+ "53": 0.21203,
+ "54": 0.20968,
+ "55": 0.20782,
+ "56": 0.20613,
+ "57": 0.20486,
+ "58": 0.21193,
+ "59": 0.20949,
+ "60": 0.20746,
+ "61": 0.20763,
+ "62": 0.20604,
+ "63": 0.20744,
+ "64": 0.21087,
+ "65": 0.20502,
+ "66": 0.20722,
+ "67": 0.20585,
+ "68": 0.2084,
+ "69": 0.20628,
+ "70": 0.20616,
+ "71": 0.21122,
+ "72": 0.20844,
+ "73": 0.60528,
+ "74": 0.22097,
+ "75": 0.22164,
+ "76": 0.20714,
+ "77": 0.20746,
+ "78": 0.2208,
+ "79": 0.2201,
+ "80": 0.22572,
+ "81": 0.21267,
+ "82": 0.21523,
+ "83": 0.26847,
+ "84": 0.23924,
+ "85": 0.22715,
+ "86": 0.23194,
+ "87": 0.20529,
+ "88": 0.2172,
+ "89": 0.25691,
+ "90": 0.23371,
+ "91": 0.22038,
+ "92": 0.22654,
+ "93": 0.23026,
+ "94": 0.23961,
+ "95": 0.24307,
+ "96": 0.2713,
+ "97": 0.25293,
+ "98": 0.24677,
+ "99": 0.23834,
+ "100": 0.24498
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances/model_config.yaml
index 8e267b178b4..159a7d1f4c2 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -60,4 +60,6 @@ MODEL_ARGS:
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
--num-distributed-optimizer-instances: 2
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_top2router/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_top2router/model_config.yaml
index 8525e285ac9..86a775807b8 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_top2router/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_top2router/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -58,4 +58,6 @@ MODEL_ARGS:
--bf16: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/golden_values_dev_dgx_gb200.json
index ce3d79128b1..610c26cc4c4 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/golden_values_dev_dgx_gb200.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.80509,
"2": 10.82386,
- "3": 10.80196,
- "4": 10.79424,
- "5": 10.8277,
- "6": 10.84005,
- "7": 10.8365,
- "8": 10.82828,
- "9": 10.83477,
- "10": 10.77496,
- "11": 10.85204,
- "12": 10.83903,
- "13": 10.85207,
- "14": 10.85914,
- "15": 10.81681,
- "16": 10.79456,
- "17": 10.77491,
- "18": 10.80399,
- "19": 10.79956,
- "20": 10.73801,
- "21": 10.72487,
- "22": 10.59177,
- "23": 10.73098,
- "24": 10.6406,
- "25": 10.59018,
- "26": 10.63555,
- "27": 10.66245,
- "28": 10.6472,
- "29": 10.64163,
- "30": 10.4518,
- "31": 10.22249,
- "32": 10.52995,
- "33": 10.51998,
- "34": 10.31247,
- "35": 10.34796,
- "36": 10.31677,
- "37": 10.42804,
- "38": 10.29194,
- "39": 10.46881,
- "40": 10.19257,
- "41": 10.23159,
- "42": 10.29766,
- "43": 9.97363,
- "44": 10.07169,
- "45": 9.97015,
- "46": 9.94713,
- "47": 10.23179,
- "48": 9.97593,
- "49": 9.67748,
- "50": 10.0144
+ "3": 10.80204,
+ "4": 10.79416,
+ "5": 10.82732,
+ "6": 10.84034,
+ "7": 10.83568,
+ "8": 10.82776,
+ "9": 10.83556,
+ "10": 10.77503,
+ "11": 10.85212,
+ "12": 10.83961,
+ "13": 10.85141,
+ "14": 10.85884,
+ "15": 10.8174,
+ "16": 10.79513,
+ "17": 10.77499,
+ "18": 10.80327,
+ "19": 10.79941,
+ "20": 10.73867,
+ "21": 10.72399,
+ "22": 10.59285,
+ "23": 10.73073,
+ "24": 10.6412,
+ "25": 10.59085,
+ "26": 10.63654,
+ "27": 10.6629,
+ "28": 10.64863,
+ "29": 10.64189,
+ "30": 10.45127,
+ "31": 10.22219,
+ "32": 10.53058,
+ "33": 10.51988,
+ "34": 10.31281,
+ "35": 10.34927,
+ "36": 10.31833,
+ "37": 10.4288,
+ "38": 10.29259,
+ "39": 10.46905,
+ "40": 10.19386,
+ "41": 10.23271,
+ "42": 10.29876,
+ "43": 9.97471,
+ "44": 10.07326,
+ "45": 9.97121,
+ "46": 9.94932,
+ "47": 10.23274,
+ "48": 9.97796,
+ "49": 9.68035,
+ "50": 10.01744
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 31182.0,
- "2": 33013.0,
- "3": 33646.0,
- "4": 32202.0,
- "5": 36913.0,
- "6": 37554.0,
- "7": 35184.0,
- "8": 32207.0,
- "9": 34523.0,
- "10": 29945.0,
- "11": 38237.0,
- "12": 35346.0,
- "13": 37426.0,
- "14": 38358.0,
- "15": 35140.0,
- "16": 36293.0,
- "17": 35645.0,
- "18": 35117.0,
- "19": 35648.0,
- "20": 32896.0,
- "21": 33511.0,
- "22": 30704.0,
- "23": 38149.0,
- "24": 32677.0,
- "25": 31055.0,
- "26": 34700.0,
- "27": 35410.0,
- "28": 37268.0,
- "29": 37953.0,
- "30": 33210.0,
- "31": 30482.0,
- "32": 36908.0,
- "33": 38308.0,
- "34": 33125.0,
- "35": 34341.0,
- "36": 34925.0,
- "37": 38767.0,
- "38": 35780.0,
- "39": 38955.0,
- "40": 36485.0,
- "41": 36015.0,
- "42": 37638.0,
- "43": 33689.0,
- "44": 33688.0,
- "45": 35448.0,
- "46": 36810.0,
- "47": 40858.0,
- "48": 35696.0,
- "49": 34729.0,
- "50": 39077.0
+ "1": 16036.0,
+ "2": 17259.0,
+ "3": 17383.0,
+ "4": 16712.0,
+ "5": 18911.0,
+ "6": 19262.0,
+ "7": 18424.0,
+ "8": 16744.0,
+ "9": 18034.0,
+ "10": 15535.0,
+ "11": 19649.0,
+ "12": 18428.0,
+ "13": 19131.0,
+ "14": 19554.0,
+ "15": 18227.0,
+ "16": 18803.0,
+ "17": 18303.0,
+ "18": 18020.0,
+ "19": 18525.0,
+ "20": 17077.0,
+ "21": 17416.0,
+ "22": 15952.0,
+ "23": 19619.0,
+ "24": 16834.0,
+ "25": 16141.0,
+ "26": 17765.0,
+ "27": 18357.0,
+ "28": 19158.0,
+ "29": 19677.0,
+ "30": 17184.0,
+ "31": 15618.0,
+ "32": 18706.0,
+ "33": 19402.0,
+ "34": 16909.0,
+ "35": 17781.0,
+ "36": 17861.0,
+ "37": 19639.0,
+ "38": 18567.0,
+ "39": 20265.0,
+ "40": 18782.0,
+ "41": 18470.0,
+ "42": 19484.0,
+ "43": 17350.0,
+ "44": 17404.0,
+ "45": 18330.0,
+ "46": 18907.0,
+ "47": 20800.0,
+ "48": 18138.0,
+ "49": 17902.0,
+ "50": 20085.0
}
},
"mem-allocated-bytes": {
@@ -120,53 +120,53 @@
"values": {
"1": 1027095040.0,
"2": 1027093504.0,
- "3": 1027094528.0,
- "4": 1027095040.0,
+ "3": 1027094016.0,
+ "4": 1027096064.0,
"5": 1027091968.0,
"6": 1027091968.0,
- "7": 1027098112.0,
+ "7": 1027098624.0,
"8": 1027097600.0,
- "9": 1027094528.0,
- "10": 1027094016.0,
- "11": 1027098624.0,
- "12": 1027094528.0,
+ "9": 1027094016.0,
+ "10": 1027092992.0,
+ "11": 1027098112.0,
+ "12": 1027093504.0,
"13": 1027092480.0,
"14": 1027095040.0,
- "15": 1027095040.0,
- "16": 1027091456.0,
- "17": 1027101184.0,
+ "15": 1027095552.0,
+ "16": 1027092480.0,
+ "17": 1027102208.0,
"18": 1027096064.0,
"19": 1027093504.0,
- "20": 1027093504.0,
- "21": 1027097088.0,
- "22": 1027100160.0,
- "23": 1027100160.0,
+ "20": 1027092992.0,
+ "21": 1027096576.0,
+ "22": 1027099136.0,
+ "23": 1027098624.0,
"24": 1027095552.0,
"25": 1027097088.0,
- "26": 1027098112.0,
+ "26": 1027097600.0,
"27": 1027091456.0,
- "28": 1027090944.0,
- "29": 1027091968.0,
+ "28": 1027090432.0,
+ "29": 1027092992.0,
"30": 1027099648.0,
- "31": 1027109888.0,
- "32": 1027095552.0,
+ "31": 1027109376.0,
+ "32": 1027096576.0,
"33": 1027090944.0,
- "34": 1027098112.0,
- "35": 1027103744.0,
- "36": 1027098112.0,
- "37": 1027092480.0,
- "38": 1027091456.0,
- "39": 1027095040.0,
+ "34": 1027099136.0,
+ "35": 1027103232.0,
+ "36": 1027097088.0,
+ "37": 1027091968.0,
+ "38": 1027091968.0,
+ "39": 1027095552.0,
"40": 1027095040.0,
"41": 1027100160.0,
"42": 1027091968.0,
"43": 1027098624.0,
- "44": 1027098624.0,
- "45": 1027096064.0,
+ "44": 1027098112.0,
+ "45": 1027095552.0,
"46": 1027104256.0,
- "47": 1027093504.0,
- "48": 1027101184.0,
- "49": 1027096064.0,
+ "47": 1027092992.0,
+ "48": 1027100672.0,
+ "49": 1027096576.0,
"50": 1027095552.0
}
},
@@ -177,54 +177,54 @@
"values": {
"1": 3059586560.0,
"2": 3299159040.0,
- "3": 3299482112.0,
- "4": 3302137344.0,
- "5": 3302137344.0,
- "6": 3302137344.0,
- "7": 3303535104.0,
- "8": 3303535104.0,
- "9": 3303535104.0,
- "10": 3303535104.0,
- "11": 3303535104.0,
- "12": 3303535104.0,
- "13": 3303535104.0,
- "14": 3303535104.0,
- "15": 3303535104.0,
- "16": 3303535104.0,
- "17": 3306910208.0,
- "18": 3306910208.0,
- "19": 3306910208.0,
- "20": 3306910208.0,
- "21": 3306910208.0,
- "22": 3306910208.0,
- "23": 3306910208.0,
- "24": 3306910208.0,
- "25": 3306910208.0,
- "26": 3306910208.0,
- "27": 3306910208.0,
- "28": 3306910208.0,
- "29": 3306910208.0,
- "30": 3306910208.0,
- "31": 3312495616.0,
- "32": 3312495616.0,
- "33": 3312495616.0,
- "34": 3312495616.0,
- "35": 3312495616.0,
- "36": 3312495616.0,
- "37": 3312495616.0,
- "38": 3312495616.0,
- "39": 3312495616.0,
- "40": 3312495616.0,
- "41": 3312495616.0,
- "42": 3312495616.0,
- "43": 3312495616.0,
- "44": 3312495616.0,
- "45": 3312495616.0,
- "46": 3312495616.0,
- "47": 3312495616.0,
- "48": 3312495616.0,
- "49": 3312495616.0,
- "50": 3312495616.0
+ "3": 3299516928.0,
+ "4": 3301408256.0,
+ "5": 3301408256.0,
+ "6": 3301408256.0,
+ "7": 3303757824.0,
+ "8": 3303757824.0,
+ "9": 3303757824.0,
+ "10": 3303757824.0,
+ "11": 3303757824.0,
+ "12": 3303757824.0,
+ "13": 3303757824.0,
+ "14": 3303757824.0,
+ "15": 3303757824.0,
+ "16": 3303757824.0,
+ "17": 3307320832.0,
+ "18": 3307320832.0,
+ "19": 3307320832.0,
+ "20": 3307320832.0,
+ "21": 3307320832.0,
+ "22": 3307320832.0,
+ "23": 3307320832.0,
+ "24": 3307320832.0,
+ "25": 3307320832.0,
+ "26": 3307320832.0,
+ "27": 3307320832.0,
+ "28": 3307320832.0,
+ "29": 3307320832.0,
+ "30": 3307320832.0,
+ "31": 3312982528.0,
+ "32": 3312982528.0,
+ "33": 3312982528.0,
+ "34": 3312982528.0,
+ "35": 3312982528.0,
+ "36": 3312982528.0,
+ "37": 3312982528.0,
+ "38": 3312982528.0,
+ "39": 3312982528.0,
+ "40": 3312982528.0,
+ "41": 3312982528.0,
+ "42": 3312982528.0,
+ "43": 3312982528.0,
+ "44": 3312982528.0,
+ "45": 3312982528.0,
+ "46": 3312982528.0,
+ "47": 3312982528.0,
+ "48": 3312982528.0,
+ "49": 3312982528.0,
+ "50": 3312982528.0
}
},
"iteration-time": {
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 3.82645,
- "3": 0.34371,
- "4": 0.32704,
- "5": 0.31536,
- "6": 0.32001,
- "7": 0.31919,
- "8": 0.31719,
- "9": 0.31876,
- "10": 0.31015,
- "11": 0.31546,
- "12": 0.31198,
- "13": 0.31518,
- "14": 0.40567,
- "15": 0.31856,
- "16": 0.30868,
- "17": 0.31352,
- "18": 0.31536,
- "19": 0.31164,
- "20": 0.31286,
- "21": 0.35519,
- "22": 0.30985,
- "23": 0.31256,
- "24": 0.31727,
- "25": 0.36651,
- "26": 0.47287,
- "27": 0.57438,
- "28": 0.3575,
- "29": 0.71431,
- "30": 0.31163,
- "31": 0.31877,
- "32": 0.34436,
- "33": 0.51773,
- "34": 0.32292,
- "35": 0.31651,
- "36": 0.34162,
- "37": 0.31339,
- "38": 0.30524,
- "39": 0.63856,
- "40": 0.31883,
- "41": 0.31475,
- "42": 0.67365,
- "43": 0.33393,
- "44": 0.31389,
- "45": 0.65089,
- "46": 0.6524,
- "47": 0.3061,
- "48": 0.30487,
- "49": 0.3295,
- "50": 0.30784
+ "2": 6.83933,
+ "3": 0.3754,
+ "4": 0.35757,
+ "5": 0.44104,
+ "6": 0.41416,
+ "7": 0.43241,
+ "8": 0.38155,
+ "9": 0.34412,
+ "10": 0.34756,
+ "11": 0.3526,
+ "12": 0.34995,
+ "13": 0.34931,
+ "14": 0.34894,
+ "15": 0.34862,
+ "16": 0.35437,
+ "17": 0.36857,
+ "18": 0.3423,
+ "19": 0.3459,
+ "20": 0.34408,
+ "21": 0.34635,
+ "22": 0.3519,
+ "23": 0.34886,
+ "24": 0.34974,
+ "25": 0.34628,
+ "26": 0.34297,
+ "27": 0.35262,
+ "28": 0.34712,
+ "29": 0.3472,
+ "30": 0.34239,
+ "31": 0.35872,
+ "32": 0.34375,
+ "33": 0.34575,
+ "34": 0.35501,
+ "35": 0.34925,
+ "36": 0.34767,
+ "37": 0.34895,
+ "38": 0.34493,
+ "39": 0.34914,
+ "40": 0.35158,
+ "41": 0.34669,
+ "42": 0.349,
+ "43": 0.35455,
+ "44": 0.35246,
+ "45": 0.34837,
+ "46": 0.34695,
+ "47": 0.35156,
+ "48": 0.35298,
+ "49": 0.35814,
+ "50": 0.34181
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/golden_values_dev_dgx_h100.json
index 38498d3139b..0d556450cec 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.80815,
- "2": 10.82612,
- "3": 10.83032,
- "4": 10.80963,
- "5": 10.84127,
- "6": 10.8581,
- "7": 10.81967,
- "8": 10.82506,
- "9": 10.83749,
- "10": 10.7783,
- "11": 10.85781,
- "12": 10.85539,
- "13": 10.85233,
- "14": 10.86699,
- "15": 10.81253,
- "16": 10.80292,
- "17": 10.78098,
- "18": 10.80788,
- "19": 10.79276,
- "20": 10.74548,
- "21": 10.72785,
- "22": 10.59608,
- "23": 10.73999,
- "24": 10.63509,
- "25": 10.59832,
- "26": 10.63517,
- "27": 10.65744,
- "28": 10.64536,
- "29": 10.65122,
- "30": 10.44144,
- "31": 10.21465,
- "32": 10.53342,
- "33": 10.52518,
- "34": 10.30171,
- "35": 10.34871,
- "36": 10.30843,
- "37": 10.42353,
- "38": 10.28859,
- "39": 10.45514,
- "40": 10.19363,
- "41": 10.22791,
- "42": 10.29725,
- "43": 9.95871,
- "44": 10.06717,
- "45": 9.95955,
- "46": 9.92614,
- "47": 10.20607,
- "48": 9.96021,
- "49": 9.65854,
- "50": 10.01296
+ "1": 10.92671,
+ "2": 10.91589,
+ "3": 10.92552,
+ "4": 10.93168,
+ "5": 10.93015,
+ "6": 10.9259,
+ "7": 10.92646,
+ "8": 10.92323,
+ "9": 10.92778,
+ "10": 10.9168,
+ "11": 10.9178,
+ "12": 10.92446,
+ "13": 10.90961,
+ "14": 10.90627,
+ "15": 10.90112,
+ "16": 10.88691,
+ "17": 10.88827,
+ "18": 10.88554,
+ "19": 10.88654,
+ "20": 10.8377,
+ "21": 10.82717,
+ "22": 10.81535,
+ "23": 10.80831,
+ "24": 10.78061,
+ "25": 10.77774,
+ "26": 10.76115,
+ "27": 10.7495,
+ "28": 10.6922,
+ "29": 10.66686,
+ "30": 10.63118,
+ "31": 10.62182,
+ "32": 10.61591,
+ "33": 10.57843,
+ "34": 10.54531,
+ "35": 10.54625,
+ "36": 10.53479,
+ "37": 10.50533,
+ "38": 10.50383,
+ "39": 10.47322,
+ "40": 10.45095,
+ "41": 10.42606,
+ "42": 10.41475,
+ "43": 10.40064,
+ "44": 10.37006,
+ "45": 10.38168,
+ "46": 10.33484,
+ "47": 10.32444,
+ "48": 10.28749,
+ "49": 10.28608,
+ "50": 10.27697
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 31590.0,
- "2": 32940.0,
- "3": 33668.0,
- "4": 31186.0,
- "5": 36214.0,
- "6": 37169.0,
- "7": 34770.0,
- "8": 31862.0,
- "9": 34102.0,
- "10": 30394.0,
- "11": 38432.0,
- "12": 35039.0,
- "13": 37236.0,
- "14": 37668.0,
- "15": 34199.0,
- "16": 36659.0,
- "17": 34831.0,
- "18": 35011.0,
- "19": 35486.0,
- "20": 33221.0,
- "21": 33971.0,
- "22": 30501.0,
- "23": 38411.0,
- "24": 32764.0,
- "25": 31363.0,
- "26": 34624.0,
- "27": 36096.0,
- "28": 37021.0,
- "29": 37900.0,
- "30": 33066.0,
- "31": 29871.0,
- "32": 36113.0,
- "33": 38168.0,
- "34": 33074.0,
- "35": 34300.0,
- "36": 35363.0,
- "37": 38150.0,
- "38": 35798.0,
- "39": 38945.0,
- "40": 35780.0,
- "41": 35999.0,
- "42": 36611.0,
- "43": 33781.0,
- "44": 34207.0,
- "45": 35198.0,
- "46": 36779.0,
- "47": 40585.0,
- "48": 36434.0,
- "49": 35787.0,
- "50": 38996.0
+ "1": 19058.0,
+ "2": 19206.0,
+ "3": 18972.0,
+ "4": 19416.0,
+ "5": 19009.0,
+ "6": 18538.0,
+ "7": 18981.0,
+ "8": 18448.0,
+ "9": 18864.0,
+ "10": 19655.0,
+ "11": 19064.0,
+ "12": 18696.0,
+ "13": 19292.0,
+ "14": 19140.0,
+ "15": 18806.0,
+ "16": 18590.0,
+ "17": 18993.0,
+ "18": 19173.0,
+ "19": 19321.0,
+ "20": 19057.0,
+ "21": 19086.0,
+ "22": 18997.0,
+ "23": 18891.0,
+ "24": 19267.0,
+ "25": 18711.0,
+ "26": 19139.0,
+ "27": 19114.0,
+ "28": 18818.0,
+ "29": 18371.0,
+ "30": 18304.0,
+ "31": 19016.0,
+ "32": 19184.0,
+ "33": 18481.0,
+ "34": 18592.0,
+ "35": 18848.0,
+ "36": 18346.0,
+ "37": 18564.0,
+ "38": 18516.0,
+ "39": 18959.0,
+ "40": 19194.0,
+ "41": 18945.0,
+ "42": 18455.0,
+ "43": 19053.0,
+ "44": 18809.0,
+ "45": 20372.0,
+ "46": 19563.0,
+ "47": 19974.0,
+ "48": 20047.0,
+ "49": 21674.0,
+ "50": 20259.0
}
},
"mem-allocated-bytes": {
@@ -118,55 +118,55 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1027090944.0,
- "2": 1027090944.0,
- "3": 1027091968.0,
- "4": 1027091456.0,
- "5": 1027091968.0,
- "6": 1027090432.0,
- "7": 1027086848.0,
- "8": 1027087872.0,
- "9": 1027094528.0,
- "10": 1027088896.0,
- "11": 1027089408.0,
- "12": 1027089408.0,
+ "1": 1027089408.0,
+ "2": 1027091968.0,
+ "3": 1027088384.0,
+ "4": 1027088384.0,
+ "5": 1027090944.0,
+ "6": 1027091968.0,
+ "7": 1027088896.0,
+ "8": 1027092992.0,
+ "9": 1027090944.0,
+ "10": 1027090432.0,
+ "11": 1027090944.0,
+ "12": 1027091456.0,
"13": 1027091968.0,
- "14": 1027088896.0,
- "15": 1027090432.0,
- "16": 1027091456.0,
- "17": 1027089408.0,
- "18": 1027093504.0,
- "19": 1027091968.0,
- "20": 1027095040.0,
- "21": 1027088384.0,
- "22": 1027091456.0,
- "23": 1027091968.0,
- "24": 1027090944.0,
- "25": 1027089408.0,
- "26": 1027090432.0,
- "27": 1027090432.0,
- "28": 1027087872.0,
- "29": 1027088896.0,
- "30": 1027087360.0,
- "31": 1027079168.0,
- "32": 1027082240.0,
- "33": 1027091456.0,
- "34": 1027088384.0,
- "35": 1027090432.0,
- "36": 1027088896.0,
- "37": 1027089408.0,
+ "14": 1027090944.0,
+ "15": 1027091456.0,
+ "16": 1027088384.0,
+ "17": 1027088896.0,
+ "18": 1027090944.0,
+ "19": 1027089920.0,
+ "20": 1027089920.0,
+ "21": 1027092480.0,
+ "22": 1027088896.0,
+ "23": 1027094528.0,
+ "24": 1027091968.0,
+ "25": 1027091456.0,
+ "26": 1027089408.0,
+ "27": 1027087872.0,
+ "28": 1027091456.0,
+ "29": 1027090432.0,
+ "30": 1027089920.0,
+ "31": 1027089408.0,
+ "32": 1027094528.0,
+ "33": 1027094016.0,
+ "34": 1027092480.0,
+ "35": 1027086848.0,
+ "36": 1027088384.0,
+ "37": 1027088896.0,
"38": 1027090432.0,
- "39": 1027085824.0,
- "40": 1027086848.0,
- "41": 1027088384.0,
- "42": 1027091968.0,
- "43": 1027084800.0,
- "44": 1027087872.0,
- "45": 1027087872.0,
- "46": 1027078656.0,
- "47": 1027087872.0,
- "48": 1027086336.0,
- "49": 1027082240.0,
+ "39": 1027090432.0,
+ "40": 1027091456.0,
+ "41": 1027088896.0,
+ "42": 1027088384.0,
+ "43": 1027088896.0,
+ "44": 1027091968.0,
+ "45": 1027091456.0,
+ "46": 1027085824.0,
+ "47": 1027089920.0,
+ "48": 1027088384.0,
+ "49": 1027086848.0,
"50": 1027089920.0
}
},
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 3057868288.0,
- "2": 3298335232.0,
- "3": 3298335232.0,
- "4": 3300084224.0,
- "5": 3300084224.0,
- "6": 3300084224.0,
- "7": 3300084224.0,
- "8": 3300084224.0,
- "9": 3300084224.0,
- "10": 3300122624.0,
- "11": 3300122624.0,
- "12": 3300122624.0,
- "13": 3300122624.0,
- "14": 3300122624.0,
- "15": 3300122624.0,
- "16": 3300122624.0,
- "17": 3300122624.0,
- "18": 3300122624.0,
- "19": 3300376576.0,
- "20": 3300416000.0,
- "21": 3300416000.0,
- "22": 3301032960.0,
- "23": 3301998080.0,
- "24": 3301998080.0,
- "25": 3301998080.0,
- "26": 3301998080.0,
- "27": 3301998080.0,
- "28": 3301998080.0,
- "29": 3301998080.0,
- "30": 3301998080.0,
- "31": 3301998080.0,
- "32": 3301998080.0,
- "33": 3301998080.0,
- "34": 3301998080.0,
- "35": 3301998080.0,
- "36": 3301998080.0,
- "37": 3301998080.0,
- "38": 3301998080.0,
- "39": 3301998080.0,
- "40": 3301998080.0,
- "41": 3301998080.0,
- "42": 3301998080.0,
- "43": 3301998080.0,
- "44": 3301998080.0,
- "45": 3301998080.0,
- "46": 3301998080.0,
- "47": 3301998080.0,
- "48": 3301998080.0,
- "49": 3301998080.0,
- "50": 3301998080.0
+ "1": 3058382848.0,
+ "2": 3298735616.0,
+ "3": 3298735616.0,
+ "4": 3298735616.0,
+ "5": 3298735616.0,
+ "6": 3298735616.0,
+ "7": 3298735616.0,
+ "8": 3299022336.0,
+ "9": 3299022336.0,
+ "10": 3299022336.0,
+ "11": 3299138048.0,
+ "12": 3299138048.0,
+ "13": 3299225088.0,
+ "14": 3299394048.0,
+ "15": 3299394048.0,
+ "16": 3299394048.0,
+ "17": 3299394048.0,
+ "18": 3299394048.0,
+ "19": 3299809792.0,
+ "20": 3299809792.0,
+ "21": 3299809792.0,
+ "22": 3299809792.0,
+ "23": 3300119552.0,
+ "24": 3300119552.0,
+ "25": 3300119552.0,
+ "26": 3300119552.0,
+ "27": 3300119552.0,
+ "28": 3300119552.0,
+ "29": 3300119552.0,
+ "30": 3300119552.0,
+ "31": 3300119552.0,
+ "32": 3300119552.0,
+ "33": 3300119552.0,
+ "34": 3300416512.0,
+ "35": 3300416512.0,
+ "36": 3300416512.0,
+ "37": 3300416512.0,
+ "38": 3300416512.0,
+ "39": 3300416512.0,
+ "40": 3300416512.0,
+ "41": 3300416512.0,
+ "42": 3300416512.0,
+ "43": 3300416512.0,
+ "44": 3300416512.0,
+ "45": 3300416512.0,
+ "46": 3300416512.0,
+ "47": 3300416512.0,
+ "48": 3300416512.0,
+ "49": 3300416512.0,
+ "50": 3300416512.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 16.45405,
- "2": 0.30024,
- "3": 0.24416,
- "4": 0.22949,
- "5": 0.21642,
- "6": 0.20677,
- "7": 0.21591,
- "8": 0.21087,
- "9": 0.20973,
- "10": 0.20724,
- "11": 0.20594,
- "12": 0.20225,
- "13": 0.21091,
- "14": 0.2028,
- "15": 0.22641,
- "16": 0.20409,
- "17": 0.21141,
- "18": 0.20363,
- "19": 0.20701,
- "20": 0.2078,
- "21": 0.20171,
- "22": 0.20432,
- "23": 0.19941,
- "24": 0.20413,
- "25": 0.20204,
- "26": 0.20188,
- "27": 0.60524,
- "28": 0.21001,
- "29": 0.20338,
- "30": 0.20253,
- "31": 0.2399,
- "32": 0.19914,
- "33": 0.20122,
- "34": 0.22929,
- "35": 0.20106,
- "36": 0.22225,
- "37": 0.20411,
- "38": 0.20267,
- "39": 0.19726,
- "40": 0.21398,
- "41": 0.21317,
- "42": 0.20362,
- "43": 0.20696,
- "44": 0.20834,
- "45": 0.21563,
- "46": 0.22195,
- "47": 0.20394,
- "48": 0.22663,
- "49": 0.24701,
- "50": 0.20255
+ "1": "nan",
+ "2": 7.89786,
+ "3": 0.24594,
+ "4": 0.22051,
+ "5": 0.222,
+ "6": 0.21926,
+ "7": 0.2078,
+ "8": 0.21045,
+ "9": 0.20823,
+ "10": 0.20524,
+ "11": 0.21966,
+ "12": 0.20488,
+ "13": 0.2063,
+ "14": 0.21021,
+ "15": 0.20599,
+ "16": 0.20609,
+ "17": 0.2069,
+ "18": 0.20307,
+ "19": 0.20538,
+ "20": 0.20251,
+ "21": 0.21207,
+ "22": 0.20145,
+ "23": 0.21042,
+ "24": 0.21155,
+ "25": 0.2081,
+ "26": 0.20377,
+ "27": 0.20504,
+ "28": 0.20365,
+ "29": 0.20313,
+ "30": 0.20266,
+ "31": 0.20257,
+ "32": 0.20336,
+ "33": 0.20201,
+ "34": 0.20295,
+ "35": 0.20399,
+ "36": 0.20262,
+ "37": 0.2058,
+ "38": 0.20263,
+ "39": 0.20527,
+ "40": 0.20348,
+ "41": 0.20601,
+ "42": 0.20448,
+ "43": 0.20532,
+ "44": 0.20505,
+ "45": 0.20908,
+ "46": 0.2037,
+ "47": 0.21773,
+ "48": 0.20684,
+ "49": 0.20825,
+ "50": 0.20302
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/golden_values_lts_dgx_a100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/golden_values_lts_dgx_a100.json
index 512f1302b5f..c23a8dad5e4 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/golden_values_lts_dgx_a100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/golden_values_lts_dgx_a100.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.81189,
"2": 10.82837,
- "3": 10.81793,
- "4": 10.80617,
- "5": 10.84927,
- "6": 10.86922,
- "7": 10.84362,
- "8": 10.83419,
- "9": 10.84988,
- "10": 10.78408,
- "11": 10.88038,
- "12": 10.85362,
- "13": 10.86619,
- "14": 10.88223,
- "15": 10.82311,
- "16": 10.81052,
- "17": 10.78202,
- "18": 10.81541,
- "19": 10.81756,
- "20": 10.74584,
- "21": 10.72627,
- "22": 10.60022,
- "23": 10.74769,
- "24": 10.6365,
- "25": 10.59453,
- "26": 10.64799,
- "27": 10.66378,
- "28": 10.64562,
- "29": 10.65242,
- "30": 10.44192,
- "31": 10.21776,
- "32": 10.52786,
- "33": 10.52222,
- "34": 10.30567,
- "35": 10.35142,
- "36": 10.32502,
- "37": 10.42585,
- "38": 10.28522,
- "39": 10.47826,
- "40": 10.19414,
- "41": 10.22953,
- "42": 10.29398,
- "43": 9.95703,
- "44": 10.06609,
- "45": 9.95939,
- "46": 9.92711,
- "47": 10.22774,
- "48": 9.96376,
- "49": 9.65724,
- "50": 10.01644
+ "3": 10.81783,
+ "4": 10.80693,
+ "5": 10.85009,
+ "6": 10.86823,
+ "7": 10.8436,
+ "8": 10.83354,
+ "9": 10.8502,
+ "10": 10.78331,
+ "11": 10.8802,
+ "12": 10.85416,
+ "13": 10.86646,
+ "14": 10.88171,
+ "15": 10.82338,
+ "16": 10.80927,
+ "17": 10.78248,
+ "18": 10.81556,
+ "19": 10.81688,
+ "20": 10.74651,
+ "21": 10.72706,
+ "22": 10.60025,
+ "23": 10.74772,
+ "24": 10.63719,
+ "25": 10.59438,
+ "26": 10.64777,
+ "27": 10.66391,
+ "28": 10.64625,
+ "29": 10.65246,
+ "30": 10.44299,
+ "31": 10.21802,
+ "32": 10.52776,
+ "33": 10.52265,
+ "34": 10.30634,
+ "35": 10.35167,
+ "36": 10.32517,
+ "37": 10.42595,
+ "38": 10.28615,
+ "39": 10.4784,
+ "40": 10.19582,
+ "41": 10.23065,
+ "42": 10.295,
+ "43": 9.95805,
+ "44": 10.06748,
+ "45": 9.96175,
+ "46": 9.92925,
+ "47": 10.22896,
+ "48": 9.96557,
+ "49": 9.65931,
+ "50": 10.01873
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 31381.0,
- "2": 33910.0,
- "3": 33570.0,
- "4": 31384.0,
- "5": 35903.0,
- "6": 37599.0,
- "7": 35525.0,
- "8": 31635.0,
- "9": 34509.0,
- "10": 30410.0,
- "11": 37757.0,
- "12": 35967.0,
- "13": 36886.0,
- "14": 38227.0,
- "15": 35788.0,
- "16": 36247.0,
- "17": 35378.0,
- "18": 35284.0,
- "19": 36213.0,
- "20": 33124.0,
- "21": 33424.0,
- "22": 31395.0,
- "23": 38414.0,
- "24": 32277.0,
- "25": 31634.0,
- "26": 35264.0,
- "27": 35935.0,
- "28": 37638.0,
- "29": 38145.0,
- "30": 33261.0,
- "31": 30972.0,
- "32": 36646.0,
- "33": 38318.0,
- "34": 33243.0,
- "35": 34813.0,
- "36": 35446.0,
- "37": 38278.0,
- "38": 35590.0,
- "39": 38681.0,
- "40": 36027.0,
- "41": 36509.0,
- "42": 37301.0,
- "43": 34597.0,
- "44": 34024.0,
- "45": 35938.0,
- "46": 37229.0,
- "47": 40862.0,
- "48": 35537.0,
- "49": 35095.0,
- "50": 39711.0
+ "1": 16326.0,
+ "2": 17442.0,
+ "3": 17806.0,
+ "4": 16293.0,
+ "5": 18637.0,
+ "6": 19380.0,
+ "7": 18359.0,
+ "8": 16562.0,
+ "9": 18232.0,
+ "10": 15931.0,
+ "11": 19907.0,
+ "12": 18526.0,
+ "13": 18903.0,
+ "14": 19504.0,
+ "15": 18657.0,
+ "16": 18757.0,
+ "17": 18379.0,
+ "18": 18182.0,
+ "19": 18941.0,
+ "20": 17178.0,
+ "21": 17492.0,
+ "22": 16165.0,
+ "23": 19782.0,
+ "24": 16643.0,
+ "25": 16211.0,
+ "26": 18084.0,
+ "27": 18542.0,
+ "28": 19648.0,
+ "29": 19321.0,
+ "30": 17124.0,
+ "31": 15926.0,
+ "32": 19151.0,
+ "33": 19418.0,
+ "34": 17302.0,
+ "35": 18108.0,
+ "36": 18306.0,
+ "37": 19706.0,
+ "38": 18570.0,
+ "39": 19948.0,
+ "40": 18854.0,
+ "41": 18733.0,
+ "42": 19354.0,
+ "43": 17827.0,
+ "44": 17748.0,
+ "45": 18541.0,
+ "46": 19035.0,
+ "47": 20764.0,
+ "48": 18583.0,
+ "49": 18143.0,
+ "50": 20370.0
}
},
"mem-allocated-bytes": {
@@ -120,54 +120,54 @@
"values": {
"1": 880441344.0,
"2": 880437760.0,
- "3": 880442368.0,
- "4": 880448512.0,
- "5": 880440832.0,
+ "3": 880442880.0,
+ "4": 880449536.0,
+ "5": 880440320.0,
"6": 880441344.0,
- "7": 880445440.0,
+ "7": 880445952.0,
"8": 880442880.0,
- "9": 880440320.0,
+ "9": 880440832.0,
"10": 880443392.0,
"11": 880442368.0,
"12": 880439808.0,
- "13": 880439808.0,
- "14": 880443904.0,
+ "13": 880440320.0,
+ "14": 880444928.0,
"15": 880443904.0,
- "16": 880441856.0,
- "17": 880448512.0,
- "18": 880437760.0,
- "19": 880440320.0,
- "20": 880442880.0,
- "21": 880445440.0,
- "22": 880450048.0,
- "23": 880449024.0,
+ "16": 880440832.0,
+ "17": 880448000.0,
+ "18": 880439296.0,
+ "19": 880438272.0,
+ "20": 880443392.0,
+ "21": 880445952.0,
+ "22": 880449024.0,
+ "23": 880448000.0,
"24": 880440320.0,
"25": 880440320.0,
- "26": 880445952.0,
- "27": 880441344.0,
- "28": 880439808.0,
- "29": 880443392.0,
- "30": 880441344.0,
+ "26": 880446464.0,
+ "27": 880439296.0,
+ "28": 880442880.0,
+ "29": 880442880.0,
+ "30": 880440320.0,
"31": 880454656.0,
"32": 880444928.0,
- "33": 880440320.0,
+ "33": 880439296.0,
"34": 880440832.0,
- "35": 880446976.0,
- "36": 880441344.0,
+ "35": 880446464.0,
+ "36": 880440832.0,
"37": 880442880.0,
"38": 880439808.0,
- "39": 880441856.0,
+ "39": 880441344.0,
"40": 880442880.0,
- "41": 880445952.0,
- "42": 880441344.0,
- "43": 880444416.0,
+ "41": 880446464.0,
+ "42": 880443392.0,
+ "43": 880443904.0,
"44": 880439808.0,
"45": 880440832.0,
"46": 880447488.0,
- "47": 880440320.0,
- "48": 880441856.0,
- "49": 880444416.0,
- "50": 880442368.0
+ "47": 880439808.0,
+ "48": 880441344.0,
+ "49": 880443904.0,
+ "50": 880441344.0
}
},
"mem-max-allocated-bytes": {
@@ -177,54 +177,54 @@
"values": {
"1": 2910130176.0,
"2": 3151821824.0,
- "3": 3152806912.0,
- "4": 3156619264.0,
- "5": 3156619264.0,
- "6": 3156619264.0,
- "7": 3156619264.0,
- "8": 3156619264.0,
- "9": 3156619264.0,
- "10": 3156619264.0,
- "11": 3156619264.0,
- "12": 3156619264.0,
- "13": 3156619264.0,
- "14": 3156619264.0,
- "15": 3156619264.0,
- "16": 3156619264.0,
- "17": 3156619264.0,
- "18": 3156619264.0,
- "19": 3156619264.0,
- "20": 3156619264.0,
- "21": 3156619264.0,
- "22": 3156619264.0,
- "23": 3156619264.0,
- "24": 3156619264.0,
- "25": 3156619264.0,
- "26": 3156619264.0,
- "27": 3156619264.0,
- "28": 3156619264.0,
- "29": 3156619264.0,
- "30": 3156619264.0,
- "31": 3167942656.0,
- "32": 3167942656.0,
- "33": 3167942656.0,
- "34": 3167942656.0,
- "35": 3167942656.0,
- "36": 3167942656.0,
- "37": 3167942656.0,
- "38": 3167942656.0,
- "39": 3167942656.0,
- "40": 3167942656.0,
- "41": 3167942656.0,
- "42": 3167942656.0,
- "43": 3167942656.0,
- "44": 3167942656.0,
- "45": 3167942656.0,
- "46": 3167942656.0,
- "47": 3167942656.0,
- "48": 3167942656.0,
- "49": 3167942656.0,
- "50": 3167942656.0
+ "3": 3152922112.0,
+ "4": 3156526080.0,
+ "5": 3156526080.0,
+ "6": 3156526080.0,
+ "7": 3156526080.0,
+ "8": 3156526080.0,
+ "9": 3156526080.0,
+ "10": 3156526080.0,
+ "11": 3156526080.0,
+ "12": 3156526080.0,
+ "13": 3156526080.0,
+ "14": 3156526080.0,
+ "15": 3156526080.0,
+ "16": 3156526080.0,
+ "17": 3156526080.0,
+ "18": 3156526080.0,
+ "19": 3156526080.0,
+ "20": 3156526080.0,
+ "21": 3156526080.0,
+ "22": 3156526080.0,
+ "23": 3156526080.0,
+ "24": 3156526080.0,
+ "25": 3156526080.0,
+ "26": 3156526080.0,
+ "27": 3156526080.0,
+ "28": 3156526080.0,
+ "29": 3156526080.0,
+ "30": 3156526080.0,
+ "31": 3167936512.0,
+ "32": 3167936512.0,
+ "33": 3167936512.0,
+ "34": 3167936512.0,
+ "35": 3167936512.0,
+ "36": 3167936512.0,
+ "37": 3167936512.0,
+ "38": 3167936512.0,
+ "39": 3167936512.0,
+ "40": 3167936512.0,
+ "41": 3167936512.0,
+ "42": 3167936512.0,
+ "43": 3167936512.0,
+ "44": 3167936512.0,
+ "45": 3167936512.0,
+ "46": 3167936512.0,
+ "47": 3167936512.0,
+ "48": 3167936512.0,
+ "49": 3167936512.0,
+ "50": 3167936512.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 6.10504,
- "2": 0.31901,
- "3": 0.30905,
- "4": 0.29474,
- "5": 0.29396,
- "6": 0.29282,
- "7": 0.29057,
- "8": 0.2914,
- "9": 0.29228,
- "10": 0.29365,
- "11": 0.29209,
- "12": 0.28885,
- "13": 0.28831,
- "14": 0.28848,
- "15": 0.29001,
- "16": 0.28893,
- "17": 0.28956,
- "18": 0.28887,
- "19": 0.28776,
- "20": 0.28952,
- "21": 0.6384,
- "22": 0.29529,
- "23": 0.29475,
- "24": 0.29441,
- "25": 0.29534,
- "26": 0.29435,
- "27": 0.29559,
- "28": 0.30134,
- "29": 0.2903,
- "30": 0.28843,
- "31": 0.28861,
- "32": 0.28817,
- "33": 0.29466,
- "34": 0.28874,
- "35": 0.28729,
- "36": 0.28824,
- "37": 0.28808,
- "38": 0.28729,
- "39": 0.28702,
- "40": 0.28605,
- "41": 0.28667,
- "42": 0.2877,
- "43": 0.28836,
- "44": 0.28722,
- "45": 0.28782,
- "46": 0.28798,
- "47": 0.28716,
- "48": 0.28759,
- "49": 0.28891,
- "50": 0.28753
+ "1": "nan",
+ "2": 3.38544,
+ "3": 0.34545,
+ "4": 0.29197,
+ "5": 0.28597,
+ "6": 0.28783,
+ "7": 0.28552,
+ "8": 0.28482,
+ "9": 0.28396,
+ "10": 0.28455,
+ "11": 0.28487,
+ "12": 0.28617,
+ "13": 0.28525,
+ "14": 0.28481,
+ "15": 0.2853,
+ "16": 0.28346,
+ "17": 0.28428,
+ "18": 0.28442,
+ "19": 0.28371,
+ "20": 0.28529,
+ "21": 0.28344,
+ "22": 0.2846,
+ "23": 0.28408,
+ "24": 0.2854,
+ "25": 0.28305,
+ "26": 0.28423,
+ "27": 0.28472,
+ "28": 0.28298,
+ "29": 0.28476,
+ "30": 0.28697,
+ "31": 0.28851,
+ "32": 0.28507,
+ "33": 0.30475,
+ "34": 0.28128,
+ "35": 0.28197,
+ "36": 0.28131,
+ "37": 0.28084,
+ "38": 0.28126,
+ "39": 0.28089,
+ "40": 0.28278,
+ "41": 0.28323,
+ "42": 0.28189,
+ "43": 0.282,
+ "44": 0.2825,
+ "45": 0.28252,
+ "46": 0.28209,
+ "47": 0.28083,
+ "48": 0.28224,
+ "49": 0.28393,
+ "50": 0.28207
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/model_config.yaml
index b84bf45b890..8ced6e37a52 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -61,4 +61,6 @@ MODEL_ARGS:
--attention-backend: unfused
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_dist_optimizer/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_dist_optimizer/model_config.yaml
index b5c774d4d3c..ec59433dc1b 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_dist_optimizer/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_dist_optimizer/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -58,4 +58,6 @@ MODEL_ARGS:
--disable-bias-linear: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM/golden_values_dev_dgx_gb200.json
index f62929eef31..f0ac8fb5e01 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM/golden_values_dev_dgx_gb200.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.80509,
"2": 10.82386,
- "3": 10.80196,
- "4": 10.79424,
- "5": 10.8277,
- "6": 10.84005,
- "7": 10.8365,
- "8": 10.82828,
- "9": 10.83477,
- "10": 10.77496,
- "11": 10.85204,
- "12": 10.83903,
- "13": 10.85207,
- "14": 10.85914,
- "15": 10.81681,
- "16": 10.79456,
- "17": 10.77491,
- "18": 10.80399,
- "19": 10.79956,
- "20": 10.73801,
- "21": 10.72487,
- "22": 10.59177,
- "23": 10.73098,
- "24": 10.6406,
- "25": 10.59018,
- "26": 10.63555,
- "27": 10.66245,
- "28": 10.6472,
- "29": 10.64163,
- "30": 10.4518,
- "31": 10.22249,
- "32": 10.52995,
- "33": 10.51998,
- "34": 10.31247,
- "35": 10.34796,
- "36": 10.31677,
- "37": 10.42804,
- "38": 10.29194,
- "39": 10.46881,
- "40": 10.19257,
- "41": 10.23159,
- "42": 10.29766,
- "43": 9.97363,
- "44": 10.07169,
- "45": 9.97015,
- "46": 9.94713,
- "47": 10.23179,
- "48": 9.97593,
- "49": 9.67748,
- "50": 10.0144
+ "3": 10.80204,
+ "4": 10.79416,
+ "5": 10.82732,
+ "6": 10.84034,
+ "7": 10.83568,
+ "8": 10.82776,
+ "9": 10.83556,
+ "10": 10.77503,
+ "11": 10.85212,
+ "12": 10.83961,
+ "13": 10.85141,
+ "14": 10.85884,
+ "15": 10.8174,
+ "16": 10.79513,
+ "17": 10.77499,
+ "18": 10.80327,
+ "19": 10.79941,
+ "20": 10.73867,
+ "21": 10.72399,
+ "22": 10.59285,
+ "23": 10.73073,
+ "24": 10.6412,
+ "25": 10.59085,
+ "26": 10.63654,
+ "27": 10.6629,
+ "28": 10.64863,
+ "29": 10.64189,
+ "30": 10.45127,
+ "31": 10.22219,
+ "32": 10.53058,
+ "33": 10.51988,
+ "34": 10.31281,
+ "35": 10.34927,
+ "36": 10.31833,
+ "37": 10.4288,
+ "38": 10.29259,
+ "39": 10.46905,
+ "40": 10.19386,
+ "41": 10.23271,
+ "42": 10.29876,
+ "43": 9.97471,
+ "44": 10.07326,
+ "45": 9.97121,
+ "46": 9.94932,
+ "47": 10.23274,
+ "48": 9.97796,
+ "49": 9.68035,
+ "50": 10.01744
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 31182.0,
- "2": 33013.0,
- "3": 33646.0,
- "4": 32202.0,
- "5": 36913.0,
- "6": 37554.0,
- "7": 35184.0,
- "8": 32207.0,
- "9": 34523.0,
- "10": 29945.0,
- "11": 38237.0,
- "12": 35346.0,
- "13": 37426.0,
- "14": 38358.0,
- "15": 35140.0,
- "16": 36293.0,
- "17": 35645.0,
- "18": 35117.0,
- "19": 35648.0,
- "20": 32896.0,
- "21": 33511.0,
- "22": 30704.0,
- "23": 38149.0,
- "24": 32677.0,
- "25": 31055.0,
- "26": 34700.0,
- "27": 35410.0,
- "28": 37268.0,
- "29": 37953.0,
- "30": 33210.0,
- "31": 30482.0,
- "32": 36908.0,
- "33": 38308.0,
- "34": 33125.0,
- "35": 34341.0,
- "36": 34925.0,
- "37": 38767.0,
- "38": 35780.0,
- "39": 38955.0,
- "40": 36485.0,
- "41": 36015.0,
- "42": 37638.0,
- "43": 33689.0,
- "44": 33688.0,
- "45": 35448.0,
- "46": 36810.0,
- "47": 40858.0,
- "48": 35696.0,
- "49": 34729.0,
- "50": 39077.0
+ "1": 16036.0,
+ "2": 17259.0,
+ "3": 17383.0,
+ "4": 16712.0,
+ "5": 18911.0,
+ "6": 19262.0,
+ "7": 18424.0,
+ "8": 16744.0,
+ "9": 18034.0,
+ "10": 15535.0,
+ "11": 19649.0,
+ "12": 18428.0,
+ "13": 19131.0,
+ "14": 19554.0,
+ "15": 18227.0,
+ "16": 18803.0,
+ "17": 18303.0,
+ "18": 18020.0,
+ "19": 18525.0,
+ "20": 17077.0,
+ "21": 17416.0,
+ "22": 15952.0,
+ "23": 19619.0,
+ "24": 16834.0,
+ "25": 16141.0,
+ "26": 17765.0,
+ "27": 18357.0,
+ "28": 19158.0,
+ "29": 19677.0,
+ "30": 17184.0,
+ "31": 15618.0,
+ "32": 18706.0,
+ "33": 19402.0,
+ "34": 16909.0,
+ "35": 17781.0,
+ "36": 17861.0,
+ "37": 19639.0,
+ "38": 18567.0,
+ "39": 20265.0,
+ "40": 18782.0,
+ "41": 18470.0,
+ "42": 19484.0,
+ "43": 17350.0,
+ "44": 17404.0,
+ "45": 18330.0,
+ "46": 18907.0,
+ "47": 20800.0,
+ "48": 18138.0,
+ "49": 17902.0,
+ "50": 20085.0
}
},
"mem-allocated-bytes": {
@@ -120,53 +120,53 @@
"values": {
"1": 1027095040.0,
"2": 1027093504.0,
- "3": 1027094528.0,
- "4": 1027095040.0,
+ "3": 1027094016.0,
+ "4": 1027096064.0,
"5": 1027091968.0,
"6": 1027091968.0,
- "7": 1027098112.0,
+ "7": 1027098624.0,
"8": 1027097600.0,
- "9": 1027094528.0,
- "10": 1027094016.0,
- "11": 1027098624.0,
- "12": 1027094528.0,
+ "9": 1027094016.0,
+ "10": 1027092992.0,
+ "11": 1027098112.0,
+ "12": 1027093504.0,
"13": 1027092480.0,
"14": 1027095040.0,
- "15": 1027095040.0,
- "16": 1027091456.0,
- "17": 1027101184.0,
+ "15": 1027095552.0,
+ "16": 1027092480.0,
+ "17": 1027102208.0,
"18": 1027096064.0,
"19": 1027093504.0,
- "20": 1027093504.0,
- "21": 1027097088.0,
- "22": 1027100160.0,
- "23": 1027100160.0,
+ "20": 1027092992.0,
+ "21": 1027096576.0,
+ "22": 1027099136.0,
+ "23": 1027098624.0,
"24": 1027095552.0,
"25": 1027097088.0,
- "26": 1027098112.0,
+ "26": 1027097600.0,
"27": 1027091456.0,
- "28": 1027090944.0,
- "29": 1027091968.0,
+ "28": 1027090432.0,
+ "29": 1027092992.0,
"30": 1027099648.0,
- "31": 1027109888.0,
- "32": 1027095552.0,
+ "31": 1027109376.0,
+ "32": 1027096576.0,
"33": 1027090944.0,
- "34": 1027098112.0,
- "35": 1027103744.0,
- "36": 1027098112.0,
- "37": 1027092480.0,
- "38": 1027091456.0,
- "39": 1027095040.0,
+ "34": 1027099136.0,
+ "35": 1027103232.0,
+ "36": 1027097088.0,
+ "37": 1027091968.0,
+ "38": 1027091968.0,
+ "39": 1027095552.0,
"40": 1027095040.0,
"41": 1027100160.0,
"42": 1027091968.0,
"43": 1027098624.0,
- "44": 1027098624.0,
- "45": 1027096064.0,
+ "44": 1027098112.0,
+ "45": 1027095552.0,
"46": 1027104256.0,
- "47": 1027093504.0,
- "48": 1027101184.0,
- "49": 1027096064.0,
+ "47": 1027092992.0,
+ "48": 1027100672.0,
+ "49": 1027096576.0,
"50": 1027095552.0
}
},
@@ -177,54 +177,54 @@
"values": {
"1": 3059586560.0,
"2": 3299159040.0,
- "3": 3299482112.0,
- "4": 3302137344.0,
- "5": 3302137344.0,
- "6": 3302137344.0,
- "7": 3303535104.0,
- "8": 3303535104.0,
- "9": 3303535104.0,
- "10": 3303535104.0,
- "11": 3303535104.0,
- "12": 3303535104.0,
- "13": 3303535104.0,
- "14": 3303535104.0,
- "15": 3303535104.0,
- "16": 3303535104.0,
- "17": 3306910208.0,
- "18": 3306910208.0,
- "19": 3306910208.0,
- "20": 3306910208.0,
- "21": 3306910208.0,
- "22": 3306910208.0,
- "23": 3306910208.0,
- "24": 3306910208.0,
- "25": 3306910208.0,
- "26": 3306910208.0,
- "27": 3306910208.0,
- "28": 3306910208.0,
- "29": 3306910208.0,
- "30": 3306910208.0,
- "31": 3312495616.0,
- "32": 3312495616.0,
- "33": 3312495616.0,
- "34": 3312495616.0,
- "35": 3312495616.0,
- "36": 3312495616.0,
- "37": 3312495616.0,
- "38": 3312495616.0,
- "39": 3312495616.0,
- "40": 3312495616.0,
- "41": 3312495616.0,
- "42": 3312495616.0,
- "43": 3312495616.0,
- "44": 3312495616.0,
- "45": 3312495616.0,
- "46": 3312495616.0,
- "47": 3312495616.0,
- "48": 3312495616.0,
- "49": 3312495616.0,
- "50": 3312495616.0
+ "3": 3299516928.0,
+ "4": 3301408256.0,
+ "5": 3301408256.0,
+ "6": 3301408256.0,
+ "7": 3303757824.0,
+ "8": 3303757824.0,
+ "9": 3303757824.0,
+ "10": 3303757824.0,
+ "11": 3303757824.0,
+ "12": 3303757824.0,
+ "13": 3303757824.0,
+ "14": 3303757824.0,
+ "15": 3303757824.0,
+ "16": 3303757824.0,
+ "17": 3307320832.0,
+ "18": 3307320832.0,
+ "19": 3307320832.0,
+ "20": 3307320832.0,
+ "21": 3307320832.0,
+ "22": 3307320832.0,
+ "23": 3307320832.0,
+ "24": 3307320832.0,
+ "25": 3307320832.0,
+ "26": 3307320832.0,
+ "27": 3307320832.0,
+ "28": 3307320832.0,
+ "29": 3307320832.0,
+ "30": 3307320832.0,
+ "31": 3312982528.0,
+ "32": 3312982528.0,
+ "33": 3312982528.0,
+ "34": 3312982528.0,
+ "35": 3312982528.0,
+ "36": 3312982528.0,
+ "37": 3312982528.0,
+ "38": 3312982528.0,
+ "39": 3312982528.0,
+ "40": 3312982528.0,
+ "41": 3312982528.0,
+ "42": 3312982528.0,
+ "43": 3312982528.0,
+ "44": 3312982528.0,
+ "45": 3312982528.0,
+ "46": 3312982528.0,
+ "47": 3312982528.0,
+ "48": 3312982528.0,
+ "49": 3312982528.0,
+ "50": 3312982528.0
}
},
"iteration-time": {
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 3.6307,
- "3": 0.3854,
- "4": 0.38116,
- "5": 0.36866,
- "6": 0.36756,
- "7": 0.37196,
- "8": 0.37096,
- "9": 0.36719,
- "10": 0.36516,
- "11": 0.36882,
- "12": 0.37126,
- "13": 0.36294,
- "14": 0.36799,
- "15": 0.3669,
- "16": 0.36835,
- "17": 0.37548,
- "18": 0.37236,
- "19": 0.36274,
- "20": 0.36388,
- "21": 0.36581,
- "22": 0.3703,
- "23": 0.36921,
- "24": 0.35712,
- "25": 0.36049,
- "26": 0.36512,
- "27": 0.36657,
- "28": 0.36074,
- "29": 0.41887,
- "30": 0.45698,
- "31": 0.54747,
- "32": 0.4695,
- "33": 0.67157,
- "34": 0.4186,
- "35": 0.39703,
- "36": 0.40139,
- "37": 0.39345,
- "38": 0.38789,
- "39": 1.0807,
- "40": 0.42023,
- "41": 0.3945,
- "42": 0.39312,
- "43": 0.41319,
- "44": 0.40657,
- "45": 0.4003,
- "46": 0.3986,
- "47": 0.38501,
- "48": 0.38618,
- "49": 0.38586,
- "50": 0.38297
+ "2": 6.76529,
+ "3": 0.37269,
+ "4": 0.35177,
+ "5": 0.3437,
+ "6": 0.34307,
+ "7": 0.35407,
+ "8": 0.351,
+ "9": 0.34371,
+ "10": 0.34471,
+ "11": 0.34856,
+ "12": 0.3428,
+ "13": 0.34482,
+ "14": 0.33823,
+ "15": 0.34058,
+ "16": 0.34565,
+ "17": 0.36095,
+ "18": 0.35119,
+ "19": 0.34454,
+ "20": 0.34311,
+ "21": 0.34742,
+ "22": 0.34649,
+ "23": 0.34449,
+ "24": 0.34322,
+ "25": 0.34366,
+ "26": 0.3456,
+ "27": 0.3515,
+ "28": 0.33973,
+ "29": 0.33934,
+ "30": 0.33919,
+ "31": 0.35138,
+ "32": 0.34004,
+ "33": 0.33992,
+ "34": 0.34399,
+ "35": 0.34028,
+ "36": 0.34132,
+ "37": 0.34535,
+ "38": 0.34494,
+ "39": 0.34035,
+ "40": 0.34755,
+ "41": 0.34682,
+ "42": 0.34284,
+ "43": 0.34981,
+ "44": 0.34838,
+ "45": 0.34237,
+ "46": 0.34685,
+ "47": 0.34282,
+ "48": 0.34833,
+ "49": 0.35569,
+ "50": 0.34633
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM/golden_values_dev_dgx_h100.json
index b626738d63e..586f94b9d87 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.80815,
- "2": 10.82612,
- "3": 10.83032,
- "4": 10.80963,
- "5": 10.84127,
- "6": 10.8581,
- "7": 10.81967,
- "8": 10.82506,
- "9": 10.83749,
- "10": 10.7783,
- "11": 10.85781,
- "12": 10.85539,
- "13": 10.85233,
- "14": 10.86699,
- "15": 10.81253,
- "16": 10.80292,
- "17": 10.78098,
- "18": 10.80788,
- "19": 10.79276,
- "20": 10.74548,
- "21": 10.72785,
- "22": 10.59608,
- "23": 10.73999,
- "24": 10.63509,
- "25": 10.59832,
- "26": 10.63517,
- "27": 10.65744,
- "28": 10.64536,
- "29": 10.65122,
- "30": 10.44144,
- "31": 10.21465,
- "32": 10.53342,
- "33": 10.52518,
- "34": 10.30171,
- "35": 10.34871,
- "36": 10.30843,
- "37": 10.42353,
- "38": 10.28859,
- "39": 10.45514,
- "40": 10.19363,
- "41": 10.22791,
- "42": 10.29725,
- "43": 9.95871,
- "44": 10.06717,
- "45": 9.95955,
- "46": 9.92614,
- "47": 10.20607,
- "48": 9.96021,
- "49": 9.65854,
- "50": 10.01296
+ "1": 10.92671,
+ "2": 10.91589,
+ "3": 10.92552,
+ "4": 10.93168,
+ "5": 10.93015,
+ "6": 10.9259,
+ "7": 10.92646,
+ "8": 10.92323,
+ "9": 10.92778,
+ "10": 10.9168,
+ "11": 10.9178,
+ "12": 10.92446,
+ "13": 10.90961,
+ "14": 10.90627,
+ "15": 10.90112,
+ "16": 10.88691,
+ "17": 10.88827,
+ "18": 10.88554,
+ "19": 10.88654,
+ "20": 10.8377,
+ "21": 10.82717,
+ "22": 10.81535,
+ "23": 10.80831,
+ "24": 10.78061,
+ "25": 10.77774,
+ "26": 10.76115,
+ "27": 10.7495,
+ "28": 10.6922,
+ "29": 10.66686,
+ "30": 10.63118,
+ "31": 10.62182,
+ "32": 10.61591,
+ "33": 10.57843,
+ "34": 10.54531,
+ "35": 10.54625,
+ "36": 10.53479,
+ "37": 10.50533,
+ "38": 10.50383,
+ "39": 10.47322,
+ "40": 10.45095,
+ "41": 10.42606,
+ "42": 10.41475,
+ "43": 10.40064,
+ "44": 10.37006,
+ "45": 10.38168,
+ "46": 10.33484,
+ "47": 10.32444,
+ "48": 10.28749,
+ "49": 10.28608,
+ "50": 10.27697
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 31590.0,
- "2": 32940.0,
- "3": 33668.0,
- "4": 31186.0,
- "5": 36214.0,
- "6": 37169.0,
- "7": 34770.0,
- "8": 31862.0,
- "9": 34102.0,
- "10": 30394.0,
- "11": 38432.0,
- "12": 35039.0,
- "13": 37236.0,
- "14": 37668.0,
- "15": 34199.0,
- "16": 36659.0,
- "17": 34831.0,
- "18": 35011.0,
- "19": 35486.0,
- "20": 33221.0,
- "21": 33971.0,
- "22": 30501.0,
- "23": 38411.0,
- "24": 32764.0,
- "25": 31363.0,
- "26": 34624.0,
- "27": 36096.0,
- "28": 37021.0,
- "29": 37900.0,
- "30": 33066.0,
- "31": 29871.0,
- "32": 36113.0,
- "33": 38168.0,
- "34": 33074.0,
- "35": 34300.0,
- "36": 35363.0,
- "37": 38150.0,
- "38": 35798.0,
- "39": 38945.0,
- "40": 35780.0,
- "41": 35999.0,
- "42": 36611.0,
- "43": 33781.0,
- "44": 34207.0,
- "45": 35198.0,
- "46": 36779.0,
- "47": 40585.0,
- "48": 36434.0,
- "49": 35787.0,
- "50": 38996.0
+ "1": 19058.0,
+ "2": 19206.0,
+ "3": 18972.0,
+ "4": 19416.0,
+ "5": 19009.0,
+ "6": 18538.0,
+ "7": 18981.0,
+ "8": 18448.0,
+ "9": 18864.0,
+ "10": 19655.0,
+ "11": 19064.0,
+ "12": 18696.0,
+ "13": 19292.0,
+ "14": 19140.0,
+ "15": 18806.0,
+ "16": 18590.0,
+ "17": 18993.0,
+ "18": 19173.0,
+ "19": 19321.0,
+ "20": 19057.0,
+ "21": 19086.0,
+ "22": 18997.0,
+ "23": 18891.0,
+ "24": 19267.0,
+ "25": 18711.0,
+ "26": 19139.0,
+ "27": 19114.0,
+ "28": 18818.0,
+ "29": 18371.0,
+ "30": 18304.0,
+ "31": 19016.0,
+ "32": 19184.0,
+ "33": 18481.0,
+ "34": 18592.0,
+ "35": 18848.0,
+ "36": 18346.0,
+ "37": 18564.0,
+ "38": 18516.0,
+ "39": 18959.0,
+ "40": 19194.0,
+ "41": 18945.0,
+ "42": 18455.0,
+ "43": 19053.0,
+ "44": 18809.0,
+ "45": 20372.0,
+ "46": 19563.0,
+ "47": 19974.0,
+ "48": 20047.0,
+ "49": 21674.0,
+ "50": 20259.0
}
},
"mem-allocated-bytes": {
@@ -118,55 +118,55 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1027090944.0,
- "2": 1027090944.0,
- "3": 1027091968.0,
- "4": 1027091456.0,
- "5": 1027091968.0,
- "6": 1027090432.0,
- "7": 1027086848.0,
- "8": 1027087872.0,
- "9": 1027094528.0,
- "10": 1027088896.0,
- "11": 1027089408.0,
- "12": 1027089408.0,
+ "1": 1027089408.0,
+ "2": 1027091968.0,
+ "3": 1027088384.0,
+ "4": 1027088384.0,
+ "5": 1027090944.0,
+ "6": 1027091968.0,
+ "7": 1027088896.0,
+ "8": 1027092992.0,
+ "9": 1027090944.0,
+ "10": 1027090432.0,
+ "11": 1027090944.0,
+ "12": 1027091456.0,
"13": 1027091968.0,
- "14": 1027088896.0,
- "15": 1027090432.0,
- "16": 1027091456.0,
- "17": 1027089408.0,
- "18": 1027093504.0,
- "19": 1027091968.0,
- "20": 1027095040.0,
- "21": 1027088384.0,
- "22": 1027091456.0,
- "23": 1027091968.0,
- "24": 1027090944.0,
- "25": 1027089408.0,
- "26": 1027090432.0,
- "27": 1027090432.0,
- "28": 1027087872.0,
- "29": 1027088896.0,
- "30": 1027087360.0,
- "31": 1027079168.0,
- "32": 1027082240.0,
- "33": 1027091456.0,
- "34": 1027088384.0,
- "35": 1027090432.0,
- "36": 1027088896.0,
- "37": 1027089408.0,
+ "14": 1027090944.0,
+ "15": 1027091456.0,
+ "16": 1027088384.0,
+ "17": 1027088896.0,
+ "18": 1027090944.0,
+ "19": 1027089920.0,
+ "20": 1027089920.0,
+ "21": 1027092480.0,
+ "22": 1027088896.0,
+ "23": 1027094528.0,
+ "24": 1027091968.0,
+ "25": 1027091456.0,
+ "26": 1027089408.0,
+ "27": 1027087872.0,
+ "28": 1027091456.0,
+ "29": 1027090432.0,
+ "30": 1027089920.0,
+ "31": 1027089408.0,
+ "32": 1027094528.0,
+ "33": 1027094016.0,
+ "34": 1027092480.0,
+ "35": 1027086848.0,
+ "36": 1027088384.0,
+ "37": 1027088896.0,
"38": 1027090432.0,
- "39": 1027085824.0,
- "40": 1027086848.0,
- "41": 1027088384.0,
- "42": 1027091968.0,
- "43": 1027084800.0,
- "44": 1027087872.0,
- "45": 1027087872.0,
- "46": 1027078656.0,
- "47": 1027087872.0,
- "48": 1027086336.0,
- "49": 1027082240.0,
+ "39": 1027090432.0,
+ "40": 1027091456.0,
+ "41": 1027088896.0,
+ "42": 1027088384.0,
+ "43": 1027088896.0,
+ "44": 1027091968.0,
+ "45": 1027091456.0,
+ "46": 1027085824.0,
+ "47": 1027089920.0,
+ "48": 1027088384.0,
+ "49": 1027086848.0,
"50": 1027089920.0
}
},
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 3057868288.0,
- "2": 3298335232.0,
- "3": 3298335232.0,
- "4": 3300084224.0,
- "5": 3300084224.0,
- "6": 3300084224.0,
- "7": 3300084224.0,
- "8": 3300084224.0,
- "9": 3300084224.0,
- "10": 3300122624.0,
- "11": 3300122624.0,
- "12": 3300122624.0,
- "13": 3300122624.0,
- "14": 3300122624.0,
- "15": 3300122624.0,
- "16": 3300122624.0,
- "17": 3300122624.0,
- "18": 3300122624.0,
- "19": 3300376576.0,
- "20": 3300416000.0,
- "21": 3300416000.0,
- "22": 3301032960.0,
- "23": 3301998080.0,
- "24": 3301998080.0,
- "25": 3301998080.0,
- "26": 3301998080.0,
- "27": 3301998080.0,
- "28": 3301998080.0,
- "29": 3301998080.0,
- "30": 3301998080.0,
- "31": 3301998080.0,
- "32": 3301998080.0,
- "33": 3301998080.0,
- "34": 3301998080.0,
- "35": 3301998080.0,
- "36": 3301998080.0,
- "37": 3301998080.0,
- "38": 3301998080.0,
- "39": 3301998080.0,
- "40": 3301998080.0,
- "41": 3301998080.0,
- "42": 3301998080.0,
- "43": 3301998080.0,
- "44": 3301998080.0,
- "45": 3301998080.0,
- "46": 3301998080.0,
- "47": 3301998080.0,
- "48": 3301998080.0,
- "49": 3301998080.0,
- "50": 3301998080.0
+ "1": 3058382848.0,
+ "2": 3298735616.0,
+ "3": 3298735616.0,
+ "4": 3298735616.0,
+ "5": 3298735616.0,
+ "6": 3298735616.0,
+ "7": 3298735616.0,
+ "8": 3299022336.0,
+ "9": 3299022336.0,
+ "10": 3299022336.0,
+ "11": 3299138048.0,
+ "12": 3299138048.0,
+ "13": 3299225088.0,
+ "14": 3299394048.0,
+ "15": 3299394048.0,
+ "16": 3299394048.0,
+ "17": 3299394048.0,
+ "18": 3299394048.0,
+ "19": 3299809792.0,
+ "20": 3299809792.0,
+ "21": 3299809792.0,
+ "22": 3299809792.0,
+ "23": 3300119552.0,
+ "24": 3300119552.0,
+ "25": 3300119552.0,
+ "26": 3300119552.0,
+ "27": 3300119552.0,
+ "28": 3300119552.0,
+ "29": 3300119552.0,
+ "30": 3300119552.0,
+ "31": 3300119552.0,
+ "32": 3300119552.0,
+ "33": 3300119552.0,
+ "34": 3300416512.0,
+ "35": 3300416512.0,
+ "36": 3300416512.0,
+ "37": 3300416512.0,
+ "38": 3300416512.0,
+ "39": 3300416512.0,
+ "40": 3300416512.0,
+ "41": 3300416512.0,
+ "42": 3300416512.0,
+ "43": 3300416512.0,
+ "44": 3300416512.0,
+ "45": 3300416512.0,
+ "46": 3300416512.0,
+ "47": 3300416512.0,
+ "48": 3300416512.0,
+ "49": 3300416512.0,
+ "50": 3300416512.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 15.57121,
- "2": 0.28312,
- "3": 0.24431,
- "4": 0.2266,
- "5": 0.21347,
- "6": 0.20803,
- "7": 0.2145,
- "8": 0.20409,
- "9": 0.2038,
- "10": 0.20378,
- "11": 0.20122,
- "12": 0.20047,
- "13": 0.2053,
- "14": 0.20008,
- "15": 0.22405,
- "16": 0.19642,
- "17": 0.20937,
- "18": 0.19918,
- "19": 0.2032,
- "20": 0.19792,
- "21": 0.19626,
- "22": 0.20047,
- "23": 0.19555,
- "24": 0.2,
- "25": 0.23371,
- "26": 0.2005,
- "27": 0.59196,
- "28": 0.19966,
- "29": 0.20231,
- "30": 0.19778,
- "31": 0.23768,
- "32": 0.20526,
- "33": 0.20518,
- "34": 0.22786,
- "35": 0.20088,
- "36": 0.21894,
- "37": 0.20033,
- "38": 0.20352,
- "39": 0.19985,
- "40": 0.20975,
- "41": 0.2189,
- "42": 0.20277,
- "43": 0.20495,
- "44": 0.20563,
- "45": 0.21473,
- "46": 0.21859,
- "47": 0.2018,
- "48": 0.22732,
- "49": 0.2668,
- "50": 0.19761
+ "1": "nan",
+ "2": 7.23651,
+ "3": 0.24222,
+ "4": 0.22131,
+ "5": 0.2271,
+ "6": 0.22305,
+ "7": 0.21362,
+ "8": 0.21345,
+ "9": 0.21177,
+ "10": 0.20554,
+ "11": 0.21683,
+ "12": 0.21959,
+ "13": 0.23214,
+ "14": 0.21046,
+ "15": 0.2093,
+ "16": 0.20781,
+ "17": 0.21094,
+ "18": 0.20855,
+ "19": 0.20679,
+ "20": 0.20604,
+ "21": 0.21437,
+ "22": 0.20598,
+ "23": 0.20879,
+ "24": 0.20414,
+ "25": 0.20266,
+ "26": 0.20454,
+ "27": 0.20634,
+ "28": 0.20309,
+ "29": 0.20238,
+ "30": 0.20203,
+ "31": 0.20437,
+ "32": 0.20127,
+ "33": 0.20216,
+ "34": 0.20283,
+ "35": 0.20336,
+ "36": 0.20293,
+ "37": 0.20654,
+ "38": 0.20237,
+ "39": 0.20306,
+ "40": 0.20384,
+ "41": 0.20686,
+ "42": 0.20485,
+ "43": 0.20433,
+ "44": 0.20288,
+ "45": 0.20816,
+ "46": 0.20343,
+ "47": 0.2071,
+ "48": 0.20408,
+ "49": 0.2097,
+ "50": 0.20466
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM/model_config.yaml
index b84bf45b890..8ced6e37a52 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -61,4 +61,6 @@ MODEL_ARGS:
--attention-backend: unfused
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_top2router/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_top2router/model_config.yaml
index 8c75b0a2e76..097beac2085 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_top2router/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_top2router/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -58,4 +58,6 @@ MODEL_ARGS:
--attention-backend: unfused
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts_etp1_ep4/golden_values_lts_dgx_a100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts_etp1_ep4/golden_values_lts_dgx_a100.json
index 43beb1e88d3..02314892489 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts_etp1_ep4/golden_values_lts_dgx_a100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts_etp1_ep4/golden_values_lts_dgx_a100.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.78519,
"2": 10.81141,
- "3": 10.79553,
- "4": 10.78363,
- "5": 10.83004,
- "6": 10.8417,
- "7": 10.81276,
- "8": 10.81576,
- "9": 10.82408,
- "10": 10.75834,
- "11": 10.86449,
- "12": 10.83694,
- "13": 10.84627,
- "14": 10.85637,
- "15": 10.79099,
- "16": 10.78206,
- "17": 10.73911,
- "18": 10.77969,
- "19": 10.77703,
- "20": 10.69044,
- "21": 10.66086,
- "22": 10.51537,
- "23": 10.68288,
- "24": 10.54469,
- "25": 10.49929,
- "26": 10.5582,
- "27": 10.57541,
- "28": 10.54334,
- "29": 10.55412,
- "30": 10.31857,
- "31": 10.06911,
- "32": 10.42007,
- "33": 10.42435,
- "34": 10.16897,
- "35": 10.22944,
- "36": 10.19358,
- "37": 10.29976,
- "38": 10.1465,
- "39": 10.37282,
- "40": 10.04287,
- "41": 10.09714,
- "42": 10.17339,
- "43": 9.79317,
- "44": 9.91871,
- "45": 9.79389,
- "46": 9.77586,
- "47": 10.10721,
- "48": 9.81704,
- "49": 9.48887,
- "50": 9.87988
+ "3": 10.79527,
+ "4": 10.78345,
+ "5": 10.83001,
+ "6": 10.84237,
+ "7": 10.81239,
+ "8": 10.81572,
+ "9": 10.82522,
+ "10": 10.75803,
+ "11": 10.86518,
+ "12": 10.83743,
+ "13": 10.84563,
+ "14": 10.85577,
+ "15": 10.79125,
+ "16": 10.78123,
+ "17": 10.73921,
+ "18": 10.7792,
+ "19": 10.77724,
+ "20": 10.69048,
+ "21": 10.66023,
+ "22": 10.51498,
+ "23": 10.6825,
+ "24": 10.54433,
+ "25": 10.49933,
+ "26": 10.55847,
+ "27": 10.57598,
+ "28": 10.5441,
+ "29": 10.5548,
+ "30": 10.31884,
+ "31": 10.06982,
+ "32": 10.42033,
+ "33": 10.42547,
+ "34": 10.16989,
+ "35": 10.23058,
+ "36": 10.19529,
+ "37": 10.30033,
+ "38": 10.14721,
+ "39": 10.37367,
+ "40": 10.0441,
+ "41": 10.09809,
+ "42": 10.17477,
+ "43": 9.79517,
+ "44": 9.92002,
+ "45": 9.79573,
+ "46": 9.77758,
+ "47": 10.10899,
+ "48": 9.81905,
+ "49": 9.4912,
+ "50": 9.88177
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 26764.0,
- "2": 29274.0,
- "3": 28827.0,
- "4": 27434.0,
- "5": 31796.0,
- "6": 32589.0,
- "7": 31434.0,
- "8": 27373.0,
- "9": 30359.0,
- "10": 25775.0,
- "11": 33420.0,
- "12": 30835.0,
- "13": 31910.0,
- "14": 33275.0,
- "15": 30931.0,
- "16": 31713.0,
- "17": 31263.0,
- "18": 30574.0,
- "19": 31200.0,
- "20": 28330.0,
- "21": 28769.0,
- "22": 27643.0,
- "23": 34373.0,
- "24": 28585.0,
- "25": 27693.0,
- "26": 30668.0,
- "27": 31536.0,
- "28": 33620.0,
- "29": 34457.0,
- "30": 29810.0,
- "31": 27999.0,
- "32": 33871.0,
- "33": 35836.0,
- "34": 30770.0,
- "35": 32783.0,
- "36": 33452.0,
- "37": 36437.0,
- "38": 34114.0,
- "39": 37433.0,
- "40": 35063.0,
- "41": 34239.0,
- "42": 36041.0,
- "43": 33976.0,
- "44": 33303.0,
- "45": 35148.0,
- "46": 36112.0,
- "47": 38996.0,
- "48": 34779.0,
- "49": 34913.0,
- "50": 38547.0
+ "1": 14142.0,
+ "2": 15107.0,
+ "3": 15001.0,
+ "4": 14175.0,
+ "5": 16507.0,
+ "6": 17067.0,
+ "7": 16089.0,
+ "8": 14137.0,
+ "9": 15664.0,
+ "10": 13397.0,
+ "11": 17523.0,
+ "12": 15835.0,
+ "13": 16584.0,
+ "14": 17253.0,
+ "15": 16019.0,
+ "16": 16521.0,
+ "17": 15891.0,
+ "18": 15875.0,
+ "19": 16058.0,
+ "20": 14544.0,
+ "21": 14905.0,
+ "22": 14052.0,
+ "23": 17975.0,
+ "24": 14536.0,
+ "25": 14079.0,
+ "26": 15519.0,
+ "27": 15908.0,
+ "28": 16967.0,
+ "29": 17412.0,
+ "30": 15152.0,
+ "31": 13936.0,
+ "32": 16635.0,
+ "33": 18077.0,
+ "34": 15000.0,
+ "35": 16125.0,
+ "36": 16723.0,
+ "37": 18480.0,
+ "38": 17564.0,
+ "39": 18982.0,
+ "40": 17474.0,
+ "41": 17245.0,
+ "42": 18461.0,
+ "43": 16861.0,
+ "44": 16907.0,
+ "45": 17626.0,
+ "46": 18187.0,
+ "47": 19746.0,
+ "48": 17876.0,
+ "49": 17401.0,
+ "50": 19524.0
}
},
"mem-allocated-bytes": {
@@ -121,52 +121,52 @@
"1": 1364012544.0,
"2": 1364020736.0,
"3": 1364009472.0,
- "4": 1364020224.0,
- "5": 1364007424.0,
+ "4": 1364019712.0,
+ "5": 1364007936.0,
"6": 1364005888.0,
- "7": 1364011520.0,
+ "7": 1364013056.0,
"8": 1364006912.0,
"9": 1364019712.0,
- "10": 1364001280.0,
- "11": 1364009984.0,
- "12": 1364022784.0,
+ "10": 1364001792.0,
+ "11": 1364010496.0,
+ "12": 1364023296.0,
"13": 1364008960.0,
- "14": 1364003840.0,
- "15": 1364007936.0,
- "16": 1363994112.0,
- "17": 1363993088.0,
- "18": 1364017152.0,
+ "14": 1364001792.0,
+ "15": 1364007424.0,
+ "16": 1363992576.0,
+ "17": 1363993600.0,
+ "18": 1364015616.0,
"19": 1364009984.0,
- "20": 1363999744.0,
- "21": 1364006400.0,
- "22": 1363983360.0,
- "23": 1363997696.0,
+ "20": 1364000256.0,
+ "21": 1364005888.0,
+ "22": 1363982848.0,
+ "23": 1363998208.0,
"24": 1364001792.0,
- "25": 1363992576.0,
- "26": 1364003328.0,
- "27": 1363998720.0,
+ "25": 1363993088.0,
+ "26": 1364003840.0,
+ "27": 1363998208.0,
"28": 1363982336.0,
"29": 1363995136.0,
- "30": 1363979776.0,
- "31": 1363991552.0,
- "32": 1363983872.0,
- "33": 1363983872.0,
+ "30": 1363980800.0,
+ "31": 1363990528.0,
+ "32": 1363982336.0,
+ "33": 1363982848.0,
"34": 1363997184.0,
"35": 1363991040.0,
"36": 1363984896.0,
- "37": 1363998720.0,
- "38": 1363991040.0,
- "39": 1363976704.0,
+ "37": 1363997696.0,
+ "38": 1363992064.0,
+ "39": 1363977216.0,
"40": 1363971584.0,
"41": 1363983872.0,
"42": 1363977728.0,
- "43": 1363967488.0,
+ "43": 1363969024.0,
"44": 1363974656.0,
- "45": 1363974144.0,
- "46": 1363965952.0,
- "47": 1363952640.0,
- "48": 1363951616.0,
- "49": 1363960320.0,
+ "45": 1363973632.0,
+ "46": 1363966976.0,
+ "47": 1363951616.0,
+ "48": 1363950592.0,
+ "49": 1363962880.0,
"50": 1363956736.0
}
},
@@ -177,54 +177,54 @@
"values": {
"1": 3405945344.0,
"2": 3972516352.0,
- "3": 3976973312.0,
- "4": 3976973312.0,
- "5": 3976973312.0,
- "6": 3976973312.0,
- "7": 3976973312.0,
- "8": 3976973312.0,
- "9": 3976973312.0,
- "10": 3976973312.0,
- "11": 3976973312.0,
- "12": 3976975872.0,
- "13": 3976975872.0,
- "14": 3976975872.0,
- "15": 3976975872.0,
- "16": 3976975872.0,
- "17": 3976975872.0,
- "18": 3976975872.0,
- "19": 3976975872.0,
- "20": 3976975872.0,
- "21": 3976975872.0,
- "22": 3976975872.0,
- "23": 3976975872.0,
- "24": 3976975872.0,
- "25": 3976975872.0,
- "26": 3976975872.0,
- "27": 3976975872.0,
- "28": 3976975872.0,
- "29": 3976975872.0,
- "30": 3976975872.0,
- "31": 3976975872.0,
- "32": 3976975872.0,
- "33": 3976975872.0,
- "34": 3976975872.0,
- "35": 3976975872.0,
- "36": 3976975872.0,
- "37": 3976975872.0,
- "38": 3976975872.0,
- "39": 3976975872.0,
- "40": 3976975872.0,
- "41": 3976975872.0,
- "42": 3976975872.0,
- "43": 3976975872.0,
- "44": 3976975872.0,
- "45": 3976975872.0,
- "46": 3976975872.0,
- "47": 3976975872.0,
- "48": 3976975872.0,
- "49": 3976975872.0,
- "50": 3976975872.0
+ "3": 3976958976.0,
+ "4": 3977828864.0,
+ "5": 3977828864.0,
+ "6": 3977828864.0,
+ "7": 3977828864.0,
+ "8": 3977828864.0,
+ "9": 3977828864.0,
+ "10": 3977828864.0,
+ "11": 3977828864.0,
+ "12": 3979322368.0,
+ "13": 3979322368.0,
+ "14": 3979322368.0,
+ "15": 3979322368.0,
+ "16": 3979322368.0,
+ "17": 3979322368.0,
+ "18": 3979322368.0,
+ "19": 3979322368.0,
+ "20": 3979322368.0,
+ "21": 3979322368.0,
+ "22": 3979322368.0,
+ "23": 3979322368.0,
+ "24": 3979322368.0,
+ "25": 3979322368.0,
+ "26": 3979322368.0,
+ "27": 3979322368.0,
+ "28": 3979322368.0,
+ "29": 3979322368.0,
+ "30": 3979322368.0,
+ "31": 3979322368.0,
+ "32": 3979322368.0,
+ "33": 3979322368.0,
+ "34": 3979322368.0,
+ "35": 3979322368.0,
+ "36": 3979322368.0,
+ "37": 3979322368.0,
+ "38": 3979322368.0,
+ "39": 3979322368.0,
+ "40": 3979322368.0,
+ "41": 3979322368.0,
+ "42": 3979322368.0,
+ "43": 3979322368.0,
+ "44": 3979322368.0,
+ "45": 3979322368.0,
+ "46": 3979322368.0,
+ "47": 3979322368.0,
+ "48": 3979322368.0,
+ "49": 3979322368.0,
+ "50": 3979322368.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 9.45286,
- "2": 0.38607,
- "3": 0.3213,
- "4": 0.29678,
- "5": 0.29879,
- "6": 0.29861,
- "7": 0.29609,
- "8": 0.29454,
- "9": 0.29554,
- "10": 0.2938,
- "11": 0.29617,
- "12": 0.29426,
- "13": 0.29354,
- "14": 0.29415,
- "15": 0.29446,
- "16": 0.29436,
- "17": 0.29604,
- "18": 0.29438,
- "19": 0.29445,
- "20": 0.2949,
- "21": 0.29462,
- "22": 0.2942,
- "23": 0.29494,
- "24": 0.29415,
- "25": 0.29456,
- "26": 0.29464,
- "27": 0.29403,
- "28": 0.29487,
- "29": 0.29396,
- "30": 0.30341,
- "31": 0.29906,
- "32": 0.29469,
- "33": 0.29821,
- "34": 0.29373,
- "35": 0.294,
- "36": 0.6955,
- "37": 0.30497,
- "38": 0.29453,
- "39": 0.29652,
- "40": 0.29409,
- "41": 0.29484,
- "42": 0.29643,
- "43": 0.29621,
- "44": 0.2949,
- "45": 0.29781,
- "46": 0.29896,
- "47": 0.29487,
- "48": 0.29896,
- "49": 0.29728,
- "50": 0.29271
+ "1": "nan",
+ "2": 4.02657,
+ "3": 0.34112,
+ "4": 0.30162,
+ "5": 0.30114,
+ "6": 0.30054,
+ "7": 0.3007,
+ "8": 0.29644,
+ "9": 0.29418,
+ "10": 0.29531,
+ "11": 0.29529,
+ "12": 0.29831,
+ "13": 0.29343,
+ "14": 0.29548,
+ "15": 0.29607,
+ "16": 0.29615,
+ "17": 0.30182,
+ "18": 0.29753,
+ "19": 0.29491,
+ "20": 0.29578,
+ "21": 0.29691,
+ "22": 0.2958,
+ "23": 0.29643,
+ "24": 0.29699,
+ "25": 0.29694,
+ "26": 0.29861,
+ "27": 0.29845,
+ "28": 0.29819,
+ "29": 0.29803,
+ "30": 0.30734,
+ "31": 0.30542,
+ "32": 0.29902,
+ "33": 0.30795,
+ "34": 0.29782,
+ "35": 0.29694,
+ "36": 0.3017,
+ "37": 0.30746,
+ "38": 0.29574,
+ "39": 0.29774,
+ "40": 0.29849,
+ "41": 0.29716,
+ "42": 0.29956,
+ "43": 0.30025,
+ "44": 0.29702,
+ "45": 0.2986,
+ "46": 0.30912,
+ "47": 0.29931,
+ "48": 0.3009,
+ "49": 0.29955,
+ "50": 0.33489
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts_etp1_ep4/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts_etp1_ep4/model_config.yaml
index 978babb72ff..8ae6dc79fe2 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts_etp1_ep4/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts_etp1_ep4/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -61,4 +61,6 @@ MODEL_ARGS:
--attention-backend: unfused
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4/golden_values_dev_dgx_gb200.json
index 682fa44a64d..192d3093a17 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4/golden_values_dev_dgx_gb200.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.82137,
"2": 10.8271,
- "3": 10.81279,
- "4": 10.80424,
- "5": 10.84481,
- "6": 10.85159,
- "7": 10.82705,
- "8": 10.83127,
- "9": 10.8396,
- "10": 10.79638,
- "11": 10.85834,
- "12": 10.8443,
- "13": 10.8625,
- "14": 10.86559,
- "15": 10.8001,
- "16": 10.78718,
- "17": 10.7639,
- "18": 10.78578,
- "19": 10.78836,
- "20": 10.71249,
- "21": 10.68241,
- "22": 10.54353,
- "23": 10.69825,
- "24": 10.58633,
- "25": 10.52721,
- "26": 10.58871,
- "27": 10.60408,
- "28": 10.57696,
- "29": 10.57897,
- "30": 10.36401,
- "31": 10.10796,
- "32": 10.44854,
- "33": 10.4401,
- "34": 10.20252,
- "35": 10.25069,
- "36": 10.21055,
- "37": 10.32849,
- "38": 10.17511,
- "39": 10.38336,
- "40": 10.05674,
- "41": 10.10841,
- "42": 10.18865,
- "43": 9.80582,
- "44": 9.91887,
- "45": 9.79924,
- "46": 9.78948,
- "47": 10.11342,
- "48": 9.82499,
- "49": 9.49844,
- "50": 9.87311
+ "3": 10.81259,
+ "4": 10.80461,
+ "5": 10.84514,
+ "6": 10.85155,
+ "7": 10.82706,
+ "8": 10.83144,
+ "9": 10.83882,
+ "10": 10.79627,
+ "11": 10.85798,
+ "12": 10.84396,
+ "13": 10.86231,
+ "14": 10.86484,
+ "15": 10.80037,
+ "16": 10.78689,
+ "17": 10.76427,
+ "18": 10.78613,
+ "19": 10.78765,
+ "20": 10.71377,
+ "21": 10.68204,
+ "22": 10.54385,
+ "23": 10.69766,
+ "24": 10.58681,
+ "25": 10.52663,
+ "26": 10.58855,
+ "27": 10.60532,
+ "28": 10.57832,
+ "29": 10.5781,
+ "30": 10.36447,
+ "31": 10.10806,
+ "32": 10.44862,
+ "33": 10.44033,
+ "34": 10.20363,
+ "35": 10.25061,
+ "36": 10.21176,
+ "37": 10.3295,
+ "38": 10.17605,
+ "39": 10.38416,
+ "40": 10.05848,
+ "41": 10.10947,
+ "42": 10.19022,
+ "43": 9.80798,
+ "44": 9.92023,
+ "45": 9.8015,
+ "46": 9.79125,
+ "47": 10.11509,
+ "48": 9.82677,
+ "49": 9.50159,
+ "50": 9.87536
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 27059.0,
- "2": 29311.0,
- "3": 28886.0,
- "4": 27768.0,
- "5": 32694.0,
- "6": 33260.0,
- "7": 31409.0,
- "8": 27342.0,
- "9": 30401.0,
- "10": 25524.0,
- "11": 33805.0,
- "12": 31146.0,
- "13": 33161.0,
- "14": 33991.0,
- "15": 31160.0,
- "16": 32445.0,
- "17": 30974.0,
- "18": 31151.0,
- "19": 31742.0,
- "20": 28624.0,
- "21": 29115.0,
- "22": 26827.0,
- "23": 34472.0,
- "24": 29096.0,
- "25": 27239.0,
- "26": 30910.0,
- "27": 31915.0,
- "28": 33968.0,
- "29": 36017.0,
- "30": 30702.0,
- "31": 27384.0,
- "32": 33681.0,
- "33": 35476.0,
- "34": 30160.0,
- "35": 31419.0,
- "36": 32568.0,
- "37": 36189.0,
- "38": 33607.0,
- "39": 37731.0,
- "40": 34463.0,
- "41": 33229.0,
- "42": 35616.0,
- "43": 32361.0,
- "44": 31908.0,
- "45": 33571.0,
- "46": 33618.0,
- "47": 38873.0,
- "48": 35034.0,
- "49": 34407.0,
- "50": 37669.0
+ "1": 14346.0,
+ "2": 15291.0,
+ "3": 15518.0,
+ "4": 14945.0,
+ "5": 16686.0,
+ "6": 17241.0,
+ "7": 16456.0,
+ "8": 14840.0,
+ "9": 16032.0,
+ "10": 13167.0,
+ "11": 17344.0,
+ "12": 15881.0,
+ "13": 16775.0,
+ "14": 17326.0,
+ "15": 16481.0,
+ "16": 16773.0,
+ "17": 16577.0,
+ "18": 16245.0,
+ "19": 16474.0,
+ "20": 14982.0,
+ "21": 15340.0,
+ "22": 14114.0,
+ "23": 17814.0,
+ "24": 15076.0,
+ "25": 14186.0,
+ "26": 15900.0,
+ "27": 16168.0,
+ "28": 17558.0,
+ "29": 18160.0,
+ "30": 15708.0,
+ "31": 14331.0,
+ "32": 17266.0,
+ "33": 17945.0,
+ "34": 16018.0,
+ "35": 16116.0,
+ "36": 16657.0,
+ "37": 18559.0,
+ "38": 17185.0,
+ "39": 18877.0,
+ "40": 17864.0,
+ "41": 17187.0,
+ "42": 18452.0,
+ "43": 16844.0,
+ "44": 16609.0,
+ "45": 17459.0,
+ "46": 17934.0,
+ "47": 20374.0,
+ "48": 18094.0,
+ "49": 18298.0,
+ "50": 19368.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1559412224.0,
- "2": 1558616064.0,
- "3": 1558495744.0,
- "4": 1559267328.0,
- "5": 1558842880.0,
- "6": 1559098368.0,
- "7": 1558495744.0,
- "8": 1558546432.0,
- "9": 1558495744.0,
- "10": 1558546432.0,
- "11": 1558597120.0,
- "12": 1558546432.0,
- "13": 1558597120.0,
- "14": 1558546432.0,
- "15": 1558904320.0,
- "16": 1558647808.0,
- "17": 1558597120.0,
- "18": 1558889472.0,
- "19": 1558597120.0,
- "20": 1559229440.0,
- "21": 1558597120.0,
- "22": 1558758400.0,
- "23": 1559698944.0,
- "24": 1559078912.0,
- "25": 1559052800.0,
- "26": 1558647808.0,
- "27": 1559382528.0,
- "28": 1558749184.0,
- "29": 1558830592.0,
- "30": 1558749184.0,
- "31": 1558915584.0,
- "32": 1559541760.0,
- "33": 1558698496.0,
- "34": 1558749184.0,
- "35": 1559422464.0,
- "36": 1558863872.0,
- "37": 1558799872.0,
- "38": 1558749184.0,
- "39": 1559397888.0,
- "40": 1559002112.0,
- "41": 1558799872.0,
- "42": 1558850560.0,
- "43": 1559724544.0,
- "44": 1558850560.0,
- "45": 1558901248.0,
- "46": 1559175168.0,
- "47": 1558901248.0,
- "48": 1558850560.0,
- "49": 1558901248.0,
- "50": 1559632896.0
+ "1": 1559687680.0,
+ "2": 1558320128.0,
+ "3": 1558141440.0,
+ "4": 1558585344.0,
+ "5": 1558673920.0,
+ "6": 1559820288.0,
+ "7": 1558721024.0,
+ "8": 1558277120.0,
+ "9": 1558141440.0,
+ "10": 1558192128.0,
+ "11": 1558824448.0,
+ "12": 1558469632.0,
+ "13": 1558242816.0,
+ "14": 1558192128.0,
+ "15": 1558558208.0,
+ "16": 1558192128.0,
+ "17": 1558242816.0,
+ "18": 1558293504.0,
+ "19": 1558927872.0,
+ "20": 1558793216.0,
+ "21": 1558242816.0,
+ "22": 1559059456.0,
+ "23": 1558344192.0,
+ "24": 1558785024.0,
+ "25": 1558811136.0,
+ "26": 1558293504.0,
+ "27": 1558344192.0,
+ "28": 1558293504.0,
+ "29": 1558782464.0,
+ "30": 1558394880.0,
+ "31": 1558344192.0,
+ "32": 1558394880.0,
+ "33": 1558586880.0,
+ "34": 1559012352.0,
+ "35": 1558756864.0,
+ "36": 1558394880.0,
+ "37": 1558445568.0,
+ "38": 1558762496.0,
+ "39": 1559027200.0,
+ "40": 1558394880.0,
+ "41": 1558445568.0,
+ "42": 1558496256.0,
+ "43": 1558445568.0,
+ "44": 1558729728.0,
+ "45": 1558969856.0,
+ "46": 1559446528.0,
+ "47": 1558946304.0,
+ "48": 1558496256.0,
+ "49": 1559235072.0,
+ "50": 1559438336.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 3495116800.0,
- "2": 4054579712.0,
- "3": 4062724096.0,
- "4": 4062724096.0,
- "5": 4062724096.0,
- "6": 4062724096.0,
- "7": 4070930432.0,
- "8": 4070930432.0,
- "9": 4073446400.0,
- "10": 4073446400.0,
- "11": 4073446400.0,
- "12": 4073446400.0,
- "13": 4073446400.0,
- "14": 4075493888.0,
- "15": 4075493888.0,
- "16": 4075493888.0,
- "17": 4075493888.0,
- "18": 4075493888.0,
- "19": 4075493888.0,
- "20": 4075493888.0,
- "21": 4075493888.0,
- "22": 4079303168.0,
- "23": 4096666624.0,
- "24": 4096666624.0,
- "25": 4096666624.0,
- "26": 4096666624.0,
- "27": 4096666624.0,
- "28": 4096666624.0,
- "29": 4096666624.0,
- "30": 4096666624.0,
- "31": 4105302016.0,
- "32": 4105302016.0,
- "33": 4105302016.0,
- "34": 4105302016.0,
- "35": 4105302016.0,
- "36": 4105302016.0,
- "37": 4105302016.0,
- "38": 4105302016.0,
- "39": 4105302016.0,
- "40": 4105302016.0,
- "41": 4105302016.0,
- "42": 4105302016.0,
- "43": 4105302016.0,
- "44": 4105302016.0,
- "45": 4105302016.0,
- "46": 4105302016.0,
- "47": 4105302016.0,
- "48": 4105302016.0,
- "49": 4105302016.0,
- "50": 4105302016.0
+ "1": 3497353216.0,
+ "2": 4057615872.0,
+ "3": 4064940032.0,
+ "4": 4064940032.0,
+ "5": 4066543616.0,
+ "6": 4066543616.0,
+ "7": 4072271872.0,
+ "8": 4072271872.0,
+ "9": 4073189376.0,
+ "10": 4074603008.0,
+ "11": 4074603008.0,
+ "12": 4074603008.0,
+ "13": 4074603008.0,
+ "14": 4076695040.0,
+ "15": 4076695040.0,
+ "16": 4080452096.0,
+ "17": 4080452096.0,
+ "18": 4080452096.0,
+ "19": 4080452096.0,
+ "20": 4080452096.0,
+ "21": 4080452096.0,
+ "22": 4080452096.0,
+ "23": 4095987712.0,
+ "24": 4095987712.0,
+ "25": 4095987712.0,
+ "26": 4095987712.0,
+ "27": 4095987712.0,
+ "28": 4095987712.0,
+ "29": 4095987712.0,
+ "30": 4095987712.0,
+ "31": 4103447552.0,
+ "32": 4103447552.0,
+ "33": 4103447552.0,
+ "34": 4103447552.0,
+ "35": 4103447552.0,
+ "36": 4103447552.0,
+ "37": 4103447552.0,
+ "38": 4103447552.0,
+ "39": 4103447552.0,
+ "40": 4103447552.0,
+ "41": 4103447552.0,
+ "42": 4103447552.0,
+ "43": 4103447552.0,
+ "44": 4103447552.0,
+ "45": 4103447552.0,
+ "46": 4103447552.0,
+ "47": 4103447552.0,
+ "48": 4103447552.0,
+ "49": 4103447552.0,
+ "50": 4103447552.0
}
},
"iteration-time": {
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 6.04776,
- "3": 0.43191,
- "4": 0.39355,
- "5": 0.39556,
- "6": 0.39818,
- "7": 0.39915,
- "8": 0.39139,
- "9": 0.41074,
- "10": 0.45245,
- "11": 0.45849,
- "12": 0.46806,
- "13": 0.46943,
- "14": 0.47411,
- "15": 0.48525,
- "16": 0.47939,
- "17": 0.47872,
- "18": 0.4715,
- "19": 0.4792,
- "20": 0.46531,
- "21": 0.46809,
- "22": 0.46348,
- "23": 0.47875,
- "24": 0.83175,
- "25": 0.50009,
- "26": 0.4884,
- "27": 0.82926,
- "28": 0.50184,
- "29": 0.50509,
- "30": 0.49725,
- "31": 0.50602,
- "32": 0.84607,
- "33": 0.50581,
- "34": 0.49849,
- "35": 0.50057,
- "36": 0.5007,
- "37": 0.50598,
- "38": 0.50147,
- "39": 0.51593,
- "40": 0.51491,
- "41": 0.50337,
- "42": 0.48945,
- "43": 0.49729,
- "44": 0.49341,
- "45": 0.4898,
- "46": 0.49624,
- "47": 0.51146,
- "48": 0.49582,
- "49": 0.49624,
- "50": 0.49469
+ "2": 9.94734,
+ "3": 0.62739,
+ "4": 0.44562,
+ "5": 0.44422,
+ "6": 0.45335,
+ "7": 0.44469,
+ "8": 0.43325,
+ "9": 0.42886,
+ "10": 0.43224,
+ "11": 0.43472,
+ "12": 0.43387,
+ "13": 0.4369,
+ "14": 0.43113,
+ "15": 0.43054,
+ "16": 0.4414,
+ "17": 0.43828,
+ "18": 0.44125,
+ "19": 0.44329,
+ "20": 0.43677,
+ "21": 0.43545,
+ "22": 0.4371,
+ "23": 0.42872,
+ "24": 0.43607,
+ "25": 0.43634,
+ "26": 0.44317,
+ "27": 0.43853,
+ "28": 0.4383,
+ "29": 0.44908,
+ "30": 0.43878,
+ "31": 0.43508,
+ "32": 0.43837,
+ "33": 0.44706,
+ "34": 0.43665,
+ "35": 0.43563,
+ "36": 0.44265,
+ "37": 0.43653,
+ "38": 0.4425,
+ "39": 0.43623,
+ "40": 0.43924,
+ "41": 0.43333,
+ "42": 0.43303,
+ "43": 0.43828,
+ "44": 0.43412,
+ "45": 0.43686,
+ "46": 0.4399,
+ "47": 0.47184,
+ "48": 0.42994,
+ "49": 0.43508,
+ "50": 0.43773
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4/golden_values_dev_dgx_h100.json
index 19b393f6369..0e601bc661a 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 10.77518,
- "2": 10.78038,
- "3": 10.79302,
- "4": 10.74107,
- "5": 10.82013,
- "6": 10.82951,
- "7": 10.7953,
- "8": 10.78263,
- "9": 10.79278,
- "10": 10.7446,
- "11": 10.85147,
- "12": 10.82613,
- "13": 10.82825,
- "14": 10.85504,
- "15": 10.75536,
- "16": 10.75777,
- "17": 10.72319,
- "18": 10.76274,
- "19": 10.75075,
- "20": 10.66587,
- "21": 10.6419,
- "22": 10.47523,
- "23": 10.66959,
- "24": 10.54157,
- "25": 10.4825,
- "26": 10.55255,
- "27": 10.57459,
- "28": 10.55159,
- "29": 10.5668,
- "30": 10.31134,
- "31": 10.01921,
- "32": 10.42655,
- "33": 10.42294,
- "34": 10.14739,
- "35": 10.21574,
- "36": 10.15811,
- "37": 10.30279,
- "38": 10.14031,
- "39": 10.36301,
- "40": 10.02669,
- "41": 10.07635,
- "42": 10.16156,
- "43": 9.74374,
- "44": 9.88962,
- "45": 9.75874,
- "46": 9.73618,
- "47": 10.0844,
- "48": 9.78532,
- "49": 9.45072,
- "50": 9.85634
+ "1": 10.90768,
+ "2": 10.90727,
+ "3": 10.9168,
+ "4": 10.90829,
+ "5": 10.91479,
+ "6": 10.89485,
+ "7": 10.90737,
+ "8": 10.90882,
+ "9": 10.90915,
+ "10": 10.91068,
+ "11": 10.90062,
+ "12": 10.89878,
+ "13": 10.88648,
+ "14": 10.88227,
+ "15": 10.87325,
+ "16": 10.85233,
+ "17": 10.85658,
+ "18": 10.84766,
+ "19": 10.85521,
+ "20": 10.77651,
+ "21": 10.76089,
+ "22": 10.75994,
+ "23": 10.7431,
+ "24": 10.70783,
+ "25": 10.70972,
+ "26": 10.69229,
+ "27": 10.66881,
+ "28": 10.60562,
+ "29": 10.57195,
+ "30": 10.54188,
+ "31": 10.53201,
+ "32": 10.51676,
+ "33": 10.48119,
+ "34": 10.44953,
+ "35": 10.44578,
+ "36": 10.42078,
+ "37": 10.40052,
+ "38": 10.4042,
+ "39": 10.36985,
+ "40": 10.35254,
+ "41": 10.33024,
+ "42": 10.31102,
+ "43": 10.29815,
+ "44": 10.27127,
+ "45": 10.28382,
+ "46": 10.24095,
+ "47": 10.23461,
+ "48": 10.19191,
+ "49": 10.19522,
+ "50": 10.19066
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 27105.0,
- "2": 28791.0,
- "3": 29282.0,
- "4": 27583.0,
- "5": 31595.0,
- "6": 32831.0,
- "7": 31023.0,
- "8": 27107.0,
- "9": 30780.0,
- "10": 25505.0,
- "11": 33684.0,
- "12": 30235.0,
- "13": 32960.0,
- "14": 32880.0,
- "15": 30405.0,
- "16": 32455.0,
- "17": 30933.0,
- "18": 30623.0,
- "19": 30803.0,
- "20": 28593.0,
- "21": 29002.0,
- "22": 27030.0,
- "23": 34463.0,
- "24": 29154.0,
- "25": 27827.0,
- "26": 31119.0,
- "27": 32108.0,
- "28": 33412.0,
- "29": 34737.0,
- "30": 30465.0,
- "31": 28775.0,
- "32": 33115.0,
- "33": 34745.0,
- "34": 30785.0,
- "35": 32116.0,
- "36": 33968.0,
- "37": 36757.0,
- "38": 34150.0,
- "39": 37240.0,
- "40": 35353.0,
- "41": 34638.0,
- "42": 36703.0,
- "43": 34601.0,
- "44": 33783.0,
- "45": 35388.0,
- "46": 35484.0,
- "47": 40591.0,
- "48": 36671.0,
- "49": 36174.0,
- "50": 38231.0
+ "1": 16760.0,
+ "2": 16448.0,
+ "3": 16457.0,
+ "4": 16370.0,
+ "5": 16128.0,
+ "6": 15986.0,
+ "7": 16871.0,
+ "8": 16013.0,
+ "9": 16494.0,
+ "10": 16663.0,
+ "11": 16350.0,
+ "12": 15746.0,
+ "13": 16650.0,
+ "14": 16501.0,
+ "15": 16034.0,
+ "16": 16079.0,
+ "17": 16562.0,
+ "18": 16415.0,
+ "19": 16924.0,
+ "20": 16413.0,
+ "21": 16333.0,
+ "22": 16350.0,
+ "23": 16127.0,
+ "24": 16475.0,
+ "25": 15776.0,
+ "26": 16841.0,
+ "27": 16638.0,
+ "28": 16195.0,
+ "29": 16566.0,
+ "30": 16223.0,
+ "31": 16976.0,
+ "32": 17083.0,
+ "33": 17181.0,
+ "34": 17125.0,
+ "35": 18157.0,
+ "36": 17429.0,
+ "37": 17735.0,
+ "38": 17898.0,
+ "39": 18505.0,
+ "40": 19060.0,
+ "41": 18163.0,
+ "42": 18145.0,
+ "43": 18663.0,
+ "44": 18601.0,
+ "45": 20633.0,
+ "46": 20049.0,
+ "47": 19866.0,
+ "48": 20108.0,
+ "49": 21960.0,
+ "50": 20138.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1561031168.0,
- "2": 1562193408.0,
- "3": 1561517056.0,
- "4": 1560948224.0,
- "5": 1562155008.0,
- "6": 1563247104.0,
- "7": 1562656768.0,
- "8": 1562246656.0,
- "9": 1561597952.0,
- "10": 1564070400.0,
- "11": 1562084352.0,
- "12": 1559892480.0,
- "13": 1562137600.0,
- "14": 1561026048.0,
- "15": 1561419776.0,
- "16": 1562166784.0,
- "17": 1560322048.0,
- "18": 1561402880.0,
- "19": 1564046336.0,
- "20": 1562059264.0,
- "21": 1560781824.0,
- "22": 1561673728.0,
- "23": 1562520064.0,
- "24": 1561093632.0,
- "25": 1561384960.0,
- "26": 1562000896.0,
- "27": 1561264128.0,
- "28": 1561458176.0,
- "29": 1561382912.0,
- "30": 1562413568.0,
- "31": 1560165376.0,
- "32": 1561413120.0,
- "33": 1562501120.0,
- "34": 1562718720.0,
- "35": 1563195392.0,
- "36": 1561894400.0,
- "37": 1560998912.0,
- "38": 1563760128.0,
- "39": 1561207808.0,
- "40": 1562625536.0,
- "41": 1561658368.0,
- "42": 1561409024.0,
- "43": 1559668736.0,
- "44": 1561136640.0,
- "45": 1560246272.0,
- "46": 1562813952.0,
- "47": 1561296896.0,
- "48": 1561900544.0,
- "49": 1562101760.0,
- "50": 1563655680.0
+ "1": 1558401536.0,
+ "2": 1558716416.0,
+ "3": 1559734784.0,
+ "4": 1558522880.0,
+ "5": 1558433280.0,
+ "6": 1558899712.0,
+ "7": 1559085568.0,
+ "8": 1559302144.0,
+ "9": 1558675968.0,
+ "10": 1559221248.0,
+ "11": 1558368768.0,
+ "12": 1558981632.0,
+ "13": 1559165440.0,
+ "14": 1558203392.0,
+ "15": 1558620672.0,
+ "16": 1558203392.0,
+ "17": 1558254080.0,
+ "18": 1559229440.0,
+ "19": 1558510080.0,
+ "20": 1558384640.0,
+ "21": 1559245312.0,
+ "22": 1558439936.0,
+ "23": 1558500864.0,
+ "24": 1558304768.0,
+ "25": 1558666752.0,
+ "26": 1558304768.0,
+ "27": 1558846976.0,
+ "28": 1558304768.0,
+ "29": 1558355456.0,
+ "30": 1559102464.0,
+ "31": 1559019008.0,
+ "32": 1559699456.0,
+ "33": 1559100928.0,
+ "34": 1558921216.0,
+ "35": 1558616576.0,
+ "36": 1558406144.0,
+ "37": 1559692800.0,
+ "38": 1558406144.0,
+ "39": 1558585856.0,
+ "40": 1559957504.0,
+ "41": 1559263744.0,
+ "42": 1558507520.0,
+ "43": 1558776320.0,
+ "44": 1559255040.0,
+ "45": 1558456832.0,
+ "46": 1558831104.0,
+ "47": 1558648320.0,
+ "48": 1558507520.0,
+ "49": 1559791104.0,
+ "50": 1558966272.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 3465706496.0,
- "2": 4045009920.0,
- "3": 4045009920.0,
- "4": 4045009920.0,
- "5": 4045009920.0,
- "6": 4067111936.0,
- "7": 4067111936.0,
- "8": 4067111936.0,
- "9": 4067111936.0,
- "10": 4067111936.0,
- "11": 4067111936.0,
- "12": 4067111936.0,
- "13": 4067111936.0,
- "14": 4067111936.0,
- "15": 4067111936.0,
- "16": 4067111936.0,
- "17": 4067111936.0,
- "18": 4067111936.0,
- "19": 4067111936.0,
- "20": 4067111936.0,
- "21": 4067111936.0,
- "22": 4067111936.0,
- "23": 4067111936.0,
- "24": 4067111936.0,
- "25": 4067111936.0,
- "26": 4067111936.0,
- "27": 4067111936.0,
- "28": 4067111936.0,
- "29": 4067111936.0,
- "30": 4067111936.0,
- "31": 4067111936.0,
- "32": 4067111936.0,
- "33": 4067111936.0,
- "34": 4067111936.0,
- "35": 4067111936.0,
- "36": 4067111936.0,
- "37": 4067111936.0,
- "38": 4067111936.0,
- "39": 4067111936.0,
- "40": 4067111936.0,
- "41": 4067111936.0,
- "42": 4067111936.0,
- "43": 4067111936.0,
- "44": 4067111936.0,
- "45": 4067111936.0,
- "46": 4067111936.0,
- "47": 4067111936.0,
- "48": 4067111936.0,
- "49": 4067111936.0,
- "50": 4067111936.0
+ "1": 3480202240.0,
+ "2": 4039383552.0,
+ "3": 4045734912.0,
+ "4": 4052123136.0,
+ "5": 4052123136.0,
+ "6": 4053458432.0,
+ "7": 4054095872.0,
+ "8": 4058186240.0,
+ "9": 4059530240.0,
+ "10": 4061010432.0,
+ "11": 4061010432.0,
+ "12": 4061010432.0,
+ "13": 4061010432.0,
+ "14": 4061010432.0,
+ "15": 4061010432.0,
+ "16": 4061010432.0,
+ "17": 4061010432.0,
+ "18": 4061010432.0,
+ "19": 4061010432.0,
+ "20": 4061010432.0,
+ "21": 4061010432.0,
+ "22": 4061010432.0,
+ "23": 4061010432.0,
+ "24": 4061010432.0,
+ "25": 4061010432.0,
+ "26": 4061010432.0,
+ "27": 4061010432.0,
+ "28": 4061010432.0,
+ "29": 4061010432.0,
+ "30": 4061010432.0,
+ "31": 4061010432.0,
+ "32": 4061010432.0,
+ "33": 4061010432.0,
+ "34": 4061010432.0,
+ "35": 4061010432.0,
+ "36": 4061010432.0,
+ "37": 4061010432.0,
+ "38": 4061010432.0,
+ "39": 4061010432.0,
+ "40": 4061010432.0,
+ "41": 4061010432.0,
+ "42": 4061010432.0,
+ "43": 4061010432.0,
+ "44": 4061010432.0,
+ "45": 4061010432.0,
+ "46": 4061010432.0,
+ "47": 4061010432.0,
+ "48": 4061010432.0,
+ "49": 4061010432.0,
+ "50": 4061010432.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 25.658,
- "2": 0.47954,
- "3": 0.41847,
- "4": 0.33258,
- "5": 0.34351,
- "6": 0.31011,
- "7": 0.31575,
- "8": 0.29238,
- "9": 0.30311,
- "10": 0.34916,
- "11": 0.30925,
- "12": 0.34341,
- "13": 0.28433,
- "14": 0.28892,
- "15": 0.29252,
- "16": 0.2927,
- "17": 0.30297,
- "18": 0.29339,
- "19": 0.2886,
- "20": 0.29686,
- "21": 0.29022,
- "22": 0.65703,
- "23": 0.29161,
- "24": 0.29821,
- "25": 0.29341,
- "26": 0.30856,
- "27": 0.2991,
- "28": 0.29279,
- "29": 0.29852,
- "30": 0.30839,
- "31": 0.29491,
- "32": 0.2896,
- "33": 0.29084,
- "34": 0.32605,
- "35": 0.29205,
- "36": 0.28559,
- "37": 0.29399,
- "38": 0.28264,
- "39": 0.28463,
- "40": 0.28019,
- "41": 0.28893,
- "42": 0.27586,
- "43": 0.28759,
- "44": 0.28318,
- "45": 0.27759,
- "46": 0.27363,
- "47": 0.27776,
- "48": 0.27855,
- "49": 1.02062,
- "50": 0.28168
+ "1": "nan",
+ "2": 11.67698,
+ "3": 0.39155,
+ "4": 0.31689,
+ "5": 0.294,
+ "6": 0.31141,
+ "7": 0.29128,
+ "8": 0.28675,
+ "9": 0.28172,
+ "10": 0.27347,
+ "11": 0.26642,
+ "12": 0.27974,
+ "13": 0.27332,
+ "14": 0.36787,
+ "15": 0.26967,
+ "16": 0.26447,
+ "17": 0.26033,
+ "18": 0.2662,
+ "19": 0.28876,
+ "20": 0.27381,
+ "21": 0.26827,
+ "22": 0.28438,
+ "23": 0.27253,
+ "24": 0.27903,
+ "25": 0.27474,
+ "26": 0.28579,
+ "27": 0.28072,
+ "28": 0.2816,
+ "29": 0.32547,
+ "30": 0.27477,
+ "31": 0.27095,
+ "32": 0.27719,
+ "33": 0.26688,
+ "34": 0.27227,
+ "35": 0.2837,
+ "36": 0.27295,
+ "37": 0.26868,
+ "38": 0.26936,
+ "39": 0.27392,
+ "40": 0.2649,
+ "41": 0.27268,
+ "42": 0.26786,
+ "43": 0.26041,
+ "44": 0.2684,
+ "45": 0.26786,
+ "46": 0.26105,
+ "47": 0.26729,
+ "48": 0.26353,
+ "49": 0.27083,
+ "50": 0.26181
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4/model_config.yaml
index b6a7c223acc..5423ee39527 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -63,4 +63,6 @@ MODEL_ARGS:
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
--exit-interval: 50
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_memory_speed/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_memory_speed/model_config.yaml
index 4c991767ca3..8f108eabdac 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_memory_speed/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_memory_speed/model_config.yaml
@@ -38,9 +38,9 @@ MODEL_ARGS:
# Data args
--seq-length: 4096
--data-cache-path: ${DATA_CACHE_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
# Add network size args
--num-layers: 16
@@ -123,6 +123,8 @@ MODEL_ARGS:
# Add mixed precision args
--bf16: true
--exit-interval: 50
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
METRICS:
- "iteration-time"
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_mtp_resume_torch_dist_fp8/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_mtp_resume_torch_dist_fp8/model_config.yaml
index fdb452c65a9..3f1ce0e8f16 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_mtp_resume_torch_dist_fp8/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_mtp_resume_torch_dist_fp8/model_config.yaml
@@ -38,9 +38,9 @@ MODEL_ARGS:
# Data args
--seq-length: 4096
--data-cache-path: ${DATA_CACHE_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
# Add network size args
--num-layers: 16
@@ -125,6 +125,8 @@ MODEL_ARGS:
--fp8-format: hybrid
--fp8-recipe: tensorwise
--exit-interval: 50
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular # Usually ckpt-resume, but as a WAR to #513 set to regular
METRICS:
- "iteration-time"
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_resume_torch_dist_attn_cudagraph/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_resume_torch_dist_attn_cudagraph/model_config.yaml
index bd565830970..02134fe47c3 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_resume_torch_dist_attn_cudagraph/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_resume_torch_dist_attn_cudagraph/model_config.yaml
@@ -38,9 +38,9 @@ MODEL_ARGS:
# Data args
--seq-length: 4096
--data-cache-path: ${DATA_CACHE_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
# Add network size args
--num-layers: 16
@@ -128,6 +128,8 @@ MODEL_ARGS:
--fp8-format: hybrid
--fp8-recipe: tensorwise
--exit-interval: 50
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular # Usually ckpt-resume, but as a WAR to #513 set to regular
METRICS:
- "iteration-time"
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_selective_recompute_experimental/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_selective_recompute_experimental/model_config.yaml
index efb1fedf93c..172fac96f6f 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_selective_recompute_experimental/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp2_ep4_etp1_selective_recompute_experimental/model_config.yaml
@@ -39,14 +39,14 @@ MODEL_ARGS:
# Data args
--seq-length: 4096
--data-cache-path: ${DATA_CACHE_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
# Add network size args
--num-layers: 16
--moe-layer-freq: ([0]*3+[1]*13)
- --pipeline-model-parallel-layout: Et*3\\|\\(tt\\|\\)*6tmL # Et*3|(tt|)*6tmL
+ --pipeline-model-parallel-layout: Et*3|(tt|)*6tmL
--hidden-size: 1024
--ffn-hidden-size: 4096
--num-attention-heads: 32
@@ -126,6 +126,8 @@ MODEL_ARGS:
--bf16: true
--exit-interval: 50
--overlap-moe-expert-parallel-comm: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular # Usually ckpt-resume, but as a WAR to #513 set to regular
METRICS:
- "iteration-time"
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_zp_z3_resume_torch_dist_te_8experts2parallel_top2router/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_zp_z3_resume_torch_dist_te_8experts2parallel_top2router/golden_values_dev_dgx_gb200.json
index 113a491b0ba..5e07fb3bfad 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_zp_z3_resume_torch_dist_te_8experts2parallel_top2router/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_zp_z3_resume_torch_dist_te_8experts2parallel_top2router/golden_values_dev_dgx_gb200.json
@@ -6,104 +6,104 @@
"values": {
"1": 10.81199,
"2": 10.82649,
- "3": 10.81384,
- "4": 10.79509,
- "5": 10.83534,
- "6": 10.84275,
- "7": 10.83571,
- "8": 10.83439,
- "9": 10.83696,
- "10": 10.78957,
- "11": 10.85974,
- "12": 10.84264,
- "13": 10.84986,
- "14": 10.86378,
- "15": 10.80482,
- "16": 10.79204,
- "17": 10.7636,
- "18": 10.78823,
- "19": 10.78841,
- "20": 10.70796,
- "21": 10.68628,
- "22": 10.53299,
- "23": 10.691,
- "24": 10.58061,
- "25": 10.5289,
- "26": 10.57723,
- "27": 10.58971,
- "28": 10.5643,
- "29": 10.56693,
- "30": 10.35124,
- "31": 10.09414,
- "32": 10.43287,
- "33": 10.43231,
- "34": 10.19673,
- "35": 10.23457,
- "36": 10.19059,
- "37": 10.31658,
- "38": 10.16469,
- "39": 10.37482,
- "40": 10.05031,
- "41": 10.10005,
- "42": 10.1774,
- "43": 9.79407,
- "44": 9.91934,
- "45": 9.7932,
- "46": 9.78104,
- "47": 10.10607,
- "48": 9.8118,
- "49": 9.48096,
- "50": 9.86752,
- "51": 9.8069,
- "52": 9.70296,
- "53": 10.03508,
- "54": 9.92052,
- "55": 9.84588,
- "56": 9.58072,
- "57": 9.43445,
- "58": 9.79856,
- "59": 9.54419,
- "60": 9.45288,
- "61": 9.65801,
- "62": 9.95366,
- "63": 9.34015,
- "64": 9.73433,
- "65": 8.90213,
- "66": 9.6667,
- "67": 9.33687,
- "68": 9.7563,
- "69": 9.77598,
- "70": 9.70281,
+ "3": 10.81402,
+ "4": 10.79444,
+ "5": 10.8356,
+ "6": 10.84311,
+ "7": 10.83557,
+ "8": 10.83498,
+ "9": 10.83668,
+ "10": 10.78964,
+ "11": 10.85912,
+ "12": 10.84339,
+ "13": 10.84997,
+ "14": 10.86414,
+ "15": 10.80576,
+ "16": 10.7918,
+ "17": 10.76394,
+ "18": 10.78766,
+ "19": 10.78774,
+ "20": 10.70812,
+ "21": 10.6864,
+ "22": 10.53307,
+ "23": 10.69044,
+ "24": 10.5809,
+ "25": 10.52886,
+ "26": 10.57744,
+ "27": 10.58939,
+ "28": 10.56471,
+ "29": 10.56607,
+ "30": 10.35103,
+ "31": 10.09367,
+ "32": 10.43199,
+ "33": 10.43216,
+ "34": 10.19633,
+ "35": 10.23455,
+ "36": 10.19036,
+ "37": 10.31682,
+ "38": 10.16475,
+ "39": 10.3741,
+ "40": 10.05088,
+ "41": 10.10003,
+ "42": 10.17734,
+ "43": 9.79377,
+ "44": 9.91897,
+ "45": 9.79315,
+ "46": 9.78119,
+ "47": 10.10601,
+ "48": 9.81175,
+ "49": 9.4813,
+ "50": 9.86738,
+ "51": 9.80706,
+ "52": 9.70288,
+ "53": 10.03514,
+ "54": 9.92065,
+ "55": 9.84605,
+ "56": 9.58055,
+ "57": 9.43481,
+ "58": 9.79877,
+ "59": 9.54386,
+ "60": 9.4523,
+ "61": 9.65803,
+ "62": 9.95373,
+ "63": 9.34019,
+ "64": 9.73453,
+ "65": 8.90212,
+ "66": 9.66653,
+ "67": 9.33709,
+ "68": 9.75619,
+ "69": 9.77579,
+ "70": 9.70272,
"71": 9.60206,
- "72": 9.543,
+ "72": 9.54307,
"73": 9.4557,
- "74": 8.87804,
- "75": 9.37677,
- "76": 9.03816,
- "77": 10.03912,
- "78": 9.69714,
- "79": 9.35195,
- "80": 9.37278,
- "81": 9.45649,
- "82": 9.6802,
- "83": 9.27723,
- "84": 9.39341,
- "85": 9.58928,
- "86": 9.05151,
- "87": 9.57623,
- "88": 9.72869,
- "89": 9.57637,
- "90": 9.80884,
- "91": 9.30719,
- "92": 9.33823,
- "93": 9.05712,
- "94": 8.80375,
- "95": 9.5091,
- "96": 9.50777,
- "97": 9.27751,
- "98": 9.65271,
- "99": 8.87009,
- "100": 9.38142
+ "74": 8.87807,
+ "75": 9.37673,
+ "76": 9.03809,
+ "77": 10.03878,
+ "78": 9.69735,
+ "79": 9.35192,
+ "80": 9.37284,
+ "81": 9.45647,
+ "82": 9.67999,
+ "83": 9.27725,
+ "84": 9.39356,
+ "85": 9.58912,
+ "86": 9.05149,
+ "87": 9.57627,
+ "88": 9.72865,
+ "89": 9.5761,
+ "90": 9.80906,
+ "91": 9.30685,
+ "92": 9.33841,
+ "93": 9.05655,
+ "94": 8.80359,
+ "95": 9.50883,
+ "96": 9.50764,
+ "97": 9.27773,
+ "98": 9.65276,
+ "99": 8.87014,
+ "100": 9.38138
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 26865.0,
- "2": 29306.0,
- "3": 29361.0,
- "4": 28339.0,
- "5": 32501.0,
- "6": 33051.0,
- "7": 31429.0,
- "8": 27274.0,
- "9": 30849.0,
- "10": 25253.0,
- "11": 34123.0,
- "12": 30710.0,
- "13": 33513.0,
- "14": 33611.0,
- "15": 31132.0,
- "16": 32283.0,
- "17": 31523.0,
- "18": 30937.0,
- "19": 31324.0,
- "20": 28686.0,
- "21": 29644.0,
- "22": 27366.0,
- "23": 34392.0,
- "24": 29052.0,
- "25": 27947.0,
- "26": 31335.0,
- "27": 31669.0,
- "28": 33909.0,
- "29": 35204.0,
- "30": 30468.0,
- "31": 27904.0,
- "32": 33358.0,
- "33": 35896.0,
- "34": 30365.0,
- "35": 31692.0,
- "36": 32966.0,
- "37": 35992.0,
- "38": 33308.0,
- "39": 38061.0,
- "40": 34579.0,
- "41": 33534.0,
- "42": 36447.0,
- "43": 32600.0,
- "44": 32178.0,
- "45": 34034.0,
- "46": 34910.0,
- "47": 39009.0,
- "48": 34943.0,
- "49": 34977.0,
- "50": 38519.0,
- "51": 36877.0,
- "52": 36443.0,
- "53": 43145.0,
- "54": 41676.0,
- "55": 38684.0,
- "56": 41454.0,
- "57": 35771.0,
- "58": 41538.0,
- "59": 39697.0,
- "60": 56137.0,
- "61": 59394.0,
- "62": 2137056.0,
- "63": 36401.0,
- "64": 50930.0,
- "65": 43788.0,
- "66": 2139459.0,
- "67": 2137025.0,
- "68": 2137005.0,
- "69": 2139555.0,
- "70": 2140268.0,
- "71": 2138613.0,
- "72": 2139093.0,
- "73": 2141321.0,
- "74": 2137048.0,
- "75": 2136852.0,
- "76": 2140757.0,
- "77": 2140654.0,
- "78": 2141929.0,
- "79": 2142543.0,
- "80": 2142157.0,
- "81": 2145547.0,
- "82": 2144670.0,
- "83": 2140858.0,
- "84": 2140984.0,
- "85": 2145921.0,
- "86": 149825.0,
- "87": 2144700.0,
- "88": 2142479.0,
- "89": 2140988.0,
- "90": 2144684.0,
- "91": 2143848.0,
- "92": 2142027.0,
- "93": 2139531.0,
- "94": 2145775.0,
- "95": 2143141.0,
- "96": 2146259.0,
- "97": 2140268.0,
- "98": 2143316.0,
- "99": 2144369.0,
- "100": 2143057.0
+ "1": 26523.0,
+ "2": 29472.0,
+ "3": 29140.0,
+ "4": 28787.0,
+ "5": 32154.0,
+ "6": 33150.0,
+ "7": 31503.0,
+ "8": 27488.0,
+ "9": 30851.0,
+ "10": 25539.0,
+ "11": 33735.0,
+ "12": 30721.0,
+ "13": 33360.0,
+ "14": 33374.0,
+ "15": 30838.0,
+ "16": 32360.0,
+ "17": 31588.0,
+ "18": 31016.0,
+ "19": 31320.0,
+ "20": 28419.0,
+ "21": 29325.0,
+ "22": 27567.0,
+ "23": 34221.0,
+ "24": 28953.0,
+ "25": 27716.0,
+ "26": 31399.0,
+ "27": 31596.0,
+ "28": 33689.0,
+ "29": 35335.0,
+ "30": 30311.0,
+ "31": 27380.0,
+ "32": 33651.0,
+ "33": 36080.0,
+ "34": 30178.0,
+ "35": 31974.0,
+ "36": 32609.0,
+ "37": 36504.0,
+ "38": 33985.0,
+ "39": 38040.0,
+ "40": 34901.0,
+ "41": 33629.0,
+ "42": 35751.0,
+ "43": 32869.0,
+ "44": 32821.0,
+ "45": 34054.0,
+ "46": 34067.0,
+ "47": 40493.0,
+ "48": 35146.0,
+ "49": 34756.0,
+ "50": 38467.0,
+ "51": 36706.0,
+ "52": 36171.0,
+ "53": 42698.0,
+ "54": 41471.0,
+ "55": 38216.0,
+ "56": 41916.0,
+ "57": 36330.0,
+ "58": 41283.0,
+ "59": 40081.0,
+ "60": 55891.0,
+ "61": 59793.0,
+ "62": 2137191.0,
+ "63": 36446.0,
+ "64": 128493.0,
+ "65": 43769.0,
+ "66": 2139269.0,
+ "67": 2137293.0,
+ "68": 2136798.0,
+ "69": 2139311.0,
+ "70": 2140379.0,
+ "71": 2138932.0,
+ "72": 2138654.0,
+ "73": 2141565.0,
+ "74": 2137087.0,
+ "75": 2137011.0,
+ "76": 2140501.0,
+ "77": 2140898.0,
+ "78": 2142043.0,
+ "79": 2142782.0,
+ "80": 2141568.0,
+ "81": 2145750.0,
+ "82": 2144812.0,
+ "83": 2141262.0,
+ "84": 2140595.0,
+ "85": 2145583.0,
+ "86": 2140562.0,
+ "87": 2144769.0,
+ "88": 2142291.0,
+ "89": 2140641.0,
+ "90": 2144794.0,
+ "91": 2143597.0,
+ "92": 2141696.0,
+ "93": 2139704.0,
+ "94": 2145839.0,
+ "95": 2142840.0,
+ "96": 2145981.0,
+ "97": 2140183.0,
+ "98": 2143585.0,
+ "99": 2143931.0,
+ "100": 2142923.0
}
},
"mem-allocated-bytes": {
@@ -220,104 +220,104 @@
"values": {
"1": 787591680.0,
"2": 787578880.0,
- "3": 787593728.0,
+ "3": 787594240.0,
"4": 787568128.0,
"5": 787563008.0,
"6": 787585536.0,
- "7": 787578368.0,
- "8": 787582976.0,
+ "7": 787578880.0,
+ "8": 787582464.0,
"9": 787581952.0,
- "10": 787592192.0,
+ "10": 787591680.0,
"11": 787569152.0,
- "12": 787570688.0,
+ "12": 787570176.0,
"13": 787579392.0,
- "14": 787582976.0,
- "15": 787565568.0,
- "16": 787572224.0,
- "17": 787566592.0,
- "18": 787547648.0,
- "19": 787566592.0,
- "20": 787537408.0,
+ "14": 787582464.0,
+ "15": 787566592.0,
+ "16": 787572736.0,
+ "17": 787567104.0,
+ "18": 787546624.0,
+ "19": 787567104.0,
+ "20": 787536384.0,
"21": 787540992.0,
- "22": 787540480.0,
- "23": 787548672.0,
- "24": 787542016.0,
- "25": 787534336.0,
+ "22": 787541504.0,
+ "23": 787549696.0,
+ "24": 787540992.0,
+ "25": 787533824.0,
"26": 787548672.0,
- "27": 787509760.0,
- "28": 787504640.0,
- "29": 787499520.0,
- "30": 787494912.0,
- "31": 787510784.0,
- "32": 787501056.0,
+ "27": 787510784.0,
+ "28": 787505152.0,
+ "29": 787500544.0,
+ "30": 787493376.0,
+ "31": 787511808.0,
+ "32": 787501568.0,
"33": 787482624.0,
"34": 787486208.0,
"35": 787483136.0,
- "36": 787482624.0,
- "37": 787460608.0,
- "38": 787457536.0,
- "39": 787461632.0,
- "40": 787457536.0,
- "41": 787466752.0,
- "42": 787432448.0,
- "43": 787450368.0,
+ "36": 787484160.0,
+ "37": 787461120.0,
+ "38": 787457024.0,
+ "39": 787462144.0,
+ "40": 787456512.0,
+ "41": 787467264.0,
+ "42": 787431936.0,
+ "43": 787449856.0,
"44": 787436032.0,
- "45": 787411456.0,
- "46": 787460608.0,
- "47": 787412992.0,
+ "45": 787411968.0,
+ "46": 787460096.0,
+ "47": 787413504.0,
"48": 787440128.0,
- "49": 787409920.0,
+ "49": 787410432.0,
"50": 787396096.0,
- "51": 787388416.0,
- "52": 787415040.0,
- "53": 787377664.0,
- "54": 787403264.0,
+ "51": 787388928.0,
+ "52": 787413504.0,
+ "53": 787377152.0,
+ "54": 787404288.0,
"55": 787375104.0,
"56": 787362304.0,
"57": 787405824.0,
- "58": 787356160.0,
- "59": 787378688.0,
- "60": 787380224.0,
- "61": 787337216.0,
- "62": 787331584.0,
- "63": 787368960.0,
+ "58": 787355648.0,
+ "59": 787378176.0,
+ "60": 787379712.0,
+ "61": 787339264.0,
+ "62": 787331072.0,
+ "63": 787369472.0,
"64": 787339264.0,
"65": 787403776.0,
- "66": 787330048.0,
- "67": 787337728.0,
- "68": 787324416.0,
- "69": 787335680.0,
- "70": 787328512.0,
- "71": 787331584.0,
- "72": 787341312.0,
- "73": 787353088.0,
- "74": 787366400.0,
- "75": 787342848.0,
- "76": 787344384.0,
- "77": 787345920.0,
+ "66": 787329024.0,
+ "67": 787337216.0,
+ "68": 787323904.0,
+ "69": 787335168.0,
+ "70": 787329536.0,
+ "71": 787331072.0,
+ "72": 787341824.0,
+ "73": 787351552.0,
+ "74": 787365376.0,
+ "75": 787343360.0,
+ "76": 787343872.0,
+ "77": 787344896.0,
"78": 787371520.0,
- "79": 787366400.0,
- "80": 787390464.0,
- "81": 787385344.0,
- "82": 787395584.0,
- "83": 787403776.0,
+ "79": 787366912.0,
+ "80": 787387904.0,
+ "81": 787384832.0,
+ "82": 787393536.0,
+ "83": 787403264.0,
"84": 787397632.0,
- "85": 787398144.0,
- "86": 787411968.0,
+ "85": 787397120.0,
+ "86": 787410432.0,
"87": 787389952.0,
- "88": 787387904.0,
- "89": 787400704.0,
- "90": 787379712.0,
- "91": 787401216.0,
- "92": 787399168.0,
- "93": 787391488.0,
- "94": 787392000.0,
+ "88": 787387392.0,
+ "89": 787400192.0,
+ "90": 787379200.0,
+ "91": 787400192.0,
+ "92": 787397632.0,
+ "93": 787390976.0,
+ "94": 787393024.0,
"95": 787398656.0,
- "96": 787395584.0,
+ "96": 787397120.0,
"97": 787403776.0,
- "98": 787396608.0,
- "99": 787406848.0,
- "100": 787410432.0
+ "98": 787398144.0,
+ "99": 787408896.0,
+ "100": 787411968.0
}
},
"mem-max-allocated-bytes": {
@@ -327,104 +327,104 @@
"values": {
"1": 2662647296.0,
"2": 2662647296.0,
- "3": 2665052672.0,
- "4": 2665052672.0,
- "5": 2665052672.0,
- "6": 2665052672.0,
- "7": 2665052672.0,
- "8": 2665052672.0,
- "9": 2665052672.0,
- "10": 2665052672.0,
- "11": 2665052672.0,
- "12": 2665052672.0,
- "13": 2665052672.0,
- "14": 2665052672.0,
- "15": 2665052672.0,
- "16": 2665052672.0,
- "17": 2665052672.0,
- "18": 2665052672.0,
- "19": 2665052672.0,
- "20": 2665052672.0,
- "21": 2665052672.0,
- "22": 2665052672.0,
- "23": 2665052672.0,
- "24": 2665052672.0,
- "25": 2665052672.0,
- "26": 2665052672.0,
- "27": 2665052672.0,
- "28": 2665052672.0,
- "29": 2665052672.0,
- "30": 2665052672.0,
- "31": 2665052672.0,
- "32": 2665052672.0,
- "33": 2665052672.0,
- "34": 2665052672.0,
- "35": 2665052672.0,
- "36": 2665052672.0,
- "37": 2665052672.0,
- "38": 2665052672.0,
- "39": 2665052672.0,
- "40": 2665052672.0,
- "41": 2665052672.0,
- "42": 2665052672.0,
- "43": 2665052672.0,
- "44": 2665052672.0,
- "45": 2665052672.0,
- "46": 2665052672.0,
- "47": 2665052672.0,
- "48": 2665052672.0,
- "49": 2665052672.0,
- "50": 2665052672.0,
- "51": 2665052672.0,
- "52": 2665052672.0,
- "53": 2665052672.0,
- "54": 2665052672.0,
- "55": 2665052672.0,
- "56": 2665052672.0,
- "57": 2665052672.0,
- "58": 2665052672.0,
- "59": 2665052672.0,
- "60": 2665052672.0,
- "61": 2665052672.0,
- "62": 2665052672.0,
- "63": 2665052672.0,
- "64": 2665052672.0,
- "65": 2665052672.0,
- "66": 2665052672.0,
- "67": 2665052672.0,
- "68": 2665052672.0,
- "69": 2665052672.0,
- "70": 2665052672.0,
- "71": 2665052672.0,
- "72": 2665052672.0,
- "73": 2665052672.0,
- "74": 2665052672.0,
- "75": 2665052672.0,
- "76": 2665052672.0,
- "77": 2665052672.0,
- "78": 2665052672.0,
- "79": 2665052672.0,
- "80": 2665052672.0,
- "81": 2665052672.0,
- "82": 2665052672.0,
- "83": 2665052672.0,
- "84": 2665052672.0,
- "85": 2665052672.0,
- "86": 2665052672.0,
- "87": 2665052672.0,
- "88": 2665052672.0,
- "89": 2665052672.0,
- "90": 2665052672.0,
- "91": 2665052672.0,
- "92": 2665052672.0,
- "93": 2665052672.0,
- "94": 2665052672.0,
- "95": 2665052672.0,
- "96": 2665052672.0,
- "97": 2665052672.0,
- "98": 2665052672.0,
- "99": 2665052672.0,
- "100": 2665052672.0
+ "3": 2665063424.0,
+ "4": 2665063424.0,
+ "5": 2665063424.0,
+ "6": 2665063424.0,
+ "7": 2665063424.0,
+ "8": 2665063424.0,
+ "9": 2665063424.0,
+ "10": 2665063424.0,
+ "11": 2665063424.0,
+ "12": 2665063424.0,
+ "13": 2665063424.0,
+ "14": 2665063424.0,
+ "15": 2665063424.0,
+ "16": 2665063424.0,
+ "17": 2665063424.0,
+ "18": 2665063424.0,
+ "19": 2665063424.0,
+ "20": 2665063424.0,
+ "21": 2665063424.0,
+ "22": 2665063424.0,
+ "23": 2665063424.0,
+ "24": 2665063424.0,
+ "25": 2665063424.0,
+ "26": 2665063424.0,
+ "27": 2665063424.0,
+ "28": 2665063424.0,
+ "29": 2665063424.0,
+ "30": 2665063424.0,
+ "31": 2665063424.0,
+ "32": 2665063424.0,
+ "33": 2665063424.0,
+ "34": 2665063424.0,
+ "35": 2665063424.0,
+ "36": 2665063424.0,
+ "37": 2665063424.0,
+ "38": 2665063424.0,
+ "39": 2665063424.0,
+ "40": 2665063424.0,
+ "41": 2665063424.0,
+ "42": 2665063424.0,
+ "43": 2665063424.0,
+ "44": 2665063424.0,
+ "45": 2665063424.0,
+ "46": 2665063424.0,
+ "47": 2665063424.0,
+ "48": 2665063424.0,
+ "49": 2665063424.0,
+ "50": 2665063424.0,
+ "51": 2665063424.0,
+ "52": 2665063424.0,
+ "53": 2665063424.0,
+ "54": 2665063424.0,
+ "55": 2665063424.0,
+ "56": 2665063424.0,
+ "57": 2665063424.0,
+ "58": 2665063424.0,
+ "59": 2665063424.0,
+ "60": 2665063424.0,
+ "61": 2665063424.0,
+ "62": 2665063424.0,
+ "63": 2665063424.0,
+ "64": 2665063424.0,
+ "65": 2665063424.0,
+ "66": 2665063424.0,
+ "67": 2665063424.0,
+ "68": 2665063424.0,
+ "69": 2665063424.0,
+ "70": 2665063424.0,
+ "71": 2665063424.0,
+ "72": 2665063424.0,
+ "73": 2665063424.0,
+ "74": 2665063424.0,
+ "75": 2665063424.0,
+ "76": 2665063424.0,
+ "77": 2665063424.0,
+ "78": 2665063424.0,
+ "79": 2665063424.0,
+ "80": 2665063424.0,
+ "81": 2665063424.0,
+ "82": 2665063424.0,
+ "83": 2665063424.0,
+ "84": 2665063424.0,
+ "85": 2665063424.0,
+ "86": 2665063424.0,
+ "87": 2665063424.0,
+ "88": 2665063424.0,
+ "89": 2665063424.0,
+ "90": 2665063424.0,
+ "91": 2665063424.0,
+ "92": 2665063424.0,
+ "93": 2665063424.0,
+ "94": 2665063424.0,
+ "95": 2665063424.0,
+ "96": 2665063424.0,
+ "97": 2665063424.0,
+ "98": 2665063424.0,
+ "99": 2665063424.0,
+ "100": 2665063424.0
}
},
"iteration-time": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 4.78579,
- "3": 0.53829,
- "4": 0.5501,
- "5": 0.52877,
- "6": 0.53341,
- "7": 0.53101,
- "8": 0.52594,
- "9": 0.52656,
- "10": 0.52721,
- "11": 0.51907,
- "12": 0.52113,
- "13": 0.52417,
- "14": 0.52392,
- "15": 0.53475,
- "16": 0.52116,
- "17": 0.52656,
- "18": 0.52034,
- "19": 0.52016,
- "20": 0.52199,
- "21": 0.53183,
- "22": 0.53661,
- "23": 0.54084,
- "24": 0.52495,
- "25": 0.53128,
- "26": 0.52735,
- "27": 0.54335,
- "28": 0.52654,
- "29": 0.53834,
- "30": 0.53606,
- "31": 0.53938,
- "32": 0.53598,
- "33": 0.53326,
- "34": 0.54444,
- "35": 0.53164,
- "36": 0.5404,
- "37": 0.54568,
- "38": 0.54552,
- "39": 0.5366,
- "40": 0.54027,
- "41": 0.53525,
- "42": 0.55075,
- "43": 0.53886,
- "44": 0.53665,
- "45": 0.55089,
- "46": 0.5331,
- "47": 0.54482,
- "48": 0.53151,
- "49": 0.53493,
- "50": 0.53302,
- "51": 0.52424,
- "52": 0.52434,
- "53": 0.51687,
- "54": 0.52816,
- "55": 0.53022,
- "56": 0.53577,
- "57": 0.53245,
- "58": 0.53568,
- "59": 0.54753,
- "60": 0.53813,
- "61": 0.53815,
- "62": 0.5366,
- "63": 0.54423,
- "64": 0.5344,
- "65": 0.53864,
- "66": 0.54089,
- "67": 0.53579,
- "68": 0.54777,
- "69": 0.54032,
- "70": 0.54348,
- "71": 0.5411,
- "72": 0.54019,
- "73": 0.53851,
- "74": 0.54021,
- "75": 0.53784,
- "76": 0.53954,
- "77": 0.54237,
- "78": 0.53049,
- "79": 0.57915,
- "80": 0.57307,
- "81": 0.56876,
- "82": 0.56781,
- "83": 0.56481,
- "84": 0.55385,
- "85": 0.56577,
- "86": 0.569,
- "87": 0.5621,
- "88": 0.56698,
- "89": 0.55835,
- "90": 0.85395,
- "91": 0.56888,
- "92": 0.55621,
- "93": 0.57143,
- "94": 0.5584,
- "95": 0.56204,
- "96": 0.5656,
- "97": 0.5491,
- "98": 0.56348,
- "99": 0.5607,
- "100": 0.56258
+ "2": 5.98635,
+ "3": 0.52282,
+ "4": 0.49693,
+ "5": 0.4928,
+ "6": 0.48259,
+ "7": 0.4965,
+ "8": 0.47878,
+ "9": 0.47293,
+ "10": 0.47259,
+ "11": 0.47135,
+ "12": 0.47375,
+ "13": 0.46469,
+ "14": 0.4653,
+ "15": 0.47382,
+ "16": 0.48208,
+ "17": 0.47932,
+ "18": 0.46393,
+ "19": 0.46346,
+ "20": 0.47236,
+ "21": 0.4714,
+ "22": 0.47499,
+ "23": 0.47258,
+ "24": 0.46914,
+ "25": 0.47024,
+ "26": 0.46574,
+ "27": 0.47482,
+ "28": 0.47982,
+ "29": 0.48899,
+ "30": 0.49411,
+ "31": 0.48791,
+ "32": 0.48868,
+ "33": 0.48565,
+ "34": 0.48033,
+ "35": 0.48225,
+ "36": 0.47838,
+ "37": 0.48688,
+ "38": 0.48265,
+ "39": 0.48609,
+ "40": 0.48829,
+ "41": 0.48993,
+ "42": 0.49163,
+ "43": 0.48738,
+ "44": 0.48033,
+ "45": 0.48298,
+ "46": 0.49224,
+ "47": 0.47934,
+ "48": 0.48869,
+ "49": 0.47492,
+ "50": 0.47463,
+ "51": 0.49442,
+ "52": 0.4729,
+ "53": 0.47381,
+ "54": 0.47741,
+ "55": 0.48415,
+ "56": 0.48472,
+ "57": 0.49879,
+ "58": 0.48585,
+ "59": 0.49378,
+ "60": 0.49224,
+ "61": 0.48445,
+ "62": 0.47883,
+ "63": 0.48658,
+ "64": 0.48416,
+ "65": 0.47652,
+ "66": 0.47867,
+ "67": 0.5028,
+ "68": 0.48553,
+ "69": 0.48415,
+ "70": 0.47946,
+ "71": 0.47869,
+ "72": 0.47973,
+ "73": 0.48056,
+ "74": 0.48003,
+ "75": 0.48769,
+ "76": 0.4697,
+ "77": 0.47534,
+ "78": 0.46682,
+ "79": 0.47552,
+ "80": 0.47839,
+ "81": 0.48653,
+ "82": 0.48245,
+ "83": 0.48713,
+ "84": 0.4737,
+ "85": 0.47339,
+ "86": 0.47528,
+ "87": 0.48514,
+ "88": 0.47048,
+ "89": 0.47146,
+ "90": 0.81332,
+ "91": 0.4747,
+ "92": 0.47449,
+ "93": 0.47825,
+ "94": 0.47459,
+ "95": 0.47757,
+ "96": 0.47444,
+ "97": 0.46924,
+ "98": 0.47068,
+ "99": 0.47128,
+ "100": 0.47481
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_zp_z3_resume_torch_dist_te_8experts2parallel_top2router/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_zp_z3_resume_torch_dist_te_8experts2parallel_top2router/golden_values_dev_dgx_h100.json
index bfbb1e850e1..2065f295d95 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_zp_z3_resume_torch_dist_te_8experts2parallel_top2router/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_zp_z3_resume_torch_dist_te_8experts2parallel_top2router/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.82922,
- "2": 10.84163,
- "3": 10.84245,
- "4": 10.82,
- "5": 10.85652,
- "6": 10.86906,
- "7": 10.83778,
- "8": 10.84312,
- "9": 10.84423,
- "10": 10.79298,
- "11": 10.86697,
- "12": 10.86875,
- "13": 10.86207,
- "14": 10.86919,
- "15": 10.8067,
- "16": 10.8057,
- "17": 10.77686,
- "18": 10.79541,
- "19": 10.78384,
- "20": 10.72654,
- "21": 10.69491,
- "22": 10.54462,
- "23": 10.6993,
- "24": 10.58151,
- "25": 10.53282,
- "26": 10.58817,
- "27": 10.601,
- "28": 10.57563,
- "29": 10.58022,
- "30": 10.35802,
- "31": 10.08769,
- "32": 10.44466,
- "33": 10.4477,
- "34": 10.18704,
- "35": 10.24483,
- "36": 10.19713,
- "37": 10.32294,
- "38": 10.17101,
- "39": 10.37026,
- "40": 10.05533,
- "41": 10.09491,
- "42": 10.17971,
- "43": 9.78263,
- "44": 9.91346,
- "45": 9.77951,
- "46": 9.75648,
- "47": 10.09647,
- "48": 9.80391,
- "49": 9.46649,
- "50": 9.86874,
- "51": 9.79428,
- "52": 9.68303,
- "53": 10.03314,
- "54": 9.9113,
- "55": 9.82995,
- "56": 9.57839,
- "57": 9.42377,
- "58": 9.80549,
- "59": 9.53292,
- "60": 9.449,
- "61": 9.65293,
- "62": 9.95672,
- "63": 9.33775,
- "64": 9.74194,
- "65": 8.89366,
- "66": 9.67317,
- "67": 9.33002,
- "68": 9.76517,
- "69": 9.76336,
- "70": 9.71127,
- "71": 9.59511,
- "72": 9.54797,
- "73": 9.47124,
- "74": 8.89297,
- "75": 9.39451,
- "76": 9.04721,
- "77": 10.04318,
- "78": 9.70313,
- "79": 9.35169,
- "80": 9.38198,
- "81": 9.45146,
- "82": 9.67546,
- "83": 9.27658,
- "84": 9.39241,
- "85": 9.58333,
- "86": 9.04518,
- "87": 9.56487,
- "88": 9.72459,
- "89": 9.57019,
- "90": 9.79944,
- "91": 9.30737,
- "92": 9.3313,
- "93": 9.04109,
- "94": 8.80259,
- "95": 9.50213,
- "96": 9.5021,
- "97": 9.28183,
- "98": 9.64883,
- "99": 8.8594,
- "100": 9.37131
+ "1": 10.93984,
+ "2": 10.92208,
+ "3": 10.93037,
+ "4": 10.9352,
+ "5": 10.9278,
+ "6": 10.9276,
+ "7": 10.92524,
+ "8": 10.93115,
+ "9": 10.93004,
+ "10": 10.92766,
+ "11": 10.91901,
+ "12": 10.92803,
+ "13": 10.90745,
+ "14": 10.9099,
+ "15": 10.89558,
+ "16": 10.8846,
+ "17": 10.87881,
+ "18": 10.87704,
+ "19": 10.87287,
+ "20": 10.8104,
+ "21": 10.79234,
+ "22": 10.77409,
+ "23": 10.76827,
+ "24": 10.73586,
+ "25": 10.734,
+ "26": 10.7154,
+ "27": 10.68878,
+ "28": 10.61761,
+ "29": 10.5821,
+ "30": 10.55879,
+ "31": 10.54423,
+ "32": 10.52791,
+ "33": 10.48645,
+ "34": 10.45812,
+ "35": 10.45949,
+ "36": 10.43353,
+ "37": 10.40252,
+ "38": 10.40328,
+ "39": 10.37326,
+ "40": 10.35199,
+ "41": 10.32451,
+ "42": 10.31529,
+ "43": 10.29129,
+ "44": 10.26864,
+ "45": 10.2805,
+ "46": 10.23617,
+ "47": 10.22646,
+ "48": 10.1904,
+ "49": 10.18862,
+ "50": 10.1893,
+ "51": 10.18496,
+ "52": 10.13744,
+ "53": 10.1439,
+ "54": 10.11277,
+ "55": 10.09192,
+ "56": 10.1151,
+ "57": 10.10245,
+ "58": 10.11155,
+ "59": 10.06017,
+ "60": 10.08389,
+ "61": 10.03655,
+ "62": 10.00288,
+ "63": 10.07352,
+ "64": 10.02434,
+ "65": 9.99463,
+ "66": 10.03273,
+ "67": 10.00897,
+ "68": 9.97384,
+ "69": 9.98138,
+ "70": 9.96363,
+ "71": 9.99158,
+ "72": 9.96777,
+ "73": 9.96022,
+ "74": 9.95217,
+ "75": 9.9044,
+ "76": 9.94704,
+ "77": 9.94762,
+ "78": 9.89288,
+ "79": 9.88794,
+ "80": 9.90818,
+ "81": 9.92915,
+ "82": 9.88598,
+ "83": 9.83627,
+ "84": 9.78909,
+ "85": 9.76321,
+ "86": 9.88299,
+ "87": 9.89218,
+ "88": 9.87307,
+ "89": 9.81984,
+ "90": 9.81305,
+ "91": 9.81874,
+ "92": 9.80382,
+ "93": 9.74126,
+ "94": 9.81461,
+ "95": 9.80568,
+ "96": 9.79241,
+ "97": 9.73889,
+ "98": 9.76528,
+ "99": 9.81148,
+ "100": 9.69975
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 27245.0,
- "2": 28958.0,
- "3": 29464.0,
- "4": 28046.0,
- "5": 31369.0,
- "6": 33287.0,
- "7": 31200.0,
- "8": 26921.0,
- "9": 30008.0,
- "10": 25870.0,
- "11": 33681.0,
- "12": 30344.0,
- "13": 32737.0,
- "14": 33315.0,
- "15": 29830.0,
- "16": 32475.0,
- "17": 30747.0,
- "18": 30381.0,
- "19": 31032.0,
- "20": 28243.0,
- "21": 29224.0,
- "22": 27340.0,
- "23": 34119.0,
- "24": 29049.0,
- "25": 27636.0,
- "26": 30662.0,
- "27": 32009.0,
- "28": 33355.0,
- "29": 34714.0,
- "30": 30387.0,
- "31": 28212.0,
- "32": 33411.0,
- "33": 34696.0,
- "34": 30053.0,
- "35": 31488.0,
- "36": 32943.0,
- "37": 35829.0,
- "38": 33740.0,
- "39": 37632.0,
- "40": 34779.0,
- "41": 33958.0,
- "42": 36396.0,
- "43": 34088.0,
- "44": 34090.0,
- "45": 35158.0,
- "46": 36174.0,
- "47": 39772.0,
- "48": 36516.0,
- "49": 36733.0,
- "50": 38234.0,
- "51": 38608.0,
- "52": 37030.0,
- "53": 42442.0,
- "54": 40944.0,
- "55": 37133.0,
- "56": 41001.0,
- "57": 37524.0,
- "58": 42317.0,
- "59": 40804.0,
- "60": 40450.0,
- "61": 41478.0,
- "62": 39766.0,
- "63": 37941.0,
- "64": 42197.0,
- "65": 40947.0,
- "66": 44094.0,
- "67": 41958.0,
- "68": 40060.0,
- "69": 42189.0,
- "70": 43436.0,
- "71": 42748.0,
- "72": 44280.0,
- "73": 47478.0,
- "74": 41456.0,
- "75": 39925.0,
- "76": 43490.0,
- "77": 45636.0,
- "78": 2141470.0,
- "79": 46055.0,
- "80": 51863.0,
- "81": 151341.0,
- "82": 49835.0,
- "83": 143360.0,
- "84": 2141546.0,
- "85": 2145177.0,
- "86": 132114.0,
- "87": 2147022.0,
- "88": 59899.0,
- "89": 162883.0,
- "90": 51330.0,
- "91": 2141901.0,
- "92": 44946.0,
- "93": 138194.0,
- "94": 2145772.0,
- "95": 45247.0,
- "96": 135045.0,
- "97": 53170.0,
- "98": 168576.0,
- "99": 2141797.0,
- "100": 163741.0
+ "1": 32290.0,
+ "2": 32206.0,
+ "3": 32568.0,
+ "4": 32429.0,
+ "5": 31781.0,
+ "6": 31950.0,
+ "7": 32860.0,
+ "8": 31816.0,
+ "9": 32801.0,
+ "10": 33288.0,
+ "11": 32443.0,
+ "12": 31532.0,
+ "13": 32773.0,
+ "14": 32823.0,
+ "15": 31610.0,
+ "16": 31534.0,
+ "17": 31799.0,
+ "18": 32808.0,
+ "19": 32272.0,
+ "20": 32467.0,
+ "21": 31900.0,
+ "22": 32813.0,
+ "23": 33007.0,
+ "24": 32944.0,
+ "25": 31918.0,
+ "26": 32699.0,
+ "27": 32939.0,
+ "28": 32573.0,
+ "29": 32904.0,
+ "30": 33064.0,
+ "31": 34207.0,
+ "32": 34748.0,
+ "33": 33826.0,
+ "34": 34171.0,
+ "35": 35392.0,
+ "36": 34707.0,
+ "37": 35389.0,
+ "38": 34919.0,
+ "39": 36711.0,
+ "40": 38102.0,
+ "41": 35865.0,
+ "42": 36168.0,
+ "43": 37646.0,
+ "44": 37042.0,
+ "45": 41280.0,
+ "46": 39410.0,
+ "47": 39175.0,
+ "48": 39959.0,
+ "49": 43657.0,
+ "50": 40565.0,
+ "51": 40263.0,
+ "52": 42960.0,
+ "53": 135456.0,
+ "54": 43085.0,
+ "55": 2135548.0,
+ "56": 42724.0,
+ "57": 36939.0,
+ "58": 57735.0,
+ "59": 42762.0,
+ "60": 131441.0,
+ "61": 41397.0,
+ "62": 64263.0,
+ "63": 140698.0,
+ "64": 49754.0,
+ "65": 40624.0,
+ "66": 48031.0,
+ "67": 2146934.0,
+ "68": 51343.0,
+ "69": 136178.0,
+ "70": 49989.0,
+ "71": 41768.0,
+ "72": 2141114.0,
+ "73": 135138.0,
+ "74": 128836.0,
+ "75": 59964.0,
+ "76": 2142917.0,
+ "77": 153063.0,
+ "78": 55870.0,
+ "79": 2140988.0,
+ "80": 2140728.0,
+ "81": 58866.0,
+ "82": 2140382.0,
+ "83": 41379.0,
+ "84": 42878.0,
+ "85": 2137910.0,
+ "86": 135858.0,
+ "87": 2139041.0,
+ "88": 2141107.0,
+ "89": 43698.0,
+ "90": 126724.0,
+ "91": 2140145.0,
+ "92": 2138706.0,
+ "93": 2139128.0,
+ "94": 2142331.0,
+ "95": 131179.0,
+ "96": 2144436.0,
+ "97": 160182.0,
+ "98": 156459.0,
+ "99": 2143948.0,
+ "100": 149451.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1668119552.0,
- "2": 1668144128.0,
- "3": 1668127232.0,
- "4": 1668115456.0,
- "5": 1668150272.0,
- "6": 1668141056.0,
- "7": 1668115968.0,
- "8": 1668127744.0,
- "9": 1668131328.0,
- "10": 1668108288.0,
- "11": 1668125184.0,
- "12": 1668123136.0,
- "13": 1668132864.0,
- "14": 1668132352.0,
- "15": 1668107264.0,
- "16": 1668116480.0,
- "17": 1668106240.0,
- "18": 1668092416.0,
- "19": 1668118016.0,
- "20": 1668108288.0,
- "21": 1668082688.0,
- "22": 1668089344.0,
- "23": 1668081664.0,
- "24": 1668089344.0,
- "25": 1668054528.0,
- "26": 1668085248.0,
- "27": 1668073984.0,
- "28": 1668053504.0,
- "29": 1668061184.0,
- "30": 1668038144.0,
- "31": 1668009984.0,
- "32": 1668027392.0,
- "33": 1668038656.0,
- "34": 1668029440.0,
- "35": 1668021760.0,
- "36": 1668039680.0,
- "37": 1668031488.0,
- "38": 1668039680.0,
- "39": 1668020736.0,
- "40": 1668018176.0,
- "41": 1668008960.0,
- "42": 1668018176.0,
- "43": 1667971072.0,
- "44": 1667995648.0,
- "45": 1668002816.0,
- "46": 1667958272.0,
- "47": 1668014592.0,
- "48": 1667957248.0,
- "49": 1667977216.0,
- "50": 1667992576.0,
- "51": 1667978752.0,
- "52": 1667986944.0,
- "53": 1667982848.0,
- "54": 1667987968.0,
- "55": 1667992064.0,
- "56": 1667992064.0,
- "57": 1667954176.0,
- "58": 1667985920.0,
- "59": 1667977216.0,
- "60": 1667998208.0,
- "61": 1668008448.0,
- "62": 1668008960.0,
- "63": 1667976192.0,
- "64": 1667992064.0,
- "65": 1667954688.0,
- "66": 1667990016.0,
- "67": 1667995136.0,
- "68": 1668002304.0,
- "69": 1667986432.0,
- "70": 1667996160.0,
- "71": 1668009984.0,
- "72": 1668003840.0,
- "73": 1668004352.0,
- "74": 1668006400.0,
- "75": 1668045824.0,
- "76": 1668047872.0,
- "77": 1668048896.0,
- "78": 1667998208.0,
- "79": 1668033536.0,
- "80": 1668013568.0,
- "81": 1668016128.0,
- "82": 1668030976.0,
- "83": 1668032000.0,
- "84": 1668015616.0,
- "85": 1668015616.0,
- "86": 1667997696.0,
- "87": 1668056064.0,
- "88": 1668017664.0,
- "89": 1668007936.0,
- "90": 1668049920.0,
- "91": 1668049408.0,
- "92": 1668049920.0,
- "93": 1668033536.0,
- "94": 1668043264.0,
- "95": 1668053504.0,
- "96": 1668058112.0,
- "97": 1668030464.0,
- "98": 1668079104.0,
- "99": 1668022272.0,
- "100": 1668041728.0
+ "1": 787530240.0,
+ "2": 787520512.0,
+ "3": 787529216.0,
+ "4": 787524608.0,
+ "5": 787529216.0,
+ "6": 787521536.0,
+ "7": 787535872.0,
+ "8": 787528704.0,
+ "9": 787534848.0,
+ "10": 787528704.0,
+ "11": 787533824.0,
+ "12": 787512832.0,
+ "13": 787518464.0,
+ "14": 787525120.0,
+ "15": 787511808.0,
+ "16": 787499520.0,
+ "17": 787522048.0,
+ "18": 787505152.0,
+ "19": 787523584.0,
+ "20": 787479552.0,
+ "21": 787496960.0,
+ "22": 787477504.0,
+ "23": 787491840.0,
+ "24": 787480576.0,
+ "25": 787476480.0,
+ "26": 787468288.0,
+ "27": 787477504.0,
+ "28": 787453440.0,
+ "29": 787453952.0,
+ "30": 787431936.0,
+ "31": 787456000.0,
+ "32": 787432960.0,
+ "33": 787440128.0,
+ "34": 787438080.0,
+ "35": 787430912.0,
+ "36": 787430912.0,
+ "37": 787439104.0,
+ "38": 787432448.0,
+ "39": 787430400.0,
+ "40": 787416576.0,
+ "41": 787436032.0,
+ "42": 787402752.0,
+ "43": 787414528.0,
+ "44": 787416064.0,
+ "45": 787401728.0,
+ "46": 787391488.0,
+ "47": 787389440.0,
+ "48": 787396096.0,
+ "49": 787383296.0,
+ "50": 787371520.0,
+ "51": 787374592.0,
+ "52": 787382784.0,
+ "53": 787399168.0,
+ "54": 787372032.0,
+ "55": 787371520.0,
+ "56": 787352064.0,
+ "57": 787386368.0,
+ "58": 787374080.0,
+ "59": 787357184.0,
+ "60": 787412480.0,
+ "61": 787359232.0,
+ "62": 787379712.0,
+ "63": 787390464.0,
+ "64": 787407360.0,
+ "65": 787392000.0,
+ "66": 787369472.0,
+ "67": 787394048.0,
+ "68": 787346432.0,
+ "69": 787356672.0,
+ "70": 787395072.0,
+ "71": 787347968.0,
+ "72": 787414528.0,
+ "73": 787433472.0,
+ "74": 787379200.0,
+ "75": 787387392.0,
+ "76": 787378176.0,
+ "77": 787430400.0,
+ "78": 787389440.0,
+ "79": 787376128.0,
+ "80": 787355136.0,
+ "81": 787488256.0,
+ "82": 787421184.0,
+ "83": 787348480.0,
+ "84": 787387904.0,
+ "85": 787452416.0,
+ "86": 787422208.0,
+ "87": 787396096.0,
+ "88": 787361792.0,
+ "89": 787382272.0,
+ "90": 787481088.0,
+ "91": 787394560.0,
+ "92": 787425792.0,
+ "93": 787386368.0,
+ "94": 787418624.0,
+ "95": 787419648.0,
+ "96": 787385344.0,
+ "97": 787439104.0,
+ "98": 787472384.0,
+ "99": 787409920.0,
+ "100": 787469312.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 3460789248.0,
- "2": 3470375936.0,
- "3": 3470375936.0,
- "4": 3470375936.0,
- "5": 3480799232.0,
- "6": 3480799232.0,
- "7": 3480799232.0,
- "8": 3480799232.0,
- "9": 3480799232.0,
- "10": 3480799232.0,
- "11": 3480799232.0,
- "12": 3480799232.0,
- "13": 3480799232.0,
- "14": 3480799232.0,
- "15": 3480799232.0,
- "16": 3480799232.0,
- "17": 3480799232.0,
- "18": 3480799232.0,
- "19": 3480799232.0,
- "20": 3480799232.0,
- "21": 3480799232.0,
- "22": 3480799232.0,
- "23": 3480799232.0,
- "24": 3480799232.0,
- "25": 3480799232.0,
- "26": 3480799232.0,
- "27": 3480799232.0,
- "28": 3480799232.0,
- "29": 3480799232.0,
- "30": 3480799232.0,
- "31": 3480799232.0,
- "32": 3480799232.0,
- "33": 3480799232.0,
- "34": 3480799232.0,
- "35": 3480799232.0,
- "36": 3480799232.0,
- "37": 3480799232.0,
- "38": 3480799232.0,
- "39": 3480799232.0,
- "40": 3480799232.0,
- "41": 3480799232.0,
- "42": 3480799232.0,
- "43": 3480799232.0,
- "44": 3480799232.0,
- "45": 3480799232.0,
- "46": 3480799232.0,
- "47": 3480799232.0,
- "48": 3480799232.0,
- "49": 3480799232.0,
- "50": 3480799232.0,
- "51": 3480799232.0,
- "52": 3480799232.0,
- "53": 3480799232.0,
- "54": 3480799232.0,
- "55": 3480799232.0,
- "56": 3480799232.0,
- "57": 3480799232.0,
- "58": 3480799232.0,
- "59": 3480799232.0,
- "60": 3480799232.0,
- "61": 3480799232.0,
- "62": 3480799232.0,
- "63": 3480799232.0,
- "64": 3480799232.0,
- "65": 3480799232.0,
- "66": 3480799232.0,
- "67": 3480799232.0,
- "68": 3480799232.0,
- "69": 3480799232.0,
- "70": 3480799232.0,
- "71": 3480799232.0,
- "72": 3480799232.0,
- "73": 3480799232.0,
- "74": 3480799232.0,
- "75": 3480799232.0,
- "76": 3480799232.0,
- "77": 3480799232.0,
- "78": 3480799232.0,
- "79": 3480799232.0,
- "80": 3480799232.0,
- "81": 3480799232.0,
- "82": 3480799232.0,
- "83": 3480799232.0,
- "84": 3480799232.0,
- "85": 3480799232.0,
- "86": 3480799232.0,
- "87": 3480799232.0,
- "88": 3480799232.0,
- "89": 3480799232.0,
- "90": 3480799232.0,
- "91": 3480799232.0,
- "92": 3480799232.0,
- "93": 3480799232.0,
- "94": 3480799232.0,
- "95": 3480799232.0,
- "96": 3480799232.0,
- "97": 3480799232.0,
- "98": 3480799232.0,
- "99": 3480799232.0,
- "100": 3480799232.0
+ "1": 2586962432.0,
+ "2": 2586962432.0,
+ "3": 2586962432.0,
+ "4": 2586962432.0,
+ "5": 2586962432.0,
+ "6": 2586962432.0,
+ "7": 2586962432.0,
+ "8": 2587204608.0,
+ "9": 2587204608.0,
+ "10": 2587204608.0,
+ "11": 2587204608.0,
+ "12": 2587204608.0,
+ "13": 2587204608.0,
+ "14": 2587204608.0,
+ "15": 2587204608.0,
+ "16": 2587204608.0,
+ "17": 2587204608.0,
+ "18": 2587204608.0,
+ "19": 2587204608.0,
+ "20": 2587204608.0,
+ "21": 2587204608.0,
+ "22": 2587204608.0,
+ "23": 2587204608.0,
+ "24": 2587204608.0,
+ "25": 2587204608.0,
+ "26": 2587204608.0,
+ "27": 2587204608.0,
+ "28": 2587204608.0,
+ "29": 2587204608.0,
+ "30": 2587204608.0,
+ "31": 2587204608.0,
+ "32": 2587204608.0,
+ "33": 2587204608.0,
+ "34": 2587204608.0,
+ "35": 2587204608.0,
+ "36": 2587204608.0,
+ "37": 2587204608.0,
+ "38": 2587204608.0,
+ "39": 2587204608.0,
+ "40": 2587204608.0,
+ "41": 2587204608.0,
+ "42": 2587204608.0,
+ "43": 2587204608.0,
+ "44": 2587204608.0,
+ "45": 2587204608.0,
+ "46": 2587204608.0,
+ "47": 2587204608.0,
+ "48": 2587204608.0,
+ "49": 2587204608.0,
+ "50": 2587204608.0,
+ "51": 2587204608.0,
+ "52": 2587204608.0,
+ "53": 2587204608.0,
+ "54": 2587204608.0,
+ "55": 2587204608.0,
+ "56": 2587204608.0,
+ "57": 2587204608.0,
+ "58": 2587204608.0,
+ "59": 2587204608.0,
+ "60": 2587204608.0,
+ "61": 2587204608.0,
+ "62": 2587204608.0,
+ "63": 2587204608.0,
+ "64": 2587204608.0,
+ "65": 2587204608.0,
+ "66": 2587204608.0,
+ "67": 2587204608.0,
+ "68": 2587204608.0,
+ "69": 2587204608.0,
+ "70": 2587204608.0,
+ "71": 2587204608.0,
+ "72": 2587204608.0,
+ "73": 2587204608.0,
+ "74": 2587204608.0,
+ "75": 2587204608.0,
+ "76": 2587204608.0,
+ "77": 2587204608.0,
+ "78": 2587204608.0,
+ "79": 2587204608.0,
+ "80": 2587204608.0,
+ "81": 2587204608.0,
+ "82": 2587204608.0,
+ "83": 2587204608.0,
+ "84": 2587204608.0,
+ "85": 2587204608.0,
+ "86": 2587204608.0,
+ "87": 2587204608.0,
+ "88": 2587204608.0,
+ "89": 2587204608.0,
+ "90": 2587204608.0,
+ "91": 2587204608.0,
+ "92": 2587204608.0,
+ "93": 2587204608.0,
+ "94": 2587204608.0,
+ "95": 2587204608.0,
+ "96": 2587204608.0,
+ "97": 2587204608.0,
+ "98": 2587204608.0,
+ "99": 2587204608.0,
+ "100": 2587204608.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 11.49667,
- "2": 0.45982,
- "3": 0.39283,
- "4": 0.37269,
- "5": 0.33438,
- "6": 0.33048,
- "7": 0.33351,
- "8": 0.32704,
- "9": 0.31789,
- "10": 0.30958,
- "11": 0.30791,
- "12": 0.30859,
- "13": 0.32053,
- "14": 0.30171,
- "15": 0.30843,
- "16": 0.30302,
- "17": 0.30464,
- "18": 0.30431,
- "19": 0.30467,
- "20": 0.29614,
- "21": 0.3034,
- "22": 0.30183,
- "23": 0.29505,
- "24": 0.29208,
- "25": 0.29678,
- "26": 0.29737,
- "27": 0.30864,
- "28": 0.31313,
- "29": 0.30795,
- "30": 0.31701,
- "31": 0.31516,
- "32": 0.32758,
- "33": 0.31728,
- "34": 0.32164,
- "35": 0.32366,
- "36": 0.3008,
- "37": 0.30816,
- "38": 0.30782,
- "39": 0.3097,
- "40": 0.31658,
- "41": 0.30749,
- "42": 0.30662,
- "43": 0.30452,
- "44": 0.32171,
- "45": 0.30874,
- "46": 0.31718,
- "47": 0.30947,
- "48": 0.30568,
- "49": 0.30559,
- "50": 0.30518,
- "51": 0.32349,
- "52": 0.30552,
- "53": 0.2972,
- "54": 0.29675,
- "55": 0.6806,
- "56": 0.30449,
- "57": 0.30268,
- "58": 0.29449,
- "59": 0.29915,
- "60": 0.30558,
- "61": 0.29817,
- "62": 0.29837,
- "63": 0.29648,
- "64": 0.30355,
- "65": 0.30526,
- "66": 0.29685,
- "67": 0.29607,
- "68": 0.30383,
- "69": 0.29497,
- "70": 0.29908,
- "71": 0.298,
- "72": 0.29482,
- "73": 0.29392,
- "74": 0.29933,
- "75": 0.29938,
- "76": 0.29472,
- "77": 0.29225,
- "78": 0.29345,
- "79": 0.29571,
- "80": 0.29379,
- "81": 0.29694,
- "82": 0.29442,
- "83": 0.29839,
- "84": 0.30064,
- "85": 0.29571,
- "86": 0.30107,
- "87": 0.29723,
- "88": 0.29324,
- "89": 0.29688,
- "90": 0.29142,
- "91": 0.29759,
- "92": 0.29347,
- "93": 0.29617,
- "94": 0.29996,
- "95": 0.29791,
- "96": 0.29236,
- "97": 0.29637,
- "98": 0.29446,
- "99": 0.293,
- "100": 0.2937
+ "1": "nan",
+ "2": 7.06508,
+ "3": 0.40657,
+ "4": 0.35024,
+ "5": 0.33876,
+ "6": 0.32872,
+ "7": 0.32215,
+ "8": 0.3196,
+ "9": 0.30928,
+ "10": 0.30478,
+ "11": 0.3081,
+ "12": 0.30809,
+ "13": 0.31112,
+ "14": 0.30616,
+ "15": 0.30369,
+ "16": 0.29613,
+ "17": 0.30022,
+ "18": 0.30557,
+ "19": 0.30486,
+ "20": 0.307,
+ "21": 0.30713,
+ "22": 0.31711,
+ "23": 0.30324,
+ "24": 0.30658,
+ "25": 0.30548,
+ "26": 0.31077,
+ "27": 0.31385,
+ "28": 0.32054,
+ "29": 0.32109,
+ "30": 0.31647,
+ "31": 0.31016,
+ "32": 0.31142,
+ "33": 0.30823,
+ "34": 0.3126,
+ "35": 0.30834,
+ "36": 0.31191,
+ "37": 0.30883,
+ "38": 0.31056,
+ "39": 0.31268,
+ "40": 0.32821,
+ "41": 0.30389,
+ "42": 0.31585,
+ "43": 0.31151,
+ "44": 0.30818,
+ "45": 0.31029,
+ "46": 0.30198,
+ "47": 0.30176,
+ "48": 0.30661,
+ "49": 0.30378,
+ "50": 0.3026,
+ "51": 0.42679,
+ "52": 0.30362,
+ "53": 0.3039,
+ "54": 0.30284,
+ "55": 0.303,
+ "56": 0.30304,
+ "57": 0.30011,
+ "58": 0.30448,
+ "59": 0.3019,
+ "60": 0.30022,
+ "61": 0.3002,
+ "62": 0.30261,
+ "63": 0.30563,
+ "64": 0.30229,
+ "65": 0.30182,
+ "66": 0.30139,
+ "67": 0.30259,
+ "68": 0.29876,
+ "69": 0.30156,
+ "70": 0.30014,
+ "71": 0.30172,
+ "72": 0.29867,
+ "73": 0.3002,
+ "74": 0.30253,
+ "75": 0.30489,
+ "76": 0.30413,
+ "77": 0.30304,
+ "78": 0.30052,
+ "79": 0.3002,
+ "80": 0.30062,
+ "81": 0.30261,
+ "82": 0.30211,
+ "83": 0.30514,
+ "84": 0.30389,
+ "85": 0.30428,
+ "86": 0.29812,
+ "87": 0.30117,
+ "88": 0.29859,
+ "89": 0.30058,
+ "90": 0.30098,
+ "91": 0.29704,
+ "92": 0.29741,
+ "93": 0.30347,
+ "94": 0.29753,
+ "95": 0.29754,
+ "96": 0.29865,
+ "97": 0.29859,
+ "98": 0.29981,
+ "99": 0.30032,
+ "100": 0.69865
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_zp_z3_resume_torch_dist_te_8experts2parallel_top2router/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_zp_z3_resume_torch_dist_te_8experts2parallel_top2router/model_config.yaml
index 8874f9cf045..15f971e9ff3 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_zp_z3_resume_torch_dist_te_8experts2parallel_top2router/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_zp_z3_resume_torch_dist_te_8experts2parallel_top2router/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_a100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_a100.json
index dffbbf25de6..37f74714dfc 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_a100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_a100.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.82207,
"2": 10.84178,
- "3": 10.81126,
- "4": 10.82219,
- "5": 10.8455,
- "6": 10.86291,
- "7": 10.84399,
- "8": 10.84652,
- "9": 10.84916,
- "10": 10.78879,
- "11": 10.8581,
- "12": 10.84415,
- "13": 10.87153,
- "14": 10.87463,
- "15": 10.83396,
- "16": 10.8091,
- "17": 10.79098,
- "18": 10.81032,
- "19": 10.80535,
- "20": 10.73557,
- "21": 10.71472,
- "22": 10.57762,
- "23": 10.72594,
- "24": 10.61811,
- "25": 10.58114,
- "26": 10.63747,
- "27": 10.63794,
- "28": 10.60614,
- "29": 10.61062,
- "30": 10.40965,
- "31": 10.16941,
- "32": 10.49897,
+ "3": 10.81113,
+ "4": 10.82234,
+ "5": 10.84489,
+ "6": 10.86304,
+ "7": 10.84427,
+ "8": 10.84693,
+ "9": 10.85001,
+ "10": 10.7897,
+ "11": 10.85766,
+ "12": 10.84396,
+ "13": 10.87181,
+ "14": 10.87455,
+ "15": 10.83399,
+ "16": 10.80919,
+ "17": 10.791,
+ "18": 10.81017,
+ "19": 10.80568,
+ "20": 10.73563,
+ "21": 10.71417,
+ "22": 10.57691,
+ "23": 10.72597,
+ "24": 10.61823,
+ "25": 10.58154,
+ "26": 10.63745,
+ "27": 10.63717,
+ "28": 10.60574,
+ "29": 10.61026,
+ "30": 10.40985,
+ "31": 10.16959,
+ "32": 10.49887,
"33": 10.49702,
- "34": 10.26142,
- "35": 10.31452,
- "36": 10.2851,
- "37": 10.3895,
- "38": 10.2473,
+ "34": 10.26149,
+ "35": 10.31439,
+ "36": 10.28489,
+ "37": 10.38874,
+ "38": 10.24737,
"39": 10.43792,
- "40": 10.14599,
- "41": 10.19691,
- "42": 10.26122,
- "43": 9.91082,
- "44": 10.02318,
- "45": 9.91674,
- "46": 9.89463,
- "47": 10.19281,
- "48": 9.93104,
- "49": 9.61208,
- "50": 9.97427
+ "40": 10.14606,
+ "41": 10.19685,
+ "42": 10.26102,
+ "43": 9.91027,
+ "44": 10.02323,
+ "45": 9.91719,
+ "46": 9.89453,
+ "47": 10.19329,
+ "48": 9.93042,
+ "49": 9.61243,
+ "50": 9.97437
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 4986.0,
- "2": 5272.0,
- "3": 5309.0,
- "4": 5162.0,
- "5": 5824.0,
- "6": 5990.0,
- "7": 5433.0,
- "8": 5101.0,
- "9": 5654.0,
- "10": 4736.0,
- "11": 6213.0,
- "12": 5723.0,
- "13": 5952.0,
- "14": 6073.0,
- "15": 5503.0,
- "16": 5808.0,
- "17": 5545.0,
- "18": 5647.0,
- "19": 5555.0,
- "20": 5120.0,
- "21": 5578.0,
- "22": 5097.0,
- "23": 5992.0,
- "24": 5204.0,
- "25": 5016.0,
- "26": 5487.0,
- "27": 5618.0,
- "28": 5994.0,
- "29": 6202.0,
- "30": 5538.0,
- "31": 4762.0,
- "32": 6010.0,
- "33": 6302.0,
- "34": 5312.0,
- "35": 5783.0,
- "36": 5716.0,
- "37": 6562.0,
- "38": 6183.0,
- "39": 6964.0,
- "40": 6220.0,
- "41": 6139.0,
- "42": 6368.0,
- "43": 5900.0,
- "44": 5754.0,
- "45": 5814.0,
- "46": 5882.0,
- "47": 6818.0,
- "48": 6495.0,
- "49": 6047.0,
- "50": 6623.0
+ "1": 5051.0,
+ "2": 5315.0,
+ "3": 5393.0,
+ "4": 5144.0,
+ "5": 5971.0,
+ "6": 5897.0,
+ "7": 5465.0,
+ "8": 5099.0,
+ "9": 5504.0,
+ "10": 4793.0,
+ "11": 6028.0,
+ "12": 5837.0,
+ "13": 6020.0,
+ "14": 5895.0,
+ "15": 5534.0,
+ "16": 5710.0,
+ "17": 5651.0,
+ "18": 5450.0,
+ "19": 5649.0,
+ "20": 5103.0,
+ "21": 5625.0,
+ "22": 5074.0,
+ "23": 6044.0,
+ "24": 5183.0,
+ "25": 4901.0,
+ "26": 5454.0,
+ "27": 5628.0,
+ "28": 6018.0,
+ "29": 6058.0,
+ "30": 5571.0,
+ "31": 4731.0,
+ "32": 5867.0,
+ "33": 6453.0,
+ "34": 5334.0,
+ "35": 5685.0,
+ "36": 5687.0,
+ "37": 6801.0,
+ "38": 6247.0,
+ "39": 6933.0,
+ "40": 6105.0,
+ "41": 6091.0,
+ "42": 6453.0,
+ "43": 5778.0,
+ "44": 5946.0,
+ "45": 5869.0,
+ "46": 6171.0,
+ "47": 6709.0,
+ "48": 6336.0,
+ "49": 6103.0,
+ "50": 6672.0
}
},
"mem-allocated-bytes": {
@@ -124,47 +124,47 @@
"4": 598360576.0,
"5": 598358016.0,
"6": 598358016.0,
- "7": 598354432.0,
- "8": 598359040.0,
- "9": 598358016.0,
+ "7": 598355456.0,
+ "8": 598360576.0,
+ "9": 598356992.0,
"10": 598356992.0,
"11": 598358016.0,
"12": 598358016.0,
"13": 598359040.0,
"14": 598359040.0,
"15": 598359040.0,
- "16": 598358528.0,
+ "16": 598360576.0,
"17": 598352384.0,
"18": 598358016.0,
"19": 598359040.0,
"20": 598357504.0,
- "21": 598358016.0,
+ "21": 598359040.0,
"22": 598354432.0,
"23": 598355968.0,
- "24": 598356480.0,
+ "24": 598357504.0,
"25": 598358528.0,
- "26": 598357504.0,
+ "26": 598356480.0,
"27": 598360064.0,
"28": 598358016.0,
- "29": 598356480.0,
+ "29": 598357504.0,
"30": 598359552.0,
"31": 598354944.0,
- "32": 598356992.0,
+ "32": 598355968.0,
"33": 598359552.0,
- "34": 598358016.0,
- "35": 598356480.0,
- "36": 598356992.0,
- "37": 598358016.0,
+ "34": 598356992.0,
+ "35": 598355968.0,
+ "36": 598357504.0,
+ "37": 598359040.0,
"38": 598358016.0,
- "39": 598357504.0,
+ "39": 598356992.0,
"40": 598357504.0,
- "41": 598352384.0,
+ "41": 598351872.0,
"42": 598357504.0,
"43": 598352384.0,
"44": 598355456.0,
"45": 598355968.0,
"46": 598351872.0,
- "47": 598359040.0,
+ "47": 598359552.0,
"48": 598354944.0,
"49": 598353408.0,
"50": 598356992.0
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 855598080.0,
- "2": 1083234304.0,
- "3": 1083234304.0,
- "4": 1083234304.0,
- "5": 1083234304.0,
- "6": 1083493888.0,
- "7": 1083493888.0,
- "8": 1083493888.0,
- "9": 1083493888.0,
- "10": 1083493888.0,
- "11": 1083493888.0,
- "12": 1083493888.0,
- "13": 1083493888.0,
- "14": 1084195840.0,
- "15": 1084195840.0,
- "16": 1084195840.0,
- "17": 1084195840.0,
- "18": 1084195840.0,
- "19": 1084195840.0,
- "20": 1084195840.0,
- "21": 1084195840.0,
- "22": 1084195840.0,
- "23": 1084195840.0,
- "24": 1084195840.0,
- "25": 1084195840.0,
- "26": 1084195840.0,
- "27": 1084195840.0,
- "28": 1084195840.0,
- "29": 1084195840.0,
- "30": 1084195840.0,
- "31": 1084195840.0,
- "32": 1084195840.0,
- "33": 1084195840.0,
- "34": 1084195840.0,
- "35": 1084195840.0,
- "36": 1084195840.0,
- "37": 1084195840.0,
- "38": 1084195840.0,
- "39": 1084195840.0,
- "40": 1084195840.0,
- "41": 1084195840.0,
- "42": 1084195840.0,
- "43": 1084195840.0,
- "44": 1084195840.0,
- "45": 1084195840.0,
- "46": 1084195840.0,
- "47": 1084195840.0,
- "48": 1084195840.0,
- "49": 1084195840.0,
- "50": 1084195840.0
+ "1": 849854464.0,
+ "2": 1083133952.0,
+ "3": 1083133952.0,
+ "4": 1083459584.0,
+ "5": 1083459584.0,
+ "6": 1084528128.0,
+ "7": 1084528128.0,
+ "8": 1084734976.0,
+ "9": 1084734976.0,
+ "10": 1084734976.0,
+ "11": 1084734976.0,
+ "12": 1084734976.0,
+ "13": 1084734976.0,
+ "14": 1084734976.0,
+ "15": 1084734976.0,
+ "16": 1084734976.0,
+ "17": 1084734976.0,
+ "18": 1084734976.0,
+ "19": 1084734976.0,
+ "20": 1084734976.0,
+ "21": 1084734976.0,
+ "22": 1084734976.0,
+ "23": 1084734976.0,
+ "24": 1084734976.0,
+ "25": 1084734976.0,
+ "26": 1084734976.0,
+ "27": 1084734976.0,
+ "28": 1084734976.0,
+ "29": 1084734976.0,
+ "30": 1084734976.0,
+ "31": 1084734976.0,
+ "32": 1084734976.0,
+ "33": 1084734976.0,
+ "34": 1084734976.0,
+ "35": 1084734976.0,
+ "36": 1084734976.0,
+ "37": 1084734976.0,
+ "38": 1084734976.0,
+ "39": 1084734976.0,
+ "40": 1084734976.0,
+ "41": 1084734976.0,
+ "42": 1084734976.0,
+ "43": 1084734976.0,
+ "44": 1084734976.0,
+ "45": 1084734976.0,
+ "46": 1084734976.0,
+ "47": 1084734976.0,
+ "48": 1084734976.0,
+ "49": 1084734976.0,
+ "50": 1084734976.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 12.15002,
- "2": 0.70236,
- "3": 0.6774,
- "4": 0.6698,
- "5": 0.66613,
- "6": 0.65685,
- "7": 0.65852,
- "8": 1.19123,
- "9": 0.65621,
- "10": 1.09603,
- "11": 0.65688,
- "12": 0.65983,
- "13": 0.6521,
- "14": 0.65135,
- "15": 0.65551,
- "16": 0.64995,
- "17": 0.6532,
- "18": 0.65306,
- "19": 0.65221,
- "20": 0.65239,
- "21": 0.65356,
- "22": 0.6536,
- "23": 0.65416,
- "24": 0.65298,
- "25": 0.65469,
- "26": 0.65391,
- "27": 0.65289,
- "28": 1.1109,
- "29": 0.65365,
- "30": 0.65326,
- "31": 0.68599,
- "32": 0.65366,
- "33": 0.65416,
- "34": 0.6538,
- "35": 0.65304,
- "36": 0.65351,
- "37": 0.65423,
- "38": 0.6542,
- "39": 0.65254,
- "40": 0.65386,
- "41": 0.65384,
- "42": 0.65434,
- "43": 0.65537,
- "44": 0.65573,
- "45": 0.65342,
- "46": 0.65451,
- "47": 0.6535,
- "48": 0.65377,
- "49": 0.65522,
- "50": 0.65221
+ "1": "nan",
+ "2": 10.29335,
+ "3": 0.67878,
+ "4": 0.6592,
+ "5": 0.65824,
+ "6": 0.65977,
+ "7": 0.65877,
+ "8": 0.66094,
+ "9": 0.65786,
+ "10": 0.65658,
+ "11": 0.6579,
+ "12": 0.65517,
+ "13": 0.65666,
+ "14": 0.65608,
+ "15": 0.65601,
+ "16": 0.65482,
+ "17": 0.65822,
+ "18": 0.65711,
+ "19": 0.65603,
+ "20": 0.65609,
+ "21": 0.6565,
+ "22": 0.67194,
+ "23": 0.65625,
+ "24": 0.66529,
+ "25": 0.66578,
+ "26": 0.67383,
+ "27": 0.65687,
+ "28": 0.65467,
+ "29": 0.65676,
+ "30": 0.65789,
+ "31": 0.65676,
+ "32": 0.65719,
+ "33": 0.65687,
+ "34": 0.65653,
+ "35": 0.65849,
+ "36": 0.65693,
+ "37": 0.65595,
+ "38": 0.6557,
+ "39": 0.65719,
+ "40": 0.65771,
+ "41": 0.65694,
+ "42": 0.6571,
+ "43": 0.65669,
+ "44": 0.65684,
+ "45": 0.65629,
+ "46": 0.65907,
+ "47": 0.65645,
+ "48": 0.6576,
+ "49": 0.65594,
+ "50": 0.65339
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_gb200.json
index bfea64b8438..ce15ba8d8b8 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_gb200.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.79193,
"2": 10.81245,
- "3": 10.79181,
- "4": 10.78209,
- "5": 10.82295,
- "6": 10.83309,
- "7": 10.81351,
- "8": 10.81215,
- "9": 10.81457,
- "10": 10.76068,
- "11": 10.84185,
- "12": 10.82404,
- "13": 10.83895,
- "14": 10.84433,
- "15": 10.79974,
- "16": 10.78654,
- "17": 10.76789,
- "18": 10.77495,
- "19": 10.77669,
- "20": 10.71893,
- "21": 10.69691,
+ "3": 10.7918,
+ "4": 10.78248,
+ "5": 10.8229,
+ "6": 10.83325,
+ "7": 10.81381,
+ "8": 10.8124,
+ "9": 10.81419,
+ "10": 10.76095,
+ "11": 10.84109,
+ "12": 10.82369,
+ "13": 10.83874,
+ "14": 10.84415,
+ "15": 10.79967,
+ "16": 10.78628,
+ "17": 10.76773,
+ "18": 10.77429,
+ "19": 10.77699,
+ "20": 10.71877,
+ "21": 10.69736,
"22": 10.5691,
- "23": 10.7131,
- "24": 10.59975,
- "25": 10.56123,
- "26": 10.60735,
- "27": 10.63093,
- "28": 10.6064,
- "29": 10.61213,
- "30": 10.39823,
- "31": 10.16422,
- "32": 10.49019,
- "33": 10.48385,
- "34": 10.26645,
- "35": 10.31743,
- "36": 10.28264,
- "37": 10.39002,
- "38": 10.25116,
- "39": 10.43811,
- "40": 10.1403,
- "41": 10.19191,
- "42": 10.25886,
- "43": 9.91588,
- "44": 10.02837,
- "45": 9.91815,
- "46": 9.89353,
- "47": 10.20144,
- "48": 9.92509,
- "49": 9.62973,
- "50": 9.97857
+ "23": 10.71318,
+ "24": 10.59944,
+ "25": 10.56048,
+ "26": 10.60755,
+ "27": 10.63083,
+ "28": 10.60651,
+ "29": 10.61165,
+ "30": 10.3982,
+ "31": 10.16412,
+ "32": 10.49049,
+ "33": 10.4843,
+ "34": 10.2663,
+ "35": 10.31711,
+ "36": 10.28281,
+ "37": 10.39018,
+ "38": 10.25141,
+ "39": 10.43825,
+ "40": 10.14073,
+ "41": 10.19205,
+ "42": 10.25794,
+ "43": 9.91532,
+ "44": 10.0282,
+ "45": 9.91826,
+ "46": 9.89373,
+ "47": 10.20145,
+ "48": 9.9251,
+ "49": 9.63015,
+ "50": 9.979
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 5192.0,
- "2": 5510.0,
- "3": 5508.0,
- "4": 5240.0,
- "5": 6136.0,
- "6": 6180.0,
- "7": 5549.0,
- "8": 5242.0,
- "9": 5717.0,
- "10": 4818.0,
- "11": 6299.0,
- "12": 5746.0,
- "13": 6110.0,
- "14": 6165.0,
- "15": 5683.0,
- "16": 5805.0,
- "17": 5758.0,
- "18": 5546.0,
- "19": 5787.0,
- "20": 5231.0,
- "21": 5741.0,
- "22": 5126.0,
- "23": 6019.0,
- "24": 5410.0,
- "25": 5100.0,
- "26": 5630.0,
- "27": 5627.0,
- "28": 6146.0,
- "29": 6174.0,
- "30": 5570.0,
- "31": 4768.0,
- "32": 5926.0,
- "33": 6348.0,
- "34": 5389.0,
- "35": 5856.0,
- "36": 5741.0,
- "37": 6611.0,
- "38": 6262.0,
- "39": 6971.0,
- "40": 6094.0,
- "41": 6227.0,
- "42": 6622.0,
- "43": 5761.0,
- "44": 5929.0,
- "45": 5769.0,
- "46": 6141.0,
- "47": 6909.0,
- "48": 6650.0,
- "49": 6100.0,
- "50": 6753.0
+ "1": 5164.0,
+ "2": 5460.0,
+ "3": 5476.0,
+ "4": 5164.0,
+ "5": 5919.0,
+ "6": 6081.0,
+ "7": 5512.0,
+ "8": 5296.0,
+ "9": 5667.0,
+ "10": 4778.0,
+ "11": 6210.0,
+ "12": 5685.0,
+ "13": 6050.0,
+ "14": 6114.0,
+ "15": 5595.0,
+ "16": 5966.0,
+ "17": 5549.0,
+ "18": 5693.0,
+ "19": 5735.0,
+ "20": 5229.0,
+ "21": 5764.0,
+ "22": 4962.0,
+ "23": 5984.0,
+ "24": 5373.0,
+ "25": 5199.0,
+ "26": 5592.0,
+ "27": 5735.0,
+ "28": 6127.0,
+ "29": 6298.0,
+ "30": 5597.0,
+ "31": 4765.0,
+ "32": 5954.0,
+ "33": 6464.0,
+ "34": 5371.0,
+ "35": 5671.0,
+ "36": 5796.0,
+ "37": 6657.0,
+ "38": 6074.0,
+ "39": 6859.0,
+ "40": 6040.0,
+ "41": 6237.0,
+ "42": 6570.0,
+ "43": 5866.0,
+ "44": 5828.0,
+ "45": 5998.0,
+ "46": 5969.0,
+ "47": 6792.0,
+ "48": 6699.0,
+ "49": 6241.0,
+ "50": 6831.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 627716608.0,
- "2": 627719168.0,
- "3": 627717632.0,
- "4": 627719680.0,
- "5": 627717120.0,
- "6": 627717120.0,
- "7": 627719680.0,
- "8": 627716608.0,
- "9": 627718144.0,
- "10": 627718144.0,
- "11": 627717632.0,
- "12": 627718144.0,
- "13": 627719168.0,
- "14": 627718144.0,
- "15": 627722240.0,
- "16": 627718144.0,
- "17": 627720704.0,
- "18": 627719680.0,
- "19": 627719168.0,
- "20": 627718144.0,
- "21": 627718656.0,
- "22": 627723264.0,
- "23": 627720192.0,
- "24": 627719680.0,
- "25": 627718144.0,
- "26": 627719168.0,
- "27": 627719168.0,
- "28": 627718144.0,
- "29": 627718144.0,
- "30": 627719168.0,
- "31": 627719168.0,
- "32": 627719168.0,
- "33": 627717632.0,
- "34": 627719680.0,
- "35": 627721216.0,
- "36": 627717120.0,
- "37": 627719168.0,
- "38": 627721216.0,
- "39": 627719168.0,
- "40": 627718656.0,
- "41": 627718144.0,
- "42": 627717632.0,
- "43": 627717120.0,
- "44": 627718656.0,
- "45": 627717632.0,
- "46": 627717120.0,
- "47": 627719168.0,
- "48": 627718144.0,
- "49": 627716608.0,
- "50": 627716096.0
+ "1": 628503040.0,
+ "2": 628505600.0,
+ "3": 628504064.0,
+ "4": 628506112.0,
+ "5": 628504576.0,
+ "6": 628503552.0,
+ "7": 628507136.0,
+ "8": 628503040.0,
+ "9": 628504576.0,
+ "10": 628503552.0,
+ "11": 628505088.0,
+ "12": 628504576.0,
+ "13": 628505600.0,
+ "14": 628504576.0,
+ "15": 628508672.0,
+ "16": 628503552.0,
+ "17": 628507136.0,
+ "18": 628506112.0,
+ "19": 628504576.0,
+ "20": 628505600.0,
+ "21": 628506112.0,
+ "22": 628509696.0,
+ "23": 628506624.0,
+ "24": 628506112.0,
+ "25": 628505600.0,
+ "26": 628504576.0,
+ "27": 628505600.0,
+ "28": 628504576.0,
+ "29": 628504576.0,
+ "30": 628504064.0,
+ "31": 628506624.0,
+ "32": 628505600.0,
+ "33": 628504064.0,
+ "34": 628506624.0,
+ "35": 628508160.0,
+ "36": 628503552.0,
+ "37": 628504576.0,
+ "38": 628506112.0,
+ "39": 628504576.0,
+ "40": 628505088.0,
+ "41": 628505088.0,
+ "42": 628504576.0,
+ "43": 628503040.0,
+ "44": 628504576.0,
+ "45": 628503040.0,
+ "46": 628503552.0,
+ "47": 628504576.0,
+ "48": 628504576.0,
+ "49": 628503040.0,
+ "50": 628502528.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 879803392.0,
- "2": 1114769920.0,
- "3": 1114769920.0,
- "4": 1116293632.0,
- "5": 1116293632.0,
- "6": 1116293632.0,
- "7": 1116293632.0,
- "8": 1116293632.0,
- "9": 1116293632.0,
- "10": 1116293632.0,
- "11": 1116293632.0,
- "12": 1116293632.0,
- "13": 1116293632.0,
- "14": 1116293632.0,
- "15": 1116293632.0,
- "16": 1116293632.0,
- "17": 1116293632.0,
- "18": 1116293632.0,
- "19": 1116293632.0,
- "20": 1116293632.0,
- "21": 1116293632.0,
- "22": 1116293632.0,
- "23": 1116293632.0,
- "24": 1116293632.0,
- "25": 1116293632.0,
- "26": 1116293632.0,
- "27": 1116293632.0,
- "28": 1116293632.0,
- "29": 1116293632.0,
- "30": 1116293632.0,
- "31": 1116293632.0,
- "32": 1116293632.0,
- "33": 1116293632.0,
- "34": 1116293632.0,
- "35": 1116293632.0,
- "36": 1116293632.0,
- "37": 1116293632.0,
- "38": 1116293632.0,
- "39": 1116293632.0,
- "40": 1116293632.0,
- "41": 1116293632.0,
- "42": 1116293632.0,
- "43": 1116293632.0,
- "44": 1116293632.0,
- "45": 1116293632.0,
- "46": 1116293632.0,
- "47": 1116293632.0,
- "48": 1116293632.0,
- "49": 1116293632.0,
- "50": 1116293632.0
+ "1": 882678784.0,
+ "2": 1114974720.0,
+ "3": 1114974720.0,
+ "4": 1116527104.0,
+ "5": 1116527104.0,
+ "6": 1116527104.0,
+ "7": 1116527104.0,
+ "8": 1116527104.0,
+ "9": 1116527104.0,
+ "10": 1116527104.0,
+ "11": 1116527104.0,
+ "12": 1116527104.0,
+ "13": 1116527104.0,
+ "14": 1116527104.0,
+ "15": 1116527104.0,
+ "16": 1116527104.0,
+ "17": 1116527104.0,
+ "18": 1116527104.0,
+ "19": 1116527104.0,
+ "20": 1116527104.0,
+ "21": 1116527104.0,
+ "22": 1116527104.0,
+ "23": 1116527104.0,
+ "24": 1116527104.0,
+ "25": 1116527104.0,
+ "26": 1116527104.0,
+ "27": 1116527104.0,
+ "28": 1116527104.0,
+ "29": 1116527104.0,
+ "30": 1116527104.0,
+ "31": 1116527104.0,
+ "32": 1116527104.0,
+ "33": 1116527104.0,
+ "34": 1116527104.0,
+ "35": 1116527104.0,
+ "36": 1116527104.0,
+ "37": 1116527104.0,
+ "38": 1116527104.0,
+ "39": 1116527104.0,
+ "40": 1116527104.0,
+ "41": 1116527104.0,
+ "42": 1116527104.0,
+ "43": 1116527104.0,
+ "44": 1116527104.0,
+ "45": 1116527104.0,
+ "46": 1116527104.0,
+ "47": 1116527104.0,
+ "48": 1116527104.0,
+ "49": 1116527104.0,
+ "50": 1116527104.0
}
},
"iteration-time": {
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 7.71846,
- "3": 0.76188,
- "4": 0.74577,
- "5": 0.73403,
- "6": 0.73193,
- "7": 0.73107,
- "8": 0.72199,
- "9": 0.726,
- "10": 0.71891,
- "11": 0.72723,
- "12": 0.71504,
- "13": 0.71448,
- "14": 0.71551,
- "15": 0.71936,
- "16": 0.71512,
- "17": 0.73948,
- "18": 0.83787,
- "19": 0.94178,
- "20": 0.98096,
- "21": 0.71399,
- "22": 0.87302,
- "23": 0.71359,
- "24": 0.7104,
- "25": 0.70807,
- "26": 0.71636,
- "27": 0.70864,
- "28": 0.72237,
- "29": 0.7163,
- "30": 0.7153,
- "31": 0.71793,
- "32": 0.70846,
- "33": 0.7079,
- "34": 0.71058,
- "35": 0.71492,
- "36": 0.72031,
- "37": 0.71537,
- "38": 0.70333,
- "39": 0.70449,
- "40": 0.71725,
- "41": 0.72322,
- "42": 0.7105,
- "43": 0.70421,
- "44": 0.70441,
- "45": 0.70449,
- "46": 0.7091,
- "47": 0.70989,
- "48": 0.70781,
- "49": 0.71985,
- "50": 0.70534
+ "2": 12.4857,
+ "3": 0.83608,
+ "4": 0.96022,
+ "5": 0.85369,
+ "6": 0.81142,
+ "7": 0.81711,
+ "8": 0.81096,
+ "9": 0.81865,
+ "10": 0.80421,
+ "11": 0.81494,
+ "12": 0.81722,
+ "13": 0.81082,
+ "14": 0.80883,
+ "15": 0.81671,
+ "16": 0.80836,
+ "17": 0.80711,
+ "18": 0.80172,
+ "19": 0.80702,
+ "20": 0.80318,
+ "21": 0.80759,
+ "22": 0.80555,
+ "23": 0.80545,
+ "24": 0.80349,
+ "25": 0.80504,
+ "26": 0.80694,
+ "27": 0.80341,
+ "28": 0.80371,
+ "29": 0.81135,
+ "30": 0.81323,
+ "31": 0.82006,
+ "32": 0.84418,
+ "33": 0.80637,
+ "34": 0.80425,
+ "35": 0.80548,
+ "36": 0.80173,
+ "37": 0.80569,
+ "38": 0.80918,
+ "39": 0.81029,
+ "40": 0.80714,
+ "41": 0.81129,
+ "42": 0.80866,
+ "43": 0.81108,
+ "44": 0.80864,
+ "45": 0.8052,
+ "46": 0.81737,
+ "47": 0.81206,
+ "48": 0.81149,
+ "49": 0.82012,
+ "50": 0.80486
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel/model_config.yaml
index c147b689e71..901cb22f005 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -57,4 +57,6 @@ MODEL_ARGS:
--disable-bias-linear: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel_dp_last/golden_values_dev_dgx_a100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel_dp_last/golden_values_dev_dgx_a100.json
index d2a07cdf1dd..cf306109a97 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel_dp_last/golden_values_dev_dgx_a100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel_dp_last/golden_values_dev_dgx_a100.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.82207,
"2": 10.84178,
- "3": 10.81126,
- "4": 10.82219,
- "5": 10.8455,
- "6": 10.86291,
- "7": 10.84399,
- "8": 10.84652,
- "9": 10.84916,
- "10": 10.78879,
- "11": 10.8581,
- "12": 10.84415,
- "13": 10.87153,
- "14": 10.87463,
- "15": 10.83396,
- "16": 10.8091,
- "17": 10.79098,
- "18": 10.81032,
- "19": 10.80535,
- "20": 10.73557,
- "21": 10.71472,
- "22": 10.57762,
- "23": 10.72594,
- "24": 10.61811,
- "25": 10.58114,
- "26": 10.63747,
- "27": 10.63794,
- "28": 10.60614,
- "29": 10.61062,
- "30": 10.40965,
- "31": 10.16941,
- "32": 10.49897,
+ "3": 10.81113,
+ "4": 10.82234,
+ "5": 10.84489,
+ "6": 10.86304,
+ "7": 10.84427,
+ "8": 10.84693,
+ "9": 10.85001,
+ "10": 10.7897,
+ "11": 10.85766,
+ "12": 10.84396,
+ "13": 10.87181,
+ "14": 10.87455,
+ "15": 10.83399,
+ "16": 10.80919,
+ "17": 10.791,
+ "18": 10.81017,
+ "19": 10.80568,
+ "20": 10.73563,
+ "21": 10.71417,
+ "22": 10.57691,
+ "23": 10.72597,
+ "24": 10.61823,
+ "25": 10.58154,
+ "26": 10.63745,
+ "27": 10.63717,
+ "28": 10.60574,
+ "29": 10.61026,
+ "30": 10.40985,
+ "31": 10.16959,
+ "32": 10.49887,
"33": 10.49702,
- "34": 10.26142,
- "35": 10.31452,
- "36": 10.2851,
- "37": 10.3895,
- "38": 10.2473,
+ "34": 10.26149,
+ "35": 10.31439,
+ "36": 10.28489,
+ "37": 10.38874,
+ "38": 10.24737,
"39": 10.43792,
- "40": 10.14599,
- "41": 10.19691,
- "42": 10.26122,
- "43": 9.91082,
- "44": 10.02318,
- "45": 9.91674,
- "46": 9.89463,
- "47": 10.19281,
- "48": 9.93104,
- "49": 9.61208,
- "50": 9.97427
+ "40": 10.14606,
+ "41": 10.19685,
+ "42": 10.26102,
+ "43": 9.91027,
+ "44": 10.02323,
+ "45": 9.91719,
+ "46": 9.89453,
+ "47": 10.19329,
+ "48": 9.93042,
+ "49": 9.61243,
+ "50": 9.97437
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 4986.0,
- "2": 5272.0,
- "3": 5309.0,
- "4": 5162.0,
- "5": 5824.0,
- "6": 5990.0,
- "7": 5433.0,
- "8": 5101.0,
- "9": 5654.0,
- "10": 4736.0,
- "11": 6213.0,
- "12": 5723.0,
- "13": 5952.0,
- "14": 6073.0,
- "15": 5503.0,
- "16": 5808.0,
- "17": 5545.0,
- "18": 5647.0,
- "19": 5555.0,
- "20": 5120.0,
- "21": 5578.0,
- "22": 5097.0,
- "23": 5992.0,
- "24": 5204.0,
- "25": 5016.0,
- "26": 5487.0,
- "27": 5618.0,
- "28": 5994.0,
- "29": 6202.0,
- "30": 5538.0,
- "31": 4762.0,
- "32": 6010.0,
- "33": 6302.0,
- "34": 5312.0,
- "35": 5783.0,
- "36": 5716.0,
- "37": 6562.0,
- "38": 6183.0,
- "39": 6964.0,
- "40": 6220.0,
- "41": 6139.0,
- "42": 6368.0,
- "43": 5900.0,
- "44": 5754.0,
- "45": 5814.0,
- "46": 5882.0,
- "47": 6818.0,
- "48": 6495.0,
- "49": 6047.0,
- "50": 6623.0
+ "1": 5051.0,
+ "2": 5315.0,
+ "3": 5393.0,
+ "4": 5144.0,
+ "5": 5971.0,
+ "6": 5897.0,
+ "7": 5465.0,
+ "8": 5099.0,
+ "9": 5504.0,
+ "10": 4793.0,
+ "11": 6028.0,
+ "12": 5837.0,
+ "13": 6020.0,
+ "14": 5895.0,
+ "15": 5534.0,
+ "16": 5710.0,
+ "17": 5651.0,
+ "18": 5450.0,
+ "19": 5649.0,
+ "20": 5103.0,
+ "21": 5625.0,
+ "22": 5074.0,
+ "23": 6044.0,
+ "24": 5183.0,
+ "25": 4901.0,
+ "26": 5454.0,
+ "27": 5628.0,
+ "28": 6018.0,
+ "29": 6058.0,
+ "30": 5571.0,
+ "31": 4731.0,
+ "32": 5867.0,
+ "33": 6453.0,
+ "34": 5334.0,
+ "35": 5685.0,
+ "36": 5687.0,
+ "37": 6801.0,
+ "38": 6247.0,
+ "39": 6933.0,
+ "40": 6105.0,
+ "41": 6091.0,
+ "42": 6453.0,
+ "43": 5778.0,
+ "44": 5946.0,
+ "45": 5869.0,
+ "46": 6171.0,
+ "47": 6709.0,
+ "48": 6336.0,
+ "49": 6103.0,
+ "50": 6672.0
}
},
"mem-allocated-bytes": {
@@ -124,47 +124,47 @@
"4": 598360576.0,
"5": 598358016.0,
"6": 598358016.0,
- "7": 598354432.0,
- "8": 598359040.0,
- "9": 598358016.0,
+ "7": 598355456.0,
+ "8": 598360576.0,
+ "9": 598356992.0,
"10": 598356992.0,
"11": 598358016.0,
"12": 598358016.0,
"13": 598359040.0,
"14": 598359040.0,
"15": 598359040.0,
- "16": 598358528.0,
+ "16": 598360576.0,
"17": 598352384.0,
"18": 598358016.0,
"19": 598359040.0,
"20": 598357504.0,
- "21": 598358016.0,
+ "21": 598359040.0,
"22": 598354432.0,
"23": 598355968.0,
- "24": 598356480.0,
+ "24": 598357504.0,
"25": 598358528.0,
- "26": 598357504.0,
+ "26": 598356480.0,
"27": 598360064.0,
"28": 598358016.0,
- "29": 598356480.0,
+ "29": 598357504.0,
"30": 598359552.0,
"31": 598354944.0,
- "32": 598356992.0,
+ "32": 598355968.0,
"33": 598359552.0,
- "34": 598358016.0,
- "35": 598356480.0,
- "36": 598356992.0,
- "37": 598358016.0,
+ "34": 598356992.0,
+ "35": 598355968.0,
+ "36": 598357504.0,
+ "37": 598359040.0,
"38": 598358016.0,
- "39": 598357504.0,
+ "39": 598356992.0,
"40": 598357504.0,
- "41": 598352384.0,
+ "41": 598351872.0,
"42": 598357504.0,
"43": 598352384.0,
"44": 598355456.0,
"45": 598355968.0,
"46": 598351872.0,
- "47": 598359040.0,
+ "47": 598359552.0,
"48": 598354944.0,
"49": 598353408.0,
"50": 598356992.0
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 855598080.0,
- "2": 1083234304.0,
- "3": 1083234304.0,
- "4": 1083234304.0,
- "5": 1083234304.0,
- "6": 1083493888.0,
- "7": 1083493888.0,
- "8": 1083493888.0,
- "9": 1083493888.0,
- "10": 1083493888.0,
- "11": 1083493888.0,
- "12": 1083493888.0,
- "13": 1083493888.0,
- "14": 1084195840.0,
- "15": 1084195840.0,
- "16": 1084195840.0,
- "17": 1084195840.0,
- "18": 1084195840.0,
- "19": 1084195840.0,
- "20": 1084195840.0,
- "21": 1084195840.0,
- "22": 1084195840.0,
- "23": 1084195840.0,
- "24": 1084195840.0,
- "25": 1084195840.0,
- "26": 1084195840.0,
- "27": 1084195840.0,
- "28": 1084195840.0,
- "29": 1084195840.0,
- "30": 1084195840.0,
- "31": 1084195840.0,
- "32": 1084195840.0,
- "33": 1084195840.0,
- "34": 1084195840.0,
- "35": 1084195840.0,
- "36": 1084195840.0,
- "37": 1084195840.0,
- "38": 1084195840.0,
- "39": 1084195840.0,
- "40": 1084195840.0,
- "41": 1084195840.0,
- "42": 1084195840.0,
- "43": 1084195840.0,
- "44": 1084195840.0,
- "45": 1084195840.0,
- "46": 1084195840.0,
- "47": 1084195840.0,
- "48": 1084195840.0,
- "49": 1084195840.0,
- "50": 1084195840.0
+ "1": 849854464.0,
+ "2": 1083133952.0,
+ "3": 1083133952.0,
+ "4": 1083459584.0,
+ "5": 1083459584.0,
+ "6": 1084528128.0,
+ "7": 1084528128.0,
+ "8": 1084734976.0,
+ "9": 1084734976.0,
+ "10": 1084734976.0,
+ "11": 1084734976.0,
+ "12": 1084734976.0,
+ "13": 1084734976.0,
+ "14": 1084734976.0,
+ "15": 1084734976.0,
+ "16": 1084734976.0,
+ "17": 1084734976.0,
+ "18": 1084734976.0,
+ "19": 1084734976.0,
+ "20": 1084734976.0,
+ "21": 1084734976.0,
+ "22": 1084734976.0,
+ "23": 1084734976.0,
+ "24": 1084734976.0,
+ "25": 1084734976.0,
+ "26": 1084734976.0,
+ "27": 1084734976.0,
+ "28": 1084734976.0,
+ "29": 1084734976.0,
+ "30": 1084734976.0,
+ "31": 1084734976.0,
+ "32": 1084734976.0,
+ "33": 1084734976.0,
+ "34": 1084734976.0,
+ "35": 1084734976.0,
+ "36": 1084734976.0,
+ "37": 1084734976.0,
+ "38": 1084734976.0,
+ "39": 1084734976.0,
+ "40": 1084734976.0,
+ "41": 1084734976.0,
+ "42": 1084734976.0,
+ "43": 1084734976.0,
+ "44": 1084734976.0,
+ "45": 1084734976.0,
+ "46": 1084734976.0,
+ "47": 1084734976.0,
+ "48": 1084734976.0,
+ "49": 1084734976.0,
+ "50": 1084734976.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 12.18178,
- "2": 0.71018,
- "3": 0.6513,
- "4": 0.63757,
- "5": 0.63692,
- "6": 1.25031,
- "7": 0.63769,
- "8": 0.6385,
- "9": 1.00487,
- "10": 0.63706,
- "11": 0.63646,
- "12": 0.63826,
- "13": 0.63654,
- "14": 0.63609,
- "15": 0.64,
- "16": 0.6373,
- "17": 0.63737,
- "18": 0.63625,
- "19": 0.63624,
- "20": 0.63844,
- "21": 0.6361,
- "22": 0.63788,
- "23": 0.63738,
- "24": 0.63546,
- "25": 0.63758,
- "26": 0.63704,
- "27": 0.63992,
- "28": 0.64468,
- "29": 0.64456,
- "30": 0.6501,
- "31": 0.64571,
- "32": 0.64554,
- "33": 0.64543,
- "34": 0.64396,
- "35": 0.64389,
- "36": 0.64513,
- "37": 0.6451,
- "38": 0.64723,
- "39": 0.6454,
- "40": 0.64512,
- "41": 0.64629,
- "42": 0.64576,
- "43": 0.64737,
- "44": 0.64709,
- "45": 0.64517,
- "46": 0.64605,
- "47": 0.64625,
- "48": 0.64627,
- "49": 0.64638,
- "50": 0.64367
+ "1": "nan",
+ "2": 7.45654,
+ "3": 0.66799,
+ "4": 0.85068,
+ "5": 0.77572,
+ "6": 0.68201,
+ "7": 0.67921,
+ "8": 0.67951,
+ "9": 0.72589,
+ "10": 0.68668,
+ "11": 0.6775,
+ "12": 0.67297,
+ "13": 0.68537,
+ "14": 0.67587,
+ "15": 0.66706,
+ "16": 0.66844,
+ "17": 0.72815,
+ "18": 0.71789,
+ "19": 0.68666,
+ "20": 0.66516,
+ "21": 0.66541,
+ "22": 0.66565,
+ "23": 0.6614,
+ "24": 0.6618,
+ "25": 0.66037,
+ "26": 0.66246,
+ "27": 0.63822,
+ "28": 0.6393,
+ "29": 0.63682,
+ "30": 0.63816,
+ "31": 0.63919,
+ "32": 0.63822,
+ "33": 0.64356,
+ "34": 0.64371,
+ "35": 0.6378,
+ "36": 0.63816,
+ "37": 0.63762,
+ "38": 0.63747,
+ "39": 0.63722,
+ "40": 0.6399,
+ "41": 0.63856,
+ "42": 0.63914,
+ "43": 0.63893,
+ "44": 0.63818,
+ "45": 0.63902,
+ "46": 0.63953,
+ "47": 0.64,
+ "48": 0.63873,
+ "49": 0.63892,
+ "50": 0.63814
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel_dp_last/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel_dp_last/golden_values_dev_dgx_gb200.json
index 2bcdb30bc50..27a9727334c 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel_dp_last/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel_dp_last/golden_values_dev_dgx_gb200.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.79193,
"2": 10.81245,
- "3": 10.79181,
- "4": 10.78209,
- "5": 10.82295,
- "6": 10.83309,
- "7": 10.81351,
- "8": 10.81215,
- "9": 10.81457,
- "10": 10.76068,
- "11": 10.84185,
- "12": 10.82404,
- "13": 10.83895,
- "14": 10.84433,
- "15": 10.79974,
- "16": 10.78654,
- "17": 10.76789,
- "18": 10.77495,
- "19": 10.77669,
- "20": 10.71893,
- "21": 10.69691,
+ "3": 10.7918,
+ "4": 10.78248,
+ "5": 10.8229,
+ "6": 10.83325,
+ "7": 10.81381,
+ "8": 10.8124,
+ "9": 10.81419,
+ "10": 10.76095,
+ "11": 10.84109,
+ "12": 10.82369,
+ "13": 10.83874,
+ "14": 10.84415,
+ "15": 10.79967,
+ "16": 10.78628,
+ "17": 10.76773,
+ "18": 10.77429,
+ "19": 10.77699,
+ "20": 10.71877,
+ "21": 10.69736,
"22": 10.5691,
- "23": 10.7131,
- "24": 10.59975,
- "25": 10.56123,
- "26": 10.60735,
- "27": 10.63093,
- "28": 10.6064,
- "29": 10.61213,
- "30": 10.39823,
- "31": 10.16422,
- "32": 10.49019,
- "33": 10.48385,
- "34": 10.26645,
- "35": 10.31743,
- "36": 10.28264,
- "37": 10.39002,
- "38": 10.25116,
- "39": 10.43811,
- "40": 10.1403,
- "41": 10.19191,
- "42": 10.25886,
- "43": 9.91588,
- "44": 10.02837,
- "45": 9.91815,
- "46": 9.89353,
- "47": 10.20144,
- "48": 9.92509,
- "49": 9.62973,
- "50": 9.97857
+ "23": 10.71318,
+ "24": 10.59944,
+ "25": 10.56048,
+ "26": 10.60755,
+ "27": 10.63083,
+ "28": 10.60651,
+ "29": 10.61165,
+ "30": 10.3982,
+ "31": 10.16412,
+ "32": 10.49049,
+ "33": 10.4843,
+ "34": 10.2663,
+ "35": 10.31711,
+ "36": 10.28281,
+ "37": 10.39018,
+ "38": 10.25141,
+ "39": 10.43825,
+ "40": 10.14073,
+ "41": 10.19205,
+ "42": 10.25794,
+ "43": 9.91532,
+ "44": 10.0282,
+ "45": 9.91826,
+ "46": 9.89373,
+ "47": 10.20145,
+ "48": 9.9251,
+ "49": 9.63015,
+ "50": 9.979
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 5192.0,
- "2": 5510.0,
- "3": 5508.0,
- "4": 5240.0,
- "5": 6136.0,
- "6": 6180.0,
- "7": 5549.0,
- "8": 5242.0,
- "9": 5717.0,
- "10": 4818.0,
- "11": 6299.0,
- "12": 5746.0,
- "13": 6110.0,
- "14": 6165.0,
- "15": 5683.0,
- "16": 5805.0,
- "17": 5758.0,
- "18": 5546.0,
- "19": 5787.0,
- "20": 5231.0,
- "21": 5741.0,
- "22": 5126.0,
- "23": 6019.0,
- "24": 5410.0,
- "25": 5100.0,
- "26": 5630.0,
- "27": 5627.0,
- "28": 6146.0,
- "29": 6174.0,
- "30": 5570.0,
- "31": 4768.0,
- "32": 5926.0,
- "33": 6348.0,
- "34": 5389.0,
- "35": 5856.0,
- "36": 5741.0,
- "37": 6611.0,
- "38": 6262.0,
- "39": 6971.0,
- "40": 6094.0,
- "41": 6227.0,
- "42": 6622.0,
- "43": 5761.0,
- "44": 5929.0,
- "45": 5769.0,
- "46": 6141.0,
- "47": 6909.0,
- "48": 6650.0,
- "49": 6100.0,
- "50": 6753.0
+ "1": 5164.0,
+ "2": 5460.0,
+ "3": 5476.0,
+ "4": 5164.0,
+ "5": 5919.0,
+ "6": 6081.0,
+ "7": 5512.0,
+ "8": 5296.0,
+ "9": 5667.0,
+ "10": 4778.0,
+ "11": 6210.0,
+ "12": 5685.0,
+ "13": 6050.0,
+ "14": 6114.0,
+ "15": 5595.0,
+ "16": 5966.0,
+ "17": 5549.0,
+ "18": 5693.0,
+ "19": 5735.0,
+ "20": 5229.0,
+ "21": 5764.0,
+ "22": 4962.0,
+ "23": 5984.0,
+ "24": 5373.0,
+ "25": 5199.0,
+ "26": 5592.0,
+ "27": 5735.0,
+ "28": 6127.0,
+ "29": 6298.0,
+ "30": 5597.0,
+ "31": 4765.0,
+ "32": 5954.0,
+ "33": 6464.0,
+ "34": 5371.0,
+ "35": 5671.0,
+ "36": 5796.0,
+ "37": 6657.0,
+ "38": 6074.0,
+ "39": 6859.0,
+ "40": 6040.0,
+ "41": 6237.0,
+ "42": 6570.0,
+ "43": 5866.0,
+ "44": 5828.0,
+ "45": 5998.0,
+ "46": 5969.0,
+ "47": 6792.0,
+ "48": 6699.0,
+ "49": 6241.0,
+ "50": 6831.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 627716608.0,
- "2": 627719168.0,
- "3": 627717632.0,
- "4": 627719680.0,
- "5": 627717120.0,
- "6": 627717120.0,
- "7": 627719680.0,
- "8": 627716608.0,
- "9": 627718144.0,
- "10": 627718144.0,
- "11": 627717632.0,
- "12": 627718144.0,
- "13": 627719168.0,
- "14": 627718144.0,
- "15": 627722240.0,
- "16": 627718144.0,
- "17": 627720704.0,
- "18": 627719680.0,
- "19": 627719168.0,
- "20": 627718144.0,
- "21": 627718656.0,
- "22": 627723264.0,
- "23": 627720192.0,
- "24": 627719680.0,
- "25": 627718144.0,
- "26": 627719168.0,
- "27": 627719168.0,
- "28": 627718144.0,
- "29": 627718144.0,
- "30": 627719168.0,
- "31": 627719168.0,
- "32": 627719168.0,
- "33": 627717632.0,
- "34": 627719680.0,
- "35": 627721216.0,
- "36": 627717120.0,
- "37": 627719168.0,
- "38": 627721216.0,
- "39": 627719168.0,
- "40": 627718656.0,
- "41": 627718144.0,
- "42": 627717632.0,
- "43": 627717120.0,
- "44": 627718656.0,
- "45": 627717632.0,
- "46": 627717120.0,
- "47": 627719168.0,
- "48": 627718144.0,
- "49": 627716608.0,
- "50": 627716096.0
+ "1": 628503040.0,
+ "2": 628505600.0,
+ "3": 628504064.0,
+ "4": 628506112.0,
+ "5": 628504576.0,
+ "6": 628503552.0,
+ "7": 628507136.0,
+ "8": 628503040.0,
+ "9": 628504576.0,
+ "10": 628503552.0,
+ "11": 628505088.0,
+ "12": 628504576.0,
+ "13": 628505600.0,
+ "14": 628504576.0,
+ "15": 628508672.0,
+ "16": 628503552.0,
+ "17": 628507136.0,
+ "18": 628506112.0,
+ "19": 628504576.0,
+ "20": 628505600.0,
+ "21": 628506112.0,
+ "22": 628509696.0,
+ "23": 628506624.0,
+ "24": 628506112.0,
+ "25": 628505600.0,
+ "26": 628504576.0,
+ "27": 628505600.0,
+ "28": 628504576.0,
+ "29": 628504576.0,
+ "30": 628504064.0,
+ "31": 628506624.0,
+ "32": 628505600.0,
+ "33": 628504064.0,
+ "34": 628506624.0,
+ "35": 628508160.0,
+ "36": 628503552.0,
+ "37": 628504576.0,
+ "38": 628506112.0,
+ "39": 628504576.0,
+ "40": 628505088.0,
+ "41": 628505088.0,
+ "42": 628504576.0,
+ "43": 628503040.0,
+ "44": 628504576.0,
+ "45": 628503040.0,
+ "46": 628503552.0,
+ "47": 628504576.0,
+ "48": 628504576.0,
+ "49": 628503040.0,
+ "50": 628502528.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 879803392.0,
- "2": 1114769920.0,
- "3": 1114769920.0,
- "4": 1116293632.0,
- "5": 1116293632.0,
- "6": 1116293632.0,
- "7": 1116293632.0,
- "8": 1116293632.0,
- "9": 1116293632.0,
- "10": 1116293632.0,
- "11": 1116293632.0,
- "12": 1116293632.0,
- "13": 1116293632.0,
- "14": 1116293632.0,
- "15": 1116293632.0,
- "16": 1116293632.0,
- "17": 1116293632.0,
- "18": 1116293632.0,
- "19": 1116293632.0,
- "20": 1116293632.0,
- "21": 1116293632.0,
- "22": 1116293632.0,
- "23": 1116293632.0,
- "24": 1116293632.0,
- "25": 1116293632.0,
- "26": 1116293632.0,
- "27": 1116293632.0,
- "28": 1116293632.0,
- "29": 1116293632.0,
- "30": 1116293632.0,
- "31": 1116293632.0,
- "32": 1116293632.0,
- "33": 1116293632.0,
- "34": 1116293632.0,
- "35": 1116293632.0,
- "36": 1116293632.0,
- "37": 1116293632.0,
- "38": 1116293632.0,
- "39": 1116293632.0,
- "40": 1116293632.0,
- "41": 1116293632.0,
- "42": 1116293632.0,
- "43": 1116293632.0,
- "44": 1116293632.0,
- "45": 1116293632.0,
- "46": 1116293632.0,
- "47": 1116293632.0,
- "48": 1116293632.0,
- "49": 1116293632.0,
- "50": 1116293632.0
+ "1": 882678784.0,
+ "2": 1114974720.0,
+ "3": 1114974720.0,
+ "4": 1116527104.0,
+ "5": 1116527104.0,
+ "6": 1116527104.0,
+ "7": 1116527104.0,
+ "8": 1116527104.0,
+ "9": 1116527104.0,
+ "10": 1116527104.0,
+ "11": 1116527104.0,
+ "12": 1116527104.0,
+ "13": 1116527104.0,
+ "14": 1116527104.0,
+ "15": 1116527104.0,
+ "16": 1116527104.0,
+ "17": 1116527104.0,
+ "18": 1116527104.0,
+ "19": 1116527104.0,
+ "20": 1116527104.0,
+ "21": 1116527104.0,
+ "22": 1116527104.0,
+ "23": 1116527104.0,
+ "24": 1116527104.0,
+ "25": 1116527104.0,
+ "26": 1116527104.0,
+ "27": 1116527104.0,
+ "28": 1116527104.0,
+ "29": 1116527104.0,
+ "30": 1116527104.0,
+ "31": 1116527104.0,
+ "32": 1116527104.0,
+ "33": 1116527104.0,
+ "34": 1116527104.0,
+ "35": 1116527104.0,
+ "36": 1116527104.0,
+ "37": 1116527104.0,
+ "38": 1116527104.0,
+ "39": 1116527104.0,
+ "40": 1116527104.0,
+ "41": 1116527104.0,
+ "42": 1116527104.0,
+ "43": 1116527104.0,
+ "44": 1116527104.0,
+ "45": 1116527104.0,
+ "46": 1116527104.0,
+ "47": 1116527104.0,
+ "48": 1116527104.0,
+ "49": 1116527104.0,
+ "50": 1116527104.0
}
},
"iteration-time": {
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 7.52257,
- "3": 0.74502,
- "4": 0.74089,
- "5": 0.73009,
- "6": 0.73041,
- "7": 0.73704,
- "8": 0.71933,
- "9": 0.72466,
- "10": 1.0546,
- "11": 0.71525,
- "12": 0.71298,
- "13": 0.71412,
- "14": 0.71521,
- "15": 0.71883,
- "16": 0.71464,
- "17": 0.72192,
- "18": 1.32991,
- "19": 0.92083,
- "20": 0.72233,
- "21": 0.71533,
- "22": 0.7144,
- "23": 0.71011,
- "24": 0.71396,
- "25": 0.70984,
- "26": 0.7111,
- "27": 0.71496,
- "28": 0.71187,
- "29": 0.71729,
- "30": 0.72095,
- "31": 0.71436,
- "32": 0.70963,
- "33": 0.71384,
- "34": 0.71534,
- "35": 0.7148,
- "36": 0.71389,
- "37": 0.71097,
- "38": 0.71244,
- "39": 0.7048,
- "40": 0.715,
- "41": 1.08196,
- "42": 0.71129,
- "43": 0.73716,
- "44": 0.72639,
- "45": 0.71182,
- "46": 0.71576,
- "47": 0.72917,
- "48": 0.72017,
- "49": 0.72166,
- "50": 0.70656
+ "2": 12.28166,
+ "3": 0.8423,
+ "4": 0.84011,
+ "5": 0.81701,
+ "6": 0.82136,
+ "7": 0.82417,
+ "8": 0.81946,
+ "9": 0.81975,
+ "10": 0.81813,
+ "11": 0.82078,
+ "12": 0.83017,
+ "13": 0.82021,
+ "14": 0.81852,
+ "15": 0.81964,
+ "16": 0.81778,
+ "17": 0.85722,
+ "18": 0.81428,
+ "19": 0.8163,
+ "20": 0.81226,
+ "21": 0.81898,
+ "22": 0.81825,
+ "23": 0.81827,
+ "24": 0.81386,
+ "25": 0.81609,
+ "26": 0.81305,
+ "27": 0.81825,
+ "28": 0.81785,
+ "29": 0.81412,
+ "30": 0.81316,
+ "31": 0.81642,
+ "32": 0.81661,
+ "33": 0.81787,
+ "34": 0.81723,
+ "35": 0.82087,
+ "36": 0.81683,
+ "37": 0.81788,
+ "38": 0.81728,
+ "39": 0.81548,
+ "40": 0.81851,
+ "41": 0.82697,
+ "42": 0.9091,
+ "43": 0.8141,
+ "44": 0.81178,
+ "45": 0.81052,
+ "46": 0.81949,
+ "47": 0.81737,
+ "48": 0.81316,
+ "49": 0.82751,
+ "50": 0.80977
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel_dp_last/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel_dp_last/model_config.yaml
index f77c2a41f68..61d25aeb356 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel_dp_last/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_cp2_pp2_ep2_te_4experts2parallel_dp_last/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -57,4 +57,6 @@ MODEL_ARGS:
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
--use-tp-pp-dp-mapping: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel/golden_values_dev_dgx_a100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel/golden_values_dev_dgx_a100.json
index e3b2e326fda..a10a199ae66 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel/golden_values_dev_dgx_a100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel/golden_values_dev_dgx_a100.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.79175,
"2": 10.80907,
- "3": 10.81011,
- "4": 10.78146,
- "5": 10.82288,
- "6": 10.84057,
- "7": 10.81192,
- "8": 10.80005,
- "9": 10.81667,
- "10": 10.7688,
- "11": 10.8618,
- "12": 10.84042,
- "13": 10.84452,
- "14": 10.86421,
- "15": 10.79157,
- "16": 10.78199,
- "17": 10.75122,
- "18": 10.79446,
- "19": 10.79523,
- "20": 10.71001,
- "21": 10.68811,
- "22": 10.53736,
- "23": 10.7066,
- "24": 10.58865,
- "25": 10.54662,
- "26": 10.59492,
- "27": 10.62142,
- "28": 10.5969,
- "29": 10.60036,
- "30": 10.39407,
- "31": 10.12951,
- "32": 10.49684,
- "33": 10.48779,
- "34": 10.24347,
- "35": 10.30461,
- "36": 10.26056,
- "37": 10.38859,
- "38": 10.24848,
- "39": 10.43799,
- "40": 10.13303,
- "41": 10.18651,
- "42": 10.25823,
- "43": 9.892,
- "44": 10.02576,
- "45": 9.90015,
- "46": 9.88387,
- "47": 10.19565,
- "48": 9.91255,
- "49": 9.60147,
- "50": 9.97874
+ "3": 10.81039,
+ "4": 10.78127,
+ "5": 10.82308,
+ "6": 10.84083,
+ "7": 10.81171,
+ "8": 10.79973,
+ "9": 10.81744,
+ "10": 10.76857,
+ "11": 10.86217,
+ "12": 10.84084,
+ "13": 10.8452,
+ "14": 10.86332,
+ "15": 10.79106,
+ "16": 10.78185,
+ "17": 10.75084,
+ "18": 10.7944,
+ "19": 10.79546,
+ "20": 10.70984,
+ "21": 10.68874,
+ "22": 10.53812,
+ "23": 10.70639,
+ "24": 10.58861,
+ "25": 10.54517,
+ "26": 10.59545,
+ "27": 10.6213,
+ "28": 10.59623,
+ "29": 10.60109,
+ "30": 10.39455,
+ "31": 10.12997,
+ "32": 10.49682,
+ "33": 10.48802,
+ "34": 10.24299,
+ "35": 10.305,
+ "36": 10.26115,
+ "37": 10.38853,
+ "38": 10.24853,
+ "39": 10.43727,
+ "40": 10.13312,
+ "41": 10.1867,
+ "42": 10.25835,
+ "43": 9.89244,
+ "44": 10.02594,
+ "45": 9.90006,
+ "46": 9.88449,
+ "47": 10.19557,
+ "48": 9.91284,
+ "49": 9.60182,
+ "50": 9.97848
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 5656.0,
- "2": 6018.0,
- "3": 5790.0,
- "4": 5941.0,
- "5": 6476.0,
- "6": 6653.0,
- "7": 6287.0,
- "8": 5875.0,
- "9": 6239.0,
- "10": 5453.0,
- "11": 6936.0,
- "12": 6711.0,
- "13": 6655.0,
- "14": 6814.0,
- "15": 6233.0,
- "16": 6533.0,
- "17": 6397.0,
- "18": 6112.0,
- "19": 6678.0,
- "20": 5837.0,
- "21": 6403.0,
- "22": 5715.0,
- "23": 6744.0,
- "24": 6051.0,
- "25": 5811.0,
- "26": 6104.0,
- "27": 6484.0,
- "28": 6884.0,
- "29": 7253.0,
- "30": 6047.0,
- "31": 5593.0,
- "32": 6625.0,
- "33": 7054.0,
- "34": 6104.0,
- "35": 6712.0,
- "36": 6684.0,
- "37": 7523.0,
- "38": 7273.0,
- "39": 7620.0,
- "40": 7062.0,
- "41": 6895.0,
- "42": 7426.0,
- "43": 6713.0,
- "44": 6664.0,
- "45": 6681.0,
- "46": 6923.0,
- "47": 7705.0,
- "48": 7248.0,
- "49": 7331.0,
- "50": 7527.0
+ "1": 5743.0,
+ "2": 5999.0,
+ "3": 5905.0,
+ "4": 5938.0,
+ "5": 6445.0,
+ "6": 6621.0,
+ "7": 6274.0,
+ "8": 5877.0,
+ "9": 6334.0,
+ "10": 5348.0,
+ "11": 6889.0,
+ "12": 6531.0,
+ "13": 6773.0,
+ "14": 6834.0,
+ "15": 6263.0,
+ "16": 6473.0,
+ "17": 6190.0,
+ "18": 6261.0,
+ "19": 6546.0,
+ "20": 5906.0,
+ "21": 6290.0,
+ "22": 5727.0,
+ "23": 6736.0,
+ "24": 5914.0,
+ "25": 5888.0,
+ "26": 6127.0,
+ "27": 6455.0,
+ "28": 6974.0,
+ "29": 7148.0,
+ "30": 6270.0,
+ "31": 5639.0,
+ "32": 6815.0,
+ "33": 7063.0,
+ "34": 6218.0,
+ "35": 6708.0,
+ "36": 6496.0,
+ "37": 7484.0,
+ "38": 7231.0,
+ "39": 7687.0,
+ "40": 7025.0,
+ "41": 7081.0,
+ "42": 7161.0,
+ "43": 6589.0,
+ "44": 6743.0,
+ "45": 6790.0,
+ "46": 6933.0,
+ "47": 7570.0,
+ "48": 7413.0,
+ "49": 7262.0,
+ "50": 7706.0
}
},
"mem-allocated-bytes": {
@@ -122,52 +122,52 @@
"2": 458211840.0,
"3": 458215424.0,
"4": 458211840.0,
- "5": 458213376.0,
+ "5": 458213888.0,
"6": 458213888.0,
- "7": 458216448.0,
+ "7": 458215936.0,
"8": 458216448.0,
"9": 458212864.0,
- "10": 458215936.0,
- "11": 458213888.0,
+ "10": 458215424.0,
+ "11": 458212864.0,
"12": 458213888.0,
"13": 458214400.0,
"14": 458215424.0,
- "15": 458215424.0,
+ "15": 458214912.0,
"16": 458212864.0,
- "17": 458214400.0,
- "18": 458214400.0,
- "19": 458214400.0,
- "20": 458214400.0,
+ "17": 458214912.0,
+ "18": 458214912.0,
+ "19": 458215424.0,
+ "20": 458214912.0,
"21": 458211840.0,
- "22": 458218496.0,
+ "22": 458219520.0,
"23": 458214912.0,
"24": 458214400.0,
"25": 458211840.0,
- "26": 458215936.0,
- "27": 458210816.0,
+ "26": 458215424.0,
+ "27": 458209792.0,
"28": 458213888.0,
- "29": 458212864.0,
+ "29": 458213888.0,
"30": 458211840.0,
"31": 458219008.0,
"32": 458214400.0,
"33": 458214912.0,
"34": 458211840.0,
- "35": 458215936.0,
+ "35": 458215424.0,
"36": 458212864.0,
- "37": 458215424.0,
+ "37": 458214400.0,
"38": 458213888.0,
"39": 458213888.0,
"40": 458213376.0,
"41": 458216960.0,
- "42": 458215424.0,
+ "42": 458214912.0,
"43": 458216960.0,
"44": 458213376.0,
- "45": 458214400.0,
+ "45": 458213888.0,
"46": 458216448.0,
- "47": 458213376.0,
+ "47": 458212352.0,
"48": 458213888.0,
- "49": 458215424.0,
- "50": 458214912.0
+ "49": 458216448.0,
+ "50": 458215424.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1029256704.0,
- "2": 1193177088.0,
- "3": 1193177088.0,
- "4": 1193686016.0,
- "5": 1193686016.0,
- "6": 1193686016.0,
- "7": 1193686016.0,
- "8": 1193686016.0,
- "9": 1193771520.0,
- "10": 1193771520.0,
- "11": 1193771520.0,
- "12": 1193771520.0,
- "13": 1193771520.0,
- "14": 1193771520.0,
- "15": 1193771520.0,
- "16": 1193771520.0,
- "17": 1193771520.0,
- "18": 1193771520.0,
- "19": 1193771520.0,
- "20": 1193771520.0,
- "21": 1193771520.0,
- "22": 1193918464.0,
- "23": 1193918464.0,
- "24": 1193918464.0,
- "25": 1193918464.0,
- "26": 1193918464.0,
- "27": 1193918464.0,
- "28": 1193918464.0,
- "29": 1193918464.0,
- "30": 1193918464.0,
- "31": 1193918464.0,
- "32": 1193918464.0,
- "33": 1193918464.0,
- "34": 1193918464.0,
- "35": 1193918464.0,
- "36": 1193918464.0,
- "37": 1193918464.0,
- "38": 1193918464.0,
- "39": 1193918464.0,
- "40": 1194139136.0,
- "41": 1194139136.0,
- "42": 1194139136.0,
- "43": 1194249728.0,
- "44": 1194249728.0,
- "45": 1194249728.0,
- "46": 1194249728.0,
- "47": 1194249728.0,
- "48": 1194249728.0,
- "49": 1194249728.0,
- "50": 1194249728.0
+ "1": 1028204032.0,
+ "2": 1193069568.0,
+ "3": 1193083392.0,
+ "4": 1194512384.0,
+ "5": 1194512384.0,
+ "6": 1194512384.0,
+ "7": 1194512384.0,
+ "8": 1194512384.0,
+ "9": 1194512384.0,
+ "10": 1194512384.0,
+ "11": 1194512384.0,
+ "12": 1194512384.0,
+ "13": 1194512384.0,
+ "14": 1194512384.0,
+ "15": 1195728896.0,
+ "16": 1195728896.0,
+ "17": 1195728896.0,
+ "18": 1195728896.0,
+ "19": 1195728896.0,
+ "20": 1195728896.0,
+ "21": 1195728896.0,
+ "22": 1195728896.0,
+ "23": 1195728896.0,
+ "24": 1195728896.0,
+ "25": 1195728896.0,
+ "26": 1195728896.0,
+ "27": 1195728896.0,
+ "28": 1195728896.0,
+ "29": 1195728896.0,
+ "30": 1195728896.0,
+ "31": 1195728896.0,
+ "32": 1195728896.0,
+ "33": 1195728896.0,
+ "34": 1195728896.0,
+ "35": 1195728896.0,
+ "36": 1195728896.0,
+ "37": 1195728896.0,
+ "38": 1195728896.0,
+ "39": 1195728896.0,
+ "40": 1195728896.0,
+ "41": 1195728896.0,
+ "42": 1195728896.0,
+ "43": 1195728896.0,
+ "44": 1195728896.0,
+ "45": 1195728896.0,
+ "46": 1195728896.0,
+ "47": 1195728896.0,
+ "48": 1195728896.0,
+ "49": 1195728896.0,
+ "50": 1195728896.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 6.67874,
- "2": 0.59048,
- "3": 0.55954,
- "4": 0.55064,
- "5": 0.54285,
- "6": 0.54344,
- "7": 0.54862,
- "8": 0.542,
- "9": 0.54738,
- "10": 0.54947,
- "11": 0.53996,
- "12": 0.54615,
- "13": 0.54407,
- "14": 0.54098,
- "15": 0.55148,
- "16": 0.54024,
- "17": 0.54784,
- "18": 0.54329,
- "19": 0.54213,
- "20": 0.55192,
- "21": 0.53901,
- "22": 0.54612,
- "23": 0.54495,
- "24": 0.54254,
- "25": 0.55242,
- "26": 0.53958,
- "27": 0.54346,
- "28": 0.5466,
- "29": 0.54048,
- "30": 0.55385,
- "31": 0.54112,
- "32": 0.54404,
- "33": 0.54779,
- "34": 0.54049,
- "35": 0.53889,
- "36": 0.53823,
- "37": 0.54013,
- "38": 0.53918,
- "39": 0.53801,
- "40": 0.5394,
- "41": 0.53905,
- "42": 0.53797,
- "43": 0.53957,
- "44": 0.5384,
- "45": 0.53795,
- "46": 0.53859,
- "47": 0.54222,
- "48": 0.53881,
- "49": 0.5401,
- "50": 0.53746
+ "1": "nan",
+ "2": 4.55953,
+ "3": 0.55516,
+ "4": 0.53746,
+ "5": 0.53478,
+ "6": 0.53498,
+ "7": 0.5367,
+ "8": 0.53629,
+ "9": 0.53522,
+ "10": 0.53431,
+ "11": 0.53368,
+ "12": 0.53492,
+ "13": 0.53533,
+ "14": 0.5449,
+ "15": 0.53453,
+ "16": 0.53447,
+ "17": 0.53496,
+ "18": 0.53336,
+ "19": 0.53494,
+ "20": 0.53505,
+ "21": 0.53514,
+ "22": 0.53519,
+ "23": 0.53691,
+ "24": 0.53531,
+ "25": 0.53509,
+ "26": 0.53425,
+ "27": 0.53381,
+ "28": 0.53396,
+ "29": 0.53282,
+ "30": 0.53481,
+ "31": 0.53441,
+ "32": 0.53325,
+ "33": 0.54597,
+ "34": 0.53535,
+ "35": 0.53466,
+ "36": 0.53377,
+ "37": 0.53356,
+ "38": 0.53288,
+ "39": 0.53419,
+ "40": 0.53467,
+ "41": 0.53334,
+ "42": 0.53455,
+ "43": 0.53489,
+ "44": 0.53505,
+ "45": 0.53371,
+ "46": 0.53538,
+ "47": 0.53461,
+ "48": 0.53464,
+ "49": 0.53506,
+ "50": 0.53332
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel/golden_values_dev_dgx_h100.json
index 6ec10f4f931..c53f8ba9f79 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel/golden_values_dev_dgx_h100.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.80475,
"2": 10.821,
- "3": 10.8216,
- "4": 10.79306,
- "5": 10.84831,
- "6": 10.85888,
- "7": 10.83177,
- "8": 10.82362,
- "9": 10.83757,
- "10": 10.78732,
- "11": 10.86732,
- "12": 10.85395,
- "13": 10.86171,
- "14": 10.88343,
- "15": 10.79765,
- "16": 10.79986,
- "17": 10.76238,
- "18": 10.80286,
- "19": 10.7945,
- "20": 10.71733,
- "21": 10.70194,
- "22": 10.55147,
- "23": 10.72167,
- "24": 10.60698,
- "25": 10.54614,
- "26": 10.6136,
- "27": 10.63974,
- "28": 10.60486,
- "29": 10.62277,
- "30": 10.41109,
- "31": 10.1456,
- "32": 10.51017,
- "33": 10.50089,
- "34": 10.25812,
- "35": 10.3154,
- "36": 10.27895,
- "37": 10.41061,
- "38": 10.25908,
- "39": 10.45334,
- "40": 10.1604,
- "41": 10.20557,
- "42": 10.26792,
- "43": 9.90468,
- "44": 10.03233,
- "45": 9.91098,
- "46": 9.87857,
- "47": 10.20952,
- "48": 9.93178,
- "49": 9.61584,
- "50": 9.98565
+ "3": 10.82137,
+ "4": 10.79315,
+ "5": 10.8483,
+ "6": 10.85935,
+ "7": 10.83174,
+ "8": 10.82386,
+ "9": 10.83754,
+ "10": 10.78771,
+ "11": 10.86699,
+ "12": 10.85365,
+ "13": 10.86137,
+ "14": 10.88332,
+ "15": 10.79759,
+ "16": 10.80014,
+ "17": 10.76189,
+ "18": 10.80285,
+ "19": 10.79428,
+ "20": 10.71656,
+ "21": 10.70165,
+ "22": 10.55146,
+ "23": 10.72122,
+ "24": 10.60742,
+ "25": 10.54634,
+ "26": 10.61335,
+ "27": 10.63973,
+ "28": 10.60466,
+ "29": 10.62274,
+ "30": 10.41087,
+ "31": 10.14603,
+ "32": 10.50965,
+ "33": 10.50142,
+ "34": 10.25863,
+ "35": 10.31568,
+ "36": 10.27941,
+ "37": 10.41098,
+ "38": 10.2593,
+ "39": 10.45366,
+ "40": 10.1605,
+ "41": 10.20637,
+ "42": 10.26762,
+ "43": 9.90459,
+ "44": 10.03234,
+ "45": 9.91164,
+ "46": 9.87875,
+ "47": 10.20947,
+ "48": 9.93144,
+ "49": 9.61602,
+ "50": 9.98541
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 5474.0,
- "2": 5853.0,
- "3": 5875.0,
- "4": 6041.0,
- "5": 6601.0,
- "6": 6654.0,
- "7": 6135.0,
- "8": 5761.0,
- "9": 6505.0,
- "10": 5497.0,
- "11": 6994.0,
- "12": 6523.0,
- "13": 6807.0,
- "14": 6969.0,
- "15": 6154.0,
- "16": 6667.0,
- "17": 6368.0,
- "18": 6298.0,
- "19": 6353.0,
- "20": 5998.0,
- "21": 6264.0,
- "22": 5628.0,
- "23": 6620.0,
- "24": 6063.0,
- "25": 5649.0,
- "26": 6226.0,
- "27": 6409.0,
- "28": 6790.0,
- "29": 7055.0,
- "30": 6430.0,
- "31": 5565.0,
- "32": 6615.0,
- "33": 6969.0,
- "34": 6107.0,
- "35": 6538.0,
- "36": 6486.0,
- "37": 7272.0,
- "38": 6923.0,
- "39": 7497.0,
- "40": 6997.0,
- "41": 6747.0,
- "42": 7228.0,
- "43": 6629.0,
- "44": 6752.0,
- "45": 6557.0,
- "46": 6904.0,
- "47": 7474.0,
- "48": 7165.0,
- "49": 7244.0,
- "50": 7331.0
+ "1": 5649.0,
+ "2": 5830.0,
+ "3": 6085.0,
+ "4": 5929.0,
+ "5": 6605.0,
+ "6": 6671.0,
+ "7": 6190.0,
+ "8": 5974.0,
+ "9": 6573.0,
+ "10": 5417.0,
+ "11": 6943.0,
+ "12": 6397.0,
+ "13": 6815.0,
+ "14": 6932.0,
+ "15": 6292.0,
+ "16": 6446.0,
+ "17": 6413.0,
+ "18": 6231.0,
+ "19": 6288.0,
+ "20": 5995.0,
+ "21": 6301.0,
+ "22": 5746.0,
+ "23": 6701.0,
+ "24": 5988.0,
+ "25": 5734.0,
+ "26": 6127.0,
+ "27": 6238.0,
+ "28": 6771.0,
+ "29": 7217.0,
+ "30": 6275.0,
+ "31": 5518.0,
+ "32": 6530.0,
+ "33": 7079.0,
+ "34": 6130.0,
+ "35": 6701.0,
+ "36": 6367.0,
+ "37": 7301.0,
+ "38": 6815.0,
+ "39": 7720.0,
+ "40": 6748.0,
+ "41": 6679.0,
+ "42": 7340.0,
+ "43": 6563.0,
+ "44": 6570.0,
+ "45": 6542.0,
+ "46": 7029.0,
+ "47": 7320.0,
+ "48": 7144.0,
+ "49": 7259.0,
+ "50": 7400.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 458212352.0,
- "2": 458212864.0,
- "3": 458211328.0,
- "4": 458212864.0,
- "5": 458212352.0,
- "6": 458213376.0,
- "7": 458212864.0,
- "8": 458213888.0,
- "9": 458213376.0,
- "10": 458212864.0,
- "11": 458210816.0,
- "12": 458210304.0,
- "13": 458211840.0,
- "14": 458213376.0,
- "15": 458214400.0,
- "16": 458215424.0,
- "17": 458212864.0,
- "18": 458210816.0,
- "19": 458211840.0,
- "20": 458212352.0,
- "21": 458213888.0,
- "22": 458213888.0,
- "23": 458211328.0,
- "24": 458211840.0,
- "25": 458211840.0,
- "26": 458212864.0,
- "27": 458212352.0,
- "28": 458212864.0,
- "29": 458211840.0,
- "30": 458211840.0,
- "31": 458213376.0,
- "32": 458211328.0,
- "33": 458210304.0,
- "34": 458213888.0,
- "35": 458214912.0,
- "36": 458211328.0,
- "37": 458210816.0,
- "38": 458211840.0,
- "39": 458213376.0,
- "40": 458211328.0,
- "41": 458213888.0,
- "42": 458211840.0,
- "43": 458214400.0,
- "44": 458213888.0,
- "45": 458210816.0,
- "46": 458213888.0,
- "47": 458211328.0,
- "48": 458212352.0,
- "49": 458212352.0,
- "50": 458210816.0
+ "1": 458736640.0,
+ "2": 458737152.0,
+ "3": 458735616.0,
+ "4": 458737152.0,
+ "5": 458736640.0,
+ "6": 458737664.0,
+ "7": 458737152.0,
+ "8": 458738176.0,
+ "9": 458738688.0,
+ "10": 458737152.0,
+ "11": 458735104.0,
+ "12": 458734592.0,
+ "13": 458737664.0,
+ "14": 458737664.0,
+ "15": 458737664.0,
+ "16": 458739712.0,
+ "17": 458737152.0,
+ "18": 458735104.0,
+ "19": 458736128.0,
+ "20": 458736640.0,
+ "21": 458735616.0,
+ "22": 458736640.0,
+ "23": 458735616.0,
+ "24": 458737152.0,
+ "25": 458737152.0,
+ "26": 458737152.0,
+ "27": 458735616.0,
+ "28": 458737152.0,
+ "29": 458735104.0,
+ "30": 458736128.0,
+ "31": 458737664.0,
+ "32": 458736128.0,
+ "33": 458734592.0,
+ "34": 458738176.0,
+ "35": 458739200.0,
+ "36": 458735616.0,
+ "37": 458735104.0,
+ "38": 458736128.0,
+ "39": 458737664.0,
+ "40": 458735616.0,
+ "41": 458737664.0,
+ "42": 458735616.0,
+ "43": 458738688.0,
+ "44": 458737664.0,
+ "45": 458735616.0,
+ "46": 458738688.0,
+ "47": 458735616.0,
+ "48": 458736640.0,
+ "49": 458736640.0,
+ "50": 458734080.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1026068480.0,
- "2": 1192152064.0,
- "3": 1192152064.0,
- "4": 1192205312.0,
- "5": 1192205312.0,
- "6": 1192205312.0,
- "7": 1192205312.0,
- "8": 1192205312.0,
- "9": 1192205312.0,
- "10": 1192205312.0,
- "11": 1192205312.0,
- "12": 1192205312.0,
- "13": 1192349184.0,
- "14": 1192349184.0,
- "15": 1192506368.0,
- "16": 1192506368.0,
- "17": 1192506368.0,
- "18": 1192506368.0,
- "19": 1192506368.0,
- "20": 1192506368.0,
- "21": 1192506368.0,
- "22": 1192506368.0,
- "23": 1192506368.0,
- "24": 1192506368.0,
- "25": 1192506368.0,
- "26": 1192506368.0,
- "27": 1192506368.0,
- "28": 1192506368.0,
- "29": 1192506368.0,
- "30": 1192506368.0,
- "31": 1192506368.0,
- "32": 1192506368.0,
- "33": 1192506368.0,
- "34": 1192506368.0,
- "35": 1192506368.0,
- "36": 1192506368.0,
- "37": 1192506368.0,
- "38": 1192506368.0,
- "39": 1192506368.0,
- "40": 1192506368.0,
- "41": 1192506368.0,
- "42": 1192506368.0,
- "43": 1192506368.0,
- "44": 1192506368.0,
- "45": 1192506368.0,
- "46": 1192506368.0,
- "47": 1192506368.0,
- "48": 1192506368.0,
- "49": 1192506368.0,
- "50": 1192506368.0
+ "1": 1026235392.0,
+ "2": 1191006720.0,
+ "3": 1191138304.0,
+ "4": 1192048640.0,
+ "5": 1192048640.0,
+ "6": 1192048640.0,
+ "7": 1192048640.0,
+ "8": 1192048640.0,
+ "9": 1192048640.0,
+ "10": 1192327680.0,
+ "11": 1192465920.0,
+ "12": 1192465920.0,
+ "13": 1192641024.0,
+ "14": 1192641024.0,
+ "15": 1192641024.0,
+ "16": 1192641024.0,
+ "17": 1192641024.0,
+ "18": 1192641024.0,
+ "19": 1192641024.0,
+ "20": 1192641024.0,
+ "21": 1192641024.0,
+ "22": 1192641024.0,
+ "23": 1192641024.0,
+ "24": 1192641024.0,
+ "25": 1192641024.0,
+ "26": 1192641024.0,
+ "27": 1192641024.0,
+ "28": 1192641024.0,
+ "29": 1192641024.0,
+ "30": 1192641024.0,
+ "31": 1192641024.0,
+ "32": 1192641024.0,
+ "33": 1192641024.0,
+ "34": 1192641024.0,
+ "35": 1192641024.0,
+ "36": 1192641024.0,
+ "37": 1192641024.0,
+ "38": 1192641024.0,
+ "39": 1193289216.0,
+ "40": 1193289216.0,
+ "41": 1193289216.0,
+ "42": 1193289216.0,
+ "43": 1193289216.0,
+ "44": 1193289216.0,
+ "45": 1193289216.0,
+ "46": 1193289216.0,
+ "47": 1193289216.0,
+ "48": 1193289216.0,
+ "49": 1193289216.0,
+ "50": 1193289216.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 13.43711,
- "2": 0.5648,
- "3": 0.46103,
- "4": 0.42843,
- "5": 0.39023,
- "6": 0.40228,
- "7": 0.39933,
- "8": 0.40801,
- "9": 0.41661,
- "10": 0.41115,
- "11": 0.40919,
- "12": 0.38713,
- "13": 0.3967,
- "14": 0.39634,
- "15": 0.3917,
- "16": 0.38895,
- "17": 0.39488,
- "18": 0.38262,
- "19": 0.38633,
- "20": 0.38778,
- "21": 0.37793,
- "22": 0.38122,
- "23": 0.3785,
- "24": 0.38176,
- "25": 0.37936,
- "26": 0.38399,
- "27": 0.37425,
- "28": 0.38373,
- "29": 0.37674,
- "30": 0.38541,
- "31": 0.38748,
- "32": 0.37483,
- "33": 0.37931,
- "34": 0.38691,
- "35": 0.39293,
- "36": 0.38011,
- "37": 0.37641,
- "38": 0.37714,
- "39": 0.37754,
- "40": 0.3929,
- "41": 0.37984,
- "42": 0.37748,
- "43": 0.39504,
- "44": 0.38155,
- "45": 0.39617,
- "46": 0.42631,
- "47": 0.39497,
- "48": 0.39432,
- "49": 0.40482,
- "50": 0.37964
+ "1": "nan",
+ "2": 7.35614,
+ "3": 0.42662,
+ "4": 0.4194,
+ "5": 0.38561,
+ "6": 0.38215,
+ "7": 0.44042,
+ "8": 0.39648,
+ "9": 0.39179,
+ "10": 0.38601,
+ "11": 0.37867,
+ "12": 0.39696,
+ "13": 0.37723,
+ "14": 0.37583,
+ "15": 0.38932,
+ "16": 0.37862,
+ "17": 0.38218,
+ "18": 0.38721,
+ "19": 0.39068,
+ "20": 0.3855,
+ "21": 0.37737,
+ "22": 0.37975,
+ "23": 0.3763,
+ "24": 0.37755,
+ "25": 0.37814,
+ "26": 0.37613,
+ "27": 0.38794,
+ "28": 0.37908,
+ "29": 0.37583,
+ "30": 0.37897,
+ "31": 0.39194,
+ "32": 0.37775,
+ "33": 0.37581,
+ "34": 0.38903,
+ "35": 0.38954,
+ "36": 0.378,
+ "37": 0.37692,
+ "38": 0.37717,
+ "39": 0.37356,
+ "40": 0.37832,
+ "41": 0.38595,
+ "42": 0.38169,
+ "43": 0.38907,
+ "44": 0.38237,
+ "45": 0.37792,
+ "46": 0.414,
+ "47": 0.3933,
+ "48": 0.38767,
+ "49": 0.39999,
+ "50": 0.37905
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel/model_config.yaml
index 12e6698a5f4..b03dd7fe023 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -56,4 +56,6 @@ MODEL_ARGS:
--disable-bias-linear: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel_dp_last/golden_values_dev_dgx_a100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel_dp_last/golden_values_dev_dgx_a100.json
index b4462fc931e..890246c909f 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel_dp_last/golden_values_dev_dgx_a100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel_dp_last/golden_values_dev_dgx_a100.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.79175,
"2": 10.80907,
- "3": 10.81011,
- "4": 10.78146,
- "5": 10.82288,
- "6": 10.84057,
- "7": 10.81192,
- "8": 10.80005,
- "9": 10.81667,
- "10": 10.7688,
- "11": 10.8618,
- "12": 10.84042,
- "13": 10.84452,
- "14": 10.86421,
- "15": 10.79157,
- "16": 10.78199,
- "17": 10.75122,
- "18": 10.79446,
- "19": 10.79523,
- "20": 10.71001,
- "21": 10.68811,
- "22": 10.53736,
- "23": 10.7066,
- "24": 10.58865,
- "25": 10.54662,
- "26": 10.59492,
- "27": 10.62142,
- "28": 10.5969,
- "29": 10.60036,
- "30": 10.39407,
- "31": 10.12951,
- "32": 10.49684,
- "33": 10.48779,
- "34": 10.24347,
- "35": 10.30461,
- "36": 10.26056,
- "37": 10.38859,
- "38": 10.24848,
- "39": 10.43799,
- "40": 10.13303,
- "41": 10.18651,
- "42": 10.25823,
- "43": 9.892,
- "44": 10.02576,
- "45": 9.90015,
- "46": 9.88387,
- "47": 10.19565,
- "48": 9.91255,
- "49": 9.60147,
- "50": 9.97874
+ "3": 10.81039,
+ "4": 10.78127,
+ "5": 10.82308,
+ "6": 10.84083,
+ "7": 10.81171,
+ "8": 10.79973,
+ "9": 10.81744,
+ "10": 10.76857,
+ "11": 10.86217,
+ "12": 10.84084,
+ "13": 10.8452,
+ "14": 10.86332,
+ "15": 10.79106,
+ "16": 10.78185,
+ "17": 10.75084,
+ "18": 10.7944,
+ "19": 10.79546,
+ "20": 10.70984,
+ "21": 10.68874,
+ "22": 10.53812,
+ "23": 10.70639,
+ "24": 10.58861,
+ "25": 10.54517,
+ "26": 10.59545,
+ "27": 10.6213,
+ "28": 10.59623,
+ "29": 10.60109,
+ "30": 10.39455,
+ "31": 10.12997,
+ "32": 10.49682,
+ "33": 10.48802,
+ "34": 10.24299,
+ "35": 10.305,
+ "36": 10.26115,
+ "37": 10.38853,
+ "38": 10.24853,
+ "39": 10.43727,
+ "40": 10.13312,
+ "41": 10.1867,
+ "42": 10.25835,
+ "43": 9.89244,
+ "44": 10.02594,
+ "45": 9.90006,
+ "46": 9.88449,
+ "47": 10.19557,
+ "48": 9.91284,
+ "49": 9.60182,
+ "50": 9.97848
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 5656.0,
- "2": 6018.0,
- "3": 5790.0,
- "4": 5941.0,
- "5": 6476.0,
- "6": 6653.0,
- "7": 6287.0,
- "8": 5875.0,
- "9": 6239.0,
- "10": 5453.0,
- "11": 6936.0,
- "12": 6711.0,
- "13": 6655.0,
- "14": 6814.0,
- "15": 6233.0,
- "16": 6533.0,
- "17": 6397.0,
- "18": 6112.0,
- "19": 6678.0,
- "20": 5837.0,
- "21": 6403.0,
- "22": 5715.0,
- "23": 6744.0,
- "24": 6051.0,
- "25": 5811.0,
- "26": 6104.0,
- "27": 6484.0,
- "28": 6884.0,
- "29": 7253.0,
- "30": 6047.0,
- "31": 5593.0,
- "32": 6625.0,
- "33": 7054.0,
- "34": 6104.0,
- "35": 6712.0,
- "36": 6684.0,
- "37": 7523.0,
- "38": 7273.0,
- "39": 7620.0,
- "40": 7062.0,
- "41": 6895.0,
- "42": 7426.0,
- "43": 6713.0,
- "44": 6664.0,
- "45": 6681.0,
- "46": 6923.0,
- "47": 7705.0,
- "48": 7248.0,
- "49": 7331.0,
- "50": 7527.0
+ "1": 5743.0,
+ "2": 5999.0,
+ "3": 5905.0,
+ "4": 5938.0,
+ "5": 6445.0,
+ "6": 6621.0,
+ "7": 6274.0,
+ "8": 5877.0,
+ "9": 6334.0,
+ "10": 5348.0,
+ "11": 6889.0,
+ "12": 6531.0,
+ "13": 6773.0,
+ "14": 6834.0,
+ "15": 6263.0,
+ "16": 6473.0,
+ "17": 6190.0,
+ "18": 6261.0,
+ "19": 6546.0,
+ "20": 5906.0,
+ "21": 6290.0,
+ "22": 5727.0,
+ "23": 6736.0,
+ "24": 5914.0,
+ "25": 5888.0,
+ "26": 6127.0,
+ "27": 6455.0,
+ "28": 6974.0,
+ "29": 7148.0,
+ "30": 6270.0,
+ "31": 5639.0,
+ "32": 6815.0,
+ "33": 7063.0,
+ "34": 6218.0,
+ "35": 6708.0,
+ "36": 6496.0,
+ "37": 7484.0,
+ "38": 7231.0,
+ "39": 7687.0,
+ "40": 7025.0,
+ "41": 7081.0,
+ "42": 7161.0,
+ "43": 6589.0,
+ "44": 6743.0,
+ "45": 6790.0,
+ "46": 6933.0,
+ "47": 7570.0,
+ "48": 7413.0,
+ "49": 7262.0,
+ "50": 7706.0
}
},
"mem-allocated-bytes": {
@@ -122,52 +122,52 @@
"2": 458211840.0,
"3": 458215424.0,
"4": 458211840.0,
- "5": 458213376.0,
+ "5": 458213888.0,
"6": 458213888.0,
- "7": 458216448.0,
+ "7": 458215936.0,
"8": 458216448.0,
"9": 458212864.0,
- "10": 458215936.0,
- "11": 458213888.0,
+ "10": 458215424.0,
+ "11": 458212864.0,
"12": 458213888.0,
"13": 458214400.0,
"14": 458215424.0,
- "15": 458215424.0,
+ "15": 458214912.0,
"16": 458212864.0,
- "17": 458214400.0,
- "18": 458214400.0,
- "19": 458214400.0,
- "20": 458214400.0,
+ "17": 458214912.0,
+ "18": 458214912.0,
+ "19": 458215424.0,
+ "20": 458214912.0,
"21": 458211840.0,
- "22": 458218496.0,
+ "22": 458219520.0,
"23": 458214912.0,
"24": 458214400.0,
"25": 458211840.0,
- "26": 458215936.0,
- "27": 458210816.0,
+ "26": 458215424.0,
+ "27": 458209792.0,
"28": 458213888.0,
- "29": 458212864.0,
+ "29": 458213888.0,
"30": 458211840.0,
"31": 458219008.0,
"32": 458214400.0,
"33": 458214912.0,
"34": 458211840.0,
- "35": 458215936.0,
+ "35": 458215424.0,
"36": 458212864.0,
- "37": 458215424.0,
+ "37": 458214400.0,
"38": 458213888.0,
"39": 458213888.0,
"40": 458213376.0,
"41": 458216960.0,
- "42": 458215424.0,
+ "42": 458214912.0,
"43": 458216960.0,
"44": 458213376.0,
- "45": 458214400.0,
+ "45": 458213888.0,
"46": 458216448.0,
- "47": 458213376.0,
+ "47": 458212352.0,
"48": 458213888.0,
- "49": 458215424.0,
- "50": 458214912.0
+ "49": 458216448.0,
+ "50": 458215424.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1029256704.0,
- "2": 1193177088.0,
- "3": 1193177088.0,
- "4": 1193686016.0,
- "5": 1193686016.0,
- "6": 1193686016.0,
- "7": 1193686016.0,
- "8": 1193686016.0,
- "9": 1193771520.0,
- "10": 1193771520.0,
- "11": 1193771520.0,
- "12": 1193771520.0,
- "13": 1193771520.0,
- "14": 1193771520.0,
- "15": 1193771520.0,
- "16": 1193771520.0,
- "17": 1193771520.0,
- "18": 1193771520.0,
- "19": 1193771520.0,
- "20": 1193771520.0,
- "21": 1193771520.0,
- "22": 1193918464.0,
- "23": 1193918464.0,
- "24": 1193918464.0,
- "25": 1193918464.0,
- "26": 1193918464.0,
- "27": 1193918464.0,
- "28": 1193918464.0,
- "29": 1193918464.0,
- "30": 1193918464.0,
- "31": 1193918464.0,
- "32": 1193918464.0,
- "33": 1193918464.0,
- "34": 1193918464.0,
- "35": 1193918464.0,
- "36": 1193918464.0,
- "37": 1193918464.0,
- "38": 1193918464.0,
- "39": 1193918464.0,
- "40": 1194139136.0,
- "41": 1194139136.0,
- "42": 1194139136.0,
- "43": 1194249728.0,
- "44": 1194249728.0,
- "45": 1194249728.0,
- "46": 1194249728.0,
- "47": 1194249728.0,
- "48": 1194249728.0,
- "49": 1194249728.0,
- "50": 1194249728.0
+ "1": 1028204032.0,
+ "2": 1193069568.0,
+ "3": 1193083392.0,
+ "4": 1194512384.0,
+ "5": 1194512384.0,
+ "6": 1194512384.0,
+ "7": 1194512384.0,
+ "8": 1194512384.0,
+ "9": 1194512384.0,
+ "10": 1194512384.0,
+ "11": 1194512384.0,
+ "12": 1194512384.0,
+ "13": 1194512384.0,
+ "14": 1194512384.0,
+ "15": 1195728896.0,
+ "16": 1195728896.0,
+ "17": 1195728896.0,
+ "18": 1195728896.0,
+ "19": 1195728896.0,
+ "20": 1195728896.0,
+ "21": 1195728896.0,
+ "22": 1195728896.0,
+ "23": 1195728896.0,
+ "24": 1195728896.0,
+ "25": 1195728896.0,
+ "26": 1195728896.0,
+ "27": 1195728896.0,
+ "28": 1195728896.0,
+ "29": 1195728896.0,
+ "30": 1195728896.0,
+ "31": 1195728896.0,
+ "32": 1195728896.0,
+ "33": 1195728896.0,
+ "34": 1195728896.0,
+ "35": 1195728896.0,
+ "36": 1195728896.0,
+ "37": 1195728896.0,
+ "38": 1195728896.0,
+ "39": 1195728896.0,
+ "40": 1195728896.0,
+ "41": 1195728896.0,
+ "42": 1195728896.0,
+ "43": 1195728896.0,
+ "44": 1195728896.0,
+ "45": 1195728896.0,
+ "46": 1195728896.0,
+ "47": 1195728896.0,
+ "48": 1195728896.0,
+ "49": 1195728896.0,
+ "50": 1195728896.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 6.42299,
- "2": 0.59069,
- "3": 0.56496,
- "4": 0.54736,
- "5": 0.54792,
- "6": 0.57731,
- "7": 0.54778,
- "8": 0.54659,
- "9": 0.54833,
- "10": 0.54497,
- "11": 0.55076,
- "12": 0.55595,
- "13": 0.54721,
- "14": 0.54614,
- "15": 0.5457,
- "16": 0.54774,
- "17": 0.54518,
- "18": 0.54582,
- "19": 0.5467,
- "20": 0.54611,
- "21": 0.54622,
- "22": 0.54617,
- "23": 0.54622,
- "24": 0.54547,
- "25": 0.54796,
- "26": 0.54413,
- "27": 0.5458,
- "28": 0.54598,
- "29": 0.54813,
- "30": 0.54556,
- "31": 0.54684,
- "32": 0.54789,
- "33": 0.57275,
- "34": 0.54705,
- "35": 0.54545,
- "36": 0.54414,
- "37": 0.54225,
- "38": 0.54504,
- "39": 0.54284,
- "40": 0.54185,
- "41": 0.54578,
- "42": 0.54542,
- "43": 0.54621,
- "44": 0.54447,
- "45": 0.54521,
- "46": 0.5449,
- "47": 0.54529,
- "48": 0.54403,
- "49": 0.56089,
- "50": 0.54374
+ "1": "nan",
+ "2": 5.39653,
+ "3": 0.54844,
+ "4": 0.53143,
+ "5": 0.52936,
+ "6": 0.5307,
+ "7": 0.53171,
+ "8": 0.52977,
+ "9": 0.5296,
+ "10": 0.52973,
+ "11": 0.52988,
+ "12": 0.52899,
+ "13": 0.53004,
+ "14": 0.52976,
+ "15": 0.5302,
+ "16": 0.52852,
+ "17": 0.52991,
+ "18": 0.52961,
+ "19": 0.52906,
+ "20": 0.53311,
+ "21": 0.53111,
+ "22": 0.53049,
+ "23": 0.52977,
+ "24": 0.52985,
+ "25": 0.52861,
+ "26": 0.52859,
+ "27": 0.52946,
+ "28": 0.5289,
+ "29": 0.5285,
+ "30": 0.53038,
+ "31": 0.53184,
+ "32": 0.52975,
+ "33": 0.5291,
+ "34": 0.52987,
+ "35": 0.52942,
+ "36": 0.52992,
+ "37": 0.52962,
+ "38": 0.52867,
+ "39": 0.52796,
+ "40": 0.53013,
+ "41": 0.52972,
+ "42": 0.52845,
+ "43": 0.52857,
+ "44": 0.52933,
+ "45": 0.52943,
+ "46": 0.52979,
+ "47": 0.5281,
+ "48": 0.52831,
+ "49": 0.52914,
+ "50": 0.52676
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel_dp_last/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel_dp_last/golden_values_dev_dgx_h100.json
index 64dc8751e92..0617e255ebc 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel_dp_last/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel_dp_last/golden_values_dev_dgx_h100.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.80475,
"2": 10.821,
- "3": 10.8216,
- "4": 10.79306,
- "5": 10.84831,
- "6": 10.85888,
- "7": 10.83177,
- "8": 10.82362,
- "9": 10.83757,
- "10": 10.78732,
- "11": 10.86732,
- "12": 10.85395,
- "13": 10.86171,
- "14": 10.88343,
- "15": 10.79765,
- "16": 10.79986,
- "17": 10.76238,
- "18": 10.80286,
- "19": 10.7945,
- "20": 10.71733,
- "21": 10.70194,
- "22": 10.55147,
- "23": 10.72167,
- "24": 10.60698,
- "25": 10.54614,
- "26": 10.6136,
- "27": 10.63974,
- "28": 10.60486,
- "29": 10.62277,
- "30": 10.41109,
- "31": 10.1456,
- "32": 10.51017,
- "33": 10.50089,
- "34": 10.25812,
- "35": 10.3154,
- "36": 10.27895,
- "37": 10.41061,
- "38": 10.25908,
- "39": 10.45334,
- "40": 10.1604,
- "41": 10.20557,
- "42": 10.26792,
- "43": 9.90468,
- "44": 10.03233,
- "45": 9.91098,
- "46": 9.87857,
- "47": 10.20952,
- "48": 9.93178,
- "49": 9.61584,
- "50": 9.98565
+ "3": 10.82137,
+ "4": 10.79315,
+ "5": 10.8483,
+ "6": 10.85935,
+ "7": 10.83174,
+ "8": 10.82386,
+ "9": 10.83754,
+ "10": 10.78771,
+ "11": 10.86699,
+ "12": 10.85365,
+ "13": 10.86137,
+ "14": 10.88332,
+ "15": 10.79759,
+ "16": 10.80014,
+ "17": 10.76189,
+ "18": 10.80285,
+ "19": 10.79428,
+ "20": 10.71656,
+ "21": 10.70165,
+ "22": 10.55146,
+ "23": 10.72122,
+ "24": 10.60742,
+ "25": 10.54634,
+ "26": 10.61335,
+ "27": 10.63973,
+ "28": 10.60466,
+ "29": 10.62274,
+ "30": 10.41087,
+ "31": 10.14603,
+ "32": 10.50965,
+ "33": 10.50142,
+ "34": 10.25863,
+ "35": 10.31568,
+ "36": 10.27941,
+ "37": 10.41098,
+ "38": 10.2593,
+ "39": 10.45366,
+ "40": 10.1605,
+ "41": 10.20637,
+ "42": 10.26762,
+ "43": 9.90459,
+ "44": 10.03234,
+ "45": 9.91164,
+ "46": 9.87875,
+ "47": 10.20947,
+ "48": 9.93144,
+ "49": 9.61602,
+ "50": 9.98541
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 5474.0,
- "2": 5853.0,
- "3": 5875.0,
- "4": 6041.0,
- "5": 6601.0,
- "6": 6654.0,
- "7": 6135.0,
- "8": 5761.0,
- "9": 6505.0,
- "10": 5497.0,
- "11": 6994.0,
- "12": 6523.0,
- "13": 6807.0,
- "14": 6969.0,
- "15": 6154.0,
- "16": 6667.0,
- "17": 6368.0,
- "18": 6298.0,
- "19": 6353.0,
- "20": 5998.0,
- "21": 6264.0,
- "22": 5628.0,
- "23": 6620.0,
- "24": 6063.0,
- "25": 5649.0,
- "26": 6226.0,
- "27": 6409.0,
- "28": 6790.0,
- "29": 7055.0,
- "30": 6430.0,
- "31": 5565.0,
- "32": 6615.0,
- "33": 6969.0,
- "34": 6107.0,
- "35": 6538.0,
- "36": 6486.0,
- "37": 7272.0,
- "38": 6923.0,
- "39": 7497.0,
- "40": 6997.0,
- "41": 6747.0,
- "42": 7228.0,
- "43": 6629.0,
- "44": 6752.0,
- "45": 6557.0,
- "46": 6904.0,
- "47": 7474.0,
- "48": 7165.0,
- "49": 7244.0,
- "50": 7331.0
+ "1": 5649.0,
+ "2": 5830.0,
+ "3": 6085.0,
+ "4": 5929.0,
+ "5": 6605.0,
+ "6": 6671.0,
+ "7": 6190.0,
+ "8": 5974.0,
+ "9": 6573.0,
+ "10": 5417.0,
+ "11": 6943.0,
+ "12": 6397.0,
+ "13": 6815.0,
+ "14": 6932.0,
+ "15": 6292.0,
+ "16": 6446.0,
+ "17": 6413.0,
+ "18": 6231.0,
+ "19": 6288.0,
+ "20": 5995.0,
+ "21": 6301.0,
+ "22": 5746.0,
+ "23": 6701.0,
+ "24": 5988.0,
+ "25": 5734.0,
+ "26": 6127.0,
+ "27": 6238.0,
+ "28": 6771.0,
+ "29": 7217.0,
+ "30": 6275.0,
+ "31": 5518.0,
+ "32": 6530.0,
+ "33": 7079.0,
+ "34": 6130.0,
+ "35": 6701.0,
+ "36": 6367.0,
+ "37": 7301.0,
+ "38": 6815.0,
+ "39": 7720.0,
+ "40": 6748.0,
+ "41": 6679.0,
+ "42": 7340.0,
+ "43": 6563.0,
+ "44": 6570.0,
+ "45": 6542.0,
+ "46": 7029.0,
+ "47": 7320.0,
+ "48": 7144.0,
+ "49": 7259.0,
+ "50": 7400.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 458212352.0,
- "2": 458212864.0,
- "3": 458211328.0,
- "4": 458212864.0,
- "5": 458212352.0,
- "6": 458213376.0,
- "7": 458212864.0,
- "8": 458213888.0,
- "9": 458213376.0,
- "10": 458212864.0,
- "11": 458210816.0,
- "12": 458210304.0,
- "13": 458211840.0,
- "14": 458213376.0,
- "15": 458214400.0,
- "16": 458215424.0,
- "17": 458212864.0,
- "18": 458210816.0,
- "19": 458211840.0,
- "20": 458212352.0,
- "21": 458213888.0,
- "22": 458213888.0,
- "23": 458211328.0,
- "24": 458211840.0,
- "25": 458211840.0,
- "26": 458212864.0,
- "27": 458212352.0,
- "28": 458212864.0,
- "29": 458211840.0,
- "30": 458211840.0,
- "31": 458213376.0,
- "32": 458211328.0,
- "33": 458210304.0,
- "34": 458213888.0,
- "35": 458214912.0,
- "36": 458211328.0,
- "37": 458210816.0,
- "38": 458211840.0,
- "39": 458213376.0,
- "40": 458211328.0,
- "41": 458213888.0,
- "42": 458211840.0,
- "43": 458214400.0,
- "44": 458213888.0,
- "45": 458210816.0,
- "46": 458213888.0,
- "47": 458211328.0,
- "48": 458212352.0,
- "49": 458212352.0,
- "50": 458210816.0
+ "1": 458736640.0,
+ "2": 458737152.0,
+ "3": 458735616.0,
+ "4": 458737152.0,
+ "5": 458736640.0,
+ "6": 458737664.0,
+ "7": 458737152.0,
+ "8": 458738176.0,
+ "9": 458738688.0,
+ "10": 458737152.0,
+ "11": 458735104.0,
+ "12": 458734592.0,
+ "13": 458737664.0,
+ "14": 458737664.0,
+ "15": 458737664.0,
+ "16": 458739712.0,
+ "17": 458737152.0,
+ "18": 458735104.0,
+ "19": 458736128.0,
+ "20": 458736640.0,
+ "21": 458735616.0,
+ "22": 458736640.0,
+ "23": 458735616.0,
+ "24": 458737152.0,
+ "25": 458737152.0,
+ "26": 458737152.0,
+ "27": 458735616.0,
+ "28": 458737152.0,
+ "29": 458735104.0,
+ "30": 458736128.0,
+ "31": 458737664.0,
+ "32": 458736128.0,
+ "33": 458734592.0,
+ "34": 458738176.0,
+ "35": 458739200.0,
+ "36": 458735616.0,
+ "37": 458735104.0,
+ "38": 458736128.0,
+ "39": 458737664.0,
+ "40": 458735616.0,
+ "41": 458737664.0,
+ "42": 458735616.0,
+ "43": 458738688.0,
+ "44": 458737664.0,
+ "45": 458735616.0,
+ "46": 458738688.0,
+ "47": 458735616.0,
+ "48": 458736640.0,
+ "49": 458736640.0,
+ "50": 458734080.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1026068480.0,
- "2": 1192152064.0,
- "3": 1192152064.0,
- "4": 1192205312.0,
- "5": 1192205312.0,
- "6": 1192205312.0,
- "7": 1192205312.0,
- "8": 1192205312.0,
- "9": 1192205312.0,
- "10": 1192205312.0,
- "11": 1192205312.0,
- "12": 1192205312.0,
- "13": 1192349184.0,
- "14": 1192349184.0,
- "15": 1192506368.0,
- "16": 1192506368.0,
- "17": 1192506368.0,
- "18": 1192506368.0,
- "19": 1192506368.0,
- "20": 1192506368.0,
- "21": 1192506368.0,
- "22": 1192506368.0,
- "23": 1192506368.0,
- "24": 1192506368.0,
- "25": 1192506368.0,
- "26": 1192506368.0,
- "27": 1192506368.0,
- "28": 1192506368.0,
- "29": 1192506368.0,
- "30": 1192506368.0,
- "31": 1192506368.0,
- "32": 1192506368.0,
- "33": 1192506368.0,
- "34": 1192506368.0,
- "35": 1192506368.0,
- "36": 1192506368.0,
- "37": 1192506368.0,
- "38": 1192506368.0,
- "39": 1192506368.0,
- "40": 1192506368.0,
- "41": 1192506368.0,
- "42": 1192506368.0,
- "43": 1192506368.0,
- "44": 1192506368.0,
- "45": 1192506368.0,
- "46": 1192506368.0,
- "47": 1192506368.0,
- "48": 1192506368.0,
- "49": 1192506368.0,
- "50": 1192506368.0
+ "1": 1026235392.0,
+ "2": 1191006720.0,
+ "3": 1191138304.0,
+ "4": 1192048640.0,
+ "5": 1192048640.0,
+ "6": 1192048640.0,
+ "7": 1192048640.0,
+ "8": 1192048640.0,
+ "9": 1192048640.0,
+ "10": 1192327680.0,
+ "11": 1192465920.0,
+ "12": 1192465920.0,
+ "13": 1192641024.0,
+ "14": 1192641024.0,
+ "15": 1192641024.0,
+ "16": 1192641024.0,
+ "17": 1192641024.0,
+ "18": 1192641024.0,
+ "19": 1192641024.0,
+ "20": 1192641024.0,
+ "21": 1192641024.0,
+ "22": 1192641024.0,
+ "23": 1192641024.0,
+ "24": 1192641024.0,
+ "25": 1192641024.0,
+ "26": 1192641024.0,
+ "27": 1192641024.0,
+ "28": 1192641024.0,
+ "29": 1192641024.0,
+ "30": 1192641024.0,
+ "31": 1192641024.0,
+ "32": 1192641024.0,
+ "33": 1192641024.0,
+ "34": 1192641024.0,
+ "35": 1192641024.0,
+ "36": 1192641024.0,
+ "37": 1192641024.0,
+ "38": 1192641024.0,
+ "39": 1193289216.0,
+ "40": 1193289216.0,
+ "41": 1193289216.0,
+ "42": 1193289216.0,
+ "43": 1193289216.0,
+ "44": 1193289216.0,
+ "45": 1193289216.0,
+ "46": 1193289216.0,
+ "47": 1193289216.0,
+ "48": 1193289216.0,
+ "49": 1193289216.0,
+ "50": 1193289216.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 13.13083,
- "2": 0.49339,
- "3": 0.43067,
- "4": 0.43124,
- "5": 0.38622,
- "6": 0.39174,
- "7": 0.39833,
- "8": 0.39421,
- "9": 0.3937,
- "10": 0.38682,
- "11": 0.39333,
- "12": 0.38647,
- "13": 0.38364,
- "14": 0.38374,
- "15": 0.38593,
- "16": 0.38263,
- "17": 0.39915,
- "18": 0.38564,
- "19": 0.38954,
- "20": 0.38955,
- "21": 0.38216,
- "22": 0.38466,
- "23": 0.38551,
- "24": 0.38195,
- "25": 0.38416,
- "26": 0.38554,
- "27": 0.38123,
- "28": 0.38882,
- "29": 0.43011,
- "30": 0.38995,
- "31": 0.39202,
- "32": 0.38203,
- "33": 0.38777,
- "34": 0.39058,
- "35": 0.39634,
- "36": 0.38496,
- "37": 0.38112,
- "38": 0.38052,
- "39": 0.37771,
- "40": 0.38438,
- "41": 0.38696,
- "42": 0.38029,
- "43": 0.39638,
- "44": 0.38187,
- "45": 0.38285,
- "46": 0.42266,
- "47": 0.3977,
- "48": 0.39566,
- "49": 0.40884,
- "50": 0.38389
+ "1": "nan",
+ "2": 7.92806,
+ "3": 0.42884,
+ "4": 0.4181,
+ "5": 0.38785,
+ "6": 0.38798,
+ "7": 0.39804,
+ "8": 0.39885,
+ "9": 0.39837,
+ "10": 0.38582,
+ "11": 0.38128,
+ "12": 0.3877,
+ "13": 0.39005,
+ "14": 0.3857,
+ "15": 0.38979,
+ "16": 0.3855,
+ "17": 0.39486,
+ "18": 0.38449,
+ "19": 0.39881,
+ "20": 0.39407,
+ "21": 0.38587,
+ "22": 0.38917,
+ "23": 0.3851,
+ "24": 0.38917,
+ "25": 0.38635,
+ "26": 0.38598,
+ "27": 0.38618,
+ "28": 0.3898,
+ "29": 0.3871,
+ "30": 0.39096,
+ "31": 0.40269,
+ "32": 0.38598,
+ "33": 0.39054,
+ "34": 0.40412,
+ "35": 0.40047,
+ "36": 0.38707,
+ "37": 0.38889,
+ "38": 0.3882,
+ "39": 0.38375,
+ "40": 0.38903,
+ "41": 0.39487,
+ "42": 0.3921,
+ "43": 0.4011,
+ "44": 0.39501,
+ "45": 0.38847,
+ "46": 0.4285,
+ "47": 0.40517,
+ "48": 0.40246,
+ "49": 0.40981,
+ "50": 0.39109
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel_dp_last/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel_dp_last/model_config.yaml
index 4d674322a23..c48348735b8 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel_dp_last/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_etp2_te_4experts2parallel_dp_last/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -57,4 +57,6 @@ MODEL_ARGS:
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
--use-tp-pp-dp-mapping: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_resume_torch_dist_te_4experts2parallel/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_resume_torch_dist_te_4experts2parallel/model_config.yaml
index 86a05a93562..192f7d09101 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_resume_torch_dist_te_4experts2parallel/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_resume_torch_dist_te_4experts2parallel/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -55,4 +55,6 @@ MODEL_ARGS:
--disable-bias-linear: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_gb200.json
index 3e910ef7869..d3e7b4da4e1 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_gb200.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.79574,
"2": 10.81485,
- "3": 10.78713,
- "4": 10.78269,
- "5": 10.82015,
- "6": 10.83331,
- "7": 10.81116,
- "8": 10.81446,
- "9": 10.81645,
- "10": 10.75997,
- "11": 10.8388,
- "12": 10.81544,
- "13": 10.84141,
- "14": 10.8476,
- "15": 10.79857,
- "16": 10.78544,
- "17": 10.77004,
- "18": 10.77906,
- "19": 10.7689,
- "20": 10.71392,
- "21": 10.69182,
- "22": 10.56438,
- "23": 10.70939,
- "24": 10.60304,
- "25": 10.55748,
- "26": 10.60238,
- "27": 10.62835,
- "28": 10.59772,
- "29": 10.61013,
- "30": 10.40394,
- "31": 10.17092,
- "32": 10.49069,
- "33": 10.48436,
- "34": 10.26719,
- "35": 10.31532,
- "36": 10.27654,
- "37": 10.39353,
- "38": 10.24536,
- "39": 10.43863,
- "40": 10.13998,
- "41": 10.19151,
- "42": 10.25868,
- "43": 9.9191,
- "44": 10.03026,
- "45": 9.92187,
- "46": 9.89763,
- "47": 10.1946,
- "48": 9.93001,
- "49": 9.62787,
- "50": 9.97966
+ "3": 10.78727,
+ "4": 10.78304,
+ "5": 10.81989,
+ "6": 10.83351,
+ "7": 10.81125,
+ "8": 10.8139,
+ "9": 10.81641,
+ "10": 10.75982,
+ "11": 10.83861,
+ "12": 10.81549,
+ "13": 10.84169,
+ "14": 10.84748,
+ "15": 10.79858,
+ "16": 10.78574,
+ "17": 10.77021,
+ "18": 10.77928,
+ "19": 10.76898,
+ "20": 10.71384,
+ "21": 10.69186,
+ "22": 10.56455,
+ "23": 10.70953,
+ "24": 10.60188,
+ "25": 10.55758,
+ "26": 10.6026,
+ "27": 10.62881,
+ "28": 10.59778,
+ "29": 10.61003,
+ "30": 10.4043,
+ "31": 10.17042,
+ "32": 10.49112,
+ "33": 10.48455,
+ "34": 10.26721,
+ "35": 10.31534,
+ "36": 10.27691,
+ "37": 10.39297,
+ "38": 10.24497,
+ "39": 10.43848,
+ "40": 10.13979,
+ "41": 10.19223,
+ "42": 10.25887,
+ "43": 9.91922,
+ "44": 10.03027,
+ "45": 9.92197,
+ "46": 9.89723,
+ "47": 10.19434,
+ "48": 9.92968,
+ "49": 9.62843,
+ "50": 9.97977
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 13025.0,
- "2": 14911.0,
- "3": 14651.0,
- "4": 13760.0,
- "5": 16297.0,
- "6": 16032.0,
- "7": 15521.0,
- "8": 13170.0,
- "9": 15403.0,
- "10": 12605.0,
- "11": 16803.0,
- "12": 15289.0,
- "13": 16415.0,
- "14": 16182.0,
- "15": 15127.0,
- "16": 16135.0,
- "17": 15282.0,
- "18": 15280.0,
- "19": 15379.0,
- "20": 13642.0,
- "21": 14281.0,
- "22": 13476.0,
- "23": 16892.0,
- "24": 13920.0,
- "25": 13236.0,
- "26": 15256.0,
- "27": 15454.0,
- "28": 15973.0,
- "29": 16892.0,
- "30": 14103.0,
- "31": 13113.0,
- "32": 16067.0,
- "33": 16788.0,
- "34": 14559.0,
- "35": 14974.0,
- "36": 15798.0,
- "37": 17569.0,
- "38": 16172.0,
- "39": 17774.0,
- "40": 16088.0,
- "41": 16616.0,
- "42": 17149.0,
- "43": 15487.0,
- "44": 15110.0,
- "45": 16499.0,
- "46": 17407.0,
- "47": 19502.0,
- "48": 16568.0,
- "49": 16613.0,
- "50": 18892.0
+ "1": 12991.0,
+ "2": 14647.0,
+ "3": 14597.0,
+ "4": 13579.0,
+ "5": 16353.0,
+ "6": 16061.0,
+ "7": 15477.0,
+ "8": 13223.0,
+ "9": 15156.0,
+ "10": 12636.0,
+ "11": 16856.0,
+ "12": 15160.0,
+ "13": 16551.0,
+ "14": 16239.0,
+ "15": 15226.0,
+ "16": 16061.0,
+ "17": 15397.0,
+ "18": 15451.0,
+ "19": 15161.0,
+ "20": 13625.0,
+ "21": 14545.0,
+ "22": 13271.0,
+ "23": 16890.0,
+ "24": 13779.0,
+ "25": 13265.0,
+ "26": 14968.0,
+ "27": 15467.0,
+ "28": 16066.0,
+ "29": 16732.0,
+ "30": 14531.0,
+ "31": 13042.0,
+ "32": 16229.0,
+ "33": 16950.0,
+ "34": 14393.0,
+ "35": 14998.0,
+ "36": 15721.0,
+ "37": 17452.0,
+ "38": 16404.0,
+ "39": 17975.0,
+ "40": 16453.0,
+ "41": 16448.0,
+ "42": 17213.0,
+ "43": 15580.0,
+ "44": 14866.0,
+ "45": 16460.0,
+ "46": 17136.0,
+ "47": 19531.0,
+ "48": 16471.0,
+ "49": 16688.0,
+ "50": 18961.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 625796096.0,
- "2": 625850368.0,
- "3": 625987072.0,
- "4": 625831424.0,
- "5": 625794048.0,
- "6": 625789952.0,
- "7": 625830912.0,
- "8": 625794048.0,
- "9": 625861120.0,
- "10": 625806848.0,
- "11": 625795584.0,
- "12": 626022912.0,
- "13": 625802240.0,
- "14": 625853952.0,
- "15": 625796608.0,
- "16": 625793024.0,
- "17": 625798144.0,
- "18": 625802240.0,
- "19": 625792000.0,
- "20": 625793536.0,
- "21": 626690048.0,
- "22": 626176000.0,
- "23": 626092032.0,
- "24": 625794560.0,
- "25": 626540544.0,
- "26": 625934848.0,
- "27": 625799168.0,
- "28": 625801728.0,
- "29": 625793536.0,
- "30": 626191360.0,
- "31": 626149376.0,
- "32": 626774016.0,
- "33": 625792512.0,
- "34": 625793024.0,
- "35": 625851904.0,
- "36": 625809408.0,
- "37": 625794048.0,
- "38": 625827328.0,
- "39": 625865216.0,
- "40": 625831936.0,
- "41": 626081280.0,
- "42": 626046464.0,
- "43": 625792000.0,
- "44": 625792000.0,
- "45": 626266112.0,
- "46": 626042880.0,
- "47": 625789440.0,
- "48": 625905152.0,
- "49": 625883648.0,
- "50": 626099712.0
+ "1": 627893248.0,
+ "2": 628091904.0,
+ "3": 627973120.0,
+ "4": 627928576.0,
+ "5": 627940352.0,
+ "6": 627909120.0,
+ "7": 627932672.0,
+ "8": 628246528.0,
+ "9": 627952128.0,
+ "10": 627889664.0,
+ "11": 627977728.0,
+ "12": 628083200.0,
+ "13": 627912704.0,
+ "14": 627889664.0,
+ "15": 627917312.0,
+ "16": 627889664.0,
+ "17": 627895296.0,
+ "18": 627897344.0,
+ "19": 627950592.0,
+ "20": 627968000.0,
+ "21": 627994624.0,
+ "22": 627895296.0,
+ "23": 627894272.0,
+ "24": 627890688.0,
+ "25": 628020224.0,
+ "26": 628020736.0,
+ "27": 628694528.0,
+ "28": 627783680.0,
+ "29": 627811840.0,
+ "30": 627985408.0,
+ "31": 628240384.0,
+ "32": 627893248.0,
+ "33": 627946496.0,
+ "34": 627910656.0,
+ "35": 627893248.0,
+ "36": 627896320.0,
+ "37": 628513280.0,
+ "38": 627889152.0,
+ "39": 627890176.0,
+ "40": 627921920.0,
+ "41": 627889664.0,
+ "42": 627889664.0,
+ "43": 627888128.0,
+ "44": 627889664.0,
+ "45": 627887616.0,
+ "46": 628792320.0,
+ "47": 627899904.0,
+ "48": 627887616.0,
+ "49": 627982848.0,
+ "50": 627890176.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1844879360.0,
- "2": 2076399104.0,
- "3": 2076399104.0,
- "4": 2079827456.0,
- "5": 2079827456.0,
- "6": 2079827456.0,
- "7": 2079827456.0,
- "8": 2079827456.0,
- "9": 2079827456.0,
- "10": 2079827456.0,
- "11": 2079827456.0,
- "12": 2079827456.0,
- "13": 2079827456.0,
- "14": 2079827456.0,
- "15": 2079827456.0,
- "16": 2079827456.0,
- "17": 2079827456.0,
- "18": 2079827456.0,
- "19": 2079827456.0,
- "20": 2079827456.0,
- "21": 2079827456.0,
- "22": 2079827456.0,
- "23": 2079827456.0,
- "24": 2079827456.0,
- "25": 2079827456.0,
- "26": 2079827456.0,
- "27": 2079827456.0,
- "28": 2079827456.0,
- "29": 2079827456.0,
- "30": 2079827456.0,
- "31": 2079827456.0,
- "32": 2079827456.0,
- "33": 2079827456.0,
- "34": 2079827456.0,
- "35": 2079827456.0,
- "36": 2079827456.0,
- "37": 2079827456.0,
- "38": 2079827456.0,
- "39": 2079827456.0,
- "40": 2079827456.0,
- "41": 2079827456.0,
- "42": 2079827456.0,
- "43": 2079827456.0,
- "44": 2079827456.0,
- "45": 2079827456.0,
- "46": 2079827456.0,
- "47": 2079827456.0,
- "48": 2079827456.0,
- "49": 2079827456.0,
- "50": 2079827456.0
+ "1": 1844857856.0,
+ "2": 2078309888.0,
+ "3": 2078309888.0,
+ "4": 2081169920.0,
+ "5": 2081169920.0,
+ "6": 2081169920.0,
+ "7": 2081169920.0,
+ "8": 2081169920.0,
+ "9": 2081169920.0,
+ "10": 2081169920.0,
+ "11": 2081169920.0,
+ "12": 2081169920.0,
+ "13": 2081169920.0,
+ "14": 2081169920.0,
+ "15": 2081276416.0,
+ "16": 2081276416.0,
+ "17": 2081276416.0,
+ "18": 2081276416.0,
+ "19": 2081276416.0,
+ "20": 2081276416.0,
+ "21": 2081276416.0,
+ "22": 2081276416.0,
+ "23": 2081276416.0,
+ "24": 2081276416.0,
+ "25": 2081276416.0,
+ "26": 2081276416.0,
+ "27": 2081276416.0,
+ "28": 2081276416.0,
+ "29": 2081276416.0,
+ "30": 2081276416.0,
+ "31": 2081276416.0,
+ "32": 2081276416.0,
+ "33": 2081276416.0,
+ "34": 2081276416.0,
+ "35": 2081276416.0,
+ "36": 2081276416.0,
+ "37": 2081276416.0,
+ "38": 2081276416.0,
+ "39": 2081276416.0,
+ "40": 2081276416.0,
+ "41": 2081276416.0,
+ "42": 2081276416.0,
+ "43": 2081276416.0,
+ "44": 2081276416.0,
+ "45": 2081276416.0,
+ "46": 2081276416.0,
+ "47": 2081276416.0,
+ "48": 2081276416.0,
+ "49": 2081276416.0,
+ "50": 2081276416.0
}
},
"iteration-time": {
@@ -233,55 +233,55 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 5.62487,
- "3": 0.39428,
- "4": 0.3711,
- "5": 0.36777,
- "6": 0.36423,
- "7": 0.35408,
- "8": 0.35462,
- "9": 0.35588,
- "10": 0.35204,
- "11": 0.35155,
- "12": 0.35049,
- "13": 0.35818,
- "14": 0.35461,
- "15": 0.36874,
- "16": 0.367,
- "17": 0.37423,
- "18": 0.36926,
- "19": 0.37139,
- "20": 0.37109,
- "21": 0.37066,
- "22": 0.37237,
- "23": 0.37636,
- "24": 0.37618,
- "25": 0.37461,
- "26": 0.37622,
- "27": 0.37576,
- "28": 0.37551,
- "29": 0.3765,
- "30": 0.3787,
- "31": 0.38695,
- "32": 0.37235,
- "33": 0.37931,
- "34": 0.37817,
- "35": 0.3749,
- "36": 0.37829,
- "37": 0.37774,
- "38": 0.3755,
- "39": 0.37889,
- "40": 0.37688,
- "41": 0.38007,
- "42": 0.37324,
- "43": 0.36948,
- "44": 0.37523,
- "45": 0.37464,
- "46": 0.38496,
- "47": 0.3737,
- "48": 0.37892,
- "49": 0.39066,
- "50": 0.37612
+ "2": 8.65526,
+ "3": 0.42519,
+ "4": 0.40037,
+ "5": 0.39893,
+ "6": 0.39093,
+ "7": 0.39009,
+ "8": 0.39,
+ "9": 0.39299,
+ "10": 0.38898,
+ "11": 0.39153,
+ "12": 0.38826,
+ "13": 0.38884,
+ "14": 0.38722,
+ "15": 0.38852,
+ "16": 0.38716,
+ "17": 0.38566,
+ "18": 0.39003,
+ "19": 0.38566,
+ "20": 0.38156,
+ "21": 0.38572,
+ "22": 0.38689,
+ "23": 0.38747,
+ "24": 0.3855,
+ "25": 0.38305,
+ "26": 0.38467,
+ "27": 0.38312,
+ "28": 0.38663,
+ "29": 0.38453,
+ "30": 0.38017,
+ "31": 0.38989,
+ "32": 0.383,
+ "33": 0.38673,
+ "34": 0.38978,
+ "35": 0.38346,
+ "36": 0.38743,
+ "37": 0.38384,
+ "38": 0.38229,
+ "39": 0.38945,
+ "40": 0.3918,
+ "41": 0.38611,
+ "42": 0.3826,
+ "43": 0.38483,
+ "44": 0.38281,
+ "45": 0.38601,
+ "46": 0.3983,
+ "47": 0.38626,
+ "48": 0.39207,
+ "49": 0.39903,
+ "50": 0.38062
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_h100.json
index 8928145fcbb..a6938ff40f2 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_te_4experts2parallel/golden_values_dev_dgx_h100.json
@@ -6,54 +6,54 @@
"values": {
"1": 10.81692,
"2": 10.82534,
- "3": 10.82401,
- "4": 10.79801,
- "5": 10.8415,
- "6": 10.85912,
- "7": 10.81927,
- "8": 10.81789,
- "9": 10.83554,
- "10": 10.78266,
- "11": 10.85455,
- "12": 10.84582,
- "13": 10.84996,
- "14": 10.87821,
- "15": 10.80684,
- "16": 10.80662,
- "17": 10.76305,
- "18": 10.80188,
- "19": 10.79303,
- "20": 10.73474,
- "21": 10.71067,
- "22": 10.57636,
- "23": 10.7196,
- "24": 10.63305,
- "25": 10.56916,
- "26": 10.62589,
- "27": 10.64466,
- "28": 10.60792,
- "29": 10.61761,
- "30": 10.42214,
- "31": 10.17719,
- "32": 10.50701,
- "33": 10.50561,
- "34": 10.27485,
- "35": 10.3276,
- "36": 10.29275,
- "37": 10.40262,
- "38": 10.25679,
- "39": 10.43615,
- "40": 10.16589,
- "41": 10.20032,
- "42": 10.27424,
- "43": 9.93044,
- "44": 10.04415,
- "45": 9.92936,
- "46": 9.89984,
- "47": 10.18573,
- "48": 9.93082,
- "49": 9.6257,
- "50": 9.98437
+ "3": 10.82425,
+ "4": 10.79835,
+ "5": 10.84149,
+ "6": 10.85958,
+ "7": 10.81932,
+ "8": 10.81811,
+ "9": 10.8359,
+ "10": 10.78241,
+ "11": 10.85501,
+ "12": 10.845,
+ "13": 10.8499,
+ "14": 10.87795,
+ "15": 10.80637,
+ "16": 10.80682,
+ "17": 10.76314,
+ "18": 10.80211,
+ "19": 10.79309,
+ "20": 10.73505,
+ "21": 10.71033,
+ "22": 10.57666,
+ "23": 10.71956,
+ "24": 10.63346,
+ "25": 10.56915,
+ "26": 10.62644,
+ "27": 10.64462,
+ "28": 10.60798,
+ "29": 10.61786,
+ "30": 10.42195,
+ "31": 10.17755,
+ "32": 10.50694,
+ "33": 10.50593,
+ "34": 10.27534,
+ "35": 10.32709,
+ "36": 10.29294,
+ "37": 10.40264,
+ "38": 10.25671,
+ "39": 10.43584,
+ "40": 10.1662,
+ "41": 10.20061,
+ "42": 10.27446,
+ "43": 9.92989,
+ "44": 10.04353,
+ "45": 9.92946,
+ "46": 9.89981,
+ "47": 10.18589,
+ "48": 9.93134,
+ "49": 9.62589,
+ "50": 9.98435
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 12899.0,
- "2": 14592.0,
- "3": 14243.0,
- "4": 13886.0,
- "5": 15732.0,
- "6": 16250.0,
- "7": 15453.0,
- "8": 13386.0,
- "9": 15159.0,
- "10": 12804.0,
- "11": 16441.0,
- "12": 14951.0,
- "13": 16151.0,
- "14": 16330.0,
- "15": 15144.0,
- "16": 15588.0,
- "17": 15315.0,
- "18": 14902.0,
- "19": 15436.0,
- "20": 13814.0,
- "21": 13977.0,
- "22": 12814.0,
- "23": 16615.0,
- "24": 13785.0,
- "25": 13451.0,
- "26": 14681.0,
- "27": 15288.0,
- "28": 16290.0,
- "29": 16880.0,
- "30": 14583.0,
- "31": 13272.0,
- "32": 15972.0,
- "33": 16904.0,
- "34": 14406.0,
- "35": 14981.0,
- "36": 15576.0,
- "37": 17584.0,
- "38": 16136.0,
- "39": 17650.0,
- "40": 16506.0,
- "41": 16391.0,
- "42": 17008.0,
- "43": 15459.0,
- "44": 15097.0,
- "45": 16136.0,
- "46": 16845.0,
- "47": 19101.0,
- "48": 16405.0,
- "49": 16558.0,
- "50": 18439.0
+ "1": 12988.0,
+ "2": 14581.0,
+ "3": 14629.0,
+ "4": 13993.0,
+ "5": 16038.0,
+ "6": 16416.0,
+ "7": 15537.0,
+ "8": 13088.0,
+ "9": 14947.0,
+ "10": 12672.0,
+ "11": 16430.0,
+ "12": 15080.0,
+ "13": 15805.0,
+ "14": 16282.0,
+ "15": 15182.0,
+ "16": 15760.0,
+ "17": 15427.0,
+ "18": 15046.0,
+ "19": 15395.0,
+ "20": 13537.0,
+ "21": 13941.0,
+ "22": 12878.0,
+ "23": 16723.0,
+ "24": 14054.0,
+ "25": 13061.0,
+ "26": 14780.0,
+ "27": 15479.0,
+ "28": 16079.0,
+ "29": 16571.0,
+ "30": 14763.0,
+ "31": 12985.0,
+ "32": 15925.0,
+ "33": 16870.0,
+ "34": 14353.0,
+ "35": 15089.0,
+ "36": 15352.0,
+ "37": 17464.0,
+ "38": 16285.0,
+ "39": 17957.0,
+ "40": 16621.0,
+ "41": 16454.0,
+ "42": 16859.0,
+ "43": 15314.0,
+ "44": 15307.0,
+ "45": 16417.0,
+ "46": 17005.0,
+ "47": 19138.0,
+ "48": 16601.0,
+ "49": 16564.0,
+ "50": 18745.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 625530880.0,
- "2": 625483264.0,
- "3": 625484800.0,
- "4": 625516032.0,
- "5": 625759744.0,
- "6": 625774592.0,
- "7": 625485312.0,
- "8": 625568256.0,
- "9": 625519616.0,
- "10": 625655808.0,
- "11": 625630720.0,
- "12": 625482240.0,
- "13": 625488384.0,
- "14": 625819136.0,
- "15": 625982976.0,
- "16": 625500160.0,
- "17": 625613312.0,
- "18": 625494016.0,
- "19": 625484288.0,
- "20": 625508864.0,
- "21": 625486336.0,
- "22": 625486848.0,
- "23": 625632768.0,
- "24": 625487872.0,
- "25": 625484288.0,
- "26": 625753088.0,
- "27": 625513984.0,
- "28": 625483264.0,
- "29": 625698816.0,
- "30": 625967104.0,
- "31": 625477632.0,
- "32": 625523200.0,
- "33": 625484288.0,
- "34": 625481216.0,
- "35": 625479680.0,
- "36": 625554432.0,
- "37": 625554944.0,
- "38": 625487360.0,
- "39": 625504768.0,
- "40": 625481216.0,
- "41": 625481728.0,
- "42": 625481728.0,
- "43": 626760192.0,
- "44": 625598464.0,
- "45": 625534464.0,
- "46": 625603072.0,
- "47": 625509376.0,
- "48": 626520576.0,
- "49": 625630720.0,
- "50": 625565696.0
+ "1": 626041856.0,
+ "2": 626082304.0,
+ "3": 626342912.0,
+ "4": 627062784.0,
+ "5": 626040320.0,
+ "6": 626118656.0,
+ "7": 626089984.0,
+ "8": 626075136.0,
+ "9": 626074624.0,
+ "10": 626041856.0,
+ "11": 626954240.0,
+ "12": 626144768.0,
+ "13": 626040832.0,
+ "14": 626037760.0,
+ "15": 626042368.0,
+ "16": 626081280.0,
+ "17": 626153984.0,
+ "18": 626056192.0,
+ "19": 626041344.0,
+ "20": 626070528.0,
+ "21": 626043904.0,
+ "22": 626040832.0,
+ "23": 626071040.0,
+ "24": 626407936.0,
+ "25": 626120704.0,
+ "26": 626217984.0,
+ "27": 626289664.0,
+ "28": 626401280.0,
+ "29": 626043904.0,
+ "30": 626057216.0,
+ "31": 626034688.0,
+ "32": 626039808.0,
+ "33": 626041856.0,
+ "34": 626091520.0,
+ "35": 626880512.0,
+ "36": 626065408.0,
+ "37": 626088448.0,
+ "38": 626063872.0,
+ "39": 626053632.0,
+ "40": 626297344.0,
+ "41": 626072576.0,
+ "42": 626039808.0,
+ "43": 626035200.0,
+ "44": 626271232.0,
+ "45": 626083328.0,
+ "46": 626032128.0,
+ "47": 626070528.0,
+ "48": 626230272.0,
+ "49": 626186752.0,
+ "50": 626039808.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1845331456.0,
- "2": 2075684352.0,
- "3": 2075684352.0,
- "4": 2078547456.0,
- "5": 2078547456.0,
- "6": 2078547456.0,
- "7": 2078547456.0,
- "8": 2078547456.0,
- "9": 2078547456.0,
- "10": 2078547456.0,
- "11": 2078547456.0,
- "12": 2078547456.0,
- "13": 2078547456.0,
- "14": 2078547456.0,
- "15": 2078547456.0,
- "16": 2078547456.0,
- "17": 2078547456.0,
- "18": 2078547456.0,
- "19": 2078547456.0,
- "20": 2078547456.0,
- "21": 2078547456.0,
- "22": 2078547456.0,
- "23": 2078547456.0,
- "24": 2078547456.0,
- "25": 2078547456.0,
- "26": 2078547456.0,
- "27": 2078547456.0,
- "28": 2078547456.0,
- "29": 2078547456.0,
- "30": 2078547456.0,
- "31": 2078547456.0,
- "32": 2078547456.0,
- "33": 2078547456.0,
- "34": 2078547456.0,
- "35": 2078547456.0,
- "36": 2078547456.0,
- "37": 2078547456.0,
- "38": 2078547456.0,
- "39": 2078547456.0,
- "40": 2078547456.0,
- "41": 2078547456.0,
- "42": 2078547456.0,
- "43": 2078547456.0,
- "44": 2078547456.0,
- "45": 2078547456.0,
- "46": 2078547456.0,
- "47": 2078547456.0,
- "48": 2078547456.0,
- "49": 2078547456.0,
- "50": 2078547456.0
+ "1": 1844949504.0,
+ "2": 2075497984.0,
+ "3": 2076485632.0,
+ "4": 2078283776.0,
+ "5": 2078283776.0,
+ "6": 2078283776.0,
+ "7": 2078283776.0,
+ "8": 2078283776.0,
+ "9": 2078283776.0,
+ "10": 2078283776.0,
+ "11": 2078283776.0,
+ "12": 2078283776.0,
+ "13": 2078292480.0,
+ "14": 2078292480.0,
+ "15": 2078292480.0,
+ "16": 2078292480.0,
+ "17": 2078292480.0,
+ "18": 2078292480.0,
+ "19": 2078292480.0,
+ "20": 2078292480.0,
+ "21": 2078387200.0,
+ "22": 2078387200.0,
+ "23": 2078387200.0,
+ "24": 2078387200.0,
+ "25": 2078387200.0,
+ "26": 2078387200.0,
+ "27": 2078387200.0,
+ "28": 2078387200.0,
+ "29": 2078387200.0,
+ "30": 2078490624.0,
+ "31": 2078490624.0,
+ "32": 2078490624.0,
+ "33": 2078490624.0,
+ "34": 2078490624.0,
+ "35": 2078490624.0,
+ "36": 2078490624.0,
+ "37": 2078490624.0,
+ "38": 2078490624.0,
+ "39": 2078490624.0,
+ "40": 2078490624.0,
+ "41": 2078490624.0,
+ "42": 2078490624.0,
+ "43": 2078490624.0,
+ "44": 2078490624.0,
+ "45": 2078490624.0,
+ "46": 2078490624.0,
+ "47": 2078490624.0,
+ "48": 2078490624.0,
+ "49": 2078490624.0,
+ "50": 2078490624.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 14.22688,
- "2": 0.36404,
- "3": 0.28777,
- "4": 0.27054,
- "5": 0.24844,
- "6": 0.23753,
- "7": 0.2541,
- "8": 0.2395,
- "9": 0.23675,
- "10": 0.23301,
- "11": 0.25454,
- "12": 0.22665,
- "13": 0.23214,
- "14": 0.22521,
- "15": 0.24748,
- "16": 0.2636,
- "17": 0.2605,
- "18": 0.24164,
- "19": 0.24627,
- "20": 0.25668,
- "21": 0.24329,
- "22": 0.24722,
- "23": 0.25378,
- "24": 0.22642,
- "25": 0.22497,
- "26": 0.22495,
- "27": 0.2239,
- "28": 0.22848,
- "29": 0.22515,
- "30": 0.22501,
- "31": 0.27252,
- "32": 0.22744,
- "33": 0.22453,
- "34": 0.23411,
- "35": 0.22556,
- "36": 0.2278,
- "37": 0.22109,
- "38": 0.22459,
- "39": 0.22077,
- "40": 0.22097,
- "41": 0.23428,
- "42": 0.22009,
- "43": 0.23227,
- "44": 0.22717,
- "45": 0.23445,
- "46": 0.23886,
- "47": 0.22667,
- "48": 0.23204,
- "49": 0.27864,
- "50": 0.22287
+ "1": "nan",
+ "2": 7.53997,
+ "3": 0.27369,
+ "4": 0.2629,
+ "5": 0.24127,
+ "6": 0.23879,
+ "7": 0.2486,
+ "8": 0.22975,
+ "9": 0.23492,
+ "10": 0.22756,
+ "11": 0.23734,
+ "12": 0.22618,
+ "13": 0.23035,
+ "14": 0.22237,
+ "15": 0.24274,
+ "16": 0.22308,
+ "17": 0.23173,
+ "18": 0.22258,
+ "19": 0.22214,
+ "20": 0.22713,
+ "21": 0.21893,
+ "22": 0.22057,
+ "23": 0.23133,
+ "24": 0.22415,
+ "25": 0.22094,
+ "26": 0.21957,
+ "27": 0.22081,
+ "28": 0.21989,
+ "29": 0.21978,
+ "30": 0.21904,
+ "31": 0.26218,
+ "32": 0.2204,
+ "33": 0.22048,
+ "34": 0.2318,
+ "35": 0.21999,
+ "36": 0.23031,
+ "37": 0.21978,
+ "38": 0.22301,
+ "39": 0.21796,
+ "40": 0.22181,
+ "41": 0.23147,
+ "42": 0.218,
+ "43": 0.2244,
+ "44": 0.22,
+ "45": 0.22268,
+ "46": 0.24114,
+ "47": 0.22482,
+ "48": 0.22419,
+ "49": 0.29207,
+ "50": 0.23757
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_te_4experts2parallel/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_te_4experts2parallel/model_config.yaml
index 5020d9d9397..f4b020370ff 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_te_4experts2parallel/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_ep2_te_4experts2parallel/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -56,4 +56,6 @@ MODEL_ARGS:
--disable-bias-linear: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular
diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_resume_torch_dist_te_2experts/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_resume_torch_dist_te_2experts/model_config.yaml
index c93bd4367f3..cf0f282e5b1 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_resume_torch_dist_te_2experts/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_tp2_pp2_resume_torch_dist_te_2experts/model_config.yaml
@@ -21,9 +21,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -55,4 +55,6 @@ MODEL_ARGS:
--disable-bias-linear: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon/golden_values_dev_dgx_gb200.json
index d9441fb83aa..632da5adb41 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon/golden_values_dev_dgx_gb200.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.80397,
- "2": 10.81064,
- "3": 10.798,
- "4": 10.78655,
- "5": 10.8299,
- "6": 10.85307,
- "7": 10.80967,
- "8": 10.8015,
- "9": 10.82484,
- "10": 10.78237,
- "11": 10.83805,
- "12": 10.84658,
- "13": 10.86137,
- "14": 10.86451,
- "15": 10.83791,
- "16": 10.83385,
- "17": 10.81249,
- "18": 10.84432,
- "19": 10.83623,
- "20": 10.8168,
- "21": 10.83551,
- "22": 10.76274,
- "23": 10.85573,
- "24": 10.8062,
- "25": 10.80042,
- "26": 10.8143,
- "27": 10.82932,
- "28": 10.8546,
- "29": 10.86594,
- "30": 10.79379,
- "31": 10.74776,
- "32": 10.84932,
- "33": 10.83727,
- "34": 10.80597,
- "35": 10.80259,
- "36": 10.79662,
- "37": 10.82559,
- "38": 10.79231,
- "39": 10.84778,
- "40": 10.77804,
- "41": 10.79895,
- "42": 10.81493,
- "43": 10.74316,
- "44": 10.76656,
- "45": 10.76408,
- "46": 10.7768,
- "47": 10.79908,
- "48": 10.77572,
- "49": 10.72207,
- "50": 10.78609,
- "51": 10.78712,
- "52": 10.7653,
- "53": 10.81235,
- "54": 10.79776,
- "55": 10.8072,
- "56": 10.7562,
- "57": 10.71334,
- "58": 10.78166,
- "59": 10.75039,
- "60": 10.72977,
- "61": 10.76435,
- "62": 10.81299,
- "63": 10.69266,
- "64": 10.76646,
- "65": 10.62484,
- "66": 10.75371,
- "67": 10.69118,
- "68": 10.77122,
- "69": 10.76048,
- "70": 10.76506,
- "71": 10.73497,
- "72": 10.72999,
- "73": 10.71715,
- "74": 10.57819,
- "75": 10.68208,
- "76": 10.6133,
- "77": 10.80786,
- "78": 10.73142,
- "79": 10.66063,
- "80": 10.68014,
- "81": 10.69828,
- "82": 10.72277,
- "83": 10.64104,
- "84": 10.66223,
- "85": 10.70251,
- "86": 10.57982,
- "87": 10.69083,
- "88": 10.73435,
- "89": 10.67796,
- "90": 10.74299,
- "91": 10.62241,
- "92": 10.64011,
- "93": 10.56628,
- "94": 10.49922,
- "95": 10.65675,
- "96": 10.65892,
- "97": 10.57941,
- "98": 10.67242,
- "99": 10.47965,
- "100": 10.59346
+ "1": 10.94254,
+ "2": 10.91055,
+ "3": 10.92749,
+ "4": 10.91714,
+ "5": 10.92102,
+ "6": 10.92742,
+ "7": 10.92275,
+ "8": 10.92023,
+ "9": 10.93267,
+ "10": 10.92713,
+ "11": 10.91817,
+ "12": 10.93474,
+ "13": 10.91968,
+ "14": 10.91019,
+ "15": 10.91861,
+ "16": 10.92068,
+ "17": 10.93501,
+ "18": 10.91631,
+ "19": 10.92772,
+ "20": 10.90291,
+ "21": 10.92688,
+ "22": 10.92298,
+ "23": 10.92978,
+ "24": 10.90443,
+ "25": 10.91399,
+ "26": 10.91366,
+ "27": 10.9297,
+ "28": 10.91921,
+ "29": 10.92045,
+ "30": 10.91206,
+ "31": 10.90628,
+ "32": 10.91853,
+ "33": 10.91624,
+ "34": 10.90372,
+ "35": 10.92707,
+ "36": 10.90711,
+ "37": 10.90561,
+ "38": 10.91719,
+ "39": 10.90998,
+ "40": 10.90346,
+ "41": 10.91124,
+ "42": 10.91566,
+ "43": 10.90796,
+ "44": 10.91688,
+ "45": 10.91036,
+ "46": 10.90611,
+ "47": 10.90071,
+ "48": 10.89355,
+ "49": 10.9046,
+ "50": 10.89361,
+ "51": 10.89863,
+ "52": 10.89701,
+ "53": 10.90064,
+ "54": 10.89127,
+ "55": 10.90659,
+ "56": 10.88375,
+ "57": 10.88811,
+ "58": 10.8802,
+ "59": 10.87887,
+ "60": 10.86686,
+ "61": 10.87051,
+ "62": 10.86349,
+ "63": 10.87017,
+ "64": 10.86218,
+ "65": 10.86533,
+ "66": 10.86787,
+ "67": 10.85597,
+ "68": 10.8481,
+ "69": 10.85361,
+ "70": 10.85688,
+ "71": 10.85026,
+ "72": 10.84978,
+ "73": 10.83918,
+ "74": 10.83515,
+ "75": 10.84164,
+ "76": 10.82842,
+ "77": 10.82972,
+ "78": 10.81244,
+ "79": 10.82921,
+ "80": 10.8342,
+ "81": 10.8158,
+ "82": 10.81396,
+ "83": 10.80416,
+ "84": 10.78077,
+ "85": 10.78142,
+ "86": 10.79777,
+ "87": 10.79943,
+ "88": 10.80181,
+ "89": 10.78453,
+ "90": 10.78191,
+ "91": 10.7827,
+ "92": 10.78237,
+ "93": 10.75808,
+ "94": 10.76788,
+ "95": 10.76574,
+ "96": 10.73902,
+ "97": 10.73031,
+ "98": 10.75382,
+ "99": 10.76047,
+ "100": 10.73769
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1158.0,
- "2": 1181.0,
- "3": 1227.0,
- "4": 1263.0,
- "5": 1308.0,
- "6": 1594.0,
- "7": 1345.0,
- "8": 1253.0,
- "9": 1239.0,
- "10": 1232.0,
- "11": 1255.0,
- "12": 1250.0,
- "13": 1516.0,
- "14": 1270.0,
- "15": 1335.0,
- "16": 1260.0,
- "17": 1249.0,
- "18": 1261.0,
- "19": 1099.0,
- "20": 1387.0,
- "21": 1298.0,
- "22": 1286.0,
- "23": 1315.0,
- "24": 1131.0,
- "25": 1157.0,
- "26": 1166.0,
- "27": 1133.0,
- "28": 1243.0,
- "29": 1348.0,
- "30": 1235.0,
- "31": 1090.0,
- "32": 1272.0,
- "33": 1355.0,
- "34": 1161.0,
- "35": 1159.0,
- "36": 1146.0,
- "37": 1222.0,
- "38": 1418.0,
- "39": 1273.0,
- "40": 1198.0,
- "41": 1160.0,
- "42": 1285.0,
- "43": 1094.0,
- "44": 1127.0,
- "45": 1130.0,
- "46": 1183.0,
- "47": 1312.0,
- "48": 1238.0,
- "49": 1068.0,
- "50": 1163.0,
- "51": 1234.0,
- "52": 1284.0,
- "53": 1380.0,
- "54": 1191.0,
- "55": 1099.0,
- "56": 1298.0,
- "57": 1241.0,
- "58": 1203.0,
- "59": 1324.0,
- "60": 1254.0,
- "61": 1120.0,
- "62": 1362.0,
- "63": 1213.0,
- "64": 1330.0,
- "65": 1057.0,
- "66": 1171.0,
- "67": 1208.0,
- "68": 1320.0,
- "69": 1304.0,
- "70": 1122.0,
- "71": 1259.0,
- "72": 1254.0,
- "73": 1203.0,
- "74": 1125.0,
- "75": 1413.0,
- "76": 1217.0,
- "77": 1412.0,
- "78": 1291.0,
- "79": 1020.0,
- "80": 1143.0,
- "81": 1243.0,
- "82": 1154.0,
- "83": 1052.0,
- "84": 1219.0,
- "85": 1360.0,
- "86": 1072.0,
- "87": 1319.0,
- "88": 1347.0,
- "89": 1127.0,
- "90": 1474.0,
- "91": 1140.0,
- "92": 1110.0,
- "93": 924.0,
- "94": 1062.0,
- "95": 1147.0,
- "96": 1128.0,
- "97": 1099.0,
- "98": 1191.0,
- "99": 1071.0,
- "100": 1214.0
+ "1": 1411.0,
+ "2": 1137.0,
+ "3": 1245.0,
+ "4": 1282.0,
+ "5": 1199.0,
+ "6": 1320.0,
+ "7": 1597.0,
+ "8": 1354.0,
+ "9": 1347.0,
+ "10": 1253.0,
+ "11": 1247.0,
+ "12": 1319.0,
+ "13": 1416.0,
+ "14": 1393.0,
+ "15": 1215.0,
+ "16": 1254.0,
+ "17": 1321.0,
+ "18": 1267.0,
+ "19": 1233.0,
+ "20": 1261.0,
+ "21": 1481.0,
+ "22": 1305.0,
+ "23": 1249.0,
+ "24": 1506.0,
+ "25": 1246.0,
+ "26": 1353.0,
+ "27": 1282.0,
+ "28": 1255.0,
+ "29": 1363.0,
+ "30": 1336.0,
+ "31": 1373.0,
+ "32": 1515.0,
+ "33": 1471.0,
+ "34": 1320.0,
+ "35": 1417.0,
+ "36": 1221.0,
+ "37": 1335.0,
+ "38": 1284.0,
+ "39": 1226.0,
+ "40": 1314.0,
+ "41": 1253.0,
+ "42": 1037.0,
+ "43": 1387.0,
+ "44": 1391.0,
+ "45": 1300.0,
+ "46": 1306.0,
+ "47": 1334.0,
+ "48": 1392.0,
+ "49": 1394.0,
+ "50": 1198.0,
+ "51": 1196.0,
+ "52": 1233.0,
+ "53": 1317.0,
+ "54": 1174.0,
+ "55": 1254.0,
+ "56": 1215.0,
+ "57": 1248.0,
+ "58": 1284.0,
+ "59": 1336.0,
+ "60": 1393.0,
+ "61": 1200.0,
+ "62": 1347.0,
+ "63": 1304.0,
+ "64": 1319.0,
+ "65": 1320.0,
+ "66": 1160.0,
+ "67": 1423.0,
+ "68": 1191.0,
+ "69": 1337.0,
+ "70": 1271.0,
+ "71": 1124.0,
+ "72": 1302.0,
+ "73": 1248.0,
+ "74": 1310.0,
+ "75": 1303.0,
+ "76": 1394.0,
+ "77": 1291.0,
+ "78": 1174.0,
+ "79": 1190.0,
+ "80": 1255.0,
+ "81": 1259.0,
+ "82": 1127.0,
+ "83": 1355.0,
+ "84": 1266.0,
+ "85": 1192.0,
+ "86": 1274.0,
+ "87": 1122.0,
+ "88": 1221.0,
+ "89": 1196.0,
+ "90": 1267.0,
+ "91": 1200.0,
+ "92": 1099.0,
+ "93": 1297.0,
+ "94": 1364.0,
+ "95": 1215.0,
+ "96": 1230.0,
+ "97": 1284.0,
+ "98": 1208.0,
+ "99": 1230.0,
+ "100": 1397.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 993802240.0,
- "2": 993814528.0,
- "3": 993790464.0,
- "4": 993820160.0,
- "5": 993728512.0,
- "6": 993758208.0,
- "7": 993780224.0,
- "8": 993795584.0,
- "9": 993800704.0,
- "10": 993771520.0,
- "11": 993752064.0,
- "12": 993737216.0,
- "13": 993804800.0,
- "14": 993734144.0,
- "15": 993773056.0,
- "16": 993763840.0,
- "17": 993744384.0,
- "18": 993743872.0,
- "19": 993764864.0,
- "20": 993719296.0,
- "21": 993772544.0,
- "22": 993796096.0,
- "23": 993748992.0,
- "24": 993819136.0,
- "25": 993792512.0,
- "26": 993776640.0,
- "27": 993804800.0,
- "28": 993701888.0,
- "29": 993717248.0,
- "30": 993766912.0,
- "31": 993802240.0,
- "32": 993705984.0,
- "33": 993759744.0,
- "34": 993780224.0,
- "35": 993740288.0,
- "36": 993748480.0,
- "37": 993697280.0,
- "38": 993763840.0,
- "39": 993747456.0,
- "40": 993781248.0,
- "41": 993752064.0,
- "42": 993757696.0,
- "43": 993793024.0,
- "44": 993833984.0,
- "45": 993780736.0,
- "46": 993798144.0,
- "47": 993789440.0,
- "48": 993793024.0,
- "49": 993743360.0,
- "50": 993754624.0,
- "51": 993786368.0,
- "52": 993749504.0,
- "53": 993812992.0,
- "54": 993750528.0,
- "55": 993732608.0,
- "56": 993777664.0,
- "57": 993859584.0,
- "58": 993849856.0,
- "59": 993761792.0,
- "60": 993774592.0,
- "61": 993771520.0,
- "62": 993786880.0,
- "63": 993787904.0,
- "64": 993761280.0,
- "65": 993792000.0,
- "66": 993746432.0,
- "67": 993782784.0,
- "68": 993783808.0,
- "69": 993741824.0,
- "70": 993747968.0,
- "71": 993736192.0,
- "72": 993762304.0,
- "73": 993784832.0,
- "74": 993809920.0,
- "75": 993753088.0,
- "76": 993797632.0,
- "77": 993720832.0,
- "78": 993729536.0,
- "79": 993730560.0,
- "80": 993763840.0,
- "81": 993728000.0,
- "82": 993711616.0,
- "83": 993772544.0,
- "84": 993782784.0,
- "85": 993787392.0,
- "86": 993804288.0,
- "87": 993737728.0,
- "88": 993731584.0,
- "89": 993755136.0,
- "90": 993742848.0,
- "91": 993763840.0,
- "92": 993774080.0,
- "93": 993792000.0,
- "94": 993779712.0,
- "95": 993723904.0,
- "96": 993714688.0,
- "97": 993752064.0,
- "98": 993708544.0,
- "99": 993792000.0,
- "100": 993812992.0
+ "1": 993735680.0,
+ "2": 993750016.0,
+ "3": 993740288.0,
+ "4": 993725440.0,
+ "5": 993733632.0,
+ "6": 993719296.0,
+ "7": 993740800.0,
+ "8": 993708544.0,
+ "9": 993737216.0,
+ "10": 993731072.0,
+ "11": 993708544.0,
+ "12": 993700352.0,
+ "13": 993785344.0,
+ "14": 993800192.0,
+ "15": 993735680.0,
+ "16": 993735680.0,
+ "17": 993747456.0,
+ "18": 993707520.0,
+ "19": 993767424.0,
+ "20": 993732096.0,
+ "21": 993782272.0,
+ "22": 993731584.0,
+ "23": 993708032.0,
+ "24": 993745408.0,
+ "25": 993696768.0,
+ "26": 993811456.0,
+ "27": 993737728.0,
+ "28": 993734656.0,
+ "29": 993771520.0,
+ "30": 993695232.0,
+ "31": 993728512.0,
+ "32": 993733632.0,
+ "33": 993721344.0,
+ "34": 993783296.0,
+ "35": 993720832.0,
+ "36": 993656320.0,
+ "37": 993731072.0,
+ "38": 993759232.0,
+ "39": 993702400.0,
+ "40": 993737728.0,
+ "41": 993740288.0,
+ "42": 993732096.0,
+ "43": 993736192.0,
+ "44": 993706496.0,
+ "45": 993702912.0,
+ "46": 993763328.0,
+ "47": 993757184.0,
+ "48": 993702400.0,
+ "49": 993729536.0,
+ "50": 993680384.0,
+ "51": 993666560.0,
+ "52": 993689088.0,
+ "53": 993729536.0,
+ "54": 993729536.0,
+ "55": 993711104.0,
+ "56": 993721344.0,
+ "57": 993755648.0,
+ "58": 993790976.0,
+ "59": 993697792.0,
+ "60": 993711616.0,
+ "61": 993682944.0,
+ "62": 993712640.0,
+ "63": 993725440.0,
+ "64": 993729024.0,
+ "65": 993713664.0,
+ "66": 993650688.0,
+ "67": 993729536.0,
+ "68": 993742336.0,
+ "69": 993720832.0,
+ "70": 993671168.0,
+ "71": 993696768.0,
+ "72": 993766400.0,
+ "73": 993696768.0,
+ "74": 993694208.0,
+ "75": 993733120.0,
+ "76": 993709056.0,
+ "77": 993723392.0,
+ "78": 993714176.0,
+ "79": 993705984.0,
+ "80": 993723392.0,
+ "81": 993709056.0,
+ "82": 993681920.0,
+ "83": 993739264.0,
+ "84": 993731072.0,
+ "85": 993740800.0,
+ "86": 993717248.0,
+ "87": 993613824.0,
+ "88": 993668608.0,
+ "89": 993677824.0,
+ "90": 993700864.0,
+ "91": 993731584.0,
+ "92": 993679872.0,
+ "93": 993647616.0,
+ "94": 993688576.0,
+ "95": 993676800.0,
+ "96": 993650688.0,
+ "97": 993678848.0,
+ "98": 993743360.0,
+ "99": 993701888.0,
+ "100": 993733632.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 3074847232.0,
- "2": 3373968896.0,
- "3": 3373968896.0,
- "4": 3378071552.0,
- "5": 3378071552.0,
- "6": 3378071552.0,
- "7": 3378071552.0,
- "8": 3378071552.0,
- "9": 3378071552.0,
- "10": 3378071552.0,
- "11": 3378071552.0,
- "12": 3378071552.0,
- "13": 3378071552.0,
- "14": 3378071552.0,
- "15": 3378071552.0,
- "16": 3378071552.0,
- "17": 3378071552.0,
- "18": 3378071552.0,
- "19": 3378071552.0,
- "20": 3378071552.0,
- "21": 3378071552.0,
- "22": 3378071552.0,
- "23": 3378071552.0,
- "24": 3378894848.0,
- "25": 3378894848.0,
- "26": 3378894848.0,
- "27": 3378894848.0,
- "28": 3378894848.0,
- "29": 3378894848.0,
- "30": 3378894848.0,
- "31": 3378894848.0,
- "32": 3378894848.0,
- "33": 3378894848.0,
- "34": 3378894848.0,
- "35": 3378894848.0,
- "36": 3378894848.0,
- "37": 3378894848.0,
- "38": 3378894848.0,
- "39": 3378894848.0,
- "40": 3378894848.0,
- "41": 3378894848.0,
- "42": 3378894848.0,
- "43": 3378894848.0,
- "44": 3392753152.0,
- "45": 3392753152.0,
- "46": 3392753152.0,
- "47": 3392753152.0,
- "48": 3392753152.0,
- "49": 3392753152.0,
- "50": 3392753152.0,
- "51": 3392753152.0,
- "52": 3392753152.0,
- "53": 3392753152.0,
- "54": 3392753152.0,
- "55": 3392753152.0,
- "56": 3392753152.0,
- "57": 3407671808.0,
- "58": 3407671808.0,
- "59": 3407671808.0,
- "60": 3407671808.0,
- "61": 3407671808.0,
- "62": 3407671808.0,
- "63": 3407671808.0,
- "64": 3407671808.0,
- "65": 3407671808.0,
- "66": 3407671808.0,
- "67": 3407671808.0,
- "68": 3407671808.0,
- "69": 3407671808.0,
- "70": 3407671808.0,
- "71": 3407671808.0,
- "72": 3407671808.0,
- "73": 3407671808.0,
- "74": 3407671808.0,
- "75": 3407671808.0,
- "76": 3407671808.0,
- "77": 3407671808.0,
- "78": 3407671808.0,
- "79": 3407671808.0,
- "80": 3407671808.0,
- "81": 3407671808.0,
- "82": 3407671808.0,
- "83": 3407671808.0,
- "84": 3407671808.0,
- "85": 3407671808.0,
- "86": 3407671808.0,
- "87": 3407671808.0,
- "88": 3407671808.0,
- "89": 3407671808.0,
- "90": 3407671808.0,
- "91": 3407671808.0,
- "92": 3407671808.0,
- "93": 3407671808.0,
- "94": 3407671808.0,
- "95": 3407671808.0,
- "96": 3407671808.0,
- "97": 3407671808.0,
- "98": 3407671808.0,
- "99": 3407671808.0,
- "100": 3407671808.0
+ "1": 3034101248.0,
+ "2": 3329965568.0,
+ "3": 3329965568.0,
+ "4": 3329965568.0,
+ "5": 3329965568.0,
+ "6": 3329965568.0,
+ "7": 3329965568.0,
+ "8": 3329965568.0,
+ "9": 3329965568.0,
+ "10": 3329965568.0,
+ "11": 3329965568.0,
+ "12": 3329965568.0,
+ "13": 3357114368.0,
+ "14": 3362752000.0,
+ "15": 3362752000.0,
+ "16": 3362752000.0,
+ "17": 3362752000.0,
+ "18": 3362752000.0,
+ "19": 3362752000.0,
+ "20": 3362752000.0,
+ "21": 3362752000.0,
+ "22": 3362752000.0,
+ "23": 3362752000.0,
+ "24": 3362752000.0,
+ "25": 3362752000.0,
+ "26": 3375598080.0,
+ "27": 3375598080.0,
+ "28": 3375598080.0,
+ "29": 3375598080.0,
+ "30": 3375598080.0,
+ "31": 3375598080.0,
+ "32": 3375598080.0,
+ "33": 3375598080.0,
+ "34": 3375598080.0,
+ "35": 3375598080.0,
+ "36": 3375598080.0,
+ "37": 3375598080.0,
+ "38": 3375598080.0,
+ "39": 3375598080.0,
+ "40": 3375598080.0,
+ "41": 3375598080.0,
+ "42": 3375598080.0,
+ "43": 3375598080.0,
+ "44": 3375598080.0,
+ "45": 3375598080.0,
+ "46": 3375598080.0,
+ "47": 3375598080.0,
+ "48": 3375598080.0,
+ "49": 3375598080.0,
+ "50": 3375598080.0,
+ "51": 3375598080.0,
+ "52": 3375598080.0,
+ "53": 3375598080.0,
+ "54": 3375598080.0,
+ "55": 3375598080.0,
+ "56": 3375598080.0,
+ "57": 3375598080.0,
+ "58": 3375598080.0,
+ "59": 3375598080.0,
+ "60": 3375598080.0,
+ "61": 3375598080.0,
+ "62": 3375598080.0,
+ "63": 3375598080.0,
+ "64": 3375598080.0,
+ "65": 3375598080.0,
+ "66": 3375598080.0,
+ "67": 3375598080.0,
+ "68": 3375598080.0,
+ "69": 3375598080.0,
+ "70": 3375598080.0,
+ "71": 3375598080.0,
+ "72": 3375598080.0,
+ "73": 3375598080.0,
+ "74": 3375598080.0,
+ "75": 3375598080.0,
+ "76": 3375598080.0,
+ "77": 3375598080.0,
+ "78": 3375598080.0,
+ "79": 3375598080.0,
+ "80": 3375598080.0,
+ "81": 3375598080.0,
+ "82": 3375598080.0,
+ "83": 3375598080.0,
+ "84": 3375598080.0,
+ "85": 3375598080.0,
+ "86": 3375598080.0,
+ "87": 3375598080.0,
+ "88": 3375598080.0,
+ "89": 3375598080.0,
+ "90": 3375598080.0,
+ "91": 3375598080.0,
+ "92": 3375598080.0,
+ "93": 3375598080.0,
+ "94": 3375598080.0,
+ "95": 3375598080.0,
+ "96": 3375598080.0,
+ "97": 3375598080.0,
+ "98": 3375598080.0,
+ "99": 3375598080.0,
+ "100": 3375598080.0
}
},
"iteration-time": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 3.79579,
- "3": 0.25267,
- "4": 0.22623,
- "5": 0.22831,
- "6": 0.57471,
- "7": 0.22109,
- "8": 0.21732,
- "9": 0.2127,
- "10": 0.20981,
- "11": 0.21771,
- "12": 0.23313,
- "13": 0.20775,
- "14": 0.19946,
- "15": 0.21125,
- "16": 0.2099,
- "17": 0.20543,
- "18": 0.19972,
- "19": 0.20265,
- "20": 0.20005,
- "21": 0.20188,
- "22": 0.19675,
- "23": 0.19822,
- "24": 0.19828,
- "25": 0.19827,
- "26": 0.19789,
- "27": 0.20238,
- "28": 0.19366,
- "29": 0.19297,
- "30": 0.19521,
- "31": 0.19886,
- "32": 0.19176,
- "33": 0.19628,
- "34": 0.19156,
- "35": 0.19683,
- "36": 0.19061,
- "37": 0.19031,
- "38": 0.19383,
- "39": 0.1966,
- "40": 0.19152,
- "41": 0.18691,
- "42": 0.1917,
- "43": 0.20258,
- "44": 0.19552,
- "45": 0.20203,
- "46": 0.18769,
- "47": 0.18872,
- "48": 0.18493,
- "49": 0.18884,
- "50": 0.18824,
- "51": 0.20579,
- "52": 0.18818,
- "53": 0.18827,
- "54": 0.19298,
- "55": 0.57299,
- "56": 0.18813,
- "57": 0.18557,
- "58": 0.18597,
- "59": 0.18577,
- "60": 0.18756,
- "61": 0.18972,
- "62": 0.18872,
- "63": 0.18937,
- "64": 0.1888,
- "65": 0.19262,
- "66": 0.1879,
- "67": 0.18498,
- "68": 0.18535,
- "69": 0.19492,
- "70": 0.1923,
- "71": 0.18822,
- "72": 0.19191,
- "73": 0.19457,
- "74": 0.19765,
- "75": 0.19091,
- "76": 0.73064,
- "77": 0.19543,
- "78": 0.19034,
- "79": 0.18715,
- "80": 0.19339,
- "81": 0.19135,
- "82": 0.18703,
- "83": 0.19082,
- "84": 0.18783,
- "85": 0.1926,
- "86": 0.19556,
- "87": 0.19127,
- "88": 0.19028,
- "89": 0.56083,
- "90": 0.19223,
- "91": 0.18622,
- "92": 0.18536,
- "93": 0.19063,
- "94": 0.18804,
- "95": 0.18711,
- "96": 0.1883,
- "97": 0.19006,
- "98": 0.18897,
- "99": 0.60361,
- "100": 0.19278
+ "2": 6.37946,
+ "3": 0.33564,
+ "4": 0.33101,
+ "5": 0.28018,
+ "6": 0.23906,
+ "7": 0.23667,
+ "8": 0.23118,
+ "9": 0.2332,
+ "10": 0.22841,
+ "11": 0.22815,
+ "12": 0.21384,
+ "13": 0.22434,
+ "14": 0.22269,
+ "15": 0.21588,
+ "16": 0.21739,
+ "17": 0.21619,
+ "18": 0.21849,
+ "19": 0.22113,
+ "20": 0.22387,
+ "21": 0.21176,
+ "22": 0.22154,
+ "23": 0.21285,
+ "24": 0.21867,
+ "25": 0.21143,
+ "26": 0.21122,
+ "27": 0.21111,
+ "28": 0.21985,
+ "29": 0.23901,
+ "30": 0.2906,
+ "31": 0.30413,
+ "32": 0.30186,
+ "33": 0.25481,
+ "34": 0.22803,
+ "35": 0.21594,
+ "36": 0.23276,
+ "37": 0.27723,
+ "38": 0.26911,
+ "39": 0.29211,
+ "40": 0.32268,
+ "41": 0.2847,
+ "42": 0.21669,
+ "43": 0.21508,
+ "44": 0.21153,
+ "45": 0.21034,
+ "46": 0.2104,
+ "47": 0.22404,
+ "48": 0.21242,
+ "49": 0.21808,
+ "50": 0.21378,
+ "51": 0.47031,
+ "52": 0.5495,
+ "53": 0.22116,
+ "54": 0.22361,
+ "55": 0.21414,
+ "56": 0.20888,
+ "57": 0.21279,
+ "58": 0.21113,
+ "59": 0.20998,
+ "60": 0.21229,
+ "61": 0.20807,
+ "62": 0.21219,
+ "63": 0.2115,
+ "64": 0.21199,
+ "65": 0.20996,
+ "66": 0.21527,
+ "67": 0.20949,
+ "68": 0.21199,
+ "69": 0.21114,
+ "70": 0.21326,
+ "71": 0.23114,
+ "72": 0.21602,
+ "73": 0.20908,
+ "74": 0.20894,
+ "75": 0.21455,
+ "76": 0.20838,
+ "77": 0.212,
+ "78": 0.21068,
+ "79": 0.20983,
+ "80": 0.207,
+ "81": 0.21523,
+ "82": 0.20995,
+ "83": 0.21319,
+ "84": 0.21048,
+ "85": 0.21115,
+ "86": 0.21141,
+ "87": 0.21395,
+ "88": 0.20931,
+ "89": 0.21408,
+ "90": 0.21324,
+ "91": 0.2195,
+ "92": 0.21065,
+ "93": 0.25469,
+ "94": 0.23409,
+ "95": 0.20775,
+ "96": 0.20726,
+ "97": 0.21584,
+ "98": 0.20765,
+ "99": 0.2071,
+ "100": 0.21821
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon/golden_values_dev_dgx_h100.json
index ccbece04f60..6ae50c1e42f 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.81103,
- "2": 10.83065,
- "3": 10.82048,
- "4": 10.81293,
- "5": 10.84375,
- "6": 10.8473,
- "7": 10.85341,
- "8": 10.83652,
- "9": 10.84691,
- "10": 10.78166,
- "11": 10.85213,
- "12": 10.8629,
- "13": 10.85433,
- "14": 10.88455,
- "15": 10.87782,
- "16": 10.84637,
- "17": 10.83054,
- "18": 10.86645,
- "19": 10.84951,
- "20": 10.84547,
- "21": 10.8476,
- "22": 10.79618,
- "23": 10.88285,
- "24": 10.83247,
- "25": 10.8246,
- "26": 10.8432,
- "27": 10.85345,
- "28": 10.87635,
- "29": 10.864,
- "30": 10.81293,
- "31": 10.78651,
- "32": 10.85541,
- "33": 10.85587,
- "34": 10.8491,
- "35": 10.83747,
- "36": 10.80362,
- "37": 10.83812,
- "38": 10.80509,
- "39": 10.84183,
- "40": 10.80312,
- "41": 10.84012,
- "42": 10.84384,
- "43": 10.80987,
- "44": 10.80275,
- "45": 10.78691,
- "46": 10.80833,
- "47": 10.81704,
- "48": 10.80337,
- "49": 10.78131,
- "50": 10.80305,
- "51": 10.82235,
- "52": 10.80371,
- "53": 10.83231,
- "54": 10.8151,
- "55": 10.82578,
- "56": 10.77729,
- "57": 10.75325,
- "58": 10.80742,
- "59": 10.79087,
- "60": 10.73998,
- "61": 10.79954,
- "62": 10.81284,
- "63": 10.72011,
- "64": 10.78598,
- "65": 10.68981,
- "66": 10.76066,
- "67": 10.73402,
- "68": 10.8022,
- "69": 10.78312,
- "70": 10.77711,
- "71": 10.76626,
- "72": 10.73591,
- "73": 10.72919,
- "74": 10.62192,
- "75": 10.69079,
- "76": 10.65398,
- "77": 10.82162,
- "78": 10.76368,
- "79": 10.70473,
- "80": 10.69368,
- "81": 10.72419,
- "82": 10.74233,
- "83": 10.66786,
- "84": 10.6983,
- "85": 10.714,
- "86": 10.6383,
- "87": 10.71809,
- "88": 10.73508,
- "89": 10.7139,
- "90": 10.74649,
- "91": 10.64861,
- "92": 10.64636,
- "93": 10.60234,
- "94": 10.53327,
- "95": 10.66155,
- "96": 10.67215,
- "97": 10.61446,
- "98": 10.68506,
- "99": 10.52056,
- "100": 10.61544
+ "1": 10.90136,
+ "2": 10.89626,
+ "3": 10.90659,
+ "4": 10.90575,
+ "5": 10.91358,
+ "6": 10.89696,
+ "7": 10.89815,
+ "8": 10.91366,
+ "9": 10.89588,
+ "10": 10.90398,
+ "11": 10.90012,
+ "12": 10.91846,
+ "13": 10.88423,
+ "14": 10.87541,
+ "15": 10.90989,
+ "16": 10.90277,
+ "17": 10.88608,
+ "18": 10.89377,
+ "19": 10.895,
+ "20": 10.89066,
+ "21": 10.8868,
+ "22": 10.90298,
+ "23": 10.91222,
+ "24": 10.88607,
+ "25": 10.90133,
+ "26": 10.88889,
+ "27": 10.89705,
+ "28": 10.88164,
+ "29": 10.88963,
+ "30": 10.91068,
+ "31": 10.89098,
+ "32": 10.88816,
+ "33": 10.89732,
+ "34": 10.87367,
+ "35": 10.89622,
+ "36": 10.90763,
+ "37": 10.8707,
+ "38": 10.87815,
+ "39": 10.88547,
+ "40": 10.89256,
+ "41": 10.88379,
+ "42": 10.89516,
+ "43": 10.87939,
+ "44": 10.88513,
+ "45": 10.88363,
+ "46": 10.88652,
+ "47": 10.88585,
+ "48": 10.86369,
+ "49": 10.87651,
+ "50": 10.8837,
+ "51": 10.89355,
+ "52": 10.87148,
+ "53": 10.85907,
+ "54": 10.87109,
+ "55": 10.86834,
+ "56": 10.87336,
+ "57": 10.84517,
+ "58": 10.856,
+ "59": 10.84639,
+ "60": 10.84315,
+ "61": 10.86095,
+ "62": 10.85706,
+ "63": 10.86067,
+ "64": 10.83924,
+ "65": 10.82737,
+ "66": 10.84584,
+ "67": 10.82868,
+ "68": 10.82937,
+ "69": 10.81915,
+ "70": 10.82806,
+ "71": 10.82662,
+ "72": 10.80781,
+ "73": 10.80771,
+ "74": 10.80585,
+ "75": 10.81493,
+ "76": 10.8112,
+ "77": 10.8075,
+ "78": 10.79154,
+ "79": 10.80235,
+ "80": 10.78921,
+ "81": 10.79518,
+ "82": 10.79608,
+ "83": 10.78587,
+ "84": 10.75736,
+ "85": 10.76238,
+ "86": 10.77717,
+ "87": 10.79631,
+ "88": 10.77499,
+ "89": 10.77556,
+ "90": 10.76448,
+ "91": 10.74031,
+ "92": 10.76017,
+ "93": 10.74713,
+ "94": 10.73416,
+ "95": 10.75291,
+ "96": 10.72342,
+ "97": 10.71521,
+ "98": 10.72656,
+ "99": 10.7464,
+ "100": 10.69534
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1248.0,
- "2": 1334.0,
- "3": 1297.0,
- "4": 1215.0,
- "5": 1398.0,
- "6": 1528.0,
- "7": 1225.0,
- "8": 1301.0,
- "9": 1348.0,
- "10": 1359.0,
- "11": 1296.0,
- "12": 1248.0,
- "13": 1286.0,
- "14": 1373.0,
- "15": 1195.0,
- "16": 1177.0,
- "17": 1266.0,
- "18": 1393.0,
- "19": 1219.0,
- "20": 1257.0,
- "21": 1244.0,
- "22": 1155.0,
- "23": 1385.0,
- "24": 1323.0,
- "25": 1226.0,
- "26": 1184.0,
- "27": 1394.0,
- "28": 1476.0,
- "29": 1300.0,
- "30": 1245.0,
- "31": 1138.0,
- "32": 1283.0,
- "33": 1247.0,
- "34": 1186.0,
- "35": 1158.0,
- "36": 1178.0,
- "37": 1232.0,
- "38": 1357.0,
- "39": 1541.0,
- "40": 1170.0,
- "41": 1369.0,
- "42": 1153.0,
- "43": 1180.0,
- "44": 1239.0,
- "45": 1189.0,
- "46": 1141.0,
- "47": 1203.0,
- "48": 1126.0,
- "49": 1194.0,
- "50": 1214.0,
- "51": 1274.0,
- "52": 1209.0,
- "53": 1360.0,
- "54": 1257.0,
- "55": 1170.0,
- "56": 1282.0,
- "57": 1296.0,
- "58": 1271.0,
- "59": 1180.0,
- "60": 1182.0,
- "61": 1202.0,
- "62": 1192.0,
- "63": 1253.0,
- "64": 1248.0,
- "65": 1180.0,
- "66": 1179.0,
- "67": 1188.0,
- "68": 1229.0,
- "69": 1232.0,
- "70": 1280.0,
- "71": 1246.0,
- "72": 1261.0,
- "73": 1148.0,
- "74": 1114.0,
- "75": 1281.0,
- "76": 1376.0,
- "77": 1373.0,
- "78": 1285.0,
- "79": 1087.0,
- "80": 1127.0,
- "81": 1135.0,
- "82": 1169.0,
- "83": 1300.0,
- "84": 1206.0,
- "85": 1269.0,
- "86": 1187.0,
- "87": 1236.0,
- "88": 1262.0,
- "89": 1197.0,
- "90": 1425.0,
- "91": 1197.0,
- "92": 1244.0,
- "93": 1142.0,
- "94": 971.0,
- "95": 1281.0,
- "96": 1243.0,
- "97": 1145.0,
- "98": 1288.0,
- "99": 1286.0,
- "100": 1212.0
+ "1": 1166.0,
+ "2": 1181.0,
+ "3": 1293.0,
+ "4": 1290.0,
+ "5": 1226.0,
+ "6": 1251.0,
+ "7": 1333.0,
+ "8": 1188.0,
+ "9": 1250.0,
+ "10": 1182.0,
+ "11": 1279.0,
+ "12": 1267.0,
+ "13": 1351.0,
+ "14": 1306.0,
+ "15": 1217.0,
+ "16": 1185.0,
+ "17": 1173.0,
+ "18": 1119.0,
+ "19": 1224.0,
+ "20": 1159.0,
+ "21": 1199.0,
+ "22": 1202.0,
+ "23": 1268.0,
+ "24": 1119.0,
+ "25": 1275.0,
+ "26": 1222.0,
+ "27": 1369.0,
+ "28": 1209.0,
+ "29": 1191.0,
+ "30": 1267.0,
+ "31": 1204.0,
+ "32": 1313.0,
+ "33": 1215.0,
+ "34": 1188.0,
+ "35": 1369.0,
+ "36": 1246.0,
+ "37": 1183.0,
+ "38": 1226.0,
+ "39": 1369.0,
+ "40": 1303.0,
+ "41": 1373.0,
+ "42": 1110.0,
+ "43": 1137.0,
+ "44": 1312.0,
+ "45": 1204.0,
+ "46": 1459.0,
+ "47": 1224.0,
+ "48": 1169.0,
+ "49": 1320.0,
+ "50": 1297.0,
+ "51": 1284.0,
+ "52": 1189.0,
+ "53": 1306.0,
+ "54": 1192.0,
+ "55": 1194.0,
+ "56": 1169.0,
+ "57": 1158.0,
+ "58": 1263.0,
+ "59": 1145.0,
+ "60": 1251.0,
+ "61": 1252.0,
+ "62": 1168.0,
+ "63": 1239.0,
+ "64": 1327.0,
+ "65": 1156.0,
+ "66": 1224.0,
+ "67": 1182.0,
+ "68": 1297.0,
+ "69": 1171.0,
+ "70": 1150.0,
+ "71": 1141.0,
+ "72": 1259.0,
+ "73": 1276.0,
+ "74": 1289.0,
+ "75": 1374.0,
+ "76": 1315.0,
+ "77": 1178.0,
+ "78": 1247.0,
+ "79": 1303.0,
+ "80": 1258.0,
+ "81": 1309.0,
+ "82": 1151.0,
+ "83": 1191.0,
+ "84": 1203.0,
+ "85": 1191.0,
+ "86": 1300.0,
+ "87": 1092.0,
+ "88": 1131.0,
+ "89": 1116.0,
+ "90": 1348.0,
+ "91": 1290.0,
+ "92": 1241.0,
+ "93": 1311.0,
+ "94": 1377.0,
+ "95": 1050.0,
+ "96": 1160.0,
+ "97": 1208.0,
+ "98": 1322.0,
+ "99": 1121.0,
+ "100": 1227.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1095885312.0,
- "2": 1095855104.0,
- "3": 1095902720.0,
- "4": 1095882752.0,
- "5": 1095905792.0,
- "6": 1095846912.0,
- "7": 1095869952.0,
- "8": 1095877120.0,
- "9": 1095892480.0,
- "10": 1095903232.0,
- "11": 1095879168.0,
- "12": 1095851008.0,
- "13": 1095903232.0,
- "14": 1095813120.0,
- "15": 1095857152.0,
- "16": 1095791104.0,
- "17": 1095911936.0,
- "18": 1095883264.0,
- "19": 1095893504.0,
- "20": 1095812096.0,
- "21": 1095832064.0,
- "22": 1095908864.0,
- "23": 1095883776.0,
- "24": 1095824384.0,
- "25": 1095956480.0,
- "26": 1095863808.0,
- "27": 1095919104.0,
- "28": 1095844864.0,
- "29": 1095879168.0,
- "30": 1095843840.0,
- "31": 1095908352.0,
- "32": 1095840768.0,
- "33": 1095850496.0,
- "34": 1095818240.0,
- "35": 1095864832.0,
- "36": 1095879680.0,
- "37": 1095839232.0,
- "38": 1095923200.0,
- "39": 1095930880.0,
- "40": 1095819264.0,
- "41": 1095848448.0,
- "42": 1095866880.0,
- "43": 1095878656.0,
- "44": 1095980544.0,
- "45": 1095855104.0,
- "46": 1095869952.0,
- "47": 1095895040.0,
- "48": 1095877632.0,
- "49": 1095844352.0,
- "50": 1095864320.0,
- "51": 1095936000.0,
- "52": 1095879680.0,
- "53": 1095939584.0,
- "54": 1095890432.0,
- "55": 1095879168.0,
- "56": 1095869440.0,
- "57": 1095916544.0,
- "58": 1095913984.0,
- "59": 1095899136.0,
- "60": 1095863296.0,
- "61": 1095864320.0,
- "62": 1095858176.0,
- "63": 1095874048.0,
- "64": 1095861760.0,
- "65": 1095869952.0,
- "66": 1095875584.0,
- "67": 1095864832.0,
- "68": 1095874048.0,
- "69": 1095860224.0,
- "70": 1095905280.0,
- "71": 1095880192.0,
- "72": 1095805440.0,
- "73": 1095907840.0,
- "74": 1095919616.0,
- "75": 1095884800.0,
- "76": 1095905792.0,
- "77": 1095855616.0,
- "78": 1095916544.0,
- "79": 1095888384.0,
- "80": 1095842304.0,
- "81": 1095875584.0,
- "82": 1095823872.0,
- "83": 1095923712.0,
- "84": 1095906304.0,
- "85": 1095871488.0,
- "86": 1095872512.0,
- "87": 1095895552.0,
- "88": 1095880192.0,
- "89": 1095869440.0,
- "90": 1095863296.0,
- "91": 1095917056.0,
- "92": 1095900160.0,
- "93": 1095879680.0,
- "94": 1095888896.0,
- "95": 1095886848.0,
- "96": 1095888384.0,
- "97": 1095897088.0,
- "98": 1095875584.0,
- "99": 1095889408.0,
- "100": 1095928320.0
+ "1": 994062848.0,
+ "2": 994053120.0,
+ "3": 994029056.0,
+ "4": 994008576.0,
+ "5": 994060288.0,
+ "6": 994045952.0,
+ "7": 993969152.0,
+ "8": 994004992.0,
+ "9": 994009088.0,
+ "10": 994040320.0,
+ "11": 994016768.0,
+ "12": 994023936.0,
+ "13": 994028032.0,
+ "14": 994029568.0,
+ "15": 994006528.0,
+ "16": 993992192.0,
+ "17": 994001920.0,
+ "18": 994003456.0,
+ "19": 994033152.0,
+ "20": 993997312.0,
+ "21": 994052096.0,
+ "22": 994078208.0,
+ "23": 994048512.0,
+ "24": 993964544.0,
+ "25": 994045952.0,
+ "26": 994076160.0,
+ "27": 994029056.0,
+ "28": 993981952.0,
+ "29": 994058240.0,
+ "30": 993992192.0,
+ "31": 993995776.0,
+ "32": 994004992.0,
+ "33": 994008064.0,
+ "34": 993987584.0,
+ "35": 993974272.0,
+ "36": 994039296.0,
+ "37": 993990144.0,
+ "38": 994047488.0,
+ "39": 994053632.0,
+ "40": 993999872.0,
+ "41": 994022400.0,
+ "42": 993965056.0,
+ "43": 994031616.0,
+ "44": 994015744.0,
+ "45": 993987584.0,
+ "46": 994053120.0,
+ "47": 994010112.0,
+ "48": 994039296.0,
+ "49": 994000384.0,
+ "50": 994046464.0,
+ "51": 994030592.0,
+ "52": 994004992.0,
+ "53": 993954816.0,
+ "54": 994049536.0,
+ "55": 994040320.0,
+ "56": 994037760.0,
+ "57": 994101248.0,
+ "58": 994033664.0,
+ "59": 994041344.0,
+ "60": 994021376.0,
+ "61": 993989632.0,
+ "62": 993978368.0,
+ "63": 993986560.0,
+ "64": 994018304.0,
+ "65": 994022912.0,
+ "66": 994036736.0,
+ "67": 994021376.0,
+ "68": 994019328.0,
+ "69": 993983488.0,
+ "70": 994068992.0,
+ "71": 994046976.0,
+ "72": 994043392.0,
+ "73": 993975296.0,
+ "74": 994014208.0,
+ "75": 994061312.0,
+ "76": 994020352.0,
+ "77": 994059776.0,
+ "78": 994006528.0,
+ "79": 994072064.0,
+ "80": 993994752.0,
+ "81": 994048512.0,
+ "82": 993993728.0,
+ "83": 994055168.0,
+ "84": 994027008.0,
+ "85": 994032128.0,
+ "86": 994016768.0,
+ "87": 994029056.0,
+ "88": 994012160.0,
+ "89": 994003968.0,
+ "90": 994024448.0,
+ "91": 994011136.0,
+ "92": 994009600.0,
+ "93": 994040832.0,
+ "94": 994014720.0,
+ "95": 994036224.0,
+ "96": 994027520.0,
+ "97": 993995264.0,
+ "98": 994034176.0,
+ "99": 994050560.0,
+ "100": 994081792.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 3260420096.0,
- "2": 3582874112.0,
- "3": 3616017408.0,
- "4": 3616017408.0,
- "5": 3616065536.0,
- "6": 3616065536.0,
- "7": 3616065536.0,
- "8": 3616065536.0,
- "9": 3616065536.0,
- "10": 3619626496.0,
- "11": 3619626496.0,
- "12": 3619626496.0,
- "13": 3619626496.0,
- "14": 3619626496.0,
- "15": 3619626496.0,
- "16": 3619626496.0,
- "17": 3619626496.0,
- "18": 3619626496.0,
- "19": 3619626496.0,
- "20": 3619626496.0,
- "21": 3619626496.0,
- "22": 3619626496.0,
- "23": 3619626496.0,
- "24": 3619626496.0,
- "25": 3648242176.0,
- "26": 3648242176.0,
- "27": 3648242176.0,
- "28": 3648242176.0,
- "29": 3648242176.0,
- "30": 3648242176.0,
- "31": 3648242176.0,
- "32": 3648242176.0,
- "33": 3648242176.0,
- "34": 3648242176.0,
- "35": 3648242176.0,
- "36": 3648242176.0,
- "37": 3648242176.0,
- "38": 3648242176.0,
- "39": 3648242176.0,
- "40": 3648242176.0,
- "41": 3648242176.0,
- "42": 3648242176.0,
- "43": 3648242176.0,
- "44": 3665209344.0,
- "45": 3665209344.0,
- "46": 3665209344.0,
- "47": 3665209344.0,
- "48": 3665209344.0,
- "49": 3665209344.0,
- "50": 3665209344.0,
- "51": 3665209344.0,
- "52": 3665209344.0,
- "53": 3665209344.0,
- "54": 3665209344.0,
- "55": 3665209344.0,
- "56": 3665209344.0,
- "57": 3665209344.0,
- "58": 3665209344.0,
- "59": 3665209344.0,
- "60": 3665209344.0,
- "61": 3665209344.0,
- "62": 3665209344.0,
- "63": 3665209344.0,
- "64": 3665209344.0,
- "65": 3665209344.0,
- "66": 3665209344.0,
- "67": 3665209344.0,
- "68": 3665209344.0,
- "69": 3665209344.0,
- "70": 3665209344.0,
- "71": 3665209344.0,
- "72": 3665209344.0,
- "73": 3665209344.0,
- "74": 3665209344.0,
- "75": 3665209344.0,
- "76": 3665209344.0,
- "77": 3665209344.0,
- "78": 3665209344.0,
- "79": 3665209344.0,
- "80": 3665209344.0,
- "81": 3665209344.0,
- "82": 3665209344.0,
- "83": 3665209344.0,
- "84": 3665209344.0,
- "85": 3665209344.0,
- "86": 3665209344.0,
- "87": 3665209344.0,
- "88": 3665209344.0,
- "89": 3665209344.0,
- "90": 3665209344.0,
- "91": 3665209344.0,
- "92": 3665209344.0,
- "93": 3665209344.0,
- "94": 3665209344.0,
- "95": 3665209344.0,
- "96": 3665209344.0,
- "97": 3665209344.0,
- "98": 3665209344.0,
- "99": 3665209344.0,
- "100": 3665209344.0
+ "1": 3207447552.0,
+ "2": 3491065344.0,
+ "3": 3491065344.0,
+ "4": 3491065344.0,
+ "5": 3497147904.0,
+ "6": 3497147904.0,
+ "7": 3497147904.0,
+ "8": 3497147904.0,
+ "9": 3497147904.0,
+ "10": 3497147904.0,
+ "11": 3497147904.0,
+ "12": 3497147904.0,
+ "13": 3497147904.0,
+ "14": 3497147904.0,
+ "15": 3497147904.0,
+ "16": 3497147904.0,
+ "17": 3497147904.0,
+ "18": 3497147904.0,
+ "19": 3497147904.0,
+ "20": 3497147904.0,
+ "21": 3497147904.0,
+ "22": 3506189312.0,
+ "23": 3506189312.0,
+ "24": 3506189312.0,
+ "25": 3506189312.0,
+ "26": 3511780352.0,
+ "27": 3511780352.0,
+ "28": 3511780352.0,
+ "29": 3511780352.0,
+ "30": 3511780352.0,
+ "31": 3511780352.0,
+ "32": 3511780352.0,
+ "33": 3511780352.0,
+ "34": 3511780352.0,
+ "35": 3511780352.0,
+ "36": 3511780352.0,
+ "37": 3511780352.0,
+ "38": 3511780352.0,
+ "39": 3511780352.0,
+ "40": 3511780352.0,
+ "41": 3511780352.0,
+ "42": 3511780352.0,
+ "43": 3511780352.0,
+ "44": 3511780352.0,
+ "45": 3511780352.0,
+ "46": 3511780352.0,
+ "47": 3511780352.0,
+ "48": 3511780352.0,
+ "49": 3511780352.0,
+ "50": 3511780352.0,
+ "51": 3511780352.0,
+ "52": 3511780352.0,
+ "53": 3511780352.0,
+ "54": 3511780352.0,
+ "55": 3511780352.0,
+ "56": 3511780352.0,
+ "57": 3527660032.0,
+ "58": 3527660032.0,
+ "59": 3527660032.0,
+ "60": 3527660032.0,
+ "61": 3527660032.0,
+ "62": 3527660032.0,
+ "63": 3527660032.0,
+ "64": 3527660032.0,
+ "65": 3527660032.0,
+ "66": 3527660032.0,
+ "67": 3527660032.0,
+ "68": 3527660032.0,
+ "69": 3527660032.0,
+ "70": 3527660032.0,
+ "71": 3527660032.0,
+ "72": 3527660032.0,
+ "73": 3527660032.0,
+ "74": 3527660032.0,
+ "75": 3527660032.0,
+ "76": 3527660032.0,
+ "77": 3527660032.0,
+ "78": 3527660032.0,
+ "79": 3527660032.0,
+ "80": 3527660032.0,
+ "81": 3527660032.0,
+ "82": 3527660032.0,
+ "83": 3527660032.0,
+ "84": 3527660032.0,
+ "85": 3527660032.0,
+ "86": 3527660032.0,
+ "87": 3527660032.0,
+ "88": 3527660032.0,
+ "89": 3527660032.0,
+ "90": 3527660032.0,
+ "91": 3527660032.0,
+ "92": 3527660032.0,
+ "93": 3527660032.0,
+ "94": 3527660032.0,
+ "95": 3527660032.0,
+ "96": 3527660032.0,
+ "97": 3527660032.0,
+ "98": 3527660032.0,
+ "99": 3527660032.0,
+ "100": 3527660032.0
}
},
"iteration-time": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 6.96692,
- "3": 0.41239,
- "4": 0.39161,
- "5": 0.40475,
- "6": 0.3904,
- "7": 0.39424,
- "8": 0.38721,
- "9": 0.37766,
- "10": 0.38826,
- "11": 0.39241,
- "12": 0.37744,
- "13": 0.37937,
- "14": 0.39891,
- "15": 0.39154,
- "16": 0.38546,
- "17": 0.36906,
- "18": 0.37961,
- "19": 0.37168,
- "20": 0.37856,
- "21": 0.37322,
- "22": 0.36901,
- "23": 0.36962,
- "24": 0.37071,
- "25": 0.36454,
- "26": 0.37164,
- "27": 0.35661,
- "28": 0.36072,
- "29": 0.37992,
- "30": 0.35418,
- "31": 0.35828,
- "32": 0.35863,
- "33": 0.36304,
- "34": 0.34938,
- "35": 0.36044,
- "36": 0.3661,
- "37": 0.36694,
- "38": 0.37046,
- "39": 0.37481,
- "40": 0.37606,
- "41": 0.35942,
- "42": 0.35928,
- "43": 0.82934,
- "44": 0.36187,
- "45": 0.36124,
- "46": 0.35574,
- "47": 0.36316,
- "48": 0.36376,
- "49": 0.35682,
- "50": 0.36509,
- "51": 0.36781,
- "52": 0.36533,
- "53": 0.85049,
- "54": 0.36057,
- "55": 0.3565,
- "56": 0.3743,
- "57": 0.36606,
- "58": 0.36355,
- "59": 0.36215,
- "60": 0.36264,
- "61": 0.36287,
- "62": 0.35671,
- "63": 0.3661,
- "64": 0.35095,
- "65": 0.38153,
- "66": 0.35893,
- "67": 0.37021,
- "68": 0.35656,
- "69": 0.35749,
- "70": 0.3687,
- "71": 0.35581,
- "72": 0.36693,
- "73": 0.35596,
- "74": 0.361,
- "75": 0.35439,
- "76": 0.35584,
- "77": 0.36297,
- "78": 0.35272,
- "79": 0.35409,
- "80": 0.35974,
- "81": 0.355,
- "82": 0.35692,
- "83": 0.3617,
- "84": 0.36038,
- "85": 0.36694,
- "86": 0.36667,
- "87": 0.36782,
- "88": 0.37457,
- "89": 0.36585,
- "90": 0.37116,
- "91": 0.36385,
- "92": 0.3564,
- "93": 0.36251,
- "94": 0.35477,
- "95": 0.35372,
- "96": 0.8695,
- "97": 0.35034,
- "98": 0.36289,
- "99": 0.35766,
- "100": 0.35116
+ "2": 5.50635,
+ "3": 0.2021,
+ "4": 0.16997,
+ "5": 0.1629,
+ "6": 0.15774,
+ "7": 0.15741,
+ "8": 0.16474,
+ "9": 0.13807,
+ "10": 0.14772,
+ "11": 0.15431,
+ "12": 0.14774,
+ "13": 0.14763,
+ "14": 0.15375,
+ "15": 0.14355,
+ "16": 0.14032,
+ "17": 0.13817,
+ "18": 0.14547,
+ "19": 0.14496,
+ "20": 0.14607,
+ "21": 0.13912,
+ "22": 0.14906,
+ "23": 0.1402,
+ "24": 0.13345,
+ "25": 0.14537,
+ "26": 0.14139,
+ "27": 0.13628,
+ "28": 0.12793,
+ "29": 0.14133,
+ "30": 0.14656,
+ "31": 0.12744,
+ "32": 0.13644,
+ "33": 0.1438,
+ "34": 0.14192,
+ "35": 0.13614,
+ "36": 0.12991,
+ "37": 0.13122,
+ "38": 0.13328,
+ "39": 0.13843,
+ "40": 0.13356,
+ "41": 0.1296,
+ "42": 0.13535,
+ "43": 0.12949,
+ "44": 0.13142,
+ "45": 0.12749,
+ "46": 0.13226,
+ "47": 0.13084,
+ "48": 0.13336,
+ "49": 0.12991,
+ "50": 0.12746,
+ "51": 0.20344,
+ "52": 0.13457,
+ "53": 0.14071,
+ "54": 0.12795,
+ "55": 0.12888,
+ "56": 0.13546,
+ "57": 0.13918,
+ "58": 0.13432,
+ "59": 0.13163,
+ "60": 0.13008,
+ "61": 0.14253,
+ "62": 0.13758,
+ "63": 0.1281,
+ "64": 0.1261,
+ "65": 0.13205,
+ "66": 0.12743,
+ "67": 0.1301,
+ "68": 0.12784,
+ "69": 0.12727,
+ "70": 0.12909,
+ "71": 0.13087,
+ "72": 0.1311,
+ "73": 0.13644,
+ "74": 0.13151,
+ "75": 0.12705,
+ "76": 0.12615,
+ "77": 0.13091,
+ "78": 0.13241,
+ "79": 0.13204,
+ "80": 0.1405,
+ "81": 0.13487,
+ "82": 0.13746,
+ "83": 0.13148,
+ "84": 0.12824,
+ "85": 0.12856,
+ "86": 0.14437,
+ "87": 0.133,
+ "88": 0.13156,
+ "89": 0.13179,
+ "90": 0.12764,
+ "91": 0.14488,
+ "92": 0.12701,
+ "93": 0.13708,
+ "94": 0.13572,
+ "95": 0.13162,
+ "96": 0.13367,
+ "97": 0.13318,
+ "98": 0.13471,
+ "99": 0.13593,
+ "100": 0.13296
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon/model_config.yaml
index 1d0ef19232e..3236582225d 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -63,4 +63,6 @@ MODEL_ARGS:
--muon-momentum: 0.9
--muon-extra-scale-factor: 0.2
--muon-scale-mode: spectral
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_h100.json
index b106daa13a1..a84fc91ee4f 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.81103,
- "2": 10.83065,
- "3": 10.82107,
- "4": 10.81304,
- "5": 10.84321,
- "6": 10.84718,
- "7": 10.85237,
- "8": 10.83499,
- "9": 10.84293,
- "10": 10.77678,
- "11": 10.84585,
- "12": 10.85174,
- "13": 10.84182,
- "14": 10.87189,
- "15": 10.83593,
- "16": 10.79751,
- "17": 10.77325,
- "18": 10.8073,
- "19": 10.78778,
- "20": 10.73435,
- "21": 10.69516,
- "22": 10.56641,
- "23": 10.71634,
- "24": 10.61287,
- "25": 10.55191,
- "26": 10.61354,
- "27": 10.62651,
- "28": 10.59524,
- "29": 10.5917,
- "30": 10.39149,
- "31": 10.1464,
- "32": 10.47402,
- "33": 10.47024,
- "34": 10.23415,
- "35": 10.2932,
- "36": 10.26667,
- "37": 10.37209,
- "38": 10.22542,
- "39": 10.42143,
- "40": 10.13017,
- "41": 10.16266,
- "42": 10.24275,
- "43": 9.88221,
- "44": 9.99119,
- "45": 9.87323,
- "46": 9.85181,
- "47": 10.15626,
- "48": 9.89157,
- "49": 9.58903,
- "50": 9.95443,
- "51": 9.88487,
- "52": 9.78018,
- "53": 10.10226,
- "54": 9.9873,
- "55": 9.9027,
- "56": 9.66818,
- "57": 9.53521,
- "58": 9.89495,
- "59": 9.6289,
- "60": 9.54307,
- "61": 9.72725,
- "62": 10.03319,
- "63": 9.45201,
- "64": 9.83185,
- "65": 8.99108,
- "66": 9.76421,
- "67": 9.40334,
- "68": 9.83107,
- "69": 9.81874,
- "70": 9.77252,
- "71": 9.65812,
- "72": 9.64065,
- "73": 9.5512,
- "74": 9.02044,
- "75": 9.47713,
- "76": 9.13591,
- "77": 10.09778,
- "78": 9.75282,
- "79": 9.41686,
- "80": 9.45072,
- "81": 9.52034,
- "82": 9.73197,
- "83": 9.36926,
- "84": 9.4504,
- "85": 9.65212,
- "86": 9.11237,
- "87": 9.61129,
- "88": 9.78679,
- "89": 9.64613,
- "90": 9.83484,
- "91": 9.39422,
- "92": 9.39187,
- "93": 9.12787,
- "94": 8.86646,
- "95": 9.54348,
- "96": 9.55708,
- "97": 9.33174,
- "98": 9.6919,
- "99": 8.92043,
- "100": 9.41916
+ "1": 10.90136,
+ "2": 10.89626,
+ "3": 10.90649,
+ "4": 10.90498,
+ "5": 10.91356,
+ "6": 10.89668,
+ "7": 10.89772,
+ "8": 10.91244,
+ "9": 10.89327,
+ "10": 10.90047,
+ "11": 10.89181,
+ "12": 10.90692,
+ "13": 10.87186,
+ "14": 10.86088,
+ "15": 10.87299,
+ "16": 10.85958,
+ "17": 10.84011,
+ "18": 10.83687,
+ "19": 10.84053,
+ "20": 10.7921,
+ "21": 10.7566,
+ "22": 10.75636,
+ "23": 10.75828,
+ "24": 10.71967,
+ "25": 10.72082,
+ "26": 10.69803,
+ "27": 10.6827,
+ "28": 10.61126,
+ "29": 10.59096,
+ "30": 10.57405,
+ "31": 10.56098,
+ "32": 10.54258,
+ "33": 10.51701,
+ "34": 10.47696,
+ "35": 10.48552,
+ "36": 10.47878,
+ "37": 10.42862,
+ "38": 10.43487,
+ "39": 10.40617,
+ "40": 10.39422,
+ "41": 10.37111,
+ "42": 10.35556,
+ "43": 10.32443,
+ "44": 10.30748,
+ "45": 10.31572,
+ "46": 10.27759,
+ "47": 10.2683,
+ "48": 10.22396,
+ "49": 10.21591,
+ "50": 10.21919,
+ "51": 10.22179,
+ "52": 10.17282,
+ "53": 10.17705,
+ "54": 10.14474,
+ "55": 10.11375,
+ "56": 10.142,
+ "57": 10.1263,
+ "58": 10.13368,
+ "59": 10.07831,
+ "60": 10.10013,
+ "61": 10.05698,
+ "62": 10.02107,
+ "63": 10.09336,
+ "64": 10.03537,
+ "65": 10.00587,
+ "66": 10.04647,
+ "67": 10.01885,
+ "68": 9.97907,
+ "69": 9.99388,
+ "70": 9.98095,
+ "71": 10.00404,
+ "72": 9.98667,
+ "73": 9.96191,
+ "74": 9.95994,
+ "75": 9.91417,
+ "76": 9.95275,
+ "77": 9.95108,
+ "78": 9.89989,
+ "79": 9.90164,
+ "80": 9.91091,
+ "81": 9.94223,
+ "82": 9.8978,
+ "83": 9.85159,
+ "84": 9.79602,
+ "85": 9.77416,
+ "86": 9.89303,
+ "87": 9.90525,
+ "88": 9.88308,
+ "89": 9.82604,
+ "90": 9.82541,
+ "91": 9.82851,
+ "92": 9.82375,
+ "93": 9.75205,
+ "94": 9.83154,
+ "95": 9.821,
+ "96": 9.80789,
+ "97": 9.75345,
+ "98": 9.78425,
+ "99": 9.83393,
+ "100": 9.72319
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1248.0,
- "2": 1334.0,
- "3": 1294.0,
- "4": 1227.0,
- "5": 1403.0,
- "6": 1427.0,
- "7": 1252.0,
- "8": 1427.0,
- "9": 1305.0,
- "10": 1282.0,
- "11": 1297.0,
- "12": 1278.0,
- "13": 1202.0,
- "14": 1425.0,
- "15": 1290.0,
- "16": 1353.0,
- "17": 1248.0,
- "18": 1308.0,
- "19": 1305.0,
- "20": 1244.0,
- "21": 1166.0,
- "22": 1145.0,
- "23": 1320.0,
- "24": 1102.0,
- "25": 1254.0,
- "26": 1241.0,
- "27": 1137.0,
- "28": 1332.0,
- "29": 1297.0,
- "30": 1138.0,
- "31": 1027.0,
- "32": 1093.0,
- "33": 1262.0,
- "34": 1095.0,
- "35": 1120.0,
- "36": 1048.0,
- "37": 1161.0,
- "38": 1211.0,
- "39": 1225.0,
- "40": 1379.0,
- "41": 1115.0,
- "42": 1175.0,
- "43": 1049.0,
- "44": 1164.0,
- "45": 1127.0,
- "46": 1334.0,
- "47": 1233.0,
- "48": 1192.0,
- "49": 1310.0,
- "50": 1125.0,
- "51": 1311.0,
- "52": 1269.0,
- "53": 1392.0,
- "54": 1266.0,
- "55": 1197.0,
- "56": 1294.0,
- "57": 1125.0,
- "58": 1380.0,
- "59": 1335.0,
- "60": 1070.0,
- "61": 1317.0,
- "62": 1323.0,
- "63": 1177.0,
- "64": 1464.0,
- "65": 1297.0,
- "66": 1459.0,
- "67": 1319.0,
- "68": 1281.0,
- "69": 1361.0,
- "70": 1439.0,
- "71": 1408.0,
- "72": 1131.0,
- "73": 1261.0,
- "74": 918.0,
- "75": 1051.0,
- "76": 1288.0,
- "77": 1472.0,
- "78": 1433.0,
- "79": 1433.0,
- "80": 1350.0,
- "81": 1576.0,
- "82": 1414.0,
- "83": 1205.0,
- "84": 1485.0,
- "85": 1339.0,
- "86": 1265.0,
- "87": 1538.0,
- "88": 1462.0,
- "89": 1499.0,
- "90": 1289.0,
- "91": 1052.0,
- "92": 1303.0,
- "93": 1235.0,
- "94": 1301.0,
- "95": 1386.0,
- "96": 2364.0,
- "97": 1408.0,
- "98": 2551.0,
- "99": 1263.0,
- "100": 1227.0
+ "1": 1166.0,
+ "2": 1181.0,
+ "3": 1363.0,
+ "4": 1284.0,
+ "5": 1232.0,
+ "6": 1243.0,
+ "7": 1333.0,
+ "8": 1202.0,
+ "9": 1220.0,
+ "10": 1150.0,
+ "11": 1194.0,
+ "12": 1122.0,
+ "13": 1432.0,
+ "14": 1291.0,
+ "15": 1143.0,
+ "16": 1275.0,
+ "17": 1306.0,
+ "18": 1206.0,
+ "19": 1149.0,
+ "20": 1169.0,
+ "21": 1086.0,
+ "22": 1175.0,
+ "23": 1247.0,
+ "24": 1218.0,
+ "25": 1182.0,
+ "26": 1206.0,
+ "27": 1234.0,
+ "28": 1165.0,
+ "29": 1139.0,
+ "30": 1189.0,
+ "31": 1120.0,
+ "32": 1165.0,
+ "33": 1212.0,
+ "34": 1126.0,
+ "35": 1194.0,
+ "36": 1091.0,
+ "37": 1205.0,
+ "38": 1166.0,
+ "39": 1239.0,
+ "40": 1169.0,
+ "41": 1251.0,
+ "42": 1216.0,
+ "43": 1188.0,
+ "44": 1223.0,
+ "45": 1275.0,
+ "46": 1274.0,
+ "47": 1258.0,
+ "48": 1250.0,
+ "49": 1221.0,
+ "50": 1312.0,
+ "51": 1446.0,
+ "52": 1317.0,
+ "53": 1238.0,
+ "54": 1340.0,
+ "55": 1217.0,
+ "56": 1327.0,
+ "57": 1202.0,
+ "58": 1050723.0,
+ "59": 1441.0,
+ "60": 1361.0,
+ "61": 42285.0,
+ "62": 49508.0,
+ "63": 1572.0,
+ "64": 42541.0,
+ "65": 1314.0,
+ "66": 1050570.0,
+ "67": 50705.0,
+ "68": 1500.0,
+ "69": 5560.0,
+ "70": 6638.0,
+ "71": 1428.0,
+ "72": 102832.0,
+ "73": 7694.0,
+ "74": 116137.0,
+ "75": 11709.0,
+ "76": 70169.0,
+ "77": 1660.0,
+ "78": 9793.0,
+ "79": 6594.0,
+ "80": 1439.0,
+ "81": 2698.0,
+ "82": 2750.0,
+ "83": 1485.0,
+ "84": 1050470.0,
+ "85": 1478.0,
+ "86": 1524.0,
+ "87": 2151910.0,
+ "88": 2428.0,
+ "89": 10820.0,
+ "90": 1060047.0,
+ "91": 1247.0,
+ "92": 2099761.0,
+ "93": 10760.0,
+ "94": 2758.0,
+ "95": 1050685.0,
+ "96": 116421.0,
+ "97": 1050925.0,
+ "98": 53896.0,
+ "99": 5468.0,
+ "100": 1050666.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 788555776.0,
- "2": 788525568.0,
- "3": 788572672.0,
- "4": 788552704.0,
- "5": 788574720.0,
- "6": 788517888.0,
- "7": 788541440.0,
- "8": 788548096.0,
- "9": 788562944.0,
- "10": 788577280.0,
- "11": 788553216.0,
- "12": 788523008.0,
- "13": 788579328.0,
- "14": 788489216.0,
- "15": 788539904.0,
- "16": 788476928.0,
- "17": 788598784.0,
- "18": 788574208.0,
- "19": 788580864.0,
- "20": 788508160.0,
- "21": 788545536.0,
- "22": 788632064.0,
- "23": 788610560.0,
- "24": 788551168.0,
- "25": 788694016.0,
- "26": 788605440.0,
- "27": 788667904.0,
- "28": 788609024.0,
- "29": 788647936.0,
- "30": 788625408.0,
- "31": 788685824.0,
- "32": 788640768.0,
- "33": 788655616.0,
- "34": 788615680.0,
- "35": 788654080.0,
- "36": 788679680.0,
- "37": 788634624.0,
- "38": 788688896.0,
- "39": 788698112.0,
- "40": 788588032.0,
- "41": 788628992.0,
- "42": 788613632.0,
- "43": 788577792.0,
- "44": 788661248.0,
- "45": 788528640.0,
- "46": 788540928.0,
- "47": 788609536.0,
- "48": 788528640.0,
- "49": 788498944.0,
- "50": 788524544.0,
- "51": 788559872.0,
- "52": 788518400.0,
- "53": 788552192.0,
- "54": 788543488.0,
- "55": 788538880.0,
- "56": 788497408.0,
- "57": 788493824.0,
- "58": 788537344.0,
- "59": 788539904.0,
- "60": 788527104.0,
- "61": 788499968.0,
- "62": 788535296.0,
- "63": 788487168.0,
- "64": 788503552.0,
- "65": 788446208.0,
- "66": 788485632.0,
- "67": 788485120.0,
- "68": 788493312.0,
- "69": 788508672.0,
- "70": 788534784.0,
- "71": 788491264.0,
- "72": 788452864.0,
- "73": 788477440.0,
- "74": 788452864.0,
- "75": 788480000.0,
- "76": 788450304.0,
- "77": 788506624.0,
- "78": 788500992.0,
- "79": 788451840.0,
- "80": 788448256.0,
- "81": 788466176.0,
- "82": 788474880.0,
- "83": 788470784.0,
- "84": 788506624.0,
- "85": 788496384.0,
- "86": 788430848.0,
- "87": 788528128.0,
- "88": 788530176.0,
- "89": 788537856.0,
- "90": 788569600.0,
- "91": 788549632.0,
- "92": 788555264.0,
- "93": 788525056.0,
- "94": 788546560.0,
- "95": 788583424.0,
- "96": 788601856.0,
- "97": 788617216.0,
- "98": 788629504.0,
- "99": 788551680.0,
- "100": 788611072.0
+ "1": 788519424.0,
+ "2": 788509696.0,
+ "3": 788487680.0,
+ "4": 788466176.0,
+ "5": 788516352.0,
+ "6": 788500992.0,
+ "7": 788425216.0,
+ "8": 788463104.0,
+ "9": 788468736.0,
+ "10": 788498432.0,
+ "11": 788475392.0,
+ "12": 788484096.0,
+ "13": 788488704.0,
+ "14": 788493312.0,
+ "15": 788476928.0,
+ "16": 788466176.0,
+ "17": 788475904.0,
+ "18": 788482048.0,
+ "19": 788513280.0,
+ "20": 788489216.0,
+ "21": 788556288.0,
+ "22": 788593152.0,
+ "23": 788578304.0,
+ "24": 788488704.0,
+ "25": 788578304.0,
+ "26": 788621824.0,
+ "27": 788564480.0,
+ "28": 788536320.0,
+ "29": 788598272.0,
+ "30": 788570112.0,
+ "31": 788587520.0,
+ "32": 788577280.0,
+ "33": 788574720.0,
+ "34": 788566528.0,
+ "35": 788553728.0,
+ "36": 788605440.0,
+ "37": 788551680.0,
+ "38": 788575232.0,
+ "39": 788594688.0,
+ "40": 788553728.0,
+ "41": 788546048.0,
+ "42": 788505600.0,
+ "43": 788537344.0,
+ "44": 788486656.0,
+ "45": 788457984.0,
+ "46": 788500992.0,
+ "47": 788451328.0,
+ "48": 788436992.0,
+ "49": 788419072.0,
+ "50": 788387328.0,
+ "51": 788402688.0,
+ "52": 788373504.0,
+ "53": 788323328.0,
+ "54": 788359168.0,
+ "55": 788352512.0,
+ "56": 788315136.0,
+ "57": 788389888.0,
+ "58": 788365312.0,
+ "59": 788350976.0,
+ "60": 788371456.0,
+ "61": 788308480.0,
+ "62": 788298752.0,
+ "63": 788332544.0,
+ "64": 788336640.0,
+ "65": 788326912.0,
+ "66": 788332544.0,
+ "67": 788338688.0,
+ "68": 788328960.0,
+ "69": 788331008.0,
+ "70": 788352512.0,
+ "71": 788346368.0,
+ "72": 788392960.0,
+ "73": 788352000.0,
+ "74": 788381184.0,
+ "75": 788321280.0,
+ "76": 788293120.0,
+ "77": 788397568.0,
+ "78": 788359168.0,
+ "79": 788318208.0,
+ "80": 788277760.0,
+ "81": 788284928.0,
+ "82": 788241920.0,
+ "83": 788276736.0,
+ "84": 788320256.0,
+ "85": 788314624.0,
+ "86": 788340736.0,
+ "87": 788148224.0,
+ "88": 788256768.0,
+ "89": 788264448.0,
+ "90": 788251648.0,
+ "91": 788319232.0,
+ "92": 788241408.0,
+ "93": 788247552.0,
+ "94": 788389888.0,
+ "95": 788252672.0,
+ "96": 788294144.0,
+ "97": 788358144.0,
+ "98": 788328448.0,
+ "99": 788267008.0,
+ "100": 788357120.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 3121186304.0,
- "2": 3272137728.0,
- "3": 3305329664.0,
- "4": 3305329664.0,
- "5": 3309687808.0,
- "6": 3309687808.0,
- "7": 3309687808.0,
- "8": 3309687808.0,
- "9": 3309687808.0,
- "10": 3309926912.0,
- "11": 3309926912.0,
- "12": 3309926912.0,
- "13": 3309926912.0,
- "14": 3309926912.0,
- "15": 3309926912.0,
- "16": 3309926912.0,
- "17": 3318584832.0,
- "18": 3318584832.0,
- "19": 3318584832.0,
- "20": 3318584832.0,
- "21": 3318584832.0,
- "22": 3346422784.0,
- "23": 3346422784.0,
- "24": 3346422784.0,
- "25": 3392057856.0,
- "26": 3392057856.0,
- "27": 3392057856.0,
- "28": 3392057856.0,
- "29": 3392057856.0,
- "30": 3392057856.0,
- "31": 3392057856.0,
- "32": 3392057856.0,
- "33": 3392057856.0,
- "34": 3392057856.0,
- "35": 3392057856.0,
- "36": 3392057856.0,
- "37": 3392057856.0,
- "38": 3392057856.0,
- "39": 3392057856.0,
- "40": 3392057856.0,
- "41": 3392057856.0,
- "42": 3392057856.0,
- "43": 3392057856.0,
- "44": 3392057856.0,
- "45": 3392057856.0,
- "46": 3392057856.0,
- "47": 3392057856.0,
- "48": 3392057856.0,
- "49": 3392057856.0,
- "50": 3392057856.0,
- "51": 3392057856.0,
- "52": 3392057856.0,
- "53": 3392057856.0,
- "54": 3392057856.0,
- "55": 3392057856.0,
- "56": 3392057856.0,
- "57": 3392057856.0,
- "58": 3392057856.0,
- "59": 3392057856.0,
- "60": 3392057856.0,
- "61": 3392057856.0,
- "62": 3392057856.0,
- "63": 3392057856.0,
- "64": 3392057856.0,
- "65": 3392057856.0,
- "66": 3392057856.0,
- "67": 3392057856.0,
- "68": 3392057856.0,
- "69": 3392057856.0,
- "70": 3392057856.0,
- "71": 3392057856.0,
- "72": 3392057856.0,
- "73": 3392057856.0,
- "74": 3392057856.0,
- "75": 3392057856.0,
- "76": 3392057856.0,
- "77": 3392057856.0,
- "78": 3392057856.0,
- "79": 3392057856.0,
- "80": 3392057856.0,
- "81": 3392057856.0,
- "82": 3392057856.0,
- "83": 3392057856.0,
- "84": 3392057856.0,
- "85": 3392057856.0,
- "86": 3392057856.0,
- "87": 3392057856.0,
- "88": 3392057856.0,
- "89": 3392057856.0,
- "90": 3392057856.0,
- "91": 3392057856.0,
- "92": 3392057856.0,
- "93": 3392057856.0,
- "94": 3392057856.0,
- "95": 3392057856.0,
- "96": 3392057856.0,
- "97": 3392057856.0,
- "98": 3392057856.0,
- "99": 3392057856.0,
- "100": 3392057856.0
+ "1": 3122096128.0,
+ "2": 3285577216.0,
+ "3": 3285577216.0,
+ "4": 3285577216.0,
+ "5": 3288475648.0,
+ "6": 3288475648.0,
+ "7": 3288475648.0,
+ "8": 3288475648.0,
+ "9": 3288475648.0,
+ "10": 3288475648.0,
+ "11": 3288475648.0,
+ "12": 3288475648.0,
+ "13": 3288475648.0,
+ "14": 3288475648.0,
+ "15": 3288475648.0,
+ "16": 3288475648.0,
+ "17": 3288475648.0,
+ "18": 3288475648.0,
+ "19": 3288475648.0,
+ "20": 3288475648.0,
+ "21": 3315433472.0,
+ "22": 3342736384.0,
+ "23": 3342736384.0,
+ "24": 3342736384.0,
+ "25": 3342736384.0,
+ "26": 3359504384.0,
+ "27": 3359504384.0,
+ "28": 3359504384.0,
+ "29": 3359504384.0,
+ "30": 3359504384.0,
+ "31": 3359504384.0,
+ "32": 3359504384.0,
+ "33": 3359504384.0,
+ "34": 3359504384.0,
+ "35": 3359504384.0,
+ "36": 3359504384.0,
+ "37": 3359504384.0,
+ "38": 3359504384.0,
+ "39": 3359504384.0,
+ "40": 3359504384.0,
+ "41": 3359504384.0,
+ "42": 3359504384.0,
+ "43": 3359504384.0,
+ "44": 3359504384.0,
+ "45": 3359504384.0,
+ "46": 3359504384.0,
+ "47": 3359504384.0,
+ "48": 3359504384.0,
+ "49": 3359504384.0,
+ "50": 3359504384.0,
+ "51": 3359504384.0,
+ "52": 3359504384.0,
+ "53": 3359504384.0,
+ "54": 3359504384.0,
+ "55": 3359504384.0,
+ "56": 3359504384.0,
+ "57": 3359504384.0,
+ "58": 3359504384.0,
+ "59": 3359504384.0,
+ "60": 3359504384.0,
+ "61": 3359504384.0,
+ "62": 3359504384.0,
+ "63": 3359504384.0,
+ "64": 3359504384.0,
+ "65": 3359504384.0,
+ "66": 3359504384.0,
+ "67": 3359504384.0,
+ "68": 3359504384.0,
+ "69": 3359504384.0,
+ "70": 3359504384.0,
+ "71": 3359504384.0,
+ "72": 3359504384.0,
+ "73": 3359504384.0,
+ "74": 3359504384.0,
+ "75": 3359504384.0,
+ "76": 3359504384.0,
+ "77": 3359504384.0,
+ "78": 3359504384.0,
+ "79": 3359504384.0,
+ "80": 3359504384.0,
+ "81": 3359504384.0,
+ "82": 3359504384.0,
+ "83": 3359504384.0,
+ "84": 3359504384.0,
+ "85": 3359504384.0,
+ "86": 3359504384.0,
+ "87": 3359504384.0,
+ "88": 3359504384.0,
+ "89": 3359504384.0,
+ "90": 3359504384.0,
+ "91": 3359504384.0,
+ "92": 3359504384.0,
+ "93": 3359504384.0,
+ "94": 3359504384.0,
+ "95": 3359504384.0,
+ "96": 3359504384.0,
+ "97": 3359504384.0,
+ "98": 3359504384.0,
+ "99": 3359504384.0,
+ "100": 3359504384.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 12.9672,
- "2": 0.18032,
- "3": 0.16621,
- "4": 0.14138,
- "5": 0.14697,
- "6": 0.12745,
- "7": 0.13018,
- "8": 0.1308,
- "9": 0.12325,
- "10": 0.11929,
- "11": 0.11868,
- "12": 0.11662,
- "13": 0.11935,
- "14": 0.12579,
- "15": 0.10685,
- "16": 0.1235,
- "17": 0.11712,
- "18": 0.11351,
- "19": 0.11956,
- "20": 0.12036,
- "21": 0.11206,
- "22": 0.12061,
- "23": 0.11918,
- "24": 0.11718,
- "25": 0.11286,
- "26": 0.11553,
- "27": 0.12325,
- "28": 0.12425,
- "29": 0.1373,
- "30": 0.14042,
- "31": 0.12588,
- "32": 0.12886,
- "33": 0.11871,
- "34": 0.1268,
- "35": 0.12631,
- "36": 0.13682,
- "37": 0.12561,
- "38": 0.12806,
- "39": 0.13203,
- "40": 0.13218,
- "41": 0.12224,
- "42": 0.13858,
- "43": 0.13174,
- "44": 0.12012,
- "45": 0.12567,
- "46": 0.13565,
- "47": 0.12427,
- "48": 0.11574,
- "49": 0.11974,
- "50": 0.12631,
- "51": 0.14169,
- "52": 0.11509,
- "53": 0.1256,
- "54": 0.1169,
- "55": 0.12608,
- "56": 0.11705,
- "57": 0.12085,
- "58": 0.11877,
- "59": 0.1187,
- "60": 0.12978,
- "61": 0.11339,
- "62": 0.1117,
- "63": 0.12276,
- "64": 0.12623,
- "65": 0.1311,
- "66": 0.1174,
- "67": 0.12925,
- "68": 0.11502,
- "69": 0.1185,
- "70": 0.12525,
- "71": 0.10756,
- "72": 0.11771,
- "73": 0.1132,
- "74": 0.12549,
- "75": 0.10854,
- "76": 0.11252,
- "77": 0.11354,
- "78": 0.10942,
- "79": 0.11618,
- "80": 0.1066,
- "81": 0.11024,
- "82": 0.10189,
- "83": 0.10909,
- "84": 0.14864,
- "85": 0.10374,
- "86": 0.10395,
- "87": 0.10291,
- "88": 0.11323,
- "89": 0.10749,
- "90": 0.10777,
- "91": 0.10528,
- "92": 0.10628,
- "93": 0.10398,
- "94": 0.11116,
- "95": 0.10621,
- "96": 0.11081,
- "97": 0.11111,
- "98": 0.09872,
- "99": 0.1051,
- "100": 0.10136
+ "1": "nan",
+ "2": 6.8759,
+ "3": 0.17485,
+ "4": 0.14494,
+ "5": 0.13544,
+ "6": 0.12337,
+ "7": 0.13263,
+ "8": 0.14855,
+ "9": 0.12809,
+ "10": 0.1206,
+ "11": 0.13386,
+ "12": 0.11969,
+ "13": 0.11431,
+ "14": 0.12701,
+ "15": 0.11886,
+ "16": 0.12188,
+ "17": 0.12931,
+ "18": 0.11477,
+ "19": 0.12377,
+ "20": 0.12332,
+ "21": 0.12052,
+ "22": 0.11921,
+ "23": 0.1154,
+ "24": 0.12101,
+ "25": 0.13066,
+ "26": 0.12004,
+ "27": 0.12764,
+ "28": 0.13841,
+ "29": 0.14529,
+ "30": 0.13385,
+ "31": 0.12841,
+ "32": 0.13691,
+ "33": 0.13668,
+ "34": 0.1402,
+ "35": 0.1298,
+ "36": 0.12886,
+ "37": 0.13783,
+ "38": 0.13713,
+ "39": 0.13075,
+ "40": 0.12028,
+ "41": 0.13382,
+ "42": 0.13672,
+ "43": 0.13903,
+ "44": 0.1337,
+ "45": 0.1247,
+ "46": 0.13228,
+ "47": 0.11765,
+ "48": 0.12127,
+ "49": 0.11427,
+ "50": 0.11221,
+ "51": 0.17671,
+ "52": 0.1284,
+ "53": 0.12087,
+ "54": 0.10991,
+ "55": 0.11918,
+ "56": 0.11083,
+ "57": 0.13169,
+ "58": 0.16886,
+ "59": 0.11912,
+ "60": 0.11761,
+ "61": 0.11521,
+ "62": 0.12652,
+ "63": 0.10929,
+ "64": 0.15753,
+ "65": 0.11806,
+ "66": 0.11134,
+ "67": 0.11208,
+ "68": 0.10895,
+ "69": 0.11551,
+ "70": 0.11444,
+ "71": 0.12117,
+ "72": 0.11983,
+ "73": 0.11787,
+ "74": 0.10632,
+ "75": 0.10978,
+ "76": 0.11257,
+ "77": 0.11854,
+ "78": 0.11671,
+ "79": 0.11719,
+ "80": 0.11469,
+ "81": 0.10308,
+ "82": 0.10761,
+ "83": 0.11344,
+ "84": 0.11203,
+ "85": 0.10343,
+ "86": 0.11222,
+ "87": 0.1061,
+ "88": 0.11771,
+ "89": 0.1112,
+ "90": 0.11039,
+ "91": 0.10705,
+ "92": 0.10731,
+ "93": 0.10316,
+ "94": 0.10265,
+ "95": 0.11107,
+ "96": 0.10482,
+ "97": 0.10541,
+ "98": 0.11291,
+ "99": 0.10379,
+ "100": 0.10477
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer/model_config.yaml
index cd7656d240f..f69a44638d6 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -61,4 +61,7 @@ MODEL_ARGS:
--bf16: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --async-strategy: mcore
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon/golden_values_dev_dgx_gb200.json
index fe8d3f78926..a452733bca6 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon/golden_values_dev_dgx_gb200.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.80397,
- "2": 10.81064,
- "3": 10.79812,
- "4": 10.78677,
- "5": 10.82981,
- "6": 10.85273,
- "7": 10.80976,
- "8": 10.80152,
- "9": 10.82476,
- "10": 10.78235,
- "11": 10.83837,
- "12": 10.84645,
- "13": 10.86121,
- "14": 10.86494,
- "15": 10.83809,
- "16": 10.8346,
- "17": 10.8121,
- "18": 10.84437,
- "19": 10.83592,
- "20": 10.81732,
- "21": 10.83519,
- "22": 10.76256,
- "23": 10.85511,
- "24": 10.80666,
- "25": 10.80025,
- "26": 10.81426,
- "27": 10.82995,
- "28": 10.855,
- "29": 10.86565,
- "30": 10.79384,
- "31": 10.74771,
- "32": 10.84943,
- "33": 10.83771,
- "34": 10.80572,
- "35": 10.80265,
- "36": 10.79622,
- "37": 10.82514,
- "38": 10.79237,
- "39": 10.84811,
- "40": 10.77883,
- "41": 10.79922,
- "42": 10.81563,
- "43": 10.74376,
- "44": 10.76683,
- "45": 10.76467,
- "46": 10.77697,
- "47": 10.79973,
- "48": 10.77586,
- "49": 10.72215,
- "50": 10.78584,
- "51": 10.78731,
- "52": 10.7657,
- "53": 10.81241,
- "54": 10.79761,
- "55": 10.80688,
- "56": 10.75611,
- "57": 10.71341,
- "58": 10.78104,
- "59": 10.7507,
- "60": 10.72941,
- "61": 10.76448,
- "62": 10.8119,
- "63": 10.69242,
- "64": 10.76661,
- "65": 10.62474,
- "66": 10.75342,
- "67": 10.69134,
- "68": 10.77079,
- "69": 10.76029,
- "70": 10.76451,
- "71": 10.73531,
- "72": 10.72951,
- "73": 10.7174,
- "74": 10.57782,
- "75": 10.68245,
- "76": 10.61342,
- "77": 10.80749,
- "78": 10.7321,
- "79": 10.66078,
- "80": 10.68008,
- "81": 10.69796,
- "82": 10.72301,
- "83": 10.6413,
- "84": 10.6619,
- "85": 10.70249,
- "86": 10.58035,
- "87": 10.69015,
- "88": 10.73441,
- "89": 10.67777,
- "90": 10.74269,
- "91": 10.62186,
- "92": 10.63964,
- "93": 10.56627,
- "94": 10.49913,
- "95": 10.65738,
- "96": 10.65873,
- "97": 10.57872,
- "98": 10.6722,
- "99": 10.4802,
- "100": 10.59334
+ "1": 10.94254,
+ "2": 10.91055,
+ "3": 10.92749,
+ "4": 10.9172,
+ "5": 10.92125,
+ "6": 10.92734,
+ "7": 10.92252,
+ "8": 10.9202,
+ "9": 10.93241,
+ "10": 10.92732,
+ "11": 10.91806,
+ "12": 10.9348,
+ "13": 10.92014,
+ "14": 10.91043,
+ "15": 10.91833,
+ "16": 10.92081,
+ "17": 10.93475,
+ "18": 10.91652,
+ "19": 10.92803,
+ "20": 10.90288,
+ "21": 10.92619,
+ "22": 10.92311,
+ "23": 10.92973,
+ "24": 10.90455,
+ "25": 10.9139,
+ "26": 10.91431,
+ "27": 10.92941,
+ "28": 10.91943,
+ "29": 10.92005,
+ "30": 10.91149,
+ "31": 10.90651,
+ "32": 10.91828,
+ "33": 10.91653,
+ "34": 10.90394,
+ "35": 10.92705,
+ "36": 10.90766,
+ "37": 10.90571,
+ "38": 10.91709,
+ "39": 10.9095,
+ "40": 10.9033,
+ "41": 10.91056,
+ "42": 10.91546,
+ "43": 10.90837,
+ "44": 10.91646,
+ "45": 10.91001,
+ "46": 10.90588,
+ "47": 10.90072,
+ "48": 10.89352,
+ "49": 10.90411,
+ "50": 10.89328,
+ "51": 10.89886,
+ "52": 10.89757,
+ "53": 10.90026,
+ "54": 10.89152,
+ "55": 10.90606,
+ "56": 10.88415,
+ "57": 10.88888,
+ "58": 10.88002,
+ "59": 10.87868,
+ "60": 10.86711,
+ "61": 10.87112,
+ "62": 10.86274,
+ "63": 10.87041,
+ "64": 10.8628,
+ "65": 10.86538,
+ "66": 10.86813,
+ "67": 10.85587,
+ "68": 10.84838,
+ "69": 10.85358,
+ "70": 10.85748,
+ "71": 10.85083,
+ "72": 10.85011,
+ "73": 10.83944,
+ "74": 10.83515,
+ "75": 10.84168,
+ "76": 10.82837,
+ "77": 10.82987,
+ "78": 10.81247,
+ "79": 10.82925,
+ "80": 10.83442,
+ "81": 10.81577,
+ "82": 10.81471,
+ "83": 10.80426,
+ "84": 10.78025,
+ "85": 10.78108,
+ "86": 10.79767,
+ "87": 10.79871,
+ "88": 10.80121,
+ "89": 10.78498,
+ "90": 10.78248,
+ "91": 10.78244,
+ "92": 10.78247,
+ "93": 10.75869,
+ "94": 10.76721,
+ "95": 10.76509,
+ "96": 10.7388,
+ "97": 10.72982,
+ "98": 10.75358,
+ "99": 10.75959,
+ "100": 10.73821
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1158.0,
- "2": 1181.0,
- "3": 1248.0,
- "4": 1219.0,
- "5": 1372.0,
- "6": 1529.0,
- "7": 1362.0,
- "8": 1179.0,
- "9": 1276.0,
- "10": 1234.0,
- "11": 1281.0,
- "12": 1249.0,
- "13": 1386.0,
- "14": 1213.0,
- "15": 1215.0,
- "16": 1299.0,
- "17": 1242.0,
- "18": 1233.0,
- "19": 1167.0,
- "20": 1392.0,
- "21": 1264.0,
- "22": 1289.0,
- "23": 1336.0,
- "24": 1168.0,
- "25": 1170.0,
- "26": 1207.0,
- "27": 1192.0,
- "28": 1327.0,
- "29": 1354.0,
- "30": 1250.0,
- "31": 1110.0,
- "32": 1331.0,
- "33": 1340.0,
- "34": 1250.0,
- "35": 1105.0,
- "36": 1138.0,
- "37": 1265.0,
- "38": 1375.0,
- "39": 1243.0,
- "40": 1306.0,
- "41": 1154.0,
- "42": 1251.0,
- "43": 1122.0,
- "44": 1139.0,
- "45": 1122.0,
- "46": 1203.0,
- "47": 1405.0,
- "48": 1282.0,
- "49": 1167.0,
- "50": 1166.0,
- "51": 1249.0,
- "52": 1320.0,
- "53": 1340.0,
- "54": 1232.0,
- "55": 1103.0,
- "56": 1275.0,
- "57": 1194.0,
- "58": 1259.0,
- "59": 1283.0,
- "60": 1265.0,
- "61": 1124.0,
- "62": 1349.0,
- "63": 1132.0,
- "64": 1272.0,
- "65": 1017.0,
- "66": 1174.0,
- "67": 1242.0,
- "68": 1291.0,
- "69": 1295.0,
- "70": 1143.0,
- "71": 1148.0,
- "72": 1266.0,
- "73": 1199.0,
- "74": 1133.0,
- "75": 1346.0,
- "76": 1224.0,
- "77": 1329.0,
- "78": 1256.0,
- "79": 997.0,
- "80": 1093.0,
- "81": 1204.0,
- "82": 1213.0,
- "83": 1128.0,
+ "1": 1411.0,
+ "2": 1137.0,
+ "3": 1245.0,
+ "4": 1197.0,
+ "5": 1177.0,
+ "6": 1316.0,
+ "7": 1570.0,
+ "8": 1317.0,
+ "9": 1281.0,
+ "10": 1251.0,
+ "11": 1267.0,
+ "12": 1255.0,
+ "13": 1444.0,
+ "14": 1368.0,
+ "15": 1136.0,
+ "16": 1301.0,
+ "17": 1336.0,
+ "18": 1255.0,
+ "19": 1236.0,
+ "20": 1240.0,
+ "21": 1399.0,
+ "22": 1274.0,
+ "23": 1277.0,
+ "24": 1493.0,
+ "25": 1306.0,
+ "26": 1354.0,
+ "27": 1239.0,
+ "28": 1249.0,
+ "29": 1319.0,
+ "30": 1388.0,
+ "31": 1298.0,
+ "32": 1468.0,
+ "33": 1420.0,
+ "34": 1451.0,
+ "35": 1327.0,
+ "36": 1239.0,
+ "37": 1368.0,
+ "38": 1269.0,
+ "39": 1232.0,
+ "40": 1339.0,
+ "41": 1246.0,
+ "42": 1079.0,
+ "43": 1367.0,
+ "44": 1322.0,
+ "45": 1395.0,
+ "46": 1200.0,
+ "47": 1387.0,
+ "48": 1355.0,
+ "49": 1369.0,
+ "50": 1280.0,
+ "51": 1218.0,
+ "52": 1274.0,
+ "53": 1271.0,
+ "54": 1221.0,
+ "55": 1261.0,
+ "56": 1198.0,
+ "57": 1270.0,
+ "58": 1341.0,
+ "59": 1289.0,
+ "60": 1405.0,
+ "61": 1093.0,
+ "62": 1387.0,
+ "63": 1236.0,
+ "64": 1276.0,
+ "65": 1430.0,
+ "66": 1236.0,
+ "67": 1439.0,
+ "68": 1194.0,
+ "69": 1406.0,
+ "70": 1278.0,
+ "71": 1150.0,
+ "72": 1357.0,
+ "73": 1329.0,
+ "74": 1282.0,
+ "75": 1361.0,
+ "76": 1378.0,
+ "77": 1325.0,
+ "78": 1284.0,
+ "79": 1300.0,
+ "80": 1283.0,
+ "81": 1350.0,
+ "82": 1132.0,
+ "83": 1316.0,
"84": 1228.0,
- "85": 1316.0,
- "86": 1101.0,
- "87": 1278.0,
- "88": 1286.0,
- "89": 1163.0,
- "90": 1415.0,
- "91": 1248.0,
- "92": 1137.0,
- "93": 912.0,
- "94": 985.0,
- "95": 1097.0,
- "96": 1087.0,
- "97": 1098.0,
- "98": 1170.0,
- "99": 1047.0,
- "100": 1205.0
+ "85": 1165.0,
+ "86": 1250.0,
+ "87": 1156.0,
+ "88": 1331.0,
+ "89": 1200.0,
+ "90": 1228.0,
+ "91": 1152.0,
+ "92": 1153.0,
+ "93": 1317.0,
+ "94": 1387.0,
+ "95": 1163.0,
+ "96": 1262.0,
+ "97": 1225.0,
+ "98": 1224.0,
+ "99": 1263.0,
+ "100": 1401.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1095588352.0,
- "2": 1095600640.0,
- "3": 1095576576.0,
- "4": 1095606272.0,
- "5": 1095514624.0,
- "6": 1095542272.0,
- "7": 1095563776.0,
- "8": 1095580160.0,
- "9": 1095585792.0,
- "10": 1095554048.0,
- "11": 1095538176.0,
- "12": 1095523328.0,
- "13": 1095589888.0,
- "14": 1095519744.0,
- "15": 1095557120.0,
- "16": 1095548928.0,
- "17": 1095531008.0,
- "18": 1095528448.0,
- "19": 1095549440.0,
- "20": 1095504384.0,
- "21": 1095561728.0,
- "22": 1095583232.0,
- "23": 1095534592.0,
- "24": 1095604736.0,
- "25": 1095577088.0,
- "26": 1095565824.0,
- "27": 1095591424.0,
- "28": 1095485952.0,
- "29": 1095502848.0,
- "30": 1095552512.0,
- "31": 1095588352.0,
- "32": 1095491072.0,
- "33": 1095547392.0,
- "34": 1095568384.0,
- "35": 1095527424.0,
- "36": 1095533568.0,
- "37": 1095482880.0,
- "38": 1095552000.0,
- "39": 1095532544.0,
- "40": 1095567360.0,
- "41": 1095537152.0,
- "42": 1095543296.0,
- "43": 1095581184.0,
- "44": 1095620096.0,
- "45": 1095569408.0,
- "46": 1095584768.0,
- "47": 1095573504.0,
- "48": 1095577088.0,
- "49": 1095530496.0,
- "50": 1095540736.0,
- "51": 1095570944.0,
- "52": 1095538176.0,
- "53": 1095597568.0,
- "54": 1095536640.0,
- "55": 1095517184.0,
- "56": 1095566848.0,
- "57": 1095645696.0,
- "58": 1095634944.0,
- "59": 1095548928.0,
- "60": 1095562752.0,
- "61": 1095553536.0,
- "62": 1095572480.0,
- "63": 1095573504.0,
- "64": 1095550464.0,
- "65": 1095578112.0,
- "66": 1095531008.0,
- "67": 1095568896.0,
- "68": 1095566848.0,
- "69": 1095527424.0,
- "70": 1095532032.0,
- "71": 1095520768.0,
- "72": 1095548928.0,
- "73": 1095569920.0,
- "74": 1095596032.0,
- "75": 1095538688.0,
- "76": 1095584768.0,
- "77": 1095507968.0,
- "78": 1095514624.0,
- "79": 1095515648.0,
- "80": 1095551488.0,
- "81": 1095513600.0,
- "82": 1095498240.0,
- "83": 1095558656.0,
- "84": 1095569408.0,
- "85": 1095576064.0,
- "86": 1095590400.0,
- "87": 1095523840.0,
- "88": 1095517696.0,
- "89": 1095539712.0,
- "90": 1095528960.0,
- "91": 1095550976.0,
- "92": 1095561216.0,
- "93": 1095579136.0,
- "94": 1095564288.0,
- "95": 1095510528.0,
- "96": 1095502336.0,
- "97": 1095537152.0,
- "98": 1095496192.0,
- "99": 1095577600.0,
- "100": 1095598592.0
+ "1": 1096852992.0,
+ "2": 1096867328.0,
+ "3": 1096857600.0,
+ "4": 1096843776.0,
+ "5": 1096850944.0,
+ "6": 1096835584.0,
+ "7": 1096857088.0,
+ "8": 1096825856.0,
+ "9": 1096852480.0,
+ "10": 1096847360.0,
+ "11": 1096826368.0,
+ "12": 1096817664.0,
+ "13": 1096904192.0,
+ "14": 1096916480.0,
+ "15": 1096852480.0,
+ "16": 1096854528.0,
+ "17": 1096863744.0,
+ "18": 1096825856.0,
+ "19": 1096884224.0,
+ "20": 1096849920.0,
+ "21": 1096899584.0,
+ "22": 1096850944.0,
+ "23": 1096825344.0,
+ "24": 1096863744.0,
+ "25": 1096814592.0,
+ "26": 1096926720.0,
+ "27": 1096856064.0,
+ "28": 1096851968.0,
+ "29": 1096888320.0,
+ "30": 1096811520.0,
+ "31": 1096846848.0,
+ "32": 1096851968.0,
+ "33": 1096839168.0,
+ "34": 1096899072.0,
+ "35": 1096836096.0,
+ "36": 1096773632.0,
+ "37": 1096848896.0,
+ "38": 1096876032.0,
+ "39": 1096820224.0,
+ "40": 1096855040.0,
+ "41": 1096857088.0,
+ "42": 1096849408.0,
+ "43": 1096853504.0,
+ "44": 1096822784.0,
+ "45": 1096819200.0,
+ "46": 1096878592.0,
+ "47": 1096875008.0,
+ "48": 1096821248.0,
+ "49": 1096846848.0,
+ "50": 1096797184.0,
+ "51": 1096783360.0,
+ "52": 1096806912.0,
+ "53": 1096848384.0,
+ "54": 1096846848.0,
+ "55": 1096829952.0,
+ "56": 1096840192.0,
+ "57": 1096873472.0,
+ "58": 1096908800.0,
+ "59": 1096815104.0,
+ "60": 1096829952.0,
+ "61": 1096800256.0,
+ "62": 1096829952.0,
+ "63": 1096842752.0,
+ "64": 1096846336.0,
+ "65": 1096830976.0,
+ "66": 1096768512.0,
+ "67": 1096847360.0,
+ "68": 1096860160.0,
+ "69": 1096836608.0,
+ "70": 1096788992.0,
+ "71": 1096812032.0,
+ "72": 1096883712.0,
+ "73": 1096814592.0,
+ "74": 1096811520.0,
+ "75": 1096850944.0,
+ "76": 1096825856.0,
+ "77": 1096840192.0,
+ "78": 1096831488.0,
+ "79": 1096822784.0,
+ "80": 1096841728.0,
+ "81": 1096826880.0,
+ "82": 1096799744.0,
+ "83": 1096858112.0,
+ "84": 1096846336.0,
+ "85": 1096859136.0,
+ "86": 1096833536.0,
+ "87": 1096730112.0,
+ "88": 1096784384.0,
+ "89": 1096793600.0,
+ "90": 1096820736.0,
+ "91": 1096846336.0,
+ "92": 1096797184.0,
+ "93": 1096763904.0,
+ "94": 1096806912.0,
+ "95": 1096793600.0,
+ "96": 1096770048.0,
+ "97": 1096795648.0,
+ "98": 1096860672.0,
+ "99": 1096819712.0,
+ "100": 1096851456.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 3125957632.0,
- "2": 3477050368.0,
- "3": 3477050368.0,
- "4": 3481636352.0,
- "5": 3481636352.0,
- "6": 3481636352.0,
- "7": 3481636352.0,
- "8": 3481636352.0,
- "9": 3481636352.0,
- "10": 3481636352.0,
- "11": 3481636352.0,
- "12": 3481636352.0,
- "13": 3481636352.0,
- "14": 3481636352.0,
- "15": 3481636352.0,
- "16": 3481636352.0,
- "17": 3481636352.0,
- "18": 3481636352.0,
- "19": 3481636352.0,
- "20": 3481636352.0,
- "21": 3481636352.0,
- "22": 3481636352.0,
- "23": 3481636352.0,
- "24": 3482527744.0,
- "25": 3482527744.0,
- "26": 3482527744.0,
- "27": 3482527744.0,
- "28": 3482527744.0,
- "29": 3482527744.0,
- "30": 3482527744.0,
- "31": 3482527744.0,
- "32": 3482527744.0,
- "33": 3482527744.0,
- "34": 3482527744.0,
- "35": 3482527744.0,
- "36": 3482527744.0,
- "37": 3482527744.0,
- "38": 3482527744.0,
- "39": 3482527744.0,
- "40": 3482527744.0,
- "41": 3482527744.0,
- "42": 3482527744.0,
- "43": 3482527744.0,
- "44": 3495770112.0,
- "45": 3495770112.0,
- "46": 3495770112.0,
- "47": 3495770112.0,
- "48": 3495770112.0,
- "49": 3495770112.0,
- "50": 3495770112.0,
- "51": 3495770112.0,
- "52": 3495770112.0,
- "53": 3495770112.0,
- "54": 3495770112.0,
- "55": 3495770112.0,
- "56": 3495770112.0,
- "57": 3505988608.0,
- "58": 3505988608.0,
- "59": 3505988608.0,
- "60": 3505988608.0,
- "61": 3505988608.0,
- "62": 3505988608.0,
- "63": 3505988608.0,
- "64": 3505988608.0,
- "65": 3505988608.0,
- "66": 3505988608.0,
- "67": 3505988608.0,
- "68": 3505988608.0,
- "69": 3505988608.0,
- "70": 3505988608.0,
- "71": 3505988608.0,
- "72": 3505988608.0,
- "73": 3505988608.0,
- "74": 3505988608.0,
- "75": 3505988608.0,
- "76": 3505988608.0,
- "77": 3505988608.0,
- "78": 3505988608.0,
- "79": 3505988608.0,
- "80": 3505988608.0,
- "81": 3505988608.0,
- "82": 3505988608.0,
- "83": 3505988608.0,
- "84": 3505988608.0,
- "85": 3505988608.0,
- "86": 3505988608.0,
- "87": 3505988608.0,
- "88": 3505988608.0,
- "89": 3505988608.0,
- "90": 3505988608.0,
- "91": 3505988608.0,
- "92": 3505988608.0,
- "93": 3505988608.0,
- "94": 3505988608.0,
- "95": 3505988608.0,
- "96": 3505988608.0,
- "97": 3505988608.0,
- "98": 3505988608.0,
- "99": 3505988608.0,
- "100": 3505988608.0
+ "1": 3084961792.0,
+ "2": 3435149312.0,
+ "3": 3435149312.0,
+ "4": 3435149312.0,
+ "5": 3435149312.0,
+ "6": 3435149312.0,
+ "7": 3435149312.0,
+ "8": 3435149312.0,
+ "9": 3435149312.0,
+ "10": 3435149312.0,
+ "11": 3435149312.0,
+ "12": 3435149312.0,
+ "13": 3461126144.0,
+ "14": 3468252160.0,
+ "15": 3468252160.0,
+ "16": 3468252160.0,
+ "17": 3468252160.0,
+ "18": 3468252160.0,
+ "19": 3468252160.0,
+ "20": 3468252160.0,
+ "21": 3468252160.0,
+ "22": 3468252160.0,
+ "23": 3468252160.0,
+ "24": 3468252160.0,
+ "25": 3468252160.0,
+ "26": 3476090880.0,
+ "27": 3476090880.0,
+ "28": 3476090880.0,
+ "29": 3476090880.0,
+ "30": 3476090880.0,
+ "31": 3476090880.0,
+ "32": 3476090880.0,
+ "33": 3476090880.0,
+ "34": 3476090880.0,
+ "35": 3476090880.0,
+ "36": 3476090880.0,
+ "37": 3476090880.0,
+ "38": 3476090880.0,
+ "39": 3476090880.0,
+ "40": 3476090880.0,
+ "41": 3476090880.0,
+ "42": 3476090880.0,
+ "43": 3476090880.0,
+ "44": 3476090880.0,
+ "45": 3476090880.0,
+ "46": 3476090880.0,
+ "47": 3476090880.0,
+ "48": 3476090880.0,
+ "49": 3476090880.0,
+ "50": 3476090880.0,
+ "51": 3476090880.0,
+ "52": 3476090880.0,
+ "53": 3476090880.0,
+ "54": 3476090880.0,
+ "55": 3476090880.0,
+ "56": 3476090880.0,
+ "57": 3476090880.0,
+ "58": 3476090880.0,
+ "59": 3476090880.0,
+ "60": 3476090880.0,
+ "61": 3476090880.0,
+ "62": 3476090880.0,
+ "63": 3476090880.0,
+ "64": 3476090880.0,
+ "65": 3476090880.0,
+ "66": 3476090880.0,
+ "67": 3476090880.0,
+ "68": 3476090880.0,
+ "69": 3476090880.0,
+ "70": 3476090880.0,
+ "71": 3476090880.0,
+ "72": 3476090880.0,
+ "73": 3476090880.0,
+ "74": 3476090880.0,
+ "75": 3476090880.0,
+ "76": 3476090880.0,
+ "77": 3476090880.0,
+ "78": 3476090880.0,
+ "79": 3476090880.0,
+ "80": 3476090880.0,
+ "81": 3476090880.0,
+ "82": 3476090880.0,
+ "83": 3476090880.0,
+ "84": 3476090880.0,
+ "85": 3476090880.0,
+ "86": 3476090880.0,
+ "87": 3476090880.0,
+ "88": 3476090880.0,
+ "89": 3476090880.0,
+ "90": 3476090880.0,
+ "91": 3476090880.0,
+ "92": 3476090880.0,
+ "93": 3476090880.0,
+ "94": 3476090880.0,
+ "95": 3476090880.0,
+ "96": 3476090880.0,
+ "97": 3476090880.0,
+ "98": 3476090880.0,
+ "99": 3476090880.0,
+ "100": 3476090880.0
}
},
"iteration-time": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 4.71174,
- "3": 0.47502,
- "4": 0.44931,
- "5": 0.44277,
- "6": 0.44844,
- "7": 0.45785,
- "8": 0.44209,
- "9": 0.43757,
- "10": 0.42772,
- "11": 0.44315,
- "12": 0.42725,
- "13": 0.42666,
- "14": 0.41928,
- "15": 0.42831,
- "16": 0.42799,
- "17": 0.42051,
- "18": 0.41469,
- "19": 0.41876,
- "20": 0.41842,
- "21": 0.43095,
- "22": 0.41003,
- "23": 0.41066,
- "24": 0.41091,
- "25": 0.40849,
- "26": 0.4098,
- "27": 0.41447,
- "28": 0.4098,
- "29": 0.40395,
- "30": 0.41016,
- "31": 0.41347,
- "32": 0.40916,
- "33": 0.41299,
- "34": 0.40596,
- "35": 0.40696,
- "36": 0.40868,
- "37": 0.40718,
- "38": 0.40736,
- "39": 0.40604,
- "40": 0.40127,
- "41": 0.4,
- "42": 0.40197,
- "43": 0.40902,
- "44": 0.40712,
- "45": 0.4098,
- "46": 0.40168,
- "47": 0.40487,
- "48": 0.40622,
- "49": 0.4089,
- "50": 0.40406,
- "51": 0.41118,
- "52": 0.40412,
- "53": 0.40027,
- "54": 0.40192,
- "55": 0.39782,
- "56": 0.39731,
- "57": 0.39836,
- "58": 0.40128,
- "59": 0.39958,
- "60": 0.39863,
- "61": 0.78712,
- "62": 0.39887,
- "63": 0.39967,
- "64": 0.40024,
- "65": 0.39891,
- "66": 0.40058,
- "67": 0.80982,
- "68": 0.39889,
- "69": 0.39895,
- "70": 0.40201,
- "71": 0.39871,
- "72": 0.39819,
- "73": 0.40638,
- "74": 0.40241,
- "75": 0.39867,
- "76": 0.40192,
- "77": 0.4032,
- "78": 0.39871,
- "79": 0.96252,
- "80": 0.39811,
- "81": 0.40176,
- "82": 0.39856,
- "83": 0.40217,
- "84": 0.3966,
- "85": 0.40212,
- "86": 0.40144,
- "87": 0.39779,
- "88": 0.3989,
- "89": 0.39982,
- "90": 0.40291,
- "91": 0.40052,
- "92": 0.39772,
- "93": 0.40147,
- "94": 0.40072,
- "95": 0.40007,
- "96": 0.40232,
- "97": 0.40777,
- "98": 0.4002,
- "99": 0.39995,
- "100": 0.39879
+ "2": 7.68365,
+ "3": 0.50311,
+ "4": 0.50121,
+ "5": 0.47301,
+ "6": 0.47548,
+ "7": 0.46747,
+ "8": 0.46103,
+ "9": 0.45984,
+ "10": 0.4589,
+ "11": 0.45806,
+ "12": 0.45531,
+ "13": 0.45587,
+ "14": 0.45344,
+ "15": 0.45584,
+ "16": 0.45844,
+ "17": 0.45245,
+ "18": 0.45157,
+ "19": 0.44557,
+ "20": 0.45325,
+ "21": 0.44533,
+ "22": 0.46013,
+ "23": 0.4395,
+ "24": 0.45705,
+ "25": 0.44719,
+ "26": 0.44382,
+ "27": 0.44184,
+ "28": 0.4486,
+ "29": 0.45059,
+ "30": 0.44613,
+ "31": 0.44946,
+ "32": 0.44174,
+ "33": 0.44725,
+ "34": 0.44875,
+ "35": 0.4463,
+ "36": 0.4415,
+ "37": 0.44526,
+ "38": 0.44277,
+ "39": 0.44987,
+ "40": 0.44783,
+ "41": 0.4436,
+ "42": 0.44693,
+ "43": 0.44926,
+ "44": 0.44584,
+ "45": 0.4484,
+ "46": 0.4509,
+ "47": 0.44027,
+ "48": 0.4512,
+ "49": 0.44852,
+ "50": 0.44597,
+ "51": 0.62527,
+ "52": 0.44237,
+ "53": 0.44022,
+ "54": 0.4416,
+ "55": 0.44591,
+ "56": 0.45406,
+ "57": 0.43865,
+ "58": 0.44163,
+ "59": 0.44026,
+ "60": 0.44156,
+ "61": 0.44057,
+ "62": 0.44793,
+ "63": 0.4396,
+ "64": 0.44658,
+ "65": 0.4415,
+ "66": 0.44225,
+ "67": 0.44189,
+ "68": 0.45217,
+ "69": 0.44544,
+ "70": 0.4435,
+ "71": 0.44643,
+ "72": 0.44423,
+ "73": 0.44399,
+ "74": 0.44372,
+ "75": 0.44489,
+ "76": 0.44178,
+ "77": 0.44514,
+ "78": 0.45071,
+ "79": 0.44323,
+ "80": 0.44497,
+ "81": 0.44432,
+ "82": 0.44246,
+ "83": 0.44567,
+ "84": 0.44533,
+ "85": 0.44698,
+ "86": 0.44072,
+ "87": 0.45027,
+ "88": 0.44587,
+ "89": 0.44519,
+ "90": 0.44239,
+ "91": 0.44192,
+ "92": 0.44151,
+ "93": 0.44687,
+ "94": 0.44048,
+ "95": 0.44589,
+ "96": 0.44807,
+ "97": 0.44298,
+ "98": 0.45472,
+ "99": 0.44178,
+ "100": 0.44633
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon/golden_values_dev_dgx_h100.json
index 7d62923f634..f969b543749 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.81103,
- "2": 10.83065,
- "3": 10.82041,
- "4": 10.81308,
- "5": 10.84384,
- "6": 10.84719,
- "7": 10.85346,
- "8": 10.83656,
- "9": 10.84673,
- "10": 10.78252,
- "11": 10.85208,
- "12": 10.86326,
- "13": 10.85438,
- "14": 10.88369,
- "15": 10.87797,
- "16": 10.84675,
- "17": 10.83091,
- "18": 10.86618,
- "19": 10.84893,
- "20": 10.84566,
- "21": 10.8476,
- "22": 10.79629,
- "23": 10.88263,
- "24": 10.83271,
- "25": 10.82477,
- "26": 10.84285,
- "27": 10.85338,
- "28": 10.87687,
- "29": 10.86419,
- "30": 10.81306,
- "31": 10.78638,
- "32": 10.85513,
- "33": 10.85601,
- "34": 10.8492,
- "35": 10.83702,
- "36": 10.80421,
- "37": 10.83785,
- "38": 10.80484,
- "39": 10.84147,
- "40": 10.80332,
- "41": 10.83988,
- "42": 10.84406,
- "43": 10.81051,
- "44": 10.8022,
- "45": 10.78682,
- "46": 10.80806,
- "47": 10.81757,
- "48": 10.80298,
- "49": 10.78142,
- "50": 10.8023,
- "51": 10.82205,
- "52": 10.80341,
- "53": 10.83273,
- "54": 10.81558,
- "55": 10.82551,
- "56": 10.77761,
- "57": 10.7527,
- "58": 10.80801,
- "59": 10.79071,
- "60": 10.73971,
- "61": 10.80018,
- "62": 10.81299,
- "63": 10.72069,
- "64": 10.78573,
- "65": 10.69001,
- "66": 10.76067,
- "67": 10.73433,
- "68": 10.80225,
- "69": 10.7835,
- "70": 10.77632,
- "71": 10.76604,
- "72": 10.736,
- "73": 10.72965,
- "74": 10.62244,
- "75": 10.69059,
- "76": 10.65429,
- "77": 10.82179,
- "78": 10.76341,
- "79": 10.70461,
- "80": 10.69433,
- "81": 10.72473,
- "82": 10.74232,
- "83": 10.66784,
- "84": 10.69896,
- "85": 10.7144,
- "86": 10.63886,
- "87": 10.71783,
- "88": 10.73541,
- "89": 10.7139,
- "90": 10.74667,
- "91": 10.64906,
- "92": 10.64667,
- "93": 10.60204,
- "94": 10.53296,
- "95": 10.66128,
- "96": 10.67208,
- "97": 10.61439,
- "98": 10.68466,
- "99": 10.52017,
- "100": 10.61535
+ "1": 10.90136,
+ "2": 10.89626,
+ "3": 10.90656,
+ "4": 10.90569,
+ "5": 10.91343,
+ "6": 10.89724,
+ "7": 10.8982,
+ "8": 10.91346,
+ "9": 10.89599,
+ "10": 10.90377,
+ "11": 10.90041,
+ "12": 10.91817,
+ "13": 10.88463,
+ "14": 10.87488,
+ "15": 10.9097,
+ "16": 10.90234,
+ "17": 10.8859,
+ "18": 10.89391,
+ "19": 10.89492,
+ "20": 10.89095,
+ "21": 10.88706,
+ "22": 10.90303,
+ "23": 10.91221,
+ "24": 10.886,
+ "25": 10.90119,
+ "26": 10.88859,
+ "27": 10.89701,
+ "28": 10.88189,
+ "29": 10.89019,
+ "30": 10.91047,
+ "31": 10.89241,
+ "32": 10.88829,
+ "33": 10.89714,
+ "34": 10.8736,
+ "35": 10.89675,
+ "36": 10.90726,
+ "37": 10.8705,
+ "38": 10.87905,
+ "39": 10.88545,
+ "40": 10.8927,
+ "41": 10.88386,
+ "42": 10.89515,
+ "43": 10.87983,
+ "44": 10.88534,
+ "45": 10.88311,
+ "46": 10.88569,
+ "47": 10.88551,
+ "48": 10.86374,
+ "49": 10.87608,
+ "50": 10.88307,
+ "51": 10.89316,
+ "52": 10.87091,
+ "53": 10.85927,
+ "54": 10.871,
+ "55": 10.86793,
+ "56": 10.87348,
+ "57": 10.84459,
+ "58": 10.85664,
+ "59": 10.84675,
+ "60": 10.8432,
+ "61": 10.86072,
+ "62": 10.85715,
+ "63": 10.86046,
+ "64": 10.83889,
+ "65": 10.8275,
+ "66": 10.84523,
+ "67": 10.82869,
+ "68": 10.82992,
+ "69": 10.81837,
+ "70": 10.82849,
+ "71": 10.82699,
+ "72": 10.80762,
+ "73": 10.80724,
+ "74": 10.80555,
+ "75": 10.81484,
+ "76": 10.81215,
+ "77": 10.80702,
+ "78": 10.79177,
+ "79": 10.80247,
+ "80": 10.78858,
+ "81": 10.79555,
+ "82": 10.79562,
+ "83": 10.78549,
+ "84": 10.75723,
+ "85": 10.76265,
+ "86": 10.77648,
+ "87": 10.79604,
+ "88": 10.775,
+ "89": 10.77563,
+ "90": 10.76454,
+ "91": 10.7402,
+ "92": 10.76035,
+ "93": 10.7468,
+ "94": 10.73441,
+ "95": 10.75291,
+ "96": 10.72327,
+ "97": 10.71558,
+ "98": 10.72628,
+ "99": 10.74627,
+ "100": 10.69573
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1248.0,
- "2": 1334.0,
- "3": 1279.0,
- "4": 1251.0,
- "5": 1382.0,
- "6": 1478.0,
- "7": 1205.0,
- "8": 1414.0,
- "9": 1410.0,
- "10": 1302.0,
- "11": 1305.0,
- "12": 1282.0,
- "13": 1254.0,
- "14": 1432.0,
- "15": 1176.0,
- "16": 1222.0,
- "17": 1256.0,
- "18": 1355.0,
- "19": 1265.0,
- "20": 1285.0,
- "21": 1258.0,
- "22": 1186.0,
- "23": 1312.0,
- "24": 1329.0,
- "25": 1270.0,
- "26": 1206.0,
- "27": 1432.0,
- "28": 1433.0,
- "29": 1297.0,
- "30": 1191.0,
- "31": 1166.0,
- "32": 1273.0,
- "33": 1273.0,
- "34": 1165.0,
- "35": 1200.0,
- "36": 1216.0,
- "37": 1241.0,
- "38": 1343.0,
- "39": 1544.0,
- "40": 1200.0,
- "41": 1350.0,
- "42": 1218.0,
- "43": 1213.0,
- "44": 1223.0,
- "45": 1179.0,
- "46": 1211.0,
- "47": 1353.0,
- "48": 1180.0,
- "49": 1180.0,
- "50": 1182.0,
- "51": 1221.0,
- "52": 1192.0,
- "53": 1460.0,
- "54": 1267.0,
- "55": 1209.0,
- "56": 1312.0,
- "57": 1287.0,
- "58": 1291.0,
- "59": 1292.0,
- "60": 1229.0,
- "61": 1153.0,
- "62": 1228.0,
- "63": 1200.0,
- "64": 1307.0,
- "65": 1183.0,
- "66": 1202.0,
- "67": 1163.0,
- "68": 1246.0,
- "69": 1316.0,
- "70": 1336.0,
- "71": 1209.0,
- "72": 1196.0,
- "73": 1115.0,
- "74": 1121.0,
- "75": 1276.0,
- "76": 1299.0,
- "77": 1349.0,
- "78": 1322.0,
- "79": 1092.0,
- "80": 1223.0,
- "81": 1098.0,
- "82": 1237.0,
- "83": 1317.0,
- "84": 1179.0,
- "85": 1286.0,
- "86": 1152.0,
- "87": 1188.0,
- "88": 1294.0,
- "89": 1227.0,
- "90": 1392.0,
- "91": 1150.0,
- "92": 1268.0,
- "93": 1105.0,
- "94": 1010.0,
- "95": 1265.0,
- "96": 1276.0,
- "97": 1181.0,
- "98": 1194.0,
- "99": 1221.0,
- "100": 1285.0
+ "1": 1166.0,
+ "2": 1181.0,
+ "3": 1349.0,
+ "4": 1286.0,
+ "5": 1216.0,
+ "6": 1264.0,
+ "7": 1284.0,
+ "8": 1203.0,
+ "9": 1246.0,
+ "10": 1153.0,
+ "11": 1210.0,
+ "12": 1191.0,
+ "13": 1313.0,
+ "14": 1273.0,
+ "15": 1234.0,
+ "16": 1223.0,
+ "17": 1200.0,
+ "18": 1098.0,
+ "19": 1216.0,
+ "20": 1141.0,
+ "21": 1209.0,
+ "22": 1235.0,
+ "23": 1292.0,
+ "24": 1142.0,
+ "25": 1255.0,
+ "26": 1172.0,
+ "27": 1367.0,
+ "28": 1129.0,
+ "29": 1181.0,
+ "30": 1231.0,
+ "31": 1206.0,
+ "32": 1247.0,
+ "33": 1126.0,
+ "34": 1184.0,
+ "35": 1275.0,
+ "36": 1206.0,
+ "37": 1202.0,
+ "38": 1291.0,
+ "39": 1324.0,
+ "40": 1271.0,
+ "41": 1355.0,
+ "42": 1206.0,
+ "43": 1243.0,
+ "44": 1269.0,
+ "45": 1151.0,
+ "46": 1381.0,
+ "47": 1218.0,
+ "48": 1146.0,
+ "49": 1381.0,
+ "50": 1297.0,
+ "51": 1281.0,
+ "52": 1203.0,
+ "53": 1237.0,
+ "54": 1160.0,
+ "55": 1166.0,
+ "56": 1182.0,
+ "57": 1210.0,
+ "58": 1241.0,
+ "59": 1100.0,
+ "60": 1269.0,
+ "61": 1248.0,
+ "62": 1204.0,
+ "63": 1281.0,
+ "64": 1333.0,
+ "65": 1182.0,
+ "66": 1205.0,
+ "67": 1204.0,
+ "68": 1253.0,
+ "69": 1153.0,
+ "70": 1238.0,
+ "71": 1193.0,
+ "72": 1281.0,
+ "73": 1218.0,
+ "74": 1341.0,
+ "75": 1334.0,
+ "76": 1204.0,
+ "77": 1210.0,
+ "78": 1207.0,
+ "79": 1286.0,
+ "80": 1194.0,
+ "81": 1341.0,
+ "82": 1190.0,
+ "83": 1233.0,
+ "84": 1181.0,
+ "85": 1169.0,
+ "86": 1251.0,
+ "87": 1079.0,
+ "88": 1218.0,
+ "89": 1132.0,
+ "90": 1340.0,
+ "91": 1240.0,
+ "92": 1206.0,
+ "93": 1336.0,
+ "94": 1303.0,
+ "95": 1134.0,
+ "96": 1190.0,
+ "97": 1180.0,
+ "98": 1385.0,
+ "99": 1161.0,
+ "100": 1265.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1095884800.0,
- "2": 1095854592.0,
- "3": 1095902720.0,
- "4": 1095882752.0,
- "5": 1095905792.0,
- "6": 1095847424.0,
- "7": 1095868928.0,
- "8": 1095876608.0,
- "9": 1095891968.0,
- "10": 1095904768.0,
- "11": 1095880704.0,
- "12": 1095849472.0,
- "13": 1095902720.0,
- "14": 1095814144.0,
- "15": 1095857152.0,
- "16": 1095790592.0,
- "17": 1095910400.0,
- "18": 1095884288.0,
- "19": 1095892480.0,
- "20": 1095812096.0,
- "21": 1095834112.0,
- "22": 1095906816.0,
- "23": 1095881728.0,
- "24": 1095824896.0,
- "25": 1095955968.0,
- "26": 1095862272.0,
- "27": 1095919616.0,
- "28": 1095846400.0,
- "29": 1095878656.0,
- "30": 1095843840.0,
- "31": 1095906816.0,
- "32": 1095840768.0,
- "33": 1095851008.0,
- "34": 1095815680.0,
- "35": 1095865856.0,
- "36": 1095880192.0,
- "37": 1095838720.0,
- "38": 1095921664.0,
- "39": 1095930368.0,
- "40": 1095818240.0,
- "41": 1095847424.0,
- "42": 1095864320.0,
- "43": 1095878144.0,
- "44": 1095982080.0,
- "45": 1095855104.0,
- "46": 1095869952.0,
- "47": 1095891968.0,
- "48": 1095878144.0,
- "49": 1095843840.0,
- "50": 1095862272.0,
- "51": 1095934464.0,
- "52": 1095880192.0,
- "53": 1095940608.0,
- "54": 1095887872.0,
- "55": 1095877632.0,
- "56": 1095868416.0,
- "57": 1095913472.0,
- "58": 1095910912.0,
- "59": 1095898112.0,
- "60": 1095865344.0,
- "61": 1095864320.0,
- "62": 1095858176.0,
- "63": 1095872000.0,
- "64": 1095862272.0,
- "65": 1095868928.0,
- "66": 1095877120.0,
- "67": 1095863808.0,
- "68": 1095873024.0,
- "69": 1095859712.0,
- "70": 1095904768.0,
- "71": 1095876608.0,
- "72": 1095805952.0,
- "73": 1095908352.0,
- "74": 1095918592.0,
- "75": 1095884288.0,
- "76": 1095903744.0,
- "77": 1095857664.0,
- "78": 1095914496.0,
- "79": 1095888896.0,
- "80": 1095839232.0,
- "81": 1095875584.0,
- "82": 1095825408.0,
- "83": 1095925248.0,
- "84": 1095904256.0,
- "85": 1095870976.0,
- "86": 1095870976.0,
- "87": 1095893504.0,
- "88": 1095882240.0,
- "89": 1095869952.0,
- "90": 1095860224.0,
- "91": 1095916032.0,
- "92": 1095900672.0,
- "93": 1095878144.0,
- "94": 1095884800.0,
- "95": 1095889920.0,
- "96": 1095886848.0,
- "97": 1095896576.0,
- "98": 1095873024.0,
- "99": 1095887872.0,
- "100": 1095927808.0
+ "1": 1096817664.0,
+ "2": 1096807936.0,
+ "3": 1096783872.0,
+ "4": 1096763392.0,
+ "5": 1096815104.0,
+ "6": 1096801280.0,
+ "7": 1096723456.0,
+ "8": 1096759808.0,
+ "9": 1096762368.0,
+ "10": 1096794112.0,
+ "11": 1096772096.0,
+ "12": 1096779264.0,
+ "13": 1096783360.0,
+ "14": 1096784896.0,
+ "15": 1096759808.0,
+ "16": 1096746496.0,
+ "17": 1096755200.0,
+ "18": 1096757760.0,
+ "19": 1096786944.0,
+ "20": 1096751104.0,
+ "21": 1096804864.0,
+ "22": 1096833024.0,
+ "23": 1096804352.0,
+ "24": 1096716800.0,
+ "25": 1096798208.0,
+ "26": 1096830976.0,
+ "27": 1096783872.0,
+ "28": 1096738816.0,
+ "29": 1096812544.0,
+ "30": 1096743936.0,
+ "31": 1096752128.0,
+ "32": 1096758784.0,
+ "33": 1096762880.0,
+ "34": 1096742912.0,
+ "35": 1096727040.0,
+ "36": 1096793088.0,
+ "37": 1096743936.0,
+ "38": 1096802816.0,
+ "39": 1096810496.0,
+ "40": 1096754176.0,
+ "41": 1096775680.0,
+ "42": 1096719872.0,
+ "43": 1096785408.0,
+ "44": 1096772096.0,
+ "45": 1096743424.0,
+ "46": 1096809472.0,
+ "47": 1096766976.0,
+ "48": 1096796672.0,
+ "49": 1096756224.0,
+ "50": 1096801792.0,
+ "51": 1096783872.0,
+ "52": 1096756224.0,
+ "53": 1096709632.0,
+ "54": 1096802304.0,
+ "55": 1096794112.0,
+ "56": 1096792576.0,
+ "57": 1096853504.0,
+ "58": 1096789504.0,
+ "59": 1096796672.0,
+ "60": 1096773632.0,
+ "61": 1096743424.0,
+ "62": 1096734208.0,
+ "63": 1096743936.0,
+ "64": 1096772096.0,
+ "65": 1096777216.0,
+ "66": 1096791552.0,
+ "67": 1096774656.0,
+ "68": 1096774656.0,
+ "69": 1096738816.0,
+ "70": 1096822784.0,
+ "71": 1096800768.0,
+ "72": 1096801792.0,
+ "73": 1096730624.0,
+ "74": 1096766976.0,
+ "75": 1096815616.0,
+ "76": 1096773120.0,
+ "77": 1096817152.0,
+ "78": 1096758784.0,
+ "79": 1096826368.0,
+ "80": 1096749056.0,
+ "81": 1096802816.0,
+ "82": 1096747520.0,
+ "83": 1096810496.0,
+ "84": 1096781312.0,
+ "85": 1096786944.0,
+ "86": 1096772608.0,
+ "87": 1096785408.0,
+ "88": 1096766976.0,
+ "89": 1096758272.0,
+ "90": 1096780800.0,
+ "91": 1096766464.0,
+ "92": 1096765952.0,
+ "93": 1096794624.0,
+ "94": 1096768512.0,
+ "95": 1096790528.0,
+ "96": 1096784896.0,
+ "97": 1096750592.0,
+ "98": 1096790016.0,
+ "99": 1096806400.0,
+ "100": 1096835584.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 3260419584.0,
- "2": 3582873600.0,
- "3": 3615975424.0,
- "4": 3615975424.0,
- "5": 3616532480.0,
- "6": 3616532480.0,
- "7": 3616532480.0,
- "8": 3616532480.0,
- "9": 3616532480.0,
- "10": 3618800640.0,
- "11": 3618800640.0,
- "12": 3618800640.0,
- "13": 3618800640.0,
- "14": 3618800640.0,
- "15": 3618800640.0,
- "16": 3618800640.0,
- "17": 3618800640.0,
- "18": 3618800640.0,
- "19": 3618800640.0,
- "20": 3618800640.0,
- "21": 3618800640.0,
- "22": 3618800640.0,
- "23": 3618800640.0,
- "24": 3618800640.0,
- "25": 3648277504.0,
- "26": 3648277504.0,
- "27": 3648277504.0,
- "28": 3648277504.0,
- "29": 3648277504.0,
- "30": 3648277504.0,
- "31": 3648277504.0,
- "32": 3648277504.0,
- "33": 3648277504.0,
- "34": 3648277504.0,
- "35": 3648277504.0,
- "36": 3648277504.0,
- "37": 3648277504.0,
- "38": 3648277504.0,
- "39": 3648277504.0,
- "40": 3648277504.0,
- "41": 3648277504.0,
- "42": 3648277504.0,
- "43": 3648277504.0,
- "44": 3667798528.0,
- "45": 3667798528.0,
- "46": 3667798528.0,
- "47": 3667798528.0,
- "48": 3667798528.0,
- "49": 3667798528.0,
- "50": 3667798528.0,
- "51": 3667798528.0,
- "52": 3667798528.0,
- "53": 3667798528.0,
- "54": 3667798528.0,
- "55": 3667798528.0,
- "56": 3667798528.0,
- "57": 3667798528.0,
- "58": 3667798528.0,
- "59": 3667798528.0,
- "60": 3667798528.0,
- "61": 3667798528.0,
- "62": 3667798528.0,
- "63": 3667798528.0,
- "64": 3667798528.0,
- "65": 3667798528.0,
- "66": 3667798528.0,
- "67": 3667798528.0,
- "68": 3667798528.0,
- "69": 3667798528.0,
- "70": 3667798528.0,
- "71": 3667798528.0,
- "72": 3667798528.0,
- "73": 3667798528.0,
- "74": 3667798528.0,
- "75": 3667798528.0,
- "76": 3667798528.0,
- "77": 3667798528.0,
- "78": 3667798528.0,
- "79": 3667798528.0,
- "80": 3667798528.0,
- "81": 3667798528.0,
- "82": 3667798528.0,
- "83": 3667798528.0,
- "84": 3667798528.0,
- "85": 3667798528.0,
- "86": 3667798528.0,
- "87": 3667798528.0,
- "88": 3667798528.0,
- "89": 3667798528.0,
- "90": 3667798528.0,
- "91": 3667798528.0,
- "92": 3667798528.0,
- "93": 3667798528.0,
- "94": 3667798528.0,
- "95": 3667798528.0,
- "96": 3667798528.0,
- "97": 3667798528.0,
- "98": 3667798528.0,
- "99": 3667798528.0,
- "100": 3667798528.0
+ "1": 3258557952.0,
+ "2": 3590636544.0,
+ "3": 3590636544.0,
+ "4": 3590636544.0,
+ "5": 3595399168.0,
+ "6": 3595399168.0,
+ "7": 3595399168.0,
+ "8": 3595399168.0,
+ "9": 3595399168.0,
+ "10": 3595399168.0,
+ "11": 3595399168.0,
+ "12": 3595399168.0,
+ "13": 3595399168.0,
+ "14": 3595399168.0,
+ "15": 3595399168.0,
+ "16": 3595399168.0,
+ "17": 3595399168.0,
+ "18": 3595399168.0,
+ "19": 3595399168.0,
+ "20": 3595399168.0,
+ "21": 3595399168.0,
+ "22": 3610098176.0,
+ "23": 3610098176.0,
+ "24": 3610098176.0,
+ "25": 3610098176.0,
+ "26": 3610098176.0,
+ "27": 3610098176.0,
+ "28": 3610098176.0,
+ "29": 3610098176.0,
+ "30": 3610098176.0,
+ "31": 3610098176.0,
+ "32": 3610098176.0,
+ "33": 3610098176.0,
+ "34": 3610098176.0,
+ "35": 3610098176.0,
+ "36": 3610098176.0,
+ "37": 3610098176.0,
+ "38": 3610098176.0,
+ "39": 3610098176.0,
+ "40": 3610098176.0,
+ "41": 3610098176.0,
+ "42": 3610098176.0,
+ "43": 3610098176.0,
+ "44": 3610098176.0,
+ "45": 3610098176.0,
+ "46": 3610098176.0,
+ "47": 3610098176.0,
+ "48": 3610098176.0,
+ "49": 3610098176.0,
+ "50": 3610098176.0,
+ "51": 3610098176.0,
+ "52": 3610098176.0,
+ "53": 3610098176.0,
+ "54": 3610098176.0,
+ "55": 3610098176.0,
+ "56": 3610098176.0,
+ "57": 3626856960.0,
+ "58": 3626856960.0,
+ "59": 3626856960.0,
+ "60": 3626856960.0,
+ "61": 3626856960.0,
+ "62": 3626856960.0,
+ "63": 3626856960.0,
+ "64": 3626856960.0,
+ "65": 3626856960.0,
+ "66": 3626856960.0,
+ "67": 3626856960.0,
+ "68": 3626856960.0,
+ "69": 3626856960.0,
+ "70": 3626856960.0,
+ "71": 3626856960.0,
+ "72": 3626856960.0,
+ "73": 3626856960.0,
+ "74": 3626856960.0,
+ "75": 3626856960.0,
+ "76": 3626856960.0,
+ "77": 3626856960.0,
+ "78": 3626856960.0,
+ "79": 3626856960.0,
+ "80": 3626856960.0,
+ "81": 3626856960.0,
+ "82": 3626856960.0,
+ "83": 3626856960.0,
+ "84": 3626856960.0,
+ "85": 3626856960.0,
+ "86": 3626856960.0,
+ "87": 3626856960.0,
+ "88": 3626856960.0,
+ "89": 3626856960.0,
+ "90": 3626856960.0,
+ "91": 3626856960.0,
+ "92": 3626856960.0,
+ "93": 3626856960.0,
+ "94": 3626856960.0,
+ "95": 3626856960.0,
+ "96": 3626856960.0,
+ "97": 3626856960.0,
+ "98": 3626856960.0,
+ "99": 3626856960.0,
+ "100": 3626856960.0
}
},
"iteration-time": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 6.73766,
- "3": 0.31869,
- "4": 0.3125,
- "5": 0.31279,
- "6": 0.29974,
- "7": 0.30628,
- "8": 0.29637,
- "9": 0.29751,
- "10": 0.28123,
- "11": 0.3055,
- "12": 0.28861,
- "13": 0.27778,
- "14": 0.28796,
- "15": 0.28678,
- "16": 0.27533,
- "17": 0.27979,
- "18": 1.87923,
- "19": 0.28574,
- "20": 0.28215,
- "21": 0.2771,
- "22": 0.27101,
- "23": 0.27311,
- "24": 1.50235,
- "25": 0.27537,
- "26": 1.04897,
- "27": 0.26783,
- "28": 0.69868,
- "29": 0.27953,
- "30": 1.54699,
- "31": 0.27899,
- "32": 0.28165,
- "33": 0.28792,
- "34": 0.27678,
- "35": 1.25378,
- "36": 0.88573,
- "37": 0.26832,
- "38": 0.26501,
- "39": 0.28399,
- "40": 0.96629,
- "41": 0.26938,
- "42": 0.31209,
- "43": 0.27968,
- "44": 0.65987,
- "45": 0.51088,
- "46": 1.37707,
- "47": 0.26575,
- "48": 0.92193,
- "49": 0.26081,
- "50": 0.27031,
- "51": 0.31353,
- "52": 0.27257,
- "53": 0.27323,
- "54": 0.27148,
- "55": 0.27248,
- "56": 0.7475,
- "57": 0.26706,
- "58": 0.28367,
- "59": 0.27716,
- "60": 1.12441,
- "61": 0.26587,
- "62": 0.68635,
- "63": 0.28123,
- "64": 0.98333,
- "65": 0.27408,
- "66": 1.22087,
- "67": 0.26407,
- "68": 0.95198,
- "69": 0.29272,
- "70": 0.52799,
- "71": 0.92323,
- "72": 0.25931,
- "73": 0.26616,
- "74": 0.28128,
- "75": 0.28947,
- "76": 0.27481,
- "77": 0.67217,
- "78": 0.28612,
- "79": 0.85039,
- "80": 0.2721,
- "81": 0.5328,
- "82": 0.57505,
- "83": 0.79918,
- "84": 0.28096,
- "85": 0.27744,
- "86": 0.273,
- "87": 0.33552,
- "88": 0.48699,
- "89": 0.28552,
- "90": 0.50386,
- "91": 0.27372,
- "92": 0.64636,
- "93": 0.26742,
- "94": 0.2649,
- "95": 0.49366,
- "96": 0.36845,
- "97": 0.29731,
- "98": 0.53051,
- "99": 0.26212,
- "100": 0.75087
+ "2": 7.17968,
+ "3": 0.40991,
+ "4": 0.38138,
+ "5": 0.37493,
+ "6": 0.36345,
+ "7": 0.37361,
+ "8": 0.37817,
+ "9": 0.37022,
+ "10": 0.36826,
+ "11": 0.38287,
+ "12": 0.35653,
+ "13": 0.35671,
+ "14": 0.36582,
+ "15": 0.36031,
+ "16": 0.36186,
+ "17": 0.36814,
+ "18": 0.36332,
+ "19": 0.35916,
+ "20": 0.3607,
+ "21": 0.3521,
+ "22": 0.361,
+ "23": 0.35545,
+ "24": 0.35155,
+ "25": 0.36572,
+ "26": 0.35368,
+ "27": 0.35075,
+ "28": 0.35568,
+ "29": 0.35186,
+ "30": 0.35402,
+ "31": 0.3513,
+ "32": 0.35424,
+ "33": 0.35891,
+ "34": 0.35477,
+ "35": 0.35845,
+ "36": 0.35291,
+ "37": 0.34892,
+ "38": 0.35121,
+ "39": 0.34548,
+ "40": 0.35302,
+ "41": 0.3477,
+ "42": 0.34969,
+ "43": 0.34977,
+ "44": 0.34495,
+ "45": 0.34752,
+ "46": 0.34445,
+ "47": 0.35615,
+ "48": 0.34549,
+ "49": 0.34997,
+ "50": 0.34623,
+ "51": 0.40812,
+ "52": 0.3508,
+ "53": 0.3514,
+ "54": 0.34796,
+ "55": 0.34845,
+ "56": 0.34796,
+ "57": 0.35013,
+ "58": 0.35407,
+ "59": 0.35061,
+ "60": 0.34534,
+ "61": 0.3563,
+ "62": 0.36043,
+ "63": 0.34748,
+ "64": 0.34922,
+ "65": 0.35369,
+ "66": 0.34981,
+ "67": 0.35187,
+ "68": 0.34605,
+ "69": 0.35527,
+ "70": 0.35126,
+ "71": 0.34843,
+ "72": 0.3524,
+ "73": 0.35005,
+ "74": 0.34515,
+ "75": 0.34524,
+ "76": 0.34871,
+ "77": 0.35762,
+ "78": 0.35088,
+ "79": 0.34689,
+ "80": 0.35296,
+ "81": 0.35338,
+ "82": 0.34935,
+ "83": 0.35314,
+ "84": 0.34855,
+ "85": 0.34812,
+ "86": 0.35659,
+ "87": 0.35911,
+ "88": 0.35548,
+ "89": 0.34997,
+ "90": 0.34987,
+ "91": 0.35492,
+ "92": 0.34795,
+ "93": 0.34591,
+ "94": 0.35798,
+ "95": 0.3514,
+ "96": 0.35409,
+ "97": 0.35835,
+ "98": 0.35841,
+ "99": 0.35403,
+ "100": 0.3505
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon/model_config.yaml
index 5c395caed56..6b14d162d9f 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -65,4 +65,6 @@ MODEL_ARGS:
--muon-extra-scale-factor: 0.2
--muon-scale-mode: spectral
--check-weight-hash-across-dp-replicas-interval: 1
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_fine_grained_offloading/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_fine_grained_offloading/golden_values_dev_dgx_h100.json
index d5ced620365..6b0e31e5923 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_fine_grained_offloading/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_fine_grained_offloading/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 11.06693,
- "2": 11.0602,
- "3": 10.16141,
- "4": 10.11145,
- "5": 10.47957,
- "6": 10.21751,
- "7": 10.56153,
- "8": 12.79501,
- "9": 12.96949,
- "10": 13.32223,
- "11": 11.63359,
- "12": 11.4938,
- "13": 12.46292,
- "14": 12.13415,
- "15": 11.90295,
- "16": 12.01307,
- "17": 12.17443,
- "18": 12.64978,
- "19": 11.81295,
- "20": 12.18673,
- "21": 11.24306,
- "22": 11.54156,
- "23": 10.98412,
- "24": 11.01925,
- "25": 10.73001,
- "26": 10.72806,
- "27": 10.79039,
- "28": 10.714,
- "29": 10.73974,
- "30": 10.75246,
- "31": 10.68874,
- "32": 10.65791,
- "33": 10.81137,
- "34": 10.79058,
- "35": 10.75368,
- "36": 10.64393,
- "37": 10.87492,
- "38": 10.90591,
- "39": 10.78825,
- "40": 10.75548,
- "41": 10.8955,
- "42": 10.70411,
- "43": 10.66907,
- "44": 10.72512,
- "45": 10.54927,
- "46": 10.46973,
- "47": 10.66311,
- "48": 10.62453,
- "49": 10.61656,
- "50": 10.21176
+ "1": 11.08538,
+ "2": 11.09904,
+ "3": 9.95225,
+ "4": 10.10271,
+ "5": 10.14422,
+ "6": 10.10111,
+ "7": 10.21424,
+ "8": 10.17907,
+ "9": 10.24606,
+ "10": 10.17794,
+ "11": 10.24115,
+ "12": 10.25975,
+ "13": 10.3654,
+ "14": 10.2976,
+ "15": 10.4152,
+ "16": 10.41561,
+ "17": 10.39063,
+ "18": 10.43635,
+ "19": 10.48229,
+ "20": 10.4958,
+ "21": 10.4449,
+ "22": 10.48072,
+ "23": 10.47255,
+ "24": 10.47574,
+ "25": 10.53493,
+ "26": 10.47164,
+ "27": 10.48921,
+ "28": 10.47788,
+ "29": 10.4831,
+ "30": 10.51378,
+ "31": 10.52498,
+ "32": 10.47843,
+ "33": 10.4765,
+ "34": 10.49449,
+ "35": 10.31486,
+ "36": 10.10124,
+ "37": 10.08548,
+ "38": 9.99805,
+ "39": 10.02259,
+ "40": 10.06908,
+ "41": 9.99792,
+ "42": 9.96682,
+ "43": 9.93446,
+ "44": 9.93088,
+ "45": 10.01076,
+ "46": 9.93176,
+ "47": 10.03119,
+ "48": 9.98395,
+ "49": 10.04856,
+ "50": 9.98221
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 47165216.0,
- "2": 46897552.0,
- "3": 52682736.0,
- "4": 70585808.0,
- "5": 1850183680.0,
- "6": 171098656.0,
- "7": 436105120.0,
- "8": 1850183680.0,
- "9": 1850183680.0,
- "10": 1850183680.0,
- "11": 1850183680.0,
- "12": 1850183680.0,
- "13": 1850183680.0,
- "14": 1850183680.0,
- "15": 555857088.0,
- "16": 1850183680.0,
- "17": 1850183680.0,
- "18": 1850183680.0,
- "19": 886404992.0,
- "20": 654826944.0,
- "21": 603993664.0,
- "22": 726709632.0,
- "23": 566656896.0,
- "24": 1850183680.0,
- "25": 799245696.0,
- "26": 978252032.0,
- "27": 1850183680.0,
- "28": 906183104.0,
- "29": 1850183680.0,
- "30": 1850183680.0,
- "31": 810874112.0,
- "32": 1850183680.0,
- "33": 1850183680.0,
- "34": 553779584.0,
- "35": 565382400.0,
- "36": 585787712.0,
- "37": 627284160.0,
- "38": 331368192.0,
- "39": 638619264.0,
- "40": 1850183680.0,
- "41": 1850183680.0,
- "42": 1850183680.0,
- "43": 1850183680.0,
- "44": 1850183680.0,
- "45": 1850183680.0,
- "46": 1850183680.0,
- "47": 434842944.0,
- "48": 1850183680.0,
- "49": 575219328.0,
- "50": 1850183680.0
+ "1": 47129288.0,
+ "2": 46932668.0,
+ "3": 68785640.0,
+ "4": 53814060.0,
+ "5": 1161269248.0,
+ "6": 1161269248.0,
+ "7": 1161269248.0,
+ "8": 1161269248.0,
+ "9": 179064480.0,
+ "10": 1161269248.0,
+ "11": 1161269248.0,
+ "12": 1161269248.0,
+ "13": 1161269248.0,
+ "14": 1161269248.0,
+ "15": 1161269248.0,
+ "16": 1161269248.0,
+ "17": 1161269248.0,
+ "18": 1161269248.0,
+ "19": 1161269248.0,
+ "20": 1161269248.0,
+ "21": 1161269248.0,
+ "22": 1161269248.0,
+ "23": 1161269248.0,
+ "24": 1161269248.0,
+ "25": 1161269248.0,
+ "26": 1161269248.0,
+ "27": 1161269248.0,
+ "28": 1161269248.0,
+ "29": 1161269248.0,
+ "30": 1161269248.0,
+ "31": 1161269248.0,
+ "32": 1161269248.0,
+ "33": 1161269248.0,
+ "34": 257282240.0,
+ "35": 1161269248.0,
+ "36": 1161269248.0,
+ "37": 471564736.0,
+ "38": 1161269248.0,
+ "39": 1161269248.0,
+ "40": 174854320.0,
+ "41": 1161269248.0,
+ "42": 1161269248.0,
+ "43": 1161269248.0,
+ "44": 286321376.0,
+ "45": 1161269248.0,
+ "46": 208607168.0,
+ "47": 1161269248.0,
+ "48": 349404864.0,
+ "49": 1161269248.0,
+ "50": 239160144.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 5283616256.0,
- "2": 5288015360.0,
- "3": 5288218112.0,
- "4": 5288420864.0,
- "5": 5288623616.0,
- "6": 5287812608.0,
- "7": 5288015360.0,
- "8": 5288218112.0,
- "9": 5287711232.0,
- "10": 5287913984.0,
- "11": 5288116736.0,
- "12": 5288319488.0,
- "13": 5288522240.0,
- "14": 5288724992.0,
- "15": 5288927744.0,
- "16": 5289130496.0,
- "17": 5289333248.0,
- "18": 5289536000.0,
- "19": 5289738752.0,
- "20": 5289941504.0,
- "21": 5290144256.0,
- "22": 5290347008.0,
- "23": 5290549760.0,
- "24": 5290752512.0,
- "25": 5290955264.0,
- "26": 5291158016.0,
- "27": 5291360768.0,
- "28": 5291563520.0,
- "29": 5291766272.0,
- "30": 5291969024.0,
- "31": 5292171776.0,
- "32": 5292374528.0,
- "33": 5292577280.0,
- "34": 5292780032.0,
- "35": 5292982784.0,
- "36": 5293185536.0,
- "37": 5293388288.0,
- "38": 5293591040.0,
- "39": 5293793792.0,
- "40": 5293996544.0,
- "41": 5294199296.0,
- "42": 5294402048.0,
- "43": 5294604800.0,
- "44": 5294807552.0,
- "45": 5295010304.0,
- "46": 5295213056.0,
- "47": 5295415808.0,
- "48": 5295618560.0,
- "49": 5295821312.0,
- "50": 5296024064.0
+ "1": 5283603968.0,
+ "2": 5283808768.0,
+ "3": 5284011520.0,
+ "4": 5283504640.0,
+ "5": 5283707392.0,
+ "6": 5283910144.0,
+ "7": 5284112896.0,
+ "8": 5284315648.0,
+ "9": 5284518400.0,
+ "10": 5284721152.0,
+ "11": 5284923904.0,
+ "12": 5285126656.0,
+ "13": 5285329408.0,
+ "14": 5285532160.0,
+ "15": 5285734912.0,
+ "16": 5285937664.0,
+ "17": 5286140416.0,
+ "18": 5286343168.0,
+ "19": 5286545920.0,
+ "20": 5286748672.0,
+ "21": 5286951424.0,
+ "22": 5287154176.0,
+ "23": 5287356928.0,
+ "24": 5287559680.0,
+ "25": 5287762432.0,
+ "26": 5287965184.0,
+ "27": 5288167936.0,
+ "28": 5288370688.0,
+ "29": 5288573440.0,
+ "30": 5288776192.0,
+ "31": 5288978944.0,
+ "32": 5289181696.0,
+ "33": 5289384448.0,
+ "34": 5289587200.0,
+ "35": 5289789952.0,
+ "36": 5289992704.0,
+ "37": 5290195456.0,
+ "38": 5290398208.0,
+ "39": 5290600960.0,
+ "40": 5290803712.0,
+ "41": 5291006464.0,
+ "42": 5291209216.0,
+ "43": 5291411968.0,
+ "44": 5291614720.0,
+ "45": 5291817472.0,
+ "46": 5292020224.0,
+ "47": 5292222976.0,
+ "48": 5292425728.0,
+ "49": 5292628480.0,
+ "50": 5292831232.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 5283618816.0,
- "2": 8185453056.0,
- "3": 8185453056.0,
- "4": 8185453056.0,
- "5": 8195318272.0,
- "6": 8195318272.0,
- "7": 8195318272.0,
- "8": 8195318272.0,
- "9": 8195318272.0,
- "10": 8195318272.0,
- "11": 8195318272.0,
- "12": 8195318272.0,
- "13": 8195318272.0,
- "14": 8195318272.0,
- "15": 8195318272.0,
- "16": 8199233024.0,
- "17": 8199233024.0,
- "18": 8199233024.0,
- "19": 8199233024.0,
- "20": 8199233024.0,
- "21": 8238446080.0,
- "22": 8238446080.0,
- "23": 8238446080.0,
- "24": 8238446080.0,
- "25": 8247293440.0,
- "26": 8247293440.0,
- "27": 8247293440.0,
- "28": 8250185216.0,
- "29": 8255527424.0,
- "30": 8255527424.0,
- "31": 8255527424.0,
- "32": 8255527424.0,
- "33": 8255527424.0,
- "34": 8255527424.0,
- "35": 8255527424.0,
- "36": 8255527424.0,
- "37": 8255527424.0,
- "38": 8255527424.0,
- "39": 8255527424.0,
- "40": 8255527424.0,
- "41": 8255527424.0,
- "42": 8255527424.0,
- "43": 8255527424.0,
- "44": 8255527424.0,
- "45": 8255527424.0,
- "46": 8255527424.0,
- "47": 8255527424.0,
- "48": 8255527424.0,
- "49": 8255527424.0,
- "50": 8255527424.0
+ "1": 5283606528.0,
+ "2": 8217088512.0,
+ "3": 8217088512.0,
+ "4": 8217088512.0,
+ "5": 8217088512.0,
+ "6": 8217088512.0,
+ "7": 8217088512.0,
+ "8": 8230964736.0,
+ "9": 8230964736.0,
+ "10": 8254195200.0,
+ "11": 8254195200.0,
+ "12": 8271429120.0,
+ "13": 8271429120.0,
+ "14": 8271429120.0,
+ "15": 8271429120.0,
+ "16": 8271429120.0,
+ "17": 8271429120.0,
+ "18": 8271429120.0,
+ "19": 8271429120.0,
+ "20": 8271429120.0,
+ "21": 8271429120.0,
+ "22": 8271835648.0,
+ "23": 8272294400.0,
+ "24": 8281385472.0,
+ "25": 8281385472.0,
+ "26": 8281385472.0,
+ "27": 8281385472.0,
+ "28": 8281385472.0,
+ "29": 8281385472.0,
+ "30": 8281385472.0,
+ "31": 8281385472.0,
+ "32": 8281385472.0,
+ "33": 8281385472.0,
+ "34": 8281385472.0,
+ "35": 8281385472.0,
+ "36": 8281385472.0,
+ "37": 8281385472.0,
+ "38": 8281385472.0,
+ "39": 8281385472.0,
+ "40": 8281385472.0,
+ "41": 8281385472.0,
+ "42": 8281385472.0,
+ "43": 8281385472.0,
+ "44": 8281385472.0,
+ "45": 8281385472.0,
+ "46": 8281385472.0,
+ "47": 8281385472.0,
+ "48": 8281385472.0,
+ "49": 8281385472.0,
+ "50": 8281385472.0
}
},
"mtp_1 loss": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 11.07401,
- "2": 11.0927,
- "3": 10.83159,
- "4": 10.61397,
- "5": 10.85768,
- "6": 9.79263,
- "7": 10.90607,
- "8": 10.19798,
- "9": 9.82717,
- "10": 9.23805,
- "11": 11.0712,
- "12": 11.11709,
- "13": 10.03407,
- "14": 10.27606,
- "15": 10.73067,
- "16": 10.91485,
- "17": 10.76886,
- "18": 10.49659,
- "19": 10.96955,
- "20": 10.45905,
- "21": 10.91629,
- "22": 10.05081,
- "23": 10.44411,
- "24": 9.74826,
- "25": 10.81497,
- "26": 10.38519,
- "27": 10.31999,
- "28": 10.27887,
- "29": 10.40945,
- "30": 10.20684,
- "31": 10.54594,
- "32": 8.85942,
- "33": 9.75619,
- "34": 10.56214,
- "35": 10.59167,
- "36": 9.30537,
- "37": 10.59407,
- "38": 10.2994,
- "39": 10.69954,
- "40": 10.37003,
- "41": 10.248,
- "42": 8.56376,
- "43": 10.49224,
- "44": 10.57211,
- "45": 9.36238,
- "46": 10.2179,
- "47": 10.63449,
- "48": 10.56697,
- "49": 10.44093,
- "50": 9.49252
+ "1": 11.03608,
+ "2": 11.05276,
+ "3": 10.88297,
+ "4": 10.86018,
+ "5": 10.84504,
+ "6": 10.82804,
+ "7": 10.82565,
+ "8": 10.83264,
+ "9": 10.80773,
+ "10": 10.74625,
+ "11": 10.75106,
+ "12": 10.71595,
+ "13": 10.77482,
+ "14": 10.63377,
+ "15": 10.70519,
+ "16": 10.67084,
+ "17": 10.68106,
+ "18": 10.71544,
+ "19": 10.71193,
+ "20": 10.7072,
+ "21": 10.61619,
+ "22": 10.62544,
+ "23": 10.65332,
+ "24": 10.59991,
+ "25": 10.70041,
+ "26": 10.65128,
+ "27": 10.59359,
+ "28": 10.63314,
+ "29": 10.6259,
+ "30": 10.6681,
+ "31": 10.68389,
+ "32": 10.53894,
+ "33": 10.62813,
+ "34": 10.66321,
+ "35": 10.68769,
+ "36": 10.54902,
+ "37": 10.59647,
+ "38": 10.56635,
+ "39": 10.53655,
+ "40": 10.63989,
+ "41": 10.60253,
+ "42": 10.5743,
+ "43": 10.54666,
+ "44": 10.5965,
+ "45": 10.62403,
+ "46": 10.52939,
+ "47": 10.62763,
+ "48": 10.54707,
+ "49": 10.64679,
+ "50": 10.58971
}
},
"iteration-time": {
@@ -289,56 +289,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 71.30157,
- "2": 2.34464,
- "3": 2.38747,
- "4": 2.10322,
- "5": 2.12945,
- "6": 2.0424,
- "7": 2.12036,
- "8": 2.0147,
- "9": 2.04925,
- "10": 2.02797,
- "11": 1.95087,
- "12": 2.04985,
- "13": 1.94106,
- "14": 1.90425,
- "15": 1.89051,
- "16": 1.89398,
- "17": 1.94082,
- "18": 1.93176,
- "19": 1.94027,
- "20": 1.90271,
- "21": 1.91097,
- "22": 1.90382,
- "23": 1.93889,
- "24": 1.90551,
- "25": 1.90947,
- "26": 1.92126,
- "27": 1.89917,
- "28": 1.89866,
- "29": 1.93981,
- "30": 1.90782,
- "31": 1.91244,
- "32": 1.93864,
- "33": 1.93947,
- "34": 1.96882,
- "35": 1.89751,
- "36": 1.94038,
- "37": 1.90603,
- "38": 1.94988,
- "39": 1.89874,
- "40": 1.90233,
- "41": 1.92861,
- "42": 1.93931,
- "43": 1.91212,
- "44": 1.92615,
- "45": 1.89555,
- "46": 1.94522,
- "47": 1.9103,
- "48": 1.94689,
- "49": 1.9355,
- "50": 1.89832
+ "1": "nan",
+ "2": 38.90311,
+ "3": 2.34916,
+ "4": 2.8016,
+ "5": 2.81521,
+ "6": 2.55136,
+ "7": 2.67655,
+ "8": 2.26895,
+ "9": 2.84909,
+ "10": 1.90067,
+ "11": 1.88083,
+ "12": 1.9061,
+ "13": 1.88093,
+ "14": 2.39479,
+ "15": 1.87455,
+ "16": 2.51045,
+ "17": 1.87702,
+ "18": 1.8748,
+ "19": 1.87706,
+ "20": 1.88036,
+ "21": 1.88232,
+ "22": 1.87871,
+ "23": 1.87259,
+ "24": 1.88165,
+ "25": 1.88056,
+ "26": 2.42597,
+ "27": 1.88499,
+ "28": 1.87369,
+ "29": 1.87582,
+ "30": 1.87682,
+ "31": 1.87626,
+ "32": 1.88702,
+ "33": 1.87716,
+ "34": 1.89392,
+ "35": 1.8889,
+ "36": 1.89199,
+ "37": 1.88025,
+ "38": 1.87713,
+ "39": 1.89655,
+ "40": 1.90178,
+ "41": 1.89724,
+ "42": 1.87668,
+ "43": 1.89246,
+ "44": 1.88668,
+ "45": 1.89939,
+ "46": 1.88,
+ "47": 1.88899,
+ "48": 1.88981,
+ "49": 1.88125,
+ "50": 1.8958
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_fine_grained_offloading/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_fine_grained_offloading/model_config.yaml
index a37dd0dc658..845f0990460 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_fine_grained_offloading/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_fine_grained_offloading/model_config.yaml
@@ -44,14 +44,14 @@ MODEL_ARGS:
# Data args
--seq-length: 4096
--data-cache-path: ${DATA_CACHE_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
# Add network size args
--num-layers: 15
--moe-layer-freq: ([0]*3+[1]*12)
- --pipeline-model-parallel-layout: Et*3\\|\\(tt\\|\\)*6mL # Et*3|(tt|)*6mL
+ --pipeline-model-parallel-layout: Et*3|(tt|)*6mL
--hidden-size: 1024
--ffn-hidden-size: 4096
--num-attention-heads: 32
@@ -131,6 +131,8 @@ MODEL_ARGS:
--bf16: true
--exit-interval: 50
--overlap-moe-expert-parallel-comm: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular # Usually ckpt-resume, but as a WAR to #513 set to regular
METRICS:
- "iteration-time"
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_no_mtp_no_a2a_ovlp_fine_grained_offloading/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_no_mtp_no_a2a_ovlp_fine_grained_offloading/golden_values_dev_dgx_h100.json
index 57848f8130e..c9b7e4b8b1d 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_no_mtp_no_a2a_ovlp_fine_grained_offloading/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_no_mtp_no_a2a_ovlp_fine_grained_offloading/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 11.01693,
- "2": 11.06263,
- "3": 10.08845,
- "4": 9.73223,
- "5": 10.41008,
- "6": 10.46377,
- "7": 11.62265,
- "8": 12.30479,
- "9": 12.258,
- "10": 12.11321,
- "11": 11.67717,
- "12": 11.60724,
- "13": 11.46408,
- "14": 11.41026,
- "15": 11.44828,
- "16": 11.31999,
- "17": 11.28503,
- "18": 11.35547,
- "19": 11.35205,
- "20": 11.50757,
- "21": 11.41181,
- "22": 11.56383,
- "23": 11.41906,
- "24": 11.39788,
- "25": 11.26438,
- "26": 11.36733,
- "27": 11.37099,
- "28": 11.40035,
- "29": 11.42808,
- "30": 11.53613,
- "31": 11.3981,
- "32": 12.00058,
- "33": 11.68213,
- "34": 11.38046,
- "35": 11.36734,
- "36": 11.77291,
- "37": 11.34584,
- "38": 11.4654,
- "39": 11.33231,
- "40": 11.43538,
- "41": 11.47405,
- "42": 12.09241,
- "43": 11.39968,
- "44": 11.38762,
- "45": 11.79356,
- "46": 11.4469,
- "47": 11.3507,
- "48": 11.30787,
- "49": 11.39251,
- "50": 11.7264
+ "1": 11.06362,
+ "2": 11.07891,
+ "3": 9.91543,
+ "4": 10.0055,
+ "5": 10.20278,
+ "6": 10.4242,
+ "7": 10.80496,
+ "8": 10.80271,
+ "9": 10.31584,
+ "10": 10.19739,
+ "11": 10.22741,
+ "12": 10.08478,
+ "13": 9.95048,
+ "14": 10.0836,
+ "15": 10.05826,
+ "16": 10.13331,
+ "17": 9.96659,
+ "18": 10.05414,
+ "19": 10.03153,
+ "20": 10.12479,
+ "21": 10.07847,
+ "22": 10.07475,
+ "23": 10.08324,
+ "24": 10.04691,
+ "25": 10.08554,
+ "26": 10.05148,
+ "27": 10.07512,
+ "28": 10.15905,
+ "29": 10.15637,
+ "30": 10.09608,
+ "31": 10.0862,
+ "32": 10.23795,
+ "33": 10.1936,
+ "34": 10.15938,
+ "35": 10.15776,
+ "36": 10.30055,
+ "37": 10.20324,
+ "38": 10.27399,
+ "39": 10.35246,
+ "40": 10.23632,
+ "41": 10.33026,
+ "42": 10.36794,
+ "43": 10.30847,
+ "44": 10.24008,
+ "45": 10.23923,
+ "46": 10.41204,
+ "47": 10.44848,
+ "48": 10.3828,
+ "49": 10.31727,
+ "50": 10.32569
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 47167880.0,
- "2": 46899772.0,
- "3": 1722086400.0,
- "4": 1722086400.0,
- "5": 188597600.0,
- "6": 120779000.0,
- "7": 527310080.0,
- "8": 1722086400.0,
- "9": 1722086400.0,
- "10": 321966144.0,
- "11": 493484608.0,
- "12": 1722086400.0,
- "13": 529395136.0,
- "14": 1722086400.0,
- "15": 1722086400.0,
- "16": 723018944.0,
- "17": 233377744.0,
- "18": 642084544.0,
- "19": 1722086400.0,
- "20": 1722086400.0,
- "21": 578776704.0,
- "22": 396416192.0,
- "23": 506872960.0,
- "24": 670044160.0,
- "25": 884090624.0,
- "26": 912192512.0,
- "27": 764026112.0,
- "28": 972234112.0,
- "29": 915345600.0,
- "30": 937728768.0,
- "31": 1722086400.0,
- "32": 976440512.0,
- "33": 984833664.0,
- "34": 802321088.0,
- "35": 1722086400.0,
- "36": 931810816.0,
- "37": 897772032.0,
- "38": 982505792.0,
- "39": 704699008.0,
- "40": 688513344.0,
- "41": 946725760.0,
- "42": 1722086400.0,
- "43": 1722086400.0,
- "44": 875336384.0,
- "45": 1722086400.0,
- "46": 909066432.0,
- "47": 900409280.0,
- "48": 890279744.0,
- "49": 597272192.0,
- "50": 921883712.0
+ "1": 47132104.0,
+ "2": 46935332.0,
+ "3": 75084320.0,
+ "4": 63257092.0,
+ "5": 1085666304.0,
+ "6": 179405216.0,
+ "7": 185295008.0,
+ "8": 518773504.0,
+ "9": 405537472.0,
+ "10": 329832896.0,
+ "11": 452344384.0,
+ "12": 349490720.0,
+ "13": 1085666304.0,
+ "14": 1085666304.0,
+ "15": 1085666304.0,
+ "16": 418744768.0,
+ "17": 380995488.0,
+ "18": 311058240.0,
+ "19": 386626944.0,
+ "20": 1085666304.0,
+ "21": 375110848.0,
+ "22": 314975904.0,
+ "23": 305120864.0,
+ "24": 1085666304.0,
+ "25": 1085666304.0,
+ "26": 1085666304.0,
+ "27": 453042048.0,
+ "28": 503786144.0,
+ "29": 525389504.0,
+ "30": 484016032.0,
+ "31": 430473664.0,
+ "32": 1085666304.0,
+ "33": 1085666304.0,
+ "34": 1085666304.0,
+ "35": 310418496.0,
+ "36": 331351232.0,
+ "37": 528177760.0,
+ "38": 1085666304.0,
+ "39": 290084544.0,
+ "40": 1085666304.0,
+ "41": 1085666304.0,
+ "42": 365336864.0,
+ "43": 1085666304.0,
+ "44": 1085666304.0,
+ "45": 461386368.0,
+ "46": 1085666304.0,
+ "47": 375535776.0,
+ "48": 1085666304.0,
+ "49": 1085666304.0,
+ "50": 1085666304.0
}
},
"mem-allocated-bytes": {
@@ -176,55 +176,55 @@
"step_interval": 1,
"values": {
"1": 4313449472.0,
- "2": 7108272640.0,
- "3": 7108272640.0,
- "4": 7108272640.0,
- "5": 7119571456.0,
- "6": 7119571456.0,
- "7": 7129409024.0,
- "8": 7158368768.0,
- "9": 7158368768.0,
- "10": 7158838784.0,
- "11": 7202046464.0,
- "12": 7202046464.0,
- "13": 7202046464.0,
- "14": 7202046464.0,
- "15": 7202046464.0,
- "16": 7202046464.0,
- "17": 7202046464.0,
- "18": 7202046464.0,
- "19": 7202046464.0,
- "20": 7202046464.0,
- "21": 7202046464.0,
- "22": 7202046464.0,
- "23": 7202046464.0,
- "24": 7202046464.0,
- "25": 7202046464.0,
- "26": 7202046464.0,
- "27": 7202046464.0,
- "28": 7202046464.0,
- "29": 7202046464.0,
- "30": 7202046464.0,
- "31": 7202046464.0,
- "32": 7202046464.0,
- "33": 7202046464.0,
- "34": 7202046464.0,
- "35": 7202046464.0,
- "36": 7202046464.0,
- "37": 7202046464.0,
- "38": 7202046464.0,
- "39": 7202046464.0,
- "40": 7202046464.0,
- "41": 7202046464.0,
- "42": 7202046464.0,
- "43": 7202046464.0,
- "44": 7202046464.0,
- "45": 7202046464.0,
- "46": 7202046464.0,
- "47": 7202046464.0,
- "48": 7202046464.0,
- "49": 7202046464.0,
- "50": 7202046464.0
+ "2": 7095137792.0,
+ "3": 7095137792.0,
+ "4": 7095137792.0,
+ "5": 7104131072.0,
+ "6": 7104131072.0,
+ "7": 7140368384.0,
+ "8": 7145902592.0,
+ "9": 7145902592.0,
+ "10": 7157680640.0,
+ "11": 7157680640.0,
+ "12": 7157680640.0,
+ "13": 7157680640.0,
+ "14": 7157680640.0,
+ "15": 7157680640.0,
+ "16": 7157680640.0,
+ "17": 7157680640.0,
+ "18": 7157680640.0,
+ "19": 7157680640.0,
+ "20": 7157680640.0,
+ "21": 7157680640.0,
+ "22": 7157680640.0,
+ "23": 7157680640.0,
+ "24": 7157680640.0,
+ "25": 7157680640.0,
+ "26": 7157680640.0,
+ "27": 7157680640.0,
+ "28": 7157680640.0,
+ "29": 7157680640.0,
+ "30": 7157680640.0,
+ "31": 7157680640.0,
+ "32": 7157680640.0,
+ "33": 7157680640.0,
+ "34": 7157680640.0,
+ "35": 7157680640.0,
+ "36": 7157680640.0,
+ "37": 7157680640.0,
+ "38": 7157680640.0,
+ "39": 7157680640.0,
+ "40": 7157680640.0,
+ "41": 7157680640.0,
+ "42": 7157680640.0,
+ "43": 7157680640.0,
+ "44": 7157680640.0,
+ "45": 7157680640.0,
+ "46": 7157680640.0,
+ "47": 7157680640.0,
+ "48": 7157680640.0,
+ "49": 7157680640.0,
+ "50": 7157680640.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 90.31742,
- "2": 2.522,
- "3": 2.42029,
- "4": 2.06158,
- "5": 2.28893,
- "6": 3.01447,
- "7": 3.96389,
- "8": 3.20878,
- "9": 2.43815,
- "10": 1.94158,
- "11": 1.95031,
- "12": 1.98877,
- "13": 1.92978,
- "14": 1.93494,
- "15": 1.92559,
- "16": 1.95925,
- "17": 2.59672,
- "18": 1.94175,
- "19": 1.92388,
- "20": 1.92283,
- "21": 1.92623,
- "22": 1.92561,
- "23": 1.92611,
- "24": 1.94339,
- "25": 2.02939,
- "26": 1.93181,
- "27": 1.92433,
- "28": 1.96842,
- "29": 1.92479,
- "30": 1.93949,
- "31": 1.96151,
- "32": 1.93071,
- "33": 1.92266,
- "34": 1.92587,
- "35": 1.92251,
- "36": 1.92324,
- "37": 1.93141,
- "38": 1.92431,
- "39": 1.93685,
- "40": 1.92592,
- "41": 1.92962,
- "42": 1.92986,
- "43": 1.92956,
- "44": 1.93019,
- "45": 1.93251,
- "46": 1.92915,
- "47": 1.93714,
- "48": 1.93564,
- "49": 1.94035,
- "50": 1.93018
+ "1": "nan",
+ "2": 39.0436,
+ "3": 2.48884,
+ "4": 3.09118,
+ "5": 2.38842,
+ "6": 3.72813,
+ "7": 2.8892,
+ "8": 1.937,
+ "9": 2.60903,
+ "10": 2.67917,
+ "11": 1.93875,
+ "12": 1.92178,
+ "13": 1.9268,
+ "14": 2.69966,
+ "15": 1.93359,
+ "16": 1.96628,
+ "17": 2.57459,
+ "18": 1.95656,
+ "19": 1.93192,
+ "20": 1.95101,
+ "21": 1.93058,
+ "22": 1.9464,
+ "23": 1.94635,
+ "24": 1.96859,
+ "25": 1.94747,
+ "26": 2.40478,
+ "27": 1.95303,
+ "28": 1.93276,
+ "29": 1.93777,
+ "30": 1.96213,
+ "31": 1.93088,
+ "32": 1.93091,
+ "33": 1.92797,
+ "34": 1.92677,
+ "35": 1.94437,
+ "36": 1.92998,
+ "37": 1.92836,
+ "38": 1.93278,
+ "39": 1.93503,
+ "40": 1.9363,
+ "41": 1.93697,
+ "42": 1.95059,
+ "43": 1.97574,
+ "44": 1.93036,
+ "45": 1.93249,
+ "46": 1.9347,
+ "47": 1.93057,
+ "48": 1.94087,
+ "49": 1.94429,
+ "50": 1.93108
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_no_mtp_no_a2a_ovlp_fine_grained_offloading/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_no_mtp_no_a2a_ovlp_fine_grained_offloading/model_config.yaml
index da78378ddae..d16c56b2264 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_no_mtp_no_a2a_ovlp_fine_grained_offloading/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_no_mtp_no_a2a_ovlp_fine_grained_offloading/model_config.yaml
@@ -44,14 +44,14 @@ MODEL_ARGS:
# Data args
--seq-length: 4096
--data-cache-path: ${DATA_CACHE_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
# Add network size args
--num-layers: 15
--moe-layer-freq: ([0]*3+[1]*12)
- --pipeline-model-parallel-layout: Et*3\\|\\(tt\\|\\)*6L # Et*3|(tt|)*6L
+ --pipeline-model-parallel-layout: Et*3|(tt|)*6L
--hidden-size: 1024
--ffn-hidden-size: 4096
--num-attention-heads: 32
@@ -127,6 +127,8 @@ MODEL_ARGS:
# Add mixed precision args
--bf16: true
--exit-interval: 50
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: regular # Usually ckpt-resume, but as a WAR to #513 set to regular
METRICS:
- "lm loss"
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_gb200.json
index a35a7574e59..ac72c629fb2 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_gb200.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.81233,
- "2": 10.82416,
- "3": 10.81841,
- "4": 10.81357,
- "5": 10.85116,
- "6": 10.85502,
- "7": 10.84363,
- "8": 10.83621,
- "9": 10.84178,
- "10": 10.77391,
- "11": 10.86217,
- "12": 10.84672,
- "13": 10.85692,
- "14": 10.8614,
- "15": 10.80709,
- "16": 10.78544,
- "17": 10.7701,
- "18": 10.79072,
- "19": 10.78529,
- "20": 10.71496,
- "21": 10.67362,
- "22": 10.5386,
- "23": 10.69608,
- "24": 10.58118,
- "25": 10.52212,
- "26": 10.58665,
- "27": 10.60344,
- "28": 10.5676,
- "29": 10.5868,
- "30": 10.36177,
- "31": 10.09661,
- "32": 10.45911,
- "33": 10.45926,
- "34": 10.21524,
- "35": 10.2617,
- "36": 10.22327,
- "37": 10.35631,
- "38": 10.20637,
- "39": 10.40825,
- "40": 10.08881,
- "41": 10.13871,
- "42": 10.22236,
- "43": 9.82978,
- "44": 9.96931,
- "45": 9.83925,
- "46": 9.81008,
- "47": 10.16408,
- "48": 9.84608,
- "49": 9.53674,
- "50": 9.91754,
- "51": 9.86341,
- "52": 9.74862,
- "53": 10.08034,
- "54": 9.96286,
- "55": 9.89221,
- "56": 9.64295,
- "57": 9.48196,
- "58": 9.85327,
- "59": 9.58985,
- "60": 9.5157,
- "61": 9.70142,
- "62": 10.01153,
- "63": 9.40557,
- "64": 9.78559,
- "65": 8.96047,
- "66": 9.72678,
- "67": 9.38244,
- "68": 9.79903,
- "69": 9.81114,
- "70": 9.74788,
- "71": 9.6452,
- "72": 9.6027,
- "73": 9.51692,
- "74": 8.95583,
- "75": 9.43449,
- "76": 9.10005,
- "77": 10.07816,
- "78": 9.72912,
- "79": 9.39357,
- "80": 9.41584,
- "81": 9.49174,
- "82": 9.71087,
- "83": 9.32591,
- "84": 9.42272,
- "85": 9.62054,
- "86": 9.08096,
- "87": 9.59797,
- "88": 9.7551,
- "89": 9.6096,
- "90": 9.83264,
- "91": 9.34163,
- "92": 9.3578,
- "93": 9.09025,
- "94": 8.83205,
- "95": 9.52868,
- "96": 9.5278,
- "97": 9.30277,
- "98": 9.66393,
- "99": 8.89773,
- "100": 9.404
+ "1": 10.92085,
+ "2": 10.91149,
+ "3": 10.9181,
+ "4": 10.91886,
+ "5": 10.90438,
+ "6": 10.90671,
+ "7": 10.91414,
+ "8": 10.91035,
+ "9": 10.91487,
+ "10": 10.9074,
+ "11": 10.89662,
+ "12": 10.89656,
+ "13": 10.90112,
+ "14": 10.89347,
+ "15": 10.87256,
+ "16": 10.86157,
+ "17": 10.87439,
+ "18": 10.85933,
+ "19": 10.86607,
+ "20": 10.78248,
+ "21": 10.77929,
+ "22": 10.76487,
+ "23": 10.75753,
+ "24": 10.72003,
+ "25": 10.72234,
+ "26": 10.71687,
+ "27": 10.68331,
+ "28": 10.62307,
+ "29": 10.58123,
+ "30": 10.56508,
+ "31": 10.55921,
+ "32": 10.54822,
+ "33": 10.51345,
+ "34": 10.48735,
+ "35": 10.47842,
+ "36": 10.45969,
+ "37": 10.43041,
+ "38": 10.4356,
+ "39": 10.4024,
+ "40": 10.37953,
+ "41": 10.36358,
+ "42": 10.33786,
+ "43": 10.32307,
+ "44": 10.28549,
+ "45": 10.311,
+ "46": 10.26833,
+ "47": 10.2563,
+ "48": 10.20775,
+ "49": 10.20838,
+ "50": 10.20921,
+ "51": 10.20906,
+ "52": 10.16628,
+ "53": 10.17227,
+ "54": 10.13282,
+ "55": 10.10824,
+ "56": 10.13284,
+ "57": 10.12132,
+ "58": 10.13382,
+ "59": 10.08181,
+ "60": 10.10004,
+ "61": 10.0469,
+ "62": 10.01977,
+ "63": 10.09429,
+ "64": 10.04944,
+ "65": 10.02337,
+ "66": 10.0478,
+ "67": 10.02657,
+ "68": 9.98514,
+ "69": 10.0076,
+ "70": 9.98453,
+ "71": 10.01265,
+ "72": 9.99391,
+ "73": 9.98651,
+ "74": 9.96479,
+ "75": 9.93791,
+ "76": 9.97187,
+ "77": 9.96294,
+ "78": 9.91595,
+ "79": 9.9223,
+ "80": 9.9326,
+ "81": 9.96098,
+ "82": 9.89457,
+ "83": 9.86427,
+ "84": 9.79821,
+ "85": 9.78963,
+ "86": 9.89018,
+ "87": 9.90946,
+ "88": 9.8867,
+ "89": 9.82931,
+ "90": 9.82367,
+ "91": 9.83441,
+ "92": 9.82052,
+ "93": 9.75665,
+ "94": 9.83609,
+ "95": 9.82849,
+ "96": 9.8084,
+ "97": 9.75432,
+ "98": 9.78611,
+ "99": 9.82586,
+ "100": 9.72089
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 5532.0,
- "2": 5934.0,
- "3": 5812.0,
- "4": 5817.0,
- "5": 6435.0,
- "6": 6641.0,
- "7": 5880.0,
- "8": 5900.0,
- "9": 6317.0,
- "10": 5314.0,
- "11": 6659.0,
- "12": 6393.0,
- "13": 6585.0,
- "14": 6649.0,
- "15": 6237.0,
- "16": 6606.0,
- "17": 6232.0,
- "18": 6059.0,
- "19": 6380.0,
- "20": 5723.0,
- "21": 6197.0,
- "22": 5714.0,
- "23": 6527.0,
- "24": 5948.0,
- "25": 5822.0,
- "26": 6271.0,
- "27": 6493.0,
- "28": 6789.0,
- "29": 6971.0,
- "30": 6252.0,
- "31": 5836.0,
- "32": 6830.0,
- "33": 7155.0,
- "34": 6428.0,
- "35": 6909.0,
- "36": 6559.0,
- "37": 7582.0,
- "38": 7325.0,
- "39": 8189.0,
- "40": 7156.0,
- "41": 7113.0,
- "42": 7783.0,
- "43": 7236.0,
- "44": 6958.0,
- "45": 7093.0,
- "46": 7385.0,
- "47": 7634.0,
- "48": 7916.0,
- "49": 7565.0,
- "50": 7795.0,
- "51": 7967.0,
- "52": 7869.0,
- "53": 9001.0,
- "54": 8408.0,
- "55": 7734.0,
- "56": 8108.0,
- "57": 7339.0,
- "58": 8677.0,
- "59": 8299.0,
- "60": 7790.0,
- "61": 8347.0,
- "62": 8345.0,
- "63": 7835.0,
- "64": 8861.0,
- "65": 8293.0,
- "66": 9180.0,
- "67": 8276.0,
- "68": 8251.0,
- "69": 8666.0,
- "70": 9836.0,
- "71": 9020.0,
- "72": 8503.0,
- "73": 8996.0,
- "74": 6967.0,
- "75": 7749.0,
- "76": 8534.0,
- "77": 10688.0,
- "78": 48163.0,
- "79": 9603.0,
- "80": 9991.0,
- "81": 55995.0,
- "82": 9533.0,
- "83": 65535.0,
- "84": 9876.0,
- "85": 15848.0,
- "86": 8732.0,
- "87": 10574.0,
- "88": 12165.0,
- "89": 9808.0,
- "90": 9644.0,
- "91": 8584.0,
- "92": 9300.0,
- "93": 8081.0,
- "94": 9101.0,
- "95": 9919.0,
- "96": 9755.0,
- "97": 11113.0,
- "98": 10522.0,
- "99": 8739.0,
- "100": 9616.0
+ "1": 2700.0,
+ "2": 2507.0,
+ "3": 2564.0,
+ "4": 2572.0,
+ "5": 2515.0,
+ "6": 2482.0,
+ "7": 2625.0,
+ "8": 2433.0,
+ "9": 2471.0,
+ "10": 2566.0,
+ "11": 2651.0,
+ "12": 2575.0,
+ "13": 2433.0,
+ "14": 2615.0,
+ "15": 2487.0,
+ "16": 2531.0,
+ "17": 2631.0,
+ "18": 2480.0,
+ "19": 2568.0,
+ "20": 2501.0,
+ "21": 2543.0,
+ "22": 2542.0,
+ "23": 2528.0,
+ "24": 2603.0,
+ "25": 2628.0,
+ "26": 2700.0,
+ "27": 2622.0,
+ "28": 2652.0,
+ "29": 2782.0,
+ "30": 2693.0,
+ "31": 2807.0,
+ "32": 2742.0,
+ "33": 2777.0,
+ "34": 2880.0,
+ "35": 2927.0,
+ "36": 2806.0,
+ "37": 2926.0,
+ "38": 2931.0,
+ "39": 2936.0,
+ "40": 3067.0,
+ "41": 3108.0,
+ "42": 2865.0,
+ "43": 3143.0,
+ "44": 3044.0,
+ "45": 3315.0,
+ "46": 3226.0,
+ "47": 3260.0,
+ "48": 3417.0,
+ "49": 3451.0,
+ "50": 3444.0,
+ "51": 3345.0,
+ "52": 3602.0,
+ "53": 3441.0,
+ "54": 7431.0,
+ "55": 3869.0,
+ "56": 3977.0,
+ "57": 5304.0,
+ "58": 528937.0,
+ "59": 6301.0,
+ "60": 3729.0,
+ "61": 4480.0,
+ "62": 4191.0,
+ "63": 6523.0,
+ "64": 4999.0,
+ "65": 29052.0,
+ "66": 5748.0,
+ "67": 4412.0,
+ "68": 36762.0,
+ "69": 7194.0,
+ "70": 26426.0,
+ "71": 4265.0,
+ "72": 27355.0,
+ "73": 6981.0,
+ "74": 530924.0,
+ "75": 28625.0,
+ "76": 556227.0,
+ "77": 1054310.0,
+ "78": 36371.0,
+ "79": 529126.0,
+ "80": 554056.0,
+ "81": 529477.0,
+ "82": 529537.0,
+ "83": 528935.0,
+ "84": 528269.0,
+ "85": 528244.0,
+ "86": 529200.0,
+ "87": 583319.0,
+ "88": 529224.0,
+ "89": 528983.0,
+ "90": 529421.0,
+ "91": 528813.0,
+ "92": 528768.0,
+ "93": 528986.0,
+ "94": 528855.0,
+ "95": 528701.0,
+ "96": 531230.0,
+ "97": 529203.0,
+ "98": 529408.0,
+ "99": 38164.0,
+ "100": 560586.0
}
},
"mem-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 982504960.0,
- "2": 1156255744.0,
- "3": 1156255744.0,
- "4": 1156255744.0,
- "5": 1156255744.0,
- "6": 1156255744.0,
- "7": 1156255744.0,
- "8": 1156255744.0,
- "9": 1156255744.0,
- "10": 1156255744.0,
- "11": 1156255744.0,
- "12": 1156255744.0,
- "13": 1156255744.0,
- "14": 1156255744.0,
- "15": 1156255744.0,
- "16": 1156255744.0,
- "17": 1156255744.0,
- "18": 1156255744.0,
- "19": 1156255744.0,
- "20": 1156255744.0,
- "21": 1156255744.0,
- "22": 1156255744.0,
- "23": 1156255744.0,
- "24": 1156255744.0,
- "25": 1156255744.0,
- "26": 1156255744.0,
- "27": 1157233664.0,
- "28": 1157233664.0,
- "29": 1157233664.0,
- "30": 1157233664.0,
- "31": 1157233664.0,
- "32": 1157233664.0,
- "33": 1157233664.0,
- "34": 1157233664.0,
- "35": 1157233664.0,
- "36": 1157233664.0,
- "37": 1157233664.0,
- "38": 1157233664.0,
- "39": 1157233664.0,
- "40": 1157233664.0,
- "41": 1158865408.0,
- "42": 1158865408.0,
- "43": 1158865408.0,
- "44": 1158865408.0,
- "45": 1158865408.0,
- "46": 1158865408.0,
- "47": 1158865408.0,
- "48": 1158865408.0,
- "49": 1158865408.0,
- "50": 1158865408.0,
- "51": 1158865408.0,
- "52": 1158865408.0,
- "53": 1158865408.0,
- "54": 1158865408.0,
- "55": 1159034368.0,
- "56": 1159063040.0,
- "57": 1159542784.0,
- "58": 1159542784.0,
- "59": 1159542784.0,
- "60": 1159542784.0,
- "61": 1165075456.0,
- "62": 1165075456.0,
- "63": 1165075456.0,
- "64": 1165075456.0,
- "65": 1165075456.0,
- "66": 1165075456.0,
- "67": 1165075456.0,
- "68": 1165075456.0,
- "69": 1165075456.0,
- "70": 1165075456.0,
- "71": 1165075456.0,
- "72": 1165075456.0,
- "73": 1165075456.0,
- "74": 1165075456.0,
- "75": 1165075456.0,
- "76": 1166216192.0,
- "77": 1166216192.0,
- "78": 1166216192.0,
- "79": 1166216192.0,
- "80": 1166216192.0,
- "81": 1166216192.0,
- "82": 1166216192.0,
- "83": 1166639104.0,
- "84": 1166639104.0,
- "85": 1166639104.0,
- "86": 1166639104.0,
- "87": 1166639104.0,
- "88": 1166639104.0,
- "89": 1166639104.0,
- "90": 1166639104.0,
- "91": 1166639104.0,
- "92": 1166639104.0,
- "93": 1166639104.0,
- "94": 1166639104.0,
- "95": 1166639104.0,
- "96": 1166639104.0,
- "97": 1166639104.0,
- "98": 1166639104.0,
- "99": 1166639104.0,
- "100": 1166639104.0
+ "1": 981952000.0,
+ "2": 1152564736.0,
+ "3": 1152564736.0,
+ "4": 1152564736.0,
+ "5": 1152564736.0,
+ "6": 1152994816.0,
+ "7": 1153227776.0,
+ "8": 1153227776.0,
+ "9": 1153227776.0,
+ "10": 1153227776.0,
+ "11": 1153227776.0,
+ "12": 1153227776.0,
+ "13": 1153227776.0,
+ "14": 1153227776.0,
+ "15": 1153227776.0,
+ "16": 1153227776.0,
+ "17": 1153227776.0,
+ "18": 1154728448.0,
+ "19": 1154728448.0,
+ "20": 1154728448.0,
+ "21": 1154728448.0,
+ "22": 1154728448.0,
+ "23": 1154728448.0,
+ "24": 1154728448.0,
+ "25": 1154728448.0,
+ "26": 1154728448.0,
+ "27": 1154728448.0,
+ "28": 1154728448.0,
+ "29": 1154728448.0,
+ "30": 1154728448.0,
+ "31": 1154728448.0,
+ "32": 1154728448.0,
+ "33": 1154728448.0,
+ "34": 1154728448.0,
+ "35": 1154728448.0,
+ "36": 1154728448.0,
+ "37": 1155855360.0,
+ "38": 1155855360.0,
+ "39": 1155855360.0,
+ "40": 1155855360.0,
+ "41": 1155855360.0,
+ "42": 1155855360.0,
+ "43": 1155855360.0,
+ "44": 1155855360.0,
+ "45": 1155855360.0,
+ "46": 1155855360.0,
+ "47": 1155855360.0,
+ "48": 1155855360.0,
+ "49": 1155855360.0,
+ "50": 1155855360.0,
+ "51": 1155855360.0,
+ "52": 1155855360.0,
+ "53": 1155855360.0,
+ "54": 1155855360.0,
+ "55": 1155855360.0,
+ "56": 1155855360.0,
+ "57": 1155855360.0,
+ "58": 1155855360.0,
+ "59": 1155855360.0,
+ "60": 1155855360.0,
+ "61": 1155855360.0,
+ "62": 1155855360.0,
+ "63": 1155855360.0,
+ "64": 1155855360.0,
+ "65": 1155855360.0,
+ "66": 1155855360.0,
+ "67": 1155855360.0,
+ "68": 1155855360.0,
+ "69": 1155855360.0,
+ "70": 1155855360.0,
+ "71": 1155855360.0,
+ "72": 1155855360.0,
+ "73": 1155855360.0,
+ "74": 1155855360.0,
+ "75": 1155855360.0,
+ "76": 1155855360.0,
+ "77": 1155855360.0,
+ "78": 1155855360.0,
+ "79": 1155855360.0,
+ "80": 1155855360.0,
+ "81": 1155855360.0,
+ "82": 1155855360.0,
+ "83": 1155855360.0,
+ "84": 1155855360.0,
+ "85": 1155855360.0,
+ "86": 1155855360.0,
+ "87": 1155855360.0,
+ "88": 1155855360.0,
+ "89": 1155855360.0,
+ "90": 1155855360.0,
+ "91": 1155855360.0,
+ "92": 1155855360.0,
+ "93": 1155855360.0,
+ "94": 1155855360.0,
+ "95": 1155855360.0,
+ "96": 1155855360.0,
+ "97": 1155855360.0,
+ "98": 1155855360.0,
+ "99": 1155855360.0,
+ "100": 1155855360.0
}
},
"iteration-time": {
@@ -433,105 +433,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 7.77547,
- "3": 1.01252,
- "4": 1.00639,
- "5": 0.9897,
- "6": 0.99553,
- "7": 0.99796,
- "8": 1.00873,
- "9": 0.99009,
- "10": 0.99264,
- "11": 0.98765,
- "12": 0.99024,
- "13": 0.98319,
- "14": 0.98552,
- "15": 0.99368,
- "16": 0.98342,
- "17": 0.97729,
- "18": 0.97272,
- "19": 0.97308,
- "20": 0.96906,
- "21": 0.9751,
- "22": 0.97375,
- "23": 0.97447,
- "24": 0.98494,
- "25": 0.9779,
- "26": 1.30939,
- "27": 0.9766,
- "28": 0.9856,
- "29": 0.99223,
- "30": 1.27178,
- "31": 0.98025,
- "32": 1.22425,
- "33": 1.27653,
- "34": 0.99358,
- "35": 1.00171,
- "36": 1.25408,
- "37": 1.60005,
- "38": 1.00572,
- "39": 0.98676,
- "40": 0.97218,
- "41": 1.30266,
- "42": 1.29066,
- "43": 0.99057,
- "44": 0.98517,
- "45": 0.97968,
- "46": 0.97289,
- "47": 0.98145,
- "48": 0.9804,
- "49": 0.98022,
- "50": 0.97431,
- "51": 0.97593,
- "52": 0.97255,
- "53": 0.97424,
- "54": 0.97043,
- "55": 0.96887,
- "56": 0.97492,
- "57": 0.97623,
- "58": 0.97423,
- "59": 0.98879,
- "60": 0.97992,
- "61": 0.97895,
- "62": 0.98829,
- "63": 0.98719,
- "64": 0.98651,
- "65": 0.97852,
- "66": 0.98045,
- "67": 0.97825,
- "68": 0.9795,
- "69": 0.97812,
- "70": 0.96297,
- "71": 0.96718,
- "72": 0.98343,
- "73": 0.978,
- "74": 0.99341,
- "75": 0.97768,
- "76": 0.97508,
- "77": 0.97891,
- "78": 0.9739,
- "79": 0.96825,
- "80": 0.96595,
- "81": 0.95551,
- "82": 0.97223,
- "83": 0.9633,
- "84": 0.96539,
- "85": 0.97065,
- "86": 0.97198,
- "87": 0.97978,
- "88": 0.98268,
- "89": 0.99894,
- "90": 1.00246,
- "91": 0.98763,
- "92": 0.98552,
- "93": 0.99698,
- "94": 0.99827,
- "95": 0.99936,
- "96": 0.99295,
- "97": 0.99144,
- "98": 0.99227,
- "99": 0.98859,
- "100": 0.99158
+ "2": 13.23153,
+ "3": 1.11646,
+ "4": 0.94408,
+ "5": 0.93465,
+ "6": 0.94217,
+ "7": 0.93541,
+ "8": 0.92261,
+ "9": 0.93068,
+ "10": 0.93344,
+ "11": 0.92694,
+ "12": 0.91536,
+ "13": 0.92189,
+ "14": 0.91465,
+ "15": 0.92909,
+ "16": 0.91698,
+ "17": 0.91333,
+ "18": 0.9146,
+ "19": 0.91747,
+ "20": 0.92809,
+ "21": 0.93419,
+ "22": 0.91615,
+ "23": 0.92013,
+ "24": 0.96556,
+ "25": 0.92629,
+ "26": 0.9319,
+ "27": 0.92617,
+ "28": 0.91952,
+ "29": 0.92605,
+ "30": 0.93204,
+ "31": 0.92494,
+ "32": 0.93212,
+ "33": 0.92975,
+ "34": 0.93187,
+ "35": 0.9295,
+ "36": 0.92469,
+ "37": 0.91703,
+ "38": 0.91579,
+ "39": 0.93997,
+ "40": 0.92526,
+ "41": 0.92436,
+ "42": 0.92746,
+ "43": 0.9179,
+ "44": 0.92543,
+ "45": 0.92879,
+ "46": 0.93442,
+ "47": 0.92447,
+ "48": 0.92246,
+ "49": 0.92876,
+ "50": 0.91986,
+ "51": 1.22825,
+ "52": 0.92564,
+ "53": 0.92895,
+ "54": 0.92632,
+ "55": 0.9433,
+ "56": 0.91624,
+ "57": 0.90853,
+ "58": 0.916,
+ "59": 0.92299,
+ "60": 0.98107,
+ "61": 0.93662,
+ "62": 0.9095,
+ "63": 0.92053,
+ "64": 0.92071,
+ "65": 0.92227,
+ "66": 0.94096,
+ "67": 0.93442,
+ "68": 0.9353,
+ "69": 0.93385,
+ "70": 0.93093,
+ "71": 0.9308,
+ "72": 0.92788,
+ "73": 0.92859,
+ "74": 0.9187,
+ "75": 0.92936,
+ "76": 0.96281,
+ "77": 0.92451,
+ "78": 0.92361,
+ "79": 0.93119,
+ "80": 0.92428,
+ "81": 0.91639,
+ "82": 0.9134,
+ "83": 0.91925,
+ "84": 0.92435,
+ "85": 0.92551,
+ "86": 0.9293,
+ "87": 0.9169,
+ "88": 0.91356,
+ "89": 0.91054,
+ "90": 0.91197,
+ "91": 0.91068,
+ "92": 0.92053,
+ "93": 0.92794,
+ "94": 0.9279,
+ "95": 0.92695,
+ "96": 0.9232,
+ "97": 0.92721,
+ "98": 0.92606,
+ "99": 0.93523,
+ "100": 0.90689
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_h100.json
index dc836c3d699..f529a646a7e 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.81455,
- "2": 10.81846,
- "3": 10.81528,
- "4": 10.80297,
- "5": 10.8513,
- "6": 10.85011,
- "7": 10.83843,
- "8": 10.83961,
- "9": 10.82224,
- "10": 10.77788,
- "11": 10.86443,
- "12": 10.83746,
- "13": 10.85841,
- "14": 10.86315,
- "15": 10.79766,
- "16": 10.79525,
- "17": 10.77133,
- "18": 10.78938,
- "19": 10.78311,
- "20": 10.71655,
- "21": 10.68376,
- "22": 10.53038,
- "23": 10.69869,
- "24": 10.5858,
- "25": 10.52379,
- "26": 10.58281,
- "27": 10.6097,
- "28": 10.57173,
- "29": 10.59005,
- "30": 10.35671,
- "31": 10.09391,
- "32": 10.45878,
- "33": 10.45658,
- "34": 10.20481,
- "35": 10.26727,
- "36": 10.22341,
- "37": 10.35319,
- "38": 10.19446,
- "39": 10.41712,
- "40": 10.08932,
- "41": 10.12772,
- "42": 10.21193,
- "43": 9.83111,
- "44": 9.96933,
- "45": 9.83615,
- "46": 9.81673,
- "47": 10.15426,
- "48": 9.85308,
- "49": 9.53436,
- "50": 9.91912,
- "51": 9.85363,
- "52": 9.74288,
- "53": 10.07163,
- "54": 9.96275,
- "55": 9.88233,
- "56": 9.63455,
- "57": 9.48649,
- "58": 9.84879,
- "59": 9.589,
- "60": 9.5109,
- "61": 9.703,
- "62": 9.99634,
- "63": 9.40054,
- "64": 9.78477,
- "65": 8.95365,
- "66": 9.71813,
- "67": 9.36915,
- "68": 9.79814,
- "69": 9.79674,
- "70": 9.74886,
- "71": 9.63185,
- "72": 9.59951,
- "73": 9.50305,
- "74": 8.95217,
- "75": 9.43098,
- "76": 9.09068,
- "77": 10.08086,
- "78": 9.7353,
- "79": 9.38859,
- "80": 9.41418,
- "81": 9.48423,
- "82": 9.70903,
- "83": 9.3151,
- "84": 9.41846,
- "85": 9.62239,
- "86": 9.07953,
- "87": 9.59204,
- "88": 9.74948,
- "89": 9.60436,
- "90": 9.82573,
- "91": 9.34231,
- "92": 9.35857,
- "93": 9.07976,
- "94": 8.82788,
- "95": 9.50877,
- "96": 9.52129,
- "97": 9.30597,
- "98": 9.66586,
- "99": 8.87711,
- "100": 9.38978
+ "1": 10.92499,
+ "2": 10.91072,
+ "3": 10.91895,
+ "4": 10.91763,
+ "5": 10.90484,
+ "6": 10.90203,
+ "7": 10.89753,
+ "8": 10.91294,
+ "9": 10.91701,
+ "10": 10.91028,
+ "11": 10.90124,
+ "12": 10.89698,
+ "13": 10.88788,
+ "14": 10.89478,
+ "15": 10.87488,
+ "16": 10.87022,
+ "17": 10.86892,
+ "18": 10.85196,
+ "19": 10.87008,
+ "20": 10.7881,
+ "21": 10.77222,
+ "22": 10.7669,
+ "23": 10.75865,
+ "24": 10.71955,
+ "25": 10.71987,
+ "26": 10.71249,
+ "27": 10.68554,
+ "28": 10.61292,
+ "29": 10.58664,
+ "30": 10.56554,
+ "31": 10.55749,
+ "32": 10.54875,
+ "33": 10.50948,
+ "34": 10.48165,
+ "35": 10.46995,
+ "36": 10.45309,
+ "37": 10.42791,
+ "38": 10.43268,
+ "39": 10.40324,
+ "40": 10.3773,
+ "41": 10.36856,
+ "42": 10.33125,
+ "43": 10.31537,
+ "44": 10.29014,
+ "45": 10.30253,
+ "46": 10.26536,
+ "47": 10.25557,
+ "48": 10.20689,
+ "49": 10.21031,
+ "50": 10.2105,
+ "51": 10.21191,
+ "52": 10.16277,
+ "53": 10.16315,
+ "54": 10.13391,
+ "55": 10.10867,
+ "56": 10.13455,
+ "57": 10.13262,
+ "58": 10.12407,
+ "59": 10.06503,
+ "60": 10.09528,
+ "61": 10.04743,
+ "62": 10.01537,
+ "63": 10.08286,
+ "64": 10.03273,
+ "65": 9.99833,
+ "66": 10.03902,
+ "67": 10.01293,
+ "68": 9.97751,
+ "69": 9.99331,
+ "70": 9.97079,
+ "71": 9.99817,
+ "72": 9.97548,
+ "73": 9.95979,
+ "74": 9.95289,
+ "75": 9.91425,
+ "76": 9.9499,
+ "77": 9.94212,
+ "78": 9.89883,
+ "79": 9.89693,
+ "80": 9.91029,
+ "81": 9.93356,
+ "82": 9.88352,
+ "83": 9.83982,
+ "84": 9.78195,
+ "85": 9.76266,
+ "86": 9.87794,
+ "87": 9.90072,
+ "88": 9.87398,
+ "89": 9.82485,
+ "90": 9.81362,
+ "91": 9.8199,
+ "92": 9.81611,
+ "93": 9.74343,
+ "94": 9.82156,
+ "95": 9.8122,
+ "96": 9.79476,
+ "97": 9.74624,
+ "98": 9.76879,
+ "99": 9.81836,
+ "100": 9.7074
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 5566.0,
- "2": 5749.0,
- "3": 5881.0,
- "4": 5840.0,
- "5": 6476.0,
- "6": 6425.0,
- "7": 5900.0,
- "8": 5783.0,
- "9": 6426.0,
- "10": 5252.0,
- "11": 6722.0,
- "12": 6169.0,
- "13": 6556.0,
- "14": 6524.0,
- "15": 6116.0,
- "16": 6245.0,
- "17": 6139.0,
- "18": 5888.0,
- "19": 6375.0,
- "20": 5773.0,
- "21": 6188.0,
- "22": 5742.0,
- "23": 6768.0,
- "24": 6000.0,
- "25": 5852.0,
- "26": 6285.0,
- "27": 6357.0,
- "28": 6586.0,
- "29": 6742.0,
- "30": 6214.0,
- "31": 5775.0,
- "32": 6746.0,
- "33": 7205.0,
- "34": 6344.0,
- "35": 6686.0,
- "36": 6743.0,
- "37": 7281.0,
- "38": 7228.0,
- "39": 7810.0,
- "40": 7116.0,
- "41": 6902.0,
- "42": 7809.0,
- "43": 7110.0,
- "44": 7040.0,
- "45": 7058.0,
- "46": 7292.0,
- "47": 7813.0,
- "48": 7672.0,
- "49": 7601.0,
- "50": 7605.0,
- "51": 8105.0,
- "52": 7792.0,
- "53": 8870.0,
- "54": 8700.0,
- "55": 7685.0,
- "56": 7975.0,
- "57": 7544.0,
- "58": 8539.0,
- "59": 8275.0,
- "60": 7822.0,
- "61": 8316.0,
- "62": 8493.0,
- "63": 7748.0,
- "64": 8801.0,
- "65": 8269.0,
- "66": 9209.0,
- "67": 8382.0,
- "68": 8362.0,
- "69": 8644.0,
- "70": 9785.0,
- "71": 9060.0,
- "72": 8909.0,
- "73": 9217.0,
- "74": 6949.0,
- "75": 7960.0,
- "76": 8489.0,
- "77": 12484.0,
- "78": 9598.0,
- "79": 12984.0,
- "80": 11398.0,
- "81": 10221.0,
- "82": 9615.0,
- "83": 62741.0,
- "84": 9936.0,
- "85": 46541.0,
- "86": 8528.0,
- "87": 14916.0,
- "88": 9710.0,
- "89": 10273.0,
- "90": 11178.0,
- "91": 8856.0,
- "92": 9337.0,
- "93": 8404.0,
- "94": 9649.0,
- "95": 9657.0,
- "96": 13226.0,
- "97": 9093.0,
- "98": 10575.0,
- "99": 15320.0,
- "100": 9363.0
+ "1": 2589.0,
+ "2": 2610.0,
+ "3": 2532.0,
+ "4": 2530.0,
+ "5": 2535.0,
+ "6": 2504.0,
+ "7": 2664.0,
+ "8": 2529.0,
+ "9": 2641.0,
+ "10": 2550.0,
+ "11": 2654.0,
+ "12": 2438.0,
+ "13": 2617.0,
+ "14": 2645.0,
+ "15": 2328.0,
+ "16": 2493.0,
+ "17": 2550.0,
+ "18": 2599.0,
+ "19": 2441.0,
+ "20": 2491.0,
+ "21": 2583.0,
+ "22": 2562.0,
+ "23": 2470.0,
+ "24": 2588.0,
+ "25": 2439.0,
+ "26": 2535.0,
+ "27": 2589.0,
+ "28": 2534.0,
+ "29": 2637.0,
+ "30": 2716.0,
+ "31": 2705.0,
+ "32": 2812.0,
+ "33": 2835.0,
+ "34": 2727.0,
+ "35": 2870.0,
+ "36": 2698.0,
+ "37": 2921.0,
+ "38": 2783.0,
+ "39": 2848.0,
+ "40": 3037.0,
+ "41": 3154.0,
+ "42": 2864.0,
+ "43": 3103.0,
+ "44": 3123.0,
+ "45": 3271.0,
+ "46": 3208.0,
+ "47": 3206.0,
+ "48": 3309.0,
+ "49": 3457.0,
+ "50": 3466.0,
+ "51": 3276.0,
+ "52": 3448.0,
+ "53": 3254.0,
+ "54": 3504.0,
+ "55": 3230.0,
+ "56": 3568.0,
+ "57": 2933.0,
+ "58": 4052.0,
+ "59": 3626.0,
+ "60": 3510.0,
+ "61": 3371.0,
+ "62": 3642.0,
+ "63": 4019.0,
+ "64": 4041.0,
+ "65": 3371.0,
+ "66": 3826.0,
+ "67": 4156.0,
+ "68": 3811.0,
+ "69": 3545.0,
+ "70": 3831.0,
+ "71": 3834.0,
+ "72": 3593.0,
+ "73": 4098.0,
+ "74": 3711.0,
+ "75": 3649.0,
+ "76": 3907.0,
+ "77": 4118.0,
+ "78": 4212.0,
+ "79": 4428.0,
+ "80": 33291.0,
+ "81": 8226.0,
+ "82": 528724.0,
+ "83": 3499.0,
+ "84": 31529.0,
+ "85": 528713.0,
+ "86": 529264.0,
+ "87": 581775.0,
+ "88": 529230.0,
+ "89": 529270.0,
+ "90": 529149.0,
+ "91": 528757.0,
+ "92": 529091.0,
+ "93": 549748.0,
+ "94": 529131.0,
+ "95": 553058.0,
+ "96": 560607.0,
+ "97": 529708.0,
+ "98": 529488.0,
+ "99": 529121.0,
+ "100": 529245.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 628645888.0,
- "2": 628646912.0,
- "3": 628646912.0,
- "4": 628646912.0,
- "5": 628646912.0,
- "6": 628646912.0,
- "7": 628646912.0,
- "8": 628646912.0,
- "9": 628646912.0,
- "10": 628646912.0,
- "11": 628646912.0,
- "12": 628646912.0,
- "13": 628646912.0,
- "14": 628646912.0,
- "15": 628646912.0,
- "16": 628646912.0,
- "17": 628646912.0,
- "18": 628646912.0,
- "19": 628646912.0,
- "20": 628646912.0,
- "21": 628646912.0,
- "22": 628646912.0,
- "23": 628646912.0,
- "24": 628646912.0,
- "25": 628646912.0,
- "26": 628646912.0,
- "27": 628646912.0,
- "28": 628646912.0,
- "29": 628646912.0,
- "30": 628646912.0,
- "31": 628646912.0,
- "32": 628646912.0,
- "33": 628646912.0,
- "34": 628646912.0,
- "35": 628646912.0,
- "36": 628646912.0,
- "37": 628646912.0,
- "38": 628646912.0,
- "39": 628646912.0,
- "40": 628646912.0,
- "41": 628646912.0,
- "42": 628646912.0,
- "43": 628646912.0,
- "44": 628646912.0,
- "45": 628646912.0,
- "46": 628646912.0,
- "47": 628646912.0,
- "48": 628646912.0,
- "49": 628646912.0,
- "50": 628646912.0,
- "51": 628646912.0,
- "52": 628646912.0,
- "53": 628646912.0,
- "54": 628646912.0,
- "55": 628646912.0,
- "56": 628646912.0,
- "57": 628646912.0,
- "58": 628646912.0,
- "59": 628646912.0,
- "60": 628646912.0,
- "61": 628646912.0,
- "62": 628646912.0,
- "63": 628646912.0,
- "64": 628646912.0,
- "65": 628646912.0,
- "66": 628646912.0,
- "67": 628646912.0,
- "68": 628646912.0,
- "69": 628646912.0,
- "70": 628646912.0,
- "71": 628646912.0,
- "72": 628646912.0,
- "73": 628646912.0,
- "74": 628646912.0,
- "75": 628646912.0,
- "76": 628646912.0,
- "77": 628646912.0,
- "78": 628646912.0,
- "79": 628646912.0,
- "80": 628646912.0,
- "81": 628646912.0,
- "82": 628646912.0,
- "83": 628646912.0,
- "84": 628646912.0,
- "85": 628646912.0,
- "86": 628646912.0,
- "87": 628646912.0,
- "88": 628646912.0,
- "89": 628646912.0,
- "90": 628646912.0,
- "91": 628646912.0,
- "92": 628646912.0,
- "93": 628646912.0,
- "94": 628646912.0,
- "95": 628646912.0,
- "96": 628646912.0,
- "97": 628646912.0,
- "98": 628646912.0,
- "99": 628646912.0,
- "100": 628646912.0
+ "1": 628064256.0,
+ "2": 628065280.0,
+ "3": 628065280.0,
+ "4": 628065280.0,
+ "5": 628065280.0,
+ "6": 628065280.0,
+ "7": 628065280.0,
+ "8": 628065280.0,
+ "9": 628065280.0,
+ "10": 628065280.0,
+ "11": 628065280.0,
+ "12": 628065280.0,
+ "13": 628065280.0,
+ "14": 628065280.0,
+ "15": 628065280.0,
+ "16": 628065280.0,
+ "17": 628065280.0,
+ "18": 628065280.0,
+ "19": 628065280.0,
+ "20": 628065280.0,
+ "21": 628065280.0,
+ "22": 628065280.0,
+ "23": 628065280.0,
+ "24": 628065280.0,
+ "25": 628065280.0,
+ "26": 628065280.0,
+ "27": 628065280.0,
+ "28": 628065280.0,
+ "29": 628065280.0,
+ "30": 628065280.0,
+ "31": 628065280.0,
+ "32": 628065280.0,
+ "33": 628065280.0,
+ "34": 628065280.0,
+ "35": 628065280.0,
+ "36": 628065280.0,
+ "37": 628065280.0,
+ "38": 628065280.0,
+ "39": 628065280.0,
+ "40": 628065280.0,
+ "41": 628065280.0,
+ "42": 628065280.0,
+ "43": 628065280.0,
+ "44": 628065280.0,
+ "45": 628065280.0,
+ "46": 628065280.0,
+ "47": 628065280.0,
+ "48": 628065280.0,
+ "49": 628065280.0,
+ "50": 628065280.0,
+ "51": 628065280.0,
+ "52": 628065280.0,
+ "53": 628065280.0,
+ "54": 628065280.0,
+ "55": 628065280.0,
+ "56": 628065280.0,
+ "57": 628065280.0,
+ "58": 628065280.0,
+ "59": 628065280.0,
+ "60": 628065280.0,
+ "61": 628065280.0,
+ "62": 628065280.0,
+ "63": 628065280.0,
+ "64": 628065280.0,
+ "65": 628065280.0,
+ "66": 628065280.0,
+ "67": 628065280.0,
+ "68": 628065280.0,
+ "69": 628065280.0,
+ "70": 628065280.0,
+ "71": 628065280.0,
+ "72": 628065280.0,
+ "73": 628065280.0,
+ "74": 628065280.0,
+ "75": 628065280.0,
+ "76": 628065280.0,
+ "77": 628065280.0,
+ "78": 628065280.0,
+ "79": 628065280.0,
+ "80": 628065280.0,
+ "81": 628065280.0,
+ "82": 628065280.0,
+ "83": 628065280.0,
+ "84": 628065280.0,
+ "85": 628065280.0,
+ "86": 628065280.0,
+ "87": 628065280.0,
+ "88": 628065280.0,
+ "89": 628065280.0,
+ "90": 628065280.0,
+ "91": 628065280.0,
+ "92": 628065280.0,
+ "93": 628065280.0,
+ "94": 628065280.0,
+ "95": 628065280.0,
+ "96": 628065280.0,
+ "97": 628065280.0,
+ "98": 628065280.0,
+ "99": 628065280.0,
+ "100": 628065280.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 982203392.0,
- "2": 1149396992.0,
- "3": 1149396992.0,
- "4": 1155475456.0,
- "5": 1155475456.0,
- "6": 1155475456.0,
- "7": 1155475456.0,
- "8": 1155475456.0,
- "9": 1155475456.0,
- "10": 1155475456.0,
- "11": 1155475456.0,
- "12": 1155475456.0,
- "13": 1155475456.0,
- "14": 1155475456.0,
- "15": 1155475456.0,
- "16": 1155475456.0,
- "17": 1155475456.0,
- "18": 1155475456.0,
- "19": 1155475456.0,
- "20": 1155475456.0,
- "21": 1155475456.0,
- "22": 1155475456.0,
- "23": 1155475456.0,
- "24": 1155475456.0,
- "25": 1155475456.0,
- "26": 1155475456.0,
- "27": 1155475456.0,
- "28": 1155475456.0,
- "29": 1155475456.0,
- "30": 1155475456.0,
- "31": 1155475456.0,
- "32": 1155475456.0,
- "33": 1155475456.0,
- "34": 1155475456.0,
- "35": 1155475456.0,
- "36": 1155475456.0,
- "37": 1155475456.0,
- "38": 1155475456.0,
- "39": 1155475456.0,
- "40": 1155475456.0,
- "41": 1155475456.0,
- "42": 1155475456.0,
- "43": 1155475456.0,
- "44": 1155475456.0,
- "45": 1155475456.0,
- "46": 1155475456.0,
- "47": 1155475456.0,
- "48": 1155475456.0,
- "49": 1155475456.0,
- "50": 1155475456.0,
- "51": 1155475456.0,
- "52": 1155475456.0,
- "53": 1155475456.0,
- "54": 1155475456.0,
- "55": 1155475456.0,
- "56": 1155475456.0,
- "57": 1155475456.0,
- "58": 1155475456.0,
- "59": 1155475456.0,
- "60": 1155975680.0,
- "61": 1159303168.0,
- "62": 1159303168.0,
- "63": 1159303168.0,
- "64": 1159303168.0,
- "65": 1159303168.0,
- "66": 1159303168.0,
- "67": 1159303168.0,
- "68": 1159303168.0,
- "69": 1159303168.0,
- "70": 1159303168.0,
- "71": 1159303168.0,
- "72": 1159303168.0,
- "73": 1159303168.0,
- "74": 1159303168.0,
- "75": 1159303168.0,
- "76": 1164697088.0,
- "77": 1164697088.0,
- "78": 1164697088.0,
- "79": 1164697088.0,
- "80": 1164697088.0,
- "81": 1164697088.0,
- "82": 1164697088.0,
- "83": 1164697088.0,
- "84": 1164697088.0,
- "85": 1164697088.0,
- "86": 1164697088.0,
- "87": 1164697088.0,
- "88": 1164697088.0,
- "89": 1164697088.0,
- "90": 1164697088.0,
- "91": 1164697088.0,
- "92": 1164697088.0,
- "93": 1164697088.0,
- "94": 1164697088.0,
- "95": 1164697088.0,
- "96": 1164697088.0,
- "97": 1164697088.0,
- "98": 1164697088.0,
- "99": 1164697088.0,
- "100": 1164697088.0
+ "1": 974333952.0,
+ "2": 1142500864.0,
+ "3": 1142675968.0,
+ "4": 1147437056.0,
+ "5": 1147925504.0,
+ "6": 1147925504.0,
+ "7": 1148942336.0,
+ "8": 1148942336.0,
+ "9": 1148942336.0,
+ "10": 1148942336.0,
+ "11": 1148942336.0,
+ "12": 1148942336.0,
+ "13": 1148942336.0,
+ "14": 1148942336.0,
+ "15": 1148942336.0,
+ "16": 1148942336.0,
+ "17": 1148942336.0,
+ "18": 1148942336.0,
+ "19": 1148942336.0,
+ "20": 1148942336.0,
+ "21": 1148942336.0,
+ "22": 1148942336.0,
+ "23": 1148942336.0,
+ "24": 1148942336.0,
+ "25": 1148942336.0,
+ "26": 1149713920.0,
+ "27": 1149713920.0,
+ "28": 1149713920.0,
+ "29": 1149713920.0,
+ "30": 1149713920.0,
+ "31": 1149713920.0,
+ "32": 1149713920.0,
+ "33": 1149713920.0,
+ "34": 1149713920.0,
+ "35": 1149713920.0,
+ "36": 1149713920.0,
+ "37": 1149713920.0,
+ "38": 1149713920.0,
+ "39": 1149713920.0,
+ "40": 1149713920.0,
+ "41": 1149713920.0,
+ "42": 1149713920.0,
+ "43": 1149713920.0,
+ "44": 1149713920.0,
+ "45": 1149713920.0,
+ "46": 1149713920.0,
+ "47": 1149713920.0,
+ "48": 1149713920.0,
+ "49": 1149713920.0,
+ "50": 1149713920.0,
+ "51": 1149713920.0,
+ "52": 1149713920.0,
+ "53": 1149713920.0,
+ "54": 1149713920.0,
+ "55": 1149713920.0,
+ "56": 1149713920.0,
+ "57": 1149713920.0,
+ "58": 1149713920.0,
+ "59": 1149713920.0,
+ "60": 1149713920.0,
+ "61": 1149713920.0,
+ "62": 1149713920.0,
+ "63": 1149713920.0,
+ "64": 1149713920.0,
+ "65": 1149713920.0,
+ "66": 1149713920.0,
+ "67": 1149713920.0,
+ "68": 1149713920.0,
+ "69": 1149713920.0,
+ "70": 1149713920.0,
+ "71": 1149713920.0,
+ "72": 1149713920.0,
+ "73": 1149713920.0,
+ "74": 1149713920.0,
+ "75": 1149713920.0,
+ "76": 1149713920.0,
+ "77": 1149713920.0,
+ "78": 1149713920.0,
+ "79": 1149713920.0,
+ "80": 1149713920.0,
+ "81": 1149713920.0,
+ "82": 1149713920.0,
+ "83": 1149713920.0,
+ "84": 1149713920.0,
+ "85": 1149713920.0,
+ "86": 1149713920.0,
+ "87": 1149713920.0,
+ "88": 1149713920.0,
+ "89": 1149713920.0,
+ "90": 1149713920.0,
+ "91": 1149713920.0,
+ "92": 1149713920.0,
+ "93": 1149713920.0,
+ "94": 1149713920.0,
+ "95": 1149713920.0,
+ "96": 1149713920.0,
+ "97": 1149713920.0,
+ "98": 1149713920.0,
+ "99": 1149713920.0,
+ "100": 1149713920.0
}
},
"iteration-time": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 19.23269,
- "2": 0.72886,
- "3": 0.65505,
- "4": 0.57926,
- "5": 0.56473,
- "6": 0.56262,
- "7": 0.55541,
- "8": 0.55169,
- "9": 0.54588,
- "10": 0.54513,
- "11": 0.54209,
- "12": 0.55074,
- "13": 0.54861,
- "14": 0.54825,
- "15": 0.54517,
- "16": 0.54378,
- "17": 0.54038,
- "18": 0.53418,
- "19": 0.54272,
- "20": 0.53786,
- "21": 0.5453,
- "22": 0.53544,
- "23": 0.5385,
- "24": 0.5306,
- "25": 0.53752,
- "26": 0.53028,
- "27": 1.14331,
- "28": 0.55476,
- "29": 0.55192,
- "30": 0.53922,
- "31": 0.53776,
- "32": 0.53422,
- "33": 0.53153,
- "34": 0.53781,
- "35": 0.53428,
- "36": 0.5321,
- "37": 0.53103,
- "38": 0.53328,
- "39": 0.53189,
- "40": 1.26265,
- "41": 0.53531,
- "42": 0.53252,
- "43": 0.53665,
- "44": 0.88396,
- "45": 0.53586,
- "46": 0.89593,
- "47": 0.53907,
- "48": 0.5309,
- "49": 0.53767,
- "50": 0.53491,
- "51": 0.55263,
- "52": 0.53343,
- "53": 0.53673,
- "54": 0.53859,
- "55": 0.5329,
- "56": 0.52954,
- "57": 0.53085,
- "58": 0.53458,
- "59": 0.53132,
- "60": 0.53967,
- "61": 0.53205,
- "62": 0.53559,
- "63": 0.53393,
- "64": 0.53143,
- "65": 0.5339,
- "66": 0.53358,
- "67": 0.53117,
- "68": 0.53709,
- "69": 0.53768,
- "70": 0.53628,
- "71": 0.53275,
- "72": 0.54058,
- "73": 0.53091,
- "74": 0.53069,
- "75": 0.53307,
- "76": 0.53389,
- "77": 0.53403,
- "78": 0.53188,
- "79": 0.53173,
- "80": 0.532,
- "81": 0.53145,
- "82": 0.5358,
- "83": 0.53475,
- "84": 0.5323,
- "85": 0.54048,
- "86": 0.53766,
- "87": 0.53212,
- "88": 0.53119,
- "89": 0.53372,
- "90": 0.53371,
- "91": 0.53164,
- "92": 0.53327,
- "93": 0.54146,
- "94": 0.53517,
- "95": 0.53542,
- "96": 0.5306,
- "97": 0.53654,
- "98": 0.53425,
- "99": 0.53223,
- "100": 0.53446
+ "1": "nan",
+ "2": 11.7836,
+ "3": 0.58975,
+ "4": 0.56544,
+ "5": 0.5504,
+ "6": 0.56842,
+ "7": 0.5491,
+ "8": 0.54138,
+ "9": 0.53371,
+ "10": 0.5342,
+ "11": 0.53224,
+ "12": 0.52891,
+ "13": 0.52976,
+ "14": 0.53162,
+ "15": 0.52297,
+ "16": 0.52336,
+ "17": 0.52793,
+ "18": 0.52225,
+ "19": 0.52121,
+ "20": 0.52937,
+ "21": 0.53168,
+ "22": 0.52349,
+ "23": 0.52045,
+ "24": 0.53318,
+ "25": 0.52745,
+ "26": 0.51972,
+ "27": 0.52474,
+ "28": 0.53885,
+ "29": 0.54406,
+ "30": 0.52979,
+ "31": 0.52273,
+ "32": 0.52354,
+ "33": 0.52179,
+ "34": 0.52809,
+ "35": 0.52207,
+ "36": 0.52789,
+ "37": 0.51996,
+ "38": 0.53223,
+ "39": 0.52549,
+ "40": 0.53308,
+ "41": 0.53147,
+ "42": 0.53153,
+ "43": 0.5292,
+ "44": 0.52056,
+ "45": 0.52578,
+ "46": 0.51549,
+ "47": 0.51842,
+ "48": 0.51917,
+ "49": 0.52488,
+ "50": 0.52255,
+ "51": 0.64477,
+ "52": 0.51979,
+ "53": 0.52383,
+ "54": 0.52192,
+ "55": 0.51931,
+ "56": 0.51907,
+ "57": 0.52009,
+ "58": 0.51807,
+ "59": 0.51736,
+ "60": 0.51892,
+ "61": 0.51809,
+ "62": 0.52089,
+ "63": 0.52315,
+ "64": 0.51504,
+ "65": 0.51491,
+ "66": 0.51739,
+ "67": 0.51455,
+ "68": 0.51564,
+ "69": 1.04071,
+ "70": 0.5162,
+ "71": 0.51607,
+ "72": 0.5156,
+ "73": 0.51835,
+ "74": 0.51882,
+ "75": 0.52265,
+ "76": 0.51863,
+ "77": 0.51483,
+ "78": 0.51774,
+ "79": 0.52634,
+ "80": 0.52171,
+ "81": 0.52135,
+ "82": 0.52168,
+ "83": 0.53375,
+ "84": 0.51785,
+ "85": 0.52358,
+ "86": 0.51614,
+ "87": 0.52652,
+ "88": 0.51691,
+ "89": 0.51638,
+ "90": 0.52191,
+ "91": 0.51655,
+ "92": 0.51846,
+ "93": 0.51379,
+ "94": 0.51835,
+ "95": 0.91609,
+ "96": 0.51869,
+ "97": 0.51813,
+ "98": 0.5255,
+ "99": 0.52418,
+ "100": 0.53762
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer/model_config.yaml
index fb438f0edda..7c292c0fa53 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer/model_config.yaml
@@ -22,9 +22,9 @@ MODEL_ARGS:
--lr-decay-iters: 320000
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--split: 949,50,1
--distributed-backend: nccl
--lr: 0.00015
@@ -64,4 +64,6 @@ MODEL_ARGS:
--bf16: true
--no-bias-gelu-fusion: true
--log-memory-to-tensorboard: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph/golden_values_dev_dgx_gb200.json
index 8710366a4a2..276cac09c4c 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph/golden_values_dev_dgx_gb200.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph/golden_values_dev_dgx_gb200.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.94839,
- "2": 10.94024,
- "3": 10.95902,
- "4": 10.9592,
- "5": 10.93942,
- "6": 10.95284,
- "7": 10.95227,
- "8": 10.93987,
- "9": 10.94518,
- "10": 10.94146,
- "11": 10.94366,
- "12": 10.93351,
- "13": 10.92937,
- "14": 10.93117,
- "15": 10.87714,
- "16": 10.88218,
- "17": 10.87388,
- "18": 10.86829,
- "19": 10.86292,
- "20": 10.78627,
- "21": 10.73278,
- "22": 10.62202,
- "23": 10.72355,
- "24": 10.61784,
- "25": 10.54739,
- "26": 10.64163,
- "27": 10.63354,
- "28": 10.59007,
- "29": 10.59937,
- "30": 10.36921,
- "31": 10.1175,
- "32": 10.457,
- "33": 10.45238,
- "34": 10.18943,
- "35": 10.24409,
- "36": 10.20779,
- "37": 10.32099,
- "38": 10.17141,
- "39": 10.39579,
- "40": 10.03318,
- "41": 10.08573,
- "42": 10.17487,
- "43": 9.7274,
- "44": 9.88257,
- "45": 9.73978,
- "46": 9.72104,
- "47": 10.08354,
- "48": 9.75251,
- "49": 9.39373,
- "50": 9.83765,
- "51": 9.76236,
- "52": 9.65444,
- "53": 10.01594,
- "54": 9.86969,
- "55": 9.79645,
- "56": 9.53492,
- "57": 9.365,
- "58": 9.75243,
+ "1": 10.94823,
+ "2": 10.94029,
+ "3": 10.95946,
+ "4": 10.95899,
+ "5": 10.93916,
+ "6": 10.95312,
+ "7": 10.95182,
+ "8": 10.94003,
+ "9": 10.94534,
+ "10": 10.94178,
+ "11": 10.94374,
+ "12": 10.93321,
+ "13": 10.92928,
+ "14": 10.93148,
+ "15": 10.87634,
+ "16": 10.88195,
+ "17": 10.87395,
+ "18": 10.86824,
+ "19": 10.8629,
+ "20": 10.78665,
+ "21": 10.73239,
+ "22": 10.6222,
+ "23": 10.72379,
+ "24": 10.61778,
+ "25": 10.54746,
+ "26": 10.64145,
+ "27": 10.63367,
+ "28": 10.59017,
+ "29": 10.59921,
+ "30": 10.36893,
+ "31": 10.11592,
+ "32": 10.45661,
+ "33": 10.45165,
+ "34": 10.18921,
+ "35": 10.24412,
+ "36": 10.20666,
+ "37": 10.32105,
+ "38": 10.1707,
+ "39": 10.39608,
+ "40": 10.03234,
+ "41": 10.0854,
+ "42": 10.17417,
+ "43": 9.7273,
+ "44": 9.88168,
+ "45": 9.73882,
+ "46": 9.71966,
+ "47": 10.0834,
+ "48": 9.75148,
+ "49": 9.39201,
+ "50": 9.83775,
+ "51": 9.7615,
+ "52": 9.6538,
+ "53": 10.01499,
+ "54": 9.86895,
+ "55": 9.79434,
+ "56": 9.5337,
+ "57": 9.36466,
+ "58": 9.75179,
"59": 9.4751,
- "60": 9.40362,
- "61": 9.59124,
- "62": 9.91012,
- "63": 9.24082,
- "64": 9.67728,
- "65": 8.79731,
- "66": 9.60544,
- "67": 9.24212,
- "68": 9.70475,
- "69": 9.70741,
- "70": 9.65988,
- "71": 9.50626,
- "72": 9.45834,
- "73": 9.38692,
- "74": 8.79461,
- "75": 9.32175,
- "76": 8.92857,
- "77": 9.99456,
- "78": 9.63104,
- "79": 9.26692,
- "80": 9.29144,
- "81": 9.37768,
- "82": 9.60984,
- "83": 9.21108,
- "84": 9.33667,
- "85": 9.52726,
- "86": 8.94539,
- "87": 9.49937,
- "88": 9.67766,
- "89": 9.49525,
- "90": 9.7509,
- "91": 9.22918,
- "92": 9.25394,
- "93": 8.96194,
- "94": 8.69021,
- "95": 9.43531,
+ "60": 9.40298,
+ "61": 9.59096,
+ "62": 9.90944,
+ "63": 9.23985,
+ "64": 9.6767,
+ "65": 8.79646,
+ "66": 9.6053,
+ "67": 9.24174,
+ "68": 9.70452,
+ "69": 9.70788,
+ "70": 9.65854,
+ "71": 9.50558,
+ "72": 9.45721,
+ "73": 9.38616,
+ "74": 8.79353,
+ "75": 9.32133,
+ "76": 8.92749,
+ "77": 9.99436,
+ "78": 9.6307,
+ "79": 9.26666,
+ "80": 9.29128,
+ "81": 9.37824,
+ "82": 9.60985,
+ "83": 9.21132,
+ "84": 9.33743,
+ "85": 9.52734,
+ "86": 8.94621,
+ "87": 9.49881,
+ "88": 9.6778,
+ "89": 9.49498,
+ "90": 9.75072,
+ "91": 9.22928,
+ "92": 9.25387,
+ "93": 8.96179,
+ "94": 8.68904,
+ "95": 9.43582,
"96": 9.39886,
- "97": 9.19199,
- "98": 9.57248,
- "99": 8.75688,
- "100": 9.2924
+ "97": 9.19176,
+ "98": 9.57251,
+ "99": 8.75773,
+ "100": 9.29251
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 22750392.0,
- "2": 22953048.0,
- "3": 22604332.0,
- "4": 23266194.0,
- "5": 22735592.0,
- "6": 23061740.0,
- "7": 22793278.0,
- "8": 22960884.0,
- "9": 22865532.0,
- "10": 22950250.0,
- "11": 22499592.0,
- "12": 22455936.0,
- "13": 22948014.0,
- "14": 22384528.0,
- "15": 22846172.0,
- "16": 22856720.0,
- "17": 22836352.0,
- "18": 22590198.0,
- "19": 22627006.0,
- "20": 22712338.0,
- "21": 22762590.0,
- "22": 22816896.0,
- "23": 22545280.0,
- "24": 22794372.0,
- "25": 22841964.0,
- "26": 22549700.0,
- "27": 22464724.0,
- "28": 22453634.0,
- "29": 22534620.0,
- "30": 22636106.0,
- "31": 22989484.0,
- "32": 22593994.0,
- "33": 22565948.0,
- "34": 22855396.0,
- "35": 22813606.0,
- "36": 22595412.0,
- "37": 22499234.0,
- "38": 22926180.0,
- "39": 22825052.0,
- "40": 22675880.0,
- "41": 22671624.0,
- "42": 22682188.0,
- "43": 23015228.0,
- "44": 22766040.0,
- "45": 22679588.0,
- "46": 22915144.0,
- "47": 22642744.0,
- "48": 24003236.0,
- "49": 23786618.0,
- "50": 22931756.0,
- "51": 23866290.0,
- "52": 23807188.0,
- "53": 24007482.0,
- "54": 23916892.0,
- "55": 23571308.0,
- "56": 23954192.0,
- "57": 24211600.0,
- "58": 23914524.0,
- "59": 23771900.0,
- "60": 23813638.0,
- "61": 23795512.0,
- "62": 23739412.0,
- "63": 23917700.0,
- "64": 23895780.0,
- "65": 24147262.0,
- "66": 23794750.0,
- "67": 23983810.0,
- "68": 23674060.0,
- "69": 23647770.0,
- "70": 23907338.0,
- "71": 23818256.0,
- "72": 23723392.0,
- "73": 22754048.0,
- "74": 25181258.0,
- "75": 24144968.0,
- "76": 23976372.0,
- "77": 22260516.0,
- "78": 23862138.0,
- "79": 23806872.0,
- "80": 23773052.0,
- "81": 25020468.0,
- "82": 22812998.0,
- "83": 23911992.0,
- "84": 25144028.0,
- "85": 22725432.0,
- "86": 24205484.0,
- "87": 24851672.0,
- "88": 23700260.0,
- "89": 22505492.0,
- "90": 24062928.0,
- "91": 22790310.0,
- "92": 24923596.0,
- "93": 23722104.0,
- "94": 23993086.0,
- "95": 24140048.0,
- "96": 23909352.0,
- "97": 23668280.0,
- "98": 23832272.0,
- "99": 23985032.0,
- "100": 24101560.0
+ "1": 22749512.0,
+ "2": 22952264.0,
+ "3": 22603510.0,
+ "4": 23265460.0,
+ "5": 22734732.0,
+ "6": 23060800.0,
+ "7": 22792468.0,
+ "8": 22959968.0,
+ "9": 22864640.0,
+ "10": 22949520.0,
+ "11": 22498684.0,
+ "12": 22455028.0,
+ "13": 22947128.0,
+ "14": 22383502.0,
+ "15": 22845306.0,
+ "16": 22855928.0,
+ "17": 22835556.0,
+ "18": 22589220.0,
+ "19": 22626116.0,
+ "20": 22711586.0,
+ "21": 22761740.0,
+ "22": 22816042.0,
+ "23": 22544336.0,
+ "24": 22793596.0,
+ "25": 22841044.0,
+ "26": 22548852.0,
+ "27": 22463864.0,
+ "28": 22452704.0,
+ "29": 22533576.0,
+ "30": 22635286.0,
+ "31": 22988692.0,
+ "32": 22593012.0,
+ "33": 22564960.0,
+ "34": 22854646.0,
+ "35": 22812672.0,
+ "36": 22594522.0,
+ "37": 22498416.0,
+ "38": 22925300.0,
+ "39": 22823992.0,
+ "40": 22674558.0,
+ "41": 22670432.0,
+ "42": 22680698.0,
+ "43": 23012430.0,
+ "44": 22763660.0,
+ "45": 22678132.0,
+ "46": 22910538.0,
+ "47": 22641894.0,
+ "48": 22953708.0,
+ "49": 22737140.0,
+ "50": 22929746.0,
+ "51": 22816672.0,
+ "52": 22757622.0,
+ "53": 22957884.0,
+ "54": 22866818.0,
+ "55": 22521728.0,
+ "56": 22904628.0,
+ "57": 23161548.0,
+ "58": 22864764.0,
+ "59": 22722182.0,
+ "60": 22763942.0,
+ "61": 22745650.0,
+ "62": 22689690.0,
+ "63": 22867000.0,
+ "64": 22845000.0,
+ "65": 23096868.0,
+ "66": 22744656.0,
+ "67": 23457800.0,
+ "68": 22620324.0,
+ "69": 23117000.0,
+ "70": 22853792.0,
+ "71": 22765820.0,
+ "72": 22667832.0,
+ "73": 22752016.0,
+ "74": 23082700.0,
+ "75": 23092680.0,
+ "76": 22926860.0,
+ "77": 22258704.0,
+ "78": 22810444.0,
+ "79": 22756700.0,
+ "80": 22722362.0,
+ "81": 22921504.0,
+ "82": 22800732.0,
+ "83": 22862068.0,
+ "84": 23045372.0,
+ "85": 22723984.0,
+ "86": 23151212.0,
+ "87": 22751020.0,
+ "88": 22649804.0,
+ "89": 22501632.0,
+ "90": 23007442.0,
+ "91": 22787770.0,
+ "92": 22825144.0,
+ "93": 22670100.0,
+ "94": 22942448.0,
+ "95": 23089812.0,
+ "96": 22858912.0,
+ "97": 22614298.0,
+ "98": 22782738.0,
+ "99": 22931148.0,
+ "100": 23051358.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 810140160.0,
- "2": 804531200.0,
- "3": 804531200.0,
- "4": 934860800.0,
- "5": 934860800.0,
- "6": 934860800.0,
- "7": 934860800.0,
- "8": 934860800.0,
- "9": 938611712.0,
- "10": 938104832.0,
- "11": 938379264.0,
- "12": 934860800.0,
- "13": 934860800.0,
- "14": 934860800.0,
- "15": 934860800.0,
- "16": 942249984.0,
- "17": 941443072.0,
- "18": 937990144.0,
- "19": 937548800.0,
- "20": 937498624.0,
- "21": 934860800.0,
- "22": 934860800.0,
- "23": 941533184.0,
- "24": 942114816.0,
- "25": 942398464.0,
- "26": 934860800.0,
- "27": 934860800.0,
- "28": 934860800.0,
- "29": 934860800.0,
- "30": 934860800.0,
- "31": 934860800.0,
- "32": 934860800.0,
- "33": 934860800.0,
- "34": 941477888.0,
- "35": 934860800.0,
- "36": 934860800.0,
- "37": 934860800.0,
- "38": 934860800.0,
- "39": 934860800.0,
- "40": 934860800.0,
- "41": 940742656.0,
- "42": 940742656.0,
- "43": 940742656.0,
- "44": 940968960.0,
- "45": 941581312.0,
- "46": 934860800.0,
- "47": 934860800.0,
- "48": 940742656.0,
- "49": 934860800.0,
- "50": 934860800.0,
- "51": 934860800.0,
- "52": 940742656.0,
- "53": 937498624.0,
- "54": 937498624.0,
- "55": 937498624.0,
- "56": 937498624.0,
- "57": 938199040.0,
- "58": 937498624.0,
- "59": 937498624.0,
- "60": 940742656.0,
- "61": 934860800.0,
- "62": 934860800.0,
- "63": 934860800.0,
- "64": 934860800.0,
- "65": 934860800.0,
- "66": 934860800.0,
- "67": 934860800.0,
- "68": 934860800.0,
- "69": 934860800.0,
- "70": 934860800.0,
- "71": 934860800.0,
- "72": 934860800.0,
- "73": 934860800.0,
- "74": 934860800.0,
- "75": 934860800.0,
- "76": 934860800.0,
- "77": 934860800.0,
- "78": 934860800.0,
- "79": 938199040.0,
- "80": 938199040.0,
- "81": 937498624.0,
- "82": 938061824.0,
- "83": 938412032.0,
- "84": 937498624.0,
- "85": 938199040.0,
- "86": 938445824.0,
- "87": 937498624.0,
- "88": 937498624.0,
- "89": 934860800.0,
- "90": 934860800.0,
- "91": 934860800.0,
- "92": 940742656.0,
- "93": 940742656.0,
- "94": 938104832.0,
- "95": 941451264.0,
- "96": 940742656.0,
- "97": 941542400.0,
- "98": 938104832.0,
- "99": 940742656.0,
- "100": 938104832.0
+ "1": 799941120.0,
+ "2": 815729664.0,
+ "3": 772426752.0,
+ "4": 800618496.0,
+ "5": 806733824.0,
+ "6": 800618496.0,
+ "7": 804676608.0,
+ "8": 800618496.0,
+ "9": 800618496.0,
+ "10": 800618496.0,
+ "11": 800618496.0,
+ "12": 806500352.0,
+ "13": 800618496.0,
+ "14": 808204288.0,
+ "15": 800618496.0,
+ "16": 804746240.0,
+ "17": 800618496.0,
+ "18": 803534848.0,
+ "19": 800618496.0,
+ "20": 804014080.0,
+ "21": 800618496.0,
+ "22": 800618496.0,
+ "23": 807139328.0,
+ "24": 800618496.0,
+ "25": 803256320.0,
+ "26": 800618496.0,
+ "27": 803256320.0,
+ "28": 807279616.0,
+ "29": 800618496.0,
+ "30": 804206592.0,
+ "31": 800618496.0,
+ "32": 803477504.0,
+ "33": 800618496.0,
+ "34": 800618496.0,
+ "35": 804508672.0,
+ "36": 800618496.0,
+ "37": 803256320.0,
+ "38": 800618496.0,
+ "39": 800618496.0,
+ "40": 807130112.0,
+ "41": 800618496.0,
+ "42": 806517760.0,
+ "43": 800618496.0,
+ "44": 803862528.0,
+ "45": 800618496.0,
+ "46": 803272704.0,
+ "47": 800618496.0,
+ "48": 800618496.0,
+ "49": 806746112.0,
+ "50": 800618496.0,
+ "51": 803256320.0,
+ "52": 806500352.0,
+ "53": 800618496.0,
+ "54": 806500352.0,
+ "55": 800618496.0,
+ "56": 806500352.0,
+ "57": 800618496.0,
+ "58": 806942720.0,
+ "59": 800618496.0,
+ "60": 806500352.0,
+ "61": 800618496.0,
+ "62": 806900736.0,
+ "63": 800618496.0,
+ "64": 800618496.0,
+ "65": 800618496.0,
+ "66": 800618496.0,
+ "67": 800618496.0,
+ "68": 806500352.0,
+ "69": 800618496.0,
+ "70": 806835200.0,
+ "71": 800618496.0,
+ "72": 806500352.0,
+ "73": 800618496.0,
+ "74": 800618496.0,
+ "75": 800618496.0,
+ "76": 806500352.0,
+ "77": 800618496.0,
+ "78": 803862528.0,
+ "79": 800618496.0,
+ "80": 806500352.0,
+ "81": 800618496.0,
+ "82": 807198720.0,
+ "83": 800618496.0,
+ "84": 806500352.0,
+ "85": 800618496.0,
+ "86": 806500352.0,
+ "87": 800618496.0,
+ "88": 803862528.0,
+ "89": 800618496.0,
+ "90": 803862528.0,
+ "91": 800618496.0,
+ "92": 803256320.0,
+ "93": 800618496.0,
+ "94": 803689472.0,
+ "95": 800618496.0,
+ "96": 803771392.0,
+ "97": 800618496.0,
+ "98": 803256320.0,
+ "99": 806500352.0,
+ "100": 800618496.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 1017976320.0,
- "2": 1226964480.0,
- "3": 1228012032.0,
- "4": 1300063744.0,
- "5": 1300063744.0,
- "6": 1300223488.0,
- "7": 1300891648.0,
- "8": 1300891648.0,
- "9": 1300891648.0,
- "10": 1303292416.0,
- "11": 1303292416.0,
- "12": 1303292416.0,
- "13": 1303292416.0,
- "14": 1303292416.0,
- "15": 1303292416.0,
- "16": 1303292416.0,
- "17": 1303292416.0,
- "18": 1303292416.0,
- "19": 1303292416.0,
- "20": 1303292416.0,
- "21": 1303292416.0,
- "22": 1303292416.0,
- "23": 1303292416.0,
- "24": 1303292416.0,
- "25": 1303292416.0,
- "26": 1303292416.0,
- "27": 1303292416.0,
- "28": 1303292416.0,
- "29": 1303292416.0,
- "30": 1303292416.0,
- "31": 1303292416.0,
- "32": 1303292416.0,
- "33": 1303292416.0,
- "34": 1303292416.0,
- "35": 1303292416.0,
- "36": 1303292416.0,
- "37": 1303292416.0,
- "38": 1303292416.0,
- "39": 1303292416.0,
- "40": 1303292416.0,
- "41": 1303292416.0,
- "42": 1303292416.0,
- "43": 1303292416.0,
- "44": 1303292416.0,
- "45": 1303292416.0,
- "46": 1303292416.0,
- "47": 1303292416.0,
- "48": 1303292416.0,
- "49": 1303292416.0,
- "50": 1303292416.0,
- "51": 1303292416.0,
- "52": 1303292416.0,
- "53": 1303292416.0,
- "54": 1303292416.0,
- "55": 1303292416.0,
- "56": 1303292416.0,
- "57": 1303292416.0,
- "58": 1303292416.0,
- "59": 1303292416.0,
- "60": 1303292416.0,
- "61": 1303292416.0,
- "62": 1303292416.0,
- "63": 1303292416.0,
- "64": 1303292416.0,
- "65": 1303292416.0,
- "66": 1303292416.0,
- "67": 1303292416.0,
- "68": 1303292416.0,
- "69": 1303292416.0,
- "70": 1303292416.0,
- "71": 1303292416.0,
- "72": 1303292416.0,
- "73": 1303292416.0,
- "74": 1303292416.0,
- "75": 1303292416.0,
- "76": 1303292416.0,
- "77": 1303292416.0,
- "78": 1303292416.0,
- "79": 1303292416.0,
- "80": 1303292416.0,
- "81": 1303292416.0,
- "82": 1303292416.0,
- "83": 1303292416.0,
- "84": 1303292416.0,
- "85": 1303292416.0,
- "86": 1303292416.0,
- "87": 1303292416.0,
- "88": 1303292416.0,
- "89": 1303292416.0,
- "90": 1303292416.0,
- "91": 1303292416.0,
- "92": 1303292416.0,
- "93": 1303292416.0,
- "94": 1303292416.0,
- "95": 1303292416.0,
- "96": 1303292416.0,
- "97": 1303292416.0,
- "98": 1303292416.0,
- "99": 1303292416.0,
- "100": 1303292416.0
+ "1": 951062016.0,
+ "2": 1174839296.0,
+ "3": 1174839296.0,
+ "4": 1174839296.0,
+ "5": 1174839296.0,
+ "6": 1174839296.0,
+ "7": 1174839296.0,
+ "8": 1174839296.0,
+ "9": 1174839296.0,
+ "10": 1174839296.0,
+ "11": 1174839296.0,
+ "12": 1174839296.0,
+ "13": 1174839296.0,
+ "14": 1174839296.0,
+ "15": 1174839296.0,
+ "16": 1174839296.0,
+ "17": 1174839296.0,
+ "18": 1174839296.0,
+ "19": 1174839296.0,
+ "20": 1174839296.0,
+ "21": 1174839296.0,
+ "22": 1174839296.0,
+ "23": 1174839296.0,
+ "24": 1174839296.0,
+ "25": 1174839296.0,
+ "26": 1174839296.0,
+ "27": 1174839296.0,
+ "28": 1174839296.0,
+ "29": 1174839296.0,
+ "30": 1174839296.0,
+ "31": 1174839296.0,
+ "32": 1174839296.0,
+ "33": 1174839296.0,
+ "34": 1174839296.0,
+ "35": 1174839296.0,
+ "36": 1174839296.0,
+ "37": 1174839296.0,
+ "38": 1174839296.0,
+ "39": 1174839296.0,
+ "40": 1174839296.0,
+ "41": 1174839296.0,
+ "42": 1174839296.0,
+ "43": 1174839296.0,
+ "44": 1174839296.0,
+ "45": 1174839296.0,
+ "46": 1174839296.0,
+ "47": 1174839296.0,
+ "48": 1174839296.0,
+ "49": 1174839296.0,
+ "50": 1174839296.0,
+ "51": 1174839296.0,
+ "52": 1174839296.0,
+ "53": 1174839296.0,
+ "54": 1174839296.0,
+ "55": 1174839296.0,
+ "56": 1174839296.0,
+ "57": 1174839296.0,
+ "58": 1174839296.0,
+ "59": 1174839296.0,
+ "60": 1174839296.0,
+ "61": 1174839296.0,
+ "62": 1174839296.0,
+ "63": 1174839296.0,
+ "64": 1174839296.0,
+ "65": 1174839296.0,
+ "66": 1174839296.0,
+ "67": 1174839296.0,
+ "68": 1174839296.0,
+ "69": 1174839296.0,
+ "70": 1174839296.0,
+ "71": 1174839296.0,
+ "72": 1174839296.0,
+ "73": 1174839296.0,
+ "74": 1174839296.0,
+ "75": 1174839296.0,
+ "76": 1174839296.0,
+ "77": 1174839296.0,
+ "78": 1174839296.0,
+ "79": 1174839296.0,
+ "80": 1174839296.0,
+ "81": 1174839296.0,
+ "82": 1174839296.0,
+ "83": 1174839296.0,
+ "84": 1174839296.0,
+ "85": 1174839296.0,
+ "86": 1174839296.0,
+ "87": 1174839296.0,
+ "88": 1174839296.0,
+ "89": 1174839296.0,
+ "90": 1174839296.0,
+ "91": 1174839296.0,
+ "92": 1174839296.0,
+ "93": 1174839296.0,
+ "94": 1174839296.0,
+ "95": 1174839296.0,
+ "96": 1174839296.0,
+ "97": 1174839296.0,
+ "98": 1174839296.0,
+ "99": 1174839296.0,
+ "100": 1174839296.0
}
},
"mtp_1 loss": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.89507,
- "2": 10.90521,
- "3": 10.90879,
- "4": 10.86406,
- "5": 10.91765,
- "6": 10.92332,
- "7": 10.90072,
- "8": 10.8906,
- "9": 10.90544,
- "10": 10.88636,
- "11": 10.93328,
- "12": 10.91582,
- "13": 10.90917,
- "14": 10.92294,
- "15": 10.89802,
- "16": 10.90337,
- "17": 10.88446,
- "18": 10.90526,
- "19": 10.90011,
- "20": 10.88775,
- "21": 10.88103,
- "22": 10.85514,
- "23": 10.89267,
+ "1": 10.89522,
+ "2": 10.90551,
+ "3": 10.90887,
+ "4": 10.86336,
+ "5": 10.9179,
+ "6": 10.92305,
+ "7": 10.9008,
+ "8": 10.8905,
+ "9": 10.9057,
+ "10": 10.88654,
+ "11": 10.93319,
+ "12": 10.91595,
+ "13": 10.9095,
+ "14": 10.92285,
+ "15": 10.89817,
+ "16": 10.90358,
+ "17": 10.88413,
+ "18": 10.90616,
+ "19": 10.90057,
+ "20": 10.8873,
+ "21": 10.88138,
+ "22": 10.85462,
+ "23": 10.89256,
"24": 10.87352,
- "25": 10.86182,
- "26": 10.87152,
- "27": 10.88847,
- "28": 10.87872,
- "29": 10.88744,
- "30": 10.87297,
- "31": 10.80177,
- "32": 10.8732,
- "33": 10.88219,
- "34": 10.83823,
- "35": 10.85291,
- "36": 10.84901,
- "37": 10.85873,
- "38": 10.83148,
- "39": 10.86289,
- "40": 10.82147,
- "41": 10.82913,
- "42": 10.84798,
- "43": 10.7908,
- "44": 10.81431,
+ "25": 10.86146,
+ "26": 10.87165,
+ "27": 10.88872,
+ "28": 10.87761,
+ "29": 10.88794,
+ "30": 10.87308,
+ "31": 10.80257,
+ "32": 10.87337,
+ "33": 10.88204,
+ "34": 10.83935,
+ "35": 10.85313,
+ "36": 10.84931,
+ "37": 10.8586,
+ "38": 10.83164,
+ "39": 10.8625,
+ "40": 10.82188,
+ "41": 10.82894,
+ "42": 10.8482,
+ "43": 10.78992,
+ "44": 10.81419,
"45": 10.7879,
- "46": 10.78018,
- "47": 10.83142,
- "48": 10.78854,
+ "46": 10.78117,
+ "47": 10.83138,
+ "48": 10.78829,
"49": 10.71024,
- "50": 10.76861,
- "51": 10.76331,
- "52": 10.73945,
- "53": 10.80126,
- "54": 10.77704,
- "55": 10.765,
- "56": 10.71649,
- "57": 10.67368,
- "58": 10.75107,
- "59": 10.69607,
- "60": 10.66236,
- "61": 10.69617,
- "62": 10.77167,
- "63": 10.6134,
- "64": 10.70881,
- "65": 10.49259,
- "66": 10.66843,
- "67": 10.58084,
- "68": 10.68215,
- "69": 10.68669,
- "70": 10.67296,
- "71": 10.64397,
- "72": 10.60997,
- "73": 10.56734,
- "74": 10.38624,
- "75": 10.53623,
- "76": 10.40297,
- "77": 10.75436,
- "78": 10.62548,
- "79": 10.47858,
- "80": 10.47388,
- "81": 10.5143,
- "82": 10.58579,
- "83": 10.43913,
- "84": 10.45418,
- "85": 10.55042,
- "86": 10.27831,
- "87": 10.51067,
- "88": 10.60469,
- "89": 10.5084,
+ "50": 10.76834,
+ "51": 10.76311,
+ "52": 10.73835,
+ "53": 10.80136,
+ "54": 10.77718,
+ "55": 10.76627,
+ "56": 10.71565,
+ "57": 10.67344,
+ "58": 10.75061,
+ "59": 10.69371,
+ "60": 10.66179,
+ "61": 10.69551,
+ "62": 10.77138,
+ "63": 10.61352,
+ "64": 10.70862,
+ "65": 10.48942,
+ "66": 10.66858,
+ "67": 10.58137,
+ "68": 10.68262,
+ "69": 10.68673,
+ "70": 10.67341,
+ "71": 10.64412,
+ "72": 10.60995,
+ "73": 10.5677,
+ "74": 10.38542,
+ "75": 10.53645,
+ "76": 10.40293,
+ "77": 10.75408,
+ "78": 10.6251,
+ "79": 10.47772,
+ "80": 10.47363,
+ "81": 10.51342,
+ "82": 10.58545,
+ "83": 10.43903,
+ "84": 10.45468,
+ "85": 10.55052,
+ "86": 10.27895,
+ "87": 10.5113,
+ "88": 10.60504,
+ "89": 10.50829,
"90": 10.60243,
- "91": 10.38487,
- "92": 10.38165,
- "93": 10.23549,
- "94": 10.07844,
- "95": 10.42709,
- "96": 10.44697,
- "97": 10.31686,
- "98": 10.4968,
- "99": 10.04966,
- "100": 10.32944
+ "91": 10.38582,
+ "92": 10.38235,
+ "93": 10.23694,
+ "94": 10.07914,
+ "95": 10.42647,
+ "96": 10.448,
+ "97": 10.31682,
+ "98": 10.49729,
+ "99": 10.05067,
+ "100": 10.32971
}
},
"iteration-time": {
@@ -540,105 +540,105 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 19.93451,
- "3": 2.31445,
- "4": 5.28856,
- "5": 1.09994,
- "6": 1.09399,
- "7": 1.09697,
- "8": 1.09872,
- "9": 1.17005,
- "10": 1.10071,
- "11": 1.0994,
- "12": 1.08313,
- "13": 1.09364,
- "14": 1.09082,
- "15": 1.09269,
- "16": 1.08133,
- "17": 1.08872,
- "18": 1.09032,
- "19": 1.10458,
- "20": 1.10126,
- "21": 1.09029,
- "22": 1.19723,
- "23": 1.36303,
- "24": 1.39758,
- "25": 1.40863,
- "26": 1.40985,
- "27": 1.40231,
- "28": 1.42816,
- "29": 1.37678,
- "30": 1.40545,
- "31": 1.40841,
- "32": 1.40385,
- "33": 1.39528,
- "34": 1.4028,
- "35": 1.41768,
- "36": 1.40649,
- "37": 1.41813,
- "38": 1.40674,
- "39": 1.38881,
- "40": 1.40998,
- "41": 1.37775,
- "42": 1.39701,
- "43": 1.3967,
- "44": 1.40408,
- "45": 1.40972,
- "46": 1.41116,
- "47": 1.40427,
- "48": 1.38905,
- "49": 1.42541,
- "50": 1.40474,
- "51": 1.40708,
- "52": 1.37484,
- "53": 1.38539,
- "54": 1.39988,
- "55": 1.39635,
- "56": 1.41326,
- "57": 1.40012,
- "58": 1.40599,
- "59": 1.41023,
- "60": 1.4209,
- "61": 1.41702,
- "62": 1.40134,
- "63": 1.40282,
- "64": 1.40573,
- "65": 1.41933,
- "66": 1.40057,
- "67": 1.41526,
- "68": 1.40285,
- "69": 1.41947,
- "70": 1.37747,
- "71": 1.41206,
- "72": 1.39123,
- "73": 1.42381,
- "74": 1.40806,
- "75": 1.40032,
- "76": 1.41783,
- "77": 1.39133,
- "78": 1.41146,
- "79": 1.42648,
- "80": 1.40774,
- "81": 1.40046,
- "82": 1.39158,
- "83": 1.4079,
- "84": 1.40469,
- "85": 1.39689,
- "86": 1.41401,
- "87": 1.40637,
- "88": 1.40569,
- "89": 1.45225,
- "90": 1.39469,
- "91": 1.39677,
- "92": 1.39569,
- "93": 1.38882,
- "94": 1.40133,
- "95": 1.41493,
- "96": 1.40659,
- "97": 1.39059,
- "98": 1.40044,
- "99": 1.41118,
- "100": 1.39159
+ "2": 37.14217,
+ "3": 2.34339,
+ "4": 6.33524,
+ "5": 1.26014,
+ "6": 1.24134,
+ "7": 1.25214,
+ "8": 1.2438,
+ "9": 1.23853,
+ "10": 1.26468,
+ "11": 1.24129,
+ "12": 1.24649,
+ "13": 1.24673,
+ "14": 1.24765,
+ "15": 1.25145,
+ "16": 1.2523,
+ "17": 1.25236,
+ "18": 1.2482,
+ "19": 1.24584,
+ "20": 1.24859,
+ "21": 1.24771,
+ "22": 1.24486,
+ "23": 1.24581,
+ "24": 1.24312,
+ "25": 1.24263,
+ "26": 1.2459,
+ "27": 1.24224,
+ "28": 1.24465,
+ "29": 1.24539,
+ "30": 1.24017,
+ "31": 1.24429,
+ "32": 1.24716,
+ "33": 1.24243,
+ "34": 1.24207,
+ "35": 1.24315,
+ "36": 1.24486,
+ "37": 1.24601,
+ "38": 1.24418,
+ "39": 1.24028,
+ "40": 1.24411,
+ "41": 1.25241,
+ "42": 1.25198,
+ "43": 1.25172,
+ "44": 1.2459,
+ "45": 1.25141,
+ "46": 1.24677,
+ "47": 1.25017,
+ "48": 1.23789,
+ "49": 1.24264,
+ "50": 1.24793,
+ "51": 1.29892,
+ "52": 1.23885,
+ "53": 1.24722,
+ "54": 1.24621,
+ "55": 1.25232,
+ "56": 1.25873,
+ "57": 1.25279,
+ "58": 1.25296,
+ "59": 1.25428,
+ "60": 1.2587,
+ "61": 1.25091,
+ "62": 1.25693,
+ "63": 1.25495,
+ "64": 1.25562,
+ "65": 1.25477,
+ "66": 1.24842,
+ "67": 1.25111,
+ "68": 1.24759,
+ "69": 1.24788,
+ "70": 1.25923,
+ "71": 1.25344,
+ "72": 1.25461,
+ "73": 1.25362,
+ "74": 1.25391,
+ "75": 1.25177,
+ "76": 1.24761,
+ "77": 1.24417,
+ "78": 1.24962,
+ "79": 1.24879,
+ "80": 1.26023,
+ "81": 1.25268,
+ "82": 1.25285,
+ "83": 1.25497,
+ "84": 1.25521,
+ "85": 1.25388,
+ "86": 1.25062,
+ "87": 1.24777,
+ "88": 1.25164,
+ "89": 1.2511,
+ "90": 1.25067,
+ "91": 1.24993,
+ "92": 1.25344,
+ "93": 1.24934,
+ "94": 1.25351,
+ "95": 1.2507,
+ "96": 1.24708,
+ "97": 1.2546,
+ "98": 1.24501,
+ "99": 1.23104,
+ "100": 1.22854
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph/golden_values_dev_dgx_h100.json
index 0954418053d..13107c98731 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph/golden_values_dev_dgx_h100.json
@@ -4,106 +4,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.93691,
- "2": 10.93262,
- "3": 10.94243,
- "4": 10.95011,
- "5": 10.9502,
- "6": 10.94175,
- "7": 10.94469,
- "8": 10.93675,
- "9": 10.94939,
- "10": 10.9367,
- "11": 10.94082,
- "12": 10.93794,
- "13": 10.92338,
- "14": 10.93415,
- "15": 10.88723,
- "16": 10.87495,
- "17": 10.86864,
- "18": 10.86127,
- "19": 10.86341,
- "20": 10.78125,
- "21": 10.73131,
- "22": 10.60371,
- "23": 10.73309,
- "24": 10.61865,
- "25": 10.55175,
- "26": 10.62651,
- "27": 10.63921,
- "28": 10.59104,
- "29": 10.5981,
- "30": 10.37817,
- "31": 10.12235,
- "32": 10.46117,
- "33": 10.45537,
- "34": 10.20087,
- "35": 10.25661,
- "36": 10.20876,
- "37": 10.33662,
- "38": 10.16683,
- "39": 10.40916,
- "40": 10.05209,
- "41": 10.09427,
- "42": 10.17821,
- "43": 9.74204,
- "44": 9.89005,
- "45": 9.74011,
- "46": 9.72669,
- "47": 10.09152,
- "48": 9.75295,
- "49": 9.40186,
- "50": 9.83645,
- "51": 9.77036,
- "52": 9.65641,
- "53": 10.03067,
- "54": 9.87916,
- "55": 9.79619,
- "56": 9.52858,
- "57": 9.36596,
- "58": 9.75327,
- "59": 9.48259,
- "60": 9.40835,
- "61": 9.60202,
- "62": 9.90742,
- "63": 9.25777,
- "64": 9.68411,
- "65": 8.79911,
- "66": 9.60796,
- "67": 9.25427,
- "68": 9.71419,
- "69": 9.71666,
- "70": 9.6613,
- "71": 9.52439,
- "72": 9.4709,
- "73": 9.38862,
- "74": 8.80286,
- "75": 9.34004,
- "76": 8.93543,
- "77": 9.99337,
- "78": 9.64723,
- "79": 9.28126,
- "80": 9.29633,
- "81": 9.39609,
- "82": 9.60877,
- "83": 9.21694,
- "84": 9.34008,
- "85": 9.53009,
- "86": 8.95652,
- "87": 9.51691,
- "88": 9.68221,
- "89": 9.50553,
- "90": 9.753,
- "91": 9.2347,
- "92": 9.26019,
- "93": 8.94568,
- "94": 8.69194,
- "95": 9.44616,
- "96": 9.41008,
- "97": 9.20125,
- "98": 9.58169,
- "99": 8.75946,
- "100": 9.29483
+ "1": 10.95659,
+ "2": 10.95273,
+ "3": 10.97293,
+ "4": 10.95528,
+ "5": 10.95882,
+ "6": 10.96034,
+ "7": 10.94954,
+ "8": 10.95612,
+ "9": 10.96238,
+ "10": 10.95507,
+ "11": 10.94969,
+ "12": 10.94911,
+ "13": 10.94317,
+ "14": 10.93962,
+ "15": 10.913,
+ "16": 10.89317,
+ "17": 10.89415,
+ "18": 10.8839,
+ "19": 10.88757,
+ "20": 10.81652,
+ "21": 10.77937,
+ "22": 10.77934,
+ "23": 10.75133,
+ "24": 10.73697,
+ "25": 10.70906,
+ "26": 10.70254,
+ "27": 10.66656,
+ "28": 10.58983,
+ "29": 10.57516,
+ "30": 10.5394,
+ "31": 10.54957,
+ "32": 10.49421,
+ "33": 10.45586,
+ "34": 10.45429,
+ "35": 10.41583,
+ "36": 10.40492,
+ "37": 10.37411,
+ "38": 10.38053,
+ "39": 10.33652,
+ "40": 10.33756,
+ "41": 10.29221,
+ "42": 10.24553,
+ "43": 10.23799,
+ "44": 10.20506,
+ "45": 10.23982,
+ "46": 10.1679,
+ "47": 10.16495,
+ "48": 10.11261,
+ "49": 10.11907,
+ "50": 10.09877,
+ "51": 10.11395,
+ "52": 10.07035,
+ "53": 10.03508,
+ "54": 10.01882,
+ "55": 9.97026,
+ "56": 10.01777,
+ "57": 10.00232,
+ "58": 10.00899,
+ "59": 9.93742,
+ "60": 9.97734,
+ "61": 9.92095,
+ "62": 9.86099,
+ "63": 9.97248,
+ "64": 9.91755,
+ "65": 9.8803,
+ "66": 9.90574,
+ "67": 9.88904,
+ "68": 9.81432,
+ "69": 9.83828,
+ "70": 9.82923,
+ "71": 9.85446,
+ "72": 9.84568,
+ "73": 9.79826,
+ "74": 9.79391,
+ "75": 9.74221,
+ "76": 9.8121,
+ "77": 9.8087,
+ "78": 9.76164,
+ "79": 9.73731,
+ "80": 9.76093,
+ "81": 9.80125,
+ "82": 9.72437,
+ "83": 9.66548,
+ "84": 9.62666,
+ "85": 9.59103,
+ "86": 9.7377,
+ "87": 9.72698,
+ "88": 9.73452,
+ "89": 9.63568,
+ "90": 9.62951,
+ "91": 9.6743,
+ "92": 9.63781,
+ "93": 9.53741,
+ "94": 9.65643,
+ "95": 9.62904,
+ "96": 9.63411,
+ "97": 9.54632,
+ "98": 9.59572,
+ "99": 9.64181,
+ "100": 9.53588
}
},
"num-zeros": {
@@ -111,106 +111,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 22750372.0,
- "2": 22953180.0,
- "3": 22604424.0,
- "4": 23266362.0,
- "5": 22735560.0,
- "6": 23061884.0,
- "7": 22793368.0,
- "8": 22960792.0,
- "9": 22865612.0,
- "10": 22950328.0,
- "11": 22499656.0,
- "12": 22456052.0,
- "13": 22948014.0,
- "14": 22384498.0,
- "15": 22846334.0,
- "16": 22856854.0,
- "17": 22836340.0,
- "18": 22590220.0,
- "19": 22627128.0,
- "20": 22712376.0,
- "21": 22762744.0,
- "22": 22816900.0,
- "23": 22545168.0,
- "24": 22794340.0,
- "25": 22841898.0,
- "26": 22549680.0,
- "27": 22464852.0,
- "28": 22453780.0,
- "29": 22534588.0,
- "30": 22636160.0,
- "31": 22989382.0,
- "32": 22594002.0,
- "33": 22566000.0,
- "34": 22855476.0,
- "35": 22813640.0,
- "36": 22595484.0,
- "37": 22499348.0,
- "38": 22926172.0,
- "39": 22825344.0,
- "40": 22675752.0,
- "41": 22671542.0,
- "42": 22682408.0,
- "43": 23014140.0,
- "44": 22768504.0,
- "45": 22679044.0,
- "46": 22912572.0,
- "47": 23691904.0,
- "48": 24003148.0,
- "49": 23786764.0,
- "50": 22931654.0,
- "51": 23866164.0,
- "52": 23807242.0,
- "53": 24007504.0,
- "54": 22867916.0,
- "55": 23571280.0,
- "56": 23954212.0,
- "57": 24211680.0,
- "58": 23914512.0,
- "59": 22722820.0,
- "60": 23813508.0,
- "61": 23796364.0,
- "62": 23739896.0,
- "63": 24965914.0,
- "64": 23898698.0,
- "65": 24150860.0,
- "66": 23796512.0,
- "67": 25032960.0,
- "68": 23673048.0,
- "69": 23644684.0,
- "70": 23903614.0,
- "71": 24864656.0,
- "72": 24766928.0,
- "73": 24850636.0,
- "74": 24133166.0,
- "75": 24143912.0,
- "76": 25025406.0,
- "77": 24358344.0,
- "78": 24910132.0,
- "79": 23808164.0,
- "80": 23772256.0,
- "81": 25020440.0,
- "82": 23851242.0,
- "83": 23911824.0,
- "84": 25143864.0,
- "85": 24823592.0,
- "86": 23153228.0,
- "87": 24850332.0,
- "88": 24749368.0,
- "89": 22505174.0,
- "90": 25108752.0,
- "91": 23838548.0,
- "92": 24923816.0,
- "93": 24769484.0,
- "94": 25041572.0,
- "95": 25189350.0,
- "96": 23909318.0,
- "97": 23664104.0,
- "98": 23832392.0,
- "99": 23981812.0,
- "100": 24101144.0
+ "1": 22985512.0,
+ "2": 22866856.0,
+ "3": 22718796.0,
+ "4": 22793112.0,
+ "5": 22800332.0,
+ "6": 22758732.0,
+ "7": 22889360.0,
+ "8": 22616952.0,
+ "9": 22770544.0,
+ "10": 22482356.0,
+ "11": 22768012.0,
+ "12": 22646636.0,
+ "13": 23376168.0,
+ "14": 23020836.0,
+ "15": 22728468.0,
+ "16": 22844216.0,
+ "17": 22956308.0,
+ "18": 23025414.0,
+ "19": 23121784.0,
+ "20": 22737782.0,
+ "21": 22939100.0,
+ "22": 22975384.0,
+ "23": 22636422.0,
+ "24": 22885688.0,
+ "25": 22646604.0,
+ "26": 23036340.0,
+ "27": 22820192.0,
+ "28": 23031660.0,
+ "29": 23007710.0,
+ "30": 22978120.0,
+ "31": 22931544.0,
+ "32": 22671892.0,
+ "33": 22753852.0,
+ "34": 23115320.0,
+ "35": 22764186.0,
+ "36": 22708208.0,
+ "37": 23140396.0,
+ "38": 22990976.0,
+ "39": 23017508.0,
+ "40": 22766752.0,
+ "41": 23101556.0,
+ "42": 22700170.0,
+ "43": 23019194.0,
+ "44": 22716630.0,
+ "45": 22868256.0,
+ "46": 22743362.0,
+ "47": 22871888.0,
+ "48": 22852526.0,
+ "49": 22908212.0,
+ "50": 22654480.0,
+ "51": 22713796.0,
+ "52": 22833128.0,
+ "53": 22987656.0,
+ "54": 22807024.0,
+ "55": 22950740.0,
+ "56": 22669944.0,
+ "57": 23234316.0,
+ "58": 22699600.0,
+ "59": 22862426.0,
+ "60": 23046704.0,
+ "61": 22688294.0,
+ "62": 22743112.0,
+ "63": 22643864.0,
+ "64": 23031820.0,
+ "65": 23243684.0,
+ "66": 22705280.0,
+ "67": 22986366.0,
+ "68": 22949460.0,
+ "69": 23193560.0,
+ "70": 22838360.0,
+ "71": 22750350.0,
+ "72": 23155256.0,
+ "73": 23168624.0,
+ "74": 22970414.0,
+ "75": 22903392.0,
+ "76": 22714040.0,
+ "77": 23011804.0,
+ "78": 23010392.0,
+ "79": 22845544.0,
+ "80": 22958276.0,
+ "81": 22850234.0,
+ "82": 22746280.0,
+ "83": 22741604.0,
+ "84": 23135624.0,
+ "85": 22945892.0,
+ "86": 23108160.0,
+ "87": 22369104.0,
+ "88": 22565104.0,
+ "89": 22738008.0,
+ "90": 22782056.0,
+ "91": 22941128.0,
+ "92": 22680628.0,
+ "93": 22647880.0,
+ "94": 23168946.0,
+ "95": 22702252.0,
+ "96": 22867296.0,
+ "97": 22852594.0,
+ "98": 22897226.0,
+ "99": 22645712.0,
+ "100": 23029136.0
}
},
"mem-allocated-bytes": {
@@ -218,106 +218,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 773784064.0,
- "2": 775203840.0,
- "3": 766700544.0,
- "4": 937129984.0,
- "5": 934836224.0,
- "6": 934836224.0,
- "7": 935983104.0,
- "8": 937129984.0,
- "9": 937129984.0,
- "10": 937129984.0,
- "11": 937129984.0,
- "12": 937129984.0,
- "13": 937129984.0,
- "14": 934836224.0,
- "15": 934836224.0,
- "16": 935376896.0,
- "17": 935983104.0,
- "18": 937129984.0,
- "19": 937129984.0,
- "20": 937129984.0,
- "21": 937129984.0,
- "22": 934836224.0,
- "23": 934836224.0,
- "24": 935376896.0,
- "25": 937129984.0,
- "26": 937129984.0,
- "27": 937129984.0,
- "28": 934836224.0,
- "29": 935376896.0,
- "30": 936523776.0,
- "31": 936523776.0,
- "32": 937129984.0,
- "33": 937129984.0,
- "34": 937129984.0,
- "35": 937129984.0,
- "36": 937129984.0,
- "37": 937129984.0,
- "38": 934836224.0,
- "39": 935376896.0,
- "40": 936523776.0,
- "41": 937129984.0,
- "42": 937129984.0,
- "43": 937129984.0,
- "44": 934836224.0,
- "45": 934836224.0,
- "46": 937129984.0,
- "47": 935376896.0,
- "48": 937129984.0,
- "49": 937129984.0,
- "50": 935376896.0,
- "51": 935376896.0,
- "52": 937129984.0,
- "53": 937129984.0,
- "54": 934836224.0,
- "55": 934836224.0,
- "56": 934836224.0,
- "57": 934836224.0,
- "58": 934836224.0,
- "59": 934836224.0,
- "60": 934836224.0,
- "61": 935376896.0,
- "62": 935376896.0,
- "63": 935983104.0,
- "64": 936523776.0,
- "65": 936523776.0,
- "66": 936523776.0,
- "67": 937129984.0,
- "68": 937129984.0,
- "69": 937129984.0,
- "70": 937129984.0,
- "71": 937129984.0,
- "72": 937129984.0,
- "73": 937129984.0,
- "74": 934836224.0,
- "75": 934836224.0,
- "76": 935376896.0,
- "77": 935376896.0,
- "78": 936523776.0,
- "79": 937129984.0,
- "80": 937129984.0,
- "81": 937129984.0,
- "82": 937129984.0,
- "83": 934836224.0,
- "84": 934836224.0,
- "85": 934836224.0,
- "86": 936523776.0,
- "87": 936523776.0,
- "88": 937129984.0,
- "89": 937129984.0,
- "90": 937129984.0,
- "91": 937129984.0,
- "92": 934836224.0,
- "93": 935376896.0,
- "94": 936523776.0,
- "95": 936523776.0,
- "96": 936523776.0,
- "97": 936523776.0,
- "98": 936523776.0,
- "99": 937129984.0,
- "100": 937129984.0
+ "1": 804504064.0,
+ "2": 766036992.0,
+ "3": 844984320.0,
+ "4": 801470464.0,
+ "5": 808671232.0,
+ "6": 808818688.0,
+ "7": 804558848.0,
+ "8": 801470464.0,
+ "9": 809105408.0,
+ "10": 807352320.0,
+ "11": 801470464.0,
+ "12": 809530368.0,
+ "13": 808635392.0,
+ "14": 801470464.0,
+ "15": 808386560.0,
+ "16": 804108288.0,
+ "17": 809097216.0,
+ "18": 801470464.0,
+ "19": 801470464.0,
+ "20": 801470464.0,
+ "21": 808868864.0,
+ "22": 808409088.0,
+ "23": 808865792.0,
+ "24": 801470464.0,
+ "25": 801470464.0,
+ "26": 807958528.0,
+ "27": 808450048.0,
+ "28": 808056832.0,
+ "29": 801470464.0,
+ "30": 808410112.0,
+ "31": 808336384.0,
+ "32": 804922368.0,
+ "33": 801470464.0,
+ "34": 808819712.0,
+ "35": 809051136.0,
+ "36": 801470464.0,
+ "37": 808127488.0,
+ "38": 808793088.0,
+ "39": 807352320.0,
+ "40": 801470464.0,
+ "41": 801470464.0,
+ "42": 808155136.0,
+ "43": 808081408.0,
+ "44": 801470464.0,
+ "45": 808409088.0,
+ "46": 808429568.0,
+ "47": 807985152.0,
+ "48": 804370432.0,
+ "49": 801470464.0,
+ "50": 808466432.0,
+ "51": 801470464.0,
+ "52": 801470464.0,
+ "53": 808466432.0,
+ "54": 808409088.0,
+ "55": 807352320.0,
+ "56": 804558848.0,
+ "57": 801470464.0,
+ "58": 808627200.0,
+ "59": 808847360.0,
+ "60": 808333312.0,
+ "61": 804354048.0,
+ "62": 801470464.0,
+ "63": 808409088.0,
+ "64": 808681472.0,
+ "65": 808024064.0,
+ "66": 808409088.0,
+ "67": 805165056.0,
+ "68": 801470464.0,
+ "69": 808627200.0,
+ "70": 808187904.0,
+ "71": 808409088.0,
+ "72": 807802880.0,
+ "73": 801470464.0,
+ "74": 801470464.0,
+ "75": 808377344.0,
+ "76": 809356288.0,
+ "77": 804843520.0,
+ "78": 801470464.0,
+ "79": 808267776.0,
+ "80": 805473280.0,
+ "81": 801470464.0,
+ "82": 808417280.0,
+ "83": 808309760.0,
+ "84": 801470464.0,
+ "85": 808409088.0,
+ "86": 808409088.0,
+ "87": 801470464.0,
+ "88": 807958528.0,
+ "89": 808089600.0,
+ "90": 801470464.0,
+ "91": 808293376.0,
+ "92": 807958528.0,
+ "93": 807663616.0,
+ "94": 801470464.0,
+ "95": 808237056.0,
+ "96": 809064448.0,
+ "97": 801470464.0,
+ "98": 801470464.0,
+ "99": 808598528.0,
+ "100": 808688640.0
}
},
"mem-max-allocated-bytes": {
@@ -325,106 +325,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 990381056.0,
- "2": 1211127808.0,
- "3": 1211127808.0,
- "4": 1296840704.0,
- "5": 1297885184.0,
- "6": 1297885184.0,
- "7": 1298358784.0,
- "8": 1299077120.0,
- "9": 1299077120.0,
- "10": 1300477952.0,
- "11": 1300477952.0,
- "12": 1300477952.0,
- "13": 1300477952.0,
- "14": 1300477952.0,
- "15": 1300477952.0,
- "16": 1300477952.0,
- "17": 1300477952.0,
- "18": 1300477952.0,
- "19": 1300779008.0,
- "20": 1300779008.0,
- "21": 1300779008.0,
- "22": 1300779008.0,
- "23": 1301612544.0,
- "24": 1301612544.0,
- "25": 1301612544.0,
- "26": 1301612544.0,
- "27": 1301612544.0,
- "28": 1301612544.0,
- "29": 1301612544.0,
- "30": 1301612544.0,
- "31": 1301612544.0,
- "32": 1301612544.0,
- "33": 1301612544.0,
- "34": 1301612544.0,
- "35": 1301612544.0,
- "36": 1301612544.0,
- "37": 1301612544.0,
- "38": 1301612544.0,
- "39": 1301612544.0,
- "40": 1301612544.0,
- "41": 1301612544.0,
- "42": 1301612544.0,
- "43": 1301612544.0,
- "44": 1301612544.0,
- "45": 1301612544.0,
- "46": 1301612544.0,
- "47": 1301612544.0,
- "48": 1301612544.0,
- "49": 1301612544.0,
- "50": 1301612544.0,
- "51": 1301612544.0,
- "52": 1301612544.0,
- "53": 1301612544.0,
- "54": 1301612544.0,
- "55": 1301612544.0,
- "56": 1301612544.0,
- "57": 1301612544.0,
- "58": 1301612544.0,
- "59": 1301612544.0,
- "60": 1301612544.0,
- "61": 1301612544.0,
- "62": 1301612544.0,
- "63": 1301612544.0,
- "64": 1301612544.0,
- "65": 1301612544.0,
- "66": 1301612544.0,
- "67": 1301612544.0,
- "68": 1301612544.0,
- "69": 1301612544.0,
- "70": 1301612544.0,
- "71": 1301612544.0,
- "72": 1301612544.0,
- "73": 1301612544.0,
- "74": 1301612544.0,
- "75": 1301612544.0,
- "76": 1301612544.0,
- "77": 1301612544.0,
- "78": 1301612544.0,
- "79": 1301612544.0,
- "80": 1301612544.0,
- "81": 1301612544.0,
- "82": 1301612544.0,
- "83": 1301612544.0,
- "84": 1301612544.0,
- "85": 1301612544.0,
- "86": 1301612544.0,
- "87": 1301612544.0,
- "88": 1301612544.0,
- "89": 1301612544.0,
- "90": 1301612544.0,
- "91": 1301612544.0,
- "92": 1301612544.0,
- "93": 1301612544.0,
- "94": 1301612544.0,
- "95": 1301612544.0,
- "96": 1301612544.0,
- "97": 1301612544.0,
- "98": 1301612544.0,
- "99": 1301612544.0,
- "100": 1301612544.0
+ "1": 978379776.0,
+ "2": 1181844992.0,
+ "3": 1184461312.0,
+ "4": 1184461312.0,
+ "5": 1184461312.0,
+ "6": 1184461312.0,
+ "7": 1184461312.0,
+ "8": 1184461312.0,
+ "9": 1184461312.0,
+ "10": 1184461312.0,
+ "11": 1184461312.0,
+ "12": 1184461312.0,
+ "13": 1184461312.0,
+ "14": 1184461312.0,
+ "15": 1184461312.0,
+ "16": 1184461312.0,
+ "17": 1184461312.0,
+ "18": 1184461312.0,
+ "19": 1184461312.0,
+ "20": 1184461312.0,
+ "21": 1184461312.0,
+ "22": 1184461312.0,
+ "23": 1184461312.0,
+ "24": 1184461312.0,
+ "25": 1184461312.0,
+ "26": 1184461312.0,
+ "27": 1184461312.0,
+ "28": 1184461312.0,
+ "29": 1184461312.0,
+ "30": 1184461312.0,
+ "31": 1184461312.0,
+ "32": 1184461312.0,
+ "33": 1184461312.0,
+ "34": 1184461312.0,
+ "35": 1184461312.0,
+ "36": 1184461312.0,
+ "37": 1184461312.0,
+ "38": 1184461312.0,
+ "39": 1184461312.0,
+ "40": 1184461312.0,
+ "41": 1184461312.0,
+ "42": 1184461312.0,
+ "43": 1184461312.0,
+ "44": 1184461312.0,
+ "45": 1184461312.0,
+ "46": 1184461312.0,
+ "47": 1184461312.0,
+ "48": 1184461312.0,
+ "49": 1184461312.0,
+ "50": 1184461312.0,
+ "51": 1184461312.0,
+ "52": 1184461312.0,
+ "53": 1184461312.0,
+ "54": 1184461312.0,
+ "55": 1184461312.0,
+ "56": 1184461312.0,
+ "57": 1184461312.0,
+ "58": 1184461312.0,
+ "59": 1184461312.0,
+ "60": 1184461312.0,
+ "61": 1184461312.0,
+ "62": 1184461312.0,
+ "63": 1184461312.0,
+ "64": 1184461312.0,
+ "65": 1184461312.0,
+ "66": 1184461312.0,
+ "67": 1184461312.0,
+ "68": 1184461312.0,
+ "69": 1184461312.0,
+ "70": 1184461312.0,
+ "71": 1184461312.0,
+ "72": 1184461312.0,
+ "73": 1184461312.0,
+ "74": 1184461312.0,
+ "75": 1184461312.0,
+ "76": 1184461312.0,
+ "77": 1184461312.0,
+ "78": 1184461312.0,
+ "79": 1184461312.0,
+ "80": 1184461312.0,
+ "81": 1184461312.0,
+ "82": 1184461312.0,
+ "83": 1184461312.0,
+ "84": 1184461312.0,
+ "85": 1184461312.0,
+ "86": 1184461312.0,
+ "87": 1184461312.0,
+ "88": 1184461312.0,
+ "89": 1184461312.0,
+ "90": 1184461312.0,
+ "91": 1184461312.0,
+ "92": 1184461312.0,
+ "93": 1184461312.0,
+ "94": 1184461312.0,
+ "95": 1184461312.0,
+ "96": 1184461312.0,
+ "97": 1184461312.0,
+ "98": 1184461312.0,
+ "99": 1184461312.0,
+ "100": 1184461312.0
}
},
"mtp_1 loss": {
@@ -432,106 +432,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 10.88691,
- "2": 10.90544,
- "3": 10.90868,
- "4": 10.86912,
- "5": 10.91636,
- "6": 10.90651,
- "7": 10.90278,
- "8": 10.88975,
- "9": 10.90453,
- "10": 10.89162,
- "11": 10.93392,
- "12": 10.91634,
- "13": 10.91136,
- "14": 10.91999,
- "15": 10.88538,
- "16": 10.90717,
- "17": 10.87525,
- "18": 10.91409,
- "19": 10.90936,
- "20": 10.87835,
- "21": 10.8786,
- "22": 10.85481,
- "23": 10.87937,
- "24": 10.87208,
- "25": 10.85798,
- "26": 10.86991,
- "27": 10.87718,
- "28": 10.88667,
- "29": 10.88859,
- "30": 10.85479,
- "31": 10.79701,
- "32": 10.86609,
- "33": 10.87789,
- "34": 10.8397,
- "35": 10.84184,
- "36": 10.85,
- "37": 10.85585,
- "38": 10.83714,
- "39": 10.86361,
- "40": 10.82866,
- "41": 10.83386,
- "42": 10.84447,
- "43": 10.78747,
- "44": 10.82127,
- "45": 10.78826,
- "46": 10.78323,
- "47": 10.82894,
- "48": 10.7901,
- "49": 10.71201,
- "50": 10.77359,
- "51": 10.76681,
- "52": 10.74029,
- "53": 10.8027,
- "54": 10.77345,
- "55": 10.76133,
- "56": 10.71153,
- "57": 10.66673,
- "58": 10.74318,
- "59": 10.69182,
- "60": 10.66418,
- "61": 10.70712,
- "62": 10.77164,
- "63": 10.61759,
- "64": 10.71667,
- "65": 10.4936,
- "66": 10.67118,
- "67": 10.57515,
- "68": 10.68716,
- "69": 10.68277,
- "70": 10.66908,
- "71": 10.64566,
- "72": 10.60905,
- "73": 10.56507,
- "74": 10.37106,
- "75": 10.5114,
- "76": 10.39856,
- "77": 10.75192,
- "78": 10.62708,
- "79": 10.4675,
- "80": 10.47474,
- "81": 10.51003,
- "82": 10.58819,
- "83": 10.43946,
- "84": 10.45015,
- "85": 10.55142,
- "86": 10.2831,
- "87": 10.51182,
- "88": 10.60318,
- "89": 10.50948,
- "90": 10.60407,
- "91": 10.38208,
- "92": 10.38708,
- "93": 10.23019,
- "94": 10.08381,
- "95": 10.4259,
- "96": 10.4489,
- "97": 10.32133,
- "98": 10.49668,
- "99": 10.04795,
- "100": 10.33446
+ "1": 10.91546,
+ "2": 10.92323,
+ "3": 10.93384,
+ "4": 10.92739,
+ "5": 10.90724,
+ "6": 10.91817,
+ "7": 10.92486,
+ "8": 10.92528,
+ "9": 10.93457,
+ "10": 10.9265,
+ "11": 10.91896,
+ "12": 10.91863,
+ "13": 10.92808,
+ "14": 10.91206,
+ "15": 10.91984,
+ "16": 10.92451,
+ "17": 10.92227,
+ "18": 10.90737,
+ "19": 10.91483,
+ "20": 10.90522,
+ "21": 10.9114,
+ "22": 10.89772,
+ "23": 10.90537,
+ "24": 10.89029,
+ "25": 10.89787,
+ "26": 10.88468,
+ "27": 10.89842,
+ "28": 10.8909,
+ "29": 10.87535,
+ "30": 10.88065,
+ "31": 10.87294,
+ "32": 10.87864,
+ "33": 10.86983,
+ "34": 10.86798,
+ "35": 10.85949,
+ "36": 10.8618,
+ "37": 10.85516,
+ "38": 10.85688,
+ "39": 10.84912,
+ "40": 10.86276,
+ "41": 10.85336,
+ "42": 10.84776,
+ "43": 10.8455,
+ "44": 10.83817,
+ "45": 10.84937,
+ "46": 10.83807,
+ "47": 10.83805,
+ "48": 10.83108,
+ "49": 10.82947,
+ "50": 10.82233,
+ "51": 10.82166,
+ "52": 10.82114,
+ "53": 10.8067,
+ "54": 10.8107,
+ "55": 10.79431,
+ "56": 10.79976,
+ "57": 10.78946,
+ "58": 10.79833,
+ "59": 10.78093,
+ "60": 10.77476,
+ "61": 10.77647,
+ "62": 10.76099,
+ "63": 10.78365,
+ "64": 10.75478,
+ "65": 10.75021,
+ "66": 10.75701,
+ "67": 10.73475,
+ "68": 10.72894,
+ "69": 10.72604,
+ "70": 10.72547,
+ "71": 10.72458,
+ "72": 10.7195,
+ "73": 10.71167,
+ "74": 10.704,
+ "75": 10.68533,
+ "76": 10.69498,
+ "77": 10.69053,
+ "78": 10.68213,
+ "79": 10.6697,
+ "80": 10.67692,
+ "81": 10.66916,
+ "82": 10.65024,
+ "83": 10.62678,
+ "84": 10.61021,
+ "85": 10.6026,
+ "86": 10.64309,
+ "87": 10.63639,
+ "88": 10.63081,
+ "89": 10.59534,
+ "90": 10.58433,
+ "91": 10.60787,
+ "92": 10.58304,
+ "93": 10.56199,
+ "94": 10.59372,
+ "95": 10.57621,
+ "96": 10.57236,
+ "97": 10.55407,
+ "98": 10.5595,
+ "99": 10.55809,
+ "100": 10.5283
}
},
"iteration-time": {
@@ -539,106 +539,106 @@
"end_step": 100,
"step_interval": 1,
"values": {
- "1": 56.96201,
- "2": 1.45193,
- "3": 1.37387,
- "4": 3.96627,
- "5": 0.7423,
- "6": 0.71394,
- "7": 0.74369,
- "8": 0.72342,
- "9": 0.70545,
- "10": 0.70125,
- "11": 0.70256,
- "12": 0.69915,
- "13": 0.70499,
- "14": 0.72329,
- "15": 0.71852,
- "16": 0.71011,
- "17": 0.70885,
- "18": 0.73035,
- "19": 0.71099,
- "20": 0.70225,
- "21": 0.70459,
- "22": 0.71823,
- "23": 0.7143,
- "24": 0.72574,
- "25": 0.72055,
- "26": 0.71722,
- "27": 0.71209,
- "28": 0.72407,
- "29": 0.72809,
- "30": 0.71187,
- "31": 0.70668,
- "32": 0.70676,
- "33": 0.70474,
- "34": 0.70406,
- "35": 0.70401,
- "36": 0.70968,
- "37": 0.71106,
- "38": 0.72458,
- "39": 0.736,
- "40": 0.71238,
- "41": 0.71868,
- "42": 0.71459,
- "43": 0.71031,
- "44": 0.70945,
- "45": 0.72444,
- "46": 0.76158,
- "47": 0.75856,
- "48": 0.7282,
- "49": 0.72448,
- "50": 0.7471,
- "51": 0.80801,
- "52": 0.73438,
- "53": 0.71695,
- "54": 0.71541,
- "55": 0.70768,
- "56": 0.70462,
- "57": 0.70705,
- "58": 0.70511,
- "59": 0.70702,
- "60": 0.70636,
- "61": 0.70372,
- "62": 0.71024,
- "63": 0.70358,
- "64": 0.70559,
- "65": 0.70617,
- "66": 0.70048,
- "67": 0.71248,
- "68": 0.7119,
- "69": 0.71093,
- "70": 0.7051,
- "71": 0.70391,
- "72": 0.70275,
- "73": 0.70876,
- "74": 0.7119,
- "75": 0.71307,
- "76": 0.718,
- "77": 0.71166,
- "78": 0.71308,
- "79": 0.70995,
- "80": 0.71153,
- "81": 0.71464,
- "82": 0.71596,
- "83": 0.71997,
- "84": 0.71197,
- "85": 0.70577,
- "86": 0.71956,
- "87": 0.70383,
- "88": 0.71047,
- "89": 0.71711,
- "90": 0.70818,
- "91": 0.71353,
- "92": 0.71401,
- "93": 0.73616,
- "94": 0.71104,
- "95": 0.70295,
- "96": 0.69995,
- "97": 0.7015,
- "98": 0.70705,
- "99": 0.70765,
- "100": 0.72052
+ "1": "nan",
+ "2": 27.18208,
+ "3": 1.39529,
+ "4": 3.6491,
+ "5": 0.67179,
+ "6": 0.67671,
+ "7": 0.66994,
+ "8": 0.66973,
+ "9": 0.67108,
+ "10": 0.67559,
+ "11": 0.67217,
+ "12": 0.67331,
+ "13": 0.66954,
+ "14": 0.67002,
+ "15": 0.67239,
+ "16": 0.76215,
+ "17": 0.69839,
+ "18": 0.68015,
+ "19": 0.69381,
+ "20": 0.68775,
+ "21": 0.69137,
+ "22": 0.68806,
+ "23": 0.69976,
+ "24": 0.77931,
+ "25": 0.76553,
+ "26": 0.68909,
+ "27": 0.68374,
+ "28": 0.68045,
+ "29": 0.6771,
+ "30": 0.67224,
+ "31": 0.67362,
+ "32": 0.67682,
+ "33": 0.672,
+ "34": 0.67674,
+ "35": 0.67276,
+ "36": 0.67257,
+ "37": 0.67332,
+ "38": 0.68112,
+ "39": 0.67781,
+ "40": 0.67515,
+ "41": 0.67587,
+ "42": 0.6733,
+ "43": 0.67545,
+ "44": 0.67335,
+ "45": 0.68357,
+ "46": 0.68261,
+ "47": 0.68097,
+ "48": 0.68067,
+ "49": 0.68428,
+ "50": 0.68183,
+ "51": 0.69468,
+ "52": 0.68108,
+ "53": 0.683,
+ "54": 0.68569,
+ "55": 0.68183,
+ "56": 0.68275,
+ "57": 0.6821,
+ "58": 0.68182,
+ "59": 0.68538,
+ "60": 0.68324,
+ "61": 0.68519,
+ "62": 0.68243,
+ "63": 0.68308,
+ "64": 0.69526,
+ "65": 0.68084,
+ "66": 0.68955,
+ "67": 0.68442,
+ "68": 0.68126,
+ "69": 0.68341,
+ "70": 0.68587,
+ "71": 0.68166,
+ "72": 0.68346,
+ "73": 0.68477,
+ "74": 0.6857,
+ "75": 0.68228,
+ "76": 0.68263,
+ "77": 0.67013,
+ "78": 0.66937,
+ "79": 0.66958,
+ "80": 0.66944,
+ "81": 0.67111,
+ "82": 0.67321,
+ "83": 0.66983,
+ "84": 0.67414,
+ "85": 0.67114,
+ "86": 0.67054,
+ "87": 0.66936,
+ "88": 0.66939,
+ "89": 0.66786,
+ "90": 0.66981,
+ "91": 0.66651,
+ "92": 0.67627,
+ "93": 0.68747,
+ "94": 0.67136,
+ "95": 0.67193,
+ "96": 0.67111,
+ "97": 0.66996,
+ "98": 0.68055,
+ "99": 0.6806,
+ "100": 0.67843
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph/model_config.yaml
index f0d1cc0afd3..badd7a0b91a 100644
--- a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph/model_config.yaml
@@ -33,7 +33,7 @@ MODEL_ARGS:
--pipeline-model-parallel-size: 2
--expert-model-parallel-size: 2
--expert-tensor-parallel-size: 2
- --pipeline-model-parallel-layout: Et\\|\\(tt\\|\\)*6mL # Et|(tt|)*6mL
+ --pipeline-model-parallel-layout: Et|(tt|)*6mL
--sequence-parallel: true
--num-experts: 8
--use-distributed-optimizer: true
@@ -77,15 +77,17 @@ MODEL_ARGS:
--save-interval: 50
--eval-interval: 1000
--eval-iters: 10
- --data-path: ${DATA_PATH}/text/the_pile/shard00/my-gpt3_00_text_document
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
--data-cache-path: ${DATA_CACHE_PATH}
- --vocab-file: ${DATA_PATH}/text/the_pile/shard00/bpe/vocab.json
- --merge-file: ${DATA_PATH}/text/the_pile/shard00/bpe/merges.txt
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+ --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
--save: ${CHECKPOINT_SAVE_PATH}
--load: ${CHECKPOINT_LOAD_PATH}
--ckpt-fully-parallel-load: true
--ckpt-format: torch_dist
--ckpt-assume-constant-structure: true
+ --async-save: true
+ --use-persistent-ckpt-worker: true
TEST_TYPE: ckpt-resume
METRICS:
# - "iteration-time"
diff --git a/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_cuda_graphs_pad_tp4_pp1_ep4_16B_logitsmatch/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_cuda_graphs_pad_tp4_pp1_ep4_16B_logitsmatch/golden_values_dev_dgx_h100.json
index 447e404810b..3f022ffde7e 100644
--- a/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_cuda_graphs_pad_tp4_pp1_ep4_16B_logitsmatch/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_cuda_graphs_pad_tp4_pp1_ep4_16B_logitsmatch/golden_values_dev_dgx_h100.json
@@ -34,7 +34,8 @@
1394,
1636
],
- "latency": 0.6956486701965332,
+ "latency": 0.6897497177124023,
+ "ttft": 0.17203974723815918,
"cuda_graph_request_count_map": {
"8": 29
},
@@ -43,253 +44,255 @@
"prompt_top_n_logprobs": null,
"prompt_logprobs": [
-10.737512588500977,
- -3.724862575531006,
- -2.833397388458252,
- -1.2464861869812012,
- -0.2549239993095398,
- -1.7607988119125366,
- -2.419379711151123,
- -1.9533929824829102,
- -2.1014301776885986,
- -6.169030666351318,
- -0.8734959363937378,
- -2.4733574390411377,
- -3.4822516441345215,
- -4.180896759033203,
- -1.9767613410949707,
- -1.8347630500793457,
- -2.2581257820129395,
- -7.180149078369141,
- -0.0453881211578846,
- -1.9841610193252563,
- -5.015386581420898,
- -8.827117919921875,
- -9.885746002197266,
- -0.8498678207397461,
- -4.770059585571289,
- -0.855280339717865,
- -2.2494924068450928,
- -0.017164958640933037,
- -0.03715415671467781,
- -3.4830124378204346,
- -8.635110855102539,
- -1.2520610094070435,
- -6.62324857711792,
- -3.639960765838623,
- -3.664339542388916,
- -4.182392597198486,
- -2.1796066761016846,
- -1.0725229978561401,
- -0.26311880350112915,
- -0.8036076426506042,
- -4.6958818435668945,
- -9.042495727539062,
- -0.013647346757352352,
- -3.1747794151306152,
- -1.322129487991333,
- -3.949110746383667,
- -0.7829495072364807,
- -0.002083513652905822,
- -2.970266580581665,
- -10.56244945526123,
- -3.2369167804718018,
- -1.1530492305755615,
- -4.917466163635254,
- -0.21241025626659393,
- -0.06490474194288254,
- -1.372581124305725,
- -2.224682092666626,
- -4.3847503662109375,
- -0.36867555975914,
- -4.035493850708008,
- -0.39869019389152527,
- -0.14373983442783356,
- -2.716118812561035,
- -10.687016487121582,
- -0.04773370549082756,
- -3.398231267929077,
- -0.8646175265312195,
- -4.74052619934082,
- -0.23649944365024567,
- -2.6610701084136963,
- -0.8428961634635925,
- -1.614527940750122,
- -5.793307781219482,
- -16.929147720336914,
- -2.6586406230926514,
- -0.1385982781648636,
- -7.435610771179199,
- -1.0483647584915161,
- -2.1261863708496094,
- -1.5261307954788208,
- -0.27082547545433044,
- -5.859070777893066,
- -0.00648513063788414,
- -7.732051849365234,
- -2.712515354156494,
- -2.9137418270111084,
- -3.041210651397705
+ -3.6886487007141113,
+ -2.8194005489349365,
+ -1.2396225929260254,
+ -0.22920642793178558,
+ -1.8583638668060303,
+ -2.384000778198242,
+ -2.008981227874756,
+ -2.09975528717041,
+ -6.182888031005859,
+ -0.890110433101654,
+ -2.478799343109131,
+ -3.502744436264038,
+ -4.090683937072754,
+ -1.964805006980896,
+ -1.8054677248001099,
+ -2.321495294570923,
+ -7.160285949707031,
+ -0.04007242992520332,
+ -1.9924155473709106,
+ -5.093112468719482,
+ -8.779500007629395,
+ -9.904478073120117,
+ -0.8523460626602173,
+ -4.82073974609375,
+ -0.86553555727005,
+ -2.36330509185791,
+ -0.01946748048067093,
+ -0.034191977232694626,
+ -3.3692376613616943,
+ -8.743470191955566,
+ -1.3306764364242554,
+ -6.645841598510742,
+ -3.7794108390808105,
+ -3.7756881713867188,
+ -4.187170028686523,
+ -2.2124571800231934,
+ -1.0734034776687622,
+ -0.22135400772094727,
+ -0.7782289981842041,
+ -4.716646671295166,
+ -9.09740161895752,
+ -0.013706612400710583,
+ -3.1654152870178223,
+ -1.3161238431930542,
+ -4.0115814208984375,
+ -0.7715368866920471,
+ -0.002078041434288025,
+ -2.9644386768341064,
+ -10.690834045410156,
+ -3.2352819442749023,
+ -1.1544448137283325,
+ -4.971055030822754,
+ -0.2275625467300415,
+ -0.05549970641732216,
+ -1.2946704626083374,
+ -2.20631742477417,
+ -4.362645149230957,
+ -0.37027108669281006,
+ -4.025010108947754,
+ -0.3656681478023529,
+ -0.13922274112701416,
+ -2.720149040222168,
+ -10.679438591003418,
+ -0.04889172688126564,
+ -3.483541965484619,
+ -0.9704694151878357,
+ -4.714925289154053,
+ -0.2623435854911804,
+ -2.60113525390625,
+ -0.8295360207557678,
+ -1.671109676361084,
+ -5.792396068572998,
+ -16.96857261657715,
+ -2.647408962249756,
+ -0.13962996006011963,
+ -7.453517436981201,
+ -1.0938549041748047,
+ -2.134143829345703,
+ -1.5537410974502563,
+ -0.2869631052017212,
+ -5.857534408569336,
+ -0.006919232662767172,
+ -7.747158527374268,
+ -2.7227447032928467,
+ -3.037987232208252,
+ -3.0312116146087646
],
"generated_logprobs": [
- -2.3559694290161133,
- -0.3973437249660492,
- -1.4338903427124023,
- -2.2967660427093506,
- -0.6096595525741577,
- -1.310807704925537,
- -1.9799187183380127,
- -1.710689663887024,
- -0.8325198888778687,
- -0.4943186938762665,
- -1.2886956930160522,
- -1.585263729095459,
- -1.101692795753479,
- -0.44188663363456726,
- -0.4740143418312073,
- -0.042198192328214645,
- -1.2899682521820068,
- -2.1242613792419434,
- -2.7151529788970947,
- -0.8274281024932861,
- -0.39647114276885986,
- -2.7846553325653076,
- -1.5348155498504639,
- -1.626529335975647,
- -0.047930192202329636,
- -1.3622726202011108,
- -1.3274445533752441,
- -1.2834383249282837,
- -1.3211638927459717,
- -0.48530423641204834
+ -2.342064619064331,
+ -0.39744827151298523,
+ -1.516434907913208,
+ -2.3665950298309326,
+ -0.6091187000274658,
+ -1.3131166696548462,
+ -1.9402395486831665,
+ -1.6906054019927979,
+ -0.8380135297775269,
+ -0.4860585927963257,
+ -1.2984997034072876,
+ -1.5767009258270264,
+ -0.9797608852386475,
+ -0.40544381737709045,
+ -0.46793943643569946,
+ -0.041146114468574524,
+ -1.2806015014648438,
+ -2.1299755573272705,
+ -2.7230143547058105,
+ -0.7635340690612793,
+ -0.43544360995292664,
+ -2.7788889408111572,
+ -1.5222126245498657,
+ -1.6613693237304688,
+ -0.05091489478945732,
+ -1.3707530498504639,
+ -1.3513561487197876,
+ -1.2351378202438354,
+ -1.2675105333328247,
+ -0.4767935872077942
],
"logprobs": [
-10.737512588500977,
- -3.724862575531006,
- -2.833397388458252,
- -1.2464861869812012,
- -0.2549239993095398,
- -1.7607988119125366,
- -2.419379711151123,
- -1.9533929824829102,
- -2.1014301776885986,
- -6.169030666351318,
- -0.8734959363937378,
- -2.4733574390411377,
- -3.4822516441345215,
- -4.180896759033203,
- -1.9767613410949707,
- -1.8347630500793457,
- -2.2581257820129395,
- -7.180149078369141,
- -0.0453881211578846,
- -1.9841610193252563,
- -5.015386581420898,
- -8.827117919921875,
- -9.885746002197266,
- -0.8498678207397461,
- -4.770059585571289,
- -0.855280339717865,
- -2.2494924068450928,
- -0.017164958640933037,
- -0.03715415671467781,
- -3.4830124378204346,
- -8.635110855102539,
- -1.2520610094070435,
- -6.62324857711792,
- -3.639960765838623,
- -3.664339542388916,
- -4.182392597198486,
- -2.1796066761016846,
- -1.0725229978561401,
- -0.26311880350112915,
- -0.8036076426506042,
- -4.6958818435668945,
- -9.042495727539062,
- -0.013647346757352352,
- -3.1747794151306152,
- -1.322129487991333,
- -3.949110746383667,
- -0.7829495072364807,
- -0.002083513652905822,
- -2.970266580581665,
- -10.56244945526123,
- -3.2369167804718018,
- -1.1530492305755615,
- -4.917466163635254,
- -0.21241025626659393,
- -0.06490474194288254,
- -1.372581124305725,
- -2.224682092666626,
- -4.3847503662109375,
- -0.36867555975914,
- -4.035493850708008,
- -0.39869019389152527,
- -0.14373983442783356,
- -2.716118812561035,
- -10.687016487121582,
- -0.04773370549082756,
- -3.398231267929077,
- -0.8646175265312195,
- -4.74052619934082,
- -0.23649944365024567,
- -2.6610701084136963,
- -0.8428961634635925,
- -1.614527940750122,
- -5.793307781219482,
- -16.929147720336914,
- -2.6586406230926514,
- -0.1385982781648636,
- -7.435610771179199,
- -1.0483647584915161,
- -2.1261863708496094,
- -1.5261307954788208,
- -0.27082547545433044,
- -5.859070777893066,
- -0.00648513063788414,
- -7.732051849365234,
- -2.712515354156494,
- -2.9137418270111084,
- -3.041210651397705,
- -2.3559694290161133,
- -0.3973437249660492,
- -1.4338903427124023,
- -2.2967660427093506,
- -0.6096595525741577,
- -1.310807704925537,
- -1.9799187183380127,
- -1.710689663887024,
- -0.8325198888778687,
- -0.4943186938762665,
- -1.2886956930160522,
- -1.585263729095459,
- -1.101692795753479,
- -0.44188663363456726,
- -0.4740143418312073,
- -0.042198192328214645,
- -1.2899682521820068,
- -2.1242613792419434,
- -2.7151529788970947,
- -0.8274281024932861,
- -0.39647114276885986,
- -2.7846553325653076,
- -1.5348155498504639,
- -1.626529335975647,
- -0.047930192202329636,
- -1.3622726202011108,
- -1.3274445533752441,
- -1.2834383249282837,
- -1.3211638927459717,
- -0.48530423641204834
+ -3.6886487007141113,
+ -2.8194005489349365,
+ -1.2396225929260254,
+ -0.22920642793178558,
+ -1.8583638668060303,
+ -2.384000778198242,
+ -2.008981227874756,
+ -2.09975528717041,
+ -6.182888031005859,
+ -0.890110433101654,
+ -2.478799343109131,
+ -3.502744436264038,
+ -4.090683937072754,
+ -1.964805006980896,
+ -1.8054677248001099,
+ -2.321495294570923,
+ -7.160285949707031,
+ -0.04007242992520332,
+ -1.9924155473709106,
+ -5.093112468719482,
+ -8.779500007629395,
+ -9.904478073120117,
+ -0.8523460626602173,
+ -4.82073974609375,
+ -0.86553555727005,
+ -2.36330509185791,
+ -0.01946748048067093,
+ -0.034191977232694626,
+ -3.3692376613616943,
+ -8.743470191955566,
+ -1.3306764364242554,
+ -6.645841598510742,
+ -3.7794108390808105,
+ -3.7756881713867188,
+ -4.187170028686523,
+ -2.2124571800231934,
+ -1.0734034776687622,
+ -0.22135400772094727,
+ -0.7782289981842041,
+ -4.716646671295166,
+ -9.09740161895752,
+ -0.013706612400710583,
+ -3.1654152870178223,
+ -1.3161238431930542,
+ -4.0115814208984375,
+ -0.7715368866920471,
+ -0.002078041434288025,
+ -2.9644386768341064,
+ -10.690834045410156,
+ -3.2352819442749023,
+ -1.1544448137283325,
+ -4.971055030822754,
+ -0.2275625467300415,
+ -0.05549970641732216,
+ -1.2946704626083374,
+ -2.20631742477417,
+ -4.362645149230957,
+ -0.37027108669281006,
+ -4.025010108947754,
+ -0.3656681478023529,
+ -0.13922274112701416,
+ -2.720149040222168,
+ -10.679438591003418,
+ -0.04889172688126564,
+ -3.483541965484619,
+ -0.9704694151878357,
+ -4.714925289154053,
+ -0.2623435854911804,
+ -2.60113525390625,
+ -0.8295360207557678,
+ -1.671109676361084,
+ -5.792396068572998,
+ -16.96857261657715,
+ -2.647408962249756,
+ -0.13962996006011963,
+ -7.453517436981201,
+ -1.0938549041748047,
+ -2.134143829345703,
+ -1.5537410974502563,
+ -0.2869631052017212,
+ -5.857534408569336,
+ -0.006919232662767172,
+ -7.747158527374268,
+ -2.7227447032928467,
+ -3.037987232208252,
+ -3.0312116146087646,
+ -2.342064619064331,
+ -0.39744827151298523,
+ -1.516434907913208,
+ -2.3665950298309326,
+ -0.6091187000274658,
+ -1.3131166696548462,
+ -1.9402395486831665,
+ -1.6906054019927979,
+ -0.8380135297775269,
+ -0.4860585927963257,
+ -1.2984997034072876,
+ -1.5767009258270264,
+ -0.9797608852386475,
+ -0.40544381737709045,
+ -0.46793943643569946,
+ -0.041146114468574524,
+ -1.2806015014648438,
+ -2.1299755573272705,
+ -2.7230143547058105,
+ -0.7635340690612793,
+ -0.43544360995292664,
+ -2.7788889408111572,
+ -1.5222126245498657,
+ -1.6613693237304688,
+ -0.05091489478945732,
+ -1.3707530498504639,
+ -1.3513561487197876,
+ -1.2351378202438354,
+ -1.2675105333328247,
+ -0.4767935872077942
]
},
"throughput": [
- 1.9407774475980641,
- 40.8327035151158,
- 42.82685786577602,
- 42.814063580843204,
- 42.88001112304976,
- 42.90077111461981,
- 42.81151438072744,
- 42.89731946981911
- ]
+ 7.8286224601594805,
+ 42.90151709390053,
+ 43.28774493800242,
+ 43.24373961471115,
+ 43.23396289817666,
+ 43.197302475580116,
+ 43.15719577445466,
+ 43.24482453812981
+ ],
+ "mem-max-allocated-bytes": 30516649984,
+ "lifetime_prefill_token_count": 88
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_cudagraph_zmq/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_cudagraph_zmq/golden_values_dev_dgx_h100.json
index b239ac96c3d..73be1228f97 100644
--- a/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_cudagraph_zmq/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_cudagraph_zmq/golden_values_dev_dgx_h100.json
@@ -34,125 +34,25633 @@
1394,
1636
],
- "latency": 1.974948332994245,
+ "latency": 2.0598746850009775,
"logprobs": [
-10.737512588500977,
- -3.724862575531006,
- -2.833397388458252,
- -1.2464861869812012,
- -0.2549239993095398,
- -1.7607988119125366,
- -2.419379711151123,
- -1.9533929824829102,
- -2.1014301776885986,
- -6.169030666351318,
- -0.8734959363937378,
- -2.4733574390411377,
- -3.4822516441345215,
- -4.180896759033203,
- -1.9767613410949707,
- -1.8347630500793457,
- -2.2581257820129395,
- -7.180149078369141,
- -0.0453881211578846,
- -1.9841610193252563,
- -5.015386581420898,
- -8.827117919921875,
- -9.885746002197266,
- -0.8498678207397461,
- -4.770059585571289,
- -0.855280339717865,
- -2.2494924068450928,
- -0.017164958640933037,
- -0.03715415671467781,
- -3.4830124378204346,
- -8.635110855102539,
- -1.2520610094070435,
- -6.604944705963135,
- -3.873375177383423,
- -3.6618807315826416,
- -4.184391975402832,
- -2.2140231132507324,
- -1.0711486339569092,
- -0.23651468753814697,
- -0.8028834462165833,
- -4.772289276123047,
- -9.09329605102539,
- -0.013350379653275013,
- -3.204096555709839,
- -1.286576271057129,
- -3.966451644897461,
- -0.784888744354248,
- -0.002081372309476137,
- -2.9676947593688965,
- -10.666797637939453,
- -3.026693105697632,
- -1.1712640523910522,
- -4.93634033203125,
- -0.2349349856376648,
- -0.06406460702419281,
- -1.3560465574264526,
- -2.1993725299835205,
- -4.378607749938965,
- -0.36503157019615173,
- -4.032698631286621,
- -0.3783165216445923,
- -0.14568912982940674,
- -2.7145652770996094,
- -10.591975212097168,
- -0.043545372784137726,
- -3.484504461288452,
- -0.9063424468040466,
- -4.716145992279053,
- -0.263183057308197,
- -2.654480457305908,
- -0.8381667137145996,
- -1.677478551864624,
- -5.816836357116699,
- -17.05772590637207,
- -2.634756088256836,
- -0.1400006264448166,
- -7.400341987609863,
- -1.0960910320281982,
- -2.1047825813293457,
- -1.5999900102615356,
- -0.26269301772117615,
- -5.727797508239746,
- -0.006663957145065069,
- -7.748298168182373,
- -2.6529595851898193,
- -3.0231595039367676,
- -2.9622015953063965,
- -2.3360800743103027,
- -0.3991503119468689,
- -1.5159229040145874,
- -2.289414882659912,
- -0.6100144386291504,
- -1.3164187669754028,
- -1.9431946277618408,
- -1.7792527675628662,
- -0.8328706622123718,
- -0.501052737236023,
- -1.278053879737854,
- -1.5683506727218628,
- -0.9720054864883423,
- -0.40760406851768494,
- -0.43419456481933594,
- -0.04328203946352005,
- -1.2999448776245117,
- -2.1266980171203613,
- -2.6690115928649902,
- -0.7812177538871765,
- -0.41717368364334106,
- -2.8806936740875244,
- -1.5312169790267944,
- -1.62917160987854,
- -0.05274559557437897,
- -1.362119436264038,
- -1.337896704673767,
- -1.2551532983779907,
- -1.256169080734253,
- -0.49199968576431274
+ -3.6886487007141113,
+ -2.8194005489349365,
+ -1.2396225929260254,
+ -0.22920642793178558,
+ -1.8583638668060303,
+ -2.384000778198242,
+ -2.008981227874756,
+ -2.09975528717041,
+ -6.182888031005859,
+ -0.890110433101654,
+ -2.478799343109131,
+ -3.502744436264038,
+ -4.090683937072754,
+ -1.964805006980896,
+ -1.8054677248001099,
+ -2.321495294570923,
+ -7.160285949707031,
+ -0.04007242992520332,
+ -1.9924155473709106,
+ -5.093112468719482,
+ -8.779500007629395,
+ -9.904478073120117,
+ -0.8523460626602173,
+ -4.82073974609375,
+ -0.86553555727005,
+ -2.36330509185791,
+ -0.01946748048067093,
+ -0.034191977232694626,
+ -3.3692376613616943,
+ -8.743470191955566,
+ -1.3306764364242554,
+ -6.645841598510742,
+ -3.7794108390808105,
+ -3.7756881713867188,
+ -4.187170028686523,
+ -2.2124571800231934,
+ -1.0734034776687622,
+ -0.22135400772094727,
+ -0.7782289981842041,
+ -4.716646671295166,
+ -9.09740161895752,
+ -0.013706612400710583,
+ -3.1654152870178223,
+ -1.3161238431930542,
+ -4.0115814208984375,
+ -0.7715368866920471,
+ -0.002078041434288025,
+ -2.9644386768341064,
+ -10.690834045410156,
+ -3.2352819442749023,
+ -1.1544448137283325,
+ -4.971055030822754,
+ -0.2275625467300415,
+ -0.05549970641732216,
+ -1.2946704626083374,
+ -2.20631742477417,
+ -4.362645149230957,
+ -0.37027108669281006,
+ -4.025010108947754,
+ -0.3656681478023529,
+ -0.13922274112701416,
+ -2.720149040222168,
+ -10.679438591003418,
+ -0.04889172688126564,
+ -3.483541965484619,
+ -0.9704694151878357,
+ -4.714925289154053,
+ -0.2623435854911804,
+ -2.60113525390625,
+ -0.8295360207557678,
+ -1.671109676361084,
+ -5.792396068572998,
+ -16.96857261657715,
+ -2.647408962249756,
+ -0.13962996006011963,
+ -7.453517436981201,
+ -1.0938549041748047,
+ -2.134143829345703,
+ -1.5537410974502563,
+ -0.2869631052017212,
+ -5.857534408569336,
+ -0.006919232662767172,
+ -7.747158527374268,
+ -2.7227447032928467,
+ -3.037987232208252,
+ -3.0312116146087646,
+ -2.342064619064331,
+ -0.39744827151298523,
+ -1.516434907913208,
+ -2.3665950298309326,
+ -0.6091187000274658,
+ -1.3131166696548462,
+ -1.9402395486831665,
+ -1.6906054019927979,
+ -0.8380135297775269,
+ -0.4860585927963257,
+ -1.2984997034072876,
+ -1.5767009258270264,
+ -0.9797608852386475,
+ -0.40544381737709045,
+ -0.46793943643569946,
+ -0.041146114468574524,
+ -1.2806015014648438,
+ -2.1299755573272705,
+ -2.7230143547058105,
+ -0.7635340690612793,
+ -0.43544360995292664,
+ -2.7788889408111572,
+ -1.5222126245498657,
+ -1.6613693237304688,
+ -0.05091489478945732,
+ -1.3707530498504639,
+ -1.3513561487197876,
+ -1.2351378202438354,
+ -1.2675105333328247,
+ -0.4767935872077942
+ ],
+ "routing_indices": [
+ [
+ [
+ 33,
+ 50,
+ 36,
+ 4,
+ 25,
+ 63
+ ],
+ [
+ 0,
+ 16,
+ 3,
+ 26,
+ 9,
+ 54
+ ],
+ [
+ 62,
+ 60,
+ 8,
+ 16,
+ 58,
+ 52
+ ],
+ [
+ 58,
+ 10,
+ 6,
+ 45,
+ 16,
+ 32
+ ],
+ [
+ 43,
+ 49,
+ 18,
+ 54,
+ 55,
+ 13
+ ],
+ [
+ 27,
+ 19,
+ 26,
+ 44,
+ 12,
+ 28
+ ],
+ [
+ 53,
+ 42,
+ 3,
+ 27,
+ 26,
+ 19
+ ],
+ [
+ 6,
+ 47,
+ 1,
+ 8,
+ 19,
+ 22
+ ],
+ [
+ 51,
+ 27,
+ 1,
+ 38,
+ 16,
+ 62
+ ],
+ [
+ 41,
+ 49,
+ 21,
+ 57,
+ 16,
+ 24
+ ],
+ [
+ 13,
+ 28,
+ 38,
+ 22,
+ 49,
+ 48
+ ],
+ [
+ 52,
+ 58,
+ 6,
+ 25,
+ 29,
+ 17
+ ],
+ [
+ 49,
+ 50,
+ 25,
+ 41,
+ 54,
+ 58
+ ],
+ [
+ 27,
+ 41,
+ 3,
+ 1,
+ 26,
+ 29
+ ],
+ [
+ 41,
+ 18,
+ 34,
+ 45,
+ 1,
+ 33
+ ],
+ [
+ 17,
+ 26,
+ 59,
+ 22,
+ 19,
+ 4
+ ],
+ [
+ 1,
+ 43,
+ 62,
+ 57,
+ 61,
+ 21
+ ],
+ [
+ 11,
+ 46,
+ 15,
+ 28,
+ 61,
+ 50
+ ],
+ [
+ 14,
+ 30,
+ 58,
+ 26,
+ 38,
+ 53
+ ],
+ [
+ 59,
+ 20,
+ 63,
+ 54,
+ 47,
+ 61
+ ],
+ [
+ 9,
+ 5,
+ 43,
+ 33,
+ 15,
+ 46
+ ],
+ [
+ 16,
+ 52,
+ 33,
+ 61,
+ 49,
+ 11
+ ],
+ [
+ 52,
+ 35,
+ 40,
+ 43,
+ 29,
+ 36
+ ],
+ [
+ 57,
+ 34,
+ 38,
+ 44,
+ 20,
+ 18
+ ],
+ [
+ 44,
+ 51,
+ 2,
+ 63,
+ 7,
+ 22
+ ],
+ [
+ 32,
+ 47,
+ 58,
+ 9,
+ 54,
+ 5
+ ],
+ [
+ 12,
+ 59,
+ 54,
+ 33,
+ 50,
+ 6
+ ]
+ ],
+ [
+ [
+ 49,
+ 43,
+ 18,
+ 28,
+ 23,
+ 25
+ ],
+ [
+ 34,
+ 24,
+ 23,
+ 60,
+ 2,
+ 18
+ ],
+ [
+ 33,
+ 40,
+ 30,
+ 3,
+ 59,
+ 48
+ ],
+ [
+ 16,
+ 58,
+ 17,
+ 48,
+ 6,
+ 45
+ ],
+ [
+ 23,
+ 58,
+ 46,
+ 37,
+ 34,
+ 48
+ ],
+ [
+ 28,
+ 26,
+ 35,
+ 33,
+ 22,
+ 43
+ ],
+ [
+ 6,
+ 31,
+ 13,
+ 46,
+ 41,
+ 37
+ ],
+ [
+ 60,
+ 29,
+ 44,
+ 36,
+ 39,
+ 15
+ ],
+ [
+ 51,
+ 18,
+ 62,
+ 5,
+ 27,
+ 35
+ ],
+ [
+ 62,
+ 45,
+ 32,
+ 56,
+ 25,
+ 3
+ ],
+ [
+ 6,
+ 20,
+ 3,
+ 16,
+ 49,
+ 41
+ ],
+ [
+ 36,
+ 41,
+ 50,
+ 45,
+ 35,
+ 48
+ ],
+ [
+ 39,
+ 46,
+ 48,
+ 25,
+ 21,
+ 33
+ ],
+ [
+ 30,
+ 38,
+ 11,
+ 22,
+ 15,
+ 9
+ ],
+ [
+ 24,
+ 2,
+ 32,
+ 56,
+ 63,
+ 14
+ ],
+ [
+ 36,
+ 55,
+ 17,
+ 32,
+ 44,
+ 35
+ ],
+ [
+ 59,
+ 46,
+ 32,
+ 44,
+ 24,
+ 14
+ ],
+ [
+ 51,
+ 15,
+ 61,
+ 43,
+ 30,
+ 16
+ ],
+ [
+ 2,
+ 32,
+ 5,
+ 39,
+ 11,
+ 50
+ ],
+ [
+ 28,
+ 42,
+ 6,
+ 30,
+ 57,
+ 37
+ ],
+ [
+ 28,
+ 55,
+ 45,
+ 0,
+ 7,
+ 41
+ ],
+ [
+ 48,
+ 40,
+ 34,
+ 3,
+ 49,
+ 22
+ ],
+ [
+ 25,
+ 6,
+ 62,
+ 50,
+ 18,
+ 53
+ ],
+ [
+ 13,
+ 22,
+ 20,
+ 28,
+ 25,
+ 59
+ ],
+ [
+ 58,
+ 36,
+ 39,
+ 47,
+ 29,
+ 37
+ ],
+ [
+ 4,
+ 33,
+ 41,
+ 12,
+ 3,
+ 17
+ ],
+ [
+ 22,
+ 13,
+ 20,
+ 52,
+ 24,
+ 62
+ ]
+ ],
+ [
+ [
+ 17,
+ 10,
+ 57,
+ 54,
+ 6,
+ 15
+ ],
+ [
+ 33,
+ 43,
+ 13,
+ 1,
+ 16,
+ 62
+ ],
+ [
+ 63,
+ 1,
+ 35,
+ 43,
+ 27,
+ 10
+ ],
+ [
+ 47,
+ 4,
+ 38,
+ 50,
+ 51,
+ 0
+ ],
+ [
+ 11,
+ 51,
+ 57,
+ 23,
+ 14,
+ 34
+ ],
+ [
+ 10,
+ 43,
+ 35,
+ 33,
+ 20,
+ 22
+ ],
+ [
+ 36,
+ 48,
+ 35,
+ 19,
+ 21,
+ 28
+ ],
+ [
+ 14,
+ 8,
+ 7,
+ 46,
+ 35,
+ 13
+ ],
+ [
+ 18,
+ 44,
+ 63,
+ 6,
+ 4,
+ 37
+ ],
+ [
+ 62,
+ 29,
+ 15,
+ 38,
+ 39,
+ 34
+ ],
+ [
+ 1,
+ 6,
+ 16,
+ 46,
+ 22,
+ 13
+ ],
+ [
+ 36,
+ 8,
+ 37,
+ 16,
+ 10,
+ 14
+ ],
+ [
+ 8,
+ 0,
+ 32,
+ 3,
+ 43,
+ 10
+ ],
+ [
+ 25,
+ 0,
+ 22,
+ 30,
+ 60,
+ 57
+ ],
+ [
+ 60,
+ 58,
+ 55,
+ 32,
+ 2,
+ 7
+ ],
+ [
+ 55,
+ 11,
+ 19,
+ 5,
+ 24,
+ 43
+ ],
+ [
+ 9,
+ 47,
+ 36,
+ 39,
+ 5,
+ 42
+ ],
+ [
+ 31,
+ 20,
+ 9,
+ 43,
+ 18,
+ 41
+ ],
+ [
+ 12,
+ 5,
+ 32,
+ 50,
+ 3,
+ 31
+ ],
+ [
+ 40,
+ 18,
+ 30,
+ 63,
+ 7,
+ 25
+ ],
+ [
+ 14,
+ 7,
+ 3,
+ 38,
+ 54,
+ 59
+ ],
+ [
+ 31,
+ 20,
+ 1,
+ 22,
+ 47,
+ 6
+ ],
+ [
+ 51,
+ 15,
+ 18,
+ 53,
+ 40,
+ 0
+ ],
+ [
+ 27,
+ 54,
+ 8,
+ 38,
+ 3,
+ 59
+ ],
+ [
+ 11,
+ 16,
+ 39,
+ 59,
+ 9,
+ 23
+ ],
+ [
+ 7,
+ 37,
+ 51,
+ 30,
+ 18,
+ 3
+ ],
+ [
+ 0,
+ 36,
+ 33,
+ 40,
+ 46,
+ 48
+ ]
+ ],
+ [
+ [
+ 48,
+ 33,
+ 52,
+ 53,
+ 24,
+ 38
+ ],
+ [
+ 3,
+ 40,
+ 45,
+ 19,
+ 22,
+ 35
+ ],
+ [
+ 46,
+ 15,
+ 39,
+ 60,
+ 25,
+ 31
+ ],
+ [
+ 49,
+ 62,
+ 61,
+ 48,
+ 55,
+ 46
+ ],
+ [
+ 41,
+ 35,
+ 53,
+ 52,
+ 13,
+ 10
+ ],
+ [
+ 52,
+ 15,
+ 45,
+ 63,
+ 46,
+ 10
+ ],
+ [
+ 14,
+ 15,
+ 35,
+ 60,
+ 49,
+ 31
+ ],
+ [
+ 44,
+ 3,
+ 6,
+ 8,
+ 23,
+ 48
+ ],
+ [
+ 32,
+ 15,
+ 44,
+ 27,
+ 4,
+ 62
+ ],
+ [
+ 51,
+ 25,
+ 62,
+ 53,
+ 48,
+ 10
+ ],
+ [
+ 8,
+ 19,
+ 5,
+ 16,
+ 22,
+ 50
+ ],
+ [
+ 36,
+ 49,
+ 60,
+ 15,
+ 44,
+ 25
+ ],
+ [
+ 13,
+ 27,
+ 53,
+ 30,
+ 56,
+ 43
+ ],
+ [
+ 26,
+ 22,
+ 7,
+ 14,
+ 32,
+ 17
+ ],
+ [
+ 26,
+ 60,
+ 58,
+ 2,
+ 54,
+ 10
+ ],
+ [
+ 50,
+ 55,
+ 17,
+ 51,
+ 47,
+ 14
+ ],
+ [
+ 0,
+ 12,
+ 16,
+ 4,
+ 23,
+ 9
+ ],
+ [
+ 10,
+ 23,
+ 27,
+ 46,
+ 55,
+ 56
+ ],
+ [
+ 3,
+ 37,
+ 4,
+ 60,
+ 16,
+ 59
+ ],
+ [
+ 38,
+ 3,
+ 29,
+ 40,
+ 25,
+ 50
+ ],
+ [
+ 31,
+ 16,
+ 62,
+ 54,
+ 42,
+ 5
+ ],
+ [
+ 47,
+ 35,
+ 11,
+ 37,
+ 46,
+ 32
+ ],
+ [
+ 27,
+ 51,
+ 20,
+ 50,
+ 55,
+ 16
+ ],
+ [
+ 63,
+ 55,
+ 46,
+ 27,
+ 48,
+ 12
+ ],
+ [
+ 53,
+ 50,
+ 30,
+ 2,
+ 39,
+ 20
+ ],
+ [
+ 53,
+ 44,
+ 24,
+ 8,
+ 51,
+ 14
+ ],
+ [
+ 19,
+ 49,
+ 37,
+ 14,
+ 44,
+ 0
+ ]
+ ],
+ [
+ [
+ 0,
+ 52,
+ 16,
+ 12,
+ 54,
+ 7
+ ],
+ [
+ 42,
+ 25,
+ 51,
+ 61,
+ 35,
+ 58
+ ],
+ [
+ 51,
+ 42,
+ 19,
+ 57,
+ 28,
+ 8
+ ],
+ [
+ 49,
+ 62,
+ 5,
+ 2,
+ 46,
+ 21
+ ],
+ [
+ 3,
+ 41,
+ 53,
+ 25,
+ 39,
+ 37
+ ],
+ [
+ 45,
+ 15,
+ 37,
+ 48,
+ 19,
+ 60
+ ],
+ [
+ 14,
+ 15,
+ 47,
+ 17,
+ 24,
+ 35
+ ],
+ [
+ 3,
+ 52,
+ 63,
+ 16,
+ 28,
+ 47
+ ],
+ [
+ 32,
+ 27,
+ 15,
+ 24,
+ 62,
+ 23
+ ],
+ [
+ 62,
+ 25,
+ 51,
+ 53,
+ 0,
+ 20
+ ],
+ [
+ 8,
+ 19,
+ 50,
+ 16,
+ 22,
+ 32
+ ],
+ [
+ 36,
+ 13,
+ 42,
+ 49,
+ 60,
+ 44
+ ],
+ [
+ 30,
+ 27,
+ 53,
+ 56,
+ 0,
+ 10
+ ],
+ [
+ 22,
+ 7,
+ 14,
+ 26,
+ 32,
+ 17
+ ],
+ [
+ 60,
+ 7,
+ 26,
+ 2,
+ 58,
+ 13
+ ],
+ [
+ 55,
+ 51,
+ 17,
+ 60,
+ 62,
+ 47
+ ],
+ [
+ 53,
+ 16,
+ 18,
+ 4,
+ 50,
+ 5
+ ],
+ [
+ 28,
+ 39,
+ 41,
+ 1,
+ 55,
+ 11
+ ],
+ [
+ 59,
+ 49,
+ 16,
+ 23,
+ 42,
+ 12
+ ],
+ [
+ 17,
+ 30,
+ 46,
+ 25,
+ 55,
+ 36
+ ],
+ [
+ 54,
+ 9,
+ 45,
+ 0,
+ 6,
+ 56
+ ],
+ [
+ 51,
+ 9,
+ 22,
+ 23,
+ 31,
+ 16
+ ],
+ [
+ 62,
+ 50,
+ 43,
+ 51,
+ 55,
+ 27
+ ],
+ [
+ 13,
+ 43,
+ 27,
+ 1,
+ 14,
+ 52
+ ],
+ [
+ 61,
+ 26,
+ 17,
+ 1,
+ 63,
+ 32
+ ],
+ [
+ 37,
+ 46,
+ 63,
+ 20,
+ 9,
+ 24
+ ],
+ [
+ 63,
+ 11,
+ 12,
+ 61,
+ 22,
+ 31
+ ]
+ ],
+ [
+ [
+ 49,
+ 54,
+ 56,
+ 11,
+ 38,
+ 3
+ ],
+ [
+ 53,
+ 46,
+ 49,
+ 38,
+ 57,
+ 17
+ ],
+ [
+ 9,
+ 48,
+ 31,
+ 12,
+ 56,
+ 6
+ ],
+ [
+ 49,
+ 62,
+ 23,
+ 5,
+ 12,
+ 63
+ ],
+ [
+ 36,
+ 26,
+ 38,
+ 7,
+ 20,
+ 23
+ ],
+ [
+ 37,
+ 33,
+ 41,
+ 58,
+ 57,
+ 32
+ ],
+ [
+ 10,
+ 14,
+ 15,
+ 31,
+ 8,
+ 43
+ ],
+ [
+ 16,
+ 52,
+ 3,
+ 2,
+ 34,
+ 14
+ ],
+ [
+ 15,
+ 32,
+ 35,
+ 27,
+ 62,
+ 54
+ ],
+ [
+ 51,
+ 62,
+ 25,
+ 53,
+ 10,
+ 20
+ ],
+ [
+ 19,
+ 8,
+ 22,
+ 50,
+ 1,
+ 5
+ ],
+ [
+ 13,
+ 49,
+ 36,
+ 60,
+ 42,
+ 20
+ ],
+ [
+ 27,
+ 30,
+ 53,
+ 54,
+ 26,
+ 43
+ ],
+ [
+ 14,
+ 32,
+ 26,
+ 22,
+ 7,
+ 17
+ ],
+ [
+ 26,
+ 60,
+ 7,
+ 2,
+ 52,
+ 54
+ ],
+ [
+ 55,
+ 17,
+ 51,
+ 47,
+ 62,
+ 26
+ ],
+ [
+ 16,
+ 18,
+ 44,
+ 4,
+ 53,
+ 50
+ ],
+ [
+ 1,
+ 47,
+ 39,
+ 45,
+ 28,
+ 56
+ ],
+ [
+ 23,
+ 16,
+ 55,
+ 49,
+ 8,
+ 32
+ ],
+ [
+ 17,
+ 55,
+ 30,
+ 62,
+ 31,
+ 23
+ ],
+ [
+ 9,
+ 54,
+ 0,
+ 44,
+ 14,
+ 56
+ ],
+ [
+ 23,
+ 9,
+ 51,
+ 22,
+ 31,
+ 50
+ ],
+ [
+ 50,
+ 27,
+ 14,
+ 51,
+ 18,
+ 19
+ ],
+ [
+ 12,
+ 13,
+ 33,
+ 1,
+ 43,
+ 5
+ ],
+ [
+ 26,
+ 32,
+ 1,
+ 37,
+ 50,
+ 57
+ ],
+ [
+ 37,
+ 47,
+ 63,
+ 46,
+ 5,
+ 4
+ ],
+ [
+ 63,
+ 11,
+ 12,
+ 19,
+ 33,
+ 61
+ ]
+ ],
+ [
+ [
+ 49,
+ 54,
+ 40,
+ 56,
+ 3,
+ 11
+ ],
+ [
+ 37,
+ 15,
+ 12,
+ 33,
+ 59,
+ 17
+ ],
+ [
+ 38,
+ 49,
+ 14,
+ 46,
+ 35,
+ 59
+ ],
+ [
+ 25,
+ 20,
+ 39,
+ 62,
+ 49,
+ 12
+ ],
+ [
+ 26,
+ 51,
+ 16,
+ 36,
+ 18,
+ 8
+ ],
+ [
+ 37,
+ 51,
+ 41,
+ 33,
+ 32,
+ 60
+ ],
+ [
+ 10,
+ 14,
+ 59,
+ 8,
+ 15,
+ 40
+ ],
+ [
+ 16,
+ 52,
+ 19,
+ 61,
+ 32,
+ 2
+ ],
+ [
+ 32,
+ 15,
+ 27,
+ 24,
+ 62,
+ 35
+ ],
+ [
+ 51,
+ 25,
+ 62,
+ 0,
+ 49,
+ 60
+ ],
+ [
+ 8,
+ 50,
+ 19,
+ 48,
+ 16,
+ 20
+ ],
+ [
+ 13,
+ 49,
+ 36,
+ 42,
+ 11,
+ 60
+ ],
+ [
+ 53,
+ 27,
+ 7,
+ 26,
+ 30,
+ 21
+ ],
+ [
+ 14,
+ 22,
+ 26,
+ 37,
+ 32,
+ 4
+ ],
+ [
+ 26,
+ 60,
+ 40,
+ 2,
+ 52,
+ 7
+ ],
+ [
+ 55,
+ 51,
+ 17,
+ 46,
+ 62,
+ 13
+ ],
+ [
+ 38,
+ 16,
+ 53,
+ 44,
+ 4,
+ 18
+ ],
+ [
+ 39,
+ 1,
+ 4,
+ 14,
+ 56,
+ 57
+ ],
+ [
+ 55,
+ 23,
+ 32,
+ 14,
+ 13,
+ 16
+ ],
+ [
+ 55,
+ 17,
+ 2,
+ 30,
+ 62,
+ 12
+ ],
+ [
+ 13,
+ 54,
+ 0,
+ 62,
+ 61,
+ 25
+ ],
+ [
+ 9,
+ 51,
+ 5,
+ 22,
+ 19,
+ 16
+ ],
+ [
+ 50,
+ 51,
+ 7,
+ 19,
+ 48,
+ 53
+ ],
+ [
+ 12,
+ 1,
+ 5,
+ 43,
+ 61,
+ 13
+ ],
+ [
+ 26,
+ 32,
+ 30,
+ 37,
+ 34,
+ 20
+ ],
+ [
+ 37,
+ 63,
+ 46,
+ 4,
+ 47,
+ 8
+ ],
+ [
+ 63,
+ 12,
+ 11,
+ 31,
+ 33,
+ 61
+ ]
+ ],
+ [
+ [
+ 47,
+ 34,
+ 30,
+ 25,
+ 31,
+ 3
+ ],
+ [
+ 15,
+ 24,
+ 46,
+ 21,
+ 8,
+ 6
+ ],
+ [
+ 34,
+ 21,
+ 18,
+ 62,
+ 28,
+ 55
+ ],
+ [
+ 35,
+ 32,
+ 20,
+ 39,
+ 59,
+ 54
+ ],
+ [
+ 26,
+ 27,
+ 15,
+ 48,
+ 60,
+ 47
+ ],
+ [
+ 37,
+ 8,
+ 50,
+ 18,
+ 54,
+ 61
+ ],
+ [
+ 35,
+ 31,
+ 8,
+ 24,
+ 14,
+ 15
+ ],
+ [
+ 16,
+ 52,
+ 34,
+ 29,
+ 48,
+ 36
+ ],
+ [
+ 11,
+ 32,
+ 62,
+ 27,
+ 46,
+ 26
+ ],
+ [
+ 61,
+ 62,
+ 25,
+ 56,
+ 46,
+ 53
+ ],
+ [
+ 56,
+ 50,
+ 63,
+ 3,
+ 45,
+ 28
+ ],
+ [
+ 11,
+ 36,
+ 5,
+ 60,
+ 35,
+ 50
+ ],
+ [
+ 21,
+ 26,
+ 41,
+ 51,
+ 46,
+ 53
+ ],
+ [
+ 14,
+ 22,
+ 33,
+ 19,
+ 41,
+ 16
+ ],
+ [
+ 2,
+ 52,
+ 34,
+ 60,
+ 21,
+ 49
+ ],
+ [
+ 59,
+ 55,
+ 29,
+ 8,
+ 61,
+ 22
+ ],
+ [
+ 51,
+ 44,
+ 2,
+ 59,
+ 47,
+ 53
+ ],
+ [
+ 39,
+ 25,
+ 18,
+ 12,
+ 51,
+ 56
+ ],
+ [
+ 34,
+ 53,
+ 32,
+ 12,
+ 38,
+ 48
+ ],
+ [
+ 30,
+ 53,
+ 56,
+ 7,
+ 40,
+ 62
+ ],
+ [
+ 40,
+ 49,
+ 28,
+ 14,
+ 23,
+ 55
+ ],
+ [
+ 15,
+ 48,
+ 40,
+ 47,
+ 9,
+ 1
+ ],
+ [
+ 50,
+ 41,
+ 53,
+ 25,
+ 18,
+ 0
+ ],
+ [
+ 22,
+ 1,
+ 59,
+ 3,
+ 55,
+ 8
+ ],
+ [
+ 1,
+ 53,
+ 32,
+ 26,
+ 47,
+ 3
+ ],
+ [
+ 4,
+ 33,
+ 28,
+ 37,
+ 55,
+ 54
+ ],
+ [
+ 30,
+ 22,
+ 57,
+ 12,
+ 33,
+ 63
+ ]
+ ],
+ [
+ [
+ 16,
+ 11,
+ 0,
+ 46,
+ 31,
+ 21
+ ],
+ [
+ 49,
+ 13,
+ 31,
+ 5,
+ 11,
+ 14
+ ],
+ [
+ 36,
+ 13,
+ 56,
+ 27,
+ 46,
+ 3
+ ],
+ [
+ 24,
+ 44,
+ 62,
+ 29,
+ 15,
+ 13
+ ],
+ [
+ 17,
+ 2,
+ 50,
+ 8,
+ 45,
+ 1
+ ],
+ [
+ 8,
+ 7,
+ 49,
+ 0,
+ 62,
+ 13
+ ],
+ [
+ 35,
+ 61,
+ 58,
+ 23,
+ 36,
+ 0
+ ],
+ [
+ 16,
+ 48,
+ 42,
+ 4,
+ 32,
+ 1
+ ],
+ [
+ 63,
+ 18,
+ 32,
+ 4,
+ 34,
+ 62
+ ],
+ [
+ 57,
+ 62,
+ 54,
+ 27,
+ 25,
+ 53
+ ],
+ [
+ 1,
+ 59,
+ 60,
+ 29,
+ 22,
+ 14
+ ],
+ [
+ 31,
+ 36,
+ 11,
+ 14,
+ 20,
+ 47
+ ],
+ [
+ 34,
+ 2,
+ 19,
+ 14,
+ 8,
+ 37
+ ],
+ [
+ 57,
+ 22,
+ 40,
+ 14,
+ 62,
+ 48
+ ],
+ [
+ 44,
+ 60,
+ 7,
+ 14,
+ 45,
+ 2
+ ],
+ [
+ 8,
+ 6,
+ 55,
+ 25,
+ 50,
+ 59
+ ],
+ [
+ 13,
+ 47,
+ 23,
+ 61,
+ 42,
+ 39
+ ],
+ [
+ 25,
+ 23,
+ 4,
+ 14,
+ 46,
+ 60
+ ],
+ [
+ 8,
+ 45,
+ 32,
+ 53,
+ 54,
+ 10
+ ],
+ [
+ 15,
+ 38,
+ 53,
+ 55,
+ 30,
+ 7
+ ],
+ [
+ 41,
+ 14,
+ 28,
+ 27,
+ 5,
+ 58
+ ],
+ [
+ 11,
+ 41,
+ 57,
+ 10,
+ 1,
+ 47
+ ],
+ [
+ 50,
+ 0,
+ 51,
+ 53,
+ 34,
+ 55
+ ],
+ [
+ 1,
+ 14,
+ 55,
+ 25,
+ 8,
+ 27
+ ],
+ [
+ 11,
+ 49,
+ 1,
+ 9,
+ 0,
+ 23
+ ],
+ [
+ 50,
+ 51,
+ 6,
+ 42,
+ 4,
+ 54
+ ],
+ [
+ 17,
+ 37,
+ 31,
+ 5,
+ 36,
+ 40
+ ]
+ ],
+ [
+ [
+ 22,
+ 53,
+ 47,
+ 6,
+ 57,
+ 21
+ ],
+ [
+ 27,
+ 11,
+ 14,
+ 6,
+ 57,
+ 16
+ ],
+ [
+ 1,
+ 11,
+ 29,
+ 26,
+ 41,
+ 17
+ ],
+ [
+ 14,
+ 22,
+ 38,
+ 31,
+ 29,
+ 36
+ ],
+ [
+ 14,
+ 59,
+ 29,
+ 61,
+ 45,
+ 52
+ ],
+ [
+ 30,
+ 8,
+ 0,
+ 21,
+ 47,
+ 58
+ ],
+ [
+ 58,
+ 35,
+ 4,
+ 61,
+ 23,
+ 36
+ ],
+ [
+ 42,
+ 20,
+ 48,
+ 16,
+ 9,
+ 4
+ ],
+ [
+ 47,
+ 29,
+ 4,
+ 18,
+ 63,
+ 32
+ ],
+ [
+ 19,
+ 54,
+ 62,
+ 53,
+ 57,
+ 29
+ ],
+ [
+ 1,
+ 60,
+ 14,
+ 59,
+ 29,
+ 40
+ ],
+ [
+ 36,
+ 0,
+ 47,
+ 3,
+ 31,
+ 8
+ ],
+ [
+ 2,
+ 19,
+ 36,
+ 8,
+ 20,
+ 37
+ ],
+ [
+ 57,
+ 22,
+ 40,
+ 49,
+ 31,
+ 14
+ ],
+ [
+ 44,
+ 37,
+ 2,
+ 5,
+ 60,
+ 21
+ ],
+ [
+ 6,
+ 43,
+ 24,
+ 5,
+ 2,
+ 59
+ ],
+ [
+ 13,
+ 61,
+ 19,
+ 47,
+ 50,
+ 39
+ ],
+ [
+ 58,
+ 14,
+ 28,
+ 4,
+ 22,
+ 11
+ ],
+ [
+ 35,
+ 32,
+ 46,
+ 31,
+ 10,
+ 45
+ ],
+ [
+ 15,
+ 13,
+ 55,
+ 45,
+ 18,
+ 63
+ ],
+ [
+ 15,
+ 27,
+ 28,
+ 14,
+ 5,
+ 60
+ ],
+ [
+ 57,
+ 41,
+ 47,
+ 19,
+ 36,
+ 10
+ ],
+ [
+ 34,
+ 10,
+ 53,
+ 55,
+ 22,
+ 19
+ ],
+ [
+ 38,
+ 55,
+ 39,
+ 27,
+ 3,
+ 25
+ ],
+ [
+ 11,
+ 39,
+ 0,
+ 9,
+ 3,
+ 49
+ ],
+ [
+ 51,
+ 6,
+ 43,
+ 18,
+ 50,
+ 53
+ ],
+ [
+ 55,
+ 43,
+ 9,
+ 36,
+ 40,
+ 5
+ ]
+ ],
+ [
+ [
+ 18,
+ 9,
+ 1,
+ 36,
+ 61,
+ 44
+ ],
+ [
+ 56,
+ 34,
+ 19,
+ 42,
+ 3,
+ 5
+ ],
+ [
+ 39,
+ 20,
+ 15,
+ 60,
+ 46,
+ 32
+ ],
+ [
+ 60,
+ 22,
+ 31,
+ 27,
+ 14,
+ 55
+ ],
+ [
+ 59,
+ 58,
+ 10,
+ 7,
+ 46,
+ 18
+ ],
+ [
+ 43,
+ 2,
+ 57,
+ 62,
+ 11,
+ 30
+ ],
+ [
+ 54,
+ 19,
+ 9,
+ 21,
+ 48,
+ 56
+ ],
+ [
+ 46,
+ 24,
+ 7,
+ 14,
+ 3,
+ 8
+ ],
+ [
+ 47,
+ 0,
+ 4,
+ 18,
+ 31,
+ 35
+ ],
+ [
+ 54,
+ 62,
+ 47,
+ 4,
+ 38,
+ 32
+ ],
+ [
+ 1,
+ 14,
+ 15,
+ 22,
+ 38,
+ 59
+ ],
+ [
+ 16,
+ 36,
+ 42,
+ 18,
+ 0,
+ 55
+ ],
+ [
+ 49,
+ 8,
+ 20,
+ 0,
+ 14,
+ 33
+ ],
+ [
+ 18,
+ 39,
+ 25,
+ 2,
+ 22,
+ 62
+ ],
+ [
+ 62,
+ 5,
+ 58,
+ 37,
+ 32,
+ 2
+ ],
+ [
+ 43,
+ 5,
+ 42,
+ 63,
+ 55,
+ 37
+ ],
+ [
+ 47,
+ 33,
+ 15,
+ 63,
+ 50,
+ 12
+ ],
+ [
+ 60,
+ 0,
+ 7,
+ 16,
+ 13,
+ 32
+ ],
+ [
+ 12,
+ 39,
+ 32,
+ 16,
+ 61,
+ 45
+ ],
+ [
+ 52,
+ 34,
+ 62,
+ 15,
+ 18,
+ 30
+ ],
+ [
+ 28,
+ 26,
+ 46,
+ 40,
+ 6,
+ 14
+ ],
+ [
+ 1,
+ 19,
+ 17,
+ 4,
+ 20,
+ 40
+ ],
+ [
+ 41,
+ 40,
+ 4,
+ 53,
+ 19,
+ 55
+ ],
+ [
+ 25,
+ 38,
+ 27,
+ 34,
+ 52,
+ 46
+ ],
+ [
+ 11,
+ 29,
+ 52,
+ 44,
+ 35,
+ 13
+ ],
+ [
+ 50,
+ 51,
+ 41,
+ 16,
+ 4,
+ 14
+ ],
+ [
+ 19,
+ 6,
+ 23,
+ 36,
+ 60,
+ 0
+ ]
+ ],
+ [
+ [
+ 17,
+ 10,
+ 57,
+ 27,
+ 5,
+ 54
+ ],
+ [
+ 33,
+ 9,
+ 43,
+ 40,
+ 56,
+ 11
+ ],
+ [
+ 63,
+ 1,
+ 35,
+ 43,
+ 10,
+ 27
+ ],
+ [
+ 51,
+ 47,
+ 20,
+ 21,
+ 28,
+ 61
+ ],
+ [
+ 25,
+ 11,
+ 58,
+ 23,
+ 55,
+ 46
+ ],
+ [
+ 43,
+ 10,
+ 12,
+ 2,
+ 30,
+ 62
+ ],
+ [
+ 48,
+ 21,
+ 19,
+ 8,
+ 7,
+ 54
+ ],
+ [
+ 14,
+ 7,
+ 24,
+ 8,
+ 46,
+ 2
+ ],
+ [
+ 4,
+ 47,
+ 37,
+ 0,
+ 44,
+ 27
+ ],
+ [
+ 54,
+ 38,
+ 62,
+ 47,
+ 15,
+ 14
+ ],
+ [
+ 1,
+ 46,
+ 15,
+ 22,
+ 51,
+ 38
+ ],
+ [
+ 36,
+ 16,
+ 42,
+ 24,
+ 55,
+ 37
+ ],
+ [
+ 49,
+ 10,
+ 0,
+ 3,
+ 43,
+ 17
+ ],
+ [
+ 39,
+ 58,
+ 0,
+ 62,
+ 22,
+ 25
+ ],
+ [
+ 58,
+ 38,
+ 7,
+ 62,
+ 55,
+ 56
+ ],
+ [
+ 42,
+ 19,
+ 55,
+ 43,
+ 11,
+ 37
+ ],
+ [
+ 9,
+ 47,
+ 43,
+ 18,
+ 52,
+ 50
+ ],
+ [
+ 31,
+ 41,
+ 32,
+ 25,
+ 20,
+ 13
+ ],
+ [
+ 12,
+ 32,
+ 61,
+ 3,
+ 43,
+ 21
+ ],
+ [
+ 36,
+ 13,
+ 40,
+ 7,
+ 16,
+ 62
+ ],
+ [
+ 14,
+ 53,
+ 50,
+ 47,
+ 51,
+ 1
+ ],
+ [
+ 1,
+ 38,
+ 19,
+ 30,
+ 18,
+ 16
+ ],
+ [
+ 0,
+ 19,
+ 51,
+ 18,
+ 52,
+ 15
+ ],
+ [
+ 8,
+ 52,
+ 27,
+ 34,
+ 38,
+ 3
+ ],
+ [
+ 27,
+ 53,
+ 59,
+ 9,
+ 40,
+ 4
+ ],
+ [
+ 37,
+ 3,
+ 26,
+ 48,
+ 8,
+ 43
+ ],
+ [
+ 46,
+ 18,
+ 11,
+ 40,
+ 33,
+ 44
+ ]
+ ],
+ [
+ [
+ 48,
+ 62,
+ 61,
+ 26,
+ 0,
+ 50
+ ],
+ [
+ 3,
+ 45,
+ 40,
+ 35,
+ 29,
+ 54
+ ],
+ [
+ 56,
+ 31,
+ 23,
+ 28,
+ 2,
+ 53
+ ],
+ [
+ 62,
+ 49,
+ 20,
+ 61,
+ 6,
+ 41
+ ],
+ [
+ 18,
+ 25,
+ 50,
+ 0,
+ 14,
+ 57
+ ],
+ [
+ 58,
+ 4,
+ 10,
+ 43,
+ 56,
+ 46
+ ],
+ [
+ 35,
+ 15,
+ 25,
+ 24,
+ 3,
+ 7
+ ],
+ [
+ 14,
+ 23,
+ 8,
+ 12,
+ 7,
+ 24
+ ],
+ [
+ 29,
+ 17,
+ 35,
+ 44,
+ 24,
+ 4
+ ],
+ [
+ 62,
+ 15,
+ 38,
+ 20,
+ 58,
+ 40
+ ],
+ [
+ 19,
+ 1,
+ 46,
+ 22,
+ 26,
+ 63
+ ],
+ [
+ 36,
+ 60,
+ 42,
+ 16,
+ 11,
+ 55
+ ],
+ [
+ 17,
+ 7,
+ 16,
+ 49,
+ 14,
+ 26
+ ],
+ [
+ 45,
+ 47,
+ 22,
+ 0,
+ 58,
+ 62
+ ],
+ [
+ 58,
+ 38,
+ 48,
+ 63,
+ 2,
+ 49
+ ],
+ [
+ 55,
+ 0,
+ 1,
+ 37,
+ 30,
+ 10
+ ],
+ [
+ 12,
+ 43,
+ 21,
+ 9,
+ 23,
+ 47
+ ],
+ [
+ 32,
+ 57,
+ 42,
+ 63,
+ 25,
+ 43
+ ],
+ [
+ 3,
+ 32,
+ 49,
+ 61,
+ 12,
+ 21
+ ],
+ [
+ 5,
+ 36,
+ 16,
+ 22,
+ 62,
+ 42
+ ],
+ [
+ 53,
+ 7,
+ 46,
+ 61,
+ 14,
+ 52
+ ],
+ [
+ 55,
+ 30,
+ 3,
+ 5,
+ 31,
+ 53
+ ],
+ [
+ 0,
+ 44,
+ 15,
+ 18,
+ 19,
+ 28
+ ],
+ [
+ 8,
+ 52,
+ 51,
+ 11,
+ 28,
+ 29
+ ],
+ [
+ 27,
+ 40,
+ 4,
+ 9,
+ 35,
+ 39
+ ],
+ [
+ 14,
+ 26,
+ 3,
+ 16,
+ 48,
+ 21
+ ],
+ [
+ 60,
+ 54,
+ 35,
+ 20,
+ 53,
+ 12
+ ]
+ ],
+ [
+ [
+ 21,
+ 50,
+ 29,
+ 41,
+ 34,
+ 60
+ ],
+ [
+ 28,
+ 51,
+ 60,
+ 33,
+ 14,
+ 45
+ ],
+ [
+ 31,
+ 2,
+ 46,
+ 33,
+ 24,
+ 49
+ ],
+ [
+ 42,
+ 3,
+ 18,
+ 62,
+ 39,
+ 49
+ ],
+ [
+ 25,
+ 15,
+ 62,
+ 27,
+ 12,
+ 11
+ ],
+ [
+ 44,
+ 50,
+ 36,
+ 57,
+ 55,
+ 41
+ ],
+ [
+ 41,
+ 37,
+ 22,
+ 15,
+ 2,
+ 40
+ ],
+ [
+ 36,
+ 62,
+ 53,
+ 30,
+ 14,
+ 57
+ ],
+ [
+ 16,
+ 58,
+ 2,
+ 29,
+ 4,
+ 3
+ ],
+ [
+ 41,
+ 38,
+ 26,
+ 16,
+ 45,
+ 46
+ ],
+ [
+ 45,
+ 46,
+ 32,
+ 41,
+ 56,
+ 26
+ ],
+ [
+ 17,
+ 53,
+ 21,
+ 11,
+ 36,
+ 35
+ ],
+ [
+ 11,
+ 16,
+ 28,
+ 14,
+ 51,
+ 39
+ ],
+ [
+ 9,
+ 35,
+ 33,
+ 22,
+ 52,
+ 62
+ ],
+ [
+ 58,
+ 50,
+ 63,
+ 30,
+ 7,
+ 27
+ ],
+ [
+ 55,
+ 3,
+ 8,
+ 41,
+ 63,
+ 37
+ ],
+ [
+ 3,
+ 51,
+ 32,
+ 46,
+ 15,
+ 6
+ ],
+ [
+ 32,
+ 12,
+ 10,
+ 25,
+ 5,
+ 16
+ ],
+ [
+ 34,
+ 2,
+ 37,
+ 61,
+ 63,
+ 39
+ ],
+ [
+ 42,
+ 22,
+ 27,
+ 53,
+ 11,
+ 56
+ ],
+ [
+ 53,
+ 12,
+ 0,
+ 47,
+ 61,
+ 49
+ ],
+ [
+ 39,
+ 45,
+ 53,
+ 17,
+ 48,
+ 14
+ ],
+ [
+ 6,
+ 0,
+ 4,
+ 53,
+ 25,
+ 11
+ ],
+ [
+ 51,
+ 11,
+ 1,
+ 63,
+ 54,
+ 59
+ ],
+ [
+ 40,
+ 56,
+ 37,
+ 53,
+ 35,
+ 5
+ ],
+ [
+ 59,
+ 28,
+ 41,
+ 10,
+ 1,
+ 45
+ ],
+ [
+ 27,
+ 30,
+ 28,
+ 24,
+ 32,
+ 57
+ ]
+ ],
+ [
+ [
+ 24,
+ 56,
+ 6,
+ 0,
+ 19,
+ 45
+ ],
+ [
+ 11,
+ 57,
+ 59,
+ 25,
+ 46,
+ 30
+ ],
+ [
+ 11,
+ 26,
+ 37,
+ 29,
+ 14,
+ 52
+ ],
+ [
+ 3,
+ 32,
+ 7,
+ 38,
+ 36,
+ 24
+ ],
+ [
+ 61,
+ 2,
+ 24,
+ 14,
+ 51,
+ 44
+ ],
+ [
+ 20,
+ 47,
+ 0,
+ 63,
+ 30,
+ 58
+ ],
+ [
+ 4,
+ 36,
+ 29,
+ 58,
+ 16,
+ 3
+ ],
+ [
+ 20,
+ 0,
+ 45,
+ 14,
+ 28,
+ 44
+ ],
+ [
+ 29,
+ 56,
+ 47,
+ 35,
+ 16,
+ 4
+ ],
+ [
+ 33,
+ 61,
+ 55,
+ 41,
+ 51,
+ 38
+ ],
+ [
+ 58,
+ 1,
+ 38,
+ 14,
+ 4,
+ 19
+ ],
+ [
+ 0,
+ 36,
+ 14,
+ 18,
+ 52,
+ 42
+ ],
+ [
+ 29,
+ 36,
+ 45,
+ 25,
+ 8,
+ 6
+ ],
+ [
+ 6,
+ 57,
+ 50,
+ 40,
+ 58,
+ 61
+ ],
+ [
+ 44,
+ 58,
+ 29,
+ 19,
+ 61,
+ 56
+ ],
+ [
+ 23,
+ 18,
+ 28,
+ 55,
+ 37,
+ 5
+ ],
+ [
+ 13,
+ 9,
+ 19,
+ 43,
+ 37,
+ 3
+ ],
+ [
+ 32,
+ 22,
+ 63,
+ 14,
+ 57,
+ 41
+ ],
+ [
+ 10,
+ 61,
+ 3,
+ 1,
+ 19,
+ 32
+ ],
+ [
+ 16,
+ 55,
+ 10,
+ 41,
+ 59,
+ 22
+ ],
+ [
+ 53,
+ 7,
+ 29,
+ 38,
+ 27,
+ 46
+ ],
+ [
+ 24,
+ 47,
+ 18,
+ 53,
+ 39,
+ 30
+ ],
+ [
+ 0,
+ 33,
+ 19,
+ 51,
+ 5,
+ 17
+ ],
+ [
+ 51,
+ 8,
+ 11,
+ 45,
+ 44,
+ 41
+ ],
+ [
+ 40,
+ 4,
+ 23,
+ 11,
+ 27,
+ 19
+ ],
+ [
+ 16,
+ 18,
+ 3,
+ 48,
+ 51,
+ 21
+ ],
+ [
+ 43,
+ 46,
+ 60,
+ 19,
+ 53,
+ 12
+ ]
+ ],
+ [
+ [
+ 48,
+ 62,
+ 61,
+ 30,
+ 0,
+ 2
+ ],
+ [
+ 45,
+ 3,
+ 35,
+ 54,
+ 29,
+ 2
+ ],
+ [
+ 56,
+ 31,
+ 53,
+ 23,
+ 49,
+ 28
+ ],
+ [
+ 60,
+ 14,
+ 57,
+ 46,
+ 41,
+ 48
+ ],
+ [
+ 18,
+ 61,
+ 59,
+ 14,
+ 44,
+ 32
+ ],
+ [
+ 45,
+ 58,
+ 47,
+ 20,
+ 4,
+ 30
+ ],
+ [
+ 54,
+ 13,
+ 25,
+ 36,
+ 26,
+ 47
+ ],
+ [
+ 20,
+ 12,
+ 0,
+ 47,
+ 30,
+ 45
+ ],
+ [
+ 56,
+ 29,
+ 47,
+ 17,
+ 35,
+ 16
+ ],
+ [
+ 33,
+ 61,
+ 55,
+ 11,
+ 38,
+ 48
+ ],
+ [
+ 58,
+ 19,
+ 14,
+ 1,
+ 38,
+ 36
+ ],
+ [
+ 14,
+ 36,
+ 0,
+ 60,
+ 11,
+ 52
+ ],
+ [
+ 29,
+ 44,
+ 7,
+ 36,
+ 16,
+ 45
+ ],
+ [
+ 6,
+ 47,
+ 50,
+ 33,
+ 42,
+ 62
+ ],
+ [
+ 44,
+ 58,
+ 61,
+ 38,
+ 29,
+ 56
+ ],
+ [
+ 23,
+ 55,
+ 18,
+ 0,
+ 57,
+ 37
+ ],
+ [
+ 9,
+ 12,
+ 43,
+ 19,
+ 13,
+ 6
+ ],
+ [
+ 32,
+ 22,
+ 63,
+ 57,
+ 42,
+ 29
+ ],
+ [
+ 3,
+ 61,
+ 1,
+ 10,
+ 49,
+ 32
+ ],
+ [
+ 5,
+ 16,
+ 55,
+ 36,
+ 22,
+ 59
+ ],
+ [
+ 53,
+ 7,
+ 29,
+ 46,
+ 9,
+ 14
+ ],
+ [
+ 24,
+ 30,
+ 18,
+ 39,
+ 55,
+ 53
+ ],
+ [
+ 33,
+ 0,
+ 19,
+ 44,
+ 51,
+ 5
+ ],
+ [
+ 51,
+ 8,
+ 53,
+ 41,
+ 4,
+ 11
+ ],
+ [
+ 40,
+ 4,
+ 27,
+ 19,
+ 23,
+ 16
+ ],
+ [
+ 16,
+ 14,
+ 48,
+ 3,
+ 21,
+ 26
+ ],
+ [
+ 60,
+ 54,
+ 35,
+ 53,
+ 12,
+ 43
+ ]
+ ],
+ [
+ [
+ 19,
+ 41,
+ 8,
+ 7,
+ 13,
+ 2
+ ],
+ [
+ 48,
+ 46,
+ 62,
+ 29,
+ 5,
+ 41
+ ],
+ [
+ 12,
+ 5,
+ 59,
+ 3,
+ 58,
+ 49
+ ],
+ [
+ 60,
+ 3,
+ 42,
+ 39,
+ 14,
+ 18
+ ],
+ [
+ 42,
+ 12,
+ 27,
+ 11,
+ 25,
+ 19
+ ],
+ [
+ 50,
+ 36,
+ 44,
+ 26,
+ 33,
+ 37
+ ],
+ [
+ 41,
+ 54,
+ 22,
+ 52,
+ 37,
+ 35
+ ],
+ [
+ 62,
+ 30,
+ 36,
+ 53,
+ 10,
+ 14
+ ],
+ [
+ 2,
+ 16,
+ 58,
+ 29,
+ 7,
+ 41
+ ],
+ [
+ 18,
+ 32,
+ 45,
+ 16,
+ 22,
+ 38
+ ],
+ [
+ 45,
+ 56,
+ 41,
+ 10,
+ 3,
+ 46
+ ],
+ [
+ 50,
+ 21,
+ 36,
+ 35,
+ 53,
+ 12
+ ],
+ [
+ 11,
+ 28,
+ 16,
+ 41,
+ 39,
+ 46
+ ],
+ [
+ 16,
+ 9,
+ 33,
+ 38,
+ 28,
+ 19
+ ],
+ [
+ 58,
+ 50,
+ 63,
+ 62,
+ 27,
+ 52
+ ],
+ [
+ 55,
+ 59,
+ 13,
+ 8,
+ 3,
+ 41
+ ],
+ [
+ 3,
+ 51,
+ 15,
+ 46,
+ 57,
+ 47
+ ],
+ [
+ 32,
+ 30,
+ 10,
+ 12,
+ 25,
+ 18
+ ],
+ [
+ 34,
+ 2,
+ 27,
+ 61,
+ 53,
+ 59
+ ],
+ [
+ 42,
+ 22,
+ 56,
+ 53,
+ 44,
+ 34
+ ],
+ [
+ 53,
+ 12,
+ 49,
+ 41,
+ 44,
+ 8
+ ],
+ [
+ 45,
+ 1,
+ 48,
+ 47,
+ 16,
+ 17
+ ],
+ [
+ 4,
+ 0,
+ 25,
+ 53,
+ 24,
+ 11
+ ],
+ [
+ 1,
+ 44,
+ 11,
+ 45,
+ 34,
+ 51
+ ],
+ [
+ 40,
+ 5,
+ 53,
+ 6,
+ 22,
+ 18
+ ],
+ [
+ 28,
+ 10,
+ 1,
+ 15,
+ 3,
+ 4
+ ],
+ [
+ 30,
+ 27,
+ 24,
+ 57,
+ 32,
+ 16
+ ]
+ ],
+ [
+ [
+ 24,
+ 56,
+ 6,
+ 0,
+ 19,
+ 45
+ ],
+ [
+ 11,
+ 57,
+ 59,
+ 46,
+ 25,
+ 30
+ ],
+ [
+ 26,
+ 11,
+ 37,
+ 14,
+ 29,
+ 49
+ ],
+ [
+ 38,
+ 36,
+ 3,
+ 24,
+ 18,
+ 20
+ ],
+ [
+ 61,
+ 51,
+ 14,
+ 2,
+ 24,
+ 1
+ ],
+ [
+ 20,
+ 0,
+ 47,
+ 30,
+ 8,
+ 35
+ ],
+ [
+ 4,
+ 58,
+ 36,
+ 54,
+ 29,
+ 12
+ ],
+ [
+ 20,
+ 58,
+ 44,
+ 28,
+ 45,
+ 9
+ ],
+ [
+ 56,
+ 47,
+ 10,
+ 29,
+ 35,
+ 27
+ ],
+ [
+ 61,
+ 33,
+ 55,
+ 54,
+ 4,
+ 36
+ ],
+ [
+ 58,
+ 1,
+ 14,
+ 4,
+ 38,
+ 52
+ ],
+ [
+ 14,
+ 0,
+ 36,
+ 63,
+ 15,
+ 23
+ ],
+ [
+ 29,
+ 36,
+ 44,
+ 8,
+ 16,
+ 2
+ ],
+ [
+ 6,
+ 40,
+ 27,
+ 57,
+ 50,
+ 42
+ ],
+ [
+ 19,
+ 44,
+ 58,
+ 61,
+ 38,
+ 37
+ ],
+ [
+ 23,
+ 18,
+ 17,
+ 57,
+ 13,
+ 40
+ ],
+ [
+ 13,
+ 9,
+ 19,
+ 37,
+ 15,
+ 50
+ ],
+ [
+ 32,
+ 14,
+ 57,
+ 58,
+ 29,
+ 22
+ ],
+ [
+ 10,
+ 61,
+ 1,
+ 3,
+ 14,
+ 59
+ ],
+ [
+ 55,
+ 16,
+ 34,
+ 18,
+ 22,
+ 49
+ ],
+ [
+ 53,
+ 27,
+ 38,
+ 28,
+ 23,
+ 44
+ ],
+ [
+ 24,
+ 47,
+ 18,
+ 41,
+ 62,
+ 30
+ ],
+ [
+ 33,
+ 51,
+ 19,
+ 5,
+ 0,
+ 44
+ ],
+ [
+ 51,
+ 8,
+ 25,
+ 53,
+ 27,
+ 16
+ ],
+ [
+ 40,
+ 11,
+ 27,
+ 4,
+ 23,
+ 19
+ ],
+ [
+ 16,
+ 18,
+ 51,
+ 48,
+ 3,
+ 47
+ ],
+ [
+ 46,
+ 43,
+ 36,
+ 9,
+ 5,
+ 12
+ ]
+ ],
+ [
+ [
+ 37,
+ 10,
+ 46,
+ 60,
+ 61,
+ 55
+ ],
+ [
+ 35,
+ 53,
+ 34,
+ 43,
+ 19,
+ 57
+ ],
+ [
+ 49,
+ 56,
+ 45,
+ 30,
+ 6,
+ 12
+ ],
+ [
+ 60,
+ 27,
+ 14,
+ 48,
+ 46,
+ 57
+ ],
+ [
+ 61,
+ 59,
+ 14,
+ 41,
+ 16,
+ 1
+ ],
+ [
+ 45,
+ 4,
+ 3,
+ 24,
+ 58,
+ 47
+ ],
+ [
+ 54,
+ 13,
+ 9,
+ 43,
+ 16,
+ 26
+ ],
+ [
+ 47,
+ 23,
+ 12,
+ 20,
+ 63,
+ 30
+ ],
+ [
+ 23,
+ 44,
+ 56,
+ 29,
+ 47,
+ 17
+ ],
+ [
+ 33,
+ 60,
+ 61,
+ 48,
+ 41,
+ 14
+ ],
+ [
+ 58,
+ 63,
+ 19,
+ 11,
+ 38,
+ 9
+ ],
+ [
+ 60,
+ 63,
+ 0,
+ 36,
+ 15,
+ 9
+ ],
+ [
+ 29,
+ 36,
+ 30,
+ 59,
+ 11,
+ 27
+ ],
+ [
+ 6,
+ 7,
+ 47,
+ 62,
+ 50,
+ 57
+ ],
+ [
+ 27,
+ 58,
+ 19,
+ 46,
+ 29,
+ 56
+ ],
+ [
+ 29,
+ 60,
+ 56,
+ 55,
+ 23,
+ 26
+ ],
+ [
+ 53,
+ 59,
+ 6,
+ 9,
+ 43,
+ 16
+ ],
+ [
+ 41,
+ 32,
+ 57,
+ 63,
+ 18,
+ 37
+ ],
+ [
+ 42,
+ 61,
+ 3,
+ 10,
+ 34,
+ 40
+ ],
+ [
+ 4,
+ 43,
+ 17,
+ 16,
+ 52,
+ 60
+ ],
+ [
+ 45,
+ 53,
+ 61,
+ 56,
+ 16,
+ 7
+ ],
+ [
+ 55,
+ 9,
+ 18,
+ 61,
+ 45,
+ 60
+ ],
+ [
+ 60,
+ 47,
+ 53,
+ 33,
+ 12,
+ 27
+ ],
+ [
+ 43,
+ 51,
+ 11,
+ 8,
+ 45,
+ 63
+ ],
+ [
+ 6,
+ 40,
+ 15,
+ 27,
+ 26,
+ 23
+ ],
+ [
+ 49,
+ 14,
+ 9,
+ 58,
+ 21,
+ 63
+ ],
+ [
+ 63,
+ 24,
+ 60,
+ 31,
+ 12,
+ 34
+ ]
+ ],
+ [
+ [
+ 16,
+ 13,
+ 44,
+ 4,
+ 23,
+ 46
+ ],
+ [
+ 16,
+ 50,
+ 9,
+ 13,
+ 23,
+ 36
+ ],
+ [
+ 11,
+ 35,
+ 21,
+ 7,
+ 59,
+ 8
+ ],
+ [
+ 1,
+ 3,
+ 25,
+ 15,
+ 60,
+ 39
+ ],
+ [
+ 54,
+ 61,
+ 31,
+ 35,
+ 55,
+ 1
+ ],
+ [
+ 51,
+ 52,
+ 46,
+ 15,
+ 4,
+ 45
+ ],
+ [
+ 60,
+ 54,
+ 59,
+ 44,
+ 10,
+ 7
+ ],
+ [
+ 12,
+ 22,
+ 14,
+ 47,
+ 0,
+ 30
+ ],
+ [
+ 42,
+ 29,
+ 23,
+ 56,
+ 33,
+ 47
+ ],
+ [
+ 33,
+ 61,
+ 20,
+ 60,
+ 0,
+ 49
+ ],
+ [
+ 35,
+ 58,
+ 63,
+ 14,
+ 51,
+ 24
+ ],
+ [
+ 29,
+ 33,
+ 36,
+ 60,
+ 0,
+ 49
+ ],
+ [
+ 29,
+ 17,
+ 30,
+ 31,
+ 12,
+ 36
+ ],
+ [
+ 6,
+ 0,
+ 61,
+ 48,
+ 50,
+ 3
+ ],
+ [
+ 8,
+ 6,
+ 58,
+ 29,
+ 37,
+ 19
+ ],
+ [
+ 60,
+ 39,
+ 27,
+ 19,
+ 1,
+ 57
+ ],
+ [
+ 56,
+ 9,
+ 30,
+ 6,
+ 43,
+ 10
+ ],
+ [
+ 32,
+ 20,
+ 13,
+ 57,
+ 63,
+ 49
+ ],
+ [
+ 8,
+ 42,
+ 21,
+ 61,
+ 37,
+ 4
+ ],
+ [
+ 45,
+ 49,
+ 16,
+ 13,
+ 2,
+ 58
+ ],
+ [
+ 53,
+ 29,
+ 14,
+ 50,
+ 61,
+ 3
+ ],
+ [
+ 37,
+ 57,
+ 27,
+ 54,
+ 46,
+ 9
+ ],
+ [
+ 52,
+ 19,
+ 22,
+ 0,
+ 18,
+ 5
+ ],
+ [
+ 14,
+ 49,
+ 30,
+ 33,
+ 53,
+ 34
+ ],
+ [
+ 13,
+ 9,
+ 4,
+ 40,
+ 23,
+ 39
+ ],
+ [
+ 27,
+ 43,
+ 47,
+ 36,
+ 49,
+ 3
+ ],
+ [
+ 59,
+ 43,
+ 40,
+ 28,
+ 0,
+ 33
+ ]
+ ],
+ [
+ [
+ 48,
+ 50,
+ 42,
+ 63,
+ 34,
+ 38
+ ],
+ [
+ 3,
+ 40,
+ 61,
+ 62,
+ 2,
+ 6
+ ],
+ [
+ 39,
+ 7,
+ 36,
+ 6,
+ 45,
+ 40
+ ],
+ [
+ 41,
+ 35,
+ 46,
+ 13,
+ 63,
+ 56
+ ],
+ [
+ 6,
+ 1,
+ 54,
+ 37,
+ 38,
+ 34
+ ],
+ [
+ 59,
+ 46,
+ 51,
+ 31,
+ 4,
+ 52
+ ],
+ [
+ 60,
+ 44,
+ 11,
+ 54,
+ 4,
+ 24
+ ],
+ [
+ 12,
+ 0,
+ 2,
+ 63,
+ 50,
+ 47
+ ],
+ [
+ 33,
+ 42,
+ 29,
+ 23,
+ 16,
+ 56
+ ],
+ [
+ 20,
+ 61,
+ 33,
+ 60,
+ 53,
+ 0
+ ],
+ [
+ 35,
+ 58,
+ 63,
+ 9,
+ 8,
+ 19
+ ],
+ [
+ 59,
+ 48,
+ 36,
+ 60,
+ 10,
+ 14
+ ],
+ [
+ 29,
+ 44,
+ 7,
+ 17,
+ 36,
+ 12
+ ],
+ [
+ 47,
+ 27,
+ 6,
+ 62,
+ 42,
+ 48
+ ],
+ [
+ 18,
+ 58,
+ 49,
+ 46,
+ 42,
+ 38
+ ],
+ [
+ 60,
+ 34,
+ 27,
+ 18,
+ 23,
+ 55
+ ],
+ [
+ 43,
+ 40,
+ 9,
+ 50,
+ 18,
+ 45
+ ],
+ [
+ 32,
+ 57,
+ 48,
+ 42,
+ 29,
+ 63
+ ],
+ [
+ 42,
+ 61,
+ 49,
+ 3,
+ 32,
+ 1
+ ],
+ [
+ 23,
+ 37,
+ 1,
+ 16,
+ 36,
+ 39
+ ],
+ [
+ 53,
+ 21,
+ 7,
+ 61,
+ 50,
+ 31
+ ],
+ [
+ 8,
+ 60,
+ 18,
+ 24,
+ 9,
+ 30
+ ],
+ [
+ 51,
+ 33,
+ 28,
+ 5,
+ 8,
+ 44
+ ],
+ [
+ 51,
+ 52,
+ 8,
+ 4,
+ 41,
+ 45
+ ],
+ [
+ 40,
+ 4,
+ 27,
+ 9,
+ 60,
+ 19
+ ],
+ [
+ 3,
+ 61,
+ 16,
+ 26,
+ 48,
+ 12
+ ],
+ [
+ 54,
+ 61,
+ 35,
+ 1,
+ 53,
+ 20
+ ]
+ ],
+ [
+ [
+ 62,
+ 28,
+ 1,
+ 42,
+ 8,
+ 55
+ ],
+ [
+ 18,
+ 12,
+ 8,
+ 41,
+ 40,
+ 31
+ ],
+ [
+ 12,
+ 6,
+ 50,
+ 4,
+ 23,
+ 45
+ ],
+ [
+ 43,
+ 35,
+ 8,
+ 20,
+ 42,
+ 46
+ ],
+ [
+ 39,
+ 41,
+ 29,
+ 22,
+ 3,
+ 56
+ ],
+ [
+ 61,
+ 45,
+ 46,
+ 48,
+ 28,
+ 51
+ ],
+ [
+ 44,
+ 4,
+ 11,
+ 25,
+ 54,
+ 59
+ ],
+ [
+ 12,
+ 33,
+ 56,
+ 52,
+ 30,
+ 17
+ ],
+ [
+ 55,
+ 29,
+ 17,
+ 42,
+ 23,
+ 14
+ ],
+ [
+ 60,
+ 12,
+ 18,
+ 61,
+ 33,
+ 28
+ ],
+ [
+ 35,
+ 58,
+ 37,
+ 63,
+ 6,
+ 27
+ ],
+ [
+ 48,
+ 59,
+ 10,
+ 36,
+ 60,
+ 58
+ ],
+ [
+ 17,
+ 7,
+ 28,
+ 31,
+ 29,
+ 44
+ ],
+ [
+ 47,
+ 42,
+ 50,
+ 6,
+ 8,
+ 14
+ ],
+ [
+ 39,
+ 58,
+ 37,
+ 59,
+ 56,
+ 18
+ ],
+ [
+ 60,
+ 57,
+ 18,
+ 9,
+ 55,
+ 23
+ ],
+ [
+ 43,
+ 63,
+ 18,
+ 60,
+ 22,
+ 19
+ ],
+ [
+ 32,
+ 1,
+ 42,
+ 57,
+ 35,
+ 63
+ ],
+ [
+ 42,
+ 61,
+ 3,
+ 32,
+ 1,
+ 50
+ ],
+ [
+ 37,
+ 36,
+ 10,
+ 23,
+ 16,
+ 57
+ ],
+ [
+ 53,
+ 61,
+ 7,
+ 57,
+ 21,
+ 23
+ ],
+ [
+ 9,
+ 39,
+ 30,
+ 18,
+ 14,
+ 8
+ ],
+ [
+ 33,
+ 44,
+ 8,
+ 5,
+ 0,
+ 56
+ ],
+ [
+ 51,
+ 53,
+ 49,
+ 52,
+ 4,
+ 41
+ ],
+ [
+ 40,
+ 27,
+ 4,
+ 6,
+ 9,
+ 16
+ ],
+ [
+ 3,
+ 16,
+ 48,
+ 26,
+ 12,
+ 4
+ ],
+ [
+ 61,
+ 14,
+ 12,
+ 54,
+ 35,
+ 53
+ ]
+ ],
+ [
+ [
+ 47,
+ 23,
+ 63,
+ 11,
+ 61,
+ 55
+ ],
+ [
+ 17,
+ 44,
+ 28,
+ 39,
+ 47,
+ 27
+ ],
+ [
+ 34,
+ 53,
+ 50,
+ 38,
+ 29,
+ 5
+ ],
+ [
+ 11,
+ 10,
+ 17,
+ 52,
+ 47,
+ 42
+ ],
+ [
+ 15,
+ 41,
+ 27,
+ 20,
+ 12,
+ 6
+ ],
+ [
+ 34,
+ 44,
+ 50,
+ 39,
+ 36,
+ 61
+ ],
+ [
+ 37,
+ 41,
+ 52,
+ 29,
+ 46,
+ 47
+ ],
+ [
+ 62,
+ 36,
+ 34,
+ 30,
+ 39,
+ 22
+ ],
+ [
+ 62,
+ 16,
+ 58,
+ 5,
+ 2,
+ 8
+ ],
+ [
+ 32,
+ 41,
+ 56,
+ 12,
+ 46,
+ 8
+ ],
+ [
+ 10,
+ 35,
+ 45,
+ 41,
+ 3,
+ 56
+ ],
+ [
+ 50,
+ 48,
+ 35,
+ 53,
+ 36,
+ 12
+ ],
+ [
+ 39,
+ 11,
+ 46,
+ 7,
+ 23,
+ 51
+ ],
+ [
+ 9,
+ 47,
+ 19,
+ 22,
+ 52,
+ 34
+ ],
+ [
+ 35,
+ 18,
+ 56,
+ 50,
+ 3,
+ 23
+ ],
+ [
+ 3,
+ 60,
+ 38,
+ 36,
+ 9,
+ 58
+ ],
+ [
+ 46,
+ 28,
+ 32,
+ 5,
+ 43,
+ 56
+ ],
+ [
+ 30,
+ 32,
+ 57,
+ 42,
+ 52,
+ 1
+ ],
+ [
+ 2,
+ 34,
+ 32,
+ 61,
+ 14,
+ 42
+ ],
+ [
+ 42,
+ 37,
+ 20,
+ 48,
+ 50,
+ 9
+ ],
+ [
+ 53,
+ 7,
+ 56,
+ 25,
+ 13,
+ 60
+ ],
+ [
+ 17,
+ 39,
+ 14,
+ 53,
+ 30,
+ 25
+ ],
+ [
+ 5,
+ 40,
+ 6,
+ 33,
+ 29,
+ 35
+ ],
+ [
+ 51,
+ 4,
+ 11,
+ 57,
+ 58,
+ 28
+ ],
+ [
+ 40,
+ 37,
+ 4,
+ 44,
+ 8,
+ 48
+ ],
+ [
+ 3,
+ 48,
+ 26,
+ 9,
+ 12,
+ 41
+ ],
+ [
+ 47,
+ 61,
+ 26,
+ 24,
+ 20,
+ 53
+ ]
+ ],
+ [
+ [
+ 31,
+ 43,
+ 41,
+ 47,
+ 11,
+ 25
+ ],
+ [
+ 50,
+ 25,
+ 31,
+ 40,
+ 24,
+ 46
+ ],
+ [
+ 23,
+ 9,
+ 62,
+ 15,
+ 20,
+ 53
+ ],
+ [
+ 4,
+ 47,
+ 44,
+ 58,
+ 48,
+ 25
+ ],
+ [
+ 2,
+ 19,
+ 12,
+ 52,
+ 0,
+ 40
+ ],
+ [
+ 49,
+ 15,
+ 24,
+ 34,
+ 60,
+ 42
+ ],
+ [
+ 12,
+ 46,
+ 17,
+ 29,
+ 41,
+ 3
+ ],
+ [
+ 39,
+ 60,
+ 44,
+ 41,
+ 33,
+ 36
+ ],
+ [
+ 21,
+ 60,
+ 16,
+ 44,
+ 51,
+ 57
+ ],
+ [
+ 24,
+ 41,
+ 12,
+ 33,
+ 13,
+ 21
+ ],
+ [
+ 43,
+ 62,
+ 3,
+ 12,
+ 28,
+ 45
+ ],
+ [
+ 58,
+ 19,
+ 39,
+ 17,
+ 49,
+ 42
+ ],
+ [
+ 25,
+ 54,
+ 4,
+ 11,
+ 7,
+ 39
+ ],
+ [
+ 34,
+ 35,
+ 4,
+ 42,
+ 62,
+ 19
+ ],
+ [
+ 43,
+ 41,
+ 42,
+ 35,
+ 40,
+ 32
+ ],
+ [
+ 21,
+ 63,
+ 3,
+ 17,
+ 20,
+ 50
+ ],
+ [
+ 58,
+ 46,
+ 44,
+ 1,
+ 25,
+ 20
+ ],
+ [
+ 26,
+ 32,
+ 16,
+ 25,
+ 46,
+ 41
+ ],
+ [
+ 37,
+ 63,
+ 61,
+ 28,
+ 24,
+ 56
+ ],
+ [
+ 33,
+ 42,
+ 40,
+ 37,
+ 48,
+ 50
+ ],
+ [
+ 11,
+ 53,
+ 25,
+ 39,
+ 4,
+ 61
+ ],
+ [
+ 12,
+ 54,
+ 27,
+ 4,
+ 50,
+ 14
+ ],
+ [
+ 47,
+ 19,
+ 42,
+ 17,
+ 35,
+ 40
+ ],
+ [
+ 54,
+ 40,
+ 60,
+ 63,
+ 45,
+ 57
+ ],
+ [
+ 44,
+ 56,
+ 40,
+ 62,
+ 37,
+ 3
+ ],
+ [
+ 59,
+ 41,
+ 57,
+ 34,
+ 48,
+ 22
+ ],
+ [
+ 56,
+ 13,
+ 59,
+ 51,
+ 26,
+ 23
+ ]
+ ],
+ [
+ [
+ 35,
+ 32,
+ 8,
+ 40,
+ 51,
+ 52
+ ],
+ [
+ 52,
+ 5,
+ 22,
+ 21,
+ 6,
+ 33
+ ],
+ [
+ 22,
+ 58,
+ 11,
+ 25,
+ 3,
+ 51
+ ],
+ [
+ 63,
+ 2,
+ 56,
+ 4,
+ 23,
+ 54
+ ],
+ [
+ 39,
+ 12,
+ 23,
+ 32,
+ 30,
+ 46
+ ],
+ [
+ 50,
+ 34,
+ 36,
+ 58,
+ 26,
+ 28
+ ],
+ [
+ 46,
+ 41,
+ 3,
+ 2,
+ 22,
+ 16
+ ],
+ [
+ 60,
+ 36,
+ 53,
+ 30,
+ 54,
+ 39
+ ],
+ [
+ 16,
+ 51,
+ 3,
+ 39,
+ 2,
+ 26
+ ],
+ [
+ 45,
+ 26,
+ 18,
+ 41,
+ 32,
+ 46
+ ],
+ [
+ 45,
+ 3,
+ 10,
+ 56,
+ 36,
+ 35
+ ],
+ [
+ 21,
+ 36,
+ 35,
+ 50,
+ 11,
+ 19
+ ],
+ [
+ 28,
+ 11,
+ 46,
+ 59,
+ 41,
+ 13
+ ],
+ [
+ 23,
+ 16,
+ 38,
+ 19,
+ 15,
+ 22
+ ],
+ [
+ 27,
+ 7,
+ 34,
+ 58,
+ 3,
+ 42
+ ],
+ [
+ 9,
+ 22,
+ 46,
+ 36,
+ 26,
+ 41
+ ],
+ [
+ 3,
+ 51,
+ 56,
+ 40,
+ 46,
+ 8
+ ],
+ [
+ 12,
+ 21,
+ 25,
+ 50,
+ 17,
+ 62
+ ],
+ [
+ 27,
+ 34,
+ 61,
+ 13,
+ 60,
+ 11
+ ],
+ [
+ 53,
+ 12,
+ 56,
+ 0,
+ 42,
+ 33
+ ],
+ [
+ 53,
+ 37,
+ 12,
+ 24,
+ 63,
+ 25
+ ],
+ [
+ 45,
+ 55,
+ 18,
+ 26,
+ 17,
+ 43
+ ],
+ [
+ 4,
+ 25,
+ 32,
+ 1,
+ 48,
+ 53
+ ],
+ [
+ 17,
+ 27,
+ 63,
+ 4,
+ 62,
+ 34
+ ],
+ [
+ 6,
+ 52,
+ 62,
+ 40,
+ 46,
+ 54
+ ],
+ [
+ 10,
+ 42,
+ 28,
+ 49,
+ 3,
+ 1
+ ],
+ [
+ 45,
+ 27,
+ 41,
+ 21,
+ 16,
+ 47
+ ]
+ ],
+ [
+ [
+ 44,
+ 24,
+ 56,
+ 33,
+ 15,
+ 41
+ ],
+ [
+ 38,
+ 26,
+ 24,
+ 29,
+ 19,
+ 53
+ ],
+ [
+ 12,
+ 15,
+ 29,
+ 9,
+ 1,
+ 63
+ ],
+ [
+ 58,
+ 38,
+ 50,
+ 0,
+ 43,
+ 61
+ ],
+ [
+ 24,
+ 51,
+ 31,
+ 34,
+ 60,
+ 7
+ ],
+ [
+ 0,
+ 7,
+ 22,
+ 43,
+ 35,
+ 1
+ ],
+ [
+ 63,
+ 36,
+ 11,
+ 1,
+ 16,
+ 4
+ ],
+ [
+ 8,
+ 50,
+ 56,
+ 4,
+ 30,
+ 55
+ ],
+ [
+ 43,
+ 16,
+ 42,
+ 29,
+ 60,
+ 35
+ ],
+ [
+ 34,
+ 0,
+ 9,
+ 22,
+ 18,
+ 26
+ ],
+ [
+ 54,
+ 51,
+ 45,
+ 35,
+ 2,
+ 36
+ ],
+ [
+ 37,
+ 36,
+ 43,
+ 60,
+ 11,
+ 59
+ ],
+ [
+ 56,
+ 38,
+ 10,
+ 28,
+ 14,
+ 43
+ ],
+ [
+ 30,
+ 0,
+ 58,
+ 62,
+ 22,
+ 19
+ ],
+ [
+ 7,
+ 55,
+ 42,
+ 58,
+ 30,
+ 38
+ ],
+ [
+ 11,
+ 33,
+ 1,
+ 39,
+ 19,
+ 16
+ ],
+ [
+ 55,
+ 20,
+ 40,
+ 9,
+ 18,
+ 30
+ ],
+ [
+ 18,
+ 20,
+ 57,
+ 32,
+ 45,
+ 1
+ ],
+ [
+ 43,
+ 61,
+ 12,
+ 32,
+ 31,
+ 30
+ ],
+ [
+ 23,
+ 25,
+ 7,
+ 28,
+ 40,
+ 19
+ ],
+ [
+ 14,
+ 51,
+ 48,
+ 58,
+ 53,
+ 25
+ ],
+ [
+ 18,
+ 30,
+ 1,
+ 49,
+ 41,
+ 9
+ ],
+ [
+ 2,
+ 51,
+ 22,
+ 0,
+ 52,
+ 5
+ ],
+ [
+ 53,
+ 4,
+ 47,
+ 52,
+ 51,
+ 40
+ ],
+ [
+ 40,
+ 16,
+ 9,
+ 47,
+ 23,
+ 11
+ ],
+ [
+ 47,
+ 3,
+ 43,
+ 46,
+ 26,
+ 53
+ ],
+ [
+ 8,
+ 40,
+ 18,
+ 46,
+ 33,
+ 63
+ ]
+ ],
+ [
+ [
+ 48,
+ 38,
+ 50,
+ 42,
+ 63,
+ 36
+ ],
+ [
+ 3,
+ 26,
+ 10,
+ 2,
+ 6,
+ 61
+ ],
+ [
+ 39,
+ 44,
+ 45,
+ 40,
+ 6,
+ 7
+ ],
+ [
+ 41,
+ 5,
+ 20,
+ 49,
+ 56,
+ 13
+ ],
+ [
+ 6,
+ 1,
+ 30,
+ 37,
+ 28,
+ 38
+ ],
+ [
+ 59,
+ 46,
+ 22,
+ 35,
+ 61,
+ 0
+ ],
+ [
+ 1,
+ 63,
+ 35,
+ 3,
+ 60,
+ 49
+ ],
+ [
+ 8,
+ 12,
+ 2,
+ 50,
+ 5,
+ 55
+ ],
+ [
+ 42,
+ 33,
+ 43,
+ 16,
+ 32,
+ 29
+ ],
+ [
+ 9,
+ 34,
+ 0,
+ 20,
+ 41,
+ 31
+ ],
+ [
+ 51,
+ 54,
+ 8,
+ 19,
+ 63,
+ 9
+ ],
+ [
+ 37,
+ 56,
+ 36,
+ 11,
+ 59,
+ 43
+ ],
+ [
+ 38,
+ 10,
+ 28,
+ 17,
+ 56,
+ 63
+ ],
+ [
+ 27,
+ 30,
+ 42,
+ 19,
+ 0,
+ 22
+ ],
+ [
+ 7,
+ 55,
+ 49,
+ 42,
+ 58,
+ 38
+ ],
+ [
+ 29,
+ 34,
+ 39,
+ 33,
+ 47,
+ 45
+ ],
+ [
+ 55,
+ 40,
+ 20,
+ 18,
+ 7,
+ 5
+ ],
+ [
+ 43,
+ 57,
+ 39,
+ 54,
+ 48,
+ 28
+ ],
+ [
+ 12,
+ 43,
+ 61,
+ 42,
+ 32,
+ 49
+ ],
+ [
+ 23,
+ 36,
+ 1,
+ 7,
+ 59,
+ 28
+ ],
+ [
+ 14,
+ 53,
+ 21,
+ 7,
+ 57,
+ 37
+ ],
+ [
+ 18,
+ 1,
+ 24,
+ 60,
+ 30,
+ 9
+ ],
+ [
+ 51,
+ 0,
+ 33,
+ 2,
+ 44,
+ 5
+ ],
+ [
+ 52,
+ 29,
+ 4,
+ 41,
+ 54,
+ 58
+ ],
+ [
+ 40,
+ 19,
+ 16,
+ 46,
+ 9,
+ 47
+ ],
+ [
+ 61,
+ 3,
+ 47,
+ 22,
+ 21,
+ 53
+ ],
+ [
+ 35,
+ 60,
+ 54,
+ 5,
+ 1,
+ 40
+ ]
+ ],
+ [
+ [
+ 17,
+ 18,
+ 8,
+ 53,
+ 25,
+ 43
+ ],
+ [
+ 9,
+ 38,
+ 24,
+ 47,
+ 25,
+ 63
+ ],
+ [
+ 20,
+ 24,
+ 5,
+ 12,
+ 54,
+ 28
+ ],
+ [
+ 43,
+ 10,
+ 20,
+ 42,
+ 11,
+ 8
+ ],
+ [
+ 53,
+ 61,
+ 30,
+ 39,
+ 29,
+ 18
+ ],
+ [
+ 61,
+ 56,
+ 25,
+ 40,
+ 5,
+ 22
+ ],
+ [
+ 62,
+ 17,
+ 24,
+ 1,
+ 47,
+ 33
+ ],
+ [
+ 41,
+ 16,
+ 34,
+ 39,
+ 29,
+ 8
+ ],
+ [
+ 39,
+ 16,
+ 36,
+ 42,
+ 29,
+ 23
+ ],
+ [
+ 9,
+ 11,
+ 41,
+ 63,
+ 56,
+ 31
+ ],
+ [
+ 48,
+ 51,
+ 10,
+ 62,
+ 63,
+ 45
+ ],
+ [
+ 36,
+ 11,
+ 37,
+ 42,
+ 58,
+ 46
+ ],
+ [
+ 51,
+ 38,
+ 25,
+ 63,
+ 29,
+ 44
+ ],
+ [
+ 4,
+ 56,
+ 44,
+ 62,
+ 58,
+ 30
+ ],
+ [
+ 3,
+ 7,
+ 42,
+ 46,
+ 33,
+ 35
+ ],
+ [
+ 39,
+ 9,
+ 33,
+ 58,
+ 60,
+ 29
+ ],
+ [
+ 40,
+ 37,
+ 20,
+ 16,
+ 55,
+ 25
+ ],
+ [
+ 54,
+ 19,
+ 11,
+ 57,
+ 0,
+ 39
+ ],
+ [
+ 12,
+ 43,
+ 61,
+ 25,
+ 49,
+ 32
+ ],
+ [
+ 4,
+ 23,
+ 54,
+ 36,
+ 7,
+ 28
+ ],
+ [
+ 40,
+ 25,
+ 26,
+ 14,
+ 2,
+ 58
+ ],
+ [
+ 18,
+ 24,
+ 58,
+ 1,
+ 22,
+ 46
+ ],
+ [
+ 2,
+ 63,
+ 22,
+ 6,
+ 44,
+ 56
+ ],
+ [
+ 52,
+ 29,
+ 51,
+ 4,
+ 40,
+ 32
+ ],
+ [
+ 40,
+ 17,
+ 15,
+ 16,
+ 46,
+ 57
+ ],
+ [
+ 9,
+ 61,
+ 3,
+ 47,
+ 24,
+ 11
+ ],
+ [
+ 2,
+ 39,
+ 24,
+ 42,
+ 0,
+ 44
+ ]
+ ],
+ [
+ [
+ 0,
+ 10,
+ 49,
+ 23,
+ 62,
+ 44
+ ],
+ [
+ 28,
+ 0,
+ 36,
+ 26,
+ 47,
+ 52
+ ],
+ [
+ 30,
+ 4,
+ 16,
+ 48,
+ 40,
+ 10
+ ],
+ [
+ 61,
+ 32,
+ 26,
+ 16,
+ 33,
+ 62
+ ],
+ [
+ 30,
+ 39,
+ 53,
+ 5,
+ 57,
+ 20
+ ],
+ [
+ 5,
+ 37,
+ 61,
+ 15,
+ 25,
+ 6
+ ],
+ [
+ 15,
+ 17,
+ 24,
+ 60,
+ 49,
+ 62
+ ],
+ [
+ 34,
+ 39,
+ 61,
+ 0,
+ 58,
+ 40
+ ],
+ [
+ 16,
+ 39,
+ 36,
+ 51,
+ 2,
+ 29
+ ],
+ [
+ 9,
+ 11,
+ 41,
+ 31,
+ 56,
+ 52
+ ],
+ [
+ 10,
+ 48,
+ 24,
+ 45,
+ 62,
+ 51
+ ],
+ [
+ 11,
+ 38,
+ 36,
+ 37,
+ 6,
+ 42
+ ],
+ [
+ 51,
+ 50,
+ 15,
+ 30,
+ 25,
+ 38
+ ],
+ [
+ 4,
+ 19,
+ 24,
+ 35,
+ 31,
+ 48
+ ],
+ [
+ 7,
+ 46,
+ 3,
+ 58,
+ 30,
+ 41
+ ],
+ [
+ 58,
+ 9,
+ 39,
+ 32,
+ 29,
+ 40
+ ],
+ [
+ 40,
+ 37,
+ 20,
+ 8,
+ 25,
+ 55
+ ],
+ [
+ 19,
+ 0,
+ 54,
+ 52,
+ 39,
+ 17
+ ],
+ [
+ 25,
+ 43,
+ 12,
+ 61,
+ 11,
+ 14
+ ],
+ [
+ 23,
+ 4,
+ 54,
+ 36,
+ 28,
+ 33
+ ],
+ [
+ 40,
+ 2,
+ 25,
+ 58,
+ 36,
+ 53
+ ],
+ [
+ 18,
+ 46,
+ 35,
+ 22,
+ 53,
+ 16
+ ],
+ [
+ 2,
+ 6,
+ 63,
+ 14,
+ 42,
+ 11
+ ],
+ [
+ 35,
+ 7,
+ 52,
+ 40,
+ 29,
+ 57
+ ],
+ [
+ 40,
+ 15,
+ 19,
+ 57,
+ 17,
+ 23
+ ],
+ [
+ 9,
+ 11,
+ 47,
+ 22,
+ 49,
+ 1
+ ],
+ [
+ 24,
+ 39,
+ 42,
+ 2,
+ 16,
+ 22
+ ]
+ ],
+ [
+ [
+ 55,
+ 39,
+ 9,
+ 43,
+ 21,
+ 46
+ ],
+ [
+ 56,
+ 0,
+ 63,
+ 39,
+ 30,
+ 41
+ ],
+ [
+ 20,
+ 1,
+ 26,
+ 58,
+ 34,
+ 19
+ ],
+ [
+ 54,
+ 24,
+ 32,
+ 51,
+ 26,
+ 44
+ ],
+ [
+ 30,
+ 53,
+ 56,
+ 39,
+ 34,
+ 40
+ ],
+ [
+ 5,
+ 37,
+ 25,
+ 50,
+ 6,
+ 61
+ ],
+ [
+ 24,
+ 49,
+ 37,
+ 15,
+ 6,
+ 29
+ ],
+ [
+ 34,
+ 16,
+ 30,
+ 61,
+ 10,
+ 36
+ ],
+ [
+ 16,
+ 29,
+ 2,
+ 5,
+ 51,
+ 26
+ ],
+ [
+ 9,
+ 56,
+ 11,
+ 31,
+ 46,
+ 45
+ ],
+ [
+ 10,
+ 45,
+ 56,
+ 62,
+ 25,
+ 36
+ ],
+ [
+ 11,
+ 6,
+ 35,
+ 36,
+ 1,
+ 52
+ ],
+ [
+ 51,
+ 50,
+ 41,
+ 46,
+ 38,
+ 4
+ ],
+ [
+ 19,
+ 33,
+ 41,
+ 16,
+ 31,
+ 52
+ ],
+ [
+ 34,
+ 7,
+ 17,
+ 47,
+ 63,
+ 3
+ ],
+ [
+ 58,
+ 9,
+ 22,
+ 61,
+ 59,
+ 8
+ ],
+ [
+ 40,
+ 37,
+ 3,
+ 51,
+ 22,
+ 25
+ ],
+ [
+ 12,
+ 52,
+ 21,
+ 54,
+ 25,
+ 19
+ ],
+ [
+ 34,
+ 53,
+ 27,
+ 43,
+ 14,
+ 13
+ ],
+ [
+ 56,
+ 53,
+ 44,
+ 24,
+ 60,
+ 43
+ ],
+ [
+ 12,
+ 53,
+ 40,
+ 49,
+ 2,
+ 62
+ ],
+ [
+ 18,
+ 39,
+ 44,
+ 61,
+ 26,
+ 23
+ ],
+ [
+ 0,
+ 4,
+ 53,
+ 41,
+ 25,
+ 21
+ ],
+ [
+ 1,
+ 7,
+ 25,
+ 10,
+ 40,
+ 56
+ ],
+ [
+ 40,
+ 22,
+ 6,
+ 29,
+ 19,
+ 48
+ ],
+ [
+ 28,
+ 10,
+ 47,
+ 55,
+ 42,
+ 44
+ ],
+ [
+ 30,
+ 27,
+ 57,
+ 16,
+ 50,
+ 59
+ ]
+ ],
+ [
+ [
+ 45,
+ 37,
+ 48,
+ 29,
+ 30,
+ 3
+ ],
+ [
+ 8,
+ 60,
+ 10,
+ 59,
+ 43,
+ 6
+ ],
+ [
+ 51,
+ 45,
+ 28,
+ 59,
+ 63,
+ 34
+ ],
+ [
+ 4,
+ 16,
+ 20,
+ 58,
+ 44,
+ 28
+ ],
+ [
+ 50,
+ 31,
+ 57,
+ 24,
+ 51,
+ 53
+ ],
+ [
+ 58,
+ 9,
+ 0,
+ 61,
+ 35,
+ 41
+ ],
+ [
+ 16,
+ 63,
+ 11,
+ 61,
+ 23,
+ 36
+ ],
+ [
+ 4,
+ 47,
+ 42,
+ 53,
+ 8,
+ 30
+ ],
+ [
+ 44,
+ 14,
+ 16,
+ 33,
+ 3,
+ 20
+ ],
+ [
+ 34,
+ 28,
+ 26,
+ 57,
+ 22,
+ 18
+ ],
+ [
+ 20,
+ 35,
+ 19,
+ 59,
+ 2,
+ 38
+ ],
+ [
+ 12,
+ 60,
+ 43,
+ 63,
+ 32,
+ 62
+ ],
+ [
+ 28,
+ 12,
+ 29,
+ 11,
+ 14,
+ 50
+ ],
+ [
+ 23,
+ 29,
+ 33,
+ 22,
+ 11,
+ 19
+ ],
+ [
+ 23,
+ 60,
+ 51,
+ 50,
+ 7,
+ 22
+ ],
+ [
+ 44,
+ 46,
+ 49,
+ 7,
+ 1,
+ 12
+ ],
+ [
+ 2,
+ 54,
+ 27,
+ 61,
+ 18,
+ 5
+ ],
+ [
+ 17,
+ 50,
+ 51,
+ 32,
+ 33,
+ 34
+ ],
+ [
+ 5,
+ 19,
+ 61,
+ 27,
+ 32,
+ 11
+ ],
+ [
+ 6,
+ 0,
+ 5,
+ 13,
+ 41,
+ 57
+ ],
+ [
+ 27,
+ 33,
+ 53,
+ 45,
+ 38,
+ 32
+ ],
+ [
+ 26,
+ 36,
+ 55,
+ 59,
+ 61,
+ 18
+ ],
+ [
+ 47,
+ 46,
+ 3,
+ 37,
+ 57,
+ 49
+ ],
+ [
+ 20,
+ 22,
+ 4,
+ 16,
+ 51,
+ 11
+ ],
+ [
+ 62,
+ 11,
+ 21,
+ 34,
+ 4,
+ 1
+ ],
+ [
+ 34,
+ 18,
+ 7,
+ 60,
+ 33,
+ 3
+ ],
+ [
+ 45,
+ 52,
+ 36,
+ 4,
+ 21,
+ 9
+ ]
+ ],
+ [
+ [
+ 18,
+ 8,
+ 20,
+ 49,
+ 30,
+ 23
+ ],
+ [
+ 1,
+ 27,
+ 26,
+ 22,
+ 59,
+ 36
+ ],
+ [
+ 43,
+ 26,
+ 15,
+ 58,
+ 0,
+ 46
+ ],
+ [
+ 55,
+ 1,
+ 35,
+ 28,
+ 16,
+ 32
+ ],
+ [
+ 59,
+ 9,
+ 10,
+ 53,
+ 12,
+ 21
+ ],
+ [
+ 9,
+ 2,
+ 27,
+ 11,
+ 61,
+ 43
+ ],
+ [
+ 16,
+ 57,
+ 63,
+ 23,
+ 19,
+ 12
+ ],
+ [
+ 46,
+ 45,
+ 26,
+ 4,
+ 30,
+ 37
+ ],
+ [
+ 43,
+ 44,
+ 20,
+ 16,
+ 14,
+ 9
+ ],
+ [
+ 34,
+ 47,
+ 42,
+ 43,
+ 26,
+ 51
+ ],
+ [
+ 2,
+ 42,
+ 38,
+ 45,
+ 20,
+ 36
+ ],
+ [
+ 18,
+ 7,
+ 12,
+ 2,
+ 43,
+ 60
+ ],
+ [
+ 1,
+ 28,
+ 12,
+ 3,
+ 29,
+ 33
+ ],
+ [
+ 25,
+ 13,
+ 0,
+ 63,
+ 2,
+ 62
+ ],
+ [
+ 18,
+ 36,
+ 6,
+ 29,
+ 19,
+ 15
+ ],
+ [
+ 1,
+ 42,
+ 63,
+ 41,
+ 57,
+ 19
+ ],
+ [
+ 57,
+ 54,
+ 5,
+ 27,
+ 31,
+ 18
+ ],
+ [
+ 50,
+ 6,
+ 13,
+ 32,
+ 17,
+ 20
+ ],
+ [
+ 17,
+ 5,
+ 27,
+ 32,
+ 1,
+ 55
+ ],
+ [
+ 49,
+ 0,
+ 61,
+ 10,
+ 5,
+ 30
+ ],
+ [
+ 29,
+ 53,
+ 51,
+ 13,
+ 33,
+ 46
+ ],
+ [
+ 29,
+ 17,
+ 30,
+ 21,
+ 14,
+ 40
+ ],
+ [
+ 5,
+ 33,
+ 17,
+ 32,
+ 18,
+ 28
+ ],
+ [
+ 4,
+ 51,
+ 20,
+ 54,
+ 58,
+ 41
+ ],
+ [
+ 47,
+ 4,
+ 27,
+ 48,
+ 37,
+ 60
+ ],
+ [
+ 3,
+ 26,
+ 12,
+ 2,
+ 59,
+ 48
+ ],
+ [
+ 46,
+ 43,
+ 18,
+ 20,
+ 9,
+ 53
+ ]
+ ],
+ [
+ [
+ 45,
+ 6,
+ 57,
+ 43,
+ 40,
+ 55
+ ],
+ [
+ 38,
+ 63,
+ 36,
+ 27,
+ 54,
+ 33
+ ],
+ [
+ 37,
+ 14,
+ 19,
+ 41,
+ 58,
+ 63
+ ],
+ [
+ 9,
+ 12,
+ 2,
+ 55,
+ 28,
+ 23
+ ],
+ [
+ 39,
+ 59,
+ 7,
+ 13,
+ 33,
+ 43
+ ],
+ [
+ 45,
+ 9,
+ 63,
+ 27,
+ 32,
+ 58
+ ],
+ [
+ 16,
+ 57,
+ 10,
+ 63,
+ 11,
+ 23
+ ],
+ [
+ 51,
+ 45,
+ 25,
+ 4,
+ 21,
+ 30
+ ],
+ [
+ 21,
+ 44,
+ 14,
+ 16,
+ 39,
+ 33
+ ],
+ [
+ 42,
+ 44,
+ 43,
+ 5,
+ 37,
+ 34
+ ],
+ [
+ 42,
+ 19,
+ 20,
+ 2,
+ 38,
+ 61
+ ],
+ [
+ 4,
+ 12,
+ 2,
+ 62,
+ 63,
+ 36
+ ],
+ [
+ 32,
+ 55,
+ 0,
+ 11,
+ 47,
+ 28
+ ],
+ [
+ 43,
+ 13,
+ 2,
+ 44,
+ 26,
+ 50
+ ],
+ [
+ 49,
+ 33,
+ 15,
+ 28,
+ 29,
+ 35
+ ],
+ [
+ 44,
+ 41,
+ 7,
+ 2,
+ 22,
+ 63
+ ],
+ [
+ 48,
+ 6,
+ 54,
+ 20,
+ 2,
+ 27
+ ],
+ [
+ 50,
+ 51,
+ 32,
+ 3,
+ 17,
+ 36
+ ],
+ [
+ 5,
+ 61,
+ 57,
+ 48,
+ 19,
+ 32
+ ],
+ [
+ 21,
+ 0,
+ 6,
+ 31,
+ 29,
+ 47
+ ],
+ [
+ 33,
+ 9,
+ 53,
+ 27,
+ 17,
+ 36
+ ],
+ [
+ 29,
+ 26,
+ 55,
+ 19,
+ 17,
+ 62
+ ],
+ [
+ 12,
+ 46,
+ 5,
+ 37,
+ 57,
+ 3
+ ],
+ [
+ 20,
+ 51,
+ 4,
+ 22,
+ 16,
+ 41
+ ],
+ [
+ 21,
+ 11,
+ 62,
+ 46,
+ 23,
+ 48
+ ],
+ [
+ 32,
+ 60,
+ 37,
+ 18,
+ 3,
+ 7
+ ],
+ [
+ 9,
+ 11,
+ 36,
+ 48,
+ 0,
+ 45
+ ]
+ ],
+ [
+ [
+ 49,
+ 42,
+ 28,
+ 23,
+ 33,
+ 61
+ ],
+ [
+ 4,
+ 2,
+ 12,
+ 6,
+ 8,
+ 55
+ ],
+ [
+ 12,
+ 0,
+ 26,
+ 41,
+ 6,
+ 27
+ ],
+ [
+ 9,
+ 57,
+ 6,
+ 23,
+ 51,
+ 28
+ ],
+ [
+ 40,
+ 7,
+ 20,
+ 15,
+ 33,
+ 16
+ ],
+ [
+ 45,
+ 59,
+ 63,
+ 62,
+ 32,
+ 3
+ ],
+ [
+ 10,
+ 39,
+ 57,
+ 13,
+ 16,
+ 19
+ ],
+ [
+ 45,
+ 23,
+ 51,
+ 33,
+ 25,
+ 46
+ ],
+ [
+ 28,
+ 21,
+ 44,
+ 11,
+ 16,
+ 59
+ ],
+ [
+ 5,
+ 42,
+ 44,
+ 24,
+ 43,
+ 47
+ ],
+ [
+ 42,
+ 53,
+ 30,
+ 18,
+ 2,
+ 27
+ ],
+ [
+ 2,
+ 62,
+ 4,
+ 43,
+ 10,
+ 36
+ ],
+ [
+ 0,
+ 56,
+ 55,
+ 47,
+ 32,
+ 49
+ ],
+ [
+ 43,
+ 25,
+ 2,
+ 5,
+ 3,
+ 49
+ ],
+ [
+ 4,
+ 28,
+ 15,
+ 8,
+ 49,
+ 58
+ ],
+ [
+ 2,
+ 42,
+ 44,
+ 41,
+ 7,
+ 63
+ ],
+ [
+ 48,
+ 27,
+ 54,
+ 20,
+ 2,
+ 18
+ ],
+ [
+ 50,
+ 51,
+ 0,
+ 36,
+ 3,
+ 32
+ ],
+ [
+ 57,
+ 5,
+ 61,
+ 19,
+ 32,
+ 38
+ ],
+ [
+ 21,
+ 0,
+ 6,
+ 63,
+ 23,
+ 51
+ ],
+ [
+ 33,
+ 53,
+ 27,
+ 36,
+ 9,
+ 38
+ ],
+ [
+ 29,
+ 26,
+ 55,
+ 62,
+ 18,
+ 19
+ ],
+ [
+ 46,
+ 56,
+ 12,
+ 53,
+ 29,
+ 0
+ ],
+ [
+ 20,
+ 16,
+ 22,
+ 4,
+ 51,
+ 61
+ ],
+ [
+ 21,
+ 62,
+ 11,
+ 46,
+ 31,
+ 33
+ ],
+ [
+ 37,
+ 60,
+ 18,
+ 7,
+ 32,
+ 44
+ ],
+ [
+ 11,
+ 9,
+ 36,
+ 0,
+ 48,
+ 63
+ ]
+ ],
+ [
+ [
+ 41,
+ 32,
+ 49,
+ 39,
+ 44,
+ 61
+ ],
+ [
+ 47,
+ 26,
+ 16,
+ 21,
+ 36,
+ 22
+ ],
+ [
+ 4,
+ 30,
+ 37,
+ 42,
+ 60,
+ 54
+ ],
+ [
+ 9,
+ 57,
+ 26,
+ 32,
+ 50,
+ 20
+ ],
+ [
+ 56,
+ 3,
+ 40,
+ 33,
+ 36,
+ 54
+ ],
+ [
+ 11,
+ 38,
+ 2,
+ 32,
+ 61,
+ 30
+ ],
+ [
+ 39,
+ 57,
+ 19,
+ 10,
+ 16,
+ 42
+ ],
+ [
+ 46,
+ 21,
+ 35,
+ 39,
+ 45,
+ 25
+ ],
+ [
+ 21,
+ 37,
+ 12,
+ 20,
+ 11,
+ 28
+ ],
+ [
+ 5,
+ 47,
+ 44,
+ 10,
+ 42,
+ 23
+ ],
+ [
+ 18,
+ 42,
+ 61,
+ 2,
+ 38,
+ 31
+ ],
+ [
+ 54,
+ 4,
+ 2,
+ 7,
+ 22,
+ 16
+ ],
+ [
+ 5,
+ 3,
+ 17,
+ 56,
+ 32,
+ 55
+ ],
+ [
+ 55,
+ 0,
+ 2,
+ 25,
+ 43,
+ 5
+ ],
+ [
+ 22,
+ 28,
+ 15,
+ 6,
+ 5,
+ 49
+ ],
+ [
+ 2,
+ 57,
+ 19,
+ 54,
+ 41,
+ 30
+ ],
+ [
+ 7,
+ 48,
+ 20,
+ 54,
+ 27,
+ 0
+ ],
+ [
+ 3,
+ 56,
+ 13,
+ 37,
+ 59,
+ 43
+ ],
+ [
+ 45,
+ 55,
+ 57,
+ 61,
+ 48,
+ 52
+ ],
+ [
+ 21,
+ 5,
+ 0,
+ 16,
+ 27,
+ 23
+ ],
+ [
+ 25,
+ 42,
+ 17,
+ 54,
+ 23,
+ 14
+ ],
+ [
+ 21,
+ 44,
+ 15,
+ 20,
+ 42,
+ 18
+ ],
+ [
+ 35,
+ 12,
+ 25,
+ 53,
+ 61,
+ 2
+ ],
+ [
+ 38,
+ 54,
+ 48,
+ 53,
+ 21,
+ 36
+ ],
+ [
+ 13,
+ 31,
+ 48,
+ 33,
+ 18,
+ 55
+ ],
+ [
+ 38,
+ 27,
+ 19,
+ 6,
+ 44,
+ 3
+ ],
+ [
+ 29,
+ 62,
+ 43,
+ 59,
+ 46,
+ 5
+ ]
+ ],
+ [
+ [
+ 57,
+ 9,
+ 19,
+ 51,
+ 18,
+ 41
+ ],
+ [
+ 28,
+ 57,
+ 36,
+ 8,
+ 48,
+ 60
+ ],
+ [
+ 2,
+ 51,
+ 59,
+ 5,
+ 34,
+ 9
+ ],
+ [
+ 9,
+ 55,
+ 59,
+ 26,
+ 4,
+ 2
+ ],
+ [
+ 49,
+ 56,
+ 35,
+ 42,
+ 30,
+ 23
+ ],
+ [
+ 18,
+ 30,
+ 22,
+ 29,
+ 19,
+ 52
+ ],
+ [
+ 39,
+ 34,
+ 33,
+ 51,
+ 56,
+ 3
+ ],
+ [
+ 32,
+ 21,
+ 7,
+ 1,
+ 46,
+ 49
+ ],
+ [
+ 33,
+ 54,
+ 23,
+ 21,
+ 12,
+ 17
+ ],
+ [
+ 5,
+ 30,
+ 60,
+ 47,
+ 15,
+ 18
+ ],
+ [
+ 4,
+ 18,
+ 46,
+ 27,
+ 20,
+ 22
+ ],
+ [
+ 22,
+ 59,
+ 54,
+ 48,
+ 19,
+ 4
+ ],
+ [
+ 17,
+ 5,
+ 56,
+ 31,
+ 49,
+ 4
+ ],
+ [
+ 29,
+ 47,
+ 55,
+ 2,
+ 53,
+ 60
+ ],
+ [
+ 8,
+ 22,
+ 11,
+ 44,
+ 36,
+ 15
+ ],
+ [
+ 60,
+ 44,
+ 30,
+ 57,
+ 54,
+ 39
+ ],
+ [
+ 7,
+ 44,
+ 27,
+ 20,
+ 2,
+ 61
+ ],
+ [
+ 48,
+ 17,
+ 37,
+ 32,
+ 21,
+ 57
+ ],
+ [
+ 48,
+ 32,
+ 46,
+ 6,
+ 61,
+ 42
+ ],
+ [
+ 4,
+ 57,
+ 1,
+ 36,
+ 0,
+ 30
+ ],
+ [
+ 7,
+ 17,
+ 61,
+ 53,
+ 21,
+ 63
+ ],
+ [
+ 60,
+ 14,
+ 53,
+ 35,
+ 18,
+ 42
+ ],
+ [
+ 10,
+ 15,
+ 33,
+ 51,
+ 36,
+ 5
+ ],
+ [
+ 11,
+ 4,
+ 19,
+ 51,
+ 21,
+ 52
+ ],
+ [
+ 47,
+ 19,
+ 48,
+ 43,
+ 4,
+ 58
+ ],
+ [
+ 3,
+ 33,
+ 26,
+ 21,
+ 52,
+ 19
+ ],
+ [
+ 24,
+ 45,
+ 60,
+ 35,
+ 49,
+ 1
+ ]
+ ],
+ [
+ [
+ 16,
+ 4,
+ 44,
+ 23,
+ 22,
+ 35
+ ],
+ [
+ 16,
+ 50,
+ 23,
+ 9,
+ 32,
+ 13
+ ],
+ [
+ 11,
+ 35,
+ 21,
+ 7,
+ 48,
+ 59
+ ],
+ [
+ 55,
+ 15,
+ 1,
+ 11,
+ 8,
+ 40
+ ],
+ [
+ 35,
+ 61,
+ 30,
+ 59,
+ 31,
+ 62
+ ],
+ [
+ 51,
+ 29,
+ 15,
+ 52,
+ 38,
+ 61
+ ],
+ [
+ 60,
+ 0,
+ 55,
+ 34,
+ 59,
+ 33
+ ],
+ [
+ 12,
+ 22,
+ 56,
+ 63,
+ 54,
+ 55
+ ],
+ [
+ 42,
+ 54,
+ 23,
+ 33,
+ 27,
+ 47
+ ],
+ [
+ 30,
+ 60,
+ 20,
+ 5,
+ 4,
+ 22
+ ],
+ [
+ 4,
+ 35,
+ 22,
+ 46,
+ 23,
+ 19
+ ],
+ [
+ 29,
+ 22,
+ 59,
+ 49,
+ 24,
+ 28
+ ],
+ [
+ 17,
+ 31,
+ 4,
+ 56,
+ 5,
+ 9
+ ],
+ [
+ 61,
+ 29,
+ 0,
+ 48,
+ 59,
+ 50
+ ],
+ [
+ 8,
+ 6,
+ 22,
+ 60,
+ 55,
+ 31
+ ],
+ [
+ 60,
+ 39,
+ 19,
+ 57,
+ 53,
+ 27
+ ],
+ [
+ 56,
+ 22,
+ 30,
+ 10,
+ 5,
+ 55
+ ],
+ [
+ 20,
+ 31,
+ 26,
+ 1,
+ 61,
+ 45
+ ],
+ [
+ 8,
+ 23,
+ 7,
+ 46,
+ 48,
+ 4
+ ],
+ [
+ 45,
+ 23,
+ 51,
+ 17,
+ 4,
+ 13
+ ],
+ [
+ 17,
+ 13,
+ 61,
+ 14,
+ 29,
+ 55
+ ],
+ [
+ 14,
+ 27,
+ 43,
+ 21,
+ 56,
+ 57
+ ],
+ [
+ 52,
+ 51,
+ 15,
+ 58,
+ 8,
+ 5
+ ],
+ [
+ 4,
+ 51,
+ 49,
+ 14,
+ 21,
+ 34
+ ],
+ [
+ 4,
+ 9,
+ 47,
+ 13,
+ 8,
+ 61
+ ],
+ [
+ 27,
+ 3,
+ 16,
+ 43,
+ 31,
+ 47
+ ],
+ [
+ 59,
+ 43,
+ 61,
+ 29,
+ 0,
+ 18
+ ]
+ ],
+ [
+ [
+ 48,
+ 21,
+ 18,
+ 49,
+ 41,
+ 23
+ ],
+ [
+ 36,
+ 4,
+ 60,
+ 8,
+ 49,
+ 44
+ ],
+ [
+ 20,
+ 39,
+ 30,
+ 59,
+ 45,
+ 55
+ ],
+ [
+ 35,
+ 46,
+ 15,
+ 48,
+ 33,
+ 2
+ ],
+ [
+ 61,
+ 37,
+ 8,
+ 15,
+ 54,
+ 10
+ ],
+ [
+ 46,
+ 51,
+ 6,
+ 29,
+ 58,
+ 4
+ ],
+ [
+ 28,
+ 11,
+ 44,
+ 60,
+ 0,
+ 1
+ ],
+ [
+ 63,
+ 12,
+ 13,
+ 27,
+ 10,
+ 0
+ ],
+ [
+ 33,
+ 42,
+ 54,
+ 44,
+ 23,
+ 14
+ ],
+ [
+ 18,
+ 60,
+ 30,
+ 22,
+ 40,
+ 14
+ ],
+ [
+ 35,
+ 4,
+ 61,
+ 9,
+ 18,
+ 33
+ ],
+ [
+ 59,
+ 45,
+ 48,
+ 28,
+ 62,
+ 22
+ ],
+ [
+ 17,
+ 56,
+ 7,
+ 5,
+ 53,
+ 36
+ ],
+ [
+ 8,
+ 47,
+ 29,
+ 59,
+ 1,
+ 6
+ ],
+ [
+ 9,
+ 8,
+ 18,
+ 22,
+ 60,
+ 15
+ ],
+ [
+ 46,
+ 60,
+ 22,
+ 44,
+ 30,
+ 57
+ ],
+ [
+ 61,
+ 2,
+ 27,
+ 34,
+ 7,
+ 60
+ ],
+ [
+ 48,
+ 21,
+ 37,
+ 17,
+ 50,
+ 57
+ ],
+ [
+ 44,
+ 42,
+ 5,
+ 2,
+ 61,
+ 48
+ ],
+ [
+ 37,
+ 57,
+ 4,
+ 36,
+ 17,
+ 59
+ ],
+ [
+ 7,
+ 17,
+ 44,
+ 61,
+ 53,
+ 33
+ ],
+ [
+ 60,
+ 45,
+ 14,
+ 42,
+ 18,
+ 9
+ ],
+ [
+ 10,
+ 8,
+ 36,
+ 33,
+ 15,
+ 58
+ ],
+ [
+ 19,
+ 20,
+ 11,
+ 4,
+ 49,
+ 51
+ ],
+ [
+ 61,
+ 37,
+ 47,
+ 23,
+ 12,
+ 3
+ ],
+ [
+ 17,
+ 33,
+ 3,
+ 40,
+ 19,
+ 16
+ ],
+ [
+ 61,
+ 45,
+ 49,
+ 1,
+ 14,
+ 63
+ ]
+ ],
+ [
+ [
+ 13,
+ 40,
+ 55,
+ 63,
+ 26,
+ 4
+ ],
+ [
+ 5,
+ 35,
+ 49,
+ 40,
+ 17,
+ 46
+ ],
+ [
+ 38,
+ 17,
+ 59,
+ 49,
+ 2,
+ 58
+ ],
+ [
+ 40,
+ 8,
+ 1,
+ 16,
+ 0,
+ 11
+ ],
+ [
+ 37,
+ 62,
+ 51,
+ 10,
+ 8,
+ 38
+ ],
+ [
+ 9,
+ 42,
+ 61,
+ 29,
+ 35,
+ 33
+ ],
+ [
+ 63,
+ 53,
+ 11,
+ 16,
+ 33,
+ 60
+ ],
+ [
+ 63,
+ 37,
+ 5,
+ 13,
+ 17,
+ 39
+ ],
+ [
+ 44,
+ 20,
+ 31,
+ 54,
+ 21,
+ 38
+ ],
+ [
+ 43,
+ 21,
+ 30,
+ 34,
+ 18,
+ 49
+ ],
+ [
+ 2,
+ 11,
+ 19,
+ 35,
+ 4,
+ 9
+ ],
+ [
+ 22,
+ 60,
+ 43,
+ 2,
+ 4,
+ 49
+ ],
+ [
+ 29,
+ 5,
+ 17,
+ 22,
+ 24,
+ 55
+ ],
+ [
+ 2,
+ 59,
+ 29,
+ 5,
+ 55,
+ 6
+ ],
+ [
+ 23,
+ 8,
+ 36,
+ 22,
+ 15,
+ 30
+ ],
+ [
+ 12,
+ 44,
+ 41,
+ 45,
+ 5,
+ 22
+ ],
+ [
+ 54,
+ 7,
+ 41,
+ 11,
+ 1,
+ 53
+ ],
+ [
+ 6,
+ 50,
+ 2,
+ 9,
+ 21,
+ 37
+ ],
+ [
+ 13,
+ 19,
+ 5,
+ 10,
+ 48,
+ 8
+ ],
+ [
+ 41,
+ 6,
+ 32,
+ 21,
+ 0,
+ 47
+ ],
+ [
+ 38,
+ 33,
+ 36,
+ 53,
+ 31,
+ 61
+ ],
+ [
+ 3,
+ 26,
+ 7,
+ 62,
+ 18,
+ 59
+ ],
+ [
+ 56,
+ 57,
+ 46,
+ 12,
+ 35,
+ 3
+ ],
+ [
+ 20,
+ 16,
+ 22,
+ 24,
+ 27,
+ 42
+ ],
+ [
+ 36,
+ 21,
+ 46,
+ 34,
+ 3,
+ 11
+ ],
+ [
+ 33,
+ 34,
+ 60,
+ 45,
+ 7,
+ 32
+ ],
+ [
+ 56,
+ 34,
+ 52,
+ 58,
+ 26,
+ 48
+ ]
+ ],
+ [
+ [
+ 54,
+ 23,
+ 53,
+ 11,
+ 58,
+ 3
+ ],
+ [
+ 11,
+ 30,
+ 59,
+ 58,
+ 63,
+ 4
+ ],
+ [
+ 20,
+ 29,
+ 58,
+ 17,
+ 42,
+ 4
+ ],
+ [
+ 1,
+ 35,
+ 40,
+ 45,
+ 53,
+ 59
+ ],
+ [
+ 40,
+ 55,
+ 21,
+ 33,
+ 38,
+ 49
+ ],
+ [
+ 45,
+ 29,
+ 61,
+ 27,
+ 63,
+ 62
+ ],
+ [
+ 33,
+ 57,
+ 11,
+ 28,
+ 53,
+ 34
+ ],
+ [
+ 11,
+ 63,
+ 39,
+ 45,
+ 10,
+ 14
+ ],
+ [
+ 30,
+ 54,
+ 57,
+ 59,
+ 33,
+ 26
+ ],
+ [
+ 43,
+ 23,
+ 5,
+ 18,
+ 21,
+ 42
+ ],
+ [
+ 11,
+ 18,
+ 2,
+ 9,
+ 34,
+ 6
+ ],
+ [
+ 22,
+ 60,
+ 28,
+ 2,
+ 63,
+ 17
+ ],
+ [
+ 5,
+ 41,
+ 6,
+ 17,
+ 56,
+ 29
+ ],
+ [
+ 55,
+ 6,
+ 5,
+ 2,
+ 48,
+ 59
+ ],
+ [
+ 23,
+ 19,
+ 62,
+ 22,
+ 11,
+ 9
+ ],
+ [
+ 12,
+ 45,
+ 41,
+ 27,
+ 8,
+ 42
+ ],
+ [
+ 11,
+ 53,
+ 41,
+ 44,
+ 51,
+ 7
+ ],
+ [
+ 21,
+ 2,
+ 36,
+ 6,
+ 50,
+ 56
+ ],
+ [
+ 13,
+ 10,
+ 48,
+ 53,
+ 61,
+ 19
+ ],
+ [
+ 47,
+ 21,
+ 56,
+ 44,
+ 6,
+ 31
+ ],
+ [
+ 44,
+ 12,
+ 3,
+ 55,
+ 41,
+ 53
+ ],
+ [
+ 44,
+ 47,
+ 28,
+ 43,
+ 45,
+ 63
+ ],
+ [
+ 1,
+ 25,
+ 53,
+ 11,
+ 39,
+ 38
+ ],
+ [
+ 1,
+ 59,
+ 38,
+ 3,
+ 37,
+ 42
+ ],
+ [
+ 45,
+ 3,
+ 0,
+ 21,
+ 22,
+ 14
+ ],
+ [
+ 10,
+ 28,
+ 42,
+ 49,
+ 11,
+ 62
+ ],
+ [
+ 30,
+ 57,
+ 15,
+ 56,
+ 16,
+ 41
+ ]
+ ],
+ [
+ [
+ 53,
+ 15,
+ 34,
+ 0,
+ 46,
+ 33
+ ],
+ [
+ 8,
+ 12,
+ 41,
+ 19,
+ 39,
+ 32
+ ],
+ [
+ 56,
+ 31,
+ 36,
+ 13,
+ 23,
+ 9
+ ],
+ [
+ 36,
+ 51,
+ 30,
+ 21,
+ 1,
+ 11
+ ],
+ [
+ 13,
+ 58,
+ 50,
+ 2,
+ 53,
+ 54
+ ],
+ [
+ 49,
+ 52,
+ 32,
+ 7,
+ 23,
+ 47
+ ],
+ [
+ 61,
+ 38,
+ 23,
+ 39,
+ 0,
+ 35
+ ],
+ [
+ 42,
+ 27,
+ 9,
+ 20,
+ 17,
+ 57
+ ],
+ [
+ 34,
+ 1,
+ 29,
+ 4,
+ 35,
+ 45
+ ],
+ [
+ 54,
+ 57,
+ 27,
+ 19,
+ 38,
+ 62
+ ],
+ [
+ 59,
+ 1,
+ 60,
+ 26,
+ 38,
+ 22
+ ],
+ [
+ 25,
+ 31,
+ 51,
+ 36,
+ 32,
+ 8
+ ],
+ [
+ 14,
+ 62,
+ 2,
+ 19,
+ 37,
+ 11
+ ],
+ [
+ 57,
+ 40,
+ 13,
+ 22,
+ 37,
+ 46
+ ],
+ [
+ 45,
+ 34,
+ 58,
+ 44,
+ 42,
+ 16
+ ],
+ [
+ 50,
+ 16,
+ 6,
+ 5,
+ 33,
+ 43
+ ],
+ [
+ 42,
+ 39,
+ 61,
+ 13,
+ 5,
+ 15
+ ],
+ [
+ 46,
+ 23,
+ 27,
+ 28,
+ 4,
+ 63
+ ],
+ [
+ 62,
+ 31,
+ 10,
+ 45,
+ 30,
+ 35
+ ],
+ [
+ 15,
+ 13,
+ 38,
+ 63,
+ 4,
+ 19
+ ],
+ [
+ 34,
+ 15,
+ 38,
+ 57,
+ 27,
+ 19
+ ],
+ [
+ 41,
+ 62,
+ 36,
+ 57,
+ 19,
+ 47
+ ],
+ [
+ 34,
+ 22,
+ 53,
+ 10,
+ 46,
+ 33
+ ],
+ [
+ 24,
+ 51,
+ 4,
+ 47,
+ 39,
+ 10
+ ],
+ [
+ 11,
+ 57,
+ 51,
+ 50,
+ 54,
+ 48
+ ],
+ [
+ 51,
+ 7,
+ 11,
+ 50,
+ 43,
+ 18
+ ],
+ [
+ 39,
+ 37,
+ 9,
+ 42,
+ 40,
+ 44
+ ]
+ ],
+ [
+ [
+ 57,
+ 17,
+ 62,
+ 42,
+ 23,
+ 60
+ ],
+ [
+ 18,
+ 7,
+ 53,
+ 43,
+ 26,
+ 60
+ ],
+ [
+ 60,
+ 5,
+ 3,
+ 53,
+ 23,
+ 57
+ ],
+ [
+ 17,
+ 10,
+ 22,
+ 19,
+ 11,
+ 31
+ ],
+ [
+ 10,
+ 15,
+ 12,
+ 27,
+ 17,
+ 4
+ ],
+ [
+ 2,
+ 44,
+ 39,
+ 36,
+ 25,
+ 54
+ ],
+ [
+ 52,
+ 62,
+ 37,
+ 21,
+ 41,
+ 42
+ ],
+ [
+ 62,
+ 7,
+ 46,
+ 30,
+ 36,
+ 14
+ ],
+ [
+ 50,
+ 4,
+ 10,
+ 58,
+ 0,
+ 16
+ ],
+ [
+ 32,
+ 47,
+ 38,
+ 54,
+ 8,
+ 41
+ ],
+ [
+ 41,
+ 32,
+ 3,
+ 18,
+ 1,
+ 22
+ ],
+ [
+ 16,
+ 50,
+ 53,
+ 7,
+ 44,
+ 12
+ ],
+ [
+ 11,
+ 39,
+ 3,
+ 35,
+ 25,
+ 48
+ ],
+ [
+ 18,
+ 9,
+ 15,
+ 13,
+ 63,
+ 28
+ ],
+ [
+ 62,
+ 58,
+ 13,
+ 5,
+ 3,
+ 17
+ ],
+ [
+ 3,
+ 31,
+ 43,
+ 53,
+ 35,
+ 57
+ ],
+ [
+ 24,
+ 51,
+ 15,
+ 46,
+ 5,
+ 59
+ ],
+ [
+ 13,
+ 30,
+ 0,
+ 32,
+ 5,
+ 59
+ ],
+ [
+ 2,
+ 39,
+ 32,
+ 38,
+ 34,
+ 22
+ ],
+ [
+ 42,
+ 26,
+ 34,
+ 28,
+ 37,
+ 54
+ ],
+ [
+ 28,
+ 43,
+ 53,
+ 41,
+ 13,
+ 0
+ ],
+ [
+ 14,
+ 15,
+ 34,
+ 1,
+ 48,
+ 40
+ ],
+ [
+ 5,
+ 25,
+ 4,
+ 33,
+ 39,
+ 53
+ ],
+ [
+ 58,
+ 4,
+ 17,
+ 57,
+ 51,
+ 12
+ ],
+ [
+ 47,
+ 37,
+ 35,
+ 31,
+ 63,
+ 29
+ ],
+ [
+ 15,
+ 3,
+ 28,
+ 33,
+ 23,
+ 9
+ ],
+ [
+ 23,
+ 6,
+ 58,
+ 47,
+ 30,
+ 56
+ ]
+ ],
+ [
+ [
+ 47,
+ 29,
+ 14,
+ 6,
+ 51,
+ 43
+ ],
+ [
+ 30,
+ 29,
+ 39,
+ 7,
+ 52,
+ 3
+ ],
+ [
+ 63,
+ 34,
+ 41,
+ 2,
+ 7,
+ 47
+ ],
+ [
+ 4,
+ 28,
+ 54,
+ 45,
+ 52,
+ 58
+ ],
+ [
+ 29,
+ 7,
+ 12,
+ 15,
+ 41,
+ 6
+ ],
+ [
+ 34,
+ 29,
+ 48,
+ 3,
+ 43,
+ 40
+ ],
+ [
+ 30,
+ 29,
+ 16,
+ 47,
+ 42,
+ 45
+ ],
+ [
+ 33,
+ 39,
+ 25,
+ 60,
+ 41,
+ 3
+ ],
+ [
+ 50,
+ 26,
+ 4,
+ 25,
+ 13,
+ 17
+ ],
+ [
+ 5,
+ 43,
+ 54,
+ 16,
+ 12,
+ 53
+ ],
+ [
+ 18,
+ 6,
+ 3,
+ 35,
+ 21,
+ 1
+ ],
+ [
+ 56,
+ 46,
+ 48,
+ 10,
+ 16,
+ 44
+ ],
+ [
+ 9,
+ 35,
+ 7,
+ 24,
+ 47,
+ 57
+ ],
+ [
+ 53,
+ 42,
+ 15,
+ 56,
+ 47,
+ 59
+ ],
+ [
+ 39,
+ 11,
+ 36,
+ 32,
+ 35,
+ 18
+ ],
+ [
+ 46,
+ 20,
+ 53,
+ 38,
+ 56,
+ 26
+ ],
+ [
+ 58,
+ 29,
+ 14,
+ 26,
+ 17,
+ 15
+ ],
+ [
+ 24,
+ 25,
+ 39,
+ 16,
+ 1,
+ 57
+ ],
+ [
+ 24,
+ 41,
+ 1,
+ 4,
+ 63,
+ 28
+ ],
+ [
+ 42,
+ 37,
+ 34,
+ 48,
+ 26,
+ 41
+ ],
+ [
+ 11,
+ 28,
+ 16,
+ 7,
+ 32,
+ 56
+ ],
+ [
+ 14,
+ 42,
+ 6,
+ 16,
+ 22,
+ 15
+ ],
+ [
+ 33,
+ 56,
+ 42,
+ 8,
+ 38,
+ 25
+ ],
+ [
+ 4,
+ 58,
+ 48,
+ 33,
+ 11,
+ 28
+ ],
+ [
+ 37,
+ 47,
+ 29,
+ 48,
+ 30,
+ 53
+ ],
+ [
+ 12,
+ 41,
+ 3,
+ 4,
+ 48,
+ 46
+ ],
+ [
+ 14,
+ 13,
+ 61,
+ 6,
+ 62,
+ 1
+ ]
+ ],
+ [
+ [
+ 45,
+ 10,
+ 44,
+ 43,
+ 53,
+ 33
+ ],
+ [
+ 32,
+ 63,
+ 22,
+ 27,
+ 30,
+ 29
+ ],
+ [
+ 54,
+ 35,
+ 37,
+ 32,
+ 26,
+ 30
+ ],
+ [
+ 24,
+ 63,
+ 0,
+ 17,
+ 25,
+ 45
+ ],
+ [
+ 40,
+ 7,
+ 0,
+ 57,
+ 29,
+ 22
+ ],
+ [
+ 10,
+ 34,
+ 20,
+ 22,
+ 43,
+ 33
+ ],
+ [
+ 42,
+ 30,
+ 5,
+ 25,
+ 19,
+ 34
+ ],
+ [
+ 33,
+ 18,
+ 35,
+ 51,
+ 7,
+ 57
+ ],
+ [
+ 50,
+ 28,
+ 25,
+ 4,
+ 10,
+ 9
+ ],
+ [
+ 5,
+ 38,
+ 16,
+ 43,
+ 54,
+ 12
+ ],
+ [
+ 18,
+ 21,
+ 3,
+ 6,
+ 39,
+ 53
+ ],
+ [
+ 56,
+ 16,
+ 53,
+ 39,
+ 46,
+ 42
+ ],
+ [
+ 9,
+ 35,
+ 57,
+ 11,
+ 47,
+ 13
+ ],
+ [
+ 56,
+ 15,
+ 3,
+ 59,
+ 9,
+ 28
+ ],
+ [
+ 4,
+ 62,
+ 39,
+ 58,
+ 63,
+ 36
+ ],
+ [
+ 20,
+ 53,
+ 57,
+ 8,
+ 51,
+ 35
+ ],
+ [
+ 51,
+ 49,
+ 11,
+ 26,
+ 15,
+ 14
+ ],
+ [
+ 0,
+ 25,
+ 62,
+ 59,
+ 21,
+ 13
+ ],
+ [
+ 39,
+ 34,
+ 48,
+ 53,
+ 61,
+ 33
+ ],
+ [
+ 26,
+ 47,
+ 28,
+ 34,
+ 21,
+ 39
+ ],
+ [
+ 28,
+ 43,
+ 12,
+ 53,
+ 41,
+ 32
+ ],
+ [
+ 14,
+ 52,
+ 17,
+ 1,
+ 15,
+ 38
+ ],
+ [
+ 25,
+ 4,
+ 5,
+ 11,
+ 58,
+ 50
+ ],
+ [
+ 58,
+ 4,
+ 17,
+ 10,
+ 25,
+ 57
+ ],
+ [
+ 29,
+ 47,
+ 35,
+ 31,
+ 52,
+ 48
+ ],
+ [
+ 55,
+ 28,
+ 23,
+ 15,
+ 3,
+ 24
+ ],
+ [
+ 6,
+ 30,
+ 57,
+ 32,
+ 34,
+ 62
+ ]
+ ],
+ [
+ [
+ 39,
+ 5,
+ 30,
+ 17,
+ 61,
+ 15
+ ],
+ [
+ 11,
+ 63,
+ 0,
+ 23,
+ 61,
+ 46
+ ],
+ [
+ 61,
+ 15,
+ 53,
+ 22,
+ 7,
+ 57
+ ],
+ [
+ 50,
+ 57,
+ 58,
+ 63,
+ 45,
+ 47
+ ],
+ [
+ 55,
+ 31,
+ 57,
+ 24,
+ 60,
+ 7
+ ],
+ [
+ 22,
+ 7,
+ 43,
+ 1,
+ 10,
+ 0
+ ],
+ [
+ 14,
+ 58,
+ 1,
+ 34,
+ 19,
+ 45
+ ],
+ [
+ 50,
+ 8,
+ 14,
+ 7,
+ 57,
+ 9
+ ],
+ [
+ 43,
+ 0,
+ 4,
+ 10,
+ 45,
+ 46
+ ],
+ [
+ 38,
+ 0,
+ 4,
+ 55,
+ 54,
+ 10
+ ],
+ [
+ 51,
+ 54,
+ 46,
+ 39,
+ 1,
+ 38
+ ],
+ [
+ 16,
+ 37,
+ 33,
+ 36,
+ 21,
+ 63
+ ],
+ [
+ 10,
+ 38,
+ 57,
+ 58,
+ 3,
+ 63
+ ],
+ [
+ 30,
+ 0,
+ 63,
+ 13,
+ 22,
+ 18
+ ],
+ [
+ 55,
+ 58,
+ 62,
+ 38,
+ 6,
+ 36
+ ],
+ [
+ 19,
+ 53,
+ 11,
+ 1,
+ 57,
+ 25
+ ],
+ [
+ 11,
+ 6,
+ 51,
+ 16,
+ 18,
+ 30
+ ],
+ [
+ 20,
+ 57,
+ 32,
+ 36,
+ 13,
+ 56
+ ],
+ [
+ 3,
+ 12,
+ 61,
+ 26,
+ 32,
+ 1
+ ],
+ [
+ 25,
+ 36,
+ 34,
+ 51,
+ 59,
+ 37
+ ],
+ [
+ 13,
+ 53,
+ 28,
+ 16,
+ 14,
+ 48
+ ],
+ [
+ 14,
+ 42,
+ 15,
+ 55,
+ 22,
+ 38
+ ],
+ [
+ 62,
+ 58,
+ 29,
+ 33,
+ 5,
+ 34
+ ],
+ [
+ 21,
+ 4,
+ 51,
+ 49,
+ 12,
+ 58
+ ],
+ [
+ 61,
+ 60,
+ 40,
+ 35,
+ 59,
+ 47
+ ],
+ [
+ 37,
+ 46,
+ 3,
+ 48,
+ 12,
+ 53
+ ],
+ [
+ 63,
+ 8,
+ 46,
+ 33,
+ 1,
+ 53
+ ]
+ ],
+ [
+ [
+ 62,
+ 61,
+ 0,
+ 9,
+ 26,
+ 41
+ ],
+ [
+ 45,
+ 3,
+ 29,
+ 35,
+ 2,
+ 25
+ ],
+ [
+ 56,
+ 31,
+ 53,
+ 28,
+ 23,
+ 2
+ ],
+ [
+ 13,
+ 41,
+ 46,
+ 49,
+ 5,
+ 45
+ ],
+ [
+ 18,
+ 25,
+ 57,
+ 55,
+ 50,
+ 15
+ ],
+ [
+ 58,
+ 22,
+ 4,
+ 46,
+ 19,
+ 12
+ ],
+ [
+ 1,
+ 25,
+ 58,
+ 22,
+ 43,
+ 35
+ ],
+ [
+ 23,
+ 50,
+ 12,
+ 8,
+ 9,
+ 16
+ ],
+ [
+ 29,
+ 17,
+ 16,
+ 43,
+ 10,
+ 4
+ ],
+ [
+ 38,
+ 55,
+ 0,
+ 40,
+ 20,
+ 10
+ ],
+ [
+ 51,
+ 19,
+ 54,
+ 39,
+ 46,
+ 1
+ ],
+ [
+ 60,
+ 16,
+ 37,
+ 11,
+ 56,
+ 36
+ ],
+ [
+ 10,
+ 7,
+ 38,
+ 57,
+ 54,
+ 44
+ ],
+ [
+ 45,
+ 0,
+ 13,
+ 30,
+ 47,
+ 63
+ ],
+ [
+ 58,
+ 48,
+ 55,
+ 38,
+ 36,
+ 29
+ ],
+ [
+ 29,
+ 53,
+ 0,
+ 1,
+ 55,
+ 57
+ ],
+ [
+ 11,
+ 12,
+ 6,
+ 29,
+ 5,
+ 40
+ ],
+ [
+ 57,
+ 32,
+ 42,
+ 8,
+ 20,
+ 36
+ ],
+ [
+ 3,
+ 61,
+ 49,
+ 32,
+ 26,
+ 1
+ ],
+ [
+ 42,
+ 5,
+ 36,
+ 16,
+ 39,
+ 51
+ ],
+ [
+ 9,
+ 13,
+ 7,
+ 53,
+ 46,
+ 28
+ ],
+ [
+ 14,
+ 55,
+ 42,
+ 24,
+ 33,
+ 47
+ ],
+ [
+ 33,
+ 58,
+ 62,
+ 15,
+ 36,
+ 5
+ ],
+ [
+ 51,
+ 4,
+ 49,
+ 58,
+ 21,
+ 41
+ ],
+ [
+ 40,
+ 60,
+ 19,
+ 35,
+ 61,
+ 15
+ ],
+ [
+ 14,
+ 21,
+ 3,
+ 12,
+ 5,
+ 37
+ ],
+ [
+ 35,
+ 60,
+ 1,
+ 10,
+ 53,
+ 54
+ ]
+ ],
+ [
+ [
+ 40,
+ 28,
+ 60,
+ 31,
+ 59,
+ 23
+ ],
+ [
+ 42,
+ 12,
+ 26,
+ 34,
+ 0,
+ 61
+ ],
+ [
+ 16,
+ 5,
+ 62,
+ 3,
+ 32,
+ 0
+ ],
+ [
+ 13,
+ 3,
+ 18,
+ 39,
+ 42,
+ 52
+ ],
+ [
+ 25,
+ 12,
+ 39,
+ 0,
+ 57,
+ 15
+ ],
+ [
+ 33,
+ 50,
+ 58,
+ 36,
+ 6,
+ 26
+ ],
+ [
+ 41,
+ 43,
+ 1,
+ 15,
+ 2,
+ 25
+ ],
+ [
+ 36,
+ 16,
+ 53,
+ 14,
+ 30,
+ 58
+ ],
+ [
+ 16,
+ 29,
+ 3,
+ 2,
+ 10,
+ 46
+ ],
+ [
+ 38,
+ 16,
+ 32,
+ 26,
+ 45,
+ 11
+ ],
+ [
+ 45,
+ 19,
+ 3,
+ 56,
+ 46,
+ 53
+ ],
+ [
+ 21,
+ 36,
+ 11,
+ 9,
+ 16,
+ 41
+ ],
+ [
+ 11,
+ 16,
+ 10,
+ 50,
+ 8,
+ 63
+ ],
+ [
+ 35,
+ 38,
+ 33,
+ 25,
+ 51,
+ 52
+ ],
+ [
+ 58,
+ 48,
+ 27,
+ 50,
+ 29,
+ 26
+ ],
+ [
+ 29,
+ 57,
+ 53,
+ 3,
+ 54,
+ 7
+ ],
+ [
+ 3,
+ 23,
+ 6,
+ 46,
+ 51,
+ 11
+ ],
+ [
+ 32,
+ 5,
+ 54,
+ 57,
+ 52,
+ 30
+ ],
+ [
+ 34,
+ 61,
+ 40,
+ 0,
+ 3,
+ 48
+ ],
+ [
+ 22,
+ 53,
+ 12,
+ 33,
+ 39,
+ 60
+ ],
+ [
+ 20,
+ 53,
+ 47,
+ 40,
+ 12,
+ 8
+ ],
+ [
+ 55,
+ 17,
+ 52,
+ 14,
+ 45,
+ 56
+ ],
+ [
+ 1,
+ 4,
+ 13,
+ 39,
+ 11,
+ 33
+ ],
+ [
+ 17,
+ 10,
+ 51,
+ 44,
+ 55,
+ 4
+ ],
+ [
+ 35,
+ 6,
+ 45,
+ 21,
+ 52,
+ 37
+ ],
+ [
+ 10,
+ 42,
+ 9,
+ 1,
+ 53,
+ 24
+ ],
+ [
+ 27,
+ 41,
+ 32,
+ 10,
+ 45,
+ 47
+ ]
+ ],
+ [
+ [
+ 11,
+ 34,
+ 44,
+ 51,
+ 41,
+ 12
+ ],
+ [
+ 20,
+ 34,
+ 3,
+ 25,
+ 63,
+ 16
+ ],
+ [
+ 40,
+ 26,
+ 37,
+ 22,
+ 15,
+ 54
+ ],
+ [
+ 4,
+ 16,
+ 25,
+ 28,
+ 45,
+ 58
+ ],
+ [
+ 33,
+ 10,
+ 32,
+ 24,
+ 3,
+ 4
+ ],
+ [
+ 0,
+ 9,
+ 58,
+ 3,
+ 34,
+ 15
+ ],
+ [
+ 16,
+ 63,
+ 43,
+ 59,
+ 25,
+ 42
+ ],
+ [
+ 16,
+ 48,
+ 36,
+ 33,
+ 25,
+ 58
+ ],
+ [
+ 20,
+ 50,
+ 16,
+ 23,
+ 42,
+ 29
+ ],
+ [
+ 60,
+ 51,
+ 16,
+ 22,
+ 38,
+ 48
+ ],
+ [
+ 31,
+ 19,
+ 38,
+ 3,
+ 2,
+ 43
+ ],
+ [
+ 56,
+ 9,
+ 21,
+ 40,
+ 18,
+ 44
+ ],
+ [
+ 11,
+ 13,
+ 10,
+ 45,
+ 27,
+ 57
+ ],
+ [
+ 56,
+ 14,
+ 35,
+ 44,
+ 45,
+ 13
+ ],
+ [
+ 58,
+ 43,
+ 26,
+ 33,
+ 31,
+ 50
+ ],
+ [
+ 13,
+ 38,
+ 46,
+ 51,
+ 28,
+ 37
+ ],
+ [
+ 58,
+ 63,
+ 42,
+ 10,
+ 16,
+ 0
+ ],
+ [
+ 0,
+ 1,
+ 53,
+ 52,
+ 24,
+ 59
+ ],
+ [
+ 17,
+ 29,
+ 0,
+ 40,
+ 60,
+ 24
+ ],
+ [
+ 17,
+ 24,
+ 2,
+ 26,
+ 35,
+ 23
+ ],
+ [
+ 44,
+ 20,
+ 22,
+ 4,
+ 50,
+ 40
+ ],
+ [
+ 9,
+ 61,
+ 17,
+ 16,
+ 27,
+ 37
+ ],
+ [
+ 63,
+ 13,
+ 16,
+ 11,
+ 24,
+ 2
+ ],
+ [
+ 44,
+ 48,
+ 23,
+ 49,
+ 56,
+ 45
+ ],
+ [
+ 6,
+ 5,
+ 49,
+ 33,
+ 31,
+ 63
+ ],
+ [
+ 49,
+ 47,
+ 1,
+ 57,
+ 4,
+ 44
+ ],
+ [
+ 2,
+ 56,
+ 19,
+ 63,
+ 39,
+ 50
+ ]
+ ],
+ [
+ [
+ 16,
+ 22,
+ 23,
+ 35,
+ 3,
+ 13
+ ],
+ [
+ 16,
+ 23,
+ 50,
+ 9,
+ 13,
+ 59
+ ],
+ [
+ 11,
+ 35,
+ 21,
+ 7,
+ 59,
+ 24
+ ],
+ [
+ 53,
+ 32,
+ 7,
+ 60,
+ 1,
+ 13
+ ],
+ [
+ 24,
+ 33,
+ 57,
+ 35,
+ 0,
+ 39
+ ],
+ [
+ 15,
+ 38,
+ 0,
+ 45,
+ 32,
+ 58
+ ],
+ [
+ 59,
+ 16,
+ 25,
+ 49,
+ 35,
+ 5
+ ],
+ [
+ 45,
+ 40,
+ 16,
+ 14,
+ 12,
+ 30
+ ],
+ [
+ 20,
+ 16,
+ 42,
+ 6,
+ 17,
+ 14
+ ],
+ [
+ 60,
+ 10,
+ 22,
+ 50,
+ 48,
+ 38
+ ],
+ [
+ 31,
+ 35,
+ 19,
+ 27,
+ 38,
+ 53
+ ],
+ [
+ 33,
+ 59,
+ 48,
+ 16,
+ 60,
+ 39
+ ],
+ [
+ 9,
+ 57,
+ 10,
+ 11,
+ 7,
+ 44
+ ],
+ [
+ 13,
+ 0,
+ 59,
+ 61,
+ 19,
+ 33
+ ],
+ [
+ 18,
+ 58,
+ 39,
+ 21,
+ 50,
+ 36
+ ],
+ [
+ 57,
+ 60,
+ 49,
+ 19,
+ 53,
+ 23
+ ],
+ [
+ 6,
+ 36,
+ 17,
+ 38,
+ 41,
+ 10
+ ],
+ [
+ 63,
+ 1,
+ 57,
+ 20,
+ 36,
+ 31
+ ],
+ [
+ 40,
+ 41,
+ 7,
+ 0,
+ 61,
+ 32
+ ],
+ [
+ 39,
+ 16,
+ 23,
+ 14,
+ 17,
+ 59
+ ],
+ [
+ 51,
+ 30,
+ 50,
+ 7,
+ 3,
+ 61
+ ],
+ [
+ 9,
+ 14,
+ 42,
+ 15,
+ 1,
+ 55
+ ],
+ [
+ 33,
+ 19,
+ 56,
+ 58,
+ 11,
+ 15
+ ],
+ [
+ 9,
+ 26,
+ 4,
+ 41,
+ 51,
+ 10
+ ],
+ [
+ 4,
+ 60,
+ 35,
+ 16,
+ 48,
+ 47
+ ],
+ [
+ 3,
+ 52,
+ 31,
+ 16,
+ 39,
+ 26
+ ],
+ [
+ 4,
+ 1,
+ 3,
+ 61,
+ 0,
+ 54
+ ]
+ ],
+ [
+ [
+ 38,
+ 45,
+ 52,
+ 43,
+ 32,
+ 39
+ ],
+ [
+ 29,
+ 48,
+ 22,
+ 60,
+ 55,
+ 57
+ ],
+ [
+ 5,
+ 49,
+ 8,
+ 20,
+ 14,
+ 55
+ ],
+ [
+ 35,
+ 46,
+ 49,
+ 32,
+ 7,
+ 48
+ ],
+ [
+ 24,
+ 29,
+ 49,
+ 16,
+ 41,
+ 0
+ ],
+ [
+ 56,
+ 3,
+ 35,
+ 15,
+ 4,
+ 40
+ ],
+ [
+ 49,
+ 59,
+ 4,
+ 16,
+ 33,
+ 11
+ ],
+ [
+ 12,
+ 51,
+ 30,
+ 40,
+ 56,
+ 27
+ ],
+ [
+ 23,
+ 14,
+ 46,
+ 55,
+ 42,
+ 17
+ ],
+ [
+ 60,
+ 22,
+ 38,
+ 6,
+ 21,
+ 50
+ ],
+ [
+ 11,
+ 19,
+ 31,
+ 35,
+ 38,
+ 22
+ ],
+ [
+ 59,
+ 48,
+ 19,
+ 33,
+ 16,
+ 9
+ ],
+ [
+ 9,
+ 28,
+ 45,
+ 55,
+ 57,
+ 16
+ ],
+ [
+ 59,
+ 1,
+ 25,
+ 47,
+ 6,
+ 3
+ ],
+ [
+ 9,
+ 39,
+ 50,
+ 18,
+ 58,
+ 21
+ ],
+ [
+ 60,
+ 57,
+ 38,
+ 53,
+ 50,
+ 35
+ ],
+ [
+ 41,
+ 53,
+ 6,
+ 38,
+ 10,
+ 25
+ ],
+ [
+ 1,
+ 63,
+ 15,
+ 24,
+ 36,
+ 29
+ ],
+ [
+ 41,
+ 40,
+ 7,
+ 42,
+ 32,
+ 62
+ ],
+ [
+ 4,
+ 10,
+ 14,
+ 37,
+ 39,
+ 17
+ ],
+ [
+ 61,
+ 30,
+ 7,
+ 50,
+ 57,
+ 3
+ ],
+ [
+ 9,
+ 14,
+ 58,
+ 30,
+ 42,
+ 8
+ ],
+ [
+ 33,
+ 56,
+ 19,
+ 60,
+ 24,
+ 59
+ ],
+ [
+ 9,
+ 26,
+ 4,
+ 51,
+ 25,
+ 41
+ ],
+ [
+ 4,
+ 35,
+ 47,
+ 41,
+ 48,
+ 60
+ ],
+ [
+ 39,
+ 52,
+ 3,
+ 26,
+ 8,
+ 30
+ ],
+ [
+ 4,
+ 1,
+ 10,
+ 61,
+ 12,
+ 0
+ ]
+ ],
+ [
+ [
+ 51,
+ 53,
+ 33,
+ 13,
+ 28,
+ 48
+ ],
+ [
+ 63,
+ 31,
+ 41,
+ 39,
+ 40,
+ 49
+ ],
+ [
+ 42,
+ 14,
+ 3,
+ 24,
+ 50,
+ 44
+ ],
+ [
+ 11,
+ 39,
+ 52,
+ 10,
+ 17,
+ 42
+ ],
+ [
+ 7,
+ 60,
+ 58,
+ 15,
+ 12,
+ 27
+ ],
+ [
+ 34,
+ 36,
+ 44,
+ 28,
+ 29,
+ 40
+ ],
+ [
+ 52,
+ 17,
+ 47,
+ 42,
+ 37,
+ 41
+ ],
+ [
+ 62,
+ 33,
+ 38,
+ 39,
+ 41,
+ 36
+ ],
+ [
+ 50,
+ 58,
+ 37,
+ 10,
+ 16,
+ 62
+ ],
+ [
+ 41,
+ 59,
+ 12,
+ 62,
+ 49,
+ 58
+ ],
+ [
+ 3,
+ 10,
+ 35,
+ 41,
+ 25,
+ 6
+ ],
+ [
+ 39,
+ 53,
+ 58,
+ 21,
+ 19,
+ 54
+ ],
+ [
+ 42,
+ 35,
+ 7,
+ 39,
+ 63,
+ 21
+ ],
+ [
+ 15,
+ 9,
+ 3,
+ 54,
+ 51,
+ 32
+ ],
+ [
+ 3,
+ 35,
+ 32,
+ 36,
+ 18,
+ 13
+ ],
+ [
+ 20,
+ 3,
+ 35,
+ 45,
+ 4,
+ 32
+ ],
+ [
+ 46,
+ 24,
+ 32,
+ 33,
+ 14,
+ 44
+ ],
+ [
+ 16,
+ 57,
+ 30,
+ 19,
+ 63,
+ 61
+ ],
+ [
+ 41,
+ 25,
+ 2,
+ 24,
+ 26,
+ 4
+ ],
+ [
+ 42,
+ 37,
+ 47,
+ 48,
+ 33,
+ 50
+ ],
+ [
+ 13,
+ 54,
+ 53,
+ 56,
+ 63,
+ 0
+ ],
+ [
+ 34,
+ 14,
+ 7,
+ 51,
+ 42,
+ 6
+ ],
+ [
+ 6,
+ 43,
+ 25,
+ 58,
+ 39,
+ 11
+ ],
+ [
+ 0,
+ 26,
+ 44,
+ 4,
+ 51,
+ 60
+ ],
+ [
+ 37,
+ 63,
+ 4,
+ 15,
+ 13,
+ 61
+ ],
+ [
+ 9,
+ 59,
+ 41,
+ 1,
+ 52,
+ 25
+ ],
+ [
+ 56,
+ 58,
+ 47,
+ 1,
+ 62,
+ 51
+ ]
+ ],
+ [
+ [
+ 52,
+ 47,
+ 27,
+ 36,
+ 38,
+ 29
+ ],
+ [
+ 43,
+ 56,
+ 4,
+ 25,
+ 52,
+ 21
+ ],
+ [
+ 25,
+ 54,
+ 35,
+ 18,
+ 11,
+ 63
+ ],
+ [
+ 17,
+ 4,
+ 1,
+ 18,
+ 50,
+ 39
+ ],
+ [
+ 36,
+ 7,
+ 32,
+ 4,
+ 30,
+ 60
+ ],
+ [
+ 34,
+ 3,
+ 61,
+ 48,
+ 24,
+ 40
+ ],
+ [
+ 17,
+ 42,
+ 3,
+ 12,
+ 29,
+ 25
+ ],
+ [
+ 33,
+ 38,
+ 39,
+ 55,
+ 17,
+ 19
+ ],
+ [
+ 17,
+ 50,
+ 41,
+ 13,
+ 16,
+ 51
+ ],
+ [
+ 41,
+ 12,
+ 60,
+ 16,
+ 62,
+ 58
+ ],
+ [
+ 3,
+ 36,
+ 28,
+ 39,
+ 35,
+ 2
+ ],
+ [
+ 48,
+ 27,
+ 53,
+ 54,
+ 19,
+ 43
+ ],
+ [
+ 42,
+ 63,
+ 7,
+ 35,
+ 62,
+ 3
+ ],
+ [
+ 56,
+ 15,
+ 9,
+ 51,
+ 42,
+ 47
+ ],
+ [
+ 35,
+ 39,
+ 18,
+ 36,
+ 43,
+ 38
+ ],
+ [
+ 60,
+ 20,
+ 0,
+ 3,
+ 35,
+ 31
+ ],
+ [
+ 24,
+ 42,
+ 14,
+ 17,
+ 32,
+ 62
+ ],
+ [
+ 24,
+ 57,
+ 63,
+ 0,
+ 42,
+ 1
+ ],
+ [
+ 41,
+ 50,
+ 28,
+ 32,
+ 26,
+ 40
+ ],
+ [
+ 2,
+ 37,
+ 46,
+ 17,
+ 42,
+ 57
+ ],
+ [
+ 49,
+ 7,
+ 11,
+ 54,
+ 13,
+ 53
+ ],
+ [
+ 14,
+ 9,
+ 42,
+ 6,
+ 19,
+ 58
+ ],
+ [
+ 63,
+ 33,
+ 43,
+ 19,
+ 58,
+ 25
+ ],
+ [
+ 9,
+ 26,
+ 48,
+ 4,
+ 41,
+ 44
+ ],
+ [
+ 4,
+ 41,
+ 5,
+ 15,
+ 9,
+ 53
+ ],
+ [
+ 14,
+ 38,
+ 9,
+ 3,
+ 52,
+ 8
+ ],
+ [
+ 23,
+ 1,
+ 61,
+ 4,
+ 47,
+ 14
+ ]
+ ],
+ [
+ [
+ 37,
+ 14,
+ 3,
+ 5,
+ 33,
+ 38
+ ],
+ [
+ 34,
+ 0,
+ 56,
+ 58,
+ 37,
+ 13
+ ],
+ [
+ 61,
+ 14,
+ 22,
+ 29,
+ 15,
+ 46
+ ],
+ [
+ 50,
+ 58,
+ 18,
+ 44,
+ 47,
+ 17
+ ],
+ [
+ 4,
+ 36,
+ 53,
+ 7,
+ 32,
+ 2
+ ],
+ [
+ 22,
+ 1,
+ 34,
+ 61,
+ 10,
+ 3
+ ],
+ [
+ 28,
+ 25,
+ 42,
+ 30,
+ 29,
+ 46
+ ],
+ [
+ 33,
+ 38,
+ 51,
+ 39,
+ 60,
+ 62
+ ],
+ [
+ 17,
+ 51,
+ 16,
+ 26,
+ 46,
+ 50
+ ],
+ [
+ 12,
+ 16,
+ 59,
+ 6,
+ 38,
+ 3
+ ],
+ [
+ 39,
+ 35,
+ 31,
+ 3,
+ 36,
+ 0
+ ],
+ [
+ 48,
+ 19,
+ 53,
+ 56,
+ 27,
+ 39
+ ],
+ [
+ 7,
+ 62,
+ 9,
+ 63,
+ 15,
+ 42
+ ],
+ [
+ 56,
+ 19,
+ 9,
+ 47,
+ 15,
+ 1
+ ],
+ [
+ 39,
+ 36,
+ 18,
+ 35,
+ 42,
+ 38
+ ],
+ [
+ 60,
+ 20,
+ 0,
+ 57,
+ 47,
+ 53
+ ],
+ [
+ 6,
+ 32,
+ 29,
+ 62,
+ 43,
+ 5
+ ],
+ [
+ 24,
+ 63,
+ 57,
+ 1,
+ 42,
+ 2
+ ],
+ [
+ 41,
+ 40,
+ 7,
+ 62,
+ 32,
+ 50
+ ],
+ [
+ 37,
+ 14,
+ 10,
+ 2,
+ 57,
+ 17
+ ],
+ [
+ 7,
+ 54,
+ 30,
+ 53,
+ 50,
+ 3
+ ],
+ [
+ 9,
+ 14,
+ 8,
+ 42,
+ 6,
+ 58
+ ],
+ [
+ 33,
+ 56,
+ 60,
+ 19,
+ 18,
+ 15
+ ],
+ [
+ 9,
+ 26,
+ 4,
+ 41,
+ 59,
+ 23
+ ],
+ [
+ 4,
+ 41,
+ 47,
+ 9,
+ 2,
+ 48
+ ],
+ [
+ 3,
+ 14,
+ 16,
+ 30,
+ 52,
+ 8
+ ],
+ [
+ 1,
+ 4,
+ 14,
+ 61,
+ 0,
+ 12
+ ]
+ ],
+ [
+ [
+ 27,
+ 21,
+ 61,
+ 30,
+ 22,
+ 40
+ ],
+ [
+ 33,
+ 12,
+ 44,
+ 0,
+ 47,
+ 20
+ ],
+ [
+ 14,
+ 39,
+ 58,
+ 37,
+ 16,
+ 61
+ ],
+ [
+ 47,
+ 11,
+ 4,
+ 18,
+ 32,
+ 35
+ ],
+ [
+ 2,
+ 19,
+ 39,
+ 20,
+ 10,
+ 3
+ ],
+ [
+ 39,
+ 61,
+ 49,
+ 15,
+ 30,
+ 47
+ ],
+ [
+ 12,
+ 47,
+ 17,
+ 49,
+ 29,
+ 46
+ ],
+ [
+ 39,
+ 41,
+ 62,
+ 30,
+ 37,
+ 52
+ ],
+ [
+ 21,
+ 15,
+ 28,
+ 48,
+ 26,
+ 27
+ ],
+ [
+ 12,
+ 32,
+ 16,
+ 24,
+ 55,
+ 41
+ ],
+ [
+ 35,
+ 43,
+ 28,
+ 31,
+ 60,
+ 47
+ ],
+ [
+ 3,
+ 17,
+ 19,
+ 49,
+ 10,
+ 42
+ ],
+ [
+ 7,
+ 54,
+ 27,
+ 59,
+ 4,
+ 23
+ ],
+ [
+ 35,
+ 33,
+ 7,
+ 6,
+ 51,
+ 14
+ ],
+ [
+ 51,
+ 28,
+ 36,
+ 42,
+ 41,
+ 32
+ ],
+ [
+ 45,
+ 52,
+ 60,
+ 21,
+ 53,
+ 12
+ ],
+ [
+ 53,
+ 1,
+ 23,
+ 5,
+ 6,
+ 46
+ ],
+ [
+ 35,
+ 9,
+ 63,
+ 16,
+ 26,
+ 22
+ ],
+ [
+ 41,
+ 60,
+ 27,
+ 40,
+ 42,
+ 15
+ ],
+ [
+ 42,
+ 48,
+ 46,
+ 9,
+ 17,
+ 37
+ ],
+ [
+ 31,
+ 11,
+ 23,
+ 58,
+ 39,
+ 44
+ ],
+ [
+ 34,
+ 2,
+ 4,
+ 14,
+ 13,
+ 9
+ ],
+ [
+ 60,
+ 59,
+ 47,
+ 40,
+ 17,
+ 27
+ ],
+ [
+ 60,
+ 63,
+ 7,
+ 42,
+ 49,
+ 40
+ ],
+ [
+ 59,
+ 15,
+ 38,
+ 62,
+ 44,
+ 41
+ ],
+ [
+ 23,
+ 57,
+ 60,
+ 31,
+ 41,
+ 3
+ ],
+ [
+ 38,
+ 59,
+ 31,
+ 51,
+ 36,
+ 7
+ ]
+ ],
+ [
+ [
+ 12,
+ 10,
+ 50,
+ 23,
+ 53,
+ 33
+ ],
+ [
+ 48,
+ 56,
+ 44,
+ 11,
+ 31,
+ 17
+ ],
+ [
+ 33,
+ 11,
+ 17,
+ 54,
+ 15,
+ 62
+ ],
+ [
+ 31,
+ 13,
+ 17,
+ 40,
+ 8,
+ 3
+ ],
+ [
+ 42,
+ 4,
+ 27,
+ 15,
+ 12,
+ 5
+ ],
+ [
+ 34,
+ 50,
+ 33,
+ 29,
+ 55,
+ 6
+ ],
+ [
+ 16,
+ 49,
+ 52,
+ 29,
+ 41,
+ 30
+ ],
+ [
+ 0,
+ 30,
+ 62,
+ 57,
+ 26,
+ 22
+ ],
+ [
+ 26,
+ 62,
+ 5,
+ 58,
+ 51,
+ 49
+ ],
+ [
+ 5,
+ 62,
+ 16,
+ 46,
+ 39,
+ 37
+ ],
+ [
+ 31,
+ 10,
+ 24,
+ 45,
+ 18,
+ 35
+ ],
+ [
+ 50,
+ 19,
+ 48,
+ 54,
+ 16,
+ 35
+ ],
+ [
+ 59,
+ 39,
+ 62,
+ 7,
+ 35,
+ 28
+ ],
+ [
+ 20,
+ 9,
+ 19,
+ 6,
+ 22,
+ 15
+ ],
+ [
+ 35,
+ 18,
+ 50,
+ 32,
+ 36,
+ 39
+ ],
+ [
+ 3,
+ 4,
+ 13,
+ 38,
+ 60,
+ 20
+ ],
+ [
+ 46,
+ 32,
+ 28,
+ 3,
+ 37,
+ 33
+ ],
+ [
+ 30,
+ 63,
+ 16,
+ 19,
+ 24,
+ 42
+ ],
+ [
+ 41,
+ 17,
+ 4,
+ 2,
+ 34,
+ 32
+ ],
+ [
+ 42,
+ 37,
+ 48,
+ 3,
+ 33,
+ 31
+ ],
+ [
+ 56,
+ 7,
+ 25,
+ 11,
+ 39,
+ 44
+ ],
+ [
+ 42,
+ 14,
+ 26,
+ 13,
+ 22,
+ 12
+ ],
+ [
+ 25,
+ 6,
+ 11,
+ 42,
+ 13,
+ 38
+ ],
+ [
+ 0,
+ 9,
+ 26,
+ 41,
+ 4,
+ 10
+ ],
+ [
+ 37,
+ 4,
+ 63,
+ 41,
+ 2,
+ 44
+ ],
+ [
+ 9,
+ 52,
+ 3,
+ 41,
+ 16,
+ 59
+ ],
+ [
+ 58,
+ 47,
+ 56,
+ 1,
+ 26,
+ 62
+ ]
+ ],
+ [
+ [
+ 54,
+ 30,
+ 26,
+ 22,
+ 3,
+ 55
+ ],
+ [
+ 44,
+ 7,
+ 49,
+ 50,
+ 25,
+ 5
+ ],
+ [
+ 54,
+ 4,
+ 48,
+ 58,
+ 26,
+ 32
+ ],
+ [
+ 20,
+ 25,
+ 3,
+ 9,
+ 55,
+ 28
+ ],
+ [
+ 7,
+ 27,
+ 42,
+ 12,
+ 58,
+ 32
+ ],
+ [
+ 50,
+ 34,
+ 6,
+ 42,
+ 29,
+ 55
+ ],
+ [
+ 16,
+ 49,
+ 40,
+ 3,
+ 27,
+ 11
+ ],
+ [
+ 51,
+ 30,
+ 26,
+ 62,
+ 57,
+ 53
+ ],
+ [
+ 3,
+ 5,
+ 49,
+ 28,
+ 26,
+ 50
+ ],
+ [
+ 16,
+ 22,
+ 46,
+ 6,
+ 49,
+ 21
+ ],
+ [
+ 31,
+ 45,
+ 11,
+ 10,
+ 56,
+ 24
+ ],
+ [
+ 21,
+ 51,
+ 50,
+ 19,
+ 9,
+ 61
+ ],
+ [
+ 28,
+ 41,
+ 13,
+ 59,
+ 34,
+ 53
+ ],
+ [
+ 23,
+ 20,
+ 16,
+ 9,
+ 38,
+ 19
+ ],
+ [
+ 50,
+ 34,
+ 58,
+ 20,
+ 27,
+ 35
+ ],
+ [
+ 8,
+ 59,
+ 61,
+ 29,
+ 35,
+ 53
+ ],
+ [
+ 3,
+ 46,
+ 51,
+ 10,
+ 25,
+ 37
+ ],
+ [
+ 62,
+ 21,
+ 25,
+ 9,
+ 18,
+ 56
+ ],
+ [
+ 34,
+ 27,
+ 10,
+ 29,
+ 53,
+ 48
+ ],
+ [
+ 56,
+ 33,
+ 44,
+ 24,
+ 9,
+ 0
+ ],
+ [
+ 44,
+ 12,
+ 19,
+ 30,
+ 8,
+ 49
+ ],
+ [
+ 47,
+ 62,
+ 12,
+ 51,
+ 16,
+ 17
+ ],
+ [
+ 11,
+ 37,
+ 25,
+ 58,
+ 35,
+ 45
+ ],
+ [
+ 4,
+ 9,
+ 10,
+ 37,
+ 39,
+ 16
+ ],
+ [
+ 22,
+ 11,
+ 21,
+ 48,
+ 45,
+ 47
+ ],
+ [
+ 10,
+ 28,
+ 55,
+ 12,
+ 24,
+ 23
+ ],
+ [
+ 30,
+ 16,
+ 27,
+ 32,
+ 57,
+ 15
+ ]
+ ],
+ [
+ [
+ 16,
+ 11,
+ 31,
+ 46,
+ 35,
+ 0
+ ],
+ [
+ 13,
+ 49,
+ 10,
+ 54,
+ 5,
+ 6
+ ],
+ [
+ 36,
+ 13,
+ 27,
+ 46,
+ 3,
+ 18
+ ],
+ [
+ 24,
+ 61,
+ 15,
+ 0,
+ 13,
+ 63
+ ],
+ [
+ 17,
+ 2,
+ 58,
+ 50,
+ 35,
+ 19
+ ],
+ [
+ 8,
+ 7,
+ 49,
+ 52,
+ 47,
+ 23
+ ],
+ [
+ 61,
+ 16,
+ 58,
+ 20,
+ 38,
+ 23
+ ],
+ [
+ 20,
+ 42,
+ 9,
+ 51,
+ 35,
+ 16
+ ],
+ [
+ 25,
+ 47,
+ 4,
+ 1,
+ 50,
+ 63
+ ],
+ [
+ 54,
+ 38,
+ 57,
+ 27,
+ 33,
+ 28
+ ],
+ [
+ 1,
+ 59,
+ 30,
+ 60,
+ 14,
+ 31
+ ],
+ [
+ 31,
+ 51,
+ 25,
+ 14,
+ 41,
+ 55
+ ],
+ [
+ 34,
+ 2,
+ 14,
+ 11,
+ 19,
+ 28
+ ],
+ [
+ 40,
+ 57,
+ 13,
+ 61,
+ 59,
+ 18
+ ],
+ [
+ 44,
+ 58,
+ 50,
+ 45,
+ 37,
+ 26
+ ],
+ [
+ 5,
+ 8,
+ 62,
+ 24,
+ 38,
+ 57
+ ],
+ [
+ 13,
+ 47,
+ 15,
+ 3,
+ 5,
+ 39
+ ],
+ [
+ 25,
+ 23,
+ 9,
+ 18,
+ 63,
+ 33
+ ],
+ [
+ 1,
+ 10,
+ 20,
+ 53,
+ 8,
+ 4
+ ],
+ [
+ 38,
+ 15,
+ 37,
+ 34,
+ 60,
+ 0
+ ],
+ [
+ 41,
+ 27,
+ 30,
+ 57,
+ 19,
+ 40
+ ],
+ [
+ 11,
+ 62,
+ 41,
+ 46,
+ 14,
+ 44
+ ],
+ [
+ 11,
+ 33,
+ 46,
+ 31,
+ 45,
+ 0
+ ],
+ [
+ 4,
+ 51,
+ 47,
+ 16,
+ 9,
+ 12
+ ],
+ [
+ 11,
+ 21,
+ 45,
+ 47,
+ 18,
+ 61
+ ],
+ [
+ 10,
+ 50,
+ 51,
+ 12,
+ 18,
+ 7
+ ],
+ [
+ 17,
+ 31,
+ 36,
+ 5,
+ 19,
+ 1
+ ]
+ ],
+ [
+ [
+ 22,
+ 6,
+ 57,
+ 39,
+ 29,
+ 47
+ ],
+ [
+ 27,
+ 6,
+ 14,
+ 17,
+ 51,
+ 32
+ ],
+ [
+ 1,
+ 29,
+ 11,
+ 26,
+ 47,
+ 51
+ ],
+ [
+ 14,
+ 38,
+ 22,
+ 31,
+ 29,
+ 53
+ ],
+ [
+ 14,
+ 61,
+ 59,
+ 1,
+ 29,
+ 49
+ ],
+ [
+ 30,
+ 8,
+ 21,
+ 47,
+ 52,
+ 0
+ ],
+ [
+ 4,
+ 58,
+ 61,
+ 23,
+ 20,
+ 29
+ ],
+ [
+ 20,
+ 9,
+ 42,
+ 35,
+ 7,
+ 24
+ ],
+ [
+ 47,
+ 25,
+ 4,
+ 1,
+ 29,
+ 0
+ ],
+ [
+ 54,
+ 19,
+ 38,
+ 29,
+ 33,
+ 44
+ ],
+ [
+ 14,
+ 1,
+ 59,
+ 40,
+ 60,
+ 20
+ ],
+ [
+ 51,
+ 0,
+ 14,
+ 62,
+ 16,
+ 52
+ ],
+ [
+ 2,
+ 36,
+ 20,
+ 29,
+ 19,
+ 52
+ ],
+ [
+ 57,
+ 40,
+ 13,
+ 22,
+ 60,
+ 6
+ ],
+ [
+ 37,
+ 44,
+ 58,
+ 5,
+ 8,
+ 50
+ ],
+ [
+ 24,
+ 5,
+ 43,
+ 23,
+ 62,
+ 59
+ ],
+ [
+ 13,
+ 19,
+ 47,
+ 39,
+ 61,
+ 15
+ ],
+ [
+ 58,
+ 33,
+ 9,
+ 7,
+ 4,
+ 28
+ ],
+ [
+ 1,
+ 35,
+ 10,
+ 19,
+ 31,
+ 20
+ ],
+ [
+ 15,
+ 55,
+ 63,
+ 18,
+ 34,
+ 38
+ ],
+ [
+ 27,
+ 15,
+ 38,
+ 30,
+ 57,
+ 42
+ ],
+ [
+ 62,
+ 41,
+ 16,
+ 29,
+ 6,
+ 19
+ ],
+ [
+ 46,
+ 33,
+ 45,
+ 10,
+ 34,
+ 23
+ ],
+ [
+ 4,
+ 51,
+ 47,
+ 1,
+ 16,
+ 41
+ ],
+ [
+ 11,
+ 21,
+ 18,
+ 7,
+ 48,
+ 28
+ ],
+ [
+ 51,
+ 18,
+ 7,
+ 50,
+ 6,
+ 32
+ ],
+ [
+ 9,
+ 11,
+ 36,
+ 55,
+ 43,
+ 48
+ ]
+ ],
+ [
+ [
+ 47,
+ 8,
+ 36,
+ 61,
+ 21,
+ 45
+ ],
+ [
+ 46,
+ 2,
+ 15,
+ 32,
+ 0,
+ 51
+ ],
+ [
+ 24,
+ 15,
+ 33,
+ 61,
+ 2,
+ 43
+ ],
+ [
+ 60,
+ 22,
+ 31,
+ 27,
+ 14,
+ 11
+ ],
+ [
+ 59,
+ 58,
+ 39,
+ 57,
+ 46,
+ 3
+ ],
+ [
+ 57,
+ 43,
+ 2,
+ 31,
+ 7,
+ 62
+ ],
+ [
+ 9,
+ 42,
+ 54,
+ 19,
+ 4,
+ 55
+ ],
+ [
+ 46,
+ 14,
+ 7,
+ 24,
+ 43,
+ 35
+ ],
+ [
+ 47,
+ 4,
+ 0,
+ 37,
+ 12,
+ 13
+ ],
+ [
+ 54,
+ 38,
+ 4,
+ 47,
+ 25,
+ 6
+ ],
+ [
+ 47,
+ 14,
+ 15,
+ 24,
+ 1,
+ 61
+ ],
+ [
+ 16,
+ 54,
+ 5,
+ 0,
+ 7,
+ 63
+ ],
+ [
+ 49,
+ 3,
+ 33,
+ 13,
+ 11,
+ 10
+ ],
+ [
+ 18,
+ 13,
+ 10,
+ 39,
+ 58,
+ 60
+ ],
+ [
+ 58,
+ 62,
+ 10,
+ 33,
+ 5,
+ 26
+ ],
+ [
+ 48,
+ 5,
+ 63,
+ 53,
+ 43,
+ 2
+ ],
+ [
+ 24,
+ 47,
+ 51,
+ 15,
+ 0,
+ 53
+ ],
+ [
+ 44,
+ 0,
+ 6,
+ 3,
+ 7,
+ 34
+ ],
+ [
+ 58,
+ 38,
+ 53,
+ 61,
+ 0,
+ 54
+ ],
+ [
+ 55,
+ 9,
+ 15,
+ 27,
+ 42,
+ 34
+ ],
+ [
+ 43,
+ 1,
+ 51,
+ 41,
+ 4,
+ 5
+ ],
+ [
+ 49,
+ 20,
+ 15,
+ 6,
+ 37,
+ 46
+ ],
+ [
+ 11,
+ 25,
+ 52,
+ 5,
+ 4,
+ 39
+ ],
+ [
+ 38,
+ 40,
+ 44,
+ 51,
+ 10,
+ 14
+ ],
+ [
+ 13,
+ 8,
+ 52,
+ 63,
+ 2,
+ 23
+ ],
+ [
+ 23,
+ 38,
+ 59,
+ 57,
+ 55,
+ 41
+ ],
+ [
+ 23,
+ 6,
+ 62,
+ 0,
+ 7,
+ 28
+ ]
+ ],
+ [
+ [
+ 41,
+ 2,
+ 42,
+ 16,
+ 50,
+ 61
+ ],
+ [
+ 51,
+ 41,
+ 5,
+ 15,
+ 10,
+ 61
+ ],
+ [
+ 43,
+ 1,
+ 29,
+ 55,
+ 21,
+ 60
+ ],
+ [
+ 24,
+ 53,
+ 25,
+ 13,
+ 51,
+ 32
+ ],
+ [
+ 31,
+ 41,
+ 57,
+ 49,
+ 34,
+ 11
+ ],
+ [
+ 17,
+ 4,
+ 35,
+ 30,
+ 10,
+ 38
+ ],
+ [
+ 34,
+ 7,
+ 56,
+ 42,
+ 21,
+ 19
+ ],
+ [
+ 14,
+ 46,
+ 7,
+ 27,
+ 25,
+ 52
+ ],
+ [
+ 0,
+ 4,
+ 6,
+ 12,
+ 60,
+ 47
+ ],
+ [
+ 54,
+ 25,
+ 4,
+ 38,
+ 47,
+ 6
+ ],
+ [
+ 24,
+ 61,
+ 15,
+ 46,
+ 7,
+ 22
+ ],
+ [
+ 5,
+ 16,
+ 57,
+ 0,
+ 22,
+ 55
+ ],
+ [
+ 49,
+ 3,
+ 26,
+ 17,
+ 57,
+ 52
+ ],
+ [
+ 13,
+ 10,
+ 61,
+ 60,
+ 0,
+ 58
+ ],
+ [
+ 58,
+ 6,
+ 49,
+ 10,
+ 5,
+ 2
+ ],
+ [
+ 25,
+ 19,
+ 2,
+ 11,
+ 54,
+ 53
+ ],
+ [
+ 47,
+ 30,
+ 27,
+ 5,
+ 18,
+ 10
+ ],
+ [
+ 44,
+ 55,
+ 63,
+ 13,
+ 22,
+ 31
+ ],
+ [
+ 58,
+ 7,
+ 38,
+ 35,
+ 32,
+ 40
+ ],
+ [
+ 13,
+ 34,
+ 16,
+ 49,
+ 45,
+ 55
+ ],
+ [
+ 51,
+ 35,
+ 30,
+ 58,
+ 55,
+ 4
+ ],
+ [
+ 46,
+ 57,
+ 15,
+ 36,
+ 30,
+ 27
+ ],
+ [
+ 52,
+ 33,
+ 23,
+ 10,
+ 51,
+ 15
+ ],
+ [
+ 40,
+ 41,
+ 4,
+ 51,
+ 10,
+ 31
+ ],
+ [
+ 8,
+ 4,
+ 59,
+ 48,
+ 34,
+ 9
+ ],
+ [
+ 43,
+ 3,
+ 27,
+ 26,
+ 19,
+ 31
+ ],
+ [
+ 46,
+ 18,
+ 8,
+ 4,
+ 40,
+ 50
+ ]
+ ],
+ [
+ [
+ 22,
+ 36,
+ 35,
+ 63,
+ 43,
+ 23
+ ],
+ [
+ 54,
+ 30,
+ 4,
+ 36,
+ 35,
+ 55
+ ],
+ [
+ 28,
+ 19,
+ 23,
+ 49,
+ 50,
+ 59
+ ],
+ [
+ 62,
+ 5,
+ 50,
+ 53,
+ 42,
+ 48
+ ],
+ [
+ 0,
+ 3,
+ 61,
+ 57,
+ 41,
+ 49
+ ],
+ [
+ 60,
+ 4,
+ 29,
+ 16,
+ 53,
+ 30
+ ],
+ [
+ 34,
+ 32,
+ 33,
+ 9,
+ 56,
+ 25
+ ],
+ [
+ 12,
+ 53,
+ 14,
+ 36,
+ 25,
+ 2
+ ],
+ [
+ 23,
+ 48,
+ 35,
+ 29,
+ 4,
+ 16
+ ],
+ [
+ 22,
+ 25,
+ 4,
+ 54,
+ 62,
+ 5
+ ],
+ [
+ 24,
+ 19,
+ 27,
+ 55,
+ 47,
+ 25
+ ],
+ [
+ 18,
+ 5,
+ 22,
+ 34,
+ 63,
+ 30
+ ],
+ [
+ 17,
+ 49,
+ 30,
+ 28,
+ 11,
+ 42
+ ],
+ [
+ 29,
+ 13,
+ 54,
+ 25,
+ 45,
+ 47
+ ],
+ [
+ 27,
+ 2,
+ 10,
+ 5,
+ 54,
+ 33
+ ],
+ [
+ 55,
+ 2,
+ 54,
+ 57,
+ 22,
+ 56
+ ],
+ [
+ 53,
+ 3,
+ 60,
+ 27,
+ 5,
+ 50
+ ],
+ [
+ 17,
+ 8,
+ 47,
+ 61,
+ 50,
+ 44
+ ],
+ [
+ 27,
+ 38,
+ 32,
+ 14,
+ 61,
+ 39
+ ],
+ [
+ 22,
+ 43,
+ 32,
+ 57,
+ 39,
+ 34
+ ],
+ [
+ 16,
+ 44,
+ 37,
+ 23,
+ 61,
+ 27
+ ],
+ [
+ 45,
+ 40,
+ 55,
+ 32,
+ 31,
+ 3
+ ],
+ [
+ 32,
+ 28,
+ 41,
+ 15,
+ 1,
+ 52
+ ],
+ [
+ 21,
+ 22,
+ 31,
+ 10,
+ 4,
+ 40
+ ],
+ [
+ 29,
+ 35,
+ 62,
+ 60,
+ 41,
+ 1
+ ],
+ [
+ 39,
+ 58,
+ 1,
+ 63,
+ 3,
+ 35
+ ],
+ [
+ 10,
+ 62,
+ 31,
+ 45,
+ 27,
+ 7
+ ]
+ ],
+ [
+ [
+ 17,
+ 5,
+ 10,
+ 57,
+ 14,
+ 27
+ ],
+ [
+ 43,
+ 9,
+ 33,
+ 56,
+ 1,
+ 20
+ ],
+ [
+ 63,
+ 1,
+ 35,
+ 43,
+ 27,
+ 10
+ ],
+ [
+ 47,
+ 6,
+ 18,
+ 3,
+ 38,
+ 15
+ ],
+ [
+ 11,
+ 51,
+ 61,
+ 34,
+ 44,
+ 55
+ ],
+ [
+ 10,
+ 63,
+ 53,
+ 60,
+ 37,
+ 58
+ ],
+ [
+ 51,
+ 45,
+ 63,
+ 34,
+ 18,
+ 60
+ ],
+ [
+ 35,
+ 51,
+ 52,
+ 53,
+ 38,
+ 45
+ ],
+ [
+ 44,
+ 0,
+ 35,
+ 55,
+ 38,
+ 9
+ ],
+ [
+ 42,
+ 43,
+ 5,
+ 25,
+ 21,
+ 6
+ ],
+ [
+ 42,
+ 13,
+ 27,
+ 25,
+ 38,
+ 32
+ ],
+ [
+ 43,
+ 5,
+ 1,
+ 38,
+ 22,
+ 2
+ ],
+ [
+ 17,
+ 49,
+ 0,
+ 37,
+ 28,
+ 30
+ ],
+ [
+ 2,
+ 29,
+ 54,
+ 5,
+ 13,
+ 60
+ ],
+ [
+ 2,
+ 27,
+ 10,
+ 58,
+ 40,
+ 28
+ ],
+ [
+ 2,
+ 49,
+ 54,
+ 62,
+ 53,
+ 57
+ ],
+ [
+ 60,
+ 53,
+ 12,
+ 27,
+ 28,
+ 0
+ ],
+ [
+ 17,
+ 50,
+ 51,
+ 33,
+ 3,
+ 43
+ ],
+ [
+ 38,
+ 5,
+ 19,
+ 27,
+ 13,
+ 32
+ ],
+ [
+ 43,
+ 22,
+ 0,
+ 57,
+ 40,
+ 63
+ ],
+ [
+ 27,
+ 16,
+ 35,
+ 52,
+ 38,
+ 39
+ ],
+ [
+ 29,
+ 31,
+ 55,
+ 40,
+ 62,
+ 27
+ ],
+ [
+ 41,
+ 48,
+ 32,
+ 46,
+ 40,
+ 57
+ ],
+ [
+ 20,
+ 54,
+ 4,
+ 22,
+ 0,
+ 55
+ ],
+ [
+ 62,
+ 23,
+ 25,
+ 33,
+ 28,
+ 20
+ ],
+ [
+ 7,
+ 18,
+ 60,
+ 22,
+ 58,
+ 3
+ ],
+ [
+ 36,
+ 9,
+ 11,
+ 0,
+ 48,
+ 31
+ ]
+ ],
+ [
+ [
+ 11,
+ 59,
+ 52,
+ 28,
+ 6,
+ 45
+ ],
+ [
+ 7,
+ 23,
+ 15,
+ 17,
+ 55,
+ 61
+ ],
+ [
+ 28,
+ 0,
+ 29,
+ 46,
+ 58,
+ 14
+ ],
+ [
+ 62,
+ 49,
+ 39,
+ 18,
+ 6,
+ 13
+ ],
+ [
+ 36,
+ 61,
+ 7,
+ 40,
+ 35,
+ 33
+ ],
+ [
+ 8,
+ 16,
+ 29,
+ 43,
+ 57,
+ 37
+ ],
+ [
+ 30,
+ 19,
+ 21,
+ 57,
+ 42,
+ 14
+ ],
+ [
+ 35,
+ 11,
+ 51,
+ 53,
+ 36,
+ 33
+ ],
+ [
+ 48,
+ 35,
+ 4,
+ 50,
+ 11,
+ 40
+ ],
+ [
+ 5,
+ 3,
+ 21,
+ 47,
+ 43,
+ 38
+ ],
+ [
+ 21,
+ 18,
+ 2,
+ 6,
+ 3,
+ 10
+ ],
+ [
+ 18,
+ 43,
+ 5,
+ 45,
+ 22,
+ 4
+ ],
+ [
+ 33,
+ 28,
+ 37,
+ 41,
+ 49,
+ 10
+ ],
+ [
+ 56,
+ 55,
+ 22,
+ 54,
+ 62,
+ 15
+ ],
+ [
+ 2,
+ 27,
+ 57,
+ 5,
+ 63,
+ 19
+ ],
+ [
+ 55,
+ 8,
+ 54,
+ 2,
+ 59,
+ 48
+ ],
+ [
+ 0,
+ 53,
+ 51,
+ 3,
+ 50,
+ 52
+ ],
+ [
+ 21,
+ 12,
+ 62,
+ 60,
+ 18,
+ 43
+ ],
+ [
+ 38,
+ 13,
+ 53,
+ 27,
+ 14,
+ 28
+ ],
+ [
+ 22,
+ 56,
+ 44,
+ 43,
+ 51,
+ 53
+ ],
+ [
+ 29,
+ 43,
+ 12,
+ 16,
+ 41,
+ 52
+ ],
+ [
+ 51,
+ 1,
+ 35,
+ 48,
+ 44,
+ 34
+ ],
+ [
+ 32,
+ 25,
+ 4,
+ 41,
+ 53,
+ 23
+ ],
+ [
+ 48,
+ 17,
+ 25,
+ 1,
+ 60,
+ 62
+ ],
+ [
+ 22,
+ 29,
+ 5,
+ 18,
+ 53,
+ 20
+ ],
+ [
+ 28,
+ 55,
+ 15,
+ 1,
+ 8,
+ 49
+ ],
+ [
+ 30,
+ 57,
+ 6,
+ 7,
+ 31,
+ 50
+ ]
+ ],
+ [
+ [
+ 11,
+ 16,
+ 31,
+ 46,
+ 35,
+ 49
+ ],
+ [
+ 13,
+ 49,
+ 31,
+ 16,
+ 34,
+ 10
+ ],
+ [
+ 36,
+ 13,
+ 27,
+ 34,
+ 42,
+ 18
+ ],
+ [
+ 24,
+ 7,
+ 61,
+ 63,
+ 34,
+ 15
+ ],
+ [
+ 17,
+ 2,
+ 35,
+ 48,
+ 44,
+ 62
+ ],
+ [
+ 8,
+ 7,
+ 23,
+ 47,
+ 51,
+ 26
+ ],
+ [
+ 61,
+ 58,
+ 29,
+ 38,
+ 33,
+ 45
+ ],
+ [
+ 20,
+ 9,
+ 42,
+ 27,
+ 54,
+ 29
+ ],
+ [
+ 25,
+ 47,
+ 34,
+ 1,
+ 4,
+ 38
+ ],
+ [
+ 54,
+ 27,
+ 57,
+ 4,
+ 28,
+ 62
+ ],
+ [
+ 1,
+ 30,
+ 59,
+ 60,
+ 17,
+ 32
+ ],
+ [
+ 31,
+ 25,
+ 51,
+ 5,
+ 58,
+ 49
+ ],
+ [
+ 34,
+ 2,
+ 19,
+ 14,
+ 29,
+ 36
+ ],
+ [
+ 57,
+ 40,
+ 6,
+ 9,
+ 32,
+ 14
+ ],
+ [
+ 8,
+ 44,
+ 16,
+ 34,
+ 45,
+ 47
+ ],
+ [
+ 8,
+ 25,
+ 40,
+ 37,
+ 14,
+ 59
+ ],
+ [
+ 13,
+ 15,
+ 19,
+ 51,
+ 25,
+ 37
+ ],
+ [
+ 23,
+ 4,
+ 44,
+ 52,
+ 15,
+ 19
+ ],
+ [
+ 8,
+ 10,
+ 45,
+ 1,
+ 31,
+ 53
+ ],
+ [
+ 38,
+ 59,
+ 15,
+ 2,
+ 0,
+ 34
+ ],
+ [
+ 41,
+ 25,
+ 57,
+ 55,
+ 27,
+ 24
+ ],
+ [
+ 11,
+ 44,
+ 41,
+ 39,
+ 53,
+ 62
+ ],
+ [
+ 31,
+ 30,
+ 42,
+ 37,
+ 34,
+ 59
+ ],
+ [
+ 33,
+ 14,
+ 12,
+ 10,
+ 51,
+ 54
+ ],
+ [
+ 11,
+ 18,
+ 45,
+ 33,
+ 20,
+ 48
+ ],
+ [
+ 50,
+ 51,
+ 54,
+ 23,
+ 10,
+ 2
+ ],
+ [
+ 17,
+ 31,
+ 37,
+ 5,
+ 19,
+ 36
+ ]
+ ],
+ [
+ [
+ 31,
+ 35,
+ 51,
+ 18,
+ 53,
+ 61
+ ],
+ [
+ 7,
+ 40,
+ 39,
+ 41,
+ 31,
+ 37
+ ],
+ [
+ 16,
+ 29,
+ 26,
+ 50,
+ 33,
+ 10
+ ],
+ [
+ 14,
+ 22,
+ 37,
+ 17,
+ 6,
+ 25
+ ],
+ [
+ 60,
+ 20,
+ 46,
+ 4,
+ 3,
+ 57
+ ],
+ [
+ 11,
+ 2,
+ 49,
+ 21,
+ 27,
+ 9
+ ],
+ [
+ 42,
+ 30,
+ 54,
+ 58,
+ 19,
+ 56
+ ],
+ [
+ 46,
+ 33,
+ 38,
+ 41,
+ 35,
+ 49
+ ],
+ [
+ 50,
+ 48,
+ 57,
+ 37,
+ 38,
+ 10
+ ],
+ [
+ 59,
+ 63,
+ 47,
+ 28,
+ 10,
+ 35
+ ],
+ [
+ 28,
+ 3,
+ 2,
+ 52,
+ 33,
+ 43
+ ],
+ [
+ 58,
+ 19,
+ 17,
+ 16,
+ 57,
+ 63
+ ],
+ [
+ 42,
+ 4,
+ 3,
+ 23,
+ 45,
+ 57
+ ],
+ [
+ 18,
+ 42,
+ 51,
+ 3,
+ 20,
+ 15
+ ],
+ [
+ 32,
+ 43,
+ 12,
+ 3,
+ 0,
+ 13
+ ],
+ [
+ 15,
+ 45,
+ 4,
+ 7,
+ 53,
+ 21
+ ],
+ [
+ 28,
+ 24,
+ 14,
+ 31,
+ 46,
+ 60
+ ],
+ [
+ 26,
+ 30,
+ 0,
+ 5,
+ 54,
+ 36
+ ],
+ [
+ 63,
+ 37,
+ 41,
+ 51,
+ 11,
+ 4
+ ],
+ [
+ 48,
+ 9,
+ 33,
+ 42,
+ 54,
+ 0
+ ],
+ [
+ 22,
+ 11,
+ 60,
+ 39,
+ 1,
+ 49
+ ],
+ [
+ 17,
+ 4,
+ 34,
+ 2,
+ 53,
+ 27
+ ],
+ [
+ 17,
+ 6,
+ 40,
+ 58,
+ 42,
+ 39
+ ],
+ [
+ 60,
+ 54,
+ 35,
+ 39,
+ 0,
+ 32
+ ],
+ [
+ 44,
+ 15,
+ 37,
+ 13,
+ 8,
+ 25
+ ],
+ [
+ 41,
+ 59,
+ 25,
+ 45,
+ 13,
+ 6
+ ],
+ [
+ 23,
+ 58,
+ 13,
+ 19,
+ 29,
+ 62
+ ]
+ ],
+ [
+ [
+ 52,
+ 47,
+ 27,
+ 36,
+ 33,
+ 38
+ ],
+ [
+ 43,
+ 56,
+ 4,
+ 21,
+ 25,
+ 6
+ ],
+ [
+ 25,
+ 54,
+ 35,
+ 18,
+ 11,
+ 57
+ ],
+ [
+ 17,
+ 16,
+ 1,
+ 6,
+ 33,
+ 45
+ ],
+ [
+ 32,
+ 36,
+ 0,
+ 60,
+ 46,
+ 57
+ ],
+ [
+ 11,
+ 1,
+ 3,
+ 21,
+ 2,
+ 34
+ ],
+ [
+ 30,
+ 42,
+ 58,
+ 54,
+ 19,
+ 13
+ ],
+ [
+ 33,
+ 35,
+ 56,
+ 42,
+ 38,
+ 45
+ ],
+ [
+ 50,
+ 37,
+ 36,
+ 14,
+ 20,
+ 48
+ ],
+ [
+ 10,
+ 28,
+ 4,
+ 37,
+ 32,
+ 6
+ ],
+ [
+ 3,
+ 18,
+ 28,
+ 61,
+ 2,
+ 44
+ ],
+ [
+ 16,
+ 56,
+ 19,
+ 27,
+ 43,
+ 46
+ ],
+ [
+ 42,
+ 33,
+ 57,
+ 3,
+ 58,
+ 26
+ ],
+ [
+ 56,
+ 15,
+ 51,
+ 50,
+ 55,
+ 13
+ ],
+ [
+ 12,
+ 43,
+ 40,
+ 13,
+ 16,
+ 29
+ ],
+ [
+ 15,
+ 7,
+ 28,
+ 53,
+ 5,
+ 20
+ ],
+ [
+ 14,
+ 58,
+ 24,
+ 60,
+ 31,
+ 51
+ ],
+ [
+ 0,
+ 60,
+ 3,
+ 24,
+ 19,
+ 44
+ ],
+ [
+ 36,
+ 28,
+ 11,
+ 53,
+ 63,
+ 15
+ ],
+ [
+ 2,
+ 26,
+ 9,
+ 34,
+ 0,
+ 3
+ ],
+ [
+ 49,
+ 28,
+ 43,
+ 41,
+ 30,
+ 11
+ ],
+ [
+ 25,
+ 51,
+ 12,
+ 6,
+ 61,
+ 16
+ ],
+ [
+ 39,
+ 17,
+ 41,
+ 50,
+ 40,
+ 21
+ ],
+ [
+ 25,
+ 58,
+ 48,
+ 12,
+ 60,
+ 33
+ ],
+ [
+ 31,
+ 49,
+ 5,
+ 52,
+ 63,
+ 3
+ ],
+ [
+ 15,
+ 55,
+ 38,
+ 47,
+ 1,
+ 49
+ ],
+ [
+ 23,
+ 6,
+ 32,
+ 19,
+ 62,
+ 7
+ ]
+ ],
+ [
+ [
+ 44,
+ 24,
+ 56,
+ 33,
+ 15,
+ 7
+ ],
+ [
+ 38,
+ 26,
+ 24,
+ 29,
+ 53,
+ 19
+ ],
+ [
+ 12,
+ 15,
+ 29,
+ 9,
+ 1,
+ 63
+ ],
+ [
+ 38,
+ 61,
+ 58,
+ 50,
+ 45,
+ 6
+ ],
+ [
+ 24,
+ 34,
+ 4,
+ 36,
+ 57,
+ 31
+ ],
+ [
+ 1,
+ 22,
+ 43,
+ 21,
+ 10,
+ 7
+ ],
+ [
+ 20,
+ 19,
+ 54,
+ 58,
+ 18,
+ 42
+ ],
+ [
+ 56,
+ 33,
+ 14,
+ 21,
+ 51,
+ 18
+ ],
+ [
+ 60,
+ 50,
+ 14,
+ 36,
+ 4,
+ 43
+ ],
+ [
+ 6,
+ 10,
+ 9,
+ 63,
+ 4,
+ 38
+ ],
+ [
+ 54,
+ 39,
+ 46,
+ 18,
+ 3,
+ 2
+ ],
+ [
+ 30,
+ 16,
+ 37,
+ 53,
+ 56,
+ 43
+ ],
+ [
+ 56,
+ 10,
+ 42,
+ 58,
+ 57,
+ 15
+ ],
+ [
+ 30,
+ 56,
+ 13,
+ 50,
+ 51,
+ 0
+ ],
+ [
+ 55,
+ 40,
+ 12,
+ 62,
+ 13,
+ 30
+ ],
+ [
+ 53,
+ 28,
+ 11,
+ 61,
+ 7,
+ 19
+ ],
+ [
+ 55,
+ 14,
+ 17,
+ 47,
+ 30,
+ 5
+ ],
+ [
+ 18,
+ 31,
+ 20,
+ 60,
+ 57,
+ 32
+ ],
+ [
+ 43,
+ 12,
+ 53,
+ 26,
+ 32,
+ 61
+ ],
+ [
+ 25,
+ 16,
+ 26,
+ 61,
+ 53,
+ 3
+ ],
+ [
+ 49,
+ 28,
+ 51,
+ 59,
+ 55,
+ 11
+ ],
+ [
+ 56,
+ 6,
+ 32,
+ 14,
+ 10,
+ 21
+ ],
+ [
+ 2,
+ 15,
+ 58,
+ 17,
+ 13,
+ 62
+ ],
+ [
+ 53,
+ 51,
+ 4,
+ 29,
+ 50,
+ 25
+ ],
+ [
+ 61,
+ 31,
+ 19,
+ 15,
+ 60,
+ 49
+ ],
+ [
+ 19,
+ 46,
+ 44,
+ 36,
+ 8,
+ 40
+ ],
+ [
+ 8,
+ 29,
+ 46,
+ 7,
+ 53,
+ 20
+ ]
+ ],
+ [
+ [
+ 48,
+ 63,
+ 38,
+ 42,
+ 50,
+ 62
+ ],
+ [
+ 3,
+ 2,
+ 33,
+ 40,
+ 14,
+ 60
+ ],
+ [
+ 39,
+ 7,
+ 45,
+ 40,
+ 6,
+ 44
+ ],
+ [
+ 41,
+ 5,
+ 20,
+ 56,
+ 13,
+ 0
+ ],
+ [
+ 6,
+ 37,
+ 30,
+ 1,
+ 38,
+ 54
+ ],
+ [
+ 59,
+ 46,
+ 4,
+ 22,
+ 5,
+ 6
+ ],
+ [
+ 20,
+ 1,
+ 44,
+ 35,
+ 13,
+ 3
+ ],
+ [
+ 12,
+ 56,
+ 8,
+ 50,
+ 31,
+ 2
+ ],
+ [
+ 33,
+ 60,
+ 41,
+ 43,
+ 37,
+ 36
+ ],
+ [
+ 9,
+ 10,
+ 0,
+ 55,
+ 40,
+ 37
+ ],
+ [
+ 54,
+ 39,
+ 9,
+ 8,
+ 61,
+ 46
+ ],
+ [
+ 30,
+ 56,
+ 53,
+ 24,
+ 16,
+ 11
+ ],
+ [
+ 7,
+ 58,
+ 57,
+ 42,
+ 10,
+ 52
+ ],
+ [
+ 27,
+ 30,
+ 42,
+ 25,
+ 59,
+ 13
+ ],
+ [
+ 9,
+ 11,
+ 55,
+ 49,
+ 0,
+ 46
+ ],
+ [
+ 34,
+ 53,
+ 54,
+ 29,
+ 57,
+ 37
+ ],
+ [
+ 55,
+ 17,
+ 5,
+ 60,
+ 18,
+ 31
+ ],
+ [
+ 57,
+ 48,
+ 43,
+ 39,
+ 32,
+ 36
+ ],
+ [
+ 12,
+ 43,
+ 42,
+ 49,
+ 7,
+ 61
+ ],
+ [
+ 23,
+ 36,
+ 1,
+ 0,
+ 16,
+ 61
+ ],
+ [
+ 21,
+ 35,
+ 11,
+ 31,
+ 55,
+ 7
+ ],
+ [
+ 60,
+ 8,
+ 42,
+ 24,
+ 14,
+ 6
+ ],
+ [
+ 15,
+ 51,
+ 58,
+ 2,
+ 33,
+ 30
+ ],
+ [
+ 52,
+ 51,
+ 4,
+ 28,
+ 21,
+ 19
+ ],
+ [
+ 19,
+ 60,
+ 15,
+ 34,
+ 54,
+ 46
+ ],
+ [
+ 19,
+ 61,
+ 58,
+ 12,
+ 40,
+ 3
+ ],
+ [
+ 35,
+ 49,
+ 54,
+ 53,
+ 1,
+ 25
+ ]
+ ],
+ [
+ [
+ 47,
+ 37,
+ 59,
+ 38,
+ 33,
+ 10
+ ],
+ [
+ 36,
+ 7,
+ 21,
+ 51,
+ 8,
+ 47
+ ],
+ [
+ 20,
+ 32,
+ 44,
+ 47,
+ 4,
+ 54
+ ],
+ [
+ 43,
+ 20,
+ 42,
+ 52,
+ 8,
+ 19
+ ],
+ [
+ 27,
+ 9,
+ 39,
+ 57,
+ 12,
+ 54
+ ],
+ [
+ 28,
+ 16,
+ 29,
+ 11,
+ 61,
+ 58
+ ],
+ [
+ 42,
+ 2,
+ 30,
+ 43,
+ 28,
+ 25
+ ],
+ [
+ 36,
+ 18,
+ 32,
+ 21,
+ 53,
+ 15
+ ],
+ [
+ 50,
+ 12,
+ 37,
+ 48,
+ 14,
+ 52
+ ],
+ [
+ 9,
+ 10,
+ 13,
+ 3,
+ 58,
+ 26
+ ],
+ [
+ 18,
+ 3,
+ 9,
+ 55,
+ 6,
+ 61
+ ],
+ [
+ 30,
+ 18,
+ 45,
+ 60,
+ 16,
+ 24
+ ],
+ [
+ 35,
+ 5,
+ 42,
+ 26,
+ 37,
+ 58
+ ],
+ [
+ 55,
+ 38,
+ 15,
+ 13,
+ 14,
+ 54
+ ],
+ [
+ 9,
+ 62,
+ 2,
+ 17,
+ 13,
+ 26
+ ],
+ [
+ 9,
+ 40,
+ 37,
+ 54,
+ 17,
+ 47
+ ],
+ [
+ 22,
+ 60,
+ 51,
+ 40,
+ 14,
+ 39
+ ],
+ [
+ 17,
+ 3,
+ 21,
+ 30,
+ 36,
+ 25
+ ],
+ [
+ 56,
+ 41,
+ 24,
+ 9,
+ 43,
+ 14
+ ],
+ [
+ 53,
+ 34,
+ 26,
+ 22,
+ 12,
+ 35
+ ],
+ [
+ 36,
+ 42,
+ 24,
+ 37,
+ 4,
+ 8
+ ],
+ [
+ 34,
+ 16,
+ 8,
+ 46,
+ 56,
+ 17
+ ],
+ [
+ 31,
+ 10,
+ 16,
+ 41,
+ 56,
+ 32
+ ],
+ [
+ 48,
+ 0,
+ 42,
+ 56,
+ 30,
+ 31
+ ],
+ [
+ 6,
+ 54,
+ 26,
+ 19,
+ 7,
+ 8
+ ],
+ [
+ 49,
+ 22,
+ 6,
+ 13,
+ 59,
+ 24
+ ],
+ [
+ 45,
+ 62,
+ 27,
+ 47,
+ 50,
+ 7
+ ]
+ ],
+ [
+ [
+ 45,
+ 37,
+ 48,
+ 29,
+ 30,
+ 3
+ ],
+ [
+ 8,
+ 60,
+ 59,
+ 43,
+ 10,
+ 48
+ ],
+ [
+ 51,
+ 45,
+ 28,
+ 34,
+ 59,
+ 63
+ ],
+ [
+ 43,
+ 2,
+ 38,
+ 12,
+ 20,
+ 4
+ ],
+ [
+ 50,
+ 57,
+ 39,
+ 31,
+ 63,
+ 9
+ ],
+ [
+ 58,
+ 53,
+ 18,
+ 9,
+ 30,
+ 21
+ ],
+ [
+ 63,
+ 51,
+ 34,
+ 7,
+ 20,
+ 27
+ ],
+ [
+ 32,
+ 21,
+ 46,
+ 47,
+ 25,
+ 18
+ ],
+ [
+ 6,
+ 12,
+ 14,
+ 50,
+ 33,
+ 37
+ ],
+ [
+ 30,
+ 10,
+ 9,
+ 6,
+ 13,
+ 29
+ ],
+ [
+ 61,
+ 18,
+ 23,
+ 20,
+ 44,
+ 6
+ ],
+ [
+ 16,
+ 12,
+ 30,
+ 60,
+ 0,
+ 62
+ ],
+ [
+ 5,
+ 26,
+ 57,
+ 35,
+ 37,
+ 61
+ ],
+ [
+ 46,
+ 55,
+ 13,
+ 1,
+ 17,
+ 31
+ ],
+ [
+ 6,
+ 10,
+ 22,
+ 2,
+ 16,
+ 15
+ ],
+ [
+ 6,
+ 54,
+ 19,
+ 25,
+ 57,
+ 46
+ ],
+ [
+ 2,
+ 30,
+ 60,
+ 61,
+ 18,
+ 49
+ ],
+ [
+ 17,
+ 4,
+ 2,
+ 27,
+ 3,
+ 44
+ ],
+ [
+ 56,
+ 46,
+ 8,
+ 32,
+ 6,
+ 14
+ ],
+ [
+ 5,
+ 13,
+ 0,
+ 34,
+ 14,
+ 30
+ ],
+ [
+ 15,
+ 23,
+ 59,
+ 57,
+ 27,
+ 53
+ ],
+ [
+ 36,
+ 61,
+ 8,
+ 43,
+ 57,
+ 37
+ ],
+ [
+ 10,
+ 15,
+ 52,
+ 31,
+ 29,
+ 23
+ ],
+ [
+ 10,
+ 38,
+ 1,
+ 4,
+ 31,
+ 57
+ ],
+ [
+ 8,
+ 32,
+ 54,
+ 33,
+ 3,
+ 50
+ ],
+ [
+ 6,
+ 33,
+ 19,
+ 50,
+ 2,
+ 3
+ ],
+ [
+ 55,
+ 43,
+ 5,
+ 4,
+ 25,
+ 8
+ ]
+ ],
+ [
+ [
+ 25,
+ 14,
+ 18,
+ 49,
+ 51,
+ 63
+ ],
+ [
+ 42,
+ 21,
+ 30,
+ 43,
+ 24,
+ 7
+ ],
+ [
+ 54,
+ 39,
+ 9,
+ 59,
+ 28,
+ 49
+ ],
+ [
+ 23,
+ 1,
+ 55,
+ 45,
+ 43,
+ 40
+ ],
+ [
+ 37,
+ 30,
+ 41,
+ 59,
+ 21,
+ 44
+ ],
+ [
+ 4,
+ 41,
+ 31,
+ 35,
+ 19,
+ 14
+ ],
+ [
+ 28,
+ 55,
+ 44,
+ 63,
+ 9,
+ 51
+ ],
+ [
+ 63,
+ 12,
+ 32,
+ 13,
+ 47,
+ 28
+ ],
+ [
+ 33,
+ 12,
+ 59,
+ 35,
+ 6,
+ 39
+ ],
+ [
+ 30,
+ 40,
+ 10,
+ 29,
+ 52,
+ 13
+ ],
+ [
+ 63,
+ 19,
+ 23,
+ 61,
+ 8,
+ 55
+ ],
+ [
+ 45,
+ 62,
+ 22,
+ 12,
+ 38,
+ 42
+ ],
+ [
+ 58,
+ 26,
+ 57,
+ 20,
+ 45,
+ 30
+ ],
+ [
+ 8,
+ 59,
+ 47,
+ 17,
+ 25,
+ 22
+ ],
+ [
+ 24,
+ 54,
+ 51,
+ 55,
+ 10,
+ 15
+ ],
+ [
+ 46,
+ 22,
+ 54,
+ 44,
+ 57,
+ 40
+ ],
+ [
+ 34,
+ 60,
+ 2,
+ 17,
+ 27,
+ 18
+ ],
+ [
+ 51,
+ 6,
+ 61,
+ 2,
+ 39,
+ 32
+ ],
+ [
+ 5,
+ 49,
+ 38,
+ 8,
+ 32,
+ 46
+ ],
+ [
+ 57,
+ 4,
+ 23,
+ 14,
+ 59,
+ 5
+ ],
+ [
+ 7,
+ 36,
+ 45,
+ 21,
+ 53,
+ 4
+ ],
+ [
+ 60,
+ 3,
+ 39,
+ 8,
+ 14,
+ 35
+ ],
+ [
+ 33,
+ 57,
+ 60,
+ 47,
+ 15,
+ 52
+ ],
+ [
+ 20,
+ 11,
+ 22,
+ 58,
+ 19,
+ 4
+ ],
+ [
+ 36,
+ 34,
+ 47,
+ 41,
+ 60,
+ 40
+ ],
+ [
+ 33,
+ 3,
+ 49,
+ 29,
+ 14,
+ 59
+ ],
+ [
+ 52,
+ 60,
+ 24,
+ 25,
+ 35,
+ 34
+ ]
+ ],
+ [
+ [
+ 23,
+ 54,
+ 53,
+ 11,
+ 58,
+ 8
+ ],
+ [
+ 11,
+ 30,
+ 15,
+ 63,
+ 59,
+ 16
+ ],
+ [
+ 20,
+ 29,
+ 58,
+ 17,
+ 52,
+ 30
+ ],
+ [
+ 18,
+ 1,
+ 43,
+ 15,
+ 3,
+ 8
+ ],
+ [
+ 59,
+ 55,
+ 13,
+ 20,
+ 44,
+ 40
+ ],
+ [
+ 33,
+ 45,
+ 27,
+ 53,
+ 52,
+ 63
+ ],
+ [
+ 28,
+ 57,
+ 51,
+ 34,
+ 53,
+ 31
+ ],
+ [
+ 13,
+ 63,
+ 3,
+ 32,
+ 44,
+ 45
+ ],
+ [
+ 59,
+ 33,
+ 12,
+ 57,
+ 6,
+ 35
+ ],
+ [
+ 23,
+ 30,
+ 42,
+ 10,
+ 29,
+ 13
+ ],
+ [
+ 63,
+ 23,
+ 18,
+ 42,
+ 38,
+ 19
+ ],
+ [
+ 62,
+ 2,
+ 22,
+ 19,
+ 45,
+ 6
+ ],
+ [
+ 6,
+ 26,
+ 20,
+ 30,
+ 5,
+ 25
+ ],
+ [
+ 3,
+ 6,
+ 8,
+ 31,
+ 17,
+ 37
+ ],
+ [
+ 59,
+ 2,
+ 10,
+ 54,
+ 55,
+ 19
+ ],
+ [
+ 45,
+ 7,
+ 57,
+ 42,
+ 54,
+ 46
+ ],
+ [
+ 60,
+ 55,
+ 53,
+ 34,
+ 41,
+ 18
+ ],
+ [
+ 36,
+ 2,
+ 8,
+ 31,
+ 32,
+ 46
+ ],
+ [
+ 36,
+ 45,
+ 38,
+ 32,
+ 61,
+ 51
+ ],
+ [
+ 57,
+ 39,
+ 6,
+ 18,
+ 19,
+ 30
+ ],
+ [
+ 37,
+ 39,
+ 59,
+ 48,
+ 53,
+ 14
+ ],
+ [
+ 52,
+ 28,
+ 46,
+ 55,
+ 47,
+ 61
+ ],
+ [
+ 1,
+ 28,
+ 52,
+ 15,
+ 31,
+ 18
+ ],
+ [
+ 22,
+ 21,
+ 8,
+ 10,
+ 46,
+ 39
+ ],
+ [
+ 45,
+ 41,
+ 35,
+ 9,
+ 54,
+ 33
+ ],
+ [
+ 42,
+ 39,
+ 58,
+ 61,
+ 24,
+ 3
+ ],
+ [
+ 41,
+ 46,
+ 10,
+ 3,
+ 15,
+ 33
+ ]
+ ],
+ [
+ [
+ 48,
+ 42,
+ 38,
+ 63,
+ 47,
+ 7
+ ],
+ [
+ 3,
+ 10,
+ 26,
+ 62,
+ 2,
+ 6
+ ],
+ [
+ 39,
+ 44,
+ 7,
+ 6,
+ 45,
+ 40
+ ],
+ [
+ 21,
+ 39,
+ 1,
+ 18,
+ 15,
+ 41
+ ],
+ [
+ 6,
+ 33,
+ 9,
+ 13,
+ 3,
+ 27
+ ],
+ [
+ 59,
+ 33,
+ 53,
+ 62,
+ 21,
+ 45
+ ],
+ [
+ 28,
+ 10,
+ 33,
+ 7,
+ 57,
+ 50
+ ],
+ [
+ 50,
+ 13,
+ 12,
+ 49,
+ 3,
+ 55
+ ],
+ [
+ 33,
+ 59,
+ 26,
+ 35,
+ 48,
+ 38
+ ],
+ [
+ 23,
+ 30,
+ 20,
+ 43,
+ 10,
+ 58
+ ],
+ [
+ 63,
+ 23,
+ 18,
+ 17,
+ 38,
+ 9
+ ],
+ [
+ 22,
+ 62,
+ 36,
+ 33,
+ 6,
+ 2
+ ],
+ [
+ 12,
+ 17,
+ 59,
+ 7,
+ 26,
+ 49
+ ],
+ [
+ 47,
+ 3,
+ 27,
+ 6,
+ 24,
+ 22
+ ],
+ [
+ 59,
+ 57,
+ 46,
+ 54,
+ 2,
+ 10
+ ],
+ [
+ 27,
+ 45,
+ 57,
+ 54,
+ 34,
+ 61
+ ],
+ [
+ 34,
+ 60,
+ 55,
+ 31,
+ 58,
+ 43
+ ],
+ [
+ 48,
+ 42,
+ 32,
+ 39,
+ 2,
+ 38
+ ],
+ [
+ 42,
+ 49,
+ 32,
+ 44,
+ 12,
+ 61
+ ],
+ [
+ 57,
+ 36,
+ 39,
+ 1,
+ 30,
+ 59
+ ],
+ [
+ 48,
+ 7,
+ 21,
+ 53,
+ 17,
+ 29
+ ],
+ [
+ 8,
+ 60,
+ 58,
+ 35,
+ 14,
+ 46
+ ],
+ [
+ 51,
+ 15,
+ 28,
+ 47,
+ 33,
+ 30
+ ],
+ [
+ 11,
+ 4,
+ 19,
+ 8,
+ 58,
+ 52
+ ],
+ [
+ 24,
+ 5,
+ 41,
+ 60,
+ 40,
+ 54
+ ],
+ [
+ 61,
+ 3,
+ 21,
+ 58,
+ 19,
+ 8
+ ],
+ [
+ 60,
+ 35,
+ 54,
+ 49,
+ 1,
+ 0
+ ]
+ ],
+ [
+ [
+ 6,
+ 24,
+ 63,
+ 25,
+ 26,
+ 45
+ ],
+ [
+ 47,
+ 13,
+ 49,
+ 44,
+ 20,
+ 19
+ ],
+ [
+ 23,
+ 32,
+ 49,
+ 20,
+ 24,
+ 2
+ ],
+ [
+ 43,
+ 21,
+ 8,
+ 40,
+ 39,
+ 45
+ ],
+ [
+ 39,
+ 29,
+ 3,
+ 5,
+ 41,
+ 12
+ ],
+ [
+ 61,
+ 33,
+ 48,
+ 40,
+ 29,
+ 62
+ ],
+ [
+ 29,
+ 28,
+ 25,
+ 33,
+ 44,
+ 31
+ ],
+ [
+ 50,
+ 33,
+ 13,
+ 11,
+ 30,
+ 54
+ ],
+ [
+ 48,
+ 26,
+ 35,
+ 17,
+ 55,
+ 6
+ ],
+ [
+ 43,
+ 23,
+ 12,
+ 9,
+ 25,
+ 62
+ ],
+ [
+ 63,
+ 35,
+ 18,
+ 53,
+ 38,
+ 17
+ ],
+ [
+ 19,
+ 48,
+ 5,
+ 36,
+ 59,
+ 17
+ ],
+ [
+ 7,
+ 26,
+ 59,
+ 17,
+ 12,
+ 30
+ ],
+ [
+ 47,
+ 22,
+ 24,
+ 62,
+ 25,
+ 42
+ ],
+ [
+ 59,
+ 46,
+ 35,
+ 39,
+ 57,
+ 54
+ ],
+ [
+ 27,
+ 9,
+ 20,
+ 0,
+ 57,
+ 52
+ ],
+ [
+ 38,
+ 58,
+ 60,
+ 34,
+ 43,
+ 29
+ ],
+ [
+ 42,
+ 32,
+ 39,
+ 54,
+ 38,
+ 57
+ ],
+ [
+ 59,
+ 32,
+ 42,
+ 6,
+ 37,
+ 18
+ ],
+ [
+ 24,
+ 36,
+ 57,
+ 4,
+ 30,
+ 60
+ ],
+ [
+ 9,
+ 48,
+ 7,
+ 53,
+ 21,
+ 58
+ ],
+ [
+ 49,
+ 31,
+ 14,
+ 8,
+ 19,
+ 22
+ ],
+ [
+ 15,
+ 33,
+ 44,
+ 8,
+ 14,
+ 3
+ ],
+ [
+ 11,
+ 4,
+ 21,
+ 28,
+ 41,
+ 23
+ ],
+ [
+ 37,
+ 27,
+ 24,
+ 12,
+ 9,
+ 20
+ ],
+ [
+ 58,
+ 9,
+ 19,
+ 3,
+ 12,
+ 48
+ ],
+ [
+ 24,
+ 14,
+ 60,
+ 47,
+ 25,
+ 13
+ ]
+ ],
+ [
+ [
+ 26,
+ 62,
+ 58,
+ 18,
+ 38,
+ 5
+ ],
+ [
+ 19,
+ 12,
+ 40,
+ 39,
+ 31,
+ 57
+ ],
+ [
+ 27,
+ 38,
+ 9,
+ 22,
+ 23,
+ 61
+ ],
+ [
+ 42,
+ 20,
+ 63,
+ 39,
+ 45,
+ 43
+ ],
+ [
+ 7,
+ 36,
+ 60,
+ 29,
+ 57,
+ 24
+ ],
+ [
+ 61,
+ 34,
+ 3,
+ 1,
+ 44,
+ 51
+ ],
+ [
+ 29,
+ 42,
+ 25,
+ 47,
+ 30,
+ 17
+ ],
+ [
+ 33,
+ 38,
+ 39,
+ 50,
+ 62,
+ 0
+ ],
+ [
+ 17,
+ 26,
+ 13,
+ 50,
+ 41,
+ 11
+ ],
+ [
+ 12,
+ 43,
+ 35,
+ 16,
+ 55,
+ 60
+ ],
+ [
+ 3,
+ 18,
+ 39,
+ 63,
+ 35,
+ 62
+ ],
+ [
+ 19,
+ 48,
+ 39,
+ 60,
+ 54,
+ 2
+ ],
+ [
+ 7,
+ 42,
+ 56,
+ 59,
+ 37,
+ 24
+ ],
+ [
+ 56,
+ 47,
+ 54,
+ 59,
+ 3,
+ 6
+ ],
+ [
+ 35,
+ 39,
+ 43,
+ 59,
+ 2,
+ 13
+ ],
+ [
+ 20,
+ 27,
+ 9,
+ 56,
+ 0,
+ 54
+ ],
+ [
+ 17,
+ 60,
+ 29,
+ 18,
+ 58,
+ 5
+ ],
+ [
+ 57,
+ 42,
+ 45,
+ 0,
+ 24,
+ 29
+ ],
+ [
+ 49,
+ 28,
+ 52,
+ 32,
+ 50,
+ 3
+ ],
+ [
+ 24,
+ 2,
+ 36,
+ 30,
+ 59,
+ 46
+ ],
+ [
+ 11,
+ 7,
+ 48,
+ 54,
+ 53,
+ 21
+ ],
+ [
+ 33,
+ 14,
+ 42,
+ 6,
+ 8,
+ 5
+ ],
+ [
+ 15,
+ 36,
+ 33,
+ 17,
+ 51,
+ 13
+ ],
+ [
+ 41,
+ 11,
+ 2,
+ 4,
+ 8,
+ 29
+ ],
+ [
+ 30,
+ 15,
+ 60,
+ 46,
+ 5,
+ 9
+ ],
+ [
+ 21,
+ 3,
+ 14,
+ 19,
+ 61,
+ 58
+ ],
+ [
+ 24,
+ 60,
+ 1,
+ 14,
+ 35,
+ 53
+ ]
+ ],
+ [
+ [
+ 8,
+ 56,
+ 54,
+ 37,
+ 4,
+ 38
+ ],
+ [
+ 36,
+ 9,
+ 24,
+ 8,
+ 1,
+ 2
+ ],
+ [
+ 16,
+ 57,
+ 29,
+ 32,
+ 58,
+ 30
+ ],
+ [
+ 32,
+ 63,
+ 42,
+ 52,
+ 4,
+ 20
+ ],
+ [
+ 27,
+ 24,
+ 44,
+ 39,
+ 63,
+ 20
+ ],
+ [
+ 24,
+ 11,
+ 28,
+ 6,
+ 15,
+ 31
+ ],
+ [
+ 30,
+ 41,
+ 42,
+ 25,
+ 52,
+ 2
+ ],
+ [
+ 26,
+ 36,
+ 62,
+ 18,
+ 50,
+ 21
+ ],
+ [
+ 48,
+ 50,
+ 32,
+ 51,
+ 27,
+ 26
+ ],
+ [
+ 13,
+ 43,
+ 3,
+ 26,
+ 12,
+ 41
+ ],
+ [
+ 3,
+ 32,
+ 18,
+ 53,
+ 39,
+ 17
+ ],
+ [
+ 21,
+ 19,
+ 9,
+ 36,
+ 48,
+ 1
+ ],
+ [
+ 35,
+ 59,
+ 54,
+ 42,
+ 63,
+ 17
+ ],
+ [
+ 15,
+ 12,
+ 38,
+ 9,
+ 51,
+ 54
+ ],
+ [
+ 3,
+ 2,
+ 48,
+ 59,
+ 57,
+ 13
+ ],
+ [
+ 27,
+ 9,
+ 4,
+ 20,
+ 22,
+ 3
+ ],
+ [
+ 60,
+ 40,
+ 22,
+ 27,
+ 46,
+ 32
+ ],
+ [
+ 30,
+ 21,
+ 25,
+ 17,
+ 0,
+ 42
+ ],
+ [
+ 33,
+ 27,
+ 34,
+ 28,
+ 38,
+ 44
+ ],
+ [
+ 53,
+ 32,
+ 33,
+ 35,
+ 31,
+ 56
+ ],
+ [
+ 40,
+ 11,
+ 20,
+ 47,
+ 48,
+ 24
+ ],
+ [
+ 45,
+ 62,
+ 33,
+ 53,
+ 17,
+ 34
+ ],
+ [
+ 41,
+ 9,
+ 39,
+ 25,
+ 17,
+ 32
+ ],
+ [
+ 44,
+ 30,
+ 2,
+ 31,
+ 0,
+ 47
+ ],
+ [
+ 6,
+ 62,
+ 37,
+ 52,
+ 55,
+ 33
+ ],
+ [
+ 10,
+ 24,
+ 57,
+ 9,
+ 49,
+ 13
+ ],
+ [
+ 45,
+ 24,
+ 27,
+ 47,
+ 19,
+ 26
+ ]
+ ],
+ [
+ [
+ 4,
+ 16,
+ 22,
+ 23,
+ 3,
+ 13
+ ],
+ [
+ 16,
+ 23,
+ 9,
+ 59,
+ 13,
+ 50
+ ],
+ [
+ 11,
+ 35,
+ 21,
+ 7,
+ 9,
+ 59
+ ],
+ [
+ 44,
+ 1,
+ 25,
+ 26,
+ 15,
+ 20
+ ],
+ [
+ 35,
+ 57,
+ 52,
+ 31,
+ 24,
+ 5
+ ],
+ [
+ 35,
+ 15,
+ 38,
+ 53,
+ 12,
+ 0
+ ],
+ [
+ 20,
+ 36,
+ 63,
+ 28,
+ 60,
+ 33
+ ],
+ [
+ 49,
+ 50,
+ 12,
+ 13,
+ 8,
+ 16
+ ],
+ [
+ 42,
+ 48,
+ 6,
+ 16,
+ 35,
+ 41
+ ],
+ [
+ 23,
+ 9,
+ 54,
+ 34,
+ 30,
+ 13
+ ],
+ [
+ 23,
+ 63,
+ 51,
+ 53,
+ 55,
+ 3
+ ],
+ [
+ 33,
+ 36,
+ 62,
+ 19,
+ 59,
+ 57
+ ],
+ [
+ 59,
+ 38,
+ 12,
+ 17,
+ 32,
+ 53
+ ],
+ [
+ 48,
+ 0,
+ 19,
+ 24,
+ 61,
+ 22
+ ],
+ [
+ 55,
+ 6,
+ 39,
+ 7,
+ 60,
+ 18
+ ],
+ [
+ 27,
+ 57,
+ 19,
+ 1,
+ 60,
+ 47
+ ],
+ [
+ 30,
+ 40,
+ 10,
+ 17,
+ 36,
+ 60
+ ],
+ [
+ 13,
+ 20,
+ 1,
+ 31,
+ 17,
+ 54
+ ],
+ [
+ 50,
+ 32,
+ 23,
+ 7,
+ 33,
+ 30
+ ],
+ [
+ 14,
+ 19,
+ 23,
+ 13,
+ 10,
+ 7
+ ],
+ [
+ 53,
+ 23,
+ 19,
+ 13,
+ 5,
+ 7
+ ],
+ [
+ 30,
+ 18,
+ 46,
+ 14,
+ 1,
+ 0
+ ],
+ [
+ 15,
+ 18,
+ 52,
+ 0,
+ 51,
+ 30
+ ],
+ [
+ 4,
+ 21,
+ 53,
+ 41,
+ 10,
+ 14
+ ],
+ [
+ 4,
+ 9,
+ 39,
+ 8,
+ 16,
+ 54
+ ],
+ [
+ 3,
+ 19,
+ 16,
+ 30,
+ 47,
+ 27
+ ],
+ [
+ 33,
+ 8,
+ 46,
+ 40,
+ 29,
+ 14
+ ]
+ ],
+ [
+ [
+ 6,
+ 26,
+ 3,
+ 24,
+ 11,
+ 38
+ ],
+ [
+ 49,
+ 19,
+ 16,
+ 57,
+ 0,
+ 18
+ ],
+ [
+ 16,
+ 60,
+ 57,
+ 0,
+ 22,
+ 30
+ ],
+ [
+ 35,
+ 46,
+ 49,
+ 44,
+ 26,
+ 17
+ ],
+ [
+ 25,
+ 54,
+ 43,
+ 9,
+ 45,
+ 27
+ ],
+ [
+ 55,
+ 33,
+ 30,
+ 6,
+ 28,
+ 57
+ ],
+ [
+ 55,
+ 31,
+ 49,
+ 52,
+ 25,
+ 15
+ ],
+ [
+ 11,
+ 36,
+ 15,
+ 29,
+ 30,
+ 62
+ ],
+ [
+ 55,
+ 42,
+ 48,
+ 54,
+ 46,
+ 6
+ ],
+ [
+ 10,
+ 3,
+ 43,
+ 21,
+ 62,
+ 54
+ ],
+ [
+ 32,
+ 56,
+ 6,
+ 3,
+ 18,
+ 55
+ ],
+ [
+ 50,
+ 21,
+ 36,
+ 19,
+ 4,
+ 42
+ ],
+ [
+ 38,
+ 35,
+ 53,
+ 39,
+ 41,
+ 32
+ ],
+ [
+ 28,
+ 24,
+ 38,
+ 41,
+ 15,
+ 12
+ ],
+ [
+ 2,
+ 40,
+ 18,
+ 60,
+ 55,
+ 13
+ ],
+ [
+ 27,
+ 22,
+ 57,
+ 8,
+ 54,
+ 37
+ ],
+ [
+ 40,
+ 27,
+ 60,
+ 46,
+ 44,
+ 50
+ ],
+ [
+ 17,
+ 21,
+ 30,
+ 12,
+ 29,
+ 26
+ ],
+ [
+ 33,
+ 38,
+ 23,
+ 2,
+ 13,
+ 27
+ ],
+ [
+ 32,
+ 56,
+ 9,
+ 44,
+ 31,
+ 60
+ ],
+ [
+ 53,
+ 12,
+ 22,
+ 40,
+ 41,
+ 5
+ ],
+ [
+ 44,
+ 45,
+ 49,
+ 17,
+ 14,
+ 56
+ ],
+ [
+ 25,
+ 39,
+ 41,
+ 4,
+ 9,
+ 53
+ ],
+ [
+ 44,
+ 1,
+ 17,
+ 0,
+ 13,
+ 58
+ ],
+ [
+ 62,
+ 52,
+ 37,
+ 54,
+ 42,
+ 36
+ ],
+ [
+ 10,
+ 9,
+ 28,
+ 55,
+ 2,
+ 34
+ ],
+ [
+ 57,
+ 30,
+ 27,
+ 45,
+ 47,
+ 16
+ ]
+ ],
+ [
+ [
+ 11,
+ 16,
+ 31,
+ 46,
+ 0,
+ 21
+ ],
+ [
+ 13,
+ 49,
+ 34,
+ 31,
+ 16,
+ 8
+ ],
+ [
+ 36,
+ 13,
+ 27,
+ 58,
+ 18,
+ 34
+ ],
+ [
+ 24,
+ 61,
+ 15,
+ 46,
+ 63,
+ 25
+ ],
+ [
+ 17,
+ 2,
+ 50,
+ 35,
+ 58,
+ 44
+ ],
+ [
+ 8,
+ 23,
+ 7,
+ 49,
+ 26,
+ 47
+ ],
+ [
+ 61,
+ 38,
+ 29,
+ 58,
+ 0,
+ 36
+ ],
+ [
+ 42,
+ 20,
+ 9,
+ 4,
+ 11,
+ 27
+ ],
+ [
+ 1,
+ 34,
+ 30,
+ 48,
+ 7,
+ 26
+ ],
+ [
+ 27,
+ 10,
+ 28,
+ 62,
+ 13,
+ 54
+ ],
+ [
+ 60,
+ 1,
+ 30,
+ 59,
+ 6,
+ 52
+ ],
+ [
+ 31,
+ 62,
+ 25,
+ 42,
+ 21,
+ 32
+ ],
+ [
+ 34,
+ 32,
+ 19,
+ 2,
+ 62,
+ 14
+ ],
+ [
+ 6,
+ 40,
+ 24,
+ 31,
+ 13,
+ 48
+ ],
+ [
+ 44,
+ 2,
+ 40,
+ 8,
+ 34,
+ 45
+ ],
+ [
+ 8,
+ 27,
+ 7,
+ 57,
+ 50,
+ 47
+ ],
+ [
+ 13,
+ 60,
+ 45,
+ 22,
+ 25,
+ 52
+ ],
+ [
+ 4,
+ 23,
+ 33,
+ 46,
+ 58,
+ 34
+ ],
+ [
+ 8,
+ 33,
+ 56,
+ 45,
+ 51,
+ 59
+ ],
+ [
+ 38,
+ 53,
+ 39,
+ 26,
+ 35,
+ 31
+ ],
+ [
+ 41,
+ 46,
+ 11,
+ 53,
+ 59,
+ 8
+ ],
+ [
+ 11,
+ 44,
+ 33,
+ 52,
+ 55,
+ 53
+ ],
+ [
+ 31,
+ 1,
+ 16,
+ 9,
+ 15,
+ 41
+ ],
+ [
+ 2,
+ 31,
+ 14,
+ 10,
+ 44,
+ 7
+ ],
+ [
+ 45,
+ 6,
+ 62,
+ 12,
+ 18,
+ 24
+ ],
+ [
+ 10,
+ 50,
+ 42,
+ 54,
+ 24,
+ 3
+ ],
+ [
+ 17,
+ 31,
+ 37,
+ 5,
+ 10,
+ 41
+ ]
+ ],
+ [
+ [
+ 45,
+ 37,
+ 48,
+ 29,
+ 30,
+ 59
+ ],
+ [
+ 8,
+ 59,
+ 10,
+ 60,
+ 43,
+ 55
+ ],
+ [
+ 45,
+ 51,
+ 28,
+ 59,
+ 34,
+ 31
+ ],
+ [
+ 14,
+ 31,
+ 22,
+ 2,
+ 19,
+ 44
+ ],
+ [
+ 59,
+ 50,
+ 4,
+ 31,
+ 9,
+ 44
+ ],
+ [
+ 58,
+ 23,
+ 47,
+ 9,
+ 53,
+ 30
+ ],
+ [
+ 54,
+ 13,
+ 51,
+ 22,
+ 29,
+ 61
+ ],
+ [
+ 47,
+ 42,
+ 32,
+ 20,
+ 24,
+ 37
+ ],
+ [
+ 33,
+ 12,
+ 47,
+ 10,
+ 30,
+ 44
+ ],
+ [
+ 30,
+ 27,
+ 10,
+ 28,
+ 33,
+ 13
+ ],
+ [
+ 61,
+ 23,
+ 14,
+ 20,
+ 1,
+ 29
+ ],
+ [
+ 12,
+ 14,
+ 32,
+ 62,
+ 31,
+ 52
+ ],
+ [
+ 5,
+ 36,
+ 19,
+ 20,
+ 2,
+ 32
+ ],
+ [
+ 46,
+ 24,
+ 1,
+ 40,
+ 17,
+ 33
+ ],
+ [
+ 44,
+ 8,
+ 16,
+ 30,
+ 2,
+ 6
+ ],
+ [
+ 6,
+ 46,
+ 15,
+ 54,
+ 21,
+ 33
+ ],
+ [
+ 2,
+ 30,
+ 60,
+ 61,
+ 18,
+ 10
+ ],
+ [
+ 6,
+ 17,
+ 33,
+ 27,
+ 58,
+ 4
+ ],
+ [
+ 56,
+ 46,
+ 8,
+ 60,
+ 22,
+ 16
+ ],
+ [
+ 5,
+ 13,
+ 14,
+ 35,
+ 36,
+ 52
+ ],
+ [
+ 15,
+ 53,
+ 59,
+ 23,
+ 6,
+ 46
+ ],
+ [
+ 36,
+ 30,
+ 60,
+ 33,
+ 53,
+ 38
+ ],
+ [
+ 10,
+ 15,
+ 16,
+ 52,
+ 4,
+ 38
+ ],
+ [
+ 31,
+ 1,
+ 10,
+ 4,
+ 41,
+ 59
+ ],
+ [
+ 32,
+ 9,
+ 8,
+ 59,
+ 33,
+ 18
+ ],
+ [
+ 33,
+ 19,
+ 6,
+ 3,
+ 45,
+ 50
+ ],
+ [
+ 55,
+ 4,
+ 43,
+ 5,
+ 25,
+ 34
+ ]
+ ],
+ [
+ [
+ 25,
+ 14,
+ 18,
+ 49,
+ 51,
+ 63
+ ],
+ [
+ 42,
+ 21,
+ 30,
+ 43,
+ 24,
+ 7
+ ],
+ [
+ 54,
+ 39,
+ 9,
+ 59,
+ 28,
+ 45
+ ],
+ [
+ 1,
+ 23,
+ 55,
+ 8,
+ 16,
+ 7
+ ],
+ [
+ 59,
+ 37,
+ 30,
+ 41,
+ 16,
+ 1
+ ],
+ [
+ 4,
+ 31,
+ 41,
+ 45,
+ 18,
+ 56
+ ],
+ [
+ 28,
+ 55,
+ 44,
+ 54,
+ 9,
+ 51
+ ],
+ [
+ 24,
+ 47,
+ 12,
+ 63,
+ 32,
+ 61
+ ],
+ [
+ 33,
+ 47,
+ 39,
+ 44,
+ 38,
+ 56
+ ],
+ [
+ 40,
+ 30,
+ 59,
+ 33,
+ 58,
+ 10
+ ],
+ [
+ 19,
+ 8,
+ 23,
+ 4,
+ 25,
+ 34
+ ],
+ [
+ 45,
+ 14,
+ 12,
+ 42,
+ 62,
+ 52
+ ],
+ [
+ 58,
+ 45,
+ 59,
+ 29,
+ 25,
+ 36
+ ],
+ [
+ 8,
+ 59,
+ 1,
+ 47,
+ 33,
+ 24
+ ],
+ [
+ 24,
+ 30,
+ 38,
+ 56,
+ 4,
+ 7
+ ],
+ [
+ 46,
+ 22,
+ 44,
+ 57,
+ 12,
+ 40
+ ],
+ [
+ 34,
+ 60,
+ 2,
+ 17,
+ 18,
+ 29
+ ],
+ [
+ 6,
+ 61,
+ 51,
+ 39,
+ 53,
+ 32
+ ],
+ [
+ 49,
+ 5,
+ 47,
+ 60,
+ 21,
+ 59
+ ],
+ [
+ 57,
+ 4,
+ 23,
+ 39,
+ 45,
+ 60
+ ],
+ [
+ 7,
+ 36,
+ 45,
+ 53,
+ 21,
+ 56
+ ],
+ [
+ 60,
+ 3,
+ 39,
+ 18,
+ 8,
+ 53
+ ],
+ [
+ 57,
+ 33,
+ 15,
+ 47,
+ 38,
+ 16
+ ],
+ [
+ 20,
+ 11,
+ 19,
+ 4,
+ 2,
+ 8
+ ],
+ [
+ 36,
+ 34,
+ 60,
+ 47,
+ 1,
+ 40
+ ],
+ [
+ 33,
+ 14,
+ 3,
+ 21,
+ 19,
+ 35
+ ],
+ [
+ 60,
+ 24,
+ 22,
+ 52,
+ 35,
+ 53
+ ]
+ ],
+ [
+ [
+ 23,
+ 54,
+ 53,
+ 58,
+ 11,
+ 8
+ ],
+ [
+ 11,
+ 30,
+ 15,
+ 59,
+ 55,
+ 63
+ ],
+ [
+ 20,
+ 58,
+ 29,
+ 17,
+ 42,
+ 30
+ ],
+ [
+ 18,
+ 1,
+ 43,
+ 15,
+ 8,
+ 3
+ ],
+ [
+ 59,
+ 55,
+ 13,
+ 28,
+ 26,
+ 63
+ ],
+ [
+ 33,
+ 45,
+ 27,
+ 53,
+ 63,
+ 19
+ ],
+ [
+ 28,
+ 57,
+ 51,
+ 54,
+ 34,
+ 53
+ ],
+ [
+ 24,
+ 13,
+ 47,
+ 3,
+ 45,
+ 50
+ ],
+ [
+ 47,
+ 59,
+ 33,
+ 57,
+ 37,
+ 35
+ ],
+ [
+ 23,
+ 42,
+ 33,
+ 41,
+ 48,
+ 30
+ ],
+ [
+ 42,
+ 63,
+ 23,
+ 25,
+ 34,
+ 17
+ ],
+ [
+ 14,
+ 62,
+ 2,
+ 19,
+ 45,
+ 43
+ ],
+ [
+ 12,
+ 6,
+ 20,
+ 30,
+ 29,
+ 17
+ ],
+ [
+ 6,
+ 3,
+ 17,
+ 8,
+ 27,
+ 31
+ ],
+ [
+ 19,
+ 59,
+ 2,
+ 10,
+ 54,
+ 26
+ ],
+ [
+ 45,
+ 61,
+ 19,
+ 57,
+ 42,
+ 17
+ ],
+ [
+ 55,
+ 60,
+ 41,
+ 34,
+ 35,
+ 53
+ ],
+ [
+ 36,
+ 2,
+ 8,
+ 31,
+ 14,
+ 5
+ ],
+ [
+ 45,
+ 36,
+ 16,
+ 38,
+ 51,
+ 21
+ ],
+ [
+ 57,
+ 39,
+ 48,
+ 6,
+ 19,
+ 60
+ ],
+ [
+ 39,
+ 37,
+ 59,
+ 48,
+ 42,
+ 40
+ ],
+ [
+ 52,
+ 28,
+ 46,
+ 33,
+ 18,
+ 55
+ ],
+ [
+ 1,
+ 15,
+ 28,
+ 18,
+ 13,
+ 52
+ ],
+ [
+ 10,
+ 21,
+ 4,
+ 2,
+ 23,
+ 31
+ ],
+ [
+ 45,
+ 9,
+ 41,
+ 18,
+ 54,
+ 3
+ ],
+ [
+ 42,
+ 61,
+ 36,
+ 3,
+ 19,
+ 27
+ ],
+ [
+ 41,
+ 46,
+ 33,
+ 3,
+ 10,
+ 15
+ ]
+ ],
+ [
+ [
+ 49,
+ 52,
+ 60,
+ 63,
+ 21,
+ 0
+ ],
+ [
+ 14,
+ 7,
+ 25,
+ 52,
+ 58,
+ 36
+ ],
+ [
+ 46,
+ 57,
+ 28,
+ 24,
+ 49,
+ 12
+ ],
+ [
+ 10,
+ 21,
+ 39,
+ 1,
+ 11,
+ 17
+ ],
+ [
+ 53,
+ 3,
+ 2,
+ 16,
+ 46,
+ 44
+ ],
+ [
+ 25,
+ 40,
+ 39,
+ 15,
+ 34,
+ 30
+ ],
+ [
+ 62,
+ 24,
+ 47,
+ 52,
+ 17,
+ 49
+ ],
+ [
+ 34,
+ 41,
+ 39,
+ 11,
+ 33,
+ 15
+ ],
+ [
+ 36,
+ 26,
+ 48,
+ 39,
+ 15,
+ 56
+ ],
+ [
+ 23,
+ 36,
+ 43,
+ 20,
+ 10,
+ 21
+ ],
+ [
+ 48,
+ 63,
+ 51,
+ 10,
+ 62,
+ 18
+ ],
+ [
+ 10,
+ 42,
+ 36,
+ 39,
+ 49,
+ 19
+ ],
+ [
+ 22,
+ 38,
+ 51,
+ 56,
+ 50,
+ 7
+ ],
+ [
+ 4,
+ 31,
+ 19,
+ 22,
+ 17,
+ 7
+ ],
+ [
+ 41,
+ 7,
+ 39,
+ 2,
+ 26,
+ 57
+ ],
+ [
+ 33,
+ 27,
+ 58,
+ 28,
+ 20,
+ 45
+ ],
+ [
+ 58,
+ 8,
+ 37,
+ 18,
+ 34,
+ 62
+ ],
+ [
+ 24,
+ 34,
+ 43,
+ 39,
+ 17,
+ 19
+ ],
+ [
+ 23,
+ 50,
+ 12,
+ 32,
+ 43,
+ 31
+ ],
+ [
+ 9,
+ 7,
+ 51,
+ 16,
+ 30,
+ 60
+ ],
+ [
+ 25,
+ 62,
+ 14,
+ 53,
+ 58,
+ 0
+ ],
+ [
+ 53,
+ 4,
+ 18,
+ 22,
+ 14,
+ 29
+ ],
+ [
+ 6,
+ 14,
+ 15,
+ 48,
+ 58,
+ 0
+ ],
+ [
+ 35,
+ 4,
+ 45,
+ 3,
+ 59,
+ 39
+ ],
+ [
+ 17,
+ 57,
+ 23,
+ 16,
+ 40,
+ 37
+ ],
+ [
+ 9,
+ 19,
+ 3,
+ 36,
+ 11,
+ 47
+ ],
+ [
+ 39,
+ 14,
+ 22,
+ 33,
+ 62,
+ 42
+ ]
+ ],
+ [
+ [
+ 14,
+ 17,
+ 2,
+ 39,
+ 47,
+ 63
+ ],
+ [
+ 23,
+ 58,
+ 2,
+ 25,
+ 5,
+ 18
+ ],
+ [
+ 27,
+ 33,
+ 0,
+ 56,
+ 6,
+ 7
+ ],
+ [
+ 23,
+ 5,
+ 30,
+ 7,
+ 21,
+ 39
+ ],
+ [
+ 39,
+ 53,
+ 27,
+ 38,
+ 54,
+ 41
+ ],
+ [
+ 37,
+ 25,
+ 50,
+ 40,
+ 33,
+ 36
+ ],
+ [
+ 49,
+ 37,
+ 24,
+ 55,
+ 29,
+ 28
+ ],
+ [
+ 34,
+ 15,
+ 36,
+ 16,
+ 30,
+ 11
+ ],
+ [
+ 5,
+ 36,
+ 2,
+ 26,
+ 16,
+ 48
+ ],
+ [
+ 56,
+ 46,
+ 10,
+ 23,
+ 39,
+ 45
+ ],
+ [
+ 10,
+ 56,
+ 62,
+ 45,
+ 49,
+ 3
+ ],
+ [
+ 35,
+ 6,
+ 50,
+ 36,
+ 11,
+ 1
+ ],
+ [
+ 50,
+ 46,
+ 39,
+ 51,
+ 38,
+ 41
+ ],
+ [
+ 41,
+ 19,
+ 31,
+ 6,
+ 48,
+ 24
+ ],
+ [
+ 2,
+ 7,
+ 57,
+ 18,
+ 47,
+ 63
+ ],
+ [
+ 27,
+ 58,
+ 22,
+ 8,
+ 57,
+ 45
+ ],
+ [
+ 37,
+ 59,
+ 51,
+ 44,
+ 55,
+ 18
+ ],
+ [
+ 52,
+ 21,
+ 25,
+ 12,
+ 34,
+ 39
+ ],
+ [
+ 23,
+ 34,
+ 60,
+ 13,
+ 27,
+ 32
+ ],
+ [
+ 32,
+ 56,
+ 26,
+ 24,
+ 44,
+ 30
+ ],
+ [
+ 41,
+ 12,
+ 53,
+ 2,
+ 5,
+ 56
+ ],
+ [
+ 44,
+ 18,
+ 39,
+ 33,
+ 38,
+ 46
+ ],
+ [
+ 25,
+ 0,
+ 59,
+ 18,
+ 11,
+ 53
+ ],
+ [
+ 1,
+ 7,
+ 17,
+ 30,
+ 4,
+ 37
+ ],
+ [
+ 62,
+ 48,
+ 19,
+ 54,
+ 37,
+ 22
+ ],
+ [
+ 10,
+ 28,
+ 3,
+ 16,
+ 11,
+ 42
+ ],
+ [
+ 26,
+ 30,
+ 57,
+ 42,
+ 41,
+ 32
+ ]
+ ],
+ [
+ [
+ 36,
+ 31,
+ 37,
+ 16,
+ 43,
+ 63
+ ],
+ [
+ 26,
+ 51,
+ 0,
+ 48,
+ 42,
+ 21
+ ],
+ [
+ 18,
+ 41,
+ 37,
+ 34,
+ 24,
+ 30
+ ],
+ [
+ 58,
+ 4,
+ 22,
+ 44,
+ 51,
+ 18
+ ],
+ [
+ 18,
+ 26,
+ 4,
+ 61,
+ 14,
+ 31
+ ],
+ [
+ 9,
+ 23,
+ 34,
+ 29,
+ 3,
+ 31
+ ],
+ [
+ 12,
+ 58,
+ 36,
+ 46,
+ 16,
+ 25
+ ],
+ [
+ 4,
+ 56,
+ 9,
+ 33,
+ 60,
+ 32
+ ],
+ [
+ 14,
+ 26,
+ 43,
+ 30,
+ 46,
+ 35
+ ],
+ [
+ 2,
+ 0,
+ 10,
+ 43,
+ 28,
+ 27
+ ],
+ [
+ 61,
+ 18,
+ 19,
+ 17,
+ 2,
+ 33
+ ],
+ [
+ 60,
+ 19,
+ 23,
+ 62,
+ 37,
+ 43
+ ],
+ [
+ 56,
+ 32,
+ 8,
+ 61,
+ 58,
+ 38
+ ],
+ [
+ 25,
+ 28,
+ 22,
+ 0,
+ 26,
+ 29
+ ],
+ [
+ 22,
+ 49,
+ 7,
+ 60,
+ 23,
+ 55
+ ],
+ [
+ 46,
+ 57,
+ 19,
+ 37,
+ 12,
+ 27
+ ],
+ [
+ 22,
+ 60,
+ 11,
+ 30,
+ 19,
+ 20
+ ],
+ [
+ 36,
+ 20,
+ 17,
+ 2,
+ 37,
+ 55
+ ],
+ [
+ 13,
+ 16,
+ 32,
+ 58,
+ 41,
+ 10
+ ],
+ [
+ 50,
+ 55,
+ 13,
+ 25,
+ 14,
+ 41
+ ],
+ [
+ 14,
+ 3,
+ 44,
+ 48,
+ 60,
+ 6
+ ],
+ [
+ 36,
+ 18,
+ 57,
+ 22,
+ 1,
+ 43
+ ],
+ [
+ 10,
+ 22,
+ 15,
+ 52,
+ 13,
+ 18
+ ],
+ [
+ 4,
+ 56,
+ 10,
+ 31,
+ 14,
+ 62
+ ],
+ [
+ 9,
+ 4,
+ 16,
+ 54,
+ 8,
+ 59
+ ],
+ [
+ 3,
+ 43,
+ 27,
+ 45,
+ 6,
+ 30
+ ],
+ [
+ 33,
+ 4,
+ 43,
+ 40,
+ 46,
+ 45
+ ]
+ ],
+ [
+ [
+ 48,
+ 42,
+ 38,
+ 63,
+ 47,
+ 7
+ ],
+ [
+ 3,
+ 62,
+ 10,
+ 6,
+ 26,
+ 2
+ ],
+ [
+ 39,
+ 44,
+ 6,
+ 7,
+ 45,
+ 40
+ ],
+ [
+ 56,
+ 33,
+ 2,
+ 41,
+ 48,
+ 16
+ ],
+ [
+ 6,
+ 18,
+ 37,
+ 27,
+ 41,
+ 30
+ ],
+ [
+ 59,
+ 45,
+ 23,
+ 31,
+ 53,
+ 9
+ ],
+ [
+ 0,
+ 13,
+ 12,
+ 44,
+ 33,
+ 16
+ ],
+ [
+ 12,
+ 56,
+ 50,
+ 60,
+ 28,
+ 63
+ ],
+ [
+ 14,
+ 33,
+ 26,
+ 24,
+ 32,
+ 16
+ ],
+ [
+ 43,
+ 2,
+ 0,
+ 50,
+ 28,
+ 22
+ ],
+ [
+ 61,
+ 63,
+ 9,
+ 8,
+ 17,
+ 39
+ ],
+ [
+ 60,
+ 19,
+ 59,
+ 23,
+ 24,
+ 30
+ ],
+ [
+ 56,
+ 59,
+ 53,
+ 32,
+ 7,
+ 38
+ ],
+ [
+ 28,
+ 25,
+ 22,
+ 24,
+ 59,
+ 42
+ ],
+ [
+ 22,
+ 49,
+ 46,
+ 23,
+ 60,
+ 7
+ ],
+ [
+ 46,
+ 34,
+ 27,
+ 29,
+ 0,
+ 12
+ ],
+ [
+ 17,
+ 22,
+ 5,
+ 31,
+ 20,
+ 27
+ ],
+ [
+ 39,
+ 48,
+ 57,
+ 37,
+ 17,
+ 36
+ ],
+ [
+ 42,
+ 49,
+ 32,
+ 7,
+ 16,
+ 61
+ ],
+ [
+ 1,
+ 7,
+ 23,
+ 14,
+ 36,
+ 39
+ ],
+ [
+ 21,
+ 7,
+ 48,
+ 14,
+ 60,
+ 35
+ ],
+ [
+ 8,
+ 18,
+ 24,
+ 60,
+ 1,
+ 0
+ ],
+ [
+ 51,
+ 15,
+ 33,
+ 36,
+ 5,
+ 30
+ ],
+ [
+ 52,
+ 4,
+ 8,
+ 29,
+ 9,
+ 26
+ ],
+ [
+ 4,
+ 9,
+ 16,
+ 40,
+ 58,
+ 60
+ ],
+ [
+ 3,
+ 17,
+ 21,
+ 61,
+ 19,
+ 8
+ ],
+ [
+ 35,
+ 54,
+ 1,
+ 53,
+ 60,
+ 0
+ ]
+ ],
+ [
+ [
+ 19,
+ 1,
+ 31,
+ 52,
+ 49,
+ 63
+ ],
+ [
+ 7,
+ 47,
+ 5,
+ 60,
+ 22,
+ 46
+ ],
+ [
+ 59,
+ 30,
+ 3,
+ 11,
+ 0,
+ 19
+ ],
+ [
+ 43,
+ 42,
+ 19,
+ 62,
+ 8,
+ 56
+ ],
+ [
+ 61,
+ 15,
+ 25,
+ 18,
+ 39,
+ 26
+ ],
+ [
+ 61,
+ 50,
+ 36,
+ 45,
+ 33,
+ 44
+ ],
+ [
+ 8,
+ 37,
+ 52,
+ 1,
+ 2,
+ 41
+ ],
+ [
+ 36,
+ 30,
+ 53,
+ 11,
+ 16,
+ 29
+ ],
+ [
+ 14,
+ 58,
+ 49,
+ 46,
+ 3,
+ 26
+ ],
+ [
+ 62,
+ 43,
+ 0,
+ 22,
+ 45,
+ 46
+ ],
+ [
+ 6,
+ 56,
+ 45,
+ 18,
+ 10,
+ 41
+ ],
+ [
+ 60,
+ 50,
+ 21,
+ 47,
+ 30,
+ 35
+ ],
+ [
+ 11,
+ 53,
+ 28,
+ 56,
+ 41,
+ 39
+ ],
+ [
+ 23,
+ 9,
+ 33,
+ 28,
+ 22,
+ 26
+ ],
+ [
+ 23,
+ 56,
+ 34,
+ 27,
+ 2,
+ 20
+ ],
+ [
+ 22,
+ 9,
+ 44,
+ 41,
+ 37,
+ 47
+ ],
+ [
+ 1,
+ 11,
+ 46,
+ 27,
+ 52,
+ 45
+ ],
+ [
+ 51,
+ 37,
+ 17,
+ 21,
+ 61,
+ 9
+ ],
+ [
+ 13,
+ 19,
+ 32,
+ 5,
+ 2,
+ 9
+ ],
+ [
+ 41,
+ 32,
+ 6,
+ 47,
+ 29,
+ 56
+ ],
+ [
+ 53,
+ 33,
+ 9,
+ 35,
+ 38,
+ 12
+ ],
+ [
+ 40,
+ 19,
+ 51,
+ 26,
+ 7,
+ 22
+ ],
+ [
+ 37,
+ 5,
+ 25,
+ 46,
+ 34,
+ 53
+ ],
+ [
+ 16,
+ 55,
+ 20,
+ 24,
+ 44,
+ 29
+ ],
+ [
+ 21,
+ 46,
+ 11,
+ 0,
+ 36,
+ 7
+ ],
+ [
+ 18,
+ 60,
+ 32,
+ 3,
+ 34,
+ 28
+ ],
+ [
+ 52,
+ 9,
+ 36,
+ 48,
+ 41,
+ 11
+ ]
+ ],
+ [
+ [
+ 53,
+ 15,
+ 34,
+ 0,
+ 36,
+ 56
+ ],
+ [
+ 8,
+ 12,
+ 41,
+ 11,
+ 19,
+ 55
+ ],
+ [
+ 56,
+ 13,
+ 31,
+ 36,
+ 23,
+ 47
+ ],
+ [
+ 36,
+ 51,
+ 30,
+ 7,
+ 26,
+ 54
+ ],
+ [
+ 58,
+ 13,
+ 50,
+ 2,
+ 53,
+ 34
+ ],
+ [
+ 49,
+ 52,
+ 23,
+ 32,
+ 7,
+ 26
+ ],
+ [
+ 61,
+ 38,
+ 23,
+ 0,
+ 4,
+ 28
+ ],
+ [
+ 42,
+ 27,
+ 17,
+ 9,
+ 18,
+ 20
+ ],
+ [
+ 1,
+ 34,
+ 45,
+ 4,
+ 12,
+ 7
+ ],
+ [
+ 54,
+ 27,
+ 57,
+ 38,
+ 44,
+ 6
+ ],
+ [
+ 59,
+ 40,
+ 1,
+ 60,
+ 48,
+ 30
+ ],
+ [
+ 25,
+ 31,
+ 32,
+ 51,
+ 62,
+ 8
+ ],
+ [
+ 62,
+ 19,
+ 14,
+ 2,
+ 37,
+ 15
+ ],
+ [
+ 40,
+ 57,
+ 37,
+ 35,
+ 22,
+ 61
+ ],
+ [
+ 45,
+ 16,
+ 34,
+ 42,
+ 37,
+ 48
+ ],
+ [
+ 50,
+ 16,
+ 62,
+ 33,
+ 25,
+ 37
+ ],
+ [
+ 42,
+ 13,
+ 39,
+ 47,
+ 3,
+ 63
+ ],
+ [
+ 46,
+ 23,
+ 28,
+ 27,
+ 4,
+ 15
+ ],
+ [
+ 56,
+ 62,
+ 31,
+ 35,
+ 59,
+ 45
+ ],
+ [
+ 15,
+ 38,
+ 13,
+ 4,
+ 63,
+ 48
+ ],
+ [
+ 34,
+ 15,
+ 57,
+ 38,
+ 13,
+ 24
+ ],
+ [
+ 62,
+ 36,
+ 41,
+ 54,
+ 46,
+ 29
+ ],
+ [
+ 22,
+ 53,
+ 46,
+ 34,
+ 30,
+ 27
+ ],
+ [
+ 24,
+ 10,
+ 4,
+ 47,
+ 18,
+ 36
+ ],
+ [
+ 50,
+ 57,
+ 51,
+ 11,
+ 49,
+ 3
+ ],
+ [
+ 51,
+ 7,
+ 0,
+ 18,
+ 11,
+ 44
+ ],
+ [
+ 39,
+ 37,
+ 9,
+ 42,
+ 40,
+ 44
+ ]
+ ],
+ [
+ [
+ 36,
+ 25,
+ 57,
+ 55,
+ 47,
+ 63
+ ],
+ [
+ 0,
+ 2,
+ 46,
+ 3,
+ 51,
+ 34
+ ],
+ [
+ 24,
+ 2,
+ 46,
+ 15,
+ 33,
+ 43
+ ],
+ [
+ 22,
+ 31,
+ 17,
+ 19,
+ 10,
+ 55
+ ],
+ [
+ 58,
+ 59,
+ 3,
+ 9,
+ 40,
+ 57
+ ],
+ [
+ 23,
+ 31,
+ 43,
+ 57,
+ 2,
+ 38
+ ],
+ [
+ 9,
+ 62,
+ 13,
+ 42,
+ 47,
+ 52
+ ],
+ [
+ 24,
+ 7,
+ 14,
+ 10,
+ 46,
+ 59
+ ],
+ [
+ 47,
+ 18,
+ 4,
+ 37,
+ 0,
+ 13
+ ],
+ [
+ 54,
+ 4,
+ 25,
+ 47,
+ 36,
+ 38
+ ],
+ [
+ 15,
+ 47,
+ 1,
+ 24,
+ 58,
+ 14
+ ],
+ [
+ 54,
+ 5,
+ 16,
+ 63,
+ 14,
+ 7
+ ],
+ [
+ 49,
+ 3,
+ 33,
+ 13,
+ 46,
+ 10
+ ],
+ [
+ 18,
+ 10,
+ 11,
+ 13,
+ 63,
+ 39
+ ],
+ [
+ 19,
+ 62,
+ 32,
+ 58,
+ 10,
+ 43
+ ],
+ [
+ 43,
+ 48,
+ 63,
+ 5,
+ 55,
+ 53
+ ],
+ [
+ 24,
+ 51,
+ 47,
+ 15,
+ 59,
+ 32
+ ],
+ [
+ 44,
+ 0,
+ 34,
+ 43,
+ 3,
+ 6
+ ],
+ [
+ 58,
+ 38,
+ 54,
+ 47,
+ 11,
+ 59
+ ],
+ [
+ 55,
+ 27,
+ 15,
+ 9,
+ 42,
+ 31
+ ],
+ [
+ 43,
+ 41,
+ 1,
+ 51,
+ 5,
+ 29
+ ],
+ [
+ 49,
+ 27,
+ 20,
+ 6,
+ 4,
+ 13
+ ],
+ [
+ 11,
+ 25,
+ 2,
+ 27,
+ 54,
+ 62
+ ],
+ [
+ 38,
+ 44,
+ 40,
+ 54,
+ 33,
+ 14
+ ],
+ [
+ 13,
+ 63,
+ 52,
+ 2,
+ 8,
+ 29
+ ],
+ [
+ 23,
+ 41,
+ 59,
+ 57,
+ 38,
+ 15
+ ],
+ [
+ 6,
+ 23,
+ 62,
+ 50,
+ 51,
+ 34
+ ]
+ ],
+ [
+ [
+ 41,
+ 2,
+ 42,
+ 16,
+ 50,
+ 23
+ ],
+ [
+ 51,
+ 41,
+ 5,
+ 15,
+ 40,
+ 21
+ ],
+ [
+ 43,
+ 1,
+ 29,
+ 55,
+ 21,
+ 35
+ ],
+ [
+ 24,
+ 53,
+ 25,
+ 51,
+ 32,
+ 29
+ ],
+ [
+ 41,
+ 31,
+ 49,
+ 57,
+ 60,
+ 34
+ ],
+ [
+ 17,
+ 4,
+ 35,
+ 30,
+ 10,
+ 38
+ ],
+ [
+ 34,
+ 7,
+ 21,
+ 9,
+ 48,
+ 31
+ ],
+ [
+ 14,
+ 24,
+ 7,
+ 46,
+ 25,
+ 27
+ ],
+ [
+ 47,
+ 0,
+ 12,
+ 6,
+ 37,
+ 60
+ ],
+ [
+ 4,
+ 30,
+ 25,
+ 47,
+ 36,
+ 54
+ ],
+ [
+ 24,
+ 61,
+ 15,
+ 47,
+ 46,
+ 1
+ ],
+ [
+ 5,
+ 14,
+ 24,
+ 16,
+ 57,
+ 63
+ ],
+ [
+ 49,
+ 3,
+ 17,
+ 26,
+ 36,
+ 44
+ ],
+ [
+ 13,
+ 10,
+ 61,
+ 0,
+ 11,
+ 22
+ ],
+ [
+ 10,
+ 6,
+ 2,
+ 49,
+ 58,
+ 46
+ ],
+ [
+ 19,
+ 11,
+ 2,
+ 25,
+ 54,
+ 18
+ ],
+ [
+ 10,
+ 5,
+ 52,
+ 24,
+ 18,
+ 17
+ ],
+ [
+ 44,
+ 13,
+ 55,
+ 31,
+ 63,
+ 38
+ ],
+ [
+ 58,
+ 7,
+ 25,
+ 32,
+ 38,
+ 14
+ ],
+ [
+ 49,
+ 55,
+ 34,
+ 13,
+ 16,
+ 40
+ ],
+ [
+ 51,
+ 23,
+ 59,
+ 35,
+ 5,
+ 4
+ ],
+ [
+ 57,
+ 15,
+ 46,
+ 27,
+ 42,
+ 32
+ ],
+ [
+ 33,
+ 15,
+ 23,
+ 52,
+ 24,
+ 27
+ ],
+ [
+ 4,
+ 41,
+ 33,
+ 10,
+ 26,
+ 40
+ ],
+ [
+ 8,
+ 13,
+ 59,
+ 4,
+ 9,
+ 39
+ ],
+ [
+ 19,
+ 3,
+ 27,
+ 31,
+ 43,
+ 2
+ ],
+ [
+ 46,
+ 61,
+ 25,
+ 8,
+ 29,
+ 50
+ ]
+ ],
+ [
+ [
+ 48,
+ 38,
+ 62,
+ 63,
+ 42,
+ 50
+ ],
+ [
+ 3,
+ 2,
+ 26,
+ 17,
+ 10,
+ 6
+ ],
+ [
+ 39,
+ 44,
+ 6,
+ 45,
+ 7,
+ 40
+ ],
+ [
+ 5,
+ 62,
+ 53,
+ 50,
+ 41,
+ 48
+ ],
+ [
+ 6,
+ 41,
+ 49,
+ 37,
+ 30,
+ 23
+ ],
+ [
+ 59,
+ 60,
+ 4,
+ 46,
+ 53,
+ 29
+ ],
+ [
+ 44,
+ 34,
+ 7,
+ 15,
+ 13,
+ 43
+ ],
+ [
+ 12,
+ 14,
+ 24,
+ 25,
+ 58,
+ 7
+ ],
+ [
+ 33,
+ 35,
+ 4,
+ 37,
+ 8,
+ 36
+ ],
+ [
+ 40,
+ 30,
+ 25,
+ 36,
+ 20,
+ 54
+ ],
+ [
+ 9,
+ 8,
+ 24,
+ 25,
+ 5,
+ 63
+ ],
+ [
+ 59,
+ 34,
+ 5,
+ 24,
+ 57,
+ 6
+ ],
+ [
+ 17,
+ 49,
+ 44,
+ 26,
+ 55,
+ 7
+ ],
+ [
+ 13,
+ 47,
+ 59,
+ 27,
+ 22,
+ 61
+ ],
+ [
+ 49,
+ 2,
+ 44,
+ 10,
+ 46,
+ 40
+ ],
+ [
+ 34,
+ 2,
+ 54,
+ 57,
+ 53,
+ 55
+ ],
+ [
+ 5,
+ 17,
+ 29,
+ 31,
+ 43,
+ 52
+ ],
+ [
+ 48,
+ 57,
+ 38,
+ 63,
+ 39,
+ 43
+ ],
+ [
+ 42,
+ 38,
+ 49,
+ 32,
+ 7,
+ 40
+ ],
+ [
+ 59,
+ 1,
+ 16,
+ 23,
+ 60,
+ 10
+ ],
+ [
+ 21,
+ 7,
+ 35,
+ 53,
+ 48,
+ 31
+ ],
+ [
+ 8,
+ 60,
+ 24,
+ 42,
+ 14,
+ 35
+ ],
+ [
+ 33,
+ 51,
+ 15,
+ 28,
+ 23,
+ 44
+ ],
+ [
+ 4,
+ 41,
+ 9,
+ 8,
+ 11,
+ 51
+ ],
+ [
+ 60,
+ 24,
+ 19,
+ 48,
+ 9,
+ 4
+ ],
+ [
+ 19,
+ 3,
+ 26,
+ 58,
+ 12,
+ 61
+ ],
+ [
+ 35,
+ 54,
+ 1,
+ 60,
+ 53,
+ 49
+ ]
+ ],
+ [
+ [
+ 21,
+ 7,
+ 53,
+ 56,
+ 63,
+ 33
+ ],
+ [
+ 3,
+ 34,
+ 57,
+ 16,
+ 20,
+ 51
+ ],
+ [
+ 55,
+ 11,
+ 16,
+ 60,
+ 0,
+ 13
+ ],
+ [
+ 62,
+ 43,
+ 5,
+ 50,
+ 8,
+ 42
+ ],
+ [
+ 35,
+ 52,
+ 9,
+ 43,
+ 0,
+ 27
+ ],
+ [
+ 16,
+ 60,
+ 29,
+ 61,
+ 28,
+ 58
+ ],
+ [
+ 31,
+ 34,
+ 43,
+ 30,
+ 2,
+ 18
+ ],
+ [
+ 36,
+ 25,
+ 53,
+ 24,
+ 35,
+ 48
+ ],
+ [
+ 48,
+ 35,
+ 4,
+ 32,
+ 8,
+ 60
+ ],
+ [
+ 25,
+ 54,
+ 30,
+ 10,
+ 4,
+ 41
+ ],
+ [
+ 9,
+ 3,
+ 19,
+ 20,
+ 61,
+ 24
+ ],
+ [
+ 30,
+ 5,
+ 17,
+ 45,
+ 18,
+ 49
+ ],
+ [
+ 49,
+ 21,
+ 17,
+ 26,
+ 42,
+ 37
+ ],
+ [
+ 38,
+ 25,
+ 10,
+ 51,
+ 54,
+ 13
+ ],
+ [
+ 2,
+ 27,
+ 26,
+ 10,
+ 58,
+ 30
+ ],
+ [
+ 9,
+ 2,
+ 54,
+ 21,
+ 25,
+ 13
+ ],
+ [
+ 22,
+ 23,
+ 33,
+ 27,
+ 51,
+ 52
+ ],
+ [
+ 30,
+ 21,
+ 35,
+ 55,
+ 5,
+ 17
+ ],
+ [
+ 38,
+ 27,
+ 24,
+ 56,
+ 21,
+ 35
+ ],
+ [
+ 53,
+ 45,
+ 22,
+ 28,
+ 0,
+ 32
+ ],
+ [
+ 40,
+ 42,
+ 48,
+ 37,
+ 8,
+ 52
+ ],
+ [
+ 46,
+ 45,
+ 57,
+ 36,
+ 51,
+ 32
+ ],
+ [
+ 62,
+ 10,
+ 38,
+ 42,
+ 41,
+ 54
+ ],
+ [
+ 60,
+ 1,
+ 10,
+ 31,
+ 44,
+ 36
+ ],
+ [
+ 29,
+ 6,
+ 9,
+ 8,
+ 56,
+ 7
+ ],
+ [
+ 9,
+ 59,
+ 22,
+ 0,
+ 35,
+ 57
+ ],
+ [
+ 45,
+ 27,
+ 62,
+ 47,
+ 3,
+ 28
+ ]
+ ],
+ [
+ [
+ 27,
+ 13,
+ 18,
+ 8,
+ 63,
+ 55
+ ],
+ [
+ 36,
+ 21,
+ 57,
+ 8,
+ 46,
+ 55
+ ],
+ [
+ 43,
+ 61,
+ 10,
+ 13,
+ 41,
+ 37
+ ],
+ [
+ 43,
+ 16,
+ 24,
+ 6,
+ 26,
+ 61
+ ],
+ [
+ 60,
+ 29,
+ 35,
+ 31,
+ 16,
+ 23
+ ],
+ [
+ 9,
+ 58,
+ 60,
+ 0,
+ 17,
+ 38
+ ],
+ [
+ 63,
+ 16,
+ 7,
+ 13,
+ 31,
+ 18
+ ],
+ [
+ 16,
+ 25,
+ 24,
+ 2,
+ 47,
+ 58
+ ],
+ [
+ 8,
+ 35,
+ 6,
+ 14,
+ 48,
+ 4
+ ],
+ [
+ 30,
+ 25,
+ 54,
+ 4,
+ 10,
+ 34
+ ],
+ [
+ 20,
+ 23,
+ 19,
+ 7,
+ 38,
+ 61
+ ],
+ [
+ 5,
+ 3,
+ 24,
+ 32,
+ 12,
+ 42
+ ],
+ [
+ 49,
+ 17,
+ 10,
+ 26,
+ 32,
+ 60
+ ],
+ [
+ 10,
+ 13,
+ 29,
+ 22,
+ 58,
+ 54
+ ],
+ [
+ 44,
+ 8,
+ 58,
+ 2,
+ 6,
+ 5
+ ],
+ [
+ 25,
+ 6,
+ 2,
+ 54,
+ 19,
+ 53
+ ],
+ [
+ 5,
+ 61,
+ 30,
+ 1,
+ 17,
+ 27
+ ],
+ [
+ 22,
+ 55,
+ 63,
+ 57,
+ 33,
+ 19
+ ],
+ [
+ 35,
+ 46,
+ 6,
+ 32,
+ 14,
+ 7
+ ],
+ [
+ 45,
+ 13,
+ 51,
+ 14,
+ 7,
+ 5
+ ],
+ [
+ 15,
+ 23,
+ 50,
+ 51,
+ 13,
+ 59
+ ],
+ [
+ 57,
+ 36,
+ 53,
+ 61,
+ 6,
+ 14
+ ],
+ [
+ 10,
+ 23,
+ 15,
+ 33,
+ 27,
+ 38
+ ],
+ [
+ 4,
+ 10,
+ 33,
+ 41,
+ 26,
+ 36
+ ],
+ [
+ 8,
+ 4,
+ 33,
+ 9,
+ 47,
+ 20
+ ],
+ [
+ 3,
+ 43,
+ 6,
+ 19,
+ 30,
+ 2
+ ],
+ [
+ 55,
+ 4,
+ 46,
+ 40,
+ 18,
+ 20
+ ]
+ ],
+ [
+ [
+ 48,
+ 38,
+ 63,
+ 37,
+ 7,
+ 42
+ ],
+ [
+ 3,
+ 26,
+ 10,
+ 35,
+ 6,
+ 2
+ ],
+ [
+ 39,
+ 44,
+ 7,
+ 45,
+ 6,
+ 40
+ ],
+ [
+ 56,
+ 23,
+ 61,
+ 16,
+ 33,
+ 41
+ ],
+ [
+ 6,
+ 37,
+ 44,
+ 16,
+ 30,
+ 19
+ ],
+ [
+ 59,
+ 17,
+ 60,
+ 19,
+ 5,
+ 52
+ ],
+ [
+ 44,
+ 55,
+ 33,
+ 63,
+ 13,
+ 31
+ ],
+ [
+ 12,
+ 25,
+ 63,
+ 2,
+ 24,
+ 54
+ ],
+ [
+ 33,
+ 8,
+ 42,
+ 4,
+ 35,
+ 41
+ ],
+ [
+ 30,
+ 40,
+ 25,
+ 21,
+ 48,
+ 51
+ ],
+ [
+ 8,
+ 23,
+ 34,
+ 5,
+ 19,
+ 22
+ ],
+ [
+ 34,
+ 59,
+ 42,
+ 11,
+ 5,
+ 3
+ ],
+ [
+ 10,
+ 26,
+ 50,
+ 55,
+ 53,
+ 59
+ ],
+ [
+ 59,
+ 54,
+ 13,
+ 10,
+ 47,
+ 25
+ ],
+ [
+ 24,
+ 2,
+ 49,
+ 46,
+ 38,
+ 8
+ ],
+ [
+ 34,
+ 2,
+ 53,
+ 54,
+ 40,
+ 57
+ ],
+ [
+ 17,
+ 29,
+ 5,
+ 27,
+ 18,
+ 43
+ ],
+ [
+ 48,
+ 57,
+ 63,
+ 39,
+ 38,
+ 32
+ ],
+ [
+ 42,
+ 49,
+ 7,
+ 32,
+ 38,
+ 61
+ ],
+ [
+ 1,
+ 59,
+ 23,
+ 14,
+ 16,
+ 10
+ ],
+ [
+ 21,
+ 7,
+ 35,
+ 53,
+ 13,
+ 11
+ ],
+ [
+ 8,
+ 60,
+ 24,
+ 25,
+ 55,
+ 42
+ ],
+ [
+ 33,
+ 15,
+ 5,
+ 51,
+ 28,
+ 44
+ ],
+ [
+ 4,
+ 9,
+ 51,
+ 8,
+ 41,
+ 39
+ ],
+ [
+ 60,
+ 4,
+ 19,
+ 48,
+ 9,
+ 24
+ ],
+ [
+ 19,
+ 3,
+ 26,
+ 12,
+ 21,
+ 53
+ ],
+ [
+ 53,
+ 35,
+ 1,
+ 54,
+ 60,
+ 20
+ ]
+ ],
+ [
+ [
+ 17,
+ 37,
+ 31,
+ 32,
+ 63,
+ 50
+ ],
+ [
+ 12,
+ 2,
+ 32,
+ 9,
+ 47,
+ 17
+ ],
+ [
+ 3,
+ 57,
+ 56,
+ 50,
+ 33,
+ 38
+ ],
+ [
+ 43,
+ 42,
+ 19,
+ 52,
+ 8,
+ 17
+ ],
+ [
+ 61,
+ 39,
+ 27,
+ 12,
+ 15,
+ 57
+ ],
+ [
+ 6,
+ 33,
+ 36,
+ 44,
+ 29,
+ 61
+ ],
+ [
+ 2,
+ 41,
+ 42,
+ 15,
+ 52,
+ 5
+ ],
+ [
+ 36,
+ 37,
+ 15,
+ 53,
+ 18,
+ 62
+ ],
+ [
+ 50,
+ 58,
+ 3,
+ 5,
+ 16,
+ 4
+ ],
+ [
+ 16,
+ 22,
+ 55,
+ 32,
+ 41,
+ 26
+ ],
+ [
+ 3,
+ 41,
+ 56,
+ 45,
+ 38,
+ 34
+ ],
+ [
+ 21,
+ 53,
+ 50,
+ 34,
+ 38,
+ 35
+ ],
+ [
+ 35,
+ 11,
+ 37,
+ 16,
+ 53,
+ 42
+ ],
+ [
+ 9,
+ 15,
+ 54,
+ 38,
+ 12,
+ 52
+ ],
+ [
+ 56,
+ 2,
+ 24,
+ 23,
+ 9,
+ 5
+ ],
+ [
+ 2,
+ 44,
+ 9,
+ 53,
+ 35,
+ 54
+ ],
+ [
+ 27,
+ 46,
+ 1,
+ 5,
+ 60,
+ 50
+ ],
+ [
+ 51,
+ 61,
+ 50,
+ 54,
+ 33,
+ 44
+ ],
+ [
+ 2,
+ 38,
+ 19,
+ 9,
+ 5,
+ 32
+ ],
+ [
+ 32,
+ 28,
+ 6,
+ 15,
+ 0,
+ 33
+ ],
+ [
+ 26,
+ 53,
+ 27,
+ 13,
+ 5,
+ 4
+ ],
+ [
+ 34,
+ 40,
+ 45,
+ 55,
+ 62,
+ 53
+ ],
+ [
+ 41,
+ 5,
+ 46,
+ 4,
+ 45,
+ 3
+ ],
+ [
+ 20,
+ 4,
+ 22,
+ 47,
+ 59,
+ 58
+ ],
+ [
+ 62,
+ 36,
+ 25,
+ 28,
+ 53,
+ 33
+ ],
+ [
+ 22,
+ 33,
+ 7,
+ 3,
+ 54,
+ 0
+ ],
+ [
+ 58,
+ 27,
+ 52,
+ 48,
+ 45,
+ 17
+ ]
+ ],
+ [
+ [
+ 3,
+ 43,
+ 17,
+ 42,
+ 4,
+ 35
+ ],
+ [
+ 0,
+ 42,
+ 36,
+ 34,
+ 32,
+ 24
+ ],
+ [
+ 43,
+ 12,
+ 19,
+ 20,
+ 2,
+ 29
+ ],
+ [
+ 28,
+ 4,
+ 54,
+ 22,
+ 58,
+ 23
+ ],
+ [
+ 12,
+ 60,
+ 52,
+ 57,
+ 46,
+ 7
+ ],
+ [
+ 43,
+ 2,
+ 1,
+ 63,
+ 20,
+ 11
+ ],
+ [
+ 42,
+ 19,
+ 57,
+ 31,
+ 15,
+ 21
+ ],
+ [
+ 35,
+ 6,
+ 46,
+ 25,
+ 36,
+ 33
+ ],
+ [
+ 11,
+ 4,
+ 35,
+ 50,
+ 32,
+ 52
+ ],
+ [
+ 21,
+ 25,
+ 38,
+ 47,
+ 31,
+ 58
+ ],
+ [
+ 3,
+ 2,
+ 22,
+ 45,
+ 42,
+ 15
+ ],
+ [
+ 2,
+ 21,
+ 5,
+ 7,
+ 43,
+ 38
+ ],
+ [
+ 33,
+ 10,
+ 41,
+ 42,
+ 16,
+ 60
+ ],
+ [
+ 44,
+ 56,
+ 30,
+ 54,
+ 9,
+ 43
+ ],
+ [
+ 9,
+ 5,
+ 28,
+ 33,
+ 58,
+ 49
+ ],
+ [
+ 8,
+ 53,
+ 59,
+ 54,
+ 31,
+ 2
+ ],
+ [
+ 47,
+ 0,
+ 48,
+ 27,
+ 24,
+ 51
+ ],
+ [
+ 0,
+ 60,
+ 21,
+ 12,
+ 56,
+ 28
+ ],
+ [
+ 38,
+ 13,
+ 53,
+ 51,
+ 9,
+ 7
+ ],
+ [
+ 19,
+ 28,
+ 56,
+ 17,
+ 21,
+ 26
+ ],
+ [
+ 12,
+ 43,
+ 26,
+ 4,
+ 0,
+ 5
+ ],
+ [
+ 43,
+ 1,
+ 55,
+ 12,
+ 4,
+ 21
+ ],
+ [
+ 41,
+ 5,
+ 21,
+ 45,
+ 25,
+ 12
+ ],
+ [
+ 4,
+ 20,
+ 1,
+ 22,
+ 30,
+ 42
+ ],
+ [
+ 5,
+ 62,
+ 25,
+ 54,
+ 48,
+ 47
+ ],
+ [
+ 10,
+ 28,
+ 55,
+ 23,
+ 0,
+ 53
+ ],
+ [
+ 30,
+ 57,
+ 8,
+ 29,
+ 17,
+ 5
+ ]
+ ],
+ [
+ [
+ 11,
+ 16,
+ 31,
+ 0,
+ 35,
+ 46
+ ],
+ [
+ 13,
+ 49,
+ 31,
+ 34,
+ 16,
+ 50
+ ],
+ [
+ 36,
+ 13,
+ 53,
+ 4,
+ 27,
+ 18
+ ],
+ [
+ 24,
+ 7,
+ 29,
+ 26,
+ 12,
+ 32
+ ],
+ [
+ 17,
+ 35,
+ 2,
+ 44,
+ 10,
+ 48
+ ],
+ [
+ 8,
+ 7,
+ 23,
+ 5,
+ 51,
+ 26
+ ],
+ [
+ 58,
+ 15,
+ 61,
+ 29,
+ 38,
+ 62
+ ],
+ [
+ 20,
+ 9,
+ 42,
+ 35,
+ 3,
+ 54
+ ],
+ [
+ 4,
+ 47,
+ 25,
+ 11,
+ 1,
+ 52
+ ],
+ [
+ 54,
+ 55,
+ 25,
+ 38,
+ 21,
+ 27
+ ],
+ [
+ 1,
+ 60,
+ 14,
+ 59,
+ 22,
+ 30
+ ],
+ [
+ 51,
+ 31,
+ 5,
+ 25,
+ 14,
+ 52
+ ],
+ [
+ 34,
+ 2,
+ 10,
+ 26,
+ 52,
+ 47
+ ],
+ [
+ 40,
+ 57,
+ 13,
+ 54,
+ 9,
+ 6
+ ],
+ [
+ 8,
+ 44,
+ 58,
+ 5,
+ 1,
+ 16
+ ],
+ [
+ 8,
+ 53,
+ 59,
+ 25,
+ 52,
+ 24
+ ],
+ [
+ 13,
+ 47,
+ 45,
+ 0,
+ 42,
+ 37
+ ],
+ [
+ 23,
+ 44,
+ 55,
+ 38,
+ 7,
+ 33
+ ],
+ [
+ 53,
+ 38,
+ 11,
+ 1,
+ 8,
+ 24
+ ],
+ [
+ 15,
+ 38,
+ 14,
+ 28,
+ 19,
+ 0
+ ],
+ [
+ 41,
+ 27,
+ 8,
+ 42,
+ 57,
+ 40
+ ],
+ [
+ 11,
+ 62,
+ 55,
+ 16,
+ 41,
+ 10
+ ],
+ [
+ 31,
+ 20,
+ 46,
+ 34,
+ 37,
+ 41
+ ],
+ [
+ 47,
+ 4,
+ 55,
+ 33,
+ 49,
+ 22
+ ],
+ [
+ 11,
+ 45,
+ 47,
+ 48,
+ 54,
+ 20
+ ],
+ [
+ 50,
+ 23,
+ 10,
+ 18,
+ 3,
+ 2
+ ],
+ [
+ 17,
+ 31,
+ 36,
+ 5,
+ 4,
+ 20
+ ]
+ ],
+ [
+ [
+ 22,
+ 6,
+ 39,
+ 57,
+ 29,
+ 28
+ ],
+ [
+ 27,
+ 6,
+ 14,
+ 17,
+ 51,
+ 55
+ ],
+ [
+ 1,
+ 11,
+ 29,
+ 26,
+ 47,
+ 4
+ ],
+ [
+ 14,
+ 38,
+ 31,
+ 22,
+ 29,
+ 6
+ ],
+ [
+ 14,
+ 59,
+ 61,
+ 16,
+ 1,
+ 19
+ ],
+ [
+ 30,
+ 8,
+ 23,
+ 21,
+ 47,
+ 1
+ ],
+ [
+ 58,
+ 4,
+ 15,
+ 61,
+ 27,
+ 31
+ ],
+ [
+ 20,
+ 42,
+ 3,
+ 9,
+ 35,
+ 6
+ ],
+ [
+ 47,
+ 4,
+ 25,
+ 8,
+ 36,
+ 0
+ ],
+ [
+ 54,
+ 55,
+ 21,
+ 19,
+ 33,
+ 25
+ ],
+ [
+ 14,
+ 4,
+ 60,
+ 20,
+ 40,
+ 24
+ ],
+ [
+ 51,
+ 0,
+ 5,
+ 32,
+ 52,
+ 3
+ ],
+ [
+ 2,
+ 36,
+ 10,
+ 52,
+ 26,
+ 32
+ ],
+ [
+ 40,
+ 13,
+ 54,
+ 36,
+ 57,
+ 46
+ ],
+ [
+ 44,
+ 8,
+ 5,
+ 37,
+ 58,
+ 2
+ ],
+ [
+ 6,
+ 53,
+ 24,
+ 2,
+ 54,
+ 37
+ ],
+ [
+ 13,
+ 47,
+ 61,
+ 5,
+ 19,
+ 17
+ ],
+ [
+ 58,
+ 55,
+ 44,
+ 38,
+ 63,
+ 6
+ ],
+ [
+ 35,
+ 46,
+ 31,
+ 1,
+ 32,
+ 19
+ ],
+ [
+ 15,
+ 13,
+ 63,
+ 45,
+ 55,
+ 9
+ ],
+ [
+ 27,
+ 15,
+ 23,
+ 6,
+ 35,
+ 63
+ ],
+ [
+ 36,
+ 62,
+ 57,
+ 41,
+ 10,
+ 16
+ ],
+ [
+ 10,
+ 33,
+ 41,
+ 20,
+ 5,
+ 23
+ ],
+ [
+ 4,
+ 47,
+ 26,
+ 55,
+ 49,
+ 10
+ ],
+ [
+ 4,
+ 11,
+ 8,
+ 36,
+ 48,
+ 9
+ ],
+ [
+ 18,
+ 51,
+ 43,
+ 33,
+ 50,
+ 3
+ ],
+ [
+ 4,
+ 55,
+ 9,
+ 36,
+ 5,
+ 43
+ ]
+ ],
+ [
+ [
+ 48,
+ 38,
+ 47,
+ 63,
+ 7,
+ 42
+ ],
+ [
+ 3,
+ 26,
+ 10,
+ 6,
+ 2,
+ 42
+ ],
+ [
+ 39,
+ 44,
+ 6,
+ 45,
+ 7,
+ 8
+ ],
+ [
+ 60,
+ 31,
+ 22,
+ 0,
+ 54,
+ 45
+ ],
+ [
+ 6,
+ 59,
+ 14,
+ 16,
+ 37,
+ 44
+ ],
+ [
+ 59,
+ 30,
+ 8,
+ 47,
+ 60,
+ 17
+ ],
+ [
+ 44,
+ 56,
+ 4,
+ 13,
+ 15,
+ 9
+ ],
+ [
+ 12,
+ 24,
+ 20,
+ 61,
+ 58,
+ 28
+ ],
+ [
+ 47,
+ 33,
+ 8,
+ 4,
+ 36,
+ 42
+ ],
+ [
+ 40,
+ 54,
+ 55,
+ 33,
+ 21,
+ 51
+ ],
+ [
+ 8,
+ 14,
+ 22,
+ 39,
+ 31,
+ 7
+ ],
+ [
+ 34,
+ 59,
+ 14,
+ 5,
+ 52,
+ 0
+ ],
+ [
+ 36,
+ 10,
+ 52,
+ 26,
+ 44,
+ 59
+ ],
+ [
+ 59,
+ 44,
+ 13,
+ 1,
+ 36,
+ 22
+ ],
+ [
+ 24,
+ 5,
+ 46,
+ 2,
+ 44,
+ 38
+ ],
+ [
+ 34,
+ 53,
+ 6,
+ 40,
+ 30,
+ 2
+ ],
+ [
+ 17,
+ 29,
+ 50,
+ 5,
+ 47,
+ 27
+ ],
+ [
+ 48,
+ 38,
+ 57,
+ 63,
+ 32,
+ 60
+ ],
+ [
+ 42,
+ 7,
+ 49,
+ 46,
+ 32,
+ 1
+ ],
+ [
+ 59,
+ 1,
+ 23,
+ 14,
+ 10,
+ 16
+ ],
+ [
+ 21,
+ 35,
+ 7,
+ 53,
+ 6,
+ 17
+ ],
+ [
+ 8,
+ 24,
+ 60,
+ 13,
+ 55,
+ 14
+ ],
+ [
+ 33,
+ 5,
+ 51,
+ 15,
+ 3,
+ 23
+ ],
+ [
+ 4,
+ 9,
+ 51,
+ 26,
+ 41,
+ 28
+ ],
+ [
+ 60,
+ 4,
+ 19,
+ 47,
+ 40,
+ 48
+ ],
+ [
+ 19,
+ 3,
+ 26,
+ 21,
+ 53,
+ 8
+ ],
+ [
+ 53,
+ 1,
+ 35,
+ 54,
+ 60,
+ 20
+ ]
+ ],
+ [
+ [
+ 37,
+ 46,
+ 39,
+ 54,
+ 27,
+ 55
+ ],
+ [
+ 34,
+ 5,
+ 16,
+ 47,
+ 6,
+ 42
+ ],
+ [
+ 32,
+ 38,
+ 16,
+ 42,
+ 3,
+ 20
+ ],
+ [
+ 43,
+ 19,
+ 60,
+ 42,
+ 52,
+ 11
+ ],
+ [
+ 7,
+ 61,
+ 39,
+ 57,
+ 12,
+ 46
+ ],
+ [
+ 50,
+ 36,
+ 44,
+ 24,
+ 28,
+ 31
+ ],
+ [
+ 15,
+ 42,
+ 41,
+ 2,
+ 40,
+ 32
+ ],
+ [
+ 36,
+ 35,
+ 38,
+ 53,
+ 58,
+ 51
+ ],
+ [
+ 50,
+ 4,
+ 16,
+ 3,
+ 36,
+ 58
+ ],
+ [
+ 59,
+ 16,
+ 32,
+ 38,
+ 26,
+ 56
+ ],
+ [
+ 3,
+ 45,
+ 41,
+ 37,
+ 21,
+ 33
+ ],
+ [
+ 21,
+ 53,
+ 34,
+ 35,
+ 5,
+ 39
+ ],
+ [
+ 35,
+ 42,
+ 63,
+ 16,
+ 11,
+ 46
+ ],
+ [
+ 23,
+ 9,
+ 51,
+ 54,
+ 15,
+ 38
+ ],
+ [
+ 2,
+ 5,
+ 20,
+ 43,
+ 24,
+ 29
+ ],
+ [
+ 53,
+ 44,
+ 9,
+ 2,
+ 54,
+ 20
+ ],
+ [
+ 27,
+ 46,
+ 1,
+ 47,
+ 50,
+ 5
+ ],
+ [
+ 61,
+ 54,
+ 57,
+ 44,
+ 51,
+ 43
+ ],
+ [
+ 19,
+ 9,
+ 33,
+ 61,
+ 38,
+ 37
+ ],
+ [
+ 0,
+ 33,
+ 15,
+ 32,
+ 6,
+ 9
+ ],
+ [
+ 26,
+ 27,
+ 53,
+ 5,
+ 47,
+ 54
+ ],
+ [
+ 55,
+ 34,
+ 12,
+ 62,
+ 3,
+ 4
+ ],
+ [
+ 41,
+ 5,
+ 46,
+ 40,
+ 4,
+ 32
+ ],
+ [
+ 55,
+ 20,
+ 26,
+ 44,
+ 4,
+ 40
+ ],
+ [
+ 62,
+ 12,
+ 28,
+ 34,
+ 23,
+ 33
+ ],
+ [
+ 18,
+ 7,
+ 22,
+ 3,
+ 54,
+ 14
+ ],
+ [
+ 36,
+ 9,
+ 27,
+ 52,
+ 48,
+ 11
+ ]
+ ],
+ [
+ [
+ 46,
+ 37,
+ 61,
+ 18,
+ 36,
+ 63
+ ],
+ [
+ 22,
+ 34,
+ 28,
+ 59,
+ 24,
+ 51
+ ],
+ [
+ 32,
+ 15,
+ 17,
+ 60,
+ 38,
+ 20
+ ],
+ [
+ 28,
+ 4,
+ 58,
+ 16,
+ 30,
+ 35
+ ],
+ [
+ 7,
+ 36,
+ 9,
+ 57,
+ 33,
+ 23
+ ],
+ [
+ 43,
+ 63,
+ 2,
+ 30,
+ 11,
+ 19
+ ],
+ [
+ 19,
+ 42,
+ 57,
+ 15,
+ 3,
+ 22
+ ],
+ [
+ 35,
+ 46,
+ 47,
+ 6,
+ 58,
+ 8
+ ],
+ [
+ 32,
+ 4,
+ 37,
+ 36,
+ 35,
+ 57
+ ],
+ [
+ 21,
+ 47,
+ 55,
+ 58,
+ 38,
+ 54
+ ],
+ [
+ 3,
+ 42,
+ 2,
+ 38,
+ 5,
+ 33
+ ],
+ [
+ 43,
+ 18,
+ 2,
+ 21,
+ 5,
+ 49
+ ],
+ [
+ 10,
+ 49,
+ 24,
+ 32,
+ 25,
+ 42
+ ],
+ [
+ 30,
+ 21,
+ 63,
+ 51,
+ 49,
+ 16
+ ],
+ [
+ 62,
+ 2,
+ 13,
+ 5,
+ 29,
+ 40
+ ],
+ [
+ 53,
+ 2,
+ 9,
+ 63,
+ 54,
+ 46
+ ],
+ [
+ 27,
+ 0,
+ 63,
+ 47,
+ 5,
+ 14
+ ],
+ [
+ 60,
+ 57,
+ 43,
+ 44,
+ 0,
+ 50
+ ],
+ [
+ 38,
+ 9,
+ 61,
+ 33,
+ 60,
+ 13
+ ],
+ [
+ 25,
+ 0,
+ 15,
+ 34,
+ 35,
+ 11
+ ],
+ [
+ 26,
+ 54,
+ 28,
+ 47,
+ 53,
+ 37
+ ],
+ [
+ 55,
+ 1,
+ 34,
+ 4,
+ 21,
+ 22
+ ],
+ [
+ 41,
+ 5,
+ 40,
+ 21,
+ 17,
+ 23
+ ],
+ [
+ 25,
+ 43,
+ 52,
+ 26,
+ 4,
+ 55
+ ],
+ [
+ 52,
+ 29,
+ 35,
+ 17,
+ 45,
+ 60
+ ],
+ [
+ 38,
+ 10,
+ 15,
+ 7,
+ 50,
+ 3
+ ],
+ [
+ 23,
+ 6,
+ 19,
+ 56,
+ 41,
+ 15
+ ]
+ ],
+ [
+ [
+ 44,
+ 14,
+ 20,
+ 47,
+ 19,
+ 56
+ ],
+ [
+ 28,
+ 34,
+ 2,
+ 56,
+ 0,
+ 11
+ ],
+ [
+ 46,
+ 15,
+ 61,
+ 14,
+ 22,
+ 60
+ ],
+ [
+ 15,
+ 50,
+ 36,
+ 47,
+ 25,
+ 21
+ ],
+ [
+ 25,
+ 12,
+ 13,
+ 36,
+ 23,
+ 57
+ ],
+ [
+ 22,
+ 43,
+ 1,
+ 37,
+ 36,
+ 30
+ ],
+ [
+ 28,
+ 42,
+ 19,
+ 31,
+ 14,
+ 21
+ ],
+ [
+ 35,
+ 51,
+ 47,
+ 33,
+ 7,
+ 46
+ ],
+ [
+ 32,
+ 4,
+ 16,
+ 11,
+ 35,
+ 19
+ ],
+ [
+ 21,
+ 38,
+ 16,
+ 47,
+ 62,
+ 15
+ ],
+ [
+ 3,
+ 45,
+ 39,
+ 50,
+ 21,
+ 2
+ ],
+ [
+ 21,
+ 5,
+ 43,
+ 56,
+ 53,
+ 52
+ ],
+ [
+ 10,
+ 41,
+ 33,
+ 11,
+ 63,
+ 37
+ ],
+ [
+ 56,
+ 16,
+ 9,
+ 63,
+ 33,
+ 45
+ ],
+ [
+ 58,
+ 52,
+ 2,
+ 5,
+ 30,
+ 56
+ ],
+ [
+ 53,
+ 8,
+ 59,
+ 55,
+ 2,
+ 50
+ ],
+ [
+ 0,
+ 47,
+ 51,
+ 45,
+ 37,
+ 14
+ ],
+ [
+ 0,
+ 43,
+ 21,
+ 12,
+ 60,
+ 62
+ ],
+ [
+ 38,
+ 53,
+ 60,
+ 36,
+ 34,
+ 9
+ ],
+ [
+ 27,
+ 9,
+ 56,
+ 0,
+ 53,
+ 11
+ ],
+ [
+ 28,
+ 12,
+ 43,
+ 54,
+ 5,
+ 62
+ ],
+ [
+ 50,
+ 55,
+ 34,
+ 16,
+ 4,
+ 21
+ ],
+ [
+ 4,
+ 21,
+ 5,
+ 40,
+ 32,
+ 54
+ ],
+ [
+ 25,
+ 57,
+ 49,
+ 1,
+ 44,
+ 43
+ ],
+ [
+ 5,
+ 35,
+ 25,
+ 42,
+ 2,
+ 22
+ ],
+ [
+ 28,
+ 15,
+ 55,
+ 38,
+ 35,
+ 37
+ ],
+ [
+ 19,
+ 30,
+ 6,
+ 57,
+ 39,
+ 51
+ ]
+ ],
+ [
+ [
+ 11,
+ 46,
+ 0,
+ 49,
+ 31,
+ 16
+ ],
+ [
+ 13,
+ 49,
+ 50,
+ 16,
+ 31,
+ 19
+ ],
+ [
+ 36,
+ 13,
+ 27,
+ 34,
+ 4,
+ 52
+ ],
+ [
+ 24,
+ 32,
+ 7,
+ 59,
+ 13,
+ 15
+ ],
+ [
+ 17,
+ 35,
+ 2,
+ 44,
+ 10,
+ 63
+ ],
+ [
+ 8,
+ 7,
+ 23,
+ 26,
+ 56,
+ 42
+ ],
+ [
+ 58,
+ 61,
+ 29,
+ 38,
+ 62,
+ 50
+ ],
+ [
+ 20,
+ 42,
+ 3,
+ 35,
+ 61,
+ 47
+ ],
+ [
+ 4,
+ 10,
+ 7,
+ 25,
+ 47,
+ 34
+ ],
+ [
+ 54,
+ 55,
+ 18,
+ 38,
+ 27,
+ 28
+ ],
+ [
+ 60,
+ 1,
+ 14,
+ 59,
+ 3,
+ 30
+ ],
+ [
+ 51,
+ 31,
+ 5,
+ 21,
+ 25,
+ 52
+ ],
+ [
+ 34,
+ 10,
+ 2,
+ 26,
+ 11,
+ 47
+ ],
+ [
+ 16,
+ 40,
+ 23,
+ 34,
+ 9,
+ 33
+ ],
+ [
+ 34,
+ 8,
+ 1,
+ 31,
+ 5,
+ 32
+ ],
+ [
+ 25,
+ 59,
+ 8,
+ 52,
+ 53,
+ 27
+ ],
+ [
+ 13,
+ 47,
+ 45,
+ 48,
+ 42,
+ 0
+ ],
+ [
+ 4,
+ 23,
+ 53,
+ 10,
+ 3,
+ 25
+ ],
+ [
+ 53,
+ 8,
+ 38,
+ 11,
+ 24,
+ 63
+ ],
+ [
+ 38,
+ 15,
+ 0,
+ 11,
+ 21,
+ 60
+ ],
+ [
+ 27,
+ 23,
+ 62,
+ 8,
+ 41,
+ 42
+ ],
+ [
+ 55,
+ 11,
+ 10,
+ 16,
+ 41,
+ 21
+ ],
+ [
+ 31,
+ 21,
+ 54,
+ 34,
+ 41,
+ 46
+ ],
+ [
+ 55,
+ 25,
+ 22,
+ 33,
+ 44,
+ 47
+ ],
+ [
+ 47,
+ 45,
+ 20,
+ 35,
+ 12,
+ 42
+ ],
+ [
+ 50,
+ 10,
+ 47,
+ 53,
+ 35,
+ 57
+ ],
+ [
+ 17,
+ 31,
+ 5,
+ 37,
+ 36,
+ 20
+ ]
+ ],
+ [
+ [
+ 22,
+ 6,
+ 29,
+ 39,
+ 44,
+ 57
+ ],
+ [
+ 27,
+ 6,
+ 14,
+ 17,
+ 55,
+ 57
+ ],
+ [
+ 11,
+ 1,
+ 29,
+ 26,
+ 47,
+ 4
+ ],
+ [
+ 14,
+ 38,
+ 31,
+ 22,
+ 36,
+ 29
+ ],
+ [
+ 14,
+ 61,
+ 59,
+ 16,
+ 44,
+ 63
+ ],
+ [
+ 30,
+ 8,
+ 47,
+ 23,
+ 21,
+ 0
+ ],
+ [
+ 58,
+ 4,
+ 54,
+ 61,
+ 15,
+ 62
+ ],
+ [
+ 20,
+ 42,
+ 58,
+ 3,
+ 35,
+ 0
+ ],
+ [
+ 10,
+ 47,
+ 4,
+ 8,
+ 41,
+ 19
+ ],
+ [
+ 54,
+ 55,
+ 33,
+ 11,
+ 38,
+ 21
+ ],
+ [
+ 14,
+ 1,
+ 4,
+ 60,
+ 20,
+ 40
+ ],
+ [
+ 51,
+ 5,
+ 14,
+ 0,
+ 32,
+ 62
+ ],
+ [
+ 2,
+ 10,
+ 36,
+ 49,
+ 32,
+ 52
+ ],
+ [
+ 13,
+ 40,
+ 54,
+ 36,
+ 22,
+ 11
+ ],
+ [
+ 2,
+ 44,
+ 37,
+ 58,
+ 5,
+ 8
+ ],
+ [
+ 6,
+ 24,
+ 2,
+ 53,
+ 19,
+ 52
+ ],
+ [
+ 13,
+ 47,
+ 61,
+ 5,
+ 50,
+ 30
+ ],
+ [
+ 55,
+ 58,
+ 4,
+ 63,
+ 22,
+ 33
+ ],
+ [
+ 35,
+ 32,
+ 46,
+ 14,
+ 7,
+ 58
+ ],
+ [
+ 15,
+ 13,
+ 45,
+ 0,
+ 60,
+ 51
+ ],
+ [
+ 15,
+ 27,
+ 23,
+ 50,
+ 6,
+ 51
+ ],
+ [
+ 57,
+ 36,
+ 10,
+ 55,
+ 14,
+ 16
+ ],
+ [
+ 10,
+ 23,
+ 33,
+ 41,
+ 54,
+ 30
+ ],
+ [
+ 33,
+ 26,
+ 4,
+ 10,
+ 49,
+ 55
+ ],
+ [
+ 4,
+ 8,
+ 33,
+ 9,
+ 20,
+ 48
+ ],
+ [
+ 43,
+ 6,
+ 3,
+ 18,
+ 50,
+ 38
+ ],
+ [
+ 4,
+ 55,
+ 36,
+ 43,
+ 46,
+ 5
+ ]
+ ],
+ [
+ [
+ 48,
+ 38,
+ 47,
+ 63,
+ 7,
+ 42
+ ],
+ [
+ 3,
+ 26,
+ 10,
+ 6,
+ 2,
+ 35
+ ],
+ [
+ 39,
+ 44,
+ 6,
+ 7,
+ 45,
+ 8
+ ],
+ [
+ 60,
+ 31,
+ 22,
+ 0,
+ 54,
+ 27
+ ],
+ [
+ 6,
+ 59,
+ 16,
+ 14,
+ 37,
+ 44
+ ],
+ [
+ 59,
+ 8,
+ 30,
+ 47,
+ 48,
+ 3
+ ],
+ [
+ 44,
+ 54,
+ 4,
+ 56,
+ 15,
+ 13
+ ],
+ [
+ 12,
+ 24,
+ 20,
+ 58,
+ 31,
+ 61
+ ],
+ [
+ 47,
+ 33,
+ 10,
+ 8,
+ 4,
+ 36
+ ],
+ [
+ 54,
+ 40,
+ 55,
+ 33,
+ 11,
+ 51
+ ],
+ [
+ 8,
+ 14,
+ 23,
+ 29,
+ 22,
+ 31
+ ],
+ [
+ 34,
+ 59,
+ 14,
+ 5,
+ 57,
+ 50
+ ],
+ [
+ 44,
+ 36,
+ 10,
+ 59,
+ 52,
+ 9
+ ],
+ [
+ 59,
+ 44,
+ 13,
+ 9,
+ 1,
+ 54
+ ],
+ [
+ 24,
+ 2,
+ 46,
+ 44,
+ 38,
+ 47
+ ],
+ [
+ 34,
+ 53,
+ 6,
+ 2,
+ 57,
+ 40
+ ],
+ [
+ 17,
+ 29,
+ 47,
+ 5,
+ 50,
+ 34
+ ],
+ [
+ 48,
+ 63,
+ 57,
+ 38,
+ 60,
+ 32
+ ],
+ [
+ 42,
+ 7,
+ 49,
+ 32,
+ 46,
+ 1
+ ],
+ [
+ 1,
+ 59,
+ 23,
+ 14,
+ 16,
+ 10
+ ],
+ [
+ 21,
+ 35,
+ 7,
+ 53,
+ 6,
+ 17
+ ],
+ [
+ 8,
+ 24,
+ 60,
+ 14,
+ 13,
+ 55
+ ],
+ [
+ 33,
+ 5,
+ 51,
+ 15,
+ 23,
+ 3
+ ],
+ [
+ 9,
+ 4,
+ 51,
+ 26,
+ 41,
+ 8
+ ],
+ [
+ 4,
+ 60,
+ 19,
+ 47,
+ 40,
+ 9
+ ],
+ [
+ 19,
+ 3,
+ 26,
+ 21,
+ 17,
+ 8
+ ],
+ [
+ 1,
+ 35,
+ 53,
+ 54,
+ 60,
+ 20
+ ]
+ ],
+ [
+ [
+ 12,
+ 41,
+ 14,
+ 62,
+ 24,
+ 10
+ ],
+ [
+ 10,
+ 53,
+ 39,
+ 35,
+ 41,
+ 58
+ ],
+ [
+ 33,
+ 32,
+ 50,
+ 31,
+ 3,
+ 34
+ ],
+ [
+ 43,
+ 10,
+ 42,
+ 11,
+ 17,
+ 47
+ ],
+ [
+ 42,
+ 12,
+ 11,
+ 19,
+ 58,
+ 54
+ ],
+ [
+ 36,
+ 50,
+ 55,
+ 61,
+ 25,
+ 56
+ ],
+ [
+ 41,
+ 22,
+ 16,
+ 52,
+ 2,
+ 15
+ ],
+ [
+ 26,
+ 36,
+ 62,
+ 53,
+ 15,
+ 51
+ ],
+ [
+ 58,
+ 16,
+ 5,
+ 53,
+ 3,
+ 49
+ ],
+ [
+ 32,
+ 46,
+ 26,
+ 45,
+ 16,
+ 62
+ ],
+ [
+ 41,
+ 45,
+ 56,
+ 49,
+ 11,
+ 3
+ ],
+ [
+ 52,
+ 34,
+ 35,
+ 50,
+ 21,
+ 53
+ ],
+ [
+ 59,
+ 53,
+ 46,
+ 30,
+ 39,
+ 37
+ ],
+ [
+ 20,
+ 9,
+ 52,
+ 2,
+ 7,
+ 33
+ ],
+ [
+ 20,
+ 50,
+ 24,
+ 29,
+ 23,
+ 2
+ ],
+ [
+ 53,
+ 2,
+ 44,
+ 41,
+ 9,
+ 13
+ ],
+ [
+ 47,
+ 27,
+ 1,
+ 5,
+ 45,
+ 46
+ ],
+ [
+ 61,
+ 7,
+ 51,
+ 30,
+ 35,
+ 9
+ ],
+ [
+ 19,
+ 34,
+ 32,
+ 17,
+ 2,
+ 14
+ ],
+ [
+ 15,
+ 32,
+ 6,
+ 45,
+ 9,
+ 11
+ ],
+ [
+ 35,
+ 27,
+ 6,
+ 53,
+ 13,
+ 60
+ ],
+ [
+ 26,
+ 62,
+ 16,
+ 28,
+ 41,
+ 14
+ ],
+ [
+ 5,
+ 20,
+ 46,
+ 37,
+ 11,
+ 55
+ ],
+ [
+ 55,
+ 47,
+ 4,
+ 16,
+ 14,
+ 27
+ ],
+ [
+ 36,
+ 11,
+ 27,
+ 62,
+ 33,
+ 48
+ ],
+ [
+ 33,
+ 54,
+ 7,
+ 3,
+ 12,
+ 32
+ ],
+ [
+ 58,
+ 52,
+ 27,
+ 26,
+ 48,
+ 38
+ ]
+ ],
+ [
+ [
+ 6,
+ 52,
+ 19,
+ 63,
+ 38,
+ 46
+ ],
+ [
+ 8,
+ 42,
+ 4,
+ 47,
+ 57,
+ 56
+ ],
+ [
+ 31,
+ 46,
+ 32,
+ 4,
+ 14,
+ 10
+ ],
+ [
+ 28,
+ 27,
+ 4,
+ 37,
+ 58,
+ 20
+ ],
+ [
+ 57,
+ 59,
+ 60,
+ 62,
+ 22,
+ 14
+ ],
+ [
+ 30,
+ 2,
+ 9,
+ 57,
+ 11,
+ 43
+ ],
+ [
+ 20,
+ 19,
+ 57,
+ 42,
+ 27,
+ 51
+ ],
+ [
+ 46,
+ 35,
+ 47,
+ 7,
+ 32,
+ 0
+ ],
+ [
+ 12,
+ 4,
+ 35,
+ 10,
+ 47,
+ 50
+ ],
+ [
+ 15,
+ 47,
+ 54,
+ 25,
+ 38,
+ 51
+ ],
+ [
+ 18,
+ 42,
+ 5,
+ 15,
+ 38,
+ 61
+ ],
+ [
+ 18,
+ 22,
+ 8,
+ 16,
+ 5,
+ 7
+ ],
+ [
+ 5,
+ 3,
+ 24,
+ 35,
+ 4,
+ 30
+ ],
+ [
+ 17,
+ 13,
+ 55,
+ 20,
+ 36,
+ 22
+ ],
+ [
+ 22,
+ 10,
+ 21,
+ 54,
+ 6,
+ 47
+ ],
+ [
+ 30,
+ 53,
+ 6,
+ 19,
+ 2,
+ 54
+ ],
+ [
+ 22,
+ 26,
+ 5,
+ 7,
+ 47,
+ 21
+ ],
+ [
+ 4,
+ 41,
+ 13,
+ 46,
+ 55,
+ 43
+ ],
+ [
+ 17,
+ 56,
+ 32,
+ 45,
+ 14,
+ 6
+ ],
+ [
+ 5,
+ 49,
+ 53,
+ 28,
+ 34,
+ 60
+ ],
+ [
+ 42,
+ 55,
+ 57,
+ 17,
+ 28,
+ 22
+ ],
+ [
+ 40,
+ 23,
+ 28,
+ 57,
+ 16,
+ 21
+ ],
+ [
+ 51,
+ 21,
+ 35,
+ 24,
+ 44,
+ 10
+ ],
+ [
+ 38,
+ 25,
+ 4,
+ 14,
+ 62,
+ 31
+ ],
+ [
+ 31,
+ 50,
+ 13,
+ 56,
+ 8,
+ 39
+ ],
+ [
+ 6,
+ 51,
+ 55,
+ 8,
+ 0,
+ 21
+ ],
+ [
+ 29,
+ 46,
+ 55,
+ 18,
+ 37,
+ 50
+ ]
+ ],
+ [
+ [
+ 46,
+ 31,
+ 35,
+ 49,
+ 14,
+ 21
+ ],
+ [
+ 13,
+ 16,
+ 50,
+ 31,
+ 33,
+ 8
+ ],
+ [
+ 13,
+ 36,
+ 27,
+ 52,
+ 3,
+ 19
+ ],
+ [
+ 9,
+ 24,
+ 29,
+ 12,
+ 55,
+ 61
+ ],
+ [
+ 35,
+ 17,
+ 2,
+ 10,
+ 41,
+ 48
+ ],
+ [
+ 51,
+ 23,
+ 30,
+ 57,
+ 52,
+ 17
+ ],
+ [
+ 34,
+ 48,
+ 0,
+ 38,
+ 27,
+ 51
+ ],
+ [
+ 47,
+ 32,
+ 2,
+ 35,
+ 58,
+ 55
+ ],
+ [
+ 15,
+ 12,
+ 35,
+ 52,
+ 45,
+ 4
+ ],
+ [
+ 36,
+ 15,
+ 4,
+ 59,
+ 11,
+ 54
+ ],
+ [
+ 60,
+ 5,
+ 59,
+ 29,
+ 50,
+ 18
+ ],
+ [
+ 18,
+ 8,
+ 31,
+ 23,
+ 24,
+ 3
+ ],
+ [
+ 34,
+ 4,
+ 5,
+ 58,
+ 14,
+ 27
+ ],
+ [
+ 17,
+ 57,
+ 36,
+ 55,
+ 51,
+ 6
+ ],
+ [
+ 10,
+ 45,
+ 8,
+ 54,
+ 22,
+ 47
+ ],
+ [
+ 30,
+ 50,
+ 14,
+ 6,
+ 16,
+ 37
+ ],
+ [
+ 26,
+ 0,
+ 22,
+ 5,
+ 13,
+ 4
+ ],
+ [
+ 4,
+ 46,
+ 23,
+ 43,
+ 60,
+ 34
+ ],
+ [
+ 45,
+ 8,
+ 56,
+ 62,
+ 17,
+ 51
+ ],
+ [
+ 4,
+ 38,
+ 53,
+ 15,
+ 58,
+ 5
+ ],
+ [
+ 57,
+ 24,
+ 34,
+ 15,
+ 55,
+ 42
+ ],
+ [
+ 23,
+ 37,
+ 40,
+ 41,
+ 54,
+ 11
+ ],
+ [
+ 48,
+ 29,
+ 51,
+ 31,
+ 34,
+ 9
+ ],
+ [
+ 62,
+ 14,
+ 18,
+ 31,
+ 4,
+ 59
+ ],
+ [
+ 50,
+ 31,
+ 49,
+ 45,
+ 56,
+ 53
+ ],
+ [
+ 51,
+ 50,
+ 56,
+ 8,
+ 12,
+ 52
+ ],
+ [
+ 17,
+ 37,
+ 5,
+ 39,
+ 44,
+ 20
+ ]
+ ],
+ [
+ [
+ 45,
+ 13,
+ 63,
+ 37,
+ 38,
+ 56
+ ],
+ [
+ 63,
+ 6,
+ 12,
+ 18,
+ 27,
+ 51
+ ],
+ [
+ 3,
+ 21,
+ 4,
+ 48,
+ 17,
+ 27
+ ],
+ [
+ 14,
+ 55,
+ 9,
+ 37,
+ 29,
+ 26
+ ],
+ [
+ 35,
+ 57,
+ 2,
+ 13,
+ 41,
+ 10
+ ],
+ [
+ 39,
+ 30,
+ 57,
+ 13,
+ 53,
+ 23
+ ],
+ [
+ 34,
+ 48,
+ 38,
+ 27,
+ 56,
+ 51
+ ],
+ [
+ 47,
+ 35,
+ 46,
+ 2,
+ 32,
+ 7
+ ],
+ [
+ 12,
+ 56,
+ 4,
+ 35,
+ 52,
+ 50
+ ],
+ [
+ 15,
+ 36,
+ 14,
+ 30,
+ 47,
+ 48
+ ],
+ [
+ 5,
+ 60,
+ 59,
+ 22,
+ 15,
+ 18
+ ],
+ [
+ 8,
+ 22,
+ 23,
+ 47,
+ 5,
+ 26
+ ],
+ [
+ 4,
+ 5,
+ 30,
+ 58,
+ 26,
+ 8
+ ],
+ [
+ 17,
+ 29,
+ 13,
+ 50,
+ 14,
+ 34
+ ],
+ [
+ 54,
+ 10,
+ 41,
+ 14,
+ 21,
+ 5
+ ],
+ [
+ 50,
+ 6,
+ 54,
+ 30,
+ 16,
+ 56
+ ],
+ [
+ 0,
+ 47,
+ 7,
+ 4,
+ 5,
+ 34
+ ],
+ [
+ 4,
+ 46,
+ 23,
+ 41,
+ 59,
+ 55
+ ],
+ [
+ 62,
+ 45,
+ 1,
+ 47,
+ 57,
+ 32
+ ],
+ [
+ 4,
+ 53,
+ 15,
+ 50,
+ 60,
+ 1
+ ],
+ [
+ 34,
+ 57,
+ 24,
+ 31,
+ 15,
+ 53
+ ],
+ [
+ 23,
+ 41,
+ 45,
+ 12,
+ 57,
+ 14
+ ],
+ [
+ 51,
+ 29,
+ 49,
+ 48,
+ 18,
+ 33
+ ],
+ [
+ 14,
+ 4,
+ 62,
+ 26,
+ 18,
+ 28
+ ],
+ [
+ 50,
+ 49,
+ 7,
+ 24,
+ 9,
+ 48
+ ],
+ [
+ 56,
+ 20,
+ 51,
+ 3,
+ 14,
+ 26
+ ],
+ [
+ 55,
+ 37,
+ 50,
+ 14,
+ 42,
+ 20
+ ]
+ ],
+ [
+ [
+ 51,
+ 43,
+ 27,
+ 30,
+ 5,
+ 12
+ ],
+ [
+ 24,
+ 16,
+ 48,
+ 15,
+ 7,
+ 30
+ ],
+ [
+ 26,
+ 21,
+ 50,
+ 52,
+ 4,
+ 56
+ ],
+ [
+ 19,
+ 17,
+ 2,
+ 14,
+ 57,
+ 22
+ ],
+ [
+ 3,
+ 35,
+ 37,
+ 45,
+ 1,
+ 6
+ ],
+ [
+ 13,
+ 30,
+ 23,
+ 39,
+ 57,
+ 60
+ ],
+ [
+ 9,
+ 34,
+ 48,
+ 17,
+ 27,
+ 26
+ ],
+ [
+ 47,
+ 32,
+ 2,
+ 59,
+ 28,
+ 57
+ ],
+ [
+ 18,
+ 12,
+ 31,
+ 20,
+ 52,
+ 4
+ ],
+ [
+ 14,
+ 15,
+ 36,
+ 30,
+ 58,
+ 22
+ ],
+ [
+ 5,
+ 4,
+ 26,
+ 19,
+ 22,
+ 59
+ ],
+ [
+ 22,
+ 8,
+ 34,
+ 26,
+ 42,
+ 60
+ ],
+ [
+ 20,
+ 5,
+ 4,
+ 52,
+ 30,
+ 33
+ ],
+ [
+ 34,
+ 17,
+ 2,
+ 5,
+ 39,
+ 20
+ ],
+ [
+ 54,
+ 61,
+ 30,
+ 14,
+ 25,
+ 20
+ ],
+ [
+ 56,
+ 41,
+ 2,
+ 5,
+ 54,
+ 44
+ ],
+ [
+ 12,
+ 34,
+ 47,
+ 7,
+ 54,
+ 59
+ ],
+ [
+ 9,
+ 11,
+ 2,
+ 43,
+ 33,
+ 50
+ ],
+ [
+ 8,
+ 10,
+ 19,
+ 20,
+ 32,
+ 18
+ ],
+ [
+ 17,
+ 6,
+ 38,
+ 29,
+ 49,
+ 41
+ ],
+ [
+ 38,
+ 45,
+ 52,
+ 57,
+ 63,
+ 27
+ ],
+ [
+ 26,
+ 28,
+ 3,
+ 40,
+ 5,
+ 47
+ ],
+ [
+ 60,
+ 49,
+ 35,
+ 34,
+ 38,
+ 55
+ ],
+ [
+ 61,
+ 24,
+ 4,
+ 55,
+ 45,
+ 16
+ ],
+ [
+ 46,
+ 11,
+ 27,
+ 0,
+ 56,
+ 48
+ ],
+ [
+ 60,
+ 45,
+ 44,
+ 25,
+ 32,
+ 3
+ ],
+ [
+ 11,
+ 38,
+ 52,
+ 48,
+ 9,
+ 21
+ ]
+ ],
+ [
+ [
+ 22,
+ 19,
+ 46,
+ 31,
+ 3,
+ 23
+ ],
+ [
+ 32,
+ 62,
+ 15,
+ 54,
+ 10,
+ 55
+ ],
+ [
+ 47,
+ 30,
+ 38,
+ 5,
+ 7,
+ 60
+ ],
+ [
+ 15,
+ 13,
+ 1,
+ 8,
+ 43,
+ 25
+ ],
+ [
+ 13,
+ 59,
+ 5,
+ 6,
+ 62,
+ 52
+ ],
+ [
+ 27,
+ 63,
+ 62,
+ 45,
+ 12,
+ 56
+ ],
+ [
+ 50,
+ 9,
+ 8,
+ 51,
+ 48,
+ 18
+ ],
+ [
+ 59,
+ 57,
+ 2,
+ 28,
+ 61,
+ 6
+ ],
+ [
+ 18,
+ 59,
+ 6,
+ 52,
+ 39,
+ 57
+ ],
+ [
+ 14,
+ 23,
+ 11,
+ 36,
+ 15,
+ 32
+ ],
+ [
+ 26,
+ 5,
+ 42,
+ 25,
+ 22,
+ 23
+ ],
+ [
+ 33,
+ 22,
+ 55,
+ 28,
+ 16,
+ 24
+ ],
+ [
+ 6,
+ 20,
+ 33,
+ 14,
+ 52,
+ 15
+ ],
+ [
+ 17,
+ 6,
+ 5,
+ 39,
+ 2,
+ 34
+ ],
+ [
+ 54,
+ 62,
+ 25,
+ 61,
+ 21,
+ 14
+ ],
+ [
+ 41,
+ 45,
+ 14,
+ 5,
+ 2,
+ 54
+ ],
+ [
+ 47,
+ 34,
+ 10,
+ 31,
+ 5,
+ 41
+ ],
+ [
+ 9,
+ 2,
+ 36,
+ 6,
+ 43,
+ 38
+ ],
+ [
+ 57,
+ 20,
+ 41,
+ 10,
+ 32,
+ 18
+ ],
+ [
+ 47,
+ 6,
+ 49,
+ 15,
+ 34,
+ 7
+ ],
+ [
+ 3,
+ 57,
+ 44,
+ 38,
+ 50,
+ 53
+ ],
+ [
+ 47,
+ 28,
+ 57,
+ 26,
+ 19,
+ 22
+ ],
+ [
+ 11,
+ 1,
+ 18,
+ 5,
+ 46,
+ 22
+ ],
+ [
+ 9,
+ 55,
+ 4,
+ 3,
+ 25,
+ 10
+ ],
+ [
+ 45,
+ 11,
+ 0,
+ 63,
+ 48,
+ 57
+ ],
+ [
+ 42,
+ 10,
+ 3,
+ 43,
+ 49,
+ 0
+ ],
+ [
+ 38,
+ 15,
+ 17,
+ 41,
+ 10,
+ 3
+ ]
+ ],
+ [
+ [
+ 19,
+ 18,
+ 51,
+ 25,
+ 60,
+ 55
+ ],
+ [
+ 56,
+ 27,
+ 61,
+ 42,
+ 55,
+ 23
+ ],
+ [
+ 32,
+ 39,
+ 37,
+ 46,
+ 20,
+ 52
+ ],
+ [
+ 41,
+ 21,
+ 37,
+ 13,
+ 57,
+ 2
+ ],
+ [
+ 10,
+ 9,
+ 3,
+ 46,
+ 58,
+ 32
+ ],
+ [
+ 62,
+ 57,
+ 27,
+ 43,
+ 2,
+ 53
+ ],
+ [
+ 50,
+ 21,
+ 19,
+ 48,
+ 15,
+ 10
+ ],
+ [
+ 14,
+ 21,
+ 54,
+ 8,
+ 57,
+ 31
+ ],
+ [
+ 6,
+ 18,
+ 4,
+ 52,
+ 39,
+ 24
+ ],
+ [
+ 32,
+ 38,
+ 23,
+ 53,
+ 17,
+ 20
+ ],
+ [
+ 26,
+ 25,
+ 22,
+ 5,
+ 33,
+ 42
+ ],
+ [
+ 5,
+ 22,
+ 16,
+ 61,
+ 9,
+ 55
+ ],
+ [
+ 49,
+ 33,
+ 30,
+ 25,
+ 20,
+ 4
+ ],
+ [
+ 18,
+ 17,
+ 25,
+ 63,
+ 39,
+ 11
+ ],
+ [
+ 62,
+ 54,
+ 10,
+ 5,
+ 58,
+ 37
+ ],
+ [
+ 42,
+ 41,
+ 5,
+ 24,
+ 37,
+ 53
+ ],
+ [
+ 47,
+ 34,
+ 10,
+ 27,
+ 33,
+ 22
+ ],
+ [
+ 60,
+ 43,
+ 44,
+ 52,
+ 7,
+ 37
+ ],
+ [
+ 12,
+ 39,
+ 32,
+ 38,
+ 1,
+ 20
+ ],
+ [
+ 34,
+ 52,
+ 49,
+ 15,
+ 40,
+ 63
+ ],
+ [
+ 28,
+ 39,
+ 3,
+ 26,
+ 22,
+ 30
+ ],
+ [
+ 29,
+ 1,
+ 40,
+ 22,
+ 19,
+ 63
+ ],
+ [
+ 41,
+ 53,
+ 5,
+ 4,
+ 33,
+ 21
+ ],
+ [
+ 25,
+ 38,
+ 4,
+ 34,
+ 51,
+ 41
+ ],
+ [
+ 29,
+ 52,
+ 48,
+ 47,
+ 33,
+ 20
+ ],
+ [
+ 50,
+ 3,
+ 16,
+ 53,
+ 38,
+ 15
+ ],
+ [
+ 19,
+ 62,
+ 6,
+ 23,
+ 36,
+ 10
+ ]
+ ],
+ [
+ [
+ 5,
+ 14,
+ 17,
+ 57,
+ 10,
+ 27
+ ],
+ [
+ 43,
+ 9,
+ 56,
+ 1,
+ 14,
+ 33
+ ],
+ [
+ 63,
+ 35,
+ 43,
+ 1,
+ 10,
+ 27
+ ],
+ [
+ 51,
+ 50,
+ 57,
+ 41,
+ 20,
+ 54
+ ],
+ [
+ 11,
+ 43,
+ 40,
+ 33,
+ 30,
+ 54
+ ],
+ [
+ 27,
+ 53,
+ 63,
+ 30,
+ 15,
+ 58
+ ],
+ [
+ 51,
+ 48,
+ 21,
+ 57,
+ 9,
+ 50
+ ],
+ [
+ 21,
+ 59,
+ 14,
+ 52,
+ 7,
+ 28
+ ],
+ [
+ 6,
+ 18,
+ 4,
+ 0,
+ 12,
+ 59
+ ],
+ [
+ 14,
+ 53,
+ 30,
+ 42,
+ 4,
+ 36
+ ],
+ [
+ 26,
+ 42,
+ 5,
+ 22,
+ 19,
+ 58
+ ],
+ [
+ 22,
+ 5,
+ 34,
+ 38,
+ 52,
+ 29
+ ],
+ [
+ 49,
+ 33,
+ 0,
+ 20,
+ 26,
+ 8
+ ],
+ [
+ 17,
+ 2,
+ 39,
+ 13,
+ 44,
+ 63
+ ],
+ [
+ 10,
+ 54,
+ 20,
+ 37,
+ 5,
+ 59
+ ],
+ [
+ 41,
+ 43,
+ 24,
+ 5,
+ 53,
+ 14
+ ],
+ [
+ 12,
+ 47,
+ 34,
+ 57,
+ 27,
+ 5
+ ],
+ [
+ 9,
+ 33,
+ 7,
+ 38,
+ 43,
+ 31
+ ],
+ [
+ 19,
+ 10,
+ 32,
+ 20,
+ 18,
+ 1
+ ],
+ [
+ 36,
+ 63,
+ 15,
+ 49,
+ 40,
+ 50
+ ],
+ [
+ 39,
+ 3,
+ 38,
+ 27,
+ 36,
+ 35
+ ],
+ [
+ 29,
+ 19,
+ 40,
+ 47,
+ 62,
+ 14
+ ],
+ [
+ 5,
+ 46,
+ 33,
+ 53,
+ 49,
+ 21
+ ],
+ [
+ 55,
+ 4,
+ 27,
+ 61,
+ 51,
+ 34
+ ],
+ [
+ 11,
+ 0,
+ 59,
+ 48,
+ 14,
+ 27
+ ],
+ [
+ 7,
+ 32,
+ 3,
+ 51,
+ 18,
+ 14
+ ],
+ [
+ 11,
+ 9,
+ 36,
+ 48,
+ 0,
+ 46
+ ]
+ ],
+ [
+ [
+ 63,
+ 62,
+ 60,
+ 19,
+ 23,
+ 56
+ ],
+ [
+ 48,
+ 32,
+ 1,
+ 35,
+ 5,
+ 21
+ ],
+ [
+ 22,
+ 24,
+ 46,
+ 58,
+ 59,
+ 60
+ ],
+ [
+ 27,
+ 37,
+ 50,
+ 28,
+ 61,
+ 6
+ ],
+ [
+ 10,
+ 12,
+ 15,
+ 58,
+ 35,
+ 23
+ ],
+ [
+ 2,
+ 43,
+ 57,
+ 36,
+ 30,
+ 20
+ ],
+ [
+ 21,
+ 14,
+ 19,
+ 63,
+ 41,
+ 42
+ ],
+ [
+ 46,
+ 7,
+ 35,
+ 43,
+ 21,
+ 36
+ ],
+ [
+ 6,
+ 18,
+ 0,
+ 35,
+ 40,
+ 4
+ ],
+ [
+ 47,
+ 53,
+ 32,
+ 38,
+ 1,
+ 58
+ ],
+ [
+ 26,
+ 18,
+ 22,
+ 15,
+ 5,
+ 0
+ ],
+ [
+ 7,
+ 22,
+ 63,
+ 24,
+ 10,
+ 53
+ ],
+ [
+ 11,
+ 3,
+ 10,
+ 18,
+ 37,
+ 24
+ ],
+ [
+ 63,
+ 18,
+ 13,
+ 39,
+ 17,
+ 0
+ ],
+ [
+ 62,
+ 54,
+ 6,
+ 18,
+ 5,
+ 14
+ ],
+ [
+ 19,
+ 43,
+ 30,
+ 31,
+ 57,
+ 42
+ ],
+ [
+ 24,
+ 5,
+ 51,
+ 34,
+ 47,
+ 21
+ ],
+ [
+ 59,
+ 13,
+ 33,
+ 44,
+ 11,
+ 41
+ ],
+ [
+ 58,
+ 32,
+ 1,
+ 7,
+ 39,
+ 38
+ ],
+ [
+ 49,
+ 13,
+ 28,
+ 9,
+ 34,
+ 7
+ ],
+ [
+ 32,
+ 51,
+ 3,
+ 53,
+ 48,
+ 13
+ ],
+ [
+ 40,
+ 15,
+ 37,
+ 57,
+ 1,
+ 23
+ ],
+ [
+ 5,
+ 53,
+ 35,
+ 51,
+ 22,
+ 38
+ ],
+ [
+ 4,
+ 14,
+ 44,
+ 41,
+ 51,
+ 57
+ ],
+ [
+ 47,
+ 48,
+ 8,
+ 61,
+ 13,
+ 4
+ ],
+ [
+ 23,
+ 3,
+ 27,
+ 43,
+ 1,
+ 0
+ ],
+ [
+ 46,
+ 23,
+ 59,
+ 62,
+ 18,
+ 40
+ ]
+ ],
+ [
+ [
+ 62,
+ 0,
+ 9,
+ 32,
+ 26,
+ 41
+ ],
+ [
+ 45,
+ 29,
+ 35,
+ 7,
+ 30,
+ 62
+ ],
+ [
+ 56,
+ 31,
+ 23,
+ 53,
+ 28,
+ 2
+ ],
+ [
+ 36,
+ 2,
+ 5,
+ 4,
+ 48,
+ 41
+ ],
+ [
+ 18,
+ 0,
+ 15,
+ 23,
+ 16,
+ 11
+ ],
+ [
+ 4,
+ 36,
+ 57,
+ 56,
+ 14,
+ 5
+ ],
+ [
+ 59,
+ 56,
+ 21,
+ 8,
+ 7,
+ 33
+ ],
+ [
+ 12,
+ 23,
+ 7,
+ 2,
+ 28,
+ 34
+ ],
+ [
+ 17,
+ 6,
+ 23,
+ 19,
+ 62,
+ 52
+ ],
+ [
+ 53,
+ 32,
+ 51,
+ 58,
+ 41,
+ 38
+ ],
+ [
+ 18,
+ 63,
+ 19,
+ 22,
+ 26,
+ 5
+ ],
+ [
+ 22,
+ 7,
+ 60,
+ 11,
+ 12,
+ 5
+ ],
+ [
+ 24,
+ 7,
+ 11,
+ 30,
+ 17,
+ 47
+ ],
+ [
+ 47,
+ 13,
+ 63,
+ 50,
+ 39,
+ 45
+ ],
+ [
+ 62,
+ 54,
+ 18,
+ 14,
+ 29,
+ 56
+ ],
+ [
+ 56,
+ 30,
+ 0,
+ 51,
+ 55,
+ 10
+ ],
+ [
+ 12,
+ 5,
+ 34,
+ 21,
+ 63,
+ 29
+ ],
+ [
+ 42,
+ 8,
+ 38,
+ 57,
+ 41,
+ 43
+ ],
+ [
+ 32,
+ 54,
+ 1,
+ 37,
+ 49,
+ 40
+ ],
+ [
+ 5,
+ 57,
+ 49,
+ 28,
+ 34,
+ 10
+ ],
+ [
+ 46,
+ 32,
+ 13,
+ 7,
+ 61,
+ 48
+ ],
+ [
+ 40,
+ 3,
+ 30,
+ 60,
+ 39,
+ 42
+ ],
+ [
+ 5,
+ 44,
+ 33,
+ 36,
+ 28,
+ 31
+ ],
+ [
+ 4,
+ 11,
+ 28,
+ 41,
+ 51,
+ 5
+ ],
+ [
+ 48,
+ 47,
+ 61,
+ 28,
+ 27,
+ 60
+ ],
+ [
+ 14,
+ 3,
+ 26,
+ 12,
+ 53,
+ 61
+ ],
+ [
+ 54,
+ 60,
+ 49,
+ 35,
+ 10,
+ 62
+ ]
+ ],
+ [
+ [
+ 27,
+ 62,
+ 63,
+ 23,
+ 47,
+ 56
+ ],
+ [
+ 7,
+ 4,
+ 2,
+ 35,
+ 10,
+ 36
+ ],
+ [
+ 3,
+ 0,
+ 27,
+ 62,
+ 50,
+ 60
+ ],
+ [
+ 36,
+ 3,
+ 42,
+ 18,
+ 2,
+ 48
+ ],
+ [
+ 61,
+ 12,
+ 27,
+ 10,
+ 15,
+ 14
+ ],
+ [
+ 36,
+ 33,
+ 50,
+ 6,
+ 29,
+ 16
+ ],
+ [
+ 2,
+ 41,
+ 8,
+ 43,
+ 40,
+ 59
+ ],
+ [
+ 36,
+ 53,
+ 15,
+ 7,
+ 37,
+ 2
+ ],
+ [
+ 3,
+ 58,
+ 4,
+ 19,
+ 5,
+ 27
+ ],
+ [
+ 22,
+ 26,
+ 53,
+ 45,
+ 25,
+ 38
+ ],
+ [
+ 18,
+ 41,
+ 22,
+ 3,
+ 34,
+ 45
+ ],
+ [
+ 21,
+ 53,
+ 5,
+ 22,
+ 9,
+ 28
+ ],
+ [
+ 11,
+ 24,
+ 28,
+ 16,
+ 8,
+ 35
+ ],
+ [
+ 38,
+ 9,
+ 21,
+ 52,
+ 41,
+ 39
+ ],
+ [
+ 27,
+ 54,
+ 14,
+ 5,
+ 8,
+ 63
+ ],
+ [
+ 56,
+ 55,
+ 43,
+ 30,
+ 51,
+ 35
+ ],
+ [
+ 51,
+ 3,
+ 14,
+ 60,
+ 5,
+ 26
+ ],
+ [
+ 62,
+ 30,
+ 8,
+ 21,
+ 44,
+ 35
+ ],
+ [
+ 38,
+ 34,
+ 27,
+ 32,
+ 22,
+ 8
+ ],
+ [
+ 22,
+ 53,
+ 28,
+ 60,
+ 12,
+ 49
+ ],
+ [
+ 49,
+ 32,
+ 41,
+ 53,
+ 12,
+ 33
+ ],
+ [
+ 40,
+ 45,
+ 22,
+ 44,
+ 1,
+ 34
+ ],
+ [
+ 5,
+ 4,
+ 53,
+ 25,
+ 41,
+ 63
+ ],
+ [
+ 4,
+ 11,
+ 25,
+ 43,
+ 31,
+ 14
+ ],
+ [
+ 31,
+ 52,
+ 5,
+ 58,
+ 48,
+ 43
+ ],
+ [
+ 23,
+ 1,
+ 28,
+ 3,
+ 0,
+ 10
+ ],
+ [
+ 27,
+ 45,
+ 32,
+ 4,
+ 30,
+ 6
+ ]
+ ],
+ [
+ [
+ 41,
+ 2,
+ 42,
+ 16,
+ 50,
+ 32
+ ],
+ [
+ 51,
+ 5,
+ 41,
+ 40,
+ 44,
+ 21
+ ],
+ [
+ 43,
+ 1,
+ 29,
+ 55,
+ 21,
+ 35
+ ],
+ [
+ 36,
+ 58,
+ 25,
+ 3,
+ 18,
+ 54
+ ],
+ [
+ 31,
+ 60,
+ 24,
+ 12,
+ 61,
+ 41
+ ],
+ [
+ 4,
+ 10,
+ 35,
+ 36,
+ 0,
+ 43
+ ],
+ [
+ 45,
+ 43,
+ 63,
+ 35,
+ 36,
+ 48
+ ],
+ [
+ 3,
+ 9,
+ 8,
+ 7,
+ 43,
+ 27
+ ],
+ [
+ 0,
+ 43,
+ 4,
+ 40,
+ 18,
+ 44
+ ],
+ [
+ 34,
+ 15,
+ 50,
+ 38,
+ 53,
+ 25
+ ],
+ [
+ 24,
+ 46,
+ 26,
+ 13,
+ 2,
+ 18
+ ],
+ [
+ 57,
+ 37,
+ 5,
+ 22,
+ 7,
+ 53
+ ],
+ [
+ 3,
+ 9,
+ 38,
+ 26,
+ 37,
+ 23
+ ],
+ [
+ 60,
+ 26,
+ 0,
+ 39,
+ 13,
+ 55
+ ],
+ [
+ 54,
+ 21,
+ 58,
+ 6,
+ 55,
+ 14
+ ],
+ [
+ 11,
+ 19,
+ 43,
+ 57,
+ 56,
+ 37
+ ],
+ [
+ 10,
+ 24,
+ 60,
+ 21,
+ 5,
+ 17
+ ],
+ [
+ 31,
+ 20,
+ 63,
+ 44,
+ 55,
+ 33
+ ],
+ [
+ 7,
+ 25,
+ 40,
+ 32,
+ 58,
+ 54
+ ],
+ [
+ 49,
+ 30,
+ 13,
+ 50,
+ 55,
+ 28
+ ],
+ [
+ 30,
+ 3,
+ 51,
+ 61,
+ 0,
+ 10
+ ],
+ [
+ 57,
+ 46,
+ 43,
+ 40,
+ 30,
+ 22
+ ],
+ [
+ 52,
+ 33,
+ 51,
+ 19,
+ 27,
+ 29
+ ],
+ [
+ 14,
+ 4,
+ 41,
+ 40,
+ 0,
+ 3
+ ],
+ [
+ 4,
+ 13,
+ 8,
+ 61,
+ 9,
+ 47
+ ],
+ [
+ 31,
+ 27,
+ 43,
+ 1,
+ 3,
+ 44
+ ],
+ [
+ 8,
+ 4,
+ 46,
+ 9,
+ 3,
+ 50
+ ]
+ ]
]
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_cudagraph_zmq/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_cudagraph_zmq/model_config.yaml
index edc5fc2eb32..80e2a37c250 100644
--- a/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_cudagraph_zmq/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_cudagraph_zmq/model_config.yaml
@@ -79,12 +79,11 @@ MODEL_ARGS:
--inference-dynamic-batching-buffer-size-gb: 20
--cuda-graph-impl: local
--moe-pad-experts-for-cuda-graph-inference: true
- --inference-dynamic-batching-buffer-size-gb: 20
- --inference-dynamic-batching-num-cuda-graphs: 1
- --inference-dynamic-batching-max-requests: 512
+ --inference-dynamic-batching-num-cuda-graphs: -1
+ --inference-dynamic-batching-max-requests: 16
--inference-logging-step-interval: 1
- --sequence-parallel: true
-
+ --moe-enable-routing-replay: true
METRICS:
- "generated_tokens"
- "logprobs"
+ - "routing_indices"
diff --git a/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq/golden_values_dev_dgx_h100.json
index 66c9e3e4121..915f3864fad 100644
--- a/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq/golden_values_dev_dgx_h100.json
@@ -34,125 +34,25633 @@
1394,
1636
],
- "latency": 28.185462809633464,
+ "latency": 20.171302660994115,
"logprobs": [
-10.737512588500977,
- -3.724862575531006,
- -2.833397388458252,
- -1.2464861869812012,
- -0.2549239993095398,
- -1.7607988119125366,
- -2.419379711151123,
- -1.9533929824829102,
- -2.1014301776885986,
- -6.169030666351318,
- -0.8734959363937378,
- -2.4733574390411377,
- -3.4822516441345215,
- -4.180896759033203,
- -1.9767613410949707,
- -1.8347630500793457,
- -2.2581257820129395,
- -7.180149078369141,
- -0.0453881211578846,
- -1.9841610193252563,
- -5.015386581420898,
- -8.827117919921875,
- -9.885746002197266,
- -0.8498678207397461,
- -4.770059585571289,
- -0.855280339717865,
- -2.2494924068450928,
- -0.017164958640933037,
- -0.03715415671467781,
- -3.4830124378204346,
- -8.635110855102539,
- -1.2520610094070435,
- -6.62324857711792,
- -3.639960765838623,
- -3.664339542388916,
- -4.182392597198486,
- -2.1796066761016846,
- -1.0725229978561401,
- -0.26311880350112915,
- -0.8036076426506042,
- -4.6958818435668945,
- -9.042495727539062,
- -0.013647346757352352,
- -3.1747794151306152,
- -1.322129487991333,
- -3.949110746383667,
- -0.7829495072364807,
- -0.002083513652905822,
- -2.970266580581665,
- -10.56244945526123,
- -3.2369167804718018,
- -1.1530492305755615,
- -4.917466163635254,
- -0.21241025626659393,
- -0.06490474194288254,
- -1.372581124305725,
- -2.224682092666626,
- -4.3847503662109375,
- -0.36867555975914,
- -4.035493850708008,
- -0.39869019389152527,
- -0.14373983442783356,
- -2.716118812561035,
- -10.687016487121582,
- -0.04773370549082756,
- -3.398231267929077,
- -0.8646175265312195,
- -4.74052619934082,
- -0.23649944365024567,
- -2.6610701084136963,
- -0.8428961634635925,
- -1.614527940750122,
- -5.793307781219482,
- -16.929147720336914,
- -2.6586406230926514,
- -0.1385982781648636,
- -7.435610771179199,
- -1.0483647584915161,
- -2.1261863708496094,
- -1.5261307954788208,
- -0.27082547545433044,
- -5.859070777893066,
- -0.00648513063788414,
- -7.732051849365234,
- -2.712515354156494,
- -2.9137418270111084,
- -3.041210651397705,
- -2.3559694290161133,
- -0.3973437249660492,
- -1.4338903427124023,
- -2.2967660427093506,
- -0.6096595525741577,
- -1.3119444847106934,
- -1.93257474899292,
- -1.726539134979248,
- -0.8397530317306519,
- -0.5014236569404602,
- -1.2989763021469116,
- -1.5857150554656982,
- -1.096572995185852,
- -0.4009067416191101,
- -0.43302634358406067,
- -0.041601795703172684,
- -1.285712718963623,
- -2.214778184890747,
- -2.6971933841705322,
- -0.8101387619972229,
- -0.43101266026496887,
- -2.808060884475708,
- -1.5226430892944336,
- -1.6209226846694946,
- -0.048716772347688675,
- -1.3497682809829712,
- -1.343377947807312,
- -1.2755295038223267,
- -1.2342015504837036,
- -0.5394397377967834
+ -3.6886487007141113,
+ -2.8194005489349365,
+ -1.2396225929260254,
+ -0.22920642793178558,
+ -1.8583638668060303,
+ -2.384000778198242,
+ -2.008981227874756,
+ -2.09975528717041,
+ -6.182888031005859,
+ -0.890110433101654,
+ -2.478799343109131,
+ -3.502744436264038,
+ -4.090683937072754,
+ -1.964805006980896,
+ -1.8054677248001099,
+ -2.321495294570923,
+ -7.160285949707031,
+ -0.04007242992520332,
+ -1.9924155473709106,
+ -5.093112468719482,
+ -8.779500007629395,
+ -9.904478073120117,
+ -0.8523460626602173,
+ -4.82073974609375,
+ -0.86553555727005,
+ -2.36330509185791,
+ -0.01946748048067093,
+ -0.034191977232694626,
+ -3.3692376613616943,
+ -8.743470191955566,
+ -1.3306764364242554,
+ -6.645841598510742,
+ -3.7794108390808105,
+ -3.7756881713867188,
+ -4.187170028686523,
+ -2.2124571800231934,
+ -1.0734034776687622,
+ -0.22135400772094727,
+ -0.7782289981842041,
+ -4.716646671295166,
+ -9.09740161895752,
+ -0.013706612400710583,
+ -3.1654152870178223,
+ -1.3161238431930542,
+ -4.0115814208984375,
+ -0.7715368866920471,
+ -0.002078041434288025,
+ -2.9644386768341064,
+ -10.690834045410156,
+ -3.2352819442749023,
+ -1.1544448137283325,
+ -4.971055030822754,
+ -0.2275625467300415,
+ -0.05549970641732216,
+ -1.2946704626083374,
+ -2.20631742477417,
+ -4.362645149230957,
+ -0.37027108669281006,
+ -4.025010108947754,
+ -0.3656681478023529,
+ -0.13922274112701416,
+ -2.720149040222168,
+ -10.679438591003418,
+ -0.04889172688126564,
+ -3.483541965484619,
+ -0.9704694151878357,
+ -4.714925289154053,
+ -0.2623435854911804,
+ -2.60113525390625,
+ -0.8295360207557678,
+ -1.671109676361084,
+ -5.792396068572998,
+ -16.96857261657715,
+ -2.647408962249756,
+ -0.13962996006011963,
+ -7.453517436981201,
+ -1.0938549041748047,
+ -2.134143829345703,
+ -1.5537410974502563,
+ -0.2869631052017212,
+ -5.857534408569336,
+ -0.006919232662767172,
+ -7.747158527374268,
+ -2.7227447032928467,
+ -3.037987232208252,
+ -3.0312116146087646,
+ -2.342064619064331,
+ -0.4008210003376007,
+ -1.5134443044662476,
+ -2.3839635848999023,
+ -0.6089661121368408,
+ -1.3122849464416504,
+ -1.9414935111999512,
+ -1.7005577087402344,
+ -0.7723040580749512,
+ -0.46768665313720703,
+ -1.2926090955734253,
+ -1.586422324180603,
+ -1.0127938985824585,
+ -0.400964617729187,
+ -0.4781394302845001,
+ -0.04295343533158302,
+ -1.282165765762329,
+ -2.125704765319824,
+ -2.6953608989715576,
+ -0.8084508776664734,
+ -0.4046926498413086,
+ -2.861154317855835,
+ -1.5114109516143799,
+ -1.6190099716186523,
+ -0.05034634843468666,
+ -1.3701083660125732,
+ -1.3019596338272095,
+ -1.2718966007232666,
+ -1.2260775566101074,
+ -0.4916250705718994
+ ],
+ "routing_indices": [
+ [
+ [
+ 33,
+ 50,
+ 36,
+ 4,
+ 25,
+ 63
+ ],
+ [
+ 0,
+ 16,
+ 3,
+ 26,
+ 9,
+ 54
+ ],
+ [
+ 62,
+ 60,
+ 8,
+ 16,
+ 58,
+ 52
+ ],
+ [
+ 58,
+ 10,
+ 6,
+ 45,
+ 16,
+ 32
+ ],
+ [
+ 43,
+ 49,
+ 18,
+ 54,
+ 55,
+ 13
+ ],
+ [
+ 27,
+ 19,
+ 26,
+ 44,
+ 12,
+ 28
+ ],
+ [
+ 53,
+ 42,
+ 3,
+ 27,
+ 26,
+ 19
+ ],
+ [
+ 6,
+ 47,
+ 1,
+ 8,
+ 19,
+ 22
+ ],
+ [
+ 51,
+ 27,
+ 1,
+ 38,
+ 16,
+ 62
+ ],
+ [
+ 41,
+ 49,
+ 21,
+ 57,
+ 16,
+ 24
+ ],
+ [
+ 13,
+ 28,
+ 38,
+ 22,
+ 49,
+ 48
+ ],
+ [
+ 52,
+ 58,
+ 6,
+ 25,
+ 29,
+ 17
+ ],
+ [
+ 49,
+ 50,
+ 25,
+ 41,
+ 54,
+ 58
+ ],
+ [
+ 27,
+ 41,
+ 3,
+ 1,
+ 26,
+ 29
+ ],
+ [
+ 41,
+ 18,
+ 34,
+ 45,
+ 1,
+ 33
+ ],
+ [
+ 17,
+ 26,
+ 59,
+ 22,
+ 19,
+ 4
+ ],
+ [
+ 1,
+ 43,
+ 62,
+ 57,
+ 61,
+ 21
+ ],
+ [
+ 11,
+ 46,
+ 15,
+ 28,
+ 61,
+ 50
+ ],
+ [
+ 14,
+ 30,
+ 58,
+ 26,
+ 38,
+ 53
+ ],
+ [
+ 59,
+ 20,
+ 63,
+ 54,
+ 47,
+ 61
+ ],
+ [
+ 9,
+ 5,
+ 43,
+ 33,
+ 15,
+ 46
+ ],
+ [
+ 16,
+ 52,
+ 33,
+ 61,
+ 49,
+ 11
+ ],
+ [
+ 52,
+ 35,
+ 40,
+ 43,
+ 29,
+ 36
+ ],
+ [
+ 57,
+ 34,
+ 38,
+ 44,
+ 20,
+ 18
+ ],
+ [
+ 44,
+ 51,
+ 2,
+ 63,
+ 7,
+ 22
+ ],
+ [
+ 32,
+ 47,
+ 58,
+ 9,
+ 54,
+ 5
+ ],
+ [
+ 12,
+ 59,
+ 54,
+ 33,
+ 50,
+ 6
+ ]
+ ],
+ [
+ [
+ 49,
+ 43,
+ 18,
+ 28,
+ 23,
+ 25
+ ],
+ [
+ 34,
+ 24,
+ 23,
+ 60,
+ 2,
+ 18
+ ],
+ [
+ 33,
+ 40,
+ 30,
+ 3,
+ 59,
+ 48
+ ],
+ [
+ 16,
+ 58,
+ 17,
+ 48,
+ 6,
+ 45
+ ],
+ [
+ 23,
+ 58,
+ 46,
+ 37,
+ 34,
+ 48
+ ],
+ [
+ 28,
+ 26,
+ 35,
+ 33,
+ 22,
+ 43
+ ],
+ [
+ 6,
+ 31,
+ 13,
+ 46,
+ 41,
+ 37
+ ],
+ [
+ 60,
+ 29,
+ 44,
+ 36,
+ 39,
+ 15
+ ],
+ [
+ 51,
+ 18,
+ 62,
+ 5,
+ 27,
+ 35
+ ],
+ [
+ 62,
+ 45,
+ 32,
+ 56,
+ 25,
+ 3
+ ],
+ [
+ 6,
+ 20,
+ 3,
+ 16,
+ 49,
+ 41
+ ],
+ [
+ 36,
+ 41,
+ 50,
+ 45,
+ 35,
+ 48
+ ],
+ [
+ 39,
+ 46,
+ 48,
+ 25,
+ 21,
+ 33
+ ],
+ [
+ 30,
+ 38,
+ 11,
+ 22,
+ 15,
+ 9
+ ],
+ [
+ 24,
+ 2,
+ 32,
+ 56,
+ 63,
+ 14
+ ],
+ [
+ 36,
+ 55,
+ 17,
+ 32,
+ 44,
+ 35
+ ],
+ [
+ 59,
+ 46,
+ 32,
+ 44,
+ 24,
+ 14
+ ],
+ [
+ 51,
+ 15,
+ 61,
+ 43,
+ 30,
+ 16
+ ],
+ [
+ 2,
+ 32,
+ 5,
+ 39,
+ 11,
+ 50
+ ],
+ [
+ 28,
+ 42,
+ 6,
+ 30,
+ 57,
+ 37
+ ],
+ [
+ 28,
+ 55,
+ 45,
+ 0,
+ 7,
+ 41
+ ],
+ [
+ 48,
+ 40,
+ 34,
+ 3,
+ 49,
+ 22
+ ],
+ [
+ 25,
+ 6,
+ 62,
+ 50,
+ 18,
+ 53
+ ],
+ [
+ 13,
+ 22,
+ 20,
+ 28,
+ 25,
+ 59
+ ],
+ [
+ 58,
+ 36,
+ 39,
+ 47,
+ 29,
+ 37
+ ],
+ [
+ 4,
+ 33,
+ 41,
+ 12,
+ 3,
+ 17
+ ],
+ [
+ 22,
+ 13,
+ 20,
+ 52,
+ 24,
+ 62
+ ]
+ ],
+ [
+ [
+ 17,
+ 10,
+ 57,
+ 54,
+ 6,
+ 15
+ ],
+ [
+ 33,
+ 43,
+ 13,
+ 1,
+ 16,
+ 62
+ ],
+ [
+ 63,
+ 1,
+ 35,
+ 43,
+ 27,
+ 10
+ ],
+ [
+ 47,
+ 4,
+ 38,
+ 50,
+ 51,
+ 0
+ ],
+ [
+ 11,
+ 51,
+ 57,
+ 23,
+ 14,
+ 34
+ ],
+ [
+ 10,
+ 43,
+ 35,
+ 33,
+ 20,
+ 22
+ ],
+ [
+ 36,
+ 48,
+ 35,
+ 19,
+ 21,
+ 28
+ ],
+ [
+ 14,
+ 8,
+ 7,
+ 46,
+ 35,
+ 13
+ ],
+ [
+ 18,
+ 44,
+ 63,
+ 6,
+ 4,
+ 37
+ ],
+ [
+ 62,
+ 29,
+ 15,
+ 38,
+ 39,
+ 34
+ ],
+ [
+ 1,
+ 6,
+ 16,
+ 46,
+ 22,
+ 13
+ ],
+ [
+ 36,
+ 8,
+ 37,
+ 16,
+ 10,
+ 14
+ ],
+ [
+ 8,
+ 0,
+ 32,
+ 3,
+ 43,
+ 10
+ ],
+ [
+ 25,
+ 0,
+ 22,
+ 30,
+ 60,
+ 57
+ ],
+ [
+ 60,
+ 58,
+ 55,
+ 32,
+ 2,
+ 7
+ ],
+ [
+ 55,
+ 11,
+ 19,
+ 5,
+ 24,
+ 43
+ ],
+ [
+ 9,
+ 47,
+ 36,
+ 39,
+ 5,
+ 42
+ ],
+ [
+ 31,
+ 20,
+ 9,
+ 43,
+ 18,
+ 41
+ ],
+ [
+ 12,
+ 5,
+ 32,
+ 50,
+ 3,
+ 31
+ ],
+ [
+ 40,
+ 18,
+ 30,
+ 63,
+ 7,
+ 25
+ ],
+ [
+ 14,
+ 7,
+ 3,
+ 38,
+ 54,
+ 59
+ ],
+ [
+ 31,
+ 20,
+ 1,
+ 22,
+ 47,
+ 6
+ ],
+ [
+ 51,
+ 15,
+ 18,
+ 53,
+ 40,
+ 0
+ ],
+ [
+ 27,
+ 54,
+ 8,
+ 38,
+ 3,
+ 59
+ ],
+ [
+ 11,
+ 16,
+ 39,
+ 59,
+ 9,
+ 23
+ ],
+ [
+ 7,
+ 37,
+ 51,
+ 30,
+ 18,
+ 3
+ ],
+ [
+ 0,
+ 36,
+ 33,
+ 40,
+ 46,
+ 48
+ ]
+ ],
+ [
+ [
+ 48,
+ 33,
+ 52,
+ 53,
+ 24,
+ 38
+ ],
+ [
+ 3,
+ 40,
+ 45,
+ 19,
+ 22,
+ 35
+ ],
+ [
+ 46,
+ 15,
+ 39,
+ 60,
+ 25,
+ 31
+ ],
+ [
+ 49,
+ 62,
+ 61,
+ 48,
+ 55,
+ 46
+ ],
+ [
+ 41,
+ 35,
+ 53,
+ 52,
+ 13,
+ 10
+ ],
+ [
+ 52,
+ 15,
+ 45,
+ 63,
+ 46,
+ 10
+ ],
+ [
+ 14,
+ 15,
+ 35,
+ 60,
+ 49,
+ 31
+ ],
+ [
+ 44,
+ 3,
+ 6,
+ 8,
+ 23,
+ 48
+ ],
+ [
+ 32,
+ 15,
+ 44,
+ 27,
+ 4,
+ 62
+ ],
+ [
+ 51,
+ 25,
+ 62,
+ 53,
+ 48,
+ 10
+ ],
+ [
+ 8,
+ 19,
+ 5,
+ 16,
+ 22,
+ 50
+ ],
+ [
+ 36,
+ 49,
+ 60,
+ 15,
+ 44,
+ 25
+ ],
+ [
+ 13,
+ 27,
+ 53,
+ 30,
+ 56,
+ 43
+ ],
+ [
+ 26,
+ 22,
+ 7,
+ 14,
+ 32,
+ 17
+ ],
+ [
+ 26,
+ 60,
+ 58,
+ 2,
+ 54,
+ 10
+ ],
+ [
+ 50,
+ 55,
+ 17,
+ 51,
+ 47,
+ 14
+ ],
+ [
+ 0,
+ 12,
+ 16,
+ 4,
+ 23,
+ 9
+ ],
+ [
+ 10,
+ 23,
+ 27,
+ 46,
+ 55,
+ 56
+ ],
+ [
+ 3,
+ 37,
+ 4,
+ 60,
+ 16,
+ 59
+ ],
+ [
+ 38,
+ 3,
+ 29,
+ 40,
+ 25,
+ 50
+ ],
+ [
+ 31,
+ 16,
+ 62,
+ 54,
+ 42,
+ 5
+ ],
+ [
+ 47,
+ 35,
+ 11,
+ 37,
+ 46,
+ 32
+ ],
+ [
+ 27,
+ 51,
+ 20,
+ 50,
+ 55,
+ 16
+ ],
+ [
+ 63,
+ 55,
+ 46,
+ 27,
+ 48,
+ 12
+ ],
+ [
+ 53,
+ 50,
+ 30,
+ 2,
+ 39,
+ 20
+ ],
+ [
+ 53,
+ 44,
+ 24,
+ 8,
+ 51,
+ 14
+ ],
+ [
+ 19,
+ 49,
+ 37,
+ 14,
+ 44,
+ 0
+ ]
+ ],
+ [
+ [
+ 0,
+ 52,
+ 16,
+ 12,
+ 54,
+ 7
+ ],
+ [
+ 42,
+ 25,
+ 51,
+ 61,
+ 35,
+ 58
+ ],
+ [
+ 51,
+ 42,
+ 19,
+ 57,
+ 28,
+ 8
+ ],
+ [
+ 49,
+ 62,
+ 5,
+ 2,
+ 46,
+ 21
+ ],
+ [
+ 3,
+ 41,
+ 53,
+ 25,
+ 39,
+ 37
+ ],
+ [
+ 45,
+ 15,
+ 37,
+ 48,
+ 19,
+ 60
+ ],
+ [
+ 14,
+ 15,
+ 47,
+ 17,
+ 24,
+ 35
+ ],
+ [
+ 3,
+ 52,
+ 63,
+ 16,
+ 28,
+ 47
+ ],
+ [
+ 32,
+ 27,
+ 15,
+ 24,
+ 62,
+ 23
+ ],
+ [
+ 62,
+ 25,
+ 51,
+ 53,
+ 0,
+ 20
+ ],
+ [
+ 8,
+ 19,
+ 50,
+ 16,
+ 22,
+ 32
+ ],
+ [
+ 36,
+ 13,
+ 42,
+ 49,
+ 60,
+ 44
+ ],
+ [
+ 30,
+ 27,
+ 53,
+ 56,
+ 0,
+ 10
+ ],
+ [
+ 22,
+ 7,
+ 14,
+ 26,
+ 32,
+ 17
+ ],
+ [
+ 60,
+ 7,
+ 26,
+ 2,
+ 58,
+ 13
+ ],
+ [
+ 55,
+ 51,
+ 17,
+ 60,
+ 62,
+ 47
+ ],
+ [
+ 53,
+ 16,
+ 18,
+ 4,
+ 50,
+ 5
+ ],
+ [
+ 28,
+ 39,
+ 41,
+ 1,
+ 55,
+ 11
+ ],
+ [
+ 59,
+ 49,
+ 16,
+ 23,
+ 42,
+ 12
+ ],
+ [
+ 17,
+ 30,
+ 46,
+ 25,
+ 55,
+ 36
+ ],
+ [
+ 54,
+ 9,
+ 45,
+ 0,
+ 6,
+ 56
+ ],
+ [
+ 51,
+ 9,
+ 22,
+ 23,
+ 31,
+ 16
+ ],
+ [
+ 62,
+ 50,
+ 43,
+ 51,
+ 55,
+ 27
+ ],
+ [
+ 13,
+ 43,
+ 27,
+ 1,
+ 14,
+ 52
+ ],
+ [
+ 61,
+ 26,
+ 17,
+ 1,
+ 63,
+ 32
+ ],
+ [
+ 37,
+ 46,
+ 63,
+ 20,
+ 9,
+ 24
+ ],
+ [
+ 63,
+ 11,
+ 12,
+ 61,
+ 22,
+ 31
+ ]
+ ],
+ [
+ [
+ 49,
+ 54,
+ 56,
+ 11,
+ 38,
+ 3
+ ],
+ [
+ 53,
+ 46,
+ 49,
+ 38,
+ 57,
+ 17
+ ],
+ [
+ 9,
+ 48,
+ 31,
+ 12,
+ 56,
+ 6
+ ],
+ [
+ 49,
+ 62,
+ 23,
+ 5,
+ 12,
+ 63
+ ],
+ [
+ 36,
+ 26,
+ 38,
+ 7,
+ 20,
+ 23
+ ],
+ [
+ 37,
+ 33,
+ 41,
+ 58,
+ 57,
+ 32
+ ],
+ [
+ 10,
+ 14,
+ 15,
+ 31,
+ 8,
+ 43
+ ],
+ [
+ 16,
+ 52,
+ 3,
+ 2,
+ 34,
+ 14
+ ],
+ [
+ 15,
+ 32,
+ 35,
+ 27,
+ 62,
+ 54
+ ],
+ [
+ 51,
+ 62,
+ 25,
+ 53,
+ 10,
+ 20
+ ],
+ [
+ 19,
+ 8,
+ 22,
+ 50,
+ 1,
+ 5
+ ],
+ [
+ 13,
+ 49,
+ 36,
+ 60,
+ 42,
+ 20
+ ],
+ [
+ 27,
+ 30,
+ 53,
+ 54,
+ 26,
+ 43
+ ],
+ [
+ 14,
+ 32,
+ 26,
+ 22,
+ 7,
+ 17
+ ],
+ [
+ 26,
+ 60,
+ 7,
+ 2,
+ 52,
+ 54
+ ],
+ [
+ 55,
+ 17,
+ 51,
+ 47,
+ 62,
+ 26
+ ],
+ [
+ 16,
+ 18,
+ 44,
+ 4,
+ 53,
+ 50
+ ],
+ [
+ 1,
+ 47,
+ 39,
+ 45,
+ 28,
+ 56
+ ],
+ [
+ 23,
+ 16,
+ 55,
+ 49,
+ 8,
+ 32
+ ],
+ [
+ 17,
+ 55,
+ 30,
+ 62,
+ 31,
+ 23
+ ],
+ [
+ 9,
+ 54,
+ 0,
+ 44,
+ 14,
+ 56
+ ],
+ [
+ 23,
+ 9,
+ 51,
+ 22,
+ 31,
+ 50
+ ],
+ [
+ 50,
+ 27,
+ 14,
+ 51,
+ 18,
+ 19
+ ],
+ [
+ 12,
+ 13,
+ 33,
+ 1,
+ 43,
+ 5
+ ],
+ [
+ 26,
+ 32,
+ 1,
+ 37,
+ 50,
+ 57
+ ],
+ [
+ 37,
+ 47,
+ 63,
+ 46,
+ 5,
+ 4
+ ],
+ [
+ 63,
+ 11,
+ 12,
+ 19,
+ 33,
+ 61
+ ]
+ ],
+ [
+ [
+ 49,
+ 54,
+ 40,
+ 56,
+ 3,
+ 11
+ ],
+ [
+ 37,
+ 15,
+ 12,
+ 33,
+ 59,
+ 17
+ ],
+ [
+ 38,
+ 49,
+ 14,
+ 46,
+ 35,
+ 59
+ ],
+ [
+ 25,
+ 20,
+ 39,
+ 62,
+ 49,
+ 12
+ ],
+ [
+ 26,
+ 51,
+ 16,
+ 36,
+ 18,
+ 8
+ ],
+ [
+ 37,
+ 51,
+ 41,
+ 33,
+ 32,
+ 60
+ ],
+ [
+ 10,
+ 14,
+ 59,
+ 8,
+ 15,
+ 40
+ ],
+ [
+ 16,
+ 52,
+ 19,
+ 61,
+ 32,
+ 2
+ ],
+ [
+ 32,
+ 15,
+ 27,
+ 24,
+ 62,
+ 35
+ ],
+ [
+ 51,
+ 25,
+ 62,
+ 0,
+ 49,
+ 60
+ ],
+ [
+ 8,
+ 50,
+ 19,
+ 48,
+ 16,
+ 20
+ ],
+ [
+ 13,
+ 49,
+ 36,
+ 42,
+ 11,
+ 60
+ ],
+ [
+ 53,
+ 27,
+ 7,
+ 26,
+ 30,
+ 21
+ ],
+ [
+ 14,
+ 22,
+ 26,
+ 37,
+ 32,
+ 4
+ ],
+ [
+ 26,
+ 60,
+ 40,
+ 2,
+ 52,
+ 7
+ ],
+ [
+ 55,
+ 51,
+ 17,
+ 46,
+ 62,
+ 13
+ ],
+ [
+ 38,
+ 16,
+ 53,
+ 44,
+ 4,
+ 18
+ ],
+ [
+ 39,
+ 1,
+ 4,
+ 14,
+ 56,
+ 57
+ ],
+ [
+ 55,
+ 23,
+ 32,
+ 14,
+ 13,
+ 16
+ ],
+ [
+ 55,
+ 17,
+ 2,
+ 30,
+ 62,
+ 12
+ ],
+ [
+ 13,
+ 54,
+ 0,
+ 62,
+ 61,
+ 25
+ ],
+ [
+ 9,
+ 51,
+ 5,
+ 22,
+ 19,
+ 16
+ ],
+ [
+ 50,
+ 51,
+ 7,
+ 19,
+ 48,
+ 53
+ ],
+ [
+ 12,
+ 1,
+ 5,
+ 43,
+ 61,
+ 13
+ ],
+ [
+ 26,
+ 32,
+ 30,
+ 37,
+ 34,
+ 20
+ ],
+ [
+ 37,
+ 63,
+ 46,
+ 4,
+ 47,
+ 8
+ ],
+ [
+ 63,
+ 12,
+ 11,
+ 31,
+ 33,
+ 61
+ ]
+ ],
+ [
+ [
+ 47,
+ 34,
+ 30,
+ 25,
+ 31,
+ 3
+ ],
+ [
+ 15,
+ 24,
+ 46,
+ 21,
+ 8,
+ 6
+ ],
+ [
+ 34,
+ 21,
+ 18,
+ 62,
+ 28,
+ 55
+ ],
+ [
+ 35,
+ 32,
+ 20,
+ 39,
+ 59,
+ 54
+ ],
+ [
+ 26,
+ 27,
+ 15,
+ 48,
+ 60,
+ 47
+ ],
+ [
+ 37,
+ 8,
+ 50,
+ 18,
+ 54,
+ 61
+ ],
+ [
+ 35,
+ 31,
+ 8,
+ 24,
+ 14,
+ 15
+ ],
+ [
+ 16,
+ 52,
+ 34,
+ 29,
+ 48,
+ 36
+ ],
+ [
+ 11,
+ 32,
+ 62,
+ 27,
+ 46,
+ 26
+ ],
+ [
+ 61,
+ 62,
+ 25,
+ 56,
+ 46,
+ 53
+ ],
+ [
+ 56,
+ 50,
+ 63,
+ 3,
+ 45,
+ 28
+ ],
+ [
+ 11,
+ 36,
+ 5,
+ 60,
+ 35,
+ 50
+ ],
+ [
+ 21,
+ 26,
+ 41,
+ 51,
+ 46,
+ 53
+ ],
+ [
+ 14,
+ 22,
+ 33,
+ 19,
+ 41,
+ 16
+ ],
+ [
+ 2,
+ 52,
+ 34,
+ 60,
+ 21,
+ 49
+ ],
+ [
+ 59,
+ 55,
+ 29,
+ 8,
+ 61,
+ 22
+ ],
+ [
+ 51,
+ 44,
+ 2,
+ 59,
+ 47,
+ 53
+ ],
+ [
+ 39,
+ 25,
+ 18,
+ 12,
+ 51,
+ 56
+ ],
+ [
+ 34,
+ 53,
+ 32,
+ 12,
+ 38,
+ 48
+ ],
+ [
+ 30,
+ 53,
+ 56,
+ 7,
+ 40,
+ 62
+ ],
+ [
+ 40,
+ 49,
+ 28,
+ 14,
+ 23,
+ 55
+ ],
+ [
+ 15,
+ 48,
+ 40,
+ 47,
+ 9,
+ 1
+ ],
+ [
+ 50,
+ 41,
+ 53,
+ 25,
+ 18,
+ 0
+ ],
+ [
+ 22,
+ 1,
+ 59,
+ 3,
+ 55,
+ 8
+ ],
+ [
+ 1,
+ 53,
+ 32,
+ 26,
+ 47,
+ 3
+ ],
+ [
+ 4,
+ 33,
+ 28,
+ 37,
+ 55,
+ 54
+ ],
+ [
+ 30,
+ 22,
+ 57,
+ 12,
+ 33,
+ 63
+ ]
+ ],
+ [
+ [
+ 16,
+ 11,
+ 0,
+ 46,
+ 31,
+ 21
+ ],
+ [
+ 49,
+ 13,
+ 31,
+ 5,
+ 11,
+ 14
+ ],
+ [
+ 36,
+ 13,
+ 56,
+ 27,
+ 46,
+ 3
+ ],
+ [
+ 24,
+ 44,
+ 62,
+ 29,
+ 15,
+ 13
+ ],
+ [
+ 17,
+ 2,
+ 50,
+ 8,
+ 45,
+ 1
+ ],
+ [
+ 8,
+ 7,
+ 49,
+ 0,
+ 62,
+ 13
+ ],
+ [
+ 35,
+ 61,
+ 58,
+ 23,
+ 36,
+ 0
+ ],
+ [
+ 16,
+ 48,
+ 42,
+ 4,
+ 32,
+ 1
+ ],
+ [
+ 63,
+ 18,
+ 32,
+ 4,
+ 34,
+ 62
+ ],
+ [
+ 57,
+ 62,
+ 54,
+ 27,
+ 25,
+ 53
+ ],
+ [
+ 1,
+ 59,
+ 60,
+ 29,
+ 22,
+ 14
+ ],
+ [
+ 31,
+ 36,
+ 11,
+ 14,
+ 20,
+ 47
+ ],
+ [
+ 34,
+ 2,
+ 19,
+ 14,
+ 8,
+ 37
+ ],
+ [
+ 57,
+ 22,
+ 40,
+ 14,
+ 62,
+ 48
+ ],
+ [
+ 44,
+ 60,
+ 7,
+ 14,
+ 45,
+ 2
+ ],
+ [
+ 8,
+ 6,
+ 55,
+ 25,
+ 50,
+ 59
+ ],
+ [
+ 13,
+ 47,
+ 23,
+ 61,
+ 42,
+ 39
+ ],
+ [
+ 25,
+ 23,
+ 4,
+ 14,
+ 46,
+ 60
+ ],
+ [
+ 8,
+ 45,
+ 32,
+ 53,
+ 54,
+ 10
+ ],
+ [
+ 15,
+ 38,
+ 53,
+ 55,
+ 30,
+ 7
+ ],
+ [
+ 41,
+ 14,
+ 28,
+ 27,
+ 5,
+ 58
+ ],
+ [
+ 11,
+ 41,
+ 57,
+ 10,
+ 1,
+ 47
+ ],
+ [
+ 50,
+ 0,
+ 51,
+ 53,
+ 34,
+ 55
+ ],
+ [
+ 1,
+ 14,
+ 55,
+ 25,
+ 8,
+ 27
+ ],
+ [
+ 11,
+ 49,
+ 1,
+ 9,
+ 0,
+ 23
+ ],
+ [
+ 50,
+ 51,
+ 6,
+ 42,
+ 4,
+ 54
+ ],
+ [
+ 17,
+ 37,
+ 31,
+ 5,
+ 36,
+ 40
+ ]
+ ],
+ [
+ [
+ 22,
+ 53,
+ 47,
+ 6,
+ 57,
+ 21
+ ],
+ [
+ 27,
+ 11,
+ 14,
+ 6,
+ 57,
+ 16
+ ],
+ [
+ 1,
+ 11,
+ 29,
+ 26,
+ 41,
+ 17
+ ],
+ [
+ 14,
+ 22,
+ 38,
+ 31,
+ 29,
+ 36
+ ],
+ [
+ 14,
+ 59,
+ 29,
+ 61,
+ 45,
+ 52
+ ],
+ [
+ 30,
+ 8,
+ 0,
+ 21,
+ 47,
+ 58
+ ],
+ [
+ 58,
+ 35,
+ 4,
+ 61,
+ 23,
+ 36
+ ],
+ [
+ 42,
+ 20,
+ 48,
+ 16,
+ 9,
+ 4
+ ],
+ [
+ 47,
+ 29,
+ 4,
+ 18,
+ 63,
+ 32
+ ],
+ [
+ 19,
+ 54,
+ 62,
+ 53,
+ 57,
+ 29
+ ],
+ [
+ 1,
+ 60,
+ 14,
+ 59,
+ 29,
+ 40
+ ],
+ [
+ 36,
+ 0,
+ 47,
+ 3,
+ 31,
+ 8
+ ],
+ [
+ 2,
+ 19,
+ 36,
+ 8,
+ 20,
+ 37
+ ],
+ [
+ 57,
+ 22,
+ 40,
+ 49,
+ 31,
+ 14
+ ],
+ [
+ 44,
+ 37,
+ 2,
+ 5,
+ 60,
+ 21
+ ],
+ [
+ 6,
+ 43,
+ 24,
+ 5,
+ 2,
+ 59
+ ],
+ [
+ 13,
+ 61,
+ 19,
+ 47,
+ 50,
+ 39
+ ],
+ [
+ 58,
+ 14,
+ 28,
+ 4,
+ 22,
+ 11
+ ],
+ [
+ 35,
+ 32,
+ 46,
+ 31,
+ 10,
+ 45
+ ],
+ [
+ 15,
+ 13,
+ 55,
+ 45,
+ 18,
+ 63
+ ],
+ [
+ 15,
+ 27,
+ 28,
+ 14,
+ 5,
+ 60
+ ],
+ [
+ 57,
+ 41,
+ 47,
+ 19,
+ 36,
+ 10
+ ],
+ [
+ 34,
+ 10,
+ 53,
+ 55,
+ 22,
+ 19
+ ],
+ [
+ 38,
+ 55,
+ 39,
+ 27,
+ 3,
+ 25
+ ],
+ [
+ 11,
+ 39,
+ 0,
+ 9,
+ 3,
+ 49
+ ],
+ [
+ 51,
+ 6,
+ 43,
+ 18,
+ 50,
+ 53
+ ],
+ [
+ 55,
+ 43,
+ 9,
+ 36,
+ 40,
+ 5
+ ]
+ ],
+ [
+ [
+ 18,
+ 9,
+ 1,
+ 36,
+ 61,
+ 44
+ ],
+ [
+ 56,
+ 34,
+ 19,
+ 42,
+ 3,
+ 5
+ ],
+ [
+ 39,
+ 20,
+ 15,
+ 60,
+ 46,
+ 32
+ ],
+ [
+ 60,
+ 22,
+ 31,
+ 27,
+ 14,
+ 55
+ ],
+ [
+ 59,
+ 58,
+ 10,
+ 7,
+ 46,
+ 18
+ ],
+ [
+ 43,
+ 2,
+ 57,
+ 62,
+ 11,
+ 30
+ ],
+ [
+ 54,
+ 19,
+ 9,
+ 21,
+ 48,
+ 56
+ ],
+ [
+ 46,
+ 24,
+ 7,
+ 14,
+ 3,
+ 8
+ ],
+ [
+ 47,
+ 0,
+ 4,
+ 18,
+ 31,
+ 35
+ ],
+ [
+ 54,
+ 62,
+ 47,
+ 4,
+ 38,
+ 32
+ ],
+ [
+ 1,
+ 14,
+ 15,
+ 22,
+ 38,
+ 59
+ ],
+ [
+ 16,
+ 36,
+ 42,
+ 18,
+ 0,
+ 55
+ ],
+ [
+ 49,
+ 8,
+ 20,
+ 0,
+ 14,
+ 33
+ ],
+ [
+ 18,
+ 39,
+ 25,
+ 2,
+ 22,
+ 62
+ ],
+ [
+ 62,
+ 5,
+ 58,
+ 37,
+ 32,
+ 2
+ ],
+ [
+ 43,
+ 5,
+ 42,
+ 63,
+ 55,
+ 37
+ ],
+ [
+ 47,
+ 33,
+ 15,
+ 63,
+ 50,
+ 12
+ ],
+ [
+ 60,
+ 0,
+ 7,
+ 16,
+ 13,
+ 32
+ ],
+ [
+ 12,
+ 39,
+ 32,
+ 16,
+ 61,
+ 45
+ ],
+ [
+ 52,
+ 34,
+ 62,
+ 15,
+ 18,
+ 30
+ ],
+ [
+ 28,
+ 26,
+ 46,
+ 40,
+ 6,
+ 14
+ ],
+ [
+ 1,
+ 19,
+ 17,
+ 4,
+ 20,
+ 40
+ ],
+ [
+ 41,
+ 40,
+ 4,
+ 53,
+ 19,
+ 55
+ ],
+ [
+ 25,
+ 38,
+ 27,
+ 34,
+ 52,
+ 46
+ ],
+ [
+ 11,
+ 29,
+ 52,
+ 44,
+ 35,
+ 13
+ ],
+ [
+ 50,
+ 51,
+ 41,
+ 16,
+ 4,
+ 14
+ ],
+ [
+ 19,
+ 6,
+ 23,
+ 36,
+ 60,
+ 0
+ ]
+ ],
+ [
+ [
+ 17,
+ 10,
+ 57,
+ 27,
+ 5,
+ 54
+ ],
+ [
+ 33,
+ 9,
+ 43,
+ 40,
+ 56,
+ 11
+ ],
+ [
+ 63,
+ 1,
+ 35,
+ 43,
+ 10,
+ 27
+ ],
+ [
+ 51,
+ 47,
+ 20,
+ 21,
+ 28,
+ 61
+ ],
+ [
+ 25,
+ 11,
+ 58,
+ 23,
+ 55,
+ 46
+ ],
+ [
+ 43,
+ 10,
+ 12,
+ 2,
+ 30,
+ 62
+ ],
+ [
+ 48,
+ 21,
+ 19,
+ 8,
+ 7,
+ 54
+ ],
+ [
+ 14,
+ 7,
+ 24,
+ 8,
+ 46,
+ 2
+ ],
+ [
+ 4,
+ 47,
+ 37,
+ 0,
+ 44,
+ 27
+ ],
+ [
+ 54,
+ 38,
+ 62,
+ 47,
+ 15,
+ 14
+ ],
+ [
+ 1,
+ 46,
+ 15,
+ 22,
+ 51,
+ 38
+ ],
+ [
+ 36,
+ 16,
+ 42,
+ 24,
+ 55,
+ 37
+ ],
+ [
+ 49,
+ 10,
+ 0,
+ 3,
+ 43,
+ 17
+ ],
+ [
+ 39,
+ 58,
+ 0,
+ 62,
+ 22,
+ 25
+ ],
+ [
+ 58,
+ 38,
+ 7,
+ 62,
+ 55,
+ 56
+ ],
+ [
+ 42,
+ 19,
+ 55,
+ 43,
+ 11,
+ 37
+ ],
+ [
+ 9,
+ 47,
+ 43,
+ 18,
+ 52,
+ 50
+ ],
+ [
+ 31,
+ 41,
+ 32,
+ 25,
+ 20,
+ 13
+ ],
+ [
+ 12,
+ 32,
+ 61,
+ 3,
+ 43,
+ 21
+ ],
+ [
+ 36,
+ 13,
+ 40,
+ 7,
+ 16,
+ 62
+ ],
+ [
+ 14,
+ 53,
+ 50,
+ 47,
+ 51,
+ 1
+ ],
+ [
+ 1,
+ 38,
+ 19,
+ 30,
+ 18,
+ 16
+ ],
+ [
+ 0,
+ 19,
+ 51,
+ 18,
+ 52,
+ 15
+ ],
+ [
+ 8,
+ 52,
+ 27,
+ 34,
+ 38,
+ 3
+ ],
+ [
+ 27,
+ 53,
+ 59,
+ 9,
+ 40,
+ 4
+ ],
+ [
+ 37,
+ 3,
+ 26,
+ 48,
+ 8,
+ 43
+ ],
+ [
+ 46,
+ 18,
+ 11,
+ 40,
+ 33,
+ 44
+ ]
+ ],
+ [
+ [
+ 48,
+ 62,
+ 61,
+ 26,
+ 0,
+ 50
+ ],
+ [
+ 3,
+ 45,
+ 40,
+ 35,
+ 29,
+ 54
+ ],
+ [
+ 56,
+ 31,
+ 23,
+ 28,
+ 2,
+ 53
+ ],
+ [
+ 62,
+ 49,
+ 20,
+ 61,
+ 6,
+ 41
+ ],
+ [
+ 18,
+ 25,
+ 50,
+ 0,
+ 14,
+ 57
+ ],
+ [
+ 58,
+ 4,
+ 10,
+ 43,
+ 56,
+ 46
+ ],
+ [
+ 35,
+ 15,
+ 25,
+ 24,
+ 3,
+ 7
+ ],
+ [
+ 14,
+ 23,
+ 8,
+ 12,
+ 7,
+ 24
+ ],
+ [
+ 29,
+ 17,
+ 35,
+ 44,
+ 24,
+ 4
+ ],
+ [
+ 62,
+ 15,
+ 38,
+ 20,
+ 58,
+ 40
+ ],
+ [
+ 19,
+ 1,
+ 46,
+ 22,
+ 26,
+ 63
+ ],
+ [
+ 36,
+ 60,
+ 42,
+ 16,
+ 11,
+ 55
+ ],
+ [
+ 17,
+ 7,
+ 16,
+ 49,
+ 14,
+ 26
+ ],
+ [
+ 45,
+ 47,
+ 22,
+ 0,
+ 58,
+ 62
+ ],
+ [
+ 58,
+ 38,
+ 48,
+ 63,
+ 2,
+ 49
+ ],
+ [
+ 55,
+ 0,
+ 1,
+ 37,
+ 30,
+ 10
+ ],
+ [
+ 12,
+ 43,
+ 21,
+ 9,
+ 23,
+ 47
+ ],
+ [
+ 32,
+ 57,
+ 42,
+ 63,
+ 25,
+ 43
+ ],
+ [
+ 3,
+ 32,
+ 49,
+ 61,
+ 12,
+ 21
+ ],
+ [
+ 5,
+ 36,
+ 16,
+ 22,
+ 62,
+ 42
+ ],
+ [
+ 53,
+ 7,
+ 46,
+ 61,
+ 14,
+ 52
+ ],
+ [
+ 55,
+ 30,
+ 3,
+ 5,
+ 31,
+ 53
+ ],
+ [
+ 0,
+ 44,
+ 15,
+ 18,
+ 19,
+ 28
+ ],
+ [
+ 8,
+ 52,
+ 51,
+ 11,
+ 28,
+ 29
+ ],
+ [
+ 27,
+ 40,
+ 4,
+ 9,
+ 35,
+ 39
+ ],
+ [
+ 14,
+ 26,
+ 3,
+ 16,
+ 48,
+ 21
+ ],
+ [
+ 60,
+ 54,
+ 35,
+ 20,
+ 53,
+ 12
+ ]
+ ],
+ [
+ [
+ 21,
+ 50,
+ 29,
+ 41,
+ 34,
+ 60
+ ],
+ [
+ 28,
+ 51,
+ 60,
+ 33,
+ 14,
+ 45
+ ],
+ [
+ 31,
+ 2,
+ 46,
+ 33,
+ 24,
+ 49
+ ],
+ [
+ 42,
+ 3,
+ 18,
+ 62,
+ 39,
+ 49
+ ],
+ [
+ 25,
+ 15,
+ 62,
+ 27,
+ 12,
+ 11
+ ],
+ [
+ 44,
+ 50,
+ 36,
+ 57,
+ 55,
+ 41
+ ],
+ [
+ 41,
+ 37,
+ 22,
+ 15,
+ 2,
+ 40
+ ],
+ [
+ 36,
+ 62,
+ 53,
+ 30,
+ 14,
+ 57
+ ],
+ [
+ 16,
+ 58,
+ 2,
+ 29,
+ 4,
+ 3
+ ],
+ [
+ 41,
+ 38,
+ 26,
+ 16,
+ 45,
+ 46
+ ],
+ [
+ 45,
+ 46,
+ 32,
+ 41,
+ 56,
+ 26
+ ],
+ [
+ 17,
+ 53,
+ 21,
+ 11,
+ 36,
+ 35
+ ],
+ [
+ 11,
+ 16,
+ 28,
+ 14,
+ 51,
+ 39
+ ],
+ [
+ 9,
+ 35,
+ 33,
+ 22,
+ 52,
+ 62
+ ],
+ [
+ 58,
+ 50,
+ 63,
+ 30,
+ 7,
+ 27
+ ],
+ [
+ 55,
+ 3,
+ 8,
+ 41,
+ 63,
+ 37
+ ],
+ [
+ 3,
+ 51,
+ 32,
+ 46,
+ 15,
+ 6
+ ],
+ [
+ 32,
+ 12,
+ 10,
+ 25,
+ 5,
+ 16
+ ],
+ [
+ 34,
+ 2,
+ 37,
+ 61,
+ 63,
+ 39
+ ],
+ [
+ 42,
+ 22,
+ 27,
+ 53,
+ 11,
+ 56
+ ],
+ [
+ 53,
+ 12,
+ 0,
+ 47,
+ 61,
+ 49
+ ],
+ [
+ 39,
+ 45,
+ 53,
+ 17,
+ 48,
+ 14
+ ],
+ [
+ 6,
+ 0,
+ 4,
+ 53,
+ 25,
+ 11
+ ],
+ [
+ 51,
+ 11,
+ 1,
+ 63,
+ 54,
+ 59
+ ],
+ [
+ 40,
+ 56,
+ 37,
+ 53,
+ 35,
+ 5
+ ],
+ [
+ 59,
+ 28,
+ 41,
+ 10,
+ 1,
+ 45
+ ],
+ [
+ 27,
+ 30,
+ 28,
+ 24,
+ 32,
+ 57
+ ]
+ ],
+ [
+ [
+ 24,
+ 56,
+ 6,
+ 0,
+ 19,
+ 45
+ ],
+ [
+ 11,
+ 57,
+ 59,
+ 25,
+ 46,
+ 30
+ ],
+ [
+ 11,
+ 26,
+ 37,
+ 29,
+ 14,
+ 52
+ ],
+ [
+ 3,
+ 32,
+ 7,
+ 38,
+ 36,
+ 24
+ ],
+ [
+ 61,
+ 2,
+ 24,
+ 14,
+ 51,
+ 44
+ ],
+ [
+ 20,
+ 47,
+ 0,
+ 63,
+ 30,
+ 58
+ ],
+ [
+ 4,
+ 36,
+ 29,
+ 58,
+ 16,
+ 3
+ ],
+ [
+ 20,
+ 0,
+ 45,
+ 14,
+ 28,
+ 44
+ ],
+ [
+ 29,
+ 56,
+ 47,
+ 35,
+ 16,
+ 4
+ ],
+ [
+ 33,
+ 61,
+ 55,
+ 41,
+ 51,
+ 38
+ ],
+ [
+ 58,
+ 1,
+ 38,
+ 14,
+ 4,
+ 19
+ ],
+ [
+ 0,
+ 36,
+ 14,
+ 18,
+ 52,
+ 42
+ ],
+ [
+ 29,
+ 36,
+ 45,
+ 25,
+ 8,
+ 6
+ ],
+ [
+ 6,
+ 57,
+ 50,
+ 40,
+ 58,
+ 61
+ ],
+ [
+ 44,
+ 58,
+ 29,
+ 19,
+ 61,
+ 56
+ ],
+ [
+ 23,
+ 18,
+ 28,
+ 55,
+ 37,
+ 5
+ ],
+ [
+ 13,
+ 9,
+ 19,
+ 43,
+ 37,
+ 3
+ ],
+ [
+ 32,
+ 22,
+ 63,
+ 14,
+ 57,
+ 41
+ ],
+ [
+ 10,
+ 61,
+ 3,
+ 1,
+ 19,
+ 32
+ ],
+ [
+ 16,
+ 55,
+ 10,
+ 41,
+ 59,
+ 22
+ ],
+ [
+ 53,
+ 7,
+ 29,
+ 38,
+ 27,
+ 46
+ ],
+ [
+ 24,
+ 47,
+ 18,
+ 53,
+ 39,
+ 30
+ ],
+ [
+ 0,
+ 33,
+ 19,
+ 51,
+ 5,
+ 17
+ ],
+ [
+ 51,
+ 8,
+ 11,
+ 45,
+ 44,
+ 41
+ ],
+ [
+ 40,
+ 4,
+ 23,
+ 11,
+ 27,
+ 19
+ ],
+ [
+ 16,
+ 18,
+ 3,
+ 48,
+ 51,
+ 21
+ ],
+ [
+ 43,
+ 46,
+ 60,
+ 19,
+ 53,
+ 12
+ ]
+ ],
+ [
+ [
+ 48,
+ 62,
+ 61,
+ 30,
+ 0,
+ 2
+ ],
+ [
+ 45,
+ 3,
+ 35,
+ 54,
+ 29,
+ 2
+ ],
+ [
+ 56,
+ 31,
+ 53,
+ 23,
+ 49,
+ 28
+ ],
+ [
+ 60,
+ 14,
+ 57,
+ 46,
+ 41,
+ 48
+ ],
+ [
+ 18,
+ 61,
+ 59,
+ 14,
+ 44,
+ 32
+ ],
+ [
+ 45,
+ 58,
+ 47,
+ 20,
+ 4,
+ 30
+ ],
+ [
+ 54,
+ 13,
+ 25,
+ 36,
+ 26,
+ 47
+ ],
+ [
+ 20,
+ 12,
+ 0,
+ 47,
+ 30,
+ 45
+ ],
+ [
+ 56,
+ 29,
+ 47,
+ 17,
+ 35,
+ 16
+ ],
+ [
+ 33,
+ 61,
+ 55,
+ 11,
+ 38,
+ 48
+ ],
+ [
+ 58,
+ 19,
+ 14,
+ 1,
+ 38,
+ 36
+ ],
+ [
+ 14,
+ 36,
+ 0,
+ 60,
+ 11,
+ 52
+ ],
+ [
+ 29,
+ 44,
+ 7,
+ 36,
+ 16,
+ 45
+ ],
+ [
+ 6,
+ 47,
+ 50,
+ 33,
+ 42,
+ 62
+ ],
+ [
+ 44,
+ 58,
+ 61,
+ 38,
+ 29,
+ 56
+ ],
+ [
+ 23,
+ 55,
+ 18,
+ 0,
+ 57,
+ 37
+ ],
+ [
+ 9,
+ 12,
+ 43,
+ 19,
+ 13,
+ 6
+ ],
+ [
+ 32,
+ 22,
+ 63,
+ 57,
+ 42,
+ 29
+ ],
+ [
+ 3,
+ 61,
+ 1,
+ 10,
+ 49,
+ 32
+ ],
+ [
+ 5,
+ 16,
+ 55,
+ 36,
+ 22,
+ 59
+ ],
+ [
+ 53,
+ 7,
+ 29,
+ 46,
+ 9,
+ 14
+ ],
+ [
+ 24,
+ 30,
+ 18,
+ 39,
+ 55,
+ 53
+ ],
+ [
+ 33,
+ 0,
+ 19,
+ 44,
+ 51,
+ 5
+ ],
+ [
+ 51,
+ 8,
+ 53,
+ 41,
+ 4,
+ 11
+ ],
+ [
+ 40,
+ 4,
+ 27,
+ 19,
+ 23,
+ 16
+ ],
+ [
+ 16,
+ 14,
+ 48,
+ 3,
+ 21,
+ 26
+ ],
+ [
+ 60,
+ 54,
+ 35,
+ 53,
+ 12,
+ 43
+ ]
+ ],
+ [
+ [
+ 19,
+ 41,
+ 8,
+ 7,
+ 13,
+ 2
+ ],
+ [
+ 48,
+ 46,
+ 62,
+ 29,
+ 5,
+ 41
+ ],
+ [
+ 12,
+ 5,
+ 59,
+ 3,
+ 58,
+ 49
+ ],
+ [
+ 60,
+ 3,
+ 42,
+ 39,
+ 14,
+ 18
+ ],
+ [
+ 42,
+ 12,
+ 27,
+ 11,
+ 25,
+ 19
+ ],
+ [
+ 50,
+ 36,
+ 44,
+ 26,
+ 33,
+ 37
+ ],
+ [
+ 41,
+ 54,
+ 22,
+ 52,
+ 37,
+ 35
+ ],
+ [
+ 62,
+ 30,
+ 36,
+ 53,
+ 10,
+ 14
+ ],
+ [
+ 2,
+ 16,
+ 58,
+ 29,
+ 7,
+ 41
+ ],
+ [
+ 18,
+ 32,
+ 45,
+ 16,
+ 22,
+ 38
+ ],
+ [
+ 45,
+ 56,
+ 41,
+ 10,
+ 3,
+ 46
+ ],
+ [
+ 50,
+ 21,
+ 36,
+ 35,
+ 53,
+ 12
+ ],
+ [
+ 11,
+ 28,
+ 16,
+ 41,
+ 39,
+ 46
+ ],
+ [
+ 16,
+ 9,
+ 33,
+ 38,
+ 28,
+ 19
+ ],
+ [
+ 58,
+ 50,
+ 63,
+ 62,
+ 27,
+ 52
+ ],
+ [
+ 55,
+ 59,
+ 13,
+ 8,
+ 3,
+ 41
+ ],
+ [
+ 3,
+ 51,
+ 15,
+ 46,
+ 57,
+ 47
+ ],
+ [
+ 32,
+ 30,
+ 10,
+ 12,
+ 25,
+ 18
+ ],
+ [
+ 34,
+ 2,
+ 27,
+ 61,
+ 53,
+ 59
+ ],
+ [
+ 42,
+ 22,
+ 56,
+ 53,
+ 44,
+ 34
+ ],
+ [
+ 53,
+ 12,
+ 49,
+ 41,
+ 44,
+ 8
+ ],
+ [
+ 45,
+ 1,
+ 48,
+ 47,
+ 16,
+ 17
+ ],
+ [
+ 4,
+ 0,
+ 25,
+ 53,
+ 24,
+ 11
+ ],
+ [
+ 1,
+ 44,
+ 11,
+ 45,
+ 34,
+ 51
+ ],
+ [
+ 40,
+ 5,
+ 53,
+ 6,
+ 22,
+ 18
+ ],
+ [
+ 28,
+ 10,
+ 1,
+ 15,
+ 3,
+ 4
+ ],
+ [
+ 30,
+ 27,
+ 24,
+ 57,
+ 32,
+ 16
+ ]
+ ],
+ [
+ [
+ 24,
+ 56,
+ 6,
+ 0,
+ 19,
+ 45
+ ],
+ [
+ 11,
+ 57,
+ 59,
+ 46,
+ 25,
+ 30
+ ],
+ [
+ 26,
+ 11,
+ 37,
+ 14,
+ 29,
+ 49
+ ],
+ [
+ 38,
+ 36,
+ 3,
+ 24,
+ 18,
+ 20
+ ],
+ [
+ 61,
+ 51,
+ 14,
+ 2,
+ 24,
+ 1
+ ],
+ [
+ 20,
+ 0,
+ 47,
+ 30,
+ 8,
+ 35
+ ],
+ [
+ 4,
+ 58,
+ 36,
+ 54,
+ 29,
+ 12
+ ],
+ [
+ 20,
+ 58,
+ 44,
+ 28,
+ 45,
+ 9
+ ],
+ [
+ 56,
+ 47,
+ 10,
+ 29,
+ 35,
+ 27
+ ],
+ [
+ 61,
+ 33,
+ 55,
+ 54,
+ 4,
+ 36
+ ],
+ [
+ 58,
+ 1,
+ 14,
+ 4,
+ 38,
+ 52
+ ],
+ [
+ 14,
+ 0,
+ 36,
+ 63,
+ 15,
+ 23
+ ],
+ [
+ 29,
+ 36,
+ 44,
+ 8,
+ 16,
+ 2
+ ],
+ [
+ 6,
+ 40,
+ 27,
+ 57,
+ 50,
+ 42
+ ],
+ [
+ 19,
+ 44,
+ 58,
+ 61,
+ 38,
+ 37
+ ],
+ [
+ 23,
+ 18,
+ 17,
+ 57,
+ 13,
+ 40
+ ],
+ [
+ 13,
+ 9,
+ 19,
+ 37,
+ 15,
+ 50
+ ],
+ [
+ 32,
+ 14,
+ 57,
+ 58,
+ 29,
+ 22
+ ],
+ [
+ 10,
+ 61,
+ 1,
+ 3,
+ 14,
+ 59
+ ],
+ [
+ 55,
+ 16,
+ 34,
+ 18,
+ 22,
+ 49
+ ],
+ [
+ 53,
+ 27,
+ 38,
+ 28,
+ 23,
+ 44
+ ],
+ [
+ 24,
+ 47,
+ 18,
+ 41,
+ 62,
+ 30
+ ],
+ [
+ 33,
+ 51,
+ 19,
+ 5,
+ 0,
+ 44
+ ],
+ [
+ 51,
+ 8,
+ 25,
+ 53,
+ 27,
+ 16
+ ],
+ [
+ 40,
+ 11,
+ 27,
+ 4,
+ 23,
+ 19
+ ],
+ [
+ 16,
+ 18,
+ 51,
+ 48,
+ 3,
+ 47
+ ],
+ [
+ 46,
+ 43,
+ 36,
+ 9,
+ 5,
+ 12
+ ]
+ ],
+ [
+ [
+ 37,
+ 10,
+ 46,
+ 60,
+ 61,
+ 55
+ ],
+ [
+ 35,
+ 53,
+ 34,
+ 43,
+ 19,
+ 57
+ ],
+ [
+ 49,
+ 56,
+ 45,
+ 30,
+ 6,
+ 12
+ ],
+ [
+ 60,
+ 27,
+ 14,
+ 48,
+ 46,
+ 57
+ ],
+ [
+ 61,
+ 59,
+ 14,
+ 41,
+ 16,
+ 1
+ ],
+ [
+ 45,
+ 4,
+ 3,
+ 24,
+ 58,
+ 47
+ ],
+ [
+ 54,
+ 13,
+ 9,
+ 43,
+ 16,
+ 26
+ ],
+ [
+ 47,
+ 23,
+ 12,
+ 20,
+ 63,
+ 30
+ ],
+ [
+ 23,
+ 44,
+ 56,
+ 29,
+ 47,
+ 17
+ ],
+ [
+ 33,
+ 60,
+ 61,
+ 48,
+ 41,
+ 14
+ ],
+ [
+ 58,
+ 63,
+ 19,
+ 11,
+ 38,
+ 9
+ ],
+ [
+ 60,
+ 63,
+ 0,
+ 36,
+ 15,
+ 9
+ ],
+ [
+ 29,
+ 36,
+ 30,
+ 59,
+ 11,
+ 27
+ ],
+ [
+ 6,
+ 7,
+ 47,
+ 62,
+ 50,
+ 57
+ ],
+ [
+ 27,
+ 58,
+ 19,
+ 46,
+ 29,
+ 56
+ ],
+ [
+ 29,
+ 60,
+ 56,
+ 55,
+ 23,
+ 26
+ ],
+ [
+ 53,
+ 59,
+ 6,
+ 9,
+ 43,
+ 16
+ ],
+ [
+ 41,
+ 32,
+ 57,
+ 63,
+ 18,
+ 37
+ ],
+ [
+ 42,
+ 61,
+ 3,
+ 10,
+ 34,
+ 40
+ ],
+ [
+ 4,
+ 43,
+ 17,
+ 16,
+ 52,
+ 60
+ ],
+ [
+ 45,
+ 53,
+ 61,
+ 56,
+ 16,
+ 7
+ ],
+ [
+ 55,
+ 9,
+ 18,
+ 61,
+ 45,
+ 60
+ ],
+ [
+ 60,
+ 47,
+ 53,
+ 33,
+ 12,
+ 27
+ ],
+ [
+ 43,
+ 51,
+ 11,
+ 8,
+ 45,
+ 63
+ ],
+ [
+ 6,
+ 40,
+ 15,
+ 27,
+ 26,
+ 23
+ ],
+ [
+ 49,
+ 14,
+ 9,
+ 58,
+ 21,
+ 63
+ ],
+ [
+ 63,
+ 24,
+ 60,
+ 31,
+ 12,
+ 34
+ ]
+ ],
+ [
+ [
+ 16,
+ 13,
+ 44,
+ 4,
+ 23,
+ 46
+ ],
+ [
+ 16,
+ 50,
+ 9,
+ 13,
+ 23,
+ 36
+ ],
+ [
+ 11,
+ 35,
+ 21,
+ 7,
+ 59,
+ 8
+ ],
+ [
+ 1,
+ 3,
+ 25,
+ 15,
+ 60,
+ 39
+ ],
+ [
+ 54,
+ 61,
+ 31,
+ 35,
+ 55,
+ 1
+ ],
+ [
+ 51,
+ 52,
+ 46,
+ 15,
+ 4,
+ 45
+ ],
+ [
+ 60,
+ 54,
+ 59,
+ 44,
+ 10,
+ 7
+ ],
+ [
+ 12,
+ 22,
+ 14,
+ 47,
+ 0,
+ 30
+ ],
+ [
+ 42,
+ 29,
+ 23,
+ 56,
+ 33,
+ 47
+ ],
+ [
+ 33,
+ 61,
+ 20,
+ 60,
+ 0,
+ 49
+ ],
+ [
+ 35,
+ 58,
+ 63,
+ 14,
+ 51,
+ 24
+ ],
+ [
+ 29,
+ 33,
+ 36,
+ 60,
+ 0,
+ 49
+ ],
+ [
+ 29,
+ 17,
+ 30,
+ 31,
+ 12,
+ 36
+ ],
+ [
+ 6,
+ 0,
+ 61,
+ 48,
+ 50,
+ 3
+ ],
+ [
+ 8,
+ 6,
+ 58,
+ 29,
+ 37,
+ 19
+ ],
+ [
+ 60,
+ 39,
+ 27,
+ 19,
+ 1,
+ 57
+ ],
+ [
+ 56,
+ 9,
+ 30,
+ 6,
+ 43,
+ 10
+ ],
+ [
+ 32,
+ 20,
+ 13,
+ 57,
+ 63,
+ 49
+ ],
+ [
+ 8,
+ 42,
+ 21,
+ 61,
+ 37,
+ 4
+ ],
+ [
+ 45,
+ 49,
+ 16,
+ 13,
+ 2,
+ 58
+ ],
+ [
+ 53,
+ 29,
+ 14,
+ 50,
+ 61,
+ 3
+ ],
+ [
+ 37,
+ 57,
+ 27,
+ 54,
+ 46,
+ 9
+ ],
+ [
+ 52,
+ 19,
+ 22,
+ 0,
+ 18,
+ 5
+ ],
+ [
+ 14,
+ 49,
+ 30,
+ 33,
+ 53,
+ 34
+ ],
+ [
+ 13,
+ 9,
+ 4,
+ 40,
+ 23,
+ 39
+ ],
+ [
+ 27,
+ 43,
+ 47,
+ 36,
+ 49,
+ 3
+ ],
+ [
+ 59,
+ 43,
+ 40,
+ 28,
+ 0,
+ 33
+ ]
+ ],
+ [
+ [
+ 48,
+ 50,
+ 42,
+ 63,
+ 34,
+ 38
+ ],
+ [
+ 3,
+ 40,
+ 61,
+ 62,
+ 2,
+ 6
+ ],
+ [
+ 39,
+ 7,
+ 36,
+ 6,
+ 45,
+ 40
+ ],
+ [
+ 41,
+ 35,
+ 46,
+ 13,
+ 63,
+ 56
+ ],
+ [
+ 6,
+ 1,
+ 54,
+ 37,
+ 38,
+ 34
+ ],
+ [
+ 59,
+ 46,
+ 51,
+ 31,
+ 4,
+ 52
+ ],
+ [
+ 60,
+ 44,
+ 11,
+ 54,
+ 4,
+ 24
+ ],
+ [
+ 12,
+ 0,
+ 2,
+ 63,
+ 50,
+ 47
+ ],
+ [
+ 33,
+ 42,
+ 29,
+ 23,
+ 16,
+ 56
+ ],
+ [
+ 20,
+ 61,
+ 33,
+ 60,
+ 53,
+ 0
+ ],
+ [
+ 35,
+ 58,
+ 63,
+ 9,
+ 8,
+ 19
+ ],
+ [
+ 59,
+ 48,
+ 36,
+ 60,
+ 10,
+ 14
+ ],
+ [
+ 29,
+ 44,
+ 7,
+ 17,
+ 36,
+ 12
+ ],
+ [
+ 47,
+ 27,
+ 6,
+ 62,
+ 42,
+ 48
+ ],
+ [
+ 18,
+ 58,
+ 49,
+ 46,
+ 42,
+ 38
+ ],
+ [
+ 60,
+ 34,
+ 27,
+ 18,
+ 23,
+ 55
+ ],
+ [
+ 43,
+ 40,
+ 9,
+ 50,
+ 18,
+ 45
+ ],
+ [
+ 32,
+ 57,
+ 48,
+ 42,
+ 29,
+ 63
+ ],
+ [
+ 42,
+ 61,
+ 49,
+ 3,
+ 32,
+ 1
+ ],
+ [
+ 23,
+ 37,
+ 1,
+ 16,
+ 36,
+ 39
+ ],
+ [
+ 53,
+ 21,
+ 7,
+ 61,
+ 50,
+ 31
+ ],
+ [
+ 8,
+ 60,
+ 18,
+ 24,
+ 9,
+ 30
+ ],
+ [
+ 51,
+ 33,
+ 28,
+ 5,
+ 8,
+ 44
+ ],
+ [
+ 51,
+ 52,
+ 8,
+ 4,
+ 41,
+ 45
+ ],
+ [
+ 40,
+ 4,
+ 27,
+ 9,
+ 60,
+ 19
+ ],
+ [
+ 3,
+ 61,
+ 16,
+ 26,
+ 48,
+ 12
+ ],
+ [
+ 54,
+ 61,
+ 35,
+ 1,
+ 53,
+ 20
+ ]
+ ],
+ [
+ [
+ 62,
+ 28,
+ 1,
+ 42,
+ 8,
+ 55
+ ],
+ [
+ 18,
+ 12,
+ 8,
+ 41,
+ 40,
+ 31
+ ],
+ [
+ 12,
+ 6,
+ 50,
+ 4,
+ 23,
+ 45
+ ],
+ [
+ 43,
+ 35,
+ 8,
+ 20,
+ 42,
+ 46
+ ],
+ [
+ 39,
+ 41,
+ 29,
+ 22,
+ 3,
+ 56
+ ],
+ [
+ 61,
+ 45,
+ 46,
+ 48,
+ 28,
+ 51
+ ],
+ [
+ 44,
+ 4,
+ 11,
+ 25,
+ 54,
+ 59
+ ],
+ [
+ 12,
+ 33,
+ 56,
+ 52,
+ 30,
+ 17
+ ],
+ [
+ 55,
+ 29,
+ 17,
+ 42,
+ 23,
+ 14
+ ],
+ [
+ 60,
+ 12,
+ 18,
+ 61,
+ 33,
+ 28
+ ],
+ [
+ 35,
+ 58,
+ 37,
+ 63,
+ 6,
+ 27
+ ],
+ [
+ 48,
+ 59,
+ 10,
+ 36,
+ 60,
+ 58
+ ],
+ [
+ 17,
+ 7,
+ 28,
+ 31,
+ 29,
+ 44
+ ],
+ [
+ 47,
+ 42,
+ 50,
+ 6,
+ 8,
+ 14
+ ],
+ [
+ 39,
+ 58,
+ 37,
+ 59,
+ 56,
+ 18
+ ],
+ [
+ 60,
+ 57,
+ 18,
+ 9,
+ 55,
+ 23
+ ],
+ [
+ 43,
+ 63,
+ 18,
+ 60,
+ 22,
+ 19
+ ],
+ [
+ 32,
+ 1,
+ 42,
+ 57,
+ 35,
+ 63
+ ],
+ [
+ 42,
+ 61,
+ 3,
+ 32,
+ 1,
+ 50
+ ],
+ [
+ 37,
+ 36,
+ 10,
+ 23,
+ 16,
+ 57
+ ],
+ [
+ 53,
+ 61,
+ 7,
+ 57,
+ 21,
+ 23
+ ],
+ [
+ 9,
+ 39,
+ 30,
+ 18,
+ 14,
+ 8
+ ],
+ [
+ 33,
+ 44,
+ 8,
+ 5,
+ 0,
+ 56
+ ],
+ [
+ 51,
+ 53,
+ 49,
+ 52,
+ 4,
+ 41
+ ],
+ [
+ 40,
+ 27,
+ 4,
+ 6,
+ 9,
+ 16
+ ],
+ [
+ 3,
+ 16,
+ 48,
+ 26,
+ 12,
+ 4
+ ],
+ [
+ 61,
+ 14,
+ 12,
+ 54,
+ 35,
+ 53
+ ]
+ ],
+ [
+ [
+ 47,
+ 23,
+ 63,
+ 11,
+ 61,
+ 55
+ ],
+ [
+ 17,
+ 44,
+ 28,
+ 39,
+ 47,
+ 27
+ ],
+ [
+ 34,
+ 53,
+ 50,
+ 38,
+ 29,
+ 5
+ ],
+ [
+ 11,
+ 10,
+ 17,
+ 52,
+ 47,
+ 42
+ ],
+ [
+ 15,
+ 41,
+ 27,
+ 20,
+ 12,
+ 6
+ ],
+ [
+ 34,
+ 44,
+ 50,
+ 39,
+ 36,
+ 61
+ ],
+ [
+ 37,
+ 41,
+ 52,
+ 29,
+ 46,
+ 47
+ ],
+ [
+ 62,
+ 36,
+ 34,
+ 30,
+ 39,
+ 22
+ ],
+ [
+ 62,
+ 16,
+ 58,
+ 5,
+ 2,
+ 8
+ ],
+ [
+ 32,
+ 41,
+ 56,
+ 12,
+ 46,
+ 8
+ ],
+ [
+ 10,
+ 35,
+ 45,
+ 41,
+ 3,
+ 56
+ ],
+ [
+ 50,
+ 48,
+ 35,
+ 53,
+ 36,
+ 12
+ ],
+ [
+ 39,
+ 11,
+ 46,
+ 7,
+ 23,
+ 51
+ ],
+ [
+ 9,
+ 47,
+ 19,
+ 22,
+ 52,
+ 34
+ ],
+ [
+ 35,
+ 18,
+ 56,
+ 50,
+ 3,
+ 23
+ ],
+ [
+ 3,
+ 60,
+ 38,
+ 36,
+ 9,
+ 58
+ ],
+ [
+ 46,
+ 28,
+ 32,
+ 5,
+ 43,
+ 56
+ ],
+ [
+ 30,
+ 32,
+ 57,
+ 42,
+ 52,
+ 1
+ ],
+ [
+ 2,
+ 34,
+ 32,
+ 61,
+ 14,
+ 42
+ ],
+ [
+ 42,
+ 37,
+ 20,
+ 48,
+ 50,
+ 9
+ ],
+ [
+ 53,
+ 7,
+ 56,
+ 25,
+ 13,
+ 60
+ ],
+ [
+ 17,
+ 39,
+ 14,
+ 53,
+ 30,
+ 25
+ ],
+ [
+ 5,
+ 40,
+ 6,
+ 33,
+ 29,
+ 35
+ ],
+ [
+ 51,
+ 4,
+ 11,
+ 57,
+ 58,
+ 28
+ ],
+ [
+ 40,
+ 37,
+ 4,
+ 44,
+ 8,
+ 48
+ ],
+ [
+ 3,
+ 48,
+ 26,
+ 9,
+ 12,
+ 41
+ ],
+ [
+ 47,
+ 61,
+ 26,
+ 24,
+ 20,
+ 53
+ ]
+ ],
+ [
+ [
+ 31,
+ 43,
+ 41,
+ 47,
+ 11,
+ 25
+ ],
+ [
+ 50,
+ 25,
+ 31,
+ 40,
+ 24,
+ 46
+ ],
+ [
+ 23,
+ 9,
+ 62,
+ 15,
+ 20,
+ 53
+ ],
+ [
+ 4,
+ 47,
+ 44,
+ 58,
+ 48,
+ 25
+ ],
+ [
+ 2,
+ 19,
+ 12,
+ 52,
+ 0,
+ 40
+ ],
+ [
+ 49,
+ 15,
+ 24,
+ 34,
+ 60,
+ 42
+ ],
+ [
+ 12,
+ 46,
+ 17,
+ 29,
+ 41,
+ 3
+ ],
+ [
+ 39,
+ 60,
+ 44,
+ 41,
+ 33,
+ 36
+ ],
+ [
+ 21,
+ 60,
+ 16,
+ 44,
+ 51,
+ 57
+ ],
+ [
+ 24,
+ 41,
+ 12,
+ 33,
+ 13,
+ 21
+ ],
+ [
+ 43,
+ 62,
+ 3,
+ 12,
+ 28,
+ 45
+ ],
+ [
+ 58,
+ 19,
+ 39,
+ 17,
+ 49,
+ 42
+ ],
+ [
+ 25,
+ 54,
+ 4,
+ 11,
+ 7,
+ 39
+ ],
+ [
+ 34,
+ 35,
+ 4,
+ 42,
+ 62,
+ 19
+ ],
+ [
+ 43,
+ 41,
+ 42,
+ 35,
+ 40,
+ 32
+ ],
+ [
+ 21,
+ 63,
+ 3,
+ 17,
+ 20,
+ 50
+ ],
+ [
+ 58,
+ 46,
+ 44,
+ 1,
+ 25,
+ 20
+ ],
+ [
+ 26,
+ 32,
+ 16,
+ 25,
+ 46,
+ 41
+ ],
+ [
+ 37,
+ 63,
+ 61,
+ 28,
+ 24,
+ 56
+ ],
+ [
+ 33,
+ 42,
+ 40,
+ 37,
+ 48,
+ 50
+ ],
+ [
+ 11,
+ 53,
+ 25,
+ 39,
+ 4,
+ 61
+ ],
+ [
+ 12,
+ 54,
+ 27,
+ 4,
+ 50,
+ 14
+ ],
+ [
+ 47,
+ 19,
+ 42,
+ 17,
+ 35,
+ 40
+ ],
+ [
+ 54,
+ 40,
+ 60,
+ 63,
+ 45,
+ 57
+ ],
+ [
+ 44,
+ 56,
+ 40,
+ 62,
+ 37,
+ 3
+ ],
+ [
+ 59,
+ 41,
+ 57,
+ 34,
+ 48,
+ 22
+ ],
+ [
+ 56,
+ 13,
+ 59,
+ 51,
+ 26,
+ 23
+ ]
+ ],
+ [
+ [
+ 35,
+ 32,
+ 8,
+ 40,
+ 51,
+ 52
+ ],
+ [
+ 52,
+ 5,
+ 22,
+ 21,
+ 6,
+ 33
+ ],
+ [
+ 22,
+ 58,
+ 11,
+ 25,
+ 3,
+ 51
+ ],
+ [
+ 63,
+ 2,
+ 56,
+ 4,
+ 23,
+ 54
+ ],
+ [
+ 39,
+ 12,
+ 23,
+ 32,
+ 30,
+ 46
+ ],
+ [
+ 50,
+ 34,
+ 36,
+ 58,
+ 26,
+ 28
+ ],
+ [
+ 46,
+ 41,
+ 3,
+ 2,
+ 22,
+ 16
+ ],
+ [
+ 60,
+ 36,
+ 53,
+ 30,
+ 54,
+ 39
+ ],
+ [
+ 16,
+ 51,
+ 3,
+ 39,
+ 2,
+ 26
+ ],
+ [
+ 45,
+ 26,
+ 18,
+ 41,
+ 32,
+ 46
+ ],
+ [
+ 45,
+ 3,
+ 10,
+ 56,
+ 36,
+ 35
+ ],
+ [
+ 21,
+ 36,
+ 35,
+ 50,
+ 11,
+ 19
+ ],
+ [
+ 28,
+ 11,
+ 46,
+ 59,
+ 41,
+ 13
+ ],
+ [
+ 23,
+ 16,
+ 38,
+ 19,
+ 15,
+ 22
+ ],
+ [
+ 27,
+ 7,
+ 34,
+ 58,
+ 3,
+ 42
+ ],
+ [
+ 9,
+ 22,
+ 46,
+ 36,
+ 26,
+ 41
+ ],
+ [
+ 3,
+ 51,
+ 56,
+ 40,
+ 46,
+ 8
+ ],
+ [
+ 12,
+ 21,
+ 25,
+ 50,
+ 17,
+ 62
+ ],
+ [
+ 27,
+ 34,
+ 61,
+ 13,
+ 60,
+ 11
+ ],
+ [
+ 53,
+ 12,
+ 56,
+ 0,
+ 42,
+ 33
+ ],
+ [
+ 53,
+ 37,
+ 12,
+ 24,
+ 63,
+ 25
+ ],
+ [
+ 45,
+ 55,
+ 18,
+ 26,
+ 17,
+ 43
+ ],
+ [
+ 4,
+ 25,
+ 32,
+ 1,
+ 48,
+ 53
+ ],
+ [
+ 17,
+ 27,
+ 63,
+ 4,
+ 62,
+ 34
+ ],
+ [
+ 6,
+ 52,
+ 62,
+ 40,
+ 46,
+ 54
+ ],
+ [
+ 10,
+ 42,
+ 28,
+ 49,
+ 3,
+ 1
+ ],
+ [
+ 45,
+ 27,
+ 41,
+ 21,
+ 16,
+ 47
+ ]
+ ],
+ [
+ [
+ 44,
+ 24,
+ 56,
+ 33,
+ 15,
+ 41
+ ],
+ [
+ 38,
+ 26,
+ 24,
+ 29,
+ 19,
+ 53
+ ],
+ [
+ 12,
+ 15,
+ 29,
+ 9,
+ 1,
+ 63
+ ],
+ [
+ 58,
+ 38,
+ 50,
+ 0,
+ 43,
+ 61
+ ],
+ [
+ 24,
+ 51,
+ 31,
+ 34,
+ 60,
+ 7
+ ],
+ [
+ 0,
+ 7,
+ 22,
+ 43,
+ 35,
+ 1
+ ],
+ [
+ 63,
+ 36,
+ 11,
+ 1,
+ 16,
+ 4
+ ],
+ [
+ 8,
+ 50,
+ 56,
+ 4,
+ 30,
+ 55
+ ],
+ [
+ 43,
+ 16,
+ 42,
+ 29,
+ 60,
+ 35
+ ],
+ [
+ 34,
+ 0,
+ 9,
+ 22,
+ 18,
+ 26
+ ],
+ [
+ 54,
+ 51,
+ 45,
+ 35,
+ 2,
+ 36
+ ],
+ [
+ 37,
+ 36,
+ 43,
+ 60,
+ 11,
+ 59
+ ],
+ [
+ 56,
+ 38,
+ 10,
+ 28,
+ 14,
+ 43
+ ],
+ [
+ 30,
+ 0,
+ 58,
+ 62,
+ 22,
+ 19
+ ],
+ [
+ 7,
+ 55,
+ 42,
+ 58,
+ 30,
+ 38
+ ],
+ [
+ 11,
+ 33,
+ 1,
+ 39,
+ 19,
+ 16
+ ],
+ [
+ 55,
+ 20,
+ 40,
+ 9,
+ 18,
+ 30
+ ],
+ [
+ 18,
+ 20,
+ 57,
+ 32,
+ 45,
+ 1
+ ],
+ [
+ 43,
+ 61,
+ 12,
+ 32,
+ 31,
+ 30
+ ],
+ [
+ 23,
+ 25,
+ 7,
+ 28,
+ 40,
+ 19
+ ],
+ [
+ 14,
+ 51,
+ 48,
+ 58,
+ 53,
+ 25
+ ],
+ [
+ 18,
+ 30,
+ 1,
+ 49,
+ 41,
+ 9
+ ],
+ [
+ 2,
+ 51,
+ 22,
+ 0,
+ 52,
+ 5
+ ],
+ [
+ 53,
+ 4,
+ 47,
+ 52,
+ 51,
+ 40
+ ],
+ [
+ 40,
+ 16,
+ 9,
+ 47,
+ 23,
+ 11
+ ],
+ [
+ 47,
+ 3,
+ 43,
+ 46,
+ 26,
+ 53
+ ],
+ [
+ 8,
+ 40,
+ 18,
+ 46,
+ 33,
+ 63
+ ]
+ ],
+ [
+ [
+ 48,
+ 38,
+ 50,
+ 42,
+ 63,
+ 36
+ ],
+ [
+ 3,
+ 26,
+ 10,
+ 2,
+ 6,
+ 61
+ ],
+ [
+ 39,
+ 44,
+ 45,
+ 40,
+ 6,
+ 7
+ ],
+ [
+ 41,
+ 5,
+ 20,
+ 49,
+ 56,
+ 13
+ ],
+ [
+ 6,
+ 1,
+ 30,
+ 37,
+ 28,
+ 38
+ ],
+ [
+ 59,
+ 46,
+ 22,
+ 35,
+ 61,
+ 0
+ ],
+ [
+ 1,
+ 63,
+ 35,
+ 3,
+ 60,
+ 49
+ ],
+ [
+ 8,
+ 12,
+ 2,
+ 50,
+ 5,
+ 55
+ ],
+ [
+ 42,
+ 33,
+ 43,
+ 16,
+ 32,
+ 29
+ ],
+ [
+ 9,
+ 34,
+ 0,
+ 20,
+ 41,
+ 31
+ ],
+ [
+ 51,
+ 54,
+ 8,
+ 19,
+ 63,
+ 9
+ ],
+ [
+ 37,
+ 56,
+ 36,
+ 11,
+ 59,
+ 43
+ ],
+ [
+ 38,
+ 10,
+ 28,
+ 17,
+ 56,
+ 63
+ ],
+ [
+ 27,
+ 30,
+ 42,
+ 19,
+ 0,
+ 22
+ ],
+ [
+ 7,
+ 55,
+ 49,
+ 42,
+ 58,
+ 38
+ ],
+ [
+ 29,
+ 34,
+ 39,
+ 33,
+ 47,
+ 45
+ ],
+ [
+ 55,
+ 40,
+ 20,
+ 18,
+ 7,
+ 5
+ ],
+ [
+ 43,
+ 57,
+ 39,
+ 54,
+ 48,
+ 28
+ ],
+ [
+ 12,
+ 43,
+ 61,
+ 42,
+ 32,
+ 49
+ ],
+ [
+ 23,
+ 36,
+ 1,
+ 7,
+ 59,
+ 28
+ ],
+ [
+ 14,
+ 53,
+ 21,
+ 7,
+ 57,
+ 37
+ ],
+ [
+ 18,
+ 1,
+ 24,
+ 60,
+ 30,
+ 9
+ ],
+ [
+ 51,
+ 0,
+ 33,
+ 2,
+ 44,
+ 5
+ ],
+ [
+ 52,
+ 29,
+ 4,
+ 41,
+ 54,
+ 58
+ ],
+ [
+ 40,
+ 19,
+ 16,
+ 46,
+ 9,
+ 47
+ ],
+ [
+ 61,
+ 3,
+ 47,
+ 22,
+ 21,
+ 53
+ ],
+ [
+ 35,
+ 60,
+ 54,
+ 5,
+ 1,
+ 40
+ ]
+ ],
+ [
+ [
+ 17,
+ 18,
+ 8,
+ 53,
+ 25,
+ 43
+ ],
+ [
+ 9,
+ 38,
+ 24,
+ 47,
+ 25,
+ 63
+ ],
+ [
+ 20,
+ 24,
+ 5,
+ 12,
+ 54,
+ 28
+ ],
+ [
+ 43,
+ 10,
+ 20,
+ 42,
+ 11,
+ 8
+ ],
+ [
+ 53,
+ 61,
+ 30,
+ 39,
+ 29,
+ 18
+ ],
+ [
+ 61,
+ 56,
+ 25,
+ 40,
+ 5,
+ 22
+ ],
+ [
+ 62,
+ 17,
+ 24,
+ 1,
+ 47,
+ 33
+ ],
+ [
+ 41,
+ 16,
+ 34,
+ 39,
+ 29,
+ 8
+ ],
+ [
+ 39,
+ 16,
+ 36,
+ 42,
+ 29,
+ 23
+ ],
+ [
+ 9,
+ 11,
+ 41,
+ 63,
+ 56,
+ 31
+ ],
+ [
+ 48,
+ 51,
+ 10,
+ 62,
+ 63,
+ 45
+ ],
+ [
+ 36,
+ 11,
+ 37,
+ 42,
+ 58,
+ 46
+ ],
+ [
+ 51,
+ 38,
+ 25,
+ 63,
+ 29,
+ 44
+ ],
+ [
+ 4,
+ 56,
+ 44,
+ 62,
+ 58,
+ 30
+ ],
+ [
+ 3,
+ 7,
+ 42,
+ 46,
+ 33,
+ 35
+ ],
+ [
+ 39,
+ 9,
+ 33,
+ 58,
+ 60,
+ 29
+ ],
+ [
+ 40,
+ 37,
+ 20,
+ 16,
+ 55,
+ 25
+ ],
+ [
+ 54,
+ 19,
+ 11,
+ 57,
+ 0,
+ 39
+ ],
+ [
+ 12,
+ 43,
+ 61,
+ 25,
+ 49,
+ 32
+ ],
+ [
+ 4,
+ 23,
+ 54,
+ 36,
+ 7,
+ 28
+ ],
+ [
+ 40,
+ 25,
+ 26,
+ 14,
+ 2,
+ 58
+ ],
+ [
+ 18,
+ 24,
+ 58,
+ 1,
+ 22,
+ 46
+ ],
+ [
+ 2,
+ 63,
+ 22,
+ 6,
+ 44,
+ 56
+ ],
+ [
+ 52,
+ 29,
+ 51,
+ 4,
+ 40,
+ 32
+ ],
+ [
+ 40,
+ 17,
+ 15,
+ 16,
+ 46,
+ 57
+ ],
+ [
+ 9,
+ 61,
+ 3,
+ 47,
+ 24,
+ 11
+ ],
+ [
+ 2,
+ 39,
+ 24,
+ 42,
+ 0,
+ 44
+ ]
+ ],
+ [
+ [
+ 0,
+ 10,
+ 49,
+ 23,
+ 62,
+ 44
+ ],
+ [
+ 28,
+ 0,
+ 36,
+ 26,
+ 47,
+ 52
+ ],
+ [
+ 30,
+ 4,
+ 16,
+ 48,
+ 40,
+ 10
+ ],
+ [
+ 61,
+ 32,
+ 26,
+ 16,
+ 33,
+ 62
+ ],
+ [
+ 30,
+ 39,
+ 53,
+ 5,
+ 57,
+ 20
+ ],
+ [
+ 5,
+ 37,
+ 61,
+ 15,
+ 25,
+ 6
+ ],
+ [
+ 15,
+ 17,
+ 24,
+ 60,
+ 49,
+ 62
+ ],
+ [
+ 34,
+ 39,
+ 61,
+ 0,
+ 58,
+ 40
+ ],
+ [
+ 16,
+ 39,
+ 36,
+ 51,
+ 2,
+ 29
+ ],
+ [
+ 9,
+ 11,
+ 41,
+ 31,
+ 56,
+ 52
+ ],
+ [
+ 10,
+ 48,
+ 24,
+ 45,
+ 62,
+ 51
+ ],
+ [
+ 11,
+ 38,
+ 36,
+ 37,
+ 6,
+ 42
+ ],
+ [
+ 51,
+ 50,
+ 15,
+ 30,
+ 25,
+ 38
+ ],
+ [
+ 4,
+ 19,
+ 24,
+ 35,
+ 31,
+ 48
+ ],
+ [
+ 7,
+ 46,
+ 3,
+ 58,
+ 30,
+ 41
+ ],
+ [
+ 58,
+ 9,
+ 39,
+ 32,
+ 29,
+ 40
+ ],
+ [
+ 40,
+ 37,
+ 20,
+ 8,
+ 25,
+ 55
+ ],
+ [
+ 19,
+ 0,
+ 54,
+ 52,
+ 39,
+ 17
+ ],
+ [
+ 25,
+ 43,
+ 12,
+ 61,
+ 11,
+ 14
+ ],
+ [
+ 23,
+ 4,
+ 54,
+ 36,
+ 28,
+ 33
+ ],
+ [
+ 40,
+ 2,
+ 25,
+ 58,
+ 36,
+ 53
+ ],
+ [
+ 18,
+ 46,
+ 35,
+ 22,
+ 53,
+ 16
+ ],
+ [
+ 2,
+ 6,
+ 63,
+ 14,
+ 42,
+ 11
+ ],
+ [
+ 35,
+ 7,
+ 52,
+ 40,
+ 29,
+ 57
+ ],
+ [
+ 40,
+ 15,
+ 19,
+ 57,
+ 17,
+ 23
+ ],
+ [
+ 9,
+ 11,
+ 47,
+ 22,
+ 49,
+ 1
+ ],
+ [
+ 24,
+ 39,
+ 42,
+ 2,
+ 16,
+ 22
+ ]
+ ],
+ [
+ [
+ 55,
+ 39,
+ 9,
+ 43,
+ 21,
+ 46
+ ],
+ [
+ 56,
+ 0,
+ 63,
+ 39,
+ 30,
+ 41
+ ],
+ [
+ 20,
+ 1,
+ 26,
+ 58,
+ 34,
+ 19
+ ],
+ [
+ 54,
+ 24,
+ 32,
+ 51,
+ 26,
+ 44
+ ],
+ [
+ 30,
+ 53,
+ 56,
+ 39,
+ 34,
+ 40
+ ],
+ [
+ 5,
+ 37,
+ 25,
+ 50,
+ 6,
+ 61
+ ],
+ [
+ 24,
+ 49,
+ 37,
+ 15,
+ 6,
+ 29
+ ],
+ [
+ 34,
+ 16,
+ 30,
+ 61,
+ 10,
+ 36
+ ],
+ [
+ 16,
+ 29,
+ 2,
+ 5,
+ 51,
+ 26
+ ],
+ [
+ 9,
+ 56,
+ 11,
+ 31,
+ 46,
+ 45
+ ],
+ [
+ 10,
+ 45,
+ 56,
+ 62,
+ 25,
+ 36
+ ],
+ [
+ 11,
+ 6,
+ 35,
+ 36,
+ 1,
+ 52
+ ],
+ [
+ 51,
+ 50,
+ 41,
+ 46,
+ 38,
+ 4
+ ],
+ [
+ 19,
+ 33,
+ 41,
+ 16,
+ 31,
+ 52
+ ],
+ [
+ 34,
+ 7,
+ 17,
+ 47,
+ 63,
+ 3
+ ],
+ [
+ 58,
+ 9,
+ 22,
+ 61,
+ 59,
+ 8
+ ],
+ [
+ 40,
+ 37,
+ 3,
+ 51,
+ 22,
+ 25
+ ],
+ [
+ 12,
+ 52,
+ 21,
+ 54,
+ 25,
+ 19
+ ],
+ [
+ 34,
+ 53,
+ 27,
+ 43,
+ 14,
+ 13
+ ],
+ [
+ 56,
+ 53,
+ 44,
+ 24,
+ 60,
+ 43
+ ],
+ [
+ 12,
+ 53,
+ 40,
+ 49,
+ 2,
+ 62
+ ],
+ [
+ 18,
+ 39,
+ 44,
+ 61,
+ 26,
+ 23
+ ],
+ [
+ 0,
+ 4,
+ 53,
+ 41,
+ 25,
+ 21
+ ],
+ [
+ 1,
+ 7,
+ 25,
+ 10,
+ 40,
+ 56
+ ],
+ [
+ 40,
+ 22,
+ 6,
+ 29,
+ 19,
+ 48
+ ],
+ [
+ 28,
+ 10,
+ 47,
+ 55,
+ 42,
+ 44
+ ],
+ [
+ 30,
+ 27,
+ 57,
+ 16,
+ 50,
+ 59
+ ]
+ ],
+ [
+ [
+ 45,
+ 37,
+ 48,
+ 29,
+ 30,
+ 3
+ ],
+ [
+ 8,
+ 60,
+ 10,
+ 59,
+ 43,
+ 6
+ ],
+ [
+ 51,
+ 45,
+ 28,
+ 59,
+ 63,
+ 34
+ ],
+ [
+ 4,
+ 16,
+ 20,
+ 58,
+ 44,
+ 28
+ ],
+ [
+ 50,
+ 31,
+ 57,
+ 24,
+ 51,
+ 53
+ ],
+ [
+ 58,
+ 9,
+ 0,
+ 61,
+ 35,
+ 41
+ ],
+ [
+ 16,
+ 63,
+ 11,
+ 61,
+ 23,
+ 36
+ ],
+ [
+ 4,
+ 47,
+ 42,
+ 53,
+ 8,
+ 30
+ ],
+ [
+ 44,
+ 14,
+ 16,
+ 33,
+ 3,
+ 20
+ ],
+ [
+ 34,
+ 28,
+ 26,
+ 57,
+ 22,
+ 18
+ ],
+ [
+ 20,
+ 35,
+ 19,
+ 59,
+ 2,
+ 38
+ ],
+ [
+ 12,
+ 60,
+ 43,
+ 63,
+ 32,
+ 62
+ ],
+ [
+ 28,
+ 12,
+ 29,
+ 11,
+ 14,
+ 50
+ ],
+ [
+ 23,
+ 29,
+ 33,
+ 22,
+ 11,
+ 19
+ ],
+ [
+ 23,
+ 60,
+ 51,
+ 50,
+ 7,
+ 22
+ ],
+ [
+ 44,
+ 46,
+ 49,
+ 7,
+ 1,
+ 12
+ ],
+ [
+ 2,
+ 54,
+ 27,
+ 61,
+ 18,
+ 5
+ ],
+ [
+ 17,
+ 50,
+ 51,
+ 32,
+ 33,
+ 34
+ ],
+ [
+ 5,
+ 19,
+ 61,
+ 27,
+ 32,
+ 11
+ ],
+ [
+ 6,
+ 0,
+ 5,
+ 13,
+ 41,
+ 57
+ ],
+ [
+ 27,
+ 33,
+ 53,
+ 45,
+ 38,
+ 32
+ ],
+ [
+ 26,
+ 36,
+ 55,
+ 59,
+ 61,
+ 18
+ ],
+ [
+ 47,
+ 46,
+ 3,
+ 37,
+ 57,
+ 49
+ ],
+ [
+ 20,
+ 22,
+ 4,
+ 16,
+ 51,
+ 11
+ ],
+ [
+ 62,
+ 11,
+ 21,
+ 34,
+ 4,
+ 1
+ ],
+ [
+ 34,
+ 18,
+ 7,
+ 60,
+ 33,
+ 3
+ ],
+ [
+ 45,
+ 52,
+ 36,
+ 4,
+ 21,
+ 9
+ ]
+ ],
+ [
+ [
+ 18,
+ 8,
+ 20,
+ 49,
+ 30,
+ 23
+ ],
+ [
+ 1,
+ 27,
+ 26,
+ 22,
+ 59,
+ 36
+ ],
+ [
+ 43,
+ 26,
+ 15,
+ 58,
+ 0,
+ 46
+ ],
+ [
+ 55,
+ 1,
+ 35,
+ 28,
+ 16,
+ 32
+ ],
+ [
+ 59,
+ 9,
+ 10,
+ 53,
+ 12,
+ 21
+ ],
+ [
+ 9,
+ 2,
+ 27,
+ 11,
+ 61,
+ 43
+ ],
+ [
+ 16,
+ 57,
+ 63,
+ 23,
+ 19,
+ 12
+ ],
+ [
+ 46,
+ 45,
+ 26,
+ 4,
+ 30,
+ 37
+ ],
+ [
+ 43,
+ 44,
+ 20,
+ 16,
+ 14,
+ 9
+ ],
+ [
+ 34,
+ 47,
+ 42,
+ 43,
+ 26,
+ 51
+ ],
+ [
+ 2,
+ 42,
+ 38,
+ 45,
+ 20,
+ 36
+ ],
+ [
+ 18,
+ 7,
+ 12,
+ 2,
+ 43,
+ 60
+ ],
+ [
+ 1,
+ 28,
+ 12,
+ 3,
+ 29,
+ 33
+ ],
+ [
+ 25,
+ 13,
+ 0,
+ 63,
+ 2,
+ 62
+ ],
+ [
+ 18,
+ 36,
+ 6,
+ 29,
+ 19,
+ 15
+ ],
+ [
+ 1,
+ 42,
+ 63,
+ 41,
+ 57,
+ 19
+ ],
+ [
+ 57,
+ 54,
+ 5,
+ 27,
+ 31,
+ 18
+ ],
+ [
+ 50,
+ 6,
+ 13,
+ 32,
+ 17,
+ 20
+ ],
+ [
+ 17,
+ 5,
+ 27,
+ 32,
+ 1,
+ 55
+ ],
+ [
+ 49,
+ 0,
+ 61,
+ 10,
+ 5,
+ 30
+ ],
+ [
+ 29,
+ 53,
+ 51,
+ 13,
+ 33,
+ 46
+ ],
+ [
+ 29,
+ 17,
+ 30,
+ 21,
+ 14,
+ 40
+ ],
+ [
+ 5,
+ 33,
+ 17,
+ 32,
+ 18,
+ 28
+ ],
+ [
+ 4,
+ 51,
+ 20,
+ 54,
+ 58,
+ 41
+ ],
+ [
+ 47,
+ 4,
+ 27,
+ 48,
+ 37,
+ 60
+ ],
+ [
+ 3,
+ 26,
+ 12,
+ 2,
+ 59,
+ 48
+ ],
+ [
+ 46,
+ 43,
+ 18,
+ 20,
+ 9,
+ 53
+ ]
+ ],
+ [
+ [
+ 45,
+ 6,
+ 57,
+ 43,
+ 40,
+ 55
+ ],
+ [
+ 38,
+ 63,
+ 36,
+ 27,
+ 54,
+ 33
+ ],
+ [
+ 37,
+ 14,
+ 19,
+ 41,
+ 58,
+ 63
+ ],
+ [
+ 9,
+ 12,
+ 2,
+ 55,
+ 28,
+ 23
+ ],
+ [
+ 39,
+ 59,
+ 7,
+ 13,
+ 33,
+ 43
+ ],
+ [
+ 45,
+ 9,
+ 63,
+ 27,
+ 32,
+ 58
+ ],
+ [
+ 16,
+ 57,
+ 10,
+ 63,
+ 11,
+ 23
+ ],
+ [
+ 51,
+ 45,
+ 25,
+ 4,
+ 21,
+ 30
+ ],
+ [
+ 21,
+ 44,
+ 14,
+ 16,
+ 39,
+ 33
+ ],
+ [
+ 42,
+ 44,
+ 43,
+ 5,
+ 37,
+ 34
+ ],
+ [
+ 42,
+ 19,
+ 20,
+ 2,
+ 38,
+ 61
+ ],
+ [
+ 4,
+ 12,
+ 2,
+ 62,
+ 63,
+ 36
+ ],
+ [
+ 32,
+ 55,
+ 0,
+ 11,
+ 47,
+ 28
+ ],
+ [
+ 43,
+ 13,
+ 2,
+ 44,
+ 26,
+ 50
+ ],
+ [
+ 49,
+ 33,
+ 15,
+ 28,
+ 29,
+ 35
+ ],
+ [
+ 44,
+ 41,
+ 7,
+ 2,
+ 22,
+ 63
+ ],
+ [
+ 48,
+ 6,
+ 54,
+ 20,
+ 2,
+ 27
+ ],
+ [
+ 50,
+ 51,
+ 32,
+ 3,
+ 17,
+ 36
+ ],
+ [
+ 5,
+ 61,
+ 57,
+ 48,
+ 19,
+ 32
+ ],
+ [
+ 21,
+ 0,
+ 6,
+ 31,
+ 29,
+ 47
+ ],
+ [
+ 33,
+ 9,
+ 53,
+ 27,
+ 17,
+ 36
+ ],
+ [
+ 29,
+ 26,
+ 55,
+ 19,
+ 17,
+ 62
+ ],
+ [
+ 12,
+ 46,
+ 5,
+ 37,
+ 57,
+ 3
+ ],
+ [
+ 20,
+ 51,
+ 4,
+ 22,
+ 16,
+ 41
+ ],
+ [
+ 21,
+ 11,
+ 62,
+ 46,
+ 23,
+ 48
+ ],
+ [
+ 32,
+ 60,
+ 37,
+ 18,
+ 3,
+ 7
+ ],
+ [
+ 9,
+ 11,
+ 36,
+ 48,
+ 0,
+ 45
+ ]
+ ],
+ [
+ [
+ 49,
+ 42,
+ 28,
+ 23,
+ 33,
+ 61
+ ],
+ [
+ 4,
+ 2,
+ 12,
+ 6,
+ 8,
+ 55
+ ],
+ [
+ 12,
+ 0,
+ 26,
+ 41,
+ 6,
+ 27
+ ],
+ [
+ 9,
+ 57,
+ 6,
+ 23,
+ 51,
+ 28
+ ],
+ [
+ 40,
+ 7,
+ 20,
+ 15,
+ 33,
+ 16
+ ],
+ [
+ 45,
+ 59,
+ 63,
+ 62,
+ 32,
+ 3
+ ],
+ [
+ 10,
+ 39,
+ 57,
+ 13,
+ 16,
+ 19
+ ],
+ [
+ 45,
+ 23,
+ 51,
+ 33,
+ 25,
+ 46
+ ],
+ [
+ 28,
+ 21,
+ 44,
+ 11,
+ 16,
+ 59
+ ],
+ [
+ 5,
+ 42,
+ 44,
+ 24,
+ 43,
+ 47
+ ],
+ [
+ 42,
+ 53,
+ 30,
+ 18,
+ 2,
+ 27
+ ],
+ [
+ 2,
+ 62,
+ 4,
+ 43,
+ 10,
+ 36
+ ],
+ [
+ 0,
+ 56,
+ 55,
+ 47,
+ 32,
+ 49
+ ],
+ [
+ 43,
+ 25,
+ 2,
+ 5,
+ 3,
+ 49
+ ],
+ [
+ 4,
+ 28,
+ 15,
+ 8,
+ 49,
+ 58
+ ],
+ [
+ 2,
+ 42,
+ 44,
+ 41,
+ 7,
+ 63
+ ],
+ [
+ 48,
+ 27,
+ 54,
+ 20,
+ 2,
+ 18
+ ],
+ [
+ 50,
+ 51,
+ 0,
+ 36,
+ 3,
+ 32
+ ],
+ [
+ 57,
+ 5,
+ 61,
+ 19,
+ 32,
+ 38
+ ],
+ [
+ 21,
+ 0,
+ 6,
+ 63,
+ 23,
+ 51
+ ],
+ [
+ 33,
+ 53,
+ 27,
+ 36,
+ 9,
+ 38
+ ],
+ [
+ 29,
+ 26,
+ 55,
+ 62,
+ 18,
+ 19
+ ],
+ [
+ 46,
+ 56,
+ 12,
+ 53,
+ 29,
+ 0
+ ],
+ [
+ 20,
+ 16,
+ 22,
+ 4,
+ 51,
+ 61
+ ],
+ [
+ 21,
+ 62,
+ 11,
+ 46,
+ 31,
+ 33
+ ],
+ [
+ 37,
+ 60,
+ 18,
+ 7,
+ 32,
+ 44
+ ],
+ [
+ 11,
+ 9,
+ 36,
+ 0,
+ 48,
+ 63
+ ]
+ ],
+ [
+ [
+ 41,
+ 32,
+ 49,
+ 39,
+ 44,
+ 61
+ ],
+ [
+ 47,
+ 26,
+ 16,
+ 21,
+ 36,
+ 22
+ ],
+ [
+ 4,
+ 30,
+ 37,
+ 42,
+ 60,
+ 54
+ ],
+ [
+ 9,
+ 57,
+ 26,
+ 32,
+ 50,
+ 20
+ ],
+ [
+ 56,
+ 3,
+ 40,
+ 33,
+ 36,
+ 54
+ ],
+ [
+ 11,
+ 38,
+ 2,
+ 32,
+ 61,
+ 30
+ ],
+ [
+ 39,
+ 57,
+ 19,
+ 10,
+ 16,
+ 42
+ ],
+ [
+ 46,
+ 21,
+ 35,
+ 39,
+ 45,
+ 25
+ ],
+ [
+ 21,
+ 37,
+ 12,
+ 20,
+ 11,
+ 28
+ ],
+ [
+ 5,
+ 47,
+ 44,
+ 10,
+ 42,
+ 23
+ ],
+ [
+ 18,
+ 42,
+ 61,
+ 2,
+ 38,
+ 31
+ ],
+ [
+ 54,
+ 4,
+ 2,
+ 7,
+ 22,
+ 16
+ ],
+ [
+ 5,
+ 3,
+ 17,
+ 56,
+ 32,
+ 55
+ ],
+ [
+ 55,
+ 0,
+ 2,
+ 25,
+ 43,
+ 5
+ ],
+ [
+ 22,
+ 28,
+ 15,
+ 6,
+ 5,
+ 49
+ ],
+ [
+ 2,
+ 57,
+ 19,
+ 54,
+ 41,
+ 30
+ ],
+ [
+ 7,
+ 48,
+ 20,
+ 54,
+ 27,
+ 0
+ ],
+ [
+ 3,
+ 56,
+ 13,
+ 37,
+ 59,
+ 43
+ ],
+ [
+ 45,
+ 55,
+ 57,
+ 61,
+ 48,
+ 52
+ ],
+ [
+ 21,
+ 5,
+ 0,
+ 16,
+ 27,
+ 23
+ ],
+ [
+ 25,
+ 42,
+ 17,
+ 54,
+ 23,
+ 14
+ ],
+ [
+ 21,
+ 44,
+ 15,
+ 20,
+ 42,
+ 18
+ ],
+ [
+ 35,
+ 12,
+ 25,
+ 53,
+ 61,
+ 2
+ ],
+ [
+ 38,
+ 54,
+ 48,
+ 53,
+ 21,
+ 36
+ ],
+ [
+ 13,
+ 31,
+ 48,
+ 33,
+ 18,
+ 55
+ ],
+ [
+ 38,
+ 27,
+ 19,
+ 6,
+ 44,
+ 3
+ ],
+ [
+ 29,
+ 62,
+ 43,
+ 59,
+ 46,
+ 5
+ ]
+ ],
+ [
+ [
+ 57,
+ 9,
+ 19,
+ 51,
+ 18,
+ 41
+ ],
+ [
+ 28,
+ 57,
+ 36,
+ 8,
+ 48,
+ 60
+ ],
+ [
+ 2,
+ 51,
+ 59,
+ 5,
+ 34,
+ 9
+ ],
+ [
+ 9,
+ 55,
+ 59,
+ 26,
+ 4,
+ 2
+ ],
+ [
+ 49,
+ 56,
+ 35,
+ 42,
+ 30,
+ 23
+ ],
+ [
+ 18,
+ 30,
+ 22,
+ 29,
+ 19,
+ 52
+ ],
+ [
+ 39,
+ 34,
+ 33,
+ 51,
+ 56,
+ 3
+ ],
+ [
+ 32,
+ 21,
+ 7,
+ 1,
+ 46,
+ 49
+ ],
+ [
+ 33,
+ 54,
+ 23,
+ 21,
+ 12,
+ 17
+ ],
+ [
+ 5,
+ 30,
+ 60,
+ 47,
+ 15,
+ 18
+ ],
+ [
+ 4,
+ 18,
+ 46,
+ 27,
+ 20,
+ 22
+ ],
+ [
+ 22,
+ 59,
+ 54,
+ 48,
+ 19,
+ 4
+ ],
+ [
+ 17,
+ 5,
+ 56,
+ 31,
+ 49,
+ 4
+ ],
+ [
+ 29,
+ 47,
+ 55,
+ 2,
+ 53,
+ 60
+ ],
+ [
+ 8,
+ 22,
+ 11,
+ 44,
+ 36,
+ 15
+ ],
+ [
+ 60,
+ 44,
+ 30,
+ 57,
+ 54,
+ 39
+ ],
+ [
+ 7,
+ 44,
+ 27,
+ 20,
+ 2,
+ 61
+ ],
+ [
+ 48,
+ 17,
+ 37,
+ 32,
+ 21,
+ 57
+ ],
+ [
+ 48,
+ 32,
+ 46,
+ 6,
+ 61,
+ 42
+ ],
+ [
+ 4,
+ 57,
+ 1,
+ 36,
+ 0,
+ 30
+ ],
+ [
+ 7,
+ 17,
+ 61,
+ 53,
+ 21,
+ 63
+ ],
+ [
+ 60,
+ 14,
+ 53,
+ 35,
+ 18,
+ 42
+ ],
+ [
+ 10,
+ 15,
+ 33,
+ 51,
+ 36,
+ 5
+ ],
+ [
+ 11,
+ 4,
+ 19,
+ 51,
+ 21,
+ 52
+ ],
+ [
+ 47,
+ 19,
+ 48,
+ 43,
+ 4,
+ 58
+ ],
+ [
+ 3,
+ 33,
+ 26,
+ 21,
+ 52,
+ 19
+ ],
+ [
+ 24,
+ 45,
+ 60,
+ 35,
+ 49,
+ 1
+ ]
+ ],
+ [
+ [
+ 16,
+ 4,
+ 44,
+ 23,
+ 22,
+ 35
+ ],
+ [
+ 16,
+ 50,
+ 23,
+ 9,
+ 32,
+ 13
+ ],
+ [
+ 11,
+ 35,
+ 21,
+ 7,
+ 48,
+ 59
+ ],
+ [
+ 55,
+ 15,
+ 1,
+ 11,
+ 8,
+ 40
+ ],
+ [
+ 35,
+ 61,
+ 30,
+ 59,
+ 31,
+ 62
+ ],
+ [
+ 51,
+ 29,
+ 15,
+ 52,
+ 38,
+ 61
+ ],
+ [
+ 60,
+ 0,
+ 55,
+ 34,
+ 59,
+ 33
+ ],
+ [
+ 12,
+ 22,
+ 56,
+ 63,
+ 54,
+ 55
+ ],
+ [
+ 42,
+ 54,
+ 23,
+ 33,
+ 27,
+ 47
+ ],
+ [
+ 30,
+ 60,
+ 20,
+ 5,
+ 4,
+ 22
+ ],
+ [
+ 4,
+ 35,
+ 22,
+ 46,
+ 23,
+ 19
+ ],
+ [
+ 29,
+ 22,
+ 59,
+ 49,
+ 24,
+ 28
+ ],
+ [
+ 17,
+ 31,
+ 4,
+ 56,
+ 5,
+ 9
+ ],
+ [
+ 61,
+ 29,
+ 0,
+ 48,
+ 59,
+ 50
+ ],
+ [
+ 8,
+ 6,
+ 22,
+ 60,
+ 55,
+ 31
+ ],
+ [
+ 60,
+ 39,
+ 19,
+ 57,
+ 53,
+ 27
+ ],
+ [
+ 56,
+ 22,
+ 30,
+ 10,
+ 5,
+ 55
+ ],
+ [
+ 20,
+ 31,
+ 26,
+ 1,
+ 61,
+ 45
+ ],
+ [
+ 8,
+ 23,
+ 7,
+ 46,
+ 48,
+ 4
+ ],
+ [
+ 45,
+ 23,
+ 51,
+ 17,
+ 4,
+ 13
+ ],
+ [
+ 17,
+ 13,
+ 61,
+ 14,
+ 29,
+ 55
+ ],
+ [
+ 14,
+ 27,
+ 43,
+ 21,
+ 56,
+ 57
+ ],
+ [
+ 52,
+ 51,
+ 15,
+ 58,
+ 8,
+ 5
+ ],
+ [
+ 4,
+ 51,
+ 49,
+ 14,
+ 21,
+ 34
+ ],
+ [
+ 4,
+ 9,
+ 47,
+ 13,
+ 8,
+ 61
+ ],
+ [
+ 27,
+ 3,
+ 16,
+ 43,
+ 31,
+ 47
+ ],
+ [
+ 59,
+ 43,
+ 61,
+ 29,
+ 0,
+ 18
+ ]
+ ],
+ [
+ [
+ 48,
+ 21,
+ 18,
+ 49,
+ 41,
+ 23
+ ],
+ [
+ 36,
+ 4,
+ 60,
+ 8,
+ 49,
+ 44
+ ],
+ [
+ 20,
+ 39,
+ 30,
+ 59,
+ 45,
+ 55
+ ],
+ [
+ 35,
+ 46,
+ 15,
+ 48,
+ 33,
+ 2
+ ],
+ [
+ 61,
+ 37,
+ 8,
+ 15,
+ 54,
+ 10
+ ],
+ [
+ 46,
+ 51,
+ 6,
+ 29,
+ 58,
+ 4
+ ],
+ [
+ 28,
+ 11,
+ 44,
+ 60,
+ 0,
+ 1
+ ],
+ [
+ 63,
+ 12,
+ 13,
+ 27,
+ 10,
+ 0
+ ],
+ [
+ 33,
+ 42,
+ 54,
+ 44,
+ 23,
+ 14
+ ],
+ [
+ 18,
+ 60,
+ 30,
+ 22,
+ 40,
+ 14
+ ],
+ [
+ 35,
+ 4,
+ 61,
+ 9,
+ 18,
+ 33
+ ],
+ [
+ 59,
+ 45,
+ 48,
+ 28,
+ 62,
+ 22
+ ],
+ [
+ 17,
+ 56,
+ 7,
+ 5,
+ 53,
+ 36
+ ],
+ [
+ 8,
+ 47,
+ 29,
+ 59,
+ 1,
+ 6
+ ],
+ [
+ 9,
+ 8,
+ 18,
+ 22,
+ 60,
+ 15
+ ],
+ [
+ 46,
+ 60,
+ 22,
+ 44,
+ 30,
+ 57
+ ],
+ [
+ 61,
+ 2,
+ 27,
+ 34,
+ 7,
+ 60
+ ],
+ [
+ 48,
+ 21,
+ 37,
+ 17,
+ 50,
+ 57
+ ],
+ [
+ 44,
+ 42,
+ 5,
+ 2,
+ 61,
+ 48
+ ],
+ [
+ 37,
+ 57,
+ 4,
+ 36,
+ 17,
+ 59
+ ],
+ [
+ 7,
+ 17,
+ 44,
+ 61,
+ 53,
+ 33
+ ],
+ [
+ 60,
+ 45,
+ 14,
+ 42,
+ 18,
+ 9
+ ],
+ [
+ 10,
+ 8,
+ 36,
+ 33,
+ 15,
+ 58
+ ],
+ [
+ 19,
+ 20,
+ 11,
+ 4,
+ 49,
+ 51
+ ],
+ [
+ 61,
+ 37,
+ 47,
+ 23,
+ 12,
+ 3
+ ],
+ [
+ 17,
+ 33,
+ 3,
+ 40,
+ 19,
+ 16
+ ],
+ [
+ 61,
+ 45,
+ 49,
+ 1,
+ 14,
+ 63
+ ]
+ ],
+ [
+ [
+ 13,
+ 40,
+ 55,
+ 63,
+ 26,
+ 4
+ ],
+ [
+ 5,
+ 35,
+ 49,
+ 40,
+ 17,
+ 46
+ ],
+ [
+ 38,
+ 17,
+ 59,
+ 49,
+ 2,
+ 58
+ ],
+ [
+ 40,
+ 8,
+ 1,
+ 16,
+ 0,
+ 11
+ ],
+ [
+ 37,
+ 62,
+ 51,
+ 10,
+ 8,
+ 38
+ ],
+ [
+ 9,
+ 42,
+ 61,
+ 29,
+ 35,
+ 33
+ ],
+ [
+ 63,
+ 53,
+ 11,
+ 16,
+ 33,
+ 60
+ ],
+ [
+ 63,
+ 37,
+ 5,
+ 13,
+ 17,
+ 39
+ ],
+ [
+ 44,
+ 20,
+ 31,
+ 54,
+ 21,
+ 38
+ ],
+ [
+ 43,
+ 21,
+ 30,
+ 34,
+ 18,
+ 49
+ ],
+ [
+ 2,
+ 11,
+ 19,
+ 35,
+ 4,
+ 9
+ ],
+ [
+ 22,
+ 60,
+ 43,
+ 2,
+ 4,
+ 49
+ ],
+ [
+ 29,
+ 5,
+ 17,
+ 22,
+ 24,
+ 55
+ ],
+ [
+ 2,
+ 59,
+ 29,
+ 5,
+ 55,
+ 6
+ ],
+ [
+ 23,
+ 8,
+ 36,
+ 22,
+ 15,
+ 30
+ ],
+ [
+ 12,
+ 44,
+ 41,
+ 45,
+ 5,
+ 22
+ ],
+ [
+ 54,
+ 7,
+ 41,
+ 11,
+ 1,
+ 53
+ ],
+ [
+ 6,
+ 50,
+ 2,
+ 9,
+ 21,
+ 37
+ ],
+ [
+ 13,
+ 19,
+ 5,
+ 10,
+ 48,
+ 8
+ ],
+ [
+ 41,
+ 6,
+ 32,
+ 21,
+ 0,
+ 47
+ ],
+ [
+ 38,
+ 33,
+ 36,
+ 53,
+ 31,
+ 61
+ ],
+ [
+ 3,
+ 26,
+ 7,
+ 62,
+ 18,
+ 59
+ ],
+ [
+ 56,
+ 57,
+ 46,
+ 12,
+ 35,
+ 3
+ ],
+ [
+ 20,
+ 16,
+ 22,
+ 24,
+ 27,
+ 42
+ ],
+ [
+ 36,
+ 21,
+ 46,
+ 34,
+ 3,
+ 11
+ ],
+ [
+ 33,
+ 34,
+ 60,
+ 45,
+ 7,
+ 32
+ ],
+ [
+ 56,
+ 34,
+ 52,
+ 58,
+ 26,
+ 48
+ ]
+ ],
+ [
+ [
+ 54,
+ 23,
+ 53,
+ 11,
+ 58,
+ 3
+ ],
+ [
+ 11,
+ 30,
+ 59,
+ 58,
+ 63,
+ 4
+ ],
+ [
+ 20,
+ 29,
+ 58,
+ 17,
+ 42,
+ 4
+ ],
+ [
+ 1,
+ 35,
+ 40,
+ 45,
+ 53,
+ 59
+ ],
+ [
+ 40,
+ 55,
+ 21,
+ 33,
+ 38,
+ 49
+ ],
+ [
+ 45,
+ 29,
+ 61,
+ 27,
+ 63,
+ 62
+ ],
+ [
+ 33,
+ 57,
+ 11,
+ 28,
+ 53,
+ 34
+ ],
+ [
+ 11,
+ 63,
+ 39,
+ 45,
+ 10,
+ 14
+ ],
+ [
+ 30,
+ 54,
+ 57,
+ 59,
+ 33,
+ 26
+ ],
+ [
+ 43,
+ 23,
+ 5,
+ 18,
+ 21,
+ 42
+ ],
+ [
+ 11,
+ 18,
+ 2,
+ 9,
+ 34,
+ 6
+ ],
+ [
+ 22,
+ 60,
+ 28,
+ 2,
+ 63,
+ 17
+ ],
+ [
+ 5,
+ 41,
+ 6,
+ 17,
+ 56,
+ 29
+ ],
+ [
+ 55,
+ 6,
+ 5,
+ 2,
+ 48,
+ 59
+ ],
+ [
+ 23,
+ 19,
+ 62,
+ 22,
+ 11,
+ 9
+ ],
+ [
+ 12,
+ 45,
+ 41,
+ 27,
+ 8,
+ 42
+ ],
+ [
+ 11,
+ 53,
+ 41,
+ 44,
+ 51,
+ 7
+ ],
+ [
+ 21,
+ 2,
+ 36,
+ 6,
+ 50,
+ 56
+ ],
+ [
+ 13,
+ 10,
+ 48,
+ 53,
+ 61,
+ 19
+ ],
+ [
+ 47,
+ 21,
+ 56,
+ 44,
+ 6,
+ 31
+ ],
+ [
+ 44,
+ 12,
+ 3,
+ 55,
+ 41,
+ 53
+ ],
+ [
+ 44,
+ 47,
+ 28,
+ 43,
+ 45,
+ 63
+ ],
+ [
+ 1,
+ 25,
+ 53,
+ 11,
+ 39,
+ 38
+ ],
+ [
+ 1,
+ 59,
+ 38,
+ 3,
+ 37,
+ 42
+ ],
+ [
+ 45,
+ 3,
+ 0,
+ 21,
+ 22,
+ 14
+ ],
+ [
+ 10,
+ 28,
+ 42,
+ 49,
+ 11,
+ 62
+ ],
+ [
+ 30,
+ 57,
+ 15,
+ 56,
+ 16,
+ 41
+ ]
+ ],
+ [
+ [
+ 53,
+ 15,
+ 34,
+ 0,
+ 46,
+ 33
+ ],
+ [
+ 8,
+ 12,
+ 41,
+ 19,
+ 39,
+ 32
+ ],
+ [
+ 56,
+ 31,
+ 36,
+ 13,
+ 23,
+ 9
+ ],
+ [
+ 36,
+ 51,
+ 30,
+ 21,
+ 1,
+ 11
+ ],
+ [
+ 13,
+ 58,
+ 50,
+ 2,
+ 53,
+ 54
+ ],
+ [
+ 49,
+ 52,
+ 32,
+ 7,
+ 23,
+ 47
+ ],
+ [
+ 61,
+ 38,
+ 23,
+ 39,
+ 0,
+ 35
+ ],
+ [
+ 42,
+ 27,
+ 9,
+ 20,
+ 17,
+ 57
+ ],
+ [
+ 34,
+ 1,
+ 29,
+ 4,
+ 35,
+ 45
+ ],
+ [
+ 54,
+ 57,
+ 27,
+ 19,
+ 38,
+ 62
+ ],
+ [
+ 59,
+ 1,
+ 60,
+ 26,
+ 38,
+ 22
+ ],
+ [
+ 25,
+ 31,
+ 51,
+ 36,
+ 32,
+ 8
+ ],
+ [
+ 14,
+ 62,
+ 2,
+ 19,
+ 37,
+ 11
+ ],
+ [
+ 57,
+ 40,
+ 13,
+ 22,
+ 37,
+ 46
+ ],
+ [
+ 45,
+ 34,
+ 58,
+ 44,
+ 42,
+ 16
+ ],
+ [
+ 50,
+ 16,
+ 6,
+ 5,
+ 33,
+ 43
+ ],
+ [
+ 42,
+ 39,
+ 61,
+ 13,
+ 5,
+ 15
+ ],
+ [
+ 46,
+ 23,
+ 27,
+ 28,
+ 4,
+ 63
+ ],
+ [
+ 62,
+ 31,
+ 10,
+ 45,
+ 30,
+ 35
+ ],
+ [
+ 15,
+ 13,
+ 38,
+ 63,
+ 4,
+ 19
+ ],
+ [
+ 34,
+ 15,
+ 38,
+ 57,
+ 27,
+ 19
+ ],
+ [
+ 41,
+ 62,
+ 36,
+ 57,
+ 19,
+ 47
+ ],
+ [
+ 34,
+ 22,
+ 53,
+ 10,
+ 46,
+ 33
+ ],
+ [
+ 24,
+ 51,
+ 4,
+ 47,
+ 39,
+ 10
+ ],
+ [
+ 11,
+ 57,
+ 51,
+ 50,
+ 54,
+ 48
+ ],
+ [
+ 51,
+ 7,
+ 11,
+ 50,
+ 43,
+ 18
+ ],
+ [
+ 39,
+ 37,
+ 9,
+ 42,
+ 40,
+ 44
+ ]
+ ],
+ [
+ [
+ 57,
+ 17,
+ 62,
+ 42,
+ 23,
+ 60
+ ],
+ [
+ 18,
+ 7,
+ 53,
+ 43,
+ 26,
+ 60
+ ],
+ [
+ 60,
+ 5,
+ 3,
+ 53,
+ 23,
+ 57
+ ],
+ [
+ 17,
+ 10,
+ 22,
+ 19,
+ 11,
+ 31
+ ],
+ [
+ 10,
+ 15,
+ 12,
+ 27,
+ 17,
+ 4
+ ],
+ [
+ 2,
+ 44,
+ 39,
+ 36,
+ 25,
+ 54
+ ],
+ [
+ 52,
+ 62,
+ 37,
+ 21,
+ 41,
+ 42
+ ],
+ [
+ 62,
+ 7,
+ 46,
+ 30,
+ 36,
+ 14
+ ],
+ [
+ 50,
+ 4,
+ 10,
+ 58,
+ 0,
+ 16
+ ],
+ [
+ 32,
+ 47,
+ 38,
+ 54,
+ 8,
+ 41
+ ],
+ [
+ 41,
+ 32,
+ 3,
+ 18,
+ 1,
+ 22
+ ],
+ [
+ 16,
+ 50,
+ 53,
+ 7,
+ 44,
+ 12
+ ],
+ [
+ 11,
+ 39,
+ 3,
+ 35,
+ 25,
+ 48
+ ],
+ [
+ 18,
+ 9,
+ 15,
+ 13,
+ 63,
+ 28
+ ],
+ [
+ 62,
+ 58,
+ 13,
+ 5,
+ 3,
+ 17
+ ],
+ [
+ 3,
+ 31,
+ 43,
+ 53,
+ 35,
+ 57
+ ],
+ [
+ 24,
+ 51,
+ 15,
+ 46,
+ 5,
+ 59
+ ],
+ [
+ 13,
+ 30,
+ 0,
+ 32,
+ 5,
+ 59
+ ],
+ [
+ 2,
+ 39,
+ 32,
+ 38,
+ 34,
+ 22
+ ],
+ [
+ 42,
+ 26,
+ 34,
+ 28,
+ 37,
+ 54
+ ],
+ [
+ 28,
+ 43,
+ 53,
+ 41,
+ 13,
+ 0
+ ],
+ [
+ 14,
+ 15,
+ 34,
+ 1,
+ 48,
+ 40
+ ],
+ [
+ 5,
+ 25,
+ 4,
+ 33,
+ 39,
+ 53
+ ],
+ [
+ 58,
+ 4,
+ 17,
+ 57,
+ 51,
+ 12
+ ],
+ [
+ 47,
+ 37,
+ 35,
+ 31,
+ 63,
+ 29
+ ],
+ [
+ 15,
+ 3,
+ 28,
+ 33,
+ 23,
+ 9
+ ],
+ [
+ 23,
+ 6,
+ 58,
+ 47,
+ 30,
+ 56
+ ]
+ ],
+ [
+ [
+ 47,
+ 29,
+ 14,
+ 6,
+ 51,
+ 43
+ ],
+ [
+ 30,
+ 29,
+ 39,
+ 7,
+ 52,
+ 3
+ ],
+ [
+ 63,
+ 34,
+ 41,
+ 2,
+ 7,
+ 47
+ ],
+ [
+ 4,
+ 28,
+ 54,
+ 45,
+ 52,
+ 58
+ ],
+ [
+ 29,
+ 7,
+ 12,
+ 15,
+ 41,
+ 6
+ ],
+ [
+ 34,
+ 29,
+ 48,
+ 3,
+ 43,
+ 40
+ ],
+ [
+ 30,
+ 29,
+ 16,
+ 47,
+ 42,
+ 45
+ ],
+ [
+ 33,
+ 39,
+ 25,
+ 60,
+ 41,
+ 3
+ ],
+ [
+ 50,
+ 26,
+ 4,
+ 25,
+ 13,
+ 17
+ ],
+ [
+ 5,
+ 43,
+ 54,
+ 16,
+ 12,
+ 53
+ ],
+ [
+ 18,
+ 6,
+ 3,
+ 35,
+ 21,
+ 1
+ ],
+ [
+ 56,
+ 46,
+ 48,
+ 10,
+ 16,
+ 44
+ ],
+ [
+ 9,
+ 35,
+ 7,
+ 24,
+ 47,
+ 57
+ ],
+ [
+ 53,
+ 42,
+ 15,
+ 56,
+ 47,
+ 59
+ ],
+ [
+ 39,
+ 11,
+ 36,
+ 32,
+ 35,
+ 18
+ ],
+ [
+ 46,
+ 20,
+ 53,
+ 38,
+ 56,
+ 26
+ ],
+ [
+ 58,
+ 29,
+ 14,
+ 26,
+ 17,
+ 15
+ ],
+ [
+ 24,
+ 25,
+ 39,
+ 16,
+ 1,
+ 57
+ ],
+ [
+ 24,
+ 41,
+ 1,
+ 4,
+ 63,
+ 28
+ ],
+ [
+ 42,
+ 37,
+ 34,
+ 48,
+ 26,
+ 41
+ ],
+ [
+ 11,
+ 28,
+ 16,
+ 7,
+ 32,
+ 56
+ ],
+ [
+ 14,
+ 42,
+ 6,
+ 16,
+ 22,
+ 15
+ ],
+ [
+ 33,
+ 56,
+ 42,
+ 8,
+ 38,
+ 25
+ ],
+ [
+ 4,
+ 58,
+ 48,
+ 33,
+ 11,
+ 28
+ ],
+ [
+ 37,
+ 47,
+ 29,
+ 48,
+ 30,
+ 53
+ ],
+ [
+ 12,
+ 41,
+ 3,
+ 4,
+ 48,
+ 46
+ ],
+ [
+ 14,
+ 13,
+ 61,
+ 6,
+ 62,
+ 1
+ ]
+ ],
+ [
+ [
+ 45,
+ 10,
+ 44,
+ 43,
+ 53,
+ 33
+ ],
+ [
+ 32,
+ 63,
+ 22,
+ 27,
+ 30,
+ 29
+ ],
+ [
+ 54,
+ 35,
+ 37,
+ 32,
+ 26,
+ 30
+ ],
+ [
+ 24,
+ 63,
+ 0,
+ 17,
+ 25,
+ 45
+ ],
+ [
+ 40,
+ 7,
+ 0,
+ 57,
+ 29,
+ 22
+ ],
+ [
+ 10,
+ 34,
+ 20,
+ 22,
+ 43,
+ 33
+ ],
+ [
+ 42,
+ 30,
+ 5,
+ 25,
+ 19,
+ 34
+ ],
+ [
+ 33,
+ 18,
+ 35,
+ 51,
+ 7,
+ 57
+ ],
+ [
+ 50,
+ 28,
+ 25,
+ 4,
+ 10,
+ 9
+ ],
+ [
+ 5,
+ 38,
+ 16,
+ 43,
+ 54,
+ 12
+ ],
+ [
+ 18,
+ 21,
+ 3,
+ 6,
+ 39,
+ 53
+ ],
+ [
+ 56,
+ 16,
+ 53,
+ 39,
+ 46,
+ 42
+ ],
+ [
+ 9,
+ 35,
+ 57,
+ 11,
+ 47,
+ 13
+ ],
+ [
+ 56,
+ 15,
+ 3,
+ 59,
+ 9,
+ 28
+ ],
+ [
+ 4,
+ 62,
+ 39,
+ 58,
+ 63,
+ 36
+ ],
+ [
+ 20,
+ 53,
+ 57,
+ 8,
+ 51,
+ 35
+ ],
+ [
+ 51,
+ 49,
+ 11,
+ 26,
+ 15,
+ 14
+ ],
+ [
+ 0,
+ 25,
+ 62,
+ 59,
+ 21,
+ 13
+ ],
+ [
+ 39,
+ 34,
+ 48,
+ 53,
+ 61,
+ 33
+ ],
+ [
+ 26,
+ 47,
+ 28,
+ 34,
+ 21,
+ 39
+ ],
+ [
+ 28,
+ 43,
+ 12,
+ 53,
+ 41,
+ 32
+ ],
+ [
+ 14,
+ 52,
+ 17,
+ 1,
+ 15,
+ 38
+ ],
+ [
+ 25,
+ 4,
+ 5,
+ 11,
+ 58,
+ 50
+ ],
+ [
+ 58,
+ 4,
+ 17,
+ 10,
+ 25,
+ 57
+ ],
+ [
+ 29,
+ 47,
+ 35,
+ 31,
+ 52,
+ 48
+ ],
+ [
+ 55,
+ 28,
+ 23,
+ 15,
+ 3,
+ 24
+ ],
+ [
+ 6,
+ 30,
+ 57,
+ 32,
+ 34,
+ 62
+ ]
+ ],
+ [
+ [
+ 39,
+ 5,
+ 30,
+ 17,
+ 61,
+ 15
+ ],
+ [
+ 11,
+ 63,
+ 0,
+ 23,
+ 61,
+ 46
+ ],
+ [
+ 61,
+ 15,
+ 53,
+ 22,
+ 7,
+ 57
+ ],
+ [
+ 50,
+ 57,
+ 58,
+ 63,
+ 45,
+ 47
+ ],
+ [
+ 55,
+ 31,
+ 57,
+ 24,
+ 60,
+ 7
+ ],
+ [
+ 22,
+ 7,
+ 43,
+ 1,
+ 10,
+ 0
+ ],
+ [
+ 14,
+ 58,
+ 1,
+ 34,
+ 19,
+ 45
+ ],
+ [
+ 50,
+ 8,
+ 14,
+ 7,
+ 57,
+ 9
+ ],
+ [
+ 43,
+ 0,
+ 4,
+ 10,
+ 45,
+ 46
+ ],
+ [
+ 38,
+ 0,
+ 4,
+ 55,
+ 54,
+ 10
+ ],
+ [
+ 51,
+ 54,
+ 46,
+ 39,
+ 1,
+ 38
+ ],
+ [
+ 16,
+ 37,
+ 33,
+ 36,
+ 21,
+ 63
+ ],
+ [
+ 10,
+ 38,
+ 57,
+ 58,
+ 3,
+ 63
+ ],
+ [
+ 30,
+ 0,
+ 63,
+ 13,
+ 22,
+ 18
+ ],
+ [
+ 55,
+ 58,
+ 62,
+ 38,
+ 6,
+ 36
+ ],
+ [
+ 19,
+ 53,
+ 11,
+ 1,
+ 57,
+ 25
+ ],
+ [
+ 11,
+ 6,
+ 51,
+ 16,
+ 18,
+ 30
+ ],
+ [
+ 20,
+ 57,
+ 32,
+ 36,
+ 13,
+ 56
+ ],
+ [
+ 3,
+ 12,
+ 61,
+ 26,
+ 32,
+ 1
+ ],
+ [
+ 25,
+ 36,
+ 34,
+ 51,
+ 59,
+ 37
+ ],
+ [
+ 13,
+ 53,
+ 28,
+ 16,
+ 14,
+ 48
+ ],
+ [
+ 14,
+ 42,
+ 15,
+ 55,
+ 22,
+ 38
+ ],
+ [
+ 62,
+ 58,
+ 29,
+ 33,
+ 5,
+ 34
+ ],
+ [
+ 21,
+ 4,
+ 51,
+ 49,
+ 12,
+ 58
+ ],
+ [
+ 61,
+ 60,
+ 40,
+ 35,
+ 59,
+ 47
+ ],
+ [
+ 37,
+ 46,
+ 3,
+ 48,
+ 12,
+ 53
+ ],
+ [
+ 63,
+ 8,
+ 46,
+ 33,
+ 1,
+ 53
+ ]
+ ],
+ [
+ [
+ 62,
+ 61,
+ 0,
+ 9,
+ 26,
+ 41
+ ],
+ [
+ 45,
+ 3,
+ 29,
+ 35,
+ 2,
+ 25
+ ],
+ [
+ 56,
+ 31,
+ 53,
+ 28,
+ 23,
+ 2
+ ],
+ [
+ 13,
+ 41,
+ 46,
+ 49,
+ 5,
+ 45
+ ],
+ [
+ 18,
+ 25,
+ 57,
+ 55,
+ 50,
+ 15
+ ],
+ [
+ 58,
+ 22,
+ 4,
+ 46,
+ 19,
+ 12
+ ],
+ [
+ 1,
+ 25,
+ 58,
+ 22,
+ 43,
+ 35
+ ],
+ [
+ 23,
+ 50,
+ 12,
+ 8,
+ 9,
+ 16
+ ],
+ [
+ 29,
+ 17,
+ 16,
+ 43,
+ 10,
+ 4
+ ],
+ [
+ 38,
+ 55,
+ 0,
+ 40,
+ 20,
+ 10
+ ],
+ [
+ 51,
+ 19,
+ 54,
+ 39,
+ 46,
+ 1
+ ],
+ [
+ 60,
+ 16,
+ 37,
+ 11,
+ 56,
+ 36
+ ],
+ [
+ 10,
+ 7,
+ 38,
+ 57,
+ 54,
+ 44
+ ],
+ [
+ 45,
+ 0,
+ 13,
+ 30,
+ 47,
+ 63
+ ],
+ [
+ 58,
+ 48,
+ 55,
+ 38,
+ 36,
+ 29
+ ],
+ [
+ 29,
+ 53,
+ 0,
+ 1,
+ 55,
+ 57
+ ],
+ [
+ 11,
+ 12,
+ 6,
+ 29,
+ 5,
+ 40
+ ],
+ [
+ 57,
+ 32,
+ 42,
+ 8,
+ 20,
+ 36
+ ],
+ [
+ 3,
+ 61,
+ 49,
+ 32,
+ 26,
+ 1
+ ],
+ [
+ 42,
+ 5,
+ 36,
+ 16,
+ 39,
+ 51
+ ],
+ [
+ 9,
+ 13,
+ 7,
+ 53,
+ 46,
+ 28
+ ],
+ [
+ 14,
+ 55,
+ 42,
+ 24,
+ 33,
+ 47
+ ],
+ [
+ 33,
+ 58,
+ 62,
+ 15,
+ 36,
+ 5
+ ],
+ [
+ 51,
+ 4,
+ 49,
+ 58,
+ 21,
+ 41
+ ],
+ [
+ 40,
+ 60,
+ 19,
+ 35,
+ 61,
+ 15
+ ],
+ [
+ 14,
+ 21,
+ 3,
+ 12,
+ 5,
+ 37
+ ],
+ [
+ 35,
+ 60,
+ 1,
+ 10,
+ 53,
+ 54
+ ]
+ ],
+ [
+ [
+ 40,
+ 28,
+ 60,
+ 31,
+ 59,
+ 23
+ ],
+ [
+ 42,
+ 12,
+ 26,
+ 34,
+ 0,
+ 61
+ ],
+ [
+ 16,
+ 5,
+ 62,
+ 3,
+ 32,
+ 0
+ ],
+ [
+ 13,
+ 3,
+ 18,
+ 39,
+ 42,
+ 52
+ ],
+ [
+ 25,
+ 12,
+ 39,
+ 0,
+ 57,
+ 15
+ ],
+ [
+ 33,
+ 50,
+ 58,
+ 36,
+ 6,
+ 26
+ ],
+ [
+ 41,
+ 43,
+ 1,
+ 15,
+ 2,
+ 25
+ ],
+ [
+ 36,
+ 16,
+ 53,
+ 14,
+ 30,
+ 58
+ ],
+ [
+ 16,
+ 29,
+ 3,
+ 2,
+ 10,
+ 46
+ ],
+ [
+ 38,
+ 16,
+ 32,
+ 26,
+ 45,
+ 11
+ ],
+ [
+ 45,
+ 19,
+ 3,
+ 56,
+ 46,
+ 53
+ ],
+ [
+ 21,
+ 36,
+ 11,
+ 9,
+ 16,
+ 41
+ ],
+ [
+ 11,
+ 16,
+ 10,
+ 50,
+ 8,
+ 63
+ ],
+ [
+ 35,
+ 38,
+ 33,
+ 25,
+ 51,
+ 52
+ ],
+ [
+ 58,
+ 48,
+ 27,
+ 50,
+ 29,
+ 26
+ ],
+ [
+ 29,
+ 57,
+ 53,
+ 3,
+ 54,
+ 7
+ ],
+ [
+ 3,
+ 23,
+ 6,
+ 46,
+ 51,
+ 11
+ ],
+ [
+ 32,
+ 5,
+ 54,
+ 57,
+ 52,
+ 30
+ ],
+ [
+ 34,
+ 61,
+ 40,
+ 0,
+ 3,
+ 48
+ ],
+ [
+ 22,
+ 53,
+ 12,
+ 33,
+ 39,
+ 60
+ ],
+ [
+ 20,
+ 53,
+ 47,
+ 40,
+ 12,
+ 8
+ ],
+ [
+ 55,
+ 17,
+ 52,
+ 14,
+ 45,
+ 56
+ ],
+ [
+ 1,
+ 4,
+ 13,
+ 39,
+ 11,
+ 33
+ ],
+ [
+ 17,
+ 10,
+ 51,
+ 44,
+ 55,
+ 4
+ ],
+ [
+ 35,
+ 6,
+ 45,
+ 21,
+ 52,
+ 37
+ ],
+ [
+ 10,
+ 42,
+ 9,
+ 1,
+ 53,
+ 24
+ ],
+ [
+ 27,
+ 41,
+ 32,
+ 10,
+ 45,
+ 47
+ ]
+ ],
+ [
+ [
+ 11,
+ 34,
+ 44,
+ 51,
+ 41,
+ 12
+ ],
+ [
+ 20,
+ 34,
+ 3,
+ 25,
+ 63,
+ 16
+ ],
+ [
+ 40,
+ 26,
+ 37,
+ 22,
+ 15,
+ 54
+ ],
+ [
+ 4,
+ 16,
+ 25,
+ 28,
+ 45,
+ 58
+ ],
+ [
+ 33,
+ 10,
+ 32,
+ 24,
+ 3,
+ 4
+ ],
+ [
+ 0,
+ 9,
+ 58,
+ 3,
+ 34,
+ 15
+ ],
+ [
+ 16,
+ 63,
+ 43,
+ 59,
+ 25,
+ 42
+ ],
+ [
+ 16,
+ 48,
+ 36,
+ 33,
+ 25,
+ 58
+ ],
+ [
+ 20,
+ 50,
+ 16,
+ 23,
+ 42,
+ 29
+ ],
+ [
+ 60,
+ 51,
+ 16,
+ 22,
+ 38,
+ 48
+ ],
+ [
+ 31,
+ 19,
+ 38,
+ 3,
+ 2,
+ 43
+ ],
+ [
+ 56,
+ 9,
+ 21,
+ 40,
+ 18,
+ 44
+ ],
+ [
+ 11,
+ 13,
+ 10,
+ 45,
+ 27,
+ 57
+ ],
+ [
+ 56,
+ 14,
+ 35,
+ 44,
+ 45,
+ 13
+ ],
+ [
+ 58,
+ 43,
+ 26,
+ 33,
+ 31,
+ 50
+ ],
+ [
+ 13,
+ 38,
+ 46,
+ 51,
+ 28,
+ 37
+ ],
+ [
+ 58,
+ 63,
+ 42,
+ 10,
+ 16,
+ 0
+ ],
+ [
+ 0,
+ 1,
+ 53,
+ 52,
+ 24,
+ 59
+ ],
+ [
+ 17,
+ 29,
+ 0,
+ 40,
+ 60,
+ 24
+ ],
+ [
+ 17,
+ 24,
+ 2,
+ 26,
+ 35,
+ 23
+ ],
+ [
+ 44,
+ 20,
+ 22,
+ 4,
+ 50,
+ 40
+ ],
+ [
+ 9,
+ 61,
+ 17,
+ 16,
+ 27,
+ 37
+ ],
+ [
+ 63,
+ 13,
+ 16,
+ 11,
+ 24,
+ 2
+ ],
+ [
+ 44,
+ 48,
+ 23,
+ 49,
+ 56,
+ 45
+ ],
+ [
+ 6,
+ 5,
+ 49,
+ 33,
+ 31,
+ 63
+ ],
+ [
+ 49,
+ 47,
+ 1,
+ 57,
+ 4,
+ 44
+ ],
+ [
+ 2,
+ 56,
+ 19,
+ 63,
+ 39,
+ 50
+ ]
+ ],
+ [
+ [
+ 16,
+ 22,
+ 23,
+ 35,
+ 3,
+ 13
+ ],
+ [
+ 16,
+ 23,
+ 50,
+ 9,
+ 13,
+ 59
+ ],
+ [
+ 11,
+ 35,
+ 21,
+ 7,
+ 59,
+ 24
+ ],
+ [
+ 53,
+ 32,
+ 7,
+ 60,
+ 1,
+ 13
+ ],
+ [
+ 24,
+ 33,
+ 57,
+ 35,
+ 0,
+ 39
+ ],
+ [
+ 15,
+ 38,
+ 0,
+ 45,
+ 32,
+ 58
+ ],
+ [
+ 59,
+ 16,
+ 25,
+ 49,
+ 35,
+ 5
+ ],
+ [
+ 45,
+ 40,
+ 16,
+ 14,
+ 12,
+ 30
+ ],
+ [
+ 20,
+ 16,
+ 42,
+ 6,
+ 17,
+ 14
+ ],
+ [
+ 60,
+ 10,
+ 22,
+ 50,
+ 48,
+ 38
+ ],
+ [
+ 31,
+ 35,
+ 19,
+ 27,
+ 38,
+ 53
+ ],
+ [
+ 33,
+ 59,
+ 48,
+ 16,
+ 60,
+ 39
+ ],
+ [
+ 9,
+ 57,
+ 10,
+ 11,
+ 7,
+ 44
+ ],
+ [
+ 13,
+ 0,
+ 59,
+ 61,
+ 19,
+ 33
+ ],
+ [
+ 18,
+ 58,
+ 39,
+ 21,
+ 50,
+ 36
+ ],
+ [
+ 57,
+ 60,
+ 49,
+ 19,
+ 53,
+ 23
+ ],
+ [
+ 6,
+ 36,
+ 17,
+ 38,
+ 41,
+ 10
+ ],
+ [
+ 63,
+ 1,
+ 57,
+ 20,
+ 36,
+ 31
+ ],
+ [
+ 40,
+ 41,
+ 7,
+ 0,
+ 61,
+ 32
+ ],
+ [
+ 39,
+ 16,
+ 23,
+ 14,
+ 17,
+ 59
+ ],
+ [
+ 51,
+ 30,
+ 50,
+ 7,
+ 3,
+ 61
+ ],
+ [
+ 9,
+ 14,
+ 42,
+ 15,
+ 1,
+ 55
+ ],
+ [
+ 33,
+ 19,
+ 56,
+ 58,
+ 11,
+ 15
+ ],
+ [
+ 9,
+ 26,
+ 4,
+ 41,
+ 51,
+ 10
+ ],
+ [
+ 4,
+ 60,
+ 35,
+ 16,
+ 48,
+ 47
+ ],
+ [
+ 3,
+ 52,
+ 31,
+ 16,
+ 39,
+ 26
+ ],
+ [
+ 4,
+ 1,
+ 3,
+ 61,
+ 0,
+ 54
+ ]
+ ],
+ [
+ [
+ 38,
+ 45,
+ 52,
+ 43,
+ 32,
+ 39
+ ],
+ [
+ 29,
+ 48,
+ 22,
+ 60,
+ 55,
+ 57
+ ],
+ [
+ 5,
+ 49,
+ 8,
+ 20,
+ 14,
+ 55
+ ],
+ [
+ 35,
+ 46,
+ 49,
+ 32,
+ 7,
+ 48
+ ],
+ [
+ 24,
+ 29,
+ 49,
+ 16,
+ 41,
+ 0
+ ],
+ [
+ 56,
+ 3,
+ 35,
+ 15,
+ 4,
+ 40
+ ],
+ [
+ 49,
+ 59,
+ 4,
+ 16,
+ 33,
+ 11
+ ],
+ [
+ 12,
+ 51,
+ 30,
+ 40,
+ 56,
+ 27
+ ],
+ [
+ 23,
+ 14,
+ 46,
+ 55,
+ 42,
+ 17
+ ],
+ [
+ 60,
+ 22,
+ 38,
+ 6,
+ 21,
+ 50
+ ],
+ [
+ 11,
+ 19,
+ 31,
+ 35,
+ 38,
+ 22
+ ],
+ [
+ 59,
+ 48,
+ 19,
+ 33,
+ 16,
+ 9
+ ],
+ [
+ 9,
+ 28,
+ 45,
+ 55,
+ 57,
+ 16
+ ],
+ [
+ 59,
+ 1,
+ 25,
+ 47,
+ 6,
+ 3
+ ],
+ [
+ 9,
+ 39,
+ 50,
+ 18,
+ 58,
+ 21
+ ],
+ [
+ 60,
+ 57,
+ 38,
+ 53,
+ 50,
+ 35
+ ],
+ [
+ 41,
+ 53,
+ 6,
+ 38,
+ 10,
+ 25
+ ],
+ [
+ 1,
+ 63,
+ 15,
+ 24,
+ 36,
+ 29
+ ],
+ [
+ 41,
+ 40,
+ 7,
+ 42,
+ 32,
+ 62
+ ],
+ [
+ 4,
+ 10,
+ 14,
+ 37,
+ 39,
+ 17
+ ],
+ [
+ 61,
+ 30,
+ 7,
+ 50,
+ 57,
+ 3
+ ],
+ [
+ 9,
+ 14,
+ 58,
+ 30,
+ 42,
+ 8
+ ],
+ [
+ 33,
+ 56,
+ 19,
+ 60,
+ 24,
+ 59
+ ],
+ [
+ 9,
+ 26,
+ 4,
+ 51,
+ 25,
+ 41
+ ],
+ [
+ 4,
+ 35,
+ 47,
+ 41,
+ 48,
+ 60
+ ],
+ [
+ 39,
+ 52,
+ 3,
+ 26,
+ 8,
+ 30
+ ],
+ [
+ 4,
+ 1,
+ 10,
+ 61,
+ 12,
+ 0
+ ]
+ ],
+ [
+ [
+ 51,
+ 53,
+ 33,
+ 13,
+ 28,
+ 48
+ ],
+ [
+ 63,
+ 31,
+ 41,
+ 39,
+ 40,
+ 49
+ ],
+ [
+ 42,
+ 14,
+ 3,
+ 24,
+ 50,
+ 44
+ ],
+ [
+ 11,
+ 39,
+ 52,
+ 10,
+ 17,
+ 42
+ ],
+ [
+ 7,
+ 60,
+ 58,
+ 15,
+ 12,
+ 27
+ ],
+ [
+ 34,
+ 36,
+ 44,
+ 28,
+ 29,
+ 40
+ ],
+ [
+ 52,
+ 17,
+ 47,
+ 42,
+ 37,
+ 41
+ ],
+ [
+ 62,
+ 33,
+ 38,
+ 39,
+ 41,
+ 36
+ ],
+ [
+ 50,
+ 58,
+ 37,
+ 10,
+ 16,
+ 62
+ ],
+ [
+ 41,
+ 59,
+ 12,
+ 62,
+ 49,
+ 58
+ ],
+ [
+ 3,
+ 10,
+ 35,
+ 41,
+ 25,
+ 6
+ ],
+ [
+ 39,
+ 53,
+ 58,
+ 21,
+ 19,
+ 54
+ ],
+ [
+ 42,
+ 35,
+ 7,
+ 39,
+ 63,
+ 21
+ ],
+ [
+ 15,
+ 9,
+ 3,
+ 54,
+ 51,
+ 32
+ ],
+ [
+ 3,
+ 35,
+ 32,
+ 36,
+ 18,
+ 13
+ ],
+ [
+ 20,
+ 3,
+ 35,
+ 45,
+ 4,
+ 32
+ ],
+ [
+ 46,
+ 24,
+ 32,
+ 33,
+ 14,
+ 44
+ ],
+ [
+ 16,
+ 57,
+ 30,
+ 19,
+ 63,
+ 61
+ ],
+ [
+ 41,
+ 25,
+ 2,
+ 24,
+ 26,
+ 4
+ ],
+ [
+ 42,
+ 37,
+ 47,
+ 48,
+ 33,
+ 50
+ ],
+ [
+ 13,
+ 54,
+ 53,
+ 56,
+ 63,
+ 0
+ ],
+ [
+ 34,
+ 14,
+ 7,
+ 51,
+ 42,
+ 6
+ ],
+ [
+ 6,
+ 43,
+ 25,
+ 58,
+ 39,
+ 11
+ ],
+ [
+ 0,
+ 26,
+ 44,
+ 4,
+ 51,
+ 60
+ ],
+ [
+ 37,
+ 63,
+ 4,
+ 15,
+ 13,
+ 61
+ ],
+ [
+ 9,
+ 59,
+ 41,
+ 1,
+ 52,
+ 25
+ ],
+ [
+ 56,
+ 58,
+ 47,
+ 1,
+ 62,
+ 51
+ ]
+ ],
+ [
+ [
+ 52,
+ 47,
+ 27,
+ 36,
+ 38,
+ 29
+ ],
+ [
+ 43,
+ 56,
+ 4,
+ 25,
+ 52,
+ 21
+ ],
+ [
+ 25,
+ 54,
+ 35,
+ 18,
+ 11,
+ 63
+ ],
+ [
+ 17,
+ 4,
+ 1,
+ 18,
+ 50,
+ 39
+ ],
+ [
+ 36,
+ 7,
+ 32,
+ 4,
+ 30,
+ 60
+ ],
+ [
+ 34,
+ 3,
+ 61,
+ 48,
+ 24,
+ 40
+ ],
+ [
+ 17,
+ 42,
+ 3,
+ 12,
+ 29,
+ 25
+ ],
+ [
+ 33,
+ 38,
+ 39,
+ 55,
+ 17,
+ 19
+ ],
+ [
+ 17,
+ 50,
+ 41,
+ 13,
+ 16,
+ 51
+ ],
+ [
+ 41,
+ 12,
+ 60,
+ 16,
+ 62,
+ 58
+ ],
+ [
+ 3,
+ 36,
+ 28,
+ 39,
+ 35,
+ 2
+ ],
+ [
+ 48,
+ 27,
+ 53,
+ 54,
+ 19,
+ 43
+ ],
+ [
+ 42,
+ 63,
+ 7,
+ 35,
+ 62,
+ 3
+ ],
+ [
+ 56,
+ 15,
+ 9,
+ 51,
+ 42,
+ 47
+ ],
+ [
+ 35,
+ 39,
+ 18,
+ 36,
+ 43,
+ 38
+ ],
+ [
+ 60,
+ 20,
+ 0,
+ 3,
+ 35,
+ 31
+ ],
+ [
+ 24,
+ 42,
+ 14,
+ 17,
+ 32,
+ 62
+ ],
+ [
+ 24,
+ 57,
+ 63,
+ 0,
+ 42,
+ 1
+ ],
+ [
+ 41,
+ 50,
+ 28,
+ 32,
+ 26,
+ 40
+ ],
+ [
+ 2,
+ 37,
+ 46,
+ 17,
+ 42,
+ 57
+ ],
+ [
+ 49,
+ 7,
+ 11,
+ 54,
+ 13,
+ 53
+ ],
+ [
+ 14,
+ 9,
+ 42,
+ 6,
+ 19,
+ 58
+ ],
+ [
+ 63,
+ 33,
+ 43,
+ 19,
+ 58,
+ 25
+ ],
+ [
+ 9,
+ 26,
+ 48,
+ 4,
+ 41,
+ 44
+ ],
+ [
+ 4,
+ 41,
+ 5,
+ 15,
+ 9,
+ 53
+ ],
+ [
+ 14,
+ 38,
+ 9,
+ 3,
+ 52,
+ 8
+ ],
+ [
+ 23,
+ 1,
+ 61,
+ 4,
+ 47,
+ 14
+ ]
+ ],
+ [
+ [
+ 37,
+ 14,
+ 3,
+ 5,
+ 33,
+ 38
+ ],
+ [
+ 34,
+ 0,
+ 56,
+ 58,
+ 37,
+ 13
+ ],
+ [
+ 61,
+ 14,
+ 22,
+ 29,
+ 15,
+ 46
+ ],
+ [
+ 50,
+ 58,
+ 18,
+ 44,
+ 47,
+ 17
+ ],
+ [
+ 4,
+ 36,
+ 53,
+ 7,
+ 32,
+ 2
+ ],
+ [
+ 22,
+ 1,
+ 34,
+ 61,
+ 10,
+ 3
+ ],
+ [
+ 28,
+ 25,
+ 42,
+ 30,
+ 29,
+ 46
+ ],
+ [
+ 33,
+ 38,
+ 51,
+ 39,
+ 60,
+ 62
+ ],
+ [
+ 17,
+ 51,
+ 16,
+ 26,
+ 46,
+ 50
+ ],
+ [
+ 12,
+ 16,
+ 59,
+ 6,
+ 38,
+ 3
+ ],
+ [
+ 39,
+ 35,
+ 31,
+ 3,
+ 36,
+ 0
+ ],
+ [
+ 48,
+ 19,
+ 53,
+ 56,
+ 27,
+ 39
+ ],
+ [
+ 7,
+ 62,
+ 9,
+ 63,
+ 15,
+ 42
+ ],
+ [
+ 56,
+ 19,
+ 9,
+ 47,
+ 15,
+ 1
+ ],
+ [
+ 39,
+ 36,
+ 18,
+ 35,
+ 42,
+ 38
+ ],
+ [
+ 60,
+ 20,
+ 0,
+ 57,
+ 47,
+ 53
+ ],
+ [
+ 6,
+ 32,
+ 29,
+ 62,
+ 43,
+ 5
+ ],
+ [
+ 24,
+ 63,
+ 57,
+ 1,
+ 42,
+ 2
+ ],
+ [
+ 41,
+ 40,
+ 7,
+ 62,
+ 32,
+ 50
+ ],
+ [
+ 37,
+ 14,
+ 10,
+ 2,
+ 57,
+ 17
+ ],
+ [
+ 7,
+ 54,
+ 30,
+ 53,
+ 50,
+ 3
+ ],
+ [
+ 9,
+ 14,
+ 8,
+ 42,
+ 6,
+ 58
+ ],
+ [
+ 33,
+ 56,
+ 60,
+ 19,
+ 18,
+ 15
+ ],
+ [
+ 9,
+ 26,
+ 4,
+ 41,
+ 59,
+ 23
+ ],
+ [
+ 4,
+ 41,
+ 47,
+ 9,
+ 2,
+ 48
+ ],
+ [
+ 3,
+ 14,
+ 16,
+ 30,
+ 52,
+ 8
+ ],
+ [
+ 1,
+ 4,
+ 14,
+ 61,
+ 0,
+ 12
+ ]
+ ],
+ [
+ [
+ 27,
+ 21,
+ 61,
+ 30,
+ 22,
+ 40
+ ],
+ [
+ 33,
+ 12,
+ 44,
+ 0,
+ 47,
+ 20
+ ],
+ [
+ 14,
+ 39,
+ 58,
+ 37,
+ 16,
+ 61
+ ],
+ [
+ 47,
+ 11,
+ 4,
+ 18,
+ 32,
+ 35
+ ],
+ [
+ 2,
+ 19,
+ 39,
+ 20,
+ 10,
+ 3
+ ],
+ [
+ 39,
+ 61,
+ 49,
+ 15,
+ 30,
+ 47
+ ],
+ [
+ 12,
+ 47,
+ 17,
+ 49,
+ 29,
+ 46
+ ],
+ [
+ 39,
+ 41,
+ 62,
+ 30,
+ 37,
+ 52
+ ],
+ [
+ 21,
+ 15,
+ 28,
+ 48,
+ 26,
+ 27
+ ],
+ [
+ 12,
+ 32,
+ 16,
+ 24,
+ 55,
+ 41
+ ],
+ [
+ 35,
+ 43,
+ 28,
+ 31,
+ 60,
+ 47
+ ],
+ [
+ 3,
+ 17,
+ 19,
+ 49,
+ 10,
+ 42
+ ],
+ [
+ 7,
+ 54,
+ 27,
+ 59,
+ 4,
+ 23
+ ],
+ [
+ 35,
+ 33,
+ 7,
+ 6,
+ 51,
+ 14
+ ],
+ [
+ 51,
+ 28,
+ 36,
+ 42,
+ 41,
+ 32
+ ],
+ [
+ 45,
+ 52,
+ 60,
+ 21,
+ 53,
+ 12
+ ],
+ [
+ 53,
+ 1,
+ 23,
+ 5,
+ 6,
+ 46
+ ],
+ [
+ 35,
+ 9,
+ 63,
+ 16,
+ 26,
+ 22
+ ],
+ [
+ 41,
+ 60,
+ 27,
+ 40,
+ 42,
+ 15
+ ],
+ [
+ 42,
+ 48,
+ 46,
+ 9,
+ 17,
+ 37
+ ],
+ [
+ 31,
+ 11,
+ 23,
+ 58,
+ 39,
+ 44
+ ],
+ [
+ 34,
+ 2,
+ 4,
+ 14,
+ 13,
+ 9
+ ],
+ [
+ 60,
+ 59,
+ 47,
+ 40,
+ 17,
+ 27
+ ],
+ [
+ 60,
+ 63,
+ 7,
+ 42,
+ 49,
+ 40
+ ],
+ [
+ 59,
+ 15,
+ 38,
+ 62,
+ 44,
+ 41
+ ],
+ [
+ 23,
+ 57,
+ 60,
+ 31,
+ 41,
+ 3
+ ],
+ [
+ 38,
+ 59,
+ 31,
+ 51,
+ 36,
+ 7
+ ]
+ ],
+ [
+ [
+ 12,
+ 10,
+ 50,
+ 23,
+ 53,
+ 33
+ ],
+ [
+ 48,
+ 56,
+ 44,
+ 11,
+ 31,
+ 17
+ ],
+ [
+ 33,
+ 11,
+ 17,
+ 54,
+ 15,
+ 62
+ ],
+ [
+ 31,
+ 13,
+ 17,
+ 40,
+ 8,
+ 3
+ ],
+ [
+ 42,
+ 4,
+ 27,
+ 15,
+ 12,
+ 5
+ ],
+ [
+ 34,
+ 50,
+ 33,
+ 29,
+ 55,
+ 6
+ ],
+ [
+ 16,
+ 49,
+ 52,
+ 29,
+ 41,
+ 30
+ ],
+ [
+ 0,
+ 30,
+ 62,
+ 57,
+ 26,
+ 22
+ ],
+ [
+ 26,
+ 62,
+ 5,
+ 58,
+ 51,
+ 49
+ ],
+ [
+ 5,
+ 62,
+ 16,
+ 46,
+ 39,
+ 37
+ ],
+ [
+ 31,
+ 10,
+ 24,
+ 45,
+ 18,
+ 35
+ ],
+ [
+ 50,
+ 19,
+ 48,
+ 54,
+ 16,
+ 35
+ ],
+ [
+ 59,
+ 39,
+ 62,
+ 7,
+ 35,
+ 28
+ ],
+ [
+ 20,
+ 9,
+ 19,
+ 6,
+ 22,
+ 15
+ ],
+ [
+ 35,
+ 18,
+ 50,
+ 32,
+ 36,
+ 39
+ ],
+ [
+ 3,
+ 4,
+ 13,
+ 38,
+ 60,
+ 20
+ ],
+ [
+ 46,
+ 32,
+ 28,
+ 3,
+ 37,
+ 33
+ ],
+ [
+ 30,
+ 63,
+ 16,
+ 19,
+ 24,
+ 42
+ ],
+ [
+ 41,
+ 17,
+ 4,
+ 2,
+ 34,
+ 32
+ ],
+ [
+ 42,
+ 37,
+ 48,
+ 3,
+ 33,
+ 31
+ ],
+ [
+ 56,
+ 7,
+ 25,
+ 11,
+ 39,
+ 44
+ ],
+ [
+ 42,
+ 14,
+ 26,
+ 13,
+ 22,
+ 12
+ ],
+ [
+ 25,
+ 6,
+ 11,
+ 42,
+ 13,
+ 38
+ ],
+ [
+ 0,
+ 9,
+ 26,
+ 41,
+ 4,
+ 10
+ ],
+ [
+ 37,
+ 4,
+ 63,
+ 41,
+ 2,
+ 44
+ ],
+ [
+ 9,
+ 52,
+ 3,
+ 41,
+ 16,
+ 59
+ ],
+ [
+ 58,
+ 47,
+ 56,
+ 1,
+ 26,
+ 62
+ ]
+ ],
+ [
+ [
+ 54,
+ 30,
+ 26,
+ 22,
+ 3,
+ 55
+ ],
+ [
+ 44,
+ 7,
+ 49,
+ 50,
+ 25,
+ 5
+ ],
+ [
+ 54,
+ 4,
+ 48,
+ 58,
+ 26,
+ 32
+ ],
+ [
+ 20,
+ 25,
+ 3,
+ 9,
+ 55,
+ 28
+ ],
+ [
+ 7,
+ 27,
+ 42,
+ 12,
+ 58,
+ 32
+ ],
+ [
+ 50,
+ 34,
+ 6,
+ 42,
+ 29,
+ 55
+ ],
+ [
+ 16,
+ 49,
+ 40,
+ 3,
+ 27,
+ 11
+ ],
+ [
+ 51,
+ 30,
+ 26,
+ 62,
+ 57,
+ 53
+ ],
+ [
+ 3,
+ 5,
+ 49,
+ 28,
+ 26,
+ 50
+ ],
+ [
+ 16,
+ 22,
+ 46,
+ 6,
+ 49,
+ 21
+ ],
+ [
+ 31,
+ 45,
+ 11,
+ 10,
+ 56,
+ 24
+ ],
+ [
+ 21,
+ 51,
+ 50,
+ 19,
+ 9,
+ 61
+ ],
+ [
+ 28,
+ 41,
+ 13,
+ 59,
+ 34,
+ 53
+ ],
+ [
+ 23,
+ 20,
+ 16,
+ 9,
+ 38,
+ 19
+ ],
+ [
+ 50,
+ 34,
+ 58,
+ 20,
+ 27,
+ 35
+ ],
+ [
+ 8,
+ 59,
+ 61,
+ 29,
+ 35,
+ 53
+ ],
+ [
+ 3,
+ 46,
+ 51,
+ 10,
+ 25,
+ 37
+ ],
+ [
+ 62,
+ 21,
+ 25,
+ 9,
+ 18,
+ 56
+ ],
+ [
+ 34,
+ 27,
+ 10,
+ 29,
+ 53,
+ 48
+ ],
+ [
+ 56,
+ 33,
+ 44,
+ 24,
+ 9,
+ 0
+ ],
+ [
+ 44,
+ 12,
+ 19,
+ 30,
+ 8,
+ 49
+ ],
+ [
+ 47,
+ 62,
+ 12,
+ 51,
+ 16,
+ 17
+ ],
+ [
+ 11,
+ 37,
+ 25,
+ 58,
+ 35,
+ 45
+ ],
+ [
+ 4,
+ 9,
+ 10,
+ 37,
+ 39,
+ 16
+ ],
+ [
+ 22,
+ 11,
+ 21,
+ 48,
+ 45,
+ 47
+ ],
+ [
+ 10,
+ 28,
+ 55,
+ 12,
+ 24,
+ 23
+ ],
+ [
+ 30,
+ 16,
+ 27,
+ 32,
+ 57,
+ 15
+ ]
+ ],
+ [
+ [
+ 16,
+ 11,
+ 31,
+ 46,
+ 35,
+ 0
+ ],
+ [
+ 13,
+ 49,
+ 10,
+ 54,
+ 5,
+ 6
+ ],
+ [
+ 36,
+ 13,
+ 27,
+ 46,
+ 3,
+ 18
+ ],
+ [
+ 24,
+ 61,
+ 15,
+ 0,
+ 13,
+ 63
+ ],
+ [
+ 17,
+ 2,
+ 58,
+ 50,
+ 35,
+ 19
+ ],
+ [
+ 8,
+ 7,
+ 49,
+ 52,
+ 47,
+ 23
+ ],
+ [
+ 61,
+ 16,
+ 58,
+ 20,
+ 38,
+ 23
+ ],
+ [
+ 20,
+ 42,
+ 9,
+ 51,
+ 35,
+ 16
+ ],
+ [
+ 25,
+ 47,
+ 4,
+ 1,
+ 50,
+ 63
+ ],
+ [
+ 54,
+ 38,
+ 57,
+ 27,
+ 33,
+ 28
+ ],
+ [
+ 1,
+ 59,
+ 30,
+ 60,
+ 14,
+ 31
+ ],
+ [
+ 31,
+ 51,
+ 25,
+ 14,
+ 41,
+ 55
+ ],
+ [
+ 34,
+ 2,
+ 14,
+ 11,
+ 19,
+ 28
+ ],
+ [
+ 40,
+ 57,
+ 13,
+ 61,
+ 59,
+ 18
+ ],
+ [
+ 44,
+ 58,
+ 50,
+ 45,
+ 37,
+ 26
+ ],
+ [
+ 5,
+ 8,
+ 62,
+ 24,
+ 38,
+ 57
+ ],
+ [
+ 13,
+ 47,
+ 15,
+ 3,
+ 5,
+ 39
+ ],
+ [
+ 25,
+ 23,
+ 9,
+ 18,
+ 63,
+ 33
+ ],
+ [
+ 1,
+ 10,
+ 20,
+ 53,
+ 8,
+ 4
+ ],
+ [
+ 38,
+ 15,
+ 37,
+ 34,
+ 60,
+ 0
+ ],
+ [
+ 41,
+ 27,
+ 30,
+ 57,
+ 19,
+ 40
+ ],
+ [
+ 11,
+ 62,
+ 41,
+ 46,
+ 14,
+ 44
+ ],
+ [
+ 11,
+ 33,
+ 46,
+ 31,
+ 45,
+ 0
+ ],
+ [
+ 4,
+ 51,
+ 47,
+ 16,
+ 9,
+ 12
+ ],
+ [
+ 11,
+ 21,
+ 45,
+ 47,
+ 18,
+ 61
+ ],
+ [
+ 10,
+ 50,
+ 51,
+ 12,
+ 18,
+ 7
+ ],
+ [
+ 17,
+ 31,
+ 36,
+ 5,
+ 19,
+ 1
+ ]
+ ],
+ [
+ [
+ 22,
+ 6,
+ 57,
+ 39,
+ 29,
+ 47
+ ],
+ [
+ 27,
+ 6,
+ 14,
+ 17,
+ 51,
+ 32
+ ],
+ [
+ 1,
+ 29,
+ 11,
+ 26,
+ 47,
+ 51
+ ],
+ [
+ 14,
+ 38,
+ 22,
+ 31,
+ 29,
+ 53
+ ],
+ [
+ 14,
+ 61,
+ 59,
+ 1,
+ 29,
+ 49
+ ],
+ [
+ 30,
+ 8,
+ 21,
+ 47,
+ 52,
+ 0
+ ],
+ [
+ 4,
+ 58,
+ 61,
+ 23,
+ 20,
+ 29
+ ],
+ [
+ 20,
+ 9,
+ 42,
+ 35,
+ 7,
+ 24
+ ],
+ [
+ 47,
+ 25,
+ 4,
+ 1,
+ 29,
+ 0
+ ],
+ [
+ 54,
+ 19,
+ 38,
+ 29,
+ 33,
+ 44
+ ],
+ [
+ 14,
+ 1,
+ 59,
+ 40,
+ 60,
+ 20
+ ],
+ [
+ 51,
+ 0,
+ 14,
+ 62,
+ 16,
+ 52
+ ],
+ [
+ 2,
+ 36,
+ 20,
+ 29,
+ 19,
+ 52
+ ],
+ [
+ 57,
+ 40,
+ 13,
+ 22,
+ 60,
+ 6
+ ],
+ [
+ 37,
+ 44,
+ 58,
+ 5,
+ 8,
+ 50
+ ],
+ [
+ 24,
+ 5,
+ 43,
+ 23,
+ 62,
+ 59
+ ],
+ [
+ 13,
+ 19,
+ 47,
+ 39,
+ 61,
+ 15
+ ],
+ [
+ 58,
+ 33,
+ 9,
+ 7,
+ 4,
+ 28
+ ],
+ [
+ 1,
+ 35,
+ 10,
+ 19,
+ 31,
+ 20
+ ],
+ [
+ 15,
+ 55,
+ 63,
+ 18,
+ 34,
+ 38
+ ],
+ [
+ 27,
+ 15,
+ 38,
+ 30,
+ 57,
+ 42
+ ],
+ [
+ 62,
+ 41,
+ 16,
+ 29,
+ 6,
+ 19
+ ],
+ [
+ 46,
+ 33,
+ 45,
+ 10,
+ 34,
+ 23
+ ],
+ [
+ 4,
+ 51,
+ 47,
+ 1,
+ 16,
+ 41
+ ],
+ [
+ 11,
+ 21,
+ 18,
+ 7,
+ 48,
+ 28
+ ],
+ [
+ 51,
+ 18,
+ 7,
+ 50,
+ 6,
+ 32
+ ],
+ [
+ 9,
+ 11,
+ 36,
+ 55,
+ 43,
+ 48
+ ]
+ ],
+ [
+ [
+ 47,
+ 8,
+ 36,
+ 61,
+ 21,
+ 45
+ ],
+ [
+ 46,
+ 2,
+ 15,
+ 32,
+ 0,
+ 51
+ ],
+ [
+ 24,
+ 15,
+ 33,
+ 61,
+ 2,
+ 43
+ ],
+ [
+ 60,
+ 22,
+ 31,
+ 27,
+ 14,
+ 11
+ ],
+ [
+ 59,
+ 58,
+ 39,
+ 57,
+ 46,
+ 3
+ ],
+ [
+ 57,
+ 43,
+ 2,
+ 31,
+ 7,
+ 62
+ ],
+ [
+ 9,
+ 42,
+ 54,
+ 19,
+ 4,
+ 55
+ ],
+ [
+ 46,
+ 14,
+ 7,
+ 24,
+ 43,
+ 35
+ ],
+ [
+ 47,
+ 4,
+ 0,
+ 37,
+ 12,
+ 13
+ ],
+ [
+ 54,
+ 38,
+ 4,
+ 47,
+ 25,
+ 6
+ ],
+ [
+ 47,
+ 14,
+ 15,
+ 24,
+ 1,
+ 61
+ ],
+ [
+ 16,
+ 54,
+ 5,
+ 0,
+ 7,
+ 63
+ ],
+ [
+ 49,
+ 3,
+ 33,
+ 13,
+ 11,
+ 10
+ ],
+ [
+ 18,
+ 13,
+ 10,
+ 39,
+ 58,
+ 60
+ ],
+ [
+ 58,
+ 62,
+ 10,
+ 33,
+ 5,
+ 26
+ ],
+ [
+ 48,
+ 5,
+ 63,
+ 53,
+ 43,
+ 2
+ ],
+ [
+ 24,
+ 47,
+ 51,
+ 15,
+ 0,
+ 53
+ ],
+ [
+ 44,
+ 0,
+ 6,
+ 3,
+ 7,
+ 34
+ ],
+ [
+ 58,
+ 38,
+ 53,
+ 61,
+ 0,
+ 54
+ ],
+ [
+ 55,
+ 9,
+ 15,
+ 27,
+ 42,
+ 34
+ ],
+ [
+ 43,
+ 1,
+ 51,
+ 41,
+ 4,
+ 5
+ ],
+ [
+ 49,
+ 20,
+ 15,
+ 6,
+ 37,
+ 46
+ ],
+ [
+ 11,
+ 25,
+ 52,
+ 5,
+ 4,
+ 39
+ ],
+ [
+ 38,
+ 40,
+ 44,
+ 51,
+ 10,
+ 14
+ ],
+ [
+ 13,
+ 8,
+ 52,
+ 63,
+ 2,
+ 23
+ ],
+ [
+ 23,
+ 38,
+ 59,
+ 57,
+ 55,
+ 41
+ ],
+ [
+ 23,
+ 6,
+ 62,
+ 0,
+ 7,
+ 28
+ ]
+ ],
+ [
+ [
+ 41,
+ 2,
+ 42,
+ 16,
+ 50,
+ 61
+ ],
+ [
+ 51,
+ 41,
+ 5,
+ 15,
+ 10,
+ 61
+ ],
+ [
+ 43,
+ 1,
+ 29,
+ 55,
+ 21,
+ 60
+ ],
+ [
+ 24,
+ 53,
+ 25,
+ 13,
+ 51,
+ 32
+ ],
+ [
+ 31,
+ 41,
+ 57,
+ 49,
+ 34,
+ 11
+ ],
+ [
+ 17,
+ 4,
+ 35,
+ 30,
+ 10,
+ 38
+ ],
+ [
+ 34,
+ 7,
+ 56,
+ 42,
+ 21,
+ 19
+ ],
+ [
+ 14,
+ 46,
+ 7,
+ 27,
+ 25,
+ 52
+ ],
+ [
+ 0,
+ 4,
+ 6,
+ 12,
+ 60,
+ 47
+ ],
+ [
+ 54,
+ 25,
+ 4,
+ 38,
+ 47,
+ 6
+ ],
+ [
+ 24,
+ 61,
+ 15,
+ 46,
+ 7,
+ 22
+ ],
+ [
+ 5,
+ 16,
+ 57,
+ 0,
+ 22,
+ 55
+ ],
+ [
+ 49,
+ 3,
+ 26,
+ 17,
+ 57,
+ 52
+ ],
+ [
+ 13,
+ 10,
+ 61,
+ 60,
+ 0,
+ 58
+ ],
+ [
+ 58,
+ 6,
+ 49,
+ 10,
+ 5,
+ 2
+ ],
+ [
+ 25,
+ 19,
+ 2,
+ 11,
+ 54,
+ 53
+ ],
+ [
+ 47,
+ 30,
+ 27,
+ 5,
+ 18,
+ 10
+ ],
+ [
+ 44,
+ 55,
+ 63,
+ 13,
+ 22,
+ 31
+ ],
+ [
+ 58,
+ 7,
+ 38,
+ 35,
+ 32,
+ 40
+ ],
+ [
+ 13,
+ 34,
+ 16,
+ 49,
+ 45,
+ 55
+ ],
+ [
+ 51,
+ 35,
+ 30,
+ 58,
+ 55,
+ 4
+ ],
+ [
+ 46,
+ 57,
+ 15,
+ 36,
+ 30,
+ 27
+ ],
+ [
+ 52,
+ 33,
+ 23,
+ 10,
+ 51,
+ 15
+ ],
+ [
+ 40,
+ 41,
+ 4,
+ 51,
+ 10,
+ 31
+ ],
+ [
+ 8,
+ 4,
+ 59,
+ 48,
+ 34,
+ 9
+ ],
+ [
+ 43,
+ 3,
+ 27,
+ 26,
+ 19,
+ 31
+ ],
+ [
+ 46,
+ 18,
+ 8,
+ 4,
+ 40,
+ 50
+ ]
+ ],
+ [
+ [
+ 22,
+ 36,
+ 35,
+ 63,
+ 43,
+ 23
+ ],
+ [
+ 54,
+ 30,
+ 4,
+ 36,
+ 35,
+ 55
+ ],
+ [
+ 28,
+ 19,
+ 23,
+ 49,
+ 50,
+ 59
+ ],
+ [
+ 62,
+ 5,
+ 50,
+ 53,
+ 42,
+ 48
+ ],
+ [
+ 0,
+ 3,
+ 61,
+ 57,
+ 41,
+ 49
+ ],
+ [
+ 60,
+ 4,
+ 29,
+ 16,
+ 53,
+ 30
+ ],
+ [
+ 34,
+ 32,
+ 33,
+ 9,
+ 56,
+ 25
+ ],
+ [
+ 12,
+ 53,
+ 14,
+ 36,
+ 25,
+ 2
+ ],
+ [
+ 23,
+ 48,
+ 35,
+ 29,
+ 4,
+ 16
+ ],
+ [
+ 22,
+ 25,
+ 4,
+ 54,
+ 62,
+ 5
+ ],
+ [
+ 24,
+ 19,
+ 27,
+ 55,
+ 47,
+ 25
+ ],
+ [
+ 18,
+ 5,
+ 22,
+ 34,
+ 63,
+ 30
+ ],
+ [
+ 17,
+ 49,
+ 30,
+ 28,
+ 11,
+ 42
+ ],
+ [
+ 29,
+ 13,
+ 54,
+ 25,
+ 45,
+ 47
+ ],
+ [
+ 27,
+ 2,
+ 10,
+ 5,
+ 54,
+ 33
+ ],
+ [
+ 55,
+ 2,
+ 54,
+ 57,
+ 22,
+ 56
+ ],
+ [
+ 53,
+ 3,
+ 60,
+ 27,
+ 5,
+ 50
+ ],
+ [
+ 17,
+ 8,
+ 47,
+ 61,
+ 50,
+ 44
+ ],
+ [
+ 27,
+ 38,
+ 32,
+ 14,
+ 61,
+ 39
+ ],
+ [
+ 22,
+ 43,
+ 32,
+ 57,
+ 39,
+ 34
+ ],
+ [
+ 16,
+ 44,
+ 37,
+ 23,
+ 61,
+ 27
+ ],
+ [
+ 45,
+ 40,
+ 55,
+ 32,
+ 31,
+ 3
+ ],
+ [
+ 32,
+ 28,
+ 41,
+ 15,
+ 1,
+ 52
+ ],
+ [
+ 21,
+ 22,
+ 31,
+ 10,
+ 4,
+ 40
+ ],
+ [
+ 29,
+ 35,
+ 62,
+ 60,
+ 41,
+ 1
+ ],
+ [
+ 39,
+ 58,
+ 1,
+ 63,
+ 3,
+ 35
+ ],
+ [
+ 10,
+ 62,
+ 31,
+ 45,
+ 27,
+ 7
+ ]
+ ],
+ [
+ [
+ 17,
+ 5,
+ 10,
+ 57,
+ 14,
+ 27
+ ],
+ [
+ 43,
+ 9,
+ 33,
+ 56,
+ 1,
+ 20
+ ],
+ [
+ 63,
+ 1,
+ 35,
+ 43,
+ 27,
+ 10
+ ],
+ [
+ 47,
+ 6,
+ 18,
+ 3,
+ 38,
+ 15
+ ],
+ [
+ 11,
+ 51,
+ 61,
+ 34,
+ 44,
+ 55
+ ],
+ [
+ 10,
+ 63,
+ 53,
+ 60,
+ 37,
+ 58
+ ],
+ [
+ 51,
+ 45,
+ 63,
+ 34,
+ 18,
+ 60
+ ],
+ [
+ 35,
+ 51,
+ 52,
+ 53,
+ 38,
+ 45
+ ],
+ [
+ 44,
+ 0,
+ 35,
+ 55,
+ 38,
+ 9
+ ],
+ [
+ 42,
+ 43,
+ 5,
+ 25,
+ 21,
+ 6
+ ],
+ [
+ 42,
+ 13,
+ 27,
+ 25,
+ 38,
+ 32
+ ],
+ [
+ 43,
+ 5,
+ 1,
+ 38,
+ 22,
+ 2
+ ],
+ [
+ 17,
+ 49,
+ 0,
+ 37,
+ 28,
+ 30
+ ],
+ [
+ 2,
+ 29,
+ 54,
+ 5,
+ 13,
+ 60
+ ],
+ [
+ 2,
+ 27,
+ 10,
+ 58,
+ 40,
+ 28
+ ],
+ [
+ 2,
+ 49,
+ 54,
+ 62,
+ 53,
+ 57
+ ],
+ [
+ 60,
+ 53,
+ 12,
+ 27,
+ 28,
+ 0
+ ],
+ [
+ 17,
+ 50,
+ 51,
+ 33,
+ 3,
+ 43
+ ],
+ [
+ 38,
+ 5,
+ 19,
+ 27,
+ 13,
+ 32
+ ],
+ [
+ 43,
+ 22,
+ 0,
+ 57,
+ 40,
+ 63
+ ],
+ [
+ 27,
+ 16,
+ 35,
+ 52,
+ 38,
+ 39
+ ],
+ [
+ 29,
+ 31,
+ 55,
+ 40,
+ 62,
+ 27
+ ],
+ [
+ 41,
+ 48,
+ 32,
+ 46,
+ 40,
+ 57
+ ],
+ [
+ 20,
+ 54,
+ 4,
+ 22,
+ 0,
+ 55
+ ],
+ [
+ 62,
+ 23,
+ 25,
+ 33,
+ 28,
+ 20
+ ],
+ [
+ 7,
+ 18,
+ 60,
+ 22,
+ 58,
+ 3
+ ],
+ [
+ 36,
+ 9,
+ 11,
+ 0,
+ 48,
+ 31
+ ]
+ ],
+ [
+ [
+ 11,
+ 59,
+ 52,
+ 28,
+ 6,
+ 45
+ ],
+ [
+ 7,
+ 23,
+ 15,
+ 17,
+ 55,
+ 61
+ ],
+ [
+ 28,
+ 0,
+ 29,
+ 46,
+ 58,
+ 14
+ ],
+ [
+ 62,
+ 49,
+ 39,
+ 18,
+ 6,
+ 13
+ ],
+ [
+ 36,
+ 61,
+ 7,
+ 40,
+ 35,
+ 33
+ ],
+ [
+ 8,
+ 16,
+ 29,
+ 43,
+ 57,
+ 37
+ ],
+ [
+ 30,
+ 19,
+ 21,
+ 57,
+ 42,
+ 14
+ ],
+ [
+ 35,
+ 11,
+ 51,
+ 53,
+ 36,
+ 33
+ ],
+ [
+ 48,
+ 35,
+ 4,
+ 50,
+ 11,
+ 40
+ ],
+ [
+ 5,
+ 3,
+ 21,
+ 47,
+ 43,
+ 38
+ ],
+ [
+ 21,
+ 18,
+ 2,
+ 6,
+ 3,
+ 10
+ ],
+ [
+ 18,
+ 43,
+ 5,
+ 45,
+ 22,
+ 4
+ ],
+ [
+ 33,
+ 28,
+ 37,
+ 41,
+ 49,
+ 10
+ ],
+ [
+ 56,
+ 55,
+ 22,
+ 54,
+ 62,
+ 15
+ ],
+ [
+ 2,
+ 27,
+ 57,
+ 5,
+ 63,
+ 19
+ ],
+ [
+ 55,
+ 8,
+ 54,
+ 2,
+ 59,
+ 48
+ ],
+ [
+ 0,
+ 53,
+ 51,
+ 3,
+ 50,
+ 52
+ ],
+ [
+ 21,
+ 12,
+ 62,
+ 60,
+ 18,
+ 43
+ ],
+ [
+ 38,
+ 13,
+ 53,
+ 27,
+ 14,
+ 28
+ ],
+ [
+ 22,
+ 56,
+ 44,
+ 43,
+ 51,
+ 53
+ ],
+ [
+ 29,
+ 43,
+ 12,
+ 16,
+ 41,
+ 52
+ ],
+ [
+ 51,
+ 1,
+ 35,
+ 48,
+ 44,
+ 34
+ ],
+ [
+ 32,
+ 25,
+ 4,
+ 41,
+ 53,
+ 23
+ ],
+ [
+ 48,
+ 17,
+ 25,
+ 1,
+ 60,
+ 62
+ ],
+ [
+ 22,
+ 29,
+ 5,
+ 18,
+ 53,
+ 20
+ ],
+ [
+ 28,
+ 55,
+ 15,
+ 1,
+ 8,
+ 49
+ ],
+ [
+ 30,
+ 57,
+ 6,
+ 7,
+ 31,
+ 50
+ ]
+ ],
+ [
+ [
+ 11,
+ 16,
+ 31,
+ 46,
+ 35,
+ 49
+ ],
+ [
+ 13,
+ 49,
+ 31,
+ 16,
+ 34,
+ 10
+ ],
+ [
+ 36,
+ 13,
+ 27,
+ 34,
+ 42,
+ 18
+ ],
+ [
+ 24,
+ 7,
+ 61,
+ 63,
+ 34,
+ 15
+ ],
+ [
+ 17,
+ 2,
+ 35,
+ 48,
+ 44,
+ 62
+ ],
+ [
+ 8,
+ 7,
+ 23,
+ 47,
+ 51,
+ 26
+ ],
+ [
+ 61,
+ 58,
+ 29,
+ 38,
+ 33,
+ 45
+ ],
+ [
+ 20,
+ 9,
+ 42,
+ 27,
+ 54,
+ 29
+ ],
+ [
+ 25,
+ 47,
+ 34,
+ 1,
+ 4,
+ 38
+ ],
+ [
+ 54,
+ 27,
+ 57,
+ 4,
+ 28,
+ 62
+ ],
+ [
+ 1,
+ 30,
+ 59,
+ 60,
+ 17,
+ 32
+ ],
+ [
+ 31,
+ 25,
+ 51,
+ 5,
+ 58,
+ 49
+ ],
+ [
+ 34,
+ 2,
+ 19,
+ 14,
+ 29,
+ 36
+ ],
+ [
+ 57,
+ 40,
+ 6,
+ 9,
+ 32,
+ 14
+ ],
+ [
+ 8,
+ 44,
+ 16,
+ 34,
+ 45,
+ 47
+ ],
+ [
+ 8,
+ 25,
+ 40,
+ 37,
+ 14,
+ 59
+ ],
+ [
+ 13,
+ 15,
+ 19,
+ 51,
+ 25,
+ 37
+ ],
+ [
+ 23,
+ 4,
+ 44,
+ 52,
+ 15,
+ 19
+ ],
+ [
+ 8,
+ 10,
+ 45,
+ 1,
+ 31,
+ 53
+ ],
+ [
+ 38,
+ 59,
+ 15,
+ 2,
+ 0,
+ 34
+ ],
+ [
+ 41,
+ 25,
+ 57,
+ 55,
+ 27,
+ 24
+ ],
+ [
+ 11,
+ 44,
+ 41,
+ 39,
+ 53,
+ 62
+ ],
+ [
+ 31,
+ 30,
+ 42,
+ 37,
+ 34,
+ 59
+ ],
+ [
+ 33,
+ 14,
+ 12,
+ 10,
+ 51,
+ 54
+ ],
+ [
+ 11,
+ 18,
+ 45,
+ 33,
+ 20,
+ 48
+ ],
+ [
+ 50,
+ 51,
+ 54,
+ 23,
+ 10,
+ 2
+ ],
+ [
+ 17,
+ 31,
+ 37,
+ 5,
+ 19,
+ 36
+ ]
+ ],
+ [
+ [
+ 31,
+ 35,
+ 51,
+ 18,
+ 53,
+ 61
+ ],
+ [
+ 7,
+ 40,
+ 39,
+ 41,
+ 31,
+ 37
+ ],
+ [
+ 16,
+ 29,
+ 26,
+ 50,
+ 33,
+ 10
+ ],
+ [
+ 14,
+ 22,
+ 37,
+ 17,
+ 6,
+ 25
+ ],
+ [
+ 60,
+ 20,
+ 46,
+ 4,
+ 3,
+ 57
+ ],
+ [
+ 11,
+ 2,
+ 49,
+ 21,
+ 27,
+ 9
+ ],
+ [
+ 42,
+ 30,
+ 54,
+ 58,
+ 19,
+ 56
+ ],
+ [
+ 46,
+ 33,
+ 38,
+ 41,
+ 35,
+ 49
+ ],
+ [
+ 50,
+ 48,
+ 57,
+ 37,
+ 38,
+ 10
+ ],
+ [
+ 59,
+ 63,
+ 47,
+ 28,
+ 10,
+ 35
+ ],
+ [
+ 28,
+ 3,
+ 2,
+ 52,
+ 33,
+ 43
+ ],
+ [
+ 58,
+ 19,
+ 17,
+ 16,
+ 57,
+ 63
+ ],
+ [
+ 42,
+ 4,
+ 3,
+ 23,
+ 45,
+ 57
+ ],
+ [
+ 18,
+ 42,
+ 51,
+ 3,
+ 20,
+ 15
+ ],
+ [
+ 32,
+ 43,
+ 12,
+ 3,
+ 0,
+ 13
+ ],
+ [
+ 15,
+ 45,
+ 4,
+ 7,
+ 53,
+ 21
+ ],
+ [
+ 28,
+ 24,
+ 14,
+ 31,
+ 46,
+ 60
+ ],
+ [
+ 26,
+ 30,
+ 0,
+ 5,
+ 54,
+ 36
+ ],
+ [
+ 63,
+ 37,
+ 41,
+ 51,
+ 11,
+ 4
+ ],
+ [
+ 48,
+ 9,
+ 33,
+ 42,
+ 54,
+ 0
+ ],
+ [
+ 22,
+ 11,
+ 60,
+ 39,
+ 1,
+ 49
+ ],
+ [
+ 17,
+ 4,
+ 34,
+ 2,
+ 53,
+ 27
+ ],
+ [
+ 17,
+ 6,
+ 40,
+ 58,
+ 42,
+ 39
+ ],
+ [
+ 60,
+ 54,
+ 35,
+ 39,
+ 0,
+ 32
+ ],
+ [
+ 44,
+ 15,
+ 37,
+ 13,
+ 8,
+ 25
+ ],
+ [
+ 41,
+ 59,
+ 25,
+ 45,
+ 13,
+ 6
+ ],
+ [
+ 23,
+ 58,
+ 13,
+ 19,
+ 29,
+ 62
+ ]
+ ],
+ [
+ [
+ 52,
+ 47,
+ 27,
+ 36,
+ 33,
+ 38
+ ],
+ [
+ 43,
+ 56,
+ 4,
+ 21,
+ 25,
+ 6
+ ],
+ [
+ 25,
+ 54,
+ 35,
+ 18,
+ 11,
+ 57
+ ],
+ [
+ 17,
+ 16,
+ 1,
+ 6,
+ 33,
+ 45
+ ],
+ [
+ 32,
+ 36,
+ 0,
+ 60,
+ 46,
+ 57
+ ],
+ [
+ 11,
+ 1,
+ 3,
+ 21,
+ 2,
+ 34
+ ],
+ [
+ 30,
+ 42,
+ 58,
+ 54,
+ 19,
+ 13
+ ],
+ [
+ 33,
+ 35,
+ 56,
+ 42,
+ 38,
+ 45
+ ],
+ [
+ 50,
+ 37,
+ 36,
+ 14,
+ 20,
+ 48
+ ],
+ [
+ 10,
+ 28,
+ 4,
+ 37,
+ 32,
+ 6
+ ],
+ [
+ 3,
+ 18,
+ 28,
+ 61,
+ 2,
+ 44
+ ],
+ [
+ 16,
+ 56,
+ 19,
+ 27,
+ 43,
+ 46
+ ],
+ [
+ 42,
+ 33,
+ 57,
+ 3,
+ 58,
+ 26
+ ],
+ [
+ 56,
+ 15,
+ 51,
+ 50,
+ 55,
+ 13
+ ],
+ [
+ 12,
+ 43,
+ 40,
+ 13,
+ 16,
+ 29
+ ],
+ [
+ 15,
+ 7,
+ 28,
+ 53,
+ 5,
+ 20
+ ],
+ [
+ 14,
+ 58,
+ 24,
+ 60,
+ 31,
+ 51
+ ],
+ [
+ 0,
+ 60,
+ 3,
+ 24,
+ 19,
+ 44
+ ],
+ [
+ 36,
+ 28,
+ 11,
+ 53,
+ 63,
+ 15
+ ],
+ [
+ 2,
+ 26,
+ 9,
+ 34,
+ 0,
+ 3
+ ],
+ [
+ 49,
+ 28,
+ 43,
+ 41,
+ 30,
+ 11
+ ],
+ [
+ 25,
+ 51,
+ 12,
+ 6,
+ 61,
+ 16
+ ],
+ [
+ 39,
+ 17,
+ 41,
+ 50,
+ 40,
+ 21
+ ],
+ [
+ 25,
+ 58,
+ 48,
+ 12,
+ 60,
+ 33
+ ],
+ [
+ 31,
+ 49,
+ 5,
+ 52,
+ 63,
+ 3
+ ],
+ [
+ 15,
+ 55,
+ 38,
+ 47,
+ 1,
+ 49
+ ],
+ [
+ 23,
+ 6,
+ 32,
+ 19,
+ 62,
+ 7
+ ]
+ ],
+ [
+ [
+ 44,
+ 24,
+ 56,
+ 33,
+ 15,
+ 7
+ ],
+ [
+ 38,
+ 26,
+ 24,
+ 29,
+ 53,
+ 19
+ ],
+ [
+ 12,
+ 15,
+ 29,
+ 9,
+ 1,
+ 63
+ ],
+ [
+ 38,
+ 61,
+ 58,
+ 50,
+ 45,
+ 6
+ ],
+ [
+ 24,
+ 34,
+ 4,
+ 36,
+ 57,
+ 31
+ ],
+ [
+ 1,
+ 22,
+ 43,
+ 21,
+ 10,
+ 7
+ ],
+ [
+ 20,
+ 19,
+ 54,
+ 58,
+ 18,
+ 42
+ ],
+ [
+ 56,
+ 33,
+ 14,
+ 21,
+ 51,
+ 18
+ ],
+ [
+ 60,
+ 50,
+ 14,
+ 36,
+ 4,
+ 43
+ ],
+ [
+ 6,
+ 10,
+ 9,
+ 63,
+ 4,
+ 38
+ ],
+ [
+ 54,
+ 39,
+ 46,
+ 18,
+ 3,
+ 2
+ ],
+ [
+ 30,
+ 16,
+ 37,
+ 53,
+ 56,
+ 43
+ ],
+ [
+ 56,
+ 10,
+ 42,
+ 58,
+ 57,
+ 15
+ ],
+ [
+ 30,
+ 56,
+ 13,
+ 50,
+ 51,
+ 0
+ ],
+ [
+ 55,
+ 40,
+ 12,
+ 62,
+ 13,
+ 30
+ ],
+ [
+ 53,
+ 28,
+ 11,
+ 61,
+ 7,
+ 19
+ ],
+ [
+ 55,
+ 14,
+ 17,
+ 47,
+ 30,
+ 5
+ ],
+ [
+ 18,
+ 31,
+ 20,
+ 60,
+ 57,
+ 32
+ ],
+ [
+ 43,
+ 12,
+ 53,
+ 26,
+ 32,
+ 61
+ ],
+ [
+ 25,
+ 16,
+ 26,
+ 61,
+ 53,
+ 3
+ ],
+ [
+ 49,
+ 28,
+ 51,
+ 59,
+ 55,
+ 11
+ ],
+ [
+ 56,
+ 6,
+ 32,
+ 14,
+ 10,
+ 21
+ ],
+ [
+ 2,
+ 15,
+ 58,
+ 17,
+ 13,
+ 62
+ ],
+ [
+ 53,
+ 51,
+ 4,
+ 29,
+ 50,
+ 25
+ ],
+ [
+ 61,
+ 31,
+ 19,
+ 15,
+ 60,
+ 49
+ ],
+ [
+ 19,
+ 46,
+ 44,
+ 36,
+ 8,
+ 40
+ ],
+ [
+ 8,
+ 29,
+ 46,
+ 7,
+ 53,
+ 20
+ ]
+ ],
+ [
+ [
+ 48,
+ 63,
+ 38,
+ 42,
+ 50,
+ 62
+ ],
+ [
+ 3,
+ 2,
+ 33,
+ 40,
+ 14,
+ 60
+ ],
+ [
+ 39,
+ 7,
+ 45,
+ 40,
+ 6,
+ 44
+ ],
+ [
+ 41,
+ 5,
+ 20,
+ 56,
+ 13,
+ 0
+ ],
+ [
+ 6,
+ 37,
+ 30,
+ 1,
+ 38,
+ 54
+ ],
+ [
+ 59,
+ 46,
+ 4,
+ 22,
+ 5,
+ 6
+ ],
+ [
+ 20,
+ 1,
+ 44,
+ 35,
+ 13,
+ 3
+ ],
+ [
+ 12,
+ 56,
+ 8,
+ 50,
+ 31,
+ 2
+ ],
+ [
+ 33,
+ 60,
+ 41,
+ 43,
+ 37,
+ 36
+ ],
+ [
+ 9,
+ 10,
+ 0,
+ 55,
+ 40,
+ 37
+ ],
+ [
+ 54,
+ 39,
+ 9,
+ 8,
+ 61,
+ 46
+ ],
+ [
+ 30,
+ 56,
+ 53,
+ 24,
+ 16,
+ 11
+ ],
+ [
+ 7,
+ 58,
+ 57,
+ 42,
+ 10,
+ 52
+ ],
+ [
+ 27,
+ 30,
+ 42,
+ 25,
+ 59,
+ 13
+ ],
+ [
+ 9,
+ 11,
+ 55,
+ 49,
+ 0,
+ 46
+ ],
+ [
+ 34,
+ 53,
+ 54,
+ 29,
+ 57,
+ 37
+ ],
+ [
+ 55,
+ 17,
+ 5,
+ 60,
+ 18,
+ 31
+ ],
+ [
+ 57,
+ 48,
+ 43,
+ 39,
+ 32,
+ 36
+ ],
+ [
+ 12,
+ 43,
+ 42,
+ 49,
+ 7,
+ 61
+ ],
+ [
+ 23,
+ 36,
+ 1,
+ 0,
+ 16,
+ 61
+ ],
+ [
+ 21,
+ 35,
+ 11,
+ 31,
+ 55,
+ 7
+ ],
+ [
+ 60,
+ 8,
+ 42,
+ 24,
+ 14,
+ 6
+ ],
+ [
+ 15,
+ 51,
+ 58,
+ 2,
+ 33,
+ 30
+ ],
+ [
+ 52,
+ 51,
+ 4,
+ 28,
+ 21,
+ 19
+ ],
+ [
+ 19,
+ 60,
+ 15,
+ 34,
+ 54,
+ 46
+ ],
+ [
+ 19,
+ 61,
+ 58,
+ 12,
+ 40,
+ 3
+ ],
+ [
+ 35,
+ 49,
+ 54,
+ 53,
+ 1,
+ 25
+ ]
+ ],
+ [
+ [
+ 47,
+ 37,
+ 59,
+ 38,
+ 33,
+ 10
+ ],
+ [
+ 36,
+ 7,
+ 21,
+ 51,
+ 8,
+ 47
+ ],
+ [
+ 20,
+ 32,
+ 44,
+ 47,
+ 4,
+ 54
+ ],
+ [
+ 43,
+ 20,
+ 42,
+ 52,
+ 8,
+ 19
+ ],
+ [
+ 27,
+ 9,
+ 39,
+ 57,
+ 12,
+ 54
+ ],
+ [
+ 28,
+ 16,
+ 29,
+ 11,
+ 61,
+ 58
+ ],
+ [
+ 42,
+ 2,
+ 30,
+ 43,
+ 28,
+ 25
+ ],
+ [
+ 36,
+ 18,
+ 32,
+ 21,
+ 53,
+ 15
+ ],
+ [
+ 50,
+ 12,
+ 37,
+ 48,
+ 14,
+ 52
+ ],
+ [
+ 9,
+ 10,
+ 13,
+ 3,
+ 58,
+ 26
+ ],
+ [
+ 18,
+ 3,
+ 9,
+ 55,
+ 6,
+ 61
+ ],
+ [
+ 30,
+ 18,
+ 45,
+ 60,
+ 16,
+ 24
+ ],
+ [
+ 35,
+ 5,
+ 42,
+ 26,
+ 37,
+ 58
+ ],
+ [
+ 55,
+ 38,
+ 15,
+ 13,
+ 14,
+ 54
+ ],
+ [
+ 9,
+ 62,
+ 2,
+ 17,
+ 13,
+ 26
+ ],
+ [
+ 9,
+ 40,
+ 37,
+ 54,
+ 17,
+ 47
+ ],
+ [
+ 22,
+ 60,
+ 51,
+ 40,
+ 14,
+ 39
+ ],
+ [
+ 17,
+ 3,
+ 21,
+ 30,
+ 36,
+ 25
+ ],
+ [
+ 56,
+ 41,
+ 24,
+ 9,
+ 43,
+ 14
+ ],
+ [
+ 53,
+ 34,
+ 26,
+ 22,
+ 12,
+ 35
+ ],
+ [
+ 36,
+ 42,
+ 24,
+ 37,
+ 4,
+ 8
+ ],
+ [
+ 34,
+ 16,
+ 8,
+ 46,
+ 56,
+ 17
+ ],
+ [
+ 31,
+ 10,
+ 16,
+ 41,
+ 56,
+ 32
+ ],
+ [
+ 48,
+ 0,
+ 42,
+ 56,
+ 30,
+ 31
+ ],
+ [
+ 6,
+ 54,
+ 26,
+ 19,
+ 7,
+ 8
+ ],
+ [
+ 49,
+ 22,
+ 6,
+ 13,
+ 59,
+ 24
+ ],
+ [
+ 45,
+ 62,
+ 27,
+ 47,
+ 50,
+ 7
+ ]
+ ],
+ [
+ [
+ 45,
+ 37,
+ 48,
+ 29,
+ 30,
+ 3
+ ],
+ [
+ 8,
+ 60,
+ 59,
+ 43,
+ 10,
+ 48
+ ],
+ [
+ 51,
+ 45,
+ 28,
+ 34,
+ 59,
+ 63
+ ],
+ [
+ 43,
+ 2,
+ 38,
+ 12,
+ 20,
+ 4
+ ],
+ [
+ 50,
+ 57,
+ 39,
+ 31,
+ 63,
+ 9
+ ],
+ [
+ 58,
+ 53,
+ 18,
+ 9,
+ 30,
+ 21
+ ],
+ [
+ 63,
+ 51,
+ 34,
+ 7,
+ 20,
+ 27
+ ],
+ [
+ 32,
+ 21,
+ 46,
+ 47,
+ 25,
+ 18
+ ],
+ [
+ 6,
+ 12,
+ 14,
+ 50,
+ 33,
+ 37
+ ],
+ [
+ 30,
+ 10,
+ 9,
+ 6,
+ 13,
+ 29
+ ],
+ [
+ 61,
+ 18,
+ 23,
+ 20,
+ 44,
+ 6
+ ],
+ [
+ 16,
+ 12,
+ 30,
+ 60,
+ 0,
+ 62
+ ],
+ [
+ 5,
+ 26,
+ 57,
+ 35,
+ 37,
+ 61
+ ],
+ [
+ 46,
+ 55,
+ 13,
+ 1,
+ 17,
+ 31
+ ],
+ [
+ 6,
+ 10,
+ 22,
+ 2,
+ 16,
+ 15
+ ],
+ [
+ 6,
+ 54,
+ 19,
+ 25,
+ 57,
+ 46
+ ],
+ [
+ 2,
+ 30,
+ 60,
+ 61,
+ 18,
+ 49
+ ],
+ [
+ 17,
+ 4,
+ 2,
+ 27,
+ 3,
+ 44
+ ],
+ [
+ 56,
+ 46,
+ 8,
+ 32,
+ 6,
+ 14
+ ],
+ [
+ 5,
+ 13,
+ 0,
+ 34,
+ 14,
+ 30
+ ],
+ [
+ 15,
+ 23,
+ 59,
+ 57,
+ 27,
+ 53
+ ],
+ [
+ 36,
+ 61,
+ 8,
+ 43,
+ 57,
+ 37
+ ],
+ [
+ 10,
+ 15,
+ 52,
+ 31,
+ 29,
+ 23
+ ],
+ [
+ 10,
+ 38,
+ 1,
+ 4,
+ 31,
+ 57
+ ],
+ [
+ 8,
+ 32,
+ 54,
+ 33,
+ 3,
+ 50
+ ],
+ [
+ 6,
+ 33,
+ 19,
+ 50,
+ 2,
+ 3
+ ],
+ [
+ 55,
+ 43,
+ 5,
+ 4,
+ 25,
+ 8
+ ]
+ ],
+ [
+ [
+ 25,
+ 14,
+ 18,
+ 49,
+ 51,
+ 63
+ ],
+ [
+ 42,
+ 21,
+ 30,
+ 43,
+ 24,
+ 7
+ ],
+ [
+ 54,
+ 39,
+ 9,
+ 59,
+ 28,
+ 49
+ ],
+ [
+ 23,
+ 1,
+ 55,
+ 45,
+ 43,
+ 40
+ ],
+ [
+ 37,
+ 30,
+ 41,
+ 59,
+ 21,
+ 44
+ ],
+ [
+ 4,
+ 41,
+ 31,
+ 35,
+ 19,
+ 14
+ ],
+ [
+ 28,
+ 55,
+ 44,
+ 63,
+ 9,
+ 51
+ ],
+ [
+ 63,
+ 12,
+ 32,
+ 13,
+ 47,
+ 28
+ ],
+ [
+ 33,
+ 12,
+ 59,
+ 35,
+ 6,
+ 39
+ ],
+ [
+ 30,
+ 40,
+ 10,
+ 29,
+ 52,
+ 13
+ ],
+ [
+ 63,
+ 19,
+ 23,
+ 61,
+ 8,
+ 55
+ ],
+ [
+ 45,
+ 62,
+ 22,
+ 12,
+ 38,
+ 42
+ ],
+ [
+ 58,
+ 26,
+ 57,
+ 20,
+ 45,
+ 30
+ ],
+ [
+ 8,
+ 59,
+ 47,
+ 17,
+ 25,
+ 22
+ ],
+ [
+ 24,
+ 54,
+ 51,
+ 55,
+ 10,
+ 15
+ ],
+ [
+ 46,
+ 22,
+ 54,
+ 44,
+ 57,
+ 40
+ ],
+ [
+ 34,
+ 60,
+ 2,
+ 17,
+ 27,
+ 18
+ ],
+ [
+ 51,
+ 6,
+ 61,
+ 2,
+ 39,
+ 32
+ ],
+ [
+ 5,
+ 49,
+ 38,
+ 8,
+ 32,
+ 46
+ ],
+ [
+ 57,
+ 4,
+ 23,
+ 14,
+ 59,
+ 5
+ ],
+ [
+ 7,
+ 36,
+ 45,
+ 21,
+ 53,
+ 4
+ ],
+ [
+ 60,
+ 3,
+ 39,
+ 8,
+ 14,
+ 35
+ ],
+ [
+ 33,
+ 57,
+ 60,
+ 47,
+ 15,
+ 52
+ ],
+ [
+ 20,
+ 11,
+ 22,
+ 58,
+ 19,
+ 4
+ ],
+ [
+ 36,
+ 34,
+ 47,
+ 41,
+ 60,
+ 40
+ ],
+ [
+ 33,
+ 3,
+ 49,
+ 29,
+ 14,
+ 59
+ ],
+ [
+ 52,
+ 60,
+ 24,
+ 25,
+ 35,
+ 34
+ ]
+ ],
+ [
+ [
+ 23,
+ 54,
+ 53,
+ 11,
+ 58,
+ 8
+ ],
+ [
+ 11,
+ 30,
+ 15,
+ 63,
+ 59,
+ 16
+ ],
+ [
+ 20,
+ 29,
+ 58,
+ 17,
+ 52,
+ 30
+ ],
+ [
+ 18,
+ 1,
+ 43,
+ 15,
+ 3,
+ 8
+ ],
+ [
+ 59,
+ 55,
+ 13,
+ 20,
+ 44,
+ 40
+ ],
+ [
+ 33,
+ 45,
+ 27,
+ 53,
+ 52,
+ 63
+ ],
+ [
+ 28,
+ 57,
+ 51,
+ 34,
+ 53,
+ 31
+ ],
+ [
+ 13,
+ 63,
+ 3,
+ 32,
+ 44,
+ 45
+ ],
+ [
+ 59,
+ 33,
+ 12,
+ 57,
+ 6,
+ 35
+ ],
+ [
+ 23,
+ 30,
+ 42,
+ 10,
+ 29,
+ 13
+ ],
+ [
+ 63,
+ 23,
+ 18,
+ 42,
+ 38,
+ 19
+ ],
+ [
+ 62,
+ 2,
+ 22,
+ 19,
+ 45,
+ 6
+ ],
+ [
+ 6,
+ 26,
+ 20,
+ 30,
+ 5,
+ 25
+ ],
+ [
+ 3,
+ 6,
+ 8,
+ 31,
+ 17,
+ 37
+ ],
+ [
+ 59,
+ 2,
+ 10,
+ 54,
+ 55,
+ 19
+ ],
+ [
+ 45,
+ 7,
+ 57,
+ 42,
+ 54,
+ 46
+ ],
+ [
+ 60,
+ 55,
+ 53,
+ 34,
+ 41,
+ 18
+ ],
+ [
+ 36,
+ 2,
+ 8,
+ 31,
+ 32,
+ 46
+ ],
+ [
+ 36,
+ 45,
+ 38,
+ 32,
+ 61,
+ 51
+ ],
+ [
+ 57,
+ 39,
+ 6,
+ 18,
+ 19,
+ 30
+ ],
+ [
+ 37,
+ 39,
+ 59,
+ 48,
+ 53,
+ 14
+ ],
+ [
+ 52,
+ 28,
+ 46,
+ 55,
+ 47,
+ 61
+ ],
+ [
+ 1,
+ 28,
+ 52,
+ 15,
+ 31,
+ 18
+ ],
+ [
+ 22,
+ 21,
+ 8,
+ 10,
+ 46,
+ 39
+ ],
+ [
+ 45,
+ 41,
+ 35,
+ 9,
+ 54,
+ 33
+ ],
+ [
+ 42,
+ 39,
+ 58,
+ 61,
+ 24,
+ 3
+ ],
+ [
+ 41,
+ 46,
+ 10,
+ 3,
+ 15,
+ 33
+ ]
+ ],
+ [
+ [
+ 48,
+ 42,
+ 38,
+ 63,
+ 47,
+ 7
+ ],
+ [
+ 3,
+ 10,
+ 26,
+ 62,
+ 2,
+ 6
+ ],
+ [
+ 39,
+ 44,
+ 7,
+ 6,
+ 45,
+ 40
+ ],
+ [
+ 21,
+ 39,
+ 1,
+ 18,
+ 15,
+ 41
+ ],
+ [
+ 6,
+ 33,
+ 9,
+ 13,
+ 3,
+ 27
+ ],
+ [
+ 59,
+ 33,
+ 53,
+ 62,
+ 21,
+ 45
+ ],
+ [
+ 28,
+ 10,
+ 33,
+ 7,
+ 57,
+ 50
+ ],
+ [
+ 50,
+ 13,
+ 12,
+ 49,
+ 3,
+ 55
+ ],
+ [
+ 33,
+ 59,
+ 26,
+ 35,
+ 48,
+ 38
+ ],
+ [
+ 23,
+ 30,
+ 20,
+ 43,
+ 10,
+ 58
+ ],
+ [
+ 63,
+ 23,
+ 18,
+ 17,
+ 38,
+ 9
+ ],
+ [
+ 22,
+ 62,
+ 36,
+ 33,
+ 6,
+ 2
+ ],
+ [
+ 12,
+ 17,
+ 59,
+ 7,
+ 26,
+ 49
+ ],
+ [
+ 47,
+ 3,
+ 27,
+ 6,
+ 24,
+ 22
+ ],
+ [
+ 59,
+ 57,
+ 46,
+ 54,
+ 2,
+ 10
+ ],
+ [
+ 27,
+ 45,
+ 57,
+ 54,
+ 34,
+ 61
+ ],
+ [
+ 34,
+ 60,
+ 55,
+ 31,
+ 58,
+ 43
+ ],
+ [
+ 48,
+ 42,
+ 32,
+ 39,
+ 2,
+ 38
+ ],
+ [
+ 42,
+ 49,
+ 32,
+ 44,
+ 12,
+ 61
+ ],
+ [
+ 57,
+ 36,
+ 39,
+ 1,
+ 30,
+ 59
+ ],
+ [
+ 48,
+ 7,
+ 21,
+ 53,
+ 17,
+ 29
+ ],
+ [
+ 8,
+ 60,
+ 58,
+ 35,
+ 14,
+ 46
+ ],
+ [
+ 51,
+ 15,
+ 28,
+ 47,
+ 33,
+ 30
+ ],
+ [
+ 11,
+ 4,
+ 19,
+ 8,
+ 58,
+ 52
+ ],
+ [
+ 24,
+ 5,
+ 41,
+ 60,
+ 40,
+ 54
+ ],
+ [
+ 61,
+ 3,
+ 21,
+ 58,
+ 19,
+ 8
+ ],
+ [
+ 60,
+ 35,
+ 54,
+ 49,
+ 1,
+ 0
+ ]
+ ],
+ [
+ [
+ 6,
+ 24,
+ 63,
+ 25,
+ 26,
+ 45
+ ],
+ [
+ 47,
+ 13,
+ 49,
+ 44,
+ 20,
+ 19
+ ],
+ [
+ 23,
+ 32,
+ 49,
+ 20,
+ 24,
+ 2
+ ],
+ [
+ 43,
+ 21,
+ 8,
+ 40,
+ 39,
+ 45
+ ],
+ [
+ 39,
+ 29,
+ 3,
+ 5,
+ 41,
+ 12
+ ],
+ [
+ 61,
+ 33,
+ 48,
+ 40,
+ 29,
+ 62
+ ],
+ [
+ 29,
+ 28,
+ 25,
+ 33,
+ 44,
+ 31
+ ],
+ [
+ 50,
+ 33,
+ 13,
+ 11,
+ 30,
+ 54
+ ],
+ [
+ 48,
+ 26,
+ 35,
+ 17,
+ 55,
+ 6
+ ],
+ [
+ 43,
+ 23,
+ 12,
+ 9,
+ 25,
+ 62
+ ],
+ [
+ 63,
+ 35,
+ 18,
+ 53,
+ 38,
+ 17
+ ],
+ [
+ 19,
+ 48,
+ 5,
+ 36,
+ 59,
+ 17
+ ],
+ [
+ 7,
+ 26,
+ 59,
+ 17,
+ 12,
+ 30
+ ],
+ [
+ 47,
+ 22,
+ 24,
+ 62,
+ 25,
+ 42
+ ],
+ [
+ 59,
+ 46,
+ 35,
+ 39,
+ 57,
+ 54
+ ],
+ [
+ 27,
+ 9,
+ 20,
+ 0,
+ 57,
+ 52
+ ],
+ [
+ 38,
+ 58,
+ 60,
+ 34,
+ 43,
+ 29
+ ],
+ [
+ 42,
+ 32,
+ 39,
+ 54,
+ 38,
+ 57
+ ],
+ [
+ 59,
+ 32,
+ 42,
+ 6,
+ 37,
+ 18
+ ],
+ [
+ 24,
+ 36,
+ 57,
+ 4,
+ 30,
+ 60
+ ],
+ [
+ 9,
+ 48,
+ 7,
+ 53,
+ 21,
+ 58
+ ],
+ [
+ 49,
+ 31,
+ 14,
+ 8,
+ 19,
+ 22
+ ],
+ [
+ 15,
+ 33,
+ 44,
+ 8,
+ 14,
+ 3
+ ],
+ [
+ 11,
+ 4,
+ 21,
+ 28,
+ 41,
+ 23
+ ],
+ [
+ 37,
+ 27,
+ 24,
+ 12,
+ 9,
+ 20
+ ],
+ [
+ 58,
+ 9,
+ 19,
+ 3,
+ 12,
+ 48
+ ],
+ [
+ 24,
+ 14,
+ 60,
+ 47,
+ 25,
+ 13
+ ]
+ ],
+ [
+ [
+ 26,
+ 62,
+ 58,
+ 18,
+ 38,
+ 5
+ ],
+ [
+ 19,
+ 12,
+ 40,
+ 39,
+ 31,
+ 57
+ ],
+ [
+ 27,
+ 38,
+ 9,
+ 22,
+ 23,
+ 61
+ ],
+ [
+ 42,
+ 20,
+ 63,
+ 39,
+ 45,
+ 43
+ ],
+ [
+ 7,
+ 36,
+ 60,
+ 29,
+ 57,
+ 24
+ ],
+ [
+ 61,
+ 34,
+ 3,
+ 1,
+ 44,
+ 51
+ ],
+ [
+ 29,
+ 42,
+ 25,
+ 47,
+ 30,
+ 17
+ ],
+ [
+ 33,
+ 38,
+ 39,
+ 50,
+ 62,
+ 0
+ ],
+ [
+ 17,
+ 26,
+ 13,
+ 50,
+ 41,
+ 11
+ ],
+ [
+ 12,
+ 43,
+ 35,
+ 16,
+ 55,
+ 60
+ ],
+ [
+ 3,
+ 18,
+ 39,
+ 63,
+ 35,
+ 62
+ ],
+ [
+ 19,
+ 48,
+ 39,
+ 60,
+ 54,
+ 2
+ ],
+ [
+ 7,
+ 42,
+ 56,
+ 59,
+ 37,
+ 24
+ ],
+ [
+ 56,
+ 47,
+ 54,
+ 59,
+ 3,
+ 6
+ ],
+ [
+ 35,
+ 39,
+ 43,
+ 59,
+ 2,
+ 13
+ ],
+ [
+ 20,
+ 27,
+ 9,
+ 56,
+ 0,
+ 54
+ ],
+ [
+ 17,
+ 60,
+ 29,
+ 18,
+ 58,
+ 5
+ ],
+ [
+ 57,
+ 42,
+ 45,
+ 0,
+ 24,
+ 29
+ ],
+ [
+ 49,
+ 28,
+ 52,
+ 32,
+ 50,
+ 3
+ ],
+ [
+ 24,
+ 2,
+ 36,
+ 30,
+ 59,
+ 46
+ ],
+ [
+ 11,
+ 7,
+ 48,
+ 54,
+ 53,
+ 21
+ ],
+ [
+ 33,
+ 14,
+ 42,
+ 6,
+ 8,
+ 5
+ ],
+ [
+ 15,
+ 36,
+ 33,
+ 17,
+ 51,
+ 13
+ ],
+ [
+ 41,
+ 11,
+ 2,
+ 4,
+ 8,
+ 29
+ ],
+ [
+ 30,
+ 15,
+ 60,
+ 46,
+ 5,
+ 9
+ ],
+ [
+ 21,
+ 3,
+ 14,
+ 19,
+ 61,
+ 58
+ ],
+ [
+ 24,
+ 60,
+ 1,
+ 14,
+ 35,
+ 53
+ ]
+ ],
+ [
+ [
+ 8,
+ 56,
+ 54,
+ 37,
+ 4,
+ 38
+ ],
+ [
+ 36,
+ 9,
+ 24,
+ 8,
+ 1,
+ 2
+ ],
+ [
+ 16,
+ 57,
+ 29,
+ 32,
+ 58,
+ 30
+ ],
+ [
+ 32,
+ 63,
+ 42,
+ 52,
+ 4,
+ 20
+ ],
+ [
+ 27,
+ 24,
+ 44,
+ 39,
+ 63,
+ 20
+ ],
+ [
+ 24,
+ 11,
+ 28,
+ 6,
+ 15,
+ 31
+ ],
+ [
+ 30,
+ 41,
+ 42,
+ 25,
+ 52,
+ 2
+ ],
+ [
+ 26,
+ 36,
+ 62,
+ 18,
+ 50,
+ 21
+ ],
+ [
+ 48,
+ 50,
+ 32,
+ 51,
+ 27,
+ 26
+ ],
+ [
+ 13,
+ 43,
+ 3,
+ 26,
+ 12,
+ 41
+ ],
+ [
+ 3,
+ 32,
+ 18,
+ 53,
+ 39,
+ 17
+ ],
+ [
+ 21,
+ 19,
+ 9,
+ 36,
+ 48,
+ 1
+ ],
+ [
+ 35,
+ 59,
+ 54,
+ 42,
+ 63,
+ 17
+ ],
+ [
+ 15,
+ 12,
+ 38,
+ 9,
+ 51,
+ 54
+ ],
+ [
+ 3,
+ 2,
+ 48,
+ 59,
+ 57,
+ 13
+ ],
+ [
+ 27,
+ 9,
+ 4,
+ 20,
+ 22,
+ 3
+ ],
+ [
+ 60,
+ 40,
+ 22,
+ 27,
+ 46,
+ 32
+ ],
+ [
+ 30,
+ 21,
+ 25,
+ 17,
+ 0,
+ 42
+ ],
+ [
+ 33,
+ 27,
+ 34,
+ 28,
+ 38,
+ 44
+ ],
+ [
+ 53,
+ 32,
+ 33,
+ 35,
+ 31,
+ 56
+ ],
+ [
+ 40,
+ 11,
+ 20,
+ 47,
+ 48,
+ 24
+ ],
+ [
+ 45,
+ 62,
+ 33,
+ 53,
+ 17,
+ 34
+ ],
+ [
+ 41,
+ 9,
+ 39,
+ 25,
+ 17,
+ 32
+ ],
+ [
+ 44,
+ 30,
+ 2,
+ 31,
+ 0,
+ 47
+ ],
+ [
+ 6,
+ 62,
+ 37,
+ 52,
+ 55,
+ 33
+ ],
+ [
+ 10,
+ 24,
+ 57,
+ 9,
+ 49,
+ 13
+ ],
+ [
+ 45,
+ 24,
+ 27,
+ 47,
+ 19,
+ 26
+ ]
+ ],
+ [
+ [
+ 4,
+ 16,
+ 22,
+ 23,
+ 3,
+ 13
+ ],
+ [
+ 16,
+ 23,
+ 9,
+ 59,
+ 13,
+ 50
+ ],
+ [
+ 11,
+ 35,
+ 21,
+ 7,
+ 9,
+ 59
+ ],
+ [
+ 44,
+ 1,
+ 25,
+ 26,
+ 15,
+ 20
+ ],
+ [
+ 35,
+ 57,
+ 52,
+ 31,
+ 24,
+ 5
+ ],
+ [
+ 35,
+ 15,
+ 38,
+ 53,
+ 12,
+ 0
+ ],
+ [
+ 20,
+ 36,
+ 63,
+ 28,
+ 60,
+ 33
+ ],
+ [
+ 49,
+ 50,
+ 12,
+ 13,
+ 8,
+ 16
+ ],
+ [
+ 42,
+ 48,
+ 6,
+ 16,
+ 35,
+ 41
+ ],
+ [
+ 23,
+ 9,
+ 54,
+ 34,
+ 30,
+ 13
+ ],
+ [
+ 23,
+ 63,
+ 51,
+ 53,
+ 55,
+ 3
+ ],
+ [
+ 33,
+ 36,
+ 62,
+ 19,
+ 59,
+ 57
+ ],
+ [
+ 59,
+ 38,
+ 12,
+ 17,
+ 32,
+ 53
+ ],
+ [
+ 48,
+ 0,
+ 19,
+ 24,
+ 61,
+ 22
+ ],
+ [
+ 55,
+ 6,
+ 39,
+ 7,
+ 60,
+ 18
+ ],
+ [
+ 27,
+ 57,
+ 19,
+ 1,
+ 60,
+ 47
+ ],
+ [
+ 30,
+ 40,
+ 10,
+ 17,
+ 36,
+ 60
+ ],
+ [
+ 13,
+ 20,
+ 1,
+ 31,
+ 17,
+ 54
+ ],
+ [
+ 50,
+ 32,
+ 23,
+ 7,
+ 33,
+ 30
+ ],
+ [
+ 14,
+ 19,
+ 23,
+ 13,
+ 10,
+ 7
+ ],
+ [
+ 53,
+ 23,
+ 19,
+ 13,
+ 5,
+ 7
+ ],
+ [
+ 30,
+ 18,
+ 46,
+ 14,
+ 1,
+ 0
+ ],
+ [
+ 15,
+ 18,
+ 52,
+ 0,
+ 51,
+ 30
+ ],
+ [
+ 4,
+ 21,
+ 53,
+ 41,
+ 10,
+ 14
+ ],
+ [
+ 4,
+ 9,
+ 39,
+ 8,
+ 16,
+ 54
+ ],
+ [
+ 3,
+ 19,
+ 16,
+ 30,
+ 47,
+ 27
+ ],
+ [
+ 33,
+ 8,
+ 46,
+ 40,
+ 29,
+ 14
+ ]
+ ],
+ [
+ [
+ 6,
+ 26,
+ 3,
+ 24,
+ 11,
+ 38
+ ],
+ [
+ 49,
+ 19,
+ 16,
+ 57,
+ 0,
+ 18
+ ],
+ [
+ 16,
+ 60,
+ 57,
+ 0,
+ 22,
+ 30
+ ],
+ [
+ 35,
+ 46,
+ 49,
+ 44,
+ 26,
+ 17
+ ],
+ [
+ 25,
+ 54,
+ 43,
+ 9,
+ 45,
+ 27
+ ],
+ [
+ 55,
+ 33,
+ 30,
+ 6,
+ 28,
+ 57
+ ],
+ [
+ 55,
+ 31,
+ 49,
+ 52,
+ 25,
+ 15
+ ],
+ [
+ 11,
+ 36,
+ 15,
+ 29,
+ 30,
+ 62
+ ],
+ [
+ 55,
+ 42,
+ 48,
+ 54,
+ 46,
+ 6
+ ],
+ [
+ 10,
+ 3,
+ 43,
+ 21,
+ 62,
+ 54
+ ],
+ [
+ 32,
+ 56,
+ 6,
+ 3,
+ 18,
+ 55
+ ],
+ [
+ 50,
+ 21,
+ 36,
+ 19,
+ 4,
+ 42
+ ],
+ [
+ 38,
+ 35,
+ 53,
+ 39,
+ 41,
+ 32
+ ],
+ [
+ 28,
+ 24,
+ 38,
+ 41,
+ 15,
+ 12
+ ],
+ [
+ 2,
+ 40,
+ 18,
+ 60,
+ 55,
+ 13
+ ],
+ [
+ 27,
+ 22,
+ 57,
+ 8,
+ 54,
+ 37
+ ],
+ [
+ 40,
+ 27,
+ 60,
+ 46,
+ 44,
+ 50
+ ],
+ [
+ 17,
+ 21,
+ 30,
+ 12,
+ 29,
+ 26
+ ],
+ [
+ 33,
+ 38,
+ 23,
+ 2,
+ 13,
+ 27
+ ],
+ [
+ 32,
+ 56,
+ 9,
+ 44,
+ 31,
+ 60
+ ],
+ [
+ 53,
+ 12,
+ 22,
+ 40,
+ 41,
+ 5
+ ],
+ [
+ 44,
+ 45,
+ 49,
+ 17,
+ 14,
+ 56
+ ],
+ [
+ 25,
+ 39,
+ 41,
+ 4,
+ 9,
+ 53
+ ],
+ [
+ 44,
+ 1,
+ 17,
+ 0,
+ 13,
+ 58
+ ],
+ [
+ 62,
+ 52,
+ 37,
+ 54,
+ 42,
+ 36
+ ],
+ [
+ 10,
+ 9,
+ 28,
+ 55,
+ 2,
+ 34
+ ],
+ [
+ 57,
+ 30,
+ 27,
+ 45,
+ 47,
+ 16
+ ]
+ ],
+ [
+ [
+ 11,
+ 16,
+ 31,
+ 46,
+ 0,
+ 21
+ ],
+ [
+ 13,
+ 49,
+ 34,
+ 31,
+ 16,
+ 8
+ ],
+ [
+ 36,
+ 13,
+ 27,
+ 58,
+ 18,
+ 34
+ ],
+ [
+ 24,
+ 61,
+ 15,
+ 46,
+ 63,
+ 25
+ ],
+ [
+ 17,
+ 2,
+ 50,
+ 35,
+ 58,
+ 44
+ ],
+ [
+ 8,
+ 23,
+ 7,
+ 49,
+ 26,
+ 47
+ ],
+ [
+ 61,
+ 38,
+ 29,
+ 58,
+ 0,
+ 36
+ ],
+ [
+ 42,
+ 20,
+ 9,
+ 4,
+ 11,
+ 27
+ ],
+ [
+ 1,
+ 34,
+ 30,
+ 48,
+ 7,
+ 26
+ ],
+ [
+ 27,
+ 10,
+ 28,
+ 62,
+ 13,
+ 54
+ ],
+ [
+ 60,
+ 1,
+ 30,
+ 59,
+ 6,
+ 52
+ ],
+ [
+ 31,
+ 62,
+ 25,
+ 42,
+ 21,
+ 32
+ ],
+ [
+ 34,
+ 32,
+ 19,
+ 2,
+ 62,
+ 14
+ ],
+ [
+ 6,
+ 40,
+ 24,
+ 31,
+ 13,
+ 48
+ ],
+ [
+ 44,
+ 2,
+ 40,
+ 8,
+ 34,
+ 45
+ ],
+ [
+ 8,
+ 27,
+ 7,
+ 57,
+ 50,
+ 47
+ ],
+ [
+ 13,
+ 60,
+ 45,
+ 22,
+ 25,
+ 52
+ ],
+ [
+ 4,
+ 23,
+ 33,
+ 46,
+ 58,
+ 34
+ ],
+ [
+ 8,
+ 33,
+ 56,
+ 45,
+ 51,
+ 59
+ ],
+ [
+ 38,
+ 53,
+ 39,
+ 26,
+ 35,
+ 31
+ ],
+ [
+ 41,
+ 46,
+ 11,
+ 53,
+ 59,
+ 8
+ ],
+ [
+ 11,
+ 44,
+ 33,
+ 52,
+ 55,
+ 53
+ ],
+ [
+ 31,
+ 1,
+ 16,
+ 9,
+ 15,
+ 41
+ ],
+ [
+ 2,
+ 31,
+ 14,
+ 10,
+ 44,
+ 7
+ ],
+ [
+ 45,
+ 6,
+ 62,
+ 12,
+ 18,
+ 24
+ ],
+ [
+ 10,
+ 50,
+ 42,
+ 54,
+ 24,
+ 3
+ ],
+ [
+ 17,
+ 31,
+ 37,
+ 5,
+ 10,
+ 41
+ ]
+ ],
+ [
+ [
+ 45,
+ 37,
+ 48,
+ 29,
+ 30,
+ 59
+ ],
+ [
+ 8,
+ 59,
+ 10,
+ 60,
+ 43,
+ 55
+ ],
+ [
+ 45,
+ 51,
+ 28,
+ 59,
+ 34,
+ 31
+ ],
+ [
+ 14,
+ 31,
+ 22,
+ 2,
+ 19,
+ 44
+ ],
+ [
+ 59,
+ 50,
+ 4,
+ 31,
+ 9,
+ 44
+ ],
+ [
+ 58,
+ 23,
+ 47,
+ 9,
+ 53,
+ 30
+ ],
+ [
+ 54,
+ 13,
+ 51,
+ 22,
+ 29,
+ 61
+ ],
+ [
+ 47,
+ 42,
+ 32,
+ 20,
+ 24,
+ 37
+ ],
+ [
+ 33,
+ 12,
+ 47,
+ 10,
+ 30,
+ 44
+ ],
+ [
+ 30,
+ 27,
+ 10,
+ 28,
+ 33,
+ 13
+ ],
+ [
+ 61,
+ 23,
+ 14,
+ 20,
+ 1,
+ 29
+ ],
+ [
+ 12,
+ 14,
+ 32,
+ 62,
+ 31,
+ 52
+ ],
+ [
+ 5,
+ 36,
+ 19,
+ 20,
+ 2,
+ 32
+ ],
+ [
+ 46,
+ 24,
+ 1,
+ 40,
+ 17,
+ 33
+ ],
+ [
+ 44,
+ 8,
+ 16,
+ 30,
+ 2,
+ 6
+ ],
+ [
+ 6,
+ 46,
+ 15,
+ 54,
+ 21,
+ 33
+ ],
+ [
+ 2,
+ 30,
+ 60,
+ 61,
+ 18,
+ 10
+ ],
+ [
+ 6,
+ 17,
+ 33,
+ 27,
+ 58,
+ 4
+ ],
+ [
+ 56,
+ 46,
+ 8,
+ 60,
+ 22,
+ 16
+ ],
+ [
+ 5,
+ 13,
+ 14,
+ 35,
+ 36,
+ 52
+ ],
+ [
+ 15,
+ 53,
+ 59,
+ 23,
+ 6,
+ 46
+ ],
+ [
+ 36,
+ 30,
+ 60,
+ 33,
+ 53,
+ 38
+ ],
+ [
+ 10,
+ 15,
+ 16,
+ 52,
+ 4,
+ 38
+ ],
+ [
+ 31,
+ 1,
+ 10,
+ 4,
+ 41,
+ 59
+ ],
+ [
+ 32,
+ 9,
+ 8,
+ 59,
+ 33,
+ 18
+ ],
+ [
+ 33,
+ 19,
+ 6,
+ 3,
+ 45,
+ 50
+ ],
+ [
+ 55,
+ 4,
+ 43,
+ 5,
+ 25,
+ 34
+ ]
+ ],
+ [
+ [
+ 25,
+ 14,
+ 18,
+ 49,
+ 51,
+ 63
+ ],
+ [
+ 42,
+ 21,
+ 30,
+ 43,
+ 24,
+ 7
+ ],
+ [
+ 54,
+ 39,
+ 9,
+ 59,
+ 28,
+ 45
+ ],
+ [
+ 1,
+ 23,
+ 55,
+ 8,
+ 16,
+ 7
+ ],
+ [
+ 59,
+ 37,
+ 30,
+ 41,
+ 16,
+ 1
+ ],
+ [
+ 4,
+ 31,
+ 41,
+ 45,
+ 18,
+ 56
+ ],
+ [
+ 28,
+ 55,
+ 44,
+ 54,
+ 9,
+ 51
+ ],
+ [
+ 24,
+ 47,
+ 12,
+ 63,
+ 32,
+ 61
+ ],
+ [
+ 33,
+ 47,
+ 39,
+ 44,
+ 38,
+ 56
+ ],
+ [
+ 40,
+ 30,
+ 59,
+ 33,
+ 58,
+ 10
+ ],
+ [
+ 19,
+ 8,
+ 23,
+ 4,
+ 25,
+ 34
+ ],
+ [
+ 45,
+ 14,
+ 12,
+ 42,
+ 62,
+ 52
+ ],
+ [
+ 58,
+ 45,
+ 59,
+ 29,
+ 25,
+ 36
+ ],
+ [
+ 8,
+ 59,
+ 1,
+ 47,
+ 33,
+ 24
+ ],
+ [
+ 24,
+ 30,
+ 38,
+ 56,
+ 4,
+ 7
+ ],
+ [
+ 46,
+ 22,
+ 44,
+ 57,
+ 12,
+ 40
+ ],
+ [
+ 34,
+ 60,
+ 2,
+ 17,
+ 18,
+ 29
+ ],
+ [
+ 6,
+ 61,
+ 51,
+ 39,
+ 53,
+ 32
+ ],
+ [
+ 49,
+ 5,
+ 47,
+ 60,
+ 21,
+ 59
+ ],
+ [
+ 57,
+ 4,
+ 23,
+ 39,
+ 45,
+ 60
+ ],
+ [
+ 7,
+ 36,
+ 45,
+ 53,
+ 21,
+ 56
+ ],
+ [
+ 60,
+ 3,
+ 39,
+ 18,
+ 8,
+ 53
+ ],
+ [
+ 57,
+ 33,
+ 15,
+ 47,
+ 38,
+ 16
+ ],
+ [
+ 20,
+ 11,
+ 19,
+ 4,
+ 2,
+ 8
+ ],
+ [
+ 36,
+ 34,
+ 60,
+ 47,
+ 1,
+ 40
+ ],
+ [
+ 33,
+ 14,
+ 3,
+ 21,
+ 19,
+ 35
+ ],
+ [
+ 60,
+ 24,
+ 22,
+ 52,
+ 35,
+ 53
+ ]
+ ],
+ [
+ [
+ 23,
+ 54,
+ 53,
+ 58,
+ 11,
+ 8
+ ],
+ [
+ 11,
+ 30,
+ 15,
+ 59,
+ 55,
+ 63
+ ],
+ [
+ 20,
+ 58,
+ 29,
+ 17,
+ 42,
+ 30
+ ],
+ [
+ 18,
+ 1,
+ 43,
+ 15,
+ 8,
+ 3
+ ],
+ [
+ 59,
+ 55,
+ 13,
+ 28,
+ 26,
+ 63
+ ],
+ [
+ 33,
+ 45,
+ 27,
+ 53,
+ 63,
+ 19
+ ],
+ [
+ 28,
+ 57,
+ 51,
+ 54,
+ 34,
+ 53
+ ],
+ [
+ 24,
+ 13,
+ 47,
+ 3,
+ 45,
+ 50
+ ],
+ [
+ 47,
+ 59,
+ 33,
+ 57,
+ 37,
+ 35
+ ],
+ [
+ 23,
+ 42,
+ 33,
+ 41,
+ 48,
+ 30
+ ],
+ [
+ 42,
+ 63,
+ 23,
+ 25,
+ 34,
+ 17
+ ],
+ [
+ 14,
+ 62,
+ 2,
+ 19,
+ 45,
+ 43
+ ],
+ [
+ 12,
+ 6,
+ 20,
+ 30,
+ 29,
+ 17
+ ],
+ [
+ 6,
+ 3,
+ 17,
+ 8,
+ 27,
+ 31
+ ],
+ [
+ 19,
+ 59,
+ 2,
+ 10,
+ 54,
+ 26
+ ],
+ [
+ 45,
+ 61,
+ 19,
+ 57,
+ 42,
+ 17
+ ],
+ [
+ 55,
+ 60,
+ 41,
+ 34,
+ 35,
+ 53
+ ],
+ [
+ 36,
+ 2,
+ 8,
+ 31,
+ 14,
+ 5
+ ],
+ [
+ 45,
+ 36,
+ 16,
+ 38,
+ 51,
+ 21
+ ],
+ [
+ 57,
+ 39,
+ 48,
+ 6,
+ 19,
+ 60
+ ],
+ [
+ 39,
+ 37,
+ 59,
+ 48,
+ 42,
+ 40
+ ],
+ [
+ 52,
+ 28,
+ 46,
+ 33,
+ 18,
+ 55
+ ],
+ [
+ 1,
+ 15,
+ 28,
+ 18,
+ 13,
+ 52
+ ],
+ [
+ 10,
+ 21,
+ 4,
+ 2,
+ 23,
+ 31
+ ],
+ [
+ 45,
+ 9,
+ 41,
+ 18,
+ 54,
+ 3
+ ],
+ [
+ 42,
+ 61,
+ 36,
+ 3,
+ 19,
+ 27
+ ],
+ [
+ 41,
+ 46,
+ 33,
+ 3,
+ 10,
+ 15
+ ]
+ ],
+ [
+ [
+ 49,
+ 52,
+ 60,
+ 63,
+ 21,
+ 0
+ ],
+ [
+ 14,
+ 7,
+ 25,
+ 52,
+ 58,
+ 36
+ ],
+ [
+ 46,
+ 57,
+ 28,
+ 24,
+ 49,
+ 12
+ ],
+ [
+ 10,
+ 21,
+ 39,
+ 1,
+ 11,
+ 17
+ ],
+ [
+ 53,
+ 3,
+ 2,
+ 16,
+ 46,
+ 44
+ ],
+ [
+ 25,
+ 40,
+ 39,
+ 15,
+ 34,
+ 30
+ ],
+ [
+ 62,
+ 24,
+ 47,
+ 52,
+ 17,
+ 49
+ ],
+ [
+ 34,
+ 41,
+ 39,
+ 11,
+ 33,
+ 15
+ ],
+ [
+ 36,
+ 26,
+ 48,
+ 39,
+ 15,
+ 56
+ ],
+ [
+ 23,
+ 36,
+ 43,
+ 20,
+ 10,
+ 21
+ ],
+ [
+ 48,
+ 63,
+ 51,
+ 10,
+ 62,
+ 18
+ ],
+ [
+ 10,
+ 42,
+ 36,
+ 39,
+ 49,
+ 19
+ ],
+ [
+ 22,
+ 38,
+ 51,
+ 56,
+ 50,
+ 7
+ ],
+ [
+ 4,
+ 31,
+ 19,
+ 22,
+ 17,
+ 7
+ ],
+ [
+ 41,
+ 7,
+ 39,
+ 2,
+ 26,
+ 57
+ ],
+ [
+ 33,
+ 27,
+ 58,
+ 28,
+ 20,
+ 45
+ ],
+ [
+ 58,
+ 8,
+ 37,
+ 18,
+ 34,
+ 62
+ ],
+ [
+ 24,
+ 34,
+ 43,
+ 39,
+ 17,
+ 19
+ ],
+ [
+ 23,
+ 50,
+ 12,
+ 32,
+ 43,
+ 31
+ ],
+ [
+ 9,
+ 7,
+ 51,
+ 16,
+ 30,
+ 60
+ ],
+ [
+ 25,
+ 62,
+ 14,
+ 53,
+ 58,
+ 0
+ ],
+ [
+ 53,
+ 4,
+ 18,
+ 22,
+ 14,
+ 29
+ ],
+ [
+ 6,
+ 14,
+ 15,
+ 48,
+ 58,
+ 0
+ ],
+ [
+ 35,
+ 4,
+ 45,
+ 3,
+ 59,
+ 39
+ ],
+ [
+ 17,
+ 57,
+ 23,
+ 16,
+ 40,
+ 37
+ ],
+ [
+ 9,
+ 19,
+ 3,
+ 36,
+ 11,
+ 47
+ ],
+ [
+ 39,
+ 14,
+ 22,
+ 33,
+ 62,
+ 42
+ ]
+ ],
+ [
+ [
+ 14,
+ 17,
+ 2,
+ 39,
+ 47,
+ 63
+ ],
+ [
+ 23,
+ 58,
+ 2,
+ 25,
+ 5,
+ 18
+ ],
+ [
+ 27,
+ 33,
+ 0,
+ 56,
+ 6,
+ 7
+ ],
+ [
+ 23,
+ 5,
+ 30,
+ 7,
+ 21,
+ 39
+ ],
+ [
+ 39,
+ 53,
+ 27,
+ 38,
+ 54,
+ 41
+ ],
+ [
+ 37,
+ 25,
+ 50,
+ 40,
+ 33,
+ 36
+ ],
+ [
+ 49,
+ 37,
+ 24,
+ 55,
+ 29,
+ 28
+ ],
+ [
+ 34,
+ 15,
+ 36,
+ 16,
+ 30,
+ 11
+ ],
+ [
+ 5,
+ 36,
+ 2,
+ 26,
+ 16,
+ 48
+ ],
+ [
+ 56,
+ 46,
+ 10,
+ 23,
+ 39,
+ 45
+ ],
+ [
+ 10,
+ 56,
+ 62,
+ 45,
+ 49,
+ 3
+ ],
+ [
+ 35,
+ 6,
+ 50,
+ 36,
+ 11,
+ 1
+ ],
+ [
+ 50,
+ 46,
+ 39,
+ 51,
+ 38,
+ 41
+ ],
+ [
+ 41,
+ 19,
+ 31,
+ 6,
+ 48,
+ 24
+ ],
+ [
+ 2,
+ 7,
+ 57,
+ 18,
+ 47,
+ 63
+ ],
+ [
+ 27,
+ 58,
+ 22,
+ 8,
+ 57,
+ 45
+ ],
+ [
+ 37,
+ 59,
+ 51,
+ 44,
+ 55,
+ 18
+ ],
+ [
+ 52,
+ 21,
+ 25,
+ 12,
+ 34,
+ 39
+ ],
+ [
+ 23,
+ 34,
+ 60,
+ 13,
+ 27,
+ 32
+ ],
+ [
+ 32,
+ 56,
+ 26,
+ 24,
+ 44,
+ 30
+ ],
+ [
+ 41,
+ 12,
+ 53,
+ 2,
+ 5,
+ 56
+ ],
+ [
+ 44,
+ 18,
+ 39,
+ 33,
+ 38,
+ 46
+ ],
+ [
+ 25,
+ 0,
+ 59,
+ 18,
+ 11,
+ 53
+ ],
+ [
+ 1,
+ 7,
+ 17,
+ 30,
+ 4,
+ 37
+ ],
+ [
+ 62,
+ 48,
+ 19,
+ 54,
+ 37,
+ 22
+ ],
+ [
+ 10,
+ 28,
+ 3,
+ 16,
+ 11,
+ 42
+ ],
+ [
+ 26,
+ 30,
+ 57,
+ 42,
+ 41,
+ 32
+ ]
+ ],
+ [
+ [
+ 36,
+ 31,
+ 37,
+ 16,
+ 43,
+ 63
+ ],
+ [
+ 26,
+ 51,
+ 0,
+ 48,
+ 42,
+ 21
+ ],
+ [
+ 18,
+ 41,
+ 37,
+ 34,
+ 24,
+ 30
+ ],
+ [
+ 58,
+ 4,
+ 22,
+ 44,
+ 51,
+ 18
+ ],
+ [
+ 18,
+ 26,
+ 4,
+ 61,
+ 14,
+ 31
+ ],
+ [
+ 9,
+ 23,
+ 34,
+ 29,
+ 3,
+ 31
+ ],
+ [
+ 12,
+ 58,
+ 36,
+ 46,
+ 16,
+ 25
+ ],
+ [
+ 4,
+ 56,
+ 9,
+ 33,
+ 60,
+ 32
+ ],
+ [
+ 14,
+ 26,
+ 43,
+ 30,
+ 46,
+ 35
+ ],
+ [
+ 2,
+ 0,
+ 10,
+ 43,
+ 28,
+ 27
+ ],
+ [
+ 61,
+ 18,
+ 19,
+ 17,
+ 2,
+ 33
+ ],
+ [
+ 60,
+ 19,
+ 23,
+ 62,
+ 37,
+ 43
+ ],
+ [
+ 56,
+ 32,
+ 8,
+ 61,
+ 58,
+ 38
+ ],
+ [
+ 25,
+ 28,
+ 22,
+ 0,
+ 26,
+ 29
+ ],
+ [
+ 22,
+ 49,
+ 7,
+ 60,
+ 23,
+ 55
+ ],
+ [
+ 46,
+ 57,
+ 19,
+ 37,
+ 12,
+ 27
+ ],
+ [
+ 22,
+ 60,
+ 11,
+ 30,
+ 19,
+ 20
+ ],
+ [
+ 36,
+ 20,
+ 17,
+ 2,
+ 37,
+ 55
+ ],
+ [
+ 13,
+ 16,
+ 32,
+ 58,
+ 41,
+ 10
+ ],
+ [
+ 50,
+ 55,
+ 13,
+ 25,
+ 14,
+ 41
+ ],
+ [
+ 14,
+ 3,
+ 44,
+ 48,
+ 60,
+ 6
+ ],
+ [
+ 36,
+ 18,
+ 57,
+ 22,
+ 1,
+ 43
+ ],
+ [
+ 10,
+ 22,
+ 15,
+ 52,
+ 13,
+ 18
+ ],
+ [
+ 4,
+ 56,
+ 10,
+ 31,
+ 14,
+ 62
+ ],
+ [
+ 9,
+ 4,
+ 16,
+ 54,
+ 8,
+ 59
+ ],
+ [
+ 3,
+ 43,
+ 27,
+ 45,
+ 6,
+ 30
+ ],
+ [
+ 33,
+ 4,
+ 43,
+ 40,
+ 46,
+ 45
+ ]
+ ],
+ [
+ [
+ 48,
+ 42,
+ 38,
+ 63,
+ 47,
+ 7
+ ],
+ [
+ 3,
+ 62,
+ 10,
+ 6,
+ 26,
+ 2
+ ],
+ [
+ 39,
+ 44,
+ 6,
+ 7,
+ 45,
+ 40
+ ],
+ [
+ 56,
+ 33,
+ 2,
+ 41,
+ 48,
+ 16
+ ],
+ [
+ 6,
+ 18,
+ 37,
+ 27,
+ 41,
+ 30
+ ],
+ [
+ 59,
+ 45,
+ 23,
+ 31,
+ 53,
+ 9
+ ],
+ [
+ 0,
+ 13,
+ 12,
+ 44,
+ 33,
+ 16
+ ],
+ [
+ 12,
+ 56,
+ 50,
+ 60,
+ 28,
+ 63
+ ],
+ [
+ 14,
+ 33,
+ 26,
+ 24,
+ 32,
+ 16
+ ],
+ [
+ 43,
+ 2,
+ 0,
+ 50,
+ 28,
+ 22
+ ],
+ [
+ 61,
+ 63,
+ 9,
+ 8,
+ 17,
+ 39
+ ],
+ [
+ 60,
+ 19,
+ 59,
+ 23,
+ 24,
+ 30
+ ],
+ [
+ 56,
+ 59,
+ 53,
+ 32,
+ 7,
+ 38
+ ],
+ [
+ 28,
+ 25,
+ 22,
+ 24,
+ 59,
+ 42
+ ],
+ [
+ 22,
+ 49,
+ 46,
+ 23,
+ 60,
+ 7
+ ],
+ [
+ 46,
+ 34,
+ 27,
+ 29,
+ 0,
+ 12
+ ],
+ [
+ 17,
+ 22,
+ 5,
+ 31,
+ 20,
+ 27
+ ],
+ [
+ 39,
+ 48,
+ 57,
+ 37,
+ 17,
+ 36
+ ],
+ [
+ 42,
+ 49,
+ 32,
+ 7,
+ 16,
+ 61
+ ],
+ [
+ 1,
+ 7,
+ 23,
+ 14,
+ 36,
+ 39
+ ],
+ [
+ 21,
+ 7,
+ 48,
+ 14,
+ 60,
+ 35
+ ],
+ [
+ 8,
+ 18,
+ 24,
+ 60,
+ 1,
+ 0
+ ],
+ [
+ 51,
+ 15,
+ 33,
+ 36,
+ 5,
+ 30
+ ],
+ [
+ 52,
+ 4,
+ 8,
+ 29,
+ 9,
+ 26
+ ],
+ [
+ 4,
+ 9,
+ 16,
+ 40,
+ 58,
+ 60
+ ],
+ [
+ 3,
+ 17,
+ 21,
+ 61,
+ 19,
+ 8
+ ],
+ [
+ 35,
+ 54,
+ 1,
+ 53,
+ 60,
+ 0
+ ]
+ ],
+ [
+ [
+ 19,
+ 1,
+ 31,
+ 52,
+ 49,
+ 63
+ ],
+ [
+ 7,
+ 47,
+ 5,
+ 60,
+ 22,
+ 46
+ ],
+ [
+ 59,
+ 30,
+ 3,
+ 11,
+ 0,
+ 19
+ ],
+ [
+ 43,
+ 42,
+ 19,
+ 62,
+ 8,
+ 56
+ ],
+ [
+ 61,
+ 15,
+ 25,
+ 18,
+ 39,
+ 26
+ ],
+ [
+ 61,
+ 50,
+ 36,
+ 45,
+ 33,
+ 44
+ ],
+ [
+ 8,
+ 37,
+ 52,
+ 1,
+ 2,
+ 41
+ ],
+ [
+ 36,
+ 30,
+ 53,
+ 11,
+ 16,
+ 29
+ ],
+ [
+ 14,
+ 58,
+ 49,
+ 46,
+ 3,
+ 26
+ ],
+ [
+ 62,
+ 43,
+ 0,
+ 22,
+ 45,
+ 46
+ ],
+ [
+ 6,
+ 56,
+ 45,
+ 18,
+ 10,
+ 41
+ ],
+ [
+ 60,
+ 50,
+ 21,
+ 47,
+ 30,
+ 35
+ ],
+ [
+ 11,
+ 53,
+ 28,
+ 56,
+ 41,
+ 39
+ ],
+ [
+ 23,
+ 9,
+ 33,
+ 28,
+ 22,
+ 26
+ ],
+ [
+ 23,
+ 56,
+ 34,
+ 27,
+ 2,
+ 20
+ ],
+ [
+ 22,
+ 9,
+ 44,
+ 41,
+ 37,
+ 47
+ ],
+ [
+ 1,
+ 11,
+ 46,
+ 27,
+ 52,
+ 45
+ ],
+ [
+ 51,
+ 37,
+ 17,
+ 21,
+ 61,
+ 9
+ ],
+ [
+ 13,
+ 19,
+ 32,
+ 5,
+ 2,
+ 9
+ ],
+ [
+ 41,
+ 32,
+ 6,
+ 47,
+ 29,
+ 56
+ ],
+ [
+ 53,
+ 33,
+ 9,
+ 35,
+ 38,
+ 12
+ ],
+ [
+ 40,
+ 19,
+ 51,
+ 26,
+ 7,
+ 22
+ ],
+ [
+ 37,
+ 5,
+ 25,
+ 46,
+ 34,
+ 53
+ ],
+ [
+ 16,
+ 55,
+ 20,
+ 24,
+ 44,
+ 29
+ ],
+ [
+ 21,
+ 46,
+ 11,
+ 0,
+ 36,
+ 7
+ ],
+ [
+ 18,
+ 60,
+ 32,
+ 3,
+ 34,
+ 28
+ ],
+ [
+ 52,
+ 9,
+ 36,
+ 48,
+ 41,
+ 11
+ ]
+ ],
+ [
+ [
+ 53,
+ 15,
+ 34,
+ 0,
+ 36,
+ 56
+ ],
+ [
+ 8,
+ 12,
+ 41,
+ 11,
+ 19,
+ 55
+ ],
+ [
+ 56,
+ 13,
+ 31,
+ 36,
+ 23,
+ 47
+ ],
+ [
+ 36,
+ 51,
+ 30,
+ 7,
+ 26,
+ 54
+ ],
+ [
+ 58,
+ 13,
+ 50,
+ 2,
+ 53,
+ 34
+ ],
+ [
+ 49,
+ 52,
+ 23,
+ 32,
+ 7,
+ 26
+ ],
+ [
+ 61,
+ 38,
+ 23,
+ 0,
+ 4,
+ 28
+ ],
+ [
+ 42,
+ 27,
+ 17,
+ 9,
+ 18,
+ 20
+ ],
+ [
+ 1,
+ 34,
+ 45,
+ 4,
+ 12,
+ 7
+ ],
+ [
+ 54,
+ 27,
+ 57,
+ 38,
+ 44,
+ 6
+ ],
+ [
+ 59,
+ 40,
+ 1,
+ 60,
+ 48,
+ 30
+ ],
+ [
+ 25,
+ 31,
+ 32,
+ 51,
+ 62,
+ 8
+ ],
+ [
+ 62,
+ 19,
+ 14,
+ 2,
+ 37,
+ 15
+ ],
+ [
+ 40,
+ 57,
+ 37,
+ 35,
+ 22,
+ 61
+ ],
+ [
+ 45,
+ 16,
+ 34,
+ 42,
+ 37,
+ 48
+ ],
+ [
+ 50,
+ 16,
+ 62,
+ 33,
+ 25,
+ 37
+ ],
+ [
+ 42,
+ 13,
+ 39,
+ 47,
+ 3,
+ 63
+ ],
+ [
+ 46,
+ 23,
+ 28,
+ 27,
+ 4,
+ 15
+ ],
+ [
+ 56,
+ 62,
+ 31,
+ 35,
+ 59,
+ 45
+ ],
+ [
+ 15,
+ 38,
+ 13,
+ 4,
+ 63,
+ 48
+ ],
+ [
+ 34,
+ 15,
+ 57,
+ 38,
+ 13,
+ 24
+ ],
+ [
+ 62,
+ 36,
+ 41,
+ 54,
+ 46,
+ 29
+ ],
+ [
+ 22,
+ 53,
+ 46,
+ 34,
+ 30,
+ 27
+ ],
+ [
+ 24,
+ 10,
+ 4,
+ 47,
+ 18,
+ 36
+ ],
+ [
+ 50,
+ 57,
+ 51,
+ 11,
+ 49,
+ 3
+ ],
+ [
+ 51,
+ 7,
+ 0,
+ 18,
+ 11,
+ 44
+ ],
+ [
+ 39,
+ 37,
+ 9,
+ 42,
+ 40,
+ 44
+ ]
+ ],
+ [
+ [
+ 36,
+ 25,
+ 57,
+ 55,
+ 47,
+ 63
+ ],
+ [
+ 0,
+ 2,
+ 46,
+ 3,
+ 51,
+ 34
+ ],
+ [
+ 24,
+ 2,
+ 46,
+ 15,
+ 33,
+ 43
+ ],
+ [
+ 22,
+ 31,
+ 17,
+ 19,
+ 10,
+ 55
+ ],
+ [
+ 58,
+ 59,
+ 3,
+ 9,
+ 40,
+ 57
+ ],
+ [
+ 23,
+ 31,
+ 43,
+ 57,
+ 2,
+ 38
+ ],
+ [
+ 9,
+ 62,
+ 13,
+ 42,
+ 47,
+ 52
+ ],
+ [
+ 24,
+ 7,
+ 14,
+ 10,
+ 46,
+ 59
+ ],
+ [
+ 47,
+ 18,
+ 4,
+ 37,
+ 0,
+ 13
+ ],
+ [
+ 54,
+ 4,
+ 25,
+ 47,
+ 36,
+ 38
+ ],
+ [
+ 15,
+ 47,
+ 1,
+ 24,
+ 58,
+ 14
+ ],
+ [
+ 54,
+ 5,
+ 16,
+ 63,
+ 14,
+ 7
+ ],
+ [
+ 49,
+ 3,
+ 33,
+ 13,
+ 46,
+ 10
+ ],
+ [
+ 18,
+ 10,
+ 11,
+ 13,
+ 63,
+ 39
+ ],
+ [
+ 19,
+ 62,
+ 32,
+ 58,
+ 10,
+ 43
+ ],
+ [
+ 43,
+ 48,
+ 63,
+ 5,
+ 55,
+ 53
+ ],
+ [
+ 24,
+ 51,
+ 47,
+ 15,
+ 59,
+ 32
+ ],
+ [
+ 44,
+ 0,
+ 34,
+ 43,
+ 3,
+ 6
+ ],
+ [
+ 58,
+ 38,
+ 54,
+ 47,
+ 11,
+ 59
+ ],
+ [
+ 55,
+ 27,
+ 15,
+ 9,
+ 42,
+ 31
+ ],
+ [
+ 43,
+ 41,
+ 1,
+ 51,
+ 5,
+ 29
+ ],
+ [
+ 49,
+ 27,
+ 20,
+ 6,
+ 4,
+ 13
+ ],
+ [
+ 11,
+ 25,
+ 2,
+ 27,
+ 54,
+ 62
+ ],
+ [
+ 38,
+ 44,
+ 40,
+ 54,
+ 33,
+ 14
+ ],
+ [
+ 13,
+ 63,
+ 52,
+ 2,
+ 8,
+ 29
+ ],
+ [
+ 23,
+ 41,
+ 59,
+ 57,
+ 38,
+ 15
+ ],
+ [
+ 6,
+ 23,
+ 62,
+ 50,
+ 34,
+ 51
+ ]
+ ],
+ [
+ [
+ 41,
+ 2,
+ 42,
+ 16,
+ 50,
+ 23
+ ],
+ [
+ 51,
+ 41,
+ 5,
+ 15,
+ 40,
+ 21
+ ],
+ [
+ 43,
+ 1,
+ 29,
+ 55,
+ 21,
+ 35
+ ],
+ [
+ 24,
+ 53,
+ 25,
+ 51,
+ 32,
+ 29
+ ],
+ [
+ 41,
+ 31,
+ 49,
+ 57,
+ 60,
+ 34
+ ],
+ [
+ 17,
+ 4,
+ 35,
+ 30,
+ 10,
+ 38
+ ],
+ [
+ 34,
+ 7,
+ 21,
+ 9,
+ 48,
+ 31
+ ],
+ [
+ 14,
+ 24,
+ 7,
+ 46,
+ 25,
+ 27
+ ],
+ [
+ 47,
+ 0,
+ 12,
+ 6,
+ 37,
+ 60
+ ],
+ [
+ 4,
+ 30,
+ 25,
+ 47,
+ 36,
+ 54
+ ],
+ [
+ 24,
+ 61,
+ 15,
+ 47,
+ 46,
+ 1
+ ],
+ [
+ 5,
+ 14,
+ 24,
+ 16,
+ 57,
+ 63
+ ],
+ [
+ 49,
+ 3,
+ 17,
+ 26,
+ 36,
+ 44
+ ],
+ [
+ 13,
+ 10,
+ 61,
+ 0,
+ 11,
+ 22
+ ],
+ [
+ 10,
+ 6,
+ 2,
+ 49,
+ 58,
+ 46
+ ],
+ [
+ 19,
+ 11,
+ 2,
+ 25,
+ 54,
+ 18
+ ],
+ [
+ 10,
+ 5,
+ 52,
+ 24,
+ 18,
+ 17
+ ],
+ [
+ 44,
+ 13,
+ 55,
+ 31,
+ 63,
+ 38
+ ],
+ [
+ 58,
+ 7,
+ 25,
+ 32,
+ 38,
+ 14
+ ],
+ [
+ 49,
+ 55,
+ 34,
+ 13,
+ 16,
+ 40
+ ],
+ [
+ 51,
+ 23,
+ 59,
+ 35,
+ 5,
+ 4
+ ],
+ [
+ 57,
+ 15,
+ 46,
+ 27,
+ 42,
+ 32
+ ],
+ [
+ 33,
+ 15,
+ 23,
+ 52,
+ 24,
+ 27
+ ],
+ [
+ 4,
+ 41,
+ 33,
+ 10,
+ 26,
+ 40
+ ],
+ [
+ 8,
+ 13,
+ 59,
+ 4,
+ 9,
+ 54
+ ],
+ [
+ 19,
+ 3,
+ 31,
+ 27,
+ 43,
+ 2
+ ],
+ [
+ 46,
+ 61,
+ 25,
+ 8,
+ 29,
+ 50
+ ]
+ ],
+ [
+ [
+ 48,
+ 38,
+ 62,
+ 63,
+ 42,
+ 50
+ ],
+ [
+ 3,
+ 2,
+ 26,
+ 17,
+ 10,
+ 6
+ ],
+ [
+ 39,
+ 44,
+ 6,
+ 45,
+ 7,
+ 40
+ ],
+ [
+ 5,
+ 62,
+ 53,
+ 50,
+ 41,
+ 48
+ ],
+ [
+ 6,
+ 41,
+ 49,
+ 37,
+ 30,
+ 23
+ ],
+ [
+ 59,
+ 60,
+ 4,
+ 46,
+ 53,
+ 29
+ ],
+ [
+ 44,
+ 34,
+ 7,
+ 15,
+ 13,
+ 43
+ ],
+ [
+ 12,
+ 14,
+ 24,
+ 25,
+ 58,
+ 7
+ ],
+ [
+ 33,
+ 35,
+ 4,
+ 37,
+ 8,
+ 36
+ ],
+ [
+ 40,
+ 30,
+ 25,
+ 36,
+ 20,
+ 54
+ ],
+ [
+ 9,
+ 8,
+ 24,
+ 25,
+ 5,
+ 63
+ ],
+ [
+ 59,
+ 34,
+ 5,
+ 24,
+ 57,
+ 6
+ ],
+ [
+ 17,
+ 49,
+ 44,
+ 26,
+ 55,
+ 7
+ ],
+ [
+ 13,
+ 47,
+ 59,
+ 27,
+ 22,
+ 61
+ ],
+ [
+ 49,
+ 2,
+ 44,
+ 10,
+ 46,
+ 40
+ ],
+ [
+ 34,
+ 2,
+ 54,
+ 57,
+ 53,
+ 55
+ ],
+ [
+ 5,
+ 17,
+ 29,
+ 31,
+ 43,
+ 52
+ ],
+ [
+ 48,
+ 57,
+ 38,
+ 63,
+ 39,
+ 43
+ ],
+ [
+ 42,
+ 38,
+ 49,
+ 32,
+ 7,
+ 40
+ ],
+ [
+ 59,
+ 1,
+ 16,
+ 23,
+ 60,
+ 10
+ ],
+ [
+ 21,
+ 7,
+ 35,
+ 53,
+ 48,
+ 31
+ ],
+ [
+ 8,
+ 60,
+ 24,
+ 42,
+ 14,
+ 35
+ ],
+ [
+ 33,
+ 51,
+ 15,
+ 28,
+ 23,
+ 5
+ ],
+ [
+ 4,
+ 41,
+ 9,
+ 11,
+ 8,
+ 51
+ ],
+ [
+ 60,
+ 24,
+ 19,
+ 48,
+ 9,
+ 4
+ ],
+ [
+ 19,
+ 3,
+ 26,
+ 58,
+ 12,
+ 61
+ ],
+ [
+ 35,
+ 54,
+ 1,
+ 60,
+ 53,
+ 49
+ ]
+ ],
+ [
+ [
+ 21,
+ 7,
+ 53,
+ 56,
+ 63,
+ 33
+ ],
+ [
+ 3,
+ 34,
+ 57,
+ 16,
+ 20,
+ 51
+ ],
+ [
+ 55,
+ 11,
+ 16,
+ 60,
+ 0,
+ 13
+ ],
+ [
+ 62,
+ 43,
+ 5,
+ 50,
+ 8,
+ 42
+ ],
+ [
+ 35,
+ 52,
+ 9,
+ 43,
+ 0,
+ 27
+ ],
+ [
+ 16,
+ 60,
+ 29,
+ 61,
+ 28,
+ 58
+ ],
+ [
+ 31,
+ 34,
+ 43,
+ 30,
+ 2,
+ 18
+ ],
+ [
+ 36,
+ 25,
+ 53,
+ 24,
+ 35,
+ 48
+ ],
+ [
+ 48,
+ 35,
+ 4,
+ 32,
+ 8,
+ 60
+ ],
+ [
+ 25,
+ 54,
+ 30,
+ 10,
+ 4,
+ 41
+ ],
+ [
+ 9,
+ 3,
+ 19,
+ 20,
+ 61,
+ 24
+ ],
+ [
+ 30,
+ 5,
+ 17,
+ 45,
+ 18,
+ 49
+ ],
+ [
+ 49,
+ 21,
+ 17,
+ 26,
+ 42,
+ 37
+ ],
+ [
+ 38,
+ 25,
+ 10,
+ 51,
+ 54,
+ 13
+ ],
+ [
+ 2,
+ 27,
+ 26,
+ 10,
+ 58,
+ 30
+ ],
+ [
+ 9,
+ 2,
+ 54,
+ 21,
+ 25,
+ 13
+ ],
+ [
+ 22,
+ 23,
+ 33,
+ 27,
+ 51,
+ 52
+ ],
+ [
+ 30,
+ 21,
+ 35,
+ 55,
+ 5,
+ 17
+ ],
+ [
+ 38,
+ 27,
+ 24,
+ 56,
+ 21,
+ 35
+ ],
+ [
+ 53,
+ 45,
+ 22,
+ 28,
+ 0,
+ 32
+ ],
+ [
+ 40,
+ 42,
+ 48,
+ 37,
+ 8,
+ 52
+ ],
+ [
+ 46,
+ 45,
+ 57,
+ 36,
+ 51,
+ 32
+ ],
+ [
+ 62,
+ 10,
+ 38,
+ 42,
+ 41,
+ 54
+ ],
+ [
+ 60,
+ 1,
+ 10,
+ 31,
+ 44,
+ 36
+ ],
+ [
+ 29,
+ 6,
+ 9,
+ 8,
+ 56,
+ 7
+ ],
+ [
+ 9,
+ 59,
+ 22,
+ 0,
+ 35,
+ 57
+ ],
+ [
+ 45,
+ 27,
+ 62,
+ 47,
+ 3,
+ 28
+ ]
+ ],
+ [
+ [
+ 27,
+ 13,
+ 18,
+ 8,
+ 63,
+ 55
+ ],
+ [
+ 36,
+ 21,
+ 57,
+ 8,
+ 46,
+ 55
+ ],
+ [
+ 43,
+ 61,
+ 10,
+ 13,
+ 41,
+ 37
+ ],
+ [
+ 43,
+ 16,
+ 24,
+ 6,
+ 26,
+ 61
+ ],
+ [
+ 60,
+ 29,
+ 35,
+ 31,
+ 16,
+ 23
+ ],
+ [
+ 9,
+ 58,
+ 60,
+ 0,
+ 17,
+ 38
+ ],
+ [
+ 63,
+ 16,
+ 7,
+ 13,
+ 31,
+ 18
+ ],
+ [
+ 16,
+ 25,
+ 24,
+ 2,
+ 47,
+ 58
+ ],
+ [
+ 8,
+ 35,
+ 6,
+ 14,
+ 48,
+ 4
+ ],
+ [
+ 30,
+ 25,
+ 54,
+ 4,
+ 10,
+ 34
+ ],
+ [
+ 20,
+ 23,
+ 19,
+ 7,
+ 38,
+ 61
+ ],
+ [
+ 5,
+ 3,
+ 24,
+ 32,
+ 12,
+ 42
+ ],
+ [
+ 49,
+ 17,
+ 10,
+ 26,
+ 32,
+ 60
+ ],
+ [
+ 10,
+ 13,
+ 29,
+ 22,
+ 58,
+ 54
+ ],
+ [
+ 44,
+ 8,
+ 58,
+ 2,
+ 6,
+ 5
+ ],
+ [
+ 25,
+ 6,
+ 2,
+ 54,
+ 19,
+ 53
+ ],
+ [
+ 61,
+ 5,
+ 30,
+ 1,
+ 17,
+ 27
+ ],
+ [
+ 22,
+ 55,
+ 63,
+ 57,
+ 33,
+ 19
+ ],
+ [
+ 35,
+ 46,
+ 6,
+ 32,
+ 14,
+ 7
+ ],
+ [
+ 45,
+ 13,
+ 51,
+ 14,
+ 7,
+ 5
+ ],
+ [
+ 15,
+ 23,
+ 50,
+ 51,
+ 13,
+ 59
+ ],
+ [
+ 57,
+ 36,
+ 53,
+ 61,
+ 6,
+ 14
+ ],
+ [
+ 10,
+ 23,
+ 15,
+ 33,
+ 27,
+ 38
+ ],
+ [
+ 4,
+ 10,
+ 33,
+ 41,
+ 26,
+ 36
+ ],
+ [
+ 8,
+ 4,
+ 33,
+ 9,
+ 47,
+ 48
+ ],
+ [
+ 3,
+ 43,
+ 6,
+ 19,
+ 2,
+ 30
+ ],
+ [
+ 55,
+ 4,
+ 46,
+ 40,
+ 18,
+ 20
+ ]
+ ],
+ [
+ [
+ 48,
+ 38,
+ 63,
+ 37,
+ 7,
+ 42
+ ],
+ [
+ 3,
+ 26,
+ 10,
+ 35,
+ 6,
+ 2
+ ],
+ [
+ 39,
+ 44,
+ 7,
+ 45,
+ 6,
+ 40
+ ],
+ [
+ 56,
+ 23,
+ 61,
+ 16,
+ 33,
+ 41
+ ],
+ [
+ 6,
+ 37,
+ 44,
+ 16,
+ 30,
+ 19
+ ],
+ [
+ 59,
+ 17,
+ 60,
+ 19,
+ 5,
+ 52
+ ],
+ [
+ 44,
+ 55,
+ 33,
+ 63,
+ 13,
+ 31
+ ],
+ [
+ 12,
+ 25,
+ 63,
+ 2,
+ 24,
+ 54
+ ],
+ [
+ 33,
+ 8,
+ 42,
+ 4,
+ 35,
+ 41
+ ],
+ [
+ 30,
+ 40,
+ 25,
+ 21,
+ 48,
+ 51
+ ],
+ [
+ 8,
+ 23,
+ 34,
+ 5,
+ 19,
+ 22
+ ],
+ [
+ 34,
+ 59,
+ 42,
+ 11,
+ 5,
+ 3
+ ],
+ [
+ 10,
+ 26,
+ 50,
+ 55,
+ 53,
+ 59
+ ],
+ [
+ 59,
+ 54,
+ 13,
+ 10,
+ 47,
+ 25
+ ],
+ [
+ 24,
+ 2,
+ 49,
+ 46,
+ 38,
+ 8
+ ],
+ [
+ 34,
+ 2,
+ 53,
+ 54,
+ 40,
+ 57
+ ],
+ [
+ 17,
+ 29,
+ 5,
+ 27,
+ 18,
+ 43
+ ],
+ [
+ 48,
+ 57,
+ 63,
+ 39,
+ 38,
+ 32
+ ],
+ [
+ 42,
+ 49,
+ 7,
+ 32,
+ 38,
+ 61
+ ],
+ [
+ 1,
+ 59,
+ 23,
+ 14,
+ 16,
+ 10
+ ],
+ [
+ 21,
+ 7,
+ 35,
+ 53,
+ 13,
+ 11
+ ],
+ [
+ 8,
+ 60,
+ 24,
+ 25,
+ 42,
+ 55
+ ],
+ [
+ 33,
+ 15,
+ 5,
+ 51,
+ 28,
+ 44
+ ],
+ [
+ 4,
+ 9,
+ 51,
+ 8,
+ 41,
+ 39
+ ],
+ [
+ 60,
+ 4,
+ 19,
+ 48,
+ 9,
+ 24
+ ],
+ [
+ 19,
+ 3,
+ 26,
+ 12,
+ 53,
+ 21
+ ],
+ [
+ 53,
+ 35,
+ 1,
+ 54,
+ 60,
+ 20
+ ]
+ ],
+ [
+ [
+ 17,
+ 37,
+ 31,
+ 32,
+ 63,
+ 50
+ ],
+ [
+ 12,
+ 2,
+ 32,
+ 9,
+ 47,
+ 17
+ ],
+ [
+ 3,
+ 57,
+ 56,
+ 50,
+ 33,
+ 38
+ ],
+ [
+ 43,
+ 42,
+ 19,
+ 52,
+ 8,
+ 17
+ ],
+ [
+ 61,
+ 39,
+ 27,
+ 12,
+ 15,
+ 57
+ ],
+ [
+ 6,
+ 33,
+ 36,
+ 44,
+ 29,
+ 61
+ ],
+ [
+ 2,
+ 41,
+ 42,
+ 15,
+ 52,
+ 5
+ ],
+ [
+ 36,
+ 37,
+ 15,
+ 53,
+ 18,
+ 62
+ ],
+ [
+ 50,
+ 58,
+ 3,
+ 5,
+ 16,
+ 4
+ ],
+ [
+ 16,
+ 22,
+ 55,
+ 41,
+ 32,
+ 26
+ ],
+ [
+ 3,
+ 41,
+ 56,
+ 45,
+ 38,
+ 34
+ ],
+ [
+ 21,
+ 53,
+ 50,
+ 34,
+ 38,
+ 35
+ ],
+ [
+ 35,
+ 11,
+ 37,
+ 16,
+ 53,
+ 42
+ ],
+ [
+ 9,
+ 15,
+ 54,
+ 38,
+ 12,
+ 52
+ ],
+ [
+ 56,
+ 2,
+ 24,
+ 23,
+ 9,
+ 5
+ ],
+ [
+ 2,
+ 44,
+ 9,
+ 53,
+ 35,
+ 54
+ ],
+ [
+ 27,
+ 46,
+ 1,
+ 5,
+ 60,
+ 50
+ ],
+ [
+ 51,
+ 61,
+ 50,
+ 54,
+ 33,
+ 44
+ ],
+ [
+ 2,
+ 38,
+ 9,
+ 19,
+ 5,
+ 32
+ ],
+ [
+ 32,
+ 28,
+ 6,
+ 15,
+ 0,
+ 33
+ ],
+ [
+ 26,
+ 53,
+ 27,
+ 13,
+ 5,
+ 4
+ ],
+ [
+ 34,
+ 40,
+ 45,
+ 55,
+ 62,
+ 53
+ ],
+ [
+ 41,
+ 5,
+ 46,
+ 4,
+ 45,
+ 3
+ ],
+ [
+ 20,
+ 4,
+ 22,
+ 47,
+ 59,
+ 58
+ ],
+ [
+ 62,
+ 36,
+ 25,
+ 28,
+ 53,
+ 33
+ ],
+ [
+ 22,
+ 33,
+ 7,
+ 3,
+ 54,
+ 0
+ ],
+ [
+ 58,
+ 27,
+ 52,
+ 48,
+ 45,
+ 17
+ ]
+ ],
+ [
+ [
+ 3,
+ 43,
+ 17,
+ 42,
+ 4,
+ 35
+ ],
+ [
+ 0,
+ 42,
+ 36,
+ 34,
+ 32,
+ 24
+ ],
+ [
+ 43,
+ 12,
+ 19,
+ 20,
+ 2,
+ 4
+ ],
+ [
+ 4,
+ 28,
+ 54,
+ 22,
+ 58,
+ 23
+ ],
+ [
+ 12,
+ 60,
+ 52,
+ 57,
+ 7,
+ 46
+ ],
+ [
+ 43,
+ 2,
+ 1,
+ 63,
+ 20,
+ 11
+ ],
+ [
+ 19,
+ 42,
+ 57,
+ 31,
+ 15,
+ 21
+ ],
+ [
+ 35,
+ 6,
+ 46,
+ 25,
+ 36,
+ 33
+ ],
+ [
+ 11,
+ 4,
+ 35,
+ 50,
+ 32,
+ 52
+ ],
+ [
+ 21,
+ 25,
+ 38,
+ 47,
+ 31,
+ 58
+ ],
+ [
+ 3,
+ 2,
+ 22,
+ 45,
+ 42,
+ 15
+ ],
+ [
+ 2,
+ 21,
+ 5,
+ 7,
+ 43,
+ 38
+ ],
+ [
+ 33,
+ 10,
+ 41,
+ 42,
+ 60,
+ 16
+ ],
+ [
+ 44,
+ 56,
+ 30,
+ 54,
+ 9,
+ 43
+ ],
+ [
+ 9,
+ 5,
+ 28,
+ 33,
+ 58,
+ 49
+ ],
+ [
+ 8,
+ 53,
+ 59,
+ 54,
+ 31,
+ 2
+ ],
+ [
+ 47,
+ 0,
+ 48,
+ 27,
+ 24,
+ 51
+ ],
+ [
+ 0,
+ 60,
+ 21,
+ 12,
+ 56,
+ 28
+ ],
+ [
+ 38,
+ 13,
+ 53,
+ 51,
+ 9,
+ 7
+ ],
+ [
+ 19,
+ 28,
+ 56,
+ 17,
+ 21,
+ 26
+ ],
+ [
+ 12,
+ 43,
+ 26,
+ 0,
+ 4,
+ 5
+ ],
+ [
+ 43,
+ 1,
+ 12,
+ 55,
+ 4,
+ 21
+ ],
+ [
+ 41,
+ 5,
+ 21,
+ 45,
+ 25,
+ 12
+ ],
+ [
+ 4,
+ 20,
+ 1,
+ 22,
+ 30,
+ 42
+ ],
+ [
+ 5,
+ 62,
+ 25,
+ 54,
+ 48,
+ 47
+ ],
+ [
+ 10,
+ 28,
+ 55,
+ 23,
+ 0,
+ 53
+ ],
+ [
+ 30,
+ 57,
+ 8,
+ 29,
+ 17,
+ 5
+ ]
+ ],
+ [
+ [
+ 11,
+ 16,
+ 31,
+ 0,
+ 35,
+ 46
+ ],
+ [
+ 13,
+ 49,
+ 31,
+ 34,
+ 16,
+ 50
+ ],
+ [
+ 36,
+ 13,
+ 53,
+ 4,
+ 27,
+ 18
+ ],
+ [
+ 24,
+ 7,
+ 29,
+ 26,
+ 12,
+ 32
+ ],
+ [
+ 17,
+ 35,
+ 2,
+ 44,
+ 10,
+ 48
+ ],
+ [
+ 8,
+ 7,
+ 23,
+ 5,
+ 51,
+ 26
+ ],
+ [
+ 58,
+ 15,
+ 61,
+ 29,
+ 38,
+ 62
+ ],
+ [
+ 20,
+ 9,
+ 42,
+ 35,
+ 3,
+ 54
+ ],
+ [
+ 4,
+ 47,
+ 25,
+ 11,
+ 1,
+ 52
+ ],
+ [
+ 54,
+ 25,
+ 55,
+ 38,
+ 21,
+ 27
+ ],
+ [
+ 1,
+ 60,
+ 14,
+ 59,
+ 22,
+ 30
+ ],
+ [
+ 51,
+ 31,
+ 5,
+ 25,
+ 14,
+ 52
+ ],
+ [
+ 34,
+ 2,
+ 10,
+ 26,
+ 52,
+ 47
+ ],
+ [
+ 40,
+ 57,
+ 13,
+ 54,
+ 9,
+ 6
+ ],
+ [
+ 8,
+ 44,
+ 58,
+ 5,
+ 16,
+ 1
+ ],
+ [
+ 8,
+ 53,
+ 59,
+ 25,
+ 52,
+ 24
+ ],
+ [
+ 13,
+ 47,
+ 45,
+ 0,
+ 42,
+ 37
+ ],
+ [
+ 23,
+ 44,
+ 55,
+ 7,
+ 38,
+ 33
+ ],
+ [
+ 53,
+ 38,
+ 11,
+ 1,
+ 8,
+ 24
+ ],
+ [
+ 15,
+ 38,
+ 14,
+ 28,
+ 19,
+ 0
+ ],
+ [
+ 41,
+ 27,
+ 8,
+ 42,
+ 57,
+ 40
+ ],
+ [
+ 11,
+ 62,
+ 55,
+ 16,
+ 41,
+ 10
+ ],
+ [
+ 31,
+ 20,
+ 46,
+ 34,
+ 37,
+ 21
+ ],
+ [
+ 47,
+ 4,
+ 55,
+ 33,
+ 49,
+ 34
+ ],
+ [
+ 11,
+ 45,
+ 47,
+ 48,
+ 20,
+ 55
+ ],
+ [
+ 50,
+ 23,
+ 18,
+ 10,
+ 3,
+ 2
+ ],
+ [
+ 17,
+ 31,
+ 36,
+ 5,
+ 4,
+ 20
+ ]
+ ],
+ [
+ [
+ 22,
+ 6,
+ 39,
+ 57,
+ 29,
+ 28
+ ],
+ [
+ 27,
+ 6,
+ 14,
+ 17,
+ 51,
+ 55
+ ],
+ [
+ 1,
+ 11,
+ 29,
+ 26,
+ 47,
+ 4
+ ],
+ [
+ 14,
+ 38,
+ 31,
+ 22,
+ 29,
+ 6
+ ],
+ [
+ 14,
+ 59,
+ 61,
+ 16,
+ 1,
+ 19
+ ],
+ [
+ 30,
+ 8,
+ 23,
+ 21,
+ 47,
+ 1
+ ],
+ [
+ 58,
+ 4,
+ 15,
+ 61,
+ 27,
+ 31
+ ],
+ [
+ 20,
+ 42,
+ 3,
+ 9,
+ 35,
+ 6
+ ],
+ [
+ 47,
+ 4,
+ 25,
+ 8,
+ 36,
+ 0
+ ],
+ [
+ 54,
+ 55,
+ 21,
+ 19,
+ 33,
+ 25
+ ],
+ [
+ 14,
+ 4,
+ 60,
+ 20,
+ 24,
+ 40
+ ],
+ [
+ 51,
+ 0,
+ 5,
+ 32,
+ 52,
+ 3
+ ],
+ [
+ 2,
+ 36,
+ 10,
+ 52,
+ 26,
+ 32
+ ],
+ [
+ 40,
+ 13,
+ 54,
+ 36,
+ 57,
+ 46
+ ],
+ [
+ 44,
+ 8,
+ 5,
+ 37,
+ 58,
+ 2
+ ],
+ [
+ 6,
+ 53,
+ 24,
+ 2,
+ 54,
+ 37
+ ],
+ [
+ 13,
+ 47,
+ 61,
+ 5,
+ 19,
+ 17
+ ],
+ [
+ 58,
+ 55,
+ 44,
+ 38,
+ 63,
+ 6
+ ],
+ [
+ 35,
+ 46,
+ 31,
+ 1,
+ 32,
+ 19
+ ],
+ [
+ 15,
+ 13,
+ 45,
+ 63,
+ 9,
+ 55
+ ],
+ [
+ 27,
+ 15,
+ 23,
+ 6,
+ 35,
+ 63
+ ],
+ [
+ 36,
+ 62,
+ 57,
+ 41,
+ 10,
+ 16
+ ],
+ [
+ 10,
+ 33,
+ 41,
+ 20,
+ 5,
+ 23
+ ],
+ [
+ 4,
+ 47,
+ 26,
+ 55,
+ 10,
+ 49
+ ],
+ [
+ 4,
+ 11,
+ 8,
+ 48,
+ 36,
+ 9
+ ],
+ [
+ 18,
+ 51,
+ 43,
+ 33,
+ 50,
+ 3
+ ],
+ [
+ 4,
+ 55,
+ 9,
+ 36,
+ 5,
+ 43
+ ]
+ ],
+ [
+ [
+ 48,
+ 38,
+ 47,
+ 63,
+ 7,
+ 42
+ ],
+ [
+ 3,
+ 26,
+ 10,
+ 6,
+ 2,
+ 42
+ ],
+ [
+ 39,
+ 44,
+ 6,
+ 7,
+ 45,
+ 8
+ ],
+ [
+ 60,
+ 31,
+ 22,
+ 0,
+ 54,
+ 45
+ ],
+ [
+ 6,
+ 59,
+ 14,
+ 16,
+ 37,
+ 19
+ ],
+ [
+ 59,
+ 30,
+ 8,
+ 47,
+ 60,
+ 17
+ ],
+ [
+ 44,
+ 56,
+ 4,
+ 13,
+ 15,
+ 9
+ ],
+ [
+ 12,
+ 24,
+ 20,
+ 58,
+ 28,
+ 61
+ ],
+ [
+ 47,
+ 33,
+ 8,
+ 4,
+ 36,
+ 42
+ ],
+ [
+ 40,
+ 54,
+ 55,
+ 33,
+ 21,
+ 51
+ ],
+ [
+ 8,
+ 14,
+ 22,
+ 39,
+ 31,
+ 7
+ ],
+ [
+ 34,
+ 59,
+ 14,
+ 5,
+ 52,
+ 0
+ ],
+ [
+ 36,
+ 10,
+ 52,
+ 26,
+ 44,
+ 59
+ ],
+ [
+ 59,
+ 44,
+ 13,
+ 1,
+ 36,
+ 22
+ ],
+ [
+ 24,
+ 5,
+ 46,
+ 2,
+ 44,
+ 38
+ ],
+ [
+ 53,
+ 34,
+ 6,
+ 40,
+ 30,
+ 2
+ ],
+ [
+ 17,
+ 29,
+ 50,
+ 47,
+ 5,
+ 27
+ ],
+ [
+ 48,
+ 38,
+ 57,
+ 63,
+ 32,
+ 60
+ ],
+ [
+ 42,
+ 7,
+ 49,
+ 46,
+ 32,
+ 1
+ ],
+ [
+ 59,
+ 1,
+ 23,
+ 14,
+ 10,
+ 16
+ ],
+ [
+ 21,
+ 35,
+ 7,
+ 53,
+ 6,
+ 17
+ ],
+ [
+ 8,
+ 24,
+ 60,
+ 13,
+ 55,
+ 14
+ ],
+ [
+ 33,
+ 5,
+ 51,
+ 15,
+ 3,
+ 23
+ ],
+ [
+ 4,
+ 9,
+ 51,
+ 26,
+ 41,
+ 28
+ ],
+ [
+ 60,
+ 4,
+ 19,
+ 47,
+ 40,
+ 48
+ ],
+ [
+ 19,
+ 3,
+ 26,
+ 21,
+ 53,
+ 8
+ ],
+ [
+ 53,
+ 1,
+ 35,
+ 54,
+ 60,
+ 20
+ ]
+ ],
+ [
+ [
+ 37,
+ 46,
+ 39,
+ 54,
+ 27,
+ 55
+ ],
+ [
+ 34,
+ 5,
+ 16,
+ 47,
+ 6,
+ 42
+ ],
+ [
+ 32,
+ 38,
+ 16,
+ 42,
+ 3,
+ 20
+ ],
+ [
+ 43,
+ 19,
+ 60,
+ 42,
+ 52,
+ 11
+ ],
+ [
+ 7,
+ 61,
+ 39,
+ 57,
+ 12,
+ 46
+ ],
+ [
+ 50,
+ 36,
+ 44,
+ 24,
+ 28,
+ 31
+ ],
+ [
+ 15,
+ 42,
+ 41,
+ 2,
+ 40,
+ 32
+ ],
+ [
+ 36,
+ 35,
+ 38,
+ 53,
+ 58,
+ 51
+ ],
+ [
+ 50,
+ 4,
+ 16,
+ 3,
+ 36,
+ 58
+ ],
+ [
+ 59,
+ 16,
+ 32,
+ 38,
+ 26,
+ 56
+ ],
+ [
+ 3,
+ 45,
+ 41,
+ 37,
+ 21,
+ 33
+ ],
+ [
+ 21,
+ 53,
+ 34,
+ 35,
+ 5,
+ 39
+ ],
+ [
+ 35,
+ 42,
+ 63,
+ 16,
+ 11,
+ 46
+ ],
+ [
+ 23,
+ 9,
+ 51,
+ 54,
+ 15,
+ 38
+ ],
+ [
+ 2,
+ 5,
+ 20,
+ 43,
+ 24,
+ 29
+ ],
+ [
+ 53,
+ 44,
+ 9,
+ 2,
+ 54,
+ 20
+ ],
+ [
+ 27,
+ 46,
+ 1,
+ 47,
+ 50,
+ 5
+ ],
+ [
+ 61,
+ 54,
+ 57,
+ 44,
+ 51,
+ 43
+ ],
+ [
+ 19,
+ 9,
+ 33,
+ 38,
+ 61,
+ 37
+ ],
+ [
+ 0,
+ 33,
+ 15,
+ 32,
+ 6,
+ 9
+ ],
+ [
+ 26,
+ 27,
+ 53,
+ 5,
+ 47,
+ 54
+ ],
+ [
+ 55,
+ 34,
+ 12,
+ 62,
+ 3,
+ 4
+ ],
+ [
+ 41,
+ 5,
+ 46,
+ 40,
+ 4,
+ 32
+ ],
+ [
+ 55,
+ 20,
+ 26,
+ 44,
+ 4,
+ 40
+ ],
+ [
+ 62,
+ 28,
+ 12,
+ 34,
+ 23,
+ 33
+ ],
+ [
+ 18,
+ 7,
+ 22,
+ 3,
+ 54,
+ 14
+ ],
+ [
+ 36,
+ 9,
+ 27,
+ 52,
+ 48,
+ 11
+ ]
+ ],
+ [
+ [
+ 46,
+ 37,
+ 61,
+ 18,
+ 36,
+ 63
+ ],
+ [
+ 22,
+ 34,
+ 28,
+ 59,
+ 24,
+ 51
+ ],
+ [
+ 32,
+ 15,
+ 17,
+ 60,
+ 38,
+ 20
+ ],
+ [
+ 28,
+ 4,
+ 58,
+ 16,
+ 30,
+ 35
+ ],
+ [
+ 7,
+ 36,
+ 9,
+ 57,
+ 33,
+ 23
+ ],
+ [
+ 43,
+ 63,
+ 2,
+ 30,
+ 11,
+ 19
+ ],
+ [
+ 19,
+ 42,
+ 57,
+ 15,
+ 3,
+ 22
+ ],
+ [
+ 35,
+ 46,
+ 47,
+ 6,
+ 58,
+ 8
+ ],
+ [
+ 32,
+ 4,
+ 37,
+ 36,
+ 35,
+ 57
+ ],
+ [
+ 21,
+ 47,
+ 55,
+ 58,
+ 38,
+ 54
+ ],
+ [
+ 3,
+ 42,
+ 2,
+ 38,
+ 5,
+ 33
+ ],
+ [
+ 43,
+ 18,
+ 2,
+ 21,
+ 5,
+ 49
+ ],
+ [
+ 10,
+ 49,
+ 24,
+ 32,
+ 25,
+ 42
+ ],
+ [
+ 30,
+ 21,
+ 63,
+ 51,
+ 49,
+ 16
+ ],
+ [
+ 62,
+ 2,
+ 13,
+ 5,
+ 29,
+ 40
+ ],
+ [
+ 53,
+ 2,
+ 9,
+ 63,
+ 54,
+ 46
+ ],
+ [
+ 27,
+ 0,
+ 63,
+ 47,
+ 5,
+ 14
+ ],
+ [
+ 60,
+ 57,
+ 43,
+ 44,
+ 0,
+ 50
+ ],
+ [
+ 38,
+ 9,
+ 61,
+ 33,
+ 60,
+ 13
+ ],
+ [
+ 25,
+ 0,
+ 15,
+ 34,
+ 35,
+ 11
+ ],
+ [
+ 26,
+ 54,
+ 28,
+ 47,
+ 53,
+ 37
+ ],
+ [
+ 55,
+ 34,
+ 1,
+ 4,
+ 21,
+ 22
+ ],
+ [
+ 41,
+ 5,
+ 40,
+ 21,
+ 17,
+ 23
+ ],
+ [
+ 25,
+ 43,
+ 52,
+ 26,
+ 4,
+ 55
+ ],
+ [
+ 52,
+ 29,
+ 35,
+ 17,
+ 45,
+ 60
+ ],
+ [
+ 38,
+ 10,
+ 15,
+ 7,
+ 50,
+ 3
+ ],
+ [
+ 23,
+ 6,
+ 19,
+ 56,
+ 41,
+ 15
+ ]
+ ],
+ [
+ [
+ 44,
+ 14,
+ 20,
+ 47,
+ 19,
+ 56
+ ],
+ [
+ 28,
+ 34,
+ 2,
+ 56,
+ 0,
+ 11
+ ],
+ [
+ 46,
+ 15,
+ 61,
+ 14,
+ 22,
+ 60
+ ],
+ [
+ 15,
+ 50,
+ 36,
+ 47,
+ 25,
+ 21
+ ],
+ [
+ 25,
+ 12,
+ 13,
+ 36,
+ 23,
+ 57
+ ],
+ [
+ 22,
+ 43,
+ 1,
+ 37,
+ 36,
+ 30
+ ],
+ [
+ 28,
+ 42,
+ 19,
+ 31,
+ 14,
+ 21
+ ],
+ [
+ 35,
+ 51,
+ 47,
+ 33,
+ 7,
+ 46
+ ],
+ [
+ 32,
+ 4,
+ 16,
+ 11,
+ 19,
+ 35
+ ],
+ [
+ 21,
+ 38,
+ 16,
+ 47,
+ 62,
+ 15
+ ],
+ [
+ 3,
+ 45,
+ 39,
+ 50,
+ 21,
+ 2
+ ],
+ [
+ 21,
+ 5,
+ 43,
+ 56,
+ 53,
+ 52
+ ],
+ [
+ 10,
+ 41,
+ 33,
+ 11,
+ 63,
+ 37
+ ],
+ [
+ 56,
+ 16,
+ 9,
+ 63,
+ 33,
+ 54
+ ],
+ [
+ 58,
+ 52,
+ 2,
+ 5,
+ 30,
+ 56
+ ],
+ [
+ 53,
+ 8,
+ 59,
+ 55,
+ 2,
+ 15
+ ],
+ [
+ 0,
+ 47,
+ 51,
+ 45,
+ 14,
+ 37
+ ],
+ [
+ 0,
+ 43,
+ 21,
+ 12,
+ 60,
+ 53
+ ],
+ [
+ 38,
+ 53,
+ 60,
+ 34,
+ 36,
+ 9
+ ],
+ [
+ 27,
+ 9,
+ 56,
+ 0,
+ 53,
+ 11
+ ],
+ [
+ 28,
+ 12,
+ 43,
+ 54,
+ 5,
+ 62
+ ],
+ [
+ 55,
+ 50,
+ 34,
+ 16,
+ 4,
+ 21
+ ],
+ [
+ 4,
+ 21,
+ 5,
+ 40,
+ 32,
+ 54
+ ],
+ [
+ 25,
+ 57,
+ 49,
+ 1,
+ 44,
+ 43
+ ],
+ [
+ 5,
+ 35,
+ 42,
+ 25,
+ 2,
+ 22
+ ],
+ [
+ 15,
+ 28,
+ 38,
+ 55,
+ 35,
+ 37
+ ],
+ [
+ 19,
+ 30,
+ 6,
+ 57,
+ 33,
+ 39
+ ]
+ ],
+ [
+ [
+ 11,
+ 46,
+ 0,
+ 49,
+ 31,
+ 16
+ ],
+ [
+ 13,
+ 49,
+ 50,
+ 16,
+ 31,
+ 19
+ ],
+ [
+ 36,
+ 13,
+ 27,
+ 34,
+ 4,
+ 52
+ ],
+ [
+ 24,
+ 32,
+ 7,
+ 59,
+ 13,
+ 15
+ ],
+ [
+ 17,
+ 35,
+ 2,
+ 44,
+ 10,
+ 63
+ ],
+ [
+ 8,
+ 7,
+ 23,
+ 26,
+ 56,
+ 42
+ ],
+ [
+ 58,
+ 61,
+ 29,
+ 38,
+ 50,
+ 62
+ ],
+ [
+ 20,
+ 42,
+ 3,
+ 35,
+ 61,
+ 47
+ ],
+ [
+ 4,
+ 10,
+ 7,
+ 47,
+ 25,
+ 34
+ ],
+ [
+ 54,
+ 55,
+ 18,
+ 38,
+ 27,
+ 28
+ ],
+ [
+ 60,
+ 1,
+ 14,
+ 59,
+ 3,
+ 30
+ ],
+ [
+ 51,
+ 31,
+ 5,
+ 21,
+ 25,
+ 52
+ ],
+ [
+ 34,
+ 10,
+ 2,
+ 26,
+ 11,
+ 47
+ ],
+ [
+ 16,
+ 40,
+ 23,
+ 34,
+ 9,
+ 33
+ ],
+ [
+ 34,
+ 8,
+ 1,
+ 31,
+ 5,
+ 32
+ ],
+ [
+ 25,
+ 59,
+ 8,
+ 52,
+ 53,
+ 27
+ ],
+ [
+ 13,
+ 47,
+ 45,
+ 48,
+ 42,
+ 0
+ ],
+ [
+ 4,
+ 23,
+ 53,
+ 10,
+ 3,
+ 25
+ ],
+ [
+ 53,
+ 8,
+ 38,
+ 11,
+ 24,
+ 63
+ ],
+ [
+ 38,
+ 15,
+ 0,
+ 11,
+ 21,
+ 8
+ ],
+ [
+ 27,
+ 23,
+ 62,
+ 8,
+ 41,
+ 42
+ ],
+ [
+ 55,
+ 11,
+ 10,
+ 41,
+ 16,
+ 34
+ ],
+ [
+ 31,
+ 21,
+ 54,
+ 41,
+ 34,
+ 46
+ ],
+ [
+ 55,
+ 25,
+ 22,
+ 33,
+ 44,
+ 47
+ ],
+ [
+ 47,
+ 45,
+ 35,
+ 20,
+ 12,
+ 42
+ ],
+ [
+ 50,
+ 10,
+ 47,
+ 57,
+ 35,
+ 53
+ ],
+ [
+ 17,
+ 31,
+ 5,
+ 37,
+ 36,
+ 20
+ ]
+ ],
+ [
+ [
+ 22,
+ 6,
+ 29,
+ 39,
+ 44,
+ 57
+ ],
+ [
+ 27,
+ 6,
+ 14,
+ 17,
+ 55,
+ 57
+ ],
+ [
+ 11,
+ 1,
+ 29,
+ 26,
+ 47,
+ 4
+ ],
+ [
+ 14,
+ 38,
+ 31,
+ 22,
+ 36,
+ 29
+ ],
+ [
+ 14,
+ 61,
+ 59,
+ 16,
+ 44,
+ 63
+ ],
+ [
+ 30,
+ 8,
+ 47,
+ 23,
+ 21,
+ 0
+ ],
+ [
+ 58,
+ 4,
+ 54,
+ 61,
+ 15,
+ 62
+ ],
+ [
+ 20,
+ 42,
+ 58,
+ 3,
+ 35,
+ 0
+ ],
+ [
+ 10,
+ 47,
+ 4,
+ 8,
+ 41,
+ 19
+ ],
+ [
+ 54,
+ 55,
+ 33,
+ 11,
+ 38,
+ 21
+ ],
+ [
+ 14,
+ 1,
+ 4,
+ 60,
+ 20,
+ 40
+ ],
+ [
+ 51,
+ 5,
+ 14,
+ 0,
+ 32,
+ 62
+ ],
+ [
+ 2,
+ 10,
+ 36,
+ 49,
+ 32,
+ 52
+ ],
+ [
+ 13,
+ 40,
+ 54,
+ 36,
+ 22,
+ 11
+ ],
+ [
+ 2,
+ 44,
+ 37,
+ 58,
+ 5,
+ 8
+ ],
+ [
+ 6,
+ 24,
+ 2,
+ 53,
+ 19,
+ 52
+ ],
+ [
+ 13,
+ 47,
+ 61,
+ 5,
+ 50,
+ 30
+ ],
+ [
+ 55,
+ 58,
+ 4,
+ 63,
+ 22,
+ 33
+ ],
+ [
+ 35,
+ 32,
+ 46,
+ 14,
+ 7,
+ 58
+ ],
+ [
+ 15,
+ 13,
+ 45,
+ 0,
+ 51,
+ 60
+ ],
+ [
+ 15,
+ 27,
+ 23,
+ 50,
+ 6,
+ 13
+ ],
+ [
+ 57,
+ 36,
+ 10,
+ 55,
+ 14,
+ 16
+ ],
+ [
+ 10,
+ 23,
+ 41,
+ 33,
+ 30,
+ 54
+ ],
+ [
+ 33,
+ 26,
+ 4,
+ 10,
+ 49,
+ 55
+ ],
+ [
+ 4,
+ 8,
+ 33,
+ 9,
+ 20,
+ 48
+ ],
+ [
+ 43,
+ 6,
+ 3,
+ 50,
+ 18,
+ 38
+ ],
+ [
+ 4,
+ 55,
+ 36,
+ 43,
+ 46,
+ 5
+ ]
+ ],
+ [
+ [
+ 48,
+ 38,
+ 47,
+ 63,
+ 7,
+ 42
+ ],
+ [
+ 3,
+ 26,
+ 10,
+ 6,
+ 2,
+ 35
+ ],
+ [
+ 39,
+ 44,
+ 6,
+ 7,
+ 45,
+ 8
+ ],
+ [
+ 60,
+ 31,
+ 22,
+ 0,
+ 54,
+ 27
+ ],
+ [
+ 6,
+ 59,
+ 16,
+ 14,
+ 37,
+ 44
+ ],
+ [
+ 59,
+ 8,
+ 30,
+ 47,
+ 48,
+ 3
+ ],
+ [
+ 44,
+ 54,
+ 4,
+ 56,
+ 15,
+ 13
+ ],
+ [
+ 12,
+ 24,
+ 20,
+ 58,
+ 31,
+ 61
+ ],
+ [
+ 47,
+ 33,
+ 10,
+ 8,
+ 4,
+ 36
+ ],
+ [
+ 54,
+ 40,
+ 55,
+ 33,
+ 11,
+ 51
+ ],
+ [
+ 8,
+ 14,
+ 23,
+ 29,
+ 22,
+ 31
+ ],
+ [
+ 34,
+ 59,
+ 14,
+ 5,
+ 57,
+ 50
+ ],
+ [
+ 44,
+ 36,
+ 10,
+ 59,
+ 52,
+ 9
+ ],
+ [
+ 59,
+ 44,
+ 13,
+ 9,
+ 1,
+ 54
+ ],
+ [
+ 24,
+ 2,
+ 38,
+ 46,
+ 44,
+ 47
+ ],
+ [
+ 34,
+ 53,
+ 6,
+ 2,
+ 57,
+ 40
+ ],
+ [
+ 17,
+ 29,
+ 47,
+ 5,
+ 50,
+ 34
+ ],
+ [
+ 48,
+ 63,
+ 57,
+ 38,
+ 60,
+ 32
+ ],
+ [
+ 42,
+ 7,
+ 49,
+ 32,
+ 46,
+ 1
+ ],
+ [
+ 1,
+ 59,
+ 23,
+ 14,
+ 16,
+ 10
+ ],
+ [
+ 21,
+ 35,
+ 7,
+ 53,
+ 6,
+ 17
+ ],
+ [
+ 8,
+ 24,
+ 60,
+ 14,
+ 13,
+ 55
+ ],
+ [
+ 33,
+ 5,
+ 51,
+ 15,
+ 23,
+ 3
+ ],
+ [
+ 9,
+ 4,
+ 51,
+ 26,
+ 41,
+ 8
+ ],
+ [
+ 4,
+ 60,
+ 19,
+ 47,
+ 40,
+ 9
+ ],
+ [
+ 19,
+ 3,
+ 26,
+ 21,
+ 17,
+ 8
+ ],
+ [
+ 1,
+ 35,
+ 53,
+ 54,
+ 60,
+ 20
+ ]
+ ],
+ [
+ [
+ 12,
+ 41,
+ 14,
+ 62,
+ 24,
+ 10
+ ],
+ [
+ 10,
+ 53,
+ 39,
+ 35,
+ 41,
+ 58
+ ],
+ [
+ 33,
+ 32,
+ 50,
+ 31,
+ 3,
+ 34
+ ],
+ [
+ 43,
+ 10,
+ 42,
+ 11,
+ 17,
+ 47
+ ],
+ [
+ 42,
+ 12,
+ 11,
+ 19,
+ 58,
+ 54
+ ],
+ [
+ 36,
+ 50,
+ 55,
+ 61,
+ 25,
+ 56
+ ],
+ [
+ 41,
+ 22,
+ 16,
+ 52,
+ 2,
+ 15
+ ],
+ [
+ 26,
+ 36,
+ 62,
+ 53,
+ 15,
+ 51
+ ],
+ [
+ 58,
+ 16,
+ 5,
+ 53,
+ 3,
+ 49
+ ],
+ [
+ 32,
+ 46,
+ 26,
+ 45,
+ 16,
+ 62
+ ],
+ [
+ 41,
+ 45,
+ 56,
+ 49,
+ 11,
+ 3
+ ],
+ [
+ 52,
+ 34,
+ 35,
+ 50,
+ 21,
+ 53
+ ],
+ [
+ 59,
+ 53,
+ 46,
+ 30,
+ 39,
+ 37
+ ],
+ [
+ 20,
+ 9,
+ 52,
+ 2,
+ 7,
+ 33
+ ],
+ [
+ 20,
+ 50,
+ 24,
+ 29,
+ 23,
+ 2
+ ],
+ [
+ 53,
+ 2,
+ 44,
+ 41,
+ 9,
+ 13
+ ],
+ [
+ 47,
+ 27,
+ 1,
+ 5,
+ 45,
+ 46
+ ],
+ [
+ 61,
+ 7,
+ 51,
+ 30,
+ 35,
+ 9
+ ],
+ [
+ 19,
+ 34,
+ 32,
+ 17,
+ 2,
+ 14
+ ],
+ [
+ 15,
+ 32,
+ 6,
+ 45,
+ 9,
+ 11
+ ],
+ [
+ 35,
+ 27,
+ 6,
+ 53,
+ 13,
+ 60
+ ],
+ [
+ 26,
+ 62,
+ 16,
+ 28,
+ 41,
+ 3
+ ],
+ [
+ 5,
+ 20,
+ 46,
+ 37,
+ 11,
+ 55
+ ],
+ [
+ 55,
+ 47,
+ 4,
+ 16,
+ 14,
+ 27
+ ],
+ [
+ 36,
+ 11,
+ 27,
+ 62,
+ 33,
+ 7
+ ],
+ [
+ 33,
+ 7,
+ 54,
+ 3,
+ 32,
+ 12
+ ],
+ [
+ 58,
+ 52,
+ 27,
+ 26,
+ 48,
+ 38
+ ]
+ ],
+ [
+ [
+ 6,
+ 52,
+ 19,
+ 63,
+ 38,
+ 46
+ ],
+ [
+ 8,
+ 42,
+ 4,
+ 47,
+ 57,
+ 56
+ ],
+ [
+ 31,
+ 46,
+ 32,
+ 4,
+ 14,
+ 10
+ ],
+ [
+ 28,
+ 27,
+ 4,
+ 37,
+ 58,
+ 20
+ ],
+ [
+ 57,
+ 59,
+ 60,
+ 62,
+ 22,
+ 14
+ ],
+ [
+ 30,
+ 2,
+ 9,
+ 57,
+ 11,
+ 43
+ ],
+ [
+ 20,
+ 19,
+ 57,
+ 42,
+ 27,
+ 51
+ ],
+ [
+ 46,
+ 35,
+ 47,
+ 7,
+ 32,
+ 0
+ ],
+ [
+ 12,
+ 4,
+ 35,
+ 10,
+ 47,
+ 50
+ ],
+ [
+ 15,
+ 47,
+ 54,
+ 25,
+ 38,
+ 51
+ ],
+ [
+ 18,
+ 42,
+ 5,
+ 15,
+ 38,
+ 61
+ ],
+ [
+ 18,
+ 22,
+ 8,
+ 16,
+ 5,
+ 7
+ ],
+ [
+ 5,
+ 3,
+ 24,
+ 35,
+ 4,
+ 30
+ ],
+ [
+ 17,
+ 13,
+ 55,
+ 20,
+ 36,
+ 22
+ ],
+ [
+ 22,
+ 10,
+ 21,
+ 54,
+ 6,
+ 47
+ ],
+ [
+ 30,
+ 53,
+ 6,
+ 19,
+ 2,
+ 54
+ ],
+ [
+ 22,
+ 26,
+ 5,
+ 7,
+ 47,
+ 21
+ ],
+ [
+ 4,
+ 41,
+ 13,
+ 46,
+ 55,
+ 43
+ ],
+ [
+ 17,
+ 56,
+ 32,
+ 45,
+ 14,
+ 6
+ ],
+ [
+ 5,
+ 49,
+ 53,
+ 28,
+ 34,
+ 60
+ ],
+ [
+ 42,
+ 55,
+ 57,
+ 17,
+ 28,
+ 22
+ ],
+ [
+ 40,
+ 23,
+ 28,
+ 57,
+ 21,
+ 16
+ ],
+ [
+ 51,
+ 21,
+ 35,
+ 24,
+ 44,
+ 10
+ ],
+ [
+ 38,
+ 25,
+ 4,
+ 14,
+ 62,
+ 31
+ ],
+ [
+ 31,
+ 50,
+ 13,
+ 56,
+ 39,
+ 8
+ ],
+ [
+ 6,
+ 51,
+ 55,
+ 8,
+ 0,
+ 21
+ ],
+ [
+ 29,
+ 46,
+ 18,
+ 55,
+ 37,
+ 50
+ ]
+ ],
+ [
+ [
+ 46,
+ 31,
+ 35,
+ 49,
+ 14,
+ 21
+ ],
+ [
+ 13,
+ 16,
+ 50,
+ 31,
+ 33,
+ 8
+ ],
+ [
+ 13,
+ 36,
+ 27,
+ 52,
+ 3,
+ 19
+ ],
+ [
+ 9,
+ 24,
+ 29,
+ 12,
+ 55,
+ 61
+ ],
+ [
+ 35,
+ 17,
+ 2,
+ 10,
+ 41,
+ 48
+ ],
+ [
+ 51,
+ 23,
+ 30,
+ 57,
+ 52,
+ 17
+ ],
+ [
+ 34,
+ 48,
+ 0,
+ 38,
+ 27,
+ 51
+ ],
+ [
+ 47,
+ 32,
+ 2,
+ 35,
+ 58,
+ 55
+ ],
+ [
+ 15,
+ 12,
+ 35,
+ 52,
+ 45,
+ 4
+ ],
+ [
+ 36,
+ 15,
+ 4,
+ 59,
+ 11,
+ 54
+ ],
+ [
+ 60,
+ 5,
+ 59,
+ 29,
+ 50,
+ 18
+ ],
+ [
+ 18,
+ 8,
+ 31,
+ 23,
+ 24,
+ 3
+ ],
+ [
+ 34,
+ 4,
+ 5,
+ 58,
+ 14,
+ 27
+ ],
+ [
+ 17,
+ 57,
+ 36,
+ 55,
+ 51,
+ 6
+ ],
+ [
+ 10,
+ 45,
+ 8,
+ 54,
+ 22,
+ 47
+ ],
+ [
+ 30,
+ 50,
+ 14,
+ 6,
+ 16,
+ 37
+ ],
+ [
+ 26,
+ 0,
+ 22,
+ 5,
+ 13,
+ 4
+ ],
+ [
+ 4,
+ 46,
+ 23,
+ 43,
+ 60,
+ 34
+ ],
+ [
+ 45,
+ 8,
+ 56,
+ 62,
+ 17,
+ 51
+ ],
+ [
+ 4,
+ 38,
+ 53,
+ 15,
+ 58,
+ 5
+ ],
+ [
+ 57,
+ 24,
+ 34,
+ 15,
+ 55,
+ 42
+ ],
+ [
+ 23,
+ 37,
+ 40,
+ 41,
+ 54,
+ 11
+ ],
+ [
+ 48,
+ 29,
+ 51,
+ 31,
+ 34,
+ 9
+ ],
+ [
+ 62,
+ 14,
+ 18,
+ 31,
+ 4,
+ 59
+ ],
+ [
+ 50,
+ 31,
+ 49,
+ 45,
+ 56,
+ 57
+ ],
+ [
+ 51,
+ 50,
+ 56,
+ 8,
+ 12,
+ 52
+ ],
+ [
+ 17,
+ 37,
+ 5,
+ 39,
+ 44,
+ 20
+ ]
+ ],
+ [
+ [
+ 45,
+ 13,
+ 63,
+ 37,
+ 38,
+ 56
+ ],
+ [
+ 63,
+ 6,
+ 12,
+ 18,
+ 27,
+ 51
+ ],
+ [
+ 3,
+ 21,
+ 4,
+ 48,
+ 17,
+ 27
+ ],
+ [
+ 14,
+ 55,
+ 9,
+ 37,
+ 29,
+ 26
+ ],
+ [
+ 35,
+ 57,
+ 2,
+ 13,
+ 41,
+ 10
+ ],
+ [
+ 39,
+ 30,
+ 57,
+ 13,
+ 53,
+ 23
+ ],
+ [
+ 34,
+ 48,
+ 38,
+ 27,
+ 56,
+ 51
+ ],
+ [
+ 47,
+ 35,
+ 46,
+ 2,
+ 32,
+ 7
+ ],
+ [
+ 12,
+ 56,
+ 4,
+ 35,
+ 52,
+ 50
+ ],
+ [
+ 15,
+ 36,
+ 14,
+ 30,
+ 47,
+ 48
+ ],
+ [
+ 5,
+ 60,
+ 59,
+ 22,
+ 15,
+ 46
+ ],
+ [
+ 8,
+ 22,
+ 23,
+ 47,
+ 5,
+ 26
+ ],
+ [
+ 4,
+ 5,
+ 30,
+ 58,
+ 26,
+ 8
+ ],
+ [
+ 17,
+ 29,
+ 13,
+ 50,
+ 14,
+ 34
+ ],
+ [
+ 54,
+ 10,
+ 41,
+ 14,
+ 21,
+ 5
+ ],
+ [
+ 50,
+ 6,
+ 54,
+ 30,
+ 16,
+ 56
+ ],
+ [
+ 0,
+ 47,
+ 7,
+ 4,
+ 5,
+ 34
+ ],
+ [
+ 4,
+ 46,
+ 23,
+ 41,
+ 59,
+ 55
+ ],
+ [
+ 62,
+ 45,
+ 1,
+ 57,
+ 47,
+ 32
+ ],
+ [
+ 4,
+ 53,
+ 15,
+ 50,
+ 60,
+ 1
+ ],
+ [
+ 34,
+ 57,
+ 24,
+ 31,
+ 15,
+ 53
+ ],
+ [
+ 23,
+ 41,
+ 45,
+ 12,
+ 57,
+ 14
+ ],
+ [
+ 51,
+ 29,
+ 49,
+ 48,
+ 18,
+ 33
+ ],
+ [
+ 14,
+ 4,
+ 62,
+ 26,
+ 18,
+ 28
+ ],
+ [
+ 50,
+ 49,
+ 7,
+ 24,
+ 9,
+ 48
+ ],
+ [
+ 56,
+ 20,
+ 51,
+ 3,
+ 14,
+ 26
+ ],
+ [
+ 55,
+ 37,
+ 50,
+ 14,
+ 42,
+ 20
+ ]
+ ],
+ [
+ [
+ 51,
+ 43,
+ 27,
+ 30,
+ 5,
+ 12
+ ],
+ [
+ 24,
+ 16,
+ 48,
+ 15,
+ 7,
+ 30
+ ],
+ [
+ 26,
+ 21,
+ 50,
+ 52,
+ 4,
+ 56
+ ],
+ [
+ 19,
+ 17,
+ 2,
+ 14,
+ 57,
+ 22
+ ],
+ [
+ 3,
+ 35,
+ 37,
+ 45,
+ 1,
+ 6
+ ],
+ [
+ 13,
+ 30,
+ 23,
+ 39,
+ 57,
+ 60
+ ],
+ [
+ 9,
+ 34,
+ 48,
+ 17,
+ 27,
+ 26
+ ],
+ [
+ 47,
+ 32,
+ 2,
+ 59,
+ 28,
+ 57
+ ],
+ [
+ 18,
+ 12,
+ 31,
+ 20,
+ 52,
+ 4
+ ],
+ [
+ 14,
+ 15,
+ 36,
+ 30,
+ 58,
+ 22
+ ],
+ [
+ 5,
+ 4,
+ 26,
+ 19,
+ 22,
+ 59
+ ],
+ [
+ 22,
+ 8,
+ 34,
+ 26,
+ 42,
+ 60
+ ],
+ [
+ 20,
+ 5,
+ 4,
+ 52,
+ 30,
+ 33
+ ],
+ [
+ 34,
+ 17,
+ 2,
+ 5,
+ 39,
+ 20
+ ],
+ [
+ 54,
+ 61,
+ 30,
+ 14,
+ 25,
+ 20
+ ],
+ [
+ 56,
+ 41,
+ 2,
+ 5,
+ 54,
+ 37
+ ],
+ [
+ 12,
+ 34,
+ 47,
+ 7,
+ 54,
+ 59
+ ],
+ [
+ 9,
+ 11,
+ 2,
+ 43,
+ 33,
+ 50
+ ],
+ [
+ 8,
+ 10,
+ 19,
+ 20,
+ 32,
+ 18
+ ],
+ [
+ 17,
+ 6,
+ 38,
+ 29,
+ 49,
+ 41
+ ],
+ [
+ 38,
+ 52,
+ 45,
+ 57,
+ 63,
+ 27
+ ],
+ [
+ 26,
+ 28,
+ 3,
+ 40,
+ 5,
+ 47
+ ],
+ [
+ 60,
+ 49,
+ 35,
+ 34,
+ 38,
+ 3
+ ],
+ [
+ 61,
+ 24,
+ 4,
+ 55,
+ 45,
+ 16
+ ],
+ [
+ 46,
+ 11,
+ 27,
+ 0,
+ 56,
+ 48
+ ],
+ [
+ 60,
+ 45,
+ 44,
+ 25,
+ 32,
+ 3
+ ],
+ [
+ 11,
+ 38,
+ 52,
+ 48,
+ 9,
+ 21
+ ]
+ ],
+ [
+ [
+ 22,
+ 19,
+ 46,
+ 31,
+ 3,
+ 23
+ ],
+ [
+ 32,
+ 62,
+ 15,
+ 54,
+ 10,
+ 55
+ ],
+ [
+ 47,
+ 30,
+ 38,
+ 5,
+ 7,
+ 60
+ ],
+ [
+ 15,
+ 13,
+ 1,
+ 8,
+ 25,
+ 43
+ ],
+ [
+ 13,
+ 59,
+ 5,
+ 6,
+ 62,
+ 52
+ ],
+ [
+ 27,
+ 63,
+ 62,
+ 45,
+ 12,
+ 56
+ ],
+ [
+ 50,
+ 9,
+ 8,
+ 51,
+ 48,
+ 18
+ ],
+ [
+ 59,
+ 57,
+ 2,
+ 28,
+ 61,
+ 6
+ ],
+ [
+ 18,
+ 59,
+ 6,
+ 52,
+ 39,
+ 57
+ ],
+ [
+ 14,
+ 23,
+ 11,
+ 36,
+ 15,
+ 32
+ ],
+ [
+ 26,
+ 5,
+ 42,
+ 25,
+ 22,
+ 23
+ ],
+ [
+ 33,
+ 22,
+ 55,
+ 28,
+ 16,
+ 24
+ ],
+ [
+ 6,
+ 20,
+ 33,
+ 14,
+ 52,
+ 15
+ ],
+ [
+ 17,
+ 6,
+ 5,
+ 39,
+ 2,
+ 34
+ ],
+ [
+ 54,
+ 62,
+ 25,
+ 61,
+ 21,
+ 14
+ ],
+ [
+ 41,
+ 45,
+ 14,
+ 5,
+ 2,
+ 54
+ ],
+ [
+ 47,
+ 34,
+ 10,
+ 31,
+ 5,
+ 41
+ ],
+ [
+ 9,
+ 2,
+ 36,
+ 6,
+ 43,
+ 38
+ ],
+ [
+ 57,
+ 20,
+ 41,
+ 10,
+ 32,
+ 18
+ ],
+ [
+ 47,
+ 6,
+ 49,
+ 15,
+ 34,
+ 7
+ ],
+ [
+ 3,
+ 57,
+ 44,
+ 38,
+ 50,
+ 53
+ ],
+ [
+ 47,
+ 28,
+ 57,
+ 26,
+ 19,
+ 22
+ ],
+ [
+ 11,
+ 1,
+ 18,
+ 5,
+ 46,
+ 22
+ ],
+ [
+ 9,
+ 55,
+ 4,
+ 3,
+ 25,
+ 10
+ ],
+ [
+ 45,
+ 11,
+ 0,
+ 63,
+ 48,
+ 57
+ ],
+ [
+ 42,
+ 10,
+ 3,
+ 43,
+ 49,
+ 0
+ ],
+ [
+ 38,
+ 15,
+ 17,
+ 41,
+ 10,
+ 3
+ ]
+ ],
+ [
+ [
+ 19,
+ 18,
+ 51,
+ 25,
+ 60,
+ 55
+ ],
+ [
+ 56,
+ 27,
+ 61,
+ 42,
+ 55,
+ 23
+ ],
+ [
+ 32,
+ 39,
+ 37,
+ 46,
+ 20,
+ 52
+ ],
+ [
+ 41,
+ 21,
+ 37,
+ 13,
+ 57,
+ 2
+ ],
+ [
+ 10,
+ 9,
+ 3,
+ 46,
+ 58,
+ 32
+ ],
+ [
+ 62,
+ 57,
+ 27,
+ 43,
+ 2,
+ 53
+ ],
+ [
+ 50,
+ 21,
+ 19,
+ 48,
+ 15,
+ 10
+ ],
+ [
+ 14,
+ 21,
+ 54,
+ 8,
+ 57,
+ 31
+ ],
+ [
+ 6,
+ 18,
+ 4,
+ 52,
+ 39,
+ 24
+ ],
+ [
+ 32,
+ 38,
+ 23,
+ 53,
+ 17,
+ 20
+ ],
+ [
+ 26,
+ 25,
+ 22,
+ 5,
+ 33,
+ 42
+ ],
+ [
+ 5,
+ 22,
+ 16,
+ 61,
+ 9,
+ 55
+ ],
+ [
+ 49,
+ 33,
+ 30,
+ 25,
+ 20,
+ 4
+ ],
+ [
+ 18,
+ 17,
+ 25,
+ 63,
+ 39,
+ 11
+ ],
+ [
+ 62,
+ 54,
+ 10,
+ 5,
+ 58,
+ 37
+ ],
+ [
+ 42,
+ 41,
+ 5,
+ 24,
+ 37,
+ 54
+ ],
+ [
+ 47,
+ 34,
+ 27,
+ 10,
+ 22,
+ 33
+ ],
+ [
+ 60,
+ 43,
+ 44,
+ 52,
+ 7,
+ 37
+ ],
+ [
+ 12,
+ 32,
+ 39,
+ 38,
+ 1,
+ 20
+ ],
+ [
+ 34,
+ 52,
+ 49,
+ 15,
+ 63,
+ 28
+ ],
+ [
+ 28,
+ 39,
+ 3,
+ 26,
+ 30,
+ 22
+ ],
+ [
+ 29,
+ 1,
+ 40,
+ 22,
+ 19,
+ 63
+ ],
+ [
+ 41,
+ 53,
+ 5,
+ 33,
+ 4,
+ 26
+ ],
+ [
+ 25,
+ 38,
+ 4,
+ 34,
+ 51,
+ 49
+ ],
+ [
+ 29,
+ 52,
+ 48,
+ 47,
+ 20,
+ 33
+ ],
+ [
+ 50,
+ 3,
+ 16,
+ 38,
+ 53,
+ 15
+ ],
+ [
+ 19,
+ 62,
+ 6,
+ 23,
+ 36,
+ 10
+ ]
+ ],
+ [
+ [
+ 5,
+ 14,
+ 17,
+ 57,
+ 10,
+ 27
+ ],
+ [
+ 43,
+ 9,
+ 56,
+ 1,
+ 14,
+ 33
+ ],
+ [
+ 63,
+ 35,
+ 43,
+ 1,
+ 10,
+ 27
+ ],
+ [
+ 51,
+ 50,
+ 57,
+ 41,
+ 20,
+ 54
+ ],
+ [
+ 11,
+ 43,
+ 40,
+ 33,
+ 30,
+ 54
+ ],
+ [
+ 27,
+ 53,
+ 63,
+ 30,
+ 15,
+ 58
+ ],
+ [
+ 51,
+ 48,
+ 21,
+ 57,
+ 9,
+ 50
+ ],
+ [
+ 21,
+ 59,
+ 14,
+ 52,
+ 7,
+ 28
+ ],
+ [
+ 6,
+ 18,
+ 4,
+ 0,
+ 12,
+ 59
+ ],
+ [
+ 14,
+ 53,
+ 30,
+ 42,
+ 4,
+ 36
+ ],
+ [
+ 26,
+ 42,
+ 5,
+ 22,
+ 19,
+ 58
+ ],
+ [
+ 22,
+ 5,
+ 34,
+ 38,
+ 52,
+ 29
+ ],
+ [
+ 49,
+ 33,
+ 0,
+ 20,
+ 26,
+ 8
+ ],
+ [
+ 17,
+ 2,
+ 39,
+ 13,
+ 44,
+ 63
+ ],
+ [
+ 10,
+ 54,
+ 20,
+ 37,
+ 5,
+ 59
+ ],
+ [
+ 41,
+ 43,
+ 24,
+ 5,
+ 53,
+ 14
+ ],
+ [
+ 12,
+ 47,
+ 34,
+ 57,
+ 27,
+ 5
+ ],
+ [
+ 9,
+ 33,
+ 7,
+ 38,
+ 43,
+ 31
+ ],
+ [
+ 19,
+ 10,
+ 32,
+ 20,
+ 1,
+ 18
+ ],
+ [
+ 36,
+ 63,
+ 15,
+ 49,
+ 40,
+ 50
+ ],
+ [
+ 39,
+ 3,
+ 38,
+ 27,
+ 36,
+ 35
+ ],
+ [
+ 29,
+ 19,
+ 40,
+ 47,
+ 62,
+ 14
+ ],
+ [
+ 5,
+ 46,
+ 33,
+ 53,
+ 49,
+ 21
+ ],
+ [
+ 55,
+ 4,
+ 27,
+ 61,
+ 51,
+ 34
+ ],
+ [
+ 11,
+ 0,
+ 59,
+ 48,
+ 14,
+ 27
+ ],
+ [
+ 7,
+ 32,
+ 3,
+ 51,
+ 18,
+ 14
+ ],
+ [
+ 11,
+ 9,
+ 36,
+ 48,
+ 0,
+ 46
+ ]
+ ],
+ [
+ [
+ 63,
+ 62,
+ 60,
+ 19,
+ 23,
+ 56
+ ],
+ [
+ 48,
+ 32,
+ 1,
+ 35,
+ 5,
+ 21
+ ],
+ [
+ 22,
+ 24,
+ 46,
+ 58,
+ 59,
+ 60
+ ],
+ [
+ 27,
+ 37,
+ 50,
+ 28,
+ 61,
+ 6
+ ],
+ [
+ 10,
+ 12,
+ 15,
+ 58,
+ 35,
+ 23
+ ],
+ [
+ 2,
+ 43,
+ 57,
+ 36,
+ 30,
+ 20
+ ],
+ [
+ 21,
+ 14,
+ 19,
+ 41,
+ 63,
+ 42
+ ],
+ [
+ 46,
+ 7,
+ 35,
+ 43,
+ 21,
+ 36
+ ],
+ [
+ 6,
+ 18,
+ 0,
+ 35,
+ 40,
+ 4
+ ],
+ [
+ 47,
+ 53,
+ 32,
+ 38,
+ 1,
+ 58
+ ],
+ [
+ 26,
+ 18,
+ 22,
+ 15,
+ 5,
+ 46
+ ],
+ [
+ 7,
+ 22,
+ 24,
+ 63,
+ 53,
+ 5
+ ],
+ [
+ 11,
+ 3,
+ 10,
+ 18,
+ 37,
+ 24
+ ],
+ [
+ 63,
+ 18,
+ 13,
+ 39,
+ 0,
+ 17
+ ],
+ [
+ 62,
+ 54,
+ 5,
+ 18,
+ 6,
+ 14
+ ],
+ [
+ 19,
+ 43,
+ 31,
+ 30,
+ 57,
+ 42
+ ],
+ [
+ 24,
+ 5,
+ 51,
+ 34,
+ 47,
+ 21
+ ],
+ [
+ 59,
+ 13,
+ 33,
+ 44,
+ 11,
+ 41
+ ],
+ [
+ 58,
+ 32,
+ 1,
+ 7,
+ 39,
+ 38
+ ],
+ [
+ 49,
+ 13,
+ 28,
+ 9,
+ 34,
+ 7
+ ],
+ [
+ 32,
+ 51,
+ 3,
+ 53,
+ 48,
+ 13
+ ],
+ [
+ 40,
+ 15,
+ 37,
+ 57,
+ 1,
+ 23
+ ],
+ [
+ 5,
+ 53,
+ 51,
+ 35,
+ 22,
+ 38
+ ],
+ [
+ 4,
+ 14,
+ 44,
+ 41,
+ 11,
+ 51
+ ],
+ [
+ 47,
+ 48,
+ 8,
+ 61,
+ 13,
+ 4
+ ],
+ [
+ 23,
+ 3,
+ 27,
+ 43,
+ 1,
+ 0
+ ],
+ [
+ 46,
+ 23,
+ 59,
+ 62,
+ 18,
+ 50
+ ]
+ ],
+ [
+ [
+ 62,
+ 0,
+ 9,
+ 32,
+ 26,
+ 41
+ ],
+ [
+ 45,
+ 29,
+ 35,
+ 7,
+ 30,
+ 62
+ ],
+ [
+ 56,
+ 31,
+ 23,
+ 53,
+ 28,
+ 2
+ ],
+ [
+ 36,
+ 2,
+ 5,
+ 4,
+ 48,
+ 41
+ ],
+ [
+ 18,
+ 0,
+ 15,
+ 23,
+ 16,
+ 11
+ ],
+ [
+ 4,
+ 36,
+ 57,
+ 56,
+ 14,
+ 5
+ ],
+ [
+ 59,
+ 56,
+ 21,
+ 8,
+ 7,
+ 33
+ ],
+ [
+ 12,
+ 23,
+ 7,
+ 2,
+ 28,
+ 34
+ ],
+ [
+ 17,
+ 6,
+ 23,
+ 19,
+ 62,
+ 27
+ ],
+ [
+ 53,
+ 32,
+ 51,
+ 38,
+ 41,
+ 58
+ ],
+ [
+ 18,
+ 63,
+ 19,
+ 22,
+ 26,
+ 5
+ ],
+ [
+ 22,
+ 7,
+ 11,
+ 60,
+ 12,
+ 19
+ ],
+ [
+ 24,
+ 7,
+ 11,
+ 17,
+ 30,
+ 37
+ ],
+ [
+ 47,
+ 13,
+ 63,
+ 50,
+ 39,
+ 45
+ ],
+ [
+ 62,
+ 54,
+ 18,
+ 14,
+ 29,
+ 56
+ ],
+ [
+ 56,
+ 30,
+ 0,
+ 55,
+ 51,
+ 10
+ ],
+ [
+ 12,
+ 5,
+ 21,
+ 34,
+ 63,
+ 29
+ ],
+ [
+ 42,
+ 8,
+ 38,
+ 57,
+ 41,
+ 44
+ ],
+ [
+ 32,
+ 54,
+ 1,
+ 37,
+ 49,
+ 40
+ ],
+ [
+ 5,
+ 57,
+ 49,
+ 28,
+ 34,
+ 10
+ ],
+ [
+ 46,
+ 32,
+ 13,
+ 7,
+ 61,
+ 48
+ ],
+ [
+ 40,
+ 3,
+ 30,
+ 60,
+ 39,
+ 14
+ ],
+ [
+ 5,
+ 44,
+ 33,
+ 36,
+ 28,
+ 31
+ ],
+ [
+ 4,
+ 11,
+ 28,
+ 41,
+ 51,
+ 5
+ ],
+ [
+ 48,
+ 47,
+ 61,
+ 28,
+ 27,
+ 60
+ ],
+ [
+ 14,
+ 3,
+ 26,
+ 12,
+ 53,
+ 61
+ ],
+ [
+ 54,
+ 60,
+ 49,
+ 35,
+ 62,
+ 10
+ ]
+ ],
+ [
+ [
+ 27,
+ 62,
+ 63,
+ 23,
+ 47,
+ 56
+ ],
+ [
+ 7,
+ 4,
+ 2,
+ 35,
+ 10,
+ 36
+ ],
+ [
+ 3,
+ 0,
+ 27,
+ 62,
+ 50,
+ 60
+ ],
+ [
+ 36,
+ 3,
+ 42,
+ 18,
+ 2,
+ 48
+ ],
+ [
+ 61,
+ 12,
+ 27,
+ 10,
+ 15,
+ 14
+ ],
+ [
+ 36,
+ 33,
+ 50,
+ 6,
+ 29,
+ 16
+ ],
+ [
+ 2,
+ 41,
+ 8,
+ 43,
+ 40,
+ 59
+ ],
+ [
+ 36,
+ 53,
+ 15,
+ 7,
+ 37,
+ 2
+ ],
+ [
+ 3,
+ 58,
+ 4,
+ 19,
+ 5,
+ 27
+ ],
+ [
+ 22,
+ 26,
+ 53,
+ 45,
+ 25,
+ 38
+ ],
+ [
+ 18,
+ 41,
+ 22,
+ 3,
+ 34,
+ 45
+ ],
+ [
+ 21,
+ 53,
+ 5,
+ 22,
+ 28,
+ 9
+ ],
+ [
+ 11,
+ 28,
+ 24,
+ 16,
+ 8,
+ 35
+ ],
+ [
+ 38,
+ 9,
+ 21,
+ 52,
+ 41,
+ 39
+ ],
+ [
+ 27,
+ 54,
+ 14,
+ 5,
+ 8,
+ 63
+ ],
+ [
+ 56,
+ 55,
+ 43,
+ 30,
+ 35,
+ 51
+ ],
+ [
+ 51,
+ 3,
+ 14,
+ 60,
+ 5,
+ 26
+ ],
+ [
+ 62,
+ 30,
+ 8,
+ 21,
+ 44,
+ 35
+ ],
+ [
+ 38,
+ 34,
+ 27,
+ 32,
+ 22,
+ 8
+ ],
+ [
+ 22,
+ 53,
+ 28,
+ 60,
+ 49,
+ 12
+ ],
+ [
+ 49,
+ 32,
+ 41,
+ 53,
+ 12,
+ 33
+ ],
+ [
+ 40,
+ 45,
+ 22,
+ 44,
+ 1,
+ 34
+ ],
+ [
+ 5,
+ 4,
+ 53,
+ 25,
+ 41,
+ 63
+ ],
+ [
+ 4,
+ 11,
+ 25,
+ 43,
+ 31,
+ 14
+ ],
+ [
+ 31,
+ 52,
+ 5,
+ 58,
+ 48,
+ 43
+ ],
+ [
+ 23,
+ 1,
+ 28,
+ 3,
+ 0,
+ 55
+ ],
+ [
+ 27,
+ 45,
+ 32,
+ 4,
+ 30,
+ 6
+ ]
+ ],
+ [
+ [
+ 41,
+ 2,
+ 42,
+ 16,
+ 50,
+ 32
+ ],
+ [
+ 51,
+ 5,
+ 41,
+ 40,
+ 21,
+ 44
+ ],
+ [
+ 43,
+ 1,
+ 29,
+ 55,
+ 21,
+ 35
+ ],
+ [
+ 36,
+ 58,
+ 25,
+ 3,
+ 18,
+ 54
+ ],
+ [
+ 31,
+ 60,
+ 24,
+ 12,
+ 61,
+ 41
+ ],
+ [
+ 4,
+ 10,
+ 35,
+ 36,
+ 0,
+ 43
+ ],
+ [
+ 45,
+ 43,
+ 63,
+ 35,
+ 36,
+ 48
+ ],
+ [
+ 3,
+ 9,
+ 8,
+ 7,
+ 43,
+ 27
+ ],
+ [
+ 0,
+ 43,
+ 4,
+ 40,
+ 18,
+ 44
+ ],
+ [
+ 34,
+ 15,
+ 50,
+ 38,
+ 53,
+ 25
+ ],
+ [
+ 24,
+ 46,
+ 26,
+ 13,
+ 2,
+ 18
+ ],
+ [
+ 57,
+ 37,
+ 5,
+ 22,
+ 7,
+ 53
+ ],
+ [
+ 3,
+ 9,
+ 38,
+ 26,
+ 37,
+ 23
+ ],
+ [
+ 60,
+ 26,
+ 0,
+ 39,
+ 13,
+ 55
+ ],
+ [
+ 54,
+ 21,
+ 58,
+ 6,
+ 55,
+ 14
+ ],
+ [
+ 11,
+ 19,
+ 43,
+ 57,
+ 56,
+ 37
+ ],
+ [
+ 10,
+ 24,
+ 60,
+ 21,
+ 5,
+ 17
+ ],
+ [
+ 31,
+ 20,
+ 63,
+ 44,
+ 55,
+ 33
+ ],
+ [
+ 7,
+ 25,
+ 40,
+ 32,
+ 54,
+ 58
+ ],
+ [
+ 49,
+ 30,
+ 13,
+ 50,
+ 55,
+ 28
+ ],
+ [
+ 30,
+ 3,
+ 51,
+ 61,
+ 0,
+ 16
+ ],
+ [
+ 57,
+ 46,
+ 43,
+ 40,
+ 30,
+ 22
+ ],
+ [
+ 52,
+ 33,
+ 51,
+ 19,
+ 27,
+ 29
+ ],
+ [
+ 14,
+ 4,
+ 40,
+ 41,
+ 0,
+ 34
+ ],
+ [
+ 4,
+ 13,
+ 8,
+ 61,
+ 47,
+ 9
+ ],
+ [
+ 31,
+ 27,
+ 43,
+ 1,
+ 3,
+ 0
+ ],
+ [
+ 8,
+ 4,
+ 46,
+ 9,
+ 3,
+ 50
+ ]
+ ]
]
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq/model_config.yaml
index d62d10db7c1..479cb7a4751 100644
--- a/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq/model_config.yaml
@@ -76,9 +76,11 @@ MODEL_ARGS:
--prompts: "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies."
--incoming-requests-per-sec: -1 # all requests arrive up front.
--inference-repeat-n: 8
- --inference-logging-step-interval: 1
- --sequence-parallel: true
-
+ --inference-dynamic-batching-buffer-size-gb: 20
+ --inference-dynamic-batching-max-requests: 16
+ --inference-logging-step-interval: 1
+ --moe-enable-routing-replay: true
METRICS:
- "generated_tokens"
- "logprobs"
+ - "routing_indices"
diff --git a/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq_suspend_resume/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq_suspend_resume/golden_values_dev_dgx_h100.json
new file mode 100644
index 00000000000..66c9e3e4121
--- /dev/null
+++ b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq_suspend_resume/golden_values_dev_dgx_h100.json
@@ -0,0 +1,158 @@
+{
+ "0": {
+ "input_prompt": "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies.",
+ "generated_text": " Wait for the moment when the music stops, and the lights come up, and the DJ says, \"I'm going to play a song for you",
+ "generated_tokens": [
+ 32844,
+ 1394,
+ 1278,
+ 4735,
+ 2200,
+ 1278,
+ 7146,
+ 30774,
+ 1044,
+ 1321,
+ 1278,
+ 26466,
+ 3930,
+ 2015,
+ 1044,
+ 1321,
+ 1278,
+ 30245,
+ 8223,
+ 1044,
+ 1429,
+ 1073,
+ 4525,
+ 4670,
+ 1317,
+ 3354,
+ 1261,
+ 6947,
+ 1394,
+ 1636
+ ],
+ "latency": 28.185462809633464,
+ "logprobs": [
+ -10.737512588500977,
+ -3.724862575531006,
+ -2.833397388458252,
+ -1.2464861869812012,
+ -0.2549239993095398,
+ -1.7607988119125366,
+ -2.419379711151123,
+ -1.9533929824829102,
+ -2.1014301776885986,
+ -6.169030666351318,
+ -0.8734959363937378,
+ -2.4733574390411377,
+ -3.4822516441345215,
+ -4.180896759033203,
+ -1.9767613410949707,
+ -1.8347630500793457,
+ -2.2581257820129395,
+ -7.180149078369141,
+ -0.0453881211578846,
+ -1.9841610193252563,
+ -5.015386581420898,
+ -8.827117919921875,
+ -9.885746002197266,
+ -0.8498678207397461,
+ -4.770059585571289,
+ -0.855280339717865,
+ -2.2494924068450928,
+ -0.017164958640933037,
+ -0.03715415671467781,
+ -3.4830124378204346,
+ -8.635110855102539,
+ -1.2520610094070435,
+ -6.62324857711792,
+ -3.639960765838623,
+ -3.664339542388916,
+ -4.182392597198486,
+ -2.1796066761016846,
+ -1.0725229978561401,
+ -0.26311880350112915,
+ -0.8036076426506042,
+ -4.6958818435668945,
+ -9.042495727539062,
+ -0.013647346757352352,
+ -3.1747794151306152,
+ -1.322129487991333,
+ -3.949110746383667,
+ -0.7829495072364807,
+ -0.002083513652905822,
+ -2.970266580581665,
+ -10.56244945526123,
+ -3.2369167804718018,
+ -1.1530492305755615,
+ -4.917466163635254,
+ -0.21241025626659393,
+ -0.06490474194288254,
+ -1.372581124305725,
+ -2.224682092666626,
+ -4.3847503662109375,
+ -0.36867555975914,
+ -4.035493850708008,
+ -0.39869019389152527,
+ -0.14373983442783356,
+ -2.716118812561035,
+ -10.687016487121582,
+ -0.04773370549082756,
+ -3.398231267929077,
+ -0.8646175265312195,
+ -4.74052619934082,
+ -0.23649944365024567,
+ -2.6610701084136963,
+ -0.8428961634635925,
+ -1.614527940750122,
+ -5.793307781219482,
+ -16.929147720336914,
+ -2.6586406230926514,
+ -0.1385982781648636,
+ -7.435610771179199,
+ -1.0483647584915161,
+ -2.1261863708496094,
+ -1.5261307954788208,
+ -0.27082547545433044,
+ -5.859070777893066,
+ -0.00648513063788414,
+ -7.732051849365234,
+ -2.712515354156494,
+ -2.9137418270111084,
+ -3.041210651397705,
+ -2.3559694290161133,
+ -0.3973437249660492,
+ -1.4338903427124023,
+ -2.2967660427093506,
+ -0.6096595525741577,
+ -1.3119444847106934,
+ -1.93257474899292,
+ -1.726539134979248,
+ -0.8397530317306519,
+ -0.5014236569404602,
+ -1.2989763021469116,
+ -1.5857150554656982,
+ -1.096572995185852,
+ -0.4009067416191101,
+ -0.43302634358406067,
+ -0.041601795703172684,
+ -1.285712718963623,
+ -2.214778184890747,
+ -2.6971933841705322,
+ -0.8101387619972229,
+ -0.43101266026496887,
+ -2.808060884475708,
+ -1.5226430892944336,
+ -1.6209226846694946,
+ -0.048716772347688675,
+ -1.3497682809829712,
+ -1.343377947807312,
+ -1.2755295038223267,
+ -1.2342015504837036,
+ -0.5394397377967834
+ ]
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq_suspend_resume/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq_suspend_resume/model_config.yaml
new file mode 100644
index 00000000000..1f302455440
--- /dev/null
+++ b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq_suspend_resume/model_config.yaml
@@ -0,0 +1,89 @@
+ENV_VARS:
+ CUDA_DEVICE_MAX_CONNECTIONS: 1
+ NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
+ NCCL_ALGO: Ring
+ CUBLAS_WORKSPACE_CONFIG: :4096:8
+TEST_TYPE: frozen-start
+MODE: inference
+MODEL_ARGS:
+ --log-num-zeros-in-grad: true
+ --log-validation-ppl-to-tensorboard: true
+ --log-timers-to-tensorboard: true
+ --log-memory-to-tensorboard: true
+ --timing-log-level: 0
+ --load: ${CHECKPOINT_LOAD_PATH}/model/deepseek_16b_pyt/dcp/mcore-v1_bf16/checkpoints
+ --tokenizer-model: ${CHECKPOINT_LOAD_PATH}/model/deepseek_16b_pyt/dcp/mcore-v1_bf16/multiMixV8.gpt4o_nc_sd.500000.128k.vocab.json
+ --tokenizer-type: TikTokenizer
+ --tiktoken-pattern: v2
+ --distributed-backend: nccl
+ --log-interval: 1
+ --transformer-impl: transformer_engine
+ --tensor-model-parallel-size: 4
+ --pipeline-model-parallel-size: 1
+ --expert-model-parallel-size: 8
+ --expert-tensor-parallel-size: 1
+ --sequence-parallel: true
+ --use-mcore-models: true
+ --moe-token-dispatcher-type: alltoall
+ --moe-grouped-gemm: true
+ --num-experts: 64
+ --moe-router-topk: 6
+ --moe-z-loss-coeff: 0
+ --moe-router-load-balancing-type: seq_aux_loss
+ --moe-aux-loss-coeff: 1e-3
+ --moe-router-score-function: sigmoid
+ --untie-embeddings-and-output-weights: true
+ --disable-bias-linear: true
+ --init-method-std: 0.014
+ --position-embedding-type: rope
+ --rotary-base: 1000000
+ --rotary-percent: 1.0
+ --num-layers: 27
+ --hidden-size: 2048
+ --moe-ffn-hidden-size: 1408
+ --moe-shared-expert-intermediate-size: 2816
+ --ffn-hidden-size: 10944
+ --num-attention-heads: 16
+ --kv-channels: 128
+ --normalization: RMSNorm
+ --swiglu: true
+ --attention-dropout: 0.0
+ --hidden-dropout: 0.0
+ --seq-length: 4096
+ --max-position-embeddings: 4096
+ --micro-batch-size: 1
+ --ckpt-format: torch_dist
+ --ckpt-fully-parallel-save: true
+ --ckpt-fully-parallel-load: true
+ --ckpt-assume-constant-structure: true
+ --dist-ckpt-strictness: log_unexpected
+ --bf16: true
+ --attention-backend: flash
+ --no-create-attention-mask-in-dataloader: true
+ --num-workers: 8
+ --use-checkpoint-args: true
+ --no-use-tokenizer-model-from-checkpoint-args: true
+ --no-load-optim: true
+ --deterministic-mode: true # moe will use different ops for determinism for inference
+ --save-interval: 2000
+ --temperature: 1.0
+ --top_k: 1
+ --return-log-probs: true
+ --num-tokens-to-generate: 30
+ --max-tokens-to-oom: 3600000
+ --inference-max-seq-length: 4096
+ --output-path: ${INFERENCE_OUTPUT_PATH}
+ --incoming-requests-per-sec: -1 # all requests arrive up front.
+ --inference-repeat-n: 8
+ --inference-dynamic-batching-buffer-size-gb: 20
+ --inference-dynamic-batching-max-requests: 16
+ --inference-logging-step-interval: 1
+ --moe-enable-routing-replay: true
+ --prompt-file: "./tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq_suspend_resume/prompts.json"
+ --suspend-resume-interval: 16
+ --suspend-timeout: 4.0
+ --rl-kv-cache-management-mode: recompute
+ --inference-dynamic-batching-unified-memory-level: 0
+ --no-rl-persist-cuda-graphs: true
+
+METRICS:
diff --git a/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq_suspend_resume/prompts.json b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq_suspend_resume/prompts.json
new file mode 100644
index 00000000000..7dcda2a3fa6
--- /dev/null
+++ b/tests/functional_tests/test_cases/moe/gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq_suspend_resume/prompts.json
@@ -0,0 +1,159 @@
+{"text": "The cat sat on the mat and"}
+{"text": "The sun rose in the east,"}
+{"text": "She opened the door and saw"}
+{"text": "He picked up the book,"}
+{"text": "Rain poured down all day,"}
+{"text": "The old house stood"}
+{"text": "Birds sang in the trees,"}
+{"text": "A car drove by,"}
+{"text": "The dog barked loudly,"}
+{"text": "The water was very"}
+{"text": "The capital of France is Paris. It is"}
+{"text": "Water boils at 100 degrees Celsius. This is"}
+{"text": "Humans have five senses: sight, sound, smell, taste, and"}
+{"text": "The Earth revolves around the Sun, taking approximately"}
+{"text": "A triangle has three sides. A square has"}
+{"text": "Computers process information using"}
+{"text": "The Amazon River is the"}
+{"text": "Photosynthesis is the process by which plants"}
+{"text": "Gravity is a force that"}
+{"text": "DNA stands for"}
+{"text": "Once upon a time, there was"}
+{"text": "All's well that ends"}
+{"text": "Every cloud has a silver"}
+{"text": "A stitch in time saves"}
+{"text": "Bite the bullet and"}
+{"text": "Cost an arm and a"}
+{"text": "Hit the road,"}
+{"text": "Let the cat out of the"}
+{"text": "Speak of the devil and"}
+{"text": "The early bird catches the"}
+{"text": "Monday, Tuesday, Wednesday,"}
+{"text": "1, 2, 3, 4,"}
+{"text": "Apple, Banana, Orange,"}
+{"text": "Red, Green, Blue,"}
+{"text": "North, South, East,"}
+{"text": "First, Second, Third,"}
+{"text": "A, B, C, D,"}
+{"text": "January, February, March,"}
+{"text": "Up, Down, Left,"}
+{"text": "Small, Medium, Large,"}
+{"text": "In a small village, nestled among the hills, lived"}
+{"text": "The detective peered through the misty window. Outside,"}
+{"text": "A sudden gust of wind swept through the forest,"}
+{"text": "She woke up with a start, realizing that"}
+{"text": "The ancient map, tattered and worn, showed"}
+{"text": "Far, far away, in a land of dragons and magic,"}
+{"text": "He hesitated at the crossroads, unsure which"}
+{"text": "The ship sailed smoothly across the calm sea,"}
+{"text": "A mysterious package arrived on her doorstep. Inside,"}
+{"text": "The old clock in the hall chimed midnight."}
+{"text": "The sky was a brilliant shade of"}
+{"text": "Her hair was long and flowing, like"}
+{"text": "The smell of freshly baked bread filled"}
+{"text": "The old, gnarled tree stood on the hill, its branches"}
+{"text": "The city lights twinkled in the distance,"}
+{"text": "His voice was deep and resonant,"}
+{"text": "The surface of the lake was smooth as"}
+{"text": "The vibrant colors of the sunset painted"}
+{"text": "The intricate pattern on the rug showed"}
+{"text": "The air was crisp and cool,"}
+{"text": "What do you call a group of"}
+{"text": "How long does it take to"}
+{"text": "Where can one find the most"}
+{"text": "Why is the sky"}
+{"text": "When did the first"}
+{"text": "Who was the last person to"}
+{"text": "Which way should we go to"}
+{"text": "What would happen if"}
+{"text": "How many stars are"}
+{"text": "Where does the river"}
+{"text": "`def main():`"}
+{"text": "``"}
+{"text": "{ \"name\": \"John\", \"age\":"}
+{"text": "`import os`"}
+{"text": "`SELECT * FROM users WHERE`"}
+{"text": "\"Hello?\" she whispered into the darkness. \"Is"}
+{"text": "He said, \"I don't know what to do.\" She replied, \"Well,"}
+{"text": "\"Can you pass the salt?\" asked Tom. Sarah reached for"}
+{"text": "\"Good morning,\" the shopkeeper greeted. \"How may I"}
+{"text": "\"I'm so tired,\" she sighed. \"Me too,\" he agreed, \"I just want to"}
+{"text": "Her laughter was music to"}
+{"text": "The city was a sleeping giant,"}
+{"text": "Time flew like an arrow,"}
+{"text": "His words were daggers,"}
+{"text": "The fog was a thick blanket,"}
+{"text": "The waves crashed like thunder,"}
+{"text": "Hope was a fragile bird,"}
+{"text": "The moon was a silver coin,"}
+{"text": "The wind whispered secrets,"}
+{"text": "Life is a journey, not"}
+{"text": "The quick brown fox jumps over"}
+{"text": "Once bitten, twice"}
+{"text": "As clear as day,"}
+{"text": "The more, the merrier."}
+{"text": "A penny for your"}
+{"text": "You can't have your cake and eat"}
+{"text": "The early bird gets the"}
+{"text": "Between a rock and a hard"}
+{"text": "Don't put all your eggs in one"}
+{"text": "When in Rome, do as the"}
+{"text": "Spring, Summer, Autumn,"}
+{"text": "Do, Re, Mi, Fa,"}
+{"text": "North America, South America, Europe,"}
+{"text": "Circle, Square, Triangle,"}
+{"text": "Tiny, Small, Medium,"}
+{"text": "The train arrived late, causing"}
+{"text": "She carefully placed the delicate vase on"}
+{"text": "He decided to take a long walk to clear"}
+{"text": "The old woman sat by the fire, knitting"}
+{"text": "The news spread quickly throughout the"}
+{"text": "The aroma of coffee filled the air, inviting"}
+{"text": "Beneath the surface of the calm water,"}
+{"text": "He adjusted his glasses and began to read"}
+{"text": "The sound of distant thunder rumbled,"}
+{"text": "The forgotten garden was overgrown with"}
+{"text": "The doctor examined the patient and prescribed"}
+{"text": "The chef prepared a delicious meal, using"}
+{"text": "The engineer designed a new bridge, considering"}
+{"text": "The artist painted a vibrant landscape, capturing"}
+{"text": "The scientist conducted an experiment, observing"}
+{"text": "A group of crows is called a"}
+{"text": "A parliament of owls is a"}
+{"text": "The opposite of hot is"}
+{"text": "The square root of 9 is"}
+{"text": "The longest river in Africa is the"}
+{"text": "The study found that regular exercise can"}
+{"text": "The theory of relativity was proposed by"}
+{"text": "The process of melting ice is called"}
+{"text": "Photosynthesis requires sunlight, water, and"}
+{"text": "The main ingredient in bread is"}
+{"text": "What is the primary function of the"}
+{"text": "How do you define the term"}
+{"text": "Where does the saying 'strike while the iron is hot' come from?"}
+{"text": "Why do leaves change color in the autumn?"}
+{"text": "When was the first personal computer"}
+{"text": "Who invented the light bulb?"}
+{"text": "Which planet is known as the Red Planet?"}
+{"text": "What happens if you mix acid with a base?"}
+{"text": "How many days are in a leap year?"}
+{"text": "Where do migratory birds go in the winter?"}
+{"text": "`function calculateArea(radius) {`"}
+{"text": "`# Python comments start with`"}
+{"text": "`CREATE TABLE employees (`"}
+{"text": "``"}
+{"text": "\"Excuse me,\" he coughed, \"but I think you've dropped something.\""}
+{"text": "She retorted, \"I'm not sure what you mean by that.\""}
+{"text": "\"It's freezing in here!\" he exclaimed. \"Can someone please\""}
+{"text": "The customer complained, \"This isn't what I ordered.\" The waiter replied,"}
+{"text": "\"Tell me about your day,\" she encouraged. He began,"}
+{"text": "His memory was a patchwork quilt of"}
+{"text": "The problem was a Gordian knot, impossible to"}
+{"text": "The news hit him like a ton of"}
+{"text": "Her words were a balm to his"}
+{"text": "The traffic moved at a snail's pace on"}
+{"text": "The silence in the room was deafening, a heavy"}
+{"text": "Opportunity knocked loudly on his"}
+{"text": "The city lights were a scatter of diamonds against"}
+{"text": "The mountain stood sentinel over the"}
+{"text": "Reading a good book is like taking a trip without"}
diff --git a/tests/functional_tests/test_cases/moe/gpt_grpo_tp8tp4_pp1_ep8ep2_dp8_throughputtest/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt_grpo_tp8tp4_pp1_ep8ep2_dp8_throughputtest/golden_values_dev_dgx_h100.json
index 1c13e432979..fdc5a1ae42b 100644
--- a/tests/functional_tests/test_cases/moe/gpt_grpo_tp8tp4_pp1_ep8ep2_dp8_throughputtest/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt_grpo_tp8tp4_pp1_ep8ep2_dp8_throughputtest/golden_values_dev_dgx_h100.json
@@ -16,11 +16,11 @@
"end_step": 5,
"step_interval": 1,
"values": {
- "1": 1116221440.0,
- "2": 1107565568.0,
- "3": 1116188160.0,
- "4": 1107525248.0,
- "5": 1116234624.0
+ "1": 622876864.0,
+ "2": 622889216.0,
+ "3": 623018176.0,
+ "4": 622966976.0,
+ "5": 623036672.0
}
},
"mem-allocated-bytes": {
@@ -28,11 +28,11 @@
"end_step": 5,
"step_interval": 1,
"values": {
- "1": 76683395072.0,
- "2": 76694667264.0,
- "3": 76694667264.0,
- "4": 76694667264.0,
- "5": 76694667264.0
+ "1": 76666388480.0,
+ "2": 76674662400.0,
+ "3": 76674662400.0,
+ "4": 76674662400.0,
+ "5": 76674662400.0
}
},
"mem-max-allocated-bytes": {
@@ -40,11 +40,11 @@
"end_step": 5,
"step_interval": 1,
"values": {
- "1": 76683403264.0,
- "2": 77029359616.0,
- "3": 77029900288.0,
- "4": 77030817792.0,
- "5": 77030817792.0
+ "1": 76666396672.0,
+ "2": 77010534400.0,
+ "3": 77014892544.0,
+ "4": 77014892544.0,
+ "5": 77014892544.0
}
},
"iteration-time": {
@@ -53,10 +53,10 @@
"step_interval": 1,
"values": {
"1": "nan",
- "2": 152.30721,
- "3": 105.10506,
- "4": 104.09995,
- "5": 102.75745
+ "2": 139.78513,
+ "3": 88.37674,
+ "4": 89.09523,
+ "5": 84.93382
}
}
-}
\ No newline at end of file
+}
diff --git a/tests/functional_tests/test_cases/moe/gpt_grpo_tp8tp4_pp1_ep8ep2_dp8_throughputtest/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt_grpo_tp8tp4_pp1_ep8ep2_dp8_throughputtest/model_config.yaml
index ed5d123892e..a334ce45edb 100644
--- a/tests/functional_tests/test_cases/moe/gpt_grpo_tp8tp4_pp1_ep8ep2_dp8_throughputtest/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt_grpo_tp8tp4_pp1_ep8ep2_dp8_throughputtest/model_config.yaml
@@ -1,5 +1,6 @@
ENV_VARS:
CUDA_DEVICE_MAX_CONNECTIONS: 1
+ THROUGHPUT_START_STEP: 1
NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0
NCCL_ALGO: Ring
CUBLAS_WORKSPACE_CONFIG: :4096:8
@@ -89,12 +90,11 @@ MODEL_ARGS:
--mock-data: true
--max-tokens-to-oom: 3600000
--inference-max-seq-length: 256
- --langrl-inference-server-type: inplace_megatron
--calculate-per-token-loss: true
--rl-use-sequence-packing: true
--rl-sequence-packing-algo: fifo
--rl-offload-optimizer-during-inference: true
- --rl-parallel-generation-tasks: 1
+ --rl-num-parallel-generations: 2
--cuda-graph-impl: local
--micro-batch-size: 1
--global-batch-size: 4
@@ -137,5 +137,3 @@ METRICS:
- "mem-allocated-bytes"
- "mem-max-allocated-bytes"
-THROUGHPUT_TEST_PARAMS:
- --start_step: 1
diff --git a/tests/functional_tests/test_cases/moe/gpt_static_inference_cuda_graphs_pad_tp4_pp1_ep4_16B_logitsmatch/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt_static_inference_cuda_graphs_pad_tp4_pp1_ep4_16B_logitsmatch/golden_values_dev_dgx_h100.json
index 0a32a2c875c..9b105e4fa2d 100644
--- a/tests/functional_tests/test_cases/moe/gpt_static_inference_cuda_graphs_pad_tp4_pp1_ep4_16B_logitsmatch/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt_static_inference_cuda_graphs_pad_tp4_pp1_ep4_16B_logitsmatch/golden_values_dev_dgx_h100.json
@@ -1 +1 @@
-{"1": {"input_prompt": "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies.", "generated_text": " Wait for the moment when the music stops, and the lights come up, and the DJ says, \"I'm going to play a song for you", "generated_tokens": [32844, 1394, 1278, 4735, 2200, 1278, 7146, 30774, 1044, 1321, 1278, 26466, 3930, 2015, 1044, 1321, 1278, 30245, 8223, 1044, 1429, 1073, 4525, 4670, 1317, 3354, 1261, 6947, 1394, 1636], "tpot": [0.4429202973842621, 0.017771262675523758, 0.01790841668844223, 0.016864703968167305, 0.016803359612822533, 0.016811424866318703, 0.016884256154298782, 0.016778528690338135, 0.016986047849059105, 0.016792479902505875, 0.016880257055163383, 0.016891231760382652, 0.016854560002684593, 0.01682988740503788, 0.016822686418890953, 0.01692502386868, 0.016792958602309227, 0.0169671680778265, 0.01693049632012844, 0.0170868169516325, 0.01686326414346695, 0.017535584047436714, 0.0174525436013937, 0.017324192449450493, 0.016914688050746918, 0.01683216169476509, 0.016909919679164886, 0.016884224489331245, 0.017001086845993996, 0.016849694773554802], "latency": 0.9421291537582874, "logprobs": [-10.737512588500977, -3.6907906532287598, -2.8354406356811523, -1.241209626197815, -0.23157396912574768, -1.8544931411743164, -2.305788278579712, -1.9660029411315918, -2.139838695526123, -6.157798767089844, -0.8562318086624146, -2.4214887619018555, -3.5043883323669434, -4.133810043334961, -2.009009838104248, -1.8133208751678467, -2.3323073387145996, -7.160175323486328, -0.040603119879961014, -1.9747259616851807, -5.076613903045654, -8.853288650512695, -9.848663330078125, -0.783089280128479, -4.775578022003174, -0.8501623868942261, -2.353159189224243, -0.019392186775803566, -0.035699184983968735, -3.369636058807373, -8.736637115478516, -1.2479770183563232, -6.668802261352539, -3.84025239944458, -3.774880886077881, -4.180184364318848, -2.2127902507781982, -1.07676362991333, -0.2321961224079132, -0.8445965647697449, -4.720583915710449, -9.144975662231445, -0.013737889938056469, -3.1713855266571045, -1.316046953201294, -3.976555824279785, -0.7929940223693848, -0.0020036876667290926, -2.9234514236450195, -10.630117416381836, -3.2423582077026367, -1.1527093648910522, -4.902451515197754, -0.20881010591983795, -0.06518254429101944, -1.3553434610366821, -2.205620765686035, -4.443068981170654, -0.3349221646785736, -4.0811614990234375, -0.40434733033180237, -0.14260707795619965, -2.7138302326202393, -10.61572551727295, -0.05091002210974693, -3.3788461685180664, -0.8990436792373657, -4.757172584533691, -0.2625967562198639, -2.6857080459594727, -0.8338347673416138, -1.5987446308135986, -5.796599388122559, -17.023239135742188, -2.5919642448425293, -0.1391627937555313, -7.425058841705322, -1.0969927310943604, -2.1373608112335205, -1.5555475950241089, -0.29913192987442017, -5.805688381195068, -0.006563534028828144, -7.741588592529297, -2.729809284210205, -2.989825487136841, -2.937342643737793, -2.452791690826416, -0.39692243933677673, -1.4191737174987793, -2.281113862991333, -0.6101264357566833, -1.3127052783966064, -1.93826162815094, -1.759519100189209, -0.8280774354934692, -0.48737525939941406, -1.2929327487945557, -1.4731515645980835, -1.0149478912353516, -0.402925580739975, -0.4662020802497864, -0.04289804771542549, -1.2809830904006958, -2.1367523670196533, -2.672316074371338, -0.832058310508728, -0.3975365459918976, -2.8649744987487793, -1.5586214065551758, -1.6164027452468872, -0.048774562776088715, -1.3553334474563599, -1.374987006187439, -1.2671791315078735, -1.29192054271698, -0.49132436513900757]}}
\ No newline at end of file
+{"1": {"input_prompt": "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies.", "generated_text": " Wait for the moment when the music stops, and the lights come up, and the DJ says, \"I'm going to play a song for you", "generated_tokens": [32844, 1394, 1278, 4735, 2200, 1278, 7146, 30774, 1044, 1321, 1278, 26466, 3930, 2015, 1044, 1321, 1278, 30245, 8223, 1044, 1429, 1073, 4525, 4670, 1317, 3354, 1261, 6947, 1394, 1636], "tpot": [0.9536803364753723, 0.016934143379330635, 0.016783935949206352, 0.016745438799262047, 0.0167548805475235, 0.01675168052315712, 0.016725631430745125, 0.016744224354624748, 0.016771871596574783, 0.01673772744834423, 0.01671331189572811, 0.016742143779993057, 0.016752831637859344, 0.016735071316361427, 0.016708416864275932, 0.01676086336374283, 0.01672108843922615, 0.01675747148692608, 0.01672768034040928, 0.01673593558371067, 0.01673644781112671, 0.016768928617239, 0.016721663996577263, 0.01674390397965908, 0.016715550795197487, 0.01673286408185959, 0.016974497586488724, 0.016759486868977547, 0.01675366424024105, 0.017026912420988083], "latency": 1.4477943610399961, "logprobs": [-10.737512588500977, -3.6907906532287598, -2.8354406356811523, -1.241209626197815, -0.23157396912574768, -1.8544931411743164, -2.3058578968048096, -1.965078592300415, -2.131650447845459, -6.1445770263671875, -0.8567591905593872, -2.4118287563323975, -3.49544095993042, -4.07368278503418, -1.9927215576171875, -1.8116189241409302, -2.220911979675293, -7.154321193695068, -0.041270580142736435, -1.897834300994873, -5.07651424407959, -8.796056747436523, -9.88559341430664, -0.7177769541740417, -4.775661468505859, -0.8519912362098694, -2.3280630111694336, -0.018436847254633904, -0.038715627044439316, -3.4831454753875732, -8.691339492797852, -1.2537559270858765, -6.647421836853027, -3.785327911376953, -3.7754249572753906, -4.1729302406311035, -2.268207550048828, -1.0762473344802856, -0.22336173057556152, -0.7773433923721313, -4.659046173095703, -9.07835865020752, -0.01364840567111969, -3.1569409370422363, -1.3136285543441772, -3.999577760696411, -0.8146029710769653, -0.0020667400676757097, -2.9257936477661133, -10.559369087219238, -3.301023483276367, -1.1468515396118164, -4.866663455963135, -0.20965954661369324, -0.06276518106460571, -1.3678232431411743, -2.2146267890930176, -4.369752883911133, -0.35328271985054016, -4.076470851898193, -0.39479735493659973, -0.14124885201454163, -2.7151336669921875, -10.645881652832031, -0.05100790411233902, -3.277879238128662, -0.859420657157898, -4.694356918334961, -0.26331964135169983, -2.601013660430908, -0.8340632319450378, -1.5944981575012207, -5.802148818969727, -16.992801666259766, -2.9630136489868164, -0.12174151837825775, -7.421735763549805, -1.090034008026123, -2.115244150161743, -1.592454195022583, -0.297377347946167, -5.607227325439453, -0.006619194056838751, -7.817281723022461, -2.701261043548584, -2.988292932510376, -2.9706215858459473, -2.4563350677490234, -0.3993130624294281, -1.5173310041427612, -2.269473075866699, -0.6111201047897339, -1.313757300376892, -1.9383023977279663, -1.6797527074813843, -0.7714957594871521, -0.49417543411254883, -1.2875804901123047, -1.5885818004608154, -1.018824577331543, -0.4005858302116394, -0.46394026279449463, -0.04470847547054291, -1.2914193868637085, -2.2410547733306885, -2.740159273147583, -0.7651359438896179, -0.4316181242465973, -2.7719383239746094, -1.5367236137390137, -1.652032732963562, -0.051836322993040085, -1.3689777851104736, -1.342658519744873, -1.26646089553833, -1.3113347291946411, -0.5160548686981201]}}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt_static_inference_cuda_graphs_pad_tp4_pp1_ep4_16B_logitsmatch/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt_static_inference_cuda_graphs_pad_tp4_pp1_ep4_16B_logitsmatch/model_config.yaml
index 549821afc8b..1c22a729f6e 100644
--- a/tests/functional_tests/test_cases/moe/gpt_static_inference_cuda_graphs_pad_tp4_pp1_ep4_16B_logitsmatch/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt_static_inference_cuda_graphs_pad_tp4_pp1_ep4_16B_logitsmatch/model_config.yaml
@@ -79,6 +79,7 @@ MODEL_ARGS:
--inference-max-seq-length: 4096
--inference-max-requests: 1
--output-path: ${INFERENCE_OUTPUT_PATH}
+ --use-legacy-static-engine: true
--prompts: "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies."
--incoming-requests-per-sec: -1
METRICS:
diff --git a/tests/functional_tests/test_cases/moe/gpt_static_inference_tp1_pp1_ep1_16B_logitsmatch/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt_static_inference_tp1_pp1_ep1_16B_logitsmatch/golden_values_dev_dgx_h100.json
index 0a088a3b4ed..16ce3cb46a0 100644
--- a/tests/functional_tests/test_cases/moe/gpt_static_inference_tp1_pp1_ep1_16B_logitsmatch/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt_static_inference_tp1_pp1_ep1_16B_logitsmatch/golden_values_dev_dgx_h100.json
@@ -1 +1 @@
-{"0": {"input_prompt": "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies.", "generated_text": " Wait for the moment when the music stops, and the lights come up, and the DJ says, \"I'm going to play a song for you", "generated_tokens": [32844, 1394, 1278, 4735, 2200, 1278, 7146, 30774, 1044, 1321, 1278, 26466, 3930, 2015, 1044, 1321, 1278, 30245, 8223, 1044, 1429, 1073, 4525, 4670, 1317, 3354, 1261, 6947, 1394, 1636], "tpot": [2.715181350708008, 11.115615844726562, 0.08171491324901581, 0.08067888021469116, 0.08026569336652756, 0.08201305568218231, 0.08297079056501389, 0.07990239560604095, 0.07923731207847595, 0.08088396489620209, 0.08342364430427551, 0.07902496308088303, 0.07883366197347641, 0.0821300819516182, 0.07879510521888733, 0.07889033854007721, 0.08096816390752792, 0.07890790700912476, 0.07898931205272675, 0.07891136407852173, 0.07872992008924484, 0.07891993969678879, 0.07908882945775986, 0.08219005167484283, 0.07928377389907837, 0.0791754499077797, 0.08204790204763412, 0.07909702509641647, 0.07994850724935532, 0.0790436789393425], "latency": 16.090769955015276, "logprobs": [-10.748703956604004, -3.675847053527832, -2.8152527809143066, -1.2499192953109741, -0.2585306465625763, -1.7650476694107056, -2.4413700103759766, -1.9855635166168213, -2.1556897163391113, -6.126346588134766, -0.8885424733161926, -2.466485023498535, -3.53129506111145, -4.1022443771362305, -1.973730444908142, -1.8129527568817139, -2.3135061264038086, -7.073224067687988, -0.0406799241900444, -1.9924827814102173, -5.044793128967285, -8.79849910736084, -9.896184921264648, -0.9244536757469177, -4.819119453430176, -0.8409886360168457, -2.3493337631225586, -0.019546041265130043, -0.03429899737238884, -3.486131429672241, -8.708669662475586, -1.2524677515029907, -6.648501396179199, -3.6543850898742676, -3.5817432403564453, -4.293689250946045, -2.213235855102539, -1.026153802871704, -0.22022850811481476, -0.7749938368797302, -4.7083001136779785, -9.260919570922852, -0.013350849971175194, -3.177624464035034, -1.3237272500991821, -3.991711139678955, -0.7711713314056396, -0.0020787552930414677, -2.9259750843048096, -10.556608200073242, -3.0338008403778076, -1.165448546409607, -4.884476184844971, -0.22491267323493958, -0.06299388408660889, -1.2974224090576172, -2.228250503540039, -4.375787258148193, -0.3615659773349762, -4.020719528198242, -0.3728649318218231, -0.16031591594219208, -2.7166409492492676, -10.650144577026367, -0.057426948100328445, -3.3819196224212646, -0.8289875388145447, -4.716109752655029, -0.2623739540576935, -2.6586318016052246, -0.846296489238739, -1.6911215782165527, -5.863524436950684, -17.074047088623047, -2.9786670207977295, -0.12697581946849823, -7.423051834106445, -1.1104215383529663, -2.125497579574585, -1.481943130493164, -0.26388564705848694, -5.852108001708984, -0.006604391150176525, -7.682407379150391, -2.7386088371276855, -2.9692039489746094, -3.0358991622924805, -2.434255838394165, -0.4008456766605377, -1.4501973390579224, -2.3068716526031494, -0.5563173294067383, -1.3114793300628662, -1.9436699151992798, -1.6950371265411377, -0.7694160342216492, -0.504065215587616, -1.2403564453125, -1.5687276124954224, -1.0141794681549072, -0.4076817035675049, -0.48037511110305786, -0.04258028045296669, -1.3669413328170776, -2.1299216747283936, -2.6870312690734863, -0.7604206800460815, -0.39242351055145264, -2.869314432144165, -1.464285135269165, -1.643430471420288, -0.04816753789782524, -1.4055166244506836, -1.3622899055480957, -1.2863339185714722, -1.2075212001800537, -0.5359172821044922]}}
\ No newline at end of file
+{"0": {"input_prompt": "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies.", "generated_text": " Wait for the moment when the music stops, and the lights come up, and the DJ says, \"I'm going to play a song for you", "generated_tokens": [32844, 1394, 1278, 4735, 2200, 1278, 7146, 30774, 1044, 1321, 1278, 26466, 3930, 2015, 1044, 1321, 1278, 30245, 8223, 1044, 1429, 1073, 4525, 4670, 1317, 3354, 1261, 6947, 1394, 1636], "tpot": [2.545225143432617, 0.8161460161209106, 0.08185821026563644, 0.08545549213886261, 0.0806993618607521, 0.08108274638652802, 0.08104605227708817, 0.08051212877035141, 0.08013814687728882, 0.08076290786266327, 0.08076870441436768, 0.08029650896787643, 0.07956188917160034, 0.07908589392900467, 0.07883257418870926, 0.0789686068892479, 0.07868800312280655, 0.0788155198097229, 0.07876496016979218, 0.07869625836610794, 0.07890035212039948, 0.08047452569007874, 0.08083260804414749, 0.08091676980257034, 0.08101955056190491, 0.08092495799064636, 0.08079087734222412, 0.08215625584125519, 0.08282729238271713, 0.08078553527593613], "latency": 5.632539719343185, "logprobs": [-10.748703956604004, -3.6647238731384277, -2.829481363296509, -1.2498102188110352, -0.2529161274433136, -1.8659480810165405, -2.381495237350464, -1.9656293392181396, -2.1487326622009277, -6.079752445220947, -0.854481041431427, -2.4210457801818848, -3.5035147666931152, -4.105377674102783, -1.9736212491989136, -1.812272310256958, -2.2293882369995117, -7.158683776855469, -0.04035309702157974, -1.8912277221679688, -5.030168056488037, -8.83056640625, -9.887261390686035, -0.783703088760376, -4.775330066680908, -0.8503050208091736, -2.3185184001922607, -0.019133294001221657, -0.035927604883909225, -3.367396831512451, -8.677278518676758, -1.249742031097412, -6.646797180175781, -3.7622885704040527, -3.5719683170318604, -4.217543125152588, -2.1486034393310547, -1.0877041816711426, -0.21807751059532166, -0.779091477394104, -4.703271865844727, -9.254538536071777, -0.013438244350254536, -3.1536498069763184, -1.347169041633606, -3.951521873474121, -0.7524824142456055, -0.0020295039284974337, -2.977275848388672, -10.547475814819336, -3.2425384521484375, -1.1805994510650635, -4.853910446166992, -0.20160463452339172, -0.06345974653959274, -1.3051351308822632, -2.185082197189331, -4.38629150390625, -0.36718395352363586, -4.047024726867676, -0.35912153124809265, -0.14624275267124176, -2.715745687484741, -10.57785415649414, -0.057624127715826035, -3.278026819229126, -0.8596221804618835, -4.68856954574585, -0.2627037465572357, -2.6609294414520264, -0.8223639726638794, -1.599161148071289, -5.80098295211792, -16.973386764526367, -2.616450548171997, -0.13072170317173004, -7.462261199951172, -1.082613229751587, -2.122760534286499, -1.5544897317886353, -0.2857922911643982, -5.874238967895508, -0.006397482007741928, -7.670827865600586, -2.7104744911193848, -2.9125661849975586, -3.0234169960021973, -2.3567330837249756, -0.39910370111465454, -1.4856780767440796, -2.293515920639038, -0.6103871464729309, -1.3627440929412842, -1.9288791418075562, -1.683215618133545, -0.7717607021331787, -0.5037432312965393, -1.2485487461090088, -1.559265375137329, -1.0295413732528687, -0.4404847025871277, -0.46411699056625366, -0.04514020308852196, -1.3507211208343506, -2.1256327629089355, -2.687516927719116, -0.759257435798645, -0.41867509484291077, -2.854245185852051, -1.5189608335494995, -1.5710458755493164, -0.05560074374079704, -1.357395052909851, -1.3819431066513062, -1.2458440065383911, -1.2788543701171875, -0.5129148364067078]}}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt_static_inference_tp1_pp1_ep1_16B_logitsmatch/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt_static_inference_tp1_pp1_ep1_16B_logitsmatch/model_config.yaml
index 4934fe6c913..03895d97ee9 100644
--- a/tests/functional_tests/test_cases/moe/gpt_static_inference_tp1_pp1_ep1_16B_logitsmatch/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt_static_inference_tp1_pp1_ep1_16B_logitsmatch/model_config.yaml
@@ -72,6 +72,7 @@ MODEL_ARGS:
--max-tokens-to-oom: 3600000
--inference-max-seq-length: 4096
--output-path: ${INFERENCE_OUTPUT_PATH}
+ --use-legacy-static-engine: true
--prompts: "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies."
--incoming-requests-per-sec: -1 # all requests arrive up front.
--inference-dynamic-batching-buffer-size-gb: 20
diff --git a/tests/functional_tests/test_cases/moe/gpt_static_inference_tp4_pp1_ep4_16B_logitsmatch/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/moe/gpt_static_inference_tp4_pp1_ep4_16B_logitsmatch/golden_values_dev_dgx_h100.json
index 46a451d0b64..9374ead29ab 100644
--- a/tests/functional_tests/test_cases/moe/gpt_static_inference_tp4_pp1_ep4_16B_logitsmatch/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/moe/gpt_static_inference_tp4_pp1_ep4_16B_logitsmatch/golden_values_dev_dgx_h100.json
@@ -1 +1 @@
-{"0": {"input_prompt": "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies.", "generated_text": " Wait for the moment when the music stops, and the lights come up, and the DJ says, \"I'm going to play a song for you", "generated_tokens": [32844, 1394, 1278, 4735, 2200, 1278, 7146, 30774, 1044, 1321, 1278, 26466, 3930, 2015, 1044, 1321, 1278, 30245, 8223, 1044, 1429, 1073, 4525, 4670, 1317, 3354, 1261, 6947, 1394, 1636], "tpot": [32.35166549682617, 0.833970844745636, 0.09404217451810837, 0.09242991358041763, 0.09451283514499664, 0.09116563200950623, 0.0916728675365448, 0.09169203042984009, 0.0920996442437172, 0.09247440099716187, 0.09316505491733551, 0.09183433651924133, 0.09311366081237793, 0.0922863706946373, 0.09139427542686462, 0.0916166678071022, 0.09881363064050674, 0.09086793661117554, 0.09085418283939362, 0.0913468450307846, 0.0913306251168251, 0.09422652423381805, 0.09134646505117416, 0.09283513575792313, 0.09112297743558884, 0.09120230376720428, 0.09097100794315338, 0.09246265143156052, 0.09317846596240997, 0.09015017747879028], "latency": 35.78577698091976, "logprobs": [-10.737512588500977, -3.6907906532287598, -2.8354406356811523, -1.241209626197815, -0.23157396912574768, -1.8544931411743164, -2.305788278579712, -1.9660029411315918, -2.139838695526123, -6.157798767089844, -0.8562318086624146, -2.4214887619018555, -3.5043883323669434, -4.133810043334961, -2.009009838104248, -1.8133208751678467, -2.3323073387145996, -7.160175323486328, -0.040603119879961014, -1.9747259616851807, -5.076613903045654, -8.853288650512695, -9.848663330078125, -0.783089280128479, -4.775578022003174, -0.8501623868942261, -2.353159189224243, -0.019392186775803566, -0.035699184983968735, -3.369636058807373, -8.736637115478516, -1.2479770183563232, -6.668802261352539, -3.84025239944458, -3.774880886077881, -4.180184364318848, -2.2127902507781982, -1.07676362991333, -0.2321961224079132, -0.8445965647697449, -4.720583915710449, -9.144975662231445, -0.013737889938056469, -3.1713855266571045, -1.316046953201294, -3.976555824279785, -0.7929940223693848, -0.0020036876667290926, -2.9234514236450195, -10.630117416381836, -3.2423582077026367, -1.1527093648910522, -4.902451515197754, -0.20881010591983795, -0.06518254429101944, -1.3553434610366821, -2.205620765686035, -4.443068981170654, -0.3349221646785736, -4.0811614990234375, -0.40434733033180237, -0.14260707795619965, -2.7138302326202393, -10.61572551727295, -0.05091002210974693, -3.3788461685180664, -0.8990436792373657, -4.757172584533691, -0.2625967562198639, -2.6857080459594727, -0.8338347673416138, -1.5987446308135986, -5.796599388122559, -17.023239135742188, -2.5919642448425293, -0.1391627937555313, -7.425058841705322, -1.0969927310943604, -2.1373608112335205, -1.5555475950241089, -0.29913192987442017, -5.805688381195068, -0.006563534028828144, -7.741588592529297, -2.729809284210205, -2.989825487136841, -2.937342643737793, -2.452791690826416, -0.39692243933677673, -1.4191737174987793, -2.281113862991333, -0.6101264357566833, -1.3127052783966064, -1.93826162815094, -1.759519100189209, -0.8280774354934692, -0.48737525939941406, -1.2929327487945557, -1.4731515645980835, -1.0149478912353516, -0.402925580739975, -0.4662020802497864, -0.04289804771542549, -1.2809830904006958, -2.1367523670196533, -2.672316074371338, -0.832058310508728, -0.3975365459918976, -2.8649744987487793, -1.5586214065551758, -1.6164027452468872, -0.048774562776088715, -1.3553334474563599, -1.374987006187439, -1.2671791315078735, -1.29192054271698, -0.49132436513900757]}}
\ No newline at end of file
+{"0": {"input_prompt": "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies.", "generated_text": " Wait for the moment when the music stops, and the lights come up, and the DJ says, \"I'm going to play a song for you", "generated_tokens": [32844, 1394, 1278, 4735, 2200, 1278, 7146, 30774, 1044, 1321, 1278, 26466, 3930, 2015, 1044, 1321, 1278, 30245, 8223, 1044, 1429, 1073, 4525, 4670, 1317, 3354, 1261, 6947, 1394, 1636], "tpot": [13.711557388305664, 0.9041259288787842, 0.09681683033704758, 0.09153660386800766, 0.09111235290765762, 0.09335695952177048, 0.09111097455024719, 0.09149472415447235, 0.09076278656721115, 0.09109959006309509, 0.09105174243450165, 0.09161145985126495, 0.09026294946670532, 0.09116349369287491, 0.09077664464712143, 0.09046704322099686, 0.09124940633773804, 0.09130454063415527, 0.09034591913223267, 0.09066786617040634, 0.0906708836555481, 0.09116076678037643, 0.09075567871332169, 0.09169074892997742, 0.0911671370267868, 0.09112399816513062, 0.09118761122226715, 0.09077664464712143, 0.09082793444395065, 0.09155046194791794], "latency": 17.191568877082318, "logprobs": [-10.737512588500977, -3.6907906532287598, -2.8354406356811523, -1.241209626197815, -0.23157396912574768, -1.8544931411743164, -2.3058578968048096, -1.965078592300415, -2.131650447845459, -6.1445770263671875, -0.8567591905593872, -2.4118287563323975, -3.49544095993042, -4.07368278503418, -1.9927215576171875, -1.8116189241409302, -2.220911979675293, -7.154321193695068, -0.041270580142736435, -1.897834300994873, -5.07651424407959, -8.796056747436523, -9.88559341430664, -0.7177769541740417, -4.775661468505859, -0.8519912362098694, -2.3280630111694336, -0.018436847254633904, -0.038715627044439316, -3.4831454753875732, -8.691339492797852, -1.2537559270858765, -6.647421836853027, -3.785327911376953, -3.7754249572753906, -4.1729302406311035, -2.268207550048828, -1.0762473344802856, -0.22336173057556152, -0.7773433923721313, -4.659046173095703, -9.07835865020752, -0.01364840567111969, -3.1569409370422363, -1.3136285543441772, -3.999577760696411, -0.8146029710769653, -0.0020667400676757097, -2.9257936477661133, -10.559369087219238, -3.301023483276367, -1.1468515396118164, -4.866663455963135, -0.20965954661369324, -0.06276518106460571, -1.3678232431411743, -2.2146267890930176, -4.369752883911133, -0.35328271985054016, -4.076470851898193, -0.39479735493659973, -0.14124885201454163, -2.7151336669921875, -10.645881652832031, -0.05100790411233902, -3.277879238128662, -0.859420657157898, -4.694356918334961, -0.26331964135169983, -2.601013660430908, -0.8340632319450378, -1.5944981575012207, -5.802148818969727, -16.992801666259766, -2.9630136489868164, -0.12174151837825775, -7.421735763549805, -1.090034008026123, -2.115244150161743, -1.592454195022583, -0.297377347946167, -5.607227325439453, -0.006619194056838751, -7.817281723022461, -2.701261043548584, -2.988292932510376, -2.9706215858459473, -2.4563350677490234, -0.3993130624294281, -1.5173310041427612, -2.269473075866699, -0.6111201047897339, -1.313757300376892, -1.9383023977279663, -1.6797527074813843, -0.7714957594871521, -0.49417543411254883, -1.2875804901123047, -1.5885818004608154, -1.018824577331543, -0.4005858302116394, -0.46394026279449463, -0.04470847547054291, -1.2914193868637085, -2.2410547733306885, -2.740159273147583, -0.7651359438896179, -0.4316181242465973, -2.7719383239746094, -1.5367236137390137, -1.652032732963562, -0.051836322993040085, -1.3689777851104736, -1.342658519744873, -1.26646089553833, -1.3113347291946411, -0.5160548686981201]}}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/moe/gpt_static_inference_tp4_pp1_ep4_16B_logitsmatch/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt_static_inference_tp4_pp1_ep4_16B_logitsmatch/model_config.yaml
index 69c0db980b0..9259d63c9d1 100644
--- a/tests/functional_tests/test_cases/moe/gpt_static_inference_tp4_pp1_ep4_16B_logitsmatch/model_config.yaml
+++ b/tests/functional_tests/test_cases/moe/gpt_static_inference_tp4_pp1_ep4_16B_logitsmatch/model_config.yaml
@@ -74,6 +74,7 @@ MODEL_ARGS:
--max-tokens-to-oom: 3600000
--inference-max-seq-length: 4096
--output-path: ${INFERENCE_OUTPUT_PATH}
+ --use-legacy-static-engine: true
--prompts: "Time travel to 2008, and go to a bar or a club or one of the myriad disco-basements on the Lower East Side that does not quite know which of those it is. Dance awkwardly in a room full of other glittered-up nerds, and wait for something to happen, buoyed on the feeling that this is the big swollen heart of life, that this is New York like the movies."
--incoming-requests-per-sec: -1 # all requests arrive up front.
METRICS:
diff --git a/tests/functional_tests/test_cases/multimodal-llava/multimodal_llava_mcore_te_tp1_pp1/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/multimodal-llava/multimodal_llava_mcore_te_tp1_pp1/golden_values_dev_dgx_h100.json
index f4357530aed..00ed37733ae 100644
--- a/tests/functional_tests/test_cases/multimodal-llava/multimodal_llava_mcore_te_tp1_pp1/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/multimodal-llava/multimodal_llava_mcore_te_tp1_pp1/golden_values_dev_dgx_h100.json
@@ -6,53 +6,53 @@
"values": {
"1": 9.14877,
"2": 9.15171,
- "3": 9.14691,
- "4": 9.15346,
+ "3": 9.1469,
+ "4": 9.15345,
"5": 9.15057,
- "6": 9.14683,
+ "6": 9.14688,
"7": 9.14378,
"8": 9.14363,
- "9": 9.15069,
- "10": 9.15231,
- "11": 9.14609,
+ "9": 9.15072,
+ "10": 9.15239,
+ "11": 9.14608,
"12": 9.14125,
- "13": 9.1414,
- "14": 9.14248,
- "15": 9.13419,
- "16": 9.12601,
- "17": 9.12407,
- "18": 9.12053,
- "19": 9.11789,
- "20": 9.09777,
- "21": 9.06948,
- "22": 9.06985,
- "23": 9.07079,
- "24": 9.06043,
- "25": 9.05505,
- "26": 9.05713,
+ "13": 9.14146,
+ "14": 9.14247,
+ "15": 9.13422,
+ "16": 9.12606,
+ "17": 9.12413,
+ "18": 9.12057,
+ "19": 9.1179,
+ "20": 9.09773,
+ "21": 9.0695,
+ "22": 9.06984,
+ "23": 9.07077,
+ "24": 9.06045,
+ "25": 9.05509,
+ "26": 9.05714,
"27": 9.04089,
"28": 9.0186,
- "29": 9.00353,
- "30": 8.99697,
- "31": 8.99484,
- "32": 8.98416,
- "33": 8.97763,
- "34": 8.98617,
- "35": 8.94993,
- "36": 8.94557,
- "37": 8.92133,
- "38": 8.94104,
- "39": 8.92482,
- "40": 8.87122,
- "41": 8.89627,
- "42": 8.87601,
+ "29": 9.00351,
+ "30": 8.99698,
+ "31": 8.9948,
+ "32": 8.98417,
+ "33": 8.97761,
+ "34": 8.9862,
+ "35": 8.94992,
+ "36": 8.9456,
+ "37": 8.92135,
+ "38": 8.94106,
+ "39": 8.92485,
+ "40": 8.87125,
+ "41": 8.89626,
+ "42": 8.87604,
"43": 8.87414,
- "44": 8.8411,
- "45": 8.81228,
- "46": 8.79564,
+ "44": 8.84111,
+ "45": 8.81225,
+ "46": 8.79568,
"47": 8.84576,
- "48": 8.77191,
- "49": 8.78047,
+ "48": 8.77194,
+ "49": 8.78043,
"50": 8.76196
}
},
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 3477955.0,
- "2": 3392302.0,
- "3": 3630021.0,
- "4": 3532452.0,
- "5": 3783960.0,
- "6": 3584449.0,
- "7": 3478372.0,
- "8": 3414330.0,
- "9": 3511649.0,
- "10": 3544311.0,
- "11": 3475468.0,
- "12": 3518965.0,
- "13": 3591786.0,
- "14": 3549396.0,
- "15": 3421163.0,
- "16": 3383319.0,
- "17": 3424120.0,
- "18": 3509184.0,
- "19": 3426107.0,
- "20": 3465915.0,
- "21": 3700118.0,
- "22": 3474397.0,
- "23": 3693474.0,
- "24": 3405657.0,
- "25": 3457588.0,
- "26": 3479130.0,
- "27": 3555371.0,
- "28": 3496999.0,
- "29": 3561842.0,
- "30": 3708011.0,
- "31": 3397663.0,
- "32": 3467970.0,
- "33": 3515742.0,
- "34": 3501589.0,
- "35": 3432484.0,
- "36": 3453953.0,
- "37": 3958777.0,
- "38": 3488640.0,
- "39": 3409958.0,
- "40": 3614258.0,
- "41": 3425709.0,
- "42": 3643603.0,
- "43": 3473029.0,
- "44": 3448331.0,
- "45": 3452202.0,
- "46": 3585738.0,
- "47": 3467386.0,
- "48": 3462962.0,
- "49": 3529813.0,
- "50": 3412019.0
+ "1": 3478044.0,
+ "2": 3392170.0,
+ "3": 3630100.0,
+ "4": 3532464.0,
+ "5": 3783909.0,
+ "6": 3584604.0,
+ "7": 3478292.0,
+ "8": 3414214.0,
+ "9": 3511551.0,
+ "10": 3544353.0,
+ "11": 3475513.0,
+ "12": 3519004.0,
+ "13": 3591792.0,
+ "14": 3549575.0,
+ "15": 3421322.0,
+ "16": 3383312.0,
+ "17": 3424142.0,
+ "18": 3509310.0,
+ "19": 3426210.0,
+ "20": 3465844.0,
+ "21": 3699866.0,
+ "22": 3474417.0,
+ "23": 3693512.0,
+ "24": 3405590.0,
+ "25": 3457789.0,
+ "26": 3479283.0,
+ "27": 3555496.0,
+ "28": 3497078.0,
+ "29": 3561734.0,
+ "30": 3708144.0,
+ "31": 3397570.0,
+ "32": 3467832.0,
+ "33": 3515682.0,
+ "34": 3501518.0,
+ "35": 3432575.0,
+ "36": 3454076.0,
+ "37": 3958864.0,
+ "38": 3488540.0,
+ "39": 3410013.0,
+ "40": 3614392.0,
+ "41": 3425670.0,
+ "42": 3643700.0,
+ "43": 3472723.0,
+ "44": 3448423.0,
+ "45": 3452103.0,
+ "46": 3585686.0,
+ "47": 3467299.0,
+ "48": 3462916.0,
+ "49": 3529603.0,
+ "50": 3411958.0
}
},
"mem-allocated-bytes": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 21.47107,
- "2": 0.21426,
- "3": 0.18485,
- "4": 0.1655,
- "5": 0.16764,
- "6": 0.16482,
- "7": 0.16761,
- "8": 0.16451,
- "9": 0.16762,
- "10": 0.16536,
- "11": 0.17999,
- "12": 0.18657,
- "13": 0.16983,
- "14": 0.16676,
- "15": 0.16908,
- "16": 0.16963,
- "17": 0.17346,
- "18": 0.17019,
- "19": 0.17052,
- "20": 0.17018,
- "21": 0.16541,
- "22": 0.16566,
- "23": 0.16521,
- "24": 0.16662,
- "25": 0.16493,
- "26": 0.16377,
- "27": 0.16515,
- "28": 0.16469,
- "29": 0.16683,
- "30": 0.16435,
- "31": 0.1697,
- "32": 0.16472,
- "33": 0.1693,
- "34": 0.16637,
- "35": 0.16593,
- "36": 0.16439,
- "37": 0.16693,
- "38": 0.16653,
- "39": 0.16645,
- "40": 0.16669,
- "41": 0.16547,
- "42": 0.16438,
- "43": 0.16787,
- "44": 0.16848,
- "45": 0.16631,
- "46": 0.16902,
- "47": 0.16588,
- "48": 0.16644,
- "49": 0.16691,
- "50": 0.1671
+ "1": "nan",
+ "2": 4.3752,
+ "3": 0.18438,
+ "4": 0.17408,
+ "5": 0.17426,
+ "6": 0.17324,
+ "7": 0.17437,
+ "8": 0.16591,
+ "9": 0.16608,
+ "10": 0.16563,
+ "11": 0.16603,
+ "12": 0.1654,
+ "13": 0.16746,
+ "14": 0.16557,
+ "15": 0.16692,
+ "16": 0.16648,
+ "17": 0.16679,
+ "18": 0.1661,
+ "19": 0.16846,
+ "20": 0.17421,
+ "21": 0.16648,
+ "22": 0.16578,
+ "23": 0.16601,
+ "24": 0.16672,
+ "25": 0.16647,
+ "26": 0.16552,
+ "27": 0.16788,
+ "28": 0.16751,
+ "29": 0.1673,
+ "30": 0.1667,
+ "31": 0.16952,
+ "32": 0.16693,
+ "33": 0.16603,
+ "34": 0.17134,
+ "35": 0.16658,
+ "36": 0.16674,
+ "37": 0.16801,
+ "38": 0.16643,
+ "39": 0.1679,
+ "40": 0.16548,
+ "41": 0.16743,
+ "42": 0.16511,
+ "43": 0.16745,
+ "44": 0.16536,
+ "45": 0.16772,
+ "46": 0.16513,
+ "47": 0.16764,
+ "48": 0.16926,
+ "49": 0.16982,
+ "50": 0.16849
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/multimodal-llava/multimodal_llava_mcore_te_tp4_sp_cp2/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/multimodal-llava/multimodal_llava_mcore_te_tp4_sp_cp2/golden_values_dev_dgx_h100.json
index b0c23087659..8fb704f4745 100644
--- a/tests/functional_tests/test_cases/multimodal-llava/multimodal_llava_mcore_te_tp4_sp_cp2/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/multimodal-llava/multimodal_llava_mcore_te_tp4_sp_cp2/golden_values_dev_dgx_h100.json
@@ -4,56 +4,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 9.28644,
- "2": 9.28396,
- "3": 9.28076,
- "4": 9.28856,
- "5": 9.27699,
- "6": 9.28726,
- "7": 9.27831,
- "8": 9.28266,
- "9": 9.28518,
- "10": 9.28294,
- "11": 9.28326,
- "12": 9.27377,
- "13": 9.27113,
- "14": 9.27209,
- "15": 9.25297,
- "16": 9.24499,
- "17": 9.24857,
- "18": 9.2295,
- "19": 9.23151,
- "20": 9.20818,
- "21": 9.1704,
+ "1": 9.28648,
+ "2": 9.28389,
+ "3": 9.2807,
+ "4": 9.2885,
+ "5": 9.27705,
+ "6": 9.28737,
+ "7": 9.27833,
+ "8": 9.2826,
+ "9": 9.28528,
+ "10": 9.28289,
+ "11": 9.28325,
+ "12": 9.27371,
+ "13": 9.27127,
+ "14": 9.27213,
+ "15": 9.253,
+ "16": 9.24492,
+ "17": 9.24859,
+ "18": 9.22958,
+ "19": 9.23149,
+ "20": 9.20816,
+ "21": 9.17058,
"22": 9.15059,
- "23": 9.16837,
- "24": 9.15073,
- "25": 9.14424,
- "26": 9.14738,
- "27": 9.12308,
- "28": 9.09717,
- "29": 9.09386,
- "30": 9.07826,
- "31": 8.97181,
- "32": 9.0315,
- "33": 9.02023,
- "34": 8.98663,
- "35": 8.95928,
- "36": 8.97134,
- "37": 8.91442,
+ "23": 9.16841,
+ "24": 9.15088,
+ "25": 9.14428,
+ "26": 9.14731,
+ "27": 9.12298,
+ "28": 9.09703,
+ "29": 9.09381,
+ "30": 9.07824,
+ "31": 8.97182,
+ "32": 9.03154,
+ "33": 9.02015,
+ "34": 8.98673,
+ "35": 8.95912,
+ "36": 8.97146,
+ "37": 8.91452,
"38": 8.88791,
- "39": 8.88879,
- "40": 8.90639,
- "41": 8.81803,
- "42": 8.87405,
- "43": 8.85655,
- "44": 8.81693,
- "45": 8.81356,
- "46": 8.84453,
- "47": 8.73701,
- "48": 8.66923,
- "49": 8.70104,
- "50": 8.73489
+ "39": 8.88878,
+ "40": 8.90648,
+ "41": 8.81814,
+ "42": 8.87399,
+ "43": 8.85661,
+ "44": 8.81711,
+ "45": 8.8137,
+ "46": 8.84467,
+ "47": 8.73715,
+ "48": 8.66933,
+ "49": 8.70117,
+ "50": 8.73514
}
},
"num-zeros": {
@@ -61,56 +61,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 5959428.0,
- "2": 6553739.0,
- "3": 7313558.0,
- "4": 6377212.0,
- "5": 6498220.0,
- "6": 7152015.0,
- "7": 6210260.0,
- "8": 6334672.0,
- "9": 6624655.0,
- "10": 6529106.0,
- "11": 7466660.0,
- "12": 6471717.0,
- "13": 6003465.0,
- "14": 8072041.0,
- "15": 6529968.0,
- "16": 7526852.0,
- "17": 6035134.0,
- "18": 6289690.0,
- "19": 6162498.0,
- "20": 6527712.0,
- "21": 6981897.0,
- "22": 7132920.0,
- "23": 5928645.0,
- "24": 6210340.0,
- "25": 6993116.0,
- "26": 6471329.0,
- "27": 6355333.0,
- "28": 6876968.0,
- "29": 6380137.0,
- "30": 6468615.0,
- "31": 8165212.0,
- "32": 6765571.0,
- "33": 6355561.0,
- "34": 6662287.0,
- "35": 7065313.0,
- "36": 6076925.0,
- "37": 7785462.0,
- "38": 6727049.0,
- "39": 7315988.0,
- "40": 6555018.0,
- "41": 7314645.0,
- "42": 6591992.0,
- "43": 6928020.0,
- "44": 7274444.0,
- "45": 6680179.0,
- "46": 6232560.0,
- "47": 6496796.0,
- "48": 6809653.0,
- "49": 6753531.0,
- "50": 6238141.0
+ "1": 5959426.0,
+ "2": 6553765.0,
+ "3": 7313448.0,
+ "4": 6377019.0,
+ "5": 6498265.0,
+ "6": 7151976.0,
+ "7": 6210453.0,
+ "8": 6334691.0,
+ "9": 6624555.0,
+ "10": 6529053.0,
+ "11": 7466628.0,
+ "12": 6471518.0,
+ "13": 6003450.0,
+ "14": 8071967.0,
+ "15": 6529964.0,
+ "16": 7526726.0,
+ "17": 6035087.0,
+ "18": 6289754.0,
+ "19": 6162432.0,
+ "20": 6527695.0,
+ "21": 6981984.0,
+ "22": 7132788.0,
+ "23": 5928504.0,
+ "24": 6210076.0,
+ "25": 6993073.0,
+ "26": 6471296.0,
+ "27": 6355325.0,
+ "28": 6877023.0,
+ "29": 6380286.0,
+ "30": 6468637.0,
+ "31": 8165049.0,
+ "32": 6765693.0,
+ "33": 6355607.0,
+ "34": 6662298.0,
+ "35": 7065050.0,
+ "36": 6077046.0,
+ "37": 7785469.0,
+ "38": 6727037.0,
+ "39": 7315675.0,
+ "40": 6555109.0,
+ "41": 7314543.0,
+ "42": 6591977.0,
+ "43": 6927941.0,
+ "44": 7274324.0,
+ "45": 6680295.0,
+ "46": 6232304.0,
+ "47": 6496696.0,
+ "48": 6809772.0,
+ "49": 6753632.0,
+ "50": 6238092.0
}
},
"mem-allocated-bytes": {
@@ -118,56 +118,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1653821440.0,
- "2": 1653821440.0,
- "3": 1653821440.0,
- "4": 1653821440.0,
- "5": 1653821440.0,
- "6": 1653821440.0,
- "7": 1653821440.0,
- "8": 1653821440.0,
- "9": 1653821440.0,
- "10": 1653821440.0,
- "11": 1653821440.0,
- "12": 1653821440.0,
- "13": 1653821440.0,
- "14": 1653821440.0,
- "15": 1653821440.0,
- "16": 1653821440.0,
- "17": 1653821440.0,
- "18": 1653821440.0,
- "19": 1653821440.0,
- "20": 1653821440.0,
- "21": 1653821440.0,
- "22": 1653821440.0,
- "23": 1653821440.0,
- "24": 1653821440.0,
- "25": 1653821440.0,
- "26": 1653821440.0,
- "27": 1653821440.0,
- "28": 1653821440.0,
- "29": 1653821440.0,
- "30": 1653821440.0,
- "31": 1653821440.0,
- "32": 1653821440.0,
- "33": 1653821440.0,
- "34": 1653821440.0,
- "35": 1653821440.0,
- "36": 1653821440.0,
- "37": 1653821440.0,
- "38": 1653821440.0,
- "39": 1653821440.0,
- "40": 1653821440.0,
- "41": 1653821440.0,
- "42": 1653821440.0,
- "43": 1653821440.0,
- "44": 1653821440.0,
- "45": 1653821440.0,
- "46": 1653821440.0,
- "47": 1653821440.0,
- "48": 1653821440.0,
- "49": 1653821440.0,
- "50": 1653821440.0
+ "1": 1650282496.0,
+ "2": 1650282496.0,
+ "3": 1650282496.0,
+ "4": 1650282496.0,
+ "5": 1650282496.0,
+ "6": 1650282496.0,
+ "7": 1650282496.0,
+ "8": 1650282496.0,
+ "9": 1650282496.0,
+ "10": 1650282496.0,
+ "11": 1650282496.0,
+ "12": 1650282496.0,
+ "13": 1650282496.0,
+ "14": 1650282496.0,
+ "15": 1650282496.0,
+ "16": 1650282496.0,
+ "17": 1650282496.0,
+ "18": 1650282496.0,
+ "19": 1650282496.0,
+ "20": 1650282496.0,
+ "21": 1650282496.0,
+ "22": 1650282496.0,
+ "23": 1650282496.0,
+ "24": 1650282496.0,
+ "25": 1650282496.0,
+ "26": 1650282496.0,
+ "27": 1650282496.0,
+ "28": 1650282496.0,
+ "29": 1650282496.0,
+ "30": 1650282496.0,
+ "31": 1650282496.0,
+ "32": 1650282496.0,
+ "33": 1650282496.0,
+ "34": 1650282496.0,
+ "35": 1650282496.0,
+ "36": 1650282496.0,
+ "37": 1650282496.0,
+ "38": 1650282496.0,
+ "39": 1650282496.0,
+ "40": 1650282496.0,
+ "41": 1650282496.0,
+ "42": 1650282496.0,
+ "43": 1650282496.0,
+ "44": 1650282496.0,
+ "45": 1650282496.0,
+ "46": 1650282496.0,
+ "47": 1650282496.0,
+ "48": 1650282496.0,
+ "49": 1650282496.0,
+ "50": 1650282496.0
}
},
"mem-max-allocated-bytes": {
@@ -175,56 +175,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 1653825536.0,
- "2": 2142998016.0,
- "3": 2142998016.0,
- "4": 2142998016.0,
- "5": 2142998016.0,
- "6": 2142998016.0,
- "7": 2142998016.0,
- "8": 2142998016.0,
- "9": 2142998016.0,
- "10": 2142998016.0,
- "11": 2142998016.0,
- "12": 2142998016.0,
- "13": 2142998016.0,
- "14": 2142998016.0,
- "15": 2142998016.0,
- "16": 2142998016.0,
- "17": 2142998016.0,
- "18": 2142998016.0,
- "19": 2142998016.0,
- "20": 2142998016.0,
- "21": 2142998016.0,
- "22": 2142998016.0,
- "23": 2142998016.0,
- "24": 2142998016.0,
- "25": 2142998016.0,
- "26": 2142998016.0,
- "27": 2142998016.0,
- "28": 2142998016.0,
- "29": 2142998016.0,
- "30": 2142998016.0,
- "31": 2142998016.0,
- "32": 2142998016.0,
- "33": 2142998016.0,
- "34": 2142998016.0,
- "35": 2142998016.0,
- "36": 2142998016.0,
- "37": 2142998016.0,
- "38": 2142998016.0,
- "39": 2142998016.0,
- "40": 2142998016.0,
- "41": 2142998016.0,
- "42": 2142998016.0,
- "43": 2142998016.0,
- "44": 2142998016.0,
- "45": 2142998016.0,
- "46": 2142998016.0,
- "47": 2142998016.0,
- "48": 2142998016.0,
- "49": 2142998016.0,
- "50": 2142998016.0
+ "1": 1650286592.0,
+ "2": 2137679360.0,
+ "3": 2137679360.0,
+ "4": 2137679360.0,
+ "5": 2137679360.0,
+ "6": 2137679360.0,
+ "7": 2137679360.0,
+ "8": 2137679360.0,
+ "9": 2137679360.0,
+ "10": 2137679360.0,
+ "11": 2137679360.0,
+ "12": 2137679360.0,
+ "13": 2137679360.0,
+ "14": 2137679360.0,
+ "15": 2137679360.0,
+ "16": 2137679360.0,
+ "17": 2137679360.0,
+ "18": 2137679360.0,
+ "19": 2137679360.0,
+ "20": 2137679360.0,
+ "21": 2137679360.0,
+ "22": 2137679360.0,
+ "23": 2137679360.0,
+ "24": 2137679360.0,
+ "25": 2137679360.0,
+ "26": 2137679360.0,
+ "27": 2137679360.0,
+ "28": 2137679360.0,
+ "29": 2137679360.0,
+ "30": 2137679360.0,
+ "31": 2137679360.0,
+ "32": 2137679360.0,
+ "33": 2137679360.0,
+ "34": 2137679360.0,
+ "35": 2137679360.0,
+ "36": 2137679360.0,
+ "37": 2137679360.0,
+ "38": 2137679360.0,
+ "39": 2137679360.0,
+ "40": 2137679360.0,
+ "41": 2137679360.0,
+ "42": 2137679360.0,
+ "43": 2137679360.0,
+ "44": 2137679360.0,
+ "45": 2137679360.0,
+ "46": 2137679360.0,
+ "47": 2137679360.0,
+ "48": 2137679360.0,
+ "49": 2137679360.0,
+ "50": 2137679360.0
}
},
"iteration-time": {
@@ -232,56 +232,56 @@
"end_step": 50,
"step_interval": 1,
"values": {
- "1": 28.88794,
- "2": 1.3875,
- "3": 1.3655,
- "4": 0.91436,
- "5": 0.92323,
- "6": 0.90862,
- "7": 0.90351,
- "8": 0.90087,
- "9": 0.90804,
- "10": 0.90099,
- "11": 1.44829,
- "12": 1.27198,
- "13": 1.47603,
- "14": 0.90715,
- "15": 0.90169,
- "16": 0.8955,
- "17": 0.91977,
- "18": 0.91161,
- "19": 0.90173,
- "20": 0.89581,
- "21": 0.89026,
- "22": 0.88949,
- "23": 0.91159,
- "24": 0.90975,
- "25": 0.90708,
- "26": 0.89948,
- "27": 0.89544,
- "28": 0.89745,
- "29": 0.90068,
- "30": 0.89534,
- "31": 0.90066,
- "32": 0.91859,
- "33": 0.91419,
- "34": 0.89878,
- "35": 0.89846,
- "36": 0.8945,
- "37": 0.89356,
- "38": 0.89475,
- "39": 0.89372,
- "40": 0.90674,
- "41": 0.90461,
- "42": 0.93092,
- "43": 0.90002,
- "44": 0.89721,
- "45": 0.89453,
- "46": 0.89499,
- "47": 0.90828,
- "48": 0.89629,
- "49": 0.90644,
- "50": 0.90588
+ "1": "nan",
+ "2": 7.91637,
+ "3": 0.92764,
+ "4": 0.91334,
+ "5": 0.91263,
+ "6": 0.91881,
+ "7": 0.91512,
+ "8": 0.91691,
+ "9": 0.91241,
+ "10": 0.91216,
+ "11": 0.90894,
+ "12": 0.90923,
+ "13": 0.91112,
+ "14": 0.91514,
+ "15": 0.91364,
+ "16": 0.91332,
+ "17": 0.91209,
+ "18": 0.9107,
+ "19": 1.32194,
+ "20": 0.91463,
+ "21": 0.91083,
+ "22": 0.91645,
+ "23": 0.91042,
+ "24": 0.91104,
+ "25": 0.90961,
+ "26": 0.90996,
+ "27": 0.91215,
+ "28": 1.30488,
+ "29": 0.91859,
+ "30": 0.91091,
+ "31": 0.91133,
+ "32": 0.91333,
+ "33": 0.9091,
+ "34": 0.91409,
+ "35": 1.30333,
+ "36": 0.91422,
+ "37": 0.91635,
+ "38": 0.91473,
+ "39": 1.50529,
+ "40": 0.97326,
+ "41": 1.31231,
+ "42": 0.91182,
+ "43": 1.30555,
+ "44": 0.91651,
+ "45": 0.91568,
+ "46": 1.35255,
+ "47": 0.91348,
+ "48": 0.91258,
+ "49": 0.91001,
+ "50": 0.90511
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/t5/t5_11b_mcore_tp4_pp1/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/t5/t5_11b_mcore_tp4_pp1/golden_values_dev_dgx_h100.json
index f4a701a2e4d..92f66b60691 100644
--- a/tests/functional_tests/test_cases/t5/t5_11b_mcore_tp4_pp1/golden_values_dev_dgx_h100.json
+++ b/tests/functional_tests/test_cases/t5/t5_11b_mcore_tp4_pp1/golden_values_dev_dgx_h100.json
@@ -4,31 +4,31 @@
"end_step": 25,
"step_interval": 1,
"values": {
- "1": 10.74903,
- "2": 10.75924,
- "3": 16.15622,
- "4": 20.1728,
- "5": 11.07413,
- "6": 10.29087,
- "7": 10.31369,
- "8": 10.31557,
- "9": 9.68992,
- "10": 9.25112,
- "11": 9.43376,
- "12": 9.8267,
- "13": 8.88334,
- "14": 8.49023,
- "15": 8.79113,
- "16": 7.95739,
- "17": 7.70005,
- "18": 7.81826,
- "19": 8.21562,
- "20": 8.16452,
- "21": 7.833,
- "22": 7.71899,
- "23": 7.88724,
- "24": 7.70093,
- "25": 7.78994
+ "1": 10.72478,
+ "2": 10.75508,
+ "3": 16.36503,
+ "4": 19.85665,
+ "5": 11.26807,
+ "6": 10.26242,
+ "7": 10.27643,
+ "8": 10.21919,
+ "9": 9.67706,
+ "10": 9.28114,
+ "11": 9.41509,
+ "12": 9.80636,
+ "13": 8.90098,
+ "14": 8.48799,
+ "15": 8.79292,
+ "16": 7.96872,
+ "17": 7.70291,
+ "18": 7.79927,
+ "19": 8.18669,
+ "20": 8.15261,
+ "21": 7.82296,
+ "22": 7.71245,
+ "23": 7.86818,
+ "24": 7.65708,
+ "25": 7.77835
}
},
"num-zeros": {
@@ -36,31 +36,31 @@
"end_step": 25,
"step_interval": 1,
"values": {
- "1": 245867.0,
- "2": 256817.0,
- "3": 248438.0,
- "4": 233541.0,
- "5": 251594.0,
- "6": 259588.0,
- "7": 256938.0,
- "8": 237612.0,
- "9": 241154.0,
- "10": 252461.0,
- "11": 288146.0,
- "12": 248712.0,
- "13": 241371.0,
- "14": 228365.0,
- "15": 261948.0,
- "16": 237032.0,
- "17": 249760.0,
- "18": 251590.0,
- "19": 257104.0,
- "20": 248292.0,
- "21": 231805.0,
- "22": 223805.0,
- "23": 247959.0,
- "24": 250798.0,
- "25": 237325.0
+ "1": 245877.0,
+ "2": 256678.0,
+ "3": 248449.0,
+ "4": 233627.0,
+ "5": 251461.0,
+ "6": 259612.0,
+ "7": 256977.0,
+ "8": 237781.0,
+ "9": 241225.0,
+ "10": 252449.0,
+ "11": 288363.0,
+ "12": 248683.0,
+ "13": 241476.0,
+ "14": 228497.0,
+ "15": 262029.0,
+ "16": 236949.0,
+ "17": 249661.0,
+ "18": 251680.0,
+ "19": 257143.0,
+ "20": 248220.0,
+ "21": 231800.0,
+ "22": 223145.0,
+ "23": 247912.0,
+ "24": 250781.0,
+ "25": 238528.0
}
},
"mem-allocated-bytes": {
@@ -101,30 +101,30 @@
"step_interval": 1,
"values": {
"1": 40735715328.0,
- "2": 44991991808.0,
- "3": 44993564672.0,
- "4": 44993564672.0,
- "5": 44993564672.0,
- "6": 44993564672.0,
- "7": 44993564672.0,
- "8": 44993564672.0,
- "9": 44993564672.0,
- "10": 44993564672.0,
- "11": 44993564672.0,
- "12": 44993564672.0,
- "13": 44993564672.0,
- "14": 44993564672.0,
- "15": 44993564672.0,
- "16": 44993564672.0,
- "17": 44993564672.0,
- "18": 44993564672.0,
- "19": 44993564672.0,
- "20": 44993564672.0,
- "21": 44993564672.0,
- "22": 44993564672.0,
- "23": 44993564672.0,
- "24": 44993564672.0,
- "25": 44993564672.0
+ "2": 44989894656.0,
+ "3": 44989894656.0,
+ "4": 44989894656.0,
+ "5": 44989894656.0,
+ "6": 44989894656.0,
+ "7": 44989894656.0,
+ "8": 44989894656.0,
+ "9": 44989894656.0,
+ "10": 44989894656.0,
+ "11": 44989894656.0,
+ "12": 44989894656.0,
+ "13": 44989894656.0,
+ "14": 44989894656.0,
+ "15": 44989894656.0,
+ "16": 44989894656.0,
+ "17": 44989894656.0,
+ "18": 44989894656.0,
+ "19": 44989894656.0,
+ "20": 44989894656.0,
+ "21": 44989894656.0,
+ "22": 44989894656.0,
+ "23": 44989894656.0,
+ "24": 44989894656.0,
+ "25": 44989894656.0
}
},
"iteration-time": {
@@ -132,31 +132,31 @@
"end_step": 25,
"step_interval": 1,
"values": {
- "1": 25.74522,
- "2": 0.73559,
- "3": 0.40581,
- "4": 0.38308,
- "5": 0.37606,
- "6": 0.37631,
- "7": 0.39269,
- "8": 0.37902,
- "9": 0.37764,
- "10": 0.8554,
- "11": 0.95952,
- "12": 0.37861,
- "13": 0.38954,
- "14": 0.42497,
- "15": 0.37698,
- "16": 0.37629,
- "17": 0.37835,
- "18": 0.3766,
- "19": 0.37494,
- "20": 0.42005,
- "21": 0.38011,
- "22": 0.37713,
- "23": 0.37617,
- "24": 0.37515,
- "25": 0.37401
+ "1": "nan",
+ "2": 6.67745,
+ "3": 0.39735,
+ "4": 0.37954,
+ "5": 0.37756,
+ "6": 0.38019,
+ "7": 0.37708,
+ "8": 0.37784,
+ "9": 0.37652,
+ "10": 0.379,
+ "11": 0.37836,
+ "12": 0.38173,
+ "13": 0.37692,
+ "14": 0.37886,
+ "15": 0.37923,
+ "16": 0.37855,
+ "17": 0.3788,
+ "18": 0.37899,
+ "19": 0.37732,
+ "20": 0.37813,
+ "21": 0.37908,
+ "22": 0.38047,
+ "23": 0.38247,
+ "24": 0.38016,
+ "25": 0.37866
}
}
}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/t5/t5_11b_mcore_tp4_pp1/model_config.yaml b/tests/functional_tests/test_cases/t5/t5_11b_mcore_tp4_pp1/model_config.yaml
index 6be0edcd91b..8dc5cf5a713 100644
--- a/tests/functional_tests/test_cases/t5/t5_11b_mcore_tp4_pp1/model_config.yaml
+++ b/tests/functional_tests/test_cases/t5/t5_11b_mcore_tp4_pp1/model_config.yaml
@@ -29,8 +29,8 @@ MODEL_ARGS:
--vocab-extra-ids: 100
--init-method-std: 0.015
--transformer-impl: transformer_engine
- --data-path: ${DATA_PATH}/text/the_pile/t5_shard00/my-t5_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/t5_shard00/bert-large-cased-vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-t5_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bert-large-cased-vocab.txt
--tokenizer-type: BertWordPieceCase
--calculate-per-token-loss: true
--split: 99982,9,9
@@ -51,7 +51,6 @@ MODEL_ARGS:
--deterministic-mode: true
--ckpt-format: torch_dist
--log-memory-to-tensorboard: true
- --legacy-tokenizer: true
TEST_TYPE: regular
METRICS:
- "lm loss"
diff --git a/tests/functional_tests/test_cases/t5/t5_mcore_te_tp1_pp1_vp1_resume_torch/model_config.yaml b/tests/functional_tests/test_cases/t5/t5_mcore_te_tp1_pp1_vp1_resume_torch/model_config.yaml
index aa0f67ff311..8fc0b04c8c8 100644
--- a/tests/functional_tests/test_cases/t5/t5_mcore_te_tp1_pp1_vp1_resume_torch/model_config.yaml
+++ b/tests/functional_tests/test_cases/t5/t5_mcore_te_tp1_pp1_vp1_resume_torch/model_config.yaml
@@ -29,8 +29,8 @@ MODEL_ARGS:
--vocab-extra-ids: 100
--init-method-std: 0.015
--transformer-impl: transformer_engine
- --data-path: ${DATA_PATH}/text/the_pile/t5_shard00/my-t5_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/t5_shard00/bert-large-cased-vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-t5_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bert-large-cased-vocab.txt
--tokenizer-type: BertWordPieceCase
--calculate-per-token-loss: true
--split: 99982,9,9
diff --git a/tests/functional_tests/test_cases/t5/t5_mcore_te_tp2_pp1_vp1/model_config.yaml b/tests/functional_tests/test_cases/t5/t5_mcore_te_tp2_pp1_vp1/model_config.yaml
index 59c1d0f280f..277d637cb4d 100644
--- a/tests/functional_tests/test_cases/t5/t5_mcore_te_tp2_pp1_vp1/model_config.yaml
+++ b/tests/functional_tests/test_cases/t5/t5_mcore_te_tp2_pp1_vp1/model_config.yaml
@@ -29,8 +29,8 @@ MODEL_ARGS:
--vocab-extra-ids: 100
--init-method-std: 0.015
--transformer-impl: transformer_engine
- --data-path: ${DATA_PATH}/text/the_pile/t5_shard00/my-t5_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/t5_shard00/bert-large-cased-vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-t5_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bert-large-cased-vocab.txt
--tokenizer-type: BertWordPieceCase
--calculate-per-token-loss: true
--split: 99982,9,9
diff --git a/tests/functional_tests/test_cases/t5/t5_mcore_te_tp2_pp1_vp1_sequence_parallel/model_config.yaml b/tests/functional_tests/test_cases/t5/t5_mcore_te_tp2_pp1_vp1_sequence_parallel/model_config.yaml
index 80a84a26e0c..8463c0ebfa4 100644
--- a/tests/functional_tests/test_cases/t5/t5_mcore_te_tp2_pp1_vp1_sequence_parallel/model_config.yaml
+++ b/tests/functional_tests/test_cases/t5/t5_mcore_te_tp2_pp1_vp1_sequence_parallel/model_config.yaml
@@ -29,8 +29,8 @@ MODEL_ARGS:
--vocab-extra-ids: 100
--init-method-std: 0.015
--transformer-impl: transformer_engine
- --data-path: ${DATA_PATH}/text/the_pile/t5_shard00/my-t5_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/t5_shard00/bert-large-cased-vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-t5_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bert-large-cased-vocab.txt
--tokenizer-type: BertWordPieceCase
--calculate-per-token-loss: true
--split: 99982,9,9
diff --git a/tests/functional_tests/test_cases/t5/t5_mcore_te_tp4_pp1/model_config.yaml b/tests/functional_tests/test_cases/t5/t5_mcore_te_tp4_pp1/model_config.yaml
index 047280dec39..f4a4e7c3b1c 100644
--- a/tests/functional_tests/test_cases/t5/t5_mcore_te_tp4_pp1/model_config.yaml
+++ b/tests/functional_tests/test_cases/t5/t5_mcore_te_tp4_pp1/model_config.yaml
@@ -29,8 +29,8 @@ MODEL_ARGS:
--vocab-extra-ids: 100
--init-method-std: 0.015
--transformer-impl: transformer_engine
- --data-path: ${DATA_PATH}/text/the_pile/t5_shard00/my-t5_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/t5_shard00/bert-large-cased-vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-t5_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bert-large-cased-vocab.txt
--tokenizer-type: BertWordPieceCase
--calculate-per-token-loss: true
--split: 99982,9,9
diff --git a/tests/functional_tests/test_cases/t5/t5_mcore_te_tp4_pp1_resume_torch_dist/model_config.yaml b/tests/functional_tests/test_cases/t5/t5_mcore_te_tp4_pp1_resume_torch_dist/model_config.yaml
index 1611c02251b..5f170ec8f43 100644
--- a/tests/functional_tests/test_cases/t5/t5_mcore_te_tp4_pp1_resume_torch_dist/model_config.yaml
+++ b/tests/functional_tests/test_cases/t5/t5_mcore_te_tp4_pp1_resume_torch_dist/model_config.yaml
@@ -29,8 +29,8 @@ MODEL_ARGS:
--vocab-extra-ids: 100
--init-method-std: 0.015
--transformer-impl: transformer_engine
- --data-path: ${DATA_PATH}/text/the_pile/t5_shard00/my-t5_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/t5_shard00/bert-large-cased-vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-t5_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bert-large-cased-vocab.txt
--tokenizer-type: BertWordPieceCase
--calculate-per-token-loss: true
--split: 99982,9,9
diff --git a/tests/functional_tests/test_cases/t5/t5_mcore_tp1_pp1_vp1/model_config.yaml b/tests/functional_tests/test_cases/t5/t5_mcore_tp1_pp1_vp1/model_config.yaml
index 12ccecb5883..f8e6db5d685 100644
--- a/tests/functional_tests/test_cases/t5/t5_mcore_tp1_pp1_vp1/model_config.yaml
+++ b/tests/functional_tests/test_cases/t5/t5_mcore_tp1_pp1_vp1/model_config.yaml
@@ -29,8 +29,8 @@ MODEL_ARGS:
--vocab-extra-ids: 100
--init-method-std: 0.015
--transformer-impl: local
- --data-path: ${DATA_PATH}/text/the_pile/t5_shard00/my-t5_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/t5_shard00/bert-large-cased-vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-t5_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bert-large-cased-vocab.txt
--tokenizer-type: BertWordPieceCase
--calculate-per-token-loss: true
--split: 99982,9,9
diff --git a/tests/functional_tests/test_cases/t5/t5_mcore_tp1_pp1_vp1_resume_torch/model_config.yaml b/tests/functional_tests/test_cases/t5/t5_mcore_tp1_pp1_vp1_resume_torch/model_config.yaml
index 8559fd587d1..314a960cc75 100644
--- a/tests/functional_tests/test_cases/t5/t5_mcore_tp1_pp1_vp1_resume_torch/model_config.yaml
+++ b/tests/functional_tests/test_cases/t5/t5_mcore_tp1_pp1_vp1_resume_torch/model_config.yaml
@@ -29,8 +29,8 @@ MODEL_ARGS:
--vocab-extra-ids: 100
--init-method-std: 0.015
--transformer-impl: local
- --data-path: ${DATA_PATH}/text/the_pile/t5_shard00/my-t5_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/t5_shard00/bert-large-cased-vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-t5_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bert-large-cased-vocab.txt
--tokenizer-type: BertWordPieceCase
--calculate-per-token-loss: true
--split: 99982,9,9
diff --git a/tests/functional_tests/test_cases/t5/t5_mcore_tp2_pp1_vp1/model_config.yaml b/tests/functional_tests/test_cases/t5/t5_mcore_tp2_pp1_vp1/model_config.yaml
index 9c6a835571c..df1bb9d1833 100644
--- a/tests/functional_tests/test_cases/t5/t5_mcore_tp2_pp1_vp1/model_config.yaml
+++ b/tests/functional_tests/test_cases/t5/t5_mcore_tp2_pp1_vp1/model_config.yaml
@@ -29,8 +29,8 @@ MODEL_ARGS:
--vocab-extra-ids: 100
--init-method-std: 0.015
--transformer-impl: local
- --data-path: ${DATA_PATH}/text/the_pile/t5_shard00/my-t5_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/t5_shard00/bert-large-cased-vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-t5_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bert-large-cased-vocab.txt
--tokenizer-type: BertWordPieceCase
--calculate-per-token-loss: true
--split: 99982,9,9
diff --git a/tests/functional_tests/test_cases/t5/t5_mcore_tp4_pp1/model_config.yaml b/tests/functional_tests/test_cases/t5/t5_mcore_tp4_pp1/model_config.yaml
index dd3896ad88a..33d79798194 100644
--- a/tests/functional_tests/test_cases/t5/t5_mcore_tp4_pp1/model_config.yaml
+++ b/tests/functional_tests/test_cases/t5/t5_mcore_tp4_pp1/model_config.yaml
@@ -29,8 +29,8 @@ MODEL_ARGS:
--vocab-extra-ids: 100
--init-method-std: 0.015
--transformer-impl: local
- --data-path: ${DATA_PATH}/text/the_pile/t5_shard00/my-t5_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/t5_shard00/bert-large-cased-vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-t5_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bert-large-cased-vocab.txt
--tokenizer-type: BertWordPieceCase
--calculate-per-token-loss: true
--split: 99982,9,9
diff --git a/tests/functional_tests/test_cases/t5/t5_mcore_tp4_pp1_resume_torch_dist/model_config.yaml b/tests/functional_tests/test_cases/t5/t5_mcore_tp4_pp1_resume_torch_dist/model_config.yaml
index 4c955dd5441..15096f11362 100644
--- a/tests/functional_tests/test_cases/t5/t5_mcore_tp4_pp1_resume_torch_dist/model_config.yaml
+++ b/tests/functional_tests/test_cases/t5/t5_mcore_tp4_pp1_resume_torch_dist/model_config.yaml
@@ -29,8 +29,8 @@ MODEL_ARGS:
--vocab-extra-ids: 100
--init-method-std: 0.015
--transformer-impl: local
- --data-path: ${DATA_PATH}/text/the_pile/t5_shard00/my-t5_00_text_document
- --vocab-file: ${DATA_PATH}/text/the_pile/t5_shard00/bert-large-cased-vocab.txt
+ --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-t5_00_text_document
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bert-large-cased-vocab.txt
--tokenizer-type: BertWordPieceCase
--calculate-per-token-loss: true
--split: 99982,9,9
diff --git a/tests/functional_tests/test_cases/t5/t5_release/model_config.yaml b/tests/functional_tests/test_cases/t5/t5_release/model_config.yaml
index a7abdc1bdd4..485b14cbfa2 100644
--- a/tests/functional_tests/test_cases/t5/t5_release/model_config.yaml
+++ b/tests/functional_tests/test_cases/t5/t5_release/model_config.yaml
@@ -37,7 +37,7 @@ MODEL_ARGS:
--pipeline-model-parallel-size: 1
# Data args
--data-path: ${DATA_BLEND}
- --vocab-file: ${DATA_PATH}/text/the_pile/t5_shard00/bert-large-cased-vocab.txt
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bert-large-cased-vocab.txt
--tokenizer-type: BertWordPieceCase
--split: 99982,9,9
--data-cache-path: ${DATA_CACHE_PATH}
diff --git a/tests/functional_tests/test_cases/t5/t5_release_sm/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/t5/t5_release_sm/golden_values_dev_dgx_gb200.json
new file mode 100644
index 00000000000..4a556d7842b
--- /dev/null
+++ b/tests/functional_tests/test_cases/t5/t5_release_sm/golden_values_dev_dgx_gb200.json
@@ -0,0 +1,10042 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 10000,
+ "step_interval": 5,
+ "values": {
+ "1": 10.35422,
+ "5": 10.35244,
+ "10": 10.26851,
+ "15": 9.88227,
+ "20": 9.69037,
+ "25": 9.56795,
+ "30": 9.46219,
+ "35": 9.39134,
+ "40": 9.3069,
+ "45": 9.22623,
+ "50": 9.17627,
+ "55": 9.10983,
+ "60": 9.02747,
+ "65": 8.96604,
+ "70": 8.97179,
+ "75": 8.89044,
+ "80": 8.90169,
+ "85": 8.8215,
+ "90": 8.80145,
+ "95": 8.76479,
+ "100": 8.71678,
+ "105": 8.65914,
+ "110": 8.59577,
+ "115": 8.56179,
+ "120": 8.56322,
+ "125": 8.49369,
+ "130": 8.4767,
+ "135": 8.38244,
+ "140": 8.38327,
+ "145": 8.32016,
+ "150": 8.2906,
+ "155": 8.19567,
+ "160": 8.19227,
+ "165": 8.12922,
+ "170": 8.12686,
+ "175": 8.07496,
+ "180": 7.96342,
+ "185": 7.96966,
+ "190": 7.89985,
+ "195": 7.87228,
+ "200": 7.80925,
+ "205": 7.75866,
+ "210": 7.69986,
+ "215": 7.64959,
+ "220": 7.61327,
+ "225": 7.52293,
+ "230": 7.45816,
+ "235": 7.43469,
+ "240": 7.32166,
+ "245": 7.26914,
+ "250": 7.17842,
+ "255": 7.15919,
+ "260": 7.03225,
+ "265": 7.00419,
+ "270": 6.97262,
+ "275": 6.9009,
+ "280": 6.84399,
+ "285": 6.79324,
+ "290": 6.80189,
+ "295": 6.69807,
+ "300": 6.66977,
+ "305": 6.61829,
+ "310": 6.56177,
+ "315": 6.49709,
+ "320": 6.55012,
+ "325": 6.52737,
+ "330": 6.48147,
+ "335": 6.42692,
+ "340": 6.44752,
+ "345": 6.38568,
+ "350": 6.40994,
+ "355": 6.34603,
+ "360": 6.3743,
+ "365": 6.33387,
+ "370": 6.30346,
+ "375": 6.28671,
+ "380": 6.26299,
+ "385": 6.25445,
+ "390": 6.21746,
+ "395": 6.2045,
+ "400": 6.25252,
+ "405": 6.17649,
+ "410": 6.26333,
+ "415": 6.18102,
+ "420": 6.15185,
+ "425": 6.11123,
+ "430": 6.08865,
+ "435": 6.11583,
+ "440": 6.05448,
+ "445": 6.02378,
+ "450": 6.0877,
+ "455": 6.01558,
+ "460": 6.01151,
+ "465": 6.01898,
+ "470": 5.96315,
+ "475": 5.956,
+ "480": 5.96648,
+ "485": 5.93978,
+ "490": 5.90354,
+ "495": 5.86746,
+ "500": 5.88068,
+ "505": 5.88819,
+ "510": 5.9053,
+ "515": 5.81479,
+ "520": 5.84294,
+ "525": 5.85994,
+ "530": 5.83191,
+ "535": 5.83363,
+ "540": 5.81726,
+ "545": 5.77191,
+ "550": 5.768,
+ "555": 5.74973,
+ "560": 5.74862,
+ "565": 5.73474,
+ "570": 5.73697,
+ "575": 5.71875,
+ "580": 5.65636,
+ "585": 5.67362,
+ "590": 5.62994,
+ "595": 5.63984,
+ "600": 5.65741,
+ "605": 5.63531,
+ "610": 5.64542,
+ "615": 5.61652,
+ "620": 5.6367,
+ "625": 5.65916,
+ "630": 5.60147,
+ "635": 5.59802,
+ "640": 5.57461,
+ "645": 5.58812,
+ "650": 5.55562,
+ "655": 5.5488,
+ "660": 5.52534,
+ "665": 5.57479,
+ "670": 5.47585,
+ "675": 5.51316,
+ "680": 5.52697,
+ "685": 5.50858,
+ "690": 5.52376,
+ "695": 5.47373,
+ "700": 5.43586,
+ "705": 5.37977,
+ "710": 5.44082,
+ "715": 5.46763,
+ "720": 5.42906,
+ "725": 5.44931,
+ "730": 5.41484,
+ "735": 5.41365,
+ "740": 5.40059,
+ "745": 5.41292,
+ "750": 5.43581,
+ "755": 5.40179,
+ "760": 5.36305,
+ "765": 5.37276,
+ "770": 5.35635,
+ "775": 5.32888,
+ "780": 5.31908,
+ "785": 5.30782,
+ "790": 5.28898,
+ "795": 5.31475,
+ "800": 5.31447,
+ "805": 5.34343,
+ "810": 5.29927,
+ "815": 5.24314,
+ "820": 5.28725,
+ "825": 5.31529,
+ "830": 5.26729,
+ "835": 5.30493,
+ "840": 5.31113,
+ "845": 5.21284,
+ "850": 5.23233,
+ "855": 5.25005,
+ "860": 5.24698,
+ "865": 5.26621,
+ "870": 5.23597,
+ "875": 5.19926,
+ "880": 5.29408,
+ "885": 5.26177,
+ "890": 5.20201,
+ "895": 5.21565,
+ "900": 5.22992,
+ "905": 5.22061,
+ "910": 5.18794,
+ "915": 5.20538,
+ "920": 5.18427,
+ "925": 5.19248,
+ "930": 5.1727,
+ "935": 5.14715,
+ "940": 5.14966,
+ "945": 5.18893,
+ "950": 5.15875,
+ "955": 5.09555,
+ "960": 5.14027,
+ "965": 5.08914,
+ "970": 5.07034,
+ "975": 5.11743,
+ "980": 5.10166,
+ "985": 5.06264,
+ "990": 5.08607,
+ "995": 5.0956,
+ "1000": 5.09103,
+ "1005": 5.09954,
+ "1010": 5.097,
+ "1015": 5.06808,
+ "1020": 5.10625,
+ "1025": 5.04915,
+ "1030": 4.99804,
+ "1035": 5.07873,
+ "1040": 5.03042,
+ "1045": 5.04968,
+ "1050": 5.08079,
+ "1055": 5.03578,
+ "1060": 5.08799,
+ "1065": 4.98247,
+ "1070": 4.97899,
+ "1075": 4.98678,
+ "1080": 4.97907,
+ "1085": 5.01173,
+ "1090": 5.02146,
+ "1095": 5.02248,
+ "1100": 5.0172,
+ "1105": 4.98326,
+ "1110": 4.92767,
+ "1115": 4.92908,
+ "1120": 4.95516,
+ "1125": 4.89882,
+ "1130": 5.04872,
+ "1135": 4.92221,
+ "1140": 4.95418,
+ "1145": 4.90174,
+ "1150": 4.9825,
+ "1155": 4.914,
+ "1160": 4.90757,
+ "1165": 4.91722,
+ "1170": 4.98739,
+ "1175": 4.95708,
+ "1180": 4.84853,
+ "1185": 4.92856,
+ "1190": 4.89222,
+ "1195": 4.92577,
+ "1200": 5.01238,
+ "1205": 4.97038,
+ "1210": 4.85084,
+ "1215": 4.88675,
+ "1220": 4.88661,
+ "1225": 4.89237,
+ "1230": 4.90525,
+ "1235": 4.89192,
+ "1240": 4.87586,
+ "1245": 4.89021,
+ "1250": 4.83298,
+ "1255": 4.88231,
+ "1260": 4.80569,
+ "1265": 4.86962,
+ "1270": 4.93569,
+ "1275": 4.84407,
+ "1280": 4.82585,
+ "1285": 4.9187,
+ "1290": 4.7207,
+ "1295": 4.83168,
+ "1300": 4.84988,
+ "1305": 4.83284,
+ "1310": 4.91649,
+ "1315": 4.80151,
+ "1320": 4.87973,
+ "1325": 4.77275,
+ "1330": 4.82154,
+ "1335": 4.7989,
+ "1340": 4.80786,
+ "1345": 4.82988,
+ "1350": 4.83467,
+ "1355": 4.80195,
+ "1360": 4.74063,
+ "1365": 4.84719,
+ "1370": 4.81709,
+ "1375": 4.83832,
+ "1380": 4.76502,
+ "1385": 4.7394,
+ "1390": 4.78736,
+ "1395": 4.80151,
+ "1400": 4.67481,
+ "1405": 4.6766,
+ "1410": 4.75678,
+ "1415": 4.79175,
+ "1420": 4.76775,
+ "1425": 4.72711,
+ "1430": 4.76087,
+ "1435": 4.77786,
+ "1440": 4.76816,
+ "1445": 4.75692,
+ "1450": 4.72069,
+ "1455": 4.66908,
+ "1460": 4.78031,
+ "1465": 4.74164,
+ "1470": 4.69459,
+ "1475": 4.71481,
+ "1480": 4.72217,
+ "1485": 4.75277,
+ "1490": 4.75536,
+ "1495": 4.73124,
+ "1500": 4.77502,
+ "1505": 4.68863,
+ "1510": 4.72556,
+ "1515": 4.71948,
+ "1520": 4.76127,
+ "1525": 4.7393,
+ "1530": 4.68131,
+ "1535": 4.70294,
+ "1540": 4.7174,
+ "1545": 4.61173,
+ "1550": 4.64164,
+ "1555": 4.61876,
+ "1560": 4.71064,
+ "1565": 4.64286,
+ "1570": 4.6759,
+ "1575": 4.67829,
+ "1580": 4.66854,
+ "1585": 4.69151,
+ "1590": 4.67386,
+ "1595": 4.72778,
+ "1600": 4.69207,
+ "1605": 4.68147,
+ "1610": 4.61474,
+ "1615": 4.62707,
+ "1620": 4.70251,
+ "1625": 4.67161,
+ "1630": 4.61246,
+ "1635": 4.61327,
+ "1640": 4.69908,
+ "1645": 4.64987,
+ "1650": 4.67226,
+ "1655": 4.62881,
+ "1660": 4.68249,
+ "1665": 4.62556,
+ "1670": 4.70554,
+ "1675": 4.65425,
+ "1680": 4.60146,
+ "1685": 4.65819,
+ "1690": 4.64996,
+ "1695": 4.67006,
+ "1700": 4.65625,
+ "1705": 4.65152,
+ "1710": 4.64685,
+ "1715": 4.69759,
+ "1720": 4.65208,
+ "1725": 4.66975,
+ "1730": 4.64425,
+ "1735": 4.64299,
+ "1740": 4.61291,
+ "1745": 4.62997,
+ "1750": 4.6083,
+ "1755": 4.64938,
+ "1760": 4.6049,
+ "1765": 4.62898,
+ "1770": 4.61893,
+ "1775": 4.59671,
+ "1780": 4.60995,
+ "1785": 4.62232,
+ "1790": 4.51589,
+ "1795": 4.5899,
+ "1800": 4.61411,
+ "1805": 4.57352,
+ "1810": 4.56953,
+ "1815": 4.58557,
+ "1820": 4.60518,
+ "1825": 4.63038,
+ "1830": 4.5886,
+ "1835": 4.57328,
+ "1840": 4.50201,
+ "1845": 4.54108,
+ "1850": 4.64782,
+ "1855": 4.59506,
+ "1860": 4.57645,
+ "1865": 4.56283,
+ "1870": 4.59126,
+ "1875": 4.5641,
+ "1880": 4.59555,
+ "1885": 4.52527,
+ "1890": 4.5125,
+ "1895": 4.64848,
+ "1900": 4.49256,
+ "1905": 4.57663,
+ "1910": 4.61161,
+ "1915": 4.59131,
+ "1920": 4.57205,
+ "1925": 4.54446,
+ "1930": 4.5772,
+ "1935": 4.55553,
+ "1940": 4.5257,
+ "1945": 4.55056,
+ "1950": 4.53022,
+ "1955": 4.56348,
+ "1960": 4.55271,
+ "1965": 4.57378,
+ "1970": 4.53254,
+ "1975": 4.56525,
+ "1980": 4.52189,
+ "1985": 4.59018,
+ "1990": 4.48917,
+ "1995": 4.57228,
+ "2000": 4.55914,
+ "2005": 4.53724,
+ "2010": 4.55658,
+ "2015": 4.56339,
+ "2020": 4.49992,
+ "2025": 4.50587,
+ "2030": 4.56106,
+ "2035": 4.53759,
+ "2040": 4.58721,
+ "2045": 4.457,
+ "2050": 4.52805,
+ "2055": 4.53119,
+ "2060": 4.50153,
+ "2065": 4.55744,
+ "2070": 4.50464,
+ "2075": 4.5742,
+ "2080": 4.56507,
+ "2085": 4.51266,
+ "2090": 4.47776,
+ "2095": 4.43338,
+ "2100": 4.41447,
+ "2105": 4.44887,
+ "2110": 4.57245,
+ "2115": 4.55797,
+ "2120": 4.58313,
+ "2125": 4.49019,
+ "2130": 4.49422,
+ "2135": 4.51235,
+ "2140": 4.514,
+ "2145": 4.45457,
+ "2150": 4.50149,
+ "2155": 4.46912,
+ "2160": 4.43112,
+ "2165": 4.56287,
+ "2170": 4.48533,
+ "2175": 4.52236,
+ "2180": 4.50345,
+ "2185": 4.43039,
+ "2190": 4.47437,
+ "2195": 4.51568,
+ "2200": 4.47286,
+ "2205": 4.3956,
+ "2210": 4.48161,
+ "2215": 4.46691,
+ "2220": 4.45638,
+ "2225": 4.4438,
+ "2230": 4.44494,
+ "2235": 4.44948,
+ "2240": 4.43393,
+ "2245": 4.49198,
+ "2250": 4.43813,
+ "2255": 4.52308,
+ "2260": 4.46955,
+ "2265": 4.42867,
+ "2270": 4.41013,
+ "2275": 4.40538,
+ "2280": 4.48383,
+ "2285": 4.45978,
+ "2290": 4.45906,
+ "2295": 4.49431,
+ "2300": 4.48019,
+ "2305": 4.44674,
+ "2310": 4.41156,
+ "2315": 4.4364,
+ "2320": 4.33316,
+ "2325": 4.44819,
+ "2330": 4.45393,
+ "2335": 4.45713,
+ "2340": 4.43146,
+ "2345": 4.45053,
+ "2350": 4.36235,
+ "2355": 4.44763,
+ "2360": 4.47316,
+ "2365": 4.4144,
+ "2370": 4.38341,
+ "2375": 4.46164,
+ "2380": 4.44698,
+ "2385": 4.35711,
+ "2390": 4.40708,
+ "2395": 4.37616,
+ "2400": 4.41473,
+ "2405": 4.44932,
+ "2410": 4.36001,
+ "2415": 4.50803,
+ "2420": 4.44674,
+ "2425": 4.40048,
+ "2430": 4.46931,
+ "2435": 4.3642,
+ "2440": 4.41901,
+ "2445": 4.41586,
+ "2450": 4.40808,
+ "2455": 4.46976,
+ "2460": 4.4024,
+ "2465": 4.42408,
+ "2470": 4.41954,
+ "2475": 4.38572,
+ "2480": 4.46519,
+ "2485": 4.46026,
+ "2490": 4.35562,
+ "2495": 4.36675,
+ "2500": 4.42544,
+ "2505": 4.38207,
+ "2510": 4.35357,
+ "2515": 4.37236,
+ "2520": 4.43953,
+ "2525": 4.33191,
+ "2530": 4.39049,
+ "2535": 4.44422,
+ "2540": 4.37879,
+ "2545": 4.35463,
+ "2550": 4.40344,
+ "2555": 4.39915,
+ "2560": 4.41186,
+ "2565": 4.38478,
+ "2570": 4.43972,
+ "2575": 4.39122,
+ "2580": 4.37909,
+ "2585": 4.3875,
+ "2590": 4.41846,
+ "2595": 4.33351,
+ "2600": 4.37468,
+ "2605": 4.43425,
+ "2610": 4.3832,
+ "2615": 4.36272,
+ "2620": 4.41136,
+ "2625": 4.43227,
+ "2630": 4.37736,
+ "2635": 4.40664,
+ "2640": 4.40595,
+ "2645": 4.38568,
+ "2650": 4.37312,
+ "2655": 4.41973,
+ "2660": 4.32712,
+ "2665": 4.27889,
+ "2670": 4.3758,
+ "2675": 4.29011,
+ "2680": 4.33746,
+ "2685": 4.31911,
+ "2690": 4.33859,
+ "2695": 4.27418,
+ "2700": 4.34345,
+ "2705": 4.30891,
+ "2710": 4.31588,
+ "2715": 4.30292,
+ "2720": 4.3503,
+ "2725": 4.33351,
+ "2730": 4.28543,
+ "2735": 4.30222,
+ "2740": 4.39965,
+ "2745": 4.30587,
+ "2750": 4.32586,
+ "2755": 4.42295,
+ "2760": 4.39183,
+ "2765": 4.38673,
+ "2770": 4.34346,
+ "2775": 4.3323,
+ "2780": 4.35649,
+ "2785": 4.28833,
+ "2790": 4.367,
+ "2795": 4.36066,
+ "2800": 4.37417,
+ "2805": 4.23619,
+ "2810": 4.29784,
+ "2815": 4.30864,
+ "2820": 4.34062,
+ "2825": 4.27944,
+ "2830": 4.34203,
+ "2835": 4.39268,
+ "2840": 4.32351,
+ "2845": 4.33238,
+ "2850": 4.31699,
+ "2855": 4.34195,
+ "2860": 4.37341,
+ "2865": 4.30534,
+ "2870": 4.30538,
+ "2875": 4.28707,
+ "2880": 4.2918,
+ "2885": 4.28004,
+ "2890": 4.27596,
+ "2895": 4.36243,
+ "2900": 4.28955,
+ "2905": 4.34758,
+ "2910": 4.28887,
+ "2915": 4.30973,
+ "2920": 4.29685,
+ "2925": 4.3276,
+ "2930": 4.29027,
+ "2935": 4.3264,
+ "2940": 4.32579,
+ "2945": 4.28513,
+ "2950": 4.24246,
+ "2955": 4.31595,
+ "2960": 4.21782,
+ "2965": 4.30204,
+ "2970": 4.31666,
+ "2975": 4.29965,
+ "2980": 4.26783,
+ "2985": 4.33124,
+ "2990": 4.25236,
+ "2995": 4.35178,
+ "3000": 4.22126,
+ "3005": 4.25027,
+ "3010": 4.31805,
+ "3015": 4.25513,
+ "3020": 4.26967,
+ "3025": 4.25007,
+ "3030": 4.24379,
+ "3035": 4.26866,
+ "3040": 4.25428,
+ "3045": 4.24621,
+ "3050": 4.30963,
+ "3055": 4.26861,
+ "3060": 4.24531,
+ "3065": 4.30289,
+ "3070": 4.24171,
+ "3075": 4.27193,
+ "3080": 4.21395,
+ "3085": 4.26482,
+ "3090": 4.23437,
+ "3095": 4.28693,
+ "3100": 4.32923,
+ "3105": 4.23148,
+ "3110": 4.24017,
+ "3115": 4.23017,
+ "3120": 4.30801,
+ "3125": 4.22123,
+ "3130": 4.24202,
+ "3135": 4.24195,
+ "3140": 4.26352,
+ "3145": 4.23895,
+ "3150": 4.2845,
+ "3155": 4.20332,
+ "3160": 4.28259,
+ "3165": 4.25178,
+ "3170": 4.19661,
+ "3175": 4.305,
+ "3180": 4.20698,
+ "3185": 4.18284,
+ "3190": 4.238,
+ "3195": 4.22101,
+ "3200": 4.26205,
+ "3205": 4.2274,
+ "3210": 4.22971,
+ "3215": 4.25478,
+ "3220": 4.20128,
+ "3225": 4.25509,
+ "3230": 4.23451,
+ "3235": 4.25631,
+ "3240": 4.18699,
+ "3245": 4.21861,
+ "3250": 4.29631,
+ "3255": 4.26241,
+ "3260": 4.20843,
+ "3265": 4.14406,
+ "3270": 4.2146,
+ "3275": 4.23503,
+ "3280": 4.24474,
+ "3285": 4.19823,
+ "3290": 4.24452,
+ "3295": 4.29653,
+ "3300": 4.28796,
+ "3305": 4.20274,
+ "3310": 4.25659,
+ "3315": 4.20445,
+ "3320": 4.25604,
+ "3325": 4.24521,
+ "3330": 4.24841,
+ "3335": 4.15592,
+ "3340": 4.24682,
+ "3345": 4.22063,
+ "3350": 4.25898,
+ "3355": 4.18246,
+ "3360": 4.14873,
+ "3365": 4.17793,
+ "3370": 4.1828,
+ "3375": 4.23422,
+ "3380": 4.19182,
+ "3385": 4.17824,
+ "3390": 4.19292,
+ "3395": 4.19439,
+ "3400": 4.16509,
+ "3405": 4.27491,
+ "3410": 4.17725,
+ "3415": 4.22617,
+ "3420": 4.1509,
+ "3425": 4.16781,
+ "3430": 4.19158,
+ "3435": 4.18525,
+ "3440": 4.22877,
+ "3445": 4.21499,
+ "3450": 4.23222,
+ "3455": 4.13823,
+ "3460": 4.15052,
+ "3465": 4.18078,
+ "3470": 4.1525,
+ "3475": 4.19416,
+ "3480": 4.23229,
+ "3485": 4.14076,
+ "3490": 4.23959,
+ "3495": 4.16724,
+ "3500": 4.15276,
+ "3505": 4.12489,
+ "3510": 4.18597,
+ "3515": 4.21082,
+ "3520": 4.14916,
+ "3525": 4.20029,
+ "3530": 4.23638,
+ "3535": 4.24227,
+ "3540": 4.12978,
+ "3545": 4.11752,
+ "3550": 4.18289,
+ "3555": 4.07565,
+ "3560": 4.10178,
+ "3565": 4.18244,
+ "3570": 4.18631,
+ "3575": 4.22151,
+ "3580": 4.12928,
+ "3585": 4.13157,
+ "3590": 4.13551,
+ "3595": 4.22328,
+ "3600": 4.16218,
+ "3605": 4.11479,
+ "3610": 4.16076,
+ "3615": 4.17481,
+ "3620": 4.18507,
+ "3625": 4.15747,
+ "3630": 4.06713,
+ "3635": 4.10508,
+ "3640": 4.13571,
+ "3645": 4.17695,
+ "3650": 4.11143,
+ "3655": 4.17768,
+ "3660": 4.0939,
+ "3665": 4.23142,
+ "3670": 4.17573,
+ "3675": 4.17312,
+ "3680": 4.13176,
+ "3685": 4.12373,
+ "3690": 4.06898,
+ "3695": 4.13323,
+ "3700": 4.15518,
+ "3705": 4.1313,
+ "3710": 4.1591,
+ "3715": 4.11361,
+ "3720": 4.05404,
+ "3725": 4.17572,
+ "3730": 4.13823,
+ "3735": 4.1374,
+ "3740": 4.08643,
+ "3745": 4.07593,
+ "3750": 4.08455,
+ "3755": 3.98679,
+ "3760": 4.13398,
+ "3765": 4.10747,
+ "3770": 4.07466,
+ "3775": 4.09411,
+ "3780": 4.13302,
+ "3785": 4.03506,
+ "3790": 4.14719,
+ "3795": 4.16785,
+ "3800": 4.13739,
+ "3805": 4.10785,
+ "3810": 4.12622,
+ "3815": 4.10852,
+ "3820": 4.15209,
+ "3825": 4.0675,
+ "3830": 4.02571,
+ "3835": 4.12846,
+ "3840": 4.07719,
+ "3845": 4.12571,
+ "3850": 4.08206,
+ "3855": 4.06548,
+ "3860": 4.14609,
+ "3865": 4.09582,
+ "3870": 4.18183,
+ "3875": 4.07501,
+ "3880": 4.15013,
+ "3885": 4.0712,
+ "3890": 4.10715,
+ "3895": 4.11649,
+ "3900": 4.11744,
+ "3905": 4.09793,
+ "3910": 4.14315,
+ "3915": 4.07271,
+ "3920": 4.10394,
+ "3925": 4.05934,
+ "3930": 4.10847,
+ "3935": 4.09814,
+ "3940": 4.13477,
+ "3945": 4.12002,
+ "3950": 4.1024,
+ "3955": 4.10915,
+ "3960": 4.06286,
+ "3965": 4.11105,
+ "3970": 4.03796,
+ "3975": 4.11092,
+ "3980": 4.03602,
+ "3985": 4.13894,
+ "3990": 4.10627,
+ "3995": 4.15195,
+ "4000": 4.07567,
+ "4005": 4.11031,
+ "4010": 4.06885,
+ "4015": 4.13442,
+ "4020": 4.13258,
+ "4025": 4.08344,
+ "4030": 4.09891,
+ "4035": 4.07746,
+ "4040": 4.08629,
+ "4045": 4.12042,
+ "4050": 4.16709,
+ "4055": 4.06068,
+ "4060": 4.09684,
+ "4065": 4.11785,
+ "4070": 4.10111,
+ "4075": 4.07772,
+ "4080": 3.99533,
+ "4085": 4.06122,
+ "4090": 4.12753,
+ "4095": 4.03309,
+ "4100": 4.07507,
+ "4105": 4.06648,
+ "4110": 4.04117,
+ "4115": 4.08931,
+ "4120": 4.09145,
+ "4125": 4.03195,
+ "4130": 4.04741,
+ "4135": 4.00654,
+ "4140": 4.09841,
+ "4145": 4.07132,
+ "4150": 4.02335,
+ "4155": 4.08403,
+ "4160": 4.00765,
+ "4165": 4.12986,
+ "4170": 4.09717,
+ "4175": 4.08868,
+ "4180": 4.1327,
+ "4185": 4.02372,
+ "4190": 4.03805,
+ "4195": 4.00586,
+ "4200": 4.09163,
+ "4205": 4.07527,
+ "4210": 4.04241,
+ "4215": 4.08333,
+ "4220": 4.09021,
+ "4225": 4.02743,
+ "4230": 4.06102,
+ "4235": 4.08189,
+ "4240": 3.99783,
+ "4245": 3.99004,
+ "4250": 4.04104,
+ "4255": 3.99634,
+ "4260": 4.01871,
+ "4265": 4.02626,
+ "4270": 3.99767,
+ "4275": 3.99513,
+ "4280": 4.07798,
+ "4285": 4.00349,
+ "4290": 3.96863,
+ "4295": 4.06779,
+ "4300": 3.96234,
+ "4305": 4.0369,
+ "4310": 4.09875,
+ "4315": 4.01874,
+ "4320": 4.06815,
+ "4325": 3.95004,
+ "4330": 4.03865,
+ "4335": 3.9892,
+ "4340": 4.03784,
+ "4345": 4.01796,
+ "4350": 4.05107,
+ "4355": 3.99237,
+ "4360": 3.98164,
+ "4365": 3.99508,
+ "4370": 4.0174,
+ "4375": 4.04336,
+ "4380": 4.02624,
+ "4385": 4.03367,
+ "4390": 4.04815,
+ "4395": 4.04338,
+ "4400": 4.04518,
+ "4405": 3.96033,
+ "4410": 3.99736,
+ "4415": 3.9979,
+ "4420": 4.0297,
+ "4425": 4.06808,
+ "4430": 3.95422,
+ "4435": 4.01626,
+ "4440": 3.98142,
+ "4445": 3.98399,
+ "4450": 3.93556,
+ "4455": 4.02788,
+ "4460": 3.91977,
+ "4465": 4.0109,
+ "4470": 3.99668,
+ "4475": 3.98255,
+ "4480": 4.05393,
+ "4485": 3.94309,
+ "4490": 3.92581,
+ "4495": 3.95859,
+ "4500": 3.97867,
+ "4505": 4.04,
+ "4510": 4.07623,
+ "4515": 3.9959,
+ "4520": 4.01708,
+ "4525": 3.9261,
+ "4530": 4.05145,
+ "4535": 3.97511,
+ "4540": 3.97856,
+ "4545": 3.94619,
+ "4550": 3.98668,
+ "4555": 4.0127,
+ "4560": 3.98697,
+ "4565": 3.98466,
+ "4570": 3.96044,
+ "4575": 3.94834,
+ "4580": 3.9363,
+ "4585": 4.03478,
+ "4590": 3.91317,
+ "4595": 4.01873,
+ "4600": 3.96146,
+ "4605": 3.94273,
+ "4610": 3.9145,
+ "4615": 3.94668,
+ "4620": 3.98377,
+ "4625": 3.98344,
+ "4630": 4.0161,
+ "4635": 3.96746,
+ "4640": 4.02648,
+ "4645": 3.99702,
+ "4650": 3.96587,
+ "4655": 3.9956,
+ "4660": 3.99201,
+ "4665": 3.93617,
+ "4670": 3.9488,
+ "4675": 3.95699,
+ "4680": 4.01655,
+ "4685": 3.93244,
+ "4690": 3.92392,
+ "4695": 3.96645,
+ "4700": 3.96368,
+ "4705": 3.94586,
+ "4710": 3.94822,
+ "4715": 3.96157,
+ "4720": 3.98896,
+ "4725": 3.96405,
+ "4730": 3.96398,
+ "4735": 3.95982,
+ "4740": 3.92778,
+ "4745": 3.93557,
+ "4750": 3.97155,
+ "4755": 3.99109,
+ "4760": 3.92608,
+ "4765": 3.8874,
+ "4770": 3.96393,
+ "4775": 3.97531,
+ "4780": 3.92213,
+ "4785": 3.99383,
+ "4790": 3.97135,
+ "4795": 3.9662,
+ "4800": 3.96044,
+ "4805": 3.9379,
+ "4810": 3.93529,
+ "4815": 4.02335,
+ "4820": 3.95373,
+ "4825": 3.8786,
+ "4830": 3.96803,
+ "4835": 3.93817,
+ "4840": 3.959,
+ "4845": 3.89322,
+ "4850": 3.95333,
+ "4855": 3.9087,
+ "4860": 3.92142,
+ "4865": 3.96204,
+ "4870": 3.96502,
+ "4875": 3.90423,
+ "4880": 3.96569,
+ "4885": 3.93421,
+ "4890": 4.01207,
+ "4895": 3.98696,
+ "4900": 3.90517,
+ "4905": 3.91283,
+ "4910": 3.9242,
+ "4915": 3.95004,
+ "4920": 3.94261,
+ "4925": 3.8883,
+ "4930": 3.87324,
+ "4935": 3.92074,
+ "4940": 3.89339,
+ "4945": 4.02103,
+ "4950": 3.8799,
+ "4955": 3.90128,
+ "4960": 3.90793,
+ "4965": 3.91786,
+ "4970": 3.94055,
+ "4975": 3.96248,
+ "4980": 3.93694,
+ "4985": 3.88853,
+ "4990": 3.88737,
+ "4995": 3.93503,
+ "5000": 3.89642,
+ "5005": 3.98184,
+ "5010": 3.94132,
+ "5015": 3.8755,
+ "5020": 3.91263,
+ "5025": 3.9462,
+ "5030": 3.85955,
+ "5035": 3.93803,
+ "5040": 3.87701,
+ "5045": 3.91926,
+ "5050": 3.87907,
+ "5055": 3.95906,
+ "5060": 3.95434,
+ "5065": 3.8793,
+ "5070": 3.90575,
+ "5075": 3.8611,
+ "5080": 3.91,
+ "5085": 3.92093,
+ "5090": 3.95191,
+ "5095": 3.8749,
+ "5100": 3.91539,
+ "5105": 3.93434,
+ "5110": 3.90326,
+ "5115": 3.85404,
+ "5120": 3.79401,
+ "5125": 3.90393,
+ "5130": 3.81418,
+ "5135": 3.88868,
+ "5140": 3.87556,
+ "5145": 3.83712,
+ "5150": 3.8787,
+ "5155": 3.89971,
+ "5160": 3.86685,
+ "5165": 3.93238,
+ "5170": 3.87219,
+ "5175": 3.89063,
+ "5180": 3.88541,
+ "5185": 3.95847,
+ "5190": 3.8774,
+ "5195": 3.87746,
+ "5200": 3.86692,
+ "5205": 3.84123,
+ "5210": 3.81744,
+ "5215": 3.80057,
+ "5220": 3.87691,
+ "5225": 3.88183,
+ "5230": 3.86878,
+ "5235": 3.88089,
+ "5240": 3.86046,
+ "5245": 3.84468,
+ "5250": 3.83607,
+ "5255": 3.95795,
+ "5260": 3.87022,
+ "5265": 3.88644,
+ "5270": 3.89624,
+ "5275": 3.89515,
+ "5280": 3.83578,
+ "5285": 3.91821,
+ "5290": 3.88595,
+ "5295": 3.87231,
+ "5300": 3.86093,
+ "5305": 3.88515,
+ "5310": 3.8582,
+ "5315": 3.81741,
+ "5320": 3.87733,
+ "5325": 3.84866,
+ "5330": 3.85341,
+ "5335": 3.88844,
+ "5340": 3.93499,
+ "5345": 3.85979,
+ "5350": 3.90037,
+ "5355": 3.89742,
+ "5360": 3.923,
+ "5365": 3.85202,
+ "5370": 3.80575,
+ "5375": 3.8934,
+ "5380": 3.79276,
+ "5385": 3.84564,
+ "5390": 3.81998,
+ "5395": 3.77963,
+ "5400": 3.89714,
+ "5405": 3.89767,
+ "5410": 3.85425,
+ "5415": 3.85532,
+ "5420": 3.90698,
+ "5425": 3.78363,
+ "5430": 3.88213,
+ "5435": 3.853,
+ "5440": 3.83905,
+ "5445": 3.78618,
+ "5450": 3.78322,
+ "5455": 3.78017,
+ "5460": 3.7813,
+ "5465": 3.82595,
+ "5470": 3.8724,
+ "5475": 3.77831,
+ "5480": 3.92076,
+ "5485": 3.85344,
+ "5490": 3.7958,
+ "5495": 3.82748,
+ "5500": 3.85562,
+ "5505": 3.87739,
+ "5510": 3.85693,
+ "5515": 3.85035,
+ "5520": 3.88153,
+ "5525": 3.84441,
+ "5530": 3.81919,
+ "5535": 3.81629,
+ "5540": 3.79298,
+ "5545": 3.87961,
+ "5550": 3.78893,
+ "5555": 3.8766,
+ "5560": 3.83823,
+ "5565": 3.80475,
+ "5570": 3.91643,
+ "5575": 3.86016,
+ "5580": 3.78753,
+ "5585": 3.8011,
+ "5590": 3.8212,
+ "5595": 3.84544,
+ "5600": 3.80037,
+ "5605": 3.78597,
+ "5610": 3.80207,
+ "5615": 3.74348,
+ "5620": 3.79289,
+ "5625": 3.79372,
+ "5630": 3.81072,
+ "5635": 3.79036,
+ "5640": 3.81213,
+ "5645": 3.82907,
+ "5650": 3.79987,
+ "5655": 3.82181,
+ "5660": 3.83562,
+ "5665": 3.82007,
+ "5670": 3.80975,
+ "5675": 3.84837,
+ "5680": 3.82712,
+ "5685": 3.80538,
+ "5690": 3.7783,
+ "5695": 3.86887,
+ "5700": 3.84075,
+ "5705": 3.79278,
+ "5710": 3.77762,
+ "5715": 3.78043,
+ "5720": 3.82044,
+ "5725": 3.83724,
+ "5730": 3.88875,
+ "5735": 3.78949,
+ "5740": 3.89389,
+ "5745": 3.83466,
+ "5750": 3.78546,
+ "5755": 3.84204,
+ "5760": 3.92068,
+ "5765": 3.79446,
+ "5770": 3.87579,
+ "5775": 3.78777,
+ "5780": 3.76695,
+ "5785": 3.84009,
+ "5790": 3.80168,
+ "5795": 3.80761,
+ "5800": 3.81834,
+ "5805": 3.80338,
+ "5810": 3.76274,
+ "5815": 3.80576,
+ "5820": 3.80576,
+ "5825": 3.7495,
+ "5830": 3.76423,
+ "5835": 3.73525,
+ "5840": 3.74656,
+ "5845": 3.81958,
+ "5850": 3.86098,
+ "5855": 3.72004,
+ "5860": 3.76578,
+ "5865": 3.76901,
+ "5870": 3.82867,
+ "5875": 3.77642,
+ "5880": 3.81761,
+ "5885": 3.74847,
+ "5890": 3.7856,
+ "5895": 3.72057,
+ "5900": 3.75574,
+ "5905": 3.76963,
+ "5910": 3.77428,
+ "5915": 3.72425,
+ "5920": 3.79122,
+ "5925": 3.7615,
+ "5930": 3.74765,
+ "5935": 3.74547,
+ "5940": 3.75824,
+ "5945": 3.78505,
+ "5950": 3.78211,
+ "5955": 3.808,
+ "5960": 3.76946,
+ "5965": 3.81091,
+ "5970": 3.73625,
+ "5975": 3.74892,
+ "5980": 3.73408,
+ "5985": 3.81086,
+ "5990": 3.82735,
+ "5995": 3.72207,
+ "6000": 3.71999,
+ "6005": 3.72812,
+ "6010": 3.75112,
+ "6015": 3.77777,
+ "6020": 3.7631,
+ "6025": 3.72233,
+ "6030": 3.79075,
+ "6035": 3.7826,
+ "6040": 3.59767,
+ "6045": 3.74448,
+ "6050": 3.68644,
+ "6055": 3.72058,
+ "6060": 3.74475,
+ "6065": 3.79522,
+ "6070": 3.72115,
+ "6075": 3.83568,
+ "6080": 3.72489,
+ "6085": 3.7565,
+ "6090": 3.73754,
+ "6095": 3.80333,
+ "6100": 3.70037,
+ "6105": 3.76789,
+ "6110": 3.70176,
+ "6115": 3.66795,
+ "6120": 3.68623,
+ "6125": 3.77625,
+ "6130": 3.721,
+ "6135": 3.72892,
+ "6140": 3.68942,
+ "6145": 3.66153,
+ "6150": 3.74429,
+ "6155": 3.70904,
+ "6160": 3.71919,
+ "6165": 3.76196,
+ "6170": 3.75498,
+ "6175": 3.70516,
+ "6180": 3.68488,
+ "6185": 3.77908,
+ "6190": 3.71575,
+ "6195": 3.74819,
+ "6200": 3.68086,
+ "6205": 3.69133,
+ "6210": 3.69776,
+ "6215": 3.77806,
+ "6220": 3.67494,
+ "6225": 3.62509,
+ "6230": 3.67324,
+ "6235": 3.66956,
+ "6240": 3.72655,
+ "6245": 3.71846,
+ "6250": 3.74448,
+ "6255": 3.67906,
+ "6260": 3.69955,
+ "6265": 3.75518,
+ "6270": 3.74026,
+ "6275": 3.67958,
+ "6280": 3.66393,
+ "6285": 3.68813,
+ "6290": 3.78248,
+ "6295": 3.68369,
+ "6300": 3.66839,
+ "6305": 3.65298,
+ "6310": 3.69071,
+ "6315": 3.73893,
+ "6320": 3.71583,
+ "6325": 3.65373,
+ "6330": 3.71955,
+ "6335": 3.6644,
+ "6340": 3.62769,
+ "6345": 3.64045,
+ "6350": 3.71678,
+ "6355": 3.73529,
+ "6360": 3.67756,
+ "6365": 3.64997,
+ "6370": 3.71229,
+ "6375": 3.70068,
+ "6380": 3.6978,
+ "6385": 3.66104,
+ "6390": 3.7039,
+ "6395": 3.70114,
+ "6400": 3.62999,
+ "6405": 3.69768,
+ "6410": 3.71836,
+ "6415": 3.64196,
+ "6420": 3.62717,
+ "6425": 3.72279,
+ "6430": 3.69813,
+ "6435": 3.72959,
+ "6440": 3.65232,
+ "6445": 3.67127,
+ "6450": 3.70032,
+ "6455": 3.62131,
+ "6460": 3.65367,
+ "6465": 3.64751,
+ "6470": 3.70033,
+ "6475": 3.60484,
+ "6480": 3.65142,
+ "6485": 3.59998,
+ "6490": 3.64795,
+ "6495": 3.64642,
+ "6500": 3.61973,
+ "6505": 3.75499,
+ "6510": 3.68629,
+ "6515": 3.69601,
+ "6520": 3.70393,
+ "6525": 3.63989,
+ "6530": 3.70465,
+ "6535": 3.71827,
+ "6540": 3.65912,
+ "6545": 3.63606,
+ "6550": 3.68576,
+ "6555": 3.67001,
+ "6560": 3.66095,
+ "6565": 3.59585,
+ "6570": 3.6113,
+ "6575": 3.6015,
+ "6580": 3.62517,
+ "6585": 3.70869,
+ "6590": 3.63788,
+ "6595": 3.64203,
+ "6600": 3.67546,
+ "6605": 3.6274,
+ "6610": 3.6506,
+ "6615": 3.64637,
+ "6620": 3.62523,
+ "6625": 3.62609,
+ "6630": 3.60413,
+ "6635": 3.67962,
+ "6640": 3.6282,
+ "6645": 3.6709,
+ "6650": 3.67407,
+ "6655": 3.6331,
+ "6660": 3.72593,
+ "6665": 3.63461,
+ "6670": 3.59253,
+ "6675": 3.72278,
+ "6680": 3.57421,
+ "6685": 3.60042,
+ "6690": 3.65416,
+ "6695": 3.62365,
+ "6700": 3.61204,
+ "6705": 3.62796,
+ "6710": 3.64208,
+ "6715": 3.64883,
+ "6720": 3.64282,
+ "6725": 3.6773,
+ "6730": 3.50297,
+ "6735": 3.6602,
+ "6740": 3.66238,
+ "6745": 3.59637,
+ "6750": 3.5933,
+ "6755": 3.68567,
+ "6760": 3.60772,
+ "6765": 3.63948,
+ "6770": 3.61203,
+ "6775": 3.62955,
+ "6780": 3.62136,
+ "6785": 3.63685,
+ "6790": 3.62904,
+ "6795": 3.62025,
+ "6800": 3.61589,
+ "6805": 3.54693,
+ "6810": 3.61795,
+ "6815": 3.62899,
+ "6820": 3.60551,
+ "6825": 3.66005,
+ "6830": 3.62524,
+ "6835": 3.58864,
+ "6840": 3.59881,
+ "6845": 3.6088,
+ "6850": 3.63634,
+ "6855": 3.59903,
+ "6860": 3.52349,
+ "6865": 3.63091,
+ "6870": 3.62102,
+ "6875": 3.66041,
+ "6880": 3.59619,
+ "6885": 3.51712,
+ "6890": 3.61751,
+ "6895": 3.55489,
+ "6900": 3.5628,
+ "6905": 3.54888,
+ "6910": 3.54538,
+ "6915": 3.58959,
+ "6920": 3.53507,
+ "6925": 3.56445,
+ "6930": 3.63621,
+ "6935": 3.57437,
+ "6940": 3.61652,
+ "6945": 3.5881,
+ "6950": 3.46509,
+ "6955": 3.60171,
+ "6960": 3.60777,
+ "6965": 3.58732,
+ "6970": 3.56151,
+ "6975": 3.65424,
+ "6980": 3.54644,
+ "6985": 3.54325,
+ "6990": 3.60162,
+ "6995": 3.474,
+ "7000": 3.51538,
+ "7005": 3.5298,
+ "7010": 3.59537,
+ "7015": 3.56136,
+ "7020": 3.53491,
+ "7025": 3.60792,
+ "7030": 3.55911,
+ "7035": 3.54955,
+ "7040": 3.58781,
+ "7045": 3.55659,
+ "7050": 3.49251,
+ "7055": 3.54942,
+ "7060": 3.51896,
+ "7065": 3.53085,
+ "7070": 3.63537,
+ "7075": 3.52605,
+ "7080": 3.48807,
+ "7085": 3.57794,
+ "7090": 3.53252,
+ "7095": 3.56878,
+ "7100": 3.52442,
+ "7105": 3.56374,
+ "7110": 3.50997,
+ "7115": 3.52615,
+ "7120": 3.49493,
+ "7125": 3.52584,
+ "7130": 3.47331,
+ "7135": 3.50532,
+ "7140": 3.57464,
+ "7145": 3.56095,
+ "7150": 3.51899,
+ "7155": 3.46688,
+ "7160": 3.49604,
+ "7165": 3.63047,
+ "7170": 3.48121,
+ "7175": 3.58857,
+ "7180": 3.5691,
+ "7185": 3.5964,
+ "7190": 3.54707,
+ "7195": 3.53401,
+ "7200": 3.52613,
+ "7205": 3.57926,
+ "7210": 3.56306,
+ "7215": 3.55787,
+ "7220": 3.51791,
+ "7225": 3.53283,
+ "7230": 3.56542,
+ "7235": 3.47165,
+ "7240": 3.45474,
+ "7245": 3.51647,
+ "7250": 3.51117,
+ "7255": 3.46447,
+ "7260": 3.53313,
+ "7265": 3.51237,
+ "7270": 3.54338,
+ "7275": 3.56454,
+ "7280": 3.48566,
+ "7285": 3.53203,
+ "7290": 3.50149,
+ "7295": 3.46415,
+ "7300": 3.53853,
+ "7305": 3.53975,
+ "7310": 3.49003,
+ "7315": 3.43979,
+ "7320": 3.50251,
+ "7325": 3.51133,
+ "7330": 3.57191,
+ "7335": 3.55164,
+ "7340": 3.45023,
+ "7345": 3.57228,
+ "7350": 3.49176,
+ "7355": 3.49032,
+ "7360": 3.48369,
+ "7365": 3.50999,
+ "7370": 3.4523,
+ "7375": 3.46708,
+ "7380": 3.49449,
+ "7385": 3.54043,
+ "7390": 3.41909,
+ "7395": 3.52501,
+ "7400": 3.48715,
+ "7405": 3.58005,
+ "7410": 3.40739,
+ "7415": 3.49153,
+ "7420": 3.45772,
+ "7425": 3.44349,
+ "7430": 3.43054,
+ "7435": 3.54587,
+ "7440": 3.50329,
+ "7445": 3.50826,
+ "7450": 3.45768,
+ "7455": 3.46178,
+ "7460": 3.47602,
+ "7465": 3.5254,
+ "7470": 3.47081,
+ "7475": 3.40666,
+ "7480": 3.43753,
+ "7485": 3.38854,
+ "7490": 3.48795,
+ "7495": 3.49051,
+ "7500": 3.39407,
+ "7505": 3.40751,
+ "7510": 3.51324,
+ "7515": 3.512,
+ "7520": 3.51883,
+ "7525": 3.47493,
+ "7530": 3.4731,
+ "7535": 3.36519,
+ "7540": 3.43764,
+ "7545": 3.4791,
+ "7550": 3.47137,
+ "7555": 3.40892,
+ "7560": 3.45804,
+ "7565": 3.45625,
+ "7570": 3.49164,
+ "7575": 3.4911,
+ "7580": 3.41372,
+ "7585": 3.44695,
+ "7590": 3.42887,
+ "7595": 3.47674,
+ "7600": 3.45118,
+ "7605": 3.43952,
+ "7610": 3.47225,
+ "7615": 3.40723,
+ "7620": 3.38321,
+ "7625": 3.49449,
+ "7630": 3.37498,
+ "7635": 3.44619,
+ "7640": 3.50168,
+ "7645": 3.4568,
+ "7650": 3.39634,
+ "7655": 3.41527,
+ "7660": 3.37672,
+ "7665": 3.40264,
+ "7670": 3.37542,
+ "7675": 3.41375,
+ "7680": 3.38996,
+ "7685": 3.44532,
+ "7690": 3.40592,
+ "7695": 3.44347,
+ "7700": 3.43885,
+ "7705": 3.35236,
+ "7710": 3.46134,
+ "7715": 3.44596,
+ "7720": 3.39064,
+ "7725": 3.46339,
+ "7730": 3.38094,
+ "7735": 3.40119,
+ "7740": 3.36481,
+ "7745": 3.37313,
+ "7750": 3.40099,
+ "7755": 3.40221,
+ "7760": 3.43674,
+ "7765": 3.35611,
+ "7770": 3.37254,
+ "7775": 3.36091,
+ "7780": 3.39603,
+ "7785": 3.34807,
+ "7790": 3.36324,
+ "7795": 3.3447,
+ "7800": 3.36116,
+ "7805": 3.35135,
+ "7810": 3.3887,
+ "7815": 3.40401,
+ "7820": 3.39129,
+ "7825": 3.33462,
+ "7830": 3.33707,
+ "7835": 3.31597,
+ "7840": 3.31659,
+ "7845": 3.40692,
+ "7850": 3.2961,
+ "7855": 3.26657,
+ "7860": 3.29978,
+ "7865": 3.29939,
+ "7870": 3.2919,
+ "7875": 3.26125,
+ "7880": 3.30015,
+ "7885": 3.33338,
+ "7890": 3.3034,
+ "7895": 3.2224,
+ "7900": 3.351,
+ "7905": 3.25547,
+ "7910": 3.25081,
+ "7915": 3.33258,
+ "7920": 3.29191,
+ "7925": 3.29955,
+ "7930": 3.27252,
+ "7935": 3.31456,
+ "7940": 3.27509,
+ "7945": 3.16313,
+ "7950": 3.2553,
+ "7955": 3.23155,
+ "7960": 3.19618,
+ "7965": 3.2654,
+ "7970": 3.29718,
+ "7975": 3.3041,
+ "7980": 3.24972,
+ "7985": 3.27419,
+ "7990": 3.26304,
+ "7995": 3.26896,
+ "8000": 3.20174,
+ "8005": 3.2351,
+ "8010": 3.26525,
+ "8015": 3.23019,
+ "8020": 3.28101,
+ "8025": 3.19867,
+ "8030": 3.2564,
+ "8035": 3.1643,
+ "8040": 3.22515,
+ "8045": 3.2746,
+ "8050": 3.254,
+ "8055": 3.26881,
+ "8060": 3.28016,
+ "8065": 3.1881,
+ "8070": 3.21356,
+ "8075": 3.22354,
+ "8080": 3.16626,
+ "8085": 3.16369,
+ "8090": 3.19929,
+ "8095": 3.16717,
+ "8100": 3.2157,
+ "8105": 3.21146,
+ "8110": 3.15722,
+ "8115": 3.15971,
+ "8120": 3.17492,
+ "8125": 3.20066,
+ "8130": 3.17685,
+ "8135": 3.22572,
+ "8140": 3.1462,
+ "8145": 3.20659,
+ "8150": 3.27097,
+ "8155": 3.19468,
+ "8160": 3.17546,
+ "8165": 3.17715,
+ "8170": 3.24992,
+ "8175": 3.17887,
+ "8180": 3.21589,
+ "8185": 3.16181,
+ "8190": 3.0422,
+ "8195": 3.22733,
+ "8200": 3.13241,
+ "8205": 3.15377,
+ "8210": 3.18169,
+ "8215": 3.13894,
+ "8220": 3.19046,
+ "8225": 3.21643,
+ "8230": 3.20372,
+ "8235": 3.19009,
+ "8240": 3.17354,
+ "8245": 3.14561,
+ "8250": 3.20224,
+ "8255": 3.18292,
+ "8260": 3.19735,
+ "8265": 3.1663,
+ "8270": 3.13552,
+ "8275": 3.19339,
+ "8280": 3.14024,
+ "8285": 3.18806,
+ "8290": 3.13996,
+ "8295": 3.11815,
+ "8300": 3.26308,
+ "8305": 3.10245,
+ "8310": 3.14159,
+ "8315": 3.13698,
+ "8320": 3.12748,
+ "8325": 3.12582,
+ "8330": 3.11988,
+ "8335": 3.07856,
+ "8340": 3.06752,
+ "8345": 3.12874,
+ "8350": 3.12254,
+ "8355": 3.18031,
+ "8360": 3.13903,
+ "8365": 3.15154,
+ "8370": 3.16924,
+ "8375": 3.07953,
+ "8380": 3.13997,
+ "8385": 3.06502,
+ "8390": 3.10852,
+ "8395": 3.07739,
+ "8400": 3.0828,
+ "8405": 3.06331,
+ "8410": 3.10589,
+ "8415": 3.12834,
+ "8420": 3.0838,
+ "8425": 3.11586,
+ "8430": 3.12859,
+ "8435": 3.12856,
+ "8440": 3.14824,
+ "8445": 3.06011,
+ "8450": 3.06665,
+ "8455": 3.12296,
+ "8460": 3.09292,
+ "8465": 3.09195,
+ "8470": 3.0569,
+ "8475": 3.07022,
+ "8480": 3.15746,
+ "8485": 3.09315,
+ "8490": 3.11908,
+ "8495": 2.96385,
+ "8500": 3.12521,
+ "8505": 3.09613,
+ "8510": 3.06092,
+ "8515": 3.11198,
+ "8520": 3.08242,
+ "8525": 3.12991,
+ "8530": 3.04905,
+ "8535": 3.10929,
+ "8540": 3.12875,
+ "8545": 3.12042,
+ "8550": 3.08551,
+ "8555": 3.04778,
+ "8560": 3.05736,
+ "8565": 3.13461,
+ "8570": 3.11584,
+ "8575": 3.07175,
+ "8580": 3.12644,
+ "8585": 3.06781,
+ "8590": 3.10401,
+ "8595": 3.0231,
+ "8600": 3.09067,
+ "8605": 3.09462,
+ "8610": 3.0909,
+ "8615": 3.06003,
+ "8620": 3.14962,
+ "8625": 3.08784,
+ "8630": 3.07936,
+ "8635": 3.12204,
+ "8640": 3.10252,
+ "8645": 3.13914,
+ "8650": 2.99369,
+ "8655": 3.05103,
+ "8660": 3.11272,
+ "8665": 3.09732,
+ "8670": 3.01974,
+ "8675": 3.03631,
+ "8680": 3.01503,
+ "8685": 3.08163,
+ "8690": 3.01551,
+ "8695": 3.04054,
+ "8700": 3.09953,
+ "8705": 3.00052,
+ "8710": 3.11245,
+ "8715": 3.04593,
+ "8720": 3.06627,
+ "8725": 3.00168,
+ "8730": 3.07992,
+ "8735": 3.02147,
+ "8740": 3.05113,
+ "8745": 2.99083,
+ "8750": 2.95244,
+ "8755": 3.01212,
+ "8760": 3.03327,
+ "8765": 3.04424,
+ "8770": 2.98771,
+ "8775": 3.00883,
+ "8780": 3.0585,
+ "8785": 3.08726,
+ "8790": 3.04395,
+ "8795": 2.95563,
+ "8800": 2.95968,
+ "8805": 2.99775,
+ "8810": 3.10815,
+ "8815": 2.95952,
+ "8820": 2.97636,
+ "8825": 3.01759,
+ "8830": 3.04101,
+ "8835": 3.06645,
+ "8840": 3.01573,
+ "8845": 3.06375,
+ "8850": 3.01064,
+ "8855": 2.99983,
+ "8860": 2.98172,
+ "8865": 2.93477,
+ "8870": 3.03435,
+ "8875": 3.02319,
+ "8880": 2.9693,
+ "8885": 2.94788,
+ "8890": 2.99851,
+ "8895": 2.95562,
+ "8900": 2.9878,
+ "8905": 3.0062,
+ "8910": 2.93783,
+ "8915": 2.99902,
+ "8920": 2.95724,
+ "8925": 2.96238,
+ "8930": 3.02703,
+ "8935": 2.97853,
+ "8940": 3.00249,
+ "8945": 3.02143,
+ "8950": 2.98576,
+ "8955": 3.00818,
+ "8960": 2.94131,
+ "8965": 2.94395,
+ "8970": 2.98049,
+ "8975": 2.986,
+ "8980": 2.96328,
+ "8985": 2.9805,
+ "8990": 2.91045,
+ "8995": 2.97136,
+ "9000": 2.88042,
+ "9005": 2.95244,
+ "9010": 2.97946,
+ "9015": 2.97742,
+ "9020": 2.95275,
+ "9025": 2.9156,
+ "9030": 3.03546,
+ "9035": 2.99646,
+ "9040": 2.93517,
+ "9045": 3.00882,
+ "9050": 2.91857,
+ "9055": 2.9753,
+ "9060": 2.98644,
+ "9065": 2.93271,
+ "9070": 2.94469,
+ "9075": 2.98122,
+ "9080": 3.05763,
+ "9085": 2.9195,
+ "9090": 2.95755,
+ "9095": 2.95071,
+ "9100": 2.89904,
+ "9105": 2.94749,
+ "9110": 2.89905,
+ "9115": 2.91987,
+ "9120": 2.93974,
+ "9125": 2.97289,
+ "9130": 2.93095,
+ "9135": 2.96125,
+ "9140": 2.91525,
+ "9145": 2.93439,
+ "9150": 2.89035,
+ "9155": 2.90334,
+ "9160": 2.95641,
+ "9165": 2.94019,
+ "9170": 2.99356,
+ "9175": 2.9094,
+ "9180": 2.95834,
+ "9185": 2.90452,
+ "9190": 2.91478,
+ "9195": 2.90828,
+ "9200": 2.94414,
+ "9205": 2.93801,
+ "9210": 2.92984,
+ "9215": 2.99611,
+ "9220": 2.96158,
+ "9225": 2.94602,
+ "9230": 2.91305,
+ "9235": 2.98214,
+ "9240": 2.92829,
+ "9245": 2.97714,
+ "9250": 2.91938,
+ "9255": 2.96502,
+ "9260": 2.86898,
+ "9265": 2.83058,
+ "9270": 2.91289,
+ "9275": 2.96145,
+ "9280": 2.91545,
+ "9285": 2.91734,
+ "9290": 2.87289,
+ "9295": 2.9314,
+ "9300": 2.94237,
+ "9305": 2.98842,
+ "9310": 2.96118,
+ "9315": 2.90266,
+ "9320": 2.89933,
+ "9325": 2.9883,
+ "9330": 2.92867,
+ "9335": 2.95319,
+ "9340": 2.94202,
+ "9345": 2.94638,
+ "9350": 2.90096,
+ "9355": 2.98461,
+ "9360": 2.88465,
+ "9365": 2.90496,
+ "9370": 2.88666,
+ "9375": 2.91118,
+ "9380": 2.89072,
+ "9385": 2.89387,
+ "9390": 2.90725,
+ "9395": 2.88876,
+ "9400": 2.91882,
+ "9405": 2.87,
+ "9410": 2.93217,
+ "9415": 2.88291,
+ "9420": 2.90393,
+ "9425": 2.90399,
+ "9430": 2.90268,
+ "9435": 2.87641,
+ "9440": 2.8581,
+ "9445": 2.82803,
+ "9450": 2.86507,
+ "9455": 2.93473,
+ "9460": 2.80685,
+ "9465": 2.88564,
+ "9470": 2.85621,
+ "9475": 2.85721,
+ "9480": 2.80664,
+ "9485": 2.87736,
+ "9490": 2.86754,
+ "9495": 2.92255,
+ "9500": 2.86093,
+ "9505": 2.90891,
+ "9510": 2.8654,
+ "9515": 2.8756,
+ "9520": 2.84784,
+ "9525": 2.91213,
+ "9530": 2.87907,
+ "9535": 2.86115,
+ "9540": 2.83086,
+ "9545": 2.8876,
+ "9550": 2.94817,
+ "9555": 2.92011,
+ "9560": 2.90837,
+ "9565": 2.9666,
+ "9570": 2.90328,
+ "9575": 2.8902,
+ "9580": 2.86849,
+ "9585": 2.84106,
+ "9590": 2.79893,
+ "9595": 2.83621,
+ "9600": 2.85848,
+ "9605": 2.88944,
+ "9610": 2.93178,
+ "9615": 2.86237,
+ "9620": 2.88815,
+ "9625": 2.80776,
+ "9630": 2.86575,
+ "9635": 2.90042,
+ "9640": 2.91494,
+ "9645": 2.92131,
+ "9650": 2.8455,
+ "9655": 2.77161,
+ "9660": 2.95357,
+ "9665": 2.87329,
+ "9670": 2.9218,
+ "9675": 2.90834,
+ "9680": 2.82953,
+ "9685": 2.79504,
+ "9690": 2.84103,
+ "9695": 2.91562,
+ "9700": 2.86643,
+ "9705": 2.92568,
+ "9710": 2.89057,
+ "9715": 2.84043,
+ "9720": 2.8267,
+ "9725": 2.8616,
+ "9730": 2.93505,
+ "9735": 2.83888,
+ "9740": 2.83365,
+ "9745": 2.85564,
+ "9750": 2.88748,
+ "9755": 2.8808,
+ "9760": 2.80671,
+ "9765": 2.93595,
+ "9770": 2.89879,
+ "9775": 2.85187,
+ "9780": 2.87577,
+ "9785": 2.83863,
+ "9790": 2.78016,
+ "9795": 2.78405,
+ "9800": 2.85774,
+ "9805": 2.84537,
+ "9810": 2.86556,
+ "9815": 2.80709,
+ "9820": 2.81843,
+ "9825": 2.85924,
+ "9830": 2.90831,
+ "9835": 2.81964,
+ "9840": 2.82465,
+ "9845": 2.87016,
+ "9850": 2.7952,
+ "9855": 2.81385,
+ "9860": 2.92471,
+ "9865": 2.83123,
+ "9870": 2.82246,
+ "9875": 2.84042,
+ "9880": 2.8571,
+ "9885": 2.82682,
+ "9890": 2.84516,
+ "9895": 2.85099,
+ "9900": 2.86065,
+ "9905": 2.79607,
+ "9910": 2.86903,
+ "9915": 2.78096,
+ "9920": 2.85961,
+ "9925": 2.81661,
+ "9930": 2.83705,
+ "9935": 2.85207,
+ "9940": 2.889,
+ "9945": 2.78996,
+ "9950": 2.92535,
+ "9955": 2.79057,
+ "9960": 2.90626,
+ "9965": 2.80602,
+ "9970": 2.80965,
+ "9975": 2.87058,
+ "9980": 2.82422,
+ "9985": 2.76524,
+ "9990": 2.81382,
+ "9995": 2.86255,
+ "10000": 2.84033
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 10000,
+ "step_interval": 5,
+ "values": {
+ "1": 40049.0,
+ "5": 40026.0,
+ "10": 41478.0,
+ "15": 37238.0,
+ "20": 40803.0,
+ "25": 40118.0,
+ "30": 40737.0,
+ "35": 39352.0,
+ "40": 41495.0,
+ "45": 42254.0,
+ "50": 38432.0,
+ "55": 39339.0,
+ "60": 42258.0,
+ "65": 39958.0,
+ "70": 40720.0,
+ "75": 41489.0,
+ "80": 40877.0,
+ "85": 40795.0,
+ "90": 40725.0,
+ "95": 40729.0,
+ "100": 41562.0,
+ "105": 40808.0,
+ "110": 38708.0,
+ "115": 41555.0,
+ "120": 39261.0,
+ "125": 40715.0,
+ "130": 40714.0,
+ "135": 37901.0,
+ "140": 38740.0,
+ "145": 40954.0,
+ "150": 40014.0,
+ "155": 40806.0,
+ "160": 40116.0,
+ "165": 38577.0,
+ "170": 37336.0,
+ "175": 40048.0,
+ "180": 40092.0,
+ "185": 38567.0,
+ "190": 39329.0,
+ "195": 41483.0,
+ "200": 41569.0,
+ "205": 40098.0,
+ "210": 40792.0,
+ "215": 40036.0,
+ "220": 40767.0,
+ "225": 39408.0,
+ "230": 41485.0,
+ "235": 37937.0,
+ "240": 39335.0,
+ "245": 38667.0,
+ "250": 40021.0,
+ "255": 41476.0,
+ "260": 39255.0,
+ "265": 41568.0,
+ "270": 40726.0,
+ "275": 40797.0,
+ "280": 41489.0,
+ "285": 41559.0,
+ "290": 40786.0,
+ "295": 38588.0,
+ "300": 38699.0,
+ "305": 41557.0,
+ "310": 39256.0,
+ "315": 40098.0,
+ "320": 39347.0,
+ "325": 40787.0,
+ "330": 40792.0,
+ "335": 40156.0,
+ "340": 37936.0,
+ "345": 39947.0,
+ "350": 40715.0,
+ "355": 39952.0,
+ "360": 41555.0,
+ "365": 37965.0,
+ "370": 40028.0,
+ "375": 41492.0,
+ "380": 38714.0,
+ "385": 37805.0,
+ "390": 39950.0,
+ "395": 40793.0,
+ "400": 39274.0,
+ "405": 40724.0,
+ "410": 39465.0,
+ "415": 40739.0,
+ "420": 41489.0,
+ "425": 39405.0,
+ "430": 39446.0,
+ "435": 42247.0,
+ "440": 35768.0,
+ "445": 40032.0,
+ "450": 41492.0,
+ "455": 39946.0,
+ "460": 38584.0,
+ "465": 39327.0,
+ "470": 40802.0,
+ "475": 40801.0,
+ "480": 40790.0,
+ "485": 39205.0,
+ "490": 38705.0,
+ "495": 40769.0,
+ "500": 40711.0,
+ "505": 41496.0,
+ "510": 37735.0,
+ "515": 40724.0,
+ "520": 40017.0,
+ "525": 39257.0,
+ "530": 38805.0,
+ "535": 40796.0,
+ "540": 40032.0,
+ "545": 41572.0,
+ "550": 40803.0,
+ "555": 40726.0,
+ "560": 39961.0,
+ "565": 40710.0,
+ "570": 40192.0,
+ "575": 39318.0,
+ "580": 40810.0,
+ "585": 40722.0,
+ "590": 40024.0,
+ "595": 40034.0,
+ "600": 40032.0,
+ "605": 40806.0,
+ "610": 40880.0,
+ "615": 39944.0,
+ "620": 41545.0,
+ "625": 40719.0,
+ "630": 42248.0,
+ "635": 39971.0,
+ "640": 37072.0,
+ "645": 40109.0,
+ "650": 39248.0,
+ "655": 39187.0,
+ "660": 37352.0,
+ "665": 40117.0,
+ "670": 40787.0,
+ "675": 39265.0,
+ "680": 39948.0,
+ "685": 39264.0,
+ "690": 42261.0,
+ "695": 38564.0,
+ "700": 40034.0,
+ "705": 38576.0,
+ "710": 42246.0,
+ "715": 42249.0,
+ "720": 41489.0,
+ "725": 40800.0,
+ "730": 38641.0,
+ "735": 40792.0,
+ "740": 41545.0,
+ "745": 41558.0,
+ "750": 40024.0,
+ "755": 38730.0,
+ "760": 38591.0,
+ "765": 39955.0,
+ "770": 40713.0,
+ "775": 40724.0,
+ "780": 39340.0,
+ "785": 41582.0,
+ "790": 42324.0,
+ "795": 41576.0,
+ "800": 40797.0,
+ "805": 37067.0,
+ "810": 39346.0,
+ "815": 40717.0,
+ "820": 43030.0,
+ "825": 40019.0,
+ "830": 39952.0,
+ "835": 41481.0,
+ "840": 38558.0,
+ "845": 41567.0,
+ "850": 38004.0,
+ "855": 36548.0,
+ "860": 41488.0,
+ "865": 39959.0,
+ "870": 35818.0,
+ "875": 38583.0,
+ "880": 39967.0,
+ "885": 40040.0,
+ "890": 39421.0,
+ "895": 40720.0,
+ "900": 40034.0,
+ "905": 40018.0,
+ "910": 40029.0,
+ "915": 39279.0,
+ "920": 40113.0,
+ "925": 40108.0,
+ "930": 40099.0,
+ "935": 40722.0,
+ "940": 40801.0,
+ "945": 39962.0,
+ "950": 37713.0,
+ "955": 40104.0,
+ "960": 40798.0,
+ "965": 40095.0,
+ "970": 40715.0,
+ "975": 39961.0,
+ "980": 40804.0,
+ "985": 30974.0,
+ "990": 39341.0,
+ "995": 42260.0,
+ "1000": 41568.0,
+ "1005": 41494.0,
+ "1010": 41633.0,
+ "1015": 40037.0,
+ "1020": 39254.0,
+ "1025": 40022.0,
+ "1030": 40191.0,
+ "1035": 39403.0,
+ "1040": 40101.0,
+ "1045": 40801.0,
+ "1050": 41497.0,
+ "1055": 37940.0,
+ "1060": 42251.0,
+ "1065": 37195.0,
+ "1070": 39282.0,
+ "1075": 43025.0,
+ "1080": 39364.0,
+ "1085": 40032.0,
+ "1090": 39972.0,
+ "1095": 38516.0,
+ "1100": 38640.0,
+ "1105": 39349.0,
+ "1110": 42323.0,
+ "1115": 40803.0,
+ "1120": 40719.0,
+ "1125": 40730.0,
+ "1130": 38659.0,
+ "1135": 40097.0,
+ "1140": 40137.0,
+ "1145": 39350.0,
+ "1150": 41575.0,
+ "1155": 40716.0,
+ "1160": 41487.0,
+ "1165": 40044.0,
+ "1170": 39257.0,
+ "1175": 40031.0,
+ "1180": 40731.0,
+ "1185": 39277.0,
+ "1190": 39252.0,
+ "1195": 40714.0,
+ "1200": 41557.0,
+ "1205": 38604.0,
+ "1210": 41564.0,
+ "1215": 39332.0,
+ "1220": 41578.0,
+ "1225": 40101.0,
+ "1230": 39349.0,
+ "1235": 41495.0,
+ "1240": 40720.0,
+ "1245": 41499.0,
+ "1250": 39412.0,
+ "1255": 38722.0,
+ "1260": 41480.0,
+ "1265": 41496.0,
+ "1270": 40723.0,
+ "1275": 38450.0,
+ "1280": 40105.0,
+ "1285": 40724.0,
+ "1290": 40031.0,
+ "1295": 40786.0,
+ "1300": 40725.0,
+ "1305": 41501.0,
+ "1310": 41506.0,
+ "1315": 40792.0,
+ "1320": 40733.0,
+ "1325": 40728.0,
+ "1330": 40791.0,
+ "1335": 38673.0,
+ "1340": 40794.0,
+ "1345": 39963.0,
+ "1350": 40786.0,
+ "1355": 40116.0,
+ "1360": 42253.0,
+ "1365": 39404.0,
+ "1370": 39352.0,
+ "1375": 40794.0,
+ "1380": 38649.0,
+ "1385": 38563.0,
+ "1390": 40060.0,
+ "1395": 40796.0,
+ "1400": 40808.0,
+ "1405": 40828.0,
+ "1410": 37800.0,
+ "1415": 40020.0,
+ "1420": 41484.0,
+ "1425": 40814.0,
+ "1430": 39979.0,
+ "1435": 35891.0,
+ "1440": 38558.0,
+ "1445": 39323.0,
+ "1450": 40110.0,
+ "1455": 41557.0,
+ "1460": 40786.0,
+ "1465": 40730.0,
+ "1470": 37187.0,
+ "1475": 40718.0,
+ "1480": 37270.0,
+ "1485": 37828.0,
+ "1490": 38505.0,
+ "1495": 38576.0,
+ "1500": 41484.0,
+ "1505": 40021.0,
+ "1510": 40728.0,
+ "1515": 40797.0,
+ "1520": 40793.0,
+ "1525": 40872.0,
+ "1530": 39293.0,
+ "1535": 39341.0,
+ "1540": 37887.0,
+ "1545": 39978.0,
+ "1550": 40895.0,
+ "1555": 40804.0,
+ "1560": 40806.0,
+ "1565": 37870.0,
+ "1570": 39956.0,
+ "1575": 40040.0,
+ "1580": 38670.0,
+ "1585": 39293.0,
+ "1590": 39345.0,
+ "1595": 37962.0,
+ "1600": 40793.0,
+ "1605": 39335.0,
+ "1610": 40729.0,
+ "1615": 37865.0,
+ "1620": 39276.0,
+ "1625": 41579.0,
+ "1630": 39951.0,
+ "1635": 40798.0,
+ "1640": 41497.0,
+ "1645": 40031.0,
+ "1650": 39342.0,
+ "1655": 41505.0,
+ "1660": 40885.0,
+ "1665": 41562.0,
+ "1670": 40816.0,
+ "1675": 40803.0,
+ "1680": 40882.0,
+ "1685": 39952.0,
+ "1690": 40048.0,
+ "1695": 39332.0,
+ "1700": 40815.0,
+ "1705": 41629.0,
+ "1710": 41506.0,
+ "1715": 39344.0,
+ "1720": 41564.0,
+ "1725": 40048.0,
+ "1730": 39359.0,
+ "1735": 38630.0,
+ "1740": 41499.0,
+ "1745": 37755.0,
+ "1750": 39284.0,
+ "1755": 39329.0,
+ "1760": 39266.0,
+ "1765": 40789.0,
+ "1770": 40796.0,
+ "1775": 39412.0,
+ "1780": 41496.0,
+ "1785": 41563.0,
+ "1790": 40719.0,
+ "1795": 41489.0,
+ "1800": 39401.0,
+ "1805": 40724.0,
+ "1810": 40036.0,
+ "1815": 37718.0,
+ "1820": 40732.0,
+ "1825": 39388.0,
+ "1830": 41505.0,
+ "1835": 39427.0,
+ "1840": 40818.0,
+ "1845": 40805.0,
+ "1850": 40792.0,
+ "1855": 40797.0,
+ "1860": 40723.0,
+ "1865": 40038.0,
+ "1870": 41651.0,
+ "1875": 40730.0,
+ "1880": 39338.0,
+ "1885": 39198.0,
+ "1890": 41574.0,
+ "1895": 40729.0,
+ "1900": 37838.0,
+ "1905": 41510.0,
+ "1910": 41487.0,
+ "1915": 40936.0,
+ "1920": 41574.0,
+ "1925": 39276.0,
+ "1930": 41485.0,
+ "1935": 40797.0,
+ "1940": 40042.0,
+ "1945": 39406.0,
+ "1950": 41503.0,
+ "1955": 41562.0,
+ "1960": 40804.0,
+ "1965": 40105.0,
+ "1970": 39968.0,
+ "1975": 42257.0,
+ "1980": 36507.0,
+ "1985": 40133.0,
+ "1990": 41555.0,
+ "1995": 40040.0,
+ "2000": 41497.0,
+ "2005": 41564.0,
+ "2010": 37826.0,
+ "2015": 38741.0,
+ "2020": 40738.0,
+ "2025": 38694.0,
+ "2030": 38084.0,
+ "2035": 40750.0,
+ "2040": 41584.0,
+ "2045": 40161.0,
+ "2050": 39496.0,
+ "2055": 41561.0,
+ "2060": 39387.0,
+ "2065": 42264.0,
+ "2070": 41495.0,
+ "2075": 39275.0,
+ "2080": 42260.0,
+ "2085": 39309.0,
+ "2090": 40822.0,
+ "2095": 41562.0,
+ "2100": 41570.0,
+ "2105": 41492.0,
+ "2110": 40813.0,
+ "2115": 40727.0,
+ "2120": 41502.0,
+ "2125": 40042.0,
+ "2130": 39328.0,
+ "2135": 39411.0,
+ "2140": 41498.0,
+ "2145": 38583.0,
+ "2150": 40037.0,
+ "2155": 41497.0,
+ "2160": 41564.0,
+ "2165": 40037.0,
+ "2170": 40800.0,
+ "2175": 39342.0,
+ "2180": 40039.0,
+ "2185": 41498.0,
+ "2190": 40873.0,
+ "2195": 38743.0,
+ "2200": 40805.0,
+ "2205": 39958.0,
+ "2210": 39422.0,
+ "2215": 40794.0,
+ "2220": 39271.0,
+ "2225": 40084.0,
+ "2230": 40799.0,
+ "2235": 37275.0,
+ "2240": 40045.0,
+ "2245": 40724.0,
+ "2250": 42263.0,
+ "2255": 40735.0,
+ "2260": 41564.0,
+ "2265": 40834.0,
+ "2270": 40037.0,
+ "2275": 40727.0,
+ "2280": 39206.0,
+ "2285": 40878.0,
+ "2290": 39421.0,
+ "2295": 37905.0,
+ "2300": 41575.0,
+ "2305": 40722.0,
+ "2310": 40037.0,
+ "2315": 38512.0,
+ "2320": 40095.0,
+ "2325": 41486.0,
+ "2330": 40737.0,
+ "2335": 37171.0,
+ "2340": 40737.0,
+ "2345": 40187.0,
+ "2350": 40124.0,
+ "2355": 40869.0,
+ "2360": 39273.0,
+ "2365": 40731.0,
+ "2370": 40739.0,
+ "2375": 38573.0,
+ "2380": 40038.0,
+ "2385": 40812.0,
+ "2390": 40123.0,
+ "2395": 40733.0,
+ "2400": 41562.0,
+ "2405": 40106.0,
+ "2410": 40809.0,
+ "2415": 39187.0,
+ "2420": 39962.0,
+ "2425": 37805.0,
+ "2430": 39274.0,
+ "2435": 41503.0,
+ "2440": 40034.0,
+ "2445": 39364.0,
+ "2450": 40810.0,
+ "2455": 42262.0,
+ "2460": 40106.0,
+ "2465": 39492.0,
+ "2470": 37838.0,
+ "2475": 40047.0,
+ "2480": 40033.0,
+ "2485": 39335.0,
+ "2490": 40109.0,
+ "2495": 41499.0,
+ "2500": 39199.0,
+ "2505": 40739.0,
+ "2510": 40729.0,
+ "2515": 38673.0,
+ "2520": 40816.0,
+ "2525": 41576.0,
+ "2530": 40734.0,
+ "2535": 39972.0,
+ "2540": 39210.0,
+ "2545": 37977.0,
+ "2550": 40809.0,
+ "2555": 41495.0,
+ "2560": 39454.0,
+ "2565": 39194.0,
+ "2570": 39357.0,
+ "2575": 40739.0,
+ "2580": 40741.0,
+ "2585": 40874.0,
+ "2590": 40125.0,
+ "2595": 37968.0,
+ "2600": 39974.0,
+ "2605": 38721.0,
+ "2610": 41556.0,
+ "2615": 40103.0,
+ "2620": 41489.0,
+ "2625": 39288.0,
+ "2630": 40024.0,
+ "2635": 39500.0,
+ "2640": 40786.0,
+ "2645": 40032.0,
+ "2650": 40062.0,
+ "2655": 39187.0,
+ "2660": 37263.0,
+ "2665": 40897.0,
+ "2670": 40884.0,
+ "2675": 38434.0,
+ "2680": 40033.0,
+ "2685": 41574.0,
+ "2690": 40026.0,
+ "2695": 40731.0,
+ "2700": 39276.0,
+ "2705": 41565.0,
+ "2710": 40802.0,
+ "2715": 38507.0,
+ "2720": 40059.0,
+ "2725": 38688.0,
+ "2730": 40880.0,
+ "2735": 40045.0,
+ "2740": 38580.0,
+ "2745": 40037.0,
+ "2750": 40795.0,
+ "2755": 38513.0,
+ "2760": 38519.0,
+ "2765": 39263.0,
+ "2770": 41585.0,
+ "2775": 39349.0,
+ "2780": 40759.0,
+ "2785": 40737.0,
+ "2790": 39294.0,
+ "2795": 40032.0,
+ "2800": 39417.0,
+ "2805": 40799.0,
+ "2810": 41555.0,
+ "2815": 42260.0,
+ "2820": 40727.0,
+ "2825": 40057.0,
+ "2830": 40788.0,
+ "2835": 41557.0,
+ "2840": 40808.0,
+ "2845": 38584.0,
+ "2850": 42261.0,
+ "2855": 40820.0,
+ "2860": 39359.0,
+ "2865": 39355.0,
+ "2870": 40203.0,
+ "2875": 42270.0,
+ "2880": 40794.0,
+ "2885": 39966.0,
+ "2890": 38579.0,
+ "2895": 40731.0,
+ "2900": 39346.0,
+ "2905": 39987.0,
+ "2910": 40731.0,
+ "2915": 40036.0,
+ "2920": 37876.0,
+ "2925": 40800.0,
+ "2930": 36513.0,
+ "2935": 41569.0,
+ "2940": 40811.0,
+ "2945": 38881.0,
+ "2950": 39426.0,
+ "2955": 38032.0,
+ "2960": 41491.0,
+ "2965": 39201.0,
+ "2970": 40809.0,
+ "2975": 38817.0,
+ "2980": 41506.0,
+ "2985": 39280.0,
+ "2990": 42330.0,
+ "2995": 40788.0,
+ "3000": 37965.0,
+ "3005": 42250.0,
+ "3010": 38723.0,
+ "3015": 40800.0,
+ "3020": 41498.0,
+ "3025": 39346.0,
+ "3030": 38449.0,
+ "3035": 40037.0,
+ "3040": 41565.0,
+ "3045": 39340.0,
+ "3050": 41493.0,
+ "3055": 40739.0,
+ "3060": 38494.0,
+ "3065": 39280.0,
+ "3070": 40806.0,
+ "3075": 37988.0,
+ "3080": 39203.0,
+ "3085": 41578.0,
+ "3090": 39282.0,
+ "3095": 39346.0,
+ "3100": 38582.0,
+ "3105": 39987.0,
+ "3110": 37137.0,
+ "3115": 38469.0,
+ "3120": 41643.0,
+ "3125": 40102.0,
+ "3130": 39310.0,
+ "3135": 39283.0,
+ "3140": 39979.0,
+ "3145": 40807.0,
+ "3150": 41493.0,
+ "3155": 38603.0,
+ "3160": 40912.0,
+ "3165": 41488.0,
+ "3170": 39269.0,
+ "3175": 37458.0,
+ "3180": 39434.0,
+ "3185": 42272.0,
+ "3190": 37996.0,
+ "3195": 39260.0,
+ "3200": 38678.0,
+ "3205": 40044.0,
+ "3210": 39956.0,
+ "3215": 38595.0,
+ "3220": 39963.0,
+ "3225": 40121.0,
+ "3230": 40110.0,
+ "3235": 42351.0,
+ "3240": 40026.0,
+ "3245": 40737.0,
+ "3250": 41490.0,
+ "3255": 42263.0,
+ "3260": 40111.0,
+ "3265": 42257.0,
+ "3270": 40796.0,
+ "3275": 41563.0,
+ "3280": 40153.0,
+ "3285": 39263.0,
+ "3290": 38660.0,
+ "3295": 40730.0,
+ "3300": 40151.0,
+ "3305": 42270.0,
+ "3310": 42356.0,
+ "3315": 41493.0,
+ "3320": 41500.0,
+ "3325": 39278.0,
+ "3330": 40809.0,
+ "3335": 40107.0,
+ "3340": 40802.0,
+ "3345": 40801.0,
+ "3350": 40816.0,
+ "3355": 41577.0,
+ "3360": 41495.0,
+ "3365": 40737.0,
+ "3370": 40047.0,
+ "3375": 40726.0,
+ "3380": 40037.0,
+ "3385": 39343.0,
+ "3390": 40030.0,
+ "3395": 39980.0,
+ "3400": 38101.0,
+ "3405": 38660.0,
+ "3410": 39982.0,
+ "3415": 40037.0,
+ "3420": 40806.0,
+ "3425": 40817.0,
+ "3430": 40730.0,
+ "3435": 40055.0,
+ "3440": 41585.0,
+ "3445": 37966.0,
+ "3450": 37910.0,
+ "3455": 40816.0,
+ "3460": 42270.0,
+ "3465": 39363.0,
+ "3470": 38601.0,
+ "3475": 40198.0,
+ "3480": 40828.0,
+ "3485": 38823.0,
+ "3490": 40047.0,
+ "3495": 40031.0,
+ "3500": 40026.0,
+ "3505": 40882.0,
+ "3510": 40807.0,
+ "3515": 40058.0,
+ "3520": 39286.0,
+ "3525": 41566.0,
+ "3530": 40723.0,
+ "3535": 41583.0,
+ "3540": 40726.0,
+ "3545": 40826.0,
+ "3550": 41584.0,
+ "3555": 40118.0,
+ "3560": 40740.0,
+ "3565": 40791.0,
+ "3570": 40813.0,
+ "3575": 40033.0,
+ "3580": 40031.0,
+ "3585": 41591.0,
+ "3590": 41500.0,
+ "3595": 39335.0,
+ "3600": 39340.0,
+ "3605": 39278.0,
+ "3610": 40037.0,
+ "3615": 42260.0,
+ "3620": 38683.0,
+ "3625": 39958.0,
+ "3630": 40808.0,
+ "3635": 40098.0,
+ "3640": 39200.0,
+ "3645": 39343.0,
+ "3650": 38656.0,
+ "3655": 40174.0,
+ "3660": 42334.0,
+ "3665": 41493.0,
+ "3670": 37358.0,
+ "3675": 39430.0,
+ "3680": 37365.0,
+ "3685": 39371.0,
+ "3690": 40818.0,
+ "3695": 42336.0,
+ "3700": 38220.0,
+ "3705": 39486.0,
+ "3710": 39280.0,
+ "3715": 38541.0,
+ "3720": 40816.0,
+ "3725": 37328.0,
+ "3730": 37820.0,
+ "3735": 40803.0,
+ "3740": 40803.0,
+ "3745": 38575.0,
+ "3750": 39428.0,
+ "3755": 39350.0,
+ "3760": 39369.0,
+ "3765": 39375.0,
+ "3770": 42353.0,
+ "3775": 40815.0,
+ "3780": 42255.0,
+ "3785": 37888.0,
+ "3790": 39362.0,
+ "3795": 40030.0,
+ "3800": 39527.0,
+ "3805": 37097.0,
+ "3810": 39436.0,
+ "3815": 40725.0,
+ "3820": 40116.0,
+ "3825": 42263.0,
+ "3830": 40034.0,
+ "3835": 40796.0,
+ "3840": 40042.0,
+ "3845": 39342.0,
+ "3850": 39405.0,
+ "3855": 39359.0,
+ "3860": 40061.0,
+ "3865": 41501.0,
+ "3870": 37759.0,
+ "3875": 39967.0,
+ "3880": 40042.0,
+ "3885": 41499.0,
+ "3890": 39982.0,
+ "3895": 40741.0,
+ "3900": 39287.0,
+ "3905": 38580.0,
+ "3910": 40028.0,
+ "3915": 40794.0,
+ "3920": 36522.0,
+ "3925": 39968.0,
+ "3930": 39243.0,
+ "3935": 39381.0,
+ "3940": 39436.0,
+ "3945": 41510.0,
+ "3950": 39974.0,
+ "3955": 39430.0,
+ "3960": 38644.0,
+ "3965": 39978.0,
+ "3970": 39281.0,
+ "3975": 39965.0,
+ "3980": 40821.0,
+ "3985": 38588.0,
+ "3990": 39214.0,
+ "3995": 39424.0,
+ "4000": 40030.0,
+ "4005": 40041.0,
+ "4010": 35676.0,
+ "4015": 41585.0,
+ "4020": 39279.0,
+ "4025": 40199.0,
+ "4030": 37025.0,
+ "4035": 41506.0,
+ "4040": 34438.0,
+ "4045": 40795.0,
+ "4050": 37758.0,
+ "4055": 39417.0,
+ "4060": 37215.0,
+ "4065": 40728.0,
+ "4070": 38510.0,
+ "4075": 40043.0,
+ "4080": 40881.0,
+ "4085": 41505.0,
+ "4090": 40731.0,
+ "4095": 41506.0,
+ "4100": 37943.0,
+ "4105": 41502.0,
+ "4110": 41502.0,
+ "4115": 40210.0,
+ "4120": 40814.0,
+ "4125": 38706.0,
+ "4130": 40741.0,
+ "4135": 39374.0,
+ "4140": 39348.0,
+ "4145": 40039.0,
+ "4150": 39376.0,
+ "4155": 39440.0,
+ "4160": 40188.0,
+ "4165": 39347.0,
+ "4170": 40750.0,
+ "4175": 39258.0,
+ "4180": 40797.0,
+ "4185": 40793.0,
+ "4190": 39267.0,
+ "4195": 40815.0,
+ "4200": 41494.0,
+ "4205": 40721.0,
+ "4210": 40732.0,
+ "4215": 40049.0,
+ "4220": 38675.0,
+ "4225": 40049.0,
+ "4230": 38797.0,
+ "4235": 39334.0,
+ "4240": 40806.0,
+ "4245": 40748.0,
+ "4250": 40084.0,
+ "4255": 39365.0,
+ "4260": 38659.0,
+ "4265": 40727.0,
+ "4270": 40790.0,
+ "4275": 40122.0,
+ "4280": 40033.0,
+ "4285": 41495.0,
+ "4290": 40745.0,
+ "4295": 37114.0,
+ "4300": 39335.0,
+ "4305": 40023.0,
+ "4310": 40131.0,
+ "4315": 40136.0,
+ "4320": 41577.0,
+ "4325": 40056.0,
+ "4330": 38726.0,
+ "4335": 42266.0,
+ "4340": 40732.0,
+ "4345": 40792.0,
+ "4350": 37987.0,
+ "4355": 41519.0,
+ "4360": 40187.0,
+ "4365": 40827.0,
+ "4370": 37216.0,
+ "4375": 40038.0,
+ "4380": 38491.0,
+ "4385": 40100.0,
+ "4390": 40730.0,
+ "4395": 39974.0,
+ "4400": 39959.0,
+ "4405": 40188.0,
+ "4410": 41587.0,
+ "4415": 41490.0,
+ "4420": 40045.0,
+ "4425": 40192.0,
+ "4430": 40727.0,
+ "4435": 37958.0,
+ "4440": 39971.0,
+ "4445": 39426.0,
+ "4450": 39966.0,
+ "4455": 40729.0,
+ "4460": 40805.0,
+ "4465": 41497.0,
+ "4470": 41493.0,
+ "4475": 39199.0,
+ "4480": 41559.0,
+ "4485": 40116.0,
+ "4490": 42262.0,
+ "4495": 39974.0,
+ "4500": 40040.0,
+ "4505": 40813.0,
+ "4510": 40823.0,
+ "4515": 37930.0,
+ "4520": 39283.0,
+ "4525": 40728.0,
+ "4530": 39960.0,
+ "4535": 39368.0,
+ "4540": 39416.0,
+ "4545": 38732.0,
+ "4550": 40035.0,
+ "4555": 40802.0,
+ "4560": 39350.0,
+ "4565": 40133.0,
+ "4570": 41659.0,
+ "4575": 40032.0,
+ "4580": 37817.0,
+ "4585": 39478.0,
+ "4590": 37315.0,
+ "4595": 39337.0,
+ "4600": 40052.0,
+ "4605": 40738.0,
+ "4610": 40822.0,
+ "4615": 40733.0,
+ "4620": 40742.0,
+ "4625": 41499.0,
+ "4630": 40051.0,
+ "4635": 40725.0,
+ "4640": 41510.0,
+ "4645": 41496.0,
+ "4650": 40815.0,
+ "4655": 39970.0,
+ "4660": 40889.0,
+ "4665": 41486.0,
+ "4670": 39271.0,
+ "4675": 40041.0,
+ "4680": 41579.0,
+ "4685": 38055.0,
+ "4690": 40740.0,
+ "4695": 39273.0,
+ "4700": 39962.0,
+ "4705": 40814.0,
+ "4710": 42259.0,
+ "4715": 41578.0,
+ "4720": 40886.0,
+ "4725": 40804.0,
+ "4730": 38534.0,
+ "4735": 40045.0,
+ "4740": 37831.0,
+ "4745": 39437.0,
+ "4750": 40066.0,
+ "4755": 39199.0,
+ "4760": 40111.0,
+ "4765": 41505.0,
+ "4770": 40117.0,
+ "4775": 36639.0,
+ "4780": 39286.0,
+ "4785": 40051.0,
+ "4790": 42345.0,
+ "4795": 39278.0,
+ "4800": 42266.0,
+ "4805": 39435.0,
+ "4810": 40027.0,
+ "4815": 42266.0,
+ "4820": 39362.0,
+ "4825": 37853.0,
+ "4830": 40127.0,
+ "4835": 41508.0,
+ "4840": 40034.0,
+ "4845": 41581.0,
+ "4850": 38640.0,
+ "4855": 40216.0,
+ "4860": 38654.0,
+ "4865": 40051.0,
+ "4870": 41565.0,
+ "4875": 40742.0,
+ "4880": 41588.0,
+ "4885": 40735.0,
+ "4890": 40733.0,
+ "4895": 40803.0,
+ "4900": 40039.0,
+ "4905": 40808.0,
+ "4910": 40039.0,
+ "4915": 40828.0,
+ "4920": 40032.0,
+ "4925": 40045.0,
+ "4930": 39981.0,
+ "4935": 40802.0,
+ "4940": 39975.0,
+ "4945": 39422.0,
+ "4950": 40048.0,
+ "4955": 39983.0,
+ "4960": 38589.0,
+ "4965": 38677.0,
+ "4970": 39354.0,
+ "4975": 39962.0,
+ "4980": 37869.0,
+ "4985": 37768.0,
+ "4990": 39340.0,
+ "4995": 40057.0,
+ "5000": 38578.0,
+ "5005": 40029.0,
+ "5010": 38656.0,
+ "5015": 40732.0,
+ "5020": 39205.0,
+ "5025": 41494.0,
+ "5030": 36520.0,
+ "5035": 37948.0,
+ "5040": 40735.0,
+ "5045": 39224.0,
+ "5050": 39275.0,
+ "5055": 39987.0,
+ "5060": 40800.0,
+ "5065": 38608.0,
+ "5070": 42256.0,
+ "5075": 40041.0,
+ "5080": 41496.0,
+ "5085": 39284.0,
+ "5090": 40797.0,
+ "5095": 40866.0,
+ "5100": 39354.0,
+ "5105": 40067.0,
+ "5110": 40830.0,
+ "5115": 38555.0,
+ "5120": 40128.0,
+ "5125": 41499.0,
+ "5130": 40131.0,
+ "5135": 41497.0,
+ "5140": 41559.0,
+ "5145": 38083.0,
+ "5150": 39350.0,
+ "5155": 40034.0,
+ "5160": 41585.0,
+ "5165": 39974.0,
+ "5170": 40734.0,
+ "5175": 38784.0,
+ "5180": 41493.0,
+ "5185": 40815.0,
+ "5190": 38647.0,
+ "5195": 38585.0,
+ "5200": 40114.0,
+ "5205": 42263.0,
+ "5210": 39274.0,
+ "5215": 37186.0,
+ "5220": 40041.0,
+ "5225": 41582.0,
+ "5230": 40186.0,
+ "5235": 40032.0,
+ "5240": 38653.0,
+ "5245": 40822.0,
+ "5250": 38501.0,
+ "5255": 40029.0,
+ "5260": 40803.0,
+ "5265": 39273.0,
+ "5270": 41499.0,
+ "5275": 41503.0,
+ "5280": 39277.0,
+ "5285": 40047.0,
+ "5290": 38735.0,
+ "5295": 39354.0,
+ "5300": 39451.0,
+ "5305": 39286.0,
+ "5310": 41572.0,
+ "5315": 39223.0,
+ "5320": 40892.0,
+ "5325": 40805.0,
+ "5330": 39335.0,
+ "5335": 40109.0,
+ "5340": 42269.0,
+ "5345": 39968.0,
+ "5350": 40810.0,
+ "5355": 40138.0,
+ "5360": 37294.0,
+ "5365": 37928.0,
+ "5370": 40798.0,
+ "5375": 40046.0,
+ "5380": 39347.0,
+ "5385": 39349.0,
+ "5390": 38084.0,
+ "5395": 39413.0,
+ "5400": 38757.0,
+ "5405": 39354.0,
+ "5410": 40037.0,
+ "5415": 40743.0,
+ "5420": 39974.0,
+ "5425": 40108.0,
+ "5430": 40799.0,
+ "5435": 42269.0,
+ "5440": 39300.0,
+ "5445": 38576.0,
+ "5450": 40051.0,
+ "5455": 40133.0,
+ "5460": 41492.0,
+ "5465": 40880.0,
+ "5470": 39431.0,
+ "5475": 39430.0,
+ "5480": 40054.0,
+ "5485": 41503.0,
+ "5490": 39433.0,
+ "5495": 40803.0,
+ "5500": 40881.0,
+ "5505": 37766.0,
+ "5510": 42261.0,
+ "5515": 39297.0,
+ "5520": 40725.0,
+ "5525": 40809.0,
+ "5530": 39296.0,
+ "5535": 40747.0,
+ "5540": 40132.0,
+ "5545": 41580.0,
+ "5550": 38719.0,
+ "5555": 39981.0,
+ "5560": 40193.0,
+ "5565": 42367.0,
+ "5570": 41487.0,
+ "5575": 39403.0,
+ "5580": 39344.0,
+ "5585": 40738.0,
+ "5590": 38574.0,
+ "5595": 40043.0,
+ "5600": 39352.0,
+ "5605": 41568.0,
+ "5610": 40811.0,
+ "5615": 40804.0,
+ "5620": 39416.0,
+ "5625": 39418.0,
+ "5630": 40139.0,
+ "5635": 39330.0,
+ "5640": 40796.0,
+ "5645": 40729.0,
+ "5650": 38590.0,
+ "5655": 39272.0,
+ "5660": 38714.0,
+ "5665": 40824.0,
+ "5670": 41572.0,
+ "5675": 40834.0,
+ "5680": 39344.0,
+ "5685": 40737.0,
+ "5690": 39974.0,
+ "5695": 40815.0,
+ "5700": 39360.0,
+ "5705": 40039.0,
+ "5710": 39337.0,
+ "5715": 40811.0,
+ "5720": 40870.0,
+ "5725": 39963.0,
+ "5730": 40004.0,
+ "5735": 39973.0,
+ "5740": 40036.0,
+ "5745": 40813.0,
+ "5750": 37197.0,
+ "5755": 41496.0,
+ "5760": 40803.0,
+ "5765": 37120.0,
+ "5770": 40722.0,
+ "5775": 40054.0,
+ "5780": 38641.0,
+ "5785": 42338.0,
+ "5790": 38014.0,
+ "5795": 40879.0,
+ "5800": 40790.0,
+ "5805": 39970.0,
+ "5810": 43035.0,
+ "5815": 39280.0,
+ "5820": 40726.0,
+ "5825": 35575.0,
+ "5830": 39431.0,
+ "5835": 40036.0,
+ "5840": 40034.0,
+ "5845": 40791.0,
+ "5850": 41508.0,
+ "5855": 42263.0,
+ "5860": 38532.0,
+ "5865": 39322.0,
+ "5870": 39428.0,
+ "5875": 40055.0,
+ "5880": 39343.0,
+ "5885": 40805.0,
+ "5890": 37907.0,
+ "5895": 40727.0,
+ "5900": 40049.0,
+ "5905": 38677.0,
+ "5910": 37956.0,
+ "5915": 37838.0,
+ "5920": 40821.0,
+ "5925": 37926.0,
+ "5930": 40891.0,
+ "5935": 39274.0,
+ "5940": 40102.0,
+ "5945": 39337.0,
+ "5950": 40122.0,
+ "5955": 39357.0,
+ "5960": 40045.0,
+ "5965": 38752.0,
+ "5970": 38435.0,
+ "5975": 36510.0,
+ "5980": 40737.0,
+ "5985": 40025.0,
+ "5990": 40809.0,
+ "5995": 38595.0,
+ "6000": 38754.0,
+ "6005": 40048.0,
+ "6010": 39967.0,
+ "6015": 40015.0,
+ "6020": 41554.0,
+ "6025": 40025.0,
+ "6030": 39367.0,
+ "6035": 41581.0,
+ "6040": 38587.0,
+ "6045": 39975.0,
+ "6050": 39980.0,
+ "6055": 41587.0,
+ "6060": 40121.0,
+ "6065": 41566.0,
+ "6070": 38741.0,
+ "6075": 41492.0,
+ "6080": 37997.0,
+ "6085": 35964.0,
+ "6090": 40738.0,
+ "6095": 41509.0,
+ "6100": 40750.0,
+ "6105": 35934.0,
+ "6110": 40119.0,
+ "6115": 41498.0,
+ "6120": 40120.0,
+ "6125": 37197.0,
+ "6130": 39262.0,
+ "6135": 39285.0,
+ "6140": 41566.0,
+ "6145": 40051.0,
+ "6150": 39288.0,
+ "6155": 41490.0,
+ "6160": 39959.0,
+ "6165": 40739.0,
+ "6170": 37313.0,
+ "6175": 40790.0,
+ "6180": 37826.0,
+ "6185": 40822.0,
+ "6190": 41504.0,
+ "6195": 38588.0,
+ "6200": 41504.0,
+ "6205": 41566.0,
+ "6210": 38747.0,
+ "6215": 41505.0,
+ "6220": 39977.0,
+ "6225": 40746.0,
+ "6230": 37896.0,
+ "6235": 40826.0,
+ "6240": 40737.0,
+ "6245": 40185.0,
+ "6250": 41489.0,
+ "6255": 40731.0,
+ "6260": 41587.0,
+ "6265": 39218.0,
+ "6270": 40803.0,
+ "6275": 40882.0,
+ "6280": 43037.0,
+ "6285": 40813.0,
+ "6290": 41586.0,
+ "6295": 40804.0,
+ "6300": 39974.0,
+ "6305": 38652.0,
+ "6310": 38664.0,
+ "6315": 41503.0,
+ "6320": 40883.0,
+ "6325": 37702.0,
+ "6330": 39282.0,
+ "6335": 41577.0,
+ "6340": 40743.0,
+ "6345": 41507.0,
+ "6350": 41598.0,
+ "6355": 40104.0,
+ "6360": 38733.0,
+ "6365": 40812.0,
+ "6370": 38736.0,
+ "6375": 40062.0,
+ "6380": 37122.0,
+ "6385": 40192.0,
+ "6390": 39512.0,
+ "6395": 41493.0,
+ "6400": 36594.0,
+ "6405": 40046.0,
+ "6410": 40103.0,
+ "6415": 41573.0,
+ "6420": 40794.0,
+ "6425": 37290.0,
+ "6430": 39964.0,
+ "6435": 40119.0,
+ "6440": 41495.0,
+ "6445": 40730.0,
+ "6450": 38495.0,
+ "6455": 38565.0,
+ "6460": 39219.0,
+ "6465": 42271.0,
+ "6470": 41503.0,
+ "6475": 40808.0,
+ "6480": 40123.0,
+ "6485": 39266.0,
+ "6490": 35805.0,
+ "6495": 39273.0,
+ "6500": 41498.0,
+ "6505": 40737.0,
+ "6510": 40050.0,
+ "6515": 40129.0,
+ "6520": 40794.0,
+ "6525": 42268.0,
+ "6530": 41498.0,
+ "6535": 39298.0,
+ "6540": 40849.0,
+ "6545": 37978.0,
+ "6550": 39352.0,
+ "6555": 40798.0,
+ "6560": 35747.0,
+ "6565": 39203.0,
+ "6570": 40137.0,
+ "6575": 41503.0,
+ "6580": 41580.0,
+ "6585": 40054.0,
+ "6590": 35479.0,
+ "6595": 40806.0,
+ "6600": 39293.0,
+ "6605": 39340.0,
+ "6610": 41505.0,
+ "6615": 40802.0,
+ "6620": 41570.0,
+ "6625": 40720.0,
+ "6630": 37819.0,
+ "6635": 38691.0,
+ "6640": 40815.0,
+ "6645": 38022.0,
+ "6650": 40022.0,
+ "6655": 40038.0,
+ "6660": 41501.0,
+ "6665": 41506.0,
+ "6670": 39285.0,
+ "6675": 39350.0,
+ "6680": 42265.0,
+ "6685": 40738.0,
+ "6690": 40037.0,
+ "6695": 40730.0,
+ "6700": 40790.0,
+ "6705": 41557.0,
+ "6710": 39208.0,
+ "6715": 38515.0,
+ "6720": 39974.0,
+ "6725": 41571.0,
+ "6730": 41500.0,
+ "6735": 38669.0,
+ "6740": 37264.0,
+ "6745": 41495.0,
+ "6750": 40733.0,
+ "6755": 38588.0,
+ "6760": 39356.0,
+ "6765": 39336.0,
+ "6770": 39971.0,
+ "6775": 38658.0,
+ "6780": 38605.0,
+ "6785": 40065.0,
+ "6790": 40734.0,
+ "6795": 40741.0,
+ "6800": 40725.0,
+ "6805": 40809.0,
+ "6810": 40198.0,
+ "6815": 40788.0,
+ "6820": 40122.0,
+ "6825": 40814.0,
+ "6830": 41577.0,
+ "6835": 38600.0,
+ "6840": 40803.0,
+ "6845": 42335.0,
+ "6850": 40120.0,
+ "6855": 39329.0,
+ "6860": 41494.0,
+ "6865": 40737.0,
+ "6870": 40056.0,
+ "6875": 40798.0,
+ "6880": 40803.0,
+ "6885": 39354.0,
+ "6890": 40034.0,
+ "6895": 40145.0,
+ "6900": 37765.0,
+ "6905": 40881.0,
+ "6910": 41582.0,
+ "6915": 41578.0,
+ "6920": 40062.0,
+ "6925": 40130.0,
+ "6930": 42269.0,
+ "6935": 38511.0,
+ "6940": 39396.0,
+ "6945": 40798.0,
+ "6950": 39977.0,
+ "6955": 39425.0,
+ "6960": 40735.0,
+ "6965": 40796.0,
+ "6970": 40806.0,
+ "6975": 41499.0,
+ "6980": 39504.0,
+ "6985": 40838.0,
+ "6990": 40112.0,
+ "6995": 39966.0,
+ "7000": 39432.0,
+ "7005": 39421.0,
+ "7010": 39962.0,
+ "7015": 38640.0,
+ "7020": 40802.0,
+ "7025": 39265.0,
+ "7030": 38673.0,
+ "7035": 39348.0,
+ "7040": 42355.0,
+ "7045": 36583.0,
+ "7050": 38577.0,
+ "7055": 40887.0,
+ "7060": 40737.0,
+ "7065": 37945.0,
+ "7070": 40046.0,
+ "7075": 39287.0,
+ "7080": 40733.0,
+ "7085": 40729.0,
+ "7090": 36618.0,
+ "7095": 40729.0,
+ "7100": 40804.0,
+ "7105": 40723.0,
+ "7110": 42346.0,
+ "7115": 39279.0,
+ "7120": 40798.0,
+ "7125": 38496.0,
+ "7130": 40888.0,
+ "7135": 39279.0,
+ "7140": 40093.0,
+ "7145": 40740.0,
+ "7150": 40802.0,
+ "7155": 42341.0,
+ "7160": 40034.0,
+ "7165": 40047.0,
+ "7170": 41496.0,
+ "7175": 40820.0,
+ "7180": 40803.0,
+ "7185": 39286.0,
+ "7190": 40820.0,
+ "7195": 39971.0,
+ "7200": 40128.0,
+ "7205": 40805.0,
+ "7210": 41569.0,
+ "7215": 41566.0,
+ "7220": 37345.0,
+ "7225": 40812.0,
+ "7230": 40095.0,
+ "7235": 41503.0,
+ "7240": 38585.0,
+ "7245": 39463.0,
+ "7250": 40823.0,
+ "7255": 39967.0,
+ "7260": 38537.0,
+ "7265": 39341.0,
+ "7270": 40743.0,
+ "7275": 38597.0,
+ "7280": 39423.0,
+ "7285": 40104.0,
+ "7290": 41575.0,
+ "7295": 40737.0,
+ "7300": 40045.0,
+ "7305": 39533.0,
+ "7310": 40144.0,
+ "7315": 40873.0,
+ "7320": 40108.0,
+ "7325": 40817.0,
+ "7330": 41492.0,
+ "7335": 39349.0,
+ "7340": 40809.0,
+ "7345": 40115.0,
+ "7350": 39349.0,
+ "7355": 40733.0,
+ "7360": 40183.0,
+ "7365": 41486.0,
+ "7370": 38559.0,
+ "7375": 38443.0,
+ "7380": 40751.0,
+ "7385": 40136.0,
+ "7390": 38651.0,
+ "7395": 38454.0,
+ "7400": 41490.0,
+ "7405": 41496.0,
+ "7410": 38493.0,
+ "7415": 38534.0,
+ "7420": 39336.0,
+ "7425": 41504.0,
+ "7430": 39353.0,
+ "7435": 39969.0,
+ "7440": 40129.0,
+ "7445": 40734.0,
+ "7450": 40812.0,
+ "7455": 42273.0,
+ "7460": 38828.0,
+ "7465": 35633.0,
+ "7470": 40814.0,
+ "7475": 37814.0,
+ "7480": 40883.0,
+ "7485": 37985.0,
+ "7490": 40810.0,
+ "7495": 40099.0,
+ "7500": 41506.0,
+ "7505": 40808.0,
+ "7510": 39205.0,
+ "7515": 40050.0,
+ "7520": 40752.0,
+ "7525": 40114.0,
+ "7530": 40818.0,
+ "7535": 39293.0,
+ "7540": 40032.0,
+ "7545": 38449.0,
+ "7550": 40739.0,
+ "7555": 40099.0,
+ "7560": 41499.0,
+ "7565": 40028.0,
+ "7570": 40732.0,
+ "7575": 40736.0,
+ "7580": 39961.0,
+ "7585": 40726.0,
+ "7590": 40808.0,
+ "7595": 40821.0,
+ "7600": 35678.0,
+ "7605": 40734.0,
+ "7610": 41500.0,
+ "7615": 38451.0,
+ "7620": 41576.0,
+ "7625": 40721.0,
+ "7630": 41562.0,
+ "7635": 41509.0,
+ "7640": 40114.0,
+ "7645": 40104.0,
+ "7650": 38659.0,
+ "7655": 41564.0,
+ "7660": 39967.0,
+ "7665": 41561.0,
+ "7670": 39987.0,
+ "7675": 39441.0,
+ "7680": 40746.0,
+ "7685": 39361.0,
+ "7690": 40144.0,
+ "7695": 40072.0,
+ "7700": 41497.0,
+ "7705": 40201.0,
+ "7710": 40792.0,
+ "7715": 40037.0,
+ "7720": 35804.0,
+ "7725": 39348.0,
+ "7730": 39363.0,
+ "7735": 37967.0,
+ "7740": 40886.0,
+ "7745": 40735.0,
+ "7750": 41495.0,
+ "7755": 39413.0,
+ "7760": 40123.0,
+ "7765": 39347.0,
+ "7770": 39355.0,
+ "7775": 39368.0,
+ "7780": 38584.0,
+ "7785": 38746.0,
+ "7790": 39342.0,
+ "7795": 40046.0,
+ "7800": 40119.0,
+ "7805": 38028.0,
+ "7810": 42348.0,
+ "7815": 38597.0,
+ "7820": 40816.0,
+ "7825": 37892.0,
+ "7830": 40876.0,
+ "7835": 39351.0,
+ "7840": 40822.0,
+ "7845": 41501.0,
+ "7850": 39354.0,
+ "7855": 38681.0,
+ "7860": 39972.0,
+ "7865": 38767.0,
+ "7870": 40804.0,
+ "7875": 40115.0,
+ "7880": 40726.0,
+ "7885": 40737.0,
+ "7890": 40042.0,
+ "7895": 42262.0,
+ "7900": 40115.0,
+ "7905": 41500.0,
+ "7910": 37907.0,
+ "7915": 39291.0,
+ "7920": 39363.0,
+ "7925": 42261.0,
+ "7930": 39286.0,
+ "7935": 41575.0,
+ "7940": 39337.0,
+ "7945": 41579.0,
+ "7950": 36604.0,
+ "7955": 37918.0,
+ "7960": 40051.0,
+ "7965": 40811.0,
+ "7970": 40870.0,
+ "7975": 37747.0,
+ "7980": 41496.0,
+ "7985": 39973.0,
+ "7990": 39416.0,
+ "7995": 39343.0,
+ "8000": 41521.0,
+ "8005": 40810.0,
+ "8010": 40725.0,
+ "8015": 40884.0,
+ "8020": 38653.0,
+ "8025": 40891.0,
+ "8030": 37840.0,
+ "8035": 40059.0,
+ "8040": 39228.0,
+ "8045": 41590.0,
+ "8050": 38601.0,
+ "8055": 39278.0,
+ "8060": 39271.0,
+ "8065": 40802.0,
+ "8070": 40818.0,
+ "8075": 41561.0,
+ "8080": 41572.0,
+ "8085": 39301.0,
+ "8090": 40815.0,
+ "8095": 39338.0,
+ "8100": 40750.0,
+ "8105": 40908.0,
+ "8110": 38677.0,
+ "8115": 40800.0,
+ "8120": 39344.0,
+ "8125": 39965.0,
+ "8130": 39294.0,
+ "8135": 41495.0,
+ "8140": 39972.0,
+ "8145": 40814.0,
+ "8150": 41569.0,
+ "8155": 41516.0,
+ "8160": 39476.0,
+ "8165": 40023.0,
+ "8170": 38714.0,
+ "8175": 40028.0,
+ "8180": 37402.0,
+ "8185": 40723.0,
+ "8190": 39358.0,
+ "8195": 40034.0,
+ "8200": 40100.0,
+ "8205": 40040.0,
+ "8210": 39270.0,
+ "8215": 40795.0,
+ "8220": 38597.0,
+ "8225": 39272.0,
+ "8230": 40824.0,
+ "8235": 40047.0,
+ "8240": 39334.0,
+ "8245": 38651.0,
+ "8250": 41566.0,
+ "8255": 39428.0,
+ "8260": 40827.0,
+ "8265": 41515.0,
+ "8270": 42283.0,
+ "8275": 40748.0,
+ "8280": 40046.0,
+ "8285": 40879.0,
+ "8290": 40101.0,
+ "8295": 39261.0,
+ "8300": 40799.0,
+ "8305": 40112.0,
+ "8310": 39284.0,
+ "8315": 39206.0,
+ "8320": 41508.0,
+ "8325": 39274.0,
+ "8330": 40736.0,
+ "8335": 40049.0,
+ "8340": 40054.0,
+ "8345": 39433.0,
+ "8350": 40801.0,
+ "8355": 40069.0,
+ "8360": 40809.0,
+ "8365": 39966.0,
+ "8370": 39444.0,
+ "8375": 38886.0,
+ "8380": 40735.0,
+ "8385": 40067.0,
+ "8390": 40808.0,
+ "8395": 39979.0,
+ "8400": 40124.0,
+ "8405": 39215.0,
+ "8410": 41495.0,
+ "8415": 38597.0,
+ "8420": 40105.0,
+ "8425": 39968.0,
+ "8430": 37823.0,
+ "8435": 39426.0,
+ "8440": 37103.0,
+ "8445": 38572.0,
+ "8450": 39432.0,
+ "8455": 39980.0,
+ "8460": 38019.0,
+ "8465": 37897.0,
+ "8470": 38587.0,
+ "8475": 40813.0,
+ "8480": 40179.0,
+ "8485": 37672.0,
+ "8490": 41501.0,
+ "8495": 40028.0,
+ "8500": 39976.0,
+ "8505": 38419.0,
+ "8510": 41506.0,
+ "8515": 39429.0,
+ "8520": 39439.0,
+ "8525": 38534.0,
+ "8530": 40811.0,
+ "8535": 40752.0,
+ "8540": 40740.0,
+ "8545": 39352.0,
+ "8550": 40828.0,
+ "8555": 40052.0,
+ "8560": 40730.0,
+ "8565": 39969.0,
+ "8570": 39288.0,
+ "8575": 40043.0,
+ "8580": 40100.0,
+ "8585": 40045.0,
+ "8590": 40844.0,
+ "8595": 40019.0,
+ "8600": 40208.0,
+ "8605": 39299.0,
+ "8610": 40885.0,
+ "8615": 38597.0,
+ "8620": 40048.0,
+ "8625": 42264.0,
+ "8630": 40742.0,
+ "8635": 39280.0,
+ "8640": 40729.0,
+ "8645": 40047.0,
+ "8650": 37204.0,
+ "8655": 40128.0,
+ "8660": 38666.0,
+ "8665": 40733.0,
+ "8670": 40803.0,
+ "8675": 40048.0,
+ "8680": 38511.0,
+ "8685": 42269.0,
+ "8690": 37899.0,
+ "8695": 39281.0,
+ "8700": 38655.0,
+ "8705": 40806.0,
+ "8710": 40038.0,
+ "8715": 40039.0,
+ "8720": 41505.0,
+ "8725": 39340.0,
+ "8730": 40802.0,
+ "8735": 38677.0,
+ "8740": 39299.0,
+ "8745": 38657.0,
+ "8750": 38565.0,
+ "8755": 40893.0,
+ "8760": 40739.0,
+ "8765": 39199.0,
+ "8770": 41571.0,
+ "8775": 40811.0,
+ "8780": 40793.0,
+ "8785": 39276.0,
+ "8790": 41493.0,
+ "8795": 40801.0,
+ "8800": 41581.0,
+ "8805": 41504.0,
+ "8810": 39284.0,
+ "8815": 40718.0,
+ "8820": 39272.0,
+ "8825": 41504.0,
+ "8830": 42268.0,
+ "8835": 40810.0,
+ "8840": 39342.0,
+ "8845": 42262.0,
+ "8850": 41497.0,
+ "8855": 42271.0,
+ "8860": 39335.0,
+ "8865": 39297.0,
+ "8870": 41584.0,
+ "8875": 38718.0,
+ "8880": 38638.0,
+ "8885": 38597.0,
+ "8890": 39219.0,
+ "8895": 40047.0,
+ "8900": 40047.0,
+ "8905": 38513.0,
+ "8910": 40742.0,
+ "8915": 37897.0,
+ "8920": 39288.0,
+ "8925": 39298.0,
+ "8930": 42263.0,
+ "8935": 37320.0,
+ "8940": 41573.0,
+ "8945": 41504.0,
+ "8950": 40032.0,
+ "8955": 41507.0,
+ "8960": 41505.0,
+ "8965": 42254.0,
+ "8970": 37917.0,
+ "8975": 36496.0,
+ "8980": 40793.0,
+ "8985": 41506.0,
+ "8990": 40745.0,
+ "8995": 39341.0,
+ "9000": 40743.0,
+ "9005": 39288.0,
+ "9010": 36497.0,
+ "9015": 39339.0,
+ "9020": 40053.0,
+ "9025": 39967.0,
+ "9030": 40071.0,
+ "9035": 40127.0,
+ "9040": 40810.0,
+ "9045": 39973.0,
+ "9050": 40099.0,
+ "9055": 40814.0,
+ "9060": 38519.0,
+ "9065": 38571.0,
+ "9070": 38599.0,
+ "9075": 40114.0,
+ "9080": 40124.0,
+ "9085": 40735.0,
+ "9090": 40045.0,
+ "9095": 39960.0,
+ "9100": 38509.0,
+ "9105": 38671.0,
+ "9110": 40108.0,
+ "9115": 39269.0,
+ "9120": 34496.0,
+ "9125": 39497.0,
+ "9130": 42336.0,
+ "9135": 39251.0,
+ "9140": 39202.0,
+ "9145": 40220.0,
+ "9150": 39277.0,
+ "9155": 42265.0,
+ "9160": 40790.0,
+ "9165": 40041.0,
+ "9170": 40031.0,
+ "9175": 38590.0,
+ "9180": 39970.0,
+ "9185": 40118.0,
+ "9190": 40040.0,
+ "9195": 40067.0,
+ "9200": 38015.0,
+ "9205": 40744.0,
+ "9210": 39285.0,
+ "9215": 40742.0,
+ "9220": 40802.0,
+ "9225": 40814.0,
+ "9230": 40801.0,
+ "9235": 38814.0,
+ "9240": 41512.0,
+ "9245": 41587.0,
+ "9250": 40729.0,
+ "9255": 40127.0,
+ "9260": 40210.0,
+ "9265": 41498.0,
+ "9270": 41499.0,
+ "9275": 41590.0,
+ "9280": 37268.0,
+ "9285": 40950.0,
+ "9290": 35887.0,
+ "9295": 40816.0,
+ "9300": 40816.0,
+ "9305": 39416.0,
+ "9310": 40731.0,
+ "9315": 40809.0,
+ "9320": 38577.0,
+ "9325": 40148.0,
+ "9330": 40809.0,
+ "9335": 38501.0,
+ "9340": 41492.0,
+ "9345": 40816.0,
+ "9350": 40803.0,
+ "9355": 38459.0,
+ "9360": 42259.0,
+ "9365": 38664.0,
+ "9370": 41494.0,
+ "9375": 40807.0,
+ "9380": 40805.0,
+ "9385": 40741.0,
+ "9390": 41569.0,
+ "9395": 40875.0,
+ "9400": 40792.0,
+ "9405": 40044.0,
+ "9410": 40812.0,
+ "9415": 40046.0,
+ "9420": 40747.0,
+ "9425": 41516.0,
+ "9430": 40735.0,
+ "9435": 39411.0,
+ "9440": 41514.0,
+ "9445": 40812.0,
+ "9450": 39359.0,
+ "9455": 42334.0,
+ "9460": 40806.0,
+ "9465": 42261.0,
+ "9470": 35927.0,
+ "9475": 40813.0,
+ "9480": 41581.0,
+ "9485": 39426.0,
+ "9490": 39299.0,
+ "9495": 38606.0,
+ "9500": 41506.0,
+ "9505": 40032.0,
+ "9510": 41591.0,
+ "9515": 40131.0,
+ "9520": 39271.0,
+ "9525": 39489.0,
+ "9530": 41564.0,
+ "9535": 38672.0,
+ "9540": 39279.0,
+ "9545": 39280.0,
+ "9550": 41583.0,
+ "9555": 40738.0,
+ "9560": 39357.0,
+ "9565": 40051.0,
+ "9570": 39442.0,
+ "9575": 42270.0,
+ "9580": 40042.0,
+ "9585": 40050.0,
+ "9590": 38500.0,
+ "9595": 38676.0,
+ "9600": 37224.0,
+ "9605": 40818.0,
+ "9610": 41582.0,
+ "9615": 39994.0,
+ "9620": 39974.0,
+ "9625": 40803.0,
+ "9630": 40792.0,
+ "9635": 38442.0,
+ "9640": 40798.0,
+ "9645": 39961.0,
+ "9650": 41497.0,
+ "9655": 42268.0,
+ "9660": 40808.0,
+ "9665": 40046.0,
+ "9670": 40061.0,
+ "9675": 38736.0,
+ "9680": 40740.0,
+ "9685": 39973.0,
+ "9690": 39389.0,
+ "9695": 39267.0,
+ "9700": 39206.0,
+ "9705": 40874.0,
+ "9710": 39270.0,
+ "9715": 39375.0,
+ "9720": 40178.0,
+ "9725": 38746.0,
+ "9730": 40802.0,
+ "9735": 41588.0,
+ "9740": 41569.0,
+ "9745": 39352.0,
+ "9750": 40816.0,
+ "9755": 40902.0,
+ "9760": 39304.0,
+ "9765": 43038.0,
+ "9770": 39415.0,
+ "9775": 39965.0,
+ "9780": 40157.0,
+ "9785": 40886.0,
+ "9790": 39218.0,
+ "9795": 42274.0,
+ "9800": 37236.0,
+ "9805": 39992.0,
+ "9810": 40815.0,
+ "9815": 40802.0,
+ "9820": 39289.0,
+ "9825": 40811.0,
+ "9830": 39520.0,
+ "9835": 39286.0,
+ "9840": 37891.0,
+ "9845": 39967.0,
+ "9850": 39291.0,
+ "9855": 38518.0,
+ "9860": 37946.0,
+ "9865": 41497.0,
+ "9870": 39345.0,
+ "9875": 37147.0,
+ "9880": 40054.0,
+ "9885": 39210.0,
+ "9890": 40035.0,
+ "9895": 41582.0,
+ "9900": 39972.0,
+ "9905": 41565.0,
+ "9910": 41586.0,
+ "9915": 41495.0,
+ "9920": 39983.0,
+ "9925": 38666.0,
+ "9930": 40046.0,
+ "9935": 37912.0,
+ "9940": 41640.0,
+ "9945": 41500.0,
+ "9950": 40798.0,
+ "9955": 41500.0,
+ "9960": 40813.0,
+ "9965": 40810.0,
+ "9970": 39354.0,
+ "9975": 40765.0,
+ "9980": 41500.0,
+ "9985": 42262.0,
+ "9990": 38524.0,
+ "9995": 42267.0,
+ "10000": 40052.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 10000,
+ "step_interval": 5,
+ "values": {
+ "1": 1083622400.0,
+ "5": 1083622400.0,
+ "10": 1083622400.0,
+ "15": 1083622400.0,
+ "20": 1083622400.0,
+ "25": 1083622400.0,
+ "30": 1083622400.0,
+ "35": 1083622400.0,
+ "40": 1083622400.0,
+ "45": 1083622400.0,
+ "50": 1083622400.0,
+ "55": 1083622400.0,
+ "60": 1083622400.0,
+ "65": 1083622400.0,
+ "70": 1083622400.0,
+ "75": 1083622400.0,
+ "80": 1083622400.0,
+ "85": 1083622400.0,
+ "90": 1083622400.0,
+ "95": 1083622400.0,
+ "100": 1083622400.0,
+ "105": 1083622400.0,
+ "110": 1083622400.0,
+ "115": 1083622400.0,
+ "120": 1083622400.0,
+ "125": 1083622400.0,
+ "130": 1083622400.0,
+ "135": 1083622400.0,
+ "140": 1083622400.0,
+ "145": 1083622400.0,
+ "150": 1083622400.0,
+ "155": 1083622400.0,
+ "160": 1083622400.0,
+ "165": 1083622400.0,
+ "170": 1083622400.0,
+ "175": 1083622400.0,
+ "180": 1083622400.0,
+ "185": 1083622400.0,
+ "190": 1083622400.0,
+ "195": 1083622400.0,
+ "200": 1083622400.0,
+ "205": 1083622400.0,
+ "210": 1083622400.0,
+ "215": 1083622400.0,
+ "220": 1083622400.0,
+ "225": 1083622400.0,
+ "230": 1083622400.0,
+ "235": 1083622400.0,
+ "240": 1083622400.0,
+ "245": 1083622400.0,
+ "250": 1083622400.0,
+ "255": 1083622400.0,
+ "260": 1083622400.0,
+ "265": 1083622400.0,
+ "270": 1083622400.0,
+ "275": 1083622400.0,
+ "280": 1083622400.0,
+ "285": 1083622400.0,
+ "290": 1083622400.0,
+ "295": 1083622400.0,
+ "300": 1083622400.0,
+ "305": 1083622400.0,
+ "310": 1083622400.0,
+ "315": 1083622400.0,
+ "320": 1083622400.0,
+ "325": 1083622400.0,
+ "330": 1083622400.0,
+ "335": 1083622400.0,
+ "340": 1083622400.0,
+ "345": 1083622400.0,
+ "350": 1083622400.0,
+ "355": 1083622400.0,
+ "360": 1083622400.0,
+ "365": 1083622400.0,
+ "370": 1083622400.0,
+ "375": 1083622400.0,
+ "380": 1083622400.0,
+ "385": 1083622400.0,
+ "390": 1083622400.0,
+ "395": 1083622400.0,
+ "400": 1083622400.0,
+ "405": 1083622400.0,
+ "410": 1083622400.0,
+ "415": 1083622400.0,
+ "420": 1083622400.0,
+ "425": 1083622400.0,
+ "430": 1083622400.0,
+ "435": 1083622400.0,
+ "440": 1083622400.0,
+ "445": 1083622400.0,
+ "450": 1083622400.0,
+ "455": 1083622400.0,
+ "460": 1083622400.0,
+ "465": 1083622400.0,
+ "470": 1083622400.0,
+ "475": 1083622400.0,
+ "480": 1083622400.0,
+ "485": 1083622400.0,
+ "490": 1083622400.0,
+ "495": 1083622400.0,
+ "500": 1083622400.0,
+ "505": 1083622400.0,
+ "510": 1083622400.0,
+ "515": 1083622400.0,
+ "520": 1083622400.0,
+ "525": 1083622400.0,
+ "530": 1083622400.0,
+ "535": 1083622400.0,
+ "540": 1083622400.0,
+ "545": 1083622400.0,
+ "550": 1083622400.0,
+ "555": 1083622400.0,
+ "560": 1083622400.0,
+ "565": 1083622400.0,
+ "570": 1083622400.0,
+ "575": 1083622400.0,
+ "580": 1083622400.0,
+ "585": 1083622400.0,
+ "590": 1083622400.0,
+ "595": 1083622400.0,
+ "600": 1083622400.0,
+ "605": 1083622400.0,
+ "610": 1083622400.0,
+ "615": 1083622400.0,
+ "620": 1083622400.0,
+ "625": 1083622400.0,
+ "630": 1083622400.0,
+ "635": 1083622400.0,
+ "640": 1083622400.0,
+ "645": 1083622400.0,
+ "650": 1083622400.0,
+ "655": 1083622400.0,
+ "660": 1083622400.0,
+ "665": 1083622400.0,
+ "670": 1083622400.0,
+ "675": 1083622400.0,
+ "680": 1083622400.0,
+ "685": 1083622400.0,
+ "690": 1083622400.0,
+ "695": 1083622400.0,
+ "700": 1083622400.0,
+ "705": 1083622400.0,
+ "710": 1083622400.0,
+ "715": 1083622400.0,
+ "720": 1083622400.0,
+ "725": 1083622400.0,
+ "730": 1083622400.0,
+ "735": 1083622400.0,
+ "740": 1083622400.0,
+ "745": 1083622400.0,
+ "750": 1083622400.0,
+ "755": 1083622400.0,
+ "760": 1083622400.0,
+ "765": 1083622400.0,
+ "770": 1083622400.0,
+ "775": 1083622400.0,
+ "780": 1083622400.0,
+ "785": 1083622400.0,
+ "790": 1083622400.0,
+ "795": 1083622400.0,
+ "800": 1083622400.0,
+ "805": 1083622400.0,
+ "810": 1083622400.0,
+ "815": 1083622400.0,
+ "820": 1083622400.0,
+ "825": 1083622400.0,
+ "830": 1083622400.0,
+ "835": 1083622400.0,
+ "840": 1083622400.0,
+ "845": 1083622400.0,
+ "850": 1083622400.0,
+ "855": 1083622400.0,
+ "860": 1083622400.0,
+ "865": 1083622400.0,
+ "870": 1083622400.0,
+ "875": 1083622400.0,
+ "880": 1083622400.0,
+ "885": 1083622400.0,
+ "890": 1083622400.0,
+ "895": 1083622400.0,
+ "900": 1083622400.0,
+ "905": 1083622400.0,
+ "910": 1083622400.0,
+ "915": 1083622400.0,
+ "920": 1083622400.0,
+ "925": 1083622400.0,
+ "930": 1083622400.0,
+ "935": 1083622400.0,
+ "940": 1083622400.0,
+ "945": 1083622400.0,
+ "950": 1083622400.0,
+ "955": 1083622400.0,
+ "960": 1083622400.0,
+ "965": 1083622400.0,
+ "970": 1083622400.0,
+ "975": 1083622400.0,
+ "980": 1083622400.0,
+ "985": 1083622400.0,
+ "990": 1083622400.0,
+ "995": 1083622400.0,
+ "1000": 1083622400.0,
+ "1005": 1083622400.0,
+ "1010": 1083622400.0,
+ "1015": 1083622400.0,
+ "1020": 1083622400.0,
+ "1025": 1083622400.0,
+ "1030": 1083622400.0,
+ "1035": 1083622400.0,
+ "1040": 1083622400.0,
+ "1045": 1083622400.0,
+ "1050": 1083622400.0,
+ "1055": 1083622400.0,
+ "1060": 1083622400.0,
+ "1065": 1083622400.0,
+ "1070": 1083622400.0,
+ "1075": 1083622400.0,
+ "1080": 1083622400.0,
+ "1085": 1083622400.0,
+ "1090": 1083622400.0,
+ "1095": 1083622400.0,
+ "1100": 1083622400.0,
+ "1105": 1083622400.0,
+ "1110": 1083622400.0,
+ "1115": 1083622400.0,
+ "1120": 1083622400.0,
+ "1125": 1083622400.0,
+ "1130": 1083622400.0,
+ "1135": 1083622400.0,
+ "1140": 1083622400.0,
+ "1145": 1083622400.0,
+ "1150": 1083622400.0,
+ "1155": 1083622400.0,
+ "1160": 1083622400.0,
+ "1165": 1083622400.0,
+ "1170": 1083622400.0,
+ "1175": 1083622400.0,
+ "1180": 1083622400.0,
+ "1185": 1083622400.0,
+ "1190": 1083622400.0,
+ "1195": 1083622400.0,
+ "1200": 1083622400.0,
+ "1205": 1083622400.0,
+ "1210": 1083622400.0,
+ "1215": 1083622400.0,
+ "1220": 1083622400.0,
+ "1225": 1083622400.0,
+ "1230": 1083622400.0,
+ "1235": 1083622400.0,
+ "1240": 1083622400.0,
+ "1245": 1083622400.0,
+ "1250": 1083622400.0,
+ "1255": 1083622400.0,
+ "1260": 1083622400.0,
+ "1265": 1083622400.0,
+ "1270": 1083622400.0,
+ "1275": 1083622400.0,
+ "1280": 1083622400.0,
+ "1285": 1083622400.0,
+ "1290": 1083622400.0,
+ "1295": 1083622400.0,
+ "1300": 1083622400.0,
+ "1305": 1083622400.0,
+ "1310": 1083622400.0,
+ "1315": 1083622400.0,
+ "1320": 1083622400.0,
+ "1325": 1083622400.0,
+ "1330": 1083622400.0,
+ "1335": 1083622400.0,
+ "1340": 1083622400.0,
+ "1345": 1083622400.0,
+ "1350": 1083622400.0,
+ "1355": 1083622400.0,
+ "1360": 1083622400.0,
+ "1365": 1083622400.0,
+ "1370": 1083622400.0,
+ "1375": 1083622400.0,
+ "1380": 1083622400.0,
+ "1385": 1083622400.0,
+ "1390": 1083622400.0,
+ "1395": 1083622400.0,
+ "1400": 1083622400.0,
+ "1405": 1083622400.0,
+ "1410": 1083622400.0,
+ "1415": 1083622400.0,
+ "1420": 1083622400.0,
+ "1425": 1083622400.0,
+ "1430": 1083622400.0,
+ "1435": 1083622400.0,
+ "1440": 1083622400.0,
+ "1445": 1083622400.0,
+ "1450": 1083622400.0,
+ "1455": 1083622400.0,
+ "1460": 1083622400.0,
+ "1465": 1083622400.0,
+ "1470": 1083622400.0,
+ "1475": 1083622400.0,
+ "1480": 1083622400.0,
+ "1485": 1083622400.0,
+ "1490": 1083622400.0,
+ "1495": 1083622400.0,
+ "1500": 1083622400.0,
+ "1505": 1083622400.0,
+ "1510": 1083622400.0,
+ "1515": 1083622400.0,
+ "1520": 1083622400.0,
+ "1525": 1083622400.0,
+ "1530": 1083622400.0,
+ "1535": 1083622400.0,
+ "1540": 1083622400.0,
+ "1545": 1083622400.0,
+ "1550": 1083622400.0,
+ "1555": 1083622400.0,
+ "1560": 1083622400.0,
+ "1565": 1083622400.0,
+ "1570": 1083622400.0,
+ "1575": 1083622400.0,
+ "1580": 1083622400.0,
+ "1585": 1083622400.0,
+ "1590": 1083622400.0,
+ "1595": 1083622400.0,
+ "1600": 1083622400.0,
+ "1605": 1083622400.0,
+ "1610": 1083622400.0,
+ "1615": 1083622400.0,
+ "1620": 1083622400.0,
+ "1625": 1083622400.0,
+ "1630": 1083622400.0,
+ "1635": 1083622400.0,
+ "1640": 1083622400.0,
+ "1645": 1083622400.0,
+ "1650": 1083622400.0,
+ "1655": 1083622400.0,
+ "1660": 1083622400.0,
+ "1665": 1083622400.0,
+ "1670": 1083622400.0,
+ "1675": 1083622400.0,
+ "1680": 1083622400.0,
+ "1685": 1083622400.0,
+ "1690": 1083622400.0,
+ "1695": 1083622400.0,
+ "1700": 1083622400.0,
+ "1705": 1083622400.0,
+ "1710": 1083622400.0,
+ "1715": 1083622400.0,
+ "1720": 1083622400.0,
+ "1725": 1083622400.0,
+ "1730": 1083622400.0,
+ "1735": 1083622400.0,
+ "1740": 1083622400.0,
+ "1745": 1083622400.0,
+ "1750": 1083622400.0,
+ "1755": 1083622400.0,
+ "1760": 1083622400.0,
+ "1765": 1083622400.0,
+ "1770": 1083622400.0,
+ "1775": 1083622400.0,
+ "1780": 1083622400.0,
+ "1785": 1083622400.0,
+ "1790": 1083622400.0,
+ "1795": 1083622400.0,
+ "1800": 1083622400.0,
+ "1805": 1083622400.0,
+ "1810": 1083622400.0,
+ "1815": 1083622400.0,
+ "1820": 1083622400.0,
+ "1825": 1083622400.0,
+ "1830": 1083622400.0,
+ "1835": 1083622400.0,
+ "1840": 1083622400.0,
+ "1845": 1083622400.0,
+ "1850": 1083622400.0,
+ "1855": 1083622400.0,
+ "1860": 1083622400.0,
+ "1865": 1083622400.0,
+ "1870": 1083622400.0,
+ "1875": 1083622400.0,
+ "1880": 1083622400.0,
+ "1885": 1083622400.0,
+ "1890": 1083622400.0,
+ "1895": 1083622400.0,
+ "1900": 1083622400.0,
+ "1905": 1083622400.0,
+ "1910": 1083622400.0,
+ "1915": 1083622400.0,
+ "1920": 1083622400.0,
+ "1925": 1083622400.0,
+ "1930": 1083622400.0,
+ "1935": 1083622400.0,
+ "1940": 1083622400.0,
+ "1945": 1083622400.0,
+ "1950": 1083622400.0,
+ "1955": 1083622400.0,
+ "1960": 1083622400.0,
+ "1965": 1083622400.0,
+ "1970": 1083622400.0,
+ "1975": 1083622400.0,
+ "1980": 1083622400.0,
+ "1985": 1083622400.0,
+ "1990": 1083622400.0,
+ "1995": 1083622400.0,
+ "2000": 1083622400.0,
+ "2005": 1083622400.0,
+ "2010": 1083622400.0,
+ "2015": 1083622400.0,
+ "2020": 1083622400.0,
+ "2025": 1083622400.0,
+ "2030": 1083622400.0,
+ "2035": 1083622400.0,
+ "2040": 1083622400.0,
+ "2045": 1083622400.0,
+ "2050": 1083622400.0,
+ "2055": 1083622400.0,
+ "2060": 1083622400.0,
+ "2065": 1083622400.0,
+ "2070": 1083622400.0,
+ "2075": 1083622400.0,
+ "2080": 1083622400.0,
+ "2085": 1083622400.0,
+ "2090": 1083622400.0,
+ "2095": 1083622400.0,
+ "2100": 1083622400.0,
+ "2105": 1083622400.0,
+ "2110": 1083622400.0,
+ "2115": 1083622400.0,
+ "2120": 1083622400.0,
+ "2125": 1083622400.0,
+ "2130": 1083622400.0,
+ "2135": 1083622400.0,
+ "2140": 1083622400.0,
+ "2145": 1083622400.0,
+ "2150": 1083622400.0,
+ "2155": 1083622400.0,
+ "2160": 1083622400.0,
+ "2165": 1083622400.0,
+ "2170": 1083622400.0,
+ "2175": 1083622400.0,
+ "2180": 1083622400.0,
+ "2185": 1083622400.0,
+ "2190": 1083622400.0,
+ "2195": 1083622400.0,
+ "2200": 1083622400.0,
+ "2205": 1083622400.0,
+ "2210": 1083622400.0,
+ "2215": 1083622400.0,
+ "2220": 1083622400.0,
+ "2225": 1083622400.0,
+ "2230": 1083622400.0,
+ "2235": 1083622400.0,
+ "2240": 1083622400.0,
+ "2245": 1083622400.0,
+ "2250": 1083622400.0,
+ "2255": 1083622400.0,
+ "2260": 1083622400.0,
+ "2265": 1083622400.0,
+ "2270": 1083622400.0,
+ "2275": 1083622400.0,
+ "2280": 1083622400.0,
+ "2285": 1083622400.0,
+ "2290": 1083622400.0,
+ "2295": 1083622400.0,
+ "2300": 1083622400.0,
+ "2305": 1083622400.0,
+ "2310": 1083622400.0,
+ "2315": 1083622400.0,
+ "2320": 1083622400.0,
+ "2325": 1083622400.0,
+ "2330": 1083622400.0,
+ "2335": 1083622400.0,
+ "2340": 1083622400.0,
+ "2345": 1083622400.0,
+ "2350": 1083622400.0,
+ "2355": 1083622400.0,
+ "2360": 1083622400.0,
+ "2365": 1083622400.0,
+ "2370": 1083622400.0,
+ "2375": 1083622400.0,
+ "2380": 1083622400.0,
+ "2385": 1083622400.0,
+ "2390": 1083622400.0,
+ "2395": 1083622400.0,
+ "2400": 1083622400.0,
+ "2405": 1083622400.0,
+ "2410": 1083622400.0,
+ "2415": 1083622400.0,
+ "2420": 1083622400.0,
+ "2425": 1083622400.0,
+ "2430": 1083622400.0,
+ "2435": 1083622400.0,
+ "2440": 1083622400.0,
+ "2445": 1083622400.0,
+ "2450": 1083622400.0,
+ "2455": 1083622400.0,
+ "2460": 1083622400.0,
+ "2465": 1083622400.0,
+ "2470": 1083622400.0,
+ "2475": 1083622400.0,
+ "2480": 1083622400.0,
+ "2485": 1083622400.0,
+ "2490": 1083622400.0,
+ "2495": 1083622400.0,
+ "2500": 1083622400.0,
+ "2505": 1083622400.0,
+ "2510": 1083622400.0,
+ "2515": 1083622400.0,
+ "2520": 1083622400.0,
+ "2525": 1083622400.0,
+ "2530": 1083622400.0,
+ "2535": 1083622400.0,
+ "2540": 1083622400.0,
+ "2545": 1083622400.0,
+ "2550": 1083622400.0,
+ "2555": 1083622400.0,
+ "2560": 1083622400.0,
+ "2565": 1083622400.0,
+ "2570": 1083622400.0,
+ "2575": 1083622400.0,
+ "2580": 1083622400.0,
+ "2585": 1083622400.0,
+ "2590": 1083622400.0,
+ "2595": 1083622400.0,
+ "2600": 1083622400.0,
+ "2605": 1083622400.0,
+ "2610": 1083622400.0,
+ "2615": 1083622400.0,
+ "2620": 1083622400.0,
+ "2625": 1083622400.0,
+ "2630": 1083622400.0,
+ "2635": 1083622400.0,
+ "2640": 1083622400.0,
+ "2645": 1083622400.0,
+ "2650": 1083622400.0,
+ "2655": 1083622400.0,
+ "2660": 1083622400.0,
+ "2665": 1083622400.0,
+ "2670": 1083622400.0,
+ "2675": 1083622400.0,
+ "2680": 1083622400.0,
+ "2685": 1083622400.0,
+ "2690": 1083622400.0,
+ "2695": 1083622400.0,
+ "2700": 1083622400.0,
+ "2705": 1083622400.0,
+ "2710": 1083622400.0,
+ "2715": 1083622400.0,
+ "2720": 1083622400.0,
+ "2725": 1083622400.0,
+ "2730": 1083622400.0,
+ "2735": 1083622400.0,
+ "2740": 1083622400.0,
+ "2745": 1083622400.0,
+ "2750": 1083622400.0,
+ "2755": 1083622400.0,
+ "2760": 1083622400.0,
+ "2765": 1083622400.0,
+ "2770": 1083622400.0,
+ "2775": 1083622400.0,
+ "2780": 1083622400.0,
+ "2785": 1083622400.0,
+ "2790": 1083622400.0,
+ "2795": 1083622400.0,
+ "2800": 1083622400.0,
+ "2805": 1083622400.0,
+ "2810": 1083622400.0,
+ "2815": 1083622400.0,
+ "2820": 1083622400.0,
+ "2825": 1083622400.0,
+ "2830": 1083622400.0,
+ "2835": 1083622400.0,
+ "2840": 1083622400.0,
+ "2845": 1083622400.0,
+ "2850": 1083622400.0,
+ "2855": 1083622400.0,
+ "2860": 1083622400.0,
+ "2865": 1083622400.0,
+ "2870": 1083622400.0,
+ "2875": 1083622400.0,
+ "2880": 1083622400.0,
+ "2885": 1083622400.0,
+ "2890": 1083622400.0,
+ "2895": 1083622400.0,
+ "2900": 1083622400.0,
+ "2905": 1083622400.0,
+ "2910": 1083622400.0,
+ "2915": 1083622400.0,
+ "2920": 1083622400.0,
+ "2925": 1083622400.0,
+ "2930": 1083622400.0,
+ "2935": 1083622400.0,
+ "2940": 1083622400.0,
+ "2945": 1083622400.0,
+ "2950": 1083622400.0,
+ "2955": 1083622400.0,
+ "2960": 1083622400.0,
+ "2965": 1083622400.0,
+ "2970": 1083622400.0,
+ "2975": 1083622400.0,
+ "2980": 1083622400.0,
+ "2985": 1083622400.0,
+ "2990": 1083622400.0,
+ "2995": 1083622400.0,
+ "3000": 1083622400.0,
+ "3005": 1083622400.0,
+ "3010": 1083622400.0,
+ "3015": 1083622400.0,
+ "3020": 1083622400.0,
+ "3025": 1083622400.0,
+ "3030": 1083622400.0,
+ "3035": 1083622400.0,
+ "3040": 1083622400.0,
+ "3045": 1083622400.0,
+ "3050": 1083622400.0,
+ "3055": 1083622400.0,
+ "3060": 1083622400.0,
+ "3065": 1083622400.0,
+ "3070": 1083622400.0,
+ "3075": 1083622400.0,
+ "3080": 1083622400.0,
+ "3085": 1083622400.0,
+ "3090": 1083622400.0,
+ "3095": 1083622400.0,
+ "3100": 1083622400.0,
+ "3105": 1083622400.0,
+ "3110": 1083622400.0,
+ "3115": 1083622400.0,
+ "3120": 1083622400.0,
+ "3125": 1083622400.0,
+ "3130": 1083622400.0,
+ "3135": 1083622400.0,
+ "3140": 1083622400.0,
+ "3145": 1083622400.0,
+ "3150": 1083622400.0,
+ "3155": 1083622400.0,
+ "3160": 1083622400.0,
+ "3165": 1083622400.0,
+ "3170": 1083622400.0,
+ "3175": 1083622400.0,
+ "3180": 1083622400.0,
+ "3185": 1083622400.0,
+ "3190": 1083622400.0,
+ "3195": 1083622400.0,
+ "3200": 1083622400.0,
+ "3205": 1083622400.0,
+ "3210": 1083622400.0,
+ "3215": 1083622400.0,
+ "3220": 1083622400.0,
+ "3225": 1083622400.0,
+ "3230": 1083622400.0,
+ "3235": 1083622400.0,
+ "3240": 1083622400.0,
+ "3245": 1083622400.0,
+ "3250": 1083622400.0,
+ "3255": 1083622400.0,
+ "3260": 1083622400.0,
+ "3265": 1083622400.0,
+ "3270": 1083622400.0,
+ "3275": 1083622400.0,
+ "3280": 1083622400.0,
+ "3285": 1083622400.0,
+ "3290": 1083622400.0,
+ "3295": 1083622400.0,
+ "3300": 1083622400.0,
+ "3305": 1083622400.0,
+ "3310": 1083622400.0,
+ "3315": 1083622400.0,
+ "3320": 1083622400.0,
+ "3325": 1083622400.0,
+ "3330": 1083622400.0,
+ "3335": 1083622400.0,
+ "3340": 1083622400.0,
+ "3345": 1083622400.0,
+ "3350": 1083622400.0,
+ "3355": 1083622400.0,
+ "3360": 1083622400.0,
+ "3365": 1083622400.0,
+ "3370": 1083622400.0,
+ "3375": 1083622400.0,
+ "3380": 1083622400.0,
+ "3385": 1083622400.0,
+ "3390": 1083622400.0,
+ "3395": 1083622400.0,
+ "3400": 1083622400.0,
+ "3405": 1083622400.0,
+ "3410": 1083622400.0,
+ "3415": 1083622400.0,
+ "3420": 1083622400.0,
+ "3425": 1083622400.0,
+ "3430": 1083622400.0,
+ "3435": 1083622400.0,
+ "3440": 1083622400.0,
+ "3445": 1083622400.0,
+ "3450": 1083622400.0,
+ "3455": 1083622400.0,
+ "3460": 1083622400.0,
+ "3465": 1083622400.0,
+ "3470": 1083622400.0,
+ "3475": 1083622400.0,
+ "3480": 1083622400.0,
+ "3485": 1083622400.0,
+ "3490": 1083622400.0,
+ "3495": 1083622400.0,
+ "3500": 1083622400.0,
+ "3505": 1083622400.0,
+ "3510": 1083622400.0,
+ "3515": 1083622400.0,
+ "3520": 1083622400.0,
+ "3525": 1083622400.0,
+ "3530": 1083622400.0,
+ "3535": 1083622400.0,
+ "3540": 1083622400.0,
+ "3545": 1083622400.0,
+ "3550": 1083622400.0,
+ "3555": 1083622400.0,
+ "3560": 1083622400.0,
+ "3565": 1083622400.0,
+ "3570": 1083622400.0,
+ "3575": 1083622400.0,
+ "3580": 1083622400.0,
+ "3585": 1083622400.0,
+ "3590": 1083622400.0,
+ "3595": 1083622400.0,
+ "3600": 1083622400.0,
+ "3605": 1083622400.0,
+ "3610": 1083622400.0,
+ "3615": 1083622400.0,
+ "3620": 1083622400.0,
+ "3625": 1083622400.0,
+ "3630": 1083622400.0,
+ "3635": 1083622400.0,
+ "3640": 1083622400.0,
+ "3645": 1083622400.0,
+ "3650": 1083622400.0,
+ "3655": 1083622400.0,
+ "3660": 1083622400.0,
+ "3665": 1083622400.0,
+ "3670": 1083622400.0,
+ "3675": 1083622400.0,
+ "3680": 1083622400.0,
+ "3685": 1083622400.0,
+ "3690": 1083622400.0,
+ "3695": 1083622400.0,
+ "3700": 1083622400.0,
+ "3705": 1083622400.0,
+ "3710": 1083622400.0,
+ "3715": 1083622400.0,
+ "3720": 1083622400.0,
+ "3725": 1083622400.0,
+ "3730": 1083622400.0,
+ "3735": 1083622400.0,
+ "3740": 1083622400.0,
+ "3745": 1083622400.0,
+ "3750": 1083622400.0,
+ "3755": 1083622400.0,
+ "3760": 1083622400.0,
+ "3765": 1083622400.0,
+ "3770": 1083622400.0,
+ "3775": 1083622400.0,
+ "3780": 1083622400.0,
+ "3785": 1083622400.0,
+ "3790": 1083622400.0,
+ "3795": 1083622400.0,
+ "3800": 1083622400.0,
+ "3805": 1083622400.0,
+ "3810": 1083622400.0,
+ "3815": 1083622400.0,
+ "3820": 1083622400.0,
+ "3825": 1083622400.0,
+ "3830": 1083622400.0,
+ "3835": 1083622400.0,
+ "3840": 1083622400.0,
+ "3845": 1083622400.0,
+ "3850": 1083622400.0,
+ "3855": 1083622400.0,
+ "3860": 1083622400.0,
+ "3865": 1083622400.0,
+ "3870": 1083622400.0,
+ "3875": 1083622400.0,
+ "3880": 1083622400.0,
+ "3885": 1083622400.0,
+ "3890": 1083622400.0,
+ "3895": 1083622400.0,
+ "3900": 1083622400.0,
+ "3905": 1083622400.0,
+ "3910": 1083622400.0,
+ "3915": 1083622400.0,
+ "3920": 1083622400.0,
+ "3925": 1083622400.0,
+ "3930": 1083622400.0,
+ "3935": 1083622400.0,
+ "3940": 1083622400.0,
+ "3945": 1083622400.0,
+ "3950": 1083622400.0,
+ "3955": 1083622400.0,
+ "3960": 1083622400.0,
+ "3965": 1083622400.0,
+ "3970": 1083622400.0,
+ "3975": 1083622400.0,
+ "3980": 1083622400.0,
+ "3985": 1083622400.0,
+ "3990": 1083622400.0,
+ "3995": 1083622400.0,
+ "4000": 1083622400.0,
+ "4005": 1083622400.0,
+ "4010": 1083622400.0,
+ "4015": 1083622400.0,
+ "4020": 1083622400.0,
+ "4025": 1083622400.0,
+ "4030": 1083622400.0,
+ "4035": 1083622400.0,
+ "4040": 1083622400.0,
+ "4045": 1083622400.0,
+ "4050": 1083622400.0,
+ "4055": 1083622400.0,
+ "4060": 1083622400.0,
+ "4065": 1083622400.0,
+ "4070": 1083622400.0,
+ "4075": 1083622400.0,
+ "4080": 1083622400.0,
+ "4085": 1083622400.0,
+ "4090": 1083622400.0,
+ "4095": 1083622400.0,
+ "4100": 1083622400.0,
+ "4105": 1083622400.0,
+ "4110": 1083622400.0,
+ "4115": 1083622400.0,
+ "4120": 1083622400.0,
+ "4125": 1083622400.0,
+ "4130": 1083622400.0,
+ "4135": 1083622400.0,
+ "4140": 1083622400.0,
+ "4145": 1083622400.0,
+ "4150": 1083622400.0,
+ "4155": 1083622400.0,
+ "4160": 1083622400.0,
+ "4165": 1083622400.0,
+ "4170": 1083622400.0,
+ "4175": 1083622400.0,
+ "4180": 1083622400.0,
+ "4185": 1083622400.0,
+ "4190": 1083622400.0,
+ "4195": 1083622400.0,
+ "4200": 1083622400.0,
+ "4205": 1083622400.0,
+ "4210": 1083622400.0,
+ "4215": 1083622400.0,
+ "4220": 1083622400.0,
+ "4225": 1083622400.0,
+ "4230": 1083622400.0,
+ "4235": 1083622400.0,
+ "4240": 1083622400.0,
+ "4245": 1083622400.0,
+ "4250": 1083622400.0,
+ "4255": 1083622400.0,
+ "4260": 1083622400.0,
+ "4265": 1083622400.0,
+ "4270": 1083622400.0,
+ "4275": 1083622400.0,
+ "4280": 1083622400.0,
+ "4285": 1083622400.0,
+ "4290": 1083622400.0,
+ "4295": 1083622400.0,
+ "4300": 1083622400.0,
+ "4305": 1083622400.0,
+ "4310": 1083622400.0,
+ "4315": 1083622400.0,
+ "4320": 1083622400.0,
+ "4325": 1083622400.0,
+ "4330": 1083622400.0,
+ "4335": 1083622400.0,
+ "4340": 1083622400.0,
+ "4345": 1083622400.0,
+ "4350": 1083622400.0,
+ "4355": 1083622400.0,
+ "4360": 1083622400.0,
+ "4365": 1083622400.0,
+ "4370": 1083622400.0,
+ "4375": 1083622400.0,
+ "4380": 1083622400.0,
+ "4385": 1083622400.0,
+ "4390": 1083622400.0,
+ "4395": 1083622400.0,
+ "4400": 1083622400.0,
+ "4405": 1083622400.0,
+ "4410": 1083622400.0,
+ "4415": 1083622400.0,
+ "4420": 1083622400.0,
+ "4425": 1083622400.0,
+ "4430": 1083622400.0,
+ "4435": 1083622400.0,
+ "4440": 1083622400.0,
+ "4445": 1083622400.0,
+ "4450": 1083622400.0,
+ "4455": 1083622400.0,
+ "4460": 1083622400.0,
+ "4465": 1083622400.0,
+ "4470": 1083622400.0,
+ "4475": 1083622400.0,
+ "4480": 1083622400.0,
+ "4485": 1083622400.0,
+ "4490": 1083622400.0,
+ "4495": 1083622400.0,
+ "4500": 1083622400.0,
+ "4505": 1083622400.0,
+ "4510": 1083622400.0,
+ "4515": 1083622400.0,
+ "4520": 1083622400.0,
+ "4525": 1083622400.0,
+ "4530": 1083622400.0,
+ "4535": 1083622400.0,
+ "4540": 1083622400.0,
+ "4545": 1083622400.0,
+ "4550": 1083622400.0,
+ "4555": 1083622400.0,
+ "4560": 1083622400.0,
+ "4565": 1083622400.0,
+ "4570": 1083622400.0,
+ "4575": 1083622400.0,
+ "4580": 1083622400.0,
+ "4585": 1083622400.0,
+ "4590": 1083622400.0,
+ "4595": 1083622400.0,
+ "4600": 1083622400.0,
+ "4605": 1083622400.0,
+ "4610": 1083622400.0,
+ "4615": 1083622400.0,
+ "4620": 1083622400.0,
+ "4625": 1083622400.0,
+ "4630": 1083622400.0,
+ "4635": 1083622400.0,
+ "4640": 1083622400.0,
+ "4645": 1083622400.0,
+ "4650": 1083622400.0,
+ "4655": 1083622400.0,
+ "4660": 1083622400.0,
+ "4665": 1083622400.0,
+ "4670": 1083622400.0,
+ "4675": 1083622400.0,
+ "4680": 1083622400.0,
+ "4685": 1083622400.0,
+ "4690": 1083622400.0,
+ "4695": 1083622400.0,
+ "4700": 1083622400.0,
+ "4705": 1083622400.0,
+ "4710": 1083622400.0,
+ "4715": 1083622400.0,
+ "4720": 1083622400.0,
+ "4725": 1083622400.0,
+ "4730": 1083622400.0,
+ "4735": 1083622400.0,
+ "4740": 1083622400.0,
+ "4745": 1083622400.0,
+ "4750": 1083622400.0,
+ "4755": 1083622400.0,
+ "4760": 1083622400.0,
+ "4765": 1083622400.0,
+ "4770": 1083622400.0,
+ "4775": 1083622400.0,
+ "4780": 1083622400.0,
+ "4785": 1083622400.0,
+ "4790": 1083622400.0,
+ "4795": 1083622400.0,
+ "4800": 1083622400.0,
+ "4805": 1083622400.0,
+ "4810": 1083622400.0,
+ "4815": 1083622400.0,
+ "4820": 1083622400.0,
+ "4825": 1083622400.0,
+ "4830": 1083622400.0,
+ "4835": 1083622400.0,
+ "4840": 1083622400.0,
+ "4845": 1083622400.0,
+ "4850": 1083622400.0,
+ "4855": 1083622400.0,
+ "4860": 1083622400.0,
+ "4865": 1083622400.0,
+ "4870": 1083622400.0,
+ "4875": 1083622400.0,
+ "4880": 1083622400.0,
+ "4885": 1083622400.0,
+ "4890": 1083622400.0,
+ "4895": 1083622400.0,
+ "4900": 1083622400.0,
+ "4905": 1083622400.0,
+ "4910": 1083622400.0,
+ "4915": 1083622400.0,
+ "4920": 1083622400.0,
+ "4925": 1083622400.0,
+ "4930": 1083622400.0,
+ "4935": 1083622400.0,
+ "4940": 1083622400.0,
+ "4945": 1083622400.0,
+ "4950": 1083622400.0,
+ "4955": 1083622400.0,
+ "4960": 1083622400.0,
+ "4965": 1083622400.0,
+ "4970": 1083622400.0,
+ "4975": 1083622400.0,
+ "4980": 1083622400.0,
+ "4985": 1083622400.0,
+ "4990": 1083622400.0,
+ "4995": 1083622400.0,
+ "5000": 1083622400.0,
+ "5005": 1083622400.0,
+ "5010": 1083622400.0,
+ "5015": 1083622400.0,
+ "5020": 1083622400.0,
+ "5025": 1083622400.0,
+ "5030": 1083622400.0,
+ "5035": 1083622400.0,
+ "5040": 1083622400.0,
+ "5045": 1083622400.0,
+ "5050": 1083622400.0,
+ "5055": 1083622400.0,
+ "5060": 1083622400.0,
+ "5065": 1083622400.0,
+ "5070": 1083622400.0,
+ "5075": 1083622400.0,
+ "5080": 1083622400.0,
+ "5085": 1083622400.0,
+ "5090": 1083622400.0,
+ "5095": 1083622400.0,
+ "5100": 1083622400.0,
+ "5105": 1083622400.0,
+ "5110": 1083622400.0,
+ "5115": 1083622400.0,
+ "5120": 1083622400.0,
+ "5125": 1083622400.0,
+ "5130": 1083622400.0,
+ "5135": 1083622400.0,
+ "5140": 1083622400.0,
+ "5145": 1083622400.0,
+ "5150": 1083622400.0,
+ "5155": 1083622400.0,
+ "5160": 1083622400.0,
+ "5165": 1083622400.0,
+ "5170": 1083622400.0,
+ "5175": 1083622400.0,
+ "5180": 1083622400.0,
+ "5185": 1083622400.0,
+ "5190": 1083622400.0,
+ "5195": 1083622400.0,
+ "5200": 1083622400.0,
+ "5205": 1083622400.0,
+ "5210": 1083622400.0,
+ "5215": 1083622400.0,
+ "5220": 1083622400.0,
+ "5225": 1083622400.0,
+ "5230": 1083622400.0,
+ "5235": 1083622400.0,
+ "5240": 1083622400.0,
+ "5245": 1083622400.0,
+ "5250": 1083622400.0,
+ "5255": 1083622400.0,
+ "5260": 1083622400.0,
+ "5265": 1083622400.0,
+ "5270": 1083622400.0,
+ "5275": 1083622400.0,
+ "5280": 1083622400.0,
+ "5285": 1083622400.0,
+ "5290": 1083622400.0,
+ "5295": 1083622400.0,
+ "5300": 1083622400.0,
+ "5305": 1083622400.0,
+ "5310": 1083622400.0,
+ "5315": 1083622400.0,
+ "5320": 1083622400.0,
+ "5325": 1083622400.0,
+ "5330": 1083622400.0,
+ "5335": 1083622400.0,
+ "5340": 1083622400.0,
+ "5345": 1083622400.0,
+ "5350": 1083622400.0,
+ "5355": 1083622400.0,
+ "5360": 1083622400.0,
+ "5365": 1083622400.0,
+ "5370": 1083622400.0,
+ "5375": 1083622400.0,
+ "5380": 1083622400.0,
+ "5385": 1083622400.0,
+ "5390": 1083622400.0,
+ "5395": 1083622400.0,
+ "5400": 1083622400.0,
+ "5405": 1083622400.0,
+ "5410": 1083622400.0,
+ "5415": 1083622400.0,
+ "5420": 1083622400.0,
+ "5425": 1083622400.0,
+ "5430": 1083622400.0,
+ "5435": 1083622400.0,
+ "5440": 1083622400.0,
+ "5445": 1083622400.0,
+ "5450": 1083622400.0,
+ "5455": 1083622400.0,
+ "5460": 1083622400.0,
+ "5465": 1083622400.0,
+ "5470": 1083622400.0,
+ "5475": 1083622400.0,
+ "5480": 1083622400.0,
+ "5485": 1083622400.0,
+ "5490": 1083622400.0,
+ "5495": 1083622400.0,
+ "5500": 1083622400.0,
+ "5505": 1083622400.0,
+ "5510": 1083622400.0,
+ "5515": 1083622400.0,
+ "5520": 1083622400.0,
+ "5525": 1083622400.0,
+ "5530": 1083622400.0,
+ "5535": 1083622400.0,
+ "5540": 1083622400.0,
+ "5545": 1083622400.0,
+ "5550": 1083622400.0,
+ "5555": 1083622400.0,
+ "5560": 1083622400.0,
+ "5565": 1083622400.0,
+ "5570": 1083622400.0,
+ "5575": 1083622400.0,
+ "5580": 1083622400.0,
+ "5585": 1083622400.0,
+ "5590": 1083622400.0,
+ "5595": 1083622400.0,
+ "5600": 1083622400.0,
+ "5605": 1083622400.0,
+ "5610": 1083622400.0,
+ "5615": 1083622400.0,
+ "5620": 1083622400.0,
+ "5625": 1083622400.0,
+ "5630": 1083622400.0,
+ "5635": 1083622400.0,
+ "5640": 1083622400.0,
+ "5645": 1083622400.0,
+ "5650": 1083622400.0,
+ "5655": 1083622400.0,
+ "5660": 1083622400.0,
+ "5665": 1083622400.0,
+ "5670": 1083622400.0,
+ "5675": 1083622400.0,
+ "5680": 1083622400.0,
+ "5685": 1083622400.0,
+ "5690": 1083622400.0,
+ "5695": 1083622400.0,
+ "5700": 1083622400.0,
+ "5705": 1083622400.0,
+ "5710": 1083622400.0,
+ "5715": 1083622400.0,
+ "5720": 1083622400.0,
+ "5725": 1083622400.0,
+ "5730": 1083622400.0,
+ "5735": 1083622400.0,
+ "5740": 1083622400.0,
+ "5745": 1083622400.0,
+ "5750": 1083622400.0,
+ "5755": 1083622400.0,
+ "5760": 1083622400.0,
+ "5765": 1083622400.0,
+ "5770": 1083622400.0,
+ "5775": 1083622400.0,
+ "5780": 1083622400.0,
+ "5785": 1083622400.0,
+ "5790": 1083622400.0,
+ "5795": 1083622400.0,
+ "5800": 1083622400.0,
+ "5805": 1083622400.0,
+ "5810": 1083622400.0,
+ "5815": 1083622400.0,
+ "5820": 1083622400.0,
+ "5825": 1083622400.0,
+ "5830": 1083622400.0,
+ "5835": 1083622400.0,
+ "5840": 1083622400.0,
+ "5845": 1083622400.0,
+ "5850": 1083622400.0,
+ "5855": 1083622400.0,
+ "5860": 1083622400.0,
+ "5865": 1083622400.0,
+ "5870": 1083622400.0,
+ "5875": 1083622400.0,
+ "5880": 1083622400.0,
+ "5885": 1083622400.0,
+ "5890": 1083622400.0,
+ "5895": 1083622400.0,
+ "5900": 1083622400.0,
+ "5905": 1083622400.0,
+ "5910": 1083622400.0,
+ "5915": 1083622400.0,
+ "5920": 1083622400.0,
+ "5925": 1083622400.0,
+ "5930": 1083622400.0,
+ "5935": 1083622400.0,
+ "5940": 1083622400.0,
+ "5945": 1083622400.0,
+ "5950": 1083622400.0,
+ "5955": 1083622400.0,
+ "5960": 1083622400.0,
+ "5965": 1083622400.0,
+ "5970": 1083622400.0,
+ "5975": 1083622400.0,
+ "5980": 1083622400.0,
+ "5985": 1083622400.0,
+ "5990": 1083622400.0,
+ "5995": 1083622400.0,
+ "6000": 1083622400.0,
+ "6005": 1083622400.0,
+ "6010": 1083622400.0,
+ "6015": 1083622400.0,
+ "6020": 1083622400.0,
+ "6025": 1083622400.0,
+ "6030": 1083622400.0,
+ "6035": 1083622400.0,
+ "6040": 1083622400.0,
+ "6045": 1083622400.0,
+ "6050": 1083622400.0,
+ "6055": 1083622400.0,
+ "6060": 1083622400.0,
+ "6065": 1083622400.0,
+ "6070": 1083622400.0,
+ "6075": 1083622400.0,
+ "6080": 1083622400.0,
+ "6085": 1083622400.0,
+ "6090": 1083622400.0,
+ "6095": 1083622400.0,
+ "6100": 1083622400.0,
+ "6105": 1083622400.0,
+ "6110": 1083622400.0,
+ "6115": 1083622400.0,
+ "6120": 1083622400.0,
+ "6125": 1083622400.0,
+ "6130": 1083622400.0,
+ "6135": 1083622400.0,
+ "6140": 1083622400.0,
+ "6145": 1083622400.0,
+ "6150": 1083622400.0,
+ "6155": 1083622400.0,
+ "6160": 1083622400.0,
+ "6165": 1083622400.0,
+ "6170": 1083622400.0,
+ "6175": 1083622400.0,
+ "6180": 1083622400.0,
+ "6185": 1083622400.0,
+ "6190": 1083622400.0,
+ "6195": 1083622400.0,
+ "6200": 1083622400.0,
+ "6205": 1083622400.0,
+ "6210": 1083622400.0,
+ "6215": 1083622400.0,
+ "6220": 1083622400.0,
+ "6225": 1083622400.0,
+ "6230": 1083622400.0,
+ "6235": 1083622400.0,
+ "6240": 1083622400.0,
+ "6245": 1083622400.0,
+ "6250": 1083622400.0,
+ "6255": 1083622400.0,
+ "6260": 1083622400.0,
+ "6265": 1083622400.0,
+ "6270": 1083622400.0,
+ "6275": 1083622400.0,
+ "6280": 1083622400.0,
+ "6285": 1083622400.0,
+ "6290": 1083622400.0,
+ "6295": 1083622400.0,
+ "6300": 1083622400.0,
+ "6305": 1083622400.0,
+ "6310": 1083622400.0,
+ "6315": 1083622400.0,
+ "6320": 1083622400.0,
+ "6325": 1083622400.0,
+ "6330": 1083622400.0,
+ "6335": 1083622400.0,
+ "6340": 1083622400.0,
+ "6345": 1083622400.0,
+ "6350": 1083622400.0,
+ "6355": 1083622400.0,
+ "6360": 1083622400.0,
+ "6365": 1083622400.0,
+ "6370": 1083622400.0,
+ "6375": 1083622400.0,
+ "6380": 1083622400.0,
+ "6385": 1083622400.0,
+ "6390": 1083622400.0,
+ "6395": 1083622400.0,
+ "6400": 1083622400.0,
+ "6405": 1083622400.0,
+ "6410": 1083622400.0,
+ "6415": 1083622400.0,
+ "6420": 1083622400.0,
+ "6425": 1083622400.0,
+ "6430": 1083622400.0,
+ "6435": 1083622400.0,
+ "6440": 1083622400.0,
+ "6445": 1083622400.0,
+ "6450": 1083622400.0,
+ "6455": 1083622400.0,
+ "6460": 1083622400.0,
+ "6465": 1083622400.0,
+ "6470": 1083622400.0,
+ "6475": 1083622400.0,
+ "6480": 1083622400.0,
+ "6485": 1083622400.0,
+ "6490": 1083622400.0,
+ "6495": 1083622400.0,
+ "6500": 1083622400.0,
+ "6505": 1083622400.0,
+ "6510": 1083622400.0,
+ "6515": 1083622400.0,
+ "6520": 1083622400.0,
+ "6525": 1083622400.0,
+ "6530": 1083622400.0,
+ "6535": 1083622400.0,
+ "6540": 1083622400.0,
+ "6545": 1083622400.0,
+ "6550": 1083622400.0,
+ "6555": 1083622400.0,
+ "6560": 1083622400.0,
+ "6565": 1083622400.0,
+ "6570": 1083622400.0,
+ "6575": 1083622400.0,
+ "6580": 1083622400.0,
+ "6585": 1083622400.0,
+ "6590": 1083622400.0,
+ "6595": 1083622400.0,
+ "6600": 1083622400.0,
+ "6605": 1083622400.0,
+ "6610": 1083622400.0,
+ "6615": 1083622400.0,
+ "6620": 1083622400.0,
+ "6625": 1083622400.0,
+ "6630": 1083622400.0,
+ "6635": 1083622400.0,
+ "6640": 1083622400.0,
+ "6645": 1083622400.0,
+ "6650": 1083622400.0,
+ "6655": 1083622400.0,
+ "6660": 1083622400.0,
+ "6665": 1083622400.0,
+ "6670": 1083622400.0,
+ "6675": 1083622400.0,
+ "6680": 1083622400.0,
+ "6685": 1083622400.0,
+ "6690": 1083622400.0,
+ "6695": 1083622400.0,
+ "6700": 1083622400.0,
+ "6705": 1083622400.0,
+ "6710": 1083622400.0,
+ "6715": 1083622400.0,
+ "6720": 1083622400.0,
+ "6725": 1083622400.0,
+ "6730": 1083622400.0,
+ "6735": 1083622400.0,
+ "6740": 1083622400.0,
+ "6745": 1083622400.0,
+ "6750": 1083622400.0,
+ "6755": 1083622400.0,
+ "6760": 1083622400.0,
+ "6765": 1083622400.0,
+ "6770": 1083622400.0,
+ "6775": 1083622400.0,
+ "6780": 1083622400.0,
+ "6785": 1083622400.0,
+ "6790": 1083622400.0,
+ "6795": 1083622400.0,
+ "6800": 1083622400.0,
+ "6805": 1083622400.0,
+ "6810": 1083622400.0,
+ "6815": 1083622400.0,
+ "6820": 1083622400.0,
+ "6825": 1083622400.0,
+ "6830": 1083622400.0,
+ "6835": 1083622400.0,
+ "6840": 1083622400.0,
+ "6845": 1083622400.0,
+ "6850": 1083622400.0,
+ "6855": 1083622400.0,
+ "6860": 1083622400.0,
+ "6865": 1083622400.0,
+ "6870": 1083622400.0,
+ "6875": 1083622400.0,
+ "6880": 1083622400.0,
+ "6885": 1083622400.0,
+ "6890": 1083622400.0,
+ "6895": 1083622400.0,
+ "6900": 1083622400.0,
+ "6905": 1083622400.0,
+ "6910": 1083622400.0,
+ "6915": 1083622400.0,
+ "6920": 1083622400.0,
+ "6925": 1083622400.0,
+ "6930": 1083622400.0,
+ "6935": 1083622400.0,
+ "6940": 1083622400.0,
+ "6945": 1083622400.0,
+ "6950": 1083622400.0,
+ "6955": 1083622400.0,
+ "6960": 1083622400.0,
+ "6965": 1083622400.0,
+ "6970": 1083622400.0,
+ "6975": 1083622400.0,
+ "6980": 1083622400.0,
+ "6985": 1083622400.0,
+ "6990": 1083622400.0,
+ "6995": 1083622400.0,
+ "7000": 1083622400.0,
+ "7005": 1083622400.0,
+ "7010": 1083622400.0,
+ "7015": 1083622400.0,
+ "7020": 1083622400.0,
+ "7025": 1083622400.0,
+ "7030": 1083622400.0,
+ "7035": 1083622400.0,
+ "7040": 1083622400.0,
+ "7045": 1083622400.0,
+ "7050": 1083622400.0,
+ "7055": 1083622400.0,
+ "7060": 1083622400.0,
+ "7065": 1083622400.0,
+ "7070": 1083622400.0,
+ "7075": 1083622400.0,
+ "7080": 1083622400.0,
+ "7085": 1083622400.0,
+ "7090": 1083622400.0,
+ "7095": 1083622400.0,
+ "7100": 1083622400.0,
+ "7105": 1083622400.0,
+ "7110": 1083622400.0,
+ "7115": 1083622400.0,
+ "7120": 1083622400.0,
+ "7125": 1083622400.0,
+ "7130": 1083622400.0,
+ "7135": 1083622400.0,
+ "7140": 1083622400.0,
+ "7145": 1083622400.0,
+ "7150": 1083622400.0,
+ "7155": 1083622400.0,
+ "7160": 1083622400.0,
+ "7165": 1083622400.0,
+ "7170": 1083622400.0,
+ "7175": 1083622400.0,
+ "7180": 1083622400.0,
+ "7185": 1083622400.0,
+ "7190": 1083622400.0,
+ "7195": 1083622400.0,
+ "7200": 1083622400.0,
+ "7205": 1083622400.0,
+ "7210": 1083622400.0,
+ "7215": 1083622400.0,
+ "7220": 1083622400.0,
+ "7225": 1083622400.0,
+ "7230": 1083622400.0,
+ "7235": 1083622400.0,
+ "7240": 1083622400.0,
+ "7245": 1083622400.0,
+ "7250": 1083622400.0,
+ "7255": 1083622400.0,
+ "7260": 1083622400.0,
+ "7265": 1083622400.0,
+ "7270": 1083622400.0,
+ "7275": 1083622400.0,
+ "7280": 1083622400.0,
+ "7285": 1083622400.0,
+ "7290": 1083622400.0,
+ "7295": 1083622400.0,
+ "7300": 1083622400.0,
+ "7305": 1083622400.0,
+ "7310": 1083622400.0,
+ "7315": 1083622400.0,
+ "7320": 1083622400.0,
+ "7325": 1083622400.0,
+ "7330": 1083622400.0,
+ "7335": 1083622400.0,
+ "7340": 1083622400.0,
+ "7345": 1083622400.0,
+ "7350": 1083622400.0,
+ "7355": 1083622400.0,
+ "7360": 1083622400.0,
+ "7365": 1083622400.0,
+ "7370": 1083622400.0,
+ "7375": 1083622400.0,
+ "7380": 1083622400.0,
+ "7385": 1083622400.0,
+ "7390": 1083622400.0,
+ "7395": 1083622400.0,
+ "7400": 1083622400.0,
+ "7405": 1083622400.0,
+ "7410": 1083622400.0,
+ "7415": 1083622400.0,
+ "7420": 1083622400.0,
+ "7425": 1083622400.0,
+ "7430": 1083622400.0,
+ "7435": 1083622400.0,
+ "7440": 1083622400.0,
+ "7445": 1083622400.0,
+ "7450": 1083622400.0,
+ "7455": 1083622400.0,
+ "7460": 1083622400.0,
+ "7465": 1083622400.0,
+ "7470": 1083622400.0,
+ "7475": 1083622400.0,
+ "7480": 1083622400.0,
+ "7485": 1083622400.0,
+ "7490": 1083622400.0,
+ "7495": 1083622400.0,
+ "7500": 1083622400.0,
+ "7505": 1083622400.0,
+ "7510": 1083622400.0,
+ "7515": 1083622400.0,
+ "7520": 1083622400.0,
+ "7525": 1083622400.0,
+ "7530": 1083622400.0,
+ "7535": 1083622400.0,
+ "7540": 1083622400.0,
+ "7545": 1083622400.0,
+ "7550": 1083622400.0,
+ "7555": 1083622400.0,
+ "7560": 1083622400.0,
+ "7565": 1083622400.0,
+ "7570": 1083622400.0,
+ "7575": 1083622400.0,
+ "7580": 1083622400.0,
+ "7585": 1083622400.0,
+ "7590": 1083622400.0,
+ "7595": 1083622400.0,
+ "7600": 1083622400.0,
+ "7605": 1083622400.0,
+ "7610": 1083622400.0,
+ "7615": 1083622400.0,
+ "7620": 1083622400.0,
+ "7625": 1083622400.0,
+ "7630": 1083622400.0,
+ "7635": 1083622400.0,
+ "7640": 1083622400.0,
+ "7645": 1083622400.0,
+ "7650": 1083622400.0,
+ "7655": 1083622400.0,
+ "7660": 1083622400.0,
+ "7665": 1083622400.0,
+ "7670": 1083622400.0,
+ "7675": 1083622400.0,
+ "7680": 1083622400.0,
+ "7685": 1083622400.0,
+ "7690": 1083622400.0,
+ "7695": 1083622400.0,
+ "7700": 1083622400.0,
+ "7705": 1083622400.0,
+ "7710": 1083622400.0,
+ "7715": 1083622400.0,
+ "7720": 1083622400.0,
+ "7725": 1083622400.0,
+ "7730": 1083622400.0,
+ "7735": 1083622400.0,
+ "7740": 1083622400.0,
+ "7745": 1083622400.0,
+ "7750": 1083622400.0,
+ "7755": 1083622400.0,
+ "7760": 1083622400.0,
+ "7765": 1083622400.0,
+ "7770": 1083622400.0,
+ "7775": 1083622400.0,
+ "7780": 1083622400.0,
+ "7785": 1083622400.0,
+ "7790": 1083622400.0,
+ "7795": 1083622400.0,
+ "7800": 1083622400.0,
+ "7805": 1083622400.0,
+ "7810": 1083622400.0,
+ "7815": 1083622400.0,
+ "7820": 1083622400.0,
+ "7825": 1083622400.0,
+ "7830": 1083622400.0,
+ "7835": 1083622400.0,
+ "7840": 1083622400.0,
+ "7845": 1083622400.0,
+ "7850": 1083622400.0,
+ "7855": 1083622400.0,
+ "7860": 1083622400.0,
+ "7865": 1083622400.0,
+ "7870": 1083622400.0,
+ "7875": 1083622400.0,
+ "7880": 1083622400.0,
+ "7885": 1083622400.0,
+ "7890": 1083622400.0,
+ "7895": 1083622400.0,
+ "7900": 1083622400.0,
+ "7905": 1083622400.0,
+ "7910": 1083622400.0,
+ "7915": 1083622400.0,
+ "7920": 1083622400.0,
+ "7925": 1083622400.0,
+ "7930": 1083622400.0,
+ "7935": 1083622400.0,
+ "7940": 1083622400.0,
+ "7945": 1083622400.0,
+ "7950": 1083622400.0,
+ "7955": 1083622400.0,
+ "7960": 1083622400.0,
+ "7965": 1083622400.0,
+ "7970": 1083622400.0,
+ "7975": 1083622400.0,
+ "7980": 1083622400.0,
+ "7985": 1083622400.0,
+ "7990": 1083622400.0,
+ "7995": 1083622400.0,
+ "8000": 1083622400.0,
+ "8005": 1083622400.0,
+ "8010": 1083622400.0,
+ "8015": 1083622400.0,
+ "8020": 1083622400.0,
+ "8025": 1083622400.0,
+ "8030": 1083622400.0,
+ "8035": 1083622400.0,
+ "8040": 1083622400.0,
+ "8045": 1083622400.0,
+ "8050": 1083622400.0,
+ "8055": 1083622400.0,
+ "8060": 1083622400.0,
+ "8065": 1083622400.0,
+ "8070": 1083622400.0,
+ "8075": 1083622400.0,
+ "8080": 1083622400.0,
+ "8085": 1083622400.0,
+ "8090": 1083622400.0,
+ "8095": 1083622400.0,
+ "8100": 1083622400.0,
+ "8105": 1083622400.0,
+ "8110": 1083622400.0,
+ "8115": 1083622400.0,
+ "8120": 1083622400.0,
+ "8125": 1083622400.0,
+ "8130": 1083622400.0,
+ "8135": 1083622400.0,
+ "8140": 1083622400.0,
+ "8145": 1083622400.0,
+ "8150": 1083622400.0,
+ "8155": 1083622400.0,
+ "8160": 1083622400.0,
+ "8165": 1083622400.0,
+ "8170": 1083622400.0,
+ "8175": 1083622400.0,
+ "8180": 1083622400.0,
+ "8185": 1083622400.0,
+ "8190": 1083622400.0,
+ "8195": 1083622400.0,
+ "8200": 1083622400.0,
+ "8205": 1083622400.0,
+ "8210": 1083622400.0,
+ "8215": 1083622400.0,
+ "8220": 1083622400.0,
+ "8225": 1083622400.0,
+ "8230": 1083622400.0,
+ "8235": 1083622400.0,
+ "8240": 1083622400.0,
+ "8245": 1083622400.0,
+ "8250": 1083622400.0,
+ "8255": 1083622400.0,
+ "8260": 1083622400.0,
+ "8265": 1083622400.0,
+ "8270": 1083622400.0,
+ "8275": 1083622400.0,
+ "8280": 1083622400.0,
+ "8285": 1083622400.0,
+ "8290": 1083622400.0,
+ "8295": 1083622400.0,
+ "8300": 1083622400.0,
+ "8305": 1083622400.0,
+ "8310": 1083622400.0,
+ "8315": 1083622400.0,
+ "8320": 1083622400.0,
+ "8325": 1083622400.0,
+ "8330": 1083622400.0,
+ "8335": 1083622400.0,
+ "8340": 1083622400.0,
+ "8345": 1083622400.0,
+ "8350": 1083622400.0,
+ "8355": 1083622400.0,
+ "8360": 1083622400.0,
+ "8365": 1083622400.0,
+ "8370": 1083622400.0,
+ "8375": 1083622400.0,
+ "8380": 1083622400.0,
+ "8385": 1083622400.0,
+ "8390": 1083622400.0,
+ "8395": 1083622400.0,
+ "8400": 1083622400.0,
+ "8405": 1083622400.0,
+ "8410": 1083622400.0,
+ "8415": 1083622400.0,
+ "8420": 1083622400.0,
+ "8425": 1083622400.0,
+ "8430": 1083622400.0,
+ "8435": 1083622400.0,
+ "8440": 1083622400.0,
+ "8445": 1083622400.0,
+ "8450": 1083622400.0,
+ "8455": 1083622400.0,
+ "8460": 1083622400.0,
+ "8465": 1083622400.0,
+ "8470": 1083622400.0,
+ "8475": 1083622400.0,
+ "8480": 1083622400.0,
+ "8485": 1083622400.0,
+ "8490": 1083622400.0,
+ "8495": 1083622400.0,
+ "8500": 1083622400.0,
+ "8505": 1083622400.0,
+ "8510": 1083622400.0,
+ "8515": 1083622400.0,
+ "8520": 1083622400.0,
+ "8525": 1083622400.0,
+ "8530": 1083622400.0,
+ "8535": 1083622400.0,
+ "8540": 1083622400.0,
+ "8545": 1083622400.0,
+ "8550": 1083622400.0,
+ "8555": 1083622400.0,
+ "8560": 1083622400.0,
+ "8565": 1083622400.0,
+ "8570": 1083622400.0,
+ "8575": 1083622400.0,
+ "8580": 1083622400.0,
+ "8585": 1083622400.0,
+ "8590": 1083622400.0,
+ "8595": 1083622400.0,
+ "8600": 1083622400.0,
+ "8605": 1083622400.0,
+ "8610": 1083622400.0,
+ "8615": 1083622400.0,
+ "8620": 1083622400.0,
+ "8625": 1083622400.0,
+ "8630": 1083622400.0,
+ "8635": 1083622400.0,
+ "8640": 1083622400.0,
+ "8645": 1083622400.0,
+ "8650": 1083622400.0,
+ "8655": 1083622400.0,
+ "8660": 1083622400.0,
+ "8665": 1083622400.0,
+ "8670": 1083622400.0,
+ "8675": 1083622400.0,
+ "8680": 1083622400.0,
+ "8685": 1083622400.0,
+ "8690": 1083622400.0,
+ "8695": 1083622400.0,
+ "8700": 1083622400.0,
+ "8705": 1083622400.0,
+ "8710": 1083622400.0,
+ "8715": 1083622400.0,
+ "8720": 1083622400.0,
+ "8725": 1083622400.0,
+ "8730": 1083622400.0,
+ "8735": 1083622400.0,
+ "8740": 1083622400.0,
+ "8745": 1083622400.0,
+ "8750": 1083622400.0,
+ "8755": 1083622400.0,
+ "8760": 1083622400.0,
+ "8765": 1083622400.0,
+ "8770": 1083622400.0,
+ "8775": 1083622400.0,
+ "8780": 1083622400.0,
+ "8785": 1083622400.0,
+ "8790": 1083622400.0,
+ "8795": 1083622400.0,
+ "8800": 1083622400.0,
+ "8805": 1083622400.0,
+ "8810": 1083622400.0,
+ "8815": 1083622400.0,
+ "8820": 1083622400.0,
+ "8825": 1083622400.0,
+ "8830": 1083622400.0,
+ "8835": 1083622400.0,
+ "8840": 1083622400.0,
+ "8845": 1083622400.0,
+ "8850": 1083622400.0,
+ "8855": 1083622400.0,
+ "8860": 1083622400.0,
+ "8865": 1083622400.0,
+ "8870": 1083622400.0,
+ "8875": 1083622400.0,
+ "8880": 1083622400.0,
+ "8885": 1083622400.0,
+ "8890": 1083622400.0,
+ "8895": 1083622400.0,
+ "8900": 1083622400.0,
+ "8905": 1083622400.0,
+ "8910": 1083622400.0,
+ "8915": 1083622400.0,
+ "8920": 1083622400.0,
+ "8925": 1083622400.0,
+ "8930": 1083622400.0,
+ "8935": 1083622400.0,
+ "8940": 1083622400.0,
+ "8945": 1083622400.0,
+ "8950": 1083622400.0,
+ "8955": 1083622400.0,
+ "8960": 1083622400.0,
+ "8965": 1083622400.0,
+ "8970": 1083622400.0,
+ "8975": 1083622400.0,
+ "8980": 1083622400.0,
+ "8985": 1083622400.0,
+ "8990": 1083622400.0,
+ "8995": 1083622400.0,
+ "9000": 1083622400.0,
+ "9005": 1083622400.0,
+ "9010": 1083622400.0,
+ "9015": 1083622400.0,
+ "9020": 1083622400.0,
+ "9025": 1083622400.0,
+ "9030": 1083622400.0,
+ "9035": 1083622400.0,
+ "9040": 1083622400.0,
+ "9045": 1083622400.0,
+ "9050": 1083622400.0,
+ "9055": 1083622400.0,
+ "9060": 1083622400.0,
+ "9065": 1083622400.0,
+ "9070": 1083622400.0,
+ "9075": 1083622400.0,
+ "9080": 1083622400.0,
+ "9085": 1083622400.0,
+ "9090": 1083622400.0,
+ "9095": 1083622400.0,
+ "9100": 1083622400.0,
+ "9105": 1083622400.0,
+ "9110": 1083622400.0,
+ "9115": 1083622400.0,
+ "9120": 1083622400.0,
+ "9125": 1083622400.0,
+ "9130": 1083622400.0,
+ "9135": 1083622400.0,
+ "9140": 1083622400.0,
+ "9145": 1083622400.0,
+ "9150": 1083622400.0,
+ "9155": 1083622400.0,
+ "9160": 1083622400.0,
+ "9165": 1083622400.0,
+ "9170": 1083622400.0,
+ "9175": 1083622400.0,
+ "9180": 1083622400.0,
+ "9185": 1083622400.0,
+ "9190": 1083622400.0,
+ "9195": 1083622400.0,
+ "9200": 1083622400.0,
+ "9205": 1083622400.0,
+ "9210": 1083622400.0,
+ "9215": 1083622400.0,
+ "9220": 1083622400.0,
+ "9225": 1083622400.0,
+ "9230": 1083622400.0,
+ "9235": 1083622400.0,
+ "9240": 1083622400.0,
+ "9245": 1083622400.0,
+ "9250": 1083622400.0,
+ "9255": 1083622400.0,
+ "9260": 1083622400.0,
+ "9265": 1083622400.0,
+ "9270": 1083622400.0,
+ "9275": 1083622400.0,
+ "9280": 1083622400.0,
+ "9285": 1083622400.0,
+ "9290": 1083622400.0,
+ "9295": 1083622400.0,
+ "9300": 1083622400.0,
+ "9305": 1083622400.0,
+ "9310": 1083622400.0,
+ "9315": 1083622400.0,
+ "9320": 1083622400.0,
+ "9325": 1083622400.0,
+ "9330": 1083622400.0,
+ "9335": 1083622400.0,
+ "9340": 1083622400.0,
+ "9345": 1083622400.0,
+ "9350": 1083622400.0,
+ "9355": 1083622400.0,
+ "9360": 1083622400.0,
+ "9365": 1083622400.0,
+ "9370": 1083622400.0,
+ "9375": 1083622400.0,
+ "9380": 1083622400.0,
+ "9385": 1083622400.0,
+ "9390": 1083622400.0,
+ "9395": 1083622400.0,
+ "9400": 1083622400.0,
+ "9405": 1083622400.0,
+ "9410": 1083622400.0,
+ "9415": 1083622400.0,
+ "9420": 1083622400.0,
+ "9425": 1083622400.0,
+ "9430": 1083622400.0,
+ "9435": 1083622400.0,
+ "9440": 1083622400.0,
+ "9445": 1083622400.0,
+ "9450": 1083622400.0,
+ "9455": 1083622400.0,
+ "9460": 1083622400.0,
+ "9465": 1083622400.0,
+ "9470": 1083622400.0,
+ "9475": 1083622400.0,
+ "9480": 1083622400.0,
+ "9485": 1083622400.0,
+ "9490": 1083622400.0,
+ "9495": 1083622400.0,
+ "9500": 1083622400.0,
+ "9505": 1083622400.0,
+ "9510": 1083622400.0,
+ "9515": 1083622400.0,
+ "9520": 1083622400.0,
+ "9525": 1083622400.0,
+ "9530": 1083622400.0,
+ "9535": 1083622400.0,
+ "9540": 1083622400.0,
+ "9545": 1083622400.0,
+ "9550": 1083622400.0,
+ "9555": 1083622400.0,
+ "9560": 1083622400.0,
+ "9565": 1083622400.0,
+ "9570": 1083622400.0,
+ "9575": 1083622400.0,
+ "9580": 1083622400.0,
+ "9585": 1083622400.0,
+ "9590": 1083622400.0,
+ "9595": 1083622400.0,
+ "9600": 1083622400.0,
+ "9605": 1083622400.0,
+ "9610": 1083622400.0,
+ "9615": 1083622400.0,
+ "9620": 1083622400.0,
+ "9625": 1083622400.0,
+ "9630": 1083622400.0,
+ "9635": 1083622400.0,
+ "9640": 1083622400.0,
+ "9645": 1083622400.0,
+ "9650": 1083622400.0,
+ "9655": 1083622400.0,
+ "9660": 1083622400.0,
+ "9665": 1083622400.0,
+ "9670": 1083622400.0,
+ "9675": 1083622400.0,
+ "9680": 1083622400.0,
+ "9685": 1083622400.0,
+ "9690": 1083622400.0,
+ "9695": 1083622400.0,
+ "9700": 1083622400.0,
+ "9705": 1083622400.0,
+ "9710": 1083622400.0,
+ "9715": 1083622400.0,
+ "9720": 1083622400.0,
+ "9725": 1083622400.0,
+ "9730": 1083622400.0,
+ "9735": 1083622400.0,
+ "9740": 1083622400.0,
+ "9745": 1083622400.0,
+ "9750": 1083622400.0,
+ "9755": 1083622400.0,
+ "9760": 1083622400.0,
+ "9765": 1083622400.0,
+ "9770": 1083622400.0,
+ "9775": 1083622400.0,
+ "9780": 1083622400.0,
+ "9785": 1083622400.0,
+ "9790": 1083622400.0,
+ "9795": 1083622400.0,
+ "9800": 1083622400.0,
+ "9805": 1083622400.0,
+ "9810": 1083622400.0,
+ "9815": 1083622400.0,
+ "9820": 1083622400.0,
+ "9825": 1083622400.0,
+ "9830": 1083622400.0,
+ "9835": 1083622400.0,
+ "9840": 1083622400.0,
+ "9845": 1083622400.0,
+ "9850": 1083622400.0,
+ "9855": 1083622400.0,
+ "9860": 1083622400.0,
+ "9865": 1083622400.0,
+ "9870": 1083622400.0,
+ "9875": 1083622400.0,
+ "9880": 1083622400.0,
+ "9885": 1083622400.0,
+ "9890": 1083622400.0,
+ "9895": 1083622400.0,
+ "9900": 1083622400.0,
+ "9905": 1083622400.0,
+ "9910": 1083622400.0,
+ "9915": 1083622400.0,
+ "9920": 1083622400.0,
+ "9925": 1083622400.0,
+ "9930": 1083622400.0,
+ "9935": 1083622400.0,
+ "9940": 1083622400.0,
+ "9945": 1083622400.0,
+ "9950": 1083622400.0,
+ "9955": 1083622400.0,
+ "9960": 1083622400.0,
+ "9965": 1083622400.0,
+ "9970": 1083622400.0,
+ "9975": 1083622400.0,
+ "9980": 1083622400.0,
+ "9985": 1083622400.0,
+ "9990": 1083622400.0,
+ "9995": 1083622400.0,
+ "10000": 1083622400.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 10000,
+ "step_interval": 5,
+ "values": {
+ "1": 6515439616.0,
+ "5": 6956085248.0,
+ "10": 6956085248.0,
+ "15": 6956085248.0,
+ "20": 6956085248.0,
+ "25": 6956085248.0,
+ "30": 6956085248.0,
+ "35": 6956085248.0,
+ "40": 6956085248.0,
+ "45": 6956085248.0,
+ "50": 6956085248.0,
+ "55": 6956085248.0,
+ "60": 6956085248.0,
+ "65": 6956085248.0,
+ "70": 6956085248.0,
+ "75": 6956085248.0,
+ "80": 6956085248.0,
+ "85": 6956085248.0,
+ "90": 6956085248.0,
+ "95": 6956085248.0,
+ "100": 6956085248.0,
+ "105": 6956085248.0,
+ "110": 6956085248.0,
+ "115": 6956085248.0,
+ "120": 6956085248.0,
+ "125": 6956085248.0,
+ "130": 6956085248.0,
+ "135": 6956085248.0,
+ "140": 6956085248.0,
+ "145": 6956085248.0,
+ "150": 6956085248.0,
+ "155": 6956085248.0,
+ "160": 6956085248.0,
+ "165": 6956085248.0,
+ "170": 6956085248.0,
+ "175": 6956085248.0,
+ "180": 6956085248.0,
+ "185": 6956085248.0,
+ "190": 6956085248.0,
+ "195": 6956085248.0,
+ "200": 6956085248.0,
+ "205": 6956085248.0,
+ "210": 6956085248.0,
+ "215": 6956085248.0,
+ "220": 6956085248.0,
+ "225": 6956085248.0,
+ "230": 6956085248.0,
+ "235": 6956085248.0,
+ "240": 6956085248.0,
+ "245": 6956085248.0,
+ "250": 6956085248.0,
+ "255": 6956085248.0,
+ "260": 6956085248.0,
+ "265": 6956085248.0,
+ "270": 6956085248.0,
+ "275": 6956085248.0,
+ "280": 6956085248.0,
+ "285": 6956085248.0,
+ "290": 6956085248.0,
+ "295": 6956085248.0,
+ "300": 6956085248.0,
+ "305": 6956085248.0,
+ "310": 6956085248.0,
+ "315": 6956085248.0,
+ "320": 6956085248.0,
+ "325": 6956085248.0,
+ "330": 6956085248.0,
+ "335": 6956085248.0,
+ "340": 6956085248.0,
+ "345": 6956085248.0,
+ "350": 6956085248.0,
+ "355": 6956085248.0,
+ "360": 6956085248.0,
+ "365": 6956085248.0,
+ "370": 6956085248.0,
+ "375": 6956085248.0,
+ "380": 6956085248.0,
+ "385": 6956085248.0,
+ "390": 6956085248.0,
+ "395": 6956085248.0,
+ "400": 6956085248.0,
+ "405": 6956085248.0,
+ "410": 6956085248.0,
+ "415": 6956085248.0,
+ "420": 6956085248.0,
+ "425": 6956085248.0,
+ "430": 6956085248.0,
+ "435": 6956085248.0,
+ "440": 6956085248.0,
+ "445": 6956085248.0,
+ "450": 6956085248.0,
+ "455": 6956085248.0,
+ "460": 6956085248.0,
+ "465": 6956085248.0,
+ "470": 6956085248.0,
+ "475": 6956085248.0,
+ "480": 6956085248.0,
+ "485": 6956085248.0,
+ "490": 6956085248.0,
+ "495": 6956085248.0,
+ "500": 6956085248.0,
+ "505": 6956085248.0,
+ "510": 6956085248.0,
+ "515": 6956085248.0,
+ "520": 6956085248.0,
+ "525": 6956085248.0,
+ "530": 6956085248.0,
+ "535": 6956085248.0,
+ "540": 6956085248.0,
+ "545": 6956085248.0,
+ "550": 6956085248.0,
+ "555": 6956085248.0,
+ "560": 6956085248.0,
+ "565": 6956085248.0,
+ "570": 6956085248.0,
+ "575": 6956085248.0,
+ "580": 6956085248.0,
+ "585": 6956085248.0,
+ "590": 6956085248.0,
+ "595": 6956085248.0,
+ "600": 6956085248.0,
+ "605": 6956085248.0,
+ "610": 6956085248.0,
+ "615": 6956085248.0,
+ "620": 6956085248.0,
+ "625": 6956085248.0,
+ "630": 6956085248.0,
+ "635": 6956085248.0,
+ "640": 6956085248.0,
+ "645": 6956085248.0,
+ "650": 6956085248.0,
+ "655": 6956085248.0,
+ "660": 6956085248.0,
+ "665": 6956085248.0,
+ "670": 6956085248.0,
+ "675": 6956085248.0,
+ "680": 6956085248.0,
+ "685": 6956085248.0,
+ "690": 6956085248.0,
+ "695": 6956085248.0,
+ "700": 6956085248.0,
+ "705": 6956085248.0,
+ "710": 6956085248.0,
+ "715": 6956085248.0,
+ "720": 6956085248.0,
+ "725": 6956085248.0,
+ "730": 6956085248.0,
+ "735": 6956085248.0,
+ "740": 6956085248.0,
+ "745": 6956085248.0,
+ "750": 6956085248.0,
+ "755": 6956085248.0,
+ "760": 6956085248.0,
+ "765": 6956085248.0,
+ "770": 6956085248.0,
+ "775": 6956085248.0,
+ "780": 6956085248.0,
+ "785": 6956085248.0,
+ "790": 6956085248.0,
+ "795": 6956085248.0,
+ "800": 6956085248.0,
+ "805": 6956085248.0,
+ "810": 6956085248.0,
+ "815": 6956085248.0,
+ "820": 6956085248.0,
+ "825": 6956085248.0,
+ "830": 6956085248.0,
+ "835": 6956085248.0,
+ "840": 6956085248.0,
+ "845": 6956085248.0,
+ "850": 6956085248.0,
+ "855": 6956085248.0,
+ "860": 6956085248.0,
+ "865": 6956085248.0,
+ "870": 6956085248.0,
+ "875": 6956085248.0,
+ "880": 6956085248.0,
+ "885": 6956085248.0,
+ "890": 6956085248.0,
+ "895": 6956085248.0,
+ "900": 6956085248.0,
+ "905": 6956085248.0,
+ "910": 6956085248.0,
+ "915": 6956085248.0,
+ "920": 6956085248.0,
+ "925": 6956085248.0,
+ "930": 6956085248.0,
+ "935": 6956085248.0,
+ "940": 6956085248.0,
+ "945": 6956085248.0,
+ "950": 6956085248.0,
+ "955": 6956085248.0,
+ "960": 6956085248.0,
+ "965": 6956085248.0,
+ "970": 6956085248.0,
+ "975": 6956085248.0,
+ "980": 6956085248.0,
+ "985": 6956085248.0,
+ "990": 6956085248.0,
+ "995": 6956085248.0,
+ "1000": 6956085248.0,
+ "1005": 6956085248.0,
+ "1010": 6956085248.0,
+ "1015": 6956085248.0,
+ "1020": 6956085248.0,
+ "1025": 6956085248.0,
+ "1030": 6956085248.0,
+ "1035": 6956085248.0,
+ "1040": 6956085248.0,
+ "1045": 6956085248.0,
+ "1050": 6956085248.0,
+ "1055": 6956085248.0,
+ "1060": 6956085248.0,
+ "1065": 6956085248.0,
+ "1070": 6956085248.0,
+ "1075": 6956085248.0,
+ "1080": 6956085248.0,
+ "1085": 6956085248.0,
+ "1090": 6956085248.0,
+ "1095": 6956085248.0,
+ "1100": 6956085248.0,
+ "1105": 6956085248.0,
+ "1110": 6956085248.0,
+ "1115": 6956085248.0,
+ "1120": 6956085248.0,
+ "1125": 6956085248.0,
+ "1130": 6956085248.0,
+ "1135": 6956085248.0,
+ "1140": 6956085248.0,
+ "1145": 6956085248.0,
+ "1150": 6956085248.0,
+ "1155": 6956085248.0,
+ "1160": 6956085248.0,
+ "1165": 6956085248.0,
+ "1170": 6956085248.0,
+ "1175": 6956085248.0,
+ "1180": 6956085248.0,
+ "1185": 6956085248.0,
+ "1190": 6956085248.0,
+ "1195": 6956085248.0,
+ "1200": 6956085248.0,
+ "1205": 6956085248.0,
+ "1210": 6956085248.0,
+ "1215": 6956085248.0,
+ "1220": 6956085248.0,
+ "1225": 6956085248.0,
+ "1230": 6956085248.0,
+ "1235": 6956085248.0,
+ "1240": 6956085248.0,
+ "1245": 6956085248.0,
+ "1250": 6956085248.0,
+ "1255": 6956085248.0,
+ "1260": 6956085248.0,
+ "1265": 6956085248.0,
+ "1270": 6956085248.0,
+ "1275": 6956085248.0,
+ "1280": 6956085248.0,
+ "1285": 6956085248.0,
+ "1290": 6956085248.0,
+ "1295": 6956085248.0,
+ "1300": 6956085248.0,
+ "1305": 6956085248.0,
+ "1310": 6956085248.0,
+ "1315": 6956085248.0,
+ "1320": 6956085248.0,
+ "1325": 6956085248.0,
+ "1330": 6956085248.0,
+ "1335": 6956085248.0,
+ "1340": 6956085248.0,
+ "1345": 6956085248.0,
+ "1350": 6956085248.0,
+ "1355": 6956085248.0,
+ "1360": 6956085248.0,
+ "1365": 6956085248.0,
+ "1370": 6956085248.0,
+ "1375": 6956085248.0,
+ "1380": 6956085248.0,
+ "1385": 6956085248.0,
+ "1390": 6956085248.0,
+ "1395": 6956085248.0,
+ "1400": 6956085248.0,
+ "1405": 6956085248.0,
+ "1410": 6956085248.0,
+ "1415": 6956085248.0,
+ "1420": 6956085248.0,
+ "1425": 6956085248.0,
+ "1430": 6956085248.0,
+ "1435": 6956085248.0,
+ "1440": 6956085248.0,
+ "1445": 6956085248.0,
+ "1450": 6956085248.0,
+ "1455": 6956085248.0,
+ "1460": 6956085248.0,
+ "1465": 6956085248.0,
+ "1470": 6956085248.0,
+ "1475": 6956085248.0,
+ "1480": 6956085248.0,
+ "1485": 6956085248.0,
+ "1490": 6956085248.0,
+ "1495": 6956085248.0,
+ "1500": 6956085248.0,
+ "1505": 6956085248.0,
+ "1510": 6956085248.0,
+ "1515": 6956085248.0,
+ "1520": 6956085248.0,
+ "1525": 6956085248.0,
+ "1530": 6956085248.0,
+ "1535": 6956085248.0,
+ "1540": 6956085248.0,
+ "1545": 6956085248.0,
+ "1550": 6956085248.0,
+ "1555": 6956085248.0,
+ "1560": 6956085248.0,
+ "1565": 6956085248.0,
+ "1570": 6956085248.0,
+ "1575": 6956085248.0,
+ "1580": 6956085248.0,
+ "1585": 6956085248.0,
+ "1590": 6956085248.0,
+ "1595": 6956085248.0,
+ "1600": 6956085248.0,
+ "1605": 6956085248.0,
+ "1610": 6956085248.0,
+ "1615": 6956085248.0,
+ "1620": 6956085248.0,
+ "1625": 6956085248.0,
+ "1630": 6956085248.0,
+ "1635": 6956085248.0,
+ "1640": 6956085248.0,
+ "1645": 6956085248.0,
+ "1650": 6956085248.0,
+ "1655": 6956085248.0,
+ "1660": 6956085248.0,
+ "1665": 6956085248.0,
+ "1670": 6956085248.0,
+ "1675": 6956085248.0,
+ "1680": 6956085248.0,
+ "1685": 6956085248.0,
+ "1690": 6956085248.0,
+ "1695": 6956085248.0,
+ "1700": 6956085248.0,
+ "1705": 6956085248.0,
+ "1710": 6956085248.0,
+ "1715": 6956085248.0,
+ "1720": 6956085248.0,
+ "1725": 6956085248.0,
+ "1730": 6956085248.0,
+ "1735": 6956085248.0,
+ "1740": 6956085248.0,
+ "1745": 6956085248.0,
+ "1750": 6956085248.0,
+ "1755": 6956085248.0,
+ "1760": 6956085248.0,
+ "1765": 6956085248.0,
+ "1770": 6956085248.0,
+ "1775": 6956085248.0,
+ "1780": 6956085248.0,
+ "1785": 6956085248.0,
+ "1790": 6956085248.0,
+ "1795": 6956085248.0,
+ "1800": 6956085248.0,
+ "1805": 6956085248.0,
+ "1810": 6956085248.0,
+ "1815": 6956085248.0,
+ "1820": 6956085248.0,
+ "1825": 6956085248.0,
+ "1830": 6956085248.0,
+ "1835": 6956085248.0,
+ "1840": 6956085248.0,
+ "1845": 6956085248.0,
+ "1850": 6956085248.0,
+ "1855": 6956085248.0,
+ "1860": 6956085248.0,
+ "1865": 6956085248.0,
+ "1870": 6956085248.0,
+ "1875": 6956085248.0,
+ "1880": 6956085248.0,
+ "1885": 6956085248.0,
+ "1890": 6956085248.0,
+ "1895": 6956085248.0,
+ "1900": 6956085248.0,
+ "1905": 6956085248.0,
+ "1910": 6956085248.0,
+ "1915": 6956085248.0,
+ "1920": 6956085248.0,
+ "1925": 6956085248.0,
+ "1930": 6956085248.0,
+ "1935": 6956085248.0,
+ "1940": 6956085248.0,
+ "1945": 6956085248.0,
+ "1950": 6956085248.0,
+ "1955": 6956085248.0,
+ "1960": 6956085248.0,
+ "1965": 6956085248.0,
+ "1970": 6956085248.0,
+ "1975": 6956085248.0,
+ "1980": 6956085248.0,
+ "1985": 6956085248.0,
+ "1990": 6956085248.0,
+ "1995": 6956085248.0,
+ "2000": 6956085248.0,
+ "2005": 6976532480.0,
+ "2010": 6976532480.0,
+ "2015": 6976532480.0,
+ "2020": 6976532480.0,
+ "2025": 6976532480.0,
+ "2030": 6976532480.0,
+ "2035": 6976532480.0,
+ "2040": 6976532480.0,
+ "2045": 6976532480.0,
+ "2050": 6976532480.0,
+ "2055": 6976532480.0,
+ "2060": 6976532480.0,
+ "2065": 6976532480.0,
+ "2070": 6976532480.0,
+ "2075": 6976532480.0,
+ "2080": 6976532480.0,
+ "2085": 6976532480.0,
+ "2090": 6976532480.0,
+ "2095": 6976532480.0,
+ "2100": 6976532480.0,
+ "2105": 6976532480.0,
+ "2110": 6976532480.0,
+ "2115": 6976532480.0,
+ "2120": 6976532480.0,
+ "2125": 6976532480.0,
+ "2130": 6976532480.0,
+ "2135": 6976532480.0,
+ "2140": 6976532480.0,
+ "2145": 6976532480.0,
+ "2150": 6976532480.0,
+ "2155": 6976532480.0,
+ "2160": 6976532480.0,
+ "2165": 6976532480.0,
+ "2170": 6976532480.0,
+ "2175": 6976532480.0,
+ "2180": 6976532480.0,
+ "2185": 6976532480.0,
+ "2190": 6976532480.0,
+ "2195": 6976532480.0,
+ "2200": 6976532480.0,
+ "2205": 6976532480.0,
+ "2210": 6976532480.0,
+ "2215": 6976532480.0,
+ "2220": 6976532480.0,
+ "2225": 6976532480.0,
+ "2230": 6976532480.0,
+ "2235": 6976532480.0,
+ "2240": 6976532480.0,
+ "2245": 6976532480.0,
+ "2250": 6976532480.0,
+ "2255": 6976532480.0,
+ "2260": 6976532480.0,
+ "2265": 6976532480.0,
+ "2270": 6976532480.0,
+ "2275": 6976532480.0,
+ "2280": 6976532480.0,
+ "2285": 6976532480.0,
+ "2290": 6976532480.0,
+ "2295": 6976532480.0,
+ "2300": 6976532480.0,
+ "2305": 6976532480.0,
+ "2310": 6976532480.0,
+ "2315": 6976532480.0,
+ "2320": 6976532480.0,
+ "2325": 6976532480.0,
+ "2330": 6976532480.0,
+ "2335": 6976532480.0,
+ "2340": 6976532480.0,
+ "2345": 6976532480.0,
+ "2350": 6976532480.0,
+ "2355": 6976532480.0,
+ "2360": 6976532480.0,
+ "2365": 6976532480.0,
+ "2370": 6976532480.0,
+ "2375": 6976532480.0,
+ "2380": 6976532480.0,
+ "2385": 6976532480.0,
+ "2390": 6976532480.0,
+ "2395": 6976532480.0,
+ "2400": 6976532480.0,
+ "2405": 6976532480.0,
+ "2410": 6976532480.0,
+ "2415": 6976532480.0,
+ "2420": 6976532480.0,
+ "2425": 6976532480.0,
+ "2430": 6976532480.0,
+ "2435": 6976532480.0,
+ "2440": 6976532480.0,
+ "2445": 6976532480.0,
+ "2450": 6976532480.0,
+ "2455": 6976532480.0,
+ "2460": 6976532480.0,
+ "2465": 6976532480.0,
+ "2470": 6976532480.0,
+ "2475": 6976532480.0,
+ "2480": 6976532480.0,
+ "2485": 6976532480.0,
+ "2490": 6976532480.0,
+ "2495": 6976532480.0,
+ "2500": 6976532480.0,
+ "2505": 6976532480.0,
+ "2510": 6976532480.0,
+ "2515": 6976532480.0,
+ "2520": 6976532480.0,
+ "2525": 6976532480.0,
+ "2530": 6976532480.0,
+ "2535": 6976532480.0,
+ "2540": 6976532480.0,
+ "2545": 6976532480.0,
+ "2550": 6976532480.0,
+ "2555": 6976532480.0,
+ "2560": 6976532480.0,
+ "2565": 6976532480.0,
+ "2570": 6976532480.0,
+ "2575": 6976532480.0,
+ "2580": 6976532480.0,
+ "2585": 6976532480.0,
+ "2590": 6976532480.0,
+ "2595": 6976532480.0,
+ "2600": 6976532480.0,
+ "2605": 6976532480.0,
+ "2610": 6976532480.0,
+ "2615": 6976532480.0,
+ "2620": 6976532480.0,
+ "2625": 6976532480.0,
+ "2630": 6976532480.0,
+ "2635": 6976532480.0,
+ "2640": 6976532480.0,
+ "2645": 6976532480.0,
+ "2650": 6976532480.0,
+ "2655": 6976532480.0,
+ "2660": 6976532480.0,
+ "2665": 6976532480.0,
+ "2670": 6976532480.0,
+ "2675": 6976532480.0,
+ "2680": 6976532480.0,
+ "2685": 6976532480.0,
+ "2690": 6976532480.0,
+ "2695": 6976532480.0,
+ "2700": 6976532480.0,
+ "2705": 6976532480.0,
+ "2710": 6976532480.0,
+ "2715": 6976532480.0,
+ "2720": 6976532480.0,
+ "2725": 6976532480.0,
+ "2730": 6976532480.0,
+ "2735": 6976532480.0,
+ "2740": 6976532480.0,
+ "2745": 6976532480.0,
+ "2750": 6976532480.0,
+ "2755": 6976532480.0,
+ "2760": 6976532480.0,
+ "2765": 6976532480.0,
+ "2770": 6976532480.0,
+ "2775": 6976532480.0,
+ "2780": 6976532480.0,
+ "2785": 6976532480.0,
+ "2790": 6976532480.0,
+ "2795": 6976532480.0,
+ "2800": 6976532480.0,
+ "2805": 6976532480.0,
+ "2810": 6976532480.0,
+ "2815": 6976532480.0,
+ "2820": 6976532480.0,
+ "2825": 6976532480.0,
+ "2830": 6976532480.0,
+ "2835": 6976532480.0,
+ "2840": 6976532480.0,
+ "2845": 6976532480.0,
+ "2850": 6976532480.0,
+ "2855": 6976532480.0,
+ "2860": 6976532480.0,
+ "2865": 6976532480.0,
+ "2870": 6976532480.0,
+ "2875": 6976532480.0,
+ "2880": 6976532480.0,
+ "2885": 6976532480.0,
+ "2890": 6976532480.0,
+ "2895": 6976532480.0,
+ "2900": 6976532480.0,
+ "2905": 6976532480.0,
+ "2910": 6976532480.0,
+ "2915": 6976532480.0,
+ "2920": 6976532480.0,
+ "2925": 6976532480.0,
+ "2930": 6976532480.0,
+ "2935": 6976532480.0,
+ "2940": 6976532480.0,
+ "2945": 6976532480.0,
+ "2950": 6976532480.0,
+ "2955": 6976532480.0,
+ "2960": 6976532480.0,
+ "2965": 6976532480.0,
+ "2970": 6976532480.0,
+ "2975": 6976532480.0,
+ "2980": 6976532480.0,
+ "2985": 6976532480.0,
+ "2990": 6976532480.0,
+ "2995": 6976532480.0,
+ "3000": 6976532480.0,
+ "3005": 6976532480.0,
+ "3010": 6976532480.0,
+ "3015": 6976532480.0,
+ "3020": 6976532480.0,
+ "3025": 6976532480.0,
+ "3030": 6976532480.0,
+ "3035": 6976532480.0,
+ "3040": 6976532480.0,
+ "3045": 6976532480.0,
+ "3050": 6976532480.0,
+ "3055": 6976532480.0,
+ "3060": 6976532480.0,
+ "3065": 6976532480.0,
+ "3070": 6976532480.0,
+ "3075": 6976532480.0,
+ "3080": 6976532480.0,
+ "3085": 6976532480.0,
+ "3090": 6976532480.0,
+ "3095": 6976532480.0,
+ "3100": 6976532480.0,
+ "3105": 6976532480.0,
+ "3110": 6976532480.0,
+ "3115": 6976532480.0,
+ "3120": 6976532480.0,
+ "3125": 6976532480.0,
+ "3130": 6976532480.0,
+ "3135": 6976532480.0,
+ "3140": 6976532480.0,
+ "3145": 6976532480.0,
+ "3150": 6976532480.0,
+ "3155": 6976532480.0,
+ "3160": 6976532480.0,
+ "3165": 6976532480.0,
+ "3170": 6976532480.0,
+ "3175": 6976532480.0,
+ "3180": 6976532480.0,
+ "3185": 6976532480.0,
+ "3190": 6976532480.0,
+ "3195": 6976532480.0,
+ "3200": 6976532480.0,
+ "3205": 6976532480.0,
+ "3210": 6976532480.0,
+ "3215": 6976532480.0,
+ "3220": 6976532480.0,
+ "3225": 6976532480.0,
+ "3230": 6976532480.0,
+ "3235": 6976532480.0,
+ "3240": 6976532480.0,
+ "3245": 6976532480.0,
+ "3250": 6976532480.0,
+ "3255": 6976532480.0,
+ "3260": 6976532480.0,
+ "3265": 6976532480.0,
+ "3270": 6976532480.0,
+ "3275": 6976532480.0,
+ "3280": 6976532480.0,
+ "3285": 6976532480.0,
+ "3290": 6976532480.0,
+ "3295": 6976532480.0,
+ "3300": 6976532480.0,
+ "3305": 6976532480.0,
+ "3310": 6976532480.0,
+ "3315": 6976532480.0,
+ "3320": 6976532480.0,
+ "3325": 6976532480.0,
+ "3330": 6976532480.0,
+ "3335": 6976532480.0,
+ "3340": 6976532480.0,
+ "3345": 6976532480.0,
+ "3350": 6976532480.0,
+ "3355": 6976532480.0,
+ "3360": 6976532480.0,
+ "3365": 6976532480.0,
+ "3370": 6976532480.0,
+ "3375": 6976532480.0,
+ "3380": 6976532480.0,
+ "3385": 6976532480.0,
+ "3390": 6976532480.0,
+ "3395": 6976532480.0,
+ "3400": 6976532480.0,
+ "3405": 6976532480.0,
+ "3410": 6976532480.0,
+ "3415": 6976532480.0,
+ "3420": 6976532480.0,
+ "3425": 6976532480.0,
+ "3430": 6976532480.0,
+ "3435": 6976532480.0,
+ "3440": 6976532480.0,
+ "3445": 6976532480.0,
+ "3450": 6976532480.0,
+ "3455": 6976532480.0,
+ "3460": 6976532480.0,
+ "3465": 6976532480.0,
+ "3470": 6976532480.0,
+ "3475": 6976532480.0,
+ "3480": 6976532480.0,
+ "3485": 6976532480.0,
+ "3490": 6976532480.0,
+ "3495": 6976532480.0,
+ "3500": 6976532480.0,
+ "3505": 6976532480.0,
+ "3510": 6976532480.0,
+ "3515": 6976532480.0,
+ "3520": 6976532480.0,
+ "3525": 6976532480.0,
+ "3530": 6976532480.0,
+ "3535": 6976532480.0,
+ "3540": 6976532480.0,
+ "3545": 6976532480.0,
+ "3550": 6976532480.0,
+ "3555": 6976532480.0,
+ "3560": 6976532480.0,
+ "3565": 6976532480.0,
+ "3570": 6976532480.0,
+ "3575": 6976532480.0,
+ "3580": 6976532480.0,
+ "3585": 6976532480.0,
+ "3590": 6976532480.0,
+ "3595": 6976532480.0,
+ "3600": 6976532480.0,
+ "3605": 6976532480.0,
+ "3610": 6976532480.0,
+ "3615": 6976532480.0,
+ "3620": 6976532480.0,
+ "3625": 6976532480.0,
+ "3630": 6976532480.0,
+ "3635": 6976532480.0,
+ "3640": 6976532480.0,
+ "3645": 6976532480.0,
+ "3650": 6976532480.0,
+ "3655": 6976532480.0,
+ "3660": 6976532480.0,
+ "3665": 6976532480.0,
+ "3670": 6976532480.0,
+ "3675": 6976532480.0,
+ "3680": 6976532480.0,
+ "3685": 6976532480.0,
+ "3690": 6976532480.0,
+ "3695": 6976532480.0,
+ "3700": 6976532480.0,
+ "3705": 6976532480.0,
+ "3710": 6976532480.0,
+ "3715": 6976532480.0,
+ "3720": 6976532480.0,
+ "3725": 6976532480.0,
+ "3730": 6976532480.0,
+ "3735": 6976532480.0,
+ "3740": 6976532480.0,
+ "3745": 6976532480.0,
+ "3750": 6976532480.0,
+ "3755": 6976532480.0,
+ "3760": 6976532480.0,
+ "3765": 6976532480.0,
+ "3770": 6976532480.0,
+ "3775": 6976532480.0,
+ "3780": 6976532480.0,
+ "3785": 6976532480.0,
+ "3790": 6976532480.0,
+ "3795": 6976532480.0,
+ "3800": 6976532480.0,
+ "3805": 6976532480.0,
+ "3810": 6976532480.0,
+ "3815": 6976532480.0,
+ "3820": 6976532480.0,
+ "3825": 6976532480.0,
+ "3830": 6976532480.0,
+ "3835": 6976532480.0,
+ "3840": 6976532480.0,
+ "3845": 6976532480.0,
+ "3850": 6976532480.0,
+ "3855": 6976532480.0,
+ "3860": 6976532480.0,
+ "3865": 6976532480.0,
+ "3870": 6976532480.0,
+ "3875": 6976532480.0,
+ "3880": 6976532480.0,
+ "3885": 6976532480.0,
+ "3890": 6976532480.0,
+ "3895": 6976532480.0,
+ "3900": 6976532480.0,
+ "3905": 6976532480.0,
+ "3910": 6976532480.0,
+ "3915": 6976532480.0,
+ "3920": 6976532480.0,
+ "3925": 6976532480.0,
+ "3930": 6976532480.0,
+ "3935": 6976532480.0,
+ "3940": 6976532480.0,
+ "3945": 6976532480.0,
+ "3950": 6976532480.0,
+ "3955": 6976532480.0,
+ "3960": 6976532480.0,
+ "3965": 6976532480.0,
+ "3970": 6976532480.0,
+ "3975": 6976532480.0,
+ "3980": 6976532480.0,
+ "3985": 6976532480.0,
+ "3990": 6976532480.0,
+ "3995": 6976532480.0,
+ "4000": 6976532480.0,
+ "4005": 6976532480.0,
+ "4010": 6976532480.0,
+ "4015": 6976532480.0,
+ "4020": 6976532480.0,
+ "4025": 6976532480.0,
+ "4030": 6976532480.0,
+ "4035": 6976532480.0,
+ "4040": 6976532480.0,
+ "4045": 6976532480.0,
+ "4050": 6976532480.0,
+ "4055": 6976532480.0,
+ "4060": 6976532480.0,
+ "4065": 6976532480.0,
+ "4070": 6976532480.0,
+ "4075": 6976532480.0,
+ "4080": 6976532480.0,
+ "4085": 6976532480.0,
+ "4090": 6976532480.0,
+ "4095": 6976532480.0,
+ "4100": 6976532480.0,
+ "4105": 6976532480.0,
+ "4110": 6976532480.0,
+ "4115": 6976532480.0,
+ "4120": 6976532480.0,
+ "4125": 6976532480.0,
+ "4130": 6976532480.0,
+ "4135": 6976532480.0,
+ "4140": 6976532480.0,
+ "4145": 6976532480.0,
+ "4150": 6976532480.0,
+ "4155": 6976532480.0,
+ "4160": 6976532480.0,
+ "4165": 6976532480.0,
+ "4170": 6976532480.0,
+ "4175": 6976532480.0,
+ "4180": 6976532480.0,
+ "4185": 6976532480.0,
+ "4190": 6976532480.0,
+ "4195": 6976532480.0,
+ "4200": 6976532480.0,
+ "4205": 6976532480.0,
+ "4210": 6976532480.0,
+ "4215": 6976532480.0,
+ "4220": 6976532480.0,
+ "4225": 6976532480.0,
+ "4230": 6976532480.0,
+ "4235": 6976532480.0,
+ "4240": 6976532480.0,
+ "4245": 6976532480.0,
+ "4250": 6976532480.0,
+ "4255": 6976532480.0,
+ "4260": 6976532480.0,
+ "4265": 6976532480.0,
+ "4270": 6976532480.0,
+ "4275": 6976532480.0,
+ "4280": 6976532480.0,
+ "4285": 6976532480.0,
+ "4290": 6976532480.0,
+ "4295": 6976532480.0,
+ "4300": 6976532480.0,
+ "4305": 6976532480.0,
+ "4310": 6976532480.0,
+ "4315": 6976532480.0,
+ "4320": 6976532480.0,
+ "4325": 6976532480.0,
+ "4330": 6976532480.0,
+ "4335": 6976532480.0,
+ "4340": 6976532480.0,
+ "4345": 6976532480.0,
+ "4350": 6976532480.0,
+ "4355": 6976532480.0,
+ "4360": 6976532480.0,
+ "4365": 6976532480.0,
+ "4370": 6976532480.0,
+ "4375": 6976532480.0,
+ "4380": 6976532480.0,
+ "4385": 6976532480.0,
+ "4390": 6976532480.0,
+ "4395": 6976532480.0,
+ "4400": 6976532480.0,
+ "4405": 6976532480.0,
+ "4410": 6976532480.0,
+ "4415": 6976532480.0,
+ "4420": 6976532480.0,
+ "4425": 6976532480.0,
+ "4430": 6976532480.0,
+ "4435": 6976532480.0,
+ "4440": 6976532480.0,
+ "4445": 6976532480.0,
+ "4450": 6976532480.0,
+ "4455": 6976532480.0,
+ "4460": 6976532480.0,
+ "4465": 6976532480.0,
+ "4470": 6976532480.0,
+ "4475": 6976532480.0,
+ "4480": 6976532480.0,
+ "4485": 6976532480.0,
+ "4490": 6976532480.0,
+ "4495": 6976532480.0,
+ "4500": 6976532480.0,
+ "4505": 6976532480.0,
+ "4510": 6976532480.0,
+ "4515": 6976532480.0,
+ "4520": 6976532480.0,
+ "4525": 6976532480.0,
+ "4530": 6976532480.0,
+ "4535": 6976532480.0,
+ "4540": 6976532480.0,
+ "4545": 6976532480.0,
+ "4550": 6976532480.0,
+ "4555": 6976532480.0,
+ "4560": 6976532480.0,
+ "4565": 6976532480.0,
+ "4570": 6976532480.0,
+ "4575": 6976532480.0,
+ "4580": 6976532480.0,
+ "4585": 6976532480.0,
+ "4590": 6976532480.0,
+ "4595": 6976532480.0,
+ "4600": 6976532480.0,
+ "4605": 6976532480.0,
+ "4610": 6976532480.0,
+ "4615": 6976532480.0,
+ "4620": 6976532480.0,
+ "4625": 6976532480.0,
+ "4630": 6976532480.0,
+ "4635": 6976532480.0,
+ "4640": 6976532480.0,
+ "4645": 6976532480.0,
+ "4650": 6976532480.0,
+ "4655": 6976532480.0,
+ "4660": 6976532480.0,
+ "4665": 6976532480.0,
+ "4670": 6976532480.0,
+ "4675": 6976532480.0,
+ "4680": 6976532480.0,
+ "4685": 6976532480.0,
+ "4690": 6976532480.0,
+ "4695": 6976532480.0,
+ "4700": 6976532480.0,
+ "4705": 6976532480.0,
+ "4710": 6976532480.0,
+ "4715": 6976532480.0,
+ "4720": 6976532480.0,
+ "4725": 6976532480.0,
+ "4730": 6976532480.0,
+ "4735": 6976532480.0,
+ "4740": 6976532480.0,
+ "4745": 6976532480.0,
+ "4750": 6976532480.0,
+ "4755": 6976532480.0,
+ "4760": 6976532480.0,
+ "4765": 6976532480.0,
+ "4770": 6976532480.0,
+ "4775": 6976532480.0,
+ "4780": 6976532480.0,
+ "4785": 6976532480.0,
+ "4790": 6976532480.0,
+ "4795": 6976532480.0,
+ "4800": 6976532480.0,
+ "4805": 6976532480.0,
+ "4810": 6976532480.0,
+ "4815": 6976532480.0,
+ "4820": 6976532480.0,
+ "4825": 6976532480.0,
+ "4830": 6976532480.0,
+ "4835": 6976532480.0,
+ "4840": 6976532480.0,
+ "4845": 6976532480.0,
+ "4850": 6976532480.0,
+ "4855": 6976532480.0,
+ "4860": 6976532480.0,
+ "4865": 6976532480.0,
+ "4870": 6976532480.0,
+ "4875": 6976532480.0,
+ "4880": 6976532480.0,
+ "4885": 6976532480.0,
+ "4890": 6976532480.0,
+ "4895": 6976532480.0,
+ "4900": 6976532480.0,
+ "4905": 6976532480.0,
+ "4910": 6976532480.0,
+ "4915": 6976532480.0,
+ "4920": 6976532480.0,
+ "4925": 6976532480.0,
+ "4930": 6976532480.0,
+ "4935": 6976532480.0,
+ "4940": 6976532480.0,
+ "4945": 6976532480.0,
+ "4950": 6976532480.0,
+ "4955": 6976532480.0,
+ "4960": 6976532480.0,
+ "4965": 6976532480.0,
+ "4970": 6976532480.0,
+ "4975": 6976532480.0,
+ "4980": 6976532480.0,
+ "4985": 6976532480.0,
+ "4990": 6976532480.0,
+ "4995": 6976532480.0,
+ "5000": 6976532480.0,
+ "5005": 6976532480.0,
+ "5010": 6976532480.0,
+ "5015": 6976532480.0,
+ "5020": 6976532480.0,
+ "5025": 6976532480.0,
+ "5030": 6976532480.0,
+ "5035": 6976532480.0,
+ "5040": 6976532480.0,
+ "5045": 6976532480.0,
+ "5050": 6976532480.0,
+ "5055": 6976532480.0,
+ "5060": 6976532480.0,
+ "5065": 6976532480.0,
+ "5070": 6976532480.0,
+ "5075": 6976532480.0,
+ "5080": 6976532480.0,
+ "5085": 6976532480.0,
+ "5090": 6976532480.0,
+ "5095": 6976532480.0,
+ "5100": 6976532480.0,
+ "5105": 6976532480.0,
+ "5110": 6976532480.0,
+ "5115": 6976532480.0,
+ "5120": 6976532480.0,
+ "5125": 6976532480.0,
+ "5130": 6976532480.0,
+ "5135": 6976532480.0,
+ "5140": 6976532480.0,
+ "5145": 6976532480.0,
+ "5150": 6976532480.0,
+ "5155": 6976532480.0,
+ "5160": 6976532480.0,
+ "5165": 6976532480.0,
+ "5170": 6976532480.0,
+ "5175": 6976532480.0,
+ "5180": 6976532480.0,
+ "5185": 6976532480.0,
+ "5190": 6976532480.0,
+ "5195": 6976532480.0,
+ "5200": 6976532480.0,
+ "5205": 6976532480.0,
+ "5210": 6976532480.0,
+ "5215": 6976532480.0,
+ "5220": 6976532480.0,
+ "5225": 6976532480.0,
+ "5230": 6976532480.0,
+ "5235": 6976532480.0,
+ "5240": 6976532480.0,
+ "5245": 6976532480.0,
+ "5250": 6976532480.0,
+ "5255": 6976532480.0,
+ "5260": 6976532480.0,
+ "5265": 6976532480.0,
+ "5270": 6976532480.0,
+ "5275": 6976532480.0,
+ "5280": 6976532480.0,
+ "5285": 6976532480.0,
+ "5290": 6976532480.0,
+ "5295": 6976532480.0,
+ "5300": 6976532480.0,
+ "5305": 6976532480.0,
+ "5310": 6976532480.0,
+ "5315": 6976532480.0,
+ "5320": 6976532480.0,
+ "5325": 6976532480.0,
+ "5330": 6976532480.0,
+ "5335": 6976532480.0,
+ "5340": 6976532480.0,
+ "5345": 6976532480.0,
+ "5350": 6976532480.0,
+ "5355": 6976532480.0,
+ "5360": 6976532480.0,
+ "5365": 6976532480.0,
+ "5370": 6976532480.0,
+ "5375": 6976532480.0,
+ "5380": 6976532480.0,
+ "5385": 6976532480.0,
+ "5390": 6976532480.0,
+ "5395": 6976532480.0,
+ "5400": 6976532480.0,
+ "5405": 6976532480.0,
+ "5410": 6976532480.0,
+ "5415": 6976532480.0,
+ "5420": 6976532480.0,
+ "5425": 6976532480.0,
+ "5430": 6976532480.0,
+ "5435": 6976532480.0,
+ "5440": 6976532480.0,
+ "5445": 6976532480.0,
+ "5450": 6976532480.0,
+ "5455": 6976532480.0,
+ "5460": 6976532480.0,
+ "5465": 6976532480.0,
+ "5470": 6976532480.0,
+ "5475": 6976532480.0,
+ "5480": 6976532480.0,
+ "5485": 6976532480.0,
+ "5490": 6976532480.0,
+ "5495": 6976532480.0,
+ "5500": 6976532480.0,
+ "5505": 6976532480.0,
+ "5510": 6976532480.0,
+ "5515": 6976532480.0,
+ "5520": 6976532480.0,
+ "5525": 6976532480.0,
+ "5530": 6976532480.0,
+ "5535": 6976532480.0,
+ "5540": 6976532480.0,
+ "5545": 6976532480.0,
+ "5550": 6976532480.0,
+ "5555": 6976532480.0,
+ "5560": 6976532480.0,
+ "5565": 6976532480.0,
+ "5570": 6976532480.0,
+ "5575": 6976532480.0,
+ "5580": 6976532480.0,
+ "5585": 6976532480.0,
+ "5590": 6976532480.0,
+ "5595": 6976532480.0,
+ "5600": 6976532480.0,
+ "5605": 6976532480.0,
+ "5610": 6976532480.0,
+ "5615": 6976532480.0,
+ "5620": 6976532480.0,
+ "5625": 6976532480.0,
+ "5630": 6976532480.0,
+ "5635": 6976532480.0,
+ "5640": 6976532480.0,
+ "5645": 6976532480.0,
+ "5650": 6976532480.0,
+ "5655": 6976532480.0,
+ "5660": 6976532480.0,
+ "5665": 6976532480.0,
+ "5670": 6976532480.0,
+ "5675": 6976532480.0,
+ "5680": 6976532480.0,
+ "5685": 6976532480.0,
+ "5690": 6976532480.0,
+ "5695": 6976532480.0,
+ "5700": 6976532480.0,
+ "5705": 6976532480.0,
+ "5710": 6976532480.0,
+ "5715": 6976532480.0,
+ "5720": 6976532480.0,
+ "5725": 6976532480.0,
+ "5730": 6976532480.0,
+ "5735": 6976532480.0,
+ "5740": 6976532480.0,
+ "5745": 6976532480.0,
+ "5750": 6976532480.0,
+ "5755": 6976532480.0,
+ "5760": 6976532480.0,
+ "5765": 6976532480.0,
+ "5770": 6976532480.0,
+ "5775": 6976532480.0,
+ "5780": 6976532480.0,
+ "5785": 6976532480.0,
+ "5790": 6976532480.0,
+ "5795": 6976532480.0,
+ "5800": 6976532480.0,
+ "5805": 6976532480.0,
+ "5810": 6976532480.0,
+ "5815": 6976532480.0,
+ "5820": 6976532480.0,
+ "5825": 6976532480.0,
+ "5830": 6976532480.0,
+ "5835": 6976532480.0,
+ "5840": 6976532480.0,
+ "5845": 6976532480.0,
+ "5850": 6976532480.0,
+ "5855": 6976532480.0,
+ "5860": 6976532480.0,
+ "5865": 6976532480.0,
+ "5870": 6976532480.0,
+ "5875": 6976532480.0,
+ "5880": 6976532480.0,
+ "5885": 6976532480.0,
+ "5890": 6976532480.0,
+ "5895": 6976532480.0,
+ "5900": 6976532480.0,
+ "5905": 6976532480.0,
+ "5910": 6976532480.0,
+ "5915": 6976532480.0,
+ "5920": 6976532480.0,
+ "5925": 6976532480.0,
+ "5930": 6976532480.0,
+ "5935": 6976532480.0,
+ "5940": 6976532480.0,
+ "5945": 6976532480.0,
+ "5950": 6976532480.0,
+ "5955": 6976532480.0,
+ "5960": 6976532480.0,
+ "5965": 6976532480.0,
+ "5970": 6976532480.0,
+ "5975": 6976532480.0,
+ "5980": 6976532480.0,
+ "5985": 6976532480.0,
+ "5990": 6976532480.0,
+ "5995": 6976532480.0,
+ "6000": 6976532480.0,
+ "6005": 6976532480.0,
+ "6010": 6976532480.0,
+ "6015": 6976532480.0,
+ "6020": 6976532480.0,
+ "6025": 6976532480.0,
+ "6030": 6976532480.0,
+ "6035": 6976532480.0,
+ "6040": 6976532480.0,
+ "6045": 6976532480.0,
+ "6050": 6976532480.0,
+ "6055": 6976532480.0,
+ "6060": 6976532480.0,
+ "6065": 6976532480.0,
+ "6070": 6976532480.0,
+ "6075": 6976532480.0,
+ "6080": 6976532480.0,
+ "6085": 6976532480.0,
+ "6090": 6976532480.0,
+ "6095": 6976532480.0,
+ "6100": 6976532480.0,
+ "6105": 6976532480.0,
+ "6110": 6976532480.0,
+ "6115": 6976532480.0,
+ "6120": 6976532480.0,
+ "6125": 6976532480.0,
+ "6130": 6976532480.0,
+ "6135": 6976532480.0,
+ "6140": 6976532480.0,
+ "6145": 6976532480.0,
+ "6150": 6976532480.0,
+ "6155": 6976532480.0,
+ "6160": 6976532480.0,
+ "6165": 6976532480.0,
+ "6170": 6976532480.0,
+ "6175": 6976532480.0,
+ "6180": 6976532480.0,
+ "6185": 6976532480.0,
+ "6190": 6976532480.0,
+ "6195": 6976532480.0,
+ "6200": 6976532480.0,
+ "6205": 6976532480.0,
+ "6210": 6976532480.0,
+ "6215": 6976532480.0,
+ "6220": 6976532480.0,
+ "6225": 6976532480.0,
+ "6230": 6976532480.0,
+ "6235": 6976532480.0,
+ "6240": 6976532480.0,
+ "6245": 6976532480.0,
+ "6250": 6976532480.0,
+ "6255": 6976532480.0,
+ "6260": 6976532480.0,
+ "6265": 6976532480.0,
+ "6270": 6976532480.0,
+ "6275": 6976532480.0,
+ "6280": 6976532480.0,
+ "6285": 6976532480.0,
+ "6290": 6976532480.0,
+ "6295": 6976532480.0,
+ "6300": 6976532480.0,
+ "6305": 6976532480.0,
+ "6310": 6976532480.0,
+ "6315": 6976532480.0,
+ "6320": 6976532480.0,
+ "6325": 6976532480.0,
+ "6330": 6976532480.0,
+ "6335": 6976532480.0,
+ "6340": 6976532480.0,
+ "6345": 6976532480.0,
+ "6350": 6976532480.0,
+ "6355": 6976532480.0,
+ "6360": 6976532480.0,
+ "6365": 6976532480.0,
+ "6370": 6976532480.0,
+ "6375": 6976532480.0,
+ "6380": 6976532480.0,
+ "6385": 6976532480.0,
+ "6390": 6976532480.0,
+ "6395": 6976532480.0,
+ "6400": 6976532480.0,
+ "6405": 6976532480.0,
+ "6410": 6976532480.0,
+ "6415": 6976532480.0,
+ "6420": 6976532480.0,
+ "6425": 6976532480.0,
+ "6430": 6976532480.0,
+ "6435": 6976532480.0,
+ "6440": 6976532480.0,
+ "6445": 6976532480.0,
+ "6450": 6976532480.0,
+ "6455": 6976532480.0,
+ "6460": 6976532480.0,
+ "6465": 6976532480.0,
+ "6470": 6976532480.0,
+ "6475": 6976532480.0,
+ "6480": 6976532480.0,
+ "6485": 6976532480.0,
+ "6490": 6976532480.0,
+ "6495": 6976532480.0,
+ "6500": 6976532480.0,
+ "6505": 6976532480.0,
+ "6510": 6976532480.0,
+ "6515": 6976532480.0,
+ "6520": 6976532480.0,
+ "6525": 6976532480.0,
+ "6530": 6976532480.0,
+ "6535": 6976532480.0,
+ "6540": 6976532480.0,
+ "6545": 6976532480.0,
+ "6550": 6976532480.0,
+ "6555": 6976532480.0,
+ "6560": 6976532480.0,
+ "6565": 6976532480.0,
+ "6570": 6976532480.0,
+ "6575": 6976532480.0,
+ "6580": 6976532480.0,
+ "6585": 6976532480.0,
+ "6590": 6976532480.0,
+ "6595": 6976532480.0,
+ "6600": 6976532480.0,
+ "6605": 6976532480.0,
+ "6610": 6976532480.0,
+ "6615": 6976532480.0,
+ "6620": 6976532480.0,
+ "6625": 6976532480.0,
+ "6630": 6976532480.0,
+ "6635": 6976532480.0,
+ "6640": 6976532480.0,
+ "6645": 6976532480.0,
+ "6650": 6976532480.0,
+ "6655": 6976532480.0,
+ "6660": 6976532480.0,
+ "6665": 6976532480.0,
+ "6670": 6976532480.0,
+ "6675": 6976532480.0,
+ "6680": 6976532480.0,
+ "6685": 6976532480.0,
+ "6690": 6976532480.0,
+ "6695": 6976532480.0,
+ "6700": 6976532480.0,
+ "6705": 6976532480.0,
+ "6710": 6976532480.0,
+ "6715": 6976532480.0,
+ "6720": 6976532480.0,
+ "6725": 6976532480.0,
+ "6730": 6976532480.0,
+ "6735": 6976532480.0,
+ "6740": 6976532480.0,
+ "6745": 6976532480.0,
+ "6750": 6976532480.0,
+ "6755": 6976532480.0,
+ "6760": 6976532480.0,
+ "6765": 6976532480.0,
+ "6770": 6976532480.0,
+ "6775": 6976532480.0,
+ "6780": 6976532480.0,
+ "6785": 6976532480.0,
+ "6790": 6976532480.0,
+ "6795": 6976532480.0,
+ "6800": 6976532480.0,
+ "6805": 6976532480.0,
+ "6810": 6976532480.0,
+ "6815": 6976532480.0,
+ "6820": 6976532480.0,
+ "6825": 6976532480.0,
+ "6830": 6976532480.0,
+ "6835": 6976532480.0,
+ "6840": 6976532480.0,
+ "6845": 6976532480.0,
+ "6850": 6976532480.0,
+ "6855": 6976532480.0,
+ "6860": 6976532480.0,
+ "6865": 6976532480.0,
+ "6870": 6976532480.0,
+ "6875": 6976532480.0,
+ "6880": 6976532480.0,
+ "6885": 6976532480.0,
+ "6890": 6976532480.0,
+ "6895": 6976532480.0,
+ "6900": 6976532480.0,
+ "6905": 6976532480.0,
+ "6910": 6976532480.0,
+ "6915": 6976532480.0,
+ "6920": 6976532480.0,
+ "6925": 6976532480.0,
+ "6930": 6976532480.0,
+ "6935": 6976532480.0,
+ "6940": 6976532480.0,
+ "6945": 6976532480.0,
+ "6950": 6976532480.0,
+ "6955": 6976532480.0,
+ "6960": 6976532480.0,
+ "6965": 6976532480.0,
+ "6970": 6976532480.0,
+ "6975": 6976532480.0,
+ "6980": 6976532480.0,
+ "6985": 6976532480.0,
+ "6990": 6976532480.0,
+ "6995": 6976532480.0,
+ "7000": 6976532480.0,
+ "7005": 6976532480.0,
+ "7010": 6976532480.0,
+ "7015": 6976532480.0,
+ "7020": 6976532480.0,
+ "7025": 6976532480.0,
+ "7030": 6976532480.0,
+ "7035": 6976532480.0,
+ "7040": 6976532480.0,
+ "7045": 6976532480.0,
+ "7050": 6976532480.0,
+ "7055": 6976532480.0,
+ "7060": 6976532480.0,
+ "7065": 6976532480.0,
+ "7070": 6976532480.0,
+ "7075": 6976532480.0,
+ "7080": 6976532480.0,
+ "7085": 6976532480.0,
+ "7090": 6976532480.0,
+ "7095": 6976532480.0,
+ "7100": 6976532480.0,
+ "7105": 6976532480.0,
+ "7110": 6976532480.0,
+ "7115": 6976532480.0,
+ "7120": 6976532480.0,
+ "7125": 6976532480.0,
+ "7130": 6976532480.0,
+ "7135": 6976532480.0,
+ "7140": 6976532480.0,
+ "7145": 6976532480.0,
+ "7150": 6976532480.0,
+ "7155": 6976532480.0,
+ "7160": 6976532480.0,
+ "7165": 6976532480.0,
+ "7170": 6976532480.0,
+ "7175": 6976532480.0,
+ "7180": 6976532480.0,
+ "7185": 6976532480.0,
+ "7190": 6976532480.0,
+ "7195": 6976532480.0,
+ "7200": 6976532480.0,
+ "7205": 6976532480.0,
+ "7210": 6976532480.0,
+ "7215": 6976532480.0,
+ "7220": 6976532480.0,
+ "7225": 6976532480.0,
+ "7230": 6976532480.0,
+ "7235": 6976532480.0,
+ "7240": 6976532480.0,
+ "7245": 6976532480.0,
+ "7250": 6976532480.0,
+ "7255": 6976532480.0,
+ "7260": 6976532480.0,
+ "7265": 6976532480.0,
+ "7270": 6976532480.0,
+ "7275": 6976532480.0,
+ "7280": 6976532480.0,
+ "7285": 6976532480.0,
+ "7290": 6976532480.0,
+ "7295": 6976532480.0,
+ "7300": 6976532480.0,
+ "7305": 6976532480.0,
+ "7310": 6976532480.0,
+ "7315": 6976532480.0,
+ "7320": 6976532480.0,
+ "7325": 6976532480.0,
+ "7330": 6976532480.0,
+ "7335": 6976532480.0,
+ "7340": 6976532480.0,
+ "7345": 6976532480.0,
+ "7350": 6976532480.0,
+ "7355": 6976532480.0,
+ "7360": 6976532480.0,
+ "7365": 6976532480.0,
+ "7370": 6976532480.0,
+ "7375": 6976532480.0,
+ "7380": 6976532480.0,
+ "7385": 6976532480.0,
+ "7390": 6976532480.0,
+ "7395": 6976532480.0,
+ "7400": 6976532480.0,
+ "7405": 6976532480.0,
+ "7410": 6976532480.0,
+ "7415": 6976532480.0,
+ "7420": 6976532480.0,
+ "7425": 6976532480.0,
+ "7430": 6976532480.0,
+ "7435": 6976532480.0,
+ "7440": 6976532480.0,
+ "7445": 6976532480.0,
+ "7450": 6976532480.0,
+ "7455": 6976532480.0,
+ "7460": 6976532480.0,
+ "7465": 6976532480.0,
+ "7470": 6976532480.0,
+ "7475": 6976532480.0,
+ "7480": 6976532480.0,
+ "7485": 6976532480.0,
+ "7490": 6976532480.0,
+ "7495": 6976532480.0,
+ "7500": 6976532480.0,
+ "7505": 6976532480.0,
+ "7510": 6976532480.0,
+ "7515": 6976532480.0,
+ "7520": 6976532480.0,
+ "7525": 6976532480.0,
+ "7530": 6976532480.0,
+ "7535": 6976532480.0,
+ "7540": 6976532480.0,
+ "7545": 6976532480.0,
+ "7550": 6976532480.0,
+ "7555": 6976532480.0,
+ "7560": 6976532480.0,
+ "7565": 6976532480.0,
+ "7570": 6976532480.0,
+ "7575": 6976532480.0,
+ "7580": 6976532480.0,
+ "7585": 6976532480.0,
+ "7590": 6976532480.0,
+ "7595": 6976532480.0,
+ "7600": 6976532480.0,
+ "7605": 6976532480.0,
+ "7610": 6976532480.0,
+ "7615": 6976532480.0,
+ "7620": 6976532480.0,
+ "7625": 6976532480.0,
+ "7630": 6976532480.0,
+ "7635": 6976532480.0,
+ "7640": 6976532480.0,
+ "7645": 6976532480.0,
+ "7650": 6976532480.0,
+ "7655": 6976532480.0,
+ "7660": 6976532480.0,
+ "7665": 6976532480.0,
+ "7670": 6976532480.0,
+ "7675": 6976532480.0,
+ "7680": 6976532480.0,
+ "7685": 6976532480.0,
+ "7690": 6976532480.0,
+ "7695": 6976532480.0,
+ "7700": 6976532480.0,
+ "7705": 6976532480.0,
+ "7710": 6976532480.0,
+ "7715": 6976532480.0,
+ "7720": 6976532480.0,
+ "7725": 6976532480.0,
+ "7730": 6976532480.0,
+ "7735": 6976532480.0,
+ "7740": 6976532480.0,
+ "7745": 6976532480.0,
+ "7750": 6976532480.0,
+ "7755": 6976532480.0,
+ "7760": 6976532480.0,
+ "7765": 6976532480.0,
+ "7770": 6976532480.0,
+ "7775": 6976532480.0,
+ "7780": 6976532480.0,
+ "7785": 6976532480.0,
+ "7790": 6976532480.0,
+ "7795": 6976532480.0,
+ "7800": 6976532480.0,
+ "7805": 6976532480.0,
+ "7810": 6976532480.0,
+ "7815": 6976532480.0,
+ "7820": 6976532480.0,
+ "7825": 6976532480.0,
+ "7830": 6976532480.0,
+ "7835": 6976532480.0,
+ "7840": 6976532480.0,
+ "7845": 6976532480.0,
+ "7850": 6976532480.0,
+ "7855": 6976532480.0,
+ "7860": 6976532480.0,
+ "7865": 6976532480.0,
+ "7870": 6976532480.0,
+ "7875": 6976532480.0,
+ "7880": 6976532480.0,
+ "7885": 6976532480.0,
+ "7890": 6976532480.0,
+ "7895": 6976532480.0,
+ "7900": 6976532480.0,
+ "7905": 6976532480.0,
+ "7910": 6976532480.0,
+ "7915": 6976532480.0,
+ "7920": 6976532480.0,
+ "7925": 6976532480.0,
+ "7930": 6976532480.0,
+ "7935": 6976532480.0,
+ "7940": 6976532480.0,
+ "7945": 6976532480.0,
+ "7950": 6976532480.0,
+ "7955": 6976532480.0,
+ "7960": 6976532480.0,
+ "7965": 6976532480.0,
+ "7970": 6976532480.0,
+ "7975": 6976532480.0,
+ "7980": 6976532480.0,
+ "7985": 6976532480.0,
+ "7990": 6976532480.0,
+ "7995": 6976532480.0,
+ "8000": 6976532480.0,
+ "8005": 6976532480.0,
+ "8010": 6976532480.0,
+ "8015": 6976532480.0,
+ "8020": 6976532480.0,
+ "8025": 6976532480.0,
+ "8030": 6976532480.0,
+ "8035": 6976532480.0,
+ "8040": 6976532480.0,
+ "8045": 6976532480.0,
+ "8050": 6976532480.0,
+ "8055": 6976532480.0,
+ "8060": 6976532480.0,
+ "8065": 6976532480.0,
+ "8070": 6976532480.0,
+ "8075": 6976532480.0,
+ "8080": 6976532480.0,
+ "8085": 6976532480.0,
+ "8090": 6976532480.0,
+ "8095": 6976532480.0,
+ "8100": 6976532480.0,
+ "8105": 6976532480.0,
+ "8110": 6976532480.0,
+ "8115": 6976532480.0,
+ "8120": 6976532480.0,
+ "8125": 6976532480.0,
+ "8130": 6976532480.0,
+ "8135": 6976532480.0,
+ "8140": 6976532480.0,
+ "8145": 6976532480.0,
+ "8150": 6976532480.0,
+ "8155": 6976532480.0,
+ "8160": 6976532480.0,
+ "8165": 6976532480.0,
+ "8170": 6976532480.0,
+ "8175": 6976532480.0,
+ "8180": 6976532480.0,
+ "8185": 6976532480.0,
+ "8190": 6976532480.0,
+ "8195": 6976532480.0,
+ "8200": 6976532480.0,
+ "8205": 6976532480.0,
+ "8210": 6976532480.0,
+ "8215": 6976532480.0,
+ "8220": 6976532480.0,
+ "8225": 6976532480.0,
+ "8230": 6976532480.0,
+ "8235": 6976532480.0,
+ "8240": 6976532480.0,
+ "8245": 6976532480.0,
+ "8250": 6976532480.0,
+ "8255": 6976532480.0,
+ "8260": 6976532480.0,
+ "8265": 6976532480.0,
+ "8270": 6976532480.0,
+ "8275": 6976532480.0,
+ "8280": 6976532480.0,
+ "8285": 6976532480.0,
+ "8290": 6976532480.0,
+ "8295": 6976532480.0,
+ "8300": 6976532480.0,
+ "8305": 6976532480.0,
+ "8310": 6976532480.0,
+ "8315": 6976532480.0,
+ "8320": 6976532480.0,
+ "8325": 6976532480.0,
+ "8330": 6976532480.0,
+ "8335": 6976532480.0,
+ "8340": 6976532480.0,
+ "8345": 6976532480.0,
+ "8350": 6976532480.0,
+ "8355": 6976532480.0,
+ "8360": 6976532480.0,
+ "8365": 6976532480.0,
+ "8370": 6976532480.0,
+ "8375": 6976532480.0,
+ "8380": 6976532480.0,
+ "8385": 6976532480.0,
+ "8390": 6976532480.0,
+ "8395": 6976532480.0,
+ "8400": 6976532480.0,
+ "8405": 6976532480.0,
+ "8410": 6976532480.0,
+ "8415": 6976532480.0,
+ "8420": 6976532480.0,
+ "8425": 6976532480.0,
+ "8430": 6976532480.0,
+ "8435": 6976532480.0,
+ "8440": 6976532480.0,
+ "8445": 6976532480.0,
+ "8450": 6976532480.0,
+ "8455": 6976532480.0,
+ "8460": 6976532480.0,
+ "8465": 6976532480.0,
+ "8470": 6976532480.0,
+ "8475": 6976532480.0,
+ "8480": 6976532480.0,
+ "8485": 6976532480.0,
+ "8490": 6976532480.0,
+ "8495": 6976532480.0,
+ "8500": 6976532480.0,
+ "8505": 6976532480.0,
+ "8510": 6976532480.0,
+ "8515": 6976532480.0,
+ "8520": 6976532480.0,
+ "8525": 6976532480.0,
+ "8530": 6976532480.0,
+ "8535": 6976532480.0,
+ "8540": 6976532480.0,
+ "8545": 6976532480.0,
+ "8550": 6976532480.0,
+ "8555": 6976532480.0,
+ "8560": 6976532480.0,
+ "8565": 6976532480.0,
+ "8570": 6976532480.0,
+ "8575": 6976532480.0,
+ "8580": 6976532480.0,
+ "8585": 6976532480.0,
+ "8590": 6976532480.0,
+ "8595": 6976532480.0,
+ "8600": 6976532480.0,
+ "8605": 6976532480.0,
+ "8610": 6976532480.0,
+ "8615": 6976532480.0,
+ "8620": 6976532480.0,
+ "8625": 6976532480.0,
+ "8630": 6976532480.0,
+ "8635": 6976532480.0,
+ "8640": 6976532480.0,
+ "8645": 6976532480.0,
+ "8650": 6976532480.0,
+ "8655": 6976532480.0,
+ "8660": 6976532480.0,
+ "8665": 6976532480.0,
+ "8670": 6976532480.0,
+ "8675": 6976532480.0,
+ "8680": 6976532480.0,
+ "8685": 6976532480.0,
+ "8690": 6976532480.0,
+ "8695": 6976532480.0,
+ "8700": 6976532480.0,
+ "8705": 6976532480.0,
+ "8710": 6976532480.0,
+ "8715": 6976532480.0,
+ "8720": 6976532480.0,
+ "8725": 6976532480.0,
+ "8730": 6976532480.0,
+ "8735": 6976532480.0,
+ "8740": 6976532480.0,
+ "8745": 6976532480.0,
+ "8750": 6976532480.0,
+ "8755": 6976532480.0,
+ "8760": 6976532480.0,
+ "8765": 6976532480.0,
+ "8770": 6976532480.0,
+ "8775": 6976532480.0,
+ "8780": 6976532480.0,
+ "8785": 6976532480.0,
+ "8790": 6976532480.0,
+ "8795": 6976532480.0,
+ "8800": 6976532480.0,
+ "8805": 6976532480.0,
+ "8810": 6976532480.0,
+ "8815": 6976532480.0,
+ "8820": 6976532480.0,
+ "8825": 6976532480.0,
+ "8830": 6976532480.0,
+ "8835": 6976532480.0,
+ "8840": 6976532480.0,
+ "8845": 6976532480.0,
+ "8850": 6976532480.0,
+ "8855": 6976532480.0,
+ "8860": 6976532480.0,
+ "8865": 6976532480.0,
+ "8870": 6976532480.0,
+ "8875": 6976532480.0,
+ "8880": 6976532480.0,
+ "8885": 6976532480.0,
+ "8890": 6976532480.0,
+ "8895": 6976532480.0,
+ "8900": 6976532480.0,
+ "8905": 6976532480.0,
+ "8910": 6976532480.0,
+ "8915": 6976532480.0,
+ "8920": 6976532480.0,
+ "8925": 6976532480.0,
+ "8930": 6976532480.0,
+ "8935": 6976532480.0,
+ "8940": 6976532480.0,
+ "8945": 6976532480.0,
+ "8950": 6976532480.0,
+ "8955": 6976532480.0,
+ "8960": 6976532480.0,
+ "8965": 6976532480.0,
+ "8970": 6976532480.0,
+ "8975": 6976532480.0,
+ "8980": 6976532480.0,
+ "8985": 6976532480.0,
+ "8990": 6976532480.0,
+ "8995": 6976532480.0,
+ "9000": 6976532480.0,
+ "9005": 6976532480.0,
+ "9010": 6976532480.0,
+ "9015": 6976532480.0,
+ "9020": 6976532480.0,
+ "9025": 6976532480.0,
+ "9030": 6976532480.0,
+ "9035": 6976532480.0,
+ "9040": 6976532480.0,
+ "9045": 6976532480.0,
+ "9050": 6976532480.0,
+ "9055": 6976532480.0,
+ "9060": 6976532480.0,
+ "9065": 6976532480.0,
+ "9070": 6976532480.0,
+ "9075": 6976532480.0,
+ "9080": 6976532480.0,
+ "9085": 6976532480.0,
+ "9090": 6976532480.0,
+ "9095": 6976532480.0,
+ "9100": 6976532480.0,
+ "9105": 6976532480.0,
+ "9110": 6976532480.0,
+ "9115": 6976532480.0,
+ "9120": 6976532480.0,
+ "9125": 6976532480.0,
+ "9130": 6976532480.0,
+ "9135": 6976532480.0,
+ "9140": 6976532480.0,
+ "9145": 6976532480.0,
+ "9150": 6976532480.0,
+ "9155": 6976532480.0,
+ "9160": 6976532480.0,
+ "9165": 6976532480.0,
+ "9170": 6976532480.0,
+ "9175": 6976532480.0,
+ "9180": 6976532480.0,
+ "9185": 6976532480.0,
+ "9190": 6976532480.0,
+ "9195": 6976532480.0,
+ "9200": 6976532480.0,
+ "9205": 6976532480.0,
+ "9210": 6976532480.0,
+ "9215": 6976532480.0,
+ "9220": 6976532480.0,
+ "9225": 6976532480.0,
+ "9230": 6976532480.0,
+ "9235": 6976532480.0,
+ "9240": 6976532480.0,
+ "9245": 6976532480.0,
+ "9250": 6976532480.0,
+ "9255": 6976532480.0,
+ "9260": 6976532480.0,
+ "9265": 6976532480.0,
+ "9270": 6976532480.0,
+ "9275": 6976532480.0,
+ "9280": 6976532480.0,
+ "9285": 6976532480.0,
+ "9290": 6976532480.0,
+ "9295": 6976532480.0,
+ "9300": 6976532480.0,
+ "9305": 6976532480.0,
+ "9310": 6976532480.0,
+ "9315": 6976532480.0,
+ "9320": 6976532480.0,
+ "9325": 6976532480.0,
+ "9330": 6976532480.0,
+ "9335": 6976532480.0,
+ "9340": 6976532480.0,
+ "9345": 6976532480.0,
+ "9350": 6976532480.0,
+ "9355": 6976532480.0,
+ "9360": 6976532480.0,
+ "9365": 6976532480.0,
+ "9370": 6976532480.0,
+ "9375": 6976532480.0,
+ "9380": 6976532480.0,
+ "9385": 6976532480.0,
+ "9390": 6976532480.0,
+ "9395": 6976532480.0,
+ "9400": 6976532480.0,
+ "9405": 6976532480.0,
+ "9410": 6976532480.0,
+ "9415": 6976532480.0,
+ "9420": 6976532480.0,
+ "9425": 6976532480.0,
+ "9430": 6976532480.0,
+ "9435": 6976532480.0,
+ "9440": 6976532480.0,
+ "9445": 6976532480.0,
+ "9450": 6976532480.0,
+ "9455": 6976532480.0,
+ "9460": 6976532480.0,
+ "9465": 6976532480.0,
+ "9470": 6976532480.0,
+ "9475": 6976532480.0,
+ "9480": 6976532480.0,
+ "9485": 6976532480.0,
+ "9490": 6976532480.0,
+ "9495": 6976532480.0,
+ "9500": 6976532480.0,
+ "9505": 6976532480.0,
+ "9510": 6976532480.0,
+ "9515": 6976532480.0,
+ "9520": 6976532480.0,
+ "9525": 6976532480.0,
+ "9530": 6976532480.0,
+ "9535": 6976532480.0,
+ "9540": 6976532480.0,
+ "9545": 6976532480.0,
+ "9550": 6976532480.0,
+ "9555": 6976532480.0,
+ "9560": 6976532480.0,
+ "9565": 6976532480.0,
+ "9570": 6976532480.0,
+ "9575": 6976532480.0,
+ "9580": 6976532480.0,
+ "9585": 6976532480.0,
+ "9590": 6976532480.0,
+ "9595": 6976532480.0,
+ "9600": 6976532480.0,
+ "9605": 6976532480.0,
+ "9610": 6976532480.0,
+ "9615": 6976532480.0,
+ "9620": 6976532480.0,
+ "9625": 6976532480.0,
+ "9630": 6976532480.0,
+ "9635": 6976532480.0,
+ "9640": 6976532480.0,
+ "9645": 6976532480.0,
+ "9650": 6976532480.0,
+ "9655": 6976532480.0,
+ "9660": 6976532480.0,
+ "9665": 6976532480.0,
+ "9670": 6976532480.0,
+ "9675": 6976532480.0,
+ "9680": 6976532480.0,
+ "9685": 6976532480.0,
+ "9690": 6976532480.0,
+ "9695": 6976532480.0,
+ "9700": 6976532480.0,
+ "9705": 6976532480.0,
+ "9710": 6976532480.0,
+ "9715": 6976532480.0,
+ "9720": 6976532480.0,
+ "9725": 6976532480.0,
+ "9730": 6976532480.0,
+ "9735": 6976532480.0,
+ "9740": 6976532480.0,
+ "9745": 6976532480.0,
+ "9750": 6976532480.0,
+ "9755": 6976532480.0,
+ "9760": 6976532480.0,
+ "9765": 6976532480.0,
+ "9770": 6976532480.0,
+ "9775": 6976532480.0,
+ "9780": 6976532480.0,
+ "9785": 6976532480.0,
+ "9790": 6976532480.0,
+ "9795": 6976532480.0,
+ "9800": 6976532480.0,
+ "9805": 6976532480.0,
+ "9810": 6976532480.0,
+ "9815": 6976532480.0,
+ "9820": 6976532480.0,
+ "9825": 6976532480.0,
+ "9830": 6976532480.0,
+ "9835": 6976532480.0,
+ "9840": 6976532480.0,
+ "9845": 6976532480.0,
+ "9850": 6976532480.0,
+ "9855": 6976532480.0,
+ "9860": 6976532480.0,
+ "9865": 6976532480.0,
+ "9870": 6976532480.0,
+ "9875": 6976532480.0,
+ "9880": 6976532480.0,
+ "9885": 6976532480.0,
+ "9890": 6976532480.0,
+ "9895": 6976532480.0,
+ "9900": 6976532480.0,
+ "9905": 6976532480.0,
+ "9910": 6976532480.0,
+ "9915": 6976532480.0,
+ "9920": 6976532480.0,
+ "9925": 6976532480.0,
+ "9930": 6976532480.0,
+ "9935": 6976532480.0,
+ "9940": 6976532480.0,
+ "9945": 6976532480.0,
+ "9950": 6976532480.0,
+ "9955": 6976532480.0,
+ "9960": 6976532480.0,
+ "9965": 6976532480.0,
+ "9970": 6976532480.0,
+ "9975": 6976532480.0,
+ "9980": 6976532480.0,
+ "9985": 6976532480.0,
+ "9990": 6976532480.0,
+ "9995": 6976532480.0,
+ "10000": 6976532480.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 10000,
+ "step_interval": 5,
+ "values": {
+ "1": "nan",
+ "5": "nan",
+ "10": "nan",
+ "15": "nan",
+ "20": "nan",
+ "25": "nan",
+ "30": "nan",
+ "35": "nan",
+ "40": "nan",
+ "45": "nan",
+ "50": "nan",
+ "55": "nan",
+ "60": "nan",
+ "65": "nan",
+ "70": "nan",
+ "75": "nan",
+ "80": "nan",
+ "85": "nan",
+ "90": "nan",
+ "95": "nan",
+ "100": 0.41745,
+ "105": "nan",
+ "110": "nan",
+ "115": "nan",
+ "120": "nan",
+ "125": "nan",
+ "130": "nan",
+ "135": "nan",
+ "140": "nan",
+ "145": "nan",
+ "150": "nan",
+ "155": "nan",
+ "160": "nan",
+ "165": "nan",
+ "170": "nan",
+ "175": "nan",
+ "180": "nan",
+ "185": "nan",
+ "190": "nan",
+ "195": "nan",
+ "200": 0.29783,
+ "205": "nan",
+ "210": "nan",
+ "215": "nan",
+ "220": "nan",
+ "225": "nan",
+ "230": "nan",
+ "235": "nan",
+ "240": "nan",
+ "245": "nan",
+ "250": "nan",
+ "255": "nan",
+ "260": "nan",
+ "265": "nan",
+ "270": "nan",
+ "275": "nan",
+ "280": "nan",
+ "285": "nan",
+ "290": "nan",
+ "295": "nan",
+ "300": 0.46526,
+ "305": "nan",
+ "310": "nan",
+ "315": "nan",
+ "320": "nan",
+ "325": "nan",
+ "330": "nan",
+ "335": "nan",
+ "340": "nan",
+ "345": "nan",
+ "350": "nan",
+ "355": "nan",
+ "360": "nan",
+ "365": "nan",
+ "370": "nan",
+ "375": "nan",
+ "380": "nan",
+ "385": "nan",
+ "390": "nan",
+ "395": "nan",
+ "400": 0.279,
+ "405": "nan",
+ "410": "nan",
+ "415": "nan",
+ "420": "nan",
+ "425": "nan",
+ "430": "nan",
+ "435": "nan",
+ "440": "nan",
+ "445": "nan",
+ "450": "nan",
+ "455": "nan",
+ "460": "nan",
+ "465": "nan",
+ "470": "nan",
+ "475": "nan",
+ "480": "nan",
+ "485": "nan",
+ "490": "nan",
+ "495": "nan",
+ "500": 0.27998,
+ "505": "nan",
+ "510": "nan",
+ "515": "nan",
+ "520": "nan",
+ "525": "nan",
+ "530": "nan",
+ "535": "nan",
+ "540": "nan",
+ "545": "nan",
+ "550": "nan",
+ "555": "nan",
+ "560": "nan",
+ "565": "nan",
+ "570": "nan",
+ "575": "nan",
+ "580": "nan",
+ "585": "nan",
+ "590": "nan",
+ "595": "nan",
+ "600": 0.27814,
+ "605": "nan",
+ "610": "nan",
+ "615": "nan",
+ "620": "nan",
+ "625": "nan",
+ "630": "nan",
+ "635": "nan",
+ "640": "nan",
+ "645": "nan",
+ "650": "nan",
+ "655": "nan",
+ "660": "nan",
+ "665": "nan",
+ "670": "nan",
+ "675": "nan",
+ "680": "nan",
+ "685": "nan",
+ "690": "nan",
+ "695": "nan",
+ "700": 0.31061,
+ "705": "nan",
+ "710": "nan",
+ "715": "nan",
+ "720": "nan",
+ "725": "nan",
+ "730": "nan",
+ "735": "nan",
+ "740": "nan",
+ "745": "nan",
+ "750": "nan",
+ "755": "nan",
+ "760": "nan",
+ "765": "nan",
+ "770": "nan",
+ "775": "nan",
+ "780": "nan",
+ "785": "nan",
+ "790": "nan",
+ "795": "nan",
+ "800": 0.36709,
+ "805": "nan",
+ "810": "nan",
+ "815": "nan",
+ "820": "nan",
+ "825": "nan",
+ "830": "nan",
+ "835": "nan",
+ "840": "nan",
+ "845": "nan",
+ "850": "nan",
+ "855": "nan",
+ "860": "nan",
+ "865": "nan",
+ "870": "nan",
+ "875": "nan",
+ "880": "nan",
+ "885": "nan",
+ "890": "nan",
+ "895": "nan",
+ "900": 0.35575,
+ "905": "nan",
+ "910": "nan",
+ "915": "nan",
+ "920": "nan",
+ "925": "nan",
+ "930": "nan",
+ "935": "nan",
+ "940": "nan",
+ "945": "nan",
+ "950": "nan",
+ "955": "nan",
+ "960": "nan",
+ "965": "nan",
+ "970": "nan",
+ "975": "nan",
+ "980": "nan",
+ "985": "nan",
+ "990": "nan",
+ "995": "nan",
+ "1000": 0.27626,
+ "1005": "nan",
+ "1010": "nan",
+ "1015": "nan",
+ "1020": "nan",
+ "1025": "nan",
+ "1030": "nan",
+ "1035": "nan",
+ "1040": "nan",
+ "1045": "nan",
+ "1050": "nan",
+ "1055": "nan",
+ "1060": "nan",
+ "1065": "nan",
+ "1070": "nan",
+ "1075": "nan",
+ "1080": "nan",
+ "1085": "nan",
+ "1090": "nan",
+ "1095": "nan",
+ "1100": 0.29956,
+ "1105": "nan",
+ "1110": "nan",
+ "1115": "nan",
+ "1120": "nan",
+ "1125": "nan",
+ "1130": "nan",
+ "1135": "nan",
+ "1140": "nan",
+ "1145": "nan",
+ "1150": "nan",
+ "1155": "nan",
+ "1160": "nan",
+ "1165": "nan",
+ "1170": "nan",
+ "1175": "nan",
+ "1180": "nan",
+ "1185": "nan",
+ "1190": "nan",
+ "1195": "nan",
+ "1200": 0.31326,
+ "1205": "nan",
+ "1210": "nan",
+ "1215": "nan",
+ "1220": "nan",
+ "1225": "nan",
+ "1230": "nan",
+ "1235": "nan",
+ "1240": "nan",
+ "1245": "nan",
+ "1250": "nan",
+ "1255": "nan",
+ "1260": "nan",
+ "1265": "nan",
+ "1270": "nan",
+ "1275": "nan",
+ "1280": "nan",
+ "1285": "nan",
+ "1290": "nan",
+ "1295": "nan",
+ "1300": 0.31493,
+ "1305": "nan",
+ "1310": "nan",
+ "1315": "nan",
+ "1320": "nan",
+ "1325": "nan",
+ "1330": "nan",
+ "1335": "nan",
+ "1340": "nan",
+ "1345": "nan",
+ "1350": "nan",
+ "1355": "nan",
+ "1360": "nan",
+ "1365": "nan",
+ "1370": "nan",
+ "1375": "nan",
+ "1380": "nan",
+ "1385": "nan",
+ "1390": "nan",
+ "1395": "nan",
+ "1400": 0.34858,
+ "1405": "nan",
+ "1410": "nan",
+ "1415": "nan",
+ "1420": "nan",
+ "1425": "nan",
+ "1430": "nan",
+ "1435": "nan",
+ "1440": "nan",
+ "1445": "nan",
+ "1450": "nan",
+ "1455": "nan",
+ "1460": "nan",
+ "1465": "nan",
+ "1470": "nan",
+ "1475": "nan",
+ "1480": "nan",
+ "1485": "nan",
+ "1490": "nan",
+ "1495": "nan",
+ "1500": 0.29301,
+ "1505": "nan",
+ "1510": "nan",
+ "1515": "nan",
+ "1520": "nan",
+ "1525": "nan",
+ "1530": "nan",
+ "1535": "nan",
+ "1540": "nan",
+ "1545": "nan",
+ "1550": "nan",
+ "1555": "nan",
+ "1560": "nan",
+ "1565": "nan",
+ "1570": "nan",
+ "1575": "nan",
+ "1580": "nan",
+ "1585": "nan",
+ "1590": "nan",
+ "1595": "nan",
+ "1600": 0.31075,
+ "1605": "nan",
+ "1610": "nan",
+ "1615": "nan",
+ "1620": "nan",
+ "1625": "nan",
+ "1630": "nan",
+ "1635": "nan",
+ "1640": "nan",
+ "1645": "nan",
+ "1650": "nan",
+ "1655": "nan",
+ "1660": "nan",
+ "1665": "nan",
+ "1670": "nan",
+ "1675": "nan",
+ "1680": "nan",
+ "1685": "nan",
+ "1690": "nan",
+ "1695": "nan",
+ "1700": 0.29068,
+ "1705": "nan",
+ "1710": "nan",
+ "1715": "nan",
+ "1720": "nan",
+ "1725": "nan",
+ "1730": "nan",
+ "1735": "nan",
+ "1740": "nan",
+ "1745": "nan",
+ "1750": "nan",
+ "1755": "nan",
+ "1760": "nan",
+ "1765": "nan",
+ "1770": "nan",
+ "1775": "nan",
+ "1780": "nan",
+ "1785": "nan",
+ "1790": "nan",
+ "1795": "nan",
+ "1800": 0.28353,
+ "1805": "nan",
+ "1810": "nan",
+ "1815": "nan",
+ "1820": "nan",
+ "1825": "nan",
+ "1830": "nan",
+ "1835": "nan",
+ "1840": "nan",
+ "1845": "nan",
+ "1850": "nan",
+ "1855": "nan",
+ "1860": "nan",
+ "1865": "nan",
+ "1870": "nan",
+ "1875": "nan",
+ "1880": "nan",
+ "1885": "nan",
+ "1890": "nan",
+ "1895": "nan",
+ "1900": 0.3545,
+ "1905": "nan",
+ "1910": "nan",
+ "1915": "nan",
+ "1920": "nan",
+ "1925": "nan",
+ "1930": "nan",
+ "1935": "nan",
+ "1940": "nan",
+ "1945": "nan",
+ "1950": "nan",
+ "1955": "nan",
+ "1960": "nan",
+ "1965": "nan",
+ "1970": "nan",
+ "1975": "nan",
+ "1980": "nan",
+ "1985": "nan",
+ "1990": "nan",
+ "1995": "nan",
+ "2000": 0.33465,
+ "2005": "nan",
+ "2010": "nan",
+ "2015": "nan",
+ "2020": "nan",
+ "2025": "nan",
+ "2030": "nan",
+ "2035": "nan",
+ "2040": "nan",
+ "2045": "nan",
+ "2050": "nan",
+ "2055": "nan",
+ "2060": "nan",
+ "2065": "nan",
+ "2070": "nan",
+ "2075": "nan",
+ "2080": "nan",
+ "2085": "nan",
+ "2090": "nan",
+ "2095": "nan",
+ "2100": 0.37474,
+ "2105": "nan",
+ "2110": "nan",
+ "2115": "nan",
+ "2120": "nan",
+ "2125": "nan",
+ "2130": "nan",
+ "2135": "nan",
+ "2140": "nan",
+ "2145": "nan",
+ "2150": "nan",
+ "2155": "nan",
+ "2160": "nan",
+ "2165": "nan",
+ "2170": "nan",
+ "2175": "nan",
+ "2180": "nan",
+ "2185": "nan",
+ "2190": "nan",
+ "2195": "nan",
+ "2200": 0.34016,
+ "2205": "nan",
+ "2210": "nan",
+ "2215": "nan",
+ "2220": "nan",
+ "2225": "nan",
+ "2230": "nan",
+ "2235": "nan",
+ "2240": "nan",
+ "2245": "nan",
+ "2250": "nan",
+ "2255": "nan",
+ "2260": "nan",
+ "2265": "nan",
+ "2270": "nan",
+ "2275": "nan",
+ "2280": "nan",
+ "2285": "nan",
+ "2290": "nan",
+ "2295": "nan",
+ "2300": 0.28193,
+ "2305": "nan",
+ "2310": "nan",
+ "2315": "nan",
+ "2320": "nan",
+ "2325": "nan",
+ "2330": "nan",
+ "2335": "nan",
+ "2340": "nan",
+ "2345": "nan",
+ "2350": "nan",
+ "2355": "nan",
+ "2360": "nan",
+ "2365": "nan",
+ "2370": "nan",
+ "2375": "nan",
+ "2380": "nan",
+ "2385": "nan",
+ "2390": "nan",
+ "2395": "nan",
+ "2400": 0.29414,
+ "2405": "nan",
+ "2410": "nan",
+ "2415": "nan",
+ "2420": "nan",
+ "2425": "nan",
+ "2430": "nan",
+ "2435": "nan",
+ "2440": "nan",
+ "2445": "nan",
+ "2450": "nan",
+ "2455": "nan",
+ "2460": "nan",
+ "2465": "nan",
+ "2470": "nan",
+ "2475": "nan",
+ "2480": "nan",
+ "2485": "nan",
+ "2490": "nan",
+ "2495": "nan",
+ "2500": 0.30252,
+ "2505": "nan",
+ "2510": "nan",
+ "2515": "nan",
+ "2520": "nan",
+ "2525": "nan",
+ "2530": "nan",
+ "2535": "nan",
+ "2540": "nan",
+ "2545": "nan",
+ "2550": "nan",
+ "2555": "nan",
+ "2560": "nan",
+ "2565": "nan",
+ "2570": "nan",
+ "2575": "nan",
+ "2580": "nan",
+ "2585": "nan",
+ "2590": "nan",
+ "2595": "nan",
+ "2600": 0.3,
+ "2605": "nan",
+ "2610": "nan",
+ "2615": "nan",
+ "2620": "nan",
+ "2625": "nan",
+ "2630": "nan",
+ "2635": "nan",
+ "2640": "nan",
+ "2645": "nan",
+ "2650": "nan",
+ "2655": "nan",
+ "2660": "nan",
+ "2665": "nan",
+ "2670": "nan",
+ "2675": "nan",
+ "2680": "nan",
+ "2685": "nan",
+ "2690": "nan",
+ "2695": "nan",
+ "2700": 0.37179,
+ "2705": "nan",
+ "2710": "nan",
+ "2715": "nan",
+ "2720": "nan",
+ "2725": "nan",
+ "2730": "nan",
+ "2735": "nan",
+ "2740": "nan",
+ "2745": "nan",
+ "2750": "nan",
+ "2755": "nan",
+ "2760": "nan",
+ "2765": "nan",
+ "2770": "nan",
+ "2775": "nan",
+ "2780": "nan",
+ "2785": "nan",
+ "2790": "nan",
+ "2795": "nan",
+ "2800": 0.29533,
+ "2805": "nan",
+ "2810": "nan",
+ "2815": "nan",
+ "2820": "nan",
+ "2825": "nan",
+ "2830": "nan",
+ "2835": "nan",
+ "2840": "nan",
+ "2845": "nan",
+ "2850": "nan",
+ "2855": "nan",
+ "2860": "nan",
+ "2865": "nan",
+ "2870": "nan",
+ "2875": "nan",
+ "2880": "nan",
+ "2885": "nan",
+ "2890": "nan",
+ "2895": "nan",
+ "2900": 0.3087,
+ "2905": "nan",
+ "2910": "nan",
+ "2915": "nan",
+ "2920": "nan",
+ "2925": "nan",
+ "2930": "nan",
+ "2935": "nan",
+ "2940": "nan",
+ "2945": "nan",
+ "2950": "nan",
+ "2955": "nan",
+ "2960": "nan",
+ "2965": "nan",
+ "2970": "nan",
+ "2975": "nan",
+ "2980": "nan",
+ "2985": "nan",
+ "2990": "nan",
+ "2995": "nan",
+ "3000": 0.29196,
+ "3005": "nan",
+ "3010": "nan",
+ "3015": "nan",
+ "3020": "nan",
+ "3025": "nan",
+ "3030": "nan",
+ "3035": "nan",
+ "3040": "nan",
+ "3045": "nan",
+ "3050": "nan",
+ "3055": "nan",
+ "3060": "nan",
+ "3065": "nan",
+ "3070": "nan",
+ "3075": "nan",
+ "3080": "nan",
+ "3085": "nan",
+ "3090": "nan",
+ "3095": "nan",
+ "3100": 0.28467,
+ "3105": "nan",
+ "3110": "nan",
+ "3115": "nan",
+ "3120": "nan",
+ "3125": "nan",
+ "3130": "nan",
+ "3135": "nan",
+ "3140": "nan",
+ "3145": "nan",
+ "3150": "nan",
+ "3155": "nan",
+ "3160": "nan",
+ "3165": "nan",
+ "3170": "nan",
+ "3175": "nan",
+ "3180": "nan",
+ "3185": "nan",
+ "3190": "nan",
+ "3195": "nan",
+ "3200": 0.33348,
+ "3205": "nan",
+ "3210": "nan",
+ "3215": "nan",
+ "3220": "nan",
+ "3225": "nan",
+ "3230": "nan",
+ "3235": "nan",
+ "3240": "nan",
+ "3245": "nan",
+ "3250": "nan",
+ "3255": "nan",
+ "3260": "nan",
+ "3265": "nan",
+ "3270": "nan",
+ "3275": "nan",
+ "3280": "nan",
+ "3285": "nan",
+ "3290": "nan",
+ "3295": "nan",
+ "3300": 0.36884,
+ "3305": "nan",
+ "3310": "nan",
+ "3315": "nan",
+ "3320": "nan",
+ "3325": "nan",
+ "3330": "nan",
+ "3335": "nan",
+ "3340": "nan",
+ "3345": "nan",
+ "3350": "nan",
+ "3355": "nan",
+ "3360": "nan",
+ "3365": "nan",
+ "3370": "nan",
+ "3375": "nan",
+ "3380": "nan",
+ "3385": "nan",
+ "3390": "nan",
+ "3395": "nan",
+ "3400": 0.29896,
+ "3405": "nan",
+ "3410": "nan",
+ "3415": "nan",
+ "3420": "nan",
+ "3425": "nan",
+ "3430": "nan",
+ "3435": "nan",
+ "3440": "nan",
+ "3445": "nan",
+ "3450": "nan",
+ "3455": "nan",
+ "3460": "nan",
+ "3465": "nan",
+ "3470": "nan",
+ "3475": "nan",
+ "3480": "nan",
+ "3485": "nan",
+ "3490": "nan",
+ "3495": "nan",
+ "3500": 0.2837,
+ "3505": "nan",
+ "3510": "nan",
+ "3515": "nan",
+ "3520": "nan",
+ "3525": "nan",
+ "3530": "nan",
+ "3535": "nan",
+ "3540": "nan",
+ "3545": "nan",
+ "3550": "nan",
+ "3555": "nan",
+ "3560": "nan",
+ "3565": "nan",
+ "3570": "nan",
+ "3575": "nan",
+ "3580": "nan",
+ "3585": "nan",
+ "3590": "nan",
+ "3595": "nan",
+ "3600": 0.28047,
+ "3605": "nan",
+ "3610": "nan",
+ "3615": "nan",
+ "3620": "nan",
+ "3625": "nan",
+ "3630": "nan",
+ "3635": "nan",
+ "3640": "nan",
+ "3645": "nan",
+ "3650": "nan",
+ "3655": "nan",
+ "3660": "nan",
+ "3665": "nan",
+ "3670": "nan",
+ "3675": "nan",
+ "3680": "nan",
+ "3685": "nan",
+ "3690": "nan",
+ "3695": "nan",
+ "3700": 0.30244,
+ "3705": "nan",
+ "3710": "nan",
+ "3715": "nan",
+ "3720": "nan",
+ "3725": "nan",
+ "3730": "nan",
+ "3735": "nan",
+ "3740": "nan",
+ "3745": "nan",
+ "3750": "nan",
+ "3755": "nan",
+ "3760": "nan",
+ "3765": "nan",
+ "3770": "nan",
+ "3775": "nan",
+ "3780": "nan",
+ "3785": "nan",
+ "3790": "nan",
+ "3795": "nan",
+ "3800": 0.37995,
+ "3805": "nan",
+ "3810": "nan",
+ "3815": "nan",
+ "3820": "nan",
+ "3825": "nan",
+ "3830": "nan",
+ "3835": "nan",
+ "3840": "nan",
+ "3845": "nan",
+ "3850": "nan",
+ "3855": "nan",
+ "3860": "nan",
+ "3865": "nan",
+ "3870": "nan",
+ "3875": "nan",
+ "3880": "nan",
+ "3885": "nan",
+ "3890": "nan",
+ "3895": "nan",
+ "3900": 0.47394,
+ "3905": "nan",
+ "3910": "nan",
+ "3915": "nan",
+ "3920": "nan",
+ "3925": "nan",
+ "3930": "nan",
+ "3935": "nan",
+ "3940": "nan",
+ "3945": "nan",
+ "3950": "nan",
+ "3955": "nan",
+ "3960": "nan",
+ "3965": "nan",
+ "3970": "nan",
+ "3975": "nan",
+ "3980": "nan",
+ "3985": "nan",
+ "3990": "nan",
+ "3995": "nan",
+ "4000": 0.45949,
+ "4005": "nan",
+ "4010": "nan",
+ "4015": "nan",
+ "4020": "nan",
+ "4025": "nan",
+ "4030": "nan",
+ "4035": "nan",
+ "4040": "nan",
+ "4045": "nan",
+ "4050": "nan",
+ "4055": "nan",
+ "4060": "nan",
+ "4065": "nan",
+ "4070": "nan",
+ "4075": "nan",
+ "4080": "nan",
+ "4085": "nan",
+ "4090": "nan",
+ "4095": "nan",
+ "4100": 0.39132,
+ "4105": "nan",
+ "4110": "nan",
+ "4115": "nan",
+ "4120": "nan",
+ "4125": "nan",
+ "4130": "nan",
+ "4135": "nan",
+ "4140": "nan",
+ "4145": "nan",
+ "4150": "nan",
+ "4155": "nan",
+ "4160": "nan",
+ "4165": "nan",
+ "4170": "nan",
+ "4175": "nan",
+ "4180": "nan",
+ "4185": "nan",
+ "4190": "nan",
+ "4195": "nan",
+ "4200": 0.27918,
+ "4205": "nan",
+ "4210": "nan",
+ "4215": "nan",
+ "4220": "nan",
+ "4225": "nan",
+ "4230": "nan",
+ "4235": "nan",
+ "4240": "nan",
+ "4245": "nan",
+ "4250": "nan",
+ "4255": "nan",
+ "4260": "nan",
+ "4265": "nan",
+ "4270": "nan",
+ "4275": "nan",
+ "4280": "nan",
+ "4285": "nan",
+ "4290": "nan",
+ "4295": "nan",
+ "4300": 0.28269,
+ "4305": "nan",
+ "4310": "nan",
+ "4315": "nan",
+ "4320": "nan",
+ "4325": "nan",
+ "4330": "nan",
+ "4335": "nan",
+ "4340": "nan",
+ "4345": "nan",
+ "4350": "nan",
+ "4355": "nan",
+ "4360": "nan",
+ "4365": "nan",
+ "4370": "nan",
+ "4375": "nan",
+ "4380": "nan",
+ "4385": "nan",
+ "4390": "nan",
+ "4395": "nan",
+ "4400": 0.30371,
+ "4405": "nan",
+ "4410": "nan",
+ "4415": "nan",
+ "4420": "nan",
+ "4425": "nan",
+ "4430": "nan",
+ "4435": "nan",
+ "4440": "nan",
+ "4445": "nan",
+ "4450": "nan",
+ "4455": "nan",
+ "4460": "nan",
+ "4465": "nan",
+ "4470": "nan",
+ "4475": "nan",
+ "4480": "nan",
+ "4485": "nan",
+ "4490": "nan",
+ "4495": "nan",
+ "4500": 0.30502,
+ "4505": "nan",
+ "4510": "nan",
+ "4515": "nan",
+ "4520": "nan",
+ "4525": "nan",
+ "4530": "nan",
+ "4535": "nan",
+ "4540": "nan",
+ "4545": "nan",
+ "4550": "nan",
+ "4555": "nan",
+ "4560": "nan",
+ "4565": "nan",
+ "4570": "nan",
+ "4575": "nan",
+ "4580": "nan",
+ "4585": "nan",
+ "4590": "nan",
+ "4595": "nan",
+ "4600": 0.36183,
+ "4605": "nan",
+ "4610": "nan",
+ "4615": "nan",
+ "4620": "nan",
+ "4625": "nan",
+ "4630": "nan",
+ "4635": "nan",
+ "4640": "nan",
+ "4645": "nan",
+ "4650": "nan",
+ "4655": "nan",
+ "4660": "nan",
+ "4665": "nan",
+ "4670": "nan",
+ "4675": "nan",
+ "4680": "nan",
+ "4685": "nan",
+ "4690": "nan",
+ "4695": "nan",
+ "4700": 0.32098,
+ "4705": "nan",
+ "4710": "nan",
+ "4715": "nan",
+ "4720": "nan",
+ "4725": "nan",
+ "4730": "nan",
+ "4735": "nan",
+ "4740": "nan",
+ "4745": "nan",
+ "4750": "nan",
+ "4755": "nan",
+ "4760": "nan",
+ "4765": "nan",
+ "4770": "nan",
+ "4775": "nan",
+ "4780": "nan",
+ "4785": "nan",
+ "4790": "nan",
+ "4795": "nan",
+ "4800": 0.30509,
+ "4805": "nan",
+ "4810": "nan",
+ "4815": "nan",
+ "4820": "nan",
+ "4825": "nan",
+ "4830": "nan",
+ "4835": "nan",
+ "4840": "nan",
+ "4845": "nan",
+ "4850": "nan",
+ "4855": "nan",
+ "4860": "nan",
+ "4865": "nan",
+ "4870": "nan",
+ "4875": "nan",
+ "4880": "nan",
+ "4885": "nan",
+ "4890": "nan",
+ "4895": "nan",
+ "4900": 0.30503,
+ "4905": "nan",
+ "4910": "nan",
+ "4915": "nan",
+ "4920": "nan",
+ "4925": "nan",
+ "4930": "nan",
+ "4935": "nan",
+ "4940": "nan",
+ "4945": "nan",
+ "4950": "nan",
+ "4955": "nan",
+ "4960": "nan",
+ "4965": "nan",
+ "4970": "nan",
+ "4975": "nan",
+ "4980": "nan",
+ "4985": "nan",
+ "4990": "nan",
+ "4995": "nan",
+ "5000": 0.28233,
+ "5005": "nan",
+ "5010": "nan",
+ "5015": "nan",
+ "5020": "nan",
+ "5025": "nan",
+ "5030": "nan",
+ "5035": "nan",
+ "5040": "nan",
+ "5045": "nan",
+ "5050": "nan",
+ "5055": "nan",
+ "5060": "nan",
+ "5065": "nan",
+ "5070": "nan",
+ "5075": "nan",
+ "5080": "nan",
+ "5085": "nan",
+ "5090": "nan",
+ "5095": "nan",
+ "5100": 0.2813,
+ "5105": "nan",
+ "5110": "nan",
+ "5115": "nan",
+ "5120": "nan",
+ "5125": "nan",
+ "5130": "nan",
+ "5135": "nan",
+ "5140": "nan",
+ "5145": "nan",
+ "5150": "nan",
+ "5155": "nan",
+ "5160": "nan",
+ "5165": "nan",
+ "5170": "nan",
+ "5175": "nan",
+ "5180": "nan",
+ "5185": "nan",
+ "5190": "nan",
+ "5195": "nan",
+ "5200": 0.40299,
+ "5205": "nan",
+ "5210": "nan",
+ "5215": "nan",
+ "5220": "nan",
+ "5225": "nan",
+ "5230": "nan",
+ "5235": "nan",
+ "5240": "nan",
+ "5245": "nan",
+ "5250": "nan",
+ "5255": "nan",
+ "5260": "nan",
+ "5265": "nan",
+ "5270": "nan",
+ "5275": "nan",
+ "5280": "nan",
+ "5285": "nan",
+ "5290": "nan",
+ "5295": "nan",
+ "5300": 0.31713,
+ "5305": "nan",
+ "5310": "nan",
+ "5315": "nan",
+ "5320": "nan",
+ "5325": "nan",
+ "5330": "nan",
+ "5335": "nan",
+ "5340": "nan",
+ "5345": "nan",
+ "5350": "nan",
+ "5355": "nan",
+ "5360": "nan",
+ "5365": "nan",
+ "5370": "nan",
+ "5375": "nan",
+ "5380": "nan",
+ "5385": "nan",
+ "5390": "nan",
+ "5395": "nan",
+ "5400": 0.284,
+ "5405": "nan",
+ "5410": "nan",
+ "5415": "nan",
+ "5420": "nan",
+ "5425": "nan",
+ "5430": "nan",
+ "5435": "nan",
+ "5440": "nan",
+ "5445": "nan",
+ "5450": "nan",
+ "5455": "nan",
+ "5460": "nan",
+ "5465": "nan",
+ "5470": "nan",
+ "5475": "nan",
+ "5480": "nan",
+ "5485": "nan",
+ "5490": "nan",
+ "5495": "nan",
+ "5500": 0.28358,
+ "5505": "nan",
+ "5510": "nan",
+ "5515": "nan",
+ "5520": "nan",
+ "5525": "nan",
+ "5530": "nan",
+ "5535": "nan",
+ "5540": "nan",
+ "5545": "nan",
+ "5550": "nan",
+ "5555": "nan",
+ "5560": "nan",
+ "5565": "nan",
+ "5570": "nan",
+ "5575": "nan",
+ "5580": "nan",
+ "5585": "nan",
+ "5590": "nan",
+ "5595": "nan",
+ "5600": 0.3057,
+ "5605": "nan",
+ "5610": "nan",
+ "5615": "nan",
+ "5620": "nan",
+ "5625": "nan",
+ "5630": "nan",
+ "5635": "nan",
+ "5640": "nan",
+ "5645": "nan",
+ "5650": "nan",
+ "5655": "nan",
+ "5660": "nan",
+ "5665": "nan",
+ "5670": "nan",
+ "5675": "nan",
+ "5680": "nan",
+ "5685": "nan",
+ "5690": "nan",
+ "5695": "nan",
+ "5700": 0.3616,
+ "5705": "nan",
+ "5710": "nan",
+ "5715": "nan",
+ "5720": "nan",
+ "5725": "nan",
+ "5730": "nan",
+ "5735": "nan",
+ "5740": "nan",
+ "5745": "nan",
+ "5750": "nan",
+ "5755": "nan",
+ "5760": "nan",
+ "5765": "nan",
+ "5770": "nan",
+ "5775": "nan",
+ "5780": "nan",
+ "5785": "nan",
+ "5790": "nan",
+ "5795": "nan",
+ "5800": 0.34267,
+ "5805": "nan",
+ "5810": "nan",
+ "5815": "nan",
+ "5820": "nan",
+ "5825": "nan",
+ "5830": "nan",
+ "5835": "nan",
+ "5840": "nan",
+ "5845": "nan",
+ "5850": "nan",
+ "5855": "nan",
+ "5860": "nan",
+ "5865": "nan",
+ "5870": "nan",
+ "5875": "nan",
+ "5880": "nan",
+ "5885": "nan",
+ "5890": "nan",
+ "5895": "nan",
+ "5900": 0.28387,
+ "5905": "nan",
+ "5910": "nan",
+ "5915": "nan",
+ "5920": "nan",
+ "5925": "nan",
+ "5930": "nan",
+ "5935": "nan",
+ "5940": "nan",
+ "5945": "nan",
+ "5950": "nan",
+ "5955": "nan",
+ "5960": "nan",
+ "5965": "nan",
+ "5970": "nan",
+ "5975": "nan",
+ "5980": "nan",
+ "5985": "nan",
+ "5990": "nan",
+ "5995": "nan",
+ "6000": 0.30908,
+ "6005": "nan",
+ "6010": "nan",
+ "6015": "nan",
+ "6020": "nan",
+ "6025": "nan",
+ "6030": "nan",
+ "6035": "nan",
+ "6040": "nan",
+ "6045": "nan",
+ "6050": "nan",
+ "6055": "nan",
+ "6060": "nan",
+ "6065": "nan",
+ "6070": "nan",
+ "6075": "nan",
+ "6080": "nan",
+ "6085": "nan",
+ "6090": "nan",
+ "6095": "nan",
+ "6100": 0.37056,
+ "6105": "nan",
+ "6110": "nan",
+ "6115": "nan",
+ "6120": "nan",
+ "6125": "nan",
+ "6130": "nan",
+ "6135": "nan",
+ "6140": "nan",
+ "6145": "nan",
+ "6150": "nan",
+ "6155": "nan",
+ "6160": "nan",
+ "6165": "nan",
+ "6170": "nan",
+ "6175": "nan",
+ "6180": "nan",
+ "6185": "nan",
+ "6190": "nan",
+ "6195": "nan",
+ "6200": 0.29676,
+ "6205": "nan",
+ "6210": "nan",
+ "6215": "nan",
+ "6220": "nan",
+ "6225": "nan",
+ "6230": "nan",
+ "6235": "nan",
+ "6240": "nan",
+ "6245": "nan",
+ "6250": "nan",
+ "6255": "nan",
+ "6260": "nan",
+ "6265": "nan",
+ "6270": "nan",
+ "6275": "nan",
+ "6280": "nan",
+ "6285": "nan",
+ "6290": "nan",
+ "6295": "nan",
+ "6300": 0.28095,
+ "6305": "nan",
+ "6310": "nan",
+ "6315": "nan",
+ "6320": "nan",
+ "6325": "nan",
+ "6330": "nan",
+ "6335": "nan",
+ "6340": "nan",
+ "6345": "nan",
+ "6350": "nan",
+ "6355": "nan",
+ "6360": "nan",
+ "6365": "nan",
+ "6370": "nan",
+ "6375": "nan",
+ "6380": "nan",
+ "6385": "nan",
+ "6390": "nan",
+ "6395": "nan",
+ "6400": 0.30156,
+ "6405": "nan",
+ "6410": "nan",
+ "6415": "nan",
+ "6420": "nan",
+ "6425": "nan",
+ "6430": "nan",
+ "6435": "nan",
+ "6440": "nan",
+ "6445": "nan",
+ "6450": "nan",
+ "6455": "nan",
+ "6460": "nan",
+ "6465": "nan",
+ "6470": "nan",
+ "6475": "nan",
+ "6480": "nan",
+ "6485": "nan",
+ "6490": "nan",
+ "6495": "nan",
+ "6500": 0.29416,
+ "6505": "nan",
+ "6510": "nan",
+ "6515": "nan",
+ "6520": "nan",
+ "6525": "nan",
+ "6530": "nan",
+ "6535": "nan",
+ "6540": "nan",
+ "6545": "nan",
+ "6550": "nan",
+ "6555": "nan",
+ "6560": "nan",
+ "6565": "nan",
+ "6570": "nan",
+ "6575": "nan",
+ "6580": "nan",
+ "6585": "nan",
+ "6590": "nan",
+ "6595": "nan",
+ "6600": 0.38742,
+ "6605": "nan",
+ "6610": "nan",
+ "6615": "nan",
+ "6620": "nan",
+ "6625": "nan",
+ "6630": "nan",
+ "6635": "nan",
+ "6640": "nan",
+ "6645": "nan",
+ "6650": "nan",
+ "6655": "nan",
+ "6660": "nan",
+ "6665": "nan",
+ "6670": "nan",
+ "6675": "nan",
+ "6680": "nan",
+ "6685": "nan",
+ "6690": "nan",
+ "6695": "nan",
+ "6700": 0.28391,
+ "6705": "nan",
+ "6710": "nan",
+ "6715": "nan",
+ "6720": "nan",
+ "6725": "nan",
+ "6730": "nan",
+ "6735": "nan",
+ "6740": "nan",
+ "6745": "nan",
+ "6750": "nan",
+ "6755": "nan",
+ "6760": "nan",
+ "6765": "nan",
+ "6770": "nan",
+ "6775": "nan",
+ "6780": "nan",
+ "6785": "nan",
+ "6790": "nan",
+ "6795": "nan",
+ "6800": 0.30487,
+ "6805": "nan",
+ "6810": "nan",
+ "6815": "nan",
+ "6820": "nan",
+ "6825": "nan",
+ "6830": "nan",
+ "6835": "nan",
+ "6840": "nan",
+ "6845": "nan",
+ "6850": "nan",
+ "6855": "nan",
+ "6860": "nan",
+ "6865": "nan",
+ "6870": "nan",
+ "6875": "nan",
+ "6880": "nan",
+ "6885": "nan",
+ "6890": "nan",
+ "6895": "nan",
+ "6900": 0.29208,
+ "6905": "nan",
+ "6910": "nan",
+ "6915": "nan",
+ "6920": "nan",
+ "6925": "nan",
+ "6930": "nan",
+ "6935": "nan",
+ "6940": "nan",
+ "6945": "nan",
+ "6950": "nan",
+ "6955": "nan",
+ "6960": "nan",
+ "6965": "nan",
+ "6970": "nan",
+ "6975": "nan",
+ "6980": "nan",
+ "6985": "nan",
+ "6990": "nan",
+ "6995": "nan",
+ "7000": 0.29311,
+ "7005": "nan",
+ "7010": "nan",
+ "7015": "nan",
+ "7020": "nan",
+ "7025": "nan",
+ "7030": "nan",
+ "7035": "nan",
+ "7040": "nan",
+ "7045": "nan",
+ "7050": "nan",
+ "7055": "nan",
+ "7060": "nan",
+ "7065": "nan",
+ "7070": "nan",
+ "7075": "nan",
+ "7080": "nan",
+ "7085": "nan",
+ "7090": "nan",
+ "7095": "nan",
+ "7100": 0.28388,
+ "7105": "nan",
+ "7110": "nan",
+ "7115": "nan",
+ "7120": "nan",
+ "7125": "nan",
+ "7130": "nan",
+ "7135": "nan",
+ "7140": "nan",
+ "7145": "nan",
+ "7150": "nan",
+ "7155": "nan",
+ "7160": "nan",
+ "7165": "nan",
+ "7170": "nan",
+ "7175": "nan",
+ "7180": "nan",
+ "7185": "nan",
+ "7190": "nan",
+ "7195": "nan",
+ "7200": 0.39837,
+ "7205": "nan",
+ "7210": "nan",
+ "7215": "nan",
+ "7220": "nan",
+ "7225": "nan",
+ "7230": "nan",
+ "7235": "nan",
+ "7240": "nan",
+ "7245": "nan",
+ "7250": "nan",
+ "7255": "nan",
+ "7260": "nan",
+ "7265": "nan",
+ "7270": "nan",
+ "7275": "nan",
+ "7280": "nan",
+ "7285": "nan",
+ "7290": "nan",
+ "7295": "nan",
+ "7300": 0.29632,
+ "7305": "nan",
+ "7310": "nan",
+ "7315": "nan",
+ "7320": "nan",
+ "7325": "nan",
+ "7330": "nan",
+ "7335": "nan",
+ "7340": "nan",
+ "7345": "nan",
+ "7350": "nan",
+ "7355": "nan",
+ "7360": "nan",
+ "7365": "nan",
+ "7370": "nan",
+ "7375": "nan",
+ "7380": "nan",
+ "7385": "nan",
+ "7390": "nan",
+ "7395": "nan",
+ "7400": 0.29124,
+ "7405": "nan",
+ "7410": "nan",
+ "7415": "nan",
+ "7420": "nan",
+ "7425": "nan",
+ "7430": "nan",
+ "7435": "nan",
+ "7440": "nan",
+ "7445": "nan",
+ "7450": "nan",
+ "7455": "nan",
+ "7460": "nan",
+ "7465": "nan",
+ "7470": "nan",
+ "7475": "nan",
+ "7480": "nan",
+ "7485": "nan",
+ "7490": "nan",
+ "7495": "nan",
+ "7500": 0.28501,
+ "7505": "nan",
+ "7510": "nan",
+ "7515": "nan",
+ "7520": "nan",
+ "7525": "nan",
+ "7530": "nan",
+ "7535": "nan",
+ "7540": "nan",
+ "7545": "nan",
+ "7550": "nan",
+ "7555": "nan",
+ "7560": "nan",
+ "7565": "nan",
+ "7570": "nan",
+ "7575": "nan",
+ "7580": "nan",
+ "7585": "nan",
+ "7590": "nan",
+ "7595": "nan",
+ "7600": 0.29897,
+ "7605": "nan",
+ "7610": "nan",
+ "7615": "nan",
+ "7620": "nan",
+ "7625": "nan",
+ "7630": "nan",
+ "7635": "nan",
+ "7640": "nan",
+ "7645": "nan",
+ "7650": "nan",
+ "7655": "nan",
+ "7660": "nan",
+ "7665": "nan",
+ "7670": "nan",
+ "7675": "nan",
+ "7680": "nan",
+ "7685": "nan",
+ "7690": "nan",
+ "7695": "nan",
+ "7700": 0.37889,
+ "7705": "nan",
+ "7710": "nan",
+ "7715": "nan",
+ "7720": "nan",
+ "7725": "nan",
+ "7730": "nan",
+ "7735": "nan",
+ "7740": "nan",
+ "7745": "nan",
+ "7750": "nan",
+ "7755": "nan",
+ "7760": "nan",
+ "7765": "nan",
+ "7770": "nan",
+ "7775": "nan",
+ "7780": "nan",
+ "7785": "nan",
+ "7790": "nan",
+ "7795": "nan",
+ "7800": 0.29964,
+ "7805": "nan",
+ "7810": "nan",
+ "7815": "nan",
+ "7820": "nan",
+ "7825": "nan",
+ "7830": "nan",
+ "7835": "nan",
+ "7840": "nan",
+ "7845": "nan",
+ "7850": "nan",
+ "7855": "nan",
+ "7860": "nan",
+ "7865": "nan",
+ "7870": "nan",
+ "7875": "nan",
+ "7880": "nan",
+ "7885": "nan",
+ "7890": "nan",
+ "7895": "nan",
+ "7900": 0.28896,
+ "7905": "nan",
+ "7910": "nan",
+ "7915": "nan",
+ "7920": "nan",
+ "7925": "nan",
+ "7930": "nan",
+ "7935": "nan",
+ "7940": "nan",
+ "7945": "nan",
+ "7950": "nan",
+ "7955": "nan",
+ "7960": "nan",
+ "7965": "nan",
+ "7970": "nan",
+ "7975": "nan",
+ "7980": "nan",
+ "7985": "nan",
+ "7990": "nan",
+ "7995": "nan",
+ "8000": 0.29143,
+ "8005": "nan",
+ "8010": "nan",
+ "8015": "nan",
+ "8020": "nan",
+ "8025": "nan",
+ "8030": "nan",
+ "8035": "nan",
+ "8040": "nan",
+ "8045": "nan",
+ "8050": "nan",
+ "8055": "nan",
+ "8060": "nan",
+ "8065": "nan",
+ "8070": "nan",
+ "8075": "nan",
+ "8080": "nan",
+ "8085": "nan",
+ "8090": "nan",
+ "8095": "nan",
+ "8100": 0.33636,
+ "8105": "nan",
+ "8110": "nan",
+ "8115": "nan",
+ "8120": "nan",
+ "8125": "nan",
+ "8130": "nan",
+ "8135": "nan",
+ "8140": "nan",
+ "8145": "nan",
+ "8150": "nan",
+ "8155": "nan",
+ "8160": "nan",
+ "8165": "nan",
+ "8170": "nan",
+ "8175": "nan",
+ "8180": "nan",
+ "8185": "nan",
+ "8190": "nan",
+ "8195": "nan",
+ "8200": 0.28154,
+ "8205": "nan",
+ "8210": "nan",
+ "8215": "nan",
+ "8220": "nan",
+ "8225": "nan",
+ "8230": "nan",
+ "8235": "nan",
+ "8240": "nan",
+ "8245": "nan",
+ "8250": "nan",
+ "8255": "nan",
+ "8260": "nan",
+ "8265": "nan",
+ "8270": "nan",
+ "8275": "nan",
+ "8280": "nan",
+ "8285": "nan",
+ "8290": "nan",
+ "8295": "nan",
+ "8300": 0.2875,
+ "8305": "nan",
+ "8310": "nan",
+ "8315": "nan",
+ "8320": "nan",
+ "8325": "nan",
+ "8330": "nan",
+ "8335": "nan",
+ "8340": "nan",
+ "8345": "nan",
+ "8350": "nan",
+ "8355": "nan",
+ "8360": "nan",
+ "8365": "nan",
+ "8370": "nan",
+ "8375": "nan",
+ "8380": "nan",
+ "8385": "nan",
+ "8390": "nan",
+ "8395": "nan",
+ "8400": 0.29238,
+ "8405": "nan",
+ "8410": "nan",
+ "8415": "nan",
+ "8420": "nan",
+ "8425": "nan",
+ "8430": "nan",
+ "8435": "nan",
+ "8440": "nan",
+ "8445": "nan",
+ "8450": "nan",
+ "8455": "nan",
+ "8460": "nan",
+ "8465": "nan",
+ "8470": "nan",
+ "8475": "nan",
+ "8480": "nan",
+ "8485": "nan",
+ "8490": "nan",
+ "8495": "nan",
+ "8500": 0.30846,
+ "8505": "nan",
+ "8510": "nan",
+ "8515": "nan",
+ "8520": "nan",
+ "8525": "nan",
+ "8530": "nan",
+ "8535": "nan",
+ "8540": "nan",
+ "8545": "nan",
+ "8550": "nan",
+ "8555": "nan",
+ "8560": "nan",
+ "8565": "nan",
+ "8570": "nan",
+ "8575": "nan",
+ "8580": "nan",
+ "8585": "nan",
+ "8590": "nan",
+ "8595": "nan",
+ "8600": 0.35999,
+ "8605": "nan",
+ "8610": "nan",
+ "8615": "nan",
+ "8620": "nan",
+ "8625": "nan",
+ "8630": "nan",
+ "8635": "nan",
+ "8640": "nan",
+ "8645": "nan",
+ "8650": "nan",
+ "8655": "nan",
+ "8660": "nan",
+ "8665": "nan",
+ "8670": "nan",
+ "8675": "nan",
+ "8680": "nan",
+ "8685": "nan",
+ "8690": "nan",
+ "8695": "nan",
+ "8700": 0.29321,
+ "8705": "nan",
+ "8710": "nan",
+ "8715": "nan",
+ "8720": "nan",
+ "8725": "nan",
+ "8730": "nan",
+ "8735": "nan",
+ "8740": "nan",
+ "8745": "nan",
+ "8750": "nan",
+ "8755": "nan",
+ "8760": "nan",
+ "8765": "nan",
+ "8770": "nan",
+ "8775": "nan",
+ "8780": "nan",
+ "8785": "nan",
+ "8790": "nan",
+ "8795": "nan",
+ "8800": 0.362,
+ "8805": "nan",
+ "8810": "nan",
+ "8815": "nan",
+ "8820": "nan",
+ "8825": "nan",
+ "8830": "nan",
+ "8835": "nan",
+ "8840": "nan",
+ "8845": "nan",
+ "8850": "nan",
+ "8855": "nan",
+ "8860": "nan",
+ "8865": "nan",
+ "8870": "nan",
+ "8875": "nan",
+ "8880": "nan",
+ "8885": "nan",
+ "8890": "nan",
+ "8895": "nan",
+ "8900": 0.29259,
+ "8905": "nan",
+ "8910": "nan",
+ "8915": "nan",
+ "8920": "nan",
+ "8925": "nan",
+ "8930": "nan",
+ "8935": "nan",
+ "8940": "nan",
+ "8945": "nan",
+ "8950": "nan",
+ "8955": "nan",
+ "8960": "nan",
+ "8965": "nan",
+ "8970": "nan",
+ "8975": "nan",
+ "8980": "nan",
+ "8985": "nan",
+ "8990": "nan",
+ "8995": "nan",
+ "9000": 0.28762,
+ "9005": "nan",
+ "9010": "nan",
+ "9015": "nan",
+ "9020": "nan",
+ "9025": "nan",
+ "9030": "nan",
+ "9035": "nan",
+ "9040": "nan",
+ "9045": "nan",
+ "9050": "nan",
+ "9055": "nan",
+ "9060": "nan",
+ "9065": "nan",
+ "9070": "nan",
+ "9075": "nan",
+ "9080": "nan",
+ "9085": "nan",
+ "9090": "nan",
+ "9095": "nan",
+ "9100": 0.29108,
+ "9105": "nan",
+ "9110": "nan",
+ "9115": "nan",
+ "9120": "nan",
+ "9125": "nan",
+ "9130": "nan",
+ "9135": "nan",
+ "9140": "nan",
+ "9145": "nan",
+ "9150": "nan",
+ "9155": "nan",
+ "9160": "nan",
+ "9165": "nan",
+ "9170": "nan",
+ "9175": "nan",
+ "9180": "nan",
+ "9185": "nan",
+ "9190": "nan",
+ "9195": "nan",
+ "9200": 0.37687,
+ "9205": "nan",
+ "9210": "nan",
+ "9215": "nan",
+ "9220": "nan",
+ "9225": "nan",
+ "9230": "nan",
+ "9235": "nan",
+ "9240": "nan",
+ "9245": "nan",
+ "9250": "nan",
+ "9255": "nan",
+ "9260": "nan",
+ "9265": "nan",
+ "9270": "nan",
+ "9275": "nan",
+ "9280": "nan",
+ "9285": "nan",
+ "9290": "nan",
+ "9295": "nan",
+ "9300": 0.302,
+ "9305": "nan",
+ "9310": "nan",
+ "9315": "nan",
+ "9320": "nan",
+ "9325": "nan",
+ "9330": "nan",
+ "9335": "nan",
+ "9340": "nan",
+ "9345": "nan",
+ "9350": "nan",
+ "9355": "nan",
+ "9360": "nan",
+ "9365": "nan",
+ "9370": "nan",
+ "9375": "nan",
+ "9380": "nan",
+ "9385": "nan",
+ "9390": "nan",
+ "9395": "nan",
+ "9400": 0.28363,
+ "9405": "nan",
+ "9410": "nan",
+ "9415": "nan",
+ "9420": "nan",
+ "9425": "nan",
+ "9430": "nan",
+ "9435": "nan",
+ "9440": "nan",
+ "9445": "nan",
+ "9450": "nan",
+ "9455": "nan",
+ "9460": "nan",
+ "9465": "nan",
+ "9470": "nan",
+ "9475": "nan",
+ "9480": "nan",
+ "9485": "nan",
+ "9490": "nan",
+ "9495": "nan",
+ "9500": 0.28919,
+ "9505": "nan",
+ "9510": "nan",
+ "9515": "nan",
+ "9520": "nan",
+ "9525": "nan",
+ "9530": "nan",
+ "9535": "nan",
+ "9540": "nan",
+ "9545": "nan",
+ "9550": "nan",
+ "9555": "nan",
+ "9560": "nan",
+ "9565": "nan",
+ "9570": "nan",
+ "9575": "nan",
+ "9580": "nan",
+ "9585": "nan",
+ "9590": "nan",
+ "9595": "nan",
+ "9600": 0.30407,
+ "9605": "nan",
+ "9610": "nan",
+ "9615": "nan",
+ "9620": "nan",
+ "9625": "nan",
+ "9630": "nan",
+ "9635": "nan",
+ "9640": "nan",
+ "9645": "nan",
+ "9650": "nan",
+ "9655": "nan",
+ "9660": "nan",
+ "9665": "nan",
+ "9670": "nan",
+ "9675": "nan",
+ "9680": "nan",
+ "9685": "nan",
+ "9690": "nan",
+ "9695": "nan",
+ "9700": 0.36124,
+ "9705": "nan",
+ "9710": "nan",
+ "9715": "nan",
+ "9720": "nan",
+ "9725": "nan",
+ "9730": "nan",
+ "9735": "nan",
+ "9740": "nan",
+ "9745": "nan",
+ "9750": "nan",
+ "9755": "nan",
+ "9760": "nan",
+ "9765": "nan",
+ "9770": "nan",
+ "9775": "nan",
+ "9780": "nan",
+ "9785": "nan",
+ "9790": "nan",
+ "9795": "nan",
+ "9800": 0.28658,
+ "9805": "nan",
+ "9810": "nan",
+ "9815": "nan",
+ "9820": "nan",
+ "9825": "nan",
+ "9830": "nan",
+ "9835": "nan",
+ "9840": "nan",
+ "9845": "nan",
+ "9850": "nan",
+ "9855": "nan",
+ "9860": "nan",
+ "9865": "nan",
+ "9870": "nan",
+ "9875": "nan",
+ "9880": "nan",
+ "9885": "nan",
+ "9890": "nan",
+ "9895": "nan",
+ "9900": 0.2893,
+ "9905": "nan",
+ "9910": "nan",
+ "9915": "nan",
+ "9920": "nan",
+ "9925": "nan",
+ "9930": "nan",
+ "9935": "nan",
+ "9940": "nan",
+ "9945": "nan",
+ "9950": "nan",
+ "9955": "nan",
+ "9960": "nan",
+ "9965": "nan",
+ "9970": "nan",
+ "9975": "nan",
+ "9980": "nan",
+ "9985": "nan",
+ "9990": "nan",
+ "9995": "nan",
+ "10000": 0.29697
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/t5/t5_release_sm/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/t5/t5_release_sm/golden_values_dev_dgx_h100.json
new file mode 100644
index 00000000000..94e5a1b1e47
--- /dev/null
+++ b/tests/functional_tests/test_cases/t5/t5_release_sm/golden_values_dev_dgx_h100.json
@@ -0,0 +1,10042 @@
+{
+ "lm loss": {
+ "start_step": 1,
+ "end_step": 10000,
+ "step_interval": 5,
+ "values": {
+ "1": 10.35099,
+ "5": 10.3436,
+ "10": 10.26416,
+ "15": 9.87885,
+ "20": 9.66512,
+ "25": 9.53844,
+ "30": 9.42539,
+ "35": 9.35299,
+ "40": 9.26966,
+ "45": 9.19156,
+ "50": 9.14193,
+ "55": 9.07697,
+ "60": 8.99378,
+ "65": 8.91984,
+ "70": 8.91932,
+ "75": 8.83179,
+ "80": 8.8355,
+ "85": 8.75482,
+ "90": 8.74895,
+ "95": 8.70083,
+ "100": 8.64718,
+ "105": 8.60014,
+ "110": 8.53088,
+ "115": 8.4881,
+ "120": 8.47064,
+ "125": 8.41542,
+ "130": 8.38274,
+ "135": 8.30817,
+ "140": 8.36998,
+ "145": 8.25486,
+ "150": 8.27637,
+ "155": 8.16245,
+ "160": 8.15625,
+ "165": 8.10985,
+ "170": 8.09773,
+ "175": 8.05865,
+ "180": 7.92768,
+ "185": 7.91717,
+ "190": 7.85148,
+ "195": 7.78971,
+ "200": 7.73666,
+ "205": 7.69571,
+ "210": 7.63986,
+ "215": 7.59606,
+ "220": 7.53309,
+ "225": 7.46291,
+ "230": 7.39309,
+ "235": 7.3628,
+ "240": 7.27791,
+ "245": 7.21834,
+ "250": 7.16482,
+ "255": 7.14221,
+ "260": 7.02847,
+ "265": 6.99788,
+ "270": 6.94442,
+ "275": 6.86106,
+ "280": 6.83388,
+ "285": 6.77036,
+ "290": 6.76952,
+ "295": 6.66727,
+ "300": 6.65503,
+ "305": 6.59312,
+ "310": 6.55051,
+ "315": 6.47276,
+ "320": 6.51845,
+ "325": 6.49574,
+ "330": 6.45443,
+ "335": 6.42204,
+ "340": 6.4226,
+ "345": 6.38149,
+ "350": 6.38269,
+ "355": 6.35305,
+ "360": 6.35159,
+ "365": 6.32742,
+ "370": 6.285,
+ "375": 6.27276,
+ "380": 6.23568,
+ "385": 6.24416,
+ "390": 6.2061,
+ "395": 6.18509,
+ "400": 6.23899,
+ "405": 6.15373,
+ "410": 6.23344,
+ "415": 6.1751,
+ "420": 6.13976,
+ "425": 6.08718,
+ "430": 6.07892,
+ "435": 6.0838,
+ "440": 6.05594,
+ "445": 6.0122,
+ "450": 6.09269,
+ "455": 5.9882,
+ "460": 6.02768,
+ "465": 6.02262,
+ "470": 5.97366,
+ "475": 5.95789,
+ "480": 5.97156,
+ "485": 5.94773,
+ "490": 5.90164,
+ "495": 5.86235,
+ "500": 5.88606,
+ "505": 5.89065,
+ "510": 5.89482,
+ "515": 5.80876,
+ "520": 5.85268,
+ "525": 5.85635,
+ "530": 5.83477,
+ "535": 5.82981,
+ "540": 5.82739,
+ "545": 5.78486,
+ "550": 5.78029,
+ "555": 5.76842,
+ "560": 5.75787,
+ "565": 5.7342,
+ "570": 5.7333,
+ "575": 5.71634,
+ "580": 5.66126,
+ "585": 5.68468,
+ "590": 5.63446,
+ "595": 5.63042,
+ "600": 5.64453,
+ "605": 5.63635,
+ "610": 5.64733,
+ "615": 5.60878,
+ "620": 5.65422,
+ "625": 5.62356,
+ "630": 5.57332,
+ "635": 5.61567,
+ "640": 5.56956,
+ "645": 5.59338,
+ "650": 5.55672,
+ "655": 5.54621,
+ "660": 5.53699,
+ "665": 5.57034,
+ "670": 5.47882,
+ "675": 5.50722,
+ "680": 5.51393,
+ "685": 5.504,
+ "690": 5.51532,
+ "695": 5.48928,
+ "700": 5.44083,
+ "705": 5.38536,
+ "710": 5.41321,
+ "715": 5.45527,
+ "720": 5.44277,
+ "725": 5.44204,
+ "730": 5.42025,
+ "735": 5.42385,
+ "740": 5.39155,
+ "745": 5.42565,
+ "750": 5.45592,
+ "755": 5.42182,
+ "760": 5.39472,
+ "765": 5.36996,
+ "770": 5.3651,
+ "775": 5.34481,
+ "780": 5.32203,
+ "785": 5.31408,
+ "790": 5.30604,
+ "795": 5.31479,
+ "800": 5.33114,
+ "805": 5.34552,
+ "810": 5.3079,
+ "815": 5.27005,
+ "820": 5.28282,
+ "825": 5.29662,
+ "830": 5.28481,
+ "835": 5.29818,
+ "840": 5.30617,
+ "845": 5.23375,
+ "850": 5.23725,
+ "855": 5.25885,
+ "860": 5.25899,
+ "865": 5.25074,
+ "870": 5.22787,
+ "875": 5.1974,
+ "880": 5.29179,
+ "885": 5.2625,
+ "890": 5.20162,
+ "895": 5.21595,
+ "900": 5.23748,
+ "905": 5.22581,
+ "910": 5.175,
+ "915": 5.20921,
+ "920": 5.17816,
+ "925": 5.19434,
+ "930": 5.18391,
+ "935": 5.16541,
+ "940": 5.16234,
+ "945": 5.19397,
+ "950": 5.13851,
+ "955": 5.0948,
+ "960": 5.13068,
+ "965": 5.0896,
+ "970": 5.06125,
+ "975": 5.11037,
+ "980": 5.09537,
+ "985": 5.06481,
+ "990": 5.07821,
+ "995": 5.0925,
+ "1000": 5.09312,
+ "1005": 5.09847,
+ "1010": 5.09649,
+ "1015": 5.06155,
+ "1020": 5.10133,
+ "1025": 5.04975,
+ "1030": 4.99371,
+ "1035": 5.07442,
+ "1040": 5.02632,
+ "1045": 5.05223,
+ "1050": 5.08019,
+ "1055": 5.0281,
+ "1060": 5.0752,
+ "1065": 4.99586,
+ "1070": 4.99149,
+ "1075": 4.99427,
+ "1080": 4.95957,
+ "1085": 5.01361,
+ "1090": 5.0299,
+ "1095": 5.02597,
+ "1100": 5.00552,
+ "1105": 4.99028,
+ "1110": 4.93743,
+ "1115": 4.93444,
+ "1120": 4.94648,
+ "1125": 4.89532,
+ "1130": 5.05492,
+ "1135": 4.92854,
+ "1140": 4.96005,
+ "1145": 4.90974,
+ "1150": 4.98478,
+ "1155": 4.91755,
+ "1160": 4.9053,
+ "1165": 4.91359,
+ "1170": 4.99305,
+ "1175": 4.95252,
+ "1180": 4.83983,
+ "1185": 4.93036,
+ "1190": 4.88168,
+ "1195": 4.92987,
+ "1200": 5.0214,
+ "1205": 4.98086,
+ "1210": 4.84283,
+ "1215": 4.88571,
+ "1220": 4.90423,
+ "1225": 4.88604,
+ "1230": 4.90307,
+ "1235": 4.90075,
+ "1240": 4.86675,
+ "1245": 4.88932,
+ "1250": 4.8321,
+ "1255": 4.88328,
+ "1260": 4.80336,
+ "1265": 4.88002,
+ "1270": 4.95227,
+ "1275": 4.87101,
+ "1280": 4.82874,
+ "1285": 4.91706,
+ "1290": 4.72442,
+ "1295": 4.83116,
+ "1300": 4.85118,
+ "1305": 4.82554,
+ "1310": 4.92081,
+ "1315": 4.79985,
+ "1320": 4.88143,
+ "1325": 4.77556,
+ "1330": 4.82122,
+ "1335": 4.80464,
+ "1340": 4.81511,
+ "1345": 4.82621,
+ "1350": 4.82697,
+ "1355": 4.79788,
+ "1360": 4.74053,
+ "1365": 4.85238,
+ "1370": 4.82647,
+ "1375": 4.84089,
+ "1380": 4.76187,
+ "1385": 4.74103,
+ "1390": 4.80249,
+ "1395": 4.80806,
+ "1400": 4.672,
+ "1405": 4.68244,
+ "1410": 4.76132,
+ "1415": 4.79833,
+ "1420": 4.76454,
+ "1425": 4.73602,
+ "1430": 4.76536,
+ "1435": 4.77699,
+ "1440": 4.76147,
+ "1445": 4.75154,
+ "1450": 4.72207,
+ "1455": 4.66141,
+ "1460": 4.76796,
+ "1465": 4.75387,
+ "1470": 4.70537,
+ "1475": 4.71697,
+ "1480": 4.72414,
+ "1485": 4.75747,
+ "1490": 4.76354,
+ "1495": 4.73035,
+ "1500": 4.76646,
+ "1505": 4.69063,
+ "1510": 4.71079,
+ "1515": 4.71211,
+ "1520": 4.76254,
+ "1525": 4.74739,
+ "1530": 4.68446,
+ "1535": 4.69399,
+ "1540": 4.7187,
+ "1545": 4.60626,
+ "1550": 4.64026,
+ "1555": 4.61861,
+ "1560": 4.70607,
+ "1565": 4.63612,
+ "1570": 4.66897,
+ "1575": 4.67503,
+ "1580": 4.66896,
+ "1585": 4.69286,
+ "1590": 4.6783,
+ "1595": 4.72607,
+ "1600": 4.69503,
+ "1605": 4.68102,
+ "1610": 4.61364,
+ "1615": 4.62838,
+ "1620": 4.70745,
+ "1625": 4.66524,
+ "1630": 4.60996,
+ "1635": 4.61055,
+ "1640": 4.69528,
+ "1645": 4.65645,
+ "1650": 4.66104,
+ "1655": 4.62383,
+ "1660": 4.69289,
+ "1665": 4.62648,
+ "1670": 4.70558,
+ "1675": 4.65207,
+ "1680": 4.59931,
+ "1685": 4.65381,
+ "1690": 4.64316,
+ "1695": 4.67082,
+ "1700": 4.65353,
+ "1705": 4.63914,
+ "1710": 4.6471,
+ "1715": 4.69655,
+ "1720": 4.64795,
+ "1725": 4.66897,
+ "1730": 4.64092,
+ "1735": 4.62773,
+ "1740": 4.61143,
+ "1745": 4.63498,
+ "1750": 4.61628,
+ "1755": 4.64613,
+ "1760": 4.58796,
+ "1765": 4.61954,
+ "1770": 4.61415,
+ "1775": 4.59366,
+ "1780": 4.60518,
+ "1785": 4.61534,
+ "1790": 4.51495,
+ "1795": 4.58016,
+ "1800": 4.60106,
+ "1805": 4.56181,
+ "1810": 4.56997,
+ "1815": 4.58289,
+ "1820": 4.60202,
+ "1825": 4.62582,
+ "1830": 4.5757,
+ "1835": 4.56536,
+ "1840": 4.4921,
+ "1845": 4.52613,
+ "1850": 4.6401,
+ "1855": 4.58938,
+ "1860": 4.56582,
+ "1865": 4.55155,
+ "1870": 4.57635,
+ "1875": 4.55854,
+ "1880": 4.58827,
+ "1885": 4.51386,
+ "1890": 4.50892,
+ "1895": 4.63655,
+ "1900": 4.48615,
+ "1905": 4.58282,
+ "1910": 4.59372,
+ "1915": 4.57399,
+ "1920": 4.55503,
+ "1925": 4.52936,
+ "1930": 4.56294,
+ "1935": 4.53376,
+ "1940": 4.51019,
+ "1945": 4.54051,
+ "1950": 4.50814,
+ "1955": 4.55756,
+ "1960": 4.53892,
+ "1965": 4.56346,
+ "1970": 4.52273,
+ "1975": 4.55359,
+ "1980": 4.50517,
+ "1985": 4.56969,
+ "1990": 4.47059,
+ "1995": 4.5668,
+ "2000": 4.55073,
+ "2005": 4.52713,
+ "2010": 4.53608,
+ "2015": 4.55755,
+ "2020": 4.484,
+ "2025": 4.48665,
+ "2030": 4.56955,
+ "2035": 4.52654,
+ "2040": 4.58349,
+ "2045": 4.44341,
+ "2050": 4.51921,
+ "2055": 4.51852,
+ "2060": 4.48778,
+ "2065": 4.54083,
+ "2070": 4.48406,
+ "2075": 4.55541,
+ "2080": 4.54549,
+ "2085": 4.49113,
+ "2090": 4.4607,
+ "2095": 4.42164,
+ "2100": 4.3972,
+ "2105": 4.43128,
+ "2110": 4.55385,
+ "2115": 4.54446,
+ "2120": 4.55788,
+ "2125": 4.47358,
+ "2130": 4.47601,
+ "2135": 4.49398,
+ "2140": 4.49407,
+ "2145": 4.43832,
+ "2150": 4.47517,
+ "2155": 4.44985,
+ "2160": 4.4143,
+ "2165": 4.54419,
+ "2170": 4.46302,
+ "2175": 4.5083,
+ "2180": 4.49196,
+ "2185": 4.40691,
+ "2190": 4.45089,
+ "2195": 4.48373,
+ "2200": 4.45135,
+ "2205": 4.3742,
+ "2210": 4.46397,
+ "2215": 4.44457,
+ "2220": 4.43303,
+ "2225": 4.417,
+ "2230": 4.42003,
+ "2235": 4.42288,
+ "2240": 4.4188,
+ "2245": 4.46464,
+ "2250": 4.42122,
+ "2255": 4.48897,
+ "2260": 4.45285,
+ "2265": 4.40447,
+ "2270": 4.39466,
+ "2275": 4.3804,
+ "2280": 4.46627,
+ "2285": 4.44014,
+ "2290": 4.43737,
+ "2295": 4.46737,
+ "2300": 4.45559,
+ "2305": 4.43233,
+ "2310": 4.39165,
+ "2315": 4.40744,
+ "2320": 4.3242,
+ "2325": 4.42584,
+ "2330": 4.43384,
+ "2335": 4.4336,
+ "2340": 4.41415,
+ "2345": 4.43936,
+ "2350": 4.33914,
+ "2355": 4.42752,
+ "2360": 4.45373,
+ "2365": 4.39588,
+ "2370": 4.36899,
+ "2375": 4.4467,
+ "2380": 4.4235,
+ "2385": 4.32849,
+ "2390": 4.39556,
+ "2395": 4.35461,
+ "2400": 4.3885,
+ "2405": 4.42229,
+ "2410": 4.33993,
+ "2415": 4.4779,
+ "2420": 4.42371,
+ "2425": 4.37481,
+ "2430": 4.44409,
+ "2435": 4.33689,
+ "2440": 4.40075,
+ "2445": 4.38926,
+ "2450": 4.3854,
+ "2455": 4.44724,
+ "2460": 4.37814,
+ "2465": 4.39853,
+ "2470": 4.39379,
+ "2475": 4.35918,
+ "2480": 4.42654,
+ "2485": 4.43062,
+ "2490": 4.32691,
+ "2495": 4.34485,
+ "2500": 4.3982,
+ "2505": 4.37206,
+ "2510": 4.33168,
+ "2515": 4.3554,
+ "2520": 4.41127,
+ "2525": 4.31352,
+ "2530": 4.36752,
+ "2535": 4.41979,
+ "2540": 4.35307,
+ "2545": 4.32644,
+ "2550": 4.38046,
+ "2555": 4.37035,
+ "2560": 4.38792,
+ "2565": 4.35609,
+ "2570": 4.40943,
+ "2575": 4.365,
+ "2580": 4.364,
+ "2585": 4.35866,
+ "2590": 4.38779,
+ "2595": 4.29822,
+ "2600": 4.34972,
+ "2605": 4.40785,
+ "2610": 4.34759,
+ "2615": 4.32107,
+ "2620": 4.39719,
+ "2625": 4.40171,
+ "2630": 4.34743,
+ "2635": 4.38139,
+ "2640": 4.36542,
+ "2645": 4.35907,
+ "2650": 4.33986,
+ "2655": 4.39717,
+ "2660": 4.30578,
+ "2665": 4.24524,
+ "2670": 4.34558,
+ "2675": 4.26804,
+ "2680": 4.31136,
+ "2685": 4.29067,
+ "2690": 4.30571,
+ "2695": 4.25172,
+ "2700": 4.31205,
+ "2705": 4.2818,
+ "2710": 4.28392,
+ "2715": 4.26854,
+ "2720": 4.32149,
+ "2725": 4.30462,
+ "2730": 4.26368,
+ "2735": 4.26661,
+ "2740": 4.37374,
+ "2745": 4.26125,
+ "2750": 4.29344,
+ "2755": 4.39107,
+ "2760": 4.35497,
+ "2765": 4.34986,
+ "2770": 4.30432,
+ "2775": 4.29965,
+ "2780": 4.32415,
+ "2785": 4.25361,
+ "2790": 4.33802,
+ "2795": 4.32879,
+ "2800": 4.33641,
+ "2805": 4.20193,
+ "2810": 4.25811,
+ "2815": 4.27512,
+ "2820": 4.31154,
+ "2825": 4.24795,
+ "2830": 4.31088,
+ "2835": 4.35118,
+ "2840": 4.29572,
+ "2845": 4.2861,
+ "2850": 4.27766,
+ "2855": 4.30747,
+ "2860": 4.33706,
+ "2865": 4.27348,
+ "2870": 4.2694,
+ "2875": 4.25889,
+ "2880": 4.25856,
+ "2885": 4.23927,
+ "2890": 4.24009,
+ "2895": 4.32037,
+ "2900": 4.25083,
+ "2905": 4.31866,
+ "2910": 4.23328,
+ "2915": 4.26491,
+ "2920": 4.25802,
+ "2925": 4.29061,
+ "2930": 4.24641,
+ "2935": 4.2858,
+ "2940": 4.29438,
+ "2945": 4.25016,
+ "2950": 4.20414,
+ "2955": 4.27722,
+ "2960": 4.18829,
+ "2965": 4.27551,
+ "2970": 4.27838,
+ "2975": 4.26903,
+ "2980": 4.23002,
+ "2985": 4.30052,
+ "2990": 4.21766,
+ "2995": 4.31721,
+ "3000": 4.18706,
+ "3005": 4.20562,
+ "3010": 4.28798,
+ "3015": 4.21546,
+ "3020": 4.23987,
+ "3025": 4.22778,
+ "3030": 4.21669,
+ "3035": 4.22512,
+ "3040": 4.22157,
+ "3045": 4.20901,
+ "3050": 4.2794,
+ "3055": 4.22791,
+ "3060": 4.21788,
+ "3065": 4.26441,
+ "3070": 4.21264,
+ "3075": 4.23899,
+ "3080": 4.17197,
+ "3085": 4.23394,
+ "3090": 4.19444,
+ "3095": 4.25219,
+ "3100": 4.29572,
+ "3105": 4.20165,
+ "3110": 4.20676,
+ "3115": 4.20654,
+ "3120": 4.27333,
+ "3125": 4.17787,
+ "3130": 4.21268,
+ "3135": 4.20154,
+ "3140": 4.22905,
+ "3145": 4.19686,
+ "3150": 4.25091,
+ "3155": 4.17797,
+ "3160": 4.24123,
+ "3165": 4.22445,
+ "3170": 4.16214,
+ "3175": 4.27084,
+ "3180": 4.16412,
+ "3185": 4.15071,
+ "3190": 4.20069,
+ "3195": 4.18747,
+ "3200": 4.21944,
+ "3205": 4.19076,
+ "3210": 4.19077,
+ "3215": 4.21687,
+ "3220": 4.16926,
+ "3225": 4.21991,
+ "3230": 4.19539,
+ "3235": 4.2267,
+ "3240": 4.15419,
+ "3245": 4.18927,
+ "3250": 4.25709,
+ "3255": 4.2296,
+ "3260": 4.1804,
+ "3265": 4.11143,
+ "3270": 4.19037,
+ "3275": 4.2043,
+ "3280": 4.22604,
+ "3285": 4.17795,
+ "3290": 4.22637,
+ "3295": 4.2708,
+ "3300": 4.2607,
+ "3305": 4.18357,
+ "3310": 4.2323,
+ "3315": 4.17617,
+ "3320": 4.23119,
+ "3325": 4.21792,
+ "3330": 4.21806,
+ "3335": 4.13101,
+ "3340": 4.22588,
+ "3345": 4.18736,
+ "3350": 4.21971,
+ "3355": 4.15337,
+ "3360": 4.12902,
+ "3365": 4.16006,
+ "3370": 4.14717,
+ "3375": 4.20985,
+ "3380": 4.15898,
+ "3385": 4.14337,
+ "3390": 4.14965,
+ "3395": 4.16708,
+ "3400": 4.13717,
+ "3405": 4.22878,
+ "3410": 4.13696,
+ "3415": 4.18529,
+ "3420": 4.12618,
+ "3425": 4.14208,
+ "3430": 4.15287,
+ "3435": 4.14748,
+ "3440": 4.20349,
+ "3445": 4.1926,
+ "3450": 4.20916,
+ "3455": 4.11284,
+ "3460": 4.12562,
+ "3465": 4.14916,
+ "3470": 4.1233,
+ "3475": 4.1662,
+ "3480": 4.1977,
+ "3485": 4.11922,
+ "3490": 4.21159,
+ "3495": 4.14066,
+ "3500": 4.11485,
+ "3505": 4.08898,
+ "3510": 4.1531,
+ "3515": 4.17981,
+ "3520": 4.12348,
+ "3525": 4.17471,
+ "3530": 4.21156,
+ "3535": 4.20444,
+ "3540": 4.1068,
+ "3545": 4.09356,
+ "3550": 4.15572,
+ "3555": 4.04826,
+ "3560": 4.07607,
+ "3565": 4.14548,
+ "3570": 4.15996,
+ "3575": 4.18942,
+ "3580": 4.10473,
+ "3585": 4.09337,
+ "3590": 4.1025,
+ "3595": 4.20653,
+ "3600": 4.13436,
+ "3605": 4.07382,
+ "3610": 4.13554,
+ "3615": 4.13738,
+ "3620": 4.14068,
+ "3625": 4.12298,
+ "3630": 4.03485,
+ "3635": 4.07717,
+ "3640": 4.10428,
+ "3645": 4.1471,
+ "3650": 4.08923,
+ "3655": 4.14025,
+ "3660": 4.06225,
+ "3665": 4.20677,
+ "3670": 4.157,
+ "3675": 4.15059,
+ "3680": 4.10657,
+ "3685": 4.09146,
+ "3690": 4.04171,
+ "3695": 4.10254,
+ "3700": 4.10906,
+ "3705": 4.10046,
+ "3710": 4.1273,
+ "3715": 4.09584,
+ "3720": 4.02522,
+ "3725": 4.14066,
+ "3730": 4.10751,
+ "3735": 4.10226,
+ "3740": 4.06307,
+ "3745": 4.04998,
+ "3750": 4.05655,
+ "3755": 3.96523,
+ "3760": 4.09974,
+ "3765": 4.08586,
+ "3770": 4.04746,
+ "3775": 4.06603,
+ "3780": 4.08909,
+ "3785": 4.00803,
+ "3790": 4.12136,
+ "3795": 4.12785,
+ "3800": 4.10331,
+ "3805": 4.07782,
+ "3810": 4.09866,
+ "3815": 4.09069,
+ "3820": 4.12218,
+ "3825": 4.03991,
+ "3830": 4.00034,
+ "3835": 4.09896,
+ "3840": 4.04966,
+ "3845": 4.09191,
+ "3850": 4.04503,
+ "3855": 4.02992,
+ "3860": 4.11681,
+ "3865": 4.08135,
+ "3870": 4.15008,
+ "3875": 4.05,
+ "3880": 4.11833,
+ "3885": 4.05003,
+ "3890": 4.0843,
+ "3895": 4.08893,
+ "3900": 4.09158,
+ "3905": 4.0653,
+ "3910": 4.12092,
+ "3915": 4.03851,
+ "3920": 4.07254,
+ "3925": 4.03294,
+ "3930": 4.07532,
+ "3935": 4.0601,
+ "3940": 4.12074,
+ "3945": 4.09458,
+ "3950": 4.06254,
+ "3955": 4.05993,
+ "3960": 4.0317,
+ "3965": 4.08048,
+ "3970": 4.0039,
+ "3975": 4.08001,
+ "3980": 4.00378,
+ "3985": 4.0992,
+ "3990": 4.0716,
+ "3995": 4.11362,
+ "4000": 4.03459,
+ "4005": 4.08811,
+ "4010": 4.04231,
+ "4015": 4.10767,
+ "4020": 4.09783,
+ "4025": 4.04823,
+ "4030": 4.06646,
+ "4035": 4.04298,
+ "4040": 4.05889,
+ "4045": 4.09779,
+ "4050": 4.13996,
+ "4055": 4.02546,
+ "4060": 4.03371,
+ "4065": 4.07116,
+ "4070": 4.06127,
+ "4075": 4.04657,
+ "4080": 3.96922,
+ "4085": 4.03884,
+ "4090": 4.09437,
+ "4095": 4.00697,
+ "4100": 4.04377,
+ "4105": 4.03881,
+ "4110": 4.02073,
+ "4115": 4.06693,
+ "4120": 4.06644,
+ "4125": 4.00379,
+ "4130": 4.0255,
+ "4135": 3.97634,
+ "4140": 4.06792,
+ "4145": 4.03542,
+ "4150": 3.98143,
+ "4155": 4.05185,
+ "4160": 3.97293,
+ "4165": 4.10035,
+ "4170": 4.07069,
+ "4175": 4.06145,
+ "4180": 4.09166,
+ "4185": 3.99495,
+ "4190": 4.00325,
+ "4195": 3.98018,
+ "4200": 4.08171,
+ "4205": 4.0369,
+ "4210": 4.00809,
+ "4215": 4.05927,
+ "4220": 4.06498,
+ "4225": 4.00336,
+ "4230": 4.03881,
+ "4235": 4.04819,
+ "4240": 3.96168,
+ "4245": 3.95712,
+ "4250": 4.00268,
+ "4255": 3.95704,
+ "4260": 3.98498,
+ "4265": 3.99519,
+ "4270": 3.96673,
+ "4275": 3.96271,
+ "4280": 4.04125,
+ "4285": 3.96884,
+ "4290": 3.94247,
+ "4295": 4.04466,
+ "4300": 3.93492,
+ "4305": 4.00695,
+ "4310": 4.06278,
+ "4315": 3.99286,
+ "4320": 4.04294,
+ "4325": 3.91276,
+ "4330": 4.00286,
+ "4335": 3.96711,
+ "4340": 3.99578,
+ "4345": 3.97972,
+ "4350": 4.00825,
+ "4355": 3.96265,
+ "4360": 3.95453,
+ "4365": 3.95367,
+ "4370": 3.98363,
+ "4375": 4.01055,
+ "4380": 3.99223,
+ "4385": 3.99996,
+ "4390": 4.00954,
+ "4395": 4.01005,
+ "4400": 4.00343,
+ "4405": 3.91442,
+ "4410": 3.96763,
+ "4415": 3.96038,
+ "4420": 3.98906,
+ "4425": 4.02776,
+ "4430": 3.91949,
+ "4435": 3.98254,
+ "4440": 3.94739,
+ "4445": 3.95384,
+ "4450": 3.89698,
+ "4455": 3.99311,
+ "4460": 3.88133,
+ "4465": 3.97858,
+ "4470": 3.95925,
+ "4475": 3.953,
+ "4480": 4.01781,
+ "4485": 3.91081,
+ "4490": 3.88833,
+ "4495": 3.92838,
+ "4500": 3.93513,
+ "4505": 4.01178,
+ "4510": 4.04171,
+ "4515": 3.96458,
+ "4520": 3.97607,
+ "4525": 3.89217,
+ "4530": 4.00963,
+ "4535": 3.93952,
+ "4540": 3.93878,
+ "4545": 3.91161,
+ "4550": 3.94636,
+ "4555": 3.97066,
+ "4560": 3.95307,
+ "4565": 3.94858,
+ "4570": 3.92087,
+ "4575": 3.90498,
+ "4580": 3.90106,
+ "4585": 3.99776,
+ "4590": 3.87143,
+ "4595": 3.9828,
+ "4600": 3.93091,
+ "4605": 3.9087,
+ "4610": 3.87772,
+ "4615": 3.91559,
+ "4620": 3.94329,
+ "4625": 3.9474,
+ "4630": 3.96836,
+ "4635": 3.92738,
+ "4640": 3.98432,
+ "4645": 3.96016,
+ "4650": 3.93422,
+ "4655": 3.95132,
+ "4660": 3.95536,
+ "4665": 3.8921,
+ "4670": 3.9188,
+ "4675": 3.91799,
+ "4680": 3.98581,
+ "4685": 3.89826,
+ "4690": 3.88573,
+ "4695": 3.92837,
+ "4700": 3.92109,
+ "4705": 3.89853,
+ "4710": 3.90372,
+ "4715": 3.91673,
+ "4720": 3.94907,
+ "4725": 3.9196,
+ "4730": 3.92298,
+ "4735": 3.91303,
+ "4740": 3.87995,
+ "4745": 3.89853,
+ "4750": 3.92622,
+ "4755": 3.94787,
+ "4760": 3.88078,
+ "4765": 3.84868,
+ "4770": 3.92419,
+ "4775": 3.92225,
+ "4780": 3.87961,
+ "4785": 3.94364,
+ "4790": 3.93538,
+ "4795": 3.92791,
+ "4800": 3.91687,
+ "4805": 3.88755,
+ "4810": 3.88191,
+ "4815": 3.98359,
+ "4820": 3.9073,
+ "4825": 3.8325,
+ "4830": 3.93939,
+ "4835": 3.89438,
+ "4840": 3.91266,
+ "4845": 3.84732,
+ "4850": 3.91204,
+ "4855": 3.86786,
+ "4860": 3.86106,
+ "4865": 3.9091,
+ "4870": 3.88998,
+ "4875": 3.8415,
+ "4880": 3.91506,
+ "4885": 3.8906,
+ "4890": 3.97429,
+ "4895": 3.94706,
+ "4900": 3.86119,
+ "4905": 3.86547,
+ "4910": 3.87253,
+ "4915": 3.90292,
+ "4920": 3.88678,
+ "4925": 3.84902,
+ "4930": 3.83402,
+ "4935": 3.87499,
+ "4940": 3.84956,
+ "4945": 3.96294,
+ "4950": 3.83645,
+ "4955": 3.85629,
+ "4960": 3.85433,
+ "4965": 3.86639,
+ "4970": 3.89784,
+ "4975": 3.90319,
+ "4980": 3.87947,
+ "4985": 3.84031,
+ "4990": 3.84166,
+ "4995": 3.88907,
+ "5000": 3.85232,
+ "5005": 3.92668,
+ "5010": 3.89292,
+ "5015": 3.81551,
+ "5020": 3.85077,
+ "5025": 3.88047,
+ "5030": 3.80173,
+ "5035": 3.89262,
+ "5040": 3.81819,
+ "5045": 3.86897,
+ "5050": 3.83488,
+ "5055": 3.90345,
+ "5060": 3.90838,
+ "5065": 3.83496,
+ "5070": 3.84427,
+ "5075": 3.80802,
+ "5080": 3.85376,
+ "5085": 3.87044,
+ "5090": 3.89095,
+ "5095": 3.81325,
+ "5100": 3.85883,
+ "5105": 3.88725,
+ "5110": 3.85313,
+ "5115": 3.79616,
+ "5120": 3.74317,
+ "5125": 3.85102,
+ "5130": 3.76819,
+ "5135": 3.83975,
+ "5140": 3.81912,
+ "5145": 3.78603,
+ "5150": 3.82399,
+ "5155": 3.85617,
+ "5160": 3.80816,
+ "5165": 3.8735,
+ "5170": 3.82091,
+ "5175": 3.82818,
+ "5180": 3.82508,
+ "5185": 3.90678,
+ "5190": 3.82168,
+ "5195": 3.82024,
+ "5200": 3.8017,
+ "5205": 3.77747,
+ "5210": 3.76047,
+ "5215": 3.72921,
+ "5220": 3.81983,
+ "5225": 3.82537,
+ "5230": 3.8128,
+ "5235": 3.81561,
+ "5240": 3.80272,
+ "5245": 3.79024,
+ "5250": 3.78792,
+ "5255": 3.90003,
+ "5260": 3.8188,
+ "5265": 3.82023,
+ "5270": 3.83169,
+ "5275": 3.83694,
+ "5280": 3.78019,
+ "5285": 3.84716,
+ "5290": 3.82177,
+ "5295": 3.81509,
+ "5300": 3.81397,
+ "5305": 3.82738,
+ "5310": 3.79654,
+ "5315": 3.75576,
+ "5320": 3.81132,
+ "5325": 3.77414,
+ "5330": 3.79485,
+ "5335": 3.81631,
+ "5340": 3.87492,
+ "5345": 3.79058,
+ "5350": 3.84091,
+ "5355": 3.82519,
+ "5360": 3.86415,
+ "5365": 3.76891,
+ "5370": 3.73729,
+ "5375": 3.84309,
+ "5380": 3.74432,
+ "5385": 3.77378,
+ "5390": 3.75401,
+ "5395": 3.72333,
+ "5400": 3.83892,
+ "5405": 3.84081,
+ "5410": 3.7956,
+ "5415": 3.78558,
+ "5420": 3.84591,
+ "5425": 3.73071,
+ "5430": 3.8303,
+ "5435": 3.80787,
+ "5440": 3.79521,
+ "5445": 3.76599,
+ "5450": 3.76531,
+ "5455": 3.75261,
+ "5460": 3.76964,
+ "5465": 3.78445,
+ "5470": 3.86175,
+ "5475": 3.78269,
+ "5480": 3.89267,
+ "5485": 3.82597,
+ "5490": 3.76814,
+ "5495": 3.78196,
+ "5500": 3.82672,
+ "5505": 3.83339,
+ "5510": 3.81159,
+ "5515": 3.80566,
+ "5520": 3.82937,
+ "5525": 3.79322,
+ "5530": 3.75251,
+ "5535": 3.75986,
+ "5540": 3.73705,
+ "5545": 3.81811,
+ "5550": 3.73185,
+ "5555": 3.82141,
+ "5560": 3.78827,
+ "5565": 3.73534,
+ "5570": 3.86506,
+ "5575": 3.78935,
+ "5580": 3.72695,
+ "5585": 3.73968,
+ "5590": 3.75168,
+ "5595": 3.78317,
+ "5600": 3.7535,
+ "5605": 3.73353,
+ "5610": 3.75158,
+ "5615": 3.6861,
+ "5620": 3.73127,
+ "5625": 3.73193,
+ "5630": 3.7502,
+ "5635": 3.72498,
+ "5640": 3.74854,
+ "5645": 3.76049,
+ "5650": 3.77236,
+ "5655": 3.76425,
+ "5660": 3.76039,
+ "5665": 3.76945,
+ "5670": 3.74434,
+ "5675": 3.79544,
+ "5680": 3.76008,
+ "5685": 3.74471,
+ "5690": 3.70749,
+ "5695": 3.79908,
+ "5700": 3.76993,
+ "5705": 3.71787,
+ "5710": 3.70626,
+ "5715": 3.71257,
+ "5720": 3.74199,
+ "5725": 3.74934,
+ "5730": 3.80807,
+ "5735": 3.71886,
+ "5740": 3.81758,
+ "5745": 3.75167,
+ "5750": 3.70682,
+ "5755": 3.75987,
+ "5760": 3.83952,
+ "5765": 3.73279,
+ "5770": 3.80644,
+ "5775": 3.71281,
+ "5780": 3.68462,
+ "5785": 3.76518,
+ "5790": 3.73226,
+ "5795": 3.75484,
+ "5800": 3.744,
+ "5805": 3.73559,
+ "5810": 3.68489,
+ "5815": 3.74029,
+ "5820": 3.72936,
+ "5825": 3.67981,
+ "5830": 3.6729,
+ "5835": 3.65572,
+ "5840": 3.6685,
+ "5845": 3.74228,
+ "5850": 3.78802,
+ "5855": 3.63877,
+ "5860": 3.68035,
+ "5865": 3.68662,
+ "5870": 3.75134,
+ "5875": 3.6997,
+ "5880": 3.74511,
+ "5885": 3.67156,
+ "5890": 3.7123,
+ "5895": 3.63771,
+ "5900": 3.67793,
+ "5905": 3.69992,
+ "5910": 3.69342,
+ "5915": 3.66509,
+ "5920": 3.72152,
+ "5925": 3.69075,
+ "5930": 3.66263,
+ "5935": 3.66013,
+ "5940": 3.66763,
+ "5945": 3.70424,
+ "5950": 3.70019,
+ "5955": 3.72691,
+ "5960": 3.68447,
+ "5965": 3.73006,
+ "5970": 3.66022,
+ "5975": 3.66787,
+ "5980": 3.66098,
+ "5985": 3.72606,
+ "5990": 3.74368,
+ "5995": 3.63169,
+ "6000": 3.63794,
+ "6005": 3.64827,
+ "6010": 3.66237,
+ "6015": 3.68262,
+ "6020": 3.67776,
+ "6025": 3.64743,
+ "6030": 3.70622,
+ "6035": 3.70191,
+ "6040": 3.51471,
+ "6045": 3.67578,
+ "6050": 3.60553,
+ "6055": 3.63191,
+ "6060": 3.66171,
+ "6065": 3.72881,
+ "6070": 3.63192,
+ "6075": 3.7436,
+ "6080": 3.64357,
+ "6085": 3.66753,
+ "6090": 3.66125,
+ "6095": 3.71801,
+ "6100": 3.63085,
+ "6105": 3.67049,
+ "6110": 3.61362,
+ "6115": 3.5912,
+ "6120": 3.60015,
+ "6125": 3.70212,
+ "6130": 3.63856,
+ "6135": 3.65367,
+ "6140": 3.6095,
+ "6145": 3.58116,
+ "6150": 3.6704,
+ "6155": 3.62769,
+ "6160": 3.63526,
+ "6165": 3.67783,
+ "6170": 3.66315,
+ "6175": 3.61693,
+ "6180": 3.60177,
+ "6185": 3.69829,
+ "6190": 3.62911,
+ "6195": 3.66449,
+ "6200": 3.60736,
+ "6205": 3.6026,
+ "6210": 3.59962,
+ "6215": 3.70297,
+ "6220": 3.58719,
+ "6225": 3.54433,
+ "6230": 3.58076,
+ "6235": 3.59217,
+ "6240": 3.63021,
+ "6245": 3.63278,
+ "6250": 3.65332,
+ "6255": 3.58478,
+ "6260": 3.61546,
+ "6265": 3.66298,
+ "6270": 3.65991,
+ "6275": 3.58848,
+ "6280": 3.57566,
+ "6285": 3.59173,
+ "6290": 3.68207,
+ "6295": 3.58893,
+ "6300": 3.57839,
+ "6305": 3.56104,
+ "6310": 3.60269,
+ "6315": 3.63978,
+ "6320": 3.62111,
+ "6325": 3.56723,
+ "6330": 3.62381,
+ "6335": 3.56472,
+ "6340": 3.53568,
+ "6345": 3.56648,
+ "6350": 3.63179,
+ "6355": 3.64163,
+ "6360": 3.58437,
+ "6365": 3.54933,
+ "6370": 3.60677,
+ "6375": 3.6199,
+ "6380": 3.60864,
+ "6385": 3.57206,
+ "6390": 3.60807,
+ "6395": 3.60167,
+ "6400": 3.54219,
+ "6405": 3.59663,
+ "6410": 3.61927,
+ "6415": 3.5454,
+ "6420": 3.52682,
+ "6425": 3.62645,
+ "6430": 3.59779,
+ "6435": 3.631,
+ "6440": 3.56304,
+ "6445": 3.57659,
+ "6450": 3.60036,
+ "6455": 3.53085,
+ "6460": 3.55408,
+ "6465": 3.55146,
+ "6470": 3.60305,
+ "6475": 3.50565,
+ "6480": 3.56853,
+ "6485": 3.50492,
+ "6490": 3.5591,
+ "6495": 3.55869,
+ "6500": 3.52703,
+ "6505": 3.66558,
+ "6510": 3.5914,
+ "6515": 3.61328,
+ "6520": 3.60955,
+ "6525": 3.55761,
+ "6530": 3.60399,
+ "6535": 3.62898,
+ "6540": 3.57438,
+ "6545": 3.54829,
+ "6550": 3.59016,
+ "6555": 3.56989,
+ "6560": 3.57337,
+ "6565": 3.49823,
+ "6570": 3.51482,
+ "6575": 3.50257,
+ "6580": 3.52887,
+ "6585": 3.61242,
+ "6590": 3.5563,
+ "6595": 3.54833,
+ "6600": 3.57552,
+ "6605": 3.53416,
+ "6610": 3.54463,
+ "6615": 3.54804,
+ "6620": 3.51665,
+ "6625": 3.53607,
+ "6630": 3.50237,
+ "6635": 3.57527,
+ "6640": 3.53314,
+ "6645": 3.56414,
+ "6650": 3.59013,
+ "6655": 3.52952,
+ "6660": 3.61313,
+ "6665": 3.53302,
+ "6670": 3.50004,
+ "6675": 3.64215,
+ "6680": 3.49245,
+ "6685": 3.50857,
+ "6690": 3.55089,
+ "6695": 3.52333,
+ "6700": 3.5042,
+ "6705": 3.52184,
+ "6710": 3.55079,
+ "6715": 3.55164,
+ "6720": 3.54675,
+ "6725": 3.57793,
+ "6730": 3.41301,
+ "6735": 3.54814,
+ "6740": 3.54832,
+ "6745": 3.48107,
+ "6750": 3.49253,
+ "6755": 3.58562,
+ "6760": 3.51154,
+ "6765": 3.54187,
+ "6770": 3.50934,
+ "6775": 3.52643,
+ "6780": 3.52314,
+ "6785": 3.54088,
+ "6790": 3.53772,
+ "6795": 3.51273,
+ "6800": 3.50033,
+ "6805": 3.44368,
+ "6810": 3.50798,
+ "6815": 3.5293,
+ "6820": 3.50708,
+ "6825": 3.55879,
+ "6830": 3.51339,
+ "6835": 3.48197,
+ "6840": 3.49093,
+ "6845": 3.49887,
+ "6850": 3.53189,
+ "6855": 3.49561,
+ "6860": 3.41624,
+ "6865": 3.5245,
+ "6870": 3.49929,
+ "6875": 3.54547,
+ "6880": 3.48665,
+ "6885": 3.4012,
+ "6890": 3.48896,
+ "6895": 3.43234,
+ "6900": 3.44905,
+ "6905": 3.42541,
+ "6910": 3.43003,
+ "6915": 3.46719,
+ "6920": 3.40083,
+ "6925": 3.44651,
+ "6930": 3.51518,
+ "6935": 3.45754,
+ "6940": 3.49963,
+ "6945": 3.46255,
+ "6950": 3.32883,
+ "6955": 3.47583,
+ "6960": 3.48123,
+ "6965": 3.45423,
+ "6970": 3.4398,
+ "6975": 3.52502,
+ "6980": 3.42735,
+ "6985": 3.42444,
+ "6990": 3.46998,
+ "6995": 3.3371,
+ "7000": 3.37682,
+ "7005": 3.38754,
+ "7010": 3.4539,
+ "7015": 3.4112,
+ "7020": 3.39445,
+ "7025": 3.46526,
+ "7030": 3.42066,
+ "7035": 3.4072,
+ "7040": 3.4623,
+ "7045": 3.40705,
+ "7050": 3.3426,
+ "7055": 3.4091,
+ "7060": 3.35191,
+ "7065": 3.38135,
+ "7070": 3.48038,
+ "7075": 3.38427,
+ "7080": 3.33994,
+ "7085": 3.40464,
+ "7090": 3.37164,
+ "7095": 3.42006,
+ "7100": 3.36606,
+ "7105": 3.40375,
+ "7110": 3.36181,
+ "7115": 3.36881,
+ "7120": 3.34598,
+ "7125": 3.36207,
+ "7130": 3.30433,
+ "7135": 3.33451,
+ "7140": 3.40053,
+ "7145": 3.39849,
+ "7150": 3.33808,
+ "7155": 3.30114,
+ "7160": 3.32182,
+ "7165": 3.45527,
+ "7170": 3.33846,
+ "7175": 3.41438,
+ "7180": 3.38941,
+ "7185": 3.40861,
+ "7190": 3.37408,
+ "7195": 3.33802,
+ "7200": 3.34143,
+ "7205": 3.39943,
+ "7210": 3.37255,
+ "7215": 3.3721,
+ "7220": 3.33346,
+ "7225": 3.33689,
+ "7230": 3.37241,
+ "7235": 3.27767,
+ "7240": 3.27538,
+ "7245": 3.33442,
+ "7250": 3.31682,
+ "7255": 3.2846,
+ "7260": 3.35439,
+ "7265": 3.32193,
+ "7270": 3.35072,
+ "7275": 3.36957,
+ "7280": 3.29185,
+ "7285": 3.33401,
+ "7290": 3.30375,
+ "7295": 3.26043,
+ "7300": 3.33857,
+ "7305": 3.33553,
+ "7310": 3.28955,
+ "7315": 3.24511,
+ "7320": 3.30933,
+ "7325": 3.30603,
+ "7330": 3.36703,
+ "7335": 3.34264,
+ "7340": 3.24938,
+ "7345": 3.35515,
+ "7350": 3.28985,
+ "7355": 3.28936,
+ "7360": 3.26968,
+ "7365": 3.30451,
+ "7370": 3.24063,
+ "7375": 3.24108,
+ "7380": 3.29109,
+ "7385": 3.33841,
+ "7390": 3.20459,
+ "7395": 3.31078,
+ "7400": 3.26896,
+ "7405": 3.36342,
+ "7410": 3.19863,
+ "7415": 3.28176,
+ "7420": 3.22978,
+ "7425": 3.22824,
+ "7430": 3.20037,
+ "7435": 3.32536,
+ "7440": 3.2787,
+ "7445": 3.27822,
+ "7450": 3.23743,
+ "7455": 3.23542,
+ "7460": 3.24453,
+ "7465": 3.28904,
+ "7470": 3.24615,
+ "7475": 3.18326,
+ "7480": 3.2111,
+ "7485": 3.15415,
+ "7490": 3.27,
+ "7495": 3.26094,
+ "7500": 3.16431,
+ "7505": 3.16803,
+ "7510": 3.27332,
+ "7515": 3.27031,
+ "7520": 3.26193,
+ "7525": 3.23394,
+ "7530": 3.22738,
+ "7535": 3.13774,
+ "7540": 3.20835,
+ "7545": 3.24259,
+ "7550": 3.23875,
+ "7555": 3.15769,
+ "7560": 3.21744,
+ "7565": 3.17494,
+ "7570": 3.22941,
+ "7575": 3.22872,
+ "7580": 3.15383,
+ "7585": 3.20582,
+ "7590": 3.18356,
+ "7595": 3.22811,
+ "7600": 3.19598,
+ "7605": 3.1894,
+ "7610": 3.21498,
+ "7615": 3.14321,
+ "7620": 3.13302,
+ "7625": 3.23964,
+ "7630": 3.11937,
+ "7635": 3.19315,
+ "7640": 3.2635,
+ "7645": 3.20638,
+ "7650": 3.1414,
+ "7655": 3.16669,
+ "7660": 3.12539,
+ "7665": 3.15098,
+ "7670": 3.10394,
+ "7675": 3.15022,
+ "7680": 3.14413,
+ "7685": 3.18761,
+ "7690": 3.1541,
+ "7695": 3.19911,
+ "7700": 3.1824,
+ "7705": 3.10679,
+ "7710": 3.21962,
+ "7715": 3.2151,
+ "7720": 3.14827,
+ "7725": 3.20891,
+ "7730": 3.13055,
+ "7735": 3.15225,
+ "7740": 3.1199,
+ "7745": 3.11032,
+ "7750": 3.16642,
+ "7755": 3.1622,
+ "7760": 3.19494,
+ "7765": 3.12263,
+ "7770": 3.14226,
+ "7775": 3.13574,
+ "7780": 3.15839,
+ "7785": 3.10935,
+ "7790": 3.1439,
+ "7795": 3.10758,
+ "7800": 3.14141,
+ "7805": 3.11809,
+ "7810": 3.14997,
+ "7815": 3.17081,
+ "7820": 3.17367,
+ "7825": 3.13338,
+ "7830": 3.10933,
+ "7835": 3.09797,
+ "7840": 3.09537,
+ "7845": 3.18511,
+ "7850": 3.06743,
+ "7855": 3.05527,
+ "7860": 3.08555,
+ "7865": 3.08367,
+ "7870": 3.07257,
+ "7875": 3.04634,
+ "7880": 3.08808,
+ "7885": 3.12011,
+ "7890": 3.08166,
+ "7895": 3.01558,
+ "7900": 3.15165,
+ "7905": 3.05007,
+ "7910": 3.04656,
+ "7915": 3.13271,
+ "7920": 3.06878,
+ "7925": 3.09724,
+ "7930": 3.06822,
+ "7935": 3.11711,
+ "7940": 3.06319,
+ "7945": 2.94985,
+ "7950": 3.06059,
+ "7955": 3.03181,
+ "7960": 2.99499,
+ "7965": 3.06894,
+ "7970": 3.10161,
+ "7975": 3.11679,
+ "7980": 3.04265,
+ "7985": 3.08104,
+ "7990": 3.06958,
+ "7995": 3.07142,
+ "8000": 3.00194,
+ "8005": 3.05021,
+ "8010": 3.06187,
+ "8015": 3.05375,
+ "8020": 3.10144,
+ "8025": 3.01091,
+ "8030": 3.07437,
+ "8035": 2.98235,
+ "8040": 3.04012,
+ "8045": 3.09009,
+ "8050": 3.07521,
+ "8055": 3.0853,
+ "8060": 3.08327,
+ "8065": 3.01433,
+ "8070": 3.03111,
+ "8075": 3.03344,
+ "8080": 2.9982,
+ "8085": 2.97416,
+ "8090": 3.02924,
+ "8095": 3.00652,
+ "8100": 3.04662,
+ "8105": 3.037,
+ "8110": 2.99193,
+ "8115": 2.98275,
+ "8120": 3.00323,
+ "8125": 3.03283,
+ "8130": 3.01137,
+ "8135": 3.04655,
+ "8140": 2.97811,
+ "8145": 3.03107,
+ "8150": 3.10562,
+ "8155": 3.02842,
+ "8160": 3.01651,
+ "8165": 3.01088,
+ "8170": 3.08901,
+ "8175": 3.01775,
+ "8180": 3.05469,
+ "8185": 3.00109,
+ "8190": 2.88217,
+ "8195": 3.07778,
+ "8200": 2.97616,
+ "8205": 2.98703,
+ "8210": 3.02127,
+ "8215": 2.98136,
+ "8220": 3.03713,
+ "8225": 3.0524,
+ "8230": 3.03954,
+ "8235": 3.0295,
+ "8240": 3.00619,
+ "8245": 2.98657,
+ "8250": 3.04377,
+ "8255": 3.01583,
+ "8260": 3.02235,
+ "8265": 3.00932,
+ "8270": 2.97415,
+ "8275": 3.04481,
+ "8280": 2.98862,
+ "8285": 3.03337,
+ "8290": 2.98486,
+ "8295": 2.96065,
+ "8300": 3.09519,
+ "8305": 2.95208,
+ "8310": 2.98079,
+ "8315": 2.99249,
+ "8320": 2.9808,
+ "8325": 2.97269,
+ "8330": 2.95839,
+ "8335": 2.92008,
+ "8340": 2.92672,
+ "8345": 2.97488,
+ "8350": 2.96557,
+ "8355": 3.03346,
+ "8360": 2.99165,
+ "8365": 3.00268,
+ "8370": 3.0323,
+ "8375": 2.93651,
+ "8380": 3.00745,
+ "8385": 2.91828,
+ "8390": 2.96404,
+ "8395": 2.94723,
+ "8400": 2.92675,
+ "8405": 2.91306,
+ "8410": 2.95279,
+ "8415": 2.99409,
+ "8420": 2.93882,
+ "8425": 2.98731,
+ "8430": 2.98076,
+ "8435": 2.98704,
+ "8440": 3.00051,
+ "8445": 2.92164,
+ "8450": 2.94232,
+ "8455": 2.96373,
+ "8460": 2.9545,
+ "8465": 2.9468,
+ "8470": 2.91866,
+ "8475": 2.93174,
+ "8480": 3.00867,
+ "8485": 2.94715,
+ "8490": 2.97952,
+ "8495": 2.8169,
+ "8500": 2.97737,
+ "8505": 2.94897,
+ "8510": 2.92843,
+ "8515": 2.96661,
+ "8520": 2.94106,
+ "8525": 2.98697,
+ "8530": 2.9068,
+ "8535": 2.97317,
+ "8540": 2.98903,
+ "8545": 2.98025,
+ "8550": 2.94124,
+ "8555": 2.89171,
+ "8560": 2.91156,
+ "8565": 3.00138,
+ "8570": 2.9654,
+ "8575": 2.94188,
+ "8580": 2.994,
+ "8585": 2.93715,
+ "8590": 2.96655,
+ "8595": 2.88697,
+ "8600": 2.95745,
+ "8605": 2.96386,
+ "8610": 2.96146,
+ "8615": 2.92197,
+ "8620": 2.99677,
+ "8625": 2.95391,
+ "8630": 2.9435,
+ "8635": 2.99451,
+ "8640": 2.96087,
+ "8645": 3.0103,
+ "8650": 2.85171,
+ "8655": 2.91985,
+ "8660": 2.95712,
+ "8665": 2.95679,
+ "8670": 2.88892,
+ "8675": 2.89363,
+ "8680": 2.88715,
+ "8685": 2.95281,
+ "8690": 2.8931,
+ "8695": 2.91653,
+ "8700": 2.96033,
+ "8705": 2.86024,
+ "8710": 2.9849,
+ "8715": 2.9153,
+ "8720": 2.92902,
+ "8725": 2.87082,
+ "8730": 2.93885,
+ "8735": 2.89104,
+ "8740": 2.91065,
+ "8745": 2.8566,
+ "8750": 2.82874,
+ "8755": 2.89383,
+ "8760": 2.90877,
+ "8765": 2.92385,
+ "8770": 2.85456,
+ "8775": 2.88889,
+ "8780": 2.93383,
+ "8785": 2.96551,
+ "8790": 2.92718,
+ "8795": 2.83905,
+ "8800": 2.82969,
+ "8805": 2.8696,
+ "8810": 2.97646,
+ "8815": 2.83331,
+ "8820": 2.85388,
+ "8825": 2.89207,
+ "8830": 2.91402,
+ "8835": 2.93624,
+ "8840": 2.89498,
+ "8845": 2.93912,
+ "8850": 2.88273,
+ "8855": 2.86641,
+ "8860": 2.85851,
+ "8865": 2.81636,
+ "8870": 2.90965,
+ "8875": 2.90412,
+ "8880": 2.85162,
+ "8885": 2.82858,
+ "8890": 2.87764,
+ "8895": 2.84269,
+ "8900": 2.86604,
+ "8905": 2.88375,
+ "8910": 2.80972,
+ "8915": 2.89041,
+ "8920": 2.84298,
+ "8925": 2.83678,
+ "8930": 2.91171,
+ "8935": 2.863,
+ "8940": 2.88761,
+ "8945": 2.90862,
+ "8950": 2.86886,
+ "8955": 2.88849,
+ "8960": 2.82783,
+ "8965": 2.826,
+ "8970": 2.87944,
+ "8975": 2.87011,
+ "8980": 2.85602,
+ "8985": 2.87026,
+ "8990": 2.79834,
+ "8995": 2.85714,
+ "9000": 2.77276,
+ "9005": 2.851,
+ "9010": 2.87519,
+ "9015": 2.86994,
+ "9020": 2.84635,
+ "9025": 2.80009,
+ "9030": 2.92976,
+ "9035": 2.89187,
+ "9040": 2.82219,
+ "9045": 2.91078,
+ "9050": 2.80982,
+ "9055": 2.872,
+ "9060": 2.86158,
+ "9065": 2.81449,
+ "9070": 2.84187,
+ "9075": 2.869,
+ "9080": 2.94704,
+ "9085": 2.8077,
+ "9090": 2.85347,
+ "9095": 2.84466,
+ "9100": 2.79375,
+ "9105": 2.83847,
+ "9110": 2.79449,
+ "9115": 2.81127,
+ "9120": 2.84302,
+ "9125": 2.86065,
+ "9130": 2.83342,
+ "9135": 2.85113,
+ "9140": 2.80045,
+ "9145": 2.82167,
+ "9150": 2.78493,
+ "9155": 2.80618,
+ "9160": 2.83696,
+ "9165": 2.83222,
+ "9170": 2.88746,
+ "9175": 2.80556,
+ "9180": 2.85392,
+ "9185": 2.80468,
+ "9190": 2.79799,
+ "9195": 2.79621,
+ "9200": 2.83834,
+ "9205": 2.83526,
+ "9210": 2.82582,
+ "9215": 2.89307,
+ "9220": 2.85758,
+ "9225": 2.84964,
+ "9230": 2.81353,
+ "9235": 2.87691,
+ "9240": 2.82266,
+ "9245": 2.877,
+ "9250": 2.81093,
+ "9255": 2.86249,
+ "9260": 2.76767,
+ "9265": 2.7352,
+ "9270": 2.82135,
+ "9275": 2.87713,
+ "9280": 2.81892,
+ "9285": 2.81767,
+ "9290": 2.77261,
+ "9295": 2.82102,
+ "9300": 2.8435,
+ "9305": 2.88097,
+ "9310": 2.86154,
+ "9315": 2.80385,
+ "9320": 2.8053,
+ "9325": 2.89561,
+ "9330": 2.83011,
+ "9335": 2.85054,
+ "9340": 2.83606,
+ "9345": 2.85149,
+ "9350": 2.79749,
+ "9355": 2.89364,
+ "9360": 2.78961,
+ "9365": 2.81195,
+ "9370": 2.79173,
+ "9375": 2.82061,
+ "9380": 2.8032,
+ "9385": 2.80666,
+ "9390": 2.8133,
+ "9395": 2.79927,
+ "9400": 2.8248,
+ "9405": 2.76771,
+ "9410": 2.82928,
+ "9415": 2.78187,
+ "9420": 2.81411,
+ "9425": 2.81987,
+ "9430": 2.81408,
+ "9435": 2.79189,
+ "9440": 2.77353,
+ "9445": 2.73783,
+ "9450": 2.7819,
+ "9455": 2.84157,
+ "9460": 2.73792,
+ "9465": 2.79481,
+ "9470": 2.76282,
+ "9475": 2.77609,
+ "9480": 2.71721,
+ "9485": 2.79126,
+ "9490": 2.77756,
+ "9495": 2.84392,
+ "9500": 2.76664,
+ "9505": 2.82406,
+ "9510": 2.78094,
+ "9515": 2.79424,
+ "9520": 2.7574,
+ "9525": 2.82816,
+ "9530": 2.79299,
+ "9535": 2.76929,
+ "9540": 2.74103,
+ "9545": 2.79346,
+ "9550": 2.85658,
+ "9555": 2.83351,
+ "9560": 2.82877,
+ "9565": 2.88969,
+ "9570": 2.81792,
+ "9575": 2.79137,
+ "9580": 2.7864,
+ "9585": 2.7408,
+ "9590": 2.7024,
+ "9595": 2.75395,
+ "9600": 2.78646,
+ "9605": 2.78734,
+ "9610": 2.84338,
+ "9615": 2.76899,
+ "9620": 2.80359,
+ "9625": 2.73596,
+ "9630": 2.77699,
+ "9635": 2.81855,
+ "9640": 2.83543,
+ "9645": 2.83833,
+ "9650": 2.76459,
+ "9655": 2.68887,
+ "9660": 2.86627,
+ "9665": 2.7891,
+ "9670": 2.83178,
+ "9675": 2.82926,
+ "9680": 2.74768,
+ "9685": 2.71754,
+ "9690": 2.75017,
+ "9695": 2.83987,
+ "9700": 2.78744,
+ "9705": 2.85662,
+ "9710": 2.80695,
+ "9715": 2.76227,
+ "9720": 2.73878,
+ "9725": 2.77904,
+ "9730": 2.84717,
+ "9735": 2.76487,
+ "9740": 2.75082,
+ "9745": 2.77276,
+ "9750": 2.81423,
+ "9755": 2.80542,
+ "9760": 2.72685,
+ "9765": 2.841,
+ "9770": 2.82005,
+ "9775": 2.77846,
+ "9780": 2.79358,
+ "9785": 2.75899,
+ "9790": 2.70122,
+ "9795": 2.70857,
+ "9800": 2.77454,
+ "9805": 2.76812,
+ "9810": 2.79558,
+ "9815": 2.72158,
+ "9820": 2.73787,
+ "9825": 2.77964,
+ "9830": 2.83434,
+ "9835": 2.73972,
+ "9840": 2.74294,
+ "9845": 2.79263,
+ "9850": 2.71785,
+ "9855": 2.74514,
+ "9860": 2.85214,
+ "9865": 2.73884,
+ "9870": 2.75013,
+ "9875": 2.76268,
+ "9880": 2.78073,
+ "9885": 2.75028,
+ "9890": 2.76406,
+ "9895": 2.76004,
+ "9900": 2.76775,
+ "9905": 2.71344,
+ "9910": 2.79481,
+ "9915": 2.69689,
+ "9920": 2.78828,
+ "9925": 2.73343,
+ "9930": 2.74746,
+ "9935": 2.77577,
+ "9940": 2.80676,
+ "9945": 2.7125,
+ "9950": 2.83891,
+ "9955": 2.70479,
+ "9960": 2.81859,
+ "9965": 2.72629,
+ "9970": 2.73337,
+ "9975": 2.79333,
+ "9980": 2.74689,
+ "9985": 2.68449,
+ "9990": 2.72662,
+ "9995": 2.77505,
+ "10000": 2.74629
+ }
+ },
+ "num-zeros": {
+ "start_step": 1,
+ "end_step": 10000,
+ "step_interval": 5,
+ "values": {
+ "1": 40052.0,
+ "5": 40028.0,
+ "10": 41482.0,
+ "15": 37263.0,
+ "20": 40804.0,
+ "25": 40129.0,
+ "30": 40745.0,
+ "35": 39351.0,
+ "40": 41491.0,
+ "45": 42267.0,
+ "50": 38437.0,
+ "55": 39345.0,
+ "60": 42258.0,
+ "65": 39975.0,
+ "70": 40728.0,
+ "75": 41490.0,
+ "80": 40874.0,
+ "85": 40796.0,
+ "90": 40712.0,
+ "95": 40727.0,
+ "100": 41554.0,
+ "105": 40780.0,
+ "110": 38713.0,
+ "115": 41558.0,
+ "120": 39259.0,
+ "125": 40718.0,
+ "130": 40727.0,
+ "135": 37920.0,
+ "140": 38721.0,
+ "145": 40941.0,
+ "150": 40017.0,
+ "155": 40783.0,
+ "160": 40100.0,
+ "165": 38583.0,
+ "170": 37361.0,
+ "175": 40035.0,
+ "180": 40114.0,
+ "185": 38579.0,
+ "190": 39330.0,
+ "195": 41506.0,
+ "200": 41563.0,
+ "205": 40091.0,
+ "210": 40799.0,
+ "215": 40024.0,
+ "220": 40790.0,
+ "225": 39410.0,
+ "230": 41486.0,
+ "235": 37965.0,
+ "240": 39318.0,
+ "245": 38631.0,
+ "250": 40030.0,
+ "255": 41474.0,
+ "260": 39256.0,
+ "265": 41562.0,
+ "270": 40729.0,
+ "275": 40790.0,
+ "280": 41489.0,
+ "285": 41564.0,
+ "290": 40793.0,
+ "295": 38575.0,
+ "300": 38709.0,
+ "305": 41552.0,
+ "310": 39263.0,
+ "315": 40114.0,
+ "320": 39325.0,
+ "325": 40795.0,
+ "330": 40807.0,
+ "335": 40174.0,
+ "340": 37978.0,
+ "345": 39954.0,
+ "350": 40730.0,
+ "355": 39948.0,
+ "360": 41555.0,
+ "365": 37956.0,
+ "370": 40037.0,
+ "375": 41492.0,
+ "380": 38720.0,
+ "385": 37817.0,
+ "390": 39954.0,
+ "395": 40787.0,
+ "400": 39251.0,
+ "405": 40720.0,
+ "410": 39503.0,
+ "415": 40720.0,
+ "420": 41487.0,
+ "425": 39392.0,
+ "430": 39380.0,
+ "435": 42256.0,
+ "440": 35797.0,
+ "445": 40017.0,
+ "450": 41493.0,
+ "455": 39960.0,
+ "460": 38571.0,
+ "465": 39343.0,
+ "470": 40786.0,
+ "475": 40800.0,
+ "480": 40786.0,
+ "485": 39210.0,
+ "490": 38713.0,
+ "495": 40780.0,
+ "500": 40716.0,
+ "505": 41497.0,
+ "510": 37746.0,
+ "515": 40719.0,
+ "520": 40020.0,
+ "525": 39276.0,
+ "530": 38825.0,
+ "535": 40821.0,
+ "540": 40041.0,
+ "545": 41543.0,
+ "550": 40814.0,
+ "555": 40723.0,
+ "560": 39956.0,
+ "565": 40712.0,
+ "570": 40156.0,
+ "575": 39330.0,
+ "580": 40781.0,
+ "585": 40718.0,
+ "590": 40026.0,
+ "595": 40046.0,
+ "600": 40048.0,
+ "605": 40801.0,
+ "610": 40875.0,
+ "615": 39949.0,
+ "620": 41562.0,
+ "625": 40735.0,
+ "630": 42257.0,
+ "635": 39962.0,
+ "640": 37049.0,
+ "645": 40118.0,
+ "650": 39253.0,
+ "655": 39188.0,
+ "660": 37373.0,
+ "665": 40122.0,
+ "670": 40783.0,
+ "675": 39253.0,
+ "680": 39950.0,
+ "685": 39249.0,
+ "690": 42258.0,
+ "695": 38595.0,
+ "700": 40052.0,
+ "705": 38569.0,
+ "710": 42250.0,
+ "715": 42247.0,
+ "720": 41487.0,
+ "725": 40792.0,
+ "730": 38638.0,
+ "735": 40784.0,
+ "740": 41568.0,
+ "745": 41568.0,
+ "750": 40036.0,
+ "755": 38685.0,
+ "760": 38586.0,
+ "765": 39957.0,
+ "770": 40723.0,
+ "775": 40718.0,
+ "780": 39327.0,
+ "785": 41581.0,
+ "790": 42331.0,
+ "795": 41550.0,
+ "800": 40802.0,
+ "805": 37053.0,
+ "810": 39342.0,
+ "815": 40718.0,
+ "820": 43022.0,
+ "825": 40017.0,
+ "830": 39956.0,
+ "835": 41487.0,
+ "840": 38551.0,
+ "845": 41565.0,
+ "850": 38031.0,
+ "855": 36482.0,
+ "860": 41487.0,
+ "865": 39957.0,
+ "870": 35796.0,
+ "875": 38586.0,
+ "880": 39970.0,
+ "885": 40045.0,
+ "890": 39426.0,
+ "895": 40728.0,
+ "900": 40020.0,
+ "905": 40030.0,
+ "910": 40044.0,
+ "915": 39271.0,
+ "920": 40085.0,
+ "925": 40106.0,
+ "930": 40091.0,
+ "935": 40714.0,
+ "940": 40781.0,
+ "945": 39957.0,
+ "950": 37730.0,
+ "955": 40085.0,
+ "960": 40799.0,
+ "965": 40122.0,
+ "970": 40724.0,
+ "975": 39956.0,
+ "980": 40801.0,
+ "985": 30945.0,
+ "990": 39362.0,
+ "995": 42256.0,
+ "1000": 41576.0,
+ "1005": 41488.0,
+ "1010": 41636.0,
+ "1015": 40040.0,
+ "1020": 39281.0,
+ "1025": 40039.0,
+ "1030": 40181.0,
+ "1035": 39417.0,
+ "1040": 40109.0,
+ "1045": 40803.0,
+ "1050": 41493.0,
+ "1055": 37959.0,
+ "1060": 42263.0,
+ "1065": 37199.0,
+ "1070": 39268.0,
+ "1075": 43018.0,
+ "1080": 39356.0,
+ "1085": 40042.0,
+ "1090": 39973.0,
+ "1095": 38495.0,
+ "1100": 38672.0,
+ "1105": 39332.0,
+ "1110": 42339.0,
+ "1115": 40780.0,
+ "1120": 40727.0,
+ "1125": 40737.0,
+ "1130": 38667.0,
+ "1135": 40103.0,
+ "1140": 40102.0,
+ "1145": 39350.0,
+ "1150": 41572.0,
+ "1155": 40718.0,
+ "1160": 41492.0,
+ "1165": 40039.0,
+ "1170": 39273.0,
+ "1175": 40034.0,
+ "1180": 40740.0,
+ "1185": 39266.0,
+ "1190": 39267.0,
+ "1195": 40716.0,
+ "1200": 41556.0,
+ "1205": 38577.0,
+ "1210": 41580.0,
+ "1215": 39346.0,
+ "1220": 41549.0,
+ "1225": 40114.0,
+ "1230": 39329.0,
+ "1235": 41489.0,
+ "1240": 40719.0,
+ "1245": 41483.0,
+ "1250": 39408.0,
+ "1255": 38729.0,
+ "1260": 41484.0,
+ "1265": 41489.0,
+ "1270": 40727.0,
+ "1275": 38445.0,
+ "1280": 40125.0,
+ "1285": 40719.0,
+ "1290": 40041.0,
+ "1295": 40787.0,
+ "1300": 40731.0,
+ "1305": 41500.0,
+ "1310": 41489.0,
+ "1315": 40801.0,
+ "1320": 40724.0,
+ "1325": 40726.0,
+ "1330": 40809.0,
+ "1335": 38665.0,
+ "1340": 40800.0,
+ "1345": 39970.0,
+ "1350": 40797.0,
+ "1355": 40107.0,
+ "1360": 42257.0,
+ "1365": 39399.0,
+ "1370": 39351.0,
+ "1375": 40795.0,
+ "1380": 38656.0,
+ "1385": 38606.0,
+ "1390": 40042.0,
+ "1395": 40801.0,
+ "1400": 40791.0,
+ "1405": 40815.0,
+ "1410": 37800.0,
+ "1415": 40018.0,
+ "1420": 41497.0,
+ "1425": 40793.0,
+ "1430": 39964.0,
+ "1435": 35892.0,
+ "1440": 38578.0,
+ "1445": 39342.0,
+ "1450": 40130.0,
+ "1455": 41571.0,
+ "1460": 40806.0,
+ "1465": 40726.0,
+ "1470": 37170.0,
+ "1475": 40728.0,
+ "1480": 37363.0,
+ "1485": 37819.0,
+ "1490": 38505.0,
+ "1495": 38599.0,
+ "1500": 41503.0,
+ "1505": 40025.0,
+ "1510": 40729.0,
+ "1515": 40777.0,
+ "1520": 40811.0,
+ "1525": 40863.0,
+ "1530": 39269.0,
+ "1535": 39344.0,
+ "1540": 37848.0,
+ "1545": 39965.0,
+ "1550": 40892.0,
+ "1555": 40821.0,
+ "1560": 40800.0,
+ "1565": 37895.0,
+ "1570": 39959.0,
+ "1575": 40026.0,
+ "1580": 38656.0,
+ "1585": 39261.0,
+ "1590": 39341.0,
+ "1595": 37971.0,
+ "1600": 40792.0,
+ "1605": 39337.0,
+ "1610": 40738.0,
+ "1615": 37889.0,
+ "1620": 39264.0,
+ "1625": 41557.0,
+ "1630": 39969.0,
+ "1635": 40802.0,
+ "1640": 41487.0,
+ "1645": 40027.0,
+ "1650": 39382.0,
+ "1655": 41496.0,
+ "1660": 40891.0,
+ "1665": 41568.0,
+ "1670": 40804.0,
+ "1675": 40795.0,
+ "1680": 40872.0,
+ "1685": 39954.0,
+ "1690": 40041.0,
+ "1695": 39343.0,
+ "1700": 40793.0,
+ "1705": 41661.0,
+ "1710": 41489.0,
+ "1715": 39339.0,
+ "1720": 41565.0,
+ "1725": 40048.0,
+ "1730": 39348.0,
+ "1735": 38649.0,
+ "1740": 41486.0,
+ "1745": 37769.0,
+ "1750": 39281.0,
+ "1755": 39348.0,
+ "1760": 39283.0,
+ "1765": 40804.0,
+ "1770": 40790.0,
+ "1775": 39409.0,
+ "1780": 41507.0,
+ "1785": 41560.0,
+ "1790": 40716.0,
+ "1795": 41485.0,
+ "1800": 39416.0,
+ "1805": 40724.0,
+ "1810": 40042.0,
+ "1815": 37734.0,
+ "1820": 40722.0,
+ "1825": 39423.0,
+ "1830": 41498.0,
+ "1835": 39416.0,
+ "1840": 40821.0,
+ "1845": 40796.0,
+ "1850": 40796.0,
+ "1855": 40795.0,
+ "1860": 40723.0,
+ "1865": 40040.0,
+ "1870": 41643.0,
+ "1875": 40729.0,
+ "1880": 39325.0,
+ "1885": 39214.0,
+ "1890": 41567.0,
+ "1895": 40736.0,
+ "1900": 37828.0,
+ "1905": 41501.0,
+ "1910": 41490.0,
+ "1915": 40973.0,
+ "1920": 41556.0,
+ "1925": 39282.0,
+ "1930": 41485.0,
+ "1935": 40807.0,
+ "1940": 40037.0,
+ "1945": 39423.0,
+ "1950": 41495.0,
+ "1955": 41551.0,
+ "1960": 40816.0,
+ "1965": 40109.0,
+ "1970": 39965.0,
+ "1975": 42260.0,
+ "1980": 36485.0,
+ "1985": 40139.0,
+ "1990": 41556.0,
+ "1995": 40029.0,
+ "2000": 41493.0,
+ "2005": 41577.0,
+ "2010": 37818.0,
+ "2015": 38739.0,
+ "2020": 40730.0,
+ "2025": 38728.0,
+ "2030": 38037.0,
+ "2035": 40755.0,
+ "2040": 41591.0,
+ "2045": 40179.0,
+ "2050": 39492.0,
+ "2055": 41576.0,
+ "2060": 39427.0,
+ "2065": 42264.0,
+ "2070": 41492.0,
+ "2075": 39261.0,
+ "2080": 42259.0,
+ "2085": 39338.0,
+ "2090": 40815.0,
+ "2095": 41564.0,
+ "2100": 41567.0,
+ "2105": 41497.0,
+ "2110": 40813.0,
+ "2115": 40731.0,
+ "2120": 41496.0,
+ "2125": 40033.0,
+ "2130": 39322.0,
+ "2135": 39422.0,
+ "2140": 41487.0,
+ "2145": 38601.0,
+ "2150": 40038.0,
+ "2155": 41507.0,
+ "2160": 41575.0,
+ "2165": 40036.0,
+ "2170": 40811.0,
+ "2175": 39343.0,
+ "2180": 40051.0,
+ "2185": 41498.0,
+ "2190": 40885.0,
+ "2195": 38822.0,
+ "2200": 40802.0,
+ "2205": 39952.0,
+ "2210": 39425.0,
+ "2215": 40787.0,
+ "2220": 39285.0,
+ "2225": 40113.0,
+ "2230": 40802.0,
+ "2235": 37291.0,
+ "2240": 40027.0,
+ "2245": 40732.0,
+ "2250": 42257.0,
+ "2255": 40741.0,
+ "2260": 41575.0,
+ "2265": 40818.0,
+ "2270": 40028.0,
+ "2275": 40729.0,
+ "2280": 39197.0,
+ "2285": 40885.0,
+ "2290": 39427.0,
+ "2295": 37895.0,
+ "2300": 41568.0,
+ "2305": 40737.0,
+ "2310": 40042.0,
+ "2315": 38495.0,
+ "2320": 40098.0,
+ "2325": 41495.0,
+ "2330": 40738.0,
+ "2335": 37174.0,
+ "2340": 40727.0,
+ "2345": 40164.0,
+ "2350": 40105.0,
+ "2355": 40892.0,
+ "2360": 39264.0,
+ "2365": 40717.0,
+ "2370": 40732.0,
+ "2375": 38568.0,
+ "2380": 40054.0,
+ "2385": 40808.0,
+ "2390": 40109.0,
+ "2395": 40735.0,
+ "2400": 41566.0,
+ "2405": 40111.0,
+ "2410": 40801.0,
+ "2415": 39199.0,
+ "2420": 39970.0,
+ "2425": 37817.0,
+ "2430": 39274.0,
+ "2435": 41500.0,
+ "2440": 40028.0,
+ "2445": 39377.0,
+ "2450": 40793.0,
+ "2455": 42271.0,
+ "2460": 40111.0,
+ "2465": 39485.0,
+ "2470": 37801.0,
+ "2475": 40036.0,
+ "2480": 40044.0,
+ "2485": 39367.0,
+ "2490": 40127.0,
+ "2495": 41503.0,
+ "2500": 39203.0,
+ "2505": 40726.0,
+ "2510": 40732.0,
+ "2515": 38645.0,
+ "2520": 40804.0,
+ "2525": 41564.0,
+ "2530": 40743.0,
+ "2535": 39972.0,
+ "2540": 39211.0,
+ "2545": 37984.0,
+ "2550": 40797.0,
+ "2555": 41497.0,
+ "2560": 39449.0,
+ "2565": 39201.0,
+ "2570": 39356.0,
+ "2575": 40740.0,
+ "2580": 40730.0,
+ "2585": 40877.0,
+ "2590": 40114.0,
+ "2595": 37988.0,
+ "2600": 39964.0,
+ "2605": 38721.0,
+ "2610": 41570.0,
+ "2615": 40108.0,
+ "2620": 41492.0,
+ "2625": 39275.0,
+ "2630": 40028.0,
+ "2635": 39527.0,
+ "2640": 40805.0,
+ "2645": 40039.0,
+ "2650": 40046.0,
+ "2655": 39194.0,
+ "2660": 37249.0,
+ "2665": 40879.0,
+ "2670": 40874.0,
+ "2675": 38432.0,
+ "2680": 40046.0,
+ "2685": 41570.0,
+ "2690": 40047.0,
+ "2695": 40724.0,
+ "2700": 39294.0,
+ "2705": 41576.0,
+ "2710": 40790.0,
+ "2715": 38507.0,
+ "2720": 40036.0,
+ "2725": 38663.0,
+ "2730": 40862.0,
+ "2735": 40047.0,
+ "2740": 38589.0,
+ "2745": 40067.0,
+ "2750": 40797.0,
+ "2755": 38526.0,
+ "2760": 38511.0,
+ "2765": 39264.0,
+ "2770": 41578.0,
+ "2775": 39362.0,
+ "2780": 40754.0,
+ "2785": 40737.0,
+ "2790": 39280.0,
+ "2795": 40040.0,
+ "2800": 39423.0,
+ "2805": 40799.0,
+ "2810": 41582.0,
+ "2815": 42256.0,
+ "2820": 40732.0,
+ "2825": 40061.0,
+ "2830": 40816.0,
+ "2835": 41580.0,
+ "2840": 40807.0,
+ "2845": 38573.0,
+ "2850": 42258.0,
+ "2855": 40787.0,
+ "2860": 39360.0,
+ "2865": 39370.0,
+ "2870": 40206.0,
+ "2875": 42268.0,
+ "2880": 40809.0,
+ "2885": 39964.0,
+ "2890": 38603.0,
+ "2895": 40733.0,
+ "2900": 39352.0,
+ "2905": 39966.0,
+ "2910": 40732.0,
+ "2915": 40062.0,
+ "2920": 37901.0,
+ "2925": 40802.0,
+ "2930": 36481.0,
+ "2935": 41581.0,
+ "2940": 40793.0,
+ "2945": 38908.0,
+ "2950": 39431.0,
+ "2955": 38030.0,
+ "2960": 41489.0,
+ "2965": 39181.0,
+ "2970": 40795.0,
+ "2975": 38809.0,
+ "2980": 41502.0,
+ "2985": 39256.0,
+ "2990": 42328.0,
+ "2995": 40812.0,
+ "3000": 37945.0,
+ "3005": 42269.0,
+ "3010": 38724.0,
+ "3015": 40810.0,
+ "3020": 41492.0,
+ "3025": 39328.0,
+ "3030": 38444.0,
+ "3035": 40046.0,
+ "3040": 41558.0,
+ "3045": 39329.0,
+ "3050": 41503.0,
+ "3055": 40750.0,
+ "3060": 38507.0,
+ "3065": 39268.0,
+ "3070": 40802.0,
+ "3075": 37954.0,
+ "3080": 39205.0,
+ "3085": 41560.0,
+ "3090": 39275.0,
+ "3095": 39331.0,
+ "3100": 38578.0,
+ "3105": 39975.0,
+ "3110": 37137.0,
+ "3115": 38441.0,
+ "3120": 41657.0,
+ "3125": 40110.0,
+ "3130": 39275.0,
+ "3135": 39265.0,
+ "3140": 39986.0,
+ "3145": 40821.0,
+ "3150": 41508.0,
+ "3155": 38581.0,
+ "3160": 40925.0,
+ "3165": 41487.0,
+ "3170": 39269.0,
+ "3175": 37417.0,
+ "3180": 39430.0,
+ "3185": 42263.0,
+ "3190": 37945.0,
+ "3195": 39259.0,
+ "3200": 38657.0,
+ "3205": 40039.0,
+ "3210": 39984.0,
+ "3215": 38575.0,
+ "3220": 39948.0,
+ "3225": 40112.0,
+ "3230": 40096.0,
+ "3235": 42343.0,
+ "3240": 40045.0,
+ "3245": 40745.0,
+ "3250": 41492.0,
+ "3255": 42260.0,
+ "3260": 40123.0,
+ "3265": 42269.0,
+ "3270": 40797.0,
+ "3275": 41567.0,
+ "3280": 40182.0,
+ "3285": 39279.0,
+ "3290": 38666.0,
+ "3295": 40726.0,
+ "3300": 40189.0,
+ "3305": 42267.0,
+ "3310": 42352.0,
+ "3315": 41494.0,
+ "3320": 41485.0,
+ "3325": 39281.0,
+ "3330": 40810.0,
+ "3335": 40126.0,
+ "3340": 40787.0,
+ "3345": 40808.0,
+ "3350": 40816.0,
+ "3355": 41579.0,
+ "3360": 41497.0,
+ "3365": 40736.0,
+ "3370": 40032.0,
+ "3375": 40722.0,
+ "3380": 40036.0,
+ "3385": 39349.0,
+ "3390": 40061.0,
+ "3395": 39967.0,
+ "3400": 38155.0,
+ "3405": 38688.0,
+ "3410": 39968.0,
+ "3415": 40052.0,
+ "3420": 40797.0,
+ "3425": 40800.0,
+ "3430": 40742.0,
+ "3435": 40039.0,
+ "3440": 41589.0,
+ "3445": 37967.0,
+ "3450": 37915.0,
+ "3455": 40806.0,
+ "3460": 42264.0,
+ "3465": 39365.0,
+ "3470": 38572.0,
+ "3475": 40199.0,
+ "3480": 40806.0,
+ "3485": 38857.0,
+ "3490": 40044.0,
+ "3495": 40052.0,
+ "3500": 40051.0,
+ "3505": 40882.0,
+ "3510": 40805.0,
+ "3515": 40038.0,
+ "3520": 39272.0,
+ "3525": 41561.0,
+ "3530": 40727.0,
+ "3535": 41573.0,
+ "3540": 40746.0,
+ "3545": 40823.0,
+ "3550": 41576.0,
+ "3555": 40111.0,
+ "3560": 40738.0,
+ "3565": 40796.0,
+ "3570": 40820.0,
+ "3575": 40041.0,
+ "3580": 40037.0,
+ "3585": 41578.0,
+ "3590": 41504.0,
+ "3595": 39357.0,
+ "3600": 39348.0,
+ "3605": 39274.0,
+ "3610": 40021.0,
+ "3615": 42264.0,
+ "3620": 38655.0,
+ "3625": 39970.0,
+ "3630": 40800.0,
+ "3635": 40092.0,
+ "3640": 39211.0,
+ "3645": 39311.0,
+ "3650": 38657.0,
+ "3655": 40215.0,
+ "3660": 42329.0,
+ "3665": 41493.0,
+ "3670": 37362.0,
+ "3675": 39421.0,
+ "3680": 37326.0,
+ "3685": 39368.0,
+ "3690": 40794.0,
+ "3695": 42340.0,
+ "3700": 38164.0,
+ "3705": 39496.0,
+ "3710": 39285.0,
+ "3715": 38535.0,
+ "3720": 40805.0,
+ "3725": 37301.0,
+ "3730": 37813.0,
+ "3735": 40800.0,
+ "3740": 40800.0,
+ "3745": 38580.0,
+ "3750": 39399.0,
+ "3755": 39360.0,
+ "3760": 39355.0,
+ "3765": 39349.0,
+ "3770": 42346.0,
+ "3775": 40815.0,
+ "3780": 42262.0,
+ "3785": 37893.0,
+ "3790": 39346.0,
+ "3795": 40031.0,
+ "3800": 39476.0,
+ "3805": 37106.0,
+ "3810": 39420.0,
+ "3815": 40732.0,
+ "3820": 40112.0,
+ "3825": 42265.0,
+ "3830": 40028.0,
+ "3835": 40809.0,
+ "3840": 40041.0,
+ "3845": 39340.0,
+ "3850": 39417.0,
+ "3855": 39357.0,
+ "3860": 40037.0,
+ "3865": 41511.0,
+ "3870": 37744.0,
+ "3875": 39962.0,
+ "3880": 40048.0,
+ "3885": 41507.0,
+ "3890": 39963.0,
+ "3895": 40725.0,
+ "3900": 39273.0,
+ "3905": 38584.0,
+ "3910": 40023.0,
+ "3915": 40808.0,
+ "3920": 36492.0,
+ "3925": 39962.0,
+ "3930": 39265.0,
+ "3935": 39345.0,
+ "3940": 39419.0,
+ "3945": 41497.0,
+ "3950": 39977.0,
+ "3955": 39417.0,
+ "3960": 38680.0,
+ "3965": 39976.0,
+ "3970": 39287.0,
+ "3975": 39969.0,
+ "3980": 40821.0,
+ "3985": 38580.0,
+ "3990": 39227.0,
+ "3995": 39440.0,
+ "4000": 40027.0,
+ "4005": 40051.0,
+ "4010": 35696.0,
+ "4015": 41576.0,
+ "4020": 39290.0,
+ "4025": 40178.0,
+ "4030": 37080.0,
+ "4035": 41515.0,
+ "4040": 34461.0,
+ "4045": 40807.0,
+ "4050": 37770.0,
+ "4055": 39411.0,
+ "4060": 37301.0,
+ "4065": 40740.0,
+ "4070": 38509.0,
+ "4075": 40035.0,
+ "4080": 40883.0,
+ "4085": 41506.0,
+ "4090": 40728.0,
+ "4095": 41508.0,
+ "4100": 37921.0,
+ "4105": 41489.0,
+ "4110": 41505.0,
+ "4115": 40192.0,
+ "4120": 40798.0,
+ "4125": 38758.0,
+ "4130": 40731.0,
+ "4135": 39382.0,
+ "4140": 39349.0,
+ "4145": 40052.0,
+ "4150": 39363.0,
+ "4155": 39425.0,
+ "4160": 40211.0,
+ "4165": 39356.0,
+ "4170": 40743.0,
+ "4175": 39267.0,
+ "4180": 40821.0,
+ "4185": 40807.0,
+ "4190": 39262.0,
+ "4195": 40796.0,
+ "4200": 41496.0,
+ "4205": 40722.0,
+ "4210": 40734.0,
+ "4215": 40038.0,
+ "4220": 38680.0,
+ "4225": 40040.0,
+ "4230": 38826.0,
+ "4235": 39346.0,
+ "4240": 40811.0,
+ "4245": 40748.0,
+ "4250": 40107.0,
+ "4255": 39328.0,
+ "4260": 38631.0,
+ "4265": 40739.0,
+ "4270": 40817.0,
+ "4275": 40118.0,
+ "4280": 40029.0,
+ "4285": 41496.0,
+ "4290": 40740.0,
+ "4295": 37122.0,
+ "4300": 39345.0,
+ "4305": 40036.0,
+ "4310": 40116.0,
+ "4315": 40117.0,
+ "4320": 41566.0,
+ "4325": 40058.0,
+ "4330": 38702.0,
+ "4335": 42261.0,
+ "4340": 40727.0,
+ "4345": 40785.0,
+ "4350": 37967.0,
+ "4355": 41504.0,
+ "4360": 40188.0,
+ "4365": 40816.0,
+ "4370": 37221.0,
+ "4375": 40047.0,
+ "4380": 38500.0,
+ "4385": 40109.0,
+ "4390": 40732.0,
+ "4395": 39982.0,
+ "4400": 39973.0,
+ "4405": 40190.0,
+ "4410": 41595.0,
+ "4415": 41495.0,
+ "4420": 40034.0,
+ "4425": 40181.0,
+ "4430": 40731.0,
+ "4435": 37958.0,
+ "4440": 39972.0,
+ "4445": 39436.0,
+ "4450": 39961.0,
+ "4455": 40730.0,
+ "4460": 40802.0,
+ "4465": 41507.0,
+ "4470": 41496.0,
+ "4475": 39195.0,
+ "4480": 41568.0,
+ "4485": 40139.0,
+ "4490": 42263.0,
+ "4495": 39970.0,
+ "4500": 40035.0,
+ "4505": 40820.0,
+ "4510": 40804.0,
+ "4515": 37908.0,
+ "4520": 39274.0,
+ "4525": 40726.0,
+ "4530": 39970.0,
+ "4535": 39348.0,
+ "4540": 39411.0,
+ "4545": 38730.0,
+ "4550": 40028.0,
+ "4555": 40799.0,
+ "4560": 39327.0,
+ "4565": 40099.0,
+ "4570": 41642.0,
+ "4575": 40053.0,
+ "4580": 37788.0,
+ "4585": 39433.0,
+ "4590": 37282.0,
+ "4595": 39344.0,
+ "4600": 40042.0,
+ "4605": 40756.0,
+ "4610": 40803.0,
+ "4615": 40737.0,
+ "4620": 40734.0,
+ "4625": 41500.0,
+ "4630": 40057.0,
+ "4635": 40753.0,
+ "4640": 41517.0,
+ "4645": 41497.0,
+ "4650": 40820.0,
+ "4655": 39983.0,
+ "4660": 40878.0,
+ "4665": 41499.0,
+ "4670": 39290.0,
+ "4675": 40051.0,
+ "4680": 41578.0,
+ "4685": 38040.0,
+ "4690": 40742.0,
+ "4695": 39287.0,
+ "4700": 39968.0,
+ "4705": 40801.0,
+ "4710": 42261.0,
+ "4715": 41571.0,
+ "4720": 40888.0,
+ "4725": 40823.0,
+ "4730": 38524.0,
+ "4735": 40045.0,
+ "4740": 37841.0,
+ "4745": 39427.0,
+ "4750": 40040.0,
+ "4755": 39206.0,
+ "4760": 40123.0,
+ "4765": 41490.0,
+ "4770": 40103.0,
+ "4775": 36650.0,
+ "4780": 39281.0,
+ "4785": 40053.0,
+ "4790": 42354.0,
+ "4795": 39288.0,
+ "4800": 42270.0,
+ "4805": 39454.0,
+ "4810": 40055.0,
+ "4815": 42259.0,
+ "4820": 39349.0,
+ "4825": 37841.0,
+ "4830": 40116.0,
+ "4835": 41510.0,
+ "4840": 40034.0,
+ "4845": 41571.0,
+ "4850": 38652.0,
+ "4855": 40177.0,
+ "4860": 38646.0,
+ "4865": 40034.0,
+ "4870": 41583.0,
+ "4875": 40750.0,
+ "4880": 41547.0,
+ "4885": 40735.0,
+ "4890": 40731.0,
+ "4895": 40793.0,
+ "4900": 40033.0,
+ "4905": 40804.0,
+ "4910": 40036.0,
+ "4915": 40806.0,
+ "4920": 40041.0,
+ "4925": 40049.0,
+ "4930": 39972.0,
+ "4935": 40814.0,
+ "4940": 40000.0,
+ "4945": 39421.0,
+ "4950": 40036.0,
+ "4955": 39993.0,
+ "4960": 38585.0,
+ "4965": 38678.0,
+ "4970": 39342.0,
+ "4975": 39968.0,
+ "4980": 37905.0,
+ "4985": 37751.0,
+ "4990": 39358.0,
+ "4995": 40033.0,
+ "5000": 38583.0,
+ "5005": 40053.0,
+ "5010": 38651.0,
+ "5015": 40733.0,
+ "5020": 39209.0,
+ "5025": 41499.0,
+ "5030": 36510.0,
+ "5035": 37972.0,
+ "5040": 40725.0,
+ "5045": 39218.0,
+ "5050": 39280.0,
+ "5055": 39997.0,
+ "5060": 40795.0,
+ "5065": 38572.0,
+ "5070": 42255.0,
+ "5075": 40031.0,
+ "5080": 41497.0,
+ "5085": 39282.0,
+ "5090": 40811.0,
+ "5095": 40895.0,
+ "5100": 39340.0,
+ "5105": 40061.0,
+ "5110": 40814.0,
+ "5115": 38576.0,
+ "5120": 40113.0,
+ "5125": 41500.0,
+ "5130": 40116.0,
+ "5135": 41504.0,
+ "5140": 41572.0,
+ "5145": 38071.0,
+ "5150": 39353.0,
+ "5155": 40011.0,
+ "5160": 41587.0,
+ "5165": 39968.0,
+ "5170": 40748.0,
+ "5175": 38839.0,
+ "5180": 41495.0,
+ "5185": 40801.0,
+ "5190": 38665.0,
+ "5195": 38602.0,
+ "5200": 40122.0,
+ "5205": 42265.0,
+ "5210": 39266.0,
+ "5215": 37204.0,
+ "5220": 40039.0,
+ "5225": 41589.0,
+ "5230": 40184.0,
+ "5235": 40032.0,
+ "5240": 38656.0,
+ "5245": 40823.0,
+ "5250": 38501.0,
+ "5255": 40054.0,
+ "5260": 40804.0,
+ "5265": 39267.0,
+ "5270": 41509.0,
+ "5275": 41503.0,
+ "5280": 39295.0,
+ "5285": 40045.0,
+ "5290": 38715.0,
+ "5295": 39357.0,
+ "5300": 39399.0,
+ "5305": 39288.0,
+ "5310": 41571.0,
+ "5315": 39226.0,
+ "5320": 40903.0,
+ "5325": 40817.0,
+ "5330": 39356.0,
+ "5335": 40106.0,
+ "5340": 42257.0,
+ "5345": 39968.0,
+ "5350": 40815.0,
+ "5355": 40145.0,
+ "5360": 37251.0,
+ "5365": 37901.0,
+ "5370": 40798.0,
+ "5375": 40016.0,
+ "5380": 39346.0,
+ "5385": 39364.0,
+ "5390": 38058.0,
+ "5395": 39445.0,
+ "5400": 38713.0,
+ "5405": 39355.0,
+ "5410": 40050.0,
+ "5415": 40732.0,
+ "5420": 39978.0,
+ "5425": 40111.0,
+ "5430": 40815.0,
+ "5435": 42257.0,
+ "5440": 39277.0,
+ "5445": 38589.0,
+ "5450": 40035.0,
+ "5455": 40150.0,
+ "5460": 41486.0,
+ "5465": 40880.0,
+ "5470": 39399.0,
+ "5475": 39424.0,
+ "5480": 40033.0,
+ "5485": 41498.0,
+ "5490": 39435.0,
+ "5495": 40798.0,
+ "5500": 40889.0,
+ "5505": 37780.0,
+ "5510": 42255.0,
+ "5515": 39278.0,
+ "5520": 40737.0,
+ "5525": 40803.0,
+ "5530": 39265.0,
+ "5535": 40749.0,
+ "5540": 40131.0,
+ "5545": 41575.0,
+ "5550": 38748.0,
+ "5555": 39965.0,
+ "5560": 40228.0,
+ "5565": 42335.0,
+ "5570": 41490.0,
+ "5575": 39418.0,
+ "5580": 39354.0,
+ "5585": 40727.0,
+ "5590": 38566.0,
+ "5595": 40062.0,
+ "5600": 39344.0,
+ "5605": 41586.0,
+ "5610": 40811.0,
+ "5615": 40793.0,
+ "5620": 39422.0,
+ "5625": 39400.0,
+ "5630": 40106.0,
+ "5635": 39352.0,
+ "5640": 40808.0,
+ "5645": 40743.0,
+ "5650": 38609.0,
+ "5655": 39284.0,
+ "5660": 38735.0,
+ "5665": 40802.0,
+ "5670": 41570.0,
+ "5675": 40828.0,
+ "5680": 39348.0,
+ "5685": 40730.0,
+ "5690": 39966.0,
+ "5695": 40798.0,
+ "5700": 39351.0,
+ "5705": 40045.0,
+ "5710": 39351.0,
+ "5715": 40799.0,
+ "5720": 40885.0,
+ "5725": 39979.0,
+ "5730": 39985.0,
+ "5735": 39972.0,
+ "5740": 40028.0,
+ "5745": 40808.0,
+ "5750": 37157.0,
+ "5755": 41510.0,
+ "5760": 40806.0,
+ "5765": 37121.0,
+ "5770": 40729.0,
+ "5775": 40070.0,
+ "5780": 38649.0,
+ "5785": 42342.0,
+ "5790": 38091.0,
+ "5795": 40870.0,
+ "5800": 40809.0,
+ "5805": 39973.0,
+ "5810": 43030.0,
+ "5815": 39278.0,
+ "5820": 40738.0,
+ "5825": 35592.0,
+ "5830": 39417.0,
+ "5835": 40054.0,
+ "5840": 40053.0,
+ "5845": 40804.0,
+ "5850": 41502.0,
+ "5855": 42268.0,
+ "5860": 38529.0,
+ "5865": 39366.0,
+ "5870": 39421.0,
+ "5875": 40047.0,
+ "5880": 39364.0,
+ "5885": 40784.0,
+ "5890": 37897.0,
+ "5895": 40745.0,
+ "5900": 40051.0,
+ "5905": 38666.0,
+ "5910": 37984.0,
+ "5915": 37835.0,
+ "5920": 40804.0,
+ "5925": 37929.0,
+ "5930": 40900.0,
+ "5935": 39283.0,
+ "5940": 40139.0,
+ "5945": 39342.0,
+ "5950": 40124.0,
+ "5955": 39355.0,
+ "5960": 40033.0,
+ "5965": 38761.0,
+ "5970": 38441.0,
+ "5975": 36488.0,
+ "5980": 40741.0,
+ "5985": 40038.0,
+ "5990": 40804.0,
+ "5995": 38582.0,
+ "6000": 38738.0,
+ "6005": 40039.0,
+ "6010": 39960.0,
+ "6015": 40035.0,
+ "6020": 41581.0,
+ "6025": 40043.0,
+ "6030": 39356.0,
+ "6035": 41576.0,
+ "6040": 38611.0,
+ "6045": 39972.0,
+ "6050": 39972.0,
+ "6055": 41580.0,
+ "6060": 40119.0,
+ "6065": 41587.0,
+ "6070": 38750.0,
+ "6075": 41499.0,
+ "6080": 37962.0,
+ "6085": 36000.0,
+ "6090": 40734.0,
+ "6095": 41493.0,
+ "6100": 40739.0,
+ "6105": 35900.0,
+ "6110": 40126.0,
+ "6115": 41499.0,
+ "6120": 40127.0,
+ "6125": 37229.0,
+ "6130": 39260.0,
+ "6135": 39288.0,
+ "6140": 41567.0,
+ "6145": 40048.0,
+ "6150": 39293.0,
+ "6155": 41499.0,
+ "6160": 39960.0,
+ "6165": 40731.0,
+ "6170": 37252.0,
+ "6175": 40813.0,
+ "6180": 37815.0,
+ "6185": 40813.0,
+ "6190": 41515.0,
+ "6195": 38585.0,
+ "6200": 41485.0,
+ "6205": 41596.0,
+ "6210": 38744.0,
+ "6215": 41497.0,
+ "6220": 39974.0,
+ "6225": 40753.0,
+ "6230": 37897.0,
+ "6235": 40819.0,
+ "6240": 40734.0,
+ "6245": 40181.0,
+ "6250": 41493.0,
+ "6255": 40737.0,
+ "6260": 41589.0,
+ "6265": 39221.0,
+ "6270": 40807.0,
+ "6275": 40869.0,
+ "6280": 43039.0,
+ "6285": 40809.0,
+ "6290": 41574.0,
+ "6295": 40790.0,
+ "6300": 39971.0,
+ "6305": 38666.0,
+ "6310": 38672.0,
+ "6315": 41507.0,
+ "6320": 40877.0,
+ "6325": 37684.0,
+ "6330": 39284.0,
+ "6335": 41586.0,
+ "6340": 40749.0,
+ "6345": 41512.0,
+ "6350": 41568.0,
+ "6355": 40106.0,
+ "6360": 38747.0,
+ "6365": 40805.0,
+ "6370": 38727.0,
+ "6375": 40062.0,
+ "6380": 37124.0,
+ "6385": 40186.0,
+ "6390": 39498.0,
+ "6395": 41496.0,
+ "6400": 36579.0,
+ "6405": 40054.0,
+ "6410": 40144.0,
+ "6415": 41558.0,
+ "6420": 40810.0,
+ "6425": 37279.0,
+ "6430": 39970.0,
+ "6435": 40113.0,
+ "6440": 41491.0,
+ "6445": 40729.0,
+ "6450": 38500.0,
+ "6455": 38603.0,
+ "6460": 39225.0,
+ "6465": 42263.0,
+ "6470": 41501.0,
+ "6475": 40815.0,
+ "6480": 40105.0,
+ "6485": 39275.0,
+ "6490": 35738.0,
+ "6495": 39283.0,
+ "6500": 41499.0,
+ "6505": 40740.0,
+ "6510": 40059.0,
+ "6515": 40110.0,
+ "6520": 40816.0,
+ "6525": 42264.0,
+ "6530": 41500.0,
+ "6535": 39285.0,
+ "6540": 40900.0,
+ "6545": 37985.0,
+ "6550": 39343.0,
+ "6555": 40804.0,
+ "6560": 35719.0,
+ "6565": 39208.0,
+ "6570": 40082.0,
+ "6575": 41504.0,
+ "6580": 41575.0,
+ "6585": 40051.0,
+ "6590": 35482.0,
+ "6595": 40804.0,
+ "6600": 39271.0,
+ "6605": 39356.0,
+ "6610": 41506.0,
+ "6615": 40819.0,
+ "6620": 41573.0,
+ "6625": 40728.0,
+ "6630": 37837.0,
+ "6635": 38687.0,
+ "6640": 40813.0,
+ "6645": 38047.0,
+ "6650": 40027.0,
+ "6655": 40062.0,
+ "6660": 41505.0,
+ "6665": 41496.0,
+ "6670": 39263.0,
+ "6675": 39343.0,
+ "6680": 42267.0,
+ "6685": 40733.0,
+ "6690": 40034.0,
+ "6695": 40728.0,
+ "6700": 40811.0,
+ "6705": 41576.0,
+ "6710": 39199.0,
+ "6715": 38511.0,
+ "6720": 39962.0,
+ "6725": 41569.0,
+ "6730": 41489.0,
+ "6735": 38649.0,
+ "6740": 37288.0,
+ "6745": 41492.0,
+ "6750": 40747.0,
+ "6755": 38608.0,
+ "6760": 39339.0,
+ "6765": 39367.0,
+ "6770": 39975.0,
+ "6775": 38645.0,
+ "6780": 38596.0,
+ "6785": 40057.0,
+ "6790": 40732.0,
+ "6795": 40740.0,
+ "6800": 40722.0,
+ "6805": 40815.0,
+ "6810": 40197.0,
+ "6815": 40806.0,
+ "6820": 40119.0,
+ "6825": 40808.0,
+ "6830": 41556.0,
+ "6835": 38574.0,
+ "6840": 40802.0,
+ "6845": 42326.0,
+ "6850": 40126.0,
+ "6855": 39363.0,
+ "6860": 41493.0,
+ "6865": 40733.0,
+ "6870": 40047.0,
+ "6875": 40797.0,
+ "6880": 40798.0,
+ "6885": 39355.0,
+ "6890": 40045.0,
+ "6895": 40160.0,
+ "6900": 37765.0,
+ "6905": 40883.0,
+ "6910": 41578.0,
+ "6915": 41581.0,
+ "6920": 40047.0,
+ "6925": 40103.0,
+ "6930": 42278.0,
+ "6935": 38503.0,
+ "6940": 39432.0,
+ "6945": 40825.0,
+ "6950": 39968.0,
+ "6955": 39432.0,
+ "6960": 40731.0,
+ "6965": 40825.0,
+ "6970": 40821.0,
+ "6975": 41501.0,
+ "6980": 39537.0,
+ "6985": 40811.0,
+ "6990": 40133.0,
+ "6995": 39963.0,
+ "7000": 39434.0,
+ "7005": 39451.0,
+ "7010": 39977.0,
+ "7015": 38654.0,
+ "7020": 40814.0,
+ "7025": 39267.0,
+ "7030": 38658.0,
+ "7035": 39348.0,
+ "7040": 42325.0,
+ "7045": 36574.0,
+ "7050": 38603.0,
+ "7055": 40889.0,
+ "7060": 40737.0,
+ "7065": 37990.0,
+ "7070": 40051.0,
+ "7075": 39261.0,
+ "7080": 40736.0,
+ "7085": 40733.0,
+ "7090": 36588.0,
+ "7095": 40732.0,
+ "7100": 40798.0,
+ "7105": 40729.0,
+ "7110": 42335.0,
+ "7115": 39282.0,
+ "7120": 40804.0,
+ "7125": 38502.0,
+ "7130": 40910.0,
+ "7135": 39309.0,
+ "7140": 40102.0,
+ "7145": 40739.0,
+ "7150": 40814.0,
+ "7155": 42342.0,
+ "7160": 40059.0,
+ "7165": 40017.0,
+ "7170": 41497.0,
+ "7175": 40816.0,
+ "7180": 40822.0,
+ "7185": 39294.0,
+ "7190": 40812.0,
+ "7195": 39973.0,
+ "7200": 40091.0,
+ "7205": 40814.0,
+ "7210": 41576.0,
+ "7215": 41561.0,
+ "7220": 37369.0,
+ "7225": 40815.0,
+ "7230": 40117.0,
+ "7235": 41492.0,
+ "7240": 38585.0,
+ "7245": 39435.0,
+ "7250": 40820.0,
+ "7255": 39967.0,
+ "7260": 38542.0,
+ "7265": 39357.0,
+ "7270": 40728.0,
+ "7275": 38598.0,
+ "7280": 39457.0,
+ "7285": 40121.0,
+ "7290": 41586.0,
+ "7295": 40728.0,
+ "7300": 40023.0,
+ "7305": 39530.0,
+ "7310": 40121.0,
+ "7315": 40888.0,
+ "7320": 40112.0,
+ "7325": 40810.0,
+ "7330": 41493.0,
+ "7335": 39342.0,
+ "7340": 40796.0,
+ "7345": 40132.0,
+ "7350": 39342.0,
+ "7355": 40730.0,
+ "7360": 40178.0,
+ "7365": 41492.0,
+ "7370": 38558.0,
+ "7375": 38436.0,
+ "7380": 40742.0,
+ "7385": 40109.0,
+ "7390": 38657.0,
+ "7395": 38463.0,
+ "7400": 41504.0,
+ "7405": 41503.0,
+ "7410": 38494.0,
+ "7415": 38500.0,
+ "7420": 39359.0,
+ "7425": 41499.0,
+ "7430": 39394.0,
+ "7435": 39972.0,
+ "7440": 40107.0,
+ "7445": 40731.0,
+ "7450": 40803.0,
+ "7455": 42257.0,
+ "7460": 38795.0,
+ "7465": 35607.0,
+ "7470": 40794.0,
+ "7475": 37826.0,
+ "7480": 40895.0,
+ "7485": 37963.0,
+ "7490": 40793.0,
+ "7495": 40130.0,
+ "7500": 41499.0,
+ "7505": 40806.0,
+ "7510": 39206.0,
+ "7515": 40037.0,
+ "7520": 40745.0,
+ "7525": 40098.0,
+ "7530": 40812.0,
+ "7535": 39279.0,
+ "7540": 40052.0,
+ "7545": 38452.0,
+ "7550": 40745.0,
+ "7555": 40112.0,
+ "7560": 41499.0,
+ "7565": 40047.0,
+ "7570": 40733.0,
+ "7575": 40743.0,
+ "7580": 39969.0,
+ "7585": 40733.0,
+ "7590": 40799.0,
+ "7595": 40795.0,
+ "7600": 35669.0,
+ "7605": 40737.0,
+ "7610": 41494.0,
+ "7615": 38472.0,
+ "7620": 41567.0,
+ "7625": 40721.0,
+ "7630": 41569.0,
+ "7635": 41518.0,
+ "7640": 40112.0,
+ "7645": 40099.0,
+ "7650": 38666.0,
+ "7655": 41569.0,
+ "7660": 39967.0,
+ "7665": 41595.0,
+ "7670": 39972.0,
+ "7675": 39434.0,
+ "7680": 40745.0,
+ "7685": 39333.0,
+ "7690": 40109.0,
+ "7695": 40045.0,
+ "7700": 41500.0,
+ "7705": 40206.0,
+ "7710": 40794.0,
+ "7715": 40024.0,
+ "7720": 35779.0,
+ "7725": 39350.0,
+ "7730": 39375.0,
+ "7735": 37974.0,
+ "7740": 40872.0,
+ "7745": 40740.0,
+ "7750": 41501.0,
+ "7755": 39428.0,
+ "7760": 40117.0,
+ "7765": 39360.0,
+ "7770": 39309.0,
+ "7775": 39354.0,
+ "7780": 38591.0,
+ "7785": 38750.0,
+ "7790": 39356.0,
+ "7795": 40044.0,
+ "7800": 40116.0,
+ "7805": 38044.0,
+ "7810": 42343.0,
+ "7815": 38572.0,
+ "7820": 40802.0,
+ "7825": 37884.0,
+ "7830": 40862.0,
+ "7835": 39363.0,
+ "7840": 40829.0,
+ "7845": 41508.0,
+ "7850": 39334.0,
+ "7855": 38659.0,
+ "7860": 39972.0,
+ "7865": 38867.0,
+ "7870": 40817.0,
+ "7875": 40105.0,
+ "7880": 40722.0,
+ "7885": 40722.0,
+ "7890": 40041.0,
+ "7895": 42261.0,
+ "7900": 40100.0,
+ "7905": 41503.0,
+ "7910": 37886.0,
+ "7915": 39274.0,
+ "7920": 39356.0,
+ "7925": 42265.0,
+ "7930": 39287.0,
+ "7935": 41563.0,
+ "7940": 39358.0,
+ "7945": 41558.0,
+ "7950": 36560.0,
+ "7955": 37921.0,
+ "7960": 40048.0,
+ "7965": 40808.0,
+ "7970": 40877.0,
+ "7975": 37771.0,
+ "7980": 41492.0,
+ "7985": 39964.0,
+ "7990": 39406.0,
+ "7995": 39359.0,
+ "8000": 41510.0,
+ "8005": 40803.0,
+ "8010": 40732.0,
+ "8015": 40886.0,
+ "8020": 38664.0,
+ "8025": 40886.0,
+ "8030": 37818.0,
+ "8035": 40052.0,
+ "8040": 39227.0,
+ "8045": 41565.0,
+ "8050": 38607.0,
+ "8055": 39272.0,
+ "8060": 39280.0,
+ "8065": 40812.0,
+ "8070": 40809.0,
+ "8075": 41548.0,
+ "8080": 41583.0,
+ "8085": 39307.0,
+ "8090": 40809.0,
+ "8095": 39365.0,
+ "8100": 40741.0,
+ "8105": 40846.0,
+ "8110": 38689.0,
+ "8115": 40805.0,
+ "8120": 39345.0,
+ "8125": 39977.0,
+ "8130": 39281.0,
+ "8135": 41499.0,
+ "8140": 39965.0,
+ "8145": 40824.0,
+ "8150": 41585.0,
+ "8155": 41513.0,
+ "8160": 39498.0,
+ "8165": 40051.0,
+ "8170": 38764.0,
+ "8175": 40034.0,
+ "8180": 37396.0,
+ "8185": 40739.0,
+ "8190": 39360.0,
+ "8195": 40046.0,
+ "8200": 40101.0,
+ "8205": 40045.0,
+ "8210": 39281.0,
+ "8215": 40805.0,
+ "8220": 38579.0,
+ "8225": 39272.0,
+ "8230": 40815.0,
+ "8235": 40044.0,
+ "8240": 39356.0,
+ "8245": 38655.0,
+ "8250": 41552.0,
+ "8255": 39421.0,
+ "8260": 40823.0,
+ "8265": 41507.0,
+ "8270": 42274.0,
+ "8275": 40741.0,
+ "8280": 40049.0,
+ "8285": 40895.0,
+ "8290": 40111.0,
+ "8295": 39285.0,
+ "8300": 40807.0,
+ "8305": 40137.0,
+ "8310": 39276.0,
+ "8315": 39215.0,
+ "8320": 41500.0,
+ "8325": 39290.0,
+ "8330": 40735.0,
+ "8335": 40038.0,
+ "8340": 40049.0,
+ "8345": 39437.0,
+ "8350": 40799.0,
+ "8355": 40042.0,
+ "8360": 40809.0,
+ "8365": 39975.0,
+ "8370": 39441.0,
+ "8375": 38870.0,
+ "8380": 40737.0,
+ "8385": 40049.0,
+ "8390": 40817.0,
+ "8395": 39980.0,
+ "8400": 40125.0,
+ "8405": 39238.0,
+ "8410": 41496.0,
+ "8415": 38601.0,
+ "8420": 40132.0,
+ "8425": 39967.0,
+ "8430": 37828.0,
+ "8435": 39425.0,
+ "8440": 37094.0,
+ "8445": 38597.0,
+ "8450": 39433.0,
+ "8455": 39976.0,
+ "8460": 38029.0,
+ "8465": 37883.0,
+ "8470": 38582.0,
+ "8475": 40810.0,
+ "8480": 40205.0,
+ "8485": 37675.0,
+ "8490": 41500.0,
+ "8495": 40029.0,
+ "8500": 39971.0,
+ "8505": 38431.0,
+ "8510": 41518.0,
+ "8515": 39411.0,
+ "8520": 39437.0,
+ "8525": 38524.0,
+ "8530": 40834.0,
+ "8535": 40747.0,
+ "8540": 40737.0,
+ "8545": 39359.0,
+ "8550": 40815.0,
+ "8555": 40038.0,
+ "8560": 40718.0,
+ "8565": 39969.0,
+ "8570": 39279.0,
+ "8575": 40047.0,
+ "8580": 40140.0,
+ "8585": 40046.0,
+ "8590": 40801.0,
+ "8595": 40038.0,
+ "8600": 40196.0,
+ "8605": 39300.0,
+ "8610": 40882.0,
+ "8615": 38595.0,
+ "8620": 40052.0,
+ "8625": 42259.0,
+ "8630": 40737.0,
+ "8635": 39287.0,
+ "8640": 40731.0,
+ "8645": 40055.0,
+ "8650": 37213.0,
+ "8655": 40122.0,
+ "8660": 38668.0,
+ "8665": 40733.0,
+ "8670": 40807.0,
+ "8675": 40074.0,
+ "8680": 38516.0,
+ "8685": 42263.0,
+ "8690": 37913.0,
+ "8695": 39290.0,
+ "8700": 38688.0,
+ "8705": 40818.0,
+ "8710": 40029.0,
+ "8715": 40046.0,
+ "8720": 41498.0,
+ "8725": 39347.0,
+ "8730": 40804.0,
+ "8735": 38649.0,
+ "8740": 39281.0,
+ "8745": 38658.0,
+ "8750": 38578.0,
+ "8755": 40887.0,
+ "8760": 40732.0,
+ "8765": 39210.0,
+ "8770": 41579.0,
+ "8775": 40812.0,
+ "8780": 40813.0,
+ "8785": 39270.0,
+ "8790": 41495.0,
+ "8795": 40808.0,
+ "8800": 41569.0,
+ "8805": 41502.0,
+ "8810": 39291.0,
+ "8815": 40733.0,
+ "8820": 39255.0,
+ "8825": 41513.0,
+ "8830": 42263.0,
+ "8835": 40819.0,
+ "8840": 39328.0,
+ "8845": 42279.0,
+ "8850": 41506.0,
+ "8855": 42271.0,
+ "8860": 39364.0,
+ "8865": 39276.0,
+ "8870": 41565.0,
+ "8875": 38738.0,
+ "8880": 38648.0,
+ "8885": 38580.0,
+ "8890": 39219.0,
+ "8895": 40043.0,
+ "8900": 40043.0,
+ "8905": 38498.0,
+ "8910": 40743.0,
+ "8915": 37904.0,
+ "8920": 39271.0,
+ "8925": 39263.0,
+ "8930": 42262.0,
+ "8935": 37273.0,
+ "8940": 41575.0,
+ "8945": 41501.0,
+ "8950": 40061.0,
+ "8955": 41513.0,
+ "8960": 41506.0,
+ "8965": 42264.0,
+ "8970": 37908.0,
+ "8975": 36535.0,
+ "8980": 40803.0,
+ "8985": 41510.0,
+ "8990": 40743.0,
+ "8995": 39349.0,
+ "9000": 40732.0,
+ "9005": 39277.0,
+ "9010": 36505.0,
+ "9015": 39353.0,
+ "9020": 40038.0,
+ "9025": 39978.0,
+ "9030": 40042.0,
+ "9035": 40134.0,
+ "9040": 40822.0,
+ "9045": 39975.0,
+ "9050": 40111.0,
+ "9055": 40825.0,
+ "9060": 38532.0,
+ "9065": 38614.0,
+ "9070": 38565.0,
+ "9075": 40138.0,
+ "9080": 40112.0,
+ "9085": 40725.0,
+ "9090": 40030.0,
+ "9095": 39968.0,
+ "9100": 38522.0,
+ "9105": 38669.0,
+ "9110": 40101.0,
+ "9115": 39273.0,
+ "9120": 34521.0,
+ "9125": 39495.0,
+ "9130": 42345.0,
+ "9135": 39275.0,
+ "9140": 39190.0,
+ "9145": 40203.0,
+ "9150": 39279.0,
+ "9155": 42253.0,
+ "9160": 40822.0,
+ "9165": 40037.0,
+ "9170": 40052.0,
+ "9175": 38607.0,
+ "9180": 39978.0,
+ "9185": 40129.0,
+ "9190": 40051.0,
+ "9195": 40044.0,
+ "9200": 38003.0,
+ "9205": 40733.0,
+ "9210": 39282.0,
+ "9215": 40732.0,
+ "9220": 40810.0,
+ "9225": 40830.0,
+ "9230": 40824.0,
+ "9235": 38806.0,
+ "9240": 41507.0,
+ "9245": 41579.0,
+ "9250": 40729.0,
+ "9255": 40135.0,
+ "9260": 40216.0,
+ "9265": 41502.0,
+ "9270": 41501.0,
+ "9275": 41579.0,
+ "9280": 37282.0,
+ "9285": 40966.0,
+ "9290": 35915.0,
+ "9295": 40821.0,
+ "9300": 40813.0,
+ "9305": 39435.0,
+ "9310": 40736.0,
+ "9315": 40818.0,
+ "9320": 38593.0,
+ "9325": 40128.0,
+ "9330": 40808.0,
+ "9335": 38505.0,
+ "9340": 41496.0,
+ "9345": 40798.0,
+ "9350": 40819.0,
+ "9355": 38449.0,
+ "9360": 42271.0,
+ "9365": 38668.0,
+ "9370": 41499.0,
+ "9375": 40806.0,
+ "9380": 40819.0,
+ "9385": 40742.0,
+ "9390": 41578.0,
+ "9395": 40890.0,
+ "9400": 40809.0,
+ "9405": 40043.0,
+ "9410": 40827.0,
+ "9415": 40034.0,
+ "9420": 40749.0,
+ "9425": 41512.0,
+ "9430": 40742.0,
+ "9435": 39419.0,
+ "9440": 41496.0,
+ "9445": 40801.0,
+ "9450": 39353.0,
+ "9455": 42326.0,
+ "9460": 40809.0,
+ "9465": 42258.0,
+ "9470": 35874.0,
+ "9475": 40796.0,
+ "9480": 41577.0,
+ "9485": 39442.0,
+ "9490": 39278.0,
+ "9495": 38590.0,
+ "9500": 41510.0,
+ "9505": 40036.0,
+ "9510": 41597.0,
+ "9515": 40100.0,
+ "9520": 39281.0,
+ "9525": 39494.0,
+ "9530": 41564.0,
+ "9535": 38670.0,
+ "9540": 39281.0,
+ "9545": 39283.0,
+ "9550": 41577.0,
+ "9555": 40735.0,
+ "9560": 39342.0,
+ "9565": 40047.0,
+ "9570": 39404.0,
+ "9575": 42273.0,
+ "9580": 40065.0,
+ "9585": 40042.0,
+ "9590": 38506.0,
+ "9595": 38665.0,
+ "9600": 37197.0,
+ "9605": 40815.0,
+ "9610": 41573.0,
+ "9615": 39989.0,
+ "9620": 39969.0,
+ "9625": 40813.0,
+ "9630": 40800.0,
+ "9635": 38431.0,
+ "9640": 40790.0,
+ "9645": 39963.0,
+ "9650": 41494.0,
+ "9655": 42265.0,
+ "9660": 40826.0,
+ "9665": 40041.0,
+ "9670": 40054.0,
+ "9675": 38721.0,
+ "9680": 40739.0,
+ "9685": 39979.0,
+ "9690": 39407.0,
+ "9695": 39273.0,
+ "9700": 39223.0,
+ "9705": 40873.0,
+ "9710": 39282.0,
+ "9715": 39336.0,
+ "9720": 40181.0,
+ "9725": 38694.0,
+ "9730": 40826.0,
+ "9735": 41583.0,
+ "9740": 41577.0,
+ "9745": 39350.0,
+ "9750": 40802.0,
+ "9755": 40906.0,
+ "9760": 39310.0,
+ "9765": 43040.0,
+ "9770": 39403.0,
+ "9775": 39963.0,
+ "9780": 40144.0,
+ "9785": 40885.0,
+ "9790": 39205.0,
+ "9795": 42274.0,
+ "9800": 37192.0,
+ "9805": 39986.0,
+ "9810": 40820.0,
+ "9815": 40803.0,
+ "9820": 39270.0,
+ "9825": 40806.0,
+ "9830": 39496.0,
+ "9835": 39265.0,
+ "9840": 37894.0,
+ "9845": 39977.0,
+ "9850": 39283.0,
+ "9855": 38526.0,
+ "9860": 37943.0,
+ "9865": 41508.0,
+ "9870": 39340.0,
+ "9875": 37139.0,
+ "9880": 40060.0,
+ "9885": 39212.0,
+ "9890": 40065.0,
+ "9895": 41587.0,
+ "9900": 39974.0,
+ "9905": 41589.0,
+ "9910": 41584.0,
+ "9915": 41504.0,
+ "9920": 39987.0,
+ "9925": 38664.0,
+ "9930": 40051.0,
+ "9935": 37914.0,
+ "9940": 41633.0,
+ "9945": 41503.0,
+ "9950": 40831.0,
+ "9955": 41505.0,
+ "9960": 40808.0,
+ "9965": 40807.0,
+ "9970": 39375.0,
+ "9975": 40745.0,
+ "9980": 41506.0,
+ "9985": 42271.0,
+ "9990": 38543.0,
+ "9995": 42270.0,
+ "10000": 40036.0
+ }
+ },
+ "mem-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 10000,
+ "step_interval": 5,
+ "values": {
+ "1": 1131332608.0,
+ "5": 1131332608.0,
+ "10": 1131332608.0,
+ "15": 1131332608.0,
+ "20": 1131332608.0,
+ "25": 1131332608.0,
+ "30": 1131332608.0,
+ "35": 1131332608.0,
+ "40": 1131332608.0,
+ "45": 1131332608.0,
+ "50": 1131332608.0,
+ "55": 1131332608.0,
+ "60": 1131332608.0,
+ "65": 1131332608.0,
+ "70": 1131332608.0,
+ "75": 1131332608.0,
+ "80": 1131332608.0,
+ "85": 1131332608.0,
+ "90": 1131332608.0,
+ "95": 1131332608.0,
+ "100": 1131332608.0,
+ "105": 1131332608.0,
+ "110": 1131332608.0,
+ "115": 1131332608.0,
+ "120": 1131332608.0,
+ "125": 1131332608.0,
+ "130": 1131332608.0,
+ "135": 1131332608.0,
+ "140": 1131332608.0,
+ "145": 1131332608.0,
+ "150": 1131332608.0,
+ "155": 1131332608.0,
+ "160": 1131332608.0,
+ "165": 1131332608.0,
+ "170": 1131332608.0,
+ "175": 1131332608.0,
+ "180": 1131332608.0,
+ "185": 1131332608.0,
+ "190": 1131332608.0,
+ "195": 1131332608.0,
+ "200": 1131332608.0,
+ "205": 1131332608.0,
+ "210": 1131332608.0,
+ "215": 1131332608.0,
+ "220": 1131332608.0,
+ "225": 1131332608.0,
+ "230": 1131332608.0,
+ "235": 1131332608.0,
+ "240": 1131332608.0,
+ "245": 1131332608.0,
+ "250": 1131332608.0,
+ "255": 1131332608.0,
+ "260": 1131332608.0,
+ "265": 1131332608.0,
+ "270": 1131332608.0,
+ "275": 1131332608.0,
+ "280": 1131332608.0,
+ "285": 1131332608.0,
+ "290": 1131332608.0,
+ "295": 1131332608.0,
+ "300": 1131332608.0,
+ "305": 1131332608.0,
+ "310": 1131332608.0,
+ "315": 1131332608.0,
+ "320": 1131332608.0,
+ "325": 1131332608.0,
+ "330": 1131332608.0,
+ "335": 1131332608.0,
+ "340": 1131332608.0,
+ "345": 1131332608.0,
+ "350": 1131332608.0,
+ "355": 1131332608.0,
+ "360": 1131332608.0,
+ "365": 1131332608.0,
+ "370": 1131332608.0,
+ "375": 1131332608.0,
+ "380": 1131332608.0,
+ "385": 1131332608.0,
+ "390": 1131332608.0,
+ "395": 1131332608.0,
+ "400": 1131332608.0,
+ "405": 1131332608.0,
+ "410": 1131332608.0,
+ "415": 1131332608.0,
+ "420": 1131332608.0,
+ "425": 1131332608.0,
+ "430": 1131332608.0,
+ "435": 1131332608.0,
+ "440": 1131332608.0,
+ "445": 1131332608.0,
+ "450": 1131332608.0,
+ "455": 1131332608.0,
+ "460": 1131332608.0,
+ "465": 1131332608.0,
+ "470": 1131332608.0,
+ "475": 1131332608.0,
+ "480": 1131332608.0,
+ "485": 1131332608.0,
+ "490": 1131332608.0,
+ "495": 1131332608.0,
+ "500": 1131332608.0,
+ "505": 1131332608.0,
+ "510": 1131332608.0,
+ "515": 1131332608.0,
+ "520": 1131332608.0,
+ "525": 1131332608.0,
+ "530": 1131332608.0,
+ "535": 1131332608.0,
+ "540": 1131332608.0,
+ "545": 1131332608.0,
+ "550": 1131332608.0,
+ "555": 1131332608.0,
+ "560": 1131332608.0,
+ "565": 1131332608.0,
+ "570": 1131332608.0,
+ "575": 1131332608.0,
+ "580": 1131332608.0,
+ "585": 1131332608.0,
+ "590": 1131332608.0,
+ "595": 1131332608.0,
+ "600": 1131332608.0,
+ "605": 1131332608.0,
+ "610": 1131332608.0,
+ "615": 1131332608.0,
+ "620": 1131332608.0,
+ "625": 1131332608.0,
+ "630": 1131332608.0,
+ "635": 1131332608.0,
+ "640": 1131332608.0,
+ "645": 1131332608.0,
+ "650": 1131332608.0,
+ "655": 1131332608.0,
+ "660": 1131332608.0,
+ "665": 1131332608.0,
+ "670": 1131332608.0,
+ "675": 1131332608.0,
+ "680": 1131332608.0,
+ "685": 1131332608.0,
+ "690": 1131332608.0,
+ "695": 1131332608.0,
+ "700": 1131332608.0,
+ "705": 1131332608.0,
+ "710": 1131332608.0,
+ "715": 1131332608.0,
+ "720": 1131332608.0,
+ "725": 1131332608.0,
+ "730": 1131332608.0,
+ "735": 1131332608.0,
+ "740": 1131332608.0,
+ "745": 1131332608.0,
+ "750": 1131332608.0,
+ "755": 1131332608.0,
+ "760": 1131332608.0,
+ "765": 1131332608.0,
+ "770": 1131332608.0,
+ "775": 1131332608.0,
+ "780": 1131332608.0,
+ "785": 1131332608.0,
+ "790": 1131332608.0,
+ "795": 1131332608.0,
+ "800": 1131332608.0,
+ "805": 1131332608.0,
+ "810": 1131332608.0,
+ "815": 1131332608.0,
+ "820": 1131332608.0,
+ "825": 1131332608.0,
+ "830": 1131332608.0,
+ "835": 1131332608.0,
+ "840": 1131332608.0,
+ "845": 1131332608.0,
+ "850": 1131332608.0,
+ "855": 1131332608.0,
+ "860": 1131332608.0,
+ "865": 1131332608.0,
+ "870": 1131332608.0,
+ "875": 1131332608.0,
+ "880": 1131332608.0,
+ "885": 1131332608.0,
+ "890": 1131332608.0,
+ "895": 1131332608.0,
+ "900": 1131332608.0,
+ "905": 1131332608.0,
+ "910": 1131332608.0,
+ "915": 1131332608.0,
+ "920": 1131332608.0,
+ "925": 1131332608.0,
+ "930": 1131332608.0,
+ "935": 1131332608.0,
+ "940": 1131332608.0,
+ "945": 1131332608.0,
+ "950": 1131332608.0,
+ "955": 1131332608.0,
+ "960": 1131332608.0,
+ "965": 1131332608.0,
+ "970": 1131332608.0,
+ "975": 1131332608.0,
+ "980": 1131332608.0,
+ "985": 1131332608.0,
+ "990": 1131332608.0,
+ "995": 1131332608.0,
+ "1000": 1131332608.0,
+ "1005": 1131332608.0,
+ "1010": 1131332608.0,
+ "1015": 1131332608.0,
+ "1020": 1131332608.0,
+ "1025": 1131332608.0,
+ "1030": 1131332608.0,
+ "1035": 1131332608.0,
+ "1040": 1131332608.0,
+ "1045": 1131332608.0,
+ "1050": 1131332608.0,
+ "1055": 1131332608.0,
+ "1060": 1131332608.0,
+ "1065": 1131332608.0,
+ "1070": 1131332608.0,
+ "1075": 1131332608.0,
+ "1080": 1131332608.0,
+ "1085": 1131332608.0,
+ "1090": 1131332608.0,
+ "1095": 1131332608.0,
+ "1100": 1131332608.0,
+ "1105": 1131332608.0,
+ "1110": 1131332608.0,
+ "1115": 1131332608.0,
+ "1120": 1131332608.0,
+ "1125": 1131332608.0,
+ "1130": 1131332608.0,
+ "1135": 1131332608.0,
+ "1140": 1131332608.0,
+ "1145": 1131332608.0,
+ "1150": 1131332608.0,
+ "1155": 1131332608.0,
+ "1160": 1131332608.0,
+ "1165": 1131332608.0,
+ "1170": 1131332608.0,
+ "1175": 1131332608.0,
+ "1180": 1131332608.0,
+ "1185": 1131332608.0,
+ "1190": 1131332608.0,
+ "1195": 1131332608.0,
+ "1200": 1131332608.0,
+ "1205": 1131332608.0,
+ "1210": 1131332608.0,
+ "1215": 1131332608.0,
+ "1220": 1131332608.0,
+ "1225": 1131332608.0,
+ "1230": 1131332608.0,
+ "1235": 1131332608.0,
+ "1240": 1131332608.0,
+ "1245": 1131332608.0,
+ "1250": 1131332608.0,
+ "1255": 1131332608.0,
+ "1260": 1131332608.0,
+ "1265": 1131332608.0,
+ "1270": 1131332608.0,
+ "1275": 1131332608.0,
+ "1280": 1131332608.0,
+ "1285": 1131332608.0,
+ "1290": 1131332608.0,
+ "1295": 1131332608.0,
+ "1300": 1131332608.0,
+ "1305": 1131332608.0,
+ "1310": 1131332608.0,
+ "1315": 1131332608.0,
+ "1320": 1131332608.0,
+ "1325": 1131332608.0,
+ "1330": 1131332608.0,
+ "1335": 1131332608.0,
+ "1340": 1131332608.0,
+ "1345": 1131332608.0,
+ "1350": 1131332608.0,
+ "1355": 1131332608.0,
+ "1360": 1131332608.0,
+ "1365": 1131332608.0,
+ "1370": 1131332608.0,
+ "1375": 1131332608.0,
+ "1380": 1131332608.0,
+ "1385": 1131332608.0,
+ "1390": 1131332608.0,
+ "1395": 1131332608.0,
+ "1400": 1131332608.0,
+ "1405": 1131332608.0,
+ "1410": 1131332608.0,
+ "1415": 1131332608.0,
+ "1420": 1131332608.0,
+ "1425": 1131332608.0,
+ "1430": 1131332608.0,
+ "1435": 1131332608.0,
+ "1440": 1131332608.0,
+ "1445": 1131332608.0,
+ "1450": 1131332608.0,
+ "1455": 1131332608.0,
+ "1460": 1131332608.0,
+ "1465": 1131332608.0,
+ "1470": 1131332608.0,
+ "1475": 1131332608.0,
+ "1480": 1131332608.0,
+ "1485": 1131332608.0,
+ "1490": 1131332608.0,
+ "1495": 1131332608.0,
+ "1500": 1131332608.0,
+ "1505": 1131332608.0,
+ "1510": 1131332608.0,
+ "1515": 1131332608.0,
+ "1520": 1131332608.0,
+ "1525": 1131332608.0,
+ "1530": 1131332608.0,
+ "1535": 1131332608.0,
+ "1540": 1131332608.0,
+ "1545": 1131332608.0,
+ "1550": 1131332608.0,
+ "1555": 1131332608.0,
+ "1560": 1131332608.0,
+ "1565": 1131332608.0,
+ "1570": 1131332608.0,
+ "1575": 1131332608.0,
+ "1580": 1131332608.0,
+ "1585": 1131332608.0,
+ "1590": 1131332608.0,
+ "1595": 1131332608.0,
+ "1600": 1131332608.0,
+ "1605": 1131332608.0,
+ "1610": 1131332608.0,
+ "1615": 1131332608.0,
+ "1620": 1131332608.0,
+ "1625": 1131332608.0,
+ "1630": 1131332608.0,
+ "1635": 1131332608.0,
+ "1640": 1131332608.0,
+ "1645": 1131332608.0,
+ "1650": 1131332608.0,
+ "1655": 1131332608.0,
+ "1660": 1131332608.0,
+ "1665": 1131332608.0,
+ "1670": 1131332608.0,
+ "1675": 1131332608.0,
+ "1680": 1131332608.0,
+ "1685": 1131332608.0,
+ "1690": 1131332608.0,
+ "1695": 1131332608.0,
+ "1700": 1131332608.0,
+ "1705": 1131332608.0,
+ "1710": 1131332608.0,
+ "1715": 1131332608.0,
+ "1720": 1131332608.0,
+ "1725": 1131332608.0,
+ "1730": 1131332608.0,
+ "1735": 1131332608.0,
+ "1740": 1131332608.0,
+ "1745": 1131332608.0,
+ "1750": 1131332608.0,
+ "1755": 1131332608.0,
+ "1760": 1131332608.0,
+ "1765": 1131332608.0,
+ "1770": 1131332608.0,
+ "1775": 1131332608.0,
+ "1780": 1131332608.0,
+ "1785": 1131332608.0,
+ "1790": 1131332608.0,
+ "1795": 1131332608.0,
+ "1800": 1131332608.0,
+ "1805": 1131332608.0,
+ "1810": 1131332608.0,
+ "1815": 1131332608.0,
+ "1820": 1131332608.0,
+ "1825": 1131332608.0,
+ "1830": 1131332608.0,
+ "1835": 1131332608.0,
+ "1840": 1131332608.0,
+ "1845": 1131332608.0,
+ "1850": 1131332608.0,
+ "1855": 1131332608.0,
+ "1860": 1131332608.0,
+ "1865": 1131332608.0,
+ "1870": 1131332608.0,
+ "1875": 1131332608.0,
+ "1880": 1131332608.0,
+ "1885": 1131332608.0,
+ "1890": 1131332608.0,
+ "1895": 1131332608.0,
+ "1900": 1131332608.0,
+ "1905": 1131332608.0,
+ "1910": 1131332608.0,
+ "1915": 1131332608.0,
+ "1920": 1131332608.0,
+ "1925": 1131332608.0,
+ "1930": 1131332608.0,
+ "1935": 1131332608.0,
+ "1940": 1131332608.0,
+ "1945": 1131332608.0,
+ "1950": 1131332608.0,
+ "1955": 1131332608.0,
+ "1960": 1131332608.0,
+ "1965": 1131332608.0,
+ "1970": 1131332608.0,
+ "1975": 1131332608.0,
+ "1980": 1131332608.0,
+ "1985": 1131332608.0,
+ "1990": 1131332608.0,
+ "1995": 1131332608.0,
+ "2000": 1131332608.0,
+ "2005": 1131332608.0,
+ "2010": 1131332608.0,
+ "2015": 1131332608.0,
+ "2020": 1131332608.0,
+ "2025": 1131332608.0,
+ "2030": 1131332608.0,
+ "2035": 1131332608.0,
+ "2040": 1131332608.0,
+ "2045": 1131332608.0,
+ "2050": 1131332608.0,
+ "2055": 1131332608.0,
+ "2060": 1131332608.0,
+ "2065": 1131332608.0,
+ "2070": 1131332608.0,
+ "2075": 1131332608.0,
+ "2080": 1131332608.0,
+ "2085": 1131332608.0,
+ "2090": 1131332608.0,
+ "2095": 1131332608.0,
+ "2100": 1131332608.0,
+ "2105": 1131332608.0,
+ "2110": 1131332608.0,
+ "2115": 1131332608.0,
+ "2120": 1131332608.0,
+ "2125": 1131332608.0,
+ "2130": 1131332608.0,
+ "2135": 1131332608.0,
+ "2140": 1131332608.0,
+ "2145": 1131332608.0,
+ "2150": 1131332608.0,
+ "2155": 1131332608.0,
+ "2160": 1131332608.0,
+ "2165": 1131332608.0,
+ "2170": 1131332608.0,
+ "2175": 1131332608.0,
+ "2180": 1131332608.0,
+ "2185": 1131332608.0,
+ "2190": 1131332608.0,
+ "2195": 1131332608.0,
+ "2200": 1131332608.0,
+ "2205": 1131332608.0,
+ "2210": 1131332608.0,
+ "2215": 1131332608.0,
+ "2220": 1131332608.0,
+ "2225": 1131332608.0,
+ "2230": 1131332608.0,
+ "2235": 1131332608.0,
+ "2240": 1131332608.0,
+ "2245": 1131332608.0,
+ "2250": 1131332608.0,
+ "2255": 1131332608.0,
+ "2260": 1131332608.0,
+ "2265": 1131332608.0,
+ "2270": 1131332608.0,
+ "2275": 1131332608.0,
+ "2280": 1131332608.0,
+ "2285": 1131332608.0,
+ "2290": 1131332608.0,
+ "2295": 1131332608.0,
+ "2300": 1131332608.0,
+ "2305": 1131332608.0,
+ "2310": 1131332608.0,
+ "2315": 1131332608.0,
+ "2320": 1131332608.0,
+ "2325": 1131332608.0,
+ "2330": 1131332608.0,
+ "2335": 1131332608.0,
+ "2340": 1131332608.0,
+ "2345": 1131332608.0,
+ "2350": 1131332608.0,
+ "2355": 1131332608.0,
+ "2360": 1131332608.0,
+ "2365": 1131332608.0,
+ "2370": 1131332608.0,
+ "2375": 1131332608.0,
+ "2380": 1131332608.0,
+ "2385": 1131332608.0,
+ "2390": 1131332608.0,
+ "2395": 1131332608.0,
+ "2400": 1131332608.0,
+ "2405": 1131332608.0,
+ "2410": 1131332608.0,
+ "2415": 1131332608.0,
+ "2420": 1131332608.0,
+ "2425": 1131332608.0,
+ "2430": 1131332608.0,
+ "2435": 1131332608.0,
+ "2440": 1131332608.0,
+ "2445": 1131332608.0,
+ "2450": 1131332608.0,
+ "2455": 1131332608.0,
+ "2460": 1131332608.0,
+ "2465": 1131332608.0,
+ "2470": 1131332608.0,
+ "2475": 1131332608.0,
+ "2480": 1131332608.0,
+ "2485": 1131332608.0,
+ "2490": 1131332608.0,
+ "2495": 1131332608.0,
+ "2500": 1131332608.0,
+ "2505": 1131332608.0,
+ "2510": 1131332608.0,
+ "2515": 1131332608.0,
+ "2520": 1131332608.0,
+ "2525": 1131332608.0,
+ "2530": 1131332608.0,
+ "2535": 1131332608.0,
+ "2540": 1131332608.0,
+ "2545": 1131332608.0,
+ "2550": 1131332608.0,
+ "2555": 1131332608.0,
+ "2560": 1131332608.0,
+ "2565": 1131332608.0,
+ "2570": 1131332608.0,
+ "2575": 1131332608.0,
+ "2580": 1131332608.0,
+ "2585": 1131332608.0,
+ "2590": 1131332608.0,
+ "2595": 1131332608.0,
+ "2600": 1131332608.0,
+ "2605": 1131332608.0,
+ "2610": 1131332608.0,
+ "2615": 1131332608.0,
+ "2620": 1131332608.0,
+ "2625": 1131332608.0,
+ "2630": 1131332608.0,
+ "2635": 1131332608.0,
+ "2640": 1131332608.0,
+ "2645": 1131332608.0,
+ "2650": 1131332608.0,
+ "2655": 1131332608.0,
+ "2660": 1131332608.0,
+ "2665": 1131332608.0,
+ "2670": 1131332608.0,
+ "2675": 1131332608.0,
+ "2680": 1131332608.0,
+ "2685": 1131332608.0,
+ "2690": 1131332608.0,
+ "2695": 1131332608.0,
+ "2700": 1131332608.0,
+ "2705": 1131332608.0,
+ "2710": 1131332608.0,
+ "2715": 1131332608.0,
+ "2720": 1131332608.0,
+ "2725": 1131332608.0,
+ "2730": 1131332608.0,
+ "2735": 1131332608.0,
+ "2740": 1131332608.0,
+ "2745": 1131332608.0,
+ "2750": 1131332608.0,
+ "2755": 1131332608.0,
+ "2760": 1131332608.0,
+ "2765": 1131332608.0,
+ "2770": 1131332608.0,
+ "2775": 1131332608.0,
+ "2780": 1131332608.0,
+ "2785": 1131332608.0,
+ "2790": 1131332608.0,
+ "2795": 1131332608.0,
+ "2800": 1131332608.0,
+ "2805": 1131332608.0,
+ "2810": 1131332608.0,
+ "2815": 1131332608.0,
+ "2820": 1131332608.0,
+ "2825": 1131332608.0,
+ "2830": 1131332608.0,
+ "2835": 1131332608.0,
+ "2840": 1131332608.0,
+ "2845": 1131332608.0,
+ "2850": 1131332608.0,
+ "2855": 1131332608.0,
+ "2860": 1131332608.0,
+ "2865": 1131332608.0,
+ "2870": 1131332608.0,
+ "2875": 1131332608.0,
+ "2880": 1131332608.0,
+ "2885": 1131332608.0,
+ "2890": 1131332608.0,
+ "2895": 1131332608.0,
+ "2900": 1131332608.0,
+ "2905": 1131332608.0,
+ "2910": 1131332608.0,
+ "2915": 1131332608.0,
+ "2920": 1131332608.0,
+ "2925": 1131332608.0,
+ "2930": 1131332608.0,
+ "2935": 1131332608.0,
+ "2940": 1131332608.0,
+ "2945": 1131332608.0,
+ "2950": 1131332608.0,
+ "2955": 1131332608.0,
+ "2960": 1131332608.0,
+ "2965": 1131332608.0,
+ "2970": 1131332608.0,
+ "2975": 1131332608.0,
+ "2980": 1131332608.0,
+ "2985": 1131332608.0,
+ "2990": 1131332608.0,
+ "2995": 1131332608.0,
+ "3000": 1131332608.0,
+ "3005": 1131332608.0,
+ "3010": 1131332608.0,
+ "3015": 1131332608.0,
+ "3020": 1131332608.0,
+ "3025": 1131332608.0,
+ "3030": 1131332608.0,
+ "3035": 1131332608.0,
+ "3040": 1131332608.0,
+ "3045": 1131332608.0,
+ "3050": 1131332608.0,
+ "3055": 1131332608.0,
+ "3060": 1131332608.0,
+ "3065": 1131332608.0,
+ "3070": 1131332608.0,
+ "3075": 1131332608.0,
+ "3080": 1131332608.0,
+ "3085": 1131332608.0,
+ "3090": 1131332608.0,
+ "3095": 1131332608.0,
+ "3100": 1131332608.0,
+ "3105": 1131332608.0,
+ "3110": 1131332608.0,
+ "3115": 1131332608.0,
+ "3120": 1131332608.0,
+ "3125": 1131332608.0,
+ "3130": 1131332608.0,
+ "3135": 1131332608.0,
+ "3140": 1131332608.0,
+ "3145": 1131332608.0,
+ "3150": 1131332608.0,
+ "3155": 1131332608.0,
+ "3160": 1131332608.0,
+ "3165": 1131332608.0,
+ "3170": 1131332608.0,
+ "3175": 1131332608.0,
+ "3180": 1131332608.0,
+ "3185": 1131332608.0,
+ "3190": 1131332608.0,
+ "3195": 1131332608.0,
+ "3200": 1131332608.0,
+ "3205": 1131332608.0,
+ "3210": 1131332608.0,
+ "3215": 1131332608.0,
+ "3220": 1131332608.0,
+ "3225": 1131332608.0,
+ "3230": 1131332608.0,
+ "3235": 1131332608.0,
+ "3240": 1131332608.0,
+ "3245": 1131332608.0,
+ "3250": 1131332608.0,
+ "3255": 1131332608.0,
+ "3260": 1131332608.0,
+ "3265": 1131332608.0,
+ "3270": 1131332608.0,
+ "3275": 1131332608.0,
+ "3280": 1131332608.0,
+ "3285": 1131332608.0,
+ "3290": 1131332608.0,
+ "3295": 1131332608.0,
+ "3300": 1131332608.0,
+ "3305": 1131332608.0,
+ "3310": 1131332608.0,
+ "3315": 1131332608.0,
+ "3320": 1131332608.0,
+ "3325": 1131332608.0,
+ "3330": 1131332608.0,
+ "3335": 1131332608.0,
+ "3340": 1131332608.0,
+ "3345": 1131332608.0,
+ "3350": 1131332608.0,
+ "3355": 1131332608.0,
+ "3360": 1131332608.0,
+ "3365": 1131332608.0,
+ "3370": 1131332608.0,
+ "3375": 1131332608.0,
+ "3380": 1131332608.0,
+ "3385": 1131332608.0,
+ "3390": 1131332608.0,
+ "3395": 1131332608.0,
+ "3400": 1131332608.0,
+ "3405": 1131332608.0,
+ "3410": 1131332608.0,
+ "3415": 1131332608.0,
+ "3420": 1131332608.0,
+ "3425": 1131332608.0,
+ "3430": 1131332608.0,
+ "3435": 1131332608.0,
+ "3440": 1131332608.0,
+ "3445": 1131332608.0,
+ "3450": 1131332608.0,
+ "3455": 1131332608.0,
+ "3460": 1131332608.0,
+ "3465": 1131332608.0,
+ "3470": 1131332608.0,
+ "3475": 1131332608.0,
+ "3480": 1131332608.0,
+ "3485": 1131332608.0,
+ "3490": 1131332608.0,
+ "3495": 1131332608.0,
+ "3500": 1131332608.0,
+ "3505": 1131332608.0,
+ "3510": 1131332608.0,
+ "3515": 1131332608.0,
+ "3520": 1131332608.0,
+ "3525": 1131332608.0,
+ "3530": 1131332608.0,
+ "3535": 1131332608.0,
+ "3540": 1131332608.0,
+ "3545": 1131332608.0,
+ "3550": 1131332608.0,
+ "3555": 1131332608.0,
+ "3560": 1131332608.0,
+ "3565": 1131332608.0,
+ "3570": 1131332608.0,
+ "3575": 1131332608.0,
+ "3580": 1131332608.0,
+ "3585": 1131332608.0,
+ "3590": 1131332608.0,
+ "3595": 1131332608.0,
+ "3600": 1131332608.0,
+ "3605": 1131332608.0,
+ "3610": 1131332608.0,
+ "3615": 1131332608.0,
+ "3620": 1131332608.0,
+ "3625": 1131332608.0,
+ "3630": 1131332608.0,
+ "3635": 1131332608.0,
+ "3640": 1131332608.0,
+ "3645": 1131332608.0,
+ "3650": 1131332608.0,
+ "3655": 1131332608.0,
+ "3660": 1131332608.0,
+ "3665": 1131332608.0,
+ "3670": 1131332608.0,
+ "3675": 1131332608.0,
+ "3680": 1131332608.0,
+ "3685": 1131332608.0,
+ "3690": 1131332608.0,
+ "3695": 1131332608.0,
+ "3700": 1131332608.0,
+ "3705": 1131332608.0,
+ "3710": 1131332608.0,
+ "3715": 1131332608.0,
+ "3720": 1131332608.0,
+ "3725": 1131332608.0,
+ "3730": 1131332608.0,
+ "3735": 1131332608.0,
+ "3740": 1131332608.0,
+ "3745": 1131332608.0,
+ "3750": 1131332608.0,
+ "3755": 1131332608.0,
+ "3760": 1131332608.0,
+ "3765": 1131332608.0,
+ "3770": 1131332608.0,
+ "3775": 1131332608.0,
+ "3780": 1131332608.0,
+ "3785": 1131332608.0,
+ "3790": 1131332608.0,
+ "3795": 1131332608.0,
+ "3800": 1131332608.0,
+ "3805": 1131332608.0,
+ "3810": 1131332608.0,
+ "3815": 1131332608.0,
+ "3820": 1131332608.0,
+ "3825": 1131332608.0,
+ "3830": 1131332608.0,
+ "3835": 1131332608.0,
+ "3840": 1131332608.0,
+ "3845": 1131332608.0,
+ "3850": 1131332608.0,
+ "3855": 1131332608.0,
+ "3860": 1131332608.0,
+ "3865": 1131332608.0,
+ "3870": 1131332608.0,
+ "3875": 1131332608.0,
+ "3880": 1131332608.0,
+ "3885": 1131332608.0,
+ "3890": 1131332608.0,
+ "3895": 1131332608.0,
+ "3900": 1131332608.0,
+ "3905": 1131332608.0,
+ "3910": 1131332608.0,
+ "3915": 1131332608.0,
+ "3920": 1131332608.0,
+ "3925": 1131332608.0,
+ "3930": 1131332608.0,
+ "3935": 1131332608.0,
+ "3940": 1131332608.0,
+ "3945": 1131332608.0,
+ "3950": 1131332608.0,
+ "3955": 1131332608.0,
+ "3960": 1131332608.0,
+ "3965": 1131332608.0,
+ "3970": 1131332608.0,
+ "3975": 1131332608.0,
+ "3980": 1131332608.0,
+ "3985": 1131332608.0,
+ "3990": 1131332608.0,
+ "3995": 1131332608.0,
+ "4000": 1131332608.0,
+ "4005": 1131332608.0,
+ "4010": 1131332608.0,
+ "4015": 1131332608.0,
+ "4020": 1131332608.0,
+ "4025": 1131332608.0,
+ "4030": 1131332608.0,
+ "4035": 1131332608.0,
+ "4040": 1131332608.0,
+ "4045": 1131332608.0,
+ "4050": 1131332608.0,
+ "4055": 1131332608.0,
+ "4060": 1131332608.0,
+ "4065": 1131332608.0,
+ "4070": 1131332608.0,
+ "4075": 1131332608.0,
+ "4080": 1131332608.0,
+ "4085": 1131332608.0,
+ "4090": 1131332608.0,
+ "4095": 1131332608.0,
+ "4100": 1131332608.0,
+ "4105": 1131332608.0,
+ "4110": 1131332608.0,
+ "4115": 1131332608.0,
+ "4120": 1131332608.0,
+ "4125": 1131332608.0,
+ "4130": 1131332608.0,
+ "4135": 1131332608.0,
+ "4140": 1131332608.0,
+ "4145": 1131332608.0,
+ "4150": 1131332608.0,
+ "4155": 1131332608.0,
+ "4160": 1131332608.0,
+ "4165": 1131332608.0,
+ "4170": 1131332608.0,
+ "4175": 1131332608.0,
+ "4180": 1131332608.0,
+ "4185": 1131332608.0,
+ "4190": 1131332608.0,
+ "4195": 1131332608.0,
+ "4200": 1131332608.0,
+ "4205": 1131332608.0,
+ "4210": 1131332608.0,
+ "4215": 1131332608.0,
+ "4220": 1131332608.0,
+ "4225": 1131332608.0,
+ "4230": 1131332608.0,
+ "4235": 1131332608.0,
+ "4240": 1131332608.0,
+ "4245": 1131332608.0,
+ "4250": 1131332608.0,
+ "4255": 1131332608.0,
+ "4260": 1131332608.0,
+ "4265": 1131332608.0,
+ "4270": 1131332608.0,
+ "4275": 1131332608.0,
+ "4280": 1131332608.0,
+ "4285": 1131332608.0,
+ "4290": 1131332608.0,
+ "4295": 1131332608.0,
+ "4300": 1131332608.0,
+ "4305": 1131332608.0,
+ "4310": 1131332608.0,
+ "4315": 1131332608.0,
+ "4320": 1131332608.0,
+ "4325": 1131332608.0,
+ "4330": 1131332608.0,
+ "4335": 1131332608.0,
+ "4340": 1131332608.0,
+ "4345": 1131332608.0,
+ "4350": 1131332608.0,
+ "4355": 1131332608.0,
+ "4360": 1131332608.0,
+ "4365": 1131332608.0,
+ "4370": 1131332608.0,
+ "4375": 1131332608.0,
+ "4380": 1131332608.0,
+ "4385": 1131332608.0,
+ "4390": 1131332608.0,
+ "4395": 1131332608.0,
+ "4400": 1131332608.0,
+ "4405": 1131332608.0,
+ "4410": 1131332608.0,
+ "4415": 1131332608.0,
+ "4420": 1131332608.0,
+ "4425": 1131332608.0,
+ "4430": 1131332608.0,
+ "4435": 1131332608.0,
+ "4440": 1131332608.0,
+ "4445": 1131332608.0,
+ "4450": 1131332608.0,
+ "4455": 1131332608.0,
+ "4460": 1131332608.0,
+ "4465": 1131332608.0,
+ "4470": 1131332608.0,
+ "4475": 1131332608.0,
+ "4480": 1131332608.0,
+ "4485": 1131332608.0,
+ "4490": 1131332608.0,
+ "4495": 1131332608.0,
+ "4500": 1131332608.0,
+ "4505": 1131332608.0,
+ "4510": 1131332608.0,
+ "4515": 1131332608.0,
+ "4520": 1131332608.0,
+ "4525": 1131332608.0,
+ "4530": 1131332608.0,
+ "4535": 1131332608.0,
+ "4540": 1131332608.0,
+ "4545": 1131332608.0,
+ "4550": 1131332608.0,
+ "4555": 1131332608.0,
+ "4560": 1131332608.0,
+ "4565": 1131332608.0,
+ "4570": 1131332608.0,
+ "4575": 1131332608.0,
+ "4580": 1131332608.0,
+ "4585": 1131332608.0,
+ "4590": 1131332608.0,
+ "4595": 1131332608.0,
+ "4600": 1131332608.0,
+ "4605": 1131332608.0,
+ "4610": 1131332608.0,
+ "4615": 1131332608.0,
+ "4620": 1131332608.0,
+ "4625": 1131332608.0,
+ "4630": 1131332608.0,
+ "4635": 1131332608.0,
+ "4640": 1131332608.0,
+ "4645": 1131332608.0,
+ "4650": 1131332608.0,
+ "4655": 1131332608.0,
+ "4660": 1131332608.0,
+ "4665": 1131332608.0,
+ "4670": 1131332608.0,
+ "4675": 1131332608.0,
+ "4680": 1131332608.0,
+ "4685": 1131332608.0,
+ "4690": 1131332608.0,
+ "4695": 1131332608.0,
+ "4700": 1131332608.0,
+ "4705": 1131332608.0,
+ "4710": 1131332608.0,
+ "4715": 1131332608.0,
+ "4720": 1131332608.0,
+ "4725": 1131332608.0,
+ "4730": 1131332608.0,
+ "4735": 1131332608.0,
+ "4740": 1131332608.0,
+ "4745": 1131332608.0,
+ "4750": 1131332608.0,
+ "4755": 1131332608.0,
+ "4760": 1131332608.0,
+ "4765": 1131332608.0,
+ "4770": 1131332608.0,
+ "4775": 1131332608.0,
+ "4780": 1131332608.0,
+ "4785": 1131332608.0,
+ "4790": 1131332608.0,
+ "4795": 1131332608.0,
+ "4800": 1131332608.0,
+ "4805": 1131332608.0,
+ "4810": 1131332608.0,
+ "4815": 1131332608.0,
+ "4820": 1131332608.0,
+ "4825": 1131332608.0,
+ "4830": 1131332608.0,
+ "4835": 1131332608.0,
+ "4840": 1131332608.0,
+ "4845": 1131332608.0,
+ "4850": 1131332608.0,
+ "4855": 1131332608.0,
+ "4860": 1131332608.0,
+ "4865": 1131332608.0,
+ "4870": 1131332608.0,
+ "4875": 1131332608.0,
+ "4880": 1131332608.0,
+ "4885": 1131332608.0,
+ "4890": 1131332608.0,
+ "4895": 1131332608.0,
+ "4900": 1131332608.0,
+ "4905": 1131332608.0,
+ "4910": 1131332608.0,
+ "4915": 1131332608.0,
+ "4920": 1131332608.0,
+ "4925": 1131332608.0,
+ "4930": 1131332608.0,
+ "4935": 1131332608.0,
+ "4940": 1131332608.0,
+ "4945": 1131332608.0,
+ "4950": 1131332608.0,
+ "4955": 1131332608.0,
+ "4960": 1131332608.0,
+ "4965": 1131332608.0,
+ "4970": 1131332608.0,
+ "4975": 1131332608.0,
+ "4980": 1131332608.0,
+ "4985": 1131332608.0,
+ "4990": 1131332608.0,
+ "4995": 1131332608.0,
+ "5000": 1131332608.0,
+ "5005": 1131332608.0,
+ "5010": 1131332608.0,
+ "5015": 1131332608.0,
+ "5020": 1131332608.0,
+ "5025": 1131332608.0,
+ "5030": 1131332608.0,
+ "5035": 1131332608.0,
+ "5040": 1131332608.0,
+ "5045": 1131332608.0,
+ "5050": 1131332608.0,
+ "5055": 1131332608.0,
+ "5060": 1131332608.0,
+ "5065": 1131332608.0,
+ "5070": 1131332608.0,
+ "5075": 1131332608.0,
+ "5080": 1131332608.0,
+ "5085": 1131332608.0,
+ "5090": 1131332608.0,
+ "5095": 1131332608.0,
+ "5100": 1131332608.0,
+ "5105": 1131332608.0,
+ "5110": 1131332608.0,
+ "5115": 1131332608.0,
+ "5120": 1131332608.0,
+ "5125": 1131332608.0,
+ "5130": 1131332608.0,
+ "5135": 1131332608.0,
+ "5140": 1131332608.0,
+ "5145": 1131332608.0,
+ "5150": 1131332608.0,
+ "5155": 1131332608.0,
+ "5160": 1131332608.0,
+ "5165": 1131332608.0,
+ "5170": 1131332608.0,
+ "5175": 1131332608.0,
+ "5180": 1131332608.0,
+ "5185": 1131332608.0,
+ "5190": 1131332608.0,
+ "5195": 1131332608.0,
+ "5200": 1131332608.0,
+ "5205": 1131332608.0,
+ "5210": 1131332608.0,
+ "5215": 1131332608.0,
+ "5220": 1131332608.0,
+ "5225": 1131332608.0,
+ "5230": 1131332608.0,
+ "5235": 1131332608.0,
+ "5240": 1131332608.0,
+ "5245": 1131332608.0,
+ "5250": 1131332608.0,
+ "5255": 1131332608.0,
+ "5260": 1131332608.0,
+ "5265": 1131332608.0,
+ "5270": 1131332608.0,
+ "5275": 1131332608.0,
+ "5280": 1131332608.0,
+ "5285": 1131332608.0,
+ "5290": 1131332608.0,
+ "5295": 1131332608.0,
+ "5300": 1131332608.0,
+ "5305": 1131332608.0,
+ "5310": 1131332608.0,
+ "5315": 1131332608.0,
+ "5320": 1131332608.0,
+ "5325": 1131332608.0,
+ "5330": 1131332608.0,
+ "5335": 1131332608.0,
+ "5340": 1131332608.0,
+ "5345": 1131332608.0,
+ "5350": 1131332608.0,
+ "5355": 1131332608.0,
+ "5360": 1131332608.0,
+ "5365": 1131332608.0,
+ "5370": 1131332608.0,
+ "5375": 1131332608.0,
+ "5380": 1131332608.0,
+ "5385": 1131332608.0,
+ "5390": 1131332608.0,
+ "5395": 1131332608.0,
+ "5400": 1131332608.0,
+ "5405": 1131332608.0,
+ "5410": 1131332608.0,
+ "5415": 1131332608.0,
+ "5420": 1131332608.0,
+ "5425": 1131332608.0,
+ "5430": 1131332608.0,
+ "5435": 1131332608.0,
+ "5440": 1131332608.0,
+ "5445": 1131332608.0,
+ "5450": 1131332608.0,
+ "5455": 1131332608.0,
+ "5460": 1131332608.0,
+ "5465": 1131332608.0,
+ "5470": 1131332608.0,
+ "5475": 1131332608.0,
+ "5480": 1131332608.0,
+ "5485": 1131332608.0,
+ "5490": 1131332608.0,
+ "5495": 1131332608.0,
+ "5500": 1131332608.0,
+ "5505": 1131332608.0,
+ "5510": 1131332608.0,
+ "5515": 1131332608.0,
+ "5520": 1131332608.0,
+ "5525": 1131332608.0,
+ "5530": 1131332608.0,
+ "5535": 1131332608.0,
+ "5540": 1131332608.0,
+ "5545": 1131332608.0,
+ "5550": 1131332608.0,
+ "5555": 1131332608.0,
+ "5560": 1131332608.0,
+ "5565": 1131332608.0,
+ "5570": 1131332608.0,
+ "5575": 1131332608.0,
+ "5580": 1131332608.0,
+ "5585": 1131332608.0,
+ "5590": 1131332608.0,
+ "5595": 1131332608.0,
+ "5600": 1131332608.0,
+ "5605": 1131332608.0,
+ "5610": 1131332608.0,
+ "5615": 1131332608.0,
+ "5620": 1131332608.0,
+ "5625": 1131332608.0,
+ "5630": 1131332608.0,
+ "5635": 1131332608.0,
+ "5640": 1131332608.0,
+ "5645": 1131332608.0,
+ "5650": 1131332608.0,
+ "5655": 1131332608.0,
+ "5660": 1131332608.0,
+ "5665": 1131332608.0,
+ "5670": 1131332608.0,
+ "5675": 1131332608.0,
+ "5680": 1131332608.0,
+ "5685": 1131332608.0,
+ "5690": 1131332608.0,
+ "5695": 1131332608.0,
+ "5700": 1131332608.0,
+ "5705": 1131332608.0,
+ "5710": 1131332608.0,
+ "5715": 1131332608.0,
+ "5720": 1131332608.0,
+ "5725": 1131332608.0,
+ "5730": 1131332608.0,
+ "5735": 1131332608.0,
+ "5740": 1131332608.0,
+ "5745": 1131332608.0,
+ "5750": 1131332608.0,
+ "5755": 1131332608.0,
+ "5760": 1131332608.0,
+ "5765": 1131332608.0,
+ "5770": 1131332608.0,
+ "5775": 1131332608.0,
+ "5780": 1131332608.0,
+ "5785": 1131332608.0,
+ "5790": 1131332608.0,
+ "5795": 1131332608.0,
+ "5800": 1131332608.0,
+ "5805": 1131332608.0,
+ "5810": 1131332608.0,
+ "5815": 1131332608.0,
+ "5820": 1131332608.0,
+ "5825": 1131332608.0,
+ "5830": 1131332608.0,
+ "5835": 1131332608.0,
+ "5840": 1131332608.0,
+ "5845": 1131332608.0,
+ "5850": 1131332608.0,
+ "5855": 1131332608.0,
+ "5860": 1131332608.0,
+ "5865": 1131332608.0,
+ "5870": 1131332608.0,
+ "5875": 1131332608.0,
+ "5880": 1131332608.0,
+ "5885": 1131332608.0,
+ "5890": 1131332608.0,
+ "5895": 1131332608.0,
+ "5900": 1131332608.0,
+ "5905": 1131332608.0,
+ "5910": 1131332608.0,
+ "5915": 1131332608.0,
+ "5920": 1131332608.0,
+ "5925": 1131332608.0,
+ "5930": 1131332608.0,
+ "5935": 1131332608.0,
+ "5940": 1131332608.0,
+ "5945": 1131332608.0,
+ "5950": 1131332608.0,
+ "5955": 1131332608.0,
+ "5960": 1131332608.0,
+ "5965": 1131332608.0,
+ "5970": 1131332608.0,
+ "5975": 1131332608.0,
+ "5980": 1131332608.0,
+ "5985": 1131332608.0,
+ "5990": 1131332608.0,
+ "5995": 1131332608.0,
+ "6000": 1131332608.0,
+ "6005": 1131332608.0,
+ "6010": 1131332608.0,
+ "6015": 1131332608.0,
+ "6020": 1131332608.0,
+ "6025": 1131332608.0,
+ "6030": 1131332608.0,
+ "6035": 1131332608.0,
+ "6040": 1131332608.0,
+ "6045": 1131332608.0,
+ "6050": 1131332608.0,
+ "6055": 1131332608.0,
+ "6060": 1131332608.0,
+ "6065": 1131332608.0,
+ "6070": 1131332608.0,
+ "6075": 1131332608.0,
+ "6080": 1131332608.0,
+ "6085": 1131332608.0,
+ "6090": 1131332608.0,
+ "6095": 1131332608.0,
+ "6100": 1131332608.0,
+ "6105": 1131332608.0,
+ "6110": 1131332608.0,
+ "6115": 1131332608.0,
+ "6120": 1131332608.0,
+ "6125": 1131332608.0,
+ "6130": 1131332608.0,
+ "6135": 1131332608.0,
+ "6140": 1131332608.0,
+ "6145": 1131332608.0,
+ "6150": 1131332608.0,
+ "6155": 1131332608.0,
+ "6160": 1131332608.0,
+ "6165": 1131332608.0,
+ "6170": 1131332608.0,
+ "6175": 1131332608.0,
+ "6180": 1131332608.0,
+ "6185": 1131332608.0,
+ "6190": 1131332608.0,
+ "6195": 1131332608.0,
+ "6200": 1131332608.0,
+ "6205": 1131332608.0,
+ "6210": 1131332608.0,
+ "6215": 1131332608.0,
+ "6220": 1131332608.0,
+ "6225": 1131332608.0,
+ "6230": 1131332608.0,
+ "6235": 1131332608.0,
+ "6240": 1131332608.0,
+ "6245": 1131332608.0,
+ "6250": 1131332608.0,
+ "6255": 1131332608.0,
+ "6260": 1131332608.0,
+ "6265": 1131332608.0,
+ "6270": 1131332608.0,
+ "6275": 1131332608.0,
+ "6280": 1131332608.0,
+ "6285": 1131332608.0,
+ "6290": 1131332608.0,
+ "6295": 1131332608.0,
+ "6300": 1131332608.0,
+ "6305": 1131332608.0,
+ "6310": 1131332608.0,
+ "6315": 1131332608.0,
+ "6320": 1131332608.0,
+ "6325": 1131332608.0,
+ "6330": 1131332608.0,
+ "6335": 1131332608.0,
+ "6340": 1131332608.0,
+ "6345": 1131332608.0,
+ "6350": 1131332608.0,
+ "6355": 1131332608.0,
+ "6360": 1131332608.0,
+ "6365": 1131332608.0,
+ "6370": 1131332608.0,
+ "6375": 1131332608.0,
+ "6380": 1131332608.0,
+ "6385": 1131332608.0,
+ "6390": 1131332608.0,
+ "6395": 1131332608.0,
+ "6400": 1131332608.0,
+ "6405": 1131332608.0,
+ "6410": 1131332608.0,
+ "6415": 1131332608.0,
+ "6420": 1131332608.0,
+ "6425": 1131332608.0,
+ "6430": 1131332608.0,
+ "6435": 1131332608.0,
+ "6440": 1131332608.0,
+ "6445": 1131332608.0,
+ "6450": 1131332608.0,
+ "6455": 1131332608.0,
+ "6460": 1131332608.0,
+ "6465": 1131332608.0,
+ "6470": 1131332608.0,
+ "6475": 1131332608.0,
+ "6480": 1131332608.0,
+ "6485": 1131332608.0,
+ "6490": 1131332608.0,
+ "6495": 1131332608.0,
+ "6500": 1131332608.0,
+ "6505": 1131332608.0,
+ "6510": 1131332608.0,
+ "6515": 1131332608.0,
+ "6520": 1131332608.0,
+ "6525": 1131332608.0,
+ "6530": 1131332608.0,
+ "6535": 1131332608.0,
+ "6540": 1131332608.0,
+ "6545": 1131332608.0,
+ "6550": 1131332608.0,
+ "6555": 1131332608.0,
+ "6560": 1131332608.0,
+ "6565": 1131332608.0,
+ "6570": 1131332608.0,
+ "6575": 1131332608.0,
+ "6580": 1131332608.0,
+ "6585": 1131332608.0,
+ "6590": 1131332608.0,
+ "6595": 1131332608.0,
+ "6600": 1131332608.0,
+ "6605": 1131332608.0,
+ "6610": 1131332608.0,
+ "6615": 1131332608.0,
+ "6620": 1131332608.0,
+ "6625": 1131332608.0,
+ "6630": 1131332608.0,
+ "6635": 1131332608.0,
+ "6640": 1131332608.0,
+ "6645": 1131332608.0,
+ "6650": 1131332608.0,
+ "6655": 1131332608.0,
+ "6660": 1131332608.0,
+ "6665": 1131332608.0,
+ "6670": 1131332608.0,
+ "6675": 1131332608.0,
+ "6680": 1131332608.0,
+ "6685": 1131332608.0,
+ "6690": 1131332608.0,
+ "6695": 1131332608.0,
+ "6700": 1131332608.0,
+ "6705": 1131332608.0,
+ "6710": 1131332608.0,
+ "6715": 1131332608.0,
+ "6720": 1131332608.0,
+ "6725": 1131332608.0,
+ "6730": 1131332608.0,
+ "6735": 1131332608.0,
+ "6740": 1131332608.0,
+ "6745": 1131332608.0,
+ "6750": 1131332608.0,
+ "6755": 1131332608.0,
+ "6760": 1131332608.0,
+ "6765": 1131332608.0,
+ "6770": 1131332608.0,
+ "6775": 1131332608.0,
+ "6780": 1131332608.0,
+ "6785": 1131332608.0,
+ "6790": 1131332608.0,
+ "6795": 1131332608.0,
+ "6800": 1131332608.0,
+ "6805": 1131332608.0,
+ "6810": 1131332608.0,
+ "6815": 1131332608.0,
+ "6820": 1131332608.0,
+ "6825": 1131332608.0,
+ "6830": 1131332608.0,
+ "6835": 1131332608.0,
+ "6840": 1131332608.0,
+ "6845": 1131332608.0,
+ "6850": 1131332608.0,
+ "6855": 1131332608.0,
+ "6860": 1131332608.0,
+ "6865": 1131332608.0,
+ "6870": 1131332608.0,
+ "6875": 1131332608.0,
+ "6880": 1131332608.0,
+ "6885": 1131332608.0,
+ "6890": 1131332608.0,
+ "6895": 1131332608.0,
+ "6900": 1131332608.0,
+ "6905": 1131332608.0,
+ "6910": 1131332608.0,
+ "6915": 1131332608.0,
+ "6920": 1131332608.0,
+ "6925": 1131332608.0,
+ "6930": 1131332608.0,
+ "6935": 1131332608.0,
+ "6940": 1131332608.0,
+ "6945": 1131332608.0,
+ "6950": 1131332608.0,
+ "6955": 1131332608.0,
+ "6960": 1131332608.0,
+ "6965": 1131332608.0,
+ "6970": 1131332608.0,
+ "6975": 1131332608.0,
+ "6980": 1131332608.0,
+ "6985": 1131332608.0,
+ "6990": 1131332608.0,
+ "6995": 1131332608.0,
+ "7000": 1131332608.0,
+ "7005": 1131332608.0,
+ "7010": 1131332608.0,
+ "7015": 1131332608.0,
+ "7020": 1131332608.0,
+ "7025": 1131332608.0,
+ "7030": 1131332608.0,
+ "7035": 1131332608.0,
+ "7040": 1131332608.0,
+ "7045": 1131332608.0,
+ "7050": 1131332608.0,
+ "7055": 1131332608.0,
+ "7060": 1131332608.0,
+ "7065": 1131332608.0,
+ "7070": 1131332608.0,
+ "7075": 1131332608.0,
+ "7080": 1131332608.0,
+ "7085": 1131332608.0,
+ "7090": 1131332608.0,
+ "7095": 1131332608.0,
+ "7100": 1131332608.0,
+ "7105": 1131332608.0,
+ "7110": 1131332608.0,
+ "7115": 1131332608.0,
+ "7120": 1131332608.0,
+ "7125": 1131332608.0,
+ "7130": 1131332608.0,
+ "7135": 1131332608.0,
+ "7140": 1131332608.0,
+ "7145": 1131332608.0,
+ "7150": 1131332608.0,
+ "7155": 1131332608.0,
+ "7160": 1131332608.0,
+ "7165": 1131332608.0,
+ "7170": 1131332608.0,
+ "7175": 1131332608.0,
+ "7180": 1131332608.0,
+ "7185": 1131332608.0,
+ "7190": 1131332608.0,
+ "7195": 1131332608.0,
+ "7200": 1131332608.0,
+ "7205": 1131332608.0,
+ "7210": 1131332608.0,
+ "7215": 1131332608.0,
+ "7220": 1131332608.0,
+ "7225": 1131332608.0,
+ "7230": 1131332608.0,
+ "7235": 1131332608.0,
+ "7240": 1131332608.0,
+ "7245": 1131332608.0,
+ "7250": 1131332608.0,
+ "7255": 1131332608.0,
+ "7260": 1131332608.0,
+ "7265": 1131332608.0,
+ "7270": 1131332608.0,
+ "7275": 1131332608.0,
+ "7280": 1131332608.0,
+ "7285": 1131332608.0,
+ "7290": 1131332608.0,
+ "7295": 1131332608.0,
+ "7300": 1131332608.0,
+ "7305": 1131332608.0,
+ "7310": 1131332608.0,
+ "7315": 1131332608.0,
+ "7320": 1131332608.0,
+ "7325": 1131332608.0,
+ "7330": 1131332608.0,
+ "7335": 1131332608.0,
+ "7340": 1131332608.0,
+ "7345": 1131332608.0,
+ "7350": 1131332608.0,
+ "7355": 1131332608.0,
+ "7360": 1131332608.0,
+ "7365": 1131332608.0,
+ "7370": 1131332608.0,
+ "7375": 1131332608.0,
+ "7380": 1131332608.0,
+ "7385": 1131332608.0,
+ "7390": 1131332608.0,
+ "7395": 1131332608.0,
+ "7400": 1131332608.0,
+ "7405": 1131332608.0,
+ "7410": 1131332608.0,
+ "7415": 1131332608.0,
+ "7420": 1131332608.0,
+ "7425": 1131332608.0,
+ "7430": 1131332608.0,
+ "7435": 1131332608.0,
+ "7440": 1131332608.0,
+ "7445": 1131332608.0,
+ "7450": 1131332608.0,
+ "7455": 1131332608.0,
+ "7460": 1131332608.0,
+ "7465": 1131332608.0,
+ "7470": 1131332608.0,
+ "7475": 1131332608.0,
+ "7480": 1131332608.0,
+ "7485": 1131332608.0,
+ "7490": 1131332608.0,
+ "7495": 1131332608.0,
+ "7500": 1131332608.0,
+ "7505": 1131332608.0,
+ "7510": 1131332608.0,
+ "7515": 1131332608.0,
+ "7520": 1131332608.0,
+ "7525": 1131332608.0,
+ "7530": 1131332608.0,
+ "7535": 1131332608.0,
+ "7540": 1131332608.0,
+ "7545": 1131332608.0,
+ "7550": 1131332608.0,
+ "7555": 1131332608.0,
+ "7560": 1131332608.0,
+ "7565": 1131332608.0,
+ "7570": 1131332608.0,
+ "7575": 1131332608.0,
+ "7580": 1131332608.0,
+ "7585": 1131332608.0,
+ "7590": 1131332608.0,
+ "7595": 1131332608.0,
+ "7600": 1131332608.0,
+ "7605": 1131332608.0,
+ "7610": 1131332608.0,
+ "7615": 1131332608.0,
+ "7620": 1131332608.0,
+ "7625": 1131332608.0,
+ "7630": 1131332608.0,
+ "7635": 1131332608.0,
+ "7640": 1131332608.0,
+ "7645": 1131332608.0,
+ "7650": 1131332608.0,
+ "7655": 1131332608.0,
+ "7660": 1131332608.0,
+ "7665": 1131332608.0,
+ "7670": 1131332608.0,
+ "7675": 1131332608.0,
+ "7680": 1131332608.0,
+ "7685": 1131332608.0,
+ "7690": 1131332608.0,
+ "7695": 1131332608.0,
+ "7700": 1131332608.0,
+ "7705": 1131332608.0,
+ "7710": 1131332608.0,
+ "7715": 1131332608.0,
+ "7720": 1131332608.0,
+ "7725": 1131332608.0,
+ "7730": 1131332608.0,
+ "7735": 1131332608.0,
+ "7740": 1131332608.0,
+ "7745": 1131332608.0,
+ "7750": 1131332608.0,
+ "7755": 1131332608.0,
+ "7760": 1131332608.0,
+ "7765": 1131332608.0,
+ "7770": 1131332608.0,
+ "7775": 1131332608.0,
+ "7780": 1131332608.0,
+ "7785": 1131332608.0,
+ "7790": 1131332608.0,
+ "7795": 1131332608.0,
+ "7800": 1131332608.0,
+ "7805": 1131332608.0,
+ "7810": 1131332608.0,
+ "7815": 1131332608.0,
+ "7820": 1131332608.0,
+ "7825": 1131332608.0,
+ "7830": 1131332608.0,
+ "7835": 1131332608.0,
+ "7840": 1131332608.0,
+ "7845": 1131332608.0,
+ "7850": 1131332608.0,
+ "7855": 1131332608.0,
+ "7860": 1131332608.0,
+ "7865": 1131332608.0,
+ "7870": 1131332608.0,
+ "7875": 1131332608.0,
+ "7880": 1131332608.0,
+ "7885": 1131332608.0,
+ "7890": 1131332608.0,
+ "7895": 1131332608.0,
+ "7900": 1131332608.0,
+ "7905": 1131332608.0,
+ "7910": 1131332608.0,
+ "7915": 1131332608.0,
+ "7920": 1131332608.0,
+ "7925": 1131332608.0,
+ "7930": 1131332608.0,
+ "7935": 1131332608.0,
+ "7940": 1131332608.0,
+ "7945": 1131332608.0,
+ "7950": 1131332608.0,
+ "7955": 1131332608.0,
+ "7960": 1131332608.0,
+ "7965": 1131332608.0,
+ "7970": 1131332608.0,
+ "7975": 1131332608.0,
+ "7980": 1131332608.0,
+ "7985": 1131332608.0,
+ "7990": 1131332608.0,
+ "7995": 1131332608.0,
+ "8000": 1131332608.0,
+ "8005": 1131332608.0,
+ "8010": 1131332608.0,
+ "8015": 1131332608.0,
+ "8020": 1131332608.0,
+ "8025": 1131332608.0,
+ "8030": 1131332608.0,
+ "8035": 1131332608.0,
+ "8040": 1131332608.0,
+ "8045": 1131332608.0,
+ "8050": 1131332608.0,
+ "8055": 1131332608.0,
+ "8060": 1131332608.0,
+ "8065": 1131332608.0,
+ "8070": 1131332608.0,
+ "8075": 1131332608.0,
+ "8080": 1131332608.0,
+ "8085": 1131332608.0,
+ "8090": 1131332608.0,
+ "8095": 1131332608.0,
+ "8100": 1131332608.0,
+ "8105": 1131332608.0,
+ "8110": 1131332608.0,
+ "8115": 1131332608.0,
+ "8120": 1131332608.0,
+ "8125": 1131332608.0,
+ "8130": 1131332608.0,
+ "8135": 1131332608.0,
+ "8140": 1131332608.0,
+ "8145": 1131332608.0,
+ "8150": 1131332608.0,
+ "8155": 1131332608.0,
+ "8160": 1131332608.0,
+ "8165": 1131332608.0,
+ "8170": 1131332608.0,
+ "8175": 1131332608.0,
+ "8180": 1131332608.0,
+ "8185": 1131332608.0,
+ "8190": 1131332608.0,
+ "8195": 1131332608.0,
+ "8200": 1131332608.0,
+ "8205": 1131332608.0,
+ "8210": 1131332608.0,
+ "8215": 1131332608.0,
+ "8220": 1131332608.0,
+ "8225": 1131332608.0,
+ "8230": 1131332608.0,
+ "8235": 1131332608.0,
+ "8240": 1131332608.0,
+ "8245": 1131332608.0,
+ "8250": 1131332608.0,
+ "8255": 1131332608.0,
+ "8260": 1131332608.0,
+ "8265": 1131332608.0,
+ "8270": 1131332608.0,
+ "8275": 1131332608.0,
+ "8280": 1131332608.0,
+ "8285": 1131332608.0,
+ "8290": 1131332608.0,
+ "8295": 1131332608.0,
+ "8300": 1131332608.0,
+ "8305": 1131332608.0,
+ "8310": 1131332608.0,
+ "8315": 1131332608.0,
+ "8320": 1131332608.0,
+ "8325": 1131332608.0,
+ "8330": 1131332608.0,
+ "8335": 1131332608.0,
+ "8340": 1131332608.0,
+ "8345": 1131332608.0,
+ "8350": 1131332608.0,
+ "8355": 1131332608.0,
+ "8360": 1131332608.0,
+ "8365": 1131332608.0,
+ "8370": 1131332608.0,
+ "8375": 1131332608.0,
+ "8380": 1131332608.0,
+ "8385": 1131332608.0,
+ "8390": 1131332608.0,
+ "8395": 1131332608.0,
+ "8400": 1131332608.0,
+ "8405": 1131332608.0,
+ "8410": 1131332608.0,
+ "8415": 1131332608.0,
+ "8420": 1131332608.0,
+ "8425": 1131332608.0,
+ "8430": 1131332608.0,
+ "8435": 1131332608.0,
+ "8440": 1131332608.0,
+ "8445": 1131332608.0,
+ "8450": 1131332608.0,
+ "8455": 1131332608.0,
+ "8460": 1131332608.0,
+ "8465": 1131332608.0,
+ "8470": 1131332608.0,
+ "8475": 1131332608.0,
+ "8480": 1131332608.0,
+ "8485": 1131332608.0,
+ "8490": 1131332608.0,
+ "8495": 1131332608.0,
+ "8500": 1131332608.0,
+ "8505": 1131332608.0,
+ "8510": 1131332608.0,
+ "8515": 1131332608.0,
+ "8520": 1131332608.0,
+ "8525": 1131332608.0,
+ "8530": 1131332608.0,
+ "8535": 1131332608.0,
+ "8540": 1131332608.0,
+ "8545": 1131332608.0,
+ "8550": 1131332608.0,
+ "8555": 1131332608.0,
+ "8560": 1131332608.0,
+ "8565": 1131332608.0,
+ "8570": 1131332608.0,
+ "8575": 1131332608.0,
+ "8580": 1131332608.0,
+ "8585": 1131332608.0,
+ "8590": 1131332608.0,
+ "8595": 1131332608.0,
+ "8600": 1131332608.0,
+ "8605": 1131332608.0,
+ "8610": 1131332608.0,
+ "8615": 1131332608.0,
+ "8620": 1131332608.0,
+ "8625": 1131332608.0,
+ "8630": 1131332608.0,
+ "8635": 1131332608.0,
+ "8640": 1131332608.0,
+ "8645": 1131332608.0,
+ "8650": 1131332608.0,
+ "8655": 1131332608.0,
+ "8660": 1131332608.0,
+ "8665": 1131332608.0,
+ "8670": 1131332608.0,
+ "8675": 1131332608.0,
+ "8680": 1131332608.0,
+ "8685": 1131332608.0,
+ "8690": 1131332608.0,
+ "8695": 1131332608.0,
+ "8700": 1131332608.0,
+ "8705": 1131332608.0,
+ "8710": 1131332608.0,
+ "8715": 1131332608.0,
+ "8720": 1131332608.0,
+ "8725": 1131332608.0,
+ "8730": 1131332608.0,
+ "8735": 1131332608.0,
+ "8740": 1131332608.0,
+ "8745": 1131332608.0,
+ "8750": 1131332608.0,
+ "8755": 1131332608.0,
+ "8760": 1131332608.0,
+ "8765": 1131332608.0,
+ "8770": 1131332608.0,
+ "8775": 1131332608.0,
+ "8780": 1131332608.0,
+ "8785": 1131332608.0,
+ "8790": 1131332608.0,
+ "8795": 1131332608.0,
+ "8800": 1131332608.0,
+ "8805": 1131332608.0,
+ "8810": 1131332608.0,
+ "8815": 1131332608.0,
+ "8820": 1131332608.0,
+ "8825": 1131332608.0,
+ "8830": 1131332608.0,
+ "8835": 1131332608.0,
+ "8840": 1131332608.0,
+ "8845": 1131332608.0,
+ "8850": 1131332608.0,
+ "8855": 1131332608.0,
+ "8860": 1131332608.0,
+ "8865": 1131332608.0,
+ "8870": 1131332608.0,
+ "8875": 1131332608.0,
+ "8880": 1131332608.0,
+ "8885": 1131332608.0,
+ "8890": 1131332608.0,
+ "8895": 1131332608.0,
+ "8900": 1131332608.0,
+ "8905": 1131332608.0,
+ "8910": 1131332608.0,
+ "8915": 1131332608.0,
+ "8920": 1131332608.0,
+ "8925": 1131332608.0,
+ "8930": 1131332608.0,
+ "8935": 1131332608.0,
+ "8940": 1131332608.0,
+ "8945": 1131332608.0,
+ "8950": 1131332608.0,
+ "8955": 1131332608.0,
+ "8960": 1131332608.0,
+ "8965": 1131332608.0,
+ "8970": 1131332608.0,
+ "8975": 1131332608.0,
+ "8980": 1131332608.0,
+ "8985": 1131332608.0,
+ "8990": 1131332608.0,
+ "8995": 1131332608.0,
+ "9000": 1131332608.0,
+ "9005": 1131332608.0,
+ "9010": 1131332608.0,
+ "9015": 1131332608.0,
+ "9020": 1131332608.0,
+ "9025": 1131332608.0,
+ "9030": 1131332608.0,
+ "9035": 1131332608.0,
+ "9040": 1131332608.0,
+ "9045": 1131332608.0,
+ "9050": 1131332608.0,
+ "9055": 1131332608.0,
+ "9060": 1131332608.0,
+ "9065": 1131332608.0,
+ "9070": 1131332608.0,
+ "9075": 1131332608.0,
+ "9080": 1131332608.0,
+ "9085": 1131332608.0,
+ "9090": 1131332608.0,
+ "9095": 1131332608.0,
+ "9100": 1131332608.0,
+ "9105": 1131332608.0,
+ "9110": 1131332608.0,
+ "9115": 1131332608.0,
+ "9120": 1131332608.0,
+ "9125": 1131332608.0,
+ "9130": 1131332608.0,
+ "9135": 1131332608.0,
+ "9140": 1131332608.0,
+ "9145": 1131332608.0,
+ "9150": 1131332608.0,
+ "9155": 1131332608.0,
+ "9160": 1131332608.0,
+ "9165": 1131332608.0,
+ "9170": 1131332608.0,
+ "9175": 1131332608.0,
+ "9180": 1131332608.0,
+ "9185": 1131332608.0,
+ "9190": 1131332608.0,
+ "9195": 1131332608.0,
+ "9200": 1131332608.0,
+ "9205": 1131332608.0,
+ "9210": 1131332608.0,
+ "9215": 1131332608.0,
+ "9220": 1131332608.0,
+ "9225": 1131332608.0,
+ "9230": 1131332608.0,
+ "9235": 1131332608.0,
+ "9240": 1131332608.0,
+ "9245": 1131332608.0,
+ "9250": 1131332608.0,
+ "9255": 1131332608.0,
+ "9260": 1131332608.0,
+ "9265": 1131332608.0,
+ "9270": 1131332608.0,
+ "9275": 1131332608.0,
+ "9280": 1131332608.0,
+ "9285": 1131332608.0,
+ "9290": 1131332608.0,
+ "9295": 1131332608.0,
+ "9300": 1131332608.0,
+ "9305": 1131332608.0,
+ "9310": 1131332608.0,
+ "9315": 1131332608.0,
+ "9320": 1131332608.0,
+ "9325": 1131332608.0,
+ "9330": 1131332608.0,
+ "9335": 1131332608.0,
+ "9340": 1131332608.0,
+ "9345": 1131332608.0,
+ "9350": 1131332608.0,
+ "9355": 1131332608.0,
+ "9360": 1131332608.0,
+ "9365": 1131332608.0,
+ "9370": 1131332608.0,
+ "9375": 1131332608.0,
+ "9380": 1131332608.0,
+ "9385": 1131332608.0,
+ "9390": 1131332608.0,
+ "9395": 1131332608.0,
+ "9400": 1131332608.0,
+ "9405": 1131332608.0,
+ "9410": 1131332608.0,
+ "9415": 1131332608.0,
+ "9420": 1131332608.0,
+ "9425": 1131332608.0,
+ "9430": 1131332608.0,
+ "9435": 1131332608.0,
+ "9440": 1131332608.0,
+ "9445": 1131332608.0,
+ "9450": 1131332608.0,
+ "9455": 1131332608.0,
+ "9460": 1131332608.0,
+ "9465": 1131332608.0,
+ "9470": 1131332608.0,
+ "9475": 1131332608.0,
+ "9480": 1131332608.0,
+ "9485": 1131332608.0,
+ "9490": 1131332608.0,
+ "9495": 1131332608.0,
+ "9500": 1131332608.0,
+ "9505": 1131332608.0,
+ "9510": 1131332608.0,
+ "9515": 1131332608.0,
+ "9520": 1131332608.0,
+ "9525": 1131332608.0,
+ "9530": 1131332608.0,
+ "9535": 1131332608.0,
+ "9540": 1131332608.0,
+ "9545": 1131332608.0,
+ "9550": 1131332608.0,
+ "9555": 1131332608.0,
+ "9560": 1131332608.0,
+ "9565": 1131332608.0,
+ "9570": 1131332608.0,
+ "9575": 1131332608.0,
+ "9580": 1131332608.0,
+ "9585": 1131332608.0,
+ "9590": 1131332608.0,
+ "9595": 1131332608.0,
+ "9600": 1131332608.0,
+ "9605": 1131332608.0,
+ "9610": 1131332608.0,
+ "9615": 1131332608.0,
+ "9620": 1131332608.0,
+ "9625": 1131332608.0,
+ "9630": 1131332608.0,
+ "9635": 1131332608.0,
+ "9640": 1131332608.0,
+ "9645": 1131332608.0,
+ "9650": 1131332608.0,
+ "9655": 1131332608.0,
+ "9660": 1131332608.0,
+ "9665": 1131332608.0,
+ "9670": 1131332608.0,
+ "9675": 1131332608.0,
+ "9680": 1131332608.0,
+ "9685": 1131332608.0,
+ "9690": 1131332608.0,
+ "9695": 1131332608.0,
+ "9700": 1131332608.0,
+ "9705": 1131332608.0,
+ "9710": 1131332608.0,
+ "9715": 1131332608.0,
+ "9720": 1131332608.0,
+ "9725": 1131332608.0,
+ "9730": 1131332608.0,
+ "9735": 1131332608.0,
+ "9740": 1131332608.0,
+ "9745": 1131332608.0,
+ "9750": 1131332608.0,
+ "9755": 1131332608.0,
+ "9760": 1131332608.0,
+ "9765": 1131332608.0,
+ "9770": 1131332608.0,
+ "9775": 1131332608.0,
+ "9780": 1131332608.0,
+ "9785": 1131332608.0,
+ "9790": 1131332608.0,
+ "9795": 1131332608.0,
+ "9800": 1131332608.0,
+ "9805": 1131332608.0,
+ "9810": 1131332608.0,
+ "9815": 1131332608.0,
+ "9820": 1131332608.0,
+ "9825": 1131332608.0,
+ "9830": 1131332608.0,
+ "9835": 1131332608.0,
+ "9840": 1131332608.0,
+ "9845": 1131332608.0,
+ "9850": 1131332608.0,
+ "9855": 1131332608.0,
+ "9860": 1131332608.0,
+ "9865": 1131332608.0,
+ "9870": 1131332608.0,
+ "9875": 1131332608.0,
+ "9880": 1131332608.0,
+ "9885": 1131332608.0,
+ "9890": 1131332608.0,
+ "9895": 1131332608.0,
+ "9900": 1131332608.0,
+ "9905": 1131332608.0,
+ "9910": 1131332608.0,
+ "9915": 1131332608.0,
+ "9920": 1131332608.0,
+ "9925": 1131332608.0,
+ "9930": 1131332608.0,
+ "9935": 1131332608.0,
+ "9940": 1131332608.0,
+ "9945": 1131332608.0,
+ "9950": 1131332608.0,
+ "9955": 1131332608.0,
+ "9960": 1131332608.0,
+ "9965": 1131332608.0,
+ "9970": 1131332608.0,
+ "9975": 1131332608.0,
+ "9980": 1131332608.0,
+ "9985": 1131332608.0,
+ "9990": 1131332608.0,
+ "9995": 1131332608.0,
+ "10000": 1131332608.0
+ }
+ },
+ "mem-max-allocated-bytes": {
+ "start_step": 1,
+ "end_step": 10000,
+ "step_interval": 5,
+ "values": {
+ "1": 6551484416.0,
+ "5": 6995931136.0,
+ "10": 6995931136.0,
+ "15": 6995931136.0,
+ "20": 6995931136.0,
+ "25": 6995931136.0,
+ "30": 6995931136.0,
+ "35": 6995931136.0,
+ "40": 6995931136.0,
+ "45": 6995931136.0,
+ "50": 6995931136.0,
+ "55": 6995931136.0,
+ "60": 6995931136.0,
+ "65": 6995931136.0,
+ "70": 6995931136.0,
+ "75": 6995931136.0,
+ "80": 6995931136.0,
+ "85": 6995931136.0,
+ "90": 6995931136.0,
+ "95": 6995931136.0,
+ "100": 6995931136.0,
+ "105": 6995931136.0,
+ "110": 6995931136.0,
+ "115": 6995931136.0,
+ "120": 6995931136.0,
+ "125": 6995931136.0,
+ "130": 6995931136.0,
+ "135": 6995931136.0,
+ "140": 6995931136.0,
+ "145": 6995931136.0,
+ "150": 6995931136.0,
+ "155": 6995931136.0,
+ "160": 6995931136.0,
+ "165": 6995931136.0,
+ "170": 6995931136.0,
+ "175": 6995931136.0,
+ "180": 6995931136.0,
+ "185": 6995931136.0,
+ "190": 6995931136.0,
+ "195": 6995931136.0,
+ "200": 6995931136.0,
+ "205": 6995931136.0,
+ "210": 6995931136.0,
+ "215": 6995931136.0,
+ "220": 6995931136.0,
+ "225": 6995931136.0,
+ "230": 6995931136.0,
+ "235": 6995931136.0,
+ "240": 6995931136.0,
+ "245": 6995931136.0,
+ "250": 6995931136.0,
+ "255": 6995931136.0,
+ "260": 6995931136.0,
+ "265": 6995931136.0,
+ "270": 6995931136.0,
+ "275": 6995931136.0,
+ "280": 6995931136.0,
+ "285": 6995931136.0,
+ "290": 6995931136.0,
+ "295": 6995931136.0,
+ "300": 6995931136.0,
+ "305": 6995931136.0,
+ "310": 6995931136.0,
+ "315": 6995931136.0,
+ "320": 6995931136.0,
+ "325": 6995931136.0,
+ "330": 6995931136.0,
+ "335": 6995931136.0,
+ "340": 6995931136.0,
+ "345": 6995931136.0,
+ "350": 6995931136.0,
+ "355": 6995931136.0,
+ "360": 6995931136.0,
+ "365": 6995931136.0,
+ "370": 6995931136.0,
+ "375": 6995931136.0,
+ "380": 6995931136.0,
+ "385": 6995931136.0,
+ "390": 6995931136.0,
+ "395": 6995931136.0,
+ "400": 6995931136.0,
+ "405": 6995931136.0,
+ "410": 6995931136.0,
+ "415": 6995931136.0,
+ "420": 6995931136.0,
+ "425": 6995931136.0,
+ "430": 6995931136.0,
+ "435": 6995931136.0,
+ "440": 6995931136.0,
+ "445": 6995931136.0,
+ "450": 6995931136.0,
+ "455": 6995931136.0,
+ "460": 6995931136.0,
+ "465": 6995931136.0,
+ "470": 6995931136.0,
+ "475": 6995931136.0,
+ "480": 6995931136.0,
+ "485": 6995931136.0,
+ "490": 6995931136.0,
+ "495": 6995931136.0,
+ "500": 6995931136.0,
+ "505": 6995931136.0,
+ "510": 6995931136.0,
+ "515": 6995931136.0,
+ "520": 6995931136.0,
+ "525": 6995931136.0,
+ "530": 6995931136.0,
+ "535": 6995931136.0,
+ "540": 6995931136.0,
+ "545": 6995931136.0,
+ "550": 6995931136.0,
+ "555": 6995931136.0,
+ "560": 6995931136.0,
+ "565": 6995931136.0,
+ "570": 6995931136.0,
+ "575": 6995931136.0,
+ "580": 6995931136.0,
+ "585": 6995931136.0,
+ "590": 6995931136.0,
+ "595": 6995931136.0,
+ "600": 6995931136.0,
+ "605": 6995931136.0,
+ "610": 6995931136.0,
+ "615": 6995931136.0,
+ "620": 6995931136.0,
+ "625": 6995931136.0,
+ "630": 6995931136.0,
+ "635": 6995931136.0,
+ "640": 6995931136.0,
+ "645": 6995931136.0,
+ "650": 6995931136.0,
+ "655": 6995931136.0,
+ "660": 6995931136.0,
+ "665": 6995931136.0,
+ "670": 6995931136.0,
+ "675": 6995931136.0,
+ "680": 6995931136.0,
+ "685": 6995931136.0,
+ "690": 6995931136.0,
+ "695": 6995931136.0,
+ "700": 6995931136.0,
+ "705": 6995931136.0,
+ "710": 6995931136.0,
+ "715": 6995931136.0,
+ "720": 6995931136.0,
+ "725": 6995931136.0,
+ "730": 6995931136.0,
+ "735": 6995931136.0,
+ "740": 6995931136.0,
+ "745": 6995931136.0,
+ "750": 6995931136.0,
+ "755": 6995931136.0,
+ "760": 6995931136.0,
+ "765": 6995931136.0,
+ "770": 6995931136.0,
+ "775": 6995931136.0,
+ "780": 6995931136.0,
+ "785": 6995931136.0,
+ "790": 6995931136.0,
+ "795": 6995931136.0,
+ "800": 6995931136.0,
+ "805": 6995931136.0,
+ "810": 6995931136.0,
+ "815": 6995931136.0,
+ "820": 6995931136.0,
+ "825": 6995931136.0,
+ "830": 6995931136.0,
+ "835": 6995931136.0,
+ "840": 6995931136.0,
+ "845": 6995931136.0,
+ "850": 6995931136.0,
+ "855": 6995931136.0,
+ "860": 6995931136.0,
+ "865": 6995931136.0,
+ "870": 6995931136.0,
+ "875": 6995931136.0,
+ "880": 6995931136.0,
+ "885": 6995931136.0,
+ "890": 6995931136.0,
+ "895": 6995931136.0,
+ "900": 6995931136.0,
+ "905": 6995931136.0,
+ "910": 6995931136.0,
+ "915": 6995931136.0,
+ "920": 6995931136.0,
+ "925": 6995931136.0,
+ "930": 6995931136.0,
+ "935": 6995931136.0,
+ "940": 6995931136.0,
+ "945": 6995931136.0,
+ "950": 6995931136.0,
+ "955": 6995931136.0,
+ "960": 6995931136.0,
+ "965": 6995931136.0,
+ "970": 6995931136.0,
+ "975": 6995931136.0,
+ "980": 6995931136.0,
+ "985": 6995931136.0,
+ "990": 6995931136.0,
+ "995": 6995931136.0,
+ "1000": 6995931136.0,
+ "1005": 6995931136.0,
+ "1010": 6995931136.0,
+ "1015": 6995931136.0,
+ "1020": 6995931136.0,
+ "1025": 6995931136.0,
+ "1030": 6995931136.0,
+ "1035": 6995931136.0,
+ "1040": 6995931136.0,
+ "1045": 6995931136.0,
+ "1050": 6995931136.0,
+ "1055": 6995931136.0,
+ "1060": 6995931136.0,
+ "1065": 6995931136.0,
+ "1070": 6995931136.0,
+ "1075": 6995931136.0,
+ "1080": 6995931136.0,
+ "1085": 6995931136.0,
+ "1090": 6995931136.0,
+ "1095": 6995931136.0,
+ "1100": 6995931136.0,
+ "1105": 6995931136.0,
+ "1110": 6995931136.0,
+ "1115": 6995931136.0,
+ "1120": 6995931136.0,
+ "1125": 6995931136.0,
+ "1130": 6995931136.0,
+ "1135": 6995931136.0,
+ "1140": 6995931136.0,
+ "1145": 6995931136.0,
+ "1150": 6995931136.0,
+ "1155": 6995931136.0,
+ "1160": 6995931136.0,
+ "1165": 6995931136.0,
+ "1170": 6995931136.0,
+ "1175": 6995931136.0,
+ "1180": 6995931136.0,
+ "1185": 6995931136.0,
+ "1190": 6995931136.0,
+ "1195": 6995931136.0,
+ "1200": 6995931136.0,
+ "1205": 6995931136.0,
+ "1210": 6995931136.0,
+ "1215": 6995931136.0,
+ "1220": 6995931136.0,
+ "1225": 6995931136.0,
+ "1230": 6995931136.0,
+ "1235": 6995931136.0,
+ "1240": 6995931136.0,
+ "1245": 6995931136.0,
+ "1250": 6995931136.0,
+ "1255": 6995931136.0,
+ "1260": 6995931136.0,
+ "1265": 6995931136.0,
+ "1270": 6995931136.0,
+ "1275": 6995931136.0,
+ "1280": 6995931136.0,
+ "1285": 6995931136.0,
+ "1290": 6995931136.0,
+ "1295": 6995931136.0,
+ "1300": 6995931136.0,
+ "1305": 6995931136.0,
+ "1310": 6995931136.0,
+ "1315": 6995931136.0,
+ "1320": 6995931136.0,
+ "1325": 6995931136.0,
+ "1330": 6995931136.0,
+ "1335": 6995931136.0,
+ "1340": 6995931136.0,
+ "1345": 6995931136.0,
+ "1350": 6995931136.0,
+ "1355": 6995931136.0,
+ "1360": 6995931136.0,
+ "1365": 6995931136.0,
+ "1370": 6995931136.0,
+ "1375": 6995931136.0,
+ "1380": 6995931136.0,
+ "1385": 6995931136.0,
+ "1390": 6995931136.0,
+ "1395": 6995931136.0,
+ "1400": 6995931136.0,
+ "1405": 6995931136.0,
+ "1410": 6995931136.0,
+ "1415": 6995931136.0,
+ "1420": 6995931136.0,
+ "1425": 6995931136.0,
+ "1430": 6995931136.0,
+ "1435": 6995931136.0,
+ "1440": 6995931136.0,
+ "1445": 6995931136.0,
+ "1450": 6995931136.0,
+ "1455": 6995931136.0,
+ "1460": 6995931136.0,
+ "1465": 6995931136.0,
+ "1470": 6995931136.0,
+ "1475": 6995931136.0,
+ "1480": 6995931136.0,
+ "1485": 6995931136.0,
+ "1490": 6995931136.0,
+ "1495": 6995931136.0,
+ "1500": 6995931136.0,
+ "1505": 6995931136.0,
+ "1510": 6995931136.0,
+ "1515": 6995931136.0,
+ "1520": 6995931136.0,
+ "1525": 6995931136.0,
+ "1530": 6995931136.0,
+ "1535": 6995931136.0,
+ "1540": 6995931136.0,
+ "1545": 6995931136.0,
+ "1550": 6995931136.0,
+ "1555": 6995931136.0,
+ "1560": 6995931136.0,
+ "1565": 6995931136.0,
+ "1570": 6995931136.0,
+ "1575": 6995931136.0,
+ "1580": 6995931136.0,
+ "1585": 6995931136.0,
+ "1590": 6995931136.0,
+ "1595": 6995931136.0,
+ "1600": 6995931136.0,
+ "1605": 6995931136.0,
+ "1610": 6995931136.0,
+ "1615": 6995931136.0,
+ "1620": 6995931136.0,
+ "1625": 6995931136.0,
+ "1630": 6995931136.0,
+ "1635": 6995931136.0,
+ "1640": 6995931136.0,
+ "1645": 6995931136.0,
+ "1650": 6995931136.0,
+ "1655": 6995931136.0,
+ "1660": 6995931136.0,
+ "1665": 6995931136.0,
+ "1670": 6995931136.0,
+ "1675": 6995931136.0,
+ "1680": 6995931136.0,
+ "1685": 6995931136.0,
+ "1690": 6995931136.0,
+ "1695": 6995931136.0,
+ "1700": 6995931136.0,
+ "1705": 6995931136.0,
+ "1710": 6995931136.0,
+ "1715": 6995931136.0,
+ "1720": 6995931136.0,
+ "1725": 6995931136.0,
+ "1730": 6995931136.0,
+ "1735": 6995931136.0,
+ "1740": 6995931136.0,
+ "1745": 6995931136.0,
+ "1750": 6995931136.0,
+ "1755": 6995931136.0,
+ "1760": 6995931136.0,
+ "1765": 6995931136.0,
+ "1770": 6995931136.0,
+ "1775": 6995931136.0,
+ "1780": 6995931136.0,
+ "1785": 6995931136.0,
+ "1790": 6995931136.0,
+ "1795": 6995931136.0,
+ "1800": 6995931136.0,
+ "1805": 6995931136.0,
+ "1810": 6995931136.0,
+ "1815": 6995931136.0,
+ "1820": 6995931136.0,
+ "1825": 6995931136.0,
+ "1830": 6995931136.0,
+ "1835": 6995931136.0,
+ "1840": 6995931136.0,
+ "1845": 6995931136.0,
+ "1850": 6995931136.0,
+ "1855": 6995931136.0,
+ "1860": 6995931136.0,
+ "1865": 6995931136.0,
+ "1870": 6995931136.0,
+ "1875": 6995931136.0,
+ "1880": 6995931136.0,
+ "1885": 6995931136.0,
+ "1890": 6995931136.0,
+ "1895": 6995931136.0,
+ "1900": 6995931136.0,
+ "1905": 6995931136.0,
+ "1910": 6995931136.0,
+ "1915": 6995931136.0,
+ "1920": 6995931136.0,
+ "1925": 6995931136.0,
+ "1930": 6995931136.0,
+ "1935": 6995931136.0,
+ "1940": 6995931136.0,
+ "1945": 6995931136.0,
+ "1950": 6995931136.0,
+ "1955": 6995931136.0,
+ "1960": 6995931136.0,
+ "1965": 6995931136.0,
+ "1970": 6995931136.0,
+ "1975": 6995931136.0,
+ "1980": 6995931136.0,
+ "1985": 6995931136.0,
+ "1990": 6995931136.0,
+ "1995": 6995931136.0,
+ "2000": 6995931136.0,
+ "2005": 7008514048.0,
+ "2010": 7008514048.0,
+ "2015": 7008514048.0,
+ "2020": 7008514048.0,
+ "2025": 7008514048.0,
+ "2030": 7008514048.0,
+ "2035": 7008514048.0,
+ "2040": 7008514048.0,
+ "2045": 7008514048.0,
+ "2050": 7008514048.0,
+ "2055": 7008514048.0,
+ "2060": 7008514048.0,
+ "2065": 7008514048.0,
+ "2070": 7008514048.0,
+ "2075": 7008514048.0,
+ "2080": 7008514048.0,
+ "2085": 7008514048.0,
+ "2090": 7008514048.0,
+ "2095": 7008514048.0,
+ "2100": 7008514048.0,
+ "2105": 7008514048.0,
+ "2110": 7008514048.0,
+ "2115": 7008514048.0,
+ "2120": 7008514048.0,
+ "2125": 7008514048.0,
+ "2130": 7008514048.0,
+ "2135": 7008514048.0,
+ "2140": 7008514048.0,
+ "2145": 7008514048.0,
+ "2150": 7008514048.0,
+ "2155": 7008514048.0,
+ "2160": 7008514048.0,
+ "2165": 7008514048.0,
+ "2170": 7008514048.0,
+ "2175": 7008514048.0,
+ "2180": 7008514048.0,
+ "2185": 7008514048.0,
+ "2190": 7008514048.0,
+ "2195": 7008514048.0,
+ "2200": 7008514048.0,
+ "2205": 7008514048.0,
+ "2210": 7008514048.0,
+ "2215": 7008514048.0,
+ "2220": 7008514048.0,
+ "2225": 7008514048.0,
+ "2230": 7008514048.0,
+ "2235": 7008514048.0,
+ "2240": 7008514048.0,
+ "2245": 7008514048.0,
+ "2250": 7008514048.0,
+ "2255": 7008514048.0,
+ "2260": 7008514048.0,
+ "2265": 7008514048.0,
+ "2270": 7008514048.0,
+ "2275": 7008514048.0,
+ "2280": 7008514048.0,
+ "2285": 7008514048.0,
+ "2290": 7008514048.0,
+ "2295": 7008514048.0,
+ "2300": 7008514048.0,
+ "2305": 7008514048.0,
+ "2310": 7008514048.0,
+ "2315": 7008514048.0,
+ "2320": 7008514048.0,
+ "2325": 7008514048.0,
+ "2330": 7008514048.0,
+ "2335": 7008514048.0,
+ "2340": 7008514048.0,
+ "2345": 7008514048.0,
+ "2350": 7008514048.0,
+ "2355": 7008514048.0,
+ "2360": 7008514048.0,
+ "2365": 7008514048.0,
+ "2370": 7008514048.0,
+ "2375": 7008514048.0,
+ "2380": 7008514048.0,
+ "2385": 7008514048.0,
+ "2390": 7008514048.0,
+ "2395": 7008514048.0,
+ "2400": 7008514048.0,
+ "2405": 7008514048.0,
+ "2410": 7008514048.0,
+ "2415": 7008514048.0,
+ "2420": 7008514048.0,
+ "2425": 7008514048.0,
+ "2430": 7008514048.0,
+ "2435": 7008514048.0,
+ "2440": 7008514048.0,
+ "2445": 7008514048.0,
+ "2450": 7008514048.0,
+ "2455": 7008514048.0,
+ "2460": 7008514048.0,
+ "2465": 7008514048.0,
+ "2470": 7008514048.0,
+ "2475": 7008514048.0,
+ "2480": 7008514048.0,
+ "2485": 7008514048.0,
+ "2490": 7008514048.0,
+ "2495": 7008514048.0,
+ "2500": 7008514048.0,
+ "2505": 7008514048.0,
+ "2510": 7008514048.0,
+ "2515": 7008514048.0,
+ "2520": 7008514048.0,
+ "2525": 7008514048.0,
+ "2530": 7008514048.0,
+ "2535": 7008514048.0,
+ "2540": 7008514048.0,
+ "2545": 7008514048.0,
+ "2550": 7008514048.0,
+ "2555": 7008514048.0,
+ "2560": 7008514048.0,
+ "2565": 7008514048.0,
+ "2570": 7008514048.0,
+ "2575": 7008514048.0,
+ "2580": 7008514048.0,
+ "2585": 7008514048.0,
+ "2590": 7008514048.0,
+ "2595": 7008514048.0,
+ "2600": 7008514048.0,
+ "2605": 7008514048.0,
+ "2610": 7008514048.0,
+ "2615": 7008514048.0,
+ "2620": 7008514048.0,
+ "2625": 7008514048.0,
+ "2630": 7008514048.0,
+ "2635": 7008514048.0,
+ "2640": 7008514048.0,
+ "2645": 7008514048.0,
+ "2650": 7008514048.0,
+ "2655": 7008514048.0,
+ "2660": 7008514048.0,
+ "2665": 7008514048.0,
+ "2670": 7008514048.0,
+ "2675": 7008514048.0,
+ "2680": 7008514048.0,
+ "2685": 7008514048.0,
+ "2690": 7008514048.0,
+ "2695": 7008514048.0,
+ "2700": 7008514048.0,
+ "2705": 7008514048.0,
+ "2710": 7008514048.0,
+ "2715": 7008514048.0,
+ "2720": 7008514048.0,
+ "2725": 7008514048.0,
+ "2730": 7008514048.0,
+ "2735": 7008514048.0,
+ "2740": 7008514048.0,
+ "2745": 7008514048.0,
+ "2750": 7008514048.0,
+ "2755": 7008514048.0,
+ "2760": 7008514048.0,
+ "2765": 7008514048.0,
+ "2770": 7008514048.0,
+ "2775": 7008514048.0,
+ "2780": 7008514048.0,
+ "2785": 7008514048.0,
+ "2790": 7008514048.0,
+ "2795": 7008514048.0,
+ "2800": 7008514048.0,
+ "2805": 7008514048.0,
+ "2810": 7008514048.0,
+ "2815": 7008514048.0,
+ "2820": 7008514048.0,
+ "2825": 7008514048.0,
+ "2830": 7008514048.0,
+ "2835": 7008514048.0,
+ "2840": 7008514048.0,
+ "2845": 7008514048.0,
+ "2850": 7008514048.0,
+ "2855": 7008514048.0,
+ "2860": 7008514048.0,
+ "2865": 7008514048.0,
+ "2870": 7008514048.0,
+ "2875": 7008514048.0,
+ "2880": 7008514048.0,
+ "2885": 7008514048.0,
+ "2890": 7008514048.0,
+ "2895": 7008514048.0,
+ "2900": 7008514048.0,
+ "2905": 7008514048.0,
+ "2910": 7008514048.0,
+ "2915": 7008514048.0,
+ "2920": 7008514048.0,
+ "2925": 7008514048.0,
+ "2930": 7008514048.0,
+ "2935": 7008514048.0,
+ "2940": 7008514048.0,
+ "2945": 7008514048.0,
+ "2950": 7008514048.0,
+ "2955": 7008514048.0,
+ "2960": 7008514048.0,
+ "2965": 7008514048.0,
+ "2970": 7008514048.0,
+ "2975": 7008514048.0,
+ "2980": 7008514048.0,
+ "2985": 7008514048.0,
+ "2990": 7008514048.0,
+ "2995": 7008514048.0,
+ "3000": 7008514048.0,
+ "3005": 7008514048.0,
+ "3010": 7008514048.0,
+ "3015": 7008514048.0,
+ "3020": 7008514048.0,
+ "3025": 7008514048.0,
+ "3030": 7008514048.0,
+ "3035": 7008514048.0,
+ "3040": 7008514048.0,
+ "3045": 7008514048.0,
+ "3050": 7008514048.0,
+ "3055": 7008514048.0,
+ "3060": 7008514048.0,
+ "3065": 7008514048.0,
+ "3070": 7008514048.0,
+ "3075": 7008514048.0,
+ "3080": 7008514048.0,
+ "3085": 7008514048.0,
+ "3090": 7008514048.0,
+ "3095": 7008514048.0,
+ "3100": 7008514048.0,
+ "3105": 7008514048.0,
+ "3110": 7008514048.0,
+ "3115": 7008514048.0,
+ "3120": 7008514048.0,
+ "3125": 7008514048.0,
+ "3130": 7008514048.0,
+ "3135": 7008514048.0,
+ "3140": 7008514048.0,
+ "3145": 7008514048.0,
+ "3150": 7008514048.0,
+ "3155": 7008514048.0,
+ "3160": 7008514048.0,
+ "3165": 7008514048.0,
+ "3170": 7008514048.0,
+ "3175": 7008514048.0,
+ "3180": 7008514048.0,
+ "3185": 7008514048.0,
+ "3190": 7008514048.0,
+ "3195": 7008514048.0,
+ "3200": 7008514048.0,
+ "3205": 7008514048.0,
+ "3210": 7008514048.0,
+ "3215": 7008514048.0,
+ "3220": 7008514048.0,
+ "3225": 7008514048.0,
+ "3230": 7008514048.0,
+ "3235": 7008514048.0,
+ "3240": 7008514048.0,
+ "3245": 7008514048.0,
+ "3250": 7008514048.0,
+ "3255": 7008514048.0,
+ "3260": 7008514048.0,
+ "3265": 7008514048.0,
+ "3270": 7008514048.0,
+ "3275": 7008514048.0,
+ "3280": 7008514048.0,
+ "3285": 7008514048.0,
+ "3290": 7008514048.0,
+ "3295": 7008514048.0,
+ "3300": 7008514048.0,
+ "3305": 7008514048.0,
+ "3310": 7008514048.0,
+ "3315": 7008514048.0,
+ "3320": 7008514048.0,
+ "3325": 7008514048.0,
+ "3330": 7008514048.0,
+ "3335": 7008514048.0,
+ "3340": 7008514048.0,
+ "3345": 7008514048.0,
+ "3350": 7008514048.0,
+ "3355": 7008514048.0,
+ "3360": 7008514048.0,
+ "3365": 7008514048.0,
+ "3370": 7008514048.0,
+ "3375": 7008514048.0,
+ "3380": 7008514048.0,
+ "3385": 7008514048.0,
+ "3390": 7008514048.0,
+ "3395": 7008514048.0,
+ "3400": 7008514048.0,
+ "3405": 7008514048.0,
+ "3410": 7008514048.0,
+ "3415": 7008514048.0,
+ "3420": 7008514048.0,
+ "3425": 7008514048.0,
+ "3430": 7008514048.0,
+ "3435": 7008514048.0,
+ "3440": 7008514048.0,
+ "3445": 7008514048.0,
+ "3450": 7008514048.0,
+ "3455": 7008514048.0,
+ "3460": 7008514048.0,
+ "3465": 7008514048.0,
+ "3470": 7008514048.0,
+ "3475": 7008514048.0,
+ "3480": 7008514048.0,
+ "3485": 7008514048.0,
+ "3490": 7008514048.0,
+ "3495": 7008514048.0,
+ "3500": 7008514048.0,
+ "3505": 7008514048.0,
+ "3510": 7008514048.0,
+ "3515": 7008514048.0,
+ "3520": 7008514048.0,
+ "3525": 7008514048.0,
+ "3530": 7008514048.0,
+ "3535": 7008514048.0,
+ "3540": 7008514048.0,
+ "3545": 7008514048.0,
+ "3550": 7008514048.0,
+ "3555": 7008514048.0,
+ "3560": 7008514048.0,
+ "3565": 7008514048.0,
+ "3570": 7008514048.0,
+ "3575": 7008514048.0,
+ "3580": 7008514048.0,
+ "3585": 7008514048.0,
+ "3590": 7008514048.0,
+ "3595": 7008514048.0,
+ "3600": 7008514048.0,
+ "3605": 7008514048.0,
+ "3610": 7008514048.0,
+ "3615": 7008514048.0,
+ "3620": 7008514048.0,
+ "3625": 7008514048.0,
+ "3630": 7008514048.0,
+ "3635": 7008514048.0,
+ "3640": 7008514048.0,
+ "3645": 7008514048.0,
+ "3650": 7008514048.0,
+ "3655": 7008514048.0,
+ "3660": 7008514048.0,
+ "3665": 7008514048.0,
+ "3670": 7008514048.0,
+ "3675": 7008514048.0,
+ "3680": 7008514048.0,
+ "3685": 7008514048.0,
+ "3690": 7008514048.0,
+ "3695": 7008514048.0,
+ "3700": 7008514048.0,
+ "3705": 7008514048.0,
+ "3710": 7008514048.0,
+ "3715": 7008514048.0,
+ "3720": 7008514048.0,
+ "3725": 7008514048.0,
+ "3730": 7008514048.0,
+ "3735": 7008514048.0,
+ "3740": 7008514048.0,
+ "3745": 7008514048.0,
+ "3750": 7008514048.0,
+ "3755": 7008514048.0,
+ "3760": 7008514048.0,
+ "3765": 7008514048.0,
+ "3770": 7008514048.0,
+ "3775": 7008514048.0,
+ "3780": 7008514048.0,
+ "3785": 7008514048.0,
+ "3790": 7008514048.0,
+ "3795": 7008514048.0,
+ "3800": 7008514048.0,
+ "3805": 7008514048.0,
+ "3810": 7008514048.0,
+ "3815": 7008514048.0,
+ "3820": 7008514048.0,
+ "3825": 7008514048.0,
+ "3830": 7008514048.0,
+ "3835": 7008514048.0,
+ "3840": 7008514048.0,
+ "3845": 7008514048.0,
+ "3850": 7008514048.0,
+ "3855": 7008514048.0,
+ "3860": 7008514048.0,
+ "3865": 7008514048.0,
+ "3870": 7008514048.0,
+ "3875": 7008514048.0,
+ "3880": 7008514048.0,
+ "3885": 7008514048.0,
+ "3890": 7008514048.0,
+ "3895": 7008514048.0,
+ "3900": 7008514048.0,
+ "3905": 7008514048.0,
+ "3910": 7008514048.0,
+ "3915": 7008514048.0,
+ "3920": 7008514048.0,
+ "3925": 7008514048.0,
+ "3930": 7008514048.0,
+ "3935": 7008514048.0,
+ "3940": 7008514048.0,
+ "3945": 7008514048.0,
+ "3950": 7008514048.0,
+ "3955": 7008514048.0,
+ "3960": 7008514048.0,
+ "3965": 7008514048.0,
+ "3970": 7008514048.0,
+ "3975": 7008514048.0,
+ "3980": 7008514048.0,
+ "3985": 7008514048.0,
+ "3990": 7008514048.0,
+ "3995": 7008514048.0,
+ "4000": 7008514048.0,
+ "4005": 7008514048.0,
+ "4010": 7008514048.0,
+ "4015": 7008514048.0,
+ "4020": 7008514048.0,
+ "4025": 7008514048.0,
+ "4030": 7008514048.0,
+ "4035": 7008514048.0,
+ "4040": 7008514048.0,
+ "4045": 7008514048.0,
+ "4050": 7008514048.0,
+ "4055": 7008514048.0,
+ "4060": 7008514048.0,
+ "4065": 7008514048.0,
+ "4070": 7008514048.0,
+ "4075": 7008514048.0,
+ "4080": 7008514048.0,
+ "4085": 7008514048.0,
+ "4090": 7008514048.0,
+ "4095": 7008514048.0,
+ "4100": 7008514048.0,
+ "4105": 7008514048.0,
+ "4110": 7008514048.0,
+ "4115": 7008514048.0,
+ "4120": 7008514048.0,
+ "4125": 7008514048.0,
+ "4130": 7008514048.0,
+ "4135": 7008514048.0,
+ "4140": 7008514048.0,
+ "4145": 7008514048.0,
+ "4150": 7008514048.0,
+ "4155": 7008514048.0,
+ "4160": 7008514048.0,
+ "4165": 7008514048.0,
+ "4170": 7008514048.0,
+ "4175": 7008514048.0,
+ "4180": 7008514048.0,
+ "4185": 7008514048.0,
+ "4190": 7008514048.0,
+ "4195": 7008514048.0,
+ "4200": 7008514048.0,
+ "4205": 7008514048.0,
+ "4210": 7008514048.0,
+ "4215": 7008514048.0,
+ "4220": 7008514048.0,
+ "4225": 7008514048.0,
+ "4230": 7008514048.0,
+ "4235": 7008514048.0,
+ "4240": 7008514048.0,
+ "4245": 7008514048.0,
+ "4250": 7008514048.0,
+ "4255": 7008514048.0,
+ "4260": 7008514048.0,
+ "4265": 7008514048.0,
+ "4270": 7008514048.0,
+ "4275": 7008514048.0,
+ "4280": 7008514048.0,
+ "4285": 7008514048.0,
+ "4290": 7008514048.0,
+ "4295": 7008514048.0,
+ "4300": 7008514048.0,
+ "4305": 7008514048.0,
+ "4310": 7008514048.0,
+ "4315": 7008514048.0,
+ "4320": 7008514048.0,
+ "4325": 7008514048.0,
+ "4330": 7008514048.0,
+ "4335": 7008514048.0,
+ "4340": 7008514048.0,
+ "4345": 7008514048.0,
+ "4350": 7008514048.0,
+ "4355": 7008514048.0,
+ "4360": 7008514048.0,
+ "4365": 7008514048.0,
+ "4370": 7008514048.0,
+ "4375": 7008514048.0,
+ "4380": 7008514048.0,
+ "4385": 7008514048.0,
+ "4390": 7008514048.0,
+ "4395": 7008514048.0,
+ "4400": 7008514048.0,
+ "4405": 7008514048.0,
+ "4410": 7008514048.0,
+ "4415": 7008514048.0,
+ "4420": 7008514048.0,
+ "4425": 7008514048.0,
+ "4430": 7008514048.0,
+ "4435": 7008514048.0,
+ "4440": 7008514048.0,
+ "4445": 7008514048.0,
+ "4450": 7008514048.0,
+ "4455": 7008514048.0,
+ "4460": 7008514048.0,
+ "4465": 7008514048.0,
+ "4470": 7008514048.0,
+ "4475": 7008514048.0,
+ "4480": 7008514048.0,
+ "4485": 7008514048.0,
+ "4490": 7008514048.0,
+ "4495": 7008514048.0,
+ "4500": 7008514048.0,
+ "4505": 7008514048.0,
+ "4510": 7008514048.0,
+ "4515": 7008514048.0,
+ "4520": 7008514048.0,
+ "4525": 7008514048.0,
+ "4530": 7008514048.0,
+ "4535": 7008514048.0,
+ "4540": 7008514048.0,
+ "4545": 7008514048.0,
+ "4550": 7008514048.0,
+ "4555": 7008514048.0,
+ "4560": 7008514048.0,
+ "4565": 7008514048.0,
+ "4570": 7008514048.0,
+ "4575": 7008514048.0,
+ "4580": 7008514048.0,
+ "4585": 7008514048.0,
+ "4590": 7008514048.0,
+ "4595": 7008514048.0,
+ "4600": 7008514048.0,
+ "4605": 7008514048.0,
+ "4610": 7008514048.0,
+ "4615": 7008514048.0,
+ "4620": 7008514048.0,
+ "4625": 7008514048.0,
+ "4630": 7008514048.0,
+ "4635": 7008514048.0,
+ "4640": 7008514048.0,
+ "4645": 7008514048.0,
+ "4650": 7008514048.0,
+ "4655": 7008514048.0,
+ "4660": 7008514048.0,
+ "4665": 7008514048.0,
+ "4670": 7008514048.0,
+ "4675": 7008514048.0,
+ "4680": 7008514048.0,
+ "4685": 7008514048.0,
+ "4690": 7008514048.0,
+ "4695": 7008514048.0,
+ "4700": 7008514048.0,
+ "4705": 7008514048.0,
+ "4710": 7008514048.0,
+ "4715": 7008514048.0,
+ "4720": 7008514048.0,
+ "4725": 7008514048.0,
+ "4730": 7008514048.0,
+ "4735": 7008514048.0,
+ "4740": 7008514048.0,
+ "4745": 7008514048.0,
+ "4750": 7008514048.0,
+ "4755": 7008514048.0,
+ "4760": 7008514048.0,
+ "4765": 7008514048.0,
+ "4770": 7008514048.0,
+ "4775": 7008514048.0,
+ "4780": 7008514048.0,
+ "4785": 7008514048.0,
+ "4790": 7008514048.0,
+ "4795": 7008514048.0,
+ "4800": 7008514048.0,
+ "4805": 7008514048.0,
+ "4810": 7008514048.0,
+ "4815": 7008514048.0,
+ "4820": 7008514048.0,
+ "4825": 7008514048.0,
+ "4830": 7008514048.0,
+ "4835": 7008514048.0,
+ "4840": 7008514048.0,
+ "4845": 7008514048.0,
+ "4850": 7008514048.0,
+ "4855": 7008514048.0,
+ "4860": 7008514048.0,
+ "4865": 7008514048.0,
+ "4870": 7008514048.0,
+ "4875": 7008514048.0,
+ "4880": 7008514048.0,
+ "4885": 7008514048.0,
+ "4890": 7008514048.0,
+ "4895": 7008514048.0,
+ "4900": 7008514048.0,
+ "4905": 7008514048.0,
+ "4910": 7008514048.0,
+ "4915": 7008514048.0,
+ "4920": 7008514048.0,
+ "4925": 7008514048.0,
+ "4930": 7008514048.0,
+ "4935": 7008514048.0,
+ "4940": 7008514048.0,
+ "4945": 7008514048.0,
+ "4950": 7008514048.0,
+ "4955": 7008514048.0,
+ "4960": 7008514048.0,
+ "4965": 7008514048.0,
+ "4970": 7008514048.0,
+ "4975": 7008514048.0,
+ "4980": 7008514048.0,
+ "4985": 7008514048.0,
+ "4990": 7008514048.0,
+ "4995": 7008514048.0,
+ "5000": 7008514048.0,
+ "5005": 7008514048.0,
+ "5010": 7008514048.0,
+ "5015": 7008514048.0,
+ "5020": 7008514048.0,
+ "5025": 7008514048.0,
+ "5030": 7008514048.0,
+ "5035": 7008514048.0,
+ "5040": 7008514048.0,
+ "5045": 7008514048.0,
+ "5050": 7008514048.0,
+ "5055": 7008514048.0,
+ "5060": 7008514048.0,
+ "5065": 7008514048.0,
+ "5070": 7008514048.0,
+ "5075": 7008514048.0,
+ "5080": 7008514048.0,
+ "5085": 7008514048.0,
+ "5090": 7008514048.0,
+ "5095": 7008514048.0,
+ "5100": 7008514048.0,
+ "5105": 7008514048.0,
+ "5110": 7008514048.0,
+ "5115": 7008514048.0,
+ "5120": 7008514048.0,
+ "5125": 7008514048.0,
+ "5130": 7008514048.0,
+ "5135": 7008514048.0,
+ "5140": 7008514048.0,
+ "5145": 7008514048.0,
+ "5150": 7008514048.0,
+ "5155": 7008514048.0,
+ "5160": 7008514048.0,
+ "5165": 7008514048.0,
+ "5170": 7008514048.0,
+ "5175": 7008514048.0,
+ "5180": 7008514048.0,
+ "5185": 7008514048.0,
+ "5190": 7008514048.0,
+ "5195": 7008514048.0,
+ "5200": 7008514048.0,
+ "5205": 7008514048.0,
+ "5210": 7008514048.0,
+ "5215": 7008514048.0,
+ "5220": 7008514048.0,
+ "5225": 7008514048.0,
+ "5230": 7008514048.0,
+ "5235": 7008514048.0,
+ "5240": 7008514048.0,
+ "5245": 7008514048.0,
+ "5250": 7008514048.0,
+ "5255": 7008514048.0,
+ "5260": 7008514048.0,
+ "5265": 7008514048.0,
+ "5270": 7008514048.0,
+ "5275": 7008514048.0,
+ "5280": 7008514048.0,
+ "5285": 7008514048.0,
+ "5290": 7008514048.0,
+ "5295": 7008514048.0,
+ "5300": 7008514048.0,
+ "5305": 7008514048.0,
+ "5310": 7008514048.0,
+ "5315": 7008514048.0,
+ "5320": 7008514048.0,
+ "5325": 7008514048.0,
+ "5330": 7008514048.0,
+ "5335": 7008514048.0,
+ "5340": 7008514048.0,
+ "5345": 7008514048.0,
+ "5350": 7008514048.0,
+ "5355": 7008514048.0,
+ "5360": 7008514048.0,
+ "5365": 7008514048.0,
+ "5370": 7008514048.0,
+ "5375": 7008514048.0,
+ "5380": 7008514048.0,
+ "5385": 7008514048.0,
+ "5390": 7008514048.0,
+ "5395": 7008514048.0,
+ "5400": 7008514048.0,
+ "5405": 7008514048.0,
+ "5410": 7008514048.0,
+ "5415": 7008514048.0,
+ "5420": 7008514048.0,
+ "5425": 7008514048.0,
+ "5430": 7008514048.0,
+ "5435": 7008514048.0,
+ "5440": 7008514048.0,
+ "5445": 7008514048.0,
+ "5450": 7008514048.0,
+ "5455": 7008514048.0,
+ "5460": 7008514048.0,
+ "5465": 7008514048.0,
+ "5470": 7008514048.0,
+ "5475": 7008514048.0,
+ "5480": 7008514048.0,
+ "5485": 7008514048.0,
+ "5490": 7008514048.0,
+ "5495": 7008514048.0,
+ "5500": 7008514048.0,
+ "5505": 7008514048.0,
+ "5510": 7008514048.0,
+ "5515": 7008514048.0,
+ "5520": 7008514048.0,
+ "5525": 7008514048.0,
+ "5530": 7008514048.0,
+ "5535": 7008514048.0,
+ "5540": 7008514048.0,
+ "5545": 7008514048.0,
+ "5550": 7008514048.0,
+ "5555": 7008514048.0,
+ "5560": 7008514048.0,
+ "5565": 7008514048.0,
+ "5570": 7008514048.0,
+ "5575": 7008514048.0,
+ "5580": 7008514048.0,
+ "5585": 7008514048.0,
+ "5590": 7008514048.0,
+ "5595": 7008514048.0,
+ "5600": 7008514048.0,
+ "5605": 7008514048.0,
+ "5610": 7008514048.0,
+ "5615": 7008514048.0,
+ "5620": 7008514048.0,
+ "5625": 7008514048.0,
+ "5630": 7008514048.0,
+ "5635": 7008514048.0,
+ "5640": 7008514048.0,
+ "5645": 7008514048.0,
+ "5650": 7008514048.0,
+ "5655": 7008514048.0,
+ "5660": 7008514048.0,
+ "5665": 7008514048.0,
+ "5670": 7008514048.0,
+ "5675": 7008514048.0,
+ "5680": 7008514048.0,
+ "5685": 7008514048.0,
+ "5690": 7008514048.0,
+ "5695": 7008514048.0,
+ "5700": 7008514048.0,
+ "5705": 7008514048.0,
+ "5710": 7008514048.0,
+ "5715": 7008514048.0,
+ "5720": 7008514048.0,
+ "5725": 7008514048.0,
+ "5730": 7008514048.0,
+ "5735": 7008514048.0,
+ "5740": 7008514048.0,
+ "5745": 7008514048.0,
+ "5750": 7008514048.0,
+ "5755": 7008514048.0,
+ "5760": 7008514048.0,
+ "5765": 7008514048.0,
+ "5770": 7008514048.0,
+ "5775": 7008514048.0,
+ "5780": 7008514048.0,
+ "5785": 7008514048.0,
+ "5790": 7008514048.0,
+ "5795": 7008514048.0,
+ "5800": 7008514048.0,
+ "5805": 7008514048.0,
+ "5810": 7008514048.0,
+ "5815": 7008514048.0,
+ "5820": 7008514048.0,
+ "5825": 7008514048.0,
+ "5830": 7008514048.0,
+ "5835": 7008514048.0,
+ "5840": 7008514048.0,
+ "5845": 7008514048.0,
+ "5850": 7008514048.0,
+ "5855": 7008514048.0,
+ "5860": 7008514048.0,
+ "5865": 7008514048.0,
+ "5870": 7008514048.0,
+ "5875": 7008514048.0,
+ "5880": 7008514048.0,
+ "5885": 7008514048.0,
+ "5890": 7008514048.0,
+ "5895": 7008514048.0,
+ "5900": 7008514048.0,
+ "5905": 7008514048.0,
+ "5910": 7008514048.0,
+ "5915": 7008514048.0,
+ "5920": 7008514048.0,
+ "5925": 7008514048.0,
+ "5930": 7008514048.0,
+ "5935": 7008514048.0,
+ "5940": 7008514048.0,
+ "5945": 7008514048.0,
+ "5950": 7008514048.0,
+ "5955": 7008514048.0,
+ "5960": 7008514048.0,
+ "5965": 7008514048.0,
+ "5970": 7008514048.0,
+ "5975": 7008514048.0,
+ "5980": 7008514048.0,
+ "5985": 7008514048.0,
+ "5990": 7008514048.0,
+ "5995": 7008514048.0,
+ "6000": 7008514048.0,
+ "6005": 7008514048.0,
+ "6010": 7008514048.0,
+ "6015": 7008514048.0,
+ "6020": 7008514048.0,
+ "6025": 7008514048.0,
+ "6030": 7008514048.0,
+ "6035": 7008514048.0,
+ "6040": 7008514048.0,
+ "6045": 7008514048.0,
+ "6050": 7008514048.0,
+ "6055": 7008514048.0,
+ "6060": 7008514048.0,
+ "6065": 7008514048.0,
+ "6070": 7008514048.0,
+ "6075": 7008514048.0,
+ "6080": 7008514048.0,
+ "6085": 7008514048.0,
+ "6090": 7008514048.0,
+ "6095": 7008514048.0,
+ "6100": 7008514048.0,
+ "6105": 7008514048.0,
+ "6110": 7008514048.0,
+ "6115": 7008514048.0,
+ "6120": 7008514048.0,
+ "6125": 7008514048.0,
+ "6130": 7008514048.0,
+ "6135": 7008514048.0,
+ "6140": 7008514048.0,
+ "6145": 7008514048.0,
+ "6150": 7008514048.0,
+ "6155": 7008514048.0,
+ "6160": 7008514048.0,
+ "6165": 7008514048.0,
+ "6170": 7008514048.0,
+ "6175": 7008514048.0,
+ "6180": 7008514048.0,
+ "6185": 7008514048.0,
+ "6190": 7008514048.0,
+ "6195": 7008514048.0,
+ "6200": 7008514048.0,
+ "6205": 7008514048.0,
+ "6210": 7008514048.0,
+ "6215": 7008514048.0,
+ "6220": 7008514048.0,
+ "6225": 7008514048.0,
+ "6230": 7008514048.0,
+ "6235": 7008514048.0,
+ "6240": 7008514048.0,
+ "6245": 7008514048.0,
+ "6250": 7008514048.0,
+ "6255": 7008514048.0,
+ "6260": 7008514048.0,
+ "6265": 7008514048.0,
+ "6270": 7008514048.0,
+ "6275": 7008514048.0,
+ "6280": 7008514048.0,
+ "6285": 7008514048.0,
+ "6290": 7008514048.0,
+ "6295": 7008514048.0,
+ "6300": 7008514048.0,
+ "6305": 7008514048.0,
+ "6310": 7008514048.0,
+ "6315": 7008514048.0,
+ "6320": 7008514048.0,
+ "6325": 7008514048.0,
+ "6330": 7008514048.0,
+ "6335": 7008514048.0,
+ "6340": 7008514048.0,
+ "6345": 7008514048.0,
+ "6350": 7008514048.0,
+ "6355": 7008514048.0,
+ "6360": 7008514048.0,
+ "6365": 7008514048.0,
+ "6370": 7008514048.0,
+ "6375": 7008514048.0,
+ "6380": 7008514048.0,
+ "6385": 7008514048.0,
+ "6390": 7008514048.0,
+ "6395": 7008514048.0,
+ "6400": 7008514048.0,
+ "6405": 7008514048.0,
+ "6410": 7008514048.0,
+ "6415": 7008514048.0,
+ "6420": 7008514048.0,
+ "6425": 7008514048.0,
+ "6430": 7008514048.0,
+ "6435": 7008514048.0,
+ "6440": 7008514048.0,
+ "6445": 7008514048.0,
+ "6450": 7008514048.0,
+ "6455": 7008514048.0,
+ "6460": 7008514048.0,
+ "6465": 7008514048.0,
+ "6470": 7008514048.0,
+ "6475": 7008514048.0,
+ "6480": 7008514048.0,
+ "6485": 7008514048.0,
+ "6490": 7008514048.0,
+ "6495": 7008514048.0,
+ "6500": 7008514048.0,
+ "6505": 7008514048.0,
+ "6510": 7008514048.0,
+ "6515": 7008514048.0,
+ "6520": 7008514048.0,
+ "6525": 7008514048.0,
+ "6530": 7008514048.0,
+ "6535": 7008514048.0,
+ "6540": 7008514048.0,
+ "6545": 7008514048.0,
+ "6550": 7008514048.0,
+ "6555": 7008514048.0,
+ "6560": 7008514048.0,
+ "6565": 7008514048.0,
+ "6570": 7008514048.0,
+ "6575": 7008514048.0,
+ "6580": 7008514048.0,
+ "6585": 7008514048.0,
+ "6590": 7008514048.0,
+ "6595": 7008514048.0,
+ "6600": 7008514048.0,
+ "6605": 7008514048.0,
+ "6610": 7008514048.0,
+ "6615": 7008514048.0,
+ "6620": 7008514048.0,
+ "6625": 7008514048.0,
+ "6630": 7008514048.0,
+ "6635": 7008514048.0,
+ "6640": 7008514048.0,
+ "6645": 7008514048.0,
+ "6650": 7008514048.0,
+ "6655": 7008514048.0,
+ "6660": 7008514048.0,
+ "6665": 7008514048.0,
+ "6670": 7008514048.0,
+ "6675": 7008514048.0,
+ "6680": 7008514048.0,
+ "6685": 7008514048.0,
+ "6690": 7008514048.0,
+ "6695": 7008514048.0,
+ "6700": 7008514048.0,
+ "6705": 7008514048.0,
+ "6710": 7008514048.0,
+ "6715": 7008514048.0,
+ "6720": 7008514048.0,
+ "6725": 7008514048.0,
+ "6730": 7008514048.0,
+ "6735": 7008514048.0,
+ "6740": 7008514048.0,
+ "6745": 7008514048.0,
+ "6750": 7008514048.0,
+ "6755": 7008514048.0,
+ "6760": 7008514048.0,
+ "6765": 7008514048.0,
+ "6770": 7008514048.0,
+ "6775": 7008514048.0,
+ "6780": 7008514048.0,
+ "6785": 7008514048.0,
+ "6790": 7008514048.0,
+ "6795": 7008514048.0,
+ "6800": 7008514048.0,
+ "6805": 7008514048.0,
+ "6810": 7008514048.0,
+ "6815": 7008514048.0,
+ "6820": 7008514048.0,
+ "6825": 7008514048.0,
+ "6830": 7008514048.0,
+ "6835": 7008514048.0,
+ "6840": 7008514048.0,
+ "6845": 7008514048.0,
+ "6850": 7008514048.0,
+ "6855": 7008514048.0,
+ "6860": 7008514048.0,
+ "6865": 7008514048.0,
+ "6870": 7008514048.0,
+ "6875": 7008514048.0,
+ "6880": 7008514048.0,
+ "6885": 7008514048.0,
+ "6890": 7008514048.0,
+ "6895": 7008514048.0,
+ "6900": 7008514048.0,
+ "6905": 7008514048.0,
+ "6910": 7008514048.0,
+ "6915": 7008514048.0,
+ "6920": 7008514048.0,
+ "6925": 7008514048.0,
+ "6930": 7008514048.0,
+ "6935": 7008514048.0,
+ "6940": 7008514048.0,
+ "6945": 7008514048.0,
+ "6950": 7008514048.0,
+ "6955": 7008514048.0,
+ "6960": 7008514048.0,
+ "6965": 7008514048.0,
+ "6970": 7008514048.0,
+ "6975": 7008514048.0,
+ "6980": 7008514048.0,
+ "6985": 7008514048.0,
+ "6990": 7008514048.0,
+ "6995": 7008514048.0,
+ "7000": 7008514048.0,
+ "7005": 7008514048.0,
+ "7010": 7008514048.0,
+ "7015": 7008514048.0,
+ "7020": 7008514048.0,
+ "7025": 7008514048.0,
+ "7030": 7008514048.0,
+ "7035": 7008514048.0,
+ "7040": 7008514048.0,
+ "7045": 7008514048.0,
+ "7050": 7008514048.0,
+ "7055": 7008514048.0,
+ "7060": 7008514048.0,
+ "7065": 7008514048.0,
+ "7070": 7008514048.0,
+ "7075": 7008514048.0,
+ "7080": 7008514048.0,
+ "7085": 7008514048.0,
+ "7090": 7008514048.0,
+ "7095": 7008514048.0,
+ "7100": 7008514048.0,
+ "7105": 7008514048.0,
+ "7110": 7008514048.0,
+ "7115": 7008514048.0,
+ "7120": 7008514048.0,
+ "7125": 7008514048.0,
+ "7130": 7008514048.0,
+ "7135": 7008514048.0,
+ "7140": 7008514048.0,
+ "7145": 7008514048.0,
+ "7150": 7008514048.0,
+ "7155": 7008514048.0,
+ "7160": 7008514048.0,
+ "7165": 7008514048.0,
+ "7170": 7008514048.0,
+ "7175": 7008514048.0,
+ "7180": 7008514048.0,
+ "7185": 7008514048.0,
+ "7190": 7008514048.0,
+ "7195": 7008514048.0,
+ "7200": 7008514048.0,
+ "7205": 7008514048.0,
+ "7210": 7008514048.0,
+ "7215": 7008514048.0,
+ "7220": 7008514048.0,
+ "7225": 7008514048.0,
+ "7230": 7008514048.0,
+ "7235": 7008514048.0,
+ "7240": 7008514048.0,
+ "7245": 7008514048.0,
+ "7250": 7008514048.0,
+ "7255": 7008514048.0,
+ "7260": 7008514048.0,
+ "7265": 7008514048.0,
+ "7270": 7008514048.0,
+ "7275": 7008514048.0,
+ "7280": 7008514048.0,
+ "7285": 7008514048.0,
+ "7290": 7008514048.0,
+ "7295": 7008514048.0,
+ "7300": 7008514048.0,
+ "7305": 7008514048.0,
+ "7310": 7008514048.0,
+ "7315": 7008514048.0,
+ "7320": 7008514048.0,
+ "7325": 7008514048.0,
+ "7330": 7008514048.0,
+ "7335": 7008514048.0,
+ "7340": 7008514048.0,
+ "7345": 7008514048.0,
+ "7350": 7008514048.0,
+ "7355": 7008514048.0,
+ "7360": 7008514048.0,
+ "7365": 7008514048.0,
+ "7370": 7008514048.0,
+ "7375": 7008514048.0,
+ "7380": 7008514048.0,
+ "7385": 7008514048.0,
+ "7390": 7008514048.0,
+ "7395": 7008514048.0,
+ "7400": 7008514048.0,
+ "7405": 7008514048.0,
+ "7410": 7008514048.0,
+ "7415": 7008514048.0,
+ "7420": 7008514048.0,
+ "7425": 7008514048.0,
+ "7430": 7008514048.0,
+ "7435": 7008514048.0,
+ "7440": 7008514048.0,
+ "7445": 7008514048.0,
+ "7450": 7008514048.0,
+ "7455": 7008514048.0,
+ "7460": 7008514048.0,
+ "7465": 7008514048.0,
+ "7470": 7008514048.0,
+ "7475": 7008514048.0,
+ "7480": 7008514048.0,
+ "7485": 7008514048.0,
+ "7490": 7008514048.0,
+ "7495": 7008514048.0,
+ "7500": 7008514048.0,
+ "7505": 7008514048.0,
+ "7510": 7008514048.0,
+ "7515": 7008514048.0,
+ "7520": 7008514048.0,
+ "7525": 7008514048.0,
+ "7530": 7008514048.0,
+ "7535": 7008514048.0,
+ "7540": 7008514048.0,
+ "7545": 7008514048.0,
+ "7550": 7008514048.0,
+ "7555": 7008514048.0,
+ "7560": 7008514048.0,
+ "7565": 7008514048.0,
+ "7570": 7008514048.0,
+ "7575": 7008514048.0,
+ "7580": 7008514048.0,
+ "7585": 7008514048.0,
+ "7590": 7008514048.0,
+ "7595": 7008514048.0,
+ "7600": 7008514048.0,
+ "7605": 7008514048.0,
+ "7610": 7008514048.0,
+ "7615": 7008514048.0,
+ "7620": 7008514048.0,
+ "7625": 7008514048.0,
+ "7630": 7008514048.0,
+ "7635": 7008514048.0,
+ "7640": 7008514048.0,
+ "7645": 7008514048.0,
+ "7650": 7008514048.0,
+ "7655": 7008514048.0,
+ "7660": 7008514048.0,
+ "7665": 7008514048.0,
+ "7670": 7008514048.0,
+ "7675": 7008514048.0,
+ "7680": 7008514048.0,
+ "7685": 7008514048.0,
+ "7690": 7008514048.0,
+ "7695": 7008514048.0,
+ "7700": 7008514048.0,
+ "7705": 7008514048.0,
+ "7710": 7008514048.0,
+ "7715": 7008514048.0,
+ "7720": 7008514048.0,
+ "7725": 7008514048.0,
+ "7730": 7008514048.0,
+ "7735": 7008514048.0,
+ "7740": 7008514048.0,
+ "7745": 7008514048.0,
+ "7750": 7008514048.0,
+ "7755": 7008514048.0,
+ "7760": 7008514048.0,
+ "7765": 7008514048.0,
+ "7770": 7008514048.0,
+ "7775": 7008514048.0,
+ "7780": 7008514048.0,
+ "7785": 7008514048.0,
+ "7790": 7008514048.0,
+ "7795": 7008514048.0,
+ "7800": 7008514048.0,
+ "7805": 7008514048.0,
+ "7810": 7008514048.0,
+ "7815": 7008514048.0,
+ "7820": 7008514048.0,
+ "7825": 7008514048.0,
+ "7830": 7008514048.0,
+ "7835": 7008514048.0,
+ "7840": 7008514048.0,
+ "7845": 7008514048.0,
+ "7850": 7008514048.0,
+ "7855": 7008514048.0,
+ "7860": 7008514048.0,
+ "7865": 7008514048.0,
+ "7870": 7008514048.0,
+ "7875": 7008514048.0,
+ "7880": 7008514048.0,
+ "7885": 7008514048.0,
+ "7890": 7008514048.0,
+ "7895": 7008514048.0,
+ "7900": 7008514048.0,
+ "7905": 7008514048.0,
+ "7910": 7008514048.0,
+ "7915": 7008514048.0,
+ "7920": 7008514048.0,
+ "7925": 7008514048.0,
+ "7930": 7008514048.0,
+ "7935": 7008514048.0,
+ "7940": 7008514048.0,
+ "7945": 7008514048.0,
+ "7950": 7008514048.0,
+ "7955": 7008514048.0,
+ "7960": 7008514048.0,
+ "7965": 7008514048.0,
+ "7970": 7008514048.0,
+ "7975": 7008514048.0,
+ "7980": 7008514048.0,
+ "7985": 7008514048.0,
+ "7990": 7008514048.0,
+ "7995": 7008514048.0,
+ "8000": 7008514048.0,
+ "8005": 7008514048.0,
+ "8010": 7008514048.0,
+ "8015": 7008514048.0,
+ "8020": 7008514048.0,
+ "8025": 7008514048.0,
+ "8030": 7008514048.0,
+ "8035": 7008514048.0,
+ "8040": 7008514048.0,
+ "8045": 7008514048.0,
+ "8050": 7008514048.0,
+ "8055": 7008514048.0,
+ "8060": 7008514048.0,
+ "8065": 7008514048.0,
+ "8070": 7008514048.0,
+ "8075": 7008514048.0,
+ "8080": 7008514048.0,
+ "8085": 7008514048.0,
+ "8090": 7008514048.0,
+ "8095": 7008514048.0,
+ "8100": 7008514048.0,
+ "8105": 7008514048.0,
+ "8110": 7008514048.0,
+ "8115": 7008514048.0,
+ "8120": 7008514048.0,
+ "8125": 7008514048.0,
+ "8130": 7008514048.0,
+ "8135": 7008514048.0,
+ "8140": 7008514048.0,
+ "8145": 7008514048.0,
+ "8150": 7008514048.0,
+ "8155": 7008514048.0,
+ "8160": 7008514048.0,
+ "8165": 7008514048.0,
+ "8170": 7008514048.0,
+ "8175": 7008514048.0,
+ "8180": 7008514048.0,
+ "8185": 7008514048.0,
+ "8190": 7008514048.0,
+ "8195": 7008514048.0,
+ "8200": 7008514048.0,
+ "8205": 7008514048.0,
+ "8210": 7008514048.0,
+ "8215": 7008514048.0,
+ "8220": 7008514048.0,
+ "8225": 7008514048.0,
+ "8230": 7008514048.0,
+ "8235": 7008514048.0,
+ "8240": 7008514048.0,
+ "8245": 7008514048.0,
+ "8250": 7008514048.0,
+ "8255": 7008514048.0,
+ "8260": 7008514048.0,
+ "8265": 7008514048.0,
+ "8270": 7008514048.0,
+ "8275": 7008514048.0,
+ "8280": 7008514048.0,
+ "8285": 7008514048.0,
+ "8290": 7008514048.0,
+ "8295": 7008514048.0,
+ "8300": 7008514048.0,
+ "8305": 7008514048.0,
+ "8310": 7008514048.0,
+ "8315": 7008514048.0,
+ "8320": 7008514048.0,
+ "8325": 7008514048.0,
+ "8330": 7008514048.0,
+ "8335": 7008514048.0,
+ "8340": 7008514048.0,
+ "8345": 7008514048.0,
+ "8350": 7008514048.0,
+ "8355": 7008514048.0,
+ "8360": 7008514048.0,
+ "8365": 7008514048.0,
+ "8370": 7008514048.0,
+ "8375": 7008514048.0,
+ "8380": 7008514048.0,
+ "8385": 7008514048.0,
+ "8390": 7008514048.0,
+ "8395": 7008514048.0,
+ "8400": 7008514048.0,
+ "8405": 7008514048.0,
+ "8410": 7008514048.0,
+ "8415": 7008514048.0,
+ "8420": 7008514048.0,
+ "8425": 7008514048.0,
+ "8430": 7008514048.0,
+ "8435": 7008514048.0,
+ "8440": 7008514048.0,
+ "8445": 7008514048.0,
+ "8450": 7008514048.0,
+ "8455": 7008514048.0,
+ "8460": 7008514048.0,
+ "8465": 7008514048.0,
+ "8470": 7008514048.0,
+ "8475": 7008514048.0,
+ "8480": 7008514048.0,
+ "8485": 7008514048.0,
+ "8490": 7008514048.0,
+ "8495": 7008514048.0,
+ "8500": 7008514048.0,
+ "8505": 7008514048.0,
+ "8510": 7008514048.0,
+ "8515": 7008514048.0,
+ "8520": 7008514048.0,
+ "8525": 7008514048.0,
+ "8530": 7008514048.0,
+ "8535": 7008514048.0,
+ "8540": 7008514048.0,
+ "8545": 7008514048.0,
+ "8550": 7008514048.0,
+ "8555": 7008514048.0,
+ "8560": 7008514048.0,
+ "8565": 7008514048.0,
+ "8570": 7008514048.0,
+ "8575": 7008514048.0,
+ "8580": 7008514048.0,
+ "8585": 7008514048.0,
+ "8590": 7008514048.0,
+ "8595": 7008514048.0,
+ "8600": 7008514048.0,
+ "8605": 7008514048.0,
+ "8610": 7008514048.0,
+ "8615": 7008514048.0,
+ "8620": 7008514048.0,
+ "8625": 7008514048.0,
+ "8630": 7008514048.0,
+ "8635": 7008514048.0,
+ "8640": 7008514048.0,
+ "8645": 7008514048.0,
+ "8650": 7008514048.0,
+ "8655": 7008514048.0,
+ "8660": 7008514048.0,
+ "8665": 7008514048.0,
+ "8670": 7008514048.0,
+ "8675": 7008514048.0,
+ "8680": 7008514048.0,
+ "8685": 7008514048.0,
+ "8690": 7008514048.0,
+ "8695": 7008514048.0,
+ "8700": 7008514048.0,
+ "8705": 7008514048.0,
+ "8710": 7008514048.0,
+ "8715": 7008514048.0,
+ "8720": 7008514048.0,
+ "8725": 7008514048.0,
+ "8730": 7008514048.0,
+ "8735": 7008514048.0,
+ "8740": 7008514048.0,
+ "8745": 7008514048.0,
+ "8750": 7008514048.0,
+ "8755": 7008514048.0,
+ "8760": 7008514048.0,
+ "8765": 7008514048.0,
+ "8770": 7008514048.0,
+ "8775": 7008514048.0,
+ "8780": 7008514048.0,
+ "8785": 7008514048.0,
+ "8790": 7008514048.0,
+ "8795": 7008514048.0,
+ "8800": 7008514048.0,
+ "8805": 7008514048.0,
+ "8810": 7008514048.0,
+ "8815": 7008514048.0,
+ "8820": 7008514048.0,
+ "8825": 7008514048.0,
+ "8830": 7008514048.0,
+ "8835": 7008514048.0,
+ "8840": 7008514048.0,
+ "8845": 7008514048.0,
+ "8850": 7008514048.0,
+ "8855": 7008514048.0,
+ "8860": 7008514048.0,
+ "8865": 7008514048.0,
+ "8870": 7008514048.0,
+ "8875": 7008514048.0,
+ "8880": 7008514048.0,
+ "8885": 7008514048.0,
+ "8890": 7008514048.0,
+ "8895": 7008514048.0,
+ "8900": 7008514048.0,
+ "8905": 7008514048.0,
+ "8910": 7008514048.0,
+ "8915": 7008514048.0,
+ "8920": 7008514048.0,
+ "8925": 7008514048.0,
+ "8930": 7008514048.0,
+ "8935": 7008514048.0,
+ "8940": 7008514048.0,
+ "8945": 7008514048.0,
+ "8950": 7008514048.0,
+ "8955": 7008514048.0,
+ "8960": 7008514048.0,
+ "8965": 7008514048.0,
+ "8970": 7008514048.0,
+ "8975": 7008514048.0,
+ "8980": 7008514048.0,
+ "8985": 7008514048.0,
+ "8990": 7008514048.0,
+ "8995": 7008514048.0,
+ "9000": 7008514048.0,
+ "9005": 7008514048.0,
+ "9010": 7008514048.0,
+ "9015": 7008514048.0,
+ "9020": 7008514048.0,
+ "9025": 7008514048.0,
+ "9030": 7008514048.0,
+ "9035": 7008514048.0,
+ "9040": 7008514048.0,
+ "9045": 7008514048.0,
+ "9050": 7008514048.0,
+ "9055": 7008514048.0,
+ "9060": 7008514048.0,
+ "9065": 7008514048.0,
+ "9070": 7008514048.0,
+ "9075": 7008514048.0,
+ "9080": 7008514048.0,
+ "9085": 7008514048.0,
+ "9090": 7008514048.0,
+ "9095": 7008514048.0,
+ "9100": 7008514048.0,
+ "9105": 7008514048.0,
+ "9110": 7008514048.0,
+ "9115": 7008514048.0,
+ "9120": 7008514048.0,
+ "9125": 7008514048.0,
+ "9130": 7008514048.0,
+ "9135": 7008514048.0,
+ "9140": 7008514048.0,
+ "9145": 7008514048.0,
+ "9150": 7008514048.0,
+ "9155": 7008514048.0,
+ "9160": 7008514048.0,
+ "9165": 7008514048.0,
+ "9170": 7008514048.0,
+ "9175": 7008514048.0,
+ "9180": 7008514048.0,
+ "9185": 7008514048.0,
+ "9190": 7008514048.0,
+ "9195": 7008514048.0,
+ "9200": 7008514048.0,
+ "9205": 7008514048.0,
+ "9210": 7008514048.0,
+ "9215": 7008514048.0,
+ "9220": 7008514048.0,
+ "9225": 7008514048.0,
+ "9230": 7008514048.0,
+ "9235": 7008514048.0,
+ "9240": 7008514048.0,
+ "9245": 7008514048.0,
+ "9250": 7008514048.0,
+ "9255": 7008514048.0,
+ "9260": 7008514048.0,
+ "9265": 7008514048.0,
+ "9270": 7008514048.0,
+ "9275": 7008514048.0,
+ "9280": 7008514048.0,
+ "9285": 7008514048.0,
+ "9290": 7008514048.0,
+ "9295": 7008514048.0,
+ "9300": 7008514048.0,
+ "9305": 7008514048.0,
+ "9310": 7008514048.0,
+ "9315": 7008514048.0,
+ "9320": 7008514048.0,
+ "9325": 7008514048.0,
+ "9330": 7008514048.0,
+ "9335": 7008514048.0,
+ "9340": 7008514048.0,
+ "9345": 7008514048.0,
+ "9350": 7008514048.0,
+ "9355": 7008514048.0,
+ "9360": 7008514048.0,
+ "9365": 7008514048.0,
+ "9370": 7008514048.0,
+ "9375": 7008514048.0,
+ "9380": 7008514048.0,
+ "9385": 7008514048.0,
+ "9390": 7008514048.0,
+ "9395": 7008514048.0,
+ "9400": 7008514048.0,
+ "9405": 7008514048.0,
+ "9410": 7008514048.0,
+ "9415": 7008514048.0,
+ "9420": 7008514048.0,
+ "9425": 7008514048.0,
+ "9430": 7008514048.0,
+ "9435": 7008514048.0,
+ "9440": 7008514048.0,
+ "9445": 7008514048.0,
+ "9450": 7008514048.0,
+ "9455": 7008514048.0,
+ "9460": 7008514048.0,
+ "9465": 7008514048.0,
+ "9470": 7008514048.0,
+ "9475": 7008514048.0,
+ "9480": 7008514048.0,
+ "9485": 7008514048.0,
+ "9490": 7008514048.0,
+ "9495": 7008514048.0,
+ "9500": 7008514048.0,
+ "9505": 7008514048.0,
+ "9510": 7008514048.0,
+ "9515": 7008514048.0,
+ "9520": 7008514048.0,
+ "9525": 7008514048.0,
+ "9530": 7008514048.0,
+ "9535": 7008514048.0,
+ "9540": 7008514048.0,
+ "9545": 7008514048.0,
+ "9550": 7008514048.0,
+ "9555": 7008514048.0,
+ "9560": 7008514048.0,
+ "9565": 7008514048.0,
+ "9570": 7008514048.0,
+ "9575": 7008514048.0,
+ "9580": 7008514048.0,
+ "9585": 7008514048.0,
+ "9590": 7008514048.0,
+ "9595": 7008514048.0,
+ "9600": 7008514048.0,
+ "9605": 7008514048.0,
+ "9610": 7008514048.0,
+ "9615": 7008514048.0,
+ "9620": 7008514048.0,
+ "9625": 7008514048.0,
+ "9630": 7008514048.0,
+ "9635": 7008514048.0,
+ "9640": 7008514048.0,
+ "9645": 7008514048.0,
+ "9650": 7008514048.0,
+ "9655": 7008514048.0,
+ "9660": 7008514048.0,
+ "9665": 7008514048.0,
+ "9670": 7008514048.0,
+ "9675": 7008514048.0,
+ "9680": 7008514048.0,
+ "9685": 7008514048.0,
+ "9690": 7008514048.0,
+ "9695": 7008514048.0,
+ "9700": 7008514048.0,
+ "9705": 7008514048.0,
+ "9710": 7008514048.0,
+ "9715": 7008514048.0,
+ "9720": 7008514048.0,
+ "9725": 7008514048.0,
+ "9730": 7008514048.0,
+ "9735": 7008514048.0,
+ "9740": 7008514048.0,
+ "9745": 7008514048.0,
+ "9750": 7008514048.0,
+ "9755": 7008514048.0,
+ "9760": 7008514048.0,
+ "9765": 7008514048.0,
+ "9770": 7008514048.0,
+ "9775": 7008514048.0,
+ "9780": 7008514048.0,
+ "9785": 7008514048.0,
+ "9790": 7008514048.0,
+ "9795": 7008514048.0,
+ "9800": 7008514048.0,
+ "9805": 7008514048.0,
+ "9810": 7008514048.0,
+ "9815": 7008514048.0,
+ "9820": 7008514048.0,
+ "9825": 7008514048.0,
+ "9830": 7008514048.0,
+ "9835": 7008514048.0,
+ "9840": 7008514048.0,
+ "9845": 7008514048.0,
+ "9850": 7008514048.0,
+ "9855": 7008514048.0,
+ "9860": 7008514048.0,
+ "9865": 7008514048.0,
+ "9870": 7008514048.0,
+ "9875": 7008514048.0,
+ "9880": 7008514048.0,
+ "9885": 7008514048.0,
+ "9890": 7008514048.0,
+ "9895": 7008514048.0,
+ "9900": 7008514048.0,
+ "9905": 7008514048.0,
+ "9910": 7008514048.0,
+ "9915": 7008514048.0,
+ "9920": 7008514048.0,
+ "9925": 7008514048.0,
+ "9930": 7008514048.0,
+ "9935": 7008514048.0,
+ "9940": 7008514048.0,
+ "9945": 7008514048.0,
+ "9950": 7008514048.0,
+ "9955": 7008514048.0,
+ "9960": 7008514048.0,
+ "9965": 7008514048.0,
+ "9970": 7008514048.0,
+ "9975": 7008514048.0,
+ "9980": 7008514048.0,
+ "9985": 7008514048.0,
+ "9990": 7008514048.0,
+ "9995": 7008514048.0,
+ "10000": 7008514048.0
+ }
+ },
+ "iteration-time": {
+ "start_step": 1,
+ "end_step": 10000,
+ "step_interval": 5,
+ "values": {
+ "1": "nan",
+ "5": "nan",
+ "10": "nan",
+ "15": "nan",
+ "20": "nan",
+ "25": "nan",
+ "30": "nan",
+ "35": "nan",
+ "40": "nan",
+ "45": "nan",
+ "50": "nan",
+ "55": "nan",
+ "60": "nan",
+ "65": "nan",
+ "70": "nan",
+ "75": "nan",
+ "80": "nan",
+ "85": "nan",
+ "90": "nan",
+ "95": "nan",
+ "100": 0.26331,
+ "105": "nan",
+ "110": "nan",
+ "115": "nan",
+ "120": "nan",
+ "125": "nan",
+ "130": "nan",
+ "135": "nan",
+ "140": "nan",
+ "145": "nan",
+ "150": "nan",
+ "155": "nan",
+ "160": "nan",
+ "165": "nan",
+ "170": "nan",
+ "175": "nan",
+ "180": "nan",
+ "185": "nan",
+ "190": "nan",
+ "195": "nan",
+ "200": 0.19746,
+ "205": "nan",
+ "210": "nan",
+ "215": "nan",
+ "220": "nan",
+ "225": "nan",
+ "230": "nan",
+ "235": "nan",
+ "240": "nan",
+ "245": "nan",
+ "250": "nan",
+ "255": "nan",
+ "260": "nan",
+ "265": "nan",
+ "270": "nan",
+ "275": "nan",
+ "280": "nan",
+ "285": "nan",
+ "290": "nan",
+ "295": "nan",
+ "300": 0.26945,
+ "305": "nan",
+ "310": "nan",
+ "315": "nan",
+ "320": "nan",
+ "325": "nan",
+ "330": "nan",
+ "335": "nan",
+ "340": "nan",
+ "345": "nan",
+ "350": "nan",
+ "355": "nan",
+ "360": "nan",
+ "365": "nan",
+ "370": "nan",
+ "375": "nan",
+ "380": "nan",
+ "385": "nan",
+ "390": "nan",
+ "395": "nan",
+ "400": 0.1767,
+ "405": "nan",
+ "410": "nan",
+ "415": "nan",
+ "420": "nan",
+ "425": "nan",
+ "430": "nan",
+ "435": "nan",
+ "440": "nan",
+ "445": "nan",
+ "450": "nan",
+ "455": "nan",
+ "460": "nan",
+ "465": "nan",
+ "470": "nan",
+ "475": "nan",
+ "480": "nan",
+ "485": "nan",
+ "490": "nan",
+ "495": "nan",
+ "500": 0.16436,
+ "505": "nan",
+ "510": "nan",
+ "515": "nan",
+ "520": "nan",
+ "525": "nan",
+ "530": "nan",
+ "535": "nan",
+ "540": "nan",
+ "545": "nan",
+ "550": "nan",
+ "555": "nan",
+ "560": "nan",
+ "565": "nan",
+ "570": "nan",
+ "575": "nan",
+ "580": "nan",
+ "585": "nan",
+ "590": "nan",
+ "595": "nan",
+ "600": 0.16453,
+ "605": "nan",
+ "610": "nan",
+ "615": "nan",
+ "620": "nan",
+ "625": "nan",
+ "630": "nan",
+ "635": "nan",
+ "640": "nan",
+ "645": "nan",
+ "650": "nan",
+ "655": "nan",
+ "660": "nan",
+ "665": "nan",
+ "670": "nan",
+ "675": "nan",
+ "680": "nan",
+ "685": "nan",
+ "690": "nan",
+ "695": "nan",
+ "700": 0.19407,
+ "705": "nan",
+ "710": "nan",
+ "715": "nan",
+ "720": "nan",
+ "725": "nan",
+ "730": "nan",
+ "735": "nan",
+ "740": "nan",
+ "745": "nan",
+ "750": "nan",
+ "755": "nan",
+ "760": "nan",
+ "765": "nan",
+ "770": "nan",
+ "775": "nan",
+ "780": "nan",
+ "785": "nan",
+ "790": "nan",
+ "795": "nan",
+ "800": 0.23124,
+ "805": "nan",
+ "810": "nan",
+ "815": "nan",
+ "820": "nan",
+ "825": "nan",
+ "830": "nan",
+ "835": "nan",
+ "840": "nan",
+ "845": "nan",
+ "850": "nan",
+ "855": "nan",
+ "860": "nan",
+ "865": "nan",
+ "870": "nan",
+ "875": "nan",
+ "880": "nan",
+ "885": "nan",
+ "890": "nan",
+ "895": "nan",
+ "900": 0.19614,
+ "905": "nan",
+ "910": "nan",
+ "915": "nan",
+ "920": "nan",
+ "925": "nan",
+ "930": "nan",
+ "935": "nan",
+ "940": "nan",
+ "945": "nan",
+ "950": "nan",
+ "955": "nan",
+ "960": "nan",
+ "965": "nan",
+ "970": "nan",
+ "975": "nan",
+ "980": "nan",
+ "985": "nan",
+ "990": "nan",
+ "995": "nan",
+ "1000": 0.16586,
+ "1005": "nan",
+ "1010": "nan",
+ "1015": "nan",
+ "1020": "nan",
+ "1025": "nan",
+ "1030": "nan",
+ "1035": "nan",
+ "1040": "nan",
+ "1045": "nan",
+ "1050": "nan",
+ "1055": "nan",
+ "1060": "nan",
+ "1065": "nan",
+ "1070": "nan",
+ "1075": "nan",
+ "1080": "nan",
+ "1085": "nan",
+ "1090": "nan",
+ "1095": "nan",
+ "1100": 0.16901,
+ "1105": "nan",
+ "1110": "nan",
+ "1115": "nan",
+ "1120": "nan",
+ "1125": "nan",
+ "1130": "nan",
+ "1135": "nan",
+ "1140": "nan",
+ "1145": "nan",
+ "1150": "nan",
+ "1155": "nan",
+ "1160": "nan",
+ "1165": "nan",
+ "1170": "nan",
+ "1175": "nan",
+ "1180": "nan",
+ "1185": "nan",
+ "1190": "nan",
+ "1195": "nan",
+ "1200": 0.21643,
+ "1205": "nan",
+ "1210": "nan",
+ "1215": "nan",
+ "1220": "nan",
+ "1225": "nan",
+ "1230": "nan",
+ "1235": "nan",
+ "1240": "nan",
+ "1245": "nan",
+ "1250": "nan",
+ "1255": "nan",
+ "1260": "nan",
+ "1265": "nan",
+ "1270": "nan",
+ "1275": "nan",
+ "1280": "nan",
+ "1285": "nan",
+ "1290": "nan",
+ "1295": "nan",
+ "1300": 0.19756,
+ "1305": "nan",
+ "1310": "nan",
+ "1315": "nan",
+ "1320": "nan",
+ "1325": "nan",
+ "1330": "nan",
+ "1335": "nan",
+ "1340": "nan",
+ "1345": "nan",
+ "1350": "nan",
+ "1355": "nan",
+ "1360": "nan",
+ "1365": "nan",
+ "1370": "nan",
+ "1375": "nan",
+ "1380": "nan",
+ "1385": "nan",
+ "1390": "nan",
+ "1395": "nan",
+ "1400": 0.19925,
+ "1405": "nan",
+ "1410": "nan",
+ "1415": "nan",
+ "1420": "nan",
+ "1425": "nan",
+ "1430": "nan",
+ "1435": "nan",
+ "1440": "nan",
+ "1445": "nan",
+ "1450": "nan",
+ "1455": "nan",
+ "1460": "nan",
+ "1465": "nan",
+ "1470": "nan",
+ "1475": "nan",
+ "1480": "nan",
+ "1485": "nan",
+ "1490": "nan",
+ "1495": "nan",
+ "1500": 0.16554,
+ "1505": "nan",
+ "1510": "nan",
+ "1515": "nan",
+ "1520": "nan",
+ "1525": "nan",
+ "1530": "nan",
+ "1535": "nan",
+ "1540": "nan",
+ "1545": "nan",
+ "1550": "nan",
+ "1555": "nan",
+ "1560": "nan",
+ "1565": "nan",
+ "1570": "nan",
+ "1575": "nan",
+ "1580": "nan",
+ "1585": "nan",
+ "1590": "nan",
+ "1595": "nan",
+ "1600": 0.17563,
+ "1605": "nan",
+ "1610": "nan",
+ "1615": "nan",
+ "1620": "nan",
+ "1625": "nan",
+ "1630": "nan",
+ "1635": "nan",
+ "1640": "nan",
+ "1645": "nan",
+ "1650": "nan",
+ "1655": "nan",
+ "1660": "nan",
+ "1665": "nan",
+ "1670": "nan",
+ "1675": "nan",
+ "1680": "nan",
+ "1685": "nan",
+ "1690": "nan",
+ "1695": "nan",
+ "1700": 0.2172,
+ "1705": "nan",
+ "1710": "nan",
+ "1715": "nan",
+ "1720": "nan",
+ "1725": "nan",
+ "1730": "nan",
+ "1735": "nan",
+ "1740": "nan",
+ "1745": "nan",
+ "1750": "nan",
+ "1755": "nan",
+ "1760": "nan",
+ "1765": "nan",
+ "1770": "nan",
+ "1775": "nan",
+ "1780": "nan",
+ "1785": "nan",
+ "1790": "nan",
+ "1795": "nan",
+ "1800": 0.17392,
+ "1805": "nan",
+ "1810": "nan",
+ "1815": "nan",
+ "1820": "nan",
+ "1825": "nan",
+ "1830": "nan",
+ "1835": "nan",
+ "1840": "nan",
+ "1845": "nan",
+ "1850": "nan",
+ "1855": "nan",
+ "1860": "nan",
+ "1865": "nan",
+ "1870": "nan",
+ "1875": "nan",
+ "1880": "nan",
+ "1885": "nan",
+ "1890": "nan",
+ "1895": "nan",
+ "1900": 0.20554,
+ "1905": "nan",
+ "1910": "nan",
+ "1915": "nan",
+ "1920": "nan",
+ "1925": "nan",
+ "1930": "nan",
+ "1935": "nan",
+ "1940": "nan",
+ "1945": "nan",
+ "1950": "nan",
+ "1955": "nan",
+ "1960": "nan",
+ "1965": "nan",
+ "1970": "nan",
+ "1975": "nan",
+ "1980": "nan",
+ "1985": "nan",
+ "1990": "nan",
+ "1995": "nan",
+ "2000": 0.18468,
+ "2005": "nan",
+ "2010": "nan",
+ "2015": "nan",
+ "2020": "nan",
+ "2025": "nan",
+ "2030": "nan",
+ "2035": "nan",
+ "2040": "nan",
+ "2045": "nan",
+ "2050": "nan",
+ "2055": "nan",
+ "2060": "nan",
+ "2065": "nan",
+ "2070": "nan",
+ "2075": "nan",
+ "2080": "nan",
+ "2085": "nan",
+ "2090": "nan",
+ "2095": "nan",
+ "2100": 0.20159,
+ "2105": "nan",
+ "2110": "nan",
+ "2115": "nan",
+ "2120": "nan",
+ "2125": "nan",
+ "2130": "nan",
+ "2135": "nan",
+ "2140": "nan",
+ "2145": "nan",
+ "2150": "nan",
+ "2155": "nan",
+ "2160": "nan",
+ "2165": "nan",
+ "2170": "nan",
+ "2175": "nan",
+ "2180": "nan",
+ "2185": "nan",
+ "2190": "nan",
+ "2195": "nan",
+ "2200": 0.18773,
+ "2205": "nan",
+ "2210": "nan",
+ "2215": "nan",
+ "2220": "nan",
+ "2225": "nan",
+ "2230": "nan",
+ "2235": "nan",
+ "2240": "nan",
+ "2245": "nan",
+ "2250": "nan",
+ "2255": "nan",
+ "2260": "nan",
+ "2265": "nan",
+ "2270": "nan",
+ "2275": "nan",
+ "2280": "nan",
+ "2285": "nan",
+ "2290": "nan",
+ "2295": "nan",
+ "2300": 0.17279,
+ "2305": "nan",
+ "2310": "nan",
+ "2315": "nan",
+ "2320": "nan",
+ "2325": "nan",
+ "2330": "nan",
+ "2335": "nan",
+ "2340": "nan",
+ "2345": "nan",
+ "2350": "nan",
+ "2355": "nan",
+ "2360": "nan",
+ "2365": "nan",
+ "2370": "nan",
+ "2375": "nan",
+ "2380": "nan",
+ "2385": "nan",
+ "2390": "nan",
+ "2395": "nan",
+ "2400": 0.16274,
+ "2405": "nan",
+ "2410": "nan",
+ "2415": "nan",
+ "2420": "nan",
+ "2425": "nan",
+ "2430": "nan",
+ "2435": "nan",
+ "2440": "nan",
+ "2445": "nan",
+ "2450": "nan",
+ "2455": "nan",
+ "2460": "nan",
+ "2465": "nan",
+ "2470": "nan",
+ "2475": "nan",
+ "2480": "nan",
+ "2485": "nan",
+ "2490": "nan",
+ "2495": "nan",
+ "2500": 0.1968,
+ "2505": "nan",
+ "2510": "nan",
+ "2515": "nan",
+ "2520": "nan",
+ "2525": "nan",
+ "2530": "nan",
+ "2535": "nan",
+ "2540": "nan",
+ "2545": "nan",
+ "2550": "nan",
+ "2555": "nan",
+ "2560": "nan",
+ "2565": "nan",
+ "2570": "nan",
+ "2575": "nan",
+ "2580": "nan",
+ "2585": "nan",
+ "2590": "nan",
+ "2595": "nan",
+ "2600": 0.18491,
+ "2605": "nan",
+ "2610": "nan",
+ "2615": "nan",
+ "2620": "nan",
+ "2625": "nan",
+ "2630": "nan",
+ "2635": "nan",
+ "2640": "nan",
+ "2645": "nan",
+ "2650": "nan",
+ "2655": "nan",
+ "2660": "nan",
+ "2665": "nan",
+ "2670": "nan",
+ "2675": "nan",
+ "2680": "nan",
+ "2685": "nan",
+ "2690": "nan",
+ "2695": "nan",
+ "2700": 0.21214,
+ "2705": "nan",
+ "2710": "nan",
+ "2715": "nan",
+ "2720": "nan",
+ "2725": "nan",
+ "2730": "nan",
+ "2735": "nan",
+ "2740": "nan",
+ "2745": "nan",
+ "2750": "nan",
+ "2755": "nan",
+ "2760": "nan",
+ "2765": "nan",
+ "2770": "nan",
+ "2775": "nan",
+ "2780": "nan",
+ "2785": "nan",
+ "2790": "nan",
+ "2795": "nan",
+ "2800": 0.19074,
+ "2805": "nan",
+ "2810": "nan",
+ "2815": "nan",
+ "2820": "nan",
+ "2825": "nan",
+ "2830": "nan",
+ "2835": "nan",
+ "2840": "nan",
+ "2845": "nan",
+ "2850": "nan",
+ "2855": "nan",
+ "2860": "nan",
+ "2865": "nan",
+ "2870": "nan",
+ "2875": "nan",
+ "2880": "nan",
+ "2885": "nan",
+ "2890": "nan",
+ "2895": "nan",
+ "2900": 0.16448,
+ "2905": "nan",
+ "2910": "nan",
+ "2915": "nan",
+ "2920": "nan",
+ "2925": "nan",
+ "2930": "nan",
+ "2935": "nan",
+ "2940": "nan",
+ "2945": "nan",
+ "2950": "nan",
+ "2955": "nan",
+ "2960": "nan",
+ "2965": "nan",
+ "2970": "nan",
+ "2975": "nan",
+ "2980": "nan",
+ "2985": "nan",
+ "2990": "nan",
+ "2995": "nan",
+ "3000": 0.2131,
+ "3005": "nan",
+ "3010": "nan",
+ "3015": "nan",
+ "3020": "nan",
+ "3025": "nan",
+ "3030": "nan",
+ "3035": "nan",
+ "3040": "nan",
+ "3045": "nan",
+ "3050": "nan",
+ "3055": "nan",
+ "3060": "nan",
+ "3065": "nan",
+ "3070": "nan",
+ "3075": "nan",
+ "3080": "nan",
+ "3085": "nan",
+ "3090": "nan",
+ "3095": "nan",
+ "3100": 0.16974,
+ "3105": "nan",
+ "3110": "nan",
+ "3115": "nan",
+ "3120": "nan",
+ "3125": "nan",
+ "3130": "nan",
+ "3135": "nan",
+ "3140": "nan",
+ "3145": "nan",
+ "3150": "nan",
+ "3155": "nan",
+ "3160": "nan",
+ "3165": "nan",
+ "3170": "nan",
+ "3175": "nan",
+ "3180": "nan",
+ "3185": "nan",
+ "3190": "nan",
+ "3195": "nan",
+ "3200": 0.19537,
+ "3205": "nan",
+ "3210": "nan",
+ "3215": "nan",
+ "3220": "nan",
+ "3225": "nan",
+ "3230": "nan",
+ "3235": "nan",
+ "3240": "nan",
+ "3245": "nan",
+ "3250": "nan",
+ "3255": "nan",
+ "3260": "nan",
+ "3265": "nan",
+ "3270": "nan",
+ "3275": "nan",
+ "3280": "nan",
+ "3285": "nan",
+ "3290": "nan",
+ "3295": "nan",
+ "3300": 0.20036,
+ "3305": "nan",
+ "3310": "nan",
+ "3315": "nan",
+ "3320": "nan",
+ "3325": "nan",
+ "3330": "nan",
+ "3335": "nan",
+ "3340": "nan",
+ "3345": "nan",
+ "3350": "nan",
+ "3355": "nan",
+ "3360": "nan",
+ "3365": "nan",
+ "3370": "nan",
+ "3375": "nan",
+ "3380": "nan",
+ "3385": "nan",
+ "3390": "nan",
+ "3395": "nan",
+ "3400": 0.18441,
+ "3405": "nan",
+ "3410": "nan",
+ "3415": "nan",
+ "3420": "nan",
+ "3425": "nan",
+ "3430": "nan",
+ "3435": "nan",
+ "3440": "nan",
+ "3445": "nan",
+ "3450": "nan",
+ "3455": "nan",
+ "3460": "nan",
+ "3465": "nan",
+ "3470": "nan",
+ "3475": "nan",
+ "3480": "nan",
+ "3485": "nan",
+ "3490": "nan",
+ "3495": "nan",
+ "3500": 0.20315,
+ "3505": "nan",
+ "3510": "nan",
+ "3515": "nan",
+ "3520": "nan",
+ "3525": "nan",
+ "3530": "nan",
+ "3535": "nan",
+ "3540": "nan",
+ "3545": "nan",
+ "3550": "nan",
+ "3555": "nan",
+ "3560": "nan",
+ "3565": "nan",
+ "3570": "nan",
+ "3575": "nan",
+ "3580": "nan",
+ "3585": "nan",
+ "3590": "nan",
+ "3595": "nan",
+ "3600": 0.17402,
+ "3605": "nan",
+ "3610": "nan",
+ "3615": "nan",
+ "3620": "nan",
+ "3625": "nan",
+ "3630": "nan",
+ "3635": "nan",
+ "3640": "nan",
+ "3645": "nan",
+ "3650": "nan",
+ "3655": "nan",
+ "3660": "nan",
+ "3665": "nan",
+ "3670": "nan",
+ "3675": "nan",
+ "3680": "nan",
+ "3685": "nan",
+ "3690": "nan",
+ "3695": "nan",
+ "3700": 0.17364,
+ "3705": "nan",
+ "3710": "nan",
+ "3715": "nan",
+ "3720": "nan",
+ "3725": "nan",
+ "3730": "nan",
+ "3735": "nan",
+ "3740": "nan",
+ "3745": "nan",
+ "3750": "nan",
+ "3755": "nan",
+ "3760": "nan",
+ "3765": "nan",
+ "3770": "nan",
+ "3775": "nan",
+ "3780": "nan",
+ "3785": "nan",
+ "3790": "nan",
+ "3795": "nan",
+ "3800": 0.2033,
+ "3805": "nan",
+ "3810": "nan",
+ "3815": "nan",
+ "3820": "nan",
+ "3825": "nan",
+ "3830": "nan",
+ "3835": "nan",
+ "3840": "nan",
+ "3845": "nan",
+ "3850": "nan",
+ "3855": "nan",
+ "3860": "nan",
+ "3865": "nan",
+ "3870": "nan",
+ "3875": "nan",
+ "3880": "nan",
+ "3885": "nan",
+ "3890": "nan",
+ "3895": "nan",
+ "3900": 0.22678,
+ "3905": "nan",
+ "3910": "nan",
+ "3915": "nan",
+ "3920": "nan",
+ "3925": "nan",
+ "3930": "nan",
+ "3935": "nan",
+ "3940": "nan",
+ "3945": "nan",
+ "3950": "nan",
+ "3955": "nan",
+ "3960": "nan",
+ "3965": "nan",
+ "3970": "nan",
+ "3975": "nan",
+ "3980": "nan",
+ "3985": "nan",
+ "3990": "nan",
+ "3995": "nan",
+ "4000": 0.19275,
+ "4005": "nan",
+ "4010": "nan",
+ "4015": "nan",
+ "4020": "nan",
+ "4025": "nan",
+ "4030": "nan",
+ "4035": "nan",
+ "4040": "nan",
+ "4045": "nan",
+ "4050": "nan",
+ "4055": "nan",
+ "4060": "nan",
+ "4065": "nan",
+ "4070": "nan",
+ "4075": "nan",
+ "4080": "nan",
+ "4085": "nan",
+ "4090": "nan",
+ "4095": "nan",
+ "4100": 0.23807,
+ "4105": "nan",
+ "4110": "nan",
+ "4115": "nan",
+ "4120": "nan",
+ "4125": "nan",
+ "4130": "nan",
+ "4135": "nan",
+ "4140": "nan",
+ "4145": "nan",
+ "4150": "nan",
+ "4155": "nan",
+ "4160": "nan",
+ "4165": "nan",
+ "4170": "nan",
+ "4175": "nan",
+ "4180": "nan",
+ "4185": "nan",
+ "4190": "nan",
+ "4195": "nan",
+ "4200": 0.20671,
+ "4205": "nan",
+ "4210": "nan",
+ "4215": "nan",
+ "4220": "nan",
+ "4225": "nan",
+ "4230": "nan",
+ "4235": "nan",
+ "4240": "nan",
+ "4245": "nan",
+ "4250": "nan",
+ "4255": "nan",
+ "4260": "nan",
+ "4265": "nan",
+ "4270": "nan",
+ "4275": "nan",
+ "4280": "nan",
+ "4285": "nan",
+ "4290": "nan",
+ "4295": "nan",
+ "4300": 0.16643,
+ "4305": "nan",
+ "4310": "nan",
+ "4315": "nan",
+ "4320": "nan",
+ "4325": "nan",
+ "4330": "nan",
+ "4335": "nan",
+ "4340": "nan",
+ "4345": "nan",
+ "4350": "nan",
+ "4355": "nan",
+ "4360": "nan",
+ "4365": "nan",
+ "4370": "nan",
+ "4375": "nan",
+ "4380": "nan",
+ "4385": "nan",
+ "4390": "nan",
+ "4395": "nan",
+ "4400": 0.16303,
+ "4405": "nan",
+ "4410": "nan",
+ "4415": "nan",
+ "4420": "nan",
+ "4425": "nan",
+ "4430": "nan",
+ "4435": "nan",
+ "4440": "nan",
+ "4445": "nan",
+ "4450": "nan",
+ "4455": "nan",
+ "4460": "nan",
+ "4465": "nan",
+ "4470": "nan",
+ "4475": "nan",
+ "4480": "nan",
+ "4485": "nan",
+ "4490": "nan",
+ "4495": "nan",
+ "4500": 0.17779,
+ "4505": "nan",
+ "4510": "nan",
+ "4515": "nan",
+ "4520": "nan",
+ "4525": "nan",
+ "4530": "nan",
+ "4535": "nan",
+ "4540": "nan",
+ "4545": "nan",
+ "4550": "nan",
+ "4555": "nan",
+ "4560": "nan",
+ "4565": "nan",
+ "4570": "nan",
+ "4575": "nan",
+ "4580": "nan",
+ "4585": "nan",
+ "4590": "nan",
+ "4595": "nan",
+ "4600": 0.22795,
+ "4605": "nan",
+ "4610": "nan",
+ "4615": "nan",
+ "4620": "nan",
+ "4625": "nan",
+ "4630": "nan",
+ "4635": "nan",
+ "4640": "nan",
+ "4645": "nan",
+ "4650": "nan",
+ "4655": "nan",
+ "4660": "nan",
+ "4665": "nan",
+ "4670": "nan",
+ "4675": "nan",
+ "4680": "nan",
+ "4685": "nan",
+ "4690": "nan",
+ "4695": "nan",
+ "4700": 0.22139,
+ "4705": "nan",
+ "4710": "nan",
+ "4715": "nan",
+ "4720": "nan",
+ "4725": "nan",
+ "4730": "nan",
+ "4735": "nan",
+ "4740": "nan",
+ "4745": "nan",
+ "4750": "nan",
+ "4755": "nan",
+ "4760": "nan",
+ "4765": "nan",
+ "4770": "nan",
+ "4775": "nan",
+ "4780": "nan",
+ "4785": "nan",
+ "4790": "nan",
+ "4795": "nan",
+ "4800": 0.17577,
+ "4805": "nan",
+ "4810": "nan",
+ "4815": "nan",
+ "4820": "nan",
+ "4825": "nan",
+ "4830": "nan",
+ "4835": "nan",
+ "4840": "nan",
+ "4845": "nan",
+ "4850": "nan",
+ "4855": "nan",
+ "4860": "nan",
+ "4865": "nan",
+ "4870": "nan",
+ "4875": "nan",
+ "4880": "nan",
+ "4885": "nan",
+ "4890": "nan",
+ "4895": "nan",
+ "4900": 0.16843,
+ "4905": "nan",
+ "4910": "nan",
+ "4915": "nan",
+ "4920": "nan",
+ "4925": "nan",
+ "4930": "nan",
+ "4935": "nan",
+ "4940": "nan",
+ "4945": "nan",
+ "4950": "nan",
+ "4955": "nan",
+ "4960": "nan",
+ "4965": "nan",
+ "4970": "nan",
+ "4975": "nan",
+ "4980": "nan",
+ "4985": "nan",
+ "4990": "nan",
+ "4995": "nan",
+ "5000": 0.19058,
+ "5005": "nan",
+ "5010": "nan",
+ "5015": "nan",
+ "5020": "nan",
+ "5025": "nan",
+ "5030": "nan",
+ "5035": "nan",
+ "5040": "nan",
+ "5045": "nan",
+ "5050": "nan",
+ "5055": "nan",
+ "5060": "nan",
+ "5065": "nan",
+ "5070": "nan",
+ "5075": "nan",
+ "5080": "nan",
+ "5085": "nan",
+ "5090": "nan",
+ "5095": "nan",
+ "5100": 0.18888,
+ "5105": "nan",
+ "5110": "nan",
+ "5115": "nan",
+ "5120": "nan",
+ "5125": "nan",
+ "5130": "nan",
+ "5135": "nan",
+ "5140": "nan",
+ "5145": "nan",
+ "5150": "nan",
+ "5155": "nan",
+ "5160": "nan",
+ "5165": "nan",
+ "5170": "nan",
+ "5175": "nan",
+ "5180": "nan",
+ "5185": "nan",
+ "5190": "nan",
+ "5195": "nan",
+ "5200": 0.22058,
+ "5205": "nan",
+ "5210": "nan",
+ "5215": "nan",
+ "5220": "nan",
+ "5225": "nan",
+ "5230": "nan",
+ "5235": "nan",
+ "5240": "nan",
+ "5245": "nan",
+ "5250": "nan",
+ "5255": "nan",
+ "5260": "nan",
+ "5265": "nan",
+ "5270": "nan",
+ "5275": "nan",
+ "5280": "nan",
+ "5285": "nan",
+ "5290": "nan",
+ "5295": "nan",
+ "5300": 0.19832,
+ "5305": "nan",
+ "5310": "nan",
+ "5315": "nan",
+ "5320": "nan",
+ "5325": "nan",
+ "5330": "nan",
+ "5335": "nan",
+ "5340": "nan",
+ "5345": "nan",
+ "5350": "nan",
+ "5355": "nan",
+ "5360": "nan",
+ "5365": "nan",
+ "5370": "nan",
+ "5375": "nan",
+ "5380": "nan",
+ "5385": "nan",
+ "5390": "nan",
+ "5395": "nan",
+ "5400": 0.17353,
+ "5405": "nan",
+ "5410": "nan",
+ "5415": "nan",
+ "5420": "nan",
+ "5425": "nan",
+ "5430": "nan",
+ "5435": "nan",
+ "5440": "nan",
+ "5445": "nan",
+ "5450": "nan",
+ "5455": "nan",
+ "5460": "nan",
+ "5465": "nan",
+ "5470": "nan",
+ "5475": "nan",
+ "5480": "nan",
+ "5485": "nan",
+ "5490": "nan",
+ "5495": "nan",
+ "5500": 0.19281,
+ "5505": "nan",
+ "5510": "nan",
+ "5515": "nan",
+ "5520": "nan",
+ "5525": "nan",
+ "5530": "nan",
+ "5535": "nan",
+ "5540": "nan",
+ "5545": "nan",
+ "5550": "nan",
+ "5555": "nan",
+ "5560": "nan",
+ "5565": "nan",
+ "5570": "nan",
+ "5575": "nan",
+ "5580": "nan",
+ "5585": "nan",
+ "5590": "nan",
+ "5595": "nan",
+ "5600": 0.17416,
+ "5605": "nan",
+ "5610": "nan",
+ "5615": "nan",
+ "5620": "nan",
+ "5625": "nan",
+ "5630": "nan",
+ "5635": "nan",
+ "5640": "nan",
+ "5645": "nan",
+ "5650": "nan",
+ "5655": "nan",
+ "5660": "nan",
+ "5665": "nan",
+ "5670": "nan",
+ "5675": "nan",
+ "5680": "nan",
+ "5685": "nan",
+ "5690": "nan",
+ "5695": "nan",
+ "5700": 0.19599,
+ "5705": "nan",
+ "5710": "nan",
+ "5715": "nan",
+ "5720": "nan",
+ "5725": "nan",
+ "5730": "nan",
+ "5735": "nan",
+ "5740": "nan",
+ "5745": "nan",
+ "5750": "nan",
+ "5755": "nan",
+ "5760": "nan",
+ "5765": "nan",
+ "5770": "nan",
+ "5775": "nan",
+ "5780": "nan",
+ "5785": "nan",
+ "5790": "nan",
+ "5795": "nan",
+ "5800": 0.20922,
+ "5805": "nan",
+ "5810": "nan",
+ "5815": "nan",
+ "5820": "nan",
+ "5825": "nan",
+ "5830": "nan",
+ "5835": "nan",
+ "5840": "nan",
+ "5845": "nan",
+ "5850": "nan",
+ "5855": "nan",
+ "5860": "nan",
+ "5865": "nan",
+ "5870": "nan",
+ "5875": "nan",
+ "5880": "nan",
+ "5885": "nan",
+ "5890": "nan",
+ "5895": "nan",
+ "5900": 0.19895,
+ "5905": "nan",
+ "5910": "nan",
+ "5915": "nan",
+ "5920": "nan",
+ "5925": "nan",
+ "5930": "nan",
+ "5935": "nan",
+ "5940": "nan",
+ "5945": "nan",
+ "5950": "nan",
+ "5955": "nan",
+ "5960": "nan",
+ "5965": "nan",
+ "5970": "nan",
+ "5975": "nan",
+ "5980": "nan",
+ "5985": "nan",
+ "5990": "nan",
+ "5995": "nan",
+ "6000": 0.19545,
+ "6005": "nan",
+ "6010": "nan",
+ "6015": "nan",
+ "6020": "nan",
+ "6025": "nan",
+ "6030": "nan",
+ "6035": "nan",
+ "6040": "nan",
+ "6045": "nan",
+ "6050": "nan",
+ "6055": "nan",
+ "6060": "nan",
+ "6065": "nan",
+ "6070": "nan",
+ "6075": "nan",
+ "6080": "nan",
+ "6085": "nan",
+ "6090": "nan",
+ "6095": "nan",
+ "6100": 0.23681,
+ "6105": "nan",
+ "6110": "nan",
+ "6115": "nan",
+ "6120": "nan",
+ "6125": "nan",
+ "6130": "nan",
+ "6135": "nan",
+ "6140": "nan",
+ "6145": "nan",
+ "6150": "nan",
+ "6155": "nan",
+ "6160": "nan",
+ "6165": "nan",
+ "6170": "nan",
+ "6175": "nan",
+ "6180": "nan",
+ "6185": "nan",
+ "6190": "nan",
+ "6195": "nan",
+ "6200": 0.1802,
+ "6205": "nan",
+ "6210": "nan",
+ "6215": "nan",
+ "6220": "nan",
+ "6225": "nan",
+ "6230": "nan",
+ "6235": "nan",
+ "6240": "nan",
+ "6245": "nan",
+ "6250": "nan",
+ "6255": "nan",
+ "6260": "nan",
+ "6265": "nan",
+ "6270": "nan",
+ "6275": "nan",
+ "6280": "nan",
+ "6285": "nan",
+ "6290": "nan",
+ "6295": "nan",
+ "6300": 0.16751,
+ "6305": "nan",
+ "6310": "nan",
+ "6315": "nan",
+ "6320": "nan",
+ "6325": "nan",
+ "6330": "nan",
+ "6335": "nan",
+ "6340": "nan",
+ "6345": "nan",
+ "6350": "nan",
+ "6355": "nan",
+ "6360": "nan",
+ "6365": "nan",
+ "6370": "nan",
+ "6375": "nan",
+ "6380": "nan",
+ "6385": "nan",
+ "6390": "nan",
+ "6395": "nan",
+ "6400": 0.17259,
+ "6405": "nan",
+ "6410": "nan",
+ "6415": "nan",
+ "6420": "nan",
+ "6425": "nan",
+ "6430": "nan",
+ "6435": "nan",
+ "6440": "nan",
+ "6445": "nan",
+ "6450": "nan",
+ "6455": "nan",
+ "6460": "nan",
+ "6465": "nan",
+ "6470": "nan",
+ "6475": "nan",
+ "6480": "nan",
+ "6485": "nan",
+ "6490": "nan",
+ "6495": "nan",
+ "6500": 0.1706,
+ "6505": "nan",
+ "6510": "nan",
+ "6515": "nan",
+ "6520": "nan",
+ "6525": "nan",
+ "6530": "nan",
+ "6535": "nan",
+ "6540": "nan",
+ "6545": "nan",
+ "6550": "nan",
+ "6555": "nan",
+ "6560": "nan",
+ "6565": "nan",
+ "6570": "nan",
+ "6575": "nan",
+ "6580": "nan",
+ "6585": "nan",
+ "6590": "nan",
+ "6595": "nan",
+ "6600": 0.26332,
+ "6605": "nan",
+ "6610": "nan",
+ "6615": "nan",
+ "6620": "nan",
+ "6625": "nan",
+ "6630": "nan",
+ "6635": "nan",
+ "6640": "nan",
+ "6645": "nan",
+ "6650": "nan",
+ "6655": "nan",
+ "6660": "nan",
+ "6665": "nan",
+ "6670": "nan",
+ "6675": "nan",
+ "6680": "nan",
+ "6685": "nan",
+ "6690": "nan",
+ "6695": "nan",
+ "6700": 0.19305,
+ "6705": "nan",
+ "6710": "nan",
+ "6715": "nan",
+ "6720": "nan",
+ "6725": "nan",
+ "6730": "nan",
+ "6735": "nan",
+ "6740": "nan",
+ "6745": "nan",
+ "6750": "nan",
+ "6755": "nan",
+ "6760": "nan",
+ "6765": "nan",
+ "6770": "nan",
+ "6775": "nan",
+ "6780": "nan",
+ "6785": "nan",
+ "6790": "nan",
+ "6795": "nan",
+ "6800": 0.16693,
+ "6805": "nan",
+ "6810": "nan",
+ "6815": "nan",
+ "6820": "nan",
+ "6825": "nan",
+ "6830": "nan",
+ "6835": "nan",
+ "6840": "nan",
+ "6845": "nan",
+ "6850": "nan",
+ "6855": "nan",
+ "6860": "nan",
+ "6865": "nan",
+ "6870": "nan",
+ "6875": "nan",
+ "6880": "nan",
+ "6885": "nan",
+ "6890": "nan",
+ "6895": "nan",
+ "6900": 0.16853,
+ "6905": "nan",
+ "6910": "nan",
+ "6915": "nan",
+ "6920": "nan",
+ "6925": "nan",
+ "6930": "nan",
+ "6935": "nan",
+ "6940": "nan",
+ "6945": "nan",
+ "6950": "nan",
+ "6955": "nan",
+ "6960": "nan",
+ "6965": "nan",
+ "6970": "nan",
+ "6975": "nan",
+ "6980": "nan",
+ "6985": "nan",
+ "6990": "nan",
+ "6995": "nan",
+ "7000": 0.18932,
+ "7005": "nan",
+ "7010": "nan",
+ "7015": "nan",
+ "7020": "nan",
+ "7025": "nan",
+ "7030": "nan",
+ "7035": "nan",
+ "7040": "nan",
+ "7045": "nan",
+ "7050": "nan",
+ "7055": "nan",
+ "7060": "nan",
+ "7065": "nan",
+ "7070": "nan",
+ "7075": "nan",
+ "7080": "nan",
+ "7085": "nan",
+ "7090": "nan",
+ "7095": "nan",
+ "7100": 0.18663,
+ "7105": "nan",
+ "7110": "nan",
+ "7115": "nan",
+ "7120": "nan",
+ "7125": "nan",
+ "7130": "nan",
+ "7135": "nan",
+ "7140": "nan",
+ "7145": "nan",
+ "7150": "nan",
+ "7155": "nan",
+ "7160": "nan",
+ "7165": "nan",
+ "7170": "nan",
+ "7175": "nan",
+ "7180": "nan",
+ "7185": "nan",
+ "7190": "nan",
+ "7195": "nan",
+ "7200": 0.24374,
+ "7205": "nan",
+ "7210": "nan",
+ "7215": "nan",
+ "7220": "nan",
+ "7225": "nan",
+ "7230": "nan",
+ "7235": "nan",
+ "7240": "nan",
+ "7245": "nan",
+ "7250": "nan",
+ "7255": "nan",
+ "7260": "nan",
+ "7265": "nan",
+ "7270": "nan",
+ "7275": "nan",
+ "7280": "nan",
+ "7285": "nan",
+ "7290": "nan",
+ "7295": "nan",
+ "7300": 0.18209,
+ "7305": "nan",
+ "7310": "nan",
+ "7315": "nan",
+ "7320": "nan",
+ "7325": "nan",
+ "7330": "nan",
+ "7335": "nan",
+ "7340": "nan",
+ "7345": "nan",
+ "7350": "nan",
+ "7355": "nan",
+ "7360": "nan",
+ "7365": "nan",
+ "7370": "nan",
+ "7375": "nan",
+ "7380": "nan",
+ "7385": "nan",
+ "7390": "nan",
+ "7395": "nan",
+ "7400": 0.18073,
+ "7405": "nan",
+ "7410": "nan",
+ "7415": "nan",
+ "7420": "nan",
+ "7425": "nan",
+ "7430": "nan",
+ "7435": "nan",
+ "7440": "nan",
+ "7445": "nan",
+ "7450": "nan",
+ "7455": "nan",
+ "7460": "nan",
+ "7465": "nan",
+ "7470": "nan",
+ "7475": "nan",
+ "7480": "nan",
+ "7485": "nan",
+ "7490": "nan",
+ "7495": "nan",
+ "7500": 0.19253,
+ "7505": "nan",
+ "7510": "nan",
+ "7515": "nan",
+ "7520": "nan",
+ "7525": "nan",
+ "7530": "nan",
+ "7535": "nan",
+ "7540": "nan",
+ "7545": "nan",
+ "7550": "nan",
+ "7555": "nan",
+ "7560": "nan",
+ "7565": "nan",
+ "7570": "nan",
+ "7575": "nan",
+ "7580": "nan",
+ "7585": "nan",
+ "7590": "nan",
+ "7595": "nan",
+ "7600": 0.17325,
+ "7605": "nan",
+ "7610": "nan",
+ "7615": "nan",
+ "7620": "nan",
+ "7625": "nan",
+ "7630": "nan",
+ "7635": "nan",
+ "7640": "nan",
+ "7645": "nan",
+ "7650": "nan",
+ "7655": "nan",
+ "7660": "nan",
+ "7665": "nan",
+ "7670": "nan",
+ "7675": "nan",
+ "7680": "nan",
+ "7685": "nan",
+ "7690": "nan",
+ "7695": "nan",
+ "7700": 0.20404,
+ "7705": "nan",
+ "7710": "nan",
+ "7715": "nan",
+ "7720": "nan",
+ "7725": "nan",
+ "7730": "nan",
+ "7735": "nan",
+ "7740": "nan",
+ "7745": "nan",
+ "7750": "nan",
+ "7755": "nan",
+ "7760": "nan",
+ "7765": "nan",
+ "7770": "nan",
+ "7775": "nan",
+ "7780": "nan",
+ "7785": "nan",
+ "7790": "nan",
+ "7795": "nan",
+ "7800": 0.20846,
+ "7805": "nan",
+ "7810": "nan",
+ "7815": "nan",
+ "7820": "nan",
+ "7825": "nan",
+ "7830": "nan",
+ "7835": "nan",
+ "7840": "nan",
+ "7845": "nan",
+ "7850": "nan",
+ "7855": "nan",
+ "7860": "nan",
+ "7865": "nan",
+ "7870": "nan",
+ "7875": "nan",
+ "7880": "nan",
+ "7885": "nan",
+ "7890": "nan",
+ "7895": "nan",
+ "7900": 0.19782,
+ "7905": "nan",
+ "7910": "nan",
+ "7915": "nan",
+ "7920": "nan",
+ "7925": "nan",
+ "7930": "nan",
+ "7935": "nan",
+ "7940": "nan",
+ "7945": "nan",
+ "7950": "nan",
+ "7955": "nan",
+ "7960": "nan",
+ "7965": "nan",
+ "7970": "nan",
+ "7975": "nan",
+ "7980": "nan",
+ "7985": "nan",
+ "7990": "nan",
+ "7995": "nan",
+ "8000": 0.18096,
+ "8005": "nan",
+ "8010": "nan",
+ "8015": "nan",
+ "8020": "nan",
+ "8025": "nan",
+ "8030": "nan",
+ "8035": "nan",
+ "8040": "nan",
+ "8045": "nan",
+ "8050": "nan",
+ "8055": "nan",
+ "8060": "nan",
+ "8065": "nan",
+ "8070": "nan",
+ "8075": "nan",
+ "8080": "nan",
+ "8085": "nan",
+ "8090": "nan",
+ "8095": "nan",
+ "8100": 0.22632,
+ "8105": "nan",
+ "8110": "nan",
+ "8115": "nan",
+ "8120": "nan",
+ "8125": "nan",
+ "8130": "nan",
+ "8135": "nan",
+ "8140": "nan",
+ "8145": "nan",
+ "8150": "nan",
+ "8155": "nan",
+ "8160": "nan",
+ "8165": "nan",
+ "8170": "nan",
+ "8175": "nan",
+ "8180": "nan",
+ "8185": "nan",
+ "8190": "nan",
+ "8195": "nan",
+ "8200": 0.17636,
+ "8205": "nan",
+ "8210": "nan",
+ "8215": "nan",
+ "8220": "nan",
+ "8225": "nan",
+ "8230": "nan",
+ "8235": "nan",
+ "8240": "nan",
+ "8245": "nan",
+ "8250": "nan",
+ "8255": "nan",
+ "8260": "nan",
+ "8265": "nan",
+ "8270": "nan",
+ "8275": "nan",
+ "8280": "nan",
+ "8285": "nan",
+ "8290": "nan",
+ "8295": "nan",
+ "8300": 0.16595,
+ "8305": "nan",
+ "8310": "nan",
+ "8315": "nan",
+ "8320": "nan",
+ "8325": "nan",
+ "8330": "nan",
+ "8335": "nan",
+ "8340": "nan",
+ "8345": "nan",
+ "8350": "nan",
+ "8355": "nan",
+ "8360": "nan",
+ "8365": "nan",
+ "8370": "nan",
+ "8375": "nan",
+ "8380": "nan",
+ "8385": "nan",
+ "8390": "nan",
+ "8395": "nan",
+ "8400": 0.16216,
+ "8405": "nan",
+ "8410": "nan",
+ "8415": "nan",
+ "8420": "nan",
+ "8425": "nan",
+ "8430": "nan",
+ "8435": "nan",
+ "8440": "nan",
+ "8445": "nan",
+ "8450": "nan",
+ "8455": "nan",
+ "8460": "nan",
+ "8465": "nan",
+ "8470": "nan",
+ "8475": "nan",
+ "8480": "nan",
+ "8485": "nan",
+ "8490": "nan",
+ "8495": "nan",
+ "8500": 0.18604,
+ "8505": "nan",
+ "8510": "nan",
+ "8515": "nan",
+ "8520": "nan",
+ "8525": "nan",
+ "8530": "nan",
+ "8535": "nan",
+ "8540": "nan",
+ "8545": "nan",
+ "8550": "nan",
+ "8555": "nan",
+ "8560": "nan",
+ "8565": "nan",
+ "8570": "nan",
+ "8575": "nan",
+ "8580": "nan",
+ "8585": "nan",
+ "8590": "nan",
+ "8595": "nan",
+ "8600": 0.24804,
+ "8605": "nan",
+ "8610": "nan",
+ "8615": "nan",
+ "8620": "nan",
+ "8625": "nan",
+ "8630": "nan",
+ "8635": "nan",
+ "8640": "nan",
+ "8645": "nan",
+ "8650": "nan",
+ "8655": "nan",
+ "8660": "nan",
+ "8665": "nan",
+ "8670": "nan",
+ "8675": "nan",
+ "8680": "nan",
+ "8685": "nan",
+ "8690": "nan",
+ "8695": "nan",
+ "8700": 0.17814,
+ "8705": "nan",
+ "8710": "nan",
+ "8715": "nan",
+ "8720": "nan",
+ "8725": "nan",
+ "8730": "nan",
+ "8735": "nan",
+ "8740": "nan",
+ "8745": "nan",
+ "8750": "nan",
+ "8755": "nan",
+ "8760": "nan",
+ "8765": "nan",
+ "8770": "nan",
+ "8775": "nan",
+ "8780": "nan",
+ "8785": "nan",
+ "8790": "nan",
+ "8795": "nan",
+ "8800": 0.16206,
+ "8805": "nan",
+ "8810": "nan",
+ "8815": "nan",
+ "8820": "nan",
+ "8825": "nan",
+ "8830": "nan",
+ "8835": "nan",
+ "8840": "nan",
+ "8845": "nan",
+ "8850": "nan",
+ "8855": "nan",
+ "8860": "nan",
+ "8865": "nan",
+ "8870": "nan",
+ "8875": "nan",
+ "8880": "nan",
+ "8885": "nan",
+ "8890": "nan",
+ "8895": "nan",
+ "8900": 0.17354,
+ "8905": "nan",
+ "8910": "nan",
+ "8915": "nan",
+ "8920": "nan",
+ "8925": "nan",
+ "8930": "nan",
+ "8935": "nan",
+ "8940": "nan",
+ "8945": "nan",
+ "8950": "nan",
+ "8955": "nan",
+ "8960": "nan",
+ "8965": "nan",
+ "8970": "nan",
+ "8975": "nan",
+ "8980": "nan",
+ "8985": "nan",
+ "8990": "nan",
+ "8995": "nan",
+ "9000": 0.19415,
+ "9005": "nan",
+ "9010": "nan",
+ "9015": "nan",
+ "9020": "nan",
+ "9025": "nan",
+ "9030": "nan",
+ "9035": "nan",
+ "9040": "nan",
+ "9045": "nan",
+ "9050": "nan",
+ "9055": "nan",
+ "9060": "nan",
+ "9065": "nan",
+ "9070": "nan",
+ "9075": "nan",
+ "9080": "nan",
+ "9085": "nan",
+ "9090": "nan",
+ "9095": "nan",
+ "9100": 0.18542,
+ "9105": "nan",
+ "9110": "nan",
+ "9115": "nan",
+ "9120": "nan",
+ "9125": "nan",
+ "9130": "nan",
+ "9135": "nan",
+ "9140": "nan",
+ "9145": "nan",
+ "9150": "nan",
+ "9155": "nan",
+ "9160": "nan",
+ "9165": "nan",
+ "9170": "nan",
+ "9175": "nan",
+ "9180": "nan",
+ "9185": "nan",
+ "9190": "nan",
+ "9195": "nan",
+ "9200": 0.23006,
+ "9205": "nan",
+ "9210": "nan",
+ "9215": "nan",
+ "9220": "nan",
+ "9225": "nan",
+ "9230": "nan",
+ "9235": "nan",
+ "9240": "nan",
+ "9245": "nan",
+ "9250": "nan",
+ "9255": "nan",
+ "9260": "nan",
+ "9265": "nan",
+ "9270": "nan",
+ "9275": "nan",
+ "9280": "nan",
+ "9285": "nan",
+ "9290": "nan",
+ "9295": "nan",
+ "9300": 0.16977,
+ "9305": "nan",
+ "9310": "nan",
+ "9315": "nan",
+ "9320": "nan",
+ "9325": "nan",
+ "9330": "nan",
+ "9335": "nan",
+ "9340": "nan",
+ "9345": "nan",
+ "9350": "nan",
+ "9355": "nan",
+ "9360": "nan",
+ "9365": "nan",
+ "9370": "nan",
+ "9375": "nan",
+ "9380": "nan",
+ "9385": "nan",
+ "9390": "nan",
+ "9395": "nan",
+ "9400": 0.17647,
+ "9405": "nan",
+ "9410": "nan",
+ "9415": "nan",
+ "9420": "nan",
+ "9425": "nan",
+ "9430": "nan",
+ "9435": "nan",
+ "9440": "nan",
+ "9445": "nan",
+ "9450": "nan",
+ "9455": "nan",
+ "9460": "nan",
+ "9465": "nan",
+ "9470": "nan",
+ "9475": "nan",
+ "9480": "nan",
+ "9485": "nan",
+ "9490": "nan",
+ "9495": "nan",
+ "9500": 0.19239,
+ "9505": "nan",
+ "9510": "nan",
+ "9515": "nan",
+ "9520": "nan",
+ "9525": "nan",
+ "9530": "nan",
+ "9535": "nan",
+ "9540": "nan",
+ "9545": "nan",
+ "9550": "nan",
+ "9555": "nan",
+ "9560": "nan",
+ "9565": "nan",
+ "9570": "nan",
+ "9575": "nan",
+ "9580": "nan",
+ "9585": "nan",
+ "9590": "nan",
+ "9595": "nan",
+ "9600": 0.17461,
+ "9605": "nan",
+ "9610": "nan",
+ "9615": "nan",
+ "9620": "nan",
+ "9625": "nan",
+ "9630": "nan",
+ "9635": "nan",
+ "9640": "nan",
+ "9645": "nan",
+ "9650": "nan",
+ "9655": "nan",
+ "9660": "nan",
+ "9665": "nan",
+ "9670": "nan",
+ "9675": "nan",
+ "9680": "nan",
+ "9685": "nan",
+ "9690": "nan",
+ "9695": "nan",
+ "9700": 0.19822,
+ "9705": "nan",
+ "9710": "nan",
+ "9715": "nan",
+ "9720": "nan",
+ "9725": "nan",
+ "9730": "nan",
+ "9735": "nan",
+ "9740": "nan",
+ "9745": "nan",
+ "9750": "nan",
+ "9755": "nan",
+ "9760": "nan",
+ "9765": "nan",
+ "9770": "nan",
+ "9775": "nan",
+ "9780": "nan",
+ "9785": "nan",
+ "9790": "nan",
+ "9795": "nan",
+ "9800": 0.20847,
+ "9805": "nan",
+ "9810": "nan",
+ "9815": "nan",
+ "9820": "nan",
+ "9825": "nan",
+ "9830": "nan",
+ "9835": "nan",
+ "9840": "nan",
+ "9845": "nan",
+ "9850": "nan",
+ "9855": "nan",
+ "9860": "nan",
+ "9865": "nan",
+ "9870": "nan",
+ "9875": "nan",
+ "9880": "nan",
+ "9885": "nan",
+ "9890": "nan",
+ "9895": "nan",
+ "9900": 0.19319,
+ "9905": "nan",
+ "9910": "nan",
+ "9915": "nan",
+ "9920": "nan",
+ "9925": "nan",
+ "9930": "nan",
+ "9935": "nan",
+ "9940": "nan",
+ "9945": "nan",
+ "9950": "nan",
+ "9955": "nan",
+ "9960": "nan",
+ "9965": "nan",
+ "9970": "nan",
+ "9975": "nan",
+ "9980": "nan",
+ "9985": "nan",
+ "9990": "nan",
+ "9995": "nan",
+ "10000": 0.18154
+ }
+ }
+}
\ No newline at end of file
diff --git a/tests/functional_tests/test_cases/t5/t5_release_sm/model_config.yaml b/tests/functional_tests/test_cases/t5/t5_release_sm/model_config.yaml
new file mode 100644
index 00000000000..6feba73c0a1
--- /dev/null
+++ b/tests/functional_tests/test_cases/t5/t5_release_sm/model_config.yaml
@@ -0,0 +1,68 @@
+ENV_VARS:
+ CUDA_DEVICE_MAX_CONNECTIONS: "1"
+ NVTE_ALLOW_NONDETERMINISTIC_ALGO: "1"
+ NON_DETERMINSTIC_RESULTS: 1
+TEST_TYPE: "release"
+MODEL_ARGS:
+ # T5 model args
+ --encoder-num-layers: 12
+ --decoder-num-layers: 12
+ --hidden-size: 768
+ --num-attention-heads: 12
+ --kv-channels: 64
+ --ffn-hidden-size: 3072
+ --encoder-seq-length: 512
+ --decoder-seq-length: 128
+ --max-position-embeddings: 512
+ --init-method-std: 0.015
+ --attention-backend: unfused
+ # Training args
+ --micro-batch-size: 32
+ --global-batch-size: 512
+ --train-iters: 100000
+ --exit-interval: 10000
+ --weight-decay: 1e-2
+ --clip-grad: 1.0
+ --bf16: true
+ --lr: 0.0001
+ --lr-decay-style: linear
+ --min-lr: 1.0e-5
+ --lr-warmup-fraction: .01
+ --distributed-backend: nccl
+ # Transformer Engine args
+ --use-mcore-models: true
+ --transformer-impl: transformer_engine
+ # Model parallel
+ --tensor-model-parallel-size: 4
+ --pipeline-model-parallel-size: 1
+ # Data args
+ --data-path: ${DATA_BLEND}
+ --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bert-large-cased-vocab.txt
+ --tokenizer-type: BertWordPieceCase
+ --split: 99982,9,9
+ --data-cache-path: ${DATA_CACHE_PATH}
+ --vocab-extra-ids: 100
+ # EVAL_AND_LOGGING_ARGS
+ --log-interval: 100
+ --save-interval: 2000
+ --eval-interval: 1000
+ --save-retain-interval: 10000
+ --save: ${CHECKPOINT_SAVE_PATH}
+ --load: ${CHECKPOINT_LOAD_PATH}
+ --eval-iters: 10
+ --tensorboard-dir: ${TENSORBOARD_PATH}
+ --log-timers-to-tensorboard: true
+ --log-memory-to-tensorboard: true
+ --log-num-zeros-in-grad: true
+ --log-params-norm: true
+ --log-validation-ppl-to-tensorboard: true
+ --timing-log-level: 0
+ --wandb-project: megatron-core-release-runs
+ --wandb-entity: adlr
+ --wandb-exp-name: ${WANDB_EXPERIMENT}
+ --wandb-save-dir: ${WANDB_SAVE_PATH}
+METRICS:
+ - "iteration-time"
+ - "lm loss"
+ - "mem-allocated-bytes"
+ - "mem-max-allocated-bytes"
diff --git a/tests/test_utils/python_scripts/auto_reminder_github.py b/tests/test_utils/python_scripts/auto_reminder_github.py
index 94e0de1ddaa..0329b2b23e0 100644
--- a/tests/test_utils/python_scripts/auto_reminder_github.py
+++ b/tests/test_utils/python_scripts/auto_reminder_github.py
@@ -40,6 +40,7 @@ class Reminder:
class PRReviewTracker:
EXPERT_REVIEW = "Expert Review"
FINAL_REVIEW = "Final Review"
+ APPROVED = "Approved"
EXCLUDED_TEAMS = {"core-adlr", "core-nemo"}
def __init__(
@@ -127,6 +128,11 @@ def get_label_date(self, pr, label: str):
]
return max(dates) if dates else None
+ def get_ready_for_review_date(self, pr):
+ """Get the date a PR was marked as ready for review."""
+ dates = [e.created_at for e in pr.as_issue().get_events() if e.event == "ready_for_review"]
+ return max(dates) if dates else None
+
def days_since(self, date):
"""Calculate days since given date."""
if not date:
@@ -138,11 +144,22 @@ def days_since(self, date):
def get_stage(self, pr):
"""Get current review stage."""
labels = {l.name for l in pr.labels}
- return self.FINAL_REVIEW if self.FINAL_REVIEW in labels else self.EXPERT_REVIEW
+ if self.APPROVED in labels:
+ return self.APPROVED
+ if self.FINAL_REVIEW in labels:
+ return self.FINAL_REVIEW
+ return self.EXPERT_REVIEW
def get_reviewers(self, pr):
"""Get filtered reviewer emails who haven't approved yet."""
stage = self.get_stage(pr)
+
+ if stage == self.APPROVED:
+ return (
+ [self.get_user_email(pr.user.login)],
+ "All reviewers have approved. Please merge the PR.",
+ )
+
org = self.github.get_organization(self.repo.organization.login)
# 1. Get the latest review state for everyone who has submitted a review
@@ -207,9 +224,19 @@ def get_reviewers(self, pr):
# 8. Handle the original edge cases
if len(reviewer_emails) == 0:
if stage == self.EXPERT_REVIEW:
+ # No reviewer activity yet — assignment hasn't completed (e.g. PR just became
+ # ready-for-review). Don't fire a spurious "all approved" message.
+ has_reviewer_activity = bool(
+ approvers
+ or non_approving_reviewers
+ or pending_individuals
+ or pending_teams_slugs
+ )
+ if not has_reviewer_activity:
+ return [], "Waiting for reviewers to be assigned."
# Assign to PR author
reviewer_emails = [self.get_user_email(pr.user.login)]
- action_message = "All Expert Reviewers approved the PR. Please attach the Final Review label to proceed with the review."
+ action_message = "All Expert Reviewers have approved the PR."
elif stage == self.FINAL_REVIEW:
# Assign to mcore-reviewers who approved
try:
@@ -217,7 +244,7 @@ def get_reviewers(self, pr):
mcore_members = {m.login for m in mcore_team.get_members()}
valid_approvers = approvers & mcore_members
reviewer_emails = sorted([self.get_user_email(u) for u in valid_approvers])
- action_message = "All Final Reviewers approved the PR. Please ping an Expert or Final Reviewer to merge the PR."
+ action_message = "All Final Reviewers approved the PR. Please ping the @mcore-oncall to merge the PR."
except Exception as e:
logger.warning(
@@ -229,7 +256,14 @@ def get_reviewers(self, pr):
def create_reminder(self, pr):
"""Create reminder for PR."""
stage = self.get_stage(pr)
- stage_days = self.days_since(self.get_label_date(pr, stage))
+ ready_date = self.get_ready_for_review_date(pr)
+ if stage == self.EXPERT_REVIEW:
+ stage_days = self.days_since(ready_date)
+ elif stage in (self.FINAL_REVIEW, self.APPROVED):
+ stage_days = self.days_since(self.get_label_date(pr, stage))
+ else:
+ stage_days = 0
+ total_review_days = self.days_since(ready_date)
author_email = self.get_user_email(pr.user.login)
reviewer_emails, action_message = self.get_reviewers(pr)
escaped_title = html.escape(pr.title, quote=False)
@@ -241,7 +275,7 @@ def create_reminder(self, pr):
author=self.get_slack_user_id(author_email),
priority="P0" if stage_days > 3 else "P1" if stage_days >= 1 else "P2",
review_stage=stage,
- total_review_time=self.days_since(self.get_label_date(pr, self.EXPERT_REVIEW)),
+ total_review_time=total_review_days,
current_stage_time=stage_days,
reviewers=[self.get_slack_user_id(email) for email in reviewer_emails],
action_message=action_message,
@@ -256,12 +290,11 @@ def generate_reminders(self):
reminders = []
for milestone in milestones:
- # Find issues with the 'Expert Review' or 'Final Review' label
+ # Find all open non-draft PRs with this milestone
query = (
f'repo:"{self.repo.full_name}" '
f'milestone:"{milestone.title}" '
- f'is:open is:pr '
- f'label:"{self.EXPERT_REVIEW}","{self.FINAL_REVIEW}"'
+ f'is:open is:pr -is:draft'
)
try:
# Use search_issues for a more direct query instead of get_issues + filtering
diff --git a/tests/test_utils/python_scripts/download_coverage_results.py b/tests/test_utils/python_scripts/download_coverage_results.py
index 8bea3d62a83..3f8f92e7f3c 100644
--- a/tests/test_utils/python_scripts/download_coverage_results.py
+++ b/tests/test_utils/python_scripts/download_coverage_results.py
@@ -1,3 +1,4 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
import glob
import logging
import os
@@ -35,7 +36,6 @@ def main(pipeline_id: int):
and pipeline_bridge.downstream_pipeline is not None
]
- ASSETS_DIR = pathlib.Path("tmp") / "results" / "iteration=0"
for pipeline_bridge in pipeline_bridges:
functional_pipeline = project.pipelines.get(pipeline_bridge.downstream_pipeline['id'])
@@ -49,6 +49,7 @@ def main(pipeline_id: int):
logger.info("Starting with job %s", job.name)
try:
+ shutil.rmtree(pathlib.Path("tmp") / "results", ignore_errors=True)
file_name = '__artifacts.zip'
with open(file_name, "wb") as f:
job.artifacts(streamed=True, action=f.write)
@@ -59,11 +60,15 @@ def main(pipeline_id: int):
continue
os.unlink(file_name)
- restart_dir = os.listdir(pathlib.Path("tmp") / "results" / "iteration=0")[-1]
+ iteration_dir = sorted(
+ (pathlib.Path("tmp") / "results").glob("iteration=*"),
+ key=lambda p: int(p.name.split("=")[1]),
+ )[-1]
+ restart_dir = sorted(os.listdir(iteration_dir))[-1]
coverage_report_source = list(
glob.glob(
str(
- pathlib.Path(ASSETS_DIR)
+ iteration_dir
/ f"{restart_dir}"
/ "assets"
/ "basic"
diff --git a/tests/test_utils/python_scripts/download_golden_values.py b/tests/test_utils/python_scripts/download_golden_values.py
index 423cd0ec254..a37f7295ecf 100644
--- a/tests/test_utils/python_scripts/download_golden_values.py
+++ b/tests/test_utils/python_scripts/download_golden_values.py
@@ -1,33 +1,28 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+import io
import logging
import os
import pathlib
+import re
import shutil
import zipfile
import click
import gitlab
+import requests
BASE_PATH = pathlib.Path(__file__).parent.resolve()
PROJECT_ID = int(os.getenv("CI_PROJECT_ID", 19378))
+GITHUB_REPO = os.getenv("GITHUB_REPOSITORY", "NVIDIA/Megatron-LM")
logger = logging.getLogger(__name__)
-@click.command()
-@click.option("--pipeline-id", required=True, type=int, help="Pipeline ID")
-@click.option(
- "--only-failing/--no-only-failing",
- default=False,
- help="Only download artifacts from failing jobs",
-)
-def main(pipeline_id: int, only_failing: bool):
- logging.basicConfig(level=logging.INFO)
- logger.info('Started')
-
- gitlab_endpoint = os.getenv('GITLAB_ENDPOINT')
- ro_api_token = os.getenv('RO_API_TOKEN')
+def download_from_gitlab(pipeline_id: int, only_failing: bool):
+ """Download golden values from Gitlab pipeline."""
+ gitlab_endpoint = os.getenv("GITLAB_ENDPOINT")
+ ro_api_token = os.getenv("RO_API_TOKEN")
if not gitlab_endpoint or not ro_api_token:
raise Exception(
@@ -48,9 +43,8 @@ def main(pipeline_id: int, only_failing: bool):
and pipeline_bridge.downstream_pipeline is not None
]
- ASSETS_DIR = pathlib.Path("tmp") / "results" / "iteration=0"
for pipeline_bridge in pipeline_bridges:
- functional_pipeline = project.pipelines.get(pipeline_bridge.downstream_pipeline['id'])
+ functional_pipeline = project.pipelines.get(pipeline_bridge.downstream_pipeline["id"])
environment = pipeline_bridge.name[len("functional:run_") :]
functional_pipeline_jobs = functional_pipeline.jobs.list(get_all=True)
logger.info("Starting with pipeline %s", pipeline_bridge.name)
@@ -62,7 +56,8 @@ def main(pipeline_id: int, only_failing: bool):
continue
try:
- file_name = '__artifacts.zip'
+ shutil.rmtree(pathlib.Path("tmp") / "results", ignore_errors=True)
+ file_name = "__artifacts.zip"
with open(file_name, "wb") as f:
job.artifacts(streamed=True, action=f.write)
zip = zipfile.ZipFile(file_name)
@@ -73,16 +68,18 @@ def main(pipeline_id: int, only_failing: bool):
continue
os.unlink(file_name)
- restart_dir = os.listdir(pathlib.Path("tmp") / "results" / "iteration=0")[-1]
+ iteration_dir = sorted(
+ (pathlib.Path("tmp") / "results").glob("iteration=*"),
+ key=lambda p: int(p.name.split("=")[1]),
+ )[-1]
+ restart_dir = sorted(os.listdir(iteration_dir))[-1]
+ job_name_hyphenated = job.name.replace("_", "-").lower()
+ assets_basic_dir = iteration_dir / f"{restart_dir}" / "assets" / "basic"
golden_values_sources = list(
- (
- pathlib.Path(ASSETS_DIR)
- / f"{restart_dir}"
- / "assets"
- / "basic"
- / f"{job.name.replace('_', '-').lower()}-{environment.replace('_', '-')}"
- ).glob("g*.json")
- )
+ (assets_basic_dir / f"{job_name_hyphenated}-{environment.replace('_', '-')}").glob(
+ "g*.json"
+ )
+ ) or list((assets_basic_dir / job_name_hyphenated).glob("g*.json"))
if len(golden_values_sources) < 1:
logger.info(
@@ -99,7 +96,7 @@ def main(pipeline_id: int, only_failing: bool):
golden_values_target = (
pathlib.Path("tests")
/ "functional_tests"
- / 'test_cases'
+ / "test_cases"
/ job.stage
/ job.name
/ golden_values_source_name
@@ -113,12 +110,394 @@ def main(pipeline_id: int, only_failing: bool):
shutil.move(golden_values_source, golden_values_target)
else:
- logger.info("Golden values for %s does not exist. Skip.", str(golden_values_source))
+ logger.info(
+ "Golden values for %s does not exist. Skip.", str(golden_values_source)
+ )
shutil.rmtree("tmp")
logger.info("beep boop: All done!")
+def _setup_github_headers(github_token: str) -> dict:
+ """Set up headers for GitHub API requests."""
+ return {
+ "Authorization": f"token {github_token}",
+ "Accept": "application/vnd.github+json",
+ "X-GitHub-Api-Version": "2022-11-28",
+ }
+
+
+def _fetch_workflow_run(workflow_id: int, headers: dict, api_base: str) -> dict:
+ """Fetch workflow run details from GitHub."""
+ logger.info("Fetching workflow run %d from %s", workflow_id, GITHUB_REPO)
+ workflow_run_url = f"{api_base}/repos/{GITHUB_REPO}/actions/runs/{workflow_id}"
+ response = requests.get(workflow_run_url, headers=headers)
+ response.raise_for_status()
+ workflow_run = response.json()
+
+ logger.info(
+ "Workflow run status: %s, conclusion: %s",
+ workflow_run["status"],
+ workflow_run["conclusion"],
+ )
+ return workflow_run
+
+
+def _fetch_all_jobs(workflow_id: int, headers: dict, api_base: str) -> list:
+ """Fetch all jobs from the workflow run with pagination."""
+ logger.info("Fetching all jobs from workflow run...")
+ all_jobs = []
+ jobs_url = f"{api_base}/repos/{GITHUB_REPO}/actions/runs/{workflow_id}/jobs"
+ page = 1
+ per_page = 100
+
+ while True:
+ response = requests.get(
+ jobs_url, headers=headers, params={"page": page, "per_page": per_page}
+ )
+ response.raise_for_status()
+ jobs_data = response.json()
+ jobs = jobs_data["jobs"]
+
+ if not jobs:
+ break
+
+ all_jobs.extend(jobs)
+ logger.info("Fetched page %d with %d jobs", page, len(jobs))
+
+ if len(jobs) < per_page:
+ break
+
+ page += 1
+
+ logger.info("Total jobs fetched: %d", len(all_jobs))
+ return all_jobs
+
+
+def _filter_cicd_jobs(all_jobs: list) -> list:
+ """Filter jobs that belong to cicd-integration-tests-latest."""
+ # Try to filter by workflow_name field
+ cicd_jobs = [
+ job
+ for job in all_jobs
+ if "workflow_name" in job
+ and "cicd-integration-tests-latest" in job.get("workflow_name", "").lower()
+ ]
+
+ # If workflow_name field doesn't exist, try filtering by job labels
+ if not cicd_jobs:
+ logger.info("No jobs found with workflow_name field. Checking job labels...")
+ cicd_jobs = [
+ job
+ for job in all_jobs
+ if "labels" in job
+ and any(
+ "cicd-integration-tests-latest" in str(label).lower() for label in job["labels"]
+ )
+ ]
+
+ # If still no jobs found, use all jobs as fallback
+ if not cicd_jobs:
+ logger.warning(
+ "Could not identify jobs belonging to cicd-integration-tests-latest. "
+ "Using all jobs from workflow run as fallback."
+ )
+ cicd_jobs = all_jobs
+ else:
+ logger.info("Filtered %d jobs belonging to cicd-integration-tests-latest", len(cicd_jobs))
+
+ return cicd_jobs
+
+
+def _create_job_name_map(cicd_jobs: list) -> dict:
+ """Create a mapping of normalized job names to job data for matching with artifacts."""
+ job_name_map = {}
+ for job in cicd_jobs:
+ job_name = job["name"]
+
+ # Extract test name from job name (between '/' and ' - ')
+ if "/" in job_name and " - " in job_name:
+ # Format: "stage/test_name - version"
+ test_name = job_name.split("/", 1)[1].split(" - ", 1)[0]
+ elif "/" in job_name:
+ # Format: "stage/test_name"
+ test_name = job_name.split("/", 1)[1]
+ else:
+ # No stage prefix, use the whole name
+ test_name = job_name.split(" - ", 1)[0] if " - " in job_name else job_name
+
+ # Normalize for matching (lowercase, preserve underscores)
+ normalized_name = test_name.lower()
+ job_name_map[normalized_name] = job
+ logger.info("Job '%s' normalized to '%s'", job["name"], normalized_name)
+
+ return job_name_map
+
+
+def _match_artifact_to_job(artifact: dict, workflow_id: int, job_name_map: dict) -> dict:
+ """Match an artifact to a job from the job name map."""
+ artifact_job_name = artifact["name"].lower().replace("logs-", "").split(f"-{workflow_id}")[0]
+ for normalized_job_name, job in job_name_map.items():
+ if normalized_job_name == artifact_job_name:
+ logger.info("Artifact '%s' matched to job '%s'", artifact["name"], job["name"])
+ return job
+
+ logger.info(
+ "Artifact '%s' does not match any cicd-integration-tests-latest jobs", artifact["name"]
+ )
+ return None
+
+
+def _fetch_and_filter_artifacts(
+ workflow_id: int, job_name_map: dict, only_failing: bool, headers: dict, api_base: str
+) -> list:
+ """Fetch and filter artifacts from the workflow run."""
+ logger.info("Fetching and filtering artifacts from workflow run...")
+ filtered_artifacts = []
+ artifacts_url = f"{api_base}/repos/{GITHUB_REPO}/actions/runs/{workflow_id}/artifacts"
+ page = 1
+ per_page = 100
+
+ while True:
+ response = requests.get(
+ artifacts_url, headers=headers, params={"page": page, "per_page": per_page}
+ )
+ response.raise_for_status()
+ artifacts_data = response.json()
+ artifacts = artifacts_data["artifacts"]
+
+ if not artifacts:
+ break
+
+ logger.info("Fetched artifacts page %d with %d artifacts", page, len(artifacts))
+
+ # Filter artifacts that match cicd jobs
+ for artifact in artifacts:
+ logger.info("Checking artifact: %s", artifact["name"])
+
+ matched_job = _match_artifact_to_job(artifact, workflow_id, job_name_map)
+
+ if matched_job:
+ # Check if we should skip based on only_failing flag
+ if only_failing and matched_job.get("conclusion") == "success":
+ logger.info(
+ "Job '%s' succeeded and only-failing is set. Skipping artifact '%s'",
+ matched_job["name"],
+ artifact["name"],
+ )
+ continue
+
+ artifact["_matched_job"] = matched_job
+ filtered_artifacts.append(artifact)
+
+ if len(artifacts) < per_page:
+ break
+
+ page += 1
+
+ logger.info(
+ "Filtered %d artifacts matching cicd-integration-tests-latest jobs", len(filtered_artifacts)
+ )
+ return filtered_artifacts
+
+
+def _extract_stage_and_test_name(job_name: str) -> tuple:
+ """Extract stage and test name from job name."""
+ if "/" in job_name:
+ # Format: "stage/test_name - version" or "stage/test_name"
+ stage, rest = job_name.split("/", 1)
+ # Remove version suffix if present (e.g., " - latest")
+ test_name = rest.split(" - ", 1)[0] if " - " in rest else rest
+ else:
+ # No stage in job name - this shouldn't happen for integration tests
+ raise ValueError(
+ f"Job name '{job_name}' does not contain a stage (expected format: 'stage/test_name - version')"
+ )
+
+ logger.info(f"Extracted stage: {stage}, test name: {test_name} from job '{job_name}'")
+ return stage, test_name
+
+
+def _download_and_extract_artifact(
+ artifact: dict, headers: dict, api_base: str, file_name: str = "__artifacts.zip"
+) -> list:
+ """Download and extract an artifact, returning list of golden values files."""
+ artifact_download_url = f"{api_base}/repos/{GITHUB_REPO}/actions/artifacts/{artifact['id']}/zip"
+
+ logger.info("Downloading artifact from %s", artifact_download_url)
+ response = requests.get(artifact_download_url, headers=headers, stream=True)
+ response.raise_for_status()
+
+ with open(file_name, "wb") as f:
+ for chunk in response.iter_content(chunk_size=8192):
+ f.write(chunk)
+
+ # Extract the artifact
+ zip_file = zipfile.ZipFile(file_name)
+ zip_file.extractall("tmp")
+ logger.info("Downloaded and extracted artifact %s", artifact["name"])
+
+ os.unlink(file_name)
+
+ # For GitHub, golden values are always in the top level of tmp
+ tmp_dir = pathlib.Path("tmp")
+ golden_values_in_tmp = list(tmp_dir.glob("golden_values*.json"))
+
+ if golden_values_in_tmp:
+ logger.info(
+ "Found %d golden values files in artifact %s",
+ len(golden_values_in_tmp),
+ artifact["name"],
+ )
+ else:
+ logger.info("No golden values found in artifact %s. Skipping.", artifact["name"])
+
+ return golden_values_in_tmp
+
+
+def _move_golden_values(golden_values_files: list, stage: str, test_name: str) -> int:
+ """Move golden values files to their target location."""
+ golden_values_moved = 0
+
+ for golden_values_file in golden_values_files:
+ golden_values_target = (
+ pathlib.Path("tests")
+ / "functional_tests"
+ / "test_cases"
+ / stage
+ / test_name
+ / golden_values_file.name
+ )
+
+ pathlib.Path(golden_values_target.parent).mkdir(parents=True, exist_ok=True)
+ logger.info(f"Moving golden values from {golden_values_file} to {golden_values_target}")
+ shutil.move(golden_values_file, golden_values_target)
+ golden_values_moved += 1
+
+ return golden_values_moved
+
+
+def _cleanup_tmp():
+ """Clean up temporary directory."""
+ if pathlib.Path("tmp").exists():
+ shutil.rmtree("tmp")
+
+
+def _process_artifact(artifact: dict, headers: dict, api_base: str) -> int:
+ """Process a single artifact (download, extract, and move golden values)."""
+ matched_job = artifact.get("_matched_job")
+ logger.info(
+ "Processing artifact: %s (matched to job: %s)",
+ artifact["name"],
+ matched_job["name"] if matched_job else "unknown",
+ )
+
+ file_name = "__artifacts.zip"
+
+ try:
+ # Download and extract the artifact
+ golden_values_files = _download_and_extract_artifact(artifact, headers, api_base, file_name)
+
+ if not golden_values_files:
+ _cleanup_tmp()
+ return 0
+
+ # Extract stage and test name from the matched job name
+ stage, test_name = _extract_stage_and_test_name(matched_job["name"])
+
+ # Move golden values to target location
+ golden_values_moved = _move_golden_values(golden_values_files, stage, test_name)
+
+ logger.info(
+ f"Successfully moved {golden_values_moved} golden values for test: {stage}/{test_name}"
+ )
+
+ # Clean up tmp directory after processing this artifact
+ _cleanup_tmp()
+
+ return golden_values_moved
+
+ except Exception as e:
+ logger.error("Failed to download/process artifact %s due to %s", artifact["name"], e)
+ # Clean up on error
+ if os.path.exists(file_name):
+ os.unlink(file_name)
+ _cleanup_tmp()
+ return 0
+
+
+def download_from_github(workflow_id: int, only_failing: bool):
+ """Download golden values from Github workflow run."""
+ github_token = os.getenv("GITHUB_TOKEN")
+
+ if not github_token:
+ raise Exception(
+ "Environment variable {GITHUB_TOKEN} has not been set. ie. GITHUB_TOKEN="
+ )
+
+ api_base = "https://api.github.com"
+ headers = _setup_github_headers(github_token)
+
+ # Fetch workflow run details
+ _fetch_workflow_run(workflow_id, headers, api_base)
+ logger.info("Setting only_failing to %s", only_failing)
+
+ # Fetch and filter jobs
+ all_jobs = _fetch_all_jobs(workflow_id, headers, api_base)
+ cicd_jobs = _filter_cicd_jobs(all_jobs)
+ job_name_map = _create_job_name_map(cicd_jobs)
+
+ # Fetch and filter artifacts
+ artifacts = _fetch_and_filter_artifacts(
+ workflow_id, job_name_map, only_failing, headers, api_base
+ )
+
+ # Process each artifact
+ total_golden_values = 0
+ total_tests_with_golden_values = 0
+
+ for artifact in artifacts:
+ golden_values_moved = _process_artifact(artifact, headers, api_base)
+
+ if golden_values_moved > 0:
+ total_golden_values += golden_values_moved
+ total_tests_with_golden_values += 1
+
+ logger.info(f"Total tests with golden values: {total_tests_with_golden_values}")
+ logger.info(f"Total golden values found: {total_golden_values}")
+ logger.info("beep boop: All done!")
+
+
+@click.command()
+@click.option(
+ "--source",
+ type=click.Choice(["gitlab", "github"], case_sensitive=False),
+ required=True,
+ help="Source platform (gitlab or github)",
+)
+@click.option(
+ "--pipeline-id", type=int, help="Gitlab Pipeline ID or Github Workflow Run ID", required=True
+)
+@click.option(
+ "--only-failing/--no-only-failing",
+ default=False,
+ help="Only download artifacts from failing jobs",
+)
+def main(source: str, pipeline_id: int, only_failing: bool):
+ """Download golden values from Gitlab or Github."""
+ logging.basicConfig(level=logging.INFO)
+ logger.info("Started")
+
+ source = source.lower()
+
+ if source == "gitlab":
+ download_from_gitlab(pipeline_id, only_failing)
+ elif source == "github":
+ download_from_github(pipeline_id, only_failing)
+ else:
+ raise click.UsageError(f"Unknown source: {source}")
+
+
if __name__ == "__main__":
main()
diff --git a/tests/test_utils/python_scripts/launch_nemo_run_workload.py b/tests/test_utils/python_scripts/launch_nemo_run_workload.py
index 8d006f70d19..cdb730fae63 100644
--- a/tests/test_utils/python_scripts/launch_nemo_run_workload.py
+++ b/tests/test_utils/python_scripts/launch_nemo_run_workload.py
@@ -54,6 +54,7 @@ def is_flaky_failure(concat_allranks_logs: str) -> bool:
"--n-repeat", required=False, type=int, help="Number of times to repeat the workload", default=1
)
@click.option("--data-dir", required=False, type=str, help="Data directory of the workload")
+@click.option("--hf-home", required=False, type=str, help="HF home directory of the workload")
@click.option("--tag", required=False, type=str, help="Tag of the workload")
@click.option(
"--enable-lightweight-mode",
@@ -73,6 +74,7 @@ def main(
container_image,
n_repeat: int = 1,
data_dir: Optional[str] = None,
+ hf_home: Optional[str] = None,
tag: Optional[str] = None,
enable_lightweight_mode: Optional[bool] = False,
):
@@ -107,6 +109,8 @@ def main(
artifacts.append(f"{os.getcwd()}:/opt/megatron-lm")
if data_dir:
artifacts.append(f"{pathlib.Path(data_dir)}:/mnt/artifacts")
+ if hf_home:
+ artifacts.append(f"{pathlib.Path(hf_home)}:/mnt/hf_home")
executor = run.DockerExecutor(
container_image=container_image,
@@ -122,6 +126,8 @@ def main(
"CLUSTER": "dgxh100_dgxc",
"NCCL_DEBUG": "INFO",
"NCCL_DEBUG_FILE": "/opt/megatron-lm/assets_dir/logs/nccl_debug.log",
+ "HF_HOME": "/mnt/hf_home",
+ "TRANSFORMERS_OFFLINE": "1",
},
packager=run.Packager(),
volumes=artifacts,
@@ -140,10 +146,10 @@ def main(
succeeded = str(job_dict["status"]) == "SUCCEEDED"
if succeeded:
- logger.info(f"Job succeeded with status: {job_dict["status"]}")
+ logger.info(f"Job succeeded with status: {job_dict['status']}")
sys.exit(0)
- logger.error(f"Job failed with status: {job_dict["status"]}")
+ logger.error(f"Job failed with status: {job_dict['status']}")
log_file_paths = pathlib.Path(os.getcwd()).glob("assets_dir/logs/*/*/attempt_0/*/std*.log")
all_ranks_all_logs = []
for log_file_path in log_file_paths:
diff --git a/tests/test_utils/python_scripts/swap_pr_labels.py b/tests/test_utils/python_scripts/swap_pr_labels.py
index aa5875443f8..1fde4520a89 100644
--- a/tests/test_utils/python_scripts/swap_pr_labels.py
+++ b/tests/test_utils/python_scripts/swap_pr_labels.py
@@ -35,6 +35,7 @@ class Reminder:
class PRReviewTracker:
EXPERT_REVIEW = "Expert Review"
FINAL_REVIEW = "Final Review"
+ APPROVED = "Approved"
EXCLUDED_TEAMS = {"core-adlr", "core-nemo"}
def __init__(self, token: str, repo_name: str, pr_number: str):
@@ -43,26 +44,110 @@ def __init__(self, token: str, repo_name: str, pr_number: str):
self.pr = self.repo.get_pull(pr_number)
self.stage = self.get_stage(self.pr)
self.org = self.github.get_organization(self.repo.organization.login)
+ self._team_cache = {}
+ self._codeowner_rules = self._parse_codeowners()
+
+ def _parse_codeowners(self):
+ """Parse CODEOWNERS into ordered list of (pattern, teams) rules."""
+ rules = []
+ try:
+ with open(".github/CODEOWNERS") as f:
+ for line in f:
+ line = line.strip()
+ if not line or line.startswith("#"):
+ continue
+ parts = line.split()
+ pattern = parts[0]
+ teams = set()
+ for part in parts[1:]:
+ if part.startswith("@NVIDIA/"):
+ teams.add(part.split("/", 1)[1])
+ rules.append((pattern, teams))
+ except FileNotFoundError:
+ logger.warning("CODEOWNERS file not found")
+ logger.info(f"Parsed {len(rules)} CODEOWNERS rules")
+ return rules
+
+ @staticmethod
+ def _match_file(filepath, pattern):
+ """Check if a file path matches a CODEOWNERS pattern.
+
+ Rules:
+ - Trailing '/' means directory pattern: matches all files under that directory.
+ - Pattern containing '/' is path-relative: exact match or directory prefix.
+ - Pattern without '/' matches the filename component anywhere.
+ """
+ if pattern.endswith('/'):
+ return filepath.startswith(pattern)
+ if '/' in pattern:
+ return filepath == pattern or filepath.startswith(pattern + '/')
+ return filepath == pattern or filepath.endswith('/' + pattern)
+
+ def _get_required_teams(self, pr):
+ """Determine required review teams from CODEOWNERS rules and PR changed files.
+
+ Uses last-match-wins semantics per file, then unions across all files.
+ """
+ required_teams = set()
+ try:
+ changed_files = [f.filename for f in pr.get_files()]
+ except Exception as e:
+ logger.warning(f"Could not get changed files for PR #{pr.number}: {e}")
+ return required_teams
+
+ for filepath in changed_files:
+ matched_teams = None
+ for pattern, teams in self._codeowner_rules:
+ if self._match_file(filepath, pattern):
+ matched_teams = teams
+ if matched_teams:
+ required_teams.update(matched_teams)
+ logger.info(f" {filepath} → {matched_teams}")
+
+ logger.info(f"Required teams from CODEOWNERS: {required_teams}")
+ return required_teams
def get_stage(self, pr):
"""Get current review stage."""
labels = {l.name for l in pr.labels}
- return self.FINAL_REVIEW if self.FINAL_REVIEW in labels else self.EXPERT_REVIEW
+ if self.APPROVED in labels:
+ return self.APPROVED
+ if self.FINAL_REVIEW in labels:
+ return self.FINAL_REVIEW
+ return self.EXPERT_REVIEW
+
+ def _get_team_members(self, slug):
+ """Get all members of a team, with caching."""
+ if slug not in self._team_cache:
+ try:
+ self._team_cache[slug] = {
+ m.login for m in self.org.get_team_by_slug(slug).get_members()
+ }
+ except Exception as e:
+ logger.warning(f"Could not get members for team {slug}: {e}")
+ self._team_cache[slug] = set()
+ return self._team_cache[slug]
+
+ def _get_teams_members(self, slugs):
+ """Get all members of multiple teams."""
+ members = set()
+ for slug in slugs:
+ members.update(self._get_team_members(slug))
+ return members
def swap_labels(self):
- """Get filtered reviewer emails who haven't approved yet."""
+ """Evaluate review state and update labels accordingly."""
pr = self.pr
- if self.stage == self.FINAL_REVIEW:
- logger.info(f"PR #{self.pr.number} is in the {self.stage} stage. No reviewers needed.")
+ if pr.draft:
+ logger.info(f"PR #{pr.number} is a draft. Skipping label swap.")
return
# 1. Get the latest review state for everyone who has submitted a review
latest_reviews = {}
try:
for review in pr.get_reviews():
- if not review.user: # Handle rare cases of deleted users
+ if not review.user:
continue
- # Only track 'APPROVED' or 'CHANGES_REQUESTED' as definitive states
if review.state in ("APPROVED", "CHANGES_REQUESTED"):
if (
review.user.login not in latest_reviews
@@ -72,61 +157,138 @@ def swap_labels(self):
except Exception as e:
logger.warning(f"Could not get reviews for PR #{pr.number}: {e}")
- # 2. Separate reviewers into approvers (List B) and non-approvers
approvers = {user for user, review in latest_reviews.items() if review.state == "APPROVED"}
- non_approving_reviewers = {
+ non_approvers = {
user for user, review in latest_reviews.items() if review.state == "CHANGES_REQUESTED"
}
- # 3. Get all *currently pending* review requests
- try:
- pending_users_req, pending_teams_req = pr.get_review_requests()
- pending_individuals = {r.login for r in pending_users_req}
- pending_teams_slugs = {t.slug for t in pending_teams_req}
- except Exception as e:
- logger.warning(f"Could not get review requests for PR #{pr.number}: {e}")
- pending_individuals = set()
- pending_teams_slugs = set()
-
- # 4. Filter pending teams based on the current stage
- teams_to_query = (
- pending_teams_slugs - self.EXCLUDED_TEAMS
- if self.stage == self.EXPERT_REVIEW
- else pending_teams_slugs & self.EXCLUDED_TEAMS
- )
-
- # 5. Get members from the required pending teams
- pending_team_members = set()
- for slug in teams_to_query:
+ # 2. Determine required teams from CODEOWNERS + changed files
+ required_teams = self._get_required_teams(pr)
+ if not required_teams:
+ logger.info(f"PR #{pr.number}: no CODEOWNERS teams matched changed files. Skipping.")
+ return
+
+ expert_required = required_teams - self.EXCLUDED_TEAMS
+ final_required = required_teams & self.EXCLUDED_TEAMS
+ logger.info(f"Expert teams required: {expert_required}")
+ logger.info(f"Final teams required: {final_required}")
+
+ # 3. Check which required teams still need approval (at least one member must approve)
+ # If _get_team_members fails (returns {}), the team stays pending — conservative.
+ pending_expert_teams = set()
+ for team in expert_required:
+ members = self._get_team_members(team)
+ if not (members & approvers):
+ pending_expert_teams.add(team)
+
+ pending_final_teams = set()
+ for team in final_required:
+ members = self._get_team_members(team)
+ if not (members & approvers):
+ pending_final_teams.add(team)
+
+ # 4. Compute pending reviewers: unsatisfied team members + individual blockers
+ all_excluded_members = self._get_teams_members(self.EXCLUDED_TEAMS)
+ expert_non_approvers = non_approvers - all_excluded_members
+ final_non_approvers = non_approvers & all_excluded_members
+
+ pending_expert = (
+ self._get_teams_members(pending_expert_teams) | expert_non_approvers
+ ) - approvers
+ pending_final = (
+ self._get_teams_members(pending_final_teams) | final_non_approvers
+ ) - approvers
+
+ logger.info(f"Pending expert teams: {pending_expert_teams}, reviewers: {pending_expert}")
+ logger.info(f"Pending final teams: {pending_final_teams}, reviewers: {pending_final}")
+
+ needs_final_review = bool(final_required)
+
+ # 5. State machine: update labels based on current stage and pending reviewers
+ if self.stage == self.APPROVED:
+ self._handle_approved_stage(pr, pending_expert, pending_final)
+ elif self.stage == self.FINAL_REVIEW:
+ self._handle_final_review_stage(pr, pending_expert, pending_final)
+ else:
+ self._handle_expert_review_stage(pr, pending_expert, pending_final, needs_final_review)
+
+ def _handle_approved_stage(self, pr, pending_expert, pending_final):
+ """Handle PRs that already have the Approved label."""
+ if len(pending_expert) > 0 or len(pending_final) > 0:
+ # New reviewers appeared — revert
try:
- pending_team_members.update(
- m.login for m in self.org.get_team_by_slug(slug).get_members()
- )
+ pr.remove_from_labels(self.APPROVED)
+ logger.info(f'Removed "{self.APPROVED}" from PR #{pr.number}')
except Exception as e:
- logger.warning(f"Could not get members for team {slug} on PR #{pr.number}: {e}")
+ logger.warning(f'Failed to remove "{self.APPROVED}" from PR #{pr.number}: {e}')
- # 6. "List A": Combine all users who *still need to review*
- all_required_reviewers = (
- pending_individuals | pending_team_members | non_approving_reviewers
- )
+ if len(pending_expert) > 0:
+ # Back to expert review — also remove Final Review if present
+ try:
+ pr.remove_from_labels(self.FINAL_REVIEW)
+ except Exception:
+ pass
+ logger.info(
+ f'Reverted PR #{pr.number} to expert review — pending: {pending_expert}'
+ )
+ else:
+ # Expert review done but final review needed again
+ try:
+ pr.add_to_labels(self.FINAL_REVIEW)
+ except Exception:
+ pass
+ logger.info(f'Reverted PR #{pr.number} to final review — pending: {pending_final}')
+ else:
+ logger.info(f"PR #{pr.number} is approved. No changes needed.")
- # 7. Final list (List A - List B):
- pending_reviewers = all_required_reviewers - approvers
- logger.info(f"Pending reviewers: {pending_reviewers}")
- if len(pending_reviewers) == 0:
+ def _handle_final_review_stage(self, pr, pending_expert, pending_final):
+ """Handle PRs in the Final Review stage."""
+ if len(pending_expert) > 0:
+ # New expert reviewers appeared — revert to expert review
try:
- pr.remove_from_labels(self.EXPERT_REVIEW)
- logger.info(f'Removed "{self.EXPERT_REVIEW}" label from PR #{pr.number}')
- except Exception as e:
- logger.warning(
- f'Failed to remove "{self.EXPERT_REVIEW}" label from PR #{pr.number}: {e}'
+ pr.remove_from_labels(self.FINAL_REVIEW)
+ logger.info(
+ f'Removed "{self.FINAL_REVIEW}" from PR #{pr.number} — '
+ f'new expert reviewers pending: {pending_expert}'
)
+ except Exception as e:
+ logger.warning(f'Failed to remove "{self.FINAL_REVIEW}" from PR #{pr.number}: {e}')
+ elif len(pending_final) == 0:
+ # All final reviewers approved — move to Approved, remove Final Review
+ try:
+ pr.remove_from_labels(self.FINAL_REVIEW)
+ logger.info(f'Removed "{self.FINAL_REVIEW}" from PR #{pr.number}')
+ except Exception as e:
+ logger.warning(f'Failed to remove "{self.FINAL_REVIEW}" from PR #{pr.number}: {e}')
+ try:
+ pr.add_to_labels(self.APPROVED)
+ logger.info(f'Added "{self.APPROVED}" to PR #{pr.number}')
+ except Exception as e:
+ logger.warning(f'Failed to add "{self.APPROVED}" to PR #{pr.number}: {e}')
+ else:
+ logger.info(f"PR #{pr.number} is in final review. Pending: {pending_final}")
+
+ def _handle_expert_review_stage(self, pr, pending_expert, pending_final, needs_final_review):
+ """Handle PRs in the Expert Review stage (no review labels yet)."""
+ if len(pending_expert) > 0:
+ logger.info(f"PR #{pr.number} is in expert review. Pending: {pending_expert}")
+ return
+ # All expert reviewers approved
+ if needs_final_review and len(pending_final) > 0:
+ # Final review teams are assigned and still pending
try:
pr.add_to_labels(self.FINAL_REVIEW)
- logger.info(f'Added "{self.FINAL_REVIEW}" label to PR #{pr.number}')
+ logger.info(f'Added "{self.FINAL_REVIEW}" to PR #{pr.number}')
+ except Exception as e:
+ logger.warning(f'Failed to add "{self.FINAL_REVIEW}" to PR #{pr.number}: {e}')
+ else:
+ # No final review needed, or final reviewers already approved
+ try:
+ pr.add_to_labels(self.APPROVED)
+ logger.info(f'Added "{self.APPROVED}" to PR #{pr.number}')
except Exception as e:
- logger.warning(f'Failed to add "{self.FINAL_REVIEW}" label to PR #{pr.number}: {e}')
+ logger.warning(f'Failed to add "{self.APPROVED}" to PR #{pr.number}: {e}')
def main():
diff --git a/tests/test_utils/recipes/gb200/gpt.yaml b/tests/test_utils/recipes/gb200/gpt.yaml
index f387fbb9a13..7c23813ae63 100644
--- a/tests/test_utils/recipes/gb200/gpt.yaml
+++ b/tests/test_utils/recipes/gb200/gpt.yaml
@@ -107,226 +107,226 @@ products:
- test_case: [gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer]
products:
- environment: [dev]
- scope: [mr, mr-github-broken]
+ scope: [mr]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
# - test_case: [gpt3_mcore_te_tp1_pp2_resume_torch_dist_cp4_a2a_p2p_nondeterministic]
# products:
# - environment: [dev]
- # scope: [mr]
+ # scope: [nightly]
- test_case: [gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp4_vp1]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
# - test_case: [gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist]
# products:
# - environment: [dev]
- # scope: [mr]
+ # scope: [nightly]
# platforms: [dgx_gb200] # Hangs: #513
- test_case: [gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp1_resume_torch_dist_cp2_nondeterministic]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2_nondeterministic]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2_etp4_nondeterministic_dp_last]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_nondeterministic]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_nondeterministic_dp_last]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_cross_entropy_loss_fusion]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_cp2_nondeterministic]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_cross_entropy_loss_fusion]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp1_gdn]
products:
- environment: [dev]
- scope: [mr, mr-github, mr-github-slim]
+ scope: [mr]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_mla]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_reshard_1x4xNone]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_tp2_pp2_uninstall_te]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_7b_tp1_pp4_memory_speed]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_7b_tp4_pp1_memory_speed]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp1_modelopt_distill_resume]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
#######################################################################
# Super important mr, mr-github tests that run for DEV per mr, mr-github #
@@ -334,89 +334,34 @@ products:
- test_case: [gpt3_mcore_reruns_persistent_1]
products:
- environment: [dev]
- scope: [mr, mr-github-broken]
+ scope: [mr]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer]
products:
- environment: [dev]
- scope: [mr, mr-github, mr-github-slim]
+ scope: [mr]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather]
products:
- environment: [dev]
- scope: [mr, mr-github, mr-github-slim]
+ scope: [mr]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp4_pp2_resume_torch_dist_reshard_8x1xNone]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- # - test_case: [gpt3_mcore_te_tp4_pp2_frozen_resume_torch_dist_reshard_8x1xNone]
- # products:
- # - environment: [dev]
- # scope: [mr]
- # platforms: [dgx_gb200]
- # - test_case: [gpt3_mcore_te_tp1_pp1_frozen_resume_torch_dist_dist_optimizer]
- # products:
- # - environment: [dev]
- # scope: [mr]
- # platforms: [dgx_gb200]
- # - test_case: [gpt3_mcore_te_tp1_pp4_frozen_resume_torch_dist_swiglu]
- # products:
- # - environment: [dev]
- # scope: [mr]
- # platforms: [dgx_gb200]
- # - test_case: [gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_cp2_nondeterministic]
- # products:
- # - environment: [dev]
- # scope: [mr]
- # platforms: [dgx_a100, dgx_gb200]
- # - test_case: [gpt3_weekly_dgx_b200_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap]
- # products:
- # - environment: [dev]
- # scope: [weekly]
- # platforms: [dgx_b200]
- # - test_case: [gpt3_weekly_dgx_b200_mcore_tp4_cp2_current_scaling_native_fp8_tp_fsdp]
- # products:
- # - environment: [dev]
- # scope: [weekly]
- # platforms: [dgx_b200]
- # - test_case: [gpt3_weekly_dgx_b200_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap]
- # products:
- # - environment: [dev]
- # scope: [weekly]
- # platforms: [dgx_b200]
- # - test_case: [gpt3_weekly_dgx_b200_mcore_tp4_cp2_mxfp8_tp_sp_cp]
- # products:
- # - environment: [dev]
- # scope: [weekly]
- # platforms: [dgx_b200]
- # - test_case: [gpt3_weekly_dgx_b200_mcore_tp4_cp2_native_fp8_tp_sp_cp_tp_overlap]
- # products:
- # - environment: [dev]
- # scope: [weekly]
- # platforms: [dgx_b200]
- - test_case: [gpt3_weekly_dgx_gb200_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap]
+ - test_case: [gpt3_weekly_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap]
products:
- environment: [dev]
scope: [weekly]
platforms: [dgx_gb200]
- # - test_case: [gpt3_weekly_dgx_gb200_mcore_tp4_cp2_current_scaling_native_fp8_tp_fsdp]
- # products:
- # - environment: [dev]
- # scope: [weekly]
- # platforms: [dgx_gb200]
- # - test_case: [gpt3_weekly_dgx_gb200_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap]
- # products:
- # - environment: [dev]
- # scope: [weekly]
- # platforms: [dgx_gb200]
- - test_case: [gpt3_weekly_dgx_gb200_mcore_tp4_cp2_native_fp8_tp_sp_cp_tp_overlap]
+ - test_case: [gpt3_weekly_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap]
products:
- environment: [dev]
scope: [weekly]
diff --git a/tests/test_utils/recipes/gb200/moe-1node.yaml b/tests/test_utils/recipes/gb200/moe-1node.yaml
new file mode 100644
index 00000000000..10fe3219c28
--- /dev/null
+++ b/tests/test_utils/recipes/gb200/moe-1node.yaml
@@ -0,0 +1,65 @@
+type: basic
+format_version: 1
+maintainers: [mcore]
+loggers: [stdout]
+spec:
+ name: "{test_case}_{environment}_{platforms}"
+ model: moe
+ build: mcore-pyt-{environment}
+ nodes: 1
+ gpus: 4
+ n_repeat: 5
+ platforms: dgx_gb200
+ script_setup: |
+ unset https_proxy
+ echo "machine gitlab-master.nvidia.com login okoenig password $RO_API_TOKEN" | tee -a /root/.netrc
+
+ # Checkout latest
+ cd /opt
+ rm -rf /opt/megatron-lm; mkdir megatron-lm; cd megatron-lm
+ git init
+ git remote add origin $MCORE_REPO
+ git fetch origin '+refs/merge-requests/*:refs/remotes/merge-requests/*'
+ git fetch origin $MCORE_MR_COMMIT
+ git checkout $MCORE_MR_COMMIT
+ git rev-parse HEAD
+
+ # Checkout backwards-ref
+ cd /opt
+ rm -rf /opt/megatron-lm-legacy; mkdir megatron-lm-legacy; cd megatron-lm-legacy
+ git init
+ git remote add origin $MCORE_REPO
+ git fetch origin $MCORE_BACKWARDS_COMMIT
+ git checkout $MCORE_BACKWARDS_COMMIT
+ git rev-parse HEAD
+ rm -rf megatron; cp -a /opt/megatron-lm/megatron ./
+ script: |-
+ ls
+ cd /opt/megatron-lm
+
+ NAME=$(echo {test_case}_{environment} | sed 's/dgx_gb200/dgx_a100/g')
+ export GPUS_PER_NODE={gpus}
+
+ ARGUMENTS=(
+ "DATA_PATH=/mnt/artifacts"
+ "DATA_CACHE_PATH=/lustre/fsw/coreai_dlalgo_mcore/mcore_ci/data/$RUN_ID/cache/"
+ "OUTPUT_PATH={assets_dir}"
+ "TENSORBOARD_PATH={assets_dir}/tensorboard"
+ "CHECKPOINT_SAVE_PATH={artifacts_dir}/checkpoints"
+ "CHECKPOINT_LOAD_PATH=/mnt/artifacts"
+ "TRAINING_SCRIPT_PATH=pretrain_gpt.py"
+ "TRAINING_PARAMS_PATH=./tests/functional_tests/test_cases/{model}/{test_case}/model_config.yaml"
+ "GOLDEN_VALUES_PATH=./tests/functional_tests/test_cases/{model}/{test_case}/golden_values_{environment}_{platforms}.json"
+ "N_REPEAT={n_repeat}"
+ "ENABLE_LIGHTWEIGHT_MODE=${{ENABLE_LIGHTWEIGHT_MODE}}"
+ "RECORD_CHECKPOINTS=${{RECORD_CHECKPOINTS}}"
+ )
+
+ bash ./tests/functional_tests/shell_test_utils/run_ci_test.sh ${{ARGUMENTS[@]}}
+
+products:
+ - test_case: [deepseek_proxy_fsdp_ep2_fsdp2_1node]
+ products:
+ - environment: [dev]
+ scope: [mr-github, mr-github-slim]
+ platforms: [dgx_gb200]
diff --git a/tests/test_utils/recipes/gb200/moe.yaml b/tests/test_utils/recipes/gb200/moe.yaml
index 28ae2415aac..eafcb874f4c 100644
--- a/tests/test_utils/recipes/gb200/moe.yaml
+++ b/tests/test_utils/recipes/gb200/moe.yaml
@@ -110,12 +110,12 @@ products:
- test_case: [gpt3_mcore_te_tp1_pp2_resume_torch_dist_reshard_2x1x4_te_8experts2parallel_dist_optimizer]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
# - test_case: [gpt3_mcore_te_tp2_pp2_ep4_etp1_mtp_resume_torch_dist_fp8]
# products:
# - environment: [dev]
- # scope: [mr]
+ # scope: [nightly]
# platforms: [dgx_gb200] # hang: #513
- test_case: [gpt3_mcore_te_tp2_pp2_ep4_etp1_resume_torch_dist_attn_cudagraph]
products:
@@ -125,67 +125,67 @@ products:
# - test_case: [gpt3_mcore_te_tp2_pp2_ep4_etp1_selective_recompute_experimental]
# products:
# - environment: [dev]
- # scope: [mr]
+ # scope: [nightly]
# platforms: [dgx_gb200] # hang: #513
- test_case: [gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_zp_z3_resume_torch_dist_te_8experts2parallel_top2router]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly-broken]
platforms: [dgx_gb200]
# - test_case: [gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_fine_grained_offloading]
# products:
# - environment: [dev]
- # scope: [mr]
+ # scope: [nightly]
# platforms: [dgx_gb200]
# - test_case: [gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_no_mtp_no_a2a_ovlp_fine_grained_offloading]
# products:
# - environment: [dev]
- # scope: [mr]
+ # scope: [nightly]
# platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph]
products:
- environment: [dev]
- scope: [mr]
+ scope: [nightly]
platforms: [dgx_gb200]
- test_case: [gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon]
products:
- environment: [dev]
- scope: [mr, mr-github, mr-slim]
+ scope: [mr, mr-slim]
platforms: [dgx_gb200]
- test_case: [gpt3_moe_mcore_te_ep8_resume_torch_dist_muon]
products:
- environment: [dev]
- scope: [mr, mr-github, mr-slim]
+ scope: [mr, mr-slim]
platforms: [dgx_gb200]
- test_case: [gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_fine_grained_offloading]
products:
@@ -198,12 +198,12 @@ products:
# - test_case: [gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_te_8experts2parallel_dist_optimizer]
# products:
# - environment: [dev]
- # scope: [mr]
+ # scope: [nightly]
# platforms: [dgx_gb200]
# - test_case: [gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_te_8experts2parallel_groupedGEMM]
# products:
# - environment: [dev]
- # scope: [mr]
+ # scope: [nightly]
# platforms: [dgx_gb200]
###########################
# Merge train tests #
@@ -211,10 +211,16 @@ products:
- test_case: [gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer]
products:
- environment: [dev]
- scope: [mr, mr-github, mr-github-slim]
+ scope: [mr]
platforms: [dgx_gb200]
- test_case: [gpt3_mcore_te_tp2_pp2_ep4_etp1_memory_speed]
products:
- environment: [dev]
scope: [mr-broken]
platforms: [dgx_gb200]
+
+ - test_case: [deepseek_proxy_fsdp_ep2_fsdp2]
+ products:
+ - environment: [dev]
+ scope: [mr]
+ platforms: [dgx_gb200]
diff --git a/tests/test_utils/recipes/h100/gpt-dynamic-inference-with-coordinator.yaml b/tests/test_utils/recipes/h100/gpt-dynamic-inference-with-coordinator.yaml
index 19d523eea8d..0349896345b 100644
--- a/tests/test_utils/recipes/h100/gpt-dynamic-inference-with-coordinator.yaml
+++ b/tests/test_utils/recipes/h100/gpt-dynamic-inference-with-coordinator.yaml
@@ -58,24 +58,25 @@ products:
- test_case: [gpt_dynamic_inference_tp8_pp1_dp1_583m_logitsmatch_zmq]
products:
- environment: [dev]
- scope: [flaky]
+ scope: [mr]
platforms: [dgx_h100]
- test_case: [gpt_dynamic_inference_tp1_pp8_dp1_583m_logitsmatch_zmq]
products:
- environment: [dev]
- scope: [flaky]
+ scope: [mr]
platforms: [dgx_h100]
- test_case: [gpt_dynamic_inference_tp1_pp1_dp8_583m_logitsmatch_zmq]
products:
- environment: [dev]
scope: [mr]
platforms: [dgx_h100]
+ - test_case: [gpt_dynamic_inference_tp2_pp2_dp2_583m_logitsmatch_zmq]
+ products:
+ - environment: [dev]
+ scope: [mr, mr-github]
+ platforms: [dgx_h100]
# - test_case: [gpt_dynamic_inference_tp1_pp1_dp8_583m_throughputtest_zmq]
# products:
# - environment: [dev]
# scope: [mr]
# platforms: [dgx_h100]
- - test_case: [gpt_dynamic_inference_tp2_pp2_dp2_583m_logitsmatch_zmq]
- products:
- - environment: [dev]
- scope: [flaky]
diff --git a/tests/test_utils/recipes/h100/gpt-grpo.yaml b/tests/test_utils/recipes/h100/gpt-grpo.yaml
index e707c1c2431..6df72dc3235 100644
--- a/tests/test_utils/recipes/h100/gpt-grpo.yaml
+++ b/tests/test_utils/recipes/h100/gpt-grpo.yaml
@@ -54,15 +54,25 @@ spec:
bash ./tests/functional_tests/shell_test_utils/run_ci_test.sh ${{ARGUMENTS[@]}}
products:
+ - test_case: [gpt_grpo_basic_function]
+ products:
+ - environment: [dev]
+ scope: [mr, mr-github]
+ platforms: [dgx_h100]
- test_case: [gpt_grpo_tp4_pp1_dp2_8b_throughput]
products:
- environment: [dev]
scope: [mr]
platforms: [dgx_h100]
+ - test_case: [gpt_grpo_tp4_pp1_dp2_8b_cudagraphs_throughput]
+ products:
+ - environment: [dev]
+ scope: [mr]
+ platforms: [dgx_h100]
- test_case: [gpt_grpo_tp4_pp1_dp2_8b_throughput_github]
products:
- environment: [dev]
- scope: [mr-github]
+ scope: [mr-github-broken]
platforms: [dgx_h100]
- test_case: [gpt_grpo_tp1tp2_pp1_dp8_583m_throughputtest]
products:
@@ -74,8 +84,3 @@ products:
- environment: [dev]
scope: [mr-github-broken]
platforms: [dgx_h100]
- - test_case: [gpt_grpo_tp2tp1_pp4pp2_dp8_583m_throughputtest]
- products:
- - environment: [dev]
- scope: [mr-broken]
- platforms: [dgx_h100]
diff --git a/tests/test_utils/recipes/h100/gpt-static-inference.yaml b/tests/test_utils/recipes/h100/gpt-static-inference.yaml
index 806762531fd..87046588b2b 100644
--- a/tests/test_utils/recipes/h100/gpt-static-inference.yaml
+++ b/tests/test_utils/recipes/h100/gpt-static-inference.yaml
@@ -63,15 +63,15 @@ products:
- test_case: [gpt_static_inference_tp1_pp1_583m_cudagraphs]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [gpt_static_inference_tp1_pp1_16b_multiprompt_tokensmatch]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
diff --git a/tests/test_utils/recipes/h100/gpt.yaml b/tests/test_utils/recipes/h100/gpt.yaml
index 90eddc55c27..52e38760f84 100644
--- a/tests/test_utils/recipes/h100/gpt.yaml
+++ b/tests/test_utils/recipes/h100/gpt.yaml
@@ -110,14 +110,14 @@ products:
- test_case: [gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer]
products:
@@ -129,7 +129,7 @@ products:
- test_case: [gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
@@ -142,28 +142,28 @@ products:
- test_case: [gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
# - environment: [lts]
# scope: [nightly] # outdated TE: #501
- test_case: [gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
# - environment: [lts]
# scope: [nightly] # non-determinism: #436
- test_case: [gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
@@ -177,28 +177,28 @@ products:
- test_case: [gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp1_pp4_vp1]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
@@ -212,7 +212,7 @@ products:
- test_case: [gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
@@ -247,83 +247,83 @@ products:
- test_case: [gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp2_pp1_resume_torch_dist_cp2_nondeterministic]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2_nondeterministic]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2_etp4_nondeterministic_dp_last]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_nondeterministic]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_nondeterministic_dp_last]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp2_pp2_cross_entropy_loss_fusion]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
@@ -331,14 +331,14 @@ products:
- test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_cp2_nondeterministic]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_cross_entropy_loss_fusion]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
@@ -350,26 +350,26 @@ products:
- test_case: [gpt3_mcore_te_tp2_pp2_mla]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
@@ -383,28 +383,28 @@ products:
- test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_reshard_1x4xNone]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_mcore_tp2_pp2_uninstall_te]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
@@ -425,14 +425,14 @@ products:
- test_case: [gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
# - environment: [lts]
# scope: [nightly]
- test_case: [gpt3_mcore_te_tp2_pp1_modelopt_distill_resume]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
# - environment: [lts]
# scope: [nightly] # Outdated: #502
@@ -469,7 +469,7 @@ products:
- test_case: [gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
@@ -483,71 +483,16 @@ products:
- test_case: [gpt3_mcore_te_tp4_pp2_resume_torch_dist_reshard_8x1xNone]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- # - test_case: [gpt3_mcore_te_tp4_pp2_frozen_resume_torch_dist_reshard_8x1xNone]
- # products:
- # - environment: [dev]
- # scope: [mr]
- # platforms: [dgx_h100]
- # - test_case: [gpt3_mcore_te_tp1_pp1_frozen_resume_torch_dist_dist_optimizer]
- # products:
- # - environment: [dev]
- # scope: [mr]
- # platforms: [dgx_h100]
- # - test_case: [gpt3_mcore_te_tp1_pp4_frozen_resume_torch_dist_swiglu]
- # products:
- # - environment: [dev]
- # scope: [mr]
- # platforms: [dgx_h100]
- # - test_case: [gpt3_mcore_te_tp2_pp1_frozen_resume_torch_dist_cp2_nondeterministic]
- # products:
- # - environment: [dev]
- # scope: [mr]
- # platforms: [dgx_a100, dgx_h100]
- # - test_case: [gpt3_weekly_dgx_b200_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap]
- # products:
- # - environment: [dev]
- # scope: [weekly]
- # platforms: [dgx_b200]
- # - test_case: [gpt3_weekly_dgx_b200_mcore_tp4_cp2_current_scaling_native_fp8_tp_fsdp]
- # products:
- # - environment: [dev]
- # scope: [weekly]
- # platforms: [dgx_b200]
- # - test_case: [gpt3_weekly_dgx_b200_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap]
- # products:
- # - environment: [dev]
- # scope: [weekly]
- # platforms: [dgx_b200]
- # - test_case: [gpt3_weekly_dgx_b200_mcore_tp4_cp2_mxfp8_tp_sp_cp]
- # products:
- # - environment: [dev]
- # scope: [weekly]
- # platforms: [dgx_b200]
- # - test_case: [gpt3_weekly_dgx_b200_mcore_tp4_cp2_native_fp8_tp_sp_cp_tp_overlap]
- # products:
- # - environment: [dev]
- # scope: [weekly]
- # platforms: [dgx_b200]
- - test_case: [gpt3_weekly_dgx_h100_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap]
+ - test_case: [gpt3_weekly_mcore_tp2_pp2_current_scaling_native_fp8_tp_pp_sp_tp_overlap]
products:
- environment: [dev]
scope: [weekly]
platforms: [dgx_h100]
- # - test_case: [gpt3_weekly_dgx_h100_mcore_tp4_cp2_current_scaling_native_fp8_tp_fsdp]
- # products:
- # - environment: [dev]
- # scope: [weekly]
- # platforms: [dgx_h100]
- # - test_case: [gpt3_weekly_dgx_h100_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap]
- # products:
- # - environment: [dev]
- # scope: [weekly]
- # platforms: [dgx_h100]
- - test_case: [gpt3_weekly_dgx_h100_mcore_tp4_cp2_native_fp8_tp_sp_cp_tp_overlap]
+ - test_case: [gpt3_weekly_mcore_tp4_cp2_current_scaling_native_fp8_tp_sp_cp_tp_overlap]
products:
- environment: [dev]
scope: [weekly]
diff --git a/tests/test_utils/recipes/h100/mamba-dynamic-inference.yaml b/tests/test_utils/recipes/h100/mamba-dynamic-inference.yaml
index c4c675746b9..495a4d130b0 100644
--- a/tests/test_utils/recipes/h100/mamba-dynamic-inference.yaml
+++ b/tests/test_utils/recipes/h100/mamba-dynamic-inference.yaml
@@ -58,10 +58,10 @@ products:
- test_case: [hybrid_dynamic_inference_tp1_pp1_dp8_583m]
products:
- environment: [dev]
- scope: [mr-github]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [hybrid_dynamic_inference_tp1_pp1_dp8_583m_chunked_prefill]
products:
- environment: [dev]
- scope: [mr-github]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
diff --git a/tests/test_utils/recipes/h100/mamba.yaml b/tests/test_utils/recipes/h100/mamba.yaml
index 47b731f7e00..703fb53160f 100644
--- a/tests/test_utils/recipes/h100/mamba.yaml
+++ b/tests/test_utils/recipes/h100/mamba.yaml
@@ -3,7 +3,7 @@ format_version: 1
maintainers: [mcore]
loggers: [stdout]
spec:
- name: '{test_case}_{environment}_{platforms}'
+ name: "{test_case}_{environment}_{platforms}"
model: hybrid
build: mcore-pyt-{environment}
nodes: 1
@@ -58,23 +58,31 @@ products:
- test_case: [hybrid_mr_mcore_te_tp1_pp1_cp1_dgx_a100_1N8G]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
# - environment: [lts] # disabled until triton is bumped
# scope: [nightly]
- # PP functional testing deferred
- # - test_case: [hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G]
- # products:
- # - environment: [dev]
- # scope: [mr]
- # - environment: [lts] # disabled until triton is bumped
- # scope: [nightly]
+ - test_case: [hybrid_mr_mcore_te_tp1_pp4_cp1_dgx_a100_1N8G]
+ products:
+ - environment: [dev]
+ scope: [mr, mr-github]
+ platforms: [dgx_h100]
+ # - environment: [lts] # disabled until triton is bumped
+ # scope: [nightly]
+
+ - test_case: [hybrid_mr_mcore_te_tp1_pp2_vpp2_cp1_dgx_a100_1N8G]
+ products:
+ - environment: [dev]
+ scope: [mr, mr-github]
+ platforms: [dgx_h100]
+ # - environment: [lts] # disabled until triton is bumped
+ # scope: [nightly]
- test_case: [hybrid_mr_mcore_te_tp2_pp1_cp1_dgx_a100_1N8G]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
# - environment: [lts] # disabled until triton is bumped
# scope: [nightly]
diff --git a/tests/test_utils/recipes/h100/mimo.yaml b/tests/test_utils/recipes/h100/mimo.yaml
index 88b17815ede..683a22feb2a 100644
--- a/tests/test_utils/recipes/h100/mimo.yaml
+++ b/tests/test_utils/recipes/h100/mimo.yaml
@@ -63,3 +63,11 @@ products:
products:
- environment: [dev]
scope: [flaky]
+ - test_case: [mimo_vlm_pretrain_convergence_tp1_pp1_cp1_dp8_seq_packing]
+ products:
+ - environment: [dev]
+ scope: [mr-github]
+ - test_case: [mimo_vlm_pretrain_convergence_tp1_pp1_cp2_dp8]
+ products:
+ - environment: [dev]
+ scope: [mr-github]
diff --git a/tests/test_utils/recipes/h100/moe-dynamic-inference-with-coordinator.yaml b/tests/test_utils/recipes/h100/moe-dynamic-inference-with-coordinator.yaml
index 513aa92834b..bb6d056dfd4 100644
--- a/tests/test_utils/recipes/h100/moe-dynamic-inference-with-coordinator.yaml
+++ b/tests/test_utils/recipes/h100/moe-dynamic-inference-with-coordinator.yaml
@@ -3,7 +3,7 @@ format_version: 1
maintainers: [mcore]
loggers: [stdout]
spec:
- name: '{test_case}_{environment}_{platforms}'
+ name: "{test_case}_{environment}_{platforms}"
model: moe
build: mcore-pyt-{environment}
nodes: 1
@@ -39,7 +39,7 @@ spec:
ARGUMENTS=(
"CHECKPOINT_LOAD_PATH=/mnt/artifacts"
"CHECKPOINT_SAVE_PATH=/tmp/checkpoints"
- "DATA_PATH=null"
+ "DATA_PATH=/mnt/artifacts/"
"DATA_CACHE_PATH=/workspace/data/cache"
"TRAINING_SCRIPT_PATH=examples/inference/gpt/gpt_dynamic_inference_with_coordinator.py"
"TRAINING_PARAMS_PATH=./tests/functional_tests/test_cases/{model}/{test_case}/model_config.yaml"
@@ -58,11 +58,15 @@ products:
- test_case: [gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq]
products:
- environment: [dev]
- scope: [flaky]
+ scope: [mr] #[mr, mr-github] - broken on github after NGC PyTorch26.02
platforms: [dgx_h100]
- test_case: [gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_cudagraph_zmq]
products:
- environment: [dev]
- scope: [flaky]
+ scope: [mr] #[mr, mr-github] - broken on github after NGC PyTorch26.02
+ platforms: [dgx_h100]
+ - test_case: [gpt_dynamic_inference_tp4_etp1_pp1_ep8_16B_logitsmatch_zmq_suspend_resume]
+ products:
+ - environment: [dev]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
-
diff --git a/tests/test_utils/recipes/h100/moe-static-inference.yaml b/tests/test_utils/recipes/h100/moe-static-inference.yaml
index f10d293e953..fdab3ff430b 100644
--- a/tests/test_utils/recipes/h100/moe-static-inference.yaml
+++ b/tests/test_utils/recipes/h100/moe-static-inference.yaml
@@ -58,15 +58,15 @@ products:
- test_case: [gpt_static_inference_tp1_pp1_ep1_16B_logitsmatch]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github-broken]
platforms: [dgx_h100]
- test_case: [gpt_static_inference_tp4_pp1_ep4_16B_logitsmatch]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github-broken]
platforms: [dgx_h100]
- test_case: [gpt_static_inference_cuda_graphs_pad_tp4_pp1_ep4_16B_logitsmatch]
products:
- environment: [dev]
- scope: [mr-broken]
+ scope: [mr, mr-github-broken]
platforms: [dgx_h100]
diff --git a/tests/test_utils/recipes/h100/moe.yaml b/tests/test_utils/recipes/h100/moe.yaml
index 06039d77440..fc7ad97e983 100644
--- a/tests/test_utils/recipes/h100/moe.yaml
+++ b/tests/test_utils/recipes/h100/moe.yaml
@@ -119,7 +119,7 @@ products:
- test_case: [gpt3_mcore_te_tp1_pp2_resume_torch_dist_reshard_2x1x4_te_8experts2parallel_dist_optimizer]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
# - test_case: [gpt3_mcore_te_tp2_pp2_ep4_etp1_mtp_resume_torch_dist_fp8]
# products:
@@ -139,17 +139,17 @@ products:
- test_case: [gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [gpt3_mcore_te_tp2_pp1_te_8experts_etp1_ep4]
products:
@@ -161,12 +161,12 @@ products:
- test_case: [gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [gpt3_mcore_te_tp2_zp_z3_resume_torch_dist_te_8experts2parallel_top2router]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr-broken, mr-github-broken]
platforms: [dgx_h100]
- test_case: [gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_fine_grained_offloading]
products:
@@ -181,19 +181,19 @@ products:
- test_case: [gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- environment: [lts]
scope: [nightly]
- test_case: [gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph]
products:
- environment: [dev]
- scope: [mr]
+ scope: [mr, mr-github]
platforms: [dgx_h100]
- test_case: [gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon]
products:
diff --git a/tests/test_utils/recipes/h100/unit-tests.yaml b/tests/test_utils/recipes/h100/unit-tests.yaml
index c3527272782..b2dba522546 100644
--- a/tests/test_utils/recipes/h100/unit-tests.yaml
+++ b/tests/test_utils/recipes/h100/unit-tests.yaml
@@ -129,6 +129,41 @@ products:
scope: [unit-tests]
n_repeat: [1]
time_limit: [1800]
+ - test_case: [tests/unit_tests/inference/**/*.py]
+ products:
+ - environment: [lts, dev]
+ tag: [latest, legacy]
+ scope: [unit-tests]
+ n_repeat: [1]
+ time_limit: [1800]
+ - test_case: [tests/unit_tests/rl/**/*.py]
+ products:
+ - environment: [lts, dev]
+ tag: [latest, legacy]
+ scope: [unit-tests]
+ n_repeat: [1]
+ time_limit: [1800]
+ - test_case: [tests/unit_tests/post_training/**/*.py]
+ products:
+ - environment: [lts, dev]
+ tag: [latest, legacy]
+ scope: [unit-tests]
+ n_repeat: [1]
+ time_limit: [1800]
+ - test_case: [tests/unit_tests/resharding/**/*.py]
+ products:
+ - environment: [lts, dev]
+ tag: [latest, legacy]
+ scope: [unit-tests]
+ n_repeat: [1]
+ time_limit: [1800]
+ - test_case: [tests/unit_tests/ssm/**/*.py]
+ products:
+ - environment: [lts, dev]
+ tag: [latest, legacy]
+ scope: [unit-tests]
+ n_repeat: [1]
+ time_limit: [1800]
- test_case: [tests/unit_tests/transformer/moe/**/*.py]
products:
- environment: [lts, dev]
diff --git a/tests/unit_tests/a2a_overlap/test_schedule_layer_1f1b.py b/tests/unit_tests/a2a_overlap/test_schedule_layer_1f1b.py
index 7e675e2333b..a4c1d60c907 100644
--- a/tests/unit_tests/a2a_overlap/test_schedule_layer_1f1b.py
+++ b/tests/unit_tests/a2a_overlap/test_schedule_layer_1f1b.py
@@ -13,6 +13,7 @@
get_gpt_mtp_block_spec,
)
from megatron.core.models.gpt.gpt_model import GPTModel
+from megatron.core.pipeline_parallel.utils import get_comm_stream, get_comp_stream, set_streams
from megatron.core.utils import is_te_min_version
from tests.unit_tests.a2a_overlap.utils import (
DummyState,
@@ -69,9 +70,8 @@ def run_transformer_layer_a2a_overlap_with_capture(model, input_tensors, microba
for i in range(len(input_tensors)):
input_tensors[i] = input_tensors[i].clone()
+ set_streams()
event = torch.cuda.Event()
- comp_stream = torch.cuda.current_stream()
- comm_stream = torch.cuda.Stream(device="cuda")
state = DummyState()
state.is_mtp = False
state.model = model
@@ -80,8 +80,8 @@ def run_transformer_layer_a2a_overlap_with_capture(model, input_tensors, microba
transformer_layer,
event,
state,
- comp_stream,
- comm_stream,
+ get_comp_stream,
+ get_comm_stream,
extra_args={"is_moe": True, "enable_deepep": False},
)
for _ in range(microbatches)
@@ -184,8 +184,7 @@ def run_mtp_layer_a2a_overlap_with_capture(
for i in range(len(hidden_states)):
hidden_states[i] = hidden_states[i].clone()
- comp_stream = torch.cuda.current_stream()
- comm_stream = torch.cuda.Stream(device="cuda")
+ set_streams()
layers = []
for _ in range(microbatches):
state = DummyState()
@@ -204,8 +203,8 @@ def run_mtp_layer_a2a_overlap_with_capture(
model.mtp.layers[0],
event,
state,
- comp_stream,
- comm_stream,
+ get_comp_stream,
+ get_comm_stream,
extra_args={
"is_moe": True,
"enable_deepep": False,
diff --git a/tests/unit_tests/conftest.py b/tests/unit_tests/conftest.py
index 5fb647b1507..c892056956f 100644
--- a/tests/unit_tests/conftest.py
+++ b/tests/unit_tests/conftest.py
@@ -1,4 +1,6 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# Copyright (c) 2026, Advanced Micro Devices, Inc. All rights reserved.
+# Modified for portability across upstream and ROCm CI environments.
import os
from pathlib import Path
@@ -11,6 +13,7 @@
from megatron.core.utils import is_te_min_version
from tests.test_utils.python_scripts.download_unit_tests_dataset import download_and_extract_asset
from tests.unit_tests.dist_checkpointing import TempNamedDir
+from tests.unit_tests.paths import unit_test_data_dir
from tests.unit_tests.test_utilities import Utils
@@ -80,16 +83,15 @@ def tmp_path_dist_ckpt(tmp_path_factory) -> Path:
@pytest.fixture(scope="session", autouse=True)
def ensure_test_data():
- """Ensure test data is available at /opt/data by downloading if necessary."""
- data_path = Path("/opt/data")
+ """Ensure unit test assets are available, downloading them if necessary."""
+ data_path = unit_test_data_dir()
# Check if data directory exists and has content
if not data_path.exists() or not any(data_path.iterdir()):
- print("Test data not found at /opt/data. Downloading...")
+ print(f"Test data not found at {data_path}. Downloading...")
try:
- # Download assets to /opt/data
- download_and_extract_asset(assets_dir=str(data_path))
+ download_and_extract_asset(assets_dir=data_path)
print("Test data downloaded successfully.")
@@ -100,7 +102,7 @@ def ensure_test_data():
print(f"Failed to download test data: {e}")
# Don't fail the tests, just warn
else:
- print("Test data already available at /opt/data")
+ print(f"Test data already available at {data_path}")
@pytest.fixture(autouse=True)
diff --git a/tests/unit_tests/data/test_builder.py b/tests/unit_tests/data/test_builder.py
index d0e86c87fb8..59f73911c82 100644
--- a/tests/unit_tests/data/test_builder.py
+++ b/tests/unit_tests/data/test_builder.py
@@ -22,7 +22,7 @@
from megatron.core.datasets.indexed_dataset import DType, IndexedDatasetBuilder
from megatron.core.datasets.megatron_dataset import LowLevelDataset, MegatronDataset
from megatron.core.datasets.utils import Split, compile_helpers, get_blend_from_list
-from megatron.training.tokenizer import build_tokenizer
+from megatron.core.tokenizers.utils.build_tokenizer import build_tokenizer
from megatron.training.utils import get_blend_and_blend_per_split
from tests.unit_tests.dist_checkpointing import TempNamedDir
from tests.unit_tests.test_utilities import Utils
diff --git a/tests/unit_tests/data/test_fim_dataset.py b/tests/unit_tests/data/test_fim_dataset.py
index 7022a4b5fa9..30d579507ad 100644
--- a/tests/unit_tests/data/test_fim_dataset.py
+++ b/tests/unit_tests/data/test_fim_dataset.py
@@ -1,4 +1,6 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2026, Advanced Micro Devices, Inc. All rights reserved.
+# Modified for portability across upstream and ROCm CI environments.
import pytest
import torch
@@ -7,11 +9,13 @@
from megatron.core.datasets.utils import compile_helpers, get_blend_from_list
from megatron.core.tokenizers import MegatronTokenizer
from megatron.training.datasets.fim_dataset import GPTFIMDataset, GPTFIMDatasetConfig
+from tests.unit_tests.paths import unit_test_data_path
from tests.unit_tests.test_utilities import Utils
@pytest.mark.parametrize("spm_rate", [0.0, 1.0])
@pytest.mark.parametrize("split_sample", [None, "python"])
+@pytest.mark.usefixtures("ensure_test_data")
def test_fim_gpt_dataset(spm_rate, split_sample):
if torch.distributed.is_available():
Utils.initialize_distributed()
@@ -22,12 +26,12 @@ def test_fim_gpt_dataset(spm_rate, split_sample):
compile_helpers()
tokenizer = MegatronTokenizer.from_pretrained(
- tokenizer_path="/opt/data/tokenizers/huggingface",
+ tokenizer_path=str(unit_test_data_path("tokenizers", "huggingface")),
metadata_path={"library": "huggingface"},
additional_special_tokens=["", "", "", "", ""],
include_special_tokens=True,
)
- blend = get_blend_from_list(["/opt/data/datasets/fim/fim_text_document"])
+ blend = get_blend_from_list([str(unit_test_data_path("datasets", "fim", "fim_text_document"))])
extra_tokens = {
"prefix": "",
"middle": "",
diff --git a/tests/unit_tests/data/test_gpt_dataset.py b/tests/unit_tests/data/test_gpt_dataset.py
index fdfa8645792..a2d25090fb8 100644
--- a/tests/unit_tests/data/test_gpt_dataset.py
+++ b/tests/unit_tests/data/test_gpt_dataset.py
@@ -13,7 +13,7 @@
from megatron.core.datasets.blended_megatron_dataset_builder import BlendedMegatronDatasetBuilder
from megatron.core.datasets.gpt_dataset import GPTDatasetConfig, MockGPTDataset
from megatron.core.datasets.utils import compile_helpers
-from megatron.training.tokenizer.tokenizer import _NullTokenizer
+from megatron.core.tokenizers import MegatronTokenizer
from tests.unit_tests.test_utilities import Utils
_MOCK_VOCAB_SIZE = 8192
@@ -37,7 +37,9 @@ def test_mock_gpt_dataset():
else:
compile_helpers()
- tokenizer = _NullTokenizer(vocab_size=_MOCK_VOCAB_SIZE)
+ tokenizer = MegatronTokenizer.from_pretrained(
+ metadata_path={"library": "null-text"}, vocab_size=_MOCK_VOCAB_SIZE
+ )
config = GPTDatasetConfig(
random_seed=1234,
diff --git a/tests/unit_tests/data/test_multimodal_dataset.py b/tests/unit_tests/data/test_multimodal_dataset.py
index f6ff575684c..2f9f76b7218 100644
--- a/tests/unit_tests/data/test_multimodal_dataset.py
+++ b/tests/unit_tests/data/test_multimodal_dataset.py
@@ -11,7 +11,7 @@
from megatron.core.datasets.blended_megatron_dataset_builder import BlendedMegatronDatasetBuilder
from megatron.core.datasets.multimodal_dataset import MockMultimodalDataset, MultimodalDatasetConfig
from megatron.core.datasets.utils import compile_helpers
-from megatron.training.tokenizer.tokenizer import _NullTokenizer
+from megatron.core.tokenizers import MegatronTokenizer
from tests.unit_tests.test_utilities import Utils
_MOCK_VOCAB_SIZE = 8192
@@ -26,6 +26,9 @@ def test_mock_multimodal_dataset():
else:
compile_helpers()
+ tokenizer = MegatronTokenizer.from_pretrained(
+ metadata_path={"library": "null-text"}, vocab_size=_MOCK_VOCAB_SIZE
+ )
config = MultimodalDatasetConfig(
random_seed=1234,
sequence_length=1024,
@@ -35,7 +38,7 @@ def test_mock_multimodal_dataset():
image_h=336,
image_w=336,
split="990,9,1",
- tokenizer=_NullTokenizer(vocab_size=_MOCK_VOCAB_SIZE),
+ tokenizer=tokenizer,
mid_level_dataset_surplus=0.005,
)
diff --git a/tests/unit_tests/data/test_preprocess_data.py b/tests/unit_tests/data/test_preprocess_data.py
index 48f3a2e7bb9..2d37104f0a3 100644
--- a/tests/unit_tests/data/test_preprocess_data.py
+++ b/tests/unit_tests/data/test_preprocess_data.py
@@ -1,7 +1,10 @@
# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2026, Advanced Micro Devices, Inc. All rights reserved.
+# Modified for portability across upstream and ROCm CI environments.
import json
import os
+import runpy
import sys
import tempfile
@@ -10,10 +13,8 @@
import requests
from megatron.core.datasets.indexed_dataset import IndexedDataset
-from megatron.training.tokenizer.gpt2_tokenization import (
- PRETRAINED_MERGES_ARCHIVE_MAP,
- PRETRAINED_VOCAB_ARCHIVE_MAP,
-)
+from megatron.core.tokenizers.text.libraries.megatron_hf_tokenizer import MEGATRON_CONFIG_MAP
+from tests.unit_tests.paths import repo_root, unit_test_data_path
from tools.merge_datasets import main as merge_main
from tools.preprocess_data import Encoder
from tools.preprocess_data import get_args as build_args
@@ -29,6 +30,8 @@
__LOCAL_GPT2_VOCAB = "/home/gitlab-runner/data/gpt3_data/gpt2-vocab.json"
+__PREPROCESS_DATA_SCRIPT = repo_root() / "tools" / "preprocess_data.py"
+
def dummy_jsonl(odir):
# numbers
@@ -170,7 +173,7 @@ def gpt2_vocab(odir):
return __LOCAL_GPT2_VOCAB
path = os.path.join(odir, "vocab.json")
with open(path, "wb") as writer:
- writer.write(requests.get(PRETRAINED_VOCAB_ARCHIVE_MAP['gpt2']).content)
+ writer.write(requests.get(MEGATRON_CONFIG_MAP['GPT2BPETokenizer']['vocab']).content)
return path
@@ -179,7 +182,7 @@ def gpt2_merge(odir):
return __LOCAL_GPT2_MERGE
path = os.path.join(odir, "merge.txt")
with open(path, "wb") as writer:
- writer.write(requests.get(PRETRAINED_MERGES_ARCHIVE_MAP['gpt2']).content)
+ writer.write(requests.get(MEGATRON_CONFIG_MAP['GPT2BPETokenizer']['merges_file']).content)
return path
@@ -191,9 +194,9 @@ def test_preprocess_data_gpt():
"--tokenizer-type",
"GPT2BPETokenizer",
"--vocab-file",
- "/opt/data/tokenizers/megatron/gpt2-vocab.json",
+ str(unit_test_data_path("tokenizers", "megatron", "gpt2-vocab.json")),
"--merge-file",
- "/opt/data/tokenizers/megatron/gpt2-merges.txt",
+ str(unit_test_data_path("tokenizers", "megatron", "gpt2-merges.txt")),
"--append-eod",
"--workers",
"10",
@@ -204,6 +207,39 @@ def test_preprocess_data_gpt():
do_test_preprocess_data(temp_dir, extra_args=gpt_args)
+@pytest.mark.usefixtures("ensure_test_data")
+def test_preprocess_data_gpt_optimal_workers():
+ with tempfile.TemporaryDirectory() as temp_dir:
+
+ # gpt specific args
+ gpt_args = [
+ "--input",
+ str(unit_test_data_path("datasets", "dclm", "dclm.jsonl")),
+ "--output-prefix",
+ f"{temp_dir}/optimal_workers",
+ "--tokenizer-type",
+ "GPT2BPETokenizer",
+ "--vocab-file",
+ str(unit_test_data_path("tokenizers", "megatron", "gpt2-vocab.json")),
+ "--merge-file",
+ str(unit_test_data_path("tokenizers", "megatron", "gpt2-merges.txt")),
+ "--append-eod",
+ "--workers",
+ "2",
+ "--log-interval",
+ "1",
+ "--find-optimal-num-workers",
+ "--workers-to-check",
+ "2",
+ "4",
+ "8",
+ "--max-documents",
+ "1002",
+ ]
+ sys.argv = [str(__PREPROCESS_DATA_SCRIPT)] + gpt_args
+ runpy.run_path(str(__PREPROCESS_DATA_SCRIPT), run_name="__main__")
+
+
def bert_vocab(odir):
if os.path.exists(__LOCAL_BERT_VOCAB):
return __LOCAL_BERT_VOCAB
@@ -223,7 +259,7 @@ def test_preprocess_data_bert():
"--tokenizer-type",
"BertWordPieceLowerCase",
"--vocab-file",
- "/opt/data/tokenizers/megatron/gpt2-vocab.json",
+ str(unit_test_data_path("tokenizers", "megatron", "gpt2-vocab.json")),
"--split-sentences",
"--workers",
"10",
@@ -240,3 +276,4 @@ def test_preprocess_data_bert():
if __name__ == "__main__":
test_preprocess_data_gpt()
test_preprocess_data_bert()
+ test_preprocess_data_gpt_optimal_workers()
diff --git a/tests/unit_tests/dist_checkpointing/conftest.py b/tests/unit_tests/dist_checkpointing/conftest.py
index e3e75fd85a8..301e78d9733 100644
--- a/tests/unit_tests/dist_checkpointing/conftest.py
+++ b/tests/unit_tests/dist_checkpointing/conftest.py
@@ -1,3 +1,5 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
from unittest import mock
import pytest
diff --git a/tests/unit_tests/dist_checkpointing/models/common.py b/tests/unit_tests/dist_checkpointing/models/common.py
index 8cb1dc4df65..d18d99d0b11 100644
--- a/tests/unit_tests/dist_checkpointing/models/common.py
+++ b/tests/unit_tests/dist_checkpointing/models/common.py
@@ -6,14 +6,14 @@
from megatron.core import parallel_state
from megatron.core.dist_checkpointing import load, load_plain_tensors, save
from megatron.core.dist_checkpointing.dict_utils import diff
-from megatron.core.dist_checkpointing.serialization import (
- get_default_load_sharded_strategy,
- get_default_save_sharded_strategy,
-)
from megatron.core.dist_checkpointing.strategies.fully_parallel import (
FullyParallelLoadStrategyWrapper,
FullyParallelSaveStrategyWrapper,
)
+from megatron.core.dist_checkpointing.strategies.torch import (
+ TorchDistLoadShardedStrategy,
+ TorchDistSaveShardedStrategy,
+)
from megatron.core.dist_checkpointing.validation import StrictHandling
from tests.unit_tests.dist_checkpointing import TempNamedDir
from tests.unit_tests.test_utilities import Utils
@@ -81,7 +81,7 @@ def common_test_parallel_reconfiguration_e2e(
pipeline_model_parallel_size=src_tp_pp[1],
**src_model_init_kwargs,
)
- save_strategy = get_default_save_sharded_strategy()
+ save_strategy = TorchDistSaveShardedStrategy()
if use_fpsl:
save_strategy = FullyParallelSaveStrategyWrapper(
save_strategy,
@@ -104,7 +104,7 @@ def common_test_parallel_reconfiguration_e2e(
**dst_model_init_kwargs,
)
if use_fpsl:
- load_strategy = get_default_load_sharded_strategy(ckpt_dir_A)
+ load_strategy = TorchDistLoadShardedStrategy()
load_strategy = FullyParallelLoadStrategyWrapper(load_strategy)
else:
load_strategy = None
diff --git a/tests/unit_tests/dist_checkpointing/models/test_mamba.py b/tests/unit_tests/dist_checkpointing/models/test_mamba.py
index 85fbe5dd045..dd2707abae5 100644
--- a/tests/unit_tests/dist_checkpointing/models/test_mamba.py
+++ b/tests/unit_tests/dist_checkpointing/models/test_mamba.py
@@ -6,14 +6,14 @@
from megatron.core import parallel_state
from megatron.core.dist_checkpointing import load, load_plain_tensors, save
from megatron.core.dist_checkpointing.dict_utils import diff
-from megatron.core.dist_checkpointing.serialization import (
- get_default_load_sharded_strategy,
- get_default_save_sharded_strategy,
-)
from megatron.core.dist_checkpointing.strategies.fully_parallel import (
FullyParallelLoadStrategyWrapper,
FullyParallelSaveStrategyWrapper,
)
+from megatron.core.dist_checkpointing.strategies.torch import (
+ TorchDistLoadShardedStrategy,
+ TorchDistSaveShardedStrategy,
+)
from megatron.core.extensions.transformer_engine import (
TELayerNormColumnParallelLinear,
TERowParallelLinear,
@@ -121,7 +121,7 @@ def test_parallel_reconfiguration_e2e(
)
sharded_state_dict = model_A.sharded_state_dict(prefix=layer_prefix, metadata=metadata)
- save_strategy = get_default_save_sharded_strategy()
+ save_strategy = TorchDistSaveShardedStrategy()
if use_fpsl:
save_strategy = FullyParallelSaveStrategyWrapper(
save_strategy,
@@ -149,7 +149,7 @@ def test_parallel_reconfiguration_e2e(
sequence_parallel=(dest_exp > 1 and dest_pp > 1),
)
if use_fpsl:
- load_strategy = get_default_load_sharded_strategy(ckpt_dir_A)
+ load_strategy = TorchDistLoadShardedStrategy()
load_strategy = FullyParallelLoadStrategyWrapper(
load_strategy,
parallel_state.get_data_parallel_group(with_context_parallel=True),
diff --git a/tests/unit_tests/dist_checkpointing/models/test_mlp_glu.py b/tests/unit_tests/dist_checkpointing/models/test_mlp_glu.py
index 864dfa65242..d4adb9d2e1e 100644
--- a/tests/unit_tests/dist_checkpointing/models/test_mlp_glu.py
+++ b/tests/unit_tests/dist_checkpointing/models/test_mlp_glu.py
@@ -13,7 +13,9 @@
get_param_id_to_sharded_param_map,
optim_state_to_sharding_state,
)
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
+from megatron.core.models.gpt.gpt_layer_specs import (
+ get_gpt_layer_with_transformer_engine_submodules,
+)
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer.mlp import MLP, apply_swiglu_sharded_factory
from megatron.core.transformer.transformer_config import TransformerConfig
@@ -32,7 +34,7 @@ def initialize_mlp(glu=True):
gated_linear_unit=glu,
)
return MLP(
- transformer_config, get_gpt_layer_with_transformer_engine_spec().submodules.mlp.submodules
+ transformer_config, get_gpt_layer_with_transformer_engine_submodules().mlp.submodules
)
diff --git a/tests/unit_tests/dist_checkpointing/models/test_moe_experts.py b/tests/unit_tests/dist_checkpointing/models/test_moe_experts.py
index dc0aad0a2a2..0b9dacd493e 100755
--- a/tests/unit_tests/dist_checkpointing/models/test_moe_experts.py
+++ b/tests/unit_tests/dist_checkpointing/models/test_moe_experts.py
@@ -1,6 +1,4 @@
# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
-import os
-
import pytest
import torch
from transformer_engine.pytorch.fp8 import check_fp8_support, fp8_autocast
@@ -8,21 +6,24 @@
from megatron.core import parallel_state
from megatron.core.dist_checkpointing import load, load_plain_tensors, save
from megatron.core.dist_checkpointing.dict_utils import diff
-from megatron.core.dist_checkpointing.serialization import (
- get_default_load_sharded_strategy,
- get_default_save_sharded_strategy,
-)
from megatron.core.dist_checkpointing.strategies.fully_parallel import (
FullyParallelLoadStrategyWrapper,
FullyParallelSaveStrategyWrapper,
)
+from megatron.core.dist_checkpointing.strategies.torch import (
+ TorchDistLoadShardedStrategy,
+ TorchDistSaveShardedStrategy,
+)
from megatron.core.models.gpt.gpt_layer_specs import (
- get_gpt_layer_local_spec,
- get_gpt_layer_with_transformer_engine_spec,
+ get_gpt_layer_local_submodules,
+ get_gpt_layer_with_transformer_engine_submodules,
)
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
-from megatron.core.transformer.moe.experts import GroupedMLP, SequentialMLP, TEGroupedMLP
+from megatron.core.transformer.mlp import MLPSubmodules
+from megatron.core.transformer.moe.experts import GroupedMLPSubmodules, SequentialMLP, TEGroupedMLP
+from megatron.core.transformer.moe.moe_layer import MoESubmodules
from megatron.core.transformer.moe.moe_utils import get_default_pg_collection
+from megatron.core.transformer.spec_utils import get_submodules
from megatron.core.transformer.transformer_config import TransformerConfig
from megatron.core.utils import is_te_min_version
from tests.unit_tests.dist_checkpointing import TempNamedDir
@@ -51,48 +52,48 @@ def initialize_expert_layer(seed, glu=True, expert_type='sequential', fp8=False,
)
default_config_kwargs.update(**config_kwargs)
transformer_config = TransformerConfig(**default_config_kwargs)
- if expert_type == 'grouped':
- model = GroupedMLP(num_local_experts, transformer_config, pg_collection)
- elif expert_type == 'te_grouped':
- transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec(
+ if expert_type == 'te_grouped':
+ layer_submodules = get_gpt_layer_with_transformer_engine_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=True
)
+ mlp_submodules = get_submodules(layer_submodules.mlp)
+ assert isinstance(mlp_submodules, MoESubmodules)
+ experts_submodules = get_submodules(mlp_submodules.experts)
+ assert isinstance(experts_submodules, GroupedMLPSubmodules)
model = TEGroupedMLP(
- num_local_experts,
- transformer_config,
- transformer_layer_spec.submodules.mlp.submodules.experts.submodules,
- pg_collection,
+ num_local_experts, transformer_config, experts_submodules, pg_collection
)
elif expert_type == 'sequential':
- transformer_layer_spec = get_gpt_layer_local_spec(
+ layer_submodules = get_gpt_layer_local_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=False
)
+ mlp_submodules = get_submodules(layer_submodules.mlp)
+ assert isinstance(mlp_submodules, MoESubmodules)
+ experts_submodules = get_submodules(mlp_submodules.experts)
+ assert isinstance(experts_submodules, MLPSubmodules)
model = SequentialMLP(
- num_local_experts,
- transformer_config,
- transformer_layer_spec.submodules.mlp.submodules.experts.submodules,
- pg_collection,
+ num_local_experts, transformer_config, experts_submodules, pg_collection
)
elif expert_type == 'te_sequential':
- transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec(
+ layer_submodules = get_gpt_layer_with_transformer_engine_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=False
)
+ mlp_submodules = get_submodules(layer_submodules.mlp)
+ assert isinstance(mlp_submodules, MoESubmodules)
+ experts_submodules = get_submodules(mlp_submodules.experts)
+ assert isinstance(experts_submodules, MLPSubmodules)
model = SequentialMLP(
- num_local_experts,
- transformer_config,
- transformer_layer_spec.submodules.mlp.submodules.experts.submodules,
- pg_collection,
+ num_local_experts, transformer_config, experts_submodules, pg_collection
)
else:
raise ValueError(
- 'expert_type can only be one of ["sequential", "te_sequential", "grouped",'
- ' "te_grouped"]'
+ 'expert_type can only be one of ["sequential", "te_sequential", "te_grouped"]'
)
return model
-expert_type = ['sequential', 'grouped']
-src_dest_expert_type = [('sequential', 'grouped'), ('grouped', 'sequential')]
+expert_type = ['sequential']
+src_dest_expert_type = []
if is_te_min_version("1.7.0.dev0"):
expert_type.append('te_sequential')
src_dest_expert_type.append(('sequential', 'te_sequential'))
@@ -181,7 +182,7 @@ def test_parallel_reconfiguration_e2e(
model_A = initialize_expert_layer(1, use_glu, expert_type)
sharded_state_dict = model_A.sharded_state_dict(prefix=layer_prefix, metadata=metadata)
- save_strategy = get_default_save_sharded_strategy()
+ save_strategy = TorchDistSaveShardedStrategy()
if use_fpsl:
save_strategy = FullyParallelSaveStrategyWrapper(
save_strategy,
@@ -205,7 +206,7 @@ def test_parallel_reconfiguration_e2e(
)
model_B = initialize_expert_layer(1, use_glu, expert_type)
if use_fpsl:
- load_strategy = get_default_load_sharded_strategy(ckpt_dir_A)
+ load_strategy = TorchDistLoadShardedStrategy()
load_strategy = FullyParallelLoadStrategyWrapper(
load_strategy,
parallel_state.get_data_parallel_group(with_context_parallel=True),
@@ -277,7 +278,7 @@ def test_sequential_grouped_mlp_interchangeable(
model_A = initialize_expert_layer(1, use_glu, expert_type=src_module)
sharded_state_dict = model_A.sharded_state_dict(prefix=layer_prefix, metadata=metadata)
- save_strategy = get_default_save_sharded_strategy()
+ save_strategy = TorchDistSaveShardedStrategy()
save(sharded_state_dict, ckpt_dir_A, save_strategy)
Utils.destroy_model_parallel()
@@ -356,7 +357,7 @@ def test_sequential_grouped_mlp_extra_state(
model_A(input_tensor, tokens_per_expert, probs)
sharded_state_dict = model_A.sharded_state_dict(prefix=layer_prefix, metadata=metadata)
- save_strategy = get_default_save_sharded_strategy()
+ save_strategy = TorchDistSaveShardedStrategy()
save(sharded_state_dict, ckpt_dir_A, save_strategy)
Utils.destroy_model_parallel()
diff --git a/tests/unit_tests/dist_checkpointing/test_async_save.py b/tests/unit_tests/dist_checkpointing/test_async_save.py
index 1c39dc0821d..cbb0b3f79b7 100755
--- a/tests/unit_tests/dist_checkpointing/test_async_save.py
+++ b/tests/unit_tests/dist_checkpointing/test_async_save.py
@@ -1,4 +1,5 @@
# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
+import sys
from unittest import mock
import pytest
@@ -9,7 +10,10 @@
from megatron.core.dist_checkpointing.dict_utils import diff
from megatron.core.dist_checkpointing.strategies.async_utils import AsyncCallsQueue
from megatron.core.dist_checkpointing.strategies.filesystem_async import FileSystemWriterAsync
-from megatron.core.dist_checkpointing.strategies.torch import TorchDistSaveShardedStrategy
+from megatron.core.dist_checkpointing.strategies.torch import (
+ TorchDistSaveShardedStrategy,
+ get_async_strategy,
+)
from tests.unit_tests.dist_checkpointing import TempNamedDir
from tests.unit_tests.test_utilities import Utils
@@ -56,7 +60,9 @@ def test_async_is_equivalent_to_sync(self, tmp_path_dist_ckpt, persistent, abort
):
# async
async_calls = AsyncCallsQueue(persistent)
- async_request = save(sharded_state_dict, async_ckpt_dir, async_sharded_save=True)
+ async_request = save(
+ sharded_state_dict, async_ckpt_dir, async_sharded_save=True, async_strategy="mcore"
+ )
async_calls.schedule_async_request(async_request)
# sync
@@ -74,37 +80,30 @@ def test_async_is_equivalent_to_sync(self, tmp_path_dist_ckpt, persistent, abort
Utils.destroy_model_parallel()
- @pytest.mark.parametrize('async_save', [False, True])
- @pytest.mark.parametrize('worker_fn', [write_data_os_err_mock_fn])
- @pytest.mark.failing_on_rocm
- def test_errors_are_reported(self, tmp_path_dist_ckpt, async_save, worker_fn):
- Utils.initialize_model_parallel(2, 4)
- orig_fn = FileSystemWriterAsync.write_preloaded_data
- FileSystemWriterAsync.write_preloaded_data = worker_fn
+ @pytest.mark.parametrize('async_strategy', ["nvrx", "mcore"])
+ def test_get_async_strategy(self, async_strategy):
+ strategy, modules = get_async_strategy(async_strategy)
- sharded_state_dict = {
- f'key{i}': ShardedTensor.from_rank_offsets(f'key{i}_rank{Utils.rank}', torch.ones(2, 4))
- for i in range(4) # make sure there is enough non-empty saving workers
- }
- save_strategy = TorchDistSaveShardedStrategy('torch_dist', 1, thread_count=8)
+ assert len(modules) > 1
+ assert strategy == async_strategy
- with (
- TempNamedDir(tmp_path_dist_ckpt / 'test_errors_are_reported') as ckpt_dir,
- pytest.raises(CheckpointException) as exc_info,
+ _, module = get_async_strategy(async_strategy, module="FileSystemWriterAsync")
+ assert type(module) is not dict
+
+ @pytest.mark.parametrize('async_strategy', ["nvrx", "mcore"])
+ def test_get_async_strategy_no_nvrx_installed(self, async_strategy):
+ with mock.patch.dict(
+ 'sys.modules', {'nvidia_resiliency_ext.checkpointing.async_ckpt.core': None}
):
- if async_save:
- async_calls = AsyncCallsQueue()
- async_request = save(
- sharded_state_dict, ckpt_dir, save_strategy, async_sharded_save=True
- )
- async_calls.schedule_async_request(async_request)
- async_calls.maybe_finalize_async_calls(blocking=True)
+ from megatron.core.dist_checkpointing.strategies.async_utils import (
+ AsyncRequest as MCoreAsyncRequest,
+ )
+
+ if async_strategy == "nvrx":
+ with pytest.raises(ModuleNotFoundError):
+ strategy, module = get_async_strategy(async_strategy, module="AsyncRequest")
else:
- save(sharded_state_dict, ckpt_dir, save_strategy)
- if Utils.rank == 0:
- assert 'Worker failure' in str(exc_info.value)
- else:
- assert 'Worker failure' not in str(exc_info.value)
+ strategy, module = get_async_strategy(async_strategy, module="AsyncRequest")
- FileSystemWriterAsync.write_preloaded_data = orig_fn
- Utils.destroy_model_parallel()
+ assert strategy == "mcore"
+ assert module == MCoreAsyncRequest
diff --git a/tests/unit_tests/dist_checkpointing/test_async_utils_shutdown.py b/tests/unit_tests/dist_checkpointing/test_async_utils_shutdown.py
new file mode 100644
index 00000000000..46cccf27f1e
--- /dev/null
+++ b/tests/unit_tests/dist_checkpointing/test_async_utils_shutdown.py
@@ -0,0 +1,48 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+"""Tests for shutdown-safety of async checkpoint callers.
+
+Verifies that ``PersistentAsyncCaller.__del__`` / ``close()`` does not raise
+when the distributed process group has already been destroyed (the scenario
+described in issue #3775).
+"""
+
+from unittest import mock
+
+import pytest
+import torch
+
+from megatron.core.dist_checkpointing.strategies.async_utils import (
+ PersistentAsyncCaller,
+ TemporalAsyncCaller,
+)
+
+
+class TestPersistentAsyncCallerShutdown:
+ """Verify ``PersistentAsyncCaller`` does not crash during GC shutdown."""
+
+ def test_close_without_process_group(self):
+ """Calling close() after process group destruction must not raise."""
+ caller = PersistentAsyncCaller()
+ # Simulate the state where no process was ever spawned (process is None)
+ # but close() still logs with the rank.
+ with mock.patch("torch.distributed.is_initialized", return_value=False):
+ # Must not raise
+ caller.close()
+
+ def test_del_without_process_group(self):
+ """``__del__`` must not raise when dist is uninitialised."""
+ caller = PersistentAsyncCaller()
+ with mock.patch("torch.distributed.is_initialized", return_value=False):
+ # Must not raise
+ caller.__del__()
+
+
+class TestTemporalAsyncCallerShutdown:
+ """Verify ``TemporalAsyncCaller`` does not crash during GC shutdown."""
+
+ def test_close_without_process_group(self):
+ """Calling close() after process group destruction must not raise."""
+ caller = TemporalAsyncCaller()
+ with mock.patch("torch.distributed.is_initialized", return_value=False):
+ caller.close()
diff --git a/tests/unit_tests/dist_checkpointing/test_fp8.py b/tests/unit_tests/dist_checkpointing/test_fp8.py
index 4fb89a8265a..136feadb370 100755
--- a/tests/unit_tests/dist_checkpointing/test_fp8.py
+++ b/tests/unit_tests/dist_checkpointing/test_fp8.py
@@ -5,14 +5,14 @@
from transformer_engine.pytorch.float8_tensor import Float8Tensor
from megatron.core.dist_checkpointing import ShardedTensor, load, save
-from megatron.core.dist_checkpointing.serialization import (
- get_default_load_sharded_strategy,
- get_default_save_sharded_strategy,
-)
from megatron.core.dist_checkpointing.strategies.fully_parallel import (
FullyParallelLoadStrategyWrapper,
FullyParallelSaveStrategyWrapper,
)
+from megatron.core.dist_checkpointing.strategies.torch import (
+ TorchDistLoadShardedStrategy,
+ TorchDistSaveShardedStrategy,
+)
from tests.unit_tests.dist_checkpointing import TempNamedDir
from tests.unit_tests.test_utilities import Utils
@@ -95,7 +95,7 @@ def get_state_dict(fill_val=1):
}
with TempNamedDir(tmp_path_dist_ckpt / 'test_fp8_save_load', sync=True) as ckpt_dir:
- save_strategy = get_default_save_sharded_strategy()
+ save_strategy = TorchDistSaveShardedStrategy()
if use_fpsl:
save_strategy = FullyParallelSaveStrategyWrapper(save_strategy, None, True)
save(get_state_dict(4), ckpt_dir, save_strategy)
@@ -104,7 +104,7 @@ def get_state_dict(fill_val=1):
Utils.initialize_model_parallel(*dest_tp_pp)
if use_fpsl:
- load_strategy = get_default_load_sharded_strategy(ckpt_dir)
+ load_strategy = TorchDistLoadShardedStrategy()
load_strategy = FullyParallelLoadStrategyWrapper(
load_strategy, None, False, load_exchange_algo
)
diff --git a/tests/unit_tests/dist_checkpointing/test_fully_parallel.py b/tests/unit_tests/dist_checkpointing/test_fully_parallel.py
index 494eaefb44c..6ca41462f82 100644
--- a/tests/unit_tests/dist_checkpointing/test_fully_parallel.py
+++ b/tests/unit_tests/dist_checkpointing/test_fully_parallel.py
@@ -27,10 +27,6 @@
ShardedTensorFactory,
is_main_replica,
)
-from megatron.core.dist_checkpointing.serialization import (
- get_default_load_sharded_strategy,
- get_default_save_sharded_strategy,
-)
from megatron.core.dist_checkpointing.strategies.base import (
LoadShardedStrategy,
SaveShardedStrategy,
@@ -68,7 +64,7 @@ def __init__(self, device='cpu'):
self.device = device
self.load_keys = set()
- def load(self, sharded_state_dict, ckpt_dir):
+ def load(self, sharded_state_dict, ckpt_dir, async_strategy="nvrx"):
for sh_ten in nested_values(sharded_state_dict):
if is_main_replica(sh_ten.replica_id):
self.load_keys.add(sh_ten.key)
@@ -591,7 +587,7 @@ def determine_cross_rank_reads(
state_dict = self.get_sharded_state_dict(ranks_placement)
with TempNamedDir(tmp_path_dist_ckpt / 'determine_cross_rank_reads') as ckpt_dir:
save_strategy = FullyParallelSaveStrategyWrapper(
- get_default_save_sharded_strategy(), parallelization_group
+ TorchDistSaveShardedStrategy(), parallelization_group
)
save_strategy.save(state_dict, ckpt_dir)
diff --git a/tests/unit_tests/dist_checkpointing/test_pipeline_parallel_layout.py b/tests/unit_tests/dist_checkpointing/test_pipeline_parallel_layout.py
index 19658986d6e..8b03262714a 100644
--- a/tests/unit_tests/dist_checkpointing/test_pipeline_parallel_layout.py
+++ b/tests/unit_tests/dist_checkpointing/test_pipeline_parallel_layout.py
@@ -288,6 +288,7 @@ def test_save_and_load_checkpoint_vpp(
args.num_attention_heads = 8
# Ckpt format
args.ckpt_format = "torch_dist"
+ args.async_strategy = "mcore"
set_args(args)
def set_tp_pp_vpp(tp, pp, vpp=None, pp_layout=None, destroy_first=True):
diff --git a/tests/unit_tests/dist_checkpointing/test_replication.py b/tests/unit_tests/dist_checkpointing/test_replication.py
index d4e792bf716..5ec9ad7ee53 100644
--- a/tests/unit_tests/dist_checkpointing/test_replication.py
+++ b/tests/unit_tests/dist_checkpointing/test_replication.py
@@ -73,7 +73,7 @@ def test_all_gather_batch(tp, pp):
# TODO: Use mock local checkpointing?
@pytest.mark.parametrize(('tp,pp'), [(2, 4), (1, 1)])
-@pytest.mark.parametrize(('async_save'), [True, False])
+@pytest.mark.parametrize(('async_save'), [False])
@pytest.mark.parametrize(('algo'), ['atomic', 'fully_parallel'])
@pytest.mark.parametrize(
("repl_groups"), [[[0, 1], [2, 3], [4, 5], [6, 7]], [[2, 6, 7], [3, 1], [5], [0, 4]]]
diff --git a/tests/unit_tests/dist_checkpointing/test_serialization.py b/tests/unit_tests/dist_checkpointing/test_serialization.py
index 2928197b872..a113363d49f 100755
--- a/tests/unit_tests/dist_checkpointing/test_serialization.py
+++ b/tests/unit_tests/dist_checkpointing/test_serialization.py
@@ -831,16 +831,15 @@ def _get_base_state_dict(self):
),
}
- @pytest.mark.parametrize('save_format', ['torch_dist'])
@pytest.mark.parametrize('validate_integrity', [True, False])
def test_unexpected_keys_handling_during_validation(
- self, caplog, tmp_path_dist_ckpt, validate_integrity, save_format
+ self, caplog, tmp_path_dist_ckpt, validate_integrity
):
sharded_state_dict = self._get_base_state_dict()
with TempNamedDir(
tmp_path_dist_ckpt / 'test_unexpected_keys_raises_error_during_validation'
) as ckpt_dir:
- save_strategy = get_default_strategy(StrategyAction.SAVE_SHARDED, save_format, 1)
+ save_strategy = get_default_strategy(StrategyAction.SAVE_SHARDED, 'torch_dist', 1)
save(sharded_state_dict, ckpt_dir, save_strategy)
def load_with_flag(strict):
@@ -865,9 +864,7 @@ def test_error(error_msg):
assert 'Missing keys' not in error_msg
# ASSUME_OK_UNEXPECTED results in an exception raised by the underlying strategy
- with pytest.raises(
- PyTCheckpointingException if save_format == 'torch_dist' else CheckpointingException
- ) as exc_info:
+ with pytest.raises(PyTCheckpointingException) as exc_info:
load_with_flag(StrictHandling.ASSUME_OK_UNEXPECTED)
# Informative exceptions with `RAISE_*` options:
with pytest.raises(CheckpointingException) as exc_info:
@@ -905,16 +902,15 @@ def test_error(error_msg):
loaded_state_dict = load_with_flag(StrictHandling.IGNORE_ALL)
assert 'TenA' in loaded_state_dict
- @pytest.mark.parametrize('save_format', ['torch_dist'])
@pytest.mark.parametrize('validate_integrity', [True, False])
def test_missing_keys_raises_error_during_validation(
- self, caplog, tmp_path_dist_ckpt, validate_integrity, save_format
+ self, caplog, tmp_path_dist_ckpt, validate_integrity
):
sharded_state_dict = self._get_base_state_dict()
with TempNamedDir(
tmp_path_dist_ckpt / 'test_missing_keys_raises_error_during_validation'
) as ckpt_dir:
- save_strategy = get_default_strategy(StrategyAction.SAVE_SHARDED, save_format, 1)
+ save_strategy = get_default_strategy(StrategyAction.SAVE_SHARDED, 'torch_dist', 1)
save(sharded_state_dict, ckpt_dir, save_strategy)
def load_with_flag(strict):
@@ -941,9 +937,21 @@ def test_error(error_msg):
loaded_state_dict = load_with_flag(StrictHandling.RAISE_UNEXPECTED)
assert 'TenB' in loaded_state_dict
+ # With LOG_UNEXPECTED and no unexpected keys, no unexpected-keys
+ # warning should be logged. Assert on WARNING-level records rather
+ # than `caplog.text == ''`: torch.distributed.checkpoint emits its
+ # own DEBUG telemetry (the `dcp_logger`, which sets its level to
+ # DEBUG and is captured regardless of `at_level`), so the raw buffer
+ # is non-deterministically non-empty.
+ caplog.clear()
with caplog.at_level(logging.WARNING):
loaded_state_dict = load_with_flag(StrictHandling.LOG_UNEXPECTED)
- assert caplog.text == ''
+ unexpected_warnings = [
+ record
+ for record in caplog.records
+ if record.levelno >= logging.WARNING and 'Unexpected keys' in record.getMessage()
+ ]
+ assert unexpected_warnings == []
assert 'TenB' in loaded_state_dict
loaded_state_dict, missing_keys, unexpected_keys = load_with_flag(
@@ -975,12 +983,11 @@ def test_error(error_msg):
assert unexpected_keys == set()
assert missing_keys == {'TenA', 'ObjB'}
- @pytest.mark.parametrize('save_format', ['torch_dist'])
@pytest.mark.parametrize('validate_integrity', [True, False])
- def test_exact_load_handling(self, caplog, tmp_path_dist_ckpt, validate_integrity, save_format):
+ def test_exact_load_handling(self, caplog, tmp_path_dist_ckpt, validate_integrity):
sharded_state_dict = self._get_base_state_dict()
with TempNamedDir(tmp_path_dist_ckpt / 'test_exact_load_handling') as ckpt_dir:
- save_strategy = get_default_strategy(StrategyAction.SAVE_SHARDED, save_format, 1)
+ save_strategy = get_default_strategy(StrategyAction.SAVE_SHARDED, 'torch_dist', 1)
save(sharded_state_dict, ckpt_dir, save_strategy)
def load_with_flag(strict):
@@ -1015,12 +1022,11 @@ def load_with_flag(strict):
assert missing_keys == set()
assert unexpected_keys == set()
- @pytest.mark.parametrize('save_format', ['torch_dist'])
- def test_sharded_metadata(self, tmp_path_dist_ckpt, save_format):
+ def test_sharded_metadata(self, tmp_path_dist_ckpt):
sharded_state_dict = self._get_base_state_dict()
with TempNamedDir(tmp_path_dist_ckpt / 'test_exact_load_handling') as ckpt_dir:
- save_strategy = get_default_strategy(StrategyAction.SAVE_SHARDED, save_format, 1)
+ save_strategy = get_default_strategy(StrategyAction.SAVE_SHARDED, 'torch_dist', 1)
save(sharded_state_dict, ckpt_dir, save_strategy)
torch.distributed.barrier()
sharded_metadata = load_sharded_metadata(ckpt_dir)
diff --git a/tests/unit_tests/dist_checkpointing/test_torch_dist.py b/tests/unit_tests/dist_checkpointing/test_torch_dist.py
index 4f4df058977..2ee7f32bb84 100644
--- a/tests/unit_tests/dist_checkpointing/test_torch_dist.py
+++ b/tests/unit_tests/dist_checkpointing/test_torch_dist.py
@@ -10,8 +10,7 @@
from megatron.core.dist_checkpointing import ShardedTensor, load, save
from megatron.core.dist_checkpointing.dict_utils import diff
-from megatron.core.dist_checkpointing.serialization import get_default_save_sharded_strategy
-from megatron.core.dist_checkpointing.strategies.async_utils import AsyncCallsQueue
+from megatron.core.dist_checkpointing.strategies.torch import TorchDistSaveShardedStrategy
from tests.unit_tests.dist_checkpointing import TempNamedDir
from tests.unit_tests.test_utilities import Utils
@@ -53,7 +52,7 @@ def test_cached_metadata(self, tmp_path_dist_ckpt):
with md_path.open('rb') as f:
md_non_cached = pickle.load(f)
- save_strategy = deepcopy(get_default_save_sharded_strategy())
+ save_strategy = deepcopy(TorchDistSaveShardedStrategy())
save_strategy.use_cached_ckpt_structure = True
# Run over 3 iterations with cached metadata enabled
# The 3rd iteration will run with cached metadata
@@ -117,7 +116,7 @@ def test_cpu_tensors_dont_take_too_much_space(self, tmp_path_dist_ckpt):
) as ckpt_dir:
save(sharded_state_dict, ckpt_dir)
- distcp_files = [(ckpt_dir / '__0_0.distcp'), (ckpt_dir / '__0_1.distcp')]
+ distcp_files = [(ckpt_dir / '__0_0.distcp')]
for file in distcp_files:
assert file.exists()
file_size = file.stat().st_size
diff --git a/tests/unit_tests/dist_checkpointing/utils.py b/tests/unit_tests/dist_checkpointing/utils.py
index dd12ecd7684..ec95602b020 100644
--- a/tests/unit_tests/dist_checkpointing/utils.py
+++ b/tests/unit_tests/dist_checkpointing/utils.py
@@ -6,6 +6,9 @@
import torch
+from megatron.core.dist_checkpointing.strategies.cached_metadata_filesystem_reader import (
+ CachedMetadataFileSystemReader,
+)
from megatron.core.models.gpt import GPTModel
from megatron.core.models.gpt.gpt_layer_specs import (
get_gpt_layer_local_spec,
@@ -167,6 +170,9 @@ def init_checkpointing_mock_args(args, ckpt_dir, fully_parallel=False):
args.dist_ckpt_optim_fully_reshardable = False
args.distrib_optim_fully_reshardable_mem_efficient = False
args.phase_transition_iterations = None
+ # Clear the metadata cache to avoid contamination between tests
+
+ CachedMetadataFileSystemReader.clear_metadata_cache()
def setup_model_and_optimizer(
@@ -224,7 +230,7 @@ def setup_model_and_optimizer(
opt.init_state_fn(opt)
optimizer.reload_model_params()
-
+ CachedMetadataFileSystemReader.clear_metadata_cache()
return unwrap_model(model), optimizer
@@ -322,5 +328,5 @@ def setup_moe_model_and_optimizer(
opt.init_state_fn(opt)
optimizer.reload_model_params()
-
+ CachedMetadataFileSystemReader.clear_metadata_cache()
return unwrap_model(model), optimizer
diff --git a/tests/unit_tests/distributed/megatron_fsdp/test_mcore_fully_sharded_data_parallel.py b/tests/unit_tests/distributed/megatron_fsdp/test_mcore_fully_sharded_data_parallel.py
index ea6fafe5fe5..2b9fb6acf69 100644
--- a/tests/unit_tests/distributed/megatron_fsdp/test_mcore_fully_sharded_data_parallel.py
+++ b/tests/unit_tests/distributed/megatron_fsdp/test_mcore_fully_sharded_data_parallel.py
@@ -87,6 +87,102 @@ def setup_class(cls):
def teardown_class(cls):
Utils.destroy_model_parallel()
+ def _build_fsdp_model(
+ self,
+ grad_reduce_in_fp32=False,
+ main_params_dtype=torch.float32,
+ main_grads_dtype=None,
+ grad_comm_dtype=None,
+ ):
+ """Helper to construct a FullyShardedDataParallel with the given dtype args."""
+ fsdp_config = DistributedDataParallelConfig(
+ data_parallel_sharding_strategy="optim_grads_params",
+ overlap_grad_reduce=True,
+ overlap_param_gather=True,
+ bucket_size=10000,
+ use_megatron_fsdp=True,
+ grad_reduce_in_fp32=grad_reduce_in_fp32,
+ megatron_fsdp_main_params_dtype=main_params_dtype,
+ megatron_fsdp_main_grads_dtype=main_grads_dtype,
+ megatron_fsdp_grad_comm_dtype=grad_comm_dtype,
+ )
+ model = TestModel(input_dim=13, output_dim=17).cuda()
+ transformer_config = TransformerConfig(
+ num_attention_heads=1, num_layers=1, context_parallel_size=1
+ )
+ fsdp_model = FullyShardedDataParallel(
+ config=transformer_config,
+ ddp_config=fsdp_config,
+ module=model,
+ fsdp_unit_modules=[torch.nn.Linear],
+ )
+ return fsdp_model
+
+ def test_fsdp_mp_policy_with_default_args(self):
+ """Test that FullyShardedDataParallel with default dtype args produces the expected policy."""
+ if not is_torch_min_version("2.4.0"):
+ pytest.skip("Megatron FSDP requires torch >= 2.4.0")
+
+ fsdp_model = self._build_fsdp_model()
+ assert fsdp_model.mp_policy.main_params_dtype == torch.float32
+ assert fsdp_model.mp_policy.main_grads_dtype is None
+ assert fsdp_model.mp_policy.grad_comm_dtype is None
+
+ @pytest.mark.parametrize(
+ ("main_params_dtype", "main_grads_dtype", "grad_comm_dtype"),
+ [
+ (torch.bfloat16, None, None),
+ (torch.float16, torch.float32, None),
+ (torch.float32, torch.bfloat16, torch.bfloat16),
+ (None, None, None),
+ ],
+ )
+ def test_fsdp_mp_policy_with_custom_dtypes(
+ self, main_params_dtype, main_grads_dtype, grad_comm_dtype
+ ):
+ """Test that FullyShardedDataParallel forwards custom dtypes to mp_policy correctly."""
+ if not is_torch_min_version("2.4.0"):
+ pytest.skip("Megatron FSDP requires torch >= 2.4.0")
+
+ fsdp_model = self._build_fsdp_model(
+ main_params_dtype=main_params_dtype,
+ main_grads_dtype=main_grads_dtype,
+ grad_comm_dtype=grad_comm_dtype,
+ )
+ assert fsdp_model.mp_policy.main_params_dtype == main_params_dtype
+ assert fsdp_model.mp_policy.main_grads_dtype == main_grads_dtype
+ assert fsdp_model.mp_policy.grad_comm_dtype == grad_comm_dtype
+
+ def test_fsdp_mp_policy_grad_reduce_in_fp32_overrides_dtypes(self):
+ """Test that grad_reduce_in_fp32=True forces main_grads and grad_comm to fp32."""
+ if not is_torch_min_version("2.4.0"):
+ pytest.skip("Megatron FSDP requires torch >= 2.4.0")
+
+ fsdp_model = self._build_fsdp_model(
+ grad_reduce_in_fp32=True,
+ main_params_dtype=torch.bfloat16,
+ main_grads_dtype=torch.bfloat16,
+ grad_comm_dtype=torch.float16,
+ )
+ assert fsdp_model.mp_policy.main_params_dtype == torch.bfloat16
+ assert fsdp_model.mp_policy.main_grads_dtype == torch.float32
+ assert fsdp_model.mp_policy.grad_comm_dtype == torch.float32
+
+ def test_fsdp_mp_policy_grad_reduce_in_fp32_disabled_preserves_dtypes(self):
+ """Test that grad_reduce_in_fp32=False preserves the user-specified grads/comm dtypes."""
+ if not is_torch_min_version("2.4.0"):
+ pytest.skip("Megatron FSDP requires torch >= 2.4.0")
+
+ fsdp_model = self._build_fsdp_model(
+ grad_reduce_in_fp32=False,
+ main_params_dtype=torch.bfloat16,
+ main_grads_dtype=torch.bfloat16,
+ grad_comm_dtype=torch.float16,
+ )
+ assert fsdp_model.mp_policy.main_params_dtype == torch.bfloat16
+ assert fsdp_model.mp_policy.main_grads_dtype == torch.bfloat16
+ assert fsdp_model.mp_policy.grad_comm_dtype == torch.float16
+
@pytest.mark.skipif(
version.parse(torch.__version__) < version.parse('2.3.0'),
reason="Device mesh feature requires PyTorch 2.3 or later",
diff --git a/tests/unit_tests/distributed/megatron_fsdp/test_mfsdp_dcp_ckpt.py b/tests/unit_tests/distributed/megatron_fsdp/test_mfsdp_dcp_ckpt.py
index 52f7d1cb423..40c6ad731d1 100644
--- a/tests/unit_tests/distributed/megatron_fsdp/test_mfsdp_dcp_ckpt.py
+++ b/tests/unit_tests/distributed/megatron_fsdp/test_mfsdp_dcp_ckpt.py
@@ -250,14 +250,14 @@ def test_dcp_checkpoint_save_and_load(
"""
from torch.distributed.tensor import DTensor
- from megatron.core.distributed.fsdp.src.megatron_fsdp.fully_shard import fully_shard
+ from megatron.core.distributed.fsdp.src.megatron_fsdp import (
+ MixedPrecisionPolicy,
+ fully_shard,
+ )
# Skip tests.
if outer_shard_strategy == OPTIM and shard_strategy != OPTIM_GRADS_PARAMS:
- # FIXME(@shjwudp, @cspades): This is an unexpected lack of support.
- # [default0]:FAILED tests/unit_tests/distributed/test_mfsdp_fully_shard.py
- # [False-True-True-True-mesh_dim_config0-optim-optim-cnn]
- # [False-True-True-True-mesh_dim_config0-optim-optim_grads-cnn]
+ # TODO(@shjwudp, @cspades): Requires various modifications to support.
pytest.skip(
f"dp_outer sharding strategy {outer_shard_strategy} requires "
"zero_dp_strategy to be full-sharded ('optim_grads_params', 3)."
@@ -288,15 +288,16 @@ def test_dcp_checkpoint_save_and_load(
module=toy_model,
optimizer=toy_adam,
device_mesh=device_mesh,
- dp_shard_dim=DP_SHARD_CP,
+ dp_shard_dim=DP_SHARD_CP if mesh_dim_config[2] > 1 else DP_SHARD,
dp_outer_dim=DP_OUTER,
tp_dim=TP,
hybrid_fsdp_group=device_mesh[HSDP].get_group(),
fsdp_unit_modules=fsdp_unit_modules,
zero_dp_strategy=shard_strategy,
outer_dp_sharding_strategy=outer_shard_strategy,
- preserve_fp32_weights=True,
- grad_reduce_in_fp32=True,
+ mixed_precision_policy=MixedPrecisionPolicy(
+ main_params_dtype=torch.float32, main_grads_dtype=torch.float32
+ ),
init_model_with_meta_device=False,
sync_model_each_microbatch=True,
)
@@ -333,11 +334,18 @@ def test_dcp_checkpoint_save_and_load(
pre_output = model(toy_input, toy_input)
pre_save_loss = mse_loss(pre_output, toy_target).item()
- # Save deep copy of the model state dictionary before checkpointing.
- s1 = deepcopy(model.state_dict())
-
- # Save deep copy of the optimizer state dictionary before checkpointing.
- o1 = deepcopy(optimizer.state_dict())
+ # Save deep copy of the model and optimizer state before checkpointing.
+ # NOTE(@cspades): deepcopy has issues with DTensors. Just clone().
+ s1 = {}
+ for key, val in model.state_dict().items():
+ s1[key] = val.clone()
+ optim_state_dict = optimizer.state_dict()
+ o1 = {"state": {}}
+ for idx, state in optim_state_dict["state"].items():
+ o1_state = o1["state"].setdefault(idx, {})
+ for key, val in state.items():
+ o1_state[key] = val.clone()
+ o1["param_groups"] = deepcopy(optim_state_dict["param_groups"])
"""
MODEL CHECKPOINT SAVE
@@ -368,15 +376,16 @@ def test_dcp_checkpoint_save_and_load(
module=toy_model,
optimizer=toy_adam,
device_mesh=device_mesh,
- dp_shard_dim=DP_SHARD_CP,
+ dp_shard_dim=DP_SHARD_CP if mesh_dim_config[2] > 1 else DP_SHARD,
dp_outer_dim=DP_OUTER,
tp_dim=TP,
hybrid_fsdp_group=device_mesh[HSDP].get_group(),
fsdp_unit_modules=fsdp_unit_modules,
zero_dp_strategy=shard_strategy,
outer_dp_sharding_strategy=outer_shard_strategy,
- preserve_fp32_weights=True,
- grad_reduce_in_fp32=True,
+ mixed_precision_policy=MixedPrecisionPolicy(
+ main_params_dtype=torch.float32, main_grads_dtype=torch.float32
+ ),
init_model_with_meta_device=False,
sync_model_each_microbatch=True,
)
@@ -386,13 +395,13 @@ def test_dcp_checkpoint_save_and_load(
torch.distributed.checkpoint.load(state_dict=ckpt_state_dict, checkpoint_id=str(CKPT_DIR))
model.load_state_dict(ckpt_state_dict["model"], strict=False)
optimizer.load_state_dict(ckpt_state_dict["optimizer"])
- s2 = deepcopy(model.state_dict())
- o2 = deepcopy(optimizer.state_dict())
"""
MODEL CHECKPOINT STATE DICT VALIDATION
"""
# Compare pre-save and post-load model state dictionaries.
+ s2 = model.state_dict()
+ nonempty_model_state = False
for key in s1.keys() | s2.keys():
v1 = s1.get(key, None)
if isinstance(v1, DTensor):
@@ -407,14 +416,17 @@ def test_dcp_checkpoint_save_and_load(
v1.shape == v2.shape
), f"[Checkpoint Param {key} Shape Mismatch] {v1.shape} != {v2.shape}"
assert torch.allclose(v1, v2), f"[Checkpoint Param {key} Value Mismatch] {v1} != {v2}"
+ nonempty_model_state = True
# Compare pre-save and post-load optimizer state dictionaries.
+ o2 = optimizer.state_dict()
+ nonempty_optim_state = False
for param_id in o1["state"].keys() | o2["state"].keys():
param_state_1 = o1["state"].get(param_id, None)
param_state_2 = o2["state"].get(param_id, None)
assert (
param_state_1 is not None and param_state_2 is not None
- ), f"[{param_id} Not Found] Original Param State: {param_state_1} | Checkpoint Param State: {param_state_2}"
+ ), f"[{param_id} Not Found] Original Optim State: {param_state_1} | Checkpoint Optim State: {param_state_2}"
for key in param_state_1.keys() | param_state_2.keys():
v1 = param_state_1.get(key, None)
if isinstance(v1, DTensor):
@@ -424,23 +436,31 @@ def test_dcp_checkpoint_save_and_load(
v2 = v2.to_local()
assert (
v1 is not None and v2 is not None
- ), f"[{param_id} {key} Not Found] Original Optimizer State: {v1} | Checkpoint Optimizer State: {v2}"
+ ), f"[{param_id} {key} Not Found] Original Optim State: {v1} | Checkpoint Optim State: {v2}"
assert (
v1.shape == v2.shape
- ), f"[Optimizer State {param_id} {key} Shape Mismatch] {v1.shape} != {v2.shape}"
+ ), f"[Optim State {param_id} {key} Shape Mismatch] {v1.shape} != {v2.shape}"
assert torch.allclose(
v1, v2
- ), f"[Optimizer State {param_id} {key} Value Mismatch] {v1} != {v2}"
+ ), f"[Optim State {param_id} {key} Value Mismatch] {v1} != {v2}"
+ nonempty_optim_state = True # Optimizer state depends on wgrad, verify this!
assert len(o1["param_groups"]) == len(
o2["param_groups"]
- ), f"[Optimizer State Param Groups Length Mismatch] {o1['param_groups']} != {o2['param_groups']}"
+ ), f"[Optim State Param Groups Length Mismatch] {o1['param_groups']} != {o2['param_groups']}"
for i in range(len(o2["param_groups"])):
for key in o1["param_groups"][i].keys():
v1 = o1["param_groups"][i][key]
v2 = o2["param_groups"][i][key]
- assert (
- v1 == v2
- ), f"[Optimizer State Param Group {i} {key} Value Mismatch] {v1} != {v2}"
+ assert v1 == v2, f"[Optim State Param Group {i} {key} Value Mismatch] {v1} != {v2}"
+
+ # Validate that at least 1 rank has a non-empty model and optimizer state.
+ # It is very possible that some ranks have completely empty state!
+ global_nonempty_model_state = [False] * torch.distributed.get_world_size()
+ torch.distributed.all_gather_object(global_nonempty_model_state, nonempty_model_state)
+ assert any(global_nonempty_model_state), "All ranks had an empty model state!"
+ global_nonempty_optim_state = [False] * torch.distributed.get_world_size()
+ torch.distributed.all_gather_object(global_nonempty_optim_state, nonempty_optim_state)
+ assert any(global_nonempty_optim_state), "All ranks had an empty optimizer state!"
"""
MODEL CHECKPOINT FORWARD PASS VALIDATION
diff --git a/tests/unit_tests/distributed/megatron_fsdp/test_mfsdp_fully_shard.py b/tests/unit_tests/distributed/megatron_fsdp/test_mfsdp_fully_shard.py
index eed9c1320f0..42ee657d338 100644
--- a/tests/unit_tests/distributed/megatron_fsdp/test_mfsdp_fully_shard.py
+++ b/tests/unit_tests/distributed/megatron_fsdp/test_mfsdp_fully_shard.py
@@ -4,6 +4,7 @@
import shutil
from contextlib import nullcontext
from copy import deepcopy
+from itertools import product
from pathlib import Path
import pytest
@@ -272,8 +273,11 @@ def teardown_class(cls):
)
@pytest.mark.parametrize("model_type", [CNN, TRANSFORMER, TE_TRANSFORMER])
@pytest.mark.parametrize(
- "dp_shard_strategy", [NO_SHARD, OPTIM, OPTIM_GRADS, OPTIM_GRADS_PARAMS]
+ # Sharding strategy for optimizer state, gradients, and parameters.
+ "dp_shard_strategy",
+ [NO_SHARD, OPTIM, OPTIM_GRADS, OPTIM_GRADS_PARAMS],
)
+ # Test FSDP, HSDP, and HFSDP.
@pytest.mark.parametrize("dp_outer_strategy", [None, NO_SHARD, OPTIM])
@pytest.mark.parametrize(
"mesh_dim_config",
@@ -312,7 +316,10 @@ def test_fully_shard(
don't add any new parameters unless absolutely necessary,
or if some combinations can be flattened or simplified.
"""
- from megatron.core.distributed.fsdp.src.megatron_fsdp.fully_shard import fully_shard
+ from megatron.core.distributed.fsdp.src.megatron_fsdp import (
+ MixedPrecisionPolicy,
+ fully_shard,
+ )
preserve_fp32_weights = common_args["preserve_fp32_weights"]
init_model_with_meta_device = common_args["init_model_with_meta_device"]
@@ -324,16 +331,12 @@ def test_fully_shard(
"Meta device initialization (init_model_with_meta_device=True) is not "
"supported or necessary for the 'no_shard' / 0 sharding strategy."
)
- elif dp_outer_strategy == OPTIM:
- if dp_shard_strategy != OPTIM_GRADS_PARAMS:
- # TODO(@shjwudp, @cspades): Requires various modifications to support.
- # [default0]:FAILED tests/unit_tests/distributed/test_mfsdp_fully_shard.py
- # [False-True-True-True-mesh_dim_config0-optim-optim-cnn]
- # [False-True-True-True-mesh_dim_config0-optim-optim_grads-cnn]
- pytest.skip(
- f"dp_outer sharding strategy {dp_outer_strategy} requires "
- "zero_dp_strategy to be full-sharded ('optim_grads_params', 3)."
- )
+ elif dp_outer_strategy == OPTIM and dp_shard_strategy != OPTIM_GRADS_PARAMS:
+ # TODO(@shjwudp, @cspades): Requires various modifications to support.
+ pytest.skip(
+ f"dp_outer sharding strategy {dp_outer_strategy} requires "
+ "zero_dp_strategy to be full-sharded ('optim_grads_params', 3)."
+ )
# Construct device mesh.
device_mesh = build_distributed_environment(mesh_dim_config)
@@ -347,7 +350,7 @@ def test_fully_shard(
module=toy_model,
optimizer=toy_adam,
device_mesh=device_mesh,
- dp_shard_dim=DP_SHARD_CP,
+ dp_shard_dim=DP_SHARD_CP if mesh_dim_config[2] > 1 else DP_SHARD,
dp_outer_dim=DP_OUTER if dp_outer_strategy is not None else None,
tp_dim=TP,
hybrid_fsdp_group=(
@@ -356,11 +359,14 @@ def test_fully_shard(
fsdp_unit_modules=fsdp_unit_modules,
zero_dp_strategy=dp_shard_strategy,
outer_dp_sharding_strategy=(
- dp_outer_strategy if dp_outer_strategy is not None else "no_shard"
+ dp_outer_strategy if dp_outer_strategy is not None else NO_SHARD
+ ),
+ mixed_precision_policy=MixedPrecisionPolicy(
+ main_params_dtype=torch.float32 if preserve_fp32_weights else None,
+ main_grads_dtype=None,
),
- preserve_fp32_weights=preserve_fp32_weights,
- grad_reduce_in_fp32=False,
init_model_with_meta_device=init_model_with_meta_device,
+ report_nan_in_param_grad=True,
)
model = torch.compile(model) if torch_compile else model
@@ -371,6 +377,8 @@ def test_fully_shard(
for step in range(NUM_STEPS):
# Synchronize model parameters and gradients on the final training step only.
if step == NUM_STEPS - 1:
+ # Triggers all-reduce / reduce-scatter across DP-Outer, and
+ # synchronizes / concludes the gradient accumulation cycle.
model.set_model_auto_sync(True)
else:
model.set_model_auto_sync(False)
@@ -392,11 +400,12 @@ def test_fully_shard(
isinstance(p.grad, torch.Tensor) and p.grad.to_local().count_nonzero().item() > 0
for p in model.parameters()
)
- sharding_group = (
- device_mesh[HSDP].get_group()
- if dp_outer_strategy == OPTIM
- else device_mesh[DP_SHARD_CP].get_group()
- )
+ sharding_dim = DP_SHARD
+ if dp_outer_strategy == OPTIM:
+ sharding_dim = HSDP
+ elif mesh_dim_config[2] > 1:
+ sharding_dim = DP_SHARD_CP
+ sharding_group = device_mesh[sharding_dim].get_group()
if dp_shard_strategy != NO_SHARD:
# Because of uneven sharding, we need to gather the result from all ranks
# to verify if any gradients exist or not at this step of training.
@@ -429,7 +438,7 @@ def test_fully_shard_ez(self, shard_strategy):
"""
Test fully_shard(device_mesh=None). Represents the easiest entrypoint to Megatron-FSDP.
"""
- from megatron.core.distributed.fsdp.src.megatron_fsdp.fully_shard import (
+ from megatron.core.distributed.fsdp.src.megatron_fsdp import (
fully_shard_model,
fully_shard_optimizer,
)
@@ -450,7 +459,7 @@ def test_fully_shard_ez(self, shard_strategy):
toy_input = torch.randn(1, DIM_SIZE, DIM_SIZE).to("cuda")
toy_target = torch.randn(1, DIM_SIZE, DIM_SIZE).to("cuda")
- for step in range(NUM_STEPS):
+ for _ in range(NUM_STEPS):
# Forward pass.
output = mfsdp_model(toy_input, toy_input)
@@ -486,7 +495,8 @@ def test_fully_shard_te_quantized(self, init_model_with_meta_device, te_recipe):
if not mxfp8_supported:
pytest.skip(mxfp8_skip_reason)
- from megatron.core.distributed.fsdp.src.megatron_fsdp.fully_shard import (
+ from megatron.core.distributed.fsdp.src.megatron_fsdp import (
+ MixedPrecisionPolicy,
fully_shard_model,
fully_shard_optimizer,
)
@@ -538,10 +548,13 @@ def test_fully_shard_te_quantized(self, init_model_with_meta_device, te_recipe):
# its own row-wise and col-wise (transpose) buffer management
# schedule that is natively managed by Megatron-FSDP.
keep_fp8_transpose_cache=True,
- # Required for FP8 parameters. The optimizer state (and gradients)
- # are never quantized, as TE produces high-precision wgrad and
- # dgrad from FP8 weights and activations. Already defaults to True.
- preserve_fp32_weights=True,
+ mixed_precision_policy=MixedPrecisionPolicy(
+ # Required for FP8 parameters. The optimizer state (and gradients)
+ # are never quantized, as TE produces high-precision wgrad and
+ # dgrad from FP8 weights and activations. Defaults to FP32.
+ main_params_dtype=torch.float32
+ ),
+ report_nan_in_param_grad=True,
)
# Initialize the distributed optimizer on the MegatronFSDP model.
@@ -571,3 +584,136 @@ def test_fully_shard_te_quantized(self, init_model_with_meta_device, te_recipe):
# Optimizer step.
optimizer.step()
optimizer.zero_grad()
+
+ @pytest.mark.skipif(
+ version.parse(torch.__version__) < version.parse('2.4.0'),
+ reason="Requires DTensor and DeviceMesh support in (approximately) PyTorch 2.4.0 or later.",
+ )
+ # Test non-FP8 and FP8 parameters.
+ @pytest.mark.parametrize("model_type", [TRANSFORMER, TE_TRANSFORMER])
+ @pytest.mark.parametrize(
+ # Test gradient all-reduce, reduce-scatter, and param all-gather.
+ "dp_shard_strategy",
+ [OPTIM, OPTIM_GRADS, OPTIM_GRADS_PARAMS],
+ )
+ # Test HSDP and HFSDP only. (FSDP collectives are a subset of HSDP.)
+ @pytest.mark.parametrize("dp_outer_strategy", [NO_SHARD, OPTIM])
+ @pytest.mark.parametrize("custom_main_params_dtype", [None, torch.float64])
+ @pytest.mark.parametrize("custom_main_grads_dtype", [None, torch.float32])
+ def test_fully_shard_custom_dtype(
+ self,
+ model_type,
+ dp_shard_strategy,
+ dp_outer_strategy,
+ custom_main_params_dtype,
+ custom_main_grads_dtype,
+ ):
+ """
+ Test custom data-types for gather and reduce communications.
+ """
+ from megatron.core.distributed.fsdp.src.megatron_fsdp import (
+ MixedPrecisionPolicy,
+ fully_shard_model,
+ fully_shard_optimizer,
+ )
+
+ if dp_outer_strategy == OPTIM and dp_shard_strategy != OPTIM_GRADS_PARAMS:
+ pytest.skip(
+ f"dp_outer sharding strategy {dp_outer_strategy} requires "
+ "zero_dp_strategy to be full-sharded ('optim_grads_params', 3)."
+ )
+ if model_type == TE_TRANSFORMER and custom_main_params_dtype is None:
+ pytest.skip(
+ f"TransformerEngine FP8 all-gather requires a main parameter buffer for FSDP."
+ )
+
+ # Construct device mesh with DP-Outer=2 and DP-Shard=4.
+ device_mesh = build_distributed_environment((2, 4, 1, 1))
+
+ # Construct toy model.
+ if model_type == TE_TRANSFORMER:
+ # Use FP8 model parameters to test data-type customization.
+ te_quant_recipe = te.common.recipe.DelayedScaling()
+ with te.pytorch.quantized_model_init(
+ recipe=te_quant_recipe,
+ # Needed for FP8 parameters with Megatron-FSDP.
+ preserve_high_precision_init_val=True,
+ ):
+ toy_model = ToyTETransformer(
+ model_dim=64,
+ num_heads=2,
+ num_layers=2,
+ output_dim=64,
+ fuse_qkv_params=True,
+ params_dtype=torch.bfloat16,
+ device="meta",
+ )
+ fsdp_unit_modules = [te.pytorch.TransformerLayer, te.pytorch.Linear]
+ else:
+ toy_model, fsdp_unit_modules = build_toy_model(model_type, True)
+
+ # Fully-shard the model.
+ mfsdp_model = fully_shard_model(
+ module=toy_model,
+ device_mesh=device_mesh,
+ dp_shard_dim=DP_SHARD,
+ dp_outer_dim=DP_OUTER,
+ tp_dim=TP,
+ hybrid_fsdp_group=device_mesh[HSDP].get_group(),
+ fsdp_unit_modules=fsdp_unit_modules,
+ zero_dp_strategy=dp_shard_strategy,
+ outer_dp_sharding_strategy=dp_outer_strategy,
+ mixed_precision_policy=MixedPrecisionPolicy(
+ main_params_dtype=custom_main_params_dtype,
+ main_grads_dtype=custom_main_grads_dtype,
+ grad_comm_dtype=None,
+ ),
+ init_model_with_meta_device=True,
+ report_nan_in_param_grad=True,
+ )
+ # Verify that the main weight and main gradient buffers have the correct dtype.
+ main_weight_buffer = getattr(
+ mfsdp_model.param_and_grad_buffer.parameter_groups[0], "main_weight_buffer", None
+ )
+ if main_weight_buffer is not None:
+ assert main_weight_buffer.data.dtype == custom_main_params_dtype
+ if custom_main_grads_dtype is not None:
+ assert (
+ mfsdp_model.param_and_grad_buffer.parameter_groups[0].main_grad_buffer.data.dtype
+ == custom_main_grads_dtype
+ )
+
+ # Initialize the distributed optimizer on the MegatronFSDP model.
+ toy_adam = Adam(params=mfsdp_model.parameters(), lr=0.001)
+ optimizer = fully_shard_optimizer(optimizer=toy_adam)
+
+ # Mock input and target.
+ if model_type == TE_TRANSFORMER:
+ toy_input = torch.randn(16, 64, 64, dtype=torch.bfloat16).to("cuda")
+ toy_target = torch.randn(16, 64, 64, dtype=torch.bfloat16).to("cuda")
+ else:
+ toy_input = torch.randn(1, DIM_SIZE, DIM_SIZE).to("cuda")
+ toy_target = torch.randn(1, DIM_SIZE, DIM_SIZE).to("cuda")
+
+ # Test a different mixed-precision policy every step.
+ for grad_comm_dtype in [None, torch.float16]:
+ # Set up mixed-precision context manager to change policy every step.
+ with mfsdp_model.mixed_precision_context(
+ MixedPrecisionPolicy(grad_comm_dtype=grad_comm_dtype)
+ ):
+ # Forward pass.
+ if model_type == TE_TRANSFORMER:
+ with te.pytorch.autocast(recipe=te_quant_recipe):
+ output = mfsdp_model(toy_input)
+ elif model_type == TRANSFORMER:
+ output = mfsdp_model(toy_input, toy_input)
+
+ # Loss.
+ loss = mse_loss(output, toy_target)
+
+ # Backward pass.
+ loss.backward()
+
+ # Optimizer step syncs gradient communication.
+ optimizer.step()
+ optimizer.zero_grad()
diff --git a/tests/unit_tests/distributed/test_grad_sync_with_expert_parallel.py b/tests/unit_tests/distributed/test_grad_sync_with_expert_parallel.py
index e83f7142284..15c479774c6 100644
--- a/tests/unit_tests/distributed/test_grad_sync_with_expert_parallel.py
+++ b/tests/unit_tests/distributed/test_grad_sync_with_expert_parallel.py
@@ -8,11 +8,12 @@
from megatron.core import parallel_state
from megatron.core.distributed import DistributedDataParallel, DistributedDataParallelConfig
-from megatron.core.distributed.param_and_grad_buffer import partition_buckets
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
+from megatron.core.models.gpt.gpt_layer_specs import (
+ get_gpt_layer_with_transformer_engine_submodules,
+)
from megatron.core.transformer import TransformerConfig
from megatron.core.transformer.moe.moe_layer import MoELayer
-from tests.unit_tests.test_utilities import TestModel, Utils
+from tests.unit_tests.test_utilities import Utils
class TestMoEModel(torch.nn.Module):
@@ -41,15 +42,13 @@ def __init__(
params_dtype=torch.bfloat16,
add_bias_linear=False,
)
- transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec(
+ submodules = get_gpt_layer_with_transformer_engine_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=moe_grouped_gemm
)
super().__init__()
self.layers = torch.nn.ModuleList(
[
- MoELayer(
- transformer_config, transformer_layer_spec.submodules.mlp.submodules
- ).cuda()
+ MoELayer(transformer_config, submodules.mlp.submodules).cuda()
for _ in range(num_layers)
]
)
diff --git a/tests/unit_tests/distributed/test_param_and_grad_buffer.py b/tests/unit_tests/distributed/test_param_and_grad_buffer.py
index 609b2cc5a71..5ea6640f4e1 100644
--- a/tests/unit_tests/distributed/test_param_and_grad_buffer.py
+++ b/tests/unit_tests/distributed/test_param_and_grad_buffer.py
@@ -26,14 +26,17 @@ def get_model_and_buffers(
overlap_grad_reduce: bool,
average_in_collective: bool,
num_distributed_optimizer_instances: int = 1,
+ grad_reduce_in_fp32: bool = True,
+ param_name_patterns_for_fp32_local_accumulation: tuple = (),
):
ddp_config = DistributedDataParallelConfig(
- grad_reduce_in_fp32=True,
+ grad_reduce_in_fp32=grad_reduce_in_fp32,
use_distributed_optimizer=use_distributed_optimizer,
overlap_grad_reduce=overlap_grad_reduce,
bucket_size=bucket_size,
average_in_collective=average_in_collective,
num_distributed_optimizer_instances=num_distributed_optimizer_instances,
+ param_name_patterns_for_fp32_local_accumulation=param_name_patterns_for_fp32_local_accumulation,
)
model = TestModel(
input_dim=input_dim,
@@ -163,6 +166,59 @@ def _pad_param_if_needed(numel_unpadded):
Utils.destroy_model_parallel()
+def test_param_to_index_alignment_with_padding():
+ """Ensure bucket-local param offsets honor padding when DistOpt pads params."""
+ Utils.initialize_model_parallel()
+
+ # With input_dim=4, output_dim=4:
+ # - weight: 4*4 = 16 elements
+ # - bias: 4 elements
+ # Since 16 % 64 != 0, the bias must be padded away from the weight,
+ # making padding observable.
+ input_dim = 4
+ output_dim = 4
+ model, param_and_grad_buffer, _ = get_model_and_buffers(
+ input_dim=input_dim,
+ output_dim=output_dim,
+ num_layers=1,
+ bias=True,
+ shared_embedding=False,
+ bucket_size=None, # single bucket
+ use_distributed_optimizer=True, # enforces 64-element alignment
+ overlap_grad_reduce=True,
+ average_in_collective=False,
+ )
+
+ bucket = param_and_grad_buffer.buckets[0]
+ naive_offset = 0
+ padding_observed = False
+
+ for param in bucket.params_list:
+ global_start, global_end, _ = param_and_grad_buffer.param_index_map[param]
+ expected_local_start = global_start - bucket.offset
+ expected_local_end = global_end - bucket.offset
+ local_start, local_end = bucket.param_to_index[param]
+
+ # param_to_index should match the padded offsets used in the global buffer.
+ assert (local_start, local_end) == (expected_local_start, expected_local_end)
+
+ # At least one param should have been padded relative to naive packing.
+ if local_start != naive_offset:
+ padding_observed = True
+ naive_offset = local_end
+
+ # Verify the slice retrieved via param_to_index matches param.data view.
+ param_slice = bucket.param_data.view(-1)[local_start:local_end]
+ torch.testing.assert_close(param_slice, param.data.view(-1))
+
+ assert padding_observed, (
+ "Expected padding to be applied between params. "
+ "Ensure model dimensions are chosen such that param sizes are not multiples of 64."
+ )
+
+ Utils.destroy_model_parallel()
+
+
@pytest.mark.parametrize("use_distributed_optimizer", [False, True])
@pytest.mark.parametrize("overlap_grad_reduce", [False, True])
@pytest.mark.parametrize("average_in_collective", [False, True])
@@ -320,3 +376,263 @@ def test_force_all_reduce_uses_correct_collective(force_all_reduce: bool):
), "Expected all_reduce NOT to be called when force_all_reduce=False"
Utils.destroy_model_parallel()
+
+
+class TestFreeOverlapBuffers:
+ """Tests for free_overlap_buffers() which releases GPU memory before async checkpoint saves."""
+
+ @staticmethod
+ def _make_model():
+ """Create a DDP-wrapped model with overlap_param_gather enabled."""
+ Utils.initialize_model_parallel()
+ ddp_config = DistributedDataParallelConfig(
+ grad_reduce_in_fp32=True,
+ use_distributed_optimizer=False,
+ overlap_grad_reduce=True,
+ overlap_param_gather=True,
+ bucket_size=None,
+ )
+ module = TestModel(
+ input_dim=32, output_dim=32, num_layers=2, bias=False, shared_embedding=False
+ ).bfloat16()
+ model = DistributedDataParallel(
+ TransformerConfig(num_attention_heads=1, num_layers=1),
+ ddp_config=ddp_config,
+ module=module,
+ )
+ return model
+
+ def test_bucket_group_clears_buffers(self):
+ """free_overlap_buffers on a bucket group should None-out per-bucket layerwise buffers."""
+ model = self._make_model()
+
+ for bg in model.bucket_groups:
+ # Simulate buffers that would be allocated by start_param_sync.
+ for bucket in bg.buckets:
+ bucket.layerwise_gather_list = [torch.empty(8), torch.empty(8)]
+ bucket._layerwise_src_buffer = torch.empty(16)
+
+ bg.free_overlap_buffers()
+
+ for bucket in bg.buckets:
+ assert (
+ bucket.layerwise_gather_list is None
+ ), "layerwise_gather_list should be None after free_overlap_buffers"
+ assert (
+ bucket._layerwise_src_buffer is None
+ ), "_layerwise_src_buffer should be None after free_overlap_buffers"
+
+ Utils.destroy_model_parallel()
+
+ def test_bucket_group_waits_on_pending_handle(self):
+ """free_overlap_buffers should wait() on any pending param_gather_handle."""
+ model = self._make_model()
+
+ for bg in model.bucket_groups:
+ mock_handle = mock.MagicMock()
+ bg.param_gather_handle = mock_handle
+
+ bg.free_overlap_buffers()
+
+ mock_handle.wait.assert_called_once()
+ assert (
+ bg.param_gather_handle is None
+ ), "param_gather_handle should be None after free_overlap_buffers"
+
+ Utils.destroy_model_parallel()
+
+ def test_bucket_group_noop_when_no_buffers(self):
+ """free_overlap_buffers should be safe to call when no buffers are allocated."""
+ model = self._make_model()
+
+ for bg in model.bucket_groups:
+ assert bg.param_gather_handle is None
+ for bucket in bg.buckets:
+ assert bucket.layerwise_gather_list is None
+ assert bucket._layerwise_src_buffer is None
+
+ # Should not raise.
+ bg.free_overlap_buffers()
+
+ Utils.destroy_model_parallel()
+
+ def test_ddp_free_overlap_buffers_delegates(self):
+ """DDP.free_overlap_buffers should call free_overlap_buffers on all bucket groups."""
+ model = self._make_model()
+
+ with mock.patch.object(type(model.bucket_groups[0]), 'free_overlap_buffers') as mock_free:
+ model.free_overlap_buffers()
+ assert mock_free.call_count == len(
+ model.bucket_groups + model.expert_parallel_bucket_groups
+ ), "free_overlap_buffers should be called on every bucket group"
+
+ Utils.destroy_model_parallel()
+
+
+class TestFP32LocalGradAccumulation:
+ """Tests for the FP32 local gradient accumulation feature
+ (param_name_patterns_for_fp32_local_accumulation)."""
+
+ @staticmethod
+ def _make_model(patterns, bucket_size=None):
+ """Create a DDP-wrapped model with FP32 local grad accumulation patterns."""
+ return get_model_and_buffers(
+ input_dim=100,
+ output_dim=100,
+ num_layers=3,
+ bias=True,
+ shared_embedding=False,
+ bucket_size=bucket_size,
+ use_distributed_optimizer=False,
+ overlap_grad_reduce=False,
+ average_in_collective=False,
+ grad_reduce_in_fp32=False,
+ param_name_patterns_for_fp32_local_accumulation=patterns,
+ )
+
+ def test_config_validation_with_grad_reduce_in_fp32(self):
+ """param_name_patterns_for_fp32_local_accumulation and grad_reduce_in_fp32 are
+ mutually exclusive."""
+ with pytest.raises(AssertionError):
+ DistributedDataParallelConfig(
+ grad_reduce_in_fp32=True, param_name_patterns_for_fp32_local_accumulation=('all',)
+ )
+
+ def test_pattern_matching_creates_fp32_main_grad(self):
+ """Params matching patterns should get a float32 main_grad and a
+ main_grad_copy_in_grad_buffer; non-matching params should not."""
+ Utils.initialize_model_parallel()
+ # Match only weight params (not bias).
+ model, buf, _ = self._make_model(patterns=('*.weight',))
+
+ for name, param in model.module.named_parameters():
+ if 'weight' in name:
+ assert param.main_grad.dtype == torch.float32, f"{name} main_grad should be float32"
+ assert hasattr(param, 'main_grad_copy_in_grad_buffer')
+ assert param.main_grad_copy_in_grad_buffer is not None
+ # The copy in grad buffer should be in the buffer's grad dtype (bf16).
+ assert param.main_grad_copy_in_grad_buffer.dtype == buf.grad_dtype
+ else:
+ # Bias params should not be promoted.
+ assert (
+ param.main_grad.dtype == buf.grad_dtype
+ ), f"{name} main_grad should remain in grad_dtype"
+ assert getattr(param, 'main_grad_copy_in_grad_buffer', None) is None
+
+ Utils.destroy_model_parallel()
+
+ def test_all_pattern_matches_every_param(self):
+ """The 'all' pattern should match every parameter."""
+ Utils.initialize_model_parallel()
+ model, buf, _ = self._make_model(patterns=('all',))
+
+ for name, param in model.module.named_parameters():
+ assert (
+ param.main_grad.dtype == torch.float32
+ ), f"{name} main_grad should be float32 with 'all' pattern"
+ assert getattr(param, 'main_grad_copy_in_grad_buffer', None) is not None
+
+ Utils.destroy_model_parallel()
+
+ def test_bucket_tracks_params_with_extra_main_grads(self):
+ """Each bucket's params_with_extra_main_grads should contain exactly
+ the params that matched the patterns."""
+ Utils.initialize_model_parallel()
+ model, buf, _ = self._make_model(patterns=('*.weight',))
+
+ promoted_params = set()
+ for name, param in model.module.named_parameters():
+ if 'weight' in name:
+ promoted_params.add(param)
+
+ bucket_promoted = set()
+ for bucket in buf.buckets:
+ for param in bucket.params_with_extra_main_grads:
+ bucket_promoted.add(param)
+ # Every param in params_with_extra_main_grads should also be in bucket.params.
+ assert bucket.params_with_extra_main_grads == [] or set(
+ bucket.params_with_extra_main_grads
+ ).issubset(bucket.params)
+
+ assert (
+ bucket_promoted == promoted_params
+ ), "Bucket-tracked promoted params should match the set of pattern-matched params"
+
+ Utils.destroy_model_parallel()
+
+ def test_no_patterns_means_no_extra_main_grads(self):
+ """With no patterns, no params should have extra main_grads."""
+ Utils.initialize_model_parallel()
+ _, buf, _ = self._make_model(patterns=())
+
+ assert len(buf.extra_main_grads) == 0
+ for bucket in buf.buckets:
+ assert len(bucket.params_with_extra_main_grads) == 0
+
+ Utils.destroy_model_parallel()
+
+ def test_reset_zeros_extra_main_grads(self):
+ """reset() should zero out both grad_data and all extra main_grads."""
+ Utils.initialize_model_parallel()
+ _, buf, _ = self._make_model(patterns=('all',))
+
+ # Fill extra main_grads and grad_data with non-zero values.
+ buf.grad_data.fill_(1.0)
+ for grad in buf.extra_main_grads:
+ grad.fill_(42.0)
+
+ buf.reset()
+
+ assert torch.all(buf.grad_data == 0), "grad_data should be zeroed after reset"
+ for grad in buf.extra_main_grads:
+ assert torch.all(grad == 0), "extra main_grads should be zeroed after reset"
+
+ Utils.destroy_model_parallel()
+
+ def test_scale_gradients_scales_extra_main_grads(self):
+ """scale_gradients() should scale both grad_data and extra main_grads."""
+ Utils.initialize_model_parallel()
+ _, buf, _ = self._make_model(patterns=('all',))
+
+ buf.grad_data.fill_(2.0)
+ for grad in buf.extra_main_grads:
+ grad.fill_(4.0)
+
+ buf.scale_gradients(0.5)
+
+ assert torch.allclose(
+ buf.grad_data, torch.tensor(1.0, dtype=buf.grad_data.dtype)
+ ), "grad_data should be scaled"
+ for grad in buf.extra_main_grads:
+ assert torch.allclose(
+ grad, torch.tensor(2.0, dtype=grad.dtype)
+ ), "extra main_grads should be scaled"
+
+ Utils.destroy_model_parallel()
+
+ def test_grad_sync_copies_to_and_from_comm_buffer(self):
+ """During grad sync, values in FP32 main_grad should be copied to the comm buffer
+ before the collective, and the reduced result should be copied back afterward."""
+ Utils.initialize_model_parallel()
+ model, buf, bucket_groups = self._make_model(patterns=('all',))
+
+ # Simulate accumulated gradients in FP32 main_grad.
+ for param in model.parameters():
+ param.main_grad.fill_(1.0)
+
+ # Run grad sync (non-overlapped, so finish_grad_sync triggers start + wait).
+ model.finish_grad_sync()
+
+ # After sync, main_grad should contain the reduced result (not the original 1.0,
+ # since the collective may have scaled / averaged). The key invariant is that
+ # main_grad should equal main_grad_copy_in_grad_buffer (the comm buffer slice)
+ # after the copy-back.
+ for param in model.parameters():
+ if getattr(param, 'main_grad_copy_in_grad_buffer', None) is not None:
+ torch.testing.assert_close(
+ param.main_grad,
+ param.main_grad_copy_in_grad_buffer.float(),
+ msg="main_grad should equal comm buffer after grad sync copy-back",
+ )
+
+ Utils.destroy_model_parallel()
diff --git a/tests/unit_tests/export/trtllm/test_distributed_fp8.py b/tests/unit_tests/export/trtllm/test_distributed_fp8.py
index cf47a864108..09732e98263 100644
--- a/tests/unit_tests/export/trtllm/test_distributed_fp8.py
+++ b/tests/unit_tests/export/trtllm/test_distributed_fp8.py
@@ -1,3 +1,5 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
from functools import partial
import pytest
@@ -15,8 +17,8 @@
from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
from megatron.core.models.gpt.gpt_model import GPTModel
from megatron.core.pipeline_parallel.schedules import get_forward_backward_func
+from megatron.core.tokenizers import MegatronTokenizer
from megatron.core.transformer.transformer_config import TransformerConfig
-from megatron.training.tokenizer.tokenizer import _NullTokenizer
from tests.unit_tests.test_utilities import Utils
VOCAB_SIZE = 256
@@ -61,13 +63,16 @@ def _get_train_data_iterator():
else:
compile_helpers()
+ tokenizer = MegatronTokenizer.from_pretrained(
+ metadata_path={"library": "null-text"}, vocab_size=50
+ )
config = GPTDatasetConfig(
random_seed=0,
sequence_length=SEQUENCE_LENGTH,
reset_position_ids=False,
reset_attention_mask=False,
eod_mask_loss=False,
- tokenizer=_NullTokenizer(vocab_size=50),
+ tokenizer=tokenizer,
mid_level_dataset_surplus=0.005,
)
diff --git a/tests/unit_tests/export/trtllm/test_single_device_fp8.py b/tests/unit_tests/export/trtllm/test_single_device_fp8.py
index 04bbfdb127d..2e0537894de 100644
--- a/tests/unit_tests/export/trtllm/test_single_device_fp8.py
+++ b/tests/unit_tests/export/trtllm/test_single_device_fp8.py
@@ -1,3 +1,5 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
from functools import partial
import pytest
@@ -15,8 +17,8 @@
from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
from megatron.core.models.gpt.gpt_model import GPTModel
from megatron.core.pipeline_parallel.schedules import get_forward_backward_func
+from megatron.core.tokenizers import MegatronTokenizer
from megatron.core.transformer.transformer_config import TransformerConfig
-from megatron.training.tokenizer.tokenizer import _NullTokenizer
from tests.unit_tests.test_utilities import Utils
SEQUENCE_LENGTH = 64
@@ -58,13 +60,16 @@ def _get_train_data_iterator():
else:
compile_helpers()
+ tokenizer = MegatronTokenizer.from_pretrained(
+ metadata_path={"library": "null-text"}, vocab_size=50
+ )
config = GPTDatasetConfig(
random_seed=0,
sequence_length=SEQUENCE_LENGTH,
reset_position_ids=False,
reset_attention_mask=False,
eod_mask_loss=False,
- tokenizer=_NullTokenizer(vocab_size=50),
+ tokenizer=tokenizer,
mid_level_dataset_surplus=0.005,
)
diff --git a/tests/unit_tests/extension/test_kitchen_sdpa.py b/tests/unit_tests/extension/test_kitchen_sdpa.py
index 6875b005c0b..0660bc226ba 100644
--- a/tests/unit_tests/extension/test_kitchen_sdpa.py
+++ b/tests/unit_tests/extension/test_kitchen_sdpa.py
@@ -8,7 +8,7 @@
import torch
from megatron.core import parallel_state
-from megatron.core.extensions.transformer_engine import TEDotProductAttention
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.process_groups_config import ProcessGroupCollection
from megatron.core.quantization.quant_config import RecipeConfig
from megatron.core.quantization.utils import get_quant_config_or_none
@@ -33,18 +33,16 @@
KitchenDotProductAttention = MagicMock()
KitchenFlashAttention = MagicMock()
-try:
- import transformer_engine # type: ignore[import-untyped]
+if HAVE_TE:
from transformer_engine.pytorch.attention import ( # type: ignore[import-untyped]
dot_product_attention,
)
- HAVE_TE = True
-except ImportError:
+ from megatron.core.extensions.transformer_engine import TEDotProductAttention
+else:
from unittest.mock import MagicMock
- HAVE_TE = False
- transformer_engine = MagicMock()
+ TEDotProductAttention = MagicMock()
dot_product_attention = MagicMock()
diff --git a/tests/unit_tests/fusions/test_bias_dropout_fusion.py b/tests/unit_tests/fusions/test_bias_dropout_fusion.py
index 6303a87ba26..f8b23900543 100644
--- a/tests/unit_tests/fusions/test_bias_dropout_fusion.py
+++ b/tests/unit_tests/fusions/test_bias_dropout_fusion.py
@@ -1,7 +1,13 @@
+# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+
import pytest
import torch
-from megatron.core.fusions.fused_bias_dropout import get_bias_dropout_add
+from megatron.core.fusions.fused_bias_dropout import _bias_dropout_add_func, get_bias_dropout_add
+
+# ---------------------------------------------------------------------------
+# Existing test: fused vs. unfused parity (same dtype)
+# ---------------------------------------------------------------------------
@pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16])
@@ -46,3 +52,270 @@ def test_bias_dropout_add(dtype, training):
# In‑place check for inference
assert out_fused.data_ptr() == x_fused.data_ptr()
assert torch.allclose(out_fused, x_fused, **tols)
+
+
+# ============================================================================
+# Tests for fp32 residual connection fix
+# ============================================================================
+#
+# The fix reverses the casting direction in _bias_dropout_add_func: when
+# x is bf16/fp16 and residual is fp32, x (and bias) are upcast to fp32
+# so that the residual stream stays in fp32. The OLD (broken) code did the
+# opposite: it downcast the fp32 residual to match x's dtype, destroying
+# the fp32 residual stream from the very first layer.
+# ============================================================================
+
+
+class TestFp32ResidualPreservation:
+ """Tests that _bias_dropout_add_func preserves fp32 residual dtype."""
+
+ device = "cuda"
+ B, H = 16, 64
+
+ # -- helpers --------------------------------------------------------
+
+ @staticmethod
+ def _reference_bias_dropout_add(x, bias, residual, prob, training):
+ """Manual reference computation in fp32 (no dropout for determinism)."""
+ x_fp32 = x.float()
+ r_fp32 = residual.float()
+ if bias is not None:
+ b_fp32 = bias.float()
+ return r_fp32 + torch.nn.functional.dropout(x_fp32 + b_fp32, p=prob, training=training)
+ return r_fp32 + torch.nn.functional.dropout(x_fp32, p=prob, training=training)
+
+ # -- core: output dtype must follow residual, not x -----------------
+
+ @pytest.mark.parametrize("x_dtype", [torch.bfloat16, torch.float16])
+ @pytest.mark.parametrize("training", [True, False])
+ @pytest.mark.parametrize("has_bias", [True, False])
+ def test_output_dtype_is_residual_dtype(self, x_dtype, training, has_bias):
+ """The output tensor must have the same dtype as the residual (fp32),
+ NOT x's dtype. This is the central invariant of the fix."""
+ x = torch.randn(self.B, self.H, dtype=x_dtype, device=self.device)
+ residual = torch.randn(self.B, self.H, dtype=torch.float32, device=self.device)
+ bias = torch.randn(self.H, dtype=x_dtype, device=self.device) if has_bias else None
+
+ out = _bias_dropout_add_func((x, bias), residual, prob=0.0, training=training)
+
+ assert (
+ out.dtype == torch.float32
+ ), f"Output dtype {out.dtype} must be fp32 (residual dtype), not {x_dtype}"
+
+ # -- numerical correctness of the upcast path ----------------------
+
+ @pytest.mark.parametrize("x_dtype", [torch.bfloat16, torch.float16])
+ @pytest.mark.parametrize("has_bias", [True, False])
+ def test_numerical_correctness_with_fp32_residual(self, x_dtype, has_bias):
+ """Forward result should match a reference computed entirely in fp32."""
+ torch.manual_seed(7)
+ x = torch.randn(self.B, self.H, dtype=x_dtype, device=self.device)
+ residual = torch.randn(self.B, self.H, dtype=torch.float32, device=self.device)
+ bias = torch.randn(self.H, dtype=x_dtype, device=self.device) if has_bias else None
+
+ out = _bias_dropout_add_func((x, bias), residual, prob=0.0, training=True)
+
+ ref = self._reference_bias_dropout_add(x, bias, residual, prob=0.0, training=True)
+ # fp32 tolerance – the only imprecision is the upcast from bf16/fp16
+ assert torch.allclose(
+ out, ref, rtol=1e-5, atol=1e-5
+ ), f"Max diff = {(out - ref).abs().max().item()}"
+
+ # -- backward: gradients flow correctly through the upcast ---------
+
+ @pytest.mark.parametrize("x_dtype", [torch.bfloat16, torch.float16])
+ @pytest.mark.parametrize("has_bias", [True, False])
+ def test_backward_with_fp32_residual(self, x_dtype, has_bias):
+ """Gradients should be computed for x and residual when dtypes differ."""
+ x = torch.randn(self.B, self.H, dtype=x_dtype, device=self.device, requires_grad=True)
+ residual = torch.randn(
+ self.B, self.H, dtype=torch.float32, device=self.device, requires_grad=True
+ )
+ bias = (
+ torch.randn(self.H, dtype=x_dtype, device=self.device, requires_grad=True)
+ if has_bias
+ else None
+ )
+
+ out = _bias_dropout_add_func((x, bias), residual, prob=0.0, training=True)
+ grad_out = torch.randn_like(out)
+ out.backward(grad_out)
+
+ assert x.grad is not None, "x.grad must not be None"
+ assert residual.grad is not None, "residual.grad must not be None"
+ assert x.grad.dtype == x_dtype, f"x.grad dtype should be {x_dtype}"
+ assert residual.grad.dtype == torch.float32, "residual.grad must stay fp32"
+ if has_bias:
+ assert bias.grad is not None, "bias.grad must not be None"
+
+ # -- same dtype: no regression (both fp32 or both bf16) ------------
+
+ @pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16])
+ @pytest.mark.parametrize("has_bias", [True, False])
+ def test_same_dtype_no_regression(self, dtype, has_bias):
+ """When x and residual share the same dtype, behaviour is unchanged."""
+ torch.manual_seed(99)
+ x = torch.randn(self.B, self.H, dtype=dtype, device=self.device)
+ residual = torch.randn(self.B, self.H, dtype=dtype, device=self.device)
+ bias = torch.randn(self.H, dtype=dtype, device=self.device) if has_bias else None
+
+ out = _bias_dropout_add_func((x, bias), residual, prob=0.0, training=True)
+
+ assert out.dtype == dtype
+ ref = self._reference_bias_dropout_add(x, bias, residual, prob=0.0, training=True)
+ ref = ref.to(dtype)
+ tols = dict(rtol=1e-5, atol=1e-5) if dtype == torch.float32 else dict(rtol=2e-2, atol=1e-2)
+ assert torch.allclose(out, ref, **tols)
+
+ # -- inference in-place optimisation still fires for same dtype ----
+
+ def test_inplace_inference_same_dtype(self):
+ """In eval mode with no grad, the in-place path should still work."""
+ x = torch.randn(self.B, self.H, dtype=torch.bfloat16, device=self.device)
+ residual = torch.randn(self.B, self.H, dtype=torch.bfloat16, device=self.device)
+ bias = torch.randn(self.H, dtype=torch.bfloat16, device=self.device)
+
+ x_ptr = x.data_ptr()
+ out = _bias_dropout_add_func((x, bias), residual, prob=0.0, training=False)
+ # In-place: output should reuse x's storage
+ assert out.data_ptr() == x_ptr
+
+ # -- inference with mixed dtypes should NOT be in-place ------------
+
+ @pytest.mark.parametrize("x_dtype", [torch.bfloat16, torch.float16])
+ def test_no_inplace_when_dtypes_differ(self, x_dtype):
+ """When x is low-precision but residual is fp32, the upcast creates
+ a new tensor so the in-place optimisation must NOT fire."""
+ x = torch.randn(self.B, self.H, dtype=x_dtype, device=self.device)
+ residual = torch.randn(self.B, self.H, dtype=torch.float32, device=self.device)
+
+ x_ptr = x.data_ptr()
+ out = _bias_dropout_add_func((x, None), residual, prob=0.0, training=False)
+ # The upcast `x = x.to(residual.dtype)` creates a new tensor,
+ # so the output must NOT alias the original x buffer.
+ assert out.dtype == torch.float32
+ assert out.data_ptr() != x_ptr
+
+ # -- dropout prob > 0 still works with mixed dtypes ----------------
+
+ @pytest.mark.parametrize("has_bias", [True, False])
+ def test_dropout_with_fp32_residual(self, has_bias):
+ """Smoke test: non-zero dropout with mixed dtypes doesn't crash."""
+ x = torch.randn(self.B, self.H, dtype=torch.bfloat16, device=self.device)
+ residual = torch.randn(self.B, self.H, dtype=torch.float32, device=self.device)
+ bias = torch.randn(self.H, dtype=torch.bfloat16, device=self.device) if has_bias else None
+
+ out = _bias_dropout_add_func((x, bias), residual, prob=0.5, training=True)
+ assert out.dtype == torch.float32
+ # With dropout, some elements should be zeroed (before residual add)
+ # so the output shouldn't be identical to the no-dropout case.
+ out_no_drop = _bias_dropout_add_func((x, bias), residual, prob=0.0, training=True)
+ # They *can* be equal with very low probability; just check dtypes
+ assert out_no_drop.dtype == torch.float32
+
+ # -- get_bias_dropout_add wrappers preserve fp32 residual ----------
+
+ @pytest.mark.parametrize("training", [True, False])
+ @pytest.mark.parametrize("fused", [True, False])
+ def test_get_bias_dropout_add_fp32_residual(self, training, fused):
+ """All four (training × fused) wrappers returned by get_bias_dropout_add
+ should preserve fp32 residual dtype."""
+ fn = get_bias_dropout_add(training=training, fused=fused)
+ x = torch.randn(self.B, self.H, dtype=torch.bfloat16, device=self.device)
+ residual = torch.randn(self.B, self.H, dtype=torch.float32, device=self.device)
+ bias = torch.randn(self.H, dtype=torch.bfloat16, device=self.device)
+
+ out = fn((x, bias), residual, prob=0.0)
+ assert out.dtype == torch.float32, (
+ f"get_bias_dropout_add(training={training}, fused={fused}) "
+ f"returned {out.dtype}, expected fp32"
+ )
+
+
+# ============================================================================
+# Tests simulating the multi-layer residual stream scenario
+# ============================================================================
+
+
+class TestFp32ResidualStreamAcrossLayers:
+ """Simulates what happens across multiple transformer layers to ensure
+ the fp32 residual stream is not degraded."""
+
+ device = "cuda"
+
+ def test_residual_stays_fp32_across_simulated_layers(self):
+ """Simulate N layers of bias-dropout-add with bf16 x and fp32 residual.
+ The residual should remain fp32 throughout — this is the scenario that
+ was broken before the fix."""
+ B, H = 4, 32
+ num_layers = 8
+
+ # Start with an fp32 residual (as the embedding layer would emit)
+ residual = torch.randn(B, H, dtype=torch.float32, device=self.device)
+
+ for layer_idx in range(num_layers):
+ # Each layer produces bf16 output (as the attention/MLP would)
+ x = torch.randn(B, H, dtype=torch.bfloat16, device=self.device)
+ bias = torch.randn(H, dtype=torch.bfloat16, device=self.device)
+
+ residual = _bias_dropout_add_func((x, bias), residual, prob=0.0, training=True)
+
+ assert (
+ residual.dtype == torch.float32
+ ), f"Layer {layer_idx}: residual dtype degraded to {residual.dtype}"
+
+ def test_residual_stays_fp32_no_bias(self):
+ """Same multi-layer simulation but without bias tensors."""
+ B, H = 4, 32
+ num_layers = 8
+
+ residual = torch.randn(B, H, dtype=torch.float32, device=self.device)
+
+ for layer_idx in range(num_layers):
+ x = torch.randn(B, H, dtype=torch.bfloat16, device=self.device)
+
+ residual = _bias_dropout_add_func((x, None), residual, prob=0.0, training=True)
+
+ assert (
+ residual.dtype == torch.float32
+ ), f"Layer {layer_idx}: residual dtype degraded to {residual.dtype}"
+
+ def test_fp32_residual_precision_advantage(self):
+ """Demonstrate that fp32 residuals accumulate more accurately than
+ bf16 residuals over many additions — the whole point of the feature."""
+ B, H = 2, 16
+ num_layers = 50
+ torch.manual_seed(42)
+
+ # Ground truth: everything in fp64
+ residual_fp64 = torch.randn(B, H, dtype=torch.float64, device=self.device)
+
+ # Track fp32 and bf16 residual streams
+ residual_fp32 = residual_fp64.float()
+ residual_bf16 = residual_fp64.to(torch.bfloat16)
+
+ for _ in range(num_layers):
+ x_fp64 = torch.randn(B, H, dtype=torch.float64, device=self.device)
+ x_bf16 = x_fp64.to(torch.bfloat16)
+
+ # fp64 reference
+ residual_fp64 = residual_fp64 + x_fp64
+
+ # fp32 residual path (the fix)
+ residual_fp32 = _bias_dropout_add_func(
+ (x_bf16, None), residual_fp32, prob=0.0, training=True
+ )
+
+ # bf16 residual path (the old broken behaviour)
+ residual_bf16 = _bias_dropout_add_func(
+ (x_bf16, None), residual_bf16.to(torch.bfloat16), prob=0.0, training=True
+ )
+
+ err_fp32 = (residual_fp32.double() - residual_fp64).abs().mean().item()
+ err_bf16 = (residual_bf16.double() - residual_fp64).abs().mean().item()
+
+ # fp32 residual should be meaningfully more precise
+ assert err_fp32 < err_bf16, (
+ f"fp32 residual error ({err_fp32:.6e}) should be less than "
+ f"bf16 residual error ({err_bf16:.6e})"
+ )
diff --git a/tests/unit_tests/fusions/test_rmsnorm_residual_fusion.py b/tests/unit_tests/fusions/test_rmsnorm_residual_fusion.py
new file mode 100644
index 00000000000..6c03e0fa801
--- /dev/null
+++ b/tests/unit_tests/fusions/test_rmsnorm_residual_fusion.py
@@ -0,0 +1,46 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+import pytest
+import torch
+from transformer_engine.pytorch import RMSNorm
+
+from megatron.core.extensions.transformer_engine import TEFusedResidualRMSNorm
+
+
+def baseline_rmsnorm_residual(x, rmsnorm: RMSNorm):
+ return rmsnorm(x), x
+
+
+@pytest.mark.parametrize("input_dtype", [torch.bfloat16, torch.float32])
+@pytest.mark.parametrize("normalized_shape", [256, 256 * 2, 256 * 4])
+def test_rmsnorm_residual_fusion(input_dtype, normalized_shape):
+ x_baseline = torch.randn(16, 32, normalized_shape, dtype=input_dtype, device="cuda")
+ x_baseline.requires_grad = True
+ x_fused = x_baseline.detach()
+ x_fused.requires_grad = True
+ baseline_rmsnorm = RMSNorm(normalized_shape=normalized_shape, dtype=input_dtype).cuda()
+ fused_rmsnorm = TEFusedResidualRMSNorm(
+ normalized_shape=normalized_shape, dtype=input_dtype
+ ).cuda()
+
+ # baseline
+ baseline_y, baseline_residual = baseline_rmsnorm_residual(x_baseline, baseline_rmsnorm)
+ baseline_loss = baseline_y.sum() + baseline_residual.sum()
+ baseline_loss.backward()
+
+ # fused
+ fused_y, fused_residual = fused_rmsnorm(x_fused)
+ fused_loss = fused_y.sum() + fused_residual.sum()
+ fused_loss.backward()
+
+ # Use tolerances appropriate for dtype (pattern from other tests)
+ tols = (
+ dict(rtol=1e-6, atol=1e-6) if input_dtype is torch.float32 else dict(rtol=2e-2, atol=1e-2)
+ )
+
+ assert fused_y.dtype == baseline_y.dtype
+ assert torch.allclose(fused_y, baseline_y, **tols)
+ assert fused_residual.dtype == baseline_residual.dtype
+ assert torch.allclose(fused_residual, baseline_residual, **tols)
+ assert x_fused.grad.dtype == x_baseline.grad.dtype
+ assert torch.allclose(x_baseline.grad, x_fused.grad, **tols)
diff --git a/tests/unit_tests/inference/contexts/attention_metadata/test_mamba_metadata.py b/tests/unit_tests/inference/contexts/attention_metadata/test_mamba_metadata.py
new file mode 100644
index 00000000000..a7f579051ac
--- /dev/null
+++ b/tests/unit_tests/inference/contexts/attention_metadata/test_mamba_metadata.py
@@ -0,0 +1,500 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+import pytest
+import torch
+
+from megatron.core.inference.batch_dimensions_utils import InferenceBatchDimensions
+from megatron.core.inference.contexts.attention_context.mamba_metadata import MambaMetadata
+
+
+class TestMambaMetadata:
+
+ @pytest.fixture
+ def metadata_context(self):
+ """Fixture to initialize MambaMetadata with standard constraints."""
+ max_requests = 16
+ max_tokens = 2048
+ metadata = MambaMetadata(max_requests=max_requests, max_tokens=max_tokens)
+
+ # Manually allocate some slots to simulate a running state.
+ # We assume request_id i maps to mamba_slot i for simplicity in assertions.
+ for i in range(max_requests):
+ metadata.request_to_mamba_state_idx[i] = i
+
+ yield metadata
+ metadata.reset()
+
+ def _run_update_test(
+ self,
+ metadata: MambaMetadata,
+ req_seq_lengths: list[int],
+ num_decode_requests: int,
+ padded_dims: InferenceBatchDimensions,
+ enable_chunked_prefill: bool,
+ ):
+ """
+ Helper to construct inputs and run update().
+
+ Args:
+ metadata: The MambaMetadata instance.
+ req_seq_lengths: List of sequence lengths for all active requests.
+ Order must be [decode_requests..., prefill_requests...].
+ num_decode_requests: Number of requests in req_seq_lengths that are in the decode phase.
+ padded_dims: The padded batch dimensions to test against.
+ enable_chunked_prefill: Whether chunked prefill is enabled.
+ """
+ num_active_requests = len(req_seq_lengths)
+ total_tokens = sum(req_seq_lengths)
+ num_prefill_requests = num_active_requests - num_decode_requests
+
+ real_dims = InferenceBatchDimensions(
+ token_count=total_tokens,
+ prefill_req_count=num_prefill_requests,
+ decode_req_count=num_decode_requests,
+ )
+
+ # Assuming 1:1 mapping (req_id i -> slot i)
+ active_mamba_indices = torch.arange(
+ num_active_requests, dtype=torch.int32, device=metadata.device
+ )
+
+ cu_seqlens = [0]
+ current_len = 0
+ for l in req_seq_lengths:
+ current_len += l
+ cu_seqlens.append(current_len)
+ cu_seqlens_tensor = torch.tensor(cu_seqlens, dtype=torch.int32, device=metadata.device)
+
+ token_to_req = []
+ for req_idx, length in enumerate(req_seq_lengths):
+ token_to_req.extend([req_idx] * length)
+ token_to_req_tensor = torch.tensor(token_to_req, dtype=torch.int32, device=metadata.device)
+
+ metadata.update(
+ active_mamba_indices=active_mamba_indices,
+ token_to_request_idx=token_to_req_tensor,
+ cu_seqlens=cu_seqlens_tensor,
+ batch_dimensions=real_dims,
+ padded_batch_dimensions=padded_dims,
+ enable_chunked_prefill=enable_chunked_prefill,
+ )
+
+ return real_dims, active_mamba_indices
+
+ # -------------------------------------------------------------------------
+ # Scenario 1: Decode Only
+ # -------------------------------------------------------------------------
+
+ @pytest.mark.internal
+ def test_update_decode_only_exact_match(self, metadata_context):
+ """Test simple decode only case where real dims match padded dims."""
+ seq_lengths = [1, 1, 1, 1] # 4 requests
+ num_decode = 4
+ padded_dims = InferenceBatchDimensions(
+ token_count=4, prefill_req_count=0, decode_req_count=4
+ )
+
+ self._run_update_test(
+ metadata_context, seq_lengths, num_decode, padded_dims, enable_chunked_prefill=False
+ )
+
+ expected_decode = torch.arange(4, dtype=torch.int32, device=metadata_context.device)
+ assert torch.equal(metadata_context.batch_indices_decode, expected_decode)
+
+ assert metadata_context.batch_indices_prefill is None
+ assert metadata_context.device_decode_prefill is None
+ assert metadata_context.cu_seqlens is None
+ assert metadata_context.seq_idx is None
+
+ @pytest.mark.internal
+ def test_update_decode_only_padded(self, metadata_context):
+ """Test decode only with padding (e.g. using CUDA graphs bucket)."""
+ seq_lengths = [1, 1] # 2 requests
+ num_decode = 2
+ # Padding to 4 requests
+ padded_dims = InferenceBatchDimensions(
+ token_count=4, prefill_req_count=0, decode_req_count=4
+ )
+
+ self._run_update_test(
+ metadata_context, seq_lengths, num_decode, padded_dims, enable_chunked_prefill=False
+ )
+
+ expected_decode = torch.tensor(
+ [0, 1, -1, -1], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.batch_indices_decode, expected_decode)
+
+ assert metadata_context.batch_indices_prefill is None
+ assert metadata_context.device_decode_prefill is None
+
+ @pytest.mark.internal
+ def test_update_chunked_enabled_no_prefill_reqs(self, metadata_context):
+ """Test edge case: Chunked prefill enabled, but only decode requests exist."""
+ seq_lengths = [1, 1]
+ num_decode = 2
+ padded_dims = InferenceBatchDimensions(
+ token_count=2, prefill_req_count=0, decode_req_count=2
+ )
+
+ self._run_update_test(
+ metadata_context, seq_lengths, num_decode, padded_dims, enable_chunked_prefill=True
+ )
+
+ # Should behave exactly like decode-only (chunked logic skipped if real_prefill == 0)
+ expected_decode = torch.tensor([0, 1], dtype=torch.int32, device=metadata_context.device)
+ assert torch.equal(metadata_context.batch_indices_decode, expected_decode)
+
+ assert metadata_context.batch_indices_prefill is None
+ assert metadata_context.cu_seqlens is None
+ assert metadata_context.seq_idx is None
+
+ # -------------------------------------------------------------------------
+ # Scenario 2: Prefill Only
+ # -------------------------------------------------------------------------
+
+ @pytest.mark.internal
+ def test_update_prefill_only_exact(self, metadata_context):
+ """Test prefill only scenario (exact match)."""
+ seq_lengths = [10, 20] # 2 requests
+ num_decode = 0
+ padded_dims = InferenceBatchDimensions(
+ token_count=30, prefill_req_count=2, decode_req_count=0
+ )
+
+ self._run_update_test(
+ metadata_context, seq_lengths, num_decode, padded_dims, enable_chunked_prefill=False
+ )
+
+ expected_prefill = torch.tensor([0, 1], dtype=torch.int32, device=metadata_context.device)
+ assert torch.equal(metadata_context.batch_indices_prefill, expected_prefill)
+
+ expected_cu_seqlens = torch.tensor(
+ [0, 10, 30], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.cu_seqlens, expected_cu_seqlens)
+
+ expected_seq_idx_0 = torch.zeros((1, 10), dtype=torch.int32, device=metadata_context.device)
+ expected_seq_idx_1 = torch.ones((1, 20), dtype=torch.int32, device=metadata_context.device)
+ expected_seq_idx = torch.cat([expected_seq_idx_0, expected_seq_idx_1], dim=1)
+ assert torch.equal(metadata_context.seq_idx, expected_seq_idx)
+
+ assert metadata_context.batch_indices_decode is None
+ assert metadata_context.device_decode_prefill is None
+
+ @pytest.mark.internal
+ def test_update_prefill_only_padded(self, metadata_context):
+ """Test prefill only with padding."""
+ seq_lengths = [10] # 1 request
+ num_decode = 0
+ # Pad to 3 prefill requests
+ padded_dims = InferenceBatchDimensions(
+ token_count=30, prefill_req_count=3, decode_req_count=0
+ )
+
+ self._run_update_test(
+ metadata_context, seq_lengths, num_decode, padded_dims, enable_chunked_prefill=False
+ )
+
+ expected_prefill = torch.tensor(
+ [0, -1, -1], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.batch_indices_prefill, expected_prefill)
+
+ expected_cu_seqlens = torch.tensor(
+ [0, 10, 10, 10], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.cu_seqlens, expected_cu_seqlens)
+
+ expected_seq_idx = torch.full(
+ (1, 30), -1, dtype=torch.int32, device=metadata_context.device
+ )
+ expected_seq_idx[:, :10] = 0
+ assert torch.equal(metadata_context.seq_idx, expected_seq_idx)
+
+ assert metadata_context.batch_indices_decode is None
+ assert metadata_context.device_decode_prefill is None
+
+ # -------------------------------------------------------------------------
+ # Scenario 3: Mixed Batch (Decode + Prefill)
+ # -------------------------------------------------------------------------
+
+ @pytest.mark.internal
+ def test_update_mixed_batch_exact(self, metadata_context):
+ """Test mix of decode and prefill requests (exact match)."""
+ # 2 decode (len 1), 2 prefill (len 10, 20)
+ seq_lengths = [1, 1, 10, 20]
+ num_decode = 2
+ padded_dims = InferenceBatchDimensions(
+ token_count=32, prefill_req_count=2, decode_req_count=2
+ )
+
+ self._run_update_test(
+ metadata_context, seq_lengths, num_decode, padded_dims, enable_chunked_prefill=False
+ )
+
+ expected_decode = torch.tensor([0, 1], dtype=torch.int32, device=metadata_context.device)
+ assert torch.equal(metadata_context.batch_indices_decode, expected_decode)
+
+ expected_prefill = torch.tensor([2, 3], dtype=torch.int32, device=metadata_context.device)
+ assert torch.equal(metadata_context.batch_indices_prefill, expected_prefill)
+
+ # device_decode_prefill stores [decode_token_count, prefill_token_count].
+ expected_device_counts = torch.tensor(
+ [2, 30], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.device_decode_prefill, expected_device_counts)
+
+ expected_cu_seqlens = torch.tensor(
+ [0, 10, 30], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.cu_seqlens, expected_cu_seqlens)
+
+ expected_seq_idx_0 = torch.zeros((1, 10), dtype=torch.int32, device=metadata_context.device)
+ expected_seq_idx_1 = torch.ones((1, 20), dtype=torch.int32, device=metadata_context.device)
+ expected_seq_idx_padding = torch.full(
+ (1, 2), -1, dtype=torch.int32, device=metadata_context.device
+ )
+ expected_seq_idx = torch.cat(
+ [expected_seq_idx_0, expected_seq_idx_1, expected_seq_idx_padding], dim=1
+ )
+ assert torch.equal(metadata_context.seq_idx, expected_seq_idx)
+
+ @pytest.mark.internal
+ def test_update_mixed_batch_mtp(self, metadata_context):
+ """Test mixed batch with MTP (decode seq_len > 1) to verify token-count split."""
+ # 2 decode requests each with 3 tokens (1 accepted + 2 speculative),
+ # 1 prefill request with 20 tokens.
+ seq_lengths = [3, 3, 20]
+ num_decode = 2
+ padded_dims = InferenceBatchDimensions(
+ token_count=26, prefill_req_count=1, decode_req_count=2
+ )
+
+ self._run_update_test(
+ metadata_context, seq_lengths, num_decode, padded_dims, enable_chunked_prefill=False
+ )
+
+ expected_decode = torch.tensor([0, 1], dtype=torch.int32, device=metadata_context.device)
+ assert torch.equal(metadata_context.batch_indices_decode, expected_decode)
+
+ expected_prefill = torch.tensor([2], dtype=torch.int32, device=metadata_context.device)
+ assert torch.equal(metadata_context.batch_indices_prefill, expected_prefill)
+
+ # device_decode_prefill stores [decode_token_count, prefill_token_count].
+ # 2 decode requests * 3 tokens each = 6 decode tokens, 20 prefill tokens.
+ expected_device_counts = torch.tensor(
+ [6, 20], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.device_decode_prefill, expected_device_counts)
+
+ @pytest.mark.internal
+ def test_update_padded_prefill_and_decode(self, metadata_context):
+ """Test scenario where padded dimensions differ from real dimensions (Mixed)."""
+ # Real: 1 decode, 1 prefill.
+ seq_lengths = [1, 10]
+ num_decode = 1
+
+ # Padded: 4 decode, 4 prefill. Total tokens 32.
+ padded_dims = InferenceBatchDimensions(
+ token_count=32, prefill_req_count=4, decode_req_count=4
+ )
+
+ self._run_update_test(
+ metadata_context, seq_lengths, num_decode, padded_dims, enable_chunked_prefill=False
+ )
+
+ expected_decode = torch.tensor(
+ [0, -1, -1, -1], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.batch_indices_decode, expected_decode)
+
+ expected_prefill = torch.tensor(
+ [1, -1, -1, -1], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.batch_indices_prefill, expected_prefill)
+
+ # device_decode_prefill stores [decode_token_count, prefill_token_count].
+ expected_device_counts = torch.tensor(
+ [1, 10], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.device_decode_prefill, expected_device_counts)
+
+ expected_cu = torch.tensor(
+ [0, 10, 10, 10, 10], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.cu_seqlens, expected_cu)
+
+ expected_seq_idx = torch.full(
+ (1, 32), -1, dtype=torch.int32, device=metadata_context.device
+ )
+ expected_seq_idx[:, :10] = 0
+ assert torch.equal(metadata_context.seq_idx, expected_seq_idx)
+
+ # -------------------------------------------------------------------------
+ # Scenario 4: Chunked Prefill
+ #
+ # In our unified implementation, all prefill requests (including chunked)
+ # go through the same varlen path and are stored in batch_indices_prefill.
+ # There is no separate batch_indices_chunked_prefill.
+ # -------------------------------------------------------------------------
+
+ @pytest.mark.internal
+ def test_update_chunked_prefill_mixed_exact(self, metadata_context):
+ """Test chunked prefill mixed with decode (Exact match)."""
+ # 1 decode, 1 chunked prefill (len 50), 1 regular prefill (len 10)
+ seq_lengths = [1, 50, 10]
+ num_decode = 1
+
+ # Exact dimensions
+ padded_dims = InferenceBatchDimensions(
+ token_count=61, prefill_req_count=2, decode_req_count=1
+ )
+
+ self._run_update_test(
+ metadata_context, seq_lengths, num_decode, padded_dims, enable_chunked_prefill=True
+ )
+
+ # All prefill requests (chunked + regular) are unified in batch_indices_prefill.
+ expected_prefill = torch.tensor([1, 2], dtype=torch.int32, device=metadata_context.device)
+ assert torch.equal(metadata_context.batch_indices_prefill, expected_prefill)
+
+ # device_decode_prefill stores [decode_token_count, prefill_token_count].
+ expected_device_counts = torch.tensor(
+ [1, 60], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.device_decode_prefill, expected_device_counts)
+
+ expected_cu_seqlens = torch.tensor(
+ [0, 50, 60], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.cu_seqlens, expected_cu_seqlens)
+
+ expected_seq_idx_0 = torch.zeros((1, 50), dtype=torch.int32, device=metadata_context.device)
+ expected_seq_idx_1 = torch.ones((1, 10), dtype=torch.int32, device=metadata_context.device)
+ expected_seq_idx_padding = torch.full(
+ (1, 1), -1, dtype=torch.int32, device=metadata_context.device
+ )
+ expected_seq_idx = torch.cat(
+ [expected_seq_idx_0, expected_seq_idx_1, expected_seq_idx_padding], dim=1
+ )
+ assert torch.equal(metadata_context.seq_idx, expected_seq_idx)
+
+ @pytest.mark.internal
+ def test_update_chunked_prefill_mixed_padded(self, metadata_context):
+ """Test chunked prefill mixed with decode (Padded)."""
+ # 2 decode, 1 chunked prefill (len 50), 1 regular prefill (len 10)
+ seq_lengths = [1, 1, 50, 10]
+ num_decode = 2
+ padded_dims = InferenceBatchDimensions(
+ token_count=62, prefill_req_count=2, decode_req_count=2
+ )
+
+ self._run_update_test(
+ metadata_context, seq_lengths, num_decode, padded_dims, enable_chunked_prefill=True
+ )
+
+ expected_decode = torch.tensor([0, 1], dtype=torch.int32, device=metadata_context.device)
+ assert torch.equal(metadata_context.batch_indices_decode, expected_decode)
+
+ # All prefill requests unified in batch_indices_prefill.
+ expected_prefill = torch.tensor([2, 3], dtype=torch.int32, device=metadata_context.device)
+ assert torch.equal(metadata_context.batch_indices_prefill, expected_prefill)
+
+ # device_decode_prefill stores [decode_token_count, prefill_token_count].
+ expected_device_counts = torch.tensor(
+ [2, 60], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.device_decode_prefill, expected_device_counts)
+
+ expected_cu = torch.tensor([0, 50, 60], dtype=torch.int32, device=metadata_context.device)
+ assert torch.equal(metadata_context.cu_seqlens, expected_cu)
+
+ expected_seq_idx = torch.full(
+ (1, 62), -1, dtype=torch.int32, device=metadata_context.device
+ )
+ expected_seq_idx[:, :50] = 0
+ expected_seq_idx[:, 50:60] = 1
+ assert torch.equal(metadata_context.seq_idx, expected_seq_idx)
+
+ @pytest.mark.internal
+ def test_update_chunked_only_padded(self, metadata_context):
+ """Test a case with only chunked prefill (no decode, no regular prefill) but with padding."""
+ # 1 chunked prefill request.
+ seq_lengths = [100]
+ num_decode = 0
+
+ padded_dims = InferenceBatchDimensions(
+ token_count=128, prefill_req_count=2, decode_req_count=0
+ )
+
+ self._run_update_test(
+ metadata_context, seq_lengths, num_decode, padded_dims, enable_chunked_prefill=True
+ )
+
+ assert metadata_context.batch_indices_decode is None
+
+ # Single prefill request unified in batch_indices_prefill, with padding.
+ expected_prefill = torch.tensor([0, -1], dtype=torch.int32, device=metadata_context.device)
+ assert torch.equal(metadata_context.batch_indices_prefill, expected_prefill)
+
+ expected_cu_seqlens = torch.tensor(
+ [0, 100, 100], dtype=torch.int32, device=metadata_context.device
+ )
+ assert torch.equal(metadata_context.cu_seqlens, expected_cu_seqlens)
+
+ expected_seq_idx = torch.full(
+ (1, 128), -1, dtype=torch.int32, device=metadata_context.device
+ )
+ expected_seq_idx[:, :100] = 0
+ assert torch.equal(metadata_context.seq_idx, expected_seq_idx)
+
+ assert metadata_context.device_decode_prefill is None
+
+ # -------------------------------------------------------------------------
+ # Scenario 5: Paused Requests (Absolute vs Relative Indices)
+ # -------------------------------------------------------------------------
+
+ @pytest.mark.internal
+ def test_update_with_paused_requests_absolute_indices(self, metadata_context):
+ """
+ Test that seq_idx correctly normalizes to 0-based indices when paused
+ requests exist, meaning absolute request IDs are > 0.
+ """
+ # Simulate 2 paused requests, meaning active requests start at index 2
+ # We'll use 2 active prefill requests of lengths 10 and 20.
+ paused_request_count = 2
+ seq_lengths = [10, 20]
+
+ # Absolute request indices: 2 and 3
+ token_to_req = [2] * 10 + [3] * 20
+ token_to_req_tensor = torch.tensor(
+ token_to_req, dtype=torch.int32, device=metadata_context.device
+ )
+
+ # Active mamba slots matching the absolute request IDs
+ active_mamba_indices = torch.tensor(
+ [2, 3], dtype=torch.int32, device=metadata_context.device
+ )
+
+ # cu_seqlens is always local to the active slice passed in
+ cu_seqlens = torch.tensor([0, 10, 30], dtype=torch.int32, device=metadata_context.device)
+
+ dims = InferenceBatchDimensions(token_count=30, prefill_req_count=2, decode_req_count=0)
+
+ metadata_context.update(
+ active_mamba_indices=active_mamba_indices,
+ token_to_request_idx=token_to_req_tensor,
+ cu_seqlens=cu_seqlens,
+ batch_dimensions=dims,
+ padded_batch_dimensions=dims,
+ enable_chunked_prefill=False,
+ )
+
+ # The key assertion: seq_idx should be strictly 0-based relative
+ # to the buffer (0s and 1s), despite the absolute request IDs being 2 and 3.
+ expected_seq_idx_0 = torch.zeros((1, 10), dtype=torch.int32, device=metadata_context.device)
+ expected_seq_idx_1 = torch.ones((1, 20), dtype=torch.int32, device=metadata_context.device)
+ expected_seq_idx = torch.cat([expected_seq_idx_0, expected_seq_idx_1], dim=1)
+
+ assert torch.equal(metadata_context.seq_idx, expected_seq_idx)
diff --git a/tests/unit_tests/inference/contexts/test_dynamic_context.py b/tests/unit_tests/inference/contexts/test_dynamic_context.py
index 05e0306bfd8..0e3a959cb00 100644
--- a/tests/unit_tests/inference/contexts/test_dynamic_context.py
+++ b/tests/unit_tests/inference/contexts/test_dynamic_context.py
@@ -1,14 +1,14 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+import contextlib
import math
+from unittest import mock
import pytest
import torch
from megatron.core import parallel_state
-from megatron.core.inference.contexts.attention_context.mamba_metadata import (
- MambaInferenceStateConfig,
-)
+from megatron.core.inference.config import InferenceConfig, MambaInferenceStateConfig
from megatron.core.inference.contexts.dynamic_context import (
DynamicInferenceContext,
RequestOverflowError,
@@ -18,28 +18,47 @@
from megatron.core.inference.sampling_params import SamplingParams
from megatron.core.ssm.mamba_hybrid_layer_allocation import Symbols
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
+from megatron.core.transformer.transformer_config import TransformerConfig
from tests.unit_tests.test_utilities import Utils
-def set_rounder(value):
- """Utility function to set the DynamicInferenceContext rounder."""
- DynamicInferenceContext.ROUNDER = value # For backwards compatibility
- DynamicInferenceContext.TOKEN_ROUNDER = value
- DynamicInferenceContext.REQUEST_ROUNDER = value
+@contextlib.contextmanager
+def rounder_override(n):
+ original_token_rounder = DynamicInferenceContext.TOKEN_ROUNDER
+ original_request_rounder = DynamicInferenceContext.REQUEST_ROUNDER
+ try:
+ DynamicInferenceContext.TOKEN_ROUNDER = n
+ DynamicInferenceContext.REQUEST_ROUNDER = n
+ yield
+ finally:
+ DynamicInferenceContext.TOKEN_ROUNDER = original_token_rounder
+ DynamicInferenceContext.REQUEST_ROUNDER = original_request_rounder
class TestDynamicContext:
- def _setup_model_parallel_group(self, tensor_parallel_size, pipeline_parallel_size):
-
- self.pp_size = pipeline_parallel_size
+ @classmethod
+ def setup_class(cls):
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=1, pipeline_model_parallel_size=1
+ )
+ model_parallel_cuda_manual_seed(123)
+ def _setup_model_parallel_group(self, tensor_parallel_size, pipeline_parallel_size):
+ Utils.destroy_model_parallel()
Utils.initialize_model_parallel(
tensor_model_parallel_size=tensor_parallel_size,
pipeline_model_parallel_size=pipeline_parallel_size,
)
model_parallel_cuda_manual_seed(123)
+ def _restore_model_parallel(self):
+ Utils.destroy_model_parallel()
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=1, pipeline_model_parallel_size=1
+ )
+ model_parallel_cuda_manual_seed(123)
+
def _get_dynamic_context(
self,
params_dtype,
@@ -52,50 +71,63 @@ def _get_dynamic_context(
max_tokens,
is_hybrid_model=False,
layer_type_list=None,
- rounder=64,
paused_buffer_size_gb=None,
+ num_cuda_graphs=None,
+ num_speculative_tokens=0,
+ enable_chunked_prefill: bool = False,
+ max_requests: int = None,
):
- set_rounder(rounder)
-
if is_hybrid_model:
if layer_type_list is None:
layer_type_list = [Symbols.MAMBA, Symbols.MLP, Symbols.ATTENTION, Symbols.MLP]
mamba_conv_states_shape = (544, 4)
mamba_ssm_states_shape = (8, 64, 16)
mamba_inference_state_config = MambaInferenceStateConfig(
- layer_type_list, mamba_conv_states_shape, mamba_ssm_states_shape
+ layer_type_list,
+ mamba_conv_states_shape,
+ mamba_ssm_states_shape,
+ params_dtype,
+ params_dtype,
)
else:
mamba_inference_state_config = None
dynamic_context = DynamicInferenceContext(
- params_dtype=params_dtype,
- num_layers=num_layers // self.pp_size,
- kv_channels=kv_channels,
- num_attention_heads=num_attention_heads,
- max_sequence_length=max_sequence_length,
- num_cuda_graphs=None,
- use_cuda_graphs_for_non_decode_steps=True,
- buffer_size_gb=buffer_size_gb,
- paused_buffer_size_gb=(
- 0.2 * buffer_size_gb if paused_buffer_size_gb is None else paused_buffer_size_gb
+ model_config=TransformerConfig(
+ params_dtype=params_dtype,
+ num_layers=num_layers,
+ kv_channels=kv_channels,
+ num_attention_heads=num_attention_heads,
+ ),
+ inference_config=InferenceConfig(
+ max_sequence_length=max_sequence_length,
+ num_cuda_graphs=num_cuda_graphs,
+ use_cuda_graphs_for_non_decode_steps=True,
+ buffer_size_gb=buffer_size_gb,
+ paused_buffer_size_gb=(
+ 0.2 * buffer_size_gb if paused_buffer_size_gb is None else paused_buffer_size_gb
+ ),
+ block_size_tokens=block_size_tokens,
+ max_tokens=max_tokens,
+ num_speculative_tokens=num_speculative_tokens,
+ mamba_inference_state_config=mamba_inference_state_config,
+ use_flashinfer_fused_rope=None, # default to using flash-infer if available
+ # this is for compatibility with the LTS environment
+ unified_memory_level=0, # unit tests currently broken with UVM
+ enable_chunked_prefill=enable_chunked_prefill,
+ max_requests=max_requests,
),
- block_size_tokens=block_size_tokens,
- max_tokens=max_tokens,
- mamba_inference_state_config=mamba_inference_state_config,
- use_flashinfer_fused_rope=None, # default to using flash-infer if available
- # this is for compatibility with the LTS environment
- unified_memory_level=0, # unit tests currently broken with UVM
)
return dynamic_context
- def teardown_method(self, method):
+ @classmethod
+ def teardown_class(cls):
Utils.destroy_model_parallel()
@pytest.mark.internal
+ @rounder_override(64)
@pytest.mark.parametrize("is_hybrid_model", [False, True])
def test_initialize_dynamic_context(self, is_hybrid_model: bool):
- self._setup_model_parallel_group(1, 1)
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
@@ -107,20 +139,19 @@ def test_initialize_dynamic_context(self, is_hybrid_model: bool):
block_size_tokens=128,
max_tokens=None,
is_hybrid_model=is_hybrid_model,
- rounder=64,
)
if not is_hybrid_model:
- assert dynamic_context.block_allocator.total_count == 491
- assert dynamic_context.block_allocator.active_count == 392
+ assert dynamic_context.kv_block_allocator.total_count == 491
+ assert dynamic_context.kv_block_allocator.active_count == 392
# We make max_requests divisible by the REQUEST_ROUNDER.
assert dynamic_context.max_requests == 448
assert dynamic_context.max_tokens == 16384
assert dynamic_context.num_mamba_layers == 0
assert dynamic_context.mamba_metadata is None
else:
- assert dynamic_context.block_allocator.total_count == 556
- assert dynamic_context.block_allocator.active_count == 444
+ assert dynamic_context.kv_block_allocator.total_count == 556
+ assert dynamic_context.kv_block_allocator.active_count == 444
assert dynamic_context.max_requests == 512
assert dynamic_context.max_tokens == 16384
assert dynamic_context.num_mamba_layers == 1
@@ -131,7 +162,7 @@ def test_initialize_dynamic_context(self, is_hybrid_model: bool):
@pytest.mark.internal
def test_is_static_batching(self):
- self._setup_model_parallel_group(1, 1)
+
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
num_layers=2,
@@ -145,9 +176,10 @@ def test_is_static_batching(self):
assert not dynamic_context.is_static_batching()
@pytest.mark.internal
+ @rounder_override(64)
@pytest.mark.parametrize("is_hybrid_model", [False, True])
def test_is_memory_available(self, is_hybrid_model):
- self._setup_model_parallel_group(1, 1)
+
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
num_layers=2,
@@ -159,18 +191,18 @@ def test_is_memory_available(self, is_hybrid_model):
max_tokens=None,
is_hybrid_model=is_hybrid_model,
)
- dynamic_context.block_allocator.total_avail = 10
- assert dynamic_context.block_allocator.is_memory_available(10)
- assert not dynamic_context.block_allocator.is_memory_available(11)
+ dynamic_context.kv_block_allocator.total_avail = 10
+ assert dynamic_context.kv_block_allocator.is_memory_available(10)
+ assert not dynamic_context.kv_block_allocator.is_memory_available(11)
- assert dynamic_context.block_allocator.is_memory_available(1)
- dynamic_context.block_allocator.total_avail = 0
- assert not dynamic_context.block_allocator.is_memory_available(1)
+ assert dynamic_context.kv_block_allocator.is_memory_available(1)
+ dynamic_context.kv_block_allocator.total_avail = 0
+ assert not dynamic_context.kv_block_allocator.is_memory_available(1)
@pytest.mark.internal
+ @rounder_override(1)
@pytest.mark.parametrize("is_hybrid_model", [False, True])
def test_request_overflow(self, is_hybrid_model: bool):
- self._setup_model_parallel_group(1, 1)
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
@@ -181,7 +213,6 @@ def test_request_overflow(self, is_hybrid_model: bool):
buffer_size_gb=0.01,
block_size_tokens=32,
max_tokens=None,
- rounder=1,
is_hybrid_model=is_hybrid_model,
)
dynamic_context.max_requests //= 2
@@ -198,9 +229,9 @@ def test_request_overflow(self, is_hybrid_model: bool):
) # Adding more than allowed requests
@pytest.mark.internal
+ @rounder_override(1)
@pytest.mark.parametrize("is_hybrid_model", [False, True])
def test_token_overflow_error(self, is_hybrid_model: bool):
- self._setup_model_parallel_group(1, 1)
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
@@ -211,7 +242,6 @@ def test_token_overflow_error(self, is_hybrid_model: bool):
buffer_size_gb=0.1,
block_size_tokens=128,
max_tokens=200, # setting low, but >= context.max_requests.
- rounder=1,
is_hybrid_model=is_hybrid_model,
)
@@ -227,9 +257,9 @@ def test_token_overflow_error(self, is_hybrid_model: bool):
) # Exceeding max token count
@pytest.mark.internal
+ @rounder_override(64)
@pytest.mark.parametrize("is_hybrid_model", [False, True])
def test_reset(self, is_hybrid_model: bool):
- self._setup_model_parallel_group(1, 1)
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
@@ -291,19 +321,20 @@ def test_reset(self, is_hybrid_model: bool):
assert torch.all(dynamic_context.token_to_block_idx == -1)
assert torch.all(dynamic_context.token_to_local_position_within_kv_block == 0)
if not is_hybrid_model:
- assert dynamic_context.block_allocator.active_count == 819
- assert dynamic_context.block_allocator.total_count == 1024
+ assert dynamic_context.kv_block_allocator.active_count == 819
+ assert dynamic_context.kv_block_allocator.total_count == 1024
else:
- assert dynamic_context.block_allocator.active_count == 1517
- assert dynamic_context.block_allocator.total_count == 1897
+ assert dynamic_context.kv_block_allocator.active_count == 1517
+ assert dynamic_context.kv_block_allocator.total_count == 1897
assert torch.all(dynamic_context.request_to_kv_block_ids == -1)
if is_hybrid_model:
assert torch.all(dynamic_context.mamba_metadata.request_to_mamba_state_idx == -1)
@pytest.mark.internal
+ @rounder_override(64)
@pytest.mark.parametrize("is_hybrid_model", [False, True])
def test_allocate_and_release_memory_blocks(self, is_hybrid_model):
- self._setup_model_parallel_group(1, 1)
+
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
num_layers=4,
@@ -323,35 +354,35 @@ def test_allocate_and_release_memory_blocks(self, is_hybrid_model):
expected_block_count_avail = expected_memory_blocks[0]
assert (
- dynamic_context.block_allocator.allocate_memory_blocks(4)
+ dynamic_context.kv_block_allocator.allocate_memory_blocks(4)
.cpu()
.detach()
.numpy()
.tolist()
== expected_memory_blocks
)
- assert dynamic_context.block_allocator.total_avail == expected_block_count_avail
- dynamic_context.block_allocator.release_memory_blocks(
+ assert dynamic_context.kv_block_allocator.total_avail == expected_block_count_avail
+ dynamic_context.kv_block_allocator.release_memory_blocks(
torch.tensor(expected_memory_blocks[-2:], device='cuda')
)
- assert dynamic_context.block_allocator.total_avail == expected_block_count_avail + 2
+ assert dynamic_context.kv_block_allocator.total_avail == expected_block_count_avail + 2
assert (
- dynamic_context.block_allocator.allocate_memory_blocks(1).item()
+ dynamic_context.kv_block_allocator.allocate_memory_blocks(1).item()
== expected_memory_blocks[-1]
)
- assert dynamic_context.block_allocator.total_avail == expected_block_count_avail + 1
+ assert dynamic_context.kv_block_allocator.total_avail == expected_block_count_avail + 1
# Should return None since we allocate more blocks than what we have.
assert (
- dynamic_context.block_allocator.allocate_memory_blocks(
- dynamic_context.block_allocator.total_avail + 100
+ dynamic_context.kv_block_allocator.allocate_memory_blocks(
+ dynamic_context.kv_block_allocator.total_avail + 100
)
== None
)
@pytest.mark.internal
+ @rounder_override(64)
@pytest.mark.parametrize("is_hybrid_model", [False, True])
def test_add_request(self, is_hybrid_model: bool):
- self._setup_model_parallel_group(1, 1)
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
@@ -422,8 +453,8 @@ def test_add_request(self, is_hybrid_model: bool):
)
@pytest.mark.internal
+ @rounder_override(64)
def test_add_dummy_requests_parallel_populates_state(self):
- self._setup_model_parallel_group(1, 1)
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
@@ -451,14 +482,14 @@ def test_add_dummy_requests_parallel_populates_state(self):
lengths = [req.remaining_prompt_length for req in requests]
total_tokens = sum(lengths)
- block_avail_before = dynamic_context.block_allocator.total_avail
+ block_avail_before = dynamic_context.kv_block_allocator.total_avail
dynamic_context.add_dummy_requests_parallel(requests, count_as_prefill=False)
assert dynamic_context.active_token_count == total_tokens
assert dynamic_context.total_request_count == len(requests)
assert dynamic_context.num_prefill_requests == 0
- assert dynamic_context.block_allocator.total_avail == block_avail_before
+ assert dynamic_context.kv_block_allocator.total_avail == block_avail_before
expected_tokens = torch.cat(
[torch.arange(0, 3, device='cuda'), torch.arange(3, 9, device='cuda')]
@@ -485,7 +516,7 @@ def test_add_dummy_requests_parallel_populates_state(self):
dynamic_context.token_to_local_position_within_kv_block[:total_tokens], expected_local
)
- dummy_block_idx = dynamic_context.block_allocator.dummy_block_idx
+ dummy_block_idx = dynamic_context.kv_block_allocator.dummy_block_idx
assert torch.all(dynamic_context.token_to_block_idx[:total_tokens] == dummy_block_idx)
assert torch.equal(
@@ -520,8 +551,8 @@ def test_add_dummy_requests_parallel_populates_state(self):
)
@pytest.mark.internal
+ @rounder_override(64)
def test_add_dummy_requests_parallel_hybrid_allocates_mamba(self):
- self._setup_model_parallel_group(1, 1)
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
@@ -550,8 +581,8 @@ def test_add_dummy_requests_parallel_hybrid_allocates_mamba(self):
assert torch.all(dynamic_context.mamba_ssm_states[:, mamba_idx] == 0)
@pytest.mark.internal
+ @rounder_override(64)
def test_add_dummy_requests_parallel_decode_does_not_count_as_prefill(self):
- self._setup_model_parallel_group(1, 1)
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
@@ -575,9 +606,9 @@ def test_add_dummy_requests_parallel_decode_does_not_count_as_prefill(self):
assert dynamic_context.num_prefill_requests == 0
@pytest.mark.internal
+ @rounder_override(64)
@pytest.mark.parametrize("is_hybrid_model", [False, True])
def test_update_request(self, is_hybrid_model: bool):
- self._setup_model_parallel_group(1, 1)
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
@@ -596,7 +627,7 @@ def test_update_request(self, is_hybrid_model: bool):
dynamic_context.paused_request_count = 0
dynamic_context.total_request_count = 3
dynamic_context.request_kv_block_counts[0:3] = 1
- new_block_ids = dynamic_context.block_allocator.allocate_memory_blocks(3)
+ new_block_ids = dynamic_context.kv_block_allocator.allocate_memory_blocks(3)
dynamic_context.request_to_kv_block_ids[0:3, 0] = new_block_ids
if is_hybrid_model:
@@ -646,16 +677,16 @@ def test_update_request(self, is_hybrid_model: bool):
)
total_request_count = 10
- dynamic_context.block_allocator.total_avail -= 11 # We align 11 blocks to the 10 requests we have. 3rd request alone we setup like it requires 2 blocks
+ dynamic_context.kv_block_allocator.total_avail -= 11 # We align 11 blocks to the 10 requests we have. 3rd request alone we setup like it requires 2 blocks
dynamic_context.total_request_count = total_request_count
dynamic_context.request_to_kv_block_ids[0:total_request_count, 0] = torch.arange(
- dynamic_context.block_allocator.total_avail,
- dynamic_context.block_allocator.total_avail + 10,
+ dynamic_context.kv_block_allocator.total_avail,
+ dynamic_context.kv_block_allocator.total_avail + 10,
)
dynamic_context.request_to_kv_block_ids[3][
1
- ] = dynamic_context.block_allocator.total_avail # Assign one extra block to request 3.
+ ] = dynamic_context.kv_block_allocator.total_avail # Assign one extra block to request 3.
dynamic_context.request_kv_length_offsets[0:total_request_count] = 10
# For 0, 1, 5, 6, the total number of tokens in last block is block size -1, so that they will all need extra blocks
dynamic_context.request_kv_length_offsets[0:2] = dynamic_context.block_size_tokens - 1
@@ -774,10 +805,10 @@ def test_update_request(self, is_hybrid_model: bool):
)
@pytest.mark.internal
+ @rounder_override(64)
@pytest.mark.parametrize("is_hybrid_model", [False, True])
def test_release_memory_blocks_for_finished_requests(self, is_hybrid_model):
"""Test that memory blocks are correctly released for finished requests."""
- self._setup_model_parallel_group(1, 1)
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
@@ -793,18 +824,22 @@ def test_release_memory_blocks_for_finished_requests(self, is_hybrid_model):
# Set up the initial state with 5 requests
# Allocate 5 blocks for 5 requests
- initial_blocks = dynamic_context.block_allocator.allocate_memory_blocks(5)
+ initial_blocks = dynamic_context.kv_block_allocator.allocate_memory_blocks(5)
dynamic_context.total_request_count = 5
dynamic_context.paused_request_count = 0
# Record the available blocks before releasing memory
- initial_available_blocks = dynamic_context.block_allocator.total_avail
+ initial_available_blocks = dynamic_context.kv_block_allocator.total_avail
# Assign blocks to the requests (one block per request)
for i in range(5):
dynamic_context.request_to_kv_block_ids[i, 0] = initial_blocks[i]
dynamic_context.request_query_lengths[i] = 1
dynamic_context.request_ids[i] = i
+ dynamic_context.request_last_kv_block_id[i] = initial_blocks[i]
+ dynamic_context.request_last_kv_block_offset[i] = 0
+ dynamic_context.request_kv_block_counts[i] = 1
+ dynamic_context.request_in_prefill_status_tensor[i] = 0
if is_hybrid_model:
dynamic_context.mamba_conv_states[:, i, :, :].fill_(
float(i + 1)
@@ -829,7 +864,7 @@ def test_release_memory_blocks_for_finished_requests(self, is_hybrid_model):
assert dynamic_context.active_token_count == 2
# Verify that 3 blocks were released by checking the available blocks
- assert dynamic_context.block_allocator.total_avail == initial_available_blocks + 3
+ assert dynamic_context.kv_block_allocator.total_avail == initial_available_blocks + 3
if is_hybrid_model:
# Request at position 3 now moves into finished request position 0
@@ -846,10 +881,10 @@ def test_release_memory_blocks_for_finished_requests(self, is_hybrid_model):
assert mamba_idx[4] == -1
@pytest.mark.internal
+ @rounder_override(64)
@pytest.mark.parametrize("is_hybrid_model", [False, True])
def test_finished_requests_with_multiple_blocks(self, is_hybrid_model):
"""Test that all memory blocks are correctly released for finished requests that use multiple blocks."""
- self._setup_model_parallel_group(1, 1)
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
@@ -865,12 +900,12 @@ def test_finished_requests_with_multiple_blocks(self, is_hybrid_model):
# Set up the initial state with 3 requests, where some use multiple blocks
# Allocate 6 blocks in total for the requests
- initial_blocks = dynamic_context.block_allocator.allocate_memory_blocks(6)
+ initial_blocks = dynamic_context.kv_block_allocator.allocate_memory_blocks(6)
dynamic_context.total_request_count = 3
dynamic_context.paused_request_count = 0
# Record the available blocks before releasing memory
- initial_available_blocks = dynamic_context.block_allocator.total_avail
+ initial_available_blocks = dynamic_context.kv_block_allocator.total_avail
# Assign blocks to the requests:
# - Request 0: 1 block
@@ -892,9 +927,16 @@ def test_finished_requests_with_multiple_blocks(self, is_hybrid_model):
for i in range(3):
dynamic_context.request_query_lengths[i] = 1
dynamic_context.request_ids[i] = i
+ dynamic_context.request_last_kv_block_id[i] = dynamic_context.request_to_kv_block_ids[
+ i, dynamic_context.request_kv_block_counts[i] - 1
+ ]
+ dynamic_context.request_last_kv_block_offset[i] = 0
+ dynamic_context.request_in_prefill_status_tensor[i] = 0
if is_hybrid_model:
dynamic_context.mamba_conv_states[:, i, :, :].fill_(float(i + 1))
dynamic_context.mamba_ssm_states[:, i, :, :, :].fill_(float(i + 1))
+ dynamic_context.mamba_metadata.request_to_mamba_state_idx[i] = i
+ dynamic_context.mamba_metadata.mamba_state_free_slot_count -= 1
# Create an active_requests_mask where all requests are finished
active_requests_mask = torch.tensor([0, 0, 0], device=torch.cuda.current_device())
@@ -910,12 +952,12 @@ def test_finished_requests_with_multiple_blocks(self, is_hybrid_model):
assert dynamic_context.active_token_count == 0
# Verify that all 6 blocks were released by checking the available blocks
- assert dynamic_context.block_allocator.total_avail == initial_available_blocks + 6
+ assert dynamic_context.kv_block_allocator.total_avail == initial_available_blocks + 6
@pytest.mark.internal
+ @rounder_override(64)
@pytest.mark.parametrize("is_hybrid_model", [False, True])
def test_mamba_states_cache(self, is_hybrid_model: bool):
- self._setup_model_parallel_group(1, 1)
if not is_hybrid_model:
# If not hybrid, mamba_states_cache should fail
@@ -988,8 +1030,9 @@ def test_mamba_states_cache(self, is_hybrid_model: bool):
assert torch.all(ssm_state_layer3 == 40.0)
@pytest.mark.internal
+ @rounder_override(64)
def test_calculate_and_store_log_probs(self):
- self._setup_model_parallel_group(1, 1)
+
dynamic_context = self._get_dynamic_context(
params_dtype=torch.float32,
num_layers=4,
@@ -1205,6 +1248,7 @@ def test_calculate_and_store_log_probs(self):
current_global_token_offset += expected_len
@pytest.mark.internal
+ @rounder_override(64)
def test_pipeline_parallel_uneven_layers(self):
"""
Test that DynamicInferenceContext synchronizes the total block count across
@@ -1215,28 +1259,49 @@ def test_pipeline_parallel_uneven_layers(self):
rank = parallel_state.get_pipeline_model_parallel_rank()
+ mamba_conv_states_shape = (544, 4)
+ mamba_ssm_states_shape = (8, 64, 16)
+ params_dtype = torch.float32
+
if rank == 0:
- local_num_layers = 12
+ mamba_inference_state_config = MambaInferenceStateConfig(
+ [Symbols.MAMBA] + [Symbols.ATTENTION] * 4,
+ mamba_conv_states_shape,
+ mamba_ssm_states_shape,
+ params_dtype,
+ params_dtype,
+ )
else:
- local_num_layers = 4
+ mamba_inference_state_config = MambaInferenceStateConfig(
+ [Symbols.MAMBA] * 4 + [Symbols.ATTENTION],
+ mamba_conv_states_shape,
+ mamba_ssm_states_shape,
+ params_dtype,
+ params_dtype,
+ )
context = DynamicInferenceContext(
- params_dtype=torch.float32,
- num_layers=local_num_layers,
- kv_channels=64,
- num_attention_heads=8,
- max_sequence_length=128,
- buffer_size_gb=0.1,
- block_size_tokens=16,
- max_tokens=1024,
- pipeline_model_parallel_size=pp_size,
- tensor_model_parallel_size=1,
- unified_memory_level=0,
+ model_config=TransformerConfig(
+ params_dtype=params_dtype,
+ num_layers=10,
+ kv_channels=64,
+ num_attention_heads=8,
+ pipeline_model_parallel_size=pp_size,
+ tensor_model_parallel_size=1,
+ pipeline_dtype=params_dtype,
+ ),
+ inference_config=InferenceConfig(
+ max_sequence_length=128,
+ buffer_size_gb=0.1,
+ block_size_tokens=16,
+ max_tokens=1024,
+ unified_memory_level=0,
+ ),
)
# Collect the total block counts on each rank
local_total_blocks = torch.tensor(
- [context.block_allocator.total_count], device='cuda', dtype=torch.long
+ [context.kv_block_allocator.total_count], device='cuda', dtype=torch.long
)
gathered_block_counts = [torch.zeros_like(local_total_blocks) for _ in range(pp_size)]
torch.distributed.all_gather(
@@ -1251,3 +1316,1491 @@ def test_pipeline_parallel_uneven_layers(self):
assert (
len(unique_counts) == 1
), f"Block counts were not synchronized across ranks. Gathered: {all_counts}"
+
+ self._restore_model_parallel()
+
+ @pytest.mark.internal
+ @pytest.mark.parametrize("ratio", [0.2, 0.4, 0.6, 0.8])
+ @rounder_override(64)
+ def test_mamba_memory_ratio_allocation(self, ratio):
+ """
+ Test that max_requests and block counts are partitioned correctly by mamba_memory_ratio.
+ """
+
+ buffer_gb = 0.05
+ paused_gb = 0.01
+ block_size = 256
+ num_attention_heads = 8
+ kv_channels = 64
+ params_dtype = torch.float32
+
+ layer_type_list = [Symbols.MAMBA, Symbols.ATTENTION]
+ mamba_conv_states_shape = (544, 4)
+ mamba_ssm_states_shape = (8, 64, 16)
+ mamba_config = MambaInferenceStateConfig(
+ layer_type_list,
+ mamba_conv_states_shape,
+ mamba_ssm_states_shape,
+ params_dtype,
+ params_dtype,
+ )
+
+ context = DynamicInferenceContext(
+ model_config=TransformerConfig(
+ params_dtype=params_dtype,
+ num_layers=2, # 1 Attn, 1 Mamba
+ kv_channels=kv_channels,
+ num_attention_heads=num_attention_heads,
+ ),
+ inference_config=InferenceConfig(
+ max_sequence_length=512,
+ buffer_size_gb=buffer_gb,
+ paused_buffer_size_gb=paused_gb,
+ block_size_tokens=block_size,
+ max_tokens=2048,
+ mamba_inference_state_config=mamba_config,
+ mamba_memory_ratio=ratio,
+ unified_memory_level=0,
+ ),
+ )
+
+ dtype_size = torch.tensor([], dtype=params_dtype).element_size()
+
+ mamba_mem_per_req = math.prod(mamba_conv_states_shape) + math.prod(mamba_ssm_states_shape)
+ mamba_mem_per_req *= dtype_size
+
+ kv_buffer_bytes = int(buffer_gb * 1024**3)
+ kv_paused_bytes = int(paused_gb * 1024**3)
+ total_mem_bytes = kv_buffer_bytes + kv_paused_bytes
+ expected_mamba_mem = total_mem_bytes * ratio
+ expected_mamba_max_reqs = int(expected_mamba_mem // mamba_mem_per_req)
+
+ # KV block calculation with buffer size reduced by Mamba memory ratio
+ kv_buffer_bytes = int(kv_buffer_bytes * (1.0 - ratio))
+ kv_paused_bytes = int(kv_paused_bytes * (1.0 - ratio))
+
+ kv_block_size_bytes = dtype_size * 2 * 1 * block_size * num_attention_heads * kv_channels
+
+ expected_active_blocks = kv_buffer_bytes // kv_block_size_bytes
+ expected_paused_blocks = kv_paused_bytes // kv_block_size_bytes
+ expected_total_blocks = expected_active_blocks + expected_paused_blocks
+
+ # Check that block allocator received the reduced block counts
+ assert context.kv_block_allocator.total_count == expected_active_blocks
+ assert context.kv_block_allocator.paused_count == expected_paused_blocks
+
+ # max_requests should be limited by the Mamba calculation if mamba_max_requests is smaller
+ # or the block count - 1 if that is smaller
+ expected_limit = min(expected_total_blocks - 1, expected_mamba_max_reqs)
+
+ # Apply rounding (REQUEST_ROUNDER = 64 in this test)
+ expected_max_requests = (expected_limit // 64) * 64
+
+ assert context.max_requests == expected_max_requests
+ assert context.is_hybrid_model is True
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_max_requests_less_than_tp_size(self):
+ tp_size = 2
+ self._setup_model_parallel_group(tensor_parallel_size=tp_size, pipeline_parallel_size=1)
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32,
+ num_layers=2,
+ kv_channels=64,
+ num_attention_heads=8,
+ tensor_model_parallel_size=tp_size,
+ )
+
+ inference_config = InferenceConfig(
+ max_sequence_length=512, buffer_size_gb=0.1, block_size_tokens=128, max_requests=1
+ )
+
+ with pytest.raises(AssertionError):
+ DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ self._restore_model_parallel()
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ @pytest.mark.parametrize("is_hybrid_model", [False, True])
+ @pytest.mark.parametrize("num_cuda_graphs", [-1, 16, 32])
+ @pytest.mark.parametrize("num_speculative_tokens", [0, 3])
+ def test_add_dummy_requests_for_expert_parallel_step_matches_slow_path(
+ self, is_hybrid_model: bool, num_cuda_graphs: int, num_speculative_tokens: int
+ ):
+ """The fast path (add_dummy_requests_for_expert_parallel_step) must leave
+ the same observable state as the slow path
+ (add_dummy_requests_for_cudagraph_capture(min(cuda_graph_dims))).
+ """
+
+ ctx = self._get_dynamic_context(
+ params_dtype=torch.float32,
+ num_layers=4,
+ kv_channels=8,
+ num_attention_heads=2,
+ max_sequence_length=512,
+ buffer_size_gb=0.03,
+ block_size_tokens=128,
+ max_tokens=None,
+ is_hybrid_model=is_hybrid_model,
+ layer_type_list=(
+ [Symbols.MAMBA, Symbols.ATTENTION, Symbols.MLP, Symbols.ATTENTION]
+ if is_hybrid_model
+ else None
+ ),
+ num_cuda_graphs=num_cuda_graphs,
+ num_speculative_tokens=num_speculative_tokens,
+ )
+
+ smallest = min(ctx.cuda_graph_batch_dimensions_list)
+ N = smallest.decode_req_count
+ T = smallest.token_count # N * (num_speculative_tokens + 1)
+ assert smallest.prefill_req_count == 0, "smallest graph must be decode-only"
+
+ # --- slow path (reference) ---
+ ctx.add_dummy_requests_for_cudagraph_capture(smallest)
+
+ slow_total_request_count = ctx.total_request_count
+ slow_active_token_count = ctx.active_token_count
+ slow_num_prefill_requests = ctx.num_prefill_requests
+ slow_request_query_lengths = ctx.request_query_lengths[:N].clone()
+ slow_request_kv_length_offsets = ctx.request_kv_length_offsets[:N].clone()
+ slow_request_to_kv_block_ids_col0 = ctx.request_to_kv_block_ids[:N, 0].clone()
+ slow_token_to_block_idx = ctx.token_to_block_idx[:T].clone()
+ slow_token_to_local_pos = ctx.token_to_local_position_within_kv_block[:T].clone()
+ if is_hybrid_model:
+ slow_token_to_request_idx = ctx.token_to_request_idx[:T].clone()
+ slow_mamba = ctx.mamba_metadata.request_to_mamba_state_idx[:N].clone()
+
+ # --- reset and run fast path ---
+ ctx.reset()
+ ctx.add_dummy_requests_for_expert_parallel_step()
+
+ # 1. Scalar counts
+ assert ctx.total_request_count == slow_total_request_count
+ assert ctx.active_token_count == slow_active_token_count
+ assert ctx.num_prefill_requests == slow_num_prefill_requests
+
+ # 2. Per-request MHA state
+ assert torch.equal(ctx.request_query_lengths[:N], slow_request_query_lengths)
+ assert torch.equal(ctx.request_kv_length_offsets[:N], slow_request_kv_length_offsets)
+ assert torch.equal(ctx.request_to_kv_block_ids[:N, 0], slow_request_to_kv_block_ids_col0)
+
+ # 3. Token-level state
+ dummy_block_idx = ctx.kv_block_allocator.dummy_block_idx
+ assert torch.all(ctx.token_to_block_idx[:T] == dummy_block_idx)
+ assert torch.equal(ctx.token_to_block_idx[:T], slow_token_to_block_idx)
+ assert torch.equal(ctx.token_to_local_position_within_kv_block[:T], slow_token_to_local_pos)
+
+ if is_hybrid_model:
+ # 4. token_to_request_idx
+ assert torch.equal(ctx.token_to_request_idx[:T], slow_token_to_request_idx)
+
+ # 5. Mamba state slots allocated (indices may differ, but must be valid and unique)
+ fast_mamba = ctx.mamba_metadata.request_to_mamba_state_idx[:N]
+ assert (fast_mamba >= 0).all(), "fast path should allocate valid mamba slots"
+ assert (slow_mamba >= 0).all(), "slow path should allocate valid mamba slots"
+ assert fast_mamba.unique().numel() == N, "fast path mamba slots must be unique"
+
+ @pytest.mark.internal
+ def test_gqa_high_tp_partition_heads(self):
+ """Tests that TP > GQA results in 1 attention head per partition."""
+ tp_size = 8
+ self._setup_model_parallel_group(tensor_parallel_size=tp_size, pipeline_parallel_size=1)
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32,
+ num_layers=2,
+ kv_channels=64,
+ num_attention_heads=16,
+ num_query_groups=2, # GQA = 2
+ tensor_model_parallel_size=tp_size,
+ )
+
+ # max_requests must be divisible by TP size (8) and REQUEST_ROUNDER
+ inference_config = InferenceConfig(
+ max_sequence_length=512, buffer_size_gb=0.1, block_size_tokens=128, max_requests=8
+ )
+
+ dynamic_context = DynamicInferenceContext(
+ model_config=model_config, inference_config=inference_config
+ )
+
+ # With TP=8 and GQA=2, num_attention_heads_per_partition should be clamped to 1
+ assert dynamic_context.num_attention_heads_per_partition == 1
+
+ self._restore_model_parallel()
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_chunked_prefill_state_preserved_across_decode_completions(self):
+ """
+ Tests that when a chunked prefill request is hidden, and active decode requests
+ finish (causing the context boundary to shrink), the hidden chunked request
+ is safely pulled to the new boundary so it doesn't lose its KV blocks or Mamba slot.
+ """
+
+ dynamic_context = self._get_dynamic_context(
+ params_dtype=torch.float32,
+ num_layers=2,
+ kv_channels=8,
+ num_attention_heads=2,
+ max_sequence_length=128,
+ buffer_size_gb=0.03,
+ block_size_tokens=4,
+ max_tokens=None,
+ is_hybrid_model=True,
+ layer_type_list=[Symbols.MAMBA, Symbols.ATTENTION],
+ enable_chunked_prefill=True,
+ )
+
+ # Add 2 normal decode requests
+ dynamic_context.add_request(
+ DynamicInferenceRequest(
+ request_id=10,
+ prompt_tokens=torch.arange(0, 2, device='cuda'),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ )
+ )
+ dynamic_context.add_request(
+ DynamicInferenceRequest(
+ request_id=11,
+ prompt_tokens=torch.arange(0, 2, device='cuda'),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ )
+ )
+
+ # Add Chunk 1 of the chunked prefill request
+ req_999 = DynamicInferenceRequest(
+ request_id=999,
+ prompt_tokens=torch.arange(0, 8, device='cuda'),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ )
+ dynamic_context.add_request(req_999, prefill_chunk_length=4)
+ dynamic_context.chunked_prefill_request_id = 999
+
+ # Capture the allocated state at index 2
+ mamba_slot_before = dynamic_context.mamba_metadata.request_to_mamba_state_idx[2].item()
+ kv_block_before = dynamic_context.request_to_kv_block_ids[2, 0].item()
+
+ assert mamba_slot_before != -1
+ assert kv_block_before != -1
+
+ # Step 1: Forward pass for all 3 requests
+ active_requests_mask = torch.tensor([1, 1, 1], dtype=torch.int32, device='cuda')
+ new_tokens = torch.tensor([100, 101, 102], dtype=torch.int32, device='cuda')
+ dynamic_context.update_requests(active_requests_mask, new_tokens)
+
+ # At this point, req 999 is hidden at index 2. total_request_count is 2 (req 10, 11).
+ assert dynamic_context.total_request_count == 2
+ assert dynamic_context.request_ids[2].item() == 999
+
+ # Step 2: Forward pass where req 10 finishes, req 11 continues. Req 999 is NOT scheduled.
+ active_requests_mask = torch.tensor([0, 1], dtype=torch.int32, device='cuda')
+ new_tokens = torch.tensor([103, 104], dtype=torch.int32, device='cuda')
+ dynamic_context.update_requests(active_requests_mask, new_tokens)
+
+ # At this point, req 10 is evicted. Req 11 shifts to index 0. total_request_count becomes 1.
+ # Req 999 should be pulled from index 2 down to index 1 (the new boundary).
+ assert dynamic_context.total_request_count == 1
+ assert dynamic_context.request_ids[0].item() == 11
+
+ # Verify that the chunked request was correctly pulled to the boundary (index 1)
+ assert dynamic_context.request_ids[1].item() == 999
+ assert (
+ dynamic_context.mamba_metadata.request_to_mamba_state_idx[1].item() == mamba_slot_before
+ )
+ assert dynamic_context.request_to_kv_block_ids[1, 0].item() == kv_block_before
+
+ # Ensure the old index 2 was properly swapped during the pull
+ assert dynamic_context.request_ids[2].item() == 11
+ assert dynamic_context.mamba_metadata.request_to_mamba_state_idx[2].item() == -1
+ assert dynamic_context.request_to_kv_block_ids[2, 0].item() == -1
+
+ # Step 3: Add the next chunk. It should sit exactly at the boundary (index 1) and inherit the state.
+ req_999.finished_chunk_token_count = 4
+ dynamic_context.add_request(req_999, prefill_chunk_length=4)
+
+ # Verify state at index 1 is active and its previous Mamba slot and KV blocks were inherited
+ assert dynamic_context.total_request_count == 2
+ assert dynamic_context.request_ids[1].item() == 999
+ assert (
+ dynamic_context.mamba_metadata.request_to_mamba_state_idx[1].item() == mamba_slot_before
+ )
+ assert dynamic_context.request_to_kv_block_ids[1, 0].item() == kv_block_before
+
+ @pytest.mark.internal
+ @rounder_override(4)
+ def test_chunked_prefill_all_active_requests_finish_while_hidden(self):
+ """
+ Tests that update_requests does not crash when ALL active decode requests
+ finish while a chunked prefill request is hidden (not scheduled this step).
+
+ This exercises the scenario where:
+ 1. A chunked prefill completes a chunk and is hidden at total_request_count
+ 2. The next chunk is not scheduled (e.g., no token budget)
+ 3. All remaining active decode requests finish in the same step
+ 4. active_request_count becomes 0 — the code must not assert-fail
+ """
+
+ dynamic_context = self._get_dynamic_context(
+ params_dtype=torch.float32,
+ num_layers=2,
+ kv_channels=8,
+ num_attention_heads=2,
+ max_sequence_length=128,
+ buffer_size_gb=0.03,
+ block_size_tokens=4,
+ max_tokens=None,
+ enable_chunked_prefill=True,
+ max_requests=16,
+ )
+
+ # Add 2 normal decode requests
+ dynamic_context.add_request(
+ DynamicInferenceRequest(
+ request_id=10,
+ prompt_tokens=torch.arange(0, 2, device='cuda'),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ )
+ )
+ dynamic_context.add_request(
+ DynamicInferenceRequest(
+ request_id=11,
+ prompt_tokens=torch.arange(0, 2, device='cuda'),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ )
+ )
+
+ # Add Chunk 1 of a chunked prefill request
+ req_999 = DynamicInferenceRequest(
+ request_id=999,
+ prompt_tokens=torch.arange(0, 8, device='cuda'),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ )
+ dynamic_context.add_request(req_999, prefill_chunk_length=4)
+ dynamic_context.chunked_prefill_request_id = 999
+
+ kv_block_before = dynamic_context.request_to_kv_block_ids[2, 0].item()
+ assert kv_block_before != -1
+
+ # Step 1: All 3 requests are active, process forward pass
+ active_requests_mask = torch.tensor([1, 1, 1], dtype=torch.int32, device='cuda')
+ new_tokens = torch.tensor([100, 101, 102], dtype=torch.int32, device='cuda')
+ dynamic_context.update_requests(active_requests_mask, new_tokens)
+
+ # Chunked prefill is now hidden at position 2, total_request_count = 2
+ assert dynamic_context.total_request_count == 2
+ assert dynamic_context.request_ids[2].item() == 999
+
+ # Step 2: Both decode requests finish, chunked prefill NOT scheduled this step.
+ # This must NOT crash even though active_request_count becomes 0.
+ active_requests_mask = torch.tensor([0, 0], dtype=torch.int32, device='cuda')
+ new_tokens = torch.tensor([103, 104], dtype=torch.int32, device='cuda')
+ dynamic_context.update_requests(active_requests_mask, new_tokens)
+
+ # total_request_count should be 0 (both finished, chunked prefill hidden)
+ assert dynamic_context.total_request_count == 0
+ assert dynamic_context.active_token_count == 0
+
+ # The hidden chunked prefill should be pulled to position 0 (the new boundary)
+ assert dynamic_context.request_ids[0].item() == 999
+ assert dynamic_context.request_to_kv_block_ids[0, 0].item() == kv_block_before
+
+ # Verify we can still add the next chunk at position 0
+ req_999.finished_chunk_token_count = 4
+ dynamic_context.add_request(req_999, prefill_chunk_length=4)
+
+ assert dynamic_context.total_request_count == 1
+ assert dynamic_context.request_ids[0].item() == 999
+ assert dynamic_context.request_to_kv_block_ids[0, 0].item() == kv_block_before
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_update_requests_speculative(self):
+ """Test update_requests correctly interleaves sampled and speculative tokens."""
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=128,
+ buffer_size_gb=0.01,
+ block_size_tokens=256,
+ num_speculative_tokens=2,
+ unified_memory_level=0,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ # Setup 2 active decode requests
+ ctx.total_request_count = 2
+ ctx.paused_request_count = 0
+ ctx.active_token_count = 2
+ ctx.request_ids[:2] = torch.tensor([10, 11])
+ ctx.request_query_lengths[:2] = 1
+ ctx.request_kv_length_offsets[:2] = torch.tensor([5, 8])
+ ctx.request_last_kv_block_offset[:2] = torch.tensor([5, 8])
+ ctx.request_to_kv_block_ids[:2, 0] = torch.tensor([0, 1])
+ ctx.request_last_kv_block_id[:2] = torch.tensor([0, 1])
+
+ active_requests_mask = torch.tensor([1, 1], device='cuda')
+ new_tokens = torch.tensor([99, 100], device='cuda') # Sampled tokens
+ new_speculative_tokens = torch.tensor(
+ [[991, 1001], [992, 1002]], device='cuda'
+ ) # Spec tokens
+
+ ctx.update_requests(
+ active_requests_mask=active_requests_mask,
+ new_tokens=new_tokens,
+ new_speculative_tokens=new_speculative_tokens,
+ )
+
+ # Each request generates 1 (sampled) + 2 (speculative) = 3 tokens.
+ assert ctx.active_token_count == 6
+ assert torch.equal(
+ ctx.request_query_lengths[:2], torch.tensor([3, 3], dtype=torch.int32, device='cuda')
+ )
+ assert torch.equal(
+ ctx.request_kv_length_offsets[:2],
+ torch.tensor([6, 9], dtype=torch.int32, device='cuda'),
+ )
+
+ # Check interleaving: [sampled_1, spec1_1, spec2_1, sampled_2, spec1_2, spec2_2]
+ expected_tokens = torch.tensor([99, 991, 992, 100, 1001, 1002], device='cuda')
+ assert torch.equal(ctx.token_to_input_ids[:6], expected_tokens)
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_speculative_boundary_crossing(self):
+ """Test token block assignment when speculative tokens cross a KV block boundary."""
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=1024,
+ buffer_size_gb=0.1,
+ block_size_tokens=256, # FA2-compatible block size to force boundary crossing
+ num_speculative_tokens=2,
+ unified_memory_level=0,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ # Setup 1 active decode request
+ ctx.total_request_count = 1
+ ctx.paused_request_count = 0
+ ctx.active_token_count = 1
+
+ ctx.request_ids[0] = 10
+ ctx.request_query_lengths[0] = 1
+ ctx.request_kv_block_counts[0] = 1
+
+ # Length is 254, meaning existing tokens are at indices 0..253.
+ # The last inserted token was at offset 253.
+ # Adding 3 tokens places them at offsets 254, 255, and 256 (crosses block size of 256).
+ ctx.request_kv_length_offsets[0] = 254
+ ctx.request_last_kv_block_offset[0] = 253
+
+ # Allocate one initial block manually
+ blocks = ctx.kv_block_allocator.allocate_memory_blocks(1)
+ first_block = blocks[0]
+ ctx.request_to_kv_block_ids[0, 0] = first_block
+ ctx.request_last_kv_block_id[0] = first_block
+
+ active_requests_mask = torch.tensor([1], device='cuda')
+ new_tokens = torch.tensor([50], device='cuda')
+ new_speculative_tokens = torch.tensor([[51], [52]], device='cuda')
+
+ # Run update_requests natively. It will automatically:
+ # 1. Detect the boundary crossing and pause the request.
+ # 2. Clone the prev_last_block_ids internally.
+ # 3. Resume the request, allocating the new block.
+ # 4. Map the 3 new tokens across the boundary.
+ ctx.update_requests(
+ active_requests_mask=active_requests_mask,
+ new_tokens=new_tokens,
+ new_speculative_tokens=new_speculative_tokens,
+ )
+
+ # Verify a new block was natively allocated by the resume logic
+ assert ctx.request_kv_block_counts[0] == 2
+ second_block = ctx.request_to_kv_block_ids[0, 1]
+ assert second_block != -1
+ assert second_block != first_block
+
+ # Expected token mapping for the 3 generated tokens (sampled, spec1, spec2)
+ # Token 0 (offset 2) -> first_block
+ # Token 1 (offset 3) -> first_block
+ # Token 2 (offset 4) -> second_block
+ expected_blocks = torch.tensor(
+ [first_block, first_block, second_block], dtype=torch.int, device='cuda'
+ )
+
+ assert torch.equal(ctx.token_to_block_idx[:3], expected_blocks)
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_paused_speculative_tokens_tracking(self):
+ """
+ Test that speculative tokens are correctly saved and concatenated
+ when requests are temporarily paused.
+ """
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=1024,
+ buffer_size_gb=0.1,
+ block_size_tokens=256,
+ num_speculative_tokens=2,
+ unified_memory_level=0,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ # Setup 2 active requests. Request 0 is about to overflow its block.
+ ctx.total_request_count = 2
+ ctx.paused_request_count = 0
+ ctx.active_token_count = 2
+ ctx.request_ids[:2] = torch.tensor([10, 11])
+ ctx.request_query_lengths[:2] = 1
+
+ # Request 0 is at offset 254. Adding 1 sampled + 2 spec = 3 tokens will push it to 257,
+ # which is >= block_size_tokens (256). It will require a new block.
+ # Request 1 is at offset 5. It will not require a new block.
+ ctx.request_kv_length_offsets[:2] = torch.tensor([254, 5])
+ ctx.request_last_kv_block_offset[:2] = torch.tensor([254, 5])
+ ctx.request_kv_block_counts[:2] = 1
+
+ # Allocate blocks
+ blocks = ctx.kv_block_allocator.allocate_memory_blocks(2)
+ ctx.request_to_kv_block_ids[0, 0] = blocks[0]
+ ctx.request_to_kv_block_ids[1, 0] = blocks[1]
+ ctx.request_last_kv_block_id[:2] = blocks
+
+ # Force the allocator to have no available blocks.
+ # This guarantees request 0 stays paused and cannot immediately resume.
+ ctx.kv_block_allocator.total_avail = 0
+ ctx.kv_block_allocator.paused_count = 100 # Ensure it doesn't get completely evicted either
+
+ active_requests_mask = torch.tensor([1, 1], device='cuda')
+ new_tokens = torch.tensor([99, 100], device='cuda') # Sampled
+ new_speculative_tokens = torch.tensor(
+ [[991, 1001], [992, 1002]], device='cuda'
+ ) # Speculative
+
+ # In update_requests, request 0 will be paused to allocate a new block.
+ # Since total_avail is 0, it will stay paused and its tokens will be cached.
+ ctx.update_requests(
+ active_requests_mask=active_requests_mask,
+ new_tokens=new_tokens,
+ new_speculative_tokens=new_speculative_tokens,
+ )
+
+ # Verify paused state was populated correctly
+ assert ctx.paused_tokens is not None
+ assert ctx.paused_speculative_tokens is not None
+
+ # Request 0 was the one paused, so its tokens should be shifted to
+ # index 0 of the paused tensors.
+ assert ctx.paused_request_count == 1
+ assert ctx.total_request_count == 2
+
+ assert ctx.paused_tokens[0].item() == 99
+ assert torch.equal(
+ ctx.paused_speculative_tokens[:, 0], torch.tensor([991, 992], device='cuda')
+ )
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_speculative_tokens_less_than_block_size_assert(self):
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=128,
+ buffer_size_gb=0.01,
+ block_size_tokens=256,
+ num_speculative_tokens=256,
+ unified_memory_level=0,
+ )
+ with pytest.raises(
+ AssertionError, match="num_speculative_tokens.*must be < block_size_tokens"
+ ):
+ DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_swap_book_keeping_tensors_with_speculative_tokens(self):
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=128,
+ buffer_size_gb=0.01,
+ block_size_tokens=256,
+ num_speculative_tokens=2,
+ unified_memory_level=0,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ ctx.request_ids[:2] = torch.tensor([10, 11])
+ next_tokens = torch.tensor([99, 100], device='cuda')
+ new_speculative_tokens = torch.tensor([[991, 1001], [992, 1002]], device='cuda')
+
+ ctx._swap_book_keeping_tensors(
+ src_idxs=torch.tensor([0]),
+ dst_idxs=torch.tensor([1]),
+ next_tokens=next_tokens,
+ new_speculative_tokens=new_speculative_tokens,
+ )
+
+ assert torch.equal(ctx.request_ids[:2], torch.tensor([11, 10], device='cuda'))
+ assert torch.equal(next_tokens[:2], torch.tensor([100, 99], device='cuda'))
+ assert torch.equal(
+ new_speculative_tokens[:, :2], torch.tensor([[1001, 991], [1002, 992]], device='cuda')
+ )
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_update_requests_with_finished_requests_and_speculative_tokens(self):
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=128,
+ buffer_size_gb=0.01,
+ block_size_tokens=32,
+ num_speculative_tokens=2,
+ unified_memory_level=0,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ # Setup 3 active requests: req0 (active), req1 (finished), req2 (active)
+ ctx.total_request_count = 3
+ ctx.paused_request_count = 0
+ ctx.active_token_count = 3
+ ctx.request_ids[:3] = torch.tensor([10, 11, 12])
+ ctx.request_query_lengths[:3] = 1
+ ctx.request_kv_length_offsets[:3] = torch.tensor([5, 8, 12])
+ ctx.request_last_kv_block_offset[:3] = torch.tensor([5, 8, 12])
+ ctx.request_to_kv_block_ids[:3, 0] = torch.tensor([0, 1, 2])
+ ctx.request_last_kv_block_id[:3] = torch.tensor([0, 1, 2])
+ ctx.request_kv_block_counts[:3] = 1
+
+ active_requests_mask = torch.tensor([1, 0, 1], device='cuda')
+ new_tokens = torch.tensor([99, 100, 101], device='cuda')
+ new_speculative_tokens = torch.tensor([[991, 1001, 1011], [992, 1002, 1012]], device='cuda')
+
+ ctx.update_requests(
+ active_requests_mask=active_requests_mask,
+ new_tokens=new_tokens,
+ new_speculative_tokens=new_speculative_tokens,
+ )
+
+ # req1 is finished. req2 moves to req1's position.
+ assert ctx.total_request_count == 2
+ assert torch.equal(
+ ctx.request_ids[:2], torch.tensor([10, 12], device='cuda', dtype=torch.int32)
+ )
+
+ # Check interleaving for req0 and req2
+ # req0: [99, 991, 992]
+ # req2: [101, 1011, 1012]
+ expected_tokens = torch.tensor([99, 991, 992, 101, 1011, 1012], device='cuda')
+ assert torch.equal(ctx.token_to_input_ids[:6], expected_tokens)
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_chunked_prefill_hidden_state_prevents_token_bloat(self):
+ """
+ Test that hiding the chunked prefill request effectively prevents
+ 'dummy' speculative tokens from bloating the active_token_count, and that the
+ next chunk seamlessly appends without needing legacy offset subtractions.
+ """
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=512,
+ buffer_size_gb=0.05,
+ block_size_tokens=128,
+ max_requests=256,
+ max_tokens=256,
+ num_speculative_tokens=3,
+ enable_chunked_prefill=True,
+ unified_memory_level=0,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+ ctx.reset_tensors()
+
+ # 1. Add a standard decode request
+ req_decode = DynamicInferenceRequest(
+ request_id=10,
+ prompt_tokens=torch.arange(0, 10, device='cuda'),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ )
+ ctx.add_request(req_decode)
+
+ # 2. Add chunk 1 of a chunked prefill request
+ req_chunked = DynamicInferenceRequest(
+ request_id=42,
+ prompt_tokens=torch.arange(0, 100, device='cuda'),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ )
+ ctx.chunked_prefill_request_id = 42
+ ctx.add_request(req_chunked, prefill_chunk_length=50)
+
+ # Verify initial active token count (10 from decode + 50 from prefill)
+ assert ctx.active_token_count == 60
+
+ # 3. Call update_requests
+ active_requests_mask = torch.tensor([1, 1], dtype=torch.int32, device='cuda')
+ new_tokens = torch.tensor([99, 199], dtype=torch.int32, device='cuda')
+ new_spec = torch.tensor(
+ [[100, 200], [101, 201], [102, 202]], dtype=torch.int32, device='cuda'
+ )
+
+ ctx.update_requests(
+ active_requests_mask=active_requests_mask,
+ new_tokens=new_tokens,
+ new_speculative_tokens=new_spec,
+ )
+
+ # 4. Verify Hiding Invariants:
+ # The chunked prefill request should be hidden safely out of bounds.
+ # The active_token_count should ONLY contain the decode request's tokens
+ # (1 base + 3 speculative = 4 tokens).
+ assert ctx.total_request_count == 1
+ assert ctx.active_token_count == 4
+ assert ctx.request_ids[1].item() == 42
+
+ # 5. Add chunk 2
+ req_chunked.finished_chunk_token_count = 50
+ ctx.add_request(req_chunked, prefill_chunk_length=50)
+
+ # 6. Verify seamless append (no legacy offset math needed)
+ # 4 active decode tokens + 50 new prefill tokens = 54
+ assert ctx.active_token_count == 54
+ assert ctx.total_request_count == 2
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_chunked_prefill_swap_with_speculative_tokens(self):
+ """Test that swapping a chunked prefill request to the end of the buffer
+ correctly brings along the 2D speculative tokens for the other decode requests.
+ """
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=128,
+ buffer_size_gb=0.01,
+ block_size_tokens=32,
+ num_speculative_tokens=2,
+ enable_chunked_prefill=True,
+ unified_memory_level=0,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ # Setup 2 active requests in the WRONG order (violating the invariant)
+ # Index 0: Chunked Prefill Request (ID 42)
+ # Index 1: Standard Decode Request (ID 99)
+ ctx.total_request_count = 2
+ ctx.paused_request_count = 0
+ ctx.active_token_count = 2
+
+ ctx.chunked_prefill_request_id = 42
+ ctx.request_ids[:2] = torch.tensor([42, 99])
+
+ # Status: 1 = Prefill, 0 = Decode
+ ctx.request_in_prefill_status_tensor[:2] = torch.tensor([1, 0])
+ ctx.request_query_lengths[:2] = 1
+ ctx.request_kv_length_offsets[:2] = torch.tensor([10, 20])
+ ctx.request_last_kv_block_offset[:2] = torch.tensor([10, 20])
+ ctx.request_to_kv_block_ids[:2, 0] = torch.tensor([0, 1])
+ ctx.request_last_kv_block_id[:2] = torch.tensor([0, 1])
+ ctx.request_kv_block_counts[:2] = 1
+
+ active_requests_mask = torch.tensor([1, 1], device='cuda')
+
+ # New base tokens: [100 (for prefill), 200 (for decode)]
+ new_tokens = torch.tensor([100, 200], device='cuda')
+
+ # New spec tokens: Col 0 for prefill (dummy), Col 1 for decode (real draft tokens)
+ new_speculative_tokens = torch.tensor([[101, 201], [102, 202]], device='cuda')
+
+ # Trigger update_requests.
+ # It must detect ID 42 is at index 0, and swap it with index 1.
+ ctx.update_requests(
+ active_requests_mask=active_requests_mask,
+ new_tokens=new_tokens,
+ new_speculative_tokens=new_speculative_tokens,
+ )
+
+ # 1. Verify the IDs were swapped successfully
+ assert torch.equal(
+ ctx.request_ids[:2], torch.tensor([99, 42], dtype=torch.int32, device='cuda')
+ )
+
+ # 2. Verify the Decode request (now at Index 0) correctly flattened its
+ # base token (200) AND its specific speculative tokens (201, 202).
+ # 3. Verify the Prefill request (now at Index 1) is hidden and does NOT
+ # flatten its dummy tokens.
+ expected_flattened_tokens = torch.tensor(
+ [200, 201, 202], device='cuda' # Decode request (ID 99)
+ )
+
+ assert ctx.active_token_count == 3
+ assert torch.equal(
+ ctx.token_to_input_ids[:3], expected_flattened_tokens
+ ), "Speculative tokens were not correctly flattened for the decode request!"
+
+ # 4. Verify that the new_speculative_tokens tensor itself was swapped so that
+ # the hidden state perfectly preserves the alignment for subsequent steps.
+ expected_swapped_spec_tokens = torch.tensor([[201, 101], [202, 102]], device='cuda')
+ assert torch.equal(
+ new_speculative_tokens, expected_swapped_spec_tokens
+ ), "new_speculative_tokens was not swapped in-place alongside the request metadata!"
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_speculative_with_prefix_caching_shared_blocks(self):
+ """Test that prefix caching correctly shares blocks when speculative decoding is enabled."""
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=512,
+ buffer_size_gb=0.1,
+ block_size_tokens=32,
+ num_speculative_tokens=2,
+ enable_prefix_caching=True,
+ unified_memory_level=0,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ bs = ctx.block_size_tokens
+ prompt = torch.arange(bs * 3, device='cuda')
+
+ # First request registers blocks.
+ req1 = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req1)
+ first_blocks = [ctx.request_to_kv_block_ids[0][i].item() for i in range(3)]
+ avail_after_first = ctx.kv_block_allocator.total_avail
+
+ # Second request with same prefix should share all blocks.
+ req2 = DynamicInferenceRequest(
+ request_id=2,
+ prompt_tokens=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req2)
+ second_blocks = [ctx.request_to_kv_block_ids[1][i].item() for i in range(3)]
+
+ # Blocks should be shared (same IDs, no pool consumption).
+ assert first_blocks == second_blocks
+ assert ctx.kv_block_allocator.total_avail == avail_after_first
+
+ # Ref counts should be 2.
+ for bid in first_blocks:
+ assert ctx.kv_block_allocator.block_ref_counts[bid].item() == 2
+
+ # Second request should skip prefix tokens (query_length == 1 for full match).
+ assert ctx.request_query_lengths[1].item() == 1
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_speculative_with_prefix_caching_kv_offset(self):
+ """Test that KV offset accounts for prefix skip when spec decoding is enabled."""
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=512,
+ buffer_size_gb=0.1,
+ block_size_tokens=32,
+ num_speculative_tokens=3,
+ enable_prefix_caching=True,
+ unified_memory_level=0,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ bs = ctx.block_size_tokens
+ prompt = torch.arange(bs * 2, device='cuda')
+
+ # First request.
+ req1 = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req1)
+
+ # Second request with same prefix: should have kv_offset = prefix_skip_tokens.
+ req2 = DynamicInferenceRequest(
+ request_id=2,
+ prompt_tokens=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req2)
+
+ # Full match: prefix_skip = min(2 * bs, 2*bs - 1) = 2*bs - 1
+ expected_skip = 2 * bs - 1
+ assert ctx.request_kv_length_offsets[1].item() == expected_skip
+ assert ctx.request_query_lengths[1].item() == 1
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_speculative_update_then_release_with_prefix_caching(self):
+ """Test that update_requests with spec tokens + block release respects ref counts."""
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=512,
+ buffer_size_gb=0.1,
+ block_size_tokens=4,
+ num_speculative_tokens=2,
+ enable_prefix_caching=True,
+ unified_memory_level=0,
+ max_requests=512,
+ max_tokens=512,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ bs = ctx.block_size_tokens
+ prompt = torch.arange(bs * 2, device='cuda')
+
+ # Two requests sharing the same prefix.
+ req1 = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req1)
+ shared_blocks = [ctx.request_to_kv_block_ids[0][i].item() for i in range(2)]
+
+ req2 = DynamicInferenceRequest(
+ request_id=2,
+ prompt_tokens=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req2)
+
+ # Verify initial ref counts are 2.
+ for bid in shared_blocks:
+ assert ctx.kv_block_allocator.block_ref_counts[bid].item() == 2
+
+ # Release one request. Ref counts should decrement to 1.
+ ctx.release_memory_blocks_from_request_indexes(torch.tensor([0]))
+ for bid in shared_blocks:
+ assert ctx.kv_block_allocator.block_ref_counts[bid].item() == 1
+
+ # Blocks should still be discoverable via hash map.
+ for bid in shared_blocks:
+ h = ctx.kv_block_allocator.block_hashes[bid].item()
+ assert h in ctx.kv_block_allocator.kv_hash_to_block_id
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_speculative_boundary_crossing_with_prefix_caching(self):
+ """Test block boundary crossing from speculative tokens does not corrupt shared blocks."""
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=512,
+ buffer_size_gb=0.1,
+ block_size_tokens=4,
+ num_speculative_tokens=2,
+ enable_prefix_caching=True,
+ unified_memory_level=0,
+ max_tokens=512,
+ max_requests=512,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ bs = ctx.block_size_tokens
+ prompt = torch.arange(bs * 2, device='cuda')
+
+ # Request 1: adds prefix blocks.
+ req1 = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req1)
+ shared_b0 = ctx.request_to_kv_block_ids[0][0].item()
+ shared_b1 = ctx.request_to_kv_block_ids[0][1].item()
+
+ # Request 2: shares prefix, gets its own decode block.
+ req2 = DynamicInferenceRequest(
+ request_id=2,
+ prompt_tokens=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req2)
+
+ # Both requests share the same 2 blocks.
+ assert ctx.request_to_kv_block_ids[1][0].item() == shared_b0
+ assert ctx.request_to_kv_block_ids[1][1].item() == shared_b1
+
+ # Set up request 0 for decode at offset that will cross block boundary.
+ # Place at offset (block_size - 1) in last block so adding 3 tokens crosses.
+ ctx.request_kv_length_offsets[0] = bs * 2 - 1 # one token from end of block 1
+ # The local offset of index 6 is (6 % bs)
+ ctx.request_last_kv_block_offset[0] = bs - 2
+ ctx.request_query_lengths[0] = 1
+ ctx.request_in_prefill_status_tensor[0] = 0
+ ctx.active_token_count = 2
+
+ active_mask = torch.tensor([1, 1], device='cuda', dtype=torch.int32)
+ new_tokens = torch.tensor([50, 50], device='cuda')
+ new_spec = torch.tensor([[51, 51], [52, 52]], device='cuda')
+
+ ctx.update_requests(
+ active_requests_mask=active_mask, new_tokens=new_tokens, new_speculative_tokens=new_spec
+ )
+
+ # A new block should have been allocated for the boundary crossing.
+ assert ctx.request_kv_block_counts[0] == 3
+ new_block = ctx.request_to_kv_block_ids[0][2].item()
+ assert new_block != -1
+ assert new_block != shared_b0
+ assert new_block != shared_b1
+
+ # Shared blocks should remain intact with ref count 2.
+ assert ctx.kv_block_allocator.block_ref_counts[shared_b0].item() == 2
+ assert ctx.kv_block_allocator.block_ref_counts[shared_b1].item() == 2
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_chunked_prefill_prefix_caching_from_hidden_state(self):
+ """Test prefix caching matching safely resolves from the hidden boundary state."""
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=512,
+ buffer_size_gb=0.1,
+ block_size_tokens=32,
+ max_requests=256,
+ max_tokens=256,
+ num_speculative_tokens=2,
+ enable_chunked_prefill=True,
+ enable_prefix_caching=True,
+ unified_memory_level=0,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+ ctx.reset_tensors()
+
+ bs = ctx.block_size_tokens
+
+ # First request: register prefix blocks (bs * 3 tokens = 3 complete blocks).
+ first_prompt = torch.arange(bs * 3, device='cuda')
+ req_first = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=first_prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req_first)
+
+ # Request 2: Chunked prefill sharing the same prefix
+ req2 = DynamicInferenceRequest(
+ request_id=42,
+ prompt_tokens=first_prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.chunked_prefill_request_id = 42
+
+ # Add chunk 1 (bs tokens)
+ req2.finished_chunk_token_count = 0
+ ctx.add_request(req2, prefill_chunk_length=bs)
+
+ # Call update_requests to move req2 to the hidden state
+ active_requests_mask = torch.tensor([1, 1], dtype=torch.int32, device='cuda')
+ new_tokens = torch.tensor([99, 199], dtype=torch.int32, device='cuda')
+ new_spec = torch.tensor([[100, 200], [101, 201]], dtype=torch.int32, device='cuda')
+ ctx.update_requests(active_requests_mask, new_tokens, new_speculative_tokens=new_spec)
+
+ # Capture active tokens before chunk 2 (which should just be the 3 tokens of req_first)
+ tokens_before_chunk_2 = ctx.active_token_count
+ assert tokens_before_chunk_2 == 3
+
+ # Add chunk 2 (bs * 2 tokens)
+ req2.finished_chunk_token_count = bs
+ chunk_length = bs * 2
+ ctx.add_request(req2, prefill_chunk_length=chunk_length)
+
+ # Prefix match should find 2 matching blocks (blocks 1 and 2 from req_first).
+ # With prefix match: 2 blocks matched -> skip (2*bs - 1) tokens
+ # effective_chunk_length = chunk_length - prefix_skip_tokens
+ # prefix_skip_tokens = min(2 * bs, chunk_length - 1) = 2 * bs - 1
+ prefix_skip = 2 * bs - 1
+ eff_chunk = chunk_length - prefix_skip
+
+ (_, _, _, _, prefix_skip, eff_chunk) = ctx._compute_prefix_match(req2, chunk_length)
+ expected_active = tokens_before_chunk_2 + eff_chunk
+ assert ctx.active_token_count == expected_active
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_prefix_caching_check_availability_with_speculative(self):
+ """Test check_availability accounts for prefix match when spec decoding is enabled."""
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=512,
+ buffer_size_gb=0.01,
+ block_size_tokens=32,
+ num_speculative_tokens=3,
+ enable_prefix_caching=True,
+ unified_memory_level=0,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ bs = ctx.block_size_tokens
+ prompt = torch.arange(bs * 2, device='cuda')
+
+ # First request registers blocks.
+ req1 = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req1)
+
+ # Exhaust the remaining pool.
+ while ctx.kv_block_allocator.total_avail > 0:
+ ctx.kv_block_allocator.allocate_memory_blocks(1)
+
+ # A new request with the same prefix should still be schedulable
+ # because prefix matching means 0 new blocks are needed from pool.
+ req2 = DynamicInferenceRequest(
+ request_id=2,
+ prompt_tokens=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ _, _, kv_available = ctx.check_availability(req2)
+ assert kv_available, "Matched blocks should not require pool allocation"
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_prefix_match_exact_block_boundary(self):
+ """Test prefix matching when the shared prefix is an exact multiple of the block size."""
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=512,
+ buffer_size_gb=0.1,
+ block_size_tokens=16,
+ enable_prefix_caching=True,
+ unified_memory_level=0,
+ max_tokens=512,
+ max_requests=512,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ bs = ctx.block_size_tokens
+
+ # req1: 32 tokens (exactly 2 complete blocks)
+ prompt1 = torch.arange(bs * 2, device='cuda')
+ req1 = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=prompt1,
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req1)
+
+ # req2: 35 tokens (first 32 tokens match req1)
+ prompt2 = torch.arange(bs * 2 + 3, device='cuda')
+ req2 = DynamicInferenceRequest(
+ request_id=2,
+ prompt_tokens=prompt2,
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req2)
+
+ # req2 should have 3 blocks total
+ assert ctx.request_kv_block_counts[1].item() == 3
+
+ # The first 2 blocks should be shared
+ assert ctx.request_to_kv_block_ids[1, 0].item() == ctx.request_to_kv_block_ids[0, 0].item()
+ assert ctx.request_to_kv_block_ids[1, 1].item() == ctx.request_to_kv_block_ids[0, 1].item()
+
+ # The 3rd block should be a newly allocated pool block
+ assert ctx.request_to_kv_block_ids[1, 2].item() != ctx.request_to_kv_block_ids[0, 1].item()
+
+ # The offset points to the last token (index 34). In the 3rd block (indices 32-47), 34 is at offset 2.
+ assert ctx.request_last_kv_block_offset[1].item() == 2
+
+ # Effective query length should be 3 (35 total - 32 skipped)
+ assert ctx.request_query_lengths[1].item() == 3
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_eviction_with_shared_prefix_blocks(self):
+ """Test that evicting a request drops ref counts correctly without destroying shared blocks."""
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=512,
+ buffer_size_gb=0.1,
+ block_size_tokens=16,
+ enable_prefix_caching=True,
+ unified_memory_level=0,
+ paused_buffer_size_gb=0.0, # 0 paused capacity to force immediate eviction
+ max_tokens=512,
+ max_requests=512,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ bs = ctx.block_size_tokens
+ prompt = torch.arange(bs * 2, device='cuda')
+
+ # Add req1 and req2 with identical prompts
+ req1 = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req1)
+
+ req2 = DynamicInferenceRequest(
+ request_id=2,
+ prompt_tokens=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req2)
+
+ shared_b0 = ctx.request_to_kv_block_ids[0, 0].item()
+ shared_b1 = ctx.request_to_kv_block_ids[0, 1].item()
+
+ # Both blocks should be safely shared with ref count 2
+ assert ctx.kv_block_allocator.block_ref_counts[shared_b0].item() == 2
+
+ # Mock the state to make req1 paused and req2 active
+ ctx.paused_request_count = 1
+ ctx.total_request_count = 2
+ ctx.request_ids[0] = 1
+ ctx.request_ids[1] = 2
+ ctx.request_kv_block_counts[0] = 2
+ ctx.request_kv_block_counts[1] = 2
+
+ # Exhaust the active block allocator
+ ctx.kv_block_allocator.total_avail = 0
+
+ # Trigger the eviction logic
+ # next_tokens must be sized to total_request_count (1 paused + 1 active = 2)
+ next_tokens = torch.tensor([50, 51], device='cuda')
+ evicted_ids = ctx.evict_overflow_paused_requests(
+ active_request_count=1, next_tokens=next_tokens
+ )
+
+ # req1 should be successfully evicted
+ assert evicted_ids is not None
+ assert evicted_ids[0].item() == 1
+
+ # req2 remains active, so the shared blocks should drop to a ref count of 1
+ assert ctx.kv_block_allocator.block_ref_counts[shared_b0].item() == 1
+ assert ctx.kv_block_allocator.block_ref_counts[shared_b1].item() == 1
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_oom_during_speculative_boundary_crossing(self):
+ """Test boundary crossing with speculative tokens pauses the request gracefully when KV cache is full, keeping other requests active."""
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=128,
+ buffer_size_gb=0.1,
+ block_size_tokens=16,
+ num_speculative_tokens=2,
+ unified_memory_level=0,
+ max_tokens=512,
+ max_requests=512,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+ bs = ctx.block_size_tokens
+
+ # Setup 2 active requests.
+ # Request 0 is exactly 1 token away from its boundary (will OOM).
+ # Request 1 has plenty of space (will remain active).
+ ctx.total_request_count = 2
+ ctx.paused_request_count = 0
+ ctx.active_token_count = 2
+
+ ctx.request_ids[:2] = torch.tensor([10, 11], device='cuda')
+ ctx.request_query_lengths[:2] = 1
+ ctx.request_kv_block_counts[:2] = 1
+
+ # Request 0 offset is 15. Adding 1 sampled + 2 spec = 3 tokens crosses the boundary (16).
+ # Request 1 offset is 5. Adding 3 tokens = 8 (does not cross).
+ ctx.request_kv_length_offsets[:2] = torch.tensor(
+ [bs - 1, 5], device='cuda', dtype=torch.int32
+ )
+ ctx.request_last_kv_block_offset[:2] = torch.tensor(
+ [bs - 1, 5], device='cuda', dtype=torch.int32
+ )
+
+ blocks = ctx.kv_block_allocator.allocate_memory_blocks(2)
+ ctx.request_to_kv_block_ids[0, 0] = blocks[0]
+ ctx.request_to_kv_block_ids[1, 0] = blocks[1]
+ ctx.request_last_kv_block_id[:2] = blocks
+
+ # Force OOM condition (no blocks left in the active pool)
+ ctx.kv_block_allocator.total_avail = 0
+ ctx.kv_block_allocator.paused_count = 100 # Prevent immediate eviction out of the system
+
+ active_mask = torch.tensor([1, 1], device='cuda', dtype=torch.int32)
+ new_tokens = torch.tensor([99, 88], device='cuda')
+ new_spec = torch.tensor([[100, 200], [101, 201]], device='cuda')
+
+ # Run update requests
+ ctx.update_requests(
+ active_requests_mask=active_mask, new_tokens=new_tokens, new_speculative_tokens=new_spec
+ )
+
+ # Request 0 should detect OOM, fail to allocate a new block, and pause.
+ # Request 1 remains active, so active_request_count goes 2 -> 1, avoiding the deadlock assert.
+ assert ctx.paused_request_count == 1
+ assert ctx.total_request_count == 2
+
+ # Request 1 generated 3 tokens (1 sampled + 2 spec)
+ assert ctx.active_token_count == 3
+
+ # Tokens must be cached in the paused buffers so Request 0 can resume cleanly later
+ assert ctx.paused_tokens is not None
+ assert ctx.paused_tokens[0].item() == 99
+
+ assert ctx.paused_speculative_tokens is not None
+ assert ctx.paused_speculative_tokens[0, 0].item() == 100
+ assert ctx.paused_speculative_tokens[1, 0].item() == 101
+
+ @pytest.mark.internal
+ @rounder_override(64)
+ def test_chunked_prefill_meets_prefix_caching(self):
+ """Test that chunks in a chunked-prefill pipeline properly hit the prefix cache mid-flight."""
+
+ model_config = TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=8, num_attention_heads=2
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=512,
+ buffer_size_gb=0.1,
+ block_size_tokens=32,
+ enable_chunked_prefill=True,
+ enable_prefix_caching=True,
+ unified_memory_level=0,
+ max_tokens=512,
+ max_requests=512,
+ )
+ ctx = DynamicInferenceContext(model_config=model_config, inference_config=inference_config)
+
+ bs = ctx.block_size_tokens
+ prompt = torch.arange(128, device='cuda')
+
+ # Cache req1 (fully processed)
+ req1 = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+ ctx.add_request(req1)
+ req1_blocks = [ctx.request_to_kv_block_ids[0, i].item() for i in range(4)]
+
+ # Start chunked prefill for req2.
+ req2 = DynamicInferenceRequest(
+ request_id=2,
+ prompt_tokens=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=bs,
+ enable_prefix_caching=True,
+ )
+
+ # Add the first chunk (64 tokens)
+ req2.finished_chunk_token_count = 0
+ ctx.chunked_prefill_request_id = 2
+ ctx.add_request(req2, prefill_chunk_length=64)
+
+ # Assert the first chunk perfectly matched the first 2 cached blocks
+ assert ctx.request_to_kv_block_ids[1, 0].item() == req1_blocks[0]
+ assert ctx.request_to_kv_block_ids[1, 1].item() == req1_blocks[1]
+ assert ctx.request_kv_block_counts[1].item() == 2
+
+ # Simulate update_requests completing the chunk
+ ctx.active_token_count += 1
+ ctx.request_in_prefill_status_tensor[1] = 0
+ ctx.total_request_count -= 1
+
+ # Add the second chunk (64 tokens)
+ req2.finished_chunk_token_count = 64
+ ctx.add_request(req2, prefill_chunk_length=64)
+
+ # It should correctly discover the remaining prefix blocks despite being mid-prefill
+ assert ctx.request_to_kv_block_ids[1, 2].item() == req1_blocks[2]
+ assert ctx.request_to_kv_block_ids[1, 3].item() == req1_blocks[3]
+ assert ctx.request_kv_block_counts[1].item() == 4
+
+ # Verify block references updated appropriately
+ assert ctx.kv_block_allocator.block_ref_counts[req1_blocks[2]].item() == 2
+ assert ctx.kv_block_allocator.block_ref_counts[req1_blocks[3]].item() == 2
diff --git a/tests/unit_tests/inference/contexts/test_dynamic_prefix_caching.py b/tests/unit_tests/inference/contexts/test_dynamic_prefix_caching.py
new file mode 100644
index 00000000000..8184aa401e7
--- /dev/null
+++ b/tests/unit_tests/inference/contexts/test_dynamic_prefix_caching.py
@@ -0,0 +1,1073 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+
+import asyncio
+from collections import deque
+
+import pytest
+import torch
+
+from megatron.core.inference.config import InferenceConfig, PrefixCachingEvictionPolicy
+from megatron.core.inference.contexts.dynamic_context import DynamicInferenceContext
+from megatron.core.inference.engines.dynamic_engine import DynamicInferenceEngine
+from megatron.core.inference.inference_request import (
+ HASH_PRIME,
+ DynamicInferenceRequest,
+ DynamicInferenceRequestRecord,
+ Status,
+ compute_block_hashes_batched,
+)
+from megatron.core.inference.sampling_params import SamplingParams
+from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
+from megatron.core.transformer.transformer_config import TransformerConfig
+from tests.unit_tests.test_utilities import Utils
+
+
+class PrefixCachingTestBase:
+
+ @classmethod
+ def setup_class(cls):
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=1, pipeline_model_parallel_size=1
+ )
+ model_parallel_cuda_manual_seed(123)
+
+ @classmethod
+ def teardown_class(cls):
+ Utils.destroy_model_parallel()
+
+ @staticmethod
+ def _mamba_config():
+ from megatron.core.inference.config import MambaInferenceStateConfig
+
+ return MambaInferenceStateConfig(
+ layer_type_list=["*", "M", "*", "M"],
+ conv_states_shape=(4, 8),
+ ssm_states_shape=(4, 16),
+ conv_states_dtype=torch.float32,
+ ssm_states_dtype=torch.float32,
+ )
+
+ def _ctx(
+ self,
+ *,
+ buffer_size_gb=0.1,
+ block_size_tokens=32,
+ max_sequence_length=512,
+ rounder=64,
+ enable_prefix_caching=True,
+ max_tokens=None,
+ prefix_caching_eviction_policy=PrefixCachingEvictionPolicy.LRU,
+ mamba_config=None,
+ prefix_caching_mamba_gb=None,
+ ):
+ DynamicInferenceContext.ROUNDER = rounder
+ DynamicInferenceContext.TOKEN_ROUNDER = rounder
+ DynamicInferenceContext.REQUEST_ROUNDER = rounder
+
+ transformer_config = TransformerConfig(
+ params_dtype=torch.float32,
+ num_layers=4,
+ kv_channels=8,
+ num_attention_heads=2,
+ hidden_size=16,
+ tensor_model_parallel_size=1,
+ pipeline_model_parallel_size=1,
+ use_cpu_initialization=True,
+ )
+ inference_config = InferenceConfig(
+ max_sequence_length=max_sequence_length,
+ buffer_size_gb=buffer_size_gb,
+ paused_buffer_size_gb=0.2 * buffer_size_gb,
+ block_size_tokens=block_size_tokens,
+ max_tokens=max_tokens,
+ mamba_inference_state_config=mamba_config,
+ use_flashinfer_fused_rope=None,
+ unified_memory_level=0,
+ enable_prefix_caching=enable_prefix_caching,
+ prefix_caching_eviction_policy=prefix_caching_eviction_policy,
+ prefix_caching_mamba_gb=prefix_caching_mamba_gb,
+ )
+ return DynamicInferenceContext(
+ model_config=transformer_config, inference_config=inference_config
+ )
+
+ @staticmethod
+ def _req(ctx, prompt_tokens, request_id=1, *, enable_prefix_caching=True):
+ return DynamicInferenceRequest(
+ request_id=request_id,
+ prompt_tokens=prompt_tokens,
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ block_size_tokens=ctx.block_size_tokens,
+ enable_prefix_caching=enable_prefix_caching,
+ )
+
+ @staticmethod
+ def _prompt(num_tokens, offset=0):
+ return torch.arange(offset, offset + num_tokens, device=torch.cuda.current_device())
+
+ @staticmethod
+ def _block_ids(ctx, req_idx, n):
+ return [ctx.request_to_kv_block_ids[req_idx][i].item() for i in range(n)]
+
+ @staticmethod
+ def _mamba_allocate_and_register(ctx, bids):
+ """Allocate Mamba cache slots and register hashes for a list of block IDs."""
+ msa = ctx.mamba_slot_allocator
+ alloc = ctx.kv_block_allocator
+ slots = msa.allocate_slots_batch(bids)
+ bid_tensor = torch.tensor(bids, dtype=torch.int64, device=alloc.block_hashes.device)
+ hashes = alloc.block_hashes[bid_tensor].tolist()
+ msa.register_block_hashes_batch(bids, hashes)
+ return slots
+
+
+class _StubEngine(DynamicInferenceEngine):
+
+ def __init__(self, context: DynamicInferenceContext, *, enable_chunked_prefill=False):
+ self.context = context
+ self.enable_chunked_prefill = enable_chunked_prefill
+ self._prefix_coordination_waits = 0
+ self._loop = asyncio.new_event_loop()
+ self.waiting_request_ids: deque = deque()
+ self.requests = {}
+ self._generation_epoch = None
+
+
+class TestPrefixCachingCore(PrefixCachingTestBase):
+
+ @pytest.mark.internal
+ def test_hash_computation(self):
+ # determinism and range
+ tokens = self._prompt(32)
+ h1 = compute_block_hashes_batched(tokens, 32)
+ h2 = compute_block_hashes_batched(tokens, 32)
+ assert h1 == h2 and len(h1) == 1 and 1 <= h1[0] <= HASH_PRIME
+ assert compute_block_hashes_batched(self._prompt(32, offset=1), 32)[0] != h1[0]
+
+ # parent chaining: 4 blocks of all-zero tokens produce distinct hashes
+ ctx = self._ctx()
+ bs = ctx.block_size_tokens
+ zeros = torch.zeros(bs * 4, device=torch.cuda.current_device(), dtype=torch.long)
+ hashes = compute_block_hashes_batched(zeros, bs)
+ assert len(hashes) == 4 and len(set(hashes)) == 4
+
+ # edge cases: short, empty, long
+ assert compute_block_hashes_batched(self._prompt(bs // 2), bs) == []
+ empty = torch.tensor([], device=torch.cuda.current_device(), dtype=torch.long)
+ assert compute_block_hashes_batched(empty, bs) == []
+ long_h = compute_block_hashes_batched(
+ torch.arange(bs * 120, device=torch.cuda.current_device(), dtype=torch.long), bs
+ )
+ assert len(long_h) == 120 and all(v > 0 for v in long_h)
+
+ @pytest.mark.internal
+ def test_registration_and_discovery(self):
+ ctx = self._ctx()
+ bs = ctx.block_size_tokens
+ alloc = ctx.kv_block_allocator
+
+ # blocks discoverable after add_request
+ prompt = self._prompt(bs * 2)
+ req = self._req(ctx, prompt)
+ ctx.add_request(req)
+ b0, b1 = self._block_ids(ctx, 0, 2)
+ h0, h1 = req.precomputed_block_hashes
+ assert alloc.kv_hash_to_block_id.get(h0) == b0
+ assert alloc.kv_hash_to_block_id.get(h1) == b1
+ assert alloc.block_hashes[b0].item() == h0 and alloc.block_hashes[b1].item() == h1
+
+ # partial block not registered
+ ctx2 = self._ctx()
+ alloc2 = ctx2.kv_block_allocator
+ ctx2.add_request(self._req(ctx2, self._prompt(bs + bs // 2)))
+ pb0, pb1 = self._block_ids(ctx2, 0, 2)
+ assert alloc2.block_hashes[pb0].item() != -1
+ assert alloc2.block_hashes[pb1].item() == -1
+
+ # decode does not register completed blocks
+ ctx3 = self._ctx()
+ alloc3 = ctx3.kv_block_allocator
+ ctx3.add_request(self._req(ctx3, self._prompt(bs + (bs - 1))))
+ db0, db1 = self._block_ids(ctx3, 0, 2)
+ assert alloc3.block_hashes[db0].item() != -1 and alloc3.block_hashes[db1].item() == -1
+ active_mask = torch.ones(1, device=torch.cuda.current_device(), dtype=torch.int32)
+ new_tokens = torch.tensor([100], device=torch.cuda.current_device())
+ ctx3.update_requests(active_mask, new_tokens)
+ assert alloc3.block_hashes[db1].item() == -1
+
+ # second request finds registered blocks
+ ctx4 = self._ctx()
+ alloc4 = ctx4.kv_block_allocator
+ p4 = self._prompt(bs * 3)
+ ctx4.add_request(self._req(ctx4, p4.clone()))
+ req2 = self._req(ctx4, p4.clone(), request_id=2)
+ for h in req2.precomputed_block_hashes:
+ assert h in alloc4.kv_hash_to_block_id
+
+ @pytest.mark.internal
+ def test_block_sharing_patterns(self):
+ ctx = self._ctx()
+ bs = ctx.block_size_tokens
+ alloc = ctx.kv_block_allocator
+
+ # N=10 identical prompts share all blocks
+ prompt = self._prompt(bs * 3)
+ ctx.add_request(self._req(ctx, prompt.clone()))
+ first_blocks = self._block_ids(ctx, 0, 3)
+ avail_after_first = alloc.total_avail
+ for i in range(2, 11):
+ ctx.add_request(self._req(ctx, prompt.clone(), request_id=i))
+ assert alloc.total_avail == avail_after_first
+ for req_idx in range(1, 10):
+ assert self._block_ids(ctx, req_idx, 3) == first_blocks
+ for bid in first_blocks:
+ assert alloc.block_ref_counts[bid].item() == 10
+
+ # divergent suffix shares common prefix
+ ctx2 = self._ctx()
+ alloc2 = ctx2.kv_block_allocator
+ p1 = self._prompt(bs * 3)
+ ctx2.add_request(self._req(ctx2, p1))
+ r1 = self._block_ids(ctx2, 0, 3)
+ p2 = p1.clone()
+ p2[bs * 2 :] += 1000
+ ctx2.add_request(self._req(ctx2, p2, request_id=2))
+ r2 = self._block_ids(ctx2, 1, 3)
+ assert r2[0] == r1[0] and r2[1] == r1[1] and r2[2] != r1[2]
+ assert alloc2.block_ref_counts[r1[0]].item() == 2
+ assert alloc2.block_ref_counts[r1[2]].item() == 1
+
+ # broken chain stops sharing: [X,W,Z] vs [X,Y,Z]
+ ctx3 = self._ctx()
+ alloc3 = ctx3.kv_block_allocator
+ p3a = self._prompt(bs * 3)
+ ctx3.add_request(self._req(ctx3, p3a))
+ r3a = self._block_ids(ctx3, 0, 3)
+ p3b = p3a.clone()
+ p3b[bs : bs * 2] += 5000
+ ctx3.add_request(self._req(ctx3, p3b, request_id=2))
+ r3b = self._block_ids(ctx3, 1, 3)
+ assert r3b[0] == r3a[0] and r3b[1] != r3a[1] and r3b[2] != r3a[2]
+ assert alloc3.block_ref_counts[r3a[0]].item() == 2
+
+ @pytest.mark.internal
+ def test_prefill_token_savings(self):
+ bs = 32
+
+ # enabled vs disabled
+ ctx_on = self._ctx()
+ prompt = self._prompt(bs * 4)
+ ctx_on.add_request(self._req(ctx_on, prompt.clone()))
+ ctx_on.add_request(self._req(ctx_on, prompt.clone(), request_id=2))
+ ctx_off = self._ctx(enable_prefix_caching=False)
+ ctx_off.add_request(self._req(ctx_off, prompt.clone(), enable_prefix_caching=False))
+ ctx_off.add_request(
+ self._req(ctx_off, prompt.clone(), request_id=2, enable_prefix_caching=False)
+ )
+ assert ctx_on.lifetime_prefill_token_count == bs * 4 + 1
+ assert ctx_off.lifetime_prefill_token_count == bs * 4 * 2
+
+ # partial match reduces proportionally
+ ctx2 = self._ctx()
+ p2a = self._prompt(bs * 3)
+ ctx2.add_request(self._req(ctx2, p2a.clone()))
+ p2b = p2a.clone()
+ p2b[bs * 2 :] += 1000
+ ctx2.add_request(self._req(ctx2, p2b, request_id=2))
+ assert ctx2.lifetime_prefill_token_count == bs * 3 + bs
+
+ # full match: recompute-based back-off (1 token recomputed per duplicate)
+ ctx3 = self._ctx()
+ alloc3 = ctx3.kv_block_allocator
+ p3 = self._prompt(bs * 3)
+ ctx3.add_request(self._req(ctx3, p3.clone()))
+ tokens_after = ctx3.active_token_count
+ first_blocks = self._block_ids(ctx3, 0, 3)
+ for i in range(5):
+ ctx3.add_request(self._req(ctx3, p3.clone(), request_id=i + 2))
+ assert ctx3.request_query_lengths[i + 1].item() == 1
+ assert ctx3.active_token_count - tokens_after == 5
+ for bid in first_blocks:
+ assert alloc3.block_ref_counts[bid].item() == 6
+ assert ctx3.lifetime_prefill_token_count == bs * 3 + 5
+
+ # no match: full prompt added
+ ctx4 = self._ctx()
+ ctx4.add_request(self._req(ctx4, self._prompt(bs * 2)))
+ ctx4.add_request(self._req(ctx4, self._prompt(bs * 2, offset=9000), request_id=2))
+ assert ctx4.lifetime_prefill_token_count == bs * 2 + bs * 2
+
+ @pytest.mark.internal
+ def test_block_allocation_with_prefix(self):
+ ctx = self._ctx()
+ bs = ctx.block_size_tokens
+ alloc = ctx.kv_block_allocator
+
+ # matched blocks not allocated from pool
+ prompt = self._prompt(bs * 4)
+ ctx.add_request(self._req(ctx, prompt.clone()))
+ first_blocks = self._block_ids(ctx, 0, 4)
+ avail = alloc.total_avail
+ ctx.add_request(self._req(ctx, prompt.clone(), request_id=2))
+ assert self._block_ids(ctx, 1, 4) == first_blocks and alloc.total_avail == avail
+
+ # extended prompt allocates only new blocks
+ ctx2 = self._ctx()
+ alloc2 = ctx2.kv_block_allocator
+ p2a = self._prompt(bs * 3)
+ ctx2.add_request(self._req(ctx2, p2a))
+ avail2 = alloc2.total_avail
+ p2b = torch.cat([p2a, self._prompt(bs * 2, offset=1000)])
+ ctx2.add_request(self._req(ctx2, p2b, request_id=2))
+ assert alloc2.total_avail == avail2 - 2
+
+ # check_availability accounts for prefix match
+ ctx3 = self._ctx(buffer_size_gb=0.01, rounder=1)
+ alloc3 = ctx3.kv_block_allocator
+ p3 = self._prompt(ctx3.block_size_tokens * 2)
+ ctx3.add_request(self._req(ctx3, p3.clone()))
+ while alloc3.total_avail > 0:
+ alloc3.allocate_memory_blocks(1)
+ _, _, kv_available = ctx3.check_availability(self._req(ctx3, p3.clone(), request_id=2))
+ assert kv_available
+
+ @pytest.mark.internal
+ def test_ref_count_lru(self):
+ ctx = self._ctx()
+ bs = ctx.block_size_tokens
+ alloc = ctx.kv_block_allocator
+ prompt = self._prompt(bs * 2)
+
+ # decrement preserves cached blocks
+ ctx.add_request(self._req(ctx, prompt.clone()))
+ ctx.add_request(self._req(ctx, prompt.clone(), request_id=2))
+ b0, b1 = self._block_ids(ctx, 0, 2)
+ b0_hash = alloc.block_hashes[b0].item()
+ assert alloc.block_ref_counts[b0].item() == 2
+ ctx.release_memory_blocks_from_request_indexes(torch.tensor([0]))
+ assert alloc.block_ref_counts[b0].item() == 1 and b0_hash in alloc.kv_hash_to_block_id
+ ctx.release_memory_blocks_from_request_indexes(torch.tensor([1]))
+ assert alloc.block_ref_counts[b0].item() == 0 and b0_hash in alloc.kv_hash_to_block_id
+
+ # cached blocks reused by new request
+ ctx2 = self._ctx()
+ alloc2 = ctx2.kv_block_allocator
+ p2 = self._prompt(bs * 2)
+ ctx2.add_request(self._req(ctx2, p2.clone()))
+ cb0, cb1 = self._block_ids(ctx2, 0, 2)
+ ctx2.release_memory_blocks_from_request_indexes(torch.tensor([0]))
+ ctx2.total_request_count = 0
+ assert alloc2.block_ref_counts[cb0].item() == 0
+ ctx2.add_request(self._req(ctx2, p2.clone(), request_id=2))
+ assert self._block_ids(ctx2, 0, 2) == [cb0, cb1]
+ assert alloc2.block_ref_counts[cb0].item() == 1
+
+ # eviction frees oldest cached first
+ ctx3 = self._ctx(buffer_size_gb=0.01, rounder=1)
+ alloc3 = ctx3.kv_block_allocator
+ ctx3.add_request(self._req(ctx3, self._prompt(bs * 2)))
+ active_blocks = ctx3.request_to_kv_block_ids[0][:2].clone()
+ ctx3.add_request(self._req(ctx3, self._prompt(bs * 2, offset=5000), request_id=2))
+ ctx3.release_memory_blocks_from_request_indexes(torch.tensor([1]))
+ ctx3.total_request_count = 1
+ for i in range(20):
+ try:
+ ctx3.add_request(
+ self._req(
+ ctx3, self._prompt(bs * 2, offset=(i + 10) * 1000), request_id=i + 100
+ )
+ )
+ except Exception:
+ break
+ for bid in active_blocks:
+ assert alloc3.block_ref_counts[bid.item()].item() == 1
+
+ @pytest.mark.internal
+ def test_ref_count_refzero(self):
+ bs = 32
+
+ # deregisters on last release
+ ctx = self._ctx(prefix_caching_eviction_policy=PrefixCachingEvictionPolicy.REF_ZERO)
+ alloc = ctx.kv_block_allocator
+ prompt = self._prompt(bs * 2)
+ ctx.add_request(self._req(ctx, prompt.clone()))
+ ctx.add_request(self._req(ctx, prompt.clone(), request_id=2))
+ b0, b1 = self._block_ids(ctx, 0, 2)
+ b0_hash = alloc.block_hashes[b0].item()
+ avail_before = alloc.total_avail
+ ctx.release_memory_blocks_from_request_indexes(torch.tensor([0]))
+ assert alloc.block_ref_counts[b0].item() == 1 and b0_hash in alloc.kv_hash_to_block_id
+ ctx.release_memory_blocks_from_request_indexes(torch.tensor([1]))
+ assert alloc.block_ref_counts[b0].item() == 0 and b0_hash not in alloc.kv_hash_to_block_id
+ assert alloc.block_hashes[b0].item() == -1 and alloc.block_hashes[b1].item() == -1
+ assert alloc.total_avail == avail_before + 2
+
+ # released blocks not discoverable
+ ctx2 = self._ctx(prefix_caching_eviction_policy=PrefixCachingEvictionPolicy.REF_ZERO)
+ alloc2 = ctx2.kv_block_allocator
+ p2 = self._prompt(bs * 2)
+ ctx2.add_request(self._req(ctx2, p2.clone()))
+ ctx2.release_memory_blocks_from_request_indexes(torch.tensor([0]))
+ ctx2.total_request_count = 0
+ ctx2.add_request(self._req(ctx2, p2.clone(), request_id=2))
+ new_blocks = self._block_ids(ctx2, 0, 2)
+ assert alloc2.block_ref_counts[new_blocks[0]].item() == 1
+
+
+class TestDisabledAndEngineScheduling(PrefixCachingTestBase):
+
+ def _engine(self, ctx, **kwargs):
+ return _StubEngine(ctx, **kwargs)
+
+ def _add_to_waiting(self, engine, ctx, req):
+ request_id = req.request_id
+ engine.requests[request_id] = type(
+ "Entry",
+ (),
+ {
+ "record": DynamicInferenceRequestRecord.from_request(req),
+ "future": engine._loop.create_future(),
+ },
+ )()
+ req.status = Status.ACTIVE_AND_GENERATING_TOKENS
+ req.sampling_params.num_tokens_to_generate = 10
+ engine.waiting_request_ids.append(request_id)
+
+ @pytest.mark.internal
+ def test_disabled_mode(self):
+ # no sharing
+ ctx = self._ctx(enable_prefix_caching=False)
+ bs = ctx.block_size_tokens
+ prompt = self._prompt(bs * 2)
+ ctx.add_request(self._req(ctx, prompt.clone(), enable_prefix_caching=False))
+ r1 = set(self._block_ids(ctx, 0, 2))
+ ctx.add_request(self._req(ctx, prompt.clone(), request_id=2, enable_prefix_caching=False))
+ r2 = set(self._block_ids(ctx, 1, 2))
+ assert r1.isdisjoint(r2)
+
+ # no caching attrs on disabled allocator
+ alloc_d = ctx.kv_block_allocator
+ assert not hasattr(alloc_d, 'block_hashes')
+ assert not hasattr(alloc_d, 'kv_hash_to_block_id')
+ assert not hasattr(alloc_d, 'block_ref_counts')
+
+ # REF_ZERO lacks timestamps
+ ctx_rz = self._ctx(prefix_caching_eviction_policy=PrefixCachingEvictionPolicy.REF_ZERO)
+ assert not hasattr(ctx_rz.kv_block_allocator, 'block_timestamps')
+
+ @pytest.mark.internal
+ def test_scheduling_deferral_and_resolution(self):
+ ctx = self._ctx()
+ bs = ctx.block_size_tokens
+ engine = self._engine(ctx)
+ prompt = self._prompt(bs * 2)
+
+ # shared prefix defers second request
+ req1 = self._req(ctx, prompt.clone())
+ self._add_to_waiting(engine, ctx, req1)
+ req2 = self._req(ctx, prompt.clone(), request_id=2)
+ self._add_to_waiting(engine, ctx, req2)
+ engine.schedule_non_chunked_prefill()
+ assert ctx.total_request_count == 1
+ assert len(engine.waiting_request_ids) == 1 and engine.waiting_request_ids[0] == 2
+ assert engine._prefix_coordination_waits == 1
+
+ # deferred request schedulable after registration (round 2)
+ engine.schedule_non_chunked_prefill()
+ assert ctx.total_request_count == 2 and len(engine.waiting_request_ids) == 0
+
+ # skip deferred to schedule non-conflicting
+ ctx2 = self._ctx()
+ engine2 = self._engine(ctx2)
+ pa = self._prompt(bs * 2)
+ pb = self._prompt(bs * 2, offset=5000)
+ self._add_to_waiting(engine2, ctx2, self._req(ctx2, pa.clone()))
+ self._add_to_waiting(engine2, ctx2, self._req(ctx2, pa.clone(), request_id=2))
+ self._add_to_waiting(engine2, ctx2, self._req(ctx2, pb.clone(), request_id=3))
+ engine2.schedule_non_chunked_prefill()
+ assert ctx2.total_request_count == 2
+ assert len(engine2.waiting_request_ids) == 1 and engine2.waiting_request_ids[0] == 2
+
+ # registered prefix allows immediate scheduling
+ ctx3 = self._ctx()
+ engine3 = self._engine(ctx3)
+ p3 = self._prompt(bs * 2)
+ ctx3.add_request(self._req(ctx3, p3.clone()))
+ self._add_to_waiting(engine3, ctx3, self._req(ctx3, p3.clone(), request_id=2))
+ engine3.schedule_non_chunked_prefill()
+ assert len(engine3.waiting_request_ids) == 0 and ctx3.total_request_count == 2
+ assert engine3._prefix_coordination_waits == 0
+
+ # metrics track deferrals
+ ctx4 = self._ctx()
+ engine4 = self._engine(ctx4)
+ p4 = self._prompt(bs * 2)
+ assert engine4.get_prefix_coordination_metrics() == {"waits": 0}
+ self._add_to_waiting(engine4, ctx4, self._req(ctx4, p4.clone()))
+ self._add_to_waiting(engine4, ctx4, self._req(ctx4, p4.clone(), request_id=2))
+ engine4.schedule_non_chunked_prefill()
+ assert engine4.get_prefix_coordination_metrics() == {"waits": 1}
+ engine4.schedule_non_chunked_prefill()
+ assert engine4.get_prefix_coordination_metrics() == {"waits": 1}
+ assert len(engine4.waiting_request_ids) == 0
+
+ @pytest.mark.internal
+ def test_chunked_prefill_deferral(self):
+ ctx = self._ctx()
+ bs = ctx.block_size_tokens
+ engine = self._engine(ctx, enable_chunked_prefill=True)
+ prompt = self._prompt(bs * 2)
+ self._add_to_waiting(engine, ctx, self._req(ctx, prompt.clone()))
+ self._add_to_waiting(engine, ctx, self._req(ctx, prompt.clone(), request_id=2))
+ engine.schedule_chunked_prefill()
+ assert ctx.total_request_count == 1
+ assert len(engine.waiting_request_ids) == 1 and engine._prefix_coordination_waits == 1
+
+
+class TestMambaPrefixCaching(PrefixCachingTestBase):
+
+ def _mctx(self, **kwargs):
+ defaults = dict(
+ mamba_config=self._mamba_config(),
+ prefix_caching_mamba_gb=0.01,
+ block_size_tokens=256,
+ max_sequence_length=4096,
+ )
+ defaults.update(kwargs)
+ return self._ctx(**defaults)
+
+ @pytest.mark.internal
+ def test_hybrid_memory_only(self):
+ # hybrid model: no prefill skipping, but blocks reused for memory savings
+ ctx = self._ctx(mamba_config=self._mamba_config())
+ bs = ctx.block_size_tokens
+ alloc = ctx.kv_block_allocator
+ prompt = self._prompt(bs * 3)
+ assert ctx.is_hybrid_model
+
+ req1 = self._req(ctx, prompt.clone())
+ ctx.add_request(req1)
+ first_blocks = self._block_ids(ctx, 0, 3)
+ avail = alloc.total_avail
+ tokens_after = ctx.active_token_count
+
+ req2 = self._req(ctx, prompt.clone(), request_id=2)
+ # no prefill skipping
+ (matched, _, _, _, prefix_skip, eff_chunk) = ctx._compute_prefix_match(req2, len(prompt))
+ assert len(matched) == 3 and prefix_skip == 0 and eff_chunk == len(prompt)
+
+ ctx.add_request(req2)
+ # blocks reused (pool unchanged), ref counts incremented
+ assert alloc.total_avail == avail
+ for bid in first_blocks:
+ assert alloc.block_ref_counts[bid].item() == 2
+ # all tokens processed (none skipped)
+ assert ctx.active_token_count - tokens_after == len(prompt)
+ assert ctx.request_kv_length_offsets[1].item() == 0
+
+ @pytest.mark.internal
+ def test_mamba_cache_lifecycle(self):
+ ctx = self._mctx()
+ bs = ctx.block_size_tokens
+
+ # allocated when prefix_caching_mamba_gb is set
+ assert ctx.mamba_slot_allocator.max_slots > 0
+ assert ctx.mamba_slot_allocator.conv_states is not None
+ assert ctx.mamba_slot_allocator.free_count == ctx.mamba_slot_allocator.max_slots
+
+ # not allocated when None
+ ctx_none = self._mctx(prefix_caching_mamba_gb=None)
+ assert ctx_none.mamba_slot_allocator is None
+
+ # store and restore round-trips
+ prompt = self._prompt(bs * 2)
+ req = self._req(ctx, prompt.clone())
+ ctx.add_request(req)
+ block_id = ctx.request_to_kv_block_ids[0][0].item()
+ slot = ctx.mamba_slot_allocator.allocate_slots_batch([block_id])[0]
+ for layer_idx in range(ctx.num_mamba_layers):
+ ssm = torch.ones_like(ctx.mamba_slot_allocator.ssm_states[layer_idx, slot]) * (
+ layer_idx + 1
+ )
+ conv = torch.ones_like(ctx.mamba_slot_allocator.conv_states[layer_idx, slot]) * (
+ layer_idx + 10
+ )
+ ctx.mamba_slot_allocator.store_from_tensors(block_id, layer_idx, ssm, conv)
+ assert ctx.mamba_slot_allocator.has_state(block_id)
+ req2 = self._req(ctx, prompt.clone(), request_id=2)
+ ctx.add_request(req2)
+ assert ctx.mamba_slot_allocator.restore_to_live(1, block_id)
+ mamba_idx = ctx.mamba_metadata.request_to_mamba_state_idx[1].item()
+ for layer_idx in range(ctx.num_mamba_layers):
+ assert torch.allclose(
+ ctx.mamba_ssm_states[layer_idx, mamba_idx],
+ torch.ones_like(ctx.mamba_ssm_states[layer_idx, mamba_idx]) * (layer_idx + 1),
+ )
+
+ # invalidate frees slot
+ ctx3 = self._mctx()
+ p3 = self._prompt(bs * 2)
+ ctx3.add_request(self._req(ctx3, p3.clone()))
+ bid3 = ctx3.request_to_kv_block_ids[0][0].item()
+ ctx3.mamba_slot_allocator.allocate_slots_batch([bid3])
+ assert ctx3.mamba_slot_allocator.has_state(bid3)
+ free_before = ctx3.mamba_slot_allocator.free_count
+ ctx3.mamba_slot_allocator.invalidate_block(bid3)
+ assert (
+ not ctx3.mamba_slot_allocator.has_state(bid3)
+ and ctx3.mamba_slot_allocator.free_count == free_before + 1
+ )
+
+ # slot reuse for same block
+ ctx4 = self._mctx()
+ ctx4.add_request(self._req(ctx4, self._prompt(bs * 2)))
+ bid4 = ctx4.request_to_kv_block_ids[0][0].item()
+ s1, s2 = ctx4.mamba_slot_allocator.allocate_slots_batch([bid4, bid4])
+ assert s1 == s2
+
+ # two-map hash design: kv and mamba maps are independent
+ ctx5 = self._mctx()
+ alloc5 = ctx5.kv_block_allocator
+ p5 = self._prompt(bs * 3)
+ ctx5.add_request(self._req(ctx5, p5.clone()))
+ msa5 = ctx5.mamba_slot_allocator
+ assert len(alloc5.kv_hash_to_block_id) == 3 and len(msa5.hash_to_block_id) == 0
+ self._mamba_allocate_and_register(ctx5, self._block_ids(ctx5, 0, 3)[:2])
+ assert len(alloc5.kv_hash_to_block_id) == 3 and len(msa5.hash_to_block_id) == 2
+
+ # find_mamba_match_count
+ ctx6 = self._mctx()
+ alloc6 = ctx6.kv_block_allocator
+ p6 = self._prompt(bs * 4)
+ ctx6.add_request(self._req(ctx6, p6.clone()))
+ msa6 = ctx6.mamba_slot_allocator
+ self._mamba_allocate_and_register(ctx6, self._block_ids(ctx6, 0, 4)[:2])
+ engine6 = _StubEngine(ctx6)
+ assert engine6._find_mamba_match_count(self._req(ctx6, p6.clone(), request_id=2)) == 2
+ # no match when no mamba hashes registered
+ ctx7 = self._mctx()
+ ctx7.add_request(self._req(ctx7, self._prompt(bs * 3)))
+ assert (
+ _StubEngine(ctx7)._find_mamba_match_count(
+ self._req(ctx7, self._prompt(bs * 3), request_id=2)
+ )
+ == 0
+ )
+
+ # allocate, free, re-allocate
+ ctx8 = self._mctx()
+ ctx8.add_request(self._req(ctx8, self._prompt(bs * 3)))
+ bids8 = self._block_ids(ctx8, 0, 3)
+ initial_free = ctx8.mamba_slot_allocator.free_count
+ ctx8.mamba_slot_allocator.allocate_slots_batch(bids8)
+ assert ctx8.mamba_slot_allocator.free_count == initial_free - 3
+ ctx8.mamba_slot_allocator.invalidate_block(bids8[0])
+ assert (
+ ctx8.mamba_slot_allocator.free_count == initial_free - 2
+ and not ctx8.mamba_slot_allocator.has_state(bids8[0])
+ )
+ ctx8.mamba_slot_allocator.allocate_slots_batch([bids8[0]])
+ assert (
+ ctx8.mamba_slot_allocator.free_count == initial_free - 3
+ and ctx8.mamba_slot_allocator.has_state(bids8[0])
+ )
+
+ @pytest.mark.internal
+ def test_mamba_prefill_skip_and_zero_prefill(self):
+ # mamba match limits prefill skip
+ ctx = self._mctx()
+ bs = ctx.block_size_tokens
+ alloc = ctx.kv_block_allocator
+ msa = ctx.mamba_slot_allocator
+ prompt = self._prompt(bs * 3)
+ ctx.add_request(self._req(ctx, prompt.clone()))
+ self._mamba_allocate_and_register(ctx, self._block_ids(ctx, 0, 3)[:1])
+ req2 = self._req(ctx, prompt.clone(), request_id=2)
+ req2._mamba_num_matched_blocks = 1
+ (matched, _, _, _, prefix_skip, eff_chunk) = ctx._compute_prefix_match(req2, len(prompt))
+ assert len(matched) == 3 and prefix_skip == bs and eff_chunk == len(prompt) - bs
+
+ # no mamba match means no skip
+ ctx2 = self._mctx()
+ p2 = self._prompt(bs * 3)
+ ctx2.add_request(self._req(ctx2, p2.clone()))
+ req2b = self._req(ctx2, p2.clone(), request_id=2)
+ req2b._mamba_num_matched_blocks = 0
+ (m2, _, _, _, ps2, ec2) = ctx2._compute_prefix_match(req2b, len(p2))
+ assert len(m2) == 3 and ps2 == 0 and ec2 == len(p2)
+
+ # zero prefill for hybrid (mamba-cached, block-aligned)
+ ctx3 = self._mctx()
+ p3 = self._prompt(bs * 3)
+ ctx3.add_request(self._req(ctx3, p3.clone()))
+ self._mamba_allocate_and_register(ctx3, self._block_ids(ctx3, 0, 3))
+ req3 = self._req(ctx3, p3.clone(), request_id=2)
+ req3._mamba_num_matched_blocks = 3
+ (m3, _, _, _, ps3, ec3) = ctx3._compute_prefix_match(req3, len(p3))
+ assert len(m3) == 3 and ps3 == 2 * bs and ec3 == bs
+
+ # recompute-based back-off for KV-only (block-aligned)
+ ctx4 = self._ctx()
+ bs4 = ctx4.block_size_tokens
+ p4 = self._prompt(bs4 * 3)
+ req4a = self._req(ctx4, p4.clone())
+ ctx4.add_request(req4a)
+ req4b = self._req(ctx4, p4.clone(), request_id=2)
+ (m4, _, _, _, ps4, ec4) = ctx4._compute_prefix_match(req4b, len(p4))
+ assert len(m4) == 3 and ps4 == 3 * bs4 - 1 and ec4 == 1
+ ctx4.add_request(req4b)
+
+ # KV eviction invalidates mamba
+ ctx5 = self._mctx(prefix_caching_eviction_policy=PrefixCachingEvictionPolicy.REF_ZERO)
+ alloc5 = ctx5.kv_block_allocator
+ msa5 = ctx5.mamba_slot_allocator
+ p5 = self._prompt(bs * 2)
+ ctx5.add_request(self._req(ctx5, p5.clone()))
+ bid5 = ctx5.request_to_kv_block_ids[0][0].item()
+ bh5 = alloc5.block_hashes[bid5].item()
+ self._mamba_allocate_and_register(ctx5, [bid5])
+ assert msa5.has_state(bid5) and bh5 in msa5.hash_to_block_id
+ ctx5.release_memory_blocks_from_request_indexes([0])
+ assert not msa5.has_state(bid5) and bh5 not in msa5.hash_to_block_id
+
+ @pytest.mark.internal
+ def test_mamba_intermediate_offsets(self):
+ bs = 256
+
+ # KV divergence offsets
+ ctx = self._mctx(block_size_tokens=bs)
+ prompt = self._prompt(bs * 4)
+ ctx.add_request(self._req(ctx, prompt.clone()))
+ msa = ctx.mamba_slot_allocator
+ self._mamba_allocate_and_register(ctx, self._block_ids(ctx, 0, 4)[:2])
+ req2 = self._req(ctx, prompt.clone(), request_id=2)
+ req2._mamba_num_matched_blocks = 2
+ (matched, _, _, overall, prefix_skip, _) = ctx._compute_prefix_match(req2, len(prompt))
+ # Copy block IDs to slot 1 so compute_and_store_offsets can resolve EOS block
+ ctx.request_to_kv_block_ids[1] = ctx.request_to_kv_block_ids[0]
+ msa.compute_and_store_offsets(
+ req2,
+ 1,
+ prefix_skip,
+ len(prompt),
+ len(matched),
+ [ctx.request_to_kv_block_ids[0][i].item() for i in range(len(matched))],
+ overall,
+ )
+ # Penultimate block offset (block 2 boundary) is a valid intermediate
+ count = msa._intermediate_counts_gpu[1].item()
+ if count > 0:
+ offsets = msa._intermediate_offsets_gpu[1, :count].tolist()
+ for o in offsets:
+ assert o > 0 and o % 128 == 0
+ assert msa._eos_cache_block_id_gpu[1].item() >= 0
+
+ # non-aligned prompt produces last_aligned intermediate offset
+ ctx2 = self._mctx(block_size_tokens=bs)
+ prompt_len = bs * 3 + bs // 2
+ p2 = self._prompt(prompt_len)
+ ctx2.add_request(self._req(ctx2, p2.clone()))
+ msa2 = ctx2.mamba_slot_allocator
+ self._mamba_allocate_and_register(ctx2, self._block_ids(ctx2, 0, 3)[:2])
+ req2b = self._req(ctx2, p2.clone(), request_id=2)
+ req2b._mamba_num_matched_blocks = 2
+ ctx2.add_request(req2b)
+ count2 = msa2._intermediate_counts_gpu[1].item()
+ if count2 > 0:
+ offsets = msa2._intermediate_offsets_gpu[1, :count2].tolist()
+ for o in offsets:
+ assert o > 0 and o % 128 == 0
+ assert msa2._eos_cache_block_id_gpu[1].item() < 0
+
+ # block-aligned prompts set EOS cache block ID
+ ctx3 = self._mctx(block_size_tokens=bs)
+ p3 = self._prompt(bs * 3)
+ ctx3.add_request(self._req(ctx3, p3.clone()))
+ req3 = self._req(ctx3, p3.clone(), request_id=2)
+ req3._mamba_num_matched_blocks = 0
+ ctx3.add_request(req3)
+ assert ctx3.mamba_slot_allocator._eos_cache_block_id_gpu[1].item() >= 0
+
+ # intermediate output buffers are pre-allocated
+ ctx4 = self._mctx()
+ msa4 = ctx4.mamba_slot_allocator
+ assert msa4.intermediate_ssm_out.shape[0] == ctx4.num_mamba_layers
+ assert msa4.intermediate_conv_out.shape[0] == ctx4.num_mamba_layers
+ assert msa4.intermediate_ssm_out.shape[1] == msa4.max_intermediate_count
+
+ # store_from_live copies all layers
+ ctx5 = self._mctx()
+ msa5 = ctx5.mamba_slot_allocator
+ p5 = self._prompt(ctx5.block_size_tokens * 2)
+ ctx5.add_request(self._req(ctx5, p5.clone()))
+ bid5 = ctx5.request_to_kv_block_ids[0][0].item()
+ slot5 = msa5.allocate_slots_batch([bid5])[0]
+ mamba_idx = ctx5.mamba_metadata.request_to_mamba_state_idx[0].item()
+ for layer in range(ctx5.num_mamba_layers):
+ ctx5.mamba_conv_states[layer, mamba_idx] = layer + 1.0
+ ctx5.mamba_ssm_states[layer, mamba_idx] = layer + 100.0
+ msa5.store_from_live_batch([slot5], [0])
+ for layer in range(ctx5.num_mamba_layers):
+ assert torch.allclose(
+ ctx5.mamba_slot_allocator.conv_states[layer, slot5],
+ torch.full_like(ctx5.mamba_slot_allocator.conv_states[layer, slot5], layer + 1.0),
+ )
+
+
+class TestMixedCachedAndFreshPrefill(PrefixCachingTestBase):
+
+ def _setup_mixed_batch(self, model_type):
+ """Set up mixed batch: req0 (decode), reqs 1-4 (mixed cached/fresh prefill)."""
+ if model_type == "gpt":
+ ctx = self._ctx(block_size_tokens=32)
+ else:
+ ctx = self._ctx(
+ mamba_config=self._mamba_config(),
+ prefix_caching_mamba_gb=0.01,
+ block_size_tokens=256,
+ max_sequence_length=4096,
+ )
+ bs = ctx.block_size_tokens
+
+ prompt0 = self._prompt(bs)
+ req0 = self._req(ctx, prompt0.clone())
+ ctx.add_request(req0)
+
+ vocab_size = bs + 50
+ block_hash = req0.precomputed_block_hashes[0]
+
+ if model_type == "hybrid":
+ block_id = ctx.kv_block_allocator.kv_hash_to_block_id[block_hash]
+ ctx.mamba_slot_allocator.register_block_hashes_batch([block_id], [block_hash])
+
+ ctx.request_kv_length_offsets[0] += bs
+ ctx.request_query_lengths[0] = 1
+ ctx.request_last_kv_block_offset[0] = 0
+ ctx.num_prefill_requests = 0
+ ctx.active_token_count = 1
+ ctx.token_to_input_ids[0] = 42
+ ctx.token_to_pos_ids[0] = bs
+ ctx.token_to_request_idx[0] = 0
+
+ req1 = self._req(ctx, prompt0.clone(), request_id=2)
+ req2 = self._req(ctx, self._prompt(bs, offset=50), request_id=3)
+ req3 = self._req(ctx, prompt0.clone(), request_id=4)
+ req4 = self._req(ctx, self._prompt(bs, offset=40), request_id=5)
+
+ if model_type == "hybrid":
+ req1._mamba_num_matched_blocks = 1
+ req2._mamba_num_matched_blocks = 0
+ req3._mamba_num_matched_blocks = 1
+ req4._mamba_num_matched_blocks = 0
+
+ for r in [req1, req2, req3, req4]:
+ ctx.add_request(r)
+
+ return ctx, bs, vocab_size, block_hash
+
+ @pytest.mark.parametrize("model_type", ["gpt", "hybrid"])
+ @pytest.mark.internal
+ def test_mixed_batch(self, model_type):
+ ctx, bs, vocab_size, block_hash = self._setup_mixed_batch(model_type)
+
+ # For GPT: req1/req3 (identical, 1-block) have effective_prefill_chunk_length=1
+ # For hybrid: req1/req3 have 1 mamba match but 1-block prompt, back-off
+ # finds no previous block, so prefix_skip_tokens=0, effective=bs
+ if model_type == "gpt":
+ cached_ql = 1
+ else:
+ cached_ql = bs
+
+ # query lengths: decode=1, cached_recompute=cached_ql, fresh=bs
+ assert ctx.request_query_lengths[0].item() == 1
+ assert ctx.request_query_lengths[1].item() == cached_ql
+ assert ctx.request_query_lengths[2].item() == bs
+ assert ctx.request_query_lengths[3].item() == cached_ql
+ assert ctx.request_query_lengths[4].item() == bs
+ assert ctx.active_token_count == 1 + 2 * cached_ql + 2 * bs
+
+ # last_token_logits
+ ctx.initialize_attention_state()
+ logits = torch.randn(
+ 1, ctx.padded_active_token_count, vocab_size, device=torch.cuda.current_device()
+ )
+ result = ctx.last_token_logits(logits)
+ assert result.shape == (5, vocab_size)
+
+ # calculate_log_probs
+ new_tokens = torch.randint(0, vocab_size, (5,), device=torch.cuda.current_device())
+ log_probs_list, _ = ctx.calculate_log_probs(logits, new_tokens)
+ assert len(log_probs_list) == 5
+ assert len(log_probs_list[0]) == 1
+ assert len(log_probs_list[1]) == cached_ql
+ assert len(log_probs_list[2]) == bs
+ assert len(log_probs_list[3]) == cached_ql
+ assert len(log_probs_list[4]) == bs
+
+
+class TestMambaSlotAllocator(PrefixCachingTestBase):
+
+ def _mctx(self, **kwargs):
+ defaults = dict(
+ mamba_config=self._mamba_config(),
+ prefix_caching_mamba_gb=0.01,
+ block_size_tokens=256,
+ max_sequence_length=4096,
+ )
+ defaults.update(kwargs)
+ return self._ctx(**defaults)
+
+ @pytest.mark.internal
+ def test_allocate_slots_batch(self):
+ ctx = self._mctx()
+ bs = ctx.block_size_tokens
+ msa = ctx.mamba_slot_allocator
+
+ # Basic batch: allocate 3 new slots, verify unique slots and mappings
+ prompt = self._prompt(bs * 4)
+ ctx.add_request(self._req(ctx, prompt.clone()))
+ bids = self._block_ids(ctx, 0, 3)
+ initial_free = msa.free_count
+ slots = msa.allocate_slots_batch(bids)
+ assert len(slots) == 3
+ assert len(set(slots)) == 3 # all unique
+ assert msa.free_count == initial_free - 3
+ for bid, slot in zip(bids, slots):
+ assert msa.block_to_slot[bid].item() == slot
+ assert msa.slot_to_block[slot].item() == bid
+
+ # Existing slots: same block_ids return same slots without consuming pool
+ free_before = msa.free_count
+ slots2 = msa.allocate_slots_batch(bids)
+ assert slots2 == slots
+ assert msa.free_count == free_before
+
+ # Dedup: same block_id twice, only one free slot consumed
+ ctx2 = self._mctx()
+ ctx2.add_request(self._req(ctx2, self._prompt(bs * 2)))
+ bid_new = self._block_ids(ctx2, 0, 1)[0]
+ msa2 = ctx2.mamba_slot_allocator
+ free_before2 = msa2.free_count
+ dup_slots = msa2.allocate_slots_batch([bid_new, bid_new])
+ assert dup_slots[0] == dup_slots[1]
+ assert msa2.free_count == free_before2 - 1
+
+ # Mixed: pre-allocated + new in one call
+ ctx3 = self._mctx()
+ ctx3.add_request(self._req(ctx3, self._prompt(bs * 3)))
+ bids3 = self._block_ids(ctx3, 0, 3)
+ msa3 = ctx3.mamba_slot_allocator
+ pre_slot = msa3.allocate_slots_batch([bids3[0]])[0]
+ free_before3 = msa3.free_count
+ mixed_slots = msa3.allocate_slots_batch(bids3)
+ assert mixed_slots[0] == pre_slot
+ assert msa3.free_count == free_before3 - 2 # only 2 new
+
+ # Eviction: exhaust free pool, verify eviction fires and returns valid slots
+ ctx4 = self._mctx(prefix_caching_mamba_gb=0.001)
+ msa4 = ctx4.mamba_slot_allocator
+ total_slots = msa4.max_slots
+ ctx4.add_request(self._req(ctx4, self._prompt(bs * 4)))
+ bids4 = self._block_ids(ctx4, 0, 4)
+ # Allocate all available slots by filling the free pool
+ fill_bids = bids4[: min(total_slots, 4)]
+ fill_slots = msa4.allocate_slots_batch(fill_bids)
+ assert len(fill_slots) == len(fill_bids)
+ # If we can exhaust the pool, test eviction
+ if total_slots <= 4:
+ assert msa4.free_count == 0
+ # Set ref counts to 0 so blocks are evictable
+ for bid in fill_bids:
+ ctx4.kv_block_allocator.block_ref_counts[bid] = 0
+ # Invalidate old slots, then reallocate to test eviction path
+ for bid in fill_bids:
+ msa4.invalidate_block(bid)
+ evict_slots = msa4.allocate_slots_batch(fill_bids)
+ assert len(evict_slots) == len(fill_bids)
+
+ @pytest.mark.internal
+ def test_commit_intermediate_states_batched(self):
+ ctx = self._mctx(block_size_tokens=256)
+ bs = ctx.block_size_tokens
+ msa = ctx.mamba_slot_allocator
+ alloc = ctx.kv_block_allocator
+ metadata = ctx.mamba_metadata
+
+ # Set up context with a prefill request that has block-aligned prompt
+ prompt = self._prompt(bs * 3)
+ req = self._req(ctx, prompt.clone())
+ req._mamba_num_matched_blocks = 0
+ ctx.add_request(req)
+
+ # initialize_attention_state sets batch_dimensions and mamba metadata
+ ctx.initialize_attention_state()
+
+ # Determine prefill_start for this batch
+ prefill_start = ctx.paused_request_count + ctx.batch_dimensions.decode_req_count
+ ctx_idx = prefill_start # first prefill request
+
+ # Write known patterns to intermediate output buffers
+ for layer in range(ctx.num_mamba_layers):
+ msa.intermediate_ssm_out[layer, 0] = layer + 1.0
+ msa.intermediate_conv_out[layer, 0] = layer + 100.0
+
+ # Set up intermediate offsets: 1 intermediate at src_offset=0
+ bid0 = ctx.request_to_kv_block_ids[ctx_idx][0].item()
+ msa._intermediate_block_ids_gpu[ctx_idx, 0] = bid0
+ msa._intermediate_offsets_gpu[ctx_idx, 0] = 128
+ msa._intermediate_counts_gpu[ctx_idx] = 1
+ msa._has_intermediates = True
+
+ # Set metadata fields that would normally be set by _update_intermediate_offsets
+ metadata.intermediate_count = 1
+ metadata.per_request_intermediate_counts = [1]
+
+ # Set up EOS block (block-aligned prompt)
+ eos_bid = ctx.request_to_kv_block_ids[ctx_idx][2].item()
+ msa._eos_cache_block_id_gpu[ctx_idx] = eos_bid
+
+ # Write known patterns to live mamba state for EOS copy
+ mamba_idx = metadata.request_to_mamba_state_idx[ctx_idx].item()
+ for layer in range(ctx.num_mamba_layers):
+ ctx.mamba_conv_states[layer, mamba_idx] = layer + 200.0
+ ctx.mamba_ssm_states[layer, mamba_idx] = layer + 300.0
+
+ # Call the batched commit
+ msa.commit_intermediate_states()
+
+ # Verify intermediate state was copied to correct slot
+ slot0 = msa.block_to_slot[bid0].item()
+ assert slot0 >= 0
+ for layer in range(ctx.num_mamba_layers):
+ assert torch.allclose(
+ msa.ssm_states[layer, slot0],
+ torch.full_like(msa.ssm_states[layer, slot0], layer + 1.0),
+ )
+ assert torch.allclose(
+ msa.conv_states[layer, slot0],
+ torch.full_like(msa.conv_states[layer, slot0], layer + 100.0),
+ )
+
+ # Verify EOS state was copied from live buffer
+ eos_slot = msa.block_to_slot[eos_bid].item()
+ assert eos_slot >= 0
+ for layer in range(ctx.num_mamba_layers):
+ assert torch.allclose(
+ msa.conv_states[layer, eos_slot],
+ torch.full_like(msa.conv_states[layer, eos_slot], layer + 200.0),
+ )
+ assert torch.allclose(
+ msa.ssm_states[layer, eos_slot],
+ torch.full_like(msa.ssm_states[layer, eos_slot], layer + 300.0),
+ )
+
+ # Verify hash_to_block_id updated for valid hashes
+ bid0_hash = alloc.block_hashes[bid0].item()
+ eos_hash = alloc.block_hashes[eos_bid].item()
+ if bid0_hash > 0:
+ assert msa.hash_to_block_id.get(bid0_hash) == bid0
+ if eos_hash > 0:
+ assert msa.hash_to_block_id.get(eos_hash) == eos_bid
+
+ # Verify _has_intermediates cleared
+ assert not msa._has_intermediates
diff --git a/tests/unit_tests/inference/coordinator_test_utils.py b/tests/unit_tests/inference/coordinator_test_utils.py
new file mode 100644
index 00000000000..d33d8790ef9
--- /dev/null
+++ b/tests/unit_tests/inference/coordinator_test_utils.py
@@ -0,0 +1,77 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""Shared test fixtures and helpers for inference tests."""
+
+import itertools
+from collections import deque
+
+import numpy as np
+
+from megatron.core.inference.config import PrefixCachingCoordinatorPolicy
+from megatron.core.inference.data_parallel_inference_coordinator import (
+ DataParallelInferenceCoordinator,
+)
+
+
+def make_coordinator_direct(
+ data_parallel_size=2,
+ block_size_tokens=4,
+ enable_prefix_caching=True,
+ deterministic_mode=True,
+ prefix_caching_routing_alpha=0.5,
+ max_requests=10,
+ policy=PrefixCachingCoordinatorPolicy.LONGEST_PREFIX,
+ tokenizer=None,
+ rank_name_template="rank_{}",
+):
+ """Create a coordinator with mock ZMQ, for unit testing routing logic.
+
+ Returns the coordinator instance with fake rank identities.
+
+ Args:
+ data_parallel_size: Number of DP ranks.
+ block_size_tokens: Block size in tokens.
+ enable_prefix_caching: Whether prefix caching is enabled.
+ deterministic_mode: If True, sort identities for deterministic ordering.
+ prefix_caching_routing_alpha: Alpha for prefix-aware scoring.
+ max_requests: Max requests per rank (None disables vectorized scoring).
+ policy: Prefix caching coordinator routing policy.
+ tokenizer: Optional tokenizer instance (set on the coordinator).
+ rank_name_template: Format string for rank names, e.g. ``"rank_{}"``
+ or ``"rank-{}"``. The integer rank index is substituted.
+ """
+ coordinator = object.__new__(DataParallelInferenceCoordinator)
+ coordinator.tokenizer = tokenizer
+ coordinator.data_parallel_size = data_parallel_size
+ coordinator.block_size_tokens = block_size_tokens
+ coordinator.enable_prefix_caching = enable_prefix_caching
+ coordinator.prefix_caching_coordinator_policy = policy
+ coordinator.prefix_caching_routing_alpha = prefix_caching_routing_alpha
+ coordinator.max_requests = max_requests
+
+ # Create fake rank identities.
+ coordinator.identities_of_data_parallel_ranks = deque(
+ [rank_name_template.format(i).encode() for i in range(data_parallel_size)]
+ )
+ if deterministic_mode:
+ coordinator.identities_of_data_parallel_ranks = deque(
+ sorted(coordinator.identities_of_data_parallel_ranks)
+ )
+ coordinator.data_parallel_rank_iterator = itertools.cycle(
+ coordinator.identities_of_data_parallel_ranks
+ )
+
+ n_ranks = data_parallel_size
+ coordinator._hash_table = {}
+ coordinator._hash_assignment_counter = 0
+ coordinator._round_robin_idx = 0
+
+ sorted_identities = sorted(coordinator.identities_of_data_parallel_ranks)
+ coordinator.identity_to_rank_index = {
+ identity: idx for idx, identity in enumerate(sorted_identities)
+ }
+
+ coordinator._pending_counts = np.zeros(n_ranks, dtype=np.int32)
+ coordinator._identities_list = list(sorted_identities)
+
+ return coordinator
diff --git a/tests/unit_tests/inference/engines/test_dynamic_engine.py b/tests/unit_tests/inference/engines/test_dynamic_engine.py
index d5803b3638e..21e008d1d43 100644
--- a/tests/unit_tests/inference/engines/test_dynamic_engine.py
+++ b/tests/unit_tests/inference/engines/test_dynamic_engine.py
@@ -1,11 +1,15 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
import asyncio
+import gc
import math
+import os
import random
import types
from dataclasses import dataclass, field
+from functools import partial
from typing import Dict, List, Optional, Tuple
+from unittest import mock
import pytest
import torch
@@ -13,7 +17,9 @@
from transformer_engine.pytorch.fp8 import check_fp8_support
from megatron.core import parallel_state
-from megatron.core.inference.contexts.attention_context.mamba_metadata import (
+from megatron.core.inference.config import (
+ InferenceConfig,
+ KVCacheManagementMode,
MambaInferenceStateConfig,
)
from megatron.core.inference.contexts.dynamic_context import (
@@ -24,13 +30,11 @@
TokenOverflowError,
)
from megatron.core.inference.engines import DynamicInferenceEngine
+from megatron.core.inference.engines.dynamic_engine import EngineState
from megatron.core.inference.inference_request import DynamicInferenceRequest, Status
from megatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapper import (
GPTInferenceWrapper,
)
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
from megatron.core.inference.sampling_params import SamplingParams
from megatron.core.inference.text_generation_controllers.text_generation_controller import (
TextGenerationController,
@@ -39,6 +43,7 @@
get_gpt_layer_local_spec,
get_gpt_layer_with_inference_spec,
get_gpt_layer_with_transformer_engine_spec,
+ get_gpt_mtp_block_spec,
)
from megatron.core.models.gpt.gpt_model import GPTModel
from megatron.core.models.mamba.mamba_layer_specs import mamba_stack_spec
@@ -48,13 +53,16 @@
from megatron.core.transformer.cuda_graphs import CudaGraphManager, _CudagraphGlobalRecord
from megatron.core.transformer.enums import CudaGraphScope
from megatron.core.transformer.transformer_config import TransformerConfig
-from megatron.core.utils import (
- get_mamba_inference_state_config_from_model,
- is_fa_min_version,
- is_te_min_version,
-)
+from megatron.core.utils import is_fa_min_version, is_te_min_version
from tests.unit_tests.test_utilities import Utils
+try:
+ from torch_memory_saver import torch_memory_saver # noqa: F401
+
+ HAVE_TORCH_MEMORY_SAVER = True
+except ImportError:
+ HAVE_TORCH_MEMORY_SAVER = False
+
def skip_if_mamba_sequence_packing_not_available(model_provider: str):
if model_provider == "mamba":
@@ -72,6 +80,17 @@ def set_rounder(value):
DynamicInferenceContext.REQUEST_ROUNDER = value
+def mock_forward(input_ids, position_ids, attention_mask, *args, **kwargs):
+ """Mock forward function to avoid numerics issues with random inputs."""
+ return torch.randn(
+ input_ids.size(0),
+ input_ids.size(1),
+ kwargs["vocab_size"],
+ device=input_ids.device,
+ dtype=torch.bfloat16,
+ )
+
+
@dataclass
class DynamicEngineTestConfig:
"""Test configuration args."""
@@ -108,8 +127,9 @@ class DynamicEngineTestConfig:
materialize_only_last_token_logits: bool = True
skip_prompt_log_probs: bool = False
enable_chunked_prefill: bool = False
+ enable_prefix_caching: bool = False
cuda_graph_scope: List[CudaGraphScope] = field(
- default_factory=lambda: [CudaGraphScope.full_iteration]
+ default_factory=lambda: [CudaGraphScope.full_iteration_inference]
)
force_build_cuda_graphs: bool = False
transformer_impl: str = "local"
@@ -120,19 +140,25 @@ class DynamicEngineTestConfig:
# relevant to the test. The tests only check if the required
# context attributes are set correctly.
suspend_resume_interval: Optional[int] = None
-
- fp8: bool = False
+ kv_cache_management_mode: str = "persist"
+ static_kv_memory_pointers: bool = True
+ track_generated_token_events: bool = False
+ num_speculative_tokens: int = 0
def __post_init__(self):
# Compute max_sequence_length.
- assert self.max_sequence_length is None
- assert self.num_tokens_to_generate is None or self.num_tokens_total is None
- if self.num_tokens_to_generate is not None:
- self.max_sequence_length = self.max_prompt_length + self.num_tokens_to_generate
- else:
- assert self.num_tokens_total is not None
- self.max_sequence_length = self.num_tokens_total
+ if self.max_sequence_length is None:
+ assert self.num_tokens_to_generate is None or self.num_tokens_total is None
+ if self.num_tokens_to_generate is not None:
+ self.max_sequence_length = (
+ self.max_prompt_length
+ + self.num_tokens_to_generate
+ + self.num_speculative_tokens
+ )
+ else:
+ assert self.num_tokens_total is not None
+ self.max_sequence_length = self.num_tokens_total + self.num_speculative_tokens
# Default paused buffer size.
if self.context_paused_buffer_size_gb is None:
@@ -164,7 +190,7 @@ def _build_requests(cls, test_config: DynamicEngineTestConfig) -> List[DynamicIn
prompt_length = test_config.min_prompt_length
else:
prompt_length = random.randint(
- test_config.min_prompt_length, test_config.max_prompt_length + 1
+ test_config.min_prompt_length, test_config.max_prompt_length
)
# Num tokens to generate.
@@ -223,26 +249,30 @@ def _build_inference_context(
# Inference context.
context = DynamicInferenceContext(
- params_dtype=transformer_config.params_dtype,
- num_layers=transformer_config.num_layers
- // transformer_config.pipeline_model_parallel_size,
- kv_channels=transformer_config.kv_channels,
- num_attention_heads=transformer_config.num_query_groups,
- max_sequence_length=test_config.max_sequence_length,
- num_cuda_graphs=test_config.num_cuda_graphs,
- use_cuda_graphs_for_non_decode_steps=True,
- buffer_size_gb=test_config.context_buffer_size_gb,
- paused_buffer_size_gb=test_config.context_paused_buffer_size_gb,
- block_size_tokens=test_config.context_block_size_tokens,
- max_requests=test_config.context_max_requests,
- max_tokens=test_config.context_max_tokens,
- tensor_model_parallel_size=transformer_config.tensor_model_parallel_size,
- pipeline_model_parallel_size=transformer_config.pipeline_model_parallel_size,
- mamba_inference_state_config=mamba_inference_state_config,
- materialize_only_last_token_logits=test_config.materialize_only_last_token_logits,
- use_flashinfer_fused_rope=None, # default to using flash-infer if available
- # this is for compatibility with the LTS environment
- unified_memory_level=0, # unit tests currently broken with UVM
+ model_config=transformer_config,
+ inference_config=InferenceConfig(
+ max_sequence_length=test_config.max_sequence_length,
+ num_cuda_graphs=test_config.num_cuda_graphs,
+ use_cuda_graphs_for_non_decode_steps=True,
+ buffer_size_gb=test_config.context_buffer_size_gb,
+ paused_buffer_size_gb=test_config.context_paused_buffer_size_gb,
+ block_size_tokens=test_config.context_block_size_tokens,
+ max_requests=test_config.context_max_requests,
+ max_tokens=test_config.context_max_tokens,
+ mamba_inference_state_config=mamba_inference_state_config,
+ materialize_only_last_token_logits=test_config.materialize_only_last_token_logits,
+ kv_cache_management_mode=KVCacheManagementMode(
+ test_config.kv_cache_management_mode
+ ),
+ static_kv_memory_pointers=test_config.static_kv_memory_pointers,
+ enable_chunked_prefill=test_config.enable_chunked_prefill,
+ enable_prefix_caching=test_config.enable_prefix_caching,
+ use_flashinfer_fused_rope=None, # default to using flash-infer if available
+ # this is for compatibility with the LTS environment
+ unified_memory_level=0, # unit tests currently broken with UVM
+ track_generated_token_events=test_config.track_generated_token_events,
+ num_speculative_tokens=test_config.num_speculative_tokens,
+ ),
)
return context
@@ -275,6 +305,7 @@ def _build_test_env(cls, test_config):
transformer_config = TransformerConfig(
params_dtype=torch.bfloat16,
num_layers=4,
+ mtp_num_layers=test_config.num_speculative_tokens,
hidden_size=128 if test_config.fp8 else 32,
num_attention_heads=4,
use_cpu_initialization=True,
@@ -316,6 +347,14 @@ def _build_test_env(cls, test_config):
elif test_config.transformer_impl == "inference_optimized":
layer_spec = get_gpt_layer_with_inference_spec()
+ # MTP block spec (needed for speculative decoding).
+ mtp_block_spec = None
+ if test_config.num_speculative_tokens > 0:
+ use_te = test_config.fp8 or test_config.transformer_impl == "transformer_engine"
+ mtp_block_spec = get_gpt_mtp_block_spec(
+ config=transformer_config, spec=layer_spec, use_transformer_engine=use_te
+ )
+
# GPT model.
model = GPTModel(
config=transformer_config,
@@ -325,6 +364,7 @@ def _build_test_env(cls, test_config):
parallel_output=True,
pre_process=parallel_state.is_pipeline_first_stage(),
post_process=parallel_state.is_pipeline_last_stage(),
+ mtp_block_spec=mtp_block_spec,
).cuda()
elif test_config.model_provider == "mamba":
pp_size = test_config.pipeline_model_parallel_size
@@ -334,6 +374,7 @@ def _build_test_env(cls, test_config):
num_layers=(
3 if pp_size == 1 else 6
), # 1 Mamba layer, 1 attention layer, 1 MLP layer
+ mtp_num_layers=test_config.num_speculative_tokens,
hidden_size=256, # The Mamba layer places several constraints on this
mamba_num_heads=16,
num_attention_heads=16,
@@ -358,7 +399,9 @@ def _build_test_env(cls, test_config):
add_bias_linear=test_config.expert_model_parallel_size == 1,
fp8="hybrid" if test_config.fp8 else None,
fp8_recipe="tensorwise" if test_config.fp8 else None,
+ inference_sampling_seed=test_config.random_seed,
cuda_graph_scope=test_config.cuda_graph_scope,
+ transformer_impl=test_config.transformer_impl,
is_hybrid_model=True, # Needs to be set for correct out_proj init
)
@@ -369,8 +412,9 @@ def _build_test_env(cls, test_config):
vocab_size=test_config.vocab_size,
max_sequence_length=test_config.max_sequence_length,
parallel_output=True,
- hybrid_attention_ratio=0.3,
- hybrid_mlp_ratio=0.3,
+ hybrid_layer_pattern=(
+ "M*-" if pp_size == 1 else "M*-|M*-"
+ ), # 3 or 6 layers (2 PP stages)
pre_process=parallel_state.is_pipeline_first_stage(),
post_process=parallel_state.is_pipeline_last_stage(),
).cuda()
@@ -382,17 +426,7 @@ def _build_test_env(cls, test_config):
model.eval()
- mamba_inference_state_config = get_mamba_inference_state_config_from_model(model)
-
- # Inference config.
- inference_config = InferenceWrapperConfig(
- hidden_size=transformer_config.hidden_size,
- inference_batch_times_seqlen_threshold=400,
- fp32_residual_connection=False,
- params_dtype=transformer_config.params_dtype,
- fp8=transformer_config.fp8,
- padded_vocab_size=test_config.vocab_size,
- )
+ mamba_inference_state_config = MambaInferenceStateConfig.from_model(model)
# Inference context.
inference_context = cls._build_inference_context(
@@ -403,7 +437,7 @@ def _build_test_env(cls, test_config):
)
# Inference model wrapper.
- inference_wrapped_model = GPTInferenceWrapper(model, inference_config, inference_context)
+ inference_wrapped_model = GPTInferenceWrapper(model, inference_context)
# Note: the following is taken from AbstractModelInferenceWrapper.prep_model_for_inference().
inference_wrapped_model.model_is_pipeline_parallel = not (
@@ -424,13 +458,7 @@ def _build_test_env(cls, test_config):
CudaGraphManager.global_mempool = None
# Inference engine.
- engine = DynamicInferenceEngine(
- text_generation_controller,
- inference_context,
- random_seed=test_config.random_seed,
- enable_cuda_graph=transformer_config.cuda_graph_impl == "local",
- enable_chunked_prefill=test_config.enable_chunked_prefill,
- )
+ engine = DynamicInferenceEngine(text_generation_controller, inference_context)
# Test env.
env = DynamicEngineTestEnv(config=test_config, requests=requests, engine=engine)
@@ -451,7 +479,7 @@ def _run_step(cls, env):
# Suspend + resume.
if (
env.config.suspend_resume_interval is not None
- and env.engine.step_count % env.config.suspend_resume_interval == 0
+ and env.engine.context.step_count % env.config.suspend_resume_interval == 0
):
suspend_resume_mems = {}
suspend_resume_mems["start"] = torch.cuda.memory_stats()
@@ -459,7 +487,7 @@ def _run_step(cls, env):
suspend_resume_mems["mid"] = torch.cuda.memory_stats()
env.engine.resume() # resume.
suspend_resume_mems["end"] = torch.cuda.memory_stats()
- env.mem_usage["suspend_resume"][env.engine.step_count] = suspend_resume_mems
+ env.mem_usage["suspend_resume"][env.engine.context.step_count] = suspend_resume_mems
# Nothing done?
finished_request_records = result["finished_request_records"]
@@ -530,7 +558,8 @@ def _run_test(cls, **test_config_kwargs):
return env
- def teardown_method(self, method):
+ @classmethod
+ def teardown_class(cls):
set_rounder(64)
Utils.destroy_model_parallel()
@@ -539,8 +568,8 @@ def teardown_method(self, method):
not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
)
@pytest.mark.parametrize("model_provider", ["gpt", "mamba"])
- @pytest.mark.parametrize("num_cuda_graphs", [None, 1, 4])
- @pytest.mark.parametrize("cuda_graph_scope", [[], [CudaGraphScope.full_iteration]])
+ @pytest.mark.parametrize("num_cuda_graphs", [None, 1, 4, -1])
+ @pytest.mark.parametrize("cuda_graph_scope", [[], [CudaGraphScope.full_iteration_inference]])
def test_simple(self, model_provider, num_cuda_graphs, cuda_graph_scope) -> None:
"""Simple test that runs without errors, and validates output."""
skip_if_mamba_sequence_packing_not_available(model_provider)
@@ -553,32 +582,45 @@ def test_simple(self, model_provider, num_cuda_graphs, cuda_graph_scope) -> None
num_cuda_graphs=num_cuda_graphs,
cuda_graph_scope=cuda_graph_scope,
force_build_cuda_graphs=True,
+ context_max_requests=128,
)
# Validate max_requests, max_tokens.
assert env.engine.context.max_tokens == DynamicInferenceContext.DEFAULT_MAX_TOKENS
+ if num_cuda_graphs is not None:
+ assert env.engine.context.cuda_graph_token_counts is not None
+ assert env.engine.context.cuda_graph_batch_dimensions_list
+ model = env.engine.controller.inference_wrapped_model.model
+ if cuda_graph_scope == [CudaGraphScope.full_iteration_inference]:
+ # check if cudagraph runners are created at the decoder level
+ assert model.decoder.cudagraph_manager.cudagraph_runners
+ else:
+ # check if cudagraph runners are created at the layer level
+ for layer in model.decoder.layers:
+ assert layer.cudagraph_manager.cudagraph_runners
+
# Validate generated tokens.
gpt_expected_generated_tokens = [
[69, 85, 55, 74, 56, 89, 64, 59, 55, 67, 15, 58, 6, 37, 54, 47],
[29, 54, 33, 72, 45, 76, 41, 56, 28, 25, 17, 2, 61, 6, 98, 76],
[35, 78, 54, 16, 79, 98, 22, 5, 60, 0, 1, 76, 77, 11, 25, 7],
[25, 75, 57, 85, 81, 37, 88, 17, 71, 15, 70, 64, 50, 0, 64, 45],
- [32, 5, 85, 75, 30, 68, 23, 33, 20, 26, 89, 20, 92, 97, 38, 81],
- [33, 69, 32, 49, 93, 24, 33, 6, 97, 36, 37, 99],
- [82, 78, 78, 65, 22, 1, 87, 42, 36, 26, 27, 56, 82, 32, 8, 80],
- [],
+ [32, 5, 85, 75, 30, 68, 23, 33, 20, 26, 89, 20, 49, 28, 38, 81],
+ [33, 69, 32, 49, 93, 24, 33, 6, 54, 89, 92, 97, 42, 80, 50, 53],
+ [82, 78, 78, 65, 26, 5, 69, 36, 37, 99],
+ [51, 70, 22, 1, 87, 42, 36, 26, 27, 56, 82, 32, 8, 80, 20, 43],
]
mamba_expected_generated_tokens = [
- [74, 72, 9, 59, 1, 70, 15, 89, 30, 52, 82, 70, 64, 16, 83, 5],
- [25, 54, 28, 14, 87, 27, 60, 92, 28, 74, 8, 63, 60, 68, 87, 82],
- [31, 21, 87, 25, 96, 13, 32, 49, 40, 54, 55, 68, 73, 2, 64, 96],
- [72, 80, 35, 72, 77, 85, 98, 36, 4, 97, 37, 46, 79, 95, 83, 25],
- [8, 80, 56, 4, 87, 1, 43, 98, 85, 7, 50, 38, 24, 28, 18, 80],
- [9, 94, 36, 16, 87, 57, 25, 76, 64, 92, 47, 86, 73, 72, 71, 97],
- [17, 5, 62, 66, 15, 52, 32, 75, 66, 18, 90, 14, 67, 37, 94, 33],
- [],
+ [69, 85, 55, 74, 85, 89, 64, 59, 55, 67, 15, 58, 6, 37, 34, 47],
+ [29, 16, 33, 30, 45, 76, 41, 46, 82, 17, 17, 2, 61, 6, 98, 76],
+ [35, 78, 54, 16, 79, 98, 22, 5, 37, 30, 1, 76, 5, 11, 25, 86],
+ [25, 75, 57, 85, 81, 59, 88, 38, 71, 15, 70, 64, 50, 0, 64, 45],
+ [32, 5, 85, 75, 30, 68, 23, 33, 20, 26, 35, 20, 49, 28, 34, 81],
+ [87, 69, 32, 49, 93, 24, 33, 6, 54, 89, 92, 97, 42, 80, 50, 53],
+ [82, 78, 78, 19, 70, 5, 97, 36, 37, 99],
+ [51, 70, 22, 1, 87, 42, 36, 26, 27, 56, 82, 32, 8, 20, 20, 43],
]
if model_provider == "gpt":
@@ -735,7 +777,7 @@ def test_generate_function(self, model_provider: str) -> None:
prompts = ["prompt1", "prompt2", "prompt3", "prompt4"]
# Mock the tokenize_prompt method to return predictable token sequences
- def mock_tokenize_prompt(prompt, add_BOS=False):
+ def mock_tokenize_prompt(tokenizer, prompt, add_BOS=False):
# Return a token sequence based on the prompt number
prompt_num = int(prompt[-1])
return [10 + i for i in range(prompt_num + 2)]
@@ -1121,78 +1163,522 @@ def test_events(self):
)
expected_event_types = [
- ['ADD', 'FINISH'],
- ['ADD', 'FINISH'],
- ['ADD', 'FINISH'],
- ['ADD', 'FINISH'],
- ['ERROR_TRANSIENT', 'ADD', 'FINISH'],
- ['ERROR_TRANSIENT', 'ADD', 'FINISH'],
- ['ADD', 'FINISH'],
- ['ERROR_NONTRANSIENT', 'FAIL'],
- ['ERROR_NONTRANSIENT', 'FAIL'],
- ['ERROR_TRANSIENT', 'ADD', 'FINISH'],
- ['ERROR_NONTRANSIENT', 'FAIL'],
- ['ERROR_TRANSIENT', 'ADD', 'FINISH'],
- ['ADD', 'FINISH'],
- ['ERROR_TRANSIENT', 'ADD', 'FINISH'],
- ['ERROR_TRANSIENT', 'ADD', 'FINISH'],
- ['ERROR_TRANSIENT', 'ADD', 'FINISH'],
+ ['ADD_ENGINE', 'ADD_CONTEXT', 'FINISH'],
+ ['ADD_ENGINE', 'ADD_CONTEXT', 'FINISH'],
+ ['ADD_ENGINE', 'ADD_CONTEXT', 'FINISH'],
+ ['ADD_ENGINE', 'ERROR_NONTRANSIENT', 'FAIL'],
+ ['ADD_ENGINE', 'ADD_CONTEXT', 'FINISH'],
+ ['ADD_ENGINE', 'ADD_CONTEXT', 'FINISH'],
+ ['ADD_ENGINE', 'ADD_CONTEXT', 'FINISH'],
+ ['ADD_ENGINE', 'ERROR_NONTRANSIENT', 'FAIL'],
+ ['ADD_ENGINE', 'ERROR_NONTRANSIENT', 'FAIL'],
+ ['ADD_ENGINE', 'ERROR_NONTRANSIENT', 'FAIL'],
+ ['ADD_ENGINE', 'ADD_CONTEXT', 'FINISH'],
+ ['ADD_ENGINE', 'ADD_CONTEXT', 'FINISH'],
+ ['ADD_ENGINE', 'ADD_CONTEXT', 'FINISH'],
+ ['ADD_ENGINE', 'ERROR_NONTRANSIENT', 'FAIL'],
+ ['ADD_ENGINE', 'ERROR_NONTRANSIENT', 'FAIL'],
+ ['ADD_ENGINE', 'ADD_CONTEXT', 'FINISH'],
+ ]
+ result_event_types = [
+ [e.type.name for e in r.events if e.type.name != 'GENERATED_TOKEN']
+ for r in env.requests
]
- result_event_types = [[e.type.name for e in r.events] for r in env.requests]
-
assert result_event_types == expected_event_types
@pytest.mark.internal
@pytest.mark.skipif(
not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
)
- @pytest.mark.parametrize("model_provider", ["gpt", "mamba"])
@torch.inference_mode()
- def test_chunked_prefill(self, model_provider: str):
- """Verify that chunked prefill output is equivalent to regular prefill."""
- skip_if_mamba_sequence_packing_not_available(model_provider)
-
- prompt_length = 1200
- num_tokens_to_generate = 16
- max_sequence_length = prompt_length + num_tokens_to_generate
+ def test_event_timestamps(self):
+ """Test that events are recorded with sensical timestamps.
- # Configure context to force chunking
+ Verifies:
+ 1. Completed requests have ADD_ENGINE, ADD_CONTEXT, GENERATED_TOKEN(s), FINISH events
+ 2. Event timestamps are monotonically increasing
+ 3. TTFT (time-to-first-token) can be computed as first GENERATED_TOKEN - ADD_ENGINE
+ """
+ num_tokens_to_generate = 8
env = self._run_test(
- num_requests=1,
- min_prompt_length=prompt_length,
- max_prompt_length=prompt_length,
+ num_requests=4,
+ max_prompt_length=16,
num_tokens_to_generate=num_tokens_to_generate,
- materialize_only_last_token_logits=False,
- model_provider=model_provider,
+ context_buffer_size_gb=0.1,
+ num_gap_steps=0,
+ track_generated_token_events=True,
+ )
+
+ # All requests should complete with this generous config (large buffer, no gap steps).
+ assert all(r.status == Status.COMPLETED for r in env.requests)
+ for request in env.requests:
+
+ # Verify event types for completed requests
+ event_types = [e.type.name for e in request.events]
+ # Should be: ADD_ENGINE, ADD_CONTEXT, GENERATED_TOKEN (repeated), FINISH
+ assert (
+ event_types[0] == 'ADD_ENGINE'
+ ), f"Request {request.request_id}: first event should be ADD_ENGINE, got {event_types[0]}"
+ assert (
+ event_types[1] == 'ADD_CONTEXT'
+ ), f"Request {request.request_id}: second event should be ADD_CONTEXT, got {event_types[1]}"
+ assert (
+ event_types[-1] == 'FINISH'
+ ), f"Request {request.request_id}: last event should be FINISH, got {event_types[-1]}"
+ # Check that GENERATED_TOKEN events are in the middle
+ gen_token_count = event_types.count('GENERATED_TOKEN')
+ assert gen_token_count == len(request.generated_tokens), (
+ f"Request {request.request_id}: GENERATED_TOKEN count ({gen_token_count}) != "
+ f"generated_tokens length ({len(request.generated_tokens)})"
+ )
+
+ # Verify timestamps are monotonically increasing
+ timestamps = [e.timestamp for e in request.events]
+ for i in range(1, len(timestamps)):
+ assert timestamps[i] >= timestamps[i - 1], (
+ f"Request {request.request_id}: timestamp[{i}] ({timestamps[i]}) < "
+ f"timestamp[{i-1}] ({timestamps[i-1]})"
+ )
+
+ # Verify TTFT is positive and sensical (first GENERATED_TOKEN - ADD_ENGINE)
+ add_engine_ts = request.events[0].timestamp
+ first_token_ts = request.events[2].timestamp # First GENERATED_TOKEN event
+ assert (
+ request.events[2].type.name == 'GENERATED_TOKEN'
+ ), f"Request {request.request_id}: event[2] should be GENERATED_TOKEN"
+ ttft = first_token_ts - add_engine_ts
+ assert ttft >= 0, f"Request {request.request_id}: TTFT is negative ({ttft})"
+
+ # Verify total request time is positive
+ finish_ts = request.events[-1].timestamp
+ total_time = finish_ts - add_engine_ts
+ assert (
+ total_time >= ttft
+ ), f"Request {request.request_id}: total_time ({total_time}) < TTFT ({ttft})"
+
+ @pytest.mark.internal
+ @pytest.mark.skipif(
+ not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
+ )
+ @torch.inference_mode()
+ def test_mamba_chunked_prefill(self):
+ """
+ Test chunked prefill with a Mamba model.
+ """
+ skip_if_mamba_sequence_packing_not_available("mamba")
+
+ # Context max tokens = 50.
+ test_config = DynamicEngineTestConfig(
+ model_provider="mamba",
+ num_requests=0,
+ num_tokens_to_generate=None,
+ num_tokens_total=200,
+ context_max_tokens=52,
+ context_max_requests=5,
+ context_block_size_tokens=256,
+ enable_chunked_prefill=True,
+ use_cuda_graphs_for_non_decode_steps=False,
+ )
+
+ env = self._build_test_env(test_config)
+ ctx = env.engine.context
+
+ # Mock the model forward function to avoid possible numerics issues
+ # caused by random inputs
+ model_instance = env.engine.controller.inference_wrapped_model.model
+ model_instance.forward = partial(mock_forward, vocab_size=test_config.vocab_size)
+
+ # Request 1: 150 tokens
+ req1_tokens = torch.randint(0, test_config.vocab_size, (130,), device='cuda')
+ req1 = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=req1_tokens,
+ sampling_params=SamplingParams(num_tokens_to_generate=3),
+ )
+
+ # Request 2: 160 tokens
+ req2_tokens = torch.randint(0, test_config.vocab_size, (160,), device='cuda')
+ req2 = DynamicInferenceRequest(
+ request_id=2,
+ prompt_tokens=req2_tokens,
+ sampling_params=SamplingParams(num_tokens_to_generate=1),
+ )
+
+ # Request 3: 24 tokens
+ req3_tokens = torch.randint(0, test_config.vocab_size, (24,), device='cuda')
+ req3 = DynamicInferenceRequest(
+ request_id=3,
+ prompt_tokens=req3_tokens,
+ sampling_params=SamplingParams(num_tokens_to_generate=1),
+ )
+
+ # Add requests 1-3
+ env.engine._add_request(req1)
+ env.engine._add_request(req2)
+ env.engine._add_request(req3)
+
+ # Run step 1
+ env.engine.schedule_waiting_requests()
+
+ env.engine.step_modern()
+ assert req1.finished_chunk_token_count == 52
+
+ # Prepare for step 2
+ env.engine.schedule_waiting_requests()
+
+ # Verify that requests 2 and 3 are queued because request 1 is still running
+ assert ctx.num_prefill_requests == 1
+ active_ids = ctx.request_ids[: ctx.total_request_count].tolist()
+ assert 1 in active_ids
+ assert 2 not in active_ids
+ assert 3 not in active_ids
+ assert list(env.engine.waiting_request_ids) == [1, 2, 3]
+
+ # Verify that active token count == max tokens
+ assert ctx.active_token_count == 52
+
+ # Verify that request 1 is the designated chunked prefill request
+ assert ctx.chunked_prefill_request_id == 1
+
+ # Run step 2
+ env.engine.step_modern()
+ assert req1.finished_chunk_token_count == 104
+
+ # Prepare for step 3
+ env.engine.schedule_waiting_requests()
+
+ # Verify that request 2 got partially scheduled and is now
+ # the designated chunked prefill request
+ req2_idx = ctx.request_ids.tolist().index(2)
+ assert req2_idx == 1
+ assert ctx.num_prefill_requests == 2
+ assert ctx.chunked_prefill_request_id == 2
+ assert ctx.get_index_of_chunked_prefill_request() == req2_idx
+ active_ids = ctx.request_ids[: ctx.total_request_count].tolist()
+ assert 1 in active_ids
+ assert 2 in active_ids
+ assert 3 not in active_ids
+
+ # Store the Mamba state tensor idx for request 2
+ req2_mamba_idx = ctx.mamba_metadata.request_to_mamba_state_idx[req2_idx].item()
+
+ # Verify that the active token count is the maximum token count
+ assert ctx.active_token_count == 52
+
+ # Run step 3
+ env.engine.step_modern()
+ assert req1.finished_chunk_token_count == 104
+
+ # Prepare for step 4
+ env.engine.schedule_waiting_requests()
+
+ # Verify that request 2 is still the first prefill request
+ assert ctx.request_ids.tolist().index(2) == 1
+ assert ctx.mamba_metadata.request_to_mamba_state_idx[1] == req2_mamba_idx
+
+ # Verify that request 1 is running decode
+ active_ids = ctx.request_ids[: ctx.total_request_count].tolist()
+ assert ctx.num_decode_requests == 1
+ assert 1 in active_ids
+
+ # Verify that request 2 is still running prefill as the designated chunked prefill request
+ assert ctx.num_prefill_requests == 1
+ assert ctx.chunked_prefill_request_id == 2
+ assert ctx.get_index_of_chunked_prefill_request() == 1
+
+ # Verify that request 3 is still waiting
+ assert 3 not in active_ids
+ assert 3 in env.engine.waiting_request_ids
+
+ # Verify that active token count == max tokens
+ assert ctx.active_token_count == 52
+
+ # Run step 4
+ env.engine.step_modern()
+
+ assert req2.finished_chunk_token_count == 77
+
+ # Prepare for step 5
+ env.engine.schedule_waiting_requests()
+
+ # Verify that request 2 is still the first prefill request
+ assert ctx.request_ids.tolist().index(2) == 1
+ assert ctx.mamba_metadata.request_to_mamba_state_idx[1] == req2_mamba_idx
+
+ # Run step 5
+ env.engine.step_modern()
+ assert req2.finished_chunk_token_count == 128
+
+ # Prepare for step 6
+ env.engine.schedule_waiting_requests()
+
+ # Verify that request 1 has completed
+ assert req1.status == Status.COMPLETED
+
+ # Verify that request 2 is still the first prefill request
+ assert ctx.request_ids.tolist().index(2) == 0
+ assert ctx.mamba_metadata.request_to_mamba_state_idx[0] == req2_mamba_idx
+
+ # Verify that request 3 is now scheduled as the chunked prefill request
+ active_ids = ctx.request_ids[: ctx.total_request_count].tolist()
+ assert 2 in active_ids
+ assert 3 in active_ids
+ assert ctx.chunked_prefill_request_id == 3
+ req3_idx = active_ids.index(3)
+ assert req3_idx == 1
+
+ # Store the Mamba state tensor idx for request 3
+ req3_mamba_idx = ctx.mamba_metadata.request_to_mamba_state_idx[req3_idx].item()
+
+ # Run step 6
+ env.engine.step_modern()
+
+ # Verify that request 2 has finished
+ assert req2.status == Status.COMPLETED
+ assert req3.finished_chunk_token_count == 20
+
+ # Prepare for step 7
+ env.engine.schedule_waiting_requests()
+
+ # Verify that request 3 is now the first prefill request
+ req3_idx = ctx.request_ids.tolist().index(3)
+ assert req3_idx == 0
+ assert ctx.mamba_metadata.request_to_mamba_state_idx[0] == req3_mamba_idx
+
+ # Run step 7
+ env.engine.step_modern()
+
+ # Verify that request 3 has finished
+ assert req3.status == Status.COMPLETED
+
+ @pytest.mark.internal
+ @pytest.mark.skipif(
+ not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
+ )
+ @torch.inference_mode()
+ def test_chunked_prefill_avoid_single_token_chunk(self):
+ """
+ Test that chunked prefill scheduling avoids leaving exactly 1 token for the final chunk.
+ This leads to a known bug in the Flash Attention kernel:
+ https://github.com/Dao-AILab/flash-attention/issues/1537
+
+ Scenario:
+ - Max tokens per step (Chunk Size): 256
+ - Request prompt length: 513
+
+ Default scheduling would do:
+ 1. Chunk 256 (Remaining 257)
+ 2. Chunk 256 (Remaining 1) -> max_seqlen_q=1 triggers decode path in kernel
+ 3. Chunk 1
+
+ Fixed scheduling should do:
+ 1. Chunk 256 (Remaining 257) -> 513 - 256 == 257. Schedule full 256.
+ 2. Chunk 255 (Remaining 2) -> 257 tokens left. If we take 256, 1 remains.
+ So we reduce chunk to 255.
+ 3. Chunk 2 (Remaining 0)
+ """
+ prefill_chunk_size = 256
+ # Prompt length designed to trigger the edge case: Chunk + (Chunk + 1)
+ # 256 + 255 + 2 = 513
+ prompt_len = 513
+
+ test_config = DynamicEngineTestConfig(
+ model_provider="gpt",
+ num_requests=0,
+ num_tokens_to_generate=None,
+ num_tokens_total=prompt_len + 1,
+ context_max_tokens=prefill_chunk_size,
+ context_max_requests=1,
+ context_block_size_tokens=256,
+ enable_chunked_prefill=True,
+ use_cuda_graphs_for_non_decode_steps=False,
+ )
+
+ env = self._build_test_env(test_config)
+ ctx = env.engine.context
+
+ # Mock the model forward function to avoid possible numerics issues
+ # caused by random inputs
+ model_instance = env.engine.controller.inference_wrapped_model.model
+ model_instance.forward = partial(mock_forward, vocab_size=test_config.vocab_size)
+
+ # Create a request with length 513
+ req_tokens = torch.randint(0, test_config.vocab_size, (prompt_len,), device='cuda')
+ req = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=req_tokens,
+ sampling_params=SamplingParams(num_tokens_to_generate=1),
+ )
+
+ env.engine._add_request(req)
+
+ assert req.status == Status.ACTIVE_AND_GENERATING_TOKENS
+
+ # --- Step 1 ---
+ # Available: 256. Remaining: 513.
+ # Logic: 513 - 256 = 257. Not 1. Schedule full 256.
+ env.engine.step_modern()
+
+ assert env.engine.context.total_request_count == 0, env.engine.context.total_request_count
+
+ assert (
+ req.finished_chunk_token_count == 256
+ ), f"Step 1: Expected 256 tokens processed, got {req.finished_chunk_token_count}"
+
+ # --- Step 2 ---
+ # Available: 256. Remaining un-prefilled: 257.
+ # Logic: 257 - 256 = 1. This is the edge case!
+ # Fix should reduce chunk size by 1 (to 255).
+ env.engine.step_modern()
+
+ assert env.engine.context.total_request_count == 0, env.engine.context.total_request_count
+
+ # 256 (previous) + 255 (this step) = 511
+ assert req.finished_chunk_token_count == 511, (
+ "Step 2: Expected 511 tokens processed (256+255), "
+ f"got {req.finished_chunk_token_count}. "
+ )
+
+ # --- Step 3 ---
+ # Remaining un-prefilled: 2. Available: 256.
+ # Logic: 2 <= 256. Schedule 2.
+ env.engine.schedule_waiting_requests()
+ env.engine.step_modern()
+
+ # Verify request finishes prefill and completes
+ assert ctx.num_prefill_requests == 0
+ assert req.status == Status.COMPLETED
+
+ @pytest.mark.internal
+ @pytest.mark.skipif(
+ not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
+ )
+ @torch.inference_mode()
+ def test_chunked_prefill_delay_scheduling_for_unavoidable_single_token_chunk(self):
+ """
+ Test that chunked prefill scheduling delays execution when the only available
+ option is to schedule a chunk of size 1 that leaves exactly 1 token remaining.
+
+ Scenario:
+ - Max tokens per step: 256
+ - Request A: 254 token prompt
+ - Request B: 2 token prompt
+
+ Sequence:
+ 1. Step 1 scheduling:
+ - Request A is scheduled (255 tokens).
+ - Context has 1 token available (256 - 255).
+ - Request B has 2 tokens remaining.
+ - If we schedule 1 token for B, it leaves exactly 1 token for its final chunk,
+ crashing FA3. Since chunk_length is 1, we can't safely reduce it.
+ The engine MUST delay scheduling Request B.
+ 2. Step 1 executes prefill for Request A only.
+ 3. Step 2 scheduling:
+ - Request A enters decode phase (takes 1 active token).
+ - Context has 255 tokens available (256 - 1).
+ - Request B is now safely scheduled for its full 2 tokens.
+ """
+ test_config = DynamicEngineTestConfig(
+ model_provider="gpt",
+ num_requests=0,
+ num_tokens_to_generate=None,
+ num_tokens_total=256,
+ context_max_tokens=256,
+ context_max_requests=2,
context_block_size_tokens=256,
- context_max_tokens=1000,
enable_chunked_prefill=True,
+ use_cuda_graphs_for_non_decode_steps=False,
+ )
+
+ env = self._build_test_env(test_config)
+ ctx = env.engine.context
+
+ # Mock the model forward function to avoid possible numerics issues
+ model_instance = env.engine.controller.inference_wrapped_model.model
+ model_instance.forward = partial(mock_forward, vocab_size=test_config.vocab_size)
+
+ # Add Request A (Length 255)
+ req_a_tokens = torch.randint(0, test_config.vocab_size, (255,), device='cuda')
+ req_a = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=req_a_tokens,
+ sampling_params=SamplingParams(num_tokens_to_generate=1),
+ )
+ env.engine._add_request(req_a)
+
+ # Add Request B (Length 2)
+ req_b_tokens = torch.randint(0, test_config.vocab_size, (2,), device='cuda')
+ req_b = DynamicInferenceRequest(
+ request_id=2,
+ prompt_tokens=req_b_tokens,
+ sampling_params=SamplingParams(num_tokens_to_generate=1),
)
+ env.engine._add_request(req_b)
+
+ # --- Step 1 ---
+ # Should schedule Request A fully (255), but delay Request B
+ env.engine.step_modern()
+
+ assert req_a.status == Status.COMPLETED
+
+ # Request B MUST be delayed (0 tokens processed) to avoid the FA3 bug
+ assert (
+ req_b.finished_chunk_token_count == 0
+ ), "Request B should have been delayed to avoid leaving a 1-token chunk"
+ assert len(env.engine.waiting_request_ids) == 1
+ assert env.engine.waiting_request_ids[0] == 2
+
+ # --- Step 2 ---
+ # Request A has completed. Context has 256 tokens available.
+ # Request B can now schedule its full 2 tokens safely.
+ env.engine.step_modern()
+
+ assert req_b.status == Status.COMPLETED
+ assert len(env.engine.waiting_request_ids) == 0
@pytest.mark.internal
@pytest.mark.skipif(
not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
)
+ @pytest.mark.parametrize("materialize_only_last_token_logits", [True, False])
+ @pytest.mark.parametrize("skip_prompt_log_probs", [True, False])
@torch.inference_mode()
- def test_chunked_prefill_with_log_probs(self):
+ def test_chunked_prefill_with_log_probs(
+ self, materialize_only_last_token_logits: bool, skip_prompt_log_probs: bool
+ ):
"""
- Test that chunked prefill correctly handles log probs with materialize_only_last_token_logits.
- This verifies that intermediate log probs are properly discarded during chunked prefill.
+ Test that chunked prefill correctly handles log probs across all branches
+ of the log-prob accumulation logic.
When materialize_only_last_token_logits=True, skip_prompt_log_probs must be True.
"""
+ if materialize_only_last_token_logits and not skip_prompt_log_probs:
+ with pytest.raises(AssertionError, match="only last token logits are materialized"):
+ self._run_test(
+ num_requests=1,
+ min_prompt_length=1200,
+ max_prompt_length=1200,
+ num_tokens_to_generate=8,
+ materialize_only_last_token_logits=True,
+ return_log_probs=True,
+ skip_prompt_log_probs=False,
+ model_provider="gpt",
+ context_block_size_tokens=256,
+ context_max_tokens=1000,
+ enable_chunked_prefill=True,
+ )
+ return
+
prompt_length = 1200
num_tokens_to_generate = 8
- # Run with chunked prefill, materialize_only_last_token_logits=True, and skip_prompt_log_probs=True
- # This is the only valid combination for chunked prefill with last-token-only logits
env = self._run_test(
num_requests=1,
min_prompt_length=prompt_length,
max_prompt_length=prompt_length,
num_tokens_to_generate=num_tokens_to_generate,
- materialize_only_last_token_logits=True,
+ materialize_only_last_token_logits=materialize_only_last_token_logits,
return_log_probs=True,
- skip_prompt_log_probs=True,
+ skip_prompt_log_probs=skip_prompt_log_probs,
model_provider="gpt",
context_block_size_tokens=256,
context_max_tokens=1000,
@@ -1213,11 +1699,17 @@ def test_chunked_prefill_with_log_probs(self):
f"generated log probs, got {len(request.generated_log_probs)}"
)
- # When skip_prompt_log_probs is True, prompt_log_probs should be empty
- assert request.prompt_log_probs is None or len(request.prompt_log_probs) == 0, (
- f"Request {request.request_id}: prompt_log_probs should be empty when "
- f"skip_prompt_log_probs=True, but got {len(request.prompt_log_probs) if request.prompt_log_probs else 0} items"
- )
+ if skip_prompt_log_probs:
+ assert request.prompt_log_probs is None or len(request.prompt_log_probs) == 0, (
+ f"Request {request.request_id}: prompt_log_probs should be empty when "
+ f"skip_prompt_log_probs=True, but got "
+ f"{len(request.prompt_log_probs) if request.prompt_log_probs else 0} items"
+ )
+ else:
+ assert len(request.prompt_log_probs) == prompt_length - 1, (
+ f"Request {request.request_id}: Expected {prompt_length - 1} "
+ f"prompt log probs, got {len(request.prompt_log_probs)}"
+ )
# Validate each generated log prob
for i, log_prob in enumerate(request.generated_log_probs):
@@ -1230,6 +1722,98 @@ def test_chunked_prefill_with_log_probs(self):
f"log_prob {log_prob} is out of expected range [-50.0, 0.0]"
)
+ @pytest.mark.internal
+ @pytest.mark.skipif(
+ not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
+ )
+ @torch.inference_mode()
+ def test_chunked_prefill_log_probs_match_baseline(self):
+ """
+ Verify that chunked prefill computes the exact same prompt log probabilities
+ as non-chunked prefill. This explicitly catches the bug where garbage
+ sampled tokens corrupt the prompt log probabilities at chunk boundaries.
+ """
+ prompt_length = 512
+ num_tokens_to_generate = 4
+
+ # Create a deterministic mock forward pass that returns logits
+ # dependent ONLY on position_ids. This guarantees the same logits
+ # whether processed in one giant chunk or split across multiple chunks.
+ def deterministic_mock_forward(input_ids, position_ids, attention_mask, *args, **kwargs):
+ vocab_size = kwargs["vocab_size"]
+ # Use torch.linspace to generate varying but 100% deterministic logits per position
+ static_logits = torch.linspace(
+ -50, 50, 4096 * vocab_size, device=input_ids.device, dtype=torch.bfloat16
+ ).view(4096, vocab_size)
+
+ return static_logits[position_ids]
+
+ def get_log_probs(chunked: bool, max_tokens: int):
+ test_config = DynamicEngineTestConfig(
+ num_requests=0, # Added manually below
+ num_tokens_to_generate=num_tokens_to_generate,
+ materialize_only_last_token_logits=False,
+ return_log_probs=True,
+ skip_prompt_log_probs=False,
+ model_provider="gpt",
+ context_block_size_tokens=256,
+ context_max_requests=1,
+ context_max_tokens=max_tokens,
+ max_sequence_length=1024,
+ enable_chunked_prefill=chunked,
+ use_cuda_graphs_for_non_decode_steps=False,
+ )
+ env = self._build_test_env(test_config)
+
+ # Patch the mock forward to be deterministic
+ model_instance = env.engine.controller.inference_wrapped_model.model
+ model_instance.forward = partial(
+ deterministic_mock_forward, vocab_size=test_config.vocab_size
+ )
+
+ # Ensure identical prompt tokens for both runs
+ torch.manual_seed(42)
+ req_tokens = torch.randint(0, test_config.vocab_size, (prompt_length,), device='cuda')
+ req = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=req_tokens,
+ sampling_params=SamplingParams(
+ num_tokens_to_generate=num_tokens_to_generate,
+ return_log_probs=True,
+ skip_prompt_log_probs=False,
+ termination_id=-1,
+ ),
+ )
+
+ env.engine._add_request(req)
+
+ # Drive the engine until the request finishes
+ while env.engine.has_unfinished_requests():
+ env.engine.schedule_waiting_requests()
+ env.engine.step_modern()
+
+ return req.prompt_log_probs
+
+ # Run non-chunked baseline (all 512 tokens in one pass)
+ baseline_log_probs = get_log_probs(chunked=False, max_tokens=1000)
+
+ # Run chunked (512 tokens split across 256-token boundaries)
+ chunked_log_probs = get_log_probs(chunked=True, max_tokens=256)
+
+ assert baseline_log_probs is not None, "Baseline prompt_log_probs is missing"
+ assert chunked_log_probs is not None, "Chunked prompt_log_probs is missing"
+
+ assert len(baseline_log_probs) == prompt_length - 1
+ assert len(chunked_log_probs) == prompt_length - 1
+
+ # Compare element-wise using math.isclose to handle minor floating point rounding
+ for i, (base_lp, chunk_lp) in enumerate(zip(baseline_log_probs, chunked_log_probs)):
+ assert math.isclose(base_lp, chunk_lp, rel_tol=1e-3, abs_tol=1e-3), (
+ f"Log prob mismatch at prompt token index {i}: "
+ f"Baseline={base_lp:.4f}, Chunked={chunk_lp:.4f}. "
+ "This indicates log prob corruption at chunk boundaries!"
+ )
+
@pytest.mark.internal
@pytest.mark.skipif(
not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
@@ -1374,16 +1958,1189 @@ def test_max_requests(self, max_requests: int | None):
env = self._run_test(
context_max_requests=max_requests, num_tokens_to_generate=16, num_gap_steps=1
)
- step_count = env.engine.step_count
+ step_count = env.engine.context.step_count
context = env.engine.context
if max_requests is None:
assert context.max_requests == 816
- assert step_count == 22
+ assert step_count == 23
else:
assert max_requests < len(env.requests), (
f"Test is only useful if max_requests ({max_requests}) < "
f"num_requests ({len(env.requests)})."
)
assert context.max_requests == 4
- assert step_count == 34
- assert context.block_allocator.active_count == 655
+ assert step_count == 35
+ assert context.kv_block_allocator.active_count == 655
+
+ @pytest.mark.internal
+ @pytest.mark.skipif(
+ not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
+ )
+ @pytest.mark.parametrize("static_kv_memory_pointers", [True, False])
+ @pytest.mark.parametrize("kv_cache_management_mode", ["persist", "offload", "recompute"])
+ @torch.inference_mode()
+ def test_suspend_resume_cycle(self, kv_cache_management_mode, static_kv_memory_pointers):
+ """Full suspend -> resume cycle with memory, data, and address checks."""
+ needs_tms = static_kv_memory_pointers and kv_cache_management_mode != "persist"
+
+ test_config = DynamicEngineTestConfig(
+ kv_cache_management_mode=kv_cache_management_mode,
+ static_kv_memory_pointers=static_kv_memory_pointers,
+ )
+
+ # Without TMS, these combos must assert on construction.
+ if needs_tms and not HAVE_TORCH_MEMORY_SAVER:
+ with pytest.raises(AssertionError, match="Static KV memory pointers"):
+ self._build_test_env(test_config)
+ return
+
+ env = self._build_test_env(test_config)
+ engine = env.engine
+ context = engine.context
+
+ assert engine.state != EngineState.SUSPENDED
+ assert context.is_tensor_state_allocated
+
+ deallocates = kv_cache_management_mode != "persist"
+ uses_tms = context._uses_torch_memory_saver
+ preserves_data = kv_cache_management_mode != "recompute"
+
+ # Write a deterministic pattern for data integrity check.
+ if preserves_data:
+ context.memory_buffer.copy_(torch.randn_like(context.memory_buffer))
+ expected = context.memory_buffer.clone()
+
+ addr_before = context.memory_buffer.data_ptr()
+
+ gc.collect()
+ torch.cuda.empty_cache()
+ torch.cuda.synchronize()
+ mem_before = torch.cuda.memory_allocated()
+ if uses_tms:
+ phys_mem_before = torch.cuda.mem_get_info()[0]
+
+ # Suspend.
+ engine.suspend()
+ assert engine.state == EngineState.SUSPENDED
+ assert not context.is_tensor_state_allocated
+
+ gc.collect()
+ torch.cuda.empty_cache()
+ torch.cuda.synchronize()
+ mem_suspended = torch.cuda.memory_allocated()
+ if uses_tms:
+ phys_mem_suspended = torch.cuda.mem_get_info()[0]
+
+ if deallocates and not uses_tms:
+ assert mem_suspended < mem_before, (
+ f"GPU memory should decrease after suspend "
+ f"(mode={kv_cache_management_mode}). "
+ f"Before: {mem_before}, After: {mem_suspended}"
+ )
+ else:
+ assert mem_suspended == mem_before, (
+ f"Memory should not change on suspend. "
+ f"Before: {mem_before}, Suspended: {mem_suspended}"
+ )
+
+ if uses_tms:
+ assert phys_mem_suspended > phys_mem_before, (
+ f"torch_memory_saver should free physical GPU memory after suspend. "
+ f"Before: {phys_mem_before}, After: {phys_mem_suspended}"
+ )
+
+ # Resume.
+ engine.resume()
+ assert engine.state != EngineState.SUSPENDED
+ assert context.is_tensor_state_allocated
+
+ if deallocates and not uses_tms:
+ torch.cuda.synchronize()
+ mem_resumed = torch.cuda.memory_allocated()
+ assert mem_resumed > mem_suspended, (
+ f"GPU memory should increase after resume. "
+ f"Suspended: {mem_suspended}, Resumed: {mem_resumed}"
+ )
+
+ if uses_tms:
+ torch.cuda.synchronize()
+ phys_mem_resumed = torch.cuda.mem_get_info()[0]
+ assert phys_mem_resumed < phys_mem_suspended, (
+ f"torch_memory_saver should re-allocate physical GPU memory after resume. "
+ f"Suspended: {phys_mem_suspended}, Resumed: {phys_mem_resumed}"
+ )
+
+ # Data integrity.
+ if preserves_data:
+ torch.testing.assert_close(
+ context.memory_buffer,
+ expected,
+ msg="memory_buffer data must be identical after suspend/resume",
+ )
+
+ # Address stability when CUDA graphs persist.
+ if static_kv_memory_pointers:
+ addr_after = context.memory_buffer.data_ptr()
+ assert addr_before == addr_after, (
+ f"Tensor address must be stable when static_kv_memory_pointers is set. "
+ f"Before: {addr_before:#x}, After: {addr_after:#x}"
+ )
+
+ @pytest.mark.skipif(
+ not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
+ )
+ @pytest.mark.parametrize("use_checkpoint", [False, True], ids=["persist", "recompute"])
+ @torch.inference_mode()
+ def test_staleness_tracking(self, use_checkpoint):
+ """Test that training-iteration stamps are correctly tracked.
+ The use_checkpoint parameter simulates the behavior of different kv_cache_management_mode.
+ """
+ PROMPT_LEN = 8
+ NUM_TOKENS = 8
+
+ test_config = DynamicEngineTestConfig(
+ num_requests=0,
+ min_prompt_length=PROMPT_LEN,
+ max_prompt_length=PROMPT_LEN,
+ num_tokens_to_generate=NUM_TOKENS,
+ )
+ env = self._build_test_env(test_config)
+ engine = env.engine
+
+ for i in range(2):
+ prompt_tokens = torch.randint(
+ 0,
+ test_config.vocab_size - 1,
+ (PROMPT_LEN,),
+ dtype=torch.int64,
+ device=torch.cuda.current_device(),
+ )
+ engine._add_request(
+ DynamicInferenceRequest(
+ request_id=i,
+ prompt_tokens=prompt_tokens,
+ sampling_params=SamplingParams(
+ num_tokens_to_generate=NUM_TOKENS, termination_id=-1
+ ),
+ )
+ )
+
+ def set_epoch(epoch):
+ """Simulate receiving a SET_GENERATION_EPOCH signal."""
+ engine._generation_epoch = epoch
+ for entry in engine.requests.values():
+ request = entry.record[-1]
+ total = len(request.prompt_tokens) + len(request.generated_tokens)
+ if total > 0:
+ boundary = (total - 1, epoch)
+ if request.policy_epoch is None:
+ request.policy_epoch = [(0, epoch)]
+ else:
+ request.policy_epoch.append(boundary)
+ if request.kv_cache_epoch is None:
+ request.kv_cache_epoch = [(0, epoch)]
+ else:
+ request.kv_cache_epoch.append(boundary)
+
+ # Steps without a generation epoch set — no stamps.
+ engine.step_modern()
+ for entry in engine.requests.values():
+ assert entry.record[-1].policy_epoch is None
+ assert entry.record[-1].kv_cache_epoch is None
+
+ # Generation epoch 0: stamps all active requests at their current length.
+ set_epoch(0)
+ for _ in range(2):
+ engine.step_modern()
+
+ for entry in engine.requests.values():
+ ps = entry.record[-1].policy_epoch
+ ks = entry.record[-1].kv_cache_epoch
+ assert ps == ks == [(0, 0)]
+
+ # Generation epoch 1: boundary at current length, before next step.
+ set_epoch(1)
+ for _ in range(3):
+ engine.step_modern()
+
+ for entry in engine.requests.values():
+ ps = entry.record[-1].policy_epoch
+ ks = entry.record[-1].kv_cache_epoch
+ assert ps == ks == [(0, 0), (PROMPT_LEN + 2, 1)]
+
+ # Simulate RECOMPUTE — checkpoint clears kv_cache so the engine's
+ # stamping logic will recreate it fresh on the next epoch signal.
+ if use_checkpoint:
+ for entry in engine.requests.values():
+ old_req = entry.record[-1]
+ event_add_engine = old_req.event_add_engine
+ entry.record.checkpoint()
+ # Prevent TTFT crash due to missing _add_request in test.
+ entry.record[-1].event_add_engine = event_add_engine
+
+ for entry in engine.requests.values():
+ assert entry.record[-1].kv_cache_epoch is None
+
+ # Generation epoch 2: stamp then generate remaining tokens.
+ set_epoch(2)
+
+ finished_records = []
+ while engine.has_unfinished_requests():
+ result = engine.step_modern()
+ finished_records.extend(result["finished_request_records"])
+
+ for record in finished_records:
+ merged = record.merge()
+
+ assert merged.policy_epoch == [(0, 0), (PROMPT_LEN + 2, 1), (PROMPT_LEN + 5, 2)]
+
+ if use_checkpoint:
+ # KV cache was cleared by checkpoint; stamping logic recreated it at epoch 2.
+ assert merged.kv_cache_epoch == [(0, 2)]
+ else:
+ assert merged.kv_cache_epoch == [(0, 0), (PROMPT_LEN + 2, 1), (PROMPT_LEN + 5, 2)]
+
+ # Verify checkpoint clears kv_cache_epoch and preserves policy.
+ record = finished_records[0]
+ record.checkpoint()
+ assert record[-1].policy_epoch == merged.policy_epoch
+ assert record[-1].kv_cache_epoch is None
+
+ @pytest.mark.internal
+ @pytest.mark.skipif(
+ not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
+ )
+ @torch.inference_mode()
+ def test_speculative_decoding_with_early_termination(self):
+ """Test that speculative decoding handles premature request termination safely
+ (e.g. hitting max_sequence_length mid-speculative-batch)."""
+
+ # Set max_sequence_length tight so it terminates during a speculative step
+ test_config = DynamicEngineTestConfig(
+ num_requests=1,
+ min_prompt_length=4,
+ max_prompt_length=4,
+ num_tokens_to_generate=3, # Prompt (4) + Gen (3) = 7
+ max_sequence_length=7, # Will force termination after 3 tokens
+ model_provider="gpt",
+ num_speculative_tokens=3,
+ materialize_only_last_token_logits=False,
+ )
+
+ env = self._build_test_env(test_config)
+ unwrapped_model = env.engine.controller.inference_wrapped_model.model
+
+ # Mock forward to return deterministic data so speculative tokens are always accepted
+ hidden_size = unwrapped_model.config.hidden_size
+
+ def mock_mtp_forward(*args, **kwargs):
+ tokens = kwargs.get("tokens", args[0] if args else kwargs.get("input_ids"))
+
+ base_logits = torch.zeros(
+ tokens.size(0),
+ tokens.size(1),
+ test_config.vocab_size,
+ device=tokens.device,
+ dtype=torch.bfloat16,
+ )
+ base_logits[:, :, 0] = 100.0 # High probability for token 0
+
+ # Cache hidden states for serial MTP computation
+ unwrapped_model._decoder_hidden_states_cache = torch.zeros(
+ tokens.size(1), 1, hidden_size, device=tokens.device, dtype=torch.bfloat16
+ )
+ return base_logits
+
+ def mock_compute_mtp_single_step(
+ hidden_states, next_token_ids, position_ids, depth, runtime_gather_output=True
+ ):
+ n = hidden_states.size(0)
+ logits = torch.zeros(
+ n, 1, test_config.vocab_size, device=hidden_states.device, dtype=torch.bfloat16
+ )
+ logits[:, :, 0] = 100.0 # High probability for token 0
+ return hidden_states, logits
+
+ unwrapped_model.forward = mock_mtp_forward
+ unwrapped_model.compute_mtp_single_step = mock_compute_mtp_single_step
+
+ env.engine._add_request(env.requests[0])
+ env.engine.schedule_waiting_requests()
+
+ # Step engine until finished naturally
+ # This allows the bookkeeping logic to gracefully truncate the
+ # speculative tokens to the max_sequence_length boundary.
+ while env.engine.has_unfinished_requests():
+ env.engine.step_modern()
+
+ assert env.requests[0].status == Status.COMPLETED
+
+ # It should trim the output to the max_sequence_length boundary
+ # Prompt was 4, Max was 7, so it should have generated exactly 3 tokens.
+ assert len(env.requests[0].generated_tokens) == 3
+
+ # Validate the engine's tracking state is clean
+ assert env.engine.context.active_token_count == 0
+ assert env.engine.context.total_request_count == 0
+
+ @pytest.mark.internal
+ @torch.inference_mode()
+ def test_speculative_block_boundary_crossing(self):
+ """Test to verify KV cache block boundary crossing logic.
+
+ When a request fills exactly one block and speculative decoding generates
+ multiple tokens, the first new token shouldn't incorrectly overwrite the old block.
+ """
+ test_config = DynamicEngineTestConfig(
+ num_requests=1,
+ min_prompt_length=256,
+ max_prompt_length=256,
+ num_tokens_to_generate=3,
+ num_speculative_tokens=2,
+ context_block_size_tokens=256, # Exactly matches prompt length
+ context_max_requests=16,
+ model_provider="gpt",
+ materialize_only_last_token_logits=False,
+ use_fixed_output_lengths=True,
+ )
+ env = self._build_test_env(test_config)
+
+ req = env.requests[0]
+ req.sampling_params.num_tokens_to_generate = 3
+ env.engine._add_request(req)
+ env.engine.schedule_waiting_requests()
+
+ # Step 1: Prefill. Processes the 4 prompt tokens.
+ # At the end of this step, `update_requests` prepares the token indices for Step 2.
+ # It assigns block indices for the 3 upcoming tokens (1 base + 2 spec).
+ env.engine.step_modern()
+
+ context = env.engine.context
+
+ # The request has 2 blocks allocated now (1 for prompt, 1 for the new 3 tokens)
+ assigned_blocks = context.request_to_kv_block_ids[0]
+ first_block = assigned_blocks[0].item()
+ second_block = assigned_blocks[1].item()
+
+ # The active_token_count for the next step should be 3
+ assert context.active_token_count == 3
+
+ # Check which blocks the 3 new tokens are assigned to.
+ # Because the prompt exactly filled the first block, ALL 3 new tokens
+ # MUST go to the second block.
+ token_blocks = context.token_to_block_idx[: context.active_token_count].tolist()
+
+ assert token_blocks == [
+ second_block,
+ second_block,
+ second_block,
+ ], f"Expected all new tokens to go to block {second_block}, but got {token_blocks}."
+
+ @pytest.mark.internal
+ @pytest.mark.skipif(
+ not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
+ )
+ @torch.inference_mode()
+ def test_speculative_stop_word_hit(self):
+ """Test that if an accepted speculative token completes a stop word,
+ the request correctly triggers the stop logic without crashing."""
+
+ test_config = DynamicEngineTestConfig(
+ num_requests=0, # We will manually add our request cleanly
+ min_prompt_length=4,
+ max_prompt_length=4,
+ num_tokens_to_generate=10,
+ num_speculative_tokens=2,
+ materialize_only_last_token_logits=False,
+ model_provider="gpt",
+ )
+ env = self._build_test_env(test_config)
+
+ unwrapped_model = env.engine.controller.inference_wrapped_model.model
+ hidden_size = unwrapped_model.config.hidden_size
+
+ # Mock forward to deterministically output an ascending sequence (1->2->3...)
+ def mock_deterministic_forward(*args, **kwargs):
+ tokens = kwargs.get("tokens", args[0] if args else kwargs.get("input_ids"))
+ b, s = tokens.shape
+
+ base_logits = torch.zeros(
+ b, s, test_config.vocab_size, device=tokens.device, dtype=torch.bfloat16
+ )
+ next_toks = (tokens + 1).clamp(max=test_config.vocab_size - 1)
+ base_logits.scatter_(2, next_toks.unsqueeze(-1), 100.0)
+
+ # Cache hidden states for serial MTP computation
+ unwrapped_model._decoder_hidden_states_cache = torch.zeros(
+ s, 1, hidden_size, device=tokens.device, dtype=torch.bfloat16
+ )
+ return base_logits
+
+ def mock_compute_mtp_single_step(
+ hidden_states, next_token_ids, position_ids, depth, runtime_gather_output=True
+ ):
+ n = hidden_states.size(0)
+ # Predict next_token_ids + 1 (continuing the ascending sequence)
+ pred_toks = (next_token_ids + 1).clamp(max=test_config.vocab_size - 1)
+ logits = torch.zeros(
+ n, 1, test_config.vocab_size, device=hidden_states.device, dtype=torch.bfloat16
+ )
+ logits.scatter_(2, pred_toks.transpose(0, 1).unsqueeze(-1), 100.0)
+ return hidden_states, logits
+
+ unwrapped_model.forward = mock_deterministic_forward
+ unwrapped_model.compute_mtp_single_step = mock_compute_mtp_single_step
+
+ # Add the request formally to ensure all internal state tensors align
+ env.engine.add_request(
+ request_id=0,
+ prompt=torch.tensor([1, 2, 3, 4], device='cuda'),
+ sampling_params=SamplingParams(
+ num_tokens_to_generate=10, termination_id=99, detokenize_stop_sequence=True
+ ),
+ )
+
+ # Inject the parsed stop word IDs
+ tracked_req = env.engine.get_request(0)
+ tracked_req.stop_word_ids = [[8, 9]] # The sequence will generate 5, 6, 7, 8, 9, ...
+
+ finished_records = []
+ while env.engine.has_unfinished_requests():
+ res = env.engine.step_modern()
+ finished_records.extend(res["finished_request_records"])
+
+ # Retrieve the finalized request from the engine's output
+ finished_req = finished_records[0].merge()
+
+ assert finished_req.status == Status.COMPLETED
+ # Since num_tokens_to_generate=10, output should stop early at ~7 tokens
+ assert len(finished_req.generated_tokens) < 10
+ # Verify the stop word was actually generated and caused the termination
+ token_pairs = [
+ finished_req.generated_tokens[i : i + 2]
+ for i in range(len(finished_req.generated_tokens) - 1)
+ ]
+ assert [8, 9] in token_pairs
+
+ @pytest.mark.internal
+ @pytest.mark.skipif(
+ not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
+ )
+ @torch.inference_mode()
+ def test_speculative_long_stop_word_hit(self):
+ """Test that if an accepted speculative token completes a long stop word
+ (length > num_speculative_tokens), it is correctly detected."""
+
+ test_config = DynamicEngineTestConfig(
+ num_requests=0,
+ min_prompt_length=4,
+ max_prompt_length=4,
+ num_tokens_to_generate=10,
+ num_speculative_tokens=2,
+ materialize_only_last_token_logits=False,
+ model_provider="gpt",
+ )
+ env = self._build_test_env(test_config)
+
+ unwrapped_model = env.engine.controller.inference_wrapped_model.model
+ hidden_size = unwrapped_model.config.hidden_size
+
+ # Mock forward to deterministically output an ascending sequence
+ def mock_deterministic_forward(*args, **kwargs):
+ tokens = kwargs.get("tokens", args[0] if args else kwargs.get("input_ids"))
+ b, s = tokens.shape
+
+ base_logits = torch.zeros(
+ b, s, test_config.vocab_size, device=tokens.device, dtype=torch.bfloat16
+ )
+ next_toks = (tokens + 1).clamp(max=test_config.vocab_size - 1)
+ base_logits.scatter_(2, next_toks.unsqueeze(-1), 100.0)
+
+ # Cache hidden states for serial MTP computation
+ unwrapped_model._decoder_hidden_states_cache = torch.zeros(
+ s, 1, hidden_size, device=tokens.device, dtype=torch.bfloat16
+ )
+ return base_logits
+
+ def mock_compute_mtp_single_step(
+ hidden_states, next_token_ids, position_ids, depth, runtime_gather_output=True
+ ):
+ n = hidden_states.size(0)
+ # Predict next_token_ids + 1 (continuing the ascending sequence)
+ pred_toks = (next_token_ids + 1).clamp(max=test_config.vocab_size - 1)
+ logits = torch.zeros(
+ n, 1, test_config.vocab_size, device=hidden_states.device, dtype=torch.bfloat16
+ )
+ logits.scatter_(2, pred_toks.transpose(0, 1).unsqueeze(-1), 100.0)
+ return hidden_states, logits
+
+ unwrapped_model.forward = mock_deterministic_forward
+ unwrapped_model.compute_mtp_single_step = mock_compute_mtp_single_step
+
+ env.engine.add_request(
+ request_id=0,
+ prompt=torch.tensor([1, 2, 3, 4], device='cuda'),
+ sampling_params=SamplingParams(
+ num_tokens_to_generate=10, termination_id=99, detokenize_stop_sequence=True
+ ),
+ )
+
+ # Stop word length 3 > num_speculative_tokens (2)
+ tracked_req = env.engine.get_request(0)
+ tracked_req.stop_word_ids = [[7, 8, 9]]
+
+ finished_records = []
+ while env.engine.has_unfinished_requests():
+ res = env.engine.step_modern()
+ finished_records.extend(res["finished_request_records"])
+
+ finished_req = finished_records[0].merge()
+
+ assert finished_req.status == Status.COMPLETED
+ assert len(finished_req.generated_tokens) < 10
+ token_triplets = [
+ finished_req.generated_tokens[i : i + 3]
+ for i in range(len(finished_req.generated_tokens) - 2)
+ ]
+ assert [7, 8, 9] in token_triplets
+
+ @pytest.mark.internal
+ @pytest.mark.skipif(
+ not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
+ )
+ @torch.inference_mode()
+ def test_speculative_stop_word_truncates_trailing_tokens(self):
+ """Test that when a stop word lands in the middle of speculative tokens,
+ the extra tokens generated after the stop word are removed.
+
+ With num_speculative_tokens=2, each step produces up to 3 tokens
+ (1 base + 2 speculative). If the stop word is [6] and the engine
+ generates [5, 6, 7] in one step, token 7 must be truncated so the
+ output ends with the stop word [6]."""
+
+ test_config = DynamicEngineTestConfig(
+ num_requests=0,
+ min_prompt_length=4,
+ max_prompt_length=4,
+ num_tokens_to_generate=10,
+ num_speculative_tokens=2,
+ materialize_only_last_token_logits=False,
+ model_provider="gpt",
+ )
+ env = self._build_test_env(test_config)
+
+ unwrapped_model = env.engine.controller.inference_wrapped_model.model
+ hidden_size = unwrapped_model.config.hidden_size
+
+ # Mock forward to deterministically output an ascending sequence (1->2->3...)
+ def mock_deterministic_forward(*args, **kwargs):
+ tokens = kwargs.get("tokens", args[0] if args else kwargs.get("input_ids"))
+ b, s = tokens.shape
+
+ base_logits = torch.zeros(
+ b, s, test_config.vocab_size, device=tokens.device, dtype=torch.bfloat16
+ )
+ next_toks = (tokens + 1).clamp(max=test_config.vocab_size - 1)
+ base_logits.scatter_(2, next_toks.unsqueeze(-1), 100.0)
+
+ # Cache hidden states for serial MTP computation
+ unwrapped_model._decoder_hidden_states_cache = torch.zeros(
+ s, 1, hidden_size, device=tokens.device, dtype=torch.bfloat16
+ )
+ return base_logits
+
+ def mock_compute_mtp_single_step(
+ hidden_states, next_token_ids, position_ids, depth, runtime_gather_output=True
+ ):
+ n = hidden_states.size(0)
+ # Predict next_token_ids + 1 (continuing the ascending sequence)
+ pred_toks = (next_token_ids + 1).clamp(max=test_config.vocab_size - 1)
+ logits = torch.zeros(
+ n, 1, test_config.vocab_size, device=hidden_states.device, dtype=torch.bfloat16
+ )
+ logits.scatter_(2, pred_toks.transpose(0, 1).unsqueeze(-1), 100.0)
+ return hidden_states, logits
+
+ unwrapped_model.forward = mock_deterministic_forward
+ unwrapped_model.compute_mtp_single_step = mock_compute_mtp_single_step
+
+ env.engine.add_request(
+ request_id=0,
+ prompt=torch.tensor([1, 2, 3, 4], device='cuda'),
+ sampling_params=SamplingParams(
+ num_tokens_to_generate=10, termination_id=99, detokenize_stop_sequence=True
+ ),
+ )
+
+ # Stop word [6] will land in the middle of a speculative batch [5, 6, 7].
+ # Token 7 should be truncated from the output.
+ tracked_req = env.engine.get_request(0)
+ tracked_req.stop_word_ids = [[6]]
+
+ finished_records = []
+ while env.engine.has_unfinished_requests():
+ res = env.engine.step_modern()
+ finished_records.extend(res["finished_request_records"])
+
+ finished_req = finished_records[0].merge()
+
+ assert finished_req.status == Status.COMPLETED
+ # The output should end exactly at the stop word, with no trailing tokens.
+ assert finished_req.generated_tokens[-1] == 6, (
+ f"Expected last token to be stop word 6, "
+ f"got {finished_req.generated_tokens[-1]}. "
+ f"Trailing tokens after stop word were not truncated. "
+ f"Full output: {finished_req.generated_tokens}"
+ )
+ # Verify no tokens after the stop word exist
+ assert 7 not in finished_req.generated_tokens, (
+ f"Token 7 should have been truncated after stop word 6. "
+ f"Full output: {finished_req.generated_tokens}"
+ )
+
+ @pytest.mark.parametrize("detokenize_stop_sequence", [True, False])
+ def test_detokenize_stop_sequence_flag(self, detokenize_stop_sequence):
+ """Test that _check_stop_words_for_request_post_append strips or keeps
+ the stop word tokens based on detokenize_stop_sequence."""
+ engine = types.SimpleNamespace(num_speculative_tokens=0)
+ check = DynamicInferenceEngine._check_stop_words_for_request_post_append
+
+ request = types.SimpleNamespace(
+ generated_tokens=[1, 2, 3, 4, 5],
+ stop_word_ids=[[4, 5]],
+ sampling_params=SamplingParams(detokenize_stop_sequence=detokenize_stop_sequence),
+ )
+ hit, trimmed = check(engine, request)
+ assert hit
+ if detokenize_stop_sequence:
+ # Stop word kept
+ assert request.generated_tokens == [1, 2, 3, 4, 5]
+ assert trimmed == 0
+ else:
+ # Stop word stripped
+ assert request.generated_tokens == [1, 2, 3]
+ assert trimmed == 2
+
+ @pytest.mark.internal
+ @torch.inference_mode()
+ def test_speculative_sequence_length_double_counting(self):
+ """Test to verify active_sequence_lengths is not double-counted.
+
+ If active sequence length is double-counted during speculative decoding,
+ the request will terminate prematurely before generating the requested tokens.
+ """
+ test_config = DynamicEngineTestConfig(
+ num_requests=0,
+ min_prompt_length=4,
+ max_prompt_length=4,
+ num_tokens_to_generate=6,
+ max_sequence_length=10, # Exactly prompt (4) + generate (6)
+ context_max_requests=16,
+ num_speculative_tokens=2,
+ model_provider="gpt",
+ materialize_only_last_token_logits=False,
+ use_fixed_output_lengths=False,
+ context_max_tokens=512,
+ )
+ env = self._build_test_env(test_config)
+
+ # Mock forward pass to return deterministic base logits.
+ # Speculative tokens will be wrong (predicted by MTP as tokens + 5)
+ # to guarantee rejection every time.
+ model = env.engine.controller.inference_wrapped_model.model
+ hidden_size = model.config.hidden_size
+
+ def mock_mtp_forward_reject(*args, **kwargs):
+ tokens = kwargs.get("tokens", args[0] if args else kwargs.get("input_ids"))
+ b, s = tokens.shape
+
+ # Base model correctly predicts tokens + 1
+ base_logits = torch.zeros(
+ b, s, test_config.vocab_size, device=tokens.device, dtype=torch.bfloat16
+ )
+ next_toks = (tokens + 1).clamp(max=test_config.vocab_size - 1)
+ base_logits.scatter_(2, next_toks.unsqueeze(-1), 100.0)
+
+ # Cache hidden states for serial MTP computation
+ model._decoder_hidden_states_cache = torch.zeros(
+ s, 1, hidden_size, device=tokens.device, dtype=torch.bfloat16
+ )
+ return base_logits
+
+ def mock_compute_mtp_single_step(
+ hidden_states, next_token_ids, position_ids, depth, runtime_gather_output=True
+ ):
+ n = hidden_states.size(0)
+ # Predict wildly wrong tokens (+ 5) to guarantee rejection
+ wrong_toks = (next_token_ids + 5).clamp(max=test_config.vocab_size - 1)
+ logits = torch.zeros(
+ n, 1, test_config.vocab_size, device=hidden_states.device, dtype=torch.bfloat16
+ )
+ logits.scatter_(2, wrong_toks.transpose(0, 1).unsqueeze(-1), 100.0)
+ return hidden_states, logits
+
+ model.forward = mock_mtp_forward_reject
+ model.compute_mtp_single_step = mock_compute_mtp_single_step
+
+ env.engine.add_request(
+ request_id=0,
+ prompt=torch.tensor([1, 2, 3, 4], device='cuda'),
+ sampling_params=SamplingParams(num_tokens_to_generate=6, termination_id=99),
+ )
+
+ finished_records = []
+ while env.engine.has_unfinished_requests():
+ res = env.engine.step_modern()
+ finished_records.extend(res["finished_request_records"])
+
+ finished_req = finished_records[0].merge()
+
+ # If there is double counting, the tracked active length will outpace the actual
+ # generated tokens, causing premature termination when it thinks it hit max_sequence_length.
+ assert finished_req.status == Status.COMPLETED
+ assert (
+ len(finished_req.generated_tokens) == 6
+ ), f"Expected 6 tokens, got {len(finished_req.generated_tokens)}. Double counting occurred."
+
+ @pytest.mark.internal
+ @pytest.mark.skipif(
+ not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
+ )
+ @torch.inference_mode()
+ def test_speculative_decoding_with_eviction_and_swapping(self):
+ """Test that speculative decoding works correctly when requests are paused and evicted.
+
+ This exercises the `_swap_book_keeping_tensors` logic with the 2D `new_speculative_tokens`
+ tensor, ensuring no dimensional mismatch or index errors occur during tensor swapping.
+ """
+ # Very constrained memory environment to force pausing and eviction
+ test_config = DynamicEngineTestConfig(
+ num_requests=3,
+ min_prompt_length=256,
+ max_prompt_length=256,
+ num_tokens_to_generate=512,
+ context_block_size_tokens=256,
+ num_speculative_tokens=2,
+ context_buffer_size_gb=0.00064, # 640 KB
+ context_paused_buffer_size_gb=0.0, # 0 paused buffer forces immediate eviction
+ model_provider="gpt",
+ materialize_only_last_token_logits=False,
+ use_fixed_output_lengths=True,
+ )
+
+ env = self._build_test_env(test_config)
+
+ unwrapped_model = env.engine.controller.inference_wrapped_model.model
+ hidden_size = unwrapped_model.config.hidden_size
+
+ # Mock forward pass to return safe, deterministic logits to avoid NaN/Inf crashes
+ # in torch.multinomial caused by randomly initialized weights.
+ def mock_safe_forward(*args, **kwargs):
+ tokens = kwargs.get("tokens", args[0] if args else kwargs.get("input_ids"))
+ b, s = tokens.shape
+
+ base_logits = torch.zeros(
+ b, s, test_config.vocab_size, device=tokens.device, dtype=torch.bfloat16
+ )
+ base_logits[:, :, 0] = 100.0 # Force model to deterministically pick token 0
+
+ # Cache hidden states for serial MTP computation
+ unwrapped_model._decoder_hidden_states_cache = torch.zeros(
+ s, 1, hidden_size, device=tokens.device, dtype=torch.bfloat16
+ )
+ return base_logits
+
+ def mock_compute_mtp_single_step(
+ hidden_states, next_token_ids, position_ids, depth, runtime_gather_output=True
+ ):
+ n = hidden_states.size(0)
+ logits = torch.zeros(
+ n, 1, test_config.vocab_size, device=hidden_states.device, dtype=torch.bfloat16
+ )
+ logits[:, :, 0] = 100.0 # Force speculative heads to also pick token 0
+ return hidden_states, logits
+
+ unwrapped_model.forward = mock_safe_forward
+ unwrapped_model.compute_mtp_single_step = mock_compute_mtp_single_step
+
+ # Add all requests at once. They will all start prefill, but as they generate
+ # and request more blocks, the engine will run out of active blocks.
+ # Since paused_buffer_size is 0, any request that pauses will immediately
+ # overflow the paused buffer and trigger an eviction.
+ for request in env.requests:
+ request.sampling_params.num_tokens_to_generate = 512
+ env.engine._add_request(request)
+
+ eviction_occurred = False
+
+ # Step the engine manually until all requests finish.
+ while env.engine.has_unfinished_requests():
+ # Record the number of evicted requests before the step
+ evicted_before = env.engine.evicted_request_count
+
+ # Step the engine
+ env.engine.schedule_waiting_requests()
+ env.engine.step_modern()
+
+ # Check if any request was evicted during this step
+ if env.engine.evicted_request_count > evicted_before:
+ eviction_occurred = True
+
+ # Assert that our constrained memory actually caused an eviction,
+ # proving we exercised the evict_overflow_paused_requests path with spec tokens.
+ assert (
+ eviction_occurred
+ ), "Test failed to trigger an eviction. The test environment memory wasn't tight enough."
+
+ # Verify all requests successfully went back through the queue and finished cleanly.
+ # We MUST check the merged records from the engine, because eviction checkpoints
+ # the requests, leaving the original instances in env.requests permanently active.
+ for request_id, entry in env.engine.requests.items():
+ merged_req = entry.record.merge()
+ assert (
+ merged_req.status == Status.COMPLETED
+ ), f"Request {request_id} failed to complete."
+ assert (
+ len(merged_req.generated_tokens) == 511
+ ), f"Request {request_id} didn't generate expected tokens."
+
+ @pytest.mark.internal
+ @pytest.mark.skipif(
+ not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
+ )
+ @torch.inference_mode()
+ def test_speculative_decoding_with_prefix_caching(self):
+ """Test that speculative decoding works correctly when prefix caching is enabled.
+
+ Two requests share the same prompt prefix. The second request should reuse
+ cached KV blocks from the first and still generate correctly with spec decoding.
+ """
+ test_config = DynamicEngineTestConfig(
+ num_requests=0, # Added manually below
+ min_prompt_length=256,
+ max_prompt_length=256,
+ num_tokens_to_generate=4,
+ num_speculative_tokens=2,
+ enable_prefix_caching=True, # Set at config level
+ context_block_size_tokens=256, # Ensure exact 1 block per prompt
+ materialize_only_last_token_logits=False,
+ model_provider="gpt",
+ context_max_tokens=4096,
+ context_max_requests=512,
+ max_sequence_length=1024,
+ )
+ env = self._build_test_env(test_config)
+
+ # Create two pairs of requests with identical shared prefixes.
+ shared_prompt_a = torch.randint(
+ 0, test_config.vocab_size - 1, (256,), dtype=torch.int64, device='cuda'
+ )
+ shared_prompt_b = torch.randint(
+ 0, test_config.vocab_size - 1, (256,), dtype=torch.int64, device='cuda'
+ )
+
+ prompts = [shared_prompt_a, shared_prompt_a, shared_prompt_b, shared_prompt_b]
+
+ for i, prompt in enumerate(prompts):
+ # Using the clean public API guarantees correct hashing and dataclass creation
+ env.engine.add_request(
+ request_id=i,
+ prompt=prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=128, termination_id=99),
+ )
+
+ # First, run schedule_waiting_requests and ONE step to allocate the prefill blocks.
+ # Req 0 and 2 will schedule immediately. Req 1 and 3 will defer because their hashes
+ # are currently pending (being registered by 0 and 2).
+ env.engine.schedule_waiting_requests()
+ env.engine.step_modern()
+
+ # After step 1, Req 0 and 2 have completely registered their cached blocks.
+ # Now, schedule the deferred ones (Req 1 and 3). They will find the registered blocks!
+ env.engine.schedule_waiting_requests()
+ env.engine.step_modern()
+
+ # 4 requests. 2 unique prefixes (1 block each).
+ # Without sharing, we'd need 8 blocks + 1 dummy = 9 active_used.
+ # With sharing, we need 2 shared blocks + 4 generation blocks + 1 dummy = 7 active_used.
+ active_used = env.engine.context.kv_block_allocator.get_active_used()
+ assert (
+ active_used <= 7
+ ), f"Prefix caching failed, expected <= 7 active blocks but got {active_used}"
+
+ while env.engine.has_unfinished_requests():
+ env.engine.step_modern()
+
+ # Context should be clean after all requests finish.
+ assert env.engine.context.active_token_count == 0
+ assert env.engine.context.total_request_count == 0
+
+ @pytest.mark.internal
+ @pytest.mark.skipif(
+ not is_fa_min_version("2.7.3"), reason="need latest flash attn for dynamic batching"
+ )
+ @torch.inference_mode()
+ def test_speculative_decoding_chunked_prefill_and_prefix_caching(self):
+ """End-to-end test combining speculative decoding, chunked prefill, and prefix caching.
+
+ Verifies that all three features interact correctly:
+ - Prefix caching shares KV blocks between requests with common prompts
+ - Chunked prefill processes long prompts in chunks
+ - Speculative decoding generates multiple tokens per step
+ """
+ test_config = DynamicEngineTestConfig(
+ num_requests=0,
+ min_prompt_length=512,
+ max_prompt_length=512,
+ num_tokens_to_generate=128,
+ num_speculative_tokens=2,
+ materialize_only_last_token_logits=False,
+ enable_chunked_prefill=True,
+ enable_prefix_caching=True, # Set at config level
+ context_block_size_tokens=256,
+ model_provider="gpt",
+ context_max_tokens=1536, # Force chunking
+ context_max_requests=48,
+ )
+ env = self._build_test_env(test_config)
+
+ # Create identical prompts for all 4 requests
+ shared_prompt = torch.randint(
+ 0, test_config.vocab_size - 1, (512,), dtype=torch.int64, device='cuda'
+ )
+
+ for i in range(4):
+ env.engine.add_request(
+ request_id=i,
+ prompt=shared_prompt.clone(),
+ sampling_params=SamplingParams(num_tokens_to_generate=128, termination_id=99),
+ )
+
+ while env.engine.has_unfinished_requests():
+ env.engine.step_modern()
+
+ assert env.engine.context.active_token_count == 0
+ assert env.engine.context.total_request_count == 0
+
+
+CHUNKED_CG_BLOCK_SIZE = 256
+CHUNKED_CG_VOCAB_SIZE = 10000
+CHUNKED_CG_MAX_SEQ_LEN = 2048
+
+
+@pytest.mark.internal
+@pytest.mark.skipif(not is_fa_min_version("2.7.3"), reason="need flash attn")
+class TestChunkedPrefillCudaGraphs:
+ """Verify correctness across chunked prefill and CUDA graph combinations.
+
+ For each model type, runs a baseline config (no chunked prefill, no CUDA graphs)
+ and compares output tokens against every other combination.
+ """
+
+ @classmethod
+ def setup_class(cls):
+ Utils.initialize_model_parallel()
+
+ @classmethod
+ def teardown_class(cls):
+ set_rounder(64)
+ Utils.destroy_model_parallel()
+
+ def _create_model(self, model_provider, num_cuda_graphs):
+ """Create a GPT or Mamba model with optional CUDA graph support."""
+ cuda_graph_impl = "local" if num_cuda_graphs else "none"
+
+ if model_provider == "gpt":
+ config = TransformerConfig(
+ params_dtype=torch.bfloat16,
+ num_layers=4,
+ hidden_size=32,
+ num_attention_heads=4,
+ use_cpu_initialization=True,
+ cuda_graph_impl=cuda_graph_impl,
+ inference_rng_tracker=True,
+ tensor_model_parallel_size=1,
+ pipeline_model_parallel_size=1,
+ pipeline_dtype=torch.bfloat16,
+ add_bias_linear=True,
+ )
+ model = GPTModel(
+ config=config,
+ transformer_layer_spec=get_gpt_layer_local_spec(),
+ vocab_size=CHUNKED_CG_VOCAB_SIZE,
+ max_sequence_length=CHUNKED_CG_MAX_SEQ_LEN,
+ parallel_output=True,
+ pre_process=parallel_state.is_pipeline_first_stage(),
+ post_process=parallel_state.is_pipeline_last_stage(),
+ ).cuda()
+ elif model_provider == "mamba":
+ config = TransformerConfig(
+ params_dtype=torch.bfloat16,
+ num_layers=3,
+ hidden_size=256,
+ mamba_num_heads=16,
+ num_attention_heads=16,
+ use_cpu_initialization=True,
+ cuda_graph_impl=cuda_graph_impl,
+ inference_rng_tracker=True,
+ tensor_model_parallel_size=1,
+ pipeline_model_parallel_size=1,
+ pipeline_dtype=torch.bfloat16,
+ add_bias_linear=True,
+ is_hybrid_model=True,
+ )
+ model = MambaModel(
+ config=config,
+ mamba_stack_spec=mamba_stack_spec,
+ vocab_size=CHUNKED_CG_VOCAB_SIZE,
+ max_sequence_length=CHUNKED_CG_MAX_SEQ_LEN,
+ parallel_output=True,
+ hybrid_layer_pattern="M*-",
+ pre_process=parallel_state.is_pipeline_first_stage(),
+ post_process=parallel_state.is_pipeline_last_stage(),
+ ).cuda()
+ else:
+ raise ValueError(f"Invalid model_provider {model_provider}")
+
+ for param in model.parameters():
+ param.data = param.data.to(config.params_dtype)
+ model.eval()
+ return model
+
+ def _reset_cuda_graph_state(self, model):
+ """Reset all CUDA graph global and per-module state."""
+ _CudagraphGlobalRecord.cudagraph_created = False
+ _CudagraphGlobalRecord.cudagraph_record = []
+ _CudagraphGlobalRecord.cudagraph_inference_record = []
+ CudaGraphManager.global_mempool = None
+ for module in model.modules():
+ if isinstance(module, CudaGraphManager):
+ module.cudagraph_runners.clear()
+ module.inference_cudagraphs_lookup_table.clear()
+
+ def _build_engine(self, model, enable_chunked_prefill, num_cuda_graphs, context_max_tokens):
+ """Build an engine with the given chunked prefill / CUDA graph config."""
+ set_rounder(4)
+ mamba_config = MambaInferenceStateConfig.from_model(model)
+
+ inference_config_kwargs = dict(
+ max_sequence_length=CHUNKED_CG_MAX_SEQ_LEN,
+ buffer_size_gb=0.5,
+ block_size_tokens=CHUNKED_CG_BLOCK_SIZE,
+ materialize_only_last_token_logits=False,
+ unified_memory_level=0,
+ num_cuda_graphs=num_cuda_graphs,
+ use_cuda_graphs_for_non_decode_steps=True,
+ enable_chunked_prefill=enable_chunked_prefill,
+ max_tokens=context_max_tokens,
+ max_requests=128,
+ )
+ if mamba_config is not None:
+ inference_config_kwargs.update(mamba_inference_state_config=mamba_config)
+ context = DynamicInferenceContext(
+ model_config=model.config, inference_config=InferenceConfig(**inference_config_kwargs)
+ )
+ wrapper = GPTInferenceWrapper(model, context)
+ wrapper.model_is_pipeline_parallel = not (
+ parallel_state.is_pipeline_first_stage() and parallel_state.is_pipeline_last_stage()
+ )
+ controller = TextGenerationController(
+ inference_wrapped_model=wrapper,
+ tokenizer=types.SimpleNamespace(
+ vocab_size=CHUNKED_CG_VOCAB_SIZE, detokenize=lambda tokens: "tokenized_prompt"
+ ),
+ )
+ self._reset_cuda_graph_state(model)
+ return DynamicInferenceEngine(controller, context)
+
+ def _run_to_completion(self, engine, prompts, num_tokens_to_generate):
+ """Add all prompts and run to completion, returning {req_id: generated_tokens}."""
+ for i, prompt in enumerate(prompts):
+ request = DynamicInferenceRequest(
+ request_id=i,
+ prompt_tokens=prompt,
+ sampling_params=SamplingParams(
+ num_tokens_to_generate=num_tokens_to_generate, termination_id=-1, top_k=1
+ ),
+ block_size_tokens=CHUNKED_CG_BLOCK_SIZE,
+ )
+ engine._add_request(request)
+
+ finished = {}
+ step_count = 0
+ while engine.has_unfinished_requests():
+ result = engine.step_modern()
+ step_count += 1
+ for record in result["finished_request_records"]:
+ merged = record.merge()
+ finished[merged.request_id] = list(merged.generated_tokens)
+
+ return finished, step_count
+
+ @pytest.mark.parametrize("model_provider", ["gpt", "mamba"])
+ @pytest.mark.parametrize("chunked_prefill", [False, True])
+ @pytest.mark.parametrize("num_cuda_graphs", [None, 2])
+ @torch.inference_mode()
+ def test_chunked_prefill_cuda_graphs(self, model_provider, chunked_prefill, num_cuda_graphs):
+ """Verify generated tokens match across chunked prefill and CUDA graph configs."""
+ skip_if_mamba_sequence_packing_not_available(model_provider)
+
+ # Clear NVTE env vars set by conftest set_env fixture.
+ os.environ.pop('NVTE_FLASH_ATTN', None)
+ os.environ.pop('NVTE_FUSED_ATTN', None)
+ os.environ.pop('NVTE_UNFUSED_ATTN', None)
+
+ random.seed(123)
+ torch.manual_seed(123)
+ model_parallel_cuda_manual_seed(
+ seed=123, inference_rng_tracker=True, use_cudagraphable_rng=False, force_reset_rng=True
+ )
+
+ # Create model with CUDA graph support so it can be used for both CG and non-CG engines.
+ model = self._create_model(model_provider, num_cuda_graphs=2)
+
+ # 3 prompts of 512 tokens each, disjoint token ranges (no prefix sharing).
+ device = torch.cuda.current_device()
+ prompts = [
+ torch.arange(i * 600, i * 600 + 512, dtype=torch.int64, device=device) for i in range(3)
+ ]
+ num_tokens_to_generate = 8
+
+ # Token budget: 768 forces chunking when chunked_prefill=True
+ # (only ~1.5 of the 512-token prompts fit per step).
+ context_max_tokens = 768 if chunked_prefill else None
+
+ # Baseline: no chunked prefill, no CUDA graphs.
+ baseline_engine = self._build_engine(
+ model, enable_chunked_prefill=False, num_cuda_graphs=None, context_max_tokens=None
+ )
+ baseline_outputs, baseline_steps = self._run_to_completion(
+ baseline_engine, prompts, num_tokens_to_generate
+ )
+
+ # Test config.
+ test_engine = self._build_engine(
+ model,
+ enable_chunked_prefill=chunked_prefill,
+ num_cuda_graphs=num_cuda_graphs,
+ context_max_tokens=context_max_tokens,
+ )
+ test_outputs, test_steps = self._run_to_completion(
+ test_engine, prompts, num_tokens_to_generate
+ )
+
+ # Correctness: generated tokens must match baseline.
+ for req_id in range(3):
+ assert baseline_outputs[req_id] == test_outputs[req_id], (
+ f"req {req_id}: baseline {baseline_outputs[req_id]} != "
+ f"test {test_outputs[req_id]} "
+ f"(chunked_prefill={chunked_prefill}, num_cuda_graphs={num_cuda_graphs})"
+ )
+
+ # When chunked prefill is enabled with a constrained token budget, the engine
+ # needs more scheduling steps than the non-chunked baseline.
+ if chunked_prefill:
+ assert test_steps > baseline_steps, (
+ f"chunked prefill should need more steps than baseline "
+ f"({test_steps} <= {baseline_steps})"
+ )
diff --git a/tests/unit_tests/inference/engines/test_dynamic_events.py b/tests/unit_tests/inference/engines/test_dynamic_events.py
new file mode 100644
index 00000000000..a991dd5329c
--- /dev/null
+++ b/tests/unit_tests/inference/engines/test_dynamic_events.py
@@ -0,0 +1,703 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""Comprehensive tests for DynamicInferenceEvent and request event lifecycle.
+
+This test suite covers all 9 event types, payload validation, serialization,
+request methods, and lifecycle sequences with exactly 9 substantial tests.
+"""
+
+import time
+from unittest.mock import patch
+
+import pytest
+import torch
+
+from megatron.core.inference.contexts.dynamic_context import (
+ BlockOverflowError,
+ MaxSequenceLengthOverflowError,
+ RequestOverflowError,
+ TokenOverflowError,
+)
+from megatron.core.inference.inference_request import (
+ DynamicInferenceEvent,
+ DynamicInferenceEventType,
+ DynamicInferenceRequest,
+ DynamicInferenceRequestRecord,
+)
+from megatron.core.inference.sampling_params import SamplingParams
+
+# ============================================================================
+# Fixtures
+# ============================================================================
+
+
+@pytest.fixture
+def basic_request():
+ """Create a basic request for testing."""
+ return DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=torch.tensor([1, 2, 3, 4], dtype=torch.int64),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ )
+
+
+# ============================================================================
+# Test 1: All Event Types Creation and Payload Validation
+# ============================================================================
+
+
+def test_all_event_types_creation_and_payload_validation():
+ """Test all 9 event types exist, are unique, and enforce payload validation rules.
+
+ Coverage:
+ - All 9 event types (ADD_ENGINE, ADD_CONTEXT, GENERATED_TOKEN, PAUSE, EVICT,
+ FINISH, FAIL, ERROR_TRANSIENT, ERROR_NONTRANSIENT)
+ - Payload requirements (int for GENERATED_TOKEN, exception for errors, None for others)
+ - Auto-timestamp generation
+ - Invalid payload rejection via AssertionError
+ """
+ # Verify exactly 9 event types exist
+ expected_types = [
+ 'ADD_ENGINE',
+ 'ADD_CONTEXT',
+ 'GENERATED_TOKEN',
+ 'PAUSE',
+ 'EVICT',
+ 'FINISH',
+ 'FAIL',
+ 'ERROR_TRANSIENT',
+ 'ERROR_NONTRANSIENT',
+ ]
+ for event_type in expected_types:
+ assert hasattr(DynamicInferenceEventType, event_type), f"Missing: {event_type}"
+ assert len(DynamicInferenceEventType) == 9
+
+ # Verify all values are unique
+ values = [e.value for e in DynamicInferenceEventType]
+ assert len(values) == len(set(values)), "Duplicate event type values"
+
+ # Create events that require no payload
+ no_payload_types = [
+ DynamicInferenceEventType.ADD_ENGINE,
+ DynamicInferenceEventType.ADD_CONTEXT,
+ DynamicInferenceEventType.PAUSE,
+ DynamicInferenceEventType.EVICT,
+ DynamicInferenceEventType.FINISH,
+ DynamicInferenceEventType.FAIL,
+ ]
+ for event_type in no_payload_types:
+ event = DynamicInferenceEvent(type=event_type)
+ assert event.type == event_type
+ assert event.payload is None
+ assert event.timestamp is not None
+ # Verify these types reject non-None payloads
+ with pytest.raises(AssertionError):
+ DynamicInferenceEvent(type=event_type, payload="not allowed")
+
+ # Create GENERATED_TOKEN event with dict payload
+ token_event = DynamicInferenceEvent(
+ type=DynamicInferenceEventType.GENERATED_TOKEN, payload={"token_id": 42}
+ )
+ assert token_event.type == DynamicInferenceEventType.GENERATED_TOKEN
+ assert token_event.payload == {"token_id": 42}
+ assert token_event.timestamp is not None
+
+ # GENERATED_TOKEN rejects None, string, float, and int payloads (must be dict with "token_id")
+ with pytest.raises(AssertionError):
+ DynamicInferenceEvent(type=DynamicInferenceEventType.GENERATED_TOKEN)
+ with pytest.raises(AssertionError):
+ DynamicInferenceEvent(type=DynamicInferenceEventType.GENERATED_TOKEN, payload="bad")
+ with pytest.raises(AssertionError):
+ DynamicInferenceEvent(type=DynamicInferenceEventType.GENERATED_TOKEN, payload=3.14)
+ with pytest.raises(AssertionError):
+ DynamicInferenceEvent(type=DynamicInferenceEventType.GENERATED_TOKEN, payload=42)
+
+ # Create error events with exception payloads
+ transient_error = RequestOverflowError(request_id=1, message="Transient")
+ event_transient = DynamicInferenceEvent(
+ type=DynamicInferenceEventType.ERROR_TRANSIENT, payload=transient_error
+ )
+ assert event_transient.type == DynamicInferenceEventType.ERROR_TRANSIENT
+ assert event_transient.payload is transient_error
+
+ nontransient_error = MaxSequenceLengthOverflowError(request_id=2, message="Fatal")
+ event_nontransient = DynamicInferenceEvent(
+ type=DynamicInferenceEventType.ERROR_NONTRANSIENT, payload=nontransient_error
+ )
+ assert event_nontransient.type == DynamicInferenceEventType.ERROR_NONTRANSIENT
+ assert event_nontransient.payload is nontransient_error
+
+ # Error events reject None payloads
+ with pytest.raises(AssertionError):
+ DynamicInferenceEvent(type=DynamicInferenceEventType.ERROR_TRANSIENT)
+ with pytest.raises(AssertionError):
+ DynamicInferenceEvent(type=DynamicInferenceEventType.ERROR_NONTRANSIENT)
+
+ # Verify auto-timestamp is reasonable
+ before = time.time()
+ event = DynamicInferenceEvent(type=DynamicInferenceEventType.ADD_ENGINE)
+ after = time.time()
+ assert before <= event.timestamp <= after
+
+
+# ============================================================================
+# Test 2: Successful Request Lifecycle with Serialization Roundtrip
+# ============================================================================
+
+
+def test_successful_request_lifecycle_with_serialization_roundtrip(basic_request):
+ """Test full request lifecycle: ADD_ENGINE -> ADD_CONTEXT -> GENERATED_TOKEN(s) -> FINISH.
+
+ Coverage:
+ - Full successful lifecycle event sequence
+ - Event order preservation
+ - Request serialization/deserialization roundtrip
+ - generated_tokens as direct List[int] field
+ """
+ # Build complete successful lifecycle
+ basic_request.add_event_add_engine()
+ basic_request.add_event_add_context()
+ basic_request.generated_tokens.extend([100, 200, 300])
+ basic_request.add_event_generated_token(100)
+ basic_request.add_event_generated_token(200)
+ basic_request.add_event_generated_token(300)
+ basic_request.add_event_finish()
+
+ # Verify event sequence
+ event_types = [e.type for e in basic_request.events]
+ assert event_types == [
+ DynamicInferenceEventType.ADD_ENGINE,
+ DynamicInferenceEventType.ADD_CONTEXT,
+ DynamicInferenceEventType.GENERATED_TOKEN,
+ DynamicInferenceEventType.GENERATED_TOKEN,
+ DynamicInferenceEventType.GENERATED_TOKEN,
+ DynamicInferenceEventType.FINISH,
+ ]
+
+ # Verify generated_tokens is a direct list field
+ assert basic_request.generated_tokens == [100, 200, 300]
+ assert isinstance(basic_request.generated_tokens, list)
+
+ # Serialize request
+ serialized = basic_request.serialize()
+ assert 'events' in serialized
+ assert len(serialized['events']) == 6
+ assert serialized['generated_tokens'] == [100, 200, 300]
+
+ # Verify GENERATED_TOKEN events serialize with {"token_id": ...} payload
+ token_events = [e for e in serialized['events'] if e['type'] == 'GENERATED_TOKEN']
+ assert len(token_events) == 3
+ assert token_events[0]['payload'] == {"token_id": 100}
+ assert token_events[1]['payload'] == {"token_id": 200}
+ assert token_events[2]['payload'] == {"token_id": 300}
+
+ # Deserialize and verify restoration
+ restored = DynamicInferenceRequest.deserialize(serialized)
+ assert len(restored.events) == 6
+ assert restored.generated_tokens == [100, 200, 300]
+
+ # Verify event order preserved after roundtrip
+ restored_types = [e.type for e in restored.events]
+ assert restored_types == event_types
+
+ # Verify timestamps preserved
+ for orig, rest in zip(basic_request.events, restored.events):
+ assert rest.timestamp == orig.timestamp
+
+
+# ============================================================================
+# Test 3: Error Event Serialization with ContextErrorFactory
+# ============================================================================
+
+
+def test_error_event_serialization_with_context_error_factory():
+ """Test error event serialization using ContextErrorFactory for all error types.
+
+ Coverage:
+ - ERROR_TRANSIENT and ERROR_NONTRANSIENT events
+ - All 4 error types: RequestOverflowError, TokenOverflowError,
+ MaxSequenceLengthOverflowError, BlockOverflowError
+ - Error attribute preservation (request_id, message, is_transient) through roundtrip
+ """
+ error_cases = [
+ # (event_type, error_class, request_id, message, is_transient)
+ (
+ DynamicInferenceEventType.ERROR_TRANSIENT,
+ RequestOverflowError,
+ 1,
+ "Max requests exceeded",
+ True,
+ ),
+ (
+ DynamicInferenceEventType.ERROR_TRANSIENT,
+ TokenOverflowError,
+ 2,
+ "Token limit exceeded",
+ True,
+ ),
+ (DynamicInferenceEventType.ERROR_TRANSIENT, BlockOverflowError, 3, "Block overflow", True),
+ (
+ DynamicInferenceEventType.ERROR_NONTRANSIENT,
+ MaxSequenceLengthOverflowError,
+ 4,
+ "Sequence too long",
+ False,
+ ),
+ ]
+
+ for event_type, error_class, req_id, message, is_transient in error_cases:
+ # Create error and event
+ error = error_class(request_id=req_id, message=message)
+ event = DynamicInferenceEvent(type=event_type, payload=error)
+
+ # Verify error attributes before serialization
+ assert event.payload.request_id == req_id
+ assert event.payload.message == message
+ assert event.payload.is_transient == is_transient
+
+ # Serialize and deserialize
+ serialized = event.serialize()
+ assert serialized['type'] == event_type.name
+ assert serialized['payload'] is not None
+
+ deserialized = DynamicInferenceEvent.deserialize(serialized)
+
+ # Verify restoration
+ assert deserialized.type == event_type
+ assert deserialized.payload.request_id == req_id
+ assert deserialized.payload.message == message
+ assert deserialized.payload.is_transient == is_transient
+ assert deserialized.timestamp == event.timestamp
+
+
+# ============================================================================
+# Test 4: Generated Tokens Property Filters and Preserves Order
+# ============================================================================
+
+
+def test_generated_tokens_is_direct_list_field(basic_request):
+ """Test generated_tokens is a direct List[int] field, independent of events.
+
+ Coverage:
+ - generated_tokens starts as empty list
+ - Direct append works independently of events
+ - Token ID edge cases: 0, large values (2^62)
+ - Field is serialized/deserialized correctly
+ """
+ # Start with empty list
+ assert basic_request.generated_tokens == []
+ assert isinstance(basic_request.generated_tokens, list)
+
+ # Direct append works
+ basic_request.generated_tokens.append(0)
+ assert basic_request.generated_tokens == [0]
+
+ basic_request.generated_tokens.append(100)
+ assert basic_request.generated_tokens == [0, 100]
+
+ # Large token ID
+ large_id = 2**62
+ basic_request.generated_tokens.append(large_id)
+ assert basic_request.generated_tokens == [0, 100, large_id]
+
+ basic_request.generated_tokens.append(200)
+ assert basic_request.generated_tokens == [0, 100, large_id, 200]
+
+ # Events are independent - add some events
+ basic_request.add_event_add_engine()
+ basic_request.add_event_add_context()
+ basic_request.add_event_finish()
+ assert len(basic_request.events) == 3
+ assert len(basic_request.generated_tokens) == 4
+
+ # Serialization roundtrip preserves generated_tokens
+ serialized = basic_request.serialize()
+ assert serialized['generated_tokens'] == [0, 100, large_id, 200]
+ restored = DynamicInferenceRequest.deserialize(serialized)
+ assert restored.generated_tokens == [0, 100, large_id, 200]
+
+
+# ============================================================================
+# Test 5: Request Record Merge Across Eviction Recovery
+# ============================================================================
+
+
+def test_request_record_merge_across_eviction_recovery():
+ """Test RequestRecord.merge() combines events from multiple checkpoints across eviction/recovery.
+
+ Coverage:
+ - RequestRecord creation and merge()
+ - Eviction lifecycle simulation (first attempt evicted, second attempt succeeds)
+ - Multi-checkpoint event combination
+ - Token aggregation from all checkpoints
+ """
+ # First request: starts, generates some tokens, then gets evicted
+ req1 = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=torch.tensor([1, 2, 3], dtype=torch.int64),
+ sampling_params=SamplingParams(num_tokens_to_generate=10),
+ )
+ req1.add_event_add_engine()
+ req1.add_event_add_context()
+ req1.generated_tokens.extend([100, 101])
+ req1.add_event_generated_token(100)
+ req1.add_event_generated_token(101)
+ req1.add_event_evict()
+
+ # Second request: recovered, generates more tokens, completes
+ req2 = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=torch.tensor([1, 2, 3, 100, 101], dtype=torch.int64),
+ sampling_params=SamplingParams(num_tokens_to_generate=8),
+ )
+ req2.add_event_add_engine()
+ req2.add_event_add_context()
+ req2.generated_tokens.extend([200, 201])
+ req2.add_event_generated_token(200)
+ req2.add_event_generated_token(201)
+ req2.add_event_finish()
+
+ # Create record with both checkpoints
+ record = DynamicInferenceRequestRecord()
+ record.requests.append(req1)
+ record.requests.append(req2)
+
+ # Merge
+ merged = record.merge()
+
+ # Verify all events present (5 from req1 + 5 from req2)
+ assert len(merged.events) == 10
+
+ # Verify event types include eviction
+ event_types = [e.type for e in merged.events]
+ assert DynamicInferenceEventType.EVICT in event_types
+ assert event_types.count(DynamicInferenceEventType.ADD_ENGINE) == 2
+ assert event_types.count(DynamicInferenceEventType.ADD_CONTEXT) == 2
+
+ # Verify all tokens combined correctly
+ assert merged.generated_tokens == [100, 101, 200, 201]
+
+ # Test from_request convenience method
+ single_record = DynamicInferenceRequestRecord.from_request(req1)
+ single_merged = single_record.merge()
+ assert len(single_merged.events) == len(req1.events)
+ assert single_merged.generated_tokens == req1.generated_tokens
+
+
+# ============================================================================
+# Test 5b: Merge does not recompute precomputed_block_hashes
+# ============================================================================
+
+
+def test_merge_preserves_precomputed_block_hashes():
+ """Test that merge() carries over precomputed_block_hashes and does not recompute them."""
+ fake_hashes = [111, 222, 333]
+
+ with patch.object(DynamicInferenceRequest, '_compute_block_hashes') as mock_compute:
+ req = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=torch.tensor([1, 2, 3, 4, 5, 6, 7, 8], dtype=torch.int64),
+ sampling_params=SamplingParams(num_tokens_to_generate=4),
+ enable_prefix_caching=True,
+ block_size_tokens=4,
+ precomputed_block_hashes=fake_hashes,
+ )
+
+ # __post_init__ should have skipped recomputation since hashes were provided.
+ mock_compute.assert_not_called()
+
+ req.generated_tokens.extend([100, 101])
+ req.add_event_finish()
+
+ record = DynamicInferenceRequestRecord.from_request(req)
+ merged = record.merge()
+
+ # Merge should carry hashes through and also skip recomputation.
+ mock_compute.assert_not_called()
+
+
+# ============================================================================
+# Test 6: TTFT Calculation from Event Timestamps
+# ============================================================================
+
+
+def test_ttft_calculation_from_event_timestamps(basic_request):
+ """Test TTFT (time-to-first-token) calculation using ADD_ENGINE and GENERATED_TOKEN timestamps.
+
+ Coverage:
+ - Timestamp accuracy
+ - ADD_ENGINE/ADD_CONTEXT split for TTFT calculation
+ - Timestamps are monotonically increasing
+ - Explicit timestamp preservation through serialization
+ """
+ # Add events with deliberate delays
+ basic_request.add_event_add_engine()
+ time.sleep(0.01)
+ basic_request.add_event_add_context()
+ time.sleep(0.01)
+ basic_request.add_event_generated_token(100)
+
+ # Find timestamps
+ add_engine_time = None
+ first_token_time = None
+ for event in basic_request.events:
+ if event.type == DynamicInferenceEventType.ADD_ENGINE:
+ add_engine_time = event.timestamp
+ elif event.type == DynamicInferenceEventType.GENERATED_TOKEN and first_token_time is None:
+ first_token_time = event.timestamp
+
+ assert add_engine_time is not None
+ assert first_token_time is not None
+
+ # Calculate TTFT
+ ttft = first_token_time - add_engine_time
+ assert ttft > 0, "TTFT should be positive"
+ assert ttft > 0.01, "TTFT should include deliberate delay"
+ assert ttft < 1.0, "TTFT should be reasonable in test"
+
+ # Verify timestamps are monotonically increasing
+ timestamps = [e.timestamp for e in basic_request.events]
+ for i in range(1, len(timestamps)):
+ assert timestamps[i] >= timestamps[i - 1], "Timestamps should not decrease"
+
+ # Test explicit timestamp preservation through serialization
+ explicit_time = 1700000000.123456
+ event_with_explicit = DynamicInferenceEvent(
+ type=DynamicInferenceEventType.FINISH, timestamp=explicit_time
+ )
+ serialized = event_with_explicit.serialize()
+ deserialized = DynamicInferenceEvent.deserialize(serialized)
+ assert deserialized.timestamp == explicit_time
+
+
+# ============================================================================
+# Test 7: Failed Request Lifecycle with Error Propagation
+# ============================================================================
+
+
+def test_failed_request_lifecycle_with_error_propagation(basic_request):
+ """Test failed request lifecycle: transient errors followed by nontransient error and FAIL.
+
+ Coverage:
+ - FAIL event
+ - Error event -> failure path
+ - Multiple transient errors followed by nontransient
+ - Error serialization preserved in full request roundtrip
+ """
+ # Build failure lifecycle
+ basic_request.add_event_add_engine()
+
+ # Transient errors (retryable)
+ transient1 = TokenOverflowError(request_id=1, message="Token overflow 1")
+ basic_request.add_event_error_transient(transient1)
+
+ transient2 = BlockOverflowError(request_id=1, message="Block overflow")
+ basic_request.add_event_error_transient(transient2)
+
+ # Fatal error
+ fatal = MaxSequenceLengthOverflowError(request_id=1, message="Sequence too long")
+ basic_request.add_event_error_nontransient(fatal)
+
+ # Request fails
+ basic_request.add_event_fail()
+
+ # Verify event sequence
+ event_types = [e.type for e in basic_request.events]
+ assert event_types == [
+ DynamicInferenceEventType.ADD_ENGINE,
+ DynamicInferenceEventType.ERROR_TRANSIENT,
+ DynamicInferenceEventType.ERROR_TRANSIENT,
+ DynamicInferenceEventType.ERROR_NONTRANSIENT,
+ DynamicInferenceEventType.FAIL,
+ ]
+
+ # Serialize entire request
+ serialized = basic_request.serialize()
+ assert len(serialized['events']) == 5
+
+ # Deserialize
+ restored = DynamicInferenceRequest.deserialize(serialized)
+ assert len(restored.events) == 5
+
+ # Verify error payloads preserved
+ error_events = [e for e in restored.events if 'ERROR' in e.type.name]
+ assert len(error_events) == 3
+
+ # Check first transient error
+ assert error_events[0].payload.message == "Token overflow 1"
+ assert error_events[0].payload.is_transient is True
+
+ # Check second transient error
+ assert error_events[1].payload.message == "Block overflow"
+ assert error_events[1].payload.is_transient is True
+
+ # Check nontransient error
+ assert error_events[2].payload.message == "Sequence too long"
+ assert error_events[2].payload.is_transient is False
+
+
+# ============================================================================
+# Test 8: Event String Representation for All Types
+# ============================================================================
+
+
+def test_event_str_representation_for_all_types():
+ """Test __str__ method produces correct format for all event types.
+
+ Coverage:
+ - Simple events show type and timestamp
+ - GENERATED_TOKEN shows token=
+ - Error events show error class name
+ - Format consistency across types
+ """
+ # Simple events (no payload)
+ simple_types = [
+ DynamicInferenceEventType.ADD_ENGINE,
+ DynamicInferenceEventType.ADD_CONTEXT,
+ DynamicInferenceEventType.PAUSE,
+ DynamicInferenceEventType.EVICT,
+ DynamicInferenceEventType.FINISH,
+ DynamicInferenceEventType.FAIL,
+ ]
+ for event_type in simple_types:
+ event = DynamicInferenceEvent(type=event_type)
+ str_repr = str(event)
+ assert event_type.name in str_repr, f"{event_type.name} not in {str_repr}"
+ # Timestamp should appear (at least first few digits)
+ assert str(event.timestamp)[:5] in str_repr
+
+ # GENERATED_TOKEN shows token=
+ token_event = DynamicInferenceEvent(
+ type=DynamicInferenceEventType.GENERATED_TOKEN, payload={"token_id": 42}
+ )
+ str_repr = str(token_event)
+ assert 'GENERATED_TOKEN' in str_repr
+ assert 'token=42' in str_repr
+
+ # Large token ID
+ large_token_event = DynamicInferenceEvent(
+ type=DynamicInferenceEventType.GENERATED_TOKEN, payload={"token_id": 9999999}
+ )
+ assert 'token=9999999' in str(large_token_event)
+
+ # Error events show error class name
+ error_cases = [
+ (DynamicInferenceEventType.ERROR_TRANSIENT, RequestOverflowError, "RequestOverflowError"),
+ (DynamicInferenceEventType.ERROR_TRANSIENT, TokenOverflowError, "TokenOverflowError"),
+ (DynamicInferenceEventType.ERROR_TRANSIENT, BlockOverflowError, "BlockOverflowError"),
+ (
+ DynamicInferenceEventType.ERROR_NONTRANSIENT,
+ MaxSequenceLengthOverflowError,
+ "MaxSequenceLengthOverflowError",
+ ),
+ ]
+ for event_type, error_class, expected_name in error_cases:
+ error = error_class(request_id=1, message="Test error")
+ event = DynamicInferenceEvent(type=event_type, payload=error)
+ str_repr = str(event)
+ assert event_type.name in str_repr
+ assert expected_name in str_repr
+
+
+# ============================================================================
+# Test 9: Complex Multi-Pause-Evict Lifecycle with Record
+# ============================================================================
+
+
+def test_complex_multi_pause_evict_lifecycle_with_record():
+ """Test complex real-world scenario with multiple pauses, evictions, and recoveries.
+
+ Coverage:
+ - Complex lifecycle with multiple PAUSE events
+ - Multiple evictions and recoveries
+ - Full roundtrip with RequestRecord
+ - Complete event history and all tokens present
+ """
+ # First attempt: starts, generates token, pauses, generates more, gets evicted
+ req1 = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=torch.tensor([1, 2, 3], dtype=torch.int64),
+ sampling_params=SamplingParams(num_tokens_to_generate=20),
+ )
+ req1.add_event_add_engine()
+ req1.add_event_add_context()
+ req1.generated_tokens.append(10)
+ req1.add_event_generated_token(10)
+ req1.add_event_pause() # Paused for higher priority request
+ req1.generated_tokens.append(11)
+ req1.add_event_generated_token(11)
+ req1.generated_tokens.append(12)
+ req1.add_event_generated_token(12)
+ req1.add_event_evict() # Memory pressure
+
+ # Second attempt: recovered, generates tokens, evicted again
+ req2 = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=torch.tensor([1, 2, 3, 10, 11, 12], dtype=torch.int64),
+ sampling_params=SamplingParams(num_tokens_to_generate=17),
+ )
+ req2.add_event_add_engine()
+ req2.add_event_add_context()
+ req2.generated_tokens.append(20)
+ req2.add_event_generated_token(20)
+ req2.add_event_pause()
+ req2.add_event_pause() # Paused twice
+ req2.generated_tokens.append(21)
+ req2.add_event_generated_token(21)
+ req2.add_event_evict()
+
+ # Third attempt: finally completes
+ req3 = DynamicInferenceRequest(
+ request_id=1,
+ prompt_tokens=torch.tensor([1, 2, 3, 10, 11, 12, 20, 21], dtype=torch.int64),
+ sampling_params=SamplingParams(num_tokens_to_generate=15),
+ )
+ req3.add_event_add_engine()
+ req3.add_event_add_context()
+ req3.generated_tokens.extend([30, 31, 32])
+ req3.add_event_generated_token(30)
+ req3.add_event_generated_token(31)
+ req3.add_event_generated_token(32)
+ req3.add_event_finish()
+
+ # Create record with all checkpoints
+ record = DynamicInferenceRequestRecord()
+ record.requests.append(req1)
+ record.requests.append(req2)
+ record.requests.append(req3)
+
+ # Serialize and deserialize record
+ serialized = record.serialize()
+ assert len(serialized['requests']) == 3
+
+ # Merge all checkpoints
+ merged = record.merge()
+
+ # Verify all tokens collected
+ assert merged.generated_tokens == [10, 11, 12, 20, 21, 30, 31, 32]
+
+ # Count event types in merged history
+ event_types = [e.type for e in merged.events]
+ assert event_types.count(DynamicInferenceEventType.ADD_ENGINE) == 3
+ assert event_types.count(DynamicInferenceEventType.ADD_CONTEXT) == 3
+ assert event_types.count(DynamicInferenceEventType.EVICT) == 2
+ assert event_types.count(DynamicInferenceEventType.PAUSE) == 3
+ assert event_types.count(DynamicInferenceEventType.FINISH) == 1
+ assert event_types.count(DynamicInferenceEventType.GENERATED_TOKEN) == 8
+
+ # Total events: 7 from req1 + 7 from req2 + 6 from req3 = 20
+ assert len(merged.events) == 20
+
+ # Test serialization roundtrip of entire record
+ serialized = record.serialize()
+ assert len(serialized['requests']) == 3
+
+ # Verify all events are dicts (no compact mode)
+ first_request_events = serialized['requests'][0]['events']
+ assert all(isinstance(e, dict) for e in first_request_events)
+
+ # Verify GENERATED_TOKEN events use {"token_id": ...} payload format
+ token_events = [e for e in first_request_events if e['type'] == 'GENERATED_TOKEN']
+ assert token_events[0]['payload'] == {"token_id": 10}
+ assert token_events[1]['payload'] == {"token_id": 11}
+ assert token_events[2]['payload'] == {"token_id": 12}
diff --git a/tests/unit_tests/inference/engines/test_mamba_prefix_caching_e2e.py b/tests/unit_tests/inference/engines/test_mamba_prefix_caching_e2e.py
new file mode 100644
index 00000000000..ce21c775b73
--- /dev/null
+++ b/tests/unit_tests/inference/engines/test_mamba_prefix_caching_e2e.py
@@ -0,0 +1,620 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""End-to-end test for Mamba prefix caching with a real hybrid model.
+
+This test exercises the 4 key indices within a Mamba prefill:
+
+ 1. num_mamba_matched — how many blocks have cached Mamba state.
+ Determines how many tokens the prefill can skip.
+
+ 2. num_kv_matched — how many KV blocks are shared with prior
+ requests. Can exceed num_mamba_matched, since KV blocks are
+ always registered for every completed block, while Mamba
+ state is only cached at divergence and last-aligned blocks.
+
+ 3. last_aligned_block — the last full-block boundary in the
+ prompt: floor(prompt_len / block_size) * block_size. Mamba
+ state is always cached here (if it falls within the
+ effective prefill). This is the "end of the known prefix"
+ state that future requests can restore from.
+
+ 4. end_of_sequence — the actual prompt length. When prompt_len
+ is block-aligned (prompt_len == last_aligned), the final
+ Mamba state is cached via the EOS path (copy from live
+ buffer). When not aligned, there's a gap between
+ last_aligned and end_of_sequence that doesn't get cached.
+
+5 requests with overlapping prefixes are processed in a specific
+order so that each request sees a different combination of these
+indices. The test verifies both internal state (mamba cache
+registration, skip counts) and output correctness (generated
+tokens match between pc=off and pc=on).
+"""
+
+import os
+import random
+import types
+
+import pytest
+import torch
+
+from megatron.core import parallel_state
+from megatron.core.inference.config import (
+ InferenceConfig,
+ MambaInferenceStateConfig,
+ PrefixCachingEvictionPolicy,
+)
+from megatron.core.inference.contexts.dynamic_context import DynamicInferenceContext
+from megatron.core.inference.engines import DynamicInferenceEngine
+from megatron.core.inference.inference_request import DynamicInferenceRequest
+from megatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapper import (
+ GPTInferenceWrapper,
+)
+from megatron.core.inference.sampling_params import SamplingParams
+from megatron.core.inference.text_generation_controllers.text_generation_controller import (
+ TextGenerationController,
+)
+from megatron.core.models.mamba.mamba_layer_specs import mamba_stack_spec
+from megatron.core.models.mamba.mamba_model import MambaModel
+from megatron.core.ssm.mamba_mixer import _check_mamba_sequence_packing_support
+from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
+from megatron.core.transformer.cuda_graphs import CudaGraphManager, _CudagraphGlobalRecord
+from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.utils import is_fa_min_version
+from tests.unit_tests.test_utilities import Utils
+
+BLOCK_SIZE = 256
+VOCAB_SIZE = 10000
+MAX_SEQ_LEN = 2048
+NUM_TOKENS_TO_GENERATE = 16
+# multi-group uses 4x the requests (20 vs 5), creating larger batch
+# composition differences between pc=off and pc=on. reduce decode steps
+# to stay within the safe bf16 rounding margin.
+MULTI_GROUP_TOKENS_TO_GENERATE = 8
+NUM_GROUPS = 4
+GROUP_TOKEN_STRIDE = 2000
+
+
+def skip_if_mamba_sequence_packing_not_available():
+ sequence_packing_available, reason = _check_mamba_sequence_packing_support()
+ if not sequence_packing_available:
+ pytest.skip(reason)
+
+
+def set_rounder(value):
+ DynamicInferenceContext.ROUNDER = value
+ DynamicInferenceContext.TOKEN_ROUNDER = value
+ DynamicInferenceContext.REQUEST_ROUNDER = value
+
+
+@pytest.mark.internal
+@pytest.mark.skipif(not is_fa_min_version("2.7.3"), reason="need flash attn")
+class TestMambaPrefixCachingE2E:
+ """End-to-end test for Mamba prefix caching with a real hybrid model."""
+
+ @classmethod
+ def setup_class(cls):
+ Utils.initialize_model_parallel()
+ random.seed(123)
+ torch.manual_seed(123)
+ model_parallel_cuda_manual_seed(
+ seed=123, inference_rng_tracker=True, use_cudagraphable_rng=False, force_reset_rng=True
+ )
+
+ def setup_method(self, method):
+ os.environ.pop('NVTE_FLASH_ATTN', None)
+ os.environ.pop('NVTE_FUSED_ATTN', None)
+ os.environ.pop('NVTE_UNFUSED_ATTN', None)
+ random.seed(123)
+ torch.manual_seed(123)
+ model_parallel_cuda_manual_seed(
+ seed=123, inference_rng_tracker=True, use_cudagraphable_rng=False, force_reset_rng=True
+ )
+
+ @classmethod
+ def teardown_class(cls):
+ Utils.destroy_model_parallel()
+
+ def _create_model(self, num_cuda_graphs=None):
+ transformer_config = TransformerConfig(
+ params_dtype=torch.bfloat16,
+ num_layers=3,
+ hidden_size=256,
+ mamba_num_heads=16,
+ num_attention_heads=16,
+ use_cpu_initialization=True,
+ cuda_graph_impl="local" if num_cuda_graphs else "none",
+ inference_rng_tracker=True,
+ tensor_model_parallel_size=1,
+ pipeline_model_parallel_size=1,
+ pipeline_dtype=torch.bfloat16,
+ add_bias_linear=True,
+ is_hybrid_model=True,
+ )
+ model = MambaModel(
+ config=transformer_config,
+ mamba_stack_spec=mamba_stack_spec,
+ vocab_size=VOCAB_SIZE,
+ max_sequence_length=MAX_SEQ_LEN,
+ parallel_output=True,
+ hybrid_layer_pattern="M*-",
+ pre_process=parallel_state.is_pipeline_first_stage(),
+ post_process=parallel_state.is_pipeline_last_stage(),
+ ).cuda()
+ for param in model.parameters():
+ param.data = param.data.to(transformer_config.params_dtype)
+ model.eval()
+ return model
+
+ def _create_prompts(self, offset=0):
+ """Build 5 prompts with carefully designed prefix sharing.
+
+ Each prompt uses disjoint token ID ranges for unique segments
+ so that parent-chained block hashes differ where content differs.
+
+ The prompts are designed so that each request hits a different
+ combination of the 4 key indices:
+
+ req 0 (300 tokens): seed request, no matches. last_aligned=256
+ req 1 (800 tokens): 1 KV match, 1 Mamba match (block 0 from req 0)
+ req 2 (800 tokens): 2 KV matches, but only 1 Mamba match
+ req 3 (800 tokens): 2 KV matches, 2 Mamba matches
+ req 4 (1100 tokens): 3 KV matches, 3 Mamba matches
+ """
+ device = torch.cuda.current_device()
+ base = torch.arange(offset, offset + 256, dtype=torch.int64, device=device)
+ seg_B = torch.arange(offset + 256, offset + 512, dtype=torch.int64, device=device)
+ seg_1rest = torch.arange(offset + 512, offset + 800, dtype=torch.int64, device=device)
+ seg_2rest = torch.arange(offset + 800, offset + 1088, dtype=torch.int64, device=device)
+ seg_3rest = torch.arange(offset + 1088, offset + 1376, dtype=torch.int64, device=device)
+ seg_4ext = torch.arange(offset + 1376, offset + 1708, dtype=torch.int64, device=device)
+ extra_0 = torch.arange(offset + 1708, offset + 1752, dtype=torch.int64, device=device)
+
+ prompts = [
+ torch.cat([base, extra_0]), # 300
+ torch.cat([base, seg_B, seg_1rest]), # 800
+ torch.cat([base, seg_B, seg_2rest]), # 800
+ torch.cat([base, seg_B, seg_3rest]), # 800
+ torch.cat([base, seg_B, seg_1rest[:256], seg_4ext]), # 1100
+ ]
+ assert [len(p) for p in prompts] == [300, 800, 800, 800, 1100]
+ return prompts
+
+ def _build_engine(
+ self,
+ model,
+ mamba_config,
+ enable_prefix_caching,
+ buffer_size_gb=0.5,
+ prefix_caching_mamba_gb=0.05,
+ request_rounder=4,
+ num_cuda_graphs=None,
+ ):
+ set_rounder(request_rounder)
+ inference_config_kwargs = dict(
+ max_sequence_length=MAX_SEQ_LEN,
+ buffer_size_gb=buffer_size_gb,
+ block_size_tokens=BLOCK_SIZE,
+ mamba_inference_state_config=mamba_config,
+ materialize_only_last_token_logits=False,
+ enable_prefix_caching=enable_prefix_caching,
+ unified_memory_level=0,
+ num_cuda_graphs=num_cuda_graphs,
+ )
+ if enable_prefix_caching:
+ inference_config_kwargs.update(
+ prefix_caching_eviction_policy=PrefixCachingEvictionPolicy.LRU,
+ prefix_caching_mamba_gb=prefix_caching_mamba_gb,
+ )
+ context = DynamicInferenceContext(
+ model_config=model.config, inference_config=InferenceConfig(**inference_config_kwargs)
+ )
+ wrapper = GPTInferenceWrapper(model, context)
+ wrapper.model_is_pipeline_parallel = not (
+ parallel_state.is_pipeline_first_stage() and parallel_state.is_pipeline_last_stage()
+ )
+ controller = TextGenerationController(
+ inference_wrapped_model=wrapper,
+ tokenizer=types.SimpleNamespace(
+ vocab_size=VOCAB_SIZE, detokenize=lambda tokens: "tokenized_prompt"
+ ),
+ )
+ _CudagraphGlobalRecord.cudagraph_created = False
+ _CudagraphGlobalRecord.cudagraph_record = []
+ _CudagraphGlobalRecord.cudagraph_inference_record = []
+ CudaGraphManager.global_mempool = None
+ for module in model.modules():
+ if isinstance(module, CudaGraphManager):
+ module.cudagraph_runners.clear()
+ module.inference_cudagraphs_lookup_table.clear()
+ return DynamicInferenceEngine(controller, context)
+
+ def _make_request(self, req_id, prompt, enable_pc, num_tokens=NUM_TOKENS_TO_GENERATE):
+ return DynamicInferenceRequest(
+ request_id=req_id,
+ prompt_tokens=prompt,
+ sampling_params=SamplingParams(
+ num_tokens_to_generate=num_tokens, termination_id=-1, top_k=1
+ ),
+ block_size_tokens=BLOCK_SIZE if enable_pc else None,
+ enable_prefix_caching=enable_pc,
+ )
+
+ def _run_simple(
+ self,
+ model,
+ mamba_config,
+ prompts,
+ enable_pc,
+ base_req_id=0,
+ num_tokens=NUM_TOKENS_TO_GENERATE,
+ **engine_kwargs,
+ ):
+ """Run all prompts with given pc setting, return (finished_dict, lifetime_prefill)."""
+ engine = self._build_engine(
+ model, mamba_config, enable_prefix_caching=enable_pc, **engine_kwargs
+ )
+ for i, prompt in enumerate(prompts):
+ engine._add_request(self._make_request(base_req_id + i, prompt, enable_pc, num_tokens))
+ finished = {}
+ while engine.has_unfinished_requests():
+ result = engine.step_modern()
+ for record in result["finished_request_records"]:
+ merged = record.merge()
+ finished[merged.request_id] = list(merged.generated_tokens)
+ return finished, engine.context.lifetime_prefill_token_count
+
+ def _get_ref_count(self, alloc, block_hash):
+ bid = alloc.kv_hash_to_block_id.get(block_hash)
+ return 0 if bid is None else alloc.block_ref_counts[bid].item()
+
+ def _assert_step(self, step, reqs_by_group, alloc, step_prefill, num_groups, ctx=None):
+ """Shared per-step verification for single-group and multi-group runs."""
+ G = num_groups
+ if step == 1:
+ for g in range(G):
+ r = reqs_by_group[g]
+ assert r[0]._mamba_num_matched_blocks == 0, f"step 1 group {g}"
+ assert r[0].precomputed_block_hashes[0] in ctx.mamba_slot_allocator.hash_to_block_id
+ assert len(ctx.mamba_slot_allocator.hash_to_block_id) == G
+ assert step_prefill == G * 300, f"step 1: expected {G * 300}, got {step_prefill}"
+ if G == 1:
+ assert (
+ self._get_ref_count(alloc, reqs_by_group[0][0].precomputed_block_hashes[0]) == 1
+ )
+
+ elif step == 2:
+ for g in range(G):
+ r = reqs_by_group[g]
+ assert r[1]._mamba_num_matched_blocks == 1, f"step 2 group {g}"
+ assert r[1].precomputed_block_hashes[2] in ctx.mamba_slot_allocator.hash_to_block_id
+ assert len(ctx.mamba_slot_allocator.hash_to_block_id) == G * 2
+ assert step_prefill == G * 544, f"step 2: expected {G * 544}, got {step_prefill}"
+ if G == 1:
+ assert (
+ self._get_ref_count(alloc, reqs_by_group[0][0].precomputed_block_hashes[0]) == 2
+ )
+
+ elif step == 3:
+ for g in range(G):
+ r = reqs_by_group[g]
+ assert r[2]._mamba_num_matched_blocks == 1, f"step 3 group {g} req 2"
+ assert r[4]._mamba_num_matched_blocks == 3, f"step 3 group {g} req 4"
+ assert r[2].precomputed_block_hashes[1] in ctx.mamba_slot_allocator.hash_to_block_id
+ assert r[2].precomputed_block_hashes[2] in ctx.mamba_slot_allocator.hash_to_block_id
+ assert r[4].precomputed_block_hashes[3] in ctx.mamba_slot_allocator.hash_to_block_id
+ h0 = r[0].precomputed_block_hashes[0]
+ h1 = r[1].precomputed_block_hashes[1]
+ assert self._get_ref_count(alloc, h0) == 4, f"step 3 group {g}"
+ assert self._get_ref_count(alloc, h1) == 3, f"step 3 group {g}"
+ assert len(ctx.mamba_slot_allocator.hash_to_block_id) == G * 5
+ assert step_prefill == G * (
+ 544 + 332
+ ), f"step 3: expected {G * 876}, got {step_prefill}"
+
+ elif step == 4:
+ for g in range(G):
+ r = reqs_by_group[g]
+ assert r[3]._mamba_num_matched_blocks == 2, f"step 4 group {g}"
+ assert r[3].precomputed_block_hashes[2] in ctx.mamba_slot_allocator.hash_to_block_id
+ h0 = r[0].precomputed_block_hashes[0]
+ h1 = r[1].precomputed_block_hashes[1]
+ assert self._get_ref_count(alloc, h0) == 5, f"step 4 group {g}"
+ assert self._get_ref_count(alloc, h1) == 4, f"step 4 group {g}"
+ assert len(ctx.mamba_slot_allocator.hash_to_block_id) == G * 6
+ assert step_prefill == G * 288, f"step 4: expected {G * 288}, got {step_prefill}"
+
+ def _run_pc_on(self, model, mamba_config, prompts):
+ """Run requests with prefix caching enabled, verifying per-step state."""
+ engine = self._build_engine(model, mamba_config, enable_prefix_caching=True)
+ alloc = engine.context.kv_block_allocator
+ ctx = engine.context
+
+ reqs = {i: self._make_request(i, p, True) for i, p in enumerate(prompts)}
+ for i in [0, 1, 2, 4]:
+ engine._add_request(reqs[i])
+
+ step = 0
+ req3_added = False
+ finished = {}
+ prev_prefill = 0
+ reqs_by_group = [{k: reqs[k] for k in reqs}]
+
+ while engine.has_unfinished_requests():
+ result = engine.step_modern()
+ step += 1
+ step_prefill = ctx.lifetime_prefill_token_count - prev_prefill
+ prev_prefill = ctx.lifetime_prefill_token_count
+ for record in result["finished_request_records"]:
+ merged = record.merge()
+ finished[merged.request_id] = list(merged.generated_tokens)
+
+ if step <= 2 or (step == 3 and not req3_added) or (step == 4 and req3_added):
+ self._assert_step(step, reqs_by_group, alloc, step_prefill, 1, ctx)
+ if step == 3 and not req3_added:
+ engine._add_request(reqs[3])
+ req3_added = True
+
+ return finished, ctx.lifetime_prefill_token_count
+
+ def _run_multi_pc_on(self, model, mamba_config, all_prompts, num_cuda_graphs=None):
+ """Run 4 groups with prefix caching enabled, verifying per-step state."""
+ engine = self._build_engine(
+ model,
+ mamba_config,
+ enable_prefix_caching=True,
+ buffer_size_gb=2.0,
+ prefix_caching_mamba_gb=0.2,
+ num_cuda_graphs=num_cuda_graphs,
+ )
+ alloc = engine.context.kv_block_allocator
+ ctx = engine.context
+
+ reqs = []
+ for g, prompts in enumerate(all_prompts):
+ group_reqs = {}
+ for lid, prompt in enumerate(prompts):
+ rid = g * 5 + lid
+ group_reqs[lid] = self._make_request(
+ rid, prompt, True, MULTI_GROUP_TOKENS_TO_GENERATE
+ )
+ reqs.append(group_reqs)
+
+ for g in range(NUM_GROUPS):
+ for lid in [0, 1, 2, 4]:
+ engine._add_request(reqs[g][lid])
+
+ step = 0
+ req3_added = False
+ finished = {}
+ prev_prefill = 0
+
+ while engine.has_unfinished_requests():
+ result = engine.step_modern()
+ step += 1
+ step_prefill = ctx.lifetime_prefill_token_count - prev_prefill
+ prev_prefill = ctx.lifetime_prefill_token_count
+ for record in result["finished_request_records"]:
+ merged = record.merge()
+ finished[merged.request_id] = list(merged.generated_tokens)
+
+ if step <= 2 or (step == 3 and not req3_added) or (step == 4 and req3_added):
+ self._assert_step(step, reqs, alloc, step_prefill, NUM_GROUPS, ctx)
+ if step == 3 and not req3_added:
+ for g in range(NUM_GROUPS):
+ engine._add_request(reqs[g][3])
+ req3_added = True
+
+ return finished, ctx.lifetime_prefill_token_count
+
+ @torch.inference_mode()
+ def test_mamba_prefix_caching_e2e(self):
+ """Verify output tokens match between pc=off and pc=on."""
+ skip_if_mamba_sequence_packing_not_available()
+ model = self._create_model()
+ mamba_config = MambaInferenceStateConfig.from_model(model)
+ prompts = self._create_prompts()
+
+ off_outputs, off_prefill = self._run_simple(model, mamba_config, prompts, False)
+ on_outputs, on_prefill = self._run_pc_on(model, mamba_config, prompts)
+
+ for req_id in range(5):
+ assert (
+ off_outputs[req_id] == on_outputs[req_id]
+ ), f"req {req_id}: pc=off {off_outputs[req_id]} != pc=on {on_outputs[req_id]}"
+ assert off_prefill == 3800 and on_prefill == 2008 and on_prefill < off_prefill
+
+ @pytest.mark.parametrize("num_cuda_graphs", [None, 2])
+ @torch.inference_mode()
+ def test_mamba_prefix_caching_multi_group_e2e(self, num_cuda_graphs):
+ """Verify multi-group prefix caching with 4 independent groups."""
+ skip_if_mamba_sequence_packing_not_available()
+ model = self._create_model(num_cuda_graphs=num_cuda_graphs)
+ mamba_config = MambaInferenceStateConfig.from_model(model)
+ all_prompts = [self._create_prompts(g * GROUP_TOKEN_STRIDE) for g in range(NUM_GROUPS)]
+
+ _, off_prefill = self._run_simple(
+ model,
+ mamba_config,
+ [p for group in all_prompts for p in group],
+ False,
+ num_tokens=MULTI_GROUP_TOKENS_TO_GENERATE,
+ num_cuda_graphs=num_cuda_graphs,
+ buffer_size_gb=2.0,
+ prefix_caching_mamba_gb=0.2,
+ )
+ on_outputs, on_prefill = self._run_multi_pc_on(
+ model, mamba_config, all_prompts, num_cuda_graphs=num_cuda_graphs
+ )
+
+ # verify per-group outputs match independent runs
+ for g in range(NUM_GROUPS):
+ ref_outputs, _ = self._run_simple(
+ model,
+ mamba_config,
+ all_prompts[g],
+ True,
+ base_req_id=g * 5,
+ num_tokens=MULTI_GROUP_TOKENS_TO_GENERATE,
+ num_cuda_graphs=num_cuda_graphs,
+ )
+ for lid in range(5):
+ rid = g * 5 + lid
+ assert (
+ on_outputs[rid] == ref_outputs[rid]
+ ), f"group {g} req {lid}: multi {on_outputs[rid]} != per-group {ref_outputs[rid]}"
+
+ assert off_prefill == NUM_GROUPS * 3800
+ assert on_prefill == NUM_GROUPS * 2008 and on_prefill < off_prefill
+
+ def _create_block_aligned_prompts(self):
+ """Build 4 prompts with block-aligned lengths for EOS path testing."""
+ device = torch.cuda.current_device()
+ seg_0 = torch.arange(8000, 8256, dtype=torch.int64, device=device)
+ seg_1 = torch.arange(8256, 8512, dtype=torch.int64, device=device)
+ prompts = [
+ seg_0.clone(),
+ seg_0.clone(),
+ torch.cat([seg_0, seg_1]),
+ torch.cat([seg_0, seg_1]),
+ ]
+ assert [len(p) for p in prompts] == [256, 256, 512, 512]
+ return prompts
+
+ def _run_eos_pc_on(self, model, mamba_config, prompts):
+ """Run block-aligned prompts with pc=on, per-step assertions.
+
+ Scheduling with pending_block_hashes coordination:
+ - step 1: A scheduled (B, C, D deferred: h0 pending)
+ - step 2: B + C co-scheduled (D deferred: h1 pending from C)
+ - step 3: D scheduled
+ """
+ engine = self._build_engine(model, mamba_config, enable_prefix_caching=True)
+ alloc = engine.context.kv_block_allocator
+ ctx = engine.context
+
+ reqs = {i: self._make_request(i, p, True) for i, p in enumerate(prompts)}
+ for i in range(4):
+ engine._add_request(reqs[i])
+
+ step = 0
+ finished = {}
+ prev_prefill = 0
+
+ while engine.has_unfinished_requests():
+ result = engine.step_modern()
+ step += 1
+ step_prefill = ctx.lifetime_prefill_token_count - prev_prefill
+ prev_prefill = ctx.lifetime_prefill_token_count
+ for record in result["finished_request_records"]:
+ merged = record.merge()
+ finished[merged.request_id] = list(merged.generated_tokens)
+
+ if step == 1:
+ assert reqs[0]._mamba_num_matched_blocks == 0, f"step 1"
+ assert len(ctx.mamba_slot_allocator.hash_to_block_id) == 1
+ assert (
+ reqs[0].precomputed_block_hashes[0] in ctx.mamba_slot_allocator.hash_to_block_id
+ )
+ assert step_prefill == 256
+ elif step == 2:
+ # B: 1 mamba match but raw_skip >= chunk_length, back off to 0 blocks, full recompute (256)
+ # C: 1 mamba match, skip 256, effective 256
+ assert reqs[1]._mamba_num_matched_blocks == 1, f"step 2 B"
+ assert reqs[2]._mamba_num_matched_blocks == 1, f"step 2 C"
+ assert len(ctx.mamba_slot_allocator.hash_to_block_id) == 2
+ assert (
+ reqs[2].precomputed_block_hashes[1] in ctx.mamba_slot_allocator.hash_to_block_id
+ )
+ assert step_prefill == 512 # B=256 (back-off recompute) + C=256
+ elif step == 3:
+ # D: 2 mamba matches, raw_skip >= chunk_length, back off to block 0, skip 256, effective 256
+ assert reqs[3]._mamba_num_matched_blocks == 2, f"step 3 D"
+ assert len(ctx.mamba_slot_allocator.hash_to_block_id) == 2
+ assert step_prefill == 256
+
+ return finished, ctx.lifetime_prefill_token_count
+
+ @torch.inference_mode()
+ def test_mamba_block_aligned_eos_e2e(self):
+ """Verify block-aligned EOS caching and recompute-based back-off."""
+ skip_if_mamba_sequence_packing_not_available()
+ model = self._create_model()
+ mamba_config = MambaInferenceStateConfig.from_model(model)
+ prompts = self._create_block_aligned_prompts()
+
+ off_outputs, off_prefill = self._run_simple(model, mamba_config, prompts, False)
+ on_outputs, on_prefill = self._run_eos_pc_on(model, mamba_config, prompts)
+
+ for req_id in range(4):
+ assert (
+ off_outputs[req_id] == on_outputs[req_id]
+ ), f"req {req_id}: pc=off {off_outputs[req_id]} != pc=on {on_outputs[req_id]}"
+ assert off_prefill == 1536 and on_prefill == 1024 and on_prefill < off_prefill
+
+ def _create_eviction_prompts(self):
+ device = torch.cuda.current_device()
+ return [
+ torch.arange(8000, 8300, dtype=torch.int64, device=device),
+ torch.arange(8300, 8600, dtype=torch.int64, device=device),
+ torch.arange(8000, 8300, dtype=torch.int64, device=device), # identical to E
+ ]
+
+ @torch.inference_mode()
+ def test_mamba_lru_eviction_e2e(self):
+ """Verify KV eviction invalidates mamba state via invalidate_mamba_state_for_block."""
+ skip_if_mamba_sequence_packing_not_available()
+ model = self._create_model()
+ mamba_config = MambaInferenceStateConfig.from_model(model)
+ prompts = self._create_eviction_prompts()
+
+ engine = self._build_engine(
+ model,
+ mamba_config,
+ enable_prefix_caching=True,
+ buffer_size_gb=0.002,
+ prefix_caching_mamba_gb=0.05,
+ request_rounder=1,
+ )
+ alloc = engine.context.kv_block_allocator
+ ctx = engine.context
+
+ assert alloc.total_count == 3, f"expected 3 total blocks, got {alloc.total_count}"
+ assert ctx.max_requests >= 1
+
+ finished = {}
+
+ def _run_one(req_id, prompt):
+ # Use num_tokens_to_generate=2 so the request survives the prefill
+ # step (commit_mamba_intermediate_states runs after update_requests)
+ req = self._make_request(req_id, prompt, True, num_tokens=2)
+ engine._add_request(req)
+ while engine.has_unfinished_requests():
+ result = engine.step_modern()
+ for record in result["finished_request_records"]:
+ merged = record.merge()
+ finished[merged.request_id] = list(merged.generated_tokens)
+ return req
+
+ # E: seed request
+ req_E = _run_one(0, prompts[0])
+ h_E0 = req_E.precomputed_block_hashes[0]
+ assert (
+ h_E0 in ctx.mamba_slot_allocator.hash_to_block_id and h_E0 in alloc.kv_hash_to_block_id
+ )
+ assert len(ctx.mamba_slot_allocator.hash_to_block_id) == 1 and alloc.total_avail == 1
+
+ # F: disjoint prefix, forces eviction of E's cached block
+ req_F = _run_one(1, prompts[1])
+ assert req_F.precomputed_block_hashes[0] in ctx.mamba_slot_allocator.hash_to_block_id
+ assert (
+ h_E0 not in alloc.kv_hash_to_block_id
+ and h_E0 not in ctx.mamba_slot_allocator.hash_to_block_id
+ )
+ assert len(ctx.mamba_slot_allocator.hash_to_block_id) == 1
+
+ # G: identical to E, but E's state was evicted
+ req_G = _run_one(2, prompts[2])
+ assert req_G._mamba_num_matched_blocks == 0
+ assert h_E0 in ctx.mamba_slot_allocator.hash_to_block_id
+ assert finished[0] == finished[2]
diff --git a/tests/unit_tests/inference/engines/test_prefix_caching_cuda_graphs.py b/tests/unit_tests/inference/engines/test_prefix_caching_cuda_graphs.py
new file mode 100644
index 00000000000..52a05f7f80f
--- /dev/null
+++ b/tests/unit_tests/inference/engines/test_prefix_caching_cuda_graphs.py
@@ -0,0 +1,536 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""Parameterized test for prefix caching with CUDA graphs.
+
+Verifies prefix caching correctness and CUDA graph usage across every
+combination of model type (transformer, hybrid) and batch structure
+(prefill, decode, mixed).
+
+For each case, the same prefix-sharing scenario is run twice (with and
+without CUDA graphs) and compared:
+ 1. Generated tokens match exactly (correctness).
+ 2. context.using_cuda_graph_this_step() returned True at expected steps.
+"""
+
+import os
+import random
+import types
+
+import pytest
+import torch
+
+from megatron.core import parallel_state
+from megatron.core.inference.config import (
+ InferenceConfig,
+ MambaInferenceStateConfig,
+ PrefixCachingEvictionPolicy,
+)
+from megatron.core.inference.contexts.dynamic_context import DynamicInferenceContext
+from megatron.core.inference.engines import DynamicInferenceEngine
+from megatron.core.inference.inference_request import DynamicInferenceRequest
+from megatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapper import (
+ GPTInferenceWrapper,
+)
+from megatron.core.inference.sampling_params import SamplingParams
+from megatron.core.inference.text_generation_controllers.text_generation_controller import (
+ TextGenerationController,
+)
+from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_spec
+from megatron.core.models.gpt.gpt_model import GPTModel
+from megatron.core.models.mamba.mamba_layer_specs import mamba_stack_spec
+from megatron.core.models.mamba.mamba_model import MambaModel
+from megatron.core.ssm.mamba_mixer import _check_mamba_sequence_packing_support
+from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
+from megatron.core.transformer.cuda_graphs import CudaGraphManager, _CudagraphGlobalRecord
+from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.utils import is_fa_min_version
+from tests.unit_tests.test_utilities import Utils
+
+BLOCK_SIZE = 256
+VOCAB_SIZE = 10000
+MAX_SEQ_LEN = 2048
+NUM_TOKENS_TO_GENERATE = 8
+
+
+def set_rounder(value):
+ DynamicInferenceContext.ROUNDER = value
+ DynamicInferenceContext.TOKEN_ROUNDER = value
+ DynamicInferenceContext.REQUEST_ROUNDER = value
+
+
+@pytest.mark.internal
+@pytest.mark.skipif(not is_fa_min_version("2.7.3"), reason="need flash attn")
+class TestPrefixCachingCudaGraphs:
+ """Verify prefix caching + CUDA graph interaction across model types and batch structures."""
+
+ def setup_method(self, method):
+ Utils.initialize_model_parallel()
+ random.seed(123)
+ torch.manual_seed(123)
+ model_parallel_cuda_manual_seed(
+ seed=123, inference_rng_tracker=True, use_cudagraphable_rng=False, force_reset_rng=True
+ )
+
+ def teardown_method(self, method):
+ Utils.destroy_model_parallel()
+
+ def _create_model(self, model_type, num_cuda_graphs=None):
+ """Create a model with optional CUDA graph support.
+
+ Returns (model, mamba_config_or_none).
+ """
+ cuda_graph_impl = "local" if num_cuda_graphs else "none"
+
+ if model_type == "transformer":
+ config = TransformerConfig(
+ params_dtype=torch.bfloat16,
+ num_layers=4,
+ hidden_size=32,
+ num_attention_heads=4,
+ use_cpu_initialization=True,
+ cuda_graph_impl=cuda_graph_impl,
+ inference_rng_tracker=True,
+ tensor_model_parallel_size=1,
+ pipeline_model_parallel_size=1,
+ pipeline_dtype=torch.bfloat16,
+ add_bias_linear=True,
+ )
+ model = GPTModel(
+ config=config,
+ transformer_layer_spec=get_gpt_layer_local_spec(),
+ vocab_size=VOCAB_SIZE,
+ max_sequence_length=MAX_SEQ_LEN,
+ parallel_output=True,
+ pre_process=parallel_state.is_pipeline_first_stage(),
+ post_process=parallel_state.is_pipeline_last_stage(),
+ ).cuda()
+ mamba_config = None
+ else: # hybrid
+ config = TransformerConfig(
+ params_dtype=torch.bfloat16,
+ num_layers=3,
+ hidden_size=256,
+ mamba_num_heads=16,
+ num_attention_heads=16,
+ use_cpu_initialization=True,
+ cuda_graph_impl=cuda_graph_impl,
+ inference_rng_tracker=True,
+ tensor_model_parallel_size=1,
+ pipeline_model_parallel_size=1,
+ pipeline_dtype=torch.bfloat16,
+ add_bias_linear=True,
+ is_hybrid_model=True,
+ )
+ model = MambaModel(
+ config=config,
+ mamba_stack_spec=mamba_stack_spec,
+ vocab_size=VOCAB_SIZE,
+ max_sequence_length=MAX_SEQ_LEN,
+ parallel_output=True,
+ hybrid_layer_pattern="M*-",
+ pre_process=parallel_state.is_pipeline_first_stage(),
+ post_process=parallel_state.is_pipeline_last_stage(),
+ ).cuda()
+ mamba_config = MambaInferenceStateConfig.from_model(model)
+
+ for param in model.parameters():
+ param.data = param.data.to(config.params_dtype)
+ model.eval()
+ return model, mamba_config
+
+ def _reset_cuda_graph_state(self, model):
+ """Reset all CUDA graph global and per-module state."""
+ _CudagraphGlobalRecord.cudagraph_created = False
+ _CudagraphGlobalRecord.cudagraph_record = []
+ _CudagraphGlobalRecord.cudagraph_inference_record = []
+ CudaGraphManager.global_mempool = None
+ for module in model.modules():
+ if isinstance(module, CudaGraphManager):
+ module.cudagraph_runners.clear()
+ module.inference_cudagraphs_lookup_table.clear()
+
+ def _build_engine(self, model, mamba_config, num_cuda_graphs):
+ """Build an engine with prefix caching and optional CUDA graphs."""
+ set_rounder(4)
+ inference_config_kwargs = dict(
+ max_sequence_length=MAX_SEQ_LEN,
+ buffer_size_gb=0.5,
+ block_size_tokens=BLOCK_SIZE,
+ materialize_only_last_token_logits=False,
+ enable_prefix_caching=True,
+ prefix_caching_eviction_policy=PrefixCachingEvictionPolicy.LRU,
+ unified_memory_level=0,
+ num_cuda_graphs=num_cuda_graphs,
+ use_cuda_graphs_for_non_decode_steps=True,
+ )
+ if mamba_config is not None:
+ inference_config_kwargs.update(
+ mamba_inference_state_config=mamba_config, prefix_caching_mamba_gb=0.05
+ )
+ context = DynamicInferenceContext(
+ model_config=model.config, inference_config=InferenceConfig(**inference_config_kwargs)
+ )
+ wrapper = GPTInferenceWrapper(model, context)
+ wrapper.model_is_pipeline_parallel = not (
+ parallel_state.is_pipeline_first_stage() and parallel_state.is_pipeline_last_stage()
+ )
+ controller = TextGenerationController(
+ inference_wrapped_model=wrapper,
+ tokenizer=types.SimpleNamespace(
+ vocab_size=VOCAB_SIZE, detokenize=lambda tokens: "tokenized_prompt"
+ ),
+ )
+ self._reset_cuda_graph_state(model)
+ return DynamicInferenceEngine(controller, context)
+
+ def _create_prompts(self):
+ """Build 4 prompts with overlapping prefixes.
+
+ req 0: tokens[0:300] — seed, no prefix match
+ req 1: tokens[0:300] — identical to req 0, full prefix match (1 block)
+ req 2: tokens[0:256] + unique[256:500] — partial match, 1 block
+ req 3: tokens[0:256] + unique[256:600] — partial match, 1 block shared + 1 unique
+ """
+ device = torch.cuda.current_device()
+ base = torch.arange(0, 256, dtype=torch.int64, device=device)
+ extra = torch.arange(256, 300, dtype=torch.int64, device=device)
+ unique_2 = torch.arange(1000, 1244, dtype=torch.int64, device=device)
+ unique_3 = torch.arange(2000, 2344, dtype=torch.int64, device=device)
+ return [
+ torch.cat([base, extra]), # req 0: 300
+ torch.cat([base, extra]), # req 1: 300
+ torch.cat([base, unique_2]), # req 2: 500
+ torch.cat([base, unique_3]), # req 3: 600
+ ]
+
+ def _make_request(self, req_id, prompt):
+ return DynamicInferenceRequest(
+ request_id=req_id,
+ prompt_tokens=prompt,
+ sampling_params=SamplingParams(
+ num_tokens_to_generate=NUM_TOKENS_TO_GENERATE, termination_id=-1, top_k=1
+ ),
+ block_size_tokens=BLOCK_SIZE,
+ enable_prefix_caching=True,
+ )
+
+ def _run_scenario(self, engine, batch_structure, prompts):
+ """Run the prefix-sharing scenario with the given batch structure.
+
+ Returns (outputs, step_log) where step_log is a list of
+ (prefill_req_count, decode_req_count, using_cuda_graph) per step.
+ """
+ ctx = engine.context
+ finished = {}
+ step_log = []
+
+ def _step_and_log():
+ result = engine.step_modern()
+ step_log.append(
+ (
+ ctx.batch_dimensions.prefill_req_count,
+ ctx.batch_dimensions.decode_req_count,
+ ctx.using_cuda_graph_this_step(),
+ )
+ )
+ for record in result["finished_request_records"]:
+ merged = record.merge()
+ finished[merged.request_id] = list(merged.generated_tokens)
+
+ if batch_structure in ("prefill", "decode"):
+ # Add all 4 requests before first step.
+ for i, prompt in enumerate(prompts):
+ engine._add_request(self._make_request(i, prompt))
+ while engine.has_unfinished_requests():
+ _step_and_log()
+
+ elif batch_structure == "mixed":
+ # Add req 0+1 first, step until both are decoding, then add 2+3.
+ for i in [0, 1]:
+ engine._add_request(self._make_request(i, prompts[i]))
+ reqs_added = False
+ while engine.has_unfinished_requests():
+ _step_and_log()
+ if not reqs_added:
+ last_p, last_d, _ = step_log[-1]
+ if last_p == 0 and last_d > 0:
+ for i in [2, 3]:
+ engine._add_request(self._make_request(i, prompts[i]))
+ reqs_added = True
+
+ return finished, step_log
+
+ @pytest.mark.parametrize("model_type", ["transformer", "hybrid"])
+ @pytest.mark.parametrize("batch_structure", ["prefill", "decode", "mixed"])
+ @torch.inference_mode()
+ def test_prefix_caching_cuda_graphs(self, model_type, batch_structure):
+ """Verify correctness and CUDA graph usage for prefix caching."""
+ if model_type == "hybrid":
+ sequence_packing_available, reason = _check_mamba_sequence_packing_support()
+ if not sequence_packing_available:
+ pytest.skip(reason)
+
+ # Create model with CUDA graph support (cuda_graph_impl="local").
+ model, mamba_config = self._create_model(model_type, num_cuda_graphs=2)
+ prompts = self._create_prompts()
+
+ # Baseline: no CUDA graphs.
+ baseline_engine = self._build_engine(model, mamba_config, num_cuda_graphs=None)
+ baseline_outputs, _ = self._run_scenario(baseline_engine, batch_structure, prompts)
+
+ # CG enabled.
+ cg_engine = self._build_engine(model, mamba_config, num_cuda_graphs=2)
+ cg_outputs, step_log = self._run_scenario(cg_engine, batch_structure, prompts)
+
+ # 1. Correctness: generated tokens must match.
+ for req_id in range(4):
+ assert baseline_outputs[req_id] == cg_outputs[req_id], (
+ f"req {req_id}: baseline {baseline_outputs[req_id]} != " f"cg {cg_outputs[req_id]}"
+ )
+
+ # 2. CUDA graph usage at expected batch types.
+ if batch_structure == "prefill":
+ assert any(
+ p > 0 and d == 0 and cg for p, d, cg in step_log
+ ), f"no prefill-only CG step found in {step_log}"
+
+ elif batch_structure == "decode":
+ decode_only = [(p, d, cg) for p, d, cg in step_log if p == 0 and d > 0]
+ assert decode_only, f"no decode-only steps found in {step_log}"
+ assert all(
+ cg for _, _, cg in decode_only
+ ), f"not all decode-only steps used CG: {decode_only}"
+
+ elif batch_structure == "mixed":
+ assert any(
+ p > 0 and d > 0 and cg for p, d, cg in step_log
+ ), f"no mixed CG step found in {step_log}"
+
+
+@pytest.mark.internal
+@pytest.mark.skipif(not is_fa_min_version("2.7.3"), reason="need flash attn")
+class TestHybridChunkedPrefillIntermediateState:
+ """Verify hybrid chunked prefill with concurrent Mamba state extraction and restoration.
+
+ Scenario: one request is mid-chunk (Mamba intermediate state being extracted during
+ forward pass) while another request has its Mamba state restored from the prefix cache.
+ """
+
+ @classmethod
+ def setup_class(cls):
+ Utils.initialize_model_parallel()
+
+ @classmethod
+ def teardown_class(cls):
+ set_rounder(64)
+ Utils.destroy_model_parallel()
+
+ def _create_hybrid_model(self, num_cuda_graphs=None):
+ """Create a hybrid (Mamba + attention) model."""
+ cuda_graph_impl = "local" if num_cuda_graphs else "none"
+ config = TransformerConfig(
+ params_dtype=torch.bfloat16,
+ num_layers=3,
+ hidden_size=256,
+ mamba_num_heads=16,
+ num_attention_heads=16,
+ use_cpu_initialization=True,
+ cuda_graph_impl=cuda_graph_impl,
+ inference_rng_tracker=True,
+ tensor_model_parallel_size=1,
+ pipeline_model_parallel_size=1,
+ pipeline_dtype=torch.bfloat16,
+ add_bias_linear=True,
+ is_hybrid_model=True,
+ )
+ model = MambaModel(
+ config=config,
+ mamba_stack_spec=mamba_stack_spec,
+ vocab_size=VOCAB_SIZE,
+ max_sequence_length=MAX_SEQ_LEN,
+ parallel_output=True,
+ hybrid_layer_pattern="M*-",
+ pre_process=parallel_state.is_pipeline_first_stage(),
+ post_process=parallel_state.is_pipeline_last_stage(),
+ ).cuda()
+ for param in model.parameters():
+ param.data = param.data.to(config.params_dtype)
+ model.eval()
+ return model
+
+ def _reset_cuda_graph_state(self, model):
+ """Reset all CUDA graph global and per-module state."""
+ _CudagraphGlobalRecord.cudagraph_created = False
+ _CudagraphGlobalRecord.cudagraph_record = []
+ _CudagraphGlobalRecord.cudagraph_inference_record = []
+ CudaGraphManager.global_mempool = None
+ for module in model.modules():
+ if isinstance(module, CudaGraphManager):
+ module.cudagraph_runners.clear()
+ module.inference_cudagraphs_lookup_table.clear()
+
+ def _build_engine(
+ self,
+ model,
+ mamba_config,
+ enable_prefix_caching,
+ enable_chunked_prefill,
+ max_tokens=None,
+ num_cuda_graphs=None,
+ ):
+ """Build an engine with the given prefix caching / chunked prefill config."""
+ set_rounder(4)
+ inference_config_kwargs = dict(
+ max_sequence_length=MAX_SEQ_LEN,
+ buffer_size_gb=0.5,
+ block_size_tokens=BLOCK_SIZE,
+ mamba_inference_state_config=mamba_config,
+ materialize_only_last_token_logits=False,
+ unified_memory_level=0,
+ num_cuda_graphs=num_cuda_graphs,
+ use_cuda_graphs_for_non_decode_steps=True,
+ enable_prefix_caching=enable_prefix_caching,
+ enable_chunked_prefill=enable_chunked_prefill,
+ max_requests=128,
+ )
+ if enable_prefix_caching:
+ inference_config_kwargs.update(
+ prefix_caching_eviction_policy=PrefixCachingEvictionPolicy.LRU,
+ prefix_caching_mamba_gb=0.05,
+ )
+ if max_tokens is not None:
+ inference_config_kwargs["max_tokens"] = max_tokens
+ context = DynamicInferenceContext(
+ model_config=model.config, inference_config=InferenceConfig(**inference_config_kwargs)
+ )
+ wrapper = GPTInferenceWrapper(model, context)
+ wrapper.model_is_pipeline_parallel = not (
+ parallel_state.is_pipeline_first_stage() and parallel_state.is_pipeline_last_stage()
+ )
+ controller = TextGenerationController(
+ inference_wrapped_model=wrapper,
+ tokenizer=types.SimpleNamespace(
+ vocab_size=VOCAB_SIZE, detokenize=lambda tokens: "tokenized_prompt"
+ ),
+ )
+ self._reset_cuda_graph_state(model)
+ return DynamicInferenceEngine(controller, context)
+
+ def _make_request(self, req_id, prompt, enable_pc):
+ return DynamicInferenceRequest(
+ request_id=req_id,
+ prompt_tokens=prompt,
+ sampling_params=SamplingParams(
+ num_tokens_to_generate=NUM_TOKENS_TO_GENERATE, termination_id=-1, top_k=1
+ ),
+ block_size_tokens=BLOCK_SIZE if enable_pc else None,
+ enable_prefix_caching=enable_pc,
+ )
+
+ @torch.inference_mode()
+ def test_hybrid_chunked_prefill_intermediate_state(self):
+ """Concurrent Mamba state extraction (mid-chunk) and restoration (prefix-cached).
+
+ req0 (300 tokens): seeds the cache with 1 block of Mamba state.
+ req1 (800 tokens): 256 shared prefix, 544 unique. With max_tokens=400 and 1
+ Mamba match (skip 256, effective=544), this request is chunked across steps.
+ req2 (300 tokens): identical to req0. Full prefix match, 1 Mamba match.
+
+ In the critical step, req2 has Mamba state restored from cache while req1 has
+ Mamba state being computed fresh with intermediate state extraction.
+ """
+ sequence_packing_available, reason = _check_mamba_sequence_packing_support()
+ if not sequence_packing_available:
+ pytest.skip(reason)
+
+ # Clear NVTE env vars set by conftest set_env fixture.
+ os.environ.pop('NVTE_FLASH_ATTN', None)
+ os.environ.pop('NVTE_FUSED_ATTN', None)
+ os.environ.pop('NVTE_UNFUSED_ATTN', None)
+
+ random.seed(123)
+ torch.manual_seed(123)
+ model_parallel_cuda_manual_seed(
+ seed=123, inference_rng_tracker=True, use_cudagraphable_rng=False, force_reset_rng=True
+ )
+
+ model = self._create_hybrid_model()
+ mamba_config = MambaInferenceStateConfig.from_model(model)
+
+ device = torch.cuda.current_device()
+ prompt0 = torch.arange(0, 300, dtype=torch.int64, device=device)
+ prompt1 = torch.cat(
+ [
+ torch.arange(0, 256, dtype=torch.int64, device=device),
+ torch.arange(5000, 5544, dtype=torch.int64, device=device),
+ ]
+ ) # 800 tokens: 256 shared + 544 unique
+ prompt2 = torch.arange(0, 300, dtype=torch.int64, device=device) # identical to prompt0
+
+ # Baseline: no prefix caching, no chunked prefill.
+ baseline_engine = self._build_engine(
+ model, mamba_config, enable_prefix_caching=False, enable_chunked_prefill=False
+ )
+ for i, prompt in enumerate([prompt0, prompt1, prompt2]):
+ baseline_engine._add_request(self._make_request(i, prompt, enable_pc=False))
+ baseline_outputs = {}
+ while baseline_engine.has_unfinished_requests():
+ result = baseline_engine.step_modern()
+ for record in result["finished_request_records"]:
+ merged = record.merge()
+ baseline_outputs[merged.request_id] = list(merged.generated_tokens)
+
+ # Test: prefix caching + chunked prefill, max_tokens=400.
+ test_engine = self._build_engine(
+ model,
+ mamba_config,
+ enable_prefix_caching=True,
+ enable_chunked_prefill=True,
+ max_tokens=400,
+ )
+ ctx = test_engine.context
+
+ test_outputs = {}
+
+ def collect_finished(result):
+ for record in result["finished_request_records"]:
+ merged = record.merge()
+ test_outputs[merged.request_id] = list(merged.generated_tokens)
+
+ # Phase 1: run req0 to completion (seeds cache).
+ req0 = self._make_request(0, prompt0, enable_pc=True)
+ test_engine._add_request(req0)
+ while test_engine.has_unfinished_requests():
+ collect_finished(test_engine.step_modern())
+
+ # Verify req0 cached its Mamba state.
+ assert (
+ len(ctx.mamba_slot_allocator.hash_to_block_id) > 0
+ ), "req0 should have cached Mamba state"
+
+ # Phase 2: add req1 + req2 simultaneously.
+ req1 = self._make_request(1, prompt1, enable_pc=True)
+ req2 = self._make_request(2, prompt2, enable_pc=True)
+ test_engine._add_request(req1)
+ test_engine._add_request(req2)
+
+ while test_engine.has_unfinished_requests():
+ collect_finished(test_engine.step_modern())
+
+ # Verify Mamba state was restored for req2 (prefix-cached).
+ assert (
+ req2._mamba_num_matched_blocks == 1
+ ), f"req2 should have 1 Mamba match, got {req2._mamba_num_matched_blocks}"
+
+ # Verify prefix caching saved prefill tokens.
+ assert ctx.lifetime_prefill_token_count < (300 + 800 + 300), (
+ f"prefix caching should reduce total prefill tokens, "
+ f"got {ctx.lifetime_prefill_token_count}"
+ )
+
+ # Correctness: generated tokens must match baseline.
+ for req_id in range(3):
+ assert baseline_outputs[req_id] == test_outputs[req_id], (
+ f"req {req_id}: baseline {baseline_outputs[req_id]} != "
+ f"test {test_outputs[req_id]}"
+ )
diff --git a/tests/unit_tests/inference/engines/test_static_engine.py b/tests/unit_tests/inference/engines/test_static_engine.py
index 03b3712e39a..483a21d13bd 100644
--- a/tests/unit_tests/inference/engines/test_static_engine.py
+++ b/tests/unit_tests/inference/engines/test_static_engine.py
@@ -20,9 +20,6 @@
from megatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapper import (
GPTInferenceWrapper,
)
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
from megatron.core.inference.sampling_params import SamplingParams
from megatron.core.inference.text_generation_controllers.text_generation_controller import (
TextGenerationController,
@@ -85,20 +82,11 @@ def setup_engine(
).cuda()
gpt_model.to(inference_config_params_dtype)
- inference_wrapper_config = InferenceWrapperConfig(
- hidden_size=self.hidden_size,
- inference_batch_times_seqlen_threshold=400,
- inference_max_requests=self.batch_size,
- fp32_residual_connection=False,
- params_dtype=inference_config_params_dtype,
- padded_vocab_size=self.vocab_size,
+ inference_context = StaticInferenceContext(
+ max_batch_size=self.batch_size, max_sequence_length=self.sequence_length
)
- inference_context = StaticInferenceContext.from_config(inference_wrapper_config)
-
- inference_wrapped_model = GPTInferenceWrapper(
- gpt_model, inference_wrapper_config, inference_context
- )
+ inference_wrapped_model = GPTInferenceWrapper(gpt_model, inference_context)
self.mock_tokenizer = mock.Mock()
# Set required tokenizer attributes before engine creation
self.mock_tokenizer.vocab_size = self.vocab_size
@@ -200,8 +188,6 @@ def test_generate_dynamic(self, batch_size: int, num_trials: int, empty_prompt:
assert len(results) == batch_size
for result in results:
- if isinstance(result, DynamicInferenceRequestRecord):
- result = result.merge()
assert isinstance(result, InferenceRequest), (
"expected ; found <%s>." % type(result).__name__
)
diff --git a/tests/unit_tests/inference/model_inference_wrappers/gpt/test_gpt_inference_wrapper.py b/tests/unit_tests/inference/model_inference_wrappers/gpt/test_gpt_inference_wrapper.py
index 07afebe1067..f49a67790e7 100644
--- a/tests/unit_tests/inference/model_inference_wrappers/gpt/test_gpt_inference_wrapper.py
+++ b/tests/unit_tests/inference/model_inference_wrappers/gpt/test_gpt_inference_wrapper.py
@@ -1,7 +1,5 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-from argparse import Namespace
-
import pytest
import torch
@@ -10,13 +8,7 @@
from megatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapper import (
GPTInferenceWrapper,
)
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
-from megatron.core.models.gpt.gpt_layer_specs import (
- get_gpt_layer_local_spec,
- get_gpt_layer_with_transformer_engine_spec,
-)
+from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_spec
from megatron.core.models.gpt.gpt_model import GPTModel
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer.transformer_config import TransformerConfig
@@ -53,27 +45,15 @@ def setup_model(self, tensor_parallel_size, pipeline_parallel_size):
post_process=parallel_state.is_pipeline_last_stage(),
).cuda()
- inference_wrapper_config = InferenceWrapperConfig(
- hidden_size=hidden_size,
- inference_batch_times_seqlen_threshold=20,
- inference_max_requests=self.batch_size,
- fp32_residual_connection=False,
- params_dtype=torch.float,
- padded_vocab_size=self.vocab_size,
- )
-
- inference_context = StaticInferenceContext.from_config(inference_wrapper_config)
+ inference_context = StaticInferenceContext(self.batch_size, self.sequence_length)
- self.inference_wrapped_model = GPTInferenceWrapper(
- gpt_model, inference_wrapper_config, inference_context
- )
+ self.inference_wrapped_model = GPTInferenceWrapper(gpt_model, inference_context)
def teardown_method(self, method):
Utils.destroy_model_parallel()
- # This will call the inference_wrapped_model.forward_pass_with_pipeline_parallel_small_input_batch()
@pytest.mark.parametrize("materialize_only_last_token_logits", [True, False])
- def test_inference_pipeline_parallel_small_size(self, materialize_only_last_token_logits):
+ def test_inference_pipeline_parallel(self, materialize_only_last_token_logits):
self.setup_model(tensor_parallel_size=2, pipeline_parallel_size=2)
batch_prompt_tokens = (
@@ -82,7 +62,7 @@ def test_inference_pipeline_parallel_small_size(self, materialize_only_last_toke
.cuda()
)
self.inference_wrapped_model.prep_model_for_inference()
- self.inference_wrapped_model.inference_context.materialize_only_last_token_logits = (
+ self.inference_wrapped_model.inference_context.config.materialize_only_last_token_logits = (
materialize_only_last_token_logits
)
@@ -107,42 +87,6 @@ def test_inference_pipeline_parallel_small_size(self, materialize_only_last_toke
self.vocab_size,
), f"Shape mismatch . Expected {(self.batch_size, logits_seq_len, self.vocab_size)}, but got {logits.shape}"
- # This will call the inference_wrapped_model.forward_pass_with_pipeline_parallel_large_input_batch()
- @pytest.mark.parametrize("materialize_only_last_token_logits", [True, False])
- def test_inference_pipeline_parallel_large_size(self, materialize_only_last_token_logits):
- self.setup_model(tensor_parallel_size=2, pipeline_parallel_size=2)
-
- batch_prompt_tokens = (
- torch.randint(low=0, high=self.vocab_size, size=(self.batch_size, self.sequence_length))
- .int()
- .cuda()
- )
- self.inference_wrapped_model.prep_model_for_inference()
- self.inference_wrapped_model.inference_context.materialize_only_last_token_logits = (
- materialize_only_last_token_logits
- )
-
- inference_input = self.inference_wrapped_model.prep_inference_input(
- prompts_tokens=batch_prompt_tokens
- )
-
- inference_input_for_context_window = (
- self.inference_wrapped_model.get_batch_for_context_window(inference_input, 0, 10)
- )
-
- logits_seq_len = 1 if materialize_only_last_token_logits else 10
-
- logits = self.inference_wrapped_model.run_one_forward_step(
- inference_input_for_context_window
- )
-
- if parallel_state.is_pipeline_last_stage():
- assert logits.shape == (
- self.batch_size,
- logits_seq_len,
- self.vocab_size,
- ), f"Shape mismatch . Expected {(self.batch_size, logits_seq_len, self.vocab_size)}, but got {logits.shape}"
-
@pytest.mark.parametrize("materialize_only_last_token_logits", [True, False])
def test_inference_only_tensor_parallel(self, materialize_only_last_token_logits):
self.setup_model(tensor_parallel_size=4, pipeline_parallel_size=1)
@@ -153,7 +97,7 @@ def test_inference_only_tensor_parallel(self, materialize_only_last_token_logits
.cuda()
)
self.inference_wrapped_model.prep_model_for_inference()
- self.inference_wrapped_model.inference_context.materialize_only_last_token_logits = (
+ self.inference_wrapped_model.inference_context.config.materialize_only_last_token_logits = (
materialize_only_last_token_logits
)
diff --git a/tests/unit_tests/inference/model_inference_wrappers/t5/test_t5_inference_wrapper.py b/tests/unit_tests/inference/model_inference_wrappers/t5/test_t5_inference_wrapper.py
index 6241330f187..d7841e34533 100755
--- a/tests/unit_tests/inference/model_inference_wrappers/t5/test_t5_inference_wrapper.py
+++ b/tests/unit_tests/inference/model_inference_wrappers/t5/test_t5_inference_wrapper.py
@@ -1,3 +1,5 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
from argparse import Namespace
from copy import deepcopy
from unittest import mock
@@ -8,9 +10,6 @@
from megatron.core import parallel_state
from megatron.core.inference.contexts import StaticInferenceContext
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
from megatron.core.inference.model_inference_wrappers.t5.t5_inference_wrapper import (
T5InferenceWrapper,
)
@@ -78,19 +77,9 @@ def setup_model(self, tensor_parallel_size, pipeline_parallel_size):
add_decoder=True,
).cuda()
- inference_wrapper_config = InferenceWrapperConfig(
- hidden_size=hidden_size,
- inference_batch_times_seqlen_threshold=-1,
- fp32_residual_connection=False,
- params_dtype=torch.float,
- padded_vocab_size=self.vocab_size,
- )
-
- inference_context = StaticInferenceContext.from_config(inference_wrapper_config)
+ inference_context = StaticInferenceContext(max_batch_size=8, max_sequence_length=2560)
- self.inference_wrapped_model = T5InferenceWrapper(
- t5_model, inference_wrapper_config, inference_context
- )
+ self.inference_wrapped_model = T5InferenceWrapper(t5_model, inference_context)
def teardown_method(self, method):
Utils.destroy_model_parallel()
diff --git a/tests/unit_tests/inference/model_inference_wrappers/test_model_inference_wrapper_config.py b/tests/unit_tests/inference/model_inference_wrappers/test_model_inference_wrapper_config.py
deleted file mode 100644
index 794634760d0..00000000000
--- a/tests/unit_tests/inference/model_inference_wrappers/test_model_inference_wrapper_config.py
+++ /dev/null
@@ -1,21 +0,0 @@
-import torch
-
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
-
-
-class TestModelInferenceWrapperConfig:
-
- def test_inference_config(self):
- inference_config = InferenceWrapperConfig(
- hidden_size=10,
- inference_batch_times_seqlen_threshold=10,
- padded_vocab_size=10,
- params_dtype=torch.float,
- fp32_residual_connection=False,
- )
- inference_config.add_attributes({"abc": 45})
- assert (
- inference_config.abc == 45
- ), f"min tokens not set correctly. it is {inference_config.min_tokens}"
diff --git a/tests/unit_tests/inference/test_batch_dimension_utils.py b/tests/unit_tests/inference/test_batch_dimension_utils.py
new file mode 100644
index 00000000000..7918ae8628e
--- /dev/null
+++ b/tests/unit_tests/inference/test_batch_dimension_utils.py
@@ -0,0 +1,447 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+
+"""
+Unit tests for CUDAGraphBatchDimensionBuilder.match_graph_config with expert parallelism.
+"""
+
+import pytest
+import torch
+import torch.distributed as dist
+
+from megatron.core import parallel_state as ps
+from megatron.core.inference.batch_dimensions_utils import (
+ CUDAGraphBatchDimensionBuilder,
+ InferenceBatchDimensions,
+)
+from tests.unit_tests.test_utilities import Utils
+
+BD = InferenceBatchDimensions
+
+# Common config shared across tests
+MAX_REQUESTS = 256
+MAX_TOKENS = 2048
+MAX_SEQ_LEN = 4096
+TP_SIZE = 1
+MIXED_PREFILL_COUNT = 16
+
+
+def _generate_graphs(num_cuda_graphs, use_non_decode=True):
+ """Generate cuda graph batch dimensions using the builder."""
+ graph_list, _ = CUDAGraphBatchDimensionBuilder.generate_cuda_graph_batch_dimensions_list(
+ tp_size=TP_SIZE,
+ num_cuda_graphs=num_cuda_graphs,
+ cuda_graph_max_tokens=MAX_REQUESTS,
+ cuda_graph_mixed_prefill_request_count=MIXED_PREFILL_COUNT,
+ max_requests=MAX_REQUESTS,
+ max_tokens=MAX_TOKENS,
+ max_sequence_length=MAX_SEQ_LEN,
+ use_cuda_graphs_for_non_decode_steps=use_non_decode,
+ )
+ return graph_list
+
+
+def _match(real, graph_list, ep_group, strict=False, decode_only=False):
+ return CUDAGraphBatchDimensionBuilder.match_graph_config(
+ real_batch_dim=real,
+ cuda_graph_batch_dimensions_list=graph_list,
+ strict=strict,
+ decode_only_cuda_graphs=decode_only,
+ ep_group=ep_group,
+ smallest_non_decode_cuda_graph_size=min(MIXED_PREFILL_COUNT, MAX_REQUESTS),
+ )
+
+
+def _assert_consistent_across_ranks(result, ep_group):
+ """Assert that the match result is the same on every EP rank.
+
+ Either all ranks return None, or all ranks return a config with the
+ same token_count (which is what the all-reduce synchronises).
+ """
+ if result is None:
+ flag = torch.zeros(1, dtype=torch.int32, device="cuda")
+ else:
+ flag = torch.ones(1, dtype=torch.int32, device="cuda")
+
+ # If any rank got None, all must get None; if any rank got a match, all must.
+ flag_sum = flag.clone()
+ dist.all_reduce(flag_sum, op=dist.ReduceOp.SUM, group=ep_group)
+ ep_size = dist.get_world_size(ep_group)
+ assert (
+ flag_sum.item() == 0 or flag_sum.item() == ep_size
+ ), f"Inconsistent match: {flag_sum.item()}/{ep_size} ranks got a match"
+
+ if result is not None:
+ tc = torch.tensor([result.token_count], dtype=torch.int32, device="cuda")
+ tc_max = tc.clone()
+ tc_min = tc.clone()
+ dist.all_reduce(tc_max, op=dist.ReduceOp.MAX, group=ep_group)
+ dist.all_reduce(tc_min, op=dist.ReduceOp.MIN, group=ep_group)
+ assert (
+ tc_max.item() == tc_min.item()
+ ), f"Token count mismatch across EP ranks: min={tc_min.item()}, max={tc_max.item()}"
+
+
+class TestCUDAGraphTokenCountAlignment:
+ """Verify that mixed/prefill graph token counts are a subset of decode graph token counts."""
+
+ @pytest.mark.parametrize("num_cuda_graphs", [1, 16, 32, -1])
+ def test_mixed_token_counts_subset_of_decode(self, num_cuda_graphs):
+ """Every token count in the mixed/prefill graph pool must also appear
+ in the decode-only pool. Otherwise, when EP syncs token counts across
+ ranks, decode-only ranks cannot find a graph at the same token count
+ as prefill ranks, causing inconsistent matching."""
+ graph_list = _generate_graphs(num_cuda_graphs)
+
+ decode_token_counts = {bd.token_count for bd in graph_list if bd.prefill_req_count == 0}
+ mixed_token_counts = {bd.token_count for bd in graph_list if bd.prefill_req_count > 0}
+
+ mixed_only = mixed_token_counts - decode_token_counts
+ assert not mixed_only, (
+ f"Mixed/prefill token counts with no decode graph: {sorted(mixed_only)}. "
+ f"This will cause EP rank mismatch when some ranks are decode-only "
+ f"and others have prefill."
+ )
+
+ # Decode-only token counts not in the mixed pool are allowed, but only
+ # below MIXED_PREFILL_COUNT. The EP adjustment elevates token counts to
+ # at least MIXED_PREFILL_COUNT when any rank has prefill, so any decode
+ # token count >= MIXED_PREFILL_COUNT must have a mixed counterpart.
+ decode_only = decode_token_counts - mixed_token_counts
+ large_decode_only = {tc for tc in decode_only if tc >= MIXED_PREFILL_COUNT}
+ assert not large_decode_only, (
+ f"Decode-only token counts >= MIXED_PREFILL_COUNT ({MIXED_PREFILL_COUNT}) "
+ f"with no mixed/prefill graph: {sorted(large_decode_only)}. "
+ f"The EP token count elevation cannot guarantee alignment for these."
+ )
+
+
+class TestMatchGraphConfigWithEP:
+ """Tests for match_graph_config with expert parallelism.
+
+ Uses the world group as the EP group (all 8 GPUs form one EP group).
+ """
+
+ def setup_class(cls):
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=1,
+ pipeline_model_parallel_size=1,
+ expert_model_parallel_size=Utils.world_size,
+ )
+
+ def teardown_class(cls):
+ Utils.destroy_model_parallel()
+
+ @staticmethod
+ def _get_ep_group():
+ """Return the EP group created by initialize_model_parallel."""
+ return ps.get_expert_model_parallel_group()
+
+ # ------------------------------------------------------------------ #
+ # 1. All ranks same decode batch → consistent match
+ # ------------------------------------------------------------------ #
+ @pytest.mark.internal
+ @pytest.mark.parametrize("num_cuda_graphs", [1, 16, 32, -1])
+ def test_uniform_decode_batch(self, num_cuda_graphs):
+ """All EP ranks have the same decode-only batch → should all match the same graph."""
+ ep_group = self._get_ep_group()
+ graph_list = _generate_graphs(num_cuda_graphs)
+ real = BD(token_count=32, prefill_req_count=0, decode_req_count=32)
+
+ result = _match(real, graph_list, ep_group=ep_group)
+ _assert_consistent_across_ranks(result, ep_group)
+ assert result is not None, "Should find a matching graph for uniform decode batch"
+
+ # ------------------------------------------------------------------ #
+ # 2. Different token counts across EP ranks → all-reduce takes max
+ # ------------------------------------------------------------------ #
+ @pytest.mark.internal
+ @pytest.mark.parametrize("num_cuda_graphs", [1, 16, 32, -1])
+ def test_varying_decode_token_counts(self, num_cuda_graphs):
+ """EP ranks have different decode token counts. The all-reduce
+ should take the max, and all ranks should match the same graph."""
+ ep_group = self._get_ep_group()
+ graph_list = _generate_graphs(num_cuda_graphs)
+ rank = dist.get_rank()
+
+ # Each rank gets a different token count: 8, 16, 24, ...
+ token_count = (rank + 1) * 8
+ real = BD(token_count=token_count, prefill_req_count=0, decode_req_count=token_count)
+
+ result = _match(real, graph_list, ep_group=ep_group)
+ _assert_consistent_across_ranks(result, ep_group)
+ assert result is not None
+
+ # ------------------------------------------------------------------ #
+ # 3. decode_only_cuda_graphs=True, some ranks have prefill → all None
+ # ------------------------------------------------------------------ #
+ @pytest.mark.internal
+ @pytest.mark.parametrize("num_cuda_graphs", [1, 16, 32, -1])
+ def test_decode_only_graphs_with_mixed_ranks(self, num_cuda_graphs):
+ """When decode_only_cuda_graphs=True and at least one EP rank has a
+ prefill request, ALL ranks should get None (eager mode)."""
+ ep_group = self._get_ep_group()
+ graph_list = _generate_graphs(num_cuda_graphs)
+ rank = dist.get_rank()
+
+ # Rank 0 has a mixed batch (prefill + decode), all others decode-only
+ if rank == 0:
+ real = BD(token_count=64, prefill_req_count=2, decode_req_count=10)
+ else:
+ real = BD(token_count=32, prefill_req_count=0, decode_req_count=32)
+
+ result = _match(real, graph_list, ep_group=ep_group, decode_only=True)
+ _assert_consistent_across_ranks(result, ep_group)
+ assert (
+ result is None
+ ), "All ranks should run eager when decode_only=True and some rank has prefill"
+
+ # ------------------------------------------------------------------ #
+ # 4. Mixed prefill graphs with strict matching
+ # ------------------------------------------------------------------ #
+ @pytest.mark.internal
+ @pytest.mark.parametrize("num_cuda_graphs", [1, 16, 32, -1])
+ def test_strict_matching_with_mixed_prefill(self, num_cuda_graphs):
+ """With strict matching, request counts are synced across EP ranks
+ via all-reduce. All ranks should still get a consistent result."""
+ ep_group = self._get_ep_group()
+ graph_list = _generate_graphs(num_cuda_graphs)
+ rank = dist.get_rank()
+
+ # Varying prefill/decode split across ranks
+ prefill = min(rank + 1, MIXED_PREFILL_COUNT)
+ decode = 16 - prefill
+ real = BD(token_count=64, prefill_req_count=prefill, decode_req_count=decode)
+
+ result = _match(real, graph_list, ep_group=ep_group, strict=True)
+ _assert_consistent_across_ranks(result, ep_group)
+
+ # ------------------------------------------------------------------ #
+ # 6. Non-strict matching with mixed prefill
+ # ------------------------------------------------------------------ #
+ @pytest.mark.internal
+ @pytest.mark.parametrize("num_cuda_graphs", [1, 16, 32, -1])
+ def test_non_strict_matching_with_mixed_prefill(self, num_cuda_graphs):
+ """Non-strict matching: prefill slots can serve decode. Token count
+ is synced across EP ranks; result must be consistent."""
+ ep_group = self._get_ep_group()
+ graph_list = _generate_graphs(num_cuda_graphs)
+ rank = dist.get_rank()
+
+ prefill = min(rank + 1, MIXED_PREFILL_COUNT)
+ decode = 16 - prefill
+ real = BD(token_count=64, prefill_req_count=prefill, decode_req_count=decode)
+
+ result = _match(real, graph_list, ep_group=ep_group)
+ _assert_consistent_across_ranks(result, ep_group)
+
+ # ------------------------------------------------------------------ #
+ # 7. Mixed decode/prefill across ranks — strict matching
+ # ------------------------------------------------------------------ #
+ @pytest.mark.internal
+ @pytest.mark.parametrize("num_cuda_graphs", [1, 16, 32, -1])
+ def test_mixed_decode_and_prefill_ranks_strict(self, num_cuda_graphs):
+ """Some EP ranks are pure decode, others have prefill requests.
+ With strict matching the all-reduce syncs request counts to the
+ max across ranks. Result must be consistent."""
+ ep_group = self._get_ep_group()
+ graph_list = _generate_graphs(num_cuda_graphs)
+ rank = dist.get_rank()
+
+ # Even ranks: pure decode (32 tokens)
+ # Odd ranks: mixed prefill (64 tokens, 2 prefill + 14 decode)
+ if rank % 2 == 0:
+ real = BD(token_count=32, prefill_req_count=0, decode_req_count=32)
+ else:
+ real = BD(token_count=64, prefill_req_count=2, decode_req_count=14)
+
+ result = _match(real, graph_list, ep_group=ep_group, strict=True)
+ _assert_consistent_across_ranks(result, ep_group)
+
+ # ------------------------------------------------------------------ #
+ # 8. Mixed decode/prefill across ranks — non-strict matching
+ # ------------------------------------------------------------------ #
+ @pytest.mark.internal
+ @pytest.mark.parametrize("num_cuda_graphs", [1, 16, 32, -1])
+ def test_mixed_decode_and_prefill_ranks_non_strict(self, num_cuda_graphs):
+ """Some EP ranks are pure decode, others have prefill requests.
+ Non-strict matching only syncs token counts (not request counts).
+ Result must be consistent."""
+ ep_group = self._get_ep_group()
+ graph_list = _generate_graphs(num_cuda_graphs)
+ rank = dist.get_rank()
+
+ # Even ranks: pure decode (32 tokens)
+ # Odd ranks: mixed prefill (64 tokens, 2 prefill + 14 decode)
+ if rank % 2 == 0:
+ real = BD(token_count=32, prefill_req_count=0, decode_req_count=32)
+ else:
+ real = BD(token_count=64, prefill_req_count=2, decode_req_count=14)
+
+ result = _match(real, graph_list, ep_group=ep_group)
+ _assert_consistent_across_ranks(result, ep_group)
+
+ # ------------------------------------------------------------------ #
+ # 9. All ranks decode-only with decode_only_cuda_graphs → should match
+ # ------------------------------------------------------------------ #
+ @pytest.mark.internal
+ @pytest.mark.parametrize("num_cuda_graphs", [1, 16, 32, -1])
+ def test_decode_only_graphs_all_decode(self, num_cuda_graphs):
+ """When all EP ranks are decode-only and decode_only_cuda_graphs=True,
+ a match should be found."""
+ ep_group = self._get_ep_group()
+ graph_list = _generate_graphs(num_cuda_graphs)
+ rank = dist.get_rank()
+
+ token_count = (rank + 1) * 4
+ real = BD(token_count=token_count, prefill_req_count=0, decode_req_count=token_count)
+
+ result = _match(real, graph_list, ep_group=ep_group, decode_only=True)
+ _assert_consistent_across_ranks(result, ep_group)
+ assert result is not None, "All-decode batch with decode_only_cuda_graphs should match"
+
+ # ------------------------------------------------------------------ #
+ # 10. Real batch exceeds all graphs → None on all ranks
+ # ------------------------------------------------------------------ #
+ @pytest.mark.internal
+ @pytest.mark.parametrize("num_cuda_graphs", [1, 16, 32, -1])
+ def test_oversized_batch_returns_none(self, num_cuda_graphs):
+ """When the real batch is larger than any available graph, all ranks
+ should get None."""
+ ep_group = self._get_ep_group()
+ graph_list = _generate_graphs(num_cuda_graphs)
+
+ # Token count exceeds MAX_TOKENS on all ranks
+ real = BD(
+ token_count=MAX_TOKENS + 100,
+ prefill_req_count=0,
+ decode_req_count=min(MAX_TOKENS + 100, MAX_REQUESTS),
+ )
+
+ result = _match(real, graph_list, ep_group=ep_group)
+ _assert_consistent_across_ranks(result, ep_group)
+ assert result is None, "Oversized batch should not match any graph"
+
+ # ------------------------------------------------------------------ #
+ # 11. One EP rank has huge batch → all-reduce lifts to max → no match
+ # ------------------------------------------------------------------ #
+ @pytest.mark.internal
+ @pytest.mark.parametrize("num_cuda_graphs", [1, 16, 32, -1])
+ def test_one_rank_oversized_forces_no_match(self, num_cuda_graphs):
+ """If one EP rank has a batch exceeding all graph capacities, the
+ all-reduce max lifts everyone → no match on any rank."""
+ ep_group = self._get_ep_group()
+ graph_list = _generate_graphs(num_cuda_graphs)
+ rank = dist.get_rank()
+
+ if rank == 0:
+ # This rank has a batch that exceeds all graphs
+ real = BD(
+ token_count=MAX_TOKENS + 100,
+ prefill_req_count=0,
+ decode_req_count=min(MAX_TOKENS + 100, MAX_REQUESTS),
+ )
+ else:
+ real = BD(token_count=8, prefill_req_count=0, decode_req_count=8)
+
+ result = _match(real, graph_list, ep_group=ep_group)
+ _assert_consistent_across_ranks(result, ep_group)
+ assert result is None, "All-reduce max from oversized rank should cause no match"
+
+
+class TestSpeculativeDecodingBatchDimensions:
+ """Tests for batch dimensions specifically handling speculative decoding."""
+
+ def setup_method(self, method):
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=1,
+ pipeline_model_parallel_size=1,
+ expert_model_parallel_size=Utils.world_size,
+ )
+
+ def teardown_method(self, method):
+ Utils.destroy_model_parallel()
+
+ @staticmethod
+ def _get_ep_group():
+ return ps.get_expert_model_parallel_group()
+
+ @pytest.mark.parametrize("num_speculative_tokens", [1, 3, 5])
+ def test_generate_graphs_with_speculative_tokens(self, num_speculative_tokens):
+ """Verify graph generation strictly adheres to the speculative token multiplier."""
+ graph_list, _ = CUDAGraphBatchDimensionBuilder.generate_cuda_graph_batch_dimensions_list(
+ tp_size=TP_SIZE,
+ num_cuda_graphs=4,
+ cuda_graph_max_tokens=MAX_REQUESTS * (num_speculative_tokens + 1),
+ cuda_graph_mixed_prefill_request_count=MIXED_PREFILL_COUNT,
+ max_requests=MAX_REQUESTS,
+ max_tokens=MAX_TOKENS,
+ max_sequence_length=MAX_SEQ_LEN,
+ use_cuda_graphs_for_non_decode_steps=True,
+ num_speculative_tokens=num_speculative_tokens,
+ )
+
+ # For pure decode graphs, token_count must exactly equal decode_req_count * (spec_tokens + 1)
+ decode_graphs = [g for g in graph_list if g.prefill_req_count == 0]
+ assert len(decode_graphs) > 0, "Should generate decode-only graphs"
+
+ for g in decode_graphs:
+ expected_tokens = g.decode_req_count * (num_speculative_tokens + 1)
+ assert g.token_count == expected_tokens, (
+ f"Mismatch in speculative token math: Expected {expected_tokens} tokens "
+ f"for {g.decode_req_count} requests with {num_speculative_tokens} spec tokens, got {g.token_count}."
+ )
+
+ def test_is_valid_with_speculative_tokens(self):
+ """Verify that validation correctly enforces speculative token budgets."""
+ num_speculative_tokens = 4
+ # 10 decode requests * (4 spec + 1 actual) = 50 tokens required.
+
+ # 49 tokens is not enough -> should be invalid
+ bd_invalid = BD(token_count=49, prefill_req_count=0, decode_req_count=10)
+ assert not bd_invalid.is_valid(
+ max_requests=MAX_REQUESTS,
+ max_sequence_length=MAX_SEQ_LEN,
+ num_speculative_tokens=num_speculative_tokens,
+ ), "Should reject batch dimension without enough tokens for speculative budget."
+
+ # Exactly 50 tokens -> should be valid
+ bd_valid = BD(token_count=50, prefill_req_count=0, decode_req_count=10)
+ assert bd_valid.is_valid(
+ max_requests=MAX_REQUESTS,
+ max_sequence_length=MAX_SEQ_LEN,
+ num_speculative_tokens=num_speculative_tokens,
+ ), "Should accept batch dimension with perfectly matched speculative budget."
+
+ @pytest.mark.internal
+ @pytest.mark.parametrize("num_cuda_graphs", [1, 16, -1])
+ def test_ep_sync_with_speculative_tokens(self, num_cuda_graphs):
+ """Verify matching and EP rank syncing scales correctly with speculative tokens."""
+ ep_group = self._get_ep_group()
+ num_speculative_tokens = 2
+
+ graph_list, _ = CUDAGraphBatchDimensionBuilder.generate_cuda_graph_batch_dimensions_list(
+ tp_size=TP_SIZE,
+ num_cuda_graphs=num_cuda_graphs,
+ cuda_graph_max_tokens=MAX_REQUESTS * (num_speculative_tokens + 1),
+ cuda_graph_mixed_prefill_request_count=MIXED_PREFILL_COUNT,
+ max_requests=MAX_REQUESTS,
+ max_tokens=MAX_TOKENS,
+ max_sequence_length=MAX_SEQ_LEN,
+ use_cuda_graphs_for_non_decode_steps=True,
+ num_speculative_tokens=num_speculative_tokens,
+ )
+
+ rank = dist.get_rank()
+
+ # Each rank has a different number of decode requests.
+ decode_reqs = (rank + 1) * 2
+ token_count = decode_reqs * (num_speculative_tokens + 1)
+ real = BD(token_count=token_count, prefill_req_count=0, decode_req_count=decode_reqs)
+
+ result = _match(real, graph_list, ep_group=ep_group)
+
+ # All ranks should end up syncing to the maximum requirement and picking the same graph
+ _assert_consistent_across_ranks(result, ep_group)
+ if result is not None:
+ # Confirm the selected graph preserves the speculative token mathematical invariance
+ assert result.token_count == result.decode_req_count * (num_speculative_tokens + 1)
diff --git a/tests/unit_tests/inference/test_data_parallel_inference_coordinator.py b/tests/unit_tests/inference/test_data_parallel_inference_coordinator.py
index 57326291a73..b2e94bc54f9 100644
--- a/tests/unit_tests/inference/test_data_parallel_inference_coordinator.py
+++ b/tests/unit_tests/inference/test_data_parallel_inference_coordinator.py
@@ -2,16 +2,28 @@
import asyncio
import itertools
+import multiprocessing
+import os
import time
+import unittest.mock
from collections import deque
from typing import Dict, Optional
import msgpack
+import numpy as np
import pytest
import torch
-from tqdm import tqdm
-from megatron.core.inference.engines.dynamic_engine import DynamicInferenceEngine, RequestEntry
+from megatron.core.inference.config import PrefixCachingCoordinatorPolicy
+from megatron.core.inference.data_parallel_inference_coordinator import (
+ DataParallelInferenceCoordinator,
+)
+from megatron.core.inference.engines.async_zmq_communicator import AsyncZMQCommunicator
+from megatron.core.inference.engines.dynamic_engine import (
+ DynamicInferenceEngine,
+ EngineState,
+ RequestEntry,
+)
from megatron.core.inference.headers import Headers
from megatron.core.inference.inference_client import InferenceClient
from megatron.core.inference.inference_request import (
@@ -34,7 +46,6 @@
NUM_REQUESTS = 10
NUM_TOKENS = 2
DEFAULT_PORT = 46581
-ZMQ_FLAKY_SHUTDOWN = True
class DummyTokenizer:
@@ -64,6 +75,11 @@ class DummyContext:
def __init__(self):
self.active_cnt = 0
+ self.step_count = 0
+ self.block_size_tokens = 64
+ self.enable_prefix_caching = False
+ self.prefix_caching_coordinator_policy = None
+ self.prefix_cache_lru_clock = 0
def get_active_request_count(self) -> int:
return self.active_cnt
@@ -86,29 +102,61 @@ def __init__(self):
"""We cannot call super().__init__() because it requires complex setup."""
self.waiting_request_ids = deque()
self.requests: Dict[int, RequestEntry] = {}
- self.suspend_signal = False
- self.is_suspended = False
self._loop = get_asyncio_loop()
self.context = DummyContext()
self.controller = DummyController()
- self.running = asyncio.Event()
- self.paused = asyncio.Event()
- self.stopped = asyncio.Event()
self.pending_microbatch = deque()
- self.received_pause: bool = False
- self.received_stop: bool = False
self.pg_collection = ProcessGroupCollection.use_mpu_process_groups()
self.rank = torch.distributed.get_rank()
+ # State machine (mirrors dynamic_engine.py reset()).
+ self.state = EngineState.RUNNING
+ self._state_events = {k: asyncio.Event() for k in self._STATE_EVENTS}
+ self._state_events[EngineState.RUNNING].set()
+ self._pending_signals = deque()
+ self.resume_request_ids = None
+ self.use_coordinator = False
+
+ self.ep_world_size = 1
+
+ self.step_start_event = unittest.mock.MagicMock()
+ self.step_end_event = unittest.mock.MagicMock()
+
+ # ZMQ-based world barrier (async-friendly, no NCCL).
+ self.zmq_context = zmq.Context()
+ total_world_size = torch.distributed.get_world_size()
+ self.world_zmq_communicator = AsyncZMQCommunicator(self.zmq_context, process_group=None)
+ self.use_synchronous_zmq_collectives = False
+
+ async def run_engine_with_coordinator(self, *, loop=None):
+ """Override to bypass @trace_async_exceptions for testability.
+
+ In production, @trace_async_exceptions converts AssertionError to sys.exit(1) -> SystemExit.
+ In Python 3.12+, asyncio re-raises SystemExit from tasks in the main thread.
+ For tests, we let AssertionErrors propagate directly so pytest.raises can catch them.
+ """
+ return await DynamicInferenceEngine.run_engine_with_coordinator.__wrapped__(self, loop=loop)
+
+ def suspend(self):
+ pass
+
+ def resume(self):
+ pass
+
def add_request(
self, request_id: int, prompt: str, sampling_params: Optional[SamplingParams] = None
) -> asyncio.Future[DynamicInferenceRequestRecord]:
"""Dummy add_request."""
+ # Mock tokenization to prevent `prompt_tokens == None`.
+ prompt_tokens = (
+ torch.arange(len(prompt.split())) if isinstance(prompt, str) else torch.tensor(prompt)
+ )
self.requests[request_id] = RequestEntry(
record=DynamicInferenceRequestRecord.from_request(
DynamicInferenceRequest(
prompt=prompt,
+ prompt_tokens=prompt_tokens,
request_id=request_id,
sampling_params=sampling_params,
status=Status.WAITING_IN_QUEUE,
@@ -122,6 +170,8 @@ def add_request(
async def async_step(self, *, verbose: Optional[bool] = False) -> Dict:
"""Dummy async_step."""
+ await asyncio.sleep(0)
+
# Finish "active" requests.
finished_request_records = []
to_remove = []
@@ -139,7 +189,8 @@ async def async_step(self, *, verbose: Optional[bool] = False) -> Dict:
# Send signal to coordinator.
if self.is_mp_coordinator:
payload = msgpack.packb(
- [Headers.ENGINE_REPLY.value, [entry.record.serialize()]], use_bin_type=True
+ [Headers.ENGINE_REPLY.value, [entry.record.merge().serialize()]],
+ use_bin_type=True,
)
self.socket_for_receiving_requests.send(payload)
@@ -163,6 +214,42 @@ async def async_step(self, *, verbose: Optional[bool] = False) -> Dict:
}
+async def cleanup_engine(engine, client=None, timeout=30.0):
+ """Disconnect an engine between tests. The coordinator stays alive."""
+ task = getattr(engine, 'engine_loop_task', None)
+ if task is not None and not task.done():
+ if client is not None:
+ client.pause_engines()
+ try:
+ await asyncio.wait_for(engine.wait_until(EngineState.PAUSED), timeout=timeout)
+ except (asyncio.TimeoutError, Exception):
+ pass
+
+ if client is not None:
+ client.stop_engines()
+ try:
+ await asyncio.wait_for(asyncio.shield(task), timeout=timeout)
+ except (asyncio.TimeoutError, asyncio.CancelledError, Exception):
+ # Graceful stop failed — fall back to forcible cleanup.
+ for attr in ('expert_parallel_zmq_communicator', 'world_zmq_communicator'):
+ comm = getattr(engine, attr, None)
+ if comm is not None:
+ comm.close()
+
+ for socket in getattr(engine, 'zmq_sockets', []):
+ if not socket.closed:
+ socket.close(linger=0)
+
+ task.cancel()
+ try:
+ await asyncio.wait_for(asyncio.shield(task), timeout=5.0)
+ except (asyncio.TimeoutError, asyncio.CancelledError, Exception):
+ pass
+
+ if client is not None:
+ client.stop()
+
+
@pytest.fixture
def initialize_model_parallel(request, monkeypatch):
"""Fixture to initialize and destroy model parallel.
@@ -179,12 +266,87 @@ def initialize_model_parallel(request, monkeypatch):
pipeline_model_parallel_size=pp,
expert_model_parallel_size=ep,
)
- dp = world_size // (tp * pp * ep)
+ dp = world_size // (tp * pp)
yield world_size, dp, tp, pp, ep
Utils.destroy_model_parallel()
-@pytest.mark.skipif(ZMQ_FLAKY_SHUTDOWN, reason="ZMQ shutdown is flaky")
+@pytest.fixture
+def test_case_communicator():
+ """A separate ZMQ communicator for test sync barriers.
+
+ Use this instead of engine._world_barrier() when the engine loop may be
+ calling _world_barrier() concurrently (e.g. during state transitions).
+ """
+ ctx = zmq.Context()
+ comm = AsyncZMQCommunicator(ctx, process_group=None)
+ yield comm
+ comm.close()
+ ctx.term()
+
+
+@pytest.fixture(scope="class")
+def coordinator():
+ """Launch a single coordinator process for the entire test class.
+
+ Only rank 0 spawns the coordinator process. Non-rank-0 processes use a
+ placeholder address; the real address is broadcast inside each test's call
+ to start_listening_to_data_parallel_coordinator (which broadcasts dp_addr
+ from dp_src within the DP process group).
+
+ The coordinator is spawned with data_parallel_size=0 so it doesn't block
+ waiting for engines; engines register dynamically via the empty-payload
+ re-registration path.
+ """
+ rank = int(os.environ.get("RANK", "0"))
+
+ if rank == 0:
+ spawn_context = multiprocessing.get_context('spawn')
+ pipe_parent, pipe_child = spawn_context.Pipe()
+ ready_event = spawn_context.Event()
+ proc = spawn_context.Process(
+ target=DataParallelInferenceCoordinator.entrypoint,
+ kwargs={
+ "pipe_connection": pipe_child,
+ "ready_event": ready_event,
+ "data_parallel_size": 0,
+ "tokenizer": DummyTokenizer(),
+ "max_requests": 16,
+ "inference_coordinator_port": DEFAULT_PORT,
+ "deterministic_mode": False,
+ },
+ )
+ proc.start()
+
+ # Wait for the coordinator to bind its socket and send the address.
+ while not pipe_parent.poll(timeout=0.1):
+ assert proc.is_alive(), "Coordinator process died during init"
+ dp_addr = pipe_parent.recv()
+ pipe_parent.close()
+ ready_event.wait(timeout=10.0)
+ else:
+ proc = None
+ # Placeholder: the engine setup broadcasts rank 0's actual address.
+ dp_addr = f"tcp://localhost:{DEFAULT_PORT}"
+
+ yield dp_addr
+
+ # Only rank 0 tears down the coordinator process.
+ if rank == 0 and proc is not None and proc.is_alive():
+ ctx = zmq.Context()
+ sock = ctx.socket(zmq.DEALER)
+ sock.connect(dp_addr)
+ sock.send(msgpack.packb([Headers.CONNECT.value], use_bin_type=True))
+ sock.recv() # CONNECT_ACK
+ sock.send(msgpack.packb([Headers.SHUTDOWN.value], use_bin_type=True))
+ sock.close(linger=1000)
+ ctx.term()
+ proc.join(timeout=10.0)
+ if proc.is_alive():
+ proc.terminate()
+ proc.join(timeout=5.0)
+
+
class TestCoordinator:
"""Test class for Data Parallel Inference Coordinator."""
@@ -195,26 +357,41 @@ def build_requests(self, num_requests=NUM_REQUESTS, num_tokens=NUM_TOKENS):
for _ in range(num_requests)
]
- async def run_coordinator_test(
- self,
- *,
- launch_coordinator=True,
- stop_engines=True,
- num_requests=NUM_REQUESTS,
- num_tokens=NUM_TOKENS,
+ @pytest.mark.internal
+ @pytest.mark.skipif(not HAVE_ZMQ, reason="pyzmq is required for this test")
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize(
+ "initialize_model_parallel",
+ [
+ pytest.param((tp, pp, ep), id=f"tp{tp}-pp{pp}-ep{ep}")
+ for tp, pp, ep in itertools.product([1, 2], [1, 2], [1, 2])
+ if tp * pp * ep <= Utils.world_size
+ ],
+ indirect=["initialize_model_parallel"],
+ )
+ async def test_parallel_configs(
+ self, initialize_model_parallel, coordinator, test_case_communicator
):
- """Run a coordinator test. Model parallel must already be initialized."""
+ """Test coordinator with various TP, PP, and EP configurations."""
+ dp_addr = coordinator
+ port = int(dp_addr.rsplit(":", 1)[-1])
+ requests = self.build_requests()
engine = DummyEngine()
- requests = self.build_requests(num_requests, num_tokens)
+ rank = torch.distributed.get_rank()
- dp_addr = await engine.start_listening_to_data_parallel_coordinator(
- inference_coordinator_port=DEFAULT_PORT, launch_inference_coordinator=launch_coordinator
+ await engine.start_listening_to_data_parallel_coordinator(
+ inference_coordinator_port=port, launch_inference_coordinator=False
)
+ # Ensure all engines are registered before submitting requests.
+ await asyncio.wait_for(test_case_communicator.all_reduce_max(1), timeout=30.0)
+
+ client = None
try:
- if torch.distributed.get_rank() == 0:
+ if rank == 0:
+ await asyncio.sleep(0)
client = InferenceClient(dp_addr)
- await client.start()
+ client.start()
futures = [
client.add_request(prompt=prompt, sampling_params=params)
@@ -222,268 +399,400 @@ async def run_coordinator_test(
]
results = await asyncio.wait_for(asyncio.gather(*futures), timeout=10.0)
- for record in results:
- assert record[-1].status == Status.COMPLETED
+ for result in results:
+ assert result["status"] == Status.COMPLETED.name
+
+ await asyncio.wait_for(test_case_communicator.all_reduce_max(1), timeout=30.0)
finally:
+ await cleanup_engine(engine, client)
+
+ @pytest.mark.internal
+ @pytest.mark.skipif(not HAVE_ZMQ, reason="pyzmq is required for this test")
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize("deserialize", [True, False], ids=["deserialize", "raw"])
+ async def test_deserialize_flag(
+ self, initialize_model_parallel, coordinator, test_case_communicator, deserialize
+ ):
+ """Test that the correct response type is returned based on the deserialize flag."""
+ dp_addr = coordinator
+ port = int(dp_addr.rsplit(":", 1)[-1])
+ engine = DummyEngine()
+ requests = self.build_requests(num_requests=2)
+
+ await engine.start_listening_to_data_parallel_coordinator(
+ inference_coordinator_port=port, launch_inference_coordinator=False
+ )
+
+ # Ensure all engines are registered before submitting requests.
+ await asyncio.wait_for(test_case_communicator.all_reduce_max(1), timeout=30.0)
+
+ client = None
+ try:
if torch.distributed.get_rank() == 0:
- if stop_engines:
- await asyncio.wait_for(client.stop_engines(), timeout=10.0)
- client.stop()
- if stop_engines:
- try:
- await asyncio.wait_for(engine.engine_loop_task, timeout=30.0)
- except asyncio.TimeoutError:
- engine.engine_loop_task.cancel()
+ await asyncio.sleep(0)
+ client = InferenceClient(dp_addr, deserialize=deserialize)
+ client.start()
+ futures = [
+ client.add_request(prompt=prompt, sampling_params=params)
+ for prompt, params in requests
+ ]
+ results = await asyncio.wait_for(asyncio.gather(*futures), timeout=10.0)
+ for result in results:
+ if deserialize:
+ assert isinstance(result, DynamicInferenceRequest)
+ else:
+ assert isinstance(result, dict)
- return dp_addr
+ await asyncio.wait_for(test_case_communicator.all_reduce_max(1), timeout=30.0)
+ finally:
+ await cleanup_engine(engine, client)
@pytest.mark.internal
@pytest.mark.skipif(not HAVE_ZMQ, reason="pyzmq is required for this test")
@pytest.mark.asyncio
@pytest.mark.parametrize(
"initialize_model_parallel",
- [
- pytest.param((tp, pp, ep), id=f"tp{tp}-pp{pp}-ep{ep}")
- for tp, pp, ep in itertools.product([1, 2], [1, 2], [1, 2])
- if tp * pp * ep <= Utils.world_size
- ],
+ [pytest.param((2, 2, 2), id="tp2-pp2-ep2")],
indirect=["initialize_model_parallel"],
)
- async def test_parallel_configs(self, initialize_model_parallel):
- """Test coordinator with various TP, PP, and EP configurations."""
- await self.run_coordinator_test()
+ async def test_control_logic_lifecycle(
+ self, initialize_model_parallel, coordinator, test_case_communicator
+ ):
+ """Comprehensive lifecycle test for the engine state machine."""
+ # States where paused stays set: once set during PAUSE, it's only cleared by UNPAUSE.
+ PAUSED_FAMILY = {
+ EngineState.PAUSED,
+ EngineState.UNPAUSING,
+ EngineState.SUSPENDING,
+ EngineState.SUSPENDED,
+ EngineState.RESUMING,
+ EngineState.STOPPING,
+ EngineState.STOPPED,
+ }
- @pytest.mark.internal
- @pytest.mark.skipif(not HAVE_ZMQ, reason="pyzmq is required for this test")
- @pytest.mark.asyncio
- async def test_coordinator_lifecycle(self, initialize_model_parallel):
- """Test coordinator connection and port conflict behavior."""
- engine1 = DummyEngine()
- engine2 = None
- engine3 = None
- third_addr = None
-
- # Launch first coordinator - binds to DEFAULT_PORT
- first_addr = await engine1.start_listening_to_data_parallel_coordinator(
- inference_coordinator_port=DEFAULT_PORT, launch_inference_coordinator=True
- )
+ def assert_state(eng, expected):
+ """Assert engine state and all four event flags are consistent."""
+ assert eng.state == expected, f"Expected state {expected}, got {eng.state}"
+ assert eng._state_events[EngineState.RUNNING].is_set() == (
+ expected == EngineState.RUNNING
+ ), f"RUNNING.is_set()={eng._state_events[EngineState.RUNNING].is_set()} for state={expected}"
+ assert eng._state_events[EngineState.PAUSED].is_set() == (
+ expected in PAUSED_FAMILY
+ ), f"PAUSED.is_set()={eng._state_events[EngineState.PAUSED].is_set()} for state={expected}"
+ assert eng._state_events[EngineState.SUSPENDED].is_set() == (
+ expected == EngineState.SUSPENDED
+ ), f"SUSPENDED.is_set()={eng._state_events[EngineState.SUSPENDED].is_set()} for state={expected}"
+ assert eng._state_events[EngineState.STOPPED].is_set() == (
+ expected == EngineState.STOPPED
+ ), f"STOPPED.is_set()={eng._state_events[EngineState.STOPPED].is_set()} for state={expected}"
+
+ dp_addr = coordinator
+ port = int(dp_addr.rsplit(":", 1)[-1])
+ requests = self.build_requests(num_requests=16)
+ engine = DummyEngine()
+ client = None
+ doomed_futures = []
+ rank = torch.distributed.get_rank()
try:
- # Cancel engine1 loop without sending stop to coordinator
- # This keeps coordinator process alive and holding the port
- engine1.engine_loop_task.cancel()
- try:
- await engine1.engine_loop_task
- except asyncio.CancelledError:
- pass
-
- # Connect engine2 to existing coordinator (don't launch new one)
- engine2 = DummyEngine()
- second_addr = await engine2.start_listening_to_data_parallel_coordinator(
- inference_coordinator_port=DEFAULT_PORT, launch_inference_coordinator=False
+ await engine.start_listening_to_data_parallel_coordinator(
+ inference_coordinator_port=port, launch_inference_coordinator=False
)
- # Should connect to same port, but will not always in CI due to port conflicts.
- first_port = int(first_addr.rsplit(":", 1)[-1])
- second_port = int(second_addr.rsplit(":", 1)[-1])
- # assert second_port == first_port
+ # Synchronize all ranks so every engine has registered.
+ # Use test_case_communicator to avoid colliding with engine-internal barriers.
+ await asyncio.wait_for(test_case_communicator.all_reduce_max(1), timeout=30.0)
- # Cancel engine2
- engine2.engine_loop_task.cancel()
- try:
- await engine2.engine_loop_task
- except asyncio.CancelledError:
- pass
+ if rank == 0:
+ client = InferenceClient(dp_addr)
+ client.start()
- # Launch new coordinator - should get different port since first is holding it
- engine3 = DummyEngine()
- third_addr = await engine3.start_listening_to_data_parallel_coordinator(
- inference_coordinator_port=DEFAULT_PORT, launch_inference_coordinator=True
- )
+ await asyncio.wait_for(engine.wait_until(EngineState.RUNNING), timeout=5.0)
+ assert_state(engine, EngineState.RUNNING)
- # Verify we got a different port due to conflict
- third_port = int(third_addr.rsplit(":", 1)[-1])
- assert (
- third_port != first_port
- ), f"Expected different port due to conflict, but got same: {third_port}"
+ # Try to submit signals out of FSM order.
+ # The coordinator's state machine filters these out.
+ client.suspend_engines()
+ await asyncio.sleep(0.1)
+ assert_state(engine, EngineState.RUNNING)
+ client.resume_engines()
+ await asyncio.sleep(0.1)
+ assert_state(engine, EngineState.RUNNING)
+ client.stop_engines()
+ await asyncio.sleep(0.1)
+ assert_state(engine, EngineState.RUNNING)
- finally:
- # Clean up engine3's coordinator
- if engine3 is not None and third_addr is not None:
- client3 = InferenceClient(third_addr)
- await client3.start()
- await asyncio.wait_for(client3.stop_engines(), timeout=10.0)
- client3.stop()
- try:
- await asyncio.wait_for(engine3.engine_loop_task, timeout=30.0)
- except asyncio.TimeoutError:
- engine3.engine_loop_task.cancel()
-
- # Rebuild engine and reconnect to engine1's coordinator
- first_port = int(first_addr.rsplit(":", 1)[-1])
- engine1 = DummyEngine()
- await engine1.start_listening_to_data_parallel_coordinator(
- inference_coordinator_port=first_port, launch_inference_coordinator=False
- )
- client1 = InferenceClient(first_addr)
- await client1.start()
- await asyncio.wait_for(client1.stop_engines(), timeout=10.0)
- client1.stop()
- try:
- await asyncio.wait_for(engine1.engine_loop_task, timeout=30.0)
- except asyncio.TimeoutError:
- engine1.engine_loop_task.cancel()
+ # Submit and complete requests while running.
+ futures = [client.add_request(prompt=p, sampling_params=s) for p, s in requests[:2]]
+ results = await asyncio.wait_for(asyncio.gather(*futures), timeout=5.0)
+ for result in results:
+ assert result["status"] == Status.COMPLETED.name
+
+ # Submit requests while RUNNING, then PAUSE before they drain.
+ # These must survive the PAUSE (not be drained during PAUSING).
+ pre_pause_futures = [
+ client.add_request(prompt=p, sampling_params=s) for p, s in requests[2:3]
+ ]
+ client.pause_engines()
+ await asyncio.wait_for(engine.wait_until(EngineState.PAUSED), timeout=5.0)
+ assert_state(engine, EngineState.PAUSED)
- @pytest.mark.internal
- @pytest.mark.skipif(not HAVE_ZMQ, reason="pyzmq is required for this test")
- @pytest.mark.asyncio
- async def test_pause(self, initialize_model_parallel):
- """Test pause and resume functionality."""
- engine = DummyEngine()
- requests = self.build_requests(num_requests=32)
+ # Pre-pause requests must NOT have been drained.
+ done, pending = await asyncio.wait(pre_pause_futures, timeout=0.5)
+ assert len(pending) > 0, "Pre-pause requests should not drain during PAUSING"
- dp_addr = await engine.start_listening_to_data_parallel_coordinator(
- inference_coordinator_port=DEFAULT_PORT, launch_inference_coordinator=True
- )
+ # Try pausing again and see if it breaks.
+ client.pause_engines()
+ await asyncio.sleep(0.1)
+ assert_state(engine, EngineState.PAUSED)
- success = True
- try:
- if torch.distributed.get_rank() == 0:
- client = InferenceClient(dp_addr)
- await client.start()
+ # Requests submitted while PAUSED should queue, not complete.
+ paused_futures = [
+ client.add_request(prompt=p, sampling_params=s) for p, s in requests[3:5]
+ ]
+ # Use asyncio.wait (not wait_for) so futures aren't cancelled.
+ done, pending = await asyncio.wait(paused_futures, timeout=0.5)
+ assert len(done) == 0, "No requests should complete while paused"
+ assert len(pending) == 2
- # Submit requests and pause after completion.
- futures = [client.add_request(prompt=p, sampling_params=s) for p, s in requests[:2]]
+ # UNPAUSE and verify all in-flight requests (pre-pause + paused) complete.
+ client.unpause_engines()
+ await asyncio.wait_for(engine.wait_until(EngineState.RUNNING), timeout=5.0)
+ all_queued = pre_pause_futures + paused_futures
+ results = await asyncio.wait_for(asyncio.gather(*all_queued), timeout=10.0)
+ for result in results:
+ assert result["status"] == Status.COMPLETED.name
+ assert_state(engine, EngineState.RUNNING)
+
+ # Engine processes new requests normally after unpause.
+ futures = [
+ client.add_request(prompt=p, sampling_params=s) for p, s in requests[5:7]
+ ]
+ results = await asyncio.wait_for(asyncio.gather(*futures), timeout=5.0)
+ for result in results:
+ assert result["status"] == Status.COMPLETED.name
+
+ # Suspend.
+ client.pause_engines()
+ await asyncio.wait_for(engine.wait_until(EngineState.PAUSED), timeout=5.0)
+ assert_state(engine, EngineState.PAUSED)
+
+ client.suspend_engines()
+ await asyncio.wait_for(engine.wait_until(EngineState.SUSPENDED), timeout=5.0)
+ assert_state(engine, EngineState.SUSPENDED)
+
+ # Try pausing again and see if it breaks.
+ client.pause_engines()
+ await asyncio.sleep(0.1)
+ assert_state(engine, EngineState.SUSPENDED)
+
+ # Try suspending again and see if it breaks.
+ client.pause_engines()
await asyncio.sleep(0.1)
- awaitables = futures + [client.pause_engines()]
- try:
- await asyncio.wait_for(asyncio.gather(*awaitables), timeout=0.5)
- except asyncio.TimeoutError:
- pytest.fail("Pause operation timed out.")
-
- # Ensure that requests can be added while paused.
- prompt, params = requests[2]
- future = client.add_request(prompt=prompt, sampling_params=params)
- with pytest.raises(asyncio.TimeoutError):
- await asyncio.wait_for(future, timeout=0.1)
-
- # Resume and verify new requests complete.
+ assert_state(engine, EngineState.SUSPENDED)
+
+ # Resume.
+ client.resume_engines()
+ await asyncio.wait_for(engine.wait_until(EngineState.RESUMED), timeout=5.0)
+ assert_state(engine, EngineState.PAUSED)
+ assert not engine._state_events[EngineState.SUSPENDED].is_set()
+
+ # Engine processes requests after suspend/resume cycle.
client.unpause_engines()
- # TODO: The system should not be incorrectly raising a cancelled error here.
- with pytest.raises(asyncio.CancelledError):
- await future
+ await asyncio.wait_for(engine.wait_until(EngineState.RUNNING), timeout=5.0)
futures = [
- client.add_request(prompt=p, sampling_params=s) for p, s in requests[3:4]
+ client.add_request(prompt=p, sampling_params=s) for p, s in requests[7:10]
]
- await asyncio.sleep(0.1)
- try:
- await asyncio.wait_for(asyncio.gather(*futures), timeout=0.5)
- except asyncio.TimeoutError:
- pytest.fail("Resumed requests did not complete in time.")
- except:
- success = False
+ results = await asyncio.wait_for(asyncio.gather(*futures), timeout=5.0)
+ for result in results:
+ assert result["status"] == Status.COMPLETED.name
+
+ # Submit requests that will be cancelled on STOP.
+ client.pause_engines()
+ await asyncio.wait_for(engine.wait_until(EngineState.PAUSED), timeout=5.0)
+ assert_state(engine, EngineState.PAUSED)
+
+ doomed_futures = [
+ client.add_request(prompt=p, sampling_params=s) for p, s in requests[10:13]
+ ]
+
+ # Synchronize all ranks before STOP.
+ await asyncio.wait_for(test_case_communicator.all_reduce_max(1), timeout=30.0)
+
+ if rank == 0:
+ # Verify doomed futures are still pending.
+ for f in doomed_futures:
+ assert not f.done(), "Client futures should still be pending"
+ client.stop_engines()
+
+ await asyncio.wait_for(engine.wait_until(EngineState.STOPPED), timeout=60.0)
+ assert_state(engine, EngineState.STOPPED)
+
finally:
- try:
- if torch.distributed.get_rank() == 0:
- await asyncio.wait_for(client.stop_engines(), timeout=5.0)
- client.stop()
- await asyncio.wait_for(engine.engine_loop_task, timeout=30.0)
- except asyncio.TimeoutError:
- engine.engine_loop_task.cancel()
- assert success, "Pause/resume test failed."
+ await cleanup_engine(engine, client)
+
+ # cleanup_engine called client.stop() which cancels pending futures.
+ if torch.distributed.get_rank() == 0:
+ for f in doomed_futures:
+ assert f.cancelled(), "Client futures should be cancelled after client.stop()"
@pytest.mark.internal
@pytest.mark.skipif(not HAVE_ZMQ, reason="pyzmq is required for this test")
@pytest.mark.asyncio
- async def test_throughput(self, initialize_model_parallel):
- """Throughput test with no TP or PP."""
- num_requests = 10**4
+ async def test_throughput(self, initialize_model_parallel, coordinator, test_case_communicator):
+ """Throughput benchmark: measures ZMQ packet rate."""
+ _, dp, _, _, _ = initialize_model_parallel
+ num_requests = 10**3
num_iterations = 10
+ dp_addr = coordinator
+ port = int(dp_addr.rsplit(":", 1)[-1])
engine = DummyEngine()
requests = self.build_requests(num_requests=num_requests)
- start_time = time.time()
- dp_addr = await engine.start_listening_to_data_parallel_coordinator(
- inference_coordinator_port=DEFAULT_PORT, launch_inference_coordinator=True
+ await engine.start_listening_to_data_parallel_coordinator(
+ inference_coordinator_port=port, launch_inference_coordinator=False
)
+ # Ensure all engines are registered before submitting requests.
+ await asyncio.wait_for(test_case_communicator.all_reduce_max(1), timeout=30.0)
+
+ client = None
try:
if torch.distributed.get_rank() == 0:
client = InferenceClient(dp_addr)
- await client.start()
- init_time = time.time()
+ client.start()
+ start_time = time.time()
for _ in range(num_iterations):
futures = []
- for prompt, sampling_params in tqdm(requests, "add_requests"):
+ for prompt, sampling_params in requests:
fut = client.add_request(prompt=prompt, sampling_params=sampling_params)
futures.append(fut)
- await asyncio.wait_for(asyncio.gather(*futures), timeout=10.0)
- done_time = time.time()
+ await asyncio.wait_for(asyncio.gather(*futures), timeout=30.0)
+ elapsed_ms = (time.time() - start_time) * 1e3
+ total = num_requests * num_iterations // dp
+ print(
+ f"ZMQ throughput: {total / elapsed_ms:.2f} requests/ms "
+ f"({total} reqs in {elapsed_ms:.0f} ms)"
+ )
+ await asyncio.wait_for(test_case_communicator.all_reduce_max(1), timeout=30.0)
finally:
- if torch.distributed.get_rank() == 0:
- await asyncio.wait_for(client.stop_engines(), timeout=10.0)
- client.stop()
- try:
- await asyncio.wait_for(engine.engine_loop_task, timeout=30.0)
- except asyncio.TimeoutError:
- engine.engine_loop_task.cancel()
+ await cleanup_engine(engine, client, timeout=60.0)
- stop_time = time.time()
- flags = torch.tensor([1, 1, 1], dtype=torch.int, device=torch.cuda.current_device())
+def _set_hash_rank(coord, h, rank_identity, timestamp):
+ """Test helper: set a hash→rank timestamp in the coordinator's dict."""
+ rank_idx = coord.identity_to_rank_index[rank_identity]
+ coord._hash_table.setdefault(h, {})[rank_idx] = timestamp
- init_duration = golden_init_duration = None
- run_duration = golden_run_duration = None
- stop_duration = golden_stop_duration = None
- if torch.distributed.get_rank() == 0:
- init_duration = (init_time - start_time) * 10**3
- golden_init_duration = 6974.43 # ms
- run_duration = (done_time - init_time) * 10**3
- golden_run_duration = 4392.63 # ms
- stop_duration = (stop_time - done_time) * 10**3
- golden_stop_duration = 931.49 # ms
-
- def clamp_to_golden_value(value, golden_value, delta=0.1):
- return value > golden_value * (1 - delta) and value < golden_value * (1 + delta)
-
- if not clamp_to_golden_value(init_duration, golden_init_duration, delta=0.5):
- flags[0] = 0
- if not clamp_to_golden_value(run_duration, golden_run_duration, delta=0.2):
- flags[1] = 0
- if not clamp_to_golden_value(stop_duration, golden_stop_duration, delta=1.0):
- flags[2] = 0
-
- # Synchronize results
- torch.distributed.broadcast(flags, src=0)
+def _make_routing_coordinator(
+ num_ranks=4, enable_prefix_caching=False, policy=PrefixCachingCoordinatorPolicy.LONGEST_PREFIX
+):
+ """Create a coordinator with fake rank identities for routing-only tests.
- if torch.distributed.get_rank() == 0:
- print(f"Initialization time: {init_duration:.2f} ms")
- print(f"Run time: {run_duration:.2f} ms")
- print(f"Stop time: {stop_duration:.2f} ms")
+ Thin wrapper around the shared helper in coordinator_test_utils.py.
+ """
+ from tests.unit_tests.inference.coordinator_test_utils import (
+ make_coordinator_direct as _make_coordinator,
+ )
- assert flags[0].item() == 1, (
- f"WARNING: Init duration {init_duration:.2f}s deviates from "
- f"golden value {golden_init_duration:.2f}s"
- )
- assert flags[1].item() == 1, (
- f"WARNING: Run duration {run_duration:.2f}s deviates from "
- f"golden value {golden_run_duration:.2f}s"
- )
- assert flags[2].item() == 1, (
- f"WARNING: Stop duration {stop_duration:.2f}s deviates from "
- f"golden value {golden_stop_duration:.2f}s"
- )
+ return _make_coordinator(
+ data_parallel_size=num_ranks,
+ block_size_tokens=64,
+ enable_prefix_caching=enable_prefix_caching,
+ policy=policy,
+ rank_name_template="rank-{}",
+ )
- print(
- f"ZMQ throughput is approximately "
- f"{num_requests * num_iterations / run_duration:.2f} "
- f"requests/ms"
- )
- else:
- assert flags[0].item() == 1
- assert flags[1].item() == 1
- assert flags[2].item() == 1
+
+class TestRoutingPolicies:
+ """Unit tests for routing behavior under different policies and load conditions."""
+
+ def test_no_prefix_caching_uses_round_robin(self):
+ """When prefix caching is off, round-robin is used regardless of load."""
+ coord = _make_routing_coordinator(num_ranks=3, enable_prefix_caching=False)
+ coord._pending_counts[coord.identity_to_rank_index[b"rank-0"]] = 2
+ coord._pending_counts[coord.identity_to_rank_index[b"rank-1"]] = 1
+
+ results = [coord.get_best_data_parallel_rank([]) for _ in range(6)]
+ assert results == [b"rank-0", b"rank-1", b"rank-2", b"rank-0", b"rank-1", b"rank-2"]
+
+ def test_empty_hashes_uses_round_robin(self):
+ """Empty hash list falls back to round-robin."""
+ coord = _make_routing_coordinator(num_ranks=4)
+ coord._pending_counts[coord.identity_to_rank_index[b"rank-1"]] = 5
+
+ results = [coord.get_best_data_parallel_rank([]) for _ in range(4)]
+ assert results == [b"rank-0", b"rank-1", b"rank-2", b"rank-3"]
+
+ def test_prefix_affinity_routing(self):
+ """When prefix caching is on with hashes, scoring picks the best rank."""
+ coord = _make_routing_coordinator(
+ num_ranks=3,
+ enable_prefix_caching=True,
+ policy=PrefixCachingCoordinatorPolicy.LONGEST_PREFIX,
+ )
+ for ident in coord.identities_of_data_parallel_ranks:
+ coord._pending_counts[coord.identity_to_rank_index[ident]] = 1
+
+ # Seed a hash on rank-2 so prefix routing prefers it.
+ fake_hash = 12345
+ _set_hash_rank(coord, fake_hash, b"rank-2", 1)
+
+ chosen = coord.get_best_data_parallel_rank([fake_hash])
+ assert chosen == b"rank-2"
+
+ def test_prefix_affinity_beats_free_capacity(self):
+ """A rank with a prefix match and capacity is preferred over a free rank."""
+ coord = _make_routing_coordinator(
+ num_ranks=3,
+ enable_prefix_caching=True,
+ policy=PrefixCachingCoordinatorPolicy.LONGEST_PREFIX,
+ )
+ coord._pending_counts[coord.identity_to_rank_index[b"rank-0"]] = 2
+ coord._pending_counts[coord.identity_to_rank_index[b"rank-1"]] = 1
+
+ fake_hash = 99999
+ _set_hash_rank(coord, fake_hash, b"rank-1", 1)
+
+ # Scoring: rank-1 gets prefix match bonus, which outweighs rank-2's
+ # free capacity advantage.
+ chosen = coord.get_best_data_parallel_rank([fake_hash])
+ assert chosen == b"rank-1"
+
+ def test_free_capacity_wins_when_prefix_rank_is_full(self):
+ """A free rank wins when the prefix-matched rank is full and alpha is low."""
+ coord = _make_routing_coordinator(
+ num_ranks=2,
+ enable_prefix_caching=True,
+ policy=PrefixCachingCoordinatorPolicy.FIRST_PREFIX_BLOCK,
+ )
+ coord.prefix_caching_routing_alpha = 0.1
+ coord._pending_counts[coord.identity_to_rank_index[b"rank-0"]] = 10
+
+ fake_hash = 42
+ _set_hash_rank(coord, fake_hash, b"rank-0", 1)
+
+ # score(rank-0) = 0.1*1 + 0.9*(0/10) = 0.1
+ # score(rank-1) = 0.1*0 + 0.9*(10/10) = 0.9
+ chosen = coord.get_best_data_parallel_rank([fake_hash])
+ assert chosen == b"rank-1"
+
+ def test_round_robin_policy_ignores_load(self):
+ """ROUND_ROBIN policy does naive round-robin regardless of load."""
+ coord = _make_routing_coordinator(
+ num_ranks=3,
+ enable_prefix_caching=True,
+ policy=PrefixCachingCoordinatorPolicy.ROUND_ROBIN,
+ )
+ coord._pending_counts[coord.identity_to_rank_index[b"rank-0"]] = 1
+ coord._pending_counts[coord.identity_to_rank_index[b"rank-1"]] = 1
+
+ coord._round_robin_idx = 0
+ identities = list(coord.identities_of_data_parallel_ranks)
+ for i in range(len(identities)):
+ chosen = coord.get_best_data_parallel_rank([99])
+ assert chosen == identities[i]
diff --git a/tests/unit_tests/inference/test_dynamic_prefix_caching_coordinator.py b/tests/unit_tests/inference/test_dynamic_prefix_caching_coordinator.py
new file mode 100644
index 00000000000..b4b8da0e538
--- /dev/null
+++ b/tests/unit_tests/inference/test_dynamic_prefix_caching_coordinator.py
@@ -0,0 +1,893 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""Tests for prefix-cache-aware coordinator routing.
+
+Validates that the DataParallelInferenceCoordinator correctly computes block
+hashes from prompts, routes requests to the DP rank with the longest consecutive
+prefix match, and maintains per-rank shadow state (cached hashes and timestamps).
+"""
+
+import asyncio
+import itertools
+from collections import deque
+from typing import Dict, Optional
+from unittest.mock import MagicMock
+
+import msgpack
+import numpy as np
+import pytest
+import torch
+
+from megatron.core.inference.config import PrefixCachingCoordinatorPolicy
+from megatron.core.inference.data_parallel_inference_coordinator import (
+ DataParallelInferenceCoordinator,
+)
+from megatron.core.inference.engines.dynamic_engine import DynamicInferenceEngine, RequestEntry
+from megatron.core.inference.headers import Headers
+from megatron.core.inference.inference_client import InferenceClient
+from megatron.core.inference.inference_request import (
+ DynamicInferenceRequest,
+ DynamicInferenceRequestRecord,
+ Status,
+ compute_block_hashes_batched,
+)
+from megatron.core.inference.sampling_params import SamplingParams
+from megatron.core.process_groups_config import ProcessGroupCollection
+from megatron.core.utils import get_asyncio_loop
+from tests.unit_tests.test_utilities import Utils
+
+try:
+ import zmq
+
+ HAVE_ZMQ = True
+except ImportError:
+ HAVE_ZMQ = False
+
+
+# ============================================================================
+# Shared fixtures and helpers
+# ============================================================================
+
+DEFAULT_PORT = 46582
+ZMQ_FLAKY_SHUTDOWN = True
+
+BLOCK_SIZE = 4
+
+
+def _set_hash_rank(coordinator, h, rank_identity, timestamp):
+ """Test helper: set a hash→rank timestamp in the coordinator's dict."""
+ rank_idx = coordinator.identity_to_rank_index[rank_identity]
+ coordinator._hash_table.setdefault(h, {})[rank_idx] = timestamp
+
+
+class DummyTokenizer:
+ """Dummy tokenizer that splits on whitespace and converts to ints."""
+
+ def __init__(self, vocab_size: int = 10, bos: int | None = None, eod: int = 0, pad: int = 0):
+ self.vocab_size = vocab_size
+ self.bos = bos
+ self.eod = eod
+ self.pad = pad
+
+ def tokenize(self, prompt):
+ if isinstance(prompt, str):
+ return [int(tok) % self.vocab_size for tok in prompt.strip().split()]
+ return list(prompt)
+
+ def detokenize(self, tokens, skip_special_tokens: bool = False):
+ if isinstance(tokens, torch.Tensor):
+ tokens = tokens.tolist()
+ if skip_special_tokens and self.eod in tokens:
+ tokens = [tok for tok in tokens if tok != self.eod]
+ return " ".join(str(tok) for tok in tokens)
+
+
+class DummyContext:
+ """Dummy inference context."""
+
+ def __init__(self):
+ self.active_cnt = 0
+
+ def get_active_request_count(self) -> int:
+ return self.active_cnt
+
+
+class DummyController:
+ """Dummy inference controller."""
+
+ def __init__(self):
+ self.tokenizer = DummyTokenizer()
+
+ def dummy_forward(self):
+ pass
+
+
+class DummyEngine(DynamicInferenceEngine):
+ """Dummy inference engine that only implements coordinator-related methods."""
+
+ def __init__(self):
+ self.waiting_request_ids = deque()
+ self.requests: Dict[int, RequestEntry] = {}
+ self.suspend_signal = False
+ self.is_suspended = False
+ self._loop = get_asyncio_loop()
+ self.context = DummyContext()
+ self.controller = DummyController()
+ self.running = asyncio.Event()
+ self.paused = asyncio.Event()
+ self.stopped = asyncio.Event()
+ self.pending_microbatch = deque()
+ self.received_pause: bool = False
+ self.received_stop: bool = False
+ self.pg_collection = ProcessGroupCollection.use_mpu_process_groups()
+ self.rank = torch.distributed.get_rank()
+
+ def add_request(
+ self, request_id: int, prompt: str, sampling_params: Optional[SamplingParams] = None
+ ) -> asyncio.Future[DynamicInferenceRequestRecord]:
+ self.requests[request_id] = RequestEntry(
+ record=DynamicInferenceRequestRecord.from_request(
+ DynamicInferenceRequest(
+ prompt=prompt,
+ request_id=request_id,
+ sampling_params=sampling_params,
+ status=Status.WAITING_IN_QUEUE,
+ )
+ ),
+ future=self._loop.create_future(),
+ )
+ self.waiting_request_ids.append(request_id)
+ return self.requests[request_id].future
+
+ async def async_step(self, *, verbose: Optional[bool] = False) -> Dict:
+ finished_request_records = []
+ to_remove = []
+ for request_id, entry in self.requests.items():
+ request = entry.record[-1]
+ if request.status == Status.ACTIVE_AND_GENERATING_TOKENS:
+ request.sampling_params.num_tokens_to_generate -= 1
+ if request.sampling_params.num_tokens_to_generate > 0:
+ continue
+ request.status = Status.COMPLETED
+ self.context.active_cnt -= 1
+ finished_request_records.append(entry.record)
+ entry.future.set_result(entry.record)
+ to_remove.append(request_id)
+ if self.is_mp_coordinator:
+ payload = msgpack.packb(
+ [Headers.ENGINE_REPLY.value, [entry.record.serialize()]], use_bin_type=True
+ )
+ self.socket_for_receiving_requests.send(payload)
+
+ for request_id in to_remove:
+ del self.requests[request_id]
+
+ active_request_ids = []
+ while self.waiting_request_ids:
+ request_id = self.waiting_request_ids.popleft()
+ record = self.requests[request_id].record
+ record[-1].status = Status.ACTIVE_AND_GENERATING_TOKENS
+ self.context.active_cnt += 1
+ active_request_ids.append(request_id)
+
+ return {
+ "active_request_ids": active_request_ids,
+ "finished_request_records": finished_request_records,
+ "step_time": 0.01,
+ "cuda_graph_request_count": 1,
+ }
+
+
+@pytest.fixture
+def initialize_model_parallel(request, monkeypatch):
+ monkeypatch.setenv("CUDA_DEVICE_MAX_CONNECTIONS", "1")
+ tp, pp, ep = getattr(request, "param", (1, 1, 1))
+ world_size = Utils.world_size
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=tp,
+ pipeline_model_parallel_size=pp,
+ expert_model_parallel_size=ep,
+ )
+ dp = world_size // (tp * pp * ep)
+ yield world_size, dp, tp, pp, ep
+ Utils.destroy_model_parallel()
+
+
+def make_coordinator_direct(
+ data_parallel_size=2,
+ block_size_tokens=BLOCK_SIZE,
+ enable_prefix_caching=True,
+ deterministic_mode=True,
+ prefix_caching_routing_alpha=0.5,
+ max_requests=10,
+):
+ """Create a coordinator with mock ZMQ, for unit testing routing logic.
+
+ Thin wrapper around the shared helper in coordinator_test_utils.py that
+ supplies a DummyTokenizer and this module's BLOCK_SIZE default.
+ """
+ from tests.unit_tests.inference.coordinator_test_utils import (
+ make_coordinator_direct as _make_coordinator,
+ )
+
+ return _make_coordinator(
+ data_parallel_size=data_parallel_size,
+ block_size_tokens=block_size_tokens,
+ enable_prefix_caching=enable_prefix_caching,
+ deterministic_mode=deterministic_mode,
+ prefix_caching_routing_alpha=prefix_caching_routing_alpha,
+ max_requests=max_requests,
+ tokenizer=DummyTokenizer(),
+ )
+
+
+# ============================================================================
+# Test classes
+# ============================================================================
+
+
+class TestCoordinatorHashComputation:
+ """Test that the coordinator computes correct block hashes from prompts."""
+
+ def test_hash_from_token_list(self):
+ """Hashes from a list of token IDs match compute_block_hashes_batched."""
+ coordinator = make_coordinator_direct()
+ tokens = [1, 2, 3, 4, 5, 6, 7, 8]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ expected = compute_block_hashes_batched(torch.tensor(tokens, dtype=torch.int64), BLOCK_SIZE)
+ assert hashes == expected
+ assert len(hashes) == 2 # 8 tokens / block_size 4 = 2 blocks
+
+ def test_hash_from_string_prompt(self):
+ """Hashes from a string prompt match hashes from tokenized form."""
+ coordinator = make_coordinator_direct()
+ prompt = "1 2 3 4 5 6 7 8"
+ hashes_from_str = coordinator.compute_request_hashes(prompt)
+
+ # DummyTokenizer tokenizes "1 2 3 4 5 6 7 8" -> [1, 2, 3, 4, 5, 6, 7, 8]
+ hashes_from_list = coordinator.compute_request_hashes([1, 2, 3, 4, 5, 6, 7, 8])
+ assert hashes_from_str == hashes_from_list
+
+ def test_hash_empty_when_disabled(self):
+ """Returns empty list when prefix caching is disabled."""
+ coordinator = make_coordinator_direct(enable_prefix_caching=False)
+ hashes = coordinator.compute_request_hashes([1, 2, 3, 4])
+ assert hashes == []
+
+ def test_hash_empty_when_no_block_size(self):
+ """Returns empty list when block_size_tokens is None."""
+ coordinator = make_coordinator_direct(block_size_tokens=None)
+ hashes = coordinator.compute_request_hashes([1, 2, 3, 4])
+ assert hashes == []
+
+ def test_hash_partial_block_ignored(self):
+ """Tokens that don't fill a complete block produce no hash."""
+ coordinator = make_coordinator_direct()
+ hashes = coordinator.compute_request_hashes([1, 2, 3])
+ assert hashes == []
+
+ def test_hash_deterministic(self):
+ """Same tokens always produce the same hashes."""
+ coordinator = make_coordinator_direct()
+ tokens = [10, 20, 30, 40, 50, 60, 70, 80]
+ h1 = coordinator.compute_request_hashes(tokens)
+ h2 = coordinator.compute_request_hashes(tokens)
+ assert h1 == h2
+
+ def test_hash_parent_chaining(self):
+ """Different prefixes produce different hashes even for same block tokens."""
+ coordinator = make_coordinator_direct()
+ # Two prompts share tokens [5,6,7,8] in block 2, but differ in block 1.
+ h1 = coordinator.compute_request_hashes([1, 2, 3, 4, 5, 6, 7, 8])
+ h2 = coordinator.compute_request_hashes([9, 8, 7, 6, 5, 6, 7, 8])
+
+ # Block 1 hashes differ.
+ assert h1[0] != h2[0]
+ # Block 2 hashes also differ due to parent chaining.
+ assert h1[1] != h2[1]
+
+
+class TestCoordinatorPrefixRouting:
+ """Test routing decisions based on prefix cache affinity."""
+
+ def test_no_match_prefers_least_loaded(self):
+ """When no rank has matching hashes, the rank with most free capacity wins."""
+ coordinator = make_coordinator_direct()
+ hashes = coordinator.compute_request_hashes([1, 2, 3, 4])
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # rank_1 has fewer pending requests, so more free capacity.
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 5
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 1
+
+ selected = coordinator.get_best_data_parallel_rank(hashes)
+ assert selected == rank_1
+
+ def test_routes_to_rank_with_longest_match(self):
+ """Request is routed to the rank with the longest consecutive prefix match."""
+ coordinator = make_coordinator_direct()
+ tokens = [1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2]
+ hashes = coordinator.compute_request_hashes(tokens)
+ assert len(hashes) == 3 # 12 tokens / 4 = 3 blocks
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # Ensure no rank is idle so prefix-matching logic is exercised.
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 1
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 1
+
+ # rank_0 has first block only.
+ _set_hash_rank(coordinator, hashes[0], rank_0, 1)
+
+ # rank_1 has first two blocks.
+ _set_hash_rank(coordinator, hashes[0], rank_1, 1)
+ _set_hash_rank(coordinator, hashes[1], rank_1, 1)
+
+ selected = coordinator.get_best_data_parallel_rank(hashes)
+ assert selected == rank_1
+
+ def test_equal_scores_tiebreak_by_rank_index(self):
+ """When two ranks have equal scores, the lower rank index wins."""
+ coordinator = make_coordinator_direct()
+ tokens = [1, 2, 3, 4, 5, 6, 7, 8]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # Both ranks have same pending counts, same match, and same timestamp.
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 1
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 1
+
+ for h in hashes:
+ _set_hash_rank(coordinator, h, rank_0, 1)
+ _set_hash_rank(coordinator, h, rank_1, 1)
+
+ # Equal scores → lowest rank index (rank_0) wins.
+ selected = coordinator.get_best_data_parallel_rank(hashes)
+ assert selected == rank_0
+
+ def test_empty_hashes_uses_round_robin(self):
+ """Empty hash list falls back to round-robin."""
+ coordinator = make_coordinator_direct()
+ for identity in coordinator.identities_of_data_parallel_ranks:
+ coordinator._pending_counts[coordinator.identity_to_rank_index[identity]] = 1
+ rank1 = coordinator.get_best_data_parallel_rank([])
+ rank2 = coordinator.get_best_data_parallel_rank([])
+ assert rank1 != rank2
+
+ def test_disabled_prefix_caching_uses_round_robin(self):
+ """With prefix caching disabled, always uses round-robin."""
+ coordinator = make_coordinator_direct(enable_prefix_caching=False)
+ for identity in coordinator.identities_of_data_parallel_ranks:
+ coordinator._pending_counts[coordinator.identity_to_rank_index[identity]] = 1
+ rank1 = coordinator.get_best_data_parallel_rank([1, 2, 3])
+ rank2 = coordinator.get_best_data_parallel_rank([1, 2, 3])
+ assert rank1 != rank2
+
+
+class TestCoordinatorShadowState:
+ """Test that shadow state (rank_cached_hashes, timestamps) is updated correctly."""
+
+ def test_update_rank_hashes_adds_to_set(self):
+ """_update_rank_hashes adds hashes to the rank's set."""
+ coordinator = make_coordinator_direct()
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ idx_0 = coordinator.identity_to_rank_index[rank_0]
+
+ coordinator._update_rank_hashes(rank_0, [100, 200, 300])
+ assert all(coordinator._hash_table.get(h, {}).get(idx_0, 0) > 0 for h in [100, 200, 300])
+
+ def test_update_rank_hashes_increments_counter(self):
+ """Each call to _update_rank_hashes increments the assignment counter."""
+ coordinator = make_coordinator_direct()
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+
+ assert coordinator._hash_assignment_counter == 0
+ coordinator._update_rank_hashes(rank_0, [100])
+ assert coordinator._hash_assignment_counter == 1
+ coordinator._update_rank_hashes(rank_0, [200])
+ assert coordinator._hash_assignment_counter == 2
+
+ def test_timestamps_updated_on_reassignment(self):
+ """Re-assigning a hash to the same rank updates its timestamp."""
+ coordinator = make_coordinator_direct()
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ idx_0 = coordinator.identity_to_rank_index[rank_0]
+
+ coordinator._update_rank_hashes(rank_0, [100])
+ ts1 = coordinator._hash_table[100][idx_0]
+
+ coordinator._update_rank_hashes(rank_0, [100])
+ ts2 = coordinator._hash_table[100][idx_0]
+
+ assert ts2 > ts1
+
+ def test_multiple_requests_accumulate_hashes(self):
+ """Multiple requests to the same rank accumulate their hashes."""
+ coordinator = make_coordinator_direct()
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ idx_0 = coordinator.identity_to_rank_index[rank_0]
+
+ coordinator._update_rank_hashes(rank_0, [10, 20])
+ coordinator._update_rank_hashes(rank_0, [30, 40])
+ assert all(coordinator._hash_table.get(h, {}).get(idx_0, 0) > 0 for h in [10, 20, 30, 40])
+
+ def test_hash_can_appear_in_multiple_ranks(self):
+ """The same hash can be owned by multiple ranks."""
+ coordinator = make_coordinator_direct()
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+ idx_0 = coordinator.identity_to_rank_index[rank_0]
+ idx_1 = coordinator.identity_to_rank_index[rank_1]
+
+ coordinator._update_rank_hashes(rank_0, [100])
+ coordinator._update_rank_hashes(rank_1, [100])
+
+ assert coordinator._hash_table[100].get(idx_0, 0) > 0
+ assert coordinator._hash_table[100].get(idx_1, 0) > 0
+
+ def test_routing_then_state_update_flow(self):
+ """Full flow: compute hashes, route, update state, then re-route to same rank."""
+ coordinator = make_coordinator_direct()
+
+ tokens = [1, 2, 3, 4, 5, 6, 7, 8]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ # First request: no matches, round-robin.
+ rank = coordinator.get_best_data_parallel_rank(hashes)
+ coordinator._update_rank_hashes(rank, hashes)
+
+ # Second request with same tokens: should go to same rank.
+ rank2 = coordinator.get_best_data_parallel_rank(hashes)
+ assert rank2 == rank
+
+ def test_recency_breaks_tie_at_equal_load(self):
+ """When two ranks match the same prefix and have equal load, the more
+ recently assigned rank wins."""
+ coordinator = make_coordinator_direct()
+ tokens = [1, 2, 3, 4, 5, 6, 7, 8]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # Equal load on both ranks.
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 1
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 1
+
+ # Both have the prefix, but rank_1 was assigned more recently.
+ for h in hashes:
+ _set_hash_rank(coordinator, h, rank_0, 1)
+ _set_hash_rank(coordinator, h, rank_1, 5)
+
+ # rank_1 wins via recency despite rank_0 having a lower index.
+ selected = coordinator.get_best_data_parallel_rank(hashes)
+ assert selected == rank_1
+
+
+@pytest.mark.skipif(ZMQ_FLAKY_SHUTDOWN, reason="ZMQ shutdown is flaky")
+class TestCoordinatorEndToEnd:
+ """End-to-end test with real ZMQ sockets and DummyEngines."""
+
+ async def run_coordinator_test(
+ self, requests, block_size_tokens=BLOCK_SIZE, enable_prefix_caching=True
+ ):
+ """Submit requests through a real coordinator and return results."""
+ engine = DummyEngine()
+
+ dp_addr = await engine.start_listening_to_data_parallel_coordinator(
+ inference_coordinator_port=DEFAULT_PORT, launch_inference_coordinator=True
+ )
+
+ try:
+ if torch.distributed.get_rank() == 0:
+ client = InferenceClient(dp_addr)
+ client.start()
+
+ futures = [
+ client.add_request(prompt=prompt, sampling_params=params)
+ for prompt, params in requests
+ ]
+ results = await asyncio.wait_for(asyncio.gather(*futures), timeout=10.0)
+
+ for record in results:
+ assert record[-1].status == Status.COMPLETED
+ finally:
+ if torch.distributed.get_rank() == 0:
+ await asyncio.wait_for(client.stop_engines(), timeout=10.0)
+ client.stop()
+ try:
+ await asyncio.wait_for(engine.engine_loop_task, timeout=30.0)
+ except asyncio.TimeoutError:
+ engine.engine_loop_task.cancel()
+
+ @pytest.mark.internal
+ @pytest.mark.skipif(not HAVE_ZMQ, reason="pyzmq is required")
+ @pytest.mark.asyncio
+ async def test_shared_prefix_requests(self, initialize_model_parallel):
+ """Requests with shared prefixes complete successfully through the coordinator."""
+ requests = [
+ ("1 2 3 4 5 6 7 8", SamplingParams(num_tokens_to_generate=2)),
+ ("1 2 3 4 9 8 7 6", SamplingParams(num_tokens_to_generate=2)),
+ ("1 2 3 4 5 6 7 8", SamplingParams(num_tokens_to_generate=2)),
+ ]
+ await self.run_coordinator_test(requests)
+
+
+def make_first_prefix_block_coordinator(**kwargs):
+ """Create a coordinator configured with FIRST_PREFIX_BLOCK policy."""
+ coordinator = make_coordinator_direct(**kwargs)
+ coordinator.prefix_caching_coordinator_policy = (
+ PrefixCachingCoordinatorPolicy.FIRST_PREFIX_BLOCK
+ )
+ return coordinator
+
+
+class TestFirstPrefixBlockRouting:
+ """Test routing decisions using the FIRST_PREFIX_BLOCK policy."""
+
+ def test_first_block_match_routes_to_rank(self):
+ """Request is routed to the rank that has the first block cached."""
+ coordinator = make_first_prefix_block_coordinator()
+ tokens = [1, 2, 3, 4, 5, 6, 7, 8]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # Ensure no rank is idle so prefix-matching logic is exercised.
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 1
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 1
+
+ # Only rank_1 has the first block.
+ _set_hash_rank(coordinator, hashes[0], rank_1, 1)
+
+ selected = coordinator.get_best_data_parallel_rank(hashes[:1])
+ assert selected == rank_1
+
+ def test_first_block_ignores_longer_match(self):
+ """Rank with more blocks cached is not preferred; only first block matters."""
+ coordinator = make_first_prefix_block_coordinator()
+ tokens = [1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2]
+ hashes = coordinator.compute_request_hashes(tokens)
+ assert len(hashes) == 3
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # Ensure no rank is idle so prefix-matching logic is exercised.
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 1
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 1
+
+ # rank_0 has first block only, with higher timestamp.
+ _set_hash_rank(coordinator, hashes[0], rank_0, 10)
+
+ # rank_1 has all three blocks, but lower timestamp on first block.
+ for h in hashes:
+ _set_hash_rank(coordinator, h, rank_1, 1)
+
+ # rank_0 wins because it has higher recency on the first block.
+ # Caller truncates to [:1] before calling get_best_data_parallel_rank.
+ selected = coordinator.get_best_data_parallel_rank(hashes[:1])
+ assert selected == rank_0
+
+ def test_first_block_equal_match_tiebreaks_by_rank_index(self):
+ """When multiple ranks have the first block with equal load, lowest index wins."""
+ coordinator = make_first_prefix_block_coordinator()
+ tokens = [1, 2, 3, 4, 5, 6, 7, 8]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 1
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 1
+
+ # Both ranks have the first block with the same timestamp.
+ _set_hash_rank(coordinator, hashes[0], rank_0, 3)
+ _set_hash_rank(coordinator, hashes[0], rank_1, 3)
+
+ # Equal scores → lowest rank index wins.
+ selected = coordinator.get_best_data_parallel_rank(hashes[:1])
+ assert selected == rank_0
+
+ def test_no_first_block_match_prefers_least_loaded(self):
+ """When no rank has the first block, the least loaded rank wins."""
+ coordinator = make_first_prefix_block_coordinator()
+ tokens = [1, 2, 3, 4, 5, 6, 7, 8]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # rank_0 has block 1 (second block), but not block 0.
+ _set_hash_rank(coordinator, hashes[1], rank_0, 1)
+
+ # rank_1 has fewer pending requests.
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 5
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 1
+
+ # No rank has the first block → load determines winner.
+ selected = coordinator.get_best_data_parallel_rank(hashes[:1])
+ assert selected == rank_1
+
+ def test_first_block_policy_with_single_block_prompt(self):
+ """Works correctly with a prompt that has only one block."""
+ coordinator = make_first_prefix_block_coordinator()
+ tokens = [1, 2, 3, 4]
+ hashes = coordinator.compute_request_hashes(tokens)
+ assert len(hashes) == 1
+
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # Ensure no rank is idle so prefix-matching logic is exercised.
+ for identity in coordinator.identities_of_data_parallel_ranks:
+ coordinator._pending_counts[coordinator.identity_to_rank_index[identity]] = 1
+
+ _set_hash_rank(coordinator, hashes[0], rank_1, 1)
+
+ selected = coordinator.get_best_data_parallel_rank(hashes[:1])
+ assert selected == rank_1
+
+
+class TestLoadAwarePrefixRouting:
+ """Test that prefix routing spreads load across ranks with the same prefix."""
+
+ def test_spreads_across_ranks_with_same_prefix(self):
+ """When three ranks all cache the same prefix, requests spread by load."""
+ coordinator = make_coordinator_direct(data_parallel_size=3)
+ tokens = [1, 2, 3, 4, 5, 6, 7, 8]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+ rank_2 = coordinator.identities_of_data_parallel_ranks[2]
+
+ # All three ranks have both blocks cached with the same timestamp.
+ for h in hashes:
+ _set_hash_rank(coordinator, h, rank_0, 1)
+ _set_hash_rank(coordinator, h, rank_1, 1)
+ _set_hash_rank(coordinator, h, rank_2, 1)
+
+ # Simulate sending 6 requests. With load-aware routing, they should
+ # spread across ranks rather than all going to one.
+ assigned_ranks = []
+ for _ in range(6):
+ rank = coordinator.get_best_data_parallel_rank(hashes)
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank]] += 1
+ assigned_ranks.append(rank)
+
+ # Each rank should get exactly 2 of the 6 requests.
+ from collections import Counter
+
+ counts = Counter(assigned_ranks)
+ assert counts[rank_0] == 2
+ assert counts[rank_1] == 2
+ assert counts[rank_2] == 2
+
+ def test_load_overrides_recency(self):
+ """A rank with a higher timestamp but more pending requests is not preferred."""
+ coordinator = make_coordinator_direct()
+ tokens = [1, 2, 3, 4]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # Both ranks have the prefix. rank_1 has a higher (more recent) timestamp.
+ _set_hash_rank(coordinator, hashes[0], rank_0, 1)
+ _set_hash_rank(coordinator, hashes[0], rank_1, 10)
+
+ # But rank_1 already has 5 pending requests while rank_0 has only 1.
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 1
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 5
+
+ selected = coordinator.get_best_data_parallel_rank(hashes)
+ assert selected == rank_0
+
+ def test_pending_count_decremented_on_completion(self):
+ """Completing a request frees capacity on the assigned rank."""
+ coordinator = make_coordinator_direct()
+ tokens = [1, 2, 3, 4]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ _set_hash_rank(coordinator, hashes[0], rank_0, 1)
+ _set_hash_rank(coordinator, hashes[0], rank_1, 1)
+
+ # Simulate assigning a request to rank_0.
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 1
+ coordinator.request_id_to_rank = {42: rank_0}
+
+ # Simulate completion: decrement pending count.
+ assigned_rank = coordinator.request_id_to_rank.pop(42, None)
+ if assigned_rank is not None:
+ idx = coordinator.identity_to_rank_index.get(assigned_rank)
+ if idx is not None:
+ coordinator._pending_counts[idx] = max(0, coordinator._pending_counts[idx] - 1)
+
+ assert coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] == 0
+
+ def test_equal_load_tiebreaks_by_rank_index(self):
+ """With equal pending counts and match, lowest rank index wins."""
+ coordinator = make_coordinator_direct()
+ tokens = [1, 2, 3, 4]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # Equal pending counts, both have the prefix cached.
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 1
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 1
+ _set_hash_rank(coordinator, hashes[0], rank_0, 10)
+ _set_hash_rank(coordinator, hashes[0], rank_1, 1)
+
+ # Equal scores → lowest rank index (rank_0) wins.
+ selected = coordinator.get_best_data_parallel_rank(hashes)
+ assert selected == rank_0
+
+
+class TestScoringFunctionRouting:
+ """Test the alpha-based scoring function: score = alpha * match + (1 - alpha) * normalized_load."""
+
+ def test_high_alpha_prefers_prefix_match(self):
+ """With alpha=1.0, a rank with a prefix hit is always preferred over a free rank."""
+ coordinator = make_coordinator_direct(prefix_caching_routing_alpha=1.0, max_requests=10)
+ tokens = [1, 2, 3, 4]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # rank_0 has the prefix but is heavily loaded (9/10 slots used).
+ _set_hash_rank(coordinator, hashes[0], rank_0, 1)
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 9
+
+ # rank_1 has no prefix match but is idle.
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 0
+
+ # alpha=1.0: score(rank_0) = 1*1 + 0*0.1 = 1.0
+ # score(rank_1) = 1*0 + 0*1.0 = 0.0
+ selected = coordinator.get_best_data_parallel_rank(hashes)
+ assert selected == rank_0
+
+ def test_low_alpha_prefers_free_capacity(self):
+ """With alpha=0.0, the rank with the most free capacity is preferred."""
+ coordinator = make_coordinator_direct(prefix_caching_routing_alpha=0.0, max_requests=10)
+ tokens = [1, 2, 3, 4]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # rank_0 has the prefix but is heavily loaded.
+ _set_hash_rank(coordinator, hashes[0], rank_0, 1)
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 8
+
+ # rank_1 has no prefix match but is nearly idle.
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 1
+
+ # alpha=0.0: score(rank_0) = 0*1 + 1*(2/10) = 0.2
+ # score(rank_1) = 0*0 + 1*(9/10) = 0.9
+ selected = coordinator.get_best_data_parallel_rank(hashes)
+ assert selected == rank_1
+
+ def test_balanced_alpha_trades_off(self):
+ """With alpha=0.5, prefix match and load are balanced."""
+ coordinator = make_coordinator_direct(prefix_caching_routing_alpha=0.5, max_requests=10)
+ tokens = [1, 2, 3, 4]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # rank_0 has prefix match, 7 pending (3 free).
+ _set_hash_rank(coordinator, hashes[0], rank_0, 1)
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 7
+
+ # rank_1 has no prefix match, 0 pending (10 free).
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 0
+
+ # alpha=0.5: score(rank_0) = 0.5*1 + 0.5*(3/10) = 0.5 + 0.15 = 0.65
+ # score(rank_1) = 0.5*0 + 0.5*(10/10) = 0.0 + 0.5 = 0.5
+ selected = coordinator.get_best_data_parallel_rank(hashes)
+ assert selected == rank_0
+
+ def test_balanced_alpha_prefers_free_when_heavily_loaded(self):
+ """With alpha=0.5, a completely free rank beats a nearly-full rank with prefix match."""
+ coordinator = make_coordinator_direct(prefix_caching_routing_alpha=0.5, max_requests=10)
+ tokens = [1, 2, 3, 4]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # rank_0 has prefix match, 10 pending (0 free).
+ _set_hash_rank(coordinator, hashes[0], rank_0, 1)
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 10
+
+ # rank_1 has no prefix match, 0 pending (10 free).
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 0
+
+ # alpha=0.5: score(rank_0) = 0.5*1 + 0.5*(0/10) = 0.5
+ # score(rank_1) = 0.5*0 + 0.5*(10/10) = 0.5
+ # Tie broken by rank index: rank_0 has lower index.
+ selected = coordinator.get_best_data_parallel_rank(hashes)
+ assert selected == rank_0
+
+ def test_scoring_tiebreak_by_rank_index(self):
+ """When scores are equal, the rank with lower index is preferred."""
+ coordinator = make_coordinator_direct(prefix_caching_routing_alpha=0.5, max_requests=10)
+ tokens = [1, 2, 3, 4]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # Both ranks have prefix match and same load.
+ _set_hash_rank(coordinator, hashes[0], rank_0, 1)
+ _set_hash_rank(coordinator, hashes[0], rank_1, 1)
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 5
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 5
+
+ selected = coordinator.get_best_data_parallel_rank(hashes)
+ assert selected == rank_0
+
+ def test_scoring_spreads_load_across_ranks(self):
+ """Scoring function distributes requests when all ranks have prefix match."""
+ coordinator = make_coordinator_direct(
+ data_parallel_size=3, prefix_caching_routing_alpha=0.5, max_requests=10
+ )
+ tokens = [1, 2, 3, 4, 5, 6, 7, 8]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+ rank_2 = coordinator.identities_of_data_parallel_ranks[2]
+
+ # All three ranks have both blocks cached.
+ for h in hashes:
+ _set_hash_rank(coordinator, h, rank_0, 1)
+ _set_hash_rank(coordinator, h, rank_1, 1)
+ _set_hash_rank(coordinator, h, rank_2, 1)
+
+ # Simulate sending 6 requests.
+ assigned_ranks = []
+ for _ in range(6):
+ rank = coordinator.get_best_data_parallel_rank(hashes)
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank]] += 1
+ assigned_ranks.append(rank)
+
+ from collections import Counter
+
+ counts = Counter(assigned_ranks)
+ # Each rank should get exactly 2 of the 6 requests.
+ assert counts[rank_0] == 2
+ assert counts[rank_1] == 2
+ assert counts[rank_2] == 2
+
+ def test_scoring_with_no_prefix_match_anywhere(self):
+ """When no rank has a prefix match, load alone determines the winner."""
+ coordinator = make_coordinator_direct(prefix_caching_routing_alpha=0.5, max_requests=10)
+ tokens = [1, 2, 3, 4]
+ hashes = coordinator.compute_request_hashes(tokens)
+
+ rank_0 = coordinator.identities_of_data_parallel_ranks[0]
+ rank_1 = coordinator.identities_of_data_parallel_ranks[1]
+
+ # No prefix matches for either rank.
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_0]] = 5
+ coordinator._pending_counts[coordinator.identity_to_rank_index[rank_1]] = 2
+
+ # alpha=0.5: score(rank_0) = 0 + 0.5*(5/10) = 0.25
+ # score(rank_1) = 0 + 0.5*(8/10) = 0.4
+ selected = coordinator.get_best_data_parallel_rank(hashes)
+ assert selected == rank_1
diff --git a/tests/unit_tests/inference/test_inference_config.py b/tests/unit_tests/inference/test_inference_config.py
new file mode 100644
index 00000000000..6d58328dade
--- /dev/null
+++ b/tests/unit_tests/inference/test_inference_config.py
@@ -0,0 +1,17 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+import dataclasses
+
+from megatron.core.inference.config import InferenceConfig
+from megatron.core.transformer.transformer_config import TransformerConfig
+
+
+class TestInferenceConfig:
+ def test_mutual_exclusivity_with_transformer_config(self):
+ """
+ Ensure mutual exclusivity between fields in `InferenceConfig` and
+ `TransformerConfig`.
+ """
+ dynamic_inference_config_fields = set(dataclasses.fields(InferenceConfig))
+ transformer_config_fields = set(dataclasses.fields(TransformerConfig))
+ assert len(dynamic_inference_config_fields.intersection(transformer_config_fields)) == 0
diff --git a/tests/unit_tests/inference/test_moe_inference.py b/tests/unit_tests/inference/test_moe_inference.py
new file mode 100644
index 00000000000..b762b5e638c
--- /dev/null
+++ b/tests/unit_tests/inference/test_moe_inference.py
@@ -0,0 +1,388 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+
+"""Unit tests for inference-optimized MoE components.
+
+Config is modeled after nanov3 (Nemotron-6 3B Hybrid MoE) with smaller
+dimensions for fast unit test execution:
+- squared_relu activation (not swiglu/gated)
+- sigmoid router score function with expert bias
+- topk=6, topk_scaling_factor=2.5
+- shared experts
+"""
+
+import pytest
+import torch
+
+from megatron.core.activations import squared_relu
+from megatron.core.inference.communication.torch_symm_triton import are_tensors_nvls_eligible
+from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.utils import is_te_min_version, is_torch_min_version
+from megatron.training.initialize import _set_random_seed
+from tests.unit_tests.test_utilities import Utils
+
+# Reusable skip decorators
+requires_te = pytest.mark.skipif(
+ not is_te_min_version("1.7.0.dev0"), reason="Requires transformer-engine >= 1.7.0"
+)
+requires_torch_grouped_mm = pytest.mark.skipif(
+ not is_torch_min_version("2.10") or not hasattr(torch, '_grouped_mm'),
+ reason="Requires PyTorch >= 2.10 with torch._grouped_mm",
+)
+
+# ──────────────────────────────────────────────────────────────────────
+# NanoV3-like config (scaled down from 2688→128 hidden, 128→8 experts)
+# ──────────────────────────────────────────────────────────────────────
+
+NANOV3_BASE = dict(
+ num_layers=1,
+ hidden_size=128,
+ ffn_hidden_size=128,
+ num_attention_heads=4,
+ num_query_groups=2,
+ num_moe_experts=8,
+ moe_ffn_hidden_size=128,
+ moe_router_topk=6,
+ moe_router_score_function="sigmoid",
+ moe_router_enable_expert_bias=True,
+ moe_router_topk_scaling_factor=2.5,
+ moe_shared_expert_intermediate_size=256,
+ moe_router_dtype='fp32',
+ moe_shared_expert_overlap=False,
+ moe_grouped_gemm=True,
+ moe_token_dispatcher_type="alltoall",
+ moe_aux_loss_coeff=0.01,
+ activation_func=squared_relu,
+ normalization="RMSNorm",
+ add_bias_linear=False,
+ bf16=True,
+ params_dtype=torch.bfloat16,
+ transformer_impl="inference_optimized",
+)
+
+
+def _make_base_config(**overrides):
+ """Create a TransformerConfig with nanov3-like defaults."""
+ params = {**NANOV3_BASE, **overrides}
+ return TransformerConfig(**params)
+
+
+# ──────────────────────────────────────────────────────────────────────
+# InferenceTopKRouter
+# ──────────────────────────────────────────────────────────────────────
+
+
+@pytest.mark.internal
+class TestInferenceTopKRouter:
+
+ @classmethod
+ def setup_class(cls):
+ Utils.initialize_model_parallel(1, 1)
+ _set_random_seed(seed_=123, data_parallel_random_init=False)
+
+ @classmethod
+ def teardown_class(cls):
+ Utils.destroy_model_parallel()
+
+ def _make_router(self, **config_overrides):
+ from megatron.core.transformer.moe.moe_utils import get_default_pg_collection
+ from megatron.core.transformer.moe.router import InferenceTopKRouter
+
+ config = _make_base_config(**config_overrides)
+ return (
+ InferenceTopKRouter(config=config, pg_collection=get_default_pg_collection())
+ .cuda()
+ .to(torch.bfloat16)
+ )
+
+ def test_init_rejects_num_groups(self):
+ """InferenceTopKRouter requires moe_router_num_groups=None."""
+ with pytest.raises(AssertionError, match="moe_router_num_groups"):
+ self._make_router(moe_router_num_groups=2)
+
+ def test_config_rejects_non_fp32_router_dtype(self):
+ """inference_optimized config requires moe_router_dtype='fp32'."""
+ with pytest.raises(ValueError, match="moe-router-dtype"):
+ _make_base_config(
+ transformer_impl="inference_optimized", add_qkv_bias=False, moe_router_dtype=None
+ )
+
+ @pytest.mark.parametrize("score_fn", ["none", "invalid"])
+ def test_init_rejects_unsupported_score_function(self, score_fn):
+ """InferenceTopKRouter requires sigmoid or softmax score function."""
+ with pytest.raises(AssertionError, match="moe_router_score_function"):
+ self._make_router(
+ moe_router_score_function=score_fn, moe_router_enable_expert_bias=False
+ )
+
+ @pytest.mark.parametrize("score_fn", ["sigmoid", "softmax"])
+ def test_init_accepts_valid_score_function(self, score_fn):
+ """InferenceTopKRouter accepts sigmoid and softmax."""
+ # Expert bias only valid with sigmoid; disable it for softmax
+ enable_bias = score_fn == "sigmoid"
+ router = self._make_router(
+ moe_router_score_function=score_fn, moe_router_enable_expert_bias=enable_bias
+ )
+ assert router is not None
+
+ def test_set_unset_inference_mode(self):
+ """Toggle is_inference_cuda_graphed_iteration flag."""
+ router = self._make_router()
+ assert not router.is_inference_cuda_graphed_iteration
+
+ router.set_inference_cuda_graphed_iteration()
+ assert router.is_inference_cuda_graphed_iteration
+
+ router.unset_inference_cuda_graphed_iteration()
+ assert not router.is_inference_cuda_graphed_iteration
+
+ def test_training_mode_forward_returns_sparse(self):
+ """In training mode, forward delegates to parent and returns sparse tensors."""
+ router = self._make_router()
+ router.train()
+ num_tokens = 16
+ num_experts = NANOV3_BASE["num_moe_experts"]
+
+ input_tensor = torch.randn(
+ num_tokens, NANOV3_BASE["hidden_size"], device="cuda", dtype=torch.bfloat16
+ )
+ probs, routing_map = router(input_tensor)
+
+ # Parent TopKRouter returns [num_tokens, num_experts] sparse routing_map
+ assert routing_map.shape == (num_tokens, num_experts)
+
+ def test_inference_vs_training_selects_same_experts(self):
+ """Inference and training modes should select the same top-k experts."""
+ router = self._make_router()
+ num_tokens = 16
+ topk = NANOV3_BASE["moe_router_topk"]
+
+ input_tensor = torch.randn(
+ num_tokens, NANOV3_BASE["hidden_size"], device="cuda", dtype=torch.bfloat16
+ )
+
+ # Training mode: get routing_map (sparse) and extract top expert indices
+ router.train()
+ _, routing_map = router(input_tensor.clone())
+ # routing_map is [num_tokens, num_experts] bool
+ training_experts = set()
+ for i in range(num_tokens):
+ experts_for_token = routing_map[i].nonzero(as_tuple=True)[0]
+ for e in experts_for_token:
+ training_experts.add((i, e.item()))
+
+ # Inference mode: get top_indices (dense)
+ router.eval()
+ router.set_inference_cuda_graphed_iteration()
+ _, top_indices = router(input_tensor.clone())
+
+ inference_experts = set()
+ for i in range(num_tokens):
+ for k in range(topk):
+ inference_experts.add((i, top_indices[i, k].item()))
+
+ # Same expert selections
+ assert training_experts == inference_experts
+
+
+# ──────────────────────────────────────────────────────────────────────
+# InferenceCUDAGraphTokenDispatcher
+# ──────────────────────────────────────────────────────────────────────
+
+
+@pytest.mark.internal
+class TestInferenceCUDAGraphTokenDispatcher:
+
+ @classmethod
+ def setup_class(cls):
+ Utils.initialize_model_parallel(1, 1, expert_model_parallel_size=Utils.world_size)
+ _set_random_seed(seed_=123, data_parallel_random_init=False)
+
+ @classmethod
+ def teardown_class(cls):
+ from megatron.core.inference.symmetric_memory import SymmetricMemoryManager
+
+ SymmetricMemoryManager.destroy()
+ Utils.destroy_model_parallel()
+
+ def _make_dispatcher(self, **config_overrides):
+ from megatron.core.transformer.moe.moe_utils import get_default_pg_collection
+ from megatron.core.transformer.moe.token_dispatcher_inference import (
+ InferenceCUDAGraphTokenDispatcher,
+ )
+
+ config_overrides.setdefault("expert_model_parallel_size", Utils.world_size)
+ config = _make_base_config(**config_overrides)
+ num_local_experts = config.num_moe_experts // Utils.world_size
+ ep_rank = torch.distributed.get_rank() if Utils.world_size > 1 else 0
+ local_expert_indices = [ep_rank * num_local_experts + i for i in range(num_local_experts)]
+
+ return InferenceCUDAGraphTokenDispatcher(
+ num_local_experts=num_local_experts,
+ local_expert_indices=local_expert_indices,
+ config=config,
+ pg_collection=get_default_pg_collection(),
+ )
+
+ def test_init(self):
+ """Dispatcher can be constructed with nanov3-like config and EP=world_size."""
+ dispatcher = self._make_dispatcher()
+ assert dispatcher.topk == NANOV3_BASE["moe_router_topk"]
+ assert dispatcher.ep_size == Utils.world_size
+
+ def test_symmetric_memory_buffer_initialized(self):
+ """EP symmetric memory buffer is lazily created via SymmetricMemoryManager."""
+ from megatron.core.inference.symmetric_memory import SymmetricMemoryManager
+
+ # Buffer should not exist yet (lazy init)
+ assert not SymmetricMemoryManager.is_initialized("ep")
+
+ # Create it explicitly and verify
+ from megatron.core.parallel_state import get_expert_model_parallel_group
+
+ buf = SymmetricMemoryManager.get_buffer(
+ "ep", process_group=get_expert_model_parallel_group()
+ )
+ assert buf is not None
+ assert SymmetricMemoryManager.is_initialized("ep")
+
+ @pytest.mark.parametrize("seed", [42, 123, 7])
+ @pytest.mark.parametrize(
+ "num_local_tokens",
+ [
+ 1,
+ 2,
+ 4,
+ 8,
+ 16,
+ 24,
+ 32,
+ 40,
+ 48,
+ 56,
+ 64,
+ 72,
+ 80,
+ 88,
+ 96,
+ 104,
+ 112,
+ 120,
+ 128,
+ 136,
+ 144,
+ 152,
+ 160,
+ 168,
+ 176,
+ 184,
+ 192,
+ 200,
+ 208,
+ 216,
+ 224,
+ 232,
+ 240,
+ 248,
+ 256,
+ 272,
+ 288,
+ 304,
+ 320,
+ 336,
+ 352,
+ 368,
+ 384,
+ 400,
+ 416,
+ 432,
+ 448,
+ 464,
+ 480,
+ 496,
+ 512,
+ ],
+ )
+ def test_cuda_graph_dispatch_combine(self, num_local_tokens, seed):
+ """Dispatch+combine can be captured in a CUDA graph and replayed.
+ Creates global buffers, shards per rank, and verifies:
+ - NVLS AllGather output matches the full globalwol buffer
+ - NVLS ReduceScatter output matches fp32-accumulated reference
+ All tensor byte sizes are 128-bit aligned for NVLS eligibility.
+ """
+
+ torch.manual_seed(seed)
+ torch.cuda.manual_seed(seed)
+
+ dispatcher = self._make_dispatcher()
+ ep_size = dispatcher.ep_size
+ hidden_size = NANOV3_BASE["hidden_size"]
+ topk = NANOV3_BASE["moe_router_topk"]
+ num_experts = NANOV3_BASE["num_moe_experts"]
+ rank = torch.distributed.get_rank() if ep_size > 1 else 0
+ num_global_tokens = num_local_tokens * ep_size
+
+ # Create global buffers on rank 0 and broadcast to all ranks
+ global_hidden = torch.randn(
+ num_global_tokens, hidden_size, device="cuda", dtype=torch.bfloat16
+ )
+ global_probs = torch.randn(num_global_tokens, topk, device="cuda", dtype=torch.float32)
+ global_routing_map = torch.randint(0, num_experts, (num_global_tokens, topk), device="cuda")
+ torch.distributed.broadcast(global_hidden, src=0)
+ torch.distributed.broadcast(global_probs, src=0)
+ torch.distributed.broadcast(global_routing_map, src=0)
+
+ # Each rank grabs their own shard
+ start = rank * num_local_tokens
+ end = start + num_local_tokens
+ static_hidden = global_hidden[start:end].contiguous()
+ static_probs = global_probs[start:end].contiguous()
+ static_routing_map = global_routing_map[start:end].contiguous()
+
+ if not are_tensors_nvls_eligible(static_hidden, static_probs, static_routing_map):
+ pytest.skip(
+ "Tensors are not NVLS-eligible (need SM>=9 and each tensor's memory to be a multiple of 16 bytes)"
+ )
+
+ # 3 warmup iterations on a side stream
+ with torch.no_grad():
+ s = torch.cuda.Stream()
+ s.wait_stream(torch.cuda.current_stream())
+ with torch.cuda.stream(s):
+ for _ in range(3):
+ dispatcher.routing_map = static_routing_map
+ d_hidden, d_probs = dispatcher.token_dispatch(static_hidden, static_probs)
+ d_hidden = d_hidden.clone()
+ d_probs = d_probs.clone()
+ dispatcher.routing_map = dispatcher.routing_map.clone()
+ dispatcher.token_combine(d_hidden.clone())
+ torch.cuda.current_stream().wait_stream(s)
+
+ # Capture
+ graph = torch.cuda.CUDAGraph()
+ with torch.cuda.graph(graph):
+ dispatcher.routing_map = static_routing_map
+ d_hidden, d_probs = dispatcher.token_dispatch(static_hidden, static_probs)
+ graph_hidden = d_hidden.clone()
+ graph_probs = d_probs.clone()
+ graph_routing_map = dispatcher.routing_map.clone()
+ graph_combined = dispatcher.token_combine(d_hidden.clone())
+
+ # Verify shapes: dispatch expands by ep_size, combine shrinks back
+ assert graph_hidden.shape == (num_global_tokens, hidden_size)
+ assert graph_probs.shape == (num_global_tokens, topk)
+ assert graph_combined.shape == (num_local_tokens, hidden_size)
+
+ # Replay
+ graph.replay()
+
+ # Verify AllGather: all gathered tensors should match global buffers
+ torch.testing.assert_close(graph_hidden, global_hidden, atol=0, rtol=0)
+ torch.testing.assert_close(graph_probs, global_probs, atol=0, rtol=0)
+ torch.testing.assert_close(graph_routing_map, global_routing_map, atol=0, rtol=0)
+
+ # Verify ReduceScatter: all ranks have the same all-gathered data, so
+ # rank r gets ep_size * chunk_r. Compute reference in fp32 then downcast.
+ # Exact match (atol=0, rtol=0) is possible because the NVLS triton kernels
+ # accumulate in fp32 before writing bf16 output.
+ expected_combined = (global_hidden[start:end].float() * ep_size).bfloat16()
+ torch.testing.assert_close(graph_combined, expected_combined, atol=0, rtol=0)
diff --git a/tests/unit_tests/inference/test_moe_permute.py b/tests/unit_tests/inference/test_moe_permute.py
new file mode 100644
index 00000000000..4664d0fa2cd
--- /dev/null
+++ b/tests/unit_tests/inference/test_moe_permute.py
@@ -0,0 +1,446 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+
+"""Unit tests for megatron.core.inference.moe.permute.
+
+Tests cover:
+- compute_local_tokens_per_expert: token counting against PyTorch reference
+- compute_expert_offsets: prefix sums with and without alignment
+- permute_tokens: expert grouping, data integrity, alignment padding
+- unpermute_tokens: weighted scatter-back, fp32 accumulation
+- permute -> unpermute roundtrip
+"""
+
+import pytest
+import torch
+
+
+def _ref_tokens_per_expert(routing_map, local_expert_start, num_local_experts):
+ """PyTorch reference for compute_local_tokens_per_expert."""
+ counts = torch.zeros(num_local_experts, dtype=torch.int32, device=routing_map.device)
+ for eid in routing_map.flatten():
+ lid = eid.item() - local_expert_start
+ if 0 <= lid < num_local_experts:
+ counts[lid] += 1
+ return counts
+
+
+def _ref_expert_offsets(tokens_per_expert, alignment):
+ """PyTorch reference for compute_expert_offsets."""
+ aligned = tokens_per_expert.clone().to(torch.int32)
+ for i in range(len(aligned)):
+ if aligned[i] > 0 and alignment > 1:
+ aligned[i] = ((aligned[i] + alignment - 1) // alignment) * alignment
+ inc = torch.cumsum(aligned, dim=0)
+ exc = inc - aligned
+ return exc.to(torch.int32), inc.to(torch.int32)
+
+
+def _make_inputs(num_tokens, hidden_dim, topk, num_experts, seed=42):
+ """Create random hidden states, probs, and routing_map."""
+ torch.manual_seed(seed)
+ hidden = torch.randn(num_tokens, hidden_dim, device="cuda", dtype=torch.bfloat16)
+ probs = torch.rand(num_tokens, topk, device="cuda", dtype=torch.float32)
+ routing_map = torch.randint(0, num_experts, (num_tokens, topk), device="cuda")
+ return hidden, probs, routing_map
+
+
+@pytest.mark.internal
+class TestComputeLocalTokensPerExpert:
+
+ @pytest.mark.parametrize("num_tokens", [1, 4, 16, 64, 128, 256, 512])
+ @pytest.mark.parametrize("topk", [1, 2, 4, 6, 8])
+ @pytest.mark.parametrize(
+ "num_experts,num_local,start",
+ [
+ (4, 4, 0), # all local, small expert count
+ (8, 8, 0), # all local (EP=1)
+ (8, 4, 0), # first half local (EP=2, rank 0)
+ (8, 4, 4), # second half local (EP=2, rank 1)
+ (8, 2, 2), # middle slice (EP=4, rank 1)
+ (8, 1, 7), # single expert local (EP=8, last rank)
+ (32, 8, 0), # 32 experts, first 8 local
+ (32, 8, 24), # 32 experts, last 8 local
+ (128, 32, 0), # 128 experts, first 32 local (EP=4, rank 0)
+ (128, 32, 96), # 128 experts, last 32 local (EP=4, rank 3)
+ ],
+ )
+ def test_matches_reference(self, num_tokens, topk, num_experts, num_local, start):
+ from megatron.core.inference.moe.permute import compute_local_tokens_per_expert
+
+ routing_map = torch.randint(0, num_experts, (num_tokens, topk), device="cuda")
+ result = compute_local_tokens_per_expert(routing_map, start, num_local)
+ expected = _ref_tokens_per_expert(routing_map, start, num_local)
+ torch.testing.assert_close(result, expected, atol=0, rtol=0)
+
+ def test_no_local_tokens(self):
+ """All tokens routed to non-local experts -> all zeros."""
+ from megatron.core.inference.moe.permute import compute_local_tokens_per_expert
+
+ routing_map = torch.full((16, 4), 99, dtype=torch.int64, device="cuda")
+ result = compute_local_tokens_per_expert(routing_map, 0, 8)
+ assert result.sum().item() == 0
+
+ def test_single_expert_all_tokens(self):
+ """All token-topk pairs route to a single local expert."""
+ from megatron.core.inference.moe.permute import compute_local_tokens_per_expert
+
+ num_tokens, topk, num_local = 32, 4, 8
+ routing_map = torch.full((num_tokens, topk), 3, dtype=torch.int64, device="cuda")
+ result = compute_local_tokens_per_expert(routing_map, 0, num_local)
+ assert result[3].item() == num_tokens * topk
+ assert result.sum().item() == num_tokens * topk
+
+ @pytest.mark.parametrize("seed", [0, 7, 42, 123, 999])
+ def test_total_count_equals_local_pairs(self, seed):
+ """Sum of tokens_per_expert equals total routing pairs hitting local experts."""
+ from megatron.core.inference.moe.permute import compute_local_tokens_per_expert
+
+ torch.manual_seed(seed)
+ num_tokens, topk, num_experts = 64, 6, 16
+ local_start, num_local = 4, 4
+ routing_map = torch.randint(0, num_experts, (num_tokens, topk), device="cuda")
+ result = compute_local_tokens_per_expert(routing_map, local_start, num_local)
+ local_mask = (routing_map >= local_start) & (routing_map < local_start + num_local)
+ assert result.sum().item() == local_mask.sum().item()
+
+
+@pytest.mark.internal
+class TestComputeExpertOffsets:
+
+ @pytest.mark.parametrize("alignment", [1, 8, 16, 32, 64, 128])
+ @pytest.mark.parametrize(
+ "tpe_values",
+ [
+ [5, 0, 12, 3, 0, 7, 1, 20],
+ [1, 1, 1, 1],
+ [0, 0, 0, 0],
+ [100, 0, 0, 50],
+ [1],
+ [33, 33, 33, 33, 33, 33, 33, 33],
+ [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16],
+ [127, 0, 129, 0, 1, 0, 255, 0],
+ ],
+ )
+ def test_matches_reference(self, alignment, tpe_values):
+ from megatron.core.inference.moe.permute import compute_expert_offsets
+
+ tpe = torch.tensor(tpe_values, dtype=torch.int32, device="cuda")
+ exc, inc = compute_expert_offsets(tpe, alignment=alignment)
+ ref_exc, ref_inc = _ref_expert_offsets(tpe, alignment)
+ torch.testing.assert_close(exc, ref_exc, atol=0, rtol=0)
+ torch.testing.assert_close(inc, ref_inc, atol=0, rtol=0)
+
+ @pytest.mark.parametrize("n_experts", [1, 2, 4, 8, 16, 32, 64, 128])
+ def test_exclusive_starts_at_zero(self, n_experts):
+ from megatron.core.inference.moe.permute import compute_expert_offsets
+
+ tpe = torch.randint(1, 50, (n_experts,), dtype=torch.int32, device="cuda")
+ exc, inc = compute_expert_offsets(tpe, alignment=1)
+ assert exc[0].item() == 0
+ assert inc[-1].item() == tpe.sum().item()
+
+ def test_zero_experts_skipped(self):
+ """Experts with 0 tokens should not consume any aligned space."""
+ from megatron.core.inference.moe.permute import compute_expert_offsets
+
+ tpe = torch.tensor([0, 5, 0, 3], dtype=torch.int32, device="cuda")
+ exc, inc = compute_expert_offsets(tpe, alignment=32)
+ # Expert 0: 0 tokens -> 0 aligned -> exc=0, inc=0
+ assert exc[0].item() == 0
+ assert inc[0].item() == 0
+ # Expert 1: 5 tokens -> 32 aligned -> exc=0, inc=32
+ assert exc[1].item() == 0
+ assert inc[1].item() == 32
+ # Expert 2: 0 tokens -> exc=32, inc=32
+ assert exc[2].item() == 32
+ assert inc[2].item() == 32
+
+ @pytest.mark.parametrize("alignment", [16, 32, 128])
+ def test_all_offsets_aligned(self, alignment):
+ """Every inclusive offset should be a multiple of alignment."""
+ from megatron.core.inference.moe.permute import compute_expert_offsets
+
+ tpe = torch.tensor([3, 7, 0, 15, 1, 0, 50, 2], dtype=torch.int32, device="cuda")
+ exc, inc = compute_expert_offsets(tpe, alignment=alignment)
+ for i in range(len(tpe)):
+ assert (
+ inc[i].item() % alignment == 0
+ ), f"inc[{i}]={inc[i].item()} not aligned to {alignment}"
+ assert (
+ exc[i].item() % alignment == 0
+ ), f"exc[{i}]={exc[i].item()} not aligned to {alignment}"
+
+
+class TestPermuteTokens:
+
+ @pytest.mark.parametrize(
+ "num_tokens,hidden_dim,topk,num_experts",
+ [
+ (1, 64, 1, 4),
+ (1, 128, 8, 8),
+ (4, 64, 2, 4),
+ (16, 128, 2, 8),
+ (32, 64, 4, 8),
+ (64, 256, 6, 8),
+ (128, 128, 8, 128),
+ (256, 64, 2, 32),
+ (512, 128, 6, 16),
+ ],
+ )
+ def test_data_integrity(self, num_tokens, hidden_dim, topk, num_experts):
+ """Every permuted row matches the original token's hidden state."""
+ from megatron.core.inference.moe.permute import permute_tokens
+
+ hidden, probs, routing_map = _make_inputs(num_tokens, hidden_dim, topk, num_experts)
+ perm_h, perm_p, perm_map, offs = permute_tokens(
+ hidden, probs, routing_map, 0, num_experts, alignment=1
+ )
+
+ # Check every non-padding row
+ for i in range(perm_map.shape[0]):
+ src = perm_map[i].item()
+ if src < 0:
+ continue
+ torch.testing.assert_close(
+ perm_h[i], hidden[src], msg=f"Row {i} (src={src}) hidden mismatch"
+ )
+
+ @pytest.mark.parametrize("alignment", [1, 16, 32, 64, 128])
+ @pytest.mark.parametrize("num_tokens,topk,num_experts", [(16, 2, 4), (64, 4, 8), (128, 8, 32)])
+ def test_offsets_are_aligned(self, alignment, num_tokens, topk, num_experts):
+ """Inclusive offsets are multiples of alignment."""
+ from megatron.core.inference.moe.permute import permute_tokens
+
+ hidden, probs, routing_map = _make_inputs(num_tokens, 128, topk, num_experts)
+ _, _, _, offs = permute_tokens(
+ hidden, probs, routing_map, 0, num_experts, alignment=alignment
+ )
+ if alignment > 1:
+ for i in range(offs.shape[0]):
+ assert (
+ offs[i].item() % alignment == 0
+ ), f"Offset {i}={offs[i].item()} not aligned to {alignment}"
+
+ @pytest.mark.parametrize(
+ "num_tokens,topk,num_experts,alignment",
+ [(8, 2, 4, 128), (32, 2, 4, 128), (16, 4, 8, 64), (64, 6, 8, 32)],
+ )
+ def test_padding_rows_have_neg1(self, num_tokens, topk, num_experts, alignment):
+ """Padding rows in permutation_map are -1."""
+ from megatron.core.inference.moe.permute import permute_tokens
+
+ hidden, probs, routing_map = _make_inputs(num_tokens, 64, topk, num_experts)
+ _, _, perm_map, _ = permute_tokens(
+ hidden, probs, routing_map, 0, num_experts, alignment=alignment
+ )
+ padding_mask = perm_map == -1
+ real_mask = perm_map >= 0
+ assert padding_mask.sum() > 0, "Expected some padding rows with large alignment"
+ assert real_mask.sum() > 0, "Expected some real rows"
+
+ @pytest.mark.parametrize(
+ "num_tokens,topk,num_experts", [(16, 2, 4), (32, 4, 8), (64, 6, 16), (128, 8, 128)]
+ )
+ @pytest.mark.parametrize("alignment", [1, 32, 128])
+ def test_total_real_rows_equals_routed_pairs(self, num_tokens, topk, num_experts, alignment):
+ """Number of non-padding rows equals total (token, topk) pairs routed locally."""
+ from megatron.core.inference.moe.permute import permute_tokens
+
+ hidden, probs, routing_map = _make_inputs(num_tokens, 64, topk, num_experts)
+ _, _, perm_map, _ = permute_tokens(
+ hidden, probs, routing_map, 0, num_experts, alignment=alignment
+ )
+ real_count = (perm_map >= 0).sum().item()
+ # All experts are local, so every pair should appear
+ assert real_count == num_tokens * topk
+
+ @pytest.mark.parametrize(
+ "num_tokens,topk,num_experts,local_start,num_local",
+ [
+ (64, 4, 8, 2, 3), # experts 2, 3, 4
+ (64, 4, 8, 0, 1), # only expert 0
+ (64, 4, 8, 7, 1), # only expert 7
+ (128, 6, 16, 4, 8), # experts 4-11
+ (32, 2, 32, 16, 16), # second half of 32
+ (256, 8, 128, 0, 32), # first 32 of 128
+ ],
+ )
+ def test_expert_subset(self, num_tokens, topk, num_experts, local_start, num_local):
+ """Only tokens routed to local experts appear in output."""
+ from megatron.core.inference.moe.permute import permute_tokens
+
+ hidden, probs, routing_map = _make_inputs(num_tokens, 64, topk, num_experts)
+ _, _, perm_map, _ = permute_tokens(
+ hidden, probs, routing_map, local_start, num_local, alignment=1
+ )
+ real_count = (perm_map >= 0).sum().item()
+ local_mask = (routing_map >= local_start) & (routing_map < local_start + num_local)
+ expected_count = local_mask.sum().item()
+ assert real_count == expected_count
+
+ @pytest.mark.parametrize("hidden_dim", [32, 64, 128, 256, 512, 1024, 2688])
+ def test_various_hidden_dims(self, hidden_dim):
+ """Permute works across various hidden dimensions including non-power-of-2."""
+ from megatron.core.inference.moe.permute import permute_tokens
+
+ hidden, probs, routing_map = _make_inputs(32, hidden_dim, 4, 8)
+ perm_h, _, perm_map, _ = permute_tokens(hidden, probs, routing_map, 0, 8, alignment=1)
+ # Spot-check first real row
+ for i in range(perm_map.shape[0]):
+ src = perm_map[i].item()
+ if src >= 0:
+ torch.testing.assert_close(perm_h[i], hidden[src])
+ break
+
+
+@pytest.mark.internal
+class TestUnpermuteTokens:
+
+ def test_weighted_scatter(self):
+ """Unpermute correctly accumulates prob-weighted expert outputs."""
+ from megatron.core.inference.moe.permute import unpermute_tokens
+
+ num_tokens, hidden_dim = 4, 8
+ # Two entries map to token 0, one to token 2
+ expert_output = torch.ones(3, hidden_dim, device="cuda", dtype=torch.bfloat16)
+ permuted_probs = torch.tensor([0.5, 0.3, 0.7], device="cuda", dtype=torch.float32)
+ perm_map = torch.tensor([0, 0, 2], dtype=torch.int32, device="cuda")
+
+ result = unpermute_tokens(expert_output, permuted_probs, perm_map, num_tokens)
+
+ assert result.dtype == torch.float32
+ # Token 0: 0.5 * 1.0 + 0.3 * 1.0 = 0.8
+ torch.testing.assert_close(
+ result[0], torch.full((hidden_dim,), 0.8, device="cuda"), atol=1e-5, rtol=1e-5
+ )
+ # Token 1: untouched -> 0
+ torch.testing.assert_close(result[1], torch.zeros(hidden_dim, device="cuda"))
+ # Token 2: 0.7 * 1.0 = 0.7
+ torch.testing.assert_close(
+ result[2], torch.full((hidden_dim,), 0.7, device="cuda"), atol=1e-5, rtol=1e-5
+ )
+
+ def test_padding_rows_ignored(self):
+ """Rows with permutation_map == -1 are skipped."""
+ from megatron.core.inference.moe.permute import unpermute_tokens
+
+ expert_output = torch.ones(4, 8, device="cuda", dtype=torch.bfloat16)
+ permuted_probs = torch.ones(4, device="cuda", dtype=torch.float32)
+ perm_map = torch.tensor([0, -1, -1, 1], dtype=torch.int32, device="cuda")
+
+ result = unpermute_tokens(expert_output, permuted_probs, perm_map, 3)
+ # Only tokens 0 and 1 get values
+ assert result[0].sum().item() != 0
+ assert result[1].sum().item() != 0
+ assert result[2].sum().item() == 0
+
+ @pytest.mark.parametrize("hidden_dim", [8, 64, 128, 256, 512, 2688])
+ def test_various_hidden_dims(self, hidden_dim):
+ """Unpermute works across various hidden dimensions."""
+ from megatron.core.inference.moe.permute import unpermute_tokens
+
+ num_tokens = 8
+ expert_output = torch.randn(4, hidden_dim, device="cuda", dtype=torch.bfloat16)
+ permuted_probs = torch.tensor([1.0, 1.0, 1.0, 1.0], device="cuda", dtype=torch.float32)
+ perm_map = torch.tensor([0, 1, 2, 3], dtype=torch.int32, device="cuda")
+
+ result = unpermute_tokens(expert_output, permuted_probs, perm_map, num_tokens)
+ assert result.shape == (num_tokens, hidden_dim)
+ # First 4 tokens should have values, rest should be zero
+ for t in range(4):
+ torch.testing.assert_close(result[t], expert_output[t].float(), atol=1e-5, rtol=1e-5)
+ for t in range(4, num_tokens):
+ assert result[t].sum().item() == 0
+
+ @pytest.mark.parametrize("topk", [1, 2, 4, 6, 8])
+ def test_multiple_topk_accumulation(self, topk):
+ """Multiple topk entries for the same token are summed correctly."""
+ from megatron.core.inference.moe.permute import unpermute_tokens
+
+ hidden_dim = 32
+ # All topk entries point to token 0
+ expert_output = torch.ones(topk, hidden_dim, device="cuda", dtype=torch.bfloat16)
+ probs = torch.full((topk,), 0.1, device="cuda", dtype=torch.float32)
+ perm_map = torch.zeros(topk, dtype=torch.int32, device="cuda")
+
+ result = unpermute_tokens(expert_output, probs, perm_map, 1)
+ expected_val = 0.1 * topk
+ torch.testing.assert_close(
+ result[0], torch.full((hidden_dim,), expected_val, device="cuda"), atol=1e-4, rtol=1e-4
+ )
+
+
+@pytest.mark.internal
+class TestPermuteUnpermuteRoundtrip:
+
+ @pytest.mark.parametrize(
+ "num_tokens,hidden_dim,topk,num_experts,alignment",
+ [
+ (1, 64, 1, 4, 1),
+ (1, 128, 1, 4, 128),
+ (8, 64, 1, 4, 1),
+ (16, 64, 2, 4, 1),
+ (16, 64, 2, 4, 32),
+ (32, 128, 4, 8, 32),
+ (32, 128, 4, 8, 128),
+ (64, 256, 6, 8, 1),
+ (64, 256, 6, 8, 128),
+ (128, 128, 8, 32, 1),
+ (128, 128, 8, 32, 128),
+ (256, 64, 2, 128, 32),
+ (64, 2688, 8, 128, 128), # nanov3-like hidden_dim
+ ],
+ )
+ def test_roundtrip_identity(self, num_tokens, hidden_dim, topk, num_experts, alignment):
+ """permute -> (identity transform) -> unpermute recovers weighted sum of inputs."""
+ from megatron.core.inference.moe.permute import permute_tokens, unpermute_tokens
+
+ torch.manual_seed(42)
+ hidden = torch.randn(num_tokens, hidden_dim, device="cuda", dtype=torch.bfloat16)
+ probs = torch.rand(num_tokens, topk, device="cuda", dtype=torch.float32)
+ routing_map = torch.randint(0, num_experts, (num_tokens, topk), device="cuda")
+
+ perm_h, perm_p, perm_map, _ = permute_tokens(
+ hidden, probs, routing_map, 0, num_experts, alignment=alignment
+ )
+ # Pass permuted hidden directly through (identity expert)
+ result = unpermute_tokens(perm_h, perm_p, perm_map, num_tokens)
+
+ # Build reference: for each token, sum prob[k] * hidden[token] over topk
+ ref = torch.zeros(num_tokens, hidden_dim, device="cuda", dtype=torch.float32)
+ for t in range(num_tokens):
+ prob_sum = probs[t].sum()
+ ref[t] = hidden[t].float() * prob_sum
+
+ torch.testing.assert_close(result, ref, atol=1e-2, rtol=1e-2)
+
+ @pytest.mark.parametrize(
+ "local_start,num_local,num_experts",
+ [(0, 4, 8), (4, 4, 8), (0, 1, 8), (0, 8, 8), (0, 32, 128), (96, 32, 128)],
+ )
+ def test_roundtrip_with_expert_subset(self, local_start, num_local, num_experts):
+ """Roundtrip works when only a subset of experts are local."""
+ from megatron.core.inference.moe.permute import permute_tokens, unpermute_tokens
+
+ torch.manual_seed(42)
+ num_tokens, hidden_dim, topk = 64, 128, 4
+ hidden = torch.randn(num_tokens, hidden_dim, device="cuda", dtype=torch.bfloat16)
+ probs = torch.rand(num_tokens, topk, device="cuda", dtype=torch.float32)
+ routing_map = torch.randint(0, num_experts, (num_tokens, topk), device="cuda")
+
+ perm_h, perm_p, perm_map, _ = permute_tokens(
+ hidden, probs, routing_map, local_start, num_local, alignment=32
+ )
+ result = unpermute_tokens(perm_h, perm_p, perm_map, num_tokens)
+
+ # Reference: only accumulate probs for local experts
+ ref = torch.zeros(num_tokens, hidden_dim, device="cuda", dtype=torch.float32)
+ for t in range(num_tokens):
+ local_prob_sum = 0.0
+ for k in range(topk):
+ eid = routing_map[t, k].item()
+ if local_start <= eid < local_start + num_local:
+ local_prob_sum += probs[t, k].item()
+ ref[t] = hidden[t].float() * local_prob_sum
+
+ torch.testing.assert_close(result, ref, atol=1e-2, rtol=1e-2)
diff --git a/tests/unit_tests/inference/test_mxfp8_utils.py b/tests/unit_tests/inference/test_mxfp8_utils.py
new file mode 100644
index 00000000000..a137dfbc820
--- /dev/null
+++ b/tests/unit_tests/inference/test_mxfp8_utils.py
@@ -0,0 +1,645 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+
+"""Unit tests for MXFP8 quantization.
+
+Tests cover:
+- mxfp8_quantize (Triton kernel): data and swizzled scales vs PyTorch reference
+- MXFP8Tensor.from_bf16: both 'triton' and 'flashinfer' backends
+- MXFP8Tensor.scale_2d: reshape correctness
+"""
+
+import pytest
+import torch
+
+pytestmark = [
+ pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA required"),
+ pytest.mark.internal,
+]
+
+
+def ceil_div(a, b):
+ return (a + b - 1) // b
+
+
+# ──────────────────────────────────────────────────────────────────────
+# Reference functions from PyTorch
+# https://github.com/pytorch/pytorch/blob/main/torch/testing/_internal/common_quantized.py#L578
+# ──────────────────────────────────────────────────────────────────────
+
+
+def ref_to_mxfp(data_hp: torch.Tensor, block_size: int = 32, format: str = "mxfp8"):
+ if data_hp.dtype not in (torch.bfloat16, torch.float):
+ raise AssertionError(f"{data_hp.dtype} is not supported yet")
+ if data_hp.shape[-1] % block_size != 0:
+ raise AssertionError(
+ f"the last dimension of shape {data_hp.shape} must be divisible by block_size {block_size}"
+ )
+ if not data_hp.is_contiguous():
+ raise AssertionError("unsupported: data_hp must be contiguous")
+
+ orig_shape = data_hp.shape
+ data_hp = data_hp.reshape(*orig_shape[:-1], orig_shape[-1] // block_size, block_size)
+
+ max_abs = torch.amax(torch.abs(data_hp), -1).unsqueeze(-1)
+
+ data_hp = data_hp.to(torch.float32)
+ max_abs = max_abs.to(torch.float32)
+
+ if format == "mxfp8":
+ F8E4M3_MAX = torch.finfo(torch.float8_e4m3fn).max # 448.0
+ max_pos = F8E4M3_MAX
+ elif format == "mxfp4":
+ F4E2M1_MAX = 6.0
+ max_pos = F4E2M1_MAX
+
+ # RCEIL
+ def _to_mx_rceil(
+ data_hp: torch.Tensor, max_abs: torch.Tensor, max_pos: float
+ ) -> tuple[torch.Tensor, torch.Tensor]:
+ E8M0_EXPONENT_BIAS = 127
+ descale = max_abs / max_pos
+ exponent = torch.where(
+ torch.isnan(descale),
+ 0xFF, # Handle biased exponent for nan
+ (
+ torch.clamp(
+ torch.ceil(torch.log2(descale)), min=-E8M0_EXPONENT_BIAS, max=E8M0_EXPONENT_BIAS
+ )
+ + E8M0_EXPONENT_BIAS
+ ).to(torch.uint8),
+ )
+
+ descale_fp = torch.where(
+ exponent == 0, 1.0, torch.exp2(E8M0_EXPONENT_BIAS - exponent.to(torch.float32))
+ )
+
+ # scale and saturated cast the data elements to max of target dtype
+ data_lp = torch.clamp(data_hp * descale_fp, min=-1 * max_pos, max=max_pos)
+ return exponent, data_lp
+
+ scale_e8m0_biased, data_lp = _to_mx_rceil(data_hp, max_abs, max_pos)
+
+ # cast to target dtype
+ data_lp = data_lp.to(torch.float8_e4m3fn)
+ data_lp = data_lp.reshape(orig_shape)
+
+ scale_e8m0_biased = scale_e8m0_biased.view(torch.float8_e8m0fnu)
+ scale_e8m0_biased = scale_e8m0_biased.squeeze(-1)
+ return scale_e8m0_biased, data_lp
+
+
+def ref_swizzle(input_matrix) -> torch.Tensor:
+ """Rearrange a scale matrix into cuBLAS 2D blocked (swizzled) layout.
+
+ See: https://docs.nvidia.com/cuda/cublas/index.html#d-block-scaling-factors-layout
+
+ Args:
+ input_matrix: Input tensor of shape (H, W)
+
+ Returns:
+ Flattened swizzled tensor.
+ """
+ rows, cols = input_matrix.shape
+ n_row_blocks = ceil_div(rows, 128)
+ n_col_blocks = ceil_div(cols, 4)
+
+ padded_rows = n_row_blocks * 128
+ padded_cols = n_col_blocks * 4
+
+ padded = input_matrix
+ if (rows, cols) != (padded_rows, padded_cols):
+ padded = torch.zeros(
+ (padded_rows, padded_cols), device=input_matrix.device, dtype=input_matrix.dtype
+ )
+ padded[:rows, :cols] = input_matrix
+
+ blocks = padded.view(n_row_blocks, 128, n_col_blocks, 4).permute(0, 2, 1, 3)
+ rearranged = blocks.reshape(-1, 4, 32, 4).transpose(1, 2).reshape(-1, 32, 16)
+
+ return rearranged.flatten()
+
+
+# ──────────────────────────────────────────────────────────────────────
+# mxfp8_quantize (Triton kernel)
+# ──────────────────────────────────────────────────────────────────────
+
+
+class TestMxfp8Quantize:
+
+ @pytest.mark.parametrize(
+ "M,K",
+ [
+ (1, 32),
+ (1, 64),
+ (1, 128),
+ (4, 32),
+ (4, 128),
+ (16, 64),
+ (16, 256),
+ (32, 128),
+ (64, 256),
+ (128, 128),
+ (128, 512),
+ (128, 2688), # nanov3 hidden_size
+ (256, 1856), # nanov3 moe_ffn_hidden_size
+ (512, 2688),
+ ],
+ )
+ def test_data_matches_reference(self, M, K):
+ """Quantized FP8 data matches PyTorch reference."""
+ from megatron.core.inference.quantization.mxfp8_quantize import mxfp8_quantize
+
+ torch.manual_seed(42)
+ x = torch.randn(M, K, device="cuda", dtype=torch.bfloat16)
+
+ triton_data, _ = mxfp8_quantize(x)
+ _, ref_data = ref_to_mxfp(x)
+
+ assert triton_data.shape == (M, K)
+ assert triton_data.dtype == torch.float8_e4m3fn
+ torch.testing.assert_close(
+ triton_data.view(torch.uint8), ref_data.view(torch.uint8), atol=0, rtol=0
+ )
+
+ @pytest.mark.parametrize(
+ "M,K",
+ [
+ (1, 32),
+ (1, 64),
+ (4, 128),
+ (16, 256),
+ (32, 128),
+ (128, 128),
+ (128, 512),
+ (128, 2688),
+ (256, 1856),
+ (512, 2688),
+ ],
+ )
+ def test_scales_match_reference(self, M, K):
+ """Swizzled scales match ref_to_mxfp scales passed through ref_swizzle."""
+ from megatron.core.inference.quantization.mxfp8_quantize import mxfp8_quantize
+
+ torch.manual_seed(42)
+ x = torch.randn(M, K, device="cuda", dtype=torch.bfloat16)
+
+ _, triton_scales = mxfp8_quantize(x)
+ ref_scales_2d, _ = ref_to_mxfp(x) # [M, K//32] e8m0
+
+ # Swizzle the reference scales
+ ref_swizzled = ref_swizzle(ref_scales_2d)
+
+ # Compare as uint8 since e8m0 is just exponent bytes
+ torch.testing.assert_close(
+ triton_scales.view(torch.uint8), ref_swizzled.view(torch.uint8), atol=0, rtol=0
+ )
+
+ @pytest.mark.parametrize("M,K", [(1, 32), (16, 128), (128, 2688)])
+ def test_all_zeros_input(self, M, K):
+ """All-zero input produces all-zero FP8 data and zero scales."""
+ from megatron.core.inference.quantization.mxfp8_quantize import mxfp8_quantize
+
+ x = torch.zeros(M, K, device="cuda", dtype=torch.bfloat16)
+ data, scales = mxfp8_quantize(x)
+ assert (data.float() == 0).all()
+ assert (scales.view(torch.uint8) == 0).all()
+
+ @pytest.mark.parametrize("M,K", [(1, 32), (16, 128), (128, 256)])
+ def test_constant_input(self, M, K):
+ """Constant input: all elements in a group have the same value."""
+ from megatron.core.inference.quantization.mxfp8_quantize import mxfp8_quantize
+
+ x = torch.full((M, K), 1.0, device="cuda", dtype=torch.bfloat16)
+ data, _ = mxfp8_quantize(x)
+ _, ref_data = ref_to_mxfp(x)
+ torch.testing.assert_close(
+ data.view(torch.uint8), ref_data.view(torch.uint8), atol=0, rtol=0
+ )
+
+ @pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16, torch.float32])
+ def test_input_dtypes(self, dtype):
+ """Kernel accepts bf16, fp16, and fp32 inputs."""
+ from megatron.core.inference.quantization.mxfp8_quantize import mxfp8_quantize
+
+ x = torch.randn(16, 128, device="cuda", dtype=dtype)
+ data, _ = mxfp8_quantize(x)
+ assert data.dtype == torch.float8_e4m3fn
+ assert data.shape == (16, 128)
+
+ @pytest.mark.parametrize("M", [1, 127, 128, 129, 255, 256, 257, 512])
+ def test_various_row_counts(self, M):
+ """Test row counts that are not multiples of 128 (macro tile boundary)."""
+ from megatron.core.inference.quantization.mxfp8_quantize import mxfp8_quantize
+
+ K = 128
+ torch.manual_seed(42)
+ x = torch.randn(M, K, device="cuda", dtype=torch.bfloat16)
+ data, _ = mxfp8_quantize(x)
+ _, ref_data = ref_to_mxfp(x)
+ torch.testing.assert_close(
+ data.view(torch.uint8), ref_data.view(torch.uint8), atol=0, rtol=0
+ )
+
+ @pytest.mark.parametrize("seed", [0, 7, 42, 123, 999])
+ def test_reproducible(self, seed):
+ """Same input always produces same output."""
+ from megatron.core.inference.quantization.mxfp8_quantize import mxfp8_quantize
+
+ torch.manual_seed(seed)
+ x = torch.randn(64, 256, device="cuda", dtype=torch.bfloat16)
+ d1, s1 = mxfp8_quantize(x)
+ d2, s2 = mxfp8_quantize(x)
+ torch.testing.assert_close(d1.view(torch.uint8), d2.view(torch.uint8), atol=0, rtol=0)
+ torch.testing.assert_close(s1.view(torch.uint8), s2.view(torch.uint8), atol=0, rtol=0)
+
+
+# ──────────────────────────────────────────────────────────────────────
+# MXFP8Tensor
+# ──────────────────────────────────────────────────────────────────────
+
+
+class TestMXFP8Tensor:
+
+ @pytest.mark.parametrize("M,K", [(16, 128), (64, 256), (128, 2688)])
+ def test_from_bf16_triton(self, M, K):
+ """from_bf16 with triton backend produces correct data and scales."""
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+
+ torch.manual_seed(42)
+ x = torch.randn(M, K, device="cuda", dtype=torch.bfloat16)
+
+ tensor = MXFP8Tensor.from_bf16(x, backend="triton")
+ _, ref_data = ref_to_mxfp(x)
+
+ assert tensor.data.shape == (M, K)
+ assert tensor.data.dtype == torch.float8_e4m3fn
+ assert tensor.backend == "triton"
+ torch.testing.assert_close(
+ tensor.data.view(torch.uint8), ref_data.view(torch.uint8), atol=0, rtol=0
+ )
+
+ @pytest.mark.parametrize("M,K", [(16, 128), (64, 256), (128, 2688)])
+ def test_from_bf16_flashinfer(self, M, K):
+ """from_bf16 with flashinfer backend produces valid output."""
+ from megatron.core.inference.quantization.mxfp8_tensor import HAVE_FLASHINFER, MXFP8Tensor
+
+ if not HAVE_FLASHINFER:
+ pytest.skip("FlashInfer not available")
+
+ torch.manual_seed(42)
+ x = torch.randn(M, K, device="cuda", dtype=torch.bfloat16)
+
+ tensor = MXFP8Tensor.from_bf16(x, backend="flashinfer")
+ assert tensor.data.shape == (M, K)
+ assert tensor.data.dtype == torch.float8_e4m3fn
+ assert tensor.backend == "flashinfer"
+
+ def test_from_bf16_invalid_backend(self):
+ """from_bf16 with invalid backend raises ValueError."""
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+
+ x = torch.randn(16, 128, device="cuda", dtype=torch.bfloat16)
+ with pytest.raises(ValueError, match="Unknown MXFP8 quantization backend"):
+ MXFP8Tensor.from_bf16(x, backend="invalid")
+
+ @pytest.mark.parametrize("M,K", [(1, 32), (16, 128), (128, 2688), (256, 1856)])
+ def test_scale_2d_shape(self, M, K):
+ """scale_2d returns correct shape: (-1, ceil(K//32, 4)*4)."""
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+
+ torch.manual_seed(42)
+ x = torch.randn(M, K, device="cuda", dtype=torch.bfloat16)
+ tensor = MXFP8Tensor.from_bf16(x, backend="triton")
+
+ scale_2d = tensor.scale_2d()
+ expected_cols = ceil_div(K // 32, 4) * 4
+ assert scale_2d.dim() == 2
+ assert scale_2d.shape[1] == expected_cols
+
+ @pytest.mark.parametrize("M,K", [(16, 128), (128, 2688)])
+ def test_scale_2d_idempotent(self, M, K):
+ """Calling scale_2d twice returns the same result."""
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+
+ torch.manual_seed(42)
+ x = torch.randn(M, K, device="cuda", dtype=torch.bfloat16)
+ tensor = MXFP8Tensor.from_bf16(x, backend="triton")
+
+ s1 = tensor.scale_2d()
+ s2 = tensor.scale_2d()
+ torch.testing.assert_close(s1.view(torch.uint8), s2.view(torch.uint8), atol=0, rtol=0)
+
+ def test_size_method(self):
+ """size() delegates to data.size()."""
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+
+ x = torch.randn(32, 128, device="cuda", dtype=torch.bfloat16)
+ tensor = MXFP8Tensor.from_bf16(x, backend="triton")
+ assert tensor.size() == torch.Size([32, 128])
+ assert tensor.size(0) == 32
+ assert tensor.size(1) == 128
+
+
+# ──────────────────────────────────────────────────────────────────────
+# Triton vs FlashInfer cross-validation
+# ──────────────────────────────────────────────────────────────────────
+
+
+@pytest.mark.skipif(
+ torch.cuda.get_device_capability()[0] < 10,
+ reason="MXFP8 FlashInfer comparison requires Blackwell (SM 100+)",
+)
+class TestTritonVsFlashinfer:
+
+ @pytest.mark.parametrize("M,K", [(1, 32), (16, 128), (64, 256), (128, 2688), (256, 1856)])
+ def test_data_matches(self, M, K):
+ """Triton and FlashInfer backends produce identical FP8 data."""
+ from megatron.core.inference.quantization.mxfp8_tensor import HAVE_FLASHINFER, MXFP8Tensor
+
+ if not HAVE_FLASHINFER:
+ pytest.skip("FlashInfer not available")
+
+ torch.manual_seed(42)
+ x = torch.randn(M, K, device="cuda", dtype=torch.bfloat16)
+
+ triton_tensor = MXFP8Tensor.from_bf16(x, backend="triton")
+ flashinfer_tensor = MXFP8Tensor.from_bf16(x, backend="flashinfer")
+
+ torch.testing.assert_close(
+ triton_tensor.data.float(), flashinfer_tensor.data.float(), atol=0, rtol=0
+ )
+
+ @pytest.mark.parametrize("M,K", [(1, 32), (16, 128), (64, 256), (128, 2688), (256, 1856)])
+ def test_scales_match(self, M, K):
+ """Triton and FlashInfer backends produce identical swizzled scales."""
+ from megatron.core.inference.quantization.mxfp8_tensor import HAVE_FLASHINFER, MXFP8Tensor
+
+ if not HAVE_FLASHINFER:
+ pytest.skip("FlashInfer not available")
+
+ torch.manual_seed(42)
+ x = torch.randn(M, K, device="cuda", dtype=torch.bfloat16)
+
+ triton_tensor = MXFP8Tensor.from_bf16(x, backend="triton")
+ flashinfer_tensor = MXFP8Tensor.from_bf16(x, backend="flashinfer")
+
+ torch.testing.assert_close(
+ triton_tensor.scale.view(torch.uint8),
+ flashinfer_tensor.scale.view(torch.uint8),
+ atol=0,
+ rtol=0,
+ )
+
+
+def _make_permutation_map(M, num_padding=0):
+ """Create a permutation_map with optional padding rows at the end."""
+ real = torch.arange(M - num_padding, dtype=torch.int32, device="cuda")
+ pad = torch.full((num_padding,), -1, dtype=torch.int32, device="cuda")
+ return torch.cat([real, pad])
+
+
+# ──────────────────────────────────────────────────────────────────────
+# squared_relu_and_quantize_mxfp8 vs PyTorch reference
+# ──────────────────────────────────────────────────────────────────────
+
+
+class TestSquaredReluAndQuantizeMxfp8:
+ """Compare fused squared_relu + mxfp8 quantize against PyTorch reference.
+
+ Reference: torch.relu(x.float()).pow(2).to(bf16) -> ref_to_mxfp -> ref_swizzle.
+ The fused kernel computes squared ReLU in fp32 and quantizes to MXFP8 in one pass,
+ so the PyTorch fp32 reference is the correct baseline (not the unfused Triton path
+ which has an intermediate bf16 roundtrip).
+ """
+
+ @pytest.mark.parametrize(
+ "M,K",
+ [
+ (1, 32),
+ (4, 64),
+ (16, 128),
+ (32, 256),
+ (64, 128),
+ (128, 128),
+ (128, 256),
+ (128, 2688),
+ (256, 1856),
+ (512, 2688),
+ ],
+ )
+ def test_data_matches_pytorch_ref(self, M, K):
+ """Fused FP8 data matches PyTorch squared ReLU + ref_to_mxfp."""
+ from megatron.core.inference.moe.activations import squared_relu_and_quantize_mxfp8
+
+ torch.manual_seed(42)
+ x = torch.randn(M, K, device="cuda", dtype=torch.bfloat16)
+ perm_map = _make_permutation_map(M, num_padding=0)
+
+ # PyTorch reference: squared ReLU in fp32, then downcast to bf16, then quantize
+ activated_ref = torch.relu(x.float()).pow(2)
+ _, ref_data = ref_to_mxfp(activated_ref)
+
+ # Fused kernel
+ fused_result = squared_relu_and_quantize_mxfp8(x, perm_map)
+
+ torch.testing.assert_close(
+ fused_result.data.view(torch.uint8), ref_data.view(torch.uint8), atol=0, rtol=0
+ )
+
+ @pytest.mark.parametrize("M,K", [(1, 32), (16, 128), (128, 128), (128, 2688), (256, 1856)])
+ def test_scales_match_pytorch_ref(self, M, K):
+ """Fused swizzled scales match PyTorch ref_to_mxfp + ref_swizzle."""
+ from megatron.core.inference.moe.activations import squared_relu_and_quantize_mxfp8
+
+ torch.manual_seed(42)
+ x = torch.randn(M, K, device="cuda", dtype=torch.bfloat16)
+ perm_map = _make_permutation_map(M, num_padding=0)
+
+ # PyTorch reference
+ activated_ref = torch.relu(x.float()).pow(2)
+ ref_scales_2d, _ = ref_to_mxfp(activated_ref)
+ ref_swizzled = ref_swizzle(ref_scales_2d)
+
+ # Fused kernel
+ fused_result = squared_relu_and_quantize_mxfp8(x, perm_map)
+
+ torch.testing.assert_close(
+ fused_result.scale.view(torch.uint8), ref_swizzled.view(torch.uint8), atol=0, rtol=0
+ )
+
+ @pytest.mark.parametrize(
+ "M,K,num_padding",
+ [(32, 128, 8), (64, 256, 16), (128, 128, 32), (128, 2688, 64), (256, 1856, 128)],
+ )
+ def test_real_rows_match_pytorch_ref_with_padding(self, M, K, num_padding):
+ """Real rows match PyTorch reference even when padding rows are present."""
+ from megatron.core.inference.moe.activations import squared_relu_and_quantize_mxfp8
+
+ torch.manual_seed(42)
+ x = torch.randn(M, K, device="cuda", dtype=torch.bfloat16)
+ perm_map = _make_permutation_map(M, num_padding=num_padding)
+
+ # PyTorch reference (only real rows)
+ real_rows = M - num_padding
+ activated_ref = torch.relu(x[:real_rows].float()).pow(2)
+ _, ref_data = ref_to_mxfp(activated_ref)
+
+ # Fused kernel
+ fused_result = squared_relu_and_quantize_mxfp8(x, perm_map)
+
+ torch.testing.assert_close(
+ fused_result.data[:real_rows].view(torch.uint8),
+ ref_data.view(torch.uint8),
+ atol=0,
+ rtol=0,
+ )
+
+
+# ──────────────────────────────────────────────────────────────────────
+# permute_and_quantize_mxfp8
+# ──────────────────────────────────────────────────────────────────────
+
+
+class TestPermuteAndQuantizeMxfp8:
+ """Compare fused permute + mxfp8 quantize against PyTorch reference.
+
+ PyTorch reference:
+ 1. For each real row, quantize the source token with ref_to_mxfp
+ 2. Compare FP8 data per source token
+ Structural checks (permutation_map, probs, offsets) verified independently.
+ """
+
+ def _make_inputs(self, num_tokens, K, topk, num_experts, seed=42):
+ torch.manual_seed(seed)
+ hidden = torch.randn(num_tokens, K, device="cuda", dtype=torch.bfloat16)
+ probs = torch.rand(num_tokens, topk, device="cuda", dtype=torch.float32)
+ routing_map = torch.randint(0, num_experts, (num_tokens, topk), device="cuda")
+ return hidden, probs, routing_map
+
+ @pytest.mark.parametrize(
+ "num_tokens,K,topk,num_experts",
+ [
+ (4, 128, 2, 4),
+ (16, 128, 2, 8),
+ (32, 256, 4, 8),
+ (64, 128, 6, 8),
+ (64, 2688, 8, 128),
+ (128, 1856, 4, 32),
+ ],
+ )
+ def test_data_matches_pytorch_ref(self, num_tokens, K, topk, num_experts):
+ """For each real row, fused FP8 data matches ref_to_mxfp of the source token."""
+ from megatron.core.inference.moe.permute import permute_and_quantize_mxfp8
+
+ hidden, probs, routing_map = self._make_inputs(num_tokens, K, topk, num_experts)
+
+ fused_mxfp8, _, fused_perm_map, _ = permute_and_quantize_mxfp8(
+ hidden, probs, routing_map, 0, num_experts, alignment=128
+ )
+
+ # For each real row, quantize the source token with PyTorch ref and compare
+ for i in range(fused_perm_map.shape[0]):
+ src = fused_perm_map[i].item()
+ if src < 0:
+ continue
+ _, ref_data = ref_to_mxfp(hidden[src].unsqueeze(0))
+ torch.testing.assert_close(
+ fused_mxfp8.data[i].view(torch.uint8),
+ ref_data.squeeze(0).view(torch.uint8),
+ atol=0,
+ rtol=0,
+ msg=f"Row {i} (src={src}) FP8 data mismatch vs PyTorch ref",
+ )
+
+ @pytest.mark.parametrize(
+ "num_tokens,K,topk,num_experts", [(16, 128, 2, 8), (32, 256, 4, 8), (64, 2688, 8, 128)]
+ )
+ def test_batch_data_matches_pytorch_ref(self, num_tokens, K, topk, num_experts):
+ """Batch comparison: gather all real rows, quantize as batch, compare."""
+ from megatron.core.inference.moe.permute import permute_and_quantize_mxfp8
+
+ hidden, probs, routing_map = self._make_inputs(num_tokens, K, topk, num_experts)
+
+ fused_mxfp8, _, fused_perm_map, _ = permute_and_quantize_mxfp8(
+ hidden, probs, routing_map, 0, num_experts, alignment=128
+ )
+
+ real_mask = fused_perm_map >= 0
+ real_indices = real_mask.nonzero(as_tuple=True)[0]
+ if len(real_indices) == 0:
+ return
+
+ src_tokens = fused_perm_map[real_indices].long()
+ permuted_bf16 = hidden[src_tokens]
+
+ _, ref_data = ref_to_mxfp(permuted_bf16)
+
+ torch.testing.assert_close(
+ fused_mxfp8.data[real_indices].view(torch.uint8),
+ ref_data.view(torch.uint8),
+ atol=0,
+ rtol=0,
+ )
+
+ @pytest.mark.parametrize(
+ "num_tokens,K,topk,num_experts", [(16, 128, 2, 8), (32, 256, 4, 8), (64, 2688, 8, 128)]
+ )
+ def test_correct_token_count(self, num_tokens, K, topk, num_experts):
+ """Number of real rows equals total (token, topk) pairs routed to local experts."""
+ from megatron.core.inference.moe.permute import permute_and_quantize_mxfp8
+
+ hidden, probs, routing_map = self._make_inputs(num_tokens, K, topk, num_experts)
+
+ _, _, fused_perm_map, _ = permute_and_quantize_mxfp8(
+ hidden, probs, routing_map, 0, num_experts, alignment=128
+ )
+
+ real_count = (fused_perm_map >= 0).sum().item()
+ # All experts are local, so every pair should appear
+ assert real_count == num_tokens * topk
+
+ @pytest.mark.parametrize(
+ "num_tokens,K,topk,num_experts,local_start,num_local",
+ [(64, 128, 4, 8, 2, 3), (64, 256, 4, 8, 0, 4), (128, 128, 8, 128, 96, 32)],
+ )
+ def test_expert_subset(self, num_tokens, K, topk, num_experts, local_start, num_local):
+ """Fused kernel correctly handles local expert subsets."""
+ from megatron.core.inference.moe.permute import permute_and_quantize_mxfp8
+
+ hidden, probs, routing_map = self._make_inputs(num_tokens, K, topk, num_experts)
+
+ _, _, fused_perm_map, _ = permute_and_quantize_mxfp8(
+ hidden, probs, routing_map, local_start, num_local, alignment=128
+ )
+
+ real_count = (fused_perm_map >= 0).sum().item()
+ local_mask = (routing_map >= local_start) & (routing_map < local_start + num_local)
+ expected_count = local_mask.sum().item()
+ assert real_count == expected_count
+
+ def test_returns_mxfp8_tensor(self):
+ """Result is an MXFP8Tensor with correct backend."""
+ from megatron.core.inference.moe.permute import permute_and_quantize_mxfp8
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+
+ hidden, probs, routing_map = self._make_inputs(16, 128, 2, 4)
+ result, _, _, _ = permute_and_quantize_mxfp8(
+ hidden, probs, routing_map, 0, 4, alignment=128
+ )
+ assert isinstance(result, MXFP8Tensor)
+ assert result.backend == "triton"
+ assert result.data.dtype == torch.float8_e4m3fn
+
+ @pytest.mark.parametrize("alignment", [128])
+ def test_offsets_aligned(self, alignment):
+ """Inclusive offsets are multiples of alignment."""
+ from megatron.core.inference.moe.permute import permute_and_quantize_mxfp8
+
+ hidden, probs, routing_map = self._make_inputs(64, 128, 4, 8)
+ _, _, _, offs = permute_and_quantize_mxfp8(
+ hidden, probs, routing_map, 0, 8, alignment=alignment
+ )
+ for i in range(offs.shape[0]):
+ assert (
+ offs[i].item() % alignment == 0
+ ), f"Offset {i}={offs[i].item()} not aligned to {alignment}"
diff --git a/tests/unit_tests/inference/test_stop_words.py b/tests/unit_tests/inference/test_stop_words.py
index 31665c0bb81..b7bb8d7410f 100644
--- a/tests/unit_tests/inference/test_stop_words.py
+++ b/tests/unit_tests/inference/test_stop_words.py
@@ -31,132 +31,339 @@ class TestStopWordDetection:
"""Test stop word detection logic."""
def _check_stop_words_for_request_post_append(
- self, request: MockDynamicInferenceRequest
- ) -> bool:
+ self, request: MockDynamicInferenceRequest, num_speculative_tokens: int = 0
+ ) -> tuple:
"""
Check if a request should stop due to stop words (after token is appended).
- This mirrors the logic in DynamicInferenceEngine._check_stop_words_for_request_post_append
+ This mirrors the logic in DynamicInferenceEngine._check_stop_words_for_request_post_append.
+ Returns (stop_word_hit, num_tokens_trimmed).
"""
- # Check if request has stop words configured
if request.stop_word_ids is None or len(request.stop_word_ids) == 0:
- return False
+ return False, 0
generated_tokens = request.generated_tokens
- # Check if the sequence ends with any stop word
for stop_word_ids in request.stop_word_ids:
stop_len = len(stop_word_ids)
if len(generated_tokens) >= stop_len:
- # Check if the last stop_len tokens match the stop word
- if list(generated_tokens[-stop_len:]) == stop_word_ids:
- return True
+ for i in range(num_speculative_tokens + 1):
+ end_idx = -i if i > 0 else None
+ if list(generated_tokens[-stop_len - i : end_idx]) == stop_word_ids:
+ if i > 0:
+ request.generated_tokens = request.generated_tokens[:-i]
+ return True, i
- return False
+ return False, 0
def test_no_stop_words_configured(self):
"""Test that requests without stop words configured don't trigger stop."""
request = MockDynamicInferenceRequest(
request_id=1, generated_tokens=[100, 200, 300], stop_word_ids=None
)
- assert self._check_stop_words_for_request_post_append(request) is False
+ hit, trim = self._check_stop_words_for_request_post_append(request)
+ assert hit is False
+ assert trim == 0
def test_empty_stop_words_list(self):
"""Test that empty stop words list doesn't trigger stop."""
request = MockDynamicInferenceRequest(
request_id=1, generated_tokens=[100, 200, 300], stop_word_ids=[]
)
- assert self._check_stop_words_for_request_post_append(request) is False
+ hit, trim = self._check_stop_words_for_request_post_append(request)
+ assert hit is False
def test_single_token_stop_word_match(self):
"""Test detection of single-token stop word."""
- # Stop word is token 300
request = MockDynamicInferenceRequest(
request_id=1, generated_tokens=[100, 200, 300], stop_word_ids=[[300]]
)
- assert self._check_stop_words_for_request_post_append(request) is True
+ hit, trim = self._check_stop_words_for_request_post_append(request)
+ assert hit is True
+ assert trim == 0
+ assert request.generated_tokens == [100, 200, 300]
def test_single_token_stop_word_no_match(self):
"""Test no detection when single-token stop word doesn't match."""
request = MockDynamicInferenceRequest(
request_id=1, generated_tokens=[100, 200, 300], stop_word_ids=[[400]]
)
- assert self._check_stop_words_for_request_post_append(request) is False
+ hit, trim = self._check_stop_words_for_request_post_append(request)
+ assert hit is False
def test_multi_token_stop_word_match(self):
"""Test detection of multi-token stop word."""
- # Stop word is tokens [200, 300]
request = MockDynamicInferenceRequest(
request_id=1, generated_tokens=[100, 200, 300], stop_word_ids=[[200, 300]]
)
- assert self._check_stop_words_for_request_post_append(request) is True
+ hit, trim = self._check_stop_words_for_request_post_append(request)
+ assert hit is True
+ assert trim == 0
def test_multi_token_stop_word_no_match_partial(self):
"""Test no detection when only partial stop word matches."""
- # Stop word is [200, 300], but generated ends with [100, 200]
request = MockDynamicInferenceRequest(
request_id=1, generated_tokens=[100, 200], stop_word_ids=[[200, 300]]
)
- assert self._check_stop_words_for_request_post_append(request) is False
+ hit, trim = self._check_stop_words_for_request_post_append(request)
+ assert hit is False
def test_multi_token_stop_word_no_match_wrong_order(self):
"""Test no detection when tokens are present but in wrong order."""
- # Stop word is [200, 300], but generated ends with [300, 200]
request = MockDynamicInferenceRequest(
request_id=1, generated_tokens=[100, 300, 200], stop_word_ids=[[200, 300]]
)
- assert self._check_stop_words_for_request_post_append(request) is False
+ hit, trim = self._check_stop_words_for_request_post_append(request)
+ assert hit is False
def test_multiple_stop_words_first_matches(self):
"""Test with multiple stop words where first one matches."""
request = MockDynamicInferenceRequest(
request_id=1, generated_tokens=[100, 200, 300], stop_word_ids=[[300], [400], [500]]
)
- assert self._check_stop_words_for_request_post_append(request) is True
+ hit, trim = self._check_stop_words_for_request_post_append(request)
+ assert hit is True
def test_multiple_stop_words_second_matches(self):
"""Test with multiple stop words where second one matches."""
request = MockDynamicInferenceRequest(
request_id=1, generated_tokens=[100, 200, 400], stop_word_ids=[[300], [400], [500]]
)
- assert self._check_stop_words_for_request_post_append(request) is True
+ hit, trim = self._check_stop_words_for_request_post_append(request)
+ assert hit is True
def test_multiple_stop_words_none_match(self):
"""Test with multiple stop words where none match."""
request = MockDynamicInferenceRequest(
request_id=1, generated_tokens=[100, 200, 600], stop_word_ids=[[300], [400], [500]]
)
- assert self._check_stop_words_for_request_post_append(request) is False
+ hit, trim = self._check_stop_words_for_request_post_append(request)
+ assert hit is False
def test_stop_word_longer_than_generated(self):
"""Test that stop word longer than generated tokens doesn't crash."""
- # Stop word is 5 tokens, but only 3 tokens generated
request = MockDynamicInferenceRequest(
request_id=1, generated_tokens=[100, 200, 300], stop_word_ids=[[1, 2, 3, 4, 5]]
)
- assert self._check_stop_words_for_request_post_append(request) is False
+ hit, trim = self._check_stop_words_for_request_post_append(request)
+ assert hit is False
def test_stop_word_exact_length_match(self):
"""Test stop word that matches entire generated sequence."""
request = MockDynamicInferenceRequest(
request_id=1, generated_tokens=[100, 200, 300], stop_word_ids=[[100, 200, 300]]
)
- assert self._check_stop_words_for_request_post_append(request) is True
+ hit, trim = self._check_stop_words_for_request_post_append(request)
+ assert hit is True
def test_empty_generated_tokens(self):
"""Test with no generated tokens."""
request = MockDynamicInferenceRequest(
request_id=1, generated_tokens=[], stop_word_ids=[[300]]
)
- assert self._check_stop_words_for_request_post_append(request) is False
+ hit, trim = self._check_stop_words_for_request_post_append(request)
+ assert hit is False
def test_stop_word_in_middle_not_end(self):
"""Test that stop word in middle of sequence doesn't trigger (only end matters)."""
- # Stop word is [200], which is in middle but not at end
request = MockDynamicInferenceRequest(
request_id=1, generated_tokens=[100, 200, 300], stop_word_ids=[[200]]
)
- assert self._check_stop_words_for_request_post_append(request) is False
+ hit, trim = self._check_stop_words_for_request_post_append(request)
+ assert hit is False
+
+
+class TestStopWordSpeculativeDecoding:
+ """Test stop word detection and truncation with speculative decoding."""
+
+ def _check_stop_words_for_request_post_append(
+ self, request: MockDynamicInferenceRequest, num_speculative_tokens: int = 0
+ ) -> tuple:
+ """Mirror of DynamicInferenceEngine._check_stop_words_for_request_post_append."""
+ if request.stop_word_ids is None or len(request.stop_word_ids) == 0:
+ return False, 0
+
+ generated_tokens = request.generated_tokens
+
+ for stop_word_ids in request.stop_word_ids:
+ stop_len = len(stop_word_ids)
+ if len(generated_tokens) >= stop_len:
+ for i in range(num_speculative_tokens + 1):
+ end_idx = -i if i > 0 else None
+ if list(generated_tokens[-stop_len - i : end_idx]) == stop_word_ids:
+ if i > 0:
+ request.generated_tokens = request.generated_tokens[:-i]
+ return True, i
+
+ return False, 0
+
+ def test_stop_word_at_end_no_trim(self):
+ """Stop word is the last token — no trimming needed."""
+ # Speculative tokens: [tok1, STOP, tok3] appended, stop word at end of accepted
+ # But here STOP is at the very end after all tokens
+ request = MockDynamicInferenceRequest(
+ request_id=1, generated_tokens=[10, 20, 42], stop_word_ids=[[42]]
+ )
+ hit, trim = self._check_stop_words_for_request_post_append(
+ request, num_speculative_tokens=2
+ )
+ assert hit is True
+ assert trim == 0
+ assert request.generated_tokens == [10, 20, 42]
+
+ def test_stop_word_with_one_extra_token(self):
+ """Stop word is second-to-last — one extra token should be trimmed."""
+ # Speculative appended [tok1, STOP, tok3], STOP=42 at position -2
+ request = MockDynamicInferenceRequest(
+ request_id=1, generated_tokens=[10, 20, 42, 99], stop_word_ids=[[42]]
+ )
+ hit, trim = self._check_stop_words_for_request_post_append(
+ request, num_speculative_tokens=2
+ )
+ assert hit is True
+ assert trim == 1
+ assert request.generated_tokens == [10, 20, 42]
+
+ def test_stop_word_with_two_extra_tokens(self):
+ """Stop word is third-to-last — two extra tokens should be trimmed."""
+ # Speculative appended [STOP, tok2, tok3], STOP=42 at position -3
+ request = MockDynamicInferenceRequest(
+ request_id=1, generated_tokens=[10, 42, 77, 88], stop_word_ids=[[42]]
+ )
+ hit, trim = self._check_stop_words_for_request_post_append(
+ request, num_speculative_tokens=2
+ )
+ assert hit is True
+ assert trim == 2
+ assert request.generated_tokens == [10, 42]
+
+ def test_multi_token_stop_word_with_extra_tokens(self):
+ """Multi-token stop word found mid-speculative-batch."""
+ # Speculative appended [tok1, STOP_A, STOP_B, tok4], stop word is [STOP_A, STOP_B]
+ request = MockDynamicInferenceRequest(
+ request_id=1, generated_tokens=[10, 20, 42, 43, 99], stop_word_ids=[[42, 43]]
+ )
+ hit, trim = self._check_stop_words_for_request_post_append(
+ request, num_speculative_tokens=2
+ )
+ assert hit is True
+ assert trim == 1
+ assert request.generated_tokens == [10, 20, 42, 43]
+
+ def test_multi_token_stop_word_with_two_extra(self):
+ """Multi-token stop word with two extra tokens after."""
+ request = MockDynamicInferenceRequest(
+ request_id=1, generated_tokens=[10, 42, 43, 77, 88], stop_word_ids=[[42, 43]]
+ )
+ hit, trim = self._check_stop_words_for_request_post_append(
+ request, num_speculative_tokens=2
+ )
+ assert hit is True
+ assert trim == 2
+ assert request.generated_tokens == [10, 42, 43]
+
+ def test_no_stop_word_speculative(self):
+ """No stop word in speculative batch — nothing happens."""
+ request = MockDynamicInferenceRequest(
+ request_id=1, generated_tokens=[10, 20, 30, 40], stop_word_ids=[[42]]
+ )
+ hit, trim = self._check_stop_words_for_request_post_append(
+ request, num_speculative_tokens=2
+ )
+ assert hit is False
+ assert trim == 0
+ assert request.generated_tokens == [10, 20, 30, 40]
+
+ def test_stop_word_outside_speculative_window(self):
+ """Stop word exists but is outside the speculative search window."""
+ # Stop word [42] is at position -4, but num_speculative_tokens=2
+ # so we only check positions -1, -2, -3 (i=0,1,2)
+ request = MockDynamicInferenceRequest(
+ request_id=1, generated_tokens=[42, 10, 20, 30], stop_word_ids=[[42]]
+ )
+ hit, trim = self._check_stop_words_for_request_post_append(
+ request, num_speculative_tokens=2
+ )
+ assert hit is False
+ assert trim == 0
+
+ def test_log_probs_trimming_scenario(self):
+ """Verify that the trim count can be used to trim log probs correctly."""
+ # Simulate: speculative batch appended [tok1, STOP, tok3]
+ # Log probs: [lp1, lp2, lp3]
+ request = MockDynamicInferenceRequest(
+ request_id=1, generated_tokens=[10, 20, 42, 99], stop_word_ids=[[42]]
+ )
+ log_probs = [-1.5, -0.3, -2.1]
+
+ hit, trim = self._check_stop_words_for_request_post_append(
+ request, num_speculative_tokens=2
+ )
+ assert hit is True
+ assert trim == 1
+
+ # Trim log probs the same way the engine does
+ if trim > 0:
+ log_probs = log_probs[:-trim]
+
+ assert log_probs == [-1.5, -0.3]
+ assert request.generated_tokens == [10, 20, 42]
+
+ def test_speculative_stop_word_at_end(self):
+ """Test stop word at end of speculative tokens (no truncation needed)."""
+ # Speculative tokens appended: [200, 300], stop word is [300]
+ request = MockDynamicInferenceRequest(
+ request_id=1, generated_tokens=[100, 200, 300], stop_word_ids=[[300]]
+ )
+ assert self._check_stop_words_for_request_post_append(
+ request, num_speculative_tokens=2
+ ) == (True, 0)
+ assert request.generated_tokens == [100, 200, 300]
+
+ def test_speculative_stop_word_in_middle_truncates(self):
+ """Test that stop word in middle of speculative tokens truncates trailing tokens."""
+ # Speculative tokens appended: [200, 300, 400], stop word is [200]
+ # Token 200 is at position -3, so tokens [300, 400] should be truncated
+ request = MockDynamicInferenceRequest(
+ request_id=1, generated_tokens=[100, 200, 300, 400], stop_word_ids=[[200]]
+ )
+ assert self._check_stop_words_for_request_post_append(
+ request, num_speculative_tokens=3
+ ) == (True, 2)
+ assert request.generated_tokens == [100, 200]
+
+ def test_speculative_multi_token_stop_word_in_middle_truncates(self):
+ """Test multi-token stop word in middle of speculative tokens truncates."""
+ # Generated: [100, 200, 300, 400, 500], stop word is [200, 300]
+ # Stop word ends at -2, so tokens [400, 500] should be truncated
+ request = MockDynamicInferenceRequest(
+ request_id=1, generated_tokens=[100, 200, 300, 400, 500], stop_word_ids=[[200, 300]]
+ )
+ assert self._check_stop_words_for_request_post_append(
+ request, num_speculative_tokens=4
+ ) == (True, 2)
+ assert request.generated_tokens == [100, 200, 300]
+
+ def test_speculative_stop_word_not_found(self):
+ """Test no stop word found even with speculative scanning."""
+ request = MockDynamicInferenceRequest(
+ request_id=1, generated_tokens=[100, 200, 300, 400], stop_word_ids=[[999]]
+ )
+ assert self._check_stop_words_for_request_post_append(
+ request, num_speculative_tokens=3
+ ) == (False, 0)
+ assert request.generated_tokens == [100, 200, 300, 400]
+
+ def test_speculative_stop_word_one_trailing_token(self):
+ """Test stop word with exactly one trailing token to truncate."""
+ # Generated: [100, 200, 300], stop word is [200], one trailing token [300]
+ request = MockDynamicInferenceRequest(
+ request_id=1, generated_tokens=[100, 200, 300], stop_word_ids=[[200]]
+ )
+ assert self._check_stop_words_for_request_post_append(
+ request, num_speculative_tokens=2
+ ) == (True, 1)
+ assert request.generated_tokens == [100, 200]
class TestStopWordTrackingFlow:
diff --git a/tests/unit_tests/inference/test_wandb_logging.py b/tests/unit_tests/inference/test_wandb_logging.py
index cab464af503..02616f96e83 100644
--- a/tests/unit_tests/inference/test_wandb_logging.py
+++ b/tests/unit_tests/inference/test_wandb_logging.py
@@ -7,6 +7,7 @@
import pytest
import torch
+from megatron.core.inference.config import InferenceConfig
from megatron.core.inference.contexts.dynamic_context import DynamicInferenceContext
from megatron.core.inference.engines import DynamicInferenceEngine
from megatron.core.inference.inference_request import DynamicInferenceRequest
@@ -15,6 +16,7 @@
TextGenerationController,
)
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
+from megatron.core.transformer.transformer_config import TransformerConfig
from tests.unit_tests.test_utilities import Utils
@@ -28,16 +30,16 @@ def set_rounder(value):
class TestInferenceWandbLogging:
"""Test suite for wandb logging in inference."""
- def setup_method(self):
- """Set up test fixtures."""
+ @classmethod
+ def setup_class(cls):
Utils.initialize_model_parallel(
tensor_model_parallel_size=1, pipeline_model_parallel_size=1
)
model_parallel_cuda_manual_seed(123)
set_rounder(64)
- def teardown_method(self):
- """Clean up test fixtures."""
+ @classmethod
+ def teardown_class(cls):
set_rounder(64)
Utils.destroy_model_parallel()
@@ -50,20 +52,26 @@ def _get_dynamic_context(
max_sequence_length=512,
buffer_size_gb=0.03,
block_size_tokens=128,
+ logging_step_interval=0,
metrics_writer=None,
):
"""Helper to create a DynamicInferenceContext."""
return DynamicInferenceContext(
- params_dtype=params_dtype,
- num_layers=num_layers,
- kv_channels=kv_channels,
- num_attention_heads=num_attention_heads,
- max_sequence_length=max_sequence_length,
- num_cuda_graphs=None,
- buffer_size_gb=buffer_size_gb,
- block_size_tokens=block_size_tokens,
- metrics_writer=metrics_writer,
- unified_memory_level=0, # unit tests currently broken with UVM
+ model_config=TransformerConfig(
+ params_dtype=params_dtype,
+ num_layers=num_layers,
+ kv_channels=kv_channels,
+ num_attention_heads=num_attention_heads,
+ ),
+ inference_config=InferenceConfig(
+ max_sequence_length=max_sequence_length,
+ num_cuda_graphs=None,
+ buffer_size_gb=buffer_size_gb,
+ block_size_tokens=block_size_tokens,
+ unified_memory_level=0, # unit tests currently broken with UVM
+ logging_step_interval=logging_step_interval,
+ metrics_writer=metrics_writer,
+ ),
)
@pytest.mark.internal
@@ -141,7 +149,7 @@ def test_get_kvcache_utilization_stats_with_requests(self):
# Verify relationship: allocated_blocks + block_count_avail + 1 (dummy) = total
assert (
stats_after['allocated_blocks'] + stats_after['block_count_avail'] + 1
- == dynamic_context.block_allocator.total_count
+ == dynamic_context.kv_block_allocator.total_count
)
# Verify utilization bounds [0, 1]
@@ -195,12 +203,14 @@ def test_kvcache_utilization_stats_types(self):
@pytest.mark.internal
@patch('megatron.core.inference.engines.dynamic_engine.HAVE_WANDB', True)
def test_engine_logging_step_interval_zero(self):
- """Test that no logging occurs when inference_logging_step_interval is 0."""
+ """Test that no logging occurs when logging_step_interval is 0."""
mock_wandb = Mock()
mock_wandb.__name__ = "wandb"
mock_wandb.log = Mock()
- dynamic_context = self._get_dynamic_context(metrics_writer=mock_wandb)
+ dynamic_context = self._get_dynamic_context(
+ logging_step_interval=0, metrics_writer=mock_wandb
+ )
# Create mock controller with proper spec to pass isinstance checks
mock_controller = create_autospec(TextGenerationController, instance=True)
@@ -210,12 +220,7 @@ def test_engine_logging_step_interval_zero(self):
mock_controller.inference_wrapped_model.model.config = Mock()
mock_controller.inference_wrapped_model.model.config.cuda_graph_impl = "none"
- engine = DynamicInferenceEngine(
- controller=mock_controller,
- context=dynamic_context,
- random_seed=123,
- inference_logging_step_interval=0, # Disabled
- )
+ engine = DynamicInferenceEngine(controller=mock_controller, context=dynamic_context)
# Verify log was never called
mock_wandb.log.assert_not_called()
@@ -225,15 +230,16 @@ def test_paused_requests_in_stats(self):
"""Test that paused requests are correctly reflected in stats."""
set_rounder(1)
dynamic_context = DynamicInferenceContext(
- params_dtype=torch.float32,
- num_layers=2,
- kv_channels=64,
- num_attention_heads=8,
- max_sequence_length=128,
- num_cuda_graphs=None,
- buffer_size_gb=0.01, # Small buffer to force pausing
- block_size_tokens=32,
- unified_memory_level=0, # unit tests currently broken with UVM
+ model_config=TransformerConfig(
+ params_dtype=torch.float32, num_layers=2, kv_channels=64, num_attention_heads=8
+ ),
+ inference_config=InferenceConfig(
+ max_sequence_length=128,
+ num_cuda_graphs=None,
+ buffer_size_gb=0.01, # Small buffer to force pausing
+ block_size_tokens=32,
+ unified_memory_level=0, # unit tests currently broken with UVM
+ ),
)
# Add multiple requests to potentially trigger pausing
@@ -257,7 +263,7 @@ def test_paused_requests_in_stats(self):
@pytest.mark.internal
def test_metrics_writer_none_handling(self):
"""Test that engine handles None metrics_writer gracefully."""
- dynamic_context = self._get_dynamic_context(metrics_writer=None)
+ dynamic_context = self._get_dynamic_context(logging_step_interval=10, metrics_writer=None)
# Create mock controller with proper spec to pass isinstance checks
mock_controller = create_autospec(TextGenerationController, instance=True)
@@ -268,13 +274,8 @@ def test_metrics_writer_none_handling(self):
mock_controller.inference_wrapped_model.model.config.cuda_graph_impl = "none"
# Should not raise error even with logging interval set
- engine = DynamicInferenceEngine(
- controller=mock_controller,
- context=dynamic_context,
- random_seed=123,
- inference_logging_step_interval=10,
- )
+ engine = DynamicInferenceEngine(controller=mock_controller, context=dynamic_context)
# Verify engine was created successfully
- assert engine.inference_logging_step_interval == 10
- assert engine.context.metrics_writer is None
+ assert engine.logging_step_interval == 10
+ assert engine.metrics_writer is None
diff --git a/tests/unit_tests/inference/text_generation_controllers/test_encoder_decoder_text_generation_controller.py b/tests/unit_tests/inference/text_generation_controllers/test_encoder_decoder_text_generation_controller.py
index 93a208710fc..5bd39ec1324 100755
--- a/tests/unit_tests/inference/text_generation_controllers/test_encoder_decoder_text_generation_controller.py
+++ b/tests/unit_tests/inference/text_generation_controllers/test_encoder_decoder_text_generation_controller.py
@@ -1,3 +1,5 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
import random
import string
import time
@@ -12,9 +14,6 @@
from megatron.core.inference.contexts import StaticInferenceContext
from megatron.core.inference.inference_request import InferenceRequest, Status
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
from megatron.core.inference.model_inference_wrappers.t5.t5_inference_wrapper import (
T5InferenceWrapper,
)
@@ -85,19 +84,9 @@ def setup_method(self, method):
add_decoder=True,
).cuda()
- inference_wrapper_config = InferenceWrapperConfig(
- hidden_size=hidden_size,
- inference_batch_times_seqlen_threshold=-1,
- fp32_residual_connection=False,
- params_dtype=torch.float,
- padded_vocab_size=self.vocab_size,
- )
-
- inference_context = StaticInferenceContext.from_config(inference_wrapper_config)
+ inference_context = StaticInferenceContext(max_batch_size=8, max_sequence_length=2560)
- inference_wrapped_model = T5InferenceWrapper(
- t5_model, inference_wrapper_config, inference_context
- )
+ inference_wrapped_model = T5InferenceWrapper(t5_model, inference_context)
self.mock_tokenizer = mock.Mock()
diff --git a/tests/unit_tests/inference/text_generation_controllers/test_simple_text_generation_controller.py b/tests/unit_tests/inference/text_generation_controllers/test_text_generation_controller.py
similarity index 67%
rename from tests/unit_tests/inference/text_generation_controllers/test_simple_text_generation_controller.py
rename to tests/unit_tests/inference/text_generation_controllers/test_text_generation_controller.py
index 0885401e7a0..22caddaa789 100644
--- a/tests/unit_tests/inference/text_generation_controllers/test_simple_text_generation_controller.py
+++ b/tests/unit_tests/inference/text_generation_controllers/test_text_generation_controller.py
@@ -14,6 +14,7 @@
from transformer_engine.pytorch.fp8 import check_fp8_support
from megatron.core import parallel_state
+from megatron.core.inference.config import InferenceConfig
from megatron.core.inference.contexts import DynamicInferenceContext, StaticInferenceContext
from megatron.core.inference.contexts.dynamic_context import MaxSequenceLengthOverflowError
from megatron.core.inference.inference_request import (
@@ -24,9 +25,6 @@
from megatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapper import (
GPTInferenceWrapper,
)
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
from megatron.core.inference.sampling_params import SamplingParams
from megatron.core.inference.text_generation_controllers.text_generation_controller import (
TextGenerationController,
@@ -54,6 +52,11 @@ def setup_model(
batch_size: int = 4,
static: bool = True,
use_training_random_init: bool = False,
+ materialize_only_last_token_logits: bool = False,
+ num_speculative_tokens: int = 0,
+ block_size_tokens: int = 256,
+ enable_prefix_caching: bool = False,
+ max_requests: int = None,
):
Utils.initialize_model_parallel(
tensor_model_parallel_size=tensor_model_parallel_size,
@@ -100,37 +103,28 @@ def setup_model(
if dtype == torch.bfloat16:
gpt_model = Float16Module(gpt_model.config, gpt_model)
- inference_wrapper_config = InferenceWrapperConfig(
- hidden_size=self.hidden_size,
- inference_batch_times_seqlen_threshold=-1,
- inference_max_seq_length=2048,
- inference_max_requests=16 if fp8 else self.batch_size,
- fp32_residual_connection=False,
- params_dtype=dtype,
- padded_vocab_size=self.vocab_size,
- )
-
if static:
- inference_context = StaticInferenceContext.from_config(inference_wrapper_config)
+ inference_context = StaticInferenceContext(
+ max_batch_size=16 if fp8 else self.batch_size, max_sequence_length=2048
+ )
else:
inference_context = DynamicInferenceContext(
- params_dtype=dtype,
- num_layers=transformer_config.num_layers // pipeline_model_parallel_size,
- kv_channels=transformer_config.kv_channels,
- num_attention_heads=transformer_config.num_attention_heads,
- tensor_model_parallel_size=transformer_config.tensor_model_parallel_size,
- pipeline_model_parallel_size=transformer_config.pipeline_model_parallel_size,
- max_sequence_length=2048,
- buffer_size_gb=0.2,
- materialize_only_last_token_logits=False,
- use_flashinfer_fused_rope=None, # default to using flash-infer if available
- # this is for compatibility with the LTS environment
- unified_memory_level=0, # unit tests currently broken with UVM
+ model_config=transformer_config,
+ inference_config=InferenceConfig(
+ max_sequence_length=2048,
+ buffer_size_gb=0.2,
+ materialize_only_last_token_logits=materialize_only_last_token_logits,
+ use_flashinfer_fused_rope=None, # default to using flash-infer if available
+ # this is for compatibility with the LTS environment
+ unified_memory_level=0, # unit tests currently broken with UVM
+ num_speculative_tokens=num_speculative_tokens,
+ block_size_tokens=block_size_tokens,
+ enable_prefix_caching=enable_prefix_caching,
+ max_requests=max_requests,
+ ),
)
- inference_wrapped_model = GPTInferenceWrapper(
- gpt_model, inference_wrapper_config, inference_context
- )
+ inference_wrapped_model = GPTInferenceWrapper(gpt_model, inference_context)
inference_wrapped_model.model_is_pipeline_parallel = not (
parallel_state.is_pipeline_first_stage() and parallel_state.is_pipeline_last_stage()
@@ -142,7 +136,8 @@ def setup_model(
inference_wrapped_model=inference_wrapped_model, tokenizer=self.mock_tokenizer
)
- def teardown_method(self, method):
+ @classmethod
+ def teardown_class(cls):
Utils.destroy_model_parallel()
def test_sample_from_logits(self):
@@ -239,11 +234,15 @@ def test_sample_from_dynamic_logits(
self, backend: str, materialize_only_last_token_logits: bool
):
batch_size = 12
- self.setup_model(torch.float32, batch_size=batch_size, static=False)
+ self.setup_model(
+ torch.float32,
+ batch_size=batch_size,
+ static=False,
+ materialize_only_last_token_logits=materialize_only_last_token_logits,
+ )
self.mock_tokenizer.eod = self.vocab_size
context = self.text_generation_controller.inference_wrapped_model.inference_context
- context.materialize_only_last_token_logits = materialize_only_last_token_logits
# Prepare sampling params in human-readable format, to aid with test maintenance.
sampling_test_cases: List[Tuple[SamplingParams, List[int]]] = [
@@ -600,7 +599,7 @@ def test_add_bos_token(self):
self.mock_tokenizer.vocab_size = self.vocab_size
self.mock_tokenizer.bos = 0
self.mock_tokenizer.eod = self.vocab_size - 1
- self.mock_tokenizer.detokenize.side_effect = lambda x: ' '.join(
+ self.mock_tokenizer.detokenize.side_effect = lambda x, **_: ' '.join(
[
''.join(random.choices(string.ascii_letters, k=random.randint(1, len(prompt))))
for _ in range(len(x))
@@ -613,35 +612,73 @@ def test_add_bos_token(self):
random.randint(0, self.vocab_size - 1) for _ in range(len(prompt))
]
+ tokenizer = self.mock_tokenizer
+
# Test on a tokenizer that does not add BOS by default
- no_bos_to_no_bos = self.text_generation_controller.tokenize_prompt(prompt, add_BOS=False)
- assert no_bos_to_no_bos[0] != self.mock_tokenizer.bos
- no_bos_to_yes_bos = self.text_generation_controller.tokenize_prompt(prompt, add_BOS=True)
- assert no_bos_to_yes_bos[0] == self.mock_tokenizer.bos
- assert no_bos_to_yes_bos[1] != self.mock_tokenizer.bos
+ no_bos_to_no_bos = TextGenerationController.tokenize_prompt(
+ tokenizer, prompt, add_BOS=False
+ )
+ assert no_bos_to_no_bos[0] != tokenizer.bos
+ no_bos_to_yes_bos = TextGenerationController.tokenize_prompt(
+ tokenizer, prompt, add_BOS=True
+ )
+ assert no_bos_to_yes_bos[0] == tokenizer.bos
+ assert no_bos_to_yes_bos[1] != tokenizer.bos
# Force the first token to be BOS to emulate a tokenizer that does add BOS by default
- self.mock_tokenizer.tokenize.return_value[0] = self.mock_tokenizer.bos
+ tokenizer.tokenize.return_value[0] = tokenizer.bos
- yes_bos_to_no_bos = self.text_generation_controller.tokenize_prompt(prompt, add_BOS=False)
- assert yes_bos_to_no_bos[0] != self.mock_tokenizer.bos
- yes_bos_to_yes_bos = self.text_generation_controller.tokenize_prompt(prompt, add_BOS=True)
- assert yes_bos_to_yes_bos[0] == self.mock_tokenizer.bos
- assert yes_bos_to_yes_bos[1] != self.mock_tokenizer.bos
+ yes_bos_to_no_bos = TextGenerationController.tokenize_prompt(
+ tokenizer, prompt, add_BOS=False
+ )
+ assert yes_bos_to_no_bos[0] != tokenizer.bos
+ yes_bos_to_yes_bos = TextGenerationController.tokenize_prompt(
+ tokenizer, prompt, add_BOS=True
+ )
+ assert yes_bos_to_yes_bos[0] == tokenizer.bos
+ assert yes_bos_to_yes_bos[1] != tokenizer.bos
# Test on an input that has had multiple BOS added
- self.mock_tokenizer.tokenize.return_value[1] = self.mock_tokenizer.bos
+ tokenizer.tokenize.return_value[1] = tokenizer.bos
- many_bos_to_no_bos = self.text_generation_controller.tokenize_prompt(prompt, add_BOS=False)
- assert many_bos_to_no_bos[0] != self.mock_tokenizer.bos
- many_bos_to_yes_bos = self.text_generation_controller.tokenize_prompt(prompt, add_BOS=True)
- assert many_bos_to_yes_bos[0] == self.mock_tokenizer.bos
- assert many_bos_to_yes_bos[1] != self.mock_tokenizer.bos
+ many_bos_to_no_bos = TextGenerationController.tokenize_prompt(
+ tokenizer, prompt, add_BOS=False
+ )
+ assert many_bos_to_no_bos[0] != tokenizer.bos
+ many_bos_to_yes_bos = TextGenerationController.tokenize_prompt(
+ tokenizer, prompt, add_BOS=True
+ )
+ assert many_bos_to_yes_bos[0] == tokenizer.bos
+ assert many_bos_to_yes_bos[1] != tokenizer.bos
# Test the assert triggered when the tokenizer has no bos
- self.mock_tokenizer.bos = None
+ tokenizer.bos = None
with pytest.raises(AssertionError):
- self.text_generation_controller.tokenize_prompt(prompt, add_BOS=True)
+ TextGenerationController.tokenize_prompt(tokenizer, prompt, add_BOS=True)
+
+ @pytest.mark.parametrize("remove_EOD", [True, False])
+ def test_remove_eod_token(self, remove_EOD):
+ self.setup_model(torch.float32)
+
+ self.mock_tokenizer.vocab_size = self.vocab_size
+ self.mock_tokenizer.bos = 0
+ self.mock_tokenizer.eod = self.vocab_size - 1
+ self.mock_tokenizer.detokenize.side_effect = lambda x, **_: ' '.join(f"T{t}" for t in x)
+
+ tokenizer = self.mock_tokenizer
+ eod = tokenizer.eod
+ detok = TextGenerationController.detokenize
+
+ # No trailing EOD.
+ assert detok(tokenizer, [1, 2, 3], remove_EOD=remove_EOD) == "T1 T2 T3"
+
+ # Single trailing EOD.
+ result = detok(tokenizer, [1, 2, eod], remove_EOD=remove_EOD)
+ assert result == ("T1 T2" if remove_EOD else f"T1 T2 T{eod}")
+
+ # Multiple trailing EOD.
+ result = detok(tokenizer, [1, eod, eod, eod], remove_EOD=remove_EOD)
+ assert result == ("T1" if remove_EOD else f"T1 T{eod} T{eod} T{eod}")
def test_zero_tokens_generated_batch_vs_single(self):
"""
@@ -758,11 +795,15 @@ def test_dynamic_top_n_logprobs_calculation(
3. Correct number of tokens are returned for each request
"""
batch_size = 4
- self.setup_model(torch.bfloat16, batch_size=batch_size, static=False)
+ self.setup_model(
+ torch.bfloat16,
+ batch_size=batch_size,
+ static=False,
+ materialize_only_last_token_logits=materialize_only_last_token_logits,
+ )
self.mock_tokenizer.eod = self.vocab_size
context = self.text_generation_controller.inference_wrapped_model.inference_context
- context.materialize_only_last_token_logits = materialize_only_last_token_logits
# Prepare sampling params
top_n = 5
@@ -1022,3 +1063,348 @@ def test_sampled_tokens_match_with_parallelism(self, static, tp_size, pp_size):
assert (
expected == actual
), f"Rank {i} tokens differ from rank {local_rank} tokens for request {j}"
+
+ @pytest.mark.internal
+ def test_speculative_verify_tokens(self):
+ """Test consecutive token acceptance logic for speculative decoding."""
+ self.setup_model(torch.float32, static=False, num_speculative_tokens=2, max_requests=2)
+
+ # Enable speculative decoding
+ self.text_generation_controller.num_speculative_tokens = 2
+ ctx = self.text_generation_controller.inference_wrapped_model.inference_context
+ ctx.total_request_count = 2
+ ctx.paused_request_count = 0
+ ctx.request_in_prefill_status_tensor = torch.tensor(
+ [0, 0], device='cuda'
+ ) # Decode requests
+ ctx.request_query_lengths = torch.tensor(
+ [3, 3], dtype=torch.int32, device='cuda'
+ ) # 1 sampled + 2 spec
+
+ # Init accepted tokens tensors
+ self.text_generation_controller._init_mtp_sampling_tensor()
+
+ # Mock inputs: [Req 1 sampled, Req 1 spec1, Req 1 spec2, Req 2 sampled, Req 2 spec1, Req 2 spec2]
+ # Target tokens (what the model was fed): [T0, T1, T2, T3, T4, T5]
+ input_ids = torch.tensor([[10, 11, 12, 20, 21, 22]], device='cuda')
+
+ # We need the sampling function to return a 1D tensor for base logits,
+ # and a 1D tensor for the flattened MTP logits.
+ def mock_sampling_func(logits, *args, **kwargs):
+ if logits.shape[0] == 6:
+ # Base logits -> return 1D tensor of shape [6]
+ # Req 1: Predicts [11, 12, 99]. Matches T1, T2. Rejects T3. -> Accepts 2 spec tokens.
+ # Req 2: Predicts [99, 22, 23]. Fails at first spec token (99 != 21). -> Accepts 0 spec tokens.
+ return torch.tensor([11, 12, 99, 99, 22, 23], dtype=torch.long, device='cuda')
+ else:
+ # MTP logits -> return 1D tensor of shape [12]
+ # The verification logic only uses base tokens, so we can return zeros here.
+ return torch.zeros((12,), dtype=torch.long, device='cuda')
+
+ # Override sampling to return our predictable mock outputs
+ self.text_generation_controller._torch_sampling_buckets = [([0, 1], 1.0, 1, 0.0)]
+ self.text_generation_controller._torch_sampling_func = mock.MagicMock(
+ side_effect=mock_sampling_func
+ )
+
+ # Mock logits matching input shape
+ logits = torch.randn(1, 6, self.vocab_size, device='cuda')
+
+ self.text_generation_controller._dynamic_step_sample_logits_and_verify_tokens(
+ logits, input_ids
+ )
+
+ # Verify acceptance counts
+ accepted_counts = self.text_generation_controller._accepted_token_counts_per_request[:2]
+ assert torch.equal(accepted_counts, torch.tensor([2, 0], device='cuda'))
+
+ # Verify accepted tokens tensor
+ accepted_tokens = self.text_generation_controller._accepted_tokens_per_request[:2]
+ # Req 1 accepted 2 tokens: 11, 12
+ assert torch.equal(accepted_tokens[0], torch.tensor([11, 12], device='cuda'))
+ # Req 2 accepted 0 tokens, should remain -1
+ assert torch.equal(accepted_tokens[1], torch.tensor([-1, -1], device='cuda'))
+
+ @pytest.mark.internal
+ @pytest.mark.parametrize("is_hybrid_model", [False, True])
+ def test_rewind_kv_cache(self, is_hybrid_model):
+ """Test KV cache state is properly rewound for rejected speculative tokens."""
+ self.setup_model(
+ torch.float32,
+ static=False,
+ num_speculative_tokens=3,
+ block_size_tokens=4,
+ max_requests=16,
+ )
+ self.text_generation_controller.num_speculative_tokens = 3
+ ctx = self.text_generation_controller.inference_wrapped_model.inference_context
+ ctx.total_request_count = 2
+ ctx.paused_request_count = 0
+ ctx.request_in_prefill_status_tensor = torch.tensor([0, 0], device='cuda')
+
+ # Initialize allocator and states
+ ctx.kv_block_allocator.total_avail = 100
+ ctx.request_kv_length_offsets[:2] = torch.tensor([10, 15], device='cuda')
+ ctx.request_kv_block_counts[:2] = torch.tensor([3, 4], device='cuda')
+
+ # Req 0: offset 2. Rewinding 2 tokens -> offset 0. No block released.
+ # Req 1: offset 1. Rewinding 3 tokens -> offset 2 (prev block). 1 block released.
+ ctx.request_last_kv_block_offset[:2] = torch.tensor([2, 1], device='cuda')
+ ctx.request_last_kv_block_id[:2] = torch.tensor([50, 60], device='cuda')
+ ctx.request_to_kv_block_ids[:2, :4] = torch.tensor(
+ [[48, 49, 50, -1], [57, 58, 59, 60]], dtype=torch.int, device='cuda'
+ )
+
+ if is_hybrid_model:
+ ctx.is_hybrid_model = True
+ ctx.mamba_metadata = mock.MagicMock()
+ ctx.mamba_metadata.request_to_mamba_state_idx = torch.tensor([0, 1], device='cuda')
+ ctx.mamba_ssm_states = torch.zeros((1, 2, 16), device='cuda')
+ ctx.mamba_intermediate_ssm_states = torch.ones((1, 2, 4, 16), device='cuda') * 99
+ ctx.mamba_conv_states = torch.zeros((1, 2, 8), device='cuda')
+ ctx.mamba_intermediate_conv_states = torch.ones((1, 2, 4, 8), device='cuda') * 77
+
+ # Mock accepted token counts: Req 0 accepts 1 (rejects 2), Req 1 accepts 0 (rejects 3)
+ self.text_generation_controller._init_mtp_sampling_tensor()
+ self.text_generation_controller._accepted_token_counts_per_request = torch.tensor(
+ [1, 0], device='cuda'
+ )
+
+ self.text_generation_controller._rewind_kv_cache()
+
+ # Assert offsets updated
+ assert torch.equal(
+ ctx.request_last_kv_block_offset[:2],
+ torch.tensor([0, 2], dtype=torch.int, device='cuda'),
+ )
+ assert torch.equal(
+ ctx.request_kv_length_offsets[:2], torch.tensor([8, 12], dtype=torch.int, device='cuda')
+ )
+
+ # Assert block counts and IDs updated for boundary crossing
+ assert torch.equal(
+ ctx.request_kv_block_counts[:2], torch.tensor([3, 3], dtype=torch.int, device='cuda')
+ )
+ assert torch.equal(
+ ctx.request_last_kv_block_id[:2], torch.tensor([50, 59], dtype=torch.int, device='cuda')
+ )
+
+ # Assert released block is cleared
+ assert ctx.request_to_kv_block_ids[1, 3].item() == -1
+ assert ctx.kv_block_allocator.total_avail == 101 # 1 block released
+
+ if is_hybrid_model:
+ # Check Mamba state was restored from intermediate cache based on accepted counts
+ assert torch.all(ctx.mamba_ssm_states[:, 0] == 99) # Req 0 accepted 1, loaded index 1
+ assert torch.all(ctx.mamba_ssm_states[:, 1] == 99) # Req 1 accepted 0, loaded index 0
+ assert torch.all(ctx.mamba_conv_states[:, 0] == 77) # Req 0 accepted 1, loaded index 1
+ assert torch.all(ctx.mamba_conv_states[:, 1] == 77) # Req 1 accepted 0, loaded index 0
+
+ @pytest.mark.internal
+ def test_speculative_multinomial_sampling(self):
+ """Test that speculative decoding can successfully use non-greedy sampling
+ (top_k > 1, top_p > 0) by flattening 3D MTP logits for torch.multinomial."""
+ num_spec = 3
+ self.setup_model(
+ torch.float32, static=False, num_speculative_tokens=num_spec, max_requests=2
+ )
+
+ # Enable speculative decoding
+ self.text_generation_controller.num_speculative_tokens = num_spec
+ ctx = self.text_generation_controller.inference_wrapped_model.inference_context
+ ctx.total_request_count = 2
+ ctx.paused_request_count = 0
+ ctx.request_in_prefill_status_tensor = torch.tensor(
+ [0, 0], device='cuda'
+ ) # Decode requests
+ # query lengths for decode with spec tokens is (1 + num_spec) = 4
+ ctx.request_query_lengths = torch.tensor([4, 4], dtype=torch.int32, device='cuda')
+
+ # Setup inputs
+ input_ids = torch.randint(0, self.vocab_size, (1, 8), device='cuda')
+
+ # Create random logits
+ # Base logits shape: [1, 8, vocab_size]
+ logits = torch.randn(1, 8, self.vocab_size, device='cuda')
+
+ # Set up a bucket that forces multinomial sampling (top_p = 0.9, top_k = 0)
+ # _torch_sampling_buckets format: (indices, temp, top_k, top_p)
+ self.text_generation_controller._torch_sampling_buckets = [([0, 1], 1.0, 0, 0.9)]
+
+ # Since we are actually testing the internal math of `_torch_sampling_func` handling the shapes,
+ # we DO NOT mock `_torch_sampling_func` here. We want it to run natively to prove it doesn't crash.
+
+ try:
+ self.text_generation_controller._dynamic_step_sample_logits_and_verify_tokens(
+ logits, input_ids
+ )
+ except RuntimeError as e:
+ if "prob_dist must be 1 or 2 dim" in str(e):
+ pytest.fail("MTP logits were not flattened before calling multinomial sampling.")
+ else:
+ raise e
+
+ # Validate that sampling produced output arrays of the correct sizes
+ active_request_count = ctx.total_request_count
+ sampled_tokens = self.text_generation_controller._sampled_tokens_cuda[:active_request_count]
+ sampled_mtp_tokens = self.text_generation_controller._sampled_mtp_tokens_cuda[
+ :, :active_request_count
+ ]
+
+ assert sampled_tokens.shape == (2,)
+ assert sampled_mtp_tokens.shape == (num_spec, 2)
+
+ @pytest.mark.internal
+ def test_rewind_kv_cache_with_prefix_caching_ref_counts(self):
+ """Test that _rewind_kv_cache correctly decrements ref counts on shared blocks
+ when speculative token rejection causes a block boundary crossing."""
+ self.setup_model(
+ torch.float32,
+ static=False,
+ num_speculative_tokens=2,
+ block_size_tokens=4,
+ enable_prefix_caching=True,
+ max_requests=16,
+ )
+
+ ctx = self.text_generation_controller.inference_wrapped_model.inference_context
+ ctx.total_request_count = 2
+ ctx.paused_request_count = 0
+ ctx.request_in_prefill_status_tensor = torch.tensor([0, 0], device='cuda')
+
+ # Req 0: 3 blocks, offset 1 in last block. Rewinding 1 token -> no block release.
+ # Req 1: 3 blocks, offset 0 in last block. Rewinding 2 tokens -> crosses back, release block.
+ ctx.request_kv_length_offsets[:2] = torch.tensor([9, 9], device='cuda')
+ ctx.request_kv_block_counts[:2] = torch.tensor([3, 3], device='cuda')
+ ctx.request_last_kv_block_offset[:2] = torch.tensor([1, 0], device='cuda')
+ ctx.request_last_kv_block_id[:2] = torch.tensor([10, 20], device='cuda')
+ ctx.request_to_kv_block_ids[:2, :3] = torch.tensor(
+ [[8, 9, 10], [18, 19, 20]], dtype=torch.int, device='cuda'
+ )
+
+ # Set ref counts: block 20 is shared (ref=2), block 10 is exclusive (ref=1).
+ ctx.kv_block_allocator.block_ref_counts[20] = 2
+ ctx.kv_block_allocator.block_ref_counts[10] = 1
+
+ initial_avail = ctx.kv_block_allocator.total_avail
+
+ # Req 0 accepts 1 (rewinds 1), Req 1 accepts 0 (rewinds 2, crosses boundary).
+ self.text_generation_controller._init_mtp_sampling_tensor()
+ self.text_generation_controller._accepted_token_counts_per_request = torch.tensor(
+ [1, 0], device='cuda'
+ )
+
+ self.text_generation_controller._rewind_kv_cache()
+
+ # Req 1 should have released block 20 (ref count decremented).
+ assert ctx.kv_block_allocator.block_ref_counts[20].item() == 1
+ # Block 10 should be untouched.
+ assert ctx.kv_block_allocator.block_ref_counts[10].item() == 1
+
+ @pytest.mark.internal
+ def test_rewind_kv_cache_does_not_release_shared_prefix_blocks(self):
+ """Test that rewinding only releases the last block, never shared prefix blocks."""
+ self.setup_model(
+ torch.float32,
+ static=False,
+ num_speculative_tokens=3,
+ block_size_tokens=4,
+ max_requests=16,
+ )
+
+ ctx = self.text_generation_controller.inference_wrapped_model.inference_context
+ ctx.total_request_count = 1
+ ctx.paused_request_count = 0
+ ctx.request_in_prefill_status_tensor = torch.tensor([0], device='cuda')
+
+ # 4 blocks. Offset 2 in last block. Rewinding 3 crosses into previous block.
+ ctx.request_kv_length_offsets[:1] = torch.tensor([14], device='cuda')
+ ctx.request_kv_block_counts[:1] = torch.tensor([4], device='cuda')
+ ctx.request_last_kv_block_offset[:1] = torch.tensor([2], device='cuda')
+ ctx.request_last_kv_block_id[:1] = torch.tensor([40], device='cuda')
+ ctx.request_to_kv_block_ids[0, :4] = torch.tensor(
+ [10, 20, 30, 40], dtype=torch.int, device='cuda'
+ )
+
+ # Blocks 10, 20 are shared prefix blocks. Block 30, 40 are exclusive.
+ ctx.kv_block_allocator.total_avail = 50
+
+ self.text_generation_controller._init_mtp_sampling_tensor()
+ self.text_generation_controller._accepted_token_counts_per_request = torch.tensor(
+ [0], device='cuda'
+ )
+
+ self.text_generation_controller._rewind_kv_cache()
+
+ # Only block 40 should be released, not blocks 10, 20, or 30.
+ assert ctx.request_kv_block_counts[0].item() == 3
+ assert ctx.request_last_kv_block_id[0].item() == 30
+ assert ctx.request_to_kv_block_ids[0, 3].item() == -1
+ assert ctx.kv_block_allocator.total_avail == 51 # exactly 1 block released
+
+ # Prefix blocks remain in request_to_kv_block_ids.
+ assert ctx.request_to_kv_block_ids[0, 0].item() == 10
+ assert ctx.request_to_kv_block_ids[0, 1].item() == 20
+ assert ctx.request_to_kv_block_ids[0, 2].item() == 30
+
+ @pytest.mark.internal
+ def test_speculative_mtp_position_ids_with_prefill(self):
+ """Test that _compute_serial_mtp_and_sample uses the correct position IDs
+ for a mixed batch of prefill and decode requests."""
+ self.setup_model(torch.float32, static=False, num_speculative_tokens=2, max_requests=2)
+
+ self.text_generation_controller.num_speculative_tokens = 2
+ self.text_generation_controller.num_mtp_heads = 2
+ ctx = self.text_generation_controller.inference_wrapped_model.inference_context
+ ctx.total_request_count = 2
+ ctx.paused_request_count = 0
+
+ # Req 0: Decode, Req 1: Prefill
+ ctx.request_in_prefill_status_tensor = torch.tensor([0, 1], device='cuda')
+
+ # Req 0 has 10 previous tokens, just processed 3 (1 base + 2 spec)
+ # Req 1 has 0 previous tokens, just processed 15 (prefill)
+ ctx.request_kv_length_offsets[:2] = torch.tensor([10, 0], dtype=torch.int32, device='cuda')
+ ctx.request_query_lengths[:2] = torch.tensor([3, 15], dtype=torch.int32, device='cuda')
+
+ self.text_generation_controller._init_mtp_sampling_tensor()
+ # Mock base token sampling (the first tokens fed into MTP)
+ self.text_generation_controller._sampled_tokens_cuda[:2] = torch.tensor(
+ [100, 200], device='cuda'
+ )
+
+ # Mock the MTP computation to record the position_ids it receives
+ unwrapped_model = self.text_generation_controller.inference_wrapped_model.model
+ unwrapped_model._decoder_hidden_states_cache = torch.randn(2, 1, 32, device='cuda')
+ self.text_generation_controller._last_accepted_seq_indices = torch.tensor(
+ [0, 1], device='cuda'
+ )
+
+ captured_position_ids = []
+
+ def mock_compute_mtp_single_step(hidden_states, next_token_ids, position_ids, depth):
+ captured_position_ids.append(position_ids.clone())
+ return hidden_states, torch.randn(2, 1, self.vocab_size, device='cuda')
+
+ unwrapped_model.compute_mtp_single_step = mock.MagicMock(
+ side_effect=mock_compute_mtp_single_step
+ )
+
+ # Mock _sample_from_logits_2d to return arbitrary dummy tokens
+ self.text_generation_controller._sample_from_logits_2d = mock.MagicMock(
+ return_value=torch.tensor([101, 201], device='cuda')
+ )
+
+ self.text_generation_controller._compute_serial_mtp_and_sample()
+
+ # The base_position for Req 0 should be 10 + 3 = 13
+ # The base_position for Req 1 should be 0 + 15 = 15
+ assert len(captured_position_ids) == 2
+ # Depth 0:
+ assert torch.equal(
+ captured_position_ids[0].squeeze(0), torch.tensor([13, 15], device='cuda')
+ )
+ # Depth 1:
+ assert torch.equal(
+ captured_position_ids[1].squeeze(0), torch.tensor([14, 16], device='cuda')
+ )
diff --git a/tests/unit_tests/inference/text_generation_controllers/test_vlm_text_generation_controller.py b/tests/unit_tests/inference/text_generation_controllers/test_vlm_text_generation_controller.py
index 31bf415ba56..5344ae9f8eb 100644
--- a/tests/unit_tests/inference/text_generation_controllers/test_vlm_text_generation_controller.py
+++ b/tests/unit_tests/inference/text_generation_controllers/test_vlm_text_generation_controller.py
@@ -1,11 +1,11 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
import copy
-import os
import random
import string
import time
-from argparse import Namespace
from collections import OrderedDict
-from typing import Dict
+from typing import Dict, List
from unittest import mock
import pytest
@@ -13,9 +13,6 @@
from megatron.core.inference.contexts import StaticInferenceContext
from megatron.core.inference.inference_request import InferenceRequest, Status, VLMInferenceRequest
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
from megatron.core.inference.model_inference_wrappers.multimodal.vlm_inference_wrapper import (
VLMInferenceWrapper,
)
@@ -23,12 +20,13 @@
from megatron.core.inference.text_generation_controllers.vlm_text_generation_controller import (
VLMTextGenerationController,
)
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_spec
+from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_submodules
from megatron.core.models.multimodal.llava_model import LLaVAModel
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
-from megatron.core.transformer.enums import AttnBackend
from megatron.core.transformer.module import Float16Module
+from megatron.core.transformer.spec_utils import ModuleSpec
from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.transformer.transformer_layer import TransformerLayer
from tests.unit_tests.test_utilities import Utils
@@ -69,15 +67,19 @@ def setup_method(self, method):
bf16=True,
)
- language_layer_spec = get_gpt_layer_local_spec()
- vision_layer_spec = copy.deepcopy(language_layer_spec)
- vision_projection_spec = copy.deepcopy(language_layer_spec.submodules.mlp.submodules)
+ language_layer_submodules = get_gpt_layer_local_submodules()
+ vision_layer_spec = ModuleSpec(
+ module=TransformerLayer, submodules=copy.deepcopy(language_layer_submodules)
+ )
+ vision_projection_spec = copy.deepcopy(language_layer_submodules.mlp.submodules)
language_config.language_model_type = "dummy"
vision_config.vision_model_type = "clip"
self.model = LLaVAModel(
language_transformer_config=language_config,
- language_transformer_layer_spec=language_layer_spec,
+ language_transformer_layer_spec=ModuleSpec(
+ module=TransformerLayer, submodules=language_layer_submodules
+ ),
language_vocab_size=self.language_vocab_size,
language_max_sequence_length=self.language_max_sequence_length,
vision_transformer_config=vision_config,
@@ -92,19 +94,9 @@ def setup_method(self, method):
self.image_token_index = self.model.image_token_index
self.model = Float16Module(self.model.config, self.model)
- inference_wrapper_config = InferenceWrapperConfig(
- hidden_size=self.language_hidden_size,
- inference_batch_times_seqlen_threshold=-1,
- fp32_residual_connection=False,
- params_dtype=torch.float,
- padded_vocab_size=self.language_vocab_size,
- )
-
- inference_context = StaticInferenceContext.from_config(inference_wrapper_config)
+ inference_context = StaticInferenceContext(max_batch_size=8, max_sequence_length=2560)
- inference_wrapped_model = VLMInferenceWrapper(
- self.model, inference_wrapper_config, inference_context
- )
+ inference_wrapped_model = VLMInferenceWrapper(self.model, inference_context)
self.mock_tokenizer = mock.Mock()
diff --git a/tests/unit_tests/models/test_bert_model.py b/tests/unit_tests/models/test_bert_model.py
index c878fd4d3b5..db7b8255776 100644
--- a/tests/unit_tests/models/test_bert_model.py
+++ b/tests/unit_tests/models/test_bert_model.py
@@ -1,21 +1,22 @@
# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
import os
-from importlib.metadata import version
import pytest
import torch
from packaging.version import Version as PkgVersion
-from pytest_mock import mocker
from megatron.core.models.bert.bert_layer_specs import (
bert_layer_local_spec,
- bert_layer_with_transformer_engine_spec,
+ get_bert_layer_with_transformer_engine_spec,
+ get_bert_layer_with_transformer_engine_submodules,
)
from megatron.core.models.bert.bert_model import BertModel
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer.enums import AttnBackend, AttnMaskType
+from megatron.core.transformer.spec_utils import ModuleSpec
from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.transformer.transformer_layer import TransformerLayer
from tests.unit_tests.test_utilities import Utils
@@ -40,7 +41,7 @@ def setup_method(self, method):
self.bert_model = BertModel(
config=transformer_config,
num_tokentypes=0,
- transformer_layer_spec=bert_layer_with_transformer_engine_spec,
+ transformer_layer_spec=get_bert_layer_with_transformer_engine_spec(),
vocab_size=100,
max_sequence_length=4,
)
@@ -112,7 +113,7 @@ def setup_method(self, method):
self.bert_model = BertModel(
config=self.transformer_config,
num_tokentypes=0,
- transformer_layer_spec=bert_layer_with_transformer_engine_spec,
+ transformer_layer_spec=get_bert_layer_with_transformer_engine_spec(),
vocab_size=100,
max_sequence_length=4,
)
@@ -139,16 +140,15 @@ def test_local_spec_exception(self, mocker):
@pytest.mark.internal
def test_transformer_engine_version_1_10(self, mocker):
- bert_layer_with_transformer_engine_spec.submodules.self_attention.params[
- 'attn_mask_type'
- ] == AttnMaskType.arbitrary
+ submodules = get_bert_layer_with_transformer_engine_submodules()
+ submodules.self_attention.params['attn_mask_type'] = AttnMaskType.arbitrary
mocker.patch("megatron.core.utils.get_te_version", return_value=PkgVersion("1.10"))
- self.bert_model.transformer_layer_spec = bert_layer_with_transformer_engine_spec
+ self.bert_model.transformer_layer_spec = ModuleSpec(
+ module=TransformerLayer, submodules=submodules
+ )
attn_mask_dimensions = self.bert_model._sanity_check_attention_and_get_attn_mask_dimension()
- attn_mask_type = self.bert_model.transformer_layer_spec.submodules.self_attention.params[
- 'attn_mask_type'
- ]
+ attn_mask_type = submodules.self_attention.params['attn_mask_type']
assert (
attn_mask_type == AttnMaskType.padding
), f"Exepcted attn mask type to be padding, but got {attn_mask_type}"
@@ -160,7 +160,7 @@ def test_transformer_engine_version_1_10(self, mocker):
def test_transformer_engine_version_1_7_to_1_10_flash_attn(self, mocker):
self.bert_model.config.attention_backend = AttnBackend.flash
mocker.patch("megatron.core.utils.get_te_version", return_value=PkgVersion("1.8"))
- self.bert_model.transformer_layer_spec = bert_layer_with_transformer_engine_spec
+ self.bert_model.transformer_layer_spec = get_bert_layer_with_transformer_engine_spec()
attn_mask_dimensions = self.bert_model._sanity_check_attention_and_get_attn_mask_dimension()
assert (
attn_mask_dimensions == "b11s"
@@ -170,15 +170,14 @@ def test_transformer_engine_version_1_7_to_1_10_flash_attn(self, mocker):
@pytest.mark.flaky
@pytest.mark.flaky_in_dev
def test_transformer_engine_version_1_7_to_1_10_rng_error(self, mocker):
- bert_layer_with_transformer_engine_spec.submodules.self_attention.params[
- 'attn_mask_type'
- ] == AttnMaskType.padding
+ submodules = get_bert_layer_with_transformer_engine_submodules()
+ submodules.self_attention.params['attn_mask_type'] = AttnMaskType.padding
mocker.patch("megatron.core.utils.get_te_version", return_value=PkgVersion("1.8"))
with pytest.raises(Exception) as exc_info:
self.bert_model = BertModel(
config=self.transformer_config,
num_tokentypes=0,
- transformer_layer_spec=bert_layer_with_transformer_engine_spec,
+ transformer_layer_spec=ModuleSpec(module=TransformerLayer, submodules=submodules),
vocab_size=100,
max_sequence_length=4,
)
@@ -191,15 +190,14 @@ def test_transformer_engine_version_1_7_to_1_10_rng_error(self, mocker):
@pytest.mark.internal
def test_transformer_engine_version_1_7_to_1_10_unfused_attention(self, mocker):
self.bert_model.config.attention_backend = AttnBackend.unfused
- bert_layer_with_transformer_engine_spec.submodules.self_attention.params[
- 'attn_mask_type'
- ] == AttnMaskType.padding
+ submodules = get_bert_layer_with_transformer_engine_submodules()
+ submodules.self_attention.params['attn_mask_type'] = AttnMaskType.padding
mocker.patch("megatron.core.utils.get_te_version", return_value=PkgVersion("1.8"))
- self.bert_model.transformer_layer_spec = bert_layer_with_transformer_engine_spec
+ self.bert_model.transformer_layer_spec = ModuleSpec(
+ module=TransformerLayer, submodules=submodules
+ )
attn_mask_dimensions = self.bert_model._sanity_check_attention_and_get_attn_mask_dimension()
- attn_mask_type = self.bert_model.transformer_layer_spec.submodules.self_attention.params[
- 'attn_mask_type'
- ]
+ attn_mask_type = submodules.self_attention.params['attn_mask_type']
assert (
attn_mask_type == AttnMaskType.arbitrary
), f"Exepcted attn mask type to be arbitrary, but got {attn_mask_type}"
@@ -218,7 +216,7 @@ def test_transformer_engine_version_less_than_1_7(self, mocker):
self.bert_model = BertModel(
config=self.transformer_config,
num_tokentypes=0,
- transformer_layer_spec=bert_layer_with_transformer_engine_spec,
+ transformer_layer_spec=get_bert_layer_with_transformer_engine_spec(),
vocab_size=100,
max_sequence_length=4,
)
diff --git a/tests/unit_tests/models/test_gpt_model.py b/tests/unit_tests/models/test_gpt_model.py
index cf3bd40ee4b..336ff0552b0 100644
--- a/tests/unit_tests/models/test_gpt_model.py
+++ b/tests/unit_tests/models/test_gpt_model.py
@@ -3,6 +3,7 @@
import inspect
import os
from datetime import timedelta
+from unittest.mock import MagicMock, patch
import pytest
import torch
@@ -12,6 +13,7 @@
from megatron.core import parallel_state
from megatron.core.hyper_comm_grid import HyperCommGrid
+from megatron.core.inference.config import InferenceConfig
from megatron.core.inference.contexts.dynamic_context import DynamicInferenceContext
from megatron.core.inference.inference_request import DynamicInferenceRequest
from megatron.core.inference.sampling_params import SamplingParams
@@ -121,7 +123,6 @@ def test_get_mlp_module_spec_interface():
"num_experts": inspect.Parameter.POSITIONAL_OR_KEYWORD,
"moe_grouped_gemm": inspect.Parameter.POSITIONAL_OR_KEYWORD,
"fp8": inspect.Parameter.POSITIONAL_OR_KEYWORD,
- "moe_use_legacy_grouped_gemm": inspect.Parameter.POSITIONAL_OR_KEYWORD,
"use_te_op_fuser": inspect.Parameter.POSITIONAL_OR_KEYWORD,
}
@@ -130,7 +131,6 @@ def test_get_mlp_module_spec_interface():
"num_experts": None,
"moe_grouped_gemm": False,
"fp8": None,
- "moe_use_legacy_grouped_gemm": False,
"use_te_op_fuser": False,
}
@@ -392,14 +392,18 @@ def test_dynamic_inference_padding_with_fp8(self):
config = self.gpt_model.config
inference_context = DynamicInferenceContext(
- params_dtype=config.params_dtype,
- num_layers=config.num_layers,
- kv_channels=config.hidden_size // config.num_attention_heads,
- num_attention_heads=config.num_attention_heads,
- max_sequence_length=self.gpt_model.module.max_sequence_length,
- buffer_size_gb=1.0,
- block_size_tokens=256,
- materialize_only_last_token_logits=False,
+ model_config=TransformerConfig(
+ params_dtype=config.params_dtype,
+ num_layers=config.num_layers,
+ kv_channels=config.hidden_size // config.num_attention_heads,
+ num_attention_heads=config.num_attention_heads,
+ ),
+ inference_config=InferenceConfig(
+ max_sequence_length=self.gpt_model.module.max_sequence_length,
+ buffer_size_gb=1.0,
+ block_size_tokens=256,
+ materialize_only_last_token_logits=False,
+ ),
)
# Add a request with 10 tokens. Since 10 is not a multiple of 64,
@@ -443,3 +447,41 @@ def test_dynamic_inference_padding_with_fp8(self):
# Assert that all padding logits are zero.
assert torch.all(padding_logits == 0.0), "Logits for padding tokens are not all zero."
+
+
+def test_get_transformer_layer_spec_forwards_use_te_activation_func():
+ """Test that _get_transformer_layer_spec forwards use_te_activation_func.
+
+ Regression test for https://github.com/NVIDIA/Megatron-LM/issues/2770
+ The --use-te-activation-func flag was silently ignored for non-MoE GPT
+ models because _get_transformer_layer_spec did not forward the parameter
+ to get_gpt_layer_with_transformer_engine_spec.
+ """
+ mock_config = MagicMock()
+ mock_config.use_te_activation_func = True
+ mock_config.use_kitchen = False
+ mock_config.use_kitchen_attention = False
+ mock_config.kitchen_attention_backend = "sdpa"
+
+ mock_args = MagicMock()
+ mock_args.num_experts = None
+ mock_args.moe_grouped_gemm = False
+ mock_args.qk_layernorm = False
+ mock_args.multi_latent_attention = False
+ mock_args.experimental_attention_variant = None
+ mock_args.moe_use_legacy_grouped_gemm = False
+ mock_args.qk_l2_norm = False
+
+ with (
+ patch('gpt_builders.get_args', return_value=mock_args),
+ patch('gpt_builders.get_gpt_layer_with_transformer_engine_spec') as mock_spec_fn,
+ ):
+ from gpt_builders import _get_transformer_layer_spec
+
+ _get_transformer_layer_spec(use_te=True, config=mock_config)
+
+ mock_spec_fn.assert_called_once()
+ _, call_kwargs = mock_spec_fn.call_args
+ assert (
+ call_kwargs.get('use_te_activation_func') is True
+ ), "use_te_activation_func must be forwarded from config"
diff --git a/tests/unit_tests/models/test_gpt_model_batch_invariant.py b/tests/unit_tests/models/test_gpt_model_batch_invariant.py
index ead9125e5ec..9ab7e445c0d 100644
--- a/tests/unit_tests/models/test_gpt_model_batch_invariant.py
+++ b/tests/unit_tests/models/test_gpt_model_batch_invariant.py
@@ -5,17 +5,15 @@
import torch
import torch.distributed as dist
+from megatron.core.inference.config import InferenceConfig
from megatron.core.inference.contexts.dynamic_context import DynamicInferenceContext
from megatron.core.inference.engines.dynamic_engine import DynamicInferenceEngine
from megatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapper import (
GPTInferenceWrapper,
)
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
from megatron.core.inference.sampling_params import SamplingParams
-from megatron.core.inference.text_generation_controllers.simple_text_generation_controller import (
- SimpleTextGenerationController,
+from megatron.core.inference.text_generation_controllers.text_generation_controller import (
+ TextGenerationController,
)
from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
from megatron.core.models.gpt.gpt_model import GPTModel
@@ -91,6 +89,8 @@ def _build_flash_attn_bik_model(seq_len: int, vocab_size: int, hidden_size: int
normalization="RMSNorm",
params_dtype=torch.bfloat16,
attention_backend=AttnBackend.flash,
+ fp32_residual_connection=False,
+ nccl_all_reduce_for_prefill=False,
)
cfg.fp16 = False
cfg.bf16 = True
@@ -184,32 +184,21 @@ def test_dynamic_engine_matches_batched_forward_rl(self):
inference_model = Float16Module(base_model.config, base_model).cuda().eval()
ctx = DynamicInferenceContext(
- params_dtype=torch.bfloat16,
- num_layers=base_model.config.num_layers,
- kv_channels=base_model.config.kv_channels,
- num_attention_heads=base_model.config.num_attention_heads,
- max_sequence_length=seq_len,
- buffer_size_gb=0.125,
- block_size_tokens=16,
- num_cuda_graphs=None,
- materialize_only_last_token_logits=False,
- use_cuda_graphs_for_non_decode_steps=False,
- unified_memory_level=0,
+ model_config=base_model.config,
+ inference_config=InferenceConfig(
+ max_sequence_length=seq_len,
+ buffer_size_gb=0.125,
+ block_size_tokens=16,
+ num_cuda_graphs=None,
+ materialize_only_last_token_logits=False,
+ use_cuda_graphs_for_non_decode_steps=False,
+ unified_memory_level=0,
+ ),
)
- wrapper_cfg = InferenceWrapperConfig(
- hidden_size=base_model.config.hidden_size,
- inference_batch_times_seqlen_threshold=-1,
- fp32_residual_connection=False,
- params_dtype=torch.bfloat16,
- padded_vocab_size=vocab_size,
- inference_max_seq_length=seq_len,
- inference_max_requests=8,
- nccl_all_reduce_for_prefill=False,
- )
- wrapper = GPTInferenceWrapper(inference_model, wrapper_cfg, ctx)
+ wrapper = GPTInferenceWrapper(inference_model, ctx)
tokenizer = DummyTokenizer(vocab_size=vocab_size, bos=None, eod=vocab_size - 1, pad=0)
- controller = SimpleTextGenerationController(wrapper, tokenizer)
+ controller = TextGenerationController(wrapper, tokenizer)
engine = DynamicInferenceEngine(
controller=controller, context=ctx, enable_cuda_graph=False, random_seed=123
)
@@ -273,32 +262,21 @@ def test_dynamic_engine_is_batch_invariant(self):
def _run_engine_with_order(order):
ctx = DynamicInferenceContext(
- params_dtype=torch.bfloat16,
- num_layers=base_model.config.num_layers,
- kv_channels=base_model.config.kv_channels,
- num_attention_heads=base_model.config.num_attention_heads,
- max_sequence_length=seq_len,
- buffer_size_gb=0.125,
- block_size_tokens=16,
- num_cuda_graphs=None,
- materialize_only_last_token_logits=False,
- use_cuda_graphs_for_non_decode_steps=False,
- unified_memory_level=0,
+ model_config=based_model.config,
+ inference_config=InferenceConfig(
+ max_sequence_length=seq_len,
+ buffer_size_gb=0.125,
+ block_size_tokens=16,
+ num_cuda_graphs=None,
+ materialize_only_last_token_logits=False,
+ use_cuda_graphs_for_non_decode_steps=False,
+ unified_memory_level=0,
+ ),
)
- wrapper_cfg = InferenceWrapperConfig(
- hidden_size=base_model.config.hidden_size,
- inference_batch_times_seqlen_threshold=-1,
- fp32_residual_connection=False,
- params_dtype=torch.bfloat16,
- padded_vocab_size=vocab_size,
- inference_max_seq_length=seq_len,
- inference_max_requests=8,
- nccl_all_reduce_for_prefill=False,
- )
- wrapper = GPTInferenceWrapper(inference_model, wrapper_cfg, ctx)
+ wrapper = GPTInferenceWrapper(inference_model, ctx)
tokenizer = DummyTokenizer(vocab_size=vocab_size, bos=None, eod=vocab_size - 1, pad=0)
- controller = SimpleTextGenerationController(wrapper, tokenizer)
+ controller = TextGenerationController(wrapper, tokenizer)
engine = DynamicInferenceEngine(
controller=controller, context=ctx, enable_cuda_graph=False, random_seed=123
)
diff --git a/tests/unit_tests/models/test_gpt_model_quantization.py b/tests/unit_tests/models/test_gpt_model_quantization.py
index e993c9be8d2..6f8d7c7b63f 100644
--- a/tests/unit_tests/models/test_gpt_model_quantization.py
+++ b/tests/unit_tests/models/test_gpt_model_quantization.py
@@ -3,6 +3,7 @@
import pytest
from megatron.core.enums import Fp8Recipe
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.models.gpt import GPTModel
from megatron.core.models.gpt.gpt_layer_specs import get_gpt_decoder_block_spec
from megatron.core.quantization.quant_config import MatchContext, RecipeConfig
@@ -10,11 +11,6 @@
from megatron.core.transformer import TransformerConfig
from tests.unit_tests.test_utilities import Utils
-try:
- from megatron.core.extensions.transformer_engine import HAVE_TE
-except ImportError:
- HAVE_TE = False
-
try:
from megatron.core.extensions.kitchen import (
HAVE_KITCHEN,
@@ -267,7 +263,6 @@ def test_kitchen_config_resolution_moe(self) -> None:
moe_router_load_balancing_type="sinkhorn",
moe_router_topk=1,
moe_grouped_gemm=True,
- moe_use_legacy_grouped_gemm=False,
num_layers=2,
hidden_size=12,
num_attention_heads=4,
diff --git a/tests/unit_tests/models/test_llava_model.py b/tests/unit_tests/models/test_llava_model.py
index 761ef9205cb..c9ed05a393b 100644
--- a/tests/unit_tests/models/test_llava_model.py
+++ b/tests/unit_tests/models/test_llava_model.py
@@ -6,16 +6,18 @@
import pytest
import torch
-from megatron.core import parallel_state as ps
from megatron.core.inference.contexts import StaticInferenceContext
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
+from megatron.core.models.gpt.gpt_layer_specs import (
+ get_gpt_layer_with_transformer_engine_submodules,
+)
from megatron.core.models.multimodal import context_parallel
from megatron.core.models.multimodal.llava_model import LLaVAModel
-from megatron.core.models.vision.vit_layer_specs import get_vit_layer_with_transformer_engine_spec
from megatron.core.packed_seq_params import PackedSeqParams
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer.enums import AttnMaskType
+from megatron.core.transformer.spec_utils import ModuleSpec
from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.transformer.transformer_layer import TransformerLayer
from megatron.core.utils import is_te_min_version
from megatron.training.global_vars import set_args
from tests.unit_tests.test_utilities import Utils
@@ -47,15 +49,19 @@ def setup_method(self, method):
use_cpu_initialization=False,
)
- language_layer_spec = get_gpt_layer_with_transformer_engine_spec()
- vision_layer_spec = deepcopy(language_layer_spec)
- vision_projection_spec = deepcopy(language_layer_spec.submodules.mlp.submodules)
+ language_layer_submodules = get_gpt_layer_with_transformer_engine_submodules()
+ vision_layer_spec = ModuleSpec(
+ module=TransformerLayer, submodules=deepcopy(language_layer_submodules)
+ )
+ vision_projection_spec = deepcopy(language_layer_submodules.mlp.submodules)
language_config.language_model_type = "dummy"
vision_config.vision_model_type = "clip"
self.model = LLaVAModel(
language_transformer_config=language_config,
- language_transformer_layer_spec=language_layer_spec,
+ language_transformer_layer_spec=ModuleSpec(
+ module=TransformerLayer, submodules=language_layer_submodules
+ ),
language_vocab_size=8192,
language_max_sequence_length=4096,
vision_transformer_config=vision_config,
@@ -483,16 +489,20 @@ def setup_and_teardown_llava_model(request):
use_cpu_initialization=False,
)
- language_layer_spec = get_gpt_layer_with_transformer_engine_spec()
- vision_layer_spec = deepcopy(language_layer_spec)
- vision_projection_spec = deepcopy(language_layer_spec.submodules.mlp.submodules)
+ language_layer_submodules = get_gpt_layer_with_transformer_engine_submodules()
+ vision_layer_spec = ModuleSpec(
+ module=TransformerLayer, submodules=deepcopy(language_layer_submodules)
+ )
+ vision_projection_spec = deepcopy(language_layer_submodules.mlp.submodules)
language_config.language_model_type = "dummy"
vision_model_type = request.param
vision_config.vision_model_type = vision_model_type
model = LLaVAModel(
language_transformer_config=language_config,
- language_transformer_layer_spec=language_layer_spec,
+ language_transformer_layer_spec=ModuleSpec(
+ module=TransformerLayer, submodules=language_layer_submodules
+ ),
language_vocab_size=2048,
language_max_sequence_length=4096,
vision_transformer_config=vision_config,
@@ -575,31 +585,33 @@ def _init_llava_model(self, cp_size, tp_size, sequence_parallel):
context_parallel_size=1,
)
- language_layer_spec = get_gpt_layer_with_transformer_engine_spec()
+ language_layer_submodules = get_gpt_layer_with_transformer_engine_submodules()
# SP/CP either requires user to ensure token lengths do not require padding OR change mask type to padding
if (
- language_layer_spec.submodules.self_attention.params.get('attn_mask_type', '')
+ language_layer_submodules.self_attention.params.get('attn_mask_type', '')
== AttnMaskType.causal
):
- language_layer_spec.submodules.self_attention.params['attn_mask_type'] = (
+ language_layer_submodules.self_attention.params['attn_mask_type'] = (
AttnMaskType.padding_causal
)
elif (
- language_layer_spec.submodules.self_attention.params.get('attn_mask_type', '')
+ language_layer_submodules.self_attention.params.get('attn_mask_type', '')
== AttnMaskType.no_mask
):
- language_layer_spec.submodules.self_attention.params['attn_mask_type'] = (
- AttnMaskType.padding
- )
+ language_layer_submodules.self_attention.params['attn_mask_type'] = AttnMaskType.padding
- vision_layer_spec = deepcopy(language_layer_spec)
- vision_projection_spec = deepcopy(language_layer_spec.submodules.mlp.submodules)
+ vision_layer_spec = ModuleSpec(
+ module=TransformerLayer, submodules=deepcopy(language_layer_submodules)
+ )
+ vision_projection_spec = deepcopy(language_layer_submodules.mlp.submodules)
language_config.language_model_type = "dummy"
vision_config.vision_model_type = "clip"
model = LLaVAModel(
language_transformer_config=language_config,
- language_transformer_layer_spec=language_layer_spec,
+ language_transformer_layer_spec=ModuleSpec(
+ module=TransformerLayer, submodules=language_layer_submodules
+ ),
language_vocab_size=8192,
language_max_sequence_length=4096,
vision_transformer_config=vision_config,
diff --git a/tests/unit_tests/models/test_mamba_model.py b/tests/unit_tests/models/test_mamba_model.py
index b83c579c6d7..b295f37b1d3 100755
--- a/tests/unit_tests/models/test_mamba_model.py
+++ b/tests/unit_tests/models/test_mamba_model.py
@@ -10,6 +10,7 @@
from megatron.core import parallel_state
from megatron.core.hyper_comm_grid import HyperCommGrid
+from megatron.core.inference.config import InferenceConfig, MambaInferenceStateConfig
from megatron.core.inference.contexts import BaseInferenceContext, StaticInferenceContext
from megatron.core.inference.contexts.dynamic_context import DynamicInferenceContext
from megatron.core.inference.inference_request import DynamicInferenceRequest
@@ -21,12 +22,7 @@
from megatron.core.transformer import TransformerConfig
from megatron.core.transformer.enums import AttnBackend
from megatron.core.transformer.module import Float16Module
-from megatron.core.utils import (
- divide,
- get_mamba_inference_state_config_from_model,
- is_fa_min_version,
- is_torch_min_version,
-)
+from megatron.core.utils import divide, is_fa_min_version, is_torch_min_version
from tests.unit_tests.test_utilities import Utils
@@ -46,8 +42,7 @@ def setup_method(self, method):
mamba_stack_spec=mamba_stack_spec,
vocab_size=100,
max_sequence_length=4,
- hybrid_attention_ratio=0.3,
- hybrid_mlp_ratio=0.3,
+ hybrid_layer_pattern="M*-", # 1 Mamba, 1 attention, 1 MLP
)
def teardown_method(self, method):
@@ -115,8 +110,7 @@ def test_forward_packed_sequence(self):
mamba_stack_spec=mamba_stack_spec,
vocab_size=vocab_size,
max_sequence_length=12,
- hybrid_attention_ratio=0.3,
- hybrid_mlp_ratio=0.3,
+ hybrid_layer_pattern="M*-", # 1 Mamba, 1 attention, 1 MLP
)
sequence_length = model.max_sequence_length
@@ -146,6 +140,7 @@ def test_forward_packed_sequence(self):
cu_seqlens_kv_padded=None,
max_seqlen_q=max_seqlen,
max_seqlen_kv=max_seqlen,
+ total_tokens=sequence_length,
)
logits = model.forward(
@@ -253,6 +248,9 @@ def test_with_custom_process_groups(self, tmp_path, tp_size, cp_size, pp_size):
tp=tp_group, cp=cp_group, pp=pp_group, embd=embd_group
)
+ # Build pattern with '|' pipeline stage separators: 3 layers per PP stage
+ hybrid_layer_pattern = "|".join(["M*-"] * pp_size)
+
# Configure model with appropriate sizes for parallelism
model_config = TransformerConfig(
num_layers=3 * pp_size, # Scale layers with PP size
@@ -270,8 +268,7 @@ def test_with_custom_process_groups(self, tmp_path, tp_size, cp_size, pp_size):
mamba_stack_spec=mamba_stack_spec,
vocab_size=128,
max_sequence_length=4,
- hybrid_attention_ratio=0.3,
- hybrid_mlp_ratio=0.3,
+ hybrid_layer_pattern=hybrid_layer_pattern,
pg_collection=pg_collection,
)
@@ -325,8 +322,7 @@ def setup_method(self, method):
mamba_stack_spec=mamba_stack_spec,
vocab_size=128,
max_sequence_length=DynamicInferenceContext.TOKEN_ROUNDER,
- hybrid_attention_ratio=0.5,
- hybrid_mlp_ratio=0.0,
+ hybrid_layer_pattern="M*", # 1 Mamba, 1 attention
)
self.model = Float16Module(self.model.config, self.model)
@@ -346,20 +342,17 @@ def test_dynamic_inference_padding_with_fp8(self):
self.model.eval()
config = self.model.config
- mamba_inference_state_config = get_mamba_inference_state_config_from_model(
- self.model.module
- )
+ mamba_inference_state_config = MambaInferenceStateConfig.from_model(self.model.module)
inference_context = DynamicInferenceContext(
- params_dtype=config.params_dtype,
- num_layers=config.num_layers,
- kv_channels=config.hidden_size // config.num_attention_heads,
- num_attention_heads=config.num_attention_heads,
- max_sequence_length=self.model.module.max_sequence_length,
- buffer_size_gb=1.0,
- block_size_tokens=256,
- materialize_only_last_token_logits=False,
- mamba_inference_state_config=mamba_inference_state_config,
+ model_config=self.model.config,
+ inference_config=InferenceConfig(
+ max_sequence_length=self.model.module.max_sequence_length,
+ buffer_size_gb=1.0,
+ block_size_tokens=256,
+ materialize_only_last_token_logits=False,
+ mamba_inference_state_config=mamba_inference_state_config,
+ ),
)
# Add a request with 10 tokens. Since 10 is not a multiple of 64 (TOKEN_ROUNDER),
diff --git a/tests/unit_tests/models/test_mamba_moe_model.py b/tests/unit_tests/models/test_mamba_moe_model.py
index ccb99e4e135..0dfc906d845 100644
--- a/tests/unit_tests/models/test_mamba_moe_model.py
+++ b/tests/unit_tests/models/test_mamba_moe_model.py
@@ -37,7 +37,6 @@
"apply_query_key_layer_scaling": False,
"apply_residual_connection_post_layernorm": False,
"apply_rope_fusion": False,
- "async_tensor_model_parallel_allreduce": True,
"attention_backend": {
"__objclass__": "megatron.core.transformer.enums.AttnBackend",
"_name_": "flash",
@@ -63,6 +62,7 @@
"cpu_offloading": False,
"cpu_offloading_activations": True,
"cpu_offloading_double_buffering": False,
+ "cpu_offloading_retain_pinned_cpu_buffers": False,
"cpu_offloading_num_layers": 0,
"cpu_offloading_weights": False,
"cross_entropy_fusion_impl": "native",
@@ -114,6 +114,7 @@
"fp8_quantizer_factory": None,
"fp8_recipe": "delayed",
"fp8_wgrad": True,
+ "fused_residual_rmsnorm": False,
"fused_single_qkv_rope": False,
"gated_linear_unit": False,
"glu_linear_offset": 0.0,
@@ -157,7 +158,6 @@
"moe_deepep_num_sms": 20,
"moe_enable_deepep": False,
"moe_expert_capacity_factor": None,
- "moe_extended_tp": False,
"moe_ffn_hidden_size": 1856,
"moe_flex_dispatcher_backend": "deepep",
"moe_grouped_gemm": True,
@@ -167,6 +167,7 @@
"moe_layer_freq": 1,
"moe_layer_recompute": False,
"moe_pad_expert_input_to_capacity": False,
+ "moe_pad_experts_for_cuda_graph_inference": False,
"moe_per_layer_logging": False,
"moe_permute_fusion": False,
"moe_router_bias_update_rate": 0.001,
@@ -190,16 +191,22 @@
"moe_token_dispatcher_type": "alltoall",
"moe_token_drop_policy": "probs",
"moe_token_dropping": False,
- "moe_use_legacy_grouped_gemm": False,
"moe_z_loss_coeff": None,
"moe_enable_routing_replay": False,
"mrope_section": None,
+ "mup_attn_scale_power": 1.0,
+ "mup_base_head_dim": None,
+ "mup_base_hidden_size": None,
+ "mup_embedding_mult": 1.0,
+ "mup_output_mult": 1.0,
+ "mup_width_mult": 1.0,
"mtp_hybrid_override_pattern": None,
"mtp_loss_scaling_factor": 0.1,
"mtp_num_layers": None,
"mtp_standalone": False,
"mtp_use_repeated_layer": False,
"multi_latent_attention": False,
+ "nccl_all_reduce_for_prefill": False,
"no_rope_freq": None,
"no_sync_func": None,
"normalization": "RMSNorm",
@@ -264,6 +271,7 @@
"use_kitchen": False,
"use_kitchen_attention": False,
"use_mamba_mem_eff_path": True,
+ "use_mup": False,
"use_ring_exchange_p2p": False,
"use_te_activation_func": False,
"use_te_rng_tracker": False,
@@ -277,6 +285,10 @@
"offload_modules": [],
"hybrid_context_parallel": False,
"max_seqlen_per_dp_cp_rank": None,
+ "inference_disable_triton_nvls_kernels": False,
+ "moe_router_force_biased": None,
+ "inference_grouped_gemm_backend": "auto",
+ "inference_moe_disable_fused_quant_kernels": False,
}
# Fields to ignore entirely (ephemeral, environment-specific, very large).
SKIP_FIELDS = set()
@@ -390,7 +402,6 @@ def create_test_args(self):
args = parse_args()
# The following args would be set from the nano v3 checkpoint.
- args.num_layers = 52
args.hidden_size = 2688
args.ffn_hidden_size = 1856
args.num_attention_heads = 32
@@ -413,10 +424,9 @@ def create_test_args(self):
args.apply_query_key_layer_scaling = False
args.attention_dropout = 0.0
args.hidden_dropout = 0.0
- args.hybrid_override_pattern = "MEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEMEM*EMEMEMEME"
+ args.hybrid_layer_pattern = "MEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEMEM*EMEMEMEME"
+ args.hybrid_override_pattern = None
args.spec = ["megatron.core.models.mamba.mamba_layer_specs", "mamba_stack_spec"]
- args.hybrid_attention_ratio = 0.0
- args.hybrid_mlp_ratio = 0.0
args.num_experts = 128
args.moe_layer_freq = 1
args.moe_ffn_hidden_size = 1856
@@ -431,14 +441,12 @@ def create_test_args(self):
args.mamba_head_dim = 64
args.mamba_num_groups = 8
args.mamba_num_heads = 64
- args.is_hybrid_model = True
args.tokenizer_type = "TikTokenizer"
args.tiktoken_pattern = "v2"
args.tokenizer_model = "/mnt/artifacts/model/nemotron6/tokenizers/multiMixV8.gpt4o_nc_sd.500000.128k.vocab.json"
args.padded_vocab_size = 131072
# The following args would be set in the user's nano v3 config.
- args.async_tensor_model_parallel_allreduce = True
args.attention_backend = AttnBackend.flash
args.bf16 = True
args.ckpt_format = 'torch_dist'
@@ -495,9 +503,7 @@ def setup_method(self, method):
mamba_stack_spec=mamba_stack_spec,
vocab_size=args.vocab_size,
max_sequence_length=args.seq_length,
- hybrid_attention_ratio=args.hybrid_attention_ratio,
- hybrid_mlp_ratio=args.hybrid_mlp_ratio,
- hybrid_override_pattern=args.hybrid_override_pattern,
+ hybrid_layer_pattern=args.hybrid_layer_pattern,
position_embedding_type=args.position_embedding_type,
rotary_base=args.rotary_base,
rotary_percent=args.rotary_percent,
@@ -515,11 +521,9 @@ def test_constructor(self):
assert self.model.pre_process is True, "pre_process should be True"
assert self.model.post_process is True, "post_process should be True"
- assert self.model.hybrid_attention_ratio == 0.0, "hybrid_attention_ratio should be 0.0"
- assert self.model.hybrid_mlp_ratio == 0.0, "hybrid_mlp_ratio should be 0.0"
assert (
- self.model.hybrid_override_pattern == args.hybrid_override_pattern
- ), f"hybrid_override_pattern should be {args.hybrid_override_pattern}"
+ self.model.hybrid_layer_pattern == args.hybrid_layer_pattern
+ ), f"hybrid_layer_pattern should be {args.hybrid_layer_pattern}"
num_weights = sum([p.numel() for p in self.model.parameters()])
assert num_weights == 8449294624, f"Expected 8449294624 parameters, got {num_weights}"
diff --git a/tests/unit_tests/models/test_mimo_1f1b_schedule.py b/tests/unit_tests/models/test_mimo_1f1b_schedule.py
new file mode 100644
index 00000000000..44be0c7911e
--- /dev/null
+++ b/tests/unit_tests/models/test_mimo_1f1b_schedule.py
@@ -0,0 +1,812 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+"""Integration tests for MIMO model with 1F1B pipeline schedule.
+
+Run with:
+ uv run python -m torch.distributed.run --nproc-per-node=2 -m pytest tests/unit_tests/models/test_mimo_1f1b_schedule.py -v
+"""
+
+import logging
+from contextlib import ExitStack, contextmanager
+from functools import partial
+
+import pytest
+import torch
+import torch.distributed as dist
+from packaging import version
+
+import megatron.core.pipeline_parallel.schedules as schedule
+from megatron.core.distributed import DistributedDataParallel, DistributedDataParallelConfig
+from megatron.core.distributed.finalize_model_grads import finalize_model_grads
+from megatron.core.hyper_comm_grid import HyperCommGrid
+from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
+from megatron.core.models.gpt.gpt_model import GPTModel
+from megatron.core.models.mimo.config.base_configs import MimoModelConfig
+from megatron.core.models.mimo.config.role import MIMO_LANGUAGE_MODULE_KEY
+from megatron.core.models.mimo.model.base import MimoModel
+from megatron.core.models.mimo.optimizer import get_mimo_optimizer
+from megatron.core.models.mimo.submodules.vision import VisionModalitySubmodules
+from megatron.core.models.vision.multimodal_projector import MultimodalProjector
+from megatron.core.optimizer.optimizer_config import OptimizerConfig
+from megatron.core.pipeline_parallel.bridge_communicator import BridgeCommunicator
+from megatron.core.pipeline_parallel.multimodule_communicator import MultiModulePipelineCommunicator
+from megatron.core.pipeline_parallel.utils import is_pp_first_stage, is_pp_last_stage
+from megatron.core.process_groups_config import (
+ MultiModuleProcessGroupCollection,
+ ProcessGroupCollection,
+)
+from megatron.core.transformer.mlp import MLP, MLPSubmodules
+from megatron.core.transformer.spec_utils import ModuleSpec
+from megatron.core.transformer.transformer_config import TransformerConfig
+from tests.unit_tests.test_utilities import Utils
+
+try:
+ from megatron.core.extensions.transformer_engine import (
+ TEColumnParallelLinear,
+ TERowParallelLinear,
+ )
+except ImportError:
+ TEColumnParallelLinear = None
+ TERowParallelLinear = None
+
+logger = logging.getLogger(__name__)
+
+
+# ============================================================================
+# Helper Functions (with grid tracking and PG caching from edc8159)
+# ============================================================================
+
+_active_grids: list = []
+_embedding_pg_cache: dict = {}
+
+
+def create_hypercomm_grid(offset=0, tp=1, cp=1, pp=1, dp=1):
+ """Create a HyperCommGrid with specified parallelism."""
+ grid = HyperCommGrid(
+ shape=[tp, cp, pp, dp, 1, 1], # [tp, cp, pp, dp, ep, expt_dp]
+ dim_names=["tp", "cp", "pp", "dp", "ep", "expt_dp"],
+ rank_offset=offset,
+ backend="nccl",
+ )
+ grid.create_pg(["tp"])
+ grid.create_pg(["cp"])
+ grid.create_pg(["pp"])
+ grid.create_pg(["dp"])
+ grid.create_pg(["dp", "cp"])
+ grid.create_pg(["ep"])
+ grid.create_pg(["expt_dp"])
+ # Required by _get_pg_collection_for_optimizer
+ grid.create_pg(["tp", "pp"])
+ grid.create_pg(["tp", "ep", "pp"])
+ grid.create_pg(["dp", "ep"])
+ grid.create_pg(["tp", "cp", "ep", "pp", "dp"])
+ _active_grids.append(grid)
+ return grid
+
+
+def destroy_all_grids():
+ """Destroy all tracked grids and bridge communicator PGs."""
+ for grid in _active_grids:
+ grid.destroy()
+ _active_grids.clear()
+ _embedding_pg_cache.clear()
+ BridgeCommunicator.destroy_broadcast_pgs()
+
+
+def get_pg_collection(grid):
+ """Get ProcessGroupCollection from grid."""
+ pg_collection = ProcessGroupCollection()
+ pg_collection.tp = grid.get_pg("tp")
+ pg_collection.cp = grid.get_pg("cp")
+ pg_collection.pp = grid.get_pg("pp")
+ pg_collection.ep = grid.get_pg("ep")
+ pg_collection.dp = grid.get_pg("dp")
+ pg_collection.dp_cp = grid.get_pg(["dp", "cp"])
+ pg_collection.expt_dp = grid.get_pg("expt_dp")
+ return pg_collection
+
+
+def create_all_embedding_groups(grids):
+ """Create embedding PGs for all grids upfront.
+
+ dist.new_group is a collective — ALL ranks must call it, even non-members.
+ We create all embedding groups in a consistent order across all ranks to
+ avoid hangs from asymmetric new_group calls.
+
+ Args:
+ grids: List of all HyperCommGrids that need embedding groups.
+ """
+ for grid in grids:
+ pp_group = grid.get_pg("pp")
+ if not pp_group:
+ continue
+
+ pp_ranks = sorted(dist.get_process_group_ranks(pp_group))
+ cache_key = tuple(pp_ranks)
+
+ if cache_key not in _embedding_pg_cache:
+ pos_embd_ranks = [pp_ranks[0]]
+ embd_ranks = [pp_ranks[0]]
+ if pp_ranks[-1] != pp_ranks[0]:
+ embd_ranks.append(pp_ranks[-1])
+ _embedding_pg_cache[cache_key] = (
+ dist.new_group(ranks=pos_embd_ranks),
+ dist.new_group(ranks=embd_ranks),
+ )
+
+
+def add_embedding_groups(pg_collection, is_language_model=False):
+ """Add cached embedding groups to a process group collection.
+
+ Must call create_all_embedding_groups() first to ensure PGs exist.
+
+ Args:
+ pg_collection: ProcessGroupCollection to add embedding groups to.
+ is_language_model: If True, set embd group for word embedding sync.
+ """
+ if not pg_collection.pp:
+ return pg_collection
+
+ pp_ranks = sorted(dist.get_process_group_ranks(pg_collection.pp))
+ cache_key = tuple(pp_ranks)
+ pos_embd_pg, embd_pg = _embedding_pg_cache[cache_key]
+
+ pg_collection.pos_embd = pos_embd_pg if is_pp_first_stage(pg_collection.pp) else None
+
+ if is_language_model:
+ pg_collection.embd = (
+ embd_pg
+ if (is_pp_last_stage(pg_collection.pp) or is_pp_first_stage(pg_collection.pp))
+ else None
+ )
+ else:
+ # Encoder submodules have no shared word embeddings to sync
+ pg_collection.embd = None
+
+ return pg_collection
+
+
+def get_pg_collection_with_embedding_groups(grid, is_language_model=False):
+ """Get ProcessGroupCollection with embedding groups (PGs must be pre-created)."""
+ return add_embedding_groups(get_pg_collection(grid), is_language_model=is_language_model)
+
+
+def is_rank_in_grid(grid):
+ """Check if current rank is in grid."""
+ rank = dist.get_rank()
+ return grid.rank_offset <= rank < grid.rank_offset + grid.size
+
+
+# ============================================================================
+# Model Spec Helpers
+# ============================================================================
+
+
+def get_language_model_spec(
+ num_layers, hidden_size, num_attention_heads, vocab_size, seq_len, pg_collection
+):
+ """Get the language model spec."""
+ pp_rank = dist.get_rank(pg_collection.pp)
+ pp_size = dist.get_world_size(pg_collection.pp)
+ tp_size = pg_collection.tp.size() if pg_collection.tp is not None else 1
+
+ lm_config = TransformerConfig(
+ num_layers=num_layers,
+ hidden_size=hidden_size,
+ num_attention_heads=num_attention_heads,
+ use_cpu_initialization=True,
+ variable_seq_lengths=True,
+ moe_token_dispatcher_type='alltoall',
+ tensor_model_parallel_size=tp_size,
+ pipeline_model_parallel_size=pp_size,
+ pipeline_dtype=torch.bfloat16,
+ bf16=True,
+ cross_entropy_loss_fusion=True,
+ cross_entropy_fusion_impl='te',
+ )
+ return ModuleSpec(
+ module=GPTModel,
+ params={
+ "config": lm_config,
+ "transformer_layer_spec": get_gpt_layer_with_transformer_engine_spec(),
+ "vocab_size": vocab_size,
+ "max_sequence_length": seq_len,
+ "pre_process": (pp_rank == 0),
+ "post_process": (pp_rank == pp_size - 1),
+ "pg_collection": pg_collection,
+ },
+ )
+
+
+def get_projection_config(hidden_size):
+ """Return a TransformerConfig for the vision projection MLP."""
+ cfg = TransformerConfig(num_layers=1, hidden_size=hidden_size, num_attention_heads=1)
+ cfg.ffn_hidden_size = hidden_size
+ cfg.bias_activation_fusion = True
+ cfg.add_bias_linear = True
+ cfg.activation_func = torch.nn.functional.gelu
+ return cfg
+
+
+def get_projection_layer_spec():
+ """Layer spec for the vision-projection MLP."""
+ if TEColumnParallelLinear is None or TERowParallelLinear is None:
+ raise RuntimeError("TEColumnParallelLinear and TERowParallelLinear are required")
+ return ModuleSpec(
+ module=MLP,
+ submodules=MLPSubmodules(linear_fc1=TEColumnParallelLinear, linear_fc2=TERowParallelLinear),
+ )
+
+
+def get_vision_submodules_spec(
+ num_layers, hidden_size, num_attention_heads, language_hidden_size, pg_collection
+):
+ """Get the submodule spec for the vision modality."""
+ from megatron.core.transformer.transformer_block import TransformerBlock
+
+ tp_size = pg_collection.tp.size() if pg_collection.tp is not None else 1
+ pp_size = pg_collection.pp.size() if pg_collection.pp is not None else 1
+ pp_rank = dist.get_rank(pg_collection.pp)
+
+ vision_config = TransformerConfig(
+ num_layers=num_layers,
+ hidden_size=hidden_size,
+ num_attention_heads=num_attention_heads,
+ use_cpu_initialization=True,
+ variable_seq_lengths=True,
+ moe_token_dispatcher_type='alltoall',
+ tensor_model_parallel_size=tp_size,
+ pipeline_model_parallel_size=pp_size,
+ pipeline_dtype=torch.bfloat16,
+ bf16=True,
+ )
+ vision_encoder_spec = ModuleSpec(
+ module=TransformerBlock,
+ params={
+ "config": vision_config,
+ "spec": get_gpt_layer_with_transformer_engine_spec(),
+ "pg_collection": pg_collection,
+ "pre_process": (pp_rank == 0),
+ "post_process": (pp_rank == pp_size - 1),
+ },
+ )
+
+ vision_projection_spec = ModuleSpec(
+ module=MultimodalProjector,
+ params={
+ "config": get_projection_config(hidden_size=language_hidden_size),
+ "submodules": get_projection_layer_spec().submodules,
+ "projector_type": "mlp",
+ "input_size": vision_config.hidden_size,
+ "tp_group": pg_collection.tp,
+ },
+ )
+
+ return ModuleSpec(
+ module=VisionModalitySubmodules,
+ params={"pg_collection": pg_collection},
+ submodules={
+ "encoders": {"clip_encoder": vision_encoder_spec},
+ "input_projections": [vision_projection_spec],
+ },
+ )
+
+
+def get_mimo_model(
+ encoder_name, encoder_grid, llm_grid, hidden_size, num_layers, vocab_size, seq_len
+):
+ """Create MIMO model with TransformerBlock encoder and GPTModel LLM."""
+ language_pg = get_pg_collection_with_embedding_groups(llm_grid, is_language_model=True)
+ vision_pg = get_pg_collection_with_embedding_groups(encoder_grid, is_language_model=False)
+
+ language_model_spec = get_language_model_spec(
+ num_layers=num_layers,
+ hidden_size=hidden_size,
+ num_attention_heads=8,
+ vocab_size=vocab_size,
+ seq_len=seq_len,
+ pg_collection=language_pg,
+ )
+ vision_submodule_spec = get_vision_submodules_spec(
+ num_layers=num_layers,
+ hidden_size=hidden_size,
+ num_attention_heads=8,
+ language_hidden_size=hidden_size,
+ pg_collection=vision_pg,
+ )
+
+ module_to_grid_map = {encoder_name: encoder_grid, MIMO_LANGUAGE_MODULE_KEY: llm_grid}
+ topology = {encoder_name: [MIMO_LANGUAGE_MODULE_KEY], MIMO_LANGUAGE_MODULE_KEY: []}
+
+ mimo_config = MimoModelConfig(
+ language_model_spec=language_model_spec,
+ modality_submodules_spec={encoder_name: vision_submodule_spec},
+ special_token_ids={encoder_name: 50257},
+ module_to_grid_map=module_to_grid_map,
+ )
+
+ mimo_model = MimoModel(mimo_config)
+ mimo_model.to(torch.device("cuda")).to(torch.bfloat16)
+
+ # Wrap with DDP
+ ddp_config = DistributedDataParallelConfig(
+ overlap_grad_reduce=True, bucket_size=10000, use_distributed_optimizer=True
+ )
+
+ if mimo_model.language_model is not None:
+ mimo_model.language_model = DistributedDataParallel(
+ config=mimo_model.language_model.config,
+ ddp_config=ddp_config,
+ module=mimo_model.language_model,
+ pg_collection=language_pg,
+ )
+
+ if encoder_name in mimo_model.modality_submodules:
+ submodule = mimo_model.modality_submodules[encoder_name]
+ if submodule is not None:
+ submodule = DistributedDataParallel(
+ config=submodule.encoders['clip_encoder'].config,
+ ddp_config=ddp_config,
+ module=submodule,
+ pg_collection=vision_pg,
+ )
+ mimo_model.modality_submodules[encoder_name] = submodule
+
+ return mimo_model, module_to_grid_map, topology, language_pg, vision_pg
+
+
+# ============================================================================
+# Data Iterator
+# ============================================================================
+
+
+class DataIterator:
+ """Simple data iterator returning VLM-like batches."""
+
+ def __init__(
+ self,
+ hidden_size,
+ seq_length,
+ micro_batch_size,
+ vocab_size,
+ encoder_name,
+ image_token_id=50257,
+ image_seq_length=None,
+ ):
+ self.hidden_size = hidden_size
+ self.seq_length = seq_length
+ self.micro_batch_size = micro_batch_size
+ self.vocab_size = vocab_size
+ self.encoder_name = encoder_name
+ self.image_token_id = image_token_id
+ self.image_seq_length = image_seq_length or (seq_length // 2)
+
+ def __iter__(self):
+ return self
+
+ def __next__(self):
+ encoder_hidden_states = torch.randn(
+ self.image_seq_length,
+ self.micro_batch_size,
+ self.hidden_size,
+ device='cuda',
+ dtype=torch.bfloat16,
+ )
+
+ image_tokens = torch.full(
+ (self.micro_batch_size, self.image_seq_length),
+ self.image_token_id,
+ dtype=torch.long,
+ device='cuda',
+ )
+ text_tokens = torch.randint(
+ 1,
+ self.vocab_size,
+ (self.micro_batch_size, self.seq_length - self.image_seq_length),
+ device='cuda',
+ )
+ input_ids = torch.cat([image_tokens, text_tokens], dim=1)
+
+ labels = input_ids.clone()
+ labels[input_ids == self.image_token_id] = -100
+
+ loss_mask = torch.ones(
+ self.micro_batch_size, self.seq_length, device='cuda', dtype=torch.float32
+ )
+ loss_mask[input_ids == self.image_token_id] = 0.0
+
+ return {
+ "input_ids": input_ids,
+ "labels": labels,
+ "loss_mask": loss_mask,
+ "position_ids": torch.arange(self.seq_length, device='cuda')
+ .unsqueeze(0)
+ .expand(self.micro_batch_size, -1)
+ .clone(),
+ "modality_inputs": {
+ self.encoder_name: {
+ "clip_encoder": {'hidden_states': encoder_hidden_states, 'attention_mask': None}
+ }
+ },
+ }
+
+
+# ============================================================================
+# Test Runner
+# ============================================================================
+
+
+def run_mimo_1f1b_test(
+ encoder_tp,
+ encoder_pp,
+ encoder_dp,
+ encoder_offset,
+ llm_tp,
+ llm_pp,
+ llm_dp,
+ llm_offset,
+ hidden_size=256,
+ num_layers=2,
+ vocab_size=1000,
+ seq_length=64,
+ micro_batch_size=2,
+ num_microbatches=4,
+):
+ """Run MIMO model through 1F1B schedule and verify."""
+ # Clear NVTE env vars that the conftest set_env fixture sets to '0'.
+ # GPTModel (LanguageModule) asserts these are unset or match the attention backend.
+ import os
+
+ os.environ.pop('NVTE_FLASH_ATTN', None)
+ os.environ.pop('NVTE_FUSED_ATTN', None)
+ os.environ.pop('NVTE_UNFUSED_ATTN', None)
+
+ encoder_name = "images"
+
+ encoder_grid = create_hypercomm_grid(
+ offset=encoder_offset, tp=encoder_tp, cp=1, pp=encoder_pp, dp=encoder_dp
+ )
+ llm_grid = create_hypercomm_grid(offset=llm_offset, tp=llm_tp, cp=1, pp=llm_pp, dp=llm_dp)
+
+ # Create all embedding PGs upfront — dist.new_group is a collective that
+ # requires ALL ranks to participate, so we must create them before any
+ # rank-specific pg_collection calls.
+ create_all_embedding_groups([encoder_grid, llm_grid])
+
+ torch.manual_seed(12345)
+
+ mimo_model, module_to_grid_map, topology, language_pg, vision_pg = get_mimo_model(
+ encoder_name=encoder_name,
+ encoder_grid=encoder_grid,
+ llm_grid=llm_grid,
+ hidden_size=hidden_size,
+ num_layers=num_layers,
+ vocab_size=vocab_size,
+ seq_len=seq_length,
+ )
+
+ # Build schedule functions using pre-created pg_collections (no leaks)
+ @contextmanager
+ def no_sync_func():
+ with ExitStack() as stack:
+ if mimo_model.language_model is not None:
+ stack.enter_context(mimo_model.language_model.no_sync())
+ for submodule in mimo_model.modality_submodules.values():
+ if submodule is not None:
+ stack.enter_context(submodule.no_sync())
+ yield
+
+ def finalize_grads_func(*args, **kwargs):
+ if mimo_model.language_model is not None:
+ finalize_model_grads(
+ [mimo_model.language_model], num_tokens=None, pg_collection=language_pg
+ )
+ for submodule in mimo_model.modality_submodules.values():
+ if submodule is not None:
+ finalize_model_grads([submodule], num_tokens=None, pg_collection=vision_pg)
+
+ mimo_model.config.no_sync_func = no_sync_func
+ mimo_model.config.finalize_model_grads_func = finalize_grads_func
+ mimo_model.config.grad_scale_func = lambda loss: (
+ torch.tensor(loss, dtype=torch.float32, device='cuda', requires_grad=True)
+ if isinstance(loss, (int, float))
+ else loss
+ )
+
+ # Create optimizer
+ opt_config = OptimizerConfig(
+ optimizer='adam',
+ lr=1e-4,
+ weight_decay=0.01,
+ clip_grad=1.0,
+ bf16=True,
+ use_distributed_optimizer=True,
+ )
+ optimizer = get_mimo_optimizer(mimo_model, opt_config)
+
+ communicator = MultiModulePipelineCommunicator(
+ module_to_grid_map,
+ topology,
+ mimo_model.config,
+ dim_mapping={'s': 0, 'h': 2, 'b': 1},
+ module_output_ndim={encoder_name: 2},
+ )
+
+ # Compute per-rank micro-batch size for asymmetric DP.
+ # The LLM's MBS is the schedule-level MBS. The encoder's MBS is adjusted
+ # by the DP ratio so that total work is conserved across the bridge.
+ llm_mbs = micro_batch_size
+ encoder_mbs = micro_batch_size * llm_dp // encoder_dp
+
+ # Create data iterator on ranks that need it, with per-role micro-batch size.
+ # Encoder ranks use encoder_mbs, LLM ranks use llm_mbs.
+ data_iterator = None
+ encoder_needs_data = is_rank_in_grid(encoder_grid) and is_pp_first_stage(
+ encoder_grid.get_pg("pp")
+ )
+ llm_needs_data = is_rank_in_grid(llm_grid) and (
+ is_pp_first_stage(llm_grid.get_pg("pp")) or is_pp_last_stage(llm_grid.get_pg("pp"))
+ )
+ if encoder_needs_data and not llm_needs_data:
+ data_iterator = DataIterator(hidden_size, seq_length, encoder_mbs, vocab_size, encoder_name)
+ elif llm_needs_data and not encoder_needs_data:
+ data_iterator = DataIterator(hidden_size, seq_length, llm_mbs, vocab_size, encoder_name)
+ elif encoder_needs_data and llm_needs_data:
+ # Colocated: both encoder and LLM on same rank. Use LLM's MBS since
+ # the LLM drives the schedule. (encoder_dp == llm_dp when colocated)
+ data_iterator = DataIterator(
+ hidden_size, seq_length, micro_batch_size, vocab_size, encoder_name
+ )
+
+ # Build MultiModuleProcessGroupCollection (reuse pre-created pg_collections)
+ module_pgs = {}
+ language_model_module_name = None
+ if is_rank_in_grid(encoder_grid):
+ module_pgs[encoder_name] = vision_pg
+ if is_rank_in_grid(llm_grid):
+ module_pgs[MIMO_LANGUAGE_MODULE_KEY] = language_pg
+ language_model_module_name = MIMO_LANGUAGE_MODULE_KEY
+
+ pg_collection = MultiModuleProcessGroupCollection(
+ module_pgs=module_pgs, language_model_module_name=language_model_module_name
+ )
+
+ def step_func(data_iterator, model):
+ def loss_func(loss_mask, output_tensor):
+ if output_tensor is None:
+ return torch.tensor(0.0, device='cuda', requires_grad=True), {'loss_reduced': 0.0}
+
+ if isinstance(output_tensor, dict):
+ output = output_tensor.get(
+ MIMO_LANGUAGE_MODULE_KEY, next(iter(output_tensor.values()), None)
+ )
+ else:
+ output = output_tensor
+
+ if output is None:
+ return torch.tensor(0.0, device='cuda', requires_grad=True), {'loss_reduced': 0.0}
+
+ loss = output.float().sum()
+ return loss, {'loss_reduced': loss}
+
+ batch = next(data_iterator) if data_iterator is not None else {'input_ids': None}
+ output_tensor, loss_mask = model(**batch)
+ return output_tensor, partial(loss_func, loss_mask)
+
+ optimizer.zero_grad()
+
+ losses = schedule.forward_backward_pipelining_without_interleaving(
+ forward_step_func=step_func,
+ data_iterator=data_iterator,
+ model=[mimo_model],
+ num_microbatches=num_microbatches,
+ seq_length=seq_length,
+ micro_batch_size=micro_batch_size,
+ forward_only=False,
+ p2p_communicator=communicator,
+ pg_collection=pg_collection,
+ )
+
+ # Optimizer step with global gradient clipping
+ success, grad_norm, num_zeros = optimizer.step()
+ assert success, "Optimizer step failed"
+ assert grad_norm is not None and grad_norm > 0, f"Expected positive grad norm, got {grad_norm}"
+
+ # Verify results on last LLM stage
+ if is_rank_in_grid(llm_grid) and is_pp_last_stage(llm_grid.get_pg("pp")):
+ assert len(losses) > 0, "Expected losses on last LLM stage"
+ for loss_dict in losses:
+ assert 'loss_reduced' in loss_dict
+
+ return losses
+
+
+# ============================================================================
+# Tests
+# ============================================================================
+
+
+@pytest.mark.skipif(
+ version.parse(torch.__version__) < version.parse('2.3.0'),
+ reason="Device mesh requires PyTorch 2.3+",
+)
+class TestMimo1F1BSchedule:
+ """Test MIMO model with 1F1B pipeline schedule."""
+
+ @classmethod
+ def setup_class(cls):
+ Utils.initialize_distributed()
+ cls.world_size = dist.get_world_size()
+
+ @classmethod
+ def teardown_class(cls):
+ Utils.destroy_model_parallel()
+
+ def teardown_method(self):
+ destroy_all_grids()
+
+ def test_baseline_2gpu(self):
+ """Encoder PP=1, LLM PP=1 on 2 GPUs."""
+ if self.world_size != 2:
+ pytest.skip(f"Requires 2 GPUs, got {self.world_size}")
+
+ run_mimo_1f1b_test(
+ encoder_tp=1,
+ encoder_pp=1,
+ encoder_dp=1,
+ encoder_offset=0,
+ llm_tp=1,
+ llm_pp=1,
+ llm_dp=1,
+ llm_offset=1,
+ hidden_size=256,
+ num_layers=2,
+ vocab_size=1000,
+ seq_length=64,
+ micro_batch_size=2,
+ num_microbatches=4,
+ )
+
+ def test_lm_pp3_4gpu(self):
+ """Encoder PP=1, LLM PP=3 on 4 GPUs."""
+ if self.world_size != 4:
+ pytest.skip(f"Requires 4 GPUs, got {self.world_size}")
+
+ run_mimo_1f1b_test(
+ encoder_tp=1,
+ encoder_pp=1,
+ encoder_dp=1,
+ encoder_offset=0,
+ llm_tp=1,
+ llm_pp=3,
+ llm_dp=1,
+ llm_offset=1,
+ hidden_size=256,
+ num_layers=2,
+ vocab_size=1000,
+ seq_length=64,
+ micro_batch_size=2,
+ num_microbatches=4,
+ )
+
+ def test_encoder_tp2_llm_tp2_pp3_8gpu(self):
+ """Encoder TP=2 PP=1, LLM TP=2 PP=3 on 8 GPUs."""
+ if self.world_size != 8:
+ pytest.skip(f"Requires 8 GPUs, got {self.world_size}")
+
+ run_mimo_1f1b_test(
+ encoder_tp=2,
+ encoder_pp=1,
+ encoder_dp=1,
+ encoder_offset=0,
+ llm_tp=2,
+ llm_pp=3,
+ llm_dp=1,
+ llm_offset=2,
+ hidden_size=256,
+ num_layers=3,
+ vocab_size=1000,
+ seq_length=64,
+ micro_batch_size=2,
+ num_microbatches=4,
+ )
+
+ def test_full_pp_8gpu(self):
+ """Encoder PP=2, LLM PP=2 with TP=2 each on 8 GPUs."""
+ if self.world_size != 8:
+ pytest.skip(f"Requires 8 GPUs, got {self.world_size}")
+
+ run_mimo_1f1b_test(
+ encoder_tp=2,
+ encoder_pp=2,
+ encoder_dp=1,
+ encoder_offset=0,
+ llm_tp=2,
+ llm_pp=2,
+ llm_dp=1,
+ llm_offset=4,
+ hidden_size=256,
+ num_layers=2,
+ vocab_size=1000,
+ seq_length=64,
+ micro_batch_size=2,
+ num_microbatches=4,
+ )
+
+ def test_fan_in_dp4_to_dp1_llm_tp2_pp2_8gpu(self):
+ """Fan-in 4→1: Encoder DP=4 → LLM TP=2 PP=2 DP=1, on 8 GPUs.
+
+ High fan-in ratio. Each encoder rank processes MBS=1, bridge concatenates
+ 4 × [img_seq, H] → [4*img_seq, H]. LLM has both TP and PP.
+ """
+ if self.world_size != 8:
+ pytest.skip(f"Requires 8 GPUs, got {self.world_size}")
+
+ run_mimo_1f1b_test(
+ encoder_tp=1,
+ encoder_pp=1,
+ encoder_dp=4,
+ encoder_offset=0,
+ llm_tp=2,
+ llm_pp=2,
+ llm_dp=1,
+ llm_offset=4,
+ hidden_size=256,
+ num_layers=2,
+ vocab_size=1000,
+ seq_length=64,
+ micro_batch_size=4,
+ num_microbatches=4,
+ )
+
+ def test_fan_out_dp1_to_dp4_enc_tp2_pp2_8gpu(self):
+ """Fan-out 1→4: Encoder TP=2 PP=2 DP=1 → LLM DP=4, on 8 GPUs.
+
+ Encoder has PP and TP. Bridge fan-out splits encoder output into
+ 4 parts for 4 LLM DP ranks each with MBS=1.
+ """
+ if self.world_size != 8:
+ pytest.skip(f"Requires 8 GPUs, got {self.world_size}")
+
+ run_mimo_1f1b_test(
+ encoder_tp=2,
+ encoder_pp=2,
+ encoder_dp=1,
+ encoder_offset=0,
+ llm_tp=1,
+ llm_pp=1,
+ llm_dp=4,
+ llm_offset=4,
+ hidden_size=256,
+ num_layers=2,
+ vocab_size=1000,
+ seq_length=64,
+ micro_batch_size=1,
+ num_microbatches=4,
+ )
+
+ def test_fan_in_dp2_to_dp1_llm_pp3_8gpu(self):
+ """Fan-in 2→1: Encoder DP=2 → LLM TP=2 PP=3, on 8 GPUs.
+
+ Tests fan-in with deep LLM pipeline (PP=3). The 2D tensor goes through
+ bridge fan-in then P2P across 3 LLM PP stages.
+ """
+ if self.world_size != 8:
+ pytest.skip(f"Requires 8 GPUs, got {self.world_size}")
+
+ run_mimo_1f1b_test(
+ encoder_tp=1,
+ encoder_pp=1,
+ encoder_dp=2,
+ encoder_offset=0,
+ llm_tp=2,
+ llm_pp=3,
+ llm_dp=1,
+ llm_offset=2,
+ hidden_size=256,
+ num_layers=3,
+ vocab_size=1000,
+ seq_length=64,
+ micro_batch_size=2,
+ num_microbatches=4,
+ )
diff --git a/tests/unit_tests/models/test_mimo_audio_submodules.py b/tests/unit_tests/models/test_mimo_audio_submodules.py
index 0f3865d940f..f9a18838f60 100644
--- a/tests/unit_tests/models/test_mimo_audio_submodules.py
+++ b/tests/unit_tests/models/test_mimo_audio_submodules.py
@@ -394,3 +394,33 @@ def test_multiple_audio_encoders(self, model_name, batch_size):
print(
f"Model {model_name} (d_model={self.d_model}) successfully processed audio and projected to dimension 768"
)
+
+
+class TestAudioSubmoduleStageAware:
+ """Tests for stage-aware forward in AudioModalitySubmodules."""
+
+ def test_stage_aware_forward(self):
+ """Test stage-aware forward: hidden_states input and projection skipping."""
+ import torch.nn as nn
+
+ hidden_size = 64
+ projection_size = 128
+ hidden_states = torch.randn(10, hidden_size)
+
+ # Non-first stage uses hidden_states, last stage projects
+ submodule_last = AudioModalitySubmodules(
+ input_projections=[nn.Linear(hidden_size, projection_size)],
+ is_first_stage=False,
+ is_last_stage=True,
+ )
+ output = submodule_last.forward(hidden_states=hidden_states)
+ assert output.shape == (10, projection_size) # Projected
+
+ # Non-last stage skips projection
+ submodule_mid = AudioModalitySubmodules(
+ input_projections=[nn.Linear(hidden_size, projection_size)],
+ is_first_stage=False,
+ is_last_stage=False,
+ )
+ output = submodule_mid.forward(hidden_states=hidden_states)
+ assert output.shape == (10, hidden_size) # Not projected
diff --git a/tests/unit_tests/models/test_mimo_checkpoint.py b/tests/unit_tests/models/test_mimo_checkpoint.py
new file mode 100644
index 00000000000..3dc75a05a87
--- /dev/null
+++ b/tests/unit_tests/models/test_mimo_checkpoint.py
@@ -0,0 +1,273 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+"""Integration tests for MIMO distributed checkpoint save/load in non-colocated mode.
+
+Run with 8 GPUs:
+ uv run python -m torch.distributed.run --nproc-per-node=8 \
+ -m pytest tests/unit_tests/models/test_mimo_checkpoint.py -v -s
+"""
+
+import os
+import shutil
+import tempfile
+
+import pytest
+import torch
+import torch.distributed as dist
+from packaging import version
+
+from megatron.core.dist_checkpointing import load, save
+from megatron.core.dist_checkpointing.validation import StrictHandling
+from megatron.core.models.mimo.optimizer import get_mimo_optimizer
+from megatron.core.optimizer.optimizer_config import OptimizerConfig
+from tests.unit_tests.models.test_mimo_1f1b_schedule import (
+ create_all_embedding_groups,
+ create_hypercomm_grid,
+ destroy_all_grids,
+ get_mimo_model,
+ is_rank_in_grid,
+)
+from tests.unit_tests.test_utilities import Utils
+
+ENCODER_NAME = "images"
+
+
+def _get_shared_tmpdir():
+ """Create a shared temp directory across all ranks."""
+ tmpdir_list = [None]
+ if dist.get_rank() == 0:
+ tmpdir_list[0] = tempfile.mkdtemp(prefix="mimo_ckpt_test_")
+ dist.broadcast_object_list(tmpdir_list, src=0)
+ return tmpdir_list[0]
+
+
+def _cleanup_tmpdir(tmpdir):
+ """Clean up temp directory (rank 0 only)."""
+ dist.barrier()
+ if dist.get_rank() == 0:
+ shutil.rmtree(tmpdir, ignore_errors=True)
+
+
+def _randomize_params(model, seed):
+ """Set all model parameters to deterministic random values."""
+ torch.manual_seed(seed)
+ with torch.no_grad():
+ for p in model.parameters():
+ p.random_()
+
+
+def _create_model_and_optimizer(encoder_grid, llm_grid, hidden_size, num_layers, vocab_size, seed):
+ """Create MIMO model with DDP + optimizer, do a fake step to populate optimizer state.
+
+ Caller must call create_all_embedding_groups() before this function.
+ """
+ torch.manual_seed(seed)
+
+ mimo_model, _, _, _, _ = get_mimo_model(
+ encoder_name=ENCODER_NAME,
+ encoder_grid=encoder_grid,
+ llm_grid=llm_grid,
+ hidden_size=hidden_size,
+ num_layers=num_layers,
+ vocab_size=vocab_size,
+ seq_len=64,
+ )
+ _randomize_params(mimo_model, seed)
+
+ # Use Float16Optimizer (not DistributedOptimizer) to exercise the MIMO-specific
+ # param_groups/grad_scaler extraction in sharded_state_dict. DistributedOptimizer
+ # handles its own checkpointing internally and our code is transparent to it.
+ opt_config = OptimizerConfig(
+ optimizer='adam',
+ lr=1e-4,
+ weight_decay=0.01,
+ clip_grad=1.0,
+ bf16=True,
+ use_distributed_optimizer=False,
+ )
+ optimizer = get_mimo_optimizer(mimo_model, opt_config)
+
+ # Fake backward + step to populate optimizer state (Adam m/v)
+ for param in mimo_model.parameters():
+ param.grad = torch.randn_like(param)
+ optimizer.step()
+
+ return mimo_model, optimizer
+
+
+def run_checkpoint_test(
+ encoder_tp,
+ encoder_pp,
+ encoder_dp,
+ encoder_offset,
+ llm_tp,
+ llm_pp,
+ llm_dp,
+ llm_offset,
+ hidden_size=256,
+ num_layers=2,
+ vocab_size=1000,
+):
+ """Save model + optimizer checkpoint, load into fresh instances, verify match."""
+ # Clear NVTE env vars that the conftest set_env fixture sets to '0'.
+ # GPTModel (LanguageModule) asserts these are unset or match the attention backend.
+ os.environ.pop('NVTE_FLASH_ATTN', None)
+ os.environ.pop('NVTE_FUSED_ATTN', None)
+ os.environ.pop('NVTE_UNFUSED_ATTN', None)
+
+ encoder_grid = create_hypercomm_grid(
+ offset=encoder_offset, tp=encoder_tp, cp=1, pp=encoder_pp, dp=encoder_dp
+ )
+ llm_grid = create_hypercomm_grid(offset=llm_offset, tp=llm_tp, cp=1, pp=llm_pp, dp=llm_dp)
+ create_all_embedding_groups([encoder_grid, llm_grid])
+
+ # --- Create model A + optimizer, snapshot state ---
+ model_a, optimizer_a = _create_model_and_optimizer(
+ encoder_grid, llm_grid, hidden_size, num_layers, vocab_size, seed=1
+ )
+ params_a = {name: p.clone() for name, p in model_a.named_parameters()}
+
+ ckpt_dir = _get_shared_tmpdir()
+ try:
+ model_ckpt = os.path.join(ckpt_dir, 'model')
+ optim_ckpt = os.path.join(ckpt_dir, 'optimizer')
+ if dist.get_rank() == 0:
+ os.makedirs(model_ckpt)
+ os.makedirs(optim_ckpt)
+ dist.barrier()
+
+ # Save model
+ save(model_a.sharded_state_dict(), model_ckpt)
+
+ # Save optimizer (needs fresh model sharded_state_dict since save() consumes tensor refs)
+ optim_sd_a = optimizer_a.sharded_state_dict(model_a.sharded_state_dict(), is_loading=False)
+ save(optim_sd_a, optim_ckpt, validate_access_integrity=False)
+
+ dist.barrier()
+
+ # --- Create model B + optimizer with different weights (reuse same grids) ---
+ model_b, optimizer_b = _create_model_and_optimizer(
+ encoder_grid, llm_grid, hidden_size, num_layers, vocab_size, seed=2
+ )
+
+ # Load model
+ model_sd_b = model_b.sharded_state_dict()
+ loaded_model_sd, missing, unexpected = load(
+ model_sd_b, model_ckpt, strict=StrictHandling.RETURN_ALL
+ )
+ real_missing = [k for k in missing if '_extra_state' not in k]
+ real_unexpected = [k for k in unexpected if '_extra_state' not in k]
+ assert not real_missing, f"Missing keys: {real_missing}"
+ assert not real_unexpected, f"Unexpected keys: {real_unexpected}"
+ model_b.load_state_dict(loaded_model_sd)
+
+ # Load optimizer
+ optim_sd_b = optimizer_b.sharded_state_dict(model_b.sharded_state_dict(), is_loading=True)
+ loaded_optim_sd = load(optim_sd_b, optim_ckpt, validate_access_integrity=False)
+ optimizer_b.load_state_dict(loaded_optim_sd)
+
+ # --- Verify model params match ---
+ mismatches = [
+ name
+ for name, p in model_b.named_parameters()
+ if name in params_a and not torch.equal(p, params_a[name])
+ ]
+ assert not mismatches, f"Model param mismatch after load: {mismatches}"
+
+ # --- Verify optimizer state matches (param_groups + Adam m/v tensors) ---
+ for name, info_b in optimizer_b.module_infos.items():
+ if not (info_b.is_active and info_b.optimizer):
+ continue
+ info_a = optimizer_a.module_infos[name]
+ sd_a = info_a.optimizer.state_dict()
+ sd_b = info_b.optimizer.state_dict()
+
+ # Verify param_groups
+ pg_a = sd_a.get('optimizer', {}).get('param_groups', [])
+ pg_b = sd_b.get('optimizer', {}).get('param_groups', [])
+ assert len(pg_a) == len(pg_b), f"Optimizer {name}: param_groups count mismatch"
+ for i, (ga, gb) in enumerate(zip(pg_a, pg_b)):
+ assert ga['lr'] == gb['lr'], f"Optimizer {name} group[{i}]: lr mismatch"
+
+ # Verify Adam state tensors (exp_avg, exp_avg_sq)
+ state_a = sd_a.get('optimizer', {}).get('state', {})
+ state_b = sd_b.get('optimizer', {}).get('state', {})
+ for param_id in state_a:
+ if param_id not in state_b:
+ continue
+ for key in ('exp_avg', 'exp_avg_sq'):
+ if key in state_a[param_id] and key in state_b[param_id]:
+ assert torch.equal(
+ state_a[param_id][key], state_b[param_id][key]
+ ), f"Optimizer {name} param {param_id} {key} mismatch"
+
+ finally:
+ _cleanup_tmpdir(ckpt_dir)
+
+
+@pytest.mark.skipif(
+ version.parse(torch.__version__) < version.parse('2.3.0'),
+ reason="Device mesh requires PyTorch 2.3+",
+)
+class TestMimoCheckpoint:
+ """Distributed checkpoint save/load tests for non-colocated MiMo (8 GPUs)."""
+
+ @classmethod
+ def setup_class(cls):
+ Utils.initialize_distributed()
+ cls.world_size = dist.get_world_size()
+
+ @classmethod
+ def teardown_class(cls):
+ Utils.destroy_model_parallel()
+
+ def teardown_method(self):
+ destroy_all_grids()
+
+ def test_encoder_tp2_llm_tp2_pp3(self):
+ if self.world_size != 8:
+ pytest.skip(f"Requires 8 GPUs, got {self.world_size}")
+ run_checkpoint_test(
+ encoder_tp=2,
+ encoder_pp=1,
+ encoder_dp=1,
+ encoder_offset=0,
+ llm_tp=2,
+ llm_pp=3,
+ llm_dp=1,
+ llm_offset=2,
+ hidden_size=256,
+ num_layers=3,
+ )
+
+ def test_encoder_tp1_llm_pp7(self):
+ if self.world_size != 8:
+ pytest.skip(f"Requires 8 GPUs, got {self.world_size}")
+ run_checkpoint_test(
+ encoder_tp=1,
+ encoder_pp=1,
+ encoder_dp=1,
+ encoder_offset=0,
+ llm_tp=1,
+ llm_pp=7,
+ llm_dp=1,
+ llm_offset=1,
+ hidden_size=256,
+ num_layers=7,
+ )
+
+ def test_encoder_tp2_pp2_llm_tp2_pp2(self):
+ if self.world_size != 8:
+ pytest.skip(f"Requires 8 GPUs, got {self.world_size}")
+ run_checkpoint_test(
+ encoder_tp=2,
+ encoder_pp=2,
+ encoder_dp=1,
+ encoder_offset=0,
+ llm_tp=2,
+ llm_pp=2,
+ llm_dp=1,
+ llm_offset=4,
+ hidden_size=256,
+ num_layers=2,
+ )
diff --git a/tests/unit_tests/models/test_mimo_embedding_alignment.py b/tests/unit_tests/models/test_mimo_embedding_alignment.py
index 74596256774..688ebe4832b 100644
--- a/tests/unit_tests/models/test_mimo_embedding_alignment.py
+++ b/tests/unit_tests/models/test_mimo_embedding_alignment.py
@@ -20,7 +20,14 @@ class TestEmbeddingAlignment:
def setup_method(self):
"""Set up for each test."""
# Create a minimal MimoModelConfig
- language_model_spec = ModuleSpec(module=MagicMock, params={'config': MagicMock()})
+ language_model_spec = ModuleSpec(
+ module=MagicMock,
+ params={
+ 'config': MagicMock(
+ context_parallel_size=1, sequence_parallel=False, pipeline_model_parallel_size=1
+ )
+ },
+ )
self.mimo_config = MimoModelConfig(
language_model_spec=language_model_spec,
modality_submodules_spec={},
diff --git a/tests/unit_tests/models/test_mimo_model.py b/tests/unit_tests/models/test_mimo_model.py
index f786f118c68..e1c4b6e89bf 100644
--- a/tests/unit_tests/models/test_mimo_model.py
+++ b/tests/unit_tests/models/test_mimo_model.py
@@ -1,19 +1,22 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
'''
-WORLD_SIZE=1 LOCAL_RANK=0 python -m pytest tests/unit_tests/models/test_mimo_model.py
+WORLD_SIZE=1 LOCAL_RANK=0 python -m pytest tests/unit_tests/models/test_mimo_model.py
'''
import math
+from unittest.mock import MagicMock, patch
import pytest
import torch
+import torch.distributed as dist
import torch.nn as nn
from transformers import WhisperConfig, WhisperModel
from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
from megatron.core.models.gpt.gpt_model import GPTModel
from megatron.core.models.mimo.config.base_configs import MimoModelConfig
+from megatron.core.models.mimo.config.role import MIMO_LANGUAGE_MODULE_KEY, ModuleLayout
from megatron.core.models.mimo.model.base import MimoModel
from megatron.core.models.mimo.submodules.audio import AudioModalitySubmodules
from megatron.core.models.mimo.submodules.vision import VisionModalitySubmodules
@@ -30,13 +33,11 @@
class AudioEncoderWrapper(torch.nn.Module):
"""Generic wrapper for audio encoder models that extracts last_hidden_state."""
- def __init__(self, config):
+ def __init__(self, **kwargs):
super().__init__()
- # Use a local Whisper model (tiny config) to avoid checkpoint download
self.encoder = WhisperModel(WhisperConfig()).encoder
def forward(self, input_features):
- # Process through encoder and extract last_hidden_state
with torch.no_grad():
return self.encoder(input_features).last_hidden_state
@@ -59,7 +60,6 @@ def get_vision_submodules_spec(hidden_size, img_h, img_w, patch_dim):
},
)
- # Create vision projection spec
vision_projection_spec = ModuleSpec(
module=nn.Linear,
params={
@@ -68,8 +68,7 @@ def get_vision_submodules_spec(hidden_size, img_h, img_w, patch_dim):
},
)
- # Create vision modality spec
- vision_submodule_spec = ModuleSpec(
+ return ModuleSpec(
module=VisionModalitySubmodules,
submodules={
"encoders": {"clip_encoder": vision_encoder_spec},
@@ -77,36 +76,17 @@ def get_vision_submodules_spec(hidden_size, img_h, img_w, patch_dim):
},
)
- return vision_submodule_spec
-
def get_audio_submodules_spec(hidden_size):
"""Get the submodule spec for the audio modality."""
-
- class AudioEncoderWrapper(torch.nn.Module):
- """Generic wrapper for audio encoder models that extracts last_hidden_state."""
-
- def __init__(self, model_name="openai/whisper-tiny"):
- super().__init__()
- # Local tiny Whisper model with random weights
- self.encoder = WhisperModel(WhisperConfig()).encoder
-
- def forward(self, input_features):
- # Process through encoder and extract last_hidden_state
- with torch.no_grad():
- return self.encoder(input_features).last_hidden_state
-
- # Audio modality configuration
- audio_encoder_spec = ModuleSpec(
- module=AudioEncoderWrapper, params={"model_name": "openai/whisper-tiny"}
- )
+ audio_encoder_spec = ModuleSpec(module=AudioEncoderWrapper, params={})
audio_projection_spec = ModuleSpec(
module=nn.Linear,
params={"in_features": 384, "out_features": hidden_size}, # Whisper tiny hidden size
)
- audio_submodule_spec = ModuleSpec(
+ return ModuleSpec(
module=AudioModalitySubmodules,
submodules={
"encoders": {"whisper_encoder": audio_encoder_spec},
@@ -114,8 +94,6 @@ def forward(self, input_features):
},
)
- return audio_submodule_spec
-
def get_language_model_spec(hidden_size, vocab_size, seq_len):
"""Get the language model spec."""
@@ -123,7 +101,7 @@ def get_language_model_spec(hidden_size, vocab_size, seq_len):
num_layers=2, hidden_size=hidden_size, num_attention_heads=4, use_cpu_initialization=True
)
language_layer_spec = get_gpt_layer_with_transformer_engine_spec()
- language_model_spec = ModuleSpec(
+ return ModuleSpec(
module=GPTModel,
params={
"config": lm_config,
@@ -134,55 +112,44 @@ def get_language_model_spec(hidden_size, vocab_size, seq_len):
"post_process": True,
},
)
- return language_model_spec
def get_avlm_mimo_model(
hidden_size, vocab_size, seq_len, img_h, img_w, patch_dim, special_token_ids
):
- language_model_spec = get_language_model_spec(hidden_size, vocab_size, seq_len)
- vision_submodule_spec = get_vision_submodules_spec(hidden_size, img_h, img_w, patch_dim)
- audio_submodule_spec = get_audio_submodules_spec(hidden_size)
-
mimo_config = MimoModelConfig(
- language_model_spec=language_model_spec,
- modality_submodules_spec={"images": vision_submodule_spec, "audio": audio_submodule_spec},
+ language_model_spec=get_language_model_spec(hidden_size, vocab_size, seq_len),
+ modality_submodules_spec={
+ "images": get_vision_submodules_spec(hidden_size, img_h, img_w, patch_dim),
+ "audio": get_audio_submodules_spec(hidden_size),
+ },
special_token_ids=special_token_ids,
)
-
- # Create MIMO model
- mimo_model = MimoModel(mimo_config)
- return mimo_model
+ return MimoModel(mimo_config)
def get_vlm_mimo_model(
hidden_size, vocab_size, seq_len, img_h, img_w, patch_dim, special_token_ids
):
- language_model_spec = get_language_model_spec(hidden_size, vocab_size, seq_len)
- vision_submodule_spec = get_vision_submodules_spec(hidden_size, img_h, img_w, patch_dim)
-
mimo_config = MimoModelConfig(
- language_model_spec=language_model_spec,
- modality_submodules_spec={"images": vision_submodule_spec},
+ language_model_spec=get_language_model_spec(hidden_size, vocab_size, seq_len),
+ modality_submodules_spec={
+ "images": get_vision_submodules_spec(hidden_size, img_h, img_w, patch_dim)
+ },
special_token_ids=special_token_ids,
)
-
- # Create MIMO model
- mimo_model = MimoModel(mimo_config)
- return mimo_model
+ return MimoModel(mimo_config)
class TestMimoModel:
"""Test the MimoModel class."""
def setup_method(self, method):
- '''setup env and model'''
try:
Utils.initialize_model_parallel(1, 1)
- except Exception as e:
- print(f"Warning: Could not initialize model parallel: {e}")
+ except Exception:
+ pass
- # Set dimensions
self.hidden_size = 64
self.batch_size = 2
self.seq_len = 2048
@@ -190,21 +157,28 @@ def setup_method(self, method):
self.img_w = 224
self.patch_dim = 16
self.vocab_size = 48000
-
- # Define special token IDs, not in LLM vocab
self.special_token_ids = {"images": 50257, "audio": 50258}
+ self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
def teardown_method(self, method):
- '''teardown env'''
try:
Utils.destroy_model_parallel()
- except Exception as e:
- print(f"Warning: Could not destroy model parallel: {e}")
+ except Exception:
+ pass
- def test_constructor(self):
- """Test constructor initialization."""
+ def _make_vlm(self):
+ return get_vlm_mimo_model(
+ self.hidden_size,
+ self.vocab_size,
+ self.seq_len,
+ self.img_h,
+ self.img_w,
+ self.patch_dim,
+ self.special_token_ids,
+ ).to(self.device)
- mimo_model = get_avlm_mimo_model(
+ def _make_avlm(self):
+ return get_avlm_mimo_model(
self.hidden_size,
self.vocab_size,
self.seq_len,
@@ -212,246 +186,438 @@ def test_constructor(self):
self.img_w,
self.patch_dim,
self.special_token_ids,
+ ).to(self.device)
+
+ def _make_input_ids(self):
+ return torch.randint(
+ 0, self.vocab_size, (self.batch_size, self.seq_len), device=self.device
+ )
+
+ def _make_position_ids(self):
+ return (
+ torch.arange(self.seq_len, device=self.device).unsqueeze(0).expand(self.batch_size, -1)
)
- # Move to device
- device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
- mimo_model = mimo_model.to(device)
+ def test_constructor(self):
+ """Test constructor initialization."""
+ mimo_model = self._make_avlm()
- # Test that modality submodules were initialized correctly
assert "images" in mimo_model.modality_submodules
assert "audio" in mimo_model.modality_submodules
assert isinstance(mimo_model.modality_submodules["images"], VisionModalitySubmodules)
assert isinstance(mimo_model.modality_submodules["audio"], AudioModalitySubmodules)
- # Test that language model was initialized
- assert hasattr(mimo_model, "language_model")
assert isinstance(mimo_model.language_model, GPTModel)
-
- # Test that special token IDs were set correctly
assert mimo_model.special_token_ids == self.special_token_ids
def test_get_text_embeddings(self):
"""Test getting text embeddings."""
- # Create random input and position IDs (within vocab size range)
- device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
- input_ids = torch.randint(
- 0, self.vocab_size, (self.batch_size, self.seq_len), device=device
- )
- position_ids = (
- torch.arange(self.seq_len, device=device).unsqueeze(0).expand(self.batch_size, -1)
- )
- mimo_model = get_avlm_mimo_model(
- self.hidden_size,
- self.vocab_size,
- self.seq_len,
- self.img_h,
- self.img_w,
- self.patch_dim,
- self.special_token_ids,
- )
- mimo_model = mimo_model.to(device)
- # Get text embeddings
+ mimo_model = self._make_avlm()
+ input_ids = self._make_input_ids()
+ position_ids = self._make_position_ids()
+
text_embeddings = mimo_model.get_text_embeddings(
input_ids, position_ids, self.special_token_ids
)
- # Verify shape
- # [b*s, h]
assert text_embeddings.shape == (self.batch_size * self.seq_len, self.hidden_size)
def test_forward_text_only(self):
"""Test forward pass with only text input."""
- # Create inputs
- device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
- input_ids = torch.randint(
- 0, self.vocab_size, (self.batch_size, self.seq_len), device=device
- )
- position_ids = (
- torch.arange(self.seq_len, device=device).unsqueeze(0).expand(self.batch_size, -1)
- )
+ mimo_model = self._make_vlm()
+ input_ids = self._make_input_ids()
+ position_ids = self._make_position_ids()
- mimo_model = get_vlm_mimo_model(
- self.hidden_size,
- self.vocab_size,
- self.seq_len,
- self.img_h,
- self.img_w,
- self.patch_dim,
- self.special_token_ids,
- )
- mimo_model = mimo_model.to(device)
- # Run forward pass with explicit parameters
outputs, _ = mimo_model(
input_ids=input_ids, position_ids=position_ids, modality_inputs=None
)
- assert outputs is not None
-
- # Verify output shape
assert outputs.shape == (self.batch_size, self.seq_len, self.vocab_size)
def test_forward_with_image_modality(self):
"""Test forward pass with text and image input."""
- # Calculate expected number of image tokens based on image size and patch dimension
- device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
expected_img_seq_len = (self.img_h // self.patch_dim) * (
self.img_w // self.patch_dim
) + 1 # +1 for CLS token
- # Create a fixed distribution of images: 3 in first sample, 2 in second sample
num_images = 5
- images_per_sample = [3, 2] # Must sum to num_images
- assert sum(images_per_sample) == num_images
- assert len(images_per_sample) == self.batch_size
+ images_per_sample = [3, 2]
+ images = torch.rand(num_images, 3, self.img_h, self.img_w, device=self.device)
+ input_ids = self._make_input_ids()
+ position_ids = self._make_position_ids()
- # Create images tensor
- images = torch.rand(
- num_images, 3, self.img_h, self.img_w, device=device
- ) # [num_images, 3, h, w] format
-
- # Create input_ids with text tokens
- input_ids = torch.randint(
- 0, self.vocab_size, (self.batch_size, self.seq_len), device=device
- )
-
- # Create position_ids
- position_ids = (
- torch.arange(self.seq_len, device=device).unsqueeze(0).expand(self.batch_size, -1)
- )
-
- # Include image special tokens in input IDs
+ # Place image special tokens in each batch sample
image_token_id = self.special_token_ids["images"]
- start_pos = 5 # Start position for image tokens
-
- # Make sure there's enough space in the sequence for all image tokens in each sample
- for b in range(self.batch_size):
- tokens_needed = images_per_sample[b] * expected_img_seq_len
- assert (
- start_pos + tokens_needed <= self.seq_len
- ), f"Sequence length too short for image tokens in sample {b}"
-
- # Add image tokens to each batch sample according to its number of images
+ start_pos = 5
for b in range(self.batch_size):
tokens_in_this_batch = images_per_sample[b] * expected_img_seq_len
- if tokens_in_this_batch > 0:
- input_ids[b, start_pos : start_pos + tokens_in_this_batch] = image_token_id
+ input_ids[b, start_pos : start_pos + tokens_in_this_batch] = image_token_id
- # Create modality inputs using the new structure
modality_inputs = {"images": {"clip_encoder": {"x": images}}}
- mimo_model = get_vlm_mimo_model(
- self.hidden_size,
- self.vocab_size,
- self.seq_len,
- self.img_h,
- self.img_w,
- self.patch_dim,
- self.special_token_ids,
- )
- mimo_model = mimo_model.to(device)
-
- # Run forward pass with new interface
+ mimo_model = self._make_vlm()
outputs, _ = mimo_model(
input_ids=input_ids, position_ids=position_ids, modality_inputs=modality_inputs
)
- assert outputs is not None
-
- # Verify output shape
assert outputs.shape == (self.batch_size, self.seq_len, self.vocab_size)
def test_forward_with_image_and_audio_modality(self):
"""Test forward pass with text, image, and audio input."""
- device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+ mimo_model = self._make_avlm()
- mimo_model = get_avlm_mimo_model(
- self.hidden_size,
- self.vocab_size,
- self.seq_len,
- self.img_h,
- self.img_w,
- self.patch_dim,
- self.special_token_ids,
- )
- mimo_model = mimo_model.to(device)
-
- # Calculate image sequence length
img_seq_len = (self.img_h // self.patch_dim) * (self.img_w // self.patch_dim) + 1
-
encoder_down_sampling = 2
-
- # Create simple audio input (30 sec)
- mel_bins = 80 # Whisper uses 80 mel bins
+ mel_bins = 80
time_bins = 3000 # 30 seconds of audio at 10ms per frame
- audio_features = torch.rand(2, mel_bins, time_bins, device=device)
+ audio_seq_len = math.ceil(time_bins / encoder_down_sampling)
- # Calculate audio sequence length using Whisper's formula
- audio_seq_len = math.ceil(time_bins / encoder_down_sampling) # 1500 tokens
+ input_ids = self._make_input_ids()
+ position_ids = self._make_position_ids()
- # Create batch data
- batch_size = 2
- seq_len = self.seq_len
-
- # Create input_ids with special tokens
- input_ids = torch.randint(0, self.vocab_size, (batch_size, seq_len), device=device)
- position_ids = torch.arange(seq_len, device=device).unsqueeze(0).expand(batch_size, -1)
-
- # Add special tokens at specific positions
+ # Place image and audio special tokens
start_pos = 5
image_token_id = self.special_token_ids["images"]
audio_token_id = self.special_token_ids["audio"]
-
- # Place image tokens followed by audio tokens in each batch item
- for i in range(batch_size):
- # Add image tokens
+ for i in range(self.batch_size):
input_ids[i, start_pos : start_pos + img_seq_len] = image_token_id
- # Add audio tokens after a gap
- input_ids[
- i, start_pos + img_seq_len + 10 : start_pos + img_seq_len + 10 + audio_seq_len
- ] = audio_token_id
+ audio_start = start_pos + img_seq_len + 10
+ input_ids[i, audio_start : audio_start + audio_seq_len] = audio_token_id
- # Prepare modality inputs
modality_inputs = {
"images": {
- "clip_encoder": {"x": torch.rand(2, 3, self.img_h, self.img_w, device=device)}
+ "clip_encoder": {"x": torch.rand(2, 3, self.img_h, self.img_w, device=self.device)}
+ },
+ "audio": {
+ "whisper_encoder": {
+ "input_features": torch.rand(2, mel_bins, time_bins, device=self.device)
+ }
},
- "audio": {"whisper_encoder": {"input_features": audio_features}},
}
- # Run forward pass
outputs, _ = mimo_model(
input_ids=input_ids, position_ids=position_ids, modality_inputs=modality_inputs
)
-
- # Verify output shape
- assert outputs is not None
- assert outputs.shape == (batch_size, seq_len, self.vocab_size)
+ assert outputs.shape == (self.batch_size, self.seq_len, self.vocab_size)
def test_state_dict(self):
"""Test state dict methods."""
- # Get state dict
- mimo_model = get_avlm_mimo_model(
+ mimo_model = self._make_avlm()
+ state_dict = mimo_model.state_dict()
+ assert len(state_dict) > 0
+ assert any(k.startswith("language_model.") for k in state_dict)
+ assert any(k.startswith("modality_submodules.") for k in state_dict)
+
+ checkpoint_dict = mimo_model.state_dict_for_save_checkpoint()
+ assert len(checkpoint_dict) > 0
+
+ def test_pipeline_model_parallel_accepted(self):
+ """Test that MimoModel accepts pipeline_model_parallel_size > 1."""
+ lm_config_pp2 = TransformerConfig(
+ num_layers=2,
+ hidden_size=self.hidden_size,
+ num_attention_heads=4,
+ use_cpu_initialization=True,
+ pipeline_model_parallel_size=2,
+ pipeline_dtype=torch.float32,
+ )
+ language_model_spec_pp2 = ModuleSpec(
+ module=GPTModel,
+ params={
+ "config": lm_config_pp2,
+ "transformer_layer_spec": get_gpt_layer_with_transformer_engine_spec(),
+ "vocab_size": self.vocab_size,
+ "max_sequence_length": self.seq_len,
+ "pre_process": True,
+ "post_process": True,
+ },
+ )
+ mimo_config = MimoModelConfig(
+ language_model_spec=language_model_spec_pp2,
+ modality_submodules_spec={},
+ special_token_ids=self.special_token_ids,
+ )
+
+ model = MimoModel(mimo_config)
+ assert model is not None
+
+ def test_partition_adapter_none_by_default(self):
+ """Test that partition_adapter is None with default config (no CP/SP)."""
+ mimo_model = self._make_vlm()
+ assert mimo_model.partition_adapter is None
+
+ def test_forward_with_packing_kwargs(self):
+ """Test that packing_kwargs builds PackedSeqParams with qkv_format='thd' and int32 seqlens."""
+ from megatron.core.packed_seq_params import PackedSeqParams
+
+ mimo_model = self._make_vlm()
+ input_ids = self._make_input_ids()
+ position_ids = self._make_position_ids()
+
+ cu_seqlens = torch.tensor(
+ [0, self.seq_len, 2 * self.seq_len], dtype=torch.int64, device=self.device
+ )
+ packing_kwargs = {"cu_seqlens_q": cu_seqlens.clone(), "cu_seqlens_kv": cu_seqlens.clone()}
+
+ text_emb = torch.zeros(self.batch_size * self.seq_len, self.hidden_size, device=self.device)
+ combined_emb = torch.zeros(
+ self.seq_len, self.batch_size, self.hidden_size, device=self.device
+ )
+
+ captured = {}
+
+ def capture_lm_forward(*args, **kwargs):
+ captured['packed_seq_params'] = kwargs.get('packed_seq_params')
+ return torch.zeros(self.batch_size, self.seq_len, self.vocab_size, device=self.device)
+
+ with (
+ patch.object(mimo_model, 'get_text_embeddings', return_value=text_emb),
+ patch.object(
+ mimo_model, 'align_embeddings_by_token_positions', return_value=combined_emb
+ ),
+ patch.object(mimo_model.language_model, 'forward', side_effect=capture_lm_forward),
+ ):
+ mimo_model(
+ input_ids=input_ids,
+ position_ids=position_ids,
+ modality_inputs=None,
+ packing_kwargs=packing_kwargs,
+ )
+
+ packed_seq_params = captured['packed_seq_params']
+ assert isinstance(packed_seq_params, PackedSeqParams)
+ assert packed_seq_params.qkv_format == 'thd'
+ assert packed_seq_params.cu_seqlens_q.dtype == torch.int32
+ assert packed_seq_params.cu_seqlens_kv.dtype == torch.int32
+
+ def test_forward_with_partition_adapter(self):
+ """Test that partition_adapter.shard() is called and embeddings are transposed correctly."""
+ mimo_model = self._make_vlm()
+ input_ids = self._make_input_ids()
+ position_ids = self._make_position_ids()
+
+ sharded_seq_len = self.seq_len // 2
+ sharded_emb = torch.zeros(
+ self.batch_size, sharded_seq_len, self.hidden_size, device=self.device
+ )
+ mock_adapter = MagicMock()
+ mock_adapter.shard.return_value = (sharded_emb, None, None, None, None)
+ mimo_model.partition_adapter = mock_adapter
+
+ text_emb = torch.zeros(self.batch_size * self.seq_len, self.hidden_size, device=self.device)
+ combined_emb = torch.zeros(
+ self.seq_len, self.batch_size, self.hidden_size, device=self.device
+ )
+
+ captured = {}
+
+ def capture_lm_forward(*args, **kwargs):
+ captured['decoder_input'] = kwargs.get('decoder_input')
+ return torch.zeros(
+ self.batch_size, sharded_seq_len, self.vocab_size, device=self.device
+ )
+
+ with (
+ patch.object(mimo_model, 'get_text_embeddings', return_value=text_emb),
+ patch.object(
+ mimo_model, 'align_embeddings_by_token_positions', return_value=combined_emb
+ ),
+ patch.object(mimo_model.language_model, 'forward', side_effect=capture_lm_forward),
+ ):
+ mimo_model(input_ids=input_ids, position_ids=position_ids, modality_inputs=None)
+
+ mock_adapter.shard.assert_called_once()
+ shard_kwargs = mock_adapter.shard.call_args[1]
+ assert shard_kwargs['embeddings'].shape == (self.batch_size, self.seq_len, self.hidden_size)
+ assert captured['decoder_input'].shape == (
+ sharded_seq_len,
+ self.batch_size,
+ self.hidden_size,
+ )
+
+
+class MockProcessGroup:
+ """Mock process group for testing."""
+
+ def __init__(self, rank, world_size):
+ self._rank = rank
+ self._size = world_size
+
+ def rank(self):
+ return self._rank
+
+ def size(self):
+ return self._size
+
+
+class MockGrid:
+ """Mock grid with HyperCommGrid-compatible interface."""
+
+ def __init__(self, rank_offset=0, size=1, dim_names=None, pp_rank=0, pp_size=1):
+ self.rank_offset = rank_offset
+ self.size = size
+ self.dim_names = dim_names or []
+ self._pp_group = MockProcessGroup(pp_rank, pp_size)
+
+ def get_pg(self, dims):
+ if dims == "pp":
+ return self._pp_group
+ raise KeyError(f"Process group for {dims} not found")
+
+
+class TestMimoModelNonColocated:
+ """Tests for non-colocated multi-module pipeline parallelism."""
+
+ def setup_method(self, method):
+ try:
+ Utils.initialize_model_parallel(1, 1)
+ except Exception:
+ pass
+ self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+ self.hidden_size = 64
+ self.vocab_size = 48000
+ self.seq_len = 256
+ self.batch_size = 2
+ self.img_h = 224
+ self.img_w = 224
+ self.patch_dim = 16
+
+ def teardown_method(self, method):
+ try:
+ Utils.destroy_model_parallel()
+ except Exception:
+ pass
+
+ def _make_config(self, encoder_in_grid=True, language_in_grid=True, pp_rank=0, pp_size=1):
+ """Helper to create MimoModelConfig with mock grids."""
+ language_model_spec = get_language_model_spec(
+ self.hidden_size, self.vocab_size, self.seq_len
+ )
+ vision_submodule_spec = get_vision_submodules_spec(
+ self.hidden_size, self.img_h, self.img_w, self.patch_dim
+ )
+
+ world_size = dist.get_world_size()
+ encoder_offset = 0 if encoder_in_grid else world_size
+ language_offset = 0 if language_in_grid else world_size
+
+ return MimoModelConfig(
+ language_model_spec=language_model_spec,
+ modality_submodules_spec={"images": vision_submodule_spec},
+ special_token_ids={"images": 50257},
+ module_to_grid_map={
+ "images": MockGrid(
+ rank_offset=encoder_offset,
+ size=world_size,
+ dim_names=["pp"] if pp_size > 1 else [],
+ pp_rank=pp_rank,
+ pp_size=pp_size,
+ ),
+ MIMO_LANGUAGE_MODULE_KEY: MockGrid(
+ rank_offset=language_offset,
+ size=world_size,
+ dim_names=["pp"] if pp_size > 1 else [],
+ pp_rank=pp_rank,
+ pp_size=pp_size,
+ ),
+ },
+ )
+
+ def test_grid_validation_rejects_mismatched_keys(self):
+ """Test validation fails when grid_map keys don't match expected modules."""
+ language_model_spec = get_language_model_spec(
+ self.hidden_size, self.vocab_size, self.seq_len
+ )
+ vision_submodule_spec = get_vision_submodules_spec(
+ self.hidden_size, self.img_h, self.img_w, self.patch_dim
+ )
+
+ mimo_config = MimoModelConfig(
+ language_model_spec=language_model_spec,
+ modality_submodules_spec={"images": vision_submodule_spec},
+ special_token_ids={"images": 50257},
+ module_to_grid_map={MIMO_LANGUAGE_MODULE_KEY: MockGrid()},
+ )
+
+ with pytest.raises(ValueError, match="module_to_grid_map keys must match"):
+ MimoModel(mimo_config)
+
+ def test_role_determination(self):
+ """Test role correctly identifies modules and stage positions."""
+ # No grid map = colocated role with all modules
+ model_no_grid = get_vlm_mimo_model(
self.hidden_size,
self.vocab_size,
self.seq_len,
self.img_h,
self.img_w,
self.patch_dim,
- self.special_token_ids,
+ {"images": 50257},
+ )
+ assert model_no_grid.role.mode == ModuleLayout.UNIFIED
+ assert model_no_grid.role.has_language_module is True
+ assert model_no_grid.role.has_modality_modules is True
+
+ # Encoder-only rank
+ model_encoder = MimoModel(self._make_config(encoder_in_grid=True, language_in_grid=False))
+ assert model_encoder.role.has_modality_modules is True
+ assert model_encoder.role.has_language_module is False
+
+ # Language-only rank
+ model_language = MimoModel(self._make_config(encoder_in_grid=False, language_in_grid=True))
+ assert model_language.role.has_modality_modules is False
+ assert model_language.role.has_language_module is True
+
+ # Stage info with PP
+ model_pp = MimoModel(
+ self._make_config(encoder_in_grid=True, language_in_grid=True, pp_rank=1, pp_size=3)
+ )
+ assert model_pp.role.is_first_stage("images") is False
+ assert model_pp.role.is_last_stage("images") is False
+
+ def test_selective_init_encoder_only(self):
+ """Test encoder-only rank initializes encoder but not language model."""
+ model = MimoModel(self._make_config(encoder_in_grid=True, language_in_grid=False))
+ assert "images" in model.modality_submodules
+ assert model.language_model is None
+
+ def test_selective_init_language_only(self):
+ """Test language-only rank initializes language model but not encoder."""
+ model = MimoModel(self._make_config(encoder_in_grid=False, language_in_grid=True))
+ assert "images" not in model.modality_submodules
+ assert model.language_model is not None
+
+ def test_forward_encoder_only(self):
+ """Test encoder-only forward returns dict of embeddings."""
+ model = MimoModel(self._make_config(encoder_in_grid=True, language_in_grid=False))
+ model = model.to(self.device)
+
+ images = torch.rand(2, 3, self.img_h, self.img_w, device=self.device)
+ input_ids = torch.randint(
+ 0, self.vocab_size, (self.batch_size, self.seq_len), device=self.device
)
- state_dict = mimo_model.state_dict()
- assert len(state_dict) > 0
- # Make sure we have keys for language model and modality submodules
- has_lm_keys = False
- has_modality_keys = False
+ outputs, _ = model(
+ input_ids=input_ids, modality_inputs={"images": {"clip_encoder": {"x": images}}}
+ )
+ assert isinstance(outputs, dict)
+ assert "images" in outputs
- for key in state_dict.keys():
- if key.startswith("language_model."):
- has_lm_keys = True
- if key.startswith("modality_submodules."):
- has_modality_keys = True
+ def test_forward_language_only(self):
+ """Test language-only forward returns tensor."""
+ model = MimoModel(self._make_config(encoder_in_grid=False, language_in_grid=True))
+ model = model.to(self.device)
- assert has_lm_keys
- assert has_modality_keys
+ img_seq_len = (self.img_h // self.patch_dim) * (self.img_w // self.patch_dim) + 1
+ input_ids = torch.randint(
+ 0, self.vocab_size, (self.batch_size, self.seq_len), device=self.device
+ )
+ input_ids[:, 5 : 5 + img_seq_len] = 50257
+ position_ids = (
+ torch.arange(self.seq_len, device=self.device).unsqueeze(0).expand(self.batch_size, -1)
+ )
- # Test checkpoint state dict
- checkpoint_dict = mimo_model.state_dict_for_save_checkpoint()
- assert len(checkpoint_dict) > 0
+ encoder_embeddings = torch.randn(
+ self.batch_size * img_seq_len, self.hidden_size, device=self.device
+ )
+ model.set_input_tensor({"images": encoder_embeddings})
+
+ outputs, _ = model(input_ids=input_ids, position_ids=position_ids, modality_inputs=None)
+ assert isinstance(outputs, torch.Tensor)
+ assert outputs.shape == (self.batch_size, self.seq_len, self.vocab_size)
diff --git a/tests/unit_tests/models/test_mimo_partition.py b/tests/unit_tests/models/test_mimo_partition.py
new file mode 100644
index 00000000000..1527fb92935
--- /dev/null
+++ b/tests/unit_tests/models/test_mimo_partition.py
@@ -0,0 +1,434 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+
+'''
+WORLD_SIZE=1 LOCAL_RANK=0 python -m torch.distributed.run \
+ --nproc_per_node=1 -m pytest \
+ tests/unit_tests/models/test_mimo_partition.py -v
+'''
+
+from unittest.mock import MagicMock, patch
+
+import pytest
+import torch
+
+from megatron.core.models.mimo.partition.utils import PartitionAdapter, PartitionConfig
+from megatron.core.transformer.transformer_config import TransformerConfig
+
+
+@pytest.mark.experimental
+class TestPartitionConfig:
+ """Tests for PartitionConfig dataclass and factory method."""
+
+ def test_is_partitioning_enabled_cp_only(self):
+ cfg = PartitionConfig(
+ seq_parallel=False, use_cp=True, tp_comm_overlap=False, max_seq_len=128
+ )
+ assert cfg.is_partitioning_enabled is True
+
+ def test_is_partitioning_enabled_sp_only(self):
+ cfg = PartitionConfig(
+ seq_parallel=True, use_cp=False, tp_comm_overlap=False, max_seq_len=128
+ )
+ assert cfg.is_partitioning_enabled is True
+
+ def test_is_partitioning_enabled_both(self):
+ cfg = PartitionConfig(
+ seq_parallel=True, use_cp=True, tp_comm_overlap=False, max_seq_len=128
+ )
+ assert cfg.is_partitioning_enabled is True
+
+ def test_is_partitioning_enabled_neither(self):
+ cfg = PartitionConfig(
+ seq_parallel=False, use_cp=False, tp_comm_overlap=False, max_seq_len=128
+ )
+ assert cfg.is_partitioning_enabled is False
+
+ def test_from_mp_config_invalid_type_raises(self):
+ with pytest.raises(TypeError, match="mp must be a ModelParallelConfig instance"):
+ PartitionConfig.from_mp_config("not_a_config", max_seq_len=128)
+
+ def test_from_mp_config_no_parallelism(self):
+ mp = TransformerConfig(
+ num_layers=1,
+ hidden_size=64,
+ num_attention_heads=4,
+ context_parallel_size=1,
+ sequence_parallel=False,
+ )
+ with patch('megatron.core.models.mimo.partition.utils.get_pg_size', return_value=1):
+ cfg = PartitionConfig.from_mp_config(mp, max_seq_len=512)
+ assert cfg.use_cp is False
+ assert cfg.seq_parallel is False
+ assert cfg.cp_group is None
+ assert cfg.tp_group is None
+ assert cfg.max_seq_len == 512
+
+ def test_from_mp_config_kv_format_thd(self):
+ mp = TransformerConfig(num_layers=1, hidden_size=64, num_attention_heads=4)
+ with patch('megatron.core.models.mimo.partition.utils.get_pg_size', return_value=1):
+ cfg = PartitionConfig.from_mp_config(mp, max_seq_len=512, kv_format='thd')
+ assert cfg.kv_format == 'thd'
+
+ def test_from_mp_config_explicit_cp_group(self):
+ mock_cp_group = MagicMock()
+ mp = TransformerConfig(
+ num_layers=1, hidden_size=64, num_attention_heads=4, context_parallel_size=2
+ )
+ with patch('megatron.core.models.mimo.partition.utils.get_pg_size', return_value=2):
+ cfg = PartitionConfig.from_mp_config(mp, max_seq_len=512, cp_group=mock_cp_group)
+ assert cfg.use_cp is True
+ assert cfg.cp_group is mock_cp_group
+
+ def test_from_mp_config_explicit_tp_group(self):
+ mock_tp_group = MagicMock()
+ mp = TransformerConfig(
+ num_layers=1,
+ hidden_size=64,
+ num_attention_heads=4,
+ tensor_model_parallel_size=2,
+ sequence_parallel=True,
+ )
+ with patch('megatron.core.models.mimo.partition.utils.get_pg_size', return_value=1):
+ cfg = PartitionConfig.from_mp_config(mp, max_seq_len=512, tp_group=mock_tp_group)
+ assert cfg.seq_parallel is True
+ assert cfg.tp_group is mock_tp_group
+
+ def test_from_mp_config_auto_fetch_cp_group(self):
+ mock_group = MagicMock()
+ mp = TransformerConfig(
+ num_layers=1, hidden_size=64, num_attention_heads=4, context_parallel_size=2
+ )
+ with (
+ patch(
+ 'megatron.core.models.mimo.partition.utils.get_context_parallel_group',
+ return_value=mock_group,
+ ),
+ patch('megatron.core.models.mimo.partition.utils.get_pg_size', return_value=2),
+ ):
+ cfg = PartitionConfig.from_mp_config(mp, max_seq_len=512)
+ assert cfg.cp_group is mock_group
+
+ def test_from_mp_config_auto_fetch_tp_group(self):
+ mock_group = MagicMock()
+ mp = TransformerConfig(
+ num_layers=1,
+ hidden_size=64,
+ num_attention_heads=4,
+ tensor_model_parallel_size=2,
+ sequence_parallel=True,
+ )
+ with (
+ patch(
+ 'megatron.core.models.mimo.partition.utils.get_tensor_model_parallel_group',
+ return_value=mock_group,
+ ),
+ patch('megatron.core.models.mimo.partition.utils.get_pg_size', return_value=1),
+ ):
+ cfg = PartitionConfig.from_mp_config(mp, max_seq_len=512)
+ assert cfg.tp_group is mock_group
+
+
+@pytest.mark.experimental
+class TestPartitionAdapterShard:
+ """Tests for PartitionAdapter.shard()."""
+
+ def _make_cfg(
+ self,
+ use_cp=False,
+ seq_parallel=False,
+ tp_comm_overlap=False,
+ max_seq_len=128,
+ cp_group=None,
+ tp_group=None,
+ ):
+ return PartitionConfig(
+ use_cp=use_cp,
+ seq_parallel=seq_parallel,
+ tp_comm_overlap=tp_comm_overlap,
+ max_seq_len=max_seq_len,
+ cp_group=cp_group,
+ tp_group=tp_group,
+ )
+
+ def _make_tensors(self, B=2, S=8, H=16):
+ embeddings = torch.rand(B, S, H)
+ labels = torch.randint(0, 100, (B, S))
+ loss_mask = torch.ones(B, S)
+ attention_mask = torch.ones(B, S)
+ return embeddings, labels, loss_mask, attention_mask
+
+ def test_noop_when_both_disabled(self):
+ """No sharding when neither CP nor SP is enabled — inputs returned as-is."""
+ cfg = self._make_cfg(use_cp=False, seq_parallel=False)
+ adapter = PartitionAdapter(cfg)
+ embeddings, labels, loss_mask, attention_mask = self._make_tensors()
+ out = adapter.shard(embeddings, labels, loss_mask, attention_mask)
+ assert out[0] is embeddings
+ assert out[1] is labels
+ assert out[2] is loss_mask
+ assert out[3] is attention_mask
+ assert out[4] is None
+
+ def test_cp_only_shards_sequence(self):
+ mock_cp_group = MagicMock()
+ cfg = self._make_cfg(use_cp=True, max_seq_len=8, cp_group=mock_cp_group)
+ adapter = PartitionAdapter(cfg)
+ embeddings, labels, loss_mask, attention_mask = self._make_tensors(B=2, S=8, H=16)
+ sharded = {
+ 'embeddings': embeddings[:, :4, :],
+ 'labels': labels[:, :4],
+ 'loss_mask': loss_mask[:, :4],
+ 'attention_mask': attention_mask[:, :4],
+ }
+ with (
+ patch('megatron.core.models.mimo.partition.utils.get_pg_size', return_value=2),
+ patch(
+ 'megatron.core.models.mimo.partition.utils.get_batch_on_this_cp_rank',
+ return_value=sharded,
+ ),
+ ):
+ out = adapter.shard(embeddings, labels, loss_mask, attention_mask)
+ assert out[0].shape == (2, 4, 16)
+ assert out[1].shape == (2, 4)
+
+ def test_sp_only_scatters(self):
+ mock_tp_group = MagicMock()
+ cfg = self._make_cfg(seq_parallel=True, max_seq_len=8, tp_group=mock_tp_group)
+ adapter = PartitionAdapter(cfg)
+ # SP uses seq_dim=0: embeddings shape [S, B, H]
+ embeddings = torch.rand(8, 2, 16)
+ labels = torch.randint(0, 100, (2, 8))
+ loss_mask = torch.ones(2, 8)
+ attention_mask = torch.ones(2, 8)
+ scattered = torch.rand(4, 2, 16)
+ with (
+ patch('megatron.core.models.mimo.partition.utils.get_pg_size', return_value=2),
+ patch(
+ 'megatron.core.models.mimo.partition.utils.tensor_parallel.scatter_to_sequence_parallel_region',
+ return_value=scattered,
+ ),
+ ):
+ out = adapter.shard(embeddings, labels, loss_mask, attention_mask)
+ assert out[0].shape == (4, 2, 16)
+
+ def test_cp_and_sp_combined(self):
+ mock_cp_group = MagicMock()
+ mock_tp_group = MagicMock()
+ cfg = self._make_cfg(
+ use_cp=True,
+ seq_parallel=True,
+ max_seq_len=16,
+ cp_group=mock_cp_group,
+ tp_group=mock_tp_group,
+ )
+ adapter = PartitionAdapter(cfg)
+ # cp_size=2, tp_size=2 → shard_factor = 2*2*2 = 8; S=16 is divisible
+ embeddings = torch.rand(2, 16, 16)
+ labels = torch.randint(0, 100, (2, 16))
+ loss_mask = torch.ones(2, 16)
+ attention_mask = torch.ones(2, 16)
+ cp_sharded = {
+ 'embeddings': embeddings[:, :8, :],
+ 'labels': labels[:, :8],
+ 'loss_mask': loss_mask[:, :8],
+ 'attention_mask': attention_mask[:, :8],
+ }
+ scattered = torch.rand(2, 4, 16)
+
+ with (
+ patch('megatron.core.models.mimo.partition.utils.get_pg_size', return_value=2),
+ patch(
+ 'megatron.core.models.mimo.partition.utils.get_batch_on_this_cp_rank',
+ return_value=cp_sharded,
+ ),
+ patch(
+ 'megatron.core.models.mimo.partition.utils.tensor_parallel.scatter_to_sequence_parallel_region',
+ return_value=scattered,
+ ),
+ ):
+ out = adapter.shard(embeddings, labels, loss_mask, attention_mask)
+ assert out[0].shape == (2, 4, 16)
+
+ def test_seq_not_divisible_raises(self):
+ mock_cp_group = MagicMock()
+ cfg = self._make_cfg(use_cp=True, max_seq_len=7, cp_group=mock_cp_group)
+ adapter = PartitionAdapter(cfg)
+ embeddings = torch.rand(2, 7, 16) # 7 % (2*2) != 0
+ labels = torch.randint(0, 100, (2, 7))
+ loss_mask = torch.ones(2, 7)
+ attention_mask = torch.ones(2, 7)
+ with (
+ patch('megatron.core.models.mimo.partition.utils.get_pg_size', return_value=2),
+ pytest.raises(AssertionError, match="divisible"),
+ ):
+ adapter.shard(embeddings, labels, loss_mask, attention_mask)
+
+ def test_tp_comm_overlap_seq_len_assertion(self):
+ mock_tp_group = MagicMock()
+ cfg = self._make_cfg(
+ seq_parallel=True, tp_comm_overlap=True, max_seq_len=16, tp_group=mock_tp_group
+ )
+ adapter = PartitionAdapter(cfg)
+ # S=8 but max_seq_len=16 → assertion fires
+ embeddings = torch.rand(8, 2, 16) # [S, B, H] for SP
+ labels = torch.randint(0, 100, (2, 8))
+ loss_mask = torch.ones(2, 8)
+ attention_mask = torch.ones(2, 8)
+ with (
+ patch('megatron.core.models.mimo.partition.utils.get_pg_size', return_value=2),
+ pytest.raises(AssertionError, match="TP Comm overlap"),
+ ):
+ adapter.shard(embeddings, labels, loss_mask, attention_mask)
+
+ def test_thd_format_skips_divisibility_check(self):
+ """PackedSeqParams with qkv_format='thd' bypasses the divisibility assertion."""
+ from megatron.core.packed_seq_params import PackedSeqParams
+
+ mock_cp_group = MagicMock()
+ cfg = self._make_cfg(use_cp=True, max_seq_len=7, cp_group=mock_cp_group)
+ adapter = PartitionAdapter(cfg)
+ embeddings = torch.rand(2, 7, 16) # seq_len=7 not divisible by cp*2, but THD skips check
+ labels = torch.randint(0, 100, (2, 7))
+ loss_mask = torch.ones(2, 7)
+ attention_mask = torch.ones(2, 7)
+ packed_seq_params = MagicMock(spec=PackedSeqParams)
+ packed_seq_params.qkv_format = 'thd'
+ packed_seq_params.cu_seqlens_q_padded = torch.tensor([0, 4, 7], dtype=torch.int32)
+
+ # THD path calls tex.thd_get_partitioned_indices — mock it to return first 4 indices
+ fake_index = torch.arange(4, dtype=torch.int32)
+ with (
+ patch('megatron.core.models.mimo.partition.utils.get_pg_size', return_value=2),
+ patch('megatron.core.models.mimo.partition.utils.get_pg_rank', return_value=0),
+ patch('megatron.core.models.mimo.partition.utils.tex') as mock_tex,
+ ):
+ mock_tex.thd_get_partitioned_indices.return_value = fake_index
+ # Should NOT raise AssertionError about divisibility
+ out = adapter.shard(embeddings, labels, loss_mask, attention_mask, packed_seq_params)
+ assert out[0] is not None
+
+ def test_none_embeddings_skips_shard_factor_check(self):
+ """When embeddings is None, the divisibility check is skipped."""
+ mock_cp_group = MagicMock()
+ cfg = self._make_cfg(use_cp=True, max_seq_len=7, cp_group=mock_cp_group)
+ adapter = PartitionAdapter(cfg)
+ labels = torch.randint(0, 100, (2, 7))
+ loss_mask = torch.ones(2, 7)
+ attention_mask = torch.ones(2, 7)
+ cp_sharded = {
+ 'labels': labels[:, :4],
+ 'loss_mask': loss_mask[:, :4],
+ 'attention_mask': attention_mask[:, :4],
+ }
+ with (
+ patch('megatron.core.models.mimo.partition.utils.get_pg_size', return_value=2),
+ patch(
+ 'megatron.core.models.mimo.partition.utils.get_batch_on_this_cp_rank',
+ return_value=cp_sharded,
+ ),
+ ):
+ out = adapter.shard(None, labels, loss_mask, attention_mask)
+ assert out[0] is None
+
+
+@pytest.mark.experimental
+class TestPartitionAdapterApplyContextParallel:
+ """Tests for PartitionAdapter._apply_context_parallel()."""
+
+ def _make_cfg(self, use_cp=True, cp_group=None):
+ return PartitionConfig(
+ use_cp=use_cp,
+ seq_parallel=False,
+ tp_comm_overlap=False,
+ max_seq_len=128,
+ cp_group=cp_group,
+ )
+
+ def test_returns_unchanged_when_cp_disabled(self):
+ cfg = self._make_cfg(use_cp=False)
+ adapter = PartitionAdapter(cfg)
+ embeddings = torch.rand(2, 8, 16)
+ labels = torch.randint(0, 100, (2, 8))
+ loss_mask = torch.ones(2, 8)
+ attention_mask = torch.ones(2, 8)
+ out = adapter._apply_context_parallel(embeddings, labels, loss_mask, attention_mask, None)
+ assert out[0] is embeddings
+ assert out[1] is labels
+ assert out[2] is loss_mask
+ assert out[3] is attention_mask
+
+ def test_sbhd_path_calls_get_batch_on_this_cp_rank(self):
+ mock_cp_group = MagicMock()
+ cfg = self._make_cfg(use_cp=True, cp_group=mock_cp_group)
+ adapter = PartitionAdapter(cfg)
+ embeddings = torch.rand(2, 8, 16)
+ labels = torch.randint(0, 100, (2, 8))
+ loss_mask = torch.ones(2, 8)
+ attention_mask = torch.ones(2, 8)
+ sharded = {
+ 'embeddings': embeddings[:, :4, :],
+ 'labels': labels[:, :4],
+ 'loss_mask': loss_mask[:, :4],
+ 'attention_mask': attention_mask[:, :4],
+ }
+ with patch(
+ 'megatron.core.models.mimo.partition.utils.get_batch_on_this_cp_rank',
+ return_value=sharded,
+ ) as mock_fn:
+ out = adapter._apply_context_parallel(
+ embeddings, labels, loss_mask, attention_mask, None
+ )
+ mock_fn.assert_called_once()
+ assert out[0].shape == (2, 4, 16)
+ assert out[1].shape == (2, 4)
+
+ def test_all_none_inputs_produces_none_outputs(self):
+ mock_cp_group = MagicMock()
+ cfg = self._make_cfg(use_cp=True, cp_group=mock_cp_group)
+ adapter = PartitionAdapter(cfg)
+ with patch(
+ 'megatron.core.models.mimo.partition.utils.get_batch_on_this_cp_rank', return_value={}
+ ):
+ out = adapter._apply_context_parallel(None, None, None, None, None)
+ assert all(v is None for v in out[:4])
+
+ def test_only_non_none_tensors_added_to_batch(self):
+ """None tensors must not appear in the batch dict passed to get_batch_on_this_cp_rank."""
+ mock_cp_group = MagicMock()
+ cfg = self._make_cfg(use_cp=True, cp_group=mock_cp_group)
+ adapter = PartitionAdapter(cfg)
+ embeddings = torch.rand(2, 8, 16)
+ sharded = {'embeddings': embeddings[:, :4, :]}
+ captured = {}
+
+ def mock_fn(batch):
+ captured.update(batch)
+ return sharded
+
+ with patch(
+ 'megatron.core.models.mimo.partition.utils.get_batch_on_this_cp_rank',
+ side_effect=mock_fn,
+ ):
+ out = adapter._apply_context_parallel(embeddings, None, None, None, None)
+
+ assert 'embeddings' in captured
+ assert 'labels' not in captured
+ assert 'loss_mask' not in captured
+ assert out[0] is not None
+ assert out[1] is None
+
+ def test_thd_path_raises_when_te_unavailable(self):
+ """THD format must assert when Transformer Engine is not available."""
+ from megatron.core.packed_seq_params import PackedSeqParams
+
+ mock_cp_group = MagicMock()
+ cfg = self._make_cfg(use_cp=True, cp_group=mock_cp_group)
+ adapter = PartitionAdapter(cfg)
+ embeddings = torch.rand(2, 5, 16)
+ packed_seq_params = MagicMock(spec=PackedSeqParams)
+ packed_seq_params.qkv_format = 'thd'
+ with (
+ patch('megatron.core.models.mimo.partition.utils._HAVE_TEX', False),
+ pytest.raises(AssertionError, match="Transformer Engine"),
+ ):
+ adapter._apply_context_parallel(embeddings, None, None, None, packed_seq_params)
diff --git a/tests/unit_tests/models/test_mimo_role.py b/tests/unit_tests/models/test_mimo_role.py
new file mode 100644
index 00000000000..e1ffe218083
--- /dev/null
+++ b/tests/unit_tests/models/test_mimo_role.py
@@ -0,0 +1,47 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+"""Tests for MIMO role data classes."""
+
+import pytest
+
+from megatron.core.models.mimo.config.role import ModuleStageInfo, RankRole
+
+
+class TestMimoRole:
+ """Tests for ModuleStageInfo and RankRole dataclasses."""
+
+ def test_module_stage_info(self):
+ """Test ModuleStageInfo creation and attributes."""
+ first = ModuleStageInfo(is_first_stage=True, is_last_stage=False)
+ last = ModuleStageInfo(is_first_stage=False, is_last_stage=True)
+ only = ModuleStageInfo(is_first_stage=True, is_last_stage=True)
+
+ assert (first.is_first_stage, first.is_last_stage) == (True, False)
+ assert (last.is_first_stage, last.is_last_stage) == (False, True)
+ assert (only.is_first_stage, only.is_last_stage) == (True, True)
+
+ def test_rank_role(self):
+ """Test RankRole properties and methods."""
+ # Encoder-only role
+ encoder_role = RankRole(modules={"vision": ModuleStageInfo(True, False)})
+ assert encoder_role.has_modality_modules is True
+ assert encoder_role.has_language_module is False
+ assert encoder_role.modality_module_names == ["vision"]
+
+ # Language-only role
+ lang_role = RankRole(modules={"language": ModuleStageInfo(True, True)})
+ assert lang_role.has_modality_modules is False
+ assert lang_role.has_language_module is True
+
+ # Mixed role with stage checks
+ mixed = RankRole(
+ modules={
+ "vision": ModuleStageInfo(is_first_stage=True, is_last_stage=False),
+ "language": ModuleStageInfo(is_first_stage=False, is_last_stage=True),
+ }
+ )
+ assert mixed.is_first_stage("vision") is True
+ assert mixed.is_last_stage("vision") is False
+ assert mixed.is_first_stage("language") is False
+ assert mixed.is_last_stage("language") is True
+ assert mixed.is_first_stage("nonexistent") is False
diff --git a/tests/unit_tests/models/test_mimo_submodules.py b/tests/unit_tests/models/test_mimo_submodules.py
index 6111394cc13..5f8de29cc0f 100644
--- a/tests/unit_tests/models/test_mimo_submodules.py
+++ b/tests/unit_tests/models/test_mimo_submodules.py
@@ -303,3 +303,42 @@ def test_empty_data_batch(self):
# Test forward pass
output = self.vision_submodule(data_batch)
assert output is None
+
+
+@pytest.mark.experimental
+class TestVisionSubmoduleStageAware:
+ """Tests for stage-aware forward in VisionModalitySubmodules."""
+
+ def test_stage_aware_forward(self):
+ """Test stage-aware forward: hidden_states input and projection skipping."""
+ hidden_size = 64
+ projection_size = 128
+ hidden_states = torch.randn(10, hidden_size)
+
+ # Default: first and last stage
+ submodule_default = VisionModalitySubmodules(
+ input_projections=[nn.Linear(hidden_size, projection_size)]
+ )
+ assert submodule_default.is_first_stage is True
+ assert submodule_default.is_last_stage is True
+
+ # Non-first stage uses hidden_states, last stage projects
+ submodule_last = VisionModalitySubmodules(
+ input_projections=[nn.Linear(hidden_size, projection_size)],
+ is_first_stage=False,
+ is_last_stage=True,
+ )
+ output = submodule_last.forward(hidden_states=hidden_states)
+ assert output.shape == (10, projection_size) # Projected
+
+ # Non-last stage skips projection
+ submodule_mid = VisionModalitySubmodules(
+ input_projections=[nn.Linear(hidden_size, projection_size)],
+ is_first_stage=False,
+ is_last_stage=False,
+ )
+ output = submodule_mid.forward(hidden_states=hidden_states)
+ assert output.shape == (10, hidden_size) # Not projected
+
+ # No input returns None
+ assert submodule_mid.forward(hidden_states=None) is None
diff --git a/tests/unit_tests/paths.py b/tests/unit_tests/paths.py
new file mode 100644
index 00000000000..122fcc90a51
--- /dev/null
+++ b/tests/unit_tests/paths.py
@@ -0,0 +1,37 @@
+# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.
+# Added for portability across upstream and ROCm CI environments.
+
+"""Shared path resolution for unit tests across CI environments."""
+
+import os
+from pathlib import Path
+
+_UNIT_TESTS_DIR = Path(__file__).resolve().parent
+
+
+def repo_root() -> Path:
+ """Megatron-LM repository root.
+
+ Upstream CI checks out to /opt/megatron-lm; ROCm CI uses /workspace/Megatron-LM.
+ Override with MEGATRON_LM_ROOT when needed.
+ """
+ if root := os.environ.get("MEGATRON_LM_ROOT"):
+ return Path(root)
+ return _UNIT_TESTS_DIR.parents[1]
+
+
+def unit_test_data_dir() -> Path:
+ """Root directory for unit test datasets and tokenizers.
+
+ Upstream and ROCm CI typically mount or populate /opt/data. The session-scoped
+ ensure_test_data fixture downloads assets here when missing. Override with
+ MEGATRON_TEST_DATA_DIR.
+ """
+ if data_dir := os.environ.get("MEGATRON_TEST_DATA_DIR"):
+ return Path(data_dir)
+ return Path("/opt/data")
+
+
+def unit_test_data_path(*parts: str) -> Path:
+ """Return a path under unit_test_data_dir()."""
+ return unit_test_data_dir().joinpath(*parts)
diff --git a/tests/unit_tests/pipeline_parallel/test_bridge_communicator.py b/tests/unit_tests/pipeline_parallel/test_bridge_communicator.py
index 4b426b718eb..326ac8b5890 100644
--- a/tests/unit_tests/pipeline_parallel/test_bridge_communicator.py
+++ b/tests/unit_tests/pipeline_parallel/test_bridge_communicator.py
@@ -1,3 +1,4 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
import logging
import os
import sys
@@ -108,6 +109,9 @@ def _shard_and_copy_(
)
+_active_grids: list = []
+
+
def create_hypercomm_grid(offset=0, tp=1, cp=1, pp=1, dp=1):
"""Create a HyperCommGrid with tensor parallelism=2, context parallelism=2, and data parallelism=2."""
# Set up environment for world size 8 if not already set
@@ -128,9 +132,18 @@ def create_hypercomm_grid(offset=0, tp=1, cp=1, pp=1, dp=1):
_ = grid.create_pg(["cp"])
_ = grid.create_pg(["pp"])
_ = grid.create_pg(["dp"])
+ _active_grids.append(grid)
return grid
+def destroy_all_grids():
+ """Destroy all tracked grids and bridge communicator PGs."""
+ for grid in _active_grids:
+ grid.destroy()
+ _active_grids.clear()
+ BridgeCommunicator.destroy_broadcast_pgs()
+
+
def _get_pg_collection_from_grid(grid):
pg_collection = ProcessGroupCollection()
pg_collection.tp = grid.get_pg("tp")
@@ -200,6 +213,9 @@ def setup_class(cls):
def teardown_class(cls):
Utils.destroy_model_parallel()
+ def teardown_method(self):
+ destroy_all_grids()
+
def test_bridge_communicator_init(self):
grid1 = create_hypercomm_grid(offset=0, tp=2, cp=1, pp=1, dp=2)
@@ -469,3 +485,47 @@ def test_get_leader_rank(self, tp, cp, pp, dp, expected_src_leaders, expected_de
assert (
dest_leaders == expected_dest_leaders
), f"Dest leaders: Expected {expected_dest_leaders}, got {dest_leaders}"
+
+ def test_2d_fan_in_fwd_bwd(self):
+ """Fan-in with 2D tensors: 4 src DP ranks → 1 dest DP group, forward + backward."""
+ src_grid = create_hypercomm_grid(offset=0, tp=1, cp=1, pp=1, dp=4)
+ dest_grid = create_hypercomm_grid(offset=4, tp=4, cp=1, pp=1, dp=1)
+ bridge = BridgeCommunicator(
+ src_grid,
+ dest_grid,
+ dim_mapping={'s': 0, 'h': 2, 'b': 1},
+ comm_dtype=torch.float32,
+ tensor_ndim=2,
+ )
+
+ rank = dist.get_rank()
+ if bridge.is_current_rank_in_grid(src_grid):
+ tensor = torch.full((577, 128), float(rank + 1), device='cuda')
+ grad = bridge.send_forward_recv_backward(tensor)
+ assert grad.shape == (577, 128)
+ else:
+ grad = torch.randn(577 * 4, 128, device='cuda')
+ activation = bridge.send_backward_recv_forward(grad)
+ assert activation.shape == (577 * 4, 128)
+
+ def test_2d_fan_out_fwd_bwd(self):
+ """Fan-out with 2D tensors: 1 src DP group → 4 dest DP ranks, forward + backward."""
+ src_grid = create_hypercomm_grid(offset=0, tp=4, cp=1, pp=1, dp=1)
+ dest_grid = create_hypercomm_grid(offset=4, tp=1, cp=1, pp=1, dp=4)
+ bridge = BridgeCommunicator(
+ src_grid,
+ dest_grid,
+ dim_mapping={'s': 0, 'h': 2, 'b': 1},
+ comm_dtype=torch.float32,
+ tensor_ndim=2,
+ )
+
+ rank = dist.get_rank()
+ if bridge.is_current_rank_in_grid(src_grid):
+ tensor = torch.randn(577 * 4, 128, device='cuda')
+ grad = bridge.send_forward_recv_backward(tensor)
+ assert grad.shape == (577 * 4, 128)
+ else:
+ grad = torch.full((577, 128), float(rank), device='cuda')
+ activation = bridge.send_backward_recv_forward(grad)
+ assert activation.shape == (577, 128)
diff --git a/tests/unit_tests/pipeline_parallel/test_fine_grained_activation_offloading.py b/tests/unit_tests/pipeline_parallel/test_fine_grained_activation_offloading.py
index ae83aa9a595..0937413fb4f 100644
--- a/tests/unit_tests/pipeline_parallel/test_fine_grained_activation_offloading.py
+++ b/tests/unit_tests/pipeline_parallel/test_fine_grained_activation_offloading.py
@@ -73,7 +73,6 @@ def _build_gpt_model(
transformer_layer_spec=get_gpt_layer_with_transformer_engine_spec(
num_experts=num_experts,
moe_grouped_gemm=num_experts is not None,
- moe_use_legacy_grouped_gemm=False,
multi_latent_attention=is_mla,
),
vocab_size=vocab_size,
@@ -133,6 +132,7 @@ def _run_one_iter_and_capture(
return logits.detach().float().cpu(), grads, peak_bytes
+@pytest.mark.flaky_in_dev
@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA is required for offloading tests.")
@pytest.mark.skipif(
not is_te_min_version("2.6.0dev0"),
@@ -309,6 +309,7 @@ def test_gpt_fine_grained_activation_offloading_correctness_and_memory(
Utils.destroy_model_parallel()
+@pytest.mark.flaky_in_dev
@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA is required for offloading tests.")
@pytest.mark.skipif(
not is_te_min_version("2.6.0dev0"),
@@ -440,10 +441,7 @@ def _build_overlap_moe_gpt(
GPTModel(
config=transformer_config,
transformer_layer_spec=get_gpt_layer_with_transformer_engine_spec(
- num_experts=num_experts,
- moe_grouped_gemm=True,
- moe_use_legacy_grouped_gemm=False,
- multi_latent_attention=is_mla,
+ num_experts=num_experts, moe_grouped_gemm=True, multi_latent_attention=is_mla
),
vocab_size=vocab_size,
max_sequence_length=seq_length,
diff --git a/tests/unit_tests/pipeline_parallel/test_multimodule_communicator.py b/tests/unit_tests/pipeline_parallel/test_multimodule_communicator.py
index 22f790cc0a9..e7026c714ea 100644
--- a/tests/unit_tests/pipeline_parallel/test_multimodule_communicator.py
+++ b/tests/unit_tests/pipeline_parallel/test_multimodule_communicator.py
@@ -18,6 +18,7 @@
_create_transformer_block,
_get_pg_collection_from_grid,
create_hypercomm_grid,
+ destroy_all_grids,
get_transformer_block_and_grid,
)
from tests.unit_tests.test_utilities import Utils
@@ -43,6 +44,9 @@ def setup_class(cls):
def teardown_class(cls):
Utils.destroy_model_parallel()
+ def teardown_method(self):
+ destroy_all_grids()
+
def test_multimodule_communicator_init(self):
"""Test MultiModulePipelineCommunicator initialization."""
diff --git a/tests/unit_tests/pipeline_parallel/test_multimodule_schedules.py b/tests/unit_tests/pipeline_parallel/test_multimodule_schedules.py
new file mode 100644
index 00000000000..f9f61e363d8
--- /dev/null
+++ b/tests/unit_tests/pipeline_parallel/test_multimodule_schedules.py
@@ -0,0 +1,551 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+
+"""Tests for multimodule pipeline schedules with heterogeneous parallelism."""
+
+from contextlib import contextmanager
+from typing import Dict, Optional
+
+import pytest
+import torch
+import torch.distributed as dist
+from packaging import version
+
+import megatron.core.pipeline_parallel.schedules as schedule
+from megatron.core import ModelParallelConfig
+from megatron.core.distributed import DistributedDataParallel, DistributedDataParallelConfig
+from megatron.core.distributed.finalize_model_grads import finalize_model_grads
+from megatron.core.hyper_comm_grid import HyperCommGrid
+from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
+from megatron.core.pipeline_parallel.bridge_communicator import BridgeCommunicator
+from megatron.core.pipeline_parallel.multimodule_communicator import MultiModulePipelineCommunicator
+from megatron.core.pipeline_parallel.utils import is_pp_first_stage, is_pp_last_stage
+from megatron.core.process_groups_config import (
+ MultiModuleProcessGroupCollection,
+ ProcessGroupCollection,
+)
+from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
+from megatron.core.transformer.transformer_block import TransformerBlock
+from megatron.core.transformer.transformer_config import TransformerConfig
+from tests.unit_tests.test_utilities import Utils
+
+# ============================================================================
+# Helper Functions
+# ============================================================================
+
+
+_active_grids: list = []
+
+
+def create_hypercomm_grid(offset=0, tp=1, pp=1, dp=1):
+ """Create a HyperCommGrid with specified parallelism."""
+ grid = HyperCommGrid(
+ shape=[tp, 1, pp, dp, 1, 1], # [tp, cp, pp, dp, ep, expt_dp]
+ dim_names=["tp", "cp", "pp", "dp", "ep", "expt_dp"],
+ rank_offset=offset,
+ backend="nccl",
+ )
+ grid.create_pg(["tp"])
+ grid.create_pg(["cp"])
+ grid.create_pg(["pp"])
+ grid.create_pg(["dp"])
+ grid.create_pg(["dp", "cp"])
+ grid.create_pg(["ep"])
+ grid.create_pg(["expt_dp"])
+ _active_grids.append(grid)
+ return grid
+
+
+def destroy_all_grids():
+ """Destroy all tracked grids and bridge communicator PGs."""
+ for grid in _active_grids:
+ grid.destroy()
+ _active_grids.clear()
+ BridgeCommunicator.destroy_broadcast_pgs()
+
+
+def get_pg_collection(grid):
+ """Get ProcessGroupCollection from grid."""
+ pg_collection = ProcessGroupCollection()
+ pg_collection.tp = grid.get_pg("tp")
+ pg_collection.cp = grid.get_pg("cp")
+ pg_collection.pp = grid.get_pg("pp")
+ pg_collection.ep = grid.get_pg("ep")
+ pg_collection.dp = grid.get_pg("dp")
+ pg_collection.dp_cp = grid.get_pg(["dp", "cp"])
+ pg_collection.expt_dp = grid.get_pg("expt_dp")
+ return pg_collection
+
+
+_embedding_pg_cache: dict = {}
+
+
+def add_embedding_groups(pg_collection):
+ """Add embedding groups to process group collection."""
+ if not pg_collection.pp:
+ return pg_collection
+
+ pp_ranks = sorted(dist.get_process_group_ranks(pg_collection.pp))
+ cache_key = tuple(pp_ranks)
+
+ if cache_key not in _embedding_pg_cache:
+ pos_embd_ranks = [pp_ranks[0]]
+ embd_ranks = [pp_ranks[0]]
+ if pp_ranks[-1] != pp_ranks[0]:
+ embd_ranks.append(pp_ranks[-1])
+ _embedding_pg_cache[cache_key] = (
+ dist.new_group(ranks=pos_embd_ranks),
+ dist.new_group(ranks=embd_ranks),
+ )
+
+ pos_embd_pg, embd_pg = _embedding_pg_cache[cache_key]
+
+ # Always set pos_embd and embd (to group or None)
+ pg_collection.pos_embd = pos_embd_pg if is_pp_first_stage(pg_collection.pp) else None
+ pg_collection.embd = (
+ embd_pg
+ if (is_pp_last_stage(pg_collection.pp) or is_pp_first_stage(pg_collection.pp))
+ else None
+ )
+
+ return pg_collection
+
+
+def create_transformer_block(hidden_size, pg_collection, dtype=torch.bfloat16):
+ """Create a transformer block for testing."""
+ torch.manual_seed(12345)
+ model_parallel_cuda_manual_seed(
+ 123,
+ tp_rank=pg_collection.tp.rank(),
+ ep_rank=pg_collection.ep.rank() if hasattr(pg_collection, 'ep') else 0,
+ etp_rank=dist.get_rank(),
+ )
+
+ config = TransformerConfig(
+ num_layers=1,
+ hidden_size=hidden_size,
+ num_attention_heads=8,
+ use_cpu_initialization=True,
+ attention_dropout=0.0,
+ hidden_dropout=0.0,
+ bf16=(dtype == torch.bfloat16),
+ )
+
+ block = (
+ TransformerBlock(
+ config, get_gpt_layer_with_transformer_engine_spec(), pg_collection=pg_collection
+ )
+ .cuda()
+ .to(dtype)
+ )
+
+ with torch.no_grad():
+ for mod in block.modules():
+ if hasattr(mod, "bias") and mod.bias is not None:
+ mod.bias.zero_()
+
+ ddp_config = DistributedDataParallelConfig(overlap_grad_reduce=True, bucket_size=10000)
+ block = DistributedDataParallel(
+ config=block.config, ddp_config=ddp_config, module=block, pg_collection=pg_collection
+ )
+ block.pre_process = False
+ block.post_process = False
+ block.share_embeddings_and_output_weights = False
+ return block
+
+
+def create_module_with_grid(tp, pp, dp, grid_offset, hidden_size):
+ """Create a module (transformer block) with its grid."""
+ rank = dist.get_rank()
+ grid = create_hypercomm_grid(offset=grid_offset, tp=tp, pp=pp, dp=dp)
+
+ if grid.rank_offset <= rank < grid.rank_offset + grid.size:
+ pg_collection = add_embedding_groups(get_pg_collection(grid))
+ module = create_transformer_block(hidden_size, pg_collection)
+ else:
+ pg_collection = None
+ module = None
+
+ return module, grid, pg_collection
+
+
+# ============================================================================
+# Model Wrapper
+# ============================================================================
+
+
+class MultiModuleModel(torch.nn.Module):
+ """Wrapper for testing multimodule schedules with multiple encoders + LLM."""
+
+ def __init__(self, encoder_configs, llm_config, hidden_size):
+ """
+ Args:
+ encoder_configs: List of dicts with keys: tp, pp, dp, grid_offset, name
+ llm_config: Dict with keys: tp, pp, dp, grid_offset
+ hidden_size: Hidden dimension size
+ """
+ super().__init__()
+ self.hidden_size = hidden_size
+ self.rank = dist.get_rank()
+
+ # Create encoders
+ self.encoders = {}
+ self.encoder_grids = {}
+ self.encoder_pg_collections = {}
+ for enc_cfg in encoder_configs:
+ name = enc_cfg['name']
+ module, grid, pg_col = create_module_with_grid(
+ enc_cfg['tp'], enc_cfg['pp'], enc_cfg['dp'], enc_cfg['grid_offset'], hidden_size
+ )
+ self.encoders[name] = module
+ self.encoder_grids[name] = grid
+ self.encoder_pg_collections[name] = pg_col
+
+ # Create LLM
+ self.llm, self.llm_grid, self.llm_pg_collection = create_module_with_grid(
+ llm_config['tp'],
+ llm_config['pp'],
+ llm_config['dp'],
+ llm_config['grid_offset'],
+ hidden_size,
+ )
+
+ # Track all modules for gradient sync
+ self.modules_and_grids = []
+ for name, module in self.encoders.items():
+ self.modules_and_grids.append(
+ (module, self.encoder_grids[name], self.encoder_pg_collections[name])
+ )
+ self.modules_and_grids.append((self.llm, self.llm_grid, self.llm_pg_collection))
+
+ # Input tensors for pipeline stages
+ self.input_tensors = {name: None for name in self.encoders.keys()}
+ self.input_tensors['llm'] = None
+
+ def is_rank_in_grid(self, grid):
+ """Check if current rank is in grid."""
+ return grid.rank_offset <= self.rank < grid.rank_offset + grid.size
+
+ @contextmanager
+ def no_sync(self):
+ """No-sync context for all active modules."""
+ contexts = []
+ for module, grid, _pg in self.modules_and_grids:
+ if module is not None and self.is_rank_in_grid(grid):
+ contexts.append(module.no_sync())
+
+ for ctx in contexts:
+ ctx.__enter__()
+ try:
+ yield
+ finally:
+ for ctx in reversed(contexts):
+ ctx.__exit__(None, None, None)
+
+ @property
+ def ddp_config(self):
+ """Get DDP config from first active module."""
+ for module, grid, _pg in self.modules_and_grids:
+ if module is not None and self.is_rank_in_grid(grid):
+ return module.ddp_config
+ raise AttributeError(f"No active modules on rank {self.rank}")
+
+ def finalize_model_grads(self, *args, **kwargs):
+ """Finalize gradients for all active modules."""
+ for module, grid, pg_collection in self.modules_and_grids:
+ if module is not None and self.is_rank_in_grid(grid):
+ finalize_model_grads([module], num_tokens=None, pg_collection=pg_collection)
+
+ def set_input_tensor(self, input_tensor):
+ """Set input tensors from previous pipeline stage."""
+ if not input_tensor or not input_tensor[0]:
+ return
+
+ tensor_dict = input_tensor[0]
+
+ # Set encoder inputs
+ for name in self.encoders.keys():
+ if name in tensor_dict:
+ self.input_tensors[name] = (
+ tensor_dict[name][0]
+ if isinstance(tensor_dict[name], list)
+ else tensor_dict[name]
+ )
+
+ # Set LLM input (from either encoder outputs or previous LLM stage)
+ # Only do this if we're on the LLM grid
+ if self.is_rank_in_grid(self.llm_grid):
+ if 'llm' in tensor_dict:
+ self.input_tensors['llm'] = (
+ tensor_dict['llm'][0]
+ if isinstance(tensor_dict['llm'], list)
+ else tensor_dict['llm']
+ )
+ elif len(self.encoders) > 0:
+ # Concatenate encoder outputs for LLM input (received via bridge)
+ encoder_outputs = []
+ for name in self.encoders.keys():
+ if name in tensor_dict:
+ tensor = tensor_dict[name]
+ # Extract tensor from list if needed (P2P sends as list)
+ if isinstance(tensor, list):
+ tensor = tensor[0]
+ encoder_outputs.append(tensor)
+ if encoder_outputs:
+ self.input_tensors['llm'] = (
+ torch.cat(encoder_outputs, dim=0)
+ if len(encoder_outputs) > 1
+ else encoder_outputs[0]
+ )
+
+ def forward(self, hidden_states):
+ """Forward pass through active modules."""
+ output_dict = {}
+
+ # Forward through encoders
+ for name, encoder in self.encoders.items():
+ if encoder is not None and self.is_rank_in_grid(self.encoder_grids[name]):
+ pp_group = self.encoder_grids[name].get_pg("pp")
+ input_tensor = (
+ hidden_states if is_pp_first_stage(pp_group) else self.input_tensors[name]
+ )
+ output_dict[name] = encoder(input_tensor, attention_mask=None)
+
+ # Forward through LLM
+ if self.llm is not None and self.is_rank_in_grid(self.llm_grid):
+ output_dict['llm'] = self.llm(self.input_tensors['llm'], attention_mask=None)
+
+ return output_dict
+
+
+# ============================================================================
+# Data Iterator
+# ============================================================================
+
+
+class DataIterator:
+ """Simple data iterator for testing."""
+
+ def __init__(self, hidden_size, seq_length, micro_batch_size):
+ self.hidden_size = hidden_size
+ self.seq_length = seq_length
+ self.micro_batch_size = micro_batch_size
+
+ def __iter__(self):
+ return self
+
+ def __next__(self):
+ return torch.randn(
+ self.seq_length,
+ self.micro_batch_size,
+ self.hidden_size,
+ device='cuda',
+ dtype=torch.bfloat16,
+ )
+
+
+# ============================================================================
+# Test Runner
+# ============================================================================
+
+
+def run_multimodule_schedule_test(
+ encoder_configs, llm_config, hidden_size, seq_length, micro_batch_size, num_microbatches
+):
+ """Run multimodule schedule test with given configuration.
+
+ Args:
+ encoder_configs: List of encoder configs
+ llm_config: LLM config dict
+ hidden_size: Hidden dimension
+ seq_length: Sequence length
+ micro_batch_size: Micro batch size
+ num_microbatches: Number of microbatches
+ """
+ # Create model
+ model = MultiModuleModel(encoder_configs, llm_config, hidden_size)
+ model.model_type = 'unit-test'
+
+ # Build module_to_grid_map and topology
+ module_to_grid_map = {name: grid for name, grid in model.encoder_grids.items()}
+ module_to_grid_map['llm'] = model.llm_grid
+
+ topology = {name: ['llm'] for name in model.encoders.keys()}
+ topology['llm'] = []
+
+ # Configure
+ config = ModelParallelConfig(pipeline_dtype=torch.bfloat16)
+ config.variable_seq_lengths = True
+ config.calculate_per_token_loss = False
+ config.fine_grained_activation_offloading = False
+ config.qk_layernorm = False
+ config.sequence_parallel = False
+ config.moe_router_enable_expert_bias = False
+ config.moe_router_load_balancing_type = "aux_loss"
+ config.no_sync_func = model.no_sync
+ config.finalize_model_grads_func = model.finalize_model_grads
+ config.grad_scale_func = lambda loss: (
+ torch.tensor(loss, dtype=torch.float32, device='cuda', requires_grad=True)
+ if isinstance(loss, (int, float))
+ else loss
+ )
+ config.hidden_size = hidden_size
+ model.config = config
+
+ # Create communicator
+ communicator = MultiModulePipelineCommunicator(
+ module_to_grid_map, topology, config, dim_mapping={'s': 0, 'h': 2, 'b': 1}
+ )
+
+ # Create data iterator (only on first encoder's first stage)
+ data_iterator = None
+ first_encoder_name = encoder_configs[0]['name']
+ first_encoder_grid = model.encoder_grids[first_encoder_name]
+ if model.is_rank_in_grid(first_encoder_grid):
+ if is_pp_first_stage(first_encoder_grid.get_pg("pp")):
+ data_iterator = DataIterator(hidden_size, seq_length, micro_batch_size)
+
+ # Build MultiModuleProcessGroupCollection for current rank
+ # Reuse pg_collections already created during model init (avoid creating duplicate PGs)
+ rank = dist.get_rank()
+ module_pgs = {}
+ language_model_module_name = None
+ for name, grid in model.encoder_grids.items():
+ if grid.rank_offset <= rank < grid.rank_offset + grid.size:
+ module_pgs[name] = model.encoder_pg_collections[name]
+ if model.llm_grid.rank_offset <= rank < model.llm_grid.rank_offset + model.llm_grid.size:
+ module_pgs['llm'] = model.llm_pg_collection
+ language_model_module_name = 'llm'
+
+ pg_collection = MultiModuleProcessGroupCollection(
+ module_pgs=module_pgs, language_model_module_name=language_model_module_name
+ )
+
+ # Define step function
+ def step_func(data_iterator, model):
+ def loss_func(output_tensor_dict: Dict[str, torch.Tensor]):
+ assert 'llm' in output_tensor_dict, f"Expected 'llm' in output"
+ loss = output_tensor_dict['llm'].sum()
+ return loss, {'loss_reduced': loss}
+
+ input_tensor = next(data_iterator) if data_iterator is not None else None
+ model_output = model(input_tensor)
+ return model_output, loss_func
+
+ # Run schedule
+ losses = schedule.forward_backward_pipelining_without_interleaving(
+ forward_step_func=step_func,
+ data_iterator=data_iterator,
+ model=[model],
+ num_microbatches=num_microbatches,
+ seq_length=seq_length,
+ micro_batch_size=micro_batch_size,
+ forward_only=False,
+ p2p_communicator=communicator,
+ pg_collection=pg_collection,
+ )
+
+ # Verify results on last LLM stage
+ if model.is_rank_in_grid(model.llm_grid):
+ if is_pp_last_stage(model.llm_grid.get_pg("pp")):
+ assert len(losses) > 0, "Expected losses on last LLM stage"
+
+ return losses
+
+
+# ============================================================================
+# Tests
+# ============================================================================
+
+
+@pytest.mark.skipif(
+ version.parse(torch.__version__) < version.parse('2.3.0'),
+ reason="Device mesh requires PyTorch 2.3+",
+)
+class TestMultimoduleSchedules:
+ """Test multimodule pipeline schedules."""
+
+ @classmethod
+ def setup_class(cls):
+ Utils.initialize_distributed()
+ cls.world_size = dist.get_world_size()
+
+ @classmethod
+ def teardown_class(cls):
+ Utils.destroy_model_parallel()
+
+ def teardown_method(self):
+ destroy_all_grids()
+
+ def test_single_encoder_2gpu(self):
+ """Test single encoder + LLM on 2 GPUs (no PP)."""
+ if self.world_size != 2:
+ pytest.skip(f"Requires 2 GPUs, got {self.world_size}")
+
+ encoder_configs = [{'name': 'encoder', 'tp': 1, 'pp': 1, 'dp': 1, 'grid_offset': 0}]
+ llm_config = {'tp': 1, 'pp': 1, 'dp': 1, 'grid_offset': 1}
+
+ run_multimodule_schedule_test(
+ encoder_configs,
+ llm_config,
+ hidden_size=512,
+ seq_length=64,
+ micro_batch_size=2,
+ num_microbatches=4,
+ )
+
+ def test_dual_encoder_2gpu(self):
+ """Test dual encoder + LLM on 2 GPUs (both encoders on rank 0)."""
+ if self.world_size != 2:
+ pytest.skip(f"Requires 2 GPUs, got {self.world_size}")
+
+ encoder_configs = [
+ {'name': 'encoder_1', 'tp': 1, 'pp': 1, 'dp': 1, 'grid_offset': 0},
+ {'name': 'encoder_2', 'tp': 1, 'pp': 1, 'dp': 1, 'grid_offset': 0},
+ ]
+ llm_config = {'tp': 1, 'pp': 1, 'dp': 1, 'grid_offset': 1}
+
+ run_multimodule_schedule_test(
+ encoder_configs,
+ llm_config,
+ hidden_size=512,
+ seq_length=64,
+ micro_batch_size=2,
+ num_microbatches=4,
+ )
+
+ def test_single_encoder_8gpu(self):
+ """Test single encoder + LLM on 8 GPUs (TP=2, PP=2 each)."""
+ if self.world_size != 8:
+ pytest.skip(f"Requires 8 GPUs, got {self.world_size}")
+
+ encoder_configs = [{'name': 'encoder', 'tp': 2, 'pp': 2, 'dp': 1, 'grid_offset': 0}]
+ llm_config = {'tp': 2, 'pp': 2, 'dp': 1, 'grid_offset': 4}
+
+ run_multimodule_schedule_test(
+ encoder_configs,
+ llm_config,
+ hidden_size=1024,
+ seq_length=512,
+ micro_batch_size=4,
+ num_microbatches=16,
+ )
+
+ def test_dual_encoder_8gpu(self):
+ """Test dual encoder + LLM on 8 GPUs (TP=2, PP=2 for each)."""
+ if self.world_size != 8:
+ pytest.skip(f"Requires 8 GPUs, got {self.world_size}")
+
+ encoder_configs = [
+ {'name': 'encoder_1', 'tp': 2, 'pp': 2, 'dp': 1, 'grid_offset': 0},
+ {'name': 'encoder_2', 'tp': 2, 'pp': 2, 'dp': 1, 'grid_offset': 0},
+ ]
+ llm_config = {'tp': 2, 'pp': 2, 'dp': 1, 'grid_offset': 4}
+
+ run_multimodule_schedule_test(
+ encoder_configs,
+ llm_config,
+ hidden_size=1024,
+ seq_length=512,
+ micro_batch_size=4,
+ num_microbatches=16,
+ )
diff --git a/tests/unit_tests/pipeline_parallel/test_pipeline_layout.py b/tests/unit_tests/pipeline_parallel/test_pipeline_layout.py
index a3990d25ecf..e7fb8159af2 100644
--- a/tests/unit_tests/pipeline_parallel/test_pipeline_layout.py
+++ b/tests/unit_tests/pipeline_parallel/test_pipeline_layout.py
@@ -134,7 +134,7 @@ def create_args():
args.consumed_valid_samples = 0
args.vocab_file = None
args.add_position_embedding = False
- args.ckpt_assume_constant_structure = True
+ args.ckpt_assume_constant_structure = False
args.dist_ckpt_strictness = "assume_ok_unexpected"
args.fp16 = False
args.bf16 = True
@@ -147,6 +147,7 @@ def create_args():
args.dist_ckpt_optim_fully_reshardable = False
args.distrib_optim_fully_reshardable_mem_efficient = False
args.phase_transition_iterations = None
+ args.async_strategy = "nvrx"
yield args
diff --git a/tests/unit_tests/post_training/test_modelopt_module_spec.py b/tests/unit_tests/post_training/test_modelopt_module_spec.py
index dac96785bc0..585be52f944 100644
--- a/tests/unit_tests/post_training/test_modelopt_module_spec.py
+++ b/tests/unit_tests/post_training/test_modelopt_module_spec.py
@@ -212,7 +212,7 @@ def setup_method(self, method):
mamba_stack_spec=mamba_stack_spec,
vocab_size=100,
max_sequence_length=4,
- hybrid_override_pattern="M*-",
+ hybrid_layer_pattern="M*-",
)
# A Hybrid MambaModel using ModelOpt spec (local + TENorm).
@@ -221,7 +221,7 @@ def setup_method(self, method):
mamba_stack_spec=get_mamba_stack_modelopt_spec(remap_te_layernorm=True),
vocab_size=100,
max_sequence_length=4,
- hybrid_override_pattern="M*-",
+ hybrid_layer_pattern="M*-",
)
@@ -272,9 +272,14 @@ def test_get_mamba_stack_modelopt_spec_interface():
expected_params = {
"local_core_attention": inspect.Parameter.POSITIONAL_OR_KEYWORD,
"remap_te_layernorm": inspect.Parameter.POSITIONAL_OR_KEYWORD,
+ "use_default_te_spec": inspect.Parameter.POSITIONAL_OR_KEYWORD,
}
- expected_defaults = {"local_core_attention": False, "remap_te_layernorm": False}
+ expected_defaults = {
+ "local_core_attention": False,
+ "remap_te_layernorm": False,
+ "use_default_te_spec": False,
+ }
# Check expected parameters are in function signature
for param_name, param_kind in expected_params.items():
@@ -291,3 +296,9 @@ def test_get_mamba_stack_modelopt_spec_interface():
assert (
k in sig_defaults and v == sig_defaults[k]
), f"Default value of {sig_defaults[k]} does not match the expected value of {v} for parameter {k}."
+
+
+def test_get_mamba_stack_modelopt_spec_use_default_te_spec():
+ """Test that use_default_te_spec=True returns the standard mamba_stack_spec."""
+ spec = get_mamba_stack_modelopt_spec(use_default_te_spec=True)
+ assert spec is mamba_stack_spec
diff --git a/tests/unit_tests/resharding/test_communication_scheduler.py b/tests/unit_tests/resharding/test_communication_scheduler.py
new file mode 100644
index 00000000000..15d25236740
--- /dev/null
+++ b/tests/unit_tests/resharding/test_communication_scheduler.py
@@ -0,0 +1,210 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+import pytest
+
+from megatron.core.resharding.nvshmem_copy_service.nvshmem_types import ScheduledBatch
+from megatron.core.resharding.nvshmem_copy_service.planning.communication_scheduler import (
+ CommunicationScheduler,
+)
+
+
+class TestCommunicationScheduler:
+ """Test suite for CommunicationScheduler."""
+
+ def test_conflict_detection(self):
+ """Test that conflicts are detected correctly."""
+ scheduler = CommunicationScheduler()
+
+ # Create test batches
+ batch1 = ScheduledBatch(src_pe=0, dest_pe=1, batch_index=0, iteration=-1, total_size=1000)
+ batch2 = ScheduledBatch(src_pe=0, dest_pe=2, batch_index=0, iteration=-1, total_size=1000)
+ batch3 = ScheduledBatch(src_pe=2, dest_pe=3, batch_index=0, iteration=-1, total_size=1000)
+ batch4 = ScheduledBatch(src_pe=4, dest_pe=5, batch_index=0, iteration=-1, total_size=1000)
+
+ # batch1 and batch2 conflict (share src PE 0)
+ # batch2 and batch3 conflict (share PE 2)
+ # batch4 doesn't conflict with any
+
+ batches = [batch1, batch2, batch3, batch4]
+ scheduler._assign_iterations(batches)
+
+ # batch1 and batch2 should be in different iterations (conflict on PE 0)
+ assert batch1.iteration != batch2.iteration
+
+ # batch2 and batch3 should be in different iterations (conflict on PE 2)
+ assert batch2.iteration != batch3.iteration
+
+ # batch4 can be in any iteration (no conflicts)
+ # Since we process by degree, batch4 (degree 0) will be placed first
+ assert batch4.iteration == 0
+
+ def test_degree_based_sorting(self):
+ """Test that batches are sorted by conflict degree."""
+ scheduler = CommunicationScheduler()
+
+ # Create batches with different conflict patterns
+ # Central hub pattern: PE 0 connects to many others
+ batches = [
+ ScheduledBatch(src_pe=0, dest_pe=1, batch_index=0, iteration=-1, total_size=1000),
+ ScheduledBatch(src_pe=0, dest_pe=2, batch_index=0, iteration=-1, total_size=1000),
+ ScheduledBatch(src_pe=0, dest_pe=3, batch_index=0, iteration=-1, total_size=1000),
+ ScheduledBatch(
+ src_pe=4, dest_pe=5, batch_index=0, iteration=-1, total_size=1000
+ ), # isolated
+ ]
+
+ scheduler._assign_iterations(batches)
+
+ # Batches involving PE 0 should be scheduled in different iterations
+ pe0_batches = [b for b in batches if b.src_pe == 0 or b.dest_pe == 0]
+ iterations = [b.iteration for b in pe0_batches]
+ # All PE 0 batches should be in different iterations
+ assert len(iterations) == len(set(iterations))
+
+ # Isolated batch should be in iteration 0 (no conflicts)
+ isolated = [b for b in batches if b.src_pe == 4][0]
+ assert isolated.iteration == 0
+
+ def test_ring_pattern(self):
+ """Test scheduling efficiency for ring communication pattern."""
+ scheduler = CommunicationScheduler()
+
+ n_pes = 8
+ # Ring pattern: each PE sends to next PE (0→1, 1→2, 2→3, ...)
+ batches = [
+ ScheduledBatch(
+ src_pe=i, dest_pe=(i + 1) % n_pes, batch_index=0, iteration=-1, total_size=1000
+ )
+ for i in range(n_pes)
+ ]
+
+ scheduler._assign_iterations(batches)
+
+ # Ring pattern needs 2 iterations because a PE can't send and receive simultaneously
+ # Iteration 0: Even-indexed PEs send (0→1, 2→3, 4→5, 6→7)
+ # Iteration 1: Odd-indexed PEs send (1→2, 3→4, 5→6, 7→0)
+ iterations_used = len(set(b.iteration for b in batches))
+ assert iterations_used == 2, f"Ring should use 2 iterations, got {iterations_used}"
+
+ # Verify no conflicts within each iteration
+ for iteration in range(iterations_used):
+ iter_batches = [b for b in batches if b.iteration == iteration]
+ used_pes = set()
+ for batch in iter_batches:
+ assert batch.src_pe not in used_pes
+ assert batch.dest_pe not in used_pes
+ used_pes.add(batch.src_pe)
+ used_pes.add(batch.dest_pe)
+
+ def test_all_to_all_pattern(self):
+ """Test scheduling for all-to-all communication."""
+ scheduler = CommunicationScheduler()
+
+ n_pes = 4
+ # All-to-all: every PE sends to every other PE
+ batches = []
+ for src in range(n_pes):
+ for dst in range(n_pes):
+ if src != dst:
+ batches.append(
+ ScheduledBatch(
+ src_pe=src, dest_pe=dst, batch_index=0, iteration=-1, total_size=1000
+ )
+ )
+
+ scheduler._assign_iterations(batches)
+
+ # Verify schedule is conflict-free
+ for iteration in range(scheduler.num_iterations):
+ iter_batches = [b for b in batches if b.iteration == iteration]
+ used_pes = set()
+ for batch in iter_batches:
+ # No PE should be used twice in same iteration
+ assert (
+ batch.src_pe not in used_pes
+ ), f"PE {batch.src_pe} used twice in iteration {iteration}"
+ assert (
+ batch.dest_pe not in used_pes
+ ), f"PE {batch.dest_pe} used twice in iteration {iteration}"
+ used_pes.add(batch.src_pe)
+ used_pes.add(batch.dest_pe)
+
+ def test_empty_workloads(self):
+ """Test handling of empty workloads."""
+ scheduler = CommunicationScheduler()
+ batches = []
+ scheduler._assign_iterations(batches)
+ assert scheduler.num_iterations == 0
+
+ def test_single_batch(self):
+ """Test scheduling with a single batch."""
+ scheduler = CommunicationScheduler()
+ batches = [
+ ScheduledBatch(src_pe=0, dest_pe=1, batch_index=0, iteration=-1, total_size=1000)
+ ]
+ scheduler._assign_iterations(batches)
+ assert scheduler.num_iterations == 1
+ assert batches[0].iteration == 0
+
+ def test_no_self_conflict(self):
+ """Test that a batch doesn't conflict with itself."""
+ scheduler = CommunicationScheduler()
+
+ # Single batch should be scheduled in iteration 0
+ batches = [
+ ScheduledBatch(src_pe=0, dest_pe=1, batch_index=0, iteration=-1, total_size=1000)
+ ]
+ scheduler._assign_iterations(batches)
+ assert batches[0].iteration == 0
+
+ def test_scatter_pattern(self):
+ """Test one-to-many scatter pattern."""
+ scheduler = CommunicationScheduler()
+
+ n_receivers = 7
+ # One PE sends to all others
+ batches = [
+ ScheduledBatch(src_pe=0, dest_pe=i + 1, batch_index=0, iteration=-1, total_size=1000)
+ for i in range(n_receivers)
+ ]
+
+ scheduler._assign_iterations(batches)
+
+ # All batches involve PE 0 as sender, so must be in different iterations
+ iterations_used = len(set(b.iteration for b in batches))
+ assert iterations_used == n_receivers
+
+ def test_gather_pattern(self):
+ """Test many-to-one gather pattern."""
+ scheduler = CommunicationScheduler()
+
+ n_senders = 7
+ # All PEs send to one PE
+ batches = [
+ ScheduledBatch(src_pe=i, dest_pe=7, batch_index=0, iteration=-1, total_size=1000)
+ for i in range(n_senders)
+ ]
+
+ scheduler._assign_iterations(batches)
+
+ # All batches involve PE 7 as receiver, so must be in different iterations
+ iterations_used = len(set(b.iteration for b in batches))
+ assert iterations_used == n_senders
+
+ def test_large_batch_priority(self):
+ """Test that larger batches get priority (tie-breaking by size)."""
+ scheduler = CommunicationScheduler()
+
+ # Create batches with different sizes
+ small_batch = ScheduledBatch(
+ src_pe=0, dest_pe=1, batch_index=0, iteration=-1, total_size=100
+ )
+ large_batch = ScheduledBatch(
+ src_pe=2, dest_pe=3, batch_index=0, iteration=-1, total_size=10000
+ )
+
+ batches = [small_batch, large_batch]
+ scheduler._assign_iterations(batches)
+
+ # Both should be in iteration 0 (no conflicts)
+ assert small_batch.iteration == 0
+ assert large_batch.iteration == 0
diff --git a/tests/unit_tests/resharding/test_dp_balancing.py b/tests/unit_tests/resharding/test_dp_balancing.py
new file mode 100644
index 00000000000..f1f8035171a
--- /dev/null
+++ b/tests/unit_tests/resharding/test_dp_balancing.py
@@ -0,0 +1,344 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+import pytest
+
+from megatron.core.resharding.utils import ParameterMetadata, select_src_metadata_balanced
+
+
+class TestDPBalancing:
+ """Test suite for DP load balancing."""
+
+ def _create_metadata(self, rank, tp_group, dp_group, ep_group=None):
+ """Helper to create ParameterMetadata for testing."""
+ return ParameterMetadata(
+ name="test.weight",
+ shape=(128, 256),
+ dtype="float32",
+ element_size=4, # 4 bytes for float32
+ owner_rank=rank,
+ tensor_parallel_group_ranks=tp_group,
+ data_parallel_group_ranks=dp_group,
+ expert_parallel_group_ranks=ep_group,
+ is_tp=True,
+ partition_dim=0,
+ )
+
+ def test_dp_balancing_basic(self):
+ """Test basic DP balancing with 2 DP groups."""
+ # Setup: TP=2, DP=2, World=4
+ # TP groups: [[0,1], [2,3]]
+ # DP groups: [[0,2], [1,3]] (TP-local-0 and TP-local-1)
+
+ # Source metadata from all ranks
+ src_meta_list = [
+ self._create_metadata(rank=0, tp_group=[0, 1], dp_group=[0, 2]),
+ self._create_metadata(rank=1, tp_group=[0, 1], dp_group=[1, 3]),
+ self._create_metadata(rank=2, tp_group=[2, 3], dp_group=[0, 2]),
+ self._create_metadata(rank=3, tp_group=[2, 3], dp_group=[1, 3]),
+ ]
+
+ # Destination metadata (TP=1, DP=4)
+ dst_meta = self._create_metadata(rank=0, tp_group=[0], dp_group=[0, 1, 2, 3])
+
+ # Test each destination rank's selection
+ selections = {}
+ for dst_rank in range(4):
+ selected = select_src_metadata_balanced(src_meta_list, dst_meta, dst_rank)
+ selections[dst_rank] = (selected.owner_rank, tuple(selected.data_parallel_group_ranks))
+
+ # Verify round-robin across DP groups
+ # dst_rank 0: 0 % 2 = 0 -> should select DP group [0,2]
+ # dst_rank 1: 1 % 2 = 1 -> should select DP group [1,3]
+ # dst_rank 2: 2 % 2 = 0 -> should select DP group [0,2]
+ # dst_rank 3: 3 % 2 = 1 -> should select DP group [1,3]
+ assert selections[0][1] in [(0, 2), (1, 3)] # DP group 0 or 1
+ assert selections[1][1] in [(0, 2), (1, 3)]
+ assert selections[2][1] == selections[0][1] # Same as rank 0
+ assert selections[3][1] == selections[1][1] # Same as rank 1
+
+ # Verify different DP groups selected
+ assert selections[0][1] != selections[1][1]
+
+ def test_dp_balancing_non_collocated(self):
+ """Test DP balancing in non-collocated mode (dst ranks not in source ranks)."""
+ # Setup: TP=2, DP=2, World=4 (non-collocated, same config on both sides)
+
+ src_meta_list = [
+ self._create_metadata(rank=0, tp_group=[0, 1], dp_group=[0, 2]),
+ self._create_metadata(rank=1, tp_group=[0, 1], dp_group=[1, 3]),
+ self._create_metadata(rank=2, tp_group=[2, 3], dp_group=[0, 2]),
+ self._create_metadata(rank=3, tp_group=[2, 3], dp_group=[1, 3]),
+ ]
+
+ # Destination with TP=2 (same as source), dst rank not in source ranks
+ dst_meta = self._create_metadata(rank=4, tp_group=[4, 5], dp_group=[4, 5])
+
+ # Should select via DP balancing (no local copy available)
+ selected = select_src_metadata_balanced(src_meta_list, dst_meta, dst_rank=4)
+
+ # dst_rank=4, 4 % 2 = 0 -> selects from first sorted DP group
+ assert selected.owner_rank in [0, 1, 2, 3]
+
+ def test_dp_balancing_distribution(self):
+ """Test that many destination ranks are evenly distributed across source DP groups."""
+ # Setup: TP=2, DP=4, World=8
+ # TP groups: [[0,1], [2,3], [4,5], [6,7]]
+ # DP groups: [[0,2,4,6], [1,3,5,7]]
+
+ src_meta_list = [
+ self._create_metadata(rank=0, tp_group=[0, 1], dp_group=[0, 2, 4, 6]),
+ self._create_metadata(rank=1, tp_group=[0, 1], dp_group=[1, 3, 5, 7]),
+ self._create_metadata(rank=2, tp_group=[2, 3], dp_group=[0, 2, 4, 6]),
+ self._create_metadata(rank=3, tp_group=[2, 3], dp_group=[1, 3, 5, 7]),
+ self._create_metadata(rank=4, tp_group=[4, 5], dp_group=[0, 2, 4, 6]),
+ self._create_metadata(rank=5, tp_group=[4, 5], dp_group=[1, 3, 5, 7]),
+ self._create_metadata(rank=6, tp_group=[6, 7], dp_group=[0, 2, 4, 6]),
+ self._create_metadata(rank=7, tp_group=[6, 7], dp_group=[1, 3, 5, 7]),
+ ]
+
+ dst_meta = self._create_metadata(rank=0, tp_group=[0], dp_group=list(range(8)))
+
+ # Count selections per DP group
+ dp_group_counts = {}
+ for dst_rank in range(16): # Test with more dst ranks than src
+ selected = select_src_metadata_balanced(src_meta_list, dst_meta, dst_rank)
+ dp_group = tuple(selected.data_parallel_group_ranks)
+ dp_group_counts[dp_group] = dp_group_counts.get(dp_group, 0) + 1
+
+ # Should have exactly 2 DP groups
+ assert len(dp_group_counts) == 2
+
+ # Each should be selected 8 times (16 ranks / 2 groups = 8)
+ assert all(count == 8 for count in dp_group_counts.values())
+
+ def test_dp_balancing_with_ep(self):
+ """Test DP balancing with expert parallelism."""
+ # Setup: TP=2, EP=2, DP=2, World=8
+ # When EP sizes match, should filter by EP local rank
+ #
+ # EP local rank is computed from ep_group.index(owner_rank):
+ # rank=0 in ep_group=[0, 2] -> EP local 0
+ # rank=2 in ep_group=[0, 2] -> EP local 1
+ # rank=4 in ep_group=[4, 6] -> EP local 0
+ # rank=6 in ep_group=[4, 6] -> EP local 1
+
+ src_meta_list = [
+ # EP local 0 (rank 0 is at index 0 in ep_group [0, 2])
+ self._create_metadata(rank=0, tp_group=[0, 1], dp_group=[0, 4], ep_group=[0, 2]),
+ # EP local 1 (rank 2 is at index 1 in ep_group [0, 2])
+ self._create_metadata(rank=2, tp_group=[2, 3], dp_group=[2, 6], ep_group=[0, 2]),
+ # EP local 0 (rank 4 is at index 0 in ep_group [4, 6])
+ self._create_metadata(rank=4, tp_group=[4, 5], dp_group=[0, 4], ep_group=[4, 6]),
+ # EP local 1 (rank 6 is at index 1 in ep_group [4, 6])
+ self._create_metadata(rank=6, tp_group=[6, 7], dp_group=[2, 6], ep_group=[4, 6]),
+ ]
+
+ # Destination with same EP size=2, EP local rank = 0
+ # (rank 8 is at index 0 in ep_group [8, 9])
+ dst_meta = self._create_metadata(rank=8, tp_group=[8, 9], dp_group=[8, 9], ep_group=[8, 9])
+
+ # When EP sizes match (2->2), should filter by EP local rank
+ selected = select_src_metadata_balanced(src_meta_list, dst_meta, dst_rank=8)
+
+ # Should select from EP-local-0 ranks only (0 or 4)
+ ep_local = selected.expert_parallel_group_ranks.index(selected.owner_rank)
+ assert ep_local == 0
+ assert selected.owner_rank in [0, 4]
+
+ def test_dp_balancing_single_dp_group(self):
+ """Test fast path when only one DP group exists."""
+ # Setup: TP=2, DP=1, World=2 (single DP group)
+
+ src_meta_list = [
+ self._create_metadata(rank=0, tp_group=[0, 1], dp_group=[0, 1]),
+ self._create_metadata(rank=1, tp_group=[0, 1], dp_group=[0, 1]),
+ ]
+
+ dst_meta = self._create_metadata(rank=0, tp_group=[0], dp_group=[0])
+
+ # Should hit fast path and return first metadata
+ selected = select_src_metadata_balanced(src_meta_list, dst_meta, dst_rank=0)
+
+ # Fast path returns first entry (after any TP/EP filtering)
+ assert selected == src_meta_list[0]
+
+ def test_tp_size_mismatch_no_filter(self):
+ """Test DP balancing when TP sizes differ (resharding mode)."""
+ # Setup: TP=4 -> TP=2 (resharding)
+
+ src_meta_list = [
+ self._create_metadata(rank=0, tp_group=[0, 1, 2, 3], dp_group=[0, 4]),
+ self._create_metadata(rank=1, tp_group=[0, 1, 2, 3], dp_group=[1, 5]),
+ self._create_metadata(rank=2, tp_group=[0, 1, 2, 3], dp_group=[2, 6]),
+ self._create_metadata(rank=3, tp_group=[0, 1, 2, 3], dp_group=[3, 7]),
+ self._create_metadata(rank=4, tp_group=[4, 5, 6, 7], dp_group=[0, 4]),
+ self._create_metadata(rank=5, tp_group=[4, 5, 6, 7], dp_group=[1, 5]),
+ self._create_metadata(rank=6, tp_group=[4, 5, 6, 7], dp_group=[2, 6]),
+ self._create_metadata(rank=7, tp_group=[4, 5, 6, 7], dp_group=[3, 7]),
+ ]
+
+ # Destination with TP=2 (different from source TP=4)
+ dst_meta = self._create_metadata(rank=8, tp_group=[8, 9], dp_group=[8, 9])
+
+ # Should only do DP balancing (no TP filtering)
+ selected = select_src_metadata_balanced(src_meta_list, dst_meta, dst_rank=8)
+
+ # Since dst_rank % 4 = 0, should select DP group 0
+ # Could be any TP local rank (not filtered)
+ assert selected.owner_rank in [0, 4] # Both have DP group [0,4]
+
+ def test_ep_size_mismatch_no_filter(self):
+ """Test that EP filtering is skipped when EP sizes differ."""
+ # Setup: EP=4 -> EP=8 (expert parallel resharding)
+
+ src_meta_list = [
+ self._create_metadata(rank=0, tp_group=[0], dp_group=[0, 4], ep_group=[0, 1, 2, 3]),
+ self._create_metadata(rank=1, tp_group=[1], dp_group=[1, 5], ep_group=[0, 1, 2, 3]),
+ self._create_metadata(rank=2, tp_group=[2], dp_group=[2, 6], ep_group=[0, 1, 2, 3]),
+ self._create_metadata(rank=3, tp_group=[3], dp_group=[3, 7], ep_group=[0, 1, 2, 3]),
+ ]
+
+ # Destination with EP=8 (different from source EP=4)
+ # EP sizes differ (4 vs 8), so EP filtering should be skipped
+ dst_meta = self._create_metadata(
+ rank=8, tp_group=[8], dp_group=[8, 9], ep_group=list(range(8, 16))
+ )
+
+ # Should NOT filter by EP local rank (sizes differ)
+ selected = select_src_metadata_balanced(src_meta_list, dst_meta, dst_rank=8)
+
+ # Should work without error (no EP filtering when sizes differ)
+ assert selected.owner_rank in [0, 1, 2, 3]
+
+ def test_load_distribution_across_parameters(self):
+ """Test that different dst ranks select different DP groups for load balancing."""
+ # Setup: TP=1, DP=4, World=4
+
+ src_meta_list = [
+ self._create_metadata(rank=0, tp_group=[0], dp_group=[0, 1, 2, 3]),
+ self._create_metadata(rank=1, tp_group=[1], dp_group=[0, 1, 2, 3]),
+ self._create_metadata(rank=2, tp_group=[2], dp_group=[0, 1, 2, 3]),
+ self._create_metadata(rank=3, tp_group=[3], dp_group=[0, 1, 2, 3]),
+ ]
+
+ dst_meta = self._create_metadata(rank=0, tp_group=[0], dp_group=[0, 1, 2, 3, 4, 5, 6, 7])
+
+ # Simulate 8 destination ranks selecting sources
+ # Since there's only 1 DP group with 4 members, all should select the same group
+ # But round-robin based on dst_rank should still distribute across src ranks
+ selections = []
+ for dst_rank in range(8):
+ selected = select_src_metadata_balanced(src_meta_list, dst_meta, dst_rank)
+ selections.append(selected.owner_rank)
+
+ # All should select the same DP group (only one exists)
+ # But within that group, should cycle through available ranks
+ # Since there's only 1 DP group, they all select from it
+ assert all(rank in [0, 1, 2, 3] for rank in selections)
+
+ def test_within_dp_group_distribution(self):
+ """Test that dst ranks distribute across source ranks within a DP group."""
+ # This tests the optimization: when multiple dst ranks map to the same DP group,
+ # they should use different source ranks within that group for load balancing.
+
+ # Setup: TP=2, World=8 -> TP=1, World=8
+ # Source TP groups: [[0,1], [2,3], [4,5], [6,7]]
+ # Source DP groups: [[0,2,4,6], [1,3,5,7]] (2 DP replicas)
+
+ src_meta_list = [
+ self._create_metadata(rank=0, tp_group=[0, 1], dp_group=[0, 2, 4, 6]),
+ self._create_metadata(rank=1, tp_group=[0, 1], dp_group=[1, 3, 5, 7]),
+ self._create_metadata(rank=2, tp_group=[2, 3], dp_group=[0, 2, 4, 6]),
+ self._create_metadata(rank=3, tp_group=[2, 3], dp_group=[1, 3, 5, 7]),
+ self._create_metadata(rank=4, tp_group=[4, 5], dp_group=[0, 2, 4, 6]),
+ self._create_metadata(rank=5, tp_group=[4, 5], dp_group=[1, 3, 5, 7]),
+ self._create_metadata(rank=6, tp_group=[6, 7], dp_group=[0, 2, 4, 6]),
+ self._create_metadata(rank=7, tp_group=[6, 7], dp_group=[1, 3, 5, 7]),
+ ]
+
+ dst_meta = self._create_metadata(rank=0, tp_group=[0], dp_group=list(range(8)))
+
+ # Test 8 destination ranks
+ selections = {}
+ for dst_rank in range(8):
+ selected = select_src_metadata_balanced(src_meta_list, dst_meta, dst_rank)
+ selections[dst_rank] = (
+ selected.owner_rank,
+ tuple(selected.tensor_parallel_group_ranks),
+ )
+
+ # Verify distribution:
+ # dst_rank 0: DP group 0 ([0,2,4,6]), within-group idx 0 -> rank 0, TP [0,1]
+ # dst_rank 1: DP group 1 ([1,3,5,7]), within-group idx 0 -> rank 1, TP [0,1]
+ # dst_rank 2: DP group 0 ([0,2,4,6]), within-group idx 1 -> rank 2, TP [2,3]
+ # dst_rank 3: DP group 1 ([1,3,5,7]), within-group idx 1 -> rank 3, TP [2,3]
+ # dst_rank 4: DP group 0 ([0,2,4,6]), within-group idx 2 -> rank 4, TP [4,5]
+ # dst_rank 5: DP group 1 ([1,3,5,7]), within-group idx 2 -> rank 5, TP [4,5]
+ # dst_rank 6: DP group 0 ([0,2,4,6]), within-group idx 3 -> rank 6, TP [6,7]
+ # dst_rank 7: DP group 1 ([1,3,5,7]), within-group idx 3 -> rank 7, TP [6,7]
+
+ assert selections[0] == (0, (0, 1))
+ assert selections[1] == (1, (0, 1))
+ assert selections[2] == (2, (2, 3))
+ assert selections[3] == (3, (2, 3))
+ assert selections[4] == (4, (4, 5))
+ assert selections[5] == (5, (4, 5))
+ assert selections[6] == (6, (6, 7))
+ assert selections[7] == (7, (6, 7))
+
+ # Verify ALL source ranks are used (good load distribution!)
+ source_ranks_used = {sel[0] for sel in selections.values()}
+ assert source_ranks_used == {0, 1, 2, 3, 4, 5, 6, 7}, "All source ranks should be used"
+
+ # Verify each TP group used by 2 dst ranks (evenly distributed)
+ tp_group_usage = {}
+ for sel in selections.values():
+ tp_group = sel[1]
+ tp_group_usage[tp_group] = tp_group_usage.get(tp_group, 0) + 1
+
+ # Each of 4 TP groups should be used by exactly 2 destination ranks
+ assert all(count == 2 for count in tp_group_usage.values())
+ assert len(tp_group_usage) == 4 # 4 different TP groups
+
+ def test_local_copy_preference_collocated(self):
+ """Test that collocated mode prefers local copies when available."""
+ # Setup: Collocated TP=2, World=8, DP=4
+ # Each rank has both src and dst models with same configuration
+ # Should always prefer local copy (dst_rank == src_rank)
+
+ # Source metadata from all ranks
+ src_meta_list = [
+ self._create_metadata(rank=0, tp_group=[0, 1], dp_group=[0, 2, 4, 6]),
+ self._create_metadata(rank=2, tp_group=[2, 3], dp_group=[0, 2, 4, 6]),
+ self._create_metadata(rank=4, tp_group=[4, 5], dp_group=[0, 2, 4, 6]),
+ self._create_metadata(rank=6, tp_group=[6, 7], dp_group=[0, 2, 4, 6]),
+ ]
+
+ # Destination rank 0 (collocated - has both src and dst)
+ dst_meta = self._create_metadata(rank=0, tp_group=[0, 1], dp_group=[0, 2, 4, 6])
+
+ # Should select rank 0 for local copy (not rank 2, 4, or 6)
+ selected = select_src_metadata_balanced(src_meta_list, dst_meta, dst_rank=0)
+ assert selected.owner_rank == 0, "Should prefer local copy in collocated mode"
+
+ # Try rank 4
+ dst_meta = self._create_metadata(rank=4, tp_group=[4, 5], dp_group=[0, 2, 4, 6])
+ selected = select_src_metadata_balanced(src_meta_list, dst_meta, dst_rank=4)
+ assert selected.owner_rank == 4, "Should prefer local copy for rank 4"
+
+ def test_no_local_copy_non_collocated(self):
+ """Test that non-collocated mode still uses DP balancing."""
+ # Setup: Non-collocated - dst rank 8 not in source ranks [0,2,4,6]
+
+ src_meta_list = [
+ self._create_metadata(rank=0, tp_group=[0, 1], dp_group=[0, 2, 4, 6]),
+ self._create_metadata(rank=2, tp_group=[2, 3], dp_group=[0, 2, 4, 6]),
+ self._create_metadata(rank=4, tp_group=[4, 5], dp_group=[0, 2, 4, 6]),
+ self._create_metadata(rank=6, tp_group=[6, 7], dp_group=[0, 2, 4, 6]),
+ ]
+
+ # Destination rank 8 (not in source ranks - non-collocated)
+ dst_meta = self._create_metadata(rank=8, tp_group=[8, 9], dp_group=[8, 9])
+
+ # Should fall back to DP balancing (not trying to find rank 8 in sources)
+ selected = select_src_metadata_balanced(src_meta_list, dst_meta, dst_rank=8)
+ assert selected.owner_rank in [0, 2, 4, 6], "Should select from available source ranks"
diff --git a/tests/unit_tests/resharding/test_model_swap.py b/tests/unit_tests/resharding/test_model_swap.py
index 73296a175ed..70d81d97829 100644
--- a/tests/unit_tests/resharding/test_model_swap.py
+++ b/tests/unit_tests/resharding/test_model_swap.py
@@ -1,7 +1,9 @@
# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
import copy
+import gc
import os
import types
+from dataclasses import fields
from typing import List, Optional, Tuple
import pytest
@@ -14,10 +16,11 @@
from megatron.core.models.gpt.gpt_layer_specs import (
get_gpt_layer_local_spec,
get_gpt_layer_with_transformer_engine_spec,
+ get_gpt_mtp_block_spec,
)
from megatron.core.models.gpt.gpt_model import GPTModel
from megatron.core.process_groups_config import ProcessGroupCollection
-from megatron.core.resharding.refit import swap_model_weights
+from megatron.core.resharding.refit import clear_all_caches, swap_model_weights
from megatron.core.tensor_parallel.layers import ColumnParallelLinear, RowParallelLinear
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer.cuda_graphs import CudaGraphManager, _CudagraphGlobalRecord
@@ -31,6 +34,16 @@
except Exception:
has_nvshmem = False
+try:
+ import mamba_ssm # noqa: F401
+
+ from megatron.core.models.mamba.mamba_layer_specs import mamba_stack_spec
+ from megatron.core.models.mamba.mamba_model import MambaModel
+
+ has_mamba_deps = True
+except Exception:
+ has_mamba_deps = False
+
def _build_pg_collection(
tp_size: int, pp_size: int = None, ep_size: int = 1
@@ -82,6 +95,70 @@ def _build_pg_collection(
)
+def _destroy_pg_collection(pgc: ProcessGroupCollection):
+ """Destroy all process groups in a ProcessGroupCollection to free NCCL communicator memory."""
+ destroyed = set()
+ for f in fields(pgc):
+ pg = getattr(pgc, f.name, None)
+ if pg is not None and id(pg) not in destroyed:
+ destroyed.add(id(pg))
+ dist.destroy_process_group(pg)
+
+
+def _pp_flags(pg_collection) -> Tuple[bool, bool]:
+ """Return (pre_process, post_process) based on pipeline-parallel rank."""
+ pp_group = pg_collection.pp
+ pp_rank = dist.get_rank(pp_group)
+ pp_size = dist.get_world_size(pp_group)
+ return pp_rank == 0, pp_rank == pp_size - 1
+
+
+def _run_forward(model, tokens, position_ids, attention_mask, pg_collection):
+ """Run a forward pass using Megatron's pipeline schedule.
+
+ For PP=1 this is a simple forward call. For PP>1 this delegates to the
+ Megatron pipeline schedule which handles P2P communication between stages.
+
+ Returns logits on the last PP stage, None on other stages.
+ """
+ from megatron.core.pipeline_parallel import get_forward_backward_func
+ from megatron.core.pipeline_parallel.p2p_communication import P2PCommunicator
+
+ pp_group = pg_collection.pp
+ pp_size = dist.get_world_size(pp_group)
+ batch, seq_len = tokens.shape
+
+ def forward_step_func(data_iterator, model):
+ output = model(tokens, position_ids, attention_mask)
+
+ def loss_func(output_tensor, non_loss_data=False):
+ if non_loss_data:
+ return output_tensor
+ return output_tensor.sum(), {"logits": output_tensor}
+
+ return output, loss_func
+
+ forward_backward_func = get_forward_backward_func(pp_size=pp_size, vp_size=None)
+ kwargs = dict(
+ forward_step_func=forward_step_func,
+ data_iterator=iter([None]),
+ model=[model],
+ num_microbatches=1,
+ seq_length=seq_len,
+ micro_batch_size=batch,
+ forward_only=True,
+ collect_non_loss_data=True,
+ pg_collection=pg_collection,
+ )
+ if pp_size > 1:
+ kwargs["p2p_communicator"] = P2PCommunicator(pp_group, model.config)
+ result = forward_backward_func(**kwargs)
+ # result is a list of per-microbatch outputs; only populated on last PP stage
+ if result and result[0] is not None:
+ return result[0]
+ return None
+
+
def _build_gpt(
config: TransformerConfig,
vocab_size: int,
@@ -90,25 +167,67 @@ def _build_gpt(
parallel_output: bool = True,
num_moe_experts: Optional[int] = None,
) -> GPTModel:
+ layer_spec = get_gpt_layer_with_transformer_engine_spec(
+ num_experts=num_moe_experts, moe_grouped_gemm=(num_moe_experts is not None)
+ )
+ mtp_block_spec = None
+ if config.mtp_num_layers:
+ mtp_block_spec = get_gpt_mtp_block_spec(
+ config=config, spec=layer_spec, use_transformer_engine=True
+ )
+ pre_process, post_process = _pp_flags(pg_collection)
model = GPTModel(
config=config,
- transformer_layer_spec=get_gpt_layer_with_transformer_engine_spec(
- num_experts=num_moe_experts, moe_grouped_gemm=(num_moe_experts is not None)
- ),
+ transformer_layer_spec=layer_spec,
vocab_size=vocab_size,
max_sequence_length=seq_len,
- pre_process=True,
- post_process=True,
+ pre_process=pre_process,
+ post_process=post_process,
fp16_lm_cross_entropy=False,
parallel_output=parallel_output,
- share_embeddings_and_output_weights=True,
+ share_embeddings_and_output_weights=False,
position_embedding_type="rope",
rotary_percent=1.0,
pg_collection=pg_collection,
+ mtp_block_spec=mtp_block_spec,
)
return model
+def _build_mamba(
+ config: TransformerConfig,
+ vocab_size: int,
+ seq_len: int,
+ pg_collection,
+ hybrid_layer_pattern: str,
+ parallel_output: bool = True,
+):
+ pre_process, post_process = _pp_flags(pg_collection)
+ model = MambaModel(
+ config=config,
+ mamba_stack_spec=mamba_stack_spec,
+ vocab_size=vocab_size,
+ max_sequence_length=seq_len,
+ hybrid_layer_pattern=hybrid_layer_pattern,
+ pre_process=pre_process,
+ post_process=post_process,
+ fp16_lm_cross_entropy=False,
+ parallel_output=parallel_output,
+ share_embeddings_and_output_weights=False,
+ pg_collection=pg_collection,
+ )
+ return model
+
+
+def _mamba_layer_pattern(base: str, num_layers: int, pp_size: int) -> str:
+ """Build hybrid_layer_pattern with '|' pipeline stage boundaries."""
+ layers_per_stage = num_layers // pp_size
+ unit_len = len(base)
+ repeats_per_stage = layers_per_stage // unit_len
+ stage = base * repeats_per_stage
+ return "|".join([stage] * pp_size)
+
+
def _mp_config() -> ModelParallelConfig:
return ModelParallelConfig(
params_dtype=torch.float32,
@@ -138,25 +257,41 @@ def _set_pg_collection(module, tp_group, dp_group):
],
)
@pytest.mark.parametrize(
- "src_tp,src_pp,src_ep,dst_tp,dst_pp,dst_ep,num_experts",
+ "src_tp,src_pp,src_ep,dst_tp,dst_pp,dst_ep,num_experts,moe_mode",
[
- # TP only changes
- (2, 1, 1, 1, 1, 1, None), # TP2 -> TP1
- (1, 1, 1, 2, 1, 1, None), # TP1 -> TP2
- (2, 1, 1, 4, 1, 1, None), # TP2 -> TP4
- # # PP only changes
- (1, 2, 1, 1, 1, 1, None), # PP2 -> PP1
- (1, 1, 1, 1, 2, 1, None), # PP1 -> PP2
- # # Both TP and PP change
- (2, 2, 1, 1, 1, 1, None), # TP2,PP2 -> TP1,PP1
- (1, 1, 1, 2, 2, 1, None), # TP1,PP1 -> TP2,PP2
- (2, 1, 1, 1, 2, 1, None), # TP2,PP1 -> TP1,PP2
- (1, 2, 1, 2, 1, 1, None), # TP1,PP2 -> TP2,PP1
- (1, 2, 1, 2, 4, 1, None), # TP1,PP2 -> TP2,PP4
- (1, 1, 2, 1, 1, 4, 4), # EP2 -> EP4
- (1, 1, 2, 1, 1, 1, 4), # EP2 -> EP1
- (1, 1, 1, 1, 1, 2, 4),
- (1, 1, 2, 1, 2, 2, 4),
+ # ---- Non-MoE: TP only changes ----
+ (2, 1, 1, 1, 1, 1, None, None), # TP2 -> TP1
+ (1, 1, 1, 2, 1, 1, None, None), # TP1 -> TP2
+ (2, 1, 1, 4, 1, 1, None, None), # TP2 -> TP4
+ # ---- Non-MoE: PP only changes ----
+ (1, 2, 1, 1, 1, 1, None, None), # PP2 -> PP1
+ (1, 1, 1, 1, 2, 1, None, None), # PP1 -> PP2
+ # ---- Non-MoE: Both TP and PP change ----
+ (2, 2, 1, 1, 1, 1, None, None), # TP2,PP2 -> TP1,PP1
+ (1, 1, 1, 2, 2, 1, None, None), # TP1,PP1 -> TP2,PP2
+ (2, 1, 1, 1, 2, 1, None, None), # TP2,PP1 -> TP1,PP2
+ (1, 2, 1, 2, 1, 1, None, None), # TP1,PP2 -> TP2,PP1
+ (1, 2, 1, 2, 4, 1, None, None), # TP1,PP2 -> TP2,PP4
+ # ---- MoE: EP changes (standard) ----
+ (1, 1, 2, 1, 1, 4, 4, None), # EP2 -> EP4
+ (1, 1, 2, 1, 1, 1, 4, None), # EP2 -> EP1
+ (1, 1, 1, 1, 1, 2, 4, None), # EP1 -> EP2
+ (1, 1, 2, 1, 2, 2, 4, None), # EP2 -> PP2,EP2
+ # ---- MoE: mixed TP + EP (standard) ----
+ (2, 1, 2, 1, 1, 1, 4, None), # TP2,EP2 -> TP1,EP1
+ (1, 1, 1, 2, 1, 2, 4, None), # TP1,EP1 -> TP2,EP2
+ (4, 1, 1, 2, 1, 2, 4, None), # TP4,EP1 -> TP2,EP2
+ (2, 1, 2, 4, 1, 1, 4, None), # TP2,EP2 -> TP4,EP1
+ (4, 1, 1, 1, 1, 4, 4, None), # TP4,EP1 -> TP1,EP4
+ (1, 1, 4, 4, 1, 1, 4, None), # EP4 -> TP4,EP1
+ # ---- MoE latent: representative configs ----
+ (1, 1, 2, 1, 1, 1, 4, "latent"), # EP2 -> EP1
+ (2, 1, 2, 1, 1, 1, 4, "latent"), # TP2,EP2 -> TP1,EP1
+ (1, 1, 1, 2, 1, 2, 4, "latent"), # TP1,EP1 -> TP2,EP2
+ # ---- MoE latent + MTP: representative configs ----
+ (1, 1, 1, 1, 1, 2, 4, "latent_mtp"), # EP1 -> EP2
+ (2, 1, 2, 1, 1, 1, 4, "latent_mtp"), # TP2,EP2 -> TP1,EP1
+ (1, 1, 1, 2, 1, 2, 4, "latent_mtp"), # TP1,EP1 -> TP2,EP2
],
)
def test_swap_gpt_parametrized(
@@ -168,20 +303,20 @@ def test_swap_gpt_parametrized(
dst_pp: int,
dst_ep: int,
num_experts: Optional[int],
+ moe_mode: Optional[str],
):
- # Initialize environment with source MP sizing
+
Utils.initialize_model_parallel(
tensor_model_parallel_size=src_tp, pipeline_model_parallel_size=src_pp
)
- # Validate divisibility post-init using the default PG safely
world = dist.get_world_size()
if (world % (src_tp * src_pp * src_ep) != 0) or (world % (dst_tp * dst_pp * dst_ep) != 0):
Utils.destroy_model_parallel()
pytest.skip(
"WORLD_SIZE must be divisible by both src_tp*src_pp*src_ep and dst_tp*dst_pp*dst_ep"
)
- model_parallel_cuda_manual_seed(1234)
+ model_parallel_cuda_manual_seed(1234)
torch.manual_seed(1234)
device = torch.device(f"cuda:{torch.cuda.current_device()}")
@@ -205,29 +340,32 @@ def test_swap_gpt_parametrized(
# Build PGs and models (always use unified PG builder so we can set EP)
src_pgs = _build_pg_collection(tp_size=src_tp, pp_size=src_pp, ep_size=src_ep)
dst_pgs = _build_pg_collection(tp_size=dst_tp, pp_size=dst_pp, ep_size=dst_ep)
- # Apply EP configuration to TransformerConfigs when MoE is requested
+ # Apply PP/EP configuration to TransformerConfigs
src_cfg = copy.deepcopy(cfg)
dst_cfg = copy.deepcopy(cfg)
+ src_cfg.pipeline_model_parallel_size = src_pp
+ dst_cfg.pipeline_model_parallel_size = dst_pp
+
if num_experts is not None:
- src_cfg.num_moe_experts = num_experts
- dst_cfg.num_moe_experts = num_experts
- # Ensure MoE MLP has an intermediate size; __post_init__ won't rerun after manual mutation
- src_cfg.moe_ffn_hidden_size = src_cfg.ffn_hidden_size
- dst_cfg.moe_ffn_hidden_size = dst_cfg.ffn_hidden_size
- src_cfg.expert_model_parallel_size = src_ep
- dst_cfg.expert_model_parallel_size = dst_ep
- # Force grouped MLP path under Transformer Engine and satisfy requirements
- src_cfg.moe_grouped_gemm = True
- dst_cfg.moe_grouped_gemm = True
- src_cfg.add_bias_linear = False
- dst_cfg.add_bias_linear = False
- # Require Transformer Engine for TEGroupedMLP; skip if unavailable
+ for c, ep in [(src_cfg, src_ep), (dst_cfg, dst_ep)]:
+ c.num_moe_experts = num_experts
+ c.moe_ffn_hidden_size = c.ffn_hidden_size
+ c.expert_model_parallel_size = ep
+ c.moe_grouped_gemm = True
+ c.add_bias_linear = False
+ if moe_mode in ("latent", "latent_mtp"):
+ c.moe_latent_size = 16
+ c.moe_shared_expert_intermediate_size = 64
+ c.activation_func = torch.nn.functional.silu
+ c.gated_linear_unit = True
+ if moe_mode == "latent_mtp":
+ c.mtp_num_layers = 1
try:
import transformer_engine
except Exception:
Utils.destroy_model_parallel()
- pytest.skip("Transformer Engine not available; skipping TE-grouped MoE test")
- # Use parallel_output=False to gather TP logits inside model and emit only on last PP stage
+ pytest.skip("Transformer Engine not available; skipping MoE refit test")
+
src_model = (
_build_gpt(
src_cfg,
@@ -263,36 +401,184 @@ def test_swap_gpt_parametrized(
)
attention_mask = torch.ones((batch, 1, seq_len, seq_len), device=device, dtype=torch.bool)
- # Collect source reference logits (parallel_output=False ensures full vocab on last PP stage)
+ # Collect source reference logits
ref_logits = torch.empty(batch, seq_len, vocab_size, device=device, dtype=torch.float32)
src_pp_ranks = dist.get_process_group_ranks(src_pgs.pp)
src_last_pp_rank = src_pp_ranks[-1]
with torch.no_grad():
- src_out = src_model(tokens, position_ids, attention_mask)
+ src_out = _run_forward(src_model, tokens, position_ids, attention_mask, src_pgs)
if dist.get_rank() == src_last_pp_rank:
- ref = src_out # [b, s, vocab]
- ref_logits.copy_(ref)
+ ref_logits.copy_(src_out)
dist.broadcast(ref_logits, src=src_last_pp_rank, group=src_pgs.pp)
# Swap weights
swap_model_weights([src_model], [dst_model], refit_method=refit_backend)
- # Collect destination logits (parallel_output=False ensures full vocab on last PP stage)
+ # Collect destination logits
dst_logits = torch.empty(batch, seq_len, vocab_size, device=device, dtype=torch.float32)
dst_pp_ranks = dist.get_process_group_ranks(dst_pgs.pp)
dst_last_pp_rank = dst_pp_ranks[-1]
with torch.no_grad():
- dst_out = dst_model(
- tokens, position_ids, attention_mask
- ) # last stage returns tensor, others return None
+ dst_out = _run_forward(dst_model, tokens, position_ids, attention_mask, dst_pgs)
if dist.get_rank() == dst_last_pp_rank:
- dst_logits.copy_(dst_out) # [b, s, vocab]
+ dst_logits.copy_(dst_out)
dist.broadcast(dst_logits, src=dst_last_pp_rank, group=dst_pgs.pp)
# Compare
assert ref_logits.shape == dst_logits.shape
- assert torch.allclose(
- dst_logits, ref_logits, atol=1e-4, rtol=1e-4
- ), f"Refit src(TP={src_tp},PP={src_pp})->dst(TP={dst_tp},PP={dst_pp}) GPT outputs differ"
+ max_diff = (dst_logits - ref_logits).abs().max().item()
+ assert torch.allclose(dst_logits, ref_logits, atol=5e-4, rtol=5e-4), (
+ f"Refit src(TP={src_tp},PP={src_pp},EP={src_ep})"
+ f"->dst(TP={dst_tp},PP={dst_pp},EP={dst_ep}) "
+ f"moe_mode={moe_mode} outputs differ (max_diff={max_diff:.6f})"
+ )
dist.barrier()
+
+ # Free GPU memory to prevent OOM across the many parametrized test cases
+ del src_model, dst_model
+ # Clear refit caches before destroying model parallel to avoid stale plans
+ clear_all_caches()
+ _destroy_pg_collection(src_pgs)
+ _destroy_pg_collection(dst_pgs)
+ Utils.destroy_model_parallel()
+ gc.collect()
+ torch.cuda.empty_cache()
+
+
+@pytest.mark.parametrize(
+ "refit_backend",
+ [
+ pytest.param(
+ "nvshmem",
+ marks=pytest.mark.skipif(
+ not has_nvshmem,
+ reason="nvshmem.core is not available (NVSHMEM Python bindings not installed)",
+ ),
+ ),
+ "nccl",
+ "gloo",
+ ],
+)
+@pytest.mark.parametrize(
+ "src_tp,src_pp,dst_tp,dst_pp",
+ [
+ # TP only changes (exercises block-interleaved planner for Mamba in_proj)
+ (2, 1, 1, 1), # TP2 -> TP1
+ (1, 1, 2, 1), # TP1 -> TP2
+ (2, 1, 4, 1), # TP2 -> TP4
+ # TP + PP change together
+ (1, 1, 2, 2), # TP1,PP1 -> TP2,PP2
+ (2, 1, 1, 2), # TP2,PP1 -> TP1,PP2
+ ],
+)
+def test_swap_mamba_parametrized(
+ refit_backend: str, src_tp: int, src_pp: int, dst_tp: int, dst_pp: int
+):
+ if not has_mamba_deps:
+ pytest.skip("Mamba dependencies (mamba_ssm, einops) not available")
+
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=src_tp, pipeline_model_parallel_size=src_pp
+ )
+ world = dist.get_world_size()
+ if (world % (src_tp * src_pp) != 0) or (world % (dst_tp * dst_pp) != 0):
+ Utils.destroy_model_parallel()
+ pytest.skip("WORLD_SIZE must be divisible by both src_tp*src_pp and dst_tp*dst_pp")
+
+ model_parallel_cuda_manual_seed(1234)
+ torch.manual_seed(1234)
+ device = torch.device(f"cuda:{torch.cuda.current_device()}")
+
+ # Small Mamba config — use "M*" hybrid pattern to test both Mamba layers
+ # (block-interleaved in_proj resharding) and attention layers together.
+ seq_len = 8
+ vocab_size = 128
+ base_pattern = "M*"
+ # Ensure enough layers for both PP configs (at least len(base_pattern) per stage)
+ min_layers = max(src_pp, dst_pp) * len(base_pattern)
+ num_layers = max(min_layers, 4 if (src_pp > 1 or dst_pp > 1) else 2)
+ # Round up to be divisible by both pp_size * unit_len
+ from math import lcm
+
+ factor = lcm(src_pp, dst_pp) * len(base_pattern)
+ num_layers = ((num_layers + factor - 1) // factor) * factor
+
+ cfg = TransformerConfig(
+ num_layers=num_layers,
+ hidden_size=256,
+ num_attention_heads=8,
+ num_query_groups=4,
+ use_cpu_initialization=True,
+ pipeline_dtype=torch.float32,
+ hidden_dropout=0.0,
+ attention_dropout=0.0,
+ )
+
+ src_pgs = _build_pg_collection(tp_size=src_tp, pp_size=src_pp)
+ dst_pgs = _build_pg_collection(tp_size=dst_tp, pp_size=dst_pp)
+
+ src_pattern = _mamba_layer_pattern(base_pattern, num_layers, src_pp)
+ dst_pattern = _mamba_layer_pattern(base_pattern, num_layers, dst_pp)
+
+ src_model = (
+ _build_mamba(cfg, vocab_size, seq_len, src_pgs, src_pattern, parallel_output=False)
+ .to(device)
+ .eval()
+ )
+ dst_model = (
+ _build_mamba(cfg, vocab_size, seq_len, dst_pgs, dst_pattern, parallel_output=False)
+ .to(device)
+ .eval()
+ )
+
+ # Inputs
+ batch = 2
+ tokens = torch.randint(
+ low=0, high=vocab_size, size=(batch, seq_len), device=device, dtype=torch.long
+ )
+ position_ids = (
+ torch.arange(seq_len, device=device, dtype=torch.long).unsqueeze(0).expand(batch, -1)
+ )
+ attention_mask = torch.ones((batch, 1, seq_len, seq_len), device=device, dtype=torch.bool)
+
+ # Collect source reference logits
+ ref_logits = torch.empty(batch, seq_len, vocab_size, device=device, dtype=torch.float32)
+ src_pp_ranks = dist.get_process_group_ranks(src_pgs.pp)
+ src_last_pp_rank = src_pp_ranks[-1]
+ with torch.no_grad():
+ src_out = _run_forward(src_model, tokens, position_ids, attention_mask, src_pgs)
+ if dist.get_rank() == src_last_pp_rank:
+ ref_logits.copy_(src_out)
+ dist.broadcast(ref_logits, src=src_last_pp_rank, group=src_pgs.pp)
+
+ # Swap weights
+ swap_model_weights([src_model], [dst_model], refit_method=refit_backend)
+
+ # Collect destination logits
+ dst_logits = torch.empty(batch, seq_len, vocab_size, device=device, dtype=torch.float32)
+ dst_pp_ranks = dist.get_process_group_ranks(dst_pgs.pp)
+ dst_last_pp_rank = dst_pp_ranks[-1]
+ with torch.no_grad():
+ dst_out = _run_forward(dst_model, tokens, position_ids, attention_mask, dst_pgs)
+ if dist.get_rank() == dst_last_pp_rank:
+ dst_logits.copy_(dst_out)
+ dist.broadcast(dst_logits, src=dst_last_pp_rank, group=dst_pgs.pp)
+
+ # Compare
+ assert ref_logits.shape == dst_logits.shape
+ max_diff = (dst_logits - ref_logits).abs().max().item()
+ assert torch.allclose(dst_logits, ref_logits, atol=1e-3, rtol=1e-3), (
+ f"Mamba refit src(TP={src_tp},PP={src_pp})"
+ f"->dst(TP={dst_tp},PP={dst_pp}) "
+ f"outputs differ (max_diff={max_diff:.6f})"
+ )
+ dist.barrier()
+
+ # Free GPU memory to prevent OOM across the many parametrized test cases
+ del src_model, dst_model
+ clear_all_caches()
+ _destroy_pg_collection(src_pgs)
+ _destroy_pg_collection(dst_pgs)
Utils.destroy_model_parallel()
+ gc.collect()
+ torch.cuda.empty_cache()
diff --git a/tests/unit_tests/resharding/test_mxfp8_refit.py b/tests/unit_tests/resharding/test_mxfp8_refit.py
new file mode 100644
index 00000000000..815d4eeedac
--- /dev/null
+++ b/tests/unit_tests/resharding/test_mxfp8_refit.py
@@ -0,0 +1,242 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+import pytest
+import torch
+
+_IS_BLACKWELL = torch.cuda.is_available() and (torch.cuda.get_device_properties(0).major >= 10)
+
+try:
+ from flashinfer import mxfp8_quantize
+
+ _HAVE_FLASHINFER = True
+except ImportError:
+ _HAVE_FLASHINFER = False
+
+pytestmark = [
+ pytest.mark.skipif(not _IS_BLACKWELL, reason="MXFP8 tests require Blackwell GPU (SM >= 10)"),
+ pytest.mark.skipif(not _HAVE_FLASHINFER, reason="MXFP8 tests require FlashInfer"),
+]
+
+
+# ===========================================================================
+# MXFP8ReshardTransform
+# ===========================================================================
+
+
+class TestMXFP8ReshardTransform:
+ """Tests for the core MXFP8 reshard transform (transforms.py).
+
+ These test the receiver-side BF16→MXFP8 conversion paths that run on
+ every refit iteration, including the critical 1D-scale accumulation
+ logic that avoids corrupting swizzled scales from partial updates.
+ """
+
+ def _make_persistent_buffers(self, shapes):
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+
+ buffers = {}
+ for name, (M, K) in shapes.items():
+ x = torch.randn(M, K, dtype=torch.bfloat16, device="cuda")
+ buffers[name] = MXFP8Tensor.from_bf16(x)
+ return buffers
+
+ def test_finalize_recv_bf16_2d_scale(self):
+ """Receiver-side conversion with 2D scale: immediate per-slice quantization."""
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+ from megatron.core.resharding.transforms import MXFP8ReshardTransform
+
+ M, K = 64, 128
+ buf = MXFP8Tensor.from_bf16(torch.randn(M, K, dtype=torch.bfloat16, device="cuda"))
+
+ if buf.scale.ndim != 2:
+ pytest.skip("FlashInfer produced 1D swizzled scale; 2D-scale test not applicable")
+
+ t = MXFP8ReshardTransform(
+ convertible_params={"decoder.weight"},
+ persistent_buffers={"weight": buf},
+ buffer_key_prefix="decoder.",
+ convert_on_send=False,
+ )
+
+ new_data = torch.randn(M, K, dtype=torch.bfloat16, device="cuda")
+ t.finalize_recv("decoder.weight", (slice(None), slice(None)), [new_data])
+
+ expected = MXFP8Tensor.from_bf16(new_data)
+ assert torch.equal(buf.data, expected.data)
+ assert torch.equal(buf.scale, expected.scale)
+
+ def test_finalize_recv_bf16_1d_scale_accumulation(self):
+ """Receiver-side conversion with 1D scale: accumulate slices then quantize."""
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+ from megatron.core.resharding.transforms import MXFP8ReshardTransform
+
+ M, K = 64, 128
+ buf = MXFP8Tensor.from_bf16(torch.randn(M, K, dtype=torch.bfloat16, device="cuda"))
+
+ if buf.scale.ndim != 1:
+ pytest.skip("FlashInfer produced 2D scale; 1D-scale accumulation test not applicable")
+
+ t = MXFP8ReshardTransform(
+ convertible_params={"decoder.weight"},
+ persistent_buffers={"weight": buf},
+ buffer_key_prefix="decoder.",
+ convert_on_send=False,
+ )
+
+ full_data = torch.randn(M, K, dtype=torch.bfloat16, device="cuda")
+ half = M // 2
+
+ # First slice: should accumulate (not finalize yet)
+ t.finalize_recv("decoder.weight", (slice(0, half), slice(None)), [full_data[:half]])
+ assert "weight" in t._pending_1d, "Should be pending after partial slice"
+
+ # Second slice: should trigger final quantization
+ t.finalize_recv("decoder.weight", (slice(half, M), slice(None)), [full_data[half:]])
+ assert "weight" not in t._pending_1d, "Should be finalized after all slices"
+
+ expected = MXFP8Tensor.from_bf16(full_data)
+ assert torch.equal(buf.data, expected.data)
+ assert torch.equal(buf.scale, expected.scale)
+
+ def test_finalize_recv_1d_scale_wrong_element_count(self):
+ """1D accumulation should raise if total elements don't match (duplicate slices)."""
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+ from megatron.core.resharding.transforms import MXFP8ReshardTransform
+
+ M, K = 64, 128
+ buf = MXFP8Tensor.from_bf16(torch.randn(M, K, dtype=torch.bfloat16, device="cuda"))
+ if buf.scale.ndim != 1:
+ pytest.skip("Need 1D scale for this test")
+
+ t = MXFP8ReshardTransform(
+ convertible_params={"decoder.weight"},
+ persistent_buffers={"weight": buf},
+ buffer_key_prefix="decoder.",
+ convert_on_send=False,
+ )
+
+ half_data = torch.randn(M // 2, K, dtype=torch.bfloat16, device="cuda")
+ t.finalize_recv("decoder.weight", (slice(0, M // 2), slice(None)), [half_data])
+
+ with pytest.raises(AssertionError, match="duplicate or missing"):
+ overlap = torch.randn(M // 2 + 1, K, dtype=torch.bfloat16, device="cuda")
+ t.finalize_recv("decoder.weight", (slice(M // 2 - 1, M), slice(None)), [overlap])
+
+
+# ===========================================================================
+# quantize_params_to_mxfp8
+# ===========================================================================
+
+
+class TestQuantizeParamsToMXFP8:
+ """Tests for persistent buffer quantization (quantization/utils.py).
+
+ The persistent buffer address stability is critical for CUDA graph
+ compatibility — if addresses change, captured graphs segfault.
+ """
+
+ def test_basic_quantization_replaces_param(self):
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+ from megatron.core.inference.quantization.utils import quantize_params_to_mxfp8
+
+ model = torch.nn.Linear(128, 64, bias=False).to(dtype=torch.bfloat16, device="cuda")
+ buffers = quantize_params_to_mxfp8(model)
+
+ assert "weight" in buffers
+ assert isinstance(buffers["weight"], MXFP8Tensor)
+ assert buffers["weight"].data.shape == (64, 128)
+ assert "weight" not in model._parameters
+
+ def test_persistent_buffer_reuse_preserves_addresses(self):
+ """Second call must copy into existing buffers (CUDA graph address stability)."""
+ from megatron.core.inference.quantization.utils import quantize_params_to_mxfp8
+
+ model = torch.nn.Linear(128, 64, bias=False).to(dtype=torch.bfloat16, device="cuda")
+ buffers = quantize_params_to_mxfp8(model)
+ data_ptr = buffers["weight"].data.data_ptr()
+ scale_ptr = buffers["weight"].scale.data_ptr()
+
+ model2 = torch.nn.Linear(128, 64, bias=False).to(dtype=torch.bfloat16, device="cuda")
+ quantize_params_to_mxfp8(model2, persistent_buffers=buffers)
+
+ assert buffers["weight"].data.data_ptr() == data_ptr
+ assert buffers["weight"].scale.data_ptr() == scale_ptr
+
+ def test_nested_module_fqn(self):
+ """Recursive quantization should produce correct fully-qualified names."""
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+ from megatron.core.inference.quantization.utils import quantize_params_to_mxfp8
+
+ model = torch.nn.Sequential(
+ torch.nn.Linear(128, 64, bias=False), torch.nn.Linear(64, 32, bias=False)
+ ).to(dtype=torch.bfloat16, device="cuda")
+ buffers = quantize_params_to_mxfp8(model)
+
+ assert "0.weight" in buffers and "1.weight" in buffers
+ assert isinstance(buffers["0.weight"], MXFP8Tensor)
+
+
+# ===========================================================================
+# End-to-end MXFP8 refit integration (single-GPU)
+# ===========================================================================
+
+
+class TestMXFP8RefitIntegration:
+ """Integration tests simulating the full send→recv→finalize refit flow."""
+
+ def test_full_transform_roundtrip_bf16_wire(self):
+ """Simulate sender sending BF16, receiver converting to MXFP8."""
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+ from megatron.core.resharding.transforms import MXFP8ReshardTransform
+
+ M, K = 64, 128
+ src_weight = torch.randn(M, K, dtype=torch.bfloat16, device="cuda")
+ src_param = torch.nn.Parameter(src_weight.clone())
+
+ dst_buf = MXFP8Tensor.from_bf16(torch.randn(M, K, dtype=torch.bfloat16, device="cuda"))
+ t = MXFP8ReshardTransform(
+ convertible_params={"decoder.weight"},
+ persistent_buffers={"weight": dst_buf},
+ buffer_key_prefix="decoder.",
+ convert_on_send=False,
+ )
+
+ # Simulate: prepare_send → wire → prepare_recv → finalize_recv
+ sent = t.prepare_send("decoder.weight", (slice(None), slice(None)), src_param)
+ recv_bufs = t.prepare_recv("decoder.weight", (slice(None), slice(None)))
+ recv_bufs[0].copy_(sent[0])
+ t.finalize_recv("decoder.weight", (slice(None), slice(None)), recv_bufs)
+
+ expected = MXFP8Tensor.from_bf16(src_weight)
+ assert torch.equal(dst_buf.data, expected.data)
+ assert torch.equal(dst_buf.scale, expected.scale)
+
+ def test_multi_slice_assembly(self):
+ """Multiple row slices should correctly assemble the full quantized weight."""
+ from megatron.core.inference.quantization.mxfp8_tensor import MXFP8Tensor
+ from megatron.core.resharding.transforms import MXFP8ReshardTransform
+
+ M, K = 128, 256
+ full_weight = torch.randn(M, K, dtype=torch.bfloat16, device="cuda")
+ dst_buf = MXFP8Tensor.from_bf16(torch.zeros(M, K, dtype=torch.bfloat16, device="cuda"))
+
+ t = MXFP8ReshardTransform(
+ convertible_params={"decoder.weight"},
+ persistent_buffers={"weight": dst_buf},
+ buffer_key_prefix="decoder.",
+ convert_on_send=False,
+ )
+
+ # Send in 4 row-slices (simulates TP=4 refit)
+ chunk = M // 4
+ for i in range(4):
+ row_slice = (slice(i * chunk, (i + 1) * chunk), slice(None))
+ src_param = torch.nn.Parameter(full_weight.clone())
+ sent = t.prepare_send("decoder.weight", row_slice, src_param)
+ recv = t.prepare_recv("decoder.weight", row_slice)
+ recv[0].copy_(sent[0])
+ t.finalize_recv("decoder.weight", row_slice, recv)
+
+ expected = MXFP8Tensor.from_bf16(full_weight)
+ assert torch.equal(dst_buf.data, expected.data)
+ assert torch.equal(dst_buf.scale, expected.scale)
diff --git a/tests/unit_tests/resharding/test_task_segmenter.py b/tests/unit_tests/resharding/test_task_segmenter.py
new file mode 100644
index 00000000000..e092f0e1c99
--- /dev/null
+++ b/tests/unit_tests/resharding/test_task_segmenter.py
@@ -0,0 +1,185 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+import pytest
+
+from megatron.core.resharding.nvshmem_copy_service.nvshmem_types import (
+ MAX_SEGMENT_SIZE,
+ ReceiveRequest,
+ SendRequest,
+)
+from megatron.core.resharding.nvshmem_copy_service.planning.task_segmenter import TaskSegmenter
+
+
+class TestTaskSegmenter:
+ """Test suite for TaskSegmenter."""
+
+ def test_segment_small_request(self):
+ """Test segmenting a request smaller than max segment size."""
+ segmenter = TaskSegmenter()
+
+ # Request smaller than 256MB should not be segmented
+ send_req = SendRequest(
+ task_id=1, src_tensor=None, src_pos=0, size=500 * 1024, dest_pe=1
+ ) # 500KB
+ recv_req = ReceiveRequest(
+ task_id=1, dest_tensor=None, dest_pos=0, size=500 * 1024, src_pe=0
+ )
+
+ send_segments = segmenter.segment_send_request(send_req)
+ recv_segments = segmenter.segment_receive_request(recv_req)
+
+ # Should produce exactly one segment (no splitting)
+ assert len(send_segments) == 1
+ assert send_segments[0].task_id == 1
+ assert send_segments[0].size == 500 * 1024
+ assert send_segments[0].dest_pe == 1
+
+ assert len(recv_segments) == 1
+ assert recv_segments[0].task_id == 1
+ assert recv_segments[0].size == 500 * 1024
+
+ def test_segment_large_request(self):
+ """Test segmenting a request larger than max segment size."""
+ segmenter = TaskSegmenter()
+
+ # Request larger than 256MB should be segmented
+ task_size = 3 * MAX_SEGMENT_SIZE # 768MB
+ send_req = SendRequest(task_id=1, src_tensor=None, src_pos=0, size=task_size, dest_pe=1)
+
+ send_segments = segmenter.segment_send_request(send_req)
+
+ # Should produce 3 segments
+ assert len(send_segments) == 3
+ for segment in send_segments:
+ assert segment.size == MAX_SEGMENT_SIZE # Each segment is max size
+ assert segment.dest_pe == 1
+
+ def test_segment_not_exact_multiple(self):
+ """Test segmenting when size is not exact multiple of max segment size."""
+ segmenter = TaskSegmenter()
+
+ # 2.5 × 256MB = 640MB -> should produce 3 segments (256MB, 256MB, 128MB)
+ task_size = int(2.5 * MAX_SEGMENT_SIZE)
+ send_req = SendRequest(task_id=1, src_tensor=None, src_pos=0, size=task_size, dest_pe=1)
+
+ send_segments = segmenter.segment_send_request(send_req)
+
+ # Should produce 3 segments
+ assert len(send_segments) == 3
+ # First two segments are full size
+ assert send_segments[0].size == MAX_SEGMENT_SIZE
+ assert send_segments[1].size == MAX_SEGMENT_SIZE
+ # Last segment is remainder
+ assert send_segments[2].size == int(0.5 * MAX_SEGMENT_SIZE)
+
+ def test_segment_send_and_receive_match(self):
+ """Test that send and receive segmentation produces matching segments."""
+ segmenter = TaskSegmenter()
+
+ task_size = int(2.5 * MAX_SEGMENT_SIZE)
+ send_req = SendRequest(task_id=1, src_tensor=None, src_pos=0, size=task_size, dest_pe=1)
+ recv_req = ReceiveRequest(task_id=1, dest_tensor=None, dest_pos=0, size=task_size, src_pe=0)
+
+ send_segments = segmenter.segment_send_request(send_req)
+ recv_segments = segmenter.segment_receive_request(recv_req)
+
+ # Should produce same number of segments
+ assert len(send_segments) == len(recv_segments)
+
+ # Sizes should match
+ for send_seg, recv_seg in zip(send_segments, recv_segments):
+ assert send_seg.size == recv_seg.size
+
+ def test_segment_very_large_request(self):
+ """Test segmenting a very large request."""
+ segmenter = TaskSegmenter()
+
+ # 10 × 256MB = 2.56GB
+ task_size = 10 * MAX_SEGMENT_SIZE
+ send_req = SendRequest(task_id=1, src_tensor=None, src_pos=0, size=task_size, dest_pe=1)
+
+ send_segments = segmenter.segment_send_request(send_req)
+
+ # Should produce 10 segments
+ assert len(send_segments) == 10
+ # All segments should be full size
+ for segment in send_segments:
+ assert segment.size == MAX_SEGMENT_SIZE
+
+ def test_segment_zero_size_request(self):
+ """Test handling of zero-size request."""
+ segmenter = TaskSegmenter()
+
+ send_req = SendRequest(task_id=1, src_tensor=None, src_pos=0, size=0, dest_pe=1)
+
+ send_segments = segmenter.segment_send_request(send_req)
+
+ # Should produce one segment with size 0
+ assert len(send_segments) == 1
+ assert send_segments[0].size == 0
+
+ def test_segment_exactly_max_size(self):
+ """Test segmenting request that is exactly max segment size."""
+ segmenter = TaskSegmenter()
+
+ # Exactly 256MB - should NOT be segmented
+ send_req = SendRequest(
+ task_id=1, src_tensor=None, src_pos=0, size=MAX_SEGMENT_SIZE, dest_pe=1
+ )
+
+ send_segments = segmenter.segment_send_request(send_req)
+
+ # Should produce exactly 1 segment (no splitting needed)
+ assert len(send_segments) == 1
+ assert send_segments[0].size == MAX_SEGMENT_SIZE
+
+ def test_segment_preserves_destination(self):
+ """Test that segmentation preserves destination PE."""
+ segmenter = TaskSegmenter()
+
+ task_size = 2 * MAX_SEGMENT_SIZE
+ send_req = SendRequest(
+ task_id=1, src_tensor=None, src_pos=0, size=task_size, dest_pe=42
+ ) # Non-standard PE
+
+ send_segments = segmenter.segment_send_request(send_req)
+
+ # All segments should have same destination
+ for segment in send_segments:
+ assert segment.dest_pe == 42
+
+ def test_segment_position_offset(self):
+ """Test that segments have correct position offsets."""
+ segmenter = TaskSegmenter()
+
+ task_size = int(2.5 * MAX_SEGMENT_SIZE)
+ start_pos = 1000
+ send_req = SendRequest(
+ task_id=1, src_tensor=None, src_pos=start_pos, size=task_size, dest_pe=1
+ )
+
+ send_segments = segmenter.segment_send_request(send_req)
+
+ # Check position offsets
+ assert send_segments[0].src_pos == start_pos
+ assert send_segments[1].src_pos == start_pos + MAX_SEGMENT_SIZE
+ assert send_segments[2].src_pos == start_pos + 2 * MAX_SEGMENT_SIZE
+
+ def test_segment_task_id_encoding(self):
+ """Test that segments have encoded task IDs."""
+ segmenter = TaskSegmenter()
+
+ task_size = 2 * MAX_SEGMENT_SIZE
+ original_task_id = 42
+ send_req = SendRequest(
+ task_id=original_task_id, src_tensor=None, src_pos=0, size=task_size, dest_pe=1
+ )
+
+ send_segments = segmenter.segment_send_request(send_req)
+
+ # Segments should have encoded task IDs (different from original)
+ # Based on the encoding: REQUEST_ID_BASE + (task_id * SEGMENT_ID_MULTIPLIER) + segment_index
+ assert len(send_segments) == 2
+ assert send_segments[0].task_id != original_task_id
+ assert send_segments[1].task_id != original_task_id
+ # Segment IDs should be different
+ assert send_segments[0].task_id != send_segments[1].task_id
diff --git a/tests/unit_tests/resharding/test_workload_packer.py b/tests/unit_tests/resharding/test_workload_packer.py
new file mode 100644
index 00000000000..f8fef6cceb1
--- /dev/null
+++ b/tests/unit_tests/resharding/test_workload_packer.py
@@ -0,0 +1,128 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+import pytest
+
+from megatron.core.resharding.nvshmem_copy_service.nvshmem_types import SendRequest
+from megatron.core.resharding.nvshmem_copy_service.planning.workload_packer import WorkloadPacker
+
+
+class TestWorkloadPacker:
+ """Test suite for WorkloadPacker."""
+
+ def test_pack_single_request(self):
+ """Test packing a single send request."""
+ packer = WorkloadPacker()
+
+ requests = [SendRequest(task_id=1, src_tensor=None, src_pos=0, size=1000, dest_pe=1)]
+ workloads = packer.pack_workloads(requests, n_pes=2)
+
+ # Should create one batch for destination PE 1
+ assert 1 in workloads
+ assert len(workloads[1]) == 1
+ assert workloads[1][0].total_size == 1000
+ assert len(workloads[1][0].tasks) == 1
+
+ def test_pack_multiple_requests_same_dest(self):
+ """Test packing multiple requests to the same destination."""
+ packer = WorkloadPacker()
+
+ requests = [
+ SendRequest(task_id=1, src_tensor=None, src_pos=0, size=1000, dest_pe=1),
+ SendRequest(task_id=2, src_tensor=None, src_pos=0, size=2000, dest_pe=1),
+ SendRequest(task_id=3, src_tensor=None, src_pos=0, size=3000, dest_pe=1),
+ ]
+ workloads = packer.pack_workloads(requests, n_pes=2)
+
+ # All requests fit in one batch (under 256MB default limit)
+ assert 1 in workloads
+ assert len(workloads[1]) == 1
+ assert workloads[1][0].total_size == 6000
+ assert len(workloads[1][0].tasks) == 3
+
+ def test_pack_exceeds_batch_size(self):
+ """Test that requests are split when size limit exceeded."""
+ packer = WorkloadPacker()
+
+ # Create requests that exceed 256MB limit
+ mb_256 = 256 * 1024 * 1024
+ requests = [
+ SendRequest(task_id=1, src_tensor=None, src_pos=0, size=mb_256 - 1000, dest_pe=1),
+ SendRequest(task_id=2, src_tensor=None, src_pos=0, size=5000, dest_pe=1),
+ SendRequest(task_id=3, src_tensor=None, src_pos=0, size=2000, dest_pe=1),
+ ]
+ workloads = packer.pack_workloads(requests, n_pes=2)
+
+ # Should create 2 batches (first request alone, others together)
+ assert 1 in workloads
+ assert len(workloads[1]) == 2
+
+ def test_pack_multiple_destinations(self):
+ """Test packing requests to multiple destinations."""
+ packer = WorkloadPacker()
+
+ requests = [
+ SendRequest(task_id=1, src_tensor=None, src_pos=0, size=1000, dest_pe=1),
+ SendRequest(task_id=2, src_tensor=None, src_pos=0, size=2000, dest_pe=2),
+ SendRequest(task_id=3, src_tensor=None, src_pos=0, size=3000, dest_pe=1),
+ SendRequest(task_id=4, src_tensor=None, src_pos=0, size=4000, dest_pe=3),
+ ]
+ workloads = packer.pack_workloads(requests, n_pes=4)
+
+ # PE 1: requests 1 and 3 (4000 total)
+ assert len(workloads[1]) == 1
+ assert workloads[1][0].total_size == 4000
+
+ # PE 2: request 2 (2000 total)
+ assert len(workloads[2]) == 1
+ assert workloads[2][0].total_size == 2000
+
+ # PE 3: request 4 (4000 total)
+ assert len(workloads[3]) == 1
+ assert workloads[3][0].total_size == 4000
+
+ def test_pack_empty_requests(self):
+ """Test packing with no requests."""
+ packer = WorkloadPacker()
+ workloads = packer.pack_workloads([], n_pes=4)
+ # All PEs should have empty lists
+ for pe in range(4):
+ assert pe in workloads
+ assert len(workloads[pe]) == 0
+
+ def test_pack_descending_size_order(self):
+ """Test that packing sorts by size descending (largest first)."""
+ packer = WorkloadPacker()
+
+ requests = [
+ SendRequest(task_id=1, src_tensor=None, src_pos=0, size=1000, dest_pe=1),
+ SendRequest(task_id=2, src_tensor=None, src_pos=0, size=5000, dest_pe=1),
+ SendRequest(task_id=3, src_tensor=None, src_pos=0, size=3000, dest_pe=1),
+ SendRequest(task_id=4, src_tensor=None, src_pos=0, size=2000, dest_pe=1),
+ ]
+ workloads = packer.pack_workloads(requests, n_pes=2)
+
+ # All should be in one batch
+ assert 1 in workloads
+ assert len(workloads[1]) == 1
+
+ # Check that tasks are sorted by size (descending)
+ sizes = [req.size for req in workloads[1][0].tasks]
+ assert sizes == sorted(sizes, reverse=True)
+
+ def test_pack_preserves_task_ids(self):
+ """Test that packing preserves task IDs."""
+ packer = WorkloadPacker()
+
+ requests = [
+ SendRequest(task_id=100, src_tensor=None, src_pos=0, size=1000, dest_pe=1),
+ SendRequest(task_id=200, src_tensor=None, src_pos=0, size=2000, dest_pe=1),
+ SendRequest(task_id=300, src_tensor=None, src_pos=0, size=3000, dest_pe=1),
+ ]
+ workloads = packer.pack_workloads(requests, n_pes=2)
+
+ # All requests should be in one batch
+ assert 1 in workloads
+ assert len(workloads[1]) == 1
+
+ # Check task IDs are preserved (sorted by size descending: 300, 200, 100)
+ task_ids = [req.task_id for req in workloads[1][0].tasks]
+ assert task_ids == [300, 200, 100]
diff --git a/tests/unit_tests/rl/test_grouped_rollouts.py b/tests/unit_tests/rl/test_grouped_rollouts.py
new file mode 100644
index 00000000000..95974c5d91b
--- /dev/null
+++ b/tests/unit_tests/rl/test_grouped_rollouts.py
@@ -0,0 +1,119 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+import asyncio
+from unittest.mock import MagicMock
+
+import pytest
+
+from megatron.rl.agent.api import (
+ GroupedRolloutGenerator,
+ GroupedRolloutRequest,
+ Rollout,
+ RolloutGenerator,
+ RolloutGroup,
+)
+from megatron.rl.agent.weighted_multi_task import AgentConfig, WeightedMultiTask
+from megatron.rl.inference import ReturnsRaw
+
+
+class MockGenerator(RolloutGenerator, GroupedRolloutGenerator):
+ """Mock generator with configurable per-call delays."""
+
+ def __init__(self, env_id="test", num_slow_calls=0, **kwargs):
+ super().__init__(**kwargs)
+ self.env_id = env_id
+ self.num_slow_calls = num_slow_calls
+ self._call_count = 0
+
+ async def rollout(self, request):
+ raise NotImplementedError
+
+ async def group_rollout(self, request):
+ idx = self._call_count
+ self._call_count += 1
+ if idx < self.num_slow_calls:
+ await asyncio.sleep(0.03)
+ return [
+ Rollout(
+ trajectory=[f"t{idx}"],
+ reward=float(idx),
+ env_id=self.env_id,
+ policy_epoch=[[(0, 0)]],
+ kv_cache_epoch=[[(0, 0)]],
+ num_evictions=[0],
+ )
+ for _ in range(request.rollouts_per_group)
+ ]
+
+
+class TestGroupedRollouts:
+
+ @pytest.mark.asyncio
+ @pytest.mark.parametrize(
+ "num_slow_calls, streaming, num_groups, expected_count, expected_batch_ids",
+ [
+ pytest.param(0, False, 1, 8, None, id="non_batched"),
+ pytest.param(4, False, 2, 8, [0, 0, 1, 1, 2, 2, 3, 3], id="batched_submission_order"),
+ pytest.param(0, True, 1, 10, None, id="streaming"),
+ ],
+ )
+ async def test_get_grouped_rollouts(
+ self, num_slow_calls, streaming, num_groups, expected_count, expected_batch_ids
+ ):
+ gen = MockGenerator(parallel_generation_tasks=8, num_slow_calls=num_slow_calls)
+ request = GroupedRolloutRequest(
+ num_groups=num_groups,
+ rollouts_per_group=1,
+ inference_interface=MagicMock(spec=ReturnsRaw),
+ streaming=streaming,
+ enforce_order=num_groups > 1,
+ )
+ groups = []
+ async for group in gen.get_grouped_rollouts(request):
+ groups.append(group)
+ if request.streaming and len(groups) >= expected_count:
+ break
+
+ assert len(groups) == expected_count
+ if expected_batch_ids is not None:
+ assert [g.batch_id for g in groups] == expected_batch_ids
+
+ @pytest.mark.asyncio
+ async def test_weighted_multi_task(self):
+ configs = [
+ AgentConfig(agent_type=MockGenerator, agent_args={"env_id": "a"}, weight=3.0),
+ AgentConfig(agent_type=MockGenerator, agent_args={"env_id": "b"}, weight=1.0),
+ ]
+ mt = WeightedMultiTask(configs)
+ mt.parallel_generation_tasks = 4
+
+ captured = []
+ for agent in mt.agents:
+ original = agent.get_grouped_rollouts
+
+ async def spy(req, orig=original):
+ captured.append(req)
+ async for group in orig(req):
+ yield group
+
+ agent.get_grouped_rollouts = spy
+
+ request = GroupedRolloutRequest(
+ num_groups=4,
+ rollouts_per_group=1,
+ inference_interface=MagicMock(spec=ReturnsRaw),
+ streaming=False,
+ enforce_order=False,
+ )
+ groups = []
+ async for group in mt.get_grouped_rollouts(request):
+ groups.append(group)
+
+ assert len(groups) == 4
+ # Weights 3:1 → agent "a" produces 3 groups, agent "b" produces 1.
+ env_ids = [g[0].env_id for g in groups]
+ assert sorted(env_ids) == ["a", "a", "a", "b"]
+ for sub_req in captured:
+ assert sub_req.num_groups in (1, 3) # distributed proportionally by weight
+ assert sub_req.enforce_order == request.enforce_order
+ assert sub_req.streaming == request.streaming
diff --git a/tests/unit_tests/rl/test_rl_utils.py b/tests/unit_tests/rl/test_rl_utils.py
index cff62d40f0e..7d05708e27c 100644
--- a/tests/unit_tests/rl/test_rl_utils.py
+++ b/tests/unit_tests/rl/test_rl_utils.py
@@ -1,9 +1,9 @@
# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
import itertools
-import os
-from types import SimpleNamespace
+from unittest.mock import MagicMock
+import numpy as np
import pytest
import torch
@@ -14,12 +14,24 @@
from megatron.core.models.gpt.gpt_model import GPTModel
from megatron.core.num_microbatches_calculator import destroy_num_microbatches_calculator
from megatron.core.optimizer import OptimizerConfig, get_megatron_optimizer
-from megatron.core.pipeline_parallel.utils import is_pp_last_stage
+from megatron.core.pipeline_parallel import get_forward_backward_func
+from megatron.core.pipeline_parallel.utils import is_pp_first_stage, is_pp_last_stage
from megatron.core.process_groups_config import ProcessGroupCollection
-from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
+from megatron.core.tensor_parallel.random import (
+ initialize_rng_tracker,
+ model_parallel_cuda_manual_seed,
+)
from megatron.core.transformer import TransformerConfig
+from megatron.core.transformer.cuda_graphs import (
+ CudaGraphManager,
+ _CudagraphGlobalRecord,
+ create_cudagraphs,
+ delete_cuda_graphs,
+)
+from megatron.core.transformer.module import Float16Module
from megatron.rl import rl_utils
from megatron.rl.agent.api import TokenRollout
+from megatron.rl.sequence_packing_utils import get_default_packed_seq_params
from megatron.training.arguments import parse_args, validate_args
from megatron.training.global_vars import destroy_global_vars, set_global_variables
from tests.unit_tests.test_utilities import Utils
@@ -66,6 +78,7 @@ def __init__(self):
self.eod = 43
self.vocab_size = VOCAB
self.bos = None
+ self.library = None
def detokenize(self, tokens):
return [str(tok) for tok in tokens]
@@ -82,9 +95,12 @@ def initialize_model_parallel(request, monkeypatch):
monkeypatch.setenv("WANDB_MODE", "disabled")
monkeypatch.setenv("LOG_TO_WANDB", "false")
+ initialize_rng_tracker(use_te_rng_tracker=True, force_reset=True)
+
tp, pp = request.param
world_size = Utils.world_size
Utils.initialize_model_parallel(tensor_model_parallel_size=tp, pipeline_model_parallel_size=pp)
+ model_parallel_cuda_manual_seed(123)
dp = world_size // (tp * pp)
yield world_size, dp, tp, pp
Utils.destroy_model_parallel()
@@ -300,6 +316,9 @@ def test_prepare_data_for_update(self, initialize_model_parallel):
logprobs=[[0.1, 0.2, 0.3]],
env_id='MEGAENV',
problem_id="2",
+ policy_epoch=[[(0, 0)]],
+ kv_cache_epoch=[[(0, 0)]],
+ num_evictions=[0],
)
r2 = TokenRollout(
trajectory=[[1, 2, 3, 4]],
@@ -308,6 +327,9 @@ def test_prepare_data_for_update(self, initialize_model_parallel):
logprobs=[[0.1, 0.2, 0.3, -1.2]],
env_id='MEGAENV',
problem_id="2",
+ policy_epoch=[[(0, 0)]],
+ kv_cache_epoch=[[(0, 0)]],
+ num_evictions=[0],
)
rollouts = [[r1, r2] for _ in range(dp)]
@@ -326,6 +348,9 @@ def test_prepare_data_for_update(self, initialize_model_parallel):
logprobs=torch.tensor([[-0.2, -0.3, -3.2]]).cuda(),
env_id='MEGAENV',
problem_id="2",
+ policy_epoch=[[(0, 0)]],
+ kv_cache_epoch=[[(0, 0)]],
+ num_evictions=[0],
)
r2 = TokenRollout(
trajectory=torch.tensor([[1, 2, 234, tokenizer.eod]], dtype=torch.float).cuda(),
@@ -334,9 +359,12 @@ def test_prepare_data_for_update(self, initialize_model_parallel):
logprobs=torch.tensor([[-0.2, -0.3, -1.2]]),
env_id='MEGAENV',
problem_id="2",
+ policy_epoch=[[(0, 0)]],
+ kv_cache_epoch=[[(0, 0)]],
+ num_evictions=[0],
)
rollouts = [[r1, r2] for _ in range(dp)]
- data_iter = rl_utils.prepare_data_for_update(
+ data_iter, _, _ = rl_utils.prepare_data_for_update(
[model], {}, rollouts, tokenizer, sequence_packing=False, is_correction=False
)
@@ -367,6 +395,9 @@ def test_prepare_trajectories(self, use_sequence_packing, num_turns):
logprobs=[[0.1, 0.2, 0.3, 0.35]] * num_turns,
env_id='MEGAENV',
problem_id="1",
+ policy_epoch=[[(0, 0)]] * num_turns,
+ kv_cache_epoch=[[(0, 0)]] * num_turns,
+ num_evictions=[0] * num_turns,
)
r2 = TokenRollout(
trajectory=[[4, 5, 6, 7, tokenizer.eod]] * num_turns,
@@ -375,6 +406,9 @@ def test_prepare_trajectories(self, use_sequence_packing, num_turns):
logprobs=[[0.4, 0.5, 0.6, 0.7, 0.75]] * num_turns,
env_id='MEGAENV',
problem_id="2",
+ policy_epoch=[[(0, 0)]] * num_turns,
+ kv_cache_epoch=[[(0, 0)]] * num_turns,
+ num_evictions=[0] * num_turns,
)
r3 = TokenRollout(
trajectory=[[8, 9, tokenizer.eod]] * num_turns,
@@ -383,6 +417,9 @@ def test_prepare_trajectories(self, use_sequence_packing, num_turns):
logprobs=[[0.8, 0.9, 0.95]] * num_turns,
env_id='MEGAENV',
problem_id="3",
+ policy_epoch=[[(0, 0)]] * num_turns,
+ kv_cache_epoch=[[(0, 0)]] * num_turns,
+ num_evictions=[0] * num_turns,
)
rollouts = [r1, r2, r3]
@@ -654,3 +691,260 @@ def get_optimizer_state_devices():
f"Expected GPU memory to increase after restore. "
f"After offload: {memory_after_offload}, After restore: {memory_after_restore}"
)
+
+ @pytest.mark.parametrize(
+ "initialize_model_parallel",
+ [
+ pytest.param((tp, pp), id=f"tp{tp}-pp{pp}")
+ for tp, pp in itertools.product([1, 2, 4], [1, 2])
+ if tp * pp <= Utils.world_size
+ ],
+ indirect=["initialize_model_parallel"],
+ )
+ def test_gpt_logprobs(self, initialize_model_parallel):
+ """Test get logprobs on an actual model, not on a mocked one.
+
+ This can be useful for quick benchmarking/analyzing regressions too.
+ """
+
+ world_size, dp, tp, pp = initialize_model_parallel
+ micro_batch_size = 1
+ self.create_test_args(
+ tensor_model_parallel_size=tp,
+ pipeline_model_parallel_size=pp,
+ bf16=True,
+ micro_batch_size=micro_batch_size,
+ )
+ model_parallel_cuda_manual_seed(123)
+
+ transformer_config = TransformerConfig(
+ num_layers=10,
+ hidden_size=128,
+ num_attention_heads=16,
+ use_cpu_initialization=True,
+ embedding_init_method_std=1.0,
+ bf16=True,
+ pipeline_dtype=torch.bfloat16, # Without this, pp!=1 runs will fail.
+ )
+ vocab_size = 10_000
+ pp_group = ProcessGroupCollection.use_mpu_process_groups().pp
+ gpt_model = GPTModel(
+ config=transformer_config,
+ transformer_layer_spec=get_gpt_layer_with_transformer_engine_spec(),
+ vocab_size=vocab_size,
+ max_sequence_length=4192,
+ pre_process=is_pp_first_stage(pp_group),
+ post_process=is_pp_last_stage(pp_group),
+ ).cuda()
+ sequence_length = gpt_model.max_sequence_length
+
+ gpt_model = Float16Module(gpt_model.config, gpt_model)
+
+ data = list(range(sequence_length))
+ input_ids = torch.tensor(data, dtype=torch.int64).repeat((micro_batch_size, 1)).cuda()
+ position_ids = torch.tensor(data, dtype=torch.int64).repeat((micro_batch_size, 1)).cuda()
+
+ with torch.no_grad():
+ logprobs = rl_utils.compute_logprobs_batch(
+ model=gpt_model,
+ data_loader=[(input_ids, position_ids)],
+ forward_backward_func=get_forward_backward_func(pp_size=pp),
+ packing_context=None,
+ trajs_batch_size=micro_batch_size,
+ seq_length=sequence_length,
+ logprobs_batch_size=micro_batch_size,
+ decoder_seq_length=sequence_length,
+ dtype=torch.bfloat16,
+ pp_group=gpt_model.pg_collection.pp,
+ is_correction=False,
+ collect_non_loss_data=True,
+ )
+ if is_pp_last_stage(pp_group):
+ assert logprobs.shape == (micro_batch_size, sequence_length - 1)
+
+ @pytest.mark.parametrize(
+ "initialize_model_parallel",
+ [pytest.param((1, 1), id="tp1-pp1")],
+ indirect=["initialize_model_parallel"],
+ )
+ def test_get_logprobs_cuda_graphs(self, initialize_model_parallel):
+ """Test that get_logprobs reuses CUDA graphs created during training forward pass.
+
+ This test verifies that rl_utils.get_logprobs can reuse CUDA graphs by:
+ 1. Running a training-style forward pass on some model to record CUDA graph runners.
+ 2. Creating the CUDA graphs.
+ 3. Running `get_logprobs` to verify it reuses the same forward graph from training.
+ """
+
+ num_layers = 2
+
+ world_size, dp, tp, pp = initialize_model_parallel
+ self.create_test_args(
+ tensor_model_parallel_size=tp,
+ pipeline_model_parallel_size=pp,
+ rl_training_cuda_graphs=True,
+ cuda_graph_impl="local",
+ bf16=True,
+ rl_sequence_packing_max_sequences_per_bin=4,
+ )
+
+ # Create a model with training CUDA graphs enabled
+ transformer_config = TransformerConfig(
+ num_layers=num_layers,
+ hidden_size=64,
+ num_attention_heads=4,
+ use_cpu_initialization=True,
+ cuda_graph_impl="local",
+ bf16=True,
+ )
+ model = GPTModel(
+ config=transformer_config,
+ transformer_layer_spec=get_gpt_layer_with_transformer_engine_spec(),
+ vocab_size=256,
+ max_sequence_length=32,
+ ).cuda()
+
+ # Wrap in Float16Module so it accepts fp32_output argument from get_logprobs
+ wrapped_model = Float16Module(transformer_config, model)
+
+ # Create test inputs (batch_size=1 required for thd format with sequence packing)
+ batch_size = 1
+ seq_length = 16
+ tokens = torch.randint(0, 256, (batch_size, seq_length), dtype=torch.long).cuda()
+ position_ids = torch.arange(seq_length).unsqueeze(0).expand(batch_size, -1).cuda()
+
+ # Create packed_seq_params for dummy data
+ packed_seq_params = get_default_packed_seq_params(
+ seq_length=seq_length, max_sequences_per_bin=4, device=tokens.device
+ )
+
+ # Run a single training forward pass to record cudagraphs
+ output = wrapped_model(
+ tokens,
+ position_ids,
+ attention_mask=None,
+ packed_seq_params=packed_seq_params,
+ runtime_gather_output=True,
+ fp32_output=False,
+ )
+
+ # Run backward to reset runner status from BWD_READY back to FWD_READY
+ # This is needed because get_logprobs runs in no_grad mode and expects FWD_READY
+ loss = output.sum()
+ loss.backward()
+
+ # Collect all CudaGraphManager instances and their runners
+ cudagraph_managers = []
+ for module in wrapped_model.modules():
+ if hasattr(module, 'cudagraph_manager') and module.cudagraph_manager is not None:
+ cudagraph_managers.append(module.cudagraph_manager)
+
+ # Record runner count before creating graphs
+ runners_before = {id(mgr): len(mgr.cudagraph_runners) for mgr in cudagraph_managers}
+
+ create_cudagraphs()
+
+ # Verify that each runner has a fwd_graph created
+ for mgr in cudagraph_managers:
+ for runner in mgr.cudagraph_runners:
+ assert runner.fwd_graph is not None, (
+ f"Expected runner to have fwd_graph created after create_cudagraphs(), "
+ f"but fwd_graph is None"
+ )
+
+ # Now test `get_logprobs`; this should reuse the existing CUDA graphs
+ # We do not pass packed_seq_params; it should be created within `get_logprobs`
+ logprobs = rl_utils.get_logprobs(
+ wrapped_model, tokens, position_ids=position_ids, sequence_packing=True
+ )
+
+ # Verify that no new runners were created and graph was reused
+ runners_after = {id(mgr): len(mgr.cudagraph_runners) for mgr in cudagraph_managers}
+ for mgr_id, count_before in runners_before.items():
+ count_after = runners_after[mgr_id]
+ assert count_after == count_before, (
+ f"Expected runner count to remain {count_before} after `get_logprobs`, "
+ f"but got {count_after}. `get_logprobs` should not create new runners."
+ )
+
+ # Verify outputs are valid
+ assert output is not None, "Training forward pass should return valid output"
+ assert logprobs is not None, "get_logprobs should return valid output"
+
+ # Destroy all captured graphs deterministically
+ for l in model.decoder.layers:
+ for runner in getattr(l.cudagraph_manager, "cudagraph_runners", []):
+ # Safely delete both graphs if present
+ if hasattr(runner, "fwd_graph"):
+ del runner.fwd_graph
+ if hasattr(runner, "bwd_graph"):
+ del runner.bwd_graph
+
+ # Ensure all pending work is complete and graph destruction runs now
+ torch.cuda.synchronize()
+
+ _CudagraphGlobalRecord.cudagraph_created = False
+ _CudagraphGlobalRecord.cudagraph_record = []
+ CudaGraphManager.global_mempool = None
+ CudaGraphManager.fwd_mempools = None
+ CudaGraphManager.bwd_mempools = None
+
+ @pytest.mark.parametrize(
+ "initialize_model_parallel",
+ [pytest.param((1, 1), id="tp1-pp1")],
+ indirect=["initialize_model_parallel"],
+ )
+ def test_prep_wandb_metrics(self, initialize_model_parallel):
+ # This tests the computation and makes us fail noisily if
+ # inputs assumptions are changed, e.g. we expect rewards to come in groups (list[list[int]]).
+ traj_lens = [[3, 3], [1, 2]]
+ turn_lens = [[1, 2, 1, 1, 1], [1, 2]]
+ rewards = [[1, 1], [-1, 2]]
+ num_turns = [[42, 2], [10, 8]]
+ advantages = [0, 1]
+ # Per-rollout min iteration stamps (2 rollouts in group 1, 2 in group 2)
+ policy_epoch = [[4, 2], [5, 0]]
+ kv_cache_epoch = [[4, 3], [5, 1]]
+ # Per-turn max epoch stamps (when each turn completed)
+ completed_epochs = [[5, 3], [5, 1]]
+ num_evictions = [[0, 1], [0, 0]]
+ current_iteration = 6
+ metrics = rl_utils.prep_wandb_metrics(
+ MagicMock(),
+ traj_lens,
+ turn_lens,
+ rewards,
+ num_turns,
+ advantages,
+ policy_epoch=policy_epoch,
+ kv_cache_epoch=kv_cache_epoch,
+ completed_epochs=completed_epochs,
+ num_evictions=num_evictions,
+ current_iteration=current_iteration,
+ )
+ assert metrics["mean_reward"] == 0.75
+ assert metrics["mean_advantage"] == 0.5
+ assert metrics["nonzero_groups_ratio"] == 0.5
+ assert metrics["max_traj_length"] == 3
+ assert metrics["min_traj_length"] == 1
+ assert metrics["mean_traj_length"] == 2.25
+ assert metrics["mean_traj_length_std"] == 0.25
+ assert metrics["max_turn_length"] == 2
+ assert metrics["min_turn_length"] == 1
+ assert metrics["mean_turn_length"] == 1.35
+ assert np.isclose(metrics["mean_turn_length_std"], 0.45)
+ assert metrics["mean_num_turns"] == 15.5
+ assert metrics["max_num_turns"] == 42
+ assert metrics["min_num_turns"] == 2
+ # true_policy_staleness = [6-4, 6-2, 6-5, 6-0] = [2, 4, 1, 6] -> mean=3.25, max=6, min=1
+ assert metrics["mean_policy_staleness"] == np.mean([2, 4, 1, 6])
+ assert metrics["max_policy_staleness"] == 6
+ assert metrics["min_policy_staleness"] == 1
+ # true_kv_staleness = [6-4, 6-3, 6-5, 6-1] = [2, 3, 1, 5] -> mean=2.75, max=5, min=1
+ assert metrics["mean_kv_cache_staleness"] == np.mean([2, 3, 1, 5])
+ assert metrics["max_kv_cache_staleness"] == 5
+ assert metrics["min_kv_cache_staleness"] == 1
+ assert metrics["total_eviction_count"] == 1
+ assert metrics["max_num_evictions"] == 1
+ # mean_completion_gap = mean([6-5, 6-3, 6-5, 6-1]) = mean([1, 3, 1, 5]) = 2.5
+ assert metrics["mean_completion_gap"] == 2.5
diff --git a/tests/unit_tests/rl/test_sequence_packing_utils.py b/tests/unit_tests/rl/test_sequence_packing_utils.py
index 44a3de762f0..06e63adf217 100644
--- a/tests/unit_tests/rl/test_sequence_packing_utils.py
+++ b/tests/unit_tests/rl/test_sequence_packing_utils.py
@@ -412,6 +412,41 @@ def test_compute_packed_inference_logprobs_stats_shape_mismatch():
assert group_stats.mean_piold_to_inf_prob is None
+@pytest.mark.parametrize("num_sequences", [1, 10, 48, 49, 50])
+def test_cu_seqlens_size(num_sequences):
+ """Test that cu_seqlens always has a fixed size regardless of how many sequences are packed."""
+ max_sequences_per_bin = 50
+ bin_size = 1024
+
+ seq_len = bin_size // max_sequences_per_bin
+ seq_lengths = [seq_len] * num_sequences
+
+ packing_info = sequence_packing_utils.PackingInfo(
+ bin_seq_indices=[list(range(num_sequences))],
+ seq_starts={0: []},
+ seq_lengths=seq_lengths,
+ seq_to_bin_idx=[0] * num_sequences,
+ packing_algo='fifo',
+ )
+
+ params = sequence_packing_utils.create_packed_seq_params_for_bin(
+ packing_info=packing_info,
+ bin_idx=0,
+ bin_size=bin_size,
+ max_sequences_per_bin=max_sequences_per_bin,
+ device=torch.device('cpu'),
+ )
+
+ expected_size = max_sequences_per_bin + 2
+ assert params.cu_seqlens_q.shape[0] == expected_size, (
+ f"cu_seqlens_q has size {params.cu_seqlens_q.shape[0]} but expected {expected_size} "
+ f"for {num_sequences} sequences"
+ )
+ assert params.cu_seqlens_kv.shape[0] == expected_size
+ assert params.cu_seqlens_q[0] == 0
+ assert params.cu_seqlens_q[-1] == bin_size
+
+
@pytest.mark.parametrize(
"ratio,local_bins,world,expected_bs",
[
diff --git a/tests/unit_tests/ssm/ops/test_causal_conv1d_varlen.py b/tests/unit_tests/ssm/ops/test_causal_conv1d_varlen.py
new file mode 100644
index 00000000000..b09457363bd
--- /dev/null
+++ b/tests/unit_tests/ssm/ops/test_causal_conv1d_varlen.py
@@ -0,0 +1,173 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""Unit tests for the Triton varlen causal conv1d kernel.
+
+Tests correctness of `causal_conv1d_varlen_fn` against a reference implementation
+that loops over requests calling `causal_conv1d_fn` with `initial_states`.
+"""
+
+import pytest
+import torch
+
+from megatron.core.ssm.ops.causal_conv1d_varlen import causal_conv1d_varlen_fn
+
+try:
+ from causal_conv1d import causal_conv1d_fn
+
+ HAS_CAUSAL_CONV1D = True
+except ImportError:
+ HAS_CAUSAL_CONV1D = False
+
+
+def _reference_conv1d_varlen(x, weight, bias, cu_seqlens, initial_states, activation="silu"):
+ """Reference: per-request loop calling causal_conv1d_fn with initial_states."""
+ num_requests = cu_seqlens.shape[0] - 1
+ conv_dim = x.shape[1]
+ d_conv = weight.shape[1]
+ parts = []
+ for r in range(num_requests):
+ start = cu_seqlens[r].item()
+ end = cu_seqlens[r + 1].item()
+ if end <= start:
+ continue
+ seq_len_r = end - start
+ if initial_states is not None:
+ init_r = initial_states[r : r + 1] # (1, conv_dim, d_conv-1)
+ # causal_conv1d_fn with initial_states requires channels-last layout
+ # for both x and initial_states: create as (1, L, C) then transpose
+ x_r = x[start:end].unsqueeze(0).transpose(1, 2) # channels-last (1, C, L)
+ init_r = init_r.permute(0, 2, 1).contiguous().transpose(1, 2) # channels-last
+ else:
+ init_r = None
+ x_r = x[start:end].T.unsqueeze(0).contiguous() # (1, conv_dim, seq_len)
+ out_r = causal_conv1d_fn(
+ x=x_r, weight=weight, bias=bias, activation=activation, initial_states=init_r
+ )
+ parts.append(out_r.squeeze(0).T.contiguous()) # (seq_len, conv_dim)
+ return torch.cat(parts, dim=0) if parts else torch.empty(0, conv_dim, device=x.device)
+
+
+@pytest.mark.skipif(not HAS_CAUSAL_CONV1D, reason="causal_conv1d not installed")
+class TestCausalConv1dVarlen:
+ """Test causal_conv1d_varlen_fn against per-request causal_conv1d_fn reference."""
+
+ @pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16])
+ def test_single_request(self, dtype):
+ """Single request should match causal_conv1d_fn exactly."""
+ torch.manual_seed(42)
+ conv_dim, d_conv, seq_len = 64, 4, 32
+ device = "cuda"
+
+ x = torch.randn(seq_len, conv_dim, dtype=dtype, device=device)
+ weight = torch.randn(conv_dim, d_conv, dtype=dtype, device=device)
+ bias = torch.randn(conv_dim, dtype=dtype, device=device)
+ cu_seqlens = torch.tensor([0, seq_len], dtype=torch.int32, device=device)
+ initial_states = torch.randn(1, conv_dim, d_conv - 1, dtype=dtype, device=device)
+
+ out = causal_conv1d_varlen_fn(x, weight, bias, cu_seqlens, initial_states)
+ ref = _reference_conv1d_varlen(x, weight, bias, cu_seqlens, initial_states)
+
+ atol = 1e-2 if dtype == torch.bfloat16 else 1e-5
+ torch.testing.assert_close(out, ref, atol=atol, rtol=1e-2)
+
+ @pytest.mark.parametrize("dtype", [torch.float32, torch.bfloat16])
+ def test_multiple_requests_varying_lengths(self, dtype):
+ """Multiple requests with different sequence lengths."""
+ torch.manual_seed(123)
+ conv_dim, d_conv = 128, 4
+ seq_lens = [10, 25, 3, 50, 8]
+ device = "cuda"
+
+ total_tokens = sum(seq_lens)
+ x = torch.randn(total_tokens, conv_dim, dtype=dtype, device=device)
+ weight = torch.randn(conv_dim, d_conv, dtype=dtype, device=device)
+ bias = torch.randn(conv_dim, dtype=dtype, device=device)
+
+ cu_seqlens_list = [0]
+ for sl in seq_lens:
+ cu_seqlens_list.append(cu_seqlens_list[-1] + sl)
+ cu_seqlens = torch.tensor(cu_seqlens_list, dtype=torch.int32, device=device)
+
+ num_requests = len(seq_lens)
+ initial_states = torch.randn(num_requests, conv_dim, d_conv - 1, dtype=dtype, device=device)
+
+ out = causal_conv1d_varlen_fn(x, weight, bias, cu_seqlens, initial_states)
+ ref = _reference_conv1d_varlen(x, weight, bias, cu_seqlens, initial_states)
+
+ atol = 1e-2 if dtype == torch.bfloat16 else 1e-5
+ torch.testing.assert_close(out, ref, atol=atol, rtol=1e-2)
+
+ def test_seqlen_shorter_than_d_conv(self):
+ """Sequence shorter than d_conv should use initial_states for all taps."""
+ torch.manual_seed(7)
+ conv_dim, d_conv = 32, 4
+ seq_lens = [2, 1, 3] # All shorter than d_conv
+ device = "cuda"
+ dtype = torch.float32
+
+ total_tokens = sum(seq_lens)
+ x = torch.randn(total_tokens, conv_dim, dtype=dtype, device=device)
+ weight = torch.randn(conv_dim, d_conv, dtype=dtype, device=device)
+ bias = torch.randn(conv_dim, dtype=dtype, device=device)
+
+ cu_seqlens_list = [0]
+ for sl in seq_lens:
+ cu_seqlens_list.append(cu_seqlens_list[-1] + sl)
+ cu_seqlens = torch.tensor(cu_seqlens_list, dtype=torch.int32, device=device)
+
+ num_requests = len(seq_lens)
+ initial_states = torch.randn(num_requests, conv_dim, d_conv - 1, dtype=dtype, device=device)
+
+ out = causal_conv1d_varlen_fn(x, weight, bias, cu_seqlens, initial_states)
+ ref = _reference_conv1d_varlen(x, weight, bias, cu_seqlens, initial_states)
+
+ torch.testing.assert_close(out, ref, atol=1e-5, rtol=1e-5)
+
+ def test_zero_initial_states(self):
+ """Zero initial_states should produce same result as None initial_states."""
+ torch.manual_seed(99)
+ conv_dim, d_conv = 64, 4
+ seq_lens = [16, 24]
+ device = "cuda"
+ dtype = torch.float32
+
+ total_tokens = sum(seq_lens)
+ x = torch.randn(total_tokens, conv_dim, dtype=dtype, device=device)
+ weight = torch.randn(conv_dim, d_conv, dtype=dtype, device=device)
+ bias = torch.randn(conv_dim, dtype=dtype, device=device)
+
+ cu_seqlens_list = [0]
+ for sl in seq_lens:
+ cu_seqlens_list.append(cu_seqlens_list[-1] + sl)
+ cu_seqlens = torch.tensor(cu_seqlens_list, dtype=torch.int32, device=device)
+
+ num_requests = len(seq_lens)
+ zero_states = torch.zeros(num_requests, conv_dim, d_conv - 1, dtype=dtype, device=device)
+
+ out_zero = causal_conv1d_varlen_fn(x, weight, bias, cu_seqlens, zero_states)
+ out_none = causal_conv1d_varlen_fn(x, weight, bias, cu_seqlens, None)
+
+ torch.testing.assert_close(out_zero, out_none, atol=1e-5, rtol=1e-5)
+
+ def test_nonzero_vs_zero_initial_states_differ(self):
+ """Non-zero initial_states should produce different results from zero."""
+ torch.manual_seed(55)
+ conv_dim, d_conv = 64, 4
+ seq_len = 16
+ device = "cuda"
+ dtype = torch.float32
+
+ x = torch.randn(seq_len, conv_dim, dtype=dtype, device=device)
+ weight = torch.randn(conv_dim, d_conv, dtype=dtype, device=device)
+ bias = torch.randn(conv_dim, dtype=dtype, device=device)
+ cu_seqlens = torch.tensor([0, seq_len], dtype=torch.int32, device=device)
+
+ nonzero_states = torch.randn(1, conv_dim, d_conv - 1, dtype=dtype, device=device)
+
+ out_nonzero = causal_conv1d_varlen_fn(x, weight, bias, cu_seqlens, nonzero_states)
+ out_none = causal_conv1d_varlen_fn(x, weight, bias, cu_seqlens, None)
+
+ # First few tokens should differ (those that depend on initial state)
+ assert not torch.allclose(
+ out_nonzero[: d_conv - 1], out_none[: d_conv - 1], atol=1e-5
+ ), "Non-zero initial states should produce different outputs for early tokens"
diff --git a/tests/unit_tests/ssm/ops/test_ops_init.py b/tests/unit_tests/ssm/ops/test_ops_init.py
new file mode 100644
index 00000000000..9241d044dce
--- /dev/null
+++ b/tests/unit_tests/ssm/ops/test_ops_init.py
@@ -0,0 +1,33 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+"""Test that the megatron.core.ssm.ops package exports the public API."""
+
+import unittest
+
+try:
+ from megatron.core.ssm import ops as ssm_ops
+
+ HAVE_SSD_OPS = True
+except (ImportError, Exception):
+ HAVE_SSD_OPS = False
+
+
+@unittest.skipIf(not HAVE_SSD_OPS, "SSD ops (Triton 3+) not available")
+class TestOpsPackagePublicAPI(unittest.TestCase):
+ """Ensure the ops package exposes the documented public API."""
+
+ def test_all_exported(self):
+ self.assertIn("mamba_chunk_scan_combined_varlen", ssm_ops.__all__)
+ self.assertIn("causal_conv1d_varlen_fn", ssm_ops.__all__)
+
+ def test_mamba_chunk_scan_combined_varlen_importable(self):
+ self.assertTrue(hasattr(ssm_ops, "mamba_chunk_scan_combined_varlen"))
+ self.assertTrue(callable(ssm_ops.mamba_chunk_scan_combined_varlen))
+
+ def test_causal_conv1d_varlen_fn_importable(self):
+ self.assertTrue(hasattr(ssm_ops, "causal_conv1d_varlen_fn"))
+ self.assertTrue(callable(ssm_ops.causal_conv1d_varlen_fn))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/unit_tests/ssm/ops/test_ssd_bmm.py b/tests/unit_tests/ssm/ops/test_ssd_bmm.py
new file mode 100644
index 00000000000..c4c4eef5404
--- /dev/null
+++ b/tests/unit_tests/ssm/ops/test_ssd_bmm.py
@@ -0,0 +1,92 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+import unittest
+
+import torch
+
+try:
+ from megatron.core.ssm.ops.ssd_bmm import _bmm_chunk_fwd
+
+ HAVE_SSD_OPS = True
+except (ImportError, Exception):
+ HAVE_SSD_OPS = False
+
+
+@unittest.skipIf(not HAVE_SSD_OPS, "SSD ops (Triton 3+) not available")
+@unittest.skipIf(not torch.cuda.is_available(), "CUDA required for SSD ops")
+class TestBmmChunkFwd(unittest.TestCase):
+ """Tests for _bmm_chunk_fwd (C^T @ B per chunk)."""
+
+ def setUp(self):
+ torch.manual_seed(42)
+ self.device = torch.device("cuda")
+ self.chunk_size = 16
+ self.seqlen = 32
+ self.ngroups = 2
+ self.dstate = 8 # K dimension
+ self.cu_chunk_seqlens = torch.tensor([0, 16, 32], dtype=torch.int32, device=self.device)
+
+ def test_bmm_chunk_fwd_shape(self):
+ # a: (seqlen, ngroups, k), b: (seqlen, ngroups, k) -> out: (nchunks, ngroups, chunk_size, chunk_size)
+ a = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+ b = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+
+ out = _bmm_chunk_fwd(
+ a, b, self.chunk_size, self.cu_chunk_seqlens, causal=True, output_dtype=torch.float32
+ )
+
+ nchunks = self.cu_chunk_seqlens.shape[0] - 1
+ self.assertEqual(out.shape, (nchunks, self.ngroups, self.chunk_size, self.chunk_size))
+ self.assertFalse(torch.isnan(out).any())
+
+ def test_bmm_chunk_fwd_vs_torch_per_chunk(self):
+ """Compare first chunk with explicit C^T @ B for that chunk."""
+ a = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+ b = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+
+ out = _bmm_chunk_fwd(
+ a, b, self.chunk_size, self.cu_chunk_seqlens, causal=False, output_dtype=torch.float32
+ )
+
+ # Chunk 0: rows 0:16 of a and b. out[0, g] = a[0:16, g] @ b[0:16, g].T
+ # Relaxed tolerances: Triton block-wise reduction order can differ from torch;
+ # atol is the main check (max abs diff was ~0.008 in practice).
+ for g in range(self.ngroups):
+ a_chunk = a[0:16, g, :].contiguous() # (16, dstate)
+ b_chunk = b[0:16, g, :].contiguous() # (16, dstate)
+ expected = torch.mm(a_chunk, b_chunk.T) # (16, 16)
+ torch.testing.assert_close(out[0, g], expected, rtol=1.0, atol=0.02)
+
+ def test_bmm_chunk_fwd_causal_vs_non_causal_shape(self):
+ a = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+ b = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+
+ out_causal = _bmm_chunk_fwd(a, b, self.chunk_size, self.cu_chunk_seqlens, causal=True)
+ out_noncausal = _bmm_chunk_fwd(a, b, self.chunk_size, self.cu_chunk_seqlens, causal=False)
+
+ self.assertEqual(out_causal.shape, out_noncausal.shape)
+ # Causal: lower triangle is correct; upper can differ
+ for c in range(out_causal.shape[0]):
+ for g in range(self.ngroups):
+ for i in range(self.chunk_size):
+ for j in range(i + 1):
+ self.assertTrue(
+ torch.allclose(out_causal[c, g, i, j], out_noncausal[c, g, i, j]),
+ f"c={c} g={g} i={i} j={j}",
+ )
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/unit_tests/ssm/ops/test_ssd_chunk_scan.py b/tests/unit_tests/ssm/ops/test_ssd_chunk_scan.py
new file mode 100644
index 00000000000..dd654743346
--- /dev/null
+++ b/tests/unit_tests/ssm/ops/test_ssd_chunk_scan.py
@@ -0,0 +1,192 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+import unittest
+
+import torch
+
+try:
+ from megatron.core.ssm.ops.ssd_chunk_scan import _chunk_scan_fwd
+
+ HAVE_SSD_OPS = True
+except (ImportError, Exception):
+ HAVE_SSD_OPS = False
+
+
+@unittest.skipIf(not HAVE_SSD_OPS, "SSD ops (Triton 3+) not available")
+@unittest.skipIf(not torch.cuda.is_available(), "CUDA required for SSD ops")
+class TestChunkScanFwd(unittest.TestCase):
+ """Tests for _chunk_scan_fwd."""
+
+ def setUp(self):
+ torch.manual_seed(42)
+ self.device = torch.device("cuda")
+ self.seqlen = 32
+ self.nheads = 4
+ self.headdim = 16
+ self.ngroups = 2
+ self.dstate = 8
+ self.chunk_size = 16
+ self.nchunks = 2
+ self.cu_chunk_seqlens = torch.tensor([0, 16, 32], dtype=torch.int32, device=self.device)
+ self.seq_idx = torch.tensor([0, 1], dtype=torch.int32, device=self.device)
+
+ def test_chunk_scan_fwd_shape_and_inplace_out(self):
+ cb = torch.randn(
+ self.nchunks,
+ self.ngroups,
+ self.chunk_size,
+ self.chunk_size,
+ device=self.device,
+ dtype=torch.float32,
+ )
+ x = torch.randn(
+ self.seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+ dt = torch.randn(
+ self.nheads, self.nchunks, self.chunk_size, device=self.device, dtype=torch.float32
+ )
+ dA_cumsum = torch.randn(
+ self.nheads, self.nchunks, self.chunk_size, device=self.device, dtype=torch.float32
+ )
+ C = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+ states = torch.randn(
+ self.nchunks,
+ self.nheads,
+ self.headdim,
+ self.dstate,
+ device=self.device,
+ dtype=torch.float32,
+ )
+ out = torch.zeros(
+ self.seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+
+ _chunk_scan_fwd(
+ cb,
+ x,
+ dt,
+ dA_cumsum,
+ C,
+ states,
+ self.cu_chunk_seqlens,
+ out,
+ self.seq_idx,
+ D=None,
+ z=None,
+ initial_states=None,
+ )
+
+ self.assertEqual(out.shape, (self.seqlen, self.nheads, self.headdim))
+ self.assertFalse(torch.isnan(out).any())
+ # Output should be non-zero (scan writes to out)
+ self.assertGreater(out.abs().max().item(), 0.0)
+
+ def test_chunk_scan_fwd_with_D(self):
+ cb = torch.randn(
+ self.nchunks,
+ self.ngroups,
+ self.chunk_size,
+ self.chunk_size,
+ device=self.device,
+ dtype=torch.float32,
+ )
+ x = torch.randn(
+ self.seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+ dt = torch.randn(
+ self.nheads, self.nchunks, self.chunk_size, device=self.device, dtype=torch.float32
+ )
+ dA_cumsum = torch.randn(
+ self.nheads, self.nchunks, self.chunk_size, device=self.device, dtype=torch.float32
+ )
+ C = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+ states = torch.randn(
+ self.nchunks,
+ self.nheads,
+ self.headdim,
+ self.dstate,
+ device=self.device,
+ dtype=torch.float32,
+ )
+ out = torch.zeros(
+ self.seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+ D = torch.ones(self.nheads, self.headdim, device=self.device, dtype=torch.float32)
+
+ _chunk_scan_fwd(
+ cb,
+ x,
+ dt,
+ dA_cumsum,
+ C,
+ states,
+ self.cu_chunk_seqlens,
+ out,
+ self.seq_idx,
+ D=D,
+ z=None,
+ initial_states=None,
+ )
+
+ self.assertFalse(torch.isnan(out).any())
+
+ def test_chunk_scan_fwd_with_z(self):
+ cb = torch.randn(
+ self.nchunks,
+ self.ngroups,
+ self.chunk_size,
+ self.chunk_size,
+ device=self.device,
+ dtype=torch.float32,
+ )
+ x = torch.randn(
+ self.seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+ z = torch.randn(
+ self.seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+ dt = torch.randn(
+ self.nheads, self.nchunks, self.chunk_size, device=self.device, dtype=torch.float32
+ )
+ dA_cumsum = torch.randn(
+ self.nheads, self.nchunks, self.chunk_size, device=self.device, dtype=torch.float32
+ )
+ C = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+ states = torch.randn(
+ self.nchunks,
+ self.nheads,
+ self.headdim,
+ self.dstate,
+ device=self.device,
+ dtype=torch.float32,
+ )
+ out = torch.zeros(
+ self.seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+
+ _chunk_scan_fwd(
+ cb,
+ x,
+ dt,
+ dA_cumsum,
+ C,
+ states,
+ self.cu_chunk_seqlens,
+ out,
+ self.seq_idx,
+ D=None,
+ z=z,
+ initial_states=None,
+ )
+
+ self.assertFalse(torch.isnan(out).any())
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/unit_tests/ssm/ops/test_ssd_chunk_state.py b/tests/unit_tests/ssm/ops/test_ssd_chunk_state.py
new file mode 100644
index 00000000000..b848b6d7c96
--- /dev/null
+++ b/tests/unit_tests/ssm/ops/test_ssd_chunk_state.py
@@ -0,0 +1,208 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+import unittest
+
+import torch
+
+try:
+ from megatron.core.ssm.ops.ssd_chunk_state import (
+ _chunk_cumsum_fwd,
+ _chunk_state_fwd,
+ chunk_state_varlen,
+ )
+
+ HAVE_SSD_OPS = True
+except (ImportError, Exception):
+ HAVE_SSD_OPS = False
+
+
+@unittest.skipIf(not HAVE_SSD_OPS, "SSD ops (Triton 3+) not available")
+@unittest.skipIf(not torch.cuda.is_available(), "CUDA required for SSD ops")
+class TestChunkCumsumFwd(unittest.TestCase):
+ """Tests for _chunk_cumsum_fwd."""
+
+ def setUp(self):
+ torch.manual_seed(42)
+ self.device = torch.device("cuda")
+ self.seqlen = 32
+ self.nheads = 4
+ self.chunk_size = 16
+ self.cu_chunk_seqlens = torch.tensor([0, 16, 32], dtype=torch.int32, device=self.device)
+
+ def test_chunk_cumsum_fwd_shape(self):
+ dt = torch.randn(self.seqlen, self.nheads, device=self.device, dtype=torch.float32)
+ A = torch.randn(self.nheads, device=self.device, dtype=torch.float32)
+
+ dA_cumsum, dt_out = _chunk_cumsum_fwd(dt, A, self.chunk_size, self.cu_chunk_seqlens)
+
+ nchunks = self.cu_chunk_seqlens.shape[0] - 1
+ self.assertEqual(dA_cumsum.shape, (self.nheads, nchunks, self.chunk_size))
+ self.assertEqual(dt_out.shape, (self.nheads, nchunks, self.chunk_size))
+ self.assertFalse(torch.isnan(dA_cumsum).any())
+ self.assertFalse(torch.isnan(dt_out).any())
+
+ def test_chunk_cumsum_fwd_cumsum_per_chunk(self):
+ """dA_cumsum should be cumsum of dt * A along the chunk dimension."""
+ dt = torch.randn(self.seqlen, self.nheads, device=self.device, dtype=torch.float32)
+ A = torch.randn(self.nheads, device=self.device, dtype=torch.float32)
+
+ dA_cumsum, dt_out = _chunk_cumsum_fwd(
+ dt,
+ A,
+ self.chunk_size,
+ self.cu_chunk_seqlens,
+ dt_bias=None,
+ dt_softplus=False,
+ dt_limit=(0.0, float("inf")),
+ )
+
+ nchunks = self.cu_chunk_seqlens.shape[0] - 1
+ for c in range(nchunks):
+ start = self.cu_chunk_seqlens[c].item()
+ end = self.cu_chunk_seqlens[c + 1].item()
+ chunk_len = end - start
+ for h in range(self.nheads):
+ dA_chunk = (dt_out[h, c, :chunk_len] * A[h]).cpu()
+ expected_cumsum = torch.cumsum(dA_chunk, dim=0)
+ torch.testing.assert_close(
+ dA_cumsum[h, c, :chunk_len].cpu(), expected_cumsum, rtol=1e-4, atol=1e-4
+ )
+
+ def test_chunk_cumsum_fwd_with_dt_bias(self):
+ dt = torch.randn(self.seqlen, self.nheads, device=self.device, dtype=torch.float32)
+ A = torch.randn(self.nheads, device=self.device, dtype=torch.float32)
+ dt_bias = torch.randn(self.nheads, device=self.device, dtype=torch.float32)
+
+ dA_cumsum, dt_out = _chunk_cumsum_fwd(
+ dt, A, self.chunk_size, self.cu_chunk_seqlens, dt_bias=dt_bias
+ )
+
+ nchunks = self.cu_chunk_seqlens.shape[0] - 1
+ self.assertEqual(dA_cumsum.shape, (self.nheads, nchunks, self.chunk_size))
+ self.assertFalse(torch.isnan(dA_cumsum).any())
+
+
+@unittest.skipIf(not HAVE_SSD_OPS, "SSD ops (Triton 3+) not available")
+@unittest.skipIf(not torch.cuda.is_available(), "CUDA required for SSD ops")
+class TestChunkStateFwd(unittest.TestCase):
+ """Tests for _chunk_state_fwd."""
+
+ def setUp(self):
+ torch.manual_seed(42)
+ self.device = torch.device("cuda")
+ self.seqlen = 32
+ self.nheads = 4
+ self.headdim = 16
+ self.ngroups = 2
+ self.dstate = 8
+ self.chunk_size = 16
+ self.cu_chunk_seqlens = torch.tensor([0, 16, 32], dtype=torch.int32, device=self.device)
+
+ def test_chunk_state_fwd_shape(self):
+ x = torch.randn(
+ self.seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+ B = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+ dt = torch.randn(self.nheads, 2, self.chunk_size, device=self.device, dtype=torch.float32)
+ dA_cumsum = torch.randn(
+ self.nheads, 2, self.chunk_size, device=self.device, dtype=torch.float32
+ )
+
+ states = _chunk_state_fwd(B, x, dt, dA_cumsum, self.cu_chunk_seqlens)
+
+ nchunks = self.cu_chunk_seqlens.shape[0] - 1
+ self.assertEqual(states.shape, (nchunks, self.nheads, self.headdim, self.dstate))
+ self.assertFalse(torch.isnan(states).any())
+
+
+@unittest.skipIf(not HAVE_SSD_OPS, "SSD ops (Triton 3+) not available")
+@unittest.skipIf(not torch.cuda.is_available(), "CUDA required for SSD ops")
+class TestChunkStateVarlen(unittest.TestCase):
+ """Tests for chunk_state_varlen."""
+
+ def setUp(self):
+ torch.manual_seed(42)
+ self.device = torch.device("cuda")
+ self.seqlen = 32
+ self.nheads = 4
+ self.headdim = 16
+ self.ngroups = 2
+ self.dstate = 8
+ self.chunk_size = 16
+ self.batch = 2
+ self.cu_seqlens = torch.tensor([0, 16, 32], dtype=torch.int32, device=self.device)
+ self.cu_chunk_seqlens = torch.tensor([0, 16, 32], dtype=torch.int32, device=self.device)
+ self.last_chunk_indices = torch.tensor([0, 1], dtype=torch.int64, device=self.device)
+
+ def test_chunk_state_varlen_shape(self):
+ x = torch.randn(
+ self.seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+ B = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+ dt = torch.randn(self.nheads, 2, self.chunk_size, device=self.device, dtype=torch.float32)
+ dA_cumsum = torch.randn(
+ self.nheads, 2, self.chunk_size, device=self.device, dtype=torch.float32
+ )
+ chunk_states = torch.randn(
+ 2, self.nheads, self.headdim, self.dstate, device=self.device, dtype=torch.float32
+ )
+
+ states = chunk_state_varlen(
+ B,
+ x,
+ dt,
+ dA_cumsum,
+ self.cu_seqlens,
+ chunk_states,
+ last_chunk_indices=self.last_chunk_indices,
+ cu_chunk_seqlens=self.cu_chunk_seqlens,
+ )
+
+ self.assertEqual(states.shape, (self.batch, self.nheads, self.headdim, self.dstate))
+ self.assertFalse(torch.isnan(states).any())
+
+ def test_chunk_state_varlen_with_initial_states(self):
+ x = torch.randn(
+ self.seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+ B = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+ dt = torch.randn(self.nheads, 2, self.chunk_size, device=self.device, dtype=torch.float32)
+ dA_cumsum = torch.randn(
+ self.nheads, 2, self.chunk_size, device=self.device, dtype=torch.float32
+ )
+ chunk_states = torch.randn(
+ 2, self.nheads, self.headdim, self.dstate, device=self.device, dtype=torch.float32
+ )
+ initial_states = torch.randn(
+ self.batch,
+ self.nheads,
+ self.headdim,
+ self.dstate,
+ device=self.device,
+ dtype=torch.float32,
+ )
+
+ states = chunk_state_varlen(
+ B,
+ x,
+ dt,
+ dA_cumsum,
+ self.cu_seqlens,
+ chunk_states,
+ initial_states=initial_states,
+ last_chunk_indices=self.last_chunk_indices,
+ cu_chunk_seqlens=self.cu_chunk_seqlens,
+ )
+
+ self.assertEqual(states.shape, (self.batch, self.nheads, self.headdim, self.dstate))
+ self.assertFalse(torch.isnan(states).any())
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/unit_tests/ssm/ops/test_ssd_combined.py b/tests/unit_tests/ssm/ops/test_ssd_combined.py
new file mode 100644
index 00000000000..b5ef14f7a79
--- /dev/null
+++ b/tests/unit_tests/ssm/ops/test_ssd_combined.py
@@ -0,0 +1,361 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+import unittest
+
+import torch
+
+try:
+ from megatron.core.ssm.ops.ssd_combined import is_int_pow_2, mamba_chunk_scan_combined_varlen
+
+ HAVE_SSD_OPS = True
+except (ImportError, Exception):
+ HAVE_SSD_OPS = False
+
+
+@unittest.skipIf(not HAVE_SSD_OPS, "SSD ops (Triton 3+) not available")
+@unittest.skipIf(not torch.cuda.is_available(), "CUDA required for SSD ops")
+class TestIsIntPow2(unittest.TestCase):
+ """Tests for is_int_pow_2 utility."""
+
+ def test_powers_of_two(self):
+ for exp in range(12):
+ n = 2**exp
+ self.assertTrue(is_int_pow_2(n), f"2^{exp}={n} should be power of 2")
+
+ def test_non_powers_of_two(self):
+ for n in [0, 3, 5, 6, 7, 9, 10, 12, 15, 18]:
+ self.assertFalse(is_int_pow_2(n), f"{n} should not be power of 2")
+
+ def test_negative_and_float(self):
+ self.assertFalse(is_int_pow_2(-1))
+ self.assertFalse(is_int_pow_2(-4))
+ self.assertFalse(is_int_pow_2(2.0))
+ self.assertFalse(is_int_pow_2(0))
+
+
+@unittest.skipIf(not HAVE_SSD_OPS, "SSD ops (Triton 3+) not available")
+@unittest.skipIf(not torch.cuda.is_available(), "CUDA required for SSD ops")
+class TestMambaChunkScanCombinedVarlen(unittest.TestCase):
+ """Tests for mamba_chunk_scan_combined_varlen."""
+
+ def setUp(self):
+ torch.manual_seed(42)
+ self.device = torch.device("cuda")
+ self.chunk_size = 16
+ self.seqlen = 32
+ self.nheads = 4
+ self.headdim = 16
+ self.ngroups = 2
+ self.dstate = 8
+ self.batch = 2
+ # 2 chunks of 16 each
+ self.cu_chunk_seqlens = torch.tensor([0, 16, 32], dtype=torch.int32, device=self.device)
+ # last chunk index per sequence: seq0 ends in chunk 0, seq1 ends in chunk 1
+ self.last_chunk_indices = torch.tensor([0, 1], dtype=torch.int64, device=self.device)
+ # seq_idx: which sequence each chunk belongs to (nchunks,)
+ self.seq_idx = torch.tensor([0, 1], dtype=torch.int32, device=self.device)
+
+ def test_mamba_chunk_scan_combined_varlen_shape_and_no_nan(self):
+ x = torch.randn(
+ self.seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+ dt = torch.randn(self.seqlen, self.nheads, device=self.device, dtype=torch.float32)
+ A = torch.randn(self.nheads, device=self.device, dtype=torch.float32)
+ B = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+ C = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+ out = torch.empty(
+ self.seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+
+ varlen_states = mamba_chunk_scan_combined_varlen(
+ x=x,
+ dt=dt,
+ A=A,
+ B=B,
+ C=C,
+ chunk_size=self.chunk_size,
+ cu_chunk_seqlens=self.cu_chunk_seqlens,
+ last_chunk_indices=self.last_chunk_indices,
+ seq_idx=self.seq_idx,
+ out=out,
+ )
+
+ self.assertEqual(varlen_states.shape, (self.batch, self.nheads, self.headdim, self.dstate))
+ self.assertEqual(out.shape, (self.seqlen, self.nheads, self.headdim))
+ self.assertFalse(torch.isnan(out).any(), "output should have no NaN")
+ self.assertFalse(torch.isnan(varlen_states).any(), "varlen_states should have no NaN")
+
+ def test_mamba_chunk_scan_combined_varlen_with_D_and_dt_bias(self):
+ x = torch.randn(
+ self.seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+ dt = torch.randn(self.seqlen, self.nheads, device=self.device, dtype=torch.float32)
+ A = torch.randn(self.nheads, device=self.device, dtype=torch.float32)
+ B = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+ C = torch.randn(
+ self.seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32
+ )
+ D = torch.ones(self.nheads, self.headdim, device=self.device, dtype=torch.float32)
+ dt_bias = torch.randn(self.nheads, device=self.device, dtype=torch.float32)
+ out = torch.empty(
+ self.seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+
+ varlen_states = mamba_chunk_scan_combined_varlen(
+ x=x,
+ dt=dt,
+ A=A,
+ B=B,
+ C=C,
+ chunk_size=self.chunk_size,
+ cu_chunk_seqlens=self.cu_chunk_seqlens,
+ last_chunk_indices=self.last_chunk_indices,
+ seq_idx=self.seq_idx,
+ out=out,
+ D=D,
+ dt_bias=dt_bias,
+ )
+
+ self.assertEqual(varlen_states.shape, (self.batch, self.nheads, self.headdim, self.dstate))
+ self.assertFalse(torch.isnan(out).any())
+
+ def test_mamba_chunk_scan_combined_varlen_single_sequence(self):
+ """Single sequence of 32 tokens, split into 2 chunks of 16."""
+ cu_chunk_seqlens = torch.tensor([0, 16, 32], dtype=torch.int32, device=self.device)
+ last_chunk_indices = torch.tensor([1], dtype=torch.int64, device=self.device)
+ seq_idx = torch.tensor([0, 0], dtype=torch.int32, device=self.device)
+
+ x = torch.randn(32, self.nheads, self.headdim, device=self.device, dtype=torch.float32)
+ dt = torch.randn(32, self.nheads, device=self.device, dtype=torch.float32)
+ A = torch.randn(self.nheads, device=self.device, dtype=torch.float32)
+ B = torch.randn(32, self.ngroups, self.dstate, device=self.device, dtype=torch.float32)
+ C = torch.randn(32, self.ngroups, self.dstate, device=self.device, dtype=torch.float32)
+ out = torch.empty(32, self.nheads, self.headdim, device=self.device, dtype=torch.float32)
+
+ varlen_states = mamba_chunk_scan_combined_varlen(
+ x=x,
+ dt=dt,
+ A=A,
+ B=B,
+ C=C,
+ chunk_size=self.chunk_size,
+ cu_chunk_seqlens=cu_chunk_seqlens,
+ last_chunk_indices=last_chunk_indices,
+ seq_idx=seq_idx,
+ out=out,
+ )
+
+ self.assertEqual(varlen_states.shape, (1, self.nheads, self.headdim, self.dstate))
+ self.assertFalse(torch.isnan(out).any())
+
+
+@unittest.skipIf(not HAVE_SSD_OPS, "SSD ops (Triton 3+) not available")
+@unittest.skipIf(not torch.cuda.is_available(), "CUDA required for SSD ops")
+class TestIntermediateStateExtraction(unittest.TestCase):
+ """Tests for intermediate_chunk_indices parameter."""
+
+ def setUp(self):
+ torch.manual_seed(42)
+ self.device = torch.device("cuda")
+ self.chunk_size = 16
+ self.nheads = 4
+ self.headdim = 16
+ self.ngroups = 2
+ self.dstate = 8
+
+ def _make_inputs(self, seqlen):
+ """Create random inputs for a single sequence of given length."""
+ x = torch.randn(seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32)
+ dt = torch.randn(seqlen, self.nheads, device=self.device, dtype=torch.float32)
+ A = torch.randn(self.nheads, device=self.device, dtype=torch.float32)
+ B = torch.randn(seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32)
+ C = torch.randn(seqlen, self.ngroups, self.dstate, device=self.device, dtype=torch.float32)
+ out = torch.empty(
+ seqlen, self.nheads, self.headdim, device=self.device, dtype=torch.float32
+ )
+ return x, dt, A, B, C, out
+
+ def test_intermediate_states_shape_and_no_nan(self):
+ """1 sequence, 4 chunks. Request intermediates at chunks [0, 1, 2]."""
+ seqlen = 64 # 4 chunks of 16
+ nchunks = seqlen // self.chunk_size
+ x, dt, A, B, C, out = self._make_inputs(seqlen)
+ cu_chunk_seqlens = torch.arange(
+ 0, seqlen + 1, self.chunk_size, dtype=torch.int32, device=self.device
+ )
+ last_chunk_indices = torch.tensor([nchunks - 1], dtype=torch.int64, device=self.device)
+ seq_idx = torch.zeros(nchunks, dtype=torch.int32, device=self.device)
+ intermediate_chunk_indices = torch.tensor([0, 1, 2], dtype=torch.int64, device=self.device)
+
+ result = mamba_chunk_scan_combined_varlen(
+ x=x,
+ dt=dt,
+ A=A,
+ B=B,
+ C=C,
+ chunk_size=self.chunk_size,
+ cu_chunk_seqlens=cu_chunk_seqlens,
+ last_chunk_indices=last_chunk_indices,
+ seq_idx=seq_idx,
+ out=out,
+ intermediate_chunk_indices=intermediate_chunk_indices,
+ )
+
+ self.assertIsInstance(result, tuple)
+ final_states, intermediate_states = result
+ self.assertEqual(final_states.shape, (1, self.nheads, self.headdim, self.dstate))
+ self.assertEqual(intermediate_states.shape, (3, self.nheads, self.headdim, self.dstate))
+ self.assertFalse(torch.isnan(final_states).any())
+ self.assertFalse(torch.isnan(intermediate_states).any())
+
+ def test_intermediate_states_match_full_states(self):
+ """Intermediate states should match corresponding entries from full states."""
+ seqlen = 64 # 4 chunks
+ nchunks = seqlen // self.chunk_size
+ x, dt, A, B, C, out = self._make_inputs(seqlen)
+ cu_chunk_seqlens = torch.arange(
+ 0, seqlen + 1, self.chunk_size, dtype=torch.int32, device=self.device
+ )
+ last_chunk_indices = torch.tensor([nchunks - 1], dtype=torch.int64, device=self.device)
+ seq_idx = torch.zeros(nchunks, dtype=torch.int32, device=self.device)
+
+ # Run with return_intermediate_states=True to get all states
+ out1 = torch.empty_like(out)
+ all_states = mamba_chunk_scan_combined_varlen(
+ x=x,
+ dt=dt,
+ A=A,
+ B=B,
+ C=C,
+ chunk_size=self.chunk_size,
+ cu_chunk_seqlens=cu_chunk_seqlens,
+ last_chunk_indices=last_chunk_indices,
+ seq_idx=seq_idx,
+ out=out1,
+ return_intermediate_states=True,
+ )
+
+ # Run with intermediate_chunk_indices
+ indices = [0, 1, 2]
+ intermediate_chunk_indices = torch.tensor(indices, dtype=torch.int64, device=self.device)
+ out2 = torch.empty_like(out)
+ final_states, intermediate_states = mamba_chunk_scan_combined_varlen(
+ x=x,
+ dt=dt,
+ A=A,
+ B=B,
+ C=C,
+ chunk_size=self.chunk_size,
+ cu_chunk_seqlens=cu_chunk_seqlens,
+ last_chunk_indices=last_chunk_indices,
+ seq_idx=seq_idx,
+ out=out2,
+ intermediate_chunk_indices=intermediate_chunk_indices,
+ )
+
+ # Intermediate states should match the corresponding all_states entries
+ for i, chunk_idx in enumerate(indices):
+ torch.testing.assert_close(
+ intermediate_states[i],
+ all_states[chunk_idx],
+ msg=f"intermediate state at index {i} (chunk {chunk_idx}) does not match",
+ )
+
+ # Final state should match last chunk
+ torch.testing.assert_close(final_states[0], all_states[nchunks - 1])
+
+ def test_intermediate_states_multi_sequence(self):
+ """2 packed sequences, verify intermediate extraction across sequence boundaries."""
+ seq1_len = 32 # 2 chunks
+ seq2_len = 48 # 3 chunks
+ total_len = seq1_len + seq2_len
+ x, dt, A, B, C, out = self._make_inputs(total_len)
+
+ # cu_chunk_seqlens: seq1 has chunks at [0, 16, 32], seq2 at [32, 48, 64, 80]
+ boundaries = list(range(0, seq1_len + 1, self.chunk_size)) + list(
+ range(seq1_len + self.chunk_size, total_len + 1, self.chunk_size)
+ )
+ cu_chunk_seqlens = torch.tensor(boundaries, dtype=torch.int32, device=self.device)
+ nchunks = len(boundaries) - 1 # 5 chunks total
+ # Last chunk for seq1 is chunk 1, for seq2 is chunk 4
+ last_chunk_indices = torch.tensor([1, 4], dtype=torch.int64, device=self.device)
+ # seq_idx: [0, 0, 1, 1, 1]
+ seq_idx = torch.tensor([0, 0, 1, 1, 1], dtype=torch.int32, device=self.device)
+
+ # Request chunk 0 from seq1 and chunks 2, 3 from seq2
+ intermediate_chunk_indices = torch.tensor([0, 2, 3], dtype=torch.int64, device=self.device)
+
+ # Also get full states for comparison
+ out_full = torch.empty_like(out)
+ all_states = mamba_chunk_scan_combined_varlen(
+ x=x,
+ dt=dt,
+ A=A,
+ B=B,
+ C=C,
+ chunk_size=self.chunk_size,
+ cu_chunk_seqlens=cu_chunk_seqlens,
+ last_chunk_indices=last_chunk_indices,
+ seq_idx=seq_idx,
+ out=out_full,
+ return_intermediate_states=True,
+ )
+
+ out2 = torch.empty_like(out)
+ final_states, intermediate_states = mamba_chunk_scan_combined_varlen(
+ x=x,
+ dt=dt,
+ A=A,
+ B=B,
+ C=C,
+ chunk_size=self.chunk_size,
+ cu_chunk_seqlens=cu_chunk_seqlens,
+ last_chunk_indices=last_chunk_indices,
+ seq_idx=seq_idx,
+ out=out2,
+ intermediate_chunk_indices=intermediate_chunk_indices,
+ )
+
+ self.assertEqual(final_states.shape, (2, self.nheads, self.headdim, self.dstate))
+ self.assertEqual(intermediate_states.shape, (3, self.nheads, self.headdim, self.dstate))
+
+ # Verify intermediate states match full states
+ for i, chunk_idx in enumerate([0, 2, 3]):
+ torch.testing.assert_close(intermediate_states[i], all_states[chunk_idx])
+
+ def test_no_intermediate_returns_tensor(self):
+ """Without intermediate_chunk_indices, result should be a plain tensor."""
+ seqlen = 32
+ nchunks = seqlen // self.chunk_size
+ x, dt, A, B, C, out = self._make_inputs(seqlen)
+ cu_chunk_seqlens = torch.arange(
+ 0, seqlen + 1, self.chunk_size, dtype=torch.int32, device=self.device
+ )
+ last_chunk_indices = torch.tensor([nchunks - 1], dtype=torch.int64, device=self.device)
+ seq_idx = torch.zeros(nchunks, dtype=torch.int32, device=self.device)
+
+ result = mamba_chunk_scan_combined_varlen(
+ x=x,
+ dt=dt,
+ A=A,
+ B=B,
+ C=C,
+ chunk_size=self.chunk_size,
+ cu_chunk_seqlens=cu_chunk_seqlens,
+ last_chunk_indices=last_chunk_indices,
+ seq_idx=seq_idx,
+ out=out,
+ )
+
+ self.assertIsInstance(result, torch.Tensor)
+ self.assertEqual(result.shape, (1, self.nheads, self.headdim, self.dstate))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/unit_tests/ssm/ops/test_ssd_state_passing.py b/tests/unit_tests/ssm/ops/test_ssd_state_passing.py
new file mode 100644
index 00000000000..f791d615078
--- /dev/null
+++ b/tests/unit_tests/ssm/ops/test_ssd_state_passing.py
@@ -0,0 +1,89 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+import unittest
+
+import torch
+
+try:
+ from megatron.core.ssm.ops.ssd_state_passing import _state_passing_fwd
+
+ HAVE_SSD_OPS = True
+except (ImportError, Exception):
+ HAVE_SSD_OPS = False
+
+
+@unittest.skipIf(not HAVE_SSD_OPS, "SSD ops (Triton 3+) not available")
+@unittest.skipIf(not torch.cuda.is_available(), "CUDA required for SSD ops")
+class TestStatePassingFwd(unittest.TestCase):
+ """Tests for _state_passing_fwd: recurrence out = exp(dA_cs_last) * prev + new_states."""
+
+ def setUp(self):
+ torch.manual_seed(42)
+ self.device = torch.device("cuda")
+ self.nchunks = 4
+ self.nheads = 2
+ self.chunk_size = 16
+ self.dim = self.chunk_size * 8 # headdim * dstate flattened
+ self.cu_chunk_seqlens = torch.tensor(
+ [0, 16, 32, 48, 64], dtype=torch.int32, device=self.device
+ )
+
+ def test_state_passing_fwd_shape(self):
+ states = torch.randn(
+ self.nchunks, self.nheads, self.dim, device=self.device, dtype=torch.float32
+ )
+ dA_cumsum = torch.randn(
+ self.nheads, self.nchunks, self.chunk_size, device=self.device, dtype=torch.float32
+ )
+ seq_idx = torch.zeros(self.nchunks, dtype=torch.int32, device=self.device)
+
+ out = _state_passing_fwd(
+ states, dA_cumsum, self.cu_chunk_seqlens, seq_idx, initial_states=None
+ )
+
+ self.assertEqual(out.shape, (self.nchunks, self.nheads, self.dim))
+ self.assertFalse(torch.isnan(out).any())
+
+ def test_state_passing_fwd_with_initial_states(self):
+ states = torch.randn(
+ self.nchunks, self.nheads, self.dim, device=self.device, dtype=torch.float32
+ )
+ dA_cumsum = torch.randn(
+ self.nheads, self.nchunks, self.chunk_size, device=self.device, dtype=torch.float32
+ )
+ seq_idx = torch.tensor([0, 0, 1, 1], dtype=torch.int32, device=self.device)
+ initial_states = torch.randn(
+ 2, self.nheads, self.dim, device=self.device, dtype=torch.float32
+ )
+
+ out = _state_passing_fwd(
+ states, dA_cumsum, self.cu_chunk_seqlens, seq_idx, initial_states=initial_states
+ )
+
+ self.assertEqual(out.shape, (self.nchunks, self.nheads, self.dim))
+ self.assertFalse(torch.isnan(out).any())
+
+ def test_state_passing_fwd_recurrence_single_head_single_dim(self):
+ """Sanity: single head, small dim, check recurrence manually for first elements."""
+ dim = 4
+ nchunks = 2
+ nheads = 1
+ chunk_size = 2
+ cu_chunk_seqlens = torch.tensor([0, 2, 4], dtype=torch.int32, device=self.device)
+ seq_idx = torch.zeros(nchunks, dtype=torch.int32, device=self.device)
+
+ states = torch.randn(nchunks, nheads, dim, device=self.device, dtype=torch.float32)
+ dA_cumsum = torch.randn(
+ nheads, nchunks, chunk_size, device=self.device, dtype=torch.float32
+ )
+
+ out = _state_passing_fwd(states, dA_cumsum, cu_chunk_seqlens, seq_idx)
+
+ # Chunk 0: out[0] = exp(dA_cumsum[0,-1]) * 0 + states[0] = states[0] (no initial state)
+ # So out[0] should equal states[0]
+ torch.testing.assert_close(out[0], states[0], rtol=1e-4, atol=1e-4)
+ self.assertEqual(out.shape, (nchunks, nheads, dim))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/unit_tests/ssm/ops/test_ssm_kernel.py b/tests/unit_tests/ssm/ops/test_ssm_kernel.py
new file mode 100644
index 00000000000..a13ee59c794
--- /dev/null
+++ b/tests/unit_tests/ssm/ops/test_ssm_kernel.py
@@ -0,0 +1,189 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+import math
+import unittest
+from unittest.mock import MagicMock
+
+import torch
+import torch.nn as nn
+
+# Assume the provided class is in mamba_mixer.py
+from megatron.core.ssm.mamba_mixer import MambaMixer
+
+
+class MockContextParallel:
+ """
+ Mocks the MambaContextParallel helper.
+ """
+
+ def __init__(self, d_inner, ngroups, nheads, d_state, device):
+ self.d_inner_local_tpcp = d_inner
+ self.ngroups_local_tpcp = ngroups
+ self.nheads_local_tpcp = nheads
+ self.cp_size = 1
+
+ # Random weights for the mock
+ self.conv1d_weight = torch.randn(d_inner + 2 * ngroups * d_state, 1, 4, device=device)
+ self.conv1d_bias = torch.randn(d_inner + 2 * ngroups * d_state, device=device)
+ self.A_log = torch.randn(nheads, device=device)
+ self.D = torch.ones(nheads, device=device)
+ self.dt_bias = torch.randn(nheads, device=device)
+
+ # Simple conv1d layer for the fallback path if needed
+ self.conv1d_layer = nn.Conv1d(
+ in_channels=self.conv1d_weight.shape[0],
+ out_channels=self.conv1d_weight.shape[0],
+ kernel_size=4,
+ groups=self.conv1d_weight.shape[0],
+ padding=3,
+ ).to(device)
+
+ def get_A_log(self):
+ return self.A_log
+
+ def get_D(self):
+ return self.D
+
+ def get_dt_bias(self):
+ return self.dt_bias
+
+ def get_conv1d_weight(self):
+ return self.conv1d_weight
+
+ def get_conv1d_bias(self):
+ return self.conv1d_bias
+
+ def conv1d(self, x):
+ return self.conv1d_layer(x)
+
+ def pre_conv_ssm(self, x):
+ return x
+
+ def post_conv_ssm(self, x):
+ return x
+
+
+class TestMambaDynamicInference(unittest.TestCase):
+
+ def setUp(self):
+ torch.manual_seed(42)
+ self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+ if self.device.type == 'cpu':
+ self.skipTest("Mamba Triton kernels require CUDA")
+
+ # --- Configuration ---
+ self.d_model = 256
+ self.d_state = 16
+ self.headdim = 64
+ self.d_conv = 4
+ self.ngroups = 1
+ self.d_inner = self.d_model * 2 # expand=2
+ self.nheads = self.d_inner // self.headdim
+
+ # Create the Mixer instance directly
+ self.mixer = MagicMock(spec=MambaMixer)
+ self.mixer.d_state = self.d_state
+ self.mixer.d_conv = self.d_conv
+ self.mixer.headdim = self.headdim
+ self.mixer.chunk_size = 256
+ self.mixer.activation = "silu"
+ self.mixer.act = nn.SiLU()
+ self.mixer.D_has_hdim = False
+ self.mixer.rmsnorm = True
+
+ # Mock the Context Parallel wrapper (used by ssm_prefill)
+ self.mixer.cp = MockContextParallel(
+ d_inner=self.d_inner,
+ ngroups=self.ngroups,
+ nheads=self.nheads,
+ d_state=self.d_state,
+ device=self.device,
+ )
+
+ # --- Setup for ssm_decode ---
+ # ssm_decode accesses attributes directly from self, not self.cp
+ self.mixer.d_inner_local_tp = self.d_inner
+ self.mixer.ngroups_local_tp = self.ngroups
+ self.mixer.nheads_local_tp = self.nheads
+
+ # Create real parameters for ssm_decode to access
+ conv_dim = self.d_inner + 2 * self.ngroups * self.d_state
+ self.mixer.conv1d = nn.Conv1d(
+ in_channels=conv_dim,
+ out_channels=conv_dim,
+ kernel_size=self.d_conv,
+ groups=conv_dim,
+ padding=self.d_conv - 1,
+ bias=True,
+ device=self.device,
+ )
+ self.mixer.dt_bias = nn.Parameter(torch.randn(self.nheads, device=self.device))
+ self.mixer.A_log = nn.Parameter(torch.randn(self.nheads, device=self.device))
+ self.mixer.D = nn.Parameter(torch.ones(self.nheads, device=self.device))
+
+ # Bind methods
+ self.mixer._ssm_prefill = MambaMixer._ssm_prefill.__get__(self.mixer, MambaMixer)
+ self.mixer._ssm_decode = MambaMixer._ssm_decode.__get__(self.mixer, MambaMixer)
+
+ def test_ssm_prefill_padding_isolation(self):
+ """
+ Tests that ssm_prefill only updates states for the real request
+ and that padding request states remain untouched.
+
+ _ssm_prefill expects inputs pre-stripped to real tokens only
+ (stripping is done by _dynamic_inference_prefill). This test
+ passes only the real tokens and verifies that only the active
+ request's state is modified.
+ """
+ num_requests = 48
+ real_seq_len = 6
+
+ # Inputs: only real tokens (padding is stripped upstream)
+ dim_inputs = self.d_inner * 2 + 2 * self.ngroups * self.d_state + self.nheads
+ zxBCdt = torch.randn(real_seq_len, 1, dim_inputs, device=self.device, dtype=torch.float32)
+
+ # Metadata: single real request
+ seq_idx = torch.zeros((1, real_seq_len), dtype=torch.int32, device=self.device)
+
+ cu_seqlens = torch.tensor([0, real_seq_len], dtype=torch.int32, device=self.device)
+
+ batch_indices = torch.tensor([0], dtype=torch.long, device=self.device)
+
+ # States
+ conv_dim = self.d_inner + 2 * self.ngroups * self.d_state
+ conv_state = torch.zeros(num_requests, conv_dim, self.d_conv, device=self.device)
+ ssm_state = torch.zeros(
+ num_requests, self.nheads, self.headdim, self.d_state, device=self.device
+ )
+
+ # Run
+ self.mixer.norm = MagicMock(side_effect=lambda x, z: x * z)
+ output = self.mixer._ssm_prefill(
+ zxBCdt=zxBCdt,
+ conv_state=conv_state,
+ ssm_state=ssm_state,
+ seq_idx=seq_idx,
+ cu_seqlens=cu_seqlens,
+ batch_indices=batch_indices,
+ )
+
+ # Output should have real_seq_len tokens
+ self.assertEqual(output.shape[0], real_seq_len)
+ self.assertTrue(conv_state[0].abs().max() > 0, "Real request conv_state should be modified")
+
+ # Verify isolation of padding states
+ remaining_conv_states = conv_state[1:num_requests]
+ remaining_ssm_states = ssm_state[1:num_requests]
+
+ self.assertTrue(
+ torch.allclose(remaining_conv_states, torch.zeros_like(remaining_conv_states)),
+ "Conv states for padding requests (indices 1 to N-1) should remain 0",
+ )
+ self.assertTrue(
+ torch.allclose(remaining_ssm_states, torch.zeros_like(remaining_ssm_states)),
+ "SSM states for padding requests (indices 1 to N-1) should remain 0",
+ )
+
+
+if __name__ == '__main__':
+ unittest.main(argv=['first-arg-is-ignored'], exit=False)
diff --git a/tests/unit_tests/ssm/test_causal_conv1d_triton.py b/tests/unit_tests/ssm/test_causal_conv1d_triton.py
new file mode 100644
index 00000000000..3015f5ed989
--- /dev/null
+++ b/tests/unit_tests/ssm/test_causal_conv1d_triton.py
@@ -0,0 +1,258 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+
+import pytest
+import torch
+
+from megatron.core.ssm.ops.causal_conv1d_triton import causal_conv1d_update
+
+
+def _requires_cuda():
+ if not torch.cuda.is_available():
+ pytest.skip("CUDA not available")
+
+
+# ---------------------- Reference Implementations ---------------------- #
+
+
+def causal_conv1d_update_ref(x, conv_state, weight, bias, silu_activation):
+ """Reference: linear (non-circular) causal conv1d update."""
+ batch, seq_len, dim = x.shape
+ width = weight.shape[1]
+ state_len = conv_state.shape[-1]
+ out = torch.empty_like(x)
+ for b in range(batch):
+ for s in range(seq_len):
+ # Shift state left by 1
+ conv_state[b, :, :-1] = conv_state[b, :, 1:].clone()
+ conv_state[b, :, -1] = x[b, s, :]
+ # Convolution over the last `width` elements
+ window = conv_state[b, :, state_len - width : state_len].float()
+ w = weight.float()
+ val = (window * w).sum(dim=1)
+ if bias is not None:
+ val = val + bias.float()
+ if silu_activation:
+ val = val * torch.sigmoid(val)
+ out[b, s, :] = val.to(x.dtype)
+ return out
+
+
+# ---------------------- Tests ---------------------- #
+
+
+@pytest.mark.internal
+class TestCausalConv1dUpdate:
+
+ def setup_method(self, method):
+ _requires_cuda()
+
+ @pytest.mark.parametrize("width", [2, 3, 4])
+ def test_linear_no_bias(self, width):
+ torch.manual_seed(42)
+ B, seq_len, D, state_len = 2, 3, 64, 8
+ x = torch.randn(B, seq_len, D, device="cuda", dtype=torch.float32)
+ conv_state_triton = torch.randn(B, D, state_len, device="cuda", dtype=torch.float32)
+ conv_state_ref = conv_state_triton.clone()
+ weight = torch.randn(D, width, device="cuda", dtype=torch.float32)
+
+ result = causal_conv1d_update(
+ x, conv_state_triton, weight, bias=None, silu_activation=False, conv_state_indices=None
+ )
+ expected = causal_conv1d_update_ref(
+ x, conv_state_ref, weight, bias=None, silu_activation=False
+ )
+
+ torch.testing.assert_close(result, expected, atol=1e-5, rtol=1e-5)
+ torch.testing.assert_close(conv_state_triton, conv_state_ref, atol=1e-5, rtol=1e-5)
+
+ @pytest.mark.parametrize("width", [2, 3, 4])
+ def test_linear_with_bias(self, width):
+ torch.manual_seed(42)
+ B, seq_len, D, state_len = 2, 3, 64, 8
+ x = torch.randn(B, seq_len, D, device="cuda", dtype=torch.float32)
+ conv_state_triton = torch.randn(B, D, state_len, device="cuda", dtype=torch.float32)
+ conv_state_ref = conv_state_triton.clone()
+ weight = torch.randn(D, width, device="cuda", dtype=torch.float32)
+ bias = torch.randn(D, device="cuda", dtype=torch.float32)
+
+ result = causal_conv1d_update(
+ x, conv_state_triton, weight, bias=bias, silu_activation=False, conv_state_indices=None
+ )
+ expected = causal_conv1d_update_ref(
+ x, conv_state_ref, weight, bias=bias, silu_activation=False
+ )
+
+ torch.testing.assert_close(result, expected, atol=1e-5, rtol=1e-5)
+
+ @pytest.mark.parametrize("width", [2, 3, 4])
+ def test_linear_with_silu(self, width):
+ torch.manual_seed(42)
+ B, seq_len, D, state_len = 2, 1, 64, 8
+ x = torch.randn(B, seq_len, D, device="cuda", dtype=torch.float32)
+ conv_state_triton = torch.randn(B, D, state_len, device="cuda", dtype=torch.float32)
+ conv_state_ref = conv_state_triton.clone()
+ weight = torch.randn(D, width, device="cuda", dtype=torch.float32)
+ bias = torch.randn(D, device="cuda", dtype=torch.float32)
+
+ result = causal_conv1d_update(
+ x, conv_state_triton, weight, bias=bias, silu_activation="silu", conv_state_indices=None
+ )
+ expected = causal_conv1d_update_ref(
+ x, conv_state_ref, weight, bias=bias, silu_activation=True
+ )
+
+ torch.testing.assert_close(result, expected, atol=1e-4, rtol=1e-4)
+
+ def test_2d_input(self):
+ """Test that 2D input (B, D) is handled correctly and returns 2D output."""
+ torch.manual_seed(42)
+ B, D, state_len, width = 2, 64, 8, 4
+ x = torch.randn(B, D, device="cuda", dtype=torch.float32)
+ conv_state = torch.randn(B, D, state_len, device="cuda", dtype=torch.float32)
+ weight = torch.randn(D, width, device="cuda", dtype=torch.float32)
+
+ result = causal_conv1d_update(
+ x, conv_state, weight, bias=None, silu_activation=False, conv_state_indices=None
+ )
+
+ assert result.dim() == 2
+ assert result.shape == (B, D)
+
+ def test_conv_state_indices(self):
+ """Test that conv_state_indices correctly maps batch to state entries."""
+ torch.manual_seed(42)
+ B, D, state_len, width = 2, 64, 8, 4
+ num_states = 4
+ x = torch.randn(B, 1, D, device="cuda", dtype=torch.float32)
+ conv_state = torch.randn(num_states, D, state_len, device="cuda", dtype=torch.float32)
+ weight = torch.randn(D, width, device="cuda", dtype=torch.float32)
+ # Map batch 0 -> state 2, batch 1 -> state 0
+ state_indices = torch.tensor([2, 0], device="cuda", dtype=torch.int32)
+
+ # Run with indices
+ conv_state_copy = conv_state.clone()
+ result = causal_conv1d_update(
+ x,
+ conv_state_copy,
+ weight,
+ bias=None,
+ silu_activation=False,
+ conv_state_indices=state_indices,
+ )
+
+ # Run without indices by manually reordering
+ conv_state_reordered = conv_state[state_indices.long()].clone()
+ expected = causal_conv1d_update(
+ x,
+ conv_state_reordered,
+ weight,
+ bias=None,
+ silu_activation=False,
+ conv_state_indices=None,
+ )
+
+ torch.testing.assert_close(result, expected, atol=1e-5, rtol=1e-5)
+
+ def test_negative_state_index_zeros_output(self):
+ """Padding batch entries (index < 0) should produce zero output."""
+ torch.manual_seed(42)
+ B, D, state_len, width = 2, 64, 8, 4
+ x = torch.randn(B, 1, D, device="cuda", dtype=torch.float32)
+ conv_state = torch.randn(B, D, state_len, device="cuda", dtype=torch.float32)
+ weight = torch.randn(D, width, device="cuda", dtype=torch.float32)
+ state_indices = torch.tensor([-1, 0], device="cuda", dtype=torch.int32)
+
+ result = causal_conv1d_update(
+ x,
+ conv_state,
+ weight,
+ bias=None,
+ silu_activation=False,
+ conv_state_indices=state_indices,
+ )
+
+ # Batch 0 (padded) should be all zeros
+ torch.testing.assert_close(result[0], torch.zeros(1, D, device="cuda", dtype=torch.float32))
+
+ @pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16])
+ def test_half_precision(self, dtype):
+ torch.manual_seed(42)
+ B, seq_len, D, state_len, width = 2, 1, 64, 8, 4
+ x = torch.randn(B, seq_len, D, device="cuda", dtype=dtype)
+ conv_state = torch.randn(B, D, state_len, device="cuda", dtype=dtype)
+ weight = torch.randn(D, width, device="cuda", dtype=dtype)
+
+ result = causal_conv1d_update(
+ x, conv_state, weight, bias=None, silu_activation=False, conv_state_indices=None
+ )
+
+ assert result.dtype == dtype
+ assert result.shape == (B, seq_len, D)
+ assert torch.isfinite(result).all()
+
+ @pytest.mark.parametrize("width", [2, 3, 4])
+ def test_intermediate_state(self, width):
+ """Test that intermediate conv states are correctly stored at each sequence step."""
+ torch.manual_seed(42)
+ B, seq_len, D, state_len = 2, 4, 64, 8
+ x = torch.randn(B, seq_len, D, device="cuda", dtype=torch.float32)
+ conv_state = torch.randn(B, D, state_len, device="cuda", dtype=torch.float32)
+ weight = torch.randn(D, width, device="cuda", dtype=torch.float32)
+
+ # Allocate intermediate state buffer: (B, seq_len, D, state_len)
+ int_states = torch.zeros(B, seq_len, D, state_len, device="cuda", dtype=torch.float32)
+
+ # Run with intermediate state recording
+ conv_state_copy = conv_state.clone()
+ result = causal_conv1d_update(
+ x,
+ conv_state_copy,
+ weight,
+ bias=None,
+ silu_activation=False,
+ conv_state_indices=None,
+ intermediate_conv_states=int_states,
+ )
+
+ # Verify by running step-by-step and checking each intermediate
+ conv_state_ref = conv_state.clone()
+ for s in range(seq_len):
+ conv_state_ref[:, :, :-1] = conv_state_ref[:, :, 1:].clone()
+ conv_state_ref[:, :, -1] = x[:, s, :]
+ torch.testing.assert_close(int_states[:, s, :, :], conv_state_ref, atol=1e-5, rtol=1e-5)
+
+ def test_intermediate_state_with_indices(self):
+ """Test intermediate states work correctly with conv_state_indices mapping."""
+ torch.manual_seed(42)
+ B, seq_len, D, state_len, width = 2, 3, 64, 8, 4
+ num_states = 4
+ x = torch.randn(B, seq_len, D, device="cuda", dtype=torch.float32)
+ conv_state = torch.randn(num_states, D, state_len, device="cuda", dtype=torch.float32)
+ weight = torch.randn(D, width, device="cuda", dtype=torch.float32)
+ state_indices = torch.tensor([2, 0], device="cuda", dtype=torch.int32)
+
+ # Intermediate states are indexed by state_batch_coord (i.e., req index, not batch index)
+ int_states = torch.zeros(
+ num_states, seq_len, D, state_len, device="cuda", dtype=torch.float32
+ )
+
+ conv_state_copy = conv_state.clone()
+ result = causal_conv1d_update(
+ x,
+ conv_state_copy,
+ weight,
+ bias=None,
+ silu_activation=False,
+ conv_state_indices=state_indices,
+ intermediate_conv_states=int_states,
+ )
+
+ # The final intermediate state at last seq step should match the final conv_state
+ for b_idx in range(B):
+ req_idx = state_indices[b_idx].item()
+ torch.testing.assert_close(
+ int_states[req_idx, seq_len - 1, :, :],
+ conv_state_copy[req_idx, :, :],
+ atol=1e-5,
+ rtol=1e-5,
+ )
diff --git a/tests/unit_tests/ssm/test_mamba_block.py b/tests/unit_tests/ssm/test_mamba_block.py
index 909ee47e836..7b743afbfad 100755
--- a/tests/unit_tests/ssm/test_mamba_block.py
+++ b/tests/unit_tests/ssm/test_mamba_block.py
@@ -5,8 +5,9 @@
from megatron.core.models.mamba.mamba_layer_specs import mamba_stack_spec
from megatron.core.process_groups_config import ProcessGroupCollection
+from megatron.core.ssm.gated_delta_net import GatedDeltaNet
from megatron.core.ssm.mamba_block import MambaStack
-from megatron.core.ssm.mamba_hybrid_layer_allocation import Symbols
+from megatron.core.ssm.mamba_hybrid_layer_allocation import Symbols, validate_segment_layers
from megatron.core.ssm.mamba_layer import MambaLayer
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer import TransformerConfig
@@ -26,12 +27,13 @@ def setup_method(self, method):
def get_pg_collection(self):
return ProcessGroupCollection.use_mpu_process_groups(required_pgs=['tp', 'pp', 'cp'])
- def get_mamba_block(self, hybrid_override_pattern):
+ def get_mamba_block(self, layer_pattern):
+ layer_type_list = validate_segment_layers(layer_pattern)
transformer_config = TransformerConfig(
hidden_size=256, # The Mamba layer places several constraints on this
# Need to specify num_attention_heads and num_layers or TransformerConfig
# will generate errors.
- num_layers=len(hybrid_override_pattern),
+ num_layers=len(layer_type_list),
num_attention_heads=4,
use_cpu_initialization=True,
)
@@ -39,7 +41,8 @@ def get_mamba_block(self, hybrid_override_pattern):
return MambaStack(
transformer_config,
modules,
- hybrid_override_pattern=hybrid_override_pattern,
+ layer_type_list=layer_type_list,
+ pp_layer_offset=0,
pg_collection=self.get_pg_collection(),
)
@@ -48,8 +51,8 @@ def teardown_method(self, method):
def test_gpu_forward(self):
"""Test GPU forward pass."""
- hybrid_override_pattern = Symbols.MAMBA + Symbols.ATTENTION + Symbols.MLP
- block = self.get_mamba_block(hybrid_override_pattern)
+ layer_pattern = Symbols.MAMBA + Symbols.ATTENTION + Symbols.MLP
+ block = self.get_mamba_block(layer_pattern)
block.cuda()
micro_batch_size = 2
sequence_length = 32
@@ -67,13 +70,13 @@ def test_gpu_forward(self):
def test_layer_types(self):
"""
- Make sure that the layer types specified with hybrid_override_pattern
+ Make sure that the layer types specified with layer_pattern
were honored.
"""
- hybrid_override_pattern = Symbols.MAMBA + Symbols.ATTENTION + Symbols.MLP
- block = self.get_mamba_block(hybrid_override_pattern)
+ layer_pattern = Symbols.MAMBA + Symbols.ATTENTION + Symbols.MLP
+ block = self.get_mamba_block(layer_pattern)
layers = block.layers
- # Note that this matches the order specified by hybrid_override_pattern in setup_method
+ # Note that this matches the order specified by layer_pattern above
assert isinstance(layers[0], MambaLayer)
assert isinstance(layers[1], TransformerLayer)
assert isinstance(layers[1].self_attention, SelfAttention)
@@ -82,8 +85,56 @@ def test_layer_types(self):
def test_invalid_layer_types_cause_failure(self):
invalid_symbol = '+'
- assert invalid_symbol not in Symbols.VALID # sanity check.
- hybrid_override_pattern = Symbols.MAMBA + Symbols.ATTENTION + Symbols.MLP + invalid_symbol
- # _allocate_override() in mamba_hybrid_layer_allocation.py throws a ValueError.
+ assert invalid_symbol not in Symbols.VALID_LAYERS # sanity check.
+ layer_pattern = Symbols.MAMBA + Symbols.ATTENTION + Symbols.MLP + invalid_symbol
+ # validate_segment_layers() in mamba_hybrid_layer_allocation.py throws a ValueError.
with pytest.raises(ValueError):
- block = self.get_mamba_block(hybrid_override_pattern)
+ block = self.get_mamba_block(layer_pattern)
+
+ def test_gdn_layer_types(self):
+ """
+ Make sure that G creates a TransformerLayer wrapping GatedDeltaNet,
+ while * creates a TransformerLayer wrapping SelfAttention.
+ """
+ layer_pattern = Symbols.GDN + Symbols.ATTENTION + Symbols.MAMBA
+ block = self.get_mamba_block(layer_pattern)
+ layers = block.layers
+ assert isinstance(layers[0], TransformerLayer)
+ assert isinstance(layers[0].self_attention, GatedDeltaNet)
+ assert isinstance(layers[1], TransformerLayer)
+ assert isinstance(layers[1].self_attention, SelfAttention)
+ assert isinstance(layers[2], MambaLayer)
+
+ def test_gdn_gpu_forward(self):
+ """Test GPU forward pass with GDN, attention, and Mamba layers."""
+ layer_pattern = Symbols.GDN + Symbols.ATTENTION + Symbols.MAMBA
+ layer_type_list = validate_segment_layers(layer_pattern)
+ transformer_config = TransformerConfig(
+ hidden_size=256,
+ num_layers=len(layer_type_list),
+ num_attention_heads=4,
+ use_cpu_initialization=True,
+ activation_func=torch.nn.functional.silu,
+ )
+ modules = mamba_stack_spec.submodules
+ block = MambaStack(
+ transformer_config,
+ modules,
+ layer_type_list=layer_type_list,
+ pp_layer_offset=0,
+ pg_collection=self.get_pg_collection(),
+ )
+ block.cuda()
+ micro_batch_size = 2
+ sequence_length = 32
+ hidden_states = torch.ones((sequence_length, micro_batch_size, block.config.hidden_size))
+ hidden_states = hidden_states.cuda()
+ attention_mask = torch.ones(
+ (micro_batch_size, 1, sequence_length, sequence_length), dtype=bool
+ )
+ attention_mask = attention_mask.cuda()
+ output = block(hidden_states, attention_mask=attention_mask)
+ assert output.shape[0] == sequence_length
+ assert output.shape[1] == micro_batch_size
+ assert output.shape[2] == block.config.hidden_size
+ assert output.dtype == torch.float32
diff --git a/tests/unit_tests/ssm/test_mamba_context_parallel.py b/tests/unit_tests/ssm/test_mamba_context_parallel.py
index 59cd1fb061c..c232aba9655 100644
--- a/tests/unit_tests/ssm/test_mamba_context_parallel.py
+++ b/tests/unit_tests/ssm/test_mamba_context_parallel.py
@@ -1,7 +1,5 @@
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
-import math
-
import pytest
import torch
import torch.nn as nn
diff --git a/tests/unit_tests/ssm/test_mamba_hybrid_layer_allocation.py b/tests/unit_tests/ssm/test_mamba_hybrid_layer_allocation.py
index 77c106c3bee..440c843bc27 100644
--- a/tests/unit_tests/ssm/test_mamba_hybrid_layer_allocation.py
+++ b/tests/unit_tests/ssm/test_mamba_hybrid_layer_allocation.py
@@ -1,85 +1,137 @@
# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
-import math
-import re
+from unittest.mock import patch
import pytest
-import torch
from megatron.core.ssm.mamba_hybrid_layer_allocation import (
ParsedHybridPattern,
Symbols,
- allocate_layers,
+ get_hybrid_layer_counts,
+ get_hybrid_total_layer_count,
+ get_hybrid_total_pipeline_segment_count,
parse_hybrid_pattern,
+ pattern_from_ratios,
+ select_pipeline_segment,
+ validate_segment_layers,
)
@pytest.mark.internal
-class TestMambaHybridLayerAllocation:
+class TestPatternFromRatios:
- def test_hybrid_layer_allocation(self):
- # The format for the test cases is:
- # (layers_count, attention_ratio, mlp_ratio, override_pattern).
+ def test_pure_mamba(self):
+ result = pattern_from_ratios(8, attention_ratio=0.0, mlp_ratio=0.0)
+ assert result == "MMMMMMMM"
+
+ def test_attention_only(self):
+ result = pattern_from_ratios(10, attention_ratio=0.3)
+ assert result.count(Symbols.ATTENTION) == 3
+ assert result.count(Symbols.MAMBA) == 7
+ assert len(result) == 10
+
+ def test_attention_and_mlp(self):
+ result = pattern_from_ratios(10, attention_ratio=0.3, mlp_ratio=0.3)
+ assert result.count(Symbols.ATTENTION) == 3
+ assert result.count(Symbols.MLP) == 3
+ assert result.count(Symbols.MAMBA) == 4
+ assert len(result) == 10
+
+ def test_attention_evenly_spaced(self):
+ result = pattern_from_ratios(10, attention_ratio=0.5)
+ assert result.count(Symbols.ATTENTION) == 5
+ assert result.count(Symbols.MAMBA) == 5
+ attn_positions = [i for i, ch in enumerate(result) if ch == Symbols.ATTENTION]
+ gaps = [attn_positions[i + 1] - attn_positions[i] for i in range(len(attn_positions) - 1)]
+ assert all(
+ g in (1, 2, 3) for g in gaps
+ ), f"Gaps between attention layers should be small, got {gaps}"
+
+ def test_mlp_does_not_replace_attention(self):
+ result = pattern_from_ratios(10, attention_ratio=0.3, mlp_ratio=0.3)
+ attn_positions = [i for i, c in enumerate(result) if c == Symbols.ATTENTION]
+ mlp_positions = [i for i, c in enumerate(result) if c == Symbols.MLP]
+ assert not set(attn_positions) & set(mlp_positions)
+
+ def test_single_layer(self):
+ assert pattern_from_ratios(1, 0.0, 0.0) == "M"
+ assert pattern_from_ratios(1, 1.0, 0.0) == "*"
+
+ def test_returns_string(self):
+ result = pattern_from_ratios(4, 0.5)
+ assert isinstance(result, str)
+
+
+@pytest.mark.internal
+class TestValidateSegmentLayers:
+
+ def test_valid_patterns(self):
+ """Test that valid segment patterns produce the correct layer type lists."""
test_cases = [
- (9, 0.0, 0.0, "M*-M*-M*-"),
- (9, 0.0, 0.0, "MMMMMMMMM"),
- (30, 0.0, 0.0, None),
- (8, 0.25, 0.25, "MM*-MM*-"),
- (8, 0.5, 0.25, "M**-M**-"),
- (48, 0.5, 0.2, None),
+ ("M*-M*-M*-", ['M', '*', '-', 'M', '*', '-', 'M', '*', '-']),
+ ("MMMMMMMMM", ['M'] * 9),
+ ("MM*-MM*-", ['M', 'M', '*', '-', 'M', 'M', '*', '-']),
+ ("E", ['E']),
+ ("", []),
+ ("GGG*GGG*", ['G', 'G', 'G', '*', 'G', 'G', 'G', '*']),
+ ("GEGEGE*E", ['G', 'E', 'G', 'E', 'G', 'E', '*', 'E']),
]
- for test in test_cases:
- (layers_count, attention_ratio, mlp_ratio, override_pattern) = test
+ for pattern, expected in test_cases:
+ result = validate_segment_layers(pattern)
+ assert result == expected, f"Failed for pattern: {pattern}"
- layer_types = allocate_layers(*test)
+ def test_all_valid_symbols(self):
+ """Make sure all returned layers are valid."""
+ for pattern in ["M*-M*-M*-", "MMMMMMMMM", "MM*-", "MEME"]:
+ layer_types = validate_segment_layers(pattern)
+ for layer_type in layer_types:
+ assert layer_type in Symbols.VALID_LAYERS
- # Check that return value is in the right format.
- assert isinstance(layer_types, list)
- assert layers_count == len(layer_types)
+ def test_invalid_symbols_cause_failure(self):
+ """Test that invalid symbols raise ValueError."""
+ with pytest.raises(ValueError):
+ validate_segment_layers("M*X")
+ with pytest.raises(ValueError):
+ validate_segment_layers("M|M") # pipe not valid in a segment
+ with pytest.raises(ValueError):
+ validate_segment_layers("M/M") # MTP separator not valid in a segment
- # Make sure all the layers are valid.
- for layer_type in layer_types:
- assert layer_type in Symbols.VALID
-
- # Make sure each layer is as requested by override_pattern.
- if override_pattern is not None:
- assert len(override_pattern) == len(layer_types)
- for index, layer_type in enumerate(layer_types):
- assert override_pattern[index] == layer_types[index]
- else:
- # Make sure the count of each type of layer is correct.
- counts = {layer_type: 0 for layer_type in Symbols.VALID} # Initialize all to zero.
- for layer_type in layer_types:
- assert layer_type in counts
- counts[layer_type] += 1
- # Check the ratios.
- remainder = 1.0 - attention_ratio - mlp_ratio
- assert remainder >= 0
- assert int(attention_ratio * layers_count + 0.5) == counts[Symbols.ATTENTION]
- assert int(mlp_ratio * layers_count + 0.5) == counts[Symbols.MLP]
- assert int(remainder * layers_count + 0.5) == counts[Symbols.MAMBA]
-
- # Make sure the ratios are as requested.
- # This code is not working yet because capsys seems broken in Megatron.
- # captured = capsys.readouterr() # Remove this output from the capture buffer.
- # out = captured.out # Get stdout.
- # if attention_ratio != 0 or mlp_ratio != 0:
- # assert (
- # match := re.search(r'Actual attention ratio: (1\.0|0\.[0-9]+)\.', out)
- # ) and math.isclose(match.group(1), attention_ratio)
- # assert (
- # match := re.search(r'Actual mlp ratio: (1\.0|0\.[0-9]+)\.', out)
- # ) and math.isclose(match.group(1), mlp_ratio)
-
- @pytest.mark.xfail(raises=ValueError)
- def test_wrong_length_override_pattern(self):
- # This override_pattern is too short.
- layer_types = allocate_layers(9, 0.0, 0.0, "M*-M*-")
-
- @pytest.mark.xfail(raises=ValueError)
- def test_wrong_number_of_layer_types_in_override_pattern(self):
- # This override_pattern has too many mlps and not enough attention
- layer_types = allocate_layers(8, 0.5, 0.25, "M*--M**-")
+
+@pytest.mark.internal
+class TestGetHybridTotalLayerCount:
+
+ def test_simple_patterns(self):
+ assert get_hybrid_total_layer_count("M*M*") == 4
+ assert get_hybrid_total_layer_count("MMMM") == 4
+ assert get_hybrid_total_layer_count("M") == 1
+
+ def test_with_pipe_separators(self):
+ assert get_hybrid_total_layer_count("M-M-|M-M*-") == 9
+ assert get_hybrid_total_layer_count("M-M-|M-M*-|M-M-|M-M*-") == 18
+ assert get_hybrid_total_layer_count("||M") == 1
+ assert get_hybrid_total_layer_count("M|M") == 2
+
+ def test_with_mtp(self):
+ assert get_hybrid_total_layer_count("M*M*/MM/MM") == 4
+ assert get_hybrid_total_layer_count("M-M-|M-M*-/MM/MM") == 9
+
+ def test_empty(self):
+ assert get_hybrid_total_layer_count("") == 0
+
+
+@pytest.mark.internal
+class TestGetHybridTotalPipelineSegmentCount:
+
+ def test_no_pipe(self):
+ assert get_hybrid_total_pipeline_segment_count("M*M*") == 1
+
+ def test_with_pipes(self):
+ assert get_hybrid_total_pipeline_segment_count("M-M-|M-M*-") == 2
+ assert get_hybrid_total_pipeline_segment_count("M|M|M|M") == 4
+ assert get_hybrid_total_pipeline_segment_count("||M") == 3
+
+ def test_with_mtp(self):
+ assert get_hybrid_total_pipeline_segment_count("M-M-|M-M*-/MM/MM") == 2
@pytest.mark.internal
@@ -101,6 +153,8 @@ def test_main_pattern_only(self):
("*M*M", "*M*M"),
("MM-*", "MM-*"),
("E", "E"),
+ ("GGG*GGG*", "GGG*GGG*"),
+ ("GEGEGE*E", "GEGEGE*E"),
]
for pattern, expected_main in test_cases:
result = parse_hybrid_pattern(pattern)
@@ -108,6 +162,15 @@ def test_main_pattern_only(self):
assert result.mtp_pattern is None
assert result.mtp_num_depths == 0
+ def test_main_pattern_with_pipes(self):
+ """Test patterns with pipe separators (no MTP)."""
+ test_cases = [("M*|M*", "M*|M*"), ("M-M-|M-M*-", "M-M-|M-M*-"), ("M|M|M|M", "M|M|M|M")]
+ for pattern, expected_main in test_cases:
+ result = parse_hybrid_pattern(pattern)
+ assert result.main_pattern == expected_main, f"Failed for pattern: {pattern}"
+ assert result.mtp_pattern is None
+ assert result.mtp_num_depths == 0
+
def test_main_with_single_mtp_depth(self):
"""Test patterns with 1 MTP depth."""
test_cases = [
@@ -136,6 +199,13 @@ def test_main_with_multiple_mtp_depths(self):
assert result.mtp_pattern == expected_mtp, f"Failed for pattern: {pattern}"
assert result.mtp_num_depths == expected_depths, f"Failed for pattern: {pattern}"
+ def test_pipe_with_mtp(self):
+ """Test patterns with both pipe and MTP separators."""
+ result = parse_hybrid_pattern("M-M-|M-M*-/MM/MM")
+ assert result.main_pattern == "M-M-|M-M*-"
+ assert result.mtp_pattern == "MM"
+ assert result.mtp_num_depths == 2
+
def test_mtp_patterns_must_be_identical(self):
"""Test that mismatched MTP patterns raise ValueError."""
invalid_patterns = [
@@ -173,6 +243,11 @@ def test_invalid_symbols_in_mtp_pattern(self):
with pytest.raises(ValueError, match="All MTP patterns must be identical"):
parse_hybrid_pattern("M*M*/MM/Ma")
+ def test_pipe_not_allowed_in_mtp(self):
+ """Test that pipe symbol in MTP pattern raises ValueError."""
+ with pytest.raises(ValueError, match="not a valid layer symbol"):
+ parse_hybrid_pattern("M*M*/M|M/M|M")
+
def test_empty_main_pattern_with_mtp(self):
"""Test pattern that starts with / (empty main pattern)."""
result = parse_hybrid_pattern("/MM/MM")
@@ -200,6 +275,8 @@ def test_complex_patterns(self):
("*****/M/M/M/M", "*****", "M", 4),
# MoE in main pattern
("MEME/MM/MM", "MEME", "MM", 2),
+ # GDN+MoE main pattern with GDN MTP
+ ("GEGEGE*E/GG/GG", "GEGEGE*E", "GG", 2),
]
for pattern, expected_main, expected_mtp, expected_depths in test_cases:
result = parse_hybrid_pattern(pattern)
@@ -212,3 +289,310 @@ def test_dataclass_equality(self):
p1 = parse_hybrid_pattern("M*M*/MM/MM")
p2 = ParsedHybridPattern(main_pattern="M*M*", mtp_pattern="MM", mtp_num_depths=2)
assert p1 == p2
+
+
+@pytest.mark.internal
+class TestGetHybridLayerCounts:
+
+ def test_simple_pattern(self):
+ assert get_hybrid_layer_counts("M*M*") == {'M': 2, 'G': 0, '*': 2, '-': 0, 'E': 0}
+
+ def test_all_layer_types(self):
+ assert get_hybrid_layer_counts("MG*-E") == {'M': 1, 'G': 1, '*': 1, '-': 1, 'E': 1}
+
+ def test_with_pipes(self):
+ # Pipes should be skipped in counting
+ assert get_hybrid_layer_counts("M*|M*") == {'M': 2, 'G': 0, '*': 2, '-': 0, 'E': 0}
+ assert get_hybrid_layer_counts("M-M-|M-M*-") == {'M': 4, 'G': 0, '*': 1, '-': 4, 'E': 0}
+
+ def test_with_mtp(self):
+ # MTP pattern "MM" repeated 2 depths -> 4 extra mamba layers
+ assert get_hybrid_layer_counts("M*M*/MM/MM") == {'M': 6, 'G': 0, '*': 2, '-': 0, 'E': 0}
+
+ def test_with_pipes_and_mtp(self):
+ # Main: M-M-|M-M*- -> 1 attn, 4 mamba, 4 mlp
+ # MTP: MM x 2 depths -> +4 mamba
+ assert get_hybrid_layer_counts("M-M-|M-M*-/MM/MM") == {
+ 'M': 8,
+ 'G': 0,
+ '*': 1,
+ '-': 4,
+ 'E': 0,
+ }
+
+ def test_moe_pattern(self):
+ assert get_hybrid_layer_counts("MEME") == {'M': 2, 'G': 0, '*': 0, '-': 0, 'E': 2}
+
+ def test_mtp_with_attention(self):
+ # MTP pattern "*M" repeated 3 depths -> 3 attn + 3 mamba from MTP
+ assert get_hybrid_layer_counts("MMMM/*M/*M/*M") == {'M': 7, 'G': 0, '*': 3, '-': 0, 'E': 0}
+
+ def test_empty_pattern(self):
+ assert get_hybrid_layer_counts("") == {'M': 0, 'G': 0, '*': 0, '-': 0, 'E': 0}
+
+ def test_gdn_pattern(self):
+ assert get_hybrid_layer_counts("GMGM") == {'M': 2, 'G': 2, '*': 0, '-': 0, 'E': 0}
+
+ def test_gdn_hybrid_pattern(self):
+ # GDN + Mamba + Attention
+ assert get_hybrid_layer_counts("G*GM*") == {'M': 1, 'G': 2, '*': 2, '-': 0, 'E': 0}
+
+
+@pytest.mark.internal
+class TestSelectPipelineSegment:
+ """Tests for select_pipeline_segment with pp_group=None (single rank).
+
+ When pp_group is None, pp_rank=0 and pp_size=1, so the segment index
+ is simply the vp_stage value.
+ """
+
+ @patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage')
+ def test_single_segment_no_vp(self, mock_log):
+ """Single segment, no VPP."""
+ layer_types, offset = select_pipeline_segment("M*M*", pp_group=None, vp_stage=None)
+ assert layer_types == ['M', '*', 'M', '*']
+ assert offset == 0
+
+ @patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage')
+ def test_two_segments_vp0(self, mock_log):
+ """Two segments, select first (vp_stage=0)."""
+ layer_types, offset = select_pipeline_segment("M-M-|M-M*-", pp_group=None, vp_stage=0)
+ assert layer_types == ['M', '-', 'M', '-']
+ assert offset == 0
+
+ @patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage')
+ def test_two_segments_vp1(self, mock_log):
+ """Two segments, select second (vp_stage=1)."""
+ layer_types, offset = select_pipeline_segment("M-M-|M-M*-", pp_group=None, vp_stage=1)
+ assert layer_types == ['M', '-', 'M', '*', '-']
+ assert offset == 4
+
+ @patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage')
+ def test_four_segments(self, mock_log):
+ """Four segments, verify each vp_stage selects correctly."""
+ pattern = "MM|M*|M-|ME"
+ expected = [(['M', 'M'], 0), (['M', '*'], 2), (['M', '-'], 4), (['M', 'E'], 6)]
+ for vp_stage, (expected_layers, expected_offset) in enumerate(expected):
+ layer_types, offset = select_pipeline_segment(pattern, pp_group=None, vp_stage=vp_stage)
+ assert layer_types == expected_layers, f"Failed for vp_stage={vp_stage}"
+ assert offset == expected_offset, f"Failed for vp_stage={vp_stage}"
+
+ @patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage')
+ def test_empty_segment(self, mock_log):
+ """Empty segments are allowed for pipeline balancing."""
+ layer_types, offset = select_pipeline_segment("||M*", pp_group=None, vp_stage=0)
+ assert layer_types == []
+ assert offset == 0
+
+ layer_types, offset = select_pipeline_segment("||M*", pp_group=None, vp_stage=2)
+ assert layer_types == ['M', '*']
+ assert offset == 0
+
+ @patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage')
+ def test_uneven_segments(self, mock_log):
+ """Segments of different lengths."""
+ pattern = "MMM|M|MMMMM"
+ layer_types, offset = select_pipeline_segment(pattern, pp_group=None, vp_stage=0)
+ assert len(layer_types) == 3
+ assert offset == 0
+
+ layer_types, offset = select_pipeline_segment(pattern, pp_group=None, vp_stage=1)
+ assert len(layer_types) == 1
+ assert offset == 3
+
+ layer_types, offset = select_pipeline_segment(pattern, pp_group=None, vp_stage=2)
+ assert len(layer_types) == 5
+ assert offset == 4
+
+ @patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage')
+ def test_empty_main_pattern(self, mock_log):
+ """Empty main pattern produces one empty segment."""
+ layer_types, offset = select_pipeline_segment("", pp_group=None, vp_stage=None)
+ assert layer_types == []
+ assert offset == 0
+
+ @patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage')
+ def test_invalid_segment_raises(self, mock_log):
+ """Invalid layer symbols in a segment should raise ValueError."""
+ with pytest.raises(ValueError):
+ select_pipeline_segment("MX|M*", pp_group=None, vp_stage=0)
+
+ @patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage')
+ def test_out_of_range_segment_raises(self, mock_log):
+ """Segment index out of range should raise ValueError."""
+ with pytest.raises(ValueError, match="out of range"):
+ select_pipeline_segment("M*|M*", pp_group=None, vp_stage=5)
+
+ @patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage')
+ def test_logging_is_called(self, mock_log):
+ """Verify that log_on_each_pipeline_stage is called."""
+ select_pipeline_segment("M*M*", pp_group=None, vp_stage=None)
+ mock_log.assert_called_once()
+
+ @patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage')
+ def test_mutual_exclusivity_pipes_with_first_stage(self, mock_log):
+ """Pipe separators + first_stage_layers should raise ValueError."""
+ with pytest.raises(ValueError, match="Cannot specify"):
+ select_pipeline_segment("M*|M*", pp_group=None, vp_stage=0, first_stage_layers=1)
+
+ @patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage')
+ def test_mutual_exclusivity_pipes_with_last_stage(self, mock_log):
+ """Pipe separators + last_stage_layers should raise ValueError."""
+ with pytest.raises(ValueError, match="Cannot specify"):
+ select_pipeline_segment("M*|M*", pp_group=None, vp_stage=0, last_stage_layers=1)
+
+ @patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage')
+ def test_segment_count_not_divisible_by_pp_size(self, mock_log):
+ """Segment count not divisible by pp_size should raise ValueError."""
+ mock_group = object()
+ with (
+ patch('torch.distributed.get_rank', return_value=0),
+ patch('torch.distributed.get_world_size', return_value=2),
+ ):
+ with pytest.raises(ValueError, match="evenly divisible"):
+ select_pipeline_segment("M|M|M", pp_group=mock_group, vp_stage=None)
+
+
+@pytest.mark.internal
+class TestSelectPipelineSegmentLegacyFallback:
+ """Tests for the no-pipes fallback path in select_pipeline_segment.
+
+ These tests exercise the backwards-compatible auto-split logic that
+ activates when the pattern has no pipe separators but pp_size > 1.
+ """
+
+ def _call_for_rank(
+ self,
+ pattern,
+ pp_rank,
+ pp_size,
+ vp_stage=None,
+ first_stage_layers=None,
+ last_stage_layers=None,
+ ):
+ """Call select_pipeline_segment with mocked PP group for a given rank."""
+ mock_group = object()
+ with (
+ patch('torch.distributed.get_rank', return_value=pp_rank),
+ patch('torch.distributed.get_world_size', return_value=pp_size),
+ patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage'),
+ patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_single_rank'),
+ ):
+ return select_pipeline_segment(
+ pattern,
+ pp_group=mock_group,
+ vp_stage=vp_stage,
+ first_stage_layers=first_stage_layers,
+ last_stage_layers=last_stage_layers,
+ )
+
+ def test_even_split_2_ranks(self):
+ """4 layers across 2 ranks -> 2 each."""
+ layers0, off0 = self._call_for_rank("M*M-", pp_rank=0, pp_size=2)
+ assert layers0 == ['M', '*']
+ assert off0 == 0
+
+ layers1, off1 = self._call_for_rank("M*M-", pp_rank=1, pp_size=2)
+ assert layers1 == ['M', '-']
+ assert off1 == 2
+
+ def test_even_split_4_ranks(self):
+ """8 layers across 4 ranks -> 2 each."""
+ pattern = "M*M*M*M*"
+ for rank in range(4):
+ layers, offset = self._call_for_rank(pattern, pp_rank=rank, pp_size=4)
+ assert len(layers) == 2
+ assert offset == rank * 2
+
+ def test_even_split_not_divisible_raises(self):
+ """6 layers across 4 ranks with no uneven PP -> ValueError."""
+ with pytest.raises(ValueError, match="evenly divisible"):
+ self._call_for_rank("M*M*M*", pp_rank=0, pp_size=4)
+
+ def test_uneven_pp_first_stage(self):
+ """6 layers, pp_size=4, first_stage=3 -> first gets 3, others get 1."""
+ pattern = "M*M*M*"
+ layers0, off0 = self._call_for_rank(pattern, pp_rank=0, pp_size=4, first_stage_layers=3)
+ assert len(layers0) == 3
+ assert off0 == 0
+
+ layers1, off1 = self._call_for_rank(pattern, pp_rank=1, pp_size=4, first_stage_layers=3)
+ assert len(layers1) == 1
+ assert off1 == 3
+
+ layers3, off3 = self._call_for_rank(pattern, pp_rank=3, pp_size=4, first_stage_layers=3)
+ assert len(layers3) == 1
+ assert off3 == 5
+
+ def test_uneven_pp_last_stage(self):
+ """6 layers, pp_size=4, last_stage=3 -> last gets 3, others get 1."""
+ pattern = "M*M*M*"
+ layers0, off0 = self._call_for_rank(pattern, pp_rank=0, pp_size=4, last_stage_layers=3)
+ assert len(layers0) == 1
+ assert off0 == 0
+
+ layers3, off3 = self._call_for_rank(pattern, pp_rank=3, pp_size=4, last_stage_layers=3)
+ assert len(layers3) == 3
+ assert off3 == 3
+
+ def test_uneven_pp_first_and_last(self):
+ """8 layers, pp_size=4, first=1, last=1 -> first 1, middle 3 each, last 1."""
+ pattern = "M*M*M*M*"
+ layers0, off0 = self._call_for_rank(
+ pattern, pp_rank=0, pp_size=4, first_stage_layers=1, last_stage_layers=1
+ )
+ assert len(layers0) == 1
+ assert off0 == 0
+
+ layers1, off1 = self._call_for_rank(
+ pattern, pp_rank=1, pp_size=4, first_stage_layers=1, last_stage_layers=1
+ )
+ assert len(layers1) == 3
+ assert off1 == 1
+
+ layers2, off2 = self._call_for_rank(
+ pattern, pp_rank=2, pp_size=4, first_stage_layers=1, last_stage_layers=1
+ )
+ assert len(layers2) == 3
+ assert off2 == 4
+
+ layers3, off3 = self._call_for_rank(
+ pattern, pp_rank=3, pp_size=4, first_stage_layers=1, last_stage_layers=1
+ )
+ assert len(layers3) == 1
+ assert off3 == 7
+
+ def test_uneven_pp_middle_not_divisible_raises(self):
+ """Middle layers not divisible by middle stages -> ValueError."""
+ with pytest.raises(ValueError, match="Middle layers"):
+ self._call_for_rank("M*M*M*M*M", pp_rank=0, pp_size=4, first_stage_layers=2)
+
+ def test_vpp_with_no_pipes_raises(self):
+ """VPP (vp_stage != None) without pipe separators -> ValueError."""
+ with pytest.raises(ValueError, match="Virtual pipeline parallelism"):
+ self._call_for_rank("M*M*", pp_rank=0, pp_size=2, vp_stage=0)
+
+ def test_deprecation_warning_logged(self):
+ """Legacy path should log a deprecation warning via log_single_rank."""
+ mock_group = object()
+ with (
+ patch('torch.distributed.get_rank', return_value=0),
+ patch('torch.distributed.get_world_size', return_value=2),
+ patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_on_each_pipeline_stage'),
+ patch('megatron.core.ssm.mamba_hybrid_layer_allocation.log_single_rank') as mock_warn,
+ ):
+ select_pipeline_segment("M*M*", pp_group=mock_group, vp_stage=None)
+ mock_warn.assert_called_once()
+ call_args = mock_warn.call_args
+ assert "DEPRECATION" in call_args[0][2]
+
+ def test_all_ranks_cover_full_pattern(self):
+ """All ranks together should reconstruct the original layer list."""
+ pattern = "M*M*M*"
+ pp_size = 3
+ all_layers = []
+ for rank in range(pp_size):
+ layers, offset = self._call_for_rank(pattern, pp_rank=rank, pp_size=pp_size)
+ assert offset == len(all_layers)
+ all_layers.extend(layers)
+ assert all_layers == ['M', '*', 'M', '*', 'M', '*']
diff --git a/tests/unit_tests/ssm/test_mamba_layer.py b/tests/unit_tests/ssm/test_mamba_layer.py
index 25d24443d38..26c7ba4d92f 100755
--- a/tests/unit_tests/ssm/test_mamba_layer.py
+++ b/tests/unit_tests/ssm/test_mamba_layer.py
@@ -5,7 +5,8 @@
from megatron.core.models.mamba.mamba_layer_specs import mamba_stack_spec
from megatron.core.process_groups_config import ProcessGroupCollection
-from megatron.core.ssm.mamba_layer import MambaLayer
+from megatron.core.ssm.mamba_block import MambaStackSubmodules
+from megatron.core.ssm.mamba_layer import MambaLayer, MambaLayerSubmodules
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer import TransformerConfig
from tests.unit_tests.test_utilities import Utils
@@ -25,9 +26,14 @@ def setup_method(self, method):
num_attention_heads=1,
use_cpu_initialization=True,
)
- modules = mamba_stack_spec.submodules.mamba_layer.submodules
+ assert isinstance(mamba_stack_spec.submodules, MambaStackSubmodules)
+ assert isinstance(mamba_stack_spec.submodules.mamba_layer.submodules, MambaLayerSubmodules)
pg_collection = ProcessGroupCollection.use_mpu_process_groups(required_pgs=['tp', 'cp'])
- self.layer = MambaLayer(transformer_config, modules, pg_collection=pg_collection)
+ self.layer = MambaLayer(
+ transformer_config,
+ mamba_stack_spec.submodules.mamba_layer.submodules,
+ pg_collection=pg_collection,
+ )
def teardown_method(self, method):
Utils.destroy_model_parallel()
diff --git a/tests/unit_tests/ssm/test_mamba_mixer.py b/tests/unit_tests/ssm/test_mamba_mixer.py
index 587cfc66967..f32b62146f8 100755
--- a/tests/unit_tests/ssm/test_mamba_mixer.py
+++ b/tests/unit_tests/ssm/test_mamba_mixer.py
@@ -6,7 +6,9 @@
from megatron.core.inference.contexts.static_context import StaticInferenceContext
from megatron.core.models.mamba.mamba_layer_specs import mamba_stack_spec
from megatron.core.process_groups_config import ProcessGroupCollection
-from megatron.core.ssm.mamba_mixer import MambaMixer
+from megatron.core.ssm.mamba_block import MambaStackSubmodules
+from megatron.core.ssm.mamba_layer import MambaLayerSubmodules
+from megatron.core.ssm.mamba_mixer import MambaMixer, MambaMixerSubmodules
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer import TransformerConfig
from tests.unit_tests.test_utilities import Utils
@@ -35,15 +37,20 @@ def get_mixer(self, tp_size=1, cp_size=1, use_mem_eff_path=True):
num_layers=1,
num_attention_heads=1,
use_cpu_initialization=True,
+ use_mamba_mem_eff_path=use_mem_eff_path,
+ )
+ assert isinstance(mamba_stack_spec.submodules, MambaStackSubmodules)
+ assert isinstance(mamba_stack_spec.submodules.mamba_layer.submodules, MambaLayerSubmodules)
+ assert isinstance(
+ mamba_stack_spec.submodules.mamba_layer.submodules.mixer.submodules,
+ MambaMixerSubmodules,
)
- modules = mamba_stack_spec.submodules.mamba_layer.submodules.mixer.submodules
pg_collection = ProcessGroupCollection.use_mpu_process_groups(required_pgs=['tp', 'cp'])
mixer = MambaMixer(
transformer_config,
- modules,
+ mamba_stack_spec.submodules.mamba_layer.submodules.mixer.submodules,
transformer_config.hidden_size,
layer_number=1,
- use_mem_eff_path=use_mem_eff_path,
pg_collection=pg_collection,
)
mixer.cuda()
@@ -119,12 +126,17 @@ def test_error_check(self, hidden_size, ngroups, tp_size, expected_error_message
use_cpu_initialization=True,
mamba_num_groups=ngroups,
)
- submodules = mamba_stack_spec.submodules.mamba_layer.submodules.mixer.submodules
+ assert isinstance(mamba_stack_spec.submodules, MambaStackSubmodules)
+ assert isinstance(mamba_stack_spec.submodules.mamba_layer.submodules, MambaLayerSubmodules)
+ assert isinstance(
+ mamba_stack_spec.submodules.mamba_layer.submodules.mixer.submodules,
+ MambaMixerSubmodules,
+ )
pg_collection = ProcessGroupCollection.use_mpu_process_groups(required_pgs=['tp', 'cp'])
with pytest.raises(AssertionError, match=expected_error_message):
MambaMixer(
transformer_config,
- submodules,
+ mamba_stack_spec.submodules.mamba_layer.submodules.mixer.submodules,
transformer_config.hidden_size,
pg_collection=pg_collection,
)
diff --git a/tests/unit_tests/tensor_parallel/test_random.py b/tests/unit_tests/tensor_parallel/test_random.py
index a15ad83cb90..4fa79733d55 100644
--- a/tests/unit_tests/tensor_parallel/test_random.py
+++ b/tests/unit_tests/tensor_parallel/test_random.py
@@ -227,4 +227,62 @@ def checkpoint_forward(input):
output2.backward(torch.ones((4, 4)), retain_graph=True)
assert torch.equal(input1.grad, input2.grad)
- Utils.destroy_model_parallel()
+
+class _ViewSavingLinear(torch.autograd.Function):
+ """Saves view tensors in forward to mimic TE GroupedLinear-style backward inputs."""
+
+ @staticmethod
+ def forward(ctx, inp, weight):
+ inp_2d = inp.reshape(-1, inp.shape[-1])
+ inputmats = torch.tensor_split(inp_2d, 2, dim=0)
+ ctx.save_for_backward(*inputmats, weight)
+ ctx.input_shape = inp.shape
+ out_2d = inp_2d.matmul(weight.t())
+ return out_2d.reshape(*inp.shape[:-1], weight.shape[0])
+
+ @staticmethod
+ def backward(ctx, grad_output):
+ *inputmats, weight = ctx.saved_tensors
+ for inputmat in inputmats:
+ if inputmat.numel() > 0 and inputmat.untyped_storage().size() == 0:
+ raise RuntimeError("Saved view tensor points to an empty storage.")
+
+ inp_2d = torch.cat(inputmats, dim=0)
+ grad_output_2d = grad_output.reshape(-1, grad_output.shape[-1])
+ grad_input_2d = grad_output_2d.matmul(weight)
+ grad_weight = grad_output_2d.t().matmul(inp_2d)
+ grad_input = grad_input_2d.reshape(ctx.input_shape)
+ return grad_input, grad_weight
+
+
+def test_checkpoint_without_output_view_sharing_regression():
+ def normal_forward(input_, weight):
+ x = torch.nn.functional.gelu(input_)
+ return _ViewSavingLinear.apply(x, weight)
+
+ def checkpoint_forward(input_, weight):
+ checkpoint = CheckpointWithoutOutput()
+ x = checkpoint.checkpoint(torch.nn.functional.gelu, input_)
+ y = _ViewSavingLinear.apply(x, weight)
+ checkpoint.discard_output_and_register_recompute(y)
+ return y
+
+ Utils.initialize_model_parallel()
+ try:
+ input1 = torch.randn((3, 2, 8), requires_grad=True)
+ weight1 = torch.randn((6, 8), requires_grad=True)
+
+ input2 = input1.detach().clone().requires_grad_(True)
+ weight2 = weight1.detach().clone().requires_grad_(True)
+
+ output1 = normal_forward(input1, weight1)
+ output2 = checkpoint_forward(input2, weight2)
+ assert torch.allclose(output1, output2)
+
+ grad = torch.randn_like(output1)
+ output1.backward(grad, retain_graph=True)
+ output2.backward(grad, retain_graph=True)
+ assert torch.allclose(input1.grad, input2.grad)
+ assert torch.allclose(weight1.grad, weight2.grad)
+ finally:
+ Utils.destroy_model_parallel()
diff --git a/tests/unit_tests/test_checkpointing.py b/tests/unit_tests/test_checkpointing.py
index 7c41c99f5a9..cacd4f0ea63 100644
--- a/tests/unit_tests/test_checkpointing.py
+++ b/tests/unit_tests/test_checkpointing.py
@@ -103,6 +103,7 @@ def create_args():
args.non_persistent_save_interval = None
args.exit_on_missing_checkpoint = True
args.async_save = False
+ args.async_strategy = "mcore"
args.data_parallel_random_init = False
args.no_save_optim = False
args.no_save_rng = False
diff --git a/tests/unit_tests/test_fp8_param.py b/tests/unit_tests/test_fp8_param.py
index 399995c843b..5b97a5b4976 100644
--- a/tests/unit_tests/test_fp8_param.py
+++ b/tests/unit_tests/test_fp8_param.py
@@ -11,7 +11,7 @@
from megatron.core.distributed import DistributedDataParallel as DDP
from megatron.core.enums import ModelType
-from megatron.core.fp8_utils import is_float8tensor
+from megatron.core.fp8_utils import is_float8tensor, is_mxfp8tensor
from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
from megatron.core.models.gpt.gpt_model import GPTModel
from megatron.core.num_microbatches_calculator import destroy_num_microbatches_calculator
@@ -256,6 +256,28 @@ def _run_test_helper(
if fp8_param_gather:
assert num_fp8_params == 4 * fp8_layers
+ # Verify that bf16 params (embedding, LN, etc.) in the MXFP8 model are mapped
+ # to the param buffer (shared with grad buffer) rather than allocated separately.
+ if args.reuse_grad_buf_for_mxfp8_param_ag:
+ for buffer in gpt_model[0].buffers:
+ if buffer.param_data is None:
+ continue
+ buf_start = buffer.param_data.data_ptr()
+ buf_end = buf_start + buffer.param_data.numel() * buffer.param_data.element_size()
+ for param in buffer.param_to_bucket:
+ if is_mxfp8tensor(param):
+ # MXFP8 params keep their own quantized storage.
+ assert not (
+ buf_start <= param.data.data_ptr() < buf_end
+ ), "MXFP8 param should not be mapped to the param buffer"
+ else:
+ # BF16 params should be views into the param buffer
+ # (no double allocation).
+ assert buf_start <= param.data.data_ptr() < buf_end, (
+ "BF16 param should be a view into the param buffer "
+ "(no separate allocation)"
+ )
+
loss_list = []
for i in range(100):
diff --git a/tests/unit_tests/test_inference.py b/tests/unit_tests/test_inference.py
index 518aa7f4126..805f3dec541 100644
--- a/tests/unit_tests/test_inference.py
+++ b/tests/unit_tests/test_inference.py
@@ -1,3 +1,7 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2026, Advanced Micro Devices, Inc. All rights reserved.
+# Modified for portability across upstream and ROCm CI environments.
+
import argparse
import unittest.mock
@@ -9,15 +13,20 @@
pytest.importorskip("flask_restful")
from megatron.core.inference.text_generation_server import MegatronServer
-from megatron.training import tokenizer
+from megatron.core.tokenizers import MegatronTokenizer
from tests.unit_tests.inference.engines.test_static_engine import StaticInferenceEngineTestHarness
-from tests.unit_tests.test_tokenizer import GPT2_VOCAB_SIZE, gpt2_tiktok_vocab
+from tests.unit_tests.paths import unit_test_data_path
from tests.unit_tests.test_utilities import Utils
@pytest.fixture(scope="module")
-def gpt2_tiktoken_tokenizer(gpt2_tiktok_vocab):
- return tokenizer.build_tokenizer(gpt2_tiktok_vocab)
+def gpt2_tiktoken_tokenizer():
+ return MegatronTokenizer.from_pretrained(
+ tokenizer_path=str(unit_test_data_path("tokenizers", "tiktoken", "tiktoken.vocab.json")),
+ vocab_size=131072,
+ num_special_tokens=1000,
+ pattern="v1",
+ )
@pytest.fixture(scope="module")
diff --git a/tests/unit_tests/test_layer_wise_optimizer.py b/tests/unit_tests/test_layer_wise_optimizer.py
index 05ce26bcfa0..c484ca104ee 100644
--- a/tests/unit_tests/test_layer_wise_optimizer.py
+++ b/tests/unit_tests/test_layer_wise_optimizer.py
@@ -10,12 +10,13 @@
from megatron.core import parallel_state
from megatron.core.distributed import DistributedDataParallel, DistributedDataParallelConfig
-from megatron.core.optimizer import OptimizerConfig, get_megatron_optimizer
+from megatron.core.optimizer import OptimizerConfig
from megatron.core.optimizer.layer_wise_optimizer import LayerWiseDistributedOptimizer
-from megatron.core.optimizer.optimizer import Float16OptimizerWithFloat16Params, FP32Optimizer
+from megatron.core.optimizer.muon import get_megatron_muon_optimizer
+from megatron.core.optimizer.optimizer import Float16OptimizerWithFloat16Params
from megatron.core.process_groups_config import ProcessGroupCollection
from megatron.core.transformer import TransformerConfig
-from megatron.core.utils import get_pg_size
+from megatron.core.utils import get_pg_rank, get_pg_size
from tests.unit_tests.test_utilities import Utils
# Skip all tests in this file for LTS versions
@@ -88,8 +89,8 @@ def create_model_and_optimizer(
model_class: Model class to instantiate
clip_grad: Optional gradient clipping value
model_kwargs: Optional kwargs for model initialization
- use_layer_wise: If True, wrap optimizer in LayerWiseDistributedOptimizer;
- if False, use get_megatron_optimizer instead (for reference)
+ use_layer_wise: If True, use LayerWiseDistributedOptimizer via dist_muon;
+ if False, use standard muon ChainedOptimizer (for reference)
Returns:
tuple: (model, optimizer, pg_collection)
@@ -110,24 +111,99 @@ def create_model_and_optimizer(
model.broadcast_params()
optimizer_config = OptimizerConfig(
- optimizer='adam',
+ optimizer='muon',
lr=0.01,
weight_decay=0.01,
- bf16=not use_layer_wise,
+ bf16=True,
use_distributed_optimizer=False,
clip_grad=clip_grad,
+ muon_tp_mode="duplicated",
)
pg_collection = ProcessGroupCollection.use_mpu_process_groups()
pg_collection.dp_cp = parallel_state.get_data_parallel_group(with_context_parallel=True)
pg_collection.expt_dp = parallel_state.get_expert_data_parallel_group()
- optimizer = get_megatron_optimizer(optimizer_config, [model])
- if use_layer_wise:
- optimizer_config.bf16 = True
- optimizer = LayerWiseDistributedOptimizer(
- optimizer.chained_optimizers, optimizer_config, pg_collection
- )
+ optimizer = get_megatron_muon_optimizer(
+ config=optimizer_config,
+ model_chunks=[model],
+ use_gloo_process_groups=True,
+ layer_wise_distributed_optimizer=use_layer_wise,
+ pg_collection=pg_collection,
+ )
+ return model, optimizer, pg_collection
+
+ def create_model_and_optimizer_with_overlap_param_gather(
+ self,
+ model_class=SimpleModel,
+ clip_grad=1.0,
+ model_kwargs=None,
+ copy_from=None,
+ async_allgather=True,
+ grad_reduce_in_fp32=False,
+ bucket_size=None,
+ ):
+ """Create model, DDP wrapper, and optimizer with overlap-param-gather enabled.
+
+ This variant sets overlap_param_gather=True in DDP config and uses
+ get_megatron_muon_optimizer with layer_wise_distributed_optimizer=True,
+ enabling the bucket-based async param gather path.
+
+ Args:
+ model_class: Model class to instantiate
+ clip_grad: Optional gradient clipping value
+ model_kwargs: Optional kwargs for model initialization
+ copy_from: Optional DDP model to copy weights from
+ async_allgather: If True, defer param all-gather to bucket infrastructure
+ grad_reduce_in_fp32: If True, reduce grads in fp32 (regression test for dtype fix)
+ bucket_size: Maximum number of parameters per bucket (None = single bucket)
+
+ Returns:
+ tuple: (model, optimizer, pg_collection)
+ """
+ if model_kwargs is None:
+ model_kwargs = {}
+
+ model = model_class(**model_kwargs).bfloat16().cuda()
+ model.requires_grad_(True)
+
+ ddp_config = DistributedDataParallelConfig(
+ use_distributed_optimizer=False,
+ overlap_param_gather=True,
+ overlap_grad_reduce=True,
+ grad_reduce_in_fp32=grad_reduce_in_fp32,
+ bucket_size=bucket_size,
+ )
+ model = DistributedDataParallel(
+ TransformerConfig(num_attention_heads=1, num_layers=1), ddp_config, model
+ )
+ if copy_from:
+ model.module.load_state_dict(copy_from.module.state_dict())
+ else:
+ model.broadcast_params()
+
+ optimizer_config = OptimizerConfig(
+ optimizer='muon',
+ lr=0.01,
+ weight_decay=0.01,
+ bf16=True,
+ use_distributed_optimizer=False,
+ clip_grad=clip_grad,
+ overlap_param_gather=async_allgather,
+ muon_tp_mode="duplicated",
+ )
+
+ pg_collection = ProcessGroupCollection.use_mpu_process_groups()
+ pg_collection.dp_cp = parallel_state.get_data_parallel_group(with_context_parallel=True)
+ pg_collection.expt_dp = parallel_state.get_expert_data_parallel_group()
+
+ optimizer = get_megatron_muon_optimizer(
+ config=optimizer_config,
+ model_chunks=[model],
+ use_gloo_process_groups=True,
+ layer_wise_distributed_optimizer=True,
+ pg_collection=pg_collection,
+ )
return model, optimizer, pg_collection
def create_reference_model(self, model):
@@ -239,11 +315,19 @@ def test_sharded_state_dict(self):
# Test sharded_state_dict
sharded_state_dict = optimizer.sharded_state_dict(model_sharded_state_dict)
- # Verify the sharded_state_dict is not None and has expected structure
+ # Verify the sharded_state_dict is not None and has expected structure.
+ # With multiple chained optimizers (muon + adam), the top-level keys are
+ # integer indices; each sub-dict should contain an 'optimizer' key.
assert sharded_state_dict is not None, "Sharded state dict should not be None"
- assert (
- 'optimizer' in sharded_state_dict
- ), "Sharded state dict should contain 'optimizer' key"
+ if isinstance(sharded_state_dict, dict) and all(
+ isinstance(k, int) for k in sharded_state_dict.keys()
+ ):
+ for idx, sub_dict in sharded_state_dict.items():
+ assert 'optimizer' in sub_dict, f"Sub-dict {idx} should contain 'optimizer' key"
+ else:
+ assert (
+ 'optimizer' in sharded_state_dict
+ ), "Sharded state dict should contain 'optimizer' key"
# Verify that replica_id is set correctly (should be 0 for DP dimension)
from megatron.core.dist_checkpointing import ShardedTensor
@@ -261,42 +345,13 @@ def test_sharded_state_dict(self):
def test_multiple_optimizers(self):
"""Test LayerWiseDistributedOptimizer with multiple chained optimizers.
- This test properly tests allgather functionality with multiple ranks.
+ Uses get_megatron_muon_optimizer which produces multiple chained optimizers
+ (muon for 2D weights + adam for 1D biases). Tests allgather with multiple ranks.
"""
- model = SimpleModel().bfloat16().cuda()
- model.requires_grad_(True)
-
- ddp_config = DistributedDataParallelConfig(use_distributed_optimizer=False)
- model = DistributedDataParallel(
- TransformerConfig(num_attention_heads=1, num_layers=1), ddp_config, model
- )
-
- optimizer_config = OptimizerConfig(
- optimizer='adam', lr=0.01, bf16=True, use_distributed_optimizer=False
- )
-
- # Split parameters into two groups for testing multiple optimizers
- params = list(model.parameters())
- mid_point = len(params) // 2
- param_groups_1 = [{'params': params[:mid_point]}]
- param_groups_2 = [{'params': params[mid_point:]}]
-
- # Create two separate base optimizers
- base_optimizer_1 = torch.optim.Adam(param_groups_1, lr=optimizer_config.lr)
- base_optimizer_2 = torch.optim.Adam(param_groups_2, lr=optimizer_config.lr)
-
- wrapped_optimizer_1 = FP32Optimizer(base_optimizer_1, optimizer_config, None)
- wrapped_optimizer_2 = FP32Optimizer(base_optimizer_2, optimizer_config, None)
-
- pg_collection = ProcessGroupCollection.use_mpu_process_groups()
- pg_collection.dp_cp = parallel_state.get_data_parallel_group(with_context_parallel=True)
- pg_collection.expt_dp = parallel_state.get_expert_data_parallel_group()
-
- optimizer = LayerWiseDistributedOptimizer(
- [wrapped_optimizer_1, wrapped_optimizer_2], optimizer_config, pg_collection
- )
+ model, optimizer, pg_collection = self.create_model_and_optimizer()
- assert len(optimizer.chained_optimizers) == 2, "Should have two chained optimizers"
+ # get_megatron_muon_optimizer produces muon + adam chained optimizers
+ assert len(optimizer.chained_optimizers) >= 2, "Should have multiple chained optimizers"
# Set gradients and test optimizer step - this will trigger allgather
for param in model.parameters():
@@ -332,26 +387,39 @@ def test_bf16_error(self):
TransformerConfig(num_attention_heads=1, num_layers=1), ddp_config, model
)
+ pg_collection = ProcessGroupCollection.use_mpu_process_groups()
+ pg_collection.dp_cp = parallel_state.get_data_parallel_group(with_context_parallel=True)
+ pg_collection.expt_dp = parallel_state.get_expert_data_parallel_group()
+
+ # Create muon optimizer (non-layer-wise) — produces Float16-wrapped chained optimizers
optimizer_config = OptimizerConfig(
- optimizer='adam', lr=0.01, bf16=True, use_distributed_optimizer=False
+ optimizer='muon',
+ lr=0.01,
+ bf16=True,
+ use_distributed_optimizer=False,
+ muon_tp_mode="duplicated",
)
-
- # Create base optimizer and manually wrap in Float16 optimizer
- param_groups = [{'params': list(model.parameters())}]
- base_optimizer = torch.optim.Adam(param_groups, lr=optimizer_config.lr)
- wrapped_optimizer = Float16OptimizerWithFloat16Params(
- base_optimizer, optimizer_config, None, None
+ muon_optimizer = get_megatron_muon_optimizer(
+ config=optimizer_config,
+ model_chunks=[model],
+ use_gloo_process_groups=True,
+ layer_wise_distributed_optimizer=False,
+ pg_collection=pg_collection,
)
- pg_collection = ProcessGroupCollection.use_mpu_process_groups()
- pg_collection.dp_cp = parallel_state.get_data_parallel_group(with_context_parallel=True)
- pg_collection.expt_dp = parallel_state.get_expert_data_parallel_group()
+ # Extract a Float16-wrapped chained optimizer
+ wrapped_optimizer = muon_optimizer.chained_optimizers[0]
+ assert isinstance(wrapped_optimizer, Float16OptimizerWithFloat16Params)
# Should raise TypeError when receiving already-wrapped Float16 optimizer
+ # Use a fresh config since get_megatron_muon_optimizer mutates config.optimizer
+ lw_config = OptimizerConfig(
+ optimizer='muon', lr=0.01, bf16=True, use_distributed_optimizer=False
+ )
with pytest.raises(
TypeError, match='LayerWiseDistributedOptimizer received Float16 optimizer already'
):
- LayerWiseDistributedOptimizer([wrapped_optimizer], optimizer_config, pg_collection)
+ LayerWiseDistributedOptimizer([wrapped_optimizer], lw_config, pg_collection)
def _run_parameter_update_test(self, model_class=SimpleModel):
"""Helper method to test parameter updates with a given model class.
@@ -438,3 +506,662 @@ def test_broadcast_vs_allgather(self):
# Verify updated values match reference optimizer
for param, ref_param in zip(model.parameters(), reference_model.parameters()):
torch.testing.assert_close(param.data, ref_param.data, rtol=0, atol=0)
+
+ # ---- Overlap-param-gather tests ----
+
+ def test_overlap_param_gather_basic(self):
+ """Test overlap-param-gather path: init, forward/backward/step, bucket-based param sync."""
+ model, optimizer, pg_collection = (
+ self.create_model_and_optimizer_with_overlap_param_gather()
+ )
+
+ assert optimizer is not None, "Optimizer should not be None"
+ assert optimizer.async_allgather, "async_allgather should be True"
+
+ reference_model = self.create_reference_model(model)
+
+ input_tensor = torch.randn(16, 80, dtype=torch.bfloat16, device='cuda')
+ output = model(input_tensor)
+ loss = output.sum()
+ loss.backward()
+
+ # step() updates local params but skips allgather (async_allgather=True)
+ update_successful, grad_norm, num_zeros = optimizer.step()
+
+ assert update_successful, "Optimizer step should be successful"
+
+ # Manually sync params through the bucket-based param sync path
+ # force_sync=True does synchronous allgather via bucket infrastructure
+ model.start_param_sync(force_sync=True)
+
+ # Verify parameters were updated
+ params_updated = 0
+ for param, ref_param in zip(model.parameters(), reference_model.parameters()):
+ if not torch.equal(param.data, ref_param.data):
+ params_updated += 1
+
+ assert params_updated > 0, "At least some parameters should be updated"
+
+ # Verify all ranks have the same updated parameters
+ dp_size = get_pg_size(pg_collection.dp_cp)
+
+ if dp_size > 1:
+ for name, param in model.named_parameters():
+ param_list = [torch.zeros_like(param.data) for _ in range(dp_size)]
+ torch.distributed.all_gather(param_list, param.data, group=pg_collection.dp_cp)
+
+ for i in range(1, dp_size):
+ torch.testing.assert_close(
+ param_list[0],
+ param_list[i],
+ msg=f"Parameter {name} differs between rank 0 and rank {i}",
+ )
+
+ def test_overlap_param_gather_parameter_updates(self):
+ """Test overlap-param-gather produces same parameter updates as standard optimizer."""
+ model, optimizer, pg_collection = (
+ self.create_model_and_optimizer_with_overlap_param_gather()
+ )
+
+ # Create reference model with standard (non-layer-wise) optimizer
+ reference_model, reference_optimizer, _ = self.create_model_and_optimizer(
+ use_layer_wise=False, copy_from=model
+ )
+
+ # Set same gradients on both models
+ for param, ref_param in zip(model.parameters(), reference_model.parameters()):
+ assert torch.equal(param.data, ref_param.data)
+ grad_value = torch.randn_like(param)
+ torch.distributed.broadcast(grad_value, src=0, group=pg_collection.dp_cp)
+ param.main_grad = grad_value.clone().detach()
+ ref_param.main_grad = grad_value.clone().detach()
+
+ # step() with async_allgather=True: updates but no allgather
+ optimizer.step()
+ # Manually sync params via bucket infrastructure
+ model.start_param_sync(force_sync=True)
+
+ reference_optimizer.step()
+
+ # Verify updated values match reference optimizer
+ for param, ref_param in zip(model.parameters(), reference_model.parameters()):
+ torch.testing.assert_close(param.data, ref_param.data, rtol=1e-5, atol=1e-5)
+
+ def test_overlap_param_gather_vs_sync_allgather(self):
+ """Key correctness test: overlap path and sync allgather produce identical updates.
+
+ Compares:
+ - Overlap path: async_allgather=True, bucket-based param sync
+ - Sync path: async_allgather=False, optimizer.allgather_params() in step()
+ """
+ # Create overlap model
+ overlap_model, overlap_optimizer, pg_collection = (
+ self.create_model_and_optimizer_with_overlap_param_gather(async_allgather=True)
+ )
+
+ # Create sync model with same weights (overlap_param_gather=True but sync allgather)
+ sync_model, sync_optimizer, _ = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=False, copy_from=overlap_model
+ )
+
+ # Verify initial parameters match
+ for op, sp in zip(overlap_model.parameters(), sync_model.parameters()):
+ assert torch.equal(op.data, sp.data)
+
+ # Set identical gradients on both
+ for op, sp in zip(overlap_model.parameters(), sync_model.parameters()):
+ grad_value = torch.randn_like(op)
+ torch.distributed.broadcast(grad_value, src=0, group=pg_collection.dp_cp)
+ op.main_grad = grad_value.clone().detach()
+ sp.main_grad = grad_value.clone().detach()
+
+ # Overlap path: step + manual sync
+ overlap_optimizer.step()
+ overlap_model.start_param_sync(force_sync=True)
+
+ # Sync path: step (includes allgather_params)
+ sync_optimizer.step()
+
+ # Both paths should produce identical parameter values
+ for op, sp in zip(overlap_model.parameters(), sync_model.parameters()):
+ torch.testing.assert_close(
+ op.data,
+ sp.data,
+ rtol=0,
+ atol=0,
+ msg="Overlap and sync allgather paths produced different parameter updates",
+ )
+
+ def test_overlap_param_gather_bucket_lw_params(self):
+ """Verify bucket.layerwise_params_list is populated when async_allgather is enabled."""
+ model, optimizer, pg_collection = (
+ self.create_model_and_optimizer_with_overlap_param_gather()
+ )
+
+ dp_size = get_pg_size(pg_collection.dp_cp)
+
+ for bucket_group in model.bucket_groups:
+ for bucket in bucket_group.buckets:
+ # layerwise_params_list should be populated by set_bucket_layerwise_params_list
+ assert (
+ bucket.layerwise_params_list is not None
+ ), "bucket.layerwise_params_list should be populated"
+ assert (
+ len(bucket.layerwise_params_list) == dp_size
+ ), f"Expected {dp_size} per-rank lists, got {len(bucket.layerwise_params_list)}"
+
+ # The union of all per-rank param lists should cover all bucket params
+ all_lw_params = set()
+ for rank_params in bucket.layerwise_params_list:
+ for p in rank_params:
+ all_lw_params.add(p)
+ assert (
+ all_lw_params == bucket.params
+ ), "Union of per-rank layerwise_params should equal bucket params"
+
+ # layerwise_param_flat_sizes should be populated and have correct length
+ assert bucket.layerwise_param_flat_sizes is not None
+ assert len(bucket.layerwise_param_flat_sizes) == dp_size
+
+ # Each flat size should equal the sum of param numels for that rank
+ for rank_idx in range(dp_size):
+ expected_size = sum(p.numel() for p in bucket.layerwise_params_list[rank_idx])
+ assert bucket.layerwise_param_flat_sizes[rank_idx] == expected_size, (
+ f"Rank {rank_idx}: expected flat_size {expected_size}, "
+ f"got {bucket.layerwise_param_flat_sizes[rank_idx]}"
+ )
+
+ def test_overlap_param_gather_vs_standard_ddp(self):
+ """Verify DDP with overlap_param_gather=True produces same results as standard DDP.
+
+ Both use LayerWiseDistributedOptimizer but with different DDP configs:
+ - Overlap path: overlap_param_gather=True (padded buffers)
+ - Standard path: overlap_param_gather=False (unpadded buffers)
+ """
+ # Create overlap-param-gather model (sync allgather for simpler comparison)
+ opg_model, opg_optimizer, pg_collection = (
+ self.create_model_and_optimizer_with_overlap_param_gather(async_allgather=False)
+ )
+
+ # Create standard model with same weights
+ std_model, std_optimizer, _ = self.create_model_and_optimizer(copy_from=opg_model)
+
+ # Set identical gradients
+ for op, sp in zip(opg_model.parameters(), std_model.parameters()):
+ assert torch.equal(op.data, sp.data)
+ grad_value = torch.randn_like(op)
+ torch.distributed.broadcast(grad_value, src=0, group=pg_collection.dp_cp)
+ op.main_grad = grad_value.clone().detach()
+ sp.main_grad = grad_value.clone().detach()
+
+ opg_optimizer.step()
+ std_optimizer.step()
+
+ # Both should produce identical parameter values
+ for op, sp in zip(opg_model.parameters(), std_model.parameters()):
+ torch.testing.assert_close(
+ op.data,
+ sp.data,
+ rtol=1e-5,
+ atol=1e-5,
+ msg="Overlap-param-gather and standard paths produced different updates",
+ )
+
+ def test_overlap_param_gather_insufficient_parameters(self):
+ """Test overlap-param-gather with TinyModel (only 2 params).
+
+ Many ranks will have no assigned params when world_size > 2.
+ """
+ model, optimizer, pg_collection = self.create_model_and_optimizer_with_overlap_param_gather(
+ model_class=TinyModel
+ )
+
+ # Create reference model with standard (non-layer-wise) optimizer
+ reference_model, reference_optimizer, _ = self.create_model_and_optimizer(
+ model_class=TinyModel, use_layer_wise=False, copy_from=model
+ )
+
+ # Set same gradients on both models
+ for param, ref_param in zip(model.parameters(), reference_model.parameters()):
+ assert torch.equal(param.data, ref_param.data)
+ grad_value = torch.randn_like(param)
+ torch.distributed.broadcast(grad_value, src=0, group=pg_collection.dp_cp)
+ param.main_grad = grad_value.clone().detach()
+ ref_param.main_grad = grad_value.clone().detach()
+
+ optimizer.step()
+ model.start_param_sync(force_sync=True)
+
+ reference_optimizer.step()
+
+ # Verify updated values match reference optimizer
+ for param, ref_param in zip(model.parameters(), reference_model.parameters()):
+ torch.testing.assert_close(param.data, ref_param.data, rtol=1e-5, atol=1e-5)
+
+ def test_overlap_param_gather_broadcast_vs_allgather(self):
+ """Test overlap-param-gather: allgather vs broadcast produce same results."""
+ model, optimizer, pg_collection = self.create_model_and_optimizer_with_overlap_param_gather(
+ model_class=SimpleModel, async_allgather=False
+ )
+
+ # Create reference model with overlap-param-gather path too
+ reference_model, reference_optimizer, _ = (
+ self.create_model_and_optimizer_with_overlap_param_gather(
+ model_class=SimpleModel, async_allgather=False, copy_from=model
+ )
+ )
+
+ # Set same gradients on both models
+ for param, ref_param in zip(model.parameters(), reference_model.parameters()):
+ assert torch.equal(param.data, ref_param.data)
+ torch.testing.assert_close(param.data, ref_param.data, rtol=0, atol=0)
+ grad_value = torch.randn_like(param)
+ torch.distributed.broadcast(grad_value, src=0, group=pg_collection.dp_cp)
+ param.main_grad = grad_value.clone().detach()
+ ref_param.main_grad = grad_value.clone().detach()
+
+ optimizer.step()
+
+ # Verify at least some parameters were updated
+ params_updated = 0
+ for param, ref_param in zip(model.parameters(), reference_model.parameters()):
+ if not torch.equal(param.data, ref_param.data):
+ params_updated += 1
+
+ assert params_updated > 0, "At least some parameters should be updated"
+
+ # step() internally calls allgather_params. Replace reference with broadcast.
+ reference_optimizer.allgather_params = reference_optimizer.broadcast_params
+ reference_optimizer.step()
+
+ # Verify updated values match reference optimizer
+ for param, ref_param in zip(model.parameters(), reference_model.parameters()):
+ torch.testing.assert_close(param.data, ref_param.data, rtol=0, atol=0)
+
+ def test_overlap_param_gather_multi_iteration(self):
+ """Test overlap-param-gather correctness over multiple training iterations.
+
+ Runs multiple forward/backward/step iterations using the async allgather path.
+ After each iteration, manually syncs params and verifies they match a reference
+ model using the sync path.
+ """
+ model, optimizer, pg_collection = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=True
+ )
+
+ # Create reference model with sync allgather for comparison
+ ref_model, ref_optimizer, _ = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=False, copy_from=model
+ )
+
+ for iteration in range(3):
+ # Set identical gradients on both models
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ grad_value = torch.randn_like(param)
+ torch.distributed.broadcast(grad_value, src=0, group=pg_collection.dp_cp)
+ param.main_grad = grad_value.clone().detach()
+ ref_param.main_grad = grad_value.clone().detach()
+
+ # Async path: step (no allgather) + manual sync
+ optimizer.step()
+ model.start_param_sync(force_sync=True)
+
+ # Sync path: step (includes allgather)
+ ref_optimizer.step()
+
+ # Verify parameters match after each iteration
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ torch.testing.assert_close(
+ param.data,
+ ref_param.data,
+ rtol=0,
+ atol=0,
+ msg=f"Parameters diverged at iteration {iteration}",
+ )
+
+ def test_overlap_param_gather_async_dispatch_and_finish(self):
+ """Test async dispatch + finish_param_sync cycle (the actual runtime path).
+
+ start_param_sync() (no force_sync) dispatches async all-gathers, then
+ finish_param_sync() waits on the handle and unflattens gathered params.
+ """
+ model, optimizer, pg_collection = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=True
+ )
+ ref_model, ref_optimizer, _ = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=False, copy_from=model
+ )
+
+ # Set identical gradients on both models
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ grad_value = torch.randn_like(param)
+ torch.distributed.broadcast(grad_value, src=0, group=pg_collection.dp_cp)
+ param.main_grad = grad_value.clone().detach()
+ ref_param.main_grad = grad_value.clone().detach()
+
+ # Async path: step (no allgather) + async dispatch + explicit finish
+ optimizer.step()
+ model.start_param_sync() # async dispatch to all bucket groups
+ for bucket_group in model.bucket_groups:
+ bucket_group.finish_param_sync(skip_next_bucket_dispatch=True)
+
+ # Sync path: step (includes allgather)
+ ref_optimizer.step()
+
+ # Verify params match sync path
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ torch.testing.assert_close(
+ param.data,
+ ref_param.data,
+ rtol=0,
+ atol=0,
+ msg="Async dispatch + finish path produced different params than sync path",
+ )
+
+ # Verify all ranks have identical parameters
+ dp_size = get_pg_size(pg_collection.dp_cp)
+ if dp_size > 1:
+ for name, param in model.named_parameters():
+ param_list = [torch.zeros_like(param.data) for _ in range(dp_size)]
+ torch.distributed.all_gather(param_list, param.data, group=pg_collection.dp_cp)
+ for i in range(1, dp_size):
+ torch.testing.assert_close(
+ param_list[0],
+ param_list[i],
+ msg=f"Parameter {name} differs between rank 0 and rank {i}",
+ )
+
+ def test_overlap_param_gather_finish_chains_next_bucket(self):
+ """Test that finish_param_sync() dispatches next_param_gather_bucket_group.
+
+ Uses a small bucket_size to force multiple bucket groups, then dispatches
+ only the last bucket group and verifies that finishing it chains to the next.
+ """
+ model, optimizer, pg_collection = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=True, bucket_size=2000
+ )
+
+ bucket_groups = model.bucket_groups
+ if len(bucket_groups) <= 1:
+ pytest.skip("Need multiple bucket groups to test chaining")
+
+ ref_model, ref_optimizer, _ = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=False, copy_from=model, bucket_size=2000
+ )
+
+ # Set identical gradients on both models
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ grad_value = torch.randn_like(param)
+ torch.distributed.broadcast(grad_value, src=0, group=pg_collection.dp_cp)
+ param.main_grad = grad_value.clone().detach()
+ ref_param.main_grad = grad_value.clone().detach()
+
+ optimizer.step()
+
+ # Dispatch ONLY the last bucket group (which has next_param_gather_bucket_group set)
+ last_bg = bucket_groups[-1]
+ last_bg.start_param_sync()
+
+ # Verify: next bucket group has NOT been dispatched yet
+ next_bg = last_bg.next_param_gather_bucket_group
+ assert next_bg is not None, "Last bucket group should have a next"
+ assert not next_bg.param_gather_dispatched, "Next bucket should not be dispatched yet"
+
+ # Finish the last bucket group — should chain-dispatch the next one
+ last_bg.finish_param_sync()
+
+ # Verify: next bucket group IS now dispatched via chaining
+ assert (
+ next_bg.param_gather_dispatched
+ ), "finish_param_sync should have dispatched next bucket group"
+
+ # Finish remaining bucket groups through the chain
+ for bg in reversed(bucket_groups[:-1]):
+ bg.finish_param_sync()
+
+ # Reference: sync step
+ ref_optimizer.step()
+
+ # Verify params match
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ torch.testing.assert_close(
+ param.data,
+ ref_param.data,
+ rtol=0,
+ atol=0,
+ msg="Chained bucket finish produced different params than sync path",
+ )
+
+ def test_overlap_param_gather_forward_pre_hook(self):
+ """Test forward pre-hooks trigger finish_param_sync during model(input).
+
+ After async dispatch, running model(input) fires forward pre-hooks that
+ call finish_param_sync() on each bucket group, completing the param sync.
+ """
+ model, optimizer, pg_collection = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=True
+ )
+ ref_model, ref_optimizer, _ = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=False, copy_from=model
+ )
+
+ # Set identical gradients on both models
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ grad_value = torch.randn_like(param)
+ torch.distributed.broadcast(grad_value, src=0, group=pg_collection.dp_cp)
+ param.main_grad = grad_value.clone().detach()
+ ref_param.main_grad = grad_value.clone().detach()
+
+ # Async path: step (no allgather) + async dispatch
+ optimizer.step()
+ model.start_param_sync() # dispatch async broadcasts
+
+ # Forward pass triggers hooks that call finish_param_sync()
+ input_tensor = torch.randn(16, 80, dtype=torch.bfloat16, device='cuda')
+ output = model(input_tensor)
+
+ # Sync path: step (includes allgather)
+ ref_optimizer.step()
+
+ # Verify params match
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ torch.testing.assert_close(
+ param.data,
+ ref_param.data,
+ rtol=0,
+ atol=0,
+ msg="Forward pre-hook path produced different params than sync path",
+ )
+
+ def test_overlap_param_gather_grad_reduce_in_fp32(self):
+ """Regression test: grad_reduce_in_fp32 must not cause dtype mismatch in broadcasts.
+
+ When grad_reduce_in_fp32=True, the grad buffer dtype is fp32 but broadcast
+ buffers must use param dtype (bf16). Without the fix (commit cbed167fc), this
+ would cause a dtype mismatch error in the per-rank broadcast calls.
+ """
+ model, optimizer, pg_collection = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=True, grad_reduce_in_fp32=True
+ )
+ ref_model, ref_optimizer, _ = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=False, copy_from=model, grad_reduce_in_fp32=True
+ )
+
+ # Set identical gradients on both models
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ grad_value = torch.randn_like(param)
+ torch.distributed.broadcast(grad_value, src=0, group=pg_collection.dp_cp)
+ param.main_grad = grad_value.clone().detach()
+ ref_param.main_grad = grad_value.clone().detach()
+
+ # Async path: step + force_sync
+ optimizer.step()
+ model.start_param_sync(force_sync=True)
+
+ # Sync path: step (includes allgather)
+ ref_optimizer.step()
+
+ # Verify params match
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ torch.testing.assert_close(
+ param.data,
+ ref_param.data,
+ rtol=1e-5,
+ atol=1e-5,
+ msg="grad_reduce_in_fp32 path produced different params than reference",
+ )
+
+ def test_overlap_param_gather_hook_enable_disable_cycle(self):
+ """Test the training loop's hook lifecycle: disable → manual sync → enable → forward.
+
+ The training loop disables hooks before iteration 1 (for initialization),
+ then enables them for subsequent iterations. This test exercises that cycle.
+ """
+ model, optimizer, pg_collection = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=True
+ )
+ ref_model, ref_optimizer, _ = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=False, copy_from=model
+ )
+
+ input_tensor = torch.randn(16, 80, dtype=torch.bfloat16, device='cuda')
+
+ # Iteration 1: hooks disabled, manual sync
+ model.disable_forward_pre_hook(param_sync=False)
+
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ grad_value = torch.randn_like(param)
+ torch.distributed.broadcast(grad_value, src=0, group=pg_collection.dp_cp)
+ param.main_grad = grad_value.clone().detach()
+ ref_param.main_grad = grad_value.clone().detach()
+
+ optimizer.step()
+ model.start_param_sync(force_sync=True) # manual sync
+
+ ref_optimizer.step()
+
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ torch.testing.assert_close(
+ param.data,
+ ref_param.data,
+ rtol=0,
+ atol=0,
+ msg="Params diverged after iteration 1 (hooks disabled)",
+ )
+
+ # Iteration 2: hooks re-enabled, forward pass triggers sync
+ model.enable_forward_pre_hook()
+
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ grad_value = torch.randn_like(param)
+ torch.distributed.broadcast(grad_value, src=0, group=pg_collection.dp_cp)
+ param.main_grad = grad_value.clone().detach()
+ ref_param.main_grad = grad_value.clone().detach()
+
+ optimizer.step()
+ model.start_param_sync() # async dispatch
+ output = model(input_tensor) # hooks finish sync
+
+ ref_optimizer.step()
+
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ torch.testing.assert_close(
+ param.data,
+ ref_param.data,
+ rtol=0,
+ atol=0,
+ msg="Params diverged after iteration 2 (hooks re-enabled)",
+ )
+
+ def test_overlap_param_gather_multi_iteration_with_hooks(self):
+ """Test multiple iterations using forward pre-hooks (not manual force_sync).
+
+ Runs 3 iterations where each iteration uses: set grads → step → async dispatch →
+ forward pass (hooks wait+unflatten). Compares against reference model using sync
+ allgather after each iteration.
+ """
+ model, optimizer, pg_collection = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=True
+ )
+ ref_model, ref_optimizer, _ = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=False, copy_from=model
+ )
+
+ input_tensor = torch.randn(16, 80, dtype=torch.bfloat16, device='cuda')
+
+ for iteration in range(3):
+ # Set identical gradients on both models
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ grad_value = torch.randn_like(param)
+ torch.distributed.broadcast(grad_value, src=0, group=pg_collection.dp_cp)
+ param.main_grad = grad_value.clone().detach()
+ ref_param.main_grad = grad_value.clone().detach()
+
+ # Async path: step + dispatch + forward (hooks wait+unflatten)
+ optimizer.step()
+ model.start_param_sync() # async dispatch
+ output = model(input_tensor) # hooks trigger finish_param_sync
+
+ # Sync path: step (includes allgather)
+ ref_optimizer.step()
+
+ # Verify parameters match after each iteration
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ torch.testing.assert_close(
+ param.data,
+ ref_param.data,
+ rtol=0,
+ atol=0,
+ msg=f"Parameters diverged at iteration {iteration}",
+ )
+
+ def test_overlap_param_gather_start_sync_with_autograd(self):
+ """Regression test: start_param_sync must work when autograd is active.
+
+ _flatten_dense_tensors on params with requires_grad=True produces a tensor
+ that also requires grad. Since all_gather writes into gather_list entries
+ in-place and the local rank's slot reuses src, this triggers:
+ RuntimeError: a view of a leaf Variable that requires grad is being
+ used in an in-place operation.
+ The fix is to .detach() the flattened tensor before using it as src.
+
+ This test calls start_param_sync (synchronous via force_sync) WITHOUT
+ torch.no_grad() to reproduce the exact scenario that occurs during the
+ forward pass when finish_param_sync chains to start_param_sync for the
+ next bucket group.
+ """
+ model, optimizer, pg_collection = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=True
+ )
+ ref_model, ref_optimizer, _ = self.create_model_and_optimizer_with_overlap_param_gather(
+ async_allgather=False, copy_from=model
+ )
+
+ # Confirm params require grad (the precondition for this bug).
+ for param in model.parameters():
+ assert param.requires_grad, "Test requires params with requires_grad=True"
+
+ # Set identical gradients on both models.
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ grad_value = torch.randn_like(param)
+ torch.distributed.broadcast(grad_value, src=0, group=pg_collection.dp_cp)
+ param.main_grad = grad_value.clone().detach()
+ ref_param.main_grad = grad_value.clone().detach()
+
+ # Step both optimizers (async path skips allgather, ref path includes it).
+ optimizer.step()
+ ref_optimizer.step()
+
+ # Call start_param_sync with autograd ENABLED (no torch.no_grad()).
+ # Before the .detach() fix, this would raise RuntimeError.
+ model.start_param_sync(force_sync=True)
+
+ # Verify gathered params match the reference.
+ for param, ref_param in zip(model.parameters(), ref_model.parameters()):
+ torch.testing.assert_close(
+ param.data,
+ ref_param.data,
+ rtol=0,
+ atol=0,
+ msg="Params incorrect after start_param_sync with autograd enabled",
+ )
diff --git a/tests/unit_tests/test_lion_optimizer.py b/tests/unit_tests/test_lion_optimizer.py
new file mode 100644
index 00000000000..589ed82764c
--- /dev/null
+++ b/tests/unit_tests/test_lion_optimizer.py
@@ -0,0 +1,255 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""Integration tests for Lion optimizer plumbing between Megatron and emerging_optimizers.
+
+Tests that require emerging_optimizers are skipped when the package is not installed.
+Tests that verify Megatron-only behavior (config, error handling) always run.
+"""
+
+from types import SimpleNamespace
+from unittest.mock import patch
+
+import pytest
+import torch
+import torch.nn as nn
+
+from megatron.core.optimizer import (
+ HAVE_EO_V02,
+ OptimizerConfig,
+ _get_megatron_optimizer_based_on_param_groups,
+ _get_param_groups,
+)
+from megatron.core.optimizer.optimizer import FP32Optimizer
+
+requires_emerging_optimizers = pytest.mark.skipif(
+ not HAVE_EO_V02, reason="emerging_optimizers package not installed"
+)
+
+
+class SimpleModel(nn.Module):
+ def __init__(self):
+ super().__init__()
+ self.linear1 = nn.Linear(16, 32)
+ self.linear2 = nn.Linear(32, 8)
+
+ def forward(self, x):
+ return self.linear2(torch.relu(self.linear1(x)))
+
+
+def _make_param_groups(model):
+ """Create simple param groups without distributed calls."""
+ return [{"params": list(model.parameters()), "is_expert_parallel": False}]
+
+
+def _make_pg_collection():
+ """Create a mock pg_collection with a tp attribute to avoid parallel_state calls."""
+ return SimpleNamespace(tp=None)
+
+
+def _create_lion_optimizer(model, config):
+ """Helper to create a Lion optimizer via Megatron's factory."""
+ return _get_megatron_optimizer_based_on_param_groups(
+ config=config,
+ model_chunks=[model],
+ param_groups=_make_param_groups(model),
+ pg_collection=_make_pg_collection(),
+ )
+
+
+class TestLionOptimizerConfig:
+ """Tests for Lion config plumbing that do not require emerging_optimizers."""
+
+ def test_lion_config_fields_exist(self):
+ """Config should accept lion_beta1, lion_beta2, and muon_scalar_optimizer."""
+ config = OptimizerConfig(
+ optimizer="lion",
+ lr=1e-4,
+ lion_beta1=0.92,
+ lion_beta2=0.97,
+ muon_scalar_optimizer="lion",
+ )
+ assert config.lion_beta1 == 0.92
+ assert config.lion_beta2 == 0.97
+ assert config.muon_scalar_optimizer == "lion"
+
+ def test_lion_config_defaults(self):
+ """Config defaults should match expected Lion defaults."""
+ config = OptimizerConfig()
+ assert config.lion_beta1 == 0.95
+ assert config.lion_beta2 == 0.98
+ assert config.muon_scalar_optimizer == "adam"
+
+ @patch("torch.distributed.get_world_size", return_value=1)
+ @patch(
+ "torch.distributed.all_gather_object",
+ lambda output_list, obj: output_list.__setitem__(0, obj),
+ )
+ def test_lion_param_groups_via_get_param_groups(self, mock_world_size):
+ """_get_param_groups should work with lion config (same as adam)."""
+ model = SimpleModel()
+ config = OptimizerConfig(optimizer="lion", lr=1e-4)
+ param_groups = _get_param_groups([model], config, {})
+
+ assert len(param_groups) == 1
+ assert param_groups[0]["params"] == list(model.parameters())
+
+ def test_lion_import_error_without_package(self):
+ """Should raise ImportError with helpful message if emerging_optimizers not installed."""
+ import megatron.core.optimizer as opt_module
+
+ original_have_lion = opt_module.HAVE_EO_V02
+ try:
+ opt_module.HAVE_EO_V02 = False
+
+ model = SimpleModel()
+ config = OptimizerConfig(optimizer="lion", lr=1e-4)
+
+ with pytest.raises(ImportError, match="emerging_optimizers"):
+ _create_lion_optimizer(model, config)
+ finally:
+ opt_module.HAVE_EO_V02 = original_have_lion
+
+
+@requires_emerging_optimizers
+class TestLionOptimizerExactness:
+ """Tests that verify bit-for-bit exactness between Megatron-plumbed Lion and standalone Lion.
+
+ These tests require emerging_optimizers to be installed.
+ """
+
+ def test_lion_factory_creates_lion_optimizer(self):
+ """_get_megatron_optimizer_based_on_param_groups should create a Lion-backed optimizer."""
+ from emerging_optimizers.scalar_optimizers import Lion
+
+ model = SimpleModel()
+ config = OptimizerConfig(
+ optimizer="lion", lr=3e-4, lion_beta1=0.93, lion_beta2=0.99, weight_decay=0.01
+ )
+
+ optimizer = _create_lion_optimizer(model, config)
+
+ # Should be wrapped in FP32Optimizer since no fp16/bf16 set.
+ assert isinstance(optimizer, FP32Optimizer)
+
+ # The underlying optimizer should be Lion from emerging_optimizers.
+ inner_opt = optimizer.optimizer
+ assert isinstance(inner_opt, Lion)
+
+ # Verify betas were plumbed through exactly.
+ for group in inner_opt.param_groups:
+ assert group["betas"] == (0.93, 0.99)
+ assert group["lr"] == 3e-4
+ assert group["weight_decay"] == 0.01
+
+ def test_lion_init_state_fn_creates_exp_avg(self):
+ """init_state_fn should pre-initialize exp_avg state for all params."""
+ model = SimpleModel()
+ config = OptimizerConfig(optimizer="lion", lr=1e-4)
+
+ optimizer = _create_lion_optimizer(model, config)
+ inner_opt = optimizer.optimizer
+
+ # State should be empty before init.
+ for p in model.parameters():
+ assert len(inner_opt.state[p]) == 0
+
+ # Call init_state_fn (stored on the FP32Optimizer wrapper).
+ optimizer.init_state_fn(inner_opt)
+
+ # State should now have exp_avg for each param, exactly zero.
+ for p in model.parameters():
+ assert "exp_avg" in inner_opt.state[p]
+ assert inner_opt.state[p]["exp_avg"].shape == p.shape
+ assert inner_opt.state[p]["exp_avg"].dtype == p.dtype
+ torch.testing.assert_close(
+ inner_opt.state[p]["exp_avg"], torch.zeros_like(p.data), atol=0, rtol=0
+ )
+
+ @pytest.mark.parametrize(
+ "lr,beta1,beta2,weight_decay",
+ [(1e-3, 0.95, 0.98, 0.0), (3e-4, 0.9, 0.99, 0.01), (1e-4, 0.85, 0.95, 0.1)],
+ )
+ def test_megatron_lion_exact_match_with_standalone(self, lr, beta1, beta2, weight_decay):
+ """Megatron-plumbed Lion must produce bit-for-bit identical results to standalone Lion.
+
+ This is the core correctness test: we run the same forward-backward-step on two
+ identical models — one using Lion directly, one through Megatron's factory — and
+ verify that all parameters and optimizer states are exactly equal.
+ """
+ from emerging_optimizers.scalar_optimizers import Lion
+
+ # Create two identical models from the same seed.
+ torch.manual_seed(0)
+ model_standalone = SimpleModel()
+ torch.manual_seed(0)
+ model_megatron = SimpleModel()
+
+ # Sanity: models start identical.
+ for (n1, p1), (n2, p2) in zip(
+ model_standalone.named_parameters(), model_megatron.named_parameters()
+ ):
+ assert n1 == n2
+ torch.testing.assert_close(p1.data, p2.data, atol=0, rtol=0)
+
+ # Create standalone Lion directly from emerging_optimizers.
+ opt_standalone = Lion(
+ model_standalone.parameters(), lr=lr, betas=(beta1, beta2), weight_decay=weight_decay
+ )
+
+ # Create Lion through Megatron's factory.
+ config = OptimizerConfig(
+ optimizer="lion", lr=lr, lion_beta1=beta1, lion_beta2=beta2, weight_decay=weight_decay
+ )
+ megatron_optimizer = _create_lion_optimizer(model_megatron, config)
+ opt_megatron = megatron_optimizer.optimizer
+
+ # Run multiple steps to accumulate momentum differences if any.
+ for step in range(3):
+ torch.manual_seed(step + 100)
+ x = torch.randn(4, 16)
+
+ # Standalone forward-backward-step.
+ loss_standalone = model_standalone(x).sum()
+ loss_standalone.backward()
+ opt_standalone.step()
+ opt_standalone.zero_grad()
+
+ # Megatron forward-backward-step (same input).
+ loss_megatron = model_megatron(x).sum()
+ loss_megatron.backward()
+ opt_megatron.step()
+ opt_megatron.zero_grad()
+
+ # Verify losses are exactly equal.
+ torch.testing.assert_close(
+ loss_standalone, loss_megatron, atol=0, rtol=0, msg=f"Step {step}: losses differ"
+ )
+
+ # Verify all parameters are exactly equal after each step.
+ for (n1, p1), (n2, p2) in zip(
+ model_standalone.named_parameters(), model_megatron.named_parameters()
+ ):
+ torch.testing.assert_close(
+ p1.data,
+ p2.data,
+ atol=0,
+ rtol=0,
+ msg=f"Step {step}, param {n1}: parameters differ",
+ )
+
+ # Verify optimizer states (exp_avg) are exactly equal.
+ for p_s, p_m in zip(model_standalone.parameters(), model_megatron.parameters()):
+ state_s = opt_standalone.state[p_s]
+ state_m = opt_megatron.state[p_m]
+ assert state_s.keys() == state_m.keys(), (
+ f"Step {step}: optimizer state keys differ: "
+ f"{state_s.keys()} vs {state_m.keys()}"
+ )
+ for key in state_s:
+ torch.testing.assert_close(
+ state_s[key],
+ state_m[key],
+ atol=0,
+ rtol=0,
+ msg=f"Step {step}, state '{key}': optimizer states differ",
+ )
diff --git a/tests/unit_tests/test_muon_optimizer.py b/tests/unit_tests/test_muon_optimizer.py
index cc99f7a16e6..0f0a90c91ed 100644
--- a/tests/unit_tests/test_muon_optimizer.py
+++ b/tests/unit_tests/test_muon_optimizer.py
@@ -10,16 +10,30 @@
from megatron.core import parallel_state
from megatron.core.distributed import DistributedDataParallel, DistributedDataParallelConfig
-from megatron.core.optimizer import OptimizerConfig
-from megatron.core.optimizer.muon import TensorParallelMuon, get_megatron_muon_optimizer
+from megatron.core.optimizer import HAVE_EMERGING_OPTIMIZERS, HAVE_EO_V02, OptimizerConfig
+from megatron.core.optimizer.muon import (
+ TensorParallelMuon,
+ get_megatron_muon_optimizer,
+ get_supported_coefficient_types,
+ validate_coefficient_type,
+)
from megatron.core.process_groups_config import ProcessGroupCollection
from megatron.core.transformer import TransformerConfig
from tests.unit_tests.test_utilities import Utils
-# Skip all tests in this file for LTS versions
-pytestmark = pytest.mark.skipif(
- Version(os.getenv('NVIDIA_PYTORCH_VERSION', "24.01")) <= Version("25.05"),
- reason="Skip muon optimizer for LTS test",
+# Skip all tests in this file for LTS versions or when emerging_optimizers is missing
+pytestmark = [
+ pytest.mark.skipif(
+ Version(os.getenv('NVIDIA_PYTORCH_VERSION', "24.01")) <= Version("25.05"),
+ reason="Skip muon optimizer for LTS test",
+ ),
+ pytest.mark.skipif(
+ not HAVE_EMERGING_OPTIMIZERS, reason="emerging_optimizers package is not installed"
+ ),
+]
+
+requires_eo_v02 = pytest.mark.skipif(
+ not HAVE_EO_V02, reason="emerging_optimizers >= 0.2 is required"
)
@@ -420,10 +434,18 @@ def test_muon_optimizer_blockwise_mode_different_result(self):
), "Weight should be updated with mode=blockwise"
-@pytest.mark.parametrize(
- "coefficient_type_and_steps", [("simple", 3), ("quintic", 5), ("polar_express", 8)]
+# All non-custom coefficient types supported by emerging_optimizers.
+_TESTABLE_COEFFICIENT_TYPES = (
+ [t for t in get_supported_coefficient_types() if t != "custom"] if HAVE_EO_V02 else []
)
-def test_muon_optimizer_coefficient_types(coefficient_type_and_steps):
+
+# A reasonable default NS step count for testing; get_coefficient_iterator
+# cycles/repeats coefficients so any step count works with any type.
+_DEFAULT_NS_STEPS = 5
+
+
+@pytest.mark.parametrize("coefficient_type", _TESTABLE_COEFFICIENT_TYPES)
+def test_muon_optimizer_coefficient_types(coefficient_type):
"""Test TensorParallelMuon optimizer with different coefficient types."""
model = torch.nn.Linear(80, 40, bias=False, dtype=torch.float32, device='cuda')
model.requires_grad_(True)
@@ -432,8 +454,8 @@ def test_muon_optimizer_coefficient_types(coefficient_type_and_steps):
optimizer = TensorParallelMuon(
params=[model.weight],
lr=0.01,
- coefficient_type=coefficient_type_and_steps[0],
- num_ns_steps=coefficient_type_and_steps[1],
+ coefficient_type=coefficient_type,
+ num_ns_steps=_DEFAULT_NS_STEPS,
pg_collection=None,
mode="duplicated",
)
@@ -448,7 +470,7 @@ def test_muon_optimizer_coefficient_types(coefficient_type_and_steps):
assert not torch.equal(
model.weight.data, original_weight
- ), f"Weight should be updated with coefficient_type={coefficient_type_and_steps[0]} and num_ns_steps={coefficient_type_and_steps[1]}"
+ ), f"Weight should be updated with coefficient_type={coefficient_type}"
@pytest.mark.parametrize("scale_mode", ["spectral", "unit_rms_norm", "shape_scaling"])
@@ -641,6 +663,106 @@ def test_muon_optimizer_extra_scale_factor():
), "Weight should be updated with extra_scale_factor"
+@requires_eo_v02
+def test_get_supported_coefficient_types_returns_tuple():
+ """Test that get_supported_coefficient_types returns a non-empty tuple of strings."""
+ supported = get_supported_coefficient_types()
+ assert isinstance(supported, tuple)
+ assert len(supported) > 0
+ for t in supported:
+ assert isinstance(t, str)
+
+
+@requires_eo_v02
+def test_get_supported_coefficient_types_contains_known_types():
+ """Test that the known coefficient types are present in the supported set."""
+ supported = get_supported_coefficient_types()
+ for expected in ("simple", "quintic", "polar_express"):
+ assert expected in supported, f"Expected '{expected}' in supported types {supported}"
+
+
+@requires_eo_v02
+def test_validate_coefficient_type_accepts_valid():
+ """Test that validate_coefficient_type does not raise for valid types."""
+ for t in get_supported_coefficient_types():
+ validate_coefficient_type(t) # should not raise
+
+
+def test_validate_coefficient_type_rejects_invalid():
+ """Test that validate_coefficient_type raises ValueError for an invalid type."""
+ with pytest.raises(ValueError, match="Unsupported muon coefficient type"):
+ validate_coefficient_type("nonexistent_type_xyz")
+
+
+def test_muon_optimizer_invalid_coefficient_type():
+ """Test that TensorParallelMuon raises ValueError for an invalid coefficient_type."""
+ model = torch.nn.Linear(80, 40, bias=False, dtype=torch.float32, device='cuda')
+ model.requires_grad_(True)
+
+ with pytest.raises(ValueError, match="Unsupported muon coefficient type"):
+ TensorParallelMuon(
+ params=[model.weight],
+ lr=0.01,
+ coefficient_type="nonexistent_type_xyz",
+ num_ns_steps=5,
+ pg_collection=None,
+ mode="duplicated",
+ )
+
+
+@pytest.mark.skipif(
+ int(os.getenv('WORLD_SIZE', '1')) == 1, reason="Multi-rank test requires WORLD_SIZE > 1"
+)
+class TestMuonCoefficientTypeMultiRank:
+ """Test coefficient_type integration through get_megatron_muon_optimizer."""
+
+ @pytest.fixture(autouse=True)
+ def setup_and_teardown(self):
+ Utils.initialize_model_parallel()
+ yield
+ Utils.destroy_model_parallel()
+
+ def create_ddp_model(self, model):
+ ddp_config = DistributedDataParallelConfig(use_distributed_optimizer=False)
+ return DistributedDataParallel(
+ TransformerConfig(num_attention_heads=1, num_layers=1), ddp_config, model
+ )
+
+ @pytest.mark.parametrize("coefficient_type", _TESTABLE_COEFFICIENT_TYPES)
+ def test_get_megatron_muon_optimizer_coefficient_type(self, coefficient_type):
+ """Test that coefficient_type flows through get_megatron_muon_optimizer."""
+ model = Net().bfloat16().cuda()
+ model.requires_grad_(True)
+ model = self.create_ddp_model(model)
+
+ optimizer_config = OptimizerConfig(
+ optimizer='muon',
+ lr=0.01,
+ weight_decay=0.01,
+ bf16=True,
+ use_distributed_optimizer=False,
+ muon_coefficient_type=coefficient_type,
+ muon_num_ns_steps=_DEFAULT_NS_STEPS,
+ muon_tp_mode="duplicated",
+ )
+
+ optimizer = get_megatron_muon_optimizer(
+ config=optimizer_config,
+ model_chunks=[model],
+ use_gloo_process_groups=True,
+ layer_wise_distributed_optimizer=False,
+ )
+
+ assert optimizer is not None
+
+ input_tensor = torch.randn(16, 80, dtype=torch.bfloat16, device='cuda')
+ output = model(input_tensor)
+ loss = output.sum()
+ loss.backward()
+
+ optimizer.step()
+
+
@pytest.mark.parametrize("num_ns_steps", [5, 15, 25])
def test_muon_optimizer_num_ns_steps(num_ns_steps):
"""Test TensorParallelMuon optimizer with different numbers of Newton-Schulz steps."""
diff --git a/tests/unit_tests/test_optimizer.py b/tests/unit_tests/test_optimizer.py
index a9dae9a47e0..98abe447a68 100644
--- a/tests/unit_tests/test_optimizer.py
+++ b/tests/unit_tests/test_optimizer.py
@@ -28,7 +28,7 @@
from megatron.core.transformer import TransformerConfig
from megatron.core.utils import is_te_min_version, is_torch_min_version
from tests.unit_tests.test_utilities import Utils
-from tests.unit_tests.test_utils import _deinit_distributed, _init_distributed
+from tests.unit_tests.test_utils import _init_distributed
try:
# Check if FP8 block scaling is available.
@@ -326,6 +326,43 @@ def to_cuda(d):
assert list(optimizer_2.state.values())[0]["momentum_buffer"].is_cuda
+def test_chained_optimizer_get_parameters():
+ """Test ChainedOptimizer.get_parameters() aggregates params from all sub-optimizers.
+
+ Regression test: without the get_parameters() override, ChainedOptimizer would
+ access self.optimizer which asserts only one optimizer exists, failing with VPP/MoE.
+ """
+
+ class MockOptimizer:
+ """Mock that mimics MegatronOptimizer's get_parameters() interface."""
+
+ def __init__(self, params):
+ self.params = list(params)
+ self.param_groups = [{"params": self.params}]
+
+ def get_parameters(self):
+ return self.params
+
+ net = Net()
+ all_params = list(net.parameters())
+
+ # Test empty
+ assert ChainedOptimizer([]).get_parameters() == []
+
+ # Test single optimizer
+ opt1 = MockOptimizer(all_params[:3])
+ assert ChainedOptimizer([opt1]).get_parameters() == opt1.params
+
+ # Test multiple optimizers (the case that previously failed)
+ opt2 = MockOptimizer(all_params[3:6])
+ opt3 = MockOptimizer(all_params[6:])
+ chained = ChainedOptimizer([opt1, opt2, opt3])
+ result = chained.get_parameters()
+
+ assert len(result) == len(all_params)
+ assert result == opt1.params + opt2.params + opt3.params
+
+
def test_precision_aware_fused_adam():
try:
from transformer_engine.pytorch.optimizers import FusedAdam
@@ -492,6 +529,136 @@ def run_model(model, input, optim, fp8_recipe, fp8_recipe_settings):
test_optim.load_state_dict(state_dict)
+@pytest.mark.parametrize("use_precision_aware", [True, False])
+def test_distrib_optimizer_save_load_with_non_tensor_state(use_precision_aware):
+ """Test that save/load of distributed optimizer handles non-tensor state entries.
+
+ Optimizers may store non-tensor entries (e.g. `found_inf: bool`) in the per-parameter
+ state dict. The distrib_optimizer's _get_main_param_and_optimizer_states and
+ _set_main_param_and_optimizer_states must skip these to avoid crashes when calling
+ tensor operations (.copy_(), get_unscaled_state, set_scaled_state) on non-tensors.
+
+ Tests both the precision-aware path (TE FusedAdam with scaled states) and the
+ non-precision-aware path (standard optimizer with .copy_()).
+ """
+ if use_precision_aware:
+ try:
+ from transformer_engine.pytorch.optimizers import FusedAdam
+ except ImportError:
+ pytest.skip("TE FusedAdam not available")
+
+ import inspect
+
+ adam_args = inspect.signature(FusedAdam).parameters
+ arg_names = [
+ "master_weight_dtype",
+ "exp_avg_dtype",
+ "exp_avg_sq_dtype",
+ "use_decoupled_grad",
+ ]
+ for name in arg_names:
+ if name not in adam_args:
+ pytest.skip("TE FusedAdam does not support precision-aware args")
+
+ world = int(os.getenv('WORLD_SIZE', '1'))
+ rank = int(os.getenv('RANK', '0'))
+
+ _init_distributed(world, rank)
+ Utils.initialize_model_parallel()
+
+ model = torch.nn.Linear(100, 100, bias=False, dtype=torch.bfloat16, device='cuda')
+ model.requires_grad_(True)
+ model.weight.data.fill_(1.0)
+ ddp_config = DistributedDataParallelConfig(use_distributed_optimizer=True)
+ model = DistributedDataParallel(
+ TransformerConfig(num_attention_heads=1, num_layers=1), ddp_config, model
+ )
+
+ optimizer_config = OptimizerConfig(
+ optimizer='adam',
+ lr=0.01,
+ bf16=True,
+ use_distributed_optimizer=True,
+ use_precision_aware_optimizer=use_precision_aware,
+ main_params_dtype=torch.float32,
+ main_grads_dtype=torch.float32,
+ exp_avg_dtype=torch.float32,
+ exp_avg_sq_dtype=torch.float32,
+ )
+ optim = get_megatron_optimizer(optimizer_config, [model])
+
+ # Run a training step to populate optimizer state
+ input_data = torch.randn(8, 100, dtype=torch.bfloat16, device='cuda')
+ output = model(input_data)
+ loss = output.sum()
+ loss.backward()
+ optim.step()
+
+ # Access the underlying distrib_optimizer
+ distrib_optim = optim.chained_optimizers[0]
+ if use_precision_aware:
+ assert distrib_optim.config.use_precision_aware_optimizer_no_fp8_or_ds_fp8
+
+ # Inject non-tensor entries into optimizer state (simulates found_inf, etc.)
+ inner_optimizer = distrib_optim.optimizer
+ for param in inner_optimizer.state:
+ inner_optimizer.state[param]['found_inf'] = False
+ inner_optimizer.state[param]['non_tensor_int'] = 42
+
+ # Test 1: _get_main_param_and_optimizer_states should skip non-tensor entries
+ for gbuf_range_maps in distrib_optim.gbuf_ranges:
+ for gbuf_range_map_for_all_buckets in gbuf_range_maps.values():
+ for gbuf_range_map in gbuf_range_map_for_all_buckets:
+ for model_param in gbuf_range_map["param_map"]:
+ tensors = distrib_optim._get_main_param_and_optimizer_states(model_param)
+ for k, v in tensors.items():
+ assert isinstance(
+ v, torch.Tensor
+ ), f"Non-tensor value for key '{k}': {type(v)}"
+
+ # Test 2: Full save/load roundtrip via dp_reshardable path
+ saved_state = distrib_optim.get_parameter_state_dp_reshardable()
+
+ # Verify saved state doesn't contain non-tensor entries (except metadata keys)
+ metadata_keys = {"per_bucket_numel", "per_bucket_numel_unpadded"}
+ for key, value in saved_state.items():
+ if key in metadata_keys:
+ continue
+ for dtype, buckets_state in value.items():
+ for bucket_state in buckets_state:
+ for param_dict in bucket_state:
+ for k, v in param_dict.items():
+ if k in ('gbuf_local_start', 'gbuf_local_end', 'padding'):
+ continue
+ assert isinstance(
+ v, torch.Tensor
+ ), f"Non-tensor in saved state key '{k}': {type(v)}"
+
+ # Test 3: load_parameter_state_from_dp_reshardable should not crash
+ # Add 'padding' key required by the load path (normally added by fully_reshardable save)
+ for key, value in saved_state.items():
+ if key in metadata_keys:
+ continue
+ for dtype, buckets_state in value.items():
+ for bucket_state in buckets_state:
+ for param_dict in bucket_state:
+ param_dict['padding'] = False
+ distrib_optim.load_parameter_state_from_dp_reshardable(saved_state)
+
+ # Test 4: Inject non-tensor entries directly into the saved state and verify load handles them
+ for key, value in saved_state.items():
+ if key in metadata_keys:
+ continue
+ for dtype, buckets_state in value.items():
+ for bucket_state in buckets_state:
+ for param_dict in bucket_state:
+ param_dict['found_inf'] = False
+ param_dict['step_count'] = 42
+
+ # This should not crash - non-tensor entries should be skipped
+ distrib_optim.load_parameter_state_from_dp_reshardable(saved_state)
+
+
@pytest.mark.parametrize("use_distributed_optimizer", [False, True])
@pytest.mark.parametrize("precision", ['bf16', 'fp32'])
def test_optim_sharded_state_dict(use_distributed_optimizer: bool, precision: str):
diff --git a/tests/unit_tests/test_optimizer_param_scheduler.py b/tests/unit_tests/test_optimizer_param_scheduler.py
index 9b781694546..670ca92c992 100644
--- a/tests/unit_tests/test_optimizer_param_scheduler.py
+++ b/tests/unit_tests/test_optimizer_param_scheduler.py
@@ -1,10 +1,13 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
import math
from unittest.mock import MagicMock
import pytest
-from megatron.core.optimizer_param_scheduler import ( # Adjust import according to your module path
+from megatron.core.optimizer_param_scheduler import (
OptimizerParamScheduler,
+ get_canonical_lr_for_logging,
)
@@ -182,6 +185,51 @@ def test_step_function(mock_optimizer):
assert math.isclose(param_group['weight_decay'], 0.01, rel_tol=1e-5)
+def test_step_updates_empty_param_groups():
+ """Empty param groups (rank-alignment stubs) must still receive lr updates.
+
+ get_canonical_lr_for_logging reads lr from default_config groups regardless
+ of whether they hold parameters, so step() must not skip them.
+ """
+ optimizer = MagicMock()
+ # lr and weight_decay are set by the scheduler's step() method
+ optimizer.param_groups = [
+ # Non-default group with its own max_lr override (lr will differ from the canonical schedule)
+ {'params': [1, 2], "min_lr": 0.001, "max_lr": 0.2, "default_config": False},
+ # Model parallelism may leave default_config groups empty on some ranks
+ {'params': [], "wd_mult": 0.0, 'default_config': True},
+ ]
+ scheduler = OptimizerParamScheduler(
+ optimizer=optimizer,
+ init_lr=0.01,
+ max_lr=0.1,
+ min_lr=0.001,
+ lr_warmup_steps=100,
+ lr_decay_steps=1000,
+ lr_decay_style='linear',
+ start_wd=0.0,
+ end_wd=0.1,
+ wd_incr_steps=1000,
+ wd_incr_style='linear',
+ )
+
+ scheduler.step(100)
+ non_empty, empty = optimizer.param_groups
+
+ # Verify learning rates: at step 100 warmup is complete so lr == max_lr
+ assert "lr" in non_empty, "non-empty param group must have an lr"
+ assert "lr" in empty, "empty param group must have an lr"
+ assert non_empty['lr'] == pytest.approx(0.2) # warmup complete → this group's max_lr override
+ assert empty['lr'] == pytest.approx(0.1) # warmup complete → scheduler's default max_lr
+ assert get_canonical_lr_for_logging(optimizer.param_groups) == pytest.approx(0.1)
+
+ # Verify weight decay: linear from 0.0 to 0.1 over 1000 steps → base wd is 0.01 at step 100
+ assert "weight_decay" in non_empty, "non-empty param group must have a weight decay"
+ assert "weight_decay" in empty, "empty param group must have a weight decay"
+ assert non_empty['weight_decay'] == pytest.approx(0.01) # base wd, no wd_mult override
+ assert empty['weight_decay'] == pytest.approx(0.0) # base wd * wd_mult=0.0
+
+
def test_state_dict(mock_optimizer):
scheduler = OptimizerParamScheduler(
optimizer=mock_optimizer,
@@ -249,3 +297,54 @@ def test_load_state_dict(mock_optimizer):
assert scheduler.end_wd == 0.2
assert scheduler.wd_incr_steps == 500
assert scheduler.wd_incr_style == 'cosine'
+
+
+# ── get_canonical_lr_for_logging tests ──────────────────────────────────────
+#
+# Returns the lr of the first default_config=True param group. In practice
+# the scheduler always sets a valid lr on every group (including empty
+# rank-alignment stubs), so a default_config=True group with a float lr is
+# always present.
+
+
+class TestGetCanonicalLrForLogging:
+ """Tests for get_canonical_lr_for_logging."""
+
+ def test_single_default_config_group(self):
+ """Typical case: one default_config group with a valid lr."""
+ param_groups = [{'lr': 0.05, 'default_config': True}]
+ assert get_canonical_lr_for_logging(param_groups) == 0.05
+
+ def test_default_config_with_non_default_groups(self):
+ """default_config group is returned even when non-default groups are present."""
+ param_groups = [{'lr': 0.001, 'default_config': True}, {'lr': 0.999}]
+ assert get_canonical_lr_for_logging(param_groups) == 0.001
+
+ def test_default_config_after_non_default(self):
+ """default_config group is found even when it is not first in the list."""
+ param_groups = [{'lr': 0.50}, {'lr': 0.01, 'default_config': True}]
+ assert get_canonical_lr_for_logging(param_groups) == 0.01
+
+ def test_no_default_config_groups(self):
+ """Returns None when no group has default_config=True."""
+ param_groups = [{'lr': 0.50}, {'lr': 0.01}]
+ assert get_canonical_lr_for_logging(param_groups) is None
+
+ def test_missing_lr_key(self):
+ """Returns None (not KeyError) when the default_config group lacks an 'lr' key."""
+ param_groups = [{'default_config': True}]
+ assert get_canonical_lr_for_logging(param_groups) is None
+
+ def test_empty_param_groups(self):
+ """Returns None when there are no param groups at all."""
+ assert get_canonical_lr_for_logging([]) is None
+
+ def test_no_default_config_no_lr(self):
+ """Returns None when groups exist but none are default_config."""
+ param_groups = [{'params': []}]
+ assert get_canonical_lr_for_logging(param_groups) is None
+
+ def test_lr_zero_is_valid(self):
+ """lr=0.0 is a legitimate value, not to be confused with None."""
+ param_groups = [{'lr': 0.0, 'default_config': True}]
+ assert get_canonical_lr_for_logging(param_groups) == 0.0
diff --git a/tests/unit_tests/test_tokenizer.py b/tests/unit_tests/test_tokenizer.py
deleted file mode 100644
index 556cef81407..00000000000
--- a/tests/unit_tests/test_tokenizer.py
+++ /dev/null
@@ -1,278 +0,0 @@
-# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-
-import base64
-import json
-from argparse import Namespace
-from pathlib import Path
-
-import numpy as np
-import pytest
-import requests
-
-from megatron.training import tokenizer
-from megatron.training.tokenizer.gpt2_tokenization import PRETRAINED_VOCAB_ARCHIVE_MAP
-from megatron.training.tokenizer.multimodal_tokenizer import MultimodalTokenizer
-
-TOKENIZER_DIR = Path("~/data/tokenizers").expanduser()
-
-# Copied over from test_preprocess_data.py
-from tests.unit_tests.data.test_preprocess_data import __LOCAL_GPT2_VOCAB
-
-GPT2_VOCAB_SIZE = 32768
-
-
-def offsets_to_substrs(offsets, string):
- return [string[start:end] for start, end in zip([0] + offsets, offsets + [len(string)])]
-
-
-def local_test_specs():
- return [
- Namespace(
- rank=0,
- tensor_model_parallel_size=8,
- make_vocab_size_divisible_by=128,
- tokenizer_type="GPTSentencePieceTokenizer",
- tokenizer_model=f"{TOKENIZER_DIR}/nemotron_2_256k.model",
- ),
- Namespace(
- rank=0,
- vocab_size=131072,
- make_vocab_size_divisible_by=128,
- tensor_model_parallel_size=8,
- tokenizer_type="TikTokenizer",
- tokenizer_model=f"{TOKENIZER_DIR}/multiMixV8.gpt4o_nc_sd.500000.128k.vocab.json",
- tiktoken_pattern="v2",
- tiktoken_num_special_tokens=1000,
- tokenizer_special_tokens=["", "", " "],
- ),
- Namespace(
- rank=0,
- vocab_size=131072,
- make_vocab_size_divisible_by=128,
- tensor_model_parallel_size=8,
- tokenizer_type="TikTokenizer",
- tokenizer_model=f"{TOKENIZER_DIR}/multiMixV5_fix_default_500000_128k.vocab.json",
- tiktoken_pattern="v1",
- tiktoken_num_special_tokens=1000,
- tokenizer_special_tokens=["", "", " "],
- ),
- Namespace(
- rank=0,
- vocab_size=128000,
- make_vocab_size_divisible_by=128,
- tensor_model_parallel_size=8,
- tokenizer_type="HuggingFaceTokenizer",
- tokenizer_model="meta-llama/Llama-2-7b-hf",
- ),
- Namespace(
- rank=0,
- vocab_size=128000,
- make_vocab_size_divisible_by=128,
- tensor_model_parallel_size=8,
- tokenizer_type="HuggingFaceTokenizer",
- tokenizer_model="meta-llama/Meta-Llama-3.1-8B",
- ),
- ]
-
-
-@pytest.fixture(scope="session")
-def gpt2_tiktok_vocab(tmp_path_factory):
-
- if Path(__LOCAL_GPT2_VOCAB).exists():
- with open(__LOCAL_GPT2_VOCAB, "r", encoding="utf-8") as reader:
- gpt2_vocab = json.load(reader)
- else:
- gpt2_vocab = json.loads(requests.get(PRETRAINED_VOCAB_ARCHIVE_MAP["gpt2"]).content)
-
- N = 256
- tiktok_vocab = [
- {"token_bytes": base64.b64encode(bytes([i])).decode("utf-8"), "token_str": str(i)}
- for i in range(N)
- ]
- tiktok_vocab_bytes = {x["token_bytes"] for x in tiktok_vocab}
-
- tiktok_vocab += [
- {"token_bytes": base64.b64encode(token.encode('utf-8')).decode("utf-8"), "token_str": token}
- for token in gpt2_vocab
- if base64.b64encode(token.encode('utf-8')).decode("utf-8") not in tiktok_vocab_bytes
- ]
-
- for i, entry in enumerate(tiktok_vocab):
- entry["rank"] = i
-
- for i, x in enumerate(tiktok_vocab):
- assert x.keys() == {"rank", "token_bytes", "token_str"}
- assert x["rank"] == i
- merge = base64.b64decode(x["token_bytes"])
- assert i >= 256 or merge == bytes([i]), f"{i} {merge} {bytes([i])}"
-
- file_name = tmp_path_factory.mktemp("data") / "gpt2_vocab.json"
- with open(file_name, "w") as f:
- json.dump(tiktok_vocab, f)
-
- return Namespace(
- rank=0,
- vocab_size=32768,
- make_vocab_size_divisible_by=128,
- tensor_model_parallel_size=8,
- tokenizer_type="TikTokenizer",
- tokenizer_model=str(file_name),
- tiktoken_pattern="v1",
- tiktoken_num_special_tokens=1000,
- tokenizer_special_tokens=["", "", " "],
- )
-
-
-@pytest.mark.parametrize("args", local_test_specs())
-def test_tokenizer(args):
- if not TOKENIZER_DIR.exists():
- pytest.skip("Skipping tokenizer tests because the tokenizer directory does not exist")
-
- tok = tokenizer.build_tokenizer(args)
- run_tokenizer_tests(tok)
-
-
-def test_gpt2_tiktok_tokenizer(gpt2_tiktok_vocab):
- tok = tokenizer.build_tokenizer(gpt2_tiktok_vocab)
- run_tokenizer_tests(tok)
-
-
-def run_tokenizer_tests(tok):
- string1 = (
- "The following are multiple choice questions (with answers) about college biology.\n"
- "Monoclonal antisera are distinguished from polyclonal antisera in which of the "
- "following ways?\n"
- "A. Each type of antibody in a monoclonal antiserum reacts against a single region of "
- "a single antigen; each type of antibody in a polyclonal antiserum reacts against "
- "multiple regions of different antigens.\n"
- "B. A monoclonal antibody reacts against multiple regions of a single antigen; a "
- "polyclonal antibody reacts against a single region of related antigens.\n"
- "C. A monoclonal antiserum contains antibodies secreted from the descendants of a "
- "single B lymphocyte; a polyclonal antiserum contains antibodies secreted from the "
- "descendants of different B lymphocytes.\n"
- "D. A monoclonal antiserum contains antibodies secreted from the descendants of a "
- "single B lymphocyte; a polyclonal antiserum contains antibodies secreted from the "
- "descendants of both B and T lymphocytes.\n"
- "Answer: C"
- )
- string2 = "Жизнь прекрасна и удивительна"
- string3 = "お誕生日おめでとう"
- strings = [string1, string2, string3]
-
- for test_string in strings:
- toks = tok.tokenize(test_string)
- offsets = tok.offsets(toks, test_string)
- dec = offsets_to_substrs(offsets, test_string)
- detok_str = ''.join(dec)
- # the following is not necessarily true by construction above,
- # since the many tokenizers may operate at the byte level and not
- # only at the character level.
- assert (
- detok_str == test_string
- ), f"Detokenized string {detok_str} does not match original {test_string}"
- assert len(toks) == len(
- offsets
- ), f"Tokenized string {toks} does not match original {offsets}"
-
-
-def test_null_tokenizer():
- args = Namespace(
- tokenizer_type="NullTokenizer",
- rank=0,
- vocab_size=128000,
- make_vocab_size_divisible_by=128,
- tensor_model_parallel_size=8,
- )
- tok = tokenizer.build_tokenizer(args)
- test_string = "1 23 456 789"
- toks = tok.tokenize(test_string)
- offsets = tok.offsets(toks, test_string)
- dec = offsets_to_substrs(offsets, test_string)
- detok_str = ''.join(dec)
-
- assert (
- detok_str == test_string
- ), f"Detokenized string {detok_str} does not match original {test_string}"
- assert len(toks) == len(offsets), f"Tokenized string {toks} does not match original {offsets}"
-
-
-class MockUnderlyingTokenizer:
- """Mock tokenizer for testing purposes."""
-
- def __init__(self):
- self.pad_token_id = 256
-
- def __len__(self):
- return 256
-
- def encode(self, text: str) -> list[int]:
- """Convert text to a list of token IDs."""
- return [ord(c) for c in text]
-
- def decode(self, tokens: list[int]) -> str:
- """Convert list of token IDs to plaintext."""
- return "".join([chr(t) for t in tokens])
-
- def apply_chat_template(self, conversation: list[dict], *args, **kwargs) -> list[int]:
- """Convert a conversation to token IDs."""
- out = []
- for turn in conversation:
- turn_tokens = self.encode(f"{turn['role']}:{turn['content']}")
- out.extend(turn_tokens)
-
- if kwargs.get("return_tensors", None) == "np":
- return [np.array(out)]
-
- return out
-
- def convert_tokens_to_ids(self, text: str) -> list[int]:
- """Convert plaintext to token IDs."""
- return self.encode(text)
-
- def add_tokens(self, extra_tokens: list[str], *args, **kwargs) -> int:
- """Add tokens to the tokenizer. No-op for this mock tokenizer."""
- return len(extra_tokens)
-
-
-def test_multimodal_tokenizer():
- """Test MultimodalTokenizer."""
- underlying = MockUnderlyingTokenizer()
- prompt_format = "chatml"
- special_tokens = [""]
- image_tag_type = ""
- tokenizer = MultimodalTokenizer(underlying, prompt_format, special_tokens, image_tag_type)
-
- # Simple encode - decode roundtrip.
- assert (
- tokenizer.detokenize(tokenizer.tokenize("abc")) == "abc"
- ), "encode-decode roundtrip failed"
-
- # Apply chat template.
- conversation = [
- {"role": "system", "content": "abc"},
- {"role": "user", "content": "123"},
- {"role": "assistant", "content": "xyz"},
- ]
- conv_tokens = tokenizer.tokenize_conversation(
- conversation, return_target=False, add_generation_prompt=False
- )
- assert len(conv_tokens) > 0, "failed to tokenize conversation"
-
- conv_tokens, target_tokens = tokenizer.tokenize_conversation(
- conversation, return_target=True, add_generation_prompt=True
- )
- assert len(conv_tokens) > 0 and len(conv_tokens) == len(
- target_tokens
- ), "failed to tokenize conversation and return target tokens"
-
- # Try converting tokens to ids.
- assert tokenizer.convert_tokens_to_ids("a"), "failed to convert tokens to ids."
-
- # Try image tags.
- image_tag_type = "nvlm"
- tokenizer = MultimodalTokenizer(underlying, prompt_format, special_tokens, image_tag_type)
-
- assert tokenizer._apply_image_tag("hello") == "hello"}]) == [
- {"role": "user", "content": "", "eos_token": ""}
tokenizer = MegatronTokenizer.from_pretrained(
- "/opt/data/tokenizers/huggingface",
+ str(unit_test_data_path("tokenizers", "huggingface")),
metadata_path=metadata,
chat_template=chat_template,
+ include_special_tokens=False,
**special_tokens,
)
@@ -112,14 +124,54 @@ def test_hf_tokenizer():
assert tokenizer.vocab_size == 128258
+# HuggingFaceTokenizer.ids_to_text and include_special_tokens (--tokenizer-hf-include-special-tokens).
+# Uses same local path as test_hf_tokenizer; tests EOS stripping vs keeping in detokenized output (e.g. RL).
+LOCAL_HF_TOKENIZER_PATH = str(unit_test_data_path("tokenizers", "huggingface"))
+
+
+def _eos_in_text(text: str, eos_token: str) -> bool:
+ return eos_token in text or text.endswith(eos_token.strip())
+
+
+@pytest.mark.skipif(not HAVE_TRANSFORMERS, reason="transformers not installed")
+@pytest.mark.parametrize("include_special_tokens", [True, False])
+@pytest.mark.parametrize("remove_special_tokens", [True, False])
+def test_hf_ids_to_text_eos_with_include_and_remove_special_tokens(
+ include_special_tokens, remove_special_tokens
+):
+ """ids_to_text EOS presence: parametrized on include_special_tokens and remove_special_tokens.
+ When remove_special_tokens=True, EOS is stripped; when False, EOS is kept (explicit overrides default).
+ """
+ try:
+ tok = HuggingFaceTokenizer(
+ LOCAL_HF_TOKENIZER_PATH, include_special_tokens=include_special_tokens
+ )
+ except Exception:
+ pytest.skip("Could not load local HuggingFace tokenizer (path not available)")
+ eos_id = tok.eos_id
+ ids = tok.text_to_ids("hello") + [eos_id]
+ text = tok.ids_to_text(ids, remove_special_tokens=remove_special_tokens)
+ eos_expected = not remove_special_tokens
+ if eos_expected:
+ assert _eos_in_text(text, tok.tokenizer.eos_token), (
+ f"Expected EOS in output for include_special_tokens={include_special_tokens}, "
+ f"remove_special_tokens={remove_special_tokens}. Got: {text!r}"
+ )
+ else:
+ assert tok.tokenizer.eos_token not in text, (
+ f"Expected EOS stripped for include_special_tokens={include_special_tokens}, "
+ f"remove_special_tokens={remove_special_tokens}. Got: {text!r}"
+ )
+
+
def test_megatron_tokenizer():
# Load tokenizer with additional special tokens
special_tokens = {}
special_tokens['additional_special_tokens'] = [f'' for i in range(100)]
metadata = {"library": "megatron", "model_type": "gpt"}
- vocab_file = "/opt/data/tokenizers/megatron/gpt2-vocab.json"
- merges_file = "/opt/data/tokenizers/megatron/gpt2-vocab.json"
+ vocab_file = str(unit_test_data_path("tokenizers", "megatron", "gpt2-vocab.json"))
+ merges_file = str(unit_test_data_path("tokenizers", "megatron", "gpt2-vocab.json"))
tokenizer = MegatronTokenizer.from_pretrained(
tokenizer_path="GPT2BPETokenizer",
metadata_path=metadata,
@@ -158,7 +210,7 @@ def test_tiktoken_tokenizer():
# Load tiktoken tokenizer
chat_template = get_chat_template()
tokenizer = MegatronTokenizer.from_pretrained(
- tokenizer_path="/opt/data/tokenizers/tiktoken/tiktoken.vocab.json",
+ tokenizer_path=str(unit_test_data_path("tokenizers", "tiktoken", "tiktoken.vocab.json")),
chat_template=chat_template,
vocab_size=131072,
)
@@ -196,7 +248,7 @@ def test_tiktoken_tokenizer():
def test_null_tokenizer():
- metadata = {"library": "null"}
+ metadata = {"library": "null-text"}
tokenizer = MegatronTokenizer.from_pretrained(metadata_path=metadata, vocab_size=131072)
ids = tokenizer.tokenize("11 325 97")
@@ -219,7 +271,7 @@ def test_bytelevel_tokenizer():
def test_write_metadata():
- tokenizer_path = "/opt/data/tokenizers/huggingface"
+ tokenizer_path = str(unit_test_data_path("tokenizers", "huggingface"))
chat_template = "test chat template"
tokenizer_library = "huggingface"
MegatronTokenizer.write_metadata(
@@ -255,3 +307,98 @@ class CustomTokenizerClass:
model_type="gpt",
overwrite=True,
)
+
+
+def test_multimodal_tokenizer():
+ """Test MegatronMultimodalTokenizer."""
+ prompt_format = "qwen2p0"
+ special_tokens = [""]
+ image_tag_type = "nvlm"
+ tokenizer = MegatronTokenizer.from_pretrained(
+ tokenizer_path=str(unit_test_data_path("tokenizers", "multimodal")),
+ metadata_path={"library": "multimodal"},
+ prompt_format=prompt_format,
+ special_tokens=special_tokens,
+ image_tag_type=image_tag_type,
+ )
+ # Simple encode - decode roundtrip.
+ assert (
+ tokenizer.detokenize(tokenizer.tokenize("abc")) == "abc"
+ ), "encode-decode roundtrip failed"
+
+ conversation = [
+ {"role": "system", "content": "You are a helpful assistant."},
+ {"role": "user", "content": "Hello! Can you summarize this image for me?"},
+ {"role": "user", "content": ""},
+ {"role": "assistant", "content": "Sure! The image shows a sunset over a mountain range."},
+ {"role": "user", "content": "Thanks! Can you also give a short poem about it?"},
+ ]
+
+ conv_tokens = tokenizer.tokenize_conversation(
+ conversation, return_target=False, add_generation_prompt=False
+ )
+ assert len(conv_tokens) > 0, "failed to tokenize conversation"
+
+ conv_tokens, target_tokens = tokenizer.tokenize_conversation(
+ conversation, return_target=True, add_generation_prompt=False
+ )
+ assert len(conv_tokens) > 0 and len(conv_tokens) == len(
+ target_tokens
+ ), "failed to tokenize conversation and return target tokens"
+
+ # Try converting tokens to ids.
+ assert tokenizer.convert_tokens_to_ids("a"), "failed to convert tokens to ids."
+
+ assert tokenizer._tokenizer._apply_image_tag("hello") == "hello"}]) == [
+ {"role": "user", "content": ""},
+ {"role": "assistant", "content": "Sure! The image shows a sunset over a mountain range."},
+ {"role": "user", "content": "Thanks! Can you also give a short poem about it?"},
+ ]
+
+ conv_tokens = tokenizer.tokenize_conversation(
+ conversation, return_target=False, add_generation_prompt=False
+ )
+ assert len(conv_tokens) > 0, "failed to tokenize conversation"
+
+ conv_tokens, target_tokens = tokenizer.tokenize_conversation(
+ conversation, return_target=True, add_generation_prompt=False
+ )
+ assert len(conv_tokens) > 0 and len(conv_tokens) == len(
+ target_tokens
+ ), "failed to tokenize conversation and return target tokens"
diff --git a/tests/unit_tests/transformer/experimental_attention_variant/test_absorbed_mla.py b/tests/unit_tests/transformer/experimental_attention_variant/test_absorbed_mla.py
new file mode 100644
index 00000000000..eb235501ad7
--- /dev/null
+++ b/tests/unit_tests/transformer/experimental_attention_variant/test_absorbed_mla.py
@@ -0,0 +1,399 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+import random
+from typing import List, Optional, Tuple
+
+import pytest
+import torch
+import torch.distributed as dist
+
+from megatron.core import parallel_state
+from megatron.core.extensions.transformer_engine_spec_provider import TESpecProvider
+from megatron.core.packed_seq_params import PackedSeqParams
+from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
+from megatron.core.transformer.enums import AttnMaskType
+from megatron.core.transformer.experimental_attention_variant.absorbed_mla import (
+ AbsorbedMLASelfAttention,
+ AbsorbedMLASelfAttentionSubmodules,
+)
+from megatron.core.transformer.identity_op import IdentityOp
+from megatron.core.transformer.multi_latent_attention import (
+ MLASelfAttention,
+ MLASelfAttentionSubmodules,
+)
+from megatron.core.transformer.transformer_config import MLATransformerConfig
+from megatron.core.utils import init_method_normal, scaled_init_method_normal
+from tests.unit_tests.test_utilities import Utils
+
+
+class MockCoreAttention(torch.nn.Module):
+ """Mock core attention for testing MLA computation flow."""
+
+ def __init__(self, *args, **kwargs):
+ super().__init__()
+ self.softmax_scale = kwargs.get("softmax_scale")
+ self.k_channels = kwargs.get("k_channels")
+ self.v_channels = kwargs.get("v_channels")
+ self.pg_collection = kwargs.get("pg_collection")
+
+ def forward(
+ self, q, k, v, *args, packed_seq_params: Optional[PackedSeqParams] = None, **kwargs
+ ):
+ """Mock forward pass."""
+ if packed_seq_params is None:
+ return self._forward_standard(q, k, v)
+ else:
+ return self._forward_thd(q, k, v, packed_seq_params)
+
+ def _forward_standard(self, q, k, v):
+ """Standard forward for [s, b, n, d] format."""
+ sq, b, n = q.shape[:3]
+ dtype = q.dtype
+ if v is None:
+ # Absorbed MLA
+ assert q.shape[-1] == self.k_channels
+ assert k.shape == (sq, b, 1, self.k_channels)
+ v = k[..., : self.v_channels]
+ k = k.expand(-1, -1, n, -1)
+ v = v.expand(-1, -1, n, -1)
+ else:
+ # Standard MLA
+ assert k.shape == q.shape
+ assert v.shape[:-1] == q.shape[:-1]
+
+ q = q.permute(1, 2, 0, 3).contiguous()
+ k = k.permute(1, 2, 3, 0).contiguous()
+ v = v.permute(1, 2, 0, 3).contiguous()
+
+ q = q.view(b * n, q.size(-2), q.size(-1)).float()
+ k = k.view(b * n, k.size(-2), k.size(-1)).float()
+ v = v.view(b * n, v.size(-2), v.size(-1)).float()
+
+ score = torch.bmm(q, k) * self.softmax_scale
+ score = torch.nn.functional.softmax(score, dim=-1, dtype=torch.float32)
+ out = torch.bmm(score, v)
+ out = out.to(dtype)
+ out = out.permute(1, 0, 2)
+ out = out.reshape(sq, b, -1)
+
+ return out
+
+ def _forward_thd(self, q, k, v, packed_seq_params):
+ """Forward for THD packed sequence format."""
+ cu_seqlens = packed_seq_params.cu_seqlens_q
+ num_seqs = len(cu_seqlens) - 1
+
+ sq, n = q.shape[:2]
+ dtype = q.dtype
+ if v is None:
+ # Absorbed MLA
+ assert q.shape[-1] == self.k_channels
+ assert k.shape == (sq, 1, self.k_channels)
+ v = k[..., : self.v_channels]
+ k = k.expand(-1, n, -1)
+ v = v.expand(-1, n, -1)
+ else:
+ # Standard MLA
+ assert k.shape == q.shape
+ assert v.shape[:-1] == q.shape[:-1]
+
+ out_list = []
+ for i in range(num_seqs):
+ start = cu_seqlens[i] // self.pg_collection.cp.size()
+ end = cu_seqlens[i + 1] // self.pg_collection.cp.size()
+ q_seq = q[start:end]
+ k_seq = k[start:end]
+ v_seq = v[start:end]
+
+ q_seq = q_seq.permute(1, 0, 2).contiguous().float()
+ k_seq = k_seq.permute(1, 2, 0).contiguous().float()
+ v_seq = v_seq.permute(1, 0, 2).contiguous().float()
+
+ score = torch.bmm(q_seq, k_seq) * self.softmax_scale
+ score = torch.nn.functional.softmax(score, dim=-1, dtype=torch.float32)
+ out = torch.bmm(score, v_seq)
+ out = out.to(dtype)
+ out = out.permute(1, 0, 2).contiguous()
+ out = out.reshape(out.shape[0], -1)
+ out_list.append(out)
+
+ return torch.cat(out_list, dim=0)
+
+
+def get_mock_mla_config(
+ tensor_model_parallel_size: int, context_parallel_size: int
+) -> MLATransformerConfig:
+ """Create test config with all attributes used in MLA."""
+ return MLATransformerConfig(
+ multi_latent_attention=True,
+ hidden_size=7168,
+ num_attention_heads=128,
+ q_lora_rank=1536,
+ kv_lora_rank=512,
+ qk_head_dim=128,
+ qk_pos_emb_head_dim=64,
+ v_head_dim=128,
+ add_bias_linear=False,
+ bf16=True,
+ params_dtype=torch.bfloat16,
+ layernorm_epsilon=1e-5,
+ normalization="RMSNorm",
+ layernorm_zero_centered_gamma=False,
+ expert_model_parallel_size=1,
+ tensor_model_parallel_size=tensor_model_parallel_size,
+ sequence_parallel=tensor_model_parallel_size > 1,
+ context_parallel_size=context_parallel_size,
+ apply_rope_fusion=False,
+ rope_type="yarn",
+ rotary_scaling_factor=40,
+ mscale=1.0,
+ mscale_all_dim=1.0,
+ rotary_base=10000,
+ original_max_position_embeddings=4096,
+ beta_fast=32,
+ beta_slow=1,
+ rotary_interleaved=False,
+ recompute_granularity=None,
+ fine_grained_activation_offloading=False,
+ gradient_accumulation_fusion=False,
+ fp8=False,
+ fp4=False,
+ init_method=init_method_normal(0.02),
+ output_layer_init_method=scaled_init_method_normal(0.02, 61, multiplier=2.0),
+ kv_channels=56,
+ num_query_groups=128,
+ batch_invariant_mode=False,
+ cache_mla_latents=False,
+ use_cpu_initialization=False,
+ perform_initialization=True,
+ symmetric_ar_type=None,
+ disable_parameter_transpose_cache=False,
+ init_model_with_meta_device=False,
+ delay_wgrad_compute=False,
+ tp_comm_overlap=False,
+ experimental_attention_variant=None,
+ softmax_scale=None,
+ )
+
+
+def get_absorbed_mla_submodules(
+ down_proj_use_column_parallel: bool, qk_layernorm: bool, rms_norm: bool
+) -> AbsorbedMLASelfAttentionSubmodules:
+ """Get submodules for AbsorbedMLASelfAttention testing."""
+ backend = TESpecProvider()
+ linear_q_down_proj = (
+ backend.column_parallel_linear() if down_proj_use_column_parallel else backend.linear()
+ )
+ linear_kv_down_proj = (
+ backend.column_parallel_linear() if down_proj_use_column_parallel else backend.linear()
+ )
+ qk_norm = backend.layer_norm(rms_norm=rms_norm, for_qk=True) if qk_layernorm else IdentityOp
+ return AbsorbedMLASelfAttentionSubmodules(
+ linear_q_proj=backend.column_parallel_linear(),
+ linear_q_down_proj=linear_q_down_proj,
+ linear_q_up_proj=backend.column_parallel_linear(),
+ linear_kv_down_proj=linear_kv_down_proj,
+ linear_k_up_proj=backend.column_parallel_linear(),
+ linear_v_up_proj=backend.column_parallel_linear(),
+ core_attention=MockCoreAttention,
+ linear_proj=backend.row_parallel_linear(),
+ q_layernorm=qk_norm,
+ kv_layernorm=qk_norm,
+ )
+
+
+def get_mla_submodules(
+ down_proj_use_column_parallel: bool, qk_layernorm: bool, rms_norm: bool
+) -> MLASelfAttentionSubmodules:
+ """Get submodules for AbsorbedMLASelfAttention testing."""
+ backend = TESpecProvider()
+ linear_q_down_proj = (
+ backend.column_parallel_linear() if down_proj_use_column_parallel else backend.linear()
+ )
+ linear_kv_down_proj = (
+ backend.column_parallel_linear() if down_proj_use_column_parallel else backend.linear()
+ )
+ qk_norm = backend.layer_norm(rms_norm=rms_norm, for_qk=True) if qk_layernorm else IdentityOp
+ return MLASelfAttentionSubmodules(
+ linear_q_proj=backend.column_parallel_linear(),
+ linear_q_down_proj=linear_q_down_proj,
+ linear_q_up_proj=backend.column_parallel_linear(),
+ linear_kv_down_proj=linear_kv_down_proj,
+ linear_kv_up_proj=backend.column_parallel_linear(),
+ core_attention=MockCoreAttention,
+ linear_proj=backend.row_parallel_linear(),
+ q_layernorm=qk_norm,
+ kv_layernorm=qk_norm,
+ )
+
+
+@pytest.mark.parametrize("tp_cp", [[1, 1], [2, 1], [1, 2], [2, 2]])
+@pytest.mark.parametrize("qkv_format", ['sbhd', 'thd'])
+@pytest.mark.parametrize("down_proj_use_column_parallel", [False, True])
+def test_functionality(tp_cp: List[int], qkv_format: str, down_proj_use_column_parallel: bool):
+ """Test that AbsorbedMLASelfAttention is equivalent to standard MLA."""
+ tp_size, cp_size = tp_cp
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=tp_size, context_parallel_size=cp_size
+ )
+ model_parallel_cuda_manual_seed(123)
+
+ # Create model
+ config = get_mock_mla_config(tensor_model_parallel_size=tp_size, context_parallel_size=cp_size)
+ absorbed_submodules = get_absorbed_mla_submodules(
+ down_proj_use_column_parallel=down_proj_use_column_parallel,
+ qk_layernorm=True,
+ rms_norm=True,
+ )
+ standard_submodules = get_mla_submodules(
+ down_proj_use_column_parallel=down_proj_use_column_parallel,
+ qk_layernorm=True,
+ rms_norm=True,
+ )
+ absorbed_mla = AbsorbedMLASelfAttention(
+ config=config,
+ submodules=absorbed_submodules,
+ layer_number=0,
+ attn_mask_type=AttnMaskType.causal,
+ cp_comm_type="all_gather" if cp_size > 1 else None,
+ pg_collection=None,
+ ).cuda()
+ standard_mla = MLASelfAttention(
+ config=config,
+ submodules=standard_submodules,
+ layer_number=0,
+ attn_mask_type=AttnMaskType.causal,
+ cp_comm_type="all_gather" if cp_size > 1 else None,
+ pg_collection=None,
+ ).cuda()
+
+ state_dict = standard_mla.state_dict()
+ absorbed_mla.load_state_dict(state_dict)
+
+ # Prepare random data
+ if qkv_format == 'thd':
+ # Create random seqlens
+ num_seqs, min_len, max_len = 3, 128, 1024
+ divisor = tp_size * cp_size * 2
+ random.seed(42)
+ seqlens = [random.randint(min_len, max_len) // divisor * divisor for _ in range(num_seqs)]
+ # Create cumulative sequence lengths
+ cu_seqlens = [0]
+ for length in seqlens:
+ cu_seqlens.append(cu_seqlens[-1] + length)
+ total_tokens = cu_seqlens[-1]
+ cu_seqlens = torch.IntTensor(cu_seqlens).cuda()
+ max_seqlen = max(seqlens)
+ # Create packed sequence parameters
+ packed_seq_params = PackedSeqParams(
+ cu_seqlens_q=cu_seqlens,
+ cu_seqlens_q_padded=cu_seqlens,
+ cu_seqlens_kv=cu_seqlens,
+ cu_seqlens_kv_padded=cu_seqlens,
+ max_seqlen_q=max_seqlen,
+ max_seqlen_kv=max_seqlen,
+ qkv_format='thd',
+ )
+ hidden_states = torch.randn(
+ (total_tokens // tp_size // cp_size, 1, config.hidden_size),
+ dtype=torch.bfloat16,
+ device='cuda',
+ )
+ grads = torch.randn_like(hidden_states)
+ else:
+ seqlen = 1024 // tp_size // cp_size
+ hidden_states = torch.randn((seqlen, 3, 7168), dtype=torch.bfloat16, device='cuda')
+ grads = torch.randn_like(hidden_states)
+ packed_seq_params = None
+
+ # Forward & Backward
+ for name, param in absorbed_mla.named_parameters():
+ if param.grad is not None:
+ param.grad.zero_()
+ absorbed_outputs, _ = absorbed_mla(
+ hidden_states, attention_mask=None, packed_seq_params=packed_seq_params
+ )
+ absorbed_outputs.backward(grads)
+
+ for name, param in standard_mla.named_parameters():
+ if param.grad is not None:
+ param.grad.zero_()
+ standard_outputs, _ = standard_mla(
+ hidden_states, attention_mask=None, packed_seq_params=packed_seq_params
+ )
+ standard_outputs.backward(grads)
+
+ def _calculate_tensor_similarity(x, y):
+ x, y = x.data.double(), y.data.double()
+ denominator = (x * x + y * y).sum()
+ if denominator == 0:
+ return 1
+ sim = 2 * (x * y).sum() / denominator
+ return sim
+
+ # Compute cosine similarity
+ absorbed_flat = absorbed_outputs.flatten().float()
+ standard_flat = standard_outputs.flatten().float()
+ cosine_sim = torch.nn.functional.cosine_similarity(
+ absorbed_flat.unsqueeze(0), standard_flat.unsqueeze(0)
+ ).item()
+ assert cosine_sim > 0.9999, f"output cosine similarity = {cosine_sim} < 0.9999"
+ assert _calculate_tensor_similarity(absorbed_outputs, standard_outputs) > 0.9999
+ torch.testing.assert_close(absorbed_outputs, standard_outputs, atol=5e-3, rtol=5e-3)
+
+ for name, param in absorbed_mla.named_parameters():
+ assert param.grad is not None
+ for name, param in standard_mla.named_parameters():
+ assert param.grad is not None
+
+ # Compare gradients with cosine similarity
+ absorbed_grads = dict(absorbed_mla.named_parameters())
+ standard_grads = dict(standard_mla.named_parameters())
+
+ # Map parameter names between absorbed and standard MLA
+ # Most parameters have the same name, except for K/V up proj
+ for name, param in standard_grads.items():
+ if 'linear_kv_up_proj' in name:
+ # Special handling: combine k and v up proj grads from absorbed_mla
+ k_name = name.replace('linear_kv_up_proj', 'linear_k_up_proj')
+ v_name = name.replace('linear_kv_up_proj', 'linear_v_up_proj')
+
+ k_grad = absorbed_grads[k_name].grad
+ v_grad = absorbed_grads[v_name].grad
+
+ # Combine k and v grads (interleaved by head)
+ # k_grad: [n * qk_head_dim, kv_lora_rank]
+ # v_grad: [n * v_head_dim, kv_lora_rank]
+ # combined: [n * (qk_head_dim + v_head_dim), kv_lora_rank]
+ n_heads = absorbed_mla.num_attention_heads_per_partition
+ qk_head_dim = absorbed_mla.config.qk_head_dim
+ v_head_dim = absorbed_mla.config.v_head_dim
+ kv_lora_rank = absorbed_mla.config.kv_lora_rank
+
+ k_grad_3d = k_grad.view(n_heads, qk_head_dim, kv_lora_rank)
+ v_grad_3d = v_grad.view(n_heads, v_head_dim, kv_lora_rank)
+ combined_grad_3d = torch.cat([k_grad_3d, v_grad_3d], dim=1)
+ combined_grad = combined_grad_3d.view(-1, kv_lora_rank)
+
+ absorbed_grad_flat = combined_grad.flatten().float()
+ standard_grad_flat = param.grad.flatten().float()
+
+ cos_sim = torch.nn.functional.cosine_similarity(
+ absorbed_grad_flat.unsqueeze(0), standard_grad_flat.unsqueeze(0)
+ ).item()
+ assert cos_sim > 0.9999, f"name: {name}, cosine similarity = {cos_sim} < 0.9999"
+ assert _calculate_tensor_similarity(combined_grad, param.grad) > 0.9999
+ else:
+ absorbed_grad = absorbed_grads[name].grad
+ standard_grad = param.grad
+
+ absorbed_grad_flat = absorbed_grad.flatten().float()
+ standard_grad_flat = standard_grad.flatten().float()
+
+ cos_sim = torch.nn.functional.cosine_similarity(
+ absorbed_grad_flat.unsqueeze(0), standard_grad_flat.unsqueeze(0)
+ ).item()
+ assert cos_sim > 0.9999, f"name: {name}, cosine similarity = {cos_sim} < 0.9999"
+ assert _calculate_tensor_similarity(absorbed_grad, standard_grad) > 0.9999
+
+ Utils.destroy_model_parallel()
diff --git a/tests/unit_tests/transformer/test_attention_variant_dsa.py b/tests/unit_tests/transformer/experimental_attention_variant/test_attention_variant_dsa.py
similarity index 80%
rename from tests/unit_tests/transformer/test_attention_variant_dsa.py
rename to tests/unit_tests/transformer/experimental_attention_variant/test_attention_variant_dsa.py
index bd106aa6f0e..96253a4ca10 100644
--- a/tests/unit_tests/transformer/test_attention_variant_dsa.py
+++ b/tests/unit_tests/transformer/experimental_attention_variant/test_attention_variant_dsa.py
@@ -17,7 +17,10 @@
DSAIndexerSubmodules,
DSAttention,
DSAttentionSubmodules,
+ FusedDSAIndexerLoss,
+ _compute_index_scores,
compute_dsa_indexer_loss,
+ fused_qk_topk_naive,
rotate_activation,
)
from megatron.core.transformer.transformer_config import MLATransformerConfig
@@ -265,6 +268,320 @@ def test_backward_pass(self):
), f"Gradient should be scaled by loss scale, expected {expected_grad_per_element}, got {dummy_input.grad[0].item()}"
+@pytest.mark.parametrize("seqlen_and_topk", [[16, 8], [32, 16], [64, 32]])
+@pytest.mark.parametrize("sparse_loss", [False, True])
+class TestFusedDSAIndexerLossGradient:
+ """Test that FusedDSAIndexerLoss manual backward matches autograd backward."""
+
+ @pytest.fixture(scope='function', autouse=True)
+ def setup_method(self):
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=1, pipeline_model_parallel_size=1
+ )
+ self.pg_collection = ProcessGroupCollection.use_mpu_process_groups(required_pgs=['tp'])
+ yield
+ Utils.destroy_model_parallel()
+
+ @pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA not available")
+ def test_fused_indexer_loss_gradient_matches_autograd(self, seqlen_and_topk, sparse_loss):
+ """
+ Test that the manually written backward in FusedDSAIndexerLoss produces
+ the same gradients as PyTorch autograd on the unfused implementation.
+ """
+ seqlen = seqlen_and_topk[0]
+ index_topk = seqlen_and_topk[1]
+ batch_size = 2
+ num_heads = 4
+ head_dim = 64
+ index_n_heads = 8
+ index_head_dim = 64
+ softmax_scale = head_dim**-0.5
+ loss_coeff = 1.0
+
+ torch.manual_seed(42)
+
+ # Create inputs for indexer
+ # q: [seqlen, batch, index_n_heads, index_head_dim]
+ q_ref = (
+ torch.randn(seqlen, batch_size, index_n_heads, index_head_dim, dtype=torch.float32)
+ .cuda()
+ .requires_grad_(True)
+ )
+ # weights: [seqlen, batch, index_n_heads]
+ weights_ref = (
+ torch.randn(seqlen, batch_size, index_n_heads, dtype=torch.float32)
+ .cuda()
+ .requires_grad_(True)
+ )
+ # k: [seqlen, batch, index_head_dim]
+ k_ref = (
+ torch.randn(seqlen, batch_size, index_head_dim, dtype=torch.float32)
+ .cuda()
+ .requires_grad_(True)
+ )
+ # query: [seqlen, batch, num_heads, head_dim] - detached, not trained
+ query = torch.randn(seqlen, batch_size, num_heads, head_dim, dtype=torch.bfloat16).cuda()
+ # key: [seqlen, batch, num_heads, head_dim] - detached, not trained
+ key = torch.randn(seqlen, batch_size, num_heads, head_dim, dtype=torch.bfloat16).cuda()
+
+ # Create causal mask
+ mask = torch.triu(
+ torch.full((seqlen, seqlen), float('-inf'), dtype=torch.float32).cuda(), diagonal=1
+ )
+
+ # =============================================
+ # Method 1: Autograd (reference)
+ # =============================================
+ # Compute index scores and apply mask (matches fused_qk_topk_naive behavior)
+ index_scores_ref = _compute_index_scores(q_ref, weights_ref, k_ref)
+ # Apply mask
+ index_scores_masked = index_scores_ref + mask.unsqueeze(0)
+ # Get topk indices from masked scores
+ topk_k = min(index_topk, seqlen)
+ topk_indices = index_scores_masked.topk(topk_k, dim=-1)[1]
+
+ # Compute loss using autograd
+ loss_ref = compute_dsa_indexer_loss(
+ index_scores=index_scores_masked,
+ topk_indices=topk_indices,
+ query=query,
+ key=key,
+ softmax_scale=softmax_scale,
+ loss_coeff=loss_coeff,
+ sparse_loss=sparse_loss,
+ pg_collection=self.pg_collection,
+ )
+
+ # Backward with autograd
+ loss_ref.backward()
+
+ # Save reference gradients
+ grad_q_ref = q_ref.grad.clone()
+ grad_weights_ref = weights_ref.grad.clone()
+ grad_k_ref = k_ref.grad.clone()
+
+ # =============================================
+ # Method 2: FusedDSAIndexerLoss (manual backward)
+ # =============================================
+ # Clone tensors from ref (detach and require grad again)
+ q_fused = q_ref.detach().clone().requires_grad_(True)
+ weights_fused = weights_ref.detach().clone().requires_grad_(True)
+ k_fused = k_ref.detach().clone().requires_grad_(True)
+
+ # Use FusedDSAIndexerLoss
+ topk_indices_fused, loss_fused = FusedDSAIndexerLoss.apply(
+ q_fused,
+ weights_fused,
+ k_fused,
+ query.detach(),
+ key.detach(),
+ softmax_scale,
+ index_topk,
+ loss_coeff,
+ mask,
+ sparse_loss,
+ self.pg_collection,
+ )
+
+ # Backward with manual implementation
+ loss_fused.backward()
+
+ # Get fused gradients
+ grad_q_fused = q_fused.grad
+ grad_weights_fused = weights_fused.grad
+ grad_k_fused = k_fused.grad
+
+ # =============================================
+ # Compare gradients
+ # =============================================
+ # Check loss values match
+ assert torch.allclose(
+ loss_fused, loss_ref, rtol=1e-5, atol=1e-5
+ ), f"Loss mismatch: fused={loss_fused.item()}, ref={loss_ref.item()}"
+
+ # Check topk indices match
+ assert torch.equal(
+ topk_indices_fused, topk_indices
+ ), "Top-k indices mismatch between fused and reference"
+
+ # Check gradients match
+ assert torch.allclose(
+ grad_q_fused, grad_q_ref, rtol=1e-5, atol=1e-5
+ ), f"grad_q mismatch: max diff = {(grad_q_fused - grad_q_ref).abs().max().item()}"
+
+ assert torch.allclose(
+ grad_weights_fused, grad_weights_ref, rtol=1e-5, atol=1e-5
+ ), f"grad_weights mismatch: max diff = {(grad_weights_fused - grad_weights_ref).abs().max().item()}"
+
+ assert torch.allclose(
+ grad_k_fused, grad_k_ref, rtol=1e-5, atol=1e-5
+ ), f"grad_k mismatch: max diff = {(grad_k_fused - grad_k_ref).abs().max().item()}"
+
+
+@pytest.mark.parametrize("tensor_model_parallel_size", [2, 4])
+@pytest.mark.parametrize("sparse_loss", [False, True])
+class TestFusedDSAIndexerLossGradientTP:
+ """Test FusedDSAIndexerLoss gradient consistency across different TP sizes."""
+
+ @pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA not available")
+ def test_fused_indexer_loss_gradient_tp_consistency(
+ self, tensor_model_parallel_size, sparse_loss
+ ):
+ """
+ Test that FusedDSAIndexerLoss produces consistent gradients across TP ranks
+ and matches TP=1 baseline.
+ """
+ seqlen = 64
+ index_topk = 32
+ batch_size = 2
+ num_heads = 8
+ head_dim = 64
+ index_n_heads = 8
+ index_head_dim = 64
+ softmax_scale = head_dim**-0.5
+ loss_coeff = 1.0
+
+ # =============================================
+ # First run with TP=1 to get baseline
+ # =============================================
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=1, pipeline_model_parallel_size=1
+ )
+ torch.manual_seed(42)
+ model_parallel_cuda_manual_seed(42)
+
+ pg_collection_tp1 = ProcessGroupCollection.use_mpu_process_groups(required_pgs=['tp'])
+
+ # Create inputs
+ q_input = torch.randn(
+ seqlen, batch_size, index_n_heads, index_head_dim, dtype=torch.float32
+ ).cuda()
+ weights_input = torch.randn(seqlen, batch_size, index_n_heads, dtype=torch.float32).cuda()
+ k_input = torch.randn(seqlen, batch_size, index_head_dim, dtype=torch.float32).cuda()
+ query_input = torch.randn(
+ seqlen, batch_size, num_heads, head_dim, dtype=torch.bfloat16
+ ).cuda()
+ key_input = torch.randn(
+ seqlen, batch_size, num_heads, head_dim, dtype=torch.bfloat16
+ ).cuda()
+ mask = torch.triu(
+ torch.full((seqlen, seqlen), float('-inf'), dtype=torch.float32).cuda(), diagonal=1
+ )
+
+ # Clone for TP=1
+ q_tp1 = q_input.clone().requires_grad_(True)
+ weights_tp1 = weights_input.clone().requires_grad_(True)
+ k_tp1 = k_input.clone().requires_grad_(True)
+
+ # Forward and backward with TP=1
+ topk_indices_tp1, loss_tp1 = FusedDSAIndexerLoss.apply(
+ q_tp1,
+ weights_tp1,
+ k_tp1,
+ query_input.detach(),
+ key_input.detach(),
+ softmax_scale,
+ index_topk,
+ loss_coeff,
+ mask,
+ sparse_loss,
+ pg_collection_tp1,
+ )
+ loss_tp1.backward()
+
+ # Save TP=1 results
+ grad_q_tp1 = q_tp1.grad.clone()
+ grad_weights_tp1 = weights_tp1.grad.clone()
+ grad_k_tp1 = k_tp1.grad.clone()
+ loss_tp1_value = loss_tp1.detach().clone()
+
+ Utils.destroy_model_parallel()
+
+ # =============================================
+ # Run with target TP size
+ # =============================================
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=tensor_model_parallel_size, pipeline_model_parallel_size=1
+ )
+ torch.manual_seed(42)
+ model_parallel_cuda_manual_seed(42)
+
+ pg_collection_tpn = ProcessGroupCollection.use_mpu_process_groups(required_pgs=['tp'])
+ tp_rank = parallel_state.get_tensor_model_parallel_rank()
+
+ # Clone inputs for TP=N (same values as TP=1)
+ q_tpn = q_input.clone().requires_grad_(True)
+ weights_tpn = weights_input.clone().requires_grad_(True)
+ k_tpn = k_input.clone().requires_grad_(True)
+
+ # query and key need to be split along heads for TP
+ head_per_rank = num_heads // tensor_model_parallel_size
+ start_head = tp_rank * head_per_rank
+ end_head = (tp_rank + 1) * head_per_rank
+ query_tpn = query_input[:, :, start_head:end_head, :].clone()
+ key_tpn = key_input[:, :, start_head:end_head, :].clone()
+
+ # Forward and backward with TP=N
+ topk_indices_tpn, loss_tpn = FusedDSAIndexerLoss.apply(
+ q_tpn,
+ weights_tpn,
+ k_tpn,
+ query_tpn.detach(),
+ key_tpn.detach(),
+ softmax_scale,
+ index_topk,
+ loss_coeff,
+ mask,
+ sparse_loss,
+ pg_collection_tpn,
+ )
+ loss_tpn.backward()
+
+ # =============================================
+ # Compare results
+ # =============================================
+ # Loss should be the same
+ assert torch.allclose(
+ loss_tpn, loss_tp1_value, rtol=1e-5, atol=1e-5
+ ), f"Loss mismatch: TP={tensor_model_parallel_size} got {loss_tpn.item()}, TP=1 got {loss_tp1_value.item()}"
+
+ # Top-k indices should be the same
+ assert torch.equal(
+ topk_indices_tpn, topk_indices_tp1
+ ), "Top-k indices mismatch between TP=1 and TP=N"
+
+ # Gradients should match exactly (indexer params are duplicated across TP)
+ assert torch.allclose(
+ q_tpn.grad, grad_q_tp1, rtol=1e-5, atol=1e-5
+ ), f"grad_q mismatch: max diff = {(q_tpn.grad - grad_q_tp1).abs().max().item()}"
+
+ assert torch.allclose(
+ weights_tpn.grad, grad_weights_tp1, rtol=1e-5, atol=1e-5
+ ), f"grad_weights mismatch: max diff = {(weights_tpn.grad - grad_weights_tp1).abs().max().item()}"
+
+ assert torch.allclose(
+ k_tpn.grad, grad_k_tp1, rtol=1e-5, atol=1e-5
+ ), f"grad_k mismatch: max diff = {(k_tpn.grad - grad_k_tp1).abs().max().item()}"
+
+ # Check gradients are identical across all TP ranks
+ tp_size = parallel_state.get_tensor_model_parallel_world_size()
+ if tp_size > 1:
+ for grad_tensor, name in [
+ (q_tpn.grad, "grad_q"),
+ (weights_tpn.grad, "grad_weights"),
+ (k_tpn.grad, "grad_k"),
+ ]:
+ grad_list = [torch.zeros_like(grad_tensor) for _ in range(tp_size)]
+ torch.distributed.all_gather(grad_list, grad_tensor, group=pg_collection_tpn.tp)
+
+ for i in range(1, tp_size):
+ assert torch.allclose(
+ grad_list[0], grad_list[i], rtol=0, atol=0
+ ), f"{name} differs between TP rank 0 and rank {i}"
+
+ Utils.destroy_model_parallel()
+
+
@pytest.mark.parametrize("seqlen", [16, 64])
class TestDSAIndexer:
"""Test DSA Indexer module basic functionality with TP=1."""
diff --git a/tests/unit_tests/transformer/moe/test_a2a_token_dispatcher.py b/tests/unit_tests/transformer/moe/test_a2a_token_dispatcher.py
index aad1fcaca5f..e20e35b84e1 100644
--- a/tests/unit_tests/transformer/moe/test_a2a_token_dispatcher.py
+++ b/tests/unit_tests/transformer/moe/test_a2a_token_dispatcher.py
@@ -95,7 +95,6 @@ def test_capacity_padding_forward_backward(self, tp_size, ep_size, permute_fusio
)
container.dispatcher_drop_and_pad_test()
- @pytest.mark.flaky_in_dev
@pytest.mark.skipif(
not is_te_min_version("1.7.0"), reason="TE 1.7.0 is required for MoE with FP8."
)
diff --git a/tests/unit_tests/transformer/moe/test_grouped_mlp.py b/tests/unit_tests/transformer/moe/test_grouped_mlp.py
index cc4c613e87d..7c0b91a9497 100644
--- a/tests/unit_tests/transformer/moe/test_grouped_mlp.py
+++ b/tests/unit_tests/transformer/moe/test_grouped_mlp.py
@@ -5,11 +5,10 @@
import torch.nn.functional as F
from megatron.core.models.gpt.gpt_layer_specs import (
- get_gpt_layer_local_spec,
- get_gpt_layer_with_transformer_engine_spec,
+ get_gpt_layer_local_submodules,
+ get_gpt_layer_with_transformer_engine_submodules,
)
from megatron.core.transformer.module import Float16Module
-from megatron.core.transformer.moe import grouped_gemm_util as gg
from megatron.core.transformer.moe.experts import TEGroupedMLP
from megatron.core.transformer.moe.moe_layer import MoELayer
from megatron.core.transformer.transformer_config import TransformerConfig
@@ -18,206 +17,6 @@
from megatron.training.initialize import _set_random_seed
from tests.unit_tests.test_utilities import Utils
-DEVICE_CAPABILITY = None
-if torch.cuda.is_available():
- DEVICE_CAPABILITY = torch.cuda.get_device_capability()
-
-
-@pytest.mark.skipif(is_te_min_version("1.9.0.dev0"), reason="Switch to TEGroupedMLP when TE>1.9.")
-class TestParallelGroupedMLP:
-
- def setup_method(self, method, use_cpu_initialization=False, swiglu=True):
- print("============")
- print(
- "Test for use_cpu_initilization={} and swiglu={}.".format(
- use_cpu_initialization, swiglu
- )
- )
- print("============")
- Utils.initialize_model_parallel(1, 1)
- num_layers = 1 # 2
- self.hidden_size = (
- 16 # must be an multiple of 16, otherwise trigger CUTLASS misaligned issue
- )
- self.num_experts = 2
- self.gated_linear_unit = swiglu
- self.activation_func = F.silu if swiglu else F.gelu
- self.use_cpu_initialization = use_cpu_initialization
-
- tf_config = TransformerConfig(
- num_layers=num_layers,
- hidden_size=self.hidden_size,
- num_attention_heads=4,
- num_moe_experts=self.num_experts,
- use_cpu_initialization=self.use_cpu_initialization,
- add_bias_linear=False,
- gated_linear_unit=self.gated_linear_unit,
- activation_func=self.activation_func,
- bias_activation_fusion=False,
- bf16=True,
- params_dtype=torch.bfloat16,
- moe_router_load_balancing_type="sinkhorn",
- moe_router_topk=1,
- )
-
- self.fc1_ffn_hidden_size = tf_config.ffn_hidden_size
- self.fc2_ffn_hidden_size = tf_config.ffn_hidden_size
- # If using swiglu double the output width, see https://arxiv.org/pdf/2002.05202.pdf
- if self.gated_linear_unit:
- self.fc1_ffn_hidden_size *= 2
-
- ## Vanilla sequential GEMM
- # Set random seed for reproducability
- _set_random_seed(seed_=123, data_parallel_random_init=False)
- transformer_layer_spec = get_gpt_layer_local_spec(self.num_experts, moe_grouped_gemm=False)
- self.sequential_mlp = MoELayer(tf_config, transformer_layer_spec.submodules.mlp.submodules)
-
- self.args = parse_args(ignore_unknown_args=True)
- self.args.bf16 = True
- # Bias is not supported in grouped gemm currently, thus we disable the
- # bias in the linear layer.
- self.args.add_bias_linear = False
- self.sequential_mlp = Float16Module(self.sequential_mlp.config, self.sequential_mlp).module
- print("done intializing for sequential gemm")
-
- ## Grouped GEMM
- _set_random_seed(seed_=123, data_parallel_random_init=False)
- tf_config.moe_grouped_gemm = True
- transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec(
- self.num_experts, moe_grouped_gemm=True
- )
- self.grouped_mlp = MoELayer(tf_config, transformer_layer_spec.submodules.mlp.submodules)
- self.grouped_mlp = Float16Module(self.grouped_mlp.config, self.grouped_mlp).module
- print("done intializing for grouped gemm")
-
- def teardown_method(self, method):
- Utils.destroy_model_parallel()
-
- @pytest.mark.internal
- def test_constructor(self):
- assert isinstance(self.sequential_mlp, MoELayer)
- assert isinstance(self.grouped_mlp, MoELayer)
-
- num_weights_smm = sum([p.numel() for p in self.sequential_mlp.parameters()])
- num_weights_gmm = sum([p.numel() for p in self.grouped_mlp.parameters()])
-
- # For the same hyper-parm model configs except the `moe_grouped_gemm`,
- # GroupedGEMM and sequential GEMMs should hold the same number of parms.
- assert num_weights_smm == num_weights_gmm
- # expected num weights: router linear weights+bias + MLP weights(no bias) of all experts
- expected_num_weights = (
- self.hidden_size * self.num_experts
- + self.hidden_size
- * (self.fc1_ffn_hidden_size + self.fc2_ffn_hidden_size)
- * self.num_experts
- )
- assert num_weights_smm == expected_num_weights
-
- assert torch.equal(self.sequential_mlp.router.weight, self.grouped_mlp.router.weight)
-
- # weight1: [h, num_experts*4h]
- # weight2: [num_experts*4h, h]
- assert self.grouped_mlp.experts.weight1.shape[0] == self.hidden_size
- assert (
- self.grouped_mlp.experts.weight1.shape[1] == self.num_experts * self.fc1_ffn_hidden_size
- )
- if self.gated_linear_unit:
- assert (
- self.grouped_mlp.experts.weight2.shape[0]
- == self.num_experts * self.fc2_ffn_hidden_size
- )
- assert self.grouped_mlp.experts.weight2.shape[1] == self.hidden_size
- else:
- assert (
- self.grouped_mlp.experts.weight1.shape == self.grouped_mlp.experts.weight2.t().shape
- )
-
- @pytest.mark.internal
- def test_weight_init_value_the_same(self):
- gmm_w1 = self.grouped_mlp.experts.weight1.view(self.num_experts, -1, self.hidden_size)
- gmm_w2 = self.grouped_mlp.experts.weight2.view(self.num_experts, self.hidden_size, -1)
- gmm_expert1_fc1 = gmm_w1[0]
- gmm_expert1_fc2 = gmm_w2[0]
- gmm_expert2_fc1 = gmm_w1[1]
- gmm_expert2_fc2 = gmm_w2[1]
-
- smm_expert1_fc1 = self.sequential_mlp.experts.local_experts[0].linear_fc1.weight
- smm_expert1_fc2 = self.sequential_mlp.experts.local_experts[0].linear_fc2.weight
- smm_expert2_fc1 = self.sequential_mlp.experts.local_experts[1].linear_fc1.weight
- smm_expert2_fc2 = self.sequential_mlp.experts.local_experts[1].linear_fc2.weight
-
- assert torch.equal(gmm_expert1_fc1, smm_expert1_fc1)
- if not self.use_cpu_initialization:
- assert torch.equal(gmm_expert1_fc2, smm_expert1_fc2)
- # the param init value is not exactly the same between gmm and smm (refer to test_weight_init_value_the_same.)
- # TODO: is it necessary to keep smm and gmm share exactly the same init params?
- # assert torch.equal(gmm_expert2_fc1, smm_expert2_fc1)
- if self.use_cpu_initialization:
- assert torch.equal(gmm_expert2_fc2, smm_expert2_fc2)
-
- @pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA not available")
- @pytest.mark.internal
- @pytest.mark.skipif(
- not DEVICE_CAPABILITY or DEVICE_CAPABILITY[0] < 8,
- reason='GroupedGEMM kernels are not supported on this device.',
- )
- def test_gpu_forward(self):
- self.sequential_mlp.cuda()
- self.grouped_mlp.cuda()
- # [sequence length, batch size, hidden size]
- seq_len = 3 # 32
- batch_size = 2
- hidden_states = torch.rand(
- (seq_len, batch_size, self.sequential_mlp.config.hidden_size), dtype=torch.bfloat16
- )
- hidden_states = hidden_states.cuda()
- output_smm, _ = self.sequential_mlp(hidden_states)
- output_gmm, _ = self.grouped_mlp(hidden_states)
-
- # The following assert fails due to the param init value is not exactly
- # the same between gmm and smm (refer to test_weight_init_value_the_same.)
- # assert torch.equal(output_smm, output_gmm)
-
- @pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA not available")
- @pytest.mark.internal
- @pytest.mark.skipif(
- not DEVICE_CAPABILITY or DEVICE_CAPABILITY[0] < 8,
- reason='GroupedGEMM kernels are not supported on this device.',
- )
- def test_gpu_forward_with_no_tokens_allocated(self):
- """Test the case when no token is allocated for groupedGEMM kernels."""
- w1 = self.grouped_mlp.experts.weight1.view(self.num_experts, -1, self.hidden_size)
- num_allocated_tokens = 0
- tokens_per_expert = torch.zeros(self.num_experts)
- hidden_states = torch.rand((num_allocated_tokens, self.hidden_size), dtype=torch.bfloat16)
- hidden_states = hidden_states.cuda()
- try:
- gg.ops.gmm(hidden_states, w1, tokens_per_expert, trans_b=False)
- except Exception as e:
- print("Expected error message from groupedGEMM:", e)
- assert str(e) == "Input batch_sizes should not be all zeros!"
-
- @pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA not available")
- @pytest.mark.internal
- @pytest.mark.skipif(
- not DEVICE_CAPABILITY or DEVICE_CAPABILITY[0] < 8,
- reason='GroupedGEMM kernels are not supported on this device.',
- )
- def test_gradient_with_no_tokens_allocated(self):
- """Test that when no token is passed in, the parameters of the grouped MLP will also have gradients."""
- self.grouped_mlp.cuda()
- num_allocated_tokens = 0
- tokens_per_expert = torch.zeros(self.num_experts)
- hidden_states = torch.rand((num_allocated_tokens, self.hidden_size), dtype=torch.bfloat16)
- hidden_states = hidden_states.cuda()
- probs = torch.rand((num_allocated_tokens,), dtype=torch.float32)
- probs = probs.cuda()
- output_gmm, _ = self.grouped_mlp.experts(
- hidden_states, tokens_per_expert=tokens_per_expert, permuted_probs=probs
- )
- output_gmm.mean().backward()
- assert self.grouped_mlp.experts.weight1.grad is not None
-
@pytest.mark.skipif(
not is_te_min_version("1.9.0.dev0"),
@@ -259,8 +58,8 @@ def setup_method(self, method, use_cpu_initialization=False, swiglu=True):
## Vanilla sequential GEMM
# Set random seed for reproducability
_set_random_seed(seed_=123, data_parallel_random_init=False)
- transformer_layer_spec = get_gpt_layer_local_spec(self.num_experts, moe_grouped_gemm=False)
- self.sequential_mlp = MoELayer(tf_config, transformer_layer_spec.submodules.mlp.submodules)
+ submodules = get_gpt_layer_local_submodules(self.num_experts, moe_grouped_gemm=False)
+ self.sequential_mlp = MoELayer(tf_config, submodules.mlp.submodules)
self.args = parse_args(ignore_unknown_args=True)
self.args.bf16 = True
@@ -271,11 +70,13 @@ def setup_method(self, method, use_cpu_initialization=False, swiglu=True):
## Grouped GEMM
_set_random_seed(seed_=123, data_parallel_random_init=False)
- transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec(
- self.num_experts, moe_grouped_gemm=True
- )
tf_config.moe_grouped_gemm = True
- self.grouped_mlp = MoELayer(tf_config, transformer_layer_spec.submodules.mlp.submodules)
+ self.grouped_mlp = MoELayer(
+ tf_config,
+ get_gpt_layer_with_transformer_engine_submodules(
+ self.num_experts, moe_grouped_gemm=True
+ ).mlp.submodules,
+ )
assert isinstance(self.grouped_mlp.experts, TEGroupedMLP)
self.grouped_mlp = Float16Module(self.grouped_mlp.config, self.grouped_mlp).module
@@ -383,17 +184,3 @@ def test_gpu_forward_backward_with_no_tokens_allocated(self):
for i in range(self.num_experts):
assert getattr(self.grouped_mlp.experts.linear_fc1, f"weight{i}").grad is not None
assert getattr(self.grouped_mlp.experts.linear_fc2, f"weight{i}").grad is not None
-
-
-if __name__ == "__main__":
- for use_cpu_unitilization in [True, False]:
- for swiglu in [True, False]:
- GMLP_test = TestParallelGroupedMLP()
- GMLP_test.setup_method(
- method=None, use_cpu_initialization=use_cpu_unitilization, swiglu=swiglu
- )
- GMLP_test.test_constructor()
- GMLP_test.test_weight_init_value_the_same()
- GMLP_test.test_gpu_forward()
- GMLP_test.test_gpu_forward_with_no_tokens_allocated()
- GMLP_test.teardown_method(method=None)
diff --git a/tests/unit_tests/transformer/moe/test_latent_moe_layer.py b/tests/unit_tests/transformer/moe/test_latent_moe_layer.py
new file mode 100644
index 00000000000..f62de67860a
--- /dev/null
+++ b/tests/unit_tests/transformer/moe/test_latent_moe_layer.py
@@ -0,0 +1,104 @@
+# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+
+import pytest
+import torch
+
+from megatron.core.models.gpt.gpt_layer_specs import (
+ get_gpt_decoder_block_spec,
+ get_gpt_layer_local_spec,
+ get_gpt_layer_with_transformer_engine_spec,
+)
+from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
+from megatron.core.transformer.moe.moe_layer import MoELayer
+from megatron.core.transformer.moe.router import Router
+from megatron.core.transformer.transformer_block import TransformerBlock
+from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.utils import is_te_min_version
+from megatron.training.initialize import _set_random_seed
+from tests.unit_tests.test_utilities import Utils
+
+
+class TestLatentMoELayer:
+ def setup_method(self, method):
+ pass
+
+ @pytest.mark.skipif(
+ not is_te_min_version("1.7.0.dev0"),
+ reason="Expert with TE Linear is only supported in TE 1.7.0 and later.",
+ )
+ @pytest.mark.parametrize("moe_token_dispatcher_type", ["allgather", "alltoall"])
+ @pytest.mark.parametrize("num_moe_experts", [4])
+ @pytest.mark.parametrize("use_te,grouped_gemm", [(True, True), (True, False), (False, False)])
+ @pytest.mark.parametrize("moe_latent_size", [8, 16])
+ def test_latent_moe_layer(
+ self, num_moe_experts, moe_token_dispatcher_type, use_te, grouped_gemm, moe_latent_size
+ ):
+ Utils.initialize_model_parallel(1, 1)
+ _set_random_seed(seed_=123, data_parallel_random_init=False)
+ self.transformer_config = TransformerConfig(
+ num_layers=1,
+ hidden_size=32,
+ num_attention_heads=4,
+ num_moe_experts=num_moe_experts,
+ use_cpu_initialization=True,
+ moe_token_dispatcher_type=moe_token_dispatcher_type,
+ moe_router_topk=2,
+ moe_aux_loss_coeff=0.01,
+ moe_grouped_gemm=grouped_gemm,
+ moe_ffn_hidden_size=128,
+ moe_shared_expert_intermediate_size=128,
+ activation_func=torch.nn.functional.silu,
+ gated_linear_unit=True,
+ add_bias_linear=False,
+ moe_latent_size=moe_latent_size,
+ )
+ if use_te:
+ transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec(
+ num_experts=num_moe_experts, moe_grouped_gemm=grouped_gemm
+ )
+ else:
+ transformer_layer_spec = get_gpt_layer_local_spec(
+ num_experts=num_moe_experts, moe_grouped_gemm=grouped_gemm
+ )
+ moe_layer = MoELayer(
+ self.transformer_config, transformer_layer_spec.submodules.mlp.submodules
+ )
+ moe_layer.cuda()
+ config = moe_layer.config
+
+ assert (
+ moe_layer.shared_experts.linear_fc1.weight.shape[1] == config.hidden_size
+ ), "Shared expert computation has to happen in hidden dimension."
+ assert (
+ moe_layer.shared_experts.linear_fc2.weight.shape[0] == config.hidden_size
+ ), "Shared expert computation has to happen in hidden dimension."
+ if grouped_gemm:
+ for i in range(num_moe_experts):
+ fc1_weight = getattr(moe_layer.experts.linear_fc1, f"weight{i}")
+ fc2_weight = getattr(moe_layer.experts.linear_fc2, f"weight{i}")
+ assert (
+ fc1_weight.shape[1] == config.moe_latent_size
+ ), f"Shape mismatch for expert {i} {fc1_weight.shape=}"
+ assert (
+ fc2_weight.shape[0] == config.moe_latent_size
+ ), f"Shape mismatch for expert {i} {fc2_weight.shape=}"
+ else:
+ for i in range(num_moe_experts):
+ expert = moe_layer.experts.local_experts[i]
+ assert (
+ expert.linear_fc1.weight.shape[1] == config.moe_latent_size
+ ), f"Shape mismatch for expert {i} {fc1_weight.shape=}"
+ assert (
+ expert.linear_fc2.weight.shape[0] == config.moe_latent_size
+ ), f"Shape mismatch for expert {i} {fc2_weight.shape=}"
+ assert (
+ moe_layer.router.weight.shape[1] == config.hidden_size
+ ), "MoE routing has to happen in hidden dimension."
+
+ # [sequence length, batch size, hidden size]
+ hidden_states = torch.ones((32, 2, config.hidden_size))
+ hidden_states = hidden_states.cuda()
+ output, _ = moe_layer(hidden_states)
+ assert output.shape[2] == config.hidden_size
+
+ Utils.destroy_model_parallel()
diff --git a/tests/unit_tests/transformer/moe/test_moe_layer.py b/tests/unit_tests/transformer/moe/test_moe_layer.py
index 11bd09f8449..0004b7fef98 100755
--- a/tests/unit_tests/transformer/moe/test_moe_layer.py
+++ b/tests/unit_tests/transformer/moe/test_moe_layer.py
@@ -5,12 +5,11 @@
from megatron.core.models.gpt.gpt_layer_specs import (
get_gpt_decoder_block_spec,
- get_gpt_layer_local_spec,
- get_gpt_layer_with_transformer_engine_spec,
+ get_gpt_layer_local_submodules,
+ get_gpt_layer_with_transformer_engine_submodules,
)
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer.moe.moe_layer import MoELayer
-from megatron.core.transformer.moe.router import Router
from megatron.core.transformer.transformer_block import TransformerBlock
from megatron.core.transformer.transformer_config import TransformerConfig
from megatron.core.utils import is_te_min_version
@@ -45,12 +44,10 @@ def test_te_moe_layer(self, num_moe_experts, moe_token_dispatcher_type, grouped_
moe_ffn_hidden_size=128,
add_bias_linear=False,
)
- transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec(
+ submodules = get_gpt_layer_with_transformer_engine_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=grouped_gemm
)
- moe_layer = MoELayer(
- self.transformer_config, transformer_layer_spec.submodules.mlp.submodules
- )
+ moe_layer = MoELayer(self.transformer_config, submodules.mlp.submodules)
Utils.destroy_model_parallel()
@pytest.mark.parametrize("moe_token_dispatcher_type", ["allgather", "alltoall"])
@@ -73,12 +70,10 @@ def test_legacy_moe_layer(self, num_moe_experts, moe_token_dispatcher_type, grou
moe_grouped_gemm=grouped_gemm,
add_bias_linear=False,
)
- transformer_layer_spec = get_gpt_layer_local_spec(
+ transformer_layer_submodules = get_gpt_layer_local_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=grouped_gemm
)
- moe_layer = MoELayer(
- self.transformer_config, transformer_layer_spec.submodules.mlp.submodules
- )
+ moe_layer = MoELayer(self.transformer_config, transformer_layer_submodules.mlp.submodules)
Utils.destroy_model_parallel()
@pytest.mark.skip(
@@ -110,7 +105,7 @@ def test_moe_with_late_initialize(
bf16=True,
params_dtype=torch.bfloat16,
)
- transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec(
+ submodules = get_gpt_layer_with_transformer_engine_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=grouped_gemm
)
@@ -118,9 +113,7 @@ def test_moe_with_late_initialize(
Utils.fake_initialize_model_parallel(
tensor_model_parallel_size=tp_size, expert_model_parallel_size=ep_size
)
- moe_layer = MoELayer(
- transformer_config, transformer_layer_spec.submodules.mlp.submodules
- ).cuda()
+ moe_layer = MoELayer(transformer_config, submodules.mlp.submodules).cuda()
Utils.initialize_model_parallel(
tensor_model_parallel_size=tp_size, expert_model_parallel_size=ep_size
@@ -236,13 +229,11 @@ def test_moe_layer_fp16_forward_backward(
params_dtype=torch.float16,
)
- transformer_layer_spec = get_gpt_layer_local_spec(
+ submodules = get_gpt_layer_local_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=False
)
- moe_layer = MoELayer(
- transformer_config, transformer_layer_spec.submodules.mlp.submodules
- ).cuda()
+ moe_layer = MoELayer(transformer_config, submodules.mlp.submodules).cuda()
hidden_states = torch.randn(
sequence_length,
@@ -341,17 +332,15 @@ def test_moe_layer_recompute_forward_backward(
# Use TE spec for fp8, local spec otherwise
if fp8:
- transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec(
+ transformer_layer_submodules = get_gpt_layer_with_transformer_engine_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=False
)
else:
- transformer_layer_spec = get_gpt_layer_local_spec(
+ transformer_layer_submodules = get_gpt_layer_local_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=False
)
- moe_layer = MoELayer(
- transformer_config, transformer_layer_spec.submodules.mlp.submodules
- ).cuda()
+ moe_layer = MoELayer(transformer_config, transformer_layer_submodules.mlp.submodules).cuda()
hidden_states = torch.randn(
sequence_length,
diff --git a/tests/unit_tests/transformer/moe/test_moe_layer_discrepancy.py b/tests/unit_tests/transformer/moe/test_moe_layer_discrepancy.py
index 4386a844dd2..d60fbe336b8 100644
--- a/tests/unit_tests/transformer/moe/test_moe_layer_discrepancy.py
+++ b/tests/unit_tests/transformer/moe/test_moe_layer_discrepancy.py
@@ -7,10 +7,8 @@
from megatron.core import parallel_state
from megatron.core.models.gpt.gpt_layer_specs import (
- get_gpt_layer_local_spec,
- get_gpt_layer_with_transformer_engine_spec,
+ get_gpt_layer_with_transformer_engine_submodules,
)
-from megatron.core.transformer.moe.moe_layer import MoELayer
from megatron.core.transformer.transformer_config import TransformerConfig
from megatron.core.transformer.transformer_layer import TransformerLayer
from megatron.training.initialize import _set_random_seed
@@ -49,7 +47,7 @@ def test_moe_layer_dispatcher_discrepancy(
expert_model_parallel_size=ep_size,
sequence_parallel=True if (tp_size > 1) else False,
)
- transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec(
+ submodules = get_gpt_layer_with_transformer_engine_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=grouped_gemm
)
# Init input and layer
@@ -58,21 +56,13 @@ def test_moe_layer_dispatcher_discrepancy(
# Init allgather moe layer
_set_random_seed(seed_=123, data_parallel_random_init=False)
- layer = (
- TransformerLayer(self.transformer_config, transformer_layer_spec.submodules)
- .cuda()
- .float()
- )
+ layer = TransformerLayer(self.transformer_config, submodules).cuda().float()
ag_moe_layer = layer.mlp
ag_moe_layer.eval()
# Init a2a moe layer
self.transformer_config.moe_token_dispatcher_type = "alltoall"
_set_random_seed(seed_=123, data_parallel_random_init=False)
- layer = (
- TransformerLayer(self.transformer_config, transformer_layer_spec.submodules)
- .cuda()
- .float()
- )
+ layer = TransformerLayer(self.transformer_config, submodules).cuda().float()
a2a_moe_layer = layer.mlp
a2a_moe_layer.eval()
@@ -134,7 +124,7 @@ def test_moe_layer_ag_dispatcher_discrepancy(
sequence_parallel=True if (tp_size > 1 and ep_size > 1) else False,
bf16=True,
)
- transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec(
+ submodules = get_gpt_layer_with_transformer_engine_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=grouped_gemm
)
# Init input and layer
@@ -143,11 +133,7 @@ def test_moe_layer_ag_dispatcher_discrepancy(
# Init allgather moe layer
_set_random_seed(seed_=123, data_parallel_random_init=False)
- layer = (
- TransformerLayer(self.transformer_config, transformer_layer_spec.submodules)
- .cuda()
- .bfloat16()
- )
+ layer = TransformerLayer(self.transformer_config, submodules).cuda().bfloat16()
ag_moe_layer = layer.mlp
ag_moe_layer.eval()
@@ -205,7 +191,7 @@ def test_moe_layer_a2a_dispatcher_discrepancy(
sequence_parallel=True if (tp_size > 1 and ep_size > 1) else False,
bf16=True,
)
- transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec(
+ submodules = get_gpt_layer_with_transformer_engine_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=grouped_gemm
)
# Init input and layer
@@ -213,11 +199,7 @@ def test_moe_layer_a2a_dispatcher_discrepancy(
input = torch.randn(1, 4096, 4096).cuda().bfloat16()
# Init a2a moe layer
- layer = (
- TransformerLayer(self.transformer_config, transformer_layer_spec.submodules)
- .cuda()
- .bfloat16()
- )
+ layer = TransformerLayer(self.transformer_config, submodules).cuda().bfloat16()
a2a_moe_layer = layer.mlp
a2a_moe_layer.eval()
diff --git a/tests/unit_tests/transformer/moe/test_routers.py b/tests/unit_tests/transformer/moe/test_routers.py
index 4d6b5ee2c3e..8f3dbbe96e0 100644
--- a/tests/unit_tests/transformer/moe/test_routers.py
+++ b/tests/unit_tests/transformer/moe/test_routers.py
@@ -1,11 +1,12 @@
# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+
from typing import cast
import pytest
import torch
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_spec
+from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_submodules
from megatron.core.transformer.moe.moe_layer import MoELayer
from megatron.core.transformer.moe.moe_utils import get_updated_expert_bias, router_gating_linear
from megatron.core.transformer.moe.router import Router
@@ -43,12 +44,10 @@ def setup_method(self, method):
params_dtype=torch.bfloat16,
add_bias_linear=False,
)
- transformer_layer_spec = get_gpt_layer_local_spec(
+ submodules = get_gpt_layer_local_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=False
)
- self.sequential_mlp = MoELayer(
- self.transformer_config, transformer_layer_spec.submodules.mlp.submodules
- )
+ self.sequential_mlp = MoELayer(self.transformer_config, submodules.mlp.submodules)
self.router = cast(Router, self.sequential_mlp.router)
def teardown_method(self, method):
@@ -314,12 +313,10 @@ def setup_method(self, method):
)
# init MoE layer
- transformer_layer_spec = get_gpt_layer_local_spec(
+ submodules = get_gpt_layer_local_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=False
)
- self.moe_layer = MoELayer(
- self.transformer_config, transformer_layer_spec.submodules.mlp.submodules
- ).cuda()
+ self.moe_layer = MoELayer(self.transformer_config, submodules.mlp.submodules).cuda()
self.router = cast(Router, self.moe_layer.router)
def teardown_method(self, method):
@@ -421,12 +418,10 @@ def setup_method(self, method):
params_dtype=torch.bfloat16,
add_bias_linear=False,
)
- transformer_layer_spec = get_gpt_layer_local_spec(
+ submodules = get_gpt_layer_local_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=False
)
- self.moe_layer = MoELayer(
- self.transformer_config, transformer_layer_spec.submodules.mlp.submodules
- )
+ self.moe_layer = MoELayer(self.transformer_config, submodules.mlp.submodules)
self.router = cast(Router, self.moe_layer.router)
assert self.router.expert_bias is not None
assert self.router.local_tokens_per_expert is not None
@@ -469,15 +464,11 @@ def test_router_forward_aux_free(self):
def test_router_forward_fusion_equivalence(self, score_function):
with torch.no_grad():
# Build two fresh routers to avoid bias update interference
- transformer_layer_spec = get_gpt_layer_local_spec(
+ submodules = get_gpt_layer_local_submodules(
num_experts=self.transformer_config.num_moe_experts, moe_grouped_gemm=False
)
- moe_layer_ref = MoELayer(
- self.transformer_config, transformer_layer_spec.submodules.mlp.submodules
- )
- moe_layer_fused = MoELayer(
- self.transformer_config, transformer_layer_spec.submodules.mlp.submodules
- )
+ moe_layer_ref = MoELayer(self.transformer_config, submodules.mlp.submodules)
+ moe_layer_fused = MoELayer(self.transformer_config, submodules.mlp.submodules)
router_ref = moe_layer_ref.router.cuda()
router_fused = moe_layer_fused.router.cuda()
diff --git a/tests/unit_tests/transformer/moe/test_sequential_mlp.py b/tests/unit_tests/transformer/moe/test_sequential_mlp.py
index a80e2a2ff1b..e618f6a8318 100644
--- a/tests/unit_tests/transformer/moe/test_sequential_mlp.py
+++ b/tests/unit_tests/transformer/moe/test_sequential_mlp.py
@@ -5,7 +5,7 @@
import torch
from megatron.core.extensions.transformer_engine import TEColumnParallelLinear, TERowParallelLinear
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_spec
+from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_submodules
from megatron.core.tensor_parallel.layers import ColumnParallelLinear, RowParallelLinear
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer.mlp import MLPSubmodules
@@ -37,12 +37,10 @@ def setup_method(self, method):
moe_router_topk=1,
add_bias_linear=False,
)
- transformer_layer_spec = get_gpt_layer_local_spec(
+ submodules = get_gpt_layer_local_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=False
)
- self.sequential_mlp = MoELayer(
- transformer_config, transformer_layer_spec.submodules.mlp.submodules
- )
+ self.sequential_mlp = MoELayer(transformer_config, submodules.mlp.submodules)
def teardown_method(self, method):
Utils.destroy_model_parallel()
diff --git a/tests/unit_tests/transformer/moe/test_shared_experts.py b/tests/unit_tests/transformer/moe/test_shared_experts.py
index 6df4d2fd369..d99dc1a0b05 100644
--- a/tests/unit_tests/transformer/moe/test_shared_experts.py
+++ b/tests/unit_tests/transformer/moe/test_shared_experts.py
@@ -3,7 +3,7 @@
import pytest
import torch
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_spec
+from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_submodules
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer.moe.moe_layer import MoELayer
from megatron.core.transformer.transformer_config import TransformerConfig
@@ -41,12 +41,10 @@ def test_gpu_forward(self, shared_expert_gate):
add_bias_linear=False,
moe_shared_expert_gate=shared_expert_gate,
)
- transformer_layer_spec = get_gpt_layer_local_spec(
+ submodules = get_gpt_layer_local_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=False
)
- self.moe_layer = MoELayer(
- transformer_config, transformer_layer_spec.submodules.mlp.submodules
- )
+ self.moe_layer = MoELayer(transformer_config, submodules.mlp.submodules)
assert isinstance(self.moe_layer, MoELayer)
@@ -103,12 +101,10 @@ def test_gpu_forward(self):
moe_router_topk=1,
add_bias_linear=False,
)
- transformer_layer_spec = get_gpt_layer_local_spec(
+ submodules = get_gpt_layer_local_submodules(
num_experts=num_moe_experts, moe_grouped_gemm=False
)
- self.moe_layer = MoELayer(
- transformer_config, transformer_layer_spec.submodules.mlp.submodules
- )
+ self.moe_layer = MoELayer(transformer_config, submodules.mlp.submodules)
assert isinstance(self.moe_layer, MoELayer)
diff --git a/tests/unit_tests/transformer/moe/test_token_dispatcher.py b/tests/unit_tests/transformer/moe/test_token_dispatcher.py
index 24617952b94..6ff8fcdc6e5 100644
--- a/tests/unit_tests/transformer/moe/test_token_dispatcher.py
+++ b/tests/unit_tests/transformer/moe/test_token_dispatcher.py
@@ -7,7 +7,7 @@
import torch
from megatron.core import config, parallel_state
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_spec
+from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_submodules
from megatron.core.transformer.moe.moe_layer import MoELayer
from megatron.core.transformer.moe.moe_utils import get_capacity
from megatron.core.transformer.transformer_config import TransformerConfig
@@ -99,15 +99,11 @@ def __init__(
self.moe_layer = self.new_moe_layer()
def new_moe_layer(self, **kargs):
- transformer_layer_spec = get_gpt_layer_local_spec(
+ submodules = get_gpt_layer_local_submodules(
num_experts=self.config.num_moe_experts, moe_grouped_gemm=self.config.moe_grouped_gemm
)
new_config = dataclasses.replace(self.config, **kargs)
- moe_layer = (
- MoELayer(new_config, transformer_layer_spec.submodules.mlp.submodules)
- .cuda()
- .to(dtype=self.test_dtype)
- )
+ moe_layer = MoELayer(new_config, submodules.mlp.submodules).cuda().to(dtype=self.test_dtype)
moe_layer.set_layer_number(0)
return moe_layer
@@ -364,7 +360,6 @@ def setup_method(self, method):
def teardown_method(self, method):
Utils.destroy_model_parallel()
- @pytest.mark.flaky_in_dev
@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA not available")
@pytest.mark.internal
@pytest.mark.parametrize("tp_size,ep_size", [(8, 1), (1, 8), (2, 4), (1, 1)])
@@ -383,7 +378,6 @@ def test_forward_backward(self, tp_size, ep_size, permute_fusion):
container.dispatcher_dropless_test()
- @pytest.mark.flaky_in_dev
@pytest.mark.skipif(not torch.cuda.is_available(), reason="CUDA not available")
@pytest.mark.internal
@pytest.mark.parametrize("permute_fusion", permute_fusion_params)
diff --git a/tests/unit_tests/transformer/moe/test_upcycling.py b/tests/unit_tests/transformer/moe/test_upcycling.py
index 9cb7aa327bd..097356d6c78 100644
--- a/tests/unit_tests/transformer/moe/test_upcycling.py
+++ b/tests/unit_tests/transformer/moe/test_upcycling.py
@@ -8,6 +8,7 @@
from megatron.core import mpu
from megatron.core.enums import ModelType
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.models.gpt.gpt_layer_specs import (
get_gpt_layer_local_spec,
get_gpt_layer_with_transformer_engine_spec,
@@ -33,12 +34,10 @@
)
from tests.unit_tests.test_utilities import Utils
-try:
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import TEColumnParallelGroupedLinear
-
- HAVE_TE = True
-except ImportError:
- HAVE_TE = False
+else:
+ TEColumnParallelGroupedLinear = None
_SEED = 42
@@ -115,7 +114,6 @@ def create_test_args(tp, grouped_gemm, swiglu, squared_relu, use_te):
args.lr = 3e-5
args.attention_dropout = 0.0
args.hidden_dropout = 0.0
- args.async_tensor_model_parallel_allreduce = False
args.no_save_optim = True
args.no_load_optim = True
args.no_load_rng = True
diff --git a/tests/unit_tests/transformer/test_attention.py b/tests/unit_tests/transformer/test_attention.py
index 1b576a3644a..0b23d250825 100755
--- a/tests/unit_tests/transformer/test_attention.py
+++ b/tests/unit_tests/transformer/test_attention.py
@@ -1,6 +1,7 @@
# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
import copy
+from unittest import mock
import pytest
import torch
@@ -8,13 +9,30 @@
import megatron.core.parallel_state as parallel_state
from megatron.core.hyper_comm_grid import HyperCommGrid
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
+from megatron.core.models.common.embeddings.rope_utils import (
+ get_pos_emb_on_this_cp_rank as get_tensor_on_this_cp_rank,
+)
+from megatron.core.models.gpt.gpt_layer_specs import (
+ get_gpt_layer_with_transformer_engine_spec,
+ get_gpt_layer_with_transformer_engine_submodules,
+)
+from megatron.core.models.gpt.gpt_model import GPTModel
from megatron.core.process_groups_config import ProcessGroupCollection
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer import TransformerConfig
from megatron.core.transformer.attention import SelfAttention
from megatron.core.transformer.enums import AttnMaskType
from megatron.core.utils import is_te_min_version
+from megatron.training.arguments import parse_args
+from megatron.training.checkpointing import load_checkpoint, save_checkpoint
+from megatron.training.global_vars import set_args
+from megatron.training.training import get_model
+from megatron.training.utils import unwrap_model
+from tests.unit_tests.dist_checkpointing import (
+ TempNamedDir,
+ init_basic_mock_args,
+ init_checkpointing_mock_args,
+)
from tests.unit_tests.test_utilities import Utils
try:
@@ -43,7 +61,7 @@ def setup_method(self, output_gate):
)
self.parallel_attention = SelfAttention(
self.transformer_config,
- get_gpt_layer_with_transformer_engine_spec().submodules.self_attention.submodules,
+ get_gpt_layer_with_transformer_engine_submodules().self_attention.submodules,
layer_number=1,
)
@@ -136,7 +154,7 @@ def test_checkpointed_gpu_forward(self):
transformer_config.recompute_granularity = 'selective'
checkpointed_parallel_attention = SelfAttention(
transformer_config,
- get_gpt_layer_with_transformer_engine_spec().submodules.self_attention.submodules,
+ get_gpt_layer_with_transformer_engine_submodules().self_attention.submodules,
layer_number=1,
)
config = checkpointed_parallel_attention.config
@@ -186,7 +204,7 @@ def test_clip_qk_disabled_raises_error(self):
)
attention = SelfAttention(
transformer_config,
- get_gpt_layer_with_transformer_engine_spec().submodules.self_attention.submodules,
+ get_gpt_layer_with_transformer_engine_submodules().self_attention.submodules,
layer_number=1,
)
@@ -206,7 +224,7 @@ def test_clip_qk_none_logits_raises_error(self):
)
attention = SelfAttention(
transformer_config,
- get_gpt_layer_with_transformer_engine_spec().submodules.self_attention.submodules,
+ get_gpt_layer_with_transformer_engine_submodules().self_attention.submodules,
layer_number=1,
)
@@ -226,7 +244,7 @@ def test_clip_qk_below_threshold_no_update(self):
)
attention = SelfAttention(
transformer_config,
- get_gpt_layer_with_transformer_engine_spec().submodules.self_attention.submodules,
+ get_gpt_layer_with_transformer_engine_submodules().self_attention.submodules,
layer_number=1,
)
attention.cuda()
@@ -260,7 +278,7 @@ def test_clip_qk_above_threshold_updates_weights(self):
)
attention = SelfAttention(
transformer_config,
- get_gpt_layer_with_transformer_engine_spec().submodules.self_attention.submodules,
+ get_gpt_layer_with_transformer_engine_submodules().self_attention.submodules,
layer_number=1,
)
attention.cuda()
@@ -295,7 +313,7 @@ def test_clip_qk_gqa_configuration(self):
)
attention = SelfAttention(
transformer_config,
- get_gpt_layer_with_transformer_engine_spec().submodules.self_attention.submodules,
+ get_gpt_layer_with_transformer_engine_submodules().self_attention.submodules,
layer_number=1,
)
attention.cuda()
@@ -329,7 +347,7 @@ def test_clip_qk_mixed_logits(self):
)
attention = SelfAttention(
transformer_config,
- get_gpt_layer_with_transformer_engine_spec().submodules.self_attention.submodules,
+ get_gpt_layer_with_transformer_engine_submodules().self_attention.submodules,
layer_number=1,
)
attention.cuda()
@@ -374,7 +392,7 @@ def run_self_attention(self, pg_collection):
)
self.self_attention = SelfAttention(
self.transformer_config,
- get_gpt_layer_with_transformer_engine_spec().submodules.self_attention.submodules,
+ get_gpt_layer_with_transformer_engine_submodules().self_attention.submodules,
layer_number=1,
attn_mask_type=AttnMaskType.causal,
pg_collection=pg_collection,
@@ -447,3 +465,262 @@ def test_self_attention_independent_pg_smoke(self):
pg_collection = ProcessGroupCollection(tp=tp_group, cp=cp_group)
self.run_self_attention(pg_collection)
+
+
+def _test_parallel_attention_correctness(
+ transformer_config,
+ transformer_layer_spec,
+ tmp_path_dist_ckpt,
+ atol,
+ rtol,
+ tp=1,
+ sp=False,
+ cp=1,
+ seed=123,
+ sequence_length=256,
+ micro_batch_size=4,
+):
+ # Model initialization function
+ def initialize_gpt_model(
+ config, pre_process=True, post_process=True, vp_stage=None, pg_collection=None
+ ):
+ gpt_model = GPTModel(
+ config=config,
+ transformer_layer_spec=transformer_layer_spec,
+ vocab_size=128,
+ max_sequence_length=sequence_length,
+ pre_process=pre_process,
+ post_process=post_process,
+ vp_stage=vp_stage,
+ pg_collection=pg_collection,
+ )
+ return gpt_model
+
+ # Initialize baseline parallel state
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=1, pipeline_model_parallel_size=1, context_parallel_size=1
+ )
+
+ # Initialize input hidden states
+ torch.manual_seed(seed)
+ model_parallel_cuda_manual_seed(seed)
+ input_hidden_states = (
+ torch.rand((sequence_length, micro_batch_size, transformer_config.hidden_size))
+ .cuda()
+ .bfloat16()
+ .requires_grad_(True)
+ )
+
+ with TempNamedDir(tmp_path_dist_ckpt / 'test_parallel_attn', sync=True) as ckpt_dir:
+ # Set argument
+ mock_args = parse_args(ignore_unknown_args=True)
+ set_args(mock_args)
+
+ # Initialize baseline model
+ init_basic_mock_args(mock_args, 1, 1, bf16=True)
+ mock_args.context_parallel_size = 1
+ mock_args.sequence_parallel = 1
+ gpt_model = unwrap_model(get_model(initialize_gpt_model, config=transformer_config))
+
+ # Initialize args and save checkpoint
+ init_checkpointing_mock_args(mock_args, ckpt_dir, False)
+ mock_args.no_save_optim = True
+ mock_args.no_save_rng = True
+ mock_args.no_load_optim = True
+ mock_args.no_load_rng = True
+ save_checkpoint(10, gpt_model, None, None, 0)
+
+ # Calculate baseline output
+ attention = gpt_model[0].decoder.layers[0].self_attention
+ output_hidden_states_baseline, bias_hidden_states_baseline = attention(
+ input_hidden_states, attention_mask=None
+ )
+ output_hidden_states_baseline.sum().backward()
+
+ # Save baseline output
+ input_grad_baseline = input_hidden_states.grad.detach()
+ output_hidden_states_baseline = output_hidden_states_baseline.detach()
+ bias_hidden_states_baseline = bias_hidden_states_baseline
+ if bias_hidden_states_baseline is not None:
+ bias_hidden_states_baseline = bias_hidden_states_baseline.detach()
+ has_bias = True
+ else:
+ has_bias = False
+
+ # Initialize parallel model
+ Utils.destroy_model_parallel()
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=tp, pipeline_model_parallel_size=1, context_parallel_size=cp
+ )
+ torch.manual_seed(seed)
+ model_parallel_cuda_manual_seed(seed)
+ transformer_config.context_parallel_size = cp
+ transformer_config.tensor_model_parallel_size = tp
+ transformer_config.sequence_parallel = sp
+ init_basic_mock_args(mock_args, tp, 1, bf16=True)
+ mock_args.context_parallel_size = cp
+ mock_args.sequence_parallel = sp
+ gpt_model = unwrap_model(get_model(initialize_gpt_model, config=transformer_config))
+ with mock.patch('megatron.training.checkpointing.check_checkpoint_args'):
+ with mock.patch('megatron.training.checkpointing.update_num_microbatches'):
+ load_checkpoint(gpt_model, None, None)
+
+ # Function to get tensor on this tp and cp rank
+ cp_group = parallel_state.get_context_parallel_group()
+ tp_rank = parallel_state.get_tensor_model_parallel_rank()
+
+ def get_tensor_on_this_rank(tensor):
+ if cp > 1:
+ tensor = get_tensor_on_this_cp_rank(tensor, 0, cp_group)
+ if tp > 1 and sp:
+ sp_seg = sequence_length // tp // cp
+ tensor = tensor[tp_rank * sp_seg : (tp_rank + 1) * sp_seg]
+ return tensor
+
+ # Calculate parallel model output
+ input_hidden_states = get_tensor_on_this_rank(input_hidden_states)
+ input_hidden_states = input_hidden_states.detach().requires_grad_(True)
+ parallel_attention = gpt_model[0].decoder.layers[0].self_attention
+ output_hidden_states_parallel, bias_hidden_states_parallel = parallel_attention(
+ input_hidden_states, attention_mask=None
+ )
+ output_hidden_states_parallel.sum().backward()
+ input_grad_parallel = input_hidden_states.grad.detach()
+
+ # Check if the output is close
+ output_hidden_states_baseline = get_tensor_on_this_rank(output_hidden_states_baseline)
+ input_grad_baseline = get_tensor_on_this_rank(input_grad_baseline)
+
+ assert torch.all(
+ ~torch.isnan(output_hidden_states_baseline)
+ ), "output_hidden_states_baseline contains nan"
+ assert torch.all(
+ ~torch.isinf(output_hidden_states_baseline)
+ ), "output_hidden_states_baseline contains inf"
+ assert torch.all(~torch.isnan(input_grad_baseline)), "input_grad_baseline contains nan"
+ assert torch.all(~torch.isinf(input_grad_baseline)), "input_grad_baseline contains inf"
+ assert torch.all(
+ ~torch.isnan(output_hidden_states_parallel)
+ ), "output_hidden_states_parallel contains nan"
+ assert torch.all(
+ ~torch.isinf(output_hidden_states_parallel)
+ ), "output_hidden_states_parallel contains inf"
+ assert torch.all(~torch.isnan(input_grad_parallel)), "input_grad_parallel contains nan"
+ assert torch.all(~torch.isinf(input_grad_parallel)), "input_grad_parallel contains inf"
+ if has_bias:
+ assert torch.all(
+ ~torch.isnan(bias_hidden_states_baseline)
+ ), "bias_hidden_states_baseline contains nan"
+ assert torch.all(
+ ~torch.isinf(bias_hidden_states_baseline)
+ ), "bias_hidden_states_baseline contains inf"
+ assert torch.all(
+ ~torch.isnan(bias_hidden_states_parallel)
+ ), "bias_hidden_states_parallel contains nan"
+ assert torch.all(
+ ~torch.isinf(bias_hidden_states_parallel)
+ ), "bias_hidden_states_parallel contains inf"
+
+ torch.testing.assert_close(
+ output_hidden_states_baseline,
+ output_hidden_states_parallel,
+ atol=atol,
+ rtol=rtol,
+ msg=lambda msg: f"Mismatch in output_hidden_states: {msg}",
+ )
+ torch.testing.assert_close(
+ input_grad_baseline,
+ input_grad_parallel,
+ atol=atol,
+ rtol=rtol,
+ msg=lambda msg: f"Mismatch in input_grad: {msg}",
+ )
+ if has_bias:
+ torch.testing.assert_close(
+ bias_hidden_states_baseline,
+ bias_hidden_states_parallel,
+ atol=atol,
+ rtol=rtol,
+ msg=lambda msg: f"Mismatch in bias_hidden_states: {msg}",
+ )
+
+ Utils.destroy_model_parallel()
+
+
+@pytest.mark.parametrize("apply_rope_fusion", [False, True])
+@pytest.mark.parametrize(
+ ("tp", "sp", "cp"),
+ [
+ (4, False, 1), # TP w/o SP
+ (4, True, 1), # TP w/ SP
+ (1, False, 4), # CP
+ (2, False, 2), # CP + TP w/o SP
+ (2, True, 2), # CP + TP w/ SP
+ ],
+)
+@pytest.mark.parametrize("qk_layernorm", [False, True])
+@pytest.mark.parametrize("output_gate", [False, True])
+def test_parallel_attention_correctness(
+ tmp_path_dist_ckpt, apply_rope_fusion, tp, sp, cp, qk_layernorm, output_gate
+):
+ transformer_config = TransformerConfig(
+ num_layers=1,
+ hidden_size=128,
+ num_attention_heads=4,
+ normalization="RMSNorm",
+ bf16=True,
+ qk_layernorm=qk_layernorm,
+ apply_rope_fusion=apply_rope_fusion,
+ attention_output_gate=output_gate,
+ hidden_dropout=0.0,
+ attention_dropout=0.0,
+ )
+
+ transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec(qk_layernorm=qk_layernorm)
+ atol, rtol = 1e-2, 1e-2
+
+ _test_parallel_attention_correctness(
+ transformer_config,
+ transformer_layer_spec,
+ tmp_path_dist_ckpt,
+ atol=atol,
+ rtol=rtol,
+ tp=tp,
+ sp=sp,
+ cp=cp,
+ seed=123,
+ sequence_length=256,
+ )
+
+
+@pytest.mark.parametrize("sp", [True, False])
+@pytest.mark.parametrize("output_gate", [False, True])
+def test_parallel_attention_correctness_num_query_groups_less_than_tp_size(
+ tmp_path_dist_ckpt, sp, output_gate
+):
+ transformer_config = TransformerConfig(
+ num_layers=1,
+ hidden_size=128,
+ num_attention_heads=8,
+ num_query_groups=2,
+ normalization="RMSNorm",
+ bf16=True,
+ attention_output_gate=output_gate,
+ hidden_dropout=0.0,
+ attention_dropout=0.0,
+ )
+
+ transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec()
+ atol, rtol = 1e-2, 1e-2
+
+ _test_parallel_attention_correctness(
+ transformer_config,
+ transformer_layer_spec,
+ tmp_path_dist_ckpt,
+ atol=atol,
+ rtol=rtol,
+ tp=4,
+ sp=sp,
+ seed=123,
+ sequence_length=256,
+ )
diff --git a/tests/unit_tests/transformer/test_attention_no_rope.py b/tests/unit_tests/transformer/test_attention_no_rope.py
index 30e11609e57..2c79d7a3011 100644
--- a/tests/unit_tests/transformer/test_attention_no_rope.py
+++ b/tests/unit_tests/transformer/test_attention_no_rope.py
@@ -3,7 +3,9 @@
import pytest
import torch
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
+from megatron.core.models.gpt.gpt_layer_specs import (
+ get_gpt_layer_with_transformer_engine_submodules,
+)
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer.attention import SelfAttention
from megatron.core.transformer.enums import AttnMaskType
@@ -30,7 +32,7 @@ def setup_method(self, method):
)
self.parallel_attention = SelfAttention(
self.transformer_config,
- get_gpt_layer_with_transformer_engine_spec().submodules.self_attention.submodules,
+ get_gpt_layer_with_transformer_engine_submodules().self_attention.submodules,
layer_number=1,
attn_mask_type=AttnMaskType.causal,
)
@@ -176,7 +178,7 @@ def test_checkpointed_gpu_forward(self):
checkpointed_parallel_attention = SelfAttention(
transformer_config,
- get_gpt_layer_with_transformer_engine_spec().submodules.self_attention.submodules,
+ get_gpt_layer_with_transformer_engine_submodules().self_attention.submodules,
layer_number=1,
attn_mask_type=AttnMaskType.causal,
)
diff --git a/tests/unit_tests/transformer/test_attention_packed_seq.py b/tests/unit_tests/transformer/test_attention_packed_seq.py
index e6e2c847395..9c8f5bd2ce6 100644
--- a/tests/unit_tests/transformer/test_attention_packed_seq.py
+++ b/tests/unit_tests/transformer/test_attention_packed_seq.py
@@ -3,7 +3,9 @@
import pytest
import torch
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
+from megatron.core.models.gpt.gpt_layer_specs import (
+ get_gpt_layer_with_transformer_engine_submodules,
+)
from megatron.core.packed_seq_params import PackedSeqParams
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer.attention import SelfAttention
@@ -62,7 +64,7 @@ def setup_method(self, method):
)
self.parallel_attention = SelfAttention(
self.transformer_config,
- get_gpt_layer_with_transformer_engine_spec().submodules.self_attention.submodules,
+ get_gpt_layer_with_transformer_engine_submodules().self_attention.submodules,
layer_number=1,
attn_mask_type=AttnMaskType.causal,
)
@@ -138,7 +140,7 @@ def test_checkpointed_gpu_forward(self):
transformer_config.recompute_granularity = 'selective'
checkpointed_parallel_attention = SelfAttention(
transformer_config,
- get_gpt_layer_with_transformer_engine_spec().submodules.self_attention.submodules,
+ get_gpt_layer_with_transformer_engine_submodules().self_attention.submodules,
layer_number=1,
attn_mask_type=AttnMaskType.causal,
)
diff --git a/tests/unit_tests/transformer/test_cuda_graphs.py b/tests/unit_tests/transformer/test_cuda_graphs.py
index 4696a3ed439..bfde9ff9cf1 100644
--- a/tests/unit_tests/transformer/test_cuda_graphs.py
+++ b/tests/unit_tests/transformer/test_cuda_graphs.py
@@ -23,6 +23,7 @@
from megatron.core.pipeline_parallel.schedules import set_current_microbatch
from megatron.core.process_groups_config import ProcessGroupCollection
from megatron.core.ssm.mamba_block import MambaStack
+from megatron.core.ssm.mamba_hybrid_layer_allocation import validate_segment_layers
from megatron.core.tensor_parallel.random import (
HAVE_TE,
initialize_rng_tracker,
@@ -37,6 +38,7 @@
from megatron.core.transformer.moe.fused_a2a import reset_hybrid_ep_buffer
from megatron.core.transformer.transformer_block import TransformerBlock
from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.transformer.transformer_layer import TransformerLayerSubmodules
from megatron.core.utils import is_fa_min_version, is_te_min_version
from megatron.training.arguments import core_transformer_config_from_args, parse_args, validate_args
from megatron.training.global_vars import (
@@ -327,6 +329,7 @@ def setup_method(self, method):
# Get layer specs
language_layer_spec = get_gpt_layer_with_transformer_engine_spec()
vision_layer_spec = get_vit_layer_with_transformer_engine_spec()
+ assert isinstance(language_layer_spec.submodules, TransformerLayerSubmodules)
vision_projection_spec = deepcopy(language_layer_spec.submodules.mlp.submodules)
# Set vision model type
@@ -472,12 +475,13 @@ def setup_method(self, method):
def get_pg_collection():
return ProcessGroupCollection.use_mpu_process_groups(required_pgs=['tp', 'pp', 'cp'])
- def get_mamba_block(hybrid_override_pattern):
+ def get_mamba_block(hybrid_layer_pattern):
+ layer_type_list = validate_segment_layers(hybrid_layer_pattern)
transformer_config = TransformerConfig(
hidden_size=256, # The Mamba layer places several constraints on this
# Need to specify num_attention_heads and num_layers or TransformerConfig
# will generate errors.
- num_layers=len(hybrid_override_pattern),
+ num_layers=len(layer_type_list),
num_attention_heads=4,
use_cpu_initialization=True,
cuda_graph_impl="local",
@@ -486,11 +490,12 @@ def get_mamba_block(hybrid_override_pattern):
return MambaStack(
transformer_config,
modules,
- hybrid_override_pattern=hybrid_override_pattern,
+ layer_type_list=layer_type_list,
+ pp_layer_offset=0,
pg_collection=get_pg_collection(),
)
- self.mamba_block = get_mamba_block(hybrid_override_pattern="M-M*-")
+ self.mamba_block = get_mamba_block(hybrid_layer_pattern="M-M*-")
self.transformer_config = self.mamba_block.config
def teardown_method(self, method):
@@ -1039,10 +1044,6 @@ def test_moe_partial_cudagraph(self, ep_size, moe_dropless_dispatcher, moe_dispa
extra_kwargs["moe_token_dispatcher_type"] = "flex"
extra_kwargs["moe_flex_dispatcher_backend"] = "deepep"
elif moe_dispatcher_type == "hybridep":
- pytest.skip(
- "Currently, the Hybrid EP is broken. "
- "Temporarily skip the test and wait for the fix."
- )
if not is_hybrid_ep_available():
pytest.skip("Hybrid EP is not available")
extra_kwargs["moe_token_dispatcher_type"] = "flex"
@@ -1052,8 +1053,6 @@ def test_moe_partial_cudagraph(self, ep_size, moe_dropless_dispatcher, moe_dispa
if not moe_dropless_dispatcher:
if moe_dispatcher_type == "deepep":
pytest.skip("Deep EP doesn't support drop&pad MoE")
- if moe_dispatcher_type == "hybridep" and ep_size == 1:
- pytest.skip("Hybrid EP doesn't support drop&pad MoE with ep_size == 1")
extra_kwargs["moe_expert_capacity_factor"] = 1.0
extra_kwargs["moe_pad_expert_input_to_capacity"] = True
diff --git a/tests/unit_tests/transformer/test_mlp.py b/tests/unit_tests/transformer/test_mlp.py
index d2c25e0cc53..45e0df9ad0a 100644
--- a/tests/unit_tests/transformer/test_mlp.py
+++ b/tests/unit_tests/transformer/test_mlp.py
@@ -1,9 +1,10 @@
# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+
import pytest
import torch
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_spec
+from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_submodules
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer.mlp import MLP
from megatron.core.transformer.transformer_config import TransformerConfig
@@ -18,7 +19,7 @@ def setup_method(self, method):
transformer_config = TransformerConfig(
num_layers=2, hidden_size=12, num_attention_heads=4, use_cpu_initialization=True
)
- self.mlp = MLP(transformer_config, get_gpt_layer_local_spec().submodules.mlp.submodules)
+ self.mlp = MLP(transformer_config, get_gpt_layer_local_submodules().mlp.submodules)
def teardown_method(self, method):
Utils.destroy_model_parallel()
diff --git a/tests/unit_tests/transformer/test_multi_latent_attention.py b/tests/unit_tests/transformer/test_multi_latent_attention.py
index beabe2885be..36f438fa07f 100644
--- a/tests/unit_tests/transformer/test_multi_latent_attention.py
+++ b/tests/unit_tests/transformer/test_multi_latent_attention.py
@@ -1,27 +1,32 @@
# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
import os
-from functools import partial
-from importlib.metadata import version
from inspect import signature
from unittest import mock
import pytest
import torch
-import transformer_engine as te
from megatron.core import parallel_state
from megatron.core.extensions.transformer_engine_spec_provider import TESpecProvider
from megatron.core.models.common.embeddings.rope_utils import (
get_pos_emb_on_this_cp_rank as get_tensor_on_this_cp_rank,
)
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
+from megatron.core.models.gpt.gpt_layer_specs import (
+ get_gpt_layer_with_transformer_engine_spec,
+ get_gpt_layer_with_transformer_engine_submodules,
+)
from megatron.core.models.gpt.gpt_model import GPTModel
from megatron.core.packed_seq_params import PackedSeqParams
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer.attention import Attention
from megatron.core.transformer.enums import AttnMaskType
-from megatron.core.transformer.multi_latent_attention import MLASelfAttention, MultiLatentAttention
+from megatron.core.transformer.multi_latent_attention import (
+ FusedMLASelfAttention,
+ MLASelfAttention,
+ MLASelfAttentionSubmodules,
+ MultiLatentAttention,
+)
from megatron.core.transformer.transformer_config import MLATransformerConfig
from megatron.core.utils import is_te_min_version, is_torch_min_version
from megatron.training.arguments import parse_args
@@ -92,15 +97,26 @@ def make_test_packed_seq_params_with_padding(
def get_mla_self_attn_submodules(linear_qkv_down_proj=None):
- submodules = get_gpt_layer_with_transformer_engine_spec(
+ submodules = get_gpt_layer_with_transformer_engine_submodules(
multi_latent_attention=True
- ).submodules.self_attention.submodules
+ ).self_attention.submodules
+ assert isinstance(submodules, MLASelfAttentionSubmodules)
if linear_qkv_down_proj is not None:
submodules.linear_q_down_proj = linear_qkv_down_proj
submodules.linear_kv_down_proj = linear_qkv_down_proj
return submodules
+def get_fused_mla_submodules():
+ """Get submodules for FusedMLASelfAttention via the mla_down_proj_fusion spec path."""
+ submodules = get_gpt_layer_with_transformer_engine_submodules(
+ multi_latent_attention=True, mla_down_proj_fusion=True
+ ).self_attention.submodules
+ assert isinstance(submodules, MLASelfAttentionSubmodules)
+ assert submodules.linear_qkv_down_proj is not None
+ return submodules
+
+
backend = TESpecProvider()
linear_qkv_down_proj_options = [backend.linear(), backend.column_parallel_linear()]
@@ -1544,3 +1560,280 @@ def get_tensor_on_this_rank(tensor):
os.environ.clear()
os.environ.update(_environ)
+
+
+@pytest.mark.parametrize("rope_type", ('yarn', 'rope'))
+class TestFusedMLASelfAttention:
+
+ @pytest.fixture(scope='function', autouse=True)
+ def setup_and_teardown(self, rope_type):
+ Utils.initialize_model_parallel(1, 1)
+ model_parallel_cuda_manual_seed(123)
+ self.transformer_config = MLATransformerConfig(
+ num_layers=2,
+ hidden_size=12,
+ num_attention_heads=4,
+ use_cpu_initialization=True,
+ q_lora_rank=32,
+ kv_lora_rank=32,
+ qk_head_dim=128,
+ v_head_dim=128,
+ qk_pos_emb_head_dim=64,
+ rope_type=rope_type,
+ rotary_base=10000,
+ original_max_position_embeddings=32,
+ )
+ self.fused_attention = FusedMLASelfAttention(
+ self.transformer_config,
+ get_fused_mla_submodules(),
+ layer_number=1,
+ attn_mask_type=AttnMaskType.causal,
+ )
+
+ def teardown_method(self, method):
+ Utils.destroy_model_parallel()
+
+ def test_constructor(self):
+ assert isinstance(self.fused_attention, FusedMLASelfAttention)
+ assert isinstance(self.fused_attention, MLASelfAttention)
+ assert self.fused_attention.layer_number == 1
+ assert hasattr(self.fused_attention, 'linear_qkv_down_proj')
+
+ def test_fused_weight_shape(self):
+ config = self.transformer_config
+ expected_out = (
+ config.q_lora_rank
+ + config.kv_lora_rank
+ + config.qk_pos_emb_head_dim
+ + self.fused_attention.kv_down_proj_mxfp8_padding
+ )
+ weight = self.fused_attention.linear_qkv_down_proj.weight
+ assert weight.shape[0] == expected_out
+ assert weight.shape[1] == config.hidden_size
+
+ def test_qkv_down_projection_split(self):
+ if not is_te_min_version("1.10.0"):
+ pytest.skip("Requires TE >= 1.10.0")
+ config = self.transformer_config
+ self.fused_attention.cuda()
+
+ seq_len, batch = 16, 2
+ hidden = torch.randn(seq_len, batch, config.hidden_size).cuda()
+ q_compressed, kv_combined = self.fused_attention._qkv_down_projection(hidden)
+
+ kv_width = (
+ config.kv_lora_rank
+ + config.qk_pos_emb_head_dim
+ + self.fused_attention.kv_down_proj_mxfp8_padding
+ )
+ assert q_compressed.shape == (seq_len, batch, config.q_lora_rank)
+ assert kv_combined.shape == (seq_len, batch, kv_width)
+
+ def test_gpu_forward(self):
+ if not is_te_min_version("1.10.0"):
+ pytest.skip("Requires TE >= 1.10.0")
+
+ config = self.fused_attention.config
+ sequence_length = 32
+ micro_batch_size = 2
+
+ self.fused_attention.cuda()
+
+ hidden_states = torch.ones((sequence_length, micro_batch_size, config.hidden_size)).cuda()
+ attention_mask = torch.ones((1, 1, sequence_length, sequence_length), dtype=bool).cuda()
+
+ output, bias = self.fused_attention(hidden_states, attention_mask)
+
+ assert output.shape[0] == sequence_length
+ assert output.shape[1] == micro_batch_size
+ assert output.shape[2] == config.hidden_size
+ assert bias.shape[0] == config.hidden_size
+
+ def test_gpu_forward_bf16(self):
+ if not is_te_min_version("1.10.0"):
+ pytest.skip("Requires TE >= 1.10.0")
+
+ config = self.fused_attention.config
+ sequence_length = 32
+ micro_batch_size = 2
+
+ self.fused_attention.cuda().bfloat16()
+
+ hidden_states = (
+ torch.ones((sequence_length, micro_batch_size, config.hidden_size)).cuda().bfloat16()
+ )
+ attention_mask = torch.ones((1, 1, sequence_length, sequence_length), dtype=bool).cuda()
+
+ output, bias = self.fused_attention(hidden_states, attention_mask)
+
+ assert output.shape[0] == sequence_length
+ assert output.shape[1] == micro_batch_size
+ assert output.shape[2] == config.hidden_size
+ assert output.dtype == torch.bfloat16
+
+
+class TestFusedMLAGradientFlow:
+
+ @pytest.fixture(scope='function', autouse=True)
+ def setup_and_teardown(self):
+ Utils.initialize_model_parallel(1, 1)
+ model_parallel_cuda_manual_seed(123)
+ self.transformer_config = MLATransformerConfig(
+ num_layers=2,
+ hidden_size=12,
+ num_attention_heads=4,
+ use_cpu_initialization=True,
+ q_lora_rank=32,
+ kv_lora_rank=32,
+ qk_head_dim=128,
+ v_head_dim=128,
+ qk_pos_emb_head_dim=64,
+ rope_type="rope",
+ rotary_base=10000,
+ original_max_position_embeddings=32,
+ )
+
+ def teardown_method(self, method):
+ Utils.destroy_model_parallel()
+
+ def test_backward_pass(self):
+ if not is_te_min_version("1.10.0"):
+ pytest.skip("Requires TE >= 1.10.0")
+
+ config = self.transformer_config
+ fused = FusedMLASelfAttention(
+ config, get_fused_mla_submodules(), layer_number=1, attn_mask_type=AttnMaskType.causal
+ )
+ fused.cuda()
+
+ seq_len, batch = 32, 2
+ hidden_states = torch.randn(
+ seq_len, batch, config.hidden_size, device='cuda', requires_grad=True
+ )
+ attention_mask = torch.ones((1, 1, seq_len, seq_len), dtype=bool, device='cuda')
+
+ output, bias = fused(hidden_states, attention_mask)
+ loss = output.sum()
+ loss.backward()
+
+ assert fused.linear_qkv_down_proj.weight.grad is not None
+ assert (
+ fused.linear_qkv_down_proj.weight.grad.shape == fused.linear_qkv_down_proj.weight.shape
+ )
+ assert hidden_states.grad is not None
+
+
+class TestFusedMLALoadFromStateDict:
+
+ @pytest.fixture(scope='function', autouse=True)
+ def setup_and_teardown(self):
+ Utils.initialize_model_parallel(1, 1)
+ model_parallel_cuda_manual_seed(123)
+ self.transformer_config = MLATransformerConfig(
+ num_layers=2,
+ hidden_size=12,
+ num_attention_heads=4,
+ use_cpu_initialization=True,
+ q_lora_rank=32,
+ kv_lora_rank=32,
+ qk_head_dim=128,
+ v_head_dim=128,
+ qk_pos_emb_head_dim=64,
+ rope_type="rope",
+ rotary_base=10000,
+ original_max_position_embeddings=32,
+ )
+
+ def teardown_method(self, method):
+ Utils.destroy_model_parallel()
+
+ def test_load_unfused_state_dict(self):
+ if not is_te_min_version("1.10.0"):
+ pytest.skip("Requires TE >= 1.10.0")
+
+ unfused = MLASelfAttention(
+ self.transformer_config,
+ get_mla_self_attn_submodules(),
+ layer_number=1,
+ attn_mask_type=AttnMaskType.causal,
+ )
+ fused = FusedMLASelfAttention(
+ self.transformer_config,
+ get_fused_mla_submodules(),
+ layer_number=1,
+ attn_mask_type=AttnMaskType.causal,
+ )
+
+ unfused_sd = unfused.state_dict()
+
+ q_down_keys = [k for k in unfused_sd if 'linear_q_down_proj' in k]
+ kv_down_keys = [k for k in unfused_sd if 'linear_kv_down_proj' in k]
+ assert len(q_down_keys) > 0, "Expected q_down_proj keys in unfused state dict"
+ assert len(kv_down_keys) > 0, "Expected kv_down_proj keys in unfused state dict"
+
+ fused.load_state_dict(unfused_sd, strict=False)
+
+ config = self.transformer_config
+ expected_out = config.q_lora_rank + config.kv_lora_rank + config.qk_pos_emb_head_dim
+ assert fused.linear_qkv_down_proj.weight.shape[0] == expected_out
+
+ q_w = unfused_sd['linear_q_down_proj.weight']
+ kv_w = unfused_sd['linear_kv_down_proj.weight']
+ expected_fused = torch.cat([q_w, kv_w], dim=0)
+ torch.testing.assert_close(fused.linear_qkv_down_proj.weight.data, expected_fused)
+
+ def test_sharded_state_dict_splits_back(self):
+ if not is_te_min_version("1.10.0"):
+ pytest.skip("Requires TE >= 1.10.0")
+
+ fused = FusedMLASelfAttention(
+ self.transformer_config,
+ get_fused_mla_submodules(),
+ layer_number=1,
+ attn_mask_type=AttnMaskType.causal,
+ )
+
+ sharded_sd = fused.sharded_state_dict(prefix="")
+ assert any(
+ 'linear_q_down_proj.weight' in k for k in sharded_sd
+ ), f"Expected linear_q_down_proj.weight in sharded state dict, got keys: {list(sharded_sd.keys())}"
+ assert any(
+ 'linear_kv_down_proj.weight' in k for k in sharded_sd
+ ), f"Expected linear_kv_down_proj.weight in sharded state dict, got keys: {list(sharded_sd.keys())}"
+ assert not any(
+ 'linear_qkv_down_proj.weight' in k for k in sharded_sd
+ ), f"Unexpected linear_qkv_down_proj.weight in sharded state dict"
+
+
+class TestFusedMLARequiresQLora:
+
+ @pytest.fixture(scope='function', autouse=True)
+ def setup_and_teardown(self):
+ Utils.initialize_model_parallel(1, 1)
+ model_parallel_cuda_manual_seed(123)
+
+ def teardown_method(self, method):
+ Utils.destroy_model_parallel()
+
+ def test_raises_without_q_lora_rank(self):
+ config = MLATransformerConfig(
+ num_layers=2,
+ hidden_size=12,
+ num_attention_heads=4,
+ use_cpu_initialization=True,
+ q_lora_rank=None,
+ kv_lora_rank=32,
+ qk_head_dim=128,
+ v_head_dim=128,
+ qk_pos_emb_head_dim=64,
+ rope_type="rope",
+ rotary_base=10000,
+ original_max_position_embeddings=32,
+ )
+ with pytest.raises(AssertionError, match="q_lora_rank"):
+ FusedMLASelfAttention(
+ config,
+ get_fused_mla_submodules(),
+ layer_number=1,
+ attn_mask_type=AttnMaskType.causal,
+ )
diff --git a/tests/unit_tests/transformer/test_multi_token_prediction.py b/tests/unit_tests/transformer/test_multi_token_prediction.py
index a6eb4749d9a..ff2ca7439ff 100644
--- a/tests/unit_tests/transformer/test_multi_token_prediction.py
+++ b/tests/unit_tests/transformer/test_multi_token_prediction.py
@@ -7,6 +7,7 @@
import torch
from megatron.core.enums import ModelType
+from megatron.core.extensions.transformer_engine import HAVE_TE
from megatron.core.models.gpt.gpt_layer_specs import (
get_gpt_layer_local_spec,
get_gpt_layer_with_transformer_engine_spec,
@@ -40,12 +41,10 @@
from tests.unit_tests.dist_checkpointing import TempNamedDir
from tests.unit_tests.test_utilities import Utils
-try:
+if HAVE_TE:
from megatron.core.extensions.transformer_engine import TEColumnParallelGroupedLinear
-
- HAVE_TE = True
-except ImportError:
- HAVE_TE = False
+else:
+ TEColumnParallelGroupedLinear = None
_SEED = 1
@@ -206,7 +205,6 @@ def create_test_args(
args.lr = 3e-5
args.attention_dropout = 0.0
args.hidden_dropout = 0.0
- args.async_tensor_model_parallel_allreduce = False
args.no_save_optim = True
args.no_load_optim = True
args.no_load_rng = True
@@ -704,7 +702,7 @@ def teardown_method(self, method):
def model_provider(self, pre_process=True, post_process=True, **config_kwargs):
"""Model provider for Mamba hybrid models with MTP.
- Uses the unified pattern syntax where MTP is configured via hybrid_override_pattern:
+ Uses the unified pattern syntax where MTP is configured via hybrid_layer_pattern:
Format: "///..."
Example: "M*M*/M*/M*" = main decoder "M*M*", MTP pattern "M*" with 2 depths
"""
@@ -712,7 +710,7 @@ def model_provider(self, pre_process=True, post_process=True, **config_kwargs):
args = get_args()
config = core_transformer_config_from_args(args)
- # MTP is configured via unified pattern in hybrid_override_pattern
+ # MTP is configured via unified pattern in hybrid_layer_pattern
# MambaModel creates the MTP block internally based on the parsed pattern
model = MambaModel(
config=config,
@@ -721,9 +719,7 @@ def model_provider(self, pre_process=True, post_process=True, **config_kwargs):
max_sequence_length=args.max_position_embeddings,
pre_process=pre_process,
post_process=post_process,
- hybrid_attention_ratio=args.hybrid_attention_ratio,
- hybrid_mlp_ratio=args.hybrid_mlp_ratio,
- hybrid_override_pattern=args.hybrid_override_pattern,
+ hybrid_layer_pattern=args.hybrid_layer_pattern,
fp16_lm_cross_entropy=args.fp16_lm_cross_entropy,
parallel_output=True,
share_embeddings_and_output_weights=not args.untie_embeddings_and_output_weights,
@@ -740,7 +736,6 @@ def create_test_args(
sys.argv = ['test_multi_token_prediction_mamba.py']
args = parse_args()
- args.num_layers = 4
args.mtp_num_layers = 2
args.mtp_loss_scaling_factor = 0.1
args.vocab_size = 128800
@@ -761,15 +756,12 @@ def create_test_args(
args.lr = 3e-5
args.attention_dropout = 0.0
args.hidden_dropout = 0.0
- args.async_tensor_model_parallel_allreduce = False
args.no_save_optim = True
args.no_load_optim = True
args.no_load_rng = True
args.bf16 = True
- args.hybrid_attention_ratio = 0.5
- args.hybrid_mlp_ratio = 0.0
# Unified pattern: "main/mtp/mtp" - main decoder "M*M*", MTP pattern "M*" with 2 depths
- args.hybrid_override_pattern = "M*M*/M*/M*"
+ args.hybrid_layer_pattern = "M*M*/M*/M*"
args.spec = "megatron.core.models.mamba.mamba_layer_specs.mamba_stack_spec"
if fp8 is not None:
diff --git a/tests/unit_tests/transformer/test_mup.py b/tests/unit_tests/transformer/test_mup.py
new file mode 100644
index 00000000000..f1d99cad1e6
--- /dev/null
+++ b/tests/unit_tests/transformer/test_mup.py
@@ -0,0 +1,663 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""Tests for Maximal Update Parameterization (MuP) implementation.
+
+These tests verify that MuP is correctly implemented in Megatron-LM:
+1. Config validation and width_mult computation
+2. Initialization scaling
+3. Attention scaling
+4. LR override computation
+"""
+
+import logging
+import math
+from unittest.mock import patch
+
+import pytest
+import torch
+
+from megatron.core.optimizer import get_mup_config_overrides, get_standard_config_overrides
+from megatron.core.optimizer.optimizer_config import OptimizerConfig
+from megatron.core.optimizer_param_scheduler import combine_param_group_overrides
+from megatron.core.transformer.multi_token_prediction import process_mtp_loss
+from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.utils import init_method_normal, mup_scaled_init_method_normal
+
+
+class TestMuPConfigValidation:
+ """Tests for MuP config validation and width_mult computation."""
+
+ def test_mup_defaults_base_hidden_size(self):
+ """use_mup without base_hidden_size defaults to hidden_size (width_mult=1.0)."""
+ config = TransformerConfig(
+ hidden_size=512,
+ num_layers=4,
+ num_attention_heads=8,
+ use_mup=True,
+ # mup_base_hidden_size not set - should default to hidden_size
+ )
+ assert config.mup_base_hidden_size == 512
+ assert config.mup_width_mult == 1.0
+
+ def test_mup_width_mult_calculation(self):
+ """width_mult = hidden_size / base_hidden_size."""
+ config = TransformerConfig(
+ hidden_size=1024,
+ num_layers=4,
+ num_attention_heads=16,
+ use_mup=True,
+ mup_base_hidden_size=256,
+ )
+ assert config.mup_width_mult == 4.0
+
+ def test_mup_width_mult_fractional(self):
+ """width_mult can be fractional (smaller than base)."""
+ config = TransformerConfig(
+ hidden_size=128,
+ num_layers=4,
+ num_attention_heads=2,
+ use_mup=True,
+ mup_base_hidden_size=256,
+ )
+ assert config.mup_width_mult == 0.5
+
+ def test_mup_backward_compatible(self):
+ """Default config unchanged when MuP disabled."""
+ config = TransformerConfig(hidden_size=512, num_layers=4, num_attention_heads=8)
+ assert config.use_mup is False
+ assert config.mup_width_mult == 1.0
+ assert config.mup_base_hidden_size is None
+
+ def test_mup_base_hidden_size_must_be_positive(self):
+ """mup_base_hidden_size must be positive."""
+ with pytest.raises(AssertionError) as exc_info:
+ TransformerConfig(
+ hidden_size=512,
+ num_layers=4,
+ num_attention_heads=8,
+ use_mup=True,
+ mup_base_hidden_size=0,
+ )
+ assert "positive" in str(exc_info.value).lower()
+
+
+class TestMuPInitMethods:
+ """Tests for MuP initialization methods."""
+
+ def test_mup_init_variance_scaling(self):
+ """Init std scales as 1/sqrt(width_mult)."""
+ base_sigma = 0.02
+ width_mult = 4.0
+
+ init_fn = init_method_normal(base_sigma / math.sqrt(width_mult))
+ tensor = torch.empty(1000, 1000)
+ init_fn(tensor)
+
+ expected_std = base_sigma / math.sqrt(width_mult) # 0.02/sqrt(4) = 0.01
+ actual_std = tensor.std().item()
+
+ # Allow 5% tolerance for statistical variance
+ assert (
+ abs(actual_std - expected_std) < 0.002
+ ), f"Expected std ~{expected_std:.4f}, got {actual_std:.4f}"
+
+ def test_mup_scaled_init_variance(self):
+ """MuP scaled init combines depth and width scaling."""
+ sigma = 0.02
+ num_layers = 8
+ width_mult = 4.0
+
+ init_fn = mup_scaled_init_method_normal(sigma, num_layers, width_mult)
+ tensor = torch.empty(1000, 1000)
+ init_fn(tensor)
+
+ # std = sigma / (sqrt(2*num_layers) * sqrt(width_mult))
+ expected_std = sigma / (math.sqrt(2 * num_layers) * math.sqrt(width_mult))
+ actual_std = tensor.std().item()
+
+ # Allow 10% tolerance
+ assert (
+ abs(actual_std - expected_std) < expected_std * 0.1
+ ), f"Expected std ~{expected_std:.6f}, got {actual_std:.6f}"
+
+
+class TestMuPAttentionScaling:
+ """Tests for MuP attention scaling."""
+
+ def test_mup_attention_scale_power_1(self):
+ """MuP uses 1/d instead of 1/sqrt(d) when power=1.0."""
+ config = TransformerConfig(
+ hidden_size=512,
+ num_layers=4,
+ num_attention_heads=8,
+ use_mup=True,
+ mup_base_hidden_size=128,
+ mup_attn_scale_power=1.0, # MuP default
+ )
+ kv_channels = config.kv_channels # 512 / 8 = 64
+
+ expected_scale = 1.0 / kv_channels # 1/64 = 0.015625
+ assert config.softmax_scale == expected_scale
+
+ def test_standard_attention_scale_power_05(self):
+ """Standard uses 1/sqrt(d) when power=0.5."""
+ config = TransformerConfig(
+ hidden_size=512,
+ num_layers=4,
+ num_attention_heads=8,
+ use_mup=True,
+ mup_base_hidden_size=128,
+ mup_attn_scale_power=0.5, # Standard default
+ )
+ kv_channels = config.kv_channels # 64
+
+ expected_scale = 1.0 / math.sqrt(kv_channels) # 1/8 = 0.125
+ assert abs(config.softmax_scale - expected_scale) < 1e-6
+
+ def test_attention_scale_not_set_when_disabled(self):
+ """softmax_scale should not be auto-set when use_mup=False."""
+ config = TransformerConfig(
+ hidden_size=512, num_layers=4, num_attention_heads=8, use_mup=False # MuP disabled
+ )
+ # softmax_scale defaults to None when MuP is disabled
+ # (actual scaling is done in the attention layer)
+ assert config.softmax_scale is None
+
+
+class TestMuPWarnings:
+ """Tests for MuP configuration warnings."""
+
+ def test_mup_warns_with_custom_init_method(self):
+ """Warn when MuP is enabled and init_method is user-provided."""
+ with pytest.warns(UserWarning, match="use_mup is enabled"):
+ TransformerConfig(
+ hidden_size=512,
+ num_layers=4,
+ num_attention_heads=8,
+ use_mup=True,
+ mup_base_hidden_size=128,
+ init_method=init_method_normal(0.01),
+ )
+
+ def test_mup_warns_with_custom_output_layer_init_method(self):
+ """Warn when MuP is enabled and output_layer_init_method is user-provided."""
+ with pytest.warns(UserWarning, match="use_mup is enabled"):
+ TransformerConfig(
+ hidden_size=512,
+ num_layers=4,
+ num_attention_heads=8,
+ use_mup=True,
+ mup_base_hidden_size=128,
+ output_layer_init_method=init_method_normal(0.01),
+ )
+
+
+class TestMuPLRScaling:
+ """Tests for MuP learning rate and Adam epsilon scaling."""
+
+ def test_mup_lr_override_computation(self):
+ """Hidden LR and Adam eps scale as 1/width_mult."""
+ optimizer_config = OptimizerConfig(lr=1e-3, min_lr=1e-5)
+ width_mult = 4.0
+
+ overrides = get_mup_config_overrides(optimizer_config, width_mult)
+
+ # Should have one override for hidden layers
+ assert len(overrides) == 1
+
+ # Get the override values
+ for param_key, override in overrides.items():
+ expected_max_lr = 1e-3 / width_mult # 2.5e-4
+ expected_min_lr = 1e-5 / width_mult # 2.5e-6
+ expected_eps = optimizer_config.adam_eps / width_mult
+
+ assert abs(override['max_lr'] - expected_max_lr) < 1e-10
+ assert abs(override['min_lr'] - expected_min_lr) < 1e-10
+ assert abs(override['eps'] - expected_eps) < 1e-15
+
+ def test_mup_lr_no_scaling_at_unity(self):
+ """No LR scaling when width_mult=1.0."""
+ optimizer_config = OptimizerConfig(lr=1e-3, min_lr=1e-5)
+ width_mult = 1.0
+
+ overrides = get_mup_config_overrides(optimizer_config, width_mult)
+
+ # Should return empty dict when no scaling needed
+ assert len(overrides) == 0
+
+ def test_mup_lr_override_has_correct_predicate(self):
+ """LR override predicate correctly identifies hidden vs vector-like params.
+
+ Per MuP paper and Microsoft's mup library:
+ - Embedding layer: base LR (fan_in is vocab, finite dimension)
+ - Hidden layers: scaled LR (fan_in is hidden, infinite dimension)
+ - Output layer: base LR when tagged as embedding-class (Table 8 symmetry)
+ """
+ optimizer_config = OptimizerConfig(lr=1e-3)
+ width_mult = 4.0
+
+ overrides = get_mup_config_overrides(optimizer_config, width_mult)
+
+ # Check the predicate is set and works correctly
+ for param_key, override in overrides.items():
+ assert param_key.with_name_predicate is not None
+ predicate_fn = param_key.with_name_predicate.fn
+
+ # Create mock parameters
+ hidden_param = torch.nn.Parameter(torch.zeros(10, 10))
+
+ # Hidden param with hidden-layer name should match (scaled LR)
+ assert predicate_fn(hidden_param, 'decoder.layer.0.weight') is True
+
+ # Output layer should NOT match when tagged as embedding-class.
+ output_param = torch.nn.Parameter(torch.zeros(10, 10))
+ output_param.is_embedding_parameter = True
+ assert predicate_fn(output_param, 'output_layer.weight') is False
+
+ # Embedding layer should NOT match (base LR) when attribute is set.
+ embedding_param = torch.nn.Parameter(torch.zeros(10, 10))
+ embedding_param.is_embedding_parameter = True
+ assert predicate_fn(embedding_param, 'decoder.layer.0.weight') is False
+
+ # 1D vector-like params (biases/LN) should also keep base LR.
+ vector_like_param = torch.nn.Parameter(torch.zeros(10))
+ assert predicate_fn(vector_like_param, 'decoder.layer.0.bias') is False
+
+ # Shared embedding copies on tied output stages should also use base LR.
+ shared_embedding_param = torch.nn.Parameter(torch.zeros(10, 10))
+ shared_embedding_param.shared_embedding = True
+ assert predicate_fn(shared_embedding_param, 'output_layer.weight') is False
+
+ # Backward-compatible fallback for older modules without the attribute.
+ assert predicate_fn(hidden_param, 'embedding.word_embeddings.weight') is False
+
+ def test_mup_with_decoupled_lr_scales_hidden_only_for_lr(self):
+ """With decoupled_lr, MuP scales hidden params only; embedding/output stay decoupled."""
+ optimizer_config = OptimizerConfig(
+ lr=1e-3, min_lr=1e-5, decoupled_lr=2e-4, decoupled_min_lr=2e-6
+ )
+ width_mult = 4.0
+
+ standard_overrides = get_standard_config_overrides(optimizer_config)
+ mup_overrides = get_mup_config_overrides(optimizer_config, width_mult)
+ combined_overrides = {**standard_overrides, **mup_overrides}
+
+ hidden_param = torch.nn.Parameter(torch.zeros(10, 10))
+ bias_param = torch.nn.Parameter(torch.zeros(10))
+ embedding_param = torch.nn.Parameter(torch.zeros(10, 10))
+ embedding_param.is_embedding_parameter = True
+ embedding_param.is_embedding_or_output_parameter = True
+ output_param = torch.nn.Parameter(torch.zeros(10, 10))
+ output_param.is_embedding_or_output_parameter = True
+ output_param.is_embedding_parameter = True
+ shared_output_param = torch.nn.Parameter(torch.zeros(10, 10))
+ shared_output_param.is_embedding_or_output_parameter = True
+ shared_output_param.shared_embedding = True
+
+ hidden_matches = [
+ override
+ for param_key, override in combined_overrides.items()
+ if param_key.matches(hidden_param, 'decoder.layer.0.weight')
+ ]
+ bias_matches = [
+ override
+ for param_key, override in combined_overrides.items()
+ if param_key.matches(bias_param, 'decoder.layer.0.bias')
+ ]
+ embedding_matches = [
+ override
+ for param_key, override in combined_overrides.items()
+ if param_key.matches(embedding_param, 'embedding.word_embeddings.weight')
+ ]
+ output_matches = [
+ override
+ for param_key, override in combined_overrides.items()
+ if param_key.matches(output_param, 'output_layer.weight')
+ ]
+ shared_output_matches = [
+ override
+ for param_key, override in combined_overrides.items()
+ if param_key.matches(shared_output_param, 'output_layer.weight')
+ ]
+
+ hidden_override = combine_param_group_overrides(hidden_matches)
+ bias_override = combine_param_group_overrides(bias_matches)
+ embedding_override = combine_param_group_overrides(embedding_matches)
+ output_override = combine_param_group_overrides(output_matches)
+ shared_output_override = combine_param_group_overrides(shared_output_matches)
+
+ # Hidden params keep MuP scaling.
+ assert hidden_override['max_lr'] == pytest.approx(1e-3 / width_mult)
+ assert hidden_override['min_lr'] == pytest.approx(1e-5 / width_mult)
+ assert hidden_override['eps'] == pytest.approx(optimizer_config.adam_eps / width_mult)
+
+ # Biases are vector-like; MuP should not override LR/eps.
+ assert 'max_lr' not in bias_override
+ assert 'min_lr' not in bias_override
+ assert 'eps' not in bias_override
+
+ # Embeddings keep decoupled LR and unscaled eps.
+ assert embedding_override['max_lr'] == pytest.approx(2e-4)
+ assert embedding_override['min_lr'] == pytest.approx(2e-6)
+ assert 'eps' not in embedding_override
+
+ # Output params keep decoupled LR and unscaled eps (Table 8 symmetry).
+ assert output_override['max_lr'] == pytest.approx(2e-4)
+ assert output_override['min_lr'] == pytest.approx(2e-6)
+ assert 'eps' not in output_override
+
+ # Shared embedding output copies stay on embedding-class (unscaled) eps.
+ assert shared_output_override['max_lr'] == pytest.approx(2e-4)
+ assert shared_output_override['min_lr'] == pytest.approx(2e-6)
+ assert 'eps' not in shared_output_override
+
+
+class TestMuPConfigIntegration:
+ """Integration tests for MuP config with init methods."""
+
+ def test_mup_output_layer_init(self):
+ """Output layer init should also scale with MuP."""
+ config = TransformerConfig(
+ hidden_size=1024,
+ num_layers=8,
+ num_attention_heads=16,
+ use_mup=True,
+ mup_base_hidden_size=256,
+ )
+
+ tensor = torch.empty(1000, 1000)
+ config.output_layer_init_method(tensor)
+
+ # Expected: sigma / (sqrt(2*L) * sqrt(m)) = 0.02 / (sqrt(16) * sqrt(4))
+ expected_std = config.init_method_std / (
+ math.sqrt(2 * config.num_layers) * math.sqrt(config.mup_width_mult)
+ )
+ actual_std = tensor.std().item()
+
+ assert abs(actual_std - expected_std) < expected_std * 0.15
+
+
+class TestMuPOptimizerTypeHandling:
+ """Tests for MuP optimizer-specific override behavior."""
+
+ def test_sgd_scales_vector_like_lr_only(self):
+ """SGD scales vector-like params by width_mult; hidden params keep base LR."""
+ optimizer_config = OptimizerConfig(lr=1e-3, min_lr=1e-5)
+ width_mult = 4.0
+
+ for sgd_variant in ['sgd', 'SGD', 'Sgd']:
+ overrides = get_mup_config_overrides(
+ optimizer_config, width_mult, optimizer_type=sgd_variant
+ )
+ assert len(overrides) == 1
+
+ param_key, override = next(iter(overrides.items()))
+ assert override['max_lr'] == pytest.approx(1e-3 * width_mult)
+ assert override['min_lr'] == pytest.approx(1e-5 * width_mult)
+ assert 'eps' not in override
+
+ hidden_param = torch.nn.Parameter(torch.zeros(10, 10))
+ bias_param = torch.nn.Parameter(torch.zeros(10))
+ embedding_param = torch.nn.Parameter(torch.zeros(10, 10))
+ embedding_param.is_embedding_parameter = True
+ output_param = torch.nn.Parameter(torch.zeros(10, 10))
+ output_param.is_embedding_parameter = True
+
+ assert param_key.matches(hidden_param, 'decoder.layer.0.weight') is False
+ assert param_key.matches(bias_param, 'decoder.layer.0.bias') is True
+ assert param_key.matches(embedding_param, 'embedding.word_embeddings.weight') is True
+ assert param_key.matches(output_param, 'output_layer.weight') is True
+
+ def test_sgd_with_decoupled_lr_preserves_embedding_output_precedence(self):
+ """With decoupled_lr, embedding/output keep decoupled LR under SGD MuP."""
+ optimizer_config = OptimizerConfig(
+ lr=1e-3, min_lr=1e-5, decoupled_lr=2e-4, decoupled_min_lr=2e-6
+ )
+ width_mult = 4.0
+
+ standard_overrides = get_standard_config_overrides(optimizer_config)
+ mup_overrides = get_mup_config_overrides(optimizer_config, width_mult, optimizer_type='sgd')
+ combined_overrides = {**standard_overrides, **mup_overrides}
+
+ hidden_param = torch.nn.Parameter(torch.zeros(10, 10))
+ bias_param = torch.nn.Parameter(torch.zeros(10))
+ embedding_param = torch.nn.Parameter(torch.zeros(10, 10))
+ embedding_param.is_embedding_parameter = True
+ embedding_param.is_embedding_or_output_parameter = True
+ output_param = torch.nn.Parameter(torch.zeros(10, 10))
+ output_param.is_embedding_parameter = True
+ output_param.is_embedding_or_output_parameter = True
+ shared_output_param = torch.nn.Parameter(torch.zeros(10, 10))
+ shared_output_param.shared_embedding = True
+ shared_output_param.is_embedding_or_output_parameter = True
+
+ hidden_matches = [
+ override
+ for param_key, override in combined_overrides.items()
+ if param_key.matches(hidden_param, 'decoder.layer.0.weight')
+ ]
+ bias_matches = [
+ override
+ for param_key, override in combined_overrides.items()
+ if param_key.matches(bias_param, 'decoder.layer.0.bias')
+ ]
+ embedding_matches = [
+ override
+ for param_key, override in combined_overrides.items()
+ if param_key.matches(embedding_param, 'embedding.word_embeddings.weight')
+ ]
+ output_matches = [
+ override
+ for param_key, override in combined_overrides.items()
+ if param_key.matches(output_param, 'output_layer.weight')
+ ]
+ shared_output_matches = [
+ override
+ for param_key, override in combined_overrides.items()
+ if param_key.matches(shared_output_param, 'output_layer.weight')
+ ]
+
+ hidden_override = combine_param_group_overrides(hidden_matches)
+ bias_override = combine_param_group_overrides(bias_matches)
+ embedding_override = combine_param_group_overrides(embedding_matches)
+ output_override = combine_param_group_overrides(output_matches)
+ shared_output_override = combine_param_group_overrides(shared_output_matches)
+
+ # Hidden params keep base LR under SGD in current uniform-width setup.
+ assert 'max_lr' not in hidden_override
+ assert 'min_lr' not in hidden_override
+ assert 'eps' not in hidden_override
+
+ # Biases are vector-like and scale up by width_mult.
+ assert bias_override['max_lr'] == pytest.approx(1e-3 * width_mult)
+ assert bias_override['min_lr'] == pytest.approx(1e-5 * width_mult)
+ assert 'eps' not in bias_override
+
+ # Embedding/output params keep explicit decoupled LR precedence.
+ assert embedding_override['max_lr'] == pytest.approx(2e-4)
+ assert embedding_override['min_lr'] == pytest.approx(2e-6)
+ assert output_override['max_lr'] == pytest.approx(2e-4)
+ assert output_override['min_lr'] == pytest.approx(2e-6)
+ assert shared_output_override['max_lr'] == pytest.approx(2e-4)
+ assert shared_output_override['min_lr'] == pytest.approx(2e-6)
+ assert 'eps' not in embedding_override
+ assert 'eps' not in output_override
+ assert 'eps' not in shared_output_override
+
+ def test_adam_scales_lr_by_default(self):
+ """Adam optimizer should scale LR and eps; default optimizer_type is adam."""
+ optimizer_config = OptimizerConfig(lr=1e-3, min_lr=1e-5)
+ width_mult = 4.0
+
+ # Explicit adam
+ overrides_explicit = get_mup_config_overrides(
+ optimizer_config, width_mult, optimizer_type='adam'
+ )
+ assert len(overrides_explicit) == 1
+
+ # Default (should behave like adam for backward compat)
+ overrides_default = get_mup_config_overrides(optimizer_config, width_mult)
+ assert len(overrides_default) == 1
+
+ for param_key, override in overrides_explicit.items():
+ expected_max_lr = 1e-3 / width_mult # 2.5e-4
+ expected_eps = optimizer_config.adam_eps / width_mult
+ assert abs(override['max_lr'] - expected_max_lr) < 1e-10
+ assert abs(override['eps'] - expected_eps) < 1e-15
+
+ def test_non_adam_does_not_set_eps_override(self):
+ """Non-Adam optimizers should not receive MuP epsilon overrides."""
+ optimizer_config = OptimizerConfig(lr=1e-3, min_lr=1e-5)
+ width_mult = 4.0
+
+ for non_adam_optimizer in ['muon', 'dist_muon']:
+ overrides = get_mup_config_overrides(
+ optimizer_config, width_mult, optimizer_type=non_adam_optimizer
+ )
+ assert len(overrides) == 1
+ for _, override in overrides.items():
+ assert override['max_lr'] == pytest.approx(1e-3 / width_mult)
+ assert override['min_lr'] == pytest.approx(1e-5 / width_mult)
+ assert 'eps' not in override
+
+ @pytest.mark.parametrize('optimizer_type', ['muon', 'dist_muon'])
+ def test_muon_excludes_muon_managed_matrices_from_mup_overrides(self, optimizer_type):
+ """Muon-managed 2D params should use Muon scaling only, not MuP LR overrides."""
+ optimizer_config = OptimizerConfig(lr=1e-3, min_lr=1e-5, muon_scale_mode='unit_rms_norm')
+ width_mult = 4.0
+
+ overrides = get_mup_config_overrides(
+ optimizer_config, width_mult, optimizer_type=optimizer_type
+ )
+
+ muon_managed_param = torch.nn.Parameter(torch.zeros(10, 10))
+ muon_managed_param.is_embedding_or_output_parameter = False
+ output_param = torch.nn.Parameter(torch.zeros(10, 10))
+ output_param.is_embedding_or_output_parameter = True
+ bias_param = torch.nn.Parameter(torch.zeros(10))
+
+ muon_managed_matches = [
+ override
+ for param_key, override in overrides.items()
+ if param_key.matches(
+ muon_managed_param, 'decoder.layers.0.self_attention.linear_proj.weight'
+ )
+ ]
+ output_matches = [
+ override
+ for param_key, override in overrides.items()
+ if param_key.matches(output_param, 'output_layer.weight')
+ ]
+ bias_matches = [
+ override
+ for param_key, override in overrides.items()
+ if param_key.matches(bias_param, 'decoder.layers.0.self_attention.linear_proj.bias')
+ ]
+
+ muon_managed_override = combine_param_group_overrides(muon_managed_matches)
+ output_override = combine_param_group_overrides(output_matches)
+ bias_override = combine_param_group_overrides(bias_matches)
+
+ # Muon-managed matrix params are excluded from Adam-style MuP LR overrides.
+ assert 'max_lr' not in muon_managed_override
+ assert 'min_lr' not in muon_managed_override
+ assert 'eps' not in muon_managed_override
+
+ # Output params remain in the MuP override path (handled by chained Adam optimizer).
+ assert output_override['max_lr'] == pytest.approx(1e-3 / width_mult)
+ assert output_override['min_lr'] == pytest.approx(1e-5 / width_mult)
+ assert 'eps' not in output_override
+
+ # Vector-like params stay unscaled.
+ assert 'max_lr' not in bias_override
+ assert 'min_lr' not in bias_override
+ assert 'eps' not in bias_override
+
+ @pytest.mark.parametrize('optimizer_type', ['muon', 'dist_muon'])
+ def test_muon_warns_for_spectral_scale_mode(self, optimizer_type):
+ """Muon+MuP should warn when scale mode is spectral."""
+ optimizer_config = OptimizerConfig(lr=1e-3, min_lr=1e-5, muon_scale_mode='spectral')
+ width_mult = 4.0
+
+ with patch('megatron.core.optimizer.log_single_rank') as mock_warn:
+ overrides = get_mup_config_overrides(
+ optimizer_config, width_mult, optimizer_type=optimizer_type
+ )
+
+ assert len(overrides) == 1
+ mock_warn.assert_called_once()
+ _, level, message = mock_warn.call_args[0]
+ assert level == logging.WARNING
+ assert "Both MuP and muon_scale_mode=spectral are enabled." in message
+ assert "--muon-scale-mode unit_rms_norm" in message
+
+ @pytest.mark.parametrize('optimizer_type', ['muon', 'dist_muon'])
+ def test_muon_unit_rms_norm_mode_has_no_warning(self, optimizer_type):
+ """Muon+MuP should not warn when scale mode is unit_rms_norm."""
+ optimizer_config = OptimizerConfig(lr=1e-3, min_lr=1e-5, muon_scale_mode='unit_rms_norm')
+ width_mult = 4.0
+
+ with patch('megatron.core.optimizer.log_single_rank') as mock_warn:
+ overrides = get_mup_config_overrides(
+ optimizer_config, width_mult, optimizer_type=optimizer_type
+ )
+
+ assert len(overrides) == 1
+ mock_warn.assert_not_called()
+
+ @pytest.mark.parametrize('optimizer_type', ['muon', 'dist_muon'])
+ def test_muon_warns_for_spectral_mode_at_unity_width_mult(self, optimizer_type):
+ """Muon+MuP warning should still fire when width_mult==1.0."""
+ optimizer_config = OptimizerConfig(lr=1e-3, min_lr=1e-5, muon_scale_mode='spectral')
+ width_mult = 1.0
+
+ with patch('megatron.core.optimizer.log_single_rank') as mock_warn:
+ overrides = get_mup_config_overrides(
+ optimizer_config, width_mult, optimizer_type=optimizer_type
+ )
+
+ assert len(overrides) == 0
+ mock_warn.assert_called_once()
+ _, level, message = mock_warn.call_args[0]
+ assert level == logging.WARNING
+ assert "Both MuP and muon_scale_mode=spectral are enabled." in message
+
+
+class TestMuPMTPLossScaling:
+ """Tests for MuP scaling integration with MTP loss processing."""
+
+ def test_process_mtp_loss_applies_scale_hook(self):
+ config = TransformerConfig(
+ hidden_size=8, num_layers=2, num_attention_heads=2, mtp_num_layers=1
+ )
+ hidden_states = torch.ones(2, 1, 4)
+ labels = torch.ones(1, 4, dtype=torch.long)
+ loss_mask = torch.ones_like(labels, dtype=torch.float32)
+ observed_logits_mean = {'value': None}
+
+ def output_layer(hidden, weight=None, runtime_gather_output=None):
+ return hidden.clone(), None
+
+ def scale_logits_fn(logits):
+ return logits * 3.0
+
+ def compute_language_model_loss(mtp_labels, mtp_logits):
+ observed_logits_mean['value'] = mtp_logits.mean().item()
+ return torch.ones_like(mtp_labels, dtype=mtp_logits.dtype)
+
+ process_mtp_loss(
+ hidden_states=hidden_states,
+ labels=labels,
+ loss_mask=loss_mask,
+ output_layer=output_layer,
+ output_weight=None,
+ runtime_gather_output=None,
+ is_training=False,
+ compute_language_model_loss=compute_language_model_loss,
+ config=config,
+ cp_group=None,
+ packed_seq_params=None,
+ scale_logits_fn=scale_logits_fn,
+ )
+
+ assert observed_logits_mean['value'] == pytest.approx(3.0)
diff --git a/tests/unit_tests/transformer/test_spec_customization.py b/tests/unit_tests/transformer/test_spec_customization.py
index d6b2a2eb623..84fb2ee706c 100755
--- a/tests/unit_tests/transformer/test_spec_customization.py
+++ b/tests/unit_tests/transformer/test_spec_customization.py
@@ -1,9 +1,8 @@
# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
import sys
-from dataclasses import dataclass, fields
+from dataclasses import fields
-import pytest
import torch
import transformer_engine as te
@@ -14,12 +13,11 @@
TERowParallelLinear,
)
from megatron.core.fusions.fused_bias_dropout import get_bias_dropout_add
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_spec
+from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_local_submodules
from megatron.core.parallel_state import get_context_parallel_group, get_tensor_model_parallel_group
from megatron.core.process_groups_config import ProcessGroupCollection
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer.attention import SelfAttention, SelfAttentionSubmodules
-from megatron.core.transformer.dot_product_attention import DotProductAttention
from megatron.core.transformer.enums import AttnMaskType
from megatron.core.transformer.identity_op import IdentityFuncOp, IdentityOp
from megatron.core.transformer.spec_utils import ModuleSpec, build_module, import_module
@@ -203,20 +201,18 @@ def test_transformer_block_custom(self):
transformer_config = TransformerConfig(
num_layers=2, hidden_size=12, num_attention_heads=4, use_cpu_initialization=True
)
- layer_local_spec = get_gpt_layer_local_spec()
+ submodules = get_gpt_layer_local_submodules()
# The following way can be used to pass a different `TransformerLayer`
# and internally the `TransformerBlock` would fan out the single
# `ModuleSpec` layer spec provided to all the layers of the block.
- layer_spec1 = ModuleSpec(module=TransformerLayer, submodules=layer_local_spec.submodules)
+ layer_spec1 = ModuleSpec(module=TransformerLayer, submodules=submodules)
model_parallel_cuda_manual_seed(123)
torch.manual_seed(0)
parallel_transformer_block1 = TransformerBlock(transformer_config, layer_spec1)
layer_spec2 = TransformerBlockSubmodules(
- layer_specs=[
- ModuleSpec(module=TransformerLayer, submodules=layer_local_spec.submodules)
- ]
+ layer_specs=[ModuleSpec(module=TransformerLayer, submodules=submodules)]
* transformer_config.num_layers,
layer_norm=TENorm,
)
@@ -252,12 +248,10 @@ def test_transformer_block_custom(self):
def test_l2_qk_norm(self):
"""Test L2 normalization for QK vectors using local spec."""
- layer_spec = get_gpt_layer_local_spec(qk_l2_norm=True)
+ submodules = get_gpt_layer_local_submodules(qk_l2_norm=True)
# Build the self-attention module from the spec
- self_attention = build_module(
- layer_spec.submodules.self_attention, config=self.config, layer_number=1
- )
+ self_attention = build_module(submodules.self_attention, config=self.config, layer_number=1)
assert isinstance(self_attention, SelfAttention)
# Verify that q_layernorm and k_layernorm are L2Norm instances
diff --git a/tests/unit_tests/transformer/test_spec_utils.py b/tests/unit_tests/transformer/test_spec_utils.py
new file mode 100644
index 00000000000..e464b09380e
--- /dev/null
+++ b/tests/unit_tests/transformer/test_spec_utils.py
@@ -0,0 +1,129 @@
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+import dataclasses
+from functools import partial
+from typing import Protocol
+
+import pytest
+
+from megatron.core.transformer.spec_utils import ModuleSpec, build_module, get_submodules
+
+
+def dummy_method(x: int, y: str) -> dict:
+ return {"x": x, "y": y}
+
+
+class ExampleA:
+ def __init__(self, x: int, y: str):
+ self.x = x
+ self.y = y
+
+
+class TestBuildModule:
+ """Unit tests for `build_module` function."""
+
+ def test_build_module_from_function(self):
+ """Test building module from a FunctionType."""
+
+ assert build_module(dummy_method) == dummy_method
+ assert build_module(ModuleSpec(module=dummy_method)) == dummy_method
+
+ def test_build_module_from_type(self):
+ """Test building module from a type."""
+
+ class Empty:
+ pass
+
+ assert isinstance(build_module(Empty), Empty)
+ assert isinstance(build_module(ModuleSpec(module=Empty)), Empty)
+
+ def test_build_module_by_import(self):
+ """Test building module by importing from a string."""
+ assert (
+ type(
+ build_module(
+ ModuleSpec(module=('megatron.core.transformer.identity_op', 'IdentityOp'))
+ )
+ ).__name__
+ == 'IdentityOp'
+ )
+
+ def test_build_module_with_params(self):
+ """Test building module with parameters."""
+
+ build_time = build_module(ExampleA, 1, 'abc')
+ assert isinstance(build_time, ExampleA)
+ assert build_time.x == 1
+ assert build_time.y == 'abc'
+
+ by_spec = build_module(ModuleSpec(module=ExampleA, params={'x': 2, 'y': 'def'}))
+ assert isinstance(by_spec, ExampleA)
+ assert by_spec.x == 2
+ assert by_spec.y == 'def'
+
+ mixed = build_module(ModuleSpec(module=ExampleA, params={'y': 'ghi'}), 3)
+ assert isinstance(mixed, ExampleA)
+ assert mixed.x == 3
+ assert mixed.y == 'ghi'
+
+ def test_build_module_by_call(self):
+ """Test building module by calling a ModuleSpec instance."""
+
+ by_spec = ModuleSpec(module=ExampleA, params={'x': 2, 'y': 'def'})()
+ assert isinstance(by_spec, ExampleA)
+ assert by_spec.x == 2
+ assert by_spec.y == 'def'
+
+ mixed = ModuleSpec(module=ExampleA, params={'y': 'ghi'})(3)
+ assert isinstance(mixed, ExampleA)
+ assert mixed.x == 3
+ assert mixed.y == 'ghi'
+
+
+class OtherChild:
+ def __init__(self, x: int):
+ self.x = x
+
+
+class ABuilder(Protocol):
+ def __call__(self, x: int) -> ExampleA: ...
+
+
+@dataclasses.dataclass
+class BSubmodules:
+ x: ModuleSpec | type
+ y: ABuilder
+
+
+class ExampleB:
+ def __init__(self, submodules: BSubmodules, z: int):
+ self.x = build_module(submodules.x, x=10)
+ self.y = submodules.y(x=10)
+ self.z = z
+
+
+class TestGetSubmodules:
+ """Test that the getter utilities work as expected."""
+
+ def test_get_submodules_missing(self):
+ """Test getting submodules from a spec without one."""
+ with pytest.raises(ValueError):
+ get_submodules(dummy_method)
+ with pytest.raises(ValueError):
+ get_submodules(ExampleA)
+ with pytest.raises(KeyError):
+ get_submodules(partial(ExampleA, x=1, y='test'))
+
+ def test_get_submodules_module_spec(self):
+ """Test getting submodules from a spec."""
+ assert get_submodules(ModuleSpec(module=dummy_method)) is None
+ assert get_submodules(ModuleSpec(module=ExampleA)) is None
+ submodules = BSubmodules(x=OtherChild, y=partial(ExampleA, y='test'))
+ assert (
+ get_submodules(ModuleSpec(module=ExampleB, submodules=submodules, params={'z': 123}))
+ == submodules
+ )
+
+ def test_get_submodules_partial(self):
+ """Test getting submodules from a use of partial."""
+ submodules = BSubmodules(x=OtherChild, y=partial(ExampleA, y='test'))
+ assert get_submodules(partial(ExampleB, submodules=submodules, z=123)) == submodules
diff --git a/tests/unit_tests/transformer/test_submodule_callables.py b/tests/unit_tests/transformer/test_submodule_callables.py
index 73059495c06..03e2d751a52 100644
--- a/tests/unit_tests/transformer/test_submodule_callables.py
+++ b/tests/unit_tests/transformer/test_submodule_callables.py
@@ -3,7 +3,9 @@
import torch
from megatron.core.models.gpt.fine_grained_callables import build_layer_callables
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
+from megatron.core.models.gpt.gpt_layer_specs import (
+ get_gpt_layer_with_transformer_engine_submodules,
+)
from megatron.core.transformer.transformer_layer import TransformerLayer
from megatron.core.utils import is_te_min_version
from tests.unit_tests.a2a_overlap.utils import (
@@ -140,13 +142,13 @@ def test_1f1b_overlap(self, dispatcher_type, grouped_gemm, permute_fusion):
config = get_test_config(extra_kwargs=extra_kwargs, moe_grouped_gemm=grouped_gemm)
microbatches = 4
with deterministic_mode():
- transformer_layer_spec = get_gpt_layer_with_transformer_engine_spec(
+ transformer_layer_submodules = get_gpt_layer_with_transformer_engine_submodules(
num_experts=8,
moe_grouped_gemm=grouped_gemm,
qk_layernorm=True,
multi_latent_attention=True,
)
- model = TransformerLayer(config, transformer_layer_spec.submodules)
+ model = TransformerLayer(config, transformer_layer_submodules)
params = reset_model(model)
input_tensors = [build_data() for _ in range(microbatches)]
diff --git a/tests/unit_tests/transformer/test_thd_correctness.py b/tests/unit_tests/transformer/test_thd_correctness.py
new file mode 100644
index 00000000000..ccf70b8a885
--- /dev/null
+++ b/tests/unit_tests/transformer/test_thd_correctness.py
@@ -0,0 +1,649 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+"""
+Compare THD format against SBHD format.
+
+Test Strategy
+-------------
+1. Generate full (unsharded) data with deterministic seed on each rank.
+2. Shard inputs for both SBHD and THD formats (zigzag CP, contiguous SP).
+3. Forward pass through the same TransformerLayer.
+4. Gather outputs back to full size (with gradient support).
+5. Backward pass with format-specific grad_output handling.
+6. Compare outputs and gradients with bitwise or similarity checks.
+
+Check Levels
+------------
+- bitwise_all: B=1, forward + backward bitwise (MockCoreAttention)
+- bitwise_fwd: B>1, forward bitwise, backward similarity (MockCoreAttention,
+ THD padded to max_len so total tokens match SBHD)
+- similarity: All parallelism configs, real TE attention, similarity checks
+"""
+
+import os
+from dataclasses import dataclass
+from typing import List
+
+import pytest
+import torch
+import torch.distributed as dist
+import torch.nn as nn
+
+from megatron.core import parallel_state
+from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
+from megatron.core.packed_seq_params import PackedSeqParams
+from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
+from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.transformer.transformer_layer import TransformerLayer
+from tests.unit_tests.test_utilities import Utils
+
+# =============================================================================
+# Constants
+# =============================================================================
+
+SIMILARITY_THRESHOLD = 0.999
+
+
+# =============================================================================
+# Test Cases
+# =============================================================================
+
+
+@dataclass
+class TestCase:
+ """Test case specification.
+
+ check_level controls comparison strictness and attention implementation:
+ "bitwise_all" - MockCoreAttention, forward + backward bitwise (B=1)
+ "bitwise_fwd" - MockCoreAttention, forward bitwise, backward similarity
+ (B>1, THD padded to max_len to match SBHD total tokens)
+ "similarity" - Real TE attention, forward + backward similarity
+ """
+
+ name: str
+ hidden_size: int
+ num_heads: int
+ num_kv_heads: int
+ ffn_hidden_size: int
+ seqlens: List[int]
+ tp_size: int = 1
+ cp_size: int = 1
+ sp_enabled: bool = False
+ check_level: str = "similarity"
+
+ @property
+ def use_mock_attention(self) -> bool:
+ return self.check_level in ("bitwise_all", "bitwise_fwd")
+
+ @property
+ def forward_bitwise(self) -> bool:
+ return self.check_level in ("bitwise_all", "bitwise_fwd")
+
+ @property
+ def backward_bitwise(self) -> bool:
+ return self.check_level == "bitwise_all"
+
+ @property
+ def pad_thd_to_max(self) -> bool:
+ """Pad each THD sequence to max_len so total tokens match SBHD."""
+ return self.check_level == "bitwise_fwd"
+
+
+# fmt: off
+TEST_CASES = [
+ # -------------------------------------------------------------------------
+ # B=1: forward + backward bitwise (MockCoreAttention)
+ # -------------------------------------------------------------------------
+ # name H heads kv_h ffn seqlens tp cp sp check_level
+ TestCase("b1_seq3891_gqa", 1024, 16, 4, 4096, [3891], 1, 1, False, "bitwise_all"),
+ TestCase("b1_seq16k_mha", 256, 4, 4, 1024, [16383], 1, 1, False, "bitwise_all"),
+
+ # -------------------------------------------------------------------------
+ # B>1 single GPU: forward bitwise, backward similarity (MockCoreAttention)
+ # THD is padded to max_len per sequence so TE GEMM sees the same M value
+ # -------------------------------------------------------------------------
+ TestCase("varlen_mixed", 1024, 16, 16, 4096, [1987, 523, 271, 1009], 1, 1, False, "bitwise_fwd"),
+ TestCase("short_seqs", 1024, 16, 16, 4096, [17, 31, 11], 1, 1, False, "bitwise_fwd"),
+ TestCase("b2_long_8k", 256, 4, 4, 1024, [8191, 8192], 1, 1, False, "bitwise_fwd"),
+
+ # -------------------------------------------------------------------------
+ # TP/CP/SP: similarity checks (TE Attention)
+ # -------------------------------------------------------------------------
+ TestCase("tp2_cp4_sp", 4096, 64, 4, 12288, [2039, 1013, 509], 2, 4, True, "similarity"),
+ TestCase("tp2_cp2_sp_longseq", 4096, 32, 8, 14336, [65536, 8191, 4096], 2, 2, True, "similarity"),
+
+ # -------------------------------------------------------------------------
+ # Edge cases
+ # -------------------------------------------------------------------------
+ TestCase("short_seqs_parallel", 1024, 16, 4, 4096, [17, 31, 11], 2, 2, True, "similarity"),
+ TestCase("extreme_mixed", 4096, 32, 8, 14336, [4093, 127, 257], 2, 2, True, "similarity"),
+ TestCase("long_short_mix", 4096, 32, 8, 14336, [65535, 512, 1024], 2, 2, True, "similarity"),
+]
+# fmt: on
+
+
+# =============================================================================
+# Padding Helpers
+# =============================================================================
+
+
+def _round_up(value: int, divisor: int) -> int:
+ return value if divisor <= 1 else (value + divisor - 1) // divisor * divisor
+
+
+def compute_sbhd_padded_max_len(
+ seqlens: List[int], cp_size: int, tp_size: int, sp_enabled: bool
+) -> int:
+ """Padded max_len for SBHD.
+
+ Must be divisible by:
+ - cp_size * 2 for zigzag CP sharding (if cp_size > 1)
+ - tp_size for SP sharding along sequence dim (if sp_enabled)
+ """
+ divisor = 1
+ if cp_size > 1:
+ divisor *= cp_size * 2
+ if sp_enabled:
+ divisor *= tp_size
+ return _round_up(max(seqlens), divisor)
+
+
+def compute_thd_padded_seqlens(
+ seqlens: List[int], cp_size: int, tp_size: int, sp_enabled: bool, pad_to_max: bool = False
+) -> List[int]:
+ """Padded per-sequence lengths for THD.
+
+ When pad_to_max=True, each sequence is padded to max(seqlens) so that
+ total THD tokens = max_len * B, matching SBHD. This ensures TE GEMM
+ kernels see identical M dimensions for bitwise comparison.
+ """
+ cp_divisor = 2 * cp_size if cp_size > 1 else 1
+ if pad_to_max:
+ max_len = _round_up(max(seqlens), cp_divisor)
+ padded = [max_len] * len(seqlens)
+ else:
+ padded = [_round_up(sl, cp_divisor) for sl in seqlens]
+ if sp_enabled:
+ remainder = sum(padded) % tp_size
+ if remainder > 0:
+ padded[-1] += tp_size - remainder
+ return padded
+
+
+# =============================================================================
+# PackedSeqParams Helper
+# =============================================================================
+
+
+def make_packed_seq_params(
+ seqlens: List[int],
+ cp_size: int = 1,
+ tp_size: int = 1,
+ sp_enabled: bool = False,
+ pad_to_max: bool = False,
+) -> PackedSeqParams:
+ """Create PackedSeqParams with cu_seqlens and cu_seqlens_padded."""
+
+ def to_cu_seqlens(lens):
+ cu = torch.zeros(len(lens) + 1, dtype=torch.int32)
+ for i, l in enumerate(lens):
+ cu[i + 1] = cu[i] + l
+ return cu.cuda()
+
+ padded = compute_thd_padded_seqlens(seqlens, cp_size, tp_size, sp_enabled, pad_to_max)
+ return PackedSeqParams(
+ cu_seqlens_q=to_cu_seqlens(seqlens),
+ cu_seqlens_kv=to_cu_seqlens(seqlens),
+ cu_seqlens_q_padded=to_cu_seqlens(padded),
+ cu_seqlens_kv_padded=to_cu_seqlens(padded),
+ max_seqlen_q=max(padded),
+ max_seqlen_kv=max(padded),
+ qkv_format='thd',
+ )
+
+
+# =============================================================================
+# Mock Core Attention (for bitwise tests)
+# =============================================================================
+
+
+class MockCoreAttention(nn.Module):
+ """Per-sequence unfused causal attention for bitwise comparison."""
+
+ def __init__(
+ self,
+ config,
+ layer_number,
+ attn_mask_type,
+ attention_type,
+ attention_dropout=None,
+ softmax_scale=None,
+ cp_comm_type=None,
+ pg_collection=None,
+ ):
+ super().__init__()
+ self.num_q_heads = config.num_attention_heads
+ self.num_kv_heads = config.num_query_groups
+ self.head_dim = config.hidden_size // config.num_attention_heads
+ self.hidden_size = config.hidden_size
+ self.scale = 1.0 / (self.head_dim**0.5)
+ self.num_rep = self.num_q_heads // self.num_kv_heads
+
+ def _repeat_kv(self, x):
+ """Repeat KV heads for GQA. [S, Hkv, D] -> [S, Hq, D]."""
+ if self.num_rep == 1:
+ return x
+ S, Hkv, D = x.shape
+ return x.unsqueeze(2).expand(S, Hkv, self.num_rep, D).reshape(S, self.num_q_heads, D)
+
+ def _attention_single_seq(self, q, k, v):
+ """Causal attention for one sequence."""
+ S = q.shape[0]
+ k, v = self._repeat_kv(k), self._repeat_kv(v)
+ q, k, v = (x.transpose(0, 1).contiguous() for x in (q, k, v))
+ q32, k32, v32 = q.float(), k.float(), v.float()
+ scores = torch.matmul(q32, k32.transpose(-2, -1)) * self.scale
+ mask = torch.triu(torch.ones(S, S, dtype=torch.bool, device=q.device), diagonal=1)
+ scores.masked_fill_(mask, float('-inf'))
+ attn = torch.softmax(scores, dim=-1)
+ out = torch.matmul(attn, v32)
+ return out.transpose(0, 1).to(q.dtype).contiguous()
+
+ def forward(
+ self,
+ query,
+ key,
+ value,
+ attention_mask=None,
+ attn_mask_type=None,
+ attention_bias=None,
+ packed_seq_params=None,
+ ):
+ if packed_seq_params is not None:
+ # THD: [T, 1, H, D] -> [T, H, D]
+ q = query.squeeze(1) if query.dim() == 4 else query
+ k = key.squeeze(1) if key.dim() == 4 else key
+ v = value.squeeze(1) if value.dim() == 4 else value
+
+ cu_valid = packed_seq_params.cu_seqlens_q.cpu().tolist()
+ cu_padded = packed_seq_params.cu_seqlens_q_padded.cpu().tolist()
+ num_seqs = len(cu_valid) - 1
+
+ outputs = []
+ for i in range(num_seqs):
+ out_seq = self._attention_single_seq(
+ q[cu_padded[i] : cu_padded[i + 1]],
+ k[cu_padded[i] : cu_padded[i + 1]],
+ v[cu_padded[i] : cu_padded[i + 1]],
+ )
+ outputs.append(out_seq)
+
+ return torch.cat(outputs, dim=0) # [T_padded, Hq, D]
+
+ else:
+ # SBHD: [S, B, H, D]
+ S, B = query.shape[:2]
+ outputs = [
+ self._attention_single_seq(query[:, b], key[:, b], value[:, b]) for b in range(B)
+ ]
+ return torch.stack(outputs, dim=1).reshape(S, B, self.hidden_size)
+
+
+# =============================================================================
+# Layer Builder
+# =============================================================================
+
+
+def build_gpt_layer(
+ hidden_size: int,
+ num_heads: int,
+ num_kv_heads: int,
+ ffn_hidden_size: int,
+ tp_size: int = 1,
+ cp_size: int = 1,
+ sp_enabled: bool = False,
+ use_mock_attention: bool = False,
+ deterministic: bool = False,
+) -> TransformerLayer:
+ """Build GPT TransformerLayer, optionally with MockCoreAttention."""
+ config = TransformerConfig(
+ num_layers=1,
+ hidden_size=hidden_size,
+ ffn_hidden_size=ffn_hidden_size,
+ num_attention_heads=num_heads,
+ num_query_groups=num_kv_heads,
+ bf16=True,
+ params_dtype=torch.bfloat16,
+ pipeline_dtype=torch.bfloat16,
+ autocast_dtype=torch.bfloat16,
+ hidden_dropout=0.0,
+ attention_dropout=0.0,
+ tensor_model_parallel_size=tp_size,
+ context_parallel_size=cp_size,
+ sequence_parallel=sp_enabled,
+ cp_comm_type="p2p" if cp_size > 1 else None,
+ deterministic_mode=deterministic,
+ )
+ spec = get_gpt_layer_with_transformer_engine_spec()
+ if use_mock_attention:
+ spec.submodules.self_attention.submodules.core_attention = MockCoreAttention
+ layer = TransformerLayer(config, spec.submodules)
+ layer.cuda()
+ return layer
+
+
+# =============================================================================
+# Sharding: full -> local
+# =============================================================================
+
+
+def _zigzag_split(tensor, cp_rank, cp_size, dim=0):
+ """Split tensor along dim using zigzag pattern for CP.
+
+ For cp_size=2: rank0 gets chunks [0,3], rank1 gets chunks [1,2]
+ For cp_size=4: rank0 gets [0,7], rank1 gets [1,6], rank2 gets [2,5], rank3 gets [3,4]
+ """
+ if cp_size <= 1:
+ return tensor
+ chunk_size = tensor.shape[dim] // (2 * cp_size)
+ i0, i1 = cp_rank, 2 * cp_size - cp_rank - 1
+ chunk0 = tensor.narrow(dim, i0 * chunk_size, chunk_size)
+ chunk1 = tensor.narrow(dim, i1 * chunk_size, chunk_size)
+ return torch.cat([chunk0, chunk1], dim=dim)
+
+
+def shard_sbhd(tensor, cp_rank, cp_size, tp_rank, tp_size, sp_enabled):
+ """Shard SBHD tensor: zigzag CP, then contiguous SP."""
+ out = _zigzag_split(tensor, cp_rank, cp_size)
+ if sp_enabled:
+ seg = out.shape[0] // tp_size
+ out = out.narrow(0, tp_rank * seg, seg)
+ return out.contiguous()
+
+
+def shard_thd(
+ seq_data_list, seqlens, cp_rank, cp_size, tp_rank, tp_size, sp_enabled, H, pad_to_max=False
+):
+ """Shard per-sequence data into local THD [local_T, 1, H]."""
+ padded = compute_thd_padded_seqlens(seqlens, cp_size, tp_size, sp_enabled, pad_to_max)
+
+ chunks = []
+ for data, sl, psl in zip(seq_data_list, seqlens, padded):
+ if psl > sl:
+ data = torch.cat([data, torch.zeros(psl - sl, H, dtype=data.dtype, device=data.device)])
+ chunks.append(_zigzag_split(data, cp_rank, cp_size))
+
+ packed = torch.cat(chunks, dim=0)
+ if sp_enabled:
+ seg = packed.shape[0] // tp_size
+ packed = packed[tp_rank * seg : (tp_rank + 1) * seg]
+ return packed.unsqueeze(1).contiguous()
+
+
+# =============================================================================
+# Gathering: local -> full (with backward support)
+# =============================================================================
+
+
+def _zigzag_merge(chunks: List[torch.Tensor], cp_size: int) -> torch.Tensor:
+ """Reconstruct full sequence from per-rank zigzag chunks."""
+ half = chunks[0].shape[0] // 2
+ parts = [None] * (2 * cp_size)
+ for r in range(cp_size):
+ parts[r] = chunks[r][:half]
+ parts[2 * cp_size - r - 1] = chunks[r][half:]
+ return torch.cat(parts, dim=0)
+
+
+def _strip_thd_padding(tensor, seqlens, padded_seqlens):
+ """Remove per-sequence padding from THD tensor, keeping autograd."""
+ total_valid = sum(seqlens)
+ if tensor.shape[0] <= total_valid:
+ return tensor
+ offset, seqs = 0, []
+ for sl, psl in zip(seqlens, padded_seqlens):
+ seqs.append(tensor[offset : offset + sl])
+ offset += psl
+ return torch.cat(seqs, dim=0)
+
+
+class _GatherSBHD(torch.autograd.Function):
+ """Gather SBHD outputs from all ranks with gradient support."""
+
+ @staticmethod
+ def forward(ctx, local, cp_size, tp_size, sp_enabled):
+ ctx.cp_size, ctx.tp_size, ctx.sp_enabled = cp_size, tp_size, sp_enabled
+ ctx.cp_rank = parallel_state.get_context_parallel_rank() if cp_size > 1 else 0
+ ctx.tp_rank = parallel_state.get_tensor_model_parallel_rank()
+
+ out = local
+ if sp_enabled:
+ gathered = [torch.empty_like(out) for _ in range(tp_size)]
+ dist.all_gather(
+ gathered, out.contiguous(), group=parallel_state.get_tensor_model_parallel_group()
+ )
+ out = torch.cat(gathered, dim=0)
+ if cp_size > 1:
+ gathered = [torch.empty_like(out) for _ in range(cp_size)]
+ dist.all_gather(
+ gathered, out.contiguous(), group=parallel_state.get_context_parallel_group()
+ )
+ out = _zigzag_merge(gathered, cp_size)
+ return out
+
+ @staticmethod
+ def backward(ctx, grad):
+ out = grad
+ if ctx.cp_size > 1:
+ out = _zigzag_split(out, ctx.cp_rank, ctx.cp_size)
+ if ctx.sp_enabled:
+ seg = out.shape[0] // ctx.tp_size
+ out = out[ctx.tp_rank * seg : (ctx.tp_rank + 1) * seg]
+ return out.contiguous(), None, None, None
+
+
+class _GatherTHD(torch.autograd.Function):
+ """Gather THD outputs from all ranks with gradient support."""
+
+ @staticmethod
+ def forward(ctx, local, seqlens, cp_size, tp_size, sp_enabled, H, pad_to_max):
+ ctx.seqlens, ctx.cp_size, ctx.tp_size, ctx.sp_enabled, ctx.H = (
+ seqlens,
+ cp_size,
+ tp_size,
+ sp_enabled,
+ H,
+ )
+ ctx.cp_rank = parallel_state.get_context_parallel_rank() if cp_size > 1 else 0
+ ctx.tp_rank = parallel_state.get_tensor_model_parallel_rank()
+ ctx.padded = compute_thd_padded_seqlens(seqlens, cp_size, tp_size, sp_enabled, pad_to_max)
+
+ out = local
+ if sp_enabled:
+ gathered = [torch.empty_like(out) for _ in range(tp_size)]
+ dist.all_gather(
+ gathered, out.contiguous(), group=parallel_state.get_tensor_model_parallel_group()
+ )
+ out = torch.cat(gathered, dim=0)
+
+ if cp_size > 1:
+ cp_group = parallel_state.get_context_parallel_group()
+ local_lens = [p // cp_size for p in ctx.padded]
+ offset, seqs = 0, []
+ for i, ll in enumerate(local_lens):
+ chunk = out[offset : offset + ll]
+ gathered = [torch.empty_like(chunk) for _ in range(cp_size)]
+ dist.all_gather(gathered, chunk.contiguous(), group=cp_group)
+ seqs.append(_zigzag_merge(gathered, cp_size)[: seqlens[i]])
+ offset += ll
+ out = torch.cat(seqs, dim=0)
+ else:
+ out = _strip_thd_padding(out, seqlens, ctx.padded)
+ return out
+
+ @staticmethod
+ def backward(ctx, grad):
+ offset, chunks = 0, []
+ for sl, psl in zip(ctx.seqlens, ctx.padded):
+ g = grad[offset : offset + sl, 0, :]
+ if psl > sl:
+ g = torch.cat([g, torch.zeros(psl - sl, ctx.H, dtype=g.dtype, device=g.device)])
+ chunks.append(_zigzag_split(g, ctx.cp_rank, ctx.cp_size))
+ offset += sl
+
+ packed = torch.cat(chunks, dim=0)
+ if ctx.sp_enabled:
+ seg = packed.shape[0] // ctx.tp_size
+ packed = packed[ctx.tp_rank * seg : (ctx.tp_rank + 1) * seg]
+ return packed.unsqueeze(1).contiguous(), None, None, None, None, None, None
+
+
+def gather_sbhd(local, cp_size, tp_size, sp_enabled):
+ if cp_size == 1 and not sp_enabled:
+ return local
+ return _GatherSBHD.apply(local, cp_size, tp_size, sp_enabled)
+
+
+def gather_thd(local, seqlens, cp_size, tp_size, sp_enabled, H, pad_to_max=False):
+ return _GatherTHD.apply(local, seqlens, cp_size, tp_size, sp_enabled, H, pad_to_max)
+
+
+# =============================================================================
+# Comparison Helpers
+# =============================================================================
+
+
+def _cosine_sim(a, b):
+ return torch.nn.functional.cosine_similarity(
+ a.flatten().float().unsqueeze(0), b.flatten().float().unsqueeze(0)
+ ).item()
+
+
+def _tensor_sim(a, b):
+ a, b = a.double(), b.double()
+ denom = (a * a + b * b).sum()
+ return (2.0 * (a * b).sum() / denom).item() if denom else 1.0
+
+
+def assert_close(name, a, b, bitwise):
+ """Assert tensors match (bitwise or similarity)."""
+ if bitwise:
+ assert torch.equal(
+ a, b
+ ), f"{name}: NOT bitwise equal, max diff = {(a-b).abs().max().item()}"
+ else:
+ cs, ts = _cosine_sim(a, b), _tensor_sim(a, b)
+ assert cs > SIMILARITY_THRESHOLD, f"{name}: cosine sim = {cs:.6f} < {SIMILARITY_THRESHOLD}"
+ assert ts > SIMILARITY_THRESHOLD, f"{name}: tensor sim = {ts:.6f} < {SIMILARITY_THRESHOLD}"
+
+
+# =============================================================================
+# Test Function
+# =============================================================================
+
+
+@pytest.mark.parametrize("tc", TEST_CASES, ids=lambda tc: tc.name)
+def test_thd_format(tc: TestCase):
+ """Compare THD vs SBHD format outputs and gradients."""
+ H, seqlens = tc.hidden_size, tc.seqlens
+ tp_size, cp_size, sp = tc.tp_size, tc.cp_size, tc.sp_enabled
+ B = len(seqlens)
+ pad_to_max = tc.pad_thd_to_max
+
+ # Deterministic mode for bitwise tests
+ if tc.forward_bitwise or tc.backward_bitwise:
+ os.environ["NVTE_ALLOW_NONDETERMINISTIC_ALGO"] = "0"
+ torch.use_deterministic_algorithms(True, warn_only=True)
+
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=tp_size, context_parallel_size=cp_size
+ )
+ model_parallel_cuda_manual_seed(42)
+
+ deterministic = tc.forward_bitwise or tc.backward_bitwise
+ layer = build_gpt_layer(
+ H,
+ tc.num_heads,
+ tc.num_kv_heads,
+ tc.ffn_hidden_size,
+ tp_size,
+ cp_size,
+ sp,
+ tc.use_mock_attention,
+ deterministic,
+ )
+
+ cp_rank = parallel_state.get_context_parallel_rank()
+ tp_rank = parallel_state.get_tensor_model_parallel_rank()
+ dp_rank = parallel_state.get_data_parallel_rank()
+
+ # Generate data
+ torch.manual_seed(42 + dp_rank)
+ seq_data = [torch.randn(sl, H, dtype=torch.bfloat16).cuda() for sl in seqlens]
+ torch.manual_seed(142 + dp_rank)
+ grad_per_seq = [torch.randn(sl, H, dtype=torch.bfloat16).cuda() for sl in seqlens]
+
+ # Prepare SBHD
+ max_len = compute_sbhd_padded_max_len(seqlens, cp_size, tp_size, sp)
+ full_sbhd = torch.zeros(max_len, B, H, dtype=torch.bfloat16, device='cuda')
+ grad_sbhd = torch.zeros_like(full_sbhd)
+ for b, sl in enumerate(seqlens):
+ full_sbhd[:sl, b] = seq_data[b]
+ grad_sbhd[:sl, b] = grad_per_seq[b]
+
+ # Prepare THD grad (valid tokens only, gather_thd backward handles re-padding)
+ grad_thd = torch.cat(grad_per_seq, dim=0).unsqueeze(1)
+
+ # --- SBHD forward/backward ---
+ local_sbhd = shard_sbhd(full_sbhd, cp_rank, cp_size, tp_rank, tp_size, sp)
+ input_sbhd = local_sbhd.detach().clone().requires_grad_(True)
+ out_sbhd, _ = layer(hidden_states=input_sbhd)
+ gathered_sbhd = gather_sbhd(out_sbhd, cp_size, tp_size, sp)
+ gathered_sbhd.backward(grad_sbhd)
+ sbhd_grads = {n: p.grad.clone() for n, p in layer.named_parameters()}
+ layer.zero_grad()
+
+ # --- THD forward/backward ---
+ local_thd = shard_thd(seq_data, seqlens, cp_rank, cp_size, tp_rank, tp_size, sp, H, pad_to_max)
+ packed_seq_params = make_packed_seq_params(seqlens, cp_size, tp_size, sp, pad_to_max)
+ input_thd = local_thd.detach().clone().requires_grad_(True)
+ out_thd, _ = layer(hidden_states=input_thd, packed_seq_params=packed_seq_params)
+ gathered_thd = gather_thd(out_thd, seqlens, cp_size, tp_size, sp, H, pad_to_max)
+ gathered_thd.backward(grad_thd)
+ thd_grads = {n: p.grad.clone() for n, p in layer.named_parameters()}
+
+ # --- Gradient sync ---
+ # Reduce across DP*CP group (each DP/CP rank sees different data/tokens)
+ dp_cp_group = parallel_state.get_data_parallel_group(with_context_parallel=True)
+ for n in sbhd_grads:
+ dist.all_reduce(sbhd_grads[n], group=dp_cp_group)
+ dist.all_reduce(thd_grads[n], group=dp_cp_group)
+ # SP params also need reduction across TP group
+ if sp:
+ tp_group = parallel_state.get_tensor_model_parallel_group()
+ for n, p in layer.named_parameters():
+ if getattr(p, "sequence_parallel", False):
+ dist.all_reduce(sbhd_grads[n], group=tp_group)
+ dist.all_reduce(thd_grads[n], group=tp_group)
+
+ # --- Forward comparison ---
+ offset = 0
+ for b, sl in enumerate(seqlens):
+ assert_close(
+ f"seq[{b}] output",
+ gathered_sbhd[:sl, b].detach(),
+ gathered_thd[offset : offset + sl, 0].detach(),
+ tc.forward_bitwise,
+ )
+ offset += sl
+
+ # --- Backward comparison ---
+ for n in sbhd_grads:
+ if n in thd_grads:
+ assert_close(f"grad[{n}]", sbhd_grads[n], thd_grads[n], tc.backward_bitwise)
+
+ # --- Cleanup ---
+ Utils.destroy_model_parallel()
+ if tc.forward_bitwise or tc.backward_bitwise:
+ torch.use_deterministic_algorithms(False)
+ os.environ.pop("NVTE_ALLOW_NONDETERMINISTIC_ALGO", None)
diff --git a/tests/unit_tests/transformer/test_transformer_block.py b/tests/unit_tests/transformer/test_transformer_block.py
index ec6e72fdbd9..0ad39e2e5e0 100755
--- a/tests/unit_tests/transformer/test_transformer_block.py
+++ b/tests/unit_tests/transformer/test_transformer_block.py
@@ -146,6 +146,220 @@ def _run_selective_checkpoint_test(self, fp8):
assert hidden_states.shape[1] == micro_batch_size
assert hidden_states.shape[2] == config.hidden_size
+ def test_feature_extraction_without_checkpoint(self):
+ """Test feature extraction in normal forward mode (no checkpointing)."""
+ parallel_transformer_block = self.parallel_transformer_block
+ config = parallel_transformer_block.config
+
+ sequence_length = 32
+ micro_batch_size = 2
+ parallel_transformer_block.cuda()
+
+ # [sequence length, batch size, hidden size]
+ hidden_states = torch.ones((sequence_length, micro_batch_size, config.hidden_size))
+ hidden_states = hidden_states.cuda()
+
+ attention_mask = torch.ones((1, 1, sequence_length, sequence_length), dtype=bool).cuda()
+
+ # Test with None (should return just hidden_states)
+ output = parallel_transformer_block(
+ hidden_states=hidden_states, attention_mask=attention_mask, extract_layer_indices=None
+ )
+ assert isinstance(output, torch.Tensor)
+ assert output.shape == (sequence_length, micro_batch_size, config.hidden_size)
+
+ # Test with single layer extraction
+ extract_indices = {0}
+ result = parallel_transformer_block(
+ hidden_states=hidden_states,
+ attention_mask=attention_mask,
+ extract_layer_indices=extract_indices,
+ )
+ assert isinstance(result, tuple)
+ assert len(result) == 2
+ output_hidden_states, intermediate_hidden_states = result
+ assert output_hidden_states.shape == (sequence_length, micro_batch_size, config.hidden_size)
+ assert len(intermediate_hidden_states) == 1
+ assert intermediate_hidden_states[0].shape == (
+ sequence_length,
+ micro_batch_size,
+ config.hidden_size,
+ )
+
+ # Test with multiple layer extraction (config has 2 layers: indices 0, 1)
+ extract_indices = {0, 1}
+ result = parallel_transformer_block(
+ hidden_states=hidden_states,
+ attention_mask=attention_mask,
+ extract_layer_indices=extract_indices,
+ )
+ assert isinstance(result, tuple)
+ assert len(result) == 2
+ output_hidden_states, intermediate_hidden_states = result
+ assert output_hidden_states.shape == (sequence_length, micro_batch_size, config.hidden_size)
+ assert len(intermediate_hidden_states) == 2
+ for intermediate in intermediate_hidden_states:
+ assert intermediate.shape == (sequence_length, micro_batch_size, config.hidden_size)
+
+ def test_feature_extraction_full_checkpoint_block(self):
+ """Test feature extraction with full checkpoint using block method."""
+ transformer_config = self.transformer_config
+ config = transformer_config
+ config.recompute_granularity = 'full'
+ config.recompute_method = 'block'
+ config.recompute_num_layers = config.num_layers # checkpoint all layers
+ block_transformer_block = TransformerBlock(
+ config, get_gpt_layer_with_transformer_engine_spec()
+ )
+ block_transformer_block.cuda()
+
+ sequence_length = 32
+ micro_batch_size = 2
+
+ # [sequence length, batch size, hidden size]
+ hidden_states = torch.ones((sequence_length, micro_batch_size, config.hidden_size))
+ hidden_states = hidden_states.cuda()
+ hidden_states.requires_grad = True
+
+ attention_mask = torch.ones((1, 1, sequence_length, sequence_length), dtype=bool).cuda()
+
+ # Test with None (should return just hidden_states)
+ output = block_transformer_block(
+ hidden_states=hidden_states, attention_mask=attention_mask, extract_layer_indices=None
+ )
+ assert isinstance(output, torch.Tensor)
+ assert output.shape == (sequence_length, micro_batch_size, config.hidden_size)
+
+ # Test with single layer extraction
+ extract_indices = {0}
+ result = block_transformer_block(
+ hidden_states=hidden_states,
+ attention_mask=attention_mask,
+ extract_layer_indices=extract_indices,
+ )
+ assert isinstance(result, tuple)
+ assert len(result) == 2
+ output_hidden_states, intermediate_hidden_states = result
+ assert output_hidden_states.shape == (sequence_length, micro_batch_size, config.hidden_size)
+ assert len(intermediate_hidden_states) == 1
+ assert intermediate_hidden_states[0].shape == (
+ sequence_length,
+ micro_batch_size,
+ config.hidden_size,
+ )
+
+ # Test with multiple layer extraction (config has 2 layers: indices 0, 1)
+ # Unlike uniform, block method supports extraction at every layer
+ extract_indices = {0, 1}
+ result = block_transformer_block(
+ hidden_states=hidden_states,
+ attention_mask=attention_mask,
+ extract_layer_indices=extract_indices,
+ )
+ assert isinstance(result, tuple)
+ assert len(result) == 2
+ output_hidden_states, intermediate_hidden_states = result
+ assert output_hidden_states.shape == (sequence_length, micro_batch_size, config.hidden_size)
+ assert len(intermediate_hidden_states) == 2
+ for intermediate in intermediate_hidden_states:
+ assert intermediate.shape == (sequence_length, micro_batch_size, config.hidden_size)
+
+ def test_feature_extraction_full_checkpoint_uniform(self):
+ """Test feature extraction with full checkpoint using uniform method.
+
+ With recompute_num_layers=2 (chunk size 2) and num_layers=2, all layers
+ fall into a single chunk [0, 1]. The uniform method can only extract
+ features at chunk boundaries (the last layer of each chunk), so
+ requesting {0, 1} returns only 1 embedding (layer 1's output).
+ Layer 0's activation is discarded during the forward pass.
+ """
+ transformer_config = self.transformer_config
+ config = transformer_config
+ config.recompute_granularity = 'full'
+ config.recompute_method = 'uniform'
+ config.recompute_num_layers = 2 # chunk size 2: single chunk [0, 1]
+ uniform_transformer_block = TransformerBlock(
+ config, get_gpt_layer_with_transformer_engine_spec()
+ )
+ uniform_transformer_block.cuda()
+
+ sequence_length = 32
+ micro_batch_size = 2
+
+ # [sequence length, batch size, hidden size]
+ hidden_states = torch.ones((sequence_length, micro_batch_size, config.hidden_size))
+ hidden_states = hidden_states.cuda()
+ hidden_states.requires_grad = True
+
+ attention_mask = torch.ones((1, 1, sequence_length, sequence_length), dtype=bool).cuda()
+
+ # Test with None (should return just hidden_states)
+ output = uniform_transformer_block(
+ hidden_states=hidden_states, attention_mask=attention_mask, extract_layer_indices=None
+ )
+ assert isinstance(output, torch.Tensor)
+ assert output.shape == (sequence_length, micro_batch_size, config.hidden_size)
+
+ # With chunk size 2, layers [0, 1] form one chunk.
+ # Only the chunk boundary (layer 1) can be extracted; layer 0 is inside the chunk.
+ extract_indices = {0, 1}
+ result = uniform_transformer_block(
+ hidden_states=hidden_states,
+ attention_mask=attention_mask,
+ extract_layer_indices=extract_indices,
+ )
+ assert isinstance(result, tuple)
+ assert len(result) == 2
+ output_hidden_states, intermediate_hidden_states = result
+ assert output_hidden_states.shape == (sequence_length, micro_batch_size, config.hidden_size)
+ # Only 1 embedding returned: layer 1 (the chunk boundary). Layer 0 is skipped.
+ assert len(intermediate_hidden_states) == 1
+ assert intermediate_hidden_states[0].shape == (
+ sequence_length,
+ micro_batch_size,
+ config.hidden_size,
+ )
+
+ def test_gpu_forward_uniform_checkpoint_non_divisible_layers(self):
+ """Test that uniform recompute works when num_layers is not evenly
+ divisible by recompute_num_layers.
+
+ With num_layers=5 and recompute_num_layers=3 the chunks should be
+ [0, 1, 2] and [3, 4]. Before the fix the last chunk would attempt
+ to access layers 3, 4, **5** which is out of bounds.
+
+ Regression test for https://github.com/NVIDIA/Megatron-LM/issues/2294
+ """
+ transformer_config = TransformerConfig(
+ num_layers=5,
+ hidden_size=64,
+ num_attention_heads=4,
+ use_cpu_initialization=True,
+ recompute_granularity='full',
+ recompute_method='uniform',
+ recompute_num_layers=3,
+ )
+ uniform_block = TransformerBlock(
+ transformer_config, get_gpt_layer_with_transformer_engine_spec()
+ )
+ uniform_block.cuda()
+
+ sequence_length = 32
+ micro_batch_size = 2
+
+ hidden_states = torch.ones(
+ (sequence_length, micro_batch_size, transformer_config.hidden_size), device='cuda'
+ )
+ hidden_states.requires_grad = True
+
+ attention_mask = torch.ones(
+ (1, 1, sequence_length, sequence_length), dtype=bool, device='cuda'
+ )
+
+ # This should not raise an IndexError
+ output = uniform_block(hidden_states=hidden_states, attention_mask=attention_mask)
+ assert output.shape == (sequence_length, micro_batch_size, transformer_config.hidden_size)
+
class TestPipelineParallelTransformerBlock:
@pytest.mark.parametrize(
diff --git a/tests/unit_tests/transformer/test_transformer_block_custom_pgs.py b/tests/unit_tests/transformer/test_transformer_block_custom_pgs.py
index bb64efe7449..c55babe35ca 100644
--- a/tests/unit_tests/transformer/test_transformer_block_custom_pgs.py
+++ b/tests/unit_tests/transformer/test_transformer_block_custom_pgs.py
@@ -68,7 +68,7 @@ def __init__(
# Temporarily replace attention and MLP with IdentityOp,
# This is a temporary workaround for the test until we have a better interface
# will rebuild them with custom process groups after super init
- def _modify_submodules(submodules):
+ def _modify_submodules(submodules: TransformerLayerSubmodules):
submodules.self_attention = IdentityOp
submodules.mlp = IdentityOp
return submodules
diff --git a/tests/unit_tests/transformer/test_transformer_layer.py b/tests/unit_tests/transformer/test_transformer_layer.py
index 7db9aa30fec..da1f9ce5860 100644
--- a/tests/unit_tests/transformer/test_transformer_layer.py
+++ b/tests/unit_tests/transformer/test_transformer_layer.py
@@ -7,7 +7,9 @@
from megatron.core import parallel_state
from megatron.core.dist_checkpointing.mapping import ShardedObject, ShardedTensor
from megatron.core.inference.contexts import StaticInferenceContext
-from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
+from megatron.core.models.gpt.gpt_layer_specs import (
+ get_gpt_layer_with_transformer_engine_submodules,
+)
from megatron.core.tensor_parallel.random import model_parallel_cuda_manual_seed
from megatron.core.transformer.transformer_config import TransformerConfig
from megatron.core.transformer.transformer_layer import (
@@ -26,7 +28,7 @@ def setup_method(self, method):
num_layers=2, hidden_size=12, num_attention_heads=4, use_cpu_initialization=True
)
self.parallel_transformer_layer = TransformerLayer(
- transformer_config, get_gpt_layer_with_transformer_engine_spec().submodules
+ transformer_config, get_gpt_layer_with_transformer_engine_submodules()
)
def teardown_method(self, method):
@@ -81,7 +83,7 @@ def test(
use_cpu_initialization=True,
)
parallel_transformer_layer = TransformerLayer(
- transformer_config, get_gpt_layer_with_transformer_engine_spec().submodules
+ transformer_config, get_gpt_layer_with_transformer_engine_submodules()
)
parallel_transformer_layer.cuda()
@@ -265,7 +267,7 @@ def test_sharded_state_dict(self, tp_pp, order):
num_layers=2, hidden_size=128, num_attention_heads=8, use_cpu_initialization=True
)
parallel_transformer_layer = TransformerLayer(
- transformer_config, get_gpt_layer_with_transformer_engine_spec().submodules
+ transformer_config, get_gpt_layer_with_transformer_engine_submodules()
)
sharded_state_dict = parallel_transformer_layer.sharded_state_dict()
diff --git a/tests/unit_tests/transformer/test_vision_cuda_graphs.py b/tests/unit_tests/transformer/test_vision_cuda_graphs.py
new file mode 100644
index 00000000000..bfd431e67a3
--- /dev/null
+++ b/tests/unit_tests/transformer/test_vision_cuda_graphs.py
@@ -0,0 +1,685 @@
+# Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+import gc
+import os
+from copy import deepcopy
+from types import SimpleNamespace
+from unittest.mock import MagicMock
+
+import pytest
+import torch
+
+from megatron.core import parallel_state
+from megatron.core.models.gpt.gpt_layer_specs import get_gpt_layer_with_transformer_engine_spec
+from megatron.core.models.vision.vit_layer_specs import get_vit_layer_with_transformer_engine_spec
+from megatron.core.tensor_parallel.random import (
+ HAVE_TE,
+ initialize_rng_tracker,
+ model_parallel_cuda_manual_seed,
+)
+from megatron.core.transformer.cuda_graphs import (
+ HAVE_TE_GRAPHS,
+ VisionTECudaGraphHelper,
+ _layer_is_graphable,
+ _wrap_graph_for_vision,
+ get_vision_cuda_graph_seq_length,
+ set_current_microbatch,
+)
+from megatron.core.transformer.transformer_config import TransformerConfig
+from megatron.core.utils import is_te_min_version
+from tests.unit_tests.test_utilities import Utils
+
+TE_MIN_VERSION = "2.13.0"
+_te_version_ok = HAVE_TE and is_te_min_version(TE_MIN_VERSION)
+if not _te_version_ok and __name__ != "__main__":
+ pytest.skip(
+ f"Vision CUDA graph tests require TransformerEngine >= {TE_MIN_VERSION}",
+ allow_module_level=True,
+ )
+
+
+# ---------------------------------------------------------------------------
+# Tests for _layer_is_graphable
+# ---------------------------------------------------------------------------
+class TestVisionLayerIsGraphable:
+ def test_non_transformer_layer_returns_false(self):
+ config = SimpleNamespace(cuda_graph_impl="transformer_engine")
+ layer = torch.nn.Linear(4, 4)
+ assert _layer_is_graphable(layer, config) is False
+
+ @pytest.mark.flaky
+ @pytest.mark.flaky_in_dev
+ def test_wrong_cuda_graph_impl_returns_false(self):
+ from megatron.core.transformer.transformer_layer import TransformerLayer
+
+ config = SimpleNamespace(cuda_graph_impl="local")
+ layer = MagicMock(spec=TransformerLayer)
+ # isinstance check with MagicMock(spec=...) should pass
+ assert _layer_is_graphable(layer, config) is False
+
+ def test_correct_config_with_transformer_layer(self):
+ """Real TransformerLayer + cuda_graph_impl='transformer_engine' -> True."""
+ initialize_rng_tracker(use_te_rng_tracker=True, force_reset=True)
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=1, pipeline_model_parallel_size=1
+ )
+ model_parallel_cuda_manual_seed(123)
+
+ config = TransformerConfig(
+ num_layers=1,
+ hidden_size=16,
+ num_attention_heads=2,
+ use_cpu_initialization=True,
+ cuda_graph_impl="transformer_engine",
+ )
+ from megatron.core.transformer.transformer_block import TransformerBlock
+
+ block = TransformerBlock(config, get_vit_layer_with_transformer_engine_spec())
+ layer = block.layers[0]
+ assert _layer_is_graphable(layer, config) is True
+
+ Utils.destroy_model_parallel()
+
+
+# ---------------------------------------------------------------------------
+# Tests for _wrap_graph_for_vision
+# ---------------------------------------------------------------------------
+class TestWrapGraphForVision:
+ def test_filters_none_from_tuple(self):
+ def fake_graph(*args, **kwargs):
+ return (torch.tensor(1.0), None)
+
+ wrapped = _wrap_graph_for_vision(fake_graph)
+ result = wrapped()
+ assert result == (torch.tensor(1.0),)
+
+ def test_returns_non_tuple_unchanged(self):
+ t = torch.tensor(42.0)
+
+ def fake_graph(*args, **kwargs):
+ return t
+
+ wrapped = _wrap_graph_for_vision(fake_graph)
+ result = wrapped()
+ assert result is t
+
+ def test_preserves_all_non_none(self):
+ a, b = torch.tensor(1.0), torch.tensor(2.0)
+
+ def fake_graph(*args, **kwargs):
+ return (a, b)
+
+ wrapped = _wrap_graph_for_vision(fake_graph)
+ result = wrapped()
+ assert result == (a, b)
+
+ def test_all_none_returns_original(self):
+ def fake_graph(*args, **kwargs):
+ return (None, None)
+
+ wrapped = _wrap_graph_for_vision(fake_graph)
+ result = wrapped()
+ # filtered is empty -> returns original tuple
+ assert result == (None, None)
+
+ def test_preserves_te_attributes(self):
+ def fake_graph(*args, **kwargs):
+ return (torch.tensor(1.0),)
+
+ fake_graph.backward_dw = "bwd_dw_fn"
+ fake_graph.reset = "reset_fn"
+
+ wrapped = _wrap_graph_for_vision(fake_graph)
+ assert wrapped.backward_dw == "bwd_dw_fn"
+ assert wrapped.reset == "reset_fn"
+
+ def test_missing_te_attributes_not_set(self):
+ def fake_graph(*args, **kwargs):
+ return (torch.tensor(1.0),)
+
+ wrapped = _wrap_graph_for_vision(fake_graph)
+ assert not hasattr(wrapped, 'backward_dw')
+ assert not hasattr(wrapped, 'reset')
+
+
+# ---------------------------------------------------------------------------
+# Tests for get_vision_cuda_graph_seq_length
+# ---------------------------------------------------------------------------
+class TestGetVisionCudaGraphSeqLength:
+ def test_explicit_max_seq_length(self):
+ config = SimpleNamespace(max_vision_cuda_graph_seq_length=2048)
+ assert get_vision_cuda_graph_seq_length(config) == 2048
+
+ def test_explicit_max_seq_length_zero_falls_through(self):
+ """max_vision_cuda_graph_seq_length=0 is falsy, should fall through."""
+ config = SimpleNamespace(max_vision_cuda_graph_seq_length=0)
+ assert get_vision_cuda_graph_seq_length(config, default_seq_length=999) == 999
+
+ def test_num_position_embeddings_only(self):
+ config = SimpleNamespace(num_position_embeddings=1024)
+ assert get_vision_cuda_graph_seq_length(config) == 1024
+
+ def test_num_position_embeddings_with_spatial_merge(self):
+ config = SimpleNamespace(num_position_embeddings=1024, spatial_merge_size=2)
+ # merge_factor = 2**2 = 4, seq = 1024 // 4 = 256
+ assert get_vision_cuda_graph_seq_length(config) == 256
+
+ def test_spatial_merge_size_3(self):
+ config = SimpleNamespace(num_position_embeddings=900, spatial_merge_size=3)
+ # merge_factor = 9, seq = 900 // 9 = 100
+ assert get_vision_cuda_graph_seq_length(config) == 100
+
+ def test_default_seq_length(self):
+ config = SimpleNamespace()
+ assert get_vision_cuda_graph_seq_length(config) == 4096
+
+ def test_custom_default(self):
+ config = SimpleNamespace()
+ assert get_vision_cuda_graph_seq_length(config, default_seq_length=512) == 512
+
+ def test_explicit_overrides_position_embeddings(self):
+ config = SimpleNamespace(
+ max_vision_cuda_graph_seq_length=8192, num_position_embeddings=1024
+ )
+ assert get_vision_cuda_graph_seq_length(config) == 8192
+
+
+# ---------------------------------------------------------------------------
+# Integration test for VisionTECudaGraphHelper with LLaVA model
+# ---------------------------------------------------------------------------
+@pytest.mark.skipif(
+ not (HAVE_TE and is_te_min_version("1.5.0")),
+ reason="use_te_rng_tracker requires TransformerEngine version >= 1.5",
+)
+class TestVisionTECudaGraphHelper:
+ """Test VisionTECudaGraphHelper initialization, sample args, and graph lifecycle."""
+
+ def setup_method(self, method):
+ initialize_rng_tracker(use_te_rng_tracker=True, force_reset=True)
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=1,
+ pipeline_model_parallel_size=1,
+ virtual_pipeline_model_parallel_size=None,
+ )
+ model_parallel_cuda_manual_seed(123)
+
+ from megatron.core.models.multimodal.llava_model import LLaVAModel
+
+ self.language_hidden_size = 64
+ self.vision_hidden_size = 16
+ self.vision_num_layers = 2
+
+ language_config = TransformerConfig(
+ num_layers=2,
+ hidden_size=self.language_hidden_size,
+ num_attention_heads=4,
+ use_cpu_initialization=True,
+ )
+
+ self.vision_config = TransformerConfig(
+ num_layers=self.vision_num_layers,
+ hidden_size=self.vision_hidden_size,
+ num_attention_heads=2,
+ use_cpu_initialization=True,
+ cuda_graph_impl="transformer_engine",
+ bf16=True,
+ pipeline_dtype=torch.bfloat16,
+ )
+
+ vision_projection_config = TransformerConfig(
+ num_layers=1,
+ hidden_size=self.language_hidden_size,
+ ffn_hidden_size=32,
+ num_attention_heads=1,
+ use_cpu_initialization=True,
+ bf16=True,
+ pipeline_dtype=torch.bfloat16,
+ )
+
+ language_layer_spec = get_gpt_layer_with_transformer_engine_spec()
+ vision_layer_spec = get_vit_layer_with_transformer_engine_spec()
+ vision_projection_spec = deepcopy(language_layer_spec.submodules.mlp.submodules)
+
+ self.vision_config.vision_model_type = "clip"
+ language_config.language_model_type = "dummy"
+
+ self.llava_model = LLaVAModel(
+ language_transformer_config=language_config,
+ language_transformer_layer_spec=language_layer_spec,
+ language_vocab_size=8192,
+ language_max_sequence_length=4096,
+ vision_transformer_config=self.vision_config,
+ vision_transformer_layer_spec=vision_layer_spec,
+ drop_vision_class_token=False,
+ vision_projection_config=vision_projection_config,
+ vision_projection_layer_spec=vision_projection_spec,
+ img_h=336,
+ img_w=336,
+ patch_dim=14,
+ pre_process=True,
+ post_process=True,
+ add_encoder=True,
+ add_decoder=True,
+ )
+ self.llava_model.bfloat16()
+
+ self.vision_seq_length = 576
+ self.micro_batch_size = 2
+
+ def teardown_method(self, method):
+ Utils.destroy_model_parallel()
+ gc.collect()
+
+ def _make_helper(self, num_microbatches=1):
+ return VisionTECudaGraphHelper(
+ model=[self.llava_model],
+ vision_config=self.vision_config,
+ vision_seq_length=self.vision_seq_length,
+ micro_batch_size=self.micro_batch_size,
+ num_microbatches=num_microbatches,
+ )
+
+ # -- Initialization tests --
+
+ def test_init_finds_vision_layers(self):
+ helper = self._make_helper()
+ assert helper.vision_model is not None, "Should find vision_model"
+ assert helper.num_layers == self.vision_num_layers
+ assert len(helper.callables) == self.vision_num_layers
+ assert helper.capture_finished() is False
+ assert helper.graphs_created() is False
+
+ def test_init_no_vision_model_warns(self):
+ """When model has no vision_model attr, helper should degrade gracefully."""
+ dummy_model = torch.nn.Linear(4, 4)
+ helper = VisionTECudaGraphHelper(
+ model=[dummy_model],
+ vision_config=self.vision_config,
+ vision_seq_length=self.vision_seq_length,
+ micro_batch_size=self.micro_batch_size,
+ )
+ assert helper.vision_model is None
+ assert len(helper.callables) == 0
+ assert helper.capture_finished() is False
+ assert helper.graphs_created() is False
+
+ # -- _get_sample_arguments tests --
+
+ def test_get_sample_arguments_shapes(self):
+ helper = self._make_helper(num_microbatches=1)
+ # order is unused by vision override; pass a dummy
+ sample_args, sample_kwargs_list = helper._get_sample_arguments(order=[1, -1])
+
+ expected_count = self.vision_num_layers * 1 # layers * microbatches
+ assert len(sample_args) == expected_count
+ assert len(sample_kwargs_list) == expected_count
+
+ for i, (args_item, kwargs_item) in enumerate(zip(sample_args, sample_kwargs_list)):
+ assert isinstance(args_item, tuple), f"sample_args[{i}] should be tuple"
+ assert len(args_item) == 1, f"sample_args[{i}] should have one element (hidden_states)"
+ hs = args_item[0]
+ assert hs.shape == (self.vision_seq_length, 1, self.vision_hidden_size), (
+ f"Expected ({self.vision_seq_length}, 1, {self.vision_hidden_size}), "
+ f"got {hs.shape}"
+ )
+ assert hs.dtype == torch.bfloat16
+ assert hs.device.type == 'cuda'
+ assert hs.requires_grad is True
+
+ def test_get_sample_arguments_multi_microbatch(self):
+ helper = self._make_helper(num_microbatches=3)
+ sample_args, sample_kwargs_list = helper._get_sample_arguments(order=[1, -1])
+
+ expected_count = self.vision_num_layers * 3
+ assert len(sample_args) == expected_count
+ assert len(sample_kwargs_list) == expected_count
+
+ def test_get_sample_arguments_empty_when_no_callables(self):
+ dummy_model = torch.nn.Linear(4, 4)
+ helper = VisionTECudaGraphHelper(
+ model=[dummy_model],
+ vision_config=self.vision_config,
+ vision_seq_length=self.vision_seq_length,
+ micro_batch_size=self.micro_batch_size,
+ )
+ sample_args, sample_kwargs_list = helper._get_sample_arguments(order=[1, -1])
+ assert sample_args == []
+ assert sample_kwargs_list == []
+
+ # -- create_cudagraphs / delete_cuda_graphs lifecycle --
+ @pytest.mark.flaky
+ @pytest.mark.flaky_in_dev
+ @pytest.mark.skipif(
+ not (HAVE_TE_GRAPHS and is_te_min_version("2.7.0")),
+ reason="TE CUDA graph capture requires TransformerEngine >= 2.7.0",
+ )
+ def test_create_and_delete_cudagraphs(self):
+ """Full lifecycle: create graphs, verify state, delete, verify cleanup."""
+ self.llava_model.cuda()
+ helper = self._make_helper(num_microbatches=1)
+
+ assert not helper.graphs_created()
+
+ helper.create_cudagraphs()
+ assert helper.graphs_created()
+
+ # Each vision layer should have cuda_graphs attached
+ for layer in helper.callables:
+ assert hasattr(layer, 'cuda_graphs'), "Layer should have cuda_graphs after capture"
+ assert len(layer.cuda_graphs) == 1 # 1 microbatch
+
+ # cudagraph_manager should have been removed during capture
+ for layer in helper.callables:
+ assert not hasattr(
+ layer, 'cudagraph_manager'
+ ), "cudagraph_manager should be removed before TE capture"
+
+ helper.delete_cuda_graphs()
+ assert not helper.graphs_created()
+
+ # cuda_graphs should be empty after delete
+ for layer in helper.callables:
+ assert layer.cuda_graphs == [], "cuda_graphs should be empty after delete"
+
+ @pytest.mark.skipif(
+ not (HAVE_TE_GRAPHS and is_te_min_version("2.7.0")),
+ reason="TE CUDA graph capture requires TransformerEngine >= 2.7.0",
+ )
+ @pytest.mark.flaky
+ @pytest.mark.flaky_in_dev
+ def test_create_cudagraphs_multi_microbatch(self):
+ """Verify that graphs are created per-microbatch per-layer."""
+ self.llava_model.cuda()
+ num_mb = 2
+ helper = self._make_helper(num_microbatches=num_mb)
+
+ helper.create_cudagraphs()
+ assert helper.graphs_created()
+
+ for layer in helper.callables:
+ assert hasattr(layer, 'cuda_graphs')
+ # PP=1 collapses to 1 microbatch internally
+ assert len(layer.cuda_graphs) == helper.num_microbatches
+
+ helper.delete_cuda_graphs()
+
+ def test_create_cudagraphs_no_callables_is_noop(self):
+ """create_cudagraphs on empty helper should not crash."""
+ dummy_model = torch.nn.Linear(4, 4)
+ helper = VisionTECudaGraphHelper(
+ model=[dummy_model],
+ vision_config=self.vision_config,
+ vision_seq_length=self.vision_seq_length,
+ micro_batch_size=self.micro_batch_size,
+ )
+ helper.create_cudagraphs()
+ assert helper.capture_finished() # Capture process finished
+ assert not helper.graphs_created() # But no graphs were created
+
+ def test_delete_cudagraphs_before_create_asserts(self):
+ """delete_cuda_graphs before creation should raise AssertionError."""
+ helper = self._make_helper()
+ with pytest.raises(AssertionError):
+ helper.delete_cuda_graphs()
+
+
+# ---------------------------------------------------------------------------
+# Integration test with PP=2: vision encoder on first pipeline stage only
+# ---------------------------------------------------------------------------
+@pytest.mark.skipif(
+ not (HAVE_TE and is_te_min_version("1.5.0")),
+ reason="use_te_rng_tracker requires TransformerEngine version >= 1.5",
+)
+class TestVisionTECudaGraphHelperPP2:
+ """Test VisionTECudaGraphHelper with PP=2.
+
+ With pipeline_model_parallel_size=2 the LLaVA model is split so that the
+ vision encoder lives exclusively on the first pipeline stage:
+ - pp_rank 0: add_encoder=True, pre_process=True, post_process=False
+ - pp_rank 1: add_encoder=False, pre_process=False, post_process=True
+
+ This test verifies that:
+ 1. On stage 0 the helper finds and captures vision layers.
+ 2. On stage 1 the helper gracefully finds no vision layers.
+ 3. With PP>1, num_microbatches is NOT collapsed to 1.
+ """
+
+ def setup_method(self, method):
+ initialize_rng_tracker(use_te_rng_tracker=True, force_reset=True)
+ Utils.initialize_model_parallel(
+ tensor_model_parallel_size=1,
+ pipeline_model_parallel_size=2,
+ virtual_pipeline_model_parallel_size=None,
+ )
+ model_parallel_cuda_manual_seed(123)
+
+ from megatron.core.models.multimodal.llava_model import LLaVAModel
+
+ self.language_hidden_size = 64
+ self.vision_hidden_size = 16
+ self.vision_num_layers = 2
+ self.language_num_layers = 4
+
+ pp_rank = parallel_state.get_pipeline_model_parallel_rank()
+ is_first_stage = pp_rank == 0
+ is_last_stage = pp_rank == (parallel_state.get_pipeline_model_parallel_world_size() - 1)
+
+ language_config = TransformerConfig(
+ num_layers=self.language_num_layers,
+ hidden_size=self.language_hidden_size,
+ num_attention_heads=4,
+ use_cpu_initialization=True,
+ pipeline_model_parallel_size=2,
+ bf16=True,
+ pipeline_dtype=torch.bfloat16,
+ )
+
+ self.vision_config = TransformerConfig(
+ num_layers=self.vision_num_layers,
+ hidden_size=self.vision_hidden_size,
+ num_attention_heads=2,
+ use_cpu_initialization=True,
+ cuda_graph_impl="transformer_engine",
+ bf16=True,
+ pipeline_dtype=torch.bfloat16,
+ )
+
+ vision_projection_config = TransformerConfig(
+ num_layers=1,
+ hidden_size=self.language_hidden_size,
+ ffn_hidden_size=32,
+ num_attention_heads=1,
+ use_cpu_initialization=True,
+ bf16=True,
+ pipeline_dtype=torch.bfloat16,
+ )
+
+ language_layer_spec = get_gpt_layer_with_transformer_engine_spec()
+ vision_layer_spec = get_vit_layer_with_transformer_engine_spec()
+ vision_projection_spec = deepcopy(language_layer_spec.submodules.mlp.submodules)
+
+ self.vision_config.vision_model_type = "clip"
+ language_config.language_model_type = "dummy"
+
+ self.is_first_stage = is_first_stage
+ self.llava_model = LLaVAModel(
+ language_transformer_config=language_config,
+ language_transformer_layer_spec=language_layer_spec,
+ language_vocab_size=8192,
+ language_max_sequence_length=4096,
+ vision_transformer_config=self.vision_config,
+ vision_transformer_layer_spec=vision_layer_spec,
+ drop_vision_class_token=False,
+ vision_projection_config=vision_projection_config,
+ vision_projection_layer_spec=vision_projection_spec,
+ img_h=336,
+ img_w=336,
+ patch_dim=14,
+ pre_process=is_first_stage,
+ post_process=is_last_stage,
+ add_encoder=is_first_stage,
+ add_decoder=True,
+ )
+ self.llava_model.bfloat16()
+
+ self.vision_seq_length = 576
+ self.micro_batch_size = 2
+
+ def teardown_method(self, method):
+ Utils.destroy_model_parallel()
+ gc.collect()
+
+ def _make_helper(self, num_microbatches=4):
+ return VisionTECudaGraphHelper(
+ model=[self.llava_model],
+ vision_config=self.vision_config,
+ vision_seq_length=self.vision_seq_length,
+ micro_batch_size=self.micro_batch_size,
+ num_microbatches=num_microbatches,
+ )
+
+ def test_pp2_first_stage_finds_vision_layers(self):
+ """Stage 0 should discover all vision encoder layers."""
+ if not self.is_first_stage:
+ pytest.skip("This assertion is only for pp_rank 0")
+
+ helper = self._make_helper(num_microbatches=4)
+ assert helper.vision_model is not None
+ assert helper.num_layers == self.vision_num_layers
+ assert len(helper.callables) == self.vision_num_layers
+
+ def test_pp2_last_stage_has_no_vision_layers(self):
+ """Stage 1 should find no vision model (encoder lives on stage 0)."""
+ if self.is_first_stage:
+ pytest.skip("This assertion is only for pp_rank 1")
+
+ helper = self._make_helper(num_microbatches=4)
+ assert helper.vision_model is None
+ assert len(helper.callables) == 0
+ assert not helper.capture_finished()
+ assert not helper.graphs_created()
+
+ def test_pp2_num_microbatches_preserved(self):
+ """With PP>1, num_microbatches should NOT be collapsed to 1."""
+ if not self.is_first_stage:
+ pytest.skip("Vision layers only on pp_rank 0")
+
+ num_mb = 8
+ helper = self._make_helper(num_microbatches=num_mb)
+ # _get_sample_arguments generates layers * microbatches entries
+ sample_args, sample_kwargs_list = helper._get_sample_arguments(order=[1, -1])
+ expected_count = self.vision_num_layers * num_mb
+ assert len(sample_args) == expected_count, (
+ f"With PP>1, expected {expected_count} sample_args "
+ f"(layers={self.vision_num_layers} * mb={num_mb}), got {len(sample_args)}"
+ )
+
+ @pytest.mark.skipif(
+ not (HAVE_TE_GRAPHS and is_te_min_version("2.7.0")),
+ reason="TE CUDA graph capture requires TransformerEngine >= 2.7.0",
+ )
+ @pytest.mark.flaky
+ @pytest.mark.flaky_in_dev
+ def test_pp2_create_cudagraphs_first_stage(self):
+ """On stage 0, CUDA graphs should be captured with the full pipeline order."""
+ if not self.is_first_stage:
+ pytest.skip("Vision layers only on pp_rank 0")
+
+ self.llava_model.cuda()
+ num_mb = 4
+ helper = self._make_helper(num_microbatches=num_mb)
+
+ assert not helper.graphs_created()
+
+ helper.create_cudagraphs()
+ assert helper.graphs_created()
+
+ # num_microbatches should be preserved (PP>1 does not collapse)
+ assert helper.num_microbatches == num_mb
+
+ # Each layer should have one graph per microbatch
+ for layer in helper.callables:
+ assert hasattr(layer, 'cuda_graphs')
+ assert (
+ len(layer.cuda_graphs) == num_mb
+ ), f"Expected {num_mb} graphs per layer, got {len(layer.cuda_graphs)}"
+
+ # Cleanup
+ helper.delete_cuda_graphs()
+ assert not helper.graphs_created()
+ for layer in helper.callables:
+ assert layer.cuda_graphs == []
+
+ @pytest.mark.skipif(
+ not (HAVE_TE_GRAPHS and is_te_min_version("2.7.0")),
+ reason="TE CUDA graph capture requires TransformerEngine >= 2.7.0",
+ )
+ def test_pp2_create_cudagraphs_last_stage_noop(self):
+ """On stage 1 (no vision model), create_cudagraphs should be a no-op."""
+ if self.is_first_stage:
+ pytest.skip("This assertion is only for pp_rank 1")
+
+ helper = self._make_helper(num_microbatches=4)
+ helper.create_cudagraphs()
+ assert helper.capture_finished() # Capture process finished
+ assert not helper.graphs_created() # But no graphs were created
+
+
+if __name__ == "__main__":
+ if not _te_version_ok:
+ print(f"SKIPPED: Vision CUDA graph tests require TransformerEngine >= {TE_MIN_VERSION}")
+ exit(0)
+
+ from _pytest.outcomes import Skipped
+
+ def run_test(test_obj, test_fn_name):
+ """Run a test method, treating pytest.skip() as a non-error."""
+ test_obj.setup_method(method=None)
+ try:
+ getattr(test_obj, test_fn_name)()
+ except Skipped as e:
+ print(f" SKIPPED {test_fn_name}: {e}")
+ finally:
+ test_obj.teardown_method(method=None)
+
+ # Quick smoke tests for pure functions
+ t = TestWrapGraphForVision()
+ t.test_filters_none_from_tuple()
+ t.test_returns_non_tuple_unchanged()
+ t.test_preserves_all_non_none()
+ t.test_all_none_returns_original()
+ t.test_preserves_te_attributes()
+ t.test_missing_te_attributes_not_set()
+ print("_wrap_graph_for_vision tests passed.")
+
+ t2 = TestGetVisionCudaGraphSeqLength()
+ t2.test_explicit_max_seq_length()
+ t2.test_explicit_max_seq_length_zero_falls_through()
+ t2.test_num_position_embeddings_only()
+ t2.test_num_position_embeddings_with_spatial_merge()
+ t2.test_spatial_merge_size_3()
+ t2.test_default_seq_length()
+ t2.test_custom_default()
+ t2.test_explicit_overrides_position_embeddings()
+ print("get_vision_cuda_graph_seq_length tests passed.")
+
+ # Integration tests (require GPU + distributed init)
+ t3 = TestVisionTECudaGraphHelper()
+ run_test(t3, "test_init_finds_vision_layers")
+ run_test(t3, "test_get_sample_arguments_shapes")
+ run_test(t3, "test_create_and_delete_cudagraphs")
+ print("TestVisionTECudaGraphHelper tests passed.")
+
+ # PP=2 integration tests (require 2+ GPUs)
+ if Utils.world_size >= 2:
+ t4 = TestVisionTECudaGraphHelperPP2()
+ run_test(t4, "test_pp2_first_stage_finds_vision_layers")
+ run_test(t4, "test_pp2_last_stage_has_no_vision_layers")
+ run_test(t4, "test_pp2_num_microbatches_preserved")
+ run_test(t4, "test_pp2_create_cudagraphs_first_stage")
+ run_test(t4, "test_pp2_create_cudagraphs_last_stage_noop")
+ print("TestVisionTECudaGraphHelperPP2 tests passed.")
+ else:
+ print("SKIPPED TestVisionTECudaGraphHelperPP2 (requires 2+ GPUs)")
+
+ print("All vision CUDA graph tests passed.")
diff --git a/tools/checkpoint/loader_base.py b/tools/checkpoint/loader_base.py
index ef9d3688a69..e12e9545b87 100644
--- a/tools/checkpoint/loader_base.py
+++ b/tools/checkpoint/loader_base.py
@@ -135,7 +135,6 @@ def initialize_megatron_env(self):
try:
from megatron.training.global_vars import set_global_variables
from megatron.core import mpu
- from megatron.legacy import fused_kernels
except ModuleNotFoundError as e:
print(f"Unable to import required Megatron modules: {e}")
self.queue.put("exit")
@@ -152,7 +151,6 @@ def initialize_megatron_env(self):
fake_ep_group = _ConverterFakeProcessGroup(size=self.margs.expert_model_parallel_size)
mpu._TENSOR_MODEL_PARALLEL_GROUP = fake_tp_group
mpu._EXPERT_MODEL_PARALLEL_GROUP = fake_ep_group
- fused_kernels.load(self.margs)
def compute_true_vocab_size(self):
"""Determine the 'true' (non-padded) vocab size."""
@@ -470,7 +468,6 @@ def build_sys_argv(self):
'--no-masked-softmax-fusion',
'--no-bias-gelu-fusion',
'--no-bias-dropout-fusion',
- '--no-async-tensor-model-parallel-allreduce',
'--use-cpu-initialization',
'--micro-batch-size', '1',
'--no-load-optim',
diff --git a/tools/checkpoint/loader_legacy.py b/tools/checkpoint/loader_legacy.py
index 0aefadacd4c..31f4e2aeb09 100644
--- a/tools/checkpoint/loader_legacy.py
+++ b/tools/checkpoint/loader_legacy.py
@@ -47,7 +47,6 @@ def _load_checkpoint(queue, args):
from megatron.legacy.model import module
from megatron.core import mpu
from megatron.core.enums import ModelType
- from megatron.legacy import fused_kernels
except ModuleNotFoundError:
print("Unable to import Megatron, please specify the path to Megatron using --megatron-path. Exiting.")
queue.put("exit")
@@ -58,7 +57,6 @@ def _load_checkpoint(queue, args):
'--no-masked-softmax-fusion',
'--no-bias-gelu-fusion',
'--no-bias-dropout-fusion',
- '--no-async-tensor-model-parallel-allreduce',
'--use-cpu-initialization',
'--micro-batch-size', '1',
'--no-load-optim',
@@ -188,7 +186,6 @@ def get_models(count, dtype):
# For backward compatibility during local parallel states refactoring
fake_tp_group = _ConverterFakeProcessGroup(size=margs.tensor_model_parallel_size)
mpu._TENSOR_MODEL_PARALLEL_GROUP = fake_tp_group
- fused_kernels.load(margs)
# Get true (non-padded) vocab size
if args.true_vocab_size is not None:
diff --git a/tools/checkpoint/loader_llama_mistral.py b/tools/checkpoint/loader_llama_mistral.py
index ce9118db375..c96615edd72 100644
--- a/tools/checkpoint/loader_llama_mistral.py
+++ b/tools/checkpoint/loader_llama_mistral.py
@@ -427,7 +427,6 @@ def _load_checkpoint(queue, args):
from megatron.legacy.model import module
from megatron.core import mpu
from megatron.core.enums import ModelType
- from megatron.legacy import fused_kernels
except ModuleNotFoundError:
print("Unable to import Megatron, please specify the path to Megatron using --megatron-path. Exiting.")
queue.put("exit")
@@ -438,7 +437,6 @@ def _load_checkpoint(queue, args):
'--no-masked-softmax-fusion',
'--no-bias-gelu-fusion',
'--no-bias-dropout-fusion',
- '--no-async-tensor-model-parallel-allreduce',
'--use-cpu-initialization',
'--micro-batch-size', '1',
'--no-load-optim',
@@ -518,7 +516,6 @@ def check_for_arg(arg_name, default=None):
mpu.set_tensor_model_parallel_world_size(margs.tensor_model_parallel_size)
mpu.set_pipeline_model_parallel_world_size(margs.pipeline_model_parallel_size)
mpu.set_virtual_pipeline_model_parallel_world_size(margs.virtual_pipeline_model_parallel_size)
- fused_kernels.load(margs)
# For backward compatibility during local parallel states refactoring
fake_tp_group = _ConverterFakeProcessGroup(size=margs.tensor_model_parallel_size)
diff --git a/tools/checkpoint/loader_mixtral_hf.py b/tools/checkpoint/loader_mixtral_hf.py
index 52a1a4d311e..3ecbdd15b7e 100644
--- a/tools/checkpoint/loader_mixtral_hf.py
+++ b/tools/checkpoint/loader_mixtral_hf.py
@@ -168,7 +168,6 @@ def _load_checkpoint(queue, args):
from megatron.legacy.model import module
from megatron.core import mpu
from megatron.core.enums import ModelType
- from megatron.legacy import fused_kernels
except ModuleNotFoundError:
print("Unable to import Megatron, please specify the path to Megatron using --megatron-path. Exiting.")
queue.put("exit")
@@ -181,7 +180,6 @@ def _load_checkpoint(queue, args):
'--no-masked-softmax-fusion',
'--no-bias-gelu-fusion',
'--no-bias-dropout-fusion',
- '--no-async-tensor-model-parallel-allreduce',
'--use-cpu-initialization',
'--micro-batch-size', '1',
'--no-load-optim',
@@ -248,7 +246,6 @@ def check_for_arg(arg_name, default=None):
fake_ep_group = _ConverterFakeProcessGroup(size=margs.expert_model_parallel_size)
mpu._TENSOR_MODEL_PARALLEL_GROUP = fake_tp_group
mpu._EXPERT_MODEL_PARALLEL_GROUP = fake_ep_group
- fused_kernels.load(margs)
# Metadata.
md = types.SimpleNamespace()
diff --git a/tools/checkpoint/saver_base.py b/tools/checkpoint/saver_base.py
index 4958dc99ed7..8f60d0515db 100644
--- a/tools/checkpoint/saver_base.py
+++ b/tools/checkpoint/saver_base.py
@@ -62,7 +62,7 @@ def _load_checkpoint_args(self, margs):
args_to_keep = ['tensor_model_parallel_size', 'pipeline_model_parallel_size', 'expert_model_parallel_size', 'world_size', 'params_dtype',
'num_layers_per_virtual_pipeline_stage', 'virtual_pipeline_model_parallel_size',
'masked_softmax_fusion', 'bias_gelu_fusion', 'bias_dropout_fusion',
- 'sequence_parallel', 'async_tensor_model_parallel_allreduce',
+ 'sequence_parallel',
'no_load_optim', 'no_load_rng', 'no_save_optim', 'no_save_rng',
'vocab_file', 'tokenizer_model',
'save_interval', 'save',
@@ -140,7 +140,6 @@ def initialize_megatron_env(self):
try:
from megatron.training.global_vars import set_global_variables, get_args
from megatron.core import mpu
- from megatron.legacy import fused_kernels
except ModuleNotFoundError as e:
print(f"Unable to import required Megatron modules: {e}")
sys.exit(1)
@@ -160,6 +159,14 @@ def initialize_megatron_env(self):
self.import_model_provider()
+ # Initialize torch.distributed with a minimal single-process backend so that
+ # process-group size queries (get_pg_size / get_tensor_model_parallel_group_if_none)
+ # return the fake groups below rather than falling back to world_size=1.
+ if not torch.distributed.is_initialized():
+ os.environ.setdefault('MASTER_ADDR', 'localhost')
+ os.environ.setdefault('MASTER_PORT', '12356')
+ torch.distributed.init_process_group(backend='gloo', rank=0, world_size=1)
+
# fake initializing distributed
mpu.set_tensor_model_parallel_world_size(self.args.target_tensor_parallel_size)
mpu.set_pipeline_model_parallel_world_size(self.args.target_pipeline_parallel_size)
@@ -170,10 +177,17 @@ def initialize_megatron_env(self):
# For backward compatibility during local parallel states refactoring
fake_tp_group = _ConverterFakeProcessGroup(size=self.args.target_tensor_parallel_size)
+ fake_pp_group = _ConverterFakeProcessGroup(size=self.args.target_pipeline_parallel_size)
fake_ep_group = _ConverterFakeProcessGroup(size=self.args.target_expert_parallel_size)
+ fake_dp_group = _ConverterFakeProcessGroup(size=1)
+ fake_dp_ep_group = _ConverterFakeProcessGroup(size=1)
mpu._TENSOR_MODEL_PARALLEL_GROUP = fake_tp_group
+ mpu._PIPELINE_MODEL_PARALLEL_GROUP = fake_pp_group
mpu._EXPERT_MODEL_PARALLEL_GROUP = fake_ep_group
- fused_kernels.load(self.margs)
+ mpu._DATA_PARALLEL_GROUP = fake_dp_group
+ mpu._DATA_PARALLEL_GROUP_WITH_CP = fake_dp_group
+ mpu._INTRA_PARTIAL_DATA_PARALLEL_GROUP_WITH_CP = fake_dp_group
+ mpu._EXPERT_DATA_PARALLEL_GROUP = fake_dp_ep_group
try:
import torch_llm_debug_tools
@@ -232,7 +246,6 @@ def build_sys_argv(self):
'--no-masked-softmax-fusion',
'--no-bias-gelu-fusion',
'--no-bias-dropout-fusion',
- '--no-async-tensor-model-parallel-allreduce',
'--use-cpu-initialization',
'--micro-batch-size', '1',
'--no-load-optim',
@@ -356,7 +369,7 @@ def receive_lm(self, schema, prefix=None):
"""
try:
from megatron.core import mpu
- from megatron.training.tokenizer.tokenizer import _vocab_size_with_padding
+ from megatron.core.tokenizers.utils.build_tokenizer import vocab_size_with_padding
except ModuleNotFoundError as e:
print(f"Unable to import required Megatron modules: {e}")
sys.exit(1)
@@ -375,7 +388,7 @@ def pad_weight(orig_word_embed, true_vocab_size):
if true_vocab_size is not None:
# figure out what our padded vocab size is
orig_vocab_size = orig_word_embed.shape[0]
- self.margs.padded_vocab_size = _vocab_size_with_padding(true_vocab_size, self.margs)
+ self.margs.padded_vocab_size = vocab_size_with_padding(true_vocab_size, self.margs)
# Cut out extra padding we don't need
if orig_vocab_size > self.margs.padded_vocab_size:
diff --git a/tools/checkpoint/saver_legacy.py b/tools/checkpoint/saver_legacy.py
index 0e582bccf89..e0e79dba3f4 100644
--- a/tools/checkpoint/saver_legacy.py
+++ b/tools/checkpoint/saver_legacy.py
@@ -39,8 +39,7 @@ def save_checkpoint(queue, args):
from megatron.training.checkpointing import save_checkpoint
from megatron.training.global_vars import set_global_variables, get_args
from megatron.core.enums import ModelType
- from megatron.training.tokenizer.tokenizer import _vocab_size_with_padding
- from megatron.legacy import fused_kernels
+ from megatron.core.tokenizers.utils.build_tokenizer import vocab_size_with_padding
from megatron.core import mpu
except ModuleNotFoundError:
print("Unable to import Megatron, please specify the path to Megatron using --megatron-path. Exiting.")
@@ -109,7 +108,6 @@ def check_message(msg):
'--no-masked-softmax-fusion',
'--no-bias-gelu-fusion',
'--no-bias-dropout-fusion',
- '--no-async-tensor-model-parallel-allreduce',
'--use-cpu-initialization',
'--micro-batch-size', '1',
'--no-load-optim',
@@ -146,7 +144,7 @@ def check_message(msg):
args_to_keep = ['tensor_model_parallel_size', 'pipeline_model_parallel_size', 'world_size', 'params_dtype',
'num_layers_per_virtual_pipeline_stage', 'virtual_pipeline_model_parallel_size',
'masked_softmax_fusion', 'bias_gelu_fusion', 'bias_dropout_fusion',
- 'sequence_parallel', 'async_tensor_model_parallel_allreduce',
+ 'sequence_parallel',
'no_load_optim', 'no_load_rng', 'no_save_optim', 'no_save_rng',
'vocab_file', 'tokenizer_model',
'save_interval', 'save',
@@ -214,7 +212,6 @@ def get_models(count, dtype, pre_process, post_process):
mpu.set_pipeline_model_parallel_world_size(args.target_pipeline_parallel_size)
mpu.set_tensor_model_parallel_rank(0)
mpu.set_pipeline_model_parallel_rank(0)
- fused_kernels.load(margs)
# For backward compatibility during local parallel states refactoring
fake_tp_group = _ConverterFakeProcessGroup(size=args.target_tensor_parallel_size)
@@ -234,7 +231,7 @@ def get_models(count, dtype, pre_process, post_process):
if md.true_vocab_size is not None:
# figure out what our padded vocab size is
orig_vocab_size = orig_word_embed.shape[0]
- margs.padded_vocab_size = _vocab_size_with_padding(md.true_vocab_size, margs)
+ margs.padded_vocab_size = vocab_size_with_padding(md.true_vocab_size, margs)
# Cut out extra padding we don't need
if orig_vocab_size > margs.padded_vocab_size:
diff --git a/tools/checkpoint/saver_llava.py b/tools/checkpoint/saver_llava.py
index 3b7f7891a9f..326e3ea665c 100644
--- a/tools/checkpoint/saver_llava.py
+++ b/tools/checkpoint/saver_llava.py
@@ -57,7 +57,7 @@ def _load_checkpoint_args(self, margs):
args_to_keep = ['tensor_model_parallel_size', 'pipeline_model_parallel_size', 'expert_model_parallel_size', 'world_size', 'params_dtype',
'num_layers_per_virtual_pipeline_stage', 'virtual_pipeline_model_parallel_size',
'masked_softmax_fusion', 'bias_gelu_fusion', 'bias_dropout_fusion',
- 'sequence_parallel', 'async_tensor_model_parallel_allreduce',
+ 'sequence_parallel',
'no_load_optim', 'no_load_rng', 'no_save_optim', 'no_save_rng',
'vocab_file',
'save_interval', 'save',
@@ -99,7 +99,6 @@ def build_sys_argv(self):
'--no-masked-softmax-fusion',
'--no-bias-gelu-fusion',
'--no-bias-dropout-fusion',
- '--no-async-tensor-model-parallel-allreduce',
'--use-cpu-initialization',
'--micro-batch-size', '1',
'--no-load-optim',
diff --git a/tools/common_pile_dataset/README.md b/tools/common_pile_dataset/README.md
new file mode 100644
index 00000000000..2431d1b01d3
--- /dev/null
+++ b/tools/common_pile_dataset/README.md
@@ -0,0 +1,223 @@
+# Common Pile CI Dataset
+
+This directory contains tools to create CI test datasets from the
+[Common Pile](https://huggingface.co/datasets/common-pile/comma_v0.1_training_dataset)
+filtered dataset, replacing the previous datasets sourced from The Pile.
+
+## Output
+
+The scripts produce Megatron-LM indexed binary datasets for three model families:
+
+```
+/
+├── my-gpt3_00_text_document.{bin,idx} # GPT (~24 GB bin)
+├── my-bert_00_text_sentence.{bin,idx} # BERT (~25 GB bin)
+├── my-t5_00_text_document.{bin,idx} # T5 (~25 GB bin)
+├── bpe/
+│ ├── vocab.json # GPT-2 BPE vocabulary
+│ └── merges.txt # GPT-2 BPE merges
+├── vocab.txt # BERT WordPiece vocabulary
+└── bert-large-cased-vocab.txt # T5 (BERT-large cased) vocabulary
+```
+
+Current production location on the HPC cluster:
+
+```
+/lustre/fsw/portfolios/coreai/projects/coreai_dlalgo_mcore/mcore_ci/text/common_pile/v01_filtered_data/
+```
+
+## Quick Start (HPC)
+
+The fastest way to recreate the dataset on the NVIDIA HPC cluster:
+
+```bash
+# 1. Copy both scripts to the remote machine
+scp tools/common_pile_dataset/setup_common_pile_dataset.sh tools/common_pile_dataset/create_common_pile_ci_dataset.py \
+ @:/tmp/
+
+# 2. Launch as a background job (survives SSH disconnection)
+ssh @ \
+ 'nohup bash /tmp/setup_common_pile_dataset.sh > /tmp/dataset_creation.log 2>&1 &'
+
+# 3. Monitor progress
+ssh @ 'tail -f /tmp/dataset_creation.log'
+```
+
+Total runtime is approximately **3-5 hours** (40 min download + ~1 hr per
+preprocessing step).
+
+## What the Setup Script Does
+
+`setup_common_pile_dataset.sh` is a self-contained wrapper that:
+
+1. **Finds Python 3.10+** (required by latest Megatron-LM for PEP 604 syntax).
+2. **Creates a virtual environment** in `/tmp` to isolate from system packages.
+3. **Clones Megatron-LM** (shallow, depth 1) for `preprocess_data.py`.
+4. **Patches `megatron/training/__init__.py`** to skip heavy imports (triton,
+ apex, transformer-engine) that are not needed for preprocessing.
+5. **Installs pip dependencies**: `datasets`, `nltk`, `torch` (CPU-only),
+ `transformers`.
+6. **Redirects HuggingFace cache** to lustre (`HF_HOME`) to avoid filling up
+ the 10 GB `/home` filesystem.
+7. **Runs `create_common_pile_ci_dataset.py`** with production parameters.
+8. **Cleans up** the temporary work directory.
+
+## Running the Python Script Directly
+
+If you already have a Megatron-LM checkout and the dependencies installed, you
+can run the Python script directly:
+
+```bash
+# Small test run (streaming, ~10K docs, a few minutes)
+python tools/common_pile_dataset/create_common_pile_ci_dataset.py \
+ --output-dir /path/to/output \
+ --num-documents 10000 \
+ --download-vocab
+
+# Full production run (~12M docs, matching existing shard00 sizes)
+python tools/common_pile_dataset/create_common_pile_ci_dataset.py \
+ --output-dir /path/to/output \
+ --num-documents 12000000 \
+ --keep-jsonl \
+ --copy-vocab-from /lustre/.../text/the_pile
+```
+
+### Key Arguments
+
+| Argument | Description |
+|---|---|
+| `--output-dir` | Where to write the output files (required) |
+| `--num-documents` | Number of documents to download (default: 10000) |
+| `--megatron-dir` | Path to Megatron-LM repo (default: auto-detect from script location) |
+| `--copy-vocab-from` | Copy vocab files from existing `the_pile` directory |
+| `--download-vocab` | Download vocab files from HuggingFace instead |
+| `--existing-jsonl` | Skip download; use a pre-existing JSONL file |
+| `--keep-jsonl` | Keep the intermediate JSONL after preprocessing |
+| `--bulk-download` | Non-streaming download (faster but requires ~460 GB HF cache) |
+| `--workers` | Number of worker processes for preprocessing (default: 4) |
+
+## Prerequisites
+
+- **Python 3.10+** (Megatron-LM uses PEP 604 `type | None` syntax)
+- **PyTorch** (CPU-only is sufficient for preprocessing)
+- **Python packages**: `datasets`, `nltk`, `transformers`
+- **Disk space**:
+ - Output directory: ~80 GB for final files
+ - Intermediate JSONL: ~43 GB (deleted unless `--keep-jsonl`)
+ - Sentence-split JSONL: ~43 GB (created during BERT preprocessing)
+ - HF cache: ~1 GB (streaming mode) or ~460 GB (`--bulk-download` mode)
+- **RAM**: ~18 GB peak (during BERT index finalization)
+
+## How It Works
+
+### Step 1: Vocabulary Files
+
+Copies tokenizer vocabularies from the existing `the_pile` dataset, or
+downloads them from HuggingFace. These are standard GPT-2 BPE and BERT
+WordPiece vocabularies.
+
+### Step 2: Download Raw Text
+
+Streams documents from `common-pile/comma_v0.1_training_dataset` on
+HuggingFace, filtering out documents shorter than 100 characters. Writes a
+JSONL file with `{"text": "..."}` per line. At 12M documents this produces a
+~43 GB file.
+
+Progress is logged with ETA:
+
+```
+[ 42.0%] 5,040,000/12,000,000 docs | 4,500 docs/s | 20,150.3 MB on disk | ETA: 25.8m
+```
+
+### Step 3: GPT Preprocessing
+
+Runs `preprocess_data.py` with `GPT2BPETokenizer` and `--append-eod` to create
+`my-gpt3_00_text_document.{bin,idx}`.
+
+### Step 4: BERT Preprocessing
+
+BERT requires sentence splitting, which is a **two-pass process** when using
+`partitions=1` (the default):
+
+1. **Pass 1**: Runs with `--split-sentences` to create a sentence-split JSONL
+ (`common_pile_raw_ss.jsonl`), then returns.
+2. **Pass 2**: Detects the `_ss.jsonl` file exists, skips splitting, and
+ encodes to binary `my-bert_00_text_sentence.{bin,idx}`.
+
+The BERT `.idx` file is much larger (~4 GB vs ~229 MB for GPT/T5) because it
+indexes individual sentences rather than whole documents.
+
+### Step 5: T5 Preprocessing
+
+Runs `preprocess_data.py` with `BertWordPieceCase` tokenizer and `--append-eod`
+to create `my-t5_00_text_document.{bin,idx}`.
+
+### Step 6: Verification
+
+Checks that all 10 expected output files exist and reports their sizes.
+
+## Troubleshooting
+
+### "No space left on device" during download
+
+The HuggingFace `datasets` library caches data under `~/.cache/huggingface/` by
+default. On HPC systems where `/home` is small, set `HF_HOME` to a path with
+sufficient space:
+
+```bash
+export HF_HOME=/lustre/path/to/.hf_cache
+```
+
+The setup script does this automatically.
+
+### "ModuleNotFoundError: No module named 'triton'" (or similar)
+
+`preprocess_data.py` imports from `megatron.training`, which eagerly loads the
+full training stack. The setup script patches `megatron/training/__init__.py` to
+comment out the heavy imports. If running manually, apply:
+
+```bash
+sed -i 's/^from \.initialize/#from .initialize/' megatron/training/__init__.py
+```
+
+### "TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'"
+
+You need Python 3.10+. The latest Megatron-LM uses PEP 604 union syntax
+(`type | None`) which is not supported in Python 3.9.
+
+### numpy/scipy binary incompatibility
+
+Use a virtual environment (`python3.10 -m venv venv`) to isolate from system
+packages. The setup script creates one automatically.
+
+### BERT produces no output files
+
+BERT with `--split-sentences` and `partitions=1` requires two invocations of
+`preprocess_data.py` (see Step 4 above). The script handles this automatically.
+
+## CI Data Path Configuration
+
+To use this dataset in CI tests, set the following paths in `model_config.yaml`:
+
+```yaml
+# GPT
+--data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document
+--vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json
+--merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt
+
+# BERT
+--data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-bert_00_text_sentence
+--vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/vocab.txt
+
+# T5
+--data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-t5_00_text_document
+--vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bert-large-cased-vocab.txt
+```
+
+## Files
+
+| File | Description |
+|---|---|
+| `create_common_pile_ci_dataset.py` | Main Python script that downloads data and runs preprocessing |
+| `setup_common_pile_dataset.sh` | Self-contained bash wrapper for HPC deployment |
+| `README.md` | This file |
diff --git a/tools/common_pile_dataset/create_common_pile_ci_dataset.py b/tools/common_pile_dataset/create_common_pile_ci_dataset.py
new file mode 100644
index 00000000000..0e8d8c282c6
--- /dev/null
+++ b/tools/common_pile_dataset/create_common_pile_ci_dataset.py
@@ -0,0 +1,531 @@
+#!/usr/bin/env python3
+# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.
+
+"""
+Create CI test dataset from Common Pile (filtered).
+
+Downloads a sample from the Common Pile filtered dataset and preprocesses it
+into Megatron-LM's indexed binary format for GPT, BERT, and T5 models.
+
+Output structure:
+ /
+ ├── my-gpt3_00_text_document.{bin,idx}
+ ├── bpe/
+ │ ├── vocab.json
+ │ └── merges.txt
+ ├── my-bert_00_text_sentence.{bin,idx}
+ ├── vocab.txt
+ ├── my-t5_00_text_document.{bin,idx}
+ └── bert-large-cased-vocab.txt
+
+Usage:
+ # Small test run (streaming):
+ python tools/common_pile_dataset/create_common_pile_ci_dataset.py \
+ --output-dir /path/to/output \
+ --num-documents 10000 \
+ --copy-vocab-from /path/to/existing/the_pile
+
+ # Large production run (~24GB, matching existing shard00):
+ python tools/common_pile_dataset/create_common_pile_ci_dataset.py \
+ --output-dir /path/to/output \
+ --num-documents 12000000 \
+ --bulk-download --keep-jsonl \
+ --copy-vocab-from /path/to/existing/the_pile
+
+ # With vocab download (no existing dataset needed):
+ python tools/common_pile_dataset/create_common_pile_ci_dataset.py \
+ --output-dir /path/to/output \
+ --num-documents 10000 \
+ --download-vocab
+"""
+
+import argparse
+import json
+import os
+import subprocess
+import sys
+import time
+import urllib.request
+
+
+def _format_eta(seconds):
+ """Format seconds into a human-readable ETA string."""
+ if seconds < 60:
+ return f"{seconds:.0f}s"
+ elif seconds < 3600:
+ return f"{seconds / 60:.1f}m"
+ else:
+ h = int(seconds // 3600)
+ m = int((seconds % 3600) // 60)
+ return f"{h}h{m:02d}m"
+
+
+def download_common_pile_sample(output_jsonl, num_documents, dataset_name):
+ """Download a sample from Common Pile using the datasets library."""
+ try:
+ from datasets import load_dataset
+ except ImportError:
+ print("ERROR: 'datasets' library not found. Install with: pip install datasets")
+ sys.exit(1)
+
+ print(f"Downloading {num_documents} documents from {dataset_name}...")
+ print(" (Using HF_TOKEN from environment for authentication if set)")
+
+ ds = load_dataset(dataset_name, split="train", streaming=True)
+
+ count = 0
+ start_time = time.time()
+ log_interval = max(1000, num_documents // 100) # Log ~100 times, min every 1000
+
+ with open(output_jsonl, 'w', encoding='utf-8') as f:
+ for sample in ds:
+ text = sample.get("text", "")
+ # Skip very short documents that wouldn't be useful for CI tests
+ if text and len(text.strip()) > 100:
+ f.write(json.dumps({"text": text}) + "\n")
+ count += 1
+ if count >= num_documents:
+ break
+ if count % log_interval == 0:
+ elapsed = time.time() - start_time
+ rate = count / elapsed if elapsed > 0 else 0
+ remaining = (num_documents - count) / rate if rate > 0 else 0
+ pct = count / num_documents * 100
+ file_size_mb = os.path.getsize(output_jsonl) / (1024 * 1024)
+ print(
+ f" [{pct:5.1f}%] {count:,}/{num_documents:,} docs | "
+ f"{rate:,.0f} docs/s | "
+ f"{file_size_mb:,.1f} MB on disk | "
+ f"ETA: {_format_eta(remaining)}"
+ )
+
+ elapsed = time.time() - start_time
+ print(f" Saved {count:,} documents to {output_jsonl} in {_format_eta(elapsed)}")
+ file_size_mb = os.path.getsize(output_jsonl) / (1024 * 1024)
+ print(f" JSONL file size: {file_size_mb:,.2f} MB")
+ return count
+
+
+def download_common_pile_bulk(output_jsonl, num_documents, dataset_name):
+ """Download from Common Pile using bulk parquet loading for speed.
+
+ This is faster than streaming for large downloads (>100K docs) because
+ it downloads full parquet files and processes them locally.
+ """
+ try:
+ from datasets import load_dataset
+ except ImportError:
+ print("ERROR: 'datasets' library not found. Install with: pip install datasets")
+ sys.exit(1)
+
+ print(f"Bulk downloading {num_documents:,} documents from {dataset_name}...")
+ print(" (Using non-streaming mode for faster throughput)")
+ print(" (Using HF_TOKEN from environment for authentication if set)")
+
+ # Load non-streaming — this downloads parquet shards to the HF cache
+ print(" Loading dataset (downloading parquet shards)...")
+ ds = load_dataset(dataset_name, split="train")
+ total_available = len(ds)
+ print(f" Dataset loaded: {total_available:,} documents available")
+
+ count = 0
+ skipped = 0
+ start_time = time.time()
+ log_interval = max(1000, num_documents // 100)
+
+ with open(output_jsonl, 'w', encoding='utf-8') as f:
+ for i in range(min(total_available, num_documents + num_documents // 10)):
+ text = ds[i].get("text", "")
+ if text and len(text.strip()) > 100:
+ f.write(json.dumps({"text": text}) + "\n")
+ count += 1
+ if count >= num_documents:
+ break
+ if count % log_interval == 0:
+ elapsed = time.time() - start_time
+ rate = count / elapsed if elapsed > 0 else 0
+ remaining = (num_documents - count) / rate if rate > 0 else 0
+ pct = count / num_documents * 100
+ file_size_mb = os.path.getsize(output_jsonl) / (1024 * 1024)
+ print(
+ f" [{pct:5.1f}%] {count:,}/{num_documents:,} docs | "
+ f"{rate:,.0f} docs/s | "
+ f"{file_size_mb:,.1f} MB on disk | "
+ f"ETA: {_format_eta(remaining)}"
+ )
+ else:
+ skipped += 1
+
+ elapsed = time.time() - start_time
+ print(
+ f" Saved {count:,} documents to {output_jsonl} in {_format_eta(elapsed)} "
+ f"({skipped:,} short docs skipped)"
+ )
+ file_size_mb = os.path.getsize(output_jsonl) / (1024 * 1024)
+ print(f" JSONL file size: {file_size_mb:,.2f} MB")
+ return count
+
+
+def copy_vocab_files(output_dir, source_base):
+ """Copy vocabulary files from existing the_pile dataset directories."""
+ copies = [
+ (
+ os.path.join(source_base, "shard00", "bpe", "vocab.json"),
+ os.path.join(output_dir, "bpe", "vocab.json"),
+ ),
+ (
+ os.path.join(source_base, "shard00", "bpe", "merges.txt"),
+ os.path.join(output_dir, "bpe", "merges.txt"),
+ ),
+ (
+ os.path.join(source_base, "bert_shard00", "vocab.txt"),
+ os.path.join(output_dir, "vocab.txt"),
+ ),
+ (
+ os.path.join(source_base, "t5_shard00", "bert-large-cased-vocab.txt"),
+ os.path.join(output_dir, "bert-large-cased-vocab.txt"),
+ ),
+ ]
+
+ for src, dst in copies:
+ os.makedirs(os.path.dirname(dst), exist_ok=True)
+ if not os.path.exists(src):
+ print(f" ERROR: Source vocab file not found: {src}")
+ sys.exit(1)
+ if os.path.exists(dst):
+ print(f" Already exists: {dst}")
+ continue
+ print(f" Copying {src} -> {dst}")
+ with open(src, 'rb') as f_in, open(dst, 'wb') as f_out:
+ f_out.write(f_in.read())
+
+
+def download_vocab_files(output_dir):
+ """Download tokenizer vocabulary files from HuggingFace."""
+ downloads = [
+ (
+ "https://huggingface.co/openai-community/gpt2/resolve/main/vocab.json",
+ os.path.join(output_dir, "bpe", "vocab.json"),
+ ),
+ (
+ "https://huggingface.co/openai-community/gpt2/resolve/main/merges.txt",
+ os.path.join(output_dir, "bpe", "merges.txt"),
+ ),
+ (
+ "https://huggingface.co/google-bert/bert-base-uncased/resolve/main/vocab.txt",
+ os.path.join(output_dir, "vocab.txt"),
+ ),
+ (
+ "https://huggingface.co/google-bert/bert-large-cased/resolve/main/vocab.txt",
+ os.path.join(output_dir, "bert-large-cased-vocab.txt"),
+ ),
+ ]
+
+ for url, dst in downloads:
+ os.makedirs(os.path.dirname(dst), exist_ok=True)
+ if os.path.exists(dst):
+ print(f" Already exists: {dst}")
+ continue
+ print(f" Downloading {url}")
+ print(f" -> {dst}")
+ hf_token = os.environ.get("HF_TOKEN", "")
+ req = urllib.request.Request(url)
+ if hf_token:
+ req.add_header("Authorization", f"Bearer {hf_token}")
+ with urllib.request.urlopen(req) as response, open(dst, 'wb') as f_out:
+ f_out.write(response.read())
+
+
+def run_preprocess(megatron_dir, jsonl_path, output_prefix, tokenizer_type,
+ vocab_file, merge_file=None, split_sentences=False,
+ append_eod=False, workers=4):
+ """Run preprocess_data.py to create .bin/.idx files."""
+ cmd = [
+ sys.executable,
+ os.path.join(megatron_dir, "tools", "preprocess_data.py"),
+ "--input", jsonl_path,
+ "--output-prefix", output_prefix,
+ "--tokenizer-type", tokenizer_type,
+ "--vocab-file", vocab_file,
+ "--workers", str(workers),
+ ]
+
+ if merge_file:
+ cmd.extend(["--merge-file", merge_file])
+ if split_sentences:
+ cmd.append("--split-sentences")
+ if append_eod:
+ cmd.append("--append-eod")
+
+ print(f"\n Running: {' '.join(cmd)}")
+ result = subprocess.run(cmd, cwd=megatron_dir)
+ if result.returncode != 0:
+ print(f" ERROR: Preprocessing failed with return code {result.returncode}")
+ sys.exit(1)
+
+
+def verify_output(output_dir):
+ """Verify all expected output files exist."""
+ expected_files = [
+ "my-gpt3_00_text_document.bin",
+ "my-gpt3_00_text_document.idx",
+ "my-bert_00_text_sentence.bin",
+ "my-bert_00_text_sentence.idx",
+ "my-t5_00_text_document.bin",
+ "my-t5_00_text_document.idx",
+ "bpe/vocab.json",
+ "bpe/merges.txt",
+ "vocab.txt",
+ "bert-large-cased-vocab.txt",
+ ]
+
+ all_ok = True
+ for f in expected_files:
+ full_path = os.path.join(output_dir, f)
+ if os.path.exists(full_path):
+ size_mb = os.path.getsize(full_path) / (1024 * 1024)
+ print(f" OK: {f} ({size_mb:.2f} MB)")
+ else:
+ print(f" MISSING: {f}")
+ all_ok = False
+
+ return all_ok
+
+
+def main():
+ parser = argparse.ArgumentParser(
+ description="Create CI test dataset from Common Pile"
+ )
+ parser.add_argument(
+ "--output-dir", type=str, required=True,
+ help="Output directory for preprocessed data",
+ )
+ parser.add_argument(
+ "--megatron-dir", type=str, default=None,
+ help="Path to Megatron-LM repo root (default: auto-detect from script location)",
+ )
+ parser.add_argument(
+ "--num-documents", type=int, default=10000,
+ help="Number of documents to download (default: 10000)",
+ )
+ parser.add_argument(
+ "--dataset-name", type=str, default="common-pile/comma_v0.1_training_dataset",
+ help="HuggingFace dataset name (default: common-pile/comma_v0.1_training_dataset)",
+ )
+ parser.add_argument(
+ "--copy-vocab-from", type=str, default=None,
+ help="Copy vocab files from existing the_pile base directory "
+ "(e.g., /lustre/.../text/the_pile)",
+ )
+ parser.add_argument(
+ "--download-vocab", action="store_true",
+ help="Download vocab files from HuggingFace instead of copying",
+ )
+ parser.add_argument(
+ "--existing-jsonl", type=str, default=None,
+ help="Path to existing JSONL file (skip download step)",
+ )
+ parser.add_argument(
+ "--workers", type=int, default=4,
+ help="Number of worker processes for preprocessing (default: 4)",
+ )
+ parser.add_argument(
+ "--keep-jsonl", action="store_true",
+ help="Keep the intermediate JSONL file after preprocessing",
+ )
+ parser.add_argument(
+ "--bulk-download", action="store_true",
+ help="Use bulk (non-streaming) download for faster throughput at scale. "
+ "Downloads full parquet shards to HF cache before writing JSONL. "
+ "Recommended for --num-documents > 100000.",
+ )
+ args = parser.parse_args()
+
+ # Auto-detect Megatron-LM directory
+ if args.megatron_dir is None:
+ args.megatron_dir = os.path.abspath(
+ os.path.join(os.path.dirname(__file__), os.path.pardir, os.path.pardir)
+ )
+ print(f"Megatron-LM directory: {args.megatron_dir}")
+ print(f"Output directory: {args.output_dir}")
+
+ # Verify megatron dir has preprocess_data.py
+ preprocess_script = os.path.join(args.megatron_dir, "tools", "preprocess_data.py")
+ if not os.path.exists(preprocess_script):
+ print(f"ERROR: preprocess_data.py not found at {preprocess_script}")
+ sys.exit(1)
+
+ os.makedirs(args.output_dir, exist_ok=True)
+
+ # ================================================================
+ # Step 1: Get vocabulary files
+ # ================================================================
+ print("\n" + "=" * 60)
+ print("Step 1: Setting up vocabulary files...")
+ print("=" * 60)
+
+ if args.copy_vocab_from:
+ print(f" Copying from: {args.copy_vocab_from}")
+ copy_vocab_files(args.output_dir, args.copy_vocab_from)
+ elif args.download_vocab:
+ print(" Downloading from HuggingFace...")
+ download_vocab_files(args.output_dir)
+ else:
+ # Default: try to copy from standard CI location, fall back to download
+ default_source = (
+ "/lustre/fsw/portfolios/coreai/projects/coreai_dlalgo_mcore"
+ "/mcore_ci/text/the_pile"
+ )
+ if os.path.exists(default_source):
+ print(f" Copying from default location: {default_source}")
+ copy_vocab_files(args.output_dir, default_source)
+ else:
+ print(" Default vocab source not found, downloading from HuggingFace...")
+ download_vocab_files(args.output_dir)
+
+ # ================================================================
+ # Step 2: Get raw text data
+ # ================================================================
+ print("\n" + "=" * 60)
+ print("Step 2: Preparing raw text data...")
+ print("=" * 60)
+
+ if args.existing_jsonl:
+ jsonl_path = args.existing_jsonl
+ print(f" Using existing JSONL: {jsonl_path}")
+ else:
+ jsonl_path = os.path.join(args.output_dir, "common_pile_raw.jsonl")
+ if os.path.exists(jsonl_path):
+ print(f" JSONL already exists: {jsonl_path}")
+ print(" (Delete it to re-download)")
+ elif args.bulk_download:
+ download_common_pile_bulk(
+ jsonl_path, args.num_documents, args.dataset_name
+ )
+ else:
+ download_common_pile_sample(
+ jsonl_path, args.num_documents, args.dataset_name
+ )
+
+ # ================================================================
+ # Step 3: Preprocess for GPT (GPT2BPETokenizer)
+ # ================================================================
+ print("\n" + "=" * 60)
+ print("Step 3: Preprocessing for GPT (GPT2BPETokenizer)...")
+ print("=" * 60)
+
+ gpt_prefix = os.path.join(args.output_dir, "my-gpt3_00")
+ gpt_bin = gpt_prefix + "_text_document.bin"
+ if os.path.exists(gpt_bin):
+ print(f" GPT data already exists: {gpt_bin}")
+ else:
+ run_preprocess(
+ megatron_dir=args.megatron_dir,
+ jsonl_path=jsonl_path,
+ output_prefix=gpt_prefix,
+ tokenizer_type="GPT2BPETokenizer",
+ vocab_file=os.path.join(args.output_dir, "bpe", "vocab.json"),
+ merge_file=os.path.join(args.output_dir, "bpe", "merges.txt"),
+ append_eod=True,
+ workers=args.workers,
+ )
+
+ # ================================================================
+ # Step 4: Preprocess for BERT (BertWordPieceLowerCase + split-sentences)
+ # ================================================================
+ print("\n" + "=" * 60)
+ print("Step 4: Preprocessing for BERT (BertWordPieceLowerCase)...")
+ print("=" * 60)
+
+ bert_prefix = os.path.join(args.output_dir, "my-bert_00")
+ bert_bin = bert_prefix + "_text_sentence.bin"
+ if os.path.exists(bert_bin):
+ print(f" BERT data already exists: {bert_bin}")
+ else:
+ # BERT with --split-sentences requires two passes when partitions=1:
+ # Pass 1: splits sentences, creates @:/tmp/
+# ssh @ 'bash /tmp/setup_common_pile_dataset.sh'
+
+set -euo pipefail
+
+# Use Python 3.10+ (required by latest Megatron-LM for PEP 604 type syntax)
+PYTHON="/usr/bin/python3.10"
+if [ ! -x "${PYTHON}" ]; then
+ echo "ERROR: Python 3.10+ required but ${PYTHON} not found"
+ exit 1
+fi
+echo "Using Python: ${PYTHON} ($("${PYTHON}" --version))"
+
+OUTPUT_DIR="/lustre/fsw/portfolios/coreai/projects/coreai_dlalgo_mcore/mcore_ci/text/common_pile/v01_filtered_data"
+EXISTING_VOCAB="/lustre/fsw/portfolios/coreai/projects/coreai_dlalgo_mcore/mcore_ci/text/the_pile"
+NUM_DOCUMENTS=12000000
+DATASET_NAME="common-pile/comma_v0.1_training_dataset"
+WORK_DIR="/tmp/mcore_dataset_setup_$$"
+
+# Redirect HuggingFace cache to lustre so it doesn't fill up /home
+export HF_HOME="/lustre/fsw/portfolios/coreai/projects/coreai_dlalgo_mcore/mcore_ci/.hf_cache"
+export HF_DATASETS_CACHE="${HF_HOME}/datasets"
+
+echo "============================================================"
+echo "Common Pile CI Dataset Setup"
+echo "============================================================"
+echo "Output: ${OUTPUT_DIR}"
+echo "Vocab from: ${EXISTING_VOCAB}"
+echo "Documents: ${NUM_DOCUMENTS}"
+echo "Dataset: ${DATASET_NAME}"
+echo "Work dir: ${WORK_DIR}"
+echo "HF cache: ${HF_HOME}"
+echo "============================================================"
+
+# Create work directory
+mkdir -p "${WORK_DIR}"
+cd "${WORK_DIR}"
+
+# Check if create_common_pile_ci_dataset.py was scp'd alongside this script
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+if [ -f "${SCRIPT_DIR}/create_common_pile_ci_dataset.py" ]; then
+ DATASET_SCRIPT="${SCRIPT_DIR}/create_common_pile_ci_dataset.py"
+ echo "Found dataset script at: ${DATASET_SCRIPT}"
+else
+ DATASET_SCRIPT=""
+fi
+
+# Clone Megatron-LM — full shallow clone needed because preprocess_data.py
+# imports from megatron.core which pulls in tensor_parallel and other submodules
+MEGATRON_DIR="${WORK_DIR}/Megatron-LM"
+if [ ! -d "${MEGATRON_DIR}" ]; then
+ echo ""
+ echo "Cloning Megatron-LM repository (full shallow clone)..."
+ git clone --depth 1 https://github.com/NVIDIA/Megatron-LM.git "${MEGATRON_DIR}" 2>&1
+fi
+
+# Patch megatron/training/__init__.py to avoid importing the full training stack.
+# preprocess_data.py only needs _add_tokenizer_args from arguments.py, but the
+# __init__.py eagerly imports initialize_megatron which pulls in triton, apex, etc.
+TRAINING_INIT="${MEGATRON_DIR}/megatron/training/__init__.py"
+if [ -f "${TRAINING_INIT}" ] && grep -q "from .initialize" "${TRAINING_INIT}"; then
+ echo "Patching megatron/training/__init__.py to skip heavy imports..."
+ sed -i 's/^from \.initialize/#from .initialize/' "${TRAINING_INIT}"
+ sed -i 's/^from \.training/#from .training/' "${TRAINING_INIT}"
+fi
+
+# Use the script from the repo if we didn't have it locally
+if [ -z "${DATASET_SCRIPT}" ]; then
+ DATASET_SCRIPT="${MEGATRON_DIR}/tools/common_pile_dataset/create_common_pile_ci_dataset.py"
+ if [ ! -f "${DATASET_SCRIPT}" ]; then
+ echo "ERROR: create_common_pile_ci_dataset.py not found"
+ exit 1
+ fi
+fi
+
+# Create a virtual environment to avoid system package conflicts
+VENV_DIR="${WORK_DIR}/venv"
+if [ ! -d "${VENV_DIR}" ]; then
+ echo ""
+ echo "Creating virtual environment..."
+ "${PYTHON}" -m venv "${VENV_DIR}"
+fi
+source "${VENV_DIR}/bin/activate"
+PYTHON="$(which python)"
+echo "Using venv Python: ${PYTHON} ($(${PYTHON} --version))"
+
+# Install Python dependencies
+echo ""
+echo "Checking Python dependencies..."
+"${PYTHON}" -c "import datasets" 2>/dev/null || {
+ echo "Installing 'datasets' library..."
+ pip install datasets --quiet
+}
+"${PYTHON}" -c "import nltk" 2>/dev/null || {
+ echo "Installing 'nltk' library..."
+ pip install nltk --quiet
+}
+"${PYTHON}" -c "import torch" 2>/dev/null || {
+ echo "Installing PyTorch (CPU-only, needed for Megatron preprocessing)..."
+ pip install torch --index-url https://download.pytorch.org/whl/cpu --quiet
+}
+"${PYTHON}" -c "import transformers" 2>/dev/null || {
+ echo "Installing 'transformers' library (needed for tokenizer in preprocessing)..."
+ pip install transformers --quiet
+}
+
+# Download NLTK punkt tokenizer data (needed for BERT sentence splitting)
+echo "Ensuring NLTK punkt tokenizer data is available..."
+"${PYTHON}" -c "import nltk; nltk.download('punkt', quiet=True); nltk.download('punkt_tab', quiet=True)" 2>/dev/null || true
+
+# Create the output directory
+mkdir -p "${OUTPUT_DIR}"
+
+# Run the dataset creation script
+echo ""
+echo "============================================================"
+echo "Running dataset creation..."
+echo "============================================================"
+
+"${PYTHON}" "${DATASET_SCRIPT}" \
+ --output-dir "${OUTPUT_DIR}" \
+ --megatron-dir "${MEGATRON_DIR}" \
+ --num-documents "${NUM_DOCUMENTS}" \
+ --dataset-name "${DATASET_NAME}" \
+ --copy-vocab-from "${EXISTING_VOCAB}" \
+ --keep-jsonl \
+ --workers 4
+
+echo ""
+echo "============================================================"
+echo "Done! Dataset saved to: ${OUTPUT_DIR}"
+echo "============================================================"
+echo ""
+echo "Final directory listing:"
+find "${OUTPUT_DIR}" -type f -exec ls -lh {} \;
+
+# Cleanup
+echo ""
+echo "Cleaning up work directory: ${WORK_DIR}"
+rm -rf "${WORK_DIR}"
diff --git a/tools/preprocess_data.py b/tools/preprocess_data.py
index 2f73f267e33..f9a7d232fe4 100644
--- a/tools/preprocess_data.py
+++ b/tools/preprocess_data.py
@@ -11,9 +11,8 @@
import time
import gzip
import glob
-import torch
-import numpy as np
import multiprocessing
+import numpy as np
try:
import nltk
from nltk.tokenize.punkt import PunktLanguageVars
@@ -22,9 +21,7 @@
PunktLanguageVars = object # Fallback to the built-in object class
nltk_available = False
-from megatron.core.tokenizers.text.utils.build_tokenizer import build_tokenizer as build_new_tokenizer
-from megatron.core.tokenizers import MegatronTokenizer
-from megatron.training.tokenizer import build_tokenizer
+from megatron.core.tokenizers.utils.build_tokenizer import build_tokenizer
from megatron.training.arguments import _add_tokenizer_args
from megatron.core.datasets import indexed_dataset
@@ -53,10 +50,7 @@ def __init__(self, args):
def initializer(self):
# Use Encoder class as a container for global data
- if self.args.legacy_tokenizer:
- Encoder.tokenizer = build_tokenizer(self.args)
- else:
- Encoder.tokenizer = build_new_tokenizer(self.args)
+ Encoder.tokenizer = build_tokenizer(self.args)
if self.args.split_sentences:
if not nltk_available:
print("NLTK is not available to split sentences.")
@@ -118,6 +112,7 @@ class Partition(object):
def __init__(self, args, workers):
self.args = args
self.workers = workers
+ self.performance = []
def print_processing_stats(self, count, proc_start, total_bytes_processed):
if count % self.args.log_interval == 0:
@@ -127,6 +122,8 @@ def print_processing_stats(self, count, proc_start, total_bytes_processed):
print(f"Processed {count} documents",
f"({count/elapsed} docs/s, {mbs} MB/s).",
file=sys.stderr)
+ if self.args.find_optimal_num_workers:
+ self.performance.append(count/elapsed)
def split_sentences(self, file_name):
input_file_name, output_file_name = file_name
@@ -148,6 +145,8 @@ def split_sentences(self, file_name):
fin.close()
fout.close()
+ pool.close()
+ pool.join()
def process_json_file(self, file_name):
input_file_name, output_prefix = file_name
@@ -159,10 +158,7 @@ def process_json_file(self, file_name):
startup_start = time.time()
encoder = Encoder(self.args)
- if self.args.legacy_tokenizer:
- tokenizer = build_tokenizer(self.args)
- else:
- tokenizer = build_new_tokenizer(self.args)
+ tokenizer = build_tokenizer(self.args)
pool = multiprocessing.Pool(self.workers, initializer=encoder.initializer)
encoded_docs = pool.imap(encoder.encode, fin, 32)
@@ -189,14 +185,22 @@ def process_json_file(self, file_name):
total_bytes_processed = 0
print("Time to startup:", startup_end - startup_start)
for i, (doc, sentence_lens, bytes_processed) in enumerate(encoded_docs, start=1):
- total_bytes_processed += bytes_processed
- for key in doc.keys():
- builders[key].add_document(doc[key], sentence_lens[key])
- self.print_processing_stats(i, proc_start, total_bytes_processed)
+ if self.args.find_optimal_num_workers and i > self.args.max_documents:
+ break
+ else:
+ total_bytes_processed += bytes_processed
+ for key in doc.keys():
+ builders[key].add_document(doc[key], sentence_lens[key])
+ self.print_processing_stats(i, proc_start, total_bytes_processed)
fin.close()
builders[key].finalize(output_idx_files[key])
+ pool.close()
+ pool.join()
+
+ return self.performance
+
def get_args():
parser = argparse.ArgumentParser()
@@ -223,6 +227,19 @@ def get_args():
help=('Number of worker processes to launch.'
'A good default for fast pre-processing '
'is: (workers * partitions) = available CPU cores.'))
+ group.add_argument('--find-optimal-num-workers', action='store_true',
+ help=('Find optimal number of workers.'
+ 'Script will run few small jobs with '
+ 'different number of workers to define '
+ 'optimal number of workers in terms of performance.'))
+ group.add_argument('--workers-to-check', nargs='+', type=int, default=[16, 32, 64],
+ help=('list of workers to run data processing with '
+ 'to find optimal number of workers. '
+ 'Works only when --find-optimal-num-workers is enabled. '))
+ group.add_argument('--max-documents', type=int, default=100_000,
+ help=('Maximum number of documents to preprocess '
+ 'to find optimal number of workers.'
+ 'Works only when --find-optimal-num-workers is enabled.'))
group.add_argument('--partitions', type=int, default=1,
help='Number of file partitions')
group.add_argument('--log-interval', type=int, default=1000,
@@ -230,8 +247,6 @@ def get_args():
group.add_argument('--keep-sequential-samples', action='store_true',
help='Ensure ordering of samples in .jsonl files is '
'preserved when using partitions>1.')
- # group.add_argument('--legacy-tokenizer', action='store_true',
- # help='Use legacy tokenizer system.')
args = parser.parse_args()
args.keep_empty = False
@@ -266,146 +281,197 @@ def check_files_exist(in_ss_out_names, key, num_partitions):
return True
+def find_optimal_num_workers(performance, partitions):
+ """Parses saved .json files with perf. numbers and prints optimal number of workers"""
+ results = []
+
+ # each file assumed to contain a single {workers: [perf_list]}
+ for workers, perf_list in performance.items():
+ workers = int(workers)
+ avg_perf = np.mean(perf_list)
+ results.append((workers, avg_perf))
+
+ # sort by average performance (descending: fastest first)
+ results.sort(key=lambda x: x[1], reverse=True)
+
+ print("\n-----------------------------------")
+ print("Performance results (fastest → slowest):")
+ for i, (workers, avg_perf) in enumerate(results):
+ print(f"{i+1}. {workers * partitions} workers → avg. docs/s: {avg_perf:.4f}")
+
+ best_workers, best_perf = results[0]
+
+ print("\n-----------------------------------")
+ print(
+ f"The most optimal num of workers is {best_workers * partitions} "
+ f"with avg. preprocessed docs/s: {best_perf:.4f}."
+ )
+ print("-----------------------------------")
+
+
def main():
args = get_args()
- if args.split_sentences:
- if nltk_available:
- nltk.download("punkt", quiet=True, download_dir=os.environ.get("NLTK_DATA"))
+ workers = args.workers_to_check if args.find_optimal_num_workers else [args.workers]
+ for num_workers in workers:
+ if num_workers % args.partitions != 0:
+ print(
+ f"Removing num_workers ({num_workers}) from workers list "
+ f"because it's not divisible by num_partitions ({args.partitions})"
+ )
+ workers.remove(num_workers)
+ assert workers, "Please, provide valid number of workers which is divisible by number of partitions."
+ if args.find_optimal_num_workers:
+ args.log_interval = 1000
+
+ performance = {}
+ for num_workers in workers:
+ print(f"Processing data with {num_workers} workers.")
+ if args.split_sentences:
+ if nltk_available:
+ nltk.download("punkt", quiet=True, download_dir=os.environ.get("NLTK_DATA"))
+ else:
+ raise Exception(
+ "nltk library required for sentence splitting is not available.")
+
+ in_ss_out_names = []
+ if args.partitions == 1:
+ file_name, extension = os.path.splitext(args.input)
+ sentence_split_file = file_name + "_ss" + extension
+ file_names = {
+ 'partition': args.input,
+ 'sentence_split': sentence_split_file,
+ 'output_prefix': args.output_prefix}
+ in_ss_out_names.append(file_names)
else:
- raise Exception(
- "nltk library required for sentence splitting is not available.")
-
- in_ss_out_names = []
- if args.partitions == 1:
- file_name, extension = os.path.splitext(args.input)
- sentence_split_file = file_name + "_ss" + extension
- file_names = {
- 'partition': args.input,
- 'sentence_split': sentence_split_file,
- 'output_prefix': args.output_prefix}
- in_ss_out_names.append(file_names)
- else:
- in_file_names = glob.glob(args.input)
-
- # Count total number of lines across .jsonl files
- if args.keep_sequential_samples:
- total_sample_count = 0
- for filename in in_file_names:
- with open(filename, "r") as fin:
- for fc, _ in enumerate(fin):
- pass
- total_sample_count += (fc + 1)
- partition_size = math.ceil(total_sample_count / args.partitions)
-
- # create .jsonl parition files
- for idx in range(args.partitions):
- in_ss_out_name = get_file_name(args, idx)
- in_ss_out_names.append(in_ss_out_name)
-
- # check to see if paritions were already created
- partitions_present = check_files_exist(in_ss_out_names, 'partition', args.partitions)
+ in_file_names = glob.glob(args.input)
+
+ # Count total number of lines across .jsonl files
+ if args.keep_sequential_samples:
+ total_sample_count = 0
+ for filename in in_file_names:
+ with open(filename, "r") as fin:
+ for fc, _ in enumerate(fin):
+ pass
+ total_sample_count += (fc + 1)
+ partition_size = math.ceil(total_sample_count / args.partitions)
+
+ # create .jsonl parition files
+ for idx in range(args.partitions):
+ in_ss_out_name = get_file_name(args, idx)
+ in_ss_out_names.append(in_ss_out_name)
+
+ # check to see if paritions were already created
+ partitions_present = check_files_exist(in_ss_out_names, 'partition', args.partitions)
+
+ # check to see if paritions with split sentences already created
+ split_sentences_present = check_files_exist(in_ss_out_names, 'sentence_split', args.partitions)
+
+ if not partitions_present and not split_sentences_present:
+ # populate .jsonl partition files from parent files
+ partitioned_input_files = []
+ for idx in range(args.partitions):
+ partitioned_input_file = open(in_ss_out_names[idx]['partition'], 'w')
+ partitioned_input_files.append(partitioned_input_file)
+
+ index = 0
+ if args.keep_sequential_samples: line_count = 0
+ for in_file_name in in_file_names:
+ # support for gzip files
+ if in_file_name.endswith(".gz"):
+ fin = gzip.open(in_file_name, 'rt')
+ else:
+ fin = open(in_file_name, 'r', encoding='utf-8')
+
+ for line in fin:
+ partitioned_input_files[index].write(line)
+ if args.keep_sequential_samples:
+ line_count += 1
+ if line_count % partition_size == 0:
+ index += 1
+ else:
+ index = (index + 1)%args.partitions
+
+ fin.close()
+
+ for idx in range(args.partitions):
+ partitioned_input_files[idx].close()
+
+ partition = Partition(args, num_workers//args.partitions)
# check to see if paritions with split sentences already created
split_sentences_present = check_files_exist(in_ss_out_names, 'sentence_split', args.partitions)
- if not partitions_present and not split_sentences_present:
- # populate .jsonl partition files from parent files
- partitioned_input_files = []
- for idx in range(args.partitions):
- partitioned_input_file = open(in_ss_out_names[idx]['partition'], 'w')
- partitioned_input_files.append(partitioned_input_file)
-
- index = 0
- if args.keep_sequential_samples: line_count = 0
- for in_file_name in in_file_names:
- # support for gzip files
- if in_file_name.endswith(".gz"):
- fin = gzip.open(in_file_name, 'rt')
- else:
- fin = open(in_file_name, 'r', encoding='utf-8')
-
- for line in fin:
- partitioned_input_files[index].write(line)
- if args.keep_sequential_samples:
- line_count += 1
- if line_count % partition_size == 0:
- index += 1
- else:
- index = (index + 1)%args.partitions
-
- fin.close()
+ # split sentences in partition files
+ if args.split_sentences and not split_sentences_present:
+ processes = []
+ for name in in_ss_out_names:
+ p = multiprocessing.Process(target=partition.split_sentences,
+ args=((name['partition'], name['sentence_split']),))
+ p.start()
+ processes.append(p)
- for idx in range(args.partitions):
- partitioned_input_files[idx].close()
+ for p in processes:
+ p.join()
- assert args.workers % args.partitions == 0
- partition = Partition(args, args.workers//args.partitions)
+ if args.partitions == 1:
+ continue
- # check to see if paritions with split sentences already created
- split_sentences_present = check_files_exist(in_ss_out_names, 'sentence_split', args.partitions)
+ def process_json_file(name, q, input_key):
+ worker_performance = partition.process_json_file((name[input_key], name['output_prefix']))
+ q.put(worker_performance)
- # split sentences in partition files
- if args.split_sentences and not split_sentences_present:
+ # encode partition files in parallel
processes = []
+ input_key = 'sentence_split' if args.split_sentences else 'partition'
+ q = multiprocessing.Queue()
for name in in_ss_out_names:
- p = multiprocessing.Process(target=partition.split_sentences,
- args=((name['partition'], name['sentence_split']),))
+ p = multiprocessing.Process(target=process_json_file, args=(name, q, input_key))
+
p.start()
processes.append(p)
+ for _ in processes:
+ worker_performance = q.get()
+ if args.find_optimal_num_workers:
+ performance[num_workers] = worker_performance
+
for p in processes:
p.join()
if args.partitions == 1:
- return
-
-
- # encode partition files in parallel
- processes = []
- input_key = 'sentence_split' if args.split_sentences else 'partition'
- for name in in_ss_out_names:
- p = multiprocessing.Process(target=partition.process_json_file,
- args=((name[input_key], name['output_prefix']),))
- p.start()
- processes.append(p)
+ continue
- for p in processes:
- p.join()
-
- if args.partitions == 1:
- return
-
- # merge bin/idx partitions
- level = "document"
- if args.split_sentences:
- level = "sentence"
+ # merge bin/idx partitions
+ level = "document"
+ if args.split_sentences:
+ level = "sentence"
- output_bin_files = {}
- output_idx_files = {}
- builders = {}
- if args.legacy_tokenizer:
+ output_bin_files = {}
+ output_idx_files = {}
+ builders = {}
tokenizer = build_tokenizer(args)
- else:
- tokenizer = build_new_tokenizer(args)
-
- for key in args.json_keys:
- output_bin_files[key] = "{}_{}_{}.bin".format(args.output_prefix,
- key, level)
- output_idx_files[key] = "{}_{}_{}.idx".format(args.output_prefix,
- key, level)
- builders[key] = indexed_dataset.IndexedDatasetBuilder(
- output_bin_files[key],
- dtype=indexed_dataset.DType.optimal_dtype(tokenizer.vocab_size),
- )
- for name in in_ss_out_names:
- parition_output_prefix = name['output_prefix']
- full_partition_output_prefix = "{}_{}_{}".format(parition_output_prefix,
- key, level)
- builders[key].add_index(full_partition_output_prefix)
- builders[key].finalize(output_idx_files[key])
+ for key in args.json_keys:
+ output_bin_files[key] = "{}_{}_{}.bin".format(args.output_prefix,
+ key, level)
+ output_idx_files[key] = "{}_{}_{}.idx".format(args.output_prefix,
+ key, level)
+ builders[key] = indexed_dataset.IndexedDatasetBuilder(
+ output_bin_files[key],
+ dtype=indexed_dataset.DType.optimal_dtype(tokenizer.vocab_size),
+ )
+
+ for name in in_ss_out_names:
+ parition_output_prefix = name['output_prefix']
+ full_partition_output_prefix = "{}_{}_{}".format(parition_output_prefix,
+ key, level)
+ builders[key].add_index(full_partition_output_prefix)
+ builders[key].finalize(output_idx_files[key])
+ # Find the most optimal number of workers
+ if args.find_optimal_num_workers:
+ find_optimal_num_workers(performance, args.partitions)
if __name__ == '__main__':
diff --git a/tools/preprocess_data_nmt.py b/tools/preprocess_data_nmt.py
index 13a04f6ee28..f753c54f578 100644
--- a/tools/preprocess_data_nmt.py
+++ b/tools/preprocess_data_nmt.py
@@ -11,7 +11,7 @@
os.path.pardir)))
import time
import torch
-from megatron.training.tokenizer import build_tokenizer
+from megatron.core.tokenizers.utils.build_tokenizer import build_tokenizer
from megatron.core.datasets import indexed_dataset
diff --git a/tools/preprocess_mmdata.py b/tools/preprocess_mmdata.py
index 8ab2c2b8673..b63c9e99cad 100755
--- a/tools/preprocess_mmdata.py
+++ b/tools/preprocess_mmdata.py
@@ -20,7 +20,7 @@
except ImportError:
PunktLanguageVars = object # Fallback to the built-in object class
-from megatron.training.tokenizer import build_tokenizer
+from megatron.core.tokenizers.utils.build_tokenizer import build_tokenizer
from megatron.core.datasets.indexed_dataset import IndexedDatasetBuilder
@@ -101,6 +101,17 @@ def get_args():
help='Language to use for NLTK-powered sentence splitting.')
group.add_argument('--tokenizer-model', type=str, default=None,
help='sentencepeice tokenizer model.')
+ group.add_argument('--tokenizer-metadata', type=str, default=None,
+ help='Path to tokenizer metadata in json format.')
+ group.add_argument('--tokenizer-special-tokens', type=str, nargs='+', default=None,
+ help='List of special tokens. For TikTokenizer needs to have '
+ '["", "", " ", "", "", "", ""]')
+ group.add_argument('--tokenizer-hf-no-use-fast', action='store_true', default=False,
+ help='Whether to use fast HuggingFace tokenizer.')
+ group.add_argument('--tokenizer-hf-no-include-special-tokens', action='store_true', default=False,
+ help='Converting text to ids will not include special for HuggingFace tokenizer.')
+ group.add_argument("--trust-remote-code", action="store_true", default=False,
+ help='Whether or not to allow PreTrainedTokenizer to execute remote code')
group = parser.add_argument_group(title='output data')
group.add_argument('--output-prefix', type=str, required=True,
@@ -110,6 +121,7 @@ def get_args():
help='Number of worker processes to launch')
group.add_argument('--log-interval', type=int, default=100,
help='Interval between progress updates')
+
args = parser.parse_args()
args.keep_empty = False
diff --git a/tools/run_dynamic_text_generation_server.py b/tools/run_dynamic_text_generation_server.py
index 615073b8fd0..74fbfd8e3d3 100644
--- a/tools/run_dynamic_text_generation_server.py
+++ b/tools/run_dynamic_text_generation_server.py
@@ -5,66 +5,73 @@
import torch
-from examples.inference.gpt.gpt_dynamic_inference import (
- add_dynamic_inference_args,
- get_inference_context,
- get_inference_controller,
- get_model,
-)
from megatron.core.inference.engines import DynamicInferenceEngine
-from megatron.core.inference.text_generation_server.dynamic_text_gen_server import run_flask_server
-from megatron.core.tokenizers.text.utils.build_tokenizer import build_tokenizer
-from megatron.core.utils import get_mamba_inference_state_config_from_model, trace_async_exceptions
+from megatron.core.inference.text_generation_server.dynamic_text_gen_server import (
+ start_text_gen_server,
+ stop_text_gen_server,
+)
+from megatron.core.utils import trace_async_exceptions
+from megatron.inference.utils import add_inference_args, get_dynamic_inference_engine
from megatron.post_training.arguments import add_modelopt_args
-from megatron.training import get_args, get_tokenizer
+from megatron.training import get_args
from megatron.training.initialize import initialize_megatron
def add_text_generation_server_args(parser: argparse.ArgumentParser):
"""Adds the required command line arguments for running the text generation server."""
parser = add_modelopt_args(parser)
- parser = add_dynamic_inference_args(parser)
+ parser = add_inference_args(parser)
parser.add_argument("--port", type=int, default=5000, help="Port for Flask server to run on")
+ parser.add_argument(
+ "--host", type=str, default=None,
+ help="Hostname or IP address to bind the server to. Defaults to 0.0.0.0 (all interfaces)."
+ )
+ parser.add_argument(
+ "--parsers", type=str, nargs="+", default=[], help="Parsers to use for parsing the response"
+ )
return parser
@trace_async_exceptions
async def run_text_generation_server(
- engine: DynamicInferenceEngine, coordinator_port: int, flask_port: int
+ engine: DynamicInferenceEngine, coordinator_port: int, server_port: int, hostname: str | None = None,
):
- """Runs the Flask server from rank 0 and initializes the DynamicInferenceEngine on all ranks.
+ """
+ Runs the text generation server from rank 0 and initializes the
+ DynamicInferenceEngine on all ranks.
Args:
engine (DynamicInferenceEngine): The dynamic inference engine.
coordinator_port (int): The network port for the dynamic inference DP coordinator.
- flask_port (int): The network for port the frontend Flask server.
+ server_port (int): The network for port the frontend text generation server.
"""
rank = torch.distributed.get_rank()
- await engine.start_listening_to_data_parallel_coordinator(
- inference_coordinator_port=coordinator_port, launch_inference_coordinator=True
+ coordinator_addr = await engine.start_listening_to_data_parallel_coordinator(
+ inference_coordinator_port=coordinator_port, launch_inference_coordinator=True,
+ hostname=hostname,
)
- server_task = None
- if rank == 0:
- server_task = asyncio.create_task(
- run_flask_server(
- coordinator_port=coordinator_port,
+ try:
+ if rank == 0:
+ start_text_gen_server(
+ coordinator_addr=coordinator_addr,
tokenizer=engine.controller.tokenizer,
+ parsers=args.parsers,
rank=rank,
- flask_port=flask_port,
+ server_port=server_port,
+ verbose=args.inference_text_gen_server_logging,
+ hostname=hostname,
)
- )
- engine_task = engine.engine_loop_task
-
- tasks_to_run = [engine_task]
- if server_task:
- assert rank == 0
- tasks_to_run.append(server_task)
+ # Await the engine loop directly since the server is running in a separate process
+ await engine.engine_loop_task
- await asyncio.gather(*tasks_to_run)
+ finally:
+ # Guarantee that the separate process is terminated when the engine loop stops or is interrupted
+ if rank == 0:
+ stop_text_gen_server()
if __name__ == "__main__":
@@ -74,36 +81,22 @@ async def run_text_generation_server(
args_defaults={'no_load_rng': True, 'no_load_optim': True},
)
- args = get_args()
- model = get_model()
-
- if args.legacy_tokenizer:
- tokenizer = get_tokenizer()
- else:
- tokenizer = build_tokenizer(args)
-
- mamba_inference_state_config = get_mamba_inference_state_config_from_model(model)
-
# Enable return_log_probs to allow prompt logprobs computation for echo=True requests
# This sets materialize_only_last_token_logits=False in the inference context,
# which is required for lm-eval compatibility (loglikelihood evaluation tasks)
+ args = get_args()
args.return_log_probs = True
- context = get_inference_context(
- None,
- None,
- calculate_max_sequence_length_from_requests=False,
- mamba_inference_state_config=mamba_inference_state_config,
- )
-
- controller = get_inference_controller(model, context)
-
- engine = DynamicInferenceEngine(
- controller,
- context,
- enable_cuda_graph=args.cuda_graph_impl == "local",
- random_seed=args.seed,
- enable_chunked_prefill=not args.disable_chunked_prefill,
- )
+ engine = get_dynamic_inference_engine()
- asyncio.run(run_text_generation_server(engine, args.inference_coordinator_port, args.port))
+ try:
+ asyncio.run(
+ run_text_generation_server(engine, args.inference_coordinator_port, args.port, args.host)
+ )
+ except KeyboardInterrupt:
+ # Catching at the top level ensures clean stdout without spamming the traceback
+ print("Server process interrupted by user.")
+ finally:
+ # Clean up PyTorch distributed groups properly
+ if torch.distributed.is_initialized():
+ torch.distributed.destroy_process_group()
diff --git a/tools/run_inference_performance_test.py b/tools/run_inference_performance_test.py
index 32d61444530..a1feef380ef 100644
--- a/tools/run_inference_performance_test.py
+++ b/tools/run_inference_performance_test.py
@@ -10,33 +10,32 @@
from gpt_builders import gpt_builder
from mamba_builders import mamba_builder
-from megatron.core.inference.contexts import DynamicInferenceContext
+from megatron.core.inference.contexts import StaticInferenceContext
from megatron.core.inference.engines import DynamicInferenceEngine, StaticInferenceEngine
from megatron.core.inference.engines.abstract_engine import AbstractEngine
-from megatron.core.inference.inference_request import InferenceRequest
+from megatron.core.inference.inference_request import (
+ DynamicInferenceRequestRecord,
+ InferenceRequest,
+)
from megatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapper import (
GPTInferenceWrapper,
)
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
from megatron.core.inference.sampling_params import SamplingParams
from megatron.core.inference.text_generation_controllers.text_generation_controller import (
TextGenerationController,
)
+from megatron.core.tokenizers.utils.build_tokenizer import build_tokenizer
from megatron.core.transformer.module import MegatronModule
-from megatron.core.utils import get_mamba_inference_state_config_from_model
+from megatron.inference.utils import add_inference_args, get_dynamic_inference_engine, get_model_for_inference
from model_provider import model_provider
sys.path.append(
os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir, os.path.pardir))
)
-import asyncio
from functools import partial
-from typing import List, Union
+from typing import List
-from examples.inference.gpt.utils import add_common_inference_args
from megatron.core import mpu
from megatron.training import get_args, get_model, get_tokenizer
from megatron.training.checkpointing import load_checkpoint
@@ -47,7 +46,7 @@
def add_inference_benchmarking_args(parser):
"""Inference benchmarking arguments."""
- parser = add_common_inference_args(parser)
+ parser = add_inference_args(parser)
group = parser.add_argument_group(title='inference_benchmarking')
@@ -60,7 +59,6 @@ def add_inference_benchmarking_args(parser):
group.add_argument(
"--benchmark-profile", action="store_true", default=False, help="If set, profile"
)
- group.add_argument('--stream', action="store_true", default=False, help="If set, stream tokens")
return parser
@@ -74,24 +72,13 @@ def get_inference_engine(args: argparse.Namespace, model: MegatronModule) -> Abs
Returns:
AbstractBackend: The chosen backend
"""
- tokenizer = get_tokenizer()
-
- inference_wrapper_config = InferenceWrapperConfig(
- hidden_size=args.hidden_size,
- inference_batch_times_seqlen_threshold=args.inference_batch_times_seqlen_threshold,
- fp32_residual_connection=args.fp32_residual_connection,
- params_dtype=args.params_dtype,
- padded_vocab_size=args.padded_vocab_size,
- inference_max_requests=args.inference_max_batch_size,
- inference_max_seq_length=args.inference_max_seq_length,
- nccl_all_reduce_for_prefill=args.nccl_all_reduce_for_prefill,
- moe_pad_experts_for_cuda_graph_inference=args.moe_pad_experts_for_cuda_graph_inference,
- )
-
- mamba_inference_state_config = get_mamba_inference_state_config_from_model(model)
if args.engine_type == "static":
- inference_wrapped_model = GPTInferenceWrapper(model, inference_wrapper_config)
+ tokenizer = get_tokenizer()
+ context = StaticInferenceContext(
+ args.inference_max_requests, args.inference_max_sequence_length
+ )
+ inference_wrapped_model = GPTInferenceWrapper(model, context)
inference_wrapped_model.model_is_pipeline_parallel = not (
mpu.is_pipeline_first_stage() and mpu.is_pipeline_last_stage()
)
@@ -100,98 +87,7 @@ def get_inference_engine(args: argparse.Namespace, model: MegatronModule) -> Abs
)
return StaticInferenceEngine(text_generation_controller=text_generation_controller)
elif args.engine_type == "dynamic":
- context = DynamicInferenceContext(
- params_dtype=args.params_dtype,
- num_layers=args.num_layers,
- kv_channels=args.kv_channels,
- num_attention_heads=(
- args.num_query_groups if args.group_query_attention else args.num_attention_heads
- ),
- max_sequence_length=args.inference_max_seq_length,
- num_cuda_graphs=(
- args.inference_dynamic_batching_num_cuda_graphs
- if args.cuda_graph_impl == "local"
- else None
- ),
- buffer_size_gb=args.inference_dynamic_batching_buffer_size_gb,
- buffer_guaranteed_fraction=args.inference_dynamic_batching_buffer_guaranteed_fraction,
- buffer_overflow_factor=args.inference_dynamic_batching_buffer_overflow_factor,
- max_requests_override=args.inference_dynamic_batching_max_requests_override,
- max_tokens_override=args.inference_dynamic_batching_max_tokens_override,
- block_size_tokens=args.inference_dynamic_batching_block_size,
- tensor_model_parallel_size=args.tensor_model_parallel_size,
- pipeline_model_parallel_size=args.pipeline_model_parallel_size,
- materialize_only_last_token_logits=not args.return_log_probs,
- mamba_inference_state_config=mamba_inference_state_config,
- cache_mla_latent=args.multi_latent_attention and args.cache_mla_latents,
- kv_lora_rank=args.kv_lora_rank if args.multi_latent_attention else None,
- qk_pos_emb_head_dim=args.qk_pos_emb_head_dim,
- use_cuda_graphs_for_non_decode_steps=not args.decode_only_cuda_graphs,
- use_flashinfer_fused_rope=args.use_flashinfer_fused_rope,
- unified_memory_level=args.inference_dynamic_batching_unified_memory_level,
- )
- inference_wrapped_model = GPTInferenceWrapper(
- model, inference_wrapper_config, inference_context=context
- )
- inference_wrapped_model.model_is_pipeline_parallel = not (
- mpu.is_pipeline_first_stage() and mpu.is_pipeline_last_stage()
- )
- text_generation_controller = TextGenerationController(
- inference_wrapped_model=inference_wrapped_model, tokenizer=tokenizer
- )
- return DynamicInferenceEngine(
- text_generation_controller,
- context,
- termination_id=-1,
- enable_cuda_graph=args.cuda_graph_impl == "local",
- random_seed=args.seed,
- )
-
-
-async def generate(
- inference_engine: Union[StaticInferenceEngine, DynamicInferenceEngine],
- sampling_params: SamplingParams,
- prompts: List[str],
- inference_requests: List[InferenceRequest] = None,
-) -> List[InferenceRequest]:
- async def collect_stream(prompt, request_id, stream_generator):
- async for output in stream_generator:
- pass
-
- if inference_requests is None:
- assert prompts is not None
- inference_requests = [None for _ in range(len(prompts))]
- elif prompts is None:
- assert inference_requests is not None
- tokenizer = get_tokenizer()
- prompts = [tokenizer.detokenize(request.prompt_tokens) for request in inference_requests]
-
- request_ids: List[int] = [
- inference_engine.add_request(
- prompt=prompt,
- inference_request=inference_request,
- inference_parameters=sampling_params,
- streaming=True,
- )
- for prompt, inference_request in zip(prompts, inference_requests)
- ]
- stream_generators = [
- inference_engine.get_stream_generator(request_id) for request_id in request_ids
- ]
-
- tasks = [
- asyncio.create_task(collect_stream(prompt, request_id, stream_generator))
- for (prompt, request_id, stream_generator) in zip(prompts, request_ids, stream_generators)
- ]
-
- await inference_engine.run_engine_async()
- await asyncio.gather(*tasks)
-
- results: List[InferenceRequest] = [
- inference_engine.scheduler.completed_request_pool[request_id] for request_id in request_ids
- ]
-
- return results
+ return get_dynamic_inference_engine(model=model)
def get_random_prompt_tokens(tokenizer, num_input_tokens) -> List[int]:
@@ -232,14 +128,12 @@ def generate_dynamic(
request_id = REQUEST_ID
REQUEST_ID += 1
prompt_tokens = request.prompt_tokens
- inference_engine.add_request(
- request_id, prompt_tokens, request.inference_parameters,
- )
+ inference_engine.add_request(request_id, prompt_tokens, request.inference_parameters)
start_time = time.perf_counter()
all_finished_requests = []
while inference_engine.has_unfinished_requests():
- result = inference_engine.step(verbose=False)
+ result = inference_engine.step()
finished_requests = result["finished_requests"]
for request in finished_requests:
req_id = request.request_id
@@ -257,8 +151,6 @@ def generate_dynamic(
def main():
"""Main program."""
- # Note: The default args passed here can be overwritten by using appropriate params (check arguments.py file)
- # Micro batch size is not needed to be set by user. (It is calculated based on inference-batch-times-seqlen-threshold argument)
initialize_megatron(
extra_args_provider=add_inference_benchmarking_args,
args_defaults={
@@ -271,19 +163,9 @@ def main():
args = get_args()
- # Set up model and load checkpoint
- if args.model_provider == "gpt":
- model_builder = gpt_builder
- elif args.model_provider == "mamba":
- model_builder = mamba_builder
- else:
- raise ValueError(f"Invalid model provider {args.model_provider}")
+ model = get_model_for_inference()
- model = get_model(partial(model_provider, model_builder), wrap_with_ddp=False)
- tokenizer = get_tokenizer()
- load_checkpoint(model, None, None)
- model = model[0]
- model.eval()
+ tokenizer = build_tokenizer(args)
assert (args.prompts is None) ^ (
args.num_input_tokens is None
@@ -298,13 +180,14 @@ def main():
return_log_probs=args.return_log_probs,
top_n_logprobs=args.top_n_logprobs,
num_tokens_to_generate=args.num_tokens_to_generate,
+ termination_id=-1,
)
sampling_params.add_attributes({"no_early_termination": True})
requests = []
if args.num_input_tokens is not None:
assert args.prompts is None
- batch_size = args.inference_max_batch_size
+ batch_size = args.inference_max_requests
for i in range(batch_size):
prompt_tokens = get_random_prompt_tokens(tokenizer, args.num_input_tokens)
requests.append(
@@ -327,33 +210,27 @@ def main():
)
)
- if args.cuda_graph_impl == "local":
- print(f"Running warmup for CUDA graphs...")
- warmup_sampling_params = SamplingParams(num_tokens_to_generate=10)
- warmup_sampling_params.add_attributes({"no_early_termination": True})
+ # TODO(ksanthanam): Use a command line argument for warmup iterations
+ for i in range(3):
+ print(f"Running warmup iteration {i+1}...")
+ warmup_sampling_params = SamplingParams(num_tokens_to_generate=10, termination_id=-1)
inference_engine.generate(prompts=["warmup"], sampling_params=warmup_sampling_params)
if args.benchmark_profile:
torch.cuda.cudart().cudaProfilerStart()
start_time = time.perf_counter()
- if args.stream:
- if args.engine_type == "dynamic":
- raise NotImplementedError("Streaming not supported with DynamicInferenceEngine")
- results: List[InferenceRequest] = asyncio.run(
- generate(
- inference_engine, sampling_params, prompts=args.prompts, inference_requests=requests
- )
+ if args.engine_type == "static":
+ results: List[InferenceRequest] = inference_engine.generate(
+ prompts=args.prompts, inference_requests=requests, sampling_params=sampling_params
)
else:
- if args.engine_type == "static":
- results: List[InferenceRequest] = inference_engine.generate(
- prompts=args.prompts, inference_requests=requests, sampling_params=sampling_params
- )
- elif args.engine_type == "dynamic":
- results: List[InferenceRequest] = generate_dynamic(
- args, requests, inference_engine,
- )
+ prompts = [request.prompt_tokens for request in requests]
+ records: List[DynamicInferenceRequestRecord] = inference_engine.generate(
+ prompts=prompts, sampling_params=sampling_params
+ )
+ results: List[InferenceRequest] = [record.merge() for record in records]
+
end_time = time.perf_counter()
latency = end_time - start_time
@@ -378,6 +255,10 @@ def main():
result_dict['generated_output'] = tokenizer.detokenize(result.generated_tokens)
print(result_dict)
+ total_output_tokens = args.num_tokens_to_generate * args.inference_max_requests
+ throughput = total_output_tokens / latency
+ print(f"Throughput: {throughput} output tokens / second")
+
if __name__ == "__main__":
main()
diff --git a/tools/run_text_generation_server.py b/tools/run_text_generation_server.py
index 350173dc16f..83a0bbc2369 100644
--- a/tools/run_text_generation_server.py
+++ b/tools/run_text_generation_server.py
@@ -22,9 +22,6 @@
from megatron.core.inference.model_inference_wrappers.gpt.gpt_inference_wrapper import (
GPTInferenceWrapper,
)
-from megatron.core.inference.model_inference_wrappers.inference_wrapper_config import (
- InferenceWrapperConfig,
-)
from megatron.core.inference.sampling_params import SamplingParams
from megatron.core.inference.text_generation_controllers.text_generation_controller import (
TextGenerationController,
@@ -63,27 +60,15 @@ def get_inference_engine(args: Namespace, model: MegatronModule) -> AbstractEngi
tokenizer = get_tokenizer()
- inference_wrapper_config = InferenceWrapperConfig(
- hidden_size=args.hidden_size,
- inference_batch_times_seqlen_threshold=args.inference_batch_times_seqlen_threshold,
- fp32_residual_connection=args.fp32_residual_connection,
- params_dtype=args.params_dtype,
- padded_vocab_size=args.padded_vocab_size,
- inference_max_seq_length=args.inference_max_seq_length,
- inference_max_requests=args.inference_max_batch_size,
- nccl_all_reduce_for_prefill=args.nccl_all_reduce_for_prefill,
- moe_pad_experts_for_cuda_graph_inference = args.moe_pad_experts_for_cuda_graph_inference
- )
- inference_context = StaticInferenceContext.from_config(inference_wrapper_config)
+ inference_context = StaticInferenceContext(args.inference_max_requests, args.inference_max_sequence_length)
inference_wrapped_model = GPTInferenceWrapper(
- model, inference_wrapper_config, inference_context
+ model, inference_context
)
text_generation_controller = TextGenerationController(
inference_wrapped_model=inference_wrapped_model, tokenizer=tokenizer
)
return StaticInferenceEngine(
text_generation_controller=text_generation_controller,
- max_batch_size=args.inference_max_batch_size,
)
@@ -113,7 +98,7 @@ def add_text_generate_args(parser):
metavar='N',
type=str,
nargs='+',
- help='Input prompts with each prompt within quotes and seperated by space',
+ help='Input prompts with each prompt within quotes and separated by space',
)
group.add_argument(
"--max-batch-size",
@@ -166,14 +151,6 @@ def main(model_type: str = "gpt"):
model = model[0]
model.eval()
- if args.max_batch_size is not None:
- assert args.inference_max_batch_size is not None
- args.inference_max_batch_size = max(args.inference_max_batch_size, args.max_batch_size)
- warnings.warn(
- "`--max-batch-size` has been deprecated in favor of `--inference-max-requests`, "
- f"setting maximum batch size to {args.inference_max_batch_size}"
- )
-
inference_engine = get_inference_engine(args, model)
if args.cuda_graph_impl == "local":
diff --git a/tools/trigger_internal_ci.md b/tools/trigger_internal_ci.md
new file mode 100644
index 00000000000..77239bcbace
--- /dev/null
+++ b/tools/trigger_internal_ci.md
@@ -0,0 +1,77 @@
+# Trigger Internal CI
+
+:warning: This is only useful to NVIDIANs.
+
+Pushes the current branch to the internal GitLab remote and triggers a CI
+pipeline — without touching the GitLab UI.
+
+## Prerequisites
+
+**1. Add the internal GitLab as a git remote** (skip if you already have one configured):
+
+```bash
+git remote add gitlab git@:ADLR/Megatron-LM.git
+```
+
+To check existing remotes: `git remote -v`
+
+**The name of the origin will be required later!**
+
+**2. Obtain a personal access token:**
+
+1. Open your internal GitLab profile: **User menu → Edit profile → Access tokens**.
+2. Click **Add new token**, give it a description, set an expiry, and select the **`api`** scope.
+3. Click **Create personal access token** and copy the generated token (starts with `glpat-`).
+4. Store it in your environment to avoid passing it on every invocation:
+
+```bash
+export GITLAB_TOKEN=glpat-
+```
+
+**Tip: Store this in your .env or .bashrc file**
+
+## Usage
+
+```bash
+python -m pip install python-gitlab
+python tools/trigger_internal_ci.py \
+ --gitlab-origin gitlab \
+ [--access-token glpat-] \
+ [--functional-test-scope mr] \
+ [--functional-test-repeat 5] \
+ [--functional-test-cases all] \
+ [--dry-run]
+```
+
+| Argument | Default | Description |
+|---|---|---|
+| `--gitlab-origin` | *(required)* | Git remote name for the internal GitLab |
+| `--access-token` | `$GITLAB_TOKEN` | Personal access token with `api` scope |
+| `--functional-test-scope` | `mr` | `FUNCTIONAL_TEST_SCOPE` pipeline variable |
+| `--functional-test-repeat` | `5` | `FUNCTIONAL_TEST_REPEAT` pipeline variable |
+| `--functional-test-cases` | `all` | `FUNCTIONAL_TEST_CASES` pipeline variable |
+| `--dry-run` | off | Print what would happen without pushing or triggering |
+
+## Example
+
+```bash
+# Dry run — no push, no trigger
+python tools/trigger_internal_ci.py --gitlab-origin gitlab --dry-run
+
+# Real run — uses token from environment
+python tools/trigger_internal_ci.py --gitlab-origin gitlab
+```
+
+## Expected behavior
+
+```
+Current branch: my-feature-branch
+Everything up-to-date
+Triggering pipeline on https:// project 19378 @ pull-request/my-feature-branch
+Pipeline triggered: https://///-/pipelines/123456
+```
+
+1. The current branch is detected from git.
+2. The branch is force-pushed to the GitLab remote as `pull-request/`.
+3. A pipeline is triggered on that ref with the configured test variables.
+4. The URL of the newly created pipeline is printed.
diff --git a/tools/trigger_internal_ci.py b/tools/trigger_internal_ci.py
new file mode 100644
index 00000000000..4b0c19f09bd
--- /dev/null
+++ b/tools/trigger_internal_ci.py
@@ -0,0 +1,174 @@
+#!/usr/bin/env python3
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""CLI tool to trigger the internal GitLab CI pipeline from a local branch.
+
+Pushes the current branch to the internal GitLab remote under the
+pull-request/ naming convention and triggers a pipeline with
+the specified test configuration.
+
+Requires a GitLab personal access token with at least the 'api' scope.
+Set the GITLAB_TOKEN environment variable or pass --access-token.
+"""
+
+import argparse
+import logging
+import os
+import subprocess
+import sys
+from urllib.parse import urlparse
+
+import gitlab # python-gitlab
+
+GITLAB_PROJECT_ID = 19378
+GITLAB_BRANCH_PREFIX = "pull-request"
+
+PIPELINE_VARIABLES_FIXED = {
+ "UNIT_TEST": "no",
+ "INTEGRATION_TEST": "no",
+}
+
+logger = logging.getLogger(__name__)
+
+
+def get_remote_url(origin):
+ """Return the fetch URL configured for the given git remote name."""
+ result = subprocess.run(
+ ["git", "remote", "get-url", origin],
+ capture_output=True,
+ text=True,
+ check=True,
+ )
+ return result.stdout.strip()
+
+
+def get_gitlab_hostname(remote_url):
+ """Extract the hostname (without port) from an SSH or HTTPS remote URL."""
+ if remote_url.startswith("git@"):
+ hostname = remote_url.split("@", 1)[1].split(":")[0]
+ else:
+ hostname = urlparse(remote_url).hostname
+ return hostname.split(":")[0]
+
+
+def get_current_branch():
+ """Return the name of the currently checked-out git branch."""
+ result = subprocess.run(
+ ["git", "rev-parse", "--abbrev-ref", "HEAD"],
+ capture_output=True,
+ text=True,
+ check=True,
+ )
+ return result.stdout.strip()
+
+
+def git_push(origin, target_branch, dry_run=False):
+ """Force-push HEAD to the given branch on the named git remote."""
+ if dry_run:
+ logger.info("[DRY RUN] Would push HEAD to remote '%s' as %s", origin, target_branch)
+ return
+ subprocess.run(
+ ["git", "push", origin, f"HEAD:{target_branch}", "--force"],
+ check=True,
+ )
+
+
+def trigger_pipeline(gitlab_url, access_token, ref, pipeline_vars, dry_run=False):
+ """Trigger a GitLab pipeline on the given ref with the provided variables."""
+ if dry_run:
+ logger.info(
+ "[DRY RUN] Would trigger pipeline on https://%s project %s @ %s",
+ gitlab_url,
+ GITLAB_PROJECT_ID,
+ ref,
+ )
+ return
+ logger.info(
+ "Triggering pipeline on https://%s project %s @ %s", gitlab_url, GITLAB_PROJECT_ID, ref
+ )
+ gl = gitlab.Gitlab(f"https://{gitlab_url}", private_token=access_token)
+ project = gl.projects.get(GITLAB_PROJECT_ID, lazy=True)
+ variables = [{"key": k, "value": v} for k, v in pipeline_vars.items()]
+ pipeline = project.pipelines.create({"ref": ref, "variables": variables})
+ logger.info("Pipeline triggered: %s", pipeline.web_url)
+
+
+def main():
+ """Parse arguments and orchestrate the push and pipeline trigger flow."""
+ parser = argparse.ArgumentParser(
+ description="Trigger the internal GitLab CI pipeline for the current branch."
+ )
+ parser.add_argument(
+ "--gitlab-origin",
+ required=True,
+ help="Name of the git remote pointing to the internal GitLab (e.g. gitlab)",
+ )
+ parser.add_argument(
+ "--access-token",
+ default=os.environ.get("GITLAB_TOKEN"),
+ help="GitLab personal access token with 'api' scope (or set GITLAB_TOKEN env var)",
+ )
+ parser.add_argument(
+ "--functional-test-scope",
+ default="mr",
+ help="FUNCTIONAL_TEST_SCOPE pipeline variable (default: mr)",
+ )
+ parser.add_argument(
+ "--functional-test-repeat",
+ type=int,
+ default=5,
+ help="FUNCTIONAL_TEST_REPEAT pipeline variable (default: 5)",
+ )
+ parser.add_argument(
+ "--functional-test-cases",
+ default="all",
+ help="FUNCTIONAL_TEST_CASES pipeline variable (default: all)",
+ )
+ parser.add_argument(
+ "--dry-run",
+ action="store_true",
+ help="Print actions without executing git push or pipeline trigger",
+ )
+ args = parser.parse_args()
+ logging.basicConfig(level=logging.INFO, format="%(message)s")
+
+ if not args.access_token:
+ logger.error("--access-token or GITLAB_TOKEN not set")
+ sys.exit(1)
+
+ branch = get_current_branch()
+ logger.info("Current branch: %s", branch)
+
+ remote_url = get_remote_url(args.gitlab_origin)
+ gitlab_hostname = get_gitlab_hostname(remote_url)
+
+ target_branch = f"{GITLAB_BRANCH_PREFIX}/{branch}"
+
+ git_push(args.gitlab_origin, target_branch, dry_run=args.dry_run)
+
+ pipeline_vars = {
+ **PIPELINE_VARIABLES_FIXED,
+ "FUNCTIONAL_TEST_SCOPE": args.functional_test_scope,
+ "FUNCTIONAL_TEST_REPEAT": str(args.functional_test_repeat),
+ "FUNCTIONAL_TEST_CASES": args.functional_test_cases,
+ }
+
+ trigger_pipeline(
+ gitlab_hostname, args.access_token, target_branch, pipeline_vars, dry_run=args.dry_run
+ )
+
+
+if __name__ == "__main__":
+ main()
\ No newline at end of file
diff --git a/tools/upgrade_dependencies.md b/tools/upgrade_dependencies.md
new file mode 100644
index 00000000000..d33172b2001
--- /dev/null
+++ b/tools/upgrade_dependencies.md
@@ -0,0 +1,35 @@
+# upgrade_dependencies.sh
+
+Updates the `uv` lockfile, optionally upgrading all dependencies to their latest allowed versions.
+
+## Prerequisites
+
+- Docker
+- `GITLAB_ENDPOINT` environment variable (see [Setup](#setup))
+
+## Setup
+
+The script pulls the `adlr/megatron-lm/mcore_ci_dev:main` image from the internal GitLab container registry for the ADLR/Megatron-LM project. Before running, export the registry hostname (no scheme, no trailing slash):
+
+```bash
+export GITLAB_ENDPOINT=
+```
+
+Ask a team member or check your local environment / CI secrets for the correct value.
+
+## Usage
+
+```bash
+# Update lockfile only (default)
+./tools/upgrade_dependencies.sh
+
+# Upgrade all dependencies and update lockfile
+./tools/upgrade_dependencies.sh --upgrade
+```
+
+## Options
+
+| Flag | Description |
+|------|-------------|
+| `--upgrade` | Upgrade dependencies in addition to updating the lockfile |
+| `--help` | Show usage information |
diff --git a/tools/upgrade_dependencies.sh b/tools/upgrade_dependencies.sh
index 14c905f8712..d96950ad79a 100755
--- a/tools/upgrade_dependencies.sh
+++ b/tools/upgrade_dependencies.sh
@@ -7,13 +7,38 @@ if [ -z "${GITLAB_ENDPOINT:-}" ]; then
exit 1
fi
+UPGRADE=false
+
+for arg in "$@"; do
+ case $arg in
+ --upgrade) UPGRADE=true ;;
+ --help)
+ echo "Usage: $(basename $0) [--upgrade]"
+ echo ""
+ echo "Options:"
+ echo " --upgrade Upgrade dependencies in addition to updating the lockfile"
+ echo " --help Show this help message"
+ echo ""
+ echo "Environment variables:"
+ echo " GITLAB_ENDPOINT Hostname of the internal GitLab registry (no scheme, e.g. gitlab.example.com)"
+ exit 0
+ ;;
+ *) echo "Unknown argument: $arg"; exit 1 ;;
+ esac
+done
+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd $SCRIPT_DIR/..
+UV_CMD="uv lock"
+if [ "$UPGRADE" = true ]; then
+ UV_CMD="uv lock --upgrade"
+fi
+
docker run \
--rm \
-v $(pwd):/workdir/ \
-w /workdir/ \
$GITLAB_ENDPOINT/adlr/megatron-lm/mcore_ci_dev:main \
- uv lock --upgrade
+ $UV_CMD
diff --git a/train_rl.py b/train_rl.py
index cfc010b3c04..f75bba12997 100644
--- a/train_rl.py
+++ b/train_rl.py
@@ -22,9 +22,11 @@
load_packed_data_by_index,
)
from megatron.training import get_args, get_timers, pretrain, print_rank_0
+from megatron.training.utils import is_hybrid_model
from megatron.training.arguments import core_transformer_config_from_args
from model_provider import model_provider
+from megatron.core.packed_seq_params import PackedSeqParams
from megatron.rl.sequence_packing_utils import get_default_packed_seq_params
stimer = StragglerDetector()
@@ -33,7 +35,6 @@
logging.basicConfig(level=logging.INFO, force=True)
-
def _gpt_builder(args, pre_process, post_process, vp_stage=None, config=None, pg_collection=None):
# TODO(Peter): This is a hack to get around the fact that we are activation recomputation for training but not
# for inference with cuda graphs. Without out this the post checks in the transformer config will assert error.
@@ -258,11 +259,22 @@ def forward_step(data_iterator, model: GPTModel, loss_only: bool = False):
model_to_use = model[0] if isinstance(model, list) else model
if packed_seq_params is None:
- packed_seq_params = get_default_packed_seq_params(
- seq_length=tokens.shape[1],
- max_sequences_per_bin=args.rl_sequence_packing_max_sequences_per_bin,
- device=tokens.device,
- )
+ if args.rl_use_sequence_packing:
+ packed_seq_params = get_default_packed_seq_params(
+ seq_length=tokens.shape[1],
+ max_sequences_per_bin=args.rl_sequence_packing_max_sequences_per_bin,
+ device=tokens.device,
+ )
+ else:
+ cu_seqlens = torch.tensor([0, tokens.shape[1]], dtype=torch.int32, device=tokens.device)
+ packed_seq_params = PackedSeqParams(
+ qkv_format='thd',
+ cu_seqlens_q=cu_seqlens,
+ cu_seqlens_kv=cu_seqlens,
+ max_seqlen_q=tokens.shape[1],
+ max_seqlen_kv=tokens.shape[1],
+ total_tokens=tokens.shape[1],
+ )
# Clear RoPE cache to avoid inference tensor errors
try:
@@ -277,7 +289,8 @@ def forward_step(data_iterator, model: GPTModel, loss_only: bool = False):
# Get current logprobs and calculate loss with straggler detection
with stimer:
logprobs_or_hidden_states = get_logprobs(
- model_to_use, tokens, position_ids, no_grad=False, packed_seq_params=packed_seq_params
+ model_to_use, tokens, position_ids, no_grad=False,
+ packed_seq_params=packed_seq_params
)
if not is_pipeline_last_stage():
@@ -370,13 +383,15 @@ def __getitem__(self, idx):
if __name__ == "__main__":
+ from megatron.inference.utils import add_inference_args
+
# Temporary for transition to core datasets
train_valid_test_datasets_provider.is_distributed = True
def _model_builder(
args, pre_process, post_process, vp_stage=None, config=None, pg_collection=None
):
- if getattr(args, "is_hybrid_model", False):
+ if is_hybrid_model(args):
return mamba_builder(
args,
pre_process,
@@ -401,4 +416,5 @@ def _model_builder(
ModelType.encoder_or_decoder,
forward_step,
args_defaults={},
+ extra_args_provider=add_inference_args,
)
diff --git a/uv.lock b/uv.lock
index 1867e8aaddf..e1e52b25a2f 100644
--- a/uv.lock
+++ b/uv.lock
@@ -1,17 +1,52 @@
version = 1
revision = 2
-requires-python = ">=3.10"
+requires-python = ">=3.12"
resolution-markers = [
- "python_full_version >= '3.14' and sys_platform == 'linux'",
- "python_full_version == '3.13.*' and sys_platform == 'linux'",
- "python_full_version == '3.12.*' and sys_platform == 'linux'",
- "python_full_version >= '3.14' and sys_platform != 'linux'",
- "python_full_version == '3.13.*' and sys_platform != 'linux'",
- "python_full_version == '3.12.*' and sys_platform != 'linux'",
- "python_full_version == '3.11.*' and sys_platform == 'linux'",
- "python_full_version == '3.11.*' and sys_platform != 'linux'",
- "python_full_version < '3.11' and sys_platform == 'linux'",
- "python_full_version < '3.11' and sys_platform != 'linux'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
]
conflicts = [[
{ package = "megatron-core", extra = "dev" },
@@ -27,11 +62,31 @@ overrides = [
[[package]]
name = "absl-py"
-version = "2.3.1"
+version = "2.4.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/64/c7/8de93764ad66968d19329a7e0c147a2bb3c7054c554d4a119111b8f9440f/absl_py-2.4.0.tar.gz", hash = "sha256:8c6af82722b35cf71e0f4d1d47dcaebfff286e27110a99fc359349b247dfb5d4", size = 116543, upload-time = "2026-01-28T10:17:05.322Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/18/a6/907a406bb7d359e6a63f99c313846d9eec4f7e6f7437809e03aa00fa3074/absl_py-2.4.0-py3-none-any.whl", hash = "sha256:88476fd881ca8aab94ffa78b7b6c632a782ab3ba1cd19c9bd423abc4fb4cd28d", size = 135750, upload-time = "2026-01-28T10:17:04.19Z" },
+]
+
+[[package]]
+name = "accelerate"
+version = "1.13.0"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/10/2a/c93173ffa1b39c1d0395b7e842bbdc62e556ca9d8d3b5572926f3e4ca752/absl_py-2.3.1.tar.gz", hash = "sha256:a97820526f7fbfd2ec1bce83f3f25e3a14840dac0d8e02a0b71cd75db3f77fc9", size = 116588, upload-time = "2025-07-03T09:31:44.05Z" }
+dependencies = [
+ { name = "huggingface-hub" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
+ { name = "psutil" },
+ { name = "pyyaml" },
+ { name = "safetensors" },
+ { name = "torch", marker = "sys_platform == 'never' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ca/14/787e5498cd062640f0f3d92ef4ae4063174f76f9afd29d13fc52a319daae/accelerate-1.13.0.tar.gz", hash = "sha256:d631b4e0f5b3de4aff2d7e9e6857d164810dfc3237d54d017f075122d057b236", size = 402835, upload-time = "2026-03-04T19:34:12.359Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/8f/aa/ba0014cc4659328dc818a28827be78e6d97312ab0cb98105a770924dc11e/absl_py-2.3.1-py3-none-any.whl", hash = "sha256:eeecf07f0c2a93ace0772c92e596ace6d3d3996c042b2128459aaae2a76de11d", size = 135811, upload-time = "2025-07-03T09:31:42.253Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/46/02ac5e262d4af18054b3e922b2baedbb2a03289ee792162de60a865defc5/accelerate-1.13.0-py3-none-any.whl", hash = "sha256:cf1a3efb96c18f7b152eb0fa7490f3710b19c3f395699358f08decca2b8b62e0", size = 383744, upload-time = "2026-03-04T19:34:10.313Z" },
]
[[package]]
@@ -48,7 +103,7 @@ wheels = [
[[package]]
name = "aiobotocore"
-version = "2.26.0"
+version = "3.3.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "aiohttp" },
@@ -59,9 +114,18 @@ dependencies = [
{ name = "python-dateutil" },
{ name = "wrapt" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/4d/f8/99fa90d9c25b78292899fd4946fce97b6353838b5ecc139ad8ba1436e70c/aiobotocore-2.26.0.tar.gz", hash = "sha256:50567feaf8dfe2b653570b4491f5bc8c6e7fb9622479d66442462c021db4fadc", size = 122026, upload-time = "2025-11-28T07:54:59.956Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/71/9f/a0568deaf008f4a7e3d57a7f80f1537df894df0e49bd4a790bb22f9a2d8e/aiobotocore-3.3.0.tar.gz", hash = "sha256:9abc21d91edd6c9c2e4a07e11bdfcbb159f0b9116ab2a0a5a349113533a18fb2", size = 122940, upload-time = "2026-03-18T09:58:49.077Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/b7/58/3bf0b7d474607dc7fd67dd1365c4e0f392c8177eaf4054e5ddee3ebd53b5/aiobotocore-2.26.0-py3-none-any.whl", hash = "sha256:a793db51c07930513b74ea7a95bd79aaa42f545bdb0f011779646eafa216abec", size = 87333, upload-time = "2025-11-28T07:54:58.457Z" },
+ { url = "https://files.pythonhosted.org/packages/16/54/a295bd8d7ac900c339b2c7024ed0ff9538afb60e92eb0979b8bb49deb20e/aiobotocore-3.3.0-py3-none-any.whl", hash = "sha256:9125ab2b63740dfe3b66b8d5a90d13aed9587b850aa53225ef214a04a1aa7fdc", size = 87817, upload-time = "2026-03-18T09:58:47.466Z" },
+]
+
+[[package]]
+name = "aiofiles"
+version = "25.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/41/c3/534eac40372d8ee36ef40df62ec129bee4fdb5ad9706e58a29be53b2c970/aiofiles-25.1.0.tar.gz", hash = "sha256:a8d728f0a29de45dc521f18f07297428d56992a742f0cd2701ba86e44d23d5b2", size = 46354, upload-time = "2025-10-09T20:51:04.358Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/bc/8a/340a1555ae33d7354dbca4faa54948d76d89a27ceef032c8c3bc661d003e/aiofiles-25.1.0-py3-none-any.whl", hash = "sha256:abe311e527c862958650f9438e859c1fa7568a141b22abcd015e120e86a85695", size = 14668, upload-time = "2025-10-09T20:51:03.174Z" },
]
[[package]]
@@ -75,122 +139,99 @@ wheels = [
[[package]]
name = "aiohttp"
-version = "3.13.3"
+version = "3.13.4"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "aiohappyeyeballs" },
{ name = "aiosignal" },
- { name = "async-timeout", marker = "python_full_version < '3.11'" },
{ name = "attrs" },
{ name = "frozenlist" },
{ name = "multidict" },
{ name = "propcache" },
{ name = "yarl" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/50/42/32cf8e7704ceb4481406eb87161349abb46a57fee3f008ba9cb610968646/aiohttp-3.13.3.tar.gz", hash = "sha256:a949eee43d3782f2daae4f4a2819b2cb9b0c5d3b7f7a927067cc84dafdbb9f88", size = 7844556, upload-time = "2026-01-03T17:33:05.204Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/36/d6/5aec9313ee6ea9c7cde8b891b69f4ff4001416867104580670a31daeba5b/aiohttp-3.13.3-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:d5a372fd5afd301b3a89582817fdcdb6c34124787c70dbcc616f259013e7eef7", size = 738950, upload-time = "2026-01-03T17:29:13.002Z" },
- { url = "https://files.pythonhosted.org/packages/68/03/8fa90a7e6d11ff20a18837a8e2b5dd23db01aabc475aa9271c8ad33299f5/aiohttp-3.13.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:147e422fd1223005c22b4fe080f5d93ced44460f5f9c105406b753612b587821", size = 496099, upload-time = "2026-01-03T17:29:15.268Z" },
- { url = "https://files.pythonhosted.org/packages/d2/23/b81f744d402510a8366b74eb420fc0cc1170d0c43daca12d10814df85f10/aiohttp-3.13.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:859bd3f2156e81dd01432f5849fc73e2243d4a487c4fd26609b1299534ee1845", size = 491072, upload-time = "2026-01-03T17:29:16.922Z" },
- { url = "https://files.pythonhosted.org/packages/d5/e1/56d1d1c0dd334cd203dd97706ce004c1aa24b34a813b0b8daf3383039706/aiohttp-3.13.3-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dca68018bf48c251ba17c72ed479f4dafe9dbd5a73707ad8d28a38d11f3d42af", size = 1671588, upload-time = "2026-01-03T17:29:18.539Z" },
- { url = "https://files.pythonhosted.org/packages/5f/34/8d7f962604f4bc2b4e39eb1220dac7d4e4cba91fb9ba0474b4ecd67db165/aiohttp-3.13.3-cp310-cp310-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:fee0c6bc7db1de362252affec009707a17478a00ec69f797d23ca256e36d5940", size = 1640334, upload-time = "2026-01-03T17:29:21.028Z" },
- { url = "https://files.pythonhosted.org/packages/94/1d/fcccf2c668d87337ddeef9881537baee13c58d8f01f12ba8a24215f2b804/aiohttp-3.13.3-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c048058117fd649334d81b4b526e94bde3ccaddb20463a815ced6ecbb7d11160", size = 1722656, upload-time = "2026-01-03T17:29:22.531Z" },
- { url = "https://files.pythonhosted.org/packages/aa/98/c6f3b081c4c606bc1e5f2ec102e87d6411c73a9ef3616fea6f2d5c98c062/aiohttp-3.13.3-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:215a685b6fbbfcf71dfe96e3eba7a6f58f10da1dfdf4889c7dd856abe430dca7", size = 1817625, upload-time = "2026-01-03T17:29:24.276Z" },
- { url = "https://files.pythonhosted.org/packages/2c/c0/cfcc3d2e11b477f86e1af2863f3858c8850d751ce8dc39c4058a072c9e54/aiohttp-3.13.3-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:de2c184bb1fe2cbd2cefba613e9db29a5ab559323f994b6737e370d3da0ac455", size = 1672604, upload-time = "2026-01-03T17:29:26.099Z" },
- { url = "https://files.pythonhosted.org/packages/1e/77/6b4ffcbcac4c6a5d041343a756f34a6dd26174ae07f977a64fe028dda5b0/aiohttp-3.13.3-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:75ca857eba4e20ce9f546cd59c7007b33906a4cd48f2ff6ccf1ccfc3b646f279", size = 1554370, upload-time = "2026-01-03T17:29:28.121Z" },
- { url = "https://files.pythonhosted.org/packages/f2/f0/e3ddfa93f17d689dbe014ba048f18e0c9f9b456033b70e94349a2e9048be/aiohttp-3.13.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:81e97251d9298386c2b7dbeb490d3d1badbdc69107fb8c9299dd04eb39bddc0e", size = 1642023, upload-time = "2026-01-03T17:29:30.002Z" },
- { url = "https://files.pythonhosted.org/packages/eb/45/c14019c9ec60a8e243d06d601b33dcc4fd92379424bde3021725859d7f99/aiohttp-3.13.3-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:c0e2d366af265797506f0283487223146af57815b388623f0357ef7eac9b209d", size = 1649680, upload-time = "2026-01-03T17:29:31.782Z" },
- { url = "https://files.pythonhosted.org/packages/9c/fd/09c9451dae5aa5c5ed756df95ff9ef549d45d4be663bafd1e4954fd836f0/aiohttp-3.13.3-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:4e239d501f73d6db1522599e14b9b321a7e3b1de66ce33d53a765d975e9f4808", size = 1692407, upload-time = "2026-01-03T17:29:33.392Z" },
- { url = "https://files.pythonhosted.org/packages/a6/81/938bc2ec33c10efd6637ccb3d22f9f3160d08e8f3aa2587a2c2d5ab578eb/aiohttp-3.13.3-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:0db318f7a6f065d84cb1e02662c526294450b314a02bd9e2a8e67f0d8564ce40", size = 1543047, upload-time = "2026-01-03T17:29:34.855Z" },
- { url = "https://files.pythonhosted.org/packages/f7/23/80488ee21c8d567c83045e412e1d9b7077d27171591a4eb7822586e8c06a/aiohttp-3.13.3-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:bfc1cc2fe31a6026a8a88e4ecfb98d7f6b1fec150cfd708adbfd1d2f42257c29", size = 1715264, upload-time = "2026-01-03T17:29:36.389Z" },
- { url = "https://files.pythonhosted.org/packages/e2/83/259a8da6683182768200b368120ab3deff5370bed93880fb9a3a86299f34/aiohttp-3.13.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:af71fff7bac6bb7508956696dce8f6eec2bbb045eceb40343944b1ae62b5ef11", size = 1657275, upload-time = "2026-01-03T17:29:38.162Z" },
- { url = "https://files.pythonhosted.org/packages/3f/4f/2c41f800a0b560785c10fb316216ac058c105f9be50bdc6a285de88db625/aiohttp-3.13.3-cp310-cp310-win32.whl", hash = "sha256:37da61e244d1749798c151421602884db5270faf479cf0ef03af0ff68954c9dd", size = 434053, upload-time = "2026-01-03T17:29:40.074Z" },
- { url = "https://files.pythonhosted.org/packages/80/df/29cd63c7ecfdb65ccc12f7d808cac4fa2a19544660c06c61a4a48462de0c/aiohttp-3.13.3-cp310-cp310-win_amd64.whl", hash = "sha256:7e63f210bc1b57ef699035f2b4b6d9ce096b5914414a49b0997c839b2bd2223c", size = 456687, upload-time = "2026-01-03T17:29:41.819Z" },
- { url = "https://files.pythonhosted.org/packages/f1/4c/a164164834f03924d9a29dc3acd9e7ee58f95857e0b467f6d04298594ebb/aiohttp-3.13.3-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:5b6073099fb654e0a068ae678b10feff95c5cae95bbfcbfa7af669d361a8aa6b", size = 746051, upload-time = "2026-01-03T17:29:43.287Z" },
- { url = "https://files.pythonhosted.org/packages/82/71/d5c31390d18d4f58115037c432b7e0348c60f6f53b727cad33172144a112/aiohttp-3.13.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:1cb93e166e6c28716c8c6aeb5f99dfb6d5ccf482d29fe9bf9a794110e6d0ab64", size = 499234, upload-time = "2026-01-03T17:29:44.822Z" },
- { url = "https://files.pythonhosted.org/packages/0e/c9/741f8ac91e14b1d2e7100690425a5b2b919a87a5075406582991fb7de920/aiohttp-3.13.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:28e027cf2f6b641693a09f631759b4d9ce9165099d2b5d92af9bd4e197690eea", size = 494979, upload-time = "2026-01-03T17:29:46.405Z" },
- { url = "https://files.pythonhosted.org/packages/75/b5/31d4d2e802dfd59f74ed47eba48869c1c21552c586d5e81a9d0d5c2ad640/aiohttp-3.13.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3b61b7169ababd7802f9568ed96142616a9118dd2be0d1866e920e77ec8fa92a", size = 1748297, upload-time = "2026-01-03T17:29:48.083Z" },
- { url = "https://files.pythonhosted.org/packages/1a/3e/eefad0ad42959f226bb79664826883f2687d602a9ae2941a18e0484a74d3/aiohttp-3.13.3-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:80dd4c21b0f6237676449c6baaa1039abae86b91636b6c91a7f8e61c87f89540", size = 1707172, upload-time = "2026-01-03T17:29:49.648Z" },
- { url = "https://files.pythonhosted.org/packages/c5/3a/54a64299fac2891c346cdcf2aa6803f994a2e4beeaf2e5a09dcc54acc842/aiohttp-3.13.3-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:65d2ccb7eabee90ce0503c17716fc77226be026dcc3e65cce859a30db715025b", size = 1805405, upload-time = "2026-01-03T17:29:51.244Z" },
- { url = "https://files.pythonhosted.org/packages/6c/70/ddc1b7169cf64075e864f64595a14b147a895a868394a48f6a8031979038/aiohttp-3.13.3-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5b179331a481cb5529fca8b432d8d3c7001cb217513c94cd72d668d1248688a3", size = 1899449, upload-time = "2026-01-03T17:29:53.938Z" },
- { url = "https://files.pythonhosted.org/packages/a1/7e/6815aab7d3a56610891c76ef79095677b8b5be6646aaf00f69b221765021/aiohttp-3.13.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d4c940f02f49483b18b079d1c27ab948721852b281f8b015c058100e9421dd1", size = 1748444, upload-time = "2026-01-03T17:29:55.484Z" },
- { url = "https://files.pythonhosted.org/packages/6b/f2/073b145c4100da5511f457dc0f7558e99b2987cf72600d42b559db856fbc/aiohttp-3.13.3-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f9444f105664c4ce47a2a7171a2418bce5b7bae45fb610f4e2c36045d85911d3", size = 1606038, upload-time = "2026-01-03T17:29:57.179Z" },
- { url = "https://files.pythonhosted.org/packages/0a/c1/778d011920cae03ae01424ec202c513dc69243cf2db303965615b81deeea/aiohttp-3.13.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:694976222c711d1d00ba131904beb60534f93966562f64440d0c9d41b8cdb440", size = 1724156, upload-time = "2026-01-03T17:29:58.914Z" },
- { url = "https://files.pythonhosted.org/packages/0e/cb/3419eabf4ec1e9ec6f242c32b689248365a1cf621891f6f0386632525494/aiohttp-3.13.3-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:f33ed1a2bf1997a36661874b017f5c4b760f41266341af36febaf271d179f6d7", size = 1722340, upload-time = "2026-01-03T17:30:01.962Z" },
- { url = "https://files.pythonhosted.org/packages/7a/e5/76cf77bdbc435bf233c1f114edad39ed4177ccbfab7c329482b179cff4f4/aiohttp-3.13.3-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:e636b3c5f61da31a92bf0d91da83e58fdfa96f178ba682f11d24f31944cdd28c", size = 1783041, upload-time = "2026-01-03T17:30:03.609Z" },
- { url = "https://files.pythonhosted.org/packages/9d/d4/dd1ca234c794fd29c057ce8c0566b8ef7fd6a51069de5f06fa84b9a1971c/aiohttp-3.13.3-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:5d2d94f1f5fcbe40838ac51a6ab5704a6f9ea42e72ceda48de5e6b898521da51", size = 1596024, upload-time = "2026-01-03T17:30:05.132Z" },
- { url = "https://files.pythonhosted.org/packages/55/58/4345b5f26661a6180afa686c473620c30a66afdf120ed3dd545bbc809e85/aiohttp-3.13.3-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:2be0e9ccf23e8a94f6f0650ce06042cefc6ac703d0d7ab6c7a917289f2539ad4", size = 1804590, upload-time = "2026-01-03T17:30:07.135Z" },
- { url = "https://files.pythonhosted.org/packages/7b/06/05950619af6c2df7e0a431d889ba2813c9f0129cec76f663e547a5ad56f2/aiohttp-3.13.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:9af5e68ee47d6534d36791bbe9b646d2a7c7deb6fc24d7943628edfbb3581f29", size = 1740355, upload-time = "2026-01-03T17:30:09.083Z" },
- { url = "https://files.pythonhosted.org/packages/3e/80/958f16de79ba0422d7c1e284b2abd0c84bc03394fbe631d0a39ffa10e1eb/aiohttp-3.13.3-cp311-cp311-win32.whl", hash = "sha256:a2212ad43c0833a873d0fb3c63fa1bacedd4cf6af2fee62bf4b739ceec3ab239", size = 433701, upload-time = "2026-01-03T17:30:10.869Z" },
- { url = "https://files.pythonhosted.org/packages/dc/f2/27cdf04c9851712d6c1b99df6821a6623c3c9e55956d4b1e318c337b5a48/aiohttp-3.13.3-cp311-cp311-win_amd64.whl", hash = "sha256:642f752c3eb117b105acbd87e2c143de710987e09860d674e068c4c2c441034f", size = 457678, upload-time = "2026-01-03T17:30:12.719Z" },
- { url = "https://files.pythonhosted.org/packages/a0/be/4fc11f202955a69e0db803a12a062b8379c970c7c84f4882b6da17337cc1/aiohttp-3.13.3-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:b903a4dfee7d347e2d87697d0713be59e0b87925be030c9178c5faa58ea58d5c", size = 739732, upload-time = "2026-01-03T17:30:14.23Z" },
- { url = "https://files.pythonhosted.org/packages/97/2c/621d5b851f94fa0bb7430d6089b3aa970a9d9b75196bc93bb624b0db237a/aiohttp-3.13.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:a45530014d7a1e09f4a55f4f43097ba0fd155089372e105e4bff4ca76cb1b168", size = 494293, upload-time = "2026-01-03T17:30:15.96Z" },
- { url = "https://files.pythonhosted.org/packages/5d/43/4be01406b78e1be8320bb8316dc9c42dbab553d281c40364e0f862d5661c/aiohttp-3.13.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:27234ef6d85c914f9efeb77ff616dbf4ad2380be0cda40b4db086ffc7ddd1b7d", size = 493533, upload-time = "2026-01-03T17:30:17.431Z" },
- { url = "https://files.pythonhosted.org/packages/8d/a8/5a35dc56a06a2c90d4742cbf35294396907027f80eea696637945a106f25/aiohttp-3.13.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d32764c6c9aafb7fb55366a224756387cd50bfa720f32b88e0e6fa45b27dcf29", size = 1737839, upload-time = "2026-01-03T17:30:19.422Z" },
- { url = "https://files.pythonhosted.org/packages/bf/62/4b9eeb331da56530bf2e198a297e5303e1c1ebdceeb00fe9b568a65c5a0c/aiohttp-3.13.3-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:b1a6102b4d3ebc07dad44fbf07b45bb600300f15b552ddf1851b5390202ea2e3", size = 1703932, upload-time = "2026-01-03T17:30:21.756Z" },
- { url = "https://files.pythonhosted.org/packages/7c/f6/af16887b5d419e6a367095994c0b1332d154f647e7dc2bd50e61876e8e3d/aiohttp-3.13.3-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c014c7ea7fb775dd015b2d3137378b7be0249a448a1612268b5a90c2d81de04d", size = 1771906, upload-time = "2026-01-03T17:30:23.932Z" },
- { url = "https://files.pythonhosted.org/packages/ce/83/397c634b1bcc24292fa1e0c7822800f9f6569e32934bdeef09dae7992dfb/aiohttp-3.13.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2b8d8ddba8f95ba17582226f80e2de99c7a7948e66490ef8d947e272a93e9463", size = 1871020, upload-time = "2026-01-03T17:30:26Z" },
- { url = "https://files.pythonhosted.org/packages/86/f6/a62cbbf13f0ac80a70f71b1672feba90fdb21fd7abd8dbf25c0105fb6fa3/aiohttp-3.13.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9ae8dd55c8e6c4257eae3a20fd2c8f41edaea5992ed67156642493b8daf3cecc", size = 1755181, upload-time = "2026-01-03T17:30:27.554Z" },
- { url = "https://files.pythonhosted.org/packages/0a/87/20a35ad487efdd3fba93d5843efdfaa62d2f1479eaafa7453398a44faf13/aiohttp-3.13.3-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:01ad2529d4b5035578f5081606a465f3b814c542882804e2e8cda61adf5c71bf", size = 1561794, upload-time = "2026-01-03T17:30:29.254Z" },
- { url = "https://files.pythonhosted.org/packages/de/95/8fd69a66682012f6716e1bc09ef8a1a2a91922c5725cb904689f112309c4/aiohttp-3.13.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:bb4f7475e359992b580559e008c598091c45b5088f28614e855e42d39c2f1033", size = 1697900, upload-time = "2026-01-03T17:30:31.033Z" },
- { url = "https://files.pythonhosted.org/packages/e5/66/7b94b3b5ba70e955ff597672dad1691333080e37f50280178967aff68657/aiohttp-3.13.3-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:c19b90316ad3b24c69cd78d5c9b4f3aa4497643685901185b65166293d36a00f", size = 1728239, upload-time = "2026-01-03T17:30:32.703Z" },
- { url = "https://files.pythonhosted.org/packages/47/71/6f72f77f9f7d74719692ab65a2a0252584bf8d5f301e2ecb4c0da734530a/aiohttp-3.13.3-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:96d604498a7c782cb15a51c406acaea70d8c027ee6b90c569baa6e7b93073679", size = 1740527, upload-time = "2026-01-03T17:30:34.695Z" },
- { url = "https://files.pythonhosted.org/packages/fa/b4/75ec16cbbd5c01bdaf4a05b19e103e78d7ce1ef7c80867eb0ace42ff4488/aiohttp-3.13.3-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:084911a532763e9d3dd95adf78a78f4096cd5f58cdc18e6fdbc1b58417a45423", size = 1554489, upload-time = "2026-01-03T17:30:36.864Z" },
- { url = "https://files.pythonhosted.org/packages/52/8f/bc518c0eea29f8406dcf7ed1f96c9b48e3bc3995a96159b3fc11f9e08321/aiohttp-3.13.3-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:7a4a94eb787e606d0a09404b9c38c113d3b099d508021faa615d70a0131907ce", size = 1767852, upload-time = "2026-01-03T17:30:39.433Z" },
- { url = "https://files.pythonhosted.org/packages/9d/f2/a07a75173124f31f11ea6f863dc44e6f09afe2bca45dd4e64979490deab1/aiohttp-3.13.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:87797e645d9d8e222e04160ee32aa06bc5c163e8499f24db719e7852ec23093a", size = 1722379, upload-time = "2026-01-03T17:30:41.081Z" },
- { url = "https://files.pythonhosted.org/packages/3c/4a/1a3fee7c21350cac78e5c5cef711bac1b94feca07399f3d406972e2d8fcd/aiohttp-3.13.3-cp312-cp312-win32.whl", hash = "sha256:b04be762396457bef43f3597c991e192ee7da460a4953d7e647ee4b1c28e7046", size = 428253, upload-time = "2026-01-03T17:30:42.644Z" },
- { url = "https://files.pythonhosted.org/packages/d9/b7/76175c7cb4eb73d91ad63c34e29fc4f77c9386bba4a65b53ba8e05ee3c39/aiohttp-3.13.3-cp312-cp312-win_amd64.whl", hash = "sha256:e3531d63d3bdfa7e3ac5e9b27b2dd7ec9df3206a98e0b3445fa906f233264c57", size = 455407, upload-time = "2026-01-03T17:30:44.195Z" },
- { url = "https://files.pythonhosted.org/packages/97/8a/12ca489246ca1faaf5432844adbfce7ff2cc4997733e0af120869345643a/aiohttp-3.13.3-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:5dff64413671b0d3e7d5918ea490bdccb97a4ad29b3f311ed423200b2203e01c", size = 734190, upload-time = "2026-01-03T17:30:45.832Z" },
- { url = "https://files.pythonhosted.org/packages/32/08/de43984c74ed1fca5c014808963cc83cb00d7bb06af228f132d33862ca76/aiohttp-3.13.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:87b9aab6d6ed88235aa2970294f496ff1a1f9adcd724d800e9b952395a80ffd9", size = 491783, upload-time = "2026-01-03T17:30:47.466Z" },
- { url = "https://files.pythonhosted.org/packages/17/f8/8dd2cf6112a5a76f81f81a5130c57ca829d101ad583ce57f889179accdda/aiohttp-3.13.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:425c126c0dc43861e22cb1c14ba4c8e45d09516d0a3ae0a3f7494b79f5f233a3", size = 490704, upload-time = "2026-01-03T17:30:49.373Z" },
- { url = "https://files.pythonhosted.org/packages/6d/40/a46b03ca03936f832bc7eaa47cfbb1ad012ba1be4790122ee4f4f8cba074/aiohttp-3.13.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7f9120f7093c2a32d9647abcaf21e6ad275b4fbec5b55969f978b1a97c7c86bf", size = 1720652, upload-time = "2026-01-03T17:30:50.974Z" },
- { url = "https://files.pythonhosted.org/packages/f7/7e/917fe18e3607af92657e4285498f500dca797ff8c918bd7d90b05abf6c2a/aiohttp-3.13.3-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:697753042d57f4bf7122cab985bf15d0cef23c770864580f5af4f52023a56bd6", size = 1692014, upload-time = "2026-01-03T17:30:52.729Z" },
- { url = "https://files.pythonhosted.org/packages/71/b6/cefa4cbc00d315d68973b671cf105b21a609c12b82d52e5d0c9ae61d2a09/aiohttp-3.13.3-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6de499a1a44e7de70735d0b39f67c8f25eb3d91eb3103be99ca0fa882cdd987d", size = 1759777, upload-time = "2026-01-03T17:30:54.537Z" },
- { url = "https://files.pythonhosted.org/packages/fb/e3/e06ee07b45e59e6d81498b591fc589629be1553abb2a82ce33efe2a7b068/aiohttp-3.13.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:37239e9f9a7ea9ac5bf6b92b0260b01f8a22281996da609206a84df860bc1261", size = 1861276, upload-time = "2026-01-03T17:30:56.512Z" },
- { url = "https://files.pythonhosted.org/packages/7c/24/75d274228acf35ceeb2850b8ce04de9dd7355ff7a0b49d607ee60c29c518/aiohttp-3.13.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f76c1e3fe7d7c8afad7ed193f89a292e1999608170dcc9751a7462a87dfd5bc0", size = 1743131, upload-time = "2026-01-03T17:30:58.256Z" },
- { url = "https://files.pythonhosted.org/packages/04/98/3d21dde21889b17ca2eea54fdcff21b27b93f45b7bb94ca029c31ab59dc3/aiohttp-3.13.3-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:fc290605db2a917f6e81b0e1e0796469871f5af381ce15c604a3c5c7e51cb730", size = 1556863, upload-time = "2026-01-03T17:31:00.445Z" },
- { url = "https://files.pythonhosted.org/packages/9e/84/da0c3ab1192eaf64782b03971ab4055b475d0db07b17eff925e8c93b3aa5/aiohttp-3.13.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4021b51936308aeea0367b8f006dc999ca02bc118a0cc78c303f50a2ff6afb91", size = 1682793, upload-time = "2026-01-03T17:31:03.024Z" },
- { url = "https://files.pythonhosted.org/packages/ff/0f/5802ada182f575afa02cbd0ec5180d7e13a402afb7c2c03a9aa5e5d49060/aiohttp-3.13.3-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:49a03727c1bba9a97d3e93c9f93ca03a57300f484b6e935463099841261195d3", size = 1716676, upload-time = "2026-01-03T17:31:04.842Z" },
- { url = "https://files.pythonhosted.org/packages/3f/8c/714d53bd8b5a4560667f7bbbb06b20c2382f9c7847d198370ec6526af39c/aiohttp-3.13.3-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:3d9908a48eb7416dc1f4524e69f1d32e5d90e3981e4e37eb0aa1cd18f9cfa2a4", size = 1733217, upload-time = "2026-01-03T17:31:06.868Z" },
- { url = "https://files.pythonhosted.org/packages/7d/79/e2176f46d2e963facea939f5be2d26368ce543622be6f00a12844d3c991f/aiohttp-3.13.3-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:2712039939ec963c237286113c68dbad80a82a4281543f3abf766d9d73228998", size = 1552303, upload-time = "2026-01-03T17:31:08.958Z" },
- { url = "https://files.pythonhosted.org/packages/ab/6a/28ed4dea1759916090587d1fe57087b03e6c784a642b85ef48217b0277ae/aiohttp-3.13.3-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:7bfdc049127717581866fa4708791220970ce291c23e28ccf3922c700740fdc0", size = 1763673, upload-time = "2026-01-03T17:31:10.676Z" },
- { url = "https://files.pythonhosted.org/packages/e8/35/4a3daeb8b9fab49240d21c04d50732313295e4bd813a465d840236dd0ce1/aiohttp-3.13.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:8057c98e0c8472d8846b9c79f56766bcc57e3e8ac7bfd510482332366c56c591", size = 1721120, upload-time = "2026-01-03T17:31:12.575Z" },
- { url = "https://files.pythonhosted.org/packages/bc/9f/d643bb3c5fb99547323e635e251c609fbbc660d983144cfebec529e09264/aiohttp-3.13.3-cp313-cp313-win32.whl", hash = "sha256:1449ceddcdbcf2e0446957863af03ebaaa03f94c090f945411b61269e2cb5daf", size = 427383, upload-time = "2026-01-03T17:31:14.382Z" },
- { url = "https://files.pythonhosted.org/packages/4e/f1/ab0395f8a79933577cdd996dd2f9aa6014af9535f65dddcf88204682fe62/aiohttp-3.13.3-cp313-cp313-win_amd64.whl", hash = "sha256:693781c45a4033d31d4187d2436f5ac701e7bbfe5df40d917736108c1cc7436e", size = 453899, upload-time = "2026-01-03T17:31:15.958Z" },
- { url = "https://files.pythonhosted.org/packages/99/36/5b6514a9f5d66f4e2597e40dea2e3db271e023eb7a5d22defe96ba560996/aiohttp-3.13.3-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:ea37047c6b367fd4bd632bff8077449b8fa034b69e812a18e0132a00fae6e808", size = 737238, upload-time = "2026-01-03T17:31:17.909Z" },
- { url = "https://files.pythonhosted.org/packages/f7/49/459327f0d5bcd8c6c9ca69e60fdeebc3622861e696490d8674a6d0cb90a6/aiohttp-3.13.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:6fc0e2337d1a4c3e6acafda6a78a39d4c14caea625124817420abceed36e2415", size = 492292, upload-time = "2026-01-03T17:31:19.919Z" },
- { url = "https://files.pythonhosted.org/packages/e8/0b/b97660c5fd05d3495b4eb27f2d0ef18dc1dc4eff7511a9bf371397ff0264/aiohttp-3.13.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c685f2d80bb67ca8c3837823ad76196b3694b0159d232206d1e461d3d434666f", size = 493021, upload-time = "2026-01-03T17:31:21.636Z" },
- { url = "https://files.pythonhosted.org/packages/54/d4/438efabdf74e30aeceb890c3290bbaa449780583b1270b00661126b8aae4/aiohttp-3.13.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:48e377758516d262bde50c2584fc6c578af272559c409eecbdd2bae1601184d6", size = 1717263, upload-time = "2026-01-03T17:31:23.296Z" },
- { url = "https://files.pythonhosted.org/packages/71/f2/7bddc7fd612367d1459c5bcf598a9e8f7092d6580d98de0e057eb42697ad/aiohttp-3.13.3-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:34749271508078b261c4abb1767d42b8d0c0cc9449c73a4df494777dc55f0687", size = 1669107, upload-time = "2026-01-03T17:31:25.334Z" },
- { url = "https://files.pythonhosted.org/packages/00/5a/1aeaecca40e22560f97610a329e0e5efef5e0b5afdf9f857f0d93839ab2e/aiohttp-3.13.3-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:82611aeec80eb144416956ec85b6ca45a64d76429c1ed46ae1b5f86c6e0c9a26", size = 1760196, upload-time = "2026-01-03T17:31:27.394Z" },
- { url = "https://files.pythonhosted.org/packages/f8/f8/0ff6992bea7bd560fc510ea1c815f87eedd745fe035589c71ce05612a19a/aiohttp-3.13.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2fff83cfc93f18f215896e3a190e8e5cb413ce01553901aca925176e7568963a", size = 1843591, upload-time = "2026-01-03T17:31:29.238Z" },
- { url = "https://files.pythonhosted.org/packages/e3/d1/e30e537a15f53485b61f5be525f2157da719819e8377298502aebac45536/aiohttp-3.13.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bbe7d4cecacb439e2e2a8a1a7b935c25b812af7a5fd26503a66dadf428e79ec1", size = 1720277, upload-time = "2026-01-03T17:31:31.053Z" },
- { url = "https://files.pythonhosted.org/packages/84/45/23f4c451d8192f553d38d838831ebbc156907ea6e05557f39563101b7717/aiohttp-3.13.3-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b928f30fe49574253644b1ca44b1b8adbd903aa0da4b9054a6c20fc7f4092a25", size = 1548575, upload-time = "2026-01-03T17:31:32.87Z" },
- { url = "https://files.pythonhosted.org/packages/6a/ed/0a42b127a43712eda7807e7892c083eadfaf8429ca8fb619662a530a3aab/aiohttp-3.13.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7b5e8fe4de30df199155baaf64f2fcd604f4c678ed20910db8e2c66dc4b11603", size = 1679455, upload-time = "2026-01-03T17:31:34.76Z" },
- { url = "https://files.pythonhosted.org/packages/2e/b5/c05f0c2b4b4fe2c9d55e73b6d3ed4fd6c9dc2684b1d81cbdf77e7fad9adb/aiohttp-3.13.3-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:8542f41a62bcc58fc7f11cf7c90e0ec324ce44950003feb70640fc2a9092c32a", size = 1687417, upload-time = "2026-01-03T17:31:36.699Z" },
- { url = "https://files.pythonhosted.org/packages/c9/6b/915bc5dad66aef602b9e459b5a973529304d4e89ca86999d9d75d80cbd0b/aiohttp-3.13.3-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:5e1d8c8b8f1d91cd08d8f4a3c2b067bfca6ec043d3ff36de0f3a715feeedf926", size = 1729968, upload-time = "2026-01-03T17:31:38.622Z" },
- { url = "https://files.pythonhosted.org/packages/11/3b/e84581290a9520024a08640b63d07673057aec5ca548177a82026187ba73/aiohttp-3.13.3-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:90455115e5da1c3c51ab619ac57f877da8fd6d73c05aacd125c5ae9819582aba", size = 1545690, upload-time = "2026-01-03T17:31:40.57Z" },
- { url = "https://files.pythonhosted.org/packages/f5/04/0c3655a566c43fd647c81b895dfe361b9f9ad6d58c19309d45cff52d6c3b/aiohttp-3.13.3-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:042e9e0bcb5fba81886c8b4fbb9a09d6b8a00245fd8d88e4d989c1f96c74164c", size = 1746390, upload-time = "2026-01-03T17:31:42.857Z" },
- { url = "https://files.pythonhosted.org/packages/1f/53/71165b26978f719c3419381514c9690bd5980e764a09440a10bb816ea4ab/aiohttp-3.13.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:2eb752b102b12a76ca02dff751a801f028b4ffbbc478840b473597fc91a9ed43", size = 1702188, upload-time = "2026-01-03T17:31:44.984Z" },
- { url = "https://files.pythonhosted.org/packages/29/a7/cbe6c9e8e136314fa1980da388a59d2f35f35395948a08b6747baebb6aa6/aiohttp-3.13.3-cp314-cp314-win32.whl", hash = "sha256:b556c85915d8efaed322bf1bdae9486aa0f3f764195a0fb6ee962e5c71ef5ce1", size = 433126, upload-time = "2026-01-03T17:31:47.463Z" },
- { url = "https://files.pythonhosted.org/packages/de/56/982704adea7d3b16614fc5936014e9af85c0e34b58f9046655817f04306e/aiohttp-3.13.3-cp314-cp314-win_amd64.whl", hash = "sha256:9bf9f7a65e7aa20dd764151fb3d616c81088f91f8df39c3893a536e279b4b984", size = 459128, upload-time = "2026-01-03T17:31:49.2Z" },
- { url = "https://files.pythonhosted.org/packages/6c/2a/3c79b638a9c3d4658d345339d22070241ea341ed4e07b5ac60fb0f418003/aiohttp-3.13.3-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:05861afbbec40650d8a07ea324367cb93e9e8cc7762e04dd4405df99fa65159c", size = 769512, upload-time = "2026-01-03T17:31:51.134Z" },
- { url = "https://files.pythonhosted.org/packages/29/b9/3e5014d46c0ab0db8707e0ac2711ed28c4da0218c358a4e7c17bae0d8722/aiohttp-3.13.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:2fc82186fadc4a8316768d61f3722c230e2c1dcab4200d52d2ebdf2482e47592", size = 506444, upload-time = "2026-01-03T17:31:52.85Z" },
- { url = "https://files.pythonhosted.org/packages/90/03/c1d4ef9a054e151cd7839cdc497f2638f00b93cbe8043983986630d7a80c/aiohttp-3.13.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:0add0900ff220d1d5c5ebbf99ed88b0c1bbf87aa7e4262300ed1376a6b13414f", size = 510798, upload-time = "2026-01-03T17:31:54.91Z" },
- { url = "https://files.pythonhosted.org/packages/ea/76/8c1e5abbfe8e127c893fe7ead569148a4d5a799f7cf958d8c09f3eedf097/aiohttp-3.13.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:568f416a4072fbfae453dcf9a99194bbb8bdeab718e08ee13dfa2ba0e4bebf29", size = 1868835, upload-time = "2026-01-03T17:31:56.733Z" },
- { url = "https://files.pythonhosted.org/packages/8e/ac/984c5a6f74c363b01ff97adc96a3976d9c98940b8969a1881575b279ac5d/aiohttp-3.13.3-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:add1da70de90a2569c5e15249ff76a631ccacfe198375eead4aadf3b8dc849dc", size = 1720486, upload-time = "2026-01-03T17:31:58.65Z" },
- { url = "https://files.pythonhosted.org/packages/b2/9a/b7039c5f099c4eb632138728828b33428585031a1e658d693d41d07d89d1/aiohttp-3.13.3-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:10b47b7ba335d2e9b1239fa571131a87e2d8ec96b333e68b2a305e7a98b0bae2", size = 1847951, upload-time = "2026-01-03T17:32:00.989Z" },
- { url = "https://files.pythonhosted.org/packages/3c/02/3bec2b9a1ba3c19ff89a43a19324202b8eb187ca1e928d8bdac9bbdddebd/aiohttp-3.13.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3dd4dce1c718e38081c8f35f323209d4c1df7d4db4bab1b5c88a6b4d12b74587", size = 1941001, upload-time = "2026-01-03T17:32:03.122Z" },
- { url = "https://files.pythonhosted.org/packages/37/df/d879401cedeef27ac4717f6426c8c36c3091c6e9f08a9178cc87549c537f/aiohttp-3.13.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:34bac00a67a812570d4a460447e1e9e06fae622946955f939051e7cc895cfab8", size = 1797246, upload-time = "2026-01-03T17:32:05.255Z" },
- { url = "https://files.pythonhosted.org/packages/8d/15/be122de1f67e6953add23335c8ece6d314ab67c8bebb3f181063010795a7/aiohttp-3.13.3-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a19884d2ee70b06d9204b2727a7b9f983d0c684c650254679e716b0b77920632", size = 1627131, upload-time = "2026-01-03T17:32:07.607Z" },
- { url = "https://files.pythonhosted.org/packages/12/12/70eedcac9134cfa3219ab7af31ea56bc877395b1ac30d65b1bc4b27d0438/aiohttp-3.13.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:5f8ca7f2bb6ba8348a3614c7918cc4bb73268c5ac2a207576b7afea19d3d9f64", size = 1795196, upload-time = "2026-01-03T17:32:09.59Z" },
- { url = "https://files.pythonhosted.org/packages/32/11/b30e1b1cd1f3054af86ebe60df96989c6a414dd87e27ad16950eee420bea/aiohttp-3.13.3-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:b0d95340658b9d2f11d9697f59b3814a9d3bb4b7a7c20b131df4bcef464037c0", size = 1782841, upload-time = "2026-01-03T17:32:11.445Z" },
- { url = "https://files.pythonhosted.org/packages/88/0d/d98a9367b38912384a17e287850f5695c528cff0f14f791ce8ee2e4f7796/aiohttp-3.13.3-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:a1e53262fd202e4b40b70c3aff944a8155059beedc8a89bba9dc1f9ef06a1b56", size = 1795193, upload-time = "2026-01-03T17:32:13.705Z" },
- { url = "https://files.pythonhosted.org/packages/43/a5/a2dfd1f5ff5581632c7f6a30e1744deda03808974f94f6534241ef60c751/aiohttp-3.13.3-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:d60ac9663f44168038586cab2157e122e46bdef09e9368b37f2d82d354c23f72", size = 1621979, upload-time = "2026-01-03T17:32:15.965Z" },
- { url = "https://files.pythonhosted.org/packages/fa/f0/12973c382ae7c1cccbc4417e129c5bf54c374dfb85af70893646e1f0e749/aiohttp-3.13.3-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:90751b8eed69435bac9ff4e3d2f6b3af1f57e37ecb0fbeee59c0174c9e2d41df", size = 1822193, upload-time = "2026-01-03T17:32:18.219Z" },
- { url = "https://files.pythonhosted.org/packages/3c/5f/24155e30ba7f8c96918af1350eb0663e2430aad9e001c0489d89cd708ab1/aiohttp-3.13.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:fc353029f176fd2b3ec6cfc71be166aba1936fe5d73dd1992ce289ca6647a9aa", size = 1769801, upload-time = "2026-01-03T17:32:20.25Z" },
- { url = "https://files.pythonhosted.org/packages/eb/f8/7314031ff5c10e6ece114da79b338ec17eeff3a079e53151f7e9f43c4723/aiohttp-3.13.3-cp314-cp314t-win32.whl", hash = "sha256:2e41b18a58da1e474a057b3d35248d8320029f61d70a37629535b16a0c8f3767", size = 466523, upload-time = "2026-01-03T17:32:22.215Z" },
- { url = "https://files.pythonhosted.org/packages/b4/63/278a98c715ae467624eafe375542d8ba9b4383a016df8fdefe0ae28382a7/aiohttp-3.13.3-cp314-cp314t-win_amd64.whl", hash = "sha256:44531a36aa2264a1860089ffd4dce7baf875ee5a6079d5fb42e261c704ef7344", size = 499694, upload-time = "2026-01-03T17:32:24.546Z" },
+sdist = { url = "https://files.pythonhosted.org/packages/45/4a/064321452809dae953c1ed6e017504e72551a26b6f5708a5a80e4bf556ff/aiohttp-3.13.4.tar.gz", hash = "sha256:d97a6d09c66087890c2ab5d49069e1e570583f7ac0314ecf98294c1b6aaebd38", size = 7859748, upload-time = "2026-03-28T17:19:40.6Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1e/bd/ede278648914cabbabfdf95e436679b5d4156e417896a9b9f4587169e376/aiohttp-3.13.4-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:ee62d4471ce86b108b19c3364db4b91180d13fe3510144872d6bad5401957360", size = 752158, upload-time = "2026-03-28T17:16:06.901Z" },
+ { url = "https://files.pythonhosted.org/packages/90/de/581c053253c07b480b03785196ca5335e3c606a37dc73e95f6527f1591fe/aiohttp-3.13.4-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:c0fd8f41b54b58636402eb493afd512c23580456f022c1ba2db0f810c959ed0d", size = 501037, upload-time = "2026-03-28T17:16:08.82Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/f9/a5ede193c08f13cc42c0a5b50d1e246ecee9115e4cf6e900d8dbd8fd6acb/aiohttp-3.13.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:4baa48ce49efd82d6b1a0be12d6a36b35e5594d1dd42f8bfba96ea9f8678b88c", size = 501556, upload-time = "2026-03-28T17:16:10.63Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/10/88ff67cd48a6ec36335b63a640abe86135791544863e0cfe1f065d6cef7a/aiohttp-3.13.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d738ebab9f71ee652d9dbd0211057690022201b11197f9a7324fd4dba128aa97", size = 1757314, upload-time = "2026-03-28T17:16:12.498Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/15/fdb90a5cf5a1f52845c276e76298c75fbbcc0ac2b4a86551906d54529965/aiohttp-3.13.4-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:0ce692c3468fa831af7dceed52edf51ac348cebfc8d3feb935927b63bd3e8576", size = 1731819, upload-time = "2026-03-28T17:16:14.558Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/df/28146785a007f7820416be05d4f28cc207493efd1e8c6c1068e9bdc29198/aiohttp-3.13.4-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8e08abcfe752a454d2cb89ff0c08f2d1ecd057ae3e8cc6d84638de853530ebab", size = 1793279, upload-time = "2026-03-28T17:16:16.594Z" },
+ { url = "https://files.pythonhosted.org/packages/10/47/689c743abf62ea7a77774d5722f220e2c912a77d65d368b884d9779ef41b/aiohttp-3.13.4-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5977f701b3fff36367a11087f30ea73c212e686d41cd363c50c022d48b011d8d", size = 1891082, upload-time = "2026-03-28T17:16:18.71Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/b6/f7f4f318c7e58c23b761c9b13b9a3c9b394e0f9d5d76fbc6622fa98509f6/aiohttp-3.13.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:54203e10405c06f8b6020bd1e076ae0fe6c194adcee12a5a78af3ffa3c57025e", size = 1773938, upload-time = "2026-03-28T17:16:21.125Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/06/f207cb3121852c989586a6fc16ff854c4fcc8651b86c5d3bd1fc83057650/aiohttp-3.13.4-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:358a6af0145bc4dda037f13167bef3cce54b132087acc4c295c739d05d16b1c3", size = 1579548, upload-time = "2026-03-28T17:16:23.588Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/58/e1289661a32161e24c1fe479711d783067210d266842523752869cc1d9c2/aiohttp-3.13.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:898ea1850656d7d61832ef06aa9846ab3ddb1621b74f46de78fbc5e1a586ba83", size = 1714669, upload-time = "2026-03-28T17:16:25.713Z" },
+ { url = "https://files.pythonhosted.org/packages/96/0a/3e86d039438a74a86e6a948a9119b22540bae037d6ba317a042ae3c22711/aiohttp-3.13.4-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:7bc30cceb710cf6a44e9617e43eebb6e3e43ad855a34da7b4b6a73537d8a6763", size = 1754175, upload-time = "2026-03-28T17:16:28.18Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/30/e717fc5df83133ba467a560b6d8ef20197037b4bb5d7075b90037de1018e/aiohttp-3.13.4-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:4a31c0c587a8a038f19a4c7e60654a6c899c9de9174593a13e7cc6e15ff271f9", size = 1762049, upload-time = "2026-03-28T17:16:30.941Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/28/8f7a2d4492e336e40005151bdd94baf344880a4707573378579f833a64c1/aiohttp-3.13.4-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:2062f675f3fe6e06d6113eb74a157fb9df58953ffed0cdb4182554b116545758", size = 1570861, upload-time = "2026-03-28T17:16:32.953Z" },
+ { url = "https://files.pythonhosted.org/packages/78/45/12e1a3d0645968b1c38de4b23fdf270b8637735ea057d4f84482ff918ad9/aiohttp-3.13.4-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:3d1ba8afb847ff80626d5e408c1fdc99f942acc877d0702fe137015903a220a9", size = 1790003, upload-time = "2026-03-28T17:16:35.468Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/0f/60374e18d590de16dcb39d6ff62f39c096c1b958e6f37727b5870026ea30/aiohttp-3.13.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:b08149419994cdd4d5eecf7fd4bc5986b5a9380285bcd01ab4c0d6bfca47b79d", size = 1737289, upload-time = "2026-03-28T17:16:38.187Z" },
+ { url = "https://files.pythonhosted.org/packages/02/bf/535e58d886cfbc40a8b0013c974afad24ef7632d645bca0b678b70033a60/aiohttp-3.13.4-cp312-cp312-win32.whl", hash = "sha256:fc432f6a2c4f720180959bc19aa37259651c1a4ed8af8afc84dd41c60f15f791", size = 434185, upload-time = "2026-03-28T17:16:40.735Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/1a/d92e3325134ebfff6f4069f270d3aac770d63320bd1fcd0eca023e74d9a8/aiohttp-3.13.4-cp312-cp312-win_amd64.whl", hash = "sha256:6148c9ae97a3e8bff9a1fc9c757fa164116f86c100468339730e717590a3fb77", size = 461285, upload-time = "2026-03-28T17:16:42.713Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/ac/892f4162df9b115b4758d615f32ec63d00f3084c705ff5526630887b9b42/aiohttp-3.13.4-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:63dd5e5b1e43b8fb1e91b79b7ceba1feba588b317d1edff385084fcc7a0a4538", size = 745744, upload-time = "2026-03-28T17:16:44.67Z" },
+ { url = "https://files.pythonhosted.org/packages/97/a9/c5b87e4443a2f0ea88cb3000c93a8fdad1ee63bffc9ded8d8c8e0d66efc6/aiohttp-3.13.4-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:746ac3cc00b5baea424dacddea3ec2c2702f9590de27d837aa67004db1eebc6e", size = 498178, upload-time = "2026-03-28T17:16:46.766Z" },
+ { url = "https://files.pythonhosted.org/packages/94/42/07e1b543a61250783650df13da8ddcdc0d0a5538b2bd15cef6e042aefc61/aiohttp-3.13.4-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:bda8f16ea99d6a6705e5946732e48487a448be874e54a4f73d514660ff7c05d3", size = 498331, upload-time = "2026-03-28T17:16:48.9Z" },
+ { url = "https://files.pythonhosted.org/packages/20/d6/492f46bf0328534124772d0cf58570acae5b286ea25006900650f69dae0e/aiohttp-3.13.4-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4b061e7b5f840391e3f64d0ddf672973e45c4cfff7a0feea425ea24e51530fc2", size = 1744414, upload-time = "2026-03-28T17:16:50.968Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/4d/e02627b2683f68051246215d2d62b2d2f249ff7a285e7a858dc47d6b6a14/aiohttp-3.13.4-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:b252e8d5cd66184b570d0d010de742736e8a4fab22c58299772b0c5a466d4b21", size = 1719226, upload-time = "2026-03-28T17:16:53.173Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/6c/5d0a3394dd2b9f9aeba6e1b6065d0439e4b75d41f1fb09a3ec010b43552b/aiohttp-3.13.4-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:20af8aad61d1803ff11152a26146d8d81c266aa8c5aa9b4504432abb965c36a0", size = 1782110, upload-time = "2026-03-28T17:16:55.362Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/2d/c20791e3437700a7441a7edfb59731150322424f5aadf635602d1d326101/aiohttp-3.13.4-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:13a5cc924b59859ad2adb1478e31f410a7ed46e92a2a619d6d1dd1a63c1a855e", size = 1884809, upload-time = "2026-03-28T17:16:57.734Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/94/d99dbfbd1924a87ef643833932eb2a3d9e5eee87656efea7d78058539eff/aiohttp-3.13.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:534913dfb0a644d537aebb4123e7d466d94e3be5549205e6a31f72368980a81a", size = 1764938, upload-time = "2026-03-28T17:17:00.221Z" },
+ { url = "https://files.pythonhosted.org/packages/49/61/3ce326a1538781deb89f6cf5e094e2029cd308ed1e21b2ba2278b08426f6/aiohttp-3.13.4-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:320e40192a2dcc1cf4b5576936e9652981ab596bf81eb309535db7e2f5b5672f", size = 1570697, upload-time = "2026-03-28T17:17:02.985Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/77/4ab5a546857bb3028fbaf34d6eea180267bdab022ee8b1168b1fcde4bfdd/aiohttp-3.13.4-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:9e587fcfce2bcf06526a43cb705bdee21ac089096f2e271d75de9c339db3100c", size = 1702258, upload-time = "2026-03-28T17:17:05.28Z" },
+ { url = "https://files.pythonhosted.org/packages/79/63/d8f29021e39bc5af8e5d5e9da1b07976fb9846487a784e11e4f4eeda4666/aiohttp-3.13.4-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:9eb9c2eea7278206b5c6c1441fdd9dc420c278ead3f3b2cc87f9b693698cc500", size = 1740287, upload-time = "2026-03-28T17:17:07.712Z" },
+ { url = "https://files.pythonhosted.org/packages/55/3a/cbc6b3b124859a11bc8055d3682c26999b393531ef926754a3445b99dfef/aiohttp-3.13.4-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:29be00c51972b04bf9d5c8f2d7f7314f48f96070ca40a873a53056e652e805f7", size = 1753011, upload-time = "2026-03-28T17:17:10.053Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/30/836278675205d58c1368b21520eab9572457cf19afd23759216c04483048/aiohttp-3.13.4-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:90c06228a6c3a7c9f776fe4fc0b7ff647fffd3bed93779a6913c804ae00c1073", size = 1566359, upload-time = "2026-03-28T17:17:12.433Z" },
+ { url = "https://files.pythonhosted.org/packages/50/b4/8032cc9b82d17e4277704ba30509eaccb39329dc18d6a35f05e424439e32/aiohttp-3.13.4-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:a533ec132f05fd9a1d959e7f34184cd7d5e8511584848dab85faefbaac573069", size = 1785537, upload-time = "2026-03-28T17:17:14.721Z" },
+ { url = "https://files.pythonhosted.org/packages/17/7d/5873e98230bde59f493bf1f7c3e327486a4b5653fa401144704df5d00211/aiohttp-3.13.4-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:1c946f10f413836f82ea4cfb90200d2a59578c549f00857e03111cf45ad01ca5", size = 1740752, upload-time = "2026-03-28T17:17:17.387Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/f2/13e46e0df051494d7d3c68b7f72d071f48c384c12716fc294f75d5b1a064/aiohttp-3.13.4-cp313-cp313-win32.whl", hash = "sha256:48708e2706106da6967eff5908c78ca3943f005ed6bcb75da2a7e4da94ef8c70", size = 433187, upload-time = "2026-03-28T17:17:19.523Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/c0/649856ee655a843c8f8664592cfccb73ac80ede6a8c8db33a25d810c12db/aiohttp-3.13.4-cp313-cp313-win_amd64.whl", hash = "sha256:74a2eb058da44fa3a877a49e2095b591d4913308bb424c418b77beb160c55ce3", size = 459778, upload-time = "2026-03-28T17:17:21.964Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/29/6657cc37ae04cacc2dbf53fb730a06b6091cc4cbe745028e047c53e6d840/aiohttp-3.13.4-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:e0a2c961fc92abeff61d6444f2ce6ad35bb982db9fc8ff8a47455beacf454a57", size = 749363, upload-time = "2026-03-28T17:17:24.044Z" },
+ { url = "https://files.pythonhosted.org/packages/90/7f/30ccdf67ca3d24b610067dc63d64dcb91e5d88e27667811640644aa4a85d/aiohttp-3.13.4-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:153274535985a0ff2bff1fb6c104ed547cec898a09213d21b0f791a44b14d933", size = 499317, upload-time = "2026-03-28T17:17:26.199Z" },
+ { url = "https://files.pythonhosted.org/packages/93/13/e372dd4e68ad04ee25dafb050c7f98b0d91ea643f7352757e87231102555/aiohttp-3.13.4-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:351f3171e2458da3d731ce83f9e6b9619e325c45cbd534c7759750cabf453ad7", size = 500477, upload-time = "2026-03-28T17:17:28.279Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/fe/ee6298e8e586096fb6f5eddd31393d8544f33ae0792c71ecbb4c2bef98ac/aiohttp-3.13.4-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f989ac8bc5595ff761a5ccd32bdb0768a117f36dd1504b1c2c074ed5d3f4df9c", size = 1737227, upload-time = "2026-03-28T17:17:30.587Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/b9/a7a0463a09e1a3fe35100f74324f23644bfc3383ac5fd5effe0722a5f0b7/aiohttp-3.13.4-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:d36fc1709110ec1e87a229b201dd3ddc32aa01e98e7868083a794609b081c349", size = 1694036, upload-time = "2026-03-28T17:17:33.29Z" },
+ { url = "https://files.pythonhosted.org/packages/57/7c/8972ae3fb7be00a91aee6b644b2a6a909aedb2c425269a3bfd90115e6f8f/aiohttp-3.13.4-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:42adaeea83cbdf069ab94f5103ce0787c21fb1a0153270da76b59d5578302329", size = 1786814, upload-time = "2026-03-28T17:17:36.035Z" },
+ { url = "https://files.pythonhosted.org/packages/93/01/c81e97e85c774decbaf0d577de7d848934e8166a3a14ad9f8aa5be329d28/aiohttp-3.13.4-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:92deb95469928cc41fd4b42a95d8012fa6df93f6b1c0a83af0ffbc4a5e218cde", size = 1866676, upload-time = "2026-03-28T17:17:38.441Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/5f/5b46fe8694a639ddea2cd035bf5729e4677ea882cb251396637e2ef1590d/aiohttp-3.13.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0c0c7c07c4257ef3a1df355f840bc62d133bcdef5c1c5ba75add3c08553e2eed", size = 1740842, upload-time = "2026-03-28T17:17:40.783Z" },
+ { url = "https://files.pythonhosted.org/packages/20/a2/0d4b03d011cca6b6b0acba8433193c1e484efa8d705ea58295590fe24203/aiohttp-3.13.4-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f062c45de8a1098cb137a1898819796a2491aec4e637a06b03f149315dff4d8f", size = 1566508, upload-time = "2026-03-28T17:17:43.235Z" },
+ { url = "https://files.pythonhosted.org/packages/98/17/e689fd500da52488ec5f889effd6404dece6a59de301e380f3c64f167beb/aiohttp-3.13.4-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:76093107c531517001114f0ebdb4f46858ce818590363e3e99a4a2280334454a", size = 1700569, upload-time = "2026-03-28T17:17:46.165Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/0d/66402894dbcf470ef7db99449e436105ea862c24f7ea4c95c683e635af35/aiohttp-3.13.4-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:6f6ec32162d293b82f8b63a16edc80769662fbd5ae6fbd4936d3206a2c2cc63b", size = 1707407, upload-time = "2026-03-28T17:17:48.825Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/eb/af0ab1a3650092cbd8e14ef29e4ab0209e1460e1c299996c3f8288b3f1ff/aiohttp-3.13.4-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:5903e2db3d202a00ad9f0ec35a122c005e85d90c9836ab4cda628f01edf425e2", size = 1752214, upload-time = "2026-03-28T17:17:51.206Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/bf/72326f8a98e4c666f292f03c385545963cc65e358835d2a7375037a97b57/aiohttp-3.13.4-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:2d5bea57be7aca98dbbac8da046d99b5557c5cf4e28538c4c786313078aca09e", size = 1562162, upload-time = "2026-03-28T17:17:53.634Z" },
+ { url = "https://files.pythonhosted.org/packages/67/9f/13b72435f99151dd9a5469c96b3b5f86aa29b7e785ca7f35cf5e538f74c0/aiohttp-3.13.4-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:bcf0c9902085976edc0232b75006ef38f89686901249ce14226b6877f88464fb", size = 1768904, upload-time = "2026-03-28T17:17:55.991Z" },
+ { url = "https://files.pythonhosted.org/packages/18/bc/28d4970e7d5452ac7776cdb5431a1164a0d9cf8bd2fffd67b4fb463aa56d/aiohttp-3.13.4-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:c3295f98bfeed2e867cab588f2a146a9db37a85e3ae9062abf46ba062bd29165", size = 1723378, upload-time = "2026-03-28T17:17:58.348Z" },
+ { url = "https://files.pythonhosted.org/packages/53/74/b32458ca1a7f34d65bdee7aef2036adbe0438123d3d53e2b083c453c24dd/aiohttp-3.13.4-cp314-cp314-win32.whl", hash = "sha256:a598a5c5767e1369d8f5b08695cab1d8160040f796c4416af76fd773d229b3c9", size = 438711, upload-time = "2026-03-28T17:18:00.728Z" },
+ { url = "https://files.pythonhosted.org/packages/40/b2/54b487316c2df3e03a8f3435e9636f8a81a42a69d942164830d193beb56a/aiohttp-3.13.4-cp314-cp314-win_amd64.whl", hash = "sha256:c555db4bc7a264bead5a7d63d92d41a1122fcd39cc62a4db815f45ad46f9c2c8", size = 464977, upload-time = "2026-03-28T17:18:03.367Z" },
+ { url = "https://files.pythonhosted.org/packages/47/fb/e41b63c6ce71b07a59243bb8f3b457ee0c3402a619acb9d2c0d21ef0e647/aiohttp-3.13.4-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:45abbbf09a129825d13c18c7d3182fecd46d9da3cfc383756145394013604ac1", size = 781549, upload-time = "2026-03-28T17:18:05.779Z" },
+ { url = "https://files.pythonhosted.org/packages/97/53/532b8d28df1e17e44c4d9a9368b78dcb6bf0b51037522136eced13afa9e8/aiohttp-3.13.4-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:74c80b2bc2c2adb7b3d1941b2b60701ee2af8296fc8aad8b8bc48bc25767266c", size = 514383, upload-time = "2026-03-28T17:18:08.096Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/1f/62e5d400603e8468cd635812d99cb81cfdc08127a3dc474c647615f31339/aiohttp-3.13.4-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c97989ae40a9746650fa196894f317dafc12227c808c774929dda0ff873a5954", size = 518304, upload-time = "2026-03-28T17:18:10.642Z" },
+ { url = "https://files.pythonhosted.org/packages/90/57/2326b37b10896447e3c6e0cbef4fe2486d30913639a5cfd1332b5d870f82/aiohttp-3.13.4-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dae86be9811493f9990ef44fff1685f5c1a3192e9061a71a109d527944eed551", size = 1893433, upload-time = "2026-03-28T17:18:13.121Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/b4/a24d82112c304afdb650167ef2fe190957d81cbddac7460bedd245f765aa/aiohttp-3.13.4-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:1db491abe852ca2fa6cc48a3341985b0174b3741838e1341b82ac82c8bd9e871", size = 1755901, upload-time = "2026-03-28T17:18:16.21Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/2d/0883ef9d878d7846287f036c162a951968f22aabeef3ac97b0bea6f76d5d/aiohttp-3.13.4-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:0e5d701c0aad02a7dce72eef6b93226cf3734330f1a31d69ebbf69f33b86666e", size = 1876093, upload-time = "2026-03-28T17:18:18.703Z" },
+ { url = "https://files.pythonhosted.org/packages/ad/52/9204bb59c014869b71971addad6778f005daa72a96eed652c496789d7468/aiohttp-3.13.4-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:8ac32a189081ae0a10ba18993f10f338ec94341f0d5df8fff348043962f3c6f8", size = 1970815, upload-time = "2026-03-28T17:18:21.858Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/b5/e4eb20275a866dde0f570f411b36c6b48f7b53edfe4f4071aa1b0728098a/aiohttp-3.13.4-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:98e968cdaba43e45c73c3f306fca418c8009a957733bac85937c9f9cf3f4de27", size = 1816223, upload-time = "2026-03-28T17:18:24.729Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/23/e98075c5bb146aa61a1239ee1ac7714c85e814838d6cebbe37d3fe19214a/aiohttp-3.13.4-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:ca114790c9144c335d538852612d3e43ea0f075288f4849cf4b05d6cd2238ce7", size = 1649145, upload-time = "2026-03-28T17:18:27.269Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/c1/7bad8be33bb06c2bb224b6468874346026092762cbec388c3bdb65a368ee/aiohttp-3.13.4-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:ea2e071661ba9cfe11eabbc81ac5376eaeb3061f6e72ec4cc86d7cdd1ffbdbbb", size = 1816562, upload-time = "2026-03-28T17:18:29.847Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/10/c00323348695e9a5e316825969c88463dcc24c7e9d443244b8a2c9cf2eae/aiohttp-3.13.4-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:34e89912b6c20e0fd80e07fa401fd218a410aa1ce9f1c2f1dad6db1bd0ce0927", size = 1800333, upload-time = "2026-03-28T17:18:32.269Z" },
+ { url = "https://files.pythonhosted.org/packages/84/43/9b2147a1df3559f49bd723e22905b46a46c068a53adb54abdca32c4de180/aiohttp-3.13.4-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:0e217cf9f6a42908c52b46e42c568bd57adc39c9286ced31aaace614b6087965", size = 1820617, upload-time = "2026-03-28T17:18:35.238Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/7f/b3481a81e7a586d02e99387b18c6dafff41285f6efd3daa2124c01f87eae/aiohttp-3.13.4-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:0c296f1221e21ba979f5ac1964c3b78cfde15c5c5f855ffd2caab337e9cd9182", size = 1643417, upload-time = "2026-03-28T17:18:37.949Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/72/07181226bc99ce1124e0f89280f5221a82d3ae6a6d9d1973ce429d48e52b/aiohttp-3.13.4-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:d99a9d168ebaffb74f36d011750e490085ac418f4db926cce3989c8fe6cb6b1b", size = 1849286, upload-time = "2026-03-28T17:18:40.534Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/e6/1b3566e103eca6da5be4ae6713e112a053725c584e96574caf117568ffef/aiohttp-3.13.4-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:cb19177205d93b881f3f89e6081593676043a6828f59c78c17a0fd6c1fbed2ba", size = 1782635, upload-time = "2026-03-28T17:18:43.073Z" },
+ { url = "https://files.pythonhosted.org/packages/37/58/1b11c71904b8d079eb0c39fe664180dd1e14bebe5608e235d8bfbadc8929/aiohttp-3.13.4-cp314-cp314t-win32.whl", hash = "sha256:c606aa5656dab6552e52ca368e43869c916338346bfaf6304e15c58fb113ea30", size = 472537, upload-time = "2026-03-28T17:18:46.286Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/8f/87c56a1a1977d7dddea5b31e12189665a140fdb48a71e9038ff90bb564ec/aiohttp-3.13.4-cp314-cp314t-win_amd64.whl", hash = "sha256:014dcc10ec8ab8db681f0d68e939d1e9286a5aa2b993cbbdb0db130853e02144", size = 506381, upload-time = "2026-03-28T17:18:48.74Z" },
+]
+
+[[package]]
+name = "aiohttp-cors"
+version = "0.8.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiohttp" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/6f/6d/d89e846a5444b3d5eb8985a6ddb0daef3774928e1bfbce8e84ec97b0ffa7/aiohttp_cors-0.8.1.tar.gz", hash = "sha256:ccacf9cb84b64939ea15f859a146af1f662a6b1d68175754a07315e305fb1403", size = 38626, upload-time = "2025-03-31T14:16:20.048Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/98/3b/40a68de458904bcc143622015fff2352b6461cd92fd66d3527bf1c6f5716/aiohttp_cors-0.8.1-py3-none-any.whl", hash = "sha256:3180cf304c5c712d626b9162b195b1db7ddf976a2a25172b35bb2448b890a80d", size = 25231, upload-time = "2025-03-31T14:16:18.478Z" },
]
[[package]]
@@ -262,7 +303,6 @@ name = "anyio"
version = "4.9.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
- { name = "exceptiongroup", marker = "python_full_version < '3.11'" },
{ name = "idna" },
{ name = "sniffio" },
{ name = "typing-extensions", marker = "python_full_version < '3.13'" },
@@ -274,142 +314,76 @@ wheels = [
[[package]]
name = "apache-tvm-ffi"
-version = "0.1.8.post2"
+version = "0.1.9"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/e3/e9/a13952726228fa6282154ecf927092396bc759739e5e045019f6ab92f3ca/apache_tvm_ffi-0.1.8.post2.tar.gz", hash = "sha256:4513e38852894f290172ecfefcbc18d34e817fd29c16a0f1770e130c82b4067e", size = 2441111, upload-time = "2026-01-13T18:11:27.864Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/cd/65/0c67653e6431716f2706e29f2e2e1ce9a6f9d9f7615c0c637a4881c3f5a5/apache_tvm_ffi-0.1.8.post2-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:e11e03c865297c65c2f206c90b8014890bc52a3059d8148b47cd2c2759bcea90", size = 1838436, upload-time = "2026-01-13T18:10:22.334Z" },
- { url = "https://files.pythonhosted.org/packages/46/8f/13fe7acbd7497312fda5faf51545fcb50c0ed5398cfe525d006ba29f1b9b/apache_tvm_ffi-0.1.8.post2-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e855f2b3f60ec16939b00e1b594ce7f488f96e387b12547e98643177f70ab2b1", size = 1996102, upload-time = "2026-01-13T18:10:23.97Z" },
- { url = "https://files.pythonhosted.org/packages/cc/f8/b469a4d91ea74f627cb220835049fb60a566f7427f27c9f66c6c54a287b6/apache_tvm_ffi-0.1.8.post2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:854ecd90a1039d542c531fa6a4928f5633452aedf1ed7f646f3bbbeca8217156", size = 2069067, upload-time = "2026-01-13T18:10:25.425Z" },
- { url = "https://files.pythonhosted.org/packages/d0/88/663e532e7ba625a3998724ae0207ce620c32a057c339b4e4ae0be6810d85/apache_tvm_ffi-0.1.8.post2-cp310-cp310-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e1894b6f9c2b45bc9df8e407d041e575128591b998ced09f974675d2bb6b8bc9", size = 1939413, upload-time = "2026-01-13T18:10:28.61Z" },
- { url = "https://files.pythonhosted.org/packages/ee/16/6ec659fd5b3b163de9adc75bf29fc90460d212b489947b77b8ed89c01472/apache_tvm_ffi-0.1.8.post2-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ef922ef3ed971a4e161a0385ef9f67af379d52b0d83d62c08b79f6707b6660b5", size = 2053058, upload-time = "2026-01-13T18:10:30.721Z" },
- { url = "https://files.pythonhosted.org/packages/ec/a8/d01f81987db9bbfc4b242575d3fe79f72aeba3582ca449fec28d19938400/apache_tvm_ffi-0.1.8.post2-cp310-cp310-win_amd64.whl", hash = "sha256:146f98dcd21052eeed96ad07472bdffd8189fb2106edc6e3de91e28e3b000bf8", size = 1809231, upload-time = "2026-01-13T18:10:32.293Z" },
- { url = "https://files.pythonhosted.org/packages/aa/86/7db24692281d80204d07d77346ad4cb87f6183f1364ed94311993a47ed1a/apache_tvm_ffi-0.1.8.post2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:40f5fba3e06617f16888a0fdaf7ab4049841ff6e741644be822400438b771fe7", size = 1840013, upload-time = "2026-01-13T18:10:33.724Z" },
- { url = "https://files.pythonhosted.org/packages/cf/cc/fbaef883c6ba8e2c56ffcca997f2c076d1c14787799a62f39bd52c7126d5/apache_tvm_ffi-0.1.8.post2-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:9eb6d228fa22b6a5da140d761962f022a154746c91fe7608c49062deaf671f9f", size = 1995159, upload-time = "2026-01-13T18:10:35.727Z" },
- { url = "https://files.pythonhosted.org/packages/49/08/f1e984e3573d0cbd6d53f3f73a12691fba153afc529fbd506d78e739b330/apache_tvm_ffi-0.1.8.post2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:581c0acf845859be0cc26ac79f3663a83393b662c97c7125ebb78f0228b69d96", size = 2068543, upload-time = "2026-01-13T18:10:39.12Z" },
- { url = "https://files.pythonhosted.org/packages/35/1f/5336d430a133cf66ca9dac8ae9b6e25d8b99275a6687656421a1deee9f1b/apache_tvm_ffi-0.1.8.post2-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:beadc7bb480ae02d02e2108543f6f4b4170d77e361ab3ccb43697d174ec185b0", size = 1939018, upload-time = "2026-01-13T18:10:40.621Z" },
- { url = "https://files.pythonhosted.org/packages/5f/67/969c66a27a128cf738d0c068e0d4451d691d8197929c797cbe8e59c6cfc9/apache_tvm_ffi-0.1.8.post2-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e593d191c7ca0726ebcd3b024a4bc8140694fdfce2e7b02493f38ad5c4c9ecf7", size = 2053068, upload-time = "2026-01-13T18:10:43.241Z" },
- { url = "https://files.pythonhosted.org/packages/d4/f1/84881a799d227fdc4a61fbf0cb8d5ceb6a72ad788fa9070e5853ed9759b6/apache_tvm_ffi-0.1.8.post2-cp311-cp311-win_amd64.whl", hash = "sha256:1c685f19d0f26d9356c7c77a1cb652a3632ec9ee6cd21aa1d8cfb968743ec1fd", size = 1809557, upload-time = "2026-01-13T18:10:44.743Z" },
- { url = "https://files.pythonhosted.org/packages/12/8b/a39d6c6eb1a87f6003e2717695cc6d44cc65ccd57dae5a0af944c0d25751/apache_tvm_ffi-0.1.8.post2-cp312-abi3-macosx_11_0_arm64.whl", hash = "sha256:c13ec7fc8f255767998b301ace0cd1e7d17ba76b48ffeb97ca9eb22a3314e250", size = 1811882, upload-time = "2026-01-13T18:10:46.317Z" },
- { url = "https://files.pythonhosted.org/packages/8e/3a/7b1c9edcaeaebb945038144896cf17eb828a40b6ace0371823e133132664/apache_tvm_ffi-0.1.8.post2-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8c78b4caf17304a1f47881bccdb2f9ac24d98b3b7fbe761a6dd4fd0585934d96", size = 1967259, upload-time = "2026-01-13T18:10:47.851Z" },
- { url = "https://files.pythonhosted.org/packages/6c/b6/463602f57dda2e1c69165c044c07061cd59404593f313a427a3ad9c02cf3/apache_tvm_ffi-0.1.8.post2-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4a48da3fa8f47130f3502134f01e97044388c5217e7b91be4b0acec4feab81a0", size = 2044821, upload-time = "2026-01-13T18:10:49.396Z" },
- { url = "https://files.pythonhosted.org/packages/fe/e6/9cdc7f4814b2fbdfceba5dc640c3704d07d8db18e3d1aef5aa49bbf1ba7e/apache_tvm_ffi-0.1.8.post2-cp312-abi3-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:61cc98e489ebc03bc96d1a966dc863eb1c0a607383f6bf4a416ff0a96170ca85", size = 1910964, upload-time = "2026-01-13T18:10:51.345Z" },
- { url = "https://files.pythonhosted.org/packages/7d/f5/a2e5487cdad575fe6cf34f8a23f8c49e08ce5808fa75dc19d98bcebc20ec/apache_tvm_ffi-0.1.8.post2-cp312-abi3-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:caa48509f0c7d9b896823b492a9ee42afac2548065c1ec7ef07f9a0dc30d2796", size = 2025814, upload-time = "2026-01-13T18:10:52.804Z" },
- { url = "https://files.pythonhosted.org/packages/8f/0d/8922c142281187ae6b989579876d00d20b84ccd3878aad487b91d951d254/apache_tvm_ffi-0.1.8.post2-cp312-abi3-win_amd64.whl", hash = "sha256:985831722d1dd562d13e8e34102fd99f42f964c53fc7cf9d80fc4f7602f89196", size = 1790204, upload-time = "2026-01-13T18:10:54.558Z" },
- { url = "https://files.pythonhosted.org/packages/2c/6e/2c21e754adf5c08fff154ee0a75b01568a4ed5da2d8f4a4a95d8451736e0/apache_tvm_ffi-0.1.8.post2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:4a3f6cb1173cfe19a1b66fd8577a6f3ce644bdc22691961c07c64304a7c3f17a", size = 1842240, upload-time = "2026-01-13T18:10:56.652Z" },
- { url = "https://files.pythonhosted.org/packages/f6/0a/342dd451d714b683143bd0d7dbd26279772dedf1d827a7efd357f05ff0aa/apache_tvm_ffi-0.1.8.post2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ac6c2d4e117ca63974bcd20fdf5715d01f3b4d0ed78921f493461050daf7c1a3", size = 1980660, upload-time = "2026-01-13T18:10:58.892Z" },
- { url = "https://files.pythonhosted.org/packages/c6/63/59f00116530cf7513866467de9044dbdd1954a536009e56c44f167743b35/apache_tvm_ffi-0.1.8.post2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0bc5456f971097dcd973daba32cb6f321893873c53235159ab6426b0c7bef7e2", size = 2052810, upload-time = "2026-01-13T18:11:01.698Z" },
- { url = "https://files.pythonhosted.org/packages/46/dc/e22c784937fdc907785a764d773ef57a925c443d8ec01ad8bff43dd8d8d6/apache_tvm_ffi-0.1.8.post2-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7f2016b4b31e7f75d71c638bbd1ae43d6e239cf8e20b539fb9de6917b3fb25bc", size = 1923716, upload-time = "2026-01-13T18:11:03.225Z" },
- { url = "https://files.pythonhosted.org/packages/ab/39/695f5642979d1d2d4cd3fca92e7b3b324ebba734b8aab9bdbacc26d4a05c/apache_tvm_ffi-0.1.8.post2-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4c0ca7be630d0888eae163a4298ddfb3f7bd837112c7e6ffcd7157e34e78215b", size = 2035440, upload-time = "2026-01-13T18:11:04.841Z" },
- { url = "https://files.pythonhosted.org/packages/ed/e0/ed152425e51b7c8a4ce81d33683b43d87e770a76a65922dc7524a0106ae8/apache_tvm_ffi-0.1.8.post2-cp314-cp314t-win_amd64.whl", hash = "sha256:ecb0d9f7f410ba3b4d92547c2477f73f8406455448f4ea8c146515671fd20210", size = 1849938, upload-time = "2026-01-13T18:11:06.312Z" },
+sdist = { url = "https://files.pythonhosted.org/packages/6f/60/1e787a0b5ebf318483235be2a689ee367173983067e441b8379564f667c0/apache_tvm_ffi-0.1.9.tar.gz", hash = "sha256:d2d402587e8906de0a07f4746aa78f3d452c7efe3625d4bb39ac2ad693bce530", size = 2513731, upload-time = "2026-02-27T19:28:06.602Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/df/f2/b8c4b151169f6d7ba8773c8af68b2e0c1013d7fb3f1bdf87573f47157ce9/apache_tvm_ffi-0.1.9-cp312-abi3-macosx_11_0_arm64.whl", hash = "sha256:49e52350b0470654847de752e65603b604a4d3323e7e9f5e8a982f44acc4c143", size = 2041756, upload-time = "2026-02-27T19:27:23.931Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/c0/6d3d54f50012255b41bc3e24944c086f63c4707c8686c7c6780e9283eb96/apache_tvm_ffi-0.1.9-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7d503029e66c43b1a1cb1a42a1e9bb428c8a28dcbdec31c28e705472ca648a3a", size = 2203712, upload-time = "2026-02-27T19:27:25.867Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/dd/2bab4c6cd86257dbf99e93452a1af833113f8dc3e25a25579f6e4e4c8a94/apache_tvm_ffi-0.1.9-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:28241371934ea8af10d5067087ba1229ebddded7b2c02d33a258ec2a96df8c46", size = 2299704, upload-time = "2026-02-27T19:27:27.477Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/4a/b469bcb2e1014cb84d336d2a59f42958a058251c577a4c2680cacad346e2/apache_tvm_ffi-0.1.9-cp312-abi3-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:87cacce81df55685fc6a76e1e3c5db1200e85e87bf5974b692c59d131b7bc622", size = 2130865, upload-time = "2026-02-27T19:27:29.092Z" },
+ { url = "https://files.pythonhosted.org/packages/70/ef/5402da5d37f5270fd88ea0348acca78dba9be8bdbf6c2bcae0935eb03ef1/apache_tvm_ffi-0.1.9-cp312-abi3-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f45eb43499acac45ff6c93564f0ff2d3ca27b69656d540fd56ce59d51c0b4c65", size = 2278991, upload-time = "2026-02-27T19:27:30.729Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/23/1b7dc5f0807f83098183a57db6ee85b2c93b646d74a6e03781c9208aaeb0/apache_tvm_ffi-0.1.9-cp312-abi3-win_amd64.whl", hash = "sha256:d1dcf4c041d5ec05e3da1d545800c33cdbb95c113baa7705085ff79fa262752b", size = 1973200, upload-time = "2026-02-27T19:27:32.367Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/1e/991ae65e64ce132c1ba665562db6638f5696d6133f580e20c653de33b9af/apache_tvm_ffi-0.1.9-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c3349f72ddb8ce206472d0380a729f213017a2180707096f8d57114b81097dd1", size = 2072944, upload-time = "2026-02-27T19:27:34.261Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/a7/1e0643949e683fb3cfababd87058c0cfef122d1a3bb6ce703f719051b842/apache_tvm_ffi-0.1.9-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d1f4d2b7ec7b1213632e9a104e9330bfc3dec48decffa62114c33aa188c9f43a", size = 2215954, upload-time = "2026-02-27T19:27:35.872Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/06/5016191ab61d2db4c3a7d754a3c1184e0836f575a7d08491669738c5e4b9/apache_tvm_ffi-0.1.9-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e4f01d16ba53fe118e363f7257253f07003797e4abe6fc9567f23b6a930dbff2", size = 2307291, upload-time = "2026-02-27T19:27:37.527Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/f5/40bf0667330938efbfc0a51743cc53c79e41b4ece1a8abad3076192c9674/apache_tvm_ffi-0.1.9-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3c0581dd6bfbce7b017ef85cfda08bbe38891cc4b3afbcfaa8bc2d383728e426", size = 2143850, upload-time = "2026-02-27T19:27:40.437Z" },
+ { url = "https://files.pythonhosted.org/packages/72/4a/421cbd4ed32e8bad3b88af3e8fa145c1f6f493bdd05be15b6f2d9b3cb7d6/apache_tvm_ffi-0.1.9-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7dfa14be2a49347791ef21222a8225ce7f99bfec17104a676cb4f1bf3a107088", size = 2289038, upload-time = "2026-02-27T19:27:41.972Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/1a/c8923d819b49872a612033b90d29299c0be73a7cbed1ddb3dc78dfe5e9f1/apache_tvm_ffi-0.1.9-cp314-cp314t-win_amd64.whl", hash = "sha256:a42d7ca27dce83efbdce7ec970fe3e773a69c31d928730ee5d9badb1229d106c", size = 2039007, upload-time = "2026-02-27T19:27:43.618Z" },
]
[[package]]
name = "astroid"
version = "3.2.4"
source = { registry = "https://pypi.org/simple" }
-dependencies = [
- { name = "typing-extensions", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
-]
sdist = { url = "https://files.pythonhosted.org/packages/9e/53/1067e1113ecaf58312357f2cd93063674924119d80d173adc3f6f2387aa2/astroid-3.2.4.tar.gz", hash = "sha256:0e14202810b30da1b735827f78f5157be2bbd4a7a59b7707ca0bfc2fb4c0063a", size = 397576, upload-time = "2024-07-20T12:57:43.26Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/80/96/b32bbbb46170a1c8b8b1f28c794202e25cfe743565e9d3469b8eb1e0cc05/astroid-3.2.4-py3-none-any.whl", hash = "sha256:413658a61eeca6202a59231abb473f932038fbcbf1666587f66d482083413a25", size = 276348, upload-time = "2024-07-20T12:57:40.886Z" },
]
-[[package]]
-name = "async-timeout"
-version = "5.0.1"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/a5/ae/136395dfbfe00dfc94da3f3e136d0b13f394cba8f4841120e34226265780/async_timeout-5.0.1.tar.gz", hash = "sha256:d9321a7a3d5a6a5e187e824d2fa0793ce379a202935782d555d6e9d2735677d3", size = 9274, upload-time = "2024-11-06T16:41:39.6Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/fe/ba/e2081de779ca30d473f21f5b30e0e737c438205440784c7dfc81efc2b029/async_timeout-5.0.1-py3-none-any.whl", hash = "sha256:39e3809566ff85354557ec2398b55e096c8364bacac9405a7a1fa429e77fe76c", size = 6233, upload-time = "2024-11-06T16:41:37.9Z" },
-]
-
[[package]]
name = "attrs"
-version = "25.4.0"
+version = "26.1.0"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/6b/5c/685e6633917e101e5dcb62b9dd76946cbb57c26e133bae9e0cd36033c0a9/attrs-25.4.0.tar.gz", hash = "sha256:16d5969b87f0859ef33a48b35d55ac1be6e42ae49d5e853b597db70c35c57e11", size = 934251, upload-time = "2025-10-06T13:54:44.725Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/9a/8e/82a0fe20a541c03148528be8cac2408564a6c9a0cc7e9171802bc1d26985/attrs-26.1.0.tar.gz", hash = "sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32", size = 952055, upload-time = "2026-03-19T14:22:25.026Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/3a/2a/7cc015f5b9f5db42b7d48157e23356022889fc354a2813c15934b7cb5c0e/attrs-25.4.0-py3-none-any.whl", hash = "sha256:adcf7e2a1fb3b36ac48d97835bb6d8ade15b8dcce26aba8bf1d14847b57a3373", size = 67615, upload-time = "2025-10-06T13:54:43.17Z" },
+ { url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548, upload-time = "2026-03-19T14:22:23.645Z" },
]
[[package]]
name = "av"
-version = "16.1.0"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/78/cd/3a83ffbc3cc25b39721d174487fb0d51a76582f4a1703f98e46170ce83d4/av-16.1.0.tar.gz", hash = "sha256:a094b4fd87a3721dacf02794d3d2c82b8d712c85b9534437e82a8a978c175ffd", size = 4285203, upload-time = "2026-01-11T07:31:33.772Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/97/51/2217a9249409d2e88e16e3f16f7c0def9fd3e7ffc4238b2ec211f9935bdb/av-16.1.0-cp310-cp310-macosx_11_0_x86_64.whl", hash = "sha256:2395748b0c34fe3a150a1721e4f3d4487b939520991b13e7b36f8926b3b12295", size = 26942590, upload-time = "2026-01-09T20:17:58.588Z" },
- { url = "https://files.pythonhosted.org/packages/bf/cd/a7070f4febc76a327c38808e01e2ff6b94531fe0b321af54ea3915165338/av-16.1.0-cp310-cp310-macosx_14_0_arm64.whl", hash = "sha256:72d7ac832710a158eeb7a93242370aa024a7646516291c562ee7f14a7ea881fd", size = 21507910, upload-time = "2026-01-09T20:18:02.309Z" },
- { url = "https://files.pythonhosted.org/packages/ae/30/ec812418cd9b297f0238fe20eb0747d8a8b68d82c5f73c56fe519a274143/av-16.1.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:6cbac833092e66b6b0ac4d81ab077970b8ca874951e9c3974d41d922aaa653ed", size = 38738309, upload-time = "2026-01-09T20:18:04.701Z" },
- { url = "https://files.pythonhosted.org/packages/3a/b8/6c5795bf1f05f45c5261f8bce6154e0e5e86b158a6676650ddd77c28805e/av-16.1.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:eb990672d97c18f99c02f31c8d5750236f770ffe354b5a52c5f4d16c5e65f619", size = 40293006, upload-time = "2026-01-09T20:18:07.238Z" },
- { url = "https://files.pythonhosted.org/packages/a7/44/5e183bcb9333fc3372ee6e683be8b0c9b515a506894b2d32ff465430c074/av-16.1.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:05ad70933ac3b8ef896a820ea64b33b6cca91a5fac5259cb9ba7fa010435be15", size = 40123516, upload-time = "2026-01-09T20:18:09.955Z" },
- { url = "https://files.pythonhosted.org/packages/12/1d/b5346d582a3c3d958b4d26a2cc63ce607233582d956121eb20d2bbe55c2e/av-16.1.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:d831a1062a3c47520bf99de6ec682bd1d64a40dfa958e5457bb613c5270e7ce3", size = 41463289, upload-time = "2026-01-09T20:18:12.459Z" },
- { url = "https://files.pythonhosted.org/packages/fa/31/acc946c0545f72b8d0d74584cb2a0ade9b7dfe2190af3ef9aa52a2e3c0b1/av-16.1.0-cp310-cp310-win_amd64.whl", hash = "sha256:358ab910fef3c5a806c55176f2b27e5663b33c4d0a692dafeb049c6ed71f8aff", size = 31754959, upload-time = "2026-01-09T20:18:14.718Z" },
- { url = "https://files.pythonhosted.org/packages/48/d0/b71b65d1b36520dcb8291a2307d98b7fc12329a45614a303ff92ada4d723/av-16.1.0-cp311-cp311-macosx_11_0_x86_64.whl", hash = "sha256:e88ad64ee9d2b9c4c5d891f16c22ae78e725188b8926eb88187538d9dd0b232f", size = 26927747, upload-time = "2026-01-09T20:18:16.976Z" },
- { url = "https://files.pythonhosted.org/packages/2f/79/720a5a6ccdee06eafa211b945b0a450e3a0b8fc3d12922f0f3c454d870d2/av-16.1.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:cb296073fa6935724de72593800ba86ae49ed48af03960a4aee34f8a611f442b", size = 21492232, upload-time = "2026-01-09T20:18:19.266Z" },
- { url = "https://files.pythonhosted.org/packages/8e/4f/a1ba8d922f2f6d1a3d52419463ef26dd6c4d43ee364164a71b424b5ae204/av-16.1.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:720edd4d25aa73723c1532bb0597806d7b9af5ee34fc02358782c358cfe2f879", size = 39291737, upload-time = "2026-01-09T20:18:21.513Z" },
- { url = "https://files.pythonhosted.org/packages/1a/31/fc62b9fe8738d2693e18d99f040b219e26e8df894c10d065f27c6b4f07e3/av-16.1.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:c7f2bc703d0df260a1fdf4de4253c7f5500ca9fc57772ea241b0cb241bcf972e", size = 40846822, upload-time = "2026-01-09T20:18:24.275Z" },
- { url = "https://files.pythonhosted.org/packages/53/10/ab446583dbce730000e8e6beec6ec3c2753e628c7f78f334a35cad0317f4/av-16.1.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d69c393809babada7d54964d56099e4b30a3e1f8b5736ca5e27bd7be0e0f3c83", size = 40675604, upload-time = "2026-01-09T20:18:26.866Z" },
- { url = "https://files.pythonhosted.org/packages/31/d7/1003be685277005f6d63fd9e64904ee222fe1f7a0ea70af313468bb597db/av-16.1.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:441892be28582356d53f282873c5a951592daaf71642c7f20165e3ddcb0b4c63", size = 42015955, upload-time = "2026-01-09T20:18:29.461Z" },
- { url = "https://files.pythonhosted.org/packages/2f/4a/fa2a38ee9306bf4579f556f94ecbc757520652eb91294d2a99c7cf7623b9/av-16.1.0-cp311-cp311-win_amd64.whl", hash = "sha256:273a3e32de64819e4a1cd96341824299fe06f70c46f2288b5dc4173944f0fd62", size = 31750339, upload-time = "2026-01-09T20:18:32.249Z" },
- { url = "https://files.pythonhosted.org/packages/9c/84/2535f55edcd426cebec02eb37b811b1b0c163f26b8d3f53b059e2ec32665/av-16.1.0-cp312-cp312-macosx_11_0_x86_64.whl", hash = "sha256:640f57b93f927fba8689f6966c956737ee95388a91bd0b8c8b5e0481f73513d6", size = 26945785, upload-time = "2026-01-09T20:18:34.486Z" },
- { url = "https://files.pythonhosted.org/packages/b6/17/ffb940c9e490bf42e86db4db1ff426ee1559cd355a69609ec1efe4d3a9eb/av-16.1.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:ae3fb658eec00852ebd7412fdc141f17f3ddce8afee2d2e1cf366263ad2a3b35", size = 21481147, upload-time = "2026-01-09T20:18:36.716Z" },
- { url = "https://files.pythonhosted.org/packages/15/c1/e0d58003d2d83c3921887d5c8c9b8f5f7de9b58dc2194356a2656a45cfdc/av-16.1.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:27ee558d9c02a142eebcbe55578a6d817fedfde42ff5676275504e16d07a7f86", size = 39517197, upload-time = "2026-01-11T09:57:31.937Z" },
- { url = "https://files.pythonhosted.org/packages/32/77/787797b43475d1b90626af76f80bfb0c12cfec5e11eafcfc4151b8c80218/av-16.1.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:7ae547f6d5fa31763f73900d43901e8c5fa6367bb9a9840978d57b5a7ae14ed2", size = 41174337, upload-time = "2026-01-11T09:57:35.792Z" },
- { url = "https://files.pythonhosted.org/packages/8e/ac/d90df7f1e3b97fc5554cf45076df5045f1e0a6adf13899e10121229b826c/av-16.1.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8cf065f9d438e1921dc31fc7aa045790b58aee71736897866420d80b5450f62a", size = 40817720, upload-time = "2026-01-11T09:57:39.039Z" },
- { url = "https://files.pythonhosted.org/packages/80/6f/13c3a35f9dbcebafd03fe0c4cbd075d71ac8968ec849a3cfce406c35a9d2/av-16.1.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:a345877a9d3cc0f08e2bc4ec163ee83176864b92587afb9d08dff50f37a9a829", size = 42267396, upload-time = "2026-01-11T09:57:42.115Z" },
- { url = "https://files.pythonhosted.org/packages/c8/b9/275df9607f7fb44317ccb1d4be74827185c0d410f52b6e2cd770fe209118/av-16.1.0-cp312-cp312-win_amd64.whl", hash = "sha256:f49243b1d27c91cd8c66fdba90a674e344eb8eb917264f36117bf2b6879118fd", size = 31752045, upload-time = "2026-01-11T09:57:45.106Z" },
- { url = "https://files.pythonhosted.org/packages/75/2a/63797a4dde34283dd8054219fcb29294ba1c25d68ba8c8c8a6ae53c62c45/av-16.1.0-cp313-cp313-macosx_11_0_x86_64.whl", hash = "sha256:ce2a1b3d8bf619f6c47a9f28cfa7518ff75ddd516c234a4ee351037b05e6a587", size = 26916715, upload-time = "2026-01-11T09:57:47.682Z" },
- { url = "https://files.pythonhosted.org/packages/d2/c4/0b49cf730d0ae8cda925402f18ae814aef351f5772d14da72dd87ff66448/av-16.1.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:408dbe6a2573ca58a855eb8cd854112b33ea598651902c36709f5f84c991ed8e", size = 21452167, upload-time = "2026-01-11T09:57:50.606Z" },
- { url = "https://files.pythonhosted.org/packages/51/23/408806503e8d5d840975aad5699b153aaa21eb6de41ade75248a79b7a37f/av-16.1.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:57f657f86652a160a8a01887aaab82282f9e629abf94c780bbdbb01595d6f0f7", size = 39215659, upload-time = "2026-01-11T09:57:53.757Z" },
- { url = "https://files.pythonhosted.org/packages/c4/19/a8528d5bba592b3903f44c28dab9cc653c95fcf7393f382d2751a1d1523e/av-16.1.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:adbad2b355c2ee4552cac59762809d791bda90586d134a33c6f13727fb86cb3a", size = 40874970, upload-time = "2026-01-11T09:57:56.802Z" },
- { url = "https://files.pythonhosted.org/packages/e8/24/2dbcdf0e929ad56b7df078e514e7bd4ca0d45cba798aff3c8caac097d2f7/av-16.1.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f42e1a68ec2aebd21f7eb6895be69efa6aa27eec1670536876399725bbda4b99", size = 40530345, upload-time = "2026-01-11T09:58:00.421Z" },
- { url = "https://files.pythonhosted.org/packages/54/27/ae91b41207f34e99602d1c72ab6ffd9c51d7c67e3fbcd4e3a6c0e54f882c/av-16.1.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:58fe47aeaef0f100c40ec8a5de9abbd37f118d3ca03829a1009cf288e9aef67c", size = 41972163, upload-time = "2026-01-11T09:58:03.756Z" },
- { url = "https://files.pythonhosted.org/packages/fc/7a/22158fb923b2a9a00dfab0e96ef2e8a1763a94dd89e666a5858412383d46/av-16.1.0-cp313-cp313-win_amd64.whl", hash = "sha256:565093ebc93b2f4b76782589564869dadfa83af5b852edebedd8fee746457d06", size = 31729230, upload-time = "2026-01-11T09:58:07.254Z" },
- { url = "https://files.pythonhosted.org/packages/7f/f1/878f8687d801d6c4565d57ebec08449c46f75126ebca8e0fed6986599627/av-16.1.0-cp313-cp313t-macosx_11_0_x86_64.whl", hash = "sha256:574081a24edb98343fd9f473e21ae155bf61443d4ec9d7708987fa597d6b04b2", size = 27008769, upload-time = "2026-01-11T09:58:10.266Z" },
- { url = "https://files.pythonhosted.org/packages/30/f1/bd4ce8c8b5cbf1d43e27048e436cbc9de628d48ede088a1d0a993768eb86/av-16.1.0-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:9ab00ea29c25ebf2ea1d1e928d7babb3532d562481c5d96c0829212b70756ad0", size = 21590588, upload-time = "2026-01-11T09:58:12.629Z" },
- { url = "https://files.pythonhosted.org/packages/1d/dd/c81f6f9209201ff0b5d5bed6da6c6e641eef52d8fbc930d738c3f4f6f75d/av-16.1.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:a84a91188c1071f238a9523fd42dbe567fb2e2607b22b779851b2ce0eac1b560", size = 40638029, upload-time = "2026-01-11T09:58:15.399Z" },
- { url = "https://files.pythonhosted.org/packages/15/4d/07edff82b78d0459a6e807e01cd280d3180ce832efc1543de80d77676722/av-16.1.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:c2cd0de4dd022a7225ff224fde8e7971496d700be41c50adaaa26c07bb50bf97", size = 41970776, upload-time = "2026-01-11T09:58:19.075Z" },
- { url = "https://files.pythonhosted.org/packages/da/9d/1f48b354b82fa135d388477cd1b11b81bdd4384bd6a42a60808e2ec2d66b/av-16.1.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:0816143530624a5a93bc5494f8c6eeaf77549b9366709c2ac8566c1e9bff6df5", size = 41764751, upload-time = "2026-01-11T09:58:22.788Z" },
- { url = "https://files.pythonhosted.org/packages/2f/c7/a509801e98db35ec552dd79da7bdbcff7104044bfeb4c7d196c1ce121593/av-16.1.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:e3a28053af29644696d0c007e897d19b1197585834660a54773e12a40b16974c", size = 43034355, upload-time = "2026-01-11T09:58:26.125Z" },
- { url = "https://files.pythonhosted.org/packages/36/8b/e5f530d9e8f640da5f5c5f681a424c65f9dd171c871cd255d8a861785a6e/av-16.1.0-cp313-cp313t-win_amd64.whl", hash = "sha256:2e3e67144a202b95ed299d165232533989390a9ea3119d37eccec697dc6dbb0c", size = 31947047, upload-time = "2026-01-11T09:58:31.867Z" },
- { url = "https://files.pythonhosted.org/packages/df/18/8812221108c27d19f7e5f486a82c827923061edf55f906824ee0fcaadf50/av-16.1.0-cp314-cp314-macosx_11_0_x86_64.whl", hash = "sha256:39a634d8e5a87e78ea80772774bfd20c0721f0d633837ff185f36c9d14ffede4", size = 26916179, upload-time = "2026-01-11T09:58:36.506Z" },
- { url = "https://files.pythonhosted.org/packages/38/ef/49d128a9ddce42a2766fe2b6595bd9c49e067ad8937a560f7838a541464e/av-16.1.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:0ba32fb9e9300948a7fa9f8a3fc686e6f7f77599a665c71eb2118fdfd2c743f9", size = 21460168, upload-time = "2026-01-11T09:58:39.231Z" },
- { url = "https://files.pythonhosted.org/packages/e6/a9/b310d390844656fa74eeb8c2750e98030877c75b97551a23a77d3f982741/av-16.1.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:ca04d17815182d34ce3edc53cbda78a4f36e956c0fd73e3bab249872a831c4d7", size = 39210194, upload-time = "2026-01-11T09:58:42.138Z" },
- { url = "https://files.pythonhosted.org/packages/0c/7b/e65aae179929d0f173af6e474ad1489b5b5ad4c968a62c42758d619e54cf/av-16.1.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:ee0e8de2e124a9ef53c955fe2add6ee7c56cc8fd83318265549e44057db77142", size = 40811675, upload-time = "2026-01-11T09:58:45.871Z" },
- { url = "https://files.pythonhosted.org/packages/54/3f/5d7edefd26b6a5187d6fac0f5065ee286109934f3dea607ef05e53f05b31/av-16.1.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:22bf77a2f658827043a1e184b479c3bf25c4c43ab32353677df2d119f080e28f", size = 40543942, upload-time = "2026-01-11T09:58:49.759Z" },
- { url = "https://files.pythonhosted.org/packages/1b/24/f8b17897b67be0900a211142f5646a99d896168f54d57c81f3e018853796/av-16.1.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:2dd419d262e6a71cab206d80bbf28e0a10d0f227b671cdf5e854c028faa2d043", size = 41924336, upload-time = "2026-01-11T09:58:53.344Z" },
- { url = "https://files.pythonhosted.org/packages/1c/cf/d32bc6bbbcf60b65f6510c54690ed3ae1c4ca5d9fafbce835b6056858686/av-16.1.0-cp314-cp314-win_amd64.whl", hash = "sha256:53585986fd431cd436f290fba662cfb44d9494fbc2949a183de00acc5b33fa88", size = 31735077, upload-time = "2026-01-11T09:58:56.684Z" },
- { url = "https://files.pythonhosted.org/packages/53/f4/9b63dc70af8636399bd933e9df4f3025a0294609510239782c1b746fc796/av-16.1.0-cp314-cp314t-macosx_11_0_x86_64.whl", hash = "sha256:76f5ed8495cf41e1209a5775d3699dc63fdc1740b94a095e2485f13586593205", size = 27014423, upload-time = "2026-01-11T09:58:59.703Z" },
- { url = "https://files.pythonhosted.org/packages/d1/da/787a07a0d6ed35a0888d7e5cfb8c2ffa202f38b7ad2c657299fac08eb046/av-16.1.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:8d55397190f12a1a3ae7538be58c356cceb2bf50df1b33523817587748ce89e5", size = 21595536, upload-time = "2026-01-11T09:59:02.508Z" },
- { url = "https://files.pythonhosted.org/packages/d8/f4/9a7d8651a611be6e7e3ab7b30bb43779899c8cac5f7293b9fb634c44a3f3/av-16.1.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:9d51d9037437218261b4bbf9df78a95e216f83d7774fbfe8d289230b5b2e28e2", size = 40642490, upload-time = "2026-01-11T09:59:05.842Z" },
- { url = "https://files.pythonhosted.org/packages/6b/e4/eb79bc538a94b4ff93cd4237d00939cba797579f3272490dd0144c165a21/av-16.1.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:0ce07a89c15644407f49d942111ca046e323bbab0a9078ff43ee57c9b4a50dad", size = 41976905, upload-time = "2026-01-11T09:59:09.169Z" },
- { url = "https://files.pythonhosted.org/packages/5e/f5/f6db0dd86b70167a4d55ee0d9d9640983c570d25504f2bde42599f38241e/av-16.1.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:cac0c074892ea97113b53556ff41c99562db7b9f09f098adac1f08318c2acad5", size = 41770481, upload-time = "2026-01-11T09:59:12.74Z" },
- { url = "https://files.pythonhosted.org/packages/9e/8b/33651d658e45e16ab7671ea5fcf3d20980ea7983234f4d8d0c63c65581a5/av-16.1.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:7dec3dcbc35a187ce450f65a2e0dda820d5a9e6553eea8344a1459af11c98649", size = 43036824, upload-time = "2026-01-11T09:59:16.507Z" },
- { url = "https://files.pythonhosted.org/packages/83/41/7f13361db54d7e02f11552575c0384dadaf0918138f4eaa82ea03a9f9580/av-16.1.0-cp314-cp314t-win_amd64.whl", hash = "sha256:6f90dc082ff2068ddbe77618400b44d698d25d9c4edac57459e250c16b33d700", size = 31948164, upload-time = "2026-01-11T09:59:19.501Z" },
-]
-
-[[package]]
-name = "babel"
-version = "2.17.0"
+version = "17.0.0"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/7d/6b/d52e42361e1aa00709585ecc30b3f9684b3ab62530771402248b1b1d6240/babel-2.17.0.tar.gz", hash = "sha256:0c54cffb19f690cdcc52a3b50bcbf71e07a808d1c80d549f2459b9d2cf0afb9d", size = 9951852, upload-time = "2025-02-01T15:17:41.026Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/b2/eb/abca886df3a091bc406feb5ff71b4c4f426beaae6b71b9697264ce8c7211/av-17.0.0.tar.gz", hash = "sha256:c53685df73775a8763c375c7b2d62a6cb149d992a26a4b098204da42ade8c3df", size = 4410769, upload-time = "2026-03-14T14:38:45.868Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/b7/b8/3fe70c75fe32afc4bb507f75563d39bc5642255d1d94f1f23604725780bf/babel-2.17.0-py3-none-any.whl", hash = "sha256:4d0b53093fdfb4b21c92b5213dba5a1b23885afa8383709427046b21c366e5f2", size = 10182537, upload-time = "2025-02-01T15:17:37.39Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/fb/55e3b5b5d1fc61466292f26fbcbabafa2642f378dc48875f8f554591e1a4/av-17.0.0-cp311-abi3-macosx_11_0_x86_64.whl", hash = "sha256:ed4013fac77c309a4a68141dcf6148f1821bb1073a36d4289379762a6372f711", size = 23238424, upload-time = "2026-03-14T14:38:05.856Z" },
+ { url = "https://files.pythonhosted.org/packages/52/03/9ace1acc08bc9ae38c14bf3a4b1360e995e4d999d1d33c2cbd7c9e77582a/av-17.0.0-cp311-abi3-macosx_14_0_arm64.whl", hash = "sha256:e44b6c83e9f3be9f79ee87d0b77a27cea9a9cd67bd630362c86b7e56a748dfbb", size = 18709043, upload-time = "2026-03-14T14:38:08.288Z" },
+ { url = "https://files.pythonhosted.org/packages/00/c0/637721f3cd5bb8bd16105a1a08efd781fc12f449931bdb3a4d0cfd63fa55/av-17.0.0-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:b440da6ac47da0629d509316f24bcd858f33158dbdd0f1b7293d71e99beb26de", size = 34018780, upload-time = "2026-03-14T14:38:10.45Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/59/d19bc3257dd985d55337d7f0414c019414b97e16cd3690ebf9941a847543/av-17.0.0-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:1060cba85f97f4a337311169d92c0b5e143452cfa5ca0e65fa499d7955e8592e", size = 36358757, upload-time = "2026-03-14T14:38:13.092Z" },
+ { url = "https://files.pythonhosted.org/packages/52/6c/a1f4f2677bae6f2ade7a8a18e90ebdcf70690c9b1c4e40e118aa30fa313f/av-17.0.0-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:deda202e6021cfc7ba3e816897760ec5431309d59a4da1f75df3c0e9413d71e7", size = 35195281, upload-time = "2026-03-14T14:38:15.789Z" },
+ { url = "https://files.pythonhosted.org/packages/90/ea/52b0fc6f69432c7bf3f5fbe6f707113650aa40a1a05b9096ffc2bba4f77d/av-17.0.0-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:ffaf266a1a9c2148072de0a4b5ae98061465178d2cfaa69ee089761149342974", size = 37444817, upload-time = "2026-03-14T14:38:18.563Z" },
+ { url = "https://files.pythonhosted.org/packages/34/ad/d2172966282cb8f146c13b6be7416efefde74186460c5e1708ddfc13dba6/av-17.0.0-cp311-abi3-win_amd64.whl", hash = "sha256:45a35a40b2875bf2f98de7c952d74d960f92f319734e6d28e03b4c62a49e6f49", size = 28888553, upload-time = "2026-03-14T14:38:21.223Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/bb/c5a4c4172c514d631fb506e6366b503576b8c7f29809cf42aca73e28ff01/av-17.0.0-cp311-abi3-win_arm64.whl", hash = "sha256:3d32e9b5c5bbcb872a0b6917b352a1db8a42142237826c9b49a36d5dbd9e9c26", size = 21916910, upload-time = "2026-03-14T14:38:23.706Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/8e/c40ac08e63f79387c59f6ecc38f47d4c942b549130eee579ec1a91f6a291/av-17.0.0-cp314-cp314t-macosx_11_0_x86_64.whl", hash = "sha256:d13250fb4b4522e9a6bec32da082556d5f257110ea223758151375748d9bbe25", size = 23483029, upload-time = "2026-03-14T14:38:25.758Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/fb/b4419494bfc249163ec393c613966d66db7e95c76da3345711cd115a79df/av-17.0.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:dbb56aa3b7ae72451d1bf6e9d37c7d83d39b97af712f73583ff419fbf08fc237", size = 18920446, upload-time = "2026-03-14T14:38:27.905Z" },
+ { url = "https://files.pythonhosted.org/packages/30/62/c2306d91602ddad2c56106f21dcb334fd51d5ea2e952f7fa025bb8aa39fc/av-17.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:a213ac9e83b7ab12c2e9f277a09cac8e9d85cf0883efdab7a87a60e2e4e48879", size = 37477266, upload-time = "2026-03-14T14:38:30.404Z" },
+ { url = "https://files.pythonhosted.org/packages/28/cd/c8510a9607886785c0b3ca019d503e888c3757529be42a7287fe2bfa92d5/av-17.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:e15c88bb0921f9435bcc5a27a0863dba571a80ad5e1389c4fcf2073833bb4a74", size = 39572988, upload-time = "2026-03-14T14:38:32.984Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/2d/207d9361e25b5abec9be335bbab4df6b6b838e2214be4b374f4cfb285427/av-17.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:096cfd1e9fc896506726c7c42aaf9b370e78c2f257cde4d6ddb6c889bfcc49ec", size = 38399591, upload-time = "2026-03-14T14:38:35.465Z" },
+ { url = "https://files.pythonhosted.org/packages/73/ca/307740c6aa2980966bf11383ffcb04bacc5b13f3d268ab4cfb274ad6f793/av-17.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:3649ab3d2c7f58049ded1a36e100c0d8fd529cf258f41dd88678ba824034d8c9", size = 40590681, upload-time = "2026-03-14T14:38:38.269Z" },
+ { url = "https://files.pythonhosted.org/packages/35/f2/6fdb26d0651adf409864cb2a0d60da107e467d3d1aabc94b234ead54324a/av-17.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:e5002271ab2135b551d980c2db8f3299d452e3b9d3633f24f6bb57fffe91cd10", size = 29216337, upload-time = "2026-03-14T14:38:40.83Z" },
+ { url = "https://files.pythonhosted.org/packages/41/0a/0896b829a39b5669a2d811e1a79598de661693685cd62b31f11d0c18e65b/av-17.0.0-cp314-cp314t-win_arm64.whl", hash = "sha256:dba98603fc4665b4f750de86fbaf6c0cfaece970671a9b529e0e3d1711e8367e", size = 22071058, upload-time = "2026-03-14T14:38:43.663Z" },
]
[[package]]
-name = "backports-asyncio-runner"
-version = "1.2.0"
+name = "babel"
+version = "2.18.0"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/8e/ff/70dca7d7cb1cbc0edb2c6cc0c38b65cba36cccc491eca64cabd5fe7f8670/backports_asyncio_runner-1.2.0.tar.gz", hash = "sha256:a5aa7b2b7d8f8bfcaa2b57313f70792df84e32a2a746f585213373f900b42162", size = 69893, upload-time = "2025-07-02T02:27:15.685Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/7d/b2/51899539b6ceeeb420d40ed3cd4b7a40519404f9baf3d4ac99dc413a834b/babel-2.18.0.tar.gz", hash = "sha256:b80b99a14bd085fcacfa15c9165f651fbb3406e66cc603abf11c5750937c992d", size = 9959554, upload-time = "2026-02-01T12:30:56.078Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/a0/59/76ab57e3fe74484f48a53f8e337171b4a2349e506eabe136d7e01d059086/backports_asyncio_runner-1.2.0-py3-none-any.whl", hash = "sha256:0da0a936a8aeb554eccb426dc55af3ba63bcdc69fa1a600b5bb305413a4477b5", size = 12313, upload-time = "2025-07-02T02:27:14.263Z" },
+ { url = "https://files.pythonhosted.org/packages/77/f5/21d2de20e8b8b0408f0681956ca2c69f1320a3848ac50e6e7f39c6159675/babel-2.18.0-py3-none-any.whl", hash = "sha256:e2b422b277c2b9a9630c1d7903c2a00d0830c409c59ac8cae9081c92f1aeba35", size = 10196845, upload-time = "2026-02-01T12:30:53.445Z" },
]
[[package]]
@@ -476,10 +450,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/c0/f6/688d2cd64bfd0b14d805ddb8a565e11ca1fb0fd6817175d58b10052b6d88/bcrypt-5.0.0-cp39-abi3-win32.whl", hash = "sha256:64d7ce196203e468c457c37ec22390f1a61c85c6f0b8160fd752940ccfb3a683", size = 153725, upload-time = "2025-09-25T19:50:34.384Z" },
{ url = "https://files.pythonhosted.org/packages/9f/b9/9d9a641194a730bda138b3dfe53f584d61c58cd5230e37566e83ec2ffa0d/bcrypt-5.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:64ee8434b0da054d830fa8e89e1c8bf30061d539044a39524ff7dec90481e5c2", size = 150912, upload-time = "2025-09-25T19:50:35.69Z" },
{ url = "https://files.pythonhosted.org/packages/27/44/d2ef5e87509158ad2187f4dd0852df80695bb1ee0cfe0a684727b01a69e0/bcrypt-5.0.0-cp39-abi3-win_arm64.whl", hash = "sha256:f2347d3534e76bf50bca5500989d6c1d05ed64b440408057a37673282c654927", size = 144953, upload-time = "2025-09-25T19:50:37.32Z" },
- { url = "https://files.pythonhosted.org/packages/8a/75/4aa9f5a4d40d762892066ba1046000b329c7cd58e888a6db878019b282dc/bcrypt-5.0.0-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:7edda91d5ab52b15636d9c30da87d2cc84f426c72b9dba7a9b4fe142ba11f534", size = 271180, upload-time = "2025-09-25T19:50:38.575Z" },
- { url = "https://files.pythonhosted.org/packages/54/79/875f9558179573d40a9cc743038ac2bf67dfb79cecb1e8b5d70e88c94c3d/bcrypt-5.0.0-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:046ad6db88edb3c5ece4369af997938fb1c19d6a699b9c1b27b0db432faae4c4", size = 273791, upload-time = "2025-09-25T19:50:39.913Z" },
- { url = "https://files.pythonhosted.org/packages/bc/fe/975adb8c216174bf70fc17535f75e85ac06ed5252ea077be10d9cff5ce24/bcrypt-5.0.0-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:dcd58e2b3a908b5ecc9b9df2f0085592506ac2d5110786018ee5e160f28e0911", size = 270746, upload-time = "2025-09-25T19:50:43.306Z" },
- { url = "https://files.pythonhosted.org/packages/e4/f8/972c96f5a2b6c4b3deca57009d93e946bbdbe2241dca9806d502f29dd3ee/bcrypt-5.0.0-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:6b8f520b61e8781efee73cba14e3e8c9556ccfb375623f4f97429544734545b4", size = 273375, upload-time = "2025-09-25T19:50:45.43Z" },
]
[[package]]
@@ -501,32 +471,6 @@ version = "3.8.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/95/06/92fdc84448d324ab8434b78e65caf4fb4c6c90b4f8ad9bdd4c8021bfaf1e/bitarray-3.8.0.tar.gz", hash = "sha256:3eae38daffd77c9621ae80c16932eea3fb3a4af141fb7cc724d4ad93eff9210d", size = 151991, upload-time = "2025-11-02T21:41:15.117Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/e8/b9/8a645fd36fc4c01ee223f97eccd4699c2f2e91681ccb33c0e963881c8e58/bitarray-3.8.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:f08342dc8d19214faa7ef99574dea6c37a2790d6d04a9793ef8fa76c188dc08d", size = 148504, upload-time = "2025-11-02T21:38:54.596Z" },
- { url = "https://files.pythonhosted.org/packages/c0/f4/11b562e13ff732bd0674376f367f0a272034ebc28b8efbafbeb924552d21/bitarray-3.8.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:792462abfeeca6cc8c6c1e6d27e14319682f0182f6b0ba37befe911af794db70", size = 145481, upload-time = "2025-11-02T21:38:56.253Z" },
- { url = "https://files.pythonhosted.org/packages/d3/7c/5a2487da579491b38abab3b437e01d3b05be6e16e69cc5eb304040dcebd5/bitarray-3.8.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0df69d26f21a9d2f1b20266f6737fa43f08aa5015c99900fb69f255fbe4dabb4", size = 322760, upload-time = "2025-11-02T21:38:57.189Z" },
- { url = "https://files.pythonhosted.org/packages/8d/59/f0ef82d6a878d4af1b4961d208a716317929aa172fc0dfa5f4115319a873/bitarray-3.8.0-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b4f10d3f304be7183fac79bf2cd997f82e16aa9a9f37343d76c026c6e435a8a8", size = 350332, upload-time = "2025-11-02T21:38:58.238Z" },
- { url = "https://files.pythonhosted.org/packages/e5/ec/d444b22fce853327d4a8adec1de9987e11b28fcc2d7204dcbc544e196ed9/bitarray-3.8.0-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fc98ff43abad61f00515ad9a06213b7716699146e46eabd256cdfe7cb522bd97", size = 360787, upload-time = "2025-11-02T21:38:59.239Z" },
- { url = "https://files.pythonhosted.org/packages/9f/9e/60b205f52ea9ff155e9f12249090475159c909039daa29e47cd95e115dd5/bitarray-3.8.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:81c6b4a6c1af800d52a6fa32389ef8f4281583f4f99dc1a40f2bb47667281541", size = 329050, upload-time = "2025-11-02T21:39:00.455Z" },
- { url = "https://files.pythonhosted.org/packages/e3/da/2ce373b423bc85a0eb93ee1cba3977971259a92a116932632f417b1b04d2/bitarray-3.8.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:f3fd8df63c41ff6a676d031956aebf68ebbc687b47c507da25501eb22eec341f", size = 320507, upload-time = "2025-11-02T21:39:01.714Z" },
- { url = "https://files.pythonhosted.org/packages/2a/88/437408a2674b8bdb02063dd1535969b9c73cb8fdd197485de431e506c50e/bitarray-3.8.0-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:f0ce9d9e07c75da8027c62b4c9f45771d1d8aae7dc9ad7fb606c6a5aedbe9741", size = 348449, upload-time = "2025-11-02T21:39:03.124Z" },
- { url = "https://files.pythonhosted.org/packages/97/46/d799e7e731c778b6dcb4627bafd395102065e5ab15a4a31f4222a3e20706/bitarray-3.8.0-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:8a9c962c64a4c08def58b9799333e33af94ec53038cf151d36edacdb41f81646", size = 344776, upload-time = "2025-11-02T21:39:04.147Z" },
- { url = "https://files.pythonhosted.org/packages/b3/9a/129fff56d22d316b1c848c6e13e64191485756b5cd6ceb08e640edb80020/bitarray-3.8.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:1a54d7e7999735faacdcbe8128e30207abc2caf9f9fd7102d180b32f1b78bfce", size = 325899, upload-time = "2025-11-02T21:39:05.118Z" },
- { url = "https://files.pythonhosted.org/packages/f8/ba/4b01e99452ecc39f4abccf9bf83fe0f01c390e9794dad2d04b2c8b893c5f/bitarray-3.8.0-cp310-cp310-win32.whl", hash = "sha256:3ea52df96566457735314794422274bd1962066bfb609e7eea9113d70cf04ffe", size = 142756, upload-time = "2025-11-02T21:39:06.402Z" },
- { url = "https://files.pythonhosted.org/packages/18/3f/c83635a67d90f45f88012468566c233eed1e9e9a9184fa882ba4039fadb3/bitarray-3.8.0-cp310-cp310-win_amd64.whl", hash = "sha256:82a07de83dce09b4fa1bccbdc8bde8f188b131666af0dc9048ba0a0e448d8a3b", size = 149527, upload-time = "2025-11-02T21:39:07.377Z" },
- { url = "https://files.pythonhosted.org/packages/33/46/391b3902a523d4555313640746460b19d317c6233d9379e150af97fa1554/bitarray-3.8.0-cp310-cp310-win_arm64.whl", hash = "sha256:c5ba07e58fd98c9782201e79eb8dd4225733d212a5a3700f9a84d329bd0463a6", size = 146453, upload-time = "2025-11-02T21:39:08.624Z" },
- { url = "https://files.pythonhosted.org/packages/bc/7d/63558f1d0eb09217a3d30c1c847890879973e224a728fcff9391fab999b8/bitarray-3.8.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:25b9cff6c9856bc396232e2f609ea0c5ec1a8a24c500cee4cca96ba8a3cd50b6", size = 148502, upload-time = "2025-11-02T21:39:09.993Z" },
- { url = "https://files.pythonhosted.org/packages/5e/7b/f957ad211cb0172965b5f0881b67b99e2b6d41512af0a1001f44a44ddf4a/bitarray-3.8.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:4d9984017314da772f5f7460add7a0301a4ffc06c72c2998bb16c300a6253607", size = 145484, upload-time = "2025-11-02T21:39:10.904Z" },
- { url = "https://files.pythonhosted.org/packages/9f/dc/897973734f14f91467a3a795a4624752238053ecffaec7c8bbda1e363fda/bitarray-3.8.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bbbbfbb7d039b20d289ce56b1beb46138d65769d04af50c199c6ac4cb6054d52", size = 330909, upload-time = "2025-11-02T21:39:12.276Z" },
- { url = "https://files.pythonhosted.org/packages/67/be/24b4b792426d92de289e73e09682915d567c2e69d47e8857586cbdc865d0/bitarray-3.8.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f1f723e260c35e1c7c57a09d3a6ebe681bd56c83e1208ae3ce1869b7c0d10d4f", size = 358469, upload-time = "2025-11-02T21:39:13.766Z" },
- { url = "https://files.pythonhosted.org/packages/3e/0e/2eda69a7a59a6998df8fb57cc9d1e0e62888c599fb5237b0a8b479a01afb/bitarray-3.8.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:cbd1660fb48827381ce3a621a4fdc237959e1cd4e98b098952a8f624a0726425", size = 369131, upload-time = "2025-11-02T21:39:15.041Z" },
- { url = "https://files.pythonhosted.org/packages/f7/7b/8a372d6635a6b2622477b2f96a569b2cd0318a62bc95a4a2144c7942c987/bitarray-3.8.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:df6d7bf3e15b7e6e202a16ff4948a51759354016026deb04ab9b5acbbe35e096", size = 337089, upload-time = "2025-11-02T21:39:16.124Z" },
- { url = "https://files.pythonhosted.org/packages/93/f0/8eca934dbe5dee47a0e5ef44eeb72e85acacc8097c27cd164337bc4ec5d3/bitarray-3.8.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d5c931ec1c03111718cabf85f6012bb2815fa0ce578175567fa8d6f2cc15d3b4", size = 328504, upload-time = "2025-11-02T21:39:17.321Z" },
- { url = "https://files.pythonhosted.org/packages/88/dd/928b8e23a9950f8a8bfc42bc1e7de41f4e27f57de01a716308be5f683c2b/bitarray-3.8.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:41b53711f89008ba2de62e4c2d2260a8b357072fd4f18e1351b28955db2719dc", size = 356461, upload-time = "2025-11-02T21:39:18.396Z" },
- { url = "https://files.pythonhosted.org/packages/a9/93/4fb58417aff47fa2fe1874a39c9346b589a1d78c93a9cb24cccede5dc737/bitarray-3.8.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:4f298daaaea58d45e245a132d6d2bdfb6f856da50dc03d75ebb761439fb626cf", size = 353008, upload-time = "2025-11-02T21:39:19.828Z" },
- { url = "https://files.pythonhosted.org/packages/da/54/aa04e4a7b45aa5913f08ee377d43319b0979925e3c0407882eb29df3be66/bitarray-3.8.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:30989a2451b693c3f9359d91098a744992b5431a0be4858f1fdf0ec76b457125", size = 334048, upload-time = "2025-11-02T21:39:20.924Z" },
- { url = "https://files.pythonhosted.org/packages/da/52/e851f41076df014c05d6ac1ce34fbf7db5fa31241da3e2f09bb2be9e283d/bitarray-3.8.0-cp311-cp311-win32.whl", hash = "sha256:e5aed4754895942ae15ffa48c52d181e1c1463236fda68d2dba29c03aa61786b", size = 142907, upload-time = "2025-11-02T21:39:22.312Z" },
- { url = "https://files.pythonhosted.org/packages/28/01/db0006148b1dd13b4ac2686df8fa57d12f5887df313a506e939af0cb0997/bitarray-3.8.0-cp311-cp311-win_amd64.whl", hash = "sha256:22c540ed20167d3dbb1e2d868ca935180247d620c40eace90efa774504a40e3b", size = 149670, upload-time = "2025-11-02T21:39:23.341Z" },
- { url = "https://files.pythonhosted.org/packages/7b/ea/b7d55ee269b1426f758a535c9ec2a07c056f20f403fa981685c3c8b4798c/bitarray-3.8.0-cp311-cp311-win_arm64.whl", hash = "sha256:84b52b2cf77bb7f703d16c4007b021078dbbe6cf8ffb57abe81a7bacfc175ef2", size = 146709, upload-time = "2025-11-02T21:39:24.343Z" },
{ url = "https://files.pythonhosted.org/packages/82/a0/0c41d893eda756315491adfdbf9bc928aee3d377a7f97a8834d453aa5de1/bitarray-3.8.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:f2fcbe9b3a5996b417e030aa33a562e7e20dfc86271e53d7e841fc5df16268b8", size = 148575, upload-time = "2025-11-02T21:39:25.718Z" },
{ url = "https://files.pythonhosted.org/packages/0e/30/12ab2f4a4429bd844b419c37877caba93d676d18be71354fbbeb21d9f4cc/bitarray-3.8.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:cd761d158f67e288fd0ebe00c3b158095ce80a4bc7c32b60c7121224003ba70d", size = 145454, upload-time = "2025-11-02T21:39:26.695Z" },
{ url = "https://files.pythonhosted.org/packages/26/58/314b3e3f219533464e120f0c51ac5123e7b1c1b91f725a4073fb70c5a858/bitarray-3.8.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c394a3f055b49f92626f83c1a0b6d6cd2c628f1ccd72481c3e3c6aa4695f3b20", size = 332949, upload-time = "2025-11-02T21:39:27.801Z" },
@@ -583,14 +527,15 @@ wheels = [
[[package]]
name = "bitstring"
-version = "4.3.1"
+version = "4.4.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "bitarray" },
+ { name = "tibs" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/15/a8/a80c890db75d5bdd5314b5de02c4144c7de94fd0cefcae51acaeb14c6a3f/bitstring-4.3.1.tar.gz", hash = "sha256:a08bc09d3857216d4c0f412a1611056f1cc2b64fd254fb1e8a0afba7cfa1a95a", size = 251426, upload-time = "2025-03-22T09:39:06.978Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/36/d3/de6fe4e7065df8c2f1ac1766f5fdccbe75bc18af2cf2dbeecd34d68e1518/bitstring-4.4.0.tar.gz", hash = "sha256:e682ac522bb63e041d16cbc9d0ca86a4f00194db16d0847c7efe066f836b2e37", size = 255209, upload-time = "2026-03-10T20:29:14.824Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/75/2d/174566b533755ddf8efb32a5503af61c756a983de379f8ad3aed6a982d38/bitstring-4.3.1-py3-none-any.whl", hash = "sha256:69d1587f0ac18dc7d93fc7e80d5f447161a33e57027e726dc18a0a8bacf1711a", size = 71930, upload-time = "2025-03-22T09:39:05.163Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/02/1a870bab76f2896d827aa4963be95e56675ffa1453e53525d13c43036edf/bitstring-4.4.0-py3-none-any.whl", hash = "sha256:feac49524fcf3ef27e6081e86f02b10d2adf6c3773bf22fbe0e7eea9534bc737", size = 76846, upload-time = "2026-03-10T20:29:12.832Z" },
]
[[package]]
@@ -600,22 +545,13 @@ source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "click" },
{ name = "mypy-extensions" },
- { name = "packaging" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
{ name = "pathspec" },
{ name = "platformdirs" },
- { name = "tomli", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "typing-extensions", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
]
sdist = { url = "https://files.pythonhosted.org/packages/a2/47/c9997eb470a7f48f7aaddd3d9a828244a2e4199569e38128715c48059ac1/black-24.4.2.tar.gz", hash = "sha256:c872b53057f000085da66a19c55d68f6f8ddcac2642392ad3a355878406fbd4d", size = 642299, upload-time = "2024-04-26T00:32:15.305Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/40/f6/3adc48c210527a7b651aaed43824a9b8bd04b3fb361a5227bad046e1c876/black-24.4.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:dd1b5a14e417189db4c7b64a6540f31730713d173f0b63e55fabd52d61d8fdce", size = 1631487, upload-time = "2024-04-26T00:40:28.969Z" },
- { url = "https://files.pythonhosted.org/packages/a2/25/70aa1bec12c841a03e333e312daa0cf2fee50ea6336ac4851c93c0e2b411/black-24.4.2-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:8e537d281831ad0e71007dcdcbe50a71470b978c453fa41ce77186bbe0ed6021", size = 1456317, upload-time = "2024-04-26T00:39:10.333Z" },
- { url = "https://files.pythonhosted.org/packages/e0/7d/7f8df0fdbbbefc4362d3eca6b69b7a8a4249a8a88dabc00a207d31fddcd7/black-24.4.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:eaea3008c281f1038edb473c1aa8ed8143a5535ff18f978a318f10302b254063", size = 1822765, upload-time = "2024-04-26T00:34:56.436Z" },
- { url = "https://files.pythonhosted.org/packages/5c/21/1ee97841c469c1551133cbe47448cdba9628c7d9431f74f114f02e3b233c/black-24.4.2-cp310-cp310-win_amd64.whl", hash = "sha256:7768a0dbf16a39aa5e9a3ded568bb545c8c2727396d063bbaf847df05b08cd96", size = 1409336, upload-time = "2024-04-26T00:35:30.392Z" },
- { url = "https://files.pythonhosted.org/packages/9b/f7/591d601c3046ceb65b97291dfe87fa25124cffac3d97aaaba89d0f0d7bdf/black-24.4.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:257d724c2c9b1660f353b36c802ccece186a30accc7742c176d29c146df6e474", size = 1615013, upload-time = "2024-04-26T00:39:49.415Z" },
- { url = "https://files.pythonhosted.org/packages/c9/17/5e0036b265bbf6bc44970d93d48febcbc03701b671db3c9603fd43ebc616/black-24.4.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:bdde6f877a18f24844e381d45e9947a49e97933573ac9d4345399be37621e26c", size = 1436163, upload-time = "2024-04-26T00:40:20.267Z" },
- { url = "https://files.pythonhosted.org/packages/c5/48/34176b522e8cff4620a5d96c2e323ff2413f574870eb25efa8025885e028/black-24.4.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e151054aa00bad1f4e1f04919542885f89f5f7d086b8a59e5000e6c616896ffb", size = 1803382, upload-time = "2024-04-26T00:34:38.665Z" },
- { url = "https://files.pythonhosted.org/packages/74/ce/e8eec1a77edbfa982bee3b5460dcdd4fe0e4e3165fc15d8ec44d04da7776/black-24.4.2-cp311-cp311-win_amd64.whl", hash = "sha256:7e122b1c4fb252fd85df3ca93578732b4749d9be076593076ef4d07a0233c3e1", size = 1417802, upload-time = "2024-04-26T00:35:08.804Z" },
{ url = "https://files.pythonhosted.org/packages/f4/75/3a29de3bda4006cc280d833b5d961cf7df3810a21f49e7a63a7e551fb351/black-24.4.2-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:accf49e151c8ed2c0cdc528691838afd217c50412534e876a19270fea1e28e2d", size = 1645176, upload-time = "2024-04-26T00:42:35.606Z" },
{ url = "https://files.pythonhosted.org/packages/be/b8/9c152301774fa62a265b035a8ede4d6280827904ea1af8c3be10a28d3187/black-24.4.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:88c57dc656038f1ab9f92b3eb5335ee9b021412feaa46330d5eba4e51fe49b04", size = 1446227, upload-time = "2024-04-26T00:40:35.195Z" },
{ url = "https://files.pythonhosted.org/packages/25/6d/eb15a1b155f755f43766cc473618c6e1de6555d6a1764965643f486dcf01/black-24.4.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:be8bef99eb46d5021bf053114442914baeb3649a89dc5f3a555c88737e5e98fc", size = 1832011, upload-time = "2024-04-26T00:34:37.825Z" },
@@ -634,16 +570,16 @@ wheels = [
[[package]]
name = "botocore"
-version = "1.41.5"
+version = "1.42.70"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "jmespath" },
{ name = "python-dateutil" },
{ name = "urllib3" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/90/22/7fe08c726a2e3b11a0aef8bf177e83891c9cb2dc1809d35c9ed91a9e60e6/botocore-1.41.5.tar.gz", hash = "sha256:0367622b811597d183bfcaab4a350f0d3ede712031ce792ef183cabdee80d3bf", size = 14668152, upload-time = "2025-11-26T20:27:38.026Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/66/54/b80e1fcee4f732e0e9314bbb8679be9d5690caa1566c4a4cd14e9724d2dd/botocore-1.42.70.tar.gz", hash = "sha256:9ee17553b7febd1a0c1253b3b62ab5d79607eb6163c8fb943470a8893c31d4fa", size = 14997068, upload-time = "2026-03-17T19:43:10.678Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/4e/4e/21cd0b8f365449f1576f93de1ec8718ed18a7a3bc086dfbdeb79437bba7a/botocore-1.41.5-py3-none-any.whl", hash = "sha256:3fef7fcda30c82c27202d232cfdbd6782cb27f20f8e7e21b20606483e66ee73a", size = 14337008, upload-time = "2025-11-26T20:27:35.208Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/51/08f32aea872253173f513ba68122f4300966290677c8e59887b4ffd5d957/botocore-1.42.70-py3-none-any.whl", hash = "sha256:54ed9d25f05f810efd22b0dfda0bb9178df3ad8952b2e4359e05156c9321bd3c", size = 14671393, upload-time = "2026-03-17T19:43:06.777Z" },
]
[[package]]
@@ -664,6 +600,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/9d/2a/9186535ce58db529927f6cf5990a849aa9e052eea3e2cfefe20b9e1802da/bracex-2.6-py3-none-any.whl", hash = "sha256:0b0049264e7340b3ec782b5cb99beb325f36c3782a32e36e876452fd49a09952", size = 11508, upload-time = "2025-06-22T19:12:29.781Z" },
]
+[[package]]
+name = "cachetools"
+version = "4.2.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/74/17/5735dd9f015f03d2d928ea108f3c02075b784ceed05d32a98e7e44ddd114/cachetools-4.2.1.tar.gz", hash = "sha256:f469e29e7aa4cff64d8de4aad95ce76de8ea1125a16c68e0d93f65c3c3dc92e9", size = 24753, upload-time = "2021-01-24T22:40:13.746Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/bb/72/8df2e0dc991f1a1d2c6869404e7622e8ee50d80bff357dbb57c3df70305b/cachetools-4.2.1-py3-none-any.whl", hash = "sha256:1d9d5f567be80f7c07d765e21b814326d78c61eb0c3a637dffc0e5d1796cb2e2", size = 12003, upload-time = "2021-01-24T22:40:11.795Z" },
+]
+
[[package]]
name = "catalogue"
version = "2.0.10"
@@ -675,22 +620,23 @@ wheels = [
[[package]]
name = "causal-conv1d"
-version = "1.6.0"
+version = "1.6.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "ninja" },
- { name = "packaging" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
{ name = "torch", marker = "sys_platform == 'never'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/db/df/63a384c49743b9fc8fec4c05dbd0b515e1c1c2b07e4559acc4fc37c69223/causal_conv1d-1.6.0.tar.gz", hash = "sha256:4eae3220d08e1e88238f3a0a88783147cbdf47f612cc610add75127c7a37ca3e", size = 29356, upload-time = "2026-01-12T17:33:32.794Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/63/15/ec51d77a2df03ee93410f8ee97fceeb7181da213813c51243e9dd6d7e144/causal_conv1d-1.6.1.tar.gz", hash = "sha256:e4a697ec2db3906f012e675125569f8b510b4559bc53e3095143d91369e1221b", size = 29426, upload-time = "2026-03-10T08:56:35.305Z" }
[[package]]
name = "certifi"
-version = "2026.1.4"
+version = "2026.2.25"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/e0/2d/a891ca51311197f6ad14a7ef42e2399f36cf2f9bd44752b3dc4eab60fdc5/certifi-2026.1.4.tar.gz", hash = "sha256:ac726dd470482006e014ad384921ed6438c457018f4b3d204aea4281258b2120", size = 154268, upload-time = "2026-01-04T02:42:41.825Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/af/2d/7bf41579a8986e348fa033a31cdd0e4121114f6bce2457e8876010b092dd/certifi-2026.2.25.tar.gz", hash = "sha256:e887ab5cee78ea814d3472169153c2d12cd43b14bd03329a39a9c6e2e80bfba7", size = 155029, upload-time = "2026-02-25T02:54:17.342Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/e6/ad/3cc14f097111b4de0040c83a525973216457bbeeb63739ef1ed275c1c021/certifi-2026.1.4-py3-none-any.whl", hash = "sha256:9943707519e4add1115f44c2bc244f782c0249876bf51b6599fee1ffbedd685c", size = 152900, upload-time = "2026-01-04T02:42:40.15Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/3c/c17fb3ca2d9c3acff52e30b309f538586f9f5b9c9cf454f3845fc9af4881/certifi-2026.2.25-py3-none-any.whl", hash = "sha256:027692e4402ad994f1c42e52a4997a9763c646b73e4096e4d5d6db8af1d6f0fa", size = 153684, upload-time = "2026-02-25T02:54:15.766Z" },
]
[[package]]
@@ -702,31 +648,6 @@ dependencies = [
]
sdist = { url = "https://files.pythonhosted.org/packages/eb/56/b1ba7935a17738ae8453301356628e8147c79dbb825bcbc73dc7401f9846/cffi-2.0.0.tar.gz", hash = "sha256:44d1b5909021139fe36001ae048dbdde8214afa20200eda0f64c068cac5d5529", size = 523588, upload-time = "2025-09-08T23:24:04.541Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/93/d7/516d984057745a6cd96575eea814fe1edd6646ee6efd552fb7b0921dec83/cffi-2.0.0-cp310-cp310-macosx_10_13_x86_64.whl", hash = "sha256:0cf2d91ecc3fcc0625c2c530fe004f82c110405f101548512cce44322fa8ac44", size = 184283, upload-time = "2025-09-08T23:22:08.01Z" },
- { url = "https://files.pythonhosted.org/packages/9e/84/ad6a0b408daa859246f57c03efd28e5dd1b33c21737c2db84cae8c237aa5/cffi-2.0.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:f73b96c41e3b2adedc34a7356e64c8eb96e03a3782b535e043a986276ce12a49", size = 180504, upload-time = "2025-09-08T23:22:10.637Z" },
- { url = "https://files.pythonhosted.org/packages/50/bd/b1a6362b80628111e6653c961f987faa55262b4002fcec42308cad1db680/cffi-2.0.0-cp310-cp310-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:53f77cbe57044e88bbd5ed26ac1d0514d2acf0591dd6bb02a3ae37f76811b80c", size = 208811, upload-time = "2025-09-08T23:22:12.267Z" },
- { url = "https://files.pythonhosted.org/packages/4f/27/6933a8b2562d7bd1fb595074cf99cc81fc3789f6a6c05cdabb46284a3188/cffi-2.0.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3e837e369566884707ddaf85fc1744b47575005c0a229de3327f8f9a20f4efeb", size = 216402, upload-time = "2025-09-08T23:22:13.455Z" },
- { url = "https://files.pythonhosted.org/packages/05/eb/b86f2a2645b62adcfff53b0dd97e8dfafb5c8aa864bd0d9a2c2049a0d551/cffi-2.0.0-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:5eda85d6d1879e692d546a078b44251cdd08dd1cfb98dfb77b670c97cee49ea0", size = 203217, upload-time = "2025-09-08T23:22:14.596Z" },
- { url = "https://files.pythonhosted.org/packages/9f/e0/6cbe77a53acf5acc7c08cc186c9928864bd7c005f9efd0d126884858a5fe/cffi-2.0.0-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:9332088d75dc3241c702d852d4671613136d90fa6881da7d770a483fd05248b4", size = 203079, upload-time = "2025-09-08T23:22:15.769Z" },
- { url = "https://files.pythonhosted.org/packages/98/29/9b366e70e243eb3d14a5cb488dfd3a0b6b2f1fb001a203f653b93ccfac88/cffi-2.0.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fc7de24befaeae77ba923797c7c87834c73648a05a4bde34b3b7e5588973a453", size = 216475, upload-time = "2025-09-08T23:22:17.427Z" },
- { url = "https://files.pythonhosted.org/packages/21/7a/13b24e70d2f90a322f2900c5d8e1f14fa7e2a6b3332b7309ba7b2ba51a5a/cffi-2.0.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:cf364028c016c03078a23b503f02058f1814320a56ad535686f90565636a9495", size = 218829, upload-time = "2025-09-08T23:22:19.069Z" },
- { url = "https://files.pythonhosted.org/packages/60/99/c9dc110974c59cc981b1f5b66e1d8af8af764e00f0293266824d9c4254bc/cffi-2.0.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:e11e82b744887154b182fd3e7e8512418446501191994dbf9c9fc1f32cc8efd5", size = 211211, upload-time = "2025-09-08T23:22:20.588Z" },
- { url = "https://files.pythonhosted.org/packages/49/72/ff2d12dbf21aca1b32a40ed792ee6b40f6dc3a9cf1644bd7ef6e95e0ac5e/cffi-2.0.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:8ea985900c5c95ce9db1745f7933eeef5d314f0565b27625d9a10ec9881e1bfb", size = 218036, upload-time = "2025-09-08T23:22:22.143Z" },
- { url = "https://files.pythonhosted.org/packages/e2/cc/027d7fb82e58c48ea717149b03bcadcbdc293553edb283af792bd4bcbb3f/cffi-2.0.0-cp310-cp310-win32.whl", hash = "sha256:1f72fb8906754ac8a2cc3f9f5aaa298070652a0ffae577e0ea9bd480dc3c931a", size = 172184, upload-time = "2025-09-08T23:22:23.328Z" },
- { url = "https://files.pythonhosted.org/packages/33/fa/072dd15ae27fbb4e06b437eb6e944e75b068deb09e2a2826039e49ee2045/cffi-2.0.0-cp310-cp310-win_amd64.whl", hash = "sha256:b18a3ed7d5b3bd8d9ef7a8cb226502c6bf8308df1525e1cc676c3680e7176739", size = 182790, upload-time = "2025-09-08T23:22:24.752Z" },
- { url = "https://files.pythonhosted.org/packages/12/4a/3dfd5f7850cbf0d06dc84ba9aa00db766b52ca38d8b86e3a38314d52498c/cffi-2.0.0-cp311-cp311-macosx_10_13_x86_64.whl", hash = "sha256:b4c854ef3adc177950a8dfc81a86f5115d2abd545751a304c5bcf2c2c7283cfe", size = 184344, upload-time = "2025-09-08T23:22:26.456Z" },
- { url = "https://files.pythonhosted.org/packages/4f/8b/f0e4c441227ba756aafbe78f117485b25bb26b1c059d01f137fa6d14896b/cffi-2.0.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:2de9a304e27f7596cd03d16f1b7c72219bd944e99cc52b84d0145aefb07cbd3c", size = 180560, upload-time = "2025-09-08T23:22:28.197Z" },
- { url = "https://files.pythonhosted.org/packages/b1/b7/1200d354378ef52ec227395d95c2576330fd22a869f7a70e88e1447eb234/cffi-2.0.0-cp311-cp311-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:baf5215e0ab74c16e2dd324e8ec067ef59e41125d3eade2b863d294fd5035c92", size = 209613, upload-time = "2025-09-08T23:22:29.475Z" },
- { url = "https://files.pythonhosted.org/packages/b8/56/6033f5e86e8cc9bb629f0077ba71679508bdf54a9a5e112a3c0b91870332/cffi-2.0.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:730cacb21e1bdff3ce90babf007d0a0917cc3e6492f336c2f0134101e0944f93", size = 216476, upload-time = "2025-09-08T23:22:31.063Z" },
- { url = "https://files.pythonhosted.org/packages/dc/7f/55fecd70f7ece178db2f26128ec41430d8720f2d12ca97bf8f0a628207d5/cffi-2.0.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:6824f87845e3396029f3820c206e459ccc91760e8fa24422f8b0c3d1731cbec5", size = 203374, upload-time = "2025-09-08T23:22:32.507Z" },
- { url = "https://files.pythonhosted.org/packages/84/ef/a7b77c8bdc0f77adc3b46888f1ad54be8f3b7821697a7b89126e829e676a/cffi-2.0.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:9de40a7b0323d889cf8d23d1ef214f565ab154443c42737dfe52ff82cf857664", size = 202597, upload-time = "2025-09-08T23:22:34.132Z" },
- { url = "https://files.pythonhosted.org/packages/d7/91/500d892b2bf36529a75b77958edfcd5ad8e2ce4064ce2ecfeab2125d72d1/cffi-2.0.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8941aaadaf67246224cee8c3803777eed332a19d909b47e29c9842ef1e79ac26", size = 215574, upload-time = "2025-09-08T23:22:35.443Z" },
- { url = "https://files.pythonhosted.org/packages/44/64/58f6255b62b101093d5df22dcb752596066c7e89dd725e0afaed242a61be/cffi-2.0.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:a05d0c237b3349096d3981b727493e22147f934b20f6f125a3eba8f994bec4a9", size = 218971, upload-time = "2025-09-08T23:22:36.805Z" },
- { url = "https://files.pythonhosted.org/packages/ab/49/fa72cebe2fd8a55fbe14956f9970fe8eb1ac59e5df042f603ef7c8ba0adc/cffi-2.0.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:94698a9c5f91f9d138526b48fe26a199609544591f859c870d477351dc7b2414", size = 211972, upload-time = "2025-09-08T23:22:38.436Z" },
- { url = "https://files.pythonhosted.org/packages/0b/28/dd0967a76aab36731b6ebfe64dec4e981aff7e0608f60c2d46b46982607d/cffi-2.0.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:5fed36fccc0612a53f1d4d9a816b50a36702c28a2aa880cb8a122b3466638743", size = 217078, upload-time = "2025-09-08T23:22:39.776Z" },
- { url = "https://files.pythonhosted.org/packages/2b/c0/015b25184413d7ab0a410775fdb4a50fca20f5589b5dab1dbbfa3baad8ce/cffi-2.0.0-cp311-cp311-win32.whl", hash = "sha256:c649e3a33450ec82378822b3dad03cc228b8f5963c0c12fc3b1e0ab940f768a5", size = 172076, upload-time = "2025-09-08T23:22:40.95Z" },
- { url = "https://files.pythonhosted.org/packages/ae/8f/dc5531155e7070361eb1b7e4c1a9d896d0cb21c49f807a6c03fd63fc877e/cffi-2.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:66f011380d0e49ed280c789fbd08ff0d40968ee7b665575489afa95c98196ab5", size = 182820, upload-time = "2025-09-08T23:22:42.463Z" },
- { url = "https://files.pythonhosted.org/packages/95/5c/1b493356429f9aecfd56bc171285a4c4ac8697f76e9bbbbb105e537853a1/cffi-2.0.0-cp311-cp311-win_arm64.whl", hash = "sha256:c6638687455baf640e37344fe26d37c404db8b80d037c3d29f58fe8d1c3b194d", size = 177635, upload-time = "2025-09-08T23:22:43.623Z" },
{ url = "https://files.pythonhosted.org/packages/ea/47/4f61023ea636104d4f16ab488e268b93008c3d0bb76893b1b31db1f96802/cffi-2.0.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6d02d6655b0e54f54c4ef0b94eb6be0607b70853c45ce98bd278dc7de718be5d", size = 185271, upload-time = "2025-09-08T23:22:44.795Z" },
{ url = "https://files.pythonhosted.org/packages/df/a2/781b623f57358e360d62cdd7a8c681f074a71d445418a776eef0aadb4ab4/cffi-2.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8eca2a813c1cb7ad4fb74d368c2ffbbb4789d377ee5bb8df98373c2cc0dee76c", size = 181048, upload-time = "2025-09-08T23:22:45.938Z" },
{ url = "https://files.pythonhosted.org/packages/ff/df/a4f0fbd47331ceeba3d37c2e51e9dfc9722498becbeec2bd8bc856c9538a/cffi-2.0.0-cp312-cp312-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:21d1152871b019407d8ac3985f6775c079416c282e431a4da6afe7aefd2bccbe", size = 212529, upload-time = "2025-09-08T23:22:47.349Z" },
@@ -777,91 +698,75 @@ wheels = [
[[package]]
name = "charset-normalizer"
-version = "3.4.4"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/13/69/33ddede1939fdd074bce5434295f38fae7136463422fe4fd3e0e89b98062/charset_normalizer-3.4.4.tar.gz", hash = "sha256:94537985111c35f28720e43603b8e7b43a6ecfb2ce1d3058bbe955b73404e21a", size = 129418, upload-time = "2025-10-14T04:42:32.879Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/1f/b8/6d51fc1d52cbd52cd4ccedd5b5b2f0f6a11bbf6765c782298b0f3e808541/charset_normalizer-3.4.4-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:e824f1492727fa856dd6eda4f7cee25f8518a12f3c4a56a74e8095695089cf6d", size = 209709, upload-time = "2025-10-14T04:40:11.385Z" },
- { url = "https://files.pythonhosted.org/packages/5c/af/1f9d7f7faafe2ddfb6f72a2e07a548a629c61ad510fe60f9630309908fef/charset_normalizer-3.4.4-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4bd5d4137d500351a30687c2d3971758aac9a19208fc110ccb9d7188fbe709e8", size = 148814, upload-time = "2025-10-14T04:40:13.135Z" },
- { url = "https://files.pythonhosted.org/packages/79/3d/f2e3ac2bbc056ca0c204298ea4e3d9db9b4afe437812638759db2c976b5f/charset_normalizer-3.4.4-cp310-cp310-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:027f6de494925c0ab2a55eab46ae5129951638a49a34d87f4c3eda90f696b4ad", size = 144467, upload-time = "2025-10-14T04:40:14.728Z" },
- { url = "https://files.pythonhosted.org/packages/ec/85/1bf997003815e60d57de7bd972c57dc6950446a3e4ccac43bc3070721856/charset_normalizer-3.4.4-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f820802628d2694cb7e56db99213f930856014862f3fd943d290ea8438d07ca8", size = 162280, upload-time = "2025-10-14T04:40:16.14Z" },
- { url = "https://files.pythonhosted.org/packages/3e/8e/6aa1952f56b192f54921c436b87f2aaf7c7a7c3d0d1a765547d64fd83c13/charset_normalizer-3.4.4-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:798d75d81754988d2565bff1b97ba5a44411867c0cf32b77a7e8f8d84796b10d", size = 159454, upload-time = "2025-10-14T04:40:17.567Z" },
- { url = "https://files.pythonhosted.org/packages/36/3b/60cbd1f8e93aa25d1c669c649b7a655b0b5fb4c571858910ea9332678558/charset_normalizer-3.4.4-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d1bb833febdff5c8927f922386db610b49db6e0d4f4ee29601d71e7c2694313", size = 153609, upload-time = "2025-10-14T04:40:19.08Z" },
- { url = "https://files.pythonhosted.org/packages/64/91/6a13396948b8fd3c4b4fd5bc74d045f5637d78c9675585e8e9fbe5636554/charset_normalizer-3.4.4-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9cd98cdc06614a2f768d2b7286d66805f94c48cde050acdbbb7db2600ab3197e", size = 151849, upload-time = "2025-10-14T04:40:20.607Z" },
- { url = "https://files.pythonhosted.org/packages/b7/7a/59482e28b9981d105691e968c544cc0df3b7d6133152fb3dcdc8f135da7a/charset_normalizer-3.4.4-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:077fbb858e903c73f6c9db43374fd213b0b6a778106bc7032446a8e8b5b38b93", size = 151586, upload-time = "2025-10-14T04:40:21.719Z" },
- { url = "https://files.pythonhosted.org/packages/92/59/f64ef6a1c4bdd2baf892b04cd78792ed8684fbc48d4c2afe467d96b4df57/charset_normalizer-3.4.4-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:244bfb999c71b35de57821b8ea746b24e863398194a4014e4c76adc2bbdfeff0", size = 145290, upload-time = "2025-10-14T04:40:23.069Z" },
- { url = "https://files.pythonhosted.org/packages/6b/63/3bf9f279ddfa641ffa1962b0db6a57a9c294361cc2f5fcac997049a00e9c/charset_normalizer-3.4.4-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:64b55f9dce520635f018f907ff1b0df1fdc31f2795a922fb49dd14fbcdf48c84", size = 163663, upload-time = "2025-10-14T04:40:24.17Z" },
- { url = "https://files.pythonhosted.org/packages/ed/09/c9e38fc8fa9e0849b172b581fd9803bdf6e694041127933934184e19f8c3/charset_normalizer-3.4.4-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:faa3a41b2b66b6e50f84ae4a68c64fcd0c44355741c6374813a800cd6695db9e", size = 151964, upload-time = "2025-10-14T04:40:25.368Z" },
- { url = "https://files.pythonhosted.org/packages/d2/d1/d28b747e512d0da79d8b6a1ac18b7ab2ecfd81b2944c4c710e166d8dd09c/charset_normalizer-3.4.4-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:6515f3182dbe4ea06ced2d9e8666d97b46ef4c75e326b79bb624110f122551db", size = 161064, upload-time = "2025-10-14T04:40:26.806Z" },
- { url = "https://files.pythonhosted.org/packages/bb/9a/31d62b611d901c3b9e5500c36aab0ff5eb442043fb3a1c254200d3d397d9/charset_normalizer-3.4.4-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:cc00f04ed596e9dc0da42ed17ac5e596c6ccba999ba6bd92b0e0aef2f170f2d6", size = 155015, upload-time = "2025-10-14T04:40:28.284Z" },
- { url = "https://files.pythonhosted.org/packages/1f/f3/107e008fa2bff0c8b9319584174418e5e5285fef32f79d8ee6a430d0039c/charset_normalizer-3.4.4-cp310-cp310-win32.whl", hash = "sha256:f34be2938726fc13801220747472850852fe6b1ea75869a048d6f896838c896f", size = 99792, upload-time = "2025-10-14T04:40:29.613Z" },
- { url = "https://files.pythonhosted.org/packages/eb/66/e396e8a408843337d7315bab30dbf106c38966f1819f123257f5520f8a96/charset_normalizer-3.4.4-cp310-cp310-win_amd64.whl", hash = "sha256:a61900df84c667873b292c3de315a786dd8dac506704dea57bc957bd31e22c7d", size = 107198, upload-time = "2025-10-14T04:40:30.644Z" },
- { url = "https://files.pythonhosted.org/packages/b5/58/01b4f815bf0312704c267f2ccb6e5d42bcc7752340cd487bc9f8c3710597/charset_normalizer-3.4.4-cp310-cp310-win_arm64.whl", hash = "sha256:cead0978fc57397645f12578bfd2d5ea9138ea0fac82b2f63f7f7c6877986a69", size = 100262, upload-time = "2025-10-14T04:40:32.108Z" },
- { url = "https://files.pythonhosted.org/packages/ed/27/c6491ff4954e58a10f69ad90aca8a1b6fe9c5d3c6f380907af3c37435b59/charset_normalizer-3.4.4-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:6e1fcf0720908f200cd21aa4e6750a48ff6ce4afe7ff5a79a90d5ed8a08296f8", size = 206988, upload-time = "2025-10-14T04:40:33.79Z" },
- { url = "https://files.pythonhosted.org/packages/94/59/2e87300fe67ab820b5428580a53cad894272dbb97f38a7a814a2a1ac1011/charset_normalizer-3.4.4-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5f819d5fe9234f9f82d75bdfa9aef3a3d72c4d24a6e57aeaebba32a704553aa0", size = 147324, upload-time = "2025-10-14T04:40:34.961Z" },
- { url = "https://files.pythonhosted.org/packages/07/fb/0cf61dc84b2b088391830f6274cb57c82e4da8bbc2efeac8c025edb88772/charset_normalizer-3.4.4-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:a59cb51917aa591b1c4e6a43c132f0cdc3c76dbad6155df4e28ee626cc77a0a3", size = 142742, upload-time = "2025-10-14T04:40:36.105Z" },
- { url = "https://files.pythonhosted.org/packages/62/8b/171935adf2312cd745d290ed93cf16cf0dfe320863ab7cbeeae1dcd6535f/charset_normalizer-3.4.4-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8ef3c867360f88ac904fd3f5e1f902f13307af9052646963ee08ff4f131adafc", size = 160863, upload-time = "2025-10-14T04:40:37.188Z" },
- { url = "https://files.pythonhosted.org/packages/09/73/ad875b192bda14f2173bfc1bc9a55e009808484a4b256748d931b6948442/charset_normalizer-3.4.4-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d9e45d7faa48ee908174d8fe84854479ef838fc6a705c9315372eacbc2f02897", size = 157837, upload-time = "2025-10-14T04:40:38.435Z" },
- { url = "https://files.pythonhosted.org/packages/6d/fc/de9cce525b2c5b94b47c70a4b4fb19f871b24995c728e957ee68ab1671ea/charset_normalizer-3.4.4-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:840c25fb618a231545cbab0564a799f101b63b9901f2569faecd6b222ac72381", size = 151550, upload-time = "2025-10-14T04:40:40.053Z" },
- { url = "https://files.pythonhosted.org/packages/55/c2/43edd615fdfba8c6f2dfbd459b25a6b3b551f24ea21981e23fb768503ce1/charset_normalizer-3.4.4-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:ca5862d5b3928c4940729dacc329aa9102900382fea192fc5e52eb69d6093815", size = 149162, upload-time = "2025-10-14T04:40:41.163Z" },
- { url = "https://files.pythonhosted.org/packages/03/86/bde4ad8b4d0e9429a4e82c1e8f5c659993a9a863ad62c7df05cf7b678d75/charset_normalizer-3.4.4-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d9c7f57c3d666a53421049053eaacdd14bbd0a528e2186fcb2e672effd053bb0", size = 150019, upload-time = "2025-10-14T04:40:42.276Z" },
- { url = "https://files.pythonhosted.org/packages/1f/86/a151eb2af293a7e7bac3a739b81072585ce36ccfb4493039f49f1d3cae8c/charset_normalizer-3.4.4-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:277e970e750505ed74c832b4bf75dac7476262ee2a013f5574dd49075879e161", size = 143310, upload-time = "2025-10-14T04:40:43.439Z" },
- { url = "https://files.pythonhosted.org/packages/b5/fe/43dae6144a7e07b87478fdfc4dbe9efd5defb0e7ec29f5f58a55aeef7bf7/charset_normalizer-3.4.4-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:31fd66405eaf47bb62e8cd575dc621c56c668f27d46a61d975a249930dd5e2a4", size = 162022, upload-time = "2025-10-14T04:40:44.547Z" },
- { url = "https://files.pythonhosted.org/packages/80/e6/7aab83774f5d2bca81f42ac58d04caf44f0cc2b65fc6db2b3b2e8a05f3b3/charset_normalizer-3.4.4-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:0d3d8f15c07f86e9ff82319b3d9ef6f4bf907608f53fe9d92b28ea9ae3d1fd89", size = 149383, upload-time = "2025-10-14T04:40:46.018Z" },
- { url = "https://files.pythonhosted.org/packages/4f/e8/b289173b4edae05c0dde07f69f8db476a0b511eac556dfe0d6bda3c43384/charset_normalizer-3.4.4-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:9f7fcd74d410a36883701fafa2482a6af2ff5ba96b9a620e9e0721e28ead5569", size = 159098, upload-time = "2025-10-14T04:40:47.081Z" },
- { url = "https://files.pythonhosted.org/packages/d8/df/fe699727754cae3f8478493c7f45f777b17c3ef0600e28abfec8619eb49c/charset_normalizer-3.4.4-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:ebf3e58c7ec8a8bed6d66a75d7fb37b55e5015b03ceae72a8e7c74495551e224", size = 152991, upload-time = "2025-10-14T04:40:48.246Z" },
- { url = "https://files.pythonhosted.org/packages/1a/86/584869fe4ddb6ffa3bd9f491b87a01568797fb9bd8933f557dba9771beaf/charset_normalizer-3.4.4-cp311-cp311-win32.whl", hash = "sha256:eecbc200c7fd5ddb9a7f16c7decb07b566c29fa2161a16cf67b8d068bd21690a", size = 99456, upload-time = "2025-10-14T04:40:49.376Z" },
- { url = "https://files.pythonhosted.org/packages/65/f6/62fdd5feb60530f50f7e38b4f6a1d5203f4d16ff4f9f0952962c044e919a/charset_normalizer-3.4.4-cp311-cp311-win_amd64.whl", hash = "sha256:5ae497466c7901d54b639cf42d5b8c1b6a4fead55215500d2f486d34db48d016", size = 106978, upload-time = "2025-10-14T04:40:50.844Z" },
- { url = "https://files.pythonhosted.org/packages/7a/9d/0710916e6c82948b3be62d9d398cb4fcf4e97b56d6a6aeccd66c4b2f2bd5/charset_normalizer-3.4.4-cp311-cp311-win_arm64.whl", hash = "sha256:65e2befcd84bc6f37095f5961e68a6f077bf44946771354a28ad434c2cce0ae1", size = 99969, upload-time = "2025-10-14T04:40:52.272Z" },
- { url = "https://files.pythonhosted.org/packages/f3/85/1637cd4af66fa687396e757dec650f28025f2a2f5a5531a3208dc0ec43f2/charset_normalizer-3.4.4-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:0a98e6759f854bd25a58a73fa88833fba3b7c491169f86ce1180c948ab3fd394", size = 208425, upload-time = "2025-10-14T04:40:53.353Z" },
- { url = "https://files.pythonhosted.org/packages/9d/6a/04130023fef2a0d9c62d0bae2649b69f7b7d8d24ea5536feef50551029df/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b5b290ccc2a263e8d185130284f8501e3e36c5e02750fc6b6bdeb2e9e96f1e25", size = 148162, upload-time = "2025-10-14T04:40:54.558Z" },
- { url = "https://files.pythonhosted.org/packages/78/29/62328d79aa60da22c9e0b9a66539feae06ca0f5a4171ac4f7dc285b83688/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:74bb723680f9f7a6234dcf67aea57e708ec1fbdf5699fb91dfd6f511b0a320ef", size = 144558, upload-time = "2025-10-14T04:40:55.677Z" },
- { url = "https://files.pythonhosted.org/packages/86/bb/b32194a4bf15b88403537c2e120b817c61cd4ecffa9b6876e941c3ee38fe/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f1e34719c6ed0b92f418c7c780480b26b5d9c50349e9a9af7d76bf757530350d", size = 161497, upload-time = "2025-10-14T04:40:57.217Z" },
- { url = "https://files.pythonhosted.org/packages/19/89/a54c82b253d5b9b111dc74aca196ba5ccfcca8242d0fb64146d4d3183ff1/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2437418e20515acec67d86e12bf70056a33abdacb5cb1655042f6538d6b085a8", size = 159240, upload-time = "2025-10-14T04:40:58.358Z" },
- { url = "https://files.pythonhosted.org/packages/c0/10/d20b513afe03acc89ec33948320a5544d31f21b05368436d580dec4e234d/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:11d694519d7f29d6cd09f6ac70028dba10f92f6cdd059096db198c283794ac86", size = 153471, upload-time = "2025-10-14T04:40:59.468Z" },
- { url = "https://files.pythonhosted.org/packages/61/fa/fbf177b55bdd727010f9c0a3c49eefa1d10f960e5f09d1d887bf93c2e698/charset_normalizer-3.4.4-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:ac1c4a689edcc530fc9d9aa11f5774b9e2f33f9a0c6a57864e90908f5208d30a", size = 150864, upload-time = "2025-10-14T04:41:00.623Z" },
- { url = "https://files.pythonhosted.org/packages/05/12/9fbc6a4d39c0198adeebbde20b619790e9236557ca59fc40e0e3cebe6f40/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:21d142cc6c0ec30d2efee5068ca36c128a30b0f2c53c1c07bd78cb6bc1d3be5f", size = 150647, upload-time = "2025-10-14T04:41:01.754Z" },
- { url = "https://files.pythonhosted.org/packages/ad/1f/6a9a593d52e3e8c5d2b167daf8c6b968808efb57ef4c210acb907c365bc4/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:5dbe56a36425d26d6cfb40ce79c314a2e4dd6211d51d6d2191c00bed34f354cc", size = 145110, upload-time = "2025-10-14T04:41:03.231Z" },
- { url = "https://files.pythonhosted.org/packages/30/42/9a52c609e72471b0fc54386dc63c3781a387bb4fe61c20231a4ebcd58bdd/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:5bfbb1b9acf3334612667b61bd3002196fe2a1eb4dd74d247e0f2a4d50ec9bbf", size = 162839, upload-time = "2025-10-14T04:41:04.715Z" },
- { url = "https://files.pythonhosted.org/packages/c4/5b/c0682bbf9f11597073052628ddd38344a3d673fda35a36773f7d19344b23/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:d055ec1e26e441f6187acf818b73564e6e6282709e9bcb5b63f5b23068356a15", size = 150667, upload-time = "2025-10-14T04:41:05.827Z" },
- { url = "https://files.pythonhosted.org/packages/e4/24/a41afeab6f990cf2daf6cb8c67419b63b48cf518e4f56022230840c9bfb2/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:af2d8c67d8e573d6de5bc30cdb27e9b95e49115cd9baad5ddbd1a6207aaa82a9", size = 160535, upload-time = "2025-10-14T04:41:06.938Z" },
- { url = "https://files.pythonhosted.org/packages/2a/e5/6a4ce77ed243c4a50a1fecca6aaaab419628c818a49434be428fe24c9957/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:780236ac706e66881f3b7f2f32dfe90507a09e67d1d454c762cf642e6e1586e0", size = 154816, upload-time = "2025-10-14T04:41:08.101Z" },
- { url = "https://files.pythonhosted.org/packages/a8/ef/89297262b8092b312d29cdb2517cb1237e51db8ecef2e9af5edbe7b683b1/charset_normalizer-3.4.4-cp312-cp312-win32.whl", hash = "sha256:5833d2c39d8896e4e19b689ffc198f08ea58116bee26dea51e362ecc7cd3ed26", size = 99694, upload-time = "2025-10-14T04:41:09.23Z" },
- { url = "https://files.pythonhosted.org/packages/3d/2d/1e5ed9dd3b3803994c155cd9aacb60c82c331bad84daf75bcb9c91b3295e/charset_normalizer-3.4.4-cp312-cp312-win_amd64.whl", hash = "sha256:a79cfe37875f822425b89a82333404539ae63dbdddf97f84dcbc3d339aae9525", size = 107131, upload-time = "2025-10-14T04:41:10.467Z" },
- { url = "https://files.pythonhosted.org/packages/d0/d9/0ed4c7098a861482a7b6a95603edce4c0d9db2311af23da1fb2b75ec26fc/charset_normalizer-3.4.4-cp312-cp312-win_arm64.whl", hash = "sha256:376bec83a63b8021bb5c8ea75e21c4ccb86e7e45ca4eb81146091b56599b80c3", size = 100390, upload-time = "2025-10-14T04:41:11.915Z" },
- { url = "https://files.pythonhosted.org/packages/97/45/4b3a1239bbacd321068ea6e7ac28875b03ab8bc0aa0966452db17cd36714/charset_normalizer-3.4.4-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:e1f185f86a6f3403aa2420e815904c67b2f9ebc443f045edd0de921108345794", size = 208091, upload-time = "2025-10-14T04:41:13.346Z" },
- { url = "https://files.pythonhosted.org/packages/7d/62/73a6d7450829655a35bb88a88fca7d736f9882a27eacdca2c6d505b57e2e/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6b39f987ae8ccdf0d2642338faf2abb1862340facc796048b604ef14919e55ed", size = 147936, upload-time = "2025-10-14T04:41:14.461Z" },
- { url = "https://files.pythonhosted.org/packages/89/c5/adb8c8b3d6625bef6d88b251bbb0d95f8205831b987631ab0c8bb5d937c2/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3162d5d8ce1bb98dd51af660f2121c55d0fa541b46dff7bb9b9f86ea1d87de72", size = 144180, upload-time = "2025-10-14T04:41:15.588Z" },
- { url = "https://files.pythonhosted.org/packages/91/ed/9706e4070682d1cc219050b6048bfd293ccf67b3d4f5a4f39207453d4b99/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:81d5eb2a312700f4ecaa977a8235b634ce853200e828fbadf3a9c50bab278328", size = 161346, upload-time = "2025-10-14T04:41:16.738Z" },
- { url = "https://files.pythonhosted.org/packages/d5/0d/031f0d95e4972901a2f6f09ef055751805ff541511dc1252ba3ca1f80cf5/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5bd2293095d766545ec1a8f612559f6b40abc0eb18bb2f5d1171872d34036ede", size = 158874, upload-time = "2025-10-14T04:41:17.923Z" },
- { url = "https://files.pythonhosted.org/packages/f5/83/6ab5883f57c9c801ce5e5677242328aa45592be8a00644310a008d04f922/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a8a8b89589086a25749f471e6a900d3f662d1d3b6e2e59dcecf787b1cc3a1894", size = 153076, upload-time = "2025-10-14T04:41:19.106Z" },
- { url = "https://files.pythonhosted.org/packages/75/1e/5ff781ddf5260e387d6419959ee89ef13878229732732ee73cdae01800f2/charset_normalizer-3.4.4-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:bc7637e2f80d8530ee4a78e878bce464f70087ce73cf7c1caf142416923b98f1", size = 150601, upload-time = "2025-10-14T04:41:20.245Z" },
- { url = "https://files.pythonhosted.org/packages/d7/57/71be810965493d3510a6ca79b90c19e48696fb1ff964da319334b12677f0/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f8bf04158c6b607d747e93949aa60618b61312fe647a6369f88ce2ff16043490", size = 150376, upload-time = "2025-10-14T04:41:21.398Z" },
- { url = "https://files.pythonhosted.org/packages/e5/d5/c3d057a78c181d007014feb7e9f2e65905a6c4ef182c0ddf0de2924edd65/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:554af85e960429cf30784dd47447d5125aaa3b99a6f0683589dbd27e2f45da44", size = 144825, upload-time = "2025-10-14T04:41:22.583Z" },
- { url = "https://files.pythonhosted.org/packages/e6/8c/d0406294828d4976f275ffbe66f00266c4b3136b7506941d87c00cab5272/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:74018750915ee7ad843a774364e13a3db91682f26142baddf775342c3f5b1133", size = 162583, upload-time = "2025-10-14T04:41:23.754Z" },
- { url = "https://files.pythonhosted.org/packages/d7/24/e2aa1f18c8f15c4c0e932d9287b8609dd30ad56dbe41d926bd846e22fb8d/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:c0463276121fdee9c49b98908b3a89c39be45d86d1dbaa22957e38f6321d4ce3", size = 150366, upload-time = "2025-10-14T04:41:25.27Z" },
- { url = "https://files.pythonhosted.org/packages/e4/5b/1e6160c7739aad1e2df054300cc618b06bf784a7a164b0f238360721ab86/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:362d61fd13843997c1c446760ef36f240cf81d3ebf74ac62652aebaf7838561e", size = 160300, upload-time = "2025-10-14T04:41:26.725Z" },
- { url = "https://files.pythonhosted.org/packages/7a/10/f882167cd207fbdd743e55534d5d9620e095089d176d55cb22d5322f2afd/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9a26f18905b8dd5d685d6d07b0cdf98a79f3c7a918906af7cc143ea2e164c8bc", size = 154465, upload-time = "2025-10-14T04:41:28.322Z" },
- { url = "https://files.pythonhosted.org/packages/89/66/c7a9e1b7429be72123441bfdbaf2bc13faab3f90b933f664db506dea5915/charset_normalizer-3.4.4-cp313-cp313-win32.whl", hash = "sha256:9b35f4c90079ff2e2edc5b26c0c77925e5d2d255c42c74fdb70fb49b172726ac", size = 99404, upload-time = "2025-10-14T04:41:29.95Z" },
- { url = "https://files.pythonhosted.org/packages/c4/26/b9924fa27db384bdcd97ab83b4f0a8058d96ad9626ead570674d5e737d90/charset_normalizer-3.4.4-cp313-cp313-win_amd64.whl", hash = "sha256:b435cba5f4f750aa6c0a0d92c541fb79f69a387c91e61f1795227e4ed9cece14", size = 107092, upload-time = "2025-10-14T04:41:31.188Z" },
- { url = "https://files.pythonhosted.org/packages/af/8f/3ed4bfa0c0c72a7ca17f0380cd9e4dd842b09f664e780c13cff1dcf2ef1b/charset_normalizer-3.4.4-cp313-cp313-win_arm64.whl", hash = "sha256:542d2cee80be6f80247095cc36c418f7bddd14f4a6de45af91dfad36d817bba2", size = 100408, upload-time = "2025-10-14T04:41:32.624Z" },
- { url = "https://files.pythonhosted.org/packages/2a/35/7051599bd493e62411d6ede36fd5af83a38f37c4767b92884df7301db25d/charset_normalizer-3.4.4-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:da3326d9e65ef63a817ecbcc0df6e94463713b754fe293eaa03da99befb9a5bd", size = 207746, upload-time = "2025-10-14T04:41:33.773Z" },
- { url = "https://files.pythonhosted.org/packages/10/9a/97c8d48ef10d6cd4fcead2415523221624bf58bcf68a802721a6bc807c8f/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8af65f14dc14a79b924524b1e7fffe304517b2bff5a58bf64f30b98bbc5079eb", size = 147889, upload-time = "2025-10-14T04:41:34.897Z" },
- { url = "https://files.pythonhosted.org/packages/10/bf/979224a919a1b606c82bd2c5fa49b5c6d5727aa47b4312bb27b1734f53cd/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:74664978bb272435107de04e36db5a9735e78232b85b77d45cfb38f758efd33e", size = 143641, upload-time = "2025-10-14T04:41:36.116Z" },
- { url = "https://files.pythonhosted.org/packages/ba/33/0ad65587441fc730dc7bd90e9716b30b4702dc7b617e6ba4997dc8651495/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:752944c7ffbfdd10c074dc58ec2d5a8a4cd9493b314d367c14d24c17684ddd14", size = 160779, upload-time = "2025-10-14T04:41:37.229Z" },
- { url = "https://files.pythonhosted.org/packages/67/ed/331d6b249259ee71ddea93f6f2f0a56cfebd46938bde6fcc6f7b9a3d0e09/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d1f13550535ad8cff21b8d757a3257963e951d96e20ec82ab44bc64aeb62a191", size = 159035, upload-time = "2025-10-14T04:41:38.368Z" },
- { url = "https://files.pythonhosted.org/packages/67/ff/f6b948ca32e4f2a4576aa129d8bed61f2e0543bf9f5f2b7fc3758ed005c9/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ecaae4149d99b1c9e7b88bb03e3221956f68fd6d50be2ef061b2381b61d20838", size = 152542, upload-time = "2025-10-14T04:41:39.862Z" },
- { url = "https://files.pythonhosted.org/packages/16/85/276033dcbcc369eb176594de22728541a925b2632f9716428c851b149e83/charset_normalizer-3.4.4-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:cb6254dc36b47a990e59e1068afacdcd02958bdcce30bb50cc1700a8b9d624a6", size = 149524, upload-time = "2025-10-14T04:41:41.319Z" },
- { url = "https://files.pythonhosted.org/packages/9e/f2/6a2a1f722b6aba37050e626530a46a68f74e63683947a8acff92569f979a/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c8ae8a0f02f57a6e61203a31428fa1d677cbe50c93622b4149d5c0f319c1d19e", size = 150395, upload-time = "2025-10-14T04:41:42.539Z" },
- { url = "https://files.pythonhosted.org/packages/60/bb/2186cb2f2bbaea6338cad15ce23a67f9b0672929744381e28b0592676824/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:47cc91b2f4dd2833fddaedd2893006b0106129d4b94fdb6af1f4ce5a9965577c", size = 143680, upload-time = "2025-10-14T04:41:43.661Z" },
- { url = "https://files.pythonhosted.org/packages/7d/a5/bf6f13b772fbb2a90360eb620d52ed8f796f3c5caee8398c3b2eb7b1c60d/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:82004af6c302b5d3ab2cfc4cc5f29db16123b1a8417f2e25f9066f91d4411090", size = 162045, upload-time = "2025-10-14T04:41:44.821Z" },
- { url = "https://files.pythonhosted.org/packages/df/c5/d1be898bf0dc3ef9030c3825e5d3b83f2c528d207d246cbabe245966808d/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:2b7d8f6c26245217bd2ad053761201e9f9680f8ce52f0fcd8d0755aeae5b2152", size = 149687, upload-time = "2025-10-14T04:41:46.442Z" },
- { url = "https://files.pythonhosted.org/packages/a5/42/90c1f7b9341eef50c8a1cb3f098ac43b0508413f33affd762855f67a410e/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:799a7a5e4fb2d5898c60b640fd4981d6a25f1c11790935a44ce38c54e985f828", size = 160014, upload-time = "2025-10-14T04:41:47.631Z" },
- { url = "https://files.pythonhosted.org/packages/76/be/4d3ee471e8145d12795ab655ece37baed0929462a86e72372fd25859047c/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:99ae2cffebb06e6c22bdc25801d7b30f503cc87dbd283479e7b606f70aff57ec", size = 154044, upload-time = "2025-10-14T04:41:48.81Z" },
- { url = "https://files.pythonhosted.org/packages/b0/6f/8f7af07237c34a1defe7defc565a9bc1807762f672c0fde711a4b22bf9c0/charset_normalizer-3.4.4-cp314-cp314-win32.whl", hash = "sha256:f9d332f8c2a2fcbffe1378594431458ddbef721c1769d78e2cbc06280d8155f9", size = 99940, upload-time = "2025-10-14T04:41:49.946Z" },
- { url = "https://files.pythonhosted.org/packages/4b/51/8ade005e5ca5b0d80fb4aff72a3775b325bdc3d27408c8113811a7cbe640/charset_normalizer-3.4.4-cp314-cp314-win_amd64.whl", hash = "sha256:8a6562c3700cce886c5be75ade4a5db4214fda19fede41d9792d100288d8f94c", size = 107104, upload-time = "2025-10-14T04:41:51.051Z" },
- { url = "https://files.pythonhosted.org/packages/da/5f/6b8f83a55bb8278772c5ae54a577f3099025f9ade59d0136ac24a0df4bde/charset_normalizer-3.4.4-cp314-cp314-win_arm64.whl", hash = "sha256:de00632ca48df9daf77a2c65a484531649261ec9f25489917f09e455cb09ddb2", size = 100743, upload-time = "2025-10-14T04:41:52.122Z" },
- { url = "https://files.pythonhosted.org/packages/0a/4c/925909008ed5a988ccbb72dcc897407e5d6d3bd72410d69e051fc0c14647/charset_normalizer-3.4.4-py3-none-any.whl", hash = "sha256:7a32c560861a02ff789ad905a2fe94e3f840803362c84fecf1851cb4cf3dc37f", size = 53402, upload-time = "2025-10-14T04:42:31.76Z" },
+version = "3.4.6"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/7b/60/e3bec1881450851b087e301bedc3daa9377a4d45f1c26aa90b0b235e38aa/charset_normalizer-3.4.6.tar.gz", hash = "sha256:1ae6b62897110aa7c79ea2f5dd38d1abca6db663687c0b1ad9aed6f6bae3d9d6", size = 143363, upload-time = "2026-03-15T18:53:25.478Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e5/62/c0815c992c9545347aeea7859b50dc9044d147e2e7278329c6e02ac9a616/charset_normalizer-3.4.6-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:2ef7fedc7a6ecbe99969cd09632516738a97eeb8bd7258bf8a0f23114c057dab", size = 295154, upload-time = "2026-03-15T18:50:50.88Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/37/bdca6613c2e3c58c7421891d80cc3efa1d32e882f7c4a7ee6039c3fc951a/charset_normalizer-3.4.6-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a4ea868bc28109052790eb2b52a9ab33f3aa7adc02f96673526ff47419490e21", size = 199191, upload-time = "2026-03-15T18:50:52.658Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/92/9934d1bbd69f7f398b38c5dae1cbf9cc672e7c34a4adf7b17c0a9c17d15d/charset_normalizer-3.4.6-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:836ab36280f21fc1a03c99cd05c6b7af70d2697e374c7af0b61ed271401a72a2", size = 218674, upload-time = "2026-03-15T18:50:54.102Z" },
+ { url = "https://files.pythonhosted.org/packages/af/90/25f6ab406659286be929fd89ab0e78e38aa183fc374e03aa3c12d730af8a/charset_normalizer-3.4.6-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f1ce721c8a7dfec21fcbdfe04e8f68174183cf4e8188e0645e92aa23985c57ff", size = 215259, upload-time = "2026-03-15T18:50:55.616Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/ef/79a463eb0fff7f96afa04c1d4c51f8fc85426f918db467854bfb6a569ce3/charset_normalizer-3.4.6-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0e28d62a8fc7a1fa411c43bd65e346f3bce9716dc51b897fbe930c5987b402d5", size = 207276, upload-time = "2026-03-15T18:50:57.054Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/72/d0426afec4b71dc159fa6b4e68f868cd5a3ecd918fec5813a15d292a7d10/charset_normalizer-3.4.6-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:530d548084c4a9f7a16ed4a294d459b4f229db50df689bfe92027452452943a0", size = 195161, upload-time = "2026-03-15T18:50:58.686Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/18/c82b06a68bfcb6ce55e508225d210c7e6a4ea122bfc0748892f3dc4e8e11/charset_normalizer-3.4.6-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:30f445ae60aad5e1f8bdbb3108e39f6fbc09f4ea16c815c66578878325f8f15a", size = 203452, upload-time = "2026-03-15T18:51:00.196Z" },
+ { url = "https://files.pythonhosted.org/packages/44/d6/0c25979b92f8adafdbb946160348d8d44aa60ce99afdc27df524379875cb/charset_normalizer-3.4.6-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ac2393c73378fea4e52aa56285a3d64be50f1a12395afef9cce47772f60334c2", size = 202272, upload-time = "2026-03-15T18:51:01.703Z" },
+ { url = "https://files.pythonhosted.org/packages/2e/3d/7fea3e8fe84136bebbac715dd1221cc25c173c57a699c030ab9b8900cbb7/charset_normalizer-3.4.6-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:90ca27cd8da8118b18a52d5f547859cc1f8354a00cd1e8e5120df3e30d6279e5", size = 195622, upload-time = "2026-03-15T18:51:03.526Z" },
+ { url = "https://files.pythonhosted.org/packages/57/8a/d6f7fd5cb96c58ef2f681424fbca01264461336d2a7fc875e4446b1f1346/charset_normalizer-3.4.6-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:8e5a94886bedca0f9b78fecd6afb6629142fd2605aa70a125d49f4edc6037ee6", size = 220056, upload-time = "2026-03-15T18:51:05.269Z" },
+ { url = "https://files.pythonhosted.org/packages/16/50/478cdda782c8c9c3fb5da3cc72dd7f331f031e7f1363a893cdd6ca0f8de0/charset_normalizer-3.4.6-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:695f5c2823691a25f17bc5d5ffe79fa90972cc34b002ac6c843bb8a1720e950d", size = 203751, upload-time = "2026-03-15T18:51:06.858Z" },
+ { url = "https://files.pythonhosted.org/packages/75/fc/cc2fcac943939c8e4d8791abfa139f685e5150cae9f94b60f12520feaa9b/charset_normalizer-3.4.6-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:231d4da14bcd9301310faf492051bee27df11f2bc7549bc0bb41fef11b82daa2", size = 216563, upload-time = "2026-03-15T18:51:08.564Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/b7/a4add1d9a5f68f3d037261aecca83abdb0ab15960a3591d340e829b37298/charset_normalizer-3.4.6-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:a056d1ad2633548ca18ffa2f85c202cfb48b68615129143915b8dc72a806a923", size = 209265, upload-time = "2026-03-15T18:51:10.312Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/18/c094561b5d64a24277707698e54b7f67bd17a4f857bbfbb1072bba07c8bf/charset_normalizer-3.4.6-cp312-cp312-win32.whl", hash = "sha256:c2274ca724536f173122f36c98ce188fd24ce3dad886ec2b7af859518ce008a4", size = 144229, upload-time = "2026-03-15T18:51:11.694Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/20/0567efb3a8fd481b8f34f739ebddc098ed062a59fed41a8d193a61939e8f/charset_normalizer-3.4.6-cp312-cp312-win_amd64.whl", hash = "sha256:c8ae56368f8cc97c7e40a7ee18e1cedaf8e780cd8bc5ed5ac8b81f238614facb", size = 154277, upload-time = "2026-03-15T18:51:13.004Z" },
+ { url = "https://files.pythonhosted.org/packages/15/57/28d79b44b51933119e21f65479d0864a8d5893e494cf5daab15df0247c17/charset_normalizer-3.4.6-cp312-cp312-win_arm64.whl", hash = "sha256:899d28f422116b08be5118ef350c292b36fc15ec2daeb9ea987c89281c7bb5c4", size = 142817, upload-time = "2026-03-15T18:51:14.408Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/1d/4fdabeef4e231153b6ed7567602f3b68265ec4e5b76d6024cf647d43d981/charset_normalizer-3.4.6-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:11afb56037cbc4b1555a34dd69151e8e069bee82e613a73bef6e714ce733585f", size = 294823, upload-time = "2026-03-15T18:51:15.755Z" },
+ { url = "https://files.pythonhosted.org/packages/47/7b/20e809b89c69d37be748d98e84dce6820bf663cf19cf6b942c951a3e8f41/charset_normalizer-3.4.6-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:423fb7e748a08f854a08a222b983f4df1912b1daedce51a72bd24fe8f26a1843", size = 198527, upload-time = "2026-03-15T18:51:17.177Z" },
+ { url = "https://files.pythonhosted.org/packages/37/a6/4f8d27527d59c039dce6f7622593cdcd3d70a8504d87d09eb11e9fdc6062/charset_normalizer-3.4.6-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d73beaac5e90173ac3deb9928a74763a6d230f494e4bfb422c217a0ad8e629bf", size = 218388, upload-time = "2026-03-15T18:51:18.934Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/9b/4770ccb3e491a9bacf1c46cc8b812214fe367c86a96353ccc6daf87b01ec/charset_normalizer-3.4.6-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d60377dce4511655582e300dc1e5a5f24ba0cb229005a1d5c8d0cb72bb758ab8", size = 214563, upload-time = "2026-03-15T18:51:20.374Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/58/a199d245894b12db0b957d627516c78e055adc3a0d978bc7f65ddaf7c399/charset_normalizer-3.4.6-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:530e8cebeea0d76bdcf93357aa5e41336f48c3dc709ac52da2bb167c5b8271d9", size = 206587, upload-time = "2026-03-15T18:51:21.807Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/70/3def227f1ec56f5c69dfc8392b8bd63b11a18ca8178d9211d7cc5e5e4f27/charset_normalizer-3.4.6-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:a26611d9987b230566f24a0a125f17fe0de6a6aff9f25c9f564aaa2721a5fb88", size = 194724, upload-time = "2026-03-15T18:51:23.508Z" },
+ { url = "https://files.pythonhosted.org/packages/58/ab/9318352e220c05efd31c2779a23b50969dc94b985a2efa643ed9077bfca5/charset_normalizer-3.4.6-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:34315ff4fc374b285ad7f4a0bf7dcbfe769e1b104230d40f49f700d4ab6bbd84", size = 202956, upload-time = "2026-03-15T18:51:25.239Z" },
+ { url = "https://files.pythonhosted.org/packages/75/13/f3550a3ac25b70f87ac98c40d3199a8503676c2f1620efbf8d42095cfc40/charset_normalizer-3.4.6-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5f8ddd609f9e1af8c7bd6e2aca279c931aefecd148a14402d4e368f3171769fd", size = 201923, upload-time = "2026-03-15T18:51:26.682Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/db/c5c643b912740b45e8eec21de1bbab8e7fc085944d37e1e709d3dcd9d72f/charset_normalizer-3.4.6-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:80d0a5615143c0b3225e5e3ef22c8d5d51f3f72ce0ea6fb84c943546c7b25b6c", size = 195366, upload-time = "2026-03-15T18:51:28.129Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/67/3b1c62744f9b2448443e0eb160d8b001c849ec3fef591e012eda6484787c/charset_normalizer-3.4.6-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:92734d4d8d187a354a556626c221cd1a892a4e0802ccb2af432a1d85ec012194", size = 219752, upload-time = "2026-03-15T18:51:29.556Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/98/32ffbaf7f0366ffb0445930b87d103f6b406bc2c271563644bde8a2b1093/charset_normalizer-3.4.6-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:613f19aa6e082cf96e17e3ffd89383343d0d589abda756b7764cf78361fd41dc", size = 203296, upload-time = "2026-03-15T18:51:30.921Z" },
+ { url = "https://files.pythonhosted.org/packages/41/12/5d308c1bbe60cabb0c5ef511574a647067e2a1f631bc8634fcafaccd8293/charset_normalizer-3.4.6-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:2b1a63e8224e401cafe7739f77efd3f9e7f5f2026bda4aead8e59afab537784f", size = 215956, upload-time = "2026-03-15T18:51:32.399Z" },
+ { url = "https://files.pythonhosted.org/packages/53/e9/5f85f6c5e20669dbe56b165c67b0260547dea97dba7e187938833d791687/charset_normalizer-3.4.6-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6cceb5473417d28edd20c6c984ab6fee6c6267d38d906823ebfe20b03d607dc2", size = 208652, upload-time = "2026-03-15T18:51:34.214Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/11/897052ea6af56df3eef3ca94edafee410ca699ca0c7b87960ad19932c55e/charset_normalizer-3.4.6-cp313-cp313-win32.whl", hash = "sha256:d7de2637729c67d67cf87614b566626057e95c303bc0a55ffe391f5205e7003d", size = 143940, upload-time = "2026-03-15T18:51:36.15Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/5c/724b6b363603e419829f561c854b87ed7c7e31231a7908708ac086cdf3e2/charset_normalizer-3.4.6-cp313-cp313-win_amd64.whl", hash = "sha256:572d7c822caf521f0525ba1bce1a622a0b85cf47ffbdae6c9c19e3b5ac3c4389", size = 154101, upload-time = "2026-03-15T18:51:37.876Z" },
+ { url = "https://files.pythonhosted.org/packages/01/a5/7abf15b4c0968e47020f9ca0935fb3274deb87cb288cd187cad92e8cdffd/charset_normalizer-3.4.6-cp313-cp313-win_arm64.whl", hash = "sha256:a4474d924a47185a06411e0064b803c68be044be2d60e50e8bddcc2649957c1f", size = 143109, upload-time = "2026-03-15T18:51:39.565Z" },
+ { url = "https://files.pythonhosted.org/packages/25/6f/ffe1e1259f384594063ea1869bfb6be5cdb8bc81020fc36c3636bc8302a1/charset_normalizer-3.4.6-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:9cc6e6d9e571d2f863fa77700701dae73ed5f78881efc8b3f9a4398772ff53e8", size = 294458, upload-time = "2026-03-15T18:51:41.134Z" },
+ { url = "https://files.pythonhosted.org/packages/56/60/09bb6c13a8c1016c2ed5c6a6488e4ffef506461aa5161662bd7636936fb1/charset_normalizer-3.4.6-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ef5960d965e67165d75b7c7ffc60a83ec5abfc5c11b764ec13ea54fbef8b4421", size = 199277, upload-time = "2026-03-15T18:51:42.953Z" },
+ { url = "https://files.pythonhosted.org/packages/00/50/dcfbb72a5138bbefdc3332e8d81a23494bf67998b4b100703fd15fa52d81/charset_normalizer-3.4.6-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b3694e3f87f8ac7ce279d4355645b3c878d24d1424581b46282f24b92f5a4ae2", size = 218758, upload-time = "2026-03-15T18:51:44.339Z" },
+ { url = "https://files.pythonhosted.org/packages/03/b3/d79a9a191bb75f5aa81f3aaaa387ef29ce7cb7a9e5074ba8ea095cc073c2/charset_normalizer-3.4.6-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5d11595abf8dd942a77883a39d81433739b287b6aa71620f15164f8096221b30", size = 215299, upload-time = "2026-03-15T18:51:45.871Z" },
+ { url = "https://files.pythonhosted.org/packages/76/7e/bc8911719f7084f72fd545f647601ea3532363927f807d296a8c88a62c0d/charset_normalizer-3.4.6-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7bda6eebafd42133efdca535b04ccb338ab29467b3f7bf79569883676fc628db", size = 206811, upload-time = "2026-03-15T18:51:47.308Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/40/c430b969d41dda0c465aa36cc7c2c068afb67177bef50905ac371b28ccc7/charset_normalizer-3.4.6-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:bbc8c8650c6e51041ad1be191742b8b421d05bbd3410f43fa2a00c8db87678e8", size = 193706, upload-time = "2026-03-15T18:51:48.849Z" },
+ { url = "https://files.pythonhosted.org/packages/48/15/e35e0590af254f7df984de1323640ef375df5761f615b6225ba8deb9799a/charset_normalizer-3.4.6-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:22c6f0c2fbc31e76c3b8a86fba1a56eda6166e238c29cdd3d14befdb4a4e4815", size = 202706, upload-time = "2026-03-15T18:51:50.257Z" },
+ { url = "https://files.pythonhosted.org/packages/5e/bd/f736f7b9cc5e93a18b794a50346bb16fbfd6b37f99e8f306f7951d27c17c/charset_normalizer-3.4.6-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7edbed096e4a4798710ed6bc75dcaa2a21b68b6c356553ac4823c3658d53743a", size = 202497, upload-time = "2026-03-15T18:51:52.012Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/ba/2cc9e3e7dfdf7760a6ed8da7446d22536f3d0ce114ac63dee2a5a3599e62/charset_normalizer-3.4.6-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:7f9019c9cb613f084481bd6a100b12e1547cf2efe362d873c2e31e4035a6fa43", size = 193511, upload-time = "2026-03-15T18:51:53.723Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/cb/5be49b5f776e5613be07298c80e1b02a2d900f7a7de807230595c85a8b2e/charset_normalizer-3.4.6-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:58c948d0d086229efc484fe2f30c2d382c86720f55cd9bc33591774348ad44e0", size = 220133, upload-time = "2026-03-15T18:51:55.333Z" },
+ { url = "https://files.pythonhosted.org/packages/83/43/99f1b5dad345accb322c80c7821071554f791a95ee50c1c90041c157ae99/charset_normalizer-3.4.6-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:419a9d91bd238052642a51938af8ac05da5b3343becde08d5cdeab9046df9ee1", size = 203035, upload-time = "2026-03-15T18:51:56.736Z" },
+ { url = "https://files.pythonhosted.org/packages/87/9a/62c2cb6a531483b55dddff1a68b3d891a8b498f3ca555fbcf2978e804d9d/charset_normalizer-3.4.6-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:5273b9f0b5835ff0350c0828faea623c68bfa65b792720c453e22b25cc72930f", size = 216321, upload-time = "2026-03-15T18:51:58.17Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/79/94a010ff81e3aec7c293eb82c28f930918e517bc144c9906a060844462eb/charset_normalizer-3.4.6-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:0e901eb1049fdb80f5bd11ed5ea1e498ec423102f7a9b9e4645d5b8204ff2815", size = 208973, upload-time = "2026-03-15T18:51:59.998Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/57/4ecff6d4ec8585342f0c71bc03efaa99cb7468f7c91a57b105bcd561cea8/charset_normalizer-3.4.6-cp314-cp314-win32.whl", hash = "sha256:b4ff1d35e8c5bd078be89349b6f3a845128e685e751b6ea1169cf2160b344c4d", size = 144610, upload-time = "2026-03-15T18:52:02.213Z" },
+ { url = "https://files.pythonhosted.org/packages/80/94/8434a02d9d7f168c25767c64671fead8d599744a05d6a6c877144c754246/charset_normalizer-3.4.6-cp314-cp314-win_amd64.whl", hash = "sha256:74119174722c4349af9708993118581686f343adc1c8c9c007d59be90d077f3f", size = 154962, upload-time = "2026-03-15T18:52:03.658Z" },
+ { url = "https://files.pythonhosted.org/packages/46/4c/48f2cdbfd923026503dfd67ccea45c94fd8fe988d9056b468579c66ed62b/charset_normalizer-3.4.6-cp314-cp314-win_arm64.whl", hash = "sha256:e5bcc1a1ae744e0bb59641171ae53743760130600da8db48cbb6e4918e186e4e", size = 143595, upload-time = "2026-03-15T18:52:05.123Z" },
+ { url = "https://files.pythonhosted.org/packages/31/93/8878be7569f87b14f1d52032946131bcb6ebbd8af3e20446bc04053dc3f1/charset_normalizer-3.4.6-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:ad8faf8df23f0378c6d527d8b0b15ea4a2e23c89376877c598c4870d1b2c7866", size = 314828, upload-time = "2026-03-15T18:52:06.831Z" },
+ { url = "https://files.pythonhosted.org/packages/06/b6/fae511ca98aac69ecc35cde828b0a3d146325dd03d99655ad38fc2cc3293/charset_normalizer-3.4.6-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f5ea69428fa1b49573eef0cc44a1d43bebd45ad0c611eb7d7eac760c7ae771bc", size = 208138, upload-time = "2026-03-15T18:52:08.239Z" },
+ { url = "https://files.pythonhosted.org/packages/54/57/64caf6e1bf07274a1e0b7c160a55ee9e8c9ec32c46846ce59b9c333f7008/charset_normalizer-3.4.6-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:06a7e86163334edfc5d20fe104db92fcd666e5a5df0977cb5680a506fe26cc8e", size = 224679, upload-time = "2026-03-15T18:52:10.043Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/cb/9ff5a25b9273ef160861b41f6937f86fae18b0792fe0a8e75e06acb08f1d/charset_normalizer-3.4.6-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:e1f6e2f00a6b8edb562826e4632e26d063ac10307e80f7461f7de3ad8ef3f077", size = 223475, upload-time = "2026-03-15T18:52:11.854Z" },
+ { url = "https://files.pythonhosted.org/packages/fc/97/440635fc093b8d7347502a377031f9605a1039c958f3cd18dcacffb37743/charset_normalizer-3.4.6-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:95b52c68d64c1878818687a473a10547b3292e82b6f6fe483808fb1468e2f52f", size = 215230, upload-time = "2026-03-15T18:52:13.325Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/24/afff630feb571a13f07c8539fbb502d2ab494019492aaffc78ef41f1d1d0/charset_normalizer-3.4.6-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:7504e9b7dc05f99a9bbb4525c67a2c155073b44d720470a148b34166a69c054e", size = 199045, upload-time = "2026-03-15T18:52:14.752Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/17/d1399ecdaf7e0498c327433e7eefdd862b41236a7e484355b8e0e5ebd64b/charset_normalizer-3.4.6-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:172985e4ff804a7ad08eebec0a1640ece87ba5041d565fff23c8f99c1f389484", size = 211658, upload-time = "2026-03-15T18:52:16.278Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/38/16baa0affb957b3d880e5ac2144caf3f9d7de7bc4a91842e447fbb5e8b67/charset_normalizer-3.4.6-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:4be9f4830ba8741527693848403e2c457c16e499100963ec711b1c6f2049b7c7", size = 210769, upload-time = "2026-03-15T18:52:17.782Z" },
+ { url = "https://files.pythonhosted.org/packages/05/34/c531bc6ac4c21da9ddfddb3107be2287188b3ea4b53b70fc58f2a77ac8d8/charset_normalizer-3.4.6-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:79090741d842f564b1b2827c0b82d846405b744d31e84f18d7a7b41c20e473ff", size = 201328, upload-time = "2026-03-15T18:52:19.553Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/73/a5a1e9ca5f234519c1953608a03fe109c306b97fdfb25f09182babad51a7/charset_normalizer-3.4.6-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:87725cfb1a4f1f8c2fc9890ae2f42094120f4b44db9360be5d99a4c6b0e03a9e", size = 225302, upload-time = "2026-03-15T18:52:21.043Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/f6/cd782923d112d296294dea4bcc7af5a7ae0f86ab79f8fefbda5526b6cfc0/charset_normalizer-3.4.6-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:fcce033e4021347d80ed9c66dcf1e7b1546319834b74445f561d2e2221de5659", size = 211127, upload-time = "2026-03-15T18:52:22.491Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/c5/0b6898950627af7d6103a449b22320372c24c6feda91aa24e201a478d161/charset_normalizer-3.4.6-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:ca0276464d148c72defa8bb4390cce01b4a0e425f3b50d1435aa6d7a18107602", size = 222840, upload-time = "2026-03-15T18:52:24.113Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/25/c4bba773bef442cbdc06111d40daa3de5050a676fa26e85090fc54dd12f0/charset_normalizer-3.4.6-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:197c1a244a274bb016dd8b79204850144ef77fe81c5b797dc389327adb552407", size = 216890, upload-time = "2026-03-15T18:52:25.541Z" },
+ { url = "https://files.pythonhosted.org/packages/35/1a/05dacadb0978da72ee287b0143097db12f2e7e8d3ffc4647da07a383b0b7/charset_normalizer-3.4.6-cp314-cp314t-win32.whl", hash = "sha256:2a24157fa36980478dd1770b585c0f30d19e18f4fb0c47c13aa568f871718579", size = 155379, upload-time = "2026-03-15T18:52:27.05Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/7a/d269d834cb3a76291651256f3b9a5945e81d0a49ab9f4a498964e83c0416/charset_normalizer-3.4.6-cp314-cp314t-win_amd64.whl", hash = "sha256:cd5e2801c89992ed8c0a3f0293ae83c159a60d9a5d685005383ef4caca77f2c4", size = 169043, upload-time = "2026-03-15T18:52:28.502Z" },
+ { url = "https://files.pythonhosted.org/packages/23/06/28b29fba521a37a8932c6a84192175c34d49f84a6d4773fa63d05f9aff22/charset_normalizer-3.4.6-cp314-cp314t-win_arm64.whl", hash = "sha256:47955475ac79cc504ef2704b192364e51d0d473ad452caedd0002605f780101c", size = 148523, upload-time = "2026-03-15T18:52:29.956Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/68/687187c7e26cb24ccbd88e5069f5ef00eba804d36dde11d99aad0838ab45/charset_normalizer-3.4.6-py3-none-any.whl", hash = "sha256:947cf925bc916d90adba35a64c82aace04fa39b46b52d4630ece166655905a69", size = 61455, upload-time = "2026-03-15T18:53:23.833Z" },
]
[[package]]
@@ -894,6 +799,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" },
]
+[[package]]
+name = "colorful"
+version = "0.5.8"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/82/31/109ef4bedeb32b4202e02ddb133162457adc4eb890a9ed9c05c9dd126ed0/colorful-0.5.8.tar.gz", hash = "sha256:bb16502b198be2f1c42ba3c52c703d5f651d826076817185f0294c1a549a7445", size = 209361, upload-time = "2025-10-29T11:53:21.663Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c3/11/25cdf9d5fc21efd30134fc74c43702c6f7ef09ebae8ed927f1283403ad8d/colorful-0.5.8-py2.py3-none-any.whl", hash = "sha256:a9381fdda3337fbaba5771991020abc69676afa102646650b759927892875992", size = 201334, upload-time = "2025-10-29T11:53:20.251Z" },
+]
+
[[package]]
name = "contextlib2"
version = "21.6.0"
@@ -905,106 +822,86 @@ wheels = [
[[package]]
name = "coverage"
-version = "7.13.1"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/23/f9/e92df5e07f3fc8d4c7f9a0f146ef75446bf870351cd37b788cf5897f8079/coverage-7.13.1.tar.gz", hash = "sha256:b7593fe7eb5feaa3fbb461ac79aac9f9fc0387a5ca8080b0c6fe2ca27b091afd", size = 825862, upload-time = "2025-12-28T15:42:56.969Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/2d/9a/3742e58fd04b233df95c012ee9f3dfe04708a5e1d32613bd2d47d4e1be0d/coverage-7.13.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e1fa280b3ad78eea5be86f94f461c04943d942697e0dac889fa18fff8f5f9147", size = 218633, upload-time = "2025-12-28T15:40:10.165Z" },
- { url = "https://files.pythonhosted.org/packages/7e/45/7e6bdc94d89cd7c8017ce735cf50478ddfe765d4fbf0c24d71d30ea33d7a/coverage-7.13.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:c3d8c679607220979434f494b139dfb00131ebf70bb406553d69c1ff01a5c33d", size = 219147, upload-time = "2025-12-28T15:40:12.069Z" },
- { url = "https://files.pythonhosted.org/packages/f7/38/0d6a258625fd7f10773fe94097dc16937a5f0e3e0cdf3adef67d3ac6baef/coverage-7.13.1-cp310-cp310-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:339dc63b3eba969067b00f41f15ad161bf2946613156fb131266d8debc8e44d0", size = 245894, upload-time = "2025-12-28T15:40:13.556Z" },
- { url = "https://files.pythonhosted.org/packages/27/58/409d15ea487986994cbd4d06376e9860e9b157cfbfd402b1236770ab8dd2/coverage-7.13.1-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:db622b999ffe49cb891f2fff3b340cdc2f9797d01a0a202a0973ba2562501d90", size = 247721, upload-time = "2025-12-28T15:40:15.37Z" },
- { url = "https://files.pythonhosted.org/packages/da/bf/6e8056a83fd7a96c93341f1ffe10df636dd89f26d5e7b9ca511ce3bcf0df/coverage-7.13.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d1443ba9acbb593fa7c1c29e011d7c9761545fe35e7652e85ce7f51a16f7e08d", size = 249585, upload-time = "2025-12-28T15:40:17.226Z" },
- { url = "https://files.pythonhosted.org/packages/f4/15/e1daff723f9f5959acb63cbe35b11203a9df77ee4b95b45fffd38b318390/coverage-7.13.1-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c832ec92c4499ac463186af72f9ed4d8daec15499b16f0a879b0d1c8e5cf4a3b", size = 246597, upload-time = "2025-12-28T15:40:19.028Z" },
- { url = "https://files.pythonhosted.org/packages/74/a6/1efd31c5433743a6ddbc9d37ac30c196bb07c7eab3d74fbb99b924c93174/coverage-7.13.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:562ec27dfa3f311e0db1ba243ec6e5f6ab96b1edfcfc6cf86f28038bc4961ce6", size = 247626, upload-time = "2025-12-28T15:40:20.846Z" },
- { url = "https://files.pythonhosted.org/packages/6d/9f/1609267dd3e749f57fdd66ca6752567d1c13b58a20a809dc409b263d0b5f/coverage-7.13.1-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:4de84e71173d4dada2897e5a0e1b7877e5eefbfe0d6a44edee6ce31d9b8ec09e", size = 245629, upload-time = "2025-12-28T15:40:22.397Z" },
- { url = "https://files.pythonhosted.org/packages/e2/f6/6815a220d5ec2466383d7cc36131b9fa6ecbe95c50ec52a631ba733f306a/coverage-7.13.1-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:a5a68357f686f8c4d527a2dc04f52e669c2fc1cbde38f6f7eb6a0e58cbd17cae", size = 245901, upload-time = "2025-12-28T15:40:23.836Z" },
- { url = "https://files.pythonhosted.org/packages/ac/58/40576554cd12e0872faf6d2c0eb3bc85f71d78427946ddd19ad65201e2c0/coverage-7.13.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:77cc258aeb29a3417062758975521eae60af6f79e930d6993555eeac6a8eac29", size = 246505, upload-time = "2025-12-28T15:40:25.421Z" },
- { url = "https://files.pythonhosted.org/packages/3b/77/9233a90253fba576b0eee81707b5781d0e21d97478e5377b226c5b096c0f/coverage-7.13.1-cp310-cp310-win32.whl", hash = "sha256:bb4f8c3c9a9f34423dba193f241f617b08ffc63e27f67159f60ae6baf2dcfe0f", size = 221257, upload-time = "2025-12-28T15:40:27.217Z" },
- { url = "https://files.pythonhosted.org/packages/e0/43/e842ff30c1a0a623ec80db89befb84a3a7aad7bfe44a6ea77d5a3e61fedd/coverage-7.13.1-cp310-cp310-win_amd64.whl", hash = "sha256:c8e2706ceb622bc63bac98ebb10ef5da80ed70fbd8a7999a5076de3afaef0fb1", size = 222191, upload-time = "2025-12-28T15:40:28.916Z" },
- { url = "https://files.pythonhosted.org/packages/b4/9b/77baf488516e9ced25fc215a6f75d803493fc3f6a1a1227ac35697910c2a/coverage-7.13.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:1a55d509a1dc5a5b708b5dad3b5334e07a16ad4c2185e27b40e4dba796ab7f88", size = 218755, upload-time = "2025-12-28T15:40:30.812Z" },
- { url = "https://files.pythonhosted.org/packages/d7/cd/7ab01154e6eb79ee2fab76bf4d89e94c6648116557307ee4ebbb85e5c1bf/coverage-7.13.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:4d010d080c4888371033baab27e47c9df7d6fb28d0b7b7adf85a4a49be9298b3", size = 219257, upload-time = "2025-12-28T15:40:32.333Z" },
- { url = "https://files.pythonhosted.org/packages/01/d5/b11ef7863ffbbdb509da0023fad1e9eda1c0eaea61a6d2ea5b17d4ac706e/coverage-7.13.1-cp311-cp311-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:d938b4a840fb1523b9dfbbb454f652967f18e197569c32266d4d13f37244c3d9", size = 249657, upload-time = "2025-12-28T15:40:34.1Z" },
- { url = "https://files.pythonhosted.org/packages/f7/7c/347280982982383621d29b8c544cf497ae07ac41e44b1ca4903024131f55/coverage-7.13.1-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:bf100a3288f9bb7f919b87eb84f87101e197535b9bd0e2c2b5b3179633324fee", size = 251581, upload-time = "2025-12-28T15:40:36.131Z" },
- { url = "https://files.pythonhosted.org/packages/82/f6/ebcfed11036ade4c0d75fa4453a6282bdd225bc073862766eec184a4c643/coverage-7.13.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ef6688db9bf91ba111ae734ba6ef1a063304a881749726e0d3575f5c10a9facf", size = 253691, upload-time = "2025-12-28T15:40:37.626Z" },
- { url = "https://files.pythonhosted.org/packages/02/92/af8f5582787f5d1a8b130b2dcba785fa5e9a7a8e121a0bb2220a6fdbdb8a/coverage-7.13.1-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0b609fc9cdbd1f02e51f67f51e5aee60a841ef58a68d00d5ee2c0faf357481a3", size = 249799, upload-time = "2025-12-28T15:40:39.47Z" },
- { url = "https://files.pythonhosted.org/packages/24/aa/0e39a2a3b16eebf7f193863323edbff38b6daba711abaaf807d4290cf61a/coverage-7.13.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:c43257717611ff5e9a1d79dce8e47566235ebda63328718d9b65dd640bc832ef", size = 251389, upload-time = "2025-12-28T15:40:40.954Z" },
- { url = "https://files.pythonhosted.org/packages/73/46/7f0c13111154dc5b978900c0ccee2e2ca239b910890e674a77f1363d483e/coverage-7.13.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:e09fbecc007f7b6afdfb3b07ce5bd9f8494b6856dd4f577d26c66c391b829851", size = 249450, upload-time = "2025-12-28T15:40:42.489Z" },
- { url = "https://files.pythonhosted.org/packages/ac/ca/e80da6769e8b669ec3695598c58eef7ad98b0e26e66333996aee6316db23/coverage-7.13.1-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:a03a4f3a19a189919c7055098790285cc5c5b0b3976f8d227aea39dbf9f8bfdb", size = 249170, upload-time = "2025-12-28T15:40:44.279Z" },
- { url = "https://files.pythonhosted.org/packages/af/18/9e29baabdec1a8644157f572541079b4658199cfd372a578f84228e860de/coverage-7.13.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:3820778ea1387c2b6a818caec01c63adc5b3750211af6447e8dcfb9b6f08dbba", size = 250081, upload-time = "2025-12-28T15:40:45.748Z" },
- { url = "https://files.pythonhosted.org/packages/00/f8/c3021625a71c3b2f516464d322e41636aea381018319050a8114105872ee/coverage-7.13.1-cp311-cp311-win32.whl", hash = "sha256:ff10896fa55167371960c5908150b434b71c876dfab97b69478f22c8b445ea19", size = 221281, upload-time = "2025-12-28T15:40:47.232Z" },
- { url = "https://files.pythonhosted.org/packages/27/56/c216625f453df6e0559ed666d246fcbaaa93f3aa99eaa5080cea1229aa3d/coverage-7.13.1-cp311-cp311-win_amd64.whl", hash = "sha256:a998cc0aeeea4c6d5622a3754da5a493055d2d95186bad877b0a34ea6e6dbe0a", size = 222215, upload-time = "2025-12-28T15:40:49.19Z" },
- { url = "https://files.pythonhosted.org/packages/5c/9a/be342e76f6e531cae6406dc46af0d350586f24d9b67fdfa6daee02df71af/coverage-7.13.1-cp311-cp311-win_arm64.whl", hash = "sha256:fea07c1a39a22614acb762e3fbbb4011f65eedafcb2948feeef641ac78b4ee5c", size = 220886, upload-time = "2025-12-28T15:40:51.067Z" },
- { url = "https://files.pythonhosted.org/packages/ce/8a/87af46cccdfa78f53db747b09f5f9a21d5fc38d796834adac09b30a8ce74/coverage-7.13.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6f34591000f06e62085b1865c9bc5f7858df748834662a51edadfd2c3bfe0dd3", size = 218927, upload-time = "2025-12-28T15:40:52.814Z" },
- { url = "https://files.pythonhosted.org/packages/82/a8/6e22fdc67242a4a5a153f9438d05944553121c8f4ba70cb072af4c41362e/coverage-7.13.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b67e47c5595b9224599016e333f5ec25392597a89d5744658f837d204e16c63e", size = 219288, upload-time = "2025-12-28T15:40:54.262Z" },
- { url = "https://files.pythonhosted.org/packages/d0/0a/853a76e03b0f7c4375e2ca025df45c918beb367f3e20a0a8e91967f6e96c/coverage-7.13.1-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:3e7b8bd70c48ffb28461ebe092c2345536fb18bbbf19d287c8913699735f505c", size = 250786, upload-time = "2025-12-28T15:40:56.059Z" },
- { url = "https://files.pythonhosted.org/packages/ea/b4/694159c15c52b9f7ec7adf49d50e5f8ee71d3e9ef38adb4445d13dd56c20/coverage-7.13.1-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:c223d078112e90dc0e5c4e35b98b9584164bea9fbbd221c0b21c5241f6d51b62", size = 253543, upload-time = "2025-12-28T15:40:57.585Z" },
- { url = "https://files.pythonhosted.org/packages/96/b2/7f1f0437a5c855f87e17cf5d0dc35920b6440ff2b58b1ba9788c059c26c8/coverage-7.13.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:794f7c05af0763b1bbd1b9e6eff0e52ad068be3b12cd96c87de037b01390c968", size = 254635, upload-time = "2025-12-28T15:40:59.443Z" },
- { url = "https://files.pythonhosted.org/packages/e9/d1/73c3fdb8d7d3bddd9473c9c6a2e0682f09fc3dfbcb9c3f36412a7368bcab/coverage-7.13.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0642eae483cc8c2902e4af7298bf886d605e80f26382124cddc3967c2a3df09e", size = 251202, upload-time = "2025-12-28T15:41:01.328Z" },
- { url = "https://files.pythonhosted.org/packages/66/3c/f0edf75dcc152f145d5598329e864bbbe04ab78660fe3e8e395f9fff010f/coverage-7.13.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:9f5e772ed5fef25b3de9f2008fe67b92d46831bd2bc5bdc5dd6bfd06b83b316f", size = 252566, upload-time = "2025-12-28T15:41:03.319Z" },
- { url = "https://files.pythonhosted.org/packages/17/b3/e64206d3c5f7dcbceafd14941345a754d3dbc78a823a6ed526e23b9cdaab/coverage-7.13.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:45980ea19277dc0a579e432aef6a504fe098ef3a9032ead15e446eb0f1191aee", size = 250711, upload-time = "2025-12-28T15:41:06.411Z" },
- { url = "https://files.pythonhosted.org/packages/dc/ad/28a3eb970a8ef5b479ee7f0c484a19c34e277479a5b70269dc652b730733/coverage-7.13.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:e4f18eca6028ffa62adbd185a8f1e1dd242f2e68164dba5c2b74a5204850b4cf", size = 250278, upload-time = "2025-12-28T15:41:08.285Z" },
- { url = "https://files.pythonhosted.org/packages/54/e3/c8f0f1a93133e3e1291ca76cbb63565bd4b5c5df63b141f539d747fff348/coverage-7.13.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f8dca5590fec7a89ed6826fce625595279e586ead52e9e958d3237821fbc750c", size = 252154, upload-time = "2025-12-28T15:41:09.969Z" },
- { url = "https://files.pythonhosted.org/packages/d0/bf/9939c5d6859c380e405b19e736321f1c7d402728792f4c752ad1adcce005/coverage-7.13.1-cp312-cp312-win32.whl", hash = "sha256:ff86d4e85188bba72cfb876df3e11fa243439882c55957184af44a35bd5880b7", size = 221487, upload-time = "2025-12-28T15:41:11.468Z" },
- { url = "https://files.pythonhosted.org/packages/fa/dc/7282856a407c621c2aad74021680a01b23010bb8ebf427cf5eacda2e876f/coverage-7.13.1-cp312-cp312-win_amd64.whl", hash = "sha256:16cc1da46c04fb0fb128b4dc430b78fa2aba8a6c0c9f8eb391fd5103409a6ac6", size = 222299, upload-time = "2025-12-28T15:41:13.386Z" },
- { url = "https://files.pythonhosted.org/packages/10/79/176a11203412c350b3e9578620013af35bcdb79b651eb976f4a4b32044fa/coverage-7.13.1-cp312-cp312-win_arm64.whl", hash = "sha256:8d9bc218650022a768f3775dd7fdac1886437325d8d295d923ebcfef4892ad5c", size = 220941, upload-time = "2025-12-28T15:41:14.975Z" },
- { url = "https://files.pythonhosted.org/packages/a3/a4/e98e689347a1ff1a7f67932ab535cef82eb5e78f32a9e4132e114bbb3a0a/coverage-7.13.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:cb237bfd0ef4d5eb6a19e29f9e528ac67ac3be932ea6b44fb6cc09b9f3ecff78", size = 218951, upload-time = "2025-12-28T15:41:16.653Z" },
- { url = "https://files.pythonhosted.org/packages/32/33/7cbfe2bdc6e2f03d6b240d23dc45fdaf3fd270aaf2d640be77b7f16989ab/coverage-7.13.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:1dcb645d7e34dcbcc96cd7c132b1fc55c39263ca62eb961c064eb3928997363b", size = 219325, upload-time = "2025-12-28T15:41:18.609Z" },
- { url = "https://files.pythonhosted.org/packages/59/f6/efdabdb4929487baeb7cb2a9f7dac457d9356f6ad1b255be283d58b16316/coverage-7.13.1-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:3d42df8201e00384736f0df9be2ced39324c3907607d17d50d50116c989d84cd", size = 250309, upload-time = "2025-12-28T15:41:20.629Z" },
- { url = "https://files.pythonhosted.org/packages/12/da/91a52516e9d5aea87d32d1523f9cdcf7a35a3b298e6be05d6509ba3cfab2/coverage-7.13.1-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:fa3edde1aa8807de1d05934982416cb3ec46d1d4d91e280bcce7cca01c507992", size = 252907, upload-time = "2025-12-28T15:41:22.257Z" },
- { url = "https://files.pythonhosted.org/packages/75/38/f1ea837e3dc1231e086db1638947e00d264e7e8c41aa8ecacf6e1e0c05f4/coverage-7.13.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9edd0e01a343766add6817bc448408858ba6b489039eaaa2018474e4001651a4", size = 254148, upload-time = "2025-12-28T15:41:23.87Z" },
- { url = "https://files.pythonhosted.org/packages/7f/43/f4f16b881aaa34954ba446318dea6b9ed5405dd725dd8daac2358eda869a/coverage-7.13.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:985b7836931d033570b94c94713c6dba5f9d3ff26045f72c3e5dbc5fe3361e5a", size = 250515, upload-time = "2025-12-28T15:41:25.437Z" },
- { url = "https://files.pythonhosted.org/packages/84/34/8cba7f00078bd468ea914134e0144263194ce849ec3baad187ffb6203d1c/coverage-7.13.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ffed1e4980889765c84a5d1a566159e363b71d6b6fbaf0bebc9d3c30bc016766", size = 252292, upload-time = "2025-12-28T15:41:28.459Z" },
- { url = "https://files.pythonhosted.org/packages/8c/a4/cffac66c7652d84ee4ac52d3ccb94c015687d3b513f9db04bfcac2ac800d/coverage-7.13.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:8842af7f175078456b8b17f1b73a0d16a65dcbdc653ecefeb00a56b3c8c298c4", size = 250242, upload-time = "2025-12-28T15:41:30.02Z" },
- { url = "https://files.pythonhosted.org/packages/f4/78/9a64d462263dde416f3c0067efade7b52b52796f489b1037a95b0dc389c9/coverage-7.13.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:ccd7a6fca48ca9c131d9b0a2972a581e28b13416fc313fb98b6d24a03ce9a398", size = 250068, upload-time = "2025-12-28T15:41:32.007Z" },
- { url = "https://files.pythonhosted.org/packages/69/c8/a8994f5fece06db7c4a97c8fc1973684e178599b42e66280dded0524ef00/coverage-7.13.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:0403f647055de2609be776965108447deb8e384fe4a553c119e3ff6bfbab4784", size = 251846, upload-time = "2025-12-28T15:41:33.946Z" },
- { url = "https://files.pythonhosted.org/packages/cc/f7/91fa73c4b80305c86598a2d4e54ba22df6bf7d0d97500944af7ef155d9f7/coverage-7.13.1-cp313-cp313-win32.whl", hash = "sha256:549d195116a1ba1e1ae2f5ca143f9777800f6636eab917d4f02b5310d6d73461", size = 221512, upload-time = "2025-12-28T15:41:35.519Z" },
- { url = "https://files.pythonhosted.org/packages/45/0b/0768b4231d5a044da8f75e097a8714ae1041246bb765d6b5563bab456735/coverage-7.13.1-cp313-cp313-win_amd64.whl", hash = "sha256:5899d28b5276f536fcf840b18b61a9fce23cc3aec1d114c44c07fe94ebeaa500", size = 222321, upload-time = "2025-12-28T15:41:37.371Z" },
- { url = "https://files.pythonhosted.org/packages/9b/b8/bdcb7253b7e85157282450262008f1366aa04663f3e3e4c30436f596c3e2/coverage-7.13.1-cp313-cp313-win_arm64.whl", hash = "sha256:868a2fae76dfb06e87291bcbd4dcbcc778a8500510b618d50496e520bd94d9b9", size = 220949, upload-time = "2025-12-28T15:41:39.553Z" },
- { url = "https://files.pythonhosted.org/packages/70/52/f2be52cc445ff75ea8397948c96c1b4ee14f7f9086ea62fc929c5ae7b717/coverage-7.13.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:67170979de0dacac3f3097d02b0ad188d8edcea44ccc44aaa0550af49150c7dc", size = 219643, upload-time = "2025-12-28T15:41:41.567Z" },
- { url = "https://files.pythonhosted.org/packages/47/79/c85e378eaa239e2edec0c5523f71542c7793fe3340954eafb0bc3904d32d/coverage-7.13.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:f80e2bb21bfab56ed7405c2d79d34b5dc0bc96c2c1d2a067b643a09fb756c43a", size = 219997, upload-time = "2025-12-28T15:41:43.418Z" },
- { url = "https://files.pythonhosted.org/packages/fe/9b/b1ade8bfb653c0bbce2d6d6e90cc6c254cbb99b7248531cc76253cb4da6d/coverage-7.13.1-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:f83351e0f7dcdb14d7326c3d8d8c4e915fa685cbfdc6281f9470d97a04e9dfe4", size = 261296, upload-time = "2025-12-28T15:41:45.207Z" },
- { url = "https://files.pythonhosted.org/packages/1f/af/ebf91e3e1a2473d523e87e87fd8581e0aa08741b96265730e2d79ce78d8d/coverage-7.13.1-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:bb3f6562e89bad0110afbe64e485aac2462efdce6232cdec7862a095dc3412f6", size = 263363, upload-time = "2025-12-28T15:41:47.163Z" },
- { url = "https://files.pythonhosted.org/packages/c4/8b/fb2423526d446596624ac7fde12ea4262e66f86f5120114c3cfd0bb2befa/coverage-7.13.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:77545b5dcda13b70f872c3b5974ac64c21d05e65b1590b441c8560115dc3a0d1", size = 265783, upload-time = "2025-12-28T15:41:49.03Z" },
- { url = "https://files.pythonhosted.org/packages/9b/26/ef2adb1e22674913b89f0fe7490ecadcef4a71fa96f5ced90c60ec358789/coverage-7.13.1-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a4d240d260a1aed814790bbe1f10a5ff31ce6c21bc78f0da4a1e8268d6c80dbd", size = 260508, upload-time = "2025-12-28T15:41:51.035Z" },
- { url = "https://files.pythonhosted.org/packages/ce/7d/f0f59b3404caf662e7b5346247883887687c074ce67ba453ea08c612b1d5/coverage-7.13.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:d2287ac9360dec3837bfdad969963a5d073a09a85d898bd86bea82aa8876ef3c", size = 263357, upload-time = "2025-12-28T15:41:52.631Z" },
- { url = "https://files.pythonhosted.org/packages/1a/b1/29896492b0b1a047604d35d6fa804f12818fa30cdad660763a5f3159e158/coverage-7.13.1-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:0d2c11f3ea4db66b5cbded23b20185c35066892c67d80ec4be4bab257b9ad1e0", size = 260978, upload-time = "2025-12-28T15:41:54.589Z" },
- { url = "https://files.pythonhosted.org/packages/48/f2/971de1238a62e6f0a4128d37adadc8bb882ee96afbe03ff1570291754629/coverage-7.13.1-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:3fc6a169517ca0d7ca6846c3c5392ef2b9e38896f61d615cb75b9e7134d4ee1e", size = 259877, upload-time = "2025-12-28T15:41:56.263Z" },
- { url = "https://files.pythonhosted.org/packages/6a/fc/0474efcbb590ff8628830e9aaec5f1831594874360e3251f1fdec31d07a3/coverage-7.13.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:d10a2ed46386e850bb3de503a54f9fe8192e5917fcbb143bfef653a9355e9a53", size = 262069, upload-time = "2025-12-28T15:41:58.093Z" },
- { url = "https://files.pythonhosted.org/packages/88/4f/3c159b7953db37a7b44c0eab8a95c37d1aa4257c47b4602c04022d5cb975/coverage-7.13.1-cp313-cp313t-win32.whl", hash = "sha256:75a6f4aa904301dab8022397a22c0039edc1f51e90b83dbd4464b8a38dc87842", size = 222184, upload-time = "2025-12-28T15:41:59.763Z" },
- { url = "https://files.pythonhosted.org/packages/58/a5/6b57d28f81417f9335774f20679d9d13b9a8fb90cd6160957aa3b54a2379/coverage-7.13.1-cp313-cp313t-win_amd64.whl", hash = "sha256:309ef5706e95e62578cda256b97f5e097916a2c26247c287bbe74794e7150df2", size = 223250, upload-time = "2025-12-28T15:42:01.52Z" },
- { url = "https://files.pythonhosted.org/packages/81/7c/160796f3b035acfbb58be80e02e484548595aa67e16a6345e7910ace0a38/coverage-7.13.1-cp313-cp313t-win_arm64.whl", hash = "sha256:92f980729e79b5d16d221038dbf2e8f9a9136afa072f9d5d6ed4cb984b126a09", size = 221521, upload-time = "2025-12-28T15:42:03.275Z" },
- { url = "https://files.pythonhosted.org/packages/aa/8e/ba0e597560c6563fc0adb902fda6526df5d4aa73bb10adf0574d03bd2206/coverage-7.13.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:97ab3647280d458a1f9adb85244e81587505a43c0c7cff851f5116cd2814b894", size = 218996, upload-time = "2025-12-28T15:42:04.978Z" },
- { url = "https://files.pythonhosted.org/packages/6b/8e/764c6e116f4221dc7aa26c4061181ff92edb9c799adae6433d18eeba7a14/coverage-7.13.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:8f572d989142e0908e6acf57ad1b9b86989ff057c006d13b76c146ec6a20216a", size = 219326, upload-time = "2025-12-28T15:42:06.691Z" },
- { url = "https://files.pythonhosted.org/packages/4f/a6/6130dc6d8da28cdcbb0f2bf8865aeca9b157622f7c0031e48c6cf9a0e591/coverage-7.13.1-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:d72140ccf8a147e94274024ff6fd8fb7811354cf7ef88b1f0a988ebaa5bc774f", size = 250374, upload-time = "2025-12-28T15:42:08.786Z" },
- { url = "https://files.pythonhosted.org/packages/82/2b/783ded568f7cd6b677762f780ad338bf4b4750205860c17c25f7c708995e/coverage-7.13.1-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:d3c9f051b028810f5a87c88e5d6e9af3c0ff32ef62763bf15d29f740453ca909", size = 252882, upload-time = "2025-12-28T15:42:10.515Z" },
- { url = "https://files.pythonhosted.org/packages/cd/b2/9808766d082e6a4d59eb0cc881a57fc1600eb2c5882813eefff8254f71b5/coverage-7.13.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f398ba4df52d30b1763f62eed9de5620dcde96e6f491f4c62686736b155aa6e4", size = 254218, upload-time = "2025-12-28T15:42:12.208Z" },
- { url = "https://files.pythonhosted.org/packages/44/ea/52a985bb447c871cb4d2e376e401116520991b597c85afdde1ea9ef54f2c/coverage-7.13.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:132718176cc723026d201e347f800cd1a9e4b62ccd3f82476950834dad501c75", size = 250391, upload-time = "2025-12-28T15:42:14.21Z" },
- { url = "https://files.pythonhosted.org/packages/7f/1d/125b36cc12310718873cfc8209ecfbc1008f14f4f5fa0662aa608e579353/coverage-7.13.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:9e549d642426e3579b3f4b92d0431543b012dcb6e825c91619d4e93b7363c3f9", size = 252239, upload-time = "2025-12-28T15:42:16.292Z" },
- { url = "https://files.pythonhosted.org/packages/6a/16/10c1c164950cade470107f9f14bbac8485f8fb8515f515fca53d337e4a7f/coverage-7.13.1-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:90480b2134999301eea795b3a9dbf606c6fbab1b489150c501da84a959442465", size = 250196, upload-time = "2025-12-28T15:42:18.54Z" },
- { url = "https://files.pythonhosted.org/packages/2a/c6/cd860fac08780c6fd659732f6ced1b40b79c35977c1356344e44d72ba6c4/coverage-7.13.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:e825dbb7f84dfa24663dd75835e7257f8882629fc11f03ecf77d84a75134b864", size = 250008, upload-time = "2025-12-28T15:42:20.365Z" },
- { url = "https://files.pythonhosted.org/packages/f0/3a/a8c58d3d38f82a5711e1e0a67268362af48e1a03df27c03072ac30feefcf/coverage-7.13.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:623dcc6d7a7ba450bbdbeedbaa0c42b329bdae16491af2282f12a7e809be7eb9", size = 251671, upload-time = "2025-12-28T15:42:22.114Z" },
- { url = "https://files.pythonhosted.org/packages/f0/bc/fd4c1da651d037a1e3d53e8cb3f8182f4b53271ffa9a95a2e211bacc0349/coverage-7.13.1-cp314-cp314-win32.whl", hash = "sha256:6e73ebb44dca5f708dc871fe0b90cf4cff1a13f9956f747cc87b535a840386f5", size = 221777, upload-time = "2025-12-28T15:42:23.919Z" },
- { url = "https://files.pythonhosted.org/packages/4b/50/71acabdc8948464c17e90b5ffd92358579bd0910732c2a1c9537d7536aa6/coverage-7.13.1-cp314-cp314-win_amd64.whl", hash = "sha256:be753b225d159feb397bd0bf91ae86f689bad0da09d3b301478cd39b878ab31a", size = 222592, upload-time = "2025-12-28T15:42:25.619Z" },
- { url = "https://files.pythonhosted.org/packages/f7/c8/a6fb943081bb0cc926499c7907731a6dc9efc2cbdc76d738c0ab752f1a32/coverage-7.13.1-cp314-cp314-win_arm64.whl", hash = "sha256:228b90f613b25ba0019361e4ab81520b343b622fc657daf7e501c4ed6a2366c0", size = 221169, upload-time = "2025-12-28T15:42:27.629Z" },
- { url = "https://files.pythonhosted.org/packages/16/61/d5b7a0a0e0e40d62e59bc8c7aa1afbd86280d82728ba97f0673b746b78e2/coverage-7.13.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:60cfb538fe9ef86e5b2ab0ca8fc8d62524777f6c611dcaf76dc16fbe9b8e698a", size = 219730, upload-time = "2025-12-28T15:42:29.306Z" },
- { url = "https://files.pythonhosted.org/packages/a3/2c/8881326445fd071bb49514d1ce97d18a46a980712b51fee84f9ab42845b4/coverage-7.13.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:57dfc8048c72ba48a8c45e188d811e5efd7e49b387effc8fb17e97936dde5bf6", size = 220001, upload-time = "2025-12-28T15:42:31.319Z" },
- { url = "https://files.pythonhosted.org/packages/b5/d7/50de63af51dfa3a7f91cc37ad8fcc1e244b734232fbc8b9ab0f3c834a5cd/coverage-7.13.1-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:3f2f725aa3e909b3c5fdb8192490bdd8e1495e85906af74fe6e34a2a77ba0673", size = 261370, upload-time = "2025-12-28T15:42:32.992Z" },
- { url = "https://files.pythonhosted.org/packages/e1/2c/d31722f0ec918fd7453b2758312729f645978d212b410cd0f7c2aed88a94/coverage-7.13.1-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9ee68b21909686eeb21dfcba2c3b81fee70dcf38b140dcd5aa70680995fa3aa5", size = 263485, upload-time = "2025-12-28T15:42:34.759Z" },
- { url = "https://files.pythonhosted.org/packages/fa/7a/2c114fa5c5fc08ba0777e4aec4c97e0b4a1afcb69c75f1f54cff78b073ab/coverage-7.13.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:724b1b270cb13ea2e6503476e34541a0b1f62280bc997eab443f87790202033d", size = 265890, upload-time = "2025-12-28T15:42:36.517Z" },
- { url = "https://files.pythonhosted.org/packages/65/d9/f0794aa1c74ceabc780fe17f6c338456bbc4e96bd950f2e969f48ac6fb20/coverage-7.13.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:916abf1ac5cf7eb16bc540a5bf75c71c43a676f5c52fcb9fe75a2bd75fb944e8", size = 260445, upload-time = "2025-12-28T15:42:38.646Z" },
- { url = "https://files.pythonhosted.org/packages/49/23/184b22a00d9bb97488863ced9454068c79e413cb23f472da6cbddc6cfc52/coverage-7.13.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:776483fd35b58d8afe3acbd9988d5de592ab6da2d2a865edfdbc9fdb43e7c486", size = 263357, upload-time = "2025-12-28T15:42:40.788Z" },
- { url = "https://files.pythonhosted.org/packages/7d/bd/58af54c0c9199ea4190284f389005779d7daf7bf3ce40dcd2d2b2f96da69/coverage-7.13.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:b6f3b96617e9852703f5b633ea01315ca45c77e879584f283c44127f0f1ec564", size = 260959, upload-time = "2025-12-28T15:42:42.808Z" },
- { url = "https://files.pythonhosted.org/packages/4b/2a/6839294e8f78a4891bf1df79d69c536880ba2f970d0ff09e7513d6e352e9/coverage-7.13.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:bd63e7b74661fed317212fab774e2a648bc4bb09b35f25474f8e3325d2945cd7", size = 259792, upload-time = "2025-12-28T15:42:44.818Z" },
- { url = "https://files.pythonhosted.org/packages/ba/c3/528674d4623283310ad676c5af7414b9850ab6d55c2300e8aa4b945ec554/coverage-7.13.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:933082f161bbb3e9f90d00990dc956120f608cdbcaeea15c4d897f56ef4fe416", size = 262123, upload-time = "2025-12-28T15:42:47.108Z" },
- { url = "https://files.pythonhosted.org/packages/06/c5/8c0515692fb4c73ac379d8dc09b18eaf0214ecb76ea6e62467ba7a1556ff/coverage-7.13.1-cp314-cp314t-win32.whl", hash = "sha256:18be793c4c87de2965e1c0f060f03d9e5aff66cfeae8e1dbe6e5b88056ec153f", size = 222562, upload-time = "2025-12-28T15:42:49.144Z" },
- { url = "https://files.pythonhosted.org/packages/05/0e/c0a0c4678cb30dac735811db529b321d7e1c9120b79bd728d4f4d6b010e9/coverage-7.13.1-cp314-cp314t-win_amd64.whl", hash = "sha256:0e42e0ec0cd3e0d851cb3c91f770c9301f48647cb2877cb78f74bdaa07639a79", size = 223670, upload-time = "2025-12-28T15:42:51.218Z" },
- { url = "https://files.pythonhosted.org/packages/f5/5f/b177aa0011f354abf03a8f30a85032686d290fdeed4222b27d36b4372a50/coverage-7.13.1-cp314-cp314t-win_arm64.whl", hash = "sha256:eaecf47ef10c72ece9a2a92118257da87e460e113b83cc0d2905cbbe931792b4", size = 221707, upload-time = "2025-12-28T15:42:53.034Z" },
- { url = "https://files.pythonhosted.org/packages/cc/48/d9f421cb8da5afaa1a64570d9989e00fb7955e6acddc5a12979f7666ef60/coverage-7.13.1-py3-none-any.whl", hash = "sha256:2016745cb3ba554469d02819d78958b571792bb68e31302610e898f80dd3a573", size = 210722, upload-time = "2025-12-28T15:42:54.901Z" },
-]
-
-[package.optional-dependencies]
-toml = [
- { name = "tomli", marker = "python_full_version <= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+version = "7.13.5"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/9d/e0/70553e3000e345daff267cec284ce4cbf3fc141b6da229ac52775b5428f1/coverage-7.13.5.tar.gz", hash = "sha256:c81f6515c4c40141f83f502b07bbfa5c240ba25bbe73da7b33f1e5b6120ff179", size = 915967, upload-time = "2026-03-17T10:33:18.341Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a0/c3/a396306ba7db865bf96fc1fb3b7fd29bcbf3d829df642e77b13555163cd6/coverage-7.13.5-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:460cf0114c5016fa841214ff5564aa4864f11948da9440bc97e21ad1f4ba1e01", size = 219554, upload-time = "2026-03-17T10:30:42.208Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/16/a68a19e5384e93f811dccc51034b1fd0b865841c390e3c931dcc4699e035/coverage-7.13.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:0e223ce4b4ed47f065bfb123687686512e37629be25cc63728557ae7db261422", size = 219908, upload-time = "2026-03-17T10:30:43.906Z" },
+ { url = "https://files.pythonhosted.org/packages/29/72/20b917c6793af3a5ceb7fb9c50033f3ec7865f2911a1416b34a7cfa0813b/coverage-7.13.5-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:6e3370441f4513c6252bf042b9c36d22491142385049243253c7e48398a15a9f", size = 251419, upload-time = "2026-03-17T10:30:45.545Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/49/cd14b789536ac6a4778c453c6a2338bc0a2fb60c5a5a41b4008328b9acc1/coverage-7.13.5-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:03ccc709a17a1de074fb1d11f217342fb0d2b1582ed544f554fc9fc3f07e95f5", size = 254159, upload-time = "2026-03-17T10:30:47.204Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/00/7b0edcfe64e2ed4c0340dac14a52ad0f4c9bd0b8b5e531af7d55b703db7c/coverage-7.13.5-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3f4818d065964db3c1c66dc0fbdac5ac692ecbc875555e13374fdbe7eedb4376", size = 255270, upload-time = "2026-03-17T10:30:48.812Z" },
+ { url = "https://files.pythonhosted.org/packages/93/89/7ffc4ba0f5d0a55c1e84ea7cee39c9fc06af7b170513d83fbf3bbefce280/coverage-7.13.5-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:012d5319e66e9d5a218834642d6c35d265515a62f01157a45bcc036ecf947256", size = 257538, upload-time = "2026-03-17T10:30:50.77Z" },
+ { url = "https://files.pythonhosted.org/packages/81/bd/73ddf85f93f7e6fa83e77ccecb6162d9415c79007b4bc124008a4995e4a7/coverage-7.13.5-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:8dd02af98971bdb956363e4827d34425cb3df19ee550ef92855b0acb9c7ce51c", size = 251821, upload-time = "2026-03-17T10:30:52.5Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/81/278aff4e8dec4926a0bcb9486320752811f543a3ce5b602cc7a29978d073/coverage-7.13.5-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:f08fd75c50a760c7eb068ae823777268daaf16a80b918fa58eea888f8e3919f5", size = 253191, upload-time = "2026-03-17T10:30:54.543Z" },
+ { url = "https://files.pythonhosted.org/packages/70/ee/fe1621488e2e0a58d7e94c4800f0d96f79671553488d401a612bebae324b/coverage-7.13.5-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:843ea8643cf967d1ac7e8ecd4bb00c99135adf4816c0c0593fdcc47b597fcf09", size = 251337, upload-time = "2026-03-17T10:30:56.663Z" },
+ { url = "https://files.pythonhosted.org/packages/37/a6/f79fb37aa104b562207cc23cb5711ab6793608e246cae1e93f26b2236ed9/coverage-7.13.5-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:9d44d7aa963820b1b971dbecd90bfe5fe8f81cff79787eb6cca15750bd2f79b9", size = 255404, upload-time = "2026-03-17T10:30:58.427Z" },
+ { url = "https://files.pythonhosted.org/packages/75/f0/ed15262a58ec81ce457ceb717b7f78752a1713556b19081b76e90896e8d4/coverage-7.13.5-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:7132bed4bd7b836200c591410ae7d97bf7ae8be6fc87d160b2bd881df929e7bf", size = 250903, upload-time = "2026-03-17T10:31:00.093Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/e9/9129958f20e7e9d4d56d51d42ccf708d15cac355ff4ac6e736e97a9393d2/coverage-7.13.5-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:a698e363641b98843c517817db75373c83254781426e94ada3197cabbc2c919c", size = 252780, upload-time = "2026-03-17T10:31:01.916Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/d7/0ad9b15812d81272db94379fe4c6df8fd17781cc7671fdfa30c76ba5ff7b/coverage-7.13.5-cp312-cp312-win32.whl", hash = "sha256:bdba0a6b8812e8c7df002d908a9a2ea3c36e92611b5708633c50869e6d922fdf", size = 222093, upload-time = "2026-03-17T10:31:03.642Z" },
+ { url = "https://files.pythonhosted.org/packages/29/3d/821a9a5799fac2556bcf0bd37a70d1d11fa9e49784b6d22e92e8b2f85f18/coverage-7.13.5-cp312-cp312-win_amd64.whl", hash = "sha256:d2c87e0c473a10bffe991502eac389220533024c8082ec1ce849f4218dded810", size = 222900, upload-time = "2026-03-17T10:31:05.651Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/fa/2238c2ad08e35cf4f020ea721f717e09ec3152aea75d191a7faf3ef009a8/coverage-7.13.5-cp312-cp312-win_arm64.whl", hash = "sha256:bf69236a9a81bdca3bff53796237aab096cdbf8d78a66ad61e992d9dac7eb2de", size = 221515, upload-time = "2026-03-17T10:31:07.293Z" },
+ { url = "https://files.pythonhosted.org/packages/74/8c/74fedc9663dcf168b0a059d4ea756ecae4da77a489048f94b5f512a8d0b3/coverage-7.13.5-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5ec4af212df513e399cf11610cc27063f1586419e814755ab362e50a85ea69c1", size = 219576, upload-time = "2026-03-17T10:31:09.045Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/c9/44fb661c55062f0818a6ffd2685c67aa30816200d5f2817543717d4b92eb/coverage-7.13.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:941617e518602e2d64942c88ec8499f7fbd49d3f6c4327d3a71d43a1973032f3", size = 219942, upload-time = "2026-03-17T10:31:10.708Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/13/93419671cee82b780bab7ea96b67c8ef448f5f295f36bf5031154ec9a790/coverage-7.13.5-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:da305e9937617ee95c2e39d8ff9f040e0487cbf1ac174f777ed5eddd7a7c1f26", size = 250935, upload-time = "2026-03-17T10:31:12.392Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/68/1666e3a4462f8202d836920114fa7a5ee9275d1fa45366d336c551a162dd/coverage-7.13.5-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:78e696e1cc714e57e8b25760b33a8b1026b7048d270140d25dafe1b0a1ee05a3", size = 253541, upload-time = "2026-03-17T10:31:14.247Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/5e/3ee3b835647be646dcf3c65a7c6c18f87c27326a858f72ab22c12730773d/coverage-7.13.5-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:02ca0eed225b2ff301c474aeeeae27d26e2537942aa0f87491d3e147e784a82b", size = 254780, upload-time = "2026-03-17T10:31:16.193Z" },
+ { url = "https://files.pythonhosted.org/packages/44/b3/cb5bd1a04cfcc49ede6cd8409d80bee17661167686741e041abc7ee1b9a9/coverage-7.13.5-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:04690832cbea4e4663d9149e05dba142546ca05cb1848816760e7f58285c970a", size = 256912, upload-time = "2026-03-17T10:31:17.89Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/66/c1dceb7b9714473800b075f5c8a84f4588f887a90eb8645282031676e242/coverage-7.13.5-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0590e44dd2745c696a778f7bab6aa95256de2cbc8b8cff4f7db8ff09813d6969", size = 251165, upload-time = "2026-03-17T10:31:19.605Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/62/5502b73b97aa2e53ea22a39cf8649ff44827bef76d90bf638777daa27a9d/coverage-7.13.5-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:d7cfad2d6d81dd298ab6b89fe72c3b7b05ec7544bdda3b707ddaecff8d25c161", size = 252908, upload-time = "2026-03-17T10:31:21.312Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/37/7792c2d69854397ca77a55c4646e5897c467928b0e27f2d235d83b5d08c6/coverage-7.13.5-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:e092b9499de38ae0fbfbc603a74660eb6ff3e869e507b50d85a13b6db9863e15", size = 250873, upload-time = "2026-03-17T10:31:23.565Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/23/bc866fb6163be52a8a9e5d708ba0d3b1283c12158cefca0a8bbb6e247a43/coverage-7.13.5-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:48c39bc4a04d983a54a705a6389512883d4a3b9862991b3617d547940e9f52b1", size = 255030, upload-time = "2026-03-17T10:31:25.58Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/8b/ef67e1c222ef49860701d346b8bbb70881bef283bd5f6cbba68a39a086c7/coverage-7.13.5-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:2d3807015f138ffea1ed9afeeb8624fd781703f2858b62a8dd8da5a0994c57b6", size = 250694, upload-time = "2026-03-17T10:31:27.316Z" },
+ { url = "https://files.pythonhosted.org/packages/46/0d/866d1f74f0acddbb906db212e096dee77a8e2158ca5e6bb44729f9d93298/coverage-7.13.5-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ee2aa19e03161671ec964004fb74b2257805d9710bf14a5c704558b9d8dbaf17", size = 252469, upload-time = "2026-03-17T10:31:29.472Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/f5/be742fec31118f02ce42b21c6af187ad6a344fed546b56ca60caacc6a9a0/coverage-7.13.5-cp313-cp313-win32.whl", hash = "sha256:ce1998c0483007608c8382f4ff50164bfc5bd07a2246dd272aa4043b75e61e85", size = 222112, upload-time = "2026-03-17T10:31:31.526Z" },
+ { url = "https://files.pythonhosted.org/packages/66/40/7732d648ab9d069a46e686043241f01206348e2bbf128daea85be4d6414b/coverage-7.13.5-cp313-cp313-win_amd64.whl", hash = "sha256:631efb83f01569670a5e866ceb80fe483e7c159fac6f167e6571522636104a0b", size = 222923, upload-time = "2026-03-17T10:31:33.633Z" },
+ { url = "https://files.pythonhosted.org/packages/48/af/fea819c12a095781f6ccd504890aaddaf88b8fab263c4940e82c7b770124/coverage-7.13.5-cp313-cp313-win_arm64.whl", hash = "sha256:f4cd16206ad171cbc2470dbea9103cf9a7607d5fe8c242fdf1edf36174020664", size = 221540, upload-time = "2026-03-17T10:31:35.445Z" },
+ { url = "https://files.pythonhosted.org/packages/23/d2/17879af479df7fbbd44bd528a31692a48f6b25055d16482fdf5cdb633805/coverage-7.13.5-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:0428cbef5783ad91fe240f673cc1f76b25e74bbfe1a13115e4aa30d3f538162d", size = 220262, upload-time = "2026-03-17T10:31:37.184Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/4c/d20e554f988c8f91d6a02c5118f9abbbf73a8768a3048cb4962230d5743f/coverage-7.13.5-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e0b216a19534b2427cc201a26c25da4a48633f29a487c61258643e89d28200c0", size = 220617, upload-time = "2026-03-17T10:31:39.245Z" },
+ { url = "https://files.pythonhosted.org/packages/29/9c/f9f5277b95184f764b24e7231e166dfdb5780a46d408a2ac665969416d61/coverage-7.13.5-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:972a9cd27894afe4bc2b1480107054e062df08e671df7c2f18c205e805ccd806", size = 261912, upload-time = "2026-03-17T10:31:41.324Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/f6/7f1ab39393eeb50cfe4747ae8ef0e4fc564b989225aa1152e13a180d74f8/coverage-7.13.5-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:4b59148601efcd2bac8c4dbf1f0ad6391693ccf7a74b8205781751637076aee3", size = 263987, upload-time = "2026-03-17T10:31:43.724Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/d7/62c084fb489ed9c6fbdf57e006752e7c516ea46fd690e5ed8b8617c7d52e/coverage-7.13.5-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:505d7083c8b0c87a8fa8c07370c285847c1f77739b22e299ad75a6af6c32c5c9", size = 266416, upload-time = "2026-03-17T10:31:45.769Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/f6/df63d8660e1a0bff6125947afda112a0502736f470d62ca68b288ea762d8/coverage-7.13.5-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:60365289c3741e4db327e7baff2a4aaacf22f788e80fa4683393891b70a89fbd", size = 267558, upload-time = "2026-03-17T10:31:48.293Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/02/353ca81d36779bd108f6d384425f7139ac3c58c750dcfaafe5d0bee6436b/coverage-7.13.5-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:1b88c69c8ef5d4b6fe7dea66d6636056a0f6a7527c440e890cf9259011f5e606", size = 261163, upload-time = "2026-03-17T10:31:50.125Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/16/2e79106d5749bcaf3aee6d309123548e3276517cd7851faa8da213bc61bf/coverage-7.13.5-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:5b13955d31d1633cf9376908089b7cebe7d15ddad7aeaabcbe969a595a97e95e", size = 263981, upload-time = "2026-03-17T10:31:51.961Z" },
+ { url = "https://files.pythonhosted.org/packages/29/c7/c29e0c59ffa6942030ae6f50b88ae49988e7e8da06de7ecdbf49c6d4feae/coverage-7.13.5-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:f70c9ab2595c56f81a89620e22899eea8b212a4041bd728ac6f4a28bf5d3ddd0", size = 261604, upload-time = "2026-03-17T10:31:53.872Z" },
+ { url = "https://files.pythonhosted.org/packages/40/48/097cdc3db342f34006a308ab41c3a7c11c3f0d84750d340f45d88a782e00/coverage-7.13.5-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:084b84a8c63e8d6fc7e3931b316a9bcafca1458d753c539db82d31ed20091a87", size = 265321, upload-time = "2026-03-17T10:31:55.997Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/1f/4994af354689e14fd03a75f8ec85a9a68d94e0188bbdab3fc1516b55e512/coverage-7.13.5-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:ad14385487393e386e2ea988b09d62dd42c397662ac2dabc3832d71253eee479", size = 260502, upload-time = "2026-03-17T10:31:58.308Z" },
+ { url = "https://files.pythonhosted.org/packages/22/c6/9bb9ef55903e628033560885f5c31aa227e46878118b63ab15dc7ba87797/coverage-7.13.5-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:7f2c47b36fe7709a6e83bfadf4eefb90bd25fbe4014d715224c4316f808e59a2", size = 262688, upload-time = "2026-03-17T10:32:00.141Z" },
+ { url = "https://files.pythonhosted.org/packages/14/4f/f5df9007e50b15e53e01edea486814783a7f019893733d9e4d6caad75557/coverage-7.13.5-cp313-cp313t-win32.whl", hash = "sha256:67e9bc5449801fad0e5dff329499fb090ba4c5800b86805c80617b4e29809b2a", size = 222788, upload-time = "2026-03-17T10:32:02.246Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/98/aa7fccaa97d0f3192bec013c4e6fd6d294a6ed44b640e6bb61f479e00ed5/coverage-7.13.5-cp313-cp313t-win_amd64.whl", hash = "sha256:da86cdcf10d2519e10cabb8ac2de03da1bcb6e4853790b7fbd48523332e3a819", size = 223851, upload-time = "2026-03-17T10:32:04.416Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/8b/e5c469f7352651e5f013198e9e21f97510b23de957dd06a84071683b4b60/coverage-7.13.5-cp313-cp313t-win_arm64.whl", hash = "sha256:0ecf12ecb326fe2c339d93fc131816f3a7367d223db37817208905c89bded911", size = 222104, upload-time = "2026-03-17T10:32:06.65Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/77/39703f0d1d4b478bfd30191d3c14f53caf596fac00efb3f8f6ee23646439/coverage-7.13.5-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:fbabfaceaeb587e16f7008f7795cd80d20ec548dc7f94fbb0d4ec2e038ce563f", size = 219621, upload-time = "2026-03-17T10:32:08.589Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/3e/51dff36d99ae14639a133d9b164d63e628532e2974d8b1edb99dd1ebc733/coverage-7.13.5-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:9bb2a28101a443669a423b665939381084412b81c3f8c0fcfbac57f4e30b5b8e", size = 219953, upload-time = "2026-03-17T10:32:10.507Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/6c/1f1917b01eb647c2f2adc9962bd66c79eb978951cab61bdc1acab3290c07/coverage-7.13.5-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:bd3a2fbc1c6cccb3c5106140d87cc6a8715110373ef42b63cf5aea29df8c217a", size = 250992, upload-time = "2026-03-17T10:32:12.41Z" },
+ { url = "https://files.pythonhosted.org/packages/22/e5/06b1f88f42a5a99df42ce61208bdec3bddb3d261412874280a19796fc09c/coverage-7.13.5-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:6c36ddb64ed9d7e496028d1d00dfec3e428e0aabf4006583bb1839958d280510", size = 253503, upload-time = "2026-03-17T10:32:14.449Z" },
+ { url = "https://files.pythonhosted.org/packages/80/28/2a148a51e5907e504fa7b85490277734e6771d8844ebcc48764a15e28155/coverage-7.13.5-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:380e8e9084d8eb38db3a9176a1a4f3c0082c3806fa0dc882d1d87abc3c789247", size = 254852, upload-time = "2026-03-17T10:32:16.56Z" },
+ { url = "https://files.pythonhosted.org/packages/61/77/50e8d3d85cc0b7ebe09f30f151d670e302c7ff4a1bf6243f71dd8b0981fa/coverage-7.13.5-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e808af52a0513762df4d945ea164a24b37f2f518cbe97e03deaa0ee66139b4d6", size = 257161, upload-time = "2026-03-17T10:32:19.004Z" },
+ { url = "https://files.pythonhosted.org/packages/3b/c4/b5fd1d4b7bf8d0e75d997afd3925c59ba629fc8616f1b3aae7605132e256/coverage-7.13.5-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e301d30dd7e95ae068671d746ba8c34e945a82682e62918e41b2679acd2051a0", size = 251021, upload-time = "2026-03-17T10:32:21.344Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/66/6ea21f910e92d69ef0b1c3346ea5922a51bad4446c9126db2ae96ee24c4c/coverage-7.13.5-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:800bc829053c80d240a687ceeb927a94fd108bbdc68dfbe505d0d75ab578a882", size = 252858, upload-time = "2026-03-17T10:32:23.506Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/ea/879c83cb5d61aa2a35fb80e72715e92672daef8191b84911a643f533840c/coverage-7.13.5-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:0b67af5492adb31940ee418a5a655c28e48165da5afab8c7fa6fd72a142f8740", size = 250823, upload-time = "2026-03-17T10:32:25.516Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/fb/616d95d3adb88b9803b275580bdeee8bd1b69a886d057652521f83d7322f/coverage-7.13.5-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:c9136ff29c3a91e25b1d1552b5308e53a1e0653a23e53b6366d7c2dcbbaf8a16", size = 255099, upload-time = "2026-03-17T10:32:27.944Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/93/25e6917c90ec1c9a56b0b26f6cad6408e5f13bb6b35d484a0d75c9cf000d/coverage-7.13.5-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:cff784eef7f0b8f6cb28804fbddcfa99f89efe4cc35fb5627e3ac58f91ed3ac0", size = 250638, upload-time = "2026-03-17T10:32:29.914Z" },
+ { url = "https://files.pythonhosted.org/packages/fc/7b/dc1776b0464145a929deed214aef9fb1493f159b59ff3c7eeeedf91eddd0/coverage-7.13.5-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:68a4953be99b17ac3c23b6efbc8a38330d99680c9458927491d18700ef23ded0", size = 252295, upload-time = "2026-03-17T10:32:31.981Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/fb/99cbbc56a26e07762a2740713f3c8f9f3f3106e3a3dd8cc4474954bccd34/coverage-7.13.5-cp314-cp314-win32.whl", hash = "sha256:35a31f2b1578185fbe6aa2e74cea1b1d0bbf4c552774247d9160d29b80ed56cc", size = 222360, upload-time = "2026-03-17T10:32:34.233Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/b7/4758d4f73fb536347cc5e4ad63662f9d60ba9118cb6785e9616b2ce5d7fa/coverage-7.13.5-cp314-cp314-win_amd64.whl", hash = "sha256:2aa055ae1857258f9e0045be26a6d62bdb47a72448b62d7b55f4820f361a2633", size = 223174, upload-time = "2026-03-17T10:32:36.369Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/f2/24d84e1dfe70f8ac9fdf30d338239860d0d1d5da0bda528959d0ebc9da28/coverage-7.13.5-cp314-cp314-win_arm64.whl", hash = "sha256:1b11eef33edeae9d142f9b4358edb76273b3bfd30bc3df9a4f95d0e49caf94e8", size = 221739, upload-time = "2026-03-17T10:32:38.736Z" },
+ { url = "https://files.pythonhosted.org/packages/60/5b/4a168591057b3668c2428bff25dd3ebc21b629d666d90bcdfa0217940e84/coverage-7.13.5-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:10a0c37f0b646eaff7cce1874c31d1f1ccb297688d4c747291f4f4c70741cc8b", size = 220351, upload-time = "2026-03-17T10:32:41.196Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/21/1fd5c4dbfe4a58b6b99649125635df46decdfd4a784c3cd6d410d303e370/coverage-7.13.5-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:b5db73ba3c41c7008037fa731ad5459fc3944cb7452fc0aa9f822ad3533c583c", size = 220612, upload-time = "2026-03-17T10:32:43.204Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/fe/2a924b3055a5e7e4512655a9d4609781b0d62334fa0140c3e742926834e2/coverage-7.13.5-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:750db93a81e3e5a9831b534be7b1229df848b2e125a604fe6651e48aa070e5f9", size = 261985, upload-time = "2026-03-17T10:32:45.514Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/0d/c8928f2bd518c45990fe1a2ab8db42e914ef9b726c975facc4282578c3eb/coverage-7.13.5-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9ddb4f4a5479f2539644be484da179b653273bca1a323947d48ab107b3ed1f29", size = 264107, upload-time = "2026-03-17T10:32:47.971Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/ae/4ae35bbd9a0af9d820362751f0766582833c211224b38665c0f8de3d487f/coverage-7.13.5-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d8a7a2049c14f413163e2bdabd37e41179b1d1ccb10ffc6ccc4b7a718429c607", size = 266513, upload-time = "2026-03-17T10:32:50.1Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/20/d326174c55af36f74eac6ae781612d9492f060ce8244b570bb9d50d9d609/coverage-7.13.5-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e1c85e0b6c05c592ea6d8768a66a254bfb3874b53774b12d4c89c481eb78cb90", size = 267650, upload-time = "2026-03-17T10:32:52.391Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/5e/31484d62cbd0eabd3412e30d74386ece4a0837d4f6c3040a653878bfc019/coverage-7.13.5-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:777c4d1eff1b67876139d24288aaf1817f6c03d6bae9c5cc8d27b83bcfe38fe3", size = 261089, upload-time = "2026-03-17T10:32:54.544Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/d8/49a72d6de146eebb0b7e48cc0f4bc2c0dd858e3d4790ab2b39a2872b62bd/coverage-7.13.5-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:6697e29b93707167687543480a40f0db8f356e86d9f67ddf2e37e2dfd91a9dab", size = 263982, upload-time = "2026-03-17T10:32:56.803Z" },
+ { url = "https://files.pythonhosted.org/packages/06/3b/0351f1bd566e6e4dd39e978efe7958bde1d32f879e85589de147654f57bb/coverage-7.13.5-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:8fdf453a942c3e4d99bd80088141c4c6960bb232c409d9c3558e2dbaa3998562", size = 261579, upload-time = "2026-03-17T10:32:59.466Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/ce/796a2a2f4017f554d7810f5c573449b35b1e46788424a548d4d19201b222/coverage-7.13.5-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:32ca0c0114c9834a43f045a87dcebd69d108d8ffb666957ea65aa132f50332e2", size = 265316, upload-time = "2026-03-17T10:33:01.847Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/16/d5ae91455541d1a78bc90abf495be600588aff8f6db5c8b0dae739fa39c9/coverage-7.13.5-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:8769751c10f339021e2638cd354e13adeac54004d1941119b2c96fe5276d45ea", size = 260427, upload-time = "2026-03-17T10:33:03.945Z" },
+ { url = "https://files.pythonhosted.org/packages/48/11/07f413dba62db21fb3fad5d0de013a50e073cc4e2dc4306e770360f6dfc8/coverage-7.13.5-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:cec2d83125531bd153175354055cdb7a09987af08a9430bd173c937c6d0fba2a", size = 262745, upload-time = "2026-03-17T10:33:06.285Z" },
+ { url = "https://files.pythonhosted.org/packages/91/15/d792371332eb4663115becf4bad47e047d16234b1aff687b1b18c58d60ae/coverage-7.13.5-cp314-cp314t-win32.whl", hash = "sha256:0cd9ed7a8b181775459296e402ca4fb27db1279740a24e93b3b41942ebe4b215", size = 223146, upload-time = "2026-03-17T10:33:08.756Z" },
+ { url = "https://files.pythonhosted.org/packages/db/51/37221f59a111dca5e85be7dbf09696323b5b9f13ff65e0641d535ed06ea8/coverage-7.13.5-cp314-cp314t-win_amd64.whl", hash = "sha256:301e3b7dfefecaca37c9f1aa6f0049b7d4ab8dd933742b607765d757aca77d43", size = 224254, upload-time = "2026-03-17T10:33:11.174Z" },
+ { url = "https://files.pythonhosted.org/packages/54/83/6acacc889de8987441aa7d5adfbdbf33d288dad28704a67e574f1df9bcbb/coverage-7.13.5-cp314-cp314t-win_arm64.whl", hash = "sha256:9dacc2ad679b292709e0f5fc1ac74a6d4d5562e424058962c7bb0c658ad25e45", size = 222276, upload-time = "2026-03-17T10:33:13.466Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/ee/a4cf96b8ce1e566ed238f0659ac2d3f007ed1d14b181bcb684e19561a69a/coverage-7.13.5-py3-none-any.whl", hash = "sha256:34b02417cf070e173989b3db962f7ed56d2f644307b2cf9d5a0f258e13084a61", size = 211346, upload-time = "2026-03-17T10:33:15.691Z" },
]
[[package]]
@@ -1012,7 +909,7 @@ name = "cryptography"
version = "42.0.8"
source = { registry = "https://pypi.org/simple" }
dependencies = [
- { name = "cffi", marker = "platform_python_implementation != 'PyPy' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "cffi", marker = "platform_python_implementation != 'PyPy'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/93/a7/1498799a2ea06148463a9a2c10ab2f6a921a74fb19e231b27dc412a748e2/cryptography-42.0.8.tar.gz", hash = "sha256:8d09d05439ce7baa8e9e95b07ec5b6c886f548deb7e0f69ef25f64b3bce842f2", size = 671250, upload-time = "2024-06-04T19:55:08.609Z" }
wheels = [
@@ -1039,58 +936,91 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/fd/2b/be327b580645927bb1a1f32d5a175b897a9b956bc085b095e15c40bac9ed/cryptography-42.0.8-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b297f90c5723d04bcc8265fc2a0f86d4ea2e0f7ab4b6994459548d3a6b992a14", size = 3938751, upload-time = "2024-06-04T19:54:37.837Z" },
{ url = "https://files.pythonhosted.org/packages/3c/d5/c6a78ffccdbe4516711ebaa9ed2c7eb6ac5dfa3dc920f2c7e920af2418b0/cryptography-42.0.8-cp39-abi3-win32.whl", hash = "sha256:2f88d197e66c65be5e42cd72e5c18afbfae3f741742070e3019ac8f4ac57262c", size = 2439281, upload-time = "2024-06-04T19:53:55.903Z" },
{ url = "https://files.pythonhosted.org/packages/a2/7b/b0d330852dd5953daee6b15f742f15d9f18e9c0154eb4cfcc8718f0436da/cryptography-42.0.8-cp39-abi3-win_amd64.whl", hash = "sha256:fa76fbb7596cc5839320000cdd5d0955313696d9511debab7ee7278fc8b5c84a", size = 2886038, upload-time = "2024-06-04T19:54:18.707Z" },
- { url = "https://files.pythonhosted.org/packages/a3/fe/1e21699f0a7904e8a30d4fc6db262958f1edf5e505a02e7d97a5b419e482/cryptography-42.0.8-pp310-pypy310_pp73-macosx_10_12_x86_64.whl", hash = "sha256:ba4f0a211697362e89ad822e667d8d340b4d8d55fae72cdd619389fb5912eefe", size = 3014449, upload-time = "2024-06-04T19:54:40.379Z" },
- { url = "https://files.pythonhosted.org/packages/d5/f3/61b398b5ec61f4b6ffbf746227df7ebb421696458d9625d634043f236a13/cryptography-42.0.8-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:81884c4d096c272f00aeb1f11cf62ccd39763581645b0812e99a91505fa48e0c", size = 3558533, upload-time = "2024-06-04T19:54:42.123Z" },
- { url = "https://files.pythonhosted.org/packages/c1/e2/60b05e720766e185ef097d07068bd878a51d613ef91e4c241750f9c9192b/cryptography-42.0.8-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:c9bb2ae11bfbab395bdd072985abde58ea9860ed84e59dbc0463a5d0159f5b71", size = 3759330, upload-time = "2024-06-04T19:54:09.258Z" },
- { url = "https://files.pythonhosted.org/packages/10/38/2c8dae407d301eaf942e377a5b2b30485cfa0df03c6c2dcc2ac044054ed9/cryptography-42.0.8-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:7016f837e15b0a1c119d27ecd89b3515f01f90a8615ed5e9427e30d9cdbfed3d", size = 2801764, upload-time = "2024-06-04T19:54:25.455Z" },
]
[[package]]
name = "cuda-bindings"
-version = "13.1.1"
+version = "13.2.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "cuda-pathfinder" },
]
wheels = [
- { url = "https://files.pythonhosted.org/packages/60/63/579402b642f5b9b8ceb79e456b39b5771f27e132a8af3b140e54d69790fc/cuda_bindings-13.1.1-cp310-cp310-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4400370a83f1538e25ed4c18c34a0e9d5fad39741e282e69ce24d1479a11017d", size = 15777291, upload-time = "2025-12-09T22:05:41.109Z" },
- { url = "https://files.pythonhosted.org/packages/df/6a/3a293cfb01cd4964444a0f75917b6edb1c31ea69d0230e329975da6991ba/cuda_bindings-13.1.1-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:81f92500e2f6aec2dac00a5a1ce77d5aa77ea77b606dc484d951f1f2cc3eaa13", size = 16311623, upload-time = "2025-12-09T22:05:43.897Z" },
- { url = "https://files.pythonhosted.org/packages/72/b8/a5860b9e70faa53658236dc61efc3ecc51846beff4a0b73de9151130ff98/cuda_bindings-13.1.1-cp310-cp310-win_amd64.whl", hash = "sha256:3f5bb8190267216f96597235252087accac4cbccefd1b60756cced114b2d6754", size = 15185932, upload-time = "2025-12-09T22:05:46.089Z" },
- { url = "https://files.pythonhosted.org/packages/b0/58/b8d4c7c5fb29ba46088a7e78d1065484219f8fe41a08adc4a85b1ee56149/cuda_bindings-13.1.1-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a5f5a6ade0ad45096568bc4dd1eb3377b65884d29124338fe9a4353130ef6631", size = 15771605, upload-time = "2025-12-09T22:05:48.266Z" },
- { url = "https://files.pythonhosted.org/packages/17/af/710403f76f2d608d483d87089465e1f666351641dbd73d19bd025e652bad/cuda_bindings-13.1.1-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9348f69b03b257f07159dd4c869615e139722c2bd81e96c66f6b8f77615efd82", size = 16338970, upload-time = "2025-12-09T22:05:50.598Z" },
- { url = "https://files.pythonhosted.org/packages/64/1c/e7ea27d4cb7d07331c88e3bbed3cacc947d2237471801086c7447b3e195d/cuda_bindings-13.1.1-cp311-cp311-win_amd64.whl", hash = "sha256:ec33b84f4bd65a86a734427f2b9cb8f221bedab2c4cfb681488cabc82f1d64ab", size = 15210672, upload-time = "2025-12-09T22:05:53.369Z" },
- { url = "https://files.pythonhosted.org/packages/53/3d/c8ed9d169843091f3f0d6b8218e826fd59520a37e0434c204feada597988/cuda_bindings-13.1.1-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1e75ad0cb863330df784236d289612d71ca855c013d19ae00e5693574abd6915", size = 15530160, upload-time = "2025-12-09T22:05:55.386Z" },
- { url = "https://files.pythonhosted.org/packages/4a/8e/368295623ee43fba622909d780fbb6863efc1638dff55f67a0f04eac6470/cuda_bindings-13.1.1-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:25785d1a3cdcd98f151240fd5efd025609319a6720a217dee2a929241749d488", size = 16110386, upload-time = "2025-12-09T22:05:57.71Z" },
- { url = "https://files.pythonhosted.org/packages/60/1f/ecc4701ade3e85f091c625a920574527b9daf7fb354189fbfbc5516af6cd/cuda_bindings-13.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:ccde9c95c0e953b31fe7731bb08da9d0a34b1770498df9a3c156fdfdbe3951ad", size = 15250028, upload-time = "2025-12-09T22:06:00.346Z" },
- { url = "https://files.pythonhosted.org/packages/fe/c1/0ee8fd94bab7e23116e0e3da8c0902e299f3d9edc95f1d7d8ef894c897ed/cuda_bindings-13.1.1-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0c9822a57c8f952dc367aacd7c32fe4cb17371104383606f455ea74635bff4c7", size = 15421116, upload-time = "2025-12-09T22:06:02.994Z" },
- { url = "https://files.pythonhosted.org/packages/f3/c2/f272fad414b96299e010dcbe510cf17fc25deaf3443e0fdb55020a8298a3/cuda_bindings-13.1.1-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5837f5ea422c5653626dcfe22e9ab68142cd19af9e67a226100f224cc25a1b99", size = 15940152, upload-time = "2025-12-09T22:06:05.079Z" },
- { url = "https://files.pythonhosted.org/packages/2a/56/433093bec0121f031edb582ea3a72f71031e8fbebecaaf329809344da4c7/cuda_bindings-13.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:9e4f348cd7a779657d51e6f71aac3965fb1738f40ff3bbe75265a3242fd6f29f", size = 15216463, upload-time = "2025-12-09T22:06:07.296Z" },
- { url = "https://files.pythonhosted.org/packages/de/38/40416d037ed25db68f1dbd50e0232775a62d90c9f25af22b196c0a13b88c/cuda_bindings-13.1.1-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:86258fe1b0d3998bea7f57dc891569e4996705b8dd00366e44c722d0a29b2090", size = 15498927, upload-time = "2025-12-09T22:06:09.476Z" },
- { url = "https://files.pythonhosted.org/packages/ac/3f/f1f88b6cdb7d41ba076f8ff10edf6d3bd17e740da9a163544b43d6349653/cuda_bindings-13.1.1-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:daf8468fd603b2724c2d16cbd499348c64916ed72b1d04643f1660ce13cd12ae", size = 15984539, upload-time = "2025-12-09T22:06:11.882Z" },
- { url = "https://files.pythonhosted.org/packages/f6/33/7739cc5e9a3373df8e7dea9060528bee5f70cf6e28b9c14f765502816c71/cuda_bindings-13.1.1-cp314-cp314-win_amd64.whl", hash = "sha256:f2e079182014dbc162562b46467815272c14c7afe5b988978fa968728b0ac726", size = 15373212, upload-time = "2025-12-09T22:06:13.989Z" },
- { url = "https://files.pythonhosted.org/packages/9e/0a/5c6d514e566ff86c4054bbbb6554bf49b9c55fefbc934eb456faecab53c9/cuda_bindings-13.1.1-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d0cd96a6ec00a78235947bff9462b2139bc5b83ce8e297d865802f0b52d1e23d", size = 15403944, upload-time = "2025-12-09T22:06:16.315Z" },
- { url = "https://files.pythonhosted.org/packages/0b/5b/319cfa491a685d4d4757aa24223b6dbc0976954afac42f49fc47290ba6a3/cuda_bindings-13.1.1-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9ff465829c6c394c2b4047250324a19925cf8c44633345b2746a4741e07bf827", size = 15911462, upload-time = "2025-12-09T22:06:18.403Z" },
- { url = "https://files.pythonhosted.org/packages/e3/5c/38b92080c5b6c4ddb09f0be2536123f81c7e9e1a89e4573f20cb00347ee3/cuda_bindings-13.1.1-cp314-cp314t-win_amd64.whl", hash = "sha256:8205eee6b8b458a2110c0384923ace206855d0f1b436fc1b145fcbaa1653b501", size = 16044390, upload-time = "2025-12-09T22:06:20.945Z" },
+ { url = "https://files.pythonhosted.org/packages/52/c8/b2589d68acf7e3d63e2be330b84bc25712e97ed799affbca7edd7eae25d6/cuda_bindings-13.2.0-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e865447abfb83d6a98ad5130ed3c70b1fc295ae3eeee39fd07b4ddb0671b6788", size = 5722404, upload-time = "2026-03-11T00:12:44.041Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/92/f899f7bbb5617bb65ec52a6eac1e9a1447a86b916c4194f8a5001b8cde0c/cuda_bindings-13.2.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:46d8776a55d6d5da9dd6e9858fba2efcda2abe6743871dee47dd06eb8cb6d955", size = 6320619, upload-time = "2026-03-11T00:12:45.939Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/a5/d7f01a415e134546248cef612adad8153c9f1eb10ec79505a7cd8294370b/cuda_bindings-13.2.0-cp312-cp312-win_amd64.whl", hash = "sha256:45815daeb595bf3b405c52671a2542b1f8e9329f3b029494acbfcc74aeaa1f2d", size = 5840830, upload-time = "2026-03-11T00:12:48.43Z" },
+ { url = "https://files.pythonhosted.org/packages/df/93/eef988860a3ca985f82c4f3174fc0cdd94e07331ba9a92e8e064c260337f/cuda_bindings-13.2.0-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6629ca2df6f795b784752409bcaedbd22a7a651b74b56a165ebc0c9dcbd504d0", size = 5614610, upload-time = "2026-03-11T00:12:50.337Z" },
+ { url = "https://files.pythonhosted.org/packages/18/23/6db3aba46864aee357ab2415135b3fe3da7e9f1fa0221fa2a86a5968099c/cuda_bindings-13.2.0-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7dca0da053d3b4cc4869eff49c61c03f3c5dbaa0bcd712317a358d5b8f3f385d", size = 6149914, upload-time = "2026-03-11T00:12:52.374Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/84/d3b6220b51cbc02ca14db7387e97445126b4ff5125aaa6c5dd7dcb75e679/cuda_bindings-13.2.0-cp313-cp313-win_amd64.whl", hash = "sha256:8cebe3ce4aeeca5af9c490e175f76c4b569bbf4a35a62294b777bc77bf7ac4d8", size = 5796512, upload-time = "2026-03-11T00:12:54.483Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/87/87a014f045b77c6de5c8527b0757fe644417b184e5367db977236a141602/cuda_bindings-13.2.0-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a6464b30f46692d6c7f65d4a0e0450d81dd29de3afc1bb515653973d01c2cd6e", size = 5685673, upload-time = "2026-03-11T00:12:56.371Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/5e/c0fe77a73aaefd3fff25ffaccaac69c5a63eafdf8b9a4c476626ef0ac703/cuda_bindings-13.2.0-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f4af9f3e1be603fa12d5ad6cfca7844c9d230befa9792b5abdf7dd79979c3626", size = 6191386, upload-time = "2026-03-11T00:12:58.965Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/73/98bcb069778fe420226db75aff54b5dd6c3ecfd0912edabab723326e80b7/cuda_bindings-13.2.0-cp314-cp314-win_amd64.whl", hash = "sha256:bd658bb5c0e55b7b3e5dd0ed509c6addb298c665db26a9bfba35e1e626000ba2", size = 5938605, upload-time = "2026-03-11T00:13:01.639Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/58/ed2c3b39c8dd5f96aa7a4abef0d47a73932c7a988e30f5fa428f00ed0da1/cuda_bindings-13.2.0-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:df850a1ff8ce1b3385257b08e47b70e959932f5f432d0a4e46a355962b4e4771", size = 5507469, upload-time = "2026-03-11T00:13:04.063Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/01/0c941b112ceeb21439b05895eace78ca1aa2eaaf695c8521a068fd9b4c00/cuda_bindings-13.2.0-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e8a16384c6494e5485f39314b0b4afb04bee48d49edb16d5d8593fd35bbd231b", size = 6059693, upload-time = "2026-03-11T00:13:06.003Z" },
+ { url = "https://files.pythonhosted.org/packages/52/49/4e01cc06447d39476e138d1b1adec8d35c0d04eccd2c8d69befc08cd66e8/cuda_bindings-13.2.0-cp314-cp314t-win_amd64.whl", hash = "sha256:6ccf14e0c1def3b7200100aafff3a9f7e210ecb6e409329e92dcf6cd2c00d5c7", size = 6662637, upload-time = "2026-03-11T00:13:07.881Z" },
]
[[package]]
name = "cuda-pathfinder"
-version = "1.3.3"
+version = "1.5.0"
source = { registry = "https://pypi.org/simple" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/0b/02/4dbe7568a42e46582248942f54dc64ad094769532adbe21e525e4edf7bc4/cuda_pathfinder-1.3.3-py3-none-any.whl", hash = "sha256:9984b664e404f7c134954a771be8775dfd6180ea1e1aef4a5a37d4be05d9bbb1", size = 27154, upload-time = "2025-12-04T22:35:08.996Z" },
+ { url = "https://files.pythonhosted.org/packages/93/66/0c02bd330e7d976f83fa68583d6198d76f23581bcbb5c0e98a6148f326e5/cuda_pathfinder-1.5.0-py3-none-any.whl", hash = "sha256:498f90a9e9de36044a7924742aecce11c50c49f735f1bc53e05aa46de9ea4110", size = 49739, upload-time = "2026-03-24T21:14:30.869Z" },
]
[[package]]
name = "cuda-python"
-version = "13.1.1"
+version = "13.2.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "cuda-bindings" },
{ name = "cuda-pathfinder" },
]
wheels = [
- { url = "https://files.pythonhosted.org/packages/cd/08/b5e3b9822662d72d540d830531e3ab6a7cabbda3dd56175696aabccfeb76/cuda_python-13.1.1-py3-none-any.whl", hash = "sha256:944cc4fe6482673d28dd545797a28840945a1668739328fa2ad1e9be4f7050d9", size = 8038, upload-time = "2025-12-09T22:13:10.719Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/da/b4dbe129f941afe1c24a09ba53521b78875626763d96414798a74763282f/cuda_python-13.2.0-py3-none-any.whl", hash = "sha256:2f092b0ec13a860115fa595411889ee939ad203450ea4f91e9461b174ea7b084", size = 8145, upload-time = "2026-03-11T13:55:19.143Z" },
+]
+
+[[package]]
+name = "cuda-toolkit"
+version = "13.0.2"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/57/b2/453099f5f3b698d7d0eab38916aac44c7f76229f451709e2eb9db6615dcd/cuda_toolkit-13.0.2-py2.py3-none-any.whl", hash = "sha256:b198824cf2f54003f50d64ada3a0f184b42ca0846c1c94192fa269ecd97a66eb", size = 2364, upload-time = "2025-12-19T23:24:07.328Z" },
+]
+
+[package.optional-dependencies]
+cublas = [
+ { name = "nvidia-cublas", marker = "sys_platform == 'linux'" },
+]
+cudart = [
+ { name = "nvidia-cuda-runtime", marker = "sys_platform == 'linux'" },
+]
+cufft = [
+ { name = "nvidia-cufft", marker = "sys_platform == 'linux'" },
+]
+cufile = [
+ { name = "nvidia-cufile", marker = "sys_platform == 'linux'" },
+]
+cupti = [
+ { name = "nvidia-cuda-cupti", marker = "sys_platform == 'linux'" },
+]
+curand = [
+ { name = "nvidia-curand", marker = "sys_platform == 'linux'" },
+]
+cusolver = [
+ { name = "nvidia-cusolver", marker = "sys_platform == 'linux'" },
+]
+cusparse = [
+ { name = "nvidia-cusparse", marker = "sys_platform == 'linux'" },
+]
+nvjitlink = [
+ { name = "nvidia-nvjitlink", marker = "sys_platform == 'linux'" },
+]
+nvrtc = [
+ { name = "nvidia-cuda-nvrtc", marker = "sys_platform == 'linux'" },
+]
+nvtx = [
+ { name = "nvidia-nvtx", marker = "sys_platform == 'linux'" },
]
[[package]]
@@ -1099,14 +1029,6 @@ version = "3.2.4"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/91/85/7574c9cd44b69a27210444b6650f6477f56c75fee1b70d7672d3e4166167/cython-3.2.4.tar.gz", hash = "sha256:84226ecd313b233da27dc2eb3601b4f222b8209c3a7216d8733b031da1dc64e6", size = 3280291, upload-time = "2026-01-04T14:14:14.473Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/a1/10/720e0fb84eab4c927c4dd6b61eb7993f7732dd83d29ba6d73083874eade9/cython-3.2.4-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:02cb0cc0f23b9874ad262d7d2b9560aed9c7e2df07b49b920bda6f2cc9cb505e", size = 2960836, upload-time = "2026-01-04T14:14:51.103Z" },
- { url = "https://files.pythonhosted.org/packages/7d/3d/b26f29092c71c36e0462752885bdfb18c23c176af4de953fdae2772a8941/cython-3.2.4-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f136f379a4a54246facd0eb6f1ee15c3837cb314ce87b677582ec014db4c6845", size = 3370134, upload-time = "2026-01-04T14:14:53.627Z" },
- { url = "https://files.pythonhosted.org/packages/56/9e/539fb0d09e4f5251b5b14f8daf77e71fee021527f1013791038234618b6b/cython-3.2.4-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:35ab0632186057406ec729374c737c37051d2eacad9d515d94e5a3b3e58a9b02", size = 3537552, upload-time = "2026-01-04T14:14:56.852Z" },
- { url = "https://files.pythonhosted.org/packages/10/c6/82d19a451c050d1be0f05b1a3302267463d391db548f013ee88b5348a8e9/cython-3.2.4-cp310-cp310-win_amd64.whl", hash = "sha256:ca2399dc75796b785f74fb85c938254fa10c80272004d573c455f9123eceed86", size = 2766191, upload-time = "2026-01-04T14:14:58.709Z" },
- { url = "https://files.pythonhosted.org/packages/85/cc/8f06145ec3efa121c8b1b67f06a640386ddacd77ee3e574da582a21b14ee/cython-3.2.4-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:ff9af2134c05e3734064808db95b4dd7341a39af06e8945d05ea358e1741aaed", size = 2953769, upload-time = "2026-01-04T14:15:00.361Z" },
- { url = "https://files.pythonhosted.org/packages/55/b0/706cf830eddd831666208af1b3058c2e0758ae157590909c1f634b53bed9/cython-3.2.4-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:67922c9de058a0bfb72d2e75222c52d09395614108c68a76d9800f150296ddb3", size = 3243841, upload-time = "2026-01-04T14:15:02.066Z" },
- { url = "https://files.pythonhosted.org/packages/ac/25/58893afd4ef45f79e3d4db82742fa4ff874b936d67a83c92939053920ccd/cython-3.2.4-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b362819d155fff1482575e804e43e3a8825332d32baa15245f4642022664a3f4", size = 3378083, upload-time = "2026-01-04T14:15:04.248Z" },
- { url = "https://files.pythonhosted.org/packages/32/e4/424a004d7c0d8a4050c81846ebbd22272ececfa9a498cb340aa44fccbec2/cython-3.2.4-cp311-cp311-win_amd64.whl", hash = "sha256:1a64a112a34ec719b47c01395647e54fb4cf088a511613f9a3a5196694e8e382", size = 2769990, upload-time = "2026-01-04T14:15:06.53Z" },
{ url = "https://files.pythonhosted.org/packages/91/4d/1eb0c7c196a136b1926f4d7f0492a96c6fabd604d77e6cd43b56a3a16d83/cython-3.2.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:64d7f71be3dd6d6d4a4c575bb3a4674ea06d1e1e5e4cd1b9882a2bc40ed3c4c9", size = 2970064, upload-time = "2026-01-04T14:15:08.567Z" },
{ url = "https://files.pythonhosted.org/packages/03/1c/46e34b08bea19a1cdd1e938a4c123e6299241074642db9d81983cef95e9f/cython-3.2.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:869487ea41d004f8b92171f42271fbfadb1ec03bede3158705d16cd570d6b891", size = 3226757, upload-time = "2026-01-04T14:15:10.812Z" },
{ url = "https://files.pythonhosted.org/packages/12/33/3298a44d201c45bcf0d769659725ae70e9c6c42adf8032f6d89c8241098d/cython-3.2.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:55b6c44cd30821f0b25220ceba6fe636ede48981d2a41b9bbfe3c7902ce44ea7", size = 3388969, upload-time = "2026-01-04T14:15:12.45Z" },
@@ -1133,28 +1055,30 @@ wheels = [
[[package]]
name = "datasets"
-version = "4.4.2"
+version = "4.8.4"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "dill" },
{ name = "filelock" },
- { name = "fsspec", extra = ["http"], marker = "extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
+ { name = "fsspec", version = "2026.2.0", source = { registry = "https://pypi.org/simple" }, extra = ["http"], marker = "extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
{ name = "httpx" },
{ name = "huggingface-hub" },
{ name = "multiprocess" },
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version < '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version >= '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "packaging" },
- { name = "pandas" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
+ { name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "pandas", version = "3.0.1", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
{ name = "pyarrow" },
{ name = "pyyaml" },
{ name = "requests" },
{ name = "tqdm" },
{ name = "xxhash" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/c4/54/9359803da96bc65439a28fbb014dc2c90b7d4d8034a93b72362b0d40191f/datasets-4.4.2.tar.gz", hash = "sha256:9de16e415c4ba4713eac0493f7c7dc74f3aa21599297f00cc6ddab409cb7b24b", size = 586474, upload-time = "2025-12-19T15:03:09.129Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/22/22/73e46ac7a8c25e7ef0b3bd6f10da3465021d90219a32eb0b4d2afea4c56e/datasets-4.8.4.tar.gz", hash = "sha256:a1429ed853275ce7943a01c6d2e25475b4501eb758934362106a280470df3a52", size = 604382, upload-time = "2026-03-23T14:21:17.987Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/7b/b5/fefa518c809de7bced5cddb7c21c010da66fa2ae494bda96844a280cc6ce/datasets-4.4.2-py3-none-any.whl", hash = "sha256:6f5ef3417504d9cd663c71c1b90b9a494ff4c2076a2cd6a6e40ceee6ad95befc", size = 512268, upload-time = "2025-12-19T15:03:07.087Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/e5/247d094108e42ac26363ab8dc57f168840cf7c05774b40ffeb0d78868fcc/datasets-4.8.4-py3-none-any.whl", hash = "sha256:cdc8bee4698e549d78bf1fed6aea2eebc760b22b084f07e6fc020c6577a6ce6d", size = 526991, upload-time = "2026-03-23T14:21:15.89Z" },
]
[[package]]
@@ -1189,11 +1113,29 @@ wheels = [
[[package]]
name = "dill"
+version = "0.4.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/81/e1/56027a71e31b02ddc53c7d65b01e68edf64dea2932122fe7746a516f75d5/dill-0.4.1.tar.gz", hash = "sha256:423092df4182177d4d8ba8290c8a5b640c66ab35ec7da59ccfa00f6fa3eea5fa", size = 187315, upload-time = "2026-01-19T02:36:56.85Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1e/77/dc8c558f7593132cf8fefec57c4f60c83b16941c574ac5f619abb3ae7933/dill-0.4.1-py3-none-any.whl", hash = "sha256:1e1ce33e978ae97fcfcff5638477032b801c46c7c65cf717f95fbc2248f79a9d", size = 120019, upload-time = "2026-01-19T02:36:55.663Z" },
+]
+
+[[package]]
+name = "distlib"
version = "0.4.0"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/12/80/630b4b88364e9a8c8c5797f4602d0f76ef820909ee32f0bacb9f90654042/dill-0.4.0.tar.gz", hash = "sha256:0633f1d2df477324f53a895b02c901fb961bdbf65a17122586ea7019292cbcf0", size = 186976, upload-time = "2025-04-16T00:41:48.867Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/96/8e/709914eb2b5749865801041647dc7f4e6d00b549cfe88b65ca192995f07c/distlib-0.4.0.tar.gz", hash = "sha256:feec40075be03a04501a973d81f633735b4b69f98b05450592310c0f401a4e0d", size = 614605, upload-time = "2025-07-17T16:52:00.465Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/33/6b/e0547afaf41bf2c42e52430072fa5658766e3d65bd4b03a563d1b6336f57/distlib-0.4.0-py2.py3-none-any.whl", hash = "sha256:9659f7d87e46584a30b5780e43ac7a2143098441670ff0a49d5f9034c54a6c16", size = 469047, upload-time = "2025-07-17T16:51:58.613Z" },
+]
+
+[[package]]
+name = "distro"
+version = "1.9.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/fc/f8/98eea607f65de6527f8a2e8885fc8015d3e6f5775df186e443e0964a11c3/distro-1.9.0.tar.gz", hash = "sha256:2fa77c6fd8940f116ee1d6b94a2f90b13b5ea8d019b98bc8bafdcabcdd9bdbed", size = 60722, upload-time = "2023-12-24T09:54:32.31Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/50/3d/9373ad9c56321fdab5b41197068e1d8c25883b3fea29dd361f9b55116869/dill-0.4.0-py3-none-any.whl", hash = "sha256:44f54bf6412c2c8464c14e8243eb163690a9800dbe2c367330883b19c7561049", size = 119668, upload-time = "2025-04-16T00:41:47.671Z" },
+ { url = "https://files.pythonhosted.org/packages/12/b3/231ffd4ab1fc9d679809f356cebee130ac7daa00d6d6f3206dd4fd137e9e/distro-1.9.0-py3-none-any.whl", hash = "sha256:7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2", size = 20277, upload-time = "2023-12-24T09:54:30.421Z" },
]
[[package]]
@@ -1221,12 +1163,22 @@ wheels = [
[[package]]
name = "docutils"
-version = "0.21.2"
+version = "0.22.4"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/ae/ed/aefcc8cd0ba62a0560c3c18c33925362d46c6075480bfa4df87b28e169a9/docutils-0.21.2.tar.gz", hash = "sha256:3a6b18732edf182daa3cd12775bbb338cf5691468f91eeeb109deff6ebfa986f", size = 2204444, upload-time = "2024-04-23T18:57:18.24Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/ae/b6/03bb70946330e88ffec97aefd3ea75ba575cb2e762061e0e62a213befee8/docutils-0.22.4.tar.gz", hash = "sha256:4db53b1fde9abecbb74d91230d32ab626d94f6badfc575d6db9194a49df29968", size = 2291750, upload-time = "2025-12-18T19:00:26.443Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/8f/d7/9322c609343d929e75e7e5e6255e614fcc67572cfd083959cdef3b7aad79/docutils-0.21.2-py3-none-any.whl", hash = "sha256:dafca5b9e384f0e419294eb4d2ff9fa826435bf15f15b7bd45723e8ad76811b2", size = 587408, upload-time = "2024-04-23T18:57:14.835Z" },
+ { url = "https://files.pythonhosted.org/packages/02/10/5da547df7a391dcde17f59520a231527b8571e6f46fc8efb02ccb370ab12/docutils-0.22.4-py3-none-any.whl", hash = "sha256:d0013f540772d1420576855455d050a2180186c91c15779301ac2ccb3eeb68de", size = 633196, upload-time = "2025-12-18T19:00:18.077Z" },
+]
+
+[[package]]
+name = "drain3"
+version = "0.9.11"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cachetools" },
+ { name = "jsonpickle" },
]
+sdist = { url = "https://files.pythonhosted.org/packages/dc/83/4da2d3a11b5e0edf1a4f4c0c2dd42126d2eb1f31c733967edd3dfac1af94/drain3-0.9.11.tar.gz", hash = "sha256:9ab4b1407fad74f56554ae371ef019c3c7985861631f4bab46a0e92585125f75", size = 27960, upload-time = "2022-07-17T06:40:11.433Z" }
[[package]]
name = "ebmlite"
@@ -1239,11 +1191,11 @@ wheels = [
[[package]]
name = "einops"
-version = "0.8.1"
+version = "0.8.2"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/e5/81/df4fbe24dff8ba3934af99044188e20a98ed441ad17a274539b74e82e126/einops-0.8.1.tar.gz", hash = "sha256:de5d960a7a761225532e0f1959e5315ebeafc0cd43394732f103ca44b9837e84", size = 54805, upload-time = "2025-02-09T03:17:00.434Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/2c/77/850bef8d72ffb9219f0b1aac23fbc1bf7d038ee6ea666f331fa273031aa2/einops-0.8.2.tar.gz", hash = "sha256:609da665570e5e265e27283aab09e7f279ade90c4f01bcfca111f3d3e13f2827", size = 56261, upload-time = "2026-01-26T04:13:17.638Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/87/62/9773de14fe6c45c23649e98b83231fffd7b9892b6cf863251dc2afa73643/einops-0.8.1-py3-none-any.whl", hash = "sha256:919387eb55330f5757c6bea9165c5ff5cfe63a642682ea788a6d472576d81737", size = 64359, upload-time = "2025-02-09T03:17:01.998Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl", hash = "sha256:54058201ac7087911181bfec4af6091bb59380360f069276601256a76af08193", size = 65638, upload-time = "2026-01-26T04:13:18.546Z" },
]
[[package]]
@@ -1261,7 +1213,7 @@ name = "exceptiongroup"
version = "1.3.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
- { name = "typing-extensions", marker = "python_full_version < '3.13'" },
+ { name = "typing-extensions", marker = "python_full_version < '3.13' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
]
sdist = { url = "https://files.pythonhosted.org/packages/50/79/66800aadf48771f6b62f7eb014e352e5d06856655206165d775e675a02c9/exceptiongroup-1.3.1.tar.gz", hash = "sha256:8b412432c6055b0b7d14c310000ae93352ed6754f70fa8f7c34141f91c4e3219", size = 30371, upload-time = "2025-11-21T23:01:54.787Z" }
wheels = [
@@ -1285,17 +1237,18 @@ wheels = [
[[package]]
name = "fastapi"
-version = "0.128.0"
+version = "0.135.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "annotated-doc" },
{ name = "pydantic" },
{ name = "starlette" },
{ name = "typing-extensions" },
+ { name = "typing-inspection" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/52/08/8c8508db6c7b9aae8f7175046af41baad690771c9bcde676419965e338c7/fastapi-0.128.0.tar.gz", hash = "sha256:1cc179e1cef10a6be60ffe429f79b829dce99d8de32d7acb7e6c8dfdf7f2645a", size = 365682, upload-time = "2025-12-27T15:21:13.714Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/c4/73/5903c4b13beae98618d64eb9870c3fac4f605523dd0312ca5c80dadbd5b9/fastapi-0.135.2.tar.gz", hash = "sha256:88a832095359755527b7f63bb4c6bc9edb8329a026189eed83d6c1afcf419d56", size = 395833, upload-time = "2026-03-23T14:12:41.697Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/5c/05/5cbb59154b093548acd0f4c7c474a118eda06da25aa75c616b72d8fcd92a/fastapi-0.128.0-py3-none-any.whl", hash = "sha256:aebd93f9716ee3b4f4fcfe13ffb7cf308d99c9f3ab5622d8877441072561582d", size = 103094, upload-time = "2025-12-27T15:21:12.154Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/ea/18f6d0457f9efb2fc6fa594857f92810cadb03024975726db6546b3d6fcf/fastapi-0.135.2-py3-none-any.whl", hash = "sha256:0af0447d541867e8db2a6a25c23a8c4bd80e2394ac5529bd87501bbb9e240ca5", size = 117407, upload-time = "2026-03-23T14:12:43.284Z" },
]
[[package]]
@@ -1315,11 +1268,11 @@ wheels = [
[[package]]
name = "filelock"
-version = "3.20.3"
+version = "3.25.2"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/1d/65/ce7f1b70157833bf3cb851b556a37d4547ceafc158aa9b34b36782f23696/filelock-3.20.3.tar.gz", hash = "sha256:18c57ee915c7ec61cff0ecf7f0f869936c7c30191bb0cf406f1341778d0834e1", size = 19485, upload-time = "2026-01-09T17:55:05.421Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/94/b8/00651a0f559862f3bb7d6f7477b192afe3f583cc5e26403b44e59a55ab34/filelock-3.25.2.tar.gz", hash = "sha256:b64ece2b38f4ca29dd3e810287aa8c48182bbecd1ae6e9ae126c9b35f1382694", size = 40480, upload-time = "2026-03-11T20:45:38.487Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/b5/36/7fb70f04bf00bc646cd5bb45aa9eddb15e19437a28b8fb2b4a5249fac770/filelock-3.20.3-py3-none-any.whl", hash = "sha256:4b0dda527ee31078689fc205ec4f1c1bf7d56cf88b6dc9426c4f230e46c2dce1", size = 16701, upload-time = "2026-01-09T17:55:04.334Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/a5/842ae8f0c08b61d6484b52f99a03510a3a72d23141942d216ebe81fefbce/filelock-3.25.2-py3-none-any.whl", hash = "sha256:ca8afb0da15f229774c9ad1b455ed96e85a81373065fb10446672f64444ddf70", size = 26759, upload-time = "2026-03-11T20:45:37.437Z" },
]
[[package]]
@@ -1333,15 +1286,15 @@ wheels = [
[[package]]
name = "fla-core"
-version = "0.4.1"
+version = "0.4.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "einops" },
{ name = "torch", marker = "sys_platform == 'never'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/f1/de/0d6bd5664ba2e711cabdde11ccb41ddcdd866c531e40900af3601bd7b8c6/fla_core-0.4.1.tar.gz", hash = "sha256:38ab28966eeadc2141b29e87c2bf72a8a4851e00af9d25bbbc3596b1fb53450d", size = 319608, upload-time = "2025-12-24T18:07:37.669Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/53/f9/9e05c48f92b1388a8a357141eb557ed0dd6d4bb936e1d05d35f01976657f/fla_core-0.4.2.tar.gz", hash = "sha256:e9fef6fcdf122029f9feb7dccfeb85eb9650e6aabc72d2a65b36558e9c590edd", size = 377722, upload-time = "2026-03-12T14:45:46.101Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/f6/43/945ef69eb48a14c30fd7323d3e0b560c821ae71e6d3ef979e06a901bc3b9/fla_core-0.4.1-py3-none-any.whl", hash = "sha256:93c6afe4c80fc7bc705fa8aeea6a46d2cf2d77383f9619a41863c7114c801bab", size = 437282, upload-time = "2025-12-24T18:07:34.41Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/36/3c303f92bafea7c3f97d68bbb83d18cc42e30cd0bfb1b7cfe589360f11d6/fla_core-0.4.2-py3-none-any.whl", hash = "sha256:cba3db29380002da3cbfc0db94d6efac19aaf528900d19c05c2765e8f3cc485b", size = 510239, upload-time = "2026-03-12T14:45:43.708Z" },
]
[[package]]
@@ -1360,15 +1313,15 @@ wheels = [
[[package]]
name = "flash-linear-attention"
-version = "0.4.1"
+version = "0.4.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "fla-core" },
{ name = "transformers" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/46/83/7d8ec7ffb5229080b1c9b772338ff588cbd63282ac355ede2a12a6e174a8/flash_linear_attention-0.4.1.tar.gz", hash = "sha256:127ee7273ed15ac17f72bcf4c75e1051719d8fbe0a2d1d047e59406f36d81ee2", size = 158280, upload-time = "2025-12-24T18:07:38.812Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/d1/cb/46cc27a829a10b308927c5dbc99176906a021bb0770253699e93f3cd81a0/flash_linear_attention-0.4.2.tar.gz", hash = "sha256:f97c01ebe7cf390323af07dd3fb65ade07da16724339bf70c78607bc0c007c34", size = 148464, upload-time = "2026-03-12T14:45:46.945Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/63/d5/6327559a9d5b9243b10c3984f1bcef256ed2ad06d105a3bb8f7b2979659c/flash_linear_attention-0.4.1-py3-none-any.whl", hash = "sha256:d18bdfe9d1f4b424676444eac9d50fb8433b70e5d4e0e0878b20bcbcdbea57ce", size = 287415, upload-time = "2025-12-24T18:07:35.815Z" },
+ { url = "https://files.pythonhosted.org/packages/60/ee/a3cba17965482b35c4990af90bad108e82c32edcb59911c37f318b5f4198/flash_linear_attention-0.4.2-py3-none-any.whl", hash = "sha256:c08be006ce4dbe1be81f54938ee8e6fc7968cfba397c8d06c7669e97b8c44c0d", size = 284661, upload-time = "2026-03-12T14:45:44.905Z" },
]
[[package]]
@@ -1385,12 +1338,13 @@ dependencies = [
{ name = "click" },
{ name = "einops" },
{ name = "ninja" },
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version < '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version >= '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
{ name = "nvidia-cudnn-frontend" },
{ name = "nvidia-cutlass-dsl" },
{ name = "nvidia-ml-py" },
- { name = "packaging" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
{ name = "requests" },
{ name = "tabulate" },
{ name = "torch", marker = "sys_platform == 'never'" },
@@ -1403,7 +1357,7 @@ wheels = [
[[package]]
name = "flask"
-version = "3.1.2"
+version = "3.1.3"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "blinker" },
@@ -1413,9 +1367,9 @@ dependencies = [
{ name = "markupsafe" },
{ name = "werkzeug" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/dc/6d/cfe3c0fcc5e477df242b98bfe186a4c34357b4847e87ecaef04507332dab/flask-3.1.2.tar.gz", hash = "sha256:bf656c15c80190ed628ad08cdfd3aaa35beb087855e2f494910aa3774cc4fd87", size = 720160, upload-time = "2025-08-19T21:03:21.205Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/26/00/35d85dcce6c57fdc871f3867d465d780f302a175ea360f62533f12b27e2b/flask-3.1.3.tar.gz", hash = "sha256:0ef0e52b8a9cd932855379197dd8f94047b359ca0a78695144304cb45f87c9eb", size = 759004, upload-time = "2026-02-19T05:00:57.678Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/ec/f9/7f9263c5695f4bd0023734af91bedb2ff8209e8de6ead162f35d8dc762fd/flask-3.1.2-py3-none-any.whl", hash = "sha256:ca1d8112ec8a6158cc29ea4858963350011b5c846a414cdb7a954aa9e967d03c", size = 103308, upload-time = "2025-08-19T21:03:19.499Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/9c/34f6962f9b9e9c71f6e5ed806e0d0ff03c9d1b0b2340088a0cf4bce09b18/flask-3.1.3-py3-none-any.whl", hash = "sha256:f4bcbefc124291925f1a26446da31a5178f9483862233b23c0c96a20701f670c", size = 103424, upload-time = "2026-02-19T05:00:56.027Z" },
]
[[package]]
@@ -1439,38 +1393,6 @@ version = "1.8.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/2d/f5/c831fac6cc817d26fd54c7eaccd04ef7e0288806943f7cc5bbf69f3ac1f0/frozenlist-1.8.0.tar.gz", hash = "sha256:3ede829ed8d842f6cd48fc7081d7a41001a56f1f38603f9d49bf3020d59a31ad", size = 45875, upload-time = "2025-10-06T05:38:17.865Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/83/4a/557715d5047da48d54e659203b9335be7bfaafda2c3f627b7c47e0b3aaf3/frozenlist-1.8.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:b37f6d31b3dcea7deb5e9696e529a6aa4a898adc33db82da12e4c60a7c4d2011", size = 86230, upload-time = "2025-10-06T05:35:23.699Z" },
- { url = "https://files.pythonhosted.org/packages/a2/fb/c85f9fed3ea8fe8740e5b46a59cc141c23b842eca617da8876cfce5f760e/frozenlist-1.8.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:ef2b7b394f208233e471abc541cc6991f907ffd47dc72584acee3147899d6565", size = 49621, upload-time = "2025-10-06T05:35:25.341Z" },
- { url = "https://files.pythonhosted.org/packages/63/70/26ca3f06aace16f2352796b08704338d74b6d1a24ca38f2771afbb7ed915/frozenlist-1.8.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:a88f062f072d1589b7b46e951698950e7da00442fc1cacbe17e19e025dc327ad", size = 49889, upload-time = "2025-10-06T05:35:26.797Z" },
- { url = "https://files.pythonhosted.org/packages/5d/ed/c7895fd2fde7f3ee70d248175f9b6cdf792fb741ab92dc59cd9ef3bd241b/frozenlist-1.8.0-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:f57fb59d9f385710aa7060e89410aeb5058b99e62f4d16b08b91986b9a2140c2", size = 219464, upload-time = "2025-10-06T05:35:28.254Z" },
- { url = "https://files.pythonhosted.org/packages/6b/83/4d587dccbfca74cb8b810472392ad62bfa100bf8108c7223eb4c4fa2f7b3/frozenlist-1.8.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:799345ab092bee59f01a915620b5d014698547afd011e691a208637312db9186", size = 221649, upload-time = "2025-10-06T05:35:29.454Z" },
- { url = "https://files.pythonhosted.org/packages/6a/c6/fd3b9cd046ec5fff9dab66831083bc2077006a874a2d3d9247dea93ddf7e/frozenlist-1.8.0-cp310-cp310-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c23c3ff005322a6e16f71bf8692fcf4d5a304aaafe1e262c98c6d4adc7be863e", size = 219188, upload-time = "2025-10-06T05:35:30.951Z" },
- { url = "https://files.pythonhosted.org/packages/ce/80/6693f55eb2e085fc8afb28cf611448fb5b90e98e068fa1d1b8d8e66e5c7d/frozenlist-1.8.0-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8a76ea0f0b9dfa06f254ee06053d93a600865b3274358ca48a352ce4f0798450", size = 231748, upload-time = "2025-10-06T05:35:32.101Z" },
- { url = "https://files.pythonhosted.org/packages/97/d6/e9459f7c5183854abd989ba384fe0cc1a0fb795a83c033f0571ec5933ca4/frozenlist-1.8.0-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c7366fe1418a6133d5aa824ee53d406550110984de7637d65a178010f759c6ef", size = 236351, upload-time = "2025-10-06T05:35:33.834Z" },
- { url = "https://files.pythonhosted.org/packages/97/92/24e97474b65c0262e9ecd076e826bfd1d3074adcc165a256e42e7b8a7249/frozenlist-1.8.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:13d23a45c4cebade99340c4165bd90eeb4a56c6d8a9d8aa49568cac19a6d0dc4", size = 218767, upload-time = "2025-10-06T05:35:35.205Z" },
- { url = "https://files.pythonhosted.org/packages/ee/bf/dc394a097508f15abff383c5108cb8ad880d1f64a725ed3b90d5c2fbf0bb/frozenlist-1.8.0-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:e4a3408834f65da56c83528fb52ce7911484f0d1eaf7b761fc66001db1646eff", size = 235887, upload-time = "2025-10-06T05:35:36.354Z" },
- { url = "https://files.pythonhosted.org/packages/40/90/25b201b9c015dbc999a5baf475a257010471a1fa8c200c843fd4abbee725/frozenlist-1.8.0-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:42145cd2748ca39f32801dad54aeea10039da6f86e303659db90db1c4b614c8c", size = 228785, upload-time = "2025-10-06T05:35:37.949Z" },
- { url = "https://files.pythonhosted.org/packages/84/f4/b5bc148df03082f05d2dd30c089e269acdbe251ac9a9cf4e727b2dbb8a3d/frozenlist-1.8.0-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:e2de870d16a7a53901e41b64ffdf26f2fbb8917b3e6ebf398098d72c5b20bd7f", size = 230312, upload-time = "2025-10-06T05:35:39.178Z" },
- { url = "https://files.pythonhosted.org/packages/db/4b/87e95b5d15097c302430e647136b7d7ab2398a702390cf4c8601975709e7/frozenlist-1.8.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:20e63c9493d33ee48536600d1a5c95eefc870cd71e7ab037763d1fbb89cc51e7", size = 217650, upload-time = "2025-10-06T05:35:40.377Z" },
- { url = "https://files.pythonhosted.org/packages/e5/70/78a0315d1fea97120591a83e0acd644da638c872f142fd72a6cebee825f3/frozenlist-1.8.0-cp310-cp310-win32.whl", hash = "sha256:adbeebaebae3526afc3c96fad434367cafbfd1b25d72369a9e5858453b1bb71a", size = 39659, upload-time = "2025-10-06T05:35:41.863Z" },
- { url = "https://files.pythonhosted.org/packages/66/aa/3f04523fb189a00e147e60c5b2205126118f216b0aa908035c45336e27e4/frozenlist-1.8.0-cp310-cp310-win_amd64.whl", hash = "sha256:667c3777ca571e5dbeb76f331562ff98b957431df140b54c85fd4d52eea8d8f6", size = 43837, upload-time = "2025-10-06T05:35:43.205Z" },
- { url = "https://files.pythonhosted.org/packages/39/75/1135feecdd7c336938bd55b4dc3b0dfc46d85b9be12ef2628574b28de776/frozenlist-1.8.0-cp310-cp310-win_arm64.whl", hash = "sha256:80f85f0a7cc86e7a54c46d99c9e1318ff01f4687c172ede30fd52d19d1da1c8e", size = 39989, upload-time = "2025-10-06T05:35:44.596Z" },
- { url = "https://files.pythonhosted.org/packages/bc/03/077f869d540370db12165c0aa51640a873fb661d8b315d1d4d67b284d7ac/frozenlist-1.8.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:09474e9831bc2b2199fad6da3c14c7b0fbdd377cce9d3d77131be28906cb7d84", size = 86912, upload-time = "2025-10-06T05:35:45.98Z" },
- { url = "https://files.pythonhosted.org/packages/df/b5/7610b6bd13e4ae77b96ba85abea1c8cb249683217ef09ac9e0ae93f25a91/frozenlist-1.8.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:17c883ab0ab67200b5f964d2b9ed6b00971917d5d8a92df149dc2c9779208ee9", size = 50046, upload-time = "2025-10-06T05:35:47.009Z" },
- { url = "https://files.pythonhosted.org/packages/6e/ef/0e8f1fe32f8a53dd26bdd1f9347efe0778b0fddf62789ea683f4cc7d787d/frozenlist-1.8.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:fa47e444b8ba08fffd1c18e8cdb9a75db1b6a27f17507522834ad13ed5922b93", size = 50119, upload-time = "2025-10-06T05:35:48.38Z" },
- { url = "https://files.pythonhosted.org/packages/11/b1/71a477adc7c36e5fb628245dfbdea2166feae310757dea848d02bd0689fd/frozenlist-1.8.0-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:2552f44204b744fba866e573be4c1f9048d6a324dfe14475103fd51613eb1d1f", size = 231067, upload-time = "2025-10-06T05:35:49.97Z" },
- { url = "https://files.pythonhosted.org/packages/45/7e/afe40eca3a2dc19b9904c0f5d7edfe82b5304cb831391edec0ac04af94c2/frozenlist-1.8.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:957e7c38f250991e48a9a73e6423db1bb9dd14e722a10f6b8bb8e16a0f55f695", size = 233160, upload-time = "2025-10-06T05:35:51.729Z" },
- { url = "https://files.pythonhosted.org/packages/a6/aa/7416eac95603ce428679d273255ffc7c998d4132cfae200103f164b108aa/frozenlist-1.8.0-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:8585e3bb2cdea02fc88ffa245069c36555557ad3609e83be0ec71f54fd4abb52", size = 228544, upload-time = "2025-10-06T05:35:53.246Z" },
- { url = "https://files.pythonhosted.org/packages/8b/3d/2a2d1f683d55ac7e3875e4263d28410063e738384d3adc294f5ff3d7105e/frozenlist-1.8.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:edee74874ce20a373d62dc28b0b18b93f645633c2943fd90ee9d898550770581", size = 243797, upload-time = "2025-10-06T05:35:54.497Z" },
- { url = "https://files.pythonhosted.org/packages/78/1e/2d5565b589e580c296d3bb54da08d206e797d941a83a6fdea42af23be79c/frozenlist-1.8.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c9a63152fe95756b85f31186bddf42e4c02c6321207fd6601a1c89ebac4fe567", size = 247923, upload-time = "2025-10-06T05:35:55.861Z" },
- { url = "https://files.pythonhosted.org/packages/aa/c3/65872fcf1d326a7f101ad4d86285c403c87be7d832b7470b77f6d2ed5ddc/frozenlist-1.8.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:b6db2185db9be0a04fecf2f241c70b63b1a242e2805be291855078f2b404dd6b", size = 230886, upload-time = "2025-10-06T05:35:57.399Z" },
- { url = "https://files.pythonhosted.org/packages/a0/76/ac9ced601d62f6956f03cc794f9e04c81719509f85255abf96e2510f4265/frozenlist-1.8.0-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:f4be2e3d8bc8aabd566f8d5b8ba7ecc09249d74ba3c9ed52e54dc23a293f0b92", size = 245731, upload-time = "2025-10-06T05:35:58.563Z" },
- { url = "https://files.pythonhosted.org/packages/b9/49/ecccb5f2598daf0b4a1415497eba4c33c1e8ce07495eb07d2860c731b8d5/frozenlist-1.8.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:c8d1634419f39ea6f5c427ea2f90ca85126b54b50837f31497f3bf38266e853d", size = 241544, upload-time = "2025-10-06T05:35:59.719Z" },
- { url = "https://files.pythonhosted.org/packages/53/4b/ddf24113323c0bbcc54cb38c8b8916f1da7165e07b8e24a717b4a12cbf10/frozenlist-1.8.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:1a7fa382a4a223773ed64242dbe1c9c326ec09457e6b8428efb4118c685c3dfd", size = 241806, upload-time = "2025-10-06T05:36:00.959Z" },
- { url = "https://files.pythonhosted.org/packages/a7/fb/9b9a084d73c67175484ba2789a59f8eebebd0827d186a8102005ce41e1ba/frozenlist-1.8.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:11847b53d722050808926e785df837353bd4d75f1d494377e59b23594d834967", size = 229382, upload-time = "2025-10-06T05:36:02.22Z" },
- { url = "https://files.pythonhosted.org/packages/95/a3/c8fb25aac55bf5e12dae5c5aa6a98f85d436c1dc658f21c3ac73f9fa95e5/frozenlist-1.8.0-cp311-cp311-win32.whl", hash = "sha256:27c6e8077956cf73eadd514be8fb04d77fc946a7fe9f7fe167648b0b9085cc25", size = 39647, upload-time = "2025-10-06T05:36:03.409Z" },
- { url = "https://files.pythonhosted.org/packages/0a/f5/603d0d6a02cfd4c8f2a095a54672b3cf967ad688a60fb9faf04fc4887f65/frozenlist-1.8.0-cp311-cp311-win_amd64.whl", hash = "sha256:ac913f8403b36a2c8610bbfd25b8013488533e71e62b4b4adce9c86c8cea905b", size = 44064, upload-time = "2025-10-06T05:36:04.368Z" },
- { url = "https://files.pythonhosted.org/packages/5d/16/c2c9ab44e181f043a86f9a8f84d5124b62dbcb3a02c0977ec72b9ac1d3e0/frozenlist-1.8.0-cp311-cp311-win_arm64.whl", hash = "sha256:d4d3214a0f8394edfa3e303136d0575eece0745ff2b47bd2cb2e66dd92d4351a", size = 39937, upload-time = "2025-10-06T05:36:05.669Z" },
{ url = "https://files.pythonhosted.org/packages/69/29/948b9aa87e75820a38650af445d2ef2b6b8a6fab1a23b6bb9e4ef0be2d59/frozenlist-1.8.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:78f7b9e5d6f2fdb88cdde9440dc147259b62b9d3b019924def9f6478be254ac1", size = 87782, upload-time = "2025-10-06T05:36:06.649Z" },
{ url = "https://files.pythonhosted.org/packages/64/80/4f6e318ee2a7c0750ed724fa33a4bdf1eacdc5a39a7a24e818a773cd91af/frozenlist-1.8.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:229bf37d2e4acdaf808fd3f06e854a4a7a3661e871b10dc1f8f1896a3b05f18b", size = 50594, upload-time = "2025-10-06T05:36:07.69Z" },
{ url = "https://files.pythonhosted.org/packages/2b/94/5c8a2b50a496b11dd519f4a24cb5496cf125681dd99e94c604ccdea9419a/frozenlist-1.8.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f833670942247a14eafbb675458b4e61c82e002a148f49e68257b79296e865c4", size = 50448, upload-time = "2025-10-06T05:36:08.78Z" },
@@ -1556,11 +1478,57 @@ wheels = [
[[package]]
name = "fsspec"
-version = "2025.10.0"
+version = "2026.2.0"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/24/7f/2747c0d332b9acfa75dc84447a066fdf812b5a6b8d30472b74d309bfe8cb/fsspec-2025.10.0.tar.gz", hash = "sha256:b6789427626f068f9a83ca4e8a3cc050850b6c0f71f99ddb4f542b8266a26a59", size = 309285, upload-time = "2025-10-30T14:58:44.036Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/eb/02/a6b21098b1d5d6249b7c5ab69dde30108a71e4e819d4a9778f1de1d5b70d/fsspec-2025.10.0-py3-none-any.whl", hash = "sha256:7c7712353ae7d875407f97715f0e1ffcc21e33d5b24556cb1e090ae9409ec61d", size = 200966, upload-time = "2025-10-30T14:58:42.53Z" },
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+]
+sdist = { url = "https://files.pythonhosted.org/packages/51/7c/f60c259dcbf4f0c47cc4ddb8f7720d2dcdc8888c8e5ad84c73ea4531cc5b/fsspec-2026.2.0.tar.gz", hash = "sha256:6544e34b16869f5aacd5b90bdf1a71acb37792ea3ddf6125ee69a22a53fb8bff", size = 313441, upload-time = "2026-02-05T21:50:53.743Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e6/ab/fb21f4c939bb440104cc2b396d3be1d9b7a9fd3c6c2a53d98c45b3d7c954/fsspec-2026.2.0-py3-none-any.whl", hash = "sha256:98de475b5cb3bd66bedd5c4679e87b4fdfe1a3bf4d707b151b3c07e58c9a2437", size = 202505, upload-time = "2026-02-05T21:50:51.819Z" },
]
[package.optional-dependencies]
@@ -1568,6 +1536,18 @@ http = [
{ name = "aiohttp" },
]
+[[package]]
+name = "fsspec"
+version = "2026.3.0"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and sys_platform == 'win32'",
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e1/cf/b50ddf667c15276a9ab15a70ef5f257564de271957933ffea49d2cdbcdfb/fsspec-2026.3.0.tar.gz", hash = "sha256:1ee6a0e28677557f8c2f994e3eea77db6392b4de9cd1f5d7a9e87a0ae9d01b41", size = 313547, upload-time = "2026-03-27T19:11:14.892Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d5/1f/5f4a3cd9e4440e9d9bc78ad0a91a1c8d46b4d429d5239ebe6793c9fe5c41/fsspec-2026.3.0-py3-none-any.whl", hash = "sha256:d2ceafaad1b3457968ed14efa28798162f1638dbb5d2a6868a2db002a5ee39a4", size = 202595, upload-time = "2026-03-27T19:11:13.595Z" },
+]
+
[[package]]
name = "gitdb"
version = "4.0.12"
@@ -1592,6 +1572,47 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/6a/09/e21df6aef1e1ffc0c816f0522ddc3f6dcded766c3261813131c78a704470/gitpython-3.1.46-py3-none-any.whl", hash = "sha256:79812ed143d9d25b6d176a10bb511de0f9c67b1fa641d82097b0ab90398a2058", size = 208620, upload-time = "2026-01-01T15:37:30.574Z" },
]
+[[package]]
+name = "google-api-core"
+version = "2.30.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "google-auth" },
+ { name = "googleapis-common-protos" },
+ { name = "proto-plus" },
+ { name = "protobuf" },
+ { name = "requests" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/22/98/586ec94553b569080caef635f98a3723db36a38eac0e3d7eb3ea9d2e4b9a/google_api_core-2.30.0.tar.gz", hash = "sha256:02edfa9fab31e17fc0befb5f161b3bf93c9096d99aed584625f38065c511ad9b", size = 176959, upload-time = "2026-02-18T20:28:11.926Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/45/27/09c33d67f7e0dcf06d7ac17d196594e66989299374bfb0d4331d1038e76b/google_api_core-2.30.0-py3-none-any.whl", hash = "sha256:80be49ee937ff9aba0fd79a6eddfde35fe658b9953ab9b79c57dd7061afa8df5", size = 173288, upload-time = "2026-02-18T20:28:10.367Z" },
+]
+
+[[package]]
+name = "google-auth"
+version = "2.49.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cryptography" },
+ { name = "pyasn1-modules" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ea/80/6a696a07d3d3b0a92488933532f03dbefa4a24ab80fb231395b9a2a1be77/google_auth-2.49.1.tar.gz", hash = "sha256:16d40da1c3c5a0533f57d268fe72e0ebb0ae1cc3b567024122651c045d879b64", size = 333825, upload-time = "2026-03-12T19:30:58.135Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e9/eb/c6c2478d8a8d633460be40e2a8a6f8f429171997a35a96f81d3b680dec83/google_auth-2.49.1-py3-none-any.whl", hash = "sha256:195ebe3dca18eddd1b3db5edc5189b76c13e96f29e73043b923ebcf3f1a860f7", size = 240737, upload-time = "2026-03-12T19:30:53.159Z" },
+]
+
+[[package]]
+name = "googleapis-common-protos"
+version = "1.73.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "protobuf" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a1/c0/4a54c386282c13449eca8bbe2ddb518181dc113e78d240458a68856b4d69/googleapis_common_protos-1.73.1.tar.gz", hash = "sha256:13114f0e9d2391756a0194c3a8131974ed7bffb06086569ba193364af59163b6", size = 147506, upload-time = "2026-03-26T22:17:38.451Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/dc/82/fcb6520612bec0c39b973a6c0954b6a0d948aadfe8f7e9487f60ceb8bfa6/googleapis_common_protos-1.73.1-py3-none-any.whl", hash = "sha256:e51f09eb0a43a8602f5a915870972e6b4a394088415c79d79605a46d8e826ee8", size = 297556, upload-time = "2026-03-26T22:15:58.455Z" },
+]
+
[[package]]
name = "graphviz"
version = "0.21"
@@ -1601,65 +1622,131 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/91/4c/e0ce1ef95d4000ebc1c11801f9b944fa5910ecc15b5e351865763d8657f8/graphviz-0.21-py3-none-any.whl", hash = "sha256:54f33de9f4f911d7e84e4191749cac8cc5653f815b06738c54db9a15ab8b1e42", size = 47300, upload-time = "2025-06-15T09:35:04.433Z" },
]
+[[package]]
+name = "greenlet"
+version = "3.3.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a3/51/1664f6b78fc6ebbd98019a1fd730e83fa78f2db7058f72b1463d3612b8db/greenlet-3.3.2.tar.gz", hash = "sha256:2eaf067fc6d886931c7962e8c6bede15d2f01965560f3359b27c80bde2d151f2", size = 188267, upload-time = "2026-02-20T20:54:15.531Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ea/ab/1608e5a7578e62113506740b88066bf09888322a311cff602105e619bd87/greenlet-3.3.2-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:ac8d61d4343b799d1e526db579833d72f23759c71e07181c2d2944e429eb09cd", size = 280358, upload-time = "2026-02-20T20:17:43.971Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/23/0eae412a4ade4e6623ff7626e38998cb9b11e9ff1ebacaa021e4e108ec15/greenlet-3.3.2-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3ceec72030dae6ac0c8ed7591b96b70410a8be370b6a477b1dbc072856ad02bd", size = 601217, upload-time = "2026-02-20T20:47:31.462Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/16/5b1678a9c07098ecb9ab2dd159fafaf12e963293e61ee8d10ecb55273e5e/greenlet-3.3.2-cp312-cp312-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a2a5be83a45ce6188c045bcc44b0ee037d6a518978de9a5d97438548b953a1ac", size = 611792, upload-time = "2026-02-20T20:55:58.423Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/c5/cc09412a29e43406eba18d61c70baa936e299bc27e074e2be3806ed29098/greenlet-3.3.2-cp312-cp312-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ae9e21c84035c490506c17002f5c8ab25f980205c3e61ddb3a2a2a2e6c411fcb", size = 626250, upload-time = "2026-02-20T21:02:46.596Z" },
+ { url = "https://files.pythonhosted.org/packages/50/1f/5155f55bd71cabd03765a4aac9ac446be129895271f73872c36ebd4b04b6/greenlet-3.3.2-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:43e99d1749147ac21dde49b99c9abffcbc1e2d55c67501465ef0930d6e78e070", size = 613875, upload-time = "2026-02-20T20:21:01.102Z" },
+ { url = "https://files.pythonhosted.org/packages/fc/dd/845f249c3fcd69e32df80cdab059b4be8b766ef5830a3d0aa9d6cad55beb/greenlet-3.3.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:4c956a19350e2c37f2c48b336a3afb4bff120b36076d9d7fb68cb44e05d95b79", size = 1571467, upload-time = "2026-02-20T20:49:33.495Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/50/2649fe21fcc2b56659a452868e695634722a6655ba245d9f77f5656010bf/greenlet-3.3.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6c6f8ba97d17a1e7d664151284cb3315fc5f8353e75221ed4324f84eb162b395", size = 1640001, upload-time = "2026-02-20T20:21:09.154Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/40/cc802e067d02af8b60b6771cea7d57e21ef5e6659912814babb42b864713/greenlet-3.3.2-cp312-cp312-win_amd64.whl", hash = "sha256:34308836d8370bddadb41f5a7ce96879b72e2fdfb4e87729330c6ab52376409f", size = 231081, upload-time = "2026-02-20T20:17:28.121Z" },
+ { url = "https://files.pythonhosted.org/packages/58/2e/fe7f36ff1982d6b10a60d5e0740c759259a7d6d2e1dc41da6d96de32fff6/greenlet-3.3.2-cp312-cp312-win_arm64.whl", hash = "sha256:d3a62fa76a32b462a97198e4c9e99afb9ab375115e74e9a83ce180e7a496f643", size = 230331, upload-time = "2026-02-20T20:17:23.34Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/48/f8b875fa7dea7dd9b33245e37f065af59df6a25af2f9561efa8d822fde51/greenlet-3.3.2-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:aa6ac98bdfd716a749b84d4034486863fd81c3abde9aa3cf8eff9127981a4ae4", size = 279120, upload-time = "2026-02-20T20:19:01.9Z" },
+ { url = "https://files.pythonhosted.org/packages/49/8d/9771d03e7a8b1ee456511961e1b97a6d77ae1dea4a34a5b98eee706689d3/greenlet-3.3.2-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ab0c7e7901a00bc0a7284907273dc165b32e0d109a6713babd04471327ff7986", size = 603238, upload-time = "2026-02-20T20:47:32.873Z" },
+ { url = "https://files.pythonhosted.org/packages/59/0e/4223c2bbb63cd5c97f28ffb2a8aee71bdfb30b323c35d409450f51b91e3e/greenlet-3.3.2-cp313-cp313-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d248d8c23c67d2291ffd47af766e2a3aa9fa1c6703155c099feb11f526c63a92", size = 614219, upload-time = "2026-02-20T20:55:59.817Z" },
+ { url = "https://files.pythonhosted.org/packages/94/2b/4d012a69759ac9d77210b8bfb128bc621125f5b20fc398bce3940d036b1c/greenlet-3.3.2-cp313-cp313-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ccd21bb86944ca9be6d967cf7691e658e43417782bce90b5d2faeda0ff78a7dd", size = 628268, upload-time = "2026-02-20T21:02:48.024Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/34/259b28ea7a2a0c904b11cd36c79b8cef8019b26ee5dbe24e73b469dea347/greenlet-3.3.2-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b6997d360a4e6a4e936c0f9625b1c20416b8a0ea18a8e19cabbefc712e7397ab", size = 616774, upload-time = "2026-02-20T20:21:02.454Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/03/996c2d1689d486a6e199cb0f1cf9e4aa940c500e01bdf201299d7d61fa69/greenlet-3.3.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:64970c33a50551c7c50491671265d8954046cb6e8e2999aacdd60e439b70418a", size = 1571277, upload-time = "2026-02-20T20:49:34.795Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/c4/2570fc07f34a39f2caf0bf9f24b0a1a0a47bc2e8e465b2c2424821389dfc/greenlet-3.3.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:1a9172f5bf6bd88e6ba5a84e0a68afeac9dc7b6b412b245dd64f52d83c81e55b", size = 1640455, upload-time = "2026-02-20T20:21:10.261Z" },
+ { url = "https://files.pythonhosted.org/packages/91/39/5ef5aa23bc545aa0d31e1b9b55822b32c8da93ba657295840b6b34124009/greenlet-3.3.2-cp313-cp313-win_amd64.whl", hash = "sha256:a7945dd0eab63ded0a48e4dcade82939783c172290a7903ebde9e184333ca124", size = 230961, upload-time = "2026-02-20T20:16:58.461Z" },
+ { url = "https://files.pythonhosted.org/packages/62/6b/a89f8456dcb06becff288f563618e9f20deed8dd29beea14f9a168aef64b/greenlet-3.3.2-cp313-cp313-win_arm64.whl", hash = "sha256:394ead29063ee3515b4e775216cb756b2e3b4a7e55ae8fd884f17fa579e6b327", size = 230221, upload-time = "2026-02-20T20:17:37.152Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/ae/8bffcbd373b57a5992cd077cbe8858fff39110480a9d50697091faea6f39/greenlet-3.3.2-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:8d1658d7291f9859beed69a776c10822a0a799bc4bfe1bd4272bb60e62507dab", size = 279650, upload-time = "2026-02-20T20:18:00.783Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/c0/45f93f348fa49abf32ac8439938726c480bd96b2a3c6f4d949ec0124b69f/greenlet-3.3.2-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:18cb1b7337bca281915b3c5d5ae19f4e76d35e1df80f4ad3c1a7be91fadf1082", size = 650295, upload-time = "2026-02-20T20:47:34.036Z" },
+ { url = "https://files.pythonhosted.org/packages/b3/de/dd7589b3f2b8372069ab3e4763ea5329940fc7ad9dcd3e272a37516d7c9b/greenlet-3.3.2-cp314-cp314-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c2e47408e8ce1c6f1ceea0dffcdf6ebb85cc09e55c7af407c99f1112016e45e9", size = 662163, upload-time = "2026-02-20T20:56:01.295Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/ac/85804f74f1ccea31ba518dcc8ee6f14c79f73fe36fa1beba38930806df09/greenlet-3.3.2-cp314-cp314-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:e3cb43ce200f59483eb82949bf1835a99cf43d7571e900d7c8d5c62cdf25d2f9", size = 675371, upload-time = "2026-02-20T21:02:49.664Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/d8/09bfa816572a4d83bccd6750df1926f79158b1c36c5f73786e26dbe4ee38/greenlet-3.3.2-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:63d10328839d1973e5ba35e98cccbca71b232b14051fd957b6f8b6e8e80d0506", size = 664160, upload-time = "2026-02-20T20:21:04.015Z" },
+ { url = "https://files.pythonhosted.org/packages/48/cf/56832f0c8255d27f6c35d41b5ec91168d74ec721d85f01a12131eec6b93c/greenlet-3.3.2-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:8e4ab3cfb02993c8cc248ea73d7dae6cec0253e9afa311c9b37e603ca9fad2ce", size = 1619181, upload-time = "2026-02-20T20:49:36.052Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/23/b90b60a4aabb4cec0796e55f25ffbfb579a907c3898cd2905c8918acaa16/greenlet-3.3.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:94ad81f0fd3c0c0681a018a976e5c2bd2ca2d9d94895f23e7bb1af4e8af4e2d5", size = 1687713, upload-time = "2026-02-20T20:21:11.684Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/ca/2101ca3d9223a1dc125140dbc063644dca76df6ff356531eb27bc267b446/greenlet-3.3.2-cp314-cp314-win_amd64.whl", hash = "sha256:8c4dd0f3997cf2512f7601563cc90dfb8957c0cff1e3a1b23991d4ea1776c492", size = 232034, upload-time = "2026-02-20T20:20:08.186Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/4a/ecf894e962a59dea60f04877eea0fd5724618da89f1867b28ee8b91e811f/greenlet-3.3.2-cp314-cp314-win_arm64.whl", hash = "sha256:cd6f9e2bbd46321ba3bbb4c8a15794d32960e3b0ae2cc4d49a1a53d314805d71", size = 231437, upload-time = "2026-02-20T20:18:59.722Z" },
+ { url = "https://files.pythonhosted.org/packages/98/6d/8f2ef704e614bcf58ed43cfb8d87afa1c285e98194ab2cfad351bf04f81e/greenlet-3.3.2-cp314-cp314t-macosx_11_0_universal2.whl", hash = "sha256:e26e72bec7ab387ac80caa7496e0f908ff954f31065b0ffc1f8ecb1338b11b54", size = 286617, upload-time = "2026-02-20T20:19:29.856Z" },
+ { url = "https://files.pythonhosted.org/packages/5e/0d/93894161d307c6ea237a43988f27eba0947b360b99ac5239ad3fe09f0b47/greenlet-3.3.2-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8b466dff7a4ffda6ca975979bab80bdadde979e29fc947ac3be4451428d8b0e4", size = 655189, upload-time = "2026-02-20T20:47:35.742Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/2c/d2d506ebd8abcb57386ec4f7ba20f4030cbe56eae541bc6fd6ef399c0b41/greenlet-3.3.2-cp314-cp314t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b8bddc5b73c9720bea487b3bffdb1840fe4e3656fba3bd40aa1489e9f37877ff", size = 658225, upload-time = "2026-02-20T20:56:02.527Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/67/8197b7e7e602150938049d8e7f30de1660cfb87e4c8ee349b42b67bdb2e1/greenlet-3.3.2-cp314-cp314t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:59b3e2c40f6706b05a9cd299c836c6aa2378cabe25d021acd80f13abf81181cf", size = 666581, upload-time = "2026-02-20T21:02:51.526Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/30/3a09155fbf728673a1dea713572d2d31159f824a37c22da82127056c44e4/greenlet-3.3.2-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b26b0f4428b871a751968285a1ac9648944cea09807177ac639b030bddebcea4", size = 657907, upload-time = "2026-02-20T20:21:05.259Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/fd/d05a4b7acd0154ed758797f0a43b4c0962a843bedfe980115e842c5b2d08/greenlet-3.3.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:1fb39a11ee2e4d94be9a76671482be9398560955c9e568550de0224e41104727", size = 1618857, upload-time = "2026-02-20T20:49:37.309Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/e1/50ee92a5db521de8f35075b5eff060dd43d39ebd46c2181a2042f7070385/greenlet-3.3.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:20154044d9085151bc309e7689d6f7ba10027f8f5a8c0676ad398b951913d89e", size = 1680010, upload-time = "2026-02-20T20:21:13.427Z" },
+ { url = "https://files.pythonhosted.org/packages/29/4b/45d90626aef8e65336bed690106d1382f7a43665e2249017e9527df8823b/greenlet-3.3.2-cp314-cp314t-win_amd64.whl", hash = "sha256:c04c5e06ec3e022cbfe2cd4a846e1d4e50087444f875ff6d2c2ad8445495cf1a", size = 237086, upload-time = "2026-02-20T20:20:45.786Z" },
+]
+
[[package]]
name = "grpcio"
-version = "1.76.0"
+version = "1.80.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/b6/e0/318c1ce3ae5a17894d5791e87aea147587c9e702f24122cc7a5c8bbaeeb1/grpcio-1.76.0.tar.gz", hash = "sha256:7be78388d6da1a25c0d5ec506523db58b18be22d9c37d8d3a32c08be4987bd73", size = 12785182, upload-time = "2025-10-21T16:23:12.106Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/88/17/ff4795dc9a34b6aee6ec379f1b66438a3789cd1315aac0cbab60d92f74b3/grpcio-1.76.0-cp310-cp310-linux_armv7l.whl", hash = "sha256:65a20de41e85648e00305c1bb09a3598f840422e522277641145a32d42dcefcc", size = 5840037, upload-time = "2025-10-21T16:20:25.069Z" },
- { url = "https://files.pythonhosted.org/packages/4e/ff/35f9b96e3fa2f12e1dcd58a4513a2e2294a001d64dec81677361b7040c9a/grpcio-1.76.0-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:40ad3afe81676fd9ec6d9d406eda00933f218038433980aa19d401490e46ecde", size = 11836482, upload-time = "2025-10-21T16:20:30.113Z" },
- { url = "https://files.pythonhosted.org/packages/3e/1c/8374990f9545e99462caacea5413ed783014b3b66ace49e35c533f07507b/grpcio-1.76.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:035d90bc79eaa4bed83f524331d55e35820725c9fbb00ffa1904d5550ed7ede3", size = 6407178, upload-time = "2025-10-21T16:20:32.733Z" },
- { url = "https://files.pythonhosted.org/packages/1e/77/36fd7d7c75a6c12542c90a6d647a27935a1ecaad03e0ffdb7c42db6b04d2/grpcio-1.76.0-cp310-cp310-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:4215d3a102bd95e2e11b5395c78562967959824156af11fa93d18fdd18050990", size = 7075684, upload-time = "2025-10-21T16:20:35.435Z" },
- { url = "https://files.pythonhosted.org/packages/38/f7/e3cdb252492278e004722306c5a8935eae91e64ea11f0af3437a7de2e2b7/grpcio-1.76.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:49ce47231818806067aea3324d4bf13825b658ad662d3b25fada0bdad9b8a6af", size = 6611133, upload-time = "2025-10-21T16:20:37.541Z" },
- { url = "https://files.pythonhosted.org/packages/7e/20/340db7af162ccd20a0893b5f3c4a5d676af7b71105517e62279b5b61d95a/grpcio-1.76.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:8cc3309d8e08fd79089e13ed4819d0af72aa935dd8f435a195fd152796752ff2", size = 7195507, upload-time = "2025-10-21T16:20:39.643Z" },
- { url = "https://files.pythonhosted.org/packages/10/f0/b2160addc1487bd8fa4810857a27132fb4ce35c1b330c2f3ac45d697b106/grpcio-1.76.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:971fd5a1d6e62e00d945423a567e42eb1fa678ba89072832185ca836a94daaa6", size = 8160651, upload-time = "2025-10-21T16:20:42.492Z" },
- { url = "https://files.pythonhosted.org/packages/2c/2c/ac6f98aa113c6ef111b3f347854e99ebb7fb9d8f7bb3af1491d438f62af4/grpcio-1.76.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:9d9adda641db7207e800a7f089068f6f645959f2df27e870ee81d44701dd9db3", size = 7620568, upload-time = "2025-10-21T16:20:45.995Z" },
- { url = "https://files.pythonhosted.org/packages/90/84/7852f7e087285e3ac17a2703bc4129fafee52d77c6c82af97d905566857e/grpcio-1.76.0-cp310-cp310-win32.whl", hash = "sha256:063065249d9e7e0782d03d2bca50787f53bd0fb89a67de9a7b521c4a01f1989b", size = 3998879, upload-time = "2025-10-21T16:20:48.592Z" },
- { url = "https://files.pythonhosted.org/packages/10/30/d3d2adcbb6dd3ff59d6ac3df6ef830e02b437fb5c90990429fd180e52f30/grpcio-1.76.0-cp310-cp310-win_amd64.whl", hash = "sha256:a6ae758eb08088d36812dd5d9af7a9859c05b1e0f714470ea243694b49278e7b", size = 4706892, upload-time = "2025-10-21T16:20:50.697Z" },
- { url = "https://files.pythonhosted.org/packages/a0/00/8163a1beeb6971f66b4bbe6ac9457b97948beba8dd2fc8e1281dce7f79ec/grpcio-1.76.0-cp311-cp311-linux_armv7l.whl", hash = "sha256:2e1743fbd7f5fa713a1b0a8ac8ebabf0ec980b5d8809ec358d488e273b9cf02a", size = 5843567, upload-time = "2025-10-21T16:20:52.829Z" },
- { url = "https://files.pythonhosted.org/packages/10/c1/934202f5cf335e6d852530ce14ddb0fef21be612ba9ecbbcbd4d748ca32d/grpcio-1.76.0-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:a8c2cf1209497cf659a667d7dea88985e834c24b7c3b605e6254cbb5076d985c", size = 11848017, upload-time = "2025-10-21T16:20:56.705Z" },
- { url = "https://files.pythonhosted.org/packages/11/0b/8dec16b1863d74af6eb3543928600ec2195af49ca58b16334972f6775663/grpcio-1.76.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:08caea849a9d3c71a542827d6df9d5a69067b0a1efbea8a855633ff5d9571465", size = 6412027, upload-time = "2025-10-21T16:20:59.3Z" },
- { url = "https://files.pythonhosted.org/packages/d7/64/7b9e6e7ab910bea9d46f2c090380bab274a0b91fb0a2fe9b0cd399fffa12/grpcio-1.76.0-cp311-cp311-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:f0e34c2079d47ae9f6188211db9e777c619a21d4faba6977774e8fa43b085e48", size = 7075913, upload-time = "2025-10-21T16:21:01.645Z" },
- { url = "https://files.pythonhosted.org/packages/68/86/093c46e9546073cefa789bd76d44c5cb2abc824ca62af0c18be590ff13ba/grpcio-1.76.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8843114c0cfce61b40ad48df65abcfc00d4dba82eae8718fab5352390848c5da", size = 6615417, upload-time = "2025-10-21T16:21:03.844Z" },
- { url = "https://files.pythonhosted.org/packages/f7/b6/5709a3a68500a9c03da6fb71740dcdd5ef245e39266461a03f31a57036d8/grpcio-1.76.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:8eddfb4d203a237da6f3cc8a540dad0517d274b5a1e9e636fd8d2c79b5c1d397", size = 7199683, upload-time = "2025-10-21T16:21:06.195Z" },
- { url = "https://files.pythonhosted.org/packages/91/d3/4b1f2bf16ed52ce0b508161df3a2d186e4935379a159a834cb4a7d687429/grpcio-1.76.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:32483fe2aab2c3794101c2a159070584e5db11d0aa091b2c0ea9c4fc43d0d749", size = 8163109, upload-time = "2025-10-21T16:21:08.498Z" },
- { url = "https://files.pythonhosted.org/packages/5c/61/d9043f95f5f4cf085ac5dd6137b469d41befb04bd80280952ffa2a4c3f12/grpcio-1.76.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:dcfe41187da8992c5f40aa8c5ec086fa3672834d2be57a32384c08d5a05b4c00", size = 7626676, upload-time = "2025-10-21T16:21:10.693Z" },
- { url = "https://files.pythonhosted.org/packages/36/95/fd9a5152ca02d8881e4dd419cdd790e11805979f499a2e5b96488b85cf27/grpcio-1.76.0-cp311-cp311-win32.whl", hash = "sha256:2107b0c024d1b35f4083f11245c0e23846ae64d02f40b2b226684840260ed054", size = 3997688, upload-time = "2025-10-21T16:21:12.746Z" },
- { url = "https://files.pythonhosted.org/packages/60/9c/5c359c8d4c9176cfa3c61ecd4efe5affe1f38d9bae81e81ac7186b4c9cc8/grpcio-1.76.0-cp311-cp311-win_amd64.whl", hash = "sha256:522175aba7af9113c48ec10cc471b9b9bd4f6ceb36aeb4544a8e2c80ed9d252d", size = 4709315, upload-time = "2025-10-21T16:21:15.26Z" },
- { url = "https://files.pythonhosted.org/packages/bf/05/8e29121994b8d959ffa0afd28996d452f291b48cfc0875619de0bde2c50c/grpcio-1.76.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:81fd9652b37b36f16138611c7e884eb82e0cec137c40d3ef7c3f9b3ed00f6ed8", size = 5799718, upload-time = "2025-10-21T16:21:17.939Z" },
- { url = "https://files.pythonhosted.org/packages/d9/75/11d0e66b3cdf998c996489581bdad8900db79ebd83513e45c19548f1cba4/grpcio-1.76.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:04bbe1bfe3a68bbfd4e52402ab7d4eb59d72d02647ae2042204326cf4bbad280", size = 11825627, upload-time = "2025-10-21T16:21:20.466Z" },
- { url = "https://files.pythonhosted.org/packages/28/50/2f0aa0498bc188048f5d9504dcc5c2c24f2eb1a9337cd0fa09a61a2e75f0/grpcio-1.76.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d388087771c837cdb6515539f43b9d4bf0b0f23593a24054ac16f7a960be16f4", size = 6359167, upload-time = "2025-10-21T16:21:23.122Z" },
- { url = "https://files.pythonhosted.org/packages/66/e5/bbf0bb97d29ede1d59d6588af40018cfc345b17ce979b7b45424628dc8bb/grpcio-1.76.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:9f8f757bebaaea112c00dba718fc0d3260052ce714e25804a03f93f5d1c6cc11", size = 7044267, upload-time = "2025-10-21T16:21:25.995Z" },
- { url = "https://files.pythonhosted.org/packages/f5/86/f6ec2164f743d9609691115ae8ece098c76b894ebe4f7c94a655c6b03e98/grpcio-1.76.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:980a846182ce88c4f2f7e2c22c56aefd515daeb36149d1c897f83cf57999e0b6", size = 6573963, upload-time = "2025-10-21T16:21:28.631Z" },
- { url = "https://files.pythonhosted.org/packages/60/bc/8d9d0d8505feccfdf38a766d262c71e73639c165b311c9457208b56d92ae/grpcio-1.76.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:f92f88e6c033db65a5ae3d97905c8fea9c725b63e28d5a75cb73b49bda5024d8", size = 7164484, upload-time = "2025-10-21T16:21:30.837Z" },
- { url = "https://files.pythonhosted.org/packages/67/e6/5d6c2fc10b95edf6df9b8f19cf10a34263b7fd48493936fffd5085521292/grpcio-1.76.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:4baf3cbe2f0be3289eb68ac8ae771156971848bb8aaff60bad42005539431980", size = 8127777, upload-time = "2025-10-21T16:21:33.577Z" },
- { url = "https://files.pythonhosted.org/packages/3f/c8/dce8ff21c86abe025efe304d9e31fdb0deaaa3b502b6a78141080f206da0/grpcio-1.76.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:615ba64c208aaceb5ec83bfdce7728b80bfeb8be97562944836a7a0a9647d882", size = 7594014, upload-time = "2025-10-21T16:21:41.882Z" },
- { url = "https://files.pythonhosted.org/packages/e0/42/ad28191ebf983a5d0ecef90bab66baa5a6b18f2bfdef9d0a63b1973d9f75/grpcio-1.76.0-cp312-cp312-win32.whl", hash = "sha256:45d59a649a82df5718fd9527ce775fd66d1af35e6d31abdcdc906a49c6822958", size = 3984750, upload-time = "2025-10-21T16:21:44.006Z" },
- { url = "https://files.pythonhosted.org/packages/9e/00/7bd478cbb851c04a48baccaa49b75abaa8e4122f7d86da797500cccdd771/grpcio-1.76.0-cp312-cp312-win_amd64.whl", hash = "sha256:c088e7a90b6017307f423efbb9d1ba97a22aa2170876223f9709e9d1de0b5347", size = 4704003, upload-time = "2025-10-21T16:21:46.244Z" },
- { url = "https://files.pythonhosted.org/packages/fc/ed/71467ab770effc9e8cef5f2e7388beb2be26ed642d567697bb103a790c72/grpcio-1.76.0-cp313-cp313-linux_armv7l.whl", hash = "sha256:26ef06c73eb53267c2b319f43e6634c7556ea37672029241a056629af27c10e2", size = 5807716, upload-time = "2025-10-21T16:21:48.475Z" },
- { url = "https://files.pythonhosted.org/packages/2c/85/c6ed56f9817fab03fa8a111ca91469941fb514e3e3ce6d793cb8f1e1347b/grpcio-1.76.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:45e0111e73f43f735d70786557dc38141185072d7ff8dc1829d6a77ac1471468", size = 11821522, upload-time = "2025-10-21T16:21:51.142Z" },
- { url = "https://files.pythonhosted.org/packages/ac/31/2b8a235ab40c39cbc141ef647f8a6eb7b0028f023015a4842933bc0d6831/grpcio-1.76.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:83d57312a58dcfe2a3a0f9d1389b299438909a02db60e2f2ea2ae2d8034909d3", size = 6362558, upload-time = "2025-10-21T16:21:54.213Z" },
- { url = "https://files.pythonhosted.org/packages/bd/64/9784eab483358e08847498ee56faf8ff6ea8e0a4592568d9f68edc97e9e9/grpcio-1.76.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:3e2a27c89eb9ac3d81ec8835e12414d73536c6e620355d65102503064a4ed6eb", size = 7049990, upload-time = "2025-10-21T16:21:56.476Z" },
- { url = "https://files.pythonhosted.org/packages/2b/94/8c12319a6369434e7a184b987e8e9f3b49a114c489b8315f029e24de4837/grpcio-1.76.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:61f69297cba3950a524f61c7c8ee12e55c486cb5f7db47ff9dcee33da6f0d3ae", size = 6575387, upload-time = "2025-10-21T16:21:59.051Z" },
- { url = "https://files.pythonhosted.org/packages/15/0f/f12c32b03f731f4a6242f771f63039df182c8b8e2cf8075b245b409259d4/grpcio-1.76.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:6a15c17af8839b6801d554263c546c69c4d7718ad4321e3166175b37eaacca77", size = 7166668, upload-time = "2025-10-21T16:22:02.049Z" },
- { url = "https://files.pythonhosted.org/packages/ff/2d/3ec9ce0c2b1d92dd59d1c3264aaec9f0f7c817d6e8ac683b97198a36ed5a/grpcio-1.76.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:25a18e9810fbc7e7f03ec2516addc116a957f8cbb8cbc95ccc80faa072743d03", size = 8124928, upload-time = "2025-10-21T16:22:04.984Z" },
- { url = "https://files.pythonhosted.org/packages/1a/74/fd3317be5672f4856bcdd1a9e7b5e17554692d3db9a3b273879dc02d657d/grpcio-1.76.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:931091142fd8cc14edccc0845a79248bc155425eee9a98b2db2ea4f00a235a42", size = 7589983, upload-time = "2025-10-21T16:22:07.881Z" },
- { url = "https://files.pythonhosted.org/packages/45/bb/ca038cf420f405971f19821c8c15bcbc875505f6ffadafe9ffd77871dc4c/grpcio-1.76.0-cp313-cp313-win32.whl", hash = "sha256:5e8571632780e08526f118f74170ad8d50fb0a48c23a746bef2a6ebade3abd6f", size = 3984727, upload-time = "2025-10-21T16:22:10.032Z" },
- { url = "https://files.pythonhosted.org/packages/41/80/84087dc56437ced7cdd4b13d7875e7439a52a261e3ab4e06488ba6173b0a/grpcio-1.76.0-cp313-cp313-win_amd64.whl", hash = "sha256:f9f7bd5faab55f47231ad8dba7787866b69f5e93bc306e3915606779bbfb4ba8", size = 4702799, upload-time = "2025-10-21T16:22:12.709Z" },
- { url = "https://files.pythonhosted.org/packages/b4/46/39adac80de49d678e6e073b70204091e76631e03e94928b9ea4ecf0f6e0e/grpcio-1.76.0-cp314-cp314-linux_armv7l.whl", hash = "sha256:ff8a59ea85a1f2191a0ffcc61298c571bc566332f82e5f5be1b83c9d8e668a62", size = 5808417, upload-time = "2025-10-21T16:22:15.02Z" },
- { url = "https://files.pythonhosted.org/packages/9c/f5/a4531f7fb8b4e2a60b94e39d5d924469b7a6988176b3422487be61fe2998/grpcio-1.76.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:06c3d6b076e7b593905d04fdba6a0525711b3466f43b3400266f04ff735de0cd", size = 11828219, upload-time = "2025-10-21T16:22:17.954Z" },
- { url = "https://files.pythonhosted.org/packages/4b/1c/de55d868ed7a8bd6acc6b1d6ddc4aa36d07a9f31d33c912c804adb1b971b/grpcio-1.76.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fd5ef5932f6475c436c4a55e4336ebbe47bd3272be04964a03d316bbf4afbcbc", size = 6367826, upload-time = "2025-10-21T16:22:20.721Z" },
- { url = "https://files.pythonhosted.org/packages/59/64/99e44c02b5adb0ad13ab3adc89cb33cb54bfa90c74770f2607eea629b86f/grpcio-1.76.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:b331680e46239e090f5b3cead313cc772f6caa7d0fc8de349337563125361a4a", size = 7049550, upload-time = "2025-10-21T16:22:23.637Z" },
- { url = "https://files.pythonhosted.org/packages/43/28/40a5be3f9a86949b83e7d6a2ad6011d993cbe9b6bd27bea881f61c7788b6/grpcio-1.76.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2229ae655ec4e8999599469559e97630185fdd53ae1e8997d147b7c9b2b72cba", size = 6575564, upload-time = "2025-10-21T16:22:26.016Z" },
- { url = "https://files.pythonhosted.org/packages/4b/a9/1be18e6055b64467440208a8559afac243c66a8b904213af6f392dc2212f/grpcio-1.76.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:490fa6d203992c47c7b9e4a9d39003a0c2bcc1c9aa3c058730884bbbb0ee9f09", size = 7176236, upload-time = "2025-10-21T16:22:28.362Z" },
- { url = "https://files.pythonhosted.org/packages/0f/55/dba05d3fcc151ce6e81327541d2cc8394f442f6b350fead67401661bf041/grpcio-1.76.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:479496325ce554792dba6548fae3df31a72cef7bad71ca2e12b0e58f9b336bfc", size = 8125795, upload-time = "2025-10-21T16:22:31.075Z" },
- { url = "https://files.pythonhosted.org/packages/4a/45/122df922d05655f63930cf42c9e3f72ba20aadb26c100ee105cad4ce4257/grpcio-1.76.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:1c9b93f79f48b03ada57ea24725d83a30284a012ec27eab2cf7e50a550cbbbcc", size = 7592214, upload-time = "2025-10-21T16:22:33.831Z" },
- { url = "https://files.pythonhosted.org/packages/4a/6e/0b899b7f6b66e5af39e377055fb4a6675c9ee28431df5708139df2e93233/grpcio-1.76.0-cp314-cp314-win32.whl", hash = "sha256:747fa73efa9b8b1488a95d0ba1039c8e2dca0f741612d80415b1e1c560febf4e", size = 4062961, upload-time = "2025-10-21T16:22:36.468Z" },
- { url = "https://files.pythonhosted.org/packages/19/41/0b430b01a2eb38ee887f88c1f07644a1df8e289353b78e82b37ef988fb64/grpcio-1.76.0-cp314-cp314-win_amd64.whl", hash = "sha256:922fa70ba549fce362d2e2871ab542082d66e2aaf0c19480ea453905b01f384e", size = 4834462, upload-time = "2025-10-21T16:22:39.772Z" },
+sdist = { url = "https://files.pythonhosted.org/packages/b7/48/af6173dbca4454f4637a4678b67f52ca7e0c1ed7d5894d89d434fecede05/grpcio-1.80.0.tar.gz", hash = "sha256:29aca15edd0688c22ba01d7cc01cb000d72b2033f4a3c72a81a19b56fd143257", size = 12978905, upload-time = "2026-03-30T08:49:10.502Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5c/e8/a2b749265eb3415abc94f2e619bbd9e9707bebdda787e61c593004ec927a/grpcio-1.80.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:c624cc9f1008361014378c9d776de7182b11fe8b2e5a81bc69f23a295f2a1ad0", size = 6015616, upload-time = "2026-03-30T08:47:13.428Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/97/b1282161a15d699d1e90c360df18d19165a045ce1c343c7f313f5e8a0b77/grpcio-1.80.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:f49eddcac43c3bf350c0385366a58f36bed8cc2c0ec35ef7b74b49e56552c0c2", size = 12014204, upload-time = "2026-03-30T08:47:15.873Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/5e/d319c6e997b50c155ac5a8cb12f5173d5b42677510e886d250d50264949d/grpcio-1.80.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d334591df610ab94714048e0d5b4f3dd5ad1bee74dfec11eee344220077a79de", size = 6563866, upload-time = "2026-03-30T08:47:18.588Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/f6/fdd975a2cb4d78eb67769a7b3b3830970bfa2e919f1decf724ae4445f42c/grpcio-1.80.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:0cb517eb1d0d0aaf1d87af7cc5b801d686557c1d88b2619f5e31fab3c2315921", size = 7273060, upload-time = "2026-03-30T08:47:21.113Z" },
+ { url = "https://files.pythonhosted.org/packages/db/f0/a3deb5feba60d9538a962913e37bd2e69a195f1c3376a3dd44fe0427e996/grpcio-1.80.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4e78c4ac0d97dc2e569b2f4bcbbb447491167cb358d1a389fc4af71ab6f70411", size = 6782121, upload-time = "2026-03-30T08:47:23.827Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/84/36c6dcfddc093e108141f757c407902a05085e0c328007cb090d56646cdf/grpcio-1.80.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2ed770b4c06984f3b47eb0517b1c69ad0b84ef3f40128f51448433be904634cd", size = 7383811, upload-time = "2026-03-30T08:47:26.517Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/ef/f3a77e3dc5b471a0ec86c564c98d6adfa3510d38f8ee99010410858d591e/grpcio-1.80.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:256507e2f524092f1473071a05e65a5b10d84b82e3ff24c5b571513cfaa61e2f", size = 8393860, upload-time = "2026-03-30T08:47:29.439Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/8d/9d4d27ed7f33d109c50d6b5ce578a9914aa68edab75d65869a17e630a8d1/grpcio-1.80.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:9a6284a5d907c37db53350645567c522be314bac859a64a7a5ca63b77bb7958f", size = 7830132, upload-time = "2026-03-30T08:47:33.254Z" },
+ { url = "https://files.pythonhosted.org/packages/14/e4/9990b41c6d7a44e1e9dee8ac11d7a9802ba1378b40d77468a7761d1ad288/grpcio-1.80.0-cp312-cp312-win32.whl", hash = "sha256:c71309cfce2f22be26aa4a847357c502db6c621f1a49825ae98aa0907595b193", size = 4140904, upload-time = "2026-03-30T08:47:35.319Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/2c/296f6138caca1f4b92a31ace4ae1b87dab692fc16a7a3417af3bb3c805bf/grpcio-1.80.0-cp312-cp312-win_amd64.whl", hash = "sha256:9fe648599c0e37594c4809d81a9e77bd138cc82eb8baa71b6a86af65426723ff", size = 4880944, upload-time = "2026-03-30T08:47:37.831Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/3a/7c3c25789e3f069e581dc342e03613c5b1cb012c4e8c7d9d5cf960a75856/grpcio-1.80.0-cp313-cp313-linux_armv7l.whl", hash = "sha256:e9e408fc016dffd20661f0126c53d8a31c2821b5c13c5d67a0f5ed5de93319ad", size = 6017243, upload-time = "2026-03-30T08:47:40.075Z" },
+ { url = "https://files.pythonhosted.org/packages/04/19/21a9806eb8240e174fd1ab0cd5b9aa948bb0e05c2f2f55f9d5d7405e6d08/grpcio-1.80.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:92d787312e613754d4d8b9ca6d3297e69994a7912a32fa38c4c4e01c272974b0", size = 12010840, upload-time = "2026-03-30T08:47:43.11Z" },
+ { url = "https://files.pythonhosted.org/packages/18/3a/23347d35f76f639e807fb7a36fad3068aed100996849a33809591f26eca6/grpcio-1.80.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8ac393b58aa16991a2f1144ec578084d544038c12242da3a215966b512904d0f", size = 6567644, upload-time = "2026-03-30T08:47:46.806Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/40/96e07ecb604a6a67ae6ab151e3e35b132875d98bc68ec65f3e5ab3e781d7/grpcio-1.80.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:68e5851ac4b9afe07e7f84483803ad167852570d65326b34d54ca560bfa53fb6", size = 7277830, upload-time = "2026-03-30T08:47:49.643Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/e2/da1506ecea1f34a5e365964644b35edef53803052b763ca214ba3870c856/grpcio-1.80.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:873ff5d17d68992ef6605330127425d2fc4e77e612fa3c3e0ed4e668685e3140", size = 6783216, upload-time = "2026-03-30T08:47:52.817Z" },
+ { url = "https://files.pythonhosted.org/packages/44/83/3b20ff58d0c3b7f6caaa3af9a4174d4023701df40a3f39f7f1c8e7c48f9d/grpcio-1.80.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:2bea16af2750fd0a899bf1abd9022244418b55d1f37da2202249ba4ba673838d", size = 7385866, upload-time = "2026-03-30T08:47:55.687Z" },
+ { url = "https://files.pythonhosted.org/packages/47/45/55c507599c5520416de5eefecc927d6a0d7af55e91cfffb2e410607e5744/grpcio-1.80.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:ba0db34f7e1d803a878284cd70e4c63cb6ae2510ba51937bf8f45ba997cefcf7", size = 8391602, upload-time = "2026-03-30T08:47:58.303Z" },
+ { url = "https://files.pythonhosted.org/packages/10/bb/dd06f4c24c01db9cf11341b547d0a016b2c90ed7dbbb086a5710df7dd1d7/grpcio-1.80.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:8eb613f02d34721f1acf3626dfdb3545bd3c8505b0e52bf8b5710a28d02e8aa7", size = 7826752, upload-time = "2026-03-30T08:48:01.311Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/1e/9d67992ba23371fd63d4527096eb8c6b76d74d52b500df992a3343fd7251/grpcio-1.80.0-cp313-cp313-win32.whl", hash = "sha256:93b6f823810720912fd131f561f91f5fed0fda372b6b7028a2681b8194d5d294", size = 4142310, upload-time = "2026-03-30T08:48:04.594Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/e6/283326a27da9e2c3038bc93eeea36fb118ce0b2d03922a9cda6688f53c5b/grpcio-1.80.0-cp313-cp313-win_amd64.whl", hash = "sha256:e172cf795a3ba5246d3529e4d34c53db70e888fa582a8ffebd2e6e48bc0cba50", size = 4882833, upload-time = "2026-03-30T08:48:07.363Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/6d/e65307ce20f5a09244ba9e9d8476e99fb039de7154f37fb85f26978b59c3/grpcio-1.80.0-cp314-cp314-linux_armv7l.whl", hash = "sha256:3d4147a97c8344d065d01bbf8b6acec2cf86fb0400d40696c8bdad34a64ffc0e", size = 6017376, upload-time = "2026-03-30T08:48:10.005Z" },
+ { url = "https://files.pythonhosted.org/packages/69/10/9cef5d9650c72625a699c549940f0abb3c4bfdb5ed45a5ce431f92f31806/grpcio-1.80.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:d8e11f167935b3eb089ac9038e1a063e6d7dbe995c0bb4a661e614583352e76f", size = 12018133, upload-time = "2026-03-30T08:48:12.927Z" },
+ { url = "https://files.pythonhosted.org/packages/04/82/983aabaad82ba26113caceeb9091706a0696b25da004fe3defb5b346e15b/grpcio-1.80.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f14b618fc30de822681ee986cfdcc2d9327229dc4c98aed16896761cacd468b9", size = 6574748, upload-time = "2026-03-30T08:48:16.386Z" },
+ { url = "https://files.pythonhosted.org/packages/07/d7/031666ef155aa0bf399ed7e19439656c38bbd143779ae0861b038ce82abd/grpcio-1.80.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:4ed39fbdcf9b87370f6e8df4e39ca7b38b3e5e9d1b0013c7b6be9639d6578d14", size = 7277711, upload-time = "2026-03-30T08:48:19.627Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/43/f437a78f7f4f1d311804189e8f11fb311a01049b2e08557c1068d470cb2e/grpcio-1.80.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2dcc70e9f0ba987526e8e8603a610fb4f460e42899e74e7a518bf3c68fe1bf05", size = 6785372, upload-time = "2026-03-30T08:48:22.373Z" },
+ { url = "https://files.pythonhosted.org/packages/93/3d/f6558e9c6296cb4227faa5c43c54a34c68d32654b829f53288313d16a86e/grpcio-1.80.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:448c884b668b868562b1bda833c5fce6272d26e1926ec46747cda05741d302c1", size = 7395268, upload-time = "2026-03-30T08:48:25.638Z" },
+ { url = "https://files.pythonhosted.org/packages/06/21/0fdd77e84720b08843c371a2efa6f2e19dbebf56adc72df73d891f5506f0/grpcio-1.80.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a1dc80fe55685b4a543555e6eef975303b36c8db1023b1599b094b92aa77965f", size = 8392000, upload-time = "2026-03-30T08:48:28.974Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/68/67f4947ed55d2e69f2cc199ab9fd85e0a0034d813bbeef84df6d2ba4d4b7/grpcio-1.80.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:31b9ac4ad1aa28ffee5503821fafd09e4da0a261ce1c1281c6c8da0423c83b6e", size = 7828477, upload-time = "2026-03-30T08:48:32.054Z" },
+ { url = "https://files.pythonhosted.org/packages/44/b6/8d4096691b2e385e8271911a0de4f35f0a6c7d05aff7098e296c3de86939/grpcio-1.80.0-cp314-cp314-win32.whl", hash = "sha256:367ce30ba67d05e0592470428f0ec1c31714cab9ef19b8f2e37be1f4c7d32fae", size = 4218563, upload-time = "2026-03-30T08:48:34.538Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/8c/bbe6baf2557262834f2070cf668515fa308b2d38a4bbf771f8f7872a7036/grpcio-1.80.0-cp314-cp314-win_amd64.whl", hash = "sha256:3b01e1f5464c583d2f567b2e46ff0d516ef979978f72091fd81f5ab7fa6e2e7f", size = 5019457, upload-time = "2026-03-30T08:48:37.308Z" },
+]
+
+[[package]]
+name = "grpcio-tools"
+version = "1.80.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "grpcio" },
+ { name = "protobuf" },
+ { name = "setuptools" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/94/c8/1223f29c84a143ae9a56c084fc96894de0ba84b6e8d60a26241abd81d278/grpcio_tools-1.80.0.tar.gz", hash = "sha256:26052b19c6ce0dcf52d1024496aea3e2bdfa864159f06dc7b97b22d041a94b26", size = 6133212, upload-time = "2026-03-30T08:52:39.077Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0c/b9/65929df8c9614792db900a8e45d4997fadbd1734c827da3f0eb1f2fe4866/grpcio_tools-1.80.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:d19d5a8244311947b96f749c417b32d144641c6953f1164824579e1f0a51d040", size = 2550856, upload-time = "2026-03-30T08:50:57.3Z" },
+ { url = "https://files.pythonhosted.org/packages/28/17/af1557544d68d1aeca9d9ea53ed16524022d521fec6ba334ab3530e9c1a6/grpcio_tools-1.80.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:fb599a3dc89ed1bb24489a2724b2f6dd4cddbbf0f7bdd69c073477bab0dc7554", size = 5710883, upload-time = "2026-03-30T08:51:00.077Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/48/aa9b4f7519ca972bc40d315d5c28f05ca28fa08de13d4e8b69f551b798ab/grpcio_tools-1.80.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:623ee31fc2ff7df9a987b4f3d139c30af17ce46a861ae0e25fb8c112daa32dd8", size = 2598004, upload-time = "2026-03-30T08:51:02.102Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/b8/b01371c119924b3beca1fe3f047b1bc2cdc66b3d37f0f3acc9d10c567a43/grpcio_tools-1.80.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:b46570a68378539ee2b75a5a43202561f8d753c832798b1047099e3c551cf5d6", size = 2909568, upload-time = "2026-03-30T08:51:04.159Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/7c/1108f7bdb58475a7e701ec89b55eb494538b6e76acd211ba0d4cc5fd28e8/grpcio_tools-1.80.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:51caf99c28999e7e0f97e9cea190c1405b7681a57bb2e0631205accd92b43fa4", size = 2660938, upload-time = "2026-03-30T08:51:06.126Z" },
+ { url = "https://files.pythonhosted.org/packages/67/59/d1c0063d4cd3b85363c7044ff3e5159d6d5df96e2692a9a5312d9c8cb290/grpcio_tools-1.80.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:cdaa1c9aa8d3a87891a96700cadd29beec214711d6522818d207277f6452567c", size = 3113814, upload-time = "2026-03-30T08:51:08.834Z" },
+ { url = "https://files.pythonhosted.org/packages/76/21/18d34a4efe524c903cf66b0cfa5260d81f277b6ae668b647edf795df9ce5/grpcio_tools-1.80.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:3399b5fd7b59bcffd59c6b9975a969d9f37a3c87f3e3d63c3a09c147907acb0d", size = 3662793, upload-time = "2026-03-30T08:51:11.094Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/40/cf2d9295a6bd593244ea703858f8fc2efd315046ca3ef7c6f9ebc5b810fa/grpcio_tools-1.80.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:9c6abc08d3485b2aac99bb58afcd31dc6cd4316ce36cf263ff09cb6df15f287f", size = 3329149, upload-time = "2026-03-30T08:51:13.066Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/1d/fc34b32167966df20d69429b71dfca83c48434b047a5ac4fd6cd91ca4eed/grpcio_tools-1.80.0-cp312-cp312-win32.whl", hash = "sha256:18c51e07652ac7386fcdbd11866f8d55a795de073337c12447b5805575339f74", size = 997519, upload-time = "2026-03-30T08:51:14.87Z" },
+ { url = "https://files.pythonhosted.org/packages/91/98/6d6563cdf51085b75f8ec24605c6f2ce84197571878ca8ab4af949c6be2d/grpcio_tools-1.80.0-cp312-cp312-win_amd64.whl", hash = "sha256:ac6fdd42d5bb18f0d903a067e2825be172deff70cf197164b6f65676cb506c9b", size = 1162407, upload-time = "2026-03-30T08:51:16.793Z" },
+ { url = "https://files.pythonhosted.org/packages/44/d9/f7887a4805939e9a85d03744b66fc02575dc1df3c3e8b4d9ec000ee7a33d/grpcio_tools-1.80.0-cp313-cp313-linux_armv7l.whl", hash = "sha256:e7046837859bbfd10b01786056145480155c16b222c9e209215b68d3be13060e", size = 2550319, upload-time = "2026-03-30T08:51:19.117Z" },
+ { url = "https://files.pythonhosted.org/packages/57/5a/c8a05b32bd7203f1b9f4c0151090a2d6179d6c97692d32f2066dc29c67a6/grpcio_tools-1.80.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:a447f28958a8fe84ff0d9d3d9473868feb27ee4a9c9c805e66f5b670121cec59", size = 5709681, upload-time = "2026-03-30T08:51:21.991Z" },
+ { url = "https://files.pythonhosted.org/packages/82/6b/794350ed645c12c310008f97068f6a6fd927150b0d0d08aad1d909e880b1/grpcio_tools-1.80.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:75f00450e08fe648ad8a1eeb25bc52219679d54cdd02f04dfdddc747309d83f6", size = 2596820, upload-time = "2026-03-30T08:51:24.323Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/b2/b39e7b79f7c878135e0784a53cd7260ee77260c8c7f2c9e46bca8e05d017/grpcio_tools-1.80.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:3db830eaff1f2c2797328f2fa86c9dcdbd7d81af573a68db81e27afa2182a611", size = 2909193, upload-time = "2026-03-30T08:51:27.025Z" },
+ { url = "https://files.pythonhosted.org/packages/10/f3/abe089b058f87f9910c9a458409505cbeb0b3e1c2d993a79721d02ee6a32/grpcio_tools-1.80.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7982b5fe42f012686b667dda12916884de95c4b1c65ff64371fb7232a1474b23", size = 2660197, upload-time = "2026-03-30T08:51:29.392Z" },
+ { url = "https://files.pythonhosted.org/packages/09/c3/3f7806ad8b731d8a89fe3c6ed496473abd1ef4c9c42c9e9a8836ce96e377/grpcio_tools-1.80.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:6451b3f4eb52d12c7f32d04bf8e0185f80521f3f088ad04b8d222b3a4819c71e", size = 3113144, upload-time = "2026-03-30T08:51:31.671Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/f5/415ef205e0b7e75d2a2005df6120145c4f02fda28d7b3715b55d924fe1a4/grpcio_tools-1.80.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:258bc30654a9a2236be4ca8e2ad443e2ac6db7c8cc20454d34cce60265922726", size = 3661897, upload-time = "2026-03-30T08:51:34.849Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/d3/2ad54764c2a9547080dd8518f4a4dc7899c7e6e747a1b1de542ce6a12066/grpcio_tools-1.80.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:865a2b8e6334c838976ab02a322cbd55c863d2eaf3c1e1a0255883c63996772a", size = 3328786, upload-time = "2026-03-30T08:51:37.265Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/63/23ab7db01f9630ab4f3742a2fc9fbff38b0cfc30c976114f913950664a75/grpcio_tools-1.80.0-cp313-cp313-win32.whl", hash = "sha256:f760ac1722f33e774814c37b6aa0444143f612e85088ead7447a0e9cd306a1f1", size = 997087, upload-time = "2026-03-30T08:51:39.137Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/af/b1c1c4423fb49cb7c8e9d2c02196b038c44160b7028b425466743c6c81fa/grpcio_tools-1.80.0-cp313-cp313-win_amd64.whl", hash = "sha256:7843b9ac6ff8ca508424d0dd968bd9a1a4559967e4a290f26be5bd6f04af2234", size = 1162167, upload-time = "2026-03-30T08:51:41.498Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/44/7beeee2348f9f412804f5bf80b7d13b81d522bf926a338ae3da46b2213b7/grpcio_tools-1.80.0-cp314-cp314-linux_armv7l.whl", hash = "sha256:12f950470449dbeec78317dbc090add7a00eb6ca812af7b0538ab7441e0a42c3", size = 2550303, upload-time = "2026-03-30T08:51:44.373Z" },
+ { url = "https://files.pythonhosted.org/packages/2d/aa/f77dd85409a1855f8c6319ffc69d81e8c3ffe122ee3a7136653e1991d8b6/grpcio_tools-1.80.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:d3f9a376a29c9adf62bb56f7ff5bc81eb4abeaf53d1e7dde5015564832901a51", size = 5709778, upload-time = "2026-03-30T08:51:47.112Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/7c/ab7af4883ebdfdc228b853de89fed409703955e8d47285b321a5794856bd/grpcio_tools-1.80.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:1ba1ffbf2cff71533615e2c5a138ed5569611eec9ae7f9c67b8898e127b54ac0", size = 2597928, upload-time = "2026-03-30T08:51:49.494Z" },
+ { url = "https://files.pythonhosted.org/packages/22/e8/4381a963d472e3ab6690ba067ed2b1f1abf8518b10f402678bd2dcb79a54/grpcio_tools-1.80.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:13f60f8d9397c514c6745a967d22b5c8c698347e88deebca1ff2e1b94555e450", size = 2909333, upload-time = "2026-03-30T08:51:52.124Z" },
+ { url = "https://files.pythonhosted.org/packages/94/cb/356b5fdf79dd99455b425fb16302fe60995554ceb721afbf3cf770a19208/grpcio_tools-1.80.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:88d77bad5dd3cd5e6f952c4ecdd0ee33e0c02ecfc2e4b0cbee3391ac19e0a431", size = 2660217, upload-time = "2026-03-30T08:51:55.066Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/d7/1752018cc2c36b2c5612051379e2e5f59f2dbe612de23e817d2f066a9487/grpcio_tools-1.80.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:017945c3e98a4ed1c4e21399781b4137fc08dfc1f802c8ace2e64ef52d32b142", size = 3113896, upload-time = "2026-03-30T08:51:57.3Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/17/695bbe454f70df35c03e22b48c5314683b913d3e6ed35ec90d065418c1ab/grpcio_tools-1.80.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a33e265d4db803495007a6c623eafb0f6b9bb123ff4a0af89e44567dad809b88", size = 3661950, upload-time = "2026-03-30T08:51:59.867Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/d0/533d87629ec823c02c9169ee20228f734c264b209dcdf55268b5a14cde0a/grpcio_tools-1.80.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6c129da370c5f85f569be2e545317dda786a60dd51d7deea29b03b0c05f6aac3", size = 3328755, upload-time = "2026-03-30T08:52:02.942Z" },
+ { url = "https://files.pythonhosted.org/packages/08/a1/504d7838770c73a9761e8a8ff4869dba1146b44f297ff0ac6641481942d3/grpcio_tools-1.80.0-cp314-cp314-win32.whl", hash = "sha256:25742de5958ae4325249a37e724e7c0e5120f8e302a24a977ebd1737b48a5e97", size = 1019620, upload-time = "2026-03-30T08:52:05.342Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/75/8b7cd281c5cdfb4ca2c308f7e9b2799bab2be6e7a9e9212ea5a82e2aecd4/grpcio_tools-1.80.0-cp314-cp314-win_amd64.whl", hash = "sha256:bbf8eeef78fda1966f732f79c1c802fadd5cfd203d845d2af4d314d18569069c", size = 1194210, upload-time = "2026-03-30T08:52:08.105Z" },
]
[[package]]
@@ -1686,47 +1773,50 @@ wheels = [
[[package]]
name = "hatchling"
-version = "1.28.0"
+version = "1.29.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
- { name = "packaging" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
{ name = "pathspec" },
{ name = "pluggy" },
- { name = "tomli", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
{ name = "trove-classifiers" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/0b/8e/e480359492affde4119a131da729dd26da742c2c9b604dff74836e47eef9/hatchling-1.28.0.tar.gz", hash = "sha256:4d50b02aece6892b8cd0b3ce6c82cb218594d3ec5836dbde75bf41a21ab004c8", size = 55365, upload-time = "2025-11-27T00:31:13.766Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/cf/9c/b4cfe330cd4f49cff17fd771154730555fa4123beb7f292cf0098b4e6c20/hatchling-1.29.0.tar.gz", hash = "sha256:793c31816d952cee405b83488ce001c719f325d9cda69f1fc4cd750527640ea6", size = 55656, upload-time = "2026-02-23T19:42:06.539Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/0d/a5/48cb7efb8b4718b1a4c0c331e3364a3a33f614ff0d6afd2b93ee883d3c47/hatchling-1.28.0-py3-none-any.whl", hash = "sha256:dc48722b68b3f4bbfa3ff618ca07cdea6750e7d03481289ffa8be1521d18a961", size = 76075, upload-time = "2025-11-27T00:31:12.544Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/8a/44032265776062a89171285ede55a0bdaadc8ac00f27f0512a71a9e3e1c8/hatchling-1.29.0-py3-none-any.whl", hash = "sha256:50af9343281f34785fab12da82e445ed987a6efb34fd8c2fc0f6e6630dbcc1b0", size = 76356, upload-time = "2026-02-23T19:42:05.197Z" },
]
[[package]]
name = "hf-xet"
-version = "1.2.0"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/5e/6e/0f11bacf08a67f7fb5ee09740f2ca54163863b07b70d579356e9222ce5d8/hf_xet-1.2.0.tar.gz", hash = "sha256:a8c27070ca547293b6890c4bf389f713f80e8c478631432962bb7f4bc0bd7d7f", size = 506020, upload-time = "2025-10-24T19:04:32.129Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/9e/a5/85ef910a0aa034a2abcfadc360ab5ac6f6bc4e9112349bd40ca97551cff0/hf_xet-1.2.0-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:ceeefcd1b7aed4956ae8499e2199607765fbd1c60510752003b6cc0b8413b649", size = 2861870, upload-time = "2025-10-24T19:04:11.422Z" },
- { url = "https://files.pythonhosted.org/packages/ea/40/e2e0a7eb9a51fe8828ba2d47fe22a7e74914ea8a0db68a18c3aa7449c767/hf_xet-1.2.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:b70218dd548e9840224df5638fdc94bd033552963cfa97f9170829381179c813", size = 2717584, upload-time = "2025-10-24T19:04:09.586Z" },
- { url = "https://files.pythonhosted.org/packages/a5/7d/daf7f8bc4594fdd59a8a596f9e3886133fdc68e675292218a5e4c1b7e834/hf_xet-1.2.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7d40b18769bb9a8bc82a9ede575ce1a44c75eb80e7375a01d76259089529b5dc", size = 3315004, upload-time = "2025-10-24T19:04:00.314Z" },
- { url = "https://files.pythonhosted.org/packages/b1/ba/45ea2f605fbf6d81c8b21e4d970b168b18a53515923010c312c06cd83164/hf_xet-1.2.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:cd3a6027d59cfb60177c12d6424e31f4b5ff13d8e3a1247b3a584bf8977e6df5", size = 3222636, upload-time = "2025-10-24T19:03:58.111Z" },
- { url = "https://files.pythonhosted.org/packages/4a/1d/04513e3cab8f29ab8c109d309ddd21a2705afab9d52f2ba1151e0c14f086/hf_xet-1.2.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:6de1fc44f58f6dd937956c8d304d8c2dea264c80680bcfa61ca4a15e7b76780f", size = 3408448, upload-time = "2025-10-24T19:04:20.951Z" },
- { url = "https://files.pythonhosted.org/packages/f0/7c/60a2756d7feec7387db3a1176c632357632fbe7849fce576c5559d4520c7/hf_xet-1.2.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:f182f264ed2acd566c514e45da9f2119110e48a87a327ca271027904c70c5832", size = 3503401, upload-time = "2025-10-24T19:04:22.549Z" },
- { url = "https://files.pythonhosted.org/packages/4e/64/48fffbd67fb418ab07451e4ce641a70de1c40c10a13e25325e24858ebe5a/hf_xet-1.2.0-cp313-cp313t-win_amd64.whl", hash = "sha256:293a7a3787e5c95d7be1857358a9130694a9c6021de3f27fa233f37267174382", size = 2900866, upload-time = "2025-10-24T19:04:33.461Z" },
- { url = "https://files.pythonhosted.org/packages/e2/51/f7e2caae42f80af886db414d4e9885fac959330509089f97cccb339c6b87/hf_xet-1.2.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:10bfab528b968c70e062607f663e21e34e2bba349e8038db546646875495179e", size = 2861861, upload-time = "2025-10-24T19:04:19.01Z" },
- { url = "https://files.pythonhosted.org/packages/6e/1d/a641a88b69994f9371bd347f1dd35e5d1e2e2460a2e350c8d5165fc62005/hf_xet-1.2.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:2a212e842647b02eb6a911187dc878e79c4aa0aa397e88dd3b26761676e8c1f8", size = 2717699, upload-time = "2025-10-24T19:04:17.306Z" },
- { url = "https://files.pythonhosted.org/packages/df/e0/e5e9bba7d15f0318955f7ec3f4af13f92e773fbb368c0b8008a5acbcb12f/hf_xet-1.2.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:30e06daccb3a7d4c065f34fc26c14c74f4653069bb2b194e7f18f17cbe9939c0", size = 3314885, upload-time = "2025-10-24T19:04:07.642Z" },
- { url = "https://files.pythonhosted.org/packages/21/90/b7fe5ff6f2b7b8cbdf1bd56145f863c90a5807d9758a549bf3d916aa4dec/hf_xet-1.2.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:29c8fc913a529ec0a91867ce3d119ac1aac966e098cf49501800c870328cc090", size = 3221550, upload-time = "2025-10-24T19:04:05.55Z" },
- { url = "https://files.pythonhosted.org/packages/6f/cb/73f276f0a7ce46cc6a6ec7d6c7d61cbfe5f2e107123d9bbd0193c355f106/hf_xet-1.2.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:66e159cbfcfbb29f920db2c09ed8b660eb894640d284f102ada929b6e3dc410a", size = 3408010, upload-time = "2025-10-24T19:04:28.598Z" },
- { url = "https://files.pythonhosted.org/packages/b8/1e/d642a12caa78171f4be64f7cd9c40e3ca5279d055d0873188a58c0f5fbb9/hf_xet-1.2.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:9c91d5ae931510107f148874e9e2de8a16052b6f1b3ca3c1b12f15ccb491390f", size = 3503264, upload-time = "2025-10-24T19:04:30.397Z" },
- { url = "https://files.pythonhosted.org/packages/17/b5/33764714923fa1ff922770f7ed18c2daae034d21ae6e10dbf4347c854154/hf_xet-1.2.0-cp314-cp314t-win_amd64.whl", hash = "sha256:210d577732b519ac6ede149d2f2f34049d44e8622bf14eb3d63bbcd2d4b332dc", size = 2901071, upload-time = "2025-10-24T19:04:37.463Z" },
- { url = "https://files.pythonhosted.org/packages/96/2d/22338486473df5923a9ab7107d375dbef9173c338ebef5098ef593d2b560/hf_xet-1.2.0-cp37-abi3-macosx_10_12_x86_64.whl", hash = "sha256:46740d4ac024a7ca9b22bebf77460ff43332868b661186a8e46c227fdae01848", size = 2866099, upload-time = "2025-10-24T19:04:15.366Z" },
- { url = "https://files.pythonhosted.org/packages/7f/8c/c5becfa53234299bc2210ba314eaaae36c2875e0045809b82e40a9544f0c/hf_xet-1.2.0-cp37-abi3-macosx_11_0_arm64.whl", hash = "sha256:27df617a076420d8845bea087f59303da8be17ed7ec0cd7ee3b9b9f579dff0e4", size = 2722178, upload-time = "2025-10-24T19:04:13.695Z" },
- { url = "https://files.pythonhosted.org/packages/9a/92/cf3ab0b652b082e66876d08da57fcc6fa2f0e6c70dfbbafbd470bb73eb47/hf_xet-1.2.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3651fd5bfe0281951b988c0facbe726aa5e347b103a675f49a3fa8144c7968fd", size = 3320214, upload-time = "2025-10-24T19:04:03.596Z" },
- { url = "https://files.pythonhosted.org/packages/46/92/3f7ec4a1b6a65bf45b059b6d4a5d38988f63e193056de2f420137e3c3244/hf_xet-1.2.0-cp37-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:d06fa97c8562fb3ee7a378dd9b51e343bc5bc8190254202c9771029152f5e08c", size = 3229054, upload-time = "2025-10-24T19:04:01.949Z" },
- { url = "https://files.pythonhosted.org/packages/0b/dd/7ac658d54b9fb7999a0ccb07ad863b413cbaf5cf172f48ebcd9497ec7263/hf_xet-1.2.0-cp37-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:4c1428c9ae73ec0939410ec73023c4f842927f39db09b063b9482dac5a3bb737", size = 3413812, upload-time = "2025-10-24T19:04:24.585Z" },
- { url = "https://files.pythonhosted.org/packages/92/68/89ac4e5b12a9ff6286a12174c8538a5930e2ed662091dd2572bbe0a18c8a/hf_xet-1.2.0-cp37-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:a55558084c16b09b5ed32ab9ed38421e2d87cf3f1f89815764d1177081b99865", size = 3508920, upload-time = "2025-10-24T19:04:26.927Z" },
- { url = "https://files.pythonhosted.org/packages/cb/44/870d44b30e1dcfb6a65932e3e1506c103a8a5aea9103c337e7a53180322c/hf_xet-1.2.0-cp37-abi3-win_amd64.whl", hash = "sha256:e6584a52253f72c9f52f9e549d5895ca7a471608495c4ecaa6cc73dba2b24d69", size = 2905735, upload-time = "2025-10-24T19:04:35.928Z" },
+version = "1.4.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/09/08/23c84a26716382c89151b5b447b4beb19e3345f3a93d3b73009a71a57ad3/hf_xet-1.4.2.tar.gz", hash = "sha256:b7457b6b482d9e0743bd116363239b1fa904a5e65deede350fbc0c4ea67c71ea", size = 672357, upload-time = "2026-03-13T06:58:51.077Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/18/06/e8cf74c3c48e5485c7acc5a990d0d8516cdfb5fdf80f799174f1287cc1b5/hf_xet-1.4.2-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:ac8202ae1e664b2c15cdfc7298cbb25e80301ae596d602ef7870099a126fcad4", size = 3796125, upload-time = "2026-03-13T06:58:33.177Z" },
+ { url = "https://files.pythonhosted.org/packages/66/d4/b73ebab01cbf60777323b7de9ef05550790451eb5172a220d6b9845385ec/hf_xet-1.4.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:6d2f8ee39fa9fba9af929f8c0d0482f8ee6e209179ad14a909b6ad78ffcb7c81", size = 3555985, upload-time = "2026-03-13T06:58:31.797Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/e7/ded6d1bd041c3f2bca9e913a0091adfe32371988e047dd3a68a2463c15a2/hf_xet-1.4.2-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4642a6cf249c09da8c1f87fe50b24b2a3450b235bf8adb55700b52f0ea6e2eb6", size = 4212085, upload-time = "2026-03-13T06:58:24.323Z" },
+ { url = "https://files.pythonhosted.org/packages/97/c1/a0a44d1f98934f7bdf17f7a915b934f9fca44bb826628c553589900f6df8/hf_xet-1.4.2-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:769431385e746c92dc05492dde6f687d304584b89c33d79def8367ace06cb555", size = 3988266, upload-time = "2026-03-13T06:58:22.887Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/82/be713b439060e7d1f1d93543c8053d4ef2fe7e6922c5b31642eaa26f3c4b/hf_xet-1.4.2-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:c9dd1c1bc4cc56168f81939b0e05b4c36dd2d28c13dc1364b17af89aa0082496", size = 4188513, upload-time = "2026-03-13T06:58:40.858Z" },
+ { url = "https://files.pythonhosted.org/packages/21/a6/cbd4188b22abd80ebd0edbb2b3e87f2633e958983519980815fb8314eae5/hf_xet-1.4.2-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:fca58a2ae4e6f6755cc971ac6fcdf777ea9284d7e540e350bb000813b9a3008d", size = 4428287, upload-time = "2026-03-13T06:58:42.601Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/4e/84e45b25e2e3e903ed3db68d7eafa96dae9a1d1f6d0e7fc85120347a852f/hf_xet-1.4.2-cp313-cp313t-win_amd64.whl", hash = "sha256:163aab46854ccae0ab6a786f8edecbbfbaa38fcaa0184db6feceebf7000c93c0", size = 3665574, upload-time = "2026-03-13T06:58:53.881Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/71/c5ac2b9a7ae39c14e91973035286e73911c31980fe44e7b1d03730c00adc/hf_xet-1.4.2-cp313-cp313t-win_arm64.whl", hash = "sha256:09b138422ecbe50fd0c84d4da5ff537d27d487d3607183cd10e3e53f05188e82", size = 3528760, upload-time = "2026-03-13T06:58:52.187Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/0f/fcd2504015eab26358d8f0f232a1aed6b8d363a011adef83fe130bff88f7/hf_xet-1.4.2-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:949dcf88b484bb9d9276ca83f6599e4aa03d493c08fc168c124ad10b2e6f75d7", size = 3796493, upload-time = "2026-03-13T06:58:39.267Z" },
+ { url = "https://files.pythonhosted.org/packages/82/56/19c25105ff81731ca6d55a188b5de2aa99d7a2644c7aa9de1810d5d3b726/hf_xet-1.4.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:41659966020d59eb9559c57de2cde8128b706a26a64c60f0531fa2318f409418", size = 3555797, upload-time = "2026-03-13T06:58:37.546Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/e3/8933c073186849b5e06762aa89847991d913d10a95d1603eb7f2c3834086/hf_xet-1.4.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5c588e21d80010119458dd5d02a69093f0d115d84e3467efe71ffb2c67c19146", size = 4212127, upload-time = "2026-03-13T06:58:30.539Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/01/f89ebba4e369b4ed699dcb60d3152753870996f41c6d22d3d7cac01310e1/hf_xet-1.4.2-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:a296744d771a8621ad1d50c098d7ab975d599800dae6d48528ba3944e5001ba0", size = 3987788, upload-time = "2026-03-13T06:58:29.139Z" },
+ { url = "https://files.pythonhosted.org/packages/84/4d/8a53e5ffbc2cc33bbf755382ac1552c6d9af13f623ed125fe67cc3e6772f/hf_xet-1.4.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:f563f7efe49588b7d0629d18d36f46d1658fe7e08dce3fa3d6526e1c98315e2d", size = 4188315, upload-time = "2026-03-13T06:58:48.017Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/b8/b7a1c1b5592254bd67050632ebbc1b42cc48588bf4757cb03c2ef87e704a/hf_xet-1.4.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5b2e0132c56d7ee1bf55bdb638c4b62e7106f6ac74f0b786fed499d5548c5570", size = 4428306, upload-time = "2026-03-13T06:58:49.502Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/0c/40779e45b20e11c7c5821a94135e0207080d6b3d76e7b78ccb413c6f839b/hf_xet-1.4.2-cp314-cp314t-win_amd64.whl", hash = "sha256:2f45c712c2fa1215713db10df6ac84b49d0e1c393465440e9cb1de73ecf7bbf6", size = 3665826, upload-time = "2026-03-13T06:58:59.88Z" },
+ { url = "https://files.pythonhosted.org/packages/51/4c/e2688c8ad1760d7c30f7c429c79f35f825932581bc7c9ec811436d2f21a0/hf_xet-1.4.2-cp314-cp314t-win_arm64.whl", hash = "sha256:6d53df40616f7168abfccff100d232e9d460583b9d86fa4912c24845f192f2b8", size = 3529113, upload-time = "2026-03-13T06:58:58.491Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/86/b40b83a2ff03ef05c4478d2672b1fc2b9683ff870e2b25f4f3af240f2e7b/hf_xet-1.4.2-cp37-abi3-macosx_10_12_x86_64.whl", hash = "sha256:71f02d6e4cdd07f344f6844845d78518cc7186bd2bc52d37c3b73dc26a3b0bc5", size = 3800339, upload-time = "2026-03-13T06:58:36.245Z" },
+ { url = "https://files.pythonhosted.org/packages/64/2e/af4475c32b4378b0e92a587adb1aa3ec53e3450fd3e5fe0372a874531c00/hf_xet-1.4.2-cp37-abi3-macosx_11_0_arm64.whl", hash = "sha256:e9b38d876e94d4bdcf650778d6ebbaa791dd28de08db9736c43faff06ede1b5a", size = 3559664, upload-time = "2026-03-13T06:58:34.787Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/4c/781267da3188db679e601de18112021a5cb16506fe86b246e22c5401a9c4/hf_xet-1.4.2-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:77e8c180b7ef12d8a96739a4e1e558847002afe9ea63b6f6358b2271a8bdda1c", size = 4217422, upload-time = "2026-03-13T06:58:27.472Z" },
+ { url = "https://files.pythonhosted.org/packages/68/47/d6cf4a39ecf6c7705f887a46f6ef5c8455b44ad9eb0d391aa7e8a2ff7fea/hf_xet-1.4.2-cp37-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:c3b3c6a882016b94b6c210957502ff7877802d0dbda8ad142c8595db8b944271", size = 3992847, upload-time = "2026-03-13T06:58:25.989Z" },
+ { url = "https://files.pythonhosted.org/packages/2d/ef/e80815061abff54697239803948abc665c6b1d237102c174f4f7a9a5ffc5/hf_xet-1.4.2-cp37-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:9d9a634cc929cfbaf2e1a50c0e532ae8c78fa98618426769480c58501e8c8ac2", size = 4193843, upload-time = "2026-03-13T06:58:44.59Z" },
+ { url = "https://files.pythonhosted.org/packages/54/75/07f6aa680575d9646c4167db6407c41340cbe2357f5654c4e72a1b01ca14/hf_xet-1.4.2-cp37-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:6b0932eb8b10317ea78b7da6bab172b17be03bbcd7809383d8d5abd6a2233e04", size = 4432751, upload-time = "2026-03-13T06:58:46.533Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/71/193eabd7e7d4b903c4aa983a215509c6114915a5a237525ec562baddb868/hf_xet-1.4.2-cp37-abi3-win_amd64.whl", hash = "sha256:ad185719fb2e8ac26f88c8100562dbf9dbdcc3d9d2add00faa94b5f106aea53f", size = 3671149, upload-time = "2026-03-13T06:58:57.07Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/7e/ccf239da366b37ba7f0b36095450efae4a64980bdc7ec2f51354205fdf39/hf_xet-1.4.2-cp37-abi3-win_arm64.whl", hash = "sha256:32c012286b581f783653e718c1862aea5b9eb140631685bb0c5e7012c8719a87", size = 3533426, upload-time = "2026-03-13T06:58:55.46Z" },
]
[[package]]
@@ -1772,23 +1862,62 @@ http2 = [
{ name = "h2" },
]
+[[package]]
+name = "httpx-aiohttp"
+version = "0.1.12"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiohttp" },
+ { name = "httpx" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/63/2c/b894861cecf030fb45675ea24aa55b5722e97c602a163d872fca66c5a6d8/httpx_aiohttp-0.1.12.tar.gz", hash = "sha256:81feec51fd82c0ecfa0e9aaf1b1a6c2591260d5e2bcbeb7eb0277a78e610df2c", size = 275945, upload-time = "2025-12-12T10:12:15.283Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/16/8d/85c9701e9af72ca132a1783e2a54364a90c6da832304416a30fc11196ab2/httpx_aiohttp-0.1.12-py3-none-any.whl", hash = "sha256:5b0eac39a7f360fa7867a60bcb46bb1024eada9c01cbfecdb54dc1edb3fb7141", size = 6367, upload-time = "2025-12-12T10:12:14.018Z" },
+]
+
+[[package]]
+name = "httpx-sse"
+version = "0.4.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/0f/4c/751061ffa58615a32c31b2d82e8482be8dd4a89154f003147acee90f2be9/httpx_sse-0.4.3.tar.gz", hash = "sha256:9b1ed0127459a66014aec3c56bebd93da3c1bc8bb6618c8082039a44889a755d", size = 15943, upload-time = "2025-10-10T21:48:22.271Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d2/fd/6668e5aec43ab844de6fc74927e155a3b37bf40d7c3790e49fc0406b6578/httpx_sse-0.4.3-py3-none-any.whl", hash = "sha256:0ac1c9fe3c0afad2e0ebb25a934a59f4c7823b60792691f779fad2c5568830fc", size = 8960, upload-time = "2025-10-10T21:48:21.158Z" },
+]
+
[[package]]
name = "huggingface-hub"
-version = "0.36.0"
+version = "0.36.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "filelock" },
- { name = "fsspec" },
+ { name = "fsspec", version = "2026.2.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.14' or sys_platform != 'win32' or extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
+ { name = "fsspec", version = "2026.3.0", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
{ name = "hf-xet", marker = "platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'" },
- { name = "packaging" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
{ name = "pyyaml" },
{ name = "requests" },
{ name = "tqdm" },
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/98/63/4910c5fa9128fdadf6a9c5ac138e8b1b6cee4ca44bf7915bbfbce4e355ee/huggingface_hub-0.36.0.tar.gz", hash = "sha256:47b3f0e2539c39bf5cde015d63b72ec49baff67b6931c3d97f3f84532e2b8d25", size = 463358, upload-time = "2025-10-23T12:12:01.413Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/7c/b7/8cb61d2eece5fb05a83271da168186721c450eb74e3c31f7ef3169fa475b/huggingface_hub-0.36.2.tar.gz", hash = "sha256:1934304d2fb224f8afa3b87007d58501acfda9215b334eed53072dd5e815ff7a", size = 649782, upload-time = "2026-02-06T09:24:13.098Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a8/af/48ac8483240de756d2438c380746e7130d1c6f75802ef22f3c6d49982787/huggingface_hub-0.36.2-py3-none-any.whl", hash = "sha256:48f0c8eac16145dfce371e9d2d7772854a4f591bcb56c9cf548accf531d54270", size = 566395, upload-time = "2026-02-06T09:24:11.133Z" },
+]
+
+[[package]]
+name = "hypercorn"
+version = "0.18.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "h11" },
+ { name = "h2" },
+ { name = "priority" },
+ { name = "wsproto" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/44/01/39f41a014b83dd5c795217362f2ca9071cf243e6a75bdcd6cd5b944658cc/hypercorn-0.18.0.tar.gz", hash = "sha256:d63267548939c46b0247dc8e5b45a9947590e35e64ee73a23c074aa3cf88e9da", size = 68420, upload-time = "2025-11-08T13:54:04.78Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/cb/bd/1a875e0d592d447cbc02805fd3fe0f497714d6a2583f59d14fa9ebad96eb/huggingface_hub-0.36.0-py3-none-any.whl", hash = "sha256:7bcc9ad17d5b3f07b57c78e79d527102d08313caa278a641993acddcb894548d", size = 566094, upload-time = "2025-10-23T12:11:59.557Z" },
+ { url = "https://files.pythonhosted.org/packages/93/35/850277d1b17b206bd10874c8a9a3f52e059452fb49bb0d22cbb908f6038b/hypercorn-0.18.0-py3-none-any.whl", hash = "sha256:225e268f2c1c2f28f6d8f6db8f40cb8c992963610c5725e13ccfcddccb24b1cd", size = 61640, upload-time = "2025-11-08T13:54:03.202Z" },
]
[[package]]
@@ -1811,11 +1940,11 @@ wheels = [
[[package]]
name = "imagesize"
-version = "1.4.1"
+version = "2.0.0"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/a7/84/62473fb57d61e31fef6e36d64a179c8781605429fd927b5dd608c997be31/imagesize-1.4.1.tar.gz", hash = "sha256:69150444affb9cb0d5cc5a92b3676f0b2fb7cd9ae39e947a5e11a36b4497cd4a", size = 1280026, upload-time = "2022-07-01T12:21:05.687Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/6c/e6/7bf14eeb8f8b7251141944835abd42eb20a658d89084b7e1f3e5fe394090/imagesize-2.0.0.tar.gz", hash = "sha256:8e8358c4a05c304f1fccf7ff96f036e7243a189e9e42e90851993c558cfe9ee3", size = 1773045, upload-time = "2026-03-03T14:18:29.941Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/ff/62/85c4c919272577931d407be5ba5d71c20f0b616d31a0befe0ae45bb79abd/imagesize-1.4.1-py2.py3-none-any.whl", hash = "sha256:0d8d18d08f840c19d0ee7ca1fd82490fdc3729b7ac93f49870406ddde8ef8d8b", size = 8769, upload-time = "2022-07-01T12:21:02.467Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/53/fb7122b71361a0d121b669dcf3d31244ef75badbbb724af388948de543e2/imagesize-2.0.0-py2.py3-none-any.whl", hash = "sha256:5667c5bbb57ab3f1fa4bc366f4fbc971db3d5ed011fd2715fd8001f782718d96", size = 9441, upload-time = "2026-03-03T14:18:27.892Z" },
]
[[package]]
@@ -1891,13 +2020,81 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" },
]
+[[package]]
+name = "jiter"
+version = "0.13.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/0d/5e/4ec91646aee381d01cdb9974e30882c9cd3b8c5d1079d6b5ff4af522439a/jiter-0.13.0.tar.gz", hash = "sha256:f2839f9c2c7e2dffc1bc5929a510e14ce0a946be9365fd1219e7ef342dae14f4", size = 164847, upload-time = "2026-02-02T12:37:56.441Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2e/30/7687e4f87086829955013ca12a9233523349767f69653ebc27036313def9/jiter-0.13.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:0a2bd69fc1d902e89925fc34d1da51b2128019423d7b339a45d9e99c894e0663", size = 307958, upload-time = "2026-02-02T12:35:57.165Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/27/e57f9a783246ed95481e6749cc5002a8a767a73177a83c63ea71f0528b90/jiter-0.13.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f917a04240ef31898182f76a332f508f2cc4b57d2b4d7ad2dbfebbfe167eb505", size = 318597, upload-time = "2026-02-02T12:35:58.591Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/52/e5719a60ac5d4d7c5995461a94ad5ef962a37c8bf5b088390e6fad59b2ff/jiter-0.13.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c1e2b199f446d3e82246b4fd9236d7cb502dc2222b18698ba0d986d2fecc6152", size = 348821, upload-time = "2026-02-02T12:36:00.093Z" },
+ { url = "https://files.pythonhosted.org/packages/61/db/c1efc32b8ba4c740ab3fc2d037d8753f67685f475e26b9d6536a4322bcdd/jiter-0.13.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:04670992b576fa65bd056dbac0c39fe8bd67681c380cb2b48efa885711d9d726", size = 364163, upload-time = "2026-02-02T12:36:01.937Z" },
+ { url = "https://files.pythonhosted.org/packages/55/8a/fb75556236047c8806995671a18e4a0ad646ed255276f51a20f32dceaeec/jiter-0.13.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5a1aff1fbdb803a376d4d22a8f63f8e7ccbce0b4890c26cc7af9e501ab339ef0", size = 483709, upload-time = "2026-02-02T12:36:03.41Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/16/43512e6ee863875693a8e6f6d532e19d650779d6ba9a81593ae40a9088ff/jiter-0.13.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3b3fb8c2053acaef8580809ac1d1f7481a0a0bdc012fd7f5d8b18fb696a5a089", size = 370480, upload-time = "2026-02-02T12:36:04.791Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/4c/09b93e30e984a187bc8aaa3510e1ec8dcbdcd71ca05d2f56aac0492453aa/jiter-0.13.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bdaba7d87e66f26a2c45d8cbadcbfc4bf7884182317907baf39cfe9775bb4d93", size = 360735, upload-time = "2026-02-02T12:36:06.994Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/1b/46c5e349019874ec5dfa508c14c37e29864ea108d376ae26d90bee238cd7/jiter-0.13.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:7b88d649135aca526da172e48083da915ec086b54e8e73a425ba50999468cc08", size = 391814, upload-time = "2026-02-02T12:36:08.368Z" },
+ { url = "https://files.pythonhosted.org/packages/15/9e/26184760e85baee7162ad37b7912797d2077718476bf91517641c92b3639/jiter-0.13.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:e404ea551d35438013c64b4f357b0474c7abf9f781c06d44fcaf7a14c69ff9e2", size = 513990, upload-time = "2026-02-02T12:36:09.993Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/34/2c9355247d6debad57a0a15e76ab1566ab799388042743656e566b3b7de1/jiter-0.13.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:1f4748aad1b4a93c8bdd70f604d0f748cdc0e8744c5547798acfa52f10e79228", size = 548021, upload-time = "2026-02-02T12:36:11.376Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/4a/9f2c23255d04a834398b9c2e0e665382116911dc4d06b795710503cdad25/jiter-0.13.0-cp312-cp312-win32.whl", hash = "sha256:0bf670e3b1445fc4d31612199f1744f67f889ee1bbae703c4b54dc097e5dd394", size = 203024, upload-time = "2026-02-02T12:36:12.682Z" },
+ { url = "https://files.pythonhosted.org/packages/09/ee/f0ae675a957ae5a8f160be3e87acea6b11dc7b89f6b7ab057e77b2d2b13a/jiter-0.13.0-cp312-cp312-win_amd64.whl", hash = "sha256:15db60e121e11fe186c0b15236bd5d18381b9ddacdcf4e659feb96fc6c969c92", size = 205424, upload-time = "2026-02-02T12:36:13.93Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/02/ae611edf913d3cbf02c97cdb90374af2082c48d7190d74c1111dde08bcdd/jiter-0.13.0-cp312-cp312-win_arm64.whl", hash = "sha256:41f92313d17989102f3cb5dd533a02787cdb99454d494344b0361355da52fcb9", size = 186818, upload-time = "2026-02-02T12:36:15.308Z" },
+ { url = "https://files.pythonhosted.org/packages/91/9c/7ee5a6ff4b9991e1a45263bfc46731634c4a2bde27dfda6c8251df2d958c/jiter-0.13.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:1f8a55b848cbabf97d861495cd65f1e5c590246fabca8b48e1747c4dfc8f85bf", size = 306897, upload-time = "2026-02-02T12:36:16.748Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/02/be5b870d1d2be5dd6a91bdfb90f248fbb7dcbd21338f092c6b89817c3dbf/jiter-0.13.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f556aa591c00f2c45eb1b89f68f52441a016034d18b65da60e2d2875bbbf344a", size = 317507, upload-time = "2026-02-02T12:36:18.351Z" },
+ { url = "https://files.pythonhosted.org/packages/da/92/b25d2ec333615f5f284f3a4024f7ce68cfa0604c322c6808b2344c7f5d2b/jiter-0.13.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f7e1d61da332ec412350463891923f960c3073cf1aae93b538f0bb4c8cd46efb", size = 350560, upload-time = "2026-02-02T12:36:19.746Z" },
+ { url = "https://files.pythonhosted.org/packages/be/ec/74dcb99fef0aca9fbe56b303bf79f6bd839010cb18ad41000bf6cc71eec0/jiter-0.13.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:3097d665a27bc96fd9bbf7f86178037db139f319f785e4757ce7ccbf390db6c2", size = 363232, upload-time = "2026-02-02T12:36:21.243Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/37/f17375e0bb2f6a812d4dd92d7616e41917f740f3e71343627da9db2824ce/jiter-0.13.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9d01ecc3a8cbdb6f25a37bd500510550b64ddf9f7d64a107d92f3ccb25035d0f", size = 483727, upload-time = "2026-02-02T12:36:22.688Z" },
+ { url = "https://files.pythonhosted.org/packages/77/d2/a71160a5ae1a1e66c1395b37ef77da67513b0adba73b993a27fbe47eb048/jiter-0.13.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ed9bbc30f5d60a3bdf63ae76beb3f9db280d7f195dfcfa61af792d6ce912d159", size = 370799, upload-time = "2026-02-02T12:36:24.106Z" },
+ { url = "https://files.pythonhosted.org/packages/01/99/ed5e478ff0eb4e8aa5fd998f9d69603c9fd3f32de3bd16c2b1194f68361c/jiter-0.13.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:98fbafb6e88256f4454de33c1f40203d09fc33ed19162a68b3b257b29ca7f663", size = 359120, upload-time = "2026-02-02T12:36:25.519Z" },
+ { url = "https://files.pythonhosted.org/packages/16/be/7ffd08203277a813f732ba897352797fa9493faf8dc7995b31f3d9cb9488/jiter-0.13.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5467696f6b827f1116556cb0db620440380434591e93ecee7fd14d1a491b6daa", size = 390664, upload-time = "2026-02-02T12:36:26.866Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/84/e0787856196d6d346264d6dcccb01f741e5f0bd014c1d9a2ebe149caf4f3/jiter-0.13.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:2d08c9475d48b92892583df9da592a0e2ac49bcd41fae1fec4f39ba6cf107820", size = 513543, upload-time = "2026-02-02T12:36:28.217Z" },
+ { url = "https://files.pythonhosted.org/packages/65/50/ecbd258181c4313cf79bca6c88fb63207d04d5bf5e4f65174114d072aa55/jiter-0.13.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:aed40e099404721d7fcaf5b89bd3b4568a4666358bcac7b6b15c09fb6252ab68", size = 547262, upload-time = "2026-02-02T12:36:29.678Z" },
+ { url = "https://files.pythonhosted.org/packages/27/da/68f38d12e7111d2016cd198161b36e1f042bd115c169255bcb7ec823a3bf/jiter-0.13.0-cp313-cp313-win32.whl", hash = "sha256:36ebfbcffafb146d0e6ffb3e74d51e03d9c35ce7c625c8066cdbfc7b953bdc72", size = 200630, upload-time = "2026-02-02T12:36:31.808Z" },
+ { url = "https://files.pythonhosted.org/packages/25/65/3bd1a972c9a08ecd22eb3b08a95d1941ebe6938aea620c246cf426ae09c2/jiter-0.13.0-cp313-cp313-win_amd64.whl", hash = "sha256:8d76029f077379374cf0dbc78dbe45b38dec4a2eb78b08b5194ce836b2517afc", size = 202602, upload-time = "2026-02-02T12:36:33.679Z" },
+ { url = "https://files.pythonhosted.org/packages/15/fe/13bd3678a311aa67686bb303654792c48206a112068f8b0b21426eb6851e/jiter-0.13.0-cp313-cp313-win_arm64.whl", hash = "sha256:bb7613e1a427cfcb6ea4544f9ac566b93d5bf67e0d48c787eca673ff9c9dff2b", size = 185939, upload-time = "2026-02-02T12:36:35.065Z" },
+ { url = "https://files.pythonhosted.org/packages/49/19/a929ec002ad3228bc97ca01dbb14f7632fffdc84a95ec92ceaf4145688ae/jiter-0.13.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:fa476ab5dd49f3bf3a168e05f89358c75a17608dbabb080ef65f96b27c19ab10", size = 316616, upload-time = "2026-02-02T12:36:36.579Z" },
+ { url = "https://files.pythonhosted.org/packages/52/56/d19a9a194afa37c1728831e5fb81b7722c3de18a3109e8f282bfc23e587a/jiter-0.13.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ade8cb6ff5632a62b7dbd4757d8c5573f7a2e9ae285d6b5b841707d8363205ef", size = 346850, upload-time = "2026-02-02T12:36:38.058Z" },
+ { url = "https://files.pythonhosted.org/packages/36/4a/94e831c6bf287754a8a019cb966ed39ff8be6ab78cadecf08df3bb02d505/jiter-0.13.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9950290340acc1adaded363edd94baebcee7dabdfa8bee4790794cd5cfad2af6", size = 358551, upload-time = "2026-02-02T12:36:39.417Z" },
+ { url = "https://files.pythonhosted.org/packages/a2/ec/a4c72c822695fa80e55d2b4142b73f0012035d9fcf90eccc56bc060db37c/jiter-0.13.0-cp313-cp313t-win_amd64.whl", hash = "sha256:2b4972c6df33731aac0742b64fd0d18e0a69bc7d6e03108ce7d40c85fd9e3e6d", size = 201950, upload-time = "2026-02-02T12:36:40.791Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/00/393553ec27b824fbc29047e9c7cd4a3951d7fbe4a76743f17e44034fa4e4/jiter-0.13.0-cp313-cp313t-win_arm64.whl", hash = "sha256:701a1e77d1e593c1b435315ff625fd071f0998c5f02792038a5ca98899261b7d", size = 185852, upload-time = "2026-02-02T12:36:42.077Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/f5/f1997e987211f6f9bd71b8083047b316208b4aca0b529bb5f8c96c89ef3e/jiter-0.13.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:cc5223ab19fe25e2f0bf2643204ad7318896fe3729bf12fde41b77bfc4fafff0", size = 308804, upload-time = "2026-02-02T12:36:43.496Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/8f/5482a7677731fd44881f0204981ce2d7175db271f82cba2085dd2212e095/jiter-0.13.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:9776ebe51713acf438fd9b4405fcd86893ae5d03487546dae7f34993217f8a91", size = 318787, upload-time = "2026-02-02T12:36:45.071Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/b9/7257ac59778f1cd025b26a23c5520a36a424f7f1b068f2442a5b499b7464/jiter-0.13.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:879e768938e7b49b5e90b7e3fecc0dbec01b8cb89595861fb39a8967c5220d09", size = 353880, upload-time = "2026-02-02T12:36:47.365Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/87/719eec4a3f0841dad99e3d3604ee4cba36af4419a76f3cb0b8e2e691ad67/jiter-0.13.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:682161a67adea11e3aae9038c06c8b4a9a71023228767477d683f69903ebc607", size = 366702, upload-time = "2026-02-02T12:36:48.871Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/65/415f0a75cf6921e43365a1bc227c565cb949caca8b7532776e430cbaa530/jiter-0.13.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a13b68cd1cd8cc9de8f244ebae18ccb3e4067ad205220ef324c39181e23bbf66", size = 486319, upload-time = "2026-02-02T12:36:53.006Z" },
+ { url = "https://files.pythonhosted.org/packages/54/a2/9e12b48e82c6bbc6081fd81abf915e1443add1b13d8fc586e1d90bb02bb8/jiter-0.13.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:87ce0f14c6c08892b610686ae8be350bf368467b6acd5085a5b65441e2bf36d2", size = 372289, upload-time = "2026-02-02T12:36:54.593Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/c1/e4693f107a1789a239c759a432e9afc592366f04e901470c2af89cfd28e1/jiter-0.13.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0c365005b05505a90d1c47856420980d0237adf82f70c4aff7aebd3c1cc143ad", size = 360165, upload-time = "2026-02-02T12:36:56.112Z" },
+ { url = "https://files.pythonhosted.org/packages/17/08/91b9ea976c1c758240614bd88442681a87672eebc3d9a6dde476874e706b/jiter-0.13.0-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1317fdffd16f5873e46ce27d0e0f7f4f90f0cdf1d86bf6abeaea9f63ca2c401d", size = 389634, upload-time = "2026-02-02T12:36:57.495Z" },
+ { url = "https://files.pythonhosted.org/packages/18/23/58325ef99390d6d40427ed6005bf1ad54f2577866594bcf13ce55675f87d/jiter-0.13.0-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:c05b450d37ba0c9e21c77fef1f205f56bcee2330bddca68d344baebfc55ae0df", size = 514933, upload-time = "2026-02-02T12:36:58.909Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/25/69f1120c7c395fd276c3996bb8adefa9c6b84c12bb7111e5c6ccdcd8526d/jiter-0.13.0-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:775e10de3849d0631a97c603f996f518159272db00fdda0a780f81752255ee9d", size = 548842, upload-time = "2026-02-02T12:37:00.433Z" },
+ { url = "https://files.pythonhosted.org/packages/18/05/981c9669d86850c5fbb0d9e62bba144787f9fba84546ba43d624ee27ef29/jiter-0.13.0-cp314-cp314-win32.whl", hash = "sha256:632bf7c1d28421c00dd8bbb8a3bac5663e1f57d5cd5ed962bce3c73bf62608e6", size = 202108, upload-time = "2026-02-02T12:37:01.718Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/96/cdcf54dd0b0341db7d25413229888a346c7130bd20820530905fdb65727b/jiter-0.13.0-cp314-cp314-win_amd64.whl", hash = "sha256:f22ef501c3f87ede88f23f9b11e608581c14f04db59b6a801f354397ae13739f", size = 204027, upload-time = "2026-02-02T12:37:03.075Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/f9/724bcaaab7a3cd727031fe4f6995cb86c4bd344909177c186699c8dec51a/jiter-0.13.0-cp314-cp314-win_arm64.whl", hash = "sha256:07b75fe09a4ee8e0c606200622e571e44943f47254f95e2436c8bdcaceb36d7d", size = 187199, upload-time = "2026-02-02T12:37:04.414Z" },
+ { url = "https://files.pythonhosted.org/packages/62/92/1661d8b9fd6a3d7a2d89831db26fe3c1509a287d83ad7838831c7b7a5c7e/jiter-0.13.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:964538479359059a35fb400e769295d4b315ae61e4105396d355a12f7fef09f0", size = 318423, upload-time = "2026-02-02T12:37:05.806Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/3b/f77d342a54d4ebcd128e520fc58ec2f5b30a423b0fd26acdfc0c6fef8e26/jiter-0.13.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e104da1db1c0991b3eaed391ccd650ae8d947eab1480c733e5a3fb28d4313e40", size = 351438, upload-time = "2026-02-02T12:37:07.189Z" },
+ { url = "https://files.pythonhosted.org/packages/76/b3/ba9a69f0e4209bd3331470c723c2f5509e6f0482e416b612431a5061ed71/jiter-0.13.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:0e3a5f0cde8ff433b8e88e41aa40131455420fb3649a3c7abdda6145f8cb7202", size = 364774, upload-time = "2026-02-02T12:37:08.579Z" },
+ { url = "https://files.pythonhosted.org/packages/b3/16/6cdb31fa342932602458dbb631bfbd47f601e03d2e4950740e0b2100b570/jiter-0.13.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:57aab48f40be1db920a582b30b116fe2435d184f77f0e4226f546794cedd9cf0", size = 487238, upload-time = "2026-02-02T12:37:10.066Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/b1/956cc7abaca8d95c13aa8d6c9b3f3797241c246cd6e792934cc4c8b250d2/jiter-0.13.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:7772115877c53f62beeb8fd853cab692dbc04374ef623b30f997959a4c0e7e95", size = 372892, upload-time = "2026-02-02T12:37:11.656Z" },
+ { url = "https://files.pythonhosted.org/packages/26/c4/97ecde8b1e74f67b8598c57c6fccf6df86ea7861ed29da84629cdbba76c4/jiter-0.13.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1211427574b17b633cfceba5040de8081e5abf114f7a7602f73d2e16f9fdaa59", size = 360309, upload-time = "2026-02-02T12:37:13.244Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/d7/eabe3cf46715854ccc80be2cd78dd4c36aedeb30751dbf85a1d08c14373c/jiter-0.13.0-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:7beae3a3d3b5212d3a55d2961db3c292e02e302feb43fce6a3f7a31b90ea6dfe", size = 389607, upload-time = "2026-02-02T12:37:14.881Z" },
+ { url = "https://files.pythonhosted.org/packages/df/2d/03963fc0804e6109b82decfb9974eb92df3797fe7222428cae12f8ccaa0c/jiter-0.13.0-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:e5562a0f0e90a6223b704163ea28e831bd3a9faa3512a711f031611e6b06c939", size = 514986, upload-time = "2026-02-02T12:37:16.326Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/6c/8c83b45eb3eb1c1e18d841fe30b4b5bc5619d781267ca9bc03e005d8fd0a/jiter-0.13.0-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:6c26a424569a59140fb51160a56df13f438a2b0967365e987889186d5fc2f6f9", size = 548756, upload-time = "2026-02-02T12:37:17.736Z" },
+ { url = "https://files.pythonhosted.org/packages/47/66/eea81dfff765ed66c68fd2ed8c96245109e13c896c2a5015c7839c92367e/jiter-0.13.0-cp314-cp314t-win32.whl", hash = "sha256:24dc96eca9f84da4131cdf87a95e6ce36765c3b156fc9ae33280873b1c32d5f6", size = 201196, upload-time = "2026-02-02T12:37:19.101Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/32/4ac9c7a76402f8f00d00842a7f6b83b284d0cf7c1e9d4227bc95aa6d17fa/jiter-0.13.0-cp314-cp314t-win_amd64.whl", hash = "sha256:0a8d76c7524087272c8ae913f5d9d608bd839154b62c4322ef65723d2e5bb0b8", size = 204215, upload-time = "2026-02-02T12:37:20.495Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/8e/7def204fea9f9be8b3c21a6f2dd6c020cf56c7d5ff753e0e23ed7f9ea57e/jiter-0.13.0-cp314-cp314t-win_arm64.whl", hash = "sha256:2c26cf47e2cad140fa23b6d58d435a7c0161f5c514284802f25e87fddfe11024", size = 187152, upload-time = "2026-02-02T12:37:22.124Z" },
+ { url = "https://files.pythonhosted.org/packages/80/60/e50fa45dd7e2eae049f0ce964663849e897300433921198aef94b6ffa23a/jiter-0.13.0-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:3d744a6061afba08dd7ae375dcde870cffb14429b7477e10f67e9e6d68772a0a", size = 305169, upload-time = "2026-02-02T12:37:50.376Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/73/a009f41c5eed71c49bec53036c4b33555afcdee70682a18c6f66e396c039/jiter-0.13.0-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:ff732bd0a0e778f43d5009840f20b935e79087b4dc65bd36f1cd0f9b04b8ff7f", size = 303808, upload-time = "2026-02-02T12:37:52.092Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/10/528b439290763bff3d939268085d03382471b442f212dca4ff5f12802d43/jiter-0.13.0-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ab44b178f7981fcaea7e0a5df20e773c663d06ffda0198f1a524e91b2fde7e59", size = 337384, upload-time = "2026-02-02T12:37:53.582Z" },
+ { url = "https://files.pythonhosted.org/packages/67/8a/a342b2f0251f3dac4ca17618265d93bf244a2a4d089126e81e4c1056ac50/jiter-0.13.0-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7bb00b6d26db67a05fe3e12c76edc75f32077fb51deed13822dc648fa373bc19", size = 343768, upload-time = "2026-02-02T12:37:55.055Z" },
+]
+
[[package]]
name = "jmespath"
-version = "1.0.1"
+version = "1.1.0"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/00/2a/e867e8531cf3e36b41201936b7fa7ba7b5702dbef42922193f05c8976cd6/jmespath-1.0.1.tar.gz", hash = "sha256:90261b206d6defd58fdd5e85f478bf633a2901798906be2ad389150c5c60edbe", size = 25843, upload-time = "2022-06-17T18:00:12.224Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/d3/59/322338183ecda247fb5d1763a6cbe46eff7222eaeebafd9fa65d4bf5cb11/jmespath-1.1.0.tar.gz", hash = "sha256:472c87d80f36026ae83c6ddd0f1d05d4e510134ed462851fd5f754c8c3cbb88d", size = 27377, upload-time = "2026-01-22T16:35:26.279Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/31/b4/b9b800c45527aadd64d5b442f9b932b00648617eb5d63d2c7a6587b7cafc/jmespath-1.0.1-py3-none-any.whl", hash = "sha256:02e2e4cc71b5bcab88332eebf907519190dd9e6e82107fa7f83b1003a6252980", size = 20256, upload-time = "2022-06-17T18:00:10.251Z" },
+ { url = "https://files.pythonhosted.org/packages/14/2f/967ba146e6d58cf6a652da73885f52fc68001525b4197effc174321d70b4/jmespath-1.1.0-py3-none-any.whl", hash = "sha256:a5663118de4908c91729bea0acadca56526eb2698e83de10cd116ae0f4e97c64", size = 20419, upload-time = "2026-01-22T16:35:24.919Z" },
]
[[package]]
@@ -1909,6 +2106,36 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl", hash = "sha256:5fc3c5039fc5ca8c0276333a188bbd59d6b7ab37fe6632daa76bc7f9ec18e713", size = 309071, upload-time = "2025-12-15T08:41:44.973Z" },
]
+[[package]]
+name = "jsonpatch"
+version = "1.33"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jsonpointer" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/42/78/18813351fe5d63acad16aec57f94ec2b70a09e53ca98145589e185423873/jsonpatch-1.33.tar.gz", hash = "sha256:9fcd4009c41e6d12348b4a0ff2563ba56a2923a7dfee731d004e212e1ee5030c", size = 21699, upload-time = "2023-06-26T12:07:29.144Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/73/07/02e16ed01e04a374e644b575638ec7987ae846d25ad97bcc9945a3ee4b0e/jsonpatch-1.33-py2.py3-none-any.whl", hash = "sha256:0ae28c0cd062bbd8b8ecc26d7d164fbbea9652a1a3693f3b956c1eae5145dade", size = 12898, upload-time = "2023-06-16T21:01:28.466Z" },
+]
+
+[[package]]
+name = "jsonpickle"
+version = "1.5.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/af/62/31ef0b58050a3731b079af69932104a9443bff07fe2b9564c161e3ec4348/jsonpickle-1.5.1.tar.gz", hash = "sha256:060f97096559d1b86aa16cac2f4ea5f7b6da0c15d8a4de150b78013a886f9a51", size = 109560, upload-time = "2021-01-31T05:57:15.037Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/77/a7/c2f527ddce3155ae9e008385963c2325cbfd52969f8b38efa2723e2af4af/jsonpickle-1.5.1-py2.py3-none-any.whl", hash = "sha256:8eb8323f0e12cb40687f0445e2115d8165901e20ac670add55bb53a95c68c0e5", size = 37124, upload-time = "2021-01-31T05:57:12.256Z" },
+]
+
+[[package]]
+name = "jsonpointer"
+version = "3.1.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/18/c7/af399a2e7a67fd18d63c40c5e62d3af4e67b836a2107468b6a5ea24c4304/jsonpointer-3.1.1.tar.gz", hash = "sha256:0b801c7db33a904024f6004d526dcc53bbb8a4a0f4e32bfd10beadf60adf1900", size = 9068, upload-time = "2026-03-23T22:32:32.458Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9e/6a/a83720e953b1682d2d109d3c2dbb0bc9bf28cc1cbc205be4ef4be5da709d/jsonpointer-3.1.1-py3-none-any.whl", hash = "sha256:8ff8b95779d071ba472cf5bc913028df06031797532f08a7d5b602d8b2a488ca", size = 7659, upload-time = "2026-03-23T22:32:31.568Z" },
+]
+
[[package]]
name = "jsonschema"
version = "4.26.0"
@@ -1936,6 +2163,89 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/41/45/1a4ed80516f02155c51f51e8cedb3c1902296743db0bbc66608a0db2814f/jsonschema_specifications-2025.9.1-py3-none-any.whl", hash = "sha256:98802fee3a11ee76ecaca44429fda8a41bff98b00a0f2838151b113f210cc6fe", size = 18437, upload-time = "2025-09-08T01:34:57.871Z" },
]
+[[package]]
+name = "langchain"
+version = "0.3.28"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "langchain-core" },
+ { name = "langchain-text-splitters" },
+ { name = "langsmith" },
+ { name = "pydantic" },
+ { name = "pyyaml" },
+ { name = "requests" },
+ { name = "sqlalchemy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/87/bb/a65e29c8e4aaf0348c2617962e427c8e760d82a67adbd197019e49c7769d/langchain-0.3.28.tar.gz", hash = "sha256:30a32f44cc6690bcc6a6fb7c14d61a15406d5eda1a0e7eab60b3660944888741", size = 10242473, upload-time = "2026-03-06T22:45:17.911Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5b/f5/ecd71e5b78e67944b2600a155ef63000bc00148e6794e8e7809b2453887a/langchain-0.3.28-py3-none-any.whl", hash = "sha256:1ba1244477b67b812b775f346209fa596e78bf055a34e45ce22acb7a45842a32", size = 1024717, upload-time = "2026-03-06T22:45:15.545Z" },
+]
+
+[[package]]
+name = "langchain-core"
+version = "0.3.83"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jsonpatch" },
+ { name = "langsmith" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" } },
+ { name = "pydantic" },
+ { name = "pyyaml" },
+ { name = "tenacity" },
+ { name = "typing-extensions" },
+ { name = "uuid-utils" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/21/a4/24f2d787bfcf56e5990924cacefe6f6e7971a3629f97c8162fc7a2a3d851/langchain_core-0.3.83.tar.gz", hash = "sha256:a0a4c7b6ea1c446d3b432116f405dc2afa1fe7891c44140d3d5acca221909415", size = 597965, upload-time = "2026-01-13T01:19:23.854Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5a/db/d71b80d3bd6193812485acea4001cdf86cf95a44bbf942f7a240120ff762/langchain_core-0.3.83-py3-none-any.whl", hash = "sha256:8c92506f8b53fc1958b1c07447f58c5783eb8833dd3cb6dc75607c80891ab1ae", size = 458890, upload-time = "2026-01-13T01:19:21.748Z" },
+]
+
+[[package]]
+name = "langchain-nvidia-ai-endpoints"
+version = "0.3.19"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiohttp" },
+ { name = "filetype" },
+ { name = "langchain-core" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/4d/6f/0fcad6732f124c0d96fd7e8487a0b51db9f167964cfe041e4669a2148dbe/langchain_nvidia_ai_endpoints-0.3.19.tar.gz", hash = "sha256:1c420c88f7c78b2b2c2fdcef8e46104c2dd19c81e036bdd03a4838a6340950fe", size = 42884, upload-time = "2025-10-31T00:17:19.352Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/de/46/d7529004de384b2abc9e5b76cf4a84a23f3028ec6381bd5f7c00ac39bfab/langchain_nvidia_ai_endpoints-0.3.19-py3-none-any.whl", hash = "sha256:40161a71646fcbe457ac5f2222c5eadcbe31a7d79d618f5a0857c37fffa3a6d5", size = 46229, upload-time = "2025-10-31T00:17:18.306Z" },
+]
+
+[[package]]
+name = "langchain-text-splitters"
+version = "0.3.11"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "langchain-core" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/11/43/dcda8fd25f0b19cb2835f2f6bb67f26ad58634f04ac2d8eae00526b0fa55/langchain_text_splitters-0.3.11.tar.gz", hash = "sha256:7a50a04ada9a133bbabb80731df7f6ddac51bc9f1b9cab7fa09304d71d38a6cc", size = 46458, upload-time = "2025-08-31T23:02:58.316Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/58/0d/41a51b40d24ff0384ec4f7ab8dd3dcea8353c05c973836b5e289f1465d4f/langchain_text_splitters-0.3.11-py3-none-any.whl", hash = "sha256:cf079131166a487f1372c8ab5d0bfaa6c0a4291733d9c43a34a16ac9bcd6a393", size = 33845, upload-time = "2025-08-31T23:02:57.195Z" },
+]
+
+[[package]]
+name = "langsmith"
+version = "0.7.22"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "httpx" },
+ { name = "orjson", marker = "platform_python_implementation != 'PyPy'" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" } },
+ { name = "pydantic" },
+ { name = "requests" },
+ { name = "requests-toolbelt" },
+ { name = "uuid-utils" },
+ { name = "xxhash" },
+ { name = "zstandard" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/be/2a/2d5e6c67396fd228670af278c4da7bd6db2b8d11deaf6f108490b6d3f561/langsmith-0.7.22.tar.gz", hash = "sha256:35bfe795d648b069958280760564632fd28ebc9921c04f3e209c0db6a6c7dc04", size = 1134923, upload-time = "2026-03-19T22:45:23.492Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1a/94/1f5d72655ab6534129540843776c40eff757387b88e798d8b3bf7e313fd4/langsmith-0.7.22-py3-none-any.whl", hash = "sha256:6e9d5148314d74e86748cb9d3898632cad0320c9323d95f70f969e5bc078eee4", size = 359927, upload-time = "2026-03-19T22:45:21.603Z" },
+]
+
[[package]]
name = "lark"
version = "1.3.1"
@@ -1956,7 +2266,7 @@ wheels = [
[[package]]
name = "leptonai"
-version = "0.26.8"
+version = "0.27.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "anyio" },
@@ -1974,14 +2284,14 @@ dependencies = [
{ name = "pydantic" },
{ name = "python-multipart" },
{ name = "pyyaml" },
- { name = "ray" },
+ { name = "ray", extra = ["default"] },
{ name = "requests" },
{ name = "rich" },
{ name = "typing-extensions" },
{ name = "uvicorn" },
]
wheels = [
- { url = "https://files.pythonhosted.org/packages/be/fd/949841aaf69cfb8086be61ddb06864426064400f831b1ca1ae1ade32b357/leptonai-0.26.8-py3-none-any.whl", hash = "sha256:ddba3afd6b82899f66cd229c4348972320f1e96f20393ebfc5153338b56aad30", size = 2467623, upload-time = "2026-01-08T00:13:55.515Z" },
+ { url = "https://files.pythonhosted.org/packages/88/80/281af82242d9e20e9c0b19fb35c2a7a6df728b14f25483271d8169ef0a9a/leptonai-0.27.0-py3-none-any.whl", hash = "sha256:2a83d77a3bfcd86b877483ab503b4cde970b0a4d4143535510dac67d565fc1a4", size = 2476455, upload-time = "2026-01-17T03:31:56.977Z" },
]
[[package]]
@@ -1994,22 +2304,6 @@ dependencies = [
]
sdist = { url = "https://files.pythonhosted.org/packages/de/cd/337df968b38d94c5aabd3e1b10630f047a2b345f6e1d4456bd9fe7417537/libcst-1.8.6.tar.gz", hash = "sha256:f729c37c9317126da9475bdd06a7208eb52fcbd180a6341648b45a56b4ba708b", size = 891354, upload-time = "2025-11-03T22:33:30.621Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/c4/52/97d5454dee9d014821fe0c88f3dc0e83131b97dd074a4d49537056a75475/libcst-1.8.6-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:a20c5182af04332cc94d8520792befda06d73daf2865e6dddc5161c72ea92cb9", size = 2211698, upload-time = "2025-11-03T22:31:50.117Z" },
- { url = "https://files.pythonhosted.org/packages/6c/a4/d1205985d378164687af3247a9c8f8bdb96278b0686ac98ab951bc6d336a/libcst-1.8.6-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:36473e47cb199b7e6531d653ee6ffed057de1d179301e6c67f651f3af0b499d6", size = 2093104, upload-time = "2025-11-03T22:31:52.189Z" },
- { url = "https://files.pythonhosted.org/packages/9e/de/1338da681b7625b51e584922576d54f1b8db8fc7ff4dc79121afc5d4d2cd/libcst-1.8.6-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:06fc56335a45d61b7c1b856bfab4587b84cfe31e9d6368f60bb3c9129d900f58", size = 2237419, upload-time = "2025-11-03T22:31:53.526Z" },
- { url = "https://files.pythonhosted.org/packages/50/06/ee66f2d83b870534756e593d464d8b33b0914c224dff3a407e0f74dc04e0/libcst-1.8.6-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:6b23d14a7fc0addd9795795763af26b185deb7c456b1e7cc4d5228e69dab5ce8", size = 2300820, upload-time = "2025-11-03T22:31:55.995Z" },
- { url = "https://files.pythonhosted.org/packages/9c/ca/959088729de8e0eac8dd516e4fb8623d8d92bad539060fa85c9e94d418a5/libcst-1.8.6-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:16cfe0cfca5fd840e1fb2c30afb628b023d3085b30c3484a79b61eae9d6fe7ba", size = 2301201, upload-time = "2025-11-03T22:31:57.347Z" },
- { url = "https://files.pythonhosted.org/packages/c2/4c/2a21a8c452436097dfe1da277f738c3517f3f728713f16d84b9a3d67ca8d/libcst-1.8.6-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:455f49a93aea4070132c30ebb6c07c2dea0ba6c1fde5ffde59fc45dbb9cfbe4b", size = 2408213, upload-time = "2025-11-03T22:31:59.221Z" },
- { url = "https://files.pythonhosted.org/packages/3e/26/8f7b671fad38a515bb20b038718fd2221ab658299119ac9bcec56c2ced27/libcst-1.8.6-cp310-cp310-win_amd64.whl", hash = "sha256:72cca15800ffc00ba25788e4626189fe0bc5fe2a0c1cb4294bce2e4df21cc073", size = 2119189, upload-time = "2025-11-03T22:32:00.696Z" },
- { url = "https://files.pythonhosted.org/packages/5b/bf/ffb23a48e27001165cc5c81c5d9b3d6583b21b7f5449109e03a0020b060c/libcst-1.8.6-cp310-cp310-win_arm64.whl", hash = "sha256:6cad63e3a26556b020b634d25a8703b605c0e0b491426b3e6b9e12ed20f09100", size = 2001736, upload-time = "2025-11-03T22:32:02.986Z" },
- { url = "https://files.pythonhosted.org/packages/dc/15/95c2ecadc0fb4af8a7057ac2012a4c0ad5921b9ef1ace6c20006b56d3b5f/libcst-1.8.6-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:3649a813660fbffd7bc24d3f810b1f75ac98bd40d9d6f56d1f0ee38579021073", size = 2211289, upload-time = "2025-11-03T22:32:04.673Z" },
- { url = "https://files.pythonhosted.org/packages/80/c3/7e1107acd5ed15cf60cc07c7bb64498a33042dc4821874aea3ec4942f3cd/libcst-1.8.6-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:0cbe17067055829607c5ba4afa46bfa4d0dd554c0b5a583546e690b7367a29b6", size = 2092927, upload-time = "2025-11-03T22:32:06.209Z" },
- { url = "https://files.pythonhosted.org/packages/c1/ff/0d2be87f67e2841a4a37d35505e74b65991d30693295c46fc0380ace0454/libcst-1.8.6-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:59a7e388c57d21d63722018978a8ddba7b176e3a99bd34b9b84a576ed53f2978", size = 2237002, upload-time = "2025-11-03T22:32:07.559Z" },
- { url = "https://files.pythonhosted.org/packages/69/99/8c4a1b35c7894ccd7d33eae01ac8967122f43da41325223181ca7e4738fe/libcst-1.8.6-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:b6c1248cc62952a3a005792b10cdef2a4e130847be9c74f33a7d617486f7e532", size = 2301048, upload-time = "2025-11-03T22:32:08.869Z" },
- { url = "https://files.pythonhosted.org/packages/9b/8b/d1aa811eacf936cccfb386ae0585aa530ea1221ccf528d67144e041f5915/libcst-1.8.6-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:6421a930b028c5ef4a943b32a5a78b7f1bf15138214525a2088f11acbb7d3d64", size = 2300675, upload-time = "2025-11-03T22:32:10.579Z" },
- { url = "https://files.pythonhosted.org/packages/c6/6b/7b65cd41f25a10c1fef2389ddc5c2b2cc23dc4d648083fa3e1aa7e0eeac2/libcst-1.8.6-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:6d8b67874f2188399a71a71731e1ba2d1a2c3173b7565d1cc7ffb32e8fbaba5b", size = 2407934, upload-time = "2025-11-03T22:32:11.856Z" },
- { url = "https://files.pythonhosted.org/packages/c5/8b/401cfff374bb3b785adfad78f05225225767ee190997176b2a9da9ed9460/libcst-1.8.6-cp311-cp311-win_amd64.whl", hash = "sha256:b0d8c364c44ae343937f474b2e492c1040df96d94530377c2f9263fb77096e4f", size = 2119247, upload-time = "2025-11-03T22:32:13.279Z" },
- { url = "https://files.pythonhosted.org/packages/f1/17/085f59eaa044b6ff6bc42148a5449df2b7f0ba567307de7782fe85c39ee2/libcst-1.8.6-cp311-cp311-win_arm64.whl", hash = "sha256:5dcaaebc835dfe5755bc85f9b186fb7e2895dda78e805e577fef1011d51d5a5c", size = 2001774, upload-time = "2025-11-03T22:32:14.647Z" },
{ url = "https://files.pythonhosted.org/packages/0c/3c/93365c17da3d42b055a8edb0e1e99f1c60c776471db6c9b7f1ddf6a44b28/libcst-1.8.6-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:0c13d5bd3d8414a129e9dccaf0e5785108a4441e9b266e1e5e9d1f82d1b943c9", size = 2206166, upload-time = "2025-11-03T22:32:16.012Z" },
{ url = "https://files.pythonhosted.org/packages/1d/cb/7530940e6ac50c6dd6022349721074e19309eb6aa296e942ede2213c1a19/libcst-1.8.6-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f1472eeafd67cdb22544e59cf3bfc25d23dc94058a68cf41f6654ff4fcb92e09", size = 2083726, upload-time = "2025-11-03T22:32:17.312Z" },
{ url = "https://files.pythonhosted.org/packages/1b/cf/7e5eaa8c8f2c54913160671575351d129170db757bb5e4b7faffed022271/libcst-1.8.6-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:089c58e75cb142ec33738a1a4ea7760a28b40c078ab2fd26b270dac7d2633a4d", size = 2235755, upload-time = "2025-11-03T22:32:18.859Z" },
@@ -2052,6 +2346,25 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/98/6d/5d6a790a02eb0d9d36c4aed4f41b277497e6178900b2fa29c35353aa45ed/libcst-1.8.6-cp314-cp314t-win_arm64.whl", hash = "sha256:819c8081e2948635cab60c603e1bbdceccdfe19104a242530ad38a36222cb88f", size = 2065000, upload-time = "2025-11-03T22:33:16.257Z" },
]
+[[package]]
+name = "logsage"
+version = "0.1.5"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "drain3" },
+ { name = "langchain" },
+ { name = "langchain-core" },
+ { name = "langchain-nvidia-ai-endpoints" },
+ { name = "nh3" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" } },
+ { name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" } },
+ { name = "pydantic-settings" },
+ { name = "requests" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e2/eb/d44608c5fc69ac216a41cd164453df5784114d07f28ece6c6ab8a351f636/logsage-0.1.5-py3-none-any.whl", hash = "sha256:4c144ea2b339f370e943929dc2d1f755b04351a75a2d8fd4201bd6d90045ed7a", size = 65098, upload-time = "2026-01-25T10:12:30.042Z" },
+]
+
[[package]]
name = "loguru"
version = "0.7.3"
@@ -2067,38 +2380,39 @@ wheels = [
[[package]]
name = "mamba-ssm"
-version = "2.3.0"
+version = "2.3.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "einops" },
{ name = "ninja" },
- { name = "packaging" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
{ name = "setuptools" },
{ name = "torch", marker = "sys_platform == 'never'" },
{ name = "transformers" },
{ name = "triton", marker = "sys_platform == 'never'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/54/69/a87f06d9dba78c041adb81f2228e978aab179477c64f1a210c0fe0d63e8d/mamba_ssm-2.3.0.tar.gz", hash = "sha256:8294e12125f76021e4e190f4137e84a84935920eeda5d0037a6917524456b303", size = 121116, upload-time = "2026-01-12T17:07:22.152Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/34/67/ec89aa703da194a813e35d2ea2de8f74a7ce6991a120a29f3a0c5e30d4b9/mamba_ssm-2.3.1.tar.gz", hash = "sha256:4d529477ad94753962216d583fc8f1c127c717b7d7c875d6bbb9376366d0d761", size = 121707, upload-time = "2026-03-10T09:27:34.798Z" }
[[package]]
name = "markdown"
-version = "3.10"
+version = "3.10.2"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/7d/ab/7dd27d9d863b3376fcf23a5a13cb5d024aed1db46f963f1b5735ae43b3be/markdown-3.10.tar.gz", hash = "sha256:37062d4f2aa4b2b6b32aefb80faa300f82cc790cb949a35b8caede34f2b68c0e", size = 364931, upload-time = "2025-11-03T19:51:15.007Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/2b/f4/69fa6ed85ae003c2378ffa8f6d2e3234662abd02c10d216c0ba96081a238/markdown-3.10.2.tar.gz", hash = "sha256:994d51325d25ad8aa7ce4ebaec003febcce822c3f8c911e3b17c52f7f589f950", size = 368805, upload-time = "2026-02-09T14:57:26.942Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/70/81/54e3ce63502cd085a0c556652a4e1b919c45a446bd1e5300e10c44c8c521/markdown-3.10-py3-none-any.whl", hash = "sha256:b5b99d6951e2e4948d939255596523444c0e677c669700b1d17aa4a8a464cb7c", size = 107678, upload-time = "2025-11-03T19:51:13.887Z" },
+ { url = "https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl", hash = "sha256:e91464b71ae3ee7afd3017d9f358ef0baf158fd9a298db92f1d4761133824c36", size = 108180, upload-time = "2026-02-09T14:57:25.787Z" },
]
[[package]]
name = "markdown-it-py"
-version = "3.0.0"
+version = "4.0.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "mdurl" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/38/71/3b932df36c1a044d397a1f92d1cf91ee0a503d91e470cbd670aa66b07ed0/markdown-it-py-3.0.0.tar.gz", hash = "sha256:e3f60a94fa066dc52ec76661e37c851cb232d92f9886b15cb560aaada2df8feb", size = 74596, upload-time = "2023-06-03T06:41:14.443Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/5b/f5/4ec618ed16cc4f8fb3b701563655a69816155e79e24a17b651541804721d/markdown_it_py-4.0.0.tar.gz", hash = "sha256:cb0a2b4aa34f932c007117b194e945bd74e0ec24133ceb5bac59009cda1cb9f3", size = 73070, upload-time = "2025-08-11T12:57:52.854Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/42/d7/1ec15b46af6af88f19b8e5ffea08fa375d433c998b8a7639e76935c14f1f/markdown_it_py-3.0.0-py3-none-any.whl", hash = "sha256:355216845c60bd96232cd8d8c40e8f9765cc86f46880e43a8fd22dc1a1a8cab1", size = 87528, upload-time = "2023-06-03T06:41:11.019Z" },
+ { url = "https://files.pythonhosted.org/packages/94/54/e7d793b573f298e1c9013b8c4dade17d481164aa517d1d7148619c2cedbf/markdown_it_py-4.0.0-py3-none-any.whl", hash = "sha256:87327c59b172c5011896038353a81343b6754500a08cd7a4973bb48c6d578147", size = 87321, upload-time = "2025-08-11T12:57:51.923Z" },
]
[[package]]
@@ -2107,28 +2421,6 @@ version = "3.0.3"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/7e/99/7690b6d4034fffd95959cbe0c02de8deb3098cc577c67bb6a24fe5d7caa7/markupsafe-3.0.3.tar.gz", hash = "sha256:722695808f4b6457b320fdc131280796bdceb04ab50fe1795cd540799ebe1698", size = 80313, upload-time = "2025-09-27T18:37:40.426Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/e8/4b/3541d44f3937ba468b75da9eebcae497dcf67adb65caa16760b0a6807ebb/markupsafe-3.0.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:2f981d352f04553a7171b8e44369f2af4055f888dfb147d55e42d29e29e74559", size = 11631, upload-time = "2025-09-27T18:36:05.558Z" },
- { url = "https://files.pythonhosted.org/packages/98/1b/fbd8eed11021cabd9226c37342fa6ca4e8a98d8188a8d9b66740494960e4/markupsafe-3.0.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:e1c1493fb6e50ab01d20a22826e57520f1284df32f2d8601fdd90b6304601419", size = 12057, upload-time = "2025-09-27T18:36:07.165Z" },
- { url = "https://files.pythonhosted.org/packages/40/01/e560d658dc0bb8ab762670ece35281dec7b6c1b33f5fbc09ebb57a185519/markupsafe-3.0.3-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1ba88449deb3de88bd40044603fafffb7bc2b055d626a330323a9ed736661695", size = 22050, upload-time = "2025-09-27T18:36:08.005Z" },
- { url = "https://files.pythonhosted.org/packages/af/cd/ce6e848bbf2c32314c9b237839119c5a564a59725b53157c856e90937b7a/markupsafe-3.0.3-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f42d0984e947b8adf7dd6dde396e720934d12c506ce84eea8476409563607591", size = 20681, upload-time = "2025-09-27T18:36:08.881Z" },
- { url = "https://files.pythonhosted.org/packages/c9/2a/b5c12c809f1c3045c4d580b035a743d12fcde53cf685dbc44660826308da/markupsafe-3.0.3-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c0c0b3ade1c0b13b936d7970b1d37a57acde9199dc2aecc4c336773e1d86049c", size = 20705, upload-time = "2025-09-27T18:36:10.131Z" },
- { url = "https://files.pythonhosted.org/packages/cf/e3/9427a68c82728d0a88c50f890d0fc072a1484de2f3ac1ad0bfc1a7214fd5/markupsafe-3.0.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:0303439a41979d9e74d18ff5e2dd8c43ed6c6001fd40e5bf2e43f7bd9bbc523f", size = 21524, upload-time = "2025-09-27T18:36:11.324Z" },
- { url = "https://files.pythonhosted.org/packages/bc/36/23578f29e9e582a4d0278e009b38081dbe363c5e7165113fad546918a232/markupsafe-3.0.3-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:d2ee202e79d8ed691ceebae8e0486bd9a2cd4794cec4824e1c99b6f5009502f6", size = 20282, upload-time = "2025-09-27T18:36:12.573Z" },
- { url = "https://files.pythonhosted.org/packages/56/21/dca11354e756ebd03e036bd8ad58d6d7168c80ce1fe5e75218e4945cbab7/markupsafe-3.0.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:177b5253b2834fe3678cb4a5f0059808258584c559193998be2601324fdeafb1", size = 20745, upload-time = "2025-09-27T18:36:13.504Z" },
- { url = "https://files.pythonhosted.org/packages/87/99/faba9369a7ad6e4d10b6a5fbf71fa2a188fe4a593b15f0963b73859a1bbd/markupsafe-3.0.3-cp310-cp310-win32.whl", hash = "sha256:2a15a08b17dd94c53a1da0438822d70ebcd13f8c3a95abe3a9ef9f11a94830aa", size = 14571, upload-time = "2025-09-27T18:36:14.779Z" },
- { url = "https://files.pythonhosted.org/packages/d6/25/55dc3ab959917602c96985cb1253efaa4ff42f71194bddeb61eb7278b8be/markupsafe-3.0.3-cp310-cp310-win_amd64.whl", hash = "sha256:c4ffb7ebf07cfe8931028e3e4c85f0357459a3f9f9490886198848f4fa002ec8", size = 15056, upload-time = "2025-09-27T18:36:16.125Z" },
- { url = "https://files.pythonhosted.org/packages/d0/9e/0a02226640c255d1da0b8d12e24ac2aa6734da68bff14c05dd53b94a0fc3/markupsafe-3.0.3-cp310-cp310-win_arm64.whl", hash = "sha256:e2103a929dfa2fcaf9bb4e7c091983a49c9ac3b19c9061b6d5427dd7d14d81a1", size = 13932, upload-time = "2025-09-27T18:36:17.311Z" },
- { url = "https://files.pythonhosted.org/packages/08/db/fefacb2136439fc8dd20e797950e749aa1f4997ed584c62cfb8ef7c2be0e/markupsafe-3.0.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:1cc7ea17a6824959616c525620e387f6dd30fec8cb44f649e31712db02123dad", size = 11631, upload-time = "2025-09-27T18:36:18.185Z" },
- { url = "https://files.pythonhosted.org/packages/e1/2e/5898933336b61975ce9dc04decbc0a7f2fee78c30353c5efba7f2d6ff27a/markupsafe-3.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:4bd4cd07944443f5a265608cc6aab442e4f74dff8088b0dfc8238647b8f6ae9a", size = 12058, upload-time = "2025-09-27T18:36:19.444Z" },
- { url = "https://files.pythonhosted.org/packages/1d/09/adf2df3699d87d1d8184038df46a9c80d78c0148492323f4693df54e17bb/markupsafe-3.0.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6b5420a1d9450023228968e7e6a9ce57f65d148ab56d2313fcd589eee96a7a50", size = 24287, upload-time = "2025-09-27T18:36:20.768Z" },
- { url = "https://files.pythonhosted.org/packages/30/ac/0273f6fcb5f42e314c6d8cd99effae6a5354604d461b8d392b5ec9530a54/markupsafe-3.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0bf2a864d67e76e5c9a34dc26ec616a66b9888e25e7b9460e1c76d3293bd9dbf", size = 22940, upload-time = "2025-09-27T18:36:22.249Z" },
- { url = "https://files.pythonhosted.org/packages/19/ae/31c1be199ef767124c042c6c3e904da327a2f7f0cd63a0337e1eca2967a8/markupsafe-3.0.3-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:bc51efed119bc9cfdf792cdeaa4d67e8f6fcccab66ed4bfdd6bde3e59bfcbb2f", size = 21887, upload-time = "2025-09-27T18:36:23.535Z" },
- { url = "https://files.pythonhosted.org/packages/b2/76/7edcab99d5349a4532a459e1fe64f0b0467a3365056ae550d3bcf3f79e1e/markupsafe-3.0.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:068f375c472b3e7acbe2d5318dea141359e6900156b5b2ba06a30b169086b91a", size = 23692, upload-time = "2025-09-27T18:36:24.823Z" },
- { url = "https://files.pythonhosted.org/packages/a4/28/6e74cdd26d7514849143d69f0bf2399f929c37dc2b31e6829fd2045b2765/markupsafe-3.0.3-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:7be7b61bb172e1ed687f1754f8e7484f1c8019780f6f6b0786e76bb01c2ae115", size = 21471, upload-time = "2025-09-27T18:36:25.95Z" },
- { url = "https://files.pythonhosted.org/packages/62/7e/a145f36a5c2945673e590850a6f8014318d5577ed7e5920a4b3448e0865d/markupsafe-3.0.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:f9e130248f4462aaa8e2552d547f36ddadbeaa573879158d721bbd33dfe4743a", size = 22923, upload-time = "2025-09-27T18:36:27.109Z" },
- { url = "https://files.pythonhosted.org/packages/0f/62/d9c46a7f5c9adbeeeda52f5b8d802e1094e9717705a645efc71b0913a0a8/markupsafe-3.0.3-cp311-cp311-win32.whl", hash = "sha256:0db14f5dafddbb6d9208827849fad01f1a2609380add406671a26386cdf15a19", size = 14572, upload-time = "2025-09-27T18:36:28.045Z" },
- { url = "https://files.pythonhosted.org/packages/83/8a/4414c03d3f891739326e1783338e48fb49781cc915b2e0ee052aa490d586/markupsafe-3.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:de8a88e63464af587c950061a5e6a67d3632e36df62b986892331d4620a35c01", size = 15077, upload-time = "2025-09-27T18:36:29.025Z" },
- { url = "https://files.pythonhosted.org/packages/35/73/893072b42e6862f319b5207adc9ae06070f095b358655f077f69a35601f0/markupsafe-3.0.3-cp311-cp311-win_arm64.whl", hash = "sha256:3b562dd9e9ea93f13d53989d23a7e775fdfd1066c33494ff43f5418bc8c58a5c", size = 13876, upload-time = "2025-09-27T18:36:29.954Z" },
{ url = "https://files.pythonhosted.org/packages/5a/72/147da192e38635ada20e0a2e1a51cf8823d2119ce8883f7053879c2199b5/markupsafe-3.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:d53197da72cc091b024dd97249dfc7794d6a56530370992a5e1a08983ad9230e", size = 11615, upload-time = "2025-09-27T18:36:30.854Z" },
{ url = "https://files.pythonhosted.org/packages/9a/81/7e4e08678a1f98521201c3079f77db69fb552acd56067661f8c2f534a718/markupsafe-3.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1872df69a4de6aead3491198eaf13810b565bdbeec3ae2dc8780f14458ec73ce", size = 12020, upload-time = "2025-09-27T18:36:31.971Z" },
{ url = "https://files.pythonhosted.org/packages/1e/2c/799f4742efc39633a1b54a92eec4082e4f815314869865d876824c257c1e/markupsafe-3.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a7e8ae81ae39e62a41ec302f972ba6ae23a5c5396c8e60113e9066ef893da0d", size = 24332, upload-time = "2025-09-27T18:36:32.813Z" },
@@ -2195,6 +2487,31 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/27/1a/1f68f9ba0c207934b35b86a8ca3aad8395a3d6dd7921c0686e23853ff5a9/mccabe-0.7.0-py2.py3-none-any.whl", hash = "sha256:6c2d30ab6be0e4a46919781807b4f0d834ebdd6c6e3dca0bda5a15f863427b6e", size = 7350, upload-time = "2022-01-24T01:14:49.62Z" },
]
+[[package]]
+name = "mcp"
+version = "1.26.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "anyio" },
+ { name = "httpx" },
+ { name = "httpx-sse" },
+ { name = "jsonschema" },
+ { name = "pydantic" },
+ { name = "pydantic-settings" },
+ { name = "pyjwt", extra = ["crypto"], marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "python-multipart" },
+ { name = "pywin32", marker = "sys_platform == 'win32'" },
+ { name = "sse-starlette" },
+ { name = "starlette" },
+ { name = "typing-extensions" },
+ { name = "typing-inspection" },
+ { name = "uvicorn", marker = "sys_platform != 'emscripten'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/fc/6d/62e76bbb8144d6ed86e202b5edd8a4cb631e7c8130f3f4893c3f90262b10/mcp-1.26.0.tar.gz", hash = "sha256:db6e2ef491eecc1a0d93711a76f28dec2e05999f93afd48795da1c1137142c66", size = 608005, upload-time = "2026-01-24T19:40:32.468Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fd/d9/eaa1f80170d2b7c5ba23f3b59f766f3a0bb41155fbc32a69adfa1adaaef9/mcp-1.26.0-py3-none-any.whl", hash = "sha256:904a21c33c25aa98ddbeb47273033c435e595bbacfdb177f4bd87f6dceebe1ca", size = 233615, upload-time = "2026-01-24T19:40:30.652Z" },
+]
+
[[package]]
name = "mdit-py-plugins"
version = "0.5.0"
@@ -2220,9 +2537,10 @@ wheels = [
name = "megatron-core"
source = { editable = "." }
dependencies = [
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "packaging" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
{ name = "torch", marker = "sys_platform == 'never' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
]
@@ -2236,17 +2554,21 @@ dev = [
{ name = "fastapi" },
{ name = "flash-linear-attention" },
{ name = "flashinfer-python" },
+ { name = "hypercorn" },
{ name = "mamba-ssm" },
{ name = "megatron-energon", extra = ["av-decode"], marker = "extra == 'extra-13-megatron-core-dev'" },
{ name = "multi-storage-client" },
- { name = "nv-grouped-gemm" },
{ name = "nvidia-modelopt", marker = "(sys_platform != 'darwin' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
{ name = "nvidia-resiliency-ext" },
{ name = "nvtx" },
- { name = "onnxscript" },
- { name = "opentelemetry-api" },
- { name = "tensorstore", version = "0.1.78", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "tensorstore", version = "0.1.80", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "onnxscript", version = "0.5.0", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.13' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "onnxscript", version = "0.6.2", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.13' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "openai", extra = ["aiohttp"], marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "opentelemetry-api", version = "1.33.1", source = { registry = "https://pypi.org/simple" } },
+ { name = "orjson" },
+ { name = "quart" },
+ { name = "tensorstore", version = "0.1.74", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.13' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "tensorstore", version = "0.1.82", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.13' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
{ name = "tqdm" },
{ name = "transformer-engine", marker = "extra == 'extra-13-megatron-core-dev'" },
{ name = "wget" },
@@ -2262,16 +2584,23 @@ lts = [
{ name = "mamba-ssm" },
{ name = "megatron-energon", extra = ["av-decode"], marker = "extra == 'extra-13-megatron-core-lts'" },
{ name = "multi-storage-client" },
- { name = "nv-grouped-gemm" },
{ name = "nvtx" },
- { name = "onnxscript" },
- { name = "opentelemetry-api" },
- { name = "tensorstore", version = "0.1.78", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "tensorstore", version = "0.1.80", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "onnxscript", version = "0.6.2", source = { registry = "https://pypi.org/simple" } },
+ { name = "opentelemetry-api", version = "1.33.1", source = { registry = "https://pypi.org/simple" } },
+ { name = "tensorstore", version = "0.1.82", source = { registry = "https://pypi.org/simple" } },
{ name = "tqdm" },
{ name = "wget" },
]
mlm = [
+ { name = "accelerate" },
+ { name = "flask-restful" },
+ { name = "sentencepiece" },
+ { name = "tiktoken" },
+ { name = "transformers" },
+ { name = "wandb" },
+]
+training = [
+ { name = "accelerate" },
{ name = "flask-restful" },
{ name = "sentencepiece" },
{ name = "tiktoken" },
@@ -2284,23 +2613,23 @@ build = [
{ name = "cython" },
{ name = "hatchling" },
{ name = "nvidia-mathdx" },
- { name = "packaging" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
{ name = "pybind11" },
{ name = "setuptools" },
{ name = "torch", marker = "sys_platform == 'never' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
]
ci = [
- { name = "pandas" },
+ { name = "pandas", version = "2.3.3", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "pandas", version = "3.0.1", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
{ name = "python-gitlab" },
{ name = "slack-sdk" },
]
docs = [
{ name = "myst-parser" },
{ name = "nvidia-sphinx-theme" },
- { name = "sphinx", version = "8.1.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinx", version = "8.2.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinx-autobuild", version = "2024.10.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinx-autobuild", version = "2025.8.25", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "sphinx" },
+ { name = "sphinx-autobuild" },
{ name = "sphinx-autodoc2" },
{ name = "sphinx-copybutton" },
]
@@ -2317,6 +2646,7 @@ no-pypi-wheels = [
]
test = [
{ name = "coverage" },
+ { name = "mock" },
{ name = "nemo-run" },
{ name = "nltk" },
{ name = "pydantic" },
@@ -2333,6 +2663,8 @@ test = [
[package.metadata]
requires-dist = [
+ { name = "accelerate", marker = "extra == 'mlm'" },
+ { name = "accelerate", marker = "extra == 'training'" },
{ name = "av", marker = "extra == 'dev'" },
{ name = "av", marker = "extra == 'lts'" },
{ name = "causal-conv1d", marker = "extra == 'dev'", specifier = "~=1.5" },
@@ -2349,6 +2681,8 @@ requires-dist = [
{ name = "flashinfer-python", marker = "extra == 'dev'", specifier = "~=0.5.0" },
{ name = "flashinfer-python", marker = "extra == 'lts'", specifier = "~=0.5.0" },
{ name = "flask-restful", marker = "extra == 'mlm'" },
+ { name = "flask-restful", marker = "extra == 'training'" },
+ { name = "hypercorn", marker = "extra == 'dev'" },
{ name = "mamba-ssm", marker = "extra == 'dev'", specifier = "~=2.2" },
{ name = "mamba-ssm", marker = "extra == 'lts'", specifier = "~=2.2" },
{ name = "megatron-energon", extras = ["av-decode"], marker = "extra == 'dev'", specifier = "~=6.0" },
@@ -2356,31 +2690,36 @@ requires-dist = [
{ name = "multi-storage-client", marker = "extra == 'dev'", specifier = "~=0.27" },
{ name = "multi-storage-client", marker = "extra == 'lts'", specifier = "~=0.27" },
{ name = "numpy" },
- { name = "nv-grouped-gemm", marker = "extra == 'dev'", specifier = "~=1.1" },
- { name = "nv-grouped-gemm", marker = "extra == 'lts'", specifier = "~=1.1" },
{ name = "nvidia-modelopt", extras = ["torch"], marker = "sys_platform != 'darwin' and extra == 'dev'" },
- { name = "nvidia-resiliency-ext", marker = "extra == 'dev'" },
+ { name = "nvidia-resiliency-ext", marker = "extra == 'dev'", git = "https://github.com/NVIDIA/nvidia-resiliency-ext.git?rev=63154570cea17f8805a7fd15cc3b8cc2919ba575" },
{ name = "nvtx", marker = "extra == 'dev'", specifier = "~=0.2" },
{ name = "nvtx", marker = "extra == 'lts'", specifier = "~=0.2" },
{ name = "onnxscript", marker = "extra == 'dev'" },
{ name = "onnxscript", marker = "extra == 'lts'" },
+ { name = "openai", extras = ["aiohttp"], marker = "extra == 'dev'" },
{ name = "opentelemetry-api", marker = "extra == 'dev'", specifier = "~=1.33.1" },
{ name = "opentelemetry-api", marker = "extra == 'lts'", specifier = "~=1.33.1" },
+ { name = "orjson", marker = "extra == 'dev'" },
{ name = "packaging", specifier = ">=24.2" },
+ { name = "quart", marker = "extra == 'dev'" },
{ name = "sentencepiece", marker = "extra == 'mlm'" },
+ { name = "sentencepiece", marker = "extra == 'training'" },
{ name = "tensorstore", marker = "extra == 'dev'", specifier = "~=0.1,!=0.1.46,!=0.1.72" },
{ name = "tensorstore", marker = "extra == 'lts'", specifier = "~=0.1,!=0.1.46,!=0.1.72" },
{ name = "tiktoken", marker = "extra == 'mlm'" },
+ { name = "tiktoken", marker = "extra == 'training'" },
{ name = "torch", specifier = ">=2.6.0" },
{ name = "tqdm", marker = "extra == 'dev'" },
{ name = "tqdm", marker = "extra == 'lts'" },
- { name = "transformer-engine", extras = ["core-cu13", "pytorch"], marker = "extra == 'dev'", git = "https://github.com/NVIDIA/TransformerEngine.git?rev=d9b7fc5770a88af06e2e9c2bd97b550614c3a69f" },
+ { name = "transformer-engine", extras = ["core-cu13", "pytorch"], marker = "extra == 'dev'", git = "https://github.com/NVIDIA/TransformerEngine.git?rev=71bbefbf153418f943640df0f7373625dc93fa46" },
{ name = "transformers", marker = "extra == 'mlm'" },
+ { name = "transformers", marker = "extra == 'training'" },
{ name = "wandb", marker = "extra == 'mlm'" },
+ { name = "wandb", marker = "extra == 'training'" },
{ name = "wget", marker = "extra == 'dev'" },
{ name = "wget", marker = "extra == 'lts'" },
]
-provides-extras = ["mlm", "dev", "lts"]
+provides-extras = ["training", "mlm", "dev", "lts"]
[package.metadata.requires-dev]
build = [
@@ -2418,6 +2757,7 @@ no-pypi-wheels = [
]
test = [
{ name = "coverage" },
+ { name = "mock" },
{ name = "nemo-run", git = "https://github.com/NVIDIA-NeMo/Run.git?rev=01a9a8ba360f7b2908728ad0516e0ad9d936966d" },
{ name = "nltk" },
{ name = "pydantic" },
@@ -2440,8 +2780,8 @@ dependencies = [
{ name = "braceexpand" },
{ name = "click" },
{ name = "multi-storage-client" },
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version < '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version >= '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
{ name = "pillow" },
{ name = "pyyaml" },
{ name = "s3fs" },
@@ -2466,23 +2806,63 @@ av-decode = [
[[package]]
name = "ml-dtypes"
-version = "0.5.4"
+version = "0.4.1"
source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
dependencies = [
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version < '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version >= '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/fd/15/76f86faa0902836cc133939732f7611ace68cf54148487a99c539c272dc8/ml_dtypes-0.4.1.tar.gz", hash = "sha256:fad5f2de464fd09127e49b7fd1252b9006fb43d2edc1ff112d390c324af5ca7a", size = 692594, upload-time = "2024-09-13T19:07:11.624Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ba/1a/99e924f12e4b62139fbac87419698c65f956d58de0dbfa7c028fa5b096aa/ml_dtypes-0.4.1-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:827d3ca2097085cf0355f8fdf092b888890bb1b1455f52801a2d7756f056f54b", size = 405077, upload-time = "2024-09-13T19:06:57.538Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/8c/7b610bd500617854c8cc6ed7c8cfb9d48d6a5c21a1437a36a4b9bc8a3598/ml_dtypes-0.4.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:772426b08a6172a891274d581ce58ea2789cc8abc1c002a27223f314aaf894e7", size = 2181554, upload-time = "2024-09-13T19:06:59.196Z" },
+ { url = "https://files.pythonhosted.org/packages/c7/c6/f89620cecc0581dc1839e218c4315171312e46c62a62da6ace204bda91c0/ml_dtypes-0.4.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:126e7d679b8676d1a958f2651949fbfa182832c3cd08020d8facd94e4114f3e9", size = 2160488, upload-time = "2024-09-13T19:07:03.131Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/11/a742d3c31b2cc8557a48efdde53427fd5f9caa2fa3c9c27d826e78a66f51/ml_dtypes-0.4.1-cp312-cp312-win_amd64.whl", hash = "sha256:df0fb650d5c582a9e72bb5bd96cfebb2cdb889d89daff621c8fbc60295eba66c", size = 127462, upload-time = "2024-09-13T19:07:04.916Z" },
+]
+
+[[package]]
+name = "ml-dtypes"
+version = "0.5.4"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.13' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/0e/4a/c27b42ed9b1c7d13d9ba8b6905dece787d6259152f2309338aed29b2447b/ml_dtypes-0.5.4.tar.gz", hash = "sha256:8ab06a50fb9bf9666dd0fe5dfb4676fa2b0ac0f31ecff72a6c3af8e22c063453", size = 692314, upload-time = "2025-11-17T22:32:31.031Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/fe/3a/c5b855752a70267ff729c349e650263adb3c206c29d28cc8ea7ace30a1d5/ml_dtypes-0.5.4-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:b95e97e470fe60ed493fd9ae3911d8da4ebac16bd21f87ffa2b7c588bf22ea2c", size = 679735, upload-time = "2025-11-17T22:31:31.367Z" },
- { url = "https://files.pythonhosted.org/packages/41/79/7433f30ee04bd4faa303844048f55e1eb939131c8e5195a00a96a0939b64/ml_dtypes-0.5.4-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b4b801ebe0b477be666696bda493a9be8356f1f0057a57f1e35cd26928823e5a", size = 5051883, upload-time = "2025-11-17T22:31:33.658Z" },
- { url = "https://files.pythonhosted.org/packages/10/b1/8938e8830b0ee2e167fc75a094dea766a1152bde46752cd9bfc57ee78a82/ml_dtypes-0.5.4-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:388d399a2152dd79a3f0456a952284a99ee5c93d3e2f8dfe25977511e0515270", size = 5030369, upload-time = "2025-11-17T22:31:35.595Z" },
- { url = "https://files.pythonhosted.org/packages/c7/a3/51886727bd16e2f47587997b802dd56398692ce8c6c03c2e5bb32ecafe26/ml_dtypes-0.5.4-cp310-cp310-win_amd64.whl", hash = "sha256:4ff7f3e7ca2972e7de850e7b8fcbb355304271e2933dd90814c1cb847414d6e2", size = 210738, upload-time = "2025-11-17T22:31:37.43Z" },
- { url = "https://files.pythonhosted.org/packages/c6/5e/712092cfe7e5eb667b8ad9ca7c54442f21ed7ca8979745f1000e24cf8737/ml_dtypes-0.5.4-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:6c7ecb74c4bd71db68a6bea1edf8da8c34f3d9fe218f038814fd1d310ac76c90", size = 679734, upload-time = "2025-11-17T22:31:39.223Z" },
- { url = "https://files.pythonhosted.org/packages/4f/cf/912146dfd4b5c0eea956836c01dcd2fce6c9c844b2691f5152aca196ce4f/ml_dtypes-0.5.4-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bc11d7e8c44a65115d05e2ab9989d1e045125d7be8e05a071a48bc76eb6d6040", size = 5056165, upload-time = "2025-11-17T22:31:41.071Z" },
- { url = "https://files.pythonhosted.org/packages/a9/80/19189ea605017473660e43762dc853d2797984b3c7bf30ce656099add30c/ml_dtypes-0.5.4-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:19b9a53598f21e453ea2fbda8aa783c20faff8e1eeb0d7ab899309a0053f1483", size = 5034975, upload-time = "2025-11-17T22:31:42.758Z" },
- { url = "https://files.pythonhosted.org/packages/b4/24/70bd59276883fdd91600ca20040b41efd4902a923283c4d6edcb1de128d2/ml_dtypes-0.5.4-cp311-cp311-win_amd64.whl", hash = "sha256:7c23c54a00ae43edf48d44066a7ec31e05fdc2eee0be2b8b50dd1903a1db94bb", size = 210742, upload-time = "2025-11-17T22:31:44.068Z" },
- { url = "https://files.pythonhosted.org/packages/a0/c9/64230ef14e40aa3f1cb254ef623bf812735e6bec7772848d19131111ac0d/ml_dtypes-0.5.4-cp311-cp311-win_arm64.whl", hash = "sha256:557a31a390b7e9439056644cb80ed0735a6e3e3bb09d67fd5687e4b04238d1de", size = 160709, upload-time = "2025-11-17T22:31:46.557Z" },
{ url = "https://files.pythonhosted.org/packages/a8/b8/3c70881695e056f8a32f8b941126cf78775d9a4d7feba8abcb52cb7b04f2/ml_dtypes-0.5.4-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:a174837a64f5b16cab6f368171a1a03a27936b31699d167684073ff1c4237dac", size = 676927, upload-time = "2025-11-17T22:31:48.182Z" },
{ url = "https://files.pythonhosted.org/packages/54/0f/428ef6881782e5ebb7eca459689448c0394fa0a80bea3aa9262cba5445ea/ml_dtypes-0.5.4-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a7f7c643e8b1320fd958bf098aa7ecf70623a42ec5154e3be3be673f4c34d900", size = 5028464, upload-time = "2025-11-17T22:31:50.135Z" },
{ url = "https://files.pythonhosted.org/packages/3a/cb/28ce52eb94390dda42599c98ea0204d74799e4d8047a0eb559b6fd648056/ml_dtypes-0.5.4-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9ad459e99793fa6e13bd5b7e6792c8f9190b4e5a1b45c63aba14a4d0a7f1d5ff", size = 5009002, upload-time = "2025-11-17T22:31:52.001Z" },
@@ -2510,6 +2890,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/ad/3f/3d42e9a78fe5edf792a83c074b13b9b770092a4fbf3462872f4303135f09/ml_dtypes-0.5.4-cp314-cp314t-win_arm64.whl", hash = "sha256:11942cbf2cf92157db91e5022633c0d9474d4dfd813a909383bd23ce828a4b7d", size = 168825, upload-time = "2025-11-17T22:32:23.766Z" },
]
+[[package]]
+name = "mock"
+version = "5.2.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/07/8c/14c2ae915e5f9dca5a22edd68b35be94400719ccfa068a03e0fb63d0f6f6/mock-5.2.0.tar.gz", hash = "sha256:4e460e818629b4b173f32d08bf30d3af8123afbb8e04bb5707a1fd4799e503f0", size = 92796, upload-time = "2025-03-03T12:31:42.911Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/bd/d9/617e6af809bf3a1d468e0d58c3997b1dc219a9a9202e650d30c2fc85d481/mock-5.2.0-py3-none-any.whl", hash = "sha256:7ba87f72ca0e915175596069dbbcc7c75af7b5e9b9bc107ad6349ede0819982f", size = 31617, upload-time = "2025-03-03T12:31:41.518Z" },
+]
+
[[package]]
name = "mpmath"
version = "1.3.0"
@@ -2525,23 +2914,6 @@ version = "1.1.2"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/4d/f2/bfb55a6236ed8725a96b0aa3acbd0ec17588e6a2c3b62a93eb513ed8783f/msgpack-1.1.2.tar.gz", hash = "sha256:3b60763c1373dd60f398488069bcdc703cd08a711477b5d480eecc9f9626f47e", size = 173581, upload-time = "2025-10-08T09:15:56.596Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/f5/a2/3b68a9e769db68668b25c6108444a35f9bd163bb848c0650d516761a59c0/msgpack-1.1.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:0051fffef5a37ca2cd16978ae4f0aef92f164df86823871b5162812bebecd8e2", size = 81318, upload-time = "2025-10-08T09:14:38.722Z" },
- { url = "https://files.pythonhosted.org/packages/5b/e1/2b720cc341325c00be44e1ed59e7cfeae2678329fbf5aa68f5bda57fe728/msgpack-1.1.2-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:a605409040f2da88676e9c9e5853b3449ba8011973616189ea5ee55ddbc5bc87", size = 83786, upload-time = "2025-10-08T09:14:40.082Z" },
- { url = "https://files.pythonhosted.org/packages/71/e5/c2241de64bfceac456b140737812a2ab310b10538a7b34a1d393b748e095/msgpack-1.1.2-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8b696e83c9f1532b4af884045ba7f3aa741a63b2bc22617293a2c6a7c645f251", size = 398240, upload-time = "2025-10-08T09:14:41.151Z" },
- { url = "https://files.pythonhosted.org/packages/b7/09/2a06956383c0fdebaef5aa9246e2356776f12ea6f2a44bd1368abf0e46c4/msgpack-1.1.2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:365c0bbe981a27d8932da71af63ef86acc59ed5c01ad929e09a0b88c6294e28a", size = 406070, upload-time = "2025-10-08T09:14:42.821Z" },
- { url = "https://files.pythonhosted.org/packages/0e/74/2957703f0e1ef20637d6aead4fbb314330c26f39aa046b348c7edcf6ca6b/msgpack-1.1.2-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:41d1a5d875680166d3ac5c38573896453bbbea7092936d2e107214daf43b1d4f", size = 393403, upload-time = "2025-10-08T09:14:44.38Z" },
- { url = "https://files.pythonhosted.org/packages/a5/09/3bfc12aa90f77b37322fc33e7a8a7c29ba7c8edeadfa27664451801b9860/msgpack-1.1.2-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:354e81bcdebaab427c3df4281187edc765d5d76bfb3a7c125af9da7a27e8458f", size = 398947, upload-time = "2025-10-08T09:14:45.56Z" },
- { url = "https://files.pythonhosted.org/packages/4b/4f/05fcebd3b4977cb3d840f7ef6b77c51f8582086de5e642f3fefee35c86fc/msgpack-1.1.2-cp310-cp310-win32.whl", hash = "sha256:e64c8d2f5e5d5fda7b842f55dec6133260ea8f53c4257d64494c534f306bf7a9", size = 64769, upload-time = "2025-10-08T09:14:47.334Z" },
- { url = "https://files.pythonhosted.org/packages/d0/3e/b4547e3a34210956382eed1c85935fff7e0f9b98be3106b3745d7dec9c5e/msgpack-1.1.2-cp310-cp310-win_amd64.whl", hash = "sha256:db6192777d943bdaaafb6ba66d44bf65aa0e9c5616fa1d2da9bb08828c6b39aa", size = 71293, upload-time = "2025-10-08T09:14:48.665Z" },
- { url = "https://files.pythonhosted.org/packages/2c/97/560d11202bcd537abca693fd85d81cebe2107ba17301de42b01ac1677b69/msgpack-1.1.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:2e86a607e558d22985d856948c12a3fa7b42efad264dca8a3ebbcfa2735d786c", size = 82271, upload-time = "2025-10-08T09:14:49.967Z" },
- { url = "https://files.pythonhosted.org/packages/83/04/28a41024ccbd67467380b6fb440ae916c1e4f25e2cd4c63abe6835ac566e/msgpack-1.1.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:283ae72fc89da59aa004ba147e8fc2f766647b1251500182fac0350d8af299c0", size = 84914, upload-time = "2025-10-08T09:14:50.958Z" },
- { url = "https://files.pythonhosted.org/packages/71/46/b817349db6886d79e57a966346cf0902a426375aadc1e8e7a86a75e22f19/msgpack-1.1.2-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:61c8aa3bd513d87c72ed0b37b53dd5c5a0f58f2ff9f26e1555d3bd7948fb7296", size = 416962, upload-time = "2025-10-08T09:14:51.997Z" },
- { url = "https://files.pythonhosted.org/packages/da/e0/6cc2e852837cd6086fe7d8406af4294e66827a60a4cf60b86575a4a65ca8/msgpack-1.1.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:454e29e186285d2ebe65be34629fa0e8605202c60fbc7c4c650ccd41870896ef", size = 426183, upload-time = "2025-10-08T09:14:53.477Z" },
- { url = "https://files.pythonhosted.org/packages/25/98/6a19f030b3d2ea906696cedd1eb251708e50a5891d0978b012cb6107234c/msgpack-1.1.2-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:7bc8813f88417599564fafa59fd6f95be417179f76b40325b500b3c98409757c", size = 411454, upload-time = "2025-10-08T09:14:54.648Z" },
- { url = "https://files.pythonhosted.org/packages/b7/cd/9098fcb6adb32187a70b7ecaabf6339da50553351558f37600e53a4a2a23/msgpack-1.1.2-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:bafca952dc13907bdfdedfc6a5f579bf4f292bdd506fadb38389afa3ac5b208e", size = 422341, upload-time = "2025-10-08T09:14:56.328Z" },
- { url = "https://files.pythonhosted.org/packages/e6/ae/270cecbcf36c1dc85ec086b33a51a4d7d08fc4f404bdbc15b582255d05ff/msgpack-1.1.2-cp311-cp311-win32.whl", hash = "sha256:602b6740e95ffc55bfb078172d279de3773d7b7db1f703b2f1323566b878b90e", size = 64747, upload-time = "2025-10-08T09:14:57.882Z" },
- { url = "https://files.pythonhosted.org/packages/2a/79/309d0e637f6f37e83c711f547308b91af02b72d2326ddd860b966080ef29/msgpack-1.1.2-cp311-cp311-win_amd64.whl", hash = "sha256:d198d275222dc54244bf3327eb8cbe00307d220241d9cec4d306d49a44e85f68", size = 71633, upload-time = "2025-10-08T09:14:59.177Z" },
- { url = "https://files.pythonhosted.org/packages/73/4d/7c4e2b3d9b1106cd0aa6cb56cc57c6267f59fa8bfab7d91df5adc802c847/msgpack-1.1.2-cp311-cp311-win_arm64.whl", hash = "sha256:86f8136dfa5c116365a8a651a7d7484b65b13339731dd6faebb9a0242151c406", size = 64755, upload-time = "2025-10-08T09:15:00.48Z" },
{ url = "https://files.pythonhosted.org/packages/ad/bd/8b0d01c756203fbab65d265859749860682ccd2a59594609aeec3a144efa/msgpack-1.1.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:70a0dff9d1f8da25179ffcf880e10cf1aad55fdb63cd59c9a49a1b82290062aa", size = 81939, upload-time = "2025-10-08T09:15:01.472Z" },
{ url = "https://files.pythonhosted.org/packages/34/68/ba4f155f793a74c1483d4bdef136e1023f7bcba557f0db4ef3db3c665cf1/msgpack-1.1.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:446abdd8b94b55c800ac34b102dffd2f6aa0ce643c55dfc017ad89347db3dbdb", size = 85064, upload-time = "2025-10-08T09:15:03.764Z" },
{ url = "https://files.pythonhosted.org/packages/f2/60/a064b0345fc36c4c3d2c743c82d9100c40388d77f0b48b2f04d6041dbec1/msgpack-1.1.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c63eea553c69ab05b6747901b97d620bb2a690633c77f23feb0c6a947a8a7b8f", size = 417131, upload-time = "2025-10-08T09:15:05.136Z" },
@@ -2582,14 +2954,14 @@ wheels = [
[[package]]
name = "multi-storage-client"
-version = "0.40.0"
+version = "0.45.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "filelock" },
{ name = "jmespath" },
{ name = "jsonschema" },
{ name = "lark" },
- { name = "opentelemetry-api" },
+ { name = "opentelemetry-api", version = "1.33.1", source = { registry = "https://pypi.org/simple" } },
{ name = "prettytable" },
{ name = "psutil" },
{ name = "python-dateutil" },
@@ -2600,179 +2972,128 @@ dependencies = [
{ name = "xattr" },
]
wheels = [
- { url = "https://files.pythonhosted.org/packages/af/63/3ecdef2bd3e627d4915497315db8c9fdd86f8443c2ea858b0ebae3116edd/multi_storage_client-0.40.0-cp310-cp310-macosx_14_0_arm64.whl", hash = "sha256:ab13383f3c70b5dfa784bc26c8e40777c03c6969c87107c72424a365140635e3", size = 8441996, upload-time = "2026-01-06T20:21:02.68Z" },
- { url = "https://files.pythonhosted.org/packages/a7/4c/1bb4945379009d2197689742ef1a932862e269a8f2267e57fa439d77bd58/multi_storage_client-0.40.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:96d7d64e06da0d0726ad9cad1d741a312d65e5be84ac93880bda5a81dcb38439", size = 4795528, upload-time = "2026-01-06T20:19:13.6Z" },
- { url = "https://files.pythonhosted.org/packages/d9/1e/d44fc5b1f1a05b7ce4b3b5edde7f8daa7b4dcf05a61b7a0c9e4fe22af1bd/multi_storage_client-0.40.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:58da2b465f8b90f6deee94d9f59b9911a752592220e323b855ee1509ae613a27", size = 5061490, upload-time = "2026-01-06T20:17:26.767Z" },
- { url = "https://files.pythonhosted.org/packages/3b/f9/73072df16c61e8927691d6d636951e8954371882f9fea8b93fdef42ae315/multi_storage_client-0.40.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:6a87296d15900a8e1e819ee6cdca9b311e892622108e947066455a3797fb8508", size = 8439893, upload-time = "2026-01-06T20:20:15.162Z" },
- { url = "https://files.pythonhosted.org/packages/92/d4/512b14589cfa739426e3852fda62f774aa7ea7ba48877e75c3d03d091ca9/multi_storage_client-0.40.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4b51c3ef6a69c7d9e98ee5b9aa2b511228b0802dd837048da5a1b79413d016fe", size = 4795279, upload-time = "2026-01-06T20:19:45.661Z" },
- { url = "https://files.pythonhosted.org/packages/49/aa/8c3a9557fb39bfb57842ac6f39f7fd614ab68e299ead20695ed3ffd90a99/multi_storage_client-0.40.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d4f85db520f058c29c0553a0b4cc7a67811879dcc72ed8cc091e1c41d65874b7", size = 5061699, upload-time = "2026-01-06T20:15:34.596Z" },
- { url = "https://files.pythonhosted.org/packages/92/65/448a08141d34629e601edca69883268801a02ef385b6c70b4bffe37074f5/multi_storage_client-0.40.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:7719d181f3be9d76c1335a5e6b83db02bbbe81b64d786a7dfbdf5fdf4edabd02", size = 8434884, upload-time = "2026-01-06T20:21:52.106Z" },
- { url = "https://files.pythonhosted.org/packages/3b/38/28ee280cab9c47c24a251f88f8b461fac00aebb7ae5dc045c8bf46fc4c05/multi_storage_client-0.40.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b36eaf7260d5bc017bba2fb77529365ca89f7180111065f820df03b55f9132ad", size = 4800088, upload-time = "2026-01-06T20:21:27.251Z" },
- { url = "https://files.pythonhosted.org/packages/1a/57/24120b57f2e30372fcf0d3ec5673e0824414ccd9a312a5669f7274980bd9/multi_storage_client-0.40.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e8fe94c6ed7861fc38f55054fb13e05a818a1a738d7625eb057920ab5292b324", size = 5061276, upload-time = "2026-01-06T20:16:37.388Z" },
- { url = "https://files.pythonhosted.org/packages/98/3d/b0cb5eb1d6f5b36c4226d74f83b14fe45e120807cf059e8db6fac3017ca0/multi_storage_client-0.40.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:a3f7998b7ddea2e4b669570384be937cab53add5734154c5543098789f8a3db2", size = 8433939, upload-time = "2026-01-06T20:17:03.222Z" },
- { url = "https://files.pythonhosted.org/packages/75/ea/fcec0e93d23fc26cc622cefb41a574f5b1697cc33d86e0e8e48f7da4248a/multi_storage_client-0.40.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7be2a973661d19434ec744e3c7b7cdf6a36d0aa22ba25b62b44b6a20a18aee88", size = 4799908, upload-time = "2026-01-06T20:17:54.691Z" },
- { url = "https://files.pythonhosted.org/packages/ae/f3/fc07bd7efcffa5422e746550231d0edd6459f9686edf03c1ad961fd4d721/multi_storage_client-0.40.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4928f1c6b0cc4011d785c6fac10114d61833d6aa10c0e22ecce85090bf868e4c", size = 5060685, upload-time = "2026-01-06T20:16:14.272Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/59/b2a05f8ecfdfe3c5324b8ff7ee17434df56130e2cd5f8b35d1c5b188c4de/multi_storage_client-0.45.1-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:ef7dff8c27c1c092b58ba06a37aac0a2441a109202f36afc986b05e1e73f97a2", size = 9011744, upload-time = "2026-03-24T17:39:22.665Z" },
+ { url = "https://files.pythonhosted.org/packages/af/b9/d0c25e18eadc5a6f14b1c4c53b781d7f669bd310046d35dc658ab0f98b4c/multi_storage_client-0.45.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1439d6acd6957c6d4e118305e08fb029c4b36814ca9c8070dcacc741b351ba9a", size = 5395849, upload-time = "2026-03-24T17:39:46.041Z" },
+ { url = "https://files.pythonhosted.org/packages/33/9b/0134d06d66a99e1fd39c6d2c1006cbca59e04db39d339a4de0bfcaacb188/multi_storage_client-0.45.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d8960b5d661cbf79349186089f59a57a9f22703d41e99e73a17c9e0f1c2737f9", size = 5585758, upload-time = "2026-03-24T17:40:10.248Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/72/565fe8f0a6ce1639719fccc5049d9dbab4d007c94978254eca3134631b4c/multi_storage_client-0.45.1-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:9de9f980d662ee741ef8ddfd84a72b6e5ab55cd91162f5182a6ad9bf75f4be8d", size = 9008456, upload-time = "2026-03-24T17:41:20.877Z" },
+ { url = "https://files.pythonhosted.org/packages/34/b2/72ca7f3d93a147edf47fe384e71146e4df0c679537acb7fe3436c6018335/multi_storage_client-0.45.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:da7285874857102b2562d2d381cde115c6cdcc3b6d0c5bc1e30c5f41a00a1518", size = 5395483, upload-time = "2026-03-24T17:38:58.779Z" },
+ { url = "https://files.pythonhosted.org/packages/5e/00/070aec0b31cae59633beaf3e696f8ecc84ae8c084aa414346bd71b95afc1/multi_storage_client-0.45.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e62bf1442931d9923d25ee7c5067a491a431294d8e9fbf56546c9ae3e1542537", size = 5585772, upload-time = "2026-03-24T17:35:53.259Z" },
]
[[package]]
name = "multidict"
-version = "6.7.0"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
- { name = "typing-extensions", marker = "python_full_version < '3.11'" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/80/1e/5492c365f222f907de1039b91f922b93fa4f764c713ee858d235495d8f50/multidict-6.7.0.tar.gz", hash = "sha256:c6e99d9a65ca282e578dfea819cfa9c0a62b2499d8677392e09feaf305e9e6f5", size = 101834, upload-time = "2025-10-06T14:52:30.657Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/a9/63/7bdd4adc330abcca54c85728db2327130e49e52e8c3ce685cec44e0f2e9f/multidict-6.7.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:9f474ad5acda359c8758c8accc22032c6abe6dc87a8be2440d097785e27a9349", size = 77153, upload-time = "2025-10-06T14:48:26.409Z" },
- { url = "https://files.pythonhosted.org/packages/3f/bb/b6c35ff175ed1a3142222b78455ee31be71a8396ed3ab5280fbe3ebe4e85/multidict-6.7.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:4b7a9db5a870f780220e931d0002bbfd88fb53aceb6293251e2c839415c1b20e", size = 44993, upload-time = "2025-10-06T14:48:28.4Z" },
- { url = "https://files.pythonhosted.org/packages/e0/1f/064c77877c5fa6df6d346e68075c0f6998547afe952d6471b4c5f6a7345d/multidict-6.7.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:03ca744319864e92721195fa28c7a3b2bc7b686246b35e4078c1e4d0eb5466d3", size = 44607, upload-time = "2025-10-06T14:48:29.581Z" },
- { url = "https://files.pythonhosted.org/packages/04/7a/bf6aa92065dd47f287690000b3d7d332edfccb2277634cadf6a810463c6a/multidict-6.7.0-cp310-cp310-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:f0e77e3c0008bc9316e662624535b88d360c3a5d3f81e15cf12c139a75250046", size = 241847, upload-time = "2025-10-06T14:48:32.107Z" },
- { url = "https://files.pythonhosted.org/packages/94/39/297a8de920f76eda343e4ce05f3b489f0ab3f9504f2576dfb37b7c08ca08/multidict-6.7.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:08325c9e5367aa379a3496aa9a022fe8837ff22e00b94db256d3a1378c76ab32", size = 242616, upload-time = "2025-10-06T14:48:34.054Z" },
- { url = "https://files.pythonhosted.org/packages/39/3a/d0eee2898cfd9d654aea6cb8c4addc2f9756e9a7e09391cfe55541f917f7/multidict-6.7.0-cp310-cp310-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:e2862408c99f84aa571ab462d25236ef9cb12a602ea959ba9c9009a54902fc73", size = 222333, upload-time = "2025-10-06T14:48:35.9Z" },
- { url = "https://files.pythonhosted.org/packages/05/48/3b328851193c7a4240815b71eea165b49248867bbb6153a0aee227a0bb47/multidict-6.7.0-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4d72a9a2d885f5c208b0cb91ff2ed43636bb7e345ec839ff64708e04f69a13cc", size = 253239, upload-time = "2025-10-06T14:48:37.302Z" },
- { url = "https://files.pythonhosted.org/packages/b1/ca/0706a98c8d126a89245413225ca4a3fefc8435014de309cf8b30acb68841/multidict-6.7.0-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:478cc36476687bac1514d651cbbaa94b86b0732fb6855c60c673794c7dd2da62", size = 251618, upload-time = "2025-10-06T14:48:38.963Z" },
- { url = "https://files.pythonhosted.org/packages/5e/4f/9c7992f245554d8b173f6f0a048ad24b3e645d883f096857ec2c0822b8bd/multidict-6.7.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6843b28b0364dc605f21481c90fadb5f60d9123b442eb8a726bb74feef588a84", size = 241655, upload-time = "2025-10-06T14:48:40.312Z" },
- { url = "https://files.pythonhosted.org/packages/31/79/26a85991ae67efd1c0b1fc2e0c275b8a6aceeb155a68861f63f87a798f16/multidict-6.7.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:23bfeee5316266e5ee2d625df2d2c602b829435fc3a235c2ba2131495706e4a0", size = 239245, upload-time = "2025-10-06T14:48:41.848Z" },
- { url = "https://files.pythonhosted.org/packages/14/1e/75fa96394478930b79d0302eaf9a6c69f34005a1a5251ac8b9c336486ec9/multidict-6.7.0-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:680878b9f3d45c31e1f730eef731f9b0bc1da456155688c6745ee84eb818e90e", size = 233523, upload-time = "2025-10-06T14:48:43.749Z" },
- { url = "https://files.pythonhosted.org/packages/b2/5e/085544cb9f9c4ad2b5d97467c15f856df8d9bac410cffd5c43991a5d878b/multidict-6.7.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:eb866162ef2f45063acc7a53a88ef6fe8bf121d45c30ea3c9cd87ce7e191a8d4", size = 243129, upload-time = "2025-10-06T14:48:45.225Z" },
- { url = "https://files.pythonhosted.org/packages/b9/c3/e9d9e2f20c9474e7a8fcef28f863c5cbd29bb5adce6b70cebe8bdad0039d/multidict-6.7.0-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:df0e3bf7993bdbeca5ac25aa859cf40d39019e015c9c91809ba7093967f7a648", size = 248999, upload-time = "2025-10-06T14:48:46.703Z" },
- { url = "https://files.pythonhosted.org/packages/b5/3f/df171b6efa3239ae33b97b887e42671cd1d94d460614bfb2c30ffdab3b95/multidict-6.7.0-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:661709cdcd919a2ece2234f9bae7174e5220c80b034585d7d8a755632d3e2111", size = 243711, upload-time = "2025-10-06T14:48:48.146Z" },
- { url = "https://files.pythonhosted.org/packages/3c/2f/9b5564888c4e14b9af64c54acf149263721a283aaf4aa0ae89b091d5d8c1/multidict-6.7.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:096f52730c3fb8ed419db2d44391932b63891b2c5ed14850a7e215c0ba9ade36", size = 237504, upload-time = "2025-10-06T14:48:49.447Z" },
- { url = "https://files.pythonhosted.org/packages/6c/3a/0bd6ca0f7d96d790542d591c8c3354c1e1b6bfd2024d4d92dc3d87485ec7/multidict-6.7.0-cp310-cp310-win32.whl", hash = "sha256:afa8a2978ec65d2336305550535c9c4ff50ee527914328c8677b3973ade52b85", size = 41422, upload-time = "2025-10-06T14:48:50.789Z" },
- { url = "https://files.pythonhosted.org/packages/00/35/f6a637ea2c75f0d3b7c7d41b1189189acff0d9deeb8b8f35536bb30f5e33/multidict-6.7.0-cp310-cp310-win_amd64.whl", hash = "sha256:b15b3afff74f707b9275d5ba6a91ae8f6429c3ffb29bbfd216b0b375a56f13d7", size = 46050, upload-time = "2025-10-06T14:48:51.938Z" },
- { url = "https://files.pythonhosted.org/packages/e7/b8/f7bf8329b39893d02d9d95cf610c75885d12fc0f402b1c894e1c8e01c916/multidict-6.7.0-cp310-cp310-win_arm64.whl", hash = "sha256:4b73189894398d59131a66ff157837b1fafea9974be486d036bb3d32331fdbf0", size = 43153, upload-time = "2025-10-06T14:48:53.146Z" },
- { url = "https://files.pythonhosted.org/packages/34/9e/5c727587644d67b2ed479041e4b1c58e30afc011e3d45d25bbe35781217c/multidict-6.7.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:4d409aa42a94c0b3fa617708ef5276dfe81012ba6753a0370fcc9d0195d0a1fc", size = 76604, upload-time = "2025-10-06T14:48:54.277Z" },
- { url = "https://files.pythonhosted.org/packages/17/e4/67b5c27bd17c085a5ea8f1ec05b8a3e5cba0ca734bfcad5560fb129e70ca/multidict-6.7.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:14c9e076eede3b54c636f8ce1c9c252b5f057c62131211f0ceeec273810c9721", size = 44715, upload-time = "2025-10-06T14:48:55.445Z" },
- { url = "https://files.pythonhosted.org/packages/4d/e1/866a5d77be6ea435711bef2a4291eed11032679b6b28b56b4776ab06ba3e/multidict-6.7.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:4c09703000a9d0fa3c3404b27041e574cc7f4df4c6563873246d0e11812a94b6", size = 44332, upload-time = "2025-10-06T14:48:56.706Z" },
- { url = "https://files.pythonhosted.org/packages/31/61/0c2d50241ada71ff61a79518db85ada85fdabfcf395d5968dae1cbda04e5/multidict-6.7.0-cp311-cp311-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:a265acbb7bb33a3a2d626afbe756371dce0279e7b17f4f4eda406459c2b5ff1c", size = 245212, upload-time = "2025-10-06T14:48:58.042Z" },
- { url = "https://files.pythonhosted.org/packages/ac/e0/919666a4e4b57fff1b57f279be1c9316e6cdc5de8a8b525d76f6598fefc7/multidict-6.7.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:51cb455de290ae462593e5b1cb1118c5c22ea7f0d3620d9940bf695cea5a4bd7", size = 246671, upload-time = "2025-10-06T14:49:00.004Z" },
- { url = "https://files.pythonhosted.org/packages/a1/cc/d027d9c5a520f3321b65adea289b965e7bcbd2c34402663f482648c716ce/multidict-6.7.0-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:db99677b4457c7a5c5a949353e125ba72d62b35f74e26da141530fbb012218a7", size = 225491, upload-time = "2025-10-06T14:49:01.393Z" },
- { url = "https://files.pythonhosted.org/packages/75/c4/bbd633980ce6155a28ff04e6a6492dd3335858394d7bb752d8b108708558/multidict-6.7.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f470f68adc395e0183b92a2f4689264d1ea4b40504a24d9882c27375e6662bb9", size = 257322, upload-time = "2025-10-06T14:49:02.745Z" },
- { url = "https://files.pythonhosted.org/packages/4c/6d/d622322d344f1f053eae47e033b0b3f965af01212de21b10bcf91be991fb/multidict-6.7.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:0db4956f82723cc1c270de9c6e799b4c341d327762ec78ef82bb962f79cc07d8", size = 254694, upload-time = "2025-10-06T14:49:04.15Z" },
- { url = "https://files.pythonhosted.org/packages/a8/9f/78f8761c2705d4c6d7516faed63c0ebdac569f6db1bef95e0d5218fdc146/multidict-6.7.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3e56d780c238f9e1ae66a22d2adf8d16f485381878250db8d496623cd38b22bd", size = 246715, upload-time = "2025-10-06T14:49:05.967Z" },
- { url = "https://files.pythonhosted.org/packages/78/59/950818e04f91b9c2b95aab3d923d9eabd01689d0dcd889563988e9ea0fd8/multidict-6.7.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:9d14baca2ee12c1a64740d4531356ba50b82543017f3ad6de0deb943c5979abb", size = 243189, upload-time = "2025-10-06T14:49:07.37Z" },
- { url = "https://files.pythonhosted.org/packages/7a/3d/77c79e1934cad2ee74991840f8a0110966d9599b3af95964c0cd79bb905b/multidict-6.7.0-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:295a92a76188917c7f99cda95858c822f9e4aae5824246bba9b6b44004ddd0a6", size = 237845, upload-time = "2025-10-06T14:49:08.759Z" },
- { url = "https://files.pythonhosted.org/packages/63/1b/834ce32a0a97a3b70f86437f685f880136677ac00d8bce0027e9fd9c2db7/multidict-6.7.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:39f1719f57adbb767ef592a50ae5ebb794220d1188f9ca93de471336401c34d2", size = 246374, upload-time = "2025-10-06T14:49:10.574Z" },
- { url = "https://files.pythonhosted.org/packages/23/ef/43d1c3ba205b5dec93dc97f3fba179dfa47910fc73aaaea4f7ceb41cec2a/multidict-6.7.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:0a13fb8e748dfc94749f622de065dd5c1def7e0d2216dba72b1d8069a389c6ff", size = 253345, upload-time = "2025-10-06T14:49:12.331Z" },
- { url = "https://files.pythonhosted.org/packages/6b/03/eaf95bcc2d19ead522001f6a650ef32811aa9e3624ff0ad37c445c7a588c/multidict-6.7.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:e3aa16de190d29a0ea1b48253c57d99a68492c8dd8948638073ab9e74dc9410b", size = 246940, upload-time = "2025-10-06T14:49:13.821Z" },
- { url = "https://files.pythonhosted.org/packages/e8/df/ec8a5fd66ea6cd6f525b1fcbb23511b033c3e9bc42b81384834ffa484a62/multidict-6.7.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:a048ce45dcdaaf1defb76b2e684f997fb5abf74437b6cb7b22ddad934a964e34", size = 242229, upload-time = "2025-10-06T14:49:15.603Z" },
- { url = "https://files.pythonhosted.org/packages/8a/a2/59b405d59fd39ec86d1142630e9049243015a5f5291ba49cadf3c090c541/multidict-6.7.0-cp311-cp311-win32.whl", hash = "sha256:a90af66facec4cebe4181b9e62a68be65e45ac9b52b67de9eec118701856e7ff", size = 41308, upload-time = "2025-10-06T14:49:16.871Z" },
- { url = "https://files.pythonhosted.org/packages/32/0f/13228f26f8b882c34da36efa776c3b7348455ec383bab4a66390e42963ae/multidict-6.7.0-cp311-cp311-win_amd64.whl", hash = "sha256:95b5ffa4349df2887518bb839409bcf22caa72d82beec453216802f475b23c81", size = 46037, upload-time = "2025-10-06T14:49:18.457Z" },
- { url = "https://files.pythonhosted.org/packages/84/1f/68588e31b000535a3207fd3c909ebeec4fb36b52c442107499c18a896a2a/multidict-6.7.0-cp311-cp311-win_arm64.whl", hash = "sha256:329aa225b085b6f004a4955271a7ba9f1087e39dcb7e65f6284a988264a63912", size = 43023, upload-time = "2025-10-06T14:49:19.648Z" },
- { url = "https://files.pythonhosted.org/packages/c2/9e/9f61ac18d9c8b475889f32ccfa91c9f59363480613fc807b6e3023d6f60b/multidict-6.7.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:8a3862568a36d26e650a19bb5cbbba14b71789032aebc0423f8cc5f150730184", size = 76877, upload-time = "2025-10-06T14:49:20.884Z" },
- { url = "https://files.pythonhosted.org/packages/38/6f/614f09a04e6184f8824268fce4bc925e9849edfa654ddd59f0b64508c595/multidict-6.7.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:960c60b5849b9b4f9dcc9bea6e3626143c252c74113df2c1540aebce70209b45", size = 45467, upload-time = "2025-10-06T14:49:22.054Z" },
- { url = "https://files.pythonhosted.org/packages/b3/93/c4f67a436dd026f2e780c433277fff72be79152894d9fc36f44569cab1a6/multidict-6.7.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:2049be98fb57a31b4ccf870bf377af2504d4ae35646a19037ec271e4c07998aa", size = 43834, upload-time = "2025-10-06T14:49:23.566Z" },
- { url = "https://files.pythonhosted.org/packages/7f/f5/013798161ca665e4a422afbc5e2d9e4070142a9ff8905e482139cd09e4d0/multidict-6.7.0-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:0934f3843a1860dd465d38895c17fce1f1cb37295149ab05cd1b9a03afacb2a7", size = 250545, upload-time = "2025-10-06T14:49:24.882Z" },
- { url = "https://files.pythonhosted.org/packages/71/2f/91dbac13e0ba94669ea5119ba267c9a832f0cb65419aca75549fcf09a3dc/multidict-6.7.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b3e34f3a1b8131ba06f1a73adab24f30934d148afcd5f5de9a73565a4404384e", size = 258305, upload-time = "2025-10-06T14:49:26.778Z" },
- { url = "https://files.pythonhosted.org/packages/ef/b0/754038b26f6e04488b48ac621f779c341338d78503fb45403755af2df477/multidict-6.7.0-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:efbb54e98446892590dc2458c19c10344ee9a883a79b5cec4bc34d6656e8d546", size = 242363, upload-time = "2025-10-06T14:49:28.562Z" },
- { url = "https://files.pythonhosted.org/packages/87/15/9da40b9336a7c9fa606c4cf2ed80a649dffeb42b905d4f63a1d7eb17d746/multidict-6.7.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a35c5fc61d4f51eb045061e7967cfe3123d622cd500e8868e7c0c592a09fedc4", size = 268375, upload-time = "2025-10-06T14:49:29.96Z" },
- { url = "https://files.pythonhosted.org/packages/82/72/c53fcade0cc94dfaad583105fd92b3a783af2091eddcb41a6d5a52474000/multidict-6.7.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:29fe6740ebccba4175af1b9b87bf553e9c15cd5868ee967e010efcf94e4fd0f1", size = 269346, upload-time = "2025-10-06T14:49:31.404Z" },
- { url = "https://files.pythonhosted.org/packages/0d/e2/9baffdae21a76f77ef8447f1a05a96ec4bc0a24dae08767abc0a2fe680b8/multidict-6.7.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:123e2a72e20537add2f33a79e605f6191fba2afda4cbb876e35c1a7074298a7d", size = 256107, upload-time = "2025-10-06T14:49:32.974Z" },
- { url = "https://files.pythonhosted.org/packages/3c/06/3f06f611087dc60d65ef775f1fb5aca7c6d61c6db4990e7cda0cef9b1651/multidict-6.7.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:b284e319754366c1aee2267a2036248b24eeb17ecd5dc16022095e747f2f4304", size = 253592, upload-time = "2025-10-06T14:49:34.52Z" },
- { url = "https://files.pythonhosted.org/packages/20/24/54e804ec7945b6023b340c412ce9c3f81e91b3bf5fa5ce65558740141bee/multidict-6.7.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:803d685de7be4303b5a657b76e2f6d1240e7e0a8aa2968ad5811fa2285553a12", size = 251024, upload-time = "2025-10-06T14:49:35.956Z" },
- { url = "https://files.pythonhosted.org/packages/14/48/011cba467ea0b17ceb938315d219391d3e421dfd35928e5dbdc3f4ae76ef/multidict-6.7.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:c04a328260dfd5db8c39538f999f02779012268f54614902d0afc775d44e0a62", size = 251484, upload-time = "2025-10-06T14:49:37.631Z" },
- { url = "https://files.pythonhosted.org/packages/0d/2f/919258b43bb35b99fa127435cfb2d91798eb3a943396631ef43e3720dcf4/multidict-6.7.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:8a19cdb57cd3df4cd865849d93ee14920fb97224300c88501f16ecfa2604b4e0", size = 263579, upload-time = "2025-10-06T14:49:39.502Z" },
- { url = "https://files.pythonhosted.org/packages/31/22/a0e884d86b5242b5a74cf08e876bdf299e413016b66e55511f7a804a366e/multidict-6.7.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:9b2fd74c52accced7e75de26023b7dccee62511a600e62311b918ec5c168fc2a", size = 259654, upload-time = "2025-10-06T14:49:41.32Z" },
- { url = "https://files.pythonhosted.org/packages/b2/e5/17e10e1b5c5f5a40f2fcbb45953c9b215f8a4098003915e46a93f5fcaa8f/multidict-6.7.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3e8bfdd0e487acf992407a140d2589fe598238eaeffa3da8448d63a63cd363f8", size = 251511, upload-time = "2025-10-06T14:49:46.021Z" },
- { url = "https://files.pythonhosted.org/packages/e3/9a/201bb1e17e7af53139597069c375e7b0dcbd47594604f65c2d5359508566/multidict-6.7.0-cp312-cp312-win32.whl", hash = "sha256:dd32a49400a2c3d52088e120ee00c1e3576cbff7e10b98467962c74fdb762ed4", size = 41895, upload-time = "2025-10-06T14:49:48.718Z" },
- { url = "https://files.pythonhosted.org/packages/46/e2/348cd32faad84eaf1d20cce80e2bb0ef8d312c55bca1f7fa9865e7770aaf/multidict-6.7.0-cp312-cp312-win_amd64.whl", hash = "sha256:92abb658ef2d7ef22ac9f8bb88e8b6c3e571671534e029359b6d9e845923eb1b", size = 46073, upload-time = "2025-10-06T14:49:50.28Z" },
- { url = "https://files.pythonhosted.org/packages/25/ec/aad2613c1910dce907480e0c3aa306905830f25df2e54ccc9dea450cb5aa/multidict-6.7.0-cp312-cp312-win_arm64.whl", hash = "sha256:490dab541a6a642ce1a9d61a4781656b346a55c13038f0b1244653828e3a83ec", size = 43226, upload-time = "2025-10-06T14:49:52.304Z" },
- { url = "https://files.pythonhosted.org/packages/d2/86/33272a544eeb36d66e4d9a920602d1a2f57d4ebea4ef3cdfe5a912574c95/multidict-6.7.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:bee7c0588aa0076ce77c0ea5d19a68d76ad81fcd9fe8501003b9a24f9d4000f6", size = 76135, upload-time = "2025-10-06T14:49:54.26Z" },
- { url = "https://files.pythonhosted.org/packages/91/1c/eb97db117a1ebe46d457a3d235a7b9d2e6dcab174f42d1b67663dd9e5371/multidict-6.7.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7ef6b61cad77091056ce0e7ce69814ef72afacb150b7ac6a3e9470def2198159", size = 45117, upload-time = "2025-10-06T14:49:55.82Z" },
- { url = "https://files.pythonhosted.org/packages/f1/d8/6c3442322e41fb1dd4de8bd67bfd11cd72352ac131f6368315617de752f1/multidict-6.7.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:9c0359b1ec12b1d6849c59f9d319610b7f20ef990a6d454ab151aa0e3b9f78ca", size = 43472, upload-time = "2025-10-06T14:49:57.048Z" },
- { url = "https://files.pythonhosted.org/packages/75/3f/e2639e80325af0b6c6febdf8e57cc07043ff15f57fa1ef808f4ccb5ac4cd/multidict-6.7.0-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:cd240939f71c64bd658f186330603aac1a9a81bf6273f523fca63673cb7378a8", size = 249342, upload-time = "2025-10-06T14:49:58.368Z" },
- { url = "https://files.pythonhosted.org/packages/5d/cc/84e0585f805cbeaa9cbdaa95f9a3d6aed745b9d25700623ac89a6ecff400/multidict-6.7.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a60a4d75718a5efa473ebd5ab685786ba0c67b8381f781d1be14da49f1a2dc60", size = 257082, upload-time = "2025-10-06T14:49:59.89Z" },
- { url = "https://files.pythonhosted.org/packages/b0/9c/ac851c107c92289acbbf5cfb485694084690c1b17e555f44952c26ddc5bd/multidict-6.7.0-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:53a42d364f323275126aff81fb67c5ca1b7a04fda0546245730a55c8c5f24bc4", size = 240704, upload-time = "2025-10-06T14:50:01.485Z" },
- { url = "https://files.pythonhosted.org/packages/50/cc/5f93e99427248c09da95b62d64b25748a5f5c98c7c2ab09825a1d6af0e15/multidict-6.7.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:3b29b980d0ddbecb736735ee5bef69bb2ddca56eff603c86f3f29a1128299b4f", size = 266355, upload-time = "2025-10-06T14:50:02.955Z" },
- { url = "https://files.pythonhosted.org/packages/ec/0c/2ec1d883ceb79c6f7f6d7ad90c919c898f5d1c6ea96d322751420211e072/multidict-6.7.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f8a93b1c0ed2d04b97a5e9336fd2d33371b9a6e29ab7dd6503d63407c20ffbaf", size = 267259, upload-time = "2025-10-06T14:50:04.446Z" },
- { url = "https://files.pythonhosted.org/packages/c6/2d/f0b184fa88d6630aa267680bdb8623fb69cb0d024b8c6f0d23f9a0f406d3/multidict-6.7.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9ff96e8815eecacc6645da76c413eb3b3d34cfca256c70b16b286a687d013c32", size = 254903, upload-time = "2025-10-06T14:50:05.98Z" },
- { url = "https://files.pythonhosted.org/packages/06/c9/11ea263ad0df7dfabcad404feb3c0dd40b131bc7f232d5537f2fb1356951/multidict-6.7.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:7516c579652f6a6be0e266aec0acd0db80829ca305c3d771ed898538804c2036", size = 252365, upload-time = "2025-10-06T14:50:07.511Z" },
- { url = "https://files.pythonhosted.org/packages/41/88/d714b86ee2c17d6e09850c70c9d310abac3d808ab49dfa16b43aba9d53fd/multidict-6.7.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:040f393368e63fb0f3330e70c26bfd336656bed925e5cbe17c9da839a6ab13ec", size = 250062, upload-time = "2025-10-06T14:50:09.074Z" },
- { url = "https://files.pythonhosted.org/packages/15/fe/ad407bb9e818c2b31383f6131ca19ea7e35ce93cf1310fce69f12e89de75/multidict-6.7.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:b3bc26a951007b1057a1c543af845f1c7e3e71cc240ed1ace7bf4484aa99196e", size = 249683, upload-time = "2025-10-06T14:50:10.714Z" },
- { url = "https://files.pythonhosted.org/packages/8c/a4/a89abdb0229e533fb925e7c6e5c40201c2873efebc9abaf14046a4536ee6/multidict-6.7.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:7b022717c748dd1992a83e219587aabe45980d88969f01b316e78683e6285f64", size = 261254, upload-time = "2025-10-06T14:50:12.28Z" },
- { url = "https://files.pythonhosted.org/packages/8d/aa/0e2b27bd88b40a4fb8dc53dd74eecac70edaa4c1dd0707eb2164da3675b3/multidict-6.7.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:9600082733859f00d79dee64effc7aef1beb26adb297416a4ad2116fd61374bd", size = 257967, upload-time = "2025-10-06T14:50:14.16Z" },
- { url = "https://files.pythonhosted.org/packages/d0/8e/0c67b7120d5d5f6d874ed85a085f9dc770a7f9d8813e80f44a9fec820bb7/multidict-6.7.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:94218fcec4d72bc61df51c198d098ce2b378e0ccbac41ddbed5ef44092913288", size = 250085, upload-time = "2025-10-06T14:50:15.639Z" },
- { url = "https://files.pythonhosted.org/packages/ba/55/b73e1d624ea4b8fd4dd07a3bb70f6e4c7c6c5d9d640a41c6ffe5cdbd2a55/multidict-6.7.0-cp313-cp313-win32.whl", hash = "sha256:a37bd74c3fa9d00be2d7b8eca074dc56bd8077ddd2917a839bd989612671ed17", size = 41713, upload-time = "2025-10-06T14:50:17.066Z" },
- { url = "https://files.pythonhosted.org/packages/32/31/75c59e7d3b4205075b4c183fa4ca398a2daf2303ddf616b04ae6ef55cffe/multidict-6.7.0-cp313-cp313-win_amd64.whl", hash = "sha256:30d193c6cc6d559db42b6bcec8a5d395d34d60c9877a0b71ecd7c204fcf15390", size = 45915, upload-time = "2025-10-06T14:50:18.264Z" },
- { url = "https://files.pythonhosted.org/packages/31/2a/8987831e811f1184c22bc2e45844934385363ee61c0a2dcfa8f71b87e608/multidict-6.7.0-cp313-cp313-win_arm64.whl", hash = "sha256:ea3334cabe4d41b7ccd01e4d349828678794edbc2d3ae97fc162a3312095092e", size = 43077, upload-time = "2025-10-06T14:50:19.853Z" },
- { url = "https://files.pythonhosted.org/packages/e8/68/7b3a5170a382a340147337b300b9eb25a9ddb573bcdfff19c0fa3f31ffba/multidict-6.7.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:ad9ce259f50abd98a1ca0aa6e490b58c316a0fce0617f609723e40804add2c00", size = 83114, upload-time = "2025-10-06T14:50:21.223Z" },
- { url = "https://files.pythonhosted.org/packages/55/5c/3fa2d07c84df4e302060f555bbf539310980362236ad49f50eeb0a1c1eb9/multidict-6.7.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:07f5594ac6d084cbb5de2df218d78baf55ef150b91f0ff8a21cc7a2e3a5a58eb", size = 48442, upload-time = "2025-10-06T14:50:22.871Z" },
- { url = "https://files.pythonhosted.org/packages/fc/56/67212d33239797f9bd91962bb899d72bb0f4c35a8652dcdb8ed049bef878/multidict-6.7.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:0591b48acf279821a579282444814a2d8d0af624ae0bc600aa4d1b920b6e924b", size = 46885, upload-time = "2025-10-06T14:50:24.258Z" },
- { url = "https://files.pythonhosted.org/packages/46/d1/908f896224290350721597a61a69cd19b89ad8ee0ae1f38b3f5cd12ea2ac/multidict-6.7.0-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:749a72584761531d2b9467cfbdfd29487ee21124c304c4b6cb760d8777b27f9c", size = 242588, upload-time = "2025-10-06T14:50:25.716Z" },
- { url = "https://files.pythonhosted.org/packages/ab/67/8604288bbd68680eee0ab568fdcb56171d8b23a01bcd5cb0c8fedf6e5d99/multidict-6.7.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6b4c3d199f953acd5b446bf7c0de1fe25d94e09e79086f8dc2f48a11a129cdf1", size = 249966, upload-time = "2025-10-06T14:50:28.192Z" },
- { url = "https://files.pythonhosted.org/packages/20/33/9228d76339f1ba51e3efef7da3ebd91964d3006217aae13211653193c3ff/multidict-6.7.0-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:9fb0211dfc3b51efea2f349ec92c114d7754dd62c01f81c3e32b765b70c45c9b", size = 228618, upload-time = "2025-10-06T14:50:29.82Z" },
- { url = "https://files.pythonhosted.org/packages/f8/2d/25d9b566d10cab1c42b3b9e5b11ef79c9111eaf4463b8c257a3bd89e0ead/multidict-6.7.0-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a027ec240fe73a8d6281872690b988eed307cd7d91b23998ff35ff577ca688b5", size = 257539, upload-time = "2025-10-06T14:50:31.731Z" },
- { url = "https://files.pythonhosted.org/packages/b6/b1/8d1a965e6637fc33de3c0d8f414485c2b7e4af00f42cab3d84e7b955c222/multidict-6.7.0-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d1d964afecdf3a8288789df2f5751dc0a8261138c3768d9af117ed384e538fad", size = 256345, upload-time = "2025-10-06T14:50:33.26Z" },
- { url = "https://files.pythonhosted.org/packages/ba/0c/06b5a8adbdeedada6f4fb8d8f193d44a347223b11939b42953eeb6530b6b/multidict-6.7.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:caf53b15b1b7df9fbd0709aa01409000a2b4dd03a5f6f5cc548183c7c8f8b63c", size = 247934, upload-time = "2025-10-06T14:50:34.808Z" },
- { url = "https://files.pythonhosted.org/packages/8f/31/b2491b5fe167ca044c6eb4b8f2c9f3b8a00b24c432c365358eadac5d7625/multidict-6.7.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:654030da3197d927f05a536a66186070e98765aa5142794c9904555d3a9d8fb5", size = 245243, upload-time = "2025-10-06T14:50:36.436Z" },
- { url = "https://files.pythonhosted.org/packages/61/1a/982913957cb90406c8c94f53001abd9eafc271cb3e70ff6371590bec478e/multidict-6.7.0-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:2090d3718829d1e484706a2f525e50c892237b2bf9b17a79b059cb98cddc2f10", size = 235878, upload-time = "2025-10-06T14:50:37.953Z" },
- { url = "https://files.pythonhosted.org/packages/be/c0/21435d804c1a1cf7a2608593f4d19bca5bcbd7a81a70b253fdd1c12af9c0/multidict-6.7.0-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:2d2cfeec3f6f45651b3d408c4acec0ebf3daa9bc8a112a084206f5db5d05b754", size = 243452, upload-time = "2025-10-06T14:50:39.574Z" },
- { url = "https://files.pythonhosted.org/packages/54/0a/4349d540d4a883863191be6eb9a928846d4ec0ea007d3dcd36323bb058ac/multidict-6.7.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:4ef089f985b8c194d341eb2c24ae6e7408c9a0e2e5658699c92f497437d88c3c", size = 252312, upload-time = "2025-10-06T14:50:41.612Z" },
- { url = "https://files.pythonhosted.org/packages/26/64/d5416038dbda1488daf16b676e4dbfd9674dde10a0cc8f4fc2b502d8125d/multidict-6.7.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:e93a0617cd16998784bf4414c7e40f17a35d2350e5c6f0bd900d3a8e02bd3762", size = 246935, upload-time = "2025-10-06T14:50:43.972Z" },
- { url = "https://files.pythonhosted.org/packages/9f/8c/8290c50d14e49f35e0bd4abc25e1bc7711149ca9588ab7d04f886cdf03d9/multidict-6.7.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:f0feece2ef8ebc42ed9e2e8c78fc4aa3cf455733b507c09ef7406364c94376c6", size = 243385, upload-time = "2025-10-06T14:50:45.648Z" },
- { url = "https://files.pythonhosted.org/packages/ef/a0/f83ae75e42d694b3fbad3e047670e511c138be747bc713cf1b10d5096416/multidict-6.7.0-cp313-cp313t-win32.whl", hash = "sha256:19a1d55338ec1be74ef62440ca9e04a2f001a04d0cc49a4983dc320ff0f3212d", size = 47777, upload-time = "2025-10-06T14:50:47.154Z" },
- { url = "https://files.pythonhosted.org/packages/dc/80/9b174a92814a3830b7357307a792300f42c9e94664b01dee8e457551fa66/multidict-6.7.0-cp313-cp313t-win_amd64.whl", hash = "sha256:3da4fb467498df97e986af166b12d01f05d2e04f978a9c1c680ea1988e0bc4b6", size = 53104, upload-time = "2025-10-06T14:50:48.851Z" },
- { url = "https://files.pythonhosted.org/packages/cc/28/04baeaf0428d95bb7a7bea0e691ba2f31394338ba424fb0679a9ed0f4c09/multidict-6.7.0-cp313-cp313t-win_arm64.whl", hash = "sha256:b4121773c49a0776461f4a904cdf6264c88e42218aaa8407e803ca8025872792", size = 45503, upload-time = "2025-10-06T14:50:50.16Z" },
- { url = "https://files.pythonhosted.org/packages/e2/b1/3da6934455dd4b261d4c72f897e3a5728eba81db59959f3a639245891baa/multidict-6.7.0-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:3bab1e4aff7adaa34410f93b1f8e57c4b36b9af0426a76003f441ee1d3c7e842", size = 75128, upload-time = "2025-10-06T14:50:51.92Z" },
- { url = "https://files.pythonhosted.org/packages/14/2c/f069cab5b51d175a1a2cb4ccdf7a2c2dabd58aa5bd933fa036a8d15e2404/multidict-6.7.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:b8512bac933afc3e45fb2b18da8e59b78d4f408399a960339598374d4ae3b56b", size = 44410, upload-time = "2025-10-06T14:50:53.275Z" },
- { url = "https://files.pythonhosted.org/packages/42/e2/64bb41266427af6642b6b128e8774ed84c11b80a90702c13ac0a86bb10cc/multidict-6.7.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:79dcf9e477bc65414ebfea98ffd013cb39552b5ecd62908752e0e413d6d06e38", size = 43205, upload-time = "2025-10-06T14:50:54.911Z" },
- { url = "https://files.pythonhosted.org/packages/02/68/6b086fef8a3f1a8541b9236c594f0c9245617c29841f2e0395d979485cde/multidict-6.7.0-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:31bae522710064b5cbeddaf2e9f32b1abab70ac6ac91d42572502299e9953128", size = 245084, upload-time = "2025-10-06T14:50:56.369Z" },
- { url = "https://files.pythonhosted.org/packages/15/ee/f524093232007cd7a75c1d132df70f235cfd590a7c9eaccd7ff422ef4ae8/multidict-6.7.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4a0df7ff02397bb63e2fd22af2c87dfa39e8c7f12947bc524dbdc528282c7e34", size = 252667, upload-time = "2025-10-06T14:50:57.991Z" },
- { url = "https://files.pythonhosted.org/packages/02/a5/eeb3f43ab45878f1895118c3ef157a480db58ede3f248e29b5354139c2c9/multidict-6.7.0-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:7a0222514e8e4c514660e182d5156a415c13ef0aabbd71682fc714e327b95e99", size = 233590, upload-time = "2025-10-06T14:50:59.589Z" },
- { url = "https://files.pythonhosted.org/packages/6a/1e/76d02f8270b97269d7e3dbd45644b1785bda457b474315f8cf999525a193/multidict-6.7.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2397ab4daaf2698eb51a76721e98db21ce4f52339e535725de03ea962b5a3202", size = 264112, upload-time = "2025-10-06T14:51:01.183Z" },
- { url = "https://files.pythonhosted.org/packages/76/0b/c28a70ecb58963847c2a8efe334904cd254812b10e535aefb3bcce513918/multidict-6.7.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:8891681594162635948a636c9fe0ff21746aeb3dd5463f6e25d9bea3a8a39ca1", size = 261194, upload-time = "2025-10-06T14:51:02.794Z" },
- { url = "https://files.pythonhosted.org/packages/b4/63/2ab26e4209773223159b83aa32721b4021ffb08102f8ac7d689c943fded1/multidict-6.7.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:18706cc31dbf402a7945916dd5cddf160251b6dab8a2c5f3d6d5a55949f676b3", size = 248510, upload-time = "2025-10-06T14:51:04.724Z" },
- { url = "https://files.pythonhosted.org/packages/93/cd/06c1fa8282af1d1c46fd55c10a7930af652afdce43999501d4d68664170c/multidict-6.7.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:f844a1bbf1d207dd311a56f383f7eda2d0e134921d45751842d8235e7778965d", size = 248395, upload-time = "2025-10-06T14:51:06.306Z" },
- { url = "https://files.pythonhosted.org/packages/99/ac/82cb419dd6b04ccf9e7e61befc00c77614fc8134362488b553402ecd55ce/multidict-6.7.0-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:d4393e3581e84e5645506923816b9cc81f5609a778c7e7534054091acc64d1c6", size = 239520, upload-time = "2025-10-06T14:51:08.091Z" },
- { url = "https://files.pythonhosted.org/packages/fa/f3/a0f9bf09493421bd8716a362e0cd1d244f5a6550f5beffdd6b47e885b331/multidict-6.7.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:fbd18dc82d7bf274b37aa48d664534330af744e03bccf696d6f4c6042e7d19e7", size = 245479, upload-time = "2025-10-06T14:51:10.365Z" },
- { url = "https://files.pythonhosted.org/packages/8d/01/476d38fc73a212843f43c852b0eee266b6971f0e28329c2184a8df90c376/multidict-6.7.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:b6234e14f9314731ec45c42fc4554b88133ad53a09092cc48a88e771c125dadb", size = 258903, upload-time = "2025-10-06T14:51:12.466Z" },
- { url = "https://files.pythonhosted.org/packages/49/6d/23faeb0868adba613b817d0e69c5f15531b24d462af8012c4f6de4fa8dc3/multidict-6.7.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:08d4379f9744d8f78d98c8673c06e202ffa88296f009c71bbafe8a6bf847d01f", size = 252333, upload-time = "2025-10-06T14:51:14.48Z" },
- { url = "https://files.pythonhosted.org/packages/1e/cc/48d02ac22b30fa247f7dad82866e4b1015431092f4ba6ebc7e77596e0b18/multidict-6.7.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:9fe04da3f79387f450fd0061d4dd2e45a72749d31bf634aecc9e27f24fdc4b3f", size = 243411, upload-time = "2025-10-06T14:51:16.072Z" },
- { url = "https://files.pythonhosted.org/packages/4a/03/29a8bf5a18abf1fe34535c88adbdfa88c9fb869b5a3b120692c64abe8284/multidict-6.7.0-cp314-cp314-win32.whl", hash = "sha256:fbafe31d191dfa7c4c51f7a6149c9fb7e914dcf9ffead27dcfd9f1ae382b3885", size = 40940, upload-time = "2025-10-06T14:51:17.544Z" },
- { url = "https://files.pythonhosted.org/packages/82/16/7ed27b680791b939de138f906d5cf2b4657b0d45ca6f5dd6236fdddafb1a/multidict-6.7.0-cp314-cp314-win_amd64.whl", hash = "sha256:2f67396ec0310764b9222a1728ced1ab638f61aadc6226f17a71dd9324f9a99c", size = 45087, upload-time = "2025-10-06T14:51:18.875Z" },
- { url = "https://files.pythonhosted.org/packages/cd/3c/e3e62eb35a1950292fe39315d3c89941e30a9d07d5d2df42965ab041da43/multidict-6.7.0-cp314-cp314-win_arm64.whl", hash = "sha256:ba672b26069957ee369cfa7fc180dde1fc6f176eaf1e6beaf61fbebbd3d9c000", size = 42368, upload-time = "2025-10-06T14:51:20.225Z" },
- { url = "https://files.pythonhosted.org/packages/8b/40/cd499bd0dbc5f1136726db3153042a735fffd0d77268e2ee20d5f33c010f/multidict-6.7.0-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:c1dcc7524066fa918c6a27d61444d4ee7900ec635779058571f70d042d86ed63", size = 82326, upload-time = "2025-10-06T14:51:21.588Z" },
- { url = "https://files.pythonhosted.org/packages/13/8a/18e031eca251c8df76daf0288e6790561806e439f5ce99a170b4af30676b/multidict-6.7.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:27e0b36c2d388dc7b6ced3406671b401e84ad7eb0656b8f3a2f46ed0ce483718", size = 48065, upload-time = "2025-10-06T14:51:22.93Z" },
- { url = "https://files.pythonhosted.org/packages/40/71/5e6701277470a87d234e433fb0a3a7deaf3bcd92566e421e7ae9776319de/multidict-6.7.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:2a7baa46a22e77f0988e3b23d4ede5513ebec1929e34ee9495be535662c0dfe2", size = 46475, upload-time = "2025-10-06T14:51:24.352Z" },
- { url = "https://files.pythonhosted.org/packages/fe/6a/bab00cbab6d9cfb57afe1663318f72ec28289ea03fd4e8236bb78429893a/multidict-6.7.0-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:7bf77f54997a9166a2f5675d1201520586439424c2511723a7312bdb4bcc034e", size = 239324, upload-time = "2025-10-06T14:51:25.822Z" },
- { url = "https://files.pythonhosted.org/packages/2a/5f/8de95f629fc22a7769ade8b41028e3e5a822c1f8904f618d175945a81ad3/multidict-6.7.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e011555abada53f1578d63389610ac8a5400fc70ce71156b0aa30d326f1a5064", size = 246877, upload-time = "2025-10-06T14:51:27.604Z" },
- { url = "https://files.pythonhosted.org/packages/23/b4/38881a960458f25b89e9f4a4fdcb02ac101cfa710190db6e5528841e67de/multidict-6.7.0-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:28b37063541b897fd6a318007373930a75ca6d6ac7c940dbe14731ffdd8d498e", size = 225824, upload-time = "2025-10-06T14:51:29.664Z" },
- { url = "https://files.pythonhosted.org/packages/1e/39/6566210c83f8a261575f18e7144736059f0c460b362e96e9cf797a24b8e7/multidict-6.7.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:05047ada7a2fde2631a0ed706f1fd68b169a681dfe5e4cf0f8e4cb6618bbc2cd", size = 253558, upload-time = "2025-10-06T14:51:31.684Z" },
- { url = "https://files.pythonhosted.org/packages/00/a3/67f18315100f64c269f46e6c0319fa87ba68f0f64f2b8e7fd7c72b913a0b/multidict-6.7.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:716133f7d1d946a4e1b91b1756b23c088881e70ff180c24e864c26192ad7534a", size = 252339, upload-time = "2025-10-06T14:51:33.699Z" },
- { url = "https://files.pythonhosted.org/packages/c8/2a/1cb77266afee2458d82f50da41beba02159b1d6b1f7973afc9a1cad1499b/multidict-6.7.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d1bed1b467ef657f2a0ae62844a607909ef1c6889562de5e1d505f74457d0b96", size = 244895, upload-time = "2025-10-06T14:51:36.189Z" },
- { url = "https://files.pythonhosted.org/packages/dd/72/09fa7dd487f119b2eb9524946ddd36e2067c08510576d43ff68469563b3b/multidict-6.7.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:ca43bdfa5d37bd6aee89d85e1d0831fb86e25541be7e9d376ead1b28974f8e5e", size = 241862, upload-time = "2025-10-06T14:51:41.291Z" },
- { url = "https://files.pythonhosted.org/packages/65/92/bc1f8bd0853d8669300f732c801974dfc3702c3eeadae2f60cef54dc69d7/multidict-6.7.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:44b546bd3eb645fd26fb949e43c02a25a2e632e2ca21a35e2e132c8105dc8599", size = 232376, upload-time = "2025-10-06T14:51:43.55Z" },
- { url = "https://files.pythonhosted.org/packages/09/86/ac39399e5cb9d0c2ac8ef6e10a768e4d3bc933ac808d49c41f9dc23337eb/multidict-6.7.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:a6ef16328011d3f468e7ebc326f24c1445f001ca1dec335b2f8e66bed3006394", size = 240272, upload-time = "2025-10-06T14:51:45.265Z" },
- { url = "https://files.pythonhosted.org/packages/3d/b6/fed5ac6b8563ec72df6cb1ea8dac6d17f0a4a1f65045f66b6d3bf1497c02/multidict-6.7.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:5aa873cbc8e593d361ae65c68f85faadd755c3295ea2c12040ee146802f23b38", size = 248774, upload-time = "2025-10-06T14:51:46.836Z" },
- { url = "https://files.pythonhosted.org/packages/6b/8d/b954d8c0dc132b68f760aefd45870978deec6818897389dace00fcde32ff/multidict-6.7.0-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:3d7b6ccce016e29df4b7ca819659f516f0bc7a4b3efa3bb2012ba06431b044f9", size = 242731, upload-time = "2025-10-06T14:51:48.541Z" },
- { url = "https://files.pythonhosted.org/packages/16/9d/a2dac7009125d3540c2f54e194829ea18ac53716c61b655d8ed300120b0f/multidict-6.7.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:171b73bd4ee683d307599b66793ac80981b06f069b62eea1c9e29c9241aa66b0", size = 240193, upload-time = "2025-10-06T14:51:50.355Z" },
- { url = "https://files.pythonhosted.org/packages/39/ca/c05f144128ea232ae2178b008d5011d4e2cea86e4ee8c85c2631b1b94802/multidict-6.7.0-cp314-cp314t-win32.whl", hash = "sha256:b2d7f80c4e1fd010b07cb26820aae86b7e73b681ee4889684fb8d2d4537aab13", size = 48023, upload-time = "2025-10-06T14:51:51.883Z" },
- { url = "https://files.pythonhosted.org/packages/ba/8f/0a60e501584145588be1af5cc829265701ba3c35a64aec8e07cbb71d39bb/multidict-6.7.0-cp314-cp314t-win_amd64.whl", hash = "sha256:09929cab6fcb68122776d575e03c6cc64ee0b8fca48d17e135474b042ce515cd", size = 53507, upload-time = "2025-10-06T14:51:53.672Z" },
- { url = "https://files.pythonhosted.org/packages/7f/ae/3148b988a9c6239903e786eac19c889fab607c31d6efa7fb2147e5680f23/multidict-6.7.0-cp314-cp314t-win_arm64.whl", hash = "sha256:cc41db090ed742f32bd2d2c721861725e6109681eddf835d0a82bd3a5c382827", size = 44804, upload-time = "2025-10-06T14:51:55.415Z" },
- { url = "https://files.pythonhosted.org/packages/b7/da/7d22601b625e241d4f23ef1ebff8acfc60da633c9e7e7922e24d10f592b3/multidict-6.7.0-py3-none-any.whl", hash = "sha256:394fc5c42a333c9ffc3e421a4c85e08580d990e08b99f6bf35b4132114c5dcb3", size = 12317, upload-time = "2025-10-06T14:52:29.272Z" },
+version = "6.7.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1a/c2/c2d94cbe6ac1753f3fc980da97b3d930efe1da3af3c9f5125354436c073d/multidict-6.7.1.tar.gz", hash = "sha256:ec6652a1bee61c53a3e5776b6049172c53b6aaba34f18c9ad04f82712bac623d", size = 102010, upload-time = "2026-01-26T02:46:45.979Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8d/9c/f20e0e2cf80e4b2e4b1c365bf5fe104ee633c751a724246262db8f1a0b13/multidict-6.7.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:a90f75c956e32891a4eda3639ce6dd86e87105271f43d43442a3aedf3cddf172", size = 76893, upload-time = "2026-01-26T02:43:52.754Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/cf/18ef143a81610136d3da8193da9d80bfe1cb548a1e2d1c775f26b23d024a/multidict-6.7.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3fccb473e87eaa1382689053e4a4618e7ba7b9b9b8d6adf2027ee474597128cd", size = 45456, upload-time = "2026-01-26T02:43:53.893Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/65/1caac9d4cd32e8433908683446eebc953e82d22b03d10d41a5f0fefe991b/multidict-6.7.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b0fa96985700739c4c7853a43c0b3e169360d6855780021bfc6d0f1ce7c123e7", size = 43872, upload-time = "2026-01-26T02:43:55.041Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/3b/d6bd75dc4f3ff7c73766e04e705b00ed6dbbaccf670d9e05a12b006f5a21/multidict-6.7.1-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:cb2a55f408c3043e42b40cc8eecd575afa27b7e0b956dfb190de0f8499a57a53", size = 251018, upload-time = "2026-01-26T02:43:56.198Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/80/c959c5933adedb9ac15152e4067c702a808ea183a8b64cf8f31af8ad3155/multidict-6.7.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eb0ce7b2a32d09892b3dd6cc44877a0d02a33241fafca5f25c8b6b62374f8b75", size = 258883, upload-time = "2026-01-26T02:43:57.499Z" },
+ { url = "https://files.pythonhosted.org/packages/86/85/7ed40adafea3d4f1c8b916e3b5cc3a8e07dfcdcb9cd72800f4ed3ca1b387/multidict-6.7.1-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c3a32d23520ee37bf327d1e1a656fec76a2edd5c038bf43eddfa0572ec49c60b", size = 242413, upload-time = "2026-01-26T02:43:58.755Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/57/b8565ff533e48595503c785f8361ff9a4fde4d67de25c207cd0ba3befd03/multidict-6.7.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9c90fed18bffc0189ba814749fdcc102b536e83a9f738a9003e569acd540a733", size = 268404, upload-time = "2026-01-26T02:44:00.216Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/50/9810c5c29350f7258180dfdcb2e52783a0632862eb334c4896ac717cebcb/multidict-6.7.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:da62917e6076f512daccfbbde27f46fed1c98fee202f0559adec8ee0de67f71a", size = 269456, upload-time = "2026-01-26T02:44:02.202Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/8d/5e5be3ced1d12966fefb5c4ea3b2a5b480afcea36406559442c6e31d4a48/multidict-6.7.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bfde23ef6ed9db7eaee6c37dcec08524cb43903c60b285b172b6c094711b3961", size = 256322, upload-time = "2026-01-26T02:44:03.56Z" },
+ { url = "https://files.pythonhosted.org/packages/31/6e/d8a26d81ac166a5592782d208dd90dfdc0a7a218adaa52b45a672b46c122/multidict-6.7.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3758692429e4e32f1ba0df23219cd0b4fc0a52f476726fff9337d1a57676a582", size = 253955, upload-time = "2026-01-26T02:44:04.845Z" },
+ { url = "https://files.pythonhosted.org/packages/59/4c/7c672c8aad41534ba619bcd4ade7a0dc87ed6b8b5c06149b85d3dd03f0cd/multidict-6.7.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:398c1478926eca669f2fd6a5856b6de9c0acf23a2cb59a14c0ba5844fa38077e", size = 251254, upload-time = "2026-01-26T02:44:06.133Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/bd/84c24de512cbafbdbc39439f74e967f19570ce7924e3007174a29c348916/multidict-6.7.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:c102791b1c4f3ab36ce4101154549105a53dc828f016356b3e3bcae2e3a039d3", size = 252059, upload-time = "2026-01-26T02:44:07.518Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/ba/f5449385510825b73d01c2d4087bf6d2fccc20a2d42ac34df93191d3dd03/multidict-6.7.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:a088b62bd733e2ad12c50dad01b7d0166c30287c166e137433d3b410add807a6", size = 263588, upload-time = "2026-01-26T02:44:09.382Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/11/afc7c677f68f75c84a69fe37184f0f82fce13ce4b92f49f3db280b7e92b3/multidict-6.7.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:3d51ff4785d58d3f6c91bdbffcb5e1f7ddfda557727043aa20d20ec4f65e324a", size = 259642, upload-time = "2026-01-26T02:44:10.73Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/17/ebb9644da78c4ab36403739e0e6e0e30ebb135b9caf3440825001a0bddcb/multidict-6.7.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fc5907494fccf3e7d3f94f95c91d6336b092b5fc83811720fae5e2765890dfba", size = 251377, upload-time = "2026-01-26T02:44:12.042Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/a4/840f5b97339e27846c46307f2530a2805d9d537d8b8bd416af031cad7fa0/multidict-6.7.1-cp312-cp312-win32.whl", hash = "sha256:28ca5ce2fd9716631133d0e9a9b9a745ad7f60bac2bccafb56aa380fc0b6c511", size = 41887, upload-time = "2026-01-26T02:44:14.245Z" },
+ { url = "https://files.pythonhosted.org/packages/80/31/0b2517913687895f5904325c2069d6a3b78f66cc641a86a2baf75a05dcbb/multidict-6.7.1-cp312-cp312-win_amd64.whl", hash = "sha256:fcee94dfbd638784645b066074b338bc9cc155d4b4bffa4adce1615c5a426c19", size = 46053, upload-time = "2026-01-26T02:44:15.371Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/5b/aba28e4ee4006ae4c7df8d327d31025d760ffa992ea23812a601d226e682/multidict-6.7.1-cp312-cp312-win_arm64.whl", hash = "sha256:ba0a9fb644d0c1a2194cf7ffb043bd852cea63a57f66fbd33959f7dae18517bf", size = 43307, upload-time = "2026-01-26T02:44:16.852Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/22/929c141d6c0dba87d3e1d38fbdf1ba8baba86b7776469f2bc2d3227a1e67/multidict-6.7.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:2b41f5fed0ed563624f1c17630cb9941cf2309d4df00e494b551b5f3e3d67a23", size = 76174, upload-time = "2026-01-26T02:44:18.509Z" },
+ { url = "https://files.pythonhosted.org/packages/c7/75/bc704ae15fee974f8fccd871305e254754167dce5f9e42d88a2def741a1d/multidict-6.7.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:84e61e3af5463c19b67ced91f6c634effb89ef8bfc5ca0267f954451ed4bb6a2", size = 45116, upload-time = "2026-01-26T02:44:19.745Z" },
+ { url = "https://files.pythonhosted.org/packages/79/76/55cd7186f498ed080a18440c9013011eb548f77ae1b297206d030eb1180a/multidict-6.7.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:935434b9853c7c112eee7ac891bc4cb86455aa631269ae35442cb316790c1445", size = 43524, upload-time = "2026-01-26T02:44:21.571Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/3c/414842ef8d5a1628d68edee29ba0e5bcf235dbfb3ccd3ea303a7fe8c72ff/multidict-6.7.1-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:432feb25a1cb67fe82a9680b4d65fb542e4635cb3166cd9c01560651ad60f177", size = 249368, upload-time = "2026-01-26T02:44:22.803Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/32/befed7f74c458b4a525e60519fe8d87eef72bb1e99924fa2b0f9d97a221e/multidict-6.7.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e82d14e3c948952a1a85503817e038cba5905a3352de76b9a465075d072fba23", size = 256952, upload-time = "2026-01-26T02:44:24.306Z" },
+ { url = "https://files.pythonhosted.org/packages/03/d6/c878a44ba877f366630c860fdf74bfb203c33778f12b6ac274936853c451/multidict-6.7.1-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:4cfb48c6ea66c83bcaaf7e4dfa7ec1b6bbcf751b7db85a328902796dfde4c060", size = 240317, upload-time = "2026-01-26T02:44:25.772Z" },
+ { url = "https://files.pythonhosted.org/packages/68/49/57421b4d7ad2e9e60e25922b08ceb37e077b90444bde6ead629095327a6f/multidict-6.7.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1d540e51b7e8e170174555edecddbd5538105443754539193e3e1061864d444d", size = 267132, upload-time = "2026-01-26T02:44:27.648Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/fe/ec0edd52ddbcea2a2e89e174f0206444a61440b40f39704e64dc807a70bd/multidict-6.7.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:273d23f4b40f3dce4d6c8a821c741a86dec62cded82e1175ba3d99be128147ed", size = 268140, upload-time = "2026-01-26T02:44:29.588Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/73/6e1b01cbeb458807aa0831742232dbdd1fa92bfa33f52a3f176b4ff3dc11/multidict-6.7.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d624335fd4fa1c08a53f8b4be7676ebde19cd092b3895c421045ca87895b429", size = 254277, upload-time = "2026-01-26T02:44:30.902Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/b2/5fb8c124d7561a4974c342bc8c778b471ebbeb3cc17df696f034a7e9afe7/multidict-6.7.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:12fad252f8b267cc75b66e8fc51b3079604e8d43a75428ffe193cd9e2195dfd6", size = 252291, upload-time = "2026-01-26T02:44:32.31Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/96/51d4e4e06bcce92577fcd488e22600bd38e4fd59c20cb49434d054903bd2/multidict-6.7.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:03ede2a6ffbe8ef936b92cb4529f27f42be7f56afcdab5ab739cd5f27fb1cbf9", size = 250156, upload-time = "2026-01-26T02:44:33.734Z" },
+ { url = "https://files.pythonhosted.org/packages/db/6b/420e173eec5fba721a50e2a9f89eda89d9c98fded1124f8d5c675f7a0c0f/multidict-6.7.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:90efbcf47dbe33dcf643a1e400d67d59abeac5db07dc3f27d6bdeae497a2198c", size = 249742, upload-time = "2026-01-26T02:44:35.222Z" },
+ { url = "https://files.pythonhosted.org/packages/44/a3/ec5b5bd98f306bc2aa297b8c6f11a46714a56b1e6ef5ebda50a4f5d7c5fb/multidict-6.7.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:5c4b9bfc148f5a91be9244d6264c53035c8a0dcd2f51f1c3c6e30e30ebaa1c84", size = 262221, upload-time = "2026-01-26T02:44:36.604Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/f7/e8c0d0da0cd1e28d10e624604e1a36bcc3353aaebdfdc3a43c72bc683a12/multidict-6.7.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:401c5a650f3add2472d1d288c26deebc540f99e2fb83e9525007a74cd2116f1d", size = 258664, upload-time = "2026-01-26T02:44:38.008Z" },
+ { url = "https://files.pythonhosted.org/packages/52/da/151a44e8016dd33feed44f730bd856a66257c1ee7aed4f44b649fb7edeb3/multidict-6.7.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:97891f3b1b3ffbded884e2916cacf3c6fc87b66bb0dde46f7357404750559f33", size = 249490, upload-time = "2026-01-26T02:44:39.386Z" },
+ { url = "https://files.pythonhosted.org/packages/87/af/a3b86bf9630b732897f6fc3f4c4714b90aa4361983ccbdcd6c0339b21b0c/multidict-6.7.1-cp313-cp313-win32.whl", hash = "sha256:e1c5988359516095535c4301af38d8a8838534158f649c05dd1050222321bcb3", size = 41695, upload-time = "2026-01-26T02:44:41.318Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/35/e994121b0e90e46134673422dd564623f93304614f5d11886b1b3e06f503/multidict-6.7.1-cp313-cp313-win_amd64.whl", hash = "sha256:960c83bf01a95b12b08fd54324a4eb1d5b52c88932b5cba5d6e712bb3ed12eb5", size = 45884, upload-time = "2026-01-26T02:44:42.488Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/61/42d3e5dbf661242a69c97ea363f2d7b46c567da8eadef8890022be6e2ab0/multidict-6.7.1-cp313-cp313-win_arm64.whl", hash = "sha256:563fe25c678aaba333d5399408f5ec3c383ca5b663e7f774dd179a520b8144df", size = 43122, upload-time = "2026-01-26T02:44:43.664Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/b3/e6b21c6c4f314bb956016b0b3ef2162590a529b84cb831c257519e7fde44/multidict-6.7.1-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:c76c4bec1538375dad9d452d246ca5368ad6e1c9039dadcf007ae59c70619ea1", size = 83175, upload-time = "2026-01-26T02:44:44.894Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/76/23ecd2abfe0957b234f6c960f4ade497f55f2c16aeb684d4ecdbf1c95791/multidict-6.7.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:57b46b24b5d5ebcc978da4ec23a819a9402b4228b8a90d9c656422b4bdd8a963", size = 48460, upload-time = "2026-01-26T02:44:46.106Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/57/a0ed92b23f3a042c36bc4227b72b97eca803f5f1801c1ab77c8a212d455e/multidict-6.7.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e954b24433c768ce78ab7929e84ccf3422e46deb45a4dc9f93438f8217fa2d34", size = 46930, upload-time = "2026-01-26T02:44:47.278Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/66/02ec7ace29162e447f6382c495dc95826bf931d3818799bbef11e8f7df1a/multidict-6.7.1-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:3bd231490fa7217cc832528e1cd8752a96f0125ddd2b5749390f7c3ec8721b65", size = 242582, upload-time = "2026-01-26T02:44:48.604Z" },
+ { url = "https://files.pythonhosted.org/packages/58/18/64f5a795e7677670e872673aca234162514696274597b3708b2c0d276cce/multidict-6.7.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:253282d70d67885a15c8a7716f3a73edf2d635793ceda8173b9ecc21f2fb8292", size = 250031, upload-time = "2026-01-26T02:44:50.544Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/ed/e192291dbbe51a8290c5686f482084d31bcd9d09af24f63358c3d42fd284/multidict-6.7.1-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:0b4c48648d7649c9335cf1927a8b87fa692de3dcb15faa676c6a6f1f1aabda43", size = 228596, upload-time = "2026-01-26T02:44:51.951Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/7e/3562a15a60cf747397e7f2180b0a11dc0c38d9175a650e75fa1b4d325e15/multidict-6.7.1-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:98bc624954ec4d2c7cb074b8eefc2b5d0ce7d482e410df446414355d158fe4ca", size = 257492, upload-time = "2026-01-26T02:44:53.902Z" },
+ { url = "https://files.pythonhosted.org/packages/24/02/7d0f9eae92b5249bb50ac1595b295f10e263dd0078ebb55115c31e0eaccd/multidict-6.7.1-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:1b99af4d9eec0b49927b4402bcbb58dea89d3e0db8806a4086117019939ad3dd", size = 255899, upload-time = "2026-01-26T02:44:55.316Z" },
+ { url = "https://files.pythonhosted.org/packages/00/e3/9b60ed9e23e64c73a5cde95269ef1330678e9c6e34dd4eb6b431b85b5a10/multidict-6.7.1-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6aac4f16b472d5b7dc6f66a0d49dd57b0e0902090be16594dc9ebfd3d17c47e7", size = 247970, upload-time = "2026-01-26T02:44:56.783Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/06/538e58a63ed5cfb0bd4517e346b91da32fde409d839720f664e9a4ae4f9d/multidict-6.7.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:21f830fe223215dffd51f538e78c172ed7c7f60c9b96a2bf05c4848ad49921c3", size = 245060, upload-time = "2026-01-26T02:44:58.195Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/2f/d743a3045a97c895d401e9bd29aaa09b94f5cbdf1bd561609e5a6c431c70/multidict-6.7.1-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:f5dd81c45b05518b9aa4da4aa74e1c93d715efa234fd3e8a179df611cc85e5f4", size = 235888, upload-time = "2026-01-26T02:44:59.57Z" },
+ { url = "https://files.pythonhosted.org/packages/38/83/5a325cac191ab28b63c52f14f1131f3b0a55ba3b9aa65a6d0bf2a9b921a0/multidict-6.7.1-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:eb304767bca2bb92fb9c5bd33cedc95baee5bb5f6c88e63706533a1c06ad08c8", size = 243554, upload-time = "2026-01-26T02:45:01.054Z" },
+ { url = "https://files.pythonhosted.org/packages/20/1f/9d2327086bd15da2725ef6aae624208e2ef828ed99892b17f60c344e57ed/multidict-6.7.1-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:c9035dde0f916702850ef66460bc4239d89d08df4d02023a5926e7446724212c", size = 252341, upload-time = "2026-01-26T02:45:02.484Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/2c/2a1aa0280cf579d0f6eed8ee5211c4f1730bd7e06c636ba2ee6aafda302e/multidict-6.7.1-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:af959b9beeb66c822380f222f0e0a1889331597e81f1ded7f374f3ecb0fd6c52", size = 246391, upload-time = "2026-01-26T02:45:03.862Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/03/7ca022ffc36c5a3f6e03b179a5ceb829be9da5783e6fe395f347c0794680/multidict-6.7.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:41f2952231456154ee479651491e94118229844dd7226541788be783be2b5108", size = 243422, upload-time = "2026-01-26T02:45:05.296Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/1d/b31650eab6c5778aceed46ba735bd97f7c7d2f54b319fa916c0f96e7805b/multidict-6.7.1-cp313-cp313t-win32.whl", hash = "sha256:df9f19c28adcb40b6aae30bbaa1478c389efd50c28d541d76760199fc1037c32", size = 47770, upload-time = "2026-01-26T02:45:06.754Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/5b/2d2d1d522e51285bd61b1e20df8f47ae1a9d80839db0b24ea783b3832832/multidict-6.7.1-cp313-cp313t-win_amd64.whl", hash = "sha256:d54ecf9f301853f2c5e802da559604b3e95bb7a3b01a9c295c6ee591b9882de8", size = 53109, upload-time = "2026-01-26T02:45:08.044Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/a3/cc409ba012c83ca024a308516703cf339bdc4b696195644a7215a5164a24/multidict-6.7.1-cp313-cp313t-win_arm64.whl", hash = "sha256:5a37ca18e360377cfda1d62f5f382ff41f2b8c4ccb329ed974cc2e1643440118", size = 45573, upload-time = "2026-01-26T02:45:09.349Z" },
+ { url = "https://files.pythonhosted.org/packages/91/cc/db74228a8be41884a567e88a62fd589a913708fcf180d029898c17a9a371/multidict-6.7.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:8f333ec9c5eb1b7105e3b84b53141e66ca05a19a605368c55450b6ba208cb9ee", size = 75190, upload-time = "2026-01-26T02:45:10.651Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/22/492f2246bb5b534abd44804292e81eeaf835388901f0c574bac4eeec73c5/multidict-6.7.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:a407f13c188f804c759fc6a9f88286a565c242a76b27626594c133b82883b5c2", size = 44486, upload-time = "2026-01-26T02:45:11.938Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/4f/733c48f270565d78b4544f2baddc2fb2a245e5a8640254b12c36ac7ac68e/multidict-6.7.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:0e161ddf326db5577c3a4cc2d8648f81456e8a20d40415541587a71620d7a7d1", size = 43219, upload-time = "2026-01-26T02:45:14.346Z" },
+ { url = "https://files.pythonhosted.org/packages/24/bb/2c0c2287963f4259c85e8bcbba9182ced8d7fca65c780c38e99e61629d11/multidict-6.7.1-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:1e3a8bb24342a8201d178c3b4984c26ba81a577c80d4d525727427460a50c22d", size = 245132, upload-time = "2026-01-26T02:45:15.712Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/f9/44d4b3064c65079d2467888794dea218d1601898ac50222ab8a9a8094460/multidict-6.7.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:97231140a50f5d447d3164f994b86a0bed7cd016e2682f8650d6a9158e14fd31", size = 252420, upload-time = "2026-01-26T02:45:17.293Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/13/78f7275e73fa17b24c9a51b0bd9d73ba64bb32d0ed51b02a746eb876abe7/multidict-6.7.1-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:6b10359683bd8806a200fd2909e7c8ca3a7b24ec1d8132e483d58e791d881048", size = 233510, upload-time = "2026-01-26T02:45:19.356Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/25/8167187f62ae3cbd52da7893f58cb036b47ea3fb67138787c76800158982/multidict-6.7.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:283ddac99f7ac25a4acadbf004cb5ae34480bbeb063520f70ce397b281859362", size = 264094, upload-time = "2026-01-26T02:45:20.834Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/e7/69a3a83b7b030cf283fb06ce074a05a02322359783424d7edf0f15fe5022/multidict-6.7.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:538cec1e18c067d0e6103aa9a74f9e832904c957adc260e61cd9d8cf0c3b3d37", size = 260786, upload-time = "2026-01-26T02:45:22.818Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/3b/8ec5074bcfc450fe84273713b4b0a0dd47c0249358f5d82eb8104ffe2520/multidict-6.7.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7eee46ccb30ff48a1e35bb818cc90846c6be2b68240e42a78599166722cea709", size = 248483, upload-time = "2026-01-26T02:45:24.368Z" },
+ { url = "https://files.pythonhosted.org/packages/48/5a/d5a99e3acbca0e29c5d9cba8f92ceb15dce78bab963b308ae692981e3a5d/multidict-6.7.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:fa263a02f4f2dd2d11a7b1bb4362aa7cb1049f84a9235d31adf63f30143469a0", size = 248403, upload-time = "2026-01-26T02:45:25.982Z" },
+ { url = "https://files.pythonhosted.org/packages/35/48/e58cd31f6c7d5102f2a4bf89f96b9cf7e00b6c6f3d04ecc44417c00a5a3c/multidict-6.7.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:2e1425e2f99ec5bd36c15a01b690a1a2456209c5deed58f95469ffb46039ccbb", size = 240315, upload-time = "2026-01-26T02:45:27.487Z" },
+ { url = "https://files.pythonhosted.org/packages/94/33/1cd210229559cb90b6786c30676bb0c58249ff42f942765f88793b41fdce/multidict-6.7.1-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:497394b3239fc6f0e13a78a3e1b61296e72bf1c5f94b4c4eb80b265c37a131cd", size = 245528, upload-time = "2026-01-26T02:45:28.991Z" },
+ { url = "https://files.pythonhosted.org/packages/64/f2/6e1107d226278c876c783056b7db43d800bb64c6131cec9c8dfb6903698e/multidict-6.7.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:233b398c29d3f1b9676b4b6f75c518a06fcb2ea0b925119fb2c1bc35c05e1601", size = 258784, upload-time = "2026-01-26T02:45:30.503Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/c1/11f664f14d525e4a1b5327a82d4de61a1db604ab34c6603bb3c2cc63ad34/multidict-6.7.1-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:93b1818e4a6e0930454f0f2af7dfce69307ca03cdcfb3739bf4d91241967b6c1", size = 251980, upload-time = "2026-01-26T02:45:32.603Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/9f/75a9ac888121d0c5bbd4ecf4eead45668b1766f6baabfb3b7f66a410e231/multidict-6.7.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f33dc2a3abe9249ea5d8360f969ec7f4142e7ac45ee7014d8f8d5acddf178b7b", size = 243602, upload-time = "2026-01-26T02:45:34.043Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/e7/50bf7b004cc8525d80dbbbedfdc7aed3e4c323810890be4413e589074032/multidict-6.7.1-cp314-cp314-win32.whl", hash = "sha256:3ab8b9d8b75aef9df299595d5388b14530839f6422333357af1339443cff777d", size = 40930, upload-time = "2026-01-26T02:45:36.278Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/bf/52f25716bbe93745595800f36fb17b73711f14da59ed0bb2eba141bc9f0f/multidict-6.7.1-cp314-cp314-win_amd64.whl", hash = "sha256:5e01429a929600e7dab7b166062d9bb54a5eed752384c7384c968c2afab8f50f", size = 45074, upload-time = "2026-01-26T02:45:37.546Z" },
+ { url = "https://files.pythonhosted.org/packages/97/ab/22803b03285fa3a525f48217963da3a65ae40f6a1b6f6cf2768879e208f9/multidict-6.7.1-cp314-cp314-win_arm64.whl", hash = "sha256:4885cb0e817aef5d00a2e8451d4665c1808378dc27c2705f1bf4ef8505c0d2e5", size = 42471, upload-time = "2026-01-26T02:45:38.889Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/6d/f9293baa6146ba9507e360ea0292b6422b016907c393e2f63fc40ab7b7b5/multidict-6.7.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:0458c978acd8e6ea53c81eefaddbbee9c6c5e591f41b3f5e8e194780fe026581", size = 82401, upload-time = "2026-01-26T02:45:40.254Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/68/53b5494738d83558d87c3c71a486504d8373421c3e0dbb6d0db48ad42ee0/multidict-6.7.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:c0abd12629b0af3cf590982c0b413b1e7395cd4ec026f30986818ab95bfaa94a", size = 48143, upload-time = "2026-01-26T02:45:41.635Z" },
+ { url = "https://files.pythonhosted.org/packages/37/e8/5284c53310dcdc99ce5d66563f6e5773531a9b9fe9ec7a615e9bc306b05f/multidict-6.7.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:14525a5f61d7d0c94b368a42cff4c9a4e7ba2d52e2672a7b23d84dc86fb02b0c", size = 46507, upload-time = "2026-01-26T02:45:42.99Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/fc/6800d0e5b3875568b4083ecf5f310dcf91d86d52573160834fb4bfcf5e4f/multidict-6.7.1-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:17307b22c217b4cf05033dabefe68255a534d637c6c9b0cc8382718f87be4262", size = 239358, upload-time = "2026-01-26T02:45:44.376Z" },
+ { url = "https://files.pythonhosted.org/packages/41/75/4ad0973179361cdf3a113905e6e088173198349131be2b390f9fa4da5fc6/multidict-6.7.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7a7e590ff876a3eaf1c02a4dfe0724b6e69a9e9de6d8f556816f29c496046e59", size = 246884, upload-time = "2026-01-26T02:45:47.167Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/9c/095bb28b5da139bd41fb9a5d5caff412584f377914bd8787c2aa98717130/multidict-6.7.1-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:5fa6a95dfee63893d80a34758cd0e0c118a30b8dcb46372bf75106c591b77889", size = 225878, upload-time = "2026-01-26T02:45:48.698Z" },
+ { url = "https://files.pythonhosted.org/packages/07/d0/c0a72000243756e8f5a277b6b514fa005f2c73d481b7d9e47cd4568aa2e4/multidict-6.7.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a0543217a6a017692aa6ae5cc39adb75e587af0f3a82288b1492eb73dd6cc2a4", size = 253542, upload-time = "2026-01-26T02:45:50.164Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/6b/f69da15289e384ecf2a68837ec8b5ad8c33e973aa18b266f50fe55f24b8c/multidict-6.7.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f99fe611c312b3c1c0ace793f92464d8cd263cc3b26b5721950d977b006b6c4d", size = 252403, upload-time = "2026-01-26T02:45:51.779Z" },
+ { url = "https://files.pythonhosted.org/packages/a2/76/b9669547afa5a1a25cd93eaca91c0da1c095b06b6d2d8ec25b713588d3a1/multidict-6.7.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9004d8386d133b7e6135679424c91b0b854d2d164af6ea3f289f8f2761064609", size = 244889, upload-time = "2026-01-26T02:45:53.27Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/a9/a50d2669e506dad33cfc45b5d574a205587b7b8a5f426f2fbb2e90882588/multidict-6.7.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e628ef0e6859ffd8273c69412a2465c4be4a9517d07261b33334b5ec6f3c7489", size = 241982, upload-time = "2026-01-26T02:45:54.919Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/bb/1609558ad8b456b4827d3c5a5b775c93b87878fd3117ed3db3423dfbce1b/multidict-6.7.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:841189848ba629c3552035a6a7f5bf3b02eb304e9fea7492ca220a8eda6b0e5c", size = 232415, upload-time = "2026-01-26T02:45:56.981Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/59/6f61039d2aa9261871e03ab9dc058a550d240f25859b05b67fd70f80d4b3/multidict-6.7.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:ce1bbd7d780bb5a0da032e095c951f7014d6b0a205f8318308140f1a6aba159e", size = 240337, upload-time = "2026-01-26T02:45:58.698Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/29/fdc6a43c203890dc2ae9249971ecd0c41deaedfe00d25cb6564b2edd99eb/multidict-6.7.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:b26684587228afed0d50cf804cc71062cc9c1cdf55051c4c6345d372947b268c", size = 248788, upload-time = "2026-01-26T02:46:00.862Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/14/a153a06101323e4cf086ecee3faadba52ff71633d471f9685c42e3736163/multidict-6.7.1-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:9f9af11306994335398293f9958071019e3ab95e9a707dc1383a35613f6abcb9", size = 242842, upload-time = "2026-01-26T02:46:02.824Z" },
+ { url = "https://files.pythonhosted.org/packages/41/5f/604ae839e64a4a6efc80db94465348d3b328ee955e37acb24badbcd24d83/multidict-6.7.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:b4938326284c4f1224178a560987b6cf8b4d38458b113d9b8c1db1a836e640a2", size = 240237, upload-time = "2026-01-26T02:46:05.898Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/60/c3a5187bf66f6fb546ff4ab8fb5a077cbdd832d7b1908d4365c7f74a1917/multidict-6.7.1-cp314-cp314t-win32.whl", hash = "sha256:98655c737850c064a65e006a3df7c997cd3b220be4ec8fe26215760b9697d4d7", size = 48008, upload-time = "2026-01-26T02:46:07.468Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/f7/addf1087b860ac60e6f382240f64fb99f8bfb532bb06f7c542b83c29ca61/multidict-6.7.1-cp314-cp314t-win_amd64.whl", hash = "sha256:497bde6223c212ba11d462853cfa4f0ae6ef97465033e7dc9940cdb3ab5b48e5", size = 53542, upload-time = "2026-01-26T02:46:08.809Z" },
+ { url = "https://files.pythonhosted.org/packages/4c/81/4629d0aa32302ef7b2ec65c75a728cc5ff4fa410c50096174c1632e70b3e/multidict-6.7.1-cp314-cp314t-win_arm64.whl", hash = "sha256:2bbd113e0d4af5db41d5ebfe9ccaff89de2120578164f86a5d17d5a576d1e5b2", size = 44719, upload-time = "2026-01-26T02:46:11.146Z" },
+ { url = "https://files.pythonhosted.org/packages/81/08/7036c080d7117f28a4af526d794aab6a84463126db031b007717c1a6676e/multidict-6.7.1-py3-none-any.whl", hash = "sha256:55d97cc6dae627efa6a6e548885712d4864b81110ac76fa4e534c03819fa4a56", size = 12319, upload-time = "2026-01-26T02:46:44.004Z" },
]
[[package]]
name = "multiprocess"
-version = "0.70.18"
+version = "0.70.19"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "dill" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/72/fd/2ae3826f5be24c6ed87266bc4e59c46ea5b059a103f3d7e7eb76a52aeecb/multiprocess-0.70.18.tar.gz", hash = "sha256:f9597128e6b3e67b23956da07cf3d2e5cba79e2f4e0fba8d7903636663ec6d0d", size = 1798503, upload-time = "2025-04-17T03:11:27.742Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/a2/f2/e783ac7f2aeeed14e9e12801f22529cc7e6b7ab80928d6dcce4e9f00922d/multiprocess-0.70.19.tar.gz", hash = "sha256:952021e0e6c55a4a9fe4cd787895b86e239a40e76802a789d6305398d3975897", size = 2079989, upload-time = "2026-01-19T06:47:39.744Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/c8/f8/7f9a8f08bf98cea1dfaa181e05cc8bbcb59cecf044b5a9ac3cce39f9c449/multiprocess-0.70.18-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:25d4012dcaaf66b9e8e955f58482b42910c2ee526d532844d8bcf661bbc604df", size = 135083, upload-time = "2025-04-17T03:11:04.223Z" },
- { url = "https://files.pythonhosted.org/packages/e5/03/b7b10dbfc17b2b3ce07d4d30b3ba8367d0ed32d6d46cd166e298f161dd46/multiprocess-0.70.18-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:06b19433de0d02afe5869aec8931dd5c01d99074664f806c73896b0d9e527213", size = 135128, upload-time = "2025-04-17T03:11:06.045Z" },
- { url = "https://files.pythonhosted.org/packages/c1/a3/5f8d3b9690ea5580bee5868ab7d7e2cfca74b7e826b28192b40aa3881cdc/multiprocess-0.70.18-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:6fa1366f994373aaf2d4738b0f56e707caeaa05486e97a7f71ee0853823180c2", size = 135132, upload-time = "2025-04-17T03:11:07.533Z" },
- { url = "https://files.pythonhosted.org/packages/55/4d/9af0d1279c84618bcd35bf5fd7e371657358c7b0a523e54a9cffb87461f8/multiprocess-0.70.18-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:8b8940ae30139e04b076da6c5b83e9398585ebdf0f2ad3250673fef5b2ff06d6", size = 144695, upload-time = "2025-04-17T03:11:09.161Z" },
- { url = "https://files.pythonhosted.org/packages/17/bf/87323e79dd0562474fad3373c21c66bc6c3c9963b68eb2a209deb4c8575e/multiprocess-0.70.18-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:0929ba95831adb938edbd5fb801ac45e705ecad9d100b3e653946b7716cb6bd3", size = 144742, upload-time = "2025-04-17T03:11:10.072Z" },
- { url = "https://files.pythonhosted.org/packages/dd/74/cb8c831e58dc6d5cf450b17c7db87f14294a1df52eb391da948b5e0a0b94/multiprocess-0.70.18-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:4d77f8e4bfe6c6e2e661925bbf9aed4d5ade9a1c6502d5dfc10129b9d1141797", size = 144745, upload-time = "2025-04-17T03:11:11.453Z" },
- { url = "https://files.pythonhosted.org/packages/ba/d8/0cba6cf51a1a31f20471fbc823a716170c73012ddc4fb85d706630ed6e8f/multiprocess-0.70.18-py310-none-any.whl", hash = "sha256:60c194974c31784019c1f459d984e8f33ee48f10fcf42c309ba97b30d9bd53ea", size = 134948, upload-time = "2025-04-17T03:11:20.223Z" },
- { url = "https://files.pythonhosted.org/packages/4b/88/9039f2fed1012ef584751d4ceff9ab4a51e5ae264898f0b7cbf44340a859/multiprocess-0.70.18-py311-none-any.whl", hash = "sha256:5aa6eef98e691281b3ad923be2832bf1c55dd2c859acd73e5ec53a66aae06a1d", size = 144462, upload-time = "2025-04-17T03:11:21.657Z" },
- { url = "https://files.pythonhosted.org/packages/bf/b6/5f922792be93b82ec6b5f270bbb1ef031fd0622847070bbcf9da816502cc/multiprocess-0.70.18-py312-none-any.whl", hash = "sha256:9b78f8e5024b573730bfb654783a13800c2c0f2dfc0c25e70b40d184d64adaa2", size = 150287, upload-time = "2025-04-17T03:11:22.69Z" },
- { url = "https://files.pythonhosted.org/packages/ee/25/7d7e78e750bc1aecfaf0efbf826c69a791d2eeaf29cf20cba93ff4cced78/multiprocess-0.70.18-py313-none-any.whl", hash = "sha256:871743755f43ef57d7910a38433cfe41319e72be1bbd90b79c7a5ac523eb9334", size = 151917, upload-time = "2025-04-17T03:11:24.044Z" },
- { url = "https://files.pythonhosted.org/packages/3b/c3/ca84c19bd14cdfc21c388fdcebf08b86a7a470ebc9f5c3c084fc2dbc50f7/multiprocess-0.70.18-py38-none-any.whl", hash = "sha256:dbf705e52a154fe5e90fb17b38f02556169557c2dd8bb084f2e06c2784d8279b", size = 132636, upload-time = "2025-04-17T03:11:24.936Z" },
- { url = "https://files.pythonhosted.org/packages/6c/28/dd72947e59a6a8c856448a5e74da6201cb5502ddff644fbc790e4bd40b9a/multiprocess-0.70.18-py39-none-any.whl", hash = "sha256:e78ca805a72b1b810c690b6b4cc32579eba34f403094bbbae962b7b5bf9dfcb8", size = 133478, upload-time = "2025-04-17T03:11:26.253Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/45/8004d1e6b9185c1a444d6b55ac5682acf9d98035e54386d967366035a03a/multiprocess-0.70.19-py310-none-any.whl", hash = "sha256:97404393419dcb2a8385910864eedf47a3cadf82c66345b44f036420eb0b5d87", size = 134948, upload-time = "2026-01-19T06:47:32.325Z" },
+ { url = "https://files.pythonhosted.org/packages/86/c2/dec9722dc3474c164a0b6bcd9a7ed7da542c98af8cabce05374abab35edd/multiprocess-0.70.19-py311-none-any.whl", hash = "sha256:928851ae7973aea4ce0eaf330bbdafb2e01398a91518d5c8818802845564f45c", size = 144457, upload-time = "2026-01-19T06:47:33.711Z" },
+ { url = "https://files.pythonhosted.org/packages/71/70/38998b950a97ea279e6bd657575d22d1a2047256caf707d9a10fbce4f065/multiprocess-0.70.19-py312-none-any.whl", hash = "sha256:3a56c0e85dd5025161bac5ce138dcac1e49174c7d8e74596537e729fd5c53c28", size = 150281, upload-time = "2026-01-19T06:47:35.037Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/74/d2c27e03cb84251dfe7249b8e82923643c6d48fa4883b9476b025e7dc7eb/multiprocess-0.70.19-py313-none-any.whl", hash = "sha256:8d5eb4ec5017ba2fab4e34a747c6d2c2b6fecfe9e7236e77988db91580ada952", size = 156414, upload-time = "2026-01-19T06:47:35.915Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/61/af9115673a5870fd885247e2f1b68c4f1197737da315b520a91c757a861a/multiprocess-0.70.19-py314-none-any.whl", hash = "sha256:e8cc7fbdff15c0613f0a1f1f8744bef961b0a164c0ca29bdff53e9d2d93c5e5f", size = 160318, upload-time = "2026-01-19T06:47:37.497Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/82/69e539c4c2027f1e1697e09aaa2449243085a0edf81ae2c6341e84d769b6/multiprocess-0.70.19-py39-none-any.whl", hash = "sha256:0d4b4397ed669d371c81dcd1ef33fd384a44d6c3de1bd0ca7ac06d837720d3c5", size = 133477, upload-time = "2026-01-19T06:47:38.619Z" },
]
[[package]]
@@ -2786,7 +3107,7 @@ wheels = [
[[package]]
name = "myst-parser"
-version = "4.0.1"
+version = "5.0.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "docutils" },
@@ -2794,12 +3115,11 @@ dependencies = [
{ name = "markdown-it-py" },
{ name = "mdit-py-plugins" },
{ name = "pyyaml" },
- { name = "sphinx", version = "8.1.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinx", version = "8.2.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "sphinx" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/66/a5/9626ba4f73555b3735ad86247a8077d4603aa8628537687c839ab08bfe44/myst_parser-4.0.1.tar.gz", hash = "sha256:5cfea715e4f3574138aecbf7d54132296bfd72bb614d31168f48c477a830a7c4", size = 93985, upload-time = "2025-02-12T10:53:03.833Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/33/fa/7b45eef11b7971f0beb29d27b7bfe0d747d063aa29e170d9edd004733c8a/myst_parser-5.0.0.tar.gz", hash = "sha256:f6f231452c56e8baa662cc352c548158f6a16fcbd6e3800fc594978002b94f3a", size = 98535, upload-time = "2026-01-15T09:08:18.036Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/5f/df/76d0321c3797b54b60fef9ec3bd6f4cfd124b9e422182156a1dd418722cf/myst_parser-4.0.1-py3-none-any.whl", hash = "sha256:9134e88959ec3b5780aedf8a99680ea242869d012e8821db3126d427edc9c95d", size = 84579, upload-time = "2025-02-12T10:53:02.078Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/ac/686789b9145413f1a61878c407210e41bfdb097976864e0913078b24098c/myst_parser-5.0.0-py3-none-any.whl", hash = "sha256:ab31e516024918296e169139072b81592336f2fef55b8986aa31c9f04b5f7211", size = 84533, upload-time = "2026-01-15T09:08:16.788Z" },
]
[[package]]
@@ -2814,10 +3134,10 @@ dependencies = [
{ name = "inquirerpy" },
{ name = "jinja2" },
{ name = "leptonai" },
- { name = "networkx", version = "3.4.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "networkx", version = "3.6.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "networkx" },
{ name = "omegaconf" },
- { name = "packaging" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
{ name = "rich" },
{ name = "toml" },
{ name = "torchx" },
@@ -2826,34 +3146,45 @@ dependencies = [
[[package]]
name = "networkx"
-version = "3.4.2"
+version = "3.6.1"
source = { registry = "https://pypi.org/simple" }
-resolution-markers = [
- "python_full_version < '3.11' and sys_platform == 'linux'",
- "python_full_version < '3.11' and sys_platform != 'linux'",
-]
-sdist = { url = "https://files.pythonhosted.org/packages/fd/1d/06475e1cd5264c0b870ea2cc6fdb3e37177c1e565c43f56ff17a10e3937f/networkx-3.4.2.tar.gz", hash = "sha256:307c3669428c5362aab27c8a1260aa8f47c4e91d3891f48be0141738d8d053e1", size = 2151368, upload-time = "2024-10-21T12:39:38.695Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/6a/51/63fe664f3908c97be9d2e4f1158eb633317598cfa6e1fc14af5383f17512/networkx-3.6.1.tar.gz", hash = "sha256:26b7c357accc0c8cde558ad486283728b65b6a95d85ee1cd66bafab4c8168509", size = 2517025, upload-time = "2025-12-08T17:02:39.908Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/b9/54/dd730b32ea14ea797530a4479b2ed46a6fb250f682a9cfb997e968bf0261/networkx-3.4.2-py3-none-any.whl", hash = "sha256:df5d4365b724cf81b8c6a7312509d0c22386097011ad1abe274afd5e9d3bbc5f", size = 1723263, upload-time = "2024-10-21T12:39:36.247Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" },
]
[[package]]
-name = "networkx"
-version = "3.6.1"
+name = "nh3"
+version = "0.3.4"
source = { registry = "https://pypi.org/simple" }
-resolution-markers = [
- "python_full_version >= '3.14' and sys_platform == 'linux'",
- "python_full_version == '3.13.*' and sys_platform == 'linux'",
- "python_full_version == '3.12.*' and sys_platform == 'linux'",
- "python_full_version >= '3.14' and sys_platform != 'linux'",
- "python_full_version == '3.13.*' and sys_platform != 'linux'",
- "python_full_version == '3.12.*' and sys_platform != 'linux'",
- "python_full_version == '3.11.*' and sys_platform == 'linux'",
- "python_full_version == '3.11.*' and sys_platform != 'linux'",
-]
-sdist = { url = "https://files.pythonhosted.org/packages/6a/51/63fe664f3908c97be9d2e4f1158eb633317598cfa6e1fc14af5383f17512/networkx-3.6.1.tar.gz", hash = "sha256:26b7c357accc0c8cde558ad486283728b65b6a95d85ee1cd66bafab4c8168509", size = 2517025, upload-time = "2025-12-08T17:02:39.908Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" },
+sdist = { url = "https://files.pythonhosted.org/packages/4e/86/f8d3a7c9bd1bbaa181f6312c757e0b74d25f71ecf84ea3c0dc5e0f01840d/nh3-0.3.4.tar.gz", hash = "sha256:96709a379997c1b28c8974146ca660b0dcd3794f4f6d50c1ea549bab39ac6ade", size = 19520, upload-time = "2026-03-25T10:57:30.789Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fd/5e/c400663d14be2216bc084ed2befc871b7b12563f85d40904f2a4bf0dd2b7/nh3-0.3.4-cp314-cp314t-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:8b61058f34c2105d44d2a4d4241bacf603a1ef5c143b08766bbd0cf23830118f", size = 1417991, upload-time = "2026-03-25T10:56:59.13Z" },
+ { url = "https://files.pythonhosted.org/packages/36/f5/109526f5002ec41322ac8cafd50f0f154bae0c26b9607c0fcb708bdca8ec/nh3-0.3.4-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:554cc2bab281758e94d770c3fb0bf2d8be5fb403ef6b2e8841dd7c1615df7a0f", size = 790566, upload-time = "2026-03-25T10:57:00.445Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/66/38950f2b4b316ffd82ee51ed8f9143d1f56fdd620312cacc91613b77b3e7/nh3-0.3.4-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:dbe76feaa44e2ef9436f345016012a591550e77818876a8de5c8bc2a248e08df", size = 837538, upload-time = "2026-03-25T10:57:01.848Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/9f/9d6da970e9524fe360ea02a2082856390c2c8ba540409d1be6e5851887b3/nh3-0.3.4-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:87dac8d611b4a478400e0821a13b35770e88c266582f065e7249d6a37b0f86e8", size = 1012154, upload-time = "2026-03-25T10:57:03.592Z" },
+ { url = "https://files.pythonhosted.org/packages/54/92/7c85c33c241e9dd51dda115bd3f765e940446588cdaaca62ef8edffe675f/nh3-0.3.4-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:8d697e19f2995b337f648204848ac3a528eaafffc39e7ce4ac6b7a2fbe6c84af", size = 1092516, upload-time = "2026-03-25T10:57:04.726Z" },
+ { url = "https://files.pythonhosted.org/packages/16/0f/597842bdb2890999a3faa2f3fcb02db8aa6ad09320d3d843ff6d0a1f737b/nh3-0.3.4-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:7cae217f031809321db962cd7e092bda8d4e95a87f78c0226628fa6c2ea8ebc5", size = 1053793, upload-time = "2026-03-25T10:57:06.171Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/32/669da65147bc10746d2e1d7a8a3dbfbffe0315f419e74b559e2ee3471a01/nh3-0.3.4-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:07999b998bf89692738f15c0eac76a416382932f855709e0b7488b595c30ec89", size = 1035975, upload-time = "2026-03-25T10:57:07.292Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/7e/9e97a8b3c5161c79b4bf21cc54e9334860a52cc54ede15bf2239ef494b73/nh3-0.3.4-cp314-cp314t-win32.whl", hash = "sha256:ca90397c8d36c1535bf1988b2bed006597337843a164c7ec269dc8813f37536b", size = 600419, upload-time = "2026-03-25T10:57:08.342Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/c7/6849d8d4295d3997d148eacb2d4b1c9faada4895ee3c1b1e12e72f4611e2/nh3-0.3.4-cp314-cp314t-win_amd64.whl", hash = "sha256:41e46b3499918ab6128b6421677b316e79869d0c140da24069d220a94f4e72d1", size = 613342, upload-time = "2026-03-25T10:57:09.593Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/0e/14a3f510f36c20b922c123a2730f071f938d006fb513aacfd46d6cbc03a7/nh3-0.3.4-cp314-cp314t-win_arm64.whl", hash = "sha256:80b955d802bf365bd42e09f6c3d64567dce777d20e97968d94b3e9d9e99b265e", size = 607025, upload-time = "2026-03-25T10:57:10.959Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/57/a97955bc95960cfb1f0517043d60a121f4ba93fde252d4d9ffd3c2a9eead/nh3-0.3.4-cp38-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:d8bebcb20ab4b91858385cd98fe58046ec4a624275b45ef9b976475604f45b49", size = 1439519, upload-time = "2026-03-25T10:57:12.019Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/60/c9a33361da8cde7c7760f091cd10467bc470634e4eea31c8bb70935b00a4/nh3-0.3.4-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0d825722a1e8cbc87d7ca1e47ffb1d2a6cf343ad4c1b8465becf7cadcabcdfd0", size = 833798, upload-time = "2026-03-25T10:57:13.264Z" },
+ { url = "https://files.pythonhosted.org/packages/6b/19/9487790780b8c94eacca37866c1270b747a4af8e244d43b3b550fddbbf62/nh3-0.3.4-cp38-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:4aa8b43e68c26b68069a3b6cef09de166d1d7fa140cf8d77e409a46cbf742e44", size = 820414, upload-time = "2026-03-25T10:57:14.236Z" },
+ { url = "https://files.pythonhosted.org/packages/6b/b4/c6a340dd321d20b1e4a663307032741da045685c87403926c43656f6f5ec/nh3-0.3.4-cp38-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl", hash = "sha256:f5f214618ad5eff4f2a6b13a8d4da4d9e7f37c569d90a13fb9f0caaf7d04fe21", size = 1061531, upload-time = "2026-03-25T10:57:15.384Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/49/f6b4b474e0032e4bcbb7174b44e4cf6915670e09c62421deb06ccfcb88b8/nh3-0.3.4-cp38-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3390e4333883673a684ce16c1716b481e91782d6f56dec5c85fed9feedb23382", size = 1021889, upload-time = "2026-03-25T10:57:16.454Z" },
+ { url = "https://files.pythonhosted.org/packages/43/da/e52a6941746d1f974752af3fc8591f1dbcdcf7fd8c726c7d99f444ba820e/nh3-0.3.4-cp38-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:18a2e44ccb29cbb45071b8f3f2dab9ebfb41a6516f328f91f1f1fd18196239a4", size = 912965, upload-time = "2026-03-25T10:57:17.624Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/b7/ec1cbc6b297a808c513f59f501656389623fc09ad6a58c640851289c7854/nh3-0.3.4-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0961a27dc2057c38d0364cb05880e1997ae1c80220cbc847db63213720b8f304", size = 804975, upload-time = "2026-03-25T10:57:18.994Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/56/b1275aa2c6510191eed76178da4626b0900402439cb9f27d6b9bf7c6d5e9/nh3-0.3.4-cp38-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:9337517edb7c10228252cce2898e20fb3d77e32ffaccbb3c66897927d74215a0", size = 833400, upload-time = "2026-03-25T10:57:20.086Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/a5/5d574ffa3c6e49a5364d1b25ebad165501c055340056671493beb467a15e/nh3-0.3.4-cp38-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:d866701affe67a5171b916b5c076e767a74c6a9efb7fb2006eb8d3c5f9a293d5", size = 854277, upload-time = "2026-03-25T10:57:21.433Z" },
+ { url = "https://files.pythonhosted.org/packages/79/36/8aeb2ab21517cefa212db109e41024e02650716cb42bf293d0a88437a92d/nh3-0.3.4-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:47d749d99ae005ab19517224140b280dd56e77b33afb82f9b600e106d0458003", size = 1022021, upload-time = "2026-03-25T10:57:22.433Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/95/9fd860997685e64abe2d5a995ca2eb5004c0fb6d6585429612a7871548b9/nh3-0.3.4-cp38-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:f987cb56458323405e8e5ea827e1befcf141ffa0c0ac797d6d02e6b646056d9a", size = 1103526, upload-time = "2026-03-25T10:57:23.487Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/0d/df545070614c1007f0109bb004230226c9000e7857c9785583ec25cda9d7/nh3-0.3.4-cp38-abi3-musllinux_1_2_i686.whl", hash = "sha256:883d5a6d6ee8078c4afc8e96e022fe579c4c265775ff6ee21e39b8c542cabab3", size = 1068050, upload-time = "2026-03-25T10:57:24.624Z" },
+ { url = "https://files.pythonhosted.org/packages/94/d5/17b016df52df052f714c53be71df26a1943551d9931e9383b92c998b88f8/nh3-0.3.4-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:75643c22f5092d8e209f766ee8108c400bc1e44760fc94d2d638eb138d18f853", size = 1046037, upload-time = "2026-03-25T10:57:25.799Z" },
+ { url = "https://files.pythonhosted.org/packages/51/39/49f737907e6ab2b4ca71855d3bd63dd7958862e9c8b94fb4e5b18ccf6988/nh3-0.3.4-cp38-abi3-win32.whl", hash = "sha256:72e4e9ca1c4bd41b4a28b0190edc2e21e3f71496acd36a0162858e1a28db3d7e", size = 609542, upload-time = "2026-03-25T10:57:27.112Z" },
+ { url = "https://files.pythonhosted.org/packages/73/4f/af8e9071d7464575a7316831938237ffc9d92d27f163dbdd964b1309cd9b/nh3-0.3.4-cp38-abi3-win_amd64.whl", hash = "sha256:c10b1f0c741e257a5cb2978d6bac86e7c784ab20572724b20c6402c2e24bce75", size = 624244, upload-time = "2026-03-25T10:57:28.302Z" },
+ { url = "https://files.pythonhosted.org/packages/44/0c/37695d6b0168f6714b5c492331636a9e6123d6ec22d25876c68d06eab1b8/nh3-0.3.4-cp38-abi3-win_arm64.whl", hash = "sha256:43ad4eedee7e049b9069bc015b7b095d320ed6d167ecec111f877de1540656e9", size = 616649, upload-time = "2026-03-25T10:57:29.623Z" },
]
[[package]]
@@ -2884,7 +3215,7 @@ wheels = [
[[package]]
name = "nltk"
-version = "3.9.2"
+version = "3.9.4"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "click" },
@@ -2892,258 +3223,318 @@ dependencies = [
{ name = "regex" },
{ name = "tqdm" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/f9/76/3a5e4312c19a028770f86fd7c058cf9f4ec4321c6cf7526bab998a5b683c/nltk-3.9.2.tar.gz", hash = "sha256:0f409e9b069ca4177c1903c3e843eef90c7e92992fa4931ae607da6de49e1419", size = 2887629, upload-time = "2025-10-01T07:19:23.764Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/74/a1/b3b4adf15585a5bc4c357adde150c01ebeeb642173ded4d871e89468767c/nltk-3.9.4.tar.gz", hash = "sha256:ed03bc098a40481310320808b2db712d95d13ca65b27372f8a403949c8b523d0", size = 2946864, upload-time = "2026-03-24T06:13:40.641Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/60/90/81ac364ef94209c100e12579629dc92bf7a709a84af32f8c551b02c07e94/nltk-3.9.2-py3-none-any.whl", hash = "sha256:1e209d2b3009110635ed9709a67a1a3e33a10f799490fa71cf4bec218c11c88a", size = 1513404, upload-time = "2025-10-01T07:19:21.648Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/91/04e965f8e717ba0ab4bdca5c112deeab11c9e750d94c4d4602f050295d39/nltk-3.9.4-py3-none-any.whl", hash = "sha256:f2fa301c3a12718ce4a0e9305c5675299da5ad9e26068218b69d692fda84828f", size = 1552087, upload-time = "2026-03-24T06:13:38.47Z" },
]
[[package]]
name = "numpy"
-version = "2.2.6"
+version = "2.0.2"
source = { registry = "https://pypi.org/simple" }
resolution-markers = [
- "python_full_version < '3.11' and sys_platform == 'linux'",
- "python_full_version < '3.11' and sys_platform != 'linux'",
-]
-sdist = { url = "https://files.pythonhosted.org/packages/76/21/7d2a95e4bba9dc13d043ee156a356c0a8f0c6309dff6b21b4d71a073b8a8/numpy-2.2.6.tar.gz", hash = "sha256:e29554e2bef54a90aa5cc07da6ce955accb83f21ab5de01a62c8478897b264fd", size = 20276440, upload-time = "2025-05-17T22:38:04.611Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/9a/3e/ed6db5be21ce87955c0cbd3009f2803f59fa08df21b5df06862e2d8e2bdd/numpy-2.2.6-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:b412caa66f72040e6d268491a59f2c43bf03eb6c96dd8f0307829feb7fa2b6fb", size = 21165245, upload-time = "2025-05-17T21:27:58.555Z" },
- { url = "https://files.pythonhosted.org/packages/22/c2/4b9221495b2a132cc9d2eb862e21d42a009f5a60e45fc44b00118c174bff/numpy-2.2.6-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:8e41fd67c52b86603a91c1a505ebaef50b3314de0213461c7a6e99c9a3beff90", size = 14360048, upload-time = "2025-05-17T21:28:21.406Z" },
- { url = "https://files.pythonhosted.org/packages/fd/77/dc2fcfc66943c6410e2bf598062f5959372735ffda175b39906d54f02349/numpy-2.2.6-cp310-cp310-macosx_14_0_arm64.whl", hash = "sha256:37e990a01ae6ec7fe7fa1c26c55ecb672dd98b19c3d0e1d1f326fa13cb38d163", size = 5340542, upload-time = "2025-05-17T21:28:30.931Z" },
- { url = "https://files.pythonhosted.org/packages/7a/4f/1cb5fdc353a5f5cc7feb692db9b8ec2c3d6405453f982435efc52561df58/numpy-2.2.6-cp310-cp310-macosx_14_0_x86_64.whl", hash = "sha256:5a6429d4be8ca66d889b7cf70f536a397dc45ba6faeb5f8c5427935d9592e9cf", size = 6878301, upload-time = "2025-05-17T21:28:41.613Z" },
- { url = "https://files.pythonhosted.org/packages/eb/17/96a3acd228cec142fcb8723bd3cc39c2a474f7dcf0a5d16731980bcafa95/numpy-2.2.6-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:efd28d4e9cd7d7a8d39074a4d44c63eda73401580c5c76acda2ce969e0a38e83", size = 14297320, upload-time = "2025-05-17T21:29:02.78Z" },
- { url = "https://files.pythonhosted.org/packages/b4/63/3de6a34ad7ad6646ac7d2f55ebc6ad439dbbf9c4370017c50cf403fb19b5/numpy-2.2.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fc7b73d02efb0e18c000e9ad8b83480dfcd5dfd11065997ed4c6747470ae8915", size = 16801050, upload-time = "2025-05-17T21:29:27.675Z" },
- { url = "https://files.pythonhosted.org/packages/07/b6/89d837eddef52b3d0cec5c6ba0456c1bf1b9ef6a6672fc2b7873c3ec4e2e/numpy-2.2.6-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:74d4531beb257d2c3f4b261bfb0fc09e0f9ebb8842d82a7b4209415896adc680", size = 15807034, upload-time = "2025-05-17T21:29:51.102Z" },
- { url = "https://files.pythonhosted.org/packages/01/c8/dc6ae86e3c61cfec1f178e5c9f7858584049b6093f843bca541f94120920/numpy-2.2.6-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:8fc377d995680230e83241d8a96def29f204b5782f371c532579b4f20607a289", size = 18614185, upload-time = "2025-05-17T21:30:18.703Z" },
- { url = "https://files.pythonhosted.org/packages/5b/c5/0064b1b7e7c89137b471ccec1fd2282fceaae0ab3a9550f2568782d80357/numpy-2.2.6-cp310-cp310-win32.whl", hash = "sha256:b093dd74e50a8cba3e873868d9e93a85b78e0daf2e98c6797566ad8044e8363d", size = 6527149, upload-time = "2025-05-17T21:30:29.788Z" },
- { url = "https://files.pythonhosted.org/packages/a3/dd/4b822569d6b96c39d1215dbae0582fd99954dcbcf0c1a13c61783feaca3f/numpy-2.2.6-cp310-cp310-win_amd64.whl", hash = "sha256:f0fd6321b839904e15c46e0d257fdd101dd7f530fe03fd6359c1ea63738703f3", size = 12904620, upload-time = "2025-05-17T21:30:48.994Z" },
- { url = "https://files.pythonhosted.org/packages/da/a8/4f83e2aa666a9fbf56d6118faaaf5f1974d456b1823fda0a176eff722839/numpy-2.2.6-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:f9f1adb22318e121c5c69a09142811a201ef17ab257a1e66ca3025065b7f53ae", size = 21176963, upload-time = "2025-05-17T21:31:19.36Z" },
- { url = "https://files.pythonhosted.org/packages/b3/2b/64e1affc7972decb74c9e29e5649fac940514910960ba25cd9af4488b66c/numpy-2.2.6-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:c820a93b0255bc360f53eca31a0e676fd1101f673dda8da93454a12e23fc5f7a", size = 14406743, upload-time = "2025-05-17T21:31:41.087Z" },
- { url = "https://files.pythonhosted.org/packages/4a/9f/0121e375000b5e50ffdd8b25bf78d8e1a5aa4cca3f185d41265198c7b834/numpy-2.2.6-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:3d70692235e759f260c3d837193090014aebdf026dfd167834bcba43e30c2a42", size = 5352616, upload-time = "2025-05-17T21:31:50.072Z" },
- { url = "https://files.pythonhosted.org/packages/31/0d/b48c405c91693635fbe2dcd7bc84a33a602add5f63286e024d3b6741411c/numpy-2.2.6-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:481b49095335f8eed42e39e8041327c05b0f6f4780488f61286ed3c01368d491", size = 6889579, upload-time = "2025-05-17T21:32:01.712Z" },
- { url = "https://files.pythonhosted.org/packages/52/b8/7f0554d49b565d0171eab6e99001846882000883998e7b7d9f0d98b1f934/numpy-2.2.6-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b64d8d4d17135e00c8e346e0a738deb17e754230d7e0810ac5012750bbd85a5a", size = 14312005, upload-time = "2025-05-17T21:32:23.332Z" },
- { url = "https://files.pythonhosted.org/packages/b3/dd/2238b898e51bd6d389b7389ffb20d7f4c10066d80351187ec8e303a5a475/numpy-2.2.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ba10f8411898fc418a521833e014a77d3ca01c15b0c6cdcce6a0d2897e6dbbdf", size = 16821570, upload-time = "2025-05-17T21:32:47.991Z" },
- { url = "https://files.pythonhosted.org/packages/83/6c/44d0325722cf644f191042bf47eedad61c1e6df2432ed65cbe28509d404e/numpy-2.2.6-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:bd48227a919f1bafbdda0583705e547892342c26fb127219d60a5c36882609d1", size = 15818548, upload-time = "2025-05-17T21:33:11.728Z" },
- { url = "https://files.pythonhosted.org/packages/ae/9d/81e8216030ce66be25279098789b665d49ff19eef08bfa8cb96d4957f422/numpy-2.2.6-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:9551a499bf125c1d4f9e250377c1ee2eddd02e01eac6644c080162c0c51778ab", size = 18620521, upload-time = "2025-05-17T21:33:39.139Z" },
- { url = "https://files.pythonhosted.org/packages/6a/fd/e19617b9530b031db51b0926eed5345ce8ddc669bb3bc0044b23e275ebe8/numpy-2.2.6-cp311-cp311-win32.whl", hash = "sha256:0678000bb9ac1475cd454c6b8c799206af8107e310843532b04d49649c717a47", size = 6525866, upload-time = "2025-05-17T21:33:50.273Z" },
- { url = "https://files.pythonhosted.org/packages/31/0a/f354fb7176b81747d870f7991dc763e157a934c717b67b58456bc63da3df/numpy-2.2.6-cp311-cp311-win_amd64.whl", hash = "sha256:e8213002e427c69c45a52bbd94163084025f533a55a59d6f9c5b820774ef3303", size = 12907455, upload-time = "2025-05-17T21:34:09.135Z" },
- { url = "https://files.pythonhosted.org/packages/82/5d/c00588b6cf18e1da539b45d3598d3557084990dcc4331960c15ee776ee41/numpy-2.2.6-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:41c5a21f4a04fa86436124d388f6ed60a9343a6f767fced1a8a71c3fbca038ff", size = 20875348, upload-time = "2025-05-17T21:34:39.648Z" },
- { url = "https://files.pythonhosted.org/packages/66/ee/560deadcdde6c2f90200450d5938f63a34b37e27ebff162810f716f6a230/numpy-2.2.6-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:de749064336d37e340f640b05f24e9e3dd678c57318c7289d222a8a2f543e90c", size = 14119362, upload-time = "2025-05-17T21:35:01.241Z" },
- { url = "https://files.pythonhosted.org/packages/3c/65/4baa99f1c53b30adf0acd9a5519078871ddde8d2339dc5a7fde80d9d87da/numpy-2.2.6-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:894b3a42502226a1cac872f840030665f33326fc3dac8e57c607905773cdcde3", size = 5084103, upload-time = "2025-05-17T21:35:10.622Z" },
- { url = "https://files.pythonhosted.org/packages/cc/89/e5a34c071a0570cc40c9a54eb472d113eea6d002e9ae12bb3a8407fb912e/numpy-2.2.6-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:71594f7c51a18e728451bb50cc60a3ce4e6538822731b2933209a1f3614e9282", size = 6625382, upload-time = "2025-05-17T21:35:21.414Z" },
- { url = "https://files.pythonhosted.org/packages/f8/35/8c80729f1ff76b3921d5c9487c7ac3de9b2a103b1cd05e905b3090513510/numpy-2.2.6-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f2618db89be1b4e05f7a1a847a9c1c0abd63e63a1607d892dd54668dd92faf87", size = 14018462, upload-time = "2025-05-17T21:35:42.174Z" },
- { url = "https://files.pythonhosted.org/packages/8c/3d/1e1db36cfd41f895d266b103df00ca5b3cbe965184df824dec5c08c6b803/numpy-2.2.6-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fd83c01228a688733f1ded5201c678f0c53ecc1006ffbc404db9f7a899ac6249", size = 16527618, upload-time = "2025-05-17T21:36:06.711Z" },
- { url = "https://files.pythonhosted.org/packages/61/c6/03ed30992602c85aa3cd95b9070a514f8b3c33e31124694438d88809ae36/numpy-2.2.6-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:37c0ca431f82cd5fa716eca9506aefcabc247fb27ba69c5062a6d3ade8cf8f49", size = 15505511, upload-time = "2025-05-17T21:36:29.965Z" },
- { url = "https://files.pythonhosted.org/packages/b7/25/5761d832a81df431e260719ec45de696414266613c9ee268394dd5ad8236/numpy-2.2.6-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fe27749d33bb772c80dcd84ae7e8df2adc920ae8297400dabec45f0dedb3f6de", size = 18313783, upload-time = "2025-05-17T21:36:56.883Z" },
- { url = "https://files.pythonhosted.org/packages/57/0a/72d5a3527c5ebffcd47bde9162c39fae1f90138c961e5296491ce778e682/numpy-2.2.6-cp312-cp312-win32.whl", hash = "sha256:4eeaae00d789f66c7a25ac5f34b71a7035bb474e679f410e5e1a94deb24cf2d4", size = 6246506, upload-time = "2025-05-17T21:37:07.368Z" },
- { url = "https://files.pythonhosted.org/packages/36/fa/8c9210162ca1b88529ab76b41ba02d433fd54fecaf6feb70ef9f124683f1/numpy-2.2.6-cp312-cp312-win_amd64.whl", hash = "sha256:c1f9540be57940698ed329904db803cf7a402f3fc200bfe599334c9bd84a40b2", size = 12614190, upload-time = "2025-05-17T21:37:26.213Z" },
- { url = "https://files.pythonhosted.org/packages/f9/5c/6657823f4f594f72b5471f1db1ab12e26e890bb2e41897522d134d2a3e81/numpy-2.2.6-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:0811bb762109d9708cca4d0b13c4f67146e3c3b7cf8d34018c722adb2d957c84", size = 20867828, upload-time = "2025-05-17T21:37:56.699Z" },
- { url = "https://files.pythonhosted.org/packages/dc/9e/14520dc3dadf3c803473bd07e9b2bd1b69bc583cb2497b47000fed2fa92f/numpy-2.2.6-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:287cc3162b6f01463ccd86be154f284d0893d2b3ed7292439ea97eafa8170e0b", size = 14143006, upload-time = "2025-05-17T21:38:18.291Z" },
- { url = "https://files.pythonhosted.org/packages/4f/06/7e96c57d90bebdce9918412087fc22ca9851cceaf5567a45c1f404480e9e/numpy-2.2.6-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:f1372f041402e37e5e633e586f62aa53de2eac8d98cbfb822806ce4bbefcb74d", size = 5076765, upload-time = "2025-05-17T21:38:27.319Z" },
- { url = "https://files.pythonhosted.org/packages/73/ed/63d920c23b4289fdac96ddbdd6132e9427790977d5457cd132f18e76eae0/numpy-2.2.6-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:55a4d33fa519660d69614a9fad433be87e5252f4b03850642f88993f7b2ca566", size = 6617736, upload-time = "2025-05-17T21:38:38.141Z" },
- { url = "https://files.pythonhosted.org/packages/85/c5/e19c8f99d83fd377ec8c7e0cf627a8049746da54afc24ef0a0cb73d5dfb5/numpy-2.2.6-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f92729c95468a2f4f15e9bb94c432a9229d0d50de67304399627a943201baa2f", size = 14010719, upload-time = "2025-05-17T21:38:58.433Z" },
- { url = "https://files.pythonhosted.org/packages/19/49/4df9123aafa7b539317bf6d342cb6d227e49f7a35b99c287a6109b13dd93/numpy-2.2.6-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1bc23a79bfabc5d056d106f9befb8d50c31ced2fbc70eedb8155aec74a45798f", size = 16526072, upload-time = "2025-05-17T21:39:22.638Z" },
- { url = "https://files.pythonhosted.org/packages/b2/6c/04b5f47f4f32f7c2b0e7260442a8cbcf8168b0e1a41ff1495da42f42a14f/numpy-2.2.6-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:e3143e4451880bed956e706a3220b4e5cf6172ef05fcc397f6f36a550b1dd868", size = 15503213, upload-time = "2025-05-17T21:39:45.865Z" },
- { url = "https://files.pythonhosted.org/packages/17/0a/5cd92e352c1307640d5b6fec1b2ffb06cd0dabe7d7b8227f97933d378422/numpy-2.2.6-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b4f13750ce79751586ae2eb824ba7e1e8dba64784086c98cdbbcc6a42112ce0d", size = 18316632, upload-time = "2025-05-17T21:40:13.331Z" },
- { url = "https://files.pythonhosted.org/packages/f0/3b/5cba2b1d88760ef86596ad0f3d484b1cbff7c115ae2429678465057c5155/numpy-2.2.6-cp313-cp313-win32.whl", hash = "sha256:5beb72339d9d4fa36522fc63802f469b13cdbe4fdab4a288f0c441b74272ebfd", size = 6244532, upload-time = "2025-05-17T21:43:46.099Z" },
- { url = "https://files.pythonhosted.org/packages/cb/3b/d58c12eafcb298d4e6d0d40216866ab15f59e55d148a5658bb3132311fcf/numpy-2.2.6-cp313-cp313-win_amd64.whl", hash = "sha256:b0544343a702fa80c95ad5d3d608ea3599dd54d4632df855e4c8d24eb6ecfa1c", size = 12610885, upload-time = "2025-05-17T21:44:05.145Z" },
- { url = "https://files.pythonhosted.org/packages/6b/9e/4bf918b818e516322db999ac25d00c75788ddfd2d2ade4fa66f1f38097e1/numpy-2.2.6-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:0bca768cd85ae743b2affdc762d617eddf3bcf8724435498a1e80132d04879e6", size = 20963467, upload-time = "2025-05-17T21:40:44Z" },
- { url = "https://files.pythonhosted.org/packages/61/66/d2de6b291507517ff2e438e13ff7b1e2cdbdb7cb40b3ed475377aece69f9/numpy-2.2.6-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:fc0c5673685c508a142ca65209b4e79ed6740a4ed6b2267dbba90f34b0b3cfda", size = 14225144, upload-time = "2025-05-17T21:41:05.695Z" },
- { url = "https://files.pythonhosted.org/packages/e4/25/480387655407ead912e28ba3a820bc69af9adf13bcbe40b299d454ec011f/numpy-2.2.6-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:5bd4fc3ac8926b3819797a7c0e2631eb889b4118a9898c84f585a54d475b7e40", size = 5200217, upload-time = "2025-05-17T21:41:15.903Z" },
- { url = "https://files.pythonhosted.org/packages/aa/4a/6e313b5108f53dcbf3aca0c0f3e9c92f4c10ce57a0a721851f9785872895/numpy-2.2.6-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:fee4236c876c4e8369388054d02d0e9bb84821feb1a64dd59e137e6511a551f8", size = 6712014, upload-time = "2025-05-17T21:41:27.321Z" },
- { url = "https://files.pythonhosted.org/packages/b7/30/172c2d5c4be71fdf476e9de553443cf8e25feddbe185e0bd88b096915bcc/numpy-2.2.6-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e1dda9c7e08dc141e0247a5b8f49cf05984955246a327d4c48bda16821947b2f", size = 14077935, upload-time = "2025-05-17T21:41:49.738Z" },
- { url = "https://files.pythonhosted.org/packages/12/fb/9e743f8d4e4d3c710902cf87af3512082ae3d43b945d5d16563f26ec251d/numpy-2.2.6-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f447e6acb680fd307f40d3da4852208af94afdfab89cf850986c3ca00562f4fa", size = 16600122, upload-time = "2025-05-17T21:42:14.046Z" },
- { url = "https://files.pythonhosted.org/packages/12/75/ee20da0e58d3a66f204f38916757e01e33a9737d0b22373b3eb5a27358f9/numpy-2.2.6-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:389d771b1623ec92636b0786bc4ae56abafad4a4c513d36a55dce14bd9ce8571", size = 15586143, upload-time = "2025-05-17T21:42:37.464Z" },
- { url = "https://files.pythonhosted.org/packages/76/95/bef5b37f29fc5e739947e9ce5179ad402875633308504a52d188302319c8/numpy-2.2.6-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:8e9ace4a37db23421249ed236fdcdd457d671e25146786dfc96835cd951aa7c1", size = 18385260, upload-time = "2025-05-17T21:43:05.189Z" },
- { url = "https://files.pythonhosted.org/packages/09/04/f2f83279d287407cf36a7a8053a5abe7be3622a4363337338f2585e4afda/numpy-2.2.6-cp313-cp313t-win32.whl", hash = "sha256:038613e9fb8c72b0a41f025a7e4c3f0b7a1b5d768ece4796b674c8f3fe13efff", size = 6377225, upload-time = "2025-05-17T21:43:16.254Z" },
- { url = "https://files.pythonhosted.org/packages/67/0e/35082d13c09c02c011cf21570543d202ad929d961c02a147493cb0c2bdf5/numpy-2.2.6-cp313-cp313t-win_amd64.whl", hash = "sha256:6031dd6dfecc0cf9f668681a37648373bddd6421fff6c66ec1624eed0180ee06", size = 12771374, upload-time = "2025-05-17T21:43:35.479Z" },
- { url = "https://files.pythonhosted.org/packages/9e/3b/d94a75f4dbf1ef5d321523ecac21ef23a3cd2ac8b78ae2aac40873590229/numpy-2.2.6-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:0b605b275d7bd0c640cad4e5d30fa701a8d59302e127e5f79138ad62762c3e3d", size = 21040391, upload-time = "2025-05-17T21:44:35.948Z" },
- { url = "https://files.pythonhosted.org/packages/17/f4/09b2fa1b58f0fb4f7c7963a1649c64c4d315752240377ed74d9cd878f7b5/numpy-2.2.6-pp310-pypy310_pp73-macosx_14_0_x86_64.whl", hash = "sha256:7befc596a7dc9da8a337f79802ee8adb30a552a94f792b9c9d18c840055907db", size = 6786754, upload-time = "2025-05-17T21:44:47.446Z" },
- { url = "https://files.pythonhosted.org/packages/af/30/feba75f143bdc868a1cc3f44ccfa6c4b9ec522b36458e738cd00f67b573f/numpy-2.2.6-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ce47521a4754c8f4593837384bd3424880629f718d87c5d44f8ed763edd63543", size = 16643476, upload-time = "2025-05-17T21:45:11.871Z" },
- { url = "https://files.pythonhosted.org/packages/37/48/ac2a9584402fb6c0cd5b5d1a91dcf176b15760130dd386bbafdbfe3640bf/numpy-2.2.6-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:d042d24c90c41b54fd506da306759e06e568864df8ec17ccc17e9e884634fd00", size = 12812666, upload-time = "2025-05-17T21:45:31.426Z" },
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a9/75/10dd1f8116a8b796cb2c737b674e02d02e80454bda953fa7e65d8c12b016/numpy-2.0.2.tar.gz", hash = "sha256:883c987dee1880e2a864ab0dc9892292582510604156762362d9326444636e78", size = 18902015, upload-time = "2024-08-26T20:19:40.945Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/45/40/2e117be60ec50d98fa08c2f8c48e09b3edea93cfcabd5a9ff6925d54b1c2/numpy-2.0.2-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:df55d490dea7934f330006d0f81e8551ba6010a5bf035a249ef61a94f21c500b", size = 20895803, upload-time = "2024-08-26T20:11:13.916Z" },
+ { url = "https://files.pythonhosted.org/packages/46/92/1b8b8dee833f53cef3e0a3f69b2374467789e0bb7399689582314df02651/numpy-2.0.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8df823f570d9adf0978347d1f926b2a867d5608f434a7cff7f7908c6570dcf5e", size = 13471835, upload-time = "2024-08-26T20:11:34.779Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/19/e2793bde475f1edaea6945be141aef6c8b4c669b90c90a300a8954d08f0a/numpy-2.0.2-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:9a92ae5c14811e390f3767053ff54eaee3bf84576d99a2456391401323f4ec2c", size = 5038499, upload-time = "2024-08-26T20:11:43.902Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/ff/ddf6dac2ff0dd50a7327bcdba45cb0264d0e96bb44d33324853f781a8f3c/numpy-2.0.2-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:a842d573724391493a97a62ebbb8e731f8a5dcc5d285dfc99141ca15a3302d0c", size = 6633497, upload-time = "2024-08-26T20:11:55.09Z" },
+ { url = "https://files.pythonhosted.org/packages/72/21/67f36eac8e2d2cd652a2e69595a54128297cdcb1ff3931cfc87838874bd4/numpy-2.0.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c05e238064fc0610c840d1cf6a13bf63d7e391717d247f1bf0318172e759e692", size = 13621158, upload-time = "2024-08-26T20:12:14.95Z" },
+ { url = "https://files.pythonhosted.org/packages/39/68/e9f1126d757653496dbc096cb429014347a36b228f5a991dae2c6b6cfd40/numpy-2.0.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0123ffdaa88fa4ab64835dcbde75dcdf89c453c922f18dced6e27c90d1d0ec5a", size = 19236173, upload-time = "2024-08-26T20:12:44.049Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/e9/1f5333281e4ebf483ba1c888b1d61ba7e78d7e910fdd8e6499667041cc35/numpy-2.0.2-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:96a55f64139912d61de9137f11bf39a55ec8faec288c75a54f93dfd39f7eb40c", size = 19634174, upload-time = "2024-08-26T20:13:13.634Z" },
+ { url = "https://files.pythonhosted.org/packages/71/af/a469674070c8d8408384e3012e064299f7a2de540738a8e414dcfd639996/numpy-2.0.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ec9852fb39354b5a45a80bdab5ac02dd02b15f44b3804e9f00c556bf24b4bded", size = 14099701, upload-time = "2024-08-26T20:13:34.851Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/3d/08ea9f239d0e0e939b6ca52ad403c84a2bce1bde301a8eb4888c1c1543f1/numpy-2.0.2-cp312-cp312-win32.whl", hash = "sha256:671bec6496f83202ed2d3c8fdc486a8fc86942f2e69ff0e986140339a63bcbe5", size = 6174313, upload-time = "2024-08-26T20:13:45.653Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/b5/4ac39baebf1fdb2e72585c8352c56d063b6126be9fc95bd2bb5ef5770c20/numpy-2.0.2-cp312-cp312-win_amd64.whl", hash = "sha256:cfd41e13fdc257aa5778496b8caa5e856dc4896d4ccf01841daee1d96465467a", size = 15606179, upload-time = "2024-08-26T20:14:08.786Z" },
]
[[package]]
name = "numpy"
-version = "2.4.1"
+version = "2.4.4"
source = { registry = "https://pypi.org/simple" }
resolution-markers = [
- "python_full_version >= '3.14' and sys_platform == 'linux'",
- "python_full_version == '3.13.*' and sys_platform == 'linux'",
- "python_full_version == '3.12.*' and sys_platform == 'linux'",
- "python_full_version >= '3.14' and sys_platform != 'linux'",
- "python_full_version == '3.13.*' and sys_platform != 'linux'",
- "python_full_version == '3.12.*' and sys_platform != 'linux'",
- "python_full_version == '3.11.*' and sys_platform == 'linux'",
- "python_full_version == '3.11.*' and sys_platform != 'linux'",
-]
-sdist = { url = "https://files.pythonhosted.org/packages/24/62/ae72ff66c0f1fd959925b4c11f8c2dea61f47f6acaea75a08512cdfe3fed/numpy-2.4.1.tar.gz", hash = "sha256:a1ceafc5042451a858231588a104093474c6a5c57dcc724841f5c888d237d690", size = 20721320, upload-time = "2026-01-10T06:44:59.619Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/a5/34/2b1bc18424f3ad9af577f6ce23600319968a70575bd7db31ce66731bbef9/numpy-2.4.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:0cce2a669e3c8ba02ee563c7835f92c153cf02edff1ae05e1823f1dde21b16a5", size = 16944563, upload-time = "2026-01-10T06:42:14.615Z" },
- { url = "https://files.pythonhosted.org/packages/2c/57/26e5f97d075aef3794045a6ca9eada6a4ed70eb9a40e7a4a93f9ac80d704/numpy-2.4.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:899d2c18024984814ac7e83f8f49d8e8180e2fbe1b2e252f2e7f1d06bea92425", size = 12645658, upload-time = "2026-01-10T06:42:17.298Z" },
- { url = "https://files.pythonhosted.org/packages/8e/ba/80fc0b1e3cb2fd5c6143f00f42eb67762aa043eaa05ca924ecc3222a7849/numpy-2.4.1-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:09aa8a87e45b55a1c2c205d42e2808849ece5c484b2aab11fecabec3841cafba", size = 5474132, upload-time = "2026-01-10T06:42:19.637Z" },
- { url = "https://files.pythonhosted.org/packages/40/ae/0a5b9a397f0e865ec171187c78d9b57e5588afc439a04ba9cab1ebb2c945/numpy-2.4.1-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:edee228f76ee2dab4579fad6f51f6a305de09d444280109e0f75df247ff21501", size = 6804159, upload-time = "2026-01-10T06:42:21.44Z" },
- { url = "https://files.pythonhosted.org/packages/86/9c/841c15e691c7085caa6fd162f063eff494099c8327aeccd509d1ab1e36ab/numpy-2.4.1-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a92f227dbcdc9e4c3e193add1a189a9909947d4f8504c576f4a732fd0b54240a", size = 14708058, upload-time = "2026-01-10T06:42:23.546Z" },
- { url = "https://files.pythonhosted.org/packages/5d/9d/7862db06743f489e6a502a3b93136d73aea27d97b2cf91504f70a27501d6/numpy-2.4.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:538bf4ec353709c765ff75ae616c34d3c3dca1a68312727e8f2676ea644f8509", size = 16651501, upload-time = "2026-01-10T06:42:25.909Z" },
- { url = "https://files.pythonhosted.org/packages/a6/9c/6fc34ebcbd4015c6e5f0c0ce38264010ce8a546cb6beacb457b84a75dfc8/numpy-2.4.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:ac08c63cb7779b85e9d5318e6c3518b424bc1f364ac4cb2c6136f12e5ff2dccc", size = 16492627, upload-time = "2026-01-10T06:42:28.938Z" },
- { url = "https://files.pythonhosted.org/packages/aa/63/2494a8597502dacda439f61b3c0db4da59928150e62be0e99395c3ad23c5/numpy-2.4.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:4f9c360ecef085e5841c539a9a12b883dff005fbd7ce46722f5e9cef52634d82", size = 18585052, upload-time = "2026-01-10T06:42:31.312Z" },
- { url = "https://files.pythonhosted.org/packages/6a/93/098e1162ae7522fc9b618d6272b77404c4656c72432ecee3abc029aa3de0/numpy-2.4.1-cp311-cp311-win32.whl", hash = "sha256:0f118ce6b972080ba0758c6087c3617b5ba243d806268623dc34216d69099ba0", size = 6236575, upload-time = "2026-01-10T06:42:33.872Z" },
- { url = "https://files.pythonhosted.org/packages/8c/de/f5e79650d23d9e12f38a7bc6b03ea0835b9575494f8ec94c11c6e773b1b1/numpy-2.4.1-cp311-cp311-win_amd64.whl", hash = "sha256:18e14c4d09d55eef39a6ab5b08406e84bc6869c1e34eef45564804f90b7e0574", size = 12604479, upload-time = "2026-01-10T06:42:35.778Z" },
- { url = "https://files.pythonhosted.org/packages/dd/65/e1097a7047cff12ce3369bd003811516b20ba1078dbdec135e1cd7c16c56/numpy-2.4.1-cp311-cp311-win_arm64.whl", hash = "sha256:6461de5113088b399d655d45c3897fa188766415d0f568f175ab071c8873bd73", size = 10578325, upload-time = "2026-01-10T06:42:38.518Z" },
- { url = "https://files.pythonhosted.org/packages/78/7f/ec53e32bf10c813604edf07a3682616bd931d026fcde7b6d13195dfb684a/numpy-2.4.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:d3703409aac693fa82c0aee023a1ae06a6e9d065dba10f5e8e80f642f1e9d0a2", size = 16656888, upload-time = "2026-01-10T06:42:40.913Z" },
- { url = "https://files.pythonhosted.org/packages/b8/e0/1f9585d7dae8f14864e948fd7fa86c6cb72dee2676ca2748e63b1c5acfe0/numpy-2.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:7211b95ca365519d3596a1d8688a95874cc94219d417504d9ecb2df99fa7bfa8", size = 12373956, upload-time = "2026-01-10T06:42:43.091Z" },
- { url = "https://files.pythonhosted.org/packages/8e/43/9762e88909ff2326f5e7536fa8cb3c49fb03a7d92705f23e6e7f553d9cb3/numpy-2.4.1-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:5adf01965456a664fc727ed69cc71848f28d063217c63e1a0e200a118d5eec9a", size = 5202567, upload-time = "2026-01-10T06:42:45.107Z" },
- { url = "https://files.pythonhosted.org/packages/4b/ee/34b7930eb61e79feb4478800a4b95b46566969d837546aa7c034c742ef98/numpy-2.4.1-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:26f0bcd9c79a00e339565b303badc74d3ea2bd6d52191eeca5f95936cad107d0", size = 6549459, upload-time = "2026-01-10T06:42:48.152Z" },
- { url = "https://files.pythonhosted.org/packages/79/e3/5f115fae982565771be994867c89bcd8d7208dbfe9469185497d70de5ddf/numpy-2.4.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0093e85df2960d7e4049664b26afc58b03236e967fb942354deef3208857a04c", size = 14404859, upload-time = "2026-01-10T06:42:49.947Z" },
- { url = "https://files.pythonhosted.org/packages/d9/7d/9c8a781c88933725445a859cac5d01b5871588a15969ee6aeb618ba99eee/numpy-2.4.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7ad270f438cbdd402c364980317fb6b117d9ec5e226fff5b4148dd9aa9fc6e02", size = 16371419, upload-time = "2026-01-10T06:42:52.409Z" },
- { url = "https://files.pythonhosted.org/packages/a6/d2/8aa084818554543f17cf4162c42f162acbd3bb42688aefdba6628a859f77/numpy-2.4.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:297c72b1b98100c2e8f873d5d35fb551fce7040ade83d67dd51d38c8d42a2162", size = 16182131, upload-time = "2026-01-10T06:42:54.694Z" },
- { url = "https://files.pythonhosted.org/packages/60/db/0425216684297c58a8df35f3284ef56ec4a043e6d283f8a59c53562caf1b/numpy-2.4.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:cf6470d91d34bf669f61d515499859fa7a4c2f7c36434afb70e82df7217933f9", size = 18295342, upload-time = "2026-01-10T06:42:56.991Z" },
- { url = "https://files.pythonhosted.org/packages/31/4c/14cb9d86240bd8c386c881bafbe43f001284b7cce3bc01623ac9475da163/numpy-2.4.1-cp312-cp312-win32.whl", hash = "sha256:b6bcf39112e956594b3331316d90c90c90fb961e39696bda97b89462f5f3943f", size = 5959015, upload-time = "2026-01-10T06:42:59.631Z" },
- { url = "https://files.pythonhosted.org/packages/51/cf/52a703dbeb0c65807540d29699fef5fda073434ff61846a564d5c296420f/numpy-2.4.1-cp312-cp312-win_amd64.whl", hash = "sha256:e1a27bb1b2dee45a2a53f5ca6ff2d1a7f135287883a1689e930d44d1ff296c87", size = 12310730, upload-time = "2026-01-10T06:43:01.627Z" },
- { url = "https://files.pythonhosted.org/packages/69/80/a828b2d0ade5e74a9fe0f4e0a17c30fdc26232ad2bc8c9f8b3197cf7cf18/numpy-2.4.1-cp312-cp312-win_arm64.whl", hash = "sha256:0e6e8f9d9ecf95399982019c01223dc130542960a12edfa8edd1122dfa66a8a8", size = 10312166, upload-time = "2026-01-10T06:43:03.673Z" },
- { url = "https://files.pythonhosted.org/packages/04/68/732d4b7811c00775f3bd522a21e8dd5a23f77eb11acdeb663e4a4ebf0ef4/numpy-2.4.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:d797454e37570cfd61143b73b8debd623c3c0952959adb817dd310a483d58a1b", size = 16652495, upload-time = "2026-01-10T06:43:06.283Z" },
- { url = "https://files.pythonhosted.org/packages/20/ca/857722353421a27f1465652b2c66813eeeccea9d76d5f7b74b99f298e60e/numpy-2.4.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:82c55962006156aeef1629b953fd359064aa47e4d82cfc8e67f0918f7da3344f", size = 12368657, upload-time = "2026-01-10T06:43:09.094Z" },
- { url = "https://files.pythonhosted.org/packages/81/0d/2377c917513449cc6240031a79d30eb9a163d32a91e79e0da47c43f2c0c8/numpy-2.4.1-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:71abbea030f2cfc3092a0ff9f8c8fdefdc5e0bf7d9d9c99663538bb0ecdac0b9", size = 5197256, upload-time = "2026-01-10T06:43:13.634Z" },
- { url = "https://files.pythonhosted.org/packages/17/39/569452228de3f5de9064ac75137082c6214be1f5c532016549a7923ab4b5/numpy-2.4.1-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:5b55aa56165b17aaf15520beb9cbd33c9039810e0d9643dd4379e44294c7303e", size = 6545212, upload-time = "2026-01-10T06:43:15.661Z" },
- { url = "https://files.pythonhosted.org/packages/8c/a4/77333f4d1e4dac4395385482557aeecf4826e6ff517e32ca48e1dafbe42a/numpy-2.4.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c0faba4a331195bfa96f93dd9dfaa10b2c7aa8cda3a02b7fd635e588fe821bf5", size = 14402871, upload-time = "2026-01-10T06:43:17.324Z" },
- { url = "https://files.pythonhosted.org/packages/ba/87/d341e519956273b39d8d47969dd1eaa1af740615394fe67d06f1efa68773/numpy-2.4.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d3e3087f53e2b4428766b54932644d148613c5a595150533ae7f00dab2f319a8", size = 16359305, upload-time = "2026-01-10T06:43:19.376Z" },
- { url = "https://files.pythonhosted.org/packages/32/91/789132c6666288eaa20ae8066bb99eba1939362e8f1a534949a215246e97/numpy-2.4.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:49e792ec351315e16da54b543db06ca8a86985ab682602d90c60ef4ff4db2a9c", size = 16181909, upload-time = "2026-01-10T06:43:21.808Z" },
- { url = "https://files.pythonhosted.org/packages/cf/b8/090b8bd27b82a844bb22ff8fdf7935cb1980b48d6e439ae116f53cdc2143/numpy-2.4.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:79e9e06c4c2379db47f3f6fc7a8652e7498251789bf8ff5bd43bf478ef314ca2", size = 18284380, upload-time = "2026-01-10T06:43:23.957Z" },
- { url = "https://files.pythonhosted.org/packages/67/78/722b62bd31842ff029412271556a1a27a98f45359dea78b1548a3a9996aa/numpy-2.4.1-cp313-cp313-win32.whl", hash = "sha256:3d1a100e48cb266090a031397863ff8a30050ceefd798f686ff92c67a486753d", size = 5957089, upload-time = "2026-01-10T06:43:27.535Z" },
- { url = "https://files.pythonhosted.org/packages/da/a6/cf32198b0b6e18d4fbfa9a21a992a7fca535b9bb2b0cdd217d4a3445b5ca/numpy-2.4.1-cp313-cp313-win_amd64.whl", hash = "sha256:92a0e65272fd60bfa0d9278e0484c2f52fe03b97aedc02b357f33fe752c52ffb", size = 12307230, upload-time = "2026-01-10T06:43:29.298Z" },
- { url = "https://files.pythonhosted.org/packages/44/6c/534d692bfb7d0afe30611320c5fb713659dcb5104d7cc182aff2aea092f5/numpy-2.4.1-cp313-cp313-win_arm64.whl", hash = "sha256:20d4649c773f66cc2fc36f663e091f57c3b7655f936a4c681b4250855d1da8f5", size = 10313125, upload-time = "2026-01-10T06:43:31.782Z" },
- { url = "https://files.pythonhosted.org/packages/da/a1/354583ac5c4caa566de6ddfbc42744409b515039e085fab6e0ff942e0df5/numpy-2.4.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:f93bc6892fe7b0663e5ffa83b61aab510aacffd58c16e012bb9352d489d90cb7", size = 12496156, upload-time = "2026-01-10T06:43:34.237Z" },
- { url = "https://files.pythonhosted.org/packages/51/b0/42807c6e8cce58c00127b1dc24d365305189991f2a7917aa694a109c8d7d/numpy-2.4.1-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:178de8f87948163d98a4c9ab5bee4ce6519ca918926ec8df195af582de28544d", size = 5324663, upload-time = "2026-01-10T06:43:36.211Z" },
- { url = "https://files.pythonhosted.org/packages/fe/55/7a621694010d92375ed82f312b2f28017694ed784775269115323e37f5e2/numpy-2.4.1-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:98b35775e03ab7f868908b524fc0a84d38932d8daf7b7e1c3c3a1b6c7a2c9f15", size = 6645224, upload-time = "2026-01-10T06:43:37.884Z" },
- { url = "https://files.pythonhosted.org/packages/50/96/9fa8635ed9d7c847d87e30c834f7109fac5e88549d79ef3324ab5c20919f/numpy-2.4.1-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:941c2a93313d030f219f3a71fd3d91a728b82979a5e8034eb2e60d394a2b83f9", size = 14462352, upload-time = "2026-01-10T06:43:39.479Z" },
- { url = "https://files.pythonhosted.org/packages/03/d1/8cf62d8bb2062da4fb82dd5d49e47c923f9c0738032f054e0a75342faba7/numpy-2.4.1-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:529050522e983e00a6c1c6b67411083630de8b57f65e853d7b03d9281b8694d2", size = 16407279, upload-time = "2026-01-10T06:43:41.93Z" },
- { url = "https://files.pythonhosted.org/packages/86/1c/95c86e17c6b0b31ce6ef219da00f71113b220bcb14938c8d9a05cee0ff53/numpy-2.4.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:2302dc0224c1cbc49bb94f7064f3f923a971bfae45c33870dcbff63a2a550505", size = 16248316, upload-time = "2026-01-10T06:43:44.121Z" },
- { url = "https://files.pythonhosted.org/packages/30/b4/e7f5ff8697274c9d0fa82398b6a372a27e5cef069b37df6355ccb1f1db1a/numpy-2.4.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:9171a42fcad32dcf3fa86f0a4faa5e9f8facefdb276f54b8b390d90447cff4e2", size = 18329884, upload-time = "2026-01-10T06:43:46.613Z" },
- { url = "https://files.pythonhosted.org/packages/37/a4/b073f3e9d77f9aec8debe8ca7f9f6a09e888ad1ba7488f0c3b36a94c03ac/numpy-2.4.1-cp313-cp313t-win32.whl", hash = "sha256:382ad67d99ef49024f11d1ce5dcb5ad8432446e4246a4b014418ba3a1175a1f4", size = 6081138, upload-time = "2026-01-10T06:43:48.854Z" },
- { url = "https://files.pythonhosted.org/packages/16/16/af42337b53844e67752a092481ab869c0523bc95c4e5c98e4dac4e9581ac/numpy-2.4.1-cp313-cp313t-win_amd64.whl", hash = "sha256:62fea415f83ad8fdb6c20840578e5fbaf5ddd65e0ec6c3c47eda0f69da172510", size = 12447478, upload-time = "2026-01-10T06:43:50.476Z" },
- { url = "https://files.pythonhosted.org/packages/6c/f8/fa85b2eac68ec631d0b631abc448552cb17d39afd17ec53dcbcc3537681a/numpy-2.4.1-cp313-cp313t-win_arm64.whl", hash = "sha256:a7870e8c5fc11aef57d6fea4b4085e537a3a60ad2cdd14322ed531fdca68d261", size = 10382981, upload-time = "2026-01-10T06:43:52.575Z" },
- { url = "https://files.pythonhosted.org/packages/1b/a7/ef08d25698e0e4b4efbad8d55251d20fe2a15f6d9aa7c9b30cd03c165e6f/numpy-2.4.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:3869ea1ee1a1edc16c29bbe3a2f2a4e515cc3a44d43903ad41e0cacdbaf733dc", size = 16652046, upload-time = "2026-01-10T06:43:54.797Z" },
- { url = "https://files.pythonhosted.org/packages/8f/39/e378b3e3ca13477e5ac70293ec027c438d1927f18637e396fe90b1addd72/numpy-2.4.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:e867df947d427cdd7a60e3e271729090b0f0df80f5f10ab7dd436f40811699c3", size = 12378858, upload-time = "2026-01-10T06:43:57.099Z" },
- { url = "https://files.pythonhosted.org/packages/c3/74/7ec6154f0006910ed1fdbb7591cf4432307033102b8a22041599935f8969/numpy-2.4.1-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:e3bd2cb07841166420d2fa7146c96ce00cb3410664cbc1a6be028e456c4ee220", size = 5207417, upload-time = "2026-01-10T06:43:59.037Z" },
- { url = "https://files.pythonhosted.org/packages/f7/b7/053ac11820d84e42f8feea5cb81cc4fcd1091499b45b1ed8c7415b1bf831/numpy-2.4.1-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:f0a90aba7d521e6954670550e561a4cb925713bd944445dbe9e729b71f6cabee", size = 6542643, upload-time = "2026-01-10T06:44:01.852Z" },
- { url = "https://files.pythonhosted.org/packages/c0/c4/2e7908915c0e32ca636b92e4e4a3bdec4cb1e7eb0f8aedf1ed3c68a0d8cd/numpy-2.4.1-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5d558123217a83b2d1ba316b986e9248a1ed1971ad495963d555ccd75dcb1556", size = 14418963, upload-time = "2026-01-10T06:44:04.047Z" },
- { url = "https://files.pythonhosted.org/packages/eb/c0/3ed5083d94e7ffd7c404e54619c088e11f2e1939a9544f5397f4adb1b8ba/numpy-2.4.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2f44de05659b67d20499cbc96d49f2650769afcb398b79b324bb6e297bfe3844", size = 16363811, upload-time = "2026-01-10T06:44:06.207Z" },
- { url = "https://files.pythonhosted.org/packages/0e/68/42b66f1852bf525050a67315a4fb94586ab7e9eaa541b1bef530fab0c5dd/numpy-2.4.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:69e7419c9012c4aaf695109564e3387f1259f001b4326dfa55907b098af082d3", size = 16197643, upload-time = "2026-01-10T06:44:08.33Z" },
- { url = "https://files.pythonhosted.org/packages/d2/40/e8714fc933d85f82c6bfc7b998a0649ad9769a32f3494ba86598aaf18a48/numpy-2.4.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:2ffd257026eb1b34352e749d7cc1678b5eeec3e329ad8c9965a797e08ccba205", size = 18289601, upload-time = "2026-01-10T06:44:10.841Z" },
- { url = "https://files.pythonhosted.org/packages/80/9a/0d44b468cad50315127e884802351723daca7cf1c98d102929468c81d439/numpy-2.4.1-cp314-cp314-win32.whl", hash = "sha256:727c6c3275ddefa0dc078524a85e064c057b4f4e71ca5ca29a19163c607be745", size = 6005722, upload-time = "2026-01-10T06:44:13.332Z" },
- { url = "https://files.pythonhosted.org/packages/7e/bb/c6513edcce5a831810e2dddc0d3452ce84d208af92405a0c2e58fd8e7881/numpy-2.4.1-cp314-cp314-win_amd64.whl", hash = "sha256:7d5d7999df434a038d75a748275cd6c0094b0ecdb0837342b332a82defc4dc4d", size = 12438590, upload-time = "2026-01-10T06:44:15.006Z" },
- { url = "https://files.pythonhosted.org/packages/e9/da/a598d5cb260780cf4d255102deba35c1d072dc028c4547832f45dd3323a8/numpy-2.4.1-cp314-cp314-win_arm64.whl", hash = "sha256:ce9ce141a505053b3c7bce3216071f3bf5c182b8b28930f14cd24d43932cd2df", size = 10596180, upload-time = "2026-01-10T06:44:17.386Z" },
- { url = "https://files.pythonhosted.org/packages/de/bc/ea3f2c96fcb382311827231f911723aeff596364eb6e1b6d1d91128aa29b/numpy-2.4.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:4e53170557d37ae404bf8d542ca5b7c629d6efa1117dac6a83e394142ea0a43f", size = 12498774, upload-time = "2026-01-10T06:44:19.467Z" },
- { url = "https://files.pythonhosted.org/packages/aa/ab/ef9d939fe4a812648c7a712610b2ca6140b0853c5efea361301006c02ae5/numpy-2.4.1-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:a73044b752f5d34d4232f25f18160a1cc418ea4507f5f11e299d8ac36875f8a0", size = 5327274, upload-time = "2026-01-10T06:44:23.189Z" },
- { url = "https://files.pythonhosted.org/packages/bd/31/d381368e2a95c3b08b8cf7faac6004849e960f4a042d920337f71cef0cae/numpy-2.4.1-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:fb1461c99de4d040666ca0444057b06541e5642f800b71c56e6ea92d6a853a0c", size = 6648306, upload-time = "2026-01-10T06:44:25.012Z" },
- { url = "https://files.pythonhosted.org/packages/c8/e5/0989b44ade47430be6323d05c23207636d67d7362a1796ccbccac6773dd2/numpy-2.4.1-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:423797bdab2eeefbe608d7c1ec7b2b4fd3c58d51460f1ee26c7500a1d9c9ee93", size = 14464653, upload-time = "2026-01-10T06:44:26.706Z" },
- { url = "https://files.pythonhosted.org/packages/10/a7/cfbe475c35371cae1358e61f20c5f075badc18c4797ab4354140e1d283cf/numpy-2.4.1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:52b5f61bdb323b566b528899cc7db2ba5d1015bda7ea811a8bcf3c89c331fa42", size = 16405144, upload-time = "2026-01-10T06:44:29.378Z" },
- { url = "https://files.pythonhosted.org/packages/f8/a3/0c63fe66b534888fa5177cc7cef061541064dbe2b4b60dcc60ffaf0d2157/numpy-2.4.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:42d7dd5fa36d16d52a84f821eb96031836fd405ee6955dd732f2023724d0aa01", size = 16247425, upload-time = "2026-01-10T06:44:31.721Z" },
- { url = "https://files.pythonhosted.org/packages/6b/2b/55d980cfa2c93bd40ff4c290bf824d792bd41d2fe3487b07707559071760/numpy-2.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:e7b6b5e28bbd47b7532698e5db2fe1db693d84b58c254e4389d99a27bb9b8f6b", size = 18330053, upload-time = "2026-01-10T06:44:34.617Z" },
- { url = "https://files.pythonhosted.org/packages/23/12/8b5fc6b9c487a09a7957188e0943c9ff08432c65e34567cabc1623b03a51/numpy-2.4.1-cp314-cp314t-win32.whl", hash = "sha256:5de60946f14ebe15e713a6f22850c2372fa72f4ff9a432ab44aa90edcadaa65a", size = 6152482, upload-time = "2026-01-10T06:44:36.798Z" },
- { url = "https://files.pythonhosted.org/packages/00/a5/9f8ca5856b8940492fc24fbe13c1bc34d65ddf4079097cf9e53164d094e1/numpy-2.4.1-cp314-cp314t-win_amd64.whl", hash = "sha256:8f085da926c0d491ffff3096f91078cc97ea67e7e6b65e490bc8dcda65663be2", size = 12627117, upload-time = "2026-01-10T06:44:38.828Z" },
- { url = "https://files.pythonhosted.org/packages/ad/0d/eca3d962f9eef265f01a8e0d20085c6dd1f443cbffc11b6dede81fd82356/numpy-2.4.1-cp314-cp314t-win_arm64.whl", hash = "sha256:6436cffb4f2bf26c974344439439c95e152c9a527013f26b3577be6c2ca64295", size = 10667121, upload-time = "2026-01-10T06:44:41.644Z" },
- { url = "https://files.pythonhosted.org/packages/1e/48/d86f97919e79314a1cdee4c832178763e6e98e623e123d0bada19e92c15a/numpy-2.4.1-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:8ad35f20be147a204e28b6a0575fbf3540c5e5f802634d4258d55b1ff5facce1", size = 16822202, upload-time = "2026-01-10T06:44:43.738Z" },
- { url = "https://files.pythonhosted.org/packages/51/e9/1e62a7f77e0f37dcfb0ad6a9744e65df00242b6ea37dfafb55debcbf5b55/numpy-2.4.1-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:8097529164c0f3e32bb89412a0905d9100bf434d9692d9fc275e18dcf53c9344", size = 12569985, upload-time = "2026-01-10T06:44:45.945Z" },
- { url = "https://files.pythonhosted.org/packages/c7/7e/914d54f0c801342306fdcdce3e994a56476f1b818c46c47fc21ae968088c/numpy-2.4.1-pp311-pypy311_pp73-macosx_14_0_arm64.whl", hash = "sha256:ea66d2b41ca4a1630aae5507ee0a71647d3124d1741980138aa8f28f44dac36e", size = 5398484, upload-time = "2026-01-10T06:44:48.012Z" },
- { url = "https://files.pythonhosted.org/packages/1c/d8/9570b68584e293a33474e7b5a77ca404f1dcc655e40050a600dee81d27fb/numpy-2.4.1-pp311-pypy311_pp73-macosx_14_0_x86_64.whl", hash = "sha256:d3f8f0df9f4b8be57b3bf74a1d087fec68f927a2fab68231fdb442bf2c12e426", size = 6713216, upload-time = "2026-01-10T06:44:49.725Z" },
- { url = "https://files.pythonhosted.org/packages/33/9b/9dd6e2db8d49eb24f86acaaa5258e5f4c8ed38209a4ee9de2d1a0ca25045/numpy-2.4.1-pp311-pypy311_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2023ef86243690c2791fd6353e5b4848eedaa88ca8a2d129f462049f6d484696", size = 14538937, upload-time = "2026-01-10T06:44:51.498Z" },
- { url = "https://files.pythonhosted.org/packages/53/87/d5bd995b0f798a37105b876350d346eea5838bd8f77ea3d7a48392f3812b/numpy-2.4.1-pp311-pypy311_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8361ea4220d763e54cff2fbe7d8c93526b744f7cd9ddab47afeff7e14e8503be", size = 16479830, upload-time = "2026-01-10T06:44:53.931Z" },
- { url = "https://files.pythonhosted.org/packages/5b/c7/b801bf98514b6ae6475e941ac05c58e6411dd863ea92916bfd6d510b08c1/numpy-2.4.1-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:4f1b68ff47680c2925f8063402a693ede215f0257f02596b1318ecdfb1d79e33", size = 12492579, upload-time = "2026-01-10T06:44:57.094Z" },
-]
-
-[[package]]
-name = "nv-grouped-gemm"
-version = "1.1.4.post8"
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d7/9f/b8cef5bffa569759033adda9481211426f12f53299629b410340795c2514/numpy-2.4.4.tar.gz", hash = "sha256:2d390634c5182175533585cc89f3608a4682ccb173cc9bb940b2881c8d6f8fa0", size = 20731587, upload-time = "2026-03-29T13:22:01.298Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/28/05/32396bec30fb2263770ee910142f49c1476d08e8ad41abf8403806b520ce/numpy-2.4.4-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:15716cfef24d3a9762e3acdf87e27f58dc823d1348f765bbea6bef8c639bfa1b", size = 16689272, upload-time = "2026-03-29T13:18:49.223Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/f3/a983d28637bfcd763a9c7aafdb6d5c0ebf3d487d1e1459ffdb57e2f01117/numpy-2.4.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:23cbfd4c17357c81021f21540da84ee282b9c8fba38a03b7b9d09ba6b951421e", size = 14699573, upload-time = "2026-03-29T13:18:52.629Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/fd/e5ecca1e78c05106d98028114f5c00d3eddb41207686b2b7de3e477b0e22/numpy-2.4.4-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:8b3b60bb7cba2c8c81837661c488637eee696f59a877788a396d33150c35d842", size = 5204782, upload-time = "2026-03-29T13:18:55.579Z" },
+ { url = "https://files.pythonhosted.org/packages/de/2f/702a4594413c1a8632092beae8aba00f1d67947389369b3777aed783fdca/numpy-2.4.4-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:e4a010c27ff6f210ff4c6ef34394cd61470d01014439b192ec22552ee867f2a8", size = 6552038, upload-time = "2026-03-29T13:18:57.769Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/37/eed308a8f56cba4d1fdf467a4fc67ef4ff4bf1c888f5fc980481890104b1/numpy-2.4.4-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f9e75681b59ddaa5e659898085ae0eaea229d054f2ac0c7e563a62205a700121", size = 15670666, upload-time = "2026-03-29T13:19:00.341Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/0d/0e3ecece05b7a7e87ab9fb587855548da437a061326fff64a223b6dcb78a/numpy-2.4.4-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:81f4a14bee47aec54f883e0cad2d73986640c1590eb9bfaaba7ad17394481e6e", size = 16645480, upload-time = "2026-03-29T13:19:03.63Z" },
+ { url = "https://files.pythonhosted.org/packages/34/49/f2312c154b82a286758ee2f1743336d50651f8b5195db18cdb63675ff649/numpy-2.4.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:62d6b0f03b694173f9fcb1fb317f7222fd0b0b103e784c6549f5e53a27718c44", size = 17020036, upload-time = "2026-03-29T13:19:07.428Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/e9/736d17bd77f1b0ec4f9901aaec129c00d59f5d84d5e79bba540ef12c2330/numpy-2.4.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fbc356aae7adf9e6336d336b9c8111d390a05df88f1805573ebb0807bd06fd1d", size = 18368643, upload-time = "2026-03-29T13:19:10.775Z" },
+ { url = "https://files.pythonhosted.org/packages/63/f6/d417977c5f519b17c8a5c3bc9e8304b0908b0e21136fe43bf628a1343914/numpy-2.4.4-cp312-cp312-win32.whl", hash = "sha256:0d35aea54ad1d420c812bfa0385c71cd7cc5bcf7c65fed95fc2cd02fe8c79827", size = 5961117, upload-time = "2026-03-29T13:19:13.464Z" },
+ { url = "https://files.pythonhosted.org/packages/2d/5b/e1deebf88ff431b01b7406ca3583ab2bbb90972bbe1c568732e49c844f7e/numpy-2.4.4-cp312-cp312-win_amd64.whl", hash = "sha256:b5f0362dc928a6ecd9db58868fca5e48485205e3855957bdedea308f8672ea4a", size = 12320584, upload-time = "2026-03-29T13:19:16.155Z" },
+ { url = "https://files.pythonhosted.org/packages/58/89/e4e856ac82a68c3ed64486a544977d0e7bdd18b8da75b78a577ca31c4395/numpy-2.4.4-cp312-cp312-win_arm64.whl", hash = "sha256:846300f379b5b12cc769334464656bc882e0735d27d9726568bc932fdc49d5ec", size = 10221450, upload-time = "2026-03-29T13:19:18.994Z" },
+ { url = "https://files.pythonhosted.org/packages/14/1d/d0a583ce4fefcc3308806a749a536c201ed6b5ad6e1322e227ee4848979d/numpy-2.4.4-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:08f2e31ed5e6f04b118e49821397f12767934cfdd12a1ce86a058f91e004ee50", size = 16684933, upload-time = "2026-03-29T13:19:22.47Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/62/2b7a48fbb745d344742c0277f01286dead15f3f68e4f359fbfcf7b48f70f/numpy-2.4.4-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e823b8b6edc81e747526f70f71a9c0a07ac4e7ad13020aa736bb7c9d67196115", size = 14694532, upload-time = "2026-03-29T13:19:25.581Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/87/499737bfba066b4a3bebff24a8f1c5b2dee410b209bc6668c9be692580f0/numpy-2.4.4-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:4a19d9dba1a76618dd86b164d608566f393f8ec6ac7c44f0cc879011c45e65af", size = 5199661, upload-time = "2026-03-29T13:19:28.31Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/da/464d551604320d1491bc345efed99b4b7034143a85787aab78d5691d5a0e/numpy-2.4.4-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:d2a8490669bfe99a233298348acc2d824d496dee0e66e31b66a6022c2ad74a5c", size = 6547539, upload-time = "2026-03-29T13:19:30.97Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/90/8d23e3b0dafd024bf31bdec225b3bb5c2dbfa6912f8a53b8659f21216cbf/numpy-2.4.4-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:45dbed2ab436a9e826e302fcdcbe9133f9b0006e5af7168afb8963a6520da103", size = 15668806, upload-time = "2026-03-29T13:19:33.887Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/73/a9d864e42a01896bb5974475438f16086be9ba1f0d19d0bb7a07427c4a8b/numpy-2.4.4-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c901b15172510173f5cb310eae652908340f8dede90fff9e3bf6c0d8dfd92f83", size = 16632682, upload-time = "2026-03-29T13:19:37.336Z" },
+ { url = "https://files.pythonhosted.org/packages/34/fb/14570d65c3bde4e202a031210475ae9cde9b7686a2e7dc97ee67d2833b35/numpy-2.4.4-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:99d838547ace2c4aace6c4f76e879ddfe02bb58a80c1549928477862b7a6d6ed", size = 17019810, upload-time = "2026-03-29T13:19:40.963Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/77/2ba9d87081fd41f6d640c83f26fb7351e536b7ce6dd9061b6af5904e8e46/numpy-2.4.4-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:0aec54fd785890ecca25a6003fd9a5aed47ad607bbac5cd64f836ad8666f4959", size = 18357394, upload-time = "2026-03-29T13:19:44.859Z" },
+ { url = "https://files.pythonhosted.org/packages/a2/23/52666c9a41708b0853fa3b1a12c90da38c507a3074883823126d4e9d5b30/numpy-2.4.4-cp313-cp313-win32.whl", hash = "sha256:07077278157d02f65c43b1b26a3886bce886f95d20aabd11f87932750dfb14ed", size = 5959556, upload-time = "2026-03-29T13:19:47.661Z" },
+ { url = "https://files.pythonhosted.org/packages/57/fb/48649b4971cde70d817cf97a2a2fdc0b4d8308569f1dd2f2611959d2e0cf/numpy-2.4.4-cp313-cp313-win_amd64.whl", hash = "sha256:5c70f1cc1c4efbe316a572e2d8b9b9cc44e89b95f79ca3331553fbb63716e2bf", size = 12317311, upload-time = "2026-03-29T13:19:50.67Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/d8/11490cddd564eb4de97b4579ef6bfe6a736cc07e94c1598590ae25415e01/numpy-2.4.4-cp313-cp313-win_arm64.whl", hash = "sha256:ef4059d6e5152fa1a39f888e344c73fdc926e1b2dd58c771d67b0acfbf2aa67d", size = 10222060, upload-time = "2026-03-29T13:19:54.229Z" },
+ { url = "https://files.pythonhosted.org/packages/99/5d/dab4339177a905aad3e2221c915b35202f1ec30d750dd2e5e9d9a72b804b/numpy-2.4.4-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:4bbc7f303d125971f60ec0aaad5e12c62d0d2c925f0ab1273debd0e4ba37aba5", size = 14822302, upload-time = "2026-03-29T13:19:57.585Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/e4/0564a65e7d3d97562ed6f9b0fd0fb0a6f559ee444092f105938b50043876/numpy-2.4.4-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:4d6d57903571f86180eb98f8f0c839fa9ebbfb031356d87f1361be91e433f5b7", size = 5327407, upload-time = "2026-03-29T13:20:00.601Z" },
+ { url = "https://files.pythonhosted.org/packages/29/8d/35a3a6ce5ad371afa58b4700f1c820f8f279948cca32524e0a695b0ded83/numpy-2.4.4-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:4636de7fd195197b7535f231b5de9e4b36d2c440b6e566d2e4e4746e6af0ca93", size = 6647631, upload-time = "2026-03-29T13:20:02.855Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/da/477731acbd5a58a946c736edfdabb2ac5b34c3d08d1ba1a7b437fa0884df/numpy-2.4.4-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ad2e2ef14e0b04e544ea2fa0a36463f847f113d314aa02e5b402fdf910ef309e", size = 15727691, upload-time = "2026-03-29T13:20:06.004Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/db/338535d9b152beabeb511579598418ba0212ce77cf9718edd70262cc4370/numpy-2.4.4-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5a285b3b96f951841799528cd1f4f01cd70e7e0204b4abebac9463eecfcf2a40", size = 16681241, upload-time = "2026-03-29T13:20:09.417Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/a9/ad248e8f58beb7a0219b413c9c7d8151c5d285f7f946c3e26695bdbbe2df/numpy-2.4.4-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:f8474c4241bc18b750be2abea9d7a9ec84f46ef861dbacf86a4f6e043401f79e", size = 17085767, upload-time = "2026-03-29T13:20:13.126Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/1a/3b88ccd3694681356f70da841630e4725a7264d6a885c8d442a697e1146b/numpy-2.4.4-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:4e874c976154687c1f71715b034739b45c7711bec81db01914770373d125e392", size = 18403169, upload-time = "2026-03-29T13:20:17.096Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/c9/fcfd5d0639222c6eac7f304829b04892ef51c96a75d479214d77e3ce6e33/numpy-2.4.4-cp313-cp313t-win32.whl", hash = "sha256:9c585a1790d5436a5374bac930dad6ed244c046ed91b2b2a3634eb2971d21008", size = 6083477, upload-time = "2026-03-29T13:20:20.195Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/e3/3938a61d1c538aaec8ed6fd6323f57b0c2d2d2219512434c5c878db76553/numpy-2.4.4-cp313-cp313t-win_amd64.whl", hash = "sha256:93e15038125dc1e5345d9b5b68aa7f996ec33b98118d18c6ca0d0b7d6198b7e8", size = 12457487, upload-time = "2026-03-29T13:20:22.946Z" },
+ { url = "https://files.pythonhosted.org/packages/97/6a/7e345032cc60501721ef94e0e30b60f6b0bd601f9174ebd36389a2b86d40/numpy-2.4.4-cp313-cp313t-win_arm64.whl", hash = "sha256:0dfd3f9d3adbe2920b68b5cd3d51444e13a10792ec7154cd0a2f6e74d4ab3233", size = 10292002, upload-time = "2026-03-29T13:20:25.909Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/06/c54062f85f673dd5c04cbe2f14c3acb8c8b95e3384869bb8cc9bff8cb9df/numpy-2.4.4-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:f169b9a863d34f5d11b8698ead99febeaa17a13ca044961aa8e2662a6c7766a0", size = 16684353, upload-time = "2026-03-29T13:20:29.504Z" },
+ { url = "https://files.pythonhosted.org/packages/4c/39/8a320264a84404c74cc7e79715de85d6130fa07a0898f67fb5cd5bd79908/numpy-2.4.4-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:2483e4584a1cb3092da4470b38866634bafb223cbcd551ee047633fd2584599a", size = 14704914, upload-time = "2026-03-29T13:20:33.547Z" },
+ { url = "https://files.pythonhosted.org/packages/91/fb/287076b2614e1d1044235f50f03748f31fa287e3dbe6abeb35cdfa351eca/numpy-2.4.4-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:2d19e6e2095506d1736b7d80595e0f252d76b89f5e715c35e06e937679ea7d7a", size = 5210005, upload-time = "2026-03-29T13:20:36.45Z" },
+ { url = "https://files.pythonhosted.org/packages/63/eb/fcc338595309910de6ecabfcef2419a9ce24399680bfb149421fa2df1280/numpy-2.4.4-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:6a246d5914aa1c820c9443ddcee9c02bec3e203b0c080349533fae17727dfd1b", size = 6544974, upload-time = "2026-03-29T13:20:39.014Z" },
+ { url = "https://files.pythonhosted.org/packages/44/5d/e7e9044032a716cdfaa3fba27a8e874bf1c5f1912a1ddd4ed071bf8a14a6/numpy-2.4.4-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:989824e9faf85f96ec9c7761cd8d29c531ad857bfa1daa930cba85baaecf1a9a", size = 15684591, upload-time = "2026-03-29T13:20:42.146Z" },
+ { url = "https://files.pythonhosted.org/packages/98/7c/21252050676612625449b4807d6b695b9ce8a7c9e1c197ee6216c8a65c7c/numpy-2.4.4-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:27a8d92cd10f1382a67d7cf4db7ce18341b66438bdd9f691d7b0e48d104c2a9d", size = 16637700, upload-time = "2026-03-29T13:20:46.204Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/29/56d2bbef9465db24ef25393383d761a1af4f446a1df9b8cded4fe3a5a5d7/numpy-2.4.4-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:e44319a2953c738205bf3354537979eaa3998ed673395b964c1176083dd46252", size = 17035781, upload-time = "2026-03-29T13:20:50.242Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/2b/a35a6d7589d21f44cea7d0a98de5ddcbb3d421b2622a5c96b1edf18707c3/numpy-2.4.4-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:e892aff75639bbef0d2a2cfd55535510df26ff92f63c92cd84ef8d4ba5a5557f", size = 18362959, upload-time = "2026-03-29T13:20:54.019Z" },
+ { url = "https://files.pythonhosted.org/packages/64/c9/d52ec581f2390e0f5f85cbfd80fb83d965fc15e9f0e1aec2195faa142cde/numpy-2.4.4-cp314-cp314-win32.whl", hash = "sha256:1378871da56ca8943c2ba674530924bb8ca40cd228358a3b5f302ad60cf875fc", size = 6008768, upload-time = "2026-03-29T13:20:56.912Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/22/4cc31a62a6c7b74a8730e31a4274c5dc80e005751e277a2ce38e675e4923/numpy-2.4.4-cp314-cp314-win_amd64.whl", hash = "sha256:715d1c092715954784bc79e1174fc2a90093dc4dc84ea15eb14dad8abdcdeb74", size = 12449181, upload-time = "2026-03-29T13:20:59.548Z" },
+ { url = "https://files.pythonhosted.org/packages/70/2e/14cda6f4d8e396c612d1bf97f22958e92148801d7e4f110cabebdc0eef4b/numpy-2.4.4-cp314-cp314-win_arm64.whl", hash = "sha256:2c194dd721e54ecad9ad387c1d35e63dce5c4450c6dc7dd5611283dda239aabb", size = 10496035, upload-time = "2026-03-29T13:21:02.524Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/e8/8fed8c8d848d7ecea092dc3469643f9d10bc3a134a815a3b033da1d2039b/numpy-2.4.4-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:2aa0613a5177c264ff5921051a5719d20095ea586ca88cc802c5c218d1c67d3e", size = 14824958, upload-time = "2026-03-29T13:21:05.671Z" },
+ { url = "https://files.pythonhosted.org/packages/05/1a/d8007a5138c179c2bf33ef44503e83d70434d2642877ee8fbb230e7c0548/numpy-2.4.4-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:42c16925aa5a02362f986765f9ebabf20de75cdefdca827d14315c568dcab113", size = 5330020, upload-time = "2026-03-29T13:21:08.635Z" },
+ { url = "https://files.pythonhosted.org/packages/99/64/ffb99ac6ae93faf117bcbd5c7ba48a7f45364a33e8e458545d3633615dda/numpy-2.4.4-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:874f200b2a981c647340f841730fc3a2b54c9d940566a3c4149099591e2c4c3d", size = 6650758, upload-time = "2026-03-29T13:21:10.949Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/6e/795cc078b78a384052e73b2f6281ff7a700e9bf53bcce2ee579d4f6dd879/numpy-2.4.4-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c9b39d38a9bd2ae1becd7eac1303d031c5c110ad31f2b319c6e7d98b135c934d", size = 15729948, upload-time = "2026-03-29T13:21:14.047Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/86/2acbda8cc2af5f3d7bfc791192863b9e3e19674da7b5e533fded124d1299/numpy-2.4.4-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b268594bccac7d7cf5844c7732e3f20c50921d94e36d7ec9b79e9857694b1b2f", size = 16679325, upload-time = "2026-03-29T13:21:17.561Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/59/cafd83018f4aa55e0ac6fa92aa066c0a1877b77a615ceff1711c260ffae8/numpy-2.4.4-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:ac6b31e35612a26483e20750126d30d0941f949426974cace8e6b5c58a3657b0", size = 17084883, upload-time = "2026-03-29T13:21:21.106Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/85/a42548db84e65ece46ab2caea3d3f78b416a47af387fcbb47ec28e660dc2/numpy-2.4.4-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:8e3ed142f2728df44263aaf5fb1f5b0b99f4070c553a0d7f033be65338329150", size = 18403474, upload-time = "2026-03-29T13:21:24.828Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/ad/483d9e262f4b831000062e5d8a45e342166ec8aaa1195264982bca267e62/numpy-2.4.4-cp314-cp314t-win32.whl", hash = "sha256:dddbbd259598d7240b18c9d87c56a9d2fb3b02fe266f49a7c101532e78c1d871", size = 6155500, upload-time = "2026-03-29T13:21:28.205Z" },
+ { url = "https://files.pythonhosted.org/packages/c7/03/2fc4e14c7bd4ff2964b74ba90ecb8552540b6315f201df70f137faa5c589/numpy-2.4.4-cp314-cp314t-win_amd64.whl", hash = "sha256:a7164afb23be6e37ad90b2f10426149fd75aee07ca55653d2aa41e66c4ef697e", size = 12637755, upload-time = "2026-03-29T13:21:31.107Z" },
+ { url = "https://files.pythonhosted.org/packages/58/78/548fb8e07b1a341746bfbecb32f2c268470f45fa028aacdbd10d9bc73aab/numpy-2.4.4-cp314-cp314t-win_arm64.whl", hash = "sha256:ba203255017337d39f89bdd58417f03c4426f12beed0440cfd933cb15f8669c7", size = 10566643, upload-time = "2026-03-29T13:21:34.339Z" },
+]
+
+[[package]]
+name = "nvdlfw-inspect"
+version = "0.2.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
- { name = "absl-py" },
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version < '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version >= '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "pyyaml" },
{ name = "torch", marker = "sys_platform == 'never'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/02/ad/046a097b63a96c1ba1d85f0031dbe7fcbdb33e6c445dfbaba2ffaefdd497/nv_grouped_gemm-1.1.4.post8.tar.gz", hash = "sha256:ab321693f0292cfd8a26dc7b6f14decd9eb00e209494de7218e4fad36191275d", size = 20821209, upload-time = "2025-12-17T02:22:38.432Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8a/86/94188e03e5d4dd7b73c390b0cddcde5618b3799c18e327b2bf15763f6137/nvdlfw_inspect-0.2.2-py3-none-any.whl", hash = "sha256:8a4dc2814c5a4cd19ae304170b9bfa514538ef3c3eb243a45a82404ec3cb279d", size = 30964, upload-time = "2025-12-03T10:52:01.933Z" },
+]
[[package]]
-name = "nv-one-logger-core"
-version = "2.3.1"
+name = "nvidia-cublas"
+version = "13.1.0.3"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e1/a5/fce49e2ae977e0ccc084e5adafceb4f0ac0c8333cb6863501618a7277f67/nvidia_cublas-13.1.0.3-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:c86fc7f7ae36d7528288c5d88098edcb7b02c633d262e7ddbb86b0ad91be5df2", size = 542851226, upload-time = "2025-10-09T08:59:04.818Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/44/423ac00af4dd95a5aeb27207e2c0d9b7118702149bf4704c3ddb55bb7429/nvidia_cublas-13.1.0.3-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:ee8722c1f0145ab246bccb9e452153b5e0515fd094c3678df50b2a0888b8b171", size = 423133236, upload-time = "2025-10-09T08:59:32.536Z" },
+ { url = "https://files.pythonhosted.org/packages/10/f5/f50bc3f5c2bb57ab8f5b4d78bc1146b57810d42cb8fcb28cbe2e14050376/nvidia_cublas-13.1.0.3-py3-none-win_amd64.whl", hash = "sha256:2a3b94a37def342471c59fad7856caee4926809a72dd5270155d6a31b5b277be", size = 404355960, upload-time = "2025-10-09T09:07:00.987Z" },
+]
+
+[[package]]
+name = "nvidia-cuda-cupti"
+version = "13.0.85"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2a/2a/80353b103fc20ce05ef51e928daed4b6015db4aaa9162ed0997090fe2250/nvidia_cuda_cupti-13.0.85-py3-none-manylinux_2_25_aarch64.whl", hash = "sha256:796bd679890ee55fb14a94629b698b6db54bcfd833d391d5e94017dd9d7d3151", size = 10310827, upload-time = "2025-09-04T08:26:42.012Z" },
+ { url = "https://files.pythonhosted.org/packages/33/6d/737d164b4837a9bbd202f5ae3078975f0525a55730fe871d8ed4e3b952b0/nvidia_cuda_cupti-13.0.85-py3-none-manylinux_2_25_x86_64.whl", hash = "sha256:4eb01c08e859bf924d222250d2e8f8b8ff6d3db4721288cf35d14252a4d933c8", size = 10715597, upload-time = "2025-09-04T08:26:51.312Z" },
+ { url = "https://files.pythonhosted.org/packages/ad/df/b74b10025c1205695c5676373f2edd3e87a7202cc62ead0dfbc373b0f6ea/nvidia_cuda_cupti-13.0.85-py3-none-win_amd64.whl", hash = "sha256:683f58d301548deeefcb8f6fac1b8d907691b9d8b18eccab417f51e362102f00", size = 7736776, upload-time = "2025-09-04T08:38:08.38Z" },
+]
+
+[[package]]
+name = "nvidia-cuda-nvrtc"
+version = "13.0.88"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c3/68/483a78f5e8f31b08fb1bb671559968c0ca3a065ac7acabfc7cee55214fd6/nvidia_cuda_nvrtc-13.0.88-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:ad9b6d2ead2435f11cbb6868809d2adeeee302e9bb94bcf0539c7a40d80e8575", size = 90215200, upload-time = "2025-09-04T08:28:44.204Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/dc/6bb80850e0b7edd6588d560758f17e0550893a1feaf436807d64d2da040f/nvidia_cuda_nvrtc-13.0.88-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d27f20a0ca67a4bb34268a5e951033496c5b74870b868bacd046b1b8e0c3267b", size = 43015449, upload-time = "2025-09-04T08:28:20.239Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/af/345fedb9f4c76c84ab4fa445b36bd4048a4d9db60e6bc76b4f913ff4b852/nvidia_cuda_nvrtc-13.0.88-py3-none-win_amd64.whl", hash = "sha256:6bcd4e7f8e205cbe644f5a98f2f799bef9556fefc89dd786e79a16312ce49872", size = 76807835, upload-time = "2025-09-04T08:39:15.274Z" },
+]
+
+[[package]]
+name = "nvidia-cuda-runtime"
+version = "13.0.96"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/87/4f/17d7b9b8e285199c58ce28e31b5c5bbaa4d8271af06a89b6405258245de2/nvidia_cuda_runtime-13.0.96-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ef9bcbe90493a2b9d810e43d249adb3d02e98dd30200d86607d8d02687c43f55", size = 2261060, upload-time = "2025-10-09T08:55:15.78Z" },
+ { url = "https://files.pythonhosted.org/packages/2e/24/d1558f3b68b1d26e706813b1d10aa1d785e4698c425af8db8edc3dced472/nvidia_cuda_runtime-13.0.96-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7f82250d7782aa23b6cfe765ecc7db554bd3c2870c43f3d1821f1d18aebf0548", size = 2243632, upload-time = "2025-10-09T08:55:36.117Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/94/6b867483bec07da24ffa32736c79fabb94ef3a7af4d787a9d4a974868576/nvidia_cuda_runtime-13.0.96-py3-none-win_amd64.whl", hash = "sha256:f79298c8a098cec150a597c8eba58ecdab96e3bdc4b9bc4f9983635031740492", size = 2927037, upload-time = "2025-10-09T09:04:23.782Z" },
+]
+
+[[package]]
+name = "nvidia-cudnn-cu13"
+version = "9.19.0.56"
source = { registry = "https://pypi.org/simple" }
dependencies = [
- { name = "overrides" },
- { name = "pydantic" },
- { name = "strenum" },
- { name = "toml" },
- { name = "typing-extensions" },
+ { name = "nvidia-cublas", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/3b/37/963095797035f371e0db6ea761f5aaccb624fc786af217115b423baeb0e2/nv_one_logger_core-2.3.1.tar.gz", hash = "sha256:cbb2f87604c78b96a302f32d87199902129d76153a73a20f8455a250b3246c1d", size = 52640, upload-time = "2025-10-29T21:11:55.812Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/ee/c4/ea91554c4fcbff66057f667690101d7a4b965605741350ac661b03fa6c46/nv_one_logger_core-2.3.1-py3-none-any.whl", hash = "sha256:0c8b77bcdac4daa1ea913bf8d4afd2a057bd5526e3654ac39f67caba157341a6", size = 63066, upload-time = "2025-10-29T21:11:52.753Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/84/26025437c1e6b61a707442184fa0c03d083b661adf3a3eecfd6d21677740/nvidia_cudnn_cu13-9.19.0.56-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:6ed29ffaee1176c612daf442e4dd6cfeb6a0caa43ddcbeb59da94953030b1be4", size = 433781201, upload-time = "2026-02-03T20:40:53.805Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/22/0b4b932655d17a6da1b92fa92ab12844b053bb2ac2475e179ba6f043da1e/nvidia_cudnn_cu13-9.19.0.56-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:d20e1734305e9d68889a96e3f35094d733ff1f83932ebe462753973e53a572bf", size = 366066321, upload-time = "2026-02-03T20:44:52.837Z" },
+ { url = "https://files.pythonhosted.org/packages/91/a2/f020386683ee9ab2c9a9f7f79290d9b0d07f7241de54dc746af2abd188d2/nvidia_cudnn_cu13-9.19.0.56-py3-none-win_amd64.whl", hash = "sha256:40d8c375005bcb01495f8edf375230b203a411a0c05fb6dc92a3781edcb23eac", size = 350547366, upload-time = "2026-02-03T20:50:49.563Z" },
]
[[package]]
-name = "nv-one-logger-training-telemetry"
-version = "2.3.1"
+name = "nvidia-cudnn-frontend"
+version = "1.21.0"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b6/61/c471907880a77d4ad88c9a2a5b22348f7ebb767d0ac78b2efb43908bf085/nvidia_cudnn_frontend-1.21.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:22f8abb69124f2bc6750e7d7e8625680964cb61caab4c3aaee641d5ad295f37b", size = 2676690, upload-time = "2026-03-25T03:09:00.102Z" },
+ { url = "https://files.pythonhosted.org/packages/87/9d/669525d19be3ea3e3e815ab0ecb21508ed0f57744a06e1e71bf9f73dda5e/nvidia_cudnn_frontend-1.21.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ae3feffb22a9c90007fc54d27642e160dab750401ac5399589dd450e8d01780f", size = 2828132, upload-time = "2026-03-25T03:13:00.829Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/c8/43ca12a36a1a7156ecff413cc1e4050a5eb5205c1fcfc92481dc4370257f/nvidia_cudnn_frontend-1.21.0-cp312-cp312-win_amd64.whl", hash = "sha256:3e5f28d29cef9a57568942ff66f6c8814a6815b369b3bf850bdcdd67c9cd41cd", size = 2253978, upload-time = "2026-03-25T03:25:21.2Z" },
+ { url = "https://files.pythonhosted.org/packages/83/14/320fbc455599073990d2b76e0658aaa9b10d4c11cd87abd591925d7565b4/nvidia_cudnn_frontend-1.21.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c3153affbefad49dba1e8643dad59642b54f7d2724bf7286acc5830619ada5ba", size = 2676443, upload-time = "2026-03-25T03:09:37.878Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/f2/5eefffb126d0bb8b81deade2b9cba466cbabf8e2ae9422cce56834fcb3fa/nvidia_cudnn_frontend-1.21.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:69ad047e47580e76f3694d8e6c74c6d939ce2fff199145f3ea1e6a494c8ea84f", size = 2827936, upload-time = "2026-03-25T03:13:25.431Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/3d/3b505280001ac79d909ff3524236a46d16947f1ee487a049a4f257059731/nvidia_cudnn_frontend-1.21.0-cp313-cp313-win_amd64.whl", hash = "sha256:2a787d4d44ecce7542fae4715630e3427ee4c1feda62069643c010b81e72aa35", size = 2253943, upload-time = "2026-03-25T03:25:44.705Z" },
+ { url = "https://files.pythonhosted.org/packages/37/17/4ce01e6c0e50968acc610962b9f1fa397f8c6437d235edb662e61a75a6db/nvidia_cudnn_frontend-1.21.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eee3f650d8573c39ae7add45881b2d80d1a1b540fae75a9fecead3d756bc2640", size = 2678599, upload-time = "2026-03-25T03:10:23.56Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/c9/33ae2104d964ee3c9f30ec59e577f961a28293253db1946a55cceb96ef8e/nvidia_cudnn_frontend-1.21.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a2c9101c7c8845225505b446be49b6eded60c9718f9befceb00c8be34cf3f954", size = 2828793, upload-time = "2026-03-25T03:13:49.696Z" },
+ { url = "https://files.pythonhosted.org/packages/68/40/84e86ed1f3ecc408492471e70aa71efc595e641d698df3cc57320cd0a3a1/nvidia_cudnn_frontend-1.21.0-cp314-cp314-win_amd64.whl", hash = "sha256:304d150ca768837eaca9f2ad8a94bb80c1f161eeeb71b696c5807d375bc27915", size = 2254903, upload-time = "2026-03-25T03:26:07.179Z" },
+]
+
+[[package]]
+name = "nvidia-cufft"
+version = "12.0.0.61"
source = { registry = "https://pypi.org/simple" }
dependencies = [
- { name = "nv-one-logger-core" },
- { name = "strenum" },
- { name = "typing-extensions" },
+ { name = "nvidia-nvjitlink", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/c5/21/016fa067967734d52f1ccf5a2a37a1a65216f2d7053bc2b85872cce956ca/nv_one_logger_training_telemetry-2.3.1.tar.gz", hash = "sha256:8c67940ea71799afaf1f46df3ba2f52f93aea26321c6f1c1d54aae02efc2a4af", size = 44435, upload-time = "2025-10-29T21:21:42.035Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/e5/15/97e6e4ddfe5fc35bcee74a45b7c33fb73abb83713c7dfa26420b971a86c3/nv_one_logger_training_telemetry-2.3.1-py3-none-any.whl", hash = "sha256:5319443829b59378a498c3c62ac98973e14f31be675c229ff2b14e2fe109aa0b", size = 44140, upload-time = "2025-10-29T21:21:40.72Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/ae/f417a75c0259e85c1d2f83ca4e960289a5f814ed0cea74d18c353d3e989d/nvidia_cufft-12.0.0.61-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2708c852ef8cd89d1d2068bdbece0aa188813a0c934db3779b9b1faa8442e5f5", size = 214053554, upload-time = "2025-09-04T08:31:38.196Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/2f/7b57e29836ea8714f81e9898409196f47d772d5ddedddf1592eadb8ab743/nvidia_cufft-12.0.0.61-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:6c44f692dce8fd5ffd3e3df134b6cdb9c2f72d99cf40b62c32dde45eea9ddad3", size = 214085489, upload-time = "2025-09-04T08:31:56.044Z" },
+ { url = "https://files.pythonhosted.org/packages/85/b2/f8af21a2ed1beed337a6a02c5a28aeb85441f4d578ec3d529543c775ea4b/nvidia_cufft-12.0.0.61-py3-none-win_amd64.whl", hash = "sha256:2abce5b39d2f5ae12730fb7e5db6696533e36c26e2d3e8fd1750bdd2853364eb", size = 213342123, upload-time = "2025-09-04T08:40:51.145Z" },
]
[[package]]
-name = "nvdlfw-inspect"
-version = "0.2.2"
+name = "nvidia-cufile"
+version = "1.15.1.6"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3f/70/4f193de89a48b71714e74602ee14d04e4019ad36a5a9f20c425776e72cd6/nvidia_cufile-1.15.1.6-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:08a3ecefae5a01c7f5117351c64f17c7c62efa5fffdbe24fc7d298da19cd0b44", size = 1223672, upload-time = "2025-09-04T08:32:22.779Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/73/cc4a14c9813a8a0d509417cf5f4bdaba76e924d58beb9864f5a7baceefbf/nvidia_cufile-1.15.1.6-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:bdc0deedc61f548bddf7733bdc216456c2fdb101d020e1ab4b88d232d5e2f6d1", size = 1136992, upload-time = "2025-09-04T08:32:14.119Z" },
+]
+
+[[package]]
+name = "nvidia-curand"
+version = "10.4.0.35"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1e/72/7c2ae24fb6b63a32e6ae5d241cc65263ea18d08802aaae087d9f013335a2/nvidia_curand-10.4.0.35-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:133df5a7509c3e292aaa2b477afd0194f06ce4ea24d714d616ff36439cee349a", size = 61962106, upload-time = "2025-08-04T10:21:41.128Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/9f/be0a41ca4a4917abf5cb9ae0daff1a6060cc5de950aec0396de9f3b52bc5/nvidia_curand-10.4.0.35-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:1aee33a5da6e1db083fe2b90082def8915f30f3248d5896bcec36a579d941bfc", size = 59544258, upload-time = "2025-08-04T10:22:03.992Z" },
+ { url = "https://files.pythonhosted.org/packages/99/27/72103153b1ffc00e09fdc40ac970235343dcd1ea8bd762e84d2d73219ffa/nvidia_curand-10.4.0.35-py3-none-win_amd64.whl", hash = "sha256:65b1710aa6961d326b411e314b374290904c5ddf41dc3f766ebc3f1d7d4ca69f", size = 55242481, upload-time = "2025-08-04T10:30:41.831Z" },
+]
+
+[[package]]
+name = "nvidia-cusolver"
+version = "12.0.4.66"
source = { registry = "https://pypi.org/simple" }
dependencies = [
- { name = "pyyaml" },
- { name = "torch", marker = "sys_platform == 'never'" },
+ { name = "nvidia-cublas", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "nvidia-cusparse", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "nvidia-nvjitlink", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
]
wheels = [
- { url = "https://files.pythonhosted.org/packages/8a/86/94188e03e5d4dd7b73c390b0cddcde5618b3799c18e327b2bf15763f6137/nvdlfw_inspect-0.2.2-py3-none-any.whl", hash = "sha256:8a4dc2814c5a4cd19ae304170b9bfa514538ef3c3eb243a45a82404ec3cb279d", size = 30964, upload-time = "2025-12-03T10:52:01.933Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/c3/b30c9e935fc01e3da443ec0116ed1b2a009bb867f5324d3f2d7e533e776b/nvidia_cusolver-12.0.4.66-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:02c2457eaa9e39de20f880f4bd8820e6a1cfb9f9a34f820eb12a155aa5bc92d2", size = 223467760, upload-time = "2025-09-04T08:33:04.222Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/67/cba3777620cdacb99102da4042883709c41c709f4b6323c10781a9c3aa34/nvidia_cusolver-12.0.4.66-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:0a759da5dea5c0ea10fd307de75cdeb59e7ea4fcb8add0924859b944babf1112", size = 200941980, upload-time = "2025-09-04T08:33:22.767Z" },
+ { url = "https://files.pythonhosted.org/packages/99/ef/332a0101260ca78a1daef046bf0b06199e8ed4dac1d2aa698289c358169c/nvidia_cusolver-12.0.4.66-py3-none-win_amd64.whl", hash = "sha256:16515bd33a8e76bb54d024cfa068fa68d30e80fc34b9e1090813ea9362e0cb65", size = 193551444, upload-time = "2025-09-04T08:41:46.813Z" },
]
[[package]]
-name = "nvidia-cudnn-frontend"
-version = "1.17.0"
+name = "nvidia-cusparse"
+version = "12.6.3.3"
source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nvidia-nvjitlink", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
wheels = [
- { url = "https://files.pythonhosted.org/packages/14/94/b224e65becfb5ab02c5b331aeb73c98f6d95cde5326d7698a2fc0d20e84a/nvidia_cudnn_frontend-1.17.0-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4835ee3fc350782c89cdd290088ade69464faaa5dd66ccb0b215ad481ab3b41b", size = 1911670, upload-time = "2025-12-20T00:26:36.302Z" },
- { url = "https://files.pythonhosted.org/packages/d5/05/54afda6fc47838bd68a029067d8019e6b495dca0570d7e970cbb2c3e0b32/nvidia_cudnn_frontend-1.17.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1da7e972dbba939ad21111f1208815b8c8024cbf72aa6c1eb223b14b2049d4b6", size = 2033618, upload-time = "2025-12-20T00:24:42.991Z" },
- { url = "https://files.pythonhosted.org/packages/83/97/77ad90fac9372b0420885f16a2afaca95f78b082fa9d6a082d51a7c96bd3/nvidia_cudnn_frontend-1.17.0-cp310-cp310-win_amd64.whl", hash = "sha256:21c5b2ce097f72c6510cbf974ce8ea9a31b34989dd9209d7187584a6100e57e5", size = 1440589, upload-time = "2025-12-20T00:29:17.641Z" },
- { url = "https://files.pythonhosted.org/packages/4e/4a/a903c57ef5aaa32aa074007ba4d50ed7cbc80a8092ddb84fe9d879a69bbb/nvidia_cudnn_frontend-1.17.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:961004000a2c21dd4a03f816534629105cf49125a643dbb49abbc97021e66d20", size = 1911775, upload-time = "2025-12-20T00:27:11.297Z" },
- { url = "https://files.pythonhosted.org/packages/15/20/80c4f5d62ebc58b8db8d25a2ee11f3246bb8947addea37c229540bcc05ac/nvidia_cudnn_frontend-1.17.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6ea44a8f2c0cfd20868b239ea13a2e0f32895dab868f6ff2bee01caf3778d273", size = 2035158, upload-time = "2025-12-20T00:25:00.9Z" },
- { url = "https://files.pythonhosted.org/packages/5f/18/c24375c8d579c53a99a2d7428397288a94c7ea411d1823e3b8dc3cef50dc/nvidia_cudnn_frontend-1.17.0-cp311-cp311-win_amd64.whl", hash = "sha256:8dd6cc197a58d63da4d146a1febc1f99d425374d159f9b00628b140c65acb486", size = 1441316, upload-time = "2025-12-20T00:29:34.951Z" },
- { url = "https://files.pythonhosted.org/packages/42/d9/f58ed6292c9396f7422812a0a2d9f80cc5a623ea6c758bcb3d34d4795bb8/nvidia_cudnn_frontend-1.17.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:de0c473f32d705abcf14f351615f7ffbeed7320e3499cf2195ae5689652a2592", size = 1917620, upload-time = "2025-12-20T00:27:46.179Z" },
- { url = "https://files.pythonhosted.org/packages/db/eb/c641135632bd2afc21339aadee96af4c5db1460dfa07ca74836de75a590f/nvidia_cudnn_frontend-1.17.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c913c87fca691a91385287f2587575531933acfebc85c33dbcecb191886c7a53", size = 2038994, upload-time = "2025-12-20T00:25:18.9Z" },
- { url = "https://files.pythonhosted.org/packages/82/49/a92da03eb43bde90be770a43666c5ab26b4f8b15f6e46c4b0b0e84f37994/nvidia_cudnn_frontend-1.17.0-cp312-cp312-win_amd64.whl", hash = "sha256:a0d4cfd03961592108abd1ba246e43c8bb7540aed984df860256d0bff181de98", size = 1441271, upload-time = "2025-12-20T00:29:52.056Z" },
- { url = "https://files.pythonhosted.org/packages/99/96/4d55a559dff3175599fe15d83c853f051526b91994b083ec36b12caae776/nvidia_cudnn_frontend-1.17.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3800a1fe3d41a9206281475b1c8c438b02cb7e3c7e262d13f0a101edec223cb6", size = 1917065, upload-time = "2025-12-20T00:28:21.402Z" },
- { url = "https://files.pythonhosted.org/packages/20/f6/5af63c254d7260dd1e974b2300eae9b157998b9d958f79c98ddaada0a0bf/nvidia_cudnn_frontend-1.17.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5adaf4a930b3be5ed019e1a25cfec7cc2bf444592a54a7639c28149b9227c2a4", size = 2039180, upload-time = "2025-12-20T00:25:36.695Z" },
- { url = "https://files.pythonhosted.org/packages/64/ee/6de6aec1e42c859134312e6d5348d6f036b2f1b825e6eae92f9a429eccc4/nvidia_cudnn_frontend-1.17.0-cp313-cp313-win_amd64.whl", hash = "sha256:5c6a120fb54b157585ce6587153fc7086081af961f284f2553e01ba7c7a80c1a", size = 1441177, upload-time = "2025-12-20T00:30:09.927Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/94/5c26f33738ae35276672f12615a64bd008ed5be6d1ebcb23579285d960a9/nvidia_cusparse-12.6.3.3-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:80bcc4662f23f1054ee334a15c72b8940402975e0eab63178fc7e670aa59472c", size = 162155568, upload-time = "2025-09-04T08:33:42.864Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/18/623c77619c31d62efd55302939756966f3ecc8d724a14dab2b75f1508850/nvidia_cusparse-12.6.3.3-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2b3c89c88d01ee0e477cb7f82ef60a11a4bcd57b6b87c33f789350b59759360b", size = 145942937, upload-time = "2025-09-04T08:33:58.029Z" },
+ { url = "https://files.pythonhosted.org/packages/02/b0/b043d6f3480f102f885cf87fc3ffd3edcb5e23b855025a50e2ef4d059185/nvidia_cusparse-12.6.3.3-py3-none-win_amd64.whl", hash = "sha256:cbcf42feb737bd7ec15b4c0a63e62351886bd3f975027b8815d7f720a2b5ea79", size = 143783033, upload-time = "2025-09-04T08:42:12.391Z" },
+]
+
+[[package]]
+name = "nvidia-cusparselt-cu13"
+version = "0.8.0"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/46/10/8dcd1175260706a2fc92a16a52e306b71d4c1ea0b0cc4a9484183399818a/nvidia_cusparselt_cu13-0.8.0-py3-none-manylinux2014_aarch64.whl", hash = "sha256:400c6ed1cf6780fc6efedd64ec9f1345871767e6a1a0a552a1ea0578117ea77c", size = 220791277, upload-time = "2025-08-13T19:22:40.982Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/53/43b0d71f4e702fa9733f8b4571fdca50a8813f1e450b656c239beff12315/nvidia_cusparselt_cu13-0.8.0-py3-none-manylinux2014_x86_64.whl", hash = "sha256:25e30a8a7323935d4ad0340b95a0b69926eee755767e8e0b1cf8dd85b197d3fd", size = 169884119, upload-time = "2025-08-13T19:23:41.967Z" },
+ { url = "https://files.pythonhosted.org/packages/57/de/8f0578928b9b1246d7b1324db0528e6b9f9fb54496a49f40bf71f09f1a27/nvidia_cusparselt_cu13-0.8.0-py3-none-win_amd64.whl", hash = "sha256:e80212ed7b1afc97102fbb2b5c82487aa73f6a0edfa6d26c5a152593e520bb8f", size = 156459710, upload-time = "2025-08-13T19:24:18.043Z" },
]
[[package]]
name = "nvidia-cutlass-dsl"
-version = "4.3.5"
+version = "4.4.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nvidia-cutlass-dsl-libs-base" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a9/03/678dab0383db1ddfc449da216220f40404189eb36eeed9d87a4fa4bdb0e6/nvidia_cutlass_dsl-4.4.2-py3-none-any.whl", hash = "sha256:7cfb9ef19062b055b9372c7a627004724e2755e4c8b16c3cc88807d64501a4ae", size = 10167, upload-time = "2026-03-16T02:18:59.043Z" },
+]
+
+[[package]]
+name = "nvidia-cutlass-dsl-libs-base"
+version = "4.4.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "cuda-python" },
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version < '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version >= '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
{ name = "typing-extensions" },
]
wheels = [
- { url = "https://files.pythonhosted.org/packages/52/3a/89f70082c24d3b88316df9b16df861e1f2cc86389a7b36a670bc7c541977/nvidia_cutlass_dsl-4.3.5-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:b4fcc50dbf9f9c6d1f4d6e1748e366c6835c95bea7b54f7111bfa6e66230f74b", size = 58736963, upload-time = "2026-01-09T01:37:55.298Z" },
- { url = "https://files.pythonhosted.org/packages/e7/92/3f39b64341e2b16dedc7434e7b63a8f457a6fdbd023346d2f00276943495/nvidia_cutlass_dsl-4.3.5-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:776f54fa72333bc8fca274e59b70552adbcd85aaef603c7d58a79ef284890046", size = 58601295, upload-time = "2026-01-09T01:39:02.461Z" },
- { url = "https://files.pythonhosted.org/packages/e8/93/9114f28351d55061d30c68dbec3ba49659ac65607966029f52dab66950e9/nvidia_cutlass_dsl-4.3.5-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:6de9a4a7150ad1832fb8c862c92df4836f347690e4c085e9044160c846010b59", size = 58736943, upload-time = "2026-01-09T01:40:25.777Z" },
- { url = "https://files.pythonhosted.org/packages/54/b5/d2f08919a9aa9052d45b2c8adfc310a724e9474e39c612358b1b24282c54/nvidia_cutlass_dsl-4.3.5-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:7a792f02ce548f311a3df313a7cdb4ac4ec1cccb6c7ff9cd68d5470b25a6daf6", size = 58602358, upload-time = "2026-01-09T01:39:28.521Z" },
- { url = "https://files.pythonhosted.org/packages/78/6c/f45c930f662e0ec7856baa5d4e6f4d1e2ca6b029678f9e05d2df54c865be/nvidia_cutlass_dsl-4.3.5-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:6a79e94d157b16ab34069dd73fb708ff0ef31f486d699b6d5a015217f754cb0b", size = 58739895, upload-time = "2026-01-09T01:38:22.076Z" },
- { url = "https://files.pythonhosted.org/packages/76/cb/998e79b6f028268bf2653250deb4a2edb618db81244e549ced71112c6f85/nvidia_cutlass_dsl-4.3.5-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:4687eef20c405023daa99dd4653a292fd875d6c9486f8d9a069ff6fcdb00834f", size = 58602784, upload-time = "2026-01-09T01:40:52.873Z" },
- { url = "https://files.pythonhosted.org/packages/97/09/78a2f9141006f6f1e371a3dfb7a921205bcad6fb27810731169939d3e63d/nvidia_cutlass_dsl-4.3.5-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:9343a5c1335169d791b05aac6fb81e33d7f17c4f8250613a091e6ee8314ed6aa", size = 58738707, upload-time = "2026-01-09T01:39:56.445Z" },
- { url = "https://files.pythonhosted.org/packages/0f/16/41b88ded92648d99f3c83880c07a54475feded9b32b4425e30d4b34f6c63/nvidia_cutlass_dsl-4.3.5-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:11d19b7e56ae1bedaf736ea3965af3be1e7af6c2482989c414b606cdd406cf32", size = 58601867, upload-time = "2026-01-09T01:37:29.895Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/7d/0df5e38d11e52cc72095a14d6448bc1c5d0d4b00b069a1189ca417fb225b/nvidia_cutlass_dsl_libs_base-4.4.2-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:2ec8812eeadcbb6fe20bda2e295ed9c00653f8253b78e33cf0ab65a47b829e73", size = 75473821, upload-time = "2026-03-16T02:27:08.371Z" },
+ { url = "https://files.pythonhosted.org/packages/56/98/e264964741d9cc9816625d9600d17a5249fd5cbd8c2d166fb0d0c34dfe5a/nvidia_cutlass_dsl_libs_base-4.4.2-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:22e37b58f7a6f2f43bba533c4df8a088012122e0b4e9a632eca23937adeafb39", size = 74355593, upload-time = "2026-03-16T02:25:11.762Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/c9/2f17950ee2deb4b5f6b82f8155515a21792fe296e81bb638f164d8e2ca9b/nvidia_cutlass_dsl_libs_base-4.4.2-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:b59a052cbfb9a25747d1b6d413615456bea38d1f377da085af07c0d86a4c8b39", size = 75477304, upload-time = "2026-03-16T02:27:35.645Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/68/27380038ebd9c8eab4be364e833fea144aef597704f44948921668f7adf4/nvidia_cutlass_dsl_libs_base-4.4.2-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:8e3324a33afa7424e93beae7e54a311e80db82b9e4ed4bba2aeeda1d6c888cd9", size = 74355765, upload-time = "2026-03-16T02:24:16.778Z" },
+ { url = "https://files.pythonhosted.org/packages/12/44/0dc7f2e5b5c65106a5bb05e60654f1a79abe92e27e9b00588a73cd26ca1f/nvidia_cutlass_dsl_libs_base-4.4.2-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:af96c1170569138b3cb965202907fbf5ab95d7c1dcc210952d00cdf9ab7b859a", size = 75472171, upload-time = "2026-03-16T02:28:03.136Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/ae/0998f328b28b956d7eb399d16f4ee681ca318b306007264444a623e86c64/nvidia_cutlass_dsl_libs_base-4.4.2-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:95db0c8d1d56992e2f5c2dcd5b3baab0297bedc0cbcefc1e70b57acd934e7b23", size = 74356280, upload-time = "2026-03-16T02:25:43.789Z" },
]
[[package]]
@@ -3156,60 +3547,90 @@ wheels = [
[[package]]
name = "nvidia-ml-py"
-version = "13.590.44"
+version = "13.595.45"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/1b/23/3871537f204aee823c574ba25cbeb08cae779979d4d43c01adddda00bab9/nvidia_ml_py-13.590.44.tar.gz", hash = "sha256:b358c7614b0fdeea4b95f046f1c90123bfe25d148ab93bb1c00248b834703373", size = 49737, upload-time = "2025-12-08T14:41:10.872Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/ce/49/c29f6e30d8662d2e94fef17739ea7309cc76aba269922ae999e4cc07f268/nvidia_ml_py-13.595.45.tar.gz", hash = "sha256:c9f34897fe0441ff35bc8f35baf80f830a20b0f4e6ce71e0a325bc0e66acf079", size = 50780, upload-time = "2026-03-19T16:59:44.956Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/e4/47/4c822bd37a008e72fd5a0eae33524ae3ac97b13f7030f63bae1728b8957e/nvidia_ml_py-13.590.44-py3-none-any.whl", hash = "sha256:18feb54eca7d0e3cdc8d1a040a771eda72d9ec3148e5443087970dbfd7377ecc", size = 50683, upload-time = "2025-12-08T14:41:09.597Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/24/fc256107d23597fa33d319505ce77160fa1a2349c096d01901ffc7cb7fc4/nvidia_ml_py-13.595.45-py3-none-any.whl", hash = "sha256:b65a7977f503d56154b14d683710125ef93594adb63fbf7e559336e3318f1376", size = 51776, upload-time = "2026-03-19T16:59:43.603Z" },
]
[[package]]
name = "nvidia-modelopt"
-version = "0.40.0"
+version = "0.42.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "ninja" },
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" } },
{ name = "nvidia-ml-py" },
- { name = "packaging" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" } },
{ name = "pulp" },
{ name = "pydantic" },
{ name = "regex" },
{ name = "rich" },
{ name = "safetensors" },
- { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "scipy", version = "1.17.0", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "scipy" },
{ name = "torch", marker = "sys_platform == 'never'" },
- { name = "torchprofile" },
{ name = "tqdm" },
]
wheels = [
- { url = "https://files.pythonhosted.org/packages/7f/4a/4b4c339637fdbd54bc98b92c87c8b22f5efee05ca9e31e40a8d49ee66187/nvidia_modelopt-0.40.0-py3-none-any.whl", hash = "sha256:0315f53aef014b902866e427038db5803e3c6787a8e1f09c3650031550885051", size = 901421, upload-time = "2025-12-12T10:35:28.506Z" },
+ { url = "https://files.pythonhosted.org/packages/61/a4/ad7f8d4ce21e1df1670aaaa05db45bece34a74c9fb44e4e77f668a24adce/nvidia_modelopt-0.42.0-py3-none-any.whl", hash = "sha256:3e8149b4d206b4ae51165f4f6a6d28fc9c2172406c948d5abcd8637b08db5c28", size = 1005332, upload-time = "2026-03-09T20:43:57.936Z" },
]
[[package]]
-name = "nvidia-resiliency-ext"
-version = "0.5.0"
+name = "nvidia-nccl-cu13"
+version = "2.28.9"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/39/55/1920646a2e43ffd4fc958536b276197ed740e9e0c54105b4bb3521591fc7/nvidia_nccl_cu13-2.28.9-py3-none-manylinux_2_18_aarch64.whl", hash = "sha256:01c873ba1626b54caa12272ed228dc5b2781545e0ae8ba3f432a8ef1c6d78643", size = 196561677, upload-time = "2025-11-18T05:49:03.45Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/b4/878fefaad5b2bcc6fcf8d474a25e3e3774bc5133e4b58adff4d0bca238bc/nvidia_nccl_cu13-2.28.9-py3-none-manylinux_2_18_x86_64.whl", hash = "sha256:e4553a30f34195f3fa1da02a6da3d6337d28f2003943aa0a3d247bbc25fefc42", size = 196493177, upload-time = "2025-11-18T05:49:17.677Z" },
+]
+
+[[package]]
+name = "nvidia-nvjitlink"
+version = "13.0.88"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/56/7a/123e033aaff487c77107195fa5a2b8686795ca537935a24efae476c41f05/nvidia_nvjitlink-13.0.88-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:13a74f429e23b921c1109976abefacc69835f2f433ebd323d3946e11d804e47b", size = 40713933, upload-time = "2025-09-04T08:35:43.553Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/2c/93c5250e64df4f894f1cbb397c6fd71f79813f9fd79d7cd61de3f97b3c2d/nvidia_nvjitlink-13.0.88-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e931536ccc7d467a98ba1d8b89ff7fa7f1fa3b13f2b0069118cd7f47bff07d0c", size = 38768748, upload-time = "2025-09-04T08:35:20.008Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/01/07530b0e37546231052e30234540289c42eaffa486f1a34a87fed340157b/nvidia_nvjitlink-13.0.88-py3-none-win_amd64.whl", hash = "sha256:634e96e3da9ef845ae744097a1f289238ecf946ce0b82e93cdce14b9782e682f", size = 36035115, upload-time = "2025-09-04T08:43:03.001Z" },
+]
+
+[[package]]
+name = "nvidia-nvshmem-cu13"
+version = "3.4.5"
source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/dc/0f/05cc9c720236dcd2db9c1ab97fff629e96821be2e63103569da0c9b72f19/nvidia_nvshmem_cu13-3.4.5-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:6dc2a197f38e5d0376ad52cd1a2a3617d3cdc150fd5966f4aee9bcebb1d68fe9", size = 60215947, upload-time = "2025-09-06T00:32:20.022Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/35/a9bf80a609e74e3b000fef598933235c908fcefcef9026042b8e6dfde2a9/nvidia_nvshmem_cu13-3.4.5-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:290f0a2ee94c9f3687a02502f3b9299a9f9fe826e6d0287ee18482e78d495b80", size = 60412546, upload-time = "2025-09-06T00:32:41.564Z" },
+]
+
+[[package]]
+name = "nvidia-nvtx"
+version = "13.0.85"
+source = { registry = "https://pypi.org/simple" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c2/f3/d86c845465a2723ad7e1e5c36dcd75ddb82898b3f53be47ebd429fb2fa5d/nvidia_nvtx-13.0.85-py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:4936d1d6780fbe68db454f5e72a42ff64d1fd6397df9f363ae786930fd5c1cd4", size = 148047, upload-time = "2025-09-04T08:29:01.761Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/64/3708a90d1ebe202ffdeb7185f878a3c84d15c2b2c31858da2ce0583e2def/nvidia_nvtx-13.0.85-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cb7780edb6b14107373c835bf8b72e7a178bac7367e23da7acb108f973f157a6", size = 148878, upload-time = "2025-09-04T08:28:53.627Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/50/0e2220f8620a177de994211186ffc5bfa9f2ce1e1282797f8f90096f9f88/nvidia_nvtx-13.0.85-py3-none-win_amd64.whl", hash = "sha256:d66ea44254dd3c6eacc300047af6e1288d2269dd072b417e0adffbf479e18519", size = 137066, upload-time = "2025-09-04T08:39:25.649Z" },
+]
+
+[[package]]
+name = "nvidia-resiliency-ext"
+version = "0.6.0"
+source = { git = "https://github.com/NVIDIA/nvidia-resiliency-ext.git?rev=63154570cea17f8805a7fd15cc3b8cc2919ba575#63154570cea17f8805a7fd15cc3b8cc2919ba575" }
dependencies = [
{ name = "defusedxml" },
- { name = "nv-one-logger-core" },
- { name = "nv-one-logger-training-telemetry" },
+ { name = "grpcio" },
+ { name = "grpcio-tools" },
+ { name = "langchain-nvidia-ai-endpoints" },
+ { name = "logsage" },
+ { name = "mcp" },
{ name = "nvidia-ml-py" },
- { name = "packaging" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" } },
{ name = "psutil" },
{ name = "pyyaml" },
{ name = "torch", marker = "sys_platform == 'never'" },
]
-wheels = [
- { url = "https://files.pythonhosted.org/packages/df/18/1898cad3bdd643c6bfa5f7aee125a5ef308ab1701ab15106e3e9c66bb416/nvidia_resiliency_ext-0.5.0-cp310-cp310-manylinux_2_39_aarch64.whl", hash = "sha256:97d4b68d3949f3b8370addb474d8662d6ac5008c3c1296420cdeb93a88d6a804", size = 402915, upload-time = "2025-11-13T21:28:34.578Z" },
- { url = "https://files.pythonhosted.org/packages/fa/48/10fc3f278898e3b2aacc3bea65f0ac4b579e6e0e8447b467742d75adeec1/nvidia_resiliency_ext-0.5.0-cp310-cp310-manylinux_2_39_x86_64.whl", hash = "sha256:ceb04ec5a7bc9301fd6f14449bda6b0d1f37ead4fbe37aa3bf1d7b2ad5b662d4", size = 406483, upload-time = "2025-11-13T21:28:58.732Z" },
- { url = "https://files.pythonhosted.org/packages/14/17/c19dfed8d4aced307a1c1404f0917ee6c1b319db8092b3cfe2af4e76de6d/nvidia_resiliency_ext-0.5.0-cp311-cp311-manylinux_2_39_aarch64.whl", hash = "sha256:62d396356adcf898cb86a54956eeece29017a41b5872db0b364c8449d23f2f66", size = 404062, upload-time = "2025-11-13T21:29:46.873Z" },
- { url = "https://files.pythonhosted.org/packages/7f/99/b4324595171c3cdffb03cef070006ab9a3de7fca90a22403576ec6423b69/nvidia_resiliency_ext-0.5.0-cp311-cp311-manylinux_2_39_x86_64.whl", hash = "sha256:c4fcd006ef69300f753bb30d17efbb6bcee6699f044e3532209b2825d22e9977", size = 407027, upload-time = "2025-11-13T21:30:09.124Z" },
- { url = "https://files.pythonhosted.org/packages/8c/73/232d9f25558f3c6165ff1d15c980a434b47c13e8f527f999cd265859abcf/nvidia_resiliency_ext-0.5.0-cp312-cp312-manylinux_2_39_aarch64.whl", hash = "sha256:81e3d827885e90bed369e67f76dda6709dd4073c2e5fa1228df85d6987cee495", size = 403317, upload-time = "2025-11-13T21:31:24.603Z" },
- { url = "https://files.pythonhosted.org/packages/44/89/4d7f39416aa3be72ee9f1260a7af56af40f2570f5add1e039d96279a8764/nvidia_resiliency_ext-0.5.0-cp312-cp312-manylinux_2_39_x86_64.whl", hash = "sha256:eb720cd25feabef07f971d4051c7bcac2f9ec73642a9031953d2663307950cb9", size = 407963, upload-time = "2025-11-13T21:30:28.998Z" },
-]
[[package]]
name = "nvidia-sphinx-theme"
@@ -3217,8 +3638,7 @@ version = "0.0.9.post1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "pydata-sphinx-theme" },
- { name = "sphinx", version = "8.1.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinx", version = "8.2.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "sphinx" },
]
wheels = [
{ url = "https://files.pythonhosted.org/packages/8c/79/017fab2f7167a9a9795665f894d04f77aafceca80821b51589bb4b23ff5c/nvidia_sphinx_theme-0.0.9.post1-py3-none-any.whl", hash = "sha256:21ca60206dff2f380d7783d64bbaf71a5b9cacae53c7d0686f089c16b5a3d45a", size = 143816, upload-time = "2025-11-09T23:16:55.719Z" },
@@ -3226,175 +3646,716 @@ wheels = [
[[package]]
name = "nvtx"
-version = "0.2.14"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/0e/03/b8a4391523a92163167fd0fee6769c223e8612043cb07aebc1173ca83fc9/nvtx-0.2.14.tar.gz", hash = "sha256:12945242a31bde70b1f15cae867f8706bdff290e2f808a11738e03ebefdf847f", size = 119864, upload-time = "2025-12-01T18:06:16.674Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/ed/ca/fa76ea4985fd8f3d8c437bffec2580b1cac7f2401671089ac842610ae466/nvtx-0.2.14-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b70b2415ab97edf19514be226d5058898922c6b6bb1d7fdd5ef92d1e086f3e0f", size = 695204, upload-time = "2025-11-27T17:28:52.688Z" },
- { url = "https://files.pythonhosted.org/packages/b9/1f/0aa62d52062d700dbed36dd2ebfddf5133c72180d448cce66545e5ccbe5d/nvtx-0.2.14-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:23ab874f9c70e5433f39e40ca318ffcfc14fb43ed6798e6be5a30f74e4ca831f", size = 686698, upload-time = "2025-11-27T17:23:19.335Z" },
- { url = "https://files.pythonhosted.org/packages/18/c9/a12d48157221a8e939f3f7ec8f8a543e232fb9248820afb164ff9eb3eaa7/nvtx-0.2.14-cp310-cp310-win_amd64.whl", hash = "sha256:3a22be895546ca609e83e54614b56739200ab6f4d13e15f5685544082b1b7908", size = 119654, upload-time = "2025-11-27T17:32:08.536Z" },
- { url = "https://files.pythonhosted.org/packages/87/a6/4d473abd7c07a6d1060c0f708e21ddf46a960258532ffc897681db5c0f46/nvtx-0.2.14-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:227f6406d2fe1a4b890be17eb1f4c1f5bd4df8f7032dd1cb8c7651d379f35541", size = 732764, upload-time = "2025-11-27T17:26:21.853Z" },
- { url = "https://files.pythonhosted.org/packages/94/06/3ab72e5a463af1b95934638cb8377e99f58e5ef21a47cbf69b92267d6602/nvtx-0.2.14-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0664aa75b24e2ad0abdd0fa52c49e9c8a120652f2194289c85dc2d93cbc6017f", size = 724555, upload-time = "2025-11-27T17:22:36.402Z" },
- { url = "https://files.pythonhosted.org/packages/18/1d/64f6078a5ab4134af91ba294035ee1ebb3512edaaa9d60d8f0f023178620/nvtx-0.2.14-cp311-cp311-win_amd64.whl", hash = "sha256:10f5971661d61c1a90cd36c3069240452c904ecec4b3a08d0d6fdba1e5398165", size = 119660, upload-time = "2025-11-27T17:32:30.406Z" },
- { url = "https://files.pythonhosted.org/packages/8a/de/2cc15bb805b1b18317b60837b853ed023757730d0db82de291635fc88bc3/nvtx-0.2.14-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3ece46f555e725db879df06549980744f89db5923a77e6f7a5aecda75292421a", size = 727708, upload-time = "2025-11-27T17:25:20.836Z" },
- { url = "https://files.pythonhosted.org/packages/81/94/b37d634fef8677ce525b5bfd2886737ea2c064bc3576fc84423973ff5b97/nvtx-0.2.14-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:17efe5d903996bceb0c8a12cae80fa9b66bee7ee895923bd9d8ec2a5af1aabd8", size = 737691, upload-time = "2025-11-27T17:21:27.87Z" },
- { url = "https://files.pythonhosted.org/packages/ad/c1/f633aa32003050ff83626a19402f03c83990a15b4df658a7bf1b590ee83e/nvtx-0.2.14-cp312-cp312-win_amd64.whl", hash = "sha256:f40db4746714d525d3020c702a0df866c2335efd6a27c41e869e577402a53a4b", size = 119193, upload-time = "2025-11-27T17:31:42.943Z" },
- { url = "https://files.pythonhosted.org/packages/04/a3/603ecdfd5cd97feee59c7e51da4929e22eac8dbe68ac78df53e74152813f/nvtx-0.2.14-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d8cd1f2b464675b4d3c2036b7bbaf975baa9307f0795107dc69c556c0c8d191d", size = 710057, upload-time = "2025-11-27T17:28:08.127Z" },
- { url = "https://files.pythonhosted.org/packages/97/29/945dd440e6bd459e6064f321ed425dbae7d03d39ffa97a38e5434fbcda27/nvtx-0.2.14-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6532556d81f782e24eb12c5e0c75e297493d6ab0431177c93c12bb29c523ea9e", size = 717825, upload-time = "2025-11-27T17:22:57.556Z" },
- { url = "https://files.pythonhosted.org/packages/16/3e/5d7872f2a0809237e3d524f81a7a3c7fbeb98bdc9dcec4723b75a45cd552/nvtx-0.2.14-cp313-cp313-win_amd64.whl", hash = "sha256:cd86f78ed56aede301b03e5ab8cb1aaeb8ba0b5ed683f98f87fbe474996d73f2", size = 118546, upload-time = "2025-11-27T17:30:32.549Z" },
- { url = "https://files.pythonhosted.org/packages/ef/04/1c8b1ce8b729a96218c1d9c0d399ea556765ab2199311ca9e1693507834d/nvtx-0.2.14-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:51d48a98db0c3f4b701d3422ef34bf34c0c9256036d036dd115d48c6286b7b82", size = 791447, upload-time = "2025-11-28T22:52:07.744Z" },
- { url = "https://files.pythonhosted.org/packages/72/a8/608bfa862de1673e63386b0e32520a05ed968524c22babe273565a1c9027/nvtx-0.2.14-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:638f66b6119fb3adfe3f5e2ba2d0cca9580bc4a898cd702b639d727a4a405c59", size = 742277, upload-time = "2025-11-28T22:56:48.341Z" },
- { url = "https://files.pythonhosted.org/packages/32/bb/579545bb24e4d1d643e42c9e323d32fcf327522027346686c12595f15ed9/nvtx-0.2.14-cp313-cp313t-win_amd64.whl", hash = "sha256:d5dfaf02a91fd2a123e104d59681dc768c07b66b05e4afc4c05ee125e45f6261", size = 131705, upload-time = "2025-11-28T22:57:30.24Z" },
- { url = "https://files.pythonhosted.org/packages/07/60/9b4ed6dd0153b17817d3344f444bed731d284907c99a4fcc0910a594b114/nvtx-0.2.14-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:12c21b01b426e85054606d5d8e78d08ab804f1231d4f24be6ded595f901b1125", size = 740863, upload-time = "2025-11-28T22:53:34.642Z" },
- { url = "https://files.pythonhosted.org/packages/93/e5/c4095778d690c8eac535048c44f4aff61e77ad0573b324655e3c8d4b7b86/nvtx-0.2.14-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:296aa978e572d2854a980506cb9de6fe641d496b46879b56c6e6df7467056303", size = 747776, upload-time = "2025-11-28T22:55:54.441Z" },
- { url = "https://files.pythonhosted.org/packages/c9/3f/05150e9953b6e818b2c103ff881a43c99063cf06f7e9b474f94f79674fcc/nvtx-0.2.14-cp314-cp314-win_amd64.whl", hash = "sha256:e265cce4d7ecfb56b9e974be736bba308be47402edfc09dd6a5f91a8eafa90c3", size = 120583, upload-time = "2025-11-28T22:58:37.289Z" },
- { url = "https://files.pythonhosted.org/packages/0d/4c/b607cb591d4600ff1771e64563cf6b395024ffad0b13fe09aa10f7b8d786/nvtx-0.2.14-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fc5522766fff59cf62e42c31324b1c405d308d7755e847e25d286f29e217f54a", size = 794398, upload-time = "2025-11-28T22:51:16.927Z" },
- { url = "https://files.pythonhosted.org/packages/f0/af/9d67e2995673e25711ee79bcc52a552926c074943fc59b42fa56996ad50f/nvtx-0.2.14-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:671b592464038054cc31a5d8c53a460d22fc38b066bbd055e086be8dd49fa43b", size = 746054, upload-time = "2025-11-28T22:55:33.65Z" },
- { url = "https://files.pythonhosted.org/packages/5f/10/143ce5b3e07921176fc2b6f808afde7335f06e93af1a29ac6f4cfa02cf4b/nvtx-0.2.14-cp314-cp314t-win_amd64.whl", hash = "sha256:2567ce29e905062c239a33ba91a46ca7307561c40fd7b37ec64c00cd78f9bdab", size = 138050, upload-time = "2025-11-28T22:57:09.773Z" },
+version = "0.2.15"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/92/dd/692765e87de30bae1522cdffaa0f2b52949658a92a0fa6d96b1a01eae9d2/nvtx-0.2.15.tar.gz", hash = "sha256:2287d3be05b85661deb386f878d1f536c2e532774aa9ec7a50c434942ed81ae5", size = 121230, upload-time = "2026-03-18T10:01:25.547Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c2/07/698355285a03a366ef63ea9762fc1feef3f9f25483e1655408f72d827090/nvtx-0.2.15-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2cc530cd0f1a2c14a3a7e683833db509888ac5ed4ead94e5c9e2c7317c6937a7", size = 807159, upload-time = "2026-03-18T10:09:49.232Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/d1/08f22448d83481408d663065764ba583df091a7de629ed38fc97e522f1af/nvtx-0.2.15-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3ca8030a6d197952318013dd1c12c22da1d4b9feb76ba72e0fcd449961183c2c", size = 806187, upload-time = "2026-03-18T10:13:32.972Z" },
+ { url = "https://files.pythonhosted.org/packages/54/23/c97c39e3b7ba256aa343cb828ca0d1c8421f705ca84795658ecd14ca95ed/nvtx-0.2.15-cp312-cp312-win_amd64.whl", hash = "sha256:70a1e768964e0520b68ccabc4df391cc227537c45936a7eba6507bc65e617e00", size = 129178, upload-time = "2026-03-18T10:02:55.299Z" },
+ { url = "https://files.pythonhosted.org/packages/05/c9/8341224b8284f7deb6a634119939de5885adc421e64b6743693b30da2186/nvtx-0.2.15-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d28660d9c46f8ba750d781572b6aa5a1e6221abba224ab32d7fb32c2d0fd67df", size = 780787, upload-time = "2026-03-18T10:10:40.634Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/c0/4a5bb7897918de7c7e0191d9342df8ae4cb797ff07276e0f20d13e497ce7/nvtx-0.2.15-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:10749686633f880ad53dcdbb2179fad41b45dcf5b7631d4a1070a577577bd386", size = 782575, upload-time = "2026-03-18T10:13:57.3Z" },
+ { url = "https://files.pythonhosted.org/packages/38/b9/6b381ac7c5a3ded331aebbf25f8959d19b51d320fb2514c76c6b6edddaaa/nvtx-0.2.15-cp313-cp313-win_amd64.whl", hash = "sha256:a6650b029263d12f8427a4dee8bd59cb9c91bccb60543bfcb20bc2b00fdcd672", size = 128764, upload-time = "2026-03-18T10:02:33.343Z" },
+ { url = "https://files.pythonhosted.org/packages/75/69/a9acb6d95d2e0e381b2956544768528dd8d7a9e827af8c2014169d838284/nvtx-0.2.15-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:25813ead4fff4d3a6e04f69a72507b096a6bdbecefa369f1100b0e584767bca8", size = 833375, upload-time = "2026-03-18T10:06:31.955Z" },
+ { url = "https://files.pythonhosted.org/packages/38/56/c7e8645061cc2fc23f3a54f33e1e340df59216f07dcfb97d46b8ae7dd26c/nvtx-0.2.15-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3741edac4678b92f03d22a3f0a2dfd469f422f85e63db71b038e02525b2404ad", size = 788639, upload-time = "2026-03-18T10:12:01.69Z" },
+ { url = "https://files.pythonhosted.org/packages/96/03/fadd82acdbca6d1c49ac517081a0c3714346f52f4c7e1d4449d77605b4aa/nvtx-0.2.15-cp313-cp313t-win_amd64.whl", hash = "sha256:8be06c3c8c267eba56a0396366b9593092e0b75ea8d3702b303d48c0a1662f0e", size = 142609, upload-time = "2026-03-18T10:01:48.832Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/5b/ca0ba6fa769d08174b7a5b4775c279e2e26611cdd5e7833aa699187871c7/nvtx-0.2.15-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b5171b8283dd3ea9ae688a86d16901b4c2c142c4eb0a4bdbf6c222f5f67f9524", size = 781769, upload-time = "2026-03-18T10:08:59.357Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/e1/e02fafc01c18f1868a2d2c030953f49e38d65f2d95884789a6c46ff308f1/nvtx-0.2.15-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3c6d0f27d4f8a2f479eb64a6b842c13aee32120348a1715d995b9bb9f75b35cf", size = 774614, upload-time = "2026-03-18T10:12:46.979Z" },
+ { url = "https://files.pythonhosted.org/packages/20/77/a2b64335bab7c75fe1c054cc4ebe2d3b3234cbdb04d2e1d6ca73551c54f5/nvtx-0.2.15-cp314-cp314-win_amd64.whl", hash = "sha256:9934fad0b441cfa6e896a848b092498ba23e2ff205c2b9a7b60520ff8367ffef", size = 130932, upload-time = "2026-03-18T10:03:43.507Z" },
+ { url = "https://files.pythonhosted.org/packages/db/24/528619230976c18364eda2340906ea67b3bf7588b7ce59e054723614abae/nvtx-0.2.15-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:aca61135c76b8107ae3c994325613afa661e1336a991c59cc9c6176829b3b32c", size = 834439, upload-time = "2026-03-18T10:05:01.181Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/7b/c1b96f13ef89bdf2a8c2f326a97bed89699271990d7c8624fda3fedc6e61/nvtx-0.2.15-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:58653bf6fd8453947b9e5153da2ad7aeb0ceafa030de7f133efb3eada5da7ca7", size = 790247, upload-time = "2026-03-18T10:11:39.124Z" },
+ { url = "https://files.pythonhosted.org/packages/14/5d/e000de781d92b732d52c572517db0e9e3a0085795f8bdc18201713c52d1f/nvtx-0.2.15-cp314-cp314t-win_amd64.whl", hash = "sha256:9d1d10db4fb4a3b0ffd6ed37bf25f0a966a3b4d34b3c9abb1f6572732959a6e5", size = 149109, upload-time = "2026-03-18T10:03:21.615Z" },
+]
+
+[[package]]
+name = "omegaconf"
+version = "2.3.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "antlr4-python3-runtime" },
+ { name = "pyyaml" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/09/48/6388f1bb9da707110532cb70ec4d2822858ddfb44f1cdf1233c20a80ea4b/omegaconf-2.3.0.tar.gz", hash = "sha256:d5d4b6d29955cc50ad50c46dc269bcd92c6e00f5f90d23ab5fee7bfca4ba4cc7", size = 3298120, upload-time = "2022-12-08T20:59:22.753Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e3/94/1843518e420fa3ed6919835845df698c7e27e183cb997394e4a670973a65/omegaconf-2.3.0-py3-none-any.whl", hash = "sha256:7b4df175cdb08ba400f45cae3bdcae7ba8365db4d165fc65fd04b050ab63b46b", size = 79500, upload-time = "2022-12-08T20:59:19.686Z" },
+]
+
+[[package]]
+name = "onnx"
+version = "1.19.0"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "ml-dtypes", version = "0.4.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" },
+ { name = "protobuf", marker = "python_full_version >= '3.13'" },
+ { name = "typing-extensions", marker = "python_full_version >= '3.13'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/5b/bf/b0a63ee9f3759dcd177b28c6f2cb22f2aecc6d9b3efecaabc298883caa5f/onnx-1.19.0.tar.gz", hash = "sha256:aa3f70b60f54a29015e41639298ace06adf1dd6b023b9b30f1bca91bb0db9473", size = 11949859, upload-time = "2025-08-27T02:34:27.107Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0d/94/f56f6ca5e2f921b28c0f0476705eab56486b279f04e1d568ed64c14e7764/onnx-1.19.0-cp312-cp312-macosx_12_0_universal2.whl", hash = "sha256:61d94e6498ca636756f8f4ee2135708434601b2892b7c09536befb19bc8ca007", size = 18322331, upload-time = "2025-08-27T02:33:20.373Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/00/8cc3f3c40b54b28f96923380f57c9176872e475face726f7d7a78bd74098/onnx-1.19.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:224473354462f005bae985c72028aaa5c85ab11de1b71d55b06fdadd64a667dd", size = 18027513, upload-time = "2025-08-27T02:33:23.44Z" },
+ { url = "https://files.pythonhosted.org/packages/61/90/17c4d2566fd0117a5e412688c9525f8950d467f477fbd574e6b32bc9cb8d/onnx-1.19.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1ae475c85c89bc4d1f16571006fd21a3e7c0e258dd2c091f6e8aafb083d1ed9b", size = 18202278, upload-time = "2025-08-27T02:33:26.103Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/6e/a9383d9cf6db4ac761a129b081e9fa5d0cd89aad43cf1e3fc6285b915c7d/onnx-1.19.0-cp312-cp312-win32.whl", hash = "sha256:323f6a96383a9cdb3960396cffea0a922593d221f3929b17312781e9f9b7fb9f", size = 16333080, upload-time = "2025-08-27T02:33:28.559Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/2e/3ff480a8c1fa7939662bdc973e41914add2d4a1f2b8572a3c39c2e4982e5/onnx-1.19.0-cp312-cp312-win_amd64.whl", hash = "sha256:50220f3499a499b1a15e19451a678a58e22ad21b34edf2c844c6ef1d9febddc2", size = 16453927, upload-time = "2025-08-27T02:33:31.177Z" },
+ { url = "https://files.pythonhosted.org/packages/57/37/ad500945b1b5c154fe9d7b826b30816ebd629d10211ea82071b5bcc30aa4/onnx-1.19.0-cp312-cp312-win_arm64.whl", hash = "sha256:efb768299580b786e21abe504e1652ae6189f0beed02ab087cd841cb4bb37e43", size = 16426022, upload-time = "2025-08-27T02:33:33.515Z" },
+ { url = "https://files.pythonhosted.org/packages/be/29/d7b731f63d243f815d9256dce0dca3c151dcaa1ac59f73e6ee06c9afbe91/onnx-1.19.0-cp313-cp313-macosx_12_0_universal2.whl", hash = "sha256:9aed51a4b01acc9ea4e0fe522f34b2220d59e9b2a47f105ac8787c2e13ec5111", size = 18322412, upload-time = "2025-08-27T02:33:36.723Z" },
+ { url = "https://files.pythonhosted.org/packages/58/f5/d3106becb42cb374f0e17ff4c9933a97f1ee1d6a798c9452067f7d3ff61b/onnx-1.19.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ce2cdc3eb518bb832668c4ea9aeeda01fbaa59d3e8e5dfaf7aa00f3d37119404", size = 18026565, upload-time = "2025-08-27T02:33:39.493Z" },
+ { url = "https://files.pythonhosted.org/packages/83/fa/b086d17bab3900754c7ffbabfb244f8e5e5da54a34dda2a27022aa2b373b/onnx-1.19.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8b546bd7958734b6abcd40cfede3d025e9c274fd96334053a288ab11106bd0aa", size = 18202077, upload-time = "2025-08-27T02:33:42.115Z" },
+ { url = "https://files.pythonhosted.org/packages/35/f2/5e2dfb9d4cf873f091c3f3c6d151f071da4295f9893fbf880f107efe3447/onnx-1.19.0-cp313-cp313-win32.whl", hash = "sha256:03086bffa1cf5837430cf92f892ca0cd28c72758d8905578c2bf8ffaf86c6743", size = 16333198, upload-time = "2025-08-27T02:33:45.172Z" },
+ { url = "https://files.pythonhosted.org/packages/79/67/b3751a35c2522f62f313156959575619b8fa66aa883db3adda9d897d8eb2/onnx-1.19.0-cp313-cp313-win_amd64.whl", hash = "sha256:1715b51eb0ab65272e34ef51cb34696160204b003566cd8aced2ad20a8f95cb8", size = 16453836, upload-time = "2025-08-27T02:33:47.779Z" },
+ { url = "https://files.pythonhosted.org/packages/14/b9/1df85effc960fbbb90bb7bc36eb3907c676b104bc2f88bce022bcfdaef63/onnx-1.19.0-cp313-cp313-win_arm64.whl", hash = "sha256:6bf5acdb97a3ddd6e70747d50b371846c313952016d0c41133cbd8f61b71a8d5", size = 16425877, upload-time = "2025-08-27T02:33:50.357Z" },
+ { url = "https://files.pythonhosted.org/packages/23/2b/089174a1427be9149f37450f8959a558ba20f79fca506ba461d59379d3a1/onnx-1.19.0-cp313-cp313t-macosx_12_0_universal2.whl", hash = "sha256:46cf29adea63e68be0403c68de45ba1b6acc9bb9592c5ddc8c13675a7c71f2cb", size = 18348546, upload-time = "2025-08-27T02:33:56.132Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/d6/3458f0e3a9dc7677675d45d7d6528cb84ad321c8670cc10c69b32c3e03da/onnx-1.19.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:246f0de1345498d990a443d55a5b5af5101a3e25a05a2c3a5fe8b7bd7a7d0707", size = 18033067, upload-time = "2025-08-27T02:33:58.661Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/16/6e4130e1b4b29465ee1fb07d04e8d6f382227615c28df8f607ba50909e2a/onnx-1.19.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ae0d163ffbc250007d984b8dd692a4e2e4506151236b50ca6e3560b612ccf9ff", size = 18205741, upload-time = "2025-08-27T02:34:01.538Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/d8/f64d010fd024b2a2b11ce0c4ee179e4f8f6d4ccc95f8184961c894c22af1/onnx-1.19.0-cp313-cp313t-win_amd64.whl", hash = "sha256:7c151604c7cca6ae26161c55923a7b9b559df3344938f93ea0074d2d49e7fe78", size = 16453839, upload-time = "2025-08-27T02:34:06.515Z" },
+ { url = "https://files.pythonhosted.org/packages/67/ec/8761048eabef4dad55af4c002c672d139b9bd47c3616abaed642a1710063/onnx-1.19.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:236bc0e60d7c0f4159300da639953dd2564df1c195bce01caba172a712e75af4", size = 18027605, upload-time = "2025-08-27T02:34:08.962Z" },
+]
+
+[[package]]
+name = "onnx"
+version = "1.21.0"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "ml-dtypes", version = "0.5.4", source = { registry = "https://pypi.org/simple" } },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.13' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
+ { name = "protobuf" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/c5/93/942d2a0f6a70538eea042ce0445c8aefd46559ad153469986f29a743c01c/onnx-1.21.0.tar.gz", hash = "sha256:4d8b67d0aaec5864c87633188b91cc520877477ec0254eda122bef8be43cd764", size = 12074608, upload-time = "2026-03-27T21:33:36.118Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7d/ae/cb644ec84c25e63575d9d8790fdcc5d1a11d67d3f62f872edb35fa38d158/onnx-1.21.0-cp312-abi3-macosx_12_0_universal2.whl", hash = "sha256:fc2635400fe39ff37ebc4e75342cc54450eadadf39c540ff132c319bf4960095", size = 17965930, upload-time = "2026-03-27T21:32:48.089Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/b6/eeb5903586645ef8a49b4b7892580438741acc3df91d7a5bd0f3a59ea9cb/onnx-1.21.0-cp312-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9003d5206c01fa2ff4b46311566865d8e493e1a6998d4009ec6de39843f1b59b", size = 17531344, upload-time = "2026-03-27T21:32:50.837Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/00/4823f06357892d1e60d6f34e7299d2ba4ed2108c487cc394f7ce85a3ff14/onnx-1.21.0-cp312-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a9261bd580fb8548c9c37b3c6750387eb8f21ea43c63880d37b2c622e1684285", size = 17613697, upload-time = "2026-03-27T21:32:54.222Z" },
+ { url = "https://files.pythonhosted.org/packages/23/1d/391f3c567ae068c8ac4f1d1316bae97c9eb45e702f05975fe0e17ad441f0/onnx-1.21.0-cp312-abi3-win32.whl", hash = "sha256:9ea4e824964082811938a9250451d89c4ec474fe42dd36c038bfa5df31993d1e", size = 16287200, upload-time = "2026-03-27T21:32:57.277Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/a6/5eefbe5b40ea96de95a766bd2e0e751f35bdea2d4b951991ec9afaa69531/onnx-1.21.0-cp312-abi3-win_amd64.whl", hash = "sha256:458d91948ad9a7729a347550553b49ab6939f9af2cddf334e2116e45467dc61f", size = 16441045, upload-time = "2026-03-27T21:33:00.081Z" },
+ { url = "https://files.pythonhosted.org/packages/63/c4/0ed8dc037a39113d2a4d66e0005e07751c299c46b993f1ad5c2c35664c20/onnx-1.21.0-cp312-abi3-win_arm64.whl", hash = "sha256:ca14bc4842fccc3187eb538f07eabeb25a779b39388b006db4356c07403a7bbb", size = 16403134, upload-time = "2026-03-27T21:33:03.987Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/89/0e1a9beb536401e2f45ac88735e123f2735e12fc7b56ff6c11727e097526/onnx-1.21.0-cp313-cp313t-macosx_12_0_universal2.whl", hash = "sha256:257d1d1deb6a652913698f1e3f33ef1ca0aa69174892fe38946d4572d89dd94f", size = 17975430, upload-time = "2026-03-27T21:33:07.005Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/46/e6dc71a7b3b317265591b20a5f71d0ff5c0d26c24e52283139dc90c66038/onnx-1.21.0-cp313-cp313t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7cd7cb8f6459311bdb557cbf6c0ccc6d8ace11c304d1bba0a30b4a4688e245f8", size = 17537435, upload-time = "2026-03-27T21:33:09.765Z" },
+ { url = "https://files.pythonhosted.org/packages/49/2e/27affcac63eaf2ef183a44fd1a1354b11da64a6c72fe6f3fdcf5571bcee5/onnx-1.21.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7b58a4cfec8d9311b73dc083e4c1fa362069267881144c05139b3eba5dc3a840", size = 17617687, upload-time = "2026-03-27T21:33:12.619Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/5c/ac8ed15e941593a3672ce424280b764979026317811f2e8508432bfc3429/onnx-1.21.0-cp313-cp313t-win_amd64.whl", hash = "sha256:1a9baf882562c4cebf79589bebb7cd71a20e30b51158cac3e3bbaf27da6163bd", size = 16449402, upload-time = "2026-03-27T21:33:15.555Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/aa/d2231e0dcaad838217afc64c306c8152a080134d2034e247cc973d577674/onnx-1.21.0-cp313-cp313t-win_arm64.whl", hash = "sha256:bba12181566acf49b35875838eba49536a327b2944664b17125577d230c637ad", size = 16408273, upload-time = "2026-03-27T21:33:18.599Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/0a/8905b14694def6ad23edf1011fdd581500384062f8c4c567e114be7aa272/onnx-1.21.0-cp314-cp314t-macosx_12_0_universal2.whl", hash = "sha256:7ee9d8fd6a4874a5fa8b44bbcabea104ce752b20469b88bc50c7dcf9030779ad", size = 17975331, upload-time = "2026-03-27T21:33:21.69Z" },
+ { url = "https://files.pythonhosted.org/packages/61/28/f4e401e5199d1b9c8b76c7e7ae1169e050515258e877b58fa8bb49d3bdcc/onnx-1.21.0-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5489f25fe461e7f32128218251a466cabbeeaf1eaa791c79daebf1a80d5a2cc9", size = 17537430, upload-time = "2026-03-27T21:33:24.547Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/cf/5d13320eb3660d5af360ea3b43aa9c63a70c92a9b4d1ea0d34501a32fcb8/onnx-1.21.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:db17fc0fec46180b6acbd1d5d8650a04e5527c02b09381da0b5b888d02a204c8", size = 17617662, upload-time = "2026-03-27T21:33:27.418Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/50/3eaa1878338247be021e6423696813d61e77e534dccbd15a703a144e703d/onnx-1.21.0-cp314-cp314t-win_amd64.whl", hash = "sha256:19d9971a3e52a12968ae6c70fd0f86c349536de0b0c33922ecdbe52d1972fe60", size = 16463688, upload-time = "2026-03-27T21:33:30.229Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/48/38d46b43bbb525e0b6a4c2c4204cc6795d67e45687a2f7403e06d8e7053d/onnx-1.21.0-cp314-cp314t-win_arm64.whl", hash = "sha256:efba467efb316baf2a9452d892c2f982b9b758c778d23e38c7f44fa211b30bb9", size = 16423387, upload-time = "2026-03-27T21:33:33.446Z" },
+]
+
+[[package]]
+name = "onnx-ir"
+version = "0.1.8"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "ml-dtypes", version = "0.4.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" },
+ { name = "onnx", version = "1.19.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" },
+ { name = "typing-extensions", marker = "python_full_version >= '3.13'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/af/4a/7ea3952e556e7281b8bfe7f7fce016a13fdac85544d6d6af8ebca5cae160/onnx_ir-0.1.8.tar.gz", hash = "sha256:85ea59eaf165b2b107788193480a260e2723cfc7a1dac1bde7085fd0b7e380d7", size = 108961, upload-time = "2025-09-05T15:45:33.887Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0f/1c/3bb51fa9e278cbc655a1943c8016163d76a6e24137e73e5198ebc20fc965/onnx_ir-0.1.8-py3-none-any.whl", hash = "sha256:61a42021b6249e566ff3b89a03342bc88dce4dc2d984b97cfb060f33ef179f8a", size = 125316, upload-time = "2025-09-05T15:45:31.211Z" },
+]
+
+[[package]]
+name = "onnx-ir"
+version = "0.2.0"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "ml-dtypes", version = "0.5.4", source = { registry = "https://pypi.org/simple" } },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.13' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
+ { name = "onnx", version = "1.21.0", source = { registry = "https://pypi.org/simple" } },
+ { name = "sympy" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/b2/a5/acc43c8fa6edbc584d127fb6bbd13ae9ebfc01b9675c74e0da2de15fa4a6/onnx_ir-0.2.0.tar.gz", hash = "sha256:8bad3906691987290789b26d05e0dbff467029a0b1e411e12e4cae02e43503e4", size = 141693, upload-time = "2026-02-24T02:31:10.998Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4a/df/a99736bcca6b16e36c687ce4996abcf4ce73c514fddd9e730cfcb6a334f2/onnx_ir-0.2.0-py3-none-any.whl", hash = "sha256:eb14d1399c2442bd1ff702719e70074e9cedfa3af5729416a32752c9e0f82591", size = 164100, upload-time = "2026-02-24T02:31:09.454Z" },
+]
+
+[[package]]
+name = "onnxscript"
+version = "0.5.0"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "ml-dtypes", version = "0.4.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" },
+ { name = "onnx", version = "1.19.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" },
+ { name = "onnx-ir", version = "0.1.8", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" },
+ { name = "typing-extensions", marker = "python_full_version >= '3.13'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/f5/2f/0bb2b6ca727e4d5173f640527f402ab4225def4bc8d667269b83047be8c4/onnxscript-0.5.0.tar.gz", hash = "sha256:4aba215e1f80fbcd07ba0d97d6bca96797fc3e9639eacb5434d35317ce1406aa", size = 588762, upload-time = "2025-09-12T16:57:46.484Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e7/f7/f0eb0b10771637a8c176a3b0594c65c5ba3cea440847741297901cef2c5e/onnxscript-0.5.0-py3-none-any.whl", hash = "sha256:da33715ac8ec80e0263a5200f1ad1b3532225804c05a13a0d6ea83712b5b4a8f", size = 684685, upload-time = "2025-09-12T16:57:48.869Z" },
+]
+
+[[package]]
+name = "onnxscript"
+version = "0.6.2"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "ml-dtypes", version = "0.5.4", source = { registry = "https://pypi.org/simple" } },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.13' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
+ { name = "onnx", version = "1.21.0", source = { registry = "https://pypi.org/simple" } },
+ { name = "onnx-ir", version = "0.2.0", source = { registry = "https://pypi.org/simple" } },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.13' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e7/2b/538fdeb0e25bed5d7e0f954af5710543e2629499fb74381afc3333f8a8ae/onnxscript-0.6.2.tar.gz", hash = "sha256:abb2e6f464db40c9b8c7fbb3e64cca04cf3f4495e67c4eda5eac17b784191ce3", size = 590865, upload-time = "2026-02-10T22:53:39.638Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/66/56/e6b179397497ab93266b6eb00743403a6a699a29063a423c4a14595d3db9/onnxscript-0.6.2-py3-none-any.whl", hash = "sha256:20e3c3fd1da19b3655549d5455a2df719db47374fe430e01e865ae69127c37b9", size = 689064, upload-time = "2026-02-10T22:53:41.663Z" },
+]
+
+[[package]]
+name = "openai"
+version = "2.30.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "anyio" },
+ { name = "distro" },
+ { name = "httpx" },
+ { name = "jiter" },
+ { name = "pydantic" },
+ { name = "sniffio" },
+ { name = "tqdm" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/88/15/52580c8fbc16d0675d516e8749806eda679b16de1e4434ea06fb6feaa610/openai-2.30.0.tar.gz", hash = "sha256:92f7661c990bda4b22a941806c83eabe4896c3094465030dd882a71abe80c885", size = 676084, upload-time = "2026-03-25T22:08:59.96Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2a/9e/5bfa2270f902d5b92ab7d41ce0475b8630572e71e349b2a4996d14bdda93/openai-2.30.0-py3-none-any.whl", hash = "sha256:9a5ae616888eb2748ec5e0c5b955a51592e0b201a11f4262db920f2a78c5231d", size = 1146656, upload-time = "2026-03-25T22:08:58.2Z" },
+]
+
+[package.optional-dependencies]
+aiohttp = [
+ { name = "aiohttp" },
+ { name = "httpx-aiohttp" },
+]
+
+[[package]]
+name = "opencensus"
+version = "0.11.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "google-api-core" },
+ { name = "opencensus-context" },
+ { name = "six" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/15/a7/a46dcffa1b63084f9f17fe3c8cb20724c4c8f91009fd0b2cfdb27d5d2b35/opencensus-0.11.4.tar.gz", hash = "sha256:cbef87d8b8773064ab60e5c2a1ced58bbaa38a6d052c41aec224958ce544eff2", size = 64966, upload-time = "2024-01-03T18:04:07.085Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b5/ed/9fbdeb23a09e430d87b7d72d430484b88184633dc50f6bfb792354b6f661/opencensus-0.11.4-py2.py3-none-any.whl", hash = "sha256:a18487ce68bc19900336e0ff4655c5a116daf10c1b3685ece8d971bddad6a864", size = 128225, upload-time = "2024-01-03T18:04:05.127Z" },
+]
+
+[[package]]
+name = "opencensus-context"
+version = "0.1.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/4c/96/3b6f638f6275a8abbd45e582448723bffa29c1fb426721dedb5c72f7d056/opencensus-context-0.1.3.tar.gz", hash = "sha256:a03108c3c10d8c80bb5ddf5c8a1f033161fa61972a9917f9b9b3a18517f0088c", size = 4066, upload-time = "2022-08-03T22:20:22.359Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/10/68/162c97ea78c957d68ecf78a5c5041d2e25bd5562bdf5d89a6cbf7f8429bf/opencensus_context-0.1.3-py2.py3-none-any.whl", hash = "sha256:073bb0590007af276853009fac7e4bab1d523c3f03baf4cb4511ca38967c6039", size = 5060, upload-time = "2022-08-03T22:20:20.352Z" },
+]
+
+[[package]]
+name = "opentelemetry-api"
+version = "1.33.1"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "deprecated", marker = "extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
+ { name = "importlib-metadata", marker = "extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/9a/8d/1f5a45fbcb9a7d87809d460f09dc3399e3fbd31d7f3e14888345e9d29951/opentelemetry_api-1.33.1.tar.gz", hash = "sha256:1c6055fc0a2d3f23a50c7e17e16ef75ad489345fd3df1f8b8af7c0bbf8a109e8", size = 65002, upload-time = "2025-05-16T18:52:41.146Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/05/44/4c45a34def3506122ae61ad684139f0bbc4e00c39555d4f7e20e0e001c8a/opentelemetry_api-1.33.1-py3-none-any.whl", hash = "sha256:4db83ebcf7ea93e64637ec6ee6fabee45c5cbe4abd9cf3da95c43828ddb50b83", size = 65771, upload-time = "2025-05-16T18:52:17.419Z" },
+]
+
+[[package]]
+name = "opentelemetry-api"
+version = "1.40.0"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "importlib-metadata", marker = "(extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts')" },
+ { name = "typing-extensions", marker = "(extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts')" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/2c/1d/4049a9e8698361cc1a1aa03a6c59e4fa4c71e0c0f94a30f988a6876a2ae6/opentelemetry_api-1.40.0.tar.gz", hash = "sha256:159be641c0b04d11e9ecd576906462773eb97ae1b657730f0ecf64d32071569f", size = 70851, upload-time = "2026-03-04T14:17:21.555Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5f/bf/93795954016c522008da367da292adceed71cca6ee1717e1d64c83089099/opentelemetry_api-1.40.0-py3-none-any.whl", hash = "sha256:82dd69331ae74b06f6a874704be0cfaa49a1650e1537d4a813b86ecef7d0ecf9", size = 68676, upload-time = "2026-03-04T14:17:01.24Z" },
+]
+
+[[package]]
+name = "opentelemetry-exporter-prometheus"
+version = "0.54b1"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "opentelemetry-api", version = "1.33.1", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
+ { name = "opentelemetry-sdk", version = "1.33.1", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
+ { name = "prometheus-client", marker = "extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/da/ef/563c6413dbf042f1b738c4e23f7ff5f80fd0ae5ba64037433a5eeb0a1f79/opentelemetry_exporter_prometheus-0.54b1.tar.gz", hash = "sha256:6a28fde40ac8693bd653b84ba9deff75721fd05edf4e4313939327ea336ad3a9", size = 14948, upload-time = "2025-05-16T18:52:46.152Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f2/4e/f351f6fe640c270158cf2521978978eb8a8fb3113fdaa5e98eedc22e8126/opentelemetry_exporter_prometheus-0.54b1-py3-none-any.whl", hash = "sha256:78052c9818140021b8b3738f653f8bf4088a34bf970144c6816b5f561c0178dc", size = 12950, upload-time = "2025-05-16T18:52:26.51Z" },
]
[[package]]
-name = "omegaconf"
-version = "2.3.0"
+name = "opentelemetry-exporter-prometheus"
+version = "0.61b0"
source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
dependencies = [
- { name = "antlr4-python3-runtime" },
- { name = "pyyaml" },
+ { name = "opentelemetry-api", version = "1.40.0", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts')" },
+ { name = "opentelemetry-sdk", version = "1.40.0", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts')" },
+ { name = "prometheus-client", marker = "(extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts')" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/09/48/6388f1bb9da707110532cb70ec4d2822858ddfb44f1cdf1233c20a80ea4b/omegaconf-2.3.0.tar.gz", hash = "sha256:d5d4b6d29955cc50ad50c46dc269bcd92c6e00f5f90d23ab5fee7bfca4ba4cc7", size = 3298120, upload-time = "2022-12-08T20:59:22.753Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/4a/20/9e818fd364d12e8d0cfdce4a3b2d82e24d98c4ceebb315de6b6770b5f214/opentelemetry_exporter_prometheus-0.61b0.tar.gz", hash = "sha256:7c4919bd8e79abd62b610767e80f42c9c3a06c5183f4dd9141eedeb57aea284b", size = 15136, upload-time = "2026-03-04T14:17:26.275Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/e3/94/1843518e420fa3ed6919835845df698c7e27e183cb997394e4a670973a65/omegaconf-2.3.0-py3-none-any.whl", hash = "sha256:7b4df175cdb08ba400f45cae3bdcae7ba8365db4d165fc65fd04b050ab63b46b", size = 79500, upload-time = "2022-12-08T20:59:19.686Z" },
+ { url = "https://files.pythonhosted.org/packages/02/4a/b65d40e94d1d930aee73a1a2857211ee6ab10ce3686cbdae5eea78cd9d34/opentelemetry_exporter_prometheus-0.61b0-py3-none-any.whl", hash = "sha256:3013b41f4370143d48d219a2351473761423e5882fa4c213811eaefacba39cb7", size = 13149, upload-time = "2026-03-04T14:17:08.983Z" },
]
[[package]]
-name = "onnx"
-version = "1.20.1"
+name = "opentelemetry-proto"
+version = "1.40.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
- { name = "ml-dtypes" },
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version < '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version >= '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
{ name = "protobuf" },
- { name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/3b/8a/335c03a8683a88a32f9a6bb98899ea6df241a41df64b37b9696772414794/onnx-1.20.1.tar.gz", hash = "sha256:ded16de1df563d51fbc1ad885f2a426f814039d8b5f4feb77febe09c0295ad67", size = 12048980, upload-time = "2026-01-10T01:40:03.043Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/79/cc/4ba3c80cfaffdb541dc5a23eaccb045a627361e94ecaeba30496270f15b3/onnx-1.20.1-cp310-cp310-macosx_12_0_universal2.whl", hash = "sha256:3fe243e83ad737637af6512708454e720d4b0864def2b28e6b0ee587b80a50be", size = 17904206, upload-time = "2026-01-10T01:38:58.574Z" },
- { url = "https://files.pythonhosted.org/packages/f3/fc/3a1c4ae2cd5cfab2d0ebc1842769b04b417fe13946144a7c8ce470dd9c85/onnx-1.20.1-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e24e96b48f27e4d6b44cb0b195b367a2665da2d819621eec51903d575fc49d38", size = 17414849, upload-time = "2026-01-10T01:39:01.494Z" },
- { url = "https://files.pythonhosted.org/packages/a4/ab/5017945291b981f2681fb620f2d5b6070e02170c648770711ef1eac79d56/onnx-1.20.1-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0903e6088ed5e8f59ebd381ab2a6e9b2a60b4c898f79aa2fe76bb79cf38a5031", size = 17513600, upload-time = "2026-01-10T01:39:04.348Z" },
- { url = "https://files.pythonhosted.org/packages/2e/b0/063e79dc365972af876d786bacc6acd8909691af2b9296615ff74ad182f3/onnx-1.20.1-cp310-cp310-win32.whl", hash = "sha256:17483e59082b2ca6cadd2b48fd8dce937e5b2c985ed5583fefc38af928be1826", size = 16239159, upload-time = "2026-01-10T01:39:07.254Z" },
- { url = "https://files.pythonhosted.org/packages/2a/73/a992271eb3683e676239d71b5a78ad3cf4d06d2223c387e701bf305da199/onnx-1.20.1-cp310-cp310-win_amd64.whl", hash = "sha256:e2b0cf797faedfd3b83491dc168ab5f1542511448c65ceb482f20f04420cbf3a", size = 16391718, upload-time = "2026-01-10T01:39:09.96Z" },
- { url = "https://files.pythonhosted.org/packages/0c/38/1a0e74d586c08833404100f5c052f92732fb5be417c0b2d7cb0838443bfe/onnx-1.20.1-cp311-cp311-macosx_12_0_universal2.whl", hash = "sha256:53426e1b458641e7a537e9f176330012ff59d90206cac1c1a9d03cdd73ed3095", size = 17904965, upload-time = "2026-01-10T01:39:13.532Z" },
- { url = "https://files.pythonhosted.org/packages/96/25/64b076e9684d17335f80b15b3bf502f7a8e1a89f08a6b208d4f2861b3011/onnx-1.20.1-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ca7281f8c576adf396c338cf43fff26faee8d4d2e2577b8e73738f37ceccf945", size = 17415179, upload-time = "2026-01-10T01:39:16.516Z" },
- { url = "https://files.pythonhosted.org/packages/ac/d5/6743b409421ced20ad5af1b3a7b4c4e568689ffaca86db431692fca409a6/onnx-1.20.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2297f428c51c7fc6d8fad0cf34384284dfeff3f86799f8e83ef905451348ade0", size = 17513672, upload-time = "2026-01-10T01:39:19.35Z" },
- { url = "https://files.pythonhosted.org/packages/9a/6b/dae82e6fdb2043302f29adca37522312ea2be55b75907b59be06fbdffe87/onnx-1.20.1-cp311-cp311-win32.whl", hash = "sha256:63d9cbcab8c96841eadeb7c930e07bfab4dde8081eb76fb68e0dfb222706b81e", size = 16239336, upload-time = "2026-01-10T01:39:22.506Z" },
- { url = "https://files.pythonhosted.org/packages/8e/17/a0d7863390c1f2067d7c02dcc1477034965c32aaa1407bfcf775305ffee4/onnx-1.20.1-cp311-cp311-win_amd64.whl", hash = "sha256:d78cde72d7ca8356a2d99c5dc0dbf67264254828cae2c5780184486c0cd7b3bf", size = 16392120, upload-time = "2026-01-10T01:39:25.106Z" },
- { url = "https://files.pythonhosted.org/packages/aa/72/9b879a46eb7a3322223791f36bf9c25d95da9ed93779eabb75a560f22e5b/onnx-1.20.1-cp311-cp311-win_arm64.whl", hash = "sha256:0104bb2d4394c179bcea3df7599a45a2932b80f4633840896fcf0d7d8daecea2", size = 16346923, upload-time = "2026-01-10T01:39:27.782Z" },
- { url = "https://files.pythonhosted.org/packages/7c/4c/4b17e82f91ab9aa07ff595771e935ca73547b035030dc5f5a76e63fbfea9/onnx-1.20.1-cp312-abi3-macosx_12_0_universal2.whl", hash = "sha256:1d923bb4f0ce1b24c6859222a7e6b2f123e7bfe7623683662805f2e7b9e95af2", size = 17903547, upload-time = "2026-01-10T01:39:31.015Z" },
- { url = "https://files.pythonhosted.org/packages/64/5e/1bfa100a9cb3f2d3d5f2f05f52f7e60323b0e20bb0abace1ae64dbc88f25/onnx-1.20.1-cp312-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ddc0b7d8b5a94627dc86c533d5e415af94cbfd103019a582669dad1f56d30281", size = 17412021, upload-time = "2026-01-10T01:39:33.885Z" },
- { url = "https://files.pythonhosted.org/packages/fb/71/d3fec0dcf9a7a99e7368112d9c765154e81da70fcba1e3121131a45c245b/onnx-1.20.1-cp312-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9336b6b8e6efcf5c490a845f6afd7e041c89a56199aeda384ed7d58fb953b080", size = 17510450, upload-time = "2026-01-10T01:39:36.589Z" },
- { url = "https://files.pythonhosted.org/packages/74/a7/edce1403e05a46e59b502fae8e3350ceeac5841f8e8f1561e98562ed9b09/onnx-1.20.1-cp312-abi3-win32.whl", hash = "sha256:564c35a94811979808ab5800d9eb4f3f32c12daedba7e33ed0845f7c61ef2431", size = 16238216, upload-time = "2026-01-10T01:39:39.46Z" },
- { url = "https://files.pythonhosted.org/packages/8b/c7/8690c81200ae652ac550c1df52f89d7795e6cc941f3cb38c9ef821419e80/onnx-1.20.1-cp312-abi3-win_amd64.whl", hash = "sha256:9fe7f9a633979d50984b94bda8ceb7807403f59a341d09d19342dc544d0ca1d5", size = 16389207, upload-time = "2026-01-10T01:39:41.955Z" },
- { url = "https://files.pythonhosted.org/packages/01/a0/4fb0e6d36eaf079af366b2c1f68bafe92df6db963e2295da84388af64abc/onnx-1.20.1-cp312-abi3-win_arm64.whl", hash = "sha256:21d747348b1c8207406fa2f3e12b82f53e0d5bb3958bcd0288bd27d3cb6ebb00", size = 16344155, upload-time = "2026-01-10T01:39:45.536Z" },
- { url = "https://files.pythonhosted.org/packages/ea/bb/715fad292b255664f0e603f1b2ef7bf2b386281775f37406beb99fa05957/onnx-1.20.1-cp313-cp313t-macosx_12_0_universal2.whl", hash = "sha256:29197b768f5acdd1568ddeb0a376407a2817844f6ac1ef8c8dd2d974c9ab27c3", size = 17912296, upload-time = "2026-01-10T01:39:48.21Z" },
- { url = "https://files.pythonhosted.org/packages/2d/c3/541af12c3d45e159a94ee701100ba9e94b7bd8b7a8ac5ca6838569f894f8/onnx-1.20.1-cp313-cp313t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1f0371aa67f51917a09cc829ada0f9a79a58f833449e03d748f7f7f53787c43c", size = 17416925, upload-time = "2026-01-10T01:39:50.82Z" },
- { url = "https://files.pythonhosted.org/packages/2c/3b/d5660a7d2ddf14f531ca66d409239f543bb290277c3f14f4b4b78e32efa3/onnx-1.20.1-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:be1e5522200b203b34327b2cf132ddec20ab063469476e1f5b02bb7bd259a489", size = 17515602, upload-time = "2026-01-10T01:39:54.132Z" },
- { url = "https://files.pythonhosted.org/packages/9c/b4/47225ab2a92562eff87ba9a1a028e3535d659a7157d7cde659003998b8e3/onnx-1.20.1-cp313-cp313t-win_amd64.whl", hash = "sha256:15c815313bbc4b2fdc7e4daeb6e26b6012012adc4d850f4e3b09ed327a7ea92a", size = 16395729, upload-time = "2026-01-10T01:39:57.577Z" },
- { url = "https://files.pythonhosted.org/packages/aa/7d/1bbe626ff6b192c844d3ad34356840cc60fca02e2dea0db95e01645758b1/onnx-1.20.1-cp313-cp313t-win_arm64.whl", hash = "sha256:eb335d7bcf9abac82a0d6a0fda0363531ae0b22cfd0fc6304bff32ee29905def", size = 16348968, upload-time = "2026-01-10T01:40:00.491Z" },
+sdist = { url = "https://files.pythonhosted.org/packages/4c/77/dd38991db037fdfce45849491cb61de5ab000f49824a00230afb112a4392/opentelemetry_proto-1.40.0.tar.gz", hash = "sha256:03f639ca129ba513f5819810f5b1f42bcb371391405d99c168fe6937c62febcd", size = 45667, upload-time = "2026-03-04T14:17:31.194Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b9/b2/189b2577dde745b15625b3214302605b1353436219d42b7912e77fa8dc24/opentelemetry_proto-1.40.0-py3-none-any.whl", hash = "sha256:266c4385d88923a23d63e353e9761af0f47a6ed0d486979777fe4de59dc9b25f", size = 72073, upload-time = "2026-03-04T14:17:16.673Z" },
]
[[package]]
-name = "onnx-ir"
-version = "0.1.14"
+name = "opentelemetry-sdk"
+version = "1.33.1"
source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
dependencies = [
- { name = "ml-dtypes" },
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version < '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version >= '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "onnx" },
- { name = "typing-extensions" },
+ { name = "opentelemetry-api", version = "1.33.1", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
+ { name = "opentelemetry-semantic-conventions", version = "0.54b1", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
+ { name = "typing-extensions", marker = "extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/a5/5b/ebd083a5c3d25ce9f95b34a11b3a492cdcf7831bf127c0f64429a4e83961/onnx_ir-0.1.14.tar.gz", hash = "sha256:bd69e3b5821046d5d7c9d0fdd023f8e1d0cc9a62cbee986fa0e5ab2b1602d7ae", size = 120732, upload-time = "2026-01-07T01:19:47.777Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/67/12/909b98a7d9b110cce4b28d49b2e311797cffdce180371f35eba13a72dd00/opentelemetry_sdk-1.33.1.tar.gz", hash = "sha256:85b9fcf7c3d23506fbc9692fd210b8b025a1920535feec50bd54ce203d57a531", size = 161885, upload-time = "2025-05-16T18:52:52.832Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/53/d1/bd9a5007448b4599a80143b0b5ccc78e9c46176e5e1bee81f6d3da68d217/onnx_ir-0.1.14-py3-none-any.whl", hash = "sha256:89b212fa7840981c5db5dc478190f1b7369536297c3c6eae68fb1c2237dd2554", size = 139128, upload-time = "2026-01-07T01:19:46.403Z" },
+ { url = "https://files.pythonhosted.org/packages/df/8e/ae2d0742041e0bd7fe0d2dcc5e7cce51dcf7d3961a26072d5b43cc8fa2a7/opentelemetry_sdk-1.33.1-py3-none-any.whl", hash = "sha256:19ea73d9a01be29cacaa5d6c8ce0adc0b7f7b4d58cc52f923e4413609f670112", size = 118950, upload-time = "2025-05-16T18:52:37.297Z" },
]
[[package]]
-name = "onnxscript"
-version = "0.5.7"
+name = "opentelemetry-sdk"
+version = "1.40.0"
source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
dependencies = [
- { name = "ml-dtypes" },
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version < '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version >= '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "onnx" },
- { name = "onnx-ir" },
- { name = "packaging" },
- { name = "typing-extensions" },
+ { name = "opentelemetry-api", version = "1.40.0", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts')" },
+ { name = "opentelemetry-semantic-conventions", version = "0.61b0", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts')" },
+ { name = "typing-extensions", marker = "(extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts')" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/2c/f8/358a7d982ea51bc1b0c264f29c08adf096c62ba9f258ba13c954b41c46f5/onnxscript-0.5.7.tar.gz", hash = "sha256:480d572451bc233ed7f742b5005cb0c899594b2fdc28e15167dab26f7fd777ad", size = 596306, upload-time = "2025-12-16T20:47:15.762Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/58/fd/3c3125b20ba18ce2155ba9ea74acb0ae5d25f8cd39cfd37455601b7955cc/opentelemetry_sdk-1.40.0.tar.gz", hash = "sha256:18e9f5ec20d859d268c7cb3c5198c8d105d073714db3de50b593b8c1345a48f2", size = 184252, upload-time = "2026-03-04T14:17:31.87Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/f6/ec/1656ea93be1e50baf429c20603dce249fa3571f3a180407cee79b1afa013/onnxscript-0.5.7-py3-none-any.whl", hash = "sha256:f94a66059c56d13b44908e9b7fd9dae4b4faa6681c784f3fd4c29cfa863e454e", size = 693353, upload-time = "2025-12-16T20:47:17.897Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/c5/6a852903d8bfac758c6dc6e9a68b015d3c33f2f1be5e9591e0f4b69c7e0a/opentelemetry_sdk-1.40.0-py3-none-any.whl", hash = "sha256:787d2154a71f4b3d81f20524a8ce061b7db667d24e46753f32a7bc48f1c1f3f1", size = 141951, upload-time = "2026-03-04T14:17:17.961Z" },
]
[[package]]
-name = "opentelemetry-api"
-version = "1.33.1"
+name = "opentelemetry-semantic-conventions"
+version = "0.54b1"
source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
dependencies = [
- { name = "deprecated" },
- { name = "importlib-metadata" },
+ { name = "deprecated", marker = "extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
+ { name = "opentelemetry-api", version = "1.33.1", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/9a/8d/1f5a45fbcb9a7d87809d460f09dc3399e3fbd31d7f3e14888345e9d29951/opentelemetry_api-1.33.1.tar.gz", hash = "sha256:1c6055fc0a2d3f23a50c7e17e16ef75ad489345fd3df1f8b8af7c0bbf8a109e8", size = 65002, upload-time = "2025-05-16T18:52:41.146Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/5b/2c/d7990fc1ffc82889d466e7cd680788ace44a26789809924813b164344393/opentelemetry_semantic_conventions-0.54b1.tar.gz", hash = "sha256:d1cecedae15d19bdaafca1e56b29a66aa286f50b5d08f036a145c7f3e9ef9cee", size = 118642, upload-time = "2025-05-16T18:52:53.962Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/05/44/4c45a34def3506122ae61ad684139f0bbc4e00c39555d4f7e20e0e001c8a/opentelemetry_api-1.33.1-py3-none-any.whl", hash = "sha256:4db83ebcf7ea93e64637ec6ee6fabee45c5cbe4abd9cf3da95c43828ddb50b83", size = 65771, upload-time = "2025-05-16T18:52:17.419Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/80/08b1698c52ff76d96ba440bf15edc2f4bc0a279868778928e947c1004bdd/opentelemetry_semantic_conventions-0.54b1-py3-none-any.whl", hash = "sha256:29dab644a7e435b58d3a3918b58c333c92686236b30f7891d5e51f02933ca60d", size = 194938, upload-time = "2025-05-16T18:52:38.796Z" },
]
[[package]]
-name = "overrides"
-version = "7.7.0"
+name = "opentelemetry-semantic-conventions"
+version = "0.61b0"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/36/86/b585f53236dec60aba864e050778b25045f857e17f6e5ea0ae95fe80edd2/overrides-7.7.0.tar.gz", hash = "sha256:55158fa3d93b98cc75299b1e67078ad9003ca27945c76162c1c0766d6f91820a", size = 22812, upload-time = "2024-01-27T21:01:33.423Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/2c/ab/fc8290c6a4c722e5514d80f62b2dc4c4df1a68a41d1364e625c35990fcf3/overrides-7.7.0-py3-none-any.whl", hash = "sha256:c7ed9d062f78b8e4c1a7b70bd8796b35ead4d9f510227ef9c5dc7626c60d7e49", size = 17832, upload-time = "2024-01-27T21:01:31.393Z" },
+resolution-markers = [
+ "python_full_version >= '3.14' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "opentelemetry-api", version = "1.40.0", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts')" },
+ { name = "typing-extensions", marker = "(extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts')" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/6d/c0/4ae7973f3c2cfd2b6e321f1675626f0dab0a97027cc7a297474c9c8f3d04/opentelemetry_semantic_conventions-0.61b0.tar.gz", hash = "sha256:072f65473c5d7c6dc0355b27d6c9d1a679d63b6d4b4b16a9773062cb7e31192a", size = 145755, upload-time = "2026-03-04T14:17:32.664Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b2/37/cc6a55e448deaa9b27377d087da8615a3416d8ad523d5960b78dbeadd02a/opentelemetry_semantic_conventions-0.61b0-py3-none-any.whl", hash = "sha256:fa530a96be229795f8cef353739b618148b0fe2b4b3f005e60e262926c4d38e2", size = 231621, upload-time = "2026-03-04T14:17:19.33Z" },
+]
+
+[[package]]
+name = "orjson"
+version = "3.11.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/53/45/b268004f745ede84e5798b48ee12b05129d19235d0e15267aa57dcdb400b/orjson-3.11.7.tar.gz", hash = "sha256:9b1a67243945819ce55d24a30b59d6a168e86220452d2c96f4d1f093e71c0c49", size = 6144992, upload-time = "2026-02-02T15:38:49.29Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/80/bf/76f4f1665f6983385938f0e2a5d7efa12a58171b8456c252f3bae8a4cf75/orjson-3.11.7-cp312-cp312-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:bd03ea7606833655048dab1a00734a2875e3e86c276e1d772b2a02556f0d895f", size = 228545, upload-time = "2026-02-02T15:37:46.376Z" },
+ { url = "https://files.pythonhosted.org/packages/79/53/6c72c002cb13b5a978a068add59b25a8bdf2800ac1c9c8ecdb26d6d97064/orjson-3.11.7-cp312-cp312-macosx_15_0_arm64.whl", hash = "sha256:89e440ebc74ce8ab5c7bc4ce6757b4a6b1041becb127df818f6997b5c71aa60b", size = 125224, upload-time = "2026-02-02T15:37:47.697Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/83/10e48852865e5dd151bdfe652c06f7da484578ed02c5fca938e3632cb0b8/orjson-3.11.7-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5ede977b5fe5ac91b1dffc0a517ca4542d2ec8a6a4ff7b2652d94f640796342a", size = 128154, upload-time = "2026-02-02T15:37:48.954Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/52/a66e22a2b9abaa374b4a081d410edab6d1e30024707b87eab7c734afe28d/orjson-3.11.7-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b7b1dae39230a393df353827c855a5f176271c23434cfd2db74e0e424e693e10", size = 123548, upload-time = "2026-02-02T15:37:50.187Z" },
+ { url = "https://files.pythonhosted.org/packages/de/38/605d371417021359f4910c496f764c48ceb8997605f8c25bf1dfe58c0ebe/orjson-3.11.7-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ed46f17096e28fb28d2975834836a639af7278aa87c84f68ab08fbe5b8bd75fa", size = 129000, upload-time = "2026-02-02T15:37:51.426Z" },
+ { url = "https://files.pythonhosted.org/packages/44/98/af32e842b0ffd2335c89714d48ca4e3917b42f5d6ee5537832e069a4b3ac/orjson-3.11.7-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3726be79e36e526e3d9c1aceaadbfb4a04ee80a72ab47b3f3c17fefb9812e7b8", size = 141686, upload-time = "2026-02-02T15:37:52.607Z" },
+ { url = "https://files.pythonhosted.org/packages/96/0b/fc793858dfa54be6feee940c1463370ece34b3c39c1ca0aa3845f5ba9892/orjson-3.11.7-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0724e265bc548af1dedebd9cb3d24b4e1c1e685a343be43e87ba922a5c5fff2f", size = 130812, upload-time = "2026-02-02T15:37:53.944Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/91/98a52415059db3f374757d0b7f0f16e3b5cd5976c90d1c2b56acaea039e6/orjson-3.11.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e7745312efa9e11c17fbd3cb3097262d079da26930ae9ae7ba28fb738367cbad", size = 133440, upload-time = "2026-02-02T15:37:55.615Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/b6/cb540117bda61791f46381f8c26c8f93e802892830a6055748d3bb1925ab/orjson-3.11.7-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:f904c24bdeabd4298f7a977ef14ca2a022ca921ed670b92ecd16ab6f3d01f867", size = 138386, upload-time = "2026-02-02T15:37:56.814Z" },
+ { url = "https://files.pythonhosted.org/packages/63/1a/50a3201c334a7f17c231eee5f841342190723794e3b06293f26e7cf87d31/orjson-3.11.7-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:b9fc4d0f81f394689e0814617aadc4f2ea0e8025f38c226cbf22d3b5ddbf025d", size = 408853, upload-time = "2026-02-02T15:37:58.291Z" },
+ { url = "https://files.pythonhosted.org/packages/87/cd/8de1c67d0be44fdc22701e5989c0d015a2adf391498ad42c4dc589cd3013/orjson-3.11.7-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:849e38203e5be40b776ed2718e587faf204d184fc9a008ae441f9442320c0cab", size = 144130, upload-time = "2026-02-02T15:38:00.163Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/fe/d605d700c35dd55f51710d159fc54516a280923cd1b7e47508982fbb387d/orjson-3.11.7-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:4682d1db3bcebd2b64757e0ddf9e87ae5f00d29d16c5cdf3a62f561d08cc3dd2", size = 134818, upload-time = "2026-02-02T15:38:01.507Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/e4/15ecc67edb3ddb3e2f46ae04475f2d294e8b60c1825fbe28a428b93b3fbd/orjson-3.11.7-cp312-cp312-win32.whl", hash = "sha256:f4f7c956b5215d949a1f65334cf9d7612dde38f20a95f2315deef167def91a6f", size = 127923, upload-time = "2026-02-02T15:38:02.75Z" },
+ { url = "https://files.pythonhosted.org/packages/34/70/2e0855361f76198a3965273048c8e50a9695d88cd75811a5b46444895845/orjson-3.11.7-cp312-cp312-win_amd64.whl", hash = "sha256:bf742e149121dc5648ba0a08ea0871e87b660467ef168a3a5e53bc1fbd64bb74", size = 125007, upload-time = "2026-02-02T15:38:04.032Z" },
+ { url = "https://files.pythonhosted.org/packages/68/40/c2051bd19fc467610fed469dc29e43ac65891571138f476834ca192bc290/orjson-3.11.7-cp312-cp312-win_arm64.whl", hash = "sha256:26c3b9132f783b7d7903bf1efb095fed8d4a3a85ec0d334ee8beff3d7a4749d5", size = 126089, upload-time = "2026-02-02T15:38:05.297Z" },
+ { url = "https://files.pythonhosted.org/packages/89/25/6e0e52cac5aab51d7b6dcd257e855e1dec1c2060f6b28566c509b4665f62/orjson-3.11.7-cp313-cp313-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:1d98b30cc1313d52d4af17d9c3d307b08389752ec5f2e5febdfada70b0f8c733", size = 228390, upload-time = "2026-02-02T15:38:06.8Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/29/a77f48d2fc8a05bbc529e5ff481fb43d914f9e383ea2469d4f3d51df3d00/orjson-3.11.7-cp313-cp313-macosx_15_0_arm64.whl", hash = "sha256:d897e81f8d0cbd2abb82226d1860ad2e1ab3ff16d7b08c96ca00df9d45409ef4", size = 125189, upload-time = "2026-02-02T15:38:08.181Z" },
+ { url = "https://files.pythonhosted.org/packages/89/25/0a16e0729a0e6a1504f9d1a13cdd365f030068aab64cec6958396b9969d7/orjson-3.11.7-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:814be4b49b228cfc0b3c565acf642dd7d13538f966e3ccde61f4f55be3e20785", size = 128106, upload-time = "2026-02-02T15:38:09.41Z" },
+ { url = "https://files.pythonhosted.org/packages/66/da/a2e505469d60666a05ab373f1a6322eb671cb2ba3a0ccfc7d4bc97196787/orjson-3.11.7-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d06e5c5fed5caedd2e540d62e5b1c25e8c82431b9e577c33537e5fa4aa909539", size = 123363, upload-time = "2026-02-02T15:38:10.73Z" },
+ { url = "https://files.pythonhosted.org/packages/23/bf/ed73f88396ea35c71b38961734ea4a4746f7ca0768bf28fd551d37e48dd0/orjson-3.11.7-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:31c80ce534ac4ea3739c5ee751270646cbc46e45aea7576a38ffec040b4029a1", size = 129007, upload-time = "2026-02-02T15:38:12.138Z" },
+ { url = "https://files.pythonhosted.org/packages/73/3c/b05d80716f0225fc9008fbf8ab22841dcc268a626aa550561743714ce3bf/orjson-3.11.7-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f50979824bde13d32b4320eedd513431c921102796d86be3eee0b58e58a3ecd1", size = 141667, upload-time = "2026-02-02T15:38:13.398Z" },
+ { url = "https://files.pythonhosted.org/packages/61/e8/0be9b0addd9bf86abfc938e97441dcd0375d494594b1c8ad10fe57479617/orjson-3.11.7-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9e54f3808e2b6b945078c41aa8d9b5834b28c50843846e97807e5adb75fa9705", size = 130832, upload-time = "2026-02-02T15:38:14.698Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/ec/c68e3b9021a31d9ec15a94931db1410136af862955854ed5dd7e7e4f5bff/orjson-3.11.7-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a12b80df61aab7b98b490fe9e4879925ba666fccdfcd175252ce4d9035865ace", size = 133373, upload-time = "2026-02-02T15:38:16.109Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/45/f3466739aaafa570cc8e77c6dbb853c48bf56e3b43738020e2661e08b0ac/orjson-3.11.7-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:996b65230271f1a97026fd0e6a753f51fbc0c335d2ad0c6201f711b0da32693b", size = 138307, upload-time = "2026-02-02T15:38:17.453Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/84/9f7f02288da1ffb31405c1be07657afd1eecbcb4b64ee2817b6fe0f785fa/orjson-3.11.7-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:ab49d4b2a6a1d415ddb9f37a21e02e0d5dbfe10b7870b21bf779fc21e9156157", size = 408695, upload-time = "2026-02-02T15:38:18.831Z" },
+ { url = "https://files.pythonhosted.org/packages/18/07/9dd2f0c0104f1a0295ffbe912bc8d63307a539b900dd9e2c48ef7810d971/orjson-3.11.7-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:390a1dce0c055ddf8adb6aa94a73b45a4a7d7177b5c584b8d1c1947f2ba60fb3", size = 144099, upload-time = "2026-02-02T15:38:20.28Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/66/857a8e4a3292e1f7b1b202883bcdeb43a91566cf59a93f97c53b44bd6801/orjson-3.11.7-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:1eb80451a9c351a71dfaf5b7ccc13ad065405217726b59fdbeadbcc544f9d223", size = 134806, upload-time = "2026-02-02T15:38:22.186Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/5b/6ebcf3defc1aab3a338ca777214966851e92efb1f30dc7fc8285216e6d1b/orjson-3.11.7-cp313-cp313-win32.whl", hash = "sha256:7477aa6a6ec6139c5cb1cc7b214643592169a5494d200397c7fc95d740d5fcf3", size = 127914, upload-time = "2026-02-02T15:38:23.511Z" },
+ { url = "https://files.pythonhosted.org/packages/00/04/c6f72daca5092e3117840a1b1e88dfc809cc1470cf0734890d0366b684a1/orjson-3.11.7-cp313-cp313-win_amd64.whl", hash = "sha256:b9f95dcdea9d4f805daa9ddf02617a89e484c6985fa03055459f90e87d7a0757", size = 124986, upload-time = "2026-02-02T15:38:24.836Z" },
+ { url = "https://files.pythonhosted.org/packages/03/ba/077a0f6f1085d6b806937246860fafbd5b17f3919c70ee3f3d8d9c713f38/orjson-3.11.7-cp313-cp313-win_arm64.whl", hash = "sha256:800988273a014a0541483dc81021247d7eacb0c845a9d1a34a422bc718f41539", size = 126045, upload-time = "2026-02-02T15:38:26.216Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/1e/745565dca749813db9a093c5ebc4bac1a9475c64d54b95654336ac3ed961/orjson-3.11.7-cp314-cp314-macosx_10_15_x86_64.macosx_11_0_arm64.macosx_10_15_universal2.whl", hash = "sha256:de0a37f21d0d364954ad5de1970491d7fbd0fb1ef7417d4d56a36dc01ba0c0a0", size = 228391, upload-time = "2026-02-02T15:38:27.757Z" },
+ { url = "https://files.pythonhosted.org/packages/46/19/e40f6225da4d3aa0c8dc6e5219c5e87c2063a560fe0d72a88deb59776794/orjson-3.11.7-cp314-cp314-macosx_15_0_arm64.whl", hash = "sha256:c2428d358d85e8da9d37cba18b8c4047c55222007a84f97156a5b22028dfbfc0", size = 125188, upload-time = "2026-02-02T15:38:29.241Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/7e/c4de2babef2c0817fd1f048fd176aa48c37bec8aef53d2fa932983032cce/orjson-3.11.7-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3c4bc6c6ac52cdaa267552544c73e486fecbd710b7ac09bc024d5a78555a22f6", size = 128097, upload-time = "2026-02-02T15:38:30.618Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/74/233d360632bafd2197f217eee7fb9c9d0229eac0c18128aee5b35b0014fe/orjson-3.11.7-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:bd0d68edd7dfca1b2eca9361a44ac9f24b078de3481003159929a0573f21a6bf", size = 123364, upload-time = "2026-02-02T15:38:32.363Z" },
+ { url = "https://files.pythonhosted.org/packages/79/51/af79504981dd31efe20a9e360eb49c15f06df2b40e7f25a0a52d9ae888e8/orjson-3.11.7-cp314-cp314-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:623ad1b9548ef63886319c16fa317848e465a21513b31a6ad7b57443c3e0dcf5", size = 129076, upload-time = "2026-02-02T15:38:33.68Z" },
+ { url = "https://files.pythonhosted.org/packages/67/e2/da898eb68b72304f8de05ca6715870d09d603ee98d30a27e8a9629abc64b/orjson-3.11.7-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:6e776b998ac37c0396093d10290e60283f59cfe0fc3fccbd0ccc4bd04dd19892", size = 141705, upload-time = "2026-02-02T15:38:34.989Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/89/15364d92acb3d903b029e28d834edb8780c2b97404cbf7929aa6b9abdb24/orjson-3.11.7-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:652c6c3af76716f4a9c290371ba2e390ede06f6603edb277b481daf37f6f464e", size = 130855, upload-time = "2026-02-02T15:38:36.379Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/8b/ecdad52d0b38d4b8f514be603e69ccd5eacf4e7241f972e37e79792212ec/orjson-3.11.7-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a56df3239294ea5964adf074c54bcc4f0ccd21636049a2cf3ca9cf03b5d03cf1", size = 133386, upload-time = "2026-02-02T15:38:37.704Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/0e/45e1dcf10e17d0924b7c9162f87ec7b4ca79e28a0548acf6a71788d3e108/orjson-3.11.7-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:bda117c4148e81f746655d5a3239ae9bd00cb7bc3ca178b5fc5a5997e9744183", size = 138295, upload-time = "2026-02-02T15:38:39.096Z" },
+ { url = "https://files.pythonhosted.org/packages/63/d7/4d2e8b03561257af0450f2845b91fbd111d7e526ccdf737267108075e0ba/orjson-3.11.7-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:23d6c20517a97a9daf1d48b580fcdc6f0516c6f4b5038823426033690b4d2650", size = 408720, upload-time = "2026-02-02T15:38:40.634Z" },
+ { url = "https://files.pythonhosted.org/packages/78/cf/d45343518282108b29c12a65892445fc51f9319dc3c552ceb51bb5905ed2/orjson-3.11.7-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:8ff206156006da5b847c9304b6308a01e8cdbc8cce824e2779a5ba71c3def141", size = 144152, upload-time = "2026-02-02T15:38:42.262Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/3a/d6001f51a7275aacd342e77b735c71fa04125a3f93c36fee4526bc8c654e/orjson-3.11.7-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:962d046ee1765f74a1da723f4b33e3b228fe3a48bd307acce5021dfefe0e29b2", size = 134814, upload-time = "2026-02-02T15:38:43.627Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/d3/f19b47ce16820cc2c480f7f1723e17f6d411b3a295c60c8ad3aa9ff1c96a/orjson-3.11.7-cp314-cp314-win32.whl", hash = "sha256:89e13dd3f89f1c38a9c9eba5fbf7cdc2d1feca82f5f290864b4b7a6aac704576", size = 127997, upload-time = "2026-02-02T15:38:45.06Z" },
+ { url = "https://files.pythonhosted.org/packages/12/df/172771902943af54bf661a8d102bdf2e7f932127968080632bda6054b62c/orjson-3.11.7-cp314-cp314-win_amd64.whl", hash = "sha256:845c3e0d8ded9c9271cd79596b9b552448b885b97110f628fb687aee2eed11c1", size = 124985, upload-time = "2026-02-02T15:38:46.388Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/1c/f2a8d8a1b17514660a614ce5f7aac74b934e69f5abc2700cc7ced882a009/orjson-3.11.7-cp314-cp314-win_arm64.whl", hash = "sha256:4a2e9c5be347b937a2e0203866f12bba36082e89b402ddb9e927d5822e43088d", size = 126038, upload-time = "2026-02-02T15:38:47.703Z" },
]
[[package]]
name = "packaging"
version = "25.0"
source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
sdist = { url = "https://files.pythonhosted.org/packages/a1/d4/1fc4078c65507b51b96ca8f8c3ba19e6a61c8253c72794544580a7b6c24d/packaging-25.0.tar.gz", hash = "sha256:d443872c98d677bf60f6a1f2f8c1cb748e8fe762d2bf9d3148b5599295b0fc4f", size = 165727, upload-time = "2025-04-19T11:48:59.673Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/20/12/38679034af332785aac8774540895e234f4d07f7545804097de4b666afd8/packaging-25.0-py3-none-any.whl", hash = "sha256:29572ef2b1f17581046b3a2227d5c611fb25ec70ca1ba8554b24b0e69331a484", size = 66469, upload-time = "2025-04-19T11:48:57.875Z" },
]
+[[package]]
+name = "packaging"
+version = "26.0"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+]
+sdist = { url = "https://files.pythonhosted.org/packages/65/ee/299d360cdc32edc7d2cf530f3accf79c4fca01e96ffc950d8a52213bd8e4/packaging-26.0.tar.gz", hash = "sha256:00243ae351a257117b6a241061796684b084ed1c516a08c48a3f7e147a9d80b4", size = 143416, upload-time = "2026-01-21T20:50:39.064Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b7/b9/c538f279a4e237a006a2c98387d081e9eb060d203d8ed34467cc0f0b9b53/packaging-26.0-py3-none-any.whl", hash = "sha256:b36f1fef9334a5588b4166f8bcd26a14e521f2b55e6b9de3aaa80d3ff7a37529", size = 74366, upload-time = "2026-01-21T20:50:37.788Z" },
+]
+
[[package]]
name = "pandas"
version = "2.3.3"
source = { registry = "https://pypi.org/simple" }
-dependencies = [
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "python-dateutil" },
- { name = "pytz" },
- { name = "tzdata" },
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "python-dateutil", marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "pytz", marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "tzdata", marker = "extra == 'extra-13-megatron-core-dev'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/33/01/d40b85317f86cf08d853a4f495195c73815fdf205eef3993821720274518/pandas-2.3.3.tar.gz", hash = "sha256:e05e1af93b977f7eafa636d043f9f94c7ee3ac81af99c13508215942e64c993b", size = 4495223, upload-time = "2025-09-29T23:34:51.853Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/3d/f7/f425a00df4fcc22b292c6895c6831c0c8ae1d9fac1e024d16f98a9ce8749/pandas-2.3.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:376c6446ae31770764215a6c937f72d917f214b43560603cd60da6408f183b6c", size = 11555763, upload-time = "2025-09-29T23:16:53.287Z" },
- { url = "https://files.pythonhosted.org/packages/13/4f/66d99628ff8ce7857aca52fed8f0066ce209f96be2fede6cef9f84e8d04f/pandas-2.3.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:e19d192383eab2f4ceb30b412b22ea30690c9e618f78870357ae1d682912015a", size = 10801217, upload-time = "2025-09-29T23:17:04.522Z" },
- { url = "https://files.pythonhosted.org/packages/1d/03/3fc4a529a7710f890a239cc496fc6d50ad4a0995657dccc1d64695adb9f4/pandas-2.3.3-cp310-cp310-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5caf26f64126b6c7aec964f74266f435afef1c1b13da3b0636c7518a1fa3e2b1", size = 12148791, upload-time = "2025-09-29T23:17:18.444Z" },
- { url = "https://files.pythonhosted.org/packages/40/a8/4dac1f8f8235e5d25b9955d02ff6f29396191d4e665d71122c3722ca83c5/pandas-2.3.3-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:dd7478f1463441ae4ca7308a70e90b33470fa593429f9d4c578dd00d1fa78838", size = 12769373, upload-time = "2025-09-29T23:17:35.846Z" },
- { url = "https://files.pythonhosted.org/packages/df/91/82cc5169b6b25440a7fc0ef3a694582418d875c8e3ebf796a6d6470aa578/pandas-2.3.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:4793891684806ae50d1288c9bae9330293ab4e083ccd1c5e383c34549c6e4250", size = 13200444, upload-time = "2025-09-29T23:17:49.341Z" },
- { url = "https://files.pythonhosted.org/packages/10/ae/89b3283800ab58f7af2952704078555fa60c807fff764395bb57ea0b0dbd/pandas-2.3.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:28083c648d9a99a5dd035ec125d42439c6c1c525098c58af0fc38dd1a7a1b3d4", size = 13858459, upload-time = "2025-09-29T23:18:03.722Z" },
- { url = "https://files.pythonhosted.org/packages/85/72/530900610650f54a35a19476eca5104f38555afccda1aa11a92ee14cb21d/pandas-2.3.3-cp310-cp310-win_amd64.whl", hash = "sha256:503cf027cf9940d2ceaa1a93cfb5f8c8c7e6e90720a2850378f0b3f3b1e06826", size = 11346086, upload-time = "2025-09-29T23:18:18.505Z" },
- { url = "https://files.pythonhosted.org/packages/c1/fa/7ac648108144a095b4fb6aa3de1954689f7af60a14cf25583f4960ecb878/pandas-2.3.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:602b8615ebcc4a0c1751e71840428ddebeb142ec02c786e8ad6b1ce3c8dec523", size = 11578790, upload-time = "2025-09-29T23:18:30.065Z" },
- { url = "https://files.pythonhosted.org/packages/9b/35/74442388c6cf008882d4d4bdfc4109be87e9b8b7ccd097ad1e7f006e2e95/pandas-2.3.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:8fe25fc7b623b0ef6b5009149627e34d2a4657e880948ec3c840e9402e5c1b45", size = 10833831, upload-time = "2025-09-29T23:38:56.071Z" },
- { url = "https://files.pythonhosted.org/packages/fe/e4/de154cbfeee13383ad58d23017da99390b91d73f8c11856f2095e813201b/pandas-2.3.3-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b468d3dad6ff947df92dcb32ede5b7bd41a9b3cceef0a30ed925f6d01fb8fa66", size = 12199267, upload-time = "2025-09-29T23:18:41.627Z" },
- { url = "https://files.pythonhosted.org/packages/bf/c9/63f8d545568d9ab91476b1818b4741f521646cbdd151c6efebf40d6de6f7/pandas-2.3.3-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b98560e98cb334799c0b07ca7967ac361a47326e9b4e5a7dfb5ab2b1c9d35a1b", size = 12789281, upload-time = "2025-09-29T23:18:56.834Z" },
- { url = "https://files.pythonhosted.org/packages/f2/00/a5ac8c7a0e67fd1a6059e40aa08fa1c52cc00709077d2300e210c3ce0322/pandas-2.3.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:1d37b5848ba49824e5c30bedb9c830ab9b7751fd049bc7914533e01c65f79791", size = 13240453, upload-time = "2025-09-29T23:19:09.247Z" },
- { url = "https://files.pythonhosted.org/packages/27/4d/5c23a5bc7bd209231618dd9e606ce076272c9bc4f12023a70e03a86b4067/pandas-2.3.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:db4301b2d1f926ae677a751eb2bd0e8c5f5319c9cb3f88b0becbbb0b07b34151", size = 13890361, upload-time = "2025-09-29T23:19:25.342Z" },
- { url = "https://files.pythonhosted.org/packages/8e/59/712db1d7040520de7a4965df15b774348980e6df45c129b8c64d0dbe74ef/pandas-2.3.3-cp311-cp311-win_amd64.whl", hash = "sha256:f086f6fe114e19d92014a1966f43a3e62285109afe874f067f5abbdcbb10e59c", size = 11348702, upload-time = "2025-09-29T23:19:38.296Z" },
{ url = "https://files.pythonhosted.org/packages/9c/fb/231d89e8637c808b997d172b18e9d4a4bc7bf31296196c260526055d1ea0/pandas-2.3.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6d21f6d74eb1725c2efaa71a2bfc661a0689579b58e9c0ca58a739ff0b002b53", size = 11597846, upload-time = "2025-09-29T23:19:48.856Z" },
{ url = "https://files.pythonhosted.org/packages/5c/bd/bf8064d9cfa214294356c2d6702b716d3cf3bb24be59287a6a21e24cae6b/pandas-2.3.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:3fd2f887589c7aa868e02632612ba39acb0b8948faf5cc58f0850e165bd46f35", size = 10729618, upload-time = "2025-09-29T23:39:08.659Z" },
{ url = "https://files.pythonhosted.org/packages/57/56/cf2dbe1a3f5271370669475ead12ce77c61726ffd19a35546e31aa8edf4e/pandas-2.3.3-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ecaf1e12bdc03c86ad4a7ea848d66c685cb6851d807a26aa245ca3d2017a1908", size = 11737212, upload-time = "2025-09-29T23:19:59.765Z" },
@@ -3430,6 +4391,87 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/70/44/5191d2e4026f86a2a109053e194d3ba7a31a2d10a9c2348368c63ed4e85a/pandas-2.3.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:3869faf4bd07b3b66a9f462417d0ca3a9df29a9f6abd5d0d0dbab15dac7abe87", size = 13202175, upload-time = "2025-09-29T23:31:59.173Z" },
]
+[[package]]
+name = "pandas"
+version = "3.0.1"
+source = { registry = "https://pypi.org/simple" }
+resolution-markers = [
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version == '3.13.*' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+ "python_full_version < '3.13' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts'",
+]
+dependencies = [
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
+ { name = "python-dateutil", marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
+ { name = "tzdata", marker = "(sys_platform == 'emscripten' and extra != 'extra-13-megatron-core-dev') or (sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/2e/0c/b28ed414f080ee0ad153f848586d61d1878f91689950f037f976ce15f6c8/pandas-3.0.1.tar.gz", hash = "sha256:4186a699674af418f655dbd420ed87f50d56b4cd6603784279d9eef6627823c8", size = 4641901, upload-time = "2026-02-17T22:20:16.434Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/37/51/b467209c08dae2c624873d7491ea47d2b47336e5403309d433ea79c38571/pandas-3.0.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:476f84f8c20c9f5bc47252b66b4bb25e1a9fc2fa98cead96744d8116cb85771d", size = 10344357, upload-time = "2026-02-17T22:18:38.262Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/f1/e2567ffc8951ab371db2e40b2fe068e36b81d8cf3260f06ae508700e5504/pandas-3.0.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:0ab749dfba921edf641d4036c4c21c0b3ea70fea478165cb98a998fb2a261955", size = 9884543, upload-time = "2026-02-17T22:18:41.476Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/39/327802e0b6d693182403c144edacbc27eb82907b57062f23ef5a4c4a5ea7/pandas-3.0.1-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b8e36891080b87823aff3640c78649b91b8ff6eea3c0d70aeabd72ea43ab069b", size = 10396030, upload-time = "2026-02-17T22:18:43.822Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/fe/89d77e424365280b79d99b3e1e7d606f5165af2f2ecfaf0c6d24c799d607/pandas-3.0.1-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:532527a701281b9dd371e2f582ed9094f4c12dd9ffb82c0c54ee28d8ac9520c4", size = 10876435, upload-time = "2026-02-17T22:18:45.954Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/a6/2a75320849dd154a793f69c951db759aedb8d1dd3939eeacda9bdcfa1629/pandas-3.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:356e5c055ed9b0da1580d465657bc7d00635af4fd47f30afb23025352ba764d1", size = 11405133, upload-time = "2026-02-17T22:18:48.533Z" },
+ { url = "https://files.pythonhosted.org/packages/58/53/1d68fafb2e02d7881df66aa53be4cd748d25cbe311f3b3c85c93ea5d30ca/pandas-3.0.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:9d810036895f9ad6345b8f2a338dd6998a74e8483847403582cab67745bff821", size = 11932065, upload-time = "2026-02-17T22:18:50.837Z" },
+ { url = "https://files.pythonhosted.org/packages/75/08/67cc404b3a966b6df27b38370ddd96b3b023030b572283d035181854aac5/pandas-3.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:536232a5fe26dd989bd633e7a0c450705fdc86a207fec7254a55e9a22950fe43", size = 9741627, upload-time = "2026-02-17T22:18:53.905Z" },
+ { url = "https://files.pythonhosted.org/packages/86/4f/caf9952948fb00d23795f09b893d11f1cacb384e666854d87249530f7cbe/pandas-3.0.1-cp312-cp312-win_arm64.whl", hash = "sha256:0f463ebfd8de7f326d38037c7363c6dacb857c5881ab8961fb387804d6daf2f7", size = 9052483, upload-time = "2026-02-17T22:18:57.31Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/48/aad6ec4f8d007534c091e9a7172b3ec1b1ee6d99a9cbb936b5eab6c6cf58/pandas-3.0.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5272627187b5d9c20e55d27caf5f2cd23e286aba25cadf73c8590e432e2b7262", size = 10317509, upload-time = "2026-02-17T22:18:59.498Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/14/5990826f779f79148ae9d3a2c39593dc04d61d5d90541e71b5749f35af95/pandas-3.0.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:661e0f665932af88c7877f31da0dc743fe9c8f2524bdffe23d24fdcb67ef9d56", size = 9860561, upload-time = "2026-02-17T22:19:02.265Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/80/f01ff54664b6d70fed71475543d108a9b7c888e923ad210795bef04ffb7d/pandas-3.0.1-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:75e6e292ff898679e47a2199172593d9f6107fd2dd3617c22c2946e97d5df46e", size = 10365506, upload-time = "2026-02-17T22:19:05.017Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/85/ab6d04733a7d6ff32bfc8382bf1b07078228f5d6ebec5266b91bfc5c4ff7/pandas-3.0.1-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1ff8cf1d2896e34343197685f432450ec99a85ba8d90cce2030c5eee2ef98791", size = 10873196, upload-time = "2026-02-17T22:19:07.204Z" },
+ { url = "https://files.pythonhosted.org/packages/48/a9/9301c83d0b47c23ac5deab91c6b39fd98d5b5db4d93b25df8d381451828f/pandas-3.0.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:eca8b4510f6763f3d37359c2105df03a7a221a508f30e396a51d0713d462e68a", size = 11370859, upload-time = "2026-02-17T22:19:09.436Z" },
+ { url = "https://files.pythonhosted.org/packages/59/fe/0c1fc5bd2d29c7db2ab372330063ad555fb83e08422829c785f5ec2176ca/pandas-3.0.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:06aff2ad6f0b94a17822cf8b83bbb563b090ed82ff4fe7712db2ce57cd50d9b8", size = 11924584, upload-time = "2026-02-17T22:19:11.562Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/7d/216a1588b65a7aa5f4535570418a599d943c85afb1d95b0876fc00aa1468/pandas-3.0.1-cp313-cp313-win_amd64.whl", hash = "sha256:9fea306c783e28884c29057a1d9baa11a349bbf99538ec1da44c8476563d1b25", size = 9742769, upload-time = "2026-02-17T22:19:13.926Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/cb/810a22a6af9a4e97c8ab1c946b47f3489c5bca5adc483ce0ffc84c9cc768/pandas-3.0.1-cp313-cp313-win_arm64.whl", hash = "sha256:a8d37a43c52917427e897cb2e429f67a449327394396a81034a4449b99afda59", size = 9043855, upload-time = "2026-02-17T22:19:16.09Z" },
+ { url = "https://files.pythonhosted.org/packages/92/fa/423c89086cca1f039cf1253c3ff5b90f157b5b3757314aa635f6bf3e30aa/pandas-3.0.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:d54855f04f8246ed7b6fc96b05d4871591143c46c0b6f4af874764ed0d2d6f06", size = 10752673, upload-time = "2026-02-17T22:19:18.304Z" },
+ { url = "https://files.pythonhosted.org/packages/22/23/b5a08ec1f40020397f0faba72f1e2c11f7596a6169c7b3e800abff0e433f/pandas-3.0.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:4e1b677accee34a09e0dc2ce5624e4a58a1870ffe56fc021e9caf7f23cd7668f", size = 10404967, upload-time = "2026-02-17T22:19:20.726Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/81/94841f1bb4afdc2b52a99daa895ac2c61600bb72e26525ecc9543d453ebc/pandas-3.0.1-cp313-cp313t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a9cabbdcd03f1b6cd254d6dda8ae09b0252524be1592594c00b7895916cb1324", size = 10320575, upload-time = "2026-02-17T22:19:24.919Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/8b/2ae37d66a5342a83adadfd0cb0b4bf9c3c7925424dd5f40d15d6cfaa35ee/pandas-3.0.1-cp313-cp313t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5ae2ab1f166668b41e770650101e7090824fd34d17915dd9cd479f5c5e0065e9", size = 10710921, upload-time = "2026-02-17T22:19:27.181Z" },
+ { url = "https://files.pythonhosted.org/packages/a2/61/772b2e2757855e232b7ccf7cb8079a5711becb3a97f291c953def15a833f/pandas-3.0.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:6bf0603c2e30e2cafac32807b06435f28741135cb8697eae8b28c7d492fc7d76", size = 11334191, upload-time = "2026-02-17T22:19:29.411Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/08/b16c6df3ef555d8495d1d265a7963b65be166785d28f06a350913a4fac78/pandas-3.0.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:6c426422973973cae1f4a23e51d4ae85974f44871b24844e4f7de752dd877098", size = 11782256, upload-time = "2026-02-17T22:19:32.34Z" },
+ { url = "https://files.pythonhosted.org/packages/55/80/178af0594890dee17e239fca96d3d8670ba0f5ff59b7d0439850924a9c09/pandas-3.0.1-cp313-cp313t-win_amd64.whl", hash = "sha256:b03f91ae8c10a85c1613102c7bef5229b5379f343030a3ccefeca8a33414cf35", size = 10485047, upload-time = "2026-02-17T22:19:34.605Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/8b/4bb774a998b97e6c2fd62a9e6cfdaae133b636fd1c468f92afb4ae9a447a/pandas-3.0.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:99d0f92ed92d3083d140bf6b97774f9f13863924cf3f52a70711f4e7588f9d0a", size = 10322465, upload-time = "2026-02-17T22:19:36.803Z" },
+ { url = "https://files.pythonhosted.org/packages/72/3a/5b39b51c64159f470f1ca3b1c2a87da290657ca022f7cd11442606f607d1/pandas-3.0.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:3b66857e983208654294bb6477b8a63dee26b37bdd0eb34d010556e91261784f", size = 9910632, upload-time = "2026-02-17T22:19:39.001Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/f7/b449ffb3f68c11da12fc06fbf6d2fa3a41c41e17d0284d23a79e1c13a7e4/pandas-3.0.1-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:56cf59638bf24dc9bdf2154c81e248b3289f9a09a6d04e63608c159022352749", size = 10440535, upload-time = "2026-02-17T22:19:41.157Z" },
+ { url = "https://files.pythonhosted.org/packages/55/77/6ea82043db22cb0f2bbfe7198da3544000ddaadb12d26be36e19b03a2dc5/pandas-3.0.1-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c1a9f55e0f46951874b863d1f3906dcb57df2d9be5c5847ba4dfb55b2c815249", size = 10893940, upload-time = "2026-02-17T22:19:43.493Z" },
+ { url = "https://files.pythonhosted.org/packages/03/30/f1b502a72468c89412c1b882a08f6eed8a4ee9dc033f35f65d0663df6081/pandas-3.0.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:1849f0bba9c8a2fb0f691d492b834cc8dadf617e29015c66e989448d58d011ee", size = 11442711, upload-time = "2026-02-17T22:19:46.074Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/f0/ebb6ddd8fc049e98cabac5c2924d14d1dda26a20adb70d41ea2e428d3ec4/pandas-3.0.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:c3d288439e11b5325b02ae6e9cc83e6805a62c40c5a6220bea9beb899c073b1c", size = 11963918, upload-time = "2026-02-17T22:19:48.838Z" },
+ { url = "https://files.pythonhosted.org/packages/09/f8/8ce132104074f977f907442790eaae24e27bce3b3b454e82faa3237ff098/pandas-3.0.1-cp314-cp314-win_amd64.whl", hash = "sha256:93325b0fe372d192965f4cca88d97667f49557398bbf94abdda3bf1b591dbe66", size = 9862099, upload-time = "2026-02-17T22:19:51.081Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/b7/6af9aac41ef2456b768ef0ae60acf8abcebb450a52043d030a65b4b7c9bd/pandas-3.0.1-cp314-cp314-win_arm64.whl", hash = "sha256:97ca08674e3287c7148f4858b01136f8bdfe7202ad25ad04fec602dd1d29d132", size = 9185333, upload-time = "2026-02-17T22:19:53.266Z" },
+ { url = "https://files.pythonhosted.org/packages/66/fc/848bb6710bc6061cb0c5badd65b92ff75c81302e0e31e496d00029fe4953/pandas-3.0.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:58eeb1b2e0fb322befcf2bbc9ba0af41e616abadb3d3414a6bc7167f6cbfce32", size = 10772664, upload-time = "2026-02-17T22:19:55.806Z" },
+ { url = "https://files.pythonhosted.org/packages/69/5c/866a9bbd0f79263b4b0db6ec1a341be13a1473323f05c122388e0f15b21d/pandas-3.0.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:cd9af1276b5ca9e298bd79a26bda32fa9cc87ed095b2a9a60978d2ca058eaf87", size = 10421286, upload-time = "2026-02-17T22:19:58.091Z" },
+ { url = "https://files.pythonhosted.org/packages/51/a4/2058fb84fb1cfbfb2d4a6d485e1940bb4ad5716e539d779852494479c580/pandas-3.0.1-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:94f87a04984d6b63788327cd9f79dda62b7f9043909d2440ceccf709249ca988", size = 10342050, upload-time = "2026-02-17T22:20:01.376Z" },
+ { url = "https://files.pythonhosted.org/packages/22/1b/674e89996cc4be74db3c4eb09240c4bb549865c9c3f5d9b086ff8fcfbf00/pandas-3.0.1-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:85fe4c4df62e1e20f9db6ebfb88c844b092c22cd5324bdcf94bfa2fc1b391221", size = 10740055, upload-time = "2026-02-17T22:20:04.328Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/f8/e954b750764298c22fa4614376531fe63c521ef517e7059a51f062b87dca/pandas-3.0.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:331ca75a2f8672c365ae25c0b29e46f5ac0c6551fdace8eec4cd65e4fac271ff", size = 11357632, upload-time = "2026-02-17T22:20:06.647Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/02/c6e04b694ffd68568297abd03588b6d30295265176a5c01b7459d3bc35a3/pandas-3.0.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:15860b1fdb1973fffade772fdb931ccf9b2f400a3f5665aef94a00445d7d8dd5", size = 11810974, upload-time = "2026-02-17T22:20:08.946Z" },
+ { url = "https://files.pythonhosted.org/packages/89/41/d7dfb63d2407f12055215070c42fc6ac41b66e90a2946cdc5e759058398b/pandas-3.0.1-cp314-cp314t-win_amd64.whl", hash = "sha256:44f1364411d5670efa692b146c748f4ed013df91ee91e9bec5677fb1fd58b937", size = 10884622, upload-time = "2026-02-17T22:20:11.711Z" },
+ { url = "https://files.pythonhosted.org/packages/68/b0/34937815889fa982613775e4b97fddd13250f11012d769949c5465af2150/pandas-3.0.1-cp314-cp314t-win_arm64.whl", hash = "sha256:108dd1790337a494aa80e38def654ca3f0968cf4f362c85f44c15e471667102d", size = 9452085, upload-time = "2026-02-17T22:20:14.331Z" },
+]
+
[[package]]
name = "paramiko"
version = "4.0.0"
@@ -3447,11 +4489,11 @@ wheels = [
[[package]]
name = "pathspec"
-version = "1.0.3"
+version = "1.0.4"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/4c/b2/bb8e495d5262bfec41ab5cb18f522f1012933347fb5d9e62452d446baca2/pathspec-1.0.3.tar.gz", hash = "sha256:bac5cf97ae2c2876e2d25ebb15078eb04d76e4b98921ee31c6f85ade8b59444d", size = 130841, upload-time = "2026-01-09T15:46:46.009Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/fa/36/e27608899f9b8d4dff0617b2d9ab17ca5608956ca44461ac14ac48b44015/pathspec-1.0.4.tar.gz", hash = "sha256:0210e2ae8a21a9137c0d470578cb0e595af87edaa6ebf12ff176f14a02e0e645", size = 131200, upload-time = "2026-01-27T03:59:46.938Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/32/2b/121e912bd60eebd623f873fd090de0e84f322972ab25a7f9044c056804ed/pathspec-1.0.3-py3-none-any.whl", hash = "sha256:e80767021c1cc524aa3fb14bedda9c34406591343cc42797b386ce7b9354fb6c", size = 55021, upload-time = "2026-01-09T15:46:44.652Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/3c/2c197d226f9ea224a9ab8d197933f9da0ae0aac5b6e0f884e2b8d9c8e9f7/pathspec-1.0.4-py3-none-any.whl", hash = "sha256:fb6ae2fd4e7c921a165808a552060e722767cfa526f99ca5156ed2ce45a5c723", size = 55206, upload-time = "2026-01-27T03:59:45.137Z" },
]
[[package]]
@@ -3465,109 +4507,80 @@ wheels = [
[[package]]
name = "pillow"
-version = "12.1.0"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/d0/02/d52c733a2452ef1ffcc123b68e6606d07276b0e358db70eabad7e40042b7/pillow-12.1.0.tar.gz", hash = "sha256:5c5ae0a06e9ea030ab786b0251b32c7e4ce10e58d983c0d5c56029455180b5b9", size = 46977283, upload-time = "2026-01-02T09:13:29.892Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/fe/41/f73d92b6b883a579e79600d391f2e21cb0df767b2714ecbd2952315dfeef/pillow-12.1.0-cp310-cp310-macosx_10_10_x86_64.whl", hash = "sha256:fb125d860738a09d363a88daa0f59c4533529a90e564785e20fe875b200b6dbd", size = 5304089, upload-time = "2026-01-02T09:10:24.953Z" },
- { url = "https://files.pythonhosted.org/packages/94/55/7aca2891560188656e4a91ed9adba305e914a4496800da6b5c0a15f09edf/pillow-12.1.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:cad302dc10fac357d3467a74a9561c90609768a6f73a1923b0fd851b6486f8b0", size = 4657815, upload-time = "2026-01-02T09:10:27.063Z" },
- { url = "https://files.pythonhosted.org/packages/e9/d2/b28221abaa7b4c40b7dba948f0f6a708bd7342c4d47ce342f0ea39643974/pillow-12.1.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a40905599d8079e09f25027423aed94f2823adaf2868940de991e53a449e14a8", size = 6222593, upload-time = "2026-01-02T09:10:29.115Z" },
- { url = "https://files.pythonhosted.org/packages/71/b8/7a61fb234df6a9b0b479f69e66901209d89ff72a435b49933f9122f94cac/pillow-12.1.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:92a7fe4225365c5e3a8e598982269c6d6698d3e783b3b1ae979e7819f9cd55c1", size = 8027579, upload-time = "2026-01-02T09:10:31.182Z" },
- { url = "https://files.pythonhosted.org/packages/ea/51/55c751a57cc524a15a0e3db20e5cde517582359508d62305a627e77fd295/pillow-12.1.0-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f10c98f49227ed8383d28174ee95155a675c4ed7f85e2e573b04414f7e371bda", size = 6335760, upload-time = "2026-01-02T09:10:33.02Z" },
- { url = "https://files.pythonhosted.org/packages/dc/7c/60e3e6f5e5891a1a06b4c910f742ac862377a6fe842f7184df4a274ce7bf/pillow-12.1.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8637e29d13f478bc4f153d8daa9ffb16455f0a6cb287da1b432fdad2bfbd66c7", size = 7027127, upload-time = "2026-01-02T09:10:35.009Z" },
- { url = "https://files.pythonhosted.org/packages/06/37/49d47266ba50b00c27ba63a7c898f1bb41a29627ced8c09e25f19ebec0ff/pillow-12.1.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:21e686a21078b0f9cb8c8a961d99e6a4ddb88e0fc5ea6e130172ddddc2e5221a", size = 6449896, upload-time = "2026-01-02T09:10:36.793Z" },
- { url = "https://files.pythonhosted.org/packages/f9/e5/67fd87d2913902462cd9b79c6211c25bfe95fcf5783d06e1367d6d9a741f/pillow-12.1.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:2415373395a831f53933c23ce051021e79c8cd7979822d8cc478547a3f4da8ef", size = 7151345, upload-time = "2026-01-02T09:10:39.064Z" },
- { url = "https://files.pythonhosted.org/packages/bd/15/f8c7abf82af68b29f50d77c227e7a1f87ce02fdc66ded9bf603bc3b41180/pillow-12.1.0-cp310-cp310-win32.whl", hash = "sha256:e75d3dba8fc1ddfec0cd752108f93b83b4f8d6ab40e524a95d35f016b9683b09", size = 6325568, upload-time = "2026-01-02T09:10:41.035Z" },
- { url = "https://files.pythonhosted.org/packages/d4/24/7d1c0e160b6b5ac2605ef7d8be537e28753c0db5363d035948073f5513d7/pillow-12.1.0-cp310-cp310-win_amd64.whl", hash = "sha256:64efdf00c09e31efd754448a383ea241f55a994fd079866b92d2bbff598aad91", size = 7032367, upload-time = "2026-01-02T09:10:43.09Z" },
- { url = "https://files.pythonhosted.org/packages/f4/03/41c038f0d7a06099254c60f618d0ec7be11e79620fc23b8e85e5b31d9a44/pillow-12.1.0-cp310-cp310-win_arm64.whl", hash = "sha256:f188028b5af6b8fb2e9a76ac0f841a575bd1bd396e46ef0840d9b88a48fdbcea", size = 2452345, upload-time = "2026-01-02T09:10:44.795Z" },
- { url = "https://files.pythonhosted.org/packages/43/c4/bf8328039de6cc22182c3ef007a2abfbbdab153661c0a9aa78af8d706391/pillow-12.1.0-cp311-cp311-macosx_10_10_x86_64.whl", hash = "sha256:a83e0850cb8f5ac975291ebfc4170ba481f41a28065277f7f735c202cd8e0af3", size = 5304057, upload-time = "2026-01-02T09:10:46.627Z" },
- { url = "https://files.pythonhosted.org/packages/43/06/7264c0597e676104cc22ca73ee48f752767cd4b1fe084662620b17e10120/pillow-12.1.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:b6e53e82ec2db0717eabb276aa56cf4e500c9a7cec2c2e189b55c24f65a3e8c0", size = 4657811, upload-time = "2026-01-02T09:10:49.548Z" },
- { url = "https://files.pythonhosted.org/packages/72/64/f9189e44474610daf83da31145fa56710b627b5c4c0b9c235e34058f6b31/pillow-12.1.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:40a8e3b9e8773876d6e30daed22f016509e3987bab61b3b7fe309d7019a87451", size = 6232243, upload-time = "2026-01-02T09:10:51.62Z" },
- { url = "https://files.pythonhosted.org/packages/ef/30/0df458009be6a4caca4ca2c52975e6275c387d4e5c95544e34138b41dc86/pillow-12.1.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:800429ac32c9b72909c671aaf17ecd13110f823ddb7db4dfef412a5587c2c24e", size = 8037872, upload-time = "2026-01-02T09:10:53.446Z" },
- { url = "https://files.pythonhosted.org/packages/e4/86/95845d4eda4f4f9557e25381d70876aa213560243ac1a6d619c46caaedd9/pillow-12.1.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0b022eaaf709541b391ee069f0022ee5b36c709df71986e3f7be312e46f42c84", size = 6345398, upload-time = "2026-01-02T09:10:55.426Z" },
- { url = "https://files.pythonhosted.org/packages/5c/1f/8e66ab9be3aaf1435bc03edd1ebdf58ffcd17f7349c1d970cafe87af27d9/pillow-12.1.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1f345e7bc9d7f368887c712aa5054558bad44d2a301ddf9248599f4161abc7c0", size = 7034667, upload-time = "2026-01-02T09:10:57.11Z" },
- { url = "https://files.pythonhosted.org/packages/f9/f6/683b83cb9b1db1fb52b87951b1c0b99bdcfceaa75febf11406c19f82cb5e/pillow-12.1.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d70347c8a5b7ccd803ec0c85c8709f036e6348f1e6a5bf048ecd9c64d3550b8b", size = 6458743, upload-time = "2026-01-02T09:10:59.331Z" },
- { url = "https://files.pythonhosted.org/packages/9a/7d/de833d63622538c1d58ce5395e7c6cb7e7dce80decdd8bde4a484e095d9f/pillow-12.1.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:1fcc52d86ce7a34fd17cb04e87cfdb164648a3662a6f20565910a99653d66c18", size = 7159342, upload-time = "2026-01-02T09:11:01.82Z" },
- { url = "https://files.pythonhosted.org/packages/8c/40/50d86571c9e5868c42b81fe7da0c76ca26373f3b95a8dd675425f4a92ec1/pillow-12.1.0-cp311-cp311-win32.whl", hash = "sha256:3ffaa2f0659e2f740473bcf03c702c39a8d4b2b7ffc629052028764324842c64", size = 6328655, upload-time = "2026-01-02T09:11:04.556Z" },
- { url = "https://files.pythonhosted.org/packages/6c/af/b1d7e301c4cd26cd45d4af884d9ee9b6fab893b0ad2450d4746d74a6968c/pillow-12.1.0-cp311-cp311-win_amd64.whl", hash = "sha256:806f3987ffe10e867bab0ddad45df1148a2b98221798457fa097ad85d6e8bc75", size = 7031469, upload-time = "2026-01-02T09:11:06.538Z" },
- { url = "https://files.pythonhosted.org/packages/48/36/d5716586d887fb2a810a4a61518a327a1e21c8b7134c89283af272efe84b/pillow-12.1.0-cp311-cp311-win_arm64.whl", hash = "sha256:9f5fefaca968e700ad1a4a9de98bf0869a94e397fe3524c4c9450c1445252304", size = 2452515, upload-time = "2026-01-02T09:11:08.226Z" },
- { url = "https://files.pythonhosted.org/packages/20/31/dc53fe21a2f2996e1b7d92bf671cdb157079385183ef7c1ae08b485db510/pillow-12.1.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:a332ac4ccb84b6dde65dbace8431f3af08874bf9770719d32a635c4ef411b18b", size = 5262642, upload-time = "2026-01-02T09:11:10.138Z" },
- { url = "https://files.pythonhosted.org/packages/ab/c1/10e45ac9cc79419cedf5121b42dcca5a50ad2b601fa080f58c22fb27626e/pillow-12.1.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:907bfa8a9cb790748a9aa4513e37c88c59660da3bcfffbd24a7d9e6abf224551", size = 4657464, upload-time = "2026-01-02T09:11:12.319Z" },
- { url = "https://files.pythonhosted.org/packages/ad/26/7b82c0ab7ef40ebede7a97c72d473bda5950f609f8e0c77b04af574a0ddb/pillow-12.1.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:efdc140e7b63b8f739d09a99033aa430accce485ff78e6d311973a67b6bf3208", size = 6234878, upload-time = "2026-01-02T09:11:14.096Z" },
- { url = "https://files.pythonhosted.org/packages/76/25/27abc9792615b5e886ca9411ba6637b675f1b77af3104710ac7353fe5605/pillow-12.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:bef9768cab184e7ae6e559c032e95ba8d07b3023c289f79a2bd36e8bf85605a5", size = 8044868, upload-time = "2026-01-02T09:11:15.903Z" },
- { url = "https://files.pythonhosted.org/packages/0a/ea/f200a4c36d836100e7bc738fc48cd963d3ba6372ebc8298a889e0cfc3359/pillow-12.1.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:742aea052cf5ab5034a53c3846165bc3ce88d7c38e954120db0ab867ca242661", size = 6349468, upload-time = "2026-01-02T09:11:17.631Z" },
- { url = "https://files.pythonhosted.org/packages/11/8f/48d0b77ab2200374c66d344459b8958c86693be99526450e7aee714e03e4/pillow-12.1.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a6dfc2af5b082b635af6e08e0d1f9f1c4e04d17d4e2ca0ef96131e85eda6eb17", size = 7041518, upload-time = "2026-01-02T09:11:19.389Z" },
- { url = "https://files.pythonhosted.org/packages/1d/23/c281182eb986b5d31f0a76d2a2c8cd41722d6fb8ed07521e802f9bba52de/pillow-12.1.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:609e89d9f90b581c8d16358c9087df76024cf058fa693dd3e1e1620823f39670", size = 6462829, upload-time = "2026-01-02T09:11:21.28Z" },
- { url = "https://files.pythonhosted.org/packages/25/ef/7018273e0faac099d7b00982abdcc39142ae6f3bd9ceb06de09779c4a9d6/pillow-12.1.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:43b4899cfd091a9693a1278c4982f3e50f7fb7cff5153b05174b4afc9593b616", size = 7166756, upload-time = "2026-01-02T09:11:23.559Z" },
- { url = "https://files.pythonhosted.org/packages/8f/c8/993d4b7ab2e341fe02ceef9576afcf5830cdec640be2ac5bee1820d693d4/pillow-12.1.0-cp312-cp312-win32.whl", hash = "sha256:aa0c9cc0b82b14766a99fbe6084409972266e82f459821cd26997a488a7261a7", size = 6328770, upload-time = "2026-01-02T09:11:25.661Z" },
- { url = "https://files.pythonhosted.org/packages/a7/87/90b358775a3f02765d87655237229ba64a997b87efa8ccaca7dd3e36e7a7/pillow-12.1.0-cp312-cp312-win_amd64.whl", hash = "sha256:d70534cea9e7966169ad29a903b99fc507e932069a881d0965a1a84bb57f6c6d", size = 7033406, upload-time = "2026-01-02T09:11:27.474Z" },
- { url = "https://files.pythonhosted.org/packages/5d/cf/881b457eccacac9e5b2ddd97d5071fb6d668307c57cbf4e3b5278e06e536/pillow-12.1.0-cp312-cp312-win_arm64.whl", hash = "sha256:65b80c1ee7e14a87d6a068dd3b0aea268ffcabfe0498d38661b00c5b4b22e74c", size = 2452612, upload-time = "2026-01-02T09:11:29.309Z" },
- { url = "https://files.pythonhosted.org/packages/dd/c7/2530a4aa28248623e9d7f27316b42e27c32ec410f695929696f2e0e4a778/pillow-12.1.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:7b5dd7cbae20285cdb597b10eb5a2c13aa9de6cde9bb64a3c1317427b1db1ae1", size = 4062543, upload-time = "2026-01-02T09:11:31.566Z" },
- { url = "https://files.pythonhosted.org/packages/8f/1f/40b8eae823dc1519b87d53c30ed9ef085506b05281d313031755c1705f73/pillow-12.1.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:29a4cef9cb672363926f0470afc516dbf7305a14d8c54f7abbb5c199cd8f8179", size = 4138373, upload-time = "2026-01-02T09:11:33.367Z" },
- { url = "https://files.pythonhosted.org/packages/d4/77/6fa60634cf06e52139fd0e89e5bbf055e8166c691c42fb162818b7fda31d/pillow-12.1.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:681088909d7e8fa9e31b9799aaa59ba5234c58e5e4f1951b4c4d1082a2e980e0", size = 3601241, upload-time = "2026-01-02T09:11:35.011Z" },
- { url = "https://files.pythonhosted.org/packages/4f/bf/28ab865de622e14b747f0cd7877510848252d950e43002e224fb1c9ababf/pillow-12.1.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:983976c2ab753166dc66d36af6e8ec15bb511e4a25856e2227e5f7e00a160587", size = 5262410, upload-time = "2026-01-02T09:11:36.682Z" },
- { url = "https://files.pythonhosted.org/packages/1c/34/583420a1b55e715937a85bd48c5c0991598247a1fd2eb5423188e765ea02/pillow-12.1.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:db44d5c160a90df2d24a24760bbd37607d53da0b34fb546c4c232af7192298ac", size = 4657312, upload-time = "2026-01-02T09:11:38.535Z" },
- { url = "https://files.pythonhosted.org/packages/1d/fd/f5a0896839762885b3376ff04878f86ab2b097c2f9a9cdccf4eda8ba8dc0/pillow-12.1.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:6b7a9d1db5dad90e2991645874f708e87d9a3c370c243c2d7684d28f7e133e6b", size = 6232605, upload-time = "2026-01-02T09:11:40.602Z" },
- { url = "https://files.pythonhosted.org/packages/98/aa/938a09d127ac1e70e6ed467bd03834350b33ef646b31edb7452d5de43792/pillow-12.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:6258f3260986990ba2fa8a874f8b6e808cf5abb51a94015ca3dc3c68aa4f30ea", size = 8041617, upload-time = "2026-01-02T09:11:42.721Z" },
- { url = "https://files.pythonhosted.org/packages/17/e8/538b24cb426ac0186e03f80f78bc8dc7246c667f58b540bdd57c71c9f79d/pillow-12.1.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e115c15e3bc727b1ca3e641a909f77f8ca72a64fff150f666fcc85e57701c26c", size = 6346509, upload-time = "2026-01-02T09:11:44.955Z" },
- { url = "https://files.pythonhosted.org/packages/01/9a/632e58ec89a32738cabfd9ec418f0e9898a2b4719afc581f07c04a05e3c9/pillow-12.1.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6741e6f3074a35e47c77b23a4e4f2d90db3ed905cb1c5e6e0d49bff2045632bc", size = 7038117, upload-time = "2026-01-02T09:11:46.736Z" },
- { url = "https://files.pythonhosted.org/packages/c7/a2/d40308cf86eada842ca1f3ffa45d0ca0df7e4ab33c83f81e73f5eaed136d/pillow-12.1.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:935b9d1aed48fcfb3f838caac506f38e29621b44ccc4f8a64d575cb1b2a88644", size = 6460151, upload-time = "2026-01-02T09:11:48.625Z" },
- { url = "https://files.pythonhosted.org/packages/f1/88/f5b058ad6453a085c5266660a1417bdad590199da1b32fb4efcff9d33b05/pillow-12.1.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:5fee4c04aad8932da9f8f710af2c1a15a83582cfb884152a9caa79d4efcdbf9c", size = 7164534, upload-time = "2026-01-02T09:11:50.445Z" },
- { url = "https://files.pythonhosted.org/packages/19/ce/c17334caea1db789163b5d855a5735e47995b0b5dc8745e9a3605d5f24c0/pillow-12.1.0-cp313-cp313-win32.whl", hash = "sha256:a786bf667724d84aa29b5db1c61b7bfdde380202aaca12c3461afd6b71743171", size = 6332551, upload-time = "2026-01-02T09:11:52.234Z" },
- { url = "https://files.pythonhosted.org/packages/e5/07/74a9d941fa45c90a0d9465098fe1ec85de3e2afbdc15cc4766622d516056/pillow-12.1.0-cp313-cp313-win_amd64.whl", hash = "sha256:461f9dfdafa394c59cd6d818bdfdbab4028b83b02caadaff0ffd433faf4c9a7a", size = 7040087, upload-time = "2026-01-02T09:11:54.822Z" },
- { url = "https://files.pythonhosted.org/packages/88/09/c99950c075a0e9053d8e880595926302575bc742b1b47fe1bbcc8d388d50/pillow-12.1.0-cp313-cp313-win_arm64.whl", hash = "sha256:9212d6b86917a2300669511ed094a9406888362e085f2431a7da985a6b124f45", size = 2452470, upload-time = "2026-01-02T09:11:56.522Z" },
- { url = "https://files.pythonhosted.org/packages/b5/ba/970b7d85ba01f348dee4d65412476321d40ee04dcb51cd3735b9dc94eb58/pillow-12.1.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:00162e9ca6d22b7c3ee8e61faa3c3253cd19b6a37f126cad04f2f88b306f557d", size = 5264816, upload-time = "2026-01-02T09:11:58.227Z" },
- { url = "https://files.pythonhosted.org/packages/10/60/650f2fb55fdba7a510d836202aa52f0baac633e50ab1cf18415d332188fb/pillow-12.1.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:7d6daa89a00b58c37cb1747ec9fb7ac3bc5ffd5949f5888657dfddde6d1312e0", size = 4660472, upload-time = "2026-01-02T09:12:00.798Z" },
- { url = "https://files.pythonhosted.org/packages/2b/c0/5273a99478956a099d533c4f46cbaa19fd69d606624f4334b85e50987a08/pillow-12.1.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e2479c7f02f9d505682dc47df8c0ea1fc5e264c4d1629a5d63fe3e2334b89554", size = 6268974, upload-time = "2026-01-02T09:12:02.572Z" },
- { url = "https://files.pythonhosted.org/packages/b4/26/0bf714bc2e73d5267887d47931d53c4ceeceea6978148ed2ab2a4e6463c4/pillow-12.1.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f188d580bd870cda1e15183790d1cc2fa78f666e76077d103edf048eed9c356e", size = 8073070, upload-time = "2026-01-02T09:12:04.75Z" },
- { url = "https://files.pythonhosted.org/packages/43/cf/1ea826200de111a9d65724c54f927f3111dc5ae297f294b370a670c17786/pillow-12.1.0-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0fde7ec5538ab5095cc02df38ee99b0443ff0e1c847a045554cf5f9af1f4aa82", size = 6380176, upload-time = "2026-01-02T09:12:06.626Z" },
- { url = "https://files.pythonhosted.org/packages/03/e0/7938dd2b2013373fd85d96e0f38d62b7a5a262af21ac274250c7ca7847c9/pillow-12.1.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0ed07dca4a8464bada6139ab38f5382f83e5f111698caf3191cb8dbf27d908b4", size = 7067061, upload-time = "2026-01-02T09:12:08.624Z" },
- { url = "https://files.pythonhosted.org/packages/86/ad/a2aa97d37272a929a98437a8c0ac37b3cf012f4f8721e1bd5154699b2518/pillow-12.1.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:f45bd71d1fa5e5749587613037b172e0b3b23159d1c00ef2fc920da6f470e6f0", size = 6491824, upload-time = "2026-01-02T09:12:10.488Z" },
- { url = "https://files.pythonhosted.org/packages/a4/44/80e46611b288d51b115826f136fb3465653c28f491068a72d3da49b54cd4/pillow-12.1.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:277518bf4fe74aa91489e1b20577473b19ee70fb97c374aa50830b279f25841b", size = 7190911, upload-time = "2026-01-02T09:12:12.772Z" },
- { url = "https://files.pythonhosted.org/packages/86/77/eacc62356b4cf81abe99ff9dbc7402750044aed02cfd6a503f7c6fc11f3e/pillow-12.1.0-cp313-cp313t-win32.whl", hash = "sha256:7315f9137087c4e0ee73a761b163fc9aa3b19f5f606a7fc08d83fd3e4379af65", size = 6336445, upload-time = "2026-01-02T09:12:14.775Z" },
- { url = "https://files.pythonhosted.org/packages/e7/3c/57d81d0b74d218706dafccb87a87ea44262c43eef98eb3b164fd000e0491/pillow-12.1.0-cp313-cp313t-win_amd64.whl", hash = "sha256:0ddedfaa8b5f0b4ffbc2fa87b556dc59f6bb4ecb14a53b33f9189713ae8053c0", size = 7045354, upload-time = "2026-01-02T09:12:16.599Z" },
- { url = "https://files.pythonhosted.org/packages/ac/82/8b9b97bba2e3576a340f93b044a3a3a09841170ab4c1eb0d5c93469fd32f/pillow-12.1.0-cp313-cp313t-win_arm64.whl", hash = "sha256:80941e6d573197a0c28f394753de529bb436b1ca990ed6e765cf42426abc39f8", size = 2454547, upload-time = "2026-01-02T09:12:18.704Z" },
- { url = "https://files.pythonhosted.org/packages/8c/87/bdf971d8bbcf80a348cc3bacfcb239f5882100fe80534b0ce67a784181d8/pillow-12.1.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:5cb7bc1966d031aec37ddb9dcf15c2da5b2e9f7cc3ca7c54473a20a927e1eb91", size = 4062533, upload-time = "2026-01-02T09:12:20.791Z" },
- { url = "https://files.pythonhosted.org/packages/ff/4f/5eb37a681c68d605eb7034c004875c81f86ec9ef51f5be4a63eadd58859a/pillow-12.1.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:97e9993d5ed946aba26baf9c1e8cf18adbab584b99f452ee72f7ee8acb882796", size = 4138546, upload-time = "2026-01-02T09:12:23.664Z" },
- { url = "https://files.pythonhosted.org/packages/11/6d/19a95acb2edbace40dcd582d077b991646b7083c41b98da4ed7555b59733/pillow-12.1.0-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:414b9a78e14ffeb98128863314e62c3f24b8a86081066625700b7985b3f529bd", size = 3601163, upload-time = "2026-01-02T09:12:26.338Z" },
- { url = "https://files.pythonhosted.org/packages/fc/36/2b8138e51cb42e4cc39c3297713455548be855a50558c3ac2beebdc251dd/pillow-12.1.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:e6bdb408f7c9dd2a5ff2b14a3b0bb6d4deb29fb9961e6eb3ae2031ae9a5cec13", size = 5266086, upload-time = "2026-01-02T09:12:28.782Z" },
- { url = "https://files.pythonhosted.org/packages/53/4b/649056e4d22e1caa90816bf99cef0884aed607ed38075bd75f091a607a38/pillow-12.1.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:3413c2ae377550f5487991d444428f1a8ae92784aac79caa8b1e3b89b175f77e", size = 4657344, upload-time = "2026-01-02T09:12:31.117Z" },
- { url = "https://files.pythonhosted.org/packages/6c/6b/c5742cea0f1ade0cd61485dc3d81f05261fc2276f537fbdc00802de56779/pillow-12.1.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e5dcbe95016e88437ecf33544ba5db21ef1b8dd6e1b434a2cb2a3d605299e643", size = 6232114, upload-time = "2026-01-02T09:12:32.936Z" },
- { url = "https://files.pythonhosted.org/packages/bf/8f/9f521268ce22d63991601aafd3d48d5ff7280a246a1ef62d626d67b44064/pillow-12.1.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d0a7735df32ccbcc98b98a1ac785cc4b19b580be1bdf0aeb5c03223220ea09d5", size = 8042708, upload-time = "2026-01-02T09:12:34.78Z" },
- { url = "https://files.pythonhosted.org/packages/1a/eb/257f38542893f021502a1bbe0c2e883c90b5cff26cc33b1584a841a06d30/pillow-12.1.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0c27407a2d1b96774cbc4a7594129cc027339fd800cd081e44497722ea1179de", size = 6347762, upload-time = "2026-01-02T09:12:36.748Z" },
- { url = "https://files.pythonhosted.org/packages/c4/5a/8ba375025701c09b309e8d5163c5a4ce0102fa86bbf8800eb0d7ac87bc51/pillow-12.1.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:15c794d74303828eaa957ff8070846d0efe8c630901a1c753fdc63850e19ecd9", size = 7039265, upload-time = "2026-01-02T09:12:39.082Z" },
- { url = "https://files.pythonhosted.org/packages/cf/dc/cf5e4cdb3db533f539e88a7bbf9f190c64ab8a08a9bc7a4ccf55067872e4/pillow-12.1.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c990547452ee2800d8506c4150280757f88532f3de2a58e3022e9b179107862a", size = 6462341, upload-time = "2026-01-02T09:12:40.946Z" },
- { url = "https://files.pythonhosted.org/packages/d0/47/0291a25ac9550677e22eda48510cfc4fa4b2ef0396448b7fbdc0a6946309/pillow-12.1.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:b63e13dd27da389ed9475b3d28510f0f954bca0041e8e551b2a4eb1eab56a39a", size = 7165395, upload-time = "2026-01-02T09:12:42.706Z" },
- { url = "https://files.pythonhosted.org/packages/4f/4c/e005a59393ec4d9416be06e6b45820403bb946a778e39ecec62f5b2b991e/pillow-12.1.0-cp314-cp314-win32.whl", hash = "sha256:1a949604f73eb07a8adab38c4fe50791f9919344398bdc8ac6b307f755fc7030", size = 6431413, upload-time = "2026-01-02T09:12:44.944Z" },
- { url = "https://files.pythonhosted.org/packages/1c/af/f23697f587ac5f9095d67e31b81c95c0249cd461a9798a061ed6709b09b5/pillow-12.1.0-cp314-cp314-win_amd64.whl", hash = "sha256:4f9f6a650743f0ddee5593ac9e954ba1bdbc5e150bc066586d4f26127853ab94", size = 7176779, upload-time = "2026-01-02T09:12:46.727Z" },
- { url = "https://files.pythonhosted.org/packages/b3/36/6a51abf8599232f3e9afbd16d52829376a68909fe14efe29084445db4b73/pillow-12.1.0-cp314-cp314-win_arm64.whl", hash = "sha256:808b99604f7873c800c4840f55ff389936ef1948e4e87645eaf3fccbc8477ac4", size = 2543105, upload-time = "2026-01-02T09:12:49.243Z" },
- { url = "https://files.pythonhosted.org/packages/82/54/2e1dd20c8749ff225080d6ba465a0cab4387f5db0d1c5fb1439e2d99923f/pillow-12.1.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:bc11908616c8a283cf7d664f77411a5ed2a02009b0097ff8abbba5e79128ccf2", size = 5268571, upload-time = "2026-01-02T09:12:51.11Z" },
- { url = "https://files.pythonhosted.org/packages/57/61/571163a5ef86ec0cf30d265ac2a70ae6fc9e28413d1dc94fa37fae6bda89/pillow-12.1.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:896866d2d436563fa2a43a9d72f417874f16b5545955c54a64941e87c1376c61", size = 4660426, upload-time = "2026-01-02T09:12:52.865Z" },
- { url = "https://files.pythonhosted.org/packages/5e/e1/53ee5163f794aef1bf84243f755ee6897a92c708505350dd1923f4afec48/pillow-12.1.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8e178e3e99d3c0ea8fc64b88447f7cac8ccf058af422a6cedc690d0eadd98c51", size = 6269908, upload-time = "2026-01-02T09:12:54.884Z" },
- { url = "https://files.pythonhosted.org/packages/bc/0b/b4b4106ff0ee1afa1dc599fde6ab230417f800279745124f6c50bcffed8e/pillow-12.1.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:079af2fb0c599c2ec144ba2c02766d1b55498e373b3ac64687e43849fbbef5bc", size = 8074733, upload-time = "2026-01-02T09:12:56.802Z" },
- { url = "https://files.pythonhosted.org/packages/19/9f/80b411cbac4a732439e629a26ad3ef11907a8c7fc5377b7602f04f6fe4e7/pillow-12.1.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bdec5e43377761c5dbca620efb69a77f6855c5a379e32ac5b158f54c84212b14", size = 6381431, upload-time = "2026-01-02T09:12:58.823Z" },
- { url = "https://files.pythonhosted.org/packages/8f/b7/d65c45db463b66ecb6abc17c6ba6917a911202a07662247e1355ce1789e7/pillow-12.1.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:565c986f4b45c020f5421a4cea13ef294dde9509a8577f29b2fc5edc7587fff8", size = 7068529, upload-time = "2026-01-02T09:13:00.885Z" },
- { url = "https://files.pythonhosted.org/packages/50/96/dfd4cd726b4a45ae6e3c669fc9e49deb2241312605d33aba50499e9d9bd1/pillow-12.1.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:43aca0a55ce1eefc0aefa6253661cb54571857b1a7b2964bd8a1e3ef4b729924", size = 6492981, upload-time = "2026-01-02T09:13:03.314Z" },
- { url = "https://files.pythonhosted.org/packages/4d/1c/b5dc52cf713ae46033359c5ca920444f18a6359ce1020dd3e9c553ea5bc6/pillow-12.1.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:0deedf2ea233722476b3a81e8cdfbad786f7adbed5d848469fa59fe52396e4ef", size = 7191878, upload-time = "2026-01-02T09:13:05.276Z" },
- { url = "https://files.pythonhosted.org/packages/53/26/c4188248bd5edaf543864fe4834aebe9c9cb4968b6f573ce014cc42d0720/pillow-12.1.0-cp314-cp314t-win32.whl", hash = "sha256:b17fbdbe01c196e7e159aacb889e091f28e61020a8abeac07b68079b6e626988", size = 6438703, upload-time = "2026-01-02T09:13:07.491Z" },
- { url = "https://files.pythonhosted.org/packages/b8/0e/69ed296de8ea05cb03ee139cee600f424ca166e632567b2d66727f08c7ed/pillow-12.1.0-cp314-cp314t-win_amd64.whl", hash = "sha256:27b9baecb428899db6c0de572d6d305cfaf38ca1596b5c0542a5182e3e74e8c6", size = 7182927, upload-time = "2026-01-02T09:13:09.841Z" },
- { url = "https://files.pythonhosted.org/packages/fc/f5/68334c015eed9b5cff77814258717dec591ded209ab5b6fb70e2ae873d1d/pillow-12.1.0-cp314-cp314t-win_arm64.whl", hash = "sha256:f61333d817698bdcdd0f9d7793e365ac3d2a21c1f1eb02b32ad6aefb8d8ea831", size = 2545104, upload-time = "2026-01-02T09:13:12.068Z" },
- { url = "https://files.pythonhosted.org/packages/8b/bc/224b1d98cffd7164b14707c91aac83c07b047fbd8f58eba4066a3e53746a/pillow-12.1.0-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:ca94b6aac0d7af2a10ba08c0f888b3d5114439b6b3ef39968378723622fed377", size = 5228605, upload-time = "2026-01-02T09:13:14.084Z" },
- { url = "https://files.pythonhosted.org/packages/0c/ca/49ca7769c4550107de049ed85208240ba0f330b3f2e316f24534795702ce/pillow-12.1.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:351889afef0f485b84078ea40fe33727a0492b9af3904661b0abbafee0355b72", size = 4622245, upload-time = "2026-01-02T09:13:15.964Z" },
- { url = "https://files.pythonhosted.org/packages/73/48/fac807ce82e5955bcc2718642b94b1bd22a82a6d452aea31cbb678cddf12/pillow-12.1.0-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:bb0984b30e973f7e2884362b7d23d0a348c7143ee559f38ef3eaab640144204c", size = 5247593, upload-time = "2026-01-02T09:13:17.913Z" },
- { url = "https://files.pythonhosted.org/packages/d2/95/3e0742fe358c4664aed4fd05d5f5373dcdad0b27af52aa0972568541e3f4/pillow-12.1.0-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:84cabc7095dd535ca934d57e9ce2a72ffd216e435a84acb06b2277b1de2689bd", size = 6989008, upload-time = "2026-01-02T09:13:20.083Z" },
- { url = "https://files.pythonhosted.org/packages/5a/74/fe2ac378e4e202e56d50540d92e1ef4ff34ed687f3c60f6a121bcf99437e/pillow-12.1.0-pp311-pypy311_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:53d8b764726d3af1a138dd353116f774e3862ec7e3794e0c8781e30db0f35dfc", size = 5313824, upload-time = "2026-01-02T09:13:22.405Z" },
- { url = "https://files.pythonhosted.org/packages/f3/77/2a60dee1adee4e2655ac328dd05c02a955c1cd683b9f1b82ec3feb44727c/pillow-12.1.0-pp311-pypy311_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5da841d81b1a05ef940a8567da92decaa15bc4d7dedb540a8c219ad83d91808a", size = 5963278, upload-time = "2026-01-02T09:13:24.706Z" },
- { url = "https://files.pythonhosted.org/packages/2d/71/64e9b1c7f04ae0027f788a248e6297d7fcc29571371fe7d45495a78172c0/pillow-12.1.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:75af0b4c229ac519b155028fa1be632d812a519abba9b46b20e50c6caa184f19", size = 7029809, upload-time = "2026-01-02T09:13:26.541Z" },
+version = "12.1.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1f/42/5c74462b4fd957fcd7b13b04fb3205ff8349236ea74c7c375766d6c82288/pillow-12.1.1.tar.gz", hash = "sha256:9ad8fa5937ab05218e2b6a4cff30295ad35afd2f83ac592e68c0d871bb0fdbc4", size = 46980264, upload-time = "2026-02-11T04:23:07.146Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/07/d3/8df65da0d4df36b094351dce696f2989bec731d4f10e743b1c5f4da4d3bf/pillow-12.1.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ab323b787d6e18b3d91a72fc99b1a2c28651e4358749842b8f8dfacd28ef2052", size = 5262803, upload-time = "2026-02-11T04:20:47.653Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/71/5026395b290ff404b836e636f51d7297e6c83beceaa87c592718747e670f/pillow-12.1.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:adebb5bee0f0af4909c30db0d890c773d1a92ffe83da908e2e9e720f8edf3984", size = 4657601, upload-time = "2026-02-11T04:20:49.328Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/2e/1001613d941c67442f745aff0f7cc66dd8df9a9c084eb497e6a543ee6f7e/pillow-12.1.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:bb66b7cc26f50977108790e2456b7921e773f23db5630261102233eb355a3b79", size = 6234995, upload-time = "2026-02-11T04:20:51.032Z" },
+ { url = "https://files.pythonhosted.org/packages/07/26/246ab11455b2549b9233dbd44d358d033a2f780fa9007b61a913c5b2d24e/pillow-12.1.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:aee2810642b2898bb187ced9b349e95d2a7272930796e022efaf12e99dccd293", size = 8045012, upload-time = "2026-02-11T04:20:52.882Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/8b/07587069c27be7535ac1fe33874e32de118fbd34e2a73b7f83436a88368c/pillow-12.1.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a0b1cd6232e2b618adcc54d9882e4e662a089d5768cd188f7c245b4c8c44a397", size = 6349638, upload-time = "2026-02-11T04:20:54.444Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/79/6df7b2ee763d619cda2fb4fea498e5f79d984dae304d45a8999b80d6cf5c/pillow-12.1.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7aac39bcf8d4770d089588a2e1dd111cbaa42df5a94be3114222057d68336bd0", size = 7041540, upload-time = "2026-02-11T04:20:55.97Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/5e/2ba19e7e7236d7529f4d873bdaf317a318896bac289abebd4bb00ef247f0/pillow-12.1.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ab174cd7d29a62dd139c44bf74b698039328f45cb03b4596c43473a46656b2f3", size = 6462613, upload-time = "2026-02-11T04:20:57.542Z" },
+ { url = "https://files.pythonhosted.org/packages/03/03/31216ec124bb5c3dacd74ce8efff4cc7f52643653bad4825f8f08c697743/pillow-12.1.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:339ffdcb7cbeaa08221cd401d517d4b1fe7a9ed5d400e4a8039719238620ca35", size = 7166745, upload-time = "2026-02-11T04:20:59.196Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/e7/7c4552d80052337eb28653b617eafdef39adfb137c49dd7e831b8dc13bc5/pillow-12.1.1-cp312-cp312-win32.whl", hash = "sha256:5d1f9575a12bed9e9eedd9a4972834b08c97a352bd17955ccdebfeca5913fa0a", size = 6328823, upload-time = "2026-02-11T04:21:01.385Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/17/688626d192d7261bbbf98846fc98995726bddc2c945344b65bec3a29d731/pillow-12.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:21329ec8c96c6e979cd0dfd29406c40c1d52521a90544463057d2aaa937d66a6", size = 7033367, upload-time = "2026-02-11T04:21:03.536Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/fe/a0ef1f73f939b0eca03ee2c108d0043a87468664770612602c63266a43c4/pillow-12.1.1-cp312-cp312-win_arm64.whl", hash = "sha256:af9a332e572978f0218686636610555ae3defd1633597be015ed50289a03c523", size = 2453811, upload-time = "2026-02-11T04:21:05.116Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/11/6db24d4bd7685583caeae54b7009584e38da3c3d4488ed4cd25b439de486/pillow-12.1.1-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:d242e8ac078781f1de88bf823d70c1a9b3c7950a44cdf4b7c012e22ccbcd8e4e", size = 4062689, upload-time = "2026-02-11T04:21:06.804Z" },
+ { url = "https://files.pythonhosted.org/packages/33/c0/ce6d3b1fe190f0021203e0d9b5b99e57843e345f15f9ef22fcd43842fd21/pillow-12.1.1-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:02f84dfad02693676692746df05b89cf25597560db2857363a208e393429f5e9", size = 4138535, upload-time = "2026-02-11T04:21:08.452Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/c6/d5eb6a4fb32a3f9c21a8c7613ec706534ea1cf9f4b3663e99f0d83f6fca8/pillow-12.1.1-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:e65498daf4b583091ccbb2556c7000abf0f3349fcd57ef7adc9a84a394ed29f6", size = 3601364, upload-time = "2026-02-11T04:21:10.194Z" },
+ { url = "https://files.pythonhosted.org/packages/14/a1/16c4b823838ba4c9c52c0e6bbda903a3fe5a1bdbf1b8eb4fff7156f3e318/pillow-12.1.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:6c6db3b84c87d48d0088943bf33440e0c42370b99b1c2a7989216f7b42eede60", size = 5262561, upload-time = "2026-02-11T04:21:11.742Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/ad/ad9dc98ff24f485008aa5cdedaf1a219876f6f6c42a4626c08bc4e80b120/pillow-12.1.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8b7e5304e34942bf62e15184219a7b5ad4ff7f3bb5cca4d984f37df1a0e1aee2", size = 4657460, upload-time = "2026-02-11T04:21:13.786Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/1b/f1a4ea9a895b5732152789326202a82464d5254759fbacae4deea3069334/pillow-12.1.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:18e5bddd742a44b7e6b1e773ab5db102bd7a94c32555ba656e76d319d19c3850", size = 6232698, upload-time = "2026-02-11T04:21:15.949Z" },
+ { url = "https://files.pythonhosted.org/packages/95/f4/86f51b8745070daf21fd2e5b1fe0eb35d4db9ca26e6d58366562fb56a743/pillow-12.1.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fc44ef1f3de4f45b50ccf9136999d71abb99dca7706bc75d222ed350b9fd2289", size = 8041706, upload-time = "2026-02-11T04:21:17.723Z" },
+ { url = "https://files.pythonhosted.org/packages/29/9b/d6ecd956bb1266dd1045e995cce9b8d77759e740953a1c9aad9502a0461e/pillow-12.1.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5a8eb7ed8d4198bccbd07058416eeec51686b498e784eda166395a23eb99138e", size = 6346621, upload-time = "2026-02-11T04:21:19.547Z" },
+ { url = "https://files.pythonhosted.org/packages/71/24/538bff45bde96535d7d998c6fed1a751c75ac7c53c37c90dc2601b243893/pillow-12.1.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:47b94983da0c642de92ced1702c5b6c292a84bd3a8e1d1702ff923f183594717", size = 7038069, upload-time = "2026-02-11T04:21:21.378Z" },
+ { url = "https://files.pythonhosted.org/packages/94/0e/58cb1a6bc48f746bc4cb3adb8cabff73e2742c92b3bf7a220b7cf69b9177/pillow-12.1.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:518a48c2aab7ce596d3bf79d0e275661b846e86e4d0e7dec34712c30fe07f02a", size = 6460040, upload-time = "2026-02-11T04:21:23.148Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/57/9045cb3ff11eeb6c1adce3b2d60d7d299d7b273a2e6c8381a524abfdc474/pillow-12.1.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:a550ae29b95c6dc13cf69e2c9dc5747f814c54eeb2e32d683e5e93af56caa029", size = 7164523, upload-time = "2026-02-11T04:21:25.01Z" },
+ { url = "https://files.pythonhosted.org/packages/73/f2/9be9cb99f2175f0d4dbadd6616ce1bf068ee54a28277ea1bf1fbf729c250/pillow-12.1.1-cp313-cp313-win32.whl", hash = "sha256:a003d7422449f6d1e3a34e3dd4110c22148336918ddbfc6a32581cd54b2e0b2b", size = 6332552, upload-time = "2026-02-11T04:21:27.238Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/eb/b0834ad8b583d7d9d42b80becff092082a1c3c156bb582590fcc973f1c7c/pillow-12.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:344cf1e3dab3be4b1fa08e449323d98a2a3f819ad20f4b22e77a0ede31f0faa1", size = 7040108, upload-time = "2026-02-11T04:21:29.462Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/7d/fc09634e2aabdd0feabaff4a32f4a7d97789223e7c2042fd805ea4b4d2c2/pillow-12.1.1-cp313-cp313-win_arm64.whl", hash = "sha256:5c0dd1636633e7e6a0afe7bf6a51a14992b7f8e60de5789018ebbdfae55b040a", size = 2453712, upload-time = "2026-02-11T04:21:31.072Z" },
+ { url = "https://files.pythonhosted.org/packages/19/2a/b9d62794fc8a0dd14c1943df68347badbd5511103e0d04c035ffe5cf2255/pillow-12.1.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:0330d233c1a0ead844fc097a7d16c0abff4c12e856c0b325f231820fee1f39da", size = 5264880, upload-time = "2026-02-11T04:21:32.865Z" },
+ { url = "https://files.pythonhosted.org/packages/26/9d/e03d857d1347fa5ed9247e123fcd2a97b6220e15e9cb73ca0a8d91702c6e/pillow-12.1.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:5dae5f21afb91322f2ff791895ddd8889e5e947ff59f71b46041c8ce6db790bc", size = 4660616, upload-time = "2026-02-11T04:21:34.97Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/ec/8a6d22afd02570d30954e043f09c32772bfe143ba9285e2fdb11284952cd/pillow-12.1.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2e0c664be47252947d870ac0d327fea7e63985a08794758aa8af5b6cb6ec0c9c", size = 6269008, upload-time = "2026-02-11T04:21:36.623Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/1d/6d875422c9f28a4a361f495a5f68d9de4a66941dc2c619103ca335fa6446/pillow-12.1.1-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:691ab2ac363b8217f7d31b3497108fb1f50faab2f75dfb03284ec2f217e87bf8", size = 8073226, upload-time = "2026-02-11T04:21:38.585Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/cd/134b0b6ee5eda6dc09e25e24b40fdafe11a520bc725c1d0bbaa5e00bf95b/pillow-12.1.1-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e9e8064fb1cc019296958595f6db671fba95209e3ceb0c4734c9baf97de04b20", size = 6380136, upload-time = "2026-02-11T04:21:40.562Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/a9/7628f013f18f001c1b98d8fffe3452f306a70dc6aba7d931019e0492f45e/pillow-12.1.1-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:472a8d7ded663e6162dafdf20015c486a7009483ca671cece7a9279b512fcb13", size = 7067129, upload-time = "2026-02-11T04:21:42.521Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/f8/66ab30a2193b277785601e82ee2d49f68ea575d9637e5e234faaa98efa4c/pillow-12.1.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:89b54027a766529136a06cfebeecb3a04900397a3590fd252160b888479517bf", size = 6491807, upload-time = "2026-02-11T04:21:44.22Z" },
+ { url = "https://files.pythonhosted.org/packages/da/0b/a877a6627dc8318fdb84e357c5e1a758c0941ab1ddffdafd231983788579/pillow-12.1.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:86172b0831b82ce4f7877f280055892b31179e1576aa00d0df3bb1bbf8c3e524", size = 7190954, upload-time = "2026-02-11T04:21:46.114Z" },
+ { url = "https://files.pythonhosted.org/packages/83/43/6f732ff85743cf746b1361b91665d9f5155e1483817f693f8d57ea93147f/pillow-12.1.1-cp313-cp313t-win32.whl", hash = "sha256:44ce27545b6efcf0fdbdceb31c9a5bdea9333e664cda58a7e674bb74608b3986", size = 6336441, upload-time = "2026-02-11T04:21:48.22Z" },
+ { url = "https://files.pythonhosted.org/packages/3b/44/e865ef3986611bb75bfabdf94a590016ea327833f434558801122979cd0e/pillow-12.1.1-cp313-cp313t-win_amd64.whl", hash = "sha256:a285e3eb7a5a45a2ff504e31f4a8d1b12ef62e84e5411c6804a42197c1cf586c", size = 7045383, upload-time = "2026-02-11T04:21:50.015Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/c6/f4fb24268d0c6908b9f04143697ea18b0379490cb74ba9e8d41b898bd005/pillow-12.1.1-cp313-cp313t-win_arm64.whl", hash = "sha256:cc7d296b5ea4d29e6570dabeaed58d31c3fea35a633a69679fb03d7664f43fb3", size = 2456104, upload-time = "2026-02-11T04:21:51.633Z" },
+ { url = "https://files.pythonhosted.org/packages/03/d0/bebb3ffbf31c5a8e97241476c4cf8b9828954693ce6744b4a2326af3e16b/pillow-12.1.1-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:417423db963cb4be8bac3fc1204fe61610f6abeed1580a7a2cbb2fbda20f12af", size = 4062652, upload-time = "2026-02-11T04:21:53.19Z" },
+ { url = "https://files.pythonhosted.org/packages/2d/c0/0e16fb0addda4851445c28f8350d8c512f09de27bbb0d6d0bbf8b6709605/pillow-12.1.1-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:b957b71c6b2387610f556a7eb0828afbe40b4a98036fc0d2acfa5a44a0c2036f", size = 4138823, upload-time = "2026-02-11T04:22:03.088Z" },
+ { url = "https://files.pythonhosted.org/packages/6b/fb/6170ec655d6f6bb6630a013dd7cf7bc218423d7b5fa9071bf63dc32175ae/pillow-12.1.1-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:097690ba1f2efdeb165a20469d59d8bb03c55fb6621eb2041a060ae8ea3e9642", size = 3601143, upload-time = "2026-02-11T04:22:04.909Z" },
+ { url = "https://files.pythonhosted.org/packages/59/04/dc5c3f297510ba9a6837cbb318b87dd2b8f73eb41a43cc63767f65cb599c/pillow-12.1.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:2815a87ab27848db0321fb78c7f0b2c8649dee134b7f2b80c6a45c6831d75ccd", size = 5266254, upload-time = "2026-02-11T04:22:07.656Z" },
+ { url = "https://files.pythonhosted.org/packages/05/30/5db1236b0d6313f03ebf97f5e17cda9ca060f524b2fcc875149a8360b21c/pillow-12.1.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:f7ed2c6543bad5a7d5530eb9e78c53132f93dfa44a28492db88b41cdab885202", size = 4657499, upload-time = "2026-02-11T04:22:09.613Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/18/008d2ca0eb612e81968e8be0bbae5051efba24d52debf930126d7eaacbba/pillow-12.1.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:652a2c9ccfb556235b2b501a3a7cf3742148cd22e04b5625c5fe057ea3e3191f", size = 6232137, upload-time = "2026-02-11T04:22:11.434Z" },
+ { url = "https://files.pythonhosted.org/packages/70/f1/f14d5b8eeb4b2cd62b9f9f847eb6605f103df89ef619ac68f92f748614ea/pillow-12.1.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d6e4571eedf43af33d0fc233a382a76e849badbccdf1ac438841308652a08e1f", size = 8042721, upload-time = "2026-02-11T04:22:13.321Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/d6/17824509146e4babbdabf04d8171491fa9d776f7061ff6e727522df9bd03/pillow-12.1.1-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b574c51cf7d5d62e9be37ba446224b59a2da26dc4c1bb2ecbe936a4fb1a7cb7f", size = 6347798, upload-time = "2026-02-11T04:22:15.449Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/ee/c85a38a9ab92037a75615aba572c85ea51e605265036e00c5b67dfafbfe2/pillow-12.1.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a37691702ed687799de29a518d63d4682d9016932db66d4e90c345831b02fb4e", size = 7039315, upload-time = "2026-02-11T04:22:17.24Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/f3/bc8ccc6e08a148290d7523bde4d9a0d6c981db34631390dc6e6ec34cacf6/pillow-12.1.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:f95c00d5d6700b2b890479664a06e754974848afaae5e21beb4d83c106923fd0", size = 6462360, upload-time = "2026-02-11T04:22:19.111Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/ab/69a42656adb1d0665ab051eec58a41f169ad295cf81ad45406963105408f/pillow-12.1.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:559b38da23606e68681337ad74622c4dbba02254fc9cb4488a305dd5975c7eeb", size = 7165438, upload-time = "2026-02-11T04:22:21.041Z" },
+ { url = "https://files.pythonhosted.org/packages/02/46/81f7aa8941873f0f01d4b55cc543b0a3d03ec2ee30d617a0448bf6bd6dec/pillow-12.1.1-cp314-cp314-win32.whl", hash = "sha256:03edcc34d688572014ff223c125a3f77fb08091e4607e7745002fc214070b35f", size = 6431503, upload-time = "2026-02-11T04:22:22.833Z" },
+ { url = "https://files.pythonhosted.org/packages/40/72/4c245f7d1044b67affc7f134a09ea619d4895333d35322b775b928180044/pillow-12.1.1-cp314-cp314-win_amd64.whl", hash = "sha256:50480dcd74fa63b8e78235957d302d98d98d82ccbfac4c7e12108ba9ecbdba15", size = 7176748, upload-time = "2026-02-11T04:22:24.64Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/ad/8a87bdbe038c5c698736e3348af5c2194ffb872ea52f11894c95f9305435/pillow-12.1.1-cp314-cp314-win_arm64.whl", hash = "sha256:5cb1785d97b0c3d1d1a16bc1d710c4a0049daefc4935f3a8f31f827f4d3d2e7f", size = 2544314, upload-time = "2026-02-11T04:22:26.685Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/9d/efd18493f9de13b87ede7c47e69184b9e859e4427225ea962e32e56a49bc/pillow-12.1.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:1f90cff8aa76835cba5769f0b3121a22bd4eb9e6884cfe338216e557a9a548b8", size = 5268612, upload-time = "2026-02-11T04:22:29.884Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/f1/4f42eb2b388eb2ffc660dcb7f7b556c1015c53ebd5f7f754965ef997585b/pillow-12.1.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1f1be78ce9466a7ee64bfda57bdba0f7cc499d9794d518b854816c41bf0aa4e9", size = 4660567, upload-time = "2026-02-11T04:22:31.799Z" },
+ { url = "https://files.pythonhosted.org/packages/01/54/df6ef130fa43e4b82e32624a7b821a2be1c5653a5fdad8469687a7db4e00/pillow-12.1.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:42fc1f4677106188ad9a55562bbade416f8b55456f522430fadab3cef7cd4e60", size = 6269951, upload-time = "2026-02-11T04:22:33.921Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/48/618752d06cc44bb4aae8ce0cd4e6426871929ed7b46215638088270d9b34/pillow-12.1.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:98edb152429ab62a1818039744d8fbb3ccab98a7c29fc3d5fcef158f3f1f68b7", size = 8074769, upload-time = "2026-02-11T04:22:35.877Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/bd/f1d71eb39a72fa088d938655afba3e00b38018d052752f435838961127d8/pillow-12.1.1-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d470ab1178551dd17fdba0fef463359c41aaa613cdcd7ff8373f54be629f9f8f", size = 6381358, upload-time = "2026-02-11T04:22:37.698Z" },
+ { url = "https://files.pythonhosted.org/packages/64/ef/c784e20b96674ed36a5af839305f55616f8b4f8aa8eeccf8531a6e312243/pillow-12.1.1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6408a7b064595afcab0a49393a413732a35788f2a5092fdc6266952ed67de586", size = 7068558, upload-time = "2026-02-11T04:22:39.597Z" },
+ { url = "https://files.pythonhosted.org/packages/73/cb/8059688b74422ae61278202c4e1ad992e8a2e7375227be0a21c6b87ca8d5/pillow-12.1.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:5d8c41325b382c07799a3682c1c258469ea2ff97103c53717b7893862d0c98ce", size = 6493028, upload-time = "2026-02-11T04:22:42.73Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/da/e3c008ed7d2dd1f905b15949325934510b9d1931e5df999bb15972756818/pillow-12.1.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:c7697918b5be27424e9ce568193efd13d925c4481dd364e43f5dff72d33e10f8", size = 7191940, upload-time = "2026-02-11T04:22:44.543Z" },
+ { url = "https://files.pythonhosted.org/packages/01/4a/9202e8d11714c1fc5951f2e1ef362f2d7fbc595e1f6717971d5dd750e969/pillow-12.1.1-cp314-cp314t-win32.whl", hash = "sha256:d2912fd8114fc5545aa3a4b5576512f64c55a03f3ebcca4c10194d593d43ea36", size = 6438736, upload-time = "2026-02-11T04:22:46.347Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/ca/cbce2327eb9885476b3957b2e82eb12c866a8b16ad77392864ad601022ce/pillow-12.1.1-cp314-cp314t-win_amd64.whl", hash = "sha256:4ceb838d4bd9dab43e06c363cab2eebf63846d6a4aeaea283bbdfd8f1a8ed58b", size = 7182894, upload-time = "2026-02-11T04:22:48.114Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/d2/de599c95ba0a973b94410477f8bf0b6f0b5e67360eb89bcb1ad365258beb/pillow-12.1.1-cp314-cp314t-win_arm64.whl", hash = "sha256:7b03048319bfc6170e93bd60728a1af51d3dd7704935feb228c4d4faab35d334", size = 2546446, upload-time = "2026-02-11T04:22:50.342Z" },
]
[[package]]
name = "platformdirs"
-version = "4.5.1"
+version = "4.9.4"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/cf/86/0248f086a84f01b37aaec0fa567b397df1a119f73c16f6c7a9aac73ea309/platformdirs-4.5.1.tar.gz", hash = "sha256:61d5cdcc6065745cdd94f0f878977f8de9437be93de97c1c12f853c9c0cdcbda", size = 21715, upload-time = "2025-12-05T13:52:58.638Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/19/56/8d4c30c8a1d07013911a8fdbd8f89440ef9f08d07a1b50ab8ca8be5a20f9/platformdirs-4.9.4.tar.gz", hash = "sha256:1ec356301b7dc906d83f371c8f487070e99d3ccf9e501686456394622a01a934", size = 28737, upload-time = "2026-03-05T18:34:13.271Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/cb/28/3bfe2fa5a7b9c46fe7e13c97bda14c895fb10fa2ebf1d0abb90e0cea7ee1/platformdirs-4.5.1-py3-none-any.whl", hash = "sha256:d03afa3963c806a9bed9d5125c8f4cb2fdaf74a55ab60e5d59b3fde758104d31", size = 18731, upload-time = "2025-12-05T13:52:56.823Z" },
+ { url = "https://files.pythonhosted.org/packages/63/d7/97f7e3a6abb67d8080dd406fd4df842c2be0efaf712d1c899c32a075027c/platformdirs-4.9.4-py3-none-any.whl", hash = "sha256:68a9a4619a666ea6439f2ff250c12a853cd1cbd5158d258bd824a7df6be2f868", size = 21216, upload-time = "2026-03-05T18:34:12.172Z" },
]
[[package]]
@@ -3591,13 +4604,22 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/ee/8c/83087ebc47ab0396ce092363001fa37c17153119ee282700c0713a195853/prettytable-3.17.0-py3-none-any.whl", hash = "sha256:aad69b294ddbe3e1f95ef8886a060ed1666a0b83018bbf56295f6f226c43d287", size = 34433, upload-time = "2025-11-14T17:33:19.093Z" },
]
+[[package]]
+name = "priority"
+version = "2.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f5/3c/eb7c35f4dcede96fca1842dac5f4f5d15511aa4b52f3a961219e68ae9204/priority-2.0.0.tar.gz", hash = "sha256:c965d54f1b8d0d0b19479db3924c7c36cf672dbf2aec92d43fbdaf4492ba18c0", size = 24792, upload-time = "2021-06-27T10:15:05.487Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5e/5f/82c8074f7e84978129347c2c6ec8b6c59f3584ff1a20bc3c940a3e061790/priority-2.0.0-py3-none-any.whl", hash = "sha256:6f8eefce5f3ad59baf2c080a664037bb4725cd0a790d53d59ab4059288faf6aa", size = 8946, upload-time = "2021-06-27T10:15:03.856Z" },
+]
+
[[package]]
name = "prometheus-client"
-version = "0.24.0"
+version = "0.24.1"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/07/8f/35d31c925f33a494b3f4f10ee25bf47757aff2d63424a06af13814293f13/prometheus_client-0.24.0.tar.gz", hash = "sha256:726b40c0d499f4904d4b5b7abe8d43e6aff090de0d468ae8f2226290b331c667", size = 85590, upload-time = "2026-01-12T20:12:48.963Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/f0/58/a794d23feb6b00fc0c72787d7e87d872a6730dd9ed7c7b3e954637d8f280/prometheus_client-0.24.1.tar.gz", hash = "sha256:7e0ced7fbbd40f7b84962d5d2ab6f17ef88a72504dcf7c0b40737b43b2a461f9", size = 85616, upload-time = "2026-01-14T15:26:26.965Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/22/dd/50260b80759f90e3be66f094e0cd1fdef680b18d9f91edc9ae1b627624ba/prometheus_client-0.24.0-py3-none-any.whl", hash = "sha256:4ab6d4fb5a1b25ad74b58e6271857e356fff3399473e599d227ab5d0ce6637f0", size = 64062, upload-time = "2026-01-12T20:12:47.501Z" },
+ { url = "https://files.pythonhosted.org/packages/74/c3/24a2f845e3917201628ecaba4f18bab4d18a337834c1df2a159ee9d22a42/prometheus_client-0.24.1-py3-none-any.whl", hash = "sha256:150db128af71a5c2482b36e588fc8a6b95e498750da4b17065947c16070f4055", size = 64057, upload-time = "2026-01-14T15:26:24.42Z" },
]
[[package]]
@@ -3631,36 +4653,6 @@ version = "0.4.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/9e/da/e9fc233cf63743258bff22b3dfa7ea5baef7b5bc324af47a0ad89b8ffc6f/propcache-0.4.1.tar.gz", hash = "sha256:f48107a8c637e80362555f37ecf49abe20370e557cc4ab374f04ec4423c97c3d", size = 46442, upload-time = "2025-10-08T19:49:02.291Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/3c/0e/934b541323035566a9af292dba85a195f7b78179114f2c6ebb24551118a9/propcache-0.4.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:7c2d1fa3201efaf55d730400d945b5b3ab6e672e100ba0f9a409d950ab25d7db", size = 79534, upload-time = "2025-10-08T19:46:02.083Z" },
- { url = "https://files.pythonhosted.org/packages/a1/6b/db0d03d96726d995dc7171286c6ba9d8d14251f37433890f88368951a44e/propcache-0.4.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:1eb2994229cc8ce7fe9b3db88f5465f5fd8651672840b2e426b88cdb1a30aac8", size = 45526, upload-time = "2025-10-08T19:46:03.884Z" },
- { url = "https://files.pythonhosted.org/packages/e4/c3/82728404aea669e1600f304f2609cde9e665c18df5a11cdd57ed73c1dceb/propcache-0.4.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:66c1f011f45a3b33d7bcb22daed4b29c0c9e2224758b6be00686731e1b46f925", size = 47263, upload-time = "2025-10-08T19:46:05.405Z" },
- { url = "https://files.pythonhosted.org/packages/df/1b/39313ddad2bf9187a1432654c38249bab4562ef535ef07f5eb6eb04d0b1b/propcache-0.4.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9a52009f2adffe195d0b605c25ec929d26b36ef986ba85244891dee3b294df21", size = 201012, upload-time = "2025-10-08T19:46:07.165Z" },
- { url = "https://files.pythonhosted.org/packages/5b/01/f1d0b57d136f294a142acf97f4ed58c8e5b974c21e543000968357115011/propcache-0.4.1-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5d4e2366a9c7b837555cf02fb9be2e3167d333aff716332ef1b7c3a142ec40c5", size = 209491, upload-time = "2025-10-08T19:46:08.909Z" },
- { url = "https://files.pythonhosted.org/packages/a1/c8/038d909c61c5bb039070b3fb02ad5cccdb1dde0d714792e251cdb17c9c05/propcache-0.4.1-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:9d2b6caef873b4f09e26ea7e33d65f42b944837563a47a94719cc3544319a0db", size = 215319, upload-time = "2025-10-08T19:46:10.7Z" },
- { url = "https://files.pythonhosted.org/packages/08/57/8c87e93142b2c1fa2408e45695205a7ba05fb5db458c0bf5c06ba0e09ea6/propcache-0.4.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2b16ec437a8c8a965ecf95739448dd938b5c7f56e67ea009f4300d8df05f32b7", size = 196856, upload-time = "2025-10-08T19:46:12.003Z" },
- { url = "https://files.pythonhosted.org/packages/42/df/5615fec76aa561987a534759b3686008a288e73107faa49a8ae5795a9f7a/propcache-0.4.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:296f4c8ed03ca7476813fe666c9ea97869a8d7aec972618671b33a38a5182ef4", size = 193241, upload-time = "2025-10-08T19:46:13.495Z" },
- { url = "https://files.pythonhosted.org/packages/d5/21/62949eb3a7a54afe8327011c90aca7e03547787a88fb8bd9726806482fea/propcache-0.4.1-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:1f0978529a418ebd1f49dad413a2b68af33f85d5c5ca5c6ca2a3bed375a7ac60", size = 190552, upload-time = "2025-10-08T19:46:14.938Z" },
- { url = "https://files.pythonhosted.org/packages/30/ee/ab4d727dd70806e5b4de96a798ae7ac6e4d42516f030ee60522474b6b332/propcache-0.4.1-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:fd138803047fb4c062b1c1dd95462f5209456bfab55c734458f15d11da288f8f", size = 200113, upload-time = "2025-10-08T19:46:16.695Z" },
- { url = "https://files.pythonhosted.org/packages/8a/0b/38b46208e6711b016aa8966a3ac793eee0d05c7159d8342aa27fc0bc365e/propcache-0.4.1-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:8c9b3cbe4584636d72ff556d9036e0c9317fa27b3ac1f0f558e7e84d1c9c5900", size = 200778, upload-time = "2025-10-08T19:46:18.023Z" },
- { url = "https://files.pythonhosted.org/packages/cf/81/5abec54355ed344476bee711e9f04815d4b00a311ab0535599204eecc257/propcache-0.4.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:f93243fdc5657247533273ac4f86ae106cc6445a0efacb9a1bfe982fcfefd90c", size = 193047, upload-time = "2025-10-08T19:46:19.449Z" },
- { url = "https://files.pythonhosted.org/packages/ec/b6/1f237c04e32063cb034acd5f6ef34ef3a394f75502e72703545631ab1ef6/propcache-0.4.1-cp310-cp310-win32.whl", hash = "sha256:a0ee98db9c5f80785b266eb805016e36058ac72c51a064040f2bc43b61101cdb", size = 38093, upload-time = "2025-10-08T19:46:20.643Z" },
- { url = "https://files.pythonhosted.org/packages/a6/67/354aac4e0603a15f76439caf0427781bcd6797f370377f75a642133bc954/propcache-0.4.1-cp310-cp310-win_amd64.whl", hash = "sha256:1cdb7988c4e5ac7f6d175a28a9aa0c94cb6f2ebe52756a3c0cda98d2809a9e37", size = 41638, upload-time = "2025-10-08T19:46:21.935Z" },
- { url = "https://files.pythonhosted.org/packages/e0/e1/74e55b9fd1a4c209ff1a9a824bf6c8b3d1fc5a1ac3eabe23462637466785/propcache-0.4.1-cp310-cp310-win_arm64.whl", hash = "sha256:d82ad62b19645419fe79dd63b3f9253e15b30e955c0170e5cebc350c1844e581", size = 38229, upload-time = "2025-10-08T19:46:23.368Z" },
- { url = "https://files.pythonhosted.org/packages/8c/d4/4e2c9aaf7ac2242b9358f98dccd8f90f2605402f5afeff6c578682c2c491/propcache-0.4.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:60a8fda9644b7dfd5dece8c61d8a85e271cb958075bfc4e01083c148b61a7caf", size = 80208, upload-time = "2025-10-08T19:46:24.597Z" },
- { url = "https://files.pythonhosted.org/packages/c2/21/d7b68e911f9c8e18e4ae43bdbc1e1e9bbd971f8866eb81608947b6f585ff/propcache-0.4.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:c30b53e7e6bda1d547cabb47c825f3843a0a1a42b0496087bb58d8fedf9f41b5", size = 45777, upload-time = "2025-10-08T19:46:25.733Z" },
- { url = "https://files.pythonhosted.org/packages/d3/1d/11605e99ac8ea9435651ee71ab4cb4bf03f0949586246476a25aadfec54a/propcache-0.4.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:6918ecbd897443087a3b7cd978d56546a812517dcaaca51b49526720571fa93e", size = 47647, upload-time = "2025-10-08T19:46:27.304Z" },
- { url = "https://files.pythonhosted.org/packages/58/1a/3c62c127a8466c9c843bccb503d40a273e5cc69838805f322e2826509e0d/propcache-0.4.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3d902a36df4e5989763425a8ab9e98cd8ad5c52c823b34ee7ef307fd50582566", size = 214929, upload-time = "2025-10-08T19:46:28.62Z" },
- { url = "https://files.pythonhosted.org/packages/56/b9/8fa98f850960b367c4b8fe0592e7fc341daa7a9462e925228f10a60cf74f/propcache-0.4.1-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a9695397f85973bb40427dedddf70d8dc4a44b22f1650dd4af9eedf443d45165", size = 221778, upload-time = "2025-10-08T19:46:30.358Z" },
- { url = "https://files.pythonhosted.org/packages/46/a6/0ab4f660eb59649d14b3d3d65c439421cf2f87fe5dd68591cbe3c1e78a89/propcache-0.4.1-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2bb07ffd7eaad486576430c89f9b215f9e4be68c4866a96e97db9e97fead85dc", size = 228144, upload-time = "2025-10-08T19:46:32.607Z" },
- { url = "https://files.pythonhosted.org/packages/52/6a/57f43e054fb3d3a56ac9fc532bc684fc6169a26c75c353e65425b3e56eef/propcache-0.4.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fd6f30fdcf9ae2a70abd34da54f18da086160e4d7d9251f81f3da0ff84fc5a48", size = 210030, upload-time = "2025-10-08T19:46:33.969Z" },
- { url = "https://files.pythonhosted.org/packages/40/e2/27e6feebb5f6b8408fa29f5efbb765cd54c153ac77314d27e457a3e993b7/propcache-0.4.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:fc38cba02d1acba4e2869eef1a57a43dfbd3d49a59bf90dda7444ec2be6a5570", size = 208252, upload-time = "2025-10-08T19:46:35.309Z" },
- { url = "https://files.pythonhosted.org/packages/9e/f8/91c27b22ccda1dbc7967f921c42825564fa5336a01ecd72eb78a9f4f53c2/propcache-0.4.1-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:67fad6162281e80e882fb3ec355398cf72864a54069d060321f6cd0ade95fe85", size = 202064, upload-time = "2025-10-08T19:46:36.993Z" },
- { url = "https://files.pythonhosted.org/packages/f2/26/7f00bd6bd1adba5aafe5f4a66390f243acab58eab24ff1a08bebb2ef9d40/propcache-0.4.1-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:f10207adf04d08bec185bae14d9606a1444715bc99180f9331c9c02093e1959e", size = 212429, upload-time = "2025-10-08T19:46:38.398Z" },
- { url = "https://files.pythonhosted.org/packages/84/89/fd108ba7815c1117ddca79c228f3f8a15fc82a73bca8b142eb5de13b2785/propcache-0.4.1-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:e9b0d8d0845bbc4cfcdcbcdbf5086886bc8157aa963c31c777ceff7846c77757", size = 216727, upload-time = "2025-10-08T19:46:39.732Z" },
- { url = "https://files.pythonhosted.org/packages/79/37/3ec3f7e3173e73f1d600495d8b545b53802cbf35506e5732dd8578db3724/propcache-0.4.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:981333cb2f4c1896a12f4ab92a9cc8f09ea664e9b7dbdc4eff74627af3a11c0f", size = 205097, upload-time = "2025-10-08T19:46:41.025Z" },
- { url = "https://files.pythonhosted.org/packages/61/b0/b2631c19793f869d35f47d5a3a56fb19e9160d3c119f15ac7344fc3ccae7/propcache-0.4.1-cp311-cp311-win32.whl", hash = "sha256:f1d2f90aeec838a52f1c1a32fe9a619fefd5e411721a9117fbf82aea638fe8a1", size = 38084, upload-time = "2025-10-08T19:46:42.693Z" },
- { url = "https://files.pythonhosted.org/packages/f4/78/6cce448e2098e9f3bfc91bb877f06aa24b6ccace872e39c53b2f707c4648/propcache-0.4.1-cp311-cp311-win_amd64.whl", hash = "sha256:364426a62660f3f699949ac8c621aad6977be7126c5807ce48c0aeb8e7333ea6", size = 41637, upload-time = "2025-10-08T19:46:43.778Z" },
- { url = "https://files.pythonhosted.org/packages/9c/e9/754f180cccd7f51a39913782c74717c581b9cc8177ad0e949f4d51812383/propcache-0.4.1-cp311-cp311-win_arm64.whl", hash = "sha256:e53f3a38d3510c11953f3e6a33f205c6d1b001129f972805ca9b42fc308bc239", size = 38064, upload-time = "2025-10-08T19:46:44.872Z" },
{ url = "https://files.pythonhosted.org/packages/a2/0f/f17b1b2b221d5ca28b4b876e8bb046ac40466513960646bda8e1853cdfa2/propcache-0.4.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:e153e9cd40cc8945138822807139367f256f89c6810c2634a4f6902b52d3b4e2", size = 80061, upload-time = "2025-10-08T19:46:46.075Z" },
{ url = "https://files.pythonhosted.org/packages/76/47/8ccf75935f51448ba9a16a71b783eb7ef6b9ee60f5d14c7f8a8a79fbeed7/propcache-0.4.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:cd547953428f7abb73c5ad82cbb32109566204260d98e41e5dfdc682eb7f8403", size = 46037, upload-time = "2025-10-08T19:46:47.23Z" },
{ url = "https://files.pythonhosted.org/packages/0a/b6/5c9a0e42df4d00bfb4a3cbbe5cf9f54260300c88a0e9af1f47ca5ce17ac0/propcache-0.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f048da1b4f243fc44f205dfd320933a951b8d89e0afd4c7cacc762a8b9165207", size = 47324, upload-time = "2025-10-08T19:46:48.384Z" },
@@ -3739,47 +4731,59 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/5b/5a/bc7b4a4ef808fa59a816c17b20c4bef6884daebbdf627ff2a161da67da19/propcache-0.4.1-py3-none-any.whl", hash = "sha256:af2a6052aeb6cf17d3e46ee169099044fd8224cbaf75c76a2ef596e8163e2237", size = 13305, upload-time = "2025-10-08T19:49:00.792Z" },
]
+[[package]]
+name = "proto-plus"
+version = "1.27.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "protobuf" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/81/0d/94dfe80193e79d55258345901acd2917523d56e8381bc4dee7fd38e3868a/proto_plus-1.27.2.tar.gz", hash = "sha256:b2adde53adadf75737c44d3dcb0104fde65250dfc83ad59168b4aa3e574b6a24", size = 57204, upload-time = "2026-03-26T22:18:57.174Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/84/f3/1fba73eeffafc998a25d59703b63f8be4fe8a5cb12eaff7386a0ba0f7125/proto_plus-1.27.2-py3-none-any.whl", hash = "sha256:6432f75893d3b9e70b9c412f1d2f03f65b11fb164b793d14ae2ca01821d22718", size = 50450, upload-time = "2026-03-26T22:13:42.927Z" },
+]
+
[[package]]
name = "protobuf"
-version = "6.33.4"
+version = "6.33.6"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/53/b8/cda15d9d46d03d4aa3a67cb6bffe05173440ccf86a9541afaf7ac59a1b6b/protobuf-6.33.4.tar.gz", hash = "sha256:dc2e61bca3b10470c1912d166fe0af67bfc20eb55971dcef8dfa48ce14f0ed91", size = 444346, upload-time = "2026-01-12T18:33:40.109Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/66/70/e908e9c5e52ef7c3a6c7902c9dfbb34c7e29c25d2f81ade3856445fd5c94/protobuf-6.33.6.tar.gz", hash = "sha256:a6768d25248312c297558af96a9f9c929e8c4cee0659cb07e780731095f38135", size = 444531, upload-time = "2026-03-18T19:05:00.988Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/e0/be/24ef9f3095bacdf95b458543334d0c4908ccdaee5130420bf064492c325f/protobuf-6.33.4-cp310-abi3-win32.whl", hash = "sha256:918966612c8232fc6c24c78e1cd89784307f5814ad7506c308ee3cf86662850d", size = 425612, upload-time = "2026-01-12T18:33:29.656Z" },
- { url = "https://files.pythonhosted.org/packages/31/ad/e5693e1974a28869e7cd244302911955c1cebc0161eb32dfa2b25b6e96f0/protobuf-6.33.4-cp310-abi3-win_amd64.whl", hash = "sha256:8f11ffae31ec67fc2554c2ef891dcb561dae9a2a3ed941f9e134c2db06657dbc", size = 436962, upload-time = "2026-01-12T18:33:31.345Z" },
- { url = "https://files.pythonhosted.org/packages/66/15/6ee23553b6bfd82670207ead921f4d8ef14c107e5e11443b04caeb5ab5ec/protobuf-6.33.4-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:2fe67f6c014c84f655ee06f6f66213f9254b3a8b6bda6cda0ccd4232c73c06f0", size = 427612, upload-time = "2026-01-12T18:33:32.646Z" },
- { url = "https://files.pythonhosted.org/packages/2b/48/d301907ce6d0db75f959ca74f44b475a9caa8fcba102d098d3c3dd0f2d3f/protobuf-6.33.4-cp39-abi3-manylinux2014_aarch64.whl", hash = "sha256:757c978f82e74d75cba88eddec479df9b99a42b31193313b75e492c06a51764e", size = 324484, upload-time = "2026-01-12T18:33:33.789Z" },
- { url = "https://files.pythonhosted.org/packages/92/1c/e53078d3f7fe710572ab2dcffd993e1e3b438ae71cfc031b71bae44fcb2d/protobuf-6.33.4-cp39-abi3-manylinux2014_s390x.whl", hash = "sha256:c7c64f259c618f0bef7bee042075e390debbf9682334be2b67408ec7c1c09ee6", size = 339256, upload-time = "2026-01-12T18:33:35.231Z" },
- { url = "https://files.pythonhosted.org/packages/e8/8e/971c0edd084914f7ee7c23aa70ba89e8903918adca179319ee94403701d5/protobuf-6.33.4-cp39-abi3-manylinux2014_x86_64.whl", hash = "sha256:3df850c2f8db9934de4cf8f9152f8dc2558f49f298f37f90c517e8e5c84c30e9", size = 323311, upload-time = "2026-01-12T18:33:36.305Z" },
- { url = "https://files.pythonhosted.org/packages/75/b1/1dc83c2c661b4c62d56cc081706ee33a4fc2835bd90f965baa2663ef7676/protobuf-6.33.4-py3-none-any.whl", hash = "sha256:1fe3730068fcf2e595816a6c34fe66eeedd37d51d0400b72fabc848811fdc1bc", size = 170532, upload-time = "2026-01-12T18:33:39.199Z" },
+ { url = "https://files.pythonhosted.org/packages/fc/9f/2f509339e89cfa6f6a4c4ff50438db9ca488dec341f7e454adad60150b00/protobuf-6.33.6-cp310-abi3-win32.whl", hash = "sha256:7d29d9b65f8afef196f8334e80d6bc1d5d4adedb449971fefd3723824e6e77d3", size = 425739, upload-time = "2026-03-18T19:04:48.373Z" },
+ { url = "https://files.pythonhosted.org/packages/76/5d/683efcd4798e0030c1bab27374fd13a89f7c2515fb1f3123efdfaa5eab57/protobuf-6.33.6-cp310-abi3-win_amd64.whl", hash = "sha256:0cd27b587afca21b7cfa59a74dcbd48a50f0a6400cfb59391340ad729d91d326", size = 437089, upload-time = "2026-03-18T19:04:50.381Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/01/a3c3ed5cd186f39e7880f8303cc51385a198a81469d53d0fdecf1f64d929/protobuf-6.33.6-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:9720e6961b251bde64edfdab7d500725a2af5280f3f4c87e57c0208376aa8c3a", size = 427737, upload-time = "2026-03-18T19:04:51.866Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/90/b3c01fdec7d2f627b3a6884243ba328c1217ed2d978def5c12dc50d328a3/protobuf-6.33.6-cp39-abi3-manylinux2014_aarch64.whl", hash = "sha256:e2afbae9b8e1825e3529f88d514754e094278bb95eadc0e199751cdd9a2e82a2", size = 324610, upload-time = "2026-03-18T19:04:53.096Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/ca/25afc144934014700c52e05103c2421997482d561f3101ff352e1292fb81/protobuf-6.33.6-cp39-abi3-manylinux2014_s390x.whl", hash = "sha256:c96c37eec15086b79762ed265d59ab204dabc53056e3443e702d2681f4b39ce3", size = 339381, upload-time = "2026-03-18T19:04:54.616Z" },
+ { url = "https://files.pythonhosted.org/packages/16/92/d1e32e3e0d894fe00b15ce28ad4944ab692713f2e7f0a99787405e43533a/protobuf-6.33.6-cp39-abi3-manylinux2014_x86_64.whl", hash = "sha256:e9db7e292e0ab79dd108d7f1a94fe31601ce1ee3f7b79e0692043423020b0593", size = 323436, upload-time = "2026-03-18T19:04:55.768Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/72/02445137af02769918a93807b2b7890047c32bfb9f90371cbc12688819eb/protobuf-6.33.6-py3-none-any.whl", hash = "sha256:77179e006c476e69bf8e8ce866640091ec42e1beb80b213c3900006ecfba6901", size = 170656, upload-time = "2026-03-18T19:04:59.826Z" },
]
[[package]]
name = "psutil"
-version = "7.2.1"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/73/cb/09e5184fb5fc0358d110fc3ca7f6b1d033800734d34cac10f4136cfac10e/psutil-7.2.1.tar.gz", hash = "sha256:f7583aec590485b43ca601dd9cea0dcd65bd7bb21d30ef4ddbf4ea6b5ed1bdd3", size = 490253, upload-time = "2025-12-29T08:26:00.169Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/77/8e/f0c242053a368c2aa89584ecd1b054a18683f13d6e5a318fc9ec36582c94/psutil-7.2.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:ba9f33bb525b14c3ea563b2fd521a84d2fa214ec59e3e6a2858f78d0844dd60d", size = 129624, upload-time = "2025-12-29T08:26:04.255Z" },
- { url = "https://files.pythonhosted.org/packages/26/97/a58a4968f8990617decee234258a2b4fc7cd9e35668387646c1963e69f26/psutil-7.2.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:81442dac7abfc2f4f4385ea9e12ddf5a796721c0f6133260687fec5c3780fa49", size = 130132, upload-time = "2025-12-29T08:26:06.228Z" },
- { url = "https://files.pythonhosted.org/packages/db/6d/ed44901e830739af5f72a85fa7ec5ff1edea7f81bfbf4875e409007149bd/psutil-7.2.1-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ea46c0d060491051d39f0d2cff4f98d5c72b288289f57a21556cc7d504db37fc", size = 180612, upload-time = "2025-12-29T08:26:08.276Z" },
- { url = "https://files.pythonhosted.org/packages/c7/65/b628f8459bca4efbfae50d4bf3feaab803de9a160b9d5f3bd9295a33f0c2/psutil-7.2.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:35630d5af80d5d0d49cfc4d64c1c13838baf6717a13effb35869a5919b854cdf", size = 183201, upload-time = "2025-12-29T08:26:10.622Z" },
- { url = "https://files.pythonhosted.org/packages/fb/23/851cadc9764edcc18f0effe7d0bf69f727d4cf2442deb4a9f78d4e4f30f2/psutil-7.2.1-cp313-cp313t-win_amd64.whl", hash = "sha256:923f8653416604e356073e6e0bccbe7c09990acef442def2f5640dd0faa9689f", size = 139081, upload-time = "2025-12-29T08:26:12.483Z" },
- { url = "https://files.pythonhosted.org/packages/59/82/d63e8494ec5758029f31c6cb06d7d161175d8281e91d011a4a441c8a43b5/psutil-7.2.1-cp313-cp313t-win_arm64.whl", hash = "sha256:cfbe6b40ca48019a51827f20d830887b3107a74a79b01ceb8cc8de4ccb17b672", size = 134767, upload-time = "2025-12-29T08:26:14.528Z" },
- { url = "https://files.pythonhosted.org/packages/05/c2/5fb764bd61e40e1fe756a44bd4c21827228394c17414ade348e28f83cd79/psutil-7.2.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:494c513ccc53225ae23eec7fe6e1482f1b8a44674241b54561f755a898650679", size = 129716, upload-time = "2025-12-29T08:26:16.017Z" },
- { url = "https://files.pythonhosted.org/packages/c9/d2/935039c20e06f615d9ca6ca0ab756cf8408a19d298ffaa08666bc18dc805/psutil-7.2.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:3fce5f92c22b00cdefd1645aa58ab4877a01679e901555067b1bd77039aa589f", size = 130133, upload-time = "2025-12-29T08:26:18.009Z" },
- { url = "https://files.pythonhosted.org/packages/77/69/19f1eb0e01d24c2b3eacbc2f78d3b5add8a89bf0bb69465bc8d563cc33de/psutil-7.2.1-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:93f3f7b0bb07711b49626e7940d6fe52aa9940ad86e8f7e74842e73189712129", size = 181518, upload-time = "2025-12-29T08:26:20.241Z" },
- { url = "https://files.pythonhosted.org/packages/e1/6d/7e18b1b4fa13ad370787626c95887b027656ad4829c156bb6569d02f3262/psutil-7.2.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d34d2ca888208eea2b5c68186841336a7f5e0b990edec929be909353a202768a", size = 184348, upload-time = "2025-12-29T08:26:22.215Z" },
- { url = "https://files.pythonhosted.org/packages/98/60/1672114392dd879586d60dd97896325df47d9a130ac7401318005aab28ec/psutil-7.2.1-cp314-cp314t-win_amd64.whl", hash = "sha256:2ceae842a78d1603753561132d5ad1b2f8a7979cb0c283f5b52fb4e6e14b1a79", size = 140400, upload-time = "2025-12-29T08:26:23.993Z" },
- { url = "https://files.pythonhosted.org/packages/fb/7b/d0e9d4513c46e46897b46bcfc410d51fc65735837ea57a25170f298326e6/psutil-7.2.1-cp314-cp314t-win_arm64.whl", hash = "sha256:08a2f175e48a898c8eb8eace45ce01777f4785bc744c90aa2cc7f2fa5462a266", size = 135430, upload-time = "2025-12-29T08:26:25.999Z" },
- { url = "https://files.pythonhosted.org/packages/c5/cf/5180eb8c8bdf6a503c6919f1da28328bd1e6b3b1b5b9d5b01ae64f019616/psutil-7.2.1-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:b2e953fcfaedcfbc952b44744f22d16575d3aa78eb4f51ae74165b4e96e55f42", size = 128137, upload-time = "2025-12-29T08:26:27.759Z" },
- { url = "https://files.pythonhosted.org/packages/c5/2c/78e4a789306a92ade5000da4f5de3255202c534acdadc3aac7b5458fadef/psutil-7.2.1-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:05cc68dbb8c174828624062e73078e7e35406f4ca2d0866c272c2410d8ef06d1", size = 128947, upload-time = "2025-12-29T08:26:29.548Z" },
- { url = "https://files.pythonhosted.org/packages/29/f8/40e01c350ad9a2b3cb4e6adbcc8a83b17ee50dd5792102b6142385937db5/psutil-7.2.1-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5e38404ca2bb30ed7267a46c02f06ff842e92da3bb8c5bfdadbd35a5722314d8", size = 154694, upload-time = "2025-12-29T08:26:32.147Z" },
- { url = "https://files.pythonhosted.org/packages/06/e4/b751cdf839c011a9714a783f120e6a86b7494eb70044d7d81a25a5cd295f/psutil-7.2.1-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ab2b98c9fc19f13f59628d94df5cc4cc4844bc572467d113a8b517d634e362c6", size = 156136, upload-time = "2025-12-29T08:26:34.079Z" },
- { url = "https://files.pythonhosted.org/packages/44/ad/bbf6595a8134ee1e94a4487af3f132cef7fce43aef4a93b49912a48c3af7/psutil-7.2.1-cp36-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:f78baafb38436d5a128f837fab2d92c276dfb48af01a240b861ae02b2413ada8", size = 148108, upload-time = "2025-12-29T08:26:36.225Z" },
- { url = "https://files.pythonhosted.org/packages/1c/15/dd6fd869753ce82ff64dcbc18356093471a5a5adf4f77ed1f805d473d859/psutil-7.2.1-cp36-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:99a4cd17a5fdd1f3d014396502daa70b5ec21bf4ffe38393e152f8e449757d67", size = 147402, upload-time = "2025-12-29T08:26:39.21Z" },
- { url = "https://files.pythonhosted.org/packages/34/68/d9317542e3f2b180c4306e3f45d3c922d7e86d8ce39f941bb9e2e9d8599e/psutil-7.2.1-cp37-abi3-win_amd64.whl", hash = "sha256:b1b0671619343aa71c20ff9767eced0483e4fc9e1f489d50923738caf6a03c17", size = 136938, upload-time = "2025-12-29T08:26:41.036Z" },
- { url = "https://files.pythonhosted.org/packages/3e/73/2ce007f4198c80fcf2cb24c169884f833fe93fbc03d55d302627b094ee91/psutil-7.2.1-cp37-abi3-win_arm64.whl", hash = "sha256:0d67c1822c355aa6f7314d92018fb4268a76668a536f133599b91edd48759442", size = 133836, upload-time = "2025-12-29T08:26:43.086Z" },
+version = "7.2.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/aa/c6/d1ddf4abb55e93cebc4f2ed8b5d6dbad109ecb8d63748dd2b20ab5e57ebe/psutil-7.2.2.tar.gz", hash = "sha256:0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372", size = 493740, upload-time = "2026-01-28T18:14:54.428Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/51/08/510cbdb69c25a96f4ae523f733cdc963ae654904e8db864c07585ef99875/psutil-7.2.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:2edccc433cbfa046b980b0df0171cd25bcaeb3a68fe9022db0979e7aa74a826b", size = 130595, upload-time = "2026-01-28T18:14:57.293Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/f5/97baea3fe7a5a9af7436301f85490905379b1c6f2dd51fe3ecf24b4c5fbf/psutil-7.2.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e78c8603dcd9a04c7364f1a3e670cea95d51ee865e4efb3556a3a63adef958ea", size = 131082, upload-time = "2026-01-28T18:14:59.732Z" },
+ { url = "https://files.pythonhosted.org/packages/37/d6/246513fbf9fa174af531f28412297dd05241d97a75911ac8febefa1a53c6/psutil-7.2.2-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1a571f2330c966c62aeda00dd24620425d4b0cc86881c89861fbc04549e5dc63", size = 181476, upload-time = "2026-01-28T18:15:01.884Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/b5/9182c9af3836cca61696dabe4fd1304e17bc56cb62f17439e1154f225dd3/psutil-7.2.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:917e891983ca3c1887b4ef36447b1e0873e70c933afc831c6b6da078ba474312", size = 184062, upload-time = "2026-01-28T18:15:04.436Z" },
+ { url = "https://files.pythonhosted.org/packages/16/ba/0756dca669f5a9300d0cbcbfae9a4c30e446dfc7440ffe43ded5724bfd93/psutil-7.2.2-cp313-cp313t-win_amd64.whl", hash = "sha256:ab486563df44c17f5173621c7b198955bd6b613fb87c71c161f827d3fb149a9b", size = 139893, upload-time = "2026-01-28T18:15:06.378Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/61/8fa0e26f33623b49949346de05ec1ddaad02ed8ba64af45f40a147dbfa97/psutil-7.2.2-cp313-cp313t-win_arm64.whl", hash = "sha256:ae0aefdd8796a7737eccea863f80f81e468a1e4cf14d926bd9b6f5f2d5f90ca9", size = 135589, upload-time = "2026-01-28T18:15:08.03Z" },
+ { url = "https://files.pythonhosted.org/packages/81/69/ef179ab5ca24f32acc1dac0c247fd6a13b501fd5534dbae0e05a1c48b66d/psutil-7.2.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:eed63d3b4d62449571547b60578c5b2c4bcccc5387148db46e0c2313dad0ee00", size = 130664, upload-time = "2026-01-28T18:15:09.469Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/64/665248b557a236d3fa9efc378d60d95ef56dd0a490c2cd37dafc7660d4a9/psutil-7.2.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7b6d09433a10592ce39b13d7be5a54fbac1d1228ed29abc880fb23df7cb694c9", size = 131087, upload-time = "2026-01-28T18:15:11.724Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/2e/e6782744700d6759ebce3043dcfa661fb61e2fb752b91cdeae9af12c2178/psutil-7.2.2-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1fa4ecf83bcdf6e6c8f4449aff98eefb5d0604bf88cb883d7da3d8d2d909546a", size = 182383, upload-time = "2026-01-28T18:15:13.445Z" },
+ { url = "https://files.pythonhosted.org/packages/57/49/0a41cefd10cb7505cdc04dab3eacf24c0c2cb158a998b8c7b1d27ee2c1f5/psutil-7.2.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e452c464a02e7dc7822a05d25db4cde564444a67e58539a00f929c51eddda0cf", size = 185210, upload-time = "2026-01-28T18:15:16.002Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/2c/ff9bfb544f283ba5f83ba725a3c5fec6d6b10b8f27ac1dc641c473dc390d/psutil-7.2.2-cp314-cp314t-win_amd64.whl", hash = "sha256:c7663d4e37f13e884d13994247449e9f8f574bc4655d509c3b95e9ec9e2b9dc1", size = 141228, upload-time = "2026-01-28T18:15:18.385Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/fc/f8d9c31db14fcec13748d373e668bc3bed94d9077dbc17fb0eebc073233c/psutil-7.2.2-cp314-cp314t-win_arm64.whl", hash = "sha256:11fe5a4f613759764e79c65cf11ebdf26e33d6dd34336f8a337aa2996d71c841", size = 136284, upload-time = "2026-01-28T18:15:19.912Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/36/5ee6e05c9bd427237b11b3937ad82bb8ad2752d72c6969314590dd0c2f6e/psutil-7.2.2-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:ed0cace939114f62738d808fdcecd4c869222507e266e574799e9c0faa17d486", size = 129090, upload-time = "2026-01-28T18:15:22.168Z" },
+ { url = "https://files.pythonhosted.org/packages/80/c4/f5af4c1ca8c1eeb2e92ccca14ce8effdeec651d5ab6053c589b074eda6e1/psutil-7.2.2-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:1a7b04c10f32cc88ab39cbf606e117fd74721c831c98a27dc04578deb0c16979", size = 129859, upload-time = "2026-01-28T18:15:23.795Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/70/5d8df3b09e25bce090399cf48e452d25c935ab72dad19406c77f4e828045/psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9", size = 155560, upload-time = "2026-01-28T18:15:25.976Z" },
+ { url = "https://files.pythonhosted.org/packages/63/65/37648c0c158dc222aba51c089eb3bdfa238e621674dc42d48706e639204f/psutil-7.2.2-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b0726cecd84f9474419d67252add4ac0cd9811b04d61123054b9fb6f57df6e9e", size = 156997, upload-time = "2026-01-28T18:15:27.794Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/13/125093eadae863ce03c6ffdbae9929430d116a246ef69866dad94da3bfbc/psutil-7.2.2-cp36-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd04ef36b4a6d599bbdb225dd1d3f51e00105f6d48a28f006da7f9822f2606d8", size = 148972, upload-time = "2026-01-28T18:15:29.342Z" },
+ { url = "https://files.pythonhosted.org/packages/04/78/0acd37ca84ce3ddffaa92ef0f571e073faa6d8ff1f0559ab1272188ea2be/psutil-7.2.2-cp36-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b58fabe35e80b264a4e3bb23e6b96f9e45a3df7fb7eed419ac0e5947c61e47cc", size = 148266, upload-time = "2026-01-28T18:15:31.597Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/90/e2159492b5426be0c1fef7acba807a03511f97c5f86b3caeda6ad92351a7/psutil-7.2.2-cp37-abi3-win_amd64.whl", hash = "sha256:eb7e81434c8d223ec4a219b5fc1c47d0417b12be7ea866e24fb5ad6e84b3d988", size = 137737, upload-time = "2026-01-28T18:15:33.849Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/c7/7bb2e321574b10df20cbde462a94e2b71d05f9bbda251ef27d104668306a/psutil-7.2.2-cp37-abi3-win_arm64.whl", hash = "sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee", size = 134617, upload-time = "2026-01-28T18:15:36.514Z" },
]
[[package]]
@@ -3791,70 +4795,92 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/99/6c/64cafaceea3f99927e84b38a362ec6a8f24f33061c90bda77dfe1cd4c3c6/pulp-3.3.0-py3-none-any.whl", hash = "sha256:dd6ad2d63f196d1254eddf9dcff5cd224912c1f046120cb7c143c5b0eda63fae", size = 16387700, upload-time = "2025-09-18T08:14:53.368Z" },
]
+[[package]]
+name = "py-spy"
+version = "0.4.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/19/e2/ff811a367028b87e86714945bb9ecb5c1cc69114a8039a67b3a862cef921/py_spy-0.4.1.tar.gz", hash = "sha256:e53aa53daa2e47c2eef97dd2455b47bb3a7e7f962796a86cc3e7dbde8e6f4db4", size = 244726, upload-time = "2025-07-31T19:33:25.172Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/14/e3/3a32500d845bdd94f6a2b4ed6244982f42ec2bc64602ea8fcfe900678ae7/py_spy-0.4.1-py2.py3-none-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:809094208c6256c8f4ccadd31e9a513fe2429253f48e20066879239ba12cd8cc", size = 3682508, upload-time = "2025-07-31T19:33:13.753Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/bf/e4d280e9e0bec71d39fc646654097027d4bbe8e04af18fb68e49afcff404/py_spy-0.4.1-py2.py3-none-macosx_11_0_arm64.whl", hash = "sha256:1fb8bf71ab8df95a95cc387deed6552934c50feef2cf6456bc06692a5508fd0c", size = 1796395, upload-time = "2025-07-31T19:33:15.325Z" },
+ { url = "https://files.pythonhosted.org/packages/df/79/9ed50bb0a9de63ed023aa2db8b6265b04a7760d98c61eb54def6a5fddb68/py_spy-0.4.1-py2.py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ee776b9d512a011d1ad3907ed53ae32ce2f3d9ff3e1782236554e22103b5c084", size = 2034938, upload-time = "2025-07-31T19:33:17.194Z" },
+ { url = "https://files.pythonhosted.org/packages/53/a5/36862e3eea59f729dfb70ee6f9e14b051d8ddce1aa7e70e0b81d9fe18536/py_spy-0.4.1-py2.py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:532d3525538254d1859b49de1fbe9744df6b8865657c9f0e444bf36ce3f19226", size = 2658968, upload-time = "2025-07-31T19:33:18.916Z" },
+ { url = "https://files.pythonhosted.org/packages/08/f8/9ea0b586b065a623f591e5e7961282ec944b5fbbdca33186c7c0296645b3/py_spy-0.4.1-py2.py3-none-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:4972c21890b6814017e39ac233c22572c4a61fd874524ebc5ccab0f2237aee0a", size = 2147541, upload-time = "2025-07-31T19:33:20.565Z" },
+ { url = "https://files.pythonhosted.org/packages/68/fb/bc7f639aed026bca6e7beb1e33f6951e16b7d315594e7635a4f7d21d63f4/py_spy-0.4.1-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:6a80ec05eb8a6883863a367c6a4d4f2d57de68466f7956b6367d4edd5c61bb29", size = 2763338, upload-time = "2025-07-31T19:33:22.202Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/da/fcc9a9fcd4ca946ff402cff20348e838b051d69f50f5d1f5dca4cd3c5eb8/py_spy-0.4.1-py2.py3-none-win_amd64.whl", hash = "sha256:d92e522bd40e9bf7d87c204033ce5bb5c828fca45fa28d970f58d71128069fdc", size = 1818784, upload-time = "2025-07-31T19:33:23.802Z" },
+]
+
[[package]]
name = "pyarrow"
-version = "22.0.0"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/30/53/04a7fdc63e6056116c9ddc8b43bc28c12cdd181b85cbeadb79278475f3ae/pyarrow-22.0.0.tar.gz", hash = "sha256:3d600dc583260d845c7d8a6db540339dd883081925da2bd1c5cb808f720b3cd9", size = 1151151, upload-time = "2025-10-24T12:30:00.762Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/d9/9b/cb3f7e0a345353def531ca879053e9ef6b9f38ed91aebcf68b09ba54dec0/pyarrow-22.0.0-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:77718810bd3066158db1e95a63c160ad7ce08c6b0710bc656055033e39cdad88", size = 34223968, upload-time = "2025-10-24T10:03:31.21Z" },
- { url = "https://files.pythonhosted.org/packages/6c/41/3184b8192a120306270c5307f105b70320fdaa592c99843c5ef78aaefdcf/pyarrow-22.0.0-cp310-cp310-macosx_12_0_x86_64.whl", hash = "sha256:44d2d26cda26d18f7af7db71453b7b783788322d756e81730acb98f24eb90ace", size = 35942085, upload-time = "2025-10-24T10:03:38.146Z" },
- { url = "https://files.pythonhosted.org/packages/d9/3d/a1eab2f6f08001f9fb714b8ed5cfb045e2fe3e3e3c0c221f2c9ed1e6d67d/pyarrow-22.0.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:b9d71701ce97c95480fecb0039ec5bb889e75f110da72005743451339262f4ce", size = 44964613, upload-time = "2025-10-24T10:03:46.516Z" },
- { url = "https://files.pythonhosted.org/packages/46/46/a1d9c24baf21cfd9ce994ac820a24608decf2710521b29223d4334985127/pyarrow-22.0.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:710624ab925dc2b05a6229d47f6f0dac1c1155e6ed559be7109f684eba048a48", size = 47627059, upload-time = "2025-10-24T10:03:55.353Z" },
- { url = "https://files.pythonhosted.org/packages/3a/4c/f711acb13075c1391fd54bc17e078587672c575f8de2a6e62509af026dcf/pyarrow-22.0.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:f963ba8c3b0199f9d6b794c90ec77545e05eadc83973897a4523c9e8d84e9340", size = 47947043, upload-time = "2025-10-24T10:04:05.408Z" },
- { url = "https://files.pythonhosted.org/packages/4e/70/1f3180dd7c2eab35c2aca2b29ace6c519f827dcd4cfeb8e0dca41612cf7a/pyarrow-22.0.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:bd0d42297ace400d8febe55f13fdf46e86754842b860c978dfec16f081e5c653", size = 50206505, upload-time = "2025-10-24T10:04:15.786Z" },
- { url = "https://files.pythonhosted.org/packages/80/07/fea6578112c8c60ffde55883a571e4c4c6bc7049f119d6b09333b5cc6f73/pyarrow-22.0.0-cp310-cp310-win_amd64.whl", hash = "sha256:00626d9dc0f5ef3a75fe63fd68b9c7c8302d2b5bbc7f74ecaedba83447a24f84", size = 28101641, upload-time = "2025-10-24T10:04:22.57Z" },
- { url = "https://files.pythonhosted.org/packages/2e/b7/18f611a8cdc43417f9394a3ccd3eace2f32183c08b9eddc3d17681819f37/pyarrow-22.0.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:3e294c5eadfb93d78b0763e859a0c16d4051fc1c5231ae8956d61cb0b5666f5a", size = 34272022, upload-time = "2025-10-24T10:04:28.973Z" },
- { url = "https://files.pythonhosted.org/packages/26/5c/f259e2526c67eb4b9e511741b19870a02363a47a35edbebc55c3178db22d/pyarrow-22.0.0-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:69763ab2445f632d90b504a815a2a033f74332997052b721002298ed6de40f2e", size = 35995834, upload-time = "2025-10-24T10:04:35.467Z" },
- { url = "https://files.pythonhosted.org/packages/50/8d/281f0f9b9376d4b7f146913b26fac0aa2829cd1ee7e997f53a27411bbb92/pyarrow-22.0.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:b41f37cabfe2463232684de44bad753d6be08a7a072f6a83447eeaf0e4d2a215", size = 45030348, upload-time = "2025-10-24T10:04:43.366Z" },
- { url = "https://files.pythonhosted.org/packages/f5/e5/53c0a1c428f0976bf22f513d79c73000926cb00b9c138d8e02daf2102e18/pyarrow-22.0.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:35ad0f0378c9359b3f297299c3309778bb03b8612f987399a0333a560b43862d", size = 47699480, upload-time = "2025-10-24T10:04:51.486Z" },
- { url = "https://files.pythonhosted.org/packages/95/e1/9dbe4c465c3365959d183e6345d0a8d1dc5b02ca3f8db4760b3bc834cf25/pyarrow-22.0.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:8382ad21458075c2e66a82a29d650f963ce51c7708c7c0ff313a8c206c4fd5e8", size = 48011148, upload-time = "2025-10-24T10:04:59.585Z" },
- { url = "https://files.pythonhosted.org/packages/c5/b4/7caf5d21930061444c3cf4fa7535c82faf5263e22ce43af7c2759ceb5b8b/pyarrow-22.0.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:1a812a5b727bc09c3d7ea072c4eebf657c2f7066155506ba31ebf4792f88f016", size = 50276964, upload-time = "2025-10-24T10:05:08.175Z" },
- { url = "https://files.pythonhosted.org/packages/ae/f3/cec89bd99fa3abf826f14d4e53d3d11340ce6f6af4d14bdcd54cd83b6576/pyarrow-22.0.0-cp311-cp311-win_amd64.whl", hash = "sha256:ec5d40dd494882704fb876c16fa7261a69791e784ae34e6b5992e977bd2e238c", size = 28106517, upload-time = "2025-10-24T10:05:14.314Z" },
- { url = "https://files.pythonhosted.org/packages/af/63/ba23862d69652f85b615ca14ad14f3bcfc5bf1b99ef3f0cd04ff93fdad5a/pyarrow-22.0.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:bea79263d55c24a32b0d79c00a1c58bb2ee5f0757ed95656b01c0fb310c5af3d", size = 34211578, upload-time = "2025-10-24T10:05:21.583Z" },
- { url = "https://files.pythonhosted.org/packages/b1/d0/f9ad86fe809efd2bcc8be32032fa72e8b0d112b01ae56a053006376c5930/pyarrow-22.0.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:12fe549c9b10ac98c91cf791d2945e878875d95508e1a5d14091a7aaa66d9cf8", size = 35989906, upload-time = "2025-10-24T10:05:29.485Z" },
- { url = "https://files.pythonhosted.org/packages/b4/a8/f910afcb14630e64d673f15904ec27dd31f1e009b77033c365c84e8c1e1d/pyarrow-22.0.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:334f900ff08ce0423407af97e6c26ad5d4e3b0763645559ece6fbf3747d6a8f5", size = 45021677, upload-time = "2025-10-24T10:05:38.274Z" },
- { url = "https://files.pythonhosted.org/packages/13/95/aec81f781c75cd10554dc17a25849c720d54feafb6f7847690478dcf5ef8/pyarrow-22.0.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:c6c791b09c57ed76a18b03f2631753a4960eefbbca80f846da8baefc6491fcfe", size = 47726315, upload-time = "2025-10-24T10:05:47.314Z" },
- { url = "https://files.pythonhosted.org/packages/bb/d4/74ac9f7a54cfde12ee42734ea25d5a3c9a45db78f9def949307a92720d37/pyarrow-22.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c3200cb41cdbc65156e5f8c908d739b0dfed57e890329413da2748d1a2cd1a4e", size = 47990906, upload-time = "2025-10-24T10:05:58.254Z" },
- { url = "https://files.pythonhosted.org/packages/2e/71/fedf2499bf7a95062eafc989ace56572f3343432570e1c54e6599d5b88da/pyarrow-22.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:ac93252226cf288753d8b46280f4edf3433bf9508b6977f8dd8526b521a1bbb9", size = 50306783, upload-time = "2025-10-24T10:06:08.08Z" },
- { url = "https://files.pythonhosted.org/packages/68/ed/b202abd5a5b78f519722f3d29063dda03c114711093c1995a33b8e2e0f4b/pyarrow-22.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:44729980b6c50a5f2bfcc2668d36c569ce17f8b17bccaf470c4313dcbbf13c9d", size = 27972883, upload-time = "2025-10-24T10:06:14.204Z" },
- { url = "https://files.pythonhosted.org/packages/a6/d6/d0fac16a2963002fc22c8fa75180a838737203d558f0ed3b564c4a54eef5/pyarrow-22.0.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:e6e95176209257803a8b3d0394f21604e796dadb643d2f7ca21b66c9c0b30c9a", size = 34204629, upload-time = "2025-10-24T10:06:20.274Z" },
- { url = "https://files.pythonhosted.org/packages/c6/9c/1d6357347fbae062ad3f17082f9ebc29cc733321e892c0d2085f42a2212b/pyarrow-22.0.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:001ea83a58024818826a9e3f89bf9310a114f7e26dfe404a4c32686f97bd7901", size = 35985783, upload-time = "2025-10-24T10:06:27.301Z" },
- { url = "https://files.pythonhosted.org/packages/ff/c0/782344c2ce58afbea010150df07e3a2f5fdad299cd631697ae7bd3bac6e3/pyarrow-22.0.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:ce20fe000754f477c8a9125543f1936ea5b8867c5406757c224d745ed033e691", size = 45020999, upload-time = "2025-10-24T10:06:35.387Z" },
- { url = "https://files.pythonhosted.org/packages/1b/8b/5362443737a5307a7b67c1017c42cd104213189b4970bf607e05faf9c525/pyarrow-22.0.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:e0a15757fccb38c410947df156f9749ae4a3c89b2393741a50521f39a8cf202a", size = 47724601, upload-time = "2025-10-24T10:06:43.551Z" },
- { url = "https://files.pythonhosted.org/packages/69/4d/76e567a4fc2e190ee6072967cb4672b7d9249ac59ae65af2d7e3047afa3b/pyarrow-22.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:cedb9dd9358e4ea1d9bce3665ce0797f6adf97ff142c8e25b46ba9cdd508e9b6", size = 48001050, upload-time = "2025-10-24T10:06:52.284Z" },
- { url = "https://files.pythonhosted.org/packages/01/5e/5653f0535d2a1aef8223cee9d92944cb6bccfee5cf1cd3f462d7cb022790/pyarrow-22.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:252be4a05f9d9185bb8c18e83764ebcfea7185076c07a7a662253af3a8c07941", size = 50307877, upload-time = "2025-10-24T10:07:02.405Z" },
- { url = "https://files.pythonhosted.org/packages/2d/f8/1d0bd75bf9328a3b826e24a16e5517cd7f9fbf8d34a3184a4566ef5a7f29/pyarrow-22.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:a4893d31e5ef780b6edcaf63122df0f8d321088bb0dee4c8c06eccb1ca28d145", size = 27977099, upload-time = "2025-10-24T10:08:07.259Z" },
- { url = "https://files.pythonhosted.org/packages/90/81/db56870c997805bf2b0f6eeeb2d68458bf4654652dccdcf1bf7a42d80903/pyarrow-22.0.0-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:f7fe3dbe871294ba70d789be16b6e7e52b418311e166e0e3cba9522f0f437fb1", size = 34336685, upload-time = "2025-10-24T10:07:11.47Z" },
- { url = "https://files.pythonhosted.org/packages/1c/98/0727947f199aba8a120f47dfc229eeb05df15bcd7a6f1b669e9f882afc58/pyarrow-22.0.0-cp313-cp313t-macosx_12_0_x86_64.whl", hash = "sha256:ba95112d15fd4f1105fb2402c4eab9068f0554435e9b7085924bcfaac2cc306f", size = 36032158, upload-time = "2025-10-24T10:07:18.626Z" },
- { url = "https://files.pythonhosted.org/packages/96/b4/9babdef9c01720a0785945c7cf550e4acd0ebcd7bdd2e6f0aa7981fa85e2/pyarrow-22.0.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:c064e28361c05d72eed8e744c9605cbd6d2bb7481a511c74071fd9b24bc65d7d", size = 44892060, upload-time = "2025-10-24T10:07:26.002Z" },
- { url = "https://files.pythonhosted.org/packages/f8/ca/2f8804edd6279f78a37062d813de3f16f29183874447ef6d1aadbb4efa0f/pyarrow-22.0.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:6f9762274496c244d951c819348afbcf212714902742225f649cf02823a6a10f", size = 47504395, upload-time = "2025-10-24T10:07:34.09Z" },
- { url = "https://files.pythonhosted.org/packages/b9/f0/77aa5198fd3943682b2e4faaf179a674f0edea0d55d326d83cb2277d9363/pyarrow-22.0.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:a9d9ffdc2ab696f6b15b4d1f7cec6658e1d788124418cb30030afbae31c64746", size = 48066216, upload-time = "2025-10-24T10:07:43.528Z" },
- { url = "https://files.pythonhosted.org/packages/79/87/a1937b6e78b2aff18b706d738c9e46ade5bfcf11b294e39c87706a0089ac/pyarrow-22.0.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:ec1a15968a9d80da01e1d30349b2b0d7cc91e96588ee324ce1b5228175043e95", size = 50288552, upload-time = "2025-10-24T10:07:53.519Z" },
- { url = "https://files.pythonhosted.org/packages/60/ae/b5a5811e11f25788ccfdaa8f26b6791c9807119dffcf80514505527c384c/pyarrow-22.0.0-cp313-cp313t-win_amd64.whl", hash = "sha256:bba208d9c7decf9961998edf5c65e3ea4355d5818dd6cd0f6809bec1afb951cc", size = 28262504, upload-time = "2025-10-24T10:08:00.932Z" },
- { url = "https://files.pythonhosted.org/packages/bd/b0/0fa4d28a8edb42b0a7144edd20befd04173ac79819547216f8a9f36f9e50/pyarrow-22.0.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:9bddc2cade6561f6820d4cd73f99a0243532ad506bc510a75a5a65a522b2d74d", size = 34224062, upload-time = "2025-10-24T10:08:14.101Z" },
- { url = "https://files.pythonhosted.org/packages/0f/a8/7a719076b3c1be0acef56a07220c586f25cd24de0e3f3102b438d18ae5df/pyarrow-22.0.0-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:e70ff90c64419709d38c8932ea9fe1cc98415c4f87ea8da81719e43f02534bc9", size = 35990057, upload-time = "2025-10-24T10:08:21.842Z" },
- { url = "https://files.pythonhosted.org/packages/89/3c/359ed54c93b47fb6fe30ed16cdf50e3f0e8b9ccfb11b86218c3619ae50a8/pyarrow-22.0.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:92843c305330aa94a36e706c16209cd4df274693e777ca47112617db7d0ef3d7", size = 45068002, upload-time = "2025-10-24T10:08:29.034Z" },
- { url = "https://files.pythonhosted.org/packages/55/fc/4945896cc8638536ee787a3bd6ce7cec8ec9acf452d78ec39ab328efa0a1/pyarrow-22.0.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:6dda1ddac033d27421c20d7a7943eec60be44e0db4e079f33cc5af3b8280ccde", size = 47737765, upload-time = "2025-10-24T10:08:38.559Z" },
- { url = "https://files.pythonhosted.org/packages/cd/5e/7cb7edeb2abfaa1f79b5d5eb89432356155c8426f75d3753cbcb9592c0fd/pyarrow-22.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:84378110dd9a6c06323b41b56e129c504d157d1a983ce8f5443761eb5256bafc", size = 48048139, upload-time = "2025-10-24T10:08:46.784Z" },
- { url = "https://files.pythonhosted.org/packages/88/c6/546baa7c48185f5e9d6e59277c4b19f30f48c94d9dd938c2a80d4d6b067c/pyarrow-22.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:854794239111d2b88b40b6ef92aa478024d1e5074f364033e73e21e3f76b25e0", size = 50314244, upload-time = "2025-10-24T10:08:55.771Z" },
- { url = "https://files.pythonhosted.org/packages/3c/79/755ff2d145aafec8d347bf18f95e4e81c00127f06d080135dfc86aea417c/pyarrow-22.0.0-cp314-cp314-win_amd64.whl", hash = "sha256:b883fe6fd85adad7932b3271c38ac289c65b7337c2c132e9569f9d3940620730", size = 28757501, upload-time = "2025-10-24T10:09:59.891Z" },
- { url = "https://files.pythonhosted.org/packages/0e/d2/237d75ac28ced3147912954e3c1a174df43a95f4f88e467809118a8165e0/pyarrow-22.0.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:7a820d8ae11facf32585507c11f04e3f38343c1e784c9b5a8b1da5c930547fe2", size = 34355506, upload-time = "2025-10-24T10:09:02.953Z" },
- { url = "https://files.pythonhosted.org/packages/1e/2c/733dfffe6d3069740f98e57ff81007809067d68626c5faef293434d11bd6/pyarrow-22.0.0-cp314-cp314t-macosx_12_0_x86_64.whl", hash = "sha256:c6ec3675d98915bf1ec8b3c7986422682f7232ea76cad276f4c8abd5b7319b70", size = 36047312, upload-time = "2025-10-24T10:09:10.334Z" },
- { url = "https://files.pythonhosted.org/packages/7c/2b/29d6e3782dc1f299727462c1543af357a0f2c1d3c160ce199950d9ca51eb/pyarrow-22.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:3e739edd001b04f654b166204fc7a9de896cf6007eaff33409ee9e50ceaff754", size = 45081609, upload-time = "2025-10-24T10:09:18.61Z" },
- { url = "https://files.pythonhosted.org/packages/8d/42/aa9355ecc05997915af1b7b947a7f66c02dcaa927f3203b87871c114ba10/pyarrow-22.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:7388ac685cab5b279a41dfe0a6ccd99e4dbf322edfb63e02fc0443bf24134e91", size = 47703663, upload-time = "2025-10-24T10:09:27.369Z" },
- { url = "https://files.pythonhosted.org/packages/ee/62/45abedde480168e83a1de005b7b7043fd553321c1e8c5a9a114425f64842/pyarrow-22.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:f633074f36dbc33d5c05b5dc75371e5660f1dbf9c8b1d95669def05e5425989c", size = 48066543, upload-time = "2025-10-24T10:09:34.908Z" },
- { url = "https://files.pythonhosted.org/packages/84/e9/7878940a5b072e4f3bf998770acafeae13b267f9893af5f6d4ab3904b67e/pyarrow-22.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:4c19236ae2402a8663a2c8f21f1870a03cc57f0bef7e4b6eb3238cc82944de80", size = 50288838, upload-time = "2025-10-24T10:09:44.394Z" },
- { url = "https://files.pythonhosted.org/packages/7b/03/f335d6c52b4a4761bcc83499789a1e2e16d9d201a58c327a9b5cc9a41bd9/pyarrow-22.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:0c34fe18094686194f204a3b1787a27456897d8a2d62caf84b61e8dfbc0252ae", size = 29185594, upload-time = "2025-10-24T10:09:53.111Z" },
+version = "23.0.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/88/22/134986a4cc224d593c1afde5494d18ff629393d74cc2eddb176669f234a4/pyarrow-23.0.1.tar.gz", hash = "sha256:b8c5873e33440b2bc2f4a79d2b47017a89c5a24116c055625e6f2ee50523f019", size = 1167336, upload-time = "2026-02-16T10:14:12.39Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9a/4b/4166bb5abbfe6f750fc60ad337c43ecf61340fa52ab386da6e8dbf9e63c4/pyarrow-23.0.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:f4b0dbfa124c0bb161f8b5ebb40f1a680b70279aa0c9901d44a2b5a20806039f", size = 34214575, upload-time = "2026-02-16T10:09:56.225Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/da/3f941e3734ac8088ea588b53e860baeddac8323ea40ce22e3d0baa865cc9/pyarrow-23.0.1-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:7707d2b6673f7de054e2e83d59f9e805939038eebe1763fe811ee8fa5c0cd1a7", size = 35832540, upload-time = "2026-02-16T10:10:03.428Z" },
+ { url = "https://files.pythonhosted.org/packages/88/7c/3d841c366620e906d54430817531b877ba646310296df42ef697308c2705/pyarrow-23.0.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:86ff03fb9f1a320266e0de855dee4b17da6794c595d207f89bba40d16b5c78b9", size = 44470940, upload-time = "2026-02-16T10:10:10.704Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/a5/da83046273d990f256cb79796a190bbf7ec999269705ddc609403f8c6b06/pyarrow-23.0.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:813d99f31275919c383aab17f0f455a04f5a429c261cc411b1e9a8f5e4aaaa05", size = 47586063, upload-time = "2026-02-16T10:10:17.95Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/3c/b7d2ebcff47a514f47f9da1e74b7949138c58cfeb108cdd4ee62f43f0cf3/pyarrow-23.0.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:bf5842f960cddd2ef757d486041d57c96483efc295a8c4a0e20e704cbbf39c67", size = 48173045, upload-time = "2026-02-16T10:10:25.363Z" },
+ { url = "https://files.pythonhosted.org/packages/43/b2/b40961262213beaba6acfc88698eb773dfce32ecdf34d19291db94c2bd73/pyarrow-23.0.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:564baf97c858ecc03ec01a41062e8f4698abc3e6e2acd79c01c2e97880a19730", size = 50621741, upload-time = "2026-02-16T10:10:33.477Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/70/1fdda42d65b28b078e93d75d371b2185a61da89dda4def8ba6ba41ebdeb4/pyarrow-23.0.1-cp312-cp312-win_amd64.whl", hash = "sha256:07deae7783782ac7250989a7b2ecde9b3c343a643f82e8a4df03d93b633006f0", size = 27620678, upload-time = "2026-02-16T10:10:39.31Z" },
+ { url = "https://files.pythonhosted.org/packages/47/10/2cbe4c6f0fb83d2de37249567373d64327a5e4d8db72f486db42875b08f6/pyarrow-23.0.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:6b8fda694640b00e8af3c824f99f789e836720aa8c9379fb435d4c4953a756b8", size = 34210066, upload-time = "2026-02-16T10:10:45.487Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/4f/679fa7e84dadbaca7a65f7cdba8d6c83febbd93ca12fa4adf40ba3b6362b/pyarrow-23.0.1-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:8ff51b1addc469b9444b7c6f3548e19dc931b172ab234e995a60aea9f6e6025f", size = 35825526, upload-time = "2026-02-16T10:10:52.266Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/63/d2747d930882c9d661e9398eefc54f15696547b8983aaaf11d4a2e8b5426/pyarrow-23.0.1-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:71c5be5cbf1e1cb6169d2a0980850bccb558ddc9b747b6206435313c47c37677", size = 44473279, upload-time = "2026-02-16T10:11:01.557Z" },
+ { url = "https://files.pythonhosted.org/packages/b3/93/10a48b5e238de6d562a411af6467e71e7aedbc9b87f8d3a35f1560ae30fb/pyarrow-23.0.1-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:9b6f4f17b43bc39d56fec96e53fe89d94bac3eb134137964371b45352d40d0c2", size = 47585798, upload-time = "2026-02-16T10:11:09.401Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/20/476943001c54ef078dbf9542280e22741219a184a0632862bca4feccd666/pyarrow-23.0.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:9fc13fc6c403d1337acab46a2c4346ca6c9dec5780c3c697cf8abfd5e19b6b37", size = 48179446, upload-time = "2026-02-16T10:11:17.781Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/b6/5dd0c47b335fcd8edba9bfab78ad961bd0fd55ebe53468cc393f45e0be60/pyarrow-23.0.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:5c16ed4f53247fa3ffb12a14d236de4213a4415d127fe9cebed33d51671113e2", size = 50623972, upload-time = "2026-02-16T10:11:26.185Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/09/a532297c9591a727d67760e2e756b83905dd89adb365a7f6e9c72578bcc1/pyarrow-23.0.1-cp313-cp313-win_amd64.whl", hash = "sha256:cecfb12ef629cf6be0b1887f9f86463b0dd3dc3195ae6224e74006be4736035a", size = 27540749, upload-time = "2026-02-16T10:12:23.297Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/8e/38749c4b1303e6ae76b3c80618f84861ae0c55dd3c2273842ea6f8258233/pyarrow-23.0.1-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:29f7f7419a0e30264ea261fdc0e5fe63ce5a6095003db2945d7cd78df391a7e1", size = 34471544, upload-time = "2026-02-16T10:11:32.535Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/73/f237b2bc8c669212f842bcfd842b04fc8d936bfc9d471630569132dc920d/pyarrow-23.0.1-cp313-cp313t-macosx_12_0_x86_64.whl", hash = "sha256:33d648dc25b51fd8055c19e4261e813dfc4d2427f068bcecc8b53d01b81b0500", size = 35949911, upload-time = "2026-02-16T10:11:39.813Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/86/b912195eee0903b5611bf596833def7d146ab2d301afeb4b722c57ffc966/pyarrow-23.0.1-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:cd395abf8f91c673dd3589cadc8cc1ee4e8674fa61b2e923c8dd215d9c7d1f41", size = 44520337, upload-time = "2026-02-16T10:11:47.764Z" },
+ { url = "https://files.pythonhosted.org/packages/69/c2/f2a717fb824f62d0be952ea724b4f6f9372a17eed6f704b5c9526f12f2f1/pyarrow-23.0.1-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:00be9576d970c31defb5c32eb72ef585bf600ef6d0a82d5eccaae96639cf9d07", size = 47548944, upload-time = "2026-02-16T10:11:56.607Z" },
+ { url = "https://files.pythonhosted.org/packages/84/a7/90007d476b9f0dc308e3bc57b832d004f848fd6c0da601375d20d92d1519/pyarrow-23.0.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:c2139549494445609f35a5cda4eb94e2c9e4d704ce60a095b342f82460c73a83", size = 48236269, upload-time = "2026-02-16T10:12:04.47Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/3f/b16fab3e77709856eb6ac328ce35f57a6d4a18462c7ca5186ef31b45e0e0/pyarrow-23.0.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:7044b442f184d84e2351e5084600f0d7343d6117aabcbc1ac78eb1ae11eb4125", size = 50604794, upload-time = "2026-02-16T10:12:11.797Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/a1/22df0620a9fac31d68397a75465c344e83c3dfe521f7612aea33e27ab6c0/pyarrow-23.0.1-cp313-cp313t-win_amd64.whl", hash = "sha256:a35581e856a2fafa12f3f54fce4331862b1cfb0bef5758347a858a4aa9d6bae8", size = 27660642, upload-time = "2026-02-16T10:12:17.746Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/1b/6da9a89583ce7b23ac611f183ae4843cd3a6cf54f079549b0e8c14031e73/pyarrow-23.0.1-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:5df1161da23636a70838099d4aaa65142777185cc0cdba4037a18cee7d8db9ca", size = 34238755, upload-time = "2026-02-16T10:12:32.819Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/b5/d58a241fbe324dbaeb8df07be6af8752c846192d78d2272e551098f74e88/pyarrow-23.0.1-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:fa8e51cb04b9f8c9c5ace6bab63af9a1f88d35c0d6cbf53e8c17c098552285e1", size = 35847826, upload-time = "2026-02-16T10:12:38.949Z" },
+ { url = "https://files.pythonhosted.org/packages/54/a5/8cbc83f04aba433ca7b331b38f39e000efd9f0c7ce47128670e737542996/pyarrow-23.0.1-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:0b95a3994f015be13c63148fef8832e8a23938128c185ee951c98908a696e0eb", size = 44536859, upload-time = "2026-02-16T10:12:45.467Z" },
+ { url = "https://files.pythonhosted.org/packages/36/2e/c0f017c405fcdc252dbccafbe05e36b0d0eb1ea9a958f081e01c6972927f/pyarrow-23.0.1-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:4982d71350b1a6e5cfe1af742c53dfb759b11ce14141870d05d9e540d13bc5d1", size = 47614443, upload-time = "2026-02-16T10:12:55.525Z" },
+ { url = "https://files.pythonhosted.org/packages/af/6b/2314a78057912f5627afa13ba43809d9d653e6630859618b0fd81a4e0759/pyarrow-23.0.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c250248f1fe266db627921c89b47b7c06fee0489ad95b04d50353537d74d6886", size = 48232991, upload-time = "2026-02-16T10:13:04.729Z" },
+ { url = "https://files.pythonhosted.org/packages/40/f2/1bcb1d3be3460832ef3370d621142216e15a2c7c62602a4ea19ec240dd64/pyarrow-23.0.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5f4763b83c11c16e5f4c15601ba6dfa849e20723b46aa2617cb4bffe8768479f", size = 50645077, upload-time = "2026-02-16T10:13:14.147Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/3f/b1da7b61cd66566a4d4c8383d376c606d1c34a906c3f1cb35c479f59d1aa/pyarrow-23.0.1-cp314-cp314-win_amd64.whl", hash = "sha256:3a4c85ef66c134161987c17b147d6bffdca4566f9a4c1d81a0a01cdf08414ea5", size = 28234271, upload-time = "2026-02-16T10:14:09.397Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/78/07f67434e910a0f7323269be7bfbf58699bd0c1d080b18a1ab49ba943fe8/pyarrow-23.0.1-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:17cd28e906c18af486a499422740298c52d7c6795344ea5002a7720b4eadf16d", size = 34488692, upload-time = "2026-02-16T10:13:21.541Z" },
+ { url = "https://files.pythonhosted.org/packages/50/76/34cf7ae93ece1f740a04910d9f7e80ba166b9b4ab9596a953e9e62b90fe1/pyarrow-23.0.1-cp314-cp314t-macosx_12_0_x86_64.whl", hash = "sha256:76e823d0e86b4fb5e1cf4a58d293036e678b5a4b03539be933d3b31f9406859f", size = 35964383, upload-time = "2026-02-16T10:13:28.63Z" },
+ { url = "https://files.pythonhosted.org/packages/46/90/459b827238936d4244214be7c684e1b366a63f8c78c380807ae25ed92199/pyarrow-23.0.1-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:a62e1899e3078bf65943078b3ad2a6ddcacf2373bc06379aac61b1e548a75814", size = 44538119, upload-time = "2026-02-16T10:13:35.506Z" },
+ { url = "https://files.pythonhosted.org/packages/28/a1/93a71ae5881e99d1f9de1d4554a87be37da11cd6b152239fb5bd924fdc64/pyarrow-23.0.1-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:df088e8f640c9fae3b1f495b3c64755c4e719091caf250f3a74d095ddf3c836d", size = 47571199, upload-time = "2026-02-16T10:13:42.504Z" },
+ { url = "https://files.pythonhosted.org/packages/88/a3/d2c462d4ef313521eaf2eff04d204ac60775263f1fb08c374b543f79f610/pyarrow-23.0.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:46718a220d64677c93bc243af1d44b55998255427588e400677d7192671845c7", size = 48259435, upload-time = "2026-02-16T10:13:49.226Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/f1/11a544b8c3d38a759eb3fbb022039117fd633e9a7b19e4841cc3da091915/pyarrow-23.0.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:a09f3876e87f48bc2f13583ab551f0379e5dfb83210391e68ace404181a20690", size = 50629149, upload-time = "2026-02-16T10:13:57.238Z" },
+ { url = "https://files.pythonhosted.org/packages/50/f2/c0e76a0b451ffdf0cf788932e182758eb7558953f4f27f1aff8e2518b653/pyarrow-23.0.1-cp314-cp314t-win_amd64.whl", hash = "sha256:527e8d899f14bd15b740cd5a54ad56b7f98044955373a17179d5956ddb93d9ce", size = 28365807, upload-time = "2026-02-16T10:14:03.892Z" },
+]
+
+[[package]]
+name = "pyasn1"
+version = "0.6.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/5c/5f/6583902b6f79b399c9c40674ac384fd9cd77805f9e6205075f828ef11fb2/pyasn1-0.6.3.tar.gz", hash = "sha256:697a8ecd6d98891189184ca1fa05d1bb00e2f84b5977c481452050549c8a72cf", size = 148685, upload-time = "2026-03-17T01:06:53.382Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5d/a0/7d793dce3fa811fe047d6ae2431c672364b462850c6235ae306c0efd025f/pyasn1-0.6.3-py3-none-any.whl", hash = "sha256:a80184d120f0864a52a073acc6fc642847d0be408e7c7252f31390c0f4eadcde", size = 83997, upload-time = "2026-03-17T01:06:52.036Z" },
+]
+
+[[package]]
+name = "pyasn1-modules"
+version = "0.4.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pyasn1" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e9/e6/78ebbb10a8c8e4b61a59249394a4a594c1a7af95593dc933a349c8d00964/pyasn1_modules-0.4.2.tar.gz", hash = "sha256:677091de870a80aae844b1ca6134f54652fa2c8c5a52aa396440ac3106e941e6", size = 307892, upload-time = "2025-03-28T02:41:22.17Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/47/8d/d529b5d697919ba8c11ad626e835d4039be708a35b0d22de83a269a6682c/pyasn1_modules-0.4.2-py3-none-any.whl", hash = "sha256:29253a9207ce32b64c3ac6600edc75368f98473906e8fd1043bd6b5b1de2c14a", size = 181259, upload-time = "2025-03-28T02:41:19.028Z" },
]
[[package]]
name = "pybind11"
-version = "3.0.1"
+version = "3.0.2"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/2f/7b/a6d8dcb83c457e24a9df1e4d8fd5fb8034d4bbc62f3c324681e8a9ba57c2/pybind11-3.0.1.tar.gz", hash = "sha256:9c0f40056a016da59bab516efb523089139fcc6f2ba7e4930854c61efb932051", size = 546914, upload-time = "2025-08-22T20:09:27.265Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/a5/98/9118a0659646f1628c592ef9bb48e0056efa6bf27c951fd12a178e0136fb/pybind11-3.0.2.tar.gz", hash = "sha256:432f01aeb68e361a3a7fc7575c2c7f497595bf640f747acd909ff238dd766e06", size = 577131, upload-time = "2026-02-17T04:46:52.556Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/cd/8a/37362fc2b949d5f733a8b0f2ff51ba423914cabefe69f1d1b6aab710f5fe/pybind11-3.0.1-py3-none-any.whl", hash = "sha256:aa8f0aa6e0a94d3b64adfc38f560f33f15e589be2175e103c0a33c6bce55ee89", size = 293611, upload-time = "2025-08-22T20:09:25.235Z" },
+ { url = "https://files.pythonhosted.org/packages/88/c5/e98d9c51f3d5300d5e40ad9037dd6b3b60736fd02ab68dcc98c96be7592d/pybind11-3.0.2-py3-none-any.whl", hash = "sha256:f8a6500548919cc33bcd220d5f984688326f574fa97f1107f2f4fdb4c6fb019f", size = 310158, upload-time = "2026-02-17T04:46:49.91Z" },
]
[[package]]
@@ -3868,11 +4894,11 @@ wheels = [
[[package]]
name = "pycparser"
-version = "2.23"
+version = "3.0"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/fe/cf/d2d3b9f5699fb1e4615c8e32ff220203e43b248e1dfcc6736ad9057731ca/pycparser-2.23.tar.gz", hash = "sha256:78816d4f24add8f10a06d6f05b4d424ad9e96cfebf68a4ddc99c65c0720d00c2", size = 173734, upload-time = "2025-09-09T13:23:47.91Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/1b/7d/92392ff7815c21062bea51aa7b87d45576f649f16458d78b7cf94b9ab2e6/pycparser-3.0.tar.gz", hash = "sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29", size = 103492, upload-time = "2026-01-21T14:26:51.89Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/a0/e3/59cd50310fc9b59512193629e1984c1f95e5c8ae6e5d8c69532ccc65a7fe/pycparser-2.23-py3-none-any.whl", hash = "sha256:e5c6e8d3fbad53479cab09ac03729e0a9faf2bee3db8208a550daf5af81a5934", size = 118140, upload-time = "2025-09-09T13:23:46.651Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" },
]
[[package]]
@@ -3899,33 +4925,6 @@ dependencies = [
]
sdist = { url = "https://files.pythonhosted.org/packages/71/70/23b021c950c2addd24ec408e9ab05d59b035b39d97cdc1130e1bce647bb6/pydantic_core-2.41.5.tar.gz", hash = "sha256:08daa51ea16ad373ffd5e7606252cc32f07bc72b28284b6bc9c6df804816476e", size = 460952, upload-time = "2025-11-04T13:43:49.098Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/c6/90/32c9941e728d564b411d574d8ee0cf09b12ec978cb22b294995bae5549a5/pydantic_core-2.41.5-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:77b63866ca88d804225eaa4af3e664c5faf3568cea95360d21f4725ab6e07146", size = 2107298, upload-time = "2025-11-04T13:39:04.116Z" },
- { url = "https://files.pythonhosted.org/packages/fb/a8/61c96a77fe28993d9a6fb0f4127e05430a267b235a124545d79fea46dd65/pydantic_core-2.41.5-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:dfa8a0c812ac681395907e71e1274819dec685fec28273a28905df579ef137e2", size = 1901475, upload-time = "2025-11-04T13:39:06.055Z" },
- { url = "https://files.pythonhosted.org/packages/5d/b6/338abf60225acc18cdc08b4faef592d0310923d19a87fba1faf05af5346e/pydantic_core-2.41.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5921a4d3ca3aee735d9fd163808f5e8dd6c6972101e4adbda9a4667908849b97", size = 1918815, upload-time = "2025-11-04T13:39:10.41Z" },
- { url = "https://files.pythonhosted.org/packages/d1/1c/2ed0433e682983d8e8cba9c8d8ef274d4791ec6a6f24c58935b90e780e0a/pydantic_core-2.41.5-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e25c479382d26a2a41b7ebea1043564a937db462816ea07afa8a44c0866d52f9", size = 2065567, upload-time = "2025-11-04T13:39:12.244Z" },
- { url = "https://files.pythonhosted.org/packages/b3/24/cf84974ee7d6eae06b9e63289b7b8f6549d416b5c199ca2d7ce13bbcf619/pydantic_core-2.41.5-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f547144f2966e1e16ae626d8ce72b4cfa0caedc7fa28052001c94fb2fcaa1c52", size = 2230442, upload-time = "2025-11-04T13:39:13.962Z" },
- { url = "https://files.pythonhosted.org/packages/fd/21/4e287865504b3edc0136c89c9c09431be326168b1eb7841911cbc877a995/pydantic_core-2.41.5-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:6f52298fbd394f9ed112d56f3d11aabd0d5bd27beb3084cc3d8ad069483b8941", size = 2350956, upload-time = "2025-11-04T13:39:15.889Z" },
- { url = "https://files.pythonhosted.org/packages/a8/76/7727ef2ffa4b62fcab916686a68a0426b9b790139720e1934e8ba797e238/pydantic_core-2.41.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:100baa204bb412b74fe285fb0f3a385256dad1d1879f0a5cb1499ed2e83d132a", size = 2068253, upload-time = "2025-11-04T13:39:17.403Z" },
- { url = "https://files.pythonhosted.org/packages/d5/8c/a4abfc79604bcb4c748e18975c44f94f756f08fb04218d5cb87eb0d3a63e/pydantic_core-2.41.5-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:05a2c8852530ad2812cb7914dc61a1125dc4e06252ee98e5638a12da6cc6fb6c", size = 2177050, upload-time = "2025-11-04T13:39:19.351Z" },
- { url = "https://files.pythonhosted.org/packages/67/b1/de2e9a9a79b480f9cb0b6e8b6ba4c50b18d4e89852426364c66aa82bb7b3/pydantic_core-2.41.5-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:29452c56df2ed968d18d7e21f4ab0ac55e71dc59524872f6fc57dcf4a3249ed2", size = 2147178, upload-time = "2025-11-04T13:39:21Z" },
- { url = "https://files.pythonhosted.org/packages/16/c1/dfb33f837a47b20417500efaa0378adc6635b3c79e8369ff7a03c494b4ac/pydantic_core-2.41.5-cp310-cp310-musllinux_1_1_armv7l.whl", hash = "sha256:d5160812ea7a8a2ffbe233d8da666880cad0cbaf5d4de74ae15c313213d62556", size = 2341833, upload-time = "2025-11-04T13:39:22.606Z" },
- { url = "https://files.pythonhosted.org/packages/47/36/00f398642a0f4b815a9a558c4f1dca1b4020a7d49562807d7bc9ff279a6c/pydantic_core-2.41.5-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:df3959765b553b9440adfd3c795617c352154e497a4eaf3752555cfb5da8fc49", size = 2321156, upload-time = "2025-11-04T13:39:25.843Z" },
- { url = "https://files.pythonhosted.org/packages/7e/70/cad3acd89fde2010807354d978725ae111ddf6d0ea46d1ea1775b5c1bd0c/pydantic_core-2.41.5-cp310-cp310-win32.whl", hash = "sha256:1f8d33a7f4d5a7889e60dc39856d76d09333d8a6ed0f5f1190635cbec70ec4ba", size = 1989378, upload-time = "2025-11-04T13:39:27.92Z" },
- { url = "https://files.pythonhosted.org/packages/76/92/d338652464c6c367e5608e4488201702cd1cbb0f33f7b6a85a60fe5f3720/pydantic_core-2.41.5-cp310-cp310-win_amd64.whl", hash = "sha256:62de39db01b8d593e45871af2af9e497295db8d73b085f6bfd0b18c83c70a8f9", size = 2013622, upload-time = "2025-11-04T13:39:29.848Z" },
- { url = "https://files.pythonhosted.org/packages/e8/72/74a989dd9f2084b3d9530b0915fdda64ac48831c30dbf7c72a41a5232db8/pydantic_core-2.41.5-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:a3a52f6156e73e7ccb0f8cced536adccb7042be67cb45f9562e12b319c119da6", size = 2105873, upload-time = "2025-11-04T13:39:31.373Z" },
- { url = "https://files.pythonhosted.org/packages/12/44/37e403fd9455708b3b942949e1d7febc02167662bf1a7da5b78ee1ea2842/pydantic_core-2.41.5-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:7f3bf998340c6d4b0c9a2f02d6a400e51f123b59565d74dc60d252ce888c260b", size = 1899826, upload-time = "2025-11-04T13:39:32.897Z" },
- { url = "https://files.pythonhosted.org/packages/33/7f/1d5cab3ccf44c1935a359d51a8a2a9e1a654b744b5e7f80d41b88d501eec/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:378bec5c66998815d224c9ca994f1e14c0c21cb95d2f52b6021cc0b2a58f2a5a", size = 1917869, upload-time = "2025-11-04T13:39:34.469Z" },
- { url = "https://files.pythonhosted.org/packages/6e/6a/30d94a9674a7fe4f4744052ed6c5e083424510be1e93da5bc47569d11810/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e7b576130c69225432866fe2f4a469a85a54ade141d96fd396dffcf607b558f8", size = 2063890, upload-time = "2025-11-04T13:39:36.053Z" },
- { url = "https://files.pythonhosted.org/packages/50/be/76e5d46203fcb2750e542f32e6c371ffa9b8ad17364cf94bb0818dbfb50c/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:6cb58b9c66f7e4179a2d5e0f849c48eff5c1fca560994d6eb6543abf955a149e", size = 2229740, upload-time = "2025-11-04T13:39:37.753Z" },
- { url = "https://files.pythonhosted.org/packages/d3/ee/fed784df0144793489f87db310a6bbf8118d7b630ed07aa180d6067e653a/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:88942d3a3dff3afc8288c21e565e476fc278902ae4d6d134f1eeda118cc830b1", size = 2350021, upload-time = "2025-11-04T13:39:40.94Z" },
- { url = "https://files.pythonhosted.org/packages/c8/be/8fed28dd0a180dca19e72c233cbf58efa36df055e5b9d90d64fd1740b828/pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f31d95a179f8d64d90f6831d71fa93290893a33148d890ba15de25642c5d075b", size = 2066378, upload-time = "2025-11-04T13:39:42.523Z" },
- { url = "https://files.pythonhosted.org/packages/b0/3b/698cf8ae1d536a010e05121b4958b1257f0b5522085e335360e53a6b1c8b/pydantic_core-2.41.5-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c1df3d34aced70add6f867a8cf413e299177e0c22660cc767218373d0779487b", size = 2175761, upload-time = "2025-11-04T13:39:44.553Z" },
- { url = "https://files.pythonhosted.org/packages/b8/ba/15d537423939553116dea94ce02f9c31be0fa9d0b806d427e0308ec17145/pydantic_core-2.41.5-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:4009935984bd36bd2c774e13f9a09563ce8de4abaa7226f5108262fa3e637284", size = 2146303, upload-time = "2025-11-04T13:39:46.238Z" },
- { url = "https://files.pythonhosted.org/packages/58/7f/0de669bf37d206723795f9c90c82966726a2ab06c336deba4735b55af431/pydantic_core-2.41.5-cp311-cp311-musllinux_1_1_armv7l.whl", hash = "sha256:34a64bc3441dc1213096a20fe27e8e128bd3ff89921706e83c0b1ac971276594", size = 2340355, upload-time = "2025-11-04T13:39:48.002Z" },
- { url = "https://files.pythonhosted.org/packages/e5/de/e7482c435b83d7e3c3ee5ee4451f6e8973cff0eb6007d2872ce6383f6398/pydantic_core-2.41.5-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:c9e19dd6e28fdcaa5a1de679aec4141f691023916427ef9bae8584f9c2fb3b0e", size = 2319875, upload-time = "2025-11-04T13:39:49.705Z" },
- { url = "https://files.pythonhosted.org/packages/fe/e6/8c9e81bb6dd7560e33b9053351c29f30c8194b72f2d6932888581f503482/pydantic_core-2.41.5-cp311-cp311-win32.whl", hash = "sha256:2c010c6ded393148374c0f6f0bf89d206bf3217f201faa0635dcd56bd1520f6b", size = 1987549, upload-time = "2025-11-04T13:39:51.842Z" },
- { url = "https://files.pythonhosted.org/packages/11/66/f14d1d978ea94d1bc21fc98fcf570f9542fe55bfcc40269d4e1a21c19bf7/pydantic_core-2.41.5-cp311-cp311-win_amd64.whl", hash = "sha256:76ee27c6e9c7f16f47db7a94157112a2f3a00e958bc626e2f4ee8bec5c328fbe", size = 2011305, upload-time = "2025-11-04T13:39:53.485Z" },
- { url = "https://files.pythonhosted.org/packages/56/d8/0e271434e8efd03186c5386671328154ee349ff0354d83c74f5caaf096ed/pydantic_core-2.41.5-cp311-cp311-win_arm64.whl", hash = "sha256:4bc36bbc0b7584de96561184ad7f012478987882ebf9f9c389b23f432ea3d90f", size = 1972902, upload-time = "2025-11-04T13:39:56.488Z" },
{ url = "https://files.pythonhosted.org/packages/5f/5d/5f6c63eebb5afee93bcaae4ce9a898f3373ca23df3ccaef086d0233a35a7/pydantic_core-2.41.5-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:f41a7489d32336dbf2199c8c0a215390a751c5b014c2c1c5366e817202e9cdf7", size = 2110990, upload-time = "2025-11-04T13:39:58.079Z" },
{ url = "https://files.pythonhosted.org/packages/aa/32/9c2e8ccb57c01111e0fd091f236c7b371c1bccea0fa85247ac55b1e2b6b6/pydantic_core-2.41.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:070259a8818988b9a84a449a2a7337c7f430a22acc0859c6b110aa7212a6d9c0", size = 1896003, upload-time = "2025-11-04T13:39:59.956Z" },
{ url = "https://files.pythonhosted.org/packages/68/b8/a01b53cb0e59139fbc9e4fda3e9724ede8de279097179be4ff31f1abb65a/pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e96cea19e34778f8d59fe40775a7a574d95816eb150850a85a7a4c8f4b94ac69", size = 1919200, upload-time = "2025-11-04T13:40:02.241Z" },
@@ -3982,30 +4981,24 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/5c/96/5fb7d8c3c17bc8c62fdb031c47d77a1af698f1d7a406b0f79aaa1338f9ad/pydantic_core-2.41.5-cp314-cp314t-win32.whl", hash = "sha256:b4ececa40ac28afa90871c2cc2b9ffd2ff0bf749380fbdf57d165fd23da353aa", size = 1988906, upload-time = "2025-11-04T13:41:56.606Z" },
{ url = "https://files.pythonhosted.org/packages/22/ed/182129d83032702912c2e2d8bbe33c036f342cc735737064668585dac28f/pydantic_core-2.41.5-cp314-cp314t-win_amd64.whl", hash = "sha256:80aa89cad80b32a912a65332f64a4450ed00966111b6615ca6816153d3585a8c", size = 1981607, upload-time = "2025-11-04T13:41:58.889Z" },
{ url = "https://files.pythonhosted.org/packages/9f/ed/068e41660b832bb0b1aa5b58011dea2a3fe0ba7861ff38c4d4904c1c1a99/pydantic_core-2.41.5-cp314-cp314t-win_arm64.whl", hash = "sha256:35b44f37a3199f771c3eaa53051bc8a70cd7b54f333531c59e29fd4db5d15008", size = 1974769, upload-time = "2025-11-04T13:42:01.186Z" },
- { url = "https://files.pythonhosted.org/packages/11/72/90fda5ee3b97e51c494938a4a44c3a35a9c96c19bba12372fb9c634d6f57/pydantic_core-2.41.5-graalpy311-graalpy242_311_native-macosx_10_12_x86_64.whl", hash = "sha256:b96d5f26b05d03cc60f11a7761a5ded1741da411e7fe0909e27a5e6a0cb7b034", size = 2115441, upload-time = "2025-11-04T13:42:39.557Z" },
- { url = "https://files.pythonhosted.org/packages/1f/53/8942f884fa33f50794f119012dc6a1a02ac43a56407adaac20463df8e98f/pydantic_core-2.41.5-graalpy311-graalpy242_311_native-macosx_11_0_arm64.whl", hash = "sha256:634e8609e89ceecea15e2d61bc9ac3718caaaa71963717bf3c8f38bfde64242c", size = 1930291, upload-time = "2025-11-04T13:42:42.169Z" },
- { url = "https://files.pythonhosted.org/packages/79/c8/ecb9ed9cd942bce09fc888ee960b52654fbdbede4ba6c2d6e0d3b1d8b49c/pydantic_core-2.41.5-graalpy311-graalpy242_311_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:93e8740d7503eb008aa2df04d3b9735f845d43ae845e6dcd2be0b55a2da43cd2", size = 1948632, upload-time = "2025-11-04T13:42:44.564Z" },
- { url = "https://files.pythonhosted.org/packages/2e/1b/687711069de7efa6af934e74f601e2a4307365e8fdc404703afc453eab26/pydantic_core-2.41.5-graalpy311-graalpy242_311_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f15489ba13d61f670dcc96772e733aad1a6f9c429cc27574c6cdaed82d0146ad", size = 2138905, upload-time = "2025-11-04T13:42:47.156Z" },
{ url = "https://files.pythonhosted.org/packages/09/32/59b0c7e63e277fa7911c2fc70ccfb45ce4b98991e7ef37110663437005af/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:7da7087d756b19037bc2c06edc6c170eeef3c3bafcb8f532ff17d64dc427adfd", size = 2110495, upload-time = "2025-11-04T13:42:49.689Z" },
{ url = "https://files.pythonhosted.org/packages/aa/81/05e400037eaf55ad400bcd318c05bb345b57e708887f07ddb2d20e3f0e98/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:aabf5777b5c8ca26f7824cb4a120a740c9588ed58df9b2d196ce92fba42ff8dc", size = 1915388, upload-time = "2025-11-04T13:42:52.215Z" },
{ url = "https://files.pythonhosted.org/packages/6e/0d/e3549b2399f71d56476b77dbf3cf8937cec5cd70536bdc0e374a421d0599/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c007fe8a43d43b3969e8469004e9845944f1a80e6acd47c150856bb87f230c56", size = 1942879, upload-time = "2025-11-04T13:42:56.483Z" },
{ url = "https://files.pythonhosted.org/packages/f7/07/34573da085946b6a313d7c42f82f16e8920bfd730665de2d11c0c37a74b5/pydantic_core-2.41.5-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:76d0819de158cd855d1cbb8fcafdf6f5cf1eb8e470abe056d5d161106e38062b", size = 2139017, upload-time = "2025-11-04T13:42:59.471Z" },
- { url = "https://files.pythonhosted.org/packages/e6/b0/1a2aa41e3b5a4ba11420aba2d091b2d17959c8d1519ece3627c371951e73/pydantic_core-2.41.5-pp310-pypy310_pp73-macosx_10_12_x86_64.whl", hash = "sha256:b5819cd790dbf0c5eb9f82c73c16b39a65dd6dd4d1439dcdea7816ec9adddab8", size = 2103351, upload-time = "2025-11-04T13:43:02.058Z" },
- { url = "https://files.pythonhosted.org/packages/a4/ee/31b1f0020baaf6d091c87900ae05c6aeae101fa4e188e1613c80e4f1ea31/pydantic_core-2.41.5-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:5a4e67afbc95fa5c34cf27d9089bca7fcab4e51e57278d710320a70b956d1b9a", size = 1925363, upload-time = "2025-11-04T13:43:05.159Z" },
- { url = "https://files.pythonhosted.org/packages/e1/89/ab8e86208467e467a80deaca4e434adac37b10a9d134cd2f99b28a01e483/pydantic_core-2.41.5-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ece5c59f0ce7d001e017643d8d24da587ea1f74f6993467d85ae8a5ef9d4f42b", size = 2135615, upload-time = "2025-11-04T13:43:08.116Z" },
- { url = "https://files.pythonhosted.org/packages/99/0a/99a53d06dd0348b2008f2f30884b34719c323f16c3be4e6cc1203b74a91d/pydantic_core-2.41.5-pp310-pypy310_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:16f80f7abe3351f8ea6858914ddc8c77e02578544a0ebc15b4c2e1a0e813b0b2", size = 2175369, upload-time = "2025-11-04T13:43:12.49Z" },
- { url = "https://files.pythonhosted.org/packages/6d/94/30ca3b73c6d485b9bb0bc66e611cff4a7138ff9736b7e66bcf0852151636/pydantic_core-2.41.5-pp310-pypy310_pp73-musllinux_1_1_aarch64.whl", hash = "sha256:33cb885e759a705b426baada1fe68cbb0a2e68e34c5d0d0289a364cf01709093", size = 2144218, upload-time = "2025-11-04T13:43:15.431Z" },
- { url = "https://files.pythonhosted.org/packages/87/57/31b4f8e12680b739a91f472b5671294236b82586889ef764b5fbc6669238/pydantic_core-2.41.5-pp310-pypy310_pp73-musllinux_1_1_armv7l.whl", hash = "sha256:c8d8b4eb992936023be7dee581270af5c6e0697a8559895f527f5b7105ecd36a", size = 2329951, upload-time = "2025-11-04T13:43:18.062Z" },
- { url = "https://files.pythonhosted.org/packages/7d/73/3c2c8edef77b8f7310e6fb012dbc4b8551386ed575b9eb6fb2506e28a7eb/pydantic_core-2.41.5-pp310-pypy310_pp73-musllinux_1_1_x86_64.whl", hash = "sha256:242a206cd0318f95cd21bdacff3fcc3aab23e79bba5cac3db5a841c9ef9c6963", size = 2318428, upload-time = "2025-11-04T13:43:20.679Z" },
- { url = "https://files.pythonhosted.org/packages/2f/02/8559b1f26ee0d502c74f9cca5c0d2fd97e967e083e006bbbb4e97f3a043a/pydantic_core-2.41.5-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:d3a978c4f57a597908b7e697229d996d77a6d3c94901e9edee593adada95ce1a", size = 2147009, upload-time = "2025-11-04T13:43:23.286Z" },
- { url = "https://files.pythonhosted.org/packages/5f/9b/1b3f0e9f9305839d7e84912f9e8bfbd191ed1b1ef48083609f0dabde978c/pydantic_core-2.41.5-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:b2379fa7ed44ddecb5bfe4e48577d752db9fc10be00a6b7446e9663ba143de26", size = 2101980, upload-time = "2025-11-04T13:43:25.97Z" },
- { url = "https://files.pythonhosted.org/packages/a4/ed/d71fefcb4263df0da6a85b5d8a7508360f2f2e9b3bf5814be9c8bccdccc1/pydantic_core-2.41.5-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:266fb4cbf5e3cbd0b53669a6d1b039c45e3ce651fd5442eff4d07c2cc8d66808", size = 1923865, upload-time = "2025-11-04T13:43:28.763Z" },
- { url = "https://files.pythonhosted.org/packages/ce/3a/626b38db460d675f873e4444b4bb030453bbe7b4ba55df821d026a0493c4/pydantic_core-2.41.5-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:58133647260ea01e4d0500089a8c4f07bd7aa6ce109682b1426394988d8aaacc", size = 2134256, upload-time = "2025-11-04T13:43:31.71Z" },
- { url = "https://files.pythonhosted.org/packages/83/d9/8412d7f06f616bbc053d30cb4e5f76786af3221462ad5eee1f202021eb4e/pydantic_core-2.41.5-pp311-pypy311_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:287dad91cfb551c363dc62899a80e9e14da1f0e2b6ebde82c806612ca2a13ef1", size = 2174762, upload-time = "2025-11-04T13:43:34.744Z" },
- { url = "https://files.pythonhosted.org/packages/55/4c/162d906b8e3ba3a99354e20faa1b49a85206c47de97a639510a0e673f5da/pydantic_core-2.41.5-pp311-pypy311_pp73-musllinux_1_1_aarch64.whl", hash = "sha256:03b77d184b9eb40240ae9fd676ca364ce1085f203e1b1256f8ab9984dca80a84", size = 2143141, upload-time = "2025-11-04T13:43:37.701Z" },
- { url = "https://files.pythonhosted.org/packages/1f/f2/f11dd73284122713f5f89fc940f370d035fa8e1e078d446b3313955157fe/pydantic_core-2.41.5-pp311-pypy311_pp73-musllinux_1_1_armv7l.whl", hash = "sha256:a668ce24de96165bb239160b3d854943128f4334822900534f2fe947930e5770", size = 2330317, upload-time = "2025-11-04T13:43:40.406Z" },
- { url = "https://files.pythonhosted.org/packages/88/9d/b06ca6acfe4abb296110fb1273a4d848a0bfb2ff65f3ee92127b3244e16b/pydantic_core-2.41.5-pp311-pypy311_pp73-musllinux_1_1_x86_64.whl", hash = "sha256:f14f8f046c14563f8eb3f45f499cc658ab8d10072961e07225e507adb700e93f", size = 2316992, upload-time = "2025-11-04T13:43:43.602Z" },
- { url = "https://files.pythonhosted.org/packages/36/c7/cfc8e811f061c841d7990b0201912c3556bfeb99cdcb7ed24adc8d6f8704/pydantic_core-2.41.5-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:56121965f7a4dc965bff783d70b907ddf3d57f6eba29b6d2e5dabfaf07799c51", size = 2145302, upload-time = "2025-11-04T13:43:46.64Z" },
+]
+
+[[package]]
+name = "pydantic-settings"
+version = "2.13.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pydantic" },
+ { name = "python-dotenv" },
+ { name = "typing-inspection" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/52/6d/fffca34caecc4a3f97bda81b2098da5e8ab7efc9a66e819074a11955d87e/pydantic_settings-2.13.1.tar.gz", hash = "sha256:b4c11847b15237fb0171e1462bf540e294affb9b86db4d9aa5c01730bdbe4025", size = 223826, upload-time = "2026-02-19T13:45:08.055Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/00/4b/ccc026168948fec4f7555b9164c724cf4125eac006e176541483d2c959be/pydantic_settings-2.13.1-py3-none-any.whl", hash = "sha256:d56fd801823dbeae7f0975e1f8c8e25c258eb75d278ea7abb5d9cebb01b56237", size = 58929, upload-time = "2026-02-19T13:45:06.034Z" },
]
[[package]]
@@ -4018,8 +5011,7 @@ dependencies = [
{ name = "beautifulsoup4" },
{ name = "docutils" },
{ name = "pygments" },
- { name = "sphinx", version = "8.1.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinx", version = "8.2.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "sphinx" },
{ name = "typing-extensions" },
]
sdist = { url = "https://files.pythonhosted.org/packages/00/20/bb50f9de3a6de69e6abd6b087b52fa2418a0418b19597601605f855ad044/pydata_sphinx_theme-0.16.1.tar.gz", hash = "sha256:a08b7f0b7f70387219dc659bff0893a7554d5eb39b59d3b8ef37b8401b7642d7", size = 2412693, upload-time = "2024-12-17T10:53:39.537Z" }
@@ -4038,7 +5030,7 @@ wheels = [
[[package]]
name = "pygithub"
-version = "2.8.1"
+version = "2.9.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "pyjwt", extra = ["crypto"] },
@@ -4047,27 +5039,27 @@ dependencies = [
{ name = "typing-extensions" },
{ name = "urllib3" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/c1/74/e560bdeffea72ecb26cff27f0fad548bbff5ecc51d6a155311ea7f9e4c4c/pygithub-2.8.1.tar.gz", hash = "sha256:341b7c78521cb07324ff670afd1baa2bf5c286f8d9fd302c1798ba594a5400c9", size = 2246994, upload-time = "2025-09-02T17:41:54.674Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/a6/9a/44f918e9be12e49cb8b053f09d5d0733b74df52bf4dabc570da1c3ecd9f6/pygithub-2.9.0.tar.gz", hash = "sha256:a26abda1222febba31238682634cad11d8b966137ed6cc3c5e445b29a11cb0a4", size = 2592289, upload-time = "2026-03-22T21:14:39.053Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/07/ba/7049ce39f653f6140aac4beb53a5aaf08b4407b6a3019aae394c1c5244ff/pygithub-2.8.1-py3-none-any.whl", hash = "sha256:23a0a5bca93baef082e03411bf0ce27204c32be8bfa7abc92fe4a3e132936df0", size = 432709, upload-time = "2025-09-02T17:41:52.947Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/de/72e02bc7674e161b155a4b5a03b2347129d0626115bc97ba5bad5070cac9/pygithub-2.9.0-py3-none-any.whl", hash = "sha256:5e2b260ce327bffce9b00f447b65953ef7078ffe93e5a5425624a3075483927c", size = 449653, upload-time = "2026-03-22T21:14:37.726Z" },
]
[[package]]
name = "pygments"
-version = "2.19.2"
+version = "2.20.0"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/b0/77/a5b8c569bf593b0140bde72ea885a803b82086995367bf2037de0159d924/pygments-2.19.2.tar.gz", hash = "sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887", size = 4968631, upload-time = "2025-06-21T13:39:12.283Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/c7/21/705964c7812476f378728bdf590ca4b771ec72385c533964653c68e86bdc/pygments-2.19.2-py3-none-any.whl", hash = "sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b", size = 1225217, upload-time = "2025-06-21T13:39:07.939Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" },
]
[[package]]
name = "pyjwt"
-version = "2.10.1"
+version = "2.12.1"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/e7/46/bd74733ff231675599650d3e47f361794b22ef3e3770998dda30d3b63726/pyjwt-2.10.1.tar.gz", hash = "sha256:3cc5772eb20009233caf06e9d8a0577824723b44e6648ee0a2aedb6cf9381953", size = 87785, upload-time = "2024-11-28T03:43:29.933Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/c2/27/a3b6e5bf6ff856d2509292e95c8f57f0df7017cf5394921fc4e4ef40308a/pyjwt-2.12.1.tar.gz", hash = "sha256:c74a7a2adf861c04d002db713dd85f84beb242228e671280bf709d765b03672b", size = 102564, upload-time = "2026-03-13T19:27:37.25Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/61/ad/689f02752eeec26aed679477e80e632ef1b682313be70793d798c1d5fc8f/PyJWT-2.10.1-py3-none-any.whl", hash = "sha256:dcdd193e30abefd5debf142f9adfcdd2b58004e644f25406ffaebd50bd98dacb", size = 22997, upload-time = "2024-11-28T03:43:27.893Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/7a/8dd906bd22e79e47397a61742927f6747fe93242ef86645ee9092e610244/pyjwt-2.12.1-py3-none-any.whl", hash = "sha256:28ca37c070cad8ba8cd9790cd940535d40274d22f80ab87f3ac6a713e6e8454c", size = 29726, upload-time = "2026-03-13T19:27:35.677Z" },
]
[package.optional-dependencies]
@@ -4086,7 +5078,6 @@ dependencies = [
{ name = "isort" },
{ name = "mccabe" },
{ name = "platformdirs" },
- { name = "tomli", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
{ name = "tomlkit" },
]
sdist = { url = "https://files.pythonhosted.org/packages/30/10/abee071c1d52b2bca48be40fe9f64ca878a77e0beef6504597e8c9c1ed84/pylint-3.2.6.tar.gz", hash = "sha256:a5d01678349454806cff6d886fb072294f56a58c4761278c97fb557d708e1eb3", size = 1510167, upload-time = "2024-07-21T19:48:38.032Z" }
@@ -4148,11 +5139,10 @@ version = "8.3.5"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "exceptiongroup", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
{ name = "iniconfig" },
- { name = "packaging" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
{ name = "pluggy" },
- { name = "tomli", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
]
sdist = { url = "https://files.pythonhosted.org/packages/ae/3c/c9d525a414d506893f0cd8a8d0de7706446213181570cdbd766691164e40/pytest-8.3.5.tar.gz", hash = "sha256:f4efe70cc14e511565ac476b57c279e12a855b11f48f212af1080ef2263d3845", size = 1450891, upload-time = "2025-03-02T12:54:54.503Z" }
wheels = [
@@ -4164,7 +5154,6 @@ name = "pytest-asyncio"
version = "1.3.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
- { name = "backports-asyncio-runner", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
{ name = "pytest" },
{ name = "typing-extensions", marker = "python_full_version < '3.13' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
]
@@ -4175,16 +5164,16 @@ wheels = [
[[package]]
name = "pytest-cov"
-version = "7.0.0"
+version = "7.1.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
- { name = "coverage", extra = ["toml"] },
+ { name = "coverage" },
{ name = "pluggy" },
{ name = "pytest" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/5e/f7/c933acc76f5208b3b00089573cf6a2bc26dc80a8aece8f52bb7d6b1855ca/pytest_cov-7.0.0.tar.gz", hash = "sha256:33c97eda2e049a0c5298e91f519302a1334c26ac65c1a483d6206fd458361af1", size = 54328, upload-time = "2025-09-09T10:57:02.113Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/b1/51/a849f96e117386044471c8ec2bd6cfebacda285da9525c9106aeb28da671/pytest_cov-7.1.0.tar.gz", hash = "sha256:30674f2b5f6351aa09702a9c8c364f6a01c27aae0c1366ae8016160d1efc56b2", size = 55592, upload-time = "2026-03-21T20:11:16.284Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/ee/49/1377b49de7d0c1ce41292161ea0f721913fa8722c19fb9c1e3aa0367eecb/pytest_cov-7.0.0-py3-none-any.whl", hash = "sha256:3b8e9558b16cc1479da72058bdecf8073661c7f57f7d3c5f22a1c23507f2d861", size = 22424, upload-time = "2025-09-09T10:57:00.695Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/7a/d968e294073affff457b041c2be9868a40c1c71f4a35fcc1e45e5493067b/pytest_cov-7.1.0-py3-none-any.whl", hash = "sha256:a0461110b7865f9a271aa1b51e516c9a95de9d696734a2f71e3e78f46e1d4678", size = 22876, upload-time = "2026-03-21T20:11:14.438Z" },
]
[[package]]
@@ -4223,35 +5212,57 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" },
]
+[[package]]
+name = "python-discovery"
+version = "1.2.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "filelock" },
+ { name = "platformdirs" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/b9/88/815e53084c5079a59df912825a279f41dd2e0df82281770eadc732f5352c/python_discovery-1.2.1.tar.gz", hash = "sha256:180c4d114bff1c32462537eac5d6a332b768242b76b69c0259c7d14b1b680c9e", size = 58457, upload-time = "2026-03-26T22:30:44.496Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/67/0f/019d3949a40280f6193b62bc010177d4ce702d0fce424322286488569cd3/python_discovery-1.2.1-py3-none-any.whl", hash = "sha256:b6a957b24c1cd79252484d3566d1b49527581d46e789aaf43181005e56201502", size = 31674, upload-time = "2026-03-26T22:30:43.396Z" },
+]
+
+[[package]]
+name = "python-dotenv"
+version = "1.2.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/82/ed/0301aeeac3e5353ef3d94b6ec08bbcabd04a72018415dcb29e588514bba8/python_dotenv-1.2.2.tar.gz", hash = "sha256:2c371a91fbd7ba082c2c1dc1f8bf89ca22564a087c2c287cd9b662adde799cf3", size = 50135, upload-time = "2026-03-01T16:00:26.196Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0b/d7/1959b9648791274998a9c3526f6d0ec8fd2233e4d4acce81bbae76b44b2a/python_dotenv-1.2.2-py3-none-any.whl", hash = "sha256:1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a", size = 22101, upload-time = "2026-03-01T16:00:25.09Z" },
+]
+
[[package]]
name = "python-gitlab"
-version = "7.1.0"
+version = "8.2.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "requests" },
{ name = "requests-toolbelt" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/31/98/0b5d0a0367b90aec818298390b60ae65e6a08989cf5140271d0ee0206882/python_gitlab-7.1.0.tar.gz", hash = "sha256:1c34da3de40ad21675d788136f73d20a60649513e692f52c5a9720434db97c46", size = 401058, upload-time = "2025-12-28T01:27:01.369Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/77/9c/ec0073c260812bca170418e8e259a2ed0728200af4b3f7120badf2e6f05a/python_gitlab-8.2.0.tar.gz", hash = "sha256:de874dc538db6dceb48929f4c8fb55d6064dd19cb3ffdad1100190f835c5b674", size = 407110, upload-time = "2026-03-28T01:50:07.66Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/14/44/70fa1e395731b6a4b1f249d5f7326f3bb6281e2cf94d6535f679239f4b93/python_gitlab-7.1.0-py3-none-any.whl", hash = "sha256:8e42030cf27674e7ec9ea1f6d2fedcaaef0a6210f5fa22c80721abaa3a4fec90", size = 144441, upload-time = "2025-12-28T01:26:59.726Z" },
+ { url = "https://files.pythonhosted.org/packages/38/ee/a61bb562bdf6f0bc6c51cdcf80ab5503cbb4b2f5053fa4b054cc0a56e48a/python_gitlab-8.2.0-py3-none-any.whl", hash = "sha256:884618d4d60beadb21bb0c5f0cca46e70c6e501784f136bf0b6f85f5bc15ce62", size = 147477, upload-time = "2026-03-28T01:50:06.237Z" },
]
[[package]]
name = "python-multipart"
-version = "0.0.21"
+version = "0.0.22"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/78/96/804520d0850c7db98e5ccb70282e29208723f0964e88ffd9d0da2f52ea09/python_multipart-0.0.21.tar.gz", hash = "sha256:7137ebd4d3bbf70ea1622998f902b97a29434a9e8dc40eb203bbcf7c2a2cba92", size = 37196, upload-time = "2025-12-17T09:24:22.446Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/94/01/979e98d542a70714b0cb2b6728ed0b7c46792b695e3eaec3e20711271ca3/python_multipart-0.0.22.tar.gz", hash = "sha256:7340bef99a7e0032613f56dc36027b959fd3b30a787ed62d310e951f7c3a3a58", size = 37612, upload-time = "2026-01-25T10:15:56.219Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/aa/76/03af049af4dcee5d27442f71b6924f01f3efb5d2bd34f23fcd563f2cc5f5/python_multipart-0.0.21-py3-none-any.whl", hash = "sha256:cf7a6713e01c87aa35387f4774e812c4361150938d20d232800f75ffcf266090", size = 24541, upload-time = "2025-12-17T09:24:21.153Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/d0/397f9626e711ff749a95d96b7af99b9c566a9bb5129b8e4c10fc4d100304/python_multipart-0.0.22-py3-none-any.whl", hash = "sha256:2b2cd894c83d21bf49d702499531c7bafd057d730c201782048f7945d82de155", size = 24579, upload-time = "2026-01-25T10:15:54.811Z" },
]
[[package]]
name = "pytz"
-version = "2025.2"
+version = "2026.1.post1"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/f8/bf/abbd3cdfb8fbc7fb3d4d38d320f2441b1e7cbe29be4f23797b4a2b5d8aac/pytz-2025.2.tar.gz", hash = "sha256:360b9e3dbb49a209c21ad61809c7fb453643e048b38924c765813546746e81c3", size = 320884, upload-time = "2025-03-25T02:25:00.538Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/56/db/b8721d71d945e6a8ac63c0fc900b2067181dbb50805958d4d4661cf7d277/pytz-2026.1.post1.tar.gz", hash = "sha256:3378dde6a0c3d26719182142c56e60c7f9af7e968076f31aae569d72a0358ee1", size = 321088, upload-time = "2026-03-03T07:47:50.683Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/81/c4/34e93fe5f5429d7570ec1fa436f1986fb1f00c3e0f43a589fe2bbcd22c3f/pytz-2025.2-py2.py3-none-any.whl", hash = "sha256:5ddf76296dd8c44c26eb8f4b6f35488f3ccbf6fbbd7adee0b7262d43f0ec2f00", size = 509225, upload-time = "2025-03-25T02:24:58.468Z" },
+ { url = "https://files.pythonhosted.org/packages/10/99/781fe0c827be2742bcc775efefccb3b048a3a9c6ce9aec0cbf4a101677e5/pytz-2026.1.post1-py2.py3-none-any.whl", hash = "sha256:f2fd16142fda348286a75e1a524be810bb05d444e5a081f37f7affc635035f7a", size = 510489, upload-time = "2026-03-03T07:47:49.167Z" },
]
[[package]]
@@ -4259,12 +5270,6 @@ name = "pywin32"
version = "311"
source = { registry = "https://pypi.org/simple" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/7b/40/44efbb0dfbd33aca6a6483191dae0716070ed99e2ecb0c53683f400a0b4f/pywin32-311-cp310-cp310-win32.whl", hash = "sha256:d03ff496d2a0cd4a5893504789d4a15399133fe82517455e78bad62efbb7f0a3", size = 8760432, upload-time = "2025-07-14T20:13:05.9Z" },
- { url = "https://files.pythonhosted.org/packages/5e/bf/360243b1e953bd254a82f12653974be395ba880e7ec23e3731d9f73921cc/pywin32-311-cp310-cp310-win_amd64.whl", hash = "sha256:797c2772017851984b97180b0bebe4b620bb86328e8a884bb626156295a63b3b", size = 9590103, upload-time = "2025-07-14T20:13:07.698Z" },
- { url = "https://files.pythonhosted.org/packages/57/38/d290720e6f138086fb3d5ffe0b6caa019a791dd57866940c82e4eeaf2012/pywin32-311-cp310-cp310-win_arm64.whl", hash = "sha256:0502d1facf1fed4839a9a51ccbcc63d952cf318f78ffc00a7e78528ac27d7a2b", size = 8778557, upload-time = "2025-07-14T20:13:11.11Z" },
- { url = "https://files.pythonhosted.org/packages/7c/af/449a6a91e5d6db51420875c54f6aff7c97a86a3b13a0b4f1a5c13b988de3/pywin32-311-cp311-cp311-win32.whl", hash = "sha256:184eb5e436dea364dcd3d2316d577d625c0351bf237c4e9a5fabbcfa5a58b151", size = 8697031, upload-time = "2025-07-14T20:13:13.266Z" },
- { url = "https://files.pythonhosted.org/packages/51/8f/9bb81dd5bb77d22243d33c8397f09377056d5c687aa6d4042bea7fbf8364/pywin32-311-cp311-cp311-win_amd64.whl", hash = "sha256:3ce80b34b22b17ccbd937a6e78e7225d80c52f5ab9940fe0506a1a16f3dab503", size = 9508308, upload-time = "2025-07-14T20:13:15.147Z" },
- { url = "https://files.pythonhosted.org/packages/44/7b/9c2ab54f74a138c491aba1b1cd0795ba61f144c711daea84a88b63dc0f6c/pywin32-311-cp311-cp311-win_arm64.whl", hash = "sha256:a733f1388e1a842abb67ffa8e7aad0e70ac519e09b0f6a784e65a136ec7cefd2", size = 8703930, upload-time = "2025-07-14T20:13:16.945Z" },
{ url = "https://files.pythonhosted.org/packages/e7/ab/01ea1943d4eba0f850c3c61e78e8dd59757ff815ff3ccd0a84de5f541f42/pywin32-311-cp312-cp312-win32.whl", hash = "sha256:750ec6e621af2b948540032557b10a2d43b0cee2ae9758c54154d711cc852d31", size = 8706543, upload-time = "2025-07-14T20:13:20.765Z" },
{ url = "https://files.pythonhosted.org/packages/d1/a8/a0e8d07d4d051ec7502cd58b291ec98dcc0c3fff027caad0470b72cfcc2f/pywin32-311-cp312-cp312-win_amd64.whl", hash = "sha256:b8c095edad5c211ff31c05223658e71bf7116daa0ecf3ad85f3201ea3190d067", size = 9495040, upload-time = "2025-07-14T20:13:22.543Z" },
{ url = "https://files.pythonhosted.org/packages/ba/3a/2ae996277b4b50f17d61f0603efd8253cb2d79cc7ae159468007b586396d/pywin32-311-cp312-cp312-win_arm64.whl", hash = "sha256:e286f46a9a39c4a18b319c28f59b61de793654af2f395c102b4f819e584b5852", size = 8710102, upload-time = "2025-07-14T20:13:24.682Z" },
@@ -4282,24 +5287,6 @@ version = "6.0.3"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/f4/a0/39350dd17dd6d6c6507025c0e53aef67a9293a6d37d3511f23ea510d5800/pyyaml-6.0.3-cp310-cp310-macosx_10_13_x86_64.whl", hash = "sha256:214ed4befebe12df36bcc8bc2b64b396ca31be9304b8f59e25c11cf94a4c033b", size = 184227, upload-time = "2025-09-25T21:31:46.04Z" },
- { url = "https://files.pythonhosted.org/packages/05/14/52d505b5c59ce73244f59c7a50ecf47093ce4765f116cdb98286a71eeca2/pyyaml-6.0.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:02ea2dfa234451bbb8772601d7b8e426c2bfa197136796224e50e35a78777956", size = 174019, upload-time = "2025-09-25T21:31:47.706Z" },
- { url = "https://files.pythonhosted.org/packages/43/f7/0e6a5ae5599c838c696adb4e6330a59f463265bfa1e116cfd1fbb0abaaae/pyyaml-6.0.3-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b30236e45cf30d2b8e7b3e85881719e98507abed1011bf463a8fa23e9c3e98a8", size = 740646, upload-time = "2025-09-25T21:31:49.21Z" },
- { url = "https://files.pythonhosted.org/packages/2f/3a/61b9db1d28f00f8fd0ae760459a5c4bf1b941baf714e207b6eb0657d2578/pyyaml-6.0.3-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:66291b10affd76d76f54fad28e22e51719ef9ba22b29e1d7d03d6777a9174198", size = 840793, upload-time = "2025-09-25T21:31:50.735Z" },
- { url = "https://files.pythonhosted.org/packages/7a/1e/7acc4f0e74c4b3d9531e24739e0ab832a5edf40e64fbae1a9c01941cabd7/pyyaml-6.0.3-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9c7708761fccb9397fe64bbc0395abcae8c4bf7b0eac081e12b809bf47700d0b", size = 770293, upload-time = "2025-09-25T21:31:51.828Z" },
- { url = "https://files.pythonhosted.org/packages/8b/ef/abd085f06853af0cd59fa5f913d61a8eab65d7639ff2a658d18a25d6a89d/pyyaml-6.0.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:418cf3f2111bc80e0933b2cd8cd04f286338bb88bdc7bc8e6dd775ebde60b5e0", size = 732872, upload-time = "2025-09-25T21:31:53.282Z" },
- { url = "https://files.pythonhosted.org/packages/1f/15/2bc9c8faf6450a8b3c9fc5448ed869c599c0a74ba2669772b1f3a0040180/pyyaml-6.0.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:5e0b74767e5f8c593e8c9b5912019159ed0533c70051e9cce3e8b6aa699fcd69", size = 758828, upload-time = "2025-09-25T21:31:54.807Z" },
- { url = "https://files.pythonhosted.org/packages/a3/00/531e92e88c00f4333ce359e50c19b8d1de9fe8d581b1534e35ccfbc5f393/pyyaml-6.0.3-cp310-cp310-win32.whl", hash = "sha256:28c8d926f98f432f88adc23edf2e6d4921ac26fb084b028c733d01868d19007e", size = 142415, upload-time = "2025-09-25T21:31:55.885Z" },
- { url = "https://files.pythonhosted.org/packages/2a/fa/926c003379b19fca39dd4634818b00dec6c62d87faf628d1394e137354d4/pyyaml-6.0.3-cp310-cp310-win_amd64.whl", hash = "sha256:bdb2c67c6c1390b63c6ff89f210c8fd09d9a1217a465701eac7316313c915e4c", size = 158561, upload-time = "2025-09-25T21:31:57.406Z" },
- { url = "https://files.pythonhosted.org/packages/6d/16/a95b6757765b7b031c9374925bb718d55e0a9ba8a1b6a12d25962ea44347/pyyaml-6.0.3-cp311-cp311-macosx_10_13_x86_64.whl", hash = "sha256:44edc647873928551a01e7a563d7452ccdebee747728c1080d881d68af7b997e", size = 185826, upload-time = "2025-09-25T21:31:58.655Z" },
- { url = "https://files.pythonhosted.org/packages/16/19/13de8e4377ed53079ee996e1ab0a9c33ec2faf808a4647b7b4c0d46dd239/pyyaml-6.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:652cb6edd41e718550aad172851962662ff2681490a8a711af6a4d288dd96824", size = 175577, upload-time = "2025-09-25T21:32:00.088Z" },
- { url = "https://files.pythonhosted.org/packages/0c/62/d2eb46264d4b157dae1275b573017abec435397aa59cbcdab6fc978a8af4/pyyaml-6.0.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:10892704fc220243f5305762e276552a0395f7beb4dbf9b14ec8fd43b57f126c", size = 775556, upload-time = "2025-09-25T21:32:01.31Z" },
- { url = "https://files.pythonhosted.org/packages/10/cb/16c3f2cf3266edd25aaa00d6c4350381c8b012ed6f5276675b9eba8d9ff4/pyyaml-6.0.3-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:850774a7879607d3a6f50d36d04f00ee69e7fc816450e5f7e58d7f17f1ae5c00", size = 882114, upload-time = "2025-09-25T21:32:03.376Z" },
- { url = "https://files.pythonhosted.org/packages/71/60/917329f640924b18ff085ab889a11c763e0b573da888e8404ff486657602/pyyaml-6.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b8bb0864c5a28024fac8a632c443c87c5aa6f215c0b126c449ae1a150412f31d", size = 806638, upload-time = "2025-09-25T21:32:04.553Z" },
- { url = "https://files.pythonhosted.org/packages/dd/6f/529b0f316a9fd167281a6c3826b5583e6192dba792dd55e3203d3f8e655a/pyyaml-6.0.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:1d37d57ad971609cf3c53ba6a7e365e40660e3be0e5175fa9f2365a379d6095a", size = 767463, upload-time = "2025-09-25T21:32:06.152Z" },
- { url = "https://files.pythonhosted.org/packages/f2/6a/b627b4e0c1dd03718543519ffb2f1deea4a1e6d42fbab8021936a4d22589/pyyaml-6.0.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:37503bfbfc9d2c40b344d06b2199cf0e96e97957ab1c1b546fd4f87e53e5d3e4", size = 794986, upload-time = "2025-09-25T21:32:07.367Z" },
- { url = "https://files.pythonhosted.org/packages/45/91/47a6e1c42d9ee337c4839208f30d9f09caa9f720ec7582917b264defc875/pyyaml-6.0.3-cp311-cp311-win32.whl", hash = "sha256:8098f252adfa6c80ab48096053f512f2321f0b998f98150cea9bd23d83e1467b", size = 142543, upload-time = "2025-09-25T21:32:08.95Z" },
- { url = "https://files.pythonhosted.org/packages/da/e3/ea007450a105ae919a72393cb06f122f288ef60bba2dc64b26e2646fa315/pyyaml-6.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:9f3bfb4965eb874431221a3ff3fdcddc7e74e3b07799e0e84ca4a0f867d449bf", size = 158763, upload-time = "2025-09-25T21:32:09.96Z" },
{ url = "https://files.pythonhosted.org/packages/d1/33/422b98d2195232ca1826284a76852ad5a86fe23e31b009c9886b2d0fb8b2/pyyaml-6.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196", size = 182063, upload-time = "2025-09-25T21:32:11.445Z" },
{ url = "https://files.pythonhosted.org/packages/89/a0/6cf41a19a1f2f3feab0e9c0b74134aa2ce6849093d5517a0c550fe37a648/pyyaml-6.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0", size = 173973, upload-time = "2025-09-25T21:32:12.492Z" },
{ url = "https://files.pythonhosted.org/packages/ed/23/7a778b6bd0b9a8039df8b1b1d80e2e2ad78aa04171592c8a5c43a56a6af4/pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28", size = 775116, upload-time = "2025-09-25T21:32:13.652Z" },
@@ -4364,36 +5351,69 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/c0/28/26534bed77109632a956977f60d8519049f545abc39215d086e33a61f1f2/pyyaml_ft-8.0.0-cp313-cp313t-win_amd64.whl", hash = "sha256:de04cfe9439565e32f178106c51dd6ca61afaa2907d143835d501d84703d3793", size = 171579, upload-time = "2025-06-10T15:32:14.34Z" },
]
+[[package]]
+name = "quart"
+version = "0.20.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiofiles" },
+ { name = "blinker" },
+ { name = "click" },
+ { name = "flask" },
+ { name = "hypercorn" },
+ { name = "itsdangerous" },
+ { name = "jinja2" },
+ { name = "markupsafe" },
+ { name = "werkzeug" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/1d/9d/12e1143a5bd2ccc05c293a6f5ae1df8fd94a8fc1440ecc6c344b2b30ce13/quart-0.20.0.tar.gz", hash = "sha256:08793c206ff832483586f5ae47018c7e40bdd75d886fee3fabbdaa70c2cf505d", size = 63874, upload-time = "2024-12-23T13:53:05.664Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7e/e9/cc28f21f52913adf333f653b9e0a3bf9cb223f5083a26422968ba73edd8d/quart-0.20.0-py3-none-any.whl", hash = "sha256:003c08f551746710acb757de49d9b768986fd431517d0eb127380b656b98b8f1", size = 77960, upload-time = "2024-12-23T13:53:02.842Z" },
+]
+
[[package]]
name = "ray"
-version = "2.53.0"
+version = "2.54.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "click" },
{ name = "filelock" },
{ name = "jsonschema" },
{ name = "msgpack" },
- { name = "packaging" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
{ name = "protobuf" },
{ name = "pyyaml" },
{ name = "requests" },
]
wheels = [
- { url = "https://files.pythonhosted.org/packages/2f/99/21986c7f8135dafbf7c49229c52faaa9d2d365db7d86fffe978dde8ee967/ray-2.53.0-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:4db914a0a6dd608fa49c066929a1282745a2dbd73caee67d7b80fe684ca65bdd", size = 69473649, upload-time = "2025-12-20T16:05:40.58Z" },
- { url = "https://files.pythonhosted.org/packages/70/d9/58b5426a3f11993851db3c93841358cebdddd948153481d355b720f31f9d/ray-2.53.0-cp310-cp310-manylinux2014_aarch64.whl", hash = "sha256:4108280d8a1cb90d7d68e5c954c35e63b8bb9a4ba15f88c5e7da0e2025647712", size = 71342662, upload-time = "2025-12-20T16:05:46.936Z" },
- { url = "https://files.pythonhosted.org/packages/c5/05/4aa32370b313481c2d1d41cb53ec786daebdb2ef665b01ef2ac43d9cf457/ray-2.53.0-cp310-cp310-manylinux2014_x86_64.whl", hash = "sha256:4dbb5fce1364763f29741055f50abe33cf726397141f9cc0e845dd3cc963e455", size = 72188620, upload-time = "2025-12-20T16:05:52.817Z" },
- { url = "https://files.pythonhosted.org/packages/f7/c6/21efe5886898421df20078a333b0984eade7d7aa4bdc68a336f0c66db27e/ray-2.53.0-cp310-cp310-win_amd64.whl", hash = "sha256:90faf630d20b6abf3135997fb3edb5842134aff92e04ee709865db04816d97ef", size = 27200553, upload-time = "2025-12-20T16:05:57.655Z" },
- { url = "https://files.pythonhosted.org/packages/bf/64/d5c29a4b014d8b9a624203a88b67630072c1d6960425dbf7a1f0fa5d6b74/ray-2.53.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:bd3ec4c342776ddac23ae2b108c64f5939f417ccc4875900d586c7c978463269", size = 69479296, upload-time = "2025-12-20T16:06:05.111Z" },
- { url = "https://files.pythonhosted.org/packages/c6/41/9e19d1e5d9458a5ba157c36642e2874bcb22fddbd7c1e77b668e5afc3f3d/ray-2.53.0-cp311-cp311-manylinux2014_aarch64.whl", hash = "sha256:a0bbb98b0b0f25a3ee075ca10171e1260e70b6bc690cd509ecd7ce1228af854d", size = 71463449, upload-time = "2025-12-20T16:06:10.983Z" },
- { url = "https://files.pythonhosted.org/packages/63/de/58c19906b0dd16ea06b4f2465b7327f5f180e6b6e1c8c9b610d7c589ea5f/ray-2.53.0-cp311-cp311-manylinux2014_x86_64.whl", hash = "sha256:eb000c17f7301071fdd15c44c4cd3ac0f7953bb4c7c227e61719fe7048195bcd", size = 72305102, upload-time = "2025-12-20T16:06:17.989Z" },
- { url = "https://files.pythonhosted.org/packages/b1/43/72cc1cfe17d26abe62a793eab10445f9546dce24192b85a6cd0cdc47ed86/ray-2.53.0-cp311-cp311-win_amd64.whl", hash = "sha256:4a1bb3fe09ab4cd0d16ddc96b9f60c9ed83b3f93b87aa8506e0d3b746fd4e825", size = 27194174, upload-time = "2025-12-20T16:06:23.042Z" },
- { url = "https://files.pythonhosted.org/packages/b2/44/562718a634e63e8ef7985285288a167d4af62bc2a7decce3300cf937776a/ray-2.53.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:d8b95d047d947493803fb8417aea31225dcacdab15afdc75b8a238901949d457", size = 69463763, upload-time = "2025-12-20T16:06:28.685Z" },
- { url = "https://files.pythonhosted.org/packages/38/68/8e59b8413f3751fe7ce8b98ee8787d13964b47a4043587950790a9dd2151/ray-2.53.0-cp312-cp312-manylinux2014_aarch64.whl", hash = "sha256:65e2ce58d3dc6baa3cf45824d889c1968ebde565ee54dfd80a98af8f31af8e4a", size = 71504450, upload-time = "2025-12-20T16:06:34.922Z" },
- { url = "https://files.pythonhosted.org/packages/2a/db/978a50d264565ca42e2a4bf115ec9a1f04f19ca5e620e6aa2f280747b644/ray-2.53.0-cp312-cp312-manylinux2014_x86_64.whl", hash = "sha256:14f46363e9b4cf0c1c8b4d8623ec337c5bd408377831b5e5b50067930137bbca", size = 72370424, upload-time = "2025-12-20T16:06:40.821Z" },
- { url = "https://files.pythonhosted.org/packages/8d/6c/bba6f22a9d83ee8f236000ba315f0c197bdc79888b4fa42fd762f729cbbd/ray-2.53.0-cp312-cp312-win_amd64.whl", hash = "sha256:b828c147f9ff2f277b1d254e4fe9a746fdfaee7e313a93a97c7edf4dae9b81a4", size = 27178106, upload-time = "2025-12-20T16:06:45.594Z" },
- { url = "https://files.pythonhosted.org/packages/3d/38/450cf9cf3c490fa4cc6d470597f819444da60f85579d2b34b95ee79fcb6f/ray-2.53.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:85b472ab6fb8f1189f8cef81913fd91b24dd69b3fa7dcca7e144827bd924f6c0", size = 69409819, upload-time = "2025-12-20T16:06:50.668Z" },
- { url = "https://files.pythonhosted.org/packages/71/5e/d452970b07174d5e4f8688abae889d01321b51ced827db1f1d1cb7d56d44/ray-2.53.0-cp313-cp313-manylinux2014_aarch64.whl", hash = "sha256:7196e5358dfcc8211be864f45e6dfe4827202df294af3c7a76ff8fbc080e0522", size = 71409529, upload-time = "2025-12-20T16:06:56.2Z" },
- { url = "https://files.pythonhosted.org/packages/cb/84/50b317a125617a638a64694c12f56183edd5df01828a35fa4c55c7b13c66/ray-2.53.0-cp313-cp313-manylinux2014_x86_64.whl", hash = "sha256:73dbbaa7962a7f5e38aa8cf9483e0e9817205e989aa3dc859c738c2af1ae01df", size = 72283961, upload-time = "2025-12-20T16:07:05.831Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/ca/f3274e0d513c44949ea9167c12c07f9971e5f25ef22b698448a6ca831434/ray-2.54.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:645ebfb73cfd32bd510a05ed9f2738a18d6db69929cae9701d749f2740dbfd9a", size = 70076126, upload-time = "2026-03-25T22:40:55.188Z" },
+ { url = "https://files.pythonhosted.org/packages/51/6f/bf1b7a6d4424c19add99eb17398c7522473502193540b679f8b94fbf2d72/ray-2.54.1-cp312-cp312-manylinux2014_aarch64.whl", hash = "sha256:cd452b61ae2e0daf9271f5a554614397429cc2731681bae10fe72316dadc2749", size = 71831684, upload-time = "2026-03-25T22:41:01.356Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/1f/b33d5006823f8c1c8760887cf1190194f4b06de858b3d17e37bd930a6a62/ray-2.54.1-cp312-cp312-manylinux2014_x86_64.whl", hash = "sha256:4c6f7e23dda62a32f94083141c3f97e9c4246e3ae4ae2bc488bcd8fd0311f54a", size = 72688748, upload-time = "2026-03-25T22:41:07.43Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/7d/02b46d3fe644e1feef62b9e4ebf8cbfc17c6b2d283763208abc52c3dc85e/ray-2.54.1-cp312-cp312-win_amd64.whl", hash = "sha256:673a895c0c4a716ed772552baa3f5b8d7d1f7a4b34e04787fdfe6fe3049ed0d8", size = 27427871, upload-time = "2026-03-25T22:41:12.485Z" },
+ { url = "https://files.pythonhosted.org/packages/80/30/90f9f8f0fcba72b898c40854e020c9d5330f33b4ccd711747cc07e061416/ray-2.54.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:d05f477d1518a00fd5880644e889a7a3eaf64ae5d1f8f239a682d052ad2a383d", size = 70023037, upload-time = "2026-03-25T22:41:17.895Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/5d/fe0e8ac47f6b362c81f391d7f8d2a6858d0bafcc2c37631dc5cc04a16545/ray-2.54.1-cp313-cp313-manylinux2014_aarch64.whl", hash = "sha256:2766f0230806480c38a9a94502087f1d4aea919f38521a28781690613b0290a4", size = 71738623, upload-time = "2026-03-25T22:41:23.898Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/22/48008a626e719baee2012080b960687cc6417b572b363c1c29fe23d119c3/ray-2.54.1-cp313-cp313-manylinux2014_x86_64.whl", hash = "sha256:0c3ae2943176e7b239c78b825a5b2bf4135d90280083a0e19c0a75a5db4d836f", size = 72603355, upload-time = "2026-03-25T22:41:29.802Z" },
+]
+
+[package.optional-dependencies]
+default = [
+ { name = "aiohttp" },
+ { name = "aiohttp-cors" },
+ { name = "colorful" },
+ { name = "grpcio" },
+ { name = "opencensus" },
+ { name = "opentelemetry-exporter-prometheus", version = "0.54b1", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
+ { name = "opentelemetry-exporter-prometheus", version = "0.61b0", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts')" },
+ { name = "opentelemetry-proto" },
+ { name = "opentelemetry-sdk", version = "1.33.1", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
+ { name = "opentelemetry-sdk", version = "1.40.0", source = { registry = "https://pypi.org/simple" }, marker = "(extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts')" },
+ { name = "prometheus-client" },
+ { name = "py-spy" },
+ { name = "pydantic" },
+ { name = "requests" },
+ { name = "smart-open" },
+ { name = "virtualenv" },
]
[[package]]
@@ -4412,114 +5432,95 @@ wheels = [
[[package]]
name = "regex"
-version = "2025.11.3"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/cc/a9/546676f25e573a4cf00fe8e119b78a37b6a8fe2dc95cda877b30889c9c45/regex-2025.11.3.tar.gz", hash = "sha256:1fedc720f9bb2494ce31a58a1631f9c82df6a09b49c19517ea5cc280b4541e01", size = 414669, upload-time = "2025-11-03T21:34:22.089Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/8a/d6/d788d52da01280a30a3f6268aef2aa71043bff359c618fea4c5b536654d5/regex-2025.11.3-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:2b441a4ae2c8049106e8b39973bfbddfb25a179dda2bdb99b0eeb60c40a6a3af", size = 488087, upload-time = "2025-11-03T21:30:47.317Z" },
- { url = "https://files.pythonhosted.org/packages/69/39/abec3bd688ec9bbea3562de0fd764ff802976185f5ff22807bf0a2697992/regex-2025.11.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:2fa2eed3f76677777345d2f81ee89f5de2f5745910e805f7af7386a920fa7313", size = 290544, upload-time = "2025-11-03T21:30:49.912Z" },
- { url = "https://files.pythonhosted.org/packages/39/b3/9a231475d5653e60002508f41205c61684bb2ffbf2401351ae2186897fc4/regex-2025.11.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:d8b4a27eebd684319bdf473d39f1d79eed36bf2cd34bd4465cdb4618d82b3d56", size = 288408, upload-time = "2025-11-03T21:30:51.344Z" },
- { url = "https://files.pythonhosted.org/packages/c3/c5/1929a0491bd5ac2d1539a866768b88965fa8c405f3e16a8cef84313098d6/regex-2025.11.3-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5cf77eac15bd264986c4a2c63353212c095b40f3affb2bc6b4ef80c4776c1a28", size = 781584, upload-time = "2025-11-03T21:30:52.596Z" },
- { url = "https://files.pythonhosted.org/packages/ce/fd/16aa16cf5d497ef727ec966f74164fbe75d6516d3d58ac9aa989bc9cdaad/regex-2025.11.3-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b7f9ee819f94c6abfa56ec7b1dbab586f41ebbdc0a57e6524bd5e7f487a878c7", size = 850733, upload-time = "2025-11-03T21:30:53.825Z" },
- { url = "https://files.pythonhosted.org/packages/e6/49/3294b988855a221cb6565189edf5dc43239957427df2d81d4a6b15244f64/regex-2025.11.3-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:838441333bc90b829406d4a03cb4b8bf7656231b84358628b0406d803931ef32", size = 898691, upload-time = "2025-11-03T21:30:55.575Z" },
- { url = "https://files.pythonhosted.org/packages/14/62/b56d29e70b03666193369bdbdedfdc23946dbe9f81dd78ce262c74d988ab/regex-2025.11.3-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:cfe6d3f0c9e3b7e8c0c694b24d25e677776f5ca26dce46fd6b0489f9c8339391", size = 791662, upload-time = "2025-11-03T21:30:57.262Z" },
- { url = "https://files.pythonhosted.org/packages/15/fc/e4c31d061eced63fbf1ce9d853975f912c61a7d406ea14eda2dd355f48e7/regex-2025.11.3-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2ab815eb8a96379a27c3b6157fcb127c8f59c36f043c1678110cea492868f1d5", size = 782587, upload-time = "2025-11-03T21:30:58.788Z" },
- { url = "https://files.pythonhosted.org/packages/b2/bb/5e30c7394bcf63f0537121c23e796be67b55a8847c3956ae6068f4c70702/regex-2025.11.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:728a9d2d173a65b62bdc380b7932dd8e74ed4295279a8fe1021204ce210803e7", size = 774709, upload-time = "2025-11-03T21:31:00.081Z" },
- { url = "https://files.pythonhosted.org/packages/c5/c4/fce773710af81b0cb37cb4ff0947e75d5d17dee304b93d940b87a67fc2f4/regex-2025.11.3-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:509dc827f89c15c66a0c216331260d777dd6c81e9a4e4f830e662b0bb296c313", size = 845773, upload-time = "2025-11-03T21:31:01.583Z" },
- { url = "https://files.pythonhosted.org/packages/7b/5e/9466a7ec4b8ec282077095c6eb50a12a389d2e036581134d4919e8ca518c/regex-2025.11.3-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:849202cd789e5f3cf5dcc7822c34b502181b4824a65ff20ce82da5524e45e8e9", size = 836164, upload-time = "2025-11-03T21:31:03.244Z" },
- { url = "https://files.pythonhosted.org/packages/95/18/82980a60e8ed1594eb3c89eb814fb276ef51b9af7caeab1340bfd8564af6/regex-2025.11.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:b6f78f98741dcc89607c16b1e9426ee46ce4bf31ac5e6b0d40e81c89f3481ea5", size = 779832, upload-time = "2025-11-03T21:31:04.876Z" },
- { url = "https://files.pythonhosted.org/packages/03/cc/90ab0fdbe6dce064a42015433f9152710139fb04a8b81b4fb57a1cb63ffa/regex-2025.11.3-cp310-cp310-win32.whl", hash = "sha256:149eb0bba95231fb4f6d37c8f760ec9fa6fabf65bab555e128dde5f2475193ec", size = 265802, upload-time = "2025-11-03T21:31:06.581Z" },
- { url = "https://files.pythonhosted.org/packages/34/9d/e9e8493a85f3b1ddc4a5014465f5c2b78c3ea1cbf238dcfde78956378041/regex-2025.11.3-cp310-cp310-win_amd64.whl", hash = "sha256:ee3a83ce492074c35a74cc76cf8235d49e77b757193a5365ff86e3f2f93db9fd", size = 277722, upload-time = "2025-11-03T21:31:08.144Z" },
- { url = "https://files.pythonhosted.org/packages/15/c4/b54b24f553966564506dbf873a3e080aef47b356a3b39b5d5aba992b50db/regex-2025.11.3-cp310-cp310-win_arm64.whl", hash = "sha256:38af559ad934a7b35147716655d4a2f79fcef2d695ddfe06a06ba40ae631fa7e", size = 270289, upload-time = "2025-11-03T21:31:10.267Z" },
- { url = "https://files.pythonhosted.org/packages/f7/90/4fb5056e5f03a7048abd2b11f598d464f0c167de4f2a51aa868c376b8c70/regex-2025.11.3-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:eadade04221641516fa25139273505a1c19f9bf97589a05bc4cfcd8b4a618031", size = 488081, upload-time = "2025-11-03T21:31:11.946Z" },
- { url = "https://files.pythonhosted.org/packages/85/23/63e481293fac8b069d84fba0299b6666df720d875110efd0338406b5d360/regex-2025.11.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:feff9e54ec0dd3833d659257f5c3f5322a12eee58ffa360984b716f8b92983f4", size = 290554, upload-time = "2025-11-03T21:31:13.387Z" },
- { url = "https://files.pythonhosted.org/packages/2b/9d/b101d0262ea293a0066b4522dfb722eb6a8785a8c3e084396a5f2c431a46/regex-2025.11.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:3b30bc921d50365775c09a7ed446359e5c0179e9e2512beec4a60cbcef6ddd50", size = 288407, upload-time = "2025-11-03T21:31:14.809Z" },
- { url = "https://files.pythonhosted.org/packages/0c/64/79241c8209d5b7e00577ec9dca35cd493cc6be35b7d147eda367d6179f6d/regex-2025.11.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f99be08cfead2020c7ca6e396c13543baea32343b7a9a5780c462e323bd8872f", size = 793418, upload-time = "2025-11-03T21:31:16.556Z" },
- { url = "https://files.pythonhosted.org/packages/3d/e2/23cd5d3573901ce8f9757c92ca4db4d09600b865919b6d3e7f69f03b1afd/regex-2025.11.3-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6dd329a1b61c0ee95ba95385fb0c07ea0d3fe1a21e1349fa2bec272636217118", size = 860448, upload-time = "2025-11-03T21:31:18.12Z" },
- { url = "https://files.pythonhosted.org/packages/2a/4c/aecf31beeaa416d0ae4ecb852148d38db35391aac19c687b5d56aedf3a8b/regex-2025.11.3-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4c5238d32f3c5269d9e87be0cf096437b7622b6920f5eac4fd202468aaeb34d2", size = 907139, upload-time = "2025-11-03T21:31:20.753Z" },
- { url = "https://files.pythonhosted.org/packages/61/22/b8cb00df7d2b5e0875f60628594d44dba283e951b1ae17c12f99e332cc0a/regex-2025.11.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:10483eefbfb0adb18ee9474498c9a32fcf4e594fbca0543bb94c48bac6183e2e", size = 800439, upload-time = "2025-11-03T21:31:22.069Z" },
- { url = "https://files.pythonhosted.org/packages/02/a8/c4b20330a5cdc7a8eb265f9ce593f389a6a88a0c5f280cf4d978f33966bc/regex-2025.11.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:78c2d02bb6e1da0720eedc0bad578049cad3f71050ef8cd065ecc87691bed2b0", size = 782965, upload-time = "2025-11-03T21:31:23.598Z" },
- { url = "https://files.pythonhosted.org/packages/b4/4c/ae3e52988ae74af4b04d2af32fee4e8077f26e51b62ec2d12d246876bea2/regex-2025.11.3-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:e6b49cd2aad93a1790ce9cffb18964f6d3a4b0b3dbdbd5de094b65296fce6e58", size = 854398, upload-time = "2025-11-03T21:31:25.008Z" },
- { url = "https://files.pythonhosted.org/packages/06/d1/a8b9cf45874eda14b2e275157ce3b304c87e10fb38d9fc26a6e14eb18227/regex-2025.11.3-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:885b26aa3ee56433b630502dc3d36ba78d186a00cc535d3806e6bfd9ed3c70ab", size = 845897, upload-time = "2025-11-03T21:31:26.427Z" },
- { url = "https://files.pythonhosted.org/packages/ea/fe/1830eb0236be93d9b145e0bd8ab499f31602fe0999b1f19e99955aa8fe20/regex-2025.11.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:ddd76a9f58e6a00f8772e72cff8ebcff78e022be95edf018766707c730593e1e", size = 788906, upload-time = "2025-11-03T21:31:28.078Z" },
- { url = "https://files.pythonhosted.org/packages/66/47/dc2577c1f95f188c1e13e2e69d8825a5ac582ac709942f8a03af42ed6e93/regex-2025.11.3-cp311-cp311-win32.whl", hash = "sha256:3e816cc9aac1cd3cc9a4ec4d860f06d40f994b5c7b4d03b93345f44e08cc68bf", size = 265812, upload-time = "2025-11-03T21:31:29.72Z" },
- { url = "https://files.pythonhosted.org/packages/50/1e/15f08b2f82a9bbb510621ec9042547b54d11e83cb620643ebb54e4eb7d71/regex-2025.11.3-cp311-cp311-win_amd64.whl", hash = "sha256:087511f5c8b7dfbe3a03f5d5ad0c2a33861b1fc387f21f6f60825a44865a385a", size = 277737, upload-time = "2025-11-03T21:31:31.422Z" },
- { url = "https://files.pythonhosted.org/packages/f4/fc/6500eb39f5f76c5e47a398df82e6b535a5e345f839581012a418b16f9cc3/regex-2025.11.3-cp311-cp311-win_arm64.whl", hash = "sha256:1ff0d190c7f68ae7769cd0313fe45820ba07ffebfddfaa89cc1eb70827ba0ddc", size = 270290, upload-time = "2025-11-03T21:31:33.041Z" },
- { url = "https://files.pythonhosted.org/packages/e8/74/18f04cb53e58e3fb107439699bd8375cf5a835eec81084e0bddbd122e4c2/regex-2025.11.3-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:bc8ab71e2e31b16e40868a40a69007bc305e1109bd4658eb6cad007e0bf67c41", size = 489312, upload-time = "2025-11-03T21:31:34.343Z" },
- { url = "https://files.pythonhosted.org/packages/78/3f/37fcdd0d2b1e78909108a876580485ea37c91e1acf66d3bb8e736348f441/regex-2025.11.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:22b29dda7e1f7062a52359fca6e58e548e28c6686f205e780b02ad8ef710de36", size = 291256, upload-time = "2025-11-03T21:31:35.675Z" },
- { url = "https://files.pythonhosted.org/packages/bf/26/0a575f58eb23b7ebd67a45fccbc02ac030b737b896b7e7a909ffe43ffd6a/regex-2025.11.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:3a91e4a29938bc1a082cc28fdea44be420bf2bebe2665343029723892eb073e1", size = 288921, upload-time = "2025-11-03T21:31:37.07Z" },
- { url = "https://files.pythonhosted.org/packages/ea/98/6a8dff667d1af907150432cf5abc05a17ccd32c72a3615410d5365ac167a/regex-2025.11.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:08b884f4226602ad40c5d55f52bf91a9df30f513864e0054bad40c0e9cf1afb7", size = 798568, upload-time = "2025-11-03T21:31:38.784Z" },
- { url = "https://files.pythonhosted.org/packages/64/15/92c1db4fa4e12733dd5a526c2dd2b6edcbfe13257e135fc0f6c57f34c173/regex-2025.11.3-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:3e0b11b2b2433d1c39c7c7a30e3f3d0aeeea44c2a8d0bae28f6b95f639927a69", size = 864165, upload-time = "2025-11-03T21:31:40.559Z" },
- { url = "https://files.pythonhosted.org/packages/f9/e7/3ad7da8cdee1ce66c7cd37ab5ab05c463a86ffeb52b1a25fe7bd9293b36c/regex-2025.11.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:87eb52a81ef58c7ba4d45c3ca74e12aa4b4e77816f72ca25258a85b3ea96cb48", size = 912182, upload-time = "2025-11-03T21:31:42.002Z" },
- { url = "https://files.pythonhosted.org/packages/84/bd/9ce9f629fcb714ffc2c3faf62b6766ecb7a585e1e885eb699bcf130a5209/regex-2025.11.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a12ab1f5c29b4e93db518f5e3872116b7e9b1646c9f9f426f777b50d44a09e8c", size = 803501, upload-time = "2025-11-03T21:31:43.815Z" },
- { url = "https://files.pythonhosted.org/packages/7c/0f/8dc2e4349d8e877283e6edd6c12bdcebc20f03744e86f197ab6e4492bf08/regex-2025.11.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:7521684c8c7c4f6e88e35ec89680ee1aa8358d3f09d27dfbdf62c446f5d4c695", size = 787842, upload-time = "2025-11-03T21:31:45.353Z" },
- { url = "https://files.pythonhosted.org/packages/f9/73/cff02702960bc185164d5619c0c62a2f598a6abff6695d391b096237d4ab/regex-2025.11.3-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:7fe6e5440584e94cc4b3f5f4d98a25e29ca12dccf8873679a635638349831b98", size = 858519, upload-time = "2025-11-03T21:31:46.814Z" },
- { url = "https://files.pythonhosted.org/packages/61/83/0e8d1ae71e15bc1dc36231c90b46ee35f9d52fab2e226b0e039e7ea9c10a/regex-2025.11.3-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:8e026094aa12b43f4fd74576714e987803a315c76edb6b098b9809db5de58f74", size = 850611, upload-time = "2025-11-03T21:31:48.289Z" },
- { url = "https://files.pythonhosted.org/packages/c8/f5/70a5cdd781dcfaa12556f2955bf170cd603cb1c96a1827479f8faea2df97/regex-2025.11.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:435bbad13e57eb5606a68443af62bed3556de2f46deb9f7d4237bc2f1c9fb3a0", size = 789759, upload-time = "2025-11-03T21:31:49.759Z" },
- { url = "https://files.pythonhosted.org/packages/59/9b/7c29be7903c318488983e7d97abcf8ebd3830e4c956c4c540005fcfb0462/regex-2025.11.3-cp312-cp312-win32.whl", hash = "sha256:3839967cf4dc4b985e1570fd8d91078f0c519f30491c60f9ac42a8db039be204", size = 266194, upload-time = "2025-11-03T21:31:51.53Z" },
- { url = "https://files.pythonhosted.org/packages/1a/67/3b92df89f179d7c367be654ab5626ae311cb28f7d5c237b6bb976cd5fbbb/regex-2025.11.3-cp312-cp312-win_amd64.whl", hash = "sha256:e721d1b46e25c481dc5ded6f4b3f66c897c58d2e8cfdf77bbced84339108b0b9", size = 277069, upload-time = "2025-11-03T21:31:53.151Z" },
- { url = "https://files.pythonhosted.org/packages/d7/55/85ba4c066fe5094d35b249c3ce8df0ba623cfd35afb22d6764f23a52a1c5/regex-2025.11.3-cp312-cp312-win_arm64.whl", hash = "sha256:64350685ff08b1d3a6fff33f45a9ca183dc1d58bbfe4981604e70ec9801bbc26", size = 270330, upload-time = "2025-11-03T21:31:54.514Z" },
- { url = "https://files.pythonhosted.org/packages/e1/a7/dda24ebd49da46a197436ad96378f17df30ceb40e52e859fc42cac45b850/regex-2025.11.3-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:c1e448051717a334891f2b9a620fe36776ebf3dd8ec46a0b877c8ae69575feb4", size = 489081, upload-time = "2025-11-03T21:31:55.9Z" },
- { url = "https://files.pythonhosted.org/packages/19/22/af2dc751aacf88089836aa088a1a11c4f21a04707eb1b0478e8e8fb32847/regex-2025.11.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:9b5aca4d5dfd7fbfbfbdaf44850fcc7709a01146a797536a8f84952e940cca76", size = 291123, upload-time = "2025-11-03T21:31:57.758Z" },
- { url = "https://files.pythonhosted.org/packages/a3/88/1a3ea5672f4b0a84802ee9891b86743438e7c04eb0b8f8c4e16a42375327/regex-2025.11.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:04d2765516395cf7dda331a244a3282c0f5ae96075f728629287dfa6f76ba70a", size = 288814, upload-time = "2025-11-03T21:32:01.12Z" },
- { url = "https://files.pythonhosted.org/packages/fb/8c/f5987895bf42b8ddeea1b315c9fedcfe07cadee28b9c98cf50d00adcb14d/regex-2025.11.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5d9903ca42bfeec4cebedba8022a7c97ad2aab22e09573ce9976ba01b65e4361", size = 798592, upload-time = "2025-11-03T21:32:03.006Z" },
- { url = "https://files.pythonhosted.org/packages/99/2a/6591ebeede78203fa77ee46a1c36649e02df9eaa77a033d1ccdf2fcd5d4e/regex-2025.11.3-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:639431bdc89d6429f6721625e8129413980ccd62e9d3f496be618a41d205f160", size = 864122, upload-time = "2025-11-03T21:32:04.553Z" },
- { url = "https://files.pythonhosted.org/packages/94/d6/be32a87cf28cf8ed064ff281cfbd49aefd90242a83e4b08b5a86b38e8eb4/regex-2025.11.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f117efad42068f9715677c8523ed2be1518116d1c49b1dd17987716695181efe", size = 912272, upload-time = "2025-11-03T21:32:06.148Z" },
- { url = "https://files.pythonhosted.org/packages/62/11/9bcef2d1445665b180ac7f230406ad80671f0fc2a6ffb93493b5dd8cd64c/regex-2025.11.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4aecb6f461316adf9f1f0f6a4a1a3d79e045f9b71ec76055a791affa3b285850", size = 803497, upload-time = "2025-11-03T21:32:08.162Z" },
- { url = "https://files.pythonhosted.org/packages/e5/a7/da0dc273d57f560399aa16d8a68ae7f9b57679476fc7ace46501d455fe84/regex-2025.11.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3b3a5f320136873cc5561098dfab677eea139521cb9a9e8db98b7e64aef44cbc", size = 787892, upload-time = "2025-11-03T21:32:09.769Z" },
- { url = "https://files.pythonhosted.org/packages/da/4b/732a0c5a9736a0b8d6d720d4945a2f1e6f38f87f48f3173559f53e8d5d82/regex-2025.11.3-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:75fa6f0056e7efb1f42a1c34e58be24072cb9e61a601340cc1196ae92326a4f9", size = 858462, upload-time = "2025-11-03T21:32:11.769Z" },
- { url = "https://files.pythonhosted.org/packages/0c/f5/a2a03df27dc4c2d0c769220f5110ba8c4084b0bfa9ab0f9b4fcfa3d2b0fc/regex-2025.11.3-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:dbe6095001465294f13f1adcd3311e50dd84e5a71525f20a10bd16689c61ce0b", size = 850528, upload-time = "2025-11-03T21:32:13.906Z" },
- { url = "https://files.pythonhosted.org/packages/d6/09/e1cd5bee3841c7f6eb37d95ca91cdee7100b8f88b81e41c2ef426910891a/regex-2025.11.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:454d9b4ae7881afbc25015b8627c16d88a597479b9dea82b8c6e7e2e07240dc7", size = 789866, upload-time = "2025-11-03T21:32:15.748Z" },
- { url = "https://files.pythonhosted.org/packages/eb/51/702f5ea74e2a9c13d855a6a85b7f80c30f9e72a95493260193c07f3f8d74/regex-2025.11.3-cp313-cp313-win32.whl", hash = "sha256:28ba4d69171fc6e9896337d4fc63a43660002b7da53fc15ac992abcf3410917c", size = 266189, upload-time = "2025-11-03T21:32:17.493Z" },
- { url = "https://files.pythonhosted.org/packages/8b/00/6e29bb314e271a743170e53649db0fdb8e8ff0b64b4f425f5602f4eb9014/regex-2025.11.3-cp313-cp313-win_amd64.whl", hash = "sha256:bac4200befe50c670c405dc33af26dad5a3b6b255dd6c000d92fe4629f9ed6a5", size = 277054, upload-time = "2025-11-03T21:32:19.042Z" },
- { url = "https://files.pythonhosted.org/packages/25/f1/b156ff9f2ec9ac441710764dda95e4edaf5f36aca48246d1eea3f1fd96ec/regex-2025.11.3-cp313-cp313-win_arm64.whl", hash = "sha256:2292cd5a90dab247f9abe892ac584cb24f0f54680c73fcb4a7493c66c2bf2467", size = 270325, upload-time = "2025-11-03T21:32:21.338Z" },
- { url = "https://files.pythonhosted.org/packages/20/28/fd0c63357caefe5680b8ea052131acbd7f456893b69cc2a90cc3e0dc90d4/regex-2025.11.3-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:1eb1ebf6822b756c723e09f5186473d93236c06c579d2cc0671a722d2ab14281", size = 491984, upload-time = "2025-11-03T21:32:23.466Z" },
- { url = "https://files.pythonhosted.org/packages/df/ec/7014c15626ab46b902b3bcc4b28a7bae46d8f281fc7ea9c95e22fcaaa917/regex-2025.11.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:1e00ec2970aab10dc5db34af535f21fcf32b4a31d99e34963419636e2f85ae39", size = 292673, upload-time = "2025-11-03T21:32:25.034Z" },
- { url = "https://files.pythonhosted.org/packages/23/ab/3b952ff7239f20d05f1f99e9e20188513905f218c81d52fb5e78d2bf7634/regex-2025.11.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:a4cb042b615245d5ff9b3794f56be4138b5adc35a4166014d31d1814744148c7", size = 291029, upload-time = "2025-11-03T21:32:26.528Z" },
- { url = "https://files.pythonhosted.org/packages/21/7e/3dc2749fc684f455f162dcafb8a187b559e2614f3826877d3844a131f37b/regex-2025.11.3-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:44f264d4bf02f3176467d90b294d59bf1db9fe53c141ff772f27a8b456b2a9ed", size = 807437, upload-time = "2025-11-03T21:32:28.363Z" },
- { url = "https://files.pythonhosted.org/packages/1b/0b/d529a85ab349c6a25d1ca783235b6e3eedf187247eab536797021f7126c6/regex-2025.11.3-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7be0277469bf3bd7a34a9c57c1b6a724532a0d235cd0dc4e7f4316f982c28b19", size = 873368, upload-time = "2025-11-03T21:32:30.4Z" },
- { url = "https://files.pythonhosted.org/packages/7d/18/2d868155f8c9e3e9d8f9e10c64e9a9f496bb8f7e037a88a8bed26b435af6/regex-2025.11.3-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:0d31e08426ff4b5b650f68839f5af51a92a5b51abd8554a60c2fbc7c71f25d0b", size = 914921, upload-time = "2025-11-03T21:32:32.123Z" },
- { url = "https://files.pythonhosted.org/packages/2d/71/9d72ff0f354fa783fe2ba913c8734c3b433b86406117a8db4ea2bf1c7a2f/regex-2025.11.3-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e43586ce5bd28f9f285a6e729466841368c4a0353f6fd08d4ce4630843d3648a", size = 812708, upload-time = "2025-11-03T21:32:34.305Z" },
- { url = "https://files.pythonhosted.org/packages/e7/19/ce4bf7f5575c97f82b6e804ffb5c4e940c62609ab2a0d9538d47a7fdf7d4/regex-2025.11.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:0f9397d561a4c16829d4e6ff75202c1c08b68a3bdbfe29dbfcdb31c9830907c6", size = 795472, upload-time = "2025-11-03T21:32:36.364Z" },
- { url = "https://files.pythonhosted.org/packages/03/86/fd1063a176ffb7b2315f9a1b08d17b18118b28d9df163132615b835a26ee/regex-2025.11.3-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:dd16e78eb18ffdb25ee33a0682d17912e8cc8a770e885aeee95020046128f1ce", size = 868341, upload-time = "2025-11-03T21:32:38.042Z" },
- { url = "https://files.pythonhosted.org/packages/12/43/103fb2e9811205e7386366501bc866a164a0430c79dd59eac886a2822950/regex-2025.11.3-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:ffcca5b9efe948ba0661e9df0fa50d2bc4b097c70b9810212d6b62f05d83b2dd", size = 854666, upload-time = "2025-11-03T21:32:40.079Z" },
- { url = "https://files.pythonhosted.org/packages/7d/22/e392e53f3869b75804762c7c848bd2dd2abf2b70fb0e526f58724638bd35/regex-2025.11.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:c56b4d162ca2b43318ac671c65bd4d563e841a694ac70e1a976ac38fcf4ca1d2", size = 799473, upload-time = "2025-11-03T21:32:42.148Z" },
- { url = "https://files.pythonhosted.org/packages/4f/f9/8bd6b656592f925b6845fcbb4d57603a3ac2fb2373344ffa1ed70aa6820a/regex-2025.11.3-cp313-cp313t-win32.whl", hash = "sha256:9ddc42e68114e161e51e272f667d640f97e84a2b9ef14b7477c53aac20c2d59a", size = 268792, upload-time = "2025-11-03T21:32:44.13Z" },
- { url = "https://files.pythonhosted.org/packages/e5/87/0e7d603467775ff65cd2aeabf1b5b50cc1c3708556a8b849a2fa4dd1542b/regex-2025.11.3-cp313-cp313t-win_amd64.whl", hash = "sha256:7a7c7fdf755032ffdd72c77e3d8096bdcb0eb92e89e17571a196f03d88b11b3c", size = 280214, upload-time = "2025-11-03T21:32:45.853Z" },
- { url = "https://files.pythonhosted.org/packages/8d/d0/2afc6f8e94e2b64bfb738a7c2b6387ac1699f09f032d363ed9447fd2bb57/regex-2025.11.3-cp313-cp313t-win_arm64.whl", hash = "sha256:df9eb838c44f570283712e7cff14c16329a9f0fb19ca492d21d4b7528ee6821e", size = 271469, upload-time = "2025-11-03T21:32:48.026Z" },
- { url = "https://files.pythonhosted.org/packages/31/e9/f6e13de7e0983837f7b6d238ad9458800a874bf37c264f7923e63409944c/regex-2025.11.3-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:9697a52e57576c83139d7c6f213d64485d3df5bf84807c35fa409e6c970801c6", size = 489089, upload-time = "2025-11-03T21:32:50.027Z" },
- { url = "https://files.pythonhosted.org/packages/a3/5c/261f4a262f1fa65141c1b74b255988bd2fa020cc599e53b080667d591cfc/regex-2025.11.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:e18bc3f73bd41243c9b38a6d9f2366cd0e0137a9aebe2d8ff76c5b67d4c0a3f4", size = 291059, upload-time = "2025-11-03T21:32:51.682Z" },
- { url = "https://files.pythonhosted.org/packages/8e/57/f14eeb7f072b0e9a5a090d1712741fd8f214ec193dba773cf5410108bb7d/regex-2025.11.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:61a08bcb0ec14ff4e0ed2044aad948d0659604f824cbd50b55e30b0ec6f09c73", size = 288900, upload-time = "2025-11-03T21:32:53.569Z" },
- { url = "https://files.pythonhosted.org/packages/3c/6b/1d650c45e99a9b327586739d926a1cd4e94666b1bd4af90428b36af66dc7/regex-2025.11.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c9c30003b9347c24bcc210958c5d167b9e4f9be786cb380a7d32f14f9b84674f", size = 799010, upload-time = "2025-11-03T21:32:55.222Z" },
- { url = "https://files.pythonhosted.org/packages/99/ee/d66dcbc6b628ce4e3f7f0cbbb84603aa2fc0ffc878babc857726b8aab2e9/regex-2025.11.3-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4e1e592789704459900728d88d41a46fe3969b82ab62945560a31732ffc19a6d", size = 864893, upload-time = "2025-11-03T21:32:57.239Z" },
- { url = "https://files.pythonhosted.org/packages/bf/2d/f238229f1caba7ac87a6c4153d79947fb0261415827ae0f77c304260c7d3/regex-2025.11.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:6538241f45eb5a25aa575dbba1069ad786f68a4f2773a29a2bd3dd1f9de787be", size = 911522, upload-time = "2025-11-03T21:32:59.274Z" },
- { url = "https://files.pythonhosted.org/packages/bd/3d/22a4eaba214a917c80e04f6025d26143690f0419511e0116508e24b11c9b/regex-2025.11.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bce22519c989bb72a7e6b36a199384c53db7722fe669ba891da75907fe3587db", size = 803272, upload-time = "2025-11-03T21:33:01.393Z" },
- { url = "https://files.pythonhosted.org/packages/84/b1/03188f634a409353a84b5ef49754b97dbcc0c0f6fd6c8ede505a8960a0a4/regex-2025.11.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:66d559b21d3640203ab9075797a55165d79017520685fb407b9234d72ab63c62", size = 787958, upload-time = "2025-11-03T21:33:03.379Z" },
- { url = "https://files.pythonhosted.org/packages/99/6a/27d072f7fbf6fadd59c64d210305e1ff865cc3b78b526fd147db768c553b/regex-2025.11.3-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:669dcfb2e38f9e8c69507bace46f4889e3abbfd9b0c29719202883c0a603598f", size = 859289, upload-time = "2025-11-03T21:33:05.374Z" },
- { url = "https://files.pythonhosted.org/packages/9a/70/1b3878f648e0b6abe023172dacb02157e685564853cc363d9961bcccde4e/regex-2025.11.3-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:32f74f35ff0f25a5021373ac61442edcb150731fbaa28286bbc8bb1582c89d02", size = 850026, upload-time = "2025-11-03T21:33:07.131Z" },
- { url = "https://files.pythonhosted.org/packages/dd/d5/68e25559b526b8baab8e66839304ede68ff6727237a47727d240006bd0ff/regex-2025.11.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:e6c7a21dffba883234baefe91bc3388e629779582038f75d2a5be918e250f0ed", size = 789499, upload-time = "2025-11-03T21:33:09.141Z" },
- { url = "https://files.pythonhosted.org/packages/fc/df/43971264857140a350910d4e33df725e8c94dd9dee8d2e4729fa0d63d49e/regex-2025.11.3-cp314-cp314-win32.whl", hash = "sha256:795ea137b1d809eb6836b43748b12634291c0ed55ad50a7d72d21edf1cd565c4", size = 271604, upload-time = "2025-11-03T21:33:10.9Z" },
- { url = "https://files.pythonhosted.org/packages/01/6f/9711b57dc6894a55faf80a4c1b5aa4f8649805cb9c7aef46f7d27e2b9206/regex-2025.11.3-cp314-cp314-win_amd64.whl", hash = "sha256:9f95fbaa0ee1610ec0fc6b26668e9917a582ba80c52cc6d9ada15e30aa9ab9ad", size = 280320, upload-time = "2025-11-03T21:33:12.572Z" },
- { url = "https://files.pythonhosted.org/packages/f1/7e/f6eaa207d4377481f5e1775cdeb5a443b5a59b392d0065f3417d31d80f87/regex-2025.11.3-cp314-cp314-win_arm64.whl", hash = "sha256:dfec44d532be4c07088c3de2876130ff0fbeeacaa89a137decbbb5f665855a0f", size = 273372, upload-time = "2025-11-03T21:33:14.219Z" },
- { url = "https://files.pythonhosted.org/packages/c3/06/49b198550ee0f5e4184271cee87ba4dfd9692c91ec55289e6282f0f86ccf/regex-2025.11.3-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:ba0d8a5d7f04f73ee7d01d974d47c5834f8a1b0224390e4fe7c12a3a92a78ecc", size = 491985, upload-time = "2025-11-03T21:33:16.555Z" },
- { url = "https://files.pythonhosted.org/packages/ce/bf/abdafade008f0b1c9da10d934034cb670432d6cf6cbe38bbb53a1cfd6cf8/regex-2025.11.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:442d86cf1cfe4faabf97db7d901ef58347efd004934da045c745e7b5bd57ac49", size = 292669, upload-time = "2025-11-03T21:33:18.32Z" },
- { url = "https://files.pythonhosted.org/packages/f9/ef/0c357bb8edbd2ad8e273fcb9e1761bc37b8acbc6e1be050bebd6475f19c1/regex-2025.11.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:fd0a5e563c756de210bb964789b5abe4f114dacae9104a47e1a649b910361536", size = 291030, upload-time = "2025-11-03T21:33:20.048Z" },
- { url = "https://files.pythonhosted.org/packages/79/06/edbb67257596649b8fb088d6aeacbcb248ac195714b18a65e018bf4c0b50/regex-2025.11.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bf3490bcbb985a1ae97b2ce9ad1c0f06a852d5b19dde9b07bdf25bf224248c95", size = 807674, upload-time = "2025-11-03T21:33:21.797Z" },
- { url = "https://files.pythonhosted.org/packages/f4/d9/ad4deccfce0ea336296bd087f1a191543bb99ee1c53093dcd4c64d951d00/regex-2025.11.3-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:3809988f0a8b8c9dcc0f92478d6501fac7200b9ec56aecf0ec21f4a2ec4b6009", size = 873451, upload-time = "2025-11-03T21:33:23.741Z" },
- { url = "https://files.pythonhosted.org/packages/13/75/a55a4724c56ef13e3e04acaab29df26582f6978c000ac9cd6810ad1f341f/regex-2025.11.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f4ff94e58e84aedb9c9fce66d4ef9f27a190285b451420f297c9a09f2b9abee9", size = 914980, upload-time = "2025-11-03T21:33:25.999Z" },
- { url = "https://files.pythonhosted.org/packages/67/1e/a1657ee15bd9116f70d4a530c736983eed997b361e20ecd8f5ca3759d5c5/regex-2025.11.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7eb542fd347ce61e1321b0a6b945d5701528dca0cd9759c2e3bb8bd57e47964d", size = 812852, upload-time = "2025-11-03T21:33:27.852Z" },
- { url = "https://files.pythonhosted.org/packages/b8/6f/f7516dde5506a588a561d296b2d0044839de06035bb486b326065b4c101e/regex-2025.11.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:d6c2d5919075a1f2e413c00b056ea0c2f065b3f5fe83c3d07d325ab92dce51d6", size = 795566, upload-time = "2025-11-03T21:33:32.364Z" },
- { url = "https://files.pythonhosted.org/packages/d9/dd/3d10b9e170cc16fb34cb2cef91513cf3df65f440b3366030631b2984a264/regex-2025.11.3-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:3f8bf11a4827cc7ce5a53d4ef6cddd5ad25595d3c1435ef08f76825851343154", size = 868463, upload-time = "2025-11-03T21:33:34.459Z" },
- { url = "https://files.pythonhosted.org/packages/f5/8e/935e6beff1695aa9085ff83195daccd72acc82c81793df480f34569330de/regex-2025.11.3-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:22c12d837298651e5550ac1d964e4ff57c3f56965fc1812c90c9fb2028eaf267", size = 854694, upload-time = "2025-11-03T21:33:36.793Z" },
- { url = "https://files.pythonhosted.org/packages/92/12/10650181a040978b2f5720a6a74d44f841371a3d984c2083fc1752e4acf6/regex-2025.11.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:62ba394a3dda9ad41c7c780f60f6e4a70988741415ae96f6d1bf6c239cf01379", size = 799691, upload-time = "2025-11-03T21:33:39.079Z" },
- { url = "https://files.pythonhosted.org/packages/67/90/8f37138181c9a7690e7e4cb388debbd389342db3c7381d636d2875940752/regex-2025.11.3-cp314-cp314t-win32.whl", hash = "sha256:4bf146dca15cdd53224a1bf46d628bd7590e4a07fbb69e720d561aea43a32b38", size = 274583, upload-time = "2025-11-03T21:33:41.302Z" },
- { url = "https://files.pythonhosted.org/packages/8f/cd/867f5ec442d56beb56f5f854f40abcfc75e11d10b11fdb1869dd39c63aaf/regex-2025.11.3-cp314-cp314t-win_amd64.whl", hash = "sha256:adad1a1bcf1c9e76346e091d22d23ac54ef28e1365117d99521631078dfec9de", size = 284286, upload-time = "2025-11-03T21:33:43.324Z" },
- { url = "https://files.pythonhosted.org/packages/20/31/32c0c4610cbc070362bf1d2e4ea86d1ea29014d400a6d6c2486fcfd57766/regex-2025.11.3-cp314-cp314t-win_arm64.whl", hash = "sha256:c54f768482cef41e219720013cd05933b6f971d9562544d691c68699bf2b6801", size = 274741, upload-time = "2025-11-03T21:33:45.557Z" },
+version = "2026.3.32"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/81/93/5ab3e899c47fa7994e524447135a71cd121685a35c8fe35029005f8b236f/regex-2026.3.32.tar.gz", hash = "sha256:f1574566457161678297a116fa5d1556c5a4159d64c5ff7c760e7c564bf66f16", size = 415605, upload-time = "2026-03-28T21:49:22.012Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/38/94/69492c45b0e61b027109d8433a5c3d4f7a90709184c057c7cfc60acb1bfa/regex-2026.3.32-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:ad8d372587e659940568afd009afeb72be939c769c552c9b28773d0337251391", size = 490572, upload-time = "2026-03-28T21:46:28.031Z" },
+ { url = "https://files.pythonhosted.org/packages/92/0a/7dcffeebe0fcac45a1f9caf80712002d3cbd66d7d69d719315ee142b280f/regex-2026.3.32-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3f5747501b69299c6b0b047853771e4ed390510bada68cb16da9c9c2078343f7", size = 292078, upload-time = "2026-03-28T21:46:29.789Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/ec/988486058ef49eb931476419bae00f164c4ceb44787c45dc7a54b7de0ea4/regex-2026.3.32-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:db976be51375bca900e008941639448d148c655c9545071965d0571ecc04f5d0", size = 289786, upload-time = "2026-03-28T21:46:31.415Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/cf/1955bb5567bc491bd63068e17f75ab0c9ff5e9d08466beec7e347f5e768d/regex-2026.3.32-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:66a5083c3ffe5a5a95f8281ea47a88072d4f24001d562d1d9d28d4cdc005fec5", size = 796431, upload-time = "2026-03-28T21:46:33.101Z" },
+ { url = "https://files.pythonhosted.org/packages/27/8a/67fcbca511b792107540181ee0690df6de877bfbcb41b7ecae7028025ca5/regex-2026.3.32-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e83ce8008b48762be296f1401f19afd9ea29f3d035d1974e0cecb74e9afbd1df", size = 865785, upload-time = "2026-03-28T21:46:35.053Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/59/0677bc44f2c28305edcabc11933777b9ad34e9e8ded7ba573d24e4bc3ee7/regex-2026.3.32-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b3aa21bad31db904e0b9055e12c8282df62d43169c4a9d2929407060066ebc74", size = 913593, upload-time = "2026-03-28T21:46:36.835Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/fe/661043d1c263b0d9d10c6ff4e9c9745f3df9641c62b51f96a3473638e7ce/regex-2026.3.32-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f54840bea73541652f1170dc63402a5b776fc851ad36a842da9e5163c1f504a0", size = 801512, upload-time = "2026-03-28T21:46:38.587Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/27/74c986061380e1811a46cf04cdf9c939db9f8c0e63953eddfe37ffd633ea/regex-2026.3.32-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:2ffbadc647325dd4e3118269bda93ded1eb5f5b0c3b7ba79a3da9fbd04f248e9", size = 776182, upload-time = "2026-03-28T21:46:40.69Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/c8/d833397b70cd1bacfcdc0a611f0e2c1f5b91fee8eedd88affcee770cbbb6/regex-2026.3.32-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:66d3126afe7eac41759cd5f0b3b246598086e88e70527c0d68c9e615b81771c4", size = 785837, upload-time = "2026-03-28T21:46:42.926Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/53/fa226b72989b5b93db6926fab5478115e085dfcf077e18d2cb386be0fd23/regex-2026.3.32-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:f785f44a44702dea89b28bce5bc82552490694ce4e144e21a4f0545e364d2150", size = 860612, upload-time = "2026-03-28T21:46:44.8Z" },
+ { url = "https://files.pythonhosted.org/packages/04/28/bdd2fc0c055a1b15702bd4084829bbb6b06095f27990e5bee52b2898ea03/regex-2026.3.32-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:b7836aa13721dbdef658aebd11f60d00de633a95726521860fe1f6be75fa225a", size = 765285, upload-time = "2026-03-28T21:46:46.625Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/da/21f5e2a35a191b27e5a47cccb3914c99e139b49b1342d3f36e64e8cc60f7/regex-2026.3.32-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:5336b1506142eb0f23c96fb4a34b37c4fefd4fed2a7042069f3c8058efe17855", size = 851963, upload-time = "2026-03-28T21:46:48.341Z" },
+ { url = "https://files.pythonhosted.org/packages/18/f4/04ed04ebf335a44083695c22772be6a42efa31900415555563acf02cb4de/regex-2026.3.32-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:b56993a7aeb4140c4770f4f7965c9e5af4f024457d06e23c01b0d47501cb18ed", size = 788332, upload-time = "2026-03-28T21:46:50.454Z" },
+ { url = "https://files.pythonhosted.org/packages/21/25/5355908f479d0dc13d044f88270cdcabc8723efc12e4c2b19e5a94ff1a96/regex-2026.3.32-cp312-cp312-win32.whl", hash = "sha256:d363660f9ef8c734495598d2f3e527fb41f745c73159dc0d743402f049fb6836", size = 266847, upload-time = "2026-03-28T21:46:52.125Z" },
+ { url = "https://files.pythonhosted.org/packages/00/e5/3be71c781a031db5df00735b613895ad5fdbf86c6e3bbea5fbbd7bfb5902/regex-2026.3.32-cp312-cp312-win_amd64.whl", hash = "sha256:c9f261ad3cd97257dc1d9355bfbaa7dd703e06574bffa0fa8fe1e31da915ee38", size = 278034, upload-time = "2026-03-28T21:46:54.096Z" },
+ { url = "https://files.pythonhosted.org/packages/31/5f/27f1e0b1eea4faa99c66daca34130af20c44fae0237bbc98b87999dbc4a8/regex-2026.3.32-cp312-cp312-win_arm64.whl", hash = "sha256:89e50667e7e8c0e7903e4d644a2764fffe9a3a5d6578f72ab7a7b4205bf204b7", size = 270673, upload-time = "2026-03-28T21:46:56.046Z" },
+ { url = "https://files.pythonhosted.org/packages/bd/ba/9c1819f302b42b5fbd4139ead6280e9ec37d19bbe33379df0039b2a57bb4/regex-2026.3.32-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:c6d9c6e783b348f719b6118bb3f187b2e138e3112576c9679eb458cc8b2e164b", size = 490394, upload-time = "2026-03-28T21:46:58.112Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/0b/f62b0ce79eb83ca82fffea1736289d29bc24400355968301406789bcebd2/regex-2026.3.32-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:0f21ae18dfd15752cdd98d03cbd7a3640be826bfd58482a93f730dbd24d7b9fb", size = 291993, upload-time = "2026-03-28T21:47:00.198Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/d8/ba0f8f81f88cd20c0b27acc123561ac5495ea33f800f0b8ebed2038b23eb/regex-2026.3.32-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:844d88509c968dd44b30daeefac72b038b1bf31ac372d5106358ab01d393c48b", size = 289618, upload-time = "2026-03-28T21:47:02.269Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/0d/b47a0e68bc511c195ff129c0311a4cd79b954b8676193a9d03a97c623a91/regex-2026.3.32-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8fc918cd003ba0d066bf0003deb05a259baaaab4dc9bd4f1207bbbe64224857a", size = 796427, upload-time = "2026-03-28T21:47:04.096Z" },
+ { url = "https://files.pythonhosted.org/packages/51/d7/32b05aa8fde7789ba316533c0f30e87b6b5d38d6d7f8765eadc5aab84671/regex-2026.3.32-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:bbc458a292aee57d572075f22c035fa32969cdb7987d454e3e34d45a40a0a8b4", size = 865850, upload-time = "2026-03-28T21:47:05.982Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/67/828d8095501f237b83f630d4069eea8c0e5cb6a204e859cf0b67c223ce12/regex-2026.3.32-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:987cdfcfb97a249abc3601ad53c7de5c370529f1981e4c8c46793e4a1e1bfe8e", size = 913578, upload-time = "2026-03-28T21:47:08.172Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/f8/acf1eb80f58852e85bd39a6ddfa78ce2243ddc8de8da7582e6ba657da593/regex-2026.3.32-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a5d88fa37ba5e8a80ca8d956b9ea03805cfa460223ac94b7d4854ee5e30f3173", size = 801536, upload-time = "2026-03-28T21:47:10.206Z" },
+ { url = "https://files.pythonhosted.org/packages/9f/05/986cdf8d12693451f5889aaf4ea4f65b2c49b1152ae814fa1fb75439e40b/regex-2026.3.32-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:4d082be64e51671dd5ee1c208c92da2ddda0f2f20d8ef387e57634f7e97b6aae", size = 776226, upload-time = "2026-03-28T21:47:12.891Z" },
+ { url = "https://files.pythonhosted.org/packages/32/02/945a6a2348ca1c6608cb1747275c8affd2ccd957d4885c25218a86377912/regex-2026.3.32-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:c1d7fa44aece1fa02b8927441614c96520253a5cad6a96994e3a81e060feed55", size = 785933, upload-time = "2026-03-28T21:47:14.795Z" },
+ { url = "https://files.pythonhosted.org/packages/53/12/c5bab6cc679ad79a45427a98c4e70809586ac963c5ad54a9217533c4763e/regex-2026.3.32-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:d478a2ca902b6ef28ffc9521e5f0f728d036abe35c0b250ee8ae78cfe7c5e44e", size = 860671, upload-time = "2026-03-28T21:47:16.985Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/68/8d85f98c2443469facabef62b82b851d369b13f92bec2ca7a3808deaa47b/regex-2026.3.32-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:2820d2231885e97aff0fcf230a19ebd5d2b5b8a1ba338c20deb34f16db1c7897", size = 765335, upload-time = "2026-03-28T21:47:18.872Z" },
+ { url = "https://files.pythonhosted.org/packages/89/a7/d8a9c270916107a501fca63b748547c6c77e570d19f16a29b557ce734f3d/regex-2026.3.32-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:fc8ced733d6cd9af5e412f256a32f7c61cd2d7371280a65c689939ac4572499f", size = 851913, upload-time = "2026-03-28T21:47:20.793Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/8e/03d392b26679914ccf21f83d18ad4443232d2f8c3e2c30a962d4e3918d9c/regex-2026.3.32-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:847087abe98b3c1ebf1eb49d6ef320dbba75a83ee4f83c94704580f1df007dd4", size = 788447, upload-time = "2026-03-28T21:47:22.628Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/df/692227d23535a50604333068b39eb262626db780ab1e1b19d83fc66853aa/regex-2026.3.32-cp313-cp313-win32.whl", hash = "sha256:d21a07edddb3e0ca12a8b8712abc8452481c3d3db19ae87fc94e9842d005964b", size = 266834, upload-time = "2026-03-28T21:47:24.778Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/37/13e4e56adc16ba607cffa1fe880f233eb9ded8ab8a8580619683c9e4ce48/regex-2026.3.32-cp313-cp313-win_amd64.whl", hash = "sha256:3c054e39a9f85a3d76c62a1d50c626c5e9306964eaa675c53f61ff7ec1204bbb", size = 277972, upload-time = "2026-03-28T21:47:26.627Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/1c/80a86dbb2b416fec003b1801462bdcebbf1d43202ed5acb176e99c1ba369/regex-2026.3.32-cp313-cp313-win_arm64.whl", hash = "sha256:b2e9c2ea2e93223579308263f359eab8837dc340530b860cb59b713651889f14", size = 270649, upload-time = "2026-03-28T21:47:28.551Z" },
+ { url = "https://files.pythonhosted.org/packages/58/08/e38372da599dc1c39c599907ec535016d110034bd3701ce36554f59767ef/regex-2026.3.32-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:5d86e3fb08c94f084a625c8dc2132a79a3a111c8bf6e2bc59351fa61753c2f6e", size = 494495, upload-time = "2026-03-28T21:47:30.642Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/27/6e29ece8c9ce01001ece1137fa21c8707529c2305b22828f63623b0eb262/regex-2026.3.32-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:b6f366a5ef66a2df4d9e68035cfe9f0eb8473cdfb922c37fac1d169b468607b0", size = 293988, upload-time = "2026-03-28T21:47:32.553Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/98/8752e18bb87a2fe728b73b0f83c082eb162a470766063f8028759fb26844/regex-2026.3.32-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:b8fca73e16c49dd972ce3a88278dfa5b93bf91ddef332a46e9443abe21ca2f7c", size = 292634, upload-time = "2026-03-28T21:47:34.651Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/7b/d7729fe294e23e9c7c3871cb69d49059fa7d65fd11e437a2cbea43f6615d/regex-2026.3.32-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b953d9d496d19786f4d46e6ba4b386c6e493e81e40f9c5392332458183b0599d", size = 810532, upload-time = "2026-03-28T21:47:36.839Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/49/4dae7b000659f611b17b9c1541fba800b0569e4060debc4635ef1b23982c/regex-2026.3.32-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b565f25171e04d4fad950d1fa837133e3af6ea6f509d96166eed745eb0cf63bc", size = 871919, upload-time = "2026-03-28T21:47:39.192Z" },
+ { url = "https://files.pythonhosted.org/packages/83/85/aa8ad3977b9399861db3df62b33fe5fef6932ee23a1b9f4f357f58f2094b/regex-2026.3.32-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f28eac18a8733a124444643a66ac96fef2c0ad65f50034e0a043b90333dc677f", size = 916550, upload-time = "2026-03-28T21:47:41.618Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/c0/6379d7f5b59ff0656ba49cf666d5013ecee55e83245275b310b0ffc79143/regex-2026.3.32-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7cdd508664430dd51b8888deb6c5b416d8de046b2e11837254378d31febe4a98", size = 814988, upload-time = "2026-03-28T21:47:43.681Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/af/2dfddc64074bd9b70e27e170ee9db900542e2870210b489ad4471416ba86/regex-2026.3.32-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5c35d097f509cf7e40d20d5bee548d35d6049b36eb9965e8d43e4659923405b9", size = 786337, upload-time = "2026-03-28T21:47:46.076Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/2f/4eb8abd705236402b4fe0e130971634deffb1855e2028bf02a2b7c0e841c/regex-2026.3.32-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:85c9b0c131427470a6423baa0a9330be6fd8c3630cc3ee6fdee03360724cbec5", size = 800029, upload-time = "2026-03-28T21:47:48.356Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/2c/77d9ca2c9df483b51b4b1291c96d79c9ae301077841c4db39bc822f6b4c6/regex-2026.3.32-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:e50af656c15e2723eeb7279c0837e07accc594b95ec18b86821a4d44b51b24bf", size = 865843, upload-time = "2026-03-28T21:47:50.762Z" },
+ { url = "https://files.pythonhosted.org/packages/48/10/306f477a509f4eed699071b1f031d89edd5a2b5fa28c8ede5b2638eaba82/regex-2026.3.32-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:4bc32b4dbdb4f9f300cf9f38f8ea2ce9511a068ffaa45ac1373ee7a943f1d810", size = 772473, upload-time = "2026-03-28T21:47:52.771Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/f6/54bd83ec46ac037de2beb049afc9dd5d2769c6ecaadf7856254ce610e62a/regex-2026.3.32-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:e3e5d1802cba785210a4a800e63fcee7a228649a880f3bf7f2aadccb151a834b", size = 856805, upload-time = "2026-03-28T21:47:55.04Z" },
+ { url = "https://files.pythonhosted.org/packages/37/e8/ee0e7d14de1fc6582d5782f072db6c61465a38a4142f88e175dda494b536/regex-2026.3.32-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:ef250a3f5e93182193f5c927c5e9575b2cb14b80d03e258bc0b89cc5de076b60", size = 801875, upload-time = "2026-03-28T21:47:57.434Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/06/0fa9daca59d07b6aabd8e0468d3b86fd578576a157206fbcddbfc2298f7d/regex-2026.3.32-cp313-cp313t-win32.whl", hash = "sha256:9cf7036dfa2370ccc8651521fcbb40391974841119e9982fa312b552929e6c85", size = 269892, upload-time = "2026-03-28T21:47:59.674Z" },
+ { url = "https://files.pythonhosted.org/packages/13/47/77f16b5ad9f10ca574f03d84a354b359b0ac33f85054f2f2daafc9f7b807/regex-2026.3.32-cp313-cp313t-win_amd64.whl", hash = "sha256:c940e00e8d3d10932c929d4b8657c2ea47d2560f31874c3e174c0d3488e8b865", size = 281318, upload-time = "2026-03-28T21:48:01.562Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/47/db4446faaea8d01c8315c9c89c7dc6abbb3305e8e712e9b23936095c4d58/regex-2026.3.32-cp313-cp313t-win_arm64.whl", hash = "sha256:ace48c5e157c1e58b7de633c5e257285ce85e567ac500c833349c363b3df69d4", size = 272366, upload-time = "2026-03-28T21:48:03.748Z" },
+ { url = "https://files.pythonhosted.org/packages/32/68/ff024bf6131b7446a791a636dbbb7fa732d586f33b276d84b3460ea49393/regex-2026.3.32-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:a416ee898ecbc5d8b283223b4cf4d560f93244f6f7615c1bd67359744b00c166", size = 490430, upload-time = "2026-03-28T21:48:05.654Z" },
+ { url = "https://files.pythonhosted.org/packages/61/72/039d9164817ee298f2a2d0246001afe662241dcbec0eedd1fe03e2a2555e/regex-2026.3.32-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:d76d62909bfb14521c3f7cfd5b94c0c75ec94b0a11f647d2f604998962ec7b6c", size = 291948, upload-time = "2026-03-28T21:48:07.666Z" },
+ { url = "https://files.pythonhosted.org/packages/06/9d/77f684d90ffe3e99b828d3cabb87a0f1601d2b9decd1333ff345809b1d02/regex-2026.3.32-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:631f7d95c83f42bccfe18946a38ad27ff6b6717fb4807e60cf24860b5eb277fc", size = 289786, upload-time = "2026-03-28T21:48:09.562Z" },
+ { url = "https://files.pythonhosted.org/packages/83/70/bd76069a0304e924682b2efd8683a01617a7e1da9b651af73039d8da76a4/regex-2026.3.32-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:12917c6c6813ffcdfb11680a04e4d63c5532b88cf089f844721c5f41f41a63ad", size = 796672, upload-time = "2026-03-28T21:48:11.568Z" },
+ { url = "https://files.pythonhosted.org/packages/80/31/c2d7d9a5671e111a2c16d57e0cb03e1ce35b28a115901590528aa928bb5b/regex-2026.3.32-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:3e221b615f83b15887636fcb90ed21f1a19541366f8b7ba14ba1ad8304f4ded4", size = 866556, upload-time = "2026-03-28T21:48:14.081Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/b9/9921a31931d0bc3416ac30205471e0e2ed60dcbd16fc922bbd69b427322b/regex-2026.3.32-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4f9ae4755fa90f1dc2d0d393d572ebc134c0fe30fcfc0ab7e67c1db15f192041", size = 912787, upload-time = "2026-03-28T21:48:16.548Z" },
+ { url = "https://files.pythonhosted.org/packages/41/ab/2c1bc8ab99f63cdabdbc7823af8f4cfcd6ddbb2babf01861826c3f1ad44d/regex-2026.3.32-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a094e9dcafedfb9d333db5cf880304946683f43a6582bb86688f123335122929", size = 800879, upload-time = "2026-03-28T21:48:18.971Z" },
+ { url = "https://files.pythonhosted.org/packages/49/e5/0be716eb2c0b2ae3a439e44432534e82b2f81848af64cb21c0473ad8ae46/regex-2026.3.32-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c1cecea3e477af105f32ef2119b8d895f297492e41d317e60d474bc4bffd62ff", size = 776332, upload-time = "2026-03-28T21:48:21.163Z" },
+ { url = "https://files.pythonhosted.org/packages/26/80/114a61bd25dec7d1070930eaef82aadf9b05961a37629e7cca7bc3fc2257/regex-2026.3.32-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:f26262900edd16272b6360014495e8d68379c6c6e95983f9b7b322dc928a1194", size = 786384, upload-time = "2026-03-28T21:48:23.277Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/78/be0a6531f8db426e8e60d6356aeef8e9cc3f541655a648c4968b63c87a88/regex-2026.3.32-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:1cb22fa9ee6a0acb22fc9aecce5f9995fe4d2426ed849357d499d62608fbd7f9", size = 861381, upload-time = "2026-03-28T21:48:25.371Z" },
+ { url = "https://files.pythonhosted.org/packages/45/b1/e5076fbe45b8fb39672584b1b606d512f5bd3a43155be68a95f6b88c1fc5/regex-2026.3.32-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:9b9118a78e031a2e4709cd2fcc3028432e89b718db70073a8da574c249b5b249", size = 765434, upload-time = "2026-03-28T21:48:27.494Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/da/fd65d68b897f8b52b1390d20d776fa753582484724a9cb4f4c26de657ae5/regex-2026.3.32-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:b193ed199848aa96618cd5959c1582a0bf23cd698b0b900cb0ffe81b02c8659c", size = 851501, upload-time = "2026-03-28T21:48:29.884Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/d6/1e9c991c32022a9312e9124cc974961b3a2501338de2cd1cce75a3612d7a/regex-2026.3.32-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:10fb2aaae1aaadf7d43c9f3c2450404253697bf8b9ce360bd5418d1d16292298", size = 788076, upload-time = "2026-03-28T21:48:32.025Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/5b/b23c72f6d607cbb24ef42acf0c7c2ef4eee1377a9f7ba43b312f889edfbb/regex-2026.3.32-cp314-cp314-win32.whl", hash = "sha256:110ba4920721374d16c4c8ea7ce27b09546d43e16aea1d7f43681b5b8f80ba61", size = 272255, upload-time = "2026-03-28T21:48:34.355Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/ec/32bbcc42366097a8cea2c481e02964be6c6fa5ccfb0fa9581686af0bec5f/regex-2026.3.32-cp314-cp314-win_amd64.whl", hash = "sha256:245667ad430745bae6a1e41081872d25819d86fbd9e0eec485ba00d9f78ad43d", size = 281160, upload-time = "2026-03-28T21:48:36.588Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/e4/89038a028cb68e719fa03ab1ad603649fc199bcda12270d2ac7b471b8f5d/regex-2026.3.32-cp314-cp314-win_arm64.whl", hash = "sha256:1ca02ff0ef33e9d8276a1fcd6d90ff6ea055a32c9149c0050b5b67e26c6d2c51", size = 273688, upload-time = "2026-03-28T21:48:38.976Z" },
+ { url = "https://files.pythonhosted.org/packages/30/6e/87caccd608837a1fa4f8c7edc48e206103452b9bbc94fc724fa39340e807/regex-2026.3.32-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:51fb7e26f91f9091fd8ec6a946f99b15d3bc3667cb5ddc73dd6cb2222dd4a1cc", size = 494506, upload-time = "2026-03-28T21:48:41.327Z" },
+ { url = "https://files.pythonhosted.org/packages/16/53/a922e6b24694d70bdd68fc3fd076950e15b1b418cff9d2cc362b3968d86f/regex-2026.3.32-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:51a93452034d671b0e21b883d48ea66c5d6a05620ee16a9d3f229e828568f3f0", size = 293986, upload-time = "2026-03-28T21:48:43.481Z" },
+ { url = "https://files.pythonhosted.org/packages/60/e4/0cb32203c1aebad0577fcd5b9af1fe764869e617d5234bc6a0ad284299ea/regex-2026.3.32-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:03c2ebd15ff51e7b13bb3dc28dd5ac18cd39e59ebb40430b14ae1a19e833cff1", size = 292677, upload-time = "2026-03-28T21:48:45.772Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/f8/5006b70291469d4174dd66ad162802e2f68419c0f2a7952d0c76c1288cfa/regex-2026.3.32-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5bf2f3c2c5bd8360d335c7dcd4a9006cf1dabae063ee2558ee1b07bbc8a20d88", size = 810661, upload-time = "2026-03-28T21:48:48.147Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/9b/438763a20d22cd1f65f95c8f030dd25df2d80a941068a891d21a5f240456/regex-2026.3.32-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8a4a3189a99ecdd1c13f42513ab3fc7fa8311b38ba7596dd98537acb8cd9acc3", size = 872156, upload-time = "2026-03-28T21:48:50.739Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/5b/1341287887ac982ed9f5f60125e440513ffe354aa7e3681940495af7c12a/regex-2026.3.32-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3c0bbfbd38506e1ea96a85da6782577f06239cb9fcf9696f1ea537c980c0680b", size = 916749, upload-time = "2026-03-28T21:48:53.57Z" },
+ { url = "https://files.pythonhosted.org/packages/42/e2/1d2b48b8e94debfffc6fefb84d2a86a178cc208652a1d6493d5f29821c70/regex-2026.3.32-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8aaf8ee8f34b677f90742ca089b9c83d64bdc410528767273c816a863ed57327", size = 814788, upload-time = "2026-03-28T21:48:55.905Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/d9/7dacb34c43adaeb954518d851f3e5d3ce495ac00a9d6010e3b4b59917c4a/regex-2026.3.32-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3ea568832eca219c2be1721afa073c1c9eb8f98a9733fdedd0a9747639fc22a5", size = 786594, upload-time = "2026-03-28T21:48:58.404Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/72/28295068c92dbd6d3ce4fd22554345cf504e957cc57dadeda4a64fa86a57/regex-2026.3.32-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8e4c8fa46aad1a11ae2f8fcd1c90b9d55e18925829ac0d98c5bb107f93351745", size = 800167, upload-time = "2026-03-28T21:49:01.226Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/17/b10745adeca5b8d52da050e7c746137f5d01dabc6dbbe6e8d9d821dc65c1/regex-2026.3.32-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:0cec365d44835b043d7b3266487797639d07d621bec9dc0ea224b00775797cc1", size = 865906, upload-time = "2026-03-28T21:49:03.484Z" },
+ { url = "https://files.pythonhosted.org/packages/45/9d/1acbcce765044ac0c87f453f4876e0897f7a61c10315262f960184310798/regex-2026.3.32-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:09e26cad1544d856da85881ad292797289e4406338afe98163f3db9f7fac816c", size = 772642, upload-time = "2026-03-28T21:49:06.811Z" },
+ { url = "https://files.pythonhosted.org/packages/24/41/1ef8b4811355ad7b9d7579d3aeca00f18b7bc043ace26c8c609b9287346d/regex-2026.3.32-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:6062c4ef581a3e9e503dccf4e1b7f2d33fdc1c13ad510b287741ac73bc4c6b27", size = 856927, upload-time = "2026-03-28T21:49:09.373Z" },
+ { url = "https://files.pythonhosted.org/packages/97/b1/0dc1d361be80ec1b8b707ada041090181133a7a29d438e432260a4b26f9a/regex-2026.3.32-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:88ebc0783907468f17fca3d7821b30f9c21865a721144eb498cb0ff99a67bcac", size = 801910, upload-time = "2026-03-28T21:49:11.818Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/db/1a23f767fa250844772a9464306d34e0fafe2c317303b88a1415096b6324/regex-2026.3.32-cp314-cp314t-win32.whl", hash = "sha256:e480d3dac06c89bc2e0fd87524cc38c546ac8b4a38177650745e64acbbcfdeba", size = 275714, upload-time = "2026-03-28T21:49:14.528Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/2b/616d31b125ca76079d74d6b1d84ec0860ffdb41c379151135d06e35a8633/regex-2026.3.32-cp314-cp314t-win_amd64.whl", hash = "sha256:67015a8162d413af9e3309d9a24e385816666fbf09e48e3ec43342c8536f7df6", size = 285722, upload-time = "2026-03-28T21:49:16.642Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/91/043d9a00d6123c5fa22a3dc96b10445ce434a8110e1d5e53efb01f243c8b/regex-2026.3.32-cp314-cp314t-win_arm64.whl", hash = "sha256:1a6ac1ed758902e664e0d95c1ee5991aa6fb355423f378ed184c6ec47a1ec0e9", size = 275700, upload-time = "2026-03-28T21:49:19.348Z" },
]
[[package]]
name = "requests"
-version = "2.32.5"
+version = "2.33.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "certifi" },
@@ -4527,9 +5528,9 @@ dependencies = [
{ name = "idna" },
{ name = "urllib3" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/c9/74/b3ff8e6c8446842c3f5c837e9c3dfcfe2018ea6ecef224c710c85ef728f4/requests-2.32.5.tar.gz", hash = "sha256:dbba0bac56e100853db0ea71b82b4dfd5fe2bf6d3754a8893c3af500cec7d7cf", size = 134517, upload-time = "2025-08-18T20:46:02.573Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/34/64/8860370b167a9721e8956ae116825caff829224fbca0ca6e7bf8ddef8430/requests-2.33.0.tar.gz", hash = "sha256:c7ebc5e8b0f21837386ad0e1c8fe8b829fa5f544d8df3b2253bff14ef29d7652", size = 134232, upload-time = "2026-03-25T15:10:41.586Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/1e/db/4254e3eabe8020b458f1a747140d32277ec7a271daf1d235b70dc0b4e6e3/requests-2.32.5-py3-none-any.whl", hash = "sha256:2462f94637a34fd532264295e186976db0f5d453d1cdd31473c85a6a161affb6", size = 64738, upload-time = "2025-08-18T20:46:00.542Z" },
+ { url = "https://files.pythonhosted.org/packages/56/5d/c814546c2333ceea4ba42262d8c4d55763003e767fa169adc693bd524478/requests-2.33.0-py3-none-any.whl", hash = "sha256:3324635456fa185245e24865e810cecec7b4caf933d7eb133dcde67d48cee69b", size = 65017, upload-time = "2026-03-25T15:10:40.382Z" },
]
[[package]]
@@ -4546,15 +5547,15 @@ wheels = [
[[package]]
name = "rich"
-version = "14.2.0"
+version = "14.3.3"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "markdown-it-py" },
{ name = "pygments" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/fb/d2/8920e102050a0de7bfabeb4c4614a49248cf8d5d7a8d01885fbb24dc767a/rich-14.2.0.tar.gz", hash = "sha256:73ff50c7c0c1c77c8243079283f4edb376f0f6442433aecb8ce7e6d0b92d1fe4", size = 219990, upload-time = "2025-10-09T14:16:53.064Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/b3/c6/f3b320c27991c46f43ee9d856302c70dc2d0fb2dba4842ff739d5f46b393/rich-14.3.3.tar.gz", hash = "sha256:b8daa0b9e4eef54dd8cf7c86c03713f53241884e814f4e2f5fb342fe520f639b", size = 230582, upload-time = "2026-02-19T17:23:12.474Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/25/7a/b0178788f8dc6cafce37a212c99565fa1fe7872c70c6c9c1e1a372d9d88f/rich-14.2.0-py3-none-any.whl", hash = "sha256:76bc51fe2e57d2b1be1f96c524b890b816e334ab4c1e45888799bfaab0021edd", size = 243393, upload-time = "2025-10-09T14:16:51.245Z" },
+ { url = "https://files.pythonhosted.org/packages/14/25/b208c5683343959b670dc001595f2f3737e051da617f66c31f7c4fa93abc/rich-14.3.3-py3-none-any.whl", hash = "sha256:793431c1f8619afa7d3b52b2cdec859562b950ea0d4b6b505397612db8d5362d", size = 310458, upload-time = "2026-02-19T17:23:13.732Z" },
]
[[package]]
@@ -4566,53 +5567,12 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/04/54/6f679c435d28e0a568d8e8a7c0a93a09010818634c3c3907fc98d8983770/roman_numerals-4.1.0-py3-none-any.whl", hash = "sha256:647ba99caddc2cc1e55a51e4360689115551bf4476d90e8162cf8c345fe233c7", size = 7676, upload-time = "2025-12-17T18:25:33.098Z" },
]
-[[package]]
-name = "roman-numerals-py"
-version = "4.1.0"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
- { name = "roman-numerals", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/cb/b5/de96fca640f4f656eb79bbee0e79aeec52e3e0e359f8a3e6a0d366378b64/roman_numerals_py-4.1.0.tar.gz", hash = "sha256:f5d7b2b4ca52dd855ef7ab8eb3590f428c0b1ea480736ce32b01fef2a5f8daf9", size = 4274, upload-time = "2025-12-17T18:25:41.153Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/27/2c/daca29684cbe9fd4bc711f8246da3c10adca1ccc4d24436b17572eb2590e/roman_numerals_py-4.1.0-py3-none-any.whl", hash = "sha256:553114c1167141c1283a51743759723ecd05604a1b6b507225e91dc1a6df0780", size = 4547, upload-time = "2025-12-17T18:25:40.136Z" },
-]
-
[[package]]
name = "rpds-py"
version = "0.30.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/20/af/3f2f423103f1113b36230496629986e0ef7e199d2aa8392452b484b38ced/rpds_py-0.30.0.tar.gz", hash = "sha256:dd8ff7cf90014af0c0f787eea34794ebf6415242ee1d6fa91eaba725cc441e84", size = 69469, upload-time = "2025-11-30T20:24:38.837Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/06/0c/0c411a0ec64ccb6d104dcabe0e713e05e153a9a2c3c2bd2b32ce412166fe/rpds_py-0.30.0-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:679ae98e00c0e8d68a7fda324e16b90fd5260945b45d3b824c892cec9eea3288", size = 370490, upload-time = "2025-11-30T20:21:33.256Z" },
- { url = "https://files.pythonhosted.org/packages/19/6a/4ba3d0fb7297ebae71171822554abe48d7cab29c28b8f9f2c04b79988c05/rpds_py-0.30.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:4cc2206b76b4f576934f0ed374b10d7ca5f457858b157ca52064bdfc26b9fc00", size = 359751, upload-time = "2025-11-30T20:21:34.591Z" },
- { url = "https://files.pythonhosted.org/packages/cd/7c/e4933565ef7f7a0818985d87c15d9d273f1a649afa6a52ea35ad011195ea/rpds_py-0.30.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:389a2d49eded1896c3d48b0136ead37c48e221b391c052fba3f4055c367f60a6", size = 389696, upload-time = "2025-11-30T20:21:36.122Z" },
- { url = "https://files.pythonhosted.org/packages/5e/01/6271a2511ad0815f00f7ed4390cf2567bec1d4b1da39e2c27a41e6e3b4de/rpds_py-0.30.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:32c8528634e1bf7121f3de08fa85b138f4e0dc47657866630611b03967f041d7", size = 403136, upload-time = "2025-11-30T20:21:37.728Z" },
- { url = "https://files.pythonhosted.org/packages/55/64/c857eb7cd7541e9b4eee9d49c196e833128a55b89a9850a9c9ac33ccf897/rpds_py-0.30.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f207f69853edd6f6700b86efb84999651baf3789e78a466431df1331608e5324", size = 524699, upload-time = "2025-11-30T20:21:38.92Z" },
- { url = "https://files.pythonhosted.org/packages/9c/ed/94816543404078af9ab26159c44f9e98e20fe47e2126d5d32c9d9948d10a/rpds_py-0.30.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:67b02ec25ba7a9e8fa74c63b6ca44cf5707f2fbfadae3ee8e7494297d56aa9df", size = 412022, upload-time = "2025-11-30T20:21:40.407Z" },
- { url = "https://files.pythonhosted.org/packages/61/b5/707f6cf0066a6412aacc11d17920ea2e19e5b2f04081c64526eb35b5c6e7/rpds_py-0.30.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0c0e95f6819a19965ff420f65578bacb0b00f251fefe2c8b23347c37174271f3", size = 390522, upload-time = "2025-11-30T20:21:42.17Z" },
- { url = "https://files.pythonhosted.org/packages/13/4e/57a85fda37a229ff4226f8cbcf09f2a455d1ed20e802ce5b2b4a7f5ed053/rpds_py-0.30.0-cp310-cp310-manylinux_2_31_riscv64.whl", hash = "sha256:a452763cc5198f2f98898eb98f7569649fe5da666c2dc6b5ddb10fde5a574221", size = 404579, upload-time = "2025-11-30T20:21:43.769Z" },
- { url = "https://files.pythonhosted.org/packages/f9/da/c9339293513ec680a721e0e16bf2bac3db6e5d7e922488de471308349bba/rpds_py-0.30.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:e0b65193a413ccc930671c55153a03ee57cecb49e6227204b04fae512eb657a7", size = 421305, upload-time = "2025-11-30T20:21:44.994Z" },
- { url = "https://files.pythonhosted.org/packages/f9/be/522cb84751114f4ad9d822ff5a1aa3c98006341895d5f084779b99596e5c/rpds_py-0.30.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:858738e9c32147f78b3ac24dc0edb6610000e56dc0f700fd5f651d0a0f0eb9ff", size = 572503, upload-time = "2025-11-30T20:21:46.91Z" },
- { url = "https://files.pythonhosted.org/packages/a2/9b/de879f7e7ceddc973ea6e4629e9b380213a6938a249e94b0cdbcc325bb66/rpds_py-0.30.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:da279aa314f00acbb803da1e76fa18666778e8a8f83484fba94526da5de2cba7", size = 598322, upload-time = "2025-11-30T20:21:48.709Z" },
- { url = "https://files.pythonhosted.org/packages/48/ac/f01fc22efec3f37d8a914fc1b2fb9bcafd56a299edbe96406f3053edea5a/rpds_py-0.30.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:7c64d38fb49b6cdeda16ab49e35fe0da2e1e9b34bc38bd78386530f218b37139", size = 560792, upload-time = "2025-11-30T20:21:50.024Z" },
- { url = "https://files.pythonhosted.org/packages/e2/da/4e2b19d0f131f35b6146425f846563d0ce036763e38913d917187307a671/rpds_py-0.30.0-cp310-cp310-win32.whl", hash = "sha256:6de2a32a1665b93233cde140ff8b3467bdb9e2af2b91079f0333a0974d12d464", size = 221901, upload-time = "2025-11-30T20:21:51.32Z" },
- { url = "https://files.pythonhosted.org/packages/96/cb/156d7a5cf4f78a7cc571465d8aec7a3c447c94f6749c5123f08438bcf7bc/rpds_py-0.30.0-cp310-cp310-win_amd64.whl", hash = "sha256:1726859cd0de969f88dc8673bdd954185b9104e05806be64bcd87badbe313169", size = 235823, upload-time = "2025-11-30T20:21:52.505Z" },
- { url = "https://files.pythonhosted.org/packages/4d/6e/f964e88b3d2abee2a82c1ac8366da848fce1c6d834dc2132c3fda3970290/rpds_py-0.30.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:a2bffea6a4ca9f01b3f8e548302470306689684e61602aa3d141e34da06cf425", size = 370157, upload-time = "2025-11-30T20:21:53.789Z" },
- { url = "https://files.pythonhosted.org/packages/94/ba/24e5ebb7c1c82e74c4e4f33b2112a5573ddc703915b13a073737b59b86e0/rpds_py-0.30.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:dc4f992dfe1e2bc3ebc7444f6c7051b4bc13cd8e33e43511e8ffd13bf407010d", size = 359676, upload-time = "2025-11-30T20:21:55.475Z" },
- { url = "https://files.pythonhosted.org/packages/84/86/04dbba1b087227747d64d80c3b74df946b986c57af0a9f0c98726d4d7a3b/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:422c3cb9856d80b09d30d2eb255d0754b23e090034e1deb4083f8004bd0761e4", size = 389938, upload-time = "2025-11-30T20:21:57.079Z" },
- { url = "https://files.pythonhosted.org/packages/42/bb/1463f0b1722b7f45431bdd468301991d1328b16cffe0b1c2918eba2c4eee/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:07ae8a593e1c3c6b82ca3292efbe73c30b61332fd612e05abee07c79359f292f", size = 402932, upload-time = "2025-11-30T20:21:58.47Z" },
- { url = "https://files.pythonhosted.org/packages/99/ee/2520700a5c1f2d76631f948b0736cdf9b0acb25abd0ca8e889b5c62ac2e3/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:12f90dd7557b6bd57f40abe7747e81e0c0b119bef015ea7726e69fe550e394a4", size = 525830, upload-time = "2025-11-30T20:21:59.699Z" },
- { url = "https://files.pythonhosted.org/packages/e0/ad/bd0331f740f5705cc555a5e17fdf334671262160270962e69a2bdef3bf76/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:99b47d6ad9a6da00bec6aabe5a6279ecd3c06a329d4aa4771034a21e335c3a97", size = 412033, upload-time = "2025-11-30T20:22:00.991Z" },
- { url = "https://files.pythonhosted.org/packages/f8/1e/372195d326549bb51f0ba0f2ecb9874579906b97e08880e7a65c3bef1a99/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:33f559f3104504506a44bb666b93a33f5d33133765b0c216a5bf2f1e1503af89", size = 390828, upload-time = "2025-11-30T20:22:02.723Z" },
- { url = "https://files.pythonhosted.org/packages/ab/2b/d88bb33294e3e0c76bc8f351a3721212713629ffca1700fa94979cb3eae8/rpds_py-0.30.0-cp311-cp311-manylinux_2_31_riscv64.whl", hash = "sha256:946fe926af6e44f3697abbc305ea168c2c31d3e3ef1058cf68f379bf0335a78d", size = 404683, upload-time = "2025-11-30T20:22:04.367Z" },
- { url = "https://files.pythonhosted.org/packages/50/32/c759a8d42bcb5289c1fac697cd92f6fe01a018dd937e62ae77e0e7f15702/rpds_py-0.30.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:495aeca4b93d465efde585977365187149e75383ad2684f81519f504f5c13038", size = 421583, upload-time = "2025-11-30T20:22:05.814Z" },
- { url = "https://files.pythonhosted.org/packages/2b/81/e729761dbd55ddf5d84ec4ff1f47857f4374b0f19bdabfcf929164da3e24/rpds_py-0.30.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d9a0ca5da0386dee0655b4ccdf46119df60e0f10da268d04fe7cc87886872ba7", size = 572496, upload-time = "2025-11-30T20:22:07.713Z" },
- { url = "https://files.pythonhosted.org/packages/14/f6/69066a924c3557c9c30baa6ec3a0aa07526305684c6f86c696b08860726c/rpds_py-0.30.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:8d6d1cc13664ec13c1b84241204ff3b12f9bb82464b8ad6e7a5d3486975c2eed", size = 598669, upload-time = "2025-11-30T20:22:09.312Z" },
- { url = "https://files.pythonhosted.org/packages/5f/48/905896b1eb8a05630d20333d1d8ffd162394127b74ce0b0784ae04498d32/rpds_py-0.30.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:3896fa1be39912cf0757753826bc8bdc8ca331a28a7c4ae46b7a21280b06bb85", size = 561011, upload-time = "2025-11-30T20:22:11.309Z" },
- { url = "https://files.pythonhosted.org/packages/22/16/cd3027c7e279d22e5eb431dd3c0fbc677bed58797fe7581e148f3f68818b/rpds_py-0.30.0-cp311-cp311-win32.whl", hash = "sha256:55f66022632205940f1827effeff17c4fa7ae1953d2b74a8581baaefb7d16f8c", size = 221406, upload-time = "2025-11-30T20:22:13.101Z" },
- { url = "https://files.pythonhosted.org/packages/fa/5b/e7b7aa136f28462b344e652ee010d4de26ee9fd16f1bfd5811f5153ccf89/rpds_py-0.30.0-cp311-cp311-win_amd64.whl", hash = "sha256:a51033ff701fca756439d641c0ad09a41d9242fa69121c7d8769604a0a629825", size = 236024, upload-time = "2025-11-30T20:22:14.853Z" },
- { url = "https://files.pythonhosted.org/packages/14/a6/364bba985e4c13658edb156640608f2c9e1d3ea3c81b27aa9d889fff0e31/rpds_py-0.30.0-cp311-cp311-win_arm64.whl", hash = "sha256:47b0ef6231c58f506ef0b74d44e330405caa8428e770fec25329ed2cb971a229", size = 229069, upload-time = "2025-11-30T20:22:16.577Z" },
{ url = "https://files.pythonhosted.org/packages/03/e7/98a2f4ac921d82f33e03f3835f5bf3a4a40aa1bfdc57975e74a97b2b4bdd/rpds_py-0.30.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:a161f20d9a43006833cd7068375a94d035714d73a172b681d8881820600abfad", size = 375086, upload-time = "2025-11-30T20:22:17.93Z" },
{ url = "https://files.pythonhosted.org/packages/4d/a1/bca7fd3d452b272e13335db8d6b0b3ecde0f90ad6f16f3328c6fb150c889/rpds_py-0.30.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6abc8880d9d036ecaafe709079969f56e876fcf107f7a8e9920ba6d5a3878d05", size = 359053, upload-time = "2025-11-30T20:22:19.297Z" },
{ url = "https://files.pythonhosted.org/packages/65/1c/ae157e83a6357eceff62ba7e52113e3ec4834a84cfe07fa4b0757a7d105f/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ca28829ae5f5d569bb62a79512c842a03a12576375d5ece7d2cadf8abe96ec28", size = 390763, upload-time = "2025-11-30T20:22:21.661Z" },
@@ -4686,18 +5646,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/85/70/92482ccffb96f5441aab93e26c4d66489eb599efdcf96fad90c14bbfb976/rpds_py-0.30.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:dbd936cde57abfee19ab3213cf9c26be06d60750e60a8e4dd85d1ab12c8b1f40", size = 556030, upload-time = "2025-11-30T20:24:10.956Z" },
{ url = "https://files.pythonhosted.org/packages/20/53/7c7e784abfa500a2b6b583b147ee4bb5a2b3747a9166bab52fec4b5b5e7d/rpds_py-0.30.0-cp314-cp314t-win32.whl", hash = "sha256:dc824125c72246d924f7f796b4f63c1e9dc810c7d9e2355864b3c3a73d59ade0", size = 211570, upload-time = "2025-11-30T20:24:12.735Z" },
{ url = "https://files.pythonhosted.org/packages/d0/02/fa464cdfbe6b26e0600b62c528b72d8608f5cc49f96b8d6e38c95d60c676/rpds_py-0.30.0-cp314-cp314t-win_amd64.whl", hash = "sha256:27f4b0e92de5bfbc6f86e43959e6edd1425c33b5e69aab0984a72047f2bcf1e3", size = 226532, upload-time = "2025-11-30T20:24:14.634Z" },
- { url = "https://files.pythonhosted.org/packages/69/71/3f34339ee70521864411f8b6992e7ab13ac30d8e4e3309e07c7361767d91/rpds_py-0.30.0-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:c2262bdba0ad4fc6fb5545660673925c2d2a5d9e2e0fb603aad545427be0fc58", size = 372292, upload-time = "2025-11-30T20:24:16.537Z" },
- { url = "https://files.pythonhosted.org/packages/57/09/f183df9b8f2d66720d2ef71075c59f7e1b336bec7ee4c48f0a2b06857653/rpds_py-0.30.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:ee6af14263f25eedc3bb918a3c04245106a42dfd4f5c2285ea6f997b1fc3f89a", size = 362128, upload-time = "2025-11-30T20:24:18.086Z" },
- { url = "https://files.pythonhosted.org/packages/7a/68/5c2594e937253457342e078f0cc1ded3dd7b2ad59afdbf2d354869110a02/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3adbb8179ce342d235c31ab8ec511e66c73faa27a47e076ccc92421add53e2bb", size = 391542, upload-time = "2025-11-30T20:24:20.092Z" },
- { url = "https://files.pythonhosted.org/packages/49/5c/31ef1afd70b4b4fbdb2800249f34c57c64beb687495b10aec0365f53dfc4/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:250fa00e9543ac9b97ac258bd37367ff5256666122c2d0f2bc97577c60a1818c", size = 404004, upload-time = "2025-11-30T20:24:22.231Z" },
- { url = "https://files.pythonhosted.org/packages/e3/63/0cfbea38d05756f3440ce6534d51a491d26176ac045e2707adc99bb6e60a/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9854cf4f488b3d57b9aaeb105f06d78e5529d3145b1e4a41750167e8c213c6d3", size = 527063, upload-time = "2025-11-30T20:24:24.302Z" },
- { url = "https://files.pythonhosted.org/packages/42/e6/01e1f72a2456678b0f618fc9a1a13f882061690893c192fcad9f2926553a/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:993914b8e560023bc0a8bf742c5f303551992dcb85e247b1e5c7f4a7d145bda5", size = 413099, upload-time = "2025-11-30T20:24:25.916Z" },
- { url = "https://files.pythonhosted.org/packages/b8/25/8df56677f209003dcbb180765520c544525e3ef21ea72279c98b9aa7c7fb/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:58edca431fb9b29950807e301826586e5bbf24163677732429770a697ffe6738", size = 392177, upload-time = "2025-11-30T20:24:27.834Z" },
- { url = "https://files.pythonhosted.org/packages/4a/b4/0a771378c5f16f8115f796d1f437950158679bcd2a7c68cf251cfb00ed5b/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_31_riscv64.whl", hash = "sha256:dea5b552272a944763b34394d04577cf0f9bd013207bc32323b5a89a53cf9c2f", size = 406015, upload-time = "2025-11-30T20:24:29.457Z" },
- { url = "https://files.pythonhosted.org/packages/36/d8/456dbba0af75049dc6f63ff295a2f92766b9d521fa00de67a2bd6427d57a/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:ba3af48635eb83d03f6c9735dfb21785303e73d22ad03d489e88adae6eab8877", size = 423736, upload-time = "2025-11-30T20:24:31.22Z" },
- { url = "https://files.pythonhosted.org/packages/13/64/b4d76f227d5c45a7e0b796c674fd81b0a6c4fbd48dc29271857d8219571c/rpds_py-0.30.0-pp311-pypy311_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:dff13836529b921e22f15cb099751209a60009731a68519630a24d61f0b1b30a", size = 573981, upload-time = "2025-11-30T20:24:32.934Z" },
- { url = "https://files.pythonhosted.org/packages/20/91/092bacadeda3edf92bf743cc96a7be133e13a39cdbfd7b5082e7ab638406/rpds_py-0.30.0-pp311-pypy311_pp73-musllinux_1_2_i686.whl", hash = "sha256:1b151685b23929ab7beec71080a8889d4d6d9fa9a983d213f07121205d48e2c4", size = 599782, upload-time = "2025-11-30T20:24:35.169Z" },
- { url = "https://files.pythonhosted.org/packages/d1/b7/b95708304cd49b7b6f82fdd039f1748b66ec2b21d6a45180910802f1abf1/rpds_py-0.30.0-pp311-pypy311_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:ac37f9f516c51e5753f27dfdef11a88330f04de2d564be3991384b2f3535d02e", size = 562191, upload-time = "2025-11-30T20:24:36.853Z" },
]
[[package]]
@@ -4727,16 +5675,16 @@ wheels = [
[[package]]
name = "s3fs"
-version = "2025.10.0"
+version = "2026.2.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "aiobotocore" },
{ name = "aiohttp" },
- { name = "fsspec" },
+ { name = "fsspec", version = "2026.2.0", source = { registry = "https://pypi.org/simple" } },
]
-sdist = { url = "https://files.pythonhosted.org/packages/bb/ee/7cf7de3b17ef6db10b027cc9f8a1108ceb6333e267943e666a35882b1474/s3fs-2025.10.0.tar.gz", hash = "sha256:e8be6cddc77aceea1681ece0f472c3a7f8ef71a0d2acddb1cc92bb6afa3e9e4f", size = 80383, upload-time = "2025-10-30T15:06:04.647Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/fa/be/392c8c5e0da9bfa139e41084690dd49a5e3e931099f78f52d3f6070105c6/s3fs-2026.2.0.tar.gz", hash = "sha256:91cb2a9f76e35643b76eeac3f47a6165172bb3def671f76b9111c8dd5779a2ac", size = 84152, upload-time = "2026-02-05T21:57:57.968Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/2d/fc/56cba14af8ad8fd020c85b6e44328520ac55939bb1f9d01444ad470504cb/s3fs-2025.10.0-py3-none-any.whl", hash = "sha256:da7ef25efc1541f5fca8e1116361e49ea1081f83f4e8001fbd77347c625da28a", size = 30357, upload-time = "2025-10-30T15:06:03.48Z" },
+ { url = "https://files.pythonhosted.org/packages/57/e1/64c264db50b68de8a438b60ceeb921b2f22da3ebb7ad6255150225d0beac/s3fs-2026.2.0-py3-none-any.whl", hash = "sha256:65198835b86b1d5771112b0085d1da52a6ede36508b1aaa6cae2aedc765dfe10", size = 31328, upload-time = "2026-02-05T21:57:56.532Z" },
]
[[package]]
@@ -4759,151 +5707,67 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/4a/d8/0c8a7dc9b41dcac53c4cbf9df2b9c83e0e0097203de8b37a712b345c0be5/safetensors-0.7.0-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b0f6d66c1c538d5a94a73aa9ddca8ccc4227e6c9ff555322ea40bdd142391dd4", size = 677368, upload-time = "2025-11-19T15:18:41.627Z" },
{ url = "https://files.pythonhosted.org/packages/05/e5/cb4b713c8a93469e3c5be7c3f8d77d307e65fe89673e731f5c2bfd0a9237/safetensors-0.7.0-cp38-abi3-win32.whl", hash = "sha256:c74af94bf3ac15ac4d0f2a7c7b4663a15f8c2ab15ed0fc7531ca61d0835eccba", size = 326423, upload-time = "2025-11-19T15:18:45.74Z" },
{ url = "https://files.pythonhosted.org/packages/5d/e6/ec8471c8072382cb91233ba7267fd931219753bb43814cbc71757bfd4dab/safetensors-0.7.0-cp38-abi3-win_amd64.whl", hash = "sha256:d1239932053f56f3456f32eb9625590cc7582e905021f94636202a864d470755", size = 341380, upload-time = "2025-11-19T15:18:44.427Z" },
- { url = "https://files.pythonhosted.org/packages/a7/6a/4d08d89a6fcbe905c5ae68b8b34f0791850882fc19782d0d02c65abbdf3b/safetensors-0.7.0-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f4729811a6640d019a4b7ba8638ee2fd21fa5ca8c7e7bdf0fed62068fcaac737", size = 492430, upload-time = "2025-11-19T15:18:11.884Z" },
- { url = "https://files.pythonhosted.org/packages/dd/29/59ed8152b30f72c42d00d241e58eaca558ae9dbfa5695206e2e0f54c7063/safetensors-0.7.0-pp310-pypy310_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:12f49080303fa6bb424b362149a12949dfbbf1e06811a88f2307276b0c131afd", size = 503977, upload-time = "2025-11-19T15:18:17.523Z" },
- { url = "https://files.pythonhosted.org/packages/d3/0b/4811bfec67fa260e791369b16dab105e4bae82686120554cc484064e22b4/safetensors-0.7.0-pp310-pypy310_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0071bffba4150c2f46cae1432d31995d77acfd9f8db598b5d1a2ce67e8440ad2", size = 623890, upload-time = "2025-11-19T15:18:22.666Z" },
- { url = "https://files.pythonhosted.org/packages/58/5b/632a58724221ef03d78ab65062e82a1010e1bef8e8e0b9d7c6d7b8044841/safetensors-0.7.0-pp310-pypy310_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:473b32699f4200e69801bf5abf93f1a4ecd432a70984df164fc22ccf39c4a6f3", size = 531885, upload-time = "2025-11-19T15:18:27.146Z" },
-]
-
-[[package]]
-name = "scipy"
-version = "1.15.3"
-source = { registry = "https://pypi.org/simple" }
-resolution-markers = [
- "python_full_version < '3.11' and sys_platform == 'linux'",
- "python_full_version < '3.11' and sys_platform != 'linux'",
-]
-dependencies = [
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/0f/37/6964b830433e654ec7485e45a00fc9a27cf868d622838f6b6d9c5ec0d532/scipy-1.15.3.tar.gz", hash = "sha256:eae3cf522bc7df64b42cad3925c876e1b0b6c35c1337c93e12c0f366f55b0eaf", size = 59419214, upload-time = "2025-05-08T16:13:05.955Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/78/2f/4966032c5f8cc7e6a60f1b2e0ad686293b9474b65246b0c642e3ef3badd0/scipy-1.15.3-cp310-cp310-macosx_10_13_x86_64.whl", hash = "sha256:a345928c86d535060c9c2b25e71e87c39ab2f22fc96e9636bd74d1dbf9de448c", size = 38702770, upload-time = "2025-05-08T16:04:20.849Z" },
- { url = "https://files.pythonhosted.org/packages/a0/6e/0c3bf90fae0e910c274db43304ebe25a6b391327f3f10b5dcc638c090795/scipy-1.15.3-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:ad3432cb0f9ed87477a8d97f03b763fd1d57709f1bbde3c9369b1dff5503b253", size = 30094511, upload-time = "2025-05-08T16:04:27.103Z" },
- { url = "https://files.pythonhosted.org/packages/ea/b1/4deb37252311c1acff7f101f6453f0440794f51b6eacb1aad4459a134081/scipy-1.15.3-cp310-cp310-macosx_14_0_arm64.whl", hash = "sha256:aef683a9ae6eb00728a542b796f52a5477b78252edede72b8327a886ab63293f", size = 22368151, upload-time = "2025-05-08T16:04:31.731Z" },
- { url = "https://files.pythonhosted.org/packages/38/7d/f457626e3cd3c29b3a49ca115a304cebb8cc6f31b04678f03b216899d3c6/scipy-1.15.3-cp310-cp310-macosx_14_0_x86_64.whl", hash = "sha256:1c832e1bd78dea67d5c16f786681b28dd695a8cb1fb90af2e27580d3d0967e92", size = 25121732, upload-time = "2025-05-08T16:04:36.596Z" },
- { url = "https://files.pythonhosted.org/packages/db/0a/92b1de4a7adc7a15dcf5bddc6e191f6f29ee663b30511ce20467ef9b82e4/scipy-1.15.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:263961f658ce2165bbd7b99fa5135195c3a12d9bef045345016b8b50c315cb82", size = 35547617, upload-time = "2025-05-08T16:04:43.546Z" },
- { url = "https://files.pythonhosted.org/packages/8e/6d/41991e503e51fc1134502694c5fa7a1671501a17ffa12716a4a9151af3df/scipy-1.15.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9e2abc762b0811e09a0d3258abee2d98e0c703eee49464ce0069590846f31d40", size = 37662964, upload-time = "2025-05-08T16:04:49.431Z" },
- { url = "https://files.pythonhosted.org/packages/25/e1/3df8f83cb15f3500478c889be8fb18700813b95e9e087328230b98d547ff/scipy-1.15.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:ed7284b21a7a0c8f1b6e5977ac05396c0d008b89e05498c8b7e8f4a1423bba0e", size = 37238749, upload-time = "2025-05-08T16:04:55.215Z" },
- { url = "https://files.pythonhosted.org/packages/93/3e/b3257cf446f2a3533ed7809757039016b74cd6f38271de91682aa844cfc5/scipy-1.15.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:5380741e53df2c566f4d234b100a484b420af85deb39ea35a1cc1be84ff53a5c", size = 40022383, upload-time = "2025-05-08T16:05:01.914Z" },
- { url = "https://files.pythonhosted.org/packages/d1/84/55bc4881973d3f79b479a5a2e2df61c8c9a04fcb986a213ac9c02cfb659b/scipy-1.15.3-cp310-cp310-win_amd64.whl", hash = "sha256:9d61e97b186a57350f6d6fd72640f9e99d5a4a2b8fbf4b9ee9a841eab327dc13", size = 41259201, upload-time = "2025-05-08T16:05:08.166Z" },
- { url = "https://files.pythonhosted.org/packages/96/ab/5cc9f80f28f6a7dff646c5756e559823614a42b1939d86dd0ed550470210/scipy-1.15.3-cp311-cp311-macosx_10_13_x86_64.whl", hash = "sha256:993439ce220d25e3696d1b23b233dd010169b62f6456488567e830654ee37a6b", size = 38714255, upload-time = "2025-05-08T16:05:14.596Z" },
- { url = "https://files.pythonhosted.org/packages/4a/4a/66ba30abe5ad1a3ad15bfb0b59d22174012e8056ff448cb1644deccbfed2/scipy-1.15.3-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:34716e281f181a02341ddeaad584205bd2fd3c242063bd3423d61ac259ca7eba", size = 30111035, upload-time = "2025-05-08T16:05:20.152Z" },
- { url = "https://files.pythonhosted.org/packages/4b/fa/a7e5b95afd80d24313307f03624acc65801846fa75599034f8ceb9e2cbf6/scipy-1.15.3-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:3b0334816afb8b91dab859281b1b9786934392aa3d527cd847e41bb6f45bee65", size = 22384499, upload-time = "2025-05-08T16:05:24.494Z" },
- { url = "https://files.pythonhosted.org/packages/17/99/f3aaddccf3588bb4aea70ba35328c204cadd89517a1612ecfda5b2dd9d7a/scipy-1.15.3-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:6db907c7368e3092e24919b5e31c76998b0ce1684d51a90943cb0ed1b4ffd6c1", size = 25152602, upload-time = "2025-05-08T16:05:29.313Z" },
- { url = "https://files.pythonhosted.org/packages/56/c5/1032cdb565f146109212153339f9cb8b993701e9fe56b1c97699eee12586/scipy-1.15.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:721d6b4ef5dc82ca8968c25b111e307083d7ca9091bc38163fb89243e85e3889", size = 35503415, upload-time = "2025-05-08T16:05:34.699Z" },
- { url = "https://files.pythonhosted.org/packages/bd/37/89f19c8c05505d0601ed5650156e50eb881ae3918786c8fd7262b4ee66d3/scipy-1.15.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:39cb9c62e471b1bb3750066ecc3a3f3052b37751c7c3dfd0fd7e48900ed52982", size = 37652622, upload-time = "2025-05-08T16:05:40.762Z" },
- { url = "https://files.pythonhosted.org/packages/7e/31/be59513aa9695519b18e1851bb9e487de66f2d31f835201f1b42f5d4d475/scipy-1.15.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:795c46999bae845966368a3c013e0e00947932d68e235702b5c3f6ea799aa8c9", size = 37244796, upload-time = "2025-05-08T16:05:48.119Z" },
- { url = "https://files.pythonhosted.org/packages/10/c0/4f5f3eeccc235632aab79b27a74a9130c6c35df358129f7ac8b29f562ac7/scipy-1.15.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:18aaacb735ab38b38db42cb01f6b92a2d0d4b6aabefeb07f02849e47f8fb3594", size = 40047684, upload-time = "2025-05-08T16:05:54.22Z" },
- { url = "https://files.pythonhosted.org/packages/ab/a7/0ddaf514ce8a8714f6ed243a2b391b41dbb65251affe21ee3077ec45ea9a/scipy-1.15.3-cp311-cp311-win_amd64.whl", hash = "sha256:ae48a786a28412d744c62fd7816a4118ef97e5be0bee968ce8f0a2fba7acf3bb", size = 41246504, upload-time = "2025-05-08T16:06:00.437Z" },
- { url = "https://files.pythonhosted.org/packages/37/4b/683aa044c4162e10ed7a7ea30527f2cbd92e6999c10a8ed8edb253836e9c/scipy-1.15.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6ac6310fdbfb7aa6612408bd2f07295bcbd3fda00d2d702178434751fe48e019", size = 38766735, upload-time = "2025-05-08T16:06:06.471Z" },
- { url = "https://files.pythonhosted.org/packages/7b/7e/f30be3d03de07f25dc0ec926d1681fed5c732d759ac8f51079708c79e680/scipy-1.15.3-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:185cd3d6d05ca4b44a8f1595af87f9c372bb6acf9c808e99aa3e9aa03bd98cf6", size = 30173284, upload-time = "2025-05-08T16:06:11.686Z" },
- { url = "https://files.pythonhosted.org/packages/07/9c/0ddb0d0abdabe0d181c1793db51f02cd59e4901da6f9f7848e1f96759f0d/scipy-1.15.3-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:05dc6abcd105e1a29f95eada46d4a3f251743cfd7d3ae8ddb4088047f24ea477", size = 22446958, upload-time = "2025-05-08T16:06:15.97Z" },
- { url = "https://files.pythonhosted.org/packages/af/43/0bce905a965f36c58ff80d8bea33f1f9351b05fad4beaad4eae34699b7a1/scipy-1.15.3-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:06efcba926324df1696931a57a176c80848ccd67ce6ad020c810736bfd58eb1c", size = 25242454, upload-time = "2025-05-08T16:06:20.394Z" },
- { url = "https://files.pythonhosted.org/packages/56/30/a6f08f84ee5b7b28b4c597aca4cbe545535c39fe911845a96414700b64ba/scipy-1.15.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c05045d8b9bfd807ee1b9f38761993297b10b245f012b11b13b91ba8945f7e45", size = 35210199, upload-time = "2025-05-08T16:06:26.159Z" },
- { url = "https://files.pythonhosted.org/packages/0b/1f/03f52c282437a168ee2c7c14a1a0d0781a9a4a8962d84ac05c06b4c5b555/scipy-1.15.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:271e3713e645149ea5ea3e97b57fdab61ce61333f97cfae392c28ba786f9bb49", size = 37309455, upload-time = "2025-05-08T16:06:32.778Z" },
- { url = "https://files.pythonhosted.org/packages/89/b1/fbb53137f42c4bf630b1ffdfc2151a62d1d1b903b249f030d2b1c0280af8/scipy-1.15.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:6cfd56fc1a8e53f6e89ba3a7a7251f7396412d655bca2aa5611c8ec9a6784a1e", size = 36885140, upload-time = "2025-05-08T16:06:39.249Z" },
- { url = "https://files.pythonhosted.org/packages/2e/2e/025e39e339f5090df1ff266d021892694dbb7e63568edcfe43f892fa381d/scipy-1.15.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:0ff17c0bb1cb32952c09217d8d1eed9b53d1463e5f1dd6052c7857f83127d539", size = 39710549, upload-time = "2025-05-08T16:06:45.729Z" },
- { url = "https://files.pythonhosted.org/packages/e6/eb/3bf6ea8ab7f1503dca3a10df2e4b9c3f6b3316df07f6c0ded94b281c7101/scipy-1.15.3-cp312-cp312-win_amd64.whl", hash = "sha256:52092bc0472cfd17df49ff17e70624345efece4e1a12b23783a1ac59a1b728ed", size = 40966184, upload-time = "2025-05-08T16:06:52.623Z" },
- { url = "https://files.pythonhosted.org/packages/73/18/ec27848c9baae6e0d6573eda6e01a602e5649ee72c27c3a8aad673ebecfd/scipy-1.15.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:2c620736bcc334782e24d173c0fdbb7590a0a436d2fdf39310a8902505008759", size = 38728256, upload-time = "2025-05-08T16:06:58.696Z" },
- { url = "https://files.pythonhosted.org/packages/74/cd/1aef2184948728b4b6e21267d53b3339762c285a46a274ebb7863c9e4742/scipy-1.15.3-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:7e11270a000969409d37ed399585ee530b9ef6aa99d50c019de4cb01e8e54e62", size = 30109540, upload-time = "2025-05-08T16:07:04.209Z" },
- { url = "https://files.pythonhosted.org/packages/5b/d8/59e452c0a255ec352bd0a833537a3bc1bfb679944c4938ab375b0a6b3a3e/scipy-1.15.3-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:8c9ed3ba2c8a2ce098163a9bdb26f891746d02136995df25227a20e71c396ebb", size = 22383115, upload-time = "2025-05-08T16:07:08.998Z" },
- { url = "https://files.pythonhosted.org/packages/08/f5/456f56bbbfccf696263b47095291040655e3cbaf05d063bdc7c7517f32ac/scipy-1.15.3-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:0bdd905264c0c9cfa74a4772cdb2070171790381a5c4d312c973382fc6eaf730", size = 25163884, upload-time = "2025-05-08T16:07:14.091Z" },
- { url = "https://files.pythonhosted.org/packages/a2/66/a9618b6a435a0f0c0b8a6d0a2efb32d4ec5a85f023c2b79d39512040355b/scipy-1.15.3-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:79167bba085c31f38603e11a267d862957cbb3ce018d8b38f79ac043bc92d825", size = 35174018, upload-time = "2025-05-08T16:07:19.427Z" },
- { url = "https://files.pythonhosted.org/packages/b5/09/c5b6734a50ad4882432b6bb7c02baf757f5b2f256041da5df242e2d7e6b6/scipy-1.15.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c9deabd6d547aee2c9a81dee6cc96c6d7e9a9b1953f74850c179f91fdc729cb7", size = 37269716, upload-time = "2025-05-08T16:07:25.712Z" },
- { url = "https://files.pythonhosted.org/packages/77/0a/eac00ff741f23bcabd352731ed9b8995a0a60ef57f5fd788d611d43d69a1/scipy-1.15.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:dde4fc32993071ac0c7dd2d82569e544f0bdaff66269cb475e0f369adad13f11", size = 36872342, upload-time = "2025-05-08T16:07:31.468Z" },
- { url = "https://files.pythonhosted.org/packages/fe/54/4379be86dd74b6ad81551689107360d9a3e18f24d20767a2d5b9253a3f0a/scipy-1.15.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:f77f853d584e72e874d87357ad70f44b437331507d1c311457bed8ed2b956126", size = 39670869, upload-time = "2025-05-08T16:07:38.002Z" },
- { url = "https://files.pythonhosted.org/packages/87/2e/892ad2862ba54f084ffe8cc4a22667eaf9c2bcec6d2bff1d15713c6c0703/scipy-1.15.3-cp313-cp313-win_amd64.whl", hash = "sha256:b90ab29d0c37ec9bf55424c064312930ca5f4bde15ee8619ee44e69319aab163", size = 40988851, upload-time = "2025-05-08T16:08:33.671Z" },
- { url = "https://files.pythonhosted.org/packages/1b/e9/7a879c137f7e55b30d75d90ce3eb468197646bc7b443ac036ae3fe109055/scipy-1.15.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:3ac07623267feb3ae308487c260ac684b32ea35fd81e12845039952f558047b8", size = 38863011, upload-time = "2025-05-08T16:07:44.039Z" },
- { url = "https://files.pythonhosted.org/packages/51/d1/226a806bbd69f62ce5ef5f3ffadc35286e9fbc802f606a07eb83bf2359de/scipy-1.15.3-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:6487aa99c2a3d509a5227d9a5e889ff05830a06b2ce08ec30df6d79db5fcd5c5", size = 30266407, upload-time = "2025-05-08T16:07:49.891Z" },
- { url = "https://files.pythonhosted.org/packages/e5/9b/f32d1d6093ab9eeabbd839b0f7619c62e46cc4b7b6dbf05b6e615bbd4400/scipy-1.15.3-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:50f9e62461c95d933d5c5ef4a1f2ebf9a2b4e83b0db374cb3f1de104d935922e", size = 22540030, upload-time = "2025-05-08T16:07:54.121Z" },
- { url = "https://files.pythonhosted.org/packages/e7/29/c278f699b095c1a884f29fda126340fcc201461ee8bfea5c8bdb1c7c958b/scipy-1.15.3-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:14ed70039d182f411ffc74789a16df3835e05dc469b898233a245cdfd7f162cb", size = 25218709, upload-time = "2025-05-08T16:07:58.506Z" },
- { url = "https://files.pythonhosted.org/packages/24/18/9e5374b617aba742a990581373cd6b68a2945d65cc588482749ef2e64467/scipy-1.15.3-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0a769105537aa07a69468a0eefcd121be52006db61cdd8cac8a0e68980bbb723", size = 34809045, upload-time = "2025-05-08T16:08:03.929Z" },
- { url = "https://files.pythonhosted.org/packages/e1/fe/9c4361e7ba2927074360856db6135ef4904d505e9b3afbbcb073c4008328/scipy-1.15.3-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9db984639887e3dffb3928d118145ffe40eff2fa40cb241a306ec57c219ebbbb", size = 36703062, upload-time = "2025-05-08T16:08:09.558Z" },
- { url = "https://files.pythonhosted.org/packages/b7/8e/038ccfe29d272b30086b25a4960f757f97122cb2ec42e62b460d02fe98e9/scipy-1.15.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:40e54d5c7e7ebf1aa596c374c49fa3135f04648a0caabcb66c52884b943f02b4", size = 36393132, upload-time = "2025-05-08T16:08:15.34Z" },
- { url = "https://files.pythonhosted.org/packages/10/7e/5c12285452970be5bdbe8352c619250b97ebf7917d7a9a9e96b8a8140f17/scipy-1.15.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:5e721fed53187e71d0ccf382b6bf977644c533e506c4d33c3fb24de89f5c3ed5", size = 38979503, upload-time = "2025-05-08T16:08:21.513Z" },
- { url = "https://files.pythonhosted.org/packages/81/06/0a5e5349474e1cbc5757975b21bd4fad0e72ebf138c5592f191646154e06/scipy-1.15.3-cp313-cp313t-win_amd64.whl", hash = "sha256:76ad1fb5f8752eabf0fa02e4cc0336b4e8f021e2d5f061ed37d6d264db35e3ca", size = 40308097, upload-time = "2025-05-08T16:08:27.627Z" },
]
[[package]]
name = "scipy"
-version = "1.17.0"
-source = { registry = "https://pypi.org/simple" }
-resolution-markers = [
- "python_full_version >= '3.14' and sys_platform == 'linux'",
- "python_full_version == '3.13.*' and sys_platform == 'linux'",
- "python_full_version == '3.12.*' and sys_platform == 'linux'",
- "python_full_version >= '3.14' and sys_platform != 'linux'",
- "python_full_version == '3.13.*' and sys_platform != 'linux'",
- "python_full_version == '3.12.*' and sys_platform != 'linux'",
- "python_full_version == '3.11.*' and sys_platform == 'linux'",
- "python_full_version == '3.11.*' and sys_platform != 'linux'",
-]
-dependencies = [
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/56/3e/9cca699f3486ce6bc12ff46dc2031f1ec8eb9ccc9a320fdaf925f1417426/scipy-1.17.0.tar.gz", hash = "sha256:2591060c8e648d8b96439e111ac41fd8342fdeff1876be2e19dea3fe8930454e", size = 30396830, upload-time = "2026-01-10T21:34:23.009Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/1e/4b/c89c131aa87cad2b77a54eb0fb94d633a842420fa7e919dc2f922037c3d8/scipy-1.17.0-cp311-cp311-macosx_10_14_x86_64.whl", hash = "sha256:2abd71643797bd8a106dff97894ff7869eeeb0af0f7a5ce02e4227c6a2e9d6fd", size = 31381316, upload-time = "2026-01-10T21:24:33.42Z" },
- { url = "https://files.pythonhosted.org/packages/5e/5f/a6b38f79a07d74989224d5f11b55267714707582908a5f1ae854cf9a9b84/scipy-1.17.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:ef28d815f4d2686503e5f4f00edc387ae58dfd7a2f42e348bb53359538f01558", size = 27966760, upload-time = "2026-01-10T21:24:38.911Z" },
- { url = "https://files.pythonhosted.org/packages/c1/20/095ad24e031ee8ed3c5975954d816b8e7e2abd731e04f8be573de8740885/scipy-1.17.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:272a9f16d6bb4667e8b50d25d71eddcc2158a214df1b566319298de0939d2ab7", size = 20138701, upload-time = "2026-01-10T21:24:43.249Z" },
- { url = "https://files.pythonhosted.org/packages/89/11/4aad2b3858d0337756f3323f8960755704e530b27eb2a94386c970c32cbe/scipy-1.17.0-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:7204fddcbec2fe6598f1c5fdf027e9f259106d05202a959a9f1aecf036adc9f6", size = 22480574, upload-time = "2026-01-10T21:24:47.266Z" },
- { url = "https://files.pythonhosted.org/packages/85/bd/f5af70c28c6da2227e510875cadf64879855193a687fb19951f0f44cfd6b/scipy-1.17.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fc02c37a5639ee67d8fb646ffded6d793c06c5622d36b35cfa8fe5ececb8f042", size = 32862414, upload-time = "2026-01-10T21:24:52.566Z" },
- { url = "https://files.pythonhosted.org/packages/ef/df/df1457c4df3826e908879fe3d76bc5b6e60aae45f4ee42539512438cfd5d/scipy-1.17.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:dac97a27520d66c12a34fd90a4fe65f43766c18c0d6e1c0a80f114d2260080e4", size = 35112380, upload-time = "2026-01-10T21:24:58.433Z" },
- { url = "https://files.pythonhosted.org/packages/5f/bb/88e2c16bd1dd4de19d80d7c5e238387182993c2fb13b4b8111e3927ad422/scipy-1.17.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:ebb7446a39b3ae0fe8f416a9a3fdc6fba3f11c634f680f16a239c5187bc487c0", size = 34922676, upload-time = "2026-01-10T21:25:04.287Z" },
- { url = "https://files.pythonhosted.org/packages/02/ba/5120242cc735f71fc002cff0303d536af4405eb265f7c60742851e7ccfe9/scipy-1.17.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:474da16199f6af66601a01546144922ce402cb17362e07d82f5a6cf8f963e449", size = 37507599, upload-time = "2026-01-10T21:25:09.851Z" },
- { url = "https://files.pythonhosted.org/packages/52/c8/08629657ac6c0da198487ce8cd3de78e02cfde42b7f34117d56a3fe249dc/scipy-1.17.0-cp311-cp311-win_amd64.whl", hash = "sha256:255c0da161bd7b32a6c898e7891509e8a9289f0b1c6c7d96142ee0d2b114c2ea", size = 36380284, upload-time = "2026-01-10T21:25:15.632Z" },
- { url = "https://files.pythonhosted.org/packages/6c/4a/465f96d42c6f33ad324a40049dfd63269891db9324aa66c4a1c108c6f994/scipy-1.17.0-cp311-cp311-win_arm64.whl", hash = "sha256:85b0ac3ad17fa3be50abd7e69d583d98792d7edc08367e01445a1e2076005379", size = 24370427, upload-time = "2026-01-10T21:25:20.514Z" },
- { url = "https://files.pythonhosted.org/packages/0b/11/7241a63e73ba5a516f1930ac8d5b44cbbfabd35ac73a2d08ca206df007c4/scipy-1.17.0-cp312-cp312-macosx_10_14_x86_64.whl", hash = "sha256:0d5018a57c24cb1dd828bcf51d7b10e65986d549f52ef5adb6b4d1ded3e32a57", size = 31364580, upload-time = "2026-01-10T21:25:25.717Z" },
- { url = "https://files.pythonhosted.org/packages/ed/1d/5057f812d4f6adc91a20a2d6f2ebcdb517fdbc87ae3acc5633c9b97c8ba5/scipy-1.17.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:88c22af9e5d5a4f9e027e26772cc7b5922fab8bcc839edb3ae33de404feebd9e", size = 27969012, upload-time = "2026-01-10T21:25:30.921Z" },
- { url = "https://files.pythonhosted.org/packages/e3/21/f6ec556c1e3b6ec4e088da667d9987bb77cc3ab3026511f427dc8451187d/scipy-1.17.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:f3cd947f20fe17013d401b64e857c6b2da83cae567adbb75b9dcba865abc66d8", size = 20140691, upload-time = "2026-01-10T21:25:34.802Z" },
- { url = "https://files.pythonhosted.org/packages/7a/fe/5e5ad04784964ba964a96f16c8d4676aa1b51357199014dce58ab7ec5670/scipy-1.17.0-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:e8c0b331c2c1f531eb51f1b4fc9ba709521a712cce58f1aa627bc007421a5306", size = 22463015, upload-time = "2026-01-10T21:25:39.277Z" },
- { url = "https://files.pythonhosted.org/packages/4a/69/7c347e857224fcaf32a34a05183b9d8a7aca25f8f2d10b8a698b8388561a/scipy-1.17.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5194c445d0a1c7a6c1a4a4681b6b7c71baad98ff66d96b949097e7513c9d6742", size = 32724197, upload-time = "2026-01-10T21:25:44.084Z" },
- { url = "https://files.pythonhosted.org/packages/d1/fe/66d73b76d378ba8cc2fe605920c0c75092e3a65ae746e1e767d9d020a75a/scipy-1.17.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9eeb9b5f5997f75507814ed9d298ab23f62cf79f5a3ef90031b1ee2506abdb5b", size = 35009148, upload-time = "2026-01-10T21:25:50.591Z" },
- { url = "https://files.pythonhosted.org/packages/af/07/07dec27d9dc41c18d8c43c69e9e413431d20c53a0339c388bcf72f353c4b/scipy-1.17.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:40052543f7bbe921df4408f46003d6f01c6af109b9e2c8a66dd1cf6cf57f7d5d", size = 34798766, upload-time = "2026-01-10T21:25:59.41Z" },
- { url = "https://files.pythonhosted.org/packages/81/61/0470810c8a093cdacd4ba7504b8a218fd49ca070d79eca23a615f5d9a0b0/scipy-1.17.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:0cf46c8013fec9d3694dc572f0b54100c28405d55d3e2cb15e2895b25057996e", size = 37405953, upload-time = "2026-01-10T21:26:07.75Z" },
- { url = "https://files.pythonhosted.org/packages/92/ce/672ed546f96d5d41ae78c4b9b02006cedd0b3d6f2bf5bb76ea455c320c28/scipy-1.17.0-cp312-cp312-win_amd64.whl", hash = "sha256:0937a0b0d8d593a198cededd4c439a0ea216a3f36653901ea1f3e4be949056f8", size = 36328121, upload-time = "2026-01-10T21:26:16.509Z" },
- { url = "https://files.pythonhosted.org/packages/9d/21/38165845392cae67b61843a52c6455d47d0cc2a40dd495c89f4362944654/scipy-1.17.0-cp312-cp312-win_arm64.whl", hash = "sha256:f603d8a5518c7426414d1d8f82e253e454471de682ce5e39c29adb0df1efb86b", size = 24314368, upload-time = "2026-01-10T21:26:23.087Z" },
- { url = "https://files.pythonhosted.org/packages/0c/51/3468fdfd49387ddefee1636f5cf6d03ce603b75205bf439bbf0e62069bfd/scipy-1.17.0-cp313-cp313-macosx_10_14_x86_64.whl", hash = "sha256:65ec32f3d32dfc48c72df4291345dae4f048749bc8d5203ee0a3f347f96c5ce6", size = 31344101, upload-time = "2026-01-10T21:26:30.25Z" },
- { url = "https://files.pythonhosted.org/packages/b2/9a/9406aec58268d437636069419e6977af953d1e246df941d42d3720b7277b/scipy-1.17.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:1f9586a58039d7229ce77b52f8472c972448cded5736eaf102d5658bbac4c269", size = 27950385, upload-time = "2026-01-10T21:26:36.801Z" },
- { url = "https://files.pythonhosted.org/packages/4f/98/e7342709e17afdfd1b26b56ae499ef4939b45a23a00e471dfb5375eea205/scipy-1.17.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:9fad7d3578c877d606b1150135c2639e9de9cecd3705caa37b66862977cc3e72", size = 20122115, upload-time = "2026-01-10T21:26:42.107Z" },
- { url = "https://files.pythonhosted.org/packages/fd/0e/9eeeb5357a64fd157cbe0302c213517c541cc16b8486d82de251f3c68ede/scipy-1.17.0-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:423ca1f6584fc03936972b5f7c06961670dbba9f234e71676a7c7ccf938a0d61", size = 22442402, upload-time = "2026-01-10T21:26:48.029Z" },
- { url = "https://files.pythonhosted.org/packages/c9/10/be13397a0e434f98e0c79552b2b584ae5bb1c8b2be95db421533bbca5369/scipy-1.17.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fe508b5690e9eaaa9467fc047f833af58f1152ae51a0d0aed67aa5801f4dd7d6", size = 32696338, upload-time = "2026-01-10T21:26:55.521Z" },
- { url = "https://files.pythonhosted.org/packages/63/1e/12fbf2a3bb240161651c94bb5cdd0eae5d4e8cc6eaeceb74ab07b12a753d/scipy-1.17.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6680f2dfd4f6182e7d6db161344537da644d1cf85cf293f015c60a17ecf08752", size = 34977201, upload-time = "2026-01-10T21:27:03.501Z" },
- { url = "https://files.pythonhosted.org/packages/19/5b/1a63923e23ccd20bd32156d7dd708af5bbde410daa993aa2500c847ab2d2/scipy-1.17.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:eec3842ec9ac9de5917899b277428886042a93db0b227ebbe3a333b64ec7643d", size = 34777384, upload-time = "2026-01-10T21:27:11.423Z" },
- { url = "https://files.pythonhosted.org/packages/39/22/b5da95d74edcf81e540e467202a988c50fef41bd2011f46e05f72ba07df6/scipy-1.17.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:d7425fcafbc09a03731e1bc05581f5fad988e48c6a861f441b7ab729a49a55ea", size = 37379586, upload-time = "2026-01-10T21:27:20.171Z" },
- { url = "https://files.pythonhosted.org/packages/b9/b6/8ac583d6da79e7b9e520579f03007cb006f063642afd6b2eeb16b890bf93/scipy-1.17.0-cp313-cp313-win_amd64.whl", hash = "sha256:87b411e42b425b84777718cc41516b8a7e0795abfa8e8e1d573bf0ef014f0812", size = 36287211, upload-time = "2026-01-10T21:28:43.122Z" },
- { url = "https://files.pythonhosted.org/packages/55/fb/7db19e0b3e52f882b420417644ec81dd57eeef1bd1705b6f689d8ff93541/scipy-1.17.0-cp313-cp313-win_arm64.whl", hash = "sha256:357ca001c6e37601066092e7c89cca2f1ce74e2a520ca78d063a6d2201101df2", size = 24312646, upload-time = "2026-01-10T21:28:49.893Z" },
- { url = "https://files.pythonhosted.org/packages/20/b6/7feaa252c21cc7aff335c6c55e1b90ab3e3306da3f048109b8b639b94648/scipy-1.17.0-cp313-cp313t-macosx_10_14_x86_64.whl", hash = "sha256:ec0827aa4d36cb79ff1b81de898e948a51ac0b9b1c43e4a372c0508c38c0f9a3", size = 31693194, upload-time = "2026-01-10T21:27:27.454Z" },
- { url = "https://files.pythonhosted.org/packages/76/bb/bbb392005abce039fb7e672cb78ac7d158700e826b0515cab6b5b60c26fb/scipy-1.17.0-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:819fc26862b4b3c73a60d486dbb919202f3d6d98c87cf20c223511429f2d1a97", size = 28365415, upload-time = "2026-01-10T21:27:34.26Z" },
- { url = "https://files.pythonhosted.org/packages/37/da/9d33196ecc99fba16a409c691ed464a3a283ac454a34a13a3a57c0d66f3a/scipy-1.17.0-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:363ad4ae2853d88ebcde3ae6ec46ccca903ea9835ee8ba543f12f575e7b07e4e", size = 20537232, upload-time = "2026-01-10T21:27:40.306Z" },
- { url = "https://files.pythonhosted.org/packages/56/9d/f4b184f6ddb28e9a5caea36a6f98e8ecd2a524f9127354087ce780885d83/scipy-1.17.0-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:979c3a0ff8e5ba254d45d59ebd38cde48fce4f10b5125c680c7a4bfe177aab07", size = 22791051, upload-time = "2026-01-10T21:27:46.539Z" },
- { url = "https://files.pythonhosted.org/packages/9b/9d/025cccdd738a72140efc582b1641d0dd4caf2e86c3fb127568dc80444e6e/scipy-1.17.0-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:130d12926ae34399d157de777472bf82e9061c60cc081372b3118edacafe1d00", size = 32815098, upload-time = "2026-01-10T21:27:54.389Z" },
- { url = "https://files.pythonhosted.org/packages/48/5f/09b879619f8bca15ce392bfc1894bd9c54377e01d1b3f2f3b595a1b4d945/scipy-1.17.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6e886000eb4919eae3a44f035e63f0fd8b651234117e8f6f29bad1cd26e7bc45", size = 35031342, upload-time = "2026-01-10T21:28:03.012Z" },
- { url = "https://files.pythonhosted.org/packages/f2/9a/f0f0a9f0aa079d2f106555b984ff0fbb11a837df280f04f71f056ea9c6e4/scipy-1.17.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:13c4096ac6bc31d706018f06a49abe0485f96499deb82066b94d19b02f664209", size = 34893199, upload-time = "2026-01-10T21:28:10.832Z" },
- { url = "https://files.pythonhosted.org/packages/90/b8/4f0f5cf0c5ea4d7548424e6533e6b17d164f34a6e2fb2e43ffebb6697b06/scipy-1.17.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:cacbaddd91fcffde703934897c5cd2c7cb0371fac195d383f4e1f1c5d3f3bd04", size = 37438061, upload-time = "2026-01-10T21:28:19.684Z" },
- { url = "https://files.pythonhosted.org/packages/f9/cc/2bd59140ed3b2fa2882fb15da0a9cb1b5a6443d67cfd0d98d4cec83a57ec/scipy-1.17.0-cp313-cp313t-win_amd64.whl", hash = "sha256:edce1a1cf66298cccdc48a1bdf8fb10a3bf58e8b58d6c3883dd1530e103f87c0", size = 36328593, upload-time = "2026-01-10T21:28:28.007Z" },
- { url = "https://files.pythonhosted.org/packages/13/1b/c87cc44a0d2c7aaf0f003aef2904c3d097b422a96c7e7c07f5efd9073c1b/scipy-1.17.0-cp313-cp313t-win_arm64.whl", hash = "sha256:30509da9dbec1c2ed8f168b8d8aa853bc6723fede1dbc23c7d43a56f5ab72a67", size = 24625083, upload-time = "2026-01-10T21:28:35.188Z" },
- { url = "https://files.pythonhosted.org/packages/1a/2d/51006cd369b8e7879e1c630999a19d1fbf6f8b5ed3e33374f29dc87e53b3/scipy-1.17.0-cp314-cp314-macosx_10_14_x86_64.whl", hash = "sha256:c17514d11b78be8f7e6331b983a65a7f5ca1fd037b95e27b280921fe5606286a", size = 31346803, upload-time = "2026-01-10T21:28:57.24Z" },
- { url = "https://files.pythonhosted.org/packages/d6/2e/2349458c3ce445f53a6c93d4386b1c4c5c0c540917304c01222ff95ff317/scipy-1.17.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:4e00562e519c09da34c31685f6acc3aa384d4d50604db0f245c14e1b4488bfa2", size = 27967182, upload-time = "2026-01-10T21:29:04.107Z" },
- { url = "https://files.pythonhosted.org/packages/5e/7c/df525fbfa77b878d1cfe625249529514dc02f4fd5f45f0f6295676a76528/scipy-1.17.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:f7df7941d71314e60a481e02d5ebcb3f0185b8d799c70d03d8258f6c80f3d467", size = 20139125, upload-time = "2026-01-10T21:29:10.179Z" },
- { url = "https://files.pythonhosted.org/packages/33/11/fcf9d43a7ed1234d31765ec643b0515a85a30b58eddccc5d5a4d12b5f194/scipy-1.17.0-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:aabf057c632798832f071a8dde013c2e26284043934f53b00489f1773b33527e", size = 22443554, upload-time = "2026-01-10T21:29:15.888Z" },
- { url = "https://files.pythonhosted.org/packages/80/5c/ea5d239cda2dd3d31399424967a24d556cf409fbea7b5b21412b0fd0a44f/scipy-1.17.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a38c3337e00be6fd8a95b4ed66b5d988bac4ec888fd922c2ea9fe5fb1603dd67", size = 32757834, upload-time = "2026-01-10T21:29:23.406Z" },
- { url = "https://files.pythonhosted.org/packages/b8/7e/8c917cc573310e5dc91cbeead76f1b600d3fb17cf0969db02c9cf92e3cfa/scipy-1.17.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:00fb5f8ec8398ad90215008d8b6009c9db9fa924fd4c7d6be307c6f945f9cd73", size = 34995775, upload-time = "2026-01-10T21:29:31.915Z" },
- { url = "https://files.pythonhosted.org/packages/c5/43/176c0c3c07b3f7df324e7cdd933d3e2c4898ca202b090bd5ba122f9fe270/scipy-1.17.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:f2a4942b0f5f7c23c7cd641a0ca1955e2ae83dedcff537e3a0259096635e186b", size = 34841240, upload-time = "2026-01-10T21:29:39.995Z" },
- { url = "https://files.pythonhosted.org/packages/44/8c/d1f5f4b491160592e7f084d997de53a8e896a3ac01cd07e59f43ca222744/scipy-1.17.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:dbf133ced83889583156566d2bdf7a07ff89228fe0c0cb727f777de92092ec6b", size = 37394463, upload-time = "2026-01-10T21:29:48.723Z" },
- { url = "https://files.pythonhosted.org/packages/9f/ec/42a6657f8d2d087e750e9a5dde0b481fd135657f09eaf1cf5688bb23c338/scipy-1.17.0-cp314-cp314-win_amd64.whl", hash = "sha256:3625c631a7acd7cfd929e4e31d2582cf00f42fcf06011f59281271746d77e061", size = 37053015, upload-time = "2026-01-10T21:30:51.418Z" },
- { url = "https://files.pythonhosted.org/packages/27/58/6b89a6afd132787d89a362d443a7bddd511b8f41336a1ae47f9e4f000dc4/scipy-1.17.0-cp314-cp314-win_arm64.whl", hash = "sha256:9244608d27eafe02b20558523ba57f15c689357c85bdcfe920b1828750aa26eb", size = 24951312, upload-time = "2026-01-10T21:30:56.771Z" },
- { url = "https://files.pythonhosted.org/packages/e9/01/f58916b9d9ae0112b86d7c3b10b9e685625ce6e8248df139d0fcb17f7397/scipy-1.17.0-cp314-cp314t-macosx_10_14_x86_64.whl", hash = "sha256:2b531f57e09c946f56ad0b4a3b2abee778789097871fc541e267d2eca081cff1", size = 31706502, upload-time = "2026-01-10T21:29:56.326Z" },
- { url = "https://files.pythonhosted.org/packages/59/8e/2912a87f94a7d1f8b38aabc0faf74b82d3b6c9e22be991c49979f0eceed8/scipy-1.17.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:13e861634a2c480bd237deb69333ac79ea1941b94568d4b0efa5db5e263d4fd1", size = 28380854, upload-time = "2026-01-10T21:30:01.554Z" },
- { url = "https://files.pythonhosted.org/packages/bd/1c/874137a52dddab7d5d595c1887089a2125d27d0601fce8c0026a24a92a0b/scipy-1.17.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:eb2651271135154aa24f6481cbae5cc8af1f0dd46e6533fb7b56aa9727b6a232", size = 20552752, upload-time = "2026-01-10T21:30:05.93Z" },
- { url = "https://files.pythonhosted.org/packages/3f/f0/7518d171cb735f6400f4576cf70f756d5b419a07fe1867da34e2c2c9c11b/scipy-1.17.0-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:c5e8647f60679790c2f5c76be17e2e9247dc6b98ad0d3b065861e082c56e078d", size = 22803972, upload-time = "2026-01-10T21:30:10.651Z" },
- { url = "https://files.pythonhosted.org/packages/7c/74/3498563a2c619e8a3ebb4d75457486c249b19b5b04a30600dfd9af06bea5/scipy-1.17.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5fb10d17e649e1446410895639f3385fd2bf4c3c7dfc9bea937bddcbc3d7b9ba", size = 32829770, upload-time = "2026-01-10T21:30:16.359Z" },
- { url = "https://files.pythonhosted.org/packages/48/d1/7b50cedd8c6c9d6f706b4b36fa8544d829c712a75e370f763b318e9638c1/scipy-1.17.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8547e7c57f932e7354a2319fab613981cde910631979f74c9b542bb167a8b9db", size = 35051093, upload-time = "2026-01-10T21:30:22.987Z" },
- { url = "https://files.pythonhosted.org/packages/e2/82/a2d684dfddb87ba1b3ea325df7c3293496ee9accb3a19abe9429bce94755/scipy-1.17.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:33af70d040e8af9d5e7a38b5ed3b772adddd281e3062ff23fec49e49681c38cf", size = 34909905, upload-time = "2026-01-10T21:30:28.704Z" },
- { url = "https://files.pythonhosted.org/packages/ef/5e/e565bd73991d42023eb82bb99e51c5b3d9e2c588ca9d4b3e2cc1d3ca62a6/scipy-1.17.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f9eb55bb97d00f8b7ab95cb64f873eb0bf54d9446264d9f3609130381233483f", size = 37457743, upload-time = "2026-01-10T21:30:34.819Z" },
- { url = "https://files.pythonhosted.org/packages/58/a8/a66a75c3d8f1fb2b83f66007d6455a06a6f6cf5618c3dc35bc9b69dd096e/scipy-1.17.0-cp314-cp314t-win_amd64.whl", hash = "sha256:1ff269abf702f6c7e67a4b7aad981d42871a11b9dd83c58d2d2ea624efbd1088", size = 37098574, upload-time = "2026-01-10T21:30:40.782Z" },
- { url = "https://files.pythonhosted.org/packages/56/a5/df8f46ef7da168f1bc52cd86e09a9de5c6f19cc1da04454d51b7d4f43408/scipy-1.17.0-cp314-cp314t-win_arm64.whl", hash = "sha256:031121914e295d9791319a1875444d55079885bbae5bdc9c5e0f2ee5f09d34ff", size = 25246266, upload-time = "2026-01-10T21:30:45.923Z" },
+version = "1.17.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" } },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/7a/97/5a3609c4f8d58b039179648e62dd220f89864f56f7357f5d4f45c29eb2cc/scipy-1.17.1.tar.gz", hash = "sha256:95d8e012d8cb8816c226aef832200b1d45109ed4464303e997c5b13122b297c0", size = 30573822, upload-time = "2026-02-23T00:26:24.851Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/35/48/b992b488d6f299dbe3f11a20b24d3dda3d46f1a635ede1c46b5b17a7b163/scipy-1.17.1-cp312-cp312-macosx_10_14_x86_64.whl", hash = "sha256:35c3a56d2ef83efc372eaec584314bd0ef2e2f0d2adb21c55e6ad5b344c0dcb8", size = 31610954, upload-time = "2026-02-23T00:17:49.855Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/02/cf107b01494c19dc100f1d0b7ac3cc08666e96ba2d64db7626066cee895e/scipy-1.17.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:fcb310ddb270a06114bb64bbe53c94926b943f5b7f0842194d585c65eb4edd76", size = 28172662, upload-time = "2026-02-23T00:18:01.64Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/a9/599c28631bad314d219cf9ffd40e985b24d603fc8a2f4ccc5ae8419a535b/scipy-1.17.1-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:cc90d2e9c7e5c7f1a482c9875007c095c3194b1cfedca3c2f3291cdc2bc7c086", size = 20344366, upload-time = "2026-02-23T00:18:12.015Z" },
+ { url = "https://files.pythonhosted.org/packages/35/f5/906eda513271c8deb5af284e5ef0206d17a96239af79f9fa0aebfe0e36b4/scipy-1.17.1-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:c80be5ede8f3f8eded4eff73cc99a25c388ce98e555b17d31da05287015ffa5b", size = 22704017, upload-time = "2026-02-23T00:18:21.502Z" },
+ { url = "https://files.pythonhosted.org/packages/da/34/16f10e3042d2f1d6b66e0428308ab52224b6a23049cb2f5c1756f713815f/scipy-1.17.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e19ebea31758fac5893a2ac360fedd00116cbb7628e650842a6691ba7ca28a21", size = 32927842, upload-time = "2026-02-23T00:18:35.367Z" },
+ { url = "https://files.pythonhosted.org/packages/01/8e/1e35281b8ab6d5d72ebe9911edcdffa3f36b04ed9d51dec6dd140396e220/scipy-1.17.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:02ae3b274fde71c5e92ac4d54bc06c42d80e399fec704383dcd99b301df37458", size = 35235890, upload-time = "2026-02-23T00:18:49.188Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/5c/9d7f4c88bea6e0d5a4f1bc0506a53a00e9fcb198de372bfe4d3652cef482/scipy-1.17.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8a604bae87c6195d8b1045eddece0514d041604b14f2727bbc2b3020172045eb", size = 35003557, upload-time = "2026-02-23T00:18:54.74Z" },
+ { url = "https://files.pythonhosted.org/packages/65/94/7698add8f276dbab7a9de9fb6b0e02fc13ee61d51c7c3f85ac28b65e1239/scipy-1.17.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f590cd684941912d10becc07325a3eeb77886fe981415660d9265c4c418d0bea", size = 37625856, upload-time = "2026-02-23T00:19:00.307Z" },
+ { url = "https://files.pythonhosted.org/packages/a2/84/dc08d77fbf3d87d3ee27f6a0c6dcce1de5829a64f2eae85a0ecc1f0daa73/scipy-1.17.1-cp312-cp312-win_amd64.whl", hash = "sha256:41b71f4a3a4cab9d366cd9065b288efc4d4f3c0b37a91a8e0947fb5bd7f31d87", size = 36549682, upload-time = "2026-02-23T00:19:07.67Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/98/fe9ae9ffb3b54b62559f52dedaebe204b408db8109a8c66fdd04869e6424/scipy-1.17.1-cp312-cp312-win_arm64.whl", hash = "sha256:f4115102802df98b2b0db3cce5cb9b92572633a1197c77b7553e5203f284a5b3", size = 24547340, upload-time = "2026-02-23T00:19:12.024Z" },
+ { url = "https://files.pythonhosted.org/packages/76/27/07ee1b57b65e92645f219b37148a7e7928b82e2b5dbeccecb4dff7c64f0b/scipy-1.17.1-cp313-cp313-macosx_10_14_x86_64.whl", hash = "sha256:5e3c5c011904115f88a39308379c17f91546f77c1667cea98739fe0fccea804c", size = 31590199, upload-time = "2026-02-23T00:19:17.192Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/ae/db19f8ab842e9b724bf5dbb7db29302a91f1e55bc4d04b1025d6d605a2c5/scipy-1.17.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:6fac755ca3d2c3edcb22f479fceaa241704111414831ddd3bc6056e18516892f", size = 28154001, upload-time = "2026-02-23T00:19:22.241Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/58/3ce96251560107b381cbd6e8413c483bbb1228a6b919fa8652b0d4090e7f/scipy-1.17.1-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:7ff200bf9d24f2e4d5dc6ee8c3ac64d739d3a89e2326ba68aaf6c4a2b838fd7d", size = 20325719, upload-time = "2026-02-23T00:19:26.329Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/83/15087d945e0e4d48ce2377498abf5ad171ae013232ae31d06f336e64c999/scipy-1.17.1-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:4b400bdc6f79fa02a4d86640310dde87a21fba0c979efff5248908c6f15fad1b", size = 22683595, upload-time = "2026-02-23T00:19:30.304Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/e0/e58fbde4a1a594c8be8114eb4aac1a55bcd6587047efc18a61eb1f5c0d30/scipy-1.17.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2b64ca7d4aee0102a97f3ba22124052b4bd2152522355073580bf4845e2550b6", size = 32896429, upload-time = "2026-02-23T00:19:35.536Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/5f/f17563f28ff03c7b6799c50d01d5d856a1d55f2676f537ca8d28c7f627cd/scipy-1.17.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:581b2264fc0aa555f3f435a5944da7504ea3a065d7029ad60e7c3d1ae09c5464", size = 35203952, upload-time = "2026-02-23T00:19:42.259Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/a5/9afd17de24f657fdfe4df9a3f1ea049b39aef7c06000c13db1530d81ccca/scipy-1.17.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:beeda3d4ae615106d7094f7e7cef6218392e4465cc95d25f900bebabfded0950", size = 34979063, upload-time = "2026-02-23T00:19:47.547Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/13/88b1d2384b424bf7c924f2038c1c409f8d88bb2a8d49d097861dd64a57b2/scipy-1.17.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6609bc224e9568f65064cfa72edc0f24ee6655b47575954ec6339534b2798369", size = 37598449, upload-time = "2026-02-23T00:19:53.238Z" },
+ { url = "https://files.pythonhosted.org/packages/35/e5/d6d0e51fc888f692a35134336866341c08655d92614f492c6860dc45bb2c/scipy-1.17.1-cp313-cp313-win_amd64.whl", hash = "sha256:37425bc9175607b0268f493d79a292c39f9d001a357bebb6b88fdfaff13f6448", size = 36510943, upload-time = "2026-02-23T00:20:50.89Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/fd/3be73c564e2a01e690e19cc618811540ba5354c67c8680dce3281123fb79/scipy-1.17.1-cp313-cp313-win_arm64.whl", hash = "sha256:5cf36e801231b6a2059bf354720274b7558746f3b1a4efb43fcf557ccd484a87", size = 24545621, upload-time = "2026-02-23T00:20:55.871Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/6b/17787db8b8114933a66f9dcc479a8272e4b4da75fe03b0c282f7b0ade8cd/scipy-1.17.1-cp313-cp313t-macosx_10_14_x86_64.whl", hash = "sha256:d59c30000a16d8edc7e64152e30220bfbd724c9bbb08368c054e24c651314f0a", size = 31936708, upload-time = "2026-02-23T00:19:58.694Z" },
+ { url = "https://files.pythonhosted.org/packages/38/2e/524405c2b6392765ab1e2b722a41d5da33dc5c7b7278184a8ad29b6cb206/scipy-1.17.1-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:010f4333c96c9bb1a4516269e33cb5917b08ef2166d5556ca2fd9f082a9e6ea0", size = 28570135, upload-time = "2026-02-23T00:20:03.934Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/c3/5bd7199f4ea8556c0c8e39f04ccb014ac37d1468e6cfa6a95c6b3562b76e/scipy-1.17.1-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:2ceb2d3e01c5f1d83c4189737a42d9cb2fc38a6eeed225e7515eef71ad301dce", size = 20741977, upload-time = "2026-02-23T00:20:07.935Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/b8/8ccd9b766ad14c78386599708eb745f6b44f08400a5fd0ade7cf89b6fc93/scipy-1.17.1-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:844e165636711ef41f80b4103ed234181646b98a53c8f05da12ca5ca289134f6", size = 23029601, upload-time = "2026-02-23T00:20:12.161Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/a0/3cb6f4d2fb3e17428ad2880333cac878909ad1a89f678527b5328b93c1d4/scipy-1.17.1-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:158dd96d2207e21c966063e1635b1063cd7787b627b6f07305315dd73d9c679e", size = 33019667, upload-time = "2026-02-23T00:20:17.208Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/c3/2d834a5ac7bf3a0c806ad1508efc02dda3c8c61472a56132d7894c312dea/scipy-1.17.1-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:74cbb80d93260fe2ffa334efa24cb8f2f0f622a9b9febf8b483c0b865bfb3475", size = 35264159, upload-time = "2026-02-23T00:20:23.087Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/77/d3ed4becfdbd217c52062fafe35a72388d1bd82c2d0ba5ca19d6fcc93e11/scipy-1.17.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:dbc12c9f3d185f5c737d801da555fb74b3dcfa1a50b66a1a93e09190f41fab50", size = 35102771, upload-time = "2026-02-23T00:20:28.636Z" },
+ { url = "https://files.pythonhosted.org/packages/bd/12/d19da97efde68ca1ee5538bb261d5d2c062f0c055575128f11a2730e3ac1/scipy-1.17.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:94055a11dfebe37c656e70317e1996dc197e1a15bbcc351bcdd4610e128fe1ca", size = 37665910, upload-time = "2026-02-23T00:20:34.743Z" },
+ { url = "https://files.pythonhosted.org/packages/06/1c/1172a88d507a4baaf72c5a09bb6c018fe2ae0ab622e5830b703a46cc9e44/scipy-1.17.1-cp313-cp313t-win_amd64.whl", hash = "sha256:e30bdeaa5deed6bc27b4cc490823cd0347d7dae09119b8803ae576ea0ce52e4c", size = 36562980, upload-time = "2026-02-23T00:20:40.575Z" },
+ { url = "https://files.pythonhosted.org/packages/70/b0/eb757336e5a76dfa7911f63252e3b7d1de00935d7705cf772db5b45ec238/scipy-1.17.1-cp313-cp313t-win_arm64.whl", hash = "sha256:a720477885a9d2411f94a93d16f9d89bad0f28ca23c3f8daa521e2dcc3f44d49", size = 24856543, upload-time = "2026-02-23T00:20:45.313Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/83/333afb452af6f0fd70414dc04f898647ee1423979ce02efa75c3b0f2c28e/scipy-1.17.1-cp314-cp314-macosx_10_14_x86_64.whl", hash = "sha256:a48a72c77a310327f6a3a920092fa2b8fd03d7deaa60f093038f22d98e096717", size = 31584510, upload-time = "2026-02-23T00:21:01.015Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/a6/d05a85fd51daeb2e4ea71d102f15b34fedca8e931af02594193ae4fd25f7/scipy-1.17.1-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:45abad819184f07240d8a696117a7aacd39787af9e0b719d00285549ed19a1e9", size = 28170131, upload-time = "2026-02-23T00:21:05.888Z" },
+ { url = "https://files.pythonhosted.org/packages/db/7b/8624a203326675d7746a254083a187398090a179335b2e4a20e2ddc46e83/scipy-1.17.1-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:3fd1fcdab3ea951b610dc4cef356d416d5802991e7e32b5254828d342f7b7e0b", size = 20342032, upload-time = "2026-02-23T00:21:09.904Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/35/2c342897c00775d688d8ff3987aced3426858fd89d5a0e26e020b660b301/scipy-1.17.1-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:7bdf2da170b67fdf10bca777614b1c7d96ae3ca5794fd9587dce41eb2966e866", size = 22678766, upload-time = "2026-02-23T00:21:14.313Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/f2/7cdb8eb308a1a6ae1e19f945913c82c23c0c442a462a46480ce487fdc0ac/scipy-1.17.1-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:adb2642e060a6549c343603a3851ba76ef0b74cc8c079a9a58121c7ec9fe2350", size = 32957007, upload-time = "2026-02-23T00:21:19.663Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/2e/7eea398450457ecb54e18e9d10110993fa65561c4f3add5e8eccd2b9cd41/scipy-1.17.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:eee2cfda04c00a857206a4330f0c5e3e56535494e30ca445eb19ec624ae75118", size = 35221333, upload-time = "2026-02-23T00:21:25.278Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/77/5b8509d03b77f093a0d52e606d3c4f79e8b06d1d38c441dacb1e26cacf46/scipy-1.17.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:d2650c1fb97e184d12d8ba010493ee7b322864f7d3d00d3f9bb97d9c21de4068", size = 35042066, upload-time = "2026-02-23T00:21:31.358Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/df/18f80fb99df40b4070328d5ae5c596f2f00fffb50167e31439e932f29e7d/scipy-1.17.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:08b900519463543aa604a06bec02461558a6e1cef8fdbb8098f77a48a83c8118", size = 37612763, upload-time = "2026-02-23T00:21:37.247Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/39/f0e8ea762a764a9dc52aa7dabcfad51a354819de1f0d4652b6a1122424d6/scipy-1.17.1-cp314-cp314-win_amd64.whl", hash = "sha256:3877ac408e14da24a6196de0ddcace62092bfc12a83823e92e49e40747e52c19", size = 37290984, upload-time = "2026-02-23T00:22:35.023Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/56/fe201e3b0f93d1a8bcf75d3379affd228a63d7e2d80ab45467a74b494947/scipy-1.17.1-cp314-cp314-win_arm64.whl", hash = "sha256:f8885db0bc2bffa59d5c1b72fad7a6a92d3e80e7257f967dd81abb553a90d293", size = 25192877, upload-time = "2026-02-23T00:22:39.798Z" },
+ { url = "https://files.pythonhosted.org/packages/96/ad/f8c414e121f82e02d76f310f16db9899c4fcde36710329502a6b2a3c0392/scipy-1.17.1-cp314-cp314t-macosx_10_14_x86_64.whl", hash = "sha256:1cc682cea2ae55524432f3cdff9e9a3be743d52a7443d0cba9017c23c87ae2f6", size = 31949750, upload-time = "2026-02-23T00:21:42.289Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/b0/c741e8865d61b67c81e255f4f0a832846c064e426636cd7de84e74d209be/scipy-1.17.1-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:2040ad4d1795a0ae89bfc7e8429677f365d45aa9fd5e4587cf1ea737f927b4a1", size = 28585858, upload-time = "2026-02-23T00:21:47.706Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/1b/3985219c6177866628fa7c2595bfd23f193ceebbe472c98a08824b9466ff/scipy-1.17.1-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:131f5aaea57602008f9822e2115029b55d4b5f7c070287699fe45c661d051e39", size = 20757723, upload-time = "2026-02-23T00:21:52.039Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/19/2a04aa25050d656d6f7b9e7b685cc83d6957fb101665bfd9369ca6534563/scipy-1.17.1-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:9cdc1a2fcfd5c52cfb3045feb399f7b3ce822abdde3a193a6b9a60b3cb5854ca", size = 23043098, upload-time = "2026-02-23T00:21:56.185Z" },
+ { url = "https://files.pythonhosted.org/packages/86/f1/3383beb9b5d0dbddd030335bf8a8b32d4317185efe495374f134d8be6cce/scipy-1.17.1-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6e3dcd57ab780c741fde8dc68619de988b966db759a3c3152e8e9142c26295ad", size = 33030397, upload-time = "2026-02-23T00:22:01.404Z" },
+ { url = "https://files.pythonhosted.org/packages/41/68/8f21e8a65a5a03f25a79165ec9d2b28c00e66dc80546cf5eb803aeeff35b/scipy-1.17.1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a9956e4d4f4a301ebf6cde39850333a6b6110799d470dbbb1e25326ac447f52a", size = 35281163, upload-time = "2026-02-23T00:22:07.024Z" },
+ { url = "https://files.pythonhosted.org/packages/84/8d/c8a5e19479554007a5632ed7529e665c315ae7492b4f946b0deb39870e39/scipy-1.17.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:a4328d245944d09fd639771de275701ccadf5f781ba0ff092ad141e017eccda4", size = 35116291, upload-time = "2026-02-23T00:22:12.585Z" },
+ { url = "https://files.pythonhosted.org/packages/52/52/e57eceff0e342a1f50e274264ed47497b59e6a4e3118808ee58ddda7b74a/scipy-1.17.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:a77cbd07b940d326d39a1d1b37817e2ee4d79cb30e7338f3d0cddffae70fcaa2", size = 37682317, upload-time = "2026-02-23T00:22:18.513Z" },
+ { url = "https://files.pythonhosted.org/packages/11/2f/b29eafe4a3fbc3d6de9662b36e028d5f039e72d345e05c250e121a230dd4/scipy-1.17.1-cp314-cp314t-win_amd64.whl", hash = "sha256:eb092099205ef62cd1782b006658db09e2fed75bffcae7cc0d44052d8aa0f484", size = 37345327, upload-time = "2026-02-23T00:22:24.442Z" },
+ { url = "https://files.pythonhosted.org/packages/07/39/338d9219c4e87f3e708f18857ecd24d22a0c3094752393319553096b98af/scipy-1.17.1-cp314-cp314t-win_arm64.whl", hash = "sha256:200e1050faffacc162be6a486a984a0497866ec54149a01270adc8a59b7c7d21", size = 25489165, upload-time = "2026-02-23T00:22:29.563Z" },
]
[[package]]
@@ -4912,22 +5776,6 @@ version = "0.2.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/15/15/2e7a025fc62d764b151ae6d0f2a92f8081755ebe8d4a64099accc6f77ba6/sentencepiece-0.2.1.tar.gz", hash = "sha256:8138cec27c2f2282f4a34d9a016e3374cd40e5c6e9cb335063db66a0a3b71fad", size = 3228515, upload-time = "2025-08-12T07:00:51.718Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/af/31/5b7cccb307b485db1a2372d6d2980b0a65d067f8be5ca943a103b4acd5b3/sentencepiece-0.2.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:e10fa50bdbaa5e2445dbd387979980d391760faf0ec99a09bd7780ff37eaec44", size = 1942557, upload-time = "2025-08-12T06:59:12.379Z" },
- { url = "https://files.pythonhosted.org/packages/1f/41/0ac923a8e685ad290c5afc8ae55c5844977b8d75076fcc04302b9a324274/sentencepiece-0.2.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:2f27ae6deea72efdb6f361750c92f6c21fd0ad087445082770cc34015213c526", size = 1325384, upload-time = "2025-08-12T06:59:14.334Z" },
- { url = "https://files.pythonhosted.org/packages/fc/ef/3751555d67daf9003384978f169d31c775cb5c7baf28633caaf1eb2b2b4d/sentencepiece-0.2.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:60937c959e6f44159fdd9f56fbdd302501f96114a5ba436829496d5f32d8de3f", size = 1253317, upload-time = "2025-08-12T06:59:16.247Z" },
- { url = "https://files.pythonhosted.org/packages/46/a5/742c69b7bd144eb32b6e5fd50dbd8abbbc7a95fce2fe16e50156fa400e3b/sentencepiece-0.2.1-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d8b1d91545578852f128650b8cce4ec20f93d39b378ff554ebe66290f2dabb92", size = 1316379, upload-time = "2025-08-12T06:59:17.825Z" },
- { url = "https://files.pythonhosted.org/packages/c8/89/8deeafbba2871e8fa10f20f17447786f4ac38085925335728d360eaf4cae/sentencepiece-0.2.1-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:27e38eee653abc3d387862e67bc5c8b6f428cd604e688b85d29170b7e725c26c", size = 1387926, upload-time = "2025-08-12T06:59:19.395Z" },
- { url = "https://files.pythonhosted.org/packages/c3/ca/67fe73005f0ab617c6a970b199754e28e524b6873aa7025224fad3cda252/sentencepiece-0.2.1-cp310-cp310-win32.whl", hash = "sha256:251874d720ac7f28024a168501f3c7bb15d1802245f6e66de565f18bbb9b5eaa", size = 999550, upload-time = "2025-08-12T06:59:20.844Z" },
- { url = "https://files.pythonhosted.org/packages/6d/33/dc5b54042050d2dda4229c3ce1f862541c99966390b6aa20f54d520d2dc2/sentencepiece-0.2.1-cp310-cp310-win_amd64.whl", hash = "sha256:e52144670738b4b477fade6c2a9b6af71a8d0094514c9853ac9f6fc1fcfabae7", size = 1054613, upload-time = "2025-08-12T06:59:22.255Z" },
- { url = "https://files.pythonhosted.org/packages/fa/19/1ea47f46ff97fe04422b78997da1a37cd632f414aae042d27a9009c5b733/sentencepiece-0.2.1-cp310-cp310-win_arm64.whl", hash = "sha256:9076430ac25dfa7147d9d05751dbc66a04bc1aaac371c07f84952979ea59f0d0", size = 1033884, upload-time = "2025-08-12T06:59:24.194Z" },
- { url = "https://files.pythonhosted.org/packages/d8/15/46afbab00733d81788b64be430ca1b93011bb9388527958e26cc31832de5/sentencepiece-0.2.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:6356d0986b8b8dc351b943150fcd81a1c6e6e4d439772e8584c64230e58ca987", size = 1942560, upload-time = "2025-08-12T06:59:25.82Z" },
- { url = "https://files.pythonhosted.org/packages/fa/79/7c01b8ef98a0567e9d84a4e7a910f8e7074fcbf398a5cd76f93f4b9316f9/sentencepiece-0.2.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:8f8ba89a3acb3dc1ae90f65ec1894b0b9596fdb98ab003ff38e058f898b39bc7", size = 1325385, upload-time = "2025-08-12T06:59:27.722Z" },
- { url = "https://files.pythonhosted.org/packages/bb/88/2b41e07bd24f33dcf2f18ec3b74247aa4af3526bad8907b8727ea3caba03/sentencepiece-0.2.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:02593eca45440ef39247cee8c47322a34bdcc1d8ae83ad28ba5a899a2cf8d79a", size = 1253319, upload-time = "2025-08-12T06:59:29.306Z" },
- { url = "https://files.pythonhosted.org/packages/a0/54/38a1af0c6210a3c6f95aa46d23d6640636d020fba7135cd0d9a84ada05a7/sentencepiece-0.2.1-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0a0d15781a171d188b661ae4bde1d998c303f6bd8621498c50c671bd45a4798e", size = 1316162, upload-time = "2025-08-12T06:59:30.914Z" },
- { url = "https://files.pythonhosted.org/packages/ef/66/fb191403ade791ad2c3c1e72fe8413e63781b08cfa3aa4c9dfc536d6e795/sentencepiece-0.2.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4f5a3e0d9f445ed9d66c0fec47d4b23d12cfc858b407a03c194c1b26c2ac2a63", size = 1387785, upload-time = "2025-08-12T06:59:32.491Z" },
- { url = "https://files.pythonhosted.org/packages/a9/2d/3bd9b08e70067b2124518b308db6a84a4f8901cc8a4317e2e4288cdd9b4d/sentencepiece-0.2.1-cp311-cp311-win32.whl", hash = "sha256:6d297a1748d429ba8534eebe5535448d78b8acc32d00a29b49acf28102eeb094", size = 999555, upload-time = "2025-08-12T06:59:34.475Z" },
- { url = "https://files.pythonhosted.org/packages/32/b8/f709977f5fda195ae1ea24f24e7c581163b6f142b1005bc3d0bbfe4d7082/sentencepiece-0.2.1-cp311-cp311-win_amd64.whl", hash = "sha256:82d9ead6591015f009cb1be1cb1c015d5e6f04046dbb8c9588b931e869a29728", size = 1054617, upload-time = "2025-08-12T06:59:36.461Z" },
- { url = "https://files.pythonhosted.org/packages/7a/40/a1fc23be23067da0f703709797b464e8a30a1c78cc8a687120cd58d4d509/sentencepiece-0.2.1-cp311-cp311-win_arm64.whl", hash = "sha256:39f8651bd10974eafb9834ce30d9bcf5b73e1fc798a7f7d2528f9820ca86e119", size = 1033877, upload-time = "2025-08-12T06:59:38.391Z" },
{ url = "https://files.pythonhosted.org/packages/4a/be/32ce495aa1d0e0c323dcb1ba87096037358edee539cac5baf8755a6bd396/sentencepiece-0.2.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:57cae326c8727de58c85977b175af132a7138d84c764635d7e71bbee7e774133", size = 1943152, upload-time = "2025-08-12T06:59:40.048Z" },
{ url = "https://files.pythonhosted.org/packages/88/7e/ff23008899a58678e98c6ff592bf4d368eee5a71af96d0df6b38a039dd4f/sentencepiece-0.2.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:56dd39a3c4d6493db3cdca7e8cc68c6b633f0d4195495cbadfcf5af8a22d05a6", size = 1325651, upload-time = "2025-08-12T06:59:41.536Z" },
{ url = "https://files.pythonhosted.org/packages/19/84/42eb3ce4796777a1b5d3699dfd4dca85113e68b637f194a6c8d786f16a04/sentencepiece-0.2.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:d9381351182ff9888cc80e41c632e7e274b106f450de33d67a9e8f6043da6f76", size = 1253645, upload-time = "2025-08-12T06:59:42.903Z" },
@@ -4972,15 +5820,15 @@ wheels = [
[[package]]
name = "sentry-sdk"
-version = "2.49.0"
+version = "2.56.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "certifi" },
{ name = "urllib3" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/02/94/23ac26616a883f492428d9ee9ad6eee391612125326b784dbfc30e1e7bab/sentry_sdk-2.49.0.tar.gz", hash = "sha256:c1878599cde410d481c04ef50ee3aedd4f600e4d0d253f4763041e468b332c30", size = 387228, upload-time = "2026-01-08T09:56:25.642Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/de/df/5008954f5466085966468612a7d1638487596ee6d2fd7fb51783a85351bf/sentry_sdk-2.56.0.tar.gz", hash = "sha256:fdab72030b69625665b2eeb9738bdde748ad254e8073085a0ce95382678e8168", size = 426820, upload-time = "2026-03-24T09:56:36.575Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/88/43/1c586f9f413765201234541857cb82fda076f4b0f7bad4a0ec248da39cf3/sentry_sdk-2.49.0-py2.py3-none-any.whl", hash = "sha256:6ea78499133874445a20fe9c826c9e960070abeb7ae0cdf930314ab16bb97aa0", size = 415693, upload-time = "2026-01-08T09:56:21.872Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/1a/b3a3e9f6520493fed7997af4d2de7965d71549c62f994a8fd15f2ecd519e/sentry_sdk-2.56.0-py2.py3-none-any.whl", hash = "sha256:5afafb744ceb91d22f4cc650c6bd048ac6af5f7412dcc6c59305a2e36f4dbc02", size = 451568, upload-time = "2026-03-24T09:56:34.807Z" },
]
[[package]]
@@ -5012,20 +5860,32 @@ wheels = [
[[package]]
name = "slack-sdk"
-version = "3.39.0"
+version = "3.41.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/22/35/fc009118a13187dd9731657c60138e5a7c2dea88681a7f04dc406af5da7d/slack_sdk-3.41.0.tar.gz", hash = "sha256:eb61eb12a65bebeca9cb5d36b3f799e836ed2be21b456d15df2627cfe34076ca", size = 250568, upload-time = "2026-03-12T16:10:11.381Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a1/df/2e4be347ff98281b505cc0ccf141408cdd25eb5ca9f3830deb361b2472d3/slack_sdk-3.41.0-py2.py3-none-any.whl", hash = "sha256:bb18dcdfff1413ec448e759cf807ec3324090993d8ab9111c74081623b692a89", size = 313885, upload-time = "2026-03-12T16:10:09.811Z" },
+]
+
+[[package]]
+name = "smart-open"
+version = "7.5.1"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/b6/dd/645f3eb93fce38eadbb649e85684730b1fc3906c2674ca59bddc2ca2bd2e/slack_sdk-3.39.0.tar.gz", hash = "sha256:6a56be10dc155c436ff658c6b776e1c082e29eae6a771fccf8b0a235822bbcb1", size = 247207, upload-time = "2025-11-20T15:27:57.556Z" }
+dependencies = [
+ { name = "wrapt" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e8/be/a66598b305763861a9ab15ff0f2fbc44e47b1ce7a776797337a4eef37c66/smart_open-7.5.1.tar.gz", hash = "sha256:3f08e16827c4733699e6b2cc40328a3568f900cb12ad9a3ad233ba6c872d9fe7", size = 54034, upload-time = "2026-02-23T11:01:28.979Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/ef/1f/32bcf088e535c1870b1a1f2e3b916129c66fdfe565a793316317241d41e5/slack_sdk-3.39.0-py2.py3-none-any.whl", hash = "sha256:b1556b2f5b8b12b94e5ea3f56c4f2c7f04462e4e1013d325c5764ff118044fa8", size = 309850, upload-time = "2025-11-20T15:27:55.729Z" },
+ { url = "https://files.pythonhosted.org/packages/5e/ea/dcdecd68acebb49d3fd560473a43499b1635076f7f1ae8641c060fe7ce74/smart_open-7.5.1-py3-none-any.whl", hash = "sha256:3e07cbbd9c8a908bcb8e25d48becf1a5cbb4886fa975e9f34c672ed171df2318", size = 64108, upload-time = "2026-02-23T11:01:27.429Z" },
]
[[package]]
name = "smmap"
-version = "5.0.2"
+version = "5.0.3"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/44/cd/a040c4b3119bbe532e5b0732286f805445375489fceaec1f48306068ee3b/smmap-5.0.2.tar.gz", hash = "sha256:26ea65a03958fa0c8a1c7e8c7a58fdc77221b8910f6be2131affade476898ad5", size = 22329, upload-time = "2025-01-02T07:14:40.909Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/1f/ea/49c993d6dfdd7338c9b1000a0f36817ed7ec84577ae2e52f890d1a4ff909/smmap-5.0.3.tar.gz", hash = "sha256:4d9debb8b99007ae47165abc08670bd74cb74b5227dda7f643eccc4e9eb5642c", size = 22506, upload-time = "2026-03-09T03:43:26.1Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/04/be/d09147ad1ec7934636ad912901c5fd7667e1c858e19d355237db0d0cd5e4/smmap-5.0.2-py3-none-any.whl", hash = "sha256:b30115f0def7d7531d22a0fb6502488d879e75b260a9db4d0819cfb25403af5e", size = 24303, upload-time = "2025-01-02T07:14:38.724Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/d4/59e74daffcb57a07668852eeeb6035af9f32cbfd7a1d2511f17d2fe6a738/smmap-5.0.3-py3-none-any.whl", hash = "sha256:c106e05d5a61449cf6ba9a1e650227ecfb141590d2a98412103ff35d89fc7b2f", size = 24390, upload-time = "2026-03-09T03:43:24.361Z" },
]
[[package]]
@@ -5061,8 +5921,8 @@ version = "0.13.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "cffi" },
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version < '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version >= '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/e1/41/9b873a8c055582859b239be17902a85339bec6a30ad162f98c9b0288a2cc/soundfile-0.13.1.tar.gz", hash = "sha256:b2c68dab1e30297317080a5b43df57e302584c49e2942defdde0acccc53f0e5b", size = 46156, upload-time = "2025-01-25T09:17:04.831Z" }
wheels = [
@@ -5077,125 +5937,53 @@ wheels = [
[[package]]
name = "soupsieve"
-version = "2.8.1"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/89/23/adf3796d740536d63a6fbda113d07e60c734b6ed5d3058d1e47fc0495e47/soupsieve-2.8.1.tar.gz", hash = "sha256:4cf733bc50fa805f5df4b8ef4740fc0e0fa6218cf3006269afd3f9d6d80fd350", size = 117856, upload-time = "2025-12-18T13:50:34.655Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/48/f3/b67d6ea49ca9154453b6d70b34ea22f3996b9fa55da105a79d8732227adc/soupsieve-2.8.1-py3-none-any.whl", hash = "sha256:a11fe2a6f3d76ab3cf2de04eb339c1be5b506a8a47f2ceb6d139803177f85434", size = 36710, upload-time = "2025-12-18T13:50:33.267Z" },
-]
-
-[[package]]
-name = "sphinx"
-version = "8.1.3"
+version = "2.8.3"
source = { registry = "https://pypi.org/simple" }
-resolution-markers = [
- "python_full_version < '3.11' and sys_platform == 'linux'",
- "python_full_version < '3.11' and sys_platform != 'linux'",
-]
-dependencies = [
- { name = "alabaster", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "babel", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "colorama", marker = "(python_full_version < '3.11' and sys_platform == 'win32') or (python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (sys_platform != 'win32' and extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "docutils", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "imagesize", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "jinja2", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "packaging", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "pygments", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "requests", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "snowballstemmer", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinxcontrib-applehelp", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinxcontrib-devhelp", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinxcontrib-htmlhelp", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinxcontrib-jsmath", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinxcontrib-qthelp", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinxcontrib-serializinghtml", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "tomli", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/6f/6d/be0b61178fe2cdcb67e2a92fc9ebb488e3c51c4f74a36a7824c0adf23425/sphinx-8.1.3.tar.gz", hash = "sha256:43c1911eecb0d3e161ad78611bc905d1ad0e523e4ddc202a58a821773dc4c927", size = 8184611, upload-time = "2024-10-13T20:27:13.93Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/7b/ae/2d9c981590ed9999a0d91755b47fc74f74de286b0f5cee14c9269041e6c4/soupsieve-2.8.3.tar.gz", hash = "sha256:3267f1eeea4251fb42728b6dfb746edc9acaffc4a45b27e19450b676586e8349", size = 118627, upload-time = "2026-01-20T04:27:02.457Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/26/60/1ddff83a56d33aaf6f10ec8ce84b4c007d9368b21008876fceda7e7381ef/sphinx-8.1.3-py3-none-any.whl", hash = "sha256:09719015511837b76bf6e03e42eb7595ac8c2e41eeb9c29c5b755c6b677992a2", size = 3487125, upload-time = "2024-10-13T20:27:10.448Z" },
+ { url = "https://files.pythonhosted.org/packages/46/2c/1462b1d0a634697ae9e55b3cecdcb64788e8b7d63f54d923fcd0bb140aed/soupsieve-2.8.3-py3-none-any.whl", hash = "sha256:ed64f2ba4eebeab06cc4962affce381647455978ffc1e36bb79a545b91f45a95", size = 37016, upload-time = "2026-01-20T04:27:01.012Z" },
]
[[package]]
name = "sphinx"
-version = "8.2.3"
-source = { registry = "https://pypi.org/simple" }
-resolution-markers = [
- "python_full_version >= '3.14' and sys_platform == 'linux'",
- "python_full_version == '3.13.*' and sys_platform == 'linux'",
- "python_full_version == '3.12.*' and sys_platform == 'linux'",
- "python_full_version >= '3.14' and sys_platform != 'linux'",
- "python_full_version == '3.13.*' and sys_platform != 'linux'",
- "python_full_version == '3.12.*' and sys_platform != 'linux'",
- "python_full_version == '3.11.*' and sys_platform == 'linux'",
- "python_full_version == '3.11.*' and sys_platform != 'linux'",
-]
-dependencies = [
- { name = "alabaster", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "babel", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "colorama", marker = "(python_full_version >= '3.11' and sys_platform == 'win32') or (python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (sys_platform != 'win32' and extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "docutils", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "imagesize", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "jinja2", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "packaging", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "pygments", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "requests", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "roman-numerals-py", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "snowballstemmer", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinxcontrib-applehelp", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinxcontrib-devhelp", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinxcontrib-htmlhelp", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinxcontrib-jsmath", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinxcontrib-qthelp", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinxcontrib-serializinghtml", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/38/ad/4360e50ed56cb483667b8e6dadf2d3fda62359593faabbe749a27c4eaca6/sphinx-8.2.3.tar.gz", hash = "sha256:398ad29dee7f63a75888314e9424d40f52ce5a6a87ae88e7071e80af296ec348", size = 8321876, upload-time = "2025-03-02T22:31:59.658Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/31/53/136e9eca6e0b9dc0e1962e2c908fbea2e5ac000c2a2fbd9a35797958c48b/sphinx-8.2.3-py3-none-any.whl", hash = "sha256:4405915165f13521d875a8c29c8970800a0141c14cc5416a38feca4ea5d9b9c3", size = 3589741, upload-time = "2025-03-02T22:31:56.836Z" },
-]
-
-[[package]]
-name = "sphinx-autobuild"
-version = "2024.10.3"
+version = "9.1.0"
source = { registry = "https://pypi.org/simple" }
-resolution-markers = [
- "python_full_version < '3.11' and sys_platform == 'linux'",
- "python_full_version < '3.11' and sys_platform != 'linux'",
-]
dependencies = [
- { name = "colorama", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinx", version = "8.1.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "starlette", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "uvicorn", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "watchfiles", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "websockets", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "alabaster" },
+ { name = "babel" },
+ { name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "docutils" },
+ { name = "imagesize" },
+ { name = "jinja2" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
+ { name = "pygments" },
+ { name = "requests" },
+ { name = "roman-numerals" },
+ { name = "snowballstemmer" },
+ { name = "sphinxcontrib-applehelp" },
+ { name = "sphinxcontrib-devhelp" },
+ { name = "sphinxcontrib-htmlhelp" },
+ { name = "sphinxcontrib-jsmath" },
+ { name = "sphinxcontrib-qthelp" },
+ { name = "sphinxcontrib-serializinghtml" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/a5/2c/155e1de2c1ba96a72e5dba152c509a8b41e047ee5c2def9e9f0d812f8be7/sphinx_autobuild-2024.10.3.tar.gz", hash = "sha256:248150f8f333e825107b6d4b86113ab28fa51750e5f9ae63b59dc339be951fb1", size = 14023, upload-time = "2024-10-02T23:15:30.172Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/cd/bd/f08eb0f4eed5c83f1ba2a3bd18f7745a2b1525fad70660a1c00224ec468a/sphinx-9.1.0.tar.gz", hash = "sha256:7741722357dd75f8190766926071fed3bdc211c74dd2d7d4df5404da95930ddb", size = 8718324, upload-time = "2025-12-31T15:09:27.646Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/18/c0/eba125db38c84d3c74717008fd3cb5000b68cd7e2cbafd1349c6a38c3d3b/sphinx_autobuild-2024.10.3-py3-none-any.whl", hash = "sha256:158e16c36f9d633e613c9aaf81c19b0fc458ca78b112533b20dafcda430d60fa", size = 11908, upload-time = "2024-10-02T23:15:28.739Z" },
+ { url = "https://files.pythonhosted.org/packages/73/f7/b1884cb3188ab181fc81fa00c266699dab600f927a964df02ec3d5d1916a/sphinx-9.1.0-py3-none-any.whl", hash = "sha256:c84fdd4e782504495fe4f2c0b3413d6c2bf388589bb352d439b2a3bb99991978", size = 3921742, upload-time = "2025-12-31T15:09:25.561Z" },
]
[[package]]
name = "sphinx-autobuild"
version = "2025.8.25"
source = { registry = "https://pypi.org/simple" }
-resolution-markers = [
- "python_full_version >= '3.14' and sys_platform == 'linux'",
- "python_full_version == '3.13.*' and sys_platform == 'linux'",
- "python_full_version == '3.12.*' and sys_platform == 'linux'",
- "python_full_version >= '3.14' and sys_platform != 'linux'",
- "python_full_version == '3.13.*' and sys_platform != 'linux'",
- "python_full_version == '3.12.*' and sys_platform != 'linux'",
- "python_full_version == '3.11.*' and sys_platform == 'linux'",
- "python_full_version == '3.11.*' and sys_platform != 'linux'",
-]
-dependencies = [
- { name = "colorama", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinx", version = "8.2.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "starlette", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "uvicorn", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "watchfiles", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "websockets", marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+dependencies = [
+ { name = "colorama" },
+ { name = "sphinx" },
+ { name = "starlette" },
+ { name = "uvicorn" },
+ { name = "watchfiles" },
+ { name = "websockets" },
]
sdist = { url = "https://files.pythonhosted.org/packages/e0/3c/a59a3a453d4133777f7ed2e83c80b7dc817d43c74b74298ca0af869662ad/sphinx_autobuild-2025.8.25.tar.gz", hash = "sha256:9cf5aab32853c8c31af572e4fecdc09c997e2b8be5a07daf2a389e270e85b213", size = 15200, upload-time = "2025-08-25T18:44:55.436Z" }
wheels = [
@@ -5208,7 +5996,6 @@ version = "0.5.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "astroid" },
- { name = "tomli", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
{ name = "typing-extensions" },
]
sdist = { url = "https://files.pythonhosted.org/packages/17/5f/5350046d1aa1a56b063ae08b9ad871025335c9d55fe2372896ea48711da9/sphinx_autodoc2-0.5.0.tar.gz", hash = "sha256:7d76044aa81d6af74447080182b6868c7eb066874edc835e8ddf810735b6565a", size = 115077, upload-time = "2023-11-27T07:27:51.407Z" }
@@ -5221,8 +6008,7 @@ name = "sphinx-copybutton"
version = "0.5.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
- { name = "sphinx", version = "8.1.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "sphinx", version = "8.2.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "sphinx" },
]
sdist = { url = "https://files.pythonhosted.org/packages/fc/2b/a964715e7f5295f77509e59309959f4125122d648f86b4fe7d70ca1d882c/sphinx-copybutton-0.5.2.tar.gz", hash = "sha256:4cf17c82fb9646d1bc9ca92ac280813a3b605d8c421225fd9913154103ee1fbd", size = 23039, upload-time = "2023-04-14T08:10:22.998Z" }
wheels = [
@@ -5284,25 +6070,75 @@ wheels = [
]
[[package]]
-name = "starlette"
-version = "0.50.0"
+name = "sqlalchemy"
+version = "2.0.48"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "greenlet", marker = "platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64'" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/1f/73/b4a9737255583b5fa858e0bb8e116eb94b88c910164ed2ed719147bde3de/sqlalchemy-2.0.48.tar.gz", hash = "sha256:5ca74f37f3369b45e1f6b7b06afb182af1fd5dde009e4ffd831830d98cbe5fe7", size = 9886075, upload-time = "2026-03-02T15:28:51.474Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ef/91/a42ae716f8925e9659df2da21ba941f158686856107a61cc97a95e7647a3/sqlalchemy-2.0.48-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:348174f228b99f33ca1f773e85510e08927620caa59ffe7803b37170df30332b", size = 2155737, upload-time = "2026-03-02T15:49:13.207Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/52/f75f516a1f3888f027c1cfb5d22d4376f4b46236f2e8669dcb0cddc60275/sqlalchemy-2.0.48-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:53667b5f668991e279d21f94ccfa6e45b4e3f4500e7591ae59a8012d0f010dcb", size = 3337020, upload-time = "2026-03-02T15:50:34.547Z" },
+ { url = "https://files.pythonhosted.org/packages/37/9a/0c28b6371e0cdcb14f8f1930778cb3123acfcbd2c95bb9cf6b4a2ba0cce3/sqlalchemy-2.0.48-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:34634e196f620c7a61d18d5cf7dc841ca6daa7961aed75d532b7e58b309ac894", size = 3349983, upload-time = "2026-03-02T15:53:25.542Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/46/0aee8f3ff20b1dcbceb46ca2d87fcc3d48b407925a383ff668218509d132/sqlalchemy-2.0.48-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:546572a1793cc35857a2ffa1fe0e58571af1779bcc1ffa7c9fb0839885ed69a9", size = 3279690, upload-time = "2026-03-02T15:50:36.277Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/8c/a957bc91293b49181350bfd55e6dfc6e30b7f7d83dc6792d72043274a390/sqlalchemy-2.0.48-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:07edba08061bc277bfdc772dd2a1a43978f5a45994dd3ede26391b405c15221e", size = 3314738, upload-time = "2026-03-02T15:53:27.519Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/44/1d257d9f9556661e7bdc83667cc414ba210acfc110c82938cb3611eea58f/sqlalchemy-2.0.48-cp312-cp312-win32.whl", hash = "sha256:908a3fa6908716f803b86896a09a2c4dde5f5ce2bb07aacc71ffebb57986ce99", size = 2115546, upload-time = "2026-03-02T15:54:31.591Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/af/c3c7e1f3a2b383155a16454df62ae8c62a30dd238e42e68c24cebebbfae6/sqlalchemy-2.0.48-cp312-cp312-win_amd64.whl", hash = "sha256:68549c403f79a8e25984376480959975212a670405e3913830614432b5daa07a", size = 2142484, upload-time = "2026-03-02T15:54:34.072Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/c6/569dc8bf3cd375abc5907e82235923e986799f301cd79a903f784b996fca/sqlalchemy-2.0.48-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e3070c03701037aa418b55d36532ecb8f8446ed0135acb71c678dbdf12f5b6e4", size = 2152599, upload-time = "2026-03-02T15:49:14.41Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/ff/f4e04a4bd5a24304f38cb0d4aa2ad4c0fb34999f8b884c656535e1b2b74c/sqlalchemy-2.0.48-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2645b7d8a738763b664a12a1542c89c940daa55196e8d73e55b169cc5c99f65f", size = 3278825, upload-time = "2026-03-02T15:50:38.269Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/88/cb59509e4668d8001818d7355d9995be90c321313078c912420603a7cb95/sqlalchemy-2.0.48-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b19151e76620a412c2ac1c6f977ab1b9fa7ad43140178345136456d5265b32ed", size = 3295200, upload-time = "2026-03-02T15:53:29.366Z" },
+ { url = "https://files.pythonhosted.org/packages/87/dc/1609a4442aefd750ea2f32629559394ec92e89ac1d621a7f462b70f736ff/sqlalchemy-2.0.48-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5b193a7e29fd9fa56e502920dca47dffe60f97c863494946bd698c6058a55658", size = 3226876, upload-time = "2026-03-02T15:50:39.802Z" },
+ { url = "https://files.pythonhosted.org/packages/37/c3/6ae2ab5ea2fa989fbac4e674de01224b7a9d744becaf59bb967d62e99bed/sqlalchemy-2.0.48-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:36ac4ddc3d33e852da9cb00ffb08cea62ca05c39711dc67062ca2bb1fae35fd8", size = 3265045, upload-time = "2026-03-02T15:53:31.421Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/82/ea4665d1bb98c50c19666e672f21b81356bd6077c4574e3d2bbb84541f53/sqlalchemy-2.0.48-cp313-cp313-win32.whl", hash = "sha256:389b984139278f97757ea9b08993e7b9d1142912e046ab7d82b3fbaeb0209131", size = 2113700, upload-time = "2026-03-02T15:54:35.825Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/2b/b9040bec58c58225f073f5b0c1870defe1940835549dafec680cbd58c3c3/sqlalchemy-2.0.48-cp313-cp313-win_amd64.whl", hash = "sha256:d612c976cbc2d17edfcc4c006874b764e85e990c29ce9bd411f926bbfb02b9a2", size = 2139487, upload-time = "2026-03-02T15:54:37.079Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/f4/7b17bd50244b78a49d22cc63c969d71dc4de54567dc152a9b46f6fae40ce/sqlalchemy-2.0.48-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:69f5bc24904d3bc3640961cddd2523e361257ef68585d6e364166dfbe8c78fae", size = 3558851, upload-time = "2026-03-02T15:57:48.607Z" },
+ { url = "https://files.pythonhosted.org/packages/20/0d/213668e9aca61d370f7d2a6449ea4ec699747fac67d4bda1bb3d129025be/sqlalchemy-2.0.48-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fd08b90d211c086181caed76931ecfa2bdfc83eea3cfccdb0f82abc6c4b876cb", size = 3525525, upload-time = "2026-03-02T16:04:38.058Z" },
+ { url = "https://files.pythonhosted.org/packages/85/d7/a84edf412979e7d59c69b89a5871f90a49228360594680e667cb2c46a828/sqlalchemy-2.0.48-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:1ccd42229aaac2df431562117ac7e667d702e8e44afdb6cf0e50fa3f18160f0b", size = 3466611, upload-time = "2026-03-02T15:57:50.759Z" },
+ { url = "https://files.pythonhosted.org/packages/86/55/42404ce5770f6be26a2b0607e7866c31b9a4176c819e9a7a5e0a055770be/sqlalchemy-2.0.48-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:f0dcbc588cd5b725162c076eb9119342f6579c7f7f55057bb7e3c6ff27e13121", size = 3475812, upload-time = "2026-03-02T16:04:40.092Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/ae/29b87775fadc43e627cf582fe3bda4d02e300f6b8f2747c764950d13784c/sqlalchemy-2.0.48-cp313-cp313t-win32.whl", hash = "sha256:9764014ef5e58aab76220c5664abb5d47d5bc858d9debf821e55cfdd0f128485", size = 2141335, upload-time = "2026-03-02T15:52:51.518Z" },
+ { url = "https://files.pythonhosted.org/packages/91/44/f39d063c90f2443e5b46ec4819abd3d8de653893aae92df42a5c4f5843de/sqlalchemy-2.0.48-cp313-cp313t-win_amd64.whl", hash = "sha256:e2f35b4cccd9ed286ad62e0a3c3ac21e06c02abc60e20aa51a3e305a30f5fa79", size = 2173095, upload-time = "2026-03-02T15:52:52.79Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/b3/f437eaa1cf028bb3c927172c7272366393e73ccd104dcf5b6963f4ab5318/sqlalchemy-2.0.48-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:e2d0d88686e3d35a76f3e15a34e8c12d73fc94c1dea1cd55782e695cc14086dd", size = 2154401, upload-time = "2026-03-02T15:49:17.24Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/1c/b3abdf0f402aa3f60f0df6ea53d92a162b458fca2321d8f1f00278506402/sqlalchemy-2.0.48-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:49b7bddc1eebf011ea5ab722fdbe67a401caa34a350d278cc7733c0e88fecb1f", size = 3274528, upload-time = "2026-03-02T15:50:41.489Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/5e/327428a034407651a048f5e624361adf3f9fbac9d0fa98e981e9c6ff2f5e/sqlalchemy-2.0.48-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:426c5ca86415d9b8945c7073597e10de9644802e2ff502b8e1f11a7a2642856b", size = 3279523, upload-time = "2026-03-02T15:53:32.962Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/ca/ece73c81a918add0965b76b868b7b5359e068380b90ef1656ee995940c02/sqlalchemy-2.0.48-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:288937433bd44e3990e7da2402fabc44a3c6c25d3704da066b85b89a85474ae0", size = 3224312, upload-time = "2026-03-02T15:50:42.996Z" },
+ { url = "https://files.pythonhosted.org/packages/88/11/fbaf1ae91fa4ee43f4fe79661cead6358644824419c26adb004941bdce7c/sqlalchemy-2.0.48-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:8183dc57ae7d9edc1346e007e840a9f3d6aa7b7f165203a99e16f447150140d2", size = 3246304, upload-time = "2026-03-02T15:53:34.937Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/a8/5fb0deb13930b4f2f698c5541ae076c18981173e27dd00376dbaea7a9c82/sqlalchemy-2.0.48-cp314-cp314-win32.whl", hash = "sha256:1182437cb2d97988cfea04cf6cdc0b0bb9c74f4d56ec3d08b81e23d621a28cc6", size = 2116565, upload-time = "2026-03-02T15:54:38.321Z" },
+ { url = "https://files.pythonhosted.org/packages/95/7e/e83615cb63f80047f18e61e31e8e32257d39458426c23006deeaf48f463b/sqlalchemy-2.0.48-cp314-cp314-win_amd64.whl", hash = "sha256:144921da96c08feb9e2b052c5c5c1d0d151a292c6135623c6b2c041f2a45f9e0", size = 2142205, upload-time = "2026-03-02T15:54:39.831Z" },
+ { url = "https://files.pythonhosted.org/packages/83/e3/69d8711b3f2c5135e9cde5f063bc1605860f0b2c53086d40c04017eb1f77/sqlalchemy-2.0.48-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5aee45fd2c6c0f2b9cdddf48c48535e7471e42d6fb81adfde801da0bd5b93241", size = 3563519, upload-time = "2026-03-02T15:57:52.387Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/4f/a7cce98facca73c149ea4578981594aaa5fd841e956834931de503359336/sqlalchemy-2.0.48-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7cddca31edf8b0653090cbb54562ca027c421c58ddde2c0685f49ff56a1690e0", size = 3528611, upload-time = "2026-03-02T16:04:42.097Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/7d/5936c7a03a0b0cb0fa0cc425998821c6029756b0855a8f7ee70fba1de955/sqlalchemy-2.0.48-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:7a936f1bb23d370b7c8cc079d5fce4c7d18da87a33c6744e51a93b0f9e97e9b3", size = 3472326, upload-time = "2026-03-02T15:57:54.423Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/33/cea7dfc31b52904efe3dcdc169eb4514078887dff1f5ae28a7f4c5d54b3c/sqlalchemy-2.0.48-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:e004aa9248e8cb0a5f9b96d003ca7c1c0a5da8decd1066e7b53f59eb8ce7c62b", size = 3478453, upload-time = "2026-03-02T16:04:44.584Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/95/32107c4d13be077a9cae61e9ae49966a35dc4bf442a8852dd871db31f62e/sqlalchemy-2.0.48-cp314-cp314t-win32.whl", hash = "sha256:b8438ec5594980d405251451c5b7ea9aa58dda38eb7ac35fb7e4c696712ee24f", size = 2147209, upload-time = "2026-03-02T15:52:54.274Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/d7/1e073da7a4bc645eb83c76067284a0374e643bc4be57f14cc6414656f92c/sqlalchemy-2.0.48-cp314-cp314t-win_amd64.whl", hash = "sha256:d854b3970067297f3a7fbd7a4683587134aa9b3877ee15aa29eea478dc68f933", size = 2182198, upload-time = "2026-03-02T15:52:55.606Z" },
+ { url = "https://files.pythonhosted.org/packages/46/2c/9664130905f03db57961b8980b05cab624afd114bf2be2576628a9f22da4/sqlalchemy-2.0.48-py3-none-any.whl", hash = "sha256:a66fe406437dd65cacd96a72689a3aaaecaebbcd62d81c5ac1c0fdbeac835096", size = 1940202, upload-time = "2026-03-02T15:52:43.285Z" },
+]
+
+[[package]]
+name = "sse-starlette"
+version = "3.3.4"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "anyio" },
- { name = "typing-extensions", marker = "python_full_version < '3.13'" },
+ { name = "starlette" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/ba/b8/73a0e6a6e079a9d9cfa64113d771e421640b6f679a52eeb9b32f72d871a1/starlette-0.50.0.tar.gz", hash = "sha256:a2a17b22203254bcbc2e1f926d2d55f3f9497f769416b3190768befe598fa3ca", size = 2646985, upload-time = "2025-11-01T15:25:27.516Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/26/8c/f9290339ef6d79badbc010f067cd769d6601ec11a57d78569c683fb4dd87/sse_starlette-3.3.4.tar.gz", hash = "sha256:aaf92fc067af8a5427192895ac028e947b484ac01edbc3caf00e7e7137c7bef1", size = 32427, upload-time = "2026-03-29T09:00:23.307Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/d9/52/1064f510b141bd54025f9b55105e26d1fa970b9be67ad766380a3c9b74b0/starlette-0.50.0-py3-none-any.whl", hash = "sha256:9e5391843ec9b6e472eed1365a78c8098cfceb7a74bfd4d6b1c0c0095efb3bca", size = 74033, upload-time = "2025-11-01T15:25:25.461Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/7f/3de5402f39890ac5660b86bcf5c03f9d855dad5c4ed764866d7b592b46fd/sse_starlette-3.3.4-py3-none-any.whl", hash = "sha256:84bb06e58939a8b38d8341f1bc9792f06c2b53f48c608dd207582b664fc8f3c1", size = 14330, upload-time = "2026-03-29T09:00:21.846Z" },
]
[[package]]
-name = "strenum"
-version = "0.4.15"
+name = "starlette"
+version = "0.52.1"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/85/ad/430fb60d90e1d112a62ff57bdd1f286ec73a2a0331272febfddd21f330e1/StrEnum-0.4.15.tar.gz", hash = "sha256:878fb5ab705442070e4dd1929bb5e2249511c0bcf2b0eeacf3bcd80875c82eff", size = 23384, upload-time = "2023-06-29T22:02:58.399Z" }
+dependencies = [
+ { name = "anyio" },
+ { name = "typing-extensions", marker = "python_full_version < '3.13'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/c4/68/79977123bb7be889ad680d79a40f339082c1978b5cfcf62c2d8d196873ac/starlette-0.52.1.tar.gz", hash = "sha256:834edd1b0a23167694292e94f597773bc3f89f362be6effee198165a35d62933", size = 2653702, upload-time = "2026-01-18T13:34:11.062Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/81/69/297302c5f5f59c862faa31e6cb9a4cd74721cd1e052b38e464c5b402df8b/StrEnum-0.4.15-py3-none-any.whl", hash = "sha256:a30cda4af7cc6b5bf52c8055bc4bf4b2b6b14a93b574626da33df53cf7740659", size = 8851, upload-time = "2023-06-29T22:02:56.947Z" },
+ { url = "https://files.pythonhosted.org/packages/81/0d/13d1d239a25cbfb19e740db83143e95c772a1fe10202dda4b76792b114dd/starlette-0.52.1-py3-none-any.whl", hash = "sha256:0029d43eb3d273bc4f83a08720b4912ea4b071087a3b48db01b7c839f7954d74", size = 74272, upload-time = "2026-01-18T13:34:09.188Z" },
]
[[package]]
@@ -5310,7 +6146,7 @@ name = "sympy"
version = "1.14.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
- { name = "mpmath", marker = "sys_platform != 'linux'" },
+ { name = "mpmath" },
]
sdist = { url = "https://files.pythonhosted.org/packages/83/d3/803453b36afefb7c2bb238361cd4ae6125a569b4db67cd9e79846ba2d68c/sympy-1.14.0.tar.gz", hash = "sha256:d3d3fe8df1e5a0b42f0e7bdf50541697dbe7d23746e894990c030e2b05e72517", size = 7793921, upload-time = "2025-04-27T18:05:01.611Z" }
wheels = [
@@ -5319,11 +6155,20 @@ wheels = [
[[package]]
name = "tabulate"
-version = "0.9.0"
+version = "0.10.0"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/ec/fe/802052aecb21e3797b8f7902564ab6ea0d60ff8ca23952079064155d1ae1/tabulate-0.9.0.tar.gz", hash = "sha256:0095b12bf5966de529c0feb1fa08671671b3368eec77d7ef7ab114be2c068b3c", size = 81090, upload-time = "2022-10-06T17:21:48.54Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/46/58/8c37dea7bbf769b20d58e7ace7e5edfe65b849442b00ffcdd56be88697c6/tabulate-0.10.0.tar.gz", hash = "sha256:e2cfde8f79420f6deeffdeda9aaec3b6bc5abce947655d17ac662b126e48a60d", size = 91754, upload-time = "2026-03-04T18:55:34.402Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/40/44/4a5f08c96eb108af5cb50b41f76142f0afa346dfa99d5296fe7202a11854/tabulate-0.9.0-py3-none-any.whl", hash = "sha256:024ca478df22e9340661486f85298cff5f6dcdba14f3813e8830015b9ed1948f", size = 35252, upload-time = "2022-10-06T17:21:44.262Z" },
+ { url = "https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl", hash = "sha256:f0b0622e567335c8fabaaa659f1b33bcb6ddfe2e496071b743aa113f8774f2d3", size = 39814, upload-time = "2026-03-04T18:55:31.284Z" },
+]
+
+[[package]]
+name = "tenacity"
+version = "9.1.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/47/c6/ee486fd809e357697ee8a44d3d69222b344920433d3b6666ccd9b374630c/tenacity-9.1.4.tar.gz", hash = "sha256:adb31d4c263f2bd041081ab33b498309a57c77f9acf2db65aadf0898179cf93a", size = 49413, upload-time = "2026-02-07T10:45:33.841Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d7/c1/eb8f9debc45d3b7918a32ab756658a0904732f75e555402972246b0b8e71/tenacity-9.1.4-py3-none-any.whl", hash = "sha256:6095a360c919085f28c6527de529e76a06ad89b23659fa881ae0649b867a9d55", size = 28926, upload-time = "2026-02-07T10:45:32.24Z" },
]
[[package]]
@@ -5334,9 +6179,10 @@ dependencies = [
{ name = "absl-py" },
{ name = "grpcio" },
{ name = "markdown" },
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "packaging" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
{ name = "pillow" },
{ name = "protobuf" },
{ name = "setuptools" },
@@ -5359,84 +6205,128 @@ wheels = [
[[package]]
name = "tensorstore"
-version = "0.1.78"
+version = "0.1.74"
source = { registry = "https://pypi.org/simple" }
resolution-markers = [
- "python_full_version < '3.11' and sys_platform == 'linux'",
- "python_full_version < '3.11' and sys_platform != 'linux'",
-]
-dependencies = [
- { name = "ml-dtypes", marker = "python_full_version < '3.11'" },
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/9f/ee/05eb424437f4db63331c90e4605025eedc0f71da3faff97161d5d7b405af/tensorstore-0.1.78.tar.gz", hash = "sha256:e26074ffe462394cf54197eb76d6569b500f347573cd74da3f4dd5f510a4ad7c", size = 6913502, upload-time = "2025-10-06T17:44:29.649Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/7a/1e/77eff7bb320f72a9cb6e9a19eee4d78bee4a6ac1c28ceef60df28b4ab670/tensorstore-0.1.78-cp310-cp310-macosx_10_14_x86_64.whl", hash = "sha256:f1bc58164ad964d9cc298d20b62ca704ab6241639a21015e47ce6ea5b5cae27f", size = 15710776, upload-time = "2025-10-06T17:43:47.469Z" },
- { url = "https://files.pythonhosted.org/packages/55/df/f74f8004b246006ae03c90c28e32d71eb8a86a5b325d2d84dda327babdcc/tensorstore-0.1.78-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:1910101ea85b6507958da28628ef53712c5311df19a795f449604f82bae6a24b", size = 13771121, upload-time = "2025-10-06T17:43:49.88Z" },
- { url = "https://files.pythonhosted.org/packages/be/b8/ab0d0b2afc53f47fbfd95c10d9ae21d393019aca45c8513657b8d7002f1f/tensorstore-0.1.78-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1e92195db0c8c3ca749f24b1e930ab93382ac27430ac4ad2e3f53fc8f739323f", size = 18154513, upload-time = "2025-10-06T17:43:51.694Z" },
- { url = "https://files.pythonhosted.org/packages/f7/ea/c1b4cc6a089a39f63e8d189a55c715e393995628b12b4c8560b3ae4874ba/tensorstore-0.1.78-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:90570b867f9100f7405e4116c73910d0bd283a101500ea5680c5a8a881ea05c6", size = 20048971, upload-time = "2025-10-06T17:43:54.358Z" },
- { url = "https://files.pythonhosted.org/packages/58/2a/7167087885b12473f20ae4fddb9a8feeed6bd44ea8d42c73ae29ad3d1591/tensorstore-0.1.78-cp310-cp310-win_amd64.whl", hash = "sha256:4de9d4ee93d712cb665890af0738f4d74cac3b9b9a0492d477a3ee63fbbf445b", size = 12707793, upload-time = "2025-10-06T17:43:56.405Z" },
- { url = "https://files.pythonhosted.org/packages/33/b1/45070c393586306cef44c7bfc47ed2eddfb8930e648aaa847f615e3ae797/tensorstore-0.1.78-cp311-cp311-macosx_10_14_x86_64.whl", hash = "sha256:1c91e7ff93561612bd9868f3ee56702b0e4fecb45079a4c152dff9a6aa751913", size = 15712387, upload-time = "2025-10-06T17:43:58.458Z" },
- { url = "https://files.pythonhosted.org/packages/a0/d8/c045da71460301f37704e1ab1eec9e7e480dc711dbd281d86dc3d792c50e/tensorstore-0.1.78-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:781e123d392b2d9115e94b01849797a4540f54cd6d34c6ee32b9491f2f2a399c", size = 13773158, upload-time = "2025-10-06T17:44:00.285Z" },
- { url = "https://files.pythonhosted.org/packages/5b/e8/2b0d48100816649ec516fca31d02ad8028c090324e77b1c309c09a172350/tensorstore-0.1.78-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e650d363ad43754626a828a242785e6359a59fedb171276e9a0c66c0bd963cd4", size = 18154388, upload-time = "2025-10-06T17:44:02.428Z" },
- { url = "https://files.pythonhosted.org/packages/3e/a1/d9be82de18afe764c0fc7fb21b3d3bb0ad12845d202861fff7189afdb99d/tensorstore-0.1.78-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:33fed0ffa7a42ad24ce203486cf039f81b211723b45bd54859ba237a9d3aedb9", size = 20050304, upload-time = "2025-10-06T17:44:04.673Z" },
- { url = "https://files.pythonhosted.org/packages/d1/fc/b980958f91a9780e4dbc1038da723d2ad91307dbe30563359606f78926e5/tensorstore-0.1.78-cp311-cp311-win_amd64.whl", hash = "sha256:c02df3d8de4703d9ee42c8f620b2288f41c19a0fd5ffa907b72a736678e22188", size = 12708115, upload-time = "2025-10-06T17:44:06.574Z" },
- { url = "https://files.pythonhosted.org/packages/d0/5f/5853c04bebaed2d3c0ada9245328ffe3fff8b0f0f1c64f4776f67b42033f/tensorstore-0.1.78-cp312-cp312-macosx_10_14_x86_64.whl", hash = "sha256:ce375a8f6621cdb94638b9cdc5266519db16a58353d4c6920e8b9d6bdd419e21", size = 15727539, upload-time = "2025-10-06T17:44:08.631Z" },
- { url = "https://files.pythonhosted.org/packages/a2/e2/f67fcca8f90258c1cf1326aa366fe10f559f4c60102f53fdcc6614159c45/tensorstore-0.1.78-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:82f68fa5a3b4c84365a667ea0a7465a53d5d969c4d3909ac990f314d1569ffc3", size = 13780753, upload-time = "2025-10-06T17:44:10.488Z" },
- { url = "https://files.pythonhosted.org/packages/57/de/95013db6ef3b6a14b4237b95184c21becdf56d16605bf42903bb141f729e/tensorstore-0.1.78-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5dc0bd6361d73e3f67d70980f96f4e8bcbd8e810b5475a01333ca9c37f0785a5", size = 18157446, upload-time = "2025-10-06T17:44:12.831Z" },
- { url = "https://files.pythonhosted.org/packages/e2/75/6e7cef68cab3a672c6668cc80c399ae6626a498a3ef04b35b3704b41e9cc/tensorstore-0.1.78-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:75a17cef99f05fad9cc6fda37f1a1868d5f1502fd577af13174382931481c948", size = 20060211, upload-time = "2025-10-06T17:44:15.189Z" },
- { url = "https://files.pythonhosted.org/packages/1e/46/4ff3e395c44348c7442523c8ddd8ccc72d9ac81838e7a8f6afdd92131c3e/tensorstore-0.1.78-cp312-cp312-win_amd64.whl", hash = "sha256:56271d4652a7cb445879089f620af47801c091765d35a005505d6bfb8d00c535", size = 12711274, upload-time = "2025-10-06T17:44:17.586Z" },
- { url = "https://files.pythonhosted.org/packages/18/36/cfb5a2acf9005896c88f80b93c2aee42f00fab9d0045369fef6e1b297242/tensorstore-0.1.78-cp313-cp313-macosx_10_14_x86_64.whl", hash = "sha256:8a1d0ae7996c80f2e623be5b8cfbc32a307d08dfef3d2dcb455f592908ecd46d", size = 15727334, upload-time = "2025-10-06T17:44:19.93Z" },
- { url = "https://files.pythonhosted.org/packages/54/cd/d1bcc3aab5be4298616dbc060b5aa2012b686270aaa16a9579c7945d0a1c/tensorstore-0.1.78-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:311846cfb2d644cd4a7861005e521a79816093e76d7924c83de5d06ca323067e", size = 13780722, upload-time = "2025-10-06T17:44:21.822Z" },
- { url = "https://files.pythonhosted.org/packages/e2/3b/b0bb4440a9d67859b1abb367e436c62b0a27991dd7109f20be9dabff488f/tensorstore-0.1.78-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:630538a66eb9964bd2975c4e09ae83be9984f2e4ebd5f7969983137bfda92071", size = 18157269, upload-time = "2025-10-06T17:44:23.743Z" },
- { url = "https://files.pythonhosted.org/packages/68/d6/d95cde18ca2475bf317051b2be168cc963c5cfcd67e9c59786326ccdca53/tensorstore-0.1.78-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6886bec93b8ba22f83c4dc9e7c1ee20b11025ea9a5a839de21d0cbf7fd7aada2", size = 20060053, upload-time = "2025-10-06T17:44:25.942Z" },
- { url = "https://files.pythonhosted.org/packages/db/a2/dbd1af0e97d5d549051309d72c6e3f2fe81fae636f9db3692d21adc9c731/tensorstore-0.1.78-cp313-cp313-win_amd64.whl", hash = "sha256:e0073de8fa3074bc4cc92ced0210310fd89851899faf42a5ba256f0ba87d095c", size = 12711250, upload-time = "2025-10-06T17:44:27.926Z" },
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "ml-dtypes", version = "0.4.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/3c/b9/ea25aba62c688a87d7d7d9cc5926d602e2f9e84fa72586825486fb180b7e/tensorstore-0.1.74.tar.gz", hash = "sha256:a062875f27283d30ce4959c408c253ecb336fce8e3f9837c064e3d30cda79203", size = 6795605, upload-time = "2025-04-24T15:42:18.829Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a1/14/2e6d1cad744af9e9a1a78d881a908a859ad95b61b15de10397069f55fbd8/tensorstore-0.1.74-cp312-cp312-macosx_10_14_x86_64.whl", hash = "sha256:7218722ee5d74e4d01f357917d3b1b7b1d6b1c068aa73e3d801cb3d58fc45116", size = 15334307, upload-time = "2025-04-24T15:41:48.315Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/ac/8d572b8c6d689eb50db0252e9d35ee6278a6aed481b64d7e025cf51e32c4/tensorstore-0.1.74-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:a6926554a8633d0210bdba619d3996fff6a6af4214237fbca626e6ddfcc8ea39", size = 13288669, upload-time = "2025-04-24T15:41:50.808Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/6c/3e76d614ad70b61670686d91abaa3ddee6b01255bf2b40f050beb15b7970/tensorstore-0.1.74-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d584e468eb4ef8195f5d21a9da4780cf96c6074b87ef219b43a89efce3d503ca", size = 17031720, upload-time = "2025-04-24T15:41:55.092Z" },
+ { url = "https://files.pythonhosted.org/packages/31/f3/09d7c3ad7c9517f89b5be9b4460b83333e98dce1c9ab0a52464ded0bab67/tensorstore-0.1.74-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e0af2225431d59f8a2bb4db4c1519252f10ee407e6550875d78212d3d34ee743", size = 18378829, upload-time = "2025-04-24T15:41:58.167Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/f2/45ece38705280ed9ebf4ccaf084ed1e76e35b1eeec8c510e589978ac8dcd/tensorstore-0.1.74-cp312-cp312-win_amd64.whl", hash = "sha256:4e35f3679873cdc488aae20b9ae2cea4589c7b147a80edb07eb3f09eba47d43d", size = 12432300, upload-time = "2025-04-24T15:42:00.761Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/e9/a08c6a6eb7d6b4b26053d4575196a06c6fccf4e89f9bc625f81e7c91bb5d/tensorstore-0.1.74-cp313-cp313-macosx_10_14_x86_64.whl", hash = "sha256:f7d2c80de9ab352ca14aeca798d6650c5670725e6f8eac73f4fcc8f3147ca614", size = 15334469, upload-time = "2025-04-24T15:42:03.731Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/a9/64b90c6e66e0b8043e641090144c6614b0c78d9a719b9110d953d13a516d/tensorstore-0.1.74-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:ceef7d2dcfd1caf61356f7eeb9a37896b4825b4be2750b00615cf5fb1ae47a8b", size = 13288791, upload-time = "2025-04-24T15:42:06.145Z" },
+ { url = "https://files.pythonhosted.org/packages/62/e8/226cfc25d7eac00e783ff2ee4994830c4a42cd8690e207c4a8b93210f3d9/tensorstore-0.1.74-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e71637002a806bc1b0f0f05556d1c33493a43f3ab35f9632b3d48855677d93dc", size = 17031815, upload-time = "2025-04-24T15:42:09.239Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/09/dce8a0942d84f6bb039b5ea3e8bc6a479b1a9535cd216b0d42dd03c4f761/tensorstore-0.1.74-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c799edf9000aee68d6676e3d2f73d4e1a56fc817c47e150732f6d3bd2b1ef46d", size = 18378091, upload-time = "2025-04-24T15:42:13.546Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/23/5218575d25de9d8debfb3faf290a1e3b9a7b6be9e77ba07ff3a63a0bc899/tensorstore-0.1.74-cp313-cp313-win_amd64.whl", hash = "sha256:5da86437ffa1ee0f0c590c38daa2f4b548890ce66b1f470ac98714cb0eabdbf5", size = 12432635, upload-time = "2025-04-24T15:42:16.275Z" },
]
[[package]]
name = "tensorstore"
-version = "0.1.80"
+version = "0.1.82"
source = { registry = "https://pypi.org/simple" }
resolution-markers = [
- "python_full_version >= '3.14' and sys_platform == 'linux'",
- "python_full_version == '3.13.*' and sys_platform == 'linux'",
- "python_full_version == '3.12.*' and sys_platform == 'linux'",
- "python_full_version >= '3.14' and sys_platform != 'linux'",
- "python_full_version == '3.13.*' and sys_platform != 'linux'",
- "python_full_version == '3.12.*' and sys_platform != 'linux'",
- "python_full_version == '3.11.*' and sys_platform == 'linux'",
- "python_full_version == '3.11.*' and sys_platform != 'linux'",
-]
-dependencies = [
- { name = "ml-dtypes", marker = "python_full_version >= '3.11'" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/88/18/7b91daa9cf29dbb6bfdd603154f355c9069a9cd8c757038fe52b0f613611/tensorstore-0.1.80.tar.gz", hash = "sha256:4158fe76b96f62d12a37d7868150d836e089b5280b2bdd363c43c5d651f10e26", size = 7090032, upload-time = "2025-12-10T21:35:10.941Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/96/1f/902d822626a6c2774229236440c85c17e384f53afb4d2c6fa4118a30c53a/tensorstore-0.1.80-cp311-cp311-macosx_10_14_x86_64.whl", hash = "sha256:246641a8780ee5e04e88bc95c8e31faac6471bab1180d1f5cdc9804b29a77c04", size = 16519587, upload-time = "2025-12-10T21:34:05.758Z" },
- { url = "https://files.pythonhosted.org/packages/21/c9/2ed6ed809946d7b0de08645800584937912c404b85900eea66361d5e2541/tensorstore-0.1.80-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:7451b30f99d9f31a2b9d70e6ef61815713dc782c58c6d817f91781341e4dac05", size = 14550336, upload-time = "2025-12-10T21:34:08.394Z" },
- { url = "https://files.pythonhosted.org/packages/d6/50/d97acbc5a4d632590dd9053697181fa41cbcb09389e88acfa6958ab8ead5/tensorstore-0.1.80-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1113a6982fc0fa8dda8fcc0495715e647ac3360909a86ff13f2e04564f82d54a", size = 19004795, upload-time = "2025-12-10T21:34:11.14Z" },
- { url = "https://files.pythonhosted.org/packages/a9/2d/fdbbf3cd6f08d41d3c1d8a2f6a67a4a2a07ac238fb6eeea852c2669184a3/tensorstore-0.1.80-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b193a7a1c4f455a61e60ed2dd67271a3daab0910ddb4bd9db51390d1b36d9996", size = 20996847, upload-time = "2025-12-10T21:34:14.031Z" },
- { url = "https://files.pythonhosted.org/packages/b6/37/4570fe93f0c5c339843042556a841cfe0073d3e7fa4dae7ba31417eb4fd3/tensorstore-0.1.80-cp311-cp311-win_amd64.whl", hash = "sha256:9c088e8c9f67c266ef4dae3703bd617f7c0cb0fd98e99c4500692e38a4328140", size = 13258296, upload-time = "2025-12-10T21:34:16.764Z" },
- { url = "https://files.pythonhosted.org/packages/c3/47/8733a99926caca2db6e8dbe22491c0623da2298a23bc649bfe6e6f645fa7/tensorstore-0.1.80-cp312-cp312-macosx_10_14_x86_64.whl", hash = "sha256:f65dfaf9e737a41389e29a5a2ea52ca5d14c8d6f48b402c723d800cd16d322b0", size = 16537887, upload-time = "2025-12-10T21:34:19.799Z" },
- { url = "https://files.pythonhosted.org/packages/50/54/59a34fee963e46f9f401c54131bdc6a17d6cfb10e5a094d586d33ae273df/tensorstore-0.1.80-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f8b51d7e685bbb63f6becd7d2ac8634d5ab67ec7e53038e597182e2db2c7aa90", size = 14551674, upload-time = "2025-12-10T21:34:22.171Z" },
- { url = "https://files.pythonhosted.org/packages/87/15/0734521f8b648e2c43a00f1bc99a7195646c9e4e31f64ab22a15ac84e75c/tensorstore-0.1.80-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:acb8d52fadcefafef4ef8ecca3fc99b1d0e3c5c5a888766484c3e39f050be7f5", size = 19013402, upload-time = "2025-12-10T21:34:24.961Z" },
- { url = "https://files.pythonhosted.org/packages/48/85/55addd16896343ea2731388028945576060139dda3c68a15d6b00158ef6f/tensorstore-0.1.80-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bc28a58c580253a526a4b6d239d18181ef96f1e285a502dbb03ff15eeec07a5b", size = 21007488, upload-time = "2025-12-10T21:34:28.093Z" },
- { url = "https://files.pythonhosted.org/packages/c3/d2/5075cfea2ffd13c5bd2e91d76cdf87a355f617e40fa0b8fbfbbdc5e7bd23/tensorstore-0.1.80-cp312-cp312-win_amd64.whl", hash = "sha256:1b2b2ed0051dfab7e25295b14e6620520729e6e2ddf505f98c8d3917569614bf", size = 13263376, upload-time = "2025-12-10T21:34:30.797Z" },
- { url = "https://files.pythonhosted.org/packages/79/3d/34e64ef1e4573419671b9aa72b69e927702d84e1d95bcef3cc98a8d63ad5/tensorstore-0.1.80-cp313-cp313-macosx_10_14_x86_64.whl", hash = "sha256:46136fe42ee6dd835d957db37073058aea0b78fdfbe2975941640131b7740824", size = 16537403, upload-time = "2025-12-10T21:34:33.404Z" },
- { url = "https://files.pythonhosted.org/packages/94/03/19f45f6134bbb98d13f8de3160271aa4f49466e1a91000c6ab2eec7d9264/tensorstore-0.1.80-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a92505189731fcb03f1c69a84ea4460abb24204bfac1f339448a0621e7def77c", size = 14551401, upload-time = "2025-12-10T21:34:36.041Z" },
- { url = "https://files.pythonhosted.org/packages/f7/fa/d5de3f1b711773e33a329b5fe11de1265b77a13f2a2447fe685ee5d0c1bc/tensorstore-0.1.80-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:de63843706fdfe9565a45567238c5b1e55a0b28bbde6524200b31d29043a9a16", size = 19013246, upload-time = "2025-12-10T21:34:38.507Z" },
- { url = "https://files.pythonhosted.org/packages/87/ee/e874b5a495a7aa14817772a91095971f3a965a4cef5b52ad06a8e15c924f/tensorstore-0.1.80-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6c8dbbdd31cbb28eccfb23dbbd4218fe67bfc32e9cb452875a485b81031c949d", size = 21008391, upload-time = "2025-12-10T21:34:41.332Z" },
- { url = "https://files.pythonhosted.org/packages/2f/99/03bcc5da6a735ffa290f888af1f2c990edc9a375b373d04152d8b6fce3e8/tensorstore-0.1.80-cp313-cp313-win_amd64.whl", hash = "sha256:c0529afab3800749dd245843d3bf0d061a109a8edb77fb345f476e8bccda51b8", size = 13262770, upload-time = "2025-12-10T21:34:43.673Z" },
- { url = "https://files.pythonhosted.org/packages/ef/57/75f65d8ba5829768e67aa978d4c0856956b9bacb279c96f0ee28564b6c41/tensorstore-0.1.80-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:04c29d979eb8b8ee48f873dc13d2701bfd49425500ffc5b848e4ec55b2548281", size = 16543698, upload-time = "2025-12-10T21:34:46.095Z" },
- { url = "https://files.pythonhosted.org/packages/9c/92/17a18eac2cfdb019c36b4362d1a5c614d769a78d10cad0aae3d368fefa0e/tensorstore-0.1.80-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:189d924eaec394c9331e284a9c513ed583e336472a925823b5151cb26f41d091", size = 14552217, upload-time = "2025-12-10T21:34:48.539Z" },
- { url = "https://files.pythonhosted.org/packages/b6/df/71f317633a0cd5270b85d185ac5ce91a749930fc076205d3fae4f1f043ed/tensorstore-0.1.80-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:07e4a84bacf70b78305831897068a9b5ad30326e63bbeb92c4bf7e565fcf5e9e", size = 19020675, upload-time = "2025-12-10T21:34:51.168Z" },
- { url = "https://files.pythonhosted.org/packages/2b/35/f03cdb5edf8e009ff73e48c0c3d0f692a70a7ffc5e393f2ea1761eff89b5/tensorstore-0.1.80-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d2b353b0bd53fedd77fc5a12a1c1a91cacc3cf59e3dd785529c5a54b31d1c7b1", size = 21009171, upload-time = "2025-12-10T21:34:53.979Z" },
- { url = "https://files.pythonhosted.org/packages/51/a9/6cf5675a7d4214ae7fd114c5c7bcf09aa71a57fce6648e187576e60c0c08/tensorstore-0.1.80-cp314-cp314-win_amd64.whl", hash = "sha256:53fd121ccd332bc4cc397f7af45889360c668b43dc3ff6bc3264df0f9886c11a", size = 13653134, upload-time = "2025-12-10T21:34:56.818Z" },
- { url = "https://files.pythonhosted.org/packages/1d/d0/8cd2725c6691387438491d0c1fbbe07235439084722f968c20f07de4119d/tensorstore-0.1.80-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:4baee67fce95f29f593fbab4866119347115eaace887732aa92cfcbb9e6b0748", size = 16620211, upload-time = "2025-12-10T21:34:59.106Z" },
- { url = "https://files.pythonhosted.org/packages/f7/c0/289b8979a08b477ce0622a6c13a59dbe8cda407e4c82c8b2ab0b4f8d1989/tensorstore-0.1.80-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:8cd11027b5a8b66db8d344085a31a1666c78621dac27039c4d571bc4974804a1", size = 14638072, upload-time = "2025-12-10T21:35:01.598Z" },
- { url = "https://files.pythonhosted.org/packages/42/47/5c63024ced48e3f440c131babedef2f5398f48ab81c1aeee6c6193491d1c/tensorstore-0.1.80-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6b7c5dd434bba4ee08fe46bbbdb25c60dd3d47ccb4b8561a9751cf1526da52b8", size = 19024739, upload-time = "2025-12-10T21:35:04.324Z" },
- { url = "https://files.pythonhosted.org/packages/6e/16/d08ade819949e0622f27e949c15b09f7b86ac18f8ac7c4d8bdfb4a711076/tensorstore-0.1.80-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e93df6d34ff5f0f6be245f4d29b99a7c1eef8ad91b50686adf57a5eeea99cb74", size = 21024449, upload-time = "2025-12-10T21:35:08.149Z" },
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.14' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version >= '3.14' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'win32'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform == 'emscripten'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform == 'emscripten'",
+ "python_full_version == '3.13.*' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.13.*' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine != 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.13' and platform_machine == 's390x' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+dependencies = [
+ { name = "ml-dtypes", version = "0.5.4", source = { registry = "https://pypi.org/simple" } },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.13' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/cd/9b/43aedb544937f214dd7c665a7edf1b8b74f2f55d53ebd351c0ce69acf81a/tensorstore-0.1.82.tar.gz", hash = "sha256:ccfceffb7611fc61330f6da24b8b0abd9251d480ac8a5bac5a1729f9ed0c3a9f", size = 7160364, upload-time = "2026-03-13T00:22:16.888Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0d/c3/5ab0b99487b2596bdc0ebd3a569e50415949a63bad90b18e6476de91a7bb/tensorstore-0.1.82-cp312-cp312-macosx_10_14_x86_64.whl", hash = "sha256:f0ac091bd47ea6f051fe11230ad2642c254b46a8fabdd5184b0600556b5529ed", size = 16570668, upload-time = "2026-03-13T00:21:36.386Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/95/92b00a4b2e6192528a9c5bac9f53007acf4aa5d54943b9e114bedb72b2da/tensorstore-0.1.82-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8cae7d0c9b2fa0653f90b147daaf9ed04664cab7d297b9772efcfa088da26cab", size = 14904517, upload-time = "2026-03-13T00:21:38.464Z" },
+ { url = "https://files.pythonhosted.org/packages/46/7e/c9c8ad65ee4015787e32d31bcf8278fcb27109e809f8334a64285bd73028/tensorstore-0.1.82-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:34c491ea3c6c1904d4618bfe40020bd83aaeb19d52a266ea0f6919eb3fdc64c4", size = 19344428, upload-time = "2026-03-13T00:21:40.575Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/8a/590bb60a190d414abd2f83dd5b5148722d0c5d310a73e21b7a60ab98cf00/tensorstore-0.1.82-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d4182300d8ffa172e961e79c6bd89e38ce6bc5cd3abf1a7dacb22c2396ce40b7", size = 20964954, upload-time = "2026-03-13T00:21:42.515Z" },
+ { url = "https://files.pythonhosted.org/packages/43/1c/34e6e97426e1718106e9cb74d3045992bdea3ee368f9ea4ea25b809bdba8/tensorstore-0.1.82-cp312-cp312-win_amd64.whl", hash = "sha256:6369809d01edf66cd487cde5c94f57138167c09561f3d906020fd53c72687f92", size = 13393361, upload-time = "2026-03-13T00:21:44.443Z" },
+ { url = "https://files.pythonhosted.org/packages/58/d1/0b39f577f047340f7c466e7f929aba0b83d33a852952ae2dc4242c141ee6/tensorstore-0.1.82-cp313-cp313-macosx_10_14_x86_64.whl", hash = "sha256:9874349ff23a9e94df361e7a0378efd3f22a1b14c1bb4d00905e6477eb56b732", size = 16570239, upload-time = "2026-03-13T00:21:46.655Z" },
+ { url = "https://files.pythonhosted.org/packages/be/41/d33bea17f9afaee862f268fc10c364997267ab29b9be2aeebe01105cb38b/tensorstore-0.1.82-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:cb2b87e8df78dc629e09a001d19b64813f249f9c78e4ade76de26e18f68bc591", size = 14904654, upload-time = "2026-03-13T00:21:48.708Z" },
+ { url = "https://files.pythonhosted.org/packages/16/b9/f9f3d00e84724968d1111bbcf5b9ec2797496f4849e86a4fdea7278f7b0d/tensorstore-0.1.82-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3e0d4f5240247986c66154c3e6c71deed5ef337ae5a52509b3125c8045717bb3", size = 19343727, upload-time = "2026-03-13T00:21:50.664Z" },
+ { url = "https://files.pythonhosted.org/packages/3b/8f/570fb1069b9789b47376bdc8129371bd3dc62bbaf57054816527e79ff88a/tensorstore-0.1.82-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9f2c51d0c40a3a4e49590a1ec07494c518c46905c8f3ec1f5583120cfba3b2cf", size = 20964994, upload-time = "2026-03-13T00:21:52.918Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/d7/e1f168c6d82fd4af1acfade95f0ba4fe3593bac9e9a81ec074a80fe6258c/tensorstore-0.1.82-cp313-cp313-win_amd64.whl", hash = "sha256:82bbac5e11eeaa80ad1aedad1c7a8f1f4f39362c5f56906820b21fc34a497100", size = 13393826, upload-time = "2026-03-13T00:21:55.459Z" },
+ { url = "https://files.pythonhosted.org/packages/95/c2/c75d42a223b5367ae0b7e10c847f6180139582cdaf51e30e28ad29721fd6/tensorstore-0.1.82-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:aa9d7b3f092a65b5573e6c9919bea1e16c909844f346c82407dc454a67a3fa11", size = 16574644, upload-time = "2026-03-13T00:21:57.382Z" },
+ { url = "https://files.pythonhosted.org/packages/37/86/b2c19cc443c9fb69d682d0e5d67ac4c165edde4e4a92adbcaa6a1ec084ed/tensorstore-0.1.82-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:32f70923d3a5dd687ebfd4eb9d0892766bff9acef92a468852c1872e96bbb440", size = 14906299, upload-time = "2026-03-13T00:21:59.563Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/71/e88cd2e6859adbd414669827800b98db646ce5156b264a34f4f0fbeb488b/tensorstore-0.1.82-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:35607c5c0135d31c1b7bd821ad0446840161708a289df52cffc796d0321f3d60", size = 19345817, upload-time = "2026-03-13T00:22:01.682Z" },
+ { url = "https://files.pythonhosted.org/packages/65/e8/48dfcf42c344980564e01052900fb2a3a28d90d515133fe69bdded70df6c/tensorstore-0.1.82-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:54d40a696115a8d13184920842a20c570bdb1cb3ba2352b05394814608290f6a", size = 20966508, upload-time = "2026-03-13T00:22:04.61Z" },
+ { url = "https://files.pythonhosted.org/packages/16/65/2e465b576f61618a8a1a0e068811298a7338e9163713bcc24f5fe4abbf6c/tensorstore-0.1.82-cp314-cp314-win_amd64.whl", hash = "sha256:c7f63af7aabdf3a3e224d5b36c924bcb59ebc4fb8e485edc8fe13b8bf8b1ba32", size = 13785613, upload-time = "2026-03-13T00:22:06.643Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/e3/49a49e0b1605a58f31aed5ee3833b3a088984b16b5c3e7efaf34bd990ccb/tensorstore-0.1.82-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:69950d352327473014299a57f4c9fc7e0caa9c9e9100b3bc0a0c37f79c47fe6d", size = 16651920, upload-time = "2026-03-13T00:22:08.539Z" },
+ { url = "https://files.pythonhosted.org/packages/77/69/bb0b929a2b1a1b72f15f6d9c5337b3ce0117de625f46345f56c815c106ee/tensorstore-0.1.82-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:0224e20fad9ca9538c3e8ac4a32ef354acaa7ab2c130e4944c2eda58c3200742", size = 14988973, upload-time = "2026-03-13T00:22:10.493Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/e6/847146a4d802fd258eb032226ce3153167c4d0f44f4176633a77beb3af14/tensorstore-0.1.82-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c45dae1b34cad5bd56796e961c35ceb5a70617e4eb182faf73dd9cc4b21f3f87", size = 19365580, upload-time = "2026-03-13T00:22:12.679Z" },
+ { url = "https://files.pythonhosted.org/packages/b3/06/46261b7ec4f6707edf9da8d4a2d68b4819b599e0f9b4906d5bfcec7fd5b2/tensorstore-0.1.82-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d8678ce55c4ca9daac815995d47aae6d3648c75dcdbb9f01326067ccc4de10a", size = 20981853, upload-time = "2026-03-13T00:22:14.817Z" },
+]
+
+[[package]]
+name = "tibs"
+version = "0.5.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/57/cd/6cf028decf1c2df4d26077dd5d0532587d93d4917233d5e004133166a940/tibs-0.5.7.tar.gz", hash = "sha256:173dfbecb2309edd9771f453580c88cf251e775613461566b23dbd756b3d54cb", size = 78255, upload-time = "2026-03-12T13:06:29.79Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a5/4f/1149a5cf2c1be6862e1dcba0c22134c43c44f05ddeef4697ecf20067e508/tibs-0.5.7-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:01ea5258bdf942d21560dc07d532082cd04f07cfef65fedd58ae84f7d0d2562a", size = 401281, upload-time = "2026-03-12T13:06:25.78Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/af/59041580d51eb06077029cc64f0b2f9165b1c87075b7fe85f400e01ec6f9/tibs-0.5.7-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:f5eea45851c960628a2bd29847765d55e19a687c5374456ad2c8cf6410eb1efa", size = 377945, upload-time = "2026-03-12T13:06:42.493Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/73/3b614d39221f02fca2f37dcdc1c65e25c963bf1da4b90ad9db393f9c130d/tibs-0.5.7-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6a9feed5931b881809a950eca0e01e757113e2383a2af06a3e6982f110c869e2", size = 409620, upload-time = "2026-03-12T13:06:28.611Z" },
+ { url = "https://files.pythonhosted.org/packages/16/a6/917ca6ca266135f0f52041700c4eb766097258dd987b81a630c061969db5/tibs-0.5.7-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:501728d096e10d9a165aa526743d47418a6bbfd7b084fa47ecb22be7641d3edb", size = 426017, upload-time = "2026-03-12T13:06:40.139Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/f6/3c795420f81bac44390d897712aebe186186d88ea5653e20f4ac5097b0b1/tibs-0.5.7-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:77103a9f1af72ac4cf5006828d0fb21578d19ce55fd990e9a1c8e46fd549561f", size = 449717, upload-time = "2026-03-12T13:06:45.416Z" },
+ { url = "https://files.pythonhosted.org/packages/98/00/700b97377b55973ac233a280d6ff81c0187710c73a5ac3356ef79bf15eb2/tibs-0.5.7-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:1f95d5db62960205a1e9eba73ce67dc14e7366ae080cd4e5b6f005ebd90faf02", size = 453131, upload-time = "2026-03-12T13:06:46.623Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/41/38ccfe6fe48432ea20f6e6a49a42aeb9662042e5f4e8f9a4029047a6c44a/tibs-0.5.7-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ace018a057459e3dccd06a4aae1c5c8cd57e352b263dcef534ae39bf3e03b5cf", size = 419054, upload-time = "2026-03-12T13:06:27.25Z" },
+ { url = "https://files.pythonhosted.org/packages/73/08/d9a66639564b92d5be07eb30bbd7a5b9052f338da09fd4ec3732346ff129/tibs-0.5.7-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:2a618de62004d9217d2d2ab0f7f9bbdd098c12642dc01f07b3fb00f0b5f3131a", size = 448585, upload-time = "2026-03-12T13:06:33.306Z" },
+ { url = "https://files.pythonhosted.org/packages/70/c1/24131985486d5bf878468226d9d0bdff5a0b04838b773a7339d22965f74e/tibs-0.5.7-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:42725200f1b02687ed6e6a1c01e0ec150dc829d21d901ffc74cc0ac4d821f57f", size = 586259, upload-time = "2026-03-12T13:06:14.095Z" },
+ { url = "https://files.pythonhosted.org/packages/02/0c/f74c6672d28054c55b6c593588792858be420dbf4b56d0adbf79fc1b7f8f/tibs-0.5.7-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:63255749f937c5e6fedcc7d54e7bd359aef711017e6855f373b0510a14ee2215", size = 701427, upload-time = "2026-03-12T13:06:37.234Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/bf/2c39836a5a1664cda596ba069d065322976245a5f86dab9f2b9a3eaff024/tibs-0.5.7-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:4b7510235379368b7523f624d46e0680f3706e3a3965877a6583cdcb598b8bac", size = 660754, upload-time = "2026-03-12T13:06:38.67Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/77/5a7a10001c38f4d1266d4f7a84fae27357c88834a0266bc401e37e1a7884/tibs-0.5.7-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:29480bf03e3372a5f9cc59ea0541f76f8efd696d4f0d214715e94247c342a037", size = 631034, upload-time = "2026-03-12T13:06:18.1Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/be/bb20938ab5d1e63ee4c5cf78be815ab2a8674e7aa0b2500db210f7db3e6d/tibs-0.5.7-cp314-cp314t-win32.whl", hash = "sha256:b9535dc7b7484904a58b51bd8e64da7efbf1d8466ff7e84ed1d78f4ddc561c99", size = 278952, upload-time = "2026-03-12T13:06:20.285Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/9a/e76888e8567dbe02a67a27d46e5acf06e3504df1268ebc6d8313942ec560/tibs-0.5.7-cp314-cp314t-win_amd64.whl", hash = "sha256:1906729038b85c3b4c040aa28a456d85bc976d0c5007177350eb73374ffa0fd0", size = 294069, upload-time = "2026-03-12T13:06:15.756Z" },
+ { url = "https://files.pythonhosted.org/packages/10/37/f74a5f4288984cb909dbccd4cc254154f3ed97b16db1913406f1bd2914c9/tibs-0.5.7-cp314-cp314t-win_arm64.whl", hash = "sha256:7d6592ed93c6748acd39df484c1ee24d40ee247c2a20ca38ba03363506fd24f3", size = 278929, upload-time = "2026-03-12T13:06:43.962Z" },
+ { url = "https://files.pythonhosted.org/packages/12/2d/de2c579d3eea0f18212b5b16decb04568b7a0ef912d00581a77492609d4e/tibs-0.5.7-cp38-abi3-macosx_10_12_x86_64.whl", hash = "sha256:859f05315ffb307d3474c505d694f3a547f00730a024c982f5f60316a5505b3c", size = 411352, upload-time = "2026-03-12T13:06:52.016Z" },
+ { url = "https://files.pythonhosted.org/packages/74/71/4c21ccc5c2e1672f9cd91ed2c46604c250cffd9d386113772dded128b5cf/tibs-0.5.7-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:a883ca13a922a66b2c1326a9c188123a574741a72510a4bf52fd6f97db191e44", size = 383971, upload-time = "2026-03-12T13:06:50.143Z" },
+ { url = "https://files.pythonhosted.org/packages/38/85/399940ac5393772792a209911a5efa42cf55cf621771e48b863211ac5a2a/tibs-0.5.7-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f70bd250769381c73110d6f24feaf8b6fcd44f680b3cb28a20ea06db3d04fb6f", size = 416256, upload-time = "2026-03-12T13:06:24.222Z" },
+ { url = "https://files.pythonhosted.org/packages/02/94/481a73e74d398949f57d297b1809a10a951d252e7ec94b6715ed952ce500/tibs-0.5.7-cp38-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:76746f01b3db9dbd802f5e615f11f68df7a29ecef521b082dca53f3fa7d0084f", size = 428003, upload-time = "2026-03-12T13:06:23.064Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/e0/72db1760a7f7fec1d5f3690e0855fbbccbcf0a4a2fd318c9d71f3b33f3a7/tibs-0.5.7-cp38-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:847709c108800ad6a45efaf9a040628278956938a4897f7427a2587013dc3b98", size = 455589, upload-time = "2026-03-12T13:06:53.144Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/26/9cd3395914bf705d6ae1e9a6c323f727e9dc88fef716327ce7f486e0b55a/tibs-0.5.7-cp38-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ad61df93b50f875b277ab736c5d37b6bce56f9abce489a22f4e02d9daa2966e3", size = 459266, upload-time = "2026-03-12T13:06:21.678Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/3b/267f19a008d13c704dc0b044138a56239272a43531ccb05464129d0fbd01/tibs-0.5.7-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e13b9c7ff2604b0146772025e1ac6f85c8c625bf6ac73736ff671eaf357dda41", size = 423466, upload-time = "2026-03-12T13:06:41.212Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/d4/424ae3515e0e013ad83186074bf3beb53399b9052c00da703415ccc316ca/tibs-0.5.7-cp38-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:0a7ce857ef05c59dc61abadc31c4b9b1e3c62f9e5fb29217988c308936aea71e", size = 452080, upload-time = "2026-03-12T13:06:32.112Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/15/ab80beba83a134745439d33763e1d3b017f994abeb9c309a3ac9fd94e90e/tibs-0.5.7-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:1d5521cc6768bfa6282a0c591ba06b079ab91b5c7d5696925ad2abac59779a54", size = 592311, upload-time = "2026-03-12T13:06:47.807Z" },
+ { url = "https://files.pythonhosted.org/packages/4c/21/f5cf41c15431e63aeaefb494e714d48d9e9061b4e01fcc01d1987e2e5faa/tibs-0.5.7-cp38-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:477608f9b87e24a22ab6d50b81da04a5cb59bfa49598ff7ec5165035a18fb392", size = 703400, upload-time = "2026-03-12T13:06:16.968Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/ec/b3bdb7dcc3de8513c5678a685f4e25bb85ef48526d7d535ddc592f9e8602/tibs-0.5.7-cp38-abi3-musllinux_1_2_i686.whl", hash = "sha256:ac0aa2aae38f7325c91c261ce1d18f769c4c7033c98d6ea3ea5534585cf16452", size = 664623, upload-time = "2026-03-12T13:06:48.894Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/71/7b85af3ad1b2cd9871c8f50ba0eb17e54e12481b467678535e58aced0d98/tibs-0.5.7-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:1b56583db148e5094d781c3d746815dbcbb6378c6f813c8ce291efd4ab21da8b", size = 635199, upload-time = "2026-03-12T13:06:34.798Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/63/60220fb502beb857306afd4a5bac4a8617ae496f3b1f4968d127380fdefe/tibs-0.5.7-cp38-abi3-win32.whl", hash = "sha256:d4f3ff613d486650816bc5516760c0382a2cc0ca8aeddd8914d011bc3b81d9a2", size = 288454, upload-time = "2026-03-12T13:06:30.978Z" },
+ { url = "https://files.pythonhosted.org/packages/46/ab/aab78827ba7e0d65fe346b86d1d61e0792c38d5f9b7547e0f71b7027c835/tibs-0.5.7-cp38-abi3-win_amd64.whl", hash = "sha256:a61d36155f8ab8642e1b6744e13822f72050fc7ec4f86ec6965295afa04949e2", size = 304135, upload-time = "2026-03-12T13:06:35.884Z" },
+ { url = "https://files.pythonhosted.org/packages/48/59/e9e6a610928a4bcbf04f0ac1436ee320aa8cbe95181f1aa32687c50e858b/tibs-0.5.7-cp38-abi3-win_arm64.whl", hash = "sha256:130bc68ff500fc8185677df7a97350b5d5339e6ba7e325bc3031337f6424ede7", size = 289272, upload-time = "2026-03-12T13:06:19.247Z" },
]
[[package]]
@@ -5449,20 +6339,6 @@ dependencies = [
]
sdist = { url = "https://files.pythonhosted.org/packages/7d/ab/4d017d0f76ec3171d469d80fc03dfbb4e48a4bcaddaa831b31d526f05edc/tiktoken-0.12.0.tar.gz", hash = "sha256:b18ba7ee2b093863978fcb14f74b3707cdc8d4d4d3836853ce7ec60772139931", size = 37806, upload-time = "2025-10-06T20:22:45.419Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/89/b3/2cb7c17b6c4cf8ca983204255d3f1d95eda7213e247e6947a0ee2c747a2c/tiktoken-0.12.0-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:3de02f5a491cfd179aec916eddb70331814bd6bf764075d39e21d5862e533970", size = 1051991, upload-time = "2025-10-06T20:21:34.098Z" },
- { url = "https://files.pythonhosted.org/packages/27/0f/df139f1df5f6167194ee5ab24634582ba9a1b62c6b996472b0277ec80f66/tiktoken-0.12.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:b6cfb6d9b7b54d20af21a912bfe63a2727d9cfa8fbda642fd8322c70340aad16", size = 995798, upload-time = "2025-10-06T20:21:35.579Z" },
- { url = "https://files.pythonhosted.org/packages/ef/5d/26a691f28ab220d5edc09b9b787399b130f24327ef824de15e5d85ef21aa/tiktoken-0.12.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:cde24cdb1b8a08368f709124f15b36ab5524aac5fa830cc3fdce9c03d4fb8030", size = 1129865, upload-time = "2025-10-06T20:21:36.675Z" },
- { url = "https://files.pythonhosted.org/packages/b2/94/443fab3d4e5ebecac895712abd3849b8da93b7b7dec61c7db5c9c7ebe40c/tiktoken-0.12.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:6de0da39f605992649b9cfa6f84071e3f9ef2cec458d08c5feb1b6f0ff62e134", size = 1152856, upload-time = "2025-10-06T20:21:37.873Z" },
- { url = "https://files.pythonhosted.org/packages/54/35/388f941251b2521c70dd4c5958e598ea6d2c88e28445d2fb8189eecc1dfc/tiktoken-0.12.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:6faa0534e0eefbcafaccb75927a4a380463a2eaa7e26000f0173b920e98b720a", size = 1195308, upload-time = "2025-10-06T20:21:39.577Z" },
- { url = "https://files.pythonhosted.org/packages/f8/00/c6681c7f833dd410576183715a530437a9873fa910265817081f65f9105f/tiktoken-0.12.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:82991e04fc860afb933efb63957affc7ad54f83e2216fe7d319007dab1ba5892", size = 1255697, upload-time = "2025-10-06T20:21:41.154Z" },
- { url = "https://files.pythonhosted.org/packages/5f/d2/82e795a6a9bafa034bf26a58e68fe9a89eeaaa610d51dbeb22106ba04f0a/tiktoken-0.12.0-cp310-cp310-win_amd64.whl", hash = "sha256:6fb2995b487c2e31acf0a9e17647e3b242235a20832642bb7a9d1a181c0c1bb1", size = 879375, upload-time = "2025-10-06T20:21:43.201Z" },
- { url = "https://files.pythonhosted.org/packages/de/46/21ea696b21f1d6d1efec8639c204bdf20fde8bafb351e1355c72c5d7de52/tiktoken-0.12.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:6e227c7f96925003487c33b1b32265fad2fbcec2b7cf4817afb76d416f40f6bb", size = 1051565, upload-time = "2025-10-06T20:21:44.566Z" },
- { url = "https://files.pythonhosted.org/packages/c9/d9/35c5d2d9e22bb2a5f74ba48266fb56c63d76ae6f66e02feb628671c0283e/tiktoken-0.12.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:c06cf0fcc24c2cb2adb5e185c7082a82cba29c17575e828518c2f11a01f445aa", size = 995284, upload-time = "2025-10-06T20:21:45.622Z" },
- { url = "https://files.pythonhosted.org/packages/01/84/961106c37b8e49b9fdcf33fe007bb3a8fdcc380c528b20cc7fbba80578b8/tiktoken-0.12.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:f18f249b041851954217e9fd8e5c00b024ab2315ffda5ed77665a05fa91f42dc", size = 1129201, upload-time = "2025-10-06T20:21:47.074Z" },
- { url = "https://files.pythonhosted.org/packages/6a/d0/3d9275198e067f8b65076a68894bb52fd253875f3644f0a321a720277b8a/tiktoken-0.12.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:47a5bc270b8c3db00bb46ece01ef34ad050e364b51d406b6f9730b64ac28eded", size = 1152444, upload-time = "2025-10-06T20:21:48.139Z" },
- { url = "https://files.pythonhosted.org/packages/78/db/a58e09687c1698a7c592e1038e01c206569b86a0377828d51635561f8ebf/tiktoken-0.12.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:508fa71810c0efdcd1b898fda574889ee62852989f7c1667414736bcb2b9a4bd", size = 1195080, upload-time = "2025-10-06T20:21:49.246Z" },
- { url = "https://files.pythonhosted.org/packages/9e/1b/a9e4d2bf91d515c0f74afc526fd773a812232dd6cda33ebea7f531202325/tiktoken-0.12.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:a1af81a6c44f008cba48494089dd98cccb8b313f55e961a52f5b222d1e507967", size = 1255240, upload-time = "2025-10-06T20:21:50.274Z" },
- { url = "https://files.pythonhosted.org/packages/9d/15/963819345f1b1fb0809070a79e9dd96938d4ca41297367d471733e79c76c/tiktoken-0.12.0-cp311-cp311-win_amd64.whl", hash = "sha256:3e68e3e593637b53e56f7237be560f7a394451cb8c11079755e80ae64b9e6def", size = 879422, upload-time = "2025-10-06T20:21:51.734Z" },
{ url = "https://files.pythonhosted.org/packages/a4/85/be65d39d6b647c79800fd9d29241d081d4eeb06271f383bb87200d74cf76/tiktoken-0.12.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b97f74aca0d78a1ff21b8cd9e9925714c15a9236d6ceacf5c7327c117e6e21e8", size = 1050728, upload-time = "2025-10-06T20:21:52.756Z" },
{ url = "https://files.pythonhosted.org/packages/4a/42/6573e9129bc55c9bf7300b3a35bef2c6b9117018acca0dc760ac2d93dffe/tiktoken-0.12.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:2b90f5ad190a4bb7c3eb30c5fa32e1e182ca1ca79f05e49b448438c3e225a49b", size = 994049, upload-time = "2025-10-06T20:21:53.782Z" },
{ url = "https://files.pythonhosted.org/packages/66/c5/ed88504d2f4a5fd6856990b230b56d85a777feab84e6129af0822f5d0f70/tiktoken-0.12.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:65b26c7a780e2139e73acc193e5c63ac754021f160df919add909c1492c0fb37", size = 1129008, upload-time = "2025-10-06T20:21:54.832Z" },
@@ -5524,10 +6400,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/fd/18/a545c4ea42af3df6effd7d13d250ba77a0a86fb20393143bbb9a92e434d4/tokenizers-0.22.2-cp39-abi3-win32.whl", hash = "sha256:a6bf3f88c554a2b653af81f3204491c818ae2ac6fbc09e76ef4773351292bc92", size = 2502363, upload-time = "2026-01-05T10:45:20.593Z" },
{ url = "https://files.pythonhosted.org/packages/65/71/0670843133a43d43070abeb1949abfdef12a86d490bea9cd9e18e37c5ff7/tokenizers-0.22.2-cp39-abi3-win_amd64.whl", hash = "sha256:c9ea31edff2968b44a88f97d784c2f16dc0729b8b143ed004699ebca91f05c48", size = 2747786, upload-time = "2026-01-05T10:45:18.411Z" },
{ url = "https://files.pythonhosted.org/packages/72/f4/0de46cfa12cdcbcd464cc59fde36912af405696f687e53a091fb432f694c/tokenizers-0.22.2-cp39-abi3-win_arm64.whl", hash = "sha256:9ce725d22864a1e965217204946f830c37876eee3b2ba6fc6255e8e903d5fcbc", size = 2612133, upload-time = "2026-01-05T10:45:17.232Z" },
- { url = "https://files.pythonhosted.org/packages/84/04/655b79dbcc9b3ac5f1479f18e931a344af67e5b7d3b251d2dcdcd7558592/tokenizers-0.22.2-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:753d47ebd4542742ef9261d9da92cd545b2cacbb48349a1225466745bb866ec4", size = 3282301, upload-time = "2026-01-05T10:40:34.858Z" },
- { url = "https://files.pythonhosted.org/packages/46/cd/e4851401f3d8f6f45d8480262ab6a5c8cb9c4302a790a35aa14eeed6d2fd/tokenizers-0.22.2-pp310-pypy310_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e10bf9113d209be7cd046d40fbabbaf3278ff6d18eb4da4c500443185dc1896c", size = 3161308, upload-time = "2026-01-05T10:40:40.737Z" },
- { url = "https://files.pythonhosted.org/packages/6f/6e/55553992a89982cd12d4a66dddb5e02126c58677ea3931efcbe601d419db/tokenizers-0.22.2-pp310-pypy310_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:64d94e84f6660764e64e7e0b22baa72f6cd942279fdbb21d46abd70d179f0195", size = 3718964, upload-time = "2026-01-05T10:40:46.56Z" },
- { url = "https://files.pythonhosted.org/packages/59/8c/b1c87148aa15e099243ec9f0cf9d0e970cc2234c3257d558c25a2c5304e6/tokenizers-0.22.2-pp310-pypy310_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:f01a9c019878532f98927d2bacb79bbb404b43d3437455522a00a30718cdedb5", size = 3373542, upload-time = "2026-01-05T10:40:52.803Z" },
]
[[package]]
@@ -5539,60 +6411,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", hash = "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b", size = 16588, upload-time = "2020-11-01T01:40:20.672Z" },
]
-[[package]]
-name = "tomli"
-version = "2.4.0"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/82/30/31573e9457673ab10aa432461bee537ce6cef177667deca369efb79df071/tomli-2.4.0.tar.gz", hash = "sha256:aa89c3f6c277dd275d8e243ad24f3b5e701491a860d5121f2cdd399fbb31fc9c", size = 17477, upload-time = "2026-01-11T11:22:38.165Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/3c/d9/3dc2289e1f3b32eb19b9785b6a006b28ee99acb37d1d47f78d4c10e28bf8/tomli-2.4.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:b5ef256a3fd497d4973c11bf142e9ed78b150d36f5773f1ca6088c230ffc5867", size = 153663, upload-time = "2026-01-11T11:21:45.27Z" },
- { url = "https://files.pythonhosted.org/packages/51/32/ef9f6845e6b9ca392cd3f64f9ec185cc6f09f0a2df3db08cbe8809d1d435/tomli-2.4.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:5572e41282d5268eb09a697c89a7bee84fae66511f87533a6f88bd2f7b652da9", size = 148469, upload-time = "2026-01-11T11:21:46.873Z" },
- { url = "https://files.pythonhosted.org/packages/d6/c2/506e44cce89a8b1b1e047d64bd495c22c9f71f21e05f380f1a950dd9c217/tomli-2.4.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:551e321c6ba03b55676970b47cb1b73f14a0a4dce6a3e1a9458fd6d921d72e95", size = 236039, upload-time = "2026-01-11T11:21:48.503Z" },
- { url = "https://files.pythonhosted.org/packages/b3/40/e1b65986dbc861b7e986e8ec394598187fa8aee85b1650b01dd925ca0be8/tomli-2.4.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5e3f639a7a8f10069d0e15408c0b96a2a828cfdec6fca05296ebcdcc28ca7c76", size = 243007, upload-time = "2026-01-11T11:21:49.456Z" },
- { url = "https://files.pythonhosted.org/packages/9c/6f/6e39ce66b58a5b7ae572a0f4352ff40c71e8573633deda43f6a379d56b3e/tomli-2.4.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:1b168f2731796b045128c45982d3a4874057626da0e2ef1fdd722848b741361d", size = 240875, upload-time = "2026-01-11T11:21:50.755Z" },
- { url = "https://files.pythonhosted.org/packages/aa/ad/cb089cb190487caa80204d503c7fd0f4d443f90b95cf4ef5cf5aa0f439b0/tomli-2.4.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:133e93646ec4300d651839d382d63edff11d8978be23da4cc106f5a18b7d0576", size = 246271, upload-time = "2026-01-11T11:21:51.81Z" },
- { url = "https://files.pythonhosted.org/packages/0b/63/69125220e47fd7a3a27fd0de0c6398c89432fec41bc739823bcc66506af6/tomli-2.4.0-cp311-cp311-win32.whl", hash = "sha256:b6c78bdf37764092d369722d9946cb65b8767bfa4110f902a1b2542d8d173c8a", size = 96770, upload-time = "2026-01-11T11:21:52.647Z" },
- { url = "https://files.pythonhosted.org/packages/1e/0d/a22bb6c83f83386b0008425a6cd1fa1c14b5f3dd4bad05e98cf3dbbf4a64/tomli-2.4.0-cp311-cp311-win_amd64.whl", hash = "sha256:d3d1654e11d724760cdb37a3d7691f0be9db5fbdaef59c9f532aabf87006dbaa", size = 107626, upload-time = "2026-01-11T11:21:53.459Z" },
- { url = "https://files.pythonhosted.org/packages/2f/6d/77be674a3485e75cacbf2ddba2b146911477bd887dda9d8c9dfb2f15e871/tomli-2.4.0-cp311-cp311-win_arm64.whl", hash = "sha256:cae9c19ed12d4e8f3ebf46d1a75090e4c0dc16271c5bce1c833ac168f08fb614", size = 94842, upload-time = "2026-01-11T11:21:54.831Z" },
- { url = "https://files.pythonhosted.org/packages/3c/43/7389a1869f2f26dba52404e1ef13b4784b6b37dac93bac53457e3ff24ca3/tomli-2.4.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:920b1de295e72887bafa3ad9f7a792f811847d57ea6b1215154030cf131f16b1", size = 154894, upload-time = "2026-01-11T11:21:56.07Z" },
- { url = "https://files.pythonhosted.org/packages/e9/05/2f9bf110b5294132b2edf13fe6ca6ae456204f3d749f623307cbb7a946f2/tomli-2.4.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:7d6d9a4aee98fac3eab4952ad1d73aee87359452d1c086b5ceb43ed02ddb16b8", size = 149053, upload-time = "2026-01-11T11:21:57.467Z" },
- { url = "https://files.pythonhosted.org/packages/e8/41/1eda3ca1abc6f6154a8db4d714a4d35c4ad90adc0bcf700657291593fbf3/tomli-2.4.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:36b9d05b51e65b254ea6c2585b59d2c4cb91c8a3d91d0ed0f17591a29aaea54a", size = 243481, upload-time = "2026-01-11T11:21:58.661Z" },
- { url = "https://files.pythonhosted.org/packages/d2/6d/02ff5ab6c8868b41e7d4b987ce2b5f6a51d3335a70aa144edd999e055a01/tomli-2.4.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1c8a885b370751837c029ef9bc014f27d80840e48bac415f3412e6593bbc18c1", size = 251720, upload-time = "2026-01-11T11:22:00.178Z" },
- { url = "https://files.pythonhosted.org/packages/7b/57/0405c59a909c45d5b6f146107c6d997825aa87568b042042f7a9c0afed34/tomli-2.4.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8768715ffc41f0008abe25d808c20c3d990f42b6e2e58305d5da280ae7d1fa3b", size = 247014, upload-time = "2026-01-11T11:22:01.238Z" },
- { url = "https://files.pythonhosted.org/packages/2c/0e/2e37568edd944b4165735687cbaf2fe3648129e440c26d02223672ee0630/tomli-2.4.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:7b438885858efd5be02a9a133caf5812b8776ee0c969fea02c45e8e3f296ba51", size = 251820, upload-time = "2026-01-11T11:22:02.727Z" },
- { url = "https://files.pythonhosted.org/packages/5a/1c/ee3b707fdac82aeeb92d1a113f803cf6d0f37bdca0849cb489553e1f417a/tomli-2.4.0-cp312-cp312-win32.whl", hash = "sha256:0408e3de5ec77cc7f81960c362543cbbd91ef883e3138e81b729fc3eea5b9729", size = 97712, upload-time = "2026-01-11T11:22:03.777Z" },
- { url = "https://files.pythonhosted.org/packages/69/13/c07a9177d0b3bab7913299b9278845fc6eaaca14a02667c6be0b0a2270c8/tomli-2.4.0-cp312-cp312-win_amd64.whl", hash = "sha256:685306e2cc7da35be4ee914fd34ab801a6acacb061b6a7abca922aaf9ad368da", size = 108296, upload-time = "2026-01-11T11:22:04.86Z" },
- { url = "https://files.pythonhosted.org/packages/18/27/e267a60bbeeee343bcc279bb9e8fbed0cbe224bc7b2a3dc2975f22809a09/tomli-2.4.0-cp312-cp312-win_arm64.whl", hash = "sha256:5aa48d7c2356055feef06a43611fc401a07337d5b006be13a30f6c58f869e3c3", size = 94553, upload-time = "2026-01-11T11:22:05.854Z" },
- { url = "https://files.pythonhosted.org/packages/34/91/7f65f9809f2936e1f4ce6268ae1903074563603b2a2bd969ebbda802744f/tomli-2.4.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:84d081fbc252d1b6a982e1870660e7330fb8f90f676f6e78b052ad4e64714bf0", size = 154915, upload-time = "2026-01-11T11:22:06.703Z" },
- { url = "https://files.pythonhosted.org/packages/20/aa/64dd73a5a849c2e8f216b755599c511badde80e91e9bc2271baa7b2cdbb1/tomli-2.4.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:9a08144fa4cba33db5255f9b74f0b89888622109bd2776148f2597447f92a94e", size = 149038, upload-time = "2026-01-11T11:22:07.56Z" },
- { url = "https://files.pythonhosted.org/packages/9e/8a/6d38870bd3d52c8d1505ce054469a73f73a0fe62c0eaf5dddf61447e32fa/tomli-2.4.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c73add4bb52a206fd0c0723432db123c0c75c280cbd67174dd9d2db228ebb1b4", size = 242245, upload-time = "2026-01-11T11:22:08.344Z" },
- { url = "https://files.pythonhosted.org/packages/59/bb/8002fadefb64ab2669e5b977df3f5e444febea60e717e755b38bb7c41029/tomli-2.4.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1fb2945cbe303b1419e2706e711b7113da57b7db31ee378d08712d678a34e51e", size = 250335, upload-time = "2026-01-11T11:22:09.951Z" },
- { url = "https://files.pythonhosted.org/packages/a5/3d/4cdb6f791682b2ea916af2de96121b3cb1284d7c203d97d92d6003e91c8d/tomli-2.4.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:bbb1b10aa643d973366dc2cb1ad94f99c1726a02343d43cbc011edbfac579e7c", size = 245962, upload-time = "2026-01-11T11:22:11.27Z" },
- { url = "https://files.pythonhosted.org/packages/f2/4a/5f25789f9a460bd858ba9756ff52d0830d825b458e13f754952dd15fb7bb/tomli-2.4.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4cbcb367d44a1f0c2be408758b43e1ffb5308abe0ea222897d6bfc8e8281ef2f", size = 250396, upload-time = "2026-01-11T11:22:12.325Z" },
- { url = "https://files.pythonhosted.org/packages/aa/2f/b73a36fea58dfa08e8b3a268750e6853a6aac2a349241a905ebd86f3047a/tomli-2.4.0-cp313-cp313-win32.whl", hash = "sha256:7d49c66a7d5e56ac959cb6fc583aff0651094ec071ba9ad43df785abc2320d86", size = 97530, upload-time = "2026-01-11T11:22:13.865Z" },
- { url = "https://files.pythonhosted.org/packages/3b/af/ca18c134b5d75de7e8dc551c5234eaba2e8e951f6b30139599b53de9c187/tomli-2.4.0-cp313-cp313-win_amd64.whl", hash = "sha256:3cf226acb51d8f1c394c1b310e0e0e61fecdd7adcb78d01e294ac297dd2e7f87", size = 108227, upload-time = "2026-01-11T11:22:15.224Z" },
- { url = "https://files.pythonhosted.org/packages/22/c3/b386b832f209fee8073c8138ec50f27b4460db2fdae9ffe022df89a57f9b/tomli-2.4.0-cp313-cp313-win_arm64.whl", hash = "sha256:d20b797a5c1ad80c516e41bc1fb0443ddb5006e9aaa7bda2d71978346aeb9132", size = 94748, upload-time = "2026-01-11T11:22:16.009Z" },
- { url = "https://files.pythonhosted.org/packages/f3/c4/84047a97eb1004418bc10bdbcfebda209fca6338002eba2dc27cc6d13563/tomli-2.4.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:26ab906a1eb794cd4e103691daa23d95c6919cc2fa9160000ac02370cc9dd3f6", size = 154725, upload-time = "2026-01-11T11:22:17.269Z" },
- { url = "https://files.pythonhosted.org/packages/a8/5d/d39038e646060b9d76274078cddf146ced86dc2b9e8bbf737ad5983609a0/tomli-2.4.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:20cedb4ee43278bc4f2fee6cb50daec836959aadaf948db5172e776dd3d993fc", size = 148901, upload-time = "2026-01-11T11:22:18.287Z" },
- { url = "https://files.pythonhosted.org/packages/73/e5/383be1724cb30f4ce44983d249645684a48c435e1cd4f8b5cded8a816d3c/tomli-2.4.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:39b0b5d1b6dd03684b3fb276407ebed7090bbec989fa55838c98560c01113b66", size = 243375, upload-time = "2026-01-11T11:22:19.154Z" },
- { url = "https://files.pythonhosted.org/packages/31/f0/bea80c17971c8d16d3cc109dc3585b0f2ce1036b5f4a8a183789023574f2/tomli-2.4.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a26d7ff68dfdb9f87a016ecfd1e1c2bacbe3108f4e0f8bcd2228ef9a766c787d", size = 250639, upload-time = "2026-01-11T11:22:20.168Z" },
- { url = "https://files.pythonhosted.org/packages/2c/8f/2853c36abbb7608e3f945d8a74e32ed3a74ee3a1f468f1ffc7d1cb3abba6/tomli-2.4.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:20ffd184fb1df76a66e34bd1b36b4a4641bd2b82954befa32fe8163e79f1a702", size = 246897, upload-time = "2026-01-11T11:22:21.544Z" },
- { url = "https://files.pythonhosted.org/packages/49/f0/6c05e3196ed5337b9fe7ea003e95fd3819a840b7a0f2bf5a408ef1dad8ed/tomli-2.4.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:75c2f8bbddf170e8effc98f5e9084a8751f8174ea6ccf4fca5398436e0320bc8", size = 254697, upload-time = "2026-01-11T11:22:23.058Z" },
- { url = "https://files.pythonhosted.org/packages/f3/f5/2922ef29c9f2951883525def7429967fc4d8208494e5ab524234f06b688b/tomli-2.4.0-cp314-cp314-win32.whl", hash = "sha256:31d556d079d72db7c584c0627ff3a24c5d3fb4f730221d3444f3efb1b2514776", size = 98567, upload-time = "2026-01-11T11:22:24.033Z" },
- { url = "https://files.pythonhosted.org/packages/7b/31/22b52e2e06dd2a5fdbc3ee73226d763b184ff21fc24e20316a44ccc4d96b/tomli-2.4.0-cp314-cp314-win_amd64.whl", hash = "sha256:43e685b9b2341681907759cf3a04e14d7104b3580f808cfde1dfdb60ada85475", size = 108556, upload-time = "2026-01-11T11:22:25.378Z" },
- { url = "https://files.pythonhosted.org/packages/48/3d/5058dff3255a3d01b705413f64f4306a141a8fd7a251e5a495e3f192a998/tomli-2.4.0-cp314-cp314-win_arm64.whl", hash = "sha256:3d895d56bd3f82ddd6faaff993c275efc2ff38e52322ea264122d72729dca2b2", size = 96014, upload-time = "2026-01-11T11:22:26.138Z" },
- { url = "https://files.pythonhosted.org/packages/b8/4e/75dab8586e268424202d3a1997ef6014919c941b50642a1682df43204c22/tomli-2.4.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:5b5807f3999fb66776dbce568cc9a828544244a8eb84b84b9bafc080c99597b9", size = 163339, upload-time = "2026-01-11T11:22:27.143Z" },
- { url = "https://files.pythonhosted.org/packages/06/e3/b904d9ab1016829a776d97f163f183a48be6a4deb87304d1e0116a349519/tomli-2.4.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c084ad935abe686bd9c898e62a02a19abfc9760b5a79bc29644463eaf2840cb0", size = 159490, upload-time = "2026-01-11T11:22:28.399Z" },
- { url = "https://files.pythonhosted.org/packages/e3/5a/fc3622c8b1ad823e8ea98a35e3c632ee316d48f66f80f9708ceb4f2a0322/tomli-2.4.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0f2e3955efea4d1cfbcb87bc321e00dc08d2bcb737fd1d5e398af111d86db5df", size = 269398, upload-time = "2026-01-11T11:22:29.345Z" },
- { url = "https://files.pythonhosted.org/packages/fd/33/62bd6152c8bdd4c305ad9faca48f51d3acb2df1f8791b1477d46ff86e7f8/tomli-2.4.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0e0fe8a0b8312acf3a88077a0802565cb09ee34107813bba1c7cd591fa6cfc8d", size = 276515, upload-time = "2026-01-11T11:22:30.327Z" },
- { url = "https://files.pythonhosted.org/packages/4b/ff/ae53619499f5235ee4211e62a8d7982ba9e439a0fb4f2f351a93d67c1dd2/tomli-2.4.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:413540dce94673591859c4c6f794dfeaa845e98bf35d72ed59636f869ef9f86f", size = 273806, upload-time = "2026-01-11T11:22:32.56Z" },
- { url = "https://files.pythonhosted.org/packages/47/71/cbca7787fa68d4d0a9f7072821980b39fbb1b6faeb5f5cf02f4a5559fa28/tomli-2.4.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:0dc56fef0e2c1c470aeac5b6ca8cc7b640bb93e92d9803ddaf9ea03e198f5b0b", size = 281340, upload-time = "2026-01-11T11:22:33.505Z" },
- { url = "https://files.pythonhosted.org/packages/f5/00/d595c120963ad42474cf6ee7771ad0d0e8a49d0f01e29576ee9195d9ecdf/tomli-2.4.0-cp314-cp314t-win32.whl", hash = "sha256:d878f2a6707cc9d53a1be1414bbb419e629c3d6e67f69230217bb663e76b5087", size = 108106, upload-time = "2026-01-11T11:22:34.451Z" },
- { url = "https://files.pythonhosted.org/packages/de/69/9aa0c6a505c2f80e519b43764f8b4ba93b5a0bbd2d9a9de6e2b24271b9a5/tomli-2.4.0-cp314-cp314t-win_amd64.whl", hash = "sha256:2add28aacc7425117ff6364fe9e06a183bb0251b03f986df0e78e974047571fd", size = 120504, upload-time = "2026-01-11T11:22:35.764Z" },
- { url = "https://files.pythonhosted.org/packages/b3/9f/f1668c281c58cfae01482f7114a4b88d345e4c140386241a1a24dcc9e7bc/tomli-2.4.0-cp314-cp314t-win_arm64.whl", hash = "sha256:2b1e3b80e1d5e52e40e9b924ec43d81570f0e7d09d11081b797bc4692765a3d4", size = 99561, upload-time = "2026-01-11T11:22:36.624Z" },
- { url = "https://files.pythonhosted.org/packages/23/d1/136eb2cb77520a31e1f64cbae9d33ec6df0d78bdf4160398e86eec8a8754/tomli-2.4.0-py3-none-any.whl", hash = "sha256:1f776e7d669ebceb01dee46484485f43a4048746235e683bcdffacdf1fb4785a", size = 14477, upload-time = "2026-01-11T11:22:37.446Z" },
-]
-
[[package]]
name = "tomlkit"
version = "0.14.0"
@@ -5604,104 +6422,45 @@ wheels = [
[[package]]
name = "torch"
-version = "2.9.1"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
- { name = "filelock", marker = "sys_platform != 'linux'" },
- { name = "fsspec", marker = "sys_platform != 'linux'" },
- { name = "jinja2", marker = "sys_platform != 'linux'" },
- { name = "networkx", version = "3.4.2", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and sys_platform != 'linux') or (python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (sys_platform == 'linux' and extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "networkx", version = "3.6.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and sys_platform != 'linux') or (python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (sys_platform == 'linux' and extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "setuptools", marker = "python_full_version >= '3.12' and sys_platform != 'linux'" },
- { name = "sympy", marker = "sys_platform != 'linux'" },
+version = "2.11.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cuda-bindings", marker = "sys_platform == 'linux'" },
+ { name = "cuda-toolkit", extra = ["cublas", "cudart", "cufft", "cufile", "cupti", "curand", "cusolver", "cusparse", "nvjitlink", "nvrtc", "nvtx"], marker = "sys_platform == 'linux' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "filelock", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "fsspec", version = "2026.2.0", source = { registry = "https://pypi.org/simple" }, marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "jinja2", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "networkx", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "nvidia-cudnn-cu13", marker = "sys_platform == 'linux'" },
+ { name = "nvidia-cusparselt-cu13", marker = "sys_platform == 'linux'" },
+ { name = "nvidia-nccl-cu13", marker = "sys_platform == 'linux'" },
+ { name = "nvidia-nvshmem-cu13", marker = "sys_platform == 'linux'" },
+ { name = "setuptools", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "sympy", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
{ name = "triton", marker = "sys_platform == 'never'" },
- { name = "typing-extensions", marker = "sys_platform != 'linux'" },
-]
-wheels = [
- { url = "https://files.pythonhosted.org/packages/5f/56/9577683b23072075ed2e40d725c52c2019d71a972fab8e083763da8e707e/torch-2.9.1-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:1cc208435f6c379f9b8fdfd5ceb5be1e3b72a6bdf1cb46c0d2812aa73472db9e", size = 104207681, upload-time = "2025-11-12T15:19:56.48Z" },
- { url = "https://files.pythonhosted.org/packages/38/45/be5a74f221df8f4b609b78ff79dc789b0cc9017624544ac4dd1c03973150/torch-2.9.1-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:9fd35c68b3679378c11f5eb73220fdcb4e6f4592295277fbb657d31fd053237c", size = 899794036, upload-time = "2025-11-12T15:21:01.886Z" },
- { url = "https://files.pythonhosted.org/packages/67/95/a581e8a382596b69385a44bab2733f1273d45c842f5d4a504c0edc3133b6/torch-2.9.1-cp310-cp310-win_amd64.whl", hash = "sha256:2af70e3be4a13becba4655d6cc07dcfec7ae844db6ac38d6c1dafeb245d17d65", size = 110969861, upload-time = "2025-11-12T15:21:30.145Z" },
- { url = "https://files.pythonhosted.org/packages/ad/51/1756dc128d2bf6ea4e0a915cb89ea5e730315ff33d60c1ff56fd626ba3eb/torch-2.9.1-cp310-none-macosx_11_0_arm64.whl", hash = "sha256:a83b0e84cc375e3318a808d032510dde99d696a85fe9473fc8575612b63ae951", size = 74452222, upload-time = "2025-11-12T15:20:46.223Z" },
- { url = "https://files.pythonhosted.org/packages/15/db/c064112ac0089af3d2f7a2b5bfbabf4aa407a78b74f87889e524b91c5402/torch-2.9.1-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:62b3fd888277946918cba4478cf849303da5359f0fb4e3bfb86b0533ba2eaf8d", size = 104220430, upload-time = "2025-11-12T15:20:31.705Z" },
- { url = "https://files.pythonhosted.org/packages/56/be/76eaa36c9cd032d3b01b001e2c5a05943df75f26211f68fae79e62f87734/torch-2.9.1-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:d033ff0ac3f5400df862a51bdde9bad83561f3739ea0046e68f5401ebfa67c1b", size = 899821446, upload-time = "2025-11-12T15:20:15.544Z" },
- { url = "https://files.pythonhosted.org/packages/47/cc/7a2949e38dfe3244c4df21f0e1c27bce8aedd6c604a587dd44fc21017cb4/torch-2.9.1-cp311-cp311-win_amd64.whl", hash = "sha256:0d06b30a9207b7c3516a9e0102114024755a07045f0c1d2f2a56b1819ac06bcb", size = 110973074, upload-time = "2025-11-12T15:21:39.958Z" },
- { url = "https://files.pythonhosted.org/packages/1e/ce/7d251155a783fb2c1bb6837b2b7023c622a2070a0a72726ca1df47e7ea34/torch-2.9.1-cp311-none-macosx_11_0_arm64.whl", hash = "sha256:52347912d868653e1528b47cafaf79b285b98be3f4f35d5955389b1b95224475", size = 74463887, upload-time = "2025-11-12T15:20:36.611Z" },
- { url = "https://files.pythonhosted.org/packages/0f/27/07c645c7673e73e53ded71705045d6cb5bae94c4b021b03aa8d03eee90ab/torch-2.9.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:da5f6f4d7f4940a173e5572791af238cb0b9e21b1aab592bd8b26da4c99f1cd6", size = 104126592, upload-time = "2025-11-12T15:20:41.62Z" },
- { url = "https://files.pythonhosted.org/packages/19/17/e377a460603132b00760511299fceba4102bd95db1a0ee788da21298ccff/torch-2.9.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:27331cd902fb4322252657f3902adf1c4f6acad9dcad81d8df3ae14c7c4f07c4", size = 899742281, upload-time = "2025-11-12T15:22:17.602Z" },
- { url = "https://files.pythonhosted.org/packages/b1/1a/64f5769025db846a82567fa5b7d21dba4558a7234ee631712ee4771c436c/torch-2.9.1-cp312-cp312-win_amd64.whl", hash = "sha256:81a285002d7b8cfd3fdf1b98aa8df138d41f1a8334fd9ea37511517cedf43083", size = 110940568, upload-time = "2025-11-12T15:21:18.689Z" },
- { url = "https://files.pythonhosted.org/packages/6e/ab/07739fd776618e5882661d04c43f5b5586323e2f6a2d7d84aac20d8f20bd/torch-2.9.1-cp312-none-macosx_11_0_arm64.whl", hash = "sha256:c0d25d1d8e531b8343bea0ed811d5d528958f1dcbd37e7245bc686273177ad7e", size = 74479191, upload-time = "2025-11-12T15:21:25.816Z" },
- { url = "https://files.pythonhosted.org/packages/20/60/8fc5e828d050bddfab469b3fe78e5ab9a7e53dda9c3bdc6a43d17ce99e63/torch-2.9.1-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:c29455d2b910b98738131990394da3e50eea8291dfeb4b12de71ecf1fdeb21cb", size = 104135743, upload-time = "2025-11-12T15:21:34.936Z" },
- { url = "https://files.pythonhosted.org/packages/f2/b7/6d3f80e6918213babddb2a37b46dbb14c15b14c5f473e347869a51f40e1f/torch-2.9.1-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:524de44cd13931208ba2c4bde9ec7741fd4ae6bfd06409a604fc32f6520c2bc9", size = 899749493, upload-time = "2025-11-12T15:24:36.356Z" },
- { url = "https://files.pythonhosted.org/packages/a6/47/c7843d69d6de8938c1cbb1eba426b1d48ddf375f101473d3e31a5fc52b74/torch-2.9.1-cp313-cp313-win_amd64.whl", hash = "sha256:545844cc16b3f91e08ce3b40e9c2d77012dd33a48d505aed34b7740ed627a1b2", size = 110944162, upload-time = "2025-11-12T15:21:53.151Z" },
- { url = "https://files.pythonhosted.org/packages/28/0e/2a37247957e72c12151b33a01e4df651d9d155dd74d8cfcbfad15a79b44a/torch-2.9.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:5be4bf7496f1e3ffb1dd44b672adb1ac3f081f204c5ca81eba6442f5f634df8e", size = 74830751, upload-time = "2025-11-12T15:21:43.792Z" },
- { url = "https://files.pythonhosted.org/packages/4b/f7/7a18745edcd7b9ca2381aa03353647bca8aace91683c4975f19ac233809d/torch-2.9.1-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:30a3e170a84894f3652434b56d59a64a2c11366b0ed5776fab33c2439396bf9a", size = 104142929, upload-time = "2025-11-12T15:21:48.319Z" },
- { url = "https://files.pythonhosted.org/packages/f4/dd/f1c0d879f2863ef209e18823a988dc7a1bf40470750e3ebe927efdb9407f/torch-2.9.1-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:8301a7b431e51764629208d0edaa4f9e4c33e6df0f2f90b90e261d623df6a4e2", size = 899748978, upload-time = "2025-11-12T15:23:04.568Z" },
- { url = "https://files.pythonhosted.org/packages/1f/9f/6986b83a53b4d043e36f3f898b798ab51f7f20fdf1a9b01a2720f445043d/torch-2.9.1-cp313-cp313t-win_amd64.whl", hash = "sha256:2e1c42c0ae92bf803a4b2409fdfed85e30f9027a66887f5e7dcdbc014c7531db", size = 111176995, upload-time = "2025-11-12T15:22:01.618Z" },
- { url = "https://files.pythonhosted.org/packages/40/60/71c698b466dd01e65d0e9514b5405faae200c52a76901baf6906856f17e4/torch-2.9.1-cp313-none-macosx_11_0_arm64.whl", hash = "sha256:2c14b3da5df416cf9cb5efab83aa3056f5b8cd8620b8fde81b4987ecab730587", size = 74480347, upload-time = "2025-11-12T15:21:57.648Z" },
- { url = "https://files.pythonhosted.org/packages/48/50/c4b5112546d0d13cc9eaa1c732b823d676a9f49ae8b6f97772f795874a03/torch-2.9.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:1edee27a7c9897f4e0b7c14cfc2f3008c571921134522d5b9b5ec4ebbc69041a", size = 74433245, upload-time = "2025-11-12T15:22:39.027Z" },
- { url = "https://files.pythonhosted.org/packages/81/c9/2628f408f0518b3bae49c95f5af3728b6ab498c8624ab1e03a43dd53d650/torch-2.9.1-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:19d144d6b3e29921f1fc70503e9f2fc572cde6a5115c0c0de2f7ca8b1483e8b6", size = 104134804, upload-time = "2025-11-12T15:22:35.222Z" },
- { url = "https://files.pythonhosted.org/packages/28/fc/5bc91d6d831ae41bf6e9e6da6468f25330522e92347c9156eb3f1cb95956/torch-2.9.1-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:c432d04376f6d9767a9852ea0def7b47a7bbc8e7af3b16ac9cf9ce02b12851c9", size = 899747132, upload-time = "2025-11-12T15:23:36.068Z" },
- { url = "https://files.pythonhosted.org/packages/63/5d/e8d4e009e52b6b2cf1684bde2a6be157b96fb873732542fb2a9a99e85a83/torch-2.9.1-cp314-cp314-win_amd64.whl", hash = "sha256:d187566a2cdc726fc80138c3cdb260970fab1c27e99f85452721f7759bbd554d", size = 110934845, upload-time = "2025-11-12T15:22:48.367Z" },
- { url = "https://files.pythonhosted.org/packages/bd/b2/2d15a52516b2ea3f414643b8de68fa4cb220d3877ac8b1028c83dc8ca1c4/torch-2.9.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:cb10896a1f7fedaddbccc2017ce6ca9ecaaf990f0973bdfcf405439750118d2c", size = 74823558, upload-time = "2025-11-12T15:22:43.392Z" },
- { url = "https://files.pythonhosted.org/packages/86/5c/5b2e5d84f5b9850cd1e71af07524d8cbb74cba19379800f1f9f7c997fc70/torch-2.9.1-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:0a2bd769944991c74acf0c4ef23603b9c777fdf7637f115605a4b2d8023110c7", size = 104145788, upload-time = "2025-11-12T15:23:52.109Z" },
- { url = "https://files.pythonhosted.org/packages/a9/8c/3da60787bcf70add986c4ad485993026ac0ca74f2fc21410bc4eb1bb7695/torch-2.9.1-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:07c8a9660bc9414c39cac530ac83b1fb1b679d7155824144a40a54f4a47bfa73", size = 899735500, upload-time = "2025-11-12T15:24:08.788Z" },
- { url = "https://files.pythonhosted.org/packages/db/2b/f7818f6ec88758dfd21da46b6cd46af9d1b3433e53ddbb19ad1e0da17f9b/torch-2.9.1-cp314-cp314t-win_amd64.whl", hash = "sha256:c88d3299ddeb2b35dcc31753305612db485ab6f1823e37fb29451c8b2732b87e", size = 111163659, upload-time = "2025-11-12T15:23:20.009Z" },
-]
-
-[[package]]
-name = "torchprofile"
-version = "0.0.4"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "torch", marker = "sys_platform == 'never'" },
- { name = "torchvision", marker = "sys_platform == 'never'" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/6f/36/574c0c46e818533b78b3c09505211162918188325ab4165ef11a3f295755/torchprofile-0.0.4.tar.gz", hash = "sha256:96b6da17d752a06b02977e078aea95614893b31d4117dd5dcd081f30ce65611b", size = 4557, upload-time = "2021-06-22T04:58:03.592Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/62/15/71ad4ed163b03cba1315f1d96e0bc8e39d5a97f92974ffa610a729b273ab/torchprofile-0.0.4-py3-none-any.whl", hash = "sha256:7151fe88dc770f0eeec241244a4c7feaec2c5e8c7852386bc2d6a8d7dde7384d", size = 7694, upload-time = "2021-06-22T04:58:02.485Z" },
-]
-
-[[package]]
-name = "torchvision"
-version = "0.24.1"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and sys_platform != 'linux' and extra == 'extra-13-megatron-core-dev') or (python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (sys_platform == 'linux' and extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and sys_platform != 'linux' and extra == 'extra-13-megatron-core-dev') or (python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts') or (sys_platform == 'linux' and extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "pillow", marker = "sys_platform != 'linux'" },
- { name = "torch", marker = "sys_platform == 'never'" },
-]
-wheels = [
- { url = "https://files.pythonhosted.org/packages/f7/09/d51aadf8591138e08b74c64a6eb783630c7a31ca2634416277115a9c3a2b/torchvision-0.24.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:ded5e625788572e4e1c4d155d1bbc48805c113794100d70e19c76e39e4d53465", size = 1891441, upload-time = "2025-11-12T15:25:01.687Z" },
- { url = "https://files.pythonhosted.org/packages/6b/49/a35df863e7c153aad82af7505abd8264a5b510306689712ef86bea862822/torchvision-0.24.1-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:54ed17c3d30e718e08d8da3fd5b30ea44b0311317e55647cb97077a29ecbc25b", size = 2386226, upload-time = "2025-11-12T15:25:05.449Z" },
- { url = "https://files.pythonhosted.org/packages/49/20/f2d7cd1eea052887c1083afff0b8df5228ec93b53e03759f20b1a3c6d22a/torchvision-0.24.1-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:f476da4e085b7307aaab6f540219617d46d5926aeda24be33e1359771c83778f", size = 8046093, upload-time = "2025-11-12T15:25:09.425Z" },
- { url = "https://files.pythonhosted.org/packages/d8/cf/0ff4007c09903199307da5f53a192ff5d62b45447069e9ef3a19bdc5ff12/torchvision-0.24.1-cp310-cp310-win_amd64.whl", hash = "sha256:fbdbdae5e540b868a681240b7dbd6473986c862445ee8a138680a6a97d6c34ff", size = 3696202, upload-time = "2025-11-12T15:25:10.657Z" },
- { url = "https://files.pythonhosted.org/packages/e7/69/30f5f03752aa1a7c23931d2519b31e557f3f10af5089d787cddf3b903ecf/torchvision-0.24.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:056c525dc875f18fe8e9c27079ada166a7b2755cea5a2199b0bc7f1f8364e600", size = 1891436, upload-time = "2025-11-12T15:25:04.3Z" },
- { url = "https://files.pythonhosted.org/packages/0c/69/49aae86edb75fe16460b59a191fcc0f568c2378f780bb063850db0fe007a/torchvision-0.24.1-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:1e39619de698e2821d71976c92c8a9e50cdfd1e993507dfb340f2688bfdd8283", size = 2387757, upload-time = "2025-11-12T15:25:06.795Z" },
- { url = "https://files.pythonhosted.org/packages/11/c9/1dfc3db98797b326f1d0c3f3bb61c83b167a813fc7eab6fcd2edb8c7eb9d/torchvision-0.24.1-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:a0f106663e60332aa4fcb1ca2159ef8c3f2ed266b0e6df88de261048a840e0df", size = 8047682, upload-time = "2025-11-12T15:25:21.125Z" },
- { url = "https://files.pythonhosted.org/packages/fa/bb/cfc6a6f6ccc84a534ed1fdf029ae5716dd6ff04e57ed9dc2dab38bf652d5/torchvision-0.24.1-cp311-cp311-win_amd64.whl", hash = "sha256:a9308cdd37d8a42e14a3e7fd9d271830c7fecb150dd929b642f3c1460514599a", size = 4037588, upload-time = "2025-11-12T15:25:14.402Z" },
- { url = "https://files.pythonhosted.org/packages/f0/af/18e2c6b9538a045f60718a0c5a058908ccb24f88fde8e6f0fc12d5ff7bd3/torchvision-0.24.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:e48bf6a8ec95872eb45763f06499f87bd2fb246b9b96cb00aae260fda2f96193", size = 1891433, upload-time = "2025-11-12T15:25:03.232Z" },
- { url = "https://files.pythonhosted.org/packages/9d/43/600e5cfb0643d10d633124f5982d7abc2170dfd7ce985584ff16edab3e76/torchvision-0.24.1-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:7fb7590c737ebe3e1c077ad60c0e5e2e56bb26e7bccc3b9d04dbfc34fd09f050", size = 2386737, upload-time = "2025-11-12T15:25:08.288Z" },
- { url = "https://files.pythonhosted.org/packages/93/b1/db2941526ecddd84884132e2742a55c9311296a6a38627f9e2627f5ac889/torchvision-0.24.1-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:66a98471fc18cad9064123106d810a75f57f0838eee20edc56233fd8484b0cc7", size = 8049868, upload-time = "2025-11-12T15:25:13.058Z" },
- { url = "https://files.pythonhosted.org/packages/69/98/16e583f59f86cd59949f59d52bfa8fc286f86341a229a9d15cbe7a694f0c/torchvision-0.24.1-cp312-cp312-win_amd64.whl", hash = "sha256:4aa6cb806eb8541e92c9b313e96192c6b826e9eb0042720e2fa250d021079952", size = 4302006, upload-time = "2025-11-12T15:25:16.184Z" },
- { url = "https://files.pythonhosted.org/packages/e4/97/ab40550f482577f2788304c27220e8ba02c63313bd74cf2f8920526aac20/torchvision-0.24.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:8a6696db7fb71eadb2c6a48602106e136c785642e598eb1533e0b27744f2cce6", size = 1891435, upload-time = "2025-11-12T15:25:28.642Z" },
- { url = "https://files.pythonhosted.org/packages/30/65/ac0a3f9be6abdbe4e1d82c915d7e20de97e7fd0e9a277970508b015309f3/torchvision-0.24.1-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:db2125c46f9cb25dc740be831ce3ce99303cfe60439249a41b04fd9f373be671", size = 2338718, upload-time = "2025-11-12T15:25:26.19Z" },
- { url = "https://files.pythonhosted.org/packages/10/b5/5bba24ff9d325181508501ed7f0c3de8ed3dd2edca0784d48b144b6c5252/torchvision-0.24.1-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:f035f0cacd1f44a8ff6cb7ca3627d84c54d685055961d73a1a9fb9827a5414c8", size = 8049661, upload-time = "2025-11-12T15:25:22.558Z" },
- { url = "https://files.pythonhosted.org/packages/5c/ec/54a96ae9ab6a0dd66d4bba27771f892e36478a9c3489fa56e51c70abcc4d/torchvision-0.24.1-cp313-cp313-win_amd64.whl", hash = "sha256:16274823b93048e0a29d83415166a2e9e0bf4e1b432668357b657612a4802864", size = 4319808, upload-time = "2025-11-12T15:25:17.318Z" },
- { url = "https://files.pythonhosted.org/packages/d5/f3/a90a389a7e547f3eb8821b13f96ea7c0563cdefbbbb60a10e08dda9720ff/torchvision-0.24.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e3f96208b4bef54cd60e415545f5200346a65024e04f29a26cd0006dbf9e8e66", size = 2005342, upload-time = "2025-11-12T15:25:11.871Z" },
- { url = "https://files.pythonhosted.org/packages/a9/fe/ff27d2ed1b524078164bea1062f23d2618a5fc3208e247d6153c18c91a76/torchvision-0.24.1-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:f231f6a4f2aa6522713326d0d2563538fa72d613741ae364f9913027fa52ea35", size = 2341708, upload-time = "2025-11-12T15:25:25.08Z" },
- { url = "https://files.pythonhosted.org/packages/b1/b9/d6c903495cbdfd2533b3ef6f7b5643ff589ea062f8feb5c206ee79b9d9e5/torchvision-0.24.1-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:1540a9e7f8cf55fe17554482f5a125a7e426347b71de07327d5de6bfd8d17caa", size = 8177239, upload-time = "2025-11-12T15:25:18.554Z" },
- { url = "https://files.pythonhosted.org/packages/4f/2b/ba02e4261369c3798310483028495cf507e6cb3f394f42e4796981ecf3a7/torchvision-0.24.1-cp313-cp313t-win_amd64.whl", hash = "sha256:d83e16d70ea85d2f196d678bfb702c36be7a655b003abed84e465988b6128938", size = 4251604, upload-time = "2025-11-12T15:25:34.069Z" },
- { url = "https://files.pythonhosted.org/packages/42/84/577b2cef8f32094add5f52887867da4c2a3e6b4261538447e9b48eb25812/torchvision-0.24.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:cccf4b4fec7fdfcd3431b9ea75d1588c0a8596d0333245dafebee0462abe3388", size = 2005319, upload-time = "2025-11-12T15:25:23.827Z" },
- { url = "https://files.pythonhosted.org/packages/5f/34/ecb786bffe0159a3b49941a61caaae089853132f3cd1e8f555e3621f7e6f/torchvision-0.24.1-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:1b495edd3a8f9911292424117544f0b4ab780452e998649425d1f4b2bed6695f", size = 2338844, upload-time = "2025-11-12T15:25:32.625Z" },
- { url = "https://files.pythonhosted.org/packages/51/99/a84623786a6969504c87f2dc3892200f586ee13503f519d282faab0bb4f0/torchvision-0.24.1-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:ab211e1807dc3e53acf8f6638df9a7444c80c0ad050466e8d652b3e83776987b", size = 8175144, upload-time = "2025-11-12T15:25:31.355Z" },
- { url = "https://files.pythonhosted.org/packages/6d/ba/8fae3525b233e109317ce6a9c1de922ab2881737b029a7e88021f81e068f/torchvision-0.24.1-cp314-cp314-win_amd64.whl", hash = "sha256:18f9cb60e64b37b551cd605a3d62c15730c086362b40682d23e24b616a697d41", size = 4234459, upload-time = "2025-11-12T15:25:19.859Z" },
- { url = "https://files.pythonhosted.org/packages/50/33/481602c1c72d0485d4b3a6b48c9534b71c2957c9d83bf860eb837bf5a620/torchvision-0.24.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ec9d7379c519428395e4ffda4dbb99ec56be64b0a75b95989e00f9ec7ae0b2d7", size = 2005336, upload-time = "2025-11-12T15:25:27.225Z" },
- { url = "https://files.pythonhosted.org/packages/d0/7f/372de60bf3dd8f5593bd0d03f4aecf0d1fd58f5bc6943618d9d913f5e6d5/torchvision-0.24.1-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:af9201184c2712d808bd4eb656899011afdfce1e83721c7cb08000034df353fe", size = 2341704, upload-time = "2025-11-12T15:25:29.857Z" },
- { url = "https://files.pythonhosted.org/packages/36/9b/0f3b9ff3d0225ee2324ec663de0e7fb3eb855615ca958ac1875f22f1f8e5/torchvision-0.24.1-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:9ef95d819fd6df81bc7cc97b8f21a15d2c0d3ac5dbfaab5cbc2d2ce57114b19e", size = 8177422, upload-time = "2025-11-12T15:25:37.357Z" },
- { url = "https://files.pythonhosted.org/packages/d6/ab/e2bcc7c2f13d882a58f8b30ff86f794210b075736587ea50f8c545834f8a/torchvision-0.24.1-cp314-cp314t-win_amd64.whl", hash = "sha256:480b271d6edff83ac2e8d69bbb4cf2073f93366516a50d48f140ccfceedb002e", size = 4335190, upload-time = "2025-11-12T15:25:35.745Z" },
+ { name = "typing-extensions", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6f/8b/69e3008d78e5cee2b30183340cc425081b78afc5eff3d080daab0adda9aa/torch-2.11.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:4b5866312ee6e52ea625cd211dcb97d6a2cdc1131a5f15cc0d87eec948f6dd34", size = 80606338, upload-time = "2026-03-23T18:11:34.781Z" },
+ { url = "https://files.pythonhosted.org/packages/13/16/42e5915ebe4868caa6bac83a8ed59db57f12e9a61b7d749d584776ed53d5/torch-2.11.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:f99924682ef0aa6a4ab3b1b76f40dc6e273fca09f367d15a524266db100a723f", size = 419731115, upload-time = "2026-03-23T18:11:06.944Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/c9/82638ef24d7877510f83baf821f5619a61b45568ce21c0a87a91576510aa/torch-2.11.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:0f68f4ac6d95d12e896c3b7a912b5871619542ec54d3649cf48cc1edd4dd2756", size = 530712279, upload-time = "2026-03-23T18:10:31.481Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/ff/6756f1c7ee302f6d202120e0f4f05b432b839908f9071157302cedfc5232/torch-2.11.0-cp312-cp312-win_amd64.whl", hash = "sha256:fbf39280699d1b869f55eac536deceaa1b60bd6788ba74f399cc67e60a5fab10", size = 114556047, upload-time = "2026-03-23T18:10:55.931Z" },
+ { url = "https://files.pythonhosted.org/packages/87/89/5ea6722763acee56b045435fb84258db7375c48165ec8be7880ab2b281c5/torch-2.11.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:1e6debd97ccd3205bbb37eb806a9d8219e1139d15419982c09e23ef7d4369d18", size = 80606801, upload-time = "2026-03-23T18:10:18.649Z" },
+ { url = "https://files.pythonhosted.org/packages/32/d1/8ed2173589cbfe744ed54e5a73efc107c0085ba5777ee93a5f4c1ab90553/torch-2.11.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:63a68fa59de8f87acc7e85a5478bb2dddbb3392b7593ec3e78827c793c4b73fd", size = 419732382, upload-time = "2026-03-23T18:08:30.835Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/e1/b73f7c575a4b8f87a5928f50a1e35416b5e27295d8be9397d5293e7e8d4c/torch-2.11.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:cc89b9b173d9adfab59fd227f0ab5e5516d9a52b658ae41d64e59d2e55a418db", size = 530711509, upload-time = "2026-03-23T18:08:47.213Z" },
+ { url = "https://files.pythonhosted.org/packages/66/82/3e3fcdd388fbe54e29fd3f991f36846ff4ac90b0d0181e9c8f7236565f82/torch-2.11.0-cp313-cp313-win_amd64.whl", hash = "sha256:4dda3b3f52d121063a731ddb835f010dc137b920d7fec2778e52f60d8e4bf0cd", size = 114555842, upload-time = "2026-03-23T18:09:52.111Z" },
+ { url = "https://files.pythonhosted.org/packages/db/38/8ac78069621b8c2b4979c2f96dc8409ef5e9c4189f6aac629189a78677ca/torch-2.11.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:8b394322f49af4362d4f80e424bcaca7efcd049619af03a4cf4501520bdf0fb4", size = 80959574, upload-time = "2026-03-23T18:10:14.214Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/6c/56bfb37073e7136e6dd86bfc6af7339946dd684e0ecf2155ac0eee687ae1/torch-2.11.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:2658f34ce7e2dabf4ec73b45e2ca68aedad7a5be87ea756ad656eaf32bf1e1ea", size = 419732324, upload-time = "2026-03-23T18:09:36.604Z" },
+ { url = "https://files.pythonhosted.org/packages/07/f4/1b666b6d61d3394cca306ea543ed03a64aad0a201b6cd159f1d41010aeb1/torch-2.11.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:98bb213c3084cfe176302949bdc360074b18a9da7ab59ef2edc9d9f742504778", size = 530596026, upload-time = "2026-03-23T18:09:20.842Z" },
+ { url = "https://files.pythonhosted.org/packages/48/6b/30d1459fa7e4b67e9e3fe1685ca1d8bb4ce7c62ef436c3a615963c6c866c/torch-2.11.0-cp313-cp313t-win_amd64.whl", hash = "sha256:a97b94bbf62992949b4730c6cd2cc9aee7b335921ee8dc207d930f2ed09ae2db", size = 114793702, upload-time = "2026-03-23T18:09:47.304Z" },
+ { url = "https://files.pythonhosted.org/packages/26/0d/8603382f61abd0db35841148ddc1ffd607bf3100b11c6e1dab6d2fc44e72/torch-2.11.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:01018087326984a33b64e04c8cb5c2795f9120e0d775ada1f6638840227b04d7", size = 80573442, upload-time = "2026-03-23T18:09:10.117Z" },
+ { url = "https://files.pythonhosted.org/packages/c7/86/7cd7c66cb9cec6be330fff36db5bd0eef386d80c031b581ec81be1d4b26c/torch-2.11.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:2bb3cc54bd0dea126b0060bb1ec9de0f9c7f7342d93d436646516b0330cd5be7", size = 419749385, upload-time = "2026-03-23T18:07:33.77Z" },
+ { url = "https://files.pythonhosted.org/packages/47/e8/b98ca2d39b2e0e4730c0ee52537e488e7008025bc77ca89552ff91021f7c/torch-2.11.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:4dc8b3809469b6c30b411bb8c4cad3828efd26236153d9beb6a3ec500f211a60", size = 530716756, upload-time = "2026-03-23T18:07:50.02Z" },
+ { url = "https://files.pythonhosted.org/packages/78/88/d4a4cda8362f8a30d1ed428564878c3cafb0d87971fbd3947d4c84552095/torch-2.11.0-cp314-cp314-win_amd64.whl", hash = "sha256:2b4e811728bd0cc58fb2b0948fe939a1ee2bf1422f6025be2fca4c7bd9d79718", size = 114552300, upload-time = "2026-03-23T18:09:05.617Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/46/4419098ed6d801750f26567b478fc185c3432e11e2cad712bc6b4c2ab0d0/torch-2.11.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:8245477871c3700d4370352ffec94b103cfcb737229445cf9946cddb7b2ca7cd", size = 80959460, upload-time = "2026-03-23T18:09:00.818Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/66/54a56a4a6ceaffb567231994a9745821d3af922a854ed33b0b3a278e0a99/torch-2.11.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:ab9a8482f475f9ba20e12db84b0e55e2f58784bdca43a854a6ccd3fd4b9f75e6", size = 419735835, upload-time = "2026-03-23T18:07:18.974Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/e7/0b6665f533aa9e337662dc190425abc0af1fe3234088f4454c52393ded61/torch-2.11.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:563ed3d25542d7e7bbc5b235ccfacfeb97fb470c7fee257eae599adb8005c8a2", size = 530613405, upload-time = "2026-03-23T18:08:07.014Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/bf/c8d12a2c86dbfd7f40fb2f56fbf5a505ccf2d9ce131eb559dfc7c51e1a04/torch-2.11.0-cp314-cp314t-win_amd64.whl", hash = "sha256:b2a43985ff5ef6ddd923bbcf99943e5f58059805787c5c9a2622bf05ca2965b0", size = 114792991, upload-time = "2026-03-23T18:08:19.216Z" },
]
[[package]]
@@ -5712,7 +6471,8 @@ dependencies = [
{ name = "docker" },
{ name = "docstring-parser" },
{ name = "filelock" },
- { name = "fsspec" },
+ { name = "fsspec", version = "2026.2.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.14' or sys_platform != 'win32' or extra == 'extra-13-megatron-core-dev' or extra == 'extra-13-megatron-core-lts'" },
+ { name = "fsspec", version = "2026.3.0", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-13-megatron-core-dev' and extra != 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
{ name = "importlib-metadata" },
{ name = "pyre-extensions" },
{ name = "pyyaml" },
@@ -5726,41 +6486,44 @@ wheels = [
[[package]]
name = "tqdm"
-version = "4.67.1"
+version = "4.67.3"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "colorama", marker = "sys_platform == 'win32'" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/a8/4b/29b4ef32e036bb34e4ab51796dd745cdba7ed47ad142a9f4a1eb8e0c744d/tqdm-4.67.1.tar.gz", hash = "sha256:f8aef9c52c08c13a65f30ea34f4e5aac3fd1a34959879d7e59e63027286627f2", size = 169737, upload-time = "2024-11-24T20:12:22.481Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/09/a9/6ba95a270c6f1fbcd8dac228323f2777d886cb206987444e4bce66338dd4/tqdm-4.67.3.tar.gz", hash = "sha256:7d825f03f89244ef73f1d4ce193cb1774a8179fd96f31d7e1dcde62092b960bb", size = 169598, upload-time = "2026-02-03T17:35:53.048Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/d0/30/dc54f88dd4a2b5dc8a0279bdd7270e735851848b762aeb1c1184ed1f6b14/tqdm-4.67.1-py3-none-any.whl", hash = "sha256:26445eca388f82e72884e0d580d5464cd801a3ea01e63e5601bdff9ba6a48de2", size = 78540, upload-time = "2024-11-24T20:12:19.698Z" },
+ { url = "https://files.pythonhosted.org/packages/16/e1/3079a9ff9b8e11b846c6ac5c8b5bfb7ff225eee721825310c91b3b50304f/tqdm-4.67.3-py3-none-any.whl", hash = "sha256:ee1e4c0e59148062281c49d80b25b67771a127c85fc9676d3be5f243206826bf", size = 78374, upload-time = "2026-02-03T17:35:50.982Z" },
]
[[package]]
name = "transformer-engine"
-version = "2.12.0+d9b7fc57"
-source = { git = "https://github.com/NVIDIA/TransformerEngine.git?rev=d9b7fc5770a88af06e2e9c2bd97b550614c3a69f#d9b7fc5770a88af06e2e9c2bd97b550614c3a69f" }
+version = "2.14.0+71bbefbf"
+source = { git = "https://github.com/NVIDIA/TransformerEngine.git?rev=71bbefbf153418f943640df0f7373625dc93fa46#71bbefbf153418f943640df0f7373625dc93fa46" }
dependencies = [
{ name = "einops" },
{ name = "importlib-metadata" },
{ name = "nvdlfw-inspect" },
- { name = "onnx" },
- { name = "onnxscript" },
- { name = "packaging" },
+ { name = "onnx", version = "1.19.0", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.13' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "onnx", version = "1.21.0", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.13' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "onnxscript", version = "0.5.0", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.13' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "onnxscript", version = "0.6.2", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.13' and extra == 'extra-13-megatron-core-dev') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" } },
{ name = "pydantic" },
{ name = "torch", marker = "sys_platform == 'never'" },
]
[[package]]
name = "transformers"
-version = "4.57.5"
+version = "4.57.6"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "filelock" },
{ name = "huggingface-hub" },
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "packaging" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
{ name = "pyyaml" },
{ name = "regex" },
{ name = "requests" },
@@ -5768,54 +6531,50 @@ dependencies = [
{ name = "tokenizers" },
{ name = "tqdm" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/35/3a/7c90ee739871495f1a5cb9bdb074b42fe69357d7ccc1a8818af858d8e63b/transformers-4.57.5.tar.gz", hash = "sha256:d631faea6bd32fc51962e482744afeaa70170c70e5e991cf8e355d7275631524", size = 10138171, upload-time = "2026-01-13T13:28:24.19Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/c4/35/67252acc1b929dc88b6602e8c4a982e64f31e733b804c14bc24b47da35e6/transformers-4.57.6.tar.gz", hash = "sha256:55e44126ece9dc0a291521b7e5492b572e6ef2766338a610b9ab5afbb70689d3", size = 10134912, upload-time = "2026-01-16T10:38:39.284Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/f8/de/4f95d22d9764659d2bd35065f383f3fe099699a9e6e89fa4728dbcd7244a/transformers-4.57.5-py3-none-any.whl", hash = "sha256:5a1e0deb989cd0b8f141b6d8c9b7c956fc029cd288d68844f57dc0acbaf2fe39", size = 11993481, upload-time = "2026-01-13T13:28:16.542Z" },
+ { url = "https://files.pythonhosted.org/packages/03/b8/e484ef633af3887baeeb4b6ad12743363af7cce68ae51e938e00aaa0529d/transformers-4.57.6-py3-none-any.whl", hash = "sha256:4c9e9de11333ddfe5114bc872c9f370509198acf0b87a832a0ab9458e2bd0550", size = 11993498, upload-time = "2026-01-16T10:38:31.289Z" },
]
[[package]]
name = "triton"
-version = "3.5.1"
+version = "3.6.0"
source = { registry = "https://pypi.org/simple" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/d9/2e/f95e673222afa2c7f0c687d8913e98fcf2589ef0b1405de76894e37fe18f/triton-3.5.1-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f63e34dcb32d7bd3a1d0195f60f30d2aee8b08a69a0424189b71017e23dfc3d2", size = 159821655, upload-time = "2025-11-11T17:51:44.09Z" },
- { url = "https://files.pythonhosted.org/packages/fd/6e/676ab5019b4dde8b9b7bab71245102fc02778ef3df48218b298686b9ffd6/triton-3.5.1-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5fc53d849f879911ea13f4a877243afc513187bc7ee92d1f2c0f1ba3169e3c94", size = 170320692, upload-time = "2025-11-11T17:40:46.074Z" },
- { url = "https://files.pythonhosted.org/packages/dc/dc/6ce44d055f2fc2403c4ec6b3cfd3a9b25f57b7d95efadccdea91497f8e81/triton-3.5.1-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:da47169e30a779bade679ce78df4810fca6d78a955843d2ddb11f226adc517dc", size = 159928005, upload-time = "2025-11-11T17:51:50.008Z" },
- { url = "https://files.pythonhosted.org/packages/b0/72/ec90c3519eaf168f22cb1757ad412f3a2add4782ad3a92861c9ad135d886/triton-3.5.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:61413522a48add32302353fdbaaf92daaaab06f6b5e3229940d21b5207f47579", size = 170425802, upload-time = "2025-11-11T17:40:53.209Z" },
- { url = "https://files.pythonhosted.org/packages/db/53/2bcc46879910991f09c063eea07627baef2bc62fe725302ba8f46a2c1ae5/triton-3.5.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:275a045b6ed670dd1bd005c3e6c2d61846c74c66f4512d6f33cc027b11de8fd4", size = 159940689, upload-time = "2025-11-11T17:51:55.938Z" },
- { url = "https://files.pythonhosted.org/packages/f2/50/9a8358d3ef58162c0a415d173cfb45b67de60176e1024f71fbc4d24c0b6d/triton-3.5.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d2c6b915a03888ab931a9fd3e55ba36785e1fe70cbea0b40c6ef93b20fc85232", size = 170470207, upload-time = "2025-11-11T17:41:00.253Z" },
- { url = "https://files.pythonhosted.org/packages/f1/ba/805684a992ee32d486b7948d36aed2f5e3c643fc63883bf8bdca1c3f3980/triton-3.5.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:56765ffe12c554cd560698398b8a268db1f616c120007bfd8829d27139abd24a", size = 159955460, upload-time = "2025-11-11T17:52:01.861Z" },
- { url = "https://files.pythonhosted.org/packages/27/46/8c3bbb5b0a19313f50edcaa363b599e5a1a5ac9683ead82b9b80fe497c8d/triton-3.5.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f3f4346b6ebbd4fad18773f5ba839114f4826037c9f2f34e0148894cd5dd3dba", size = 170470410, upload-time = "2025-11-11T17:41:06.319Z" },
- { url = "https://files.pythonhosted.org/packages/84/1e/7df59baef41931e21159371c481c31a517ff4c2517343b62503d0cd2be99/triton-3.5.1-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:02c770856f5e407d24d28ddc66e33cf026e6f4d360dcb8b2fabe6ea1fc758621", size = 160072799, upload-time = "2025-11-11T17:52:07.293Z" },
- { url = "https://files.pythonhosted.org/packages/37/92/e97fcc6b2c27cdb87ce5ee063d77f8f26f19f06916aa680464c8104ef0f6/triton-3.5.1-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0b4d2c70127fca6a23e247f9348b8adde979d2e7a20391bfbabaac6aebc7e6a8", size = 170579924, upload-time = "2025-11-11T17:41:12.455Z" },
- { url = "https://files.pythonhosted.org/packages/14/f9/0430e879c1e63a1016cb843261528fd3187c872c3a9539132efc39514753/triton-3.5.1-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f617aa7925f9ea9968ec2e1adaf93e87864ff51549c8f04ce658f29bbdb71e2d", size = 159956163, upload-time = "2025-11-11T17:52:12.999Z" },
- { url = "https://files.pythonhosted.org/packages/a4/e6/c595c35e5c50c4bc56a7bac96493dad321e9e29b953b526bbbe20f9911d0/triton-3.5.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d0637b1efb1db599a8e9dc960d53ab6e4637db7d4ab6630a0974705d77b14b60", size = 170480488, upload-time = "2025-11-11T17:41:18.222Z" },
- { url = "https://files.pythonhosted.org/packages/41/1e/63d367c576c75919e268e4fbc33c1cb33b6dc12bb85e8bfe531c2a8bd5d3/triton-3.5.1-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8932391d7f93698dfe5bc9bead77c47a24f97329e9f20c10786bb230a9083f56", size = 160073620, upload-time = "2025-11-11T17:52:18.403Z" },
- { url = "https://files.pythonhosted.org/packages/16/b5/b0d3d8b901b6a04ca38df5e24c27e53afb15b93624d7fd7d658c7cd9352a/triton-3.5.1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bac7f7d959ad0f48c0e97d6643a1cc0fd5786fe61cb1f83b537c6b2d54776478", size = 170582192, upload-time = "2025-11-11T17:41:23.963Z" },
+ { url = "https://files.pythonhosted.org/packages/17/5d/08201db32823bdf77a0e2b9039540080b2e5c23a20706ddba942924ebcd6/triton-3.6.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:374f52c11a711fd062b4bfbb201fd9ac0a5febd28a96fb41b4a0f51dde3157f4", size = 176128243, upload-time = "2026-01-20T16:16:07.857Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/a8/cdf8b3e4c98132f965f88c2313a4b493266832ad47fb52f23d14d4f86bb5/triton-3.6.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:74caf5e34b66d9f3a429af689c1c7128daba1d8208df60e81106b115c00d6fca", size = 188266850, upload-time = "2026-01-20T16:00:43.041Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/12/34d71b350e89a204c2c7777a9bba0dcf2f19a5bfdd70b57c4dbc5ffd7154/triton-3.6.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:448e02fe6dc898e9e5aa89cf0ee5c371e99df5aa5e8ad976a80b93334f3494fd", size = 176133521, upload-time = "2026-01-20T16:16:13.321Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/0b/37d991d8c130ce81a8728ae3c25b6e60935838e9be1b58791f5997b24a54/triton-3.6.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:10c7f76c6e72d2ef08df639e3d0d30729112f47a56b0c81672edc05ee5116ac9", size = 188289450, upload-time = "2026-01-20T16:00:49.136Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/4e/41b0c8033b503fd3cfcd12392cdd256945026a91ff02452bef40ec34bee7/triton-3.6.0-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1722e172d34e32abc3eb7711d0025bb69d7959ebea84e3b7f7a341cd7ed694d6", size = 176276087, upload-time = "2026-01-20T16:16:18.989Z" },
+ { url = "https://files.pythonhosted.org/packages/35/f8/9c66bfc55361ec6d0e4040a0337fb5924ceb23de4648b8a81ae9d33b2b38/triton-3.6.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d002e07d7180fd65e622134fbd980c9a3d4211fb85224b56a0a0efbd422ab72f", size = 188400296, upload-time = "2026-01-20T16:00:56.042Z" },
+ { url = "https://files.pythonhosted.org/packages/49/55/5ecf0dcaa0f2fbbd4420f7ef227ee3cb172e91e5fede9d0ecaddc43363b4/triton-3.6.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ef5523241e7d1abca00f1d240949eebdd7c673b005edbbce0aca95b8191f1d43", size = 176138577, upload-time = "2026-01-20T16:16:25.426Z" },
+ { url = "https://files.pythonhosted.org/packages/df/3d/9e7eee57b37c80cec63322c0231bb6da3cfe535a91d7a4d64896fcb89357/triton-3.6.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a17a5d5985f0ac494ed8a8e54568f092f7057ef60e1b0fa09d3fd1512064e803", size = 188273063, upload-time = "2026-01-20T16:01:07.278Z" },
+ { url = "https://files.pythonhosted.org/packages/48/db/56ee649cab5eaff4757541325aca81f52d02d4a7cd3506776cad2451e060/triton-3.6.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0b3a97e8ed304dfa9bd23bb41ca04cdf6b2e617d5e782a8653d616037a5d537d", size = 176274804, upload-time = "2026-01-20T16:16:31.528Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/56/6113c23ff46c00aae423333eb58b3e60bdfe9179d542781955a5e1514cb3/triton-3.6.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:46bd1c1af4b6704e554cad2eeb3b0a6513a980d470ccfa63189737340c7746a7", size = 188397994, upload-time = "2026-01-20T16:01:14.236Z" },
]
[[package]]
name = "trove-classifiers"
-version = "2026.1.12.15"
+version = "2026.1.14.14"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/4c/3b/600fa0e35b353a66d1134a233d67feee4d934b7878aef10a21f39b17c6ab/trove_classifiers-2026.1.12.15.tar.gz", hash = "sha256:832a7e89ccc43b64b89f8f9d9150c069ebcd17d2dc68279bc00bb53f2a9ae112", size = 16978, upload-time = "2026-01-12T15:15:10.479Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/d8/43/7935f8ea93fcb6680bc10a6fdbf534075c198eeead59150dd5ed68449642/trove_classifiers-2026.1.14.14.tar.gz", hash = "sha256:00492545a1402b09d4858605ba190ea33243d361e2b01c9c296ce06b5c3325f3", size = 16997, upload-time = "2026-01-14T14:54:50.526Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/7e/c3/4b74a82b58dbfb0a4dafa8149fc60f20ce5228be5d133ca0c3f2e90f6e7d/trove_classifiers-2026.1.12.15-py3-none-any.whl", hash = "sha256:8832dfbc226fc4df986666b9cb3a018818b1498aeb79f5f66a31a918b47a98f1", size = 14192, upload-time = "2026-01-12T15:15:09.413Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/4a/2e5583e544bc437d5e8e54b47db87430df9031b29b48d17f26d129fa60c0/trove_classifiers-2026.1.14.14-py3-none-any.whl", hash = "sha256:1f9553927f18d0513d8e5ff80ab8980b8202ce37ecae0e3274ed2ef11880e74d", size = 14197, upload-time = "2026-01-14T14:54:49.067Z" },
]
[[package]]
name = "typer"
-version = "0.21.1"
+version = "0.24.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
+ { name = "annotated-doc" },
{ name = "click" },
{ name = "rich" },
{ name = "shellingham" },
- { name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/36/bf/8825b5929afd84d0dabd606c67cd57b8388cb3ec385f7ef19c5cc2202069/typer-0.21.1.tar.gz", hash = "sha256:ea835607cd752343b6b2b7ce676893e5a0324082268b48f27aa058bdb7d2145d", size = 110371, upload-time = "2026-01-06T11:21:10.989Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/f5/24/cb09efec5cc954f7f9b930bf8279447d24618bb6758d4f6adf2574c41780/typer-0.24.1.tar.gz", hash = "sha256:e39b4732d65fbdcde189ae76cf7cd48aeae72919dea1fdfc16593be016256b45", size = 118613, upload-time = "2026-02-21T16:54:40.609Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/a0/1d/d9257dd49ff2ca23ea5f132edf1281a0c4f9de8a762b9ae399b670a59235/typer-0.21.1-py3-none-any.whl", hash = "sha256:7985e89081c636b88d172c2ee0cfe33c253160994d47bdfdc302defd7d1f1d01", size = 47381, upload-time = "2026-01-06T11:21:09.824Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/91/48db081e7a63bb37284f9fbcefda7c44c277b18b0e13fbc36ea2335b71e6/typer-0.24.1-py3-none-any.whl", hash = "sha256:112c1f0ce578bfb4cab9ffdabc68f031416ebcc216536611ba21f04e9aa84c9e", size = 56085, upload-time = "2026-02-21T16:54:41.616Z" },
]
[[package]]
@@ -5870,28 +6629,65 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/33/cf/8435d5a7159e2a9c83a95896ed596f68cf798005fe107cc655b5c5c14704/urllib3-1.26.20-py2.py3-none-any.whl", hash = "sha256:0ed14ccfbf1c30a9072c7ca157e4319b70d65f623e91e7b32fadb2853431016e", size = 144225, upload-time = "2024-08-29T15:43:08.921Z" },
]
+[[package]]
+name = "uuid-utils"
+version = "0.14.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/7b/d1/38a573f0c631c062cf42fa1f5d021d4dd3c31fb23e4376e4b56b0c9fbbed/uuid_utils-0.14.1.tar.gz", hash = "sha256:9bfc95f64af80ccf129c604fb6b8ca66c6f256451e32bc4570f760e4309c9b69", size = 22195, upload-time = "2026-02-20T22:50:38.833Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/43/b7/add4363039a34506a58457d96d4aa2126061df3a143eb4d042aedd6a2e76/uuid_utils-0.14.1-cp39-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:93a3b5dc798a54a1feb693f2d1cb4cf08258c32ff05ae4929b5f0a2ca624a4f0", size = 604679, upload-time = "2026-02-20T22:50:27.469Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/84/d1d0bef50d9e66d31b2019997c741b42274d53dde2e001b7a83e9511c339/uuid_utils-0.14.1-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:ccd65a4b8e83af23eae5e56d88034b2fe7264f465d3e830845f10d1591b81741", size = 309346, upload-time = "2026-02-20T22:50:31.857Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/ed/b6d6fd52a6636d7c3eddf97d68da50910bf17cd5ac221992506fb56cf12e/uuid_utils-0.14.1-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b56b0cacd81583834820588378e432b0696186683b813058b707aedc1e16c4b1", size = 344714, upload-time = "2026-02-20T22:50:42.642Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/a7/a19a1719fb626fe0b31882db36056d44fe904dc0cf15b06fdf56b2679cf7/uuid_utils-0.14.1-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:bb3cf14de789097320a3c56bfdfdd51b1225d11d67298afbedee7e84e3837c96", size = 350914, upload-time = "2026-02-20T22:50:36.487Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/fc/f6690e667fdc3bb1a73f57951f97497771c56fe23e3d302d7404be394d4f/uuid_utils-0.14.1-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:60e0854a90d67f4b0cc6e54773deb8be618f4c9bad98d3326f081423b5d14fae", size = 482609, upload-time = "2026-02-20T22:50:37.511Z" },
+ { url = "https://files.pythonhosted.org/packages/54/6e/dcd3fa031320921a12ec7b4672dea3bd1dd90ddffa363a91831ba834d559/uuid_utils-0.14.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ce6743ba194de3910b5feb1a62590cd2587e33a73ab6af8a01b642ceb5055862", size = 345699, upload-time = "2026-02-20T22:50:46.87Z" },
+ { url = "https://files.pythonhosted.org/packages/04/28/e5220204b58b44ac0047226a9d016a113fde039280cc8732d9e6da43b39f/uuid_utils-0.14.1-cp39-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:043fb58fde6cf1620a6c066382f04f87a8e74feb0f95a585e4ed46f5d44af57b", size = 372205, upload-time = "2026-02-20T22:50:28.438Z" },
+ { url = "https://files.pythonhosted.org/packages/c7/d9/3d2eb98af94b8dfffc82b6a33b4dfc87b0a5de2c68a28f6dde0db1f8681b/uuid_utils-0.14.1-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:c915d53f22945e55fe0d3d3b0b87fd965a57f5fd15666fd92d6593a73b1dd297", size = 521836, upload-time = "2026-02-20T22:50:23.057Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/15/0eb106cc6fe182f7577bc0ab6e2f0a40be247f35c5e297dbf7bbc460bd02/uuid_utils-0.14.1-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:0972488e3f9b449e83f006ead5a0e0a33ad4a13e4462e865b7c286ab7d7566a3", size = 625260, upload-time = "2026-02-20T22:50:25.949Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/17/f539507091334b109e7496830af2f093d9fc8082411eafd3ece58af1f8ba/uuid_utils-0.14.1-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:1c238812ae0c8ffe77d8d447a32c6dfd058ea4631246b08b5a71df586ff08531", size = 587824, upload-time = "2026-02-20T22:50:35.225Z" },
+ { url = "https://files.pythonhosted.org/packages/2e/c2/d37a7b2e41f153519367d4db01f0526e0d4b06f1a4a87f1c5dfca5d70a8b/uuid_utils-0.14.1-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:bec8f8ef627af86abf8298e7ec50926627e29b34fa907fcfbedb45aaa72bca43", size = 551407, upload-time = "2026-02-20T22:50:44.915Z" },
+ { url = "https://files.pythonhosted.org/packages/65/36/2d24b2cbe78547c6532da33fb8613debd3126eccc33a6374ab788f5e46e9/uuid_utils-0.14.1-cp39-abi3-win32.whl", hash = "sha256:b54d6aa6252d96bac1fdbc80d26ba71bad9f220b2724d692ad2f2310c22ef523", size = 183476, upload-time = "2026-02-20T22:50:32.745Z" },
+ { url = "https://files.pythonhosted.org/packages/83/92/2d7e90df8b1a69ec4cff33243ce02b7a62f926ef9e2f0eca5a026889cd73/uuid_utils-0.14.1-cp39-abi3-win_amd64.whl", hash = "sha256:fc27638c2ce267a0ce3e06828aff786f91367f093c80625ee21dad0208e0f5ba", size = 187147, upload-time = "2026-02-20T22:50:45.807Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/26/529f4beee17e5248e37e0bc17a2761d34c0fa3b1e5729c88adb2065bae6e/uuid_utils-0.14.1-cp39-abi3-win_arm64.whl", hash = "sha256:b04cb49b42afbc4ff8dbc60cf054930afc479d6f4dd7f1ec3bbe5dbfdde06b7a", size = 188132, upload-time = "2026-02-20T22:50:41.718Z" },
+]
+
[[package]]
name = "uvicorn"
-version = "0.40.0"
+version = "0.42.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "click" },
{ name = "h11" },
- { name = "typing-extensions", marker = "python_full_version < '3.11' or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/c3/d1/8f3c683c9561a4e6689dd3b1d345c815f10f86acd044ee1fb9a4dcd0b8c5/uvicorn-0.40.0.tar.gz", hash = "sha256:839676675e87e73694518b5574fd0f24c9d97b46bea16df7b8c05ea1a51071ea", size = 81761, upload-time = "2025-12-21T14:16:22.45Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/e3/ad/4a96c425be6fb67e0621e62d86c402b4a17ab2be7f7c055d9bd2f638b9e2/uvicorn-0.42.0.tar.gz", hash = "sha256:9b1f190ce15a2dd22e7758651d9b6d12df09a13d51ba5bf4fc33c383a48e1775", size = 85393, upload-time = "2026-03-16T06:19:50.077Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0a/89/f8827ccff89c1586027a105e5630ff6139a64da2515e24dafe860bd9ae4d/uvicorn-0.42.0-py3-none-any.whl", hash = "sha256:96c30f5c7abe6f74ae8900a70e92b85ad6613b745d4879eb9b16ccad15645359", size = 68830, upload-time = "2026-03-16T06:19:48.325Z" },
+]
+
+[[package]]
+name = "virtualenv"
+version = "21.2.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "distlib" },
+ { name = "filelock" },
+ { name = "platformdirs" },
+ { name = "python-discovery" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/aa/92/58199fe10049f9703c2666e809c4f686c54ef0a68b0f6afccf518c0b1eb9/virtualenv-21.2.0.tar.gz", hash = "sha256:1720dc3a62ef5b443092e3f499228599045d7fea4c79199770499df8becf9098", size = 5840618, upload-time = "2026-03-09T17:24:38.013Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/3d/d8/2083a1daa7439a66f3a48589a57d576aa117726762618f6bb09fe3798796/uvicorn-0.40.0-py3-none-any.whl", hash = "sha256:c6c8f55bc8bf13eb6fa9ff87ad62308bbbc33d0b67f84293151efe87e0d5f2ee", size = 68502, upload-time = "2025-12-21T14:16:21.041Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/59/7d02447a55b2e55755011a647479041bc92a82e143f96a8195cb33bd0a1c/virtualenv-21.2.0-py3-none-any.whl", hash = "sha256:1bd755b504931164a5a496d217c014d098426cddc79363ad66ac78125f9d908f", size = 5825084, upload-time = "2026-03-09T17:24:35.378Z" },
]
[[package]]
name = "wandb"
-version = "0.24.0"
+version = "0.25.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "click" },
{ name = "gitpython" },
- { name = "packaging" },
+ { name = "packaging", version = "25.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "packaging", version = "26.0", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts' or extra != 'extra-13-megatron-core-dev'" },
{ name = "platformdirs" },
{ name = "protobuf" },
{ name = "pydantic" },
@@ -5900,17 +6696,17 @@ dependencies = [
{ name = "sentry-sdk" },
{ name = "typing-extensions" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/27/7e/aad6e943012ea4d88f3a037f1a5a7c6898263c60fbef8c9cdb95a8ff9fd9/wandb-0.24.0.tar.gz", hash = "sha256:4715a243b3d460b6434b9562e935dfd9dfdf5d6e428cfb4c3e7ce4fd44460ab3", size = 44197947, upload-time = "2026-01-13T22:59:59.767Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/60/bb/eb579bf9abac70934a014a9d4e45346aab307994f3021d201bebe5fa25ec/wandb-0.25.1.tar.gz", hash = "sha256:b2a95cd777ecbe7499599a43158834983448a0048329bc7210ef46ca18d21994", size = 43983308, upload-time = "2026-03-10T23:51:44.227Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/5f/8a/efec186dcc5dcf3c806040e3f33e58997878b2d30b87aa02b26f046858b6/wandb-0.24.0-py3-none-macosx_12_0_arm64.whl", hash = "sha256:aa9777398ff4b0f04c41359f7d1b95b5d656cb12c37c63903666799212e50299", size = 21464901, upload-time = "2026-01-13T22:59:31.86Z" },
- { url = "https://files.pythonhosted.org/packages/ed/84/fadf0d5f1d86c3ba662d2b33a15d2b1f08ff1e4e196c77e455f028b0fda2/wandb-0.24.0-py3-none-macosx_12_0_x86_64.whl", hash = "sha256:0423fbd58c3926949724feae8aab89d20c68846f9f4f596b80f9ffe1fc298130", size = 22697817, upload-time = "2026-01-13T22:59:35.267Z" },
- { url = "https://files.pythonhosted.org/packages/6e/5f/e3124e68d02b30c62856175ce714e07904730be06eecb00f66bb1a59aacf/wandb-0.24.0-py3-none-manylinux_2_28_aarch64.whl", hash = "sha256:2b25fc0c123daac97ed32912ac55642c65013cc6e3a898e88ca2d917fc8eadc0", size = 21118798, upload-time = "2026-01-13T22:59:38.453Z" },
- { url = "https://files.pythonhosted.org/packages/22/a1/8d68a914c030e897c306c876d47c73aa5d9ca72be608971290d3a5749570/wandb-0.24.0-py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:9485344b4667944b5b77294185bae8469cfa4074869bec0e74f54f8492234cc2", size = 22849954, upload-time = "2026-01-13T22:59:41.265Z" },
- { url = "https://files.pythonhosted.org/packages/e9/f8/3e68841a4282a4fb6a8935534e6064acc6c9708e8fb76953ec73bbc72a5e/wandb-0.24.0-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:51b2b9a9d7d6b35640f12a46a48814fd4516807ad44f586b819ed6560f8de1fd", size = 21160339, upload-time = "2026-01-13T22:59:43.967Z" },
- { url = "https://files.pythonhosted.org/packages/16/e5/d851868ce5b4b437a7cc90405979cd83809790e4e2a2f1e454f63f116e52/wandb-0.24.0-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:11f7e7841f31eff82c82a677988889ad3aa684c6de61ff82145333b5214ec860", size = 22936978, upload-time = "2026-01-13T22:59:46.911Z" },
- { url = "https://files.pythonhosted.org/packages/d2/34/43b7f18870051047ce6fe18e7eb24ba7ebdc71663a8f1c58e31e855eb8ac/wandb-0.24.0-py3-none-win32.whl", hash = "sha256:42af348998b00d4309ae790c5374040ac6cc353ab21567f4e29c98c9376dee8e", size = 22118243, upload-time = "2026-01-13T22:59:49.555Z" },
- { url = "https://files.pythonhosted.org/packages/a1/92/909c81173cf1399111f57f9ca5399a8f165607b024e406e080178c878f70/wandb-0.24.0-py3-none-win_amd64.whl", hash = "sha256:32604eddcd362e1ed4a2e2ce5f3a239369c4a193af223f3e66603481ac91f336", size = 22118246, upload-time = "2026-01-13T22:59:52.126Z" },
- { url = "https://files.pythonhosted.org/packages/87/85/a845aefd9c2285f98261fa6ffa0a14466366c1ac106d35bc84b654c0ad7f/wandb-0.24.0-py3-none-win_arm64.whl", hash = "sha256:e0f2367552abfca21b0f3a03405fbf48f1e14de9846e70f73c6af5da57afd8ef", size = 20077678, upload-time = "2026-01-13T22:59:56.112Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/d8/873553b6818499d1b1de314067d528b892897baf0dc81fedc0e845abc2dd/wandb-0.25.1-py3-none-macosx_12_0_arm64.whl", hash = "sha256:9bb0679a3e2dcd96db9d9b6d3e17d046241d8d122974b24facb85cc93309a8c9", size = 23615900, upload-time = "2026-03-10T23:51:06.278Z" },
+ { url = "https://files.pythonhosted.org/packages/71/ea/b131f319aaa5d0bf7572b6bfcff3dd89e1cf92b17eee443bbab71d12d74c/wandb-0.25.1-py3-none-macosx_12_0_x86_64.whl", hash = "sha256:0fb13ed18914027523e7b4fc20380c520e0d10da0ee452f924a13f84509fbe12", size = 25576144, upload-time = "2026-03-10T23:51:11.527Z" },
+ { url = "https://files.pythonhosted.org/packages/70/5f/81508581f0bb77b0495665c1c78e77606a48e66e855ca71ba7c8ae29efa4/wandb-0.25.1-py3-none-manylinux_2_28_aarch64.whl", hash = "sha256:cc4521eb5223429ddab5e8eee9b42fdf4caabdf0bc4e0e809042720e5fbef0ed", size = 23070425, upload-time = "2026-03-10T23:51:15.71Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/c7/445155ef010e2e35d190797d7c36ff441e062a5b566a6da4778e22233395/wandb-0.25.1-py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:e73b4c55b947edae349232d5845204d30fac88e18eb4ad1d4b96bf7cf898405a", size = 25628142, upload-time = "2026-03-10T23:51:19.326Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/63/f5c55ee00cf481ef1ccd3c385a0585ad52e7840d08419d4f82ddbeeea959/wandb-0.25.1-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:22b84065aa398e1624d2e5ad79e08bc4d2af41a6db61697b03b3aaba332977c6", size = 23123172, upload-time = "2026-03-10T23:51:23.418Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/d9/19eb7974c0e9253bcbaee655222c0f0e1a52e63e9479ee711b4208f8ac31/wandb-0.25.1-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:005c4c6b5126ef8f4b4110e5372d950918b00637d6dc4b615ad17445f9739478", size = 25714479, upload-time = "2026-03-10T23:51:27.421Z" },
+ { url = "https://files.pythonhosted.org/packages/11/19/466c1d03323a4a0ed7d4036a59b18d6b6f67cb5032e444205927e226b18d/wandb-0.25.1-py3-none-win32.whl", hash = "sha256:8f2d04f16b88d65bfba9d79fb945f6c64e2686215469a841936e0972be8ec6a5", size = 24967338, upload-time = "2026-03-10T23:51:31.833Z" },
+ { url = "https://files.pythonhosted.org/packages/89/22/680d34c1587f3a979c701b66d71aa7c42b4ef2fdf0774f67034e618e834e/wandb-0.25.1-py3-none-win_amd64.whl", hash = "sha256:62db5166de14456156d7a85953a58733a631228e6d4248a753605f75f75fb845", size = 24967343, upload-time = "2026-03-10T23:51:36.026Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/e8/76836b75d401ff5912aaf513176e64557ceaec4c4946bfd38a698ff84d48/wandb-0.25.1-py3-none-win_arm64.whl", hash = "sha256:cc7c34b70cf4b7be4d395541e82e325fd9d2be978d62c9ec01f1a7141523b6bb", size = 22080774, upload-time = "2026-03-10T23:51:40.196Z" },
]
[[package]]
@@ -5922,31 +6718,6 @@ dependencies = [
]
sdist = { url = "https://files.pythonhosted.org/packages/c2/c9/8869df9b2a2d6c59d79220a4db37679e74f807c559ffe5265e08b227a210/watchfiles-1.1.1.tar.gz", hash = "sha256:a173cb5c16c4f40ab19cecf48a534c409f7ea983ab8fed0741304a1c0a31b3f2", size = 94440, upload-time = "2025-10-14T15:06:21.08Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/a7/1a/206e8cf2dd86fddf939165a57b4df61607a1e0add2785f170a3f616b7d9f/watchfiles-1.1.1-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:eef58232d32daf2ac67f42dea51a2c80f0d03379075d44a587051e63cc2e368c", size = 407318, upload-time = "2025-10-14T15:04:18.753Z" },
- { url = "https://files.pythonhosted.org/packages/b3/0f/abaf5262b9c496b5dad4ed3c0e799cbecb1f8ea512ecb6ddd46646a9fca3/watchfiles-1.1.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:03fa0f5237118a0c5e496185cafa92878568b652a2e9a9382a5151b1a0380a43", size = 394478, upload-time = "2025-10-14T15:04:20.297Z" },
- { url = "https://files.pythonhosted.org/packages/b1/04/9cc0ba88697b34b755371f5ace8d3a4d9a15719c07bdc7bd13d7d8c6a341/watchfiles-1.1.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8ca65483439f9c791897f7db49202301deb6e15fe9f8fe2fed555bf986d10c31", size = 449894, upload-time = "2025-10-14T15:04:21.527Z" },
- { url = "https://files.pythonhosted.org/packages/d2/9c/eda4615863cd8621e89aed4df680d8c3ec3da6a4cf1da113c17decd87c7f/watchfiles-1.1.1-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f0ab1c1af0cb38e3f598244c17919fb1a84d1629cc08355b0074b6d7f53138ac", size = 459065, upload-time = "2025-10-14T15:04:22.795Z" },
- { url = "https://files.pythonhosted.org/packages/84/13/f28b3f340157d03cbc8197629bc109d1098764abe1e60874622a0be5c112/watchfiles-1.1.1-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3bc570d6c01c206c46deb6e935a260be44f186a2f05179f52f7fcd2be086a94d", size = 488377, upload-time = "2025-10-14T15:04:24.138Z" },
- { url = "https://files.pythonhosted.org/packages/86/93/cfa597fa9389e122488f7ffdbd6db505b3b915ca7435ecd7542e855898c2/watchfiles-1.1.1-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e84087b432b6ac94778de547e08611266f1f8ffad28c0ee4c82e028b0fc5966d", size = 595837, upload-time = "2025-10-14T15:04:25.057Z" },
- { url = "https://files.pythonhosted.org/packages/57/1e/68c1ed5652b48d89fc24d6af905d88ee4f82fa8bc491e2666004e307ded1/watchfiles-1.1.1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:620bae625f4cb18427b1bb1a2d9426dc0dd5a5ba74c7c2cdb9de405f7b129863", size = 473456, upload-time = "2025-10-14T15:04:26.497Z" },
- { url = "https://files.pythonhosted.org/packages/d5/dc/1a680b7458ffa3b14bb64878112aefc8f2e4f73c5af763cbf0bd43100658/watchfiles-1.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:544364b2b51a9b0c7000a4b4b02f90e9423d97fbbf7e06689236443ebcad81ab", size = 455614, upload-time = "2025-10-14T15:04:27.539Z" },
- { url = "https://files.pythonhosted.org/packages/61/a5/3d782a666512e01eaa6541a72ebac1d3aae191ff4a31274a66b8dd85760c/watchfiles-1.1.1-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:bbe1ef33d45bc71cf21364df962af171f96ecaeca06bd9e3d0b583efb12aec82", size = 630690, upload-time = "2025-10-14T15:04:28.495Z" },
- { url = "https://files.pythonhosted.org/packages/9b/73/bb5f38590e34687b2a9c47a244aa4dd50c56a825969c92c9c5fc7387cea1/watchfiles-1.1.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:1a0bb430adb19ef49389e1ad368450193a90038b5b752f4ac089ec6942c4dff4", size = 622459, upload-time = "2025-10-14T15:04:29.491Z" },
- { url = "https://files.pythonhosted.org/packages/f1/ac/c9bb0ec696e07a20bd58af5399aeadaef195fb2c73d26baf55180fe4a942/watchfiles-1.1.1-cp310-cp310-win32.whl", hash = "sha256:3f6d37644155fb5beca5378feb8c1708d5783145f2a0f1c4d5a061a210254844", size = 272663, upload-time = "2025-10-14T15:04:30.435Z" },
- { url = "https://files.pythonhosted.org/packages/11/a0/a60c5a7c2ec59fa062d9a9c61d02e3b6abd94d32aac2d8344c4bdd033326/watchfiles-1.1.1-cp310-cp310-win_amd64.whl", hash = "sha256:a36d8efe0f290835fd0f33da35042a1bb5dc0e83cbc092dcf69bce442579e88e", size = 287453, upload-time = "2025-10-14T15:04:31.53Z" },
- { url = "https://files.pythonhosted.org/packages/1f/f8/2c5f479fb531ce2f0564eda479faecf253d886b1ab3630a39b7bf7362d46/watchfiles-1.1.1-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:f57b396167a2565a4e8b5e56a5a1c537571733992b226f4f1197d79e94cf0ae5", size = 406529, upload-time = "2025-10-14T15:04:32.899Z" },
- { url = "https://files.pythonhosted.org/packages/fe/cd/f515660b1f32f65df671ddf6f85bfaca621aee177712874dc30a97397977/watchfiles-1.1.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:421e29339983e1bebc281fab40d812742268ad057db4aee8c4d2bce0af43b741", size = 394384, upload-time = "2025-10-14T15:04:33.761Z" },
- { url = "https://files.pythonhosted.org/packages/7b/c3/28b7dc99733eab43fca2d10f55c86e03bd6ab11ca31b802abac26b23d161/watchfiles-1.1.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6e43d39a741e972bab5d8100b5cdacf69db64e34eb19b6e9af162bccf63c5cc6", size = 448789, upload-time = "2025-10-14T15:04:34.679Z" },
- { url = "https://files.pythonhosted.org/packages/4a/24/33e71113b320030011c8e4316ccca04194bf0cbbaeee207f00cbc7d6b9f5/watchfiles-1.1.1-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f537afb3276d12814082a2e9b242bdcf416c2e8fd9f799a737990a1dbe906e5b", size = 460521, upload-time = "2025-10-14T15:04:35.963Z" },
- { url = "https://files.pythonhosted.org/packages/f4/c3/3c9a55f255aa57b91579ae9e98c88704955fa9dac3e5614fb378291155df/watchfiles-1.1.1-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b2cd9e04277e756a2e2d2543d65d1e2166d6fd4c9b183f8808634fda23f17b14", size = 488722, upload-time = "2025-10-14T15:04:37.091Z" },
- { url = "https://files.pythonhosted.org/packages/49/36/506447b73eb46c120169dc1717fe2eff07c234bb3232a7200b5f5bd816e9/watchfiles-1.1.1-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5f3f58818dc0b07f7d9aa7fe9eb1037aecb9700e63e1f6acfed13e9fef648f5d", size = 596088, upload-time = "2025-10-14T15:04:38.39Z" },
- { url = "https://files.pythonhosted.org/packages/82/ab/5f39e752a9838ec4d52e9b87c1e80f1ee3ccdbe92e183c15b6577ab9de16/watchfiles-1.1.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9bb9f66367023ae783551042d31b1d7fd422e8289eedd91f26754a66f44d5cff", size = 472923, upload-time = "2025-10-14T15:04:39.666Z" },
- { url = "https://files.pythonhosted.org/packages/af/b9/a419292f05e302dea372fa7e6fda5178a92998411f8581b9830d28fb9edb/watchfiles-1.1.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:aebfd0861a83e6c3d1110b78ad54704486555246e542be3e2bb94195eabb2606", size = 456080, upload-time = "2025-10-14T15:04:40.643Z" },
- { url = "https://files.pythonhosted.org/packages/b0/c3/d5932fd62bde1a30c36e10c409dc5d54506726f08cb3e1d8d0ba5e2bc8db/watchfiles-1.1.1-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:5fac835b4ab3c6487b5dbad78c4b3724e26bcc468e886f8ba8cc4306f68f6701", size = 629432, upload-time = "2025-10-14T15:04:41.789Z" },
- { url = "https://files.pythonhosted.org/packages/f7/77/16bddd9779fafb795f1a94319dc965209c5641db5bf1edbbccace6d1b3c0/watchfiles-1.1.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:399600947b170270e80134ac854e21b3ccdefa11a9529a3decc1327088180f10", size = 623046, upload-time = "2025-10-14T15:04:42.718Z" },
- { url = "https://files.pythonhosted.org/packages/46/ef/f2ecb9a0f342b4bfad13a2787155c6ee7ce792140eac63a34676a2feeef2/watchfiles-1.1.1-cp311-cp311-win32.whl", hash = "sha256:de6da501c883f58ad50db3a32ad397b09ad29865b5f26f64c24d3e3281685849", size = 271473, upload-time = "2025-10-14T15:04:43.624Z" },
- { url = "https://files.pythonhosted.org/packages/94/bc/f42d71125f19731ea435c3948cad148d31a64fccde3867e5ba4edee901f9/watchfiles-1.1.1-cp311-cp311-win_amd64.whl", hash = "sha256:35c53bd62a0b885bf653ebf6b700d1bf05debb78ad9292cf2a942b23513dc4c4", size = 287598, upload-time = "2025-10-14T15:04:44.516Z" },
- { url = "https://files.pythonhosted.org/packages/57/c9/a30f897351f95bbbfb6abcadafbaca711ce1162f4db95fc908c98a9165f3/watchfiles-1.1.1-cp311-cp311-win_arm64.whl", hash = "sha256:57ca5281a8b5e27593cb7d82c2ac927ad88a96ed406aa446f6344e4328208e9e", size = 277210, upload-time = "2025-10-14T15:04:45.883Z" },
{ url = "https://files.pythonhosted.org/packages/74/d5/f039e7e3c639d9b1d09b07ea412a6806d38123f0508e5f9b48a87b0a76cc/watchfiles-1.1.1-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:8c89f9f2f740a6b7dcc753140dd5e1ab9215966f7a3530d0c0705c83b401bd7d", size = 404745, upload-time = "2025-10-14T15:04:46.731Z" },
{ url = "https://files.pythonhosted.org/packages/a5/96/a881a13aa1349827490dab2d363c8039527060cfcc2c92cc6d13d1b1049e/watchfiles-1.1.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:bd404be08018c37350f0d6e34676bd1e2889990117a2b90070b3007f172d0610", size = 391769, upload-time = "2025-10-14T15:04:48.003Z" },
{ url = "https://files.pythonhosted.org/packages/4b/5b/d3b460364aeb8da471c1989238ea0e56bec24b6042a68046adf3d9ddb01c/watchfiles-1.1.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8526e8f916bb5b9a0a777c8317c23ce65de259422bba5b31325a6fa6029d33af", size = 449374, upload-time = "2025-10-14T15:04:49.179Z" },
@@ -6006,14 +6777,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/4f/55/2af26693fd15165c4ff7857e38330e1b61ab8c37d15dc79118cdba115b7a/watchfiles-1.1.1-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8c91ed27800188c2ae96d16e3149f199d62f86c7af5f5f4d2c61a3ed8cd3666c", size = 455072, upload-time = "2025-10-14T15:05:48.928Z" },
{ url = "https://files.pythonhosted.org/packages/66/1d/d0d200b10c9311ec25d2273f8aad8c3ef7cc7ea11808022501811208a750/watchfiles-1.1.1-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:311ff15a0bae3714ffb603e6ba6dbfba4065ab60865d15a6ec544133bdb21099", size = 629104, upload-time = "2025-10-14T15:05:49.908Z" },
{ url = "https://files.pythonhosted.org/packages/e3/bd/fa9bb053192491b3867ba07d2343d9f2252e00811567d30ae8d0f78136fe/watchfiles-1.1.1-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:a916a2932da8f8ab582f242c065f5c81bed3462849ca79ee357dd9551b0e9b01", size = 622112, upload-time = "2025-10-14T15:05:50.941Z" },
- { url = "https://files.pythonhosted.org/packages/ba/4c/a888c91e2e326872fa4705095d64acd8aa2fb9c1f7b9bd0588f33850516c/watchfiles-1.1.1-pp310-pypy310_pp73-macosx_10_12_x86_64.whl", hash = "sha256:17ef139237dfced9da49fb7f2232c86ca9421f666d78c264c7ffca6601d154c3", size = 409611, upload-time = "2025-10-14T15:06:05.809Z" },
- { url = "https://files.pythonhosted.org/packages/1e/c7/5420d1943c8e3ce1a21c0a9330bcf7edafb6aa65d26b21dbb3267c9e8112/watchfiles-1.1.1-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:672b8adf25b1a0d35c96b5888b7b18699d27d4194bac8beeae75be4b7a3fc9b2", size = 396889, upload-time = "2025-10-14T15:06:07.035Z" },
- { url = "https://files.pythonhosted.org/packages/0c/e5/0072cef3804ce8d3aaddbfe7788aadff6b3d3f98a286fdbee9fd74ca59a7/watchfiles-1.1.1-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:77a13aea58bc2b90173bc69f2a90de8e282648939a00a602e1dc4ee23e26b66d", size = 451616, upload-time = "2025-10-14T15:06:08.072Z" },
- { url = "https://files.pythonhosted.org/packages/83/4e/b87b71cbdfad81ad7e83358b3e447fedd281b880a03d64a760fe0a11fc2e/watchfiles-1.1.1-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0b495de0bb386df6a12b18335a0285dda90260f51bdb505503c02bcd1ce27a8b", size = 458413, upload-time = "2025-10-14T15:06:09.209Z" },
- { url = "https://files.pythonhosted.org/packages/d3/8e/e500f8b0b77be4ff753ac94dc06b33d8f0d839377fee1b78e8c8d8f031bf/watchfiles-1.1.1-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:db476ab59b6765134de1d4fe96a1a9c96ddf091683599be0f26147ea1b2e4b88", size = 408250, upload-time = "2025-10-14T15:06:10.264Z" },
- { url = "https://files.pythonhosted.org/packages/bd/95/615e72cd27b85b61eec764a5ca51bd94d40b5adea5ff47567d9ebc4d275a/watchfiles-1.1.1-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:89eef07eee5e9d1fda06e38822ad167a044153457e6fd997f8a858ab7564a336", size = 396117, upload-time = "2025-10-14T15:06:11.28Z" },
- { url = "https://files.pythonhosted.org/packages/c9/81/e7fe958ce8a7fb5c73cc9fb07f5aeaf755e6aa72498c57d760af760c91f8/watchfiles-1.1.1-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ce19e06cbda693e9e7686358af9cd6f5d61312ab8b00488bc36f5aabbaf77e24", size = 450493, upload-time = "2025-10-14T15:06:12.321Z" },
- { url = "https://files.pythonhosted.org/packages/6e/d4/ed38dd3b1767193de971e694aa544356e63353c33a85d948166b5ff58b9e/watchfiles-1.1.1-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3e6f39af2eab0118338902798b5aa6664f46ff66bc0280de76fca67a7f262a49", size = 457546, upload-time = "2025-10-14T15:06:13.372Z" },
]
[[package]]
@@ -6030,11 +6793,11 @@ wheels = [
[[package]]
name = "wcwidth"
-version = "0.2.14"
+version = "0.6.0"
source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/24/30/6b0809f4510673dc723187aeaf24c7f5459922d01e2f794277a3dfb90345/wcwidth-0.2.14.tar.gz", hash = "sha256:4d478375d31bc5395a3c55c40ccdf3354688364cd61c4f6adacaa9215d0b3605", size = 102293, upload-time = "2025-09-22T16:29:53.023Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/35/a2/8e3becb46433538a38726c948d3399905a4c7cabd0df578ede5dc51f0ec2/wcwidth-0.6.0.tar.gz", hash = "sha256:cdc4e4262d6ef9a1a57e018384cbeb1208d8abbc64176027e2c2455c81313159", size = 159684, upload-time = "2026-02-06T19:19:40.919Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/af/b5/123f13c975e9f27ab9c0770f514345bd406d0e8d3b7a0723af9d43f710af/wcwidth-0.2.14-py2.py3-none-any.whl", hash = "sha256:a7bb560c8aee30f9957e5f9895805edd20602f2d7f720186dfd906e82b4982e1", size = 37286, upload-time = "2025-09-22T16:29:51.641Z" },
+ { url = "https://files.pythonhosted.org/packages/68/5a/199c59e0a824a3db2b89c5d2dade7ab5f9624dbf6448dc291b46d5ec94d3/wcwidth-0.6.0-py3-none-any.whl", hash = "sha256:1a3a1e510b553315f8e146c54764f4fb6264ffad731b3d78088cdb1478ffbdad", size = 94189, upload-time = "2026-02-06T19:19:39.646Z" },
]
[[package]]
@@ -6043,8 +6806,8 @@ version = "1.0.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "braceexpand" },
- { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version < '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version < '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
- { name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple" }, marker = "(python_full_version >= '3.11' and extra == 'extra-13-megatron-core-dev') or (python_full_version >= '3.11' and extra == 'extra-13-megatron-core-lts') or (extra == 'extra-13-megatron-core-dev' and extra == 'extra-13-megatron-core-lts')" },
+ { name = "numpy", version = "2.0.2", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-dev'" },
+ { name = "numpy", version = "2.4.4", source = { registry = "https://pypi.org/simple" }, marker = "extra == 'extra-13-megatron-core-lts'" },
{ name = "pyyaml" },
]
sdist = { url = "https://files.pythonhosted.org/packages/5a/3a/68800d92e065cf4750ebecf973b13979c0c929b439e1293012938862038d/webdataset-1.0.2.tar.gz", hash = "sha256:7f0498be827cfa46cc5430a58768a24e2c6a410676a61be1838f53d61afdaab4", size = 80090, upload-time = "2025-06-19T23:26:21.945Z" }
@@ -6058,24 +6821,6 @@ version = "16.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/04/24/4b2031d72e840ce4c1ccb255f693b15c334757fc50023e4db9537080b8c4/websockets-16.0.tar.gz", hash = "sha256:5f6261a5e56e8d5c42a4497b364ea24d94d9563e8fbd44e78ac40879c60179b5", size = 179346, upload-time = "2026-01-10T09:23:47.181Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/20/74/221f58decd852f4b59cc3354cccaf87e8ef695fede361d03dc9a7396573b/websockets-16.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:04cdd5d2d1dacbad0a7bf36ccbcd3ccd5a30ee188f2560b7a62a30d14107b31a", size = 177343, upload-time = "2026-01-10T09:22:21.28Z" },
- { url = "https://files.pythonhosted.org/packages/19/0f/22ef6107ee52ab7f0b710d55d36f5a5d3ef19e8a205541a6d7ffa7994e5a/websockets-16.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:8ff32bb86522a9e5e31439a58addbb0166f0204d64066fb955265c4e214160f0", size = 175021, upload-time = "2026-01-10T09:22:22.696Z" },
- { url = "https://files.pythonhosted.org/packages/10/40/904a4cb30d9b61c0e278899bf36342e9b0208eb3c470324a9ecbaac2a30f/websockets-16.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:583b7c42688636f930688d712885cf1531326ee05effd982028212ccc13e5957", size = 175320, upload-time = "2026-01-10T09:22:23.94Z" },
- { url = "https://files.pythonhosted.org/packages/9d/2f/4b3ca7e106bc608744b1cdae041e005e446124bebb037b18799c2d356864/websockets-16.0-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:7d837379b647c0c4c2355c2499723f82f1635fd2c26510e1f587d89bc2199e72", size = 183815, upload-time = "2026-01-10T09:22:25.469Z" },
- { url = "https://files.pythonhosted.org/packages/86/26/d40eaa2a46d4302becec8d15b0fc5e45bdde05191e7628405a19cf491ccd/websockets-16.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:df57afc692e517a85e65b72e165356ed1df12386ecb879ad5693be08fac65dde", size = 185054, upload-time = "2026-01-10T09:22:27.101Z" },
- { url = "https://files.pythonhosted.org/packages/b0/ba/6500a0efc94f7373ee8fefa8c271acdfd4dca8bd49a90d4be7ccabfc397e/websockets-16.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:2b9f1e0d69bc60a4a87349d50c09a037a2607918746f07de04df9e43252c77a3", size = 184565, upload-time = "2026-01-10T09:22:28.293Z" },
- { url = "https://files.pythonhosted.org/packages/04/b4/96bf2cee7c8d8102389374a2616200574f5f01128d1082f44102140344cc/websockets-16.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:335c23addf3d5e6a8633f9f8eda77efad001671e80b95c491dd0924587ece0b3", size = 183848, upload-time = "2026-01-10T09:22:30.394Z" },
- { url = "https://files.pythonhosted.org/packages/02/8e/81f40fb00fd125357814e8c3025738fc4ffc3da4b6b4a4472a82ba304b41/websockets-16.0-cp310-cp310-win32.whl", hash = "sha256:37b31c1623c6605e4c00d466c9d633f9b812ea430c11c8a278774a1fde1acfa9", size = 178249, upload-time = "2026-01-10T09:22:32.083Z" },
- { url = "https://files.pythonhosted.org/packages/b4/5f/7e40efe8df57db9b91c88a43690ac66f7b7aa73a11aa6a66b927e44f26fa/websockets-16.0-cp310-cp310-win_amd64.whl", hash = "sha256:8e1dab317b6e77424356e11e99a432b7cb2f3ec8c5ab4dabbcee6add48f72b35", size = 178685, upload-time = "2026-01-10T09:22:33.345Z" },
- { url = "https://files.pythonhosted.org/packages/f2/db/de907251b4ff46ae804ad0409809504153b3f30984daf82a1d84a9875830/websockets-16.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:31a52addea25187bde0797a97d6fc3d2f92b6f72a9370792d65a6e84615ac8a8", size = 177340, upload-time = "2026-01-10T09:22:34.539Z" },
- { url = "https://files.pythonhosted.org/packages/f3/fa/abe89019d8d8815c8781e90d697dec52523fb8ebe308bf11664e8de1877e/websockets-16.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:417b28978cdccab24f46400586d128366313e8a96312e4b9362a4af504f3bbad", size = 175022, upload-time = "2026-01-10T09:22:36.332Z" },
- { url = "https://files.pythonhosted.org/packages/58/5d/88ea17ed1ded2079358b40d31d48abe90a73c9e5819dbcde1606e991e2ad/websockets-16.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:af80d74d4edfa3cb9ed973a0a5ba2b2a549371f8a741e0800cb07becdd20f23d", size = 175319, upload-time = "2026-01-10T09:22:37.602Z" },
- { url = "https://files.pythonhosted.org/packages/d2/ae/0ee92b33087a33632f37a635e11e1d99d429d3d323329675a6022312aac2/websockets-16.0-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:08d7af67b64d29823fed316505a89b86705f2b7981c07848fb5e3ea3020c1abe", size = 184631, upload-time = "2026-01-10T09:22:38.789Z" },
- { url = "https://files.pythonhosted.org/packages/c8/c5/27178df583b6c5b31b29f526ba2da5e2f864ecc79c99dae630a85d68c304/websockets-16.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7be95cfb0a4dae143eaed2bcba8ac23f4892d8971311f1b06f3c6b78952ee70b", size = 185870, upload-time = "2026-01-10T09:22:39.893Z" },
- { url = "https://files.pythonhosted.org/packages/87/05/536652aa84ddc1c018dbb7e2c4cbcd0db884580bf8e95aece7593fde526f/websockets-16.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d6297ce39ce5c2e6feb13c1a996a2ded3b6832155fcfc920265c76f24c7cceb5", size = 185361, upload-time = "2026-01-10T09:22:41.016Z" },
- { url = "https://files.pythonhosted.org/packages/6d/e2/d5332c90da12b1e01f06fb1b85c50cfc489783076547415bf9f0a659ec19/websockets-16.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:1c1b30e4f497b0b354057f3467f56244c603a79c0d1dafce1d16c283c25f6e64", size = 184615, upload-time = "2026-01-10T09:22:42.442Z" },
- { url = "https://files.pythonhosted.org/packages/77/fb/d3f9576691cae9253b51555f841bc6600bf0a983a461c79500ace5a5b364/websockets-16.0-cp311-cp311-win32.whl", hash = "sha256:5f451484aeb5cafee1ccf789b1b66f535409d038c56966d6101740c1614b86c6", size = 178246, upload-time = "2026-01-10T09:22:43.654Z" },
- { url = "https://files.pythonhosted.org/packages/54/67/eaff76b3dbaf18dcddabc3b8c1dba50b483761cccff67793897945b37408/websockets-16.0-cp311-cp311-win_amd64.whl", hash = "sha256:8d7f0659570eefb578dacde98e24fb60af35350193e4f56e11190787bee77dac", size = 178684, upload-time = "2026-01-10T09:22:44.941Z" },
{ url = "https://files.pythonhosted.org/packages/84/7b/bac442e6b96c9d25092695578dda82403c77936104b5682307bd4deb1ad4/websockets-16.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:71c989cbf3254fbd5e84d3bff31e4da39c43f884e64f2551d14bb3c186230f00", size = 177365, upload-time = "2026-01-10T09:22:46.787Z" },
{ url = "https://files.pythonhosted.org/packages/b0/fe/136ccece61bd690d9c1f715baaeefd953bb2360134de73519d5df19d29ca/websockets-16.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:8b6e209ffee39ff1b6d0fa7bfef6de950c60dfb91b8fcead17da4ee539121a79", size = 175038, upload-time = "2026-01-10T09:22:47.999Z" },
{ url = "https://files.pythonhosted.org/packages/40/1e/9771421ac2286eaab95b8575b0cb701ae3663abf8b5e1f64f1fd90d0a673/websockets-16.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:86890e837d61574c92a97496d590968b23c2ef0aeb8a9bc9421d174cd378ae39", size = 175328, upload-time = "2026-01-10T09:22:49.809Z" },
@@ -6112,24 +6857,19 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/88/a8/a080593f89b0138b6cba1b28f8df5673b5506f72879322288b031337c0b8/websockets-16.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:32da954ffa2814258030e5a57bc73a3635463238e797c7375dc8091327434206", size = 185356, upload-time = "2026-01-10T09:23:32.627Z" },
{ url = "https://files.pythonhosted.org/packages/c2/b6/b9afed2afadddaf5ebb2afa801abf4b0868f42f8539bfe4b071b5266c9fe/websockets-16.0-cp314-cp314t-win32.whl", hash = "sha256:5a4b4cc550cb665dd8a47f868c8d04c8230f857363ad3c9caf7a0c3bf8c61ca6", size = 178085, upload-time = "2026-01-10T09:23:33.816Z" },
{ url = "https://files.pythonhosted.org/packages/9f/3e/28135a24e384493fa804216b79a6a6759a38cc4ff59118787b9fb693df93/websockets-16.0-cp314-cp314t-win_amd64.whl", hash = "sha256:b14dc141ed6d2dde437cddb216004bcac6a1df0935d79656387bd41632ba0bbd", size = 178531, upload-time = "2026-01-10T09:23:35.016Z" },
- { url = "https://files.pythonhosted.org/packages/72/07/c98a68571dcf256e74f1f816b8cc5eae6eb2d3d5cfa44d37f801619d9166/websockets-16.0-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:349f83cd6c9a415428ee1005cadb5c2c56f4389bc06a9af16103c3bc3dcc8b7d", size = 174947, upload-time = "2026-01-10T09:23:36.166Z" },
- { url = "https://files.pythonhosted.org/packages/7e/52/93e166a81e0305b33fe416338be92ae863563fe7bce446b0f687b9df5aea/websockets-16.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:4a1aba3340a8dca8db6eb5a7986157f52eb9e436b74813764241981ca4888f03", size = 175260, upload-time = "2026-01-10T09:23:37.409Z" },
- { url = "https://files.pythonhosted.org/packages/56/0c/2dbf513bafd24889d33de2ff0368190a0e69f37bcfa19009ef819fe4d507/websockets-16.0-pp311-pypy311_pp73-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:f4a32d1bd841d4bcbffdcb3d2ce50c09c3909fbead375ab28d0181af89fd04da", size = 176071, upload-time = "2026-01-10T09:23:39.158Z" },
- { url = "https://files.pythonhosted.org/packages/a5/8f/aea9c71cc92bf9b6cc0f7f70df8f0b420636b6c96ef4feee1e16f80f75dd/websockets-16.0-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0298d07ee155e2e9fda5be8a9042200dd2e3bb0b8a38482156576f863a9d457c", size = 176968, upload-time = "2026-01-10T09:23:41.031Z" },
- { url = "https://files.pythonhosted.org/packages/9a/3f/f70e03f40ffc9a30d817eef7da1be72ee4956ba8d7255c399a01b135902a/websockets-16.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:a653aea902e0324b52f1613332ddf50b00c06fdaf7e92624fbf8c77c78fa5767", size = 178735, upload-time = "2026-01-10T09:23:42.259Z" },
{ url = "https://files.pythonhosted.org/packages/6f/28/258ebab549c2bf3e64d2b0217b973467394a9cea8c42f70418ca2c5d0d2e/websockets-16.0-py3-none-any.whl", hash = "sha256:1637db62fad1dc833276dded54215f2c7fa46912301a24bd94d45d46a011ceec", size = 171598, upload-time = "2026-01-10T09:23:45.395Z" },
]
[[package]]
name = "werkzeug"
-version = "3.1.5"
+version = "3.1.7"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "markupsafe" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/5a/70/1469ef1d3542ae7c2c7b72bd5e3a4e6ee69d7978fa8a3af05a38eca5becf/werkzeug-3.1.5.tar.gz", hash = "sha256:6a548b0e88955dd07ccb25539d7d0cc97417ee9e179677d22c7041c8f078ce67", size = 864754, upload-time = "2026-01-08T17:49:23.247Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/b5/43/76ded108b296a49f52de6bac5192ca1c4be84e886f9b5c9ba8427d9694fd/werkzeug-3.1.7.tar.gz", hash = "sha256:fb8c01fe6ab13b9b7cdb46892b99b1d66754e1d7ab8e542e865ec13f526b5351", size = 875700, upload-time = "2026-03-24T01:08:07.687Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/ad/e4/8d97cca767bcc1be76d16fb76951608305561c6e056811587f36cb1316a8/werkzeug-3.1.5-py3-none-any.whl", hash = "sha256:5111e36e91086ece91f93268bb39b4a35c1e6f1feac762c9c822ded0a4e322dc", size = 225025, upload-time = "2026-01-08T17:49:21.859Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/b2/0bba9bbb4596d2d2f285a16c2ab04118f6b957d8441566e1abb892e6a6b2/werkzeug-3.1.7-py3-none-any.whl", hash = "sha256:4b314d81163a3e1a169b6a0be2a000a0e204e8873c5de6586f453c55688d422f", size = 226295, upload-time = "2026-03-24T01:08:06.133Z" },
]
[[package]]
@@ -6149,71 +6889,78 @@ wheels = [
[[package]]
name = "wrapt"
-version = "1.17.3"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/95/8f/aeb76c5b46e273670962298c23e7ddde79916cb74db802131d49a85e4b7d/wrapt-1.17.3.tar.gz", hash = "sha256:f66eb08feaa410fe4eebd17f2a2c8e2e46d3476e9f8c783daa8e09e0faa666d0", size = 55547, upload-time = "2025-08-12T05:53:21.714Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/3f/23/bb82321b86411eb51e5a5db3fb8f8032fd30bd7c2d74bfe936136b2fa1d6/wrapt-1.17.3-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:88bbae4d40d5a46142e70d58bf664a89b6b4befaea7b2ecc14e03cedb8e06c04", size = 53482, upload-time = "2025-08-12T05:51:44.467Z" },
- { url = "https://files.pythonhosted.org/packages/45/69/f3c47642b79485a30a59c63f6d739ed779fb4cc8323205d047d741d55220/wrapt-1.17.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e6b13af258d6a9ad602d57d889f83b9d5543acd471eee12eb51f5b01f8eb1bc2", size = 38676, upload-time = "2025-08-12T05:51:32.636Z" },
- { url = "https://files.pythonhosted.org/packages/d1/71/e7e7f5670c1eafd9e990438e69d8fb46fa91a50785332e06b560c869454f/wrapt-1.17.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:fd341868a4b6714a5962c1af0bd44f7c404ef78720c7de4892901e540417111c", size = 38957, upload-time = "2025-08-12T05:51:54.655Z" },
- { url = "https://files.pythonhosted.org/packages/de/17/9f8f86755c191d6779d7ddead1a53c7a8aa18bccb7cea8e7e72dfa6a8a09/wrapt-1.17.3-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:f9b2601381be482f70e5d1051a5965c25fb3625455a2bf520b5a077b22afb775", size = 81975, upload-time = "2025-08-12T05:52:30.109Z" },
- { url = "https://files.pythonhosted.org/packages/f2/15/dd576273491f9f43dd09fce517f6c2ce6eb4fe21681726068db0d0467096/wrapt-1.17.3-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:343e44b2a8e60e06a7e0d29c1671a0d9951f59174f3709962b5143f60a2a98bd", size = 83149, upload-time = "2025-08-12T05:52:09.316Z" },
- { url = "https://files.pythonhosted.org/packages/0c/c4/5eb4ce0d4814521fee7aa806264bf7a114e748ad05110441cd5b8a5c744b/wrapt-1.17.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:33486899acd2d7d3066156b03465b949da3fd41a5da6e394ec49d271baefcf05", size = 82209, upload-time = "2025-08-12T05:52:10.331Z" },
- { url = "https://files.pythonhosted.org/packages/31/4b/819e9e0eb5c8dc86f60dfc42aa4e2c0d6c3db8732bce93cc752e604bb5f5/wrapt-1.17.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:e6f40a8aa5a92f150bdb3e1c44b7e98fb7113955b2e5394122fa5532fec4b418", size = 81551, upload-time = "2025-08-12T05:52:31.137Z" },
- { url = "https://files.pythonhosted.org/packages/f8/83/ed6baf89ba3a56694700139698cf703aac9f0f9eb03dab92f57551bd5385/wrapt-1.17.3-cp310-cp310-win32.whl", hash = "sha256:a36692b8491d30a8c75f1dfee65bef119d6f39ea84ee04d9f9311f83c5ad9390", size = 36464, upload-time = "2025-08-12T05:53:01.204Z" },
- { url = "https://files.pythonhosted.org/packages/2f/90/ee61d36862340ad7e9d15a02529df6b948676b9a5829fd5e16640156627d/wrapt-1.17.3-cp310-cp310-win_amd64.whl", hash = "sha256:afd964fd43b10c12213574db492cb8f73b2f0826c8df07a68288f8f19af2ebe6", size = 38748, upload-time = "2025-08-12T05:53:00.209Z" },
- { url = "https://files.pythonhosted.org/packages/bd/c3/cefe0bd330d389c9983ced15d326f45373f4073c9f4a8c2f99b50bfea329/wrapt-1.17.3-cp310-cp310-win_arm64.whl", hash = "sha256:af338aa93554be859173c39c85243970dc6a289fa907402289eeae7543e1ae18", size = 36810, upload-time = "2025-08-12T05:52:51.906Z" },
- { url = "https://files.pythonhosted.org/packages/52/db/00e2a219213856074a213503fdac0511203dceefff26e1daa15250cc01a0/wrapt-1.17.3-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:273a736c4645e63ac582c60a56b0acb529ef07f78e08dc6bfadf6a46b19c0da7", size = 53482, upload-time = "2025-08-12T05:51:45.79Z" },
- { url = "https://files.pythonhosted.org/packages/5e/30/ca3c4a5eba478408572096fe9ce36e6e915994dd26a4e9e98b4f729c06d9/wrapt-1.17.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:5531d911795e3f935a9c23eb1c8c03c211661a5060aab167065896bbf62a5f85", size = 38674, upload-time = "2025-08-12T05:51:34.629Z" },
- { url = "https://files.pythonhosted.org/packages/31/25/3e8cc2c46b5329c5957cec959cb76a10718e1a513309c31399a4dad07eb3/wrapt-1.17.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:0610b46293c59a3adbae3dee552b648b984176f8562ee0dba099a56cfbe4df1f", size = 38959, upload-time = "2025-08-12T05:51:56.074Z" },
- { url = "https://files.pythonhosted.org/packages/5d/8f/a32a99fc03e4b37e31b57cb9cefc65050ea08147a8ce12f288616b05ef54/wrapt-1.17.3-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b32888aad8b6e68f83a8fdccbf3165f5469702a7544472bdf41f582970ed3311", size = 82376, upload-time = "2025-08-12T05:52:32.134Z" },
- { url = "https://files.pythonhosted.org/packages/31/57/4930cb8d9d70d59c27ee1332a318c20291749b4fba31f113c2f8ac49a72e/wrapt-1.17.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8cccf4f81371f257440c88faed6b74f1053eef90807b77e31ca057b2db74edb1", size = 83604, upload-time = "2025-08-12T05:52:11.663Z" },
- { url = "https://files.pythonhosted.org/packages/a8/f3/1afd48de81d63dd66e01b263a6fbb86e1b5053b419b9b33d13e1f6d0f7d0/wrapt-1.17.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d8a210b158a34164de8bb68b0e7780041a903d7b00c87e906fb69928bf7890d5", size = 82782, upload-time = "2025-08-12T05:52:12.626Z" },
- { url = "https://files.pythonhosted.org/packages/1e/d7/4ad5327612173b144998232f98a85bb24b60c352afb73bc48e3e0d2bdc4e/wrapt-1.17.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:79573c24a46ce11aab457b472efd8d125e5a51da2d1d24387666cd85f54c05b2", size = 82076, upload-time = "2025-08-12T05:52:33.168Z" },
- { url = "https://files.pythonhosted.org/packages/bb/59/e0adfc831674a65694f18ea6dc821f9fcb9ec82c2ce7e3d73a88ba2e8718/wrapt-1.17.3-cp311-cp311-win32.whl", hash = "sha256:c31eebe420a9a5d2887b13000b043ff6ca27c452a9a22fa71f35f118e8d4bf89", size = 36457, upload-time = "2025-08-12T05:53:03.936Z" },
- { url = "https://files.pythonhosted.org/packages/83/88/16b7231ba49861b6f75fc309b11012ede4d6b0a9c90969d9e0db8d991aeb/wrapt-1.17.3-cp311-cp311-win_amd64.whl", hash = "sha256:0b1831115c97f0663cb77aa27d381237e73ad4f721391a9bfb2fe8bc25fa6e77", size = 38745, upload-time = "2025-08-12T05:53:02.885Z" },
- { url = "https://files.pythonhosted.org/packages/9a/1e/c4d4f3398ec073012c51d1c8d87f715f56765444e1a4b11e5180577b7e6e/wrapt-1.17.3-cp311-cp311-win_arm64.whl", hash = "sha256:5a7b3c1ee8265eb4c8f1b7d29943f195c00673f5ab60c192eba2d4a7eae5f46a", size = 36806, upload-time = "2025-08-12T05:52:53.368Z" },
- { url = "https://files.pythonhosted.org/packages/9f/41/cad1aba93e752f1f9268c77270da3c469883d56e2798e7df6240dcb2287b/wrapt-1.17.3-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:ab232e7fdb44cdfbf55fc3afa31bcdb0d8980b9b95c38b6405df2acb672af0e0", size = 53998, upload-time = "2025-08-12T05:51:47.138Z" },
- { url = "https://files.pythonhosted.org/packages/60/f8/096a7cc13097a1869fe44efe68dace40d2a16ecb853141394047f0780b96/wrapt-1.17.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:9baa544e6acc91130e926e8c802a17f3b16fbea0fd441b5a60f5cf2cc5c3deba", size = 39020, upload-time = "2025-08-12T05:51:35.906Z" },
- { url = "https://files.pythonhosted.org/packages/33/df/bdf864b8997aab4febb96a9ae5c124f700a5abd9b5e13d2a3214ec4be705/wrapt-1.17.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6b538e31eca1a7ea4605e44f81a48aa24c4632a277431a6ed3f328835901f4fd", size = 39098, upload-time = "2025-08-12T05:51:57.474Z" },
- { url = "https://files.pythonhosted.org/packages/9f/81/5d931d78d0eb732b95dc3ddaeeb71c8bb572fb01356e9133916cd729ecdd/wrapt-1.17.3-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:042ec3bb8f319c147b1301f2393bc19dba6e176b7da446853406d041c36c7828", size = 88036, upload-time = "2025-08-12T05:52:34.784Z" },
- { url = "https://files.pythonhosted.org/packages/ca/38/2e1785df03b3d72d34fc6252d91d9d12dc27a5c89caef3335a1bbb8908ca/wrapt-1.17.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3af60380ba0b7b5aeb329bc4e402acd25bd877e98b3727b0135cb5c2efdaefe9", size = 88156, upload-time = "2025-08-12T05:52:13.599Z" },
- { url = "https://files.pythonhosted.org/packages/b3/8b/48cdb60fe0603e34e05cffda0b2a4adab81fd43718e11111a4b0100fd7c1/wrapt-1.17.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:0b02e424deef65c9f7326d8c19220a2c9040c51dc165cddb732f16198c168396", size = 87102, upload-time = "2025-08-12T05:52:14.56Z" },
- { url = "https://files.pythonhosted.org/packages/3c/51/d81abca783b58f40a154f1b2c56db1d2d9e0d04fa2d4224e357529f57a57/wrapt-1.17.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:74afa28374a3c3a11b3b5e5fca0ae03bef8450d6aa3ab3a1e2c30e3a75d023dc", size = 87732, upload-time = "2025-08-12T05:52:36.165Z" },
- { url = "https://files.pythonhosted.org/packages/9e/b1/43b286ca1392a006d5336412d41663eeef1ad57485f3e52c767376ba7e5a/wrapt-1.17.3-cp312-cp312-win32.whl", hash = "sha256:4da9f45279fff3543c371d5ababc57a0384f70be244de7759c85a7f989cb4ebe", size = 36705, upload-time = "2025-08-12T05:53:07.123Z" },
- { url = "https://files.pythonhosted.org/packages/28/de/49493f962bd3c586ab4b88066e967aa2e0703d6ef2c43aa28cb83bf7b507/wrapt-1.17.3-cp312-cp312-win_amd64.whl", hash = "sha256:e71d5c6ebac14875668a1e90baf2ea0ef5b7ac7918355850c0908ae82bcb297c", size = 38877, upload-time = "2025-08-12T05:53:05.436Z" },
- { url = "https://files.pythonhosted.org/packages/f1/48/0f7102fe9cb1e8a5a77f80d4f0956d62d97034bbe88d33e94699f99d181d/wrapt-1.17.3-cp312-cp312-win_arm64.whl", hash = "sha256:604d076c55e2fdd4c1c03d06dc1a31b95130010517b5019db15365ec4a405fc6", size = 36885, upload-time = "2025-08-12T05:52:54.367Z" },
- { url = "https://files.pythonhosted.org/packages/fc/f6/759ece88472157acb55fc195e5b116e06730f1b651b5b314c66291729193/wrapt-1.17.3-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:a47681378a0439215912ef542c45a783484d4dd82bac412b71e59cf9c0e1cea0", size = 54003, upload-time = "2025-08-12T05:51:48.627Z" },
- { url = "https://files.pythonhosted.org/packages/4f/a9/49940b9dc6d47027dc850c116d79b4155f15c08547d04db0f07121499347/wrapt-1.17.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:54a30837587c6ee3cd1a4d1c2ec5d24e77984d44e2f34547e2323ddb4e22eb77", size = 39025, upload-time = "2025-08-12T05:51:37.156Z" },
- { url = "https://files.pythonhosted.org/packages/45/35/6a08de0f2c96dcdd7fe464d7420ddb9a7655a6561150e5fc4da9356aeaab/wrapt-1.17.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:16ecf15d6af39246fe33e507105d67e4b81d8f8d2c6598ff7e3ca1b8a37213f7", size = 39108, upload-time = "2025-08-12T05:51:58.425Z" },
- { url = "https://files.pythonhosted.org/packages/0c/37/6faf15cfa41bf1f3dba80cd3f5ccc6622dfccb660ab26ed79f0178c7497f/wrapt-1.17.3-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:6fd1ad24dc235e4ab88cda009e19bf347aabb975e44fd5c2fb22a3f6e4141277", size = 88072, upload-time = "2025-08-12T05:52:37.53Z" },
- { url = "https://files.pythonhosted.org/packages/78/f2/efe19ada4a38e4e15b6dff39c3e3f3f73f5decf901f66e6f72fe79623a06/wrapt-1.17.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0ed61b7c2d49cee3c027372df5809a59d60cf1b6c2f81ee980a091f3afed6a2d", size = 88214, upload-time = "2025-08-12T05:52:15.886Z" },
- { url = "https://files.pythonhosted.org/packages/40/90/ca86701e9de1622b16e09689fc24b76f69b06bb0150990f6f4e8b0eeb576/wrapt-1.17.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:423ed5420ad5f5529db9ce89eac09c8a2f97da18eb1c870237e84c5a5c2d60aa", size = 87105, upload-time = "2025-08-12T05:52:17.914Z" },
- { url = "https://files.pythonhosted.org/packages/fd/e0/d10bd257c9a3e15cbf5523025252cc14d77468e8ed644aafb2d6f54cb95d/wrapt-1.17.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e01375f275f010fcbf7f643b4279896d04e571889b8a5b3f848423d91bf07050", size = 87766, upload-time = "2025-08-12T05:52:39.243Z" },
- { url = "https://files.pythonhosted.org/packages/e8/cf/7d848740203c7b4b27eb55dbfede11aca974a51c3d894f6cc4b865f42f58/wrapt-1.17.3-cp313-cp313-win32.whl", hash = "sha256:53e5e39ff71b3fc484df8a522c933ea2b7cdd0d5d15ae82e5b23fde87d44cbd8", size = 36711, upload-time = "2025-08-12T05:53:10.074Z" },
- { url = "https://files.pythonhosted.org/packages/57/54/35a84d0a4d23ea675994104e667ceff49227ce473ba6a59ba2c84f250b74/wrapt-1.17.3-cp313-cp313-win_amd64.whl", hash = "sha256:1f0b2f40cf341ee8cc1a97d51ff50dddb9fcc73241b9143ec74b30fc4f44f6cb", size = 38885, upload-time = "2025-08-12T05:53:08.695Z" },
- { url = "https://files.pythonhosted.org/packages/01/77/66e54407c59d7b02a3c4e0af3783168fff8e5d61def52cda8728439d86bc/wrapt-1.17.3-cp313-cp313-win_arm64.whl", hash = "sha256:7425ac3c54430f5fc5e7b6f41d41e704db073309acfc09305816bc6a0b26bb16", size = 36896, upload-time = "2025-08-12T05:52:55.34Z" },
- { url = "https://files.pythonhosted.org/packages/02/a2/cd864b2a14f20d14f4c496fab97802001560f9f41554eef6df201cd7f76c/wrapt-1.17.3-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:cf30f6e3c077c8e6a9a7809c94551203c8843e74ba0c960f4a98cd80d4665d39", size = 54132, upload-time = "2025-08-12T05:51:49.864Z" },
- { url = "https://files.pythonhosted.org/packages/d5/46/d011725b0c89e853dc44cceb738a307cde5d240d023d6d40a82d1b4e1182/wrapt-1.17.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:e228514a06843cae89621384cfe3a80418f3c04aadf8a3b14e46a7be704e4235", size = 39091, upload-time = "2025-08-12T05:51:38.935Z" },
- { url = "https://files.pythonhosted.org/packages/2e/9e/3ad852d77c35aae7ddebdbc3b6d35ec8013af7d7dddad0ad911f3d891dae/wrapt-1.17.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:5ea5eb3c0c071862997d6f3e02af1d055f381b1d25b286b9d6644b79db77657c", size = 39172, upload-time = "2025-08-12T05:51:59.365Z" },
- { url = "https://files.pythonhosted.org/packages/c3/f7/c983d2762bcce2326c317c26a6a1e7016f7eb039c27cdf5c4e30f4160f31/wrapt-1.17.3-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:281262213373b6d5e4bb4353bc36d1ba4084e6d6b5d242863721ef2bf2c2930b", size = 87163, upload-time = "2025-08-12T05:52:40.965Z" },
- { url = "https://files.pythonhosted.org/packages/e4/0f/f673f75d489c7f22d17fe0193e84b41540d962f75fce579cf6873167c29b/wrapt-1.17.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dc4a8d2b25efb6681ecacad42fca8859f88092d8732b170de6a5dddd80a1c8fa", size = 87963, upload-time = "2025-08-12T05:52:20.326Z" },
- { url = "https://files.pythonhosted.org/packages/df/61/515ad6caca68995da2fac7a6af97faab8f78ebe3bf4f761e1b77efbc47b5/wrapt-1.17.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:373342dd05b1d07d752cecbec0c41817231f29f3a89aa8b8843f7b95992ed0c7", size = 86945, upload-time = "2025-08-12T05:52:21.581Z" },
- { url = "https://files.pythonhosted.org/packages/d3/bd/4e70162ce398462a467bc09e768bee112f1412e563620adc353de9055d33/wrapt-1.17.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:d40770d7c0fd5cbed9d84b2c3f2e156431a12c9a37dc6284060fb4bec0b7ffd4", size = 86857, upload-time = "2025-08-12T05:52:43.043Z" },
- { url = "https://files.pythonhosted.org/packages/2b/b8/da8560695e9284810b8d3df8a19396a6e40e7518059584a1a394a2b35e0a/wrapt-1.17.3-cp314-cp314-win32.whl", hash = "sha256:fbd3c8319de8e1dc79d346929cd71d523622da527cca14e0c1d257e31c2b8b10", size = 37178, upload-time = "2025-08-12T05:53:12.605Z" },
- { url = "https://files.pythonhosted.org/packages/db/c8/b71eeb192c440d67a5a0449aaee2310a1a1e8eca41676046f99ed2487e9f/wrapt-1.17.3-cp314-cp314-win_amd64.whl", hash = "sha256:e1a4120ae5705f673727d3253de3ed0e016f7cd78dc463db1b31e2463e1f3cf6", size = 39310, upload-time = "2025-08-12T05:53:11.106Z" },
- { url = "https://files.pythonhosted.org/packages/45/20/2cda20fd4865fa40f86f6c46ed37a2a8356a7a2fde0773269311f2af56c7/wrapt-1.17.3-cp314-cp314-win_arm64.whl", hash = "sha256:507553480670cab08a800b9463bdb881b2edeed77dc677b0a5915e6106e91a58", size = 37266, upload-time = "2025-08-12T05:52:56.531Z" },
- { url = "https://files.pythonhosted.org/packages/77/ed/dd5cf21aec36c80443c6f900449260b80e2a65cf963668eaef3b9accce36/wrapt-1.17.3-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:ed7c635ae45cfbc1a7371f708727bf74690daedc49b4dba310590ca0bd28aa8a", size = 56544, upload-time = "2025-08-12T05:51:51.109Z" },
- { url = "https://files.pythonhosted.org/packages/8d/96/450c651cc753877ad100c7949ab4d2e2ecc4d97157e00fa8f45df682456a/wrapt-1.17.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:249f88ed15503f6492a71f01442abddd73856a0032ae860de6d75ca62eed8067", size = 40283, upload-time = "2025-08-12T05:51:39.912Z" },
- { url = "https://files.pythonhosted.org/packages/d1/86/2fcad95994d9b572db57632acb6f900695a648c3e063f2cd344b3f5c5a37/wrapt-1.17.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:5a03a38adec8066d5a37bea22f2ba6bbf39fcdefbe2d91419ab864c3fb515454", size = 40366, upload-time = "2025-08-12T05:52:00.693Z" },
- { url = "https://files.pythonhosted.org/packages/64/0e/f4472f2fdde2d4617975144311f8800ef73677a159be7fe61fa50997d6c0/wrapt-1.17.3-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:5d4478d72eb61c36e5b446e375bbc49ed002430d17cdec3cecb36993398e1a9e", size = 108571, upload-time = "2025-08-12T05:52:44.521Z" },
- { url = "https://files.pythonhosted.org/packages/cc/01/9b85a99996b0a97c8a17484684f206cbb6ba73c1ce6890ac668bcf3838fb/wrapt-1.17.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:223db574bb38637e8230eb14b185565023ab624474df94d2af18f1cdb625216f", size = 113094, upload-time = "2025-08-12T05:52:22.618Z" },
- { url = "https://files.pythonhosted.org/packages/25/02/78926c1efddcc7b3aa0bc3d6b33a822f7d898059f7cd9ace8c8318e559ef/wrapt-1.17.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e405adefb53a435f01efa7ccdec012c016b5a1d3f35459990afc39b6be4d5056", size = 110659, upload-time = "2025-08-12T05:52:24.057Z" },
- { url = "https://files.pythonhosted.org/packages/dc/ee/c414501ad518ac3e6fe184753632fe5e5ecacdcf0effc23f31c1e4f7bfcf/wrapt-1.17.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:88547535b787a6c9ce4086917b6e1d291aa8ed914fdd3a838b3539dc95c12804", size = 106946, upload-time = "2025-08-12T05:52:45.976Z" },
- { url = "https://files.pythonhosted.org/packages/be/44/a1bd64b723d13bb151d6cc91b986146a1952385e0392a78567e12149c7b4/wrapt-1.17.3-cp314-cp314t-win32.whl", hash = "sha256:41b1d2bc74c2cac6f9074df52b2efbef2b30bdfe5f40cb78f8ca22963bc62977", size = 38717, upload-time = "2025-08-12T05:53:15.214Z" },
- { url = "https://files.pythonhosted.org/packages/79/d9/7cfd5a312760ac4dd8bf0184a6ee9e43c33e47f3dadc303032ce012b8fa3/wrapt-1.17.3-cp314-cp314t-win_amd64.whl", hash = "sha256:73d496de46cd2cdbdbcce4ae4bcdb4afb6a11234a1df9c085249d55166b95116", size = 41334, upload-time = "2025-08-12T05:53:14.178Z" },
- { url = "https://files.pythonhosted.org/packages/46/78/10ad9781128ed2f99dbc474f43283b13fea8ba58723e98844367531c18e9/wrapt-1.17.3-cp314-cp314t-win_arm64.whl", hash = "sha256:f38e60678850c42461d4202739f9bf1e3a737c7ad283638251e79cc49effb6b6", size = 38471, upload-time = "2025-08-12T05:52:57.784Z" },
- { url = "https://files.pythonhosted.org/packages/1f/f6/a933bd70f98e9cf3e08167fc5cd7aaaca49147e48411c0bd5ae701bb2194/wrapt-1.17.3-py3-none-any.whl", hash = "sha256:7171ae35d2c33d326ac19dd8facb1e82e5fd04ef8c6c0e394d7af55a55051c22", size = 23591, upload-time = "2025-08-12T05:53:20.674Z" },
+version = "2.1.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/2e/64/925f213fdcbb9baeb1530449ac71a4d57fc361c053d06bf78d0c5c7cd80c/wrapt-2.1.2.tar.gz", hash = "sha256:3996a67eecc2c68fd47b4e3c564405a5777367adfd9b8abb58387b63ee83b21e", size = 81678, upload-time = "2026-03-06T02:53:25.134Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4c/b6/1db817582c49c7fcbb7df6809d0f515af29d7c2fbf57eb44c36e98fb1492/wrapt-2.1.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ff2aad9c4cda28a8f0653fc2d487596458c2a3f475e56ba02909e950a9efa6a9", size = 61255, upload-time = "2026-03-06T02:52:45.663Z" },
+ { url = "https://files.pythonhosted.org/packages/a2/16/9b02a6b99c09227c93cd4b73acc3678114154ec38da53043c0ddc1fba0dc/wrapt-2.1.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6433ea84e1cfacf32021d2a4ee909554ade7fd392caa6f7c13f1f4bf7b8e8748", size = 61848, upload-time = "2026-03-06T02:53:48.728Z" },
+ { url = "https://files.pythonhosted.org/packages/af/aa/ead46a88f9ec3a432a4832dfedb84092fc35af2d0ba40cd04aea3889f247/wrapt-2.1.2-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:c20b757c268d30d6215916a5fa8461048d023865d888e437fab451139cad6c8e", size = 121433, upload-time = "2026-03-06T02:54:40.328Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/9f/742c7c7cdf58b59085a1ee4b6c37b013f66ac33673a7ef4aaed5e992bc33/wrapt-2.1.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:79847b83eb38e70d93dc392c7c5b587efe65b3e7afcc167aa8abd5d60e8761c8", size = 123013, upload-time = "2026-03-06T02:53:26.58Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/44/2c3dd45d53236b7ed7c646fcf212251dc19e48e599debd3926b52310fafb/wrapt-2.1.2-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f8fba1bae256186a83d1875b2b1f4e2d1242e8fac0f58ec0d7e41b26967b965c", size = 117326, upload-time = "2026-03-06T02:53:11.547Z" },
+ { url = "https://files.pythonhosted.org/packages/74/e2/b17d66abc26bd96f89dec0ecd0ef03da4a1286e6ff793839ec431b9fae57/wrapt-2.1.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:e3d3b35eedcf5f7d022291ecd7533321c4775f7b9cd0050a31a68499ba45757c", size = 121444, upload-time = "2026-03-06T02:54:09.5Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/62/e2977843fdf9f03daf1586a0ff49060b1b2fc7ff85a7ea82b6217c1ae36e/wrapt-2.1.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:6f2c5390460de57fa9582bc8a1b7a6c86e1a41dfad74c5225fc07044c15cc8d1", size = 116237, upload-time = "2026-03-06T02:54:03.884Z" },
+ { url = "https://files.pythonhosted.org/packages/88/dd/27fc67914e68d740bce512f11734aec08696e6b17641fef8867c00c949fc/wrapt-2.1.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:7dfa9f2cf65d027b951d05c662cc99ee3bd01f6e4691ed39848a7a5fffc902b2", size = 120563, upload-time = "2026-03-06T02:53:20.412Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/9f/b750b3692ed2ef4705cb305bd68858e73010492b80e43d2a4faa5573cbe7/wrapt-2.1.2-cp312-cp312-win32.whl", hash = "sha256:eba8155747eb2cae4a0b913d9ebd12a1db4d860fc4c829d7578c7b989bd3f2f0", size = 58198, upload-time = "2026-03-06T02:53:37.732Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/b2/feecfe29f28483d888d76a48f03c4c4d8afea944dbee2b0cd3380f9df032/wrapt-2.1.2-cp312-cp312-win_amd64.whl", hash = "sha256:1c51c738d7d9faa0b3601708e7e2eda9bf779e1b601dce6c77411f2a1b324a63", size = 60441, upload-time = "2026-03-06T02:52:47.138Z" },
+ { url = "https://files.pythonhosted.org/packages/44/e1/e328f605d6e208547ea9fd120804fcdec68536ac748987a68c47c606eea8/wrapt-2.1.2-cp312-cp312-win_arm64.whl", hash = "sha256:c8e46ae8e4032792eb2f677dbd0d557170a8e5524d22acc55199f43efedd39bf", size = 58836, upload-time = "2026-03-06T02:53:22.053Z" },
+ { url = "https://files.pythonhosted.org/packages/4c/7a/d936840735c828b38d26a854e85d5338894cda544cb7a85a9d5b8b9c4df7/wrapt-2.1.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:787fd6f4d67befa6fe2abdffcbd3de2d82dfc6fb8a6d850407c53332709d030b", size = 61259, upload-time = "2026-03-06T02:53:41.922Z" },
+ { url = "https://files.pythonhosted.org/packages/5e/88/9a9b9a90ac8ca11c2fdb6a286cb3a1fc7dd774c00ed70929a6434f6bc634/wrapt-2.1.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:4bdf26e03e6d0da3f0e9422fd36bcebf7bc0eeb55fdf9c727a09abc6b9fe472e", size = 61851, upload-time = "2026-03-06T02:52:48.672Z" },
+ { url = "https://files.pythonhosted.org/packages/03/a9/5b7d6a16fd6533fed2756900fc8fc923f678179aea62ada6d65c92718c00/wrapt-2.1.2-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:bbac24d879aa22998e87f6b3f481a5216311e7d53c7db87f189a7a0266dafffb", size = 121446, upload-time = "2026-03-06T02:54:14.013Z" },
+ { url = "https://files.pythonhosted.org/packages/45/bb/34c443690c847835cfe9f892be78c533d4f32366ad2888972c094a897e39/wrapt-2.1.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:16997dfb9d67addc2e3f41b62a104341e80cac52f91110dece393923c0ebd5ca", size = 123056, upload-time = "2026-03-06T02:54:10.829Z" },
+ { url = "https://files.pythonhosted.org/packages/93/b9/ff205f391cb708f67f41ea148545f2b53ff543a7ac293b30d178af4d2271/wrapt-2.1.2-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:162e4e2ba7542da9027821cb6e7c5e068d64f9a10b5f15512ea28e954893a267", size = 117359, upload-time = "2026-03-06T02:53:03.623Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/3d/1ea04d7747825119c3c9a5e0874a40b33594ada92e5649347c457d982805/wrapt-2.1.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f29c827a8d9936ac320746747a016c4bc66ef639f5cd0d32df24f5eacbf9c69f", size = 121479, upload-time = "2026-03-06T02:53:45.844Z" },
+ { url = "https://files.pythonhosted.org/packages/78/cc/ee3a011920c7a023b25e8df26f306b2484a531ab84ca5c96260a73de76c0/wrapt-2.1.2-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:a9dd9813825f7ecb018c17fd147a01845eb330254dff86d3b5816f20f4d6aaf8", size = 116271, upload-time = "2026-03-06T02:54:46.356Z" },
+ { url = "https://files.pythonhosted.org/packages/98/fd/e5ff7ded41b76d802cf1191288473e850d24ba2e39a6ec540f21ae3b57cb/wrapt-2.1.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6f8dbdd3719e534860d6a78526aafc220e0241f981367018c2875178cf83a413", size = 120573, upload-time = "2026-03-06T02:52:50.163Z" },
+ { url = "https://files.pythonhosted.org/packages/47/c5/242cae3b5b080cd09bacef0591691ba1879739050cc7c801ff35c8886b66/wrapt-2.1.2-cp313-cp313-win32.whl", hash = "sha256:5c35b5d82b16a3bc6e0a04349b606a0582bc29f573786aebe98e0c159bc48db6", size = 58205, upload-time = "2026-03-06T02:53:47.494Z" },
+ { url = "https://files.pythonhosted.org/packages/12/69/c358c61e7a50f290958809b3c61ebe8b3838ea3e070d7aac9814f95a0528/wrapt-2.1.2-cp313-cp313-win_amd64.whl", hash = "sha256:f8bc1c264d8d1cf5b3560a87bbdd31131573eb25f9f9447bb6252b8d4c44a3a1", size = 60452, upload-time = "2026-03-06T02:53:30.038Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/66/c8a6fcfe321295fd8c0ab1bd685b5a01462a9b3aa2f597254462fc2bc975/wrapt-2.1.2-cp313-cp313-win_arm64.whl", hash = "sha256:3beb22f674550d5634642c645aba4c72a2c66fb185ae1aebe1e955fae5a13baf", size = 58842, upload-time = "2026-03-06T02:52:52.114Z" },
+ { url = "https://files.pythonhosted.org/packages/da/55/9c7052c349106e0b3f17ae8db4b23a691a963c334de7f9dbd60f8f74a831/wrapt-2.1.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:0fc04bc8664a8bc4c8e00b37b5355cffca2535209fba1abb09ae2b7c76ddf82b", size = 63075, upload-time = "2026-03-06T02:53:19.108Z" },
+ { url = "https://files.pythonhosted.org/packages/09/a8/ce7b4006f7218248dd71b7b2b732d0710845a0e49213b18faef64811ffef/wrapt-2.1.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:a9b9d50c9af998875a1482a038eb05755dfd6fe303a313f6a940bb53a83c3f18", size = 63719, upload-time = "2026-03-06T02:54:33.452Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/e5/2ca472e80b9e2b7a17f106bb8f9df1db11e62101652ce210f66935c6af67/wrapt-2.1.2-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:2d3ff4f0024dd224290c0eabf0240f1bfc1f26363431505fb1b0283d3b08f11d", size = 152643, upload-time = "2026-03-06T02:52:42.721Z" },
+ { url = "https://files.pythonhosted.org/packages/36/42/30f0f2cefca9d9cbf6835f544d825064570203c3e70aa873d8ae12e23791/wrapt-2.1.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3278c471f4468ad544a691b31bb856374fbdefb7fee1a152153e64019379f015", size = 158805, upload-time = "2026-03-06T02:54:25.441Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/67/d08672f801f604889dcf58f1a0b424fe3808860ede9e03affc1876b295af/wrapt-2.1.2-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a8914c754d3134a3032601c6984db1c576e6abaf3fc68094bb8ab1379d75ff92", size = 145990, upload-time = "2026-03-06T02:53:57.456Z" },
+ { url = "https://files.pythonhosted.org/packages/68/a7/fd371b02e73babec1de6ade596e8cd9691051058cfdadbfd62a5898f3295/wrapt-2.1.2-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:ff95d4264e55839be37bafe1536db2ab2de19da6b65f9244f01f332b5286cfbf", size = 155670, upload-time = "2026-03-06T02:54:55.309Z" },
+ { url = "https://files.pythonhosted.org/packages/86/2d/9fe0095dfdb621009f40117dcebf41d7396c2c22dca6eac779f4c007b86c/wrapt-2.1.2-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:76405518ca4e1b76fbb1b9f686cff93aebae03920cc55ceeec48ff9f719c5f67", size = 144357, upload-time = "2026-03-06T02:54:24.092Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/b6/ec7b4a254abbe4cde9fa15c5d2cca4518f6b07d0f1b77d4ee9655e30280e/wrapt-2.1.2-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:c0be8b5a74c5824e9359b53e7e58bef71a729bacc82e16587db1c4ebc91f7c5a", size = 150269, upload-time = "2026-03-06T02:53:31.268Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/6b/2fabe8ebf148f4ee3c782aae86a795cc68ffe7d432ef550f234025ce0cfa/wrapt-2.1.2-cp313-cp313t-win32.whl", hash = "sha256:f01277d9a5fc1862f26f7626da9cf443bebc0abd2f303f41c5e995b15887dabd", size = 59894, upload-time = "2026-03-06T02:54:15.391Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/fb/9ba66fc2dedc936de5f8073c0217b5d4484e966d87723415cc8262c5d9c2/wrapt-2.1.2-cp313-cp313t-win_amd64.whl", hash = "sha256:84ce8f1c2104d2f6daa912b1b5b039f331febfeee74f8042ad4e04992bd95c8f", size = 63197, upload-time = "2026-03-06T02:54:41.943Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/1c/012d7423c95d0e337117723eb8ecf73c622ce15a97847e84cf3f8f26cd7e/wrapt-2.1.2-cp313-cp313t-win_arm64.whl", hash = "sha256:a93cd767e37faeddbe07d8fc4212d5cba660af59bdb0f6372c93faaa13e6e679", size = 60363, upload-time = "2026-03-06T02:54:48.093Z" },
+ { url = "https://files.pythonhosted.org/packages/39/25/e7ea0b417db02bb796182a5316398a75792cd9a22528783d868755e1f669/wrapt-2.1.2-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:1370e516598854e5b4366e09ce81e08bfe94d42b0fd569b88ec46cc56d9164a9", size = 61418, upload-time = "2026-03-06T02:53:55.706Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/0f/fa539e2f6a770249907757eaeb9a5ff4deb41c026f8466c1c6d799088a9b/wrapt-2.1.2-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:6de1a3851c27e0bd6a04ca993ea6f80fc53e6c742ee1601f486c08e9f9b900a9", size = 61914, upload-time = "2026-03-06T02:52:53.37Z" },
+ { url = "https://files.pythonhosted.org/packages/53/37/02af1867f5b1441aaeda9c82deed061b7cd1372572ddcd717f6df90b5e93/wrapt-2.1.2-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:de9f1a2bbc5ac7f6012ec24525bdd444765a2ff64b5985ac6e0692144838542e", size = 120417, upload-time = "2026-03-06T02:54:30.74Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/b7/0138a6238c8ba7476c77cf786a807f871672b37f37a422970342308276e7/wrapt-2.1.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:970d57ed83fa040d8b20c52fe74a6ae7e3775ae8cff5efd6a81e06b19078484c", size = 122797, upload-time = "2026-03-06T02:54:51.539Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/ad/819ae558036d6a15b7ed290d5b14e209ca795dd4da9c58e50c067d5927b0/wrapt-2.1.2-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3969c56e4563c375861c8df14fa55146e81ac11c8db49ea6fb7f2ba58bc1ff9a", size = 117350, upload-time = "2026-03-06T02:54:37.651Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/2d/afc18dc57a4600a6e594f77a9ae09db54f55ba455440a54886694a84c71b/wrapt-2.1.2-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:57d7c0c980abdc5f1d98b11a2aa3bb159790add80258c717fa49a99921456d90", size = 121223, upload-time = "2026-03-06T02:54:35.221Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/5b/5ec189b22205697bc56eb3b62aed87a1e0423e9c8285d0781c7a83170d15/wrapt-2.1.2-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:776867878e83130c7a04237010463372e877c1c994d449ca6aaafeab6aab2586", size = 116287, upload-time = "2026-03-06T02:54:19.654Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/2d/f84939a7c9b5e6cdd8a8d0f6a26cabf36a0f7e468b967720e8b0cd2bdf69/wrapt-2.1.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:fab036efe5464ec3291411fabb80a7a39e2dd80bae9bcbeeca5087fdfa891e19", size = 119593, upload-time = "2026-03-06T02:54:16.697Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/fe/ccd22a1263159c4ac811ab9374c061bcb4a702773f6e06e38de5f81a1bdc/wrapt-2.1.2-cp314-cp314-win32.whl", hash = "sha256:e6ed62c82ddf58d001096ae84ce7f833db97ae2263bff31c9b336ba8cfe3f508", size = 58631, upload-time = "2026-03-06T02:53:06.498Z" },
+ { url = "https://files.pythonhosted.org/packages/65/0a/6bd83be7bff2e7efaac7b4ac9748da9d75a34634bbbbc8ad077d527146df/wrapt-2.1.2-cp314-cp314-win_amd64.whl", hash = "sha256:467e7c76315390331c67073073d00662015bb730c566820c9ca9b54e4d67fd04", size = 60875, upload-time = "2026-03-06T02:53:50.252Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/c0/0b3056397fe02ff80e5a5d72d627c11eb885d1ca78e71b1a5c1e8c7d45de/wrapt-2.1.2-cp314-cp314-win_arm64.whl", hash = "sha256:da1f00a557c66225d53b095a97eace0fc5349e3bfda28fa34ffae238978ee575", size = 59164, upload-time = "2026-03-06T02:53:59.128Z" },
+ { url = "https://files.pythonhosted.org/packages/71/ed/5d89c798741993b2371396eb9d4634f009ff1ad8a6c78d366fe2883ea7a6/wrapt-2.1.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:62503ffbc2d3a69891cf29beeaccdb4d5e0a126e2b6a851688d4777e01428dbb", size = 63163, upload-time = "2026-03-06T02:52:54.873Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/8c/05d277d182bf36b0a13d6bd393ed1dec3468a25b59d01fba2dd70fe4d6ae/wrapt-2.1.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c7e6cd120ef837d5b6f860a6ea3745f8763805c418bb2f12eeb1fa6e25f22d22", size = 63723, upload-time = "2026-03-06T02:52:56.374Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/27/6c51ec1eff4413c57e72d6106bb8dec6f0c7cdba6503d78f0fa98767bcc9/wrapt-2.1.2-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:3769a77df8e756d65fbc050333f423c01ae012b4f6731aaf70cf2bef61b34596", size = 152652, upload-time = "2026-03-06T02:53:23.79Z" },
+ { url = "https://files.pythonhosted.org/packages/db/4c/d7dd662d6963fc7335bfe29d512b02b71cdfa23eeca7ab3ac74a67505deb/wrapt-2.1.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a76d61a2e851996150ba0f80582dd92a870643fa481f3b3846f229de88caf044", size = 158807, upload-time = "2026-03-06T02:53:35.742Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/4d/1e5eea1a78d539d346765727422976676615814029522c76b87a95f6bcdd/wrapt-2.1.2-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:6f97edc9842cf215312b75fe737ee7c8adda75a89979f8e11558dfff6343cc4b", size = 146061, upload-time = "2026-03-06T02:52:57.574Z" },
+ { url = "https://files.pythonhosted.org/packages/89/bc/62cabea7695cd12a288023251eeefdcb8465056ddaab6227cb78a2de005b/wrapt-2.1.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:4006c351de6d5007aa33a551f600404ba44228a89e833d2fadc5caa5de8edfbf", size = 155667, upload-time = "2026-03-06T02:53:39.422Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/99/6f2888cd68588f24df3a76572c69c2de28287acb9e1972bf0c83ce97dbc1/wrapt-2.1.2-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:a9372fc3639a878c8e7d87e1556fa209091b0a66e912c611e3f833e2c4202be2", size = 144392, upload-time = "2026-03-06T02:54:22.41Z" },
+ { url = "https://files.pythonhosted.org/packages/40/51/1dfc783a6c57971614c48e361a82ca3b6da9055879952587bc99fe1a7171/wrapt-2.1.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:3144b027ff30cbd2fca07c0a87e67011adb717eb5f5bd8496325c17e454257a3", size = 150296, upload-time = "2026-03-06T02:54:07.848Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/38/cbb8b933a0201076c1f64fc42883b0023002bdc14a4964219154e6ff3350/wrapt-2.1.2-cp314-cp314t-win32.whl", hash = "sha256:3b8d15e52e195813efe5db8cec156eebe339aaf84222f4f4f051a6c01f237ed7", size = 60539, upload-time = "2026-03-06T02:54:00.594Z" },
+ { url = "https://files.pythonhosted.org/packages/82/dd/e5176e4b241c9f528402cebb238a36785a628179d7d8b71091154b3e4c9e/wrapt-2.1.2-cp314-cp314t-win_amd64.whl", hash = "sha256:08ffa54146a7559f5b8df4b289b46d963a8e74ed16ba3687f99896101a3990c5", size = 63969, upload-time = "2026-03-06T02:54:39Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/99/79f17046cf67e4a95b9987ea129632ba8bcec0bc81f3fb3d19bdb0bd60cd/wrapt-2.1.2-cp314-cp314t-win_arm64.whl", hash = "sha256:72aaa9d0d8e4ed0e2e98019cea47a21f823c9dd4b43c7b77bba6679ffcca6a00", size = 60554, upload-time = "2026-03-06T02:53:14.132Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/c7/8528ac2dfa2c1e6708f647df7ae144ead13f0a31146f43c7264b4942bf12/wrapt-2.1.2-py3-none-any.whl", hash = "sha256:b8fd6fa2b2c4e7621808f8c62e8317f4aae56e59721ad933bac5239d913cf0e8", size = 43993, upload-time = "2026-03-06T02:53:12.905Z" },
+]
+
+[[package]]
+name = "wsproto"
+version = "1.3.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "h11" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/c7/79/12135bdf8b9c9367b8701c2c19a14c913c120b882d50b014ca0d38083c2c/wsproto-1.3.2.tar.gz", hash = "sha256:b86885dcf294e15204919950f666e06ffc6c7c114ca900b060d6e16293528294", size = 50116, upload-time = "2025-11-20T18:18:01.871Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a4/f5/10b68b7b1544245097b2a1b8238f66f2fc6dcaeb24ba5d917f52bd2eed4f/wsproto-1.3.2-py3-none-any.whl", hash = "sha256:61eea322cdf56e8cc904bd3ad7573359a242ba65688716b0710a5eb12beab584", size = 24405, upload-time = "2025-11-20T18:18:00.454Z" },
]
[[package]]
@@ -6225,20 +6972,6 @@ dependencies = [
]
sdist = { url = "https://files.pythonhosted.org/packages/08/d5/25f7b19af3a2cb4000cac4f9e5525a40bec79f4f5d0ac9b517c0544586a0/xattr-1.3.0.tar.gz", hash = "sha256:30439fabd7de0787b27e9a6e1d569c5959854cb322f64ce7380fedbfa5035036", size = 17148, upload-time = "2025-10-13T22:16:47.353Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/ab/11/bbb25ab921e02efb789efcab5b7d03581b5d28f71d829f21e4ea6aba09fb/xattr-1.3.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:a80c4617e08670cdc3ba71f1dbb275c1627744c5c3641280879cb3bc95a07237", size = 23453, upload-time = "2025-10-13T22:15:50.753Z" },
- { url = "https://files.pythonhosted.org/packages/be/88/66021fdfbb2037a94fc5b16c1dce1894b8e9da7a1829e4be0b491b3f24ff/xattr-1.3.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:51cdaa359f5cd2861178ae01ea3647b56dbdfd98e724a8aa3c04f77123b78217", size = 18551, upload-time = "2025-10-13T22:15:51.961Z" },
- { url = "https://files.pythonhosted.org/packages/be/f7/5dd21fcfc48487a59fcec33ffe02eb671f256424869e9aef87e33c65d95b/xattr-1.3.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2fea070768d7d2d25797817bea93bf0a6fda6449e88cfee8bb3d75de9ed11c7b", size = 18852, upload-time = "2025-10-13T22:15:53.104Z" },
- { url = "https://files.pythonhosted.org/packages/af/2a/e29753ac17a92aadf27b9e16b1d600584d9f10acd0b399d2c06f47af2dff/xattr-1.3.0-cp310-cp310-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:69bca34be2d7a928389aff4e32f27857e1c62d04c91ec7c1519b1636870bd58f", size = 38547, upload-time = "2025-10-13T22:15:54.385Z" },
- { url = "https://files.pythonhosted.org/packages/f4/46/b2c9185d24b93542e4307ce30cd3d4eb6af8efdc843d98ff9f07fcb048d9/xattr-1.3.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:05f8e068409742d246babba60cff8310b2c577745491f498b08bf068e0c867a3", size = 38755, upload-time = "2025-10-13T22:15:55.738Z" },
- { url = "https://files.pythonhosted.org/packages/c0/0a/93cf1f03536bf38e8fd3fe57eb04124e4dfe2e16c0c5ced589d3360a1858/xattr-1.3.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:bbd06987102bc11f5cbd08b15d1029832b862cf5bc61780573fc0828812f01ca", size = 38052, upload-time = "2025-10-13T22:15:57.031Z" },
- { url = "https://files.pythonhosted.org/packages/55/ad/60e43f7e1037cee671e14c2a283e3e7168b756c9938eba62f0616e6599aa/xattr-1.3.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:b8589744116d2c37928b771c50383cb281675cd6dcfd740abfab6883e3d4af85", size = 37560, upload-time = "2025-10-13T22:15:58.295Z" },
- { url = "https://files.pythonhosted.org/packages/8a/64/292426ad5653e72c6e1325bbff22868a20077290d967cebb9c0624ad08b6/xattr-1.3.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:331a51bf8f20c27822f44054b0d760588462d3ed472d5e52ba135cf0bea510e8", size = 23448, upload-time = "2025-10-13T22:15:59.229Z" },
- { url = "https://files.pythonhosted.org/packages/63/84/6539fbe620da8e5927406e76b9c8abad8953025d5f578d792747c38a8c0e/xattr-1.3.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:196360f068b74fa0132a8c6001ce1333f095364b8f43b6fd8cdaf2f18741ef89", size = 18553, upload-time = "2025-10-13T22:16:00.151Z" },
- { url = "https://files.pythonhosted.org/packages/cc/bb/c1c2e24a49f8d13ff878fb85aabc42ea1b2f98ce08d8205b9661d517a9cc/xattr-1.3.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:405d2e4911d37f2b9400fa501acd920fe0c97fe2b2ec252cb23df4b59c000811", size = 18848, upload-time = "2025-10-13T22:16:01.046Z" },
- { url = "https://files.pythonhosted.org/packages/02/c2/a60aad150322b217dfe33695d8d9f32bc01e8f300641b6ba4b73f4b3c03f/xattr-1.3.0-cp311-cp311-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:4ae3a66ae1effd40994f64defeeaa97da369406485e60bfb421f2d781be3b75d", size = 38547, upload-time = "2025-10-13T22:16:01.973Z" },
- { url = "https://files.pythonhosted.org/packages/c6/58/2eca142bad4ea0a2be6b58d3122d0acce310c4e53fa7defd168202772178/xattr-1.3.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:69cd3bfe779f7ba87abe6473fdfa428460cf9e78aeb7e390cfd737b784edf1b5", size = 38753, upload-time = "2025-10-13T22:16:03.244Z" },
- { url = "https://files.pythonhosted.org/packages/2b/50/d032e5254c2c27d36bdb02abdf2735db6768a441f0e3d0f139e0f9f56638/xattr-1.3.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:c5742ca61761a99ae0c522f90a39d5fb8139280f27b254e3128482296d1df2db", size = 38054, upload-time = "2025-10-13T22:16:04.656Z" },
- { url = "https://files.pythonhosted.org/packages/04/24/458a306439aabe0083ca0a7b14c3e6a800ab9782b5ec0bdcec4ec9f3dc6c/xattr-1.3.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:4a04ada131e9bdfd32db3ab1efa9f852646f4f7c9d6fde0596c3825c67161be3", size = 37562, upload-time = "2025-10-13T22:16:05.97Z" },
{ url = "https://files.pythonhosted.org/packages/bf/78/00bdc9290066173e53e1e734d8d8e1a84a6faa9c66aee9df81e4d9aeec1c/xattr-1.3.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:dd4e63614722d183e81842cb237fd1cc978d43384166f9fe22368bfcb187ebe5", size = 23476, upload-time = "2025-10-13T22:16:06.942Z" },
{ url = "https://files.pythonhosted.org/packages/53/16/5243722294eb982514fa7b6b87a29dfb7b29b8e5e1486500c5babaf6e4b3/xattr-1.3.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:995843ef374af73e3370b0c107319611f3cdcdb6d151d629449efecad36be4c4", size = 18556, upload-time = "2025-10-13T22:16:08.209Z" },
{ url = "https://files.pythonhosted.org/packages/d6/5c/d7ab0e547bea885b55f097206459bd612cefb652c5fc1f747130cbc0d42c/xattr-1.3.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fa23a25220e29d956cedf75746e3df6cc824cc1553326d6516479967c540e386", size = 18869, upload-time = "2025-10-13T22:16:10.319Z" },
@@ -6275,36 +7008,6 @@ version = "3.6.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/02/84/30869e01909fb37a6cc7e18688ee8bf1e42d57e7e0777636bd47524c43c7/xxhash-3.6.0.tar.gz", hash = "sha256:f0162a78b13a0d7617b2845b90c763339d1f1d82bb04a4b07f4ab535cc5e05d6", size = 85160, upload-time = "2025-10-02T14:37:08.097Z" }
wheels = [
- { url = "https://files.pythonhosted.org/packages/34/ee/f9f1d656ad168681bb0f6b092372c1e533c4416b8069b1896a175c46e484/xxhash-3.6.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:87ff03d7e35c61435976554477a7f4cd1704c3596a89a8300d5ce7fc83874a71", size = 32845, upload-time = "2025-10-02T14:33:51.573Z" },
- { url = "https://files.pythonhosted.org/packages/a3/b1/93508d9460b292c74a09b83d16750c52a0ead89c51eea9951cb97a60d959/xxhash-3.6.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:f572dfd3d0e2eb1a57511831cf6341242f5a9f8298a45862d085f5b93394a27d", size = 30807, upload-time = "2025-10-02T14:33:52.964Z" },
- { url = "https://files.pythonhosted.org/packages/07/55/28c93a3662f2d200c70704efe74aab9640e824f8ce330d8d3943bf7c9b3c/xxhash-3.6.0-cp310-cp310-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:89952ea539566b9fed2bbd94e589672794b4286f342254fad28b149f9615fef8", size = 193786, upload-time = "2025-10-02T14:33:54.272Z" },
- { url = "https://files.pythonhosted.org/packages/c1/96/fec0be9bb4b8f5d9c57d76380a366f31a1781fb802f76fc7cda6c84893c7/xxhash-3.6.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:48e6f2ffb07a50b52465a1032c3cf1f4a5683f944acaca8a134a2f23674c2058", size = 212830, upload-time = "2025-10-02T14:33:55.706Z" },
- { url = "https://files.pythonhosted.org/packages/c4/a0/c706845ba77b9611f81fd2e93fad9859346b026e8445e76f8c6fd057cc6d/xxhash-3.6.0-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b5b848ad6c16d308c3ac7ad4ba6bede80ed5df2ba8ed382f8932df63158dd4b2", size = 211606, upload-time = "2025-10-02T14:33:57.133Z" },
- { url = "https://files.pythonhosted.org/packages/67/1e/164126a2999e5045f04a69257eea946c0dc3e86541b400d4385d646b53d7/xxhash-3.6.0-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a034590a727b44dd8ac5914236a7b8504144447a9682586c3327e935f33ec8cc", size = 444872, upload-time = "2025-10-02T14:33:58.446Z" },
- { url = "https://files.pythonhosted.org/packages/2d/4b/55ab404c56cd70a2cf5ecfe484838865d0fea5627365c6c8ca156bd09c8f/xxhash-3.6.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8a8f1972e75ebdd161d7896743122834fe87378160c20e97f8b09166213bf8cc", size = 193217, upload-time = "2025-10-02T14:33:59.724Z" },
- { url = "https://files.pythonhosted.org/packages/45/e6/52abf06bac316db33aa269091ae7311bd53cfc6f4b120ae77bac1b348091/xxhash-3.6.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:ee34327b187f002a596d7b167ebc59a1b729e963ce645964bbc050d2f1b73d07", size = 210139, upload-time = "2025-10-02T14:34:02.041Z" },
- { url = "https://files.pythonhosted.org/packages/34/37/db94d490b8691236d356bc249c08819cbcef9273a1a30acf1254ff9ce157/xxhash-3.6.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:339f518c3c7a850dd033ab416ea25a692759dc7478a71131fe8869010d2b75e4", size = 197669, upload-time = "2025-10-02T14:34:03.664Z" },
- { url = "https://files.pythonhosted.org/packages/b7/36/c4f219ef4a17a4f7a64ed3569bc2b5a9c8311abdb22249ac96093625b1a4/xxhash-3.6.0-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:bf48889c9630542d4709192578aebbd836177c9f7a4a2778a7d6340107c65f06", size = 210018, upload-time = "2025-10-02T14:34:05.325Z" },
- { url = "https://files.pythonhosted.org/packages/fd/06/bfac889a374fc2fc439a69223d1750eed2e18a7db8514737ab630534fa08/xxhash-3.6.0-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:5576b002a56207f640636056b4160a378fe36a58db73ae5c27a7ec8db35f71d4", size = 413058, upload-time = "2025-10-02T14:34:06.925Z" },
- { url = "https://files.pythonhosted.org/packages/c9/d1/555d8447e0dd32ad0930a249a522bb2e289f0d08b6b16204cfa42c1f5a0c/xxhash-3.6.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:af1f3278bd02814d6dedc5dec397993b549d6f16c19379721e5a1d31e132c49b", size = 190628, upload-time = "2025-10-02T14:34:08.669Z" },
- { url = "https://files.pythonhosted.org/packages/d1/15/8751330b5186cedc4ed4b597989882ea05e0408b53fa47bcb46a6125bfc6/xxhash-3.6.0-cp310-cp310-win32.whl", hash = "sha256:aed058764db109dc9052720da65fafe84873b05eb8b07e5e653597951af57c3b", size = 30577, upload-time = "2025-10-02T14:34:10.234Z" },
- { url = "https://files.pythonhosted.org/packages/bb/cc/53f87e8b5871a6eb2ff7e89c48c66093bda2be52315a8161ddc54ea550c4/xxhash-3.6.0-cp310-cp310-win_amd64.whl", hash = "sha256:e82da5670f2d0d98950317f82a0e4a0197150ff19a6df2ba40399c2a3b9ae5fb", size = 31487, upload-time = "2025-10-02T14:34:11.618Z" },
- { url = "https://files.pythonhosted.org/packages/9f/00/60f9ea3bb697667a14314d7269956f58bf56bb73864f8f8d52a3c2535e9a/xxhash-3.6.0-cp310-cp310-win_arm64.whl", hash = "sha256:4a082ffff8c6ac07707fb6b671caf7c6e020c75226c561830b73d862060f281d", size = 27863, upload-time = "2025-10-02T14:34:12.619Z" },
- { url = "https://files.pythonhosted.org/packages/17/d4/cc2f0400e9154df4b9964249da78ebd72f318e35ccc425e9f403c392f22a/xxhash-3.6.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:b47bbd8cf2d72797f3c2772eaaac0ded3d3af26481a26d7d7d41dc2d3c46b04a", size = 32844, upload-time = "2025-10-02T14:34:14.037Z" },
- { url = "https://files.pythonhosted.org/packages/5e/ec/1cc11cd13e26ea8bc3cb4af4eaadd8d46d5014aebb67be3f71fb0b68802a/xxhash-3.6.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:2b6821e94346f96db75abaa6e255706fb06ebd530899ed76d32cd99f20dc52fa", size = 30809, upload-time = "2025-10-02T14:34:15.484Z" },
- { url = "https://files.pythonhosted.org/packages/04/5f/19fe357ea348d98ca22f456f75a30ac0916b51c753e1f8b2e0e6fb884cce/xxhash-3.6.0-cp311-cp311-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:d0a9751f71a1a65ce3584e9cae4467651c7e70c9d31017fa57574583a4540248", size = 194665, upload-time = "2025-10-02T14:34:16.541Z" },
- { url = "https://files.pythonhosted.org/packages/90/3b/d1f1a8f5442a5fd8beedae110c5af7604dc37349a8e16519c13c19a9a2de/xxhash-3.6.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8b29ee68625ab37b04c0b40c3fafdf24d2f75ccd778333cfb698f65f6c463f62", size = 213550, upload-time = "2025-10-02T14:34:17.878Z" },
- { url = "https://files.pythonhosted.org/packages/c4/ef/3a9b05eb527457d5db13a135a2ae1a26c80fecd624d20f3e8dcc4cb170f3/xxhash-3.6.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6812c25fe0d6c36a46ccb002f40f27ac903bf18af9f6dd8f9669cb4d176ab18f", size = 212384, upload-time = "2025-10-02T14:34:19.182Z" },
- { url = "https://files.pythonhosted.org/packages/0f/18/ccc194ee698c6c623acbf0f8c2969811a8a4b6185af5e824cd27b9e4fd3e/xxhash-3.6.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:4ccbff013972390b51a18ef1255ef5ac125c92dc9143b2d1909f59abc765540e", size = 445749, upload-time = "2025-10-02T14:34:20.659Z" },
- { url = "https://files.pythonhosted.org/packages/a5/86/cf2c0321dc3940a7aa73076f4fd677a0fb3e405cb297ead7d864fd90847e/xxhash-3.6.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:297b7fbf86c82c550e12e8fb71968b3f033d27b874276ba3624ea868c11165a8", size = 193880, upload-time = "2025-10-02T14:34:22.431Z" },
- { url = "https://files.pythonhosted.org/packages/82/fb/96213c8560e6f948a1ecc9a7613f8032b19ee45f747f4fca4eb31bb6d6ed/xxhash-3.6.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:dea26ae1eb293db089798d3973a5fc928a18fdd97cc8801226fae705b02b14b0", size = 210912, upload-time = "2025-10-02T14:34:23.937Z" },
- { url = "https://files.pythonhosted.org/packages/40/aa/4395e669b0606a096d6788f40dbdf2b819d6773aa290c19e6e83cbfc312f/xxhash-3.6.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:7a0b169aafb98f4284f73635a8e93f0735f9cbde17bd5ec332480484241aaa77", size = 198654, upload-time = "2025-10-02T14:34:25.644Z" },
- { url = "https://files.pythonhosted.org/packages/67/74/b044fcd6b3d89e9b1b665924d85d3f400636c23590226feb1eb09e1176ce/xxhash-3.6.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:08d45aef063a4531b785cd72de4887766d01dc8f362a515693df349fdb825e0c", size = 210867, upload-time = "2025-10-02T14:34:27.203Z" },
- { url = "https://files.pythonhosted.org/packages/bc/fd/3ce73bf753b08cb19daee1eb14aa0d7fe331f8da9c02dd95316ddfe5275e/xxhash-3.6.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:929142361a48ee07f09121fe9e96a84950e8d4df3bb298ca5d88061969f34d7b", size = 414012, upload-time = "2025-10-02T14:34:28.409Z" },
- { url = "https://files.pythonhosted.org/packages/ba/b3/5a4241309217c5c876f156b10778f3ab3af7ba7e3259e6d5f5c7d0129eb2/xxhash-3.6.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:51312c768403d8540487dbbfb557454cfc55589bbde6424456951f7fcd4facb3", size = 191409, upload-time = "2025-10-02T14:34:29.696Z" },
- { url = "https://files.pythonhosted.org/packages/c0/01/99bfbc15fb9abb9a72b088c1d95219fc4782b7d01fc835bd5744d66dd0b8/xxhash-3.6.0-cp311-cp311-win32.whl", hash = "sha256:d1927a69feddc24c987b337ce81ac15c4720955b667fe9b588e02254b80446fd", size = 30574, upload-time = "2025-10-02T14:34:31.028Z" },
- { url = "https://files.pythonhosted.org/packages/65/79/9d24d7f53819fe301b231044ea362ce64e86c74f6e8c8e51320de248b3e5/xxhash-3.6.0-cp311-cp311-win_amd64.whl", hash = "sha256:26734cdc2d4ffe449b41d186bbeac416f704a482ed835d375a5c0cb02bc63fef", size = 31481, upload-time = "2025-10-02T14:34:32.062Z" },
- { url = "https://files.pythonhosted.org/packages/30/4e/15cd0e3e8772071344eab2961ce83f6e485111fed8beb491a3f1ce100270/xxhash-3.6.0-cp311-cp311-win_arm64.whl", hash = "sha256:d72f67ef8bf36e05f5b6c65e8524f265bd61071471cd4cf1d36743ebeeeb06b7", size = 27861, upload-time = "2025-10-02T14:34:33.555Z" },
{ url = "https://files.pythonhosted.org/packages/9a/07/d9412f3d7d462347e4511181dea65e47e0d0e16e26fbee2ea86a2aefb657/xxhash-3.6.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:01362c4331775398e7bb34e3ab403bc9ee9f7c497bc7dee6272114055277dd3c", size = 32744, upload-time = "2025-10-02T14:34:34.622Z" },
{ url = "https://files.pythonhosted.org/packages/79/35/0429ee11d035fc33abe32dca1b2b69e8c18d236547b9a9b72c1929189b9a/xxhash-3.6.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b7b2df81a23f8cb99656378e72501b2cb41b1827c0f5a86f87d6b06b69f9f204", size = 30816, upload-time = "2025-10-02T14:34:36.043Z" },
{ url = "https://files.pythonhosted.org/packages/b7/f2/57eb99aa0f7d98624c0932c5b9a170e1806406cdbcdb510546634a1359e0/xxhash-3.6.0-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:dc94790144e66b14f67b10ac8ed75b39ca47536bf8800eb7c24b50271ea0c490", size = 194035, upload-time = "2025-10-02T14:34:37.354Z" },
@@ -6380,137 +7083,110 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/9a/9a/c19c42c5b3f5a4aad748a6d5b4f23df3bed7ee5445accc65a0fb3ff03953/xxhash-3.6.0-cp314-cp314t-win32.whl", hash = "sha256:5851f033c3030dd95c086b4a36a2683c2ff4a799b23af60977188b057e467119", size = 31586, upload-time = "2025-10-02T14:36:15.603Z" },
{ url = "https://files.pythonhosted.org/packages/03/d6/4cc450345be9924fd5dc8c590ceda1db5b43a0a889587b0ae81a95511360/xxhash-3.6.0-cp314-cp314t-win_amd64.whl", hash = "sha256:0444e7967dac37569052d2409b00a8860c2135cff05502df4da80267d384849f", size = 32526, upload-time = "2025-10-02T14:36:16.708Z" },
{ url = "https://files.pythonhosted.org/packages/0f/c9/7243eb3f9eaabd1a88a5a5acadf06df2d83b100c62684b7425c6a11bcaa8/xxhash-3.6.0-cp314-cp314t-win_arm64.whl", hash = "sha256:bb79b1e63f6fd84ec778a4b1916dfe0a7c3fdb986c06addd5db3a0d413819d95", size = 28898, upload-time = "2025-10-02T14:36:17.843Z" },
- { url = "https://files.pythonhosted.org/packages/93/1e/8aec23647a34a249f62e2398c42955acd9b4c6ed5cf08cbea94dc46f78d2/xxhash-3.6.0-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:0f7b7e2ec26c1666ad5fc9dbfa426a6a3367ceaf79db5dd76264659d509d73b0", size = 30662, upload-time = "2025-10-02T14:37:01.743Z" },
- { url = "https://files.pythonhosted.org/packages/b8/0b/b14510b38ba91caf43006209db846a696ceea6a847a0c9ba0a5b1adc53d6/xxhash-3.6.0-pp311-pypy311_pp73-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:5dc1e14d14fa0f5789ec29a7062004b5933964bb9b02aae6622b8f530dc40296", size = 41056, upload-time = "2025-10-02T14:37:02.879Z" },
- { url = "https://files.pythonhosted.org/packages/50/55/15a7b8a56590e66ccd374bbfa3f9ffc45b810886c8c3b614e3f90bd2367c/xxhash-3.6.0-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:881b47fc47e051b37d94d13e7455131054b56749b91b508b0907eb07900d1c13", size = 36251, upload-time = "2025-10-02T14:37:04.44Z" },
- { url = "https://files.pythonhosted.org/packages/62/b2/5ac99a041a29e58e95f907876b04f7067a0242cb85b5f39e726153981503/xxhash-3.6.0-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c6dc31591899f5e5666f04cc2e529e69b4072827085c1ef15294d91a004bc1bd", size = 32481, upload-time = "2025-10-02T14:37:05.869Z" },
- { url = "https://files.pythonhosted.org/packages/7b/d9/8d95e906764a386a3d3b596f3c68bb63687dfca806373509f51ce8eea81f/xxhash-3.6.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:15e0dac10eb9309508bfc41f7f9deaa7755c69e35af835db9cb10751adebc35d", size = 31565, upload-time = "2025-10-02T14:37:06.966Z" },
]
[[package]]
name = "yarl"
-version = "1.22.0"
+version = "1.23.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "idna" },
{ name = "multidict" },
{ name = "propcache" },
]
-sdist = { url = "https://files.pythonhosted.org/packages/57/63/0c6ebca57330cd313f6102b16dd57ffaf3ec4c83403dcb45dbd15c6f3ea1/yarl-1.22.0.tar.gz", hash = "sha256:bebf8557577d4401ba8bd9ff33906f1376c877aa78d1fe216ad01b4d6745af71", size = 187169, upload-time = "2025-10-06T14:12:55.963Z" }
-wheels = [
- { url = "https://files.pythonhosted.org/packages/d1/43/a2204825342f37c337f5edb6637040fa14e365b2fcc2346960201d457579/yarl-1.22.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:c7bd6683587567e5a49ee6e336e0612bec8329be1b7d4c8af5687dcdeb67ee1e", size = 140517, upload-time = "2025-10-06T14:08:42.494Z" },
- { url = "https://files.pythonhosted.org/packages/44/6f/674f3e6f02266428c56f704cd2501c22f78e8b2eeb23f153117cc86fb28a/yarl-1.22.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:5cdac20da754f3a723cceea5b3448e1a2074866406adeb4ef35b469d089adb8f", size = 93495, upload-time = "2025-10-06T14:08:46.2Z" },
- { url = "https://files.pythonhosted.org/packages/b8/12/5b274d8a0f30c07b91b2f02cba69152600b47830fcfb465c108880fcee9c/yarl-1.22.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:07a524d84df0c10f41e3ee918846e1974aba4ec017f990dc735aad487a0bdfdf", size = 94400, upload-time = "2025-10-06T14:08:47.855Z" },
- { url = "https://files.pythonhosted.org/packages/e2/7f/df1b6949b1fa1aa9ff6de6e2631876ad4b73c4437822026e85d8acb56bb1/yarl-1.22.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e1b329cb8146d7b736677a2440e422eadd775d1806a81db2d4cded80a48efc1a", size = 347545, upload-time = "2025-10-06T14:08:49.683Z" },
- { url = "https://files.pythonhosted.org/packages/84/09/f92ed93bd6cd77872ab6c3462df45ca45cd058d8f1d0c9b4f54c1704429f/yarl-1.22.0-cp310-cp310-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:75976c6945d85dbb9ee6308cd7ff7b1fb9409380c82d6119bd778d8fcfe2931c", size = 319598, upload-time = "2025-10-06T14:08:51.215Z" },
- { url = "https://files.pythonhosted.org/packages/c3/97/ac3f3feae7d522cf7ccec3d340bb0b2b61c56cb9767923df62a135092c6b/yarl-1.22.0-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:80ddf7a5f8c86cb3eb4bc9028b07bbbf1f08a96c5c0bc1244be5e8fefcb94147", size = 363893, upload-time = "2025-10-06T14:08:53.144Z" },
- { url = "https://files.pythonhosted.org/packages/06/49/f3219097403b9c84a4d079b1d7bda62dd9b86d0d6e4428c02d46ab2c77fc/yarl-1.22.0-cp310-cp310-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d332fc2e3c94dad927f2112395772a4e4fedbcf8f80efc21ed7cdfae4d574fdb", size = 371240, upload-time = "2025-10-06T14:08:55.036Z" },
- { url = "https://files.pythonhosted.org/packages/35/9f/06b765d45c0e44e8ecf0fe15c9eacbbde342bb5b7561c46944f107bfb6c3/yarl-1.22.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0cf71bf877efeac18b38d3930594c0948c82b64547c1cf420ba48722fe5509f6", size = 346965, upload-time = "2025-10-06T14:08:56.722Z" },
- { url = "https://files.pythonhosted.org/packages/c5/69/599e7cea8d0fcb1694323b0db0dda317fa3162f7b90166faddecf532166f/yarl-1.22.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:663e1cadaddae26be034a6ab6072449a8426ddb03d500f43daf952b74553bba0", size = 342026, upload-time = "2025-10-06T14:08:58.563Z" },
- { url = "https://files.pythonhosted.org/packages/95/6f/9dfd12c8bc90fea9eab39832ee32ea48f8e53d1256252a77b710c065c89f/yarl-1.22.0-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:6dcbb0829c671f305be48a7227918cfcd11276c2d637a8033a99a02b67bf9eda", size = 335637, upload-time = "2025-10-06T14:09:00.506Z" },
- { url = "https://files.pythonhosted.org/packages/57/2e/34c5b4eb9b07e16e873db5b182c71e5f06f9b5af388cdaa97736d79dd9a6/yarl-1.22.0-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:f0d97c18dfd9a9af4490631905a3f131a8e4c9e80a39353919e2cfed8f00aedc", size = 359082, upload-time = "2025-10-06T14:09:01.936Z" },
- { url = "https://files.pythonhosted.org/packages/31/71/fa7e10fb772d273aa1f096ecb8ab8594117822f683bab7d2c5a89914c92a/yarl-1.22.0-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:437840083abe022c978470b942ff832c3940b2ad3734d424b7eaffcd07f76737", size = 357811, upload-time = "2025-10-06T14:09:03.445Z" },
- { url = "https://files.pythonhosted.org/packages/26/da/11374c04e8e1184a6a03cf9c8f5688d3e5cec83ed6f31ad3481b3207f709/yarl-1.22.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:a899cbd98dce6f5d8de1aad31cb712ec0a530abc0a86bd6edaa47c1090138467", size = 351223, upload-time = "2025-10-06T14:09:05.401Z" },
- { url = "https://files.pythonhosted.org/packages/82/8f/e2d01f161b0c034a30410e375e191a5d27608c1f8693bab1a08b089ca096/yarl-1.22.0-cp310-cp310-win32.whl", hash = "sha256:595697f68bd1f0c1c159fcb97b661fc9c3f5db46498043555d04805430e79bea", size = 82118, upload-time = "2025-10-06T14:09:11.148Z" },
- { url = "https://files.pythonhosted.org/packages/62/46/94c76196642dbeae634c7a61ba3da88cd77bed875bf6e4a8bed037505aa6/yarl-1.22.0-cp310-cp310-win_amd64.whl", hash = "sha256:cb95a9b1adaa48e41815a55ae740cfda005758104049a640a398120bf02515ca", size = 86852, upload-time = "2025-10-06T14:09:12.958Z" },
- { url = "https://files.pythonhosted.org/packages/af/af/7df4f179d3b1a6dcb9a4bd2ffbc67642746fcafdb62580e66876ce83fff4/yarl-1.22.0-cp310-cp310-win_arm64.whl", hash = "sha256:b85b982afde6df99ecc996990d4ad7ccbdbb70e2a4ba4de0aecde5922ba98a0b", size = 82012, upload-time = "2025-10-06T14:09:14.664Z" },
- { url = "https://files.pythonhosted.org/packages/4d/27/5ab13fc84c76a0250afd3d26d5936349a35be56ce5785447d6c423b26d92/yarl-1.22.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:1ab72135b1f2db3fed3997d7e7dc1b80573c67138023852b6efb336a5eae6511", size = 141607, upload-time = "2025-10-06T14:09:16.298Z" },
- { url = "https://files.pythonhosted.org/packages/6a/a1/d065d51d02dc02ce81501d476b9ed2229d9a990818332242a882d5d60340/yarl-1.22.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:669930400e375570189492dc8d8341301578e8493aec04aebc20d4717f899dd6", size = 94027, upload-time = "2025-10-06T14:09:17.786Z" },
- { url = "https://files.pythonhosted.org/packages/c1/da/8da9f6a53f67b5106ffe902c6fa0164e10398d4e150d85838b82f424072a/yarl-1.22.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:792a2af6d58177ef7c19cbf0097aba92ca1b9cb3ffdd9c7470e156c8f9b5e028", size = 94963, upload-time = "2025-10-06T14:09:19.662Z" },
- { url = "https://files.pythonhosted.org/packages/68/fe/2c1f674960c376e29cb0bec1249b117d11738db92a6ccc4a530b972648db/yarl-1.22.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3ea66b1c11c9150f1372f69afb6b8116f2dd7286f38e14ea71a44eee9ec51b9d", size = 368406, upload-time = "2025-10-06T14:09:21.402Z" },
- { url = "https://files.pythonhosted.org/packages/95/26/812a540e1c3c6418fec60e9bbd38e871eaba9545e94fa5eff8f4a8e28e1e/yarl-1.22.0-cp311-cp311-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3e2daa88dc91870215961e96a039ec73e4937da13cf77ce17f9cad0c18df3503", size = 336581, upload-time = "2025-10-06T14:09:22.98Z" },
- { url = "https://files.pythonhosted.org/packages/0b/f5/5777b19e26fdf98563985e481f8be3d8a39f8734147a6ebf459d0dab5a6b/yarl-1.22.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ba440ae430c00eee41509353628600212112cd5018d5def7e9b05ea7ac34eb65", size = 388924, upload-time = "2025-10-06T14:09:24.655Z" },
- { url = "https://files.pythonhosted.org/packages/86/08/24bd2477bd59c0bbd994fe1d93b126e0472e4e3df5a96a277b0a55309e89/yarl-1.22.0-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:e6438cc8f23a9c1478633d216b16104a586b9761db62bfacb6425bac0a36679e", size = 392890, upload-time = "2025-10-06T14:09:26.617Z" },
- { url = "https://files.pythonhosted.org/packages/46/00/71b90ed48e895667ecfb1eaab27c1523ee2fa217433ed77a73b13205ca4b/yarl-1.22.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4c52a6e78aef5cf47a98ef8e934755abf53953379b7d53e68b15ff4420e6683d", size = 365819, upload-time = "2025-10-06T14:09:28.544Z" },
- { url = "https://files.pythonhosted.org/packages/30/2d/f715501cae832651d3282387c6a9236cd26bd00d0ff1e404b3dc52447884/yarl-1.22.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:3b06bcadaac49c70f4c88af4ffcfbe3dc155aab3163e75777818092478bcbbe7", size = 363601, upload-time = "2025-10-06T14:09:30.568Z" },
- { url = "https://files.pythonhosted.org/packages/f8/f9/a678c992d78e394e7126ee0b0e4e71bd2775e4334d00a9278c06a6cce96a/yarl-1.22.0-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:6944b2dc72c4d7f7052683487e3677456050ff77fcf5e6204e98caf785ad1967", size = 358072, upload-time = "2025-10-06T14:09:32.528Z" },
- { url = "https://files.pythonhosted.org/packages/2c/d1/b49454411a60edb6fefdcad4f8e6dbba7d8019e3a508a1c5836cba6d0781/yarl-1.22.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:d5372ca1df0f91a86b047d1277c2aaf1edb32d78bbcefffc81b40ffd18f027ed", size = 385311, upload-time = "2025-10-06T14:09:34.634Z" },
- { url = "https://files.pythonhosted.org/packages/87/e5/40d7a94debb8448c7771a916d1861d6609dddf7958dc381117e7ba36d9e8/yarl-1.22.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:51af598701f5299012b8416486b40fceef8c26fc87dc6d7d1f6fc30609ea0aa6", size = 381094, upload-time = "2025-10-06T14:09:36.268Z" },
- { url = "https://files.pythonhosted.org/packages/35/d8/611cc282502381ad855448643e1ad0538957fc82ae83dfe7762c14069e14/yarl-1.22.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:b266bd01fedeffeeac01a79ae181719ff848a5a13ce10075adbefc8f1daee70e", size = 370944, upload-time = "2025-10-06T14:09:37.872Z" },
- { url = "https://files.pythonhosted.org/packages/2d/df/fadd00fb1c90e1a5a8bd731fa3d3de2e165e5a3666a095b04e31b04d9cb6/yarl-1.22.0-cp311-cp311-win32.whl", hash = "sha256:a9b1ba5610a4e20f655258d5a1fdc7ebe3d837bb0e45b581398b99eb98b1f5ca", size = 81804, upload-time = "2025-10-06T14:09:39.359Z" },
- { url = "https://files.pythonhosted.org/packages/b5/f7/149bb6f45f267cb5c074ac40c01c6b3ea6d8a620d34b337f6321928a1b4d/yarl-1.22.0-cp311-cp311-win_amd64.whl", hash = "sha256:078278b9b0b11568937d9509b589ee83ef98ed6d561dfe2020e24a9fd08eaa2b", size = 86858, upload-time = "2025-10-06T14:09:41.068Z" },
- { url = "https://files.pythonhosted.org/packages/2b/13/88b78b93ad3f2f0b78e13bfaaa24d11cbc746e93fe76d8c06bf139615646/yarl-1.22.0-cp311-cp311-win_arm64.whl", hash = "sha256:b6a6f620cfe13ccec221fa312139135166e47ae169f8253f72a0abc0dae94376", size = 81637, upload-time = "2025-10-06T14:09:42.712Z" },
- { url = "https://files.pythonhosted.org/packages/75/ff/46736024fee3429b80a165a732e38e5d5a238721e634ab41b040d49f8738/yarl-1.22.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:e340382d1afa5d32b892b3ff062436d592ec3d692aeea3bef3a5cfe11bbf8c6f", size = 142000, upload-time = "2025-10-06T14:09:44.631Z" },
- { url = "https://files.pythonhosted.org/packages/5a/9a/b312ed670df903145598914770eb12de1bac44599549b3360acc96878df8/yarl-1.22.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:f1e09112a2c31ffe8d80be1b0988fa6a18c5d5cad92a9ffbb1c04c91bfe52ad2", size = 94338, upload-time = "2025-10-06T14:09:46.372Z" },
- { url = "https://files.pythonhosted.org/packages/ba/f5/0601483296f09c3c65e303d60c070a5c19fcdbc72daa061e96170785bc7d/yarl-1.22.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:939fe60db294c786f6b7c2d2e121576628468f65453d86b0fe36cb52f987bd74", size = 94909, upload-time = "2025-10-06T14:09:48.648Z" },
- { url = "https://files.pythonhosted.org/packages/60/41/9a1fe0b73dbcefce72e46cf149b0e0a67612d60bfc90fb59c2b2efdfbd86/yarl-1.22.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e1651bf8e0398574646744c1885a41198eba53dc8a9312b954073f845c90a8df", size = 372940, upload-time = "2025-10-06T14:09:50.089Z" },
- { url = "https://files.pythonhosted.org/packages/17/7a/795cb6dfee561961c30b800f0ed616b923a2ec6258b5def2a00bf8231334/yarl-1.22.0-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:b8a0588521a26bf92a57a1705b77b8b59044cdceccac7151bd8d229e66b8dedb", size = 345825, upload-time = "2025-10-06T14:09:52.142Z" },
- { url = "https://files.pythonhosted.org/packages/d7/93/a58f4d596d2be2ae7bab1a5846c4d270b894958845753b2c606d666744d3/yarl-1.22.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:42188e6a615c1a75bcaa6e150c3fe8f3e8680471a6b10150c5f7e83f47cc34d2", size = 386705, upload-time = "2025-10-06T14:09:54.128Z" },
- { url = "https://files.pythonhosted.org/packages/61/92/682279d0e099d0e14d7fd2e176bd04f48de1484f56546a3e1313cd6c8e7c/yarl-1.22.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f6d2cb59377d99718913ad9a151030d6f83ef420a2b8f521d94609ecc106ee82", size = 396518, upload-time = "2025-10-06T14:09:55.762Z" },
- { url = "https://files.pythonhosted.org/packages/db/0f/0d52c98b8a885aeda831224b78f3be7ec2e1aa4a62091f9f9188c3c65b56/yarl-1.22.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:50678a3b71c751d58d7908edc96d332af328839eea883bb554a43f539101277a", size = 377267, upload-time = "2025-10-06T14:09:57.958Z" },
- { url = "https://files.pythonhosted.org/packages/22/42/d2685e35908cbeaa6532c1fc73e89e7f2efb5d8a7df3959ea8e37177c5a3/yarl-1.22.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1e8fbaa7cec507aa24ea27a01456e8dd4b6fab829059b69844bd348f2d467124", size = 365797, upload-time = "2025-10-06T14:09:59.527Z" },
- { url = "https://files.pythonhosted.org/packages/a2/83/cf8c7bcc6355631762f7d8bdab920ad09b82efa6b722999dfb05afa6cfac/yarl-1.22.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:433885ab5431bc3d3d4f2f9bd15bfa1614c522b0f1405d62c4f926ccd69d04fa", size = 365535, upload-time = "2025-10-06T14:10:01.139Z" },
- { url = "https://files.pythonhosted.org/packages/25/e1/5302ff9b28f0c59cac913b91fe3f16c59a033887e57ce9ca5d41a3a94737/yarl-1.22.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:b790b39c7e9a4192dc2e201a282109ed2985a1ddbd5ac08dc56d0e121400a8f7", size = 382324, upload-time = "2025-10-06T14:10:02.756Z" },
- { url = "https://files.pythonhosted.org/packages/bf/cd/4617eb60f032f19ae3a688dc990d8f0d89ee0ea378b61cac81ede3e52fae/yarl-1.22.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:31f0b53913220599446872d757257be5898019c85e7971599065bc55065dc99d", size = 383803, upload-time = "2025-10-06T14:10:04.552Z" },
- { url = "https://files.pythonhosted.org/packages/59/65/afc6e62bb506a319ea67b694551dab4a7e6fb7bf604e9bd9f3e11d575fec/yarl-1.22.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:a49370e8f711daec68d09b821a34e1167792ee2d24d405cbc2387be4f158b520", size = 374220, upload-time = "2025-10-06T14:10:06.489Z" },
- { url = "https://files.pythonhosted.org/packages/e7/3d/68bf18d50dc674b942daec86a9ba922d3113d8399b0e52b9897530442da2/yarl-1.22.0-cp312-cp312-win32.whl", hash = "sha256:70dfd4f241c04bd9239d53b17f11e6ab672b9f1420364af63e8531198e3f5fe8", size = 81589, upload-time = "2025-10-06T14:10:09.254Z" },
- { url = "https://files.pythonhosted.org/packages/c8/9a/6ad1a9b37c2f72874f93e691b2e7ecb6137fb2b899983125db4204e47575/yarl-1.22.0-cp312-cp312-win_amd64.whl", hash = "sha256:8884d8b332a5e9b88e23f60bb166890009429391864c685e17bd73a9eda9105c", size = 87213, upload-time = "2025-10-06T14:10:11.369Z" },
- { url = "https://files.pythonhosted.org/packages/44/c5/c21b562d1680a77634d748e30c653c3ca918beb35555cff24986fff54598/yarl-1.22.0-cp312-cp312-win_arm64.whl", hash = "sha256:ea70f61a47f3cc93bdf8b2f368ed359ef02a01ca6393916bc8ff877427181e74", size = 81330, upload-time = "2025-10-06T14:10:13.112Z" },
- { url = "https://files.pythonhosted.org/packages/ea/f3/d67de7260456ee105dc1d162d43a019ecad6b91e2f51809d6cddaa56690e/yarl-1.22.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:8dee9c25c74997f6a750cd317b8ca63545169c098faee42c84aa5e506c819b53", size = 139980, upload-time = "2025-10-06T14:10:14.601Z" },
- { url = "https://files.pythonhosted.org/packages/01/88/04d98af0b47e0ef42597b9b28863b9060bb515524da0a65d5f4db160b2d5/yarl-1.22.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:01e73b85a5434f89fc4fe27dcda2aff08ddf35e4d47bbbea3bdcd25321af538a", size = 93424, upload-time = "2025-10-06T14:10:16.115Z" },
- { url = "https://files.pythonhosted.org/packages/18/91/3274b215fd8442a03975ce6bee5fe6aa57a8326b29b9d3d56234a1dca244/yarl-1.22.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:22965c2af250d20c873cdbee8ff958fb809940aeb2e74ba5f20aaf6b7ac8c70c", size = 93821, upload-time = "2025-10-06T14:10:17.993Z" },
- { url = "https://files.pythonhosted.org/packages/61/3a/caf4e25036db0f2da4ca22a353dfeb3c9d3c95d2761ebe9b14df8fc16eb0/yarl-1.22.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b4f15793aa49793ec8d1c708ab7f9eded1aa72edc5174cae703651555ed1b601", size = 373243, upload-time = "2025-10-06T14:10:19.44Z" },
- { url = "https://files.pythonhosted.org/packages/6e/9e/51a77ac7516e8e7803b06e01f74e78649c24ee1021eca3d6a739cb6ea49c/yarl-1.22.0-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:e5542339dcf2747135c5c85f68680353d5cb9ffd741c0f2e8d832d054d41f35a", size = 342361, upload-time = "2025-10-06T14:10:21.124Z" },
- { url = "https://files.pythonhosted.org/packages/d4/f8/33b92454789dde8407f156c00303e9a891f1f51a0330b0fad7c909f87692/yarl-1.22.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5c401e05ad47a75869c3ab3e35137f8468b846770587e70d71e11de797d113df", size = 387036, upload-time = "2025-10-06T14:10:22.902Z" },
- { url = "https://files.pythonhosted.org/packages/d9/9a/c5db84ea024f76838220280f732970aa4ee154015d7f5c1bfb60a267af6f/yarl-1.22.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:243dda95d901c733f5b59214d28b0120893d91777cb8aa043e6ef059d3cddfe2", size = 397671, upload-time = "2025-10-06T14:10:24.523Z" },
- { url = "https://files.pythonhosted.org/packages/11/c9/cd8538dc2e7727095e0c1d867bad1e40c98f37763e6d995c1939f5fdc7b1/yarl-1.22.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bec03d0d388060058f5d291a813f21c011041938a441c593374da6077fe21b1b", size = 377059, upload-time = "2025-10-06T14:10:26.406Z" },
- { url = "https://files.pythonhosted.org/packages/a1/b9/ab437b261702ced75122ed78a876a6dec0a1b0f5e17a4ac7a9a2482d8abe/yarl-1.22.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:b0748275abb8c1e1e09301ee3cf90c8a99678a4e92e4373705f2a2570d581273", size = 365356, upload-time = "2025-10-06T14:10:28.461Z" },
- { url = "https://files.pythonhosted.org/packages/b2/9d/8e1ae6d1d008a9567877b08f0ce4077a29974c04c062dabdb923ed98e6fe/yarl-1.22.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:47fdb18187e2a4e18fda2c25c05d8251a9e4a521edaed757fef033e7d8498d9a", size = 361331, upload-time = "2025-10-06T14:10:30.541Z" },
- { url = "https://files.pythonhosted.org/packages/ca/5a/09b7be3905962f145b73beb468cdd53db8aa171cf18c80400a54c5b82846/yarl-1.22.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:c7044802eec4524fde550afc28edda0dd5784c4c45f0be151a2d3ba017daca7d", size = 382590, upload-time = "2025-10-06T14:10:33.352Z" },
- { url = "https://files.pythonhosted.org/packages/aa/7f/59ec509abf90eda5048b0bc3e2d7b5099dffdb3e6b127019895ab9d5ef44/yarl-1.22.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:139718f35149ff544caba20fce6e8a2f71f1e39b92c700d8438a0b1d2a631a02", size = 385316, upload-time = "2025-10-06T14:10:35.034Z" },
- { url = "https://files.pythonhosted.org/packages/e5/84/891158426bc8036bfdfd862fabd0e0fa25df4176ec793e447f4b85cf1be4/yarl-1.22.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e1b51bebd221006d3d2f95fbe124b22b247136647ae5dcc8c7acafba66e5ee67", size = 374431, upload-time = "2025-10-06T14:10:37.76Z" },
- { url = "https://files.pythonhosted.org/packages/bb/49/03da1580665baa8bef5e8ed34c6df2c2aca0a2f28bf397ed238cc1bbc6f2/yarl-1.22.0-cp313-cp313-win32.whl", hash = "sha256:d3e32536234a95f513bd374e93d717cf6b2231a791758de6c509e3653f234c95", size = 81555, upload-time = "2025-10-06T14:10:39.649Z" },
- { url = "https://files.pythonhosted.org/packages/9a/ee/450914ae11b419eadd067c6183ae08381cfdfcb9798b90b2b713bbebddda/yarl-1.22.0-cp313-cp313-win_amd64.whl", hash = "sha256:47743b82b76d89a1d20b83e60d5c20314cbd5ba2befc9cda8f28300c4a08ed4d", size = 86965, upload-time = "2025-10-06T14:10:41.313Z" },
- { url = "https://files.pythonhosted.org/packages/98/4d/264a01eae03b6cf629ad69bae94e3b0e5344741e929073678e84bf7a3e3b/yarl-1.22.0-cp313-cp313-win_arm64.whl", hash = "sha256:5d0fcda9608875f7d052eff120c7a5da474a6796fe4d83e152e0e4d42f6d1a9b", size = 81205, upload-time = "2025-10-06T14:10:43.167Z" },
- { url = "https://files.pythonhosted.org/packages/88/fc/6908f062a2f77b5f9f6d69cecb1747260831ff206adcbc5b510aff88df91/yarl-1.22.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:719ae08b6972befcba4310e49edb1161a88cdd331e3a694b84466bd938a6ab10", size = 146209, upload-time = "2025-10-06T14:10:44.643Z" },
- { url = "https://files.pythonhosted.org/packages/65/47/76594ae8eab26210b4867be6f49129861ad33da1f1ebdf7051e98492bf62/yarl-1.22.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:47d8a5c446df1c4db9d21b49619ffdba90e77c89ec6e283f453856c74b50b9e3", size = 95966, upload-time = "2025-10-06T14:10:46.554Z" },
- { url = "https://files.pythonhosted.org/packages/ab/ce/05e9828a49271ba6b5b038b15b3934e996980dd78abdfeb52a04cfb9467e/yarl-1.22.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:cfebc0ac8333520d2d0423cbbe43ae43c8838862ddb898f5ca68565e395516e9", size = 97312, upload-time = "2025-10-06T14:10:48.007Z" },
- { url = "https://files.pythonhosted.org/packages/d1/c5/7dffad5e4f2265b29c9d7ec869c369e4223166e4f9206fc2243ee9eea727/yarl-1.22.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4398557cbf484207df000309235979c79c4356518fd5c99158c7d38203c4da4f", size = 361967, upload-time = "2025-10-06T14:10:49.997Z" },
- { url = "https://files.pythonhosted.org/packages/50/b2/375b933c93a54bff7fc041e1a6ad2c0f6f733ffb0c6e642ce56ee3b39970/yarl-1.22.0-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:2ca6fd72a8cd803be290d42f2dec5cdcd5299eeb93c2d929bf060ad9efaf5de0", size = 323949, upload-time = "2025-10-06T14:10:52.004Z" },
- { url = "https://files.pythonhosted.org/packages/66/50/bfc2a29a1d78644c5a7220ce2f304f38248dc94124a326794e677634b6cf/yarl-1.22.0-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ca1f59c4e1ab6e72f0a23c13fca5430f889634166be85dbf1013683e49e3278e", size = 361818, upload-time = "2025-10-06T14:10:54.078Z" },
- { url = "https://files.pythonhosted.org/packages/46/96/f3941a46af7d5d0f0498f86d71275696800ddcdd20426298e572b19b91ff/yarl-1.22.0-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:6c5010a52015e7c70f86eb967db0f37f3c8bd503a695a49f8d45700144667708", size = 372626, upload-time = "2025-10-06T14:10:55.767Z" },
- { url = "https://files.pythonhosted.org/packages/c1/42/8b27c83bb875cd89448e42cd627e0fb971fa1675c9ec546393d18826cb50/yarl-1.22.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d7672ecf7557476642c88497c2f8d8542f8e36596e928e9bcba0e42e1e7d71f", size = 341129, upload-time = "2025-10-06T14:10:57.985Z" },
- { url = "https://files.pythonhosted.org/packages/49/36/99ca3122201b382a3cf7cc937b95235b0ac944f7e9f2d5331d50821ed352/yarl-1.22.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:3b7c88eeef021579d600e50363e0b6ee4f7f6f728cd3486b9d0f3ee7b946398d", size = 346776, upload-time = "2025-10-06T14:10:59.633Z" },
- { url = "https://files.pythonhosted.org/packages/85/b4/47328bf996acd01a4c16ef9dcd2f59c969f495073616586f78cd5f2efb99/yarl-1.22.0-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:f4afb5c34f2c6fecdcc182dfcfc6af6cccf1aa923eed4d6a12e9d96904e1a0d8", size = 334879, upload-time = "2025-10-06T14:11:01.454Z" },
- { url = "https://files.pythonhosted.org/packages/c2/ad/b77d7b3f14a4283bffb8e92c6026496f6de49751c2f97d4352242bba3990/yarl-1.22.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:59c189e3e99a59cf8d83cbb31d4db02d66cda5a1a4374e8a012b51255341abf5", size = 350996, upload-time = "2025-10-06T14:11:03.452Z" },
- { url = "https://files.pythonhosted.org/packages/81/c8/06e1d69295792ba54d556f06686cbd6a7ce39c22307100e3fb4a2c0b0a1d/yarl-1.22.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:5a3bf7f62a289fa90f1990422dc8dff5a458469ea71d1624585ec3a4c8d6960f", size = 356047, upload-time = "2025-10-06T14:11:05.115Z" },
- { url = "https://files.pythonhosted.org/packages/4b/b8/4c0e9e9f597074b208d18cef227d83aac36184bfbc6eab204ea55783dbc5/yarl-1.22.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:de6b9a04c606978fdfe72666fa216ffcf2d1a9f6a381058d4378f8d7b1e5de62", size = 342947, upload-time = "2025-10-06T14:11:08.137Z" },
- { url = "https://files.pythonhosted.org/packages/e0/e5/11f140a58bf4c6ad7aca69a892bff0ee638c31bea4206748fc0df4ebcb3a/yarl-1.22.0-cp313-cp313t-win32.whl", hash = "sha256:1834bb90991cc2999f10f97f5f01317f99b143284766d197e43cd5b45eb18d03", size = 86943, upload-time = "2025-10-06T14:11:10.284Z" },
- { url = "https://files.pythonhosted.org/packages/31/74/8b74bae38ed7fe6793d0c15a0c8207bbb819cf287788459e5ed230996cdd/yarl-1.22.0-cp313-cp313t-win_amd64.whl", hash = "sha256:ff86011bd159a9d2dfc89c34cfd8aff12875980e3bd6a39ff097887520e60249", size = 93715, upload-time = "2025-10-06T14:11:11.739Z" },
- { url = "https://files.pythonhosted.org/packages/69/66/991858aa4b5892d57aef7ee1ba6b4d01ec3b7eb3060795d34090a3ca3278/yarl-1.22.0-cp313-cp313t-win_arm64.whl", hash = "sha256:7861058d0582b847bc4e3a4a4c46828a410bca738673f35a29ba3ca5db0b473b", size = 83857, upload-time = "2025-10-06T14:11:13.586Z" },
- { url = "https://files.pythonhosted.org/packages/46/b3/e20ef504049f1a1c54a814b4b9bed96d1ac0e0610c3b4da178f87209db05/yarl-1.22.0-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:34b36c2c57124530884d89d50ed2c1478697ad7473efd59cfd479945c95650e4", size = 140520, upload-time = "2025-10-06T14:11:15.465Z" },
- { url = "https://files.pythonhosted.org/packages/e4/04/3532d990fdbab02e5ede063676b5c4260e7f3abea2151099c2aa745acc4c/yarl-1.22.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:0dd9a702591ca2e543631c2a017e4a547e38a5c0f29eece37d9097e04a7ac683", size = 93504, upload-time = "2025-10-06T14:11:17.106Z" },
- { url = "https://files.pythonhosted.org/packages/11/63/ff458113c5c2dac9a9719ac68ee7c947cb621432bcf28c9972b1c0e83938/yarl-1.22.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:594fcab1032e2d2cc3321bb2e51271e7cd2b516c7d9aee780ece81b07ff8244b", size = 94282, upload-time = "2025-10-06T14:11:19.064Z" },
- { url = "https://files.pythonhosted.org/packages/a7/bc/315a56aca762d44a6aaaf7ad253f04d996cb6b27bad34410f82d76ea8038/yarl-1.22.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f3d7a87a78d46a2e3d5b72587ac14b4c16952dd0887dbb051451eceac774411e", size = 372080, upload-time = "2025-10-06T14:11:20.996Z" },
- { url = "https://files.pythonhosted.org/packages/3f/3f/08e9b826ec2e099ea6e7c69a61272f4f6da62cb5b1b63590bb80ca2e4a40/yarl-1.22.0-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:852863707010316c973162e703bddabec35e8757e67fcb8ad58829de1ebc8590", size = 338696, upload-time = "2025-10-06T14:11:22.847Z" },
- { url = "https://files.pythonhosted.org/packages/e3/9f/90360108e3b32bd76789088e99538febfea24a102380ae73827f62073543/yarl-1.22.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:131a085a53bfe839a477c0845acf21efc77457ba2bcf5899618136d64f3303a2", size = 387121, upload-time = "2025-10-06T14:11:24.889Z" },
- { url = "https://files.pythonhosted.org/packages/98/92/ab8d4657bd5b46a38094cfaea498f18bb70ce6b63508fd7e909bd1f93066/yarl-1.22.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:078a8aefd263f4d4f923a9677b942b445a2be970ca24548a8102689a3a8ab8da", size = 394080, upload-time = "2025-10-06T14:11:27.307Z" },
- { url = "https://files.pythonhosted.org/packages/f5/e7/d8c5a7752fef68205296201f8ec2bf718f5c805a7a7e9880576c67600658/yarl-1.22.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bca03b91c323036913993ff5c738d0842fc9c60c4648e5c8d98331526df89784", size = 372661, upload-time = "2025-10-06T14:11:29.387Z" },
- { url = "https://files.pythonhosted.org/packages/b6/2e/f4d26183c8db0bb82d491b072f3127fb8c381a6206a3a56332714b79b751/yarl-1.22.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:68986a61557d37bb90d3051a45b91fa3d5c516d177dfc6dd6f2f436a07ff2b6b", size = 364645, upload-time = "2025-10-06T14:11:31.423Z" },
- { url = "https://files.pythonhosted.org/packages/80/7c/428e5812e6b87cd00ee8e898328a62c95825bf37c7fa87f0b6bb2ad31304/yarl-1.22.0-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:4792b262d585ff0dff6bcb787f8492e40698443ec982a3568c2096433660c694", size = 355361, upload-time = "2025-10-06T14:11:33.055Z" },
- { url = "https://files.pythonhosted.org/packages/ec/2a/249405fd26776f8b13c067378ef4d7dd49c9098d1b6457cdd152a99e96a9/yarl-1.22.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:ebd4549b108d732dba1d4ace67614b9545b21ece30937a63a65dd34efa19732d", size = 381451, upload-time = "2025-10-06T14:11:35.136Z" },
- { url = "https://files.pythonhosted.org/packages/67/a8/fb6b1adbe98cf1e2dd9fad71003d3a63a1bc22459c6e15f5714eb9323b93/yarl-1.22.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:f87ac53513d22240c7d59203f25cc3beac1e574c6cd681bbfd321987b69f95fd", size = 383814, upload-time = "2025-10-06T14:11:37.094Z" },
- { url = "https://files.pythonhosted.org/packages/d9/f9/3aa2c0e480fb73e872ae2814c43bc1e734740bb0d54e8cb2a95925f98131/yarl-1.22.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:22b029f2881599e2f1b06f8f1db2ee63bd309e2293ba2d566e008ba12778b8da", size = 370799, upload-time = "2025-10-06T14:11:38.83Z" },
- { url = "https://files.pythonhosted.org/packages/50/3c/af9dba3b8b5eeb302f36f16f92791f3ea62e3f47763406abf6d5a4a3333b/yarl-1.22.0-cp314-cp314-win32.whl", hash = "sha256:6a635ea45ba4ea8238463b4f7d0e721bad669f80878b7bfd1f89266e2ae63da2", size = 82990, upload-time = "2025-10-06T14:11:40.624Z" },
- { url = "https://files.pythonhosted.org/packages/ac/30/ac3a0c5bdc1d6efd1b41fa24d4897a4329b3b1e98de9449679dd327af4f0/yarl-1.22.0-cp314-cp314-win_amd64.whl", hash = "sha256:0d6e6885777af0f110b0e5d7e5dda8b704efed3894da26220b7f3d887b839a79", size = 88292, upload-time = "2025-10-06T14:11:42.578Z" },
- { url = "https://files.pythonhosted.org/packages/df/0a/227ab4ff5b998a1b7410abc7b46c9b7a26b0ca9e86c34ba4b8d8bc7c63d5/yarl-1.22.0-cp314-cp314-win_arm64.whl", hash = "sha256:8218f4e98d3c10d683584cb40f0424f4b9fd6e95610232dd75e13743b070ee33", size = 82888, upload-time = "2025-10-06T14:11:44.863Z" },
- { url = "https://files.pythonhosted.org/packages/06/5e/a15eb13db90abd87dfbefb9760c0f3f257ac42a5cac7e75dbc23bed97a9f/yarl-1.22.0-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:45c2842ff0e0d1b35a6bf1cd6c690939dacb617a70827f715232b2e0494d55d1", size = 146223, upload-time = "2025-10-06T14:11:46.796Z" },
- { url = "https://files.pythonhosted.org/packages/18/82/9665c61910d4d84f41a5bf6837597c89e665fa88aa4941080704645932a9/yarl-1.22.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:d947071e6ebcf2e2bee8fce76e10faca8f7a14808ca36a910263acaacef08eca", size = 95981, upload-time = "2025-10-06T14:11:48.845Z" },
- { url = "https://files.pythonhosted.org/packages/5d/9a/2f65743589809af4d0a6d3aa749343c4b5f4c380cc24a8e94a3c6625a808/yarl-1.22.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:334b8721303e61b00019474cc103bdac3d7b1f65e91f0bfedeec2d56dfe74b53", size = 97303, upload-time = "2025-10-06T14:11:50.897Z" },
- { url = "https://files.pythonhosted.org/packages/b0/ab/5b13d3e157505c43c3b43b5a776cbf7b24a02bc4cccc40314771197e3508/yarl-1.22.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1e7ce67c34138a058fd092f67d07a72b8e31ff0c9236e751957465a24b28910c", size = 361820, upload-time = "2025-10-06T14:11:52.549Z" },
- { url = "https://files.pythonhosted.org/packages/fb/76/242a5ef4677615cf95330cfc1b4610e78184400699bdda0acb897ef5e49a/yarl-1.22.0-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:d77e1b2c6d04711478cb1c4ab90db07f1609ccf06a287d5607fcd90dc9863acf", size = 323203, upload-time = "2025-10-06T14:11:54.225Z" },
- { url = "https://files.pythonhosted.org/packages/8c/96/475509110d3f0153b43d06164cf4195c64d16999e0c7e2d8a099adcd6907/yarl-1.22.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c4647674b6150d2cae088fc07de2738a84b8bcedebef29802cf0b0a82ab6face", size = 363173, upload-time = "2025-10-06T14:11:56.069Z" },
- { url = "https://files.pythonhosted.org/packages/c9/66/59db471aecfbd559a1fd48aedd954435558cd98c7d0da8b03cc6c140a32c/yarl-1.22.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:efb07073be061c8f79d03d04139a80ba33cbd390ca8f0297aae9cce6411e4c6b", size = 373562, upload-time = "2025-10-06T14:11:58.783Z" },
- { url = "https://files.pythonhosted.org/packages/03/1f/c5d94abc91557384719da10ff166b916107c1b45e4d0423a88457071dd88/yarl-1.22.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e51ac5435758ba97ad69617e13233da53908beccc6cfcd6c34bbed8dcbede486", size = 339828, upload-time = "2025-10-06T14:12:00.686Z" },
- { url = "https://files.pythonhosted.org/packages/5f/97/aa6a143d3afba17b6465733681c70cf175af89f76ec8d9286e08437a7454/yarl-1.22.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:33e32a0dd0c8205efa8e83d04fc9f19313772b78522d1bdc7d9aed706bfd6138", size = 347551, upload-time = "2025-10-06T14:12:02.628Z" },
- { url = "https://files.pythonhosted.org/packages/43/3c/45a2b6d80195959239a7b2a8810506d4eea5487dce61c2a3393e7fc3c52e/yarl-1.22.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:bf4a21e58b9cde0e401e683ebd00f6ed30a06d14e93f7c8fd059f8b6e8f87b6a", size = 334512, upload-time = "2025-10-06T14:12:04.871Z" },
- { url = "https://files.pythonhosted.org/packages/86/a0/c2ab48d74599c7c84cb104ebd799c5813de252bea0f360ffc29d270c2caa/yarl-1.22.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:e4b582bab49ac33c8deb97e058cd67c2c50dac0dd134874106d9c774fd272529", size = 352400, upload-time = "2025-10-06T14:12:06.624Z" },
- { url = "https://files.pythonhosted.org/packages/32/75/f8919b2eafc929567d3d8411f72bdb1a2109c01caaab4ebfa5f8ffadc15b/yarl-1.22.0-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:0b5bcc1a9c4839e7e30b7b30dd47fe5e7e44fb7054ec29b5bb8d526aa1041093", size = 357140, upload-time = "2025-10-06T14:12:08.362Z" },
- { url = "https://files.pythonhosted.org/packages/cf/72/6a85bba382f22cf78add705d8c3731748397d986e197e53ecc7835e76de7/yarl-1.22.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:c0232bce2170103ec23c454e54a57008a9a72b5d1c3105dc2496750da8cfa47c", size = 341473, upload-time = "2025-10-06T14:12:10.994Z" },
- { url = "https://files.pythonhosted.org/packages/35/18/55e6011f7c044dc80b98893060773cefcfdbf60dfefb8cb2f58b9bacbd83/yarl-1.22.0-cp314-cp314t-win32.whl", hash = "sha256:8009b3173bcd637be650922ac455946197d858b3630b6d8787aa9e5c4564533e", size = 89056, upload-time = "2025-10-06T14:12:13.317Z" },
- { url = "https://files.pythonhosted.org/packages/f9/86/0f0dccb6e59a9e7f122c5afd43568b1d31b8ab7dda5f1b01fb5c7025c9a9/yarl-1.22.0-cp314-cp314t-win_amd64.whl", hash = "sha256:9fb17ea16e972c63d25d4a97f016d235c78dd2344820eb35bc034bc32012ee27", size = 96292, upload-time = "2025-10-06T14:12:15.398Z" },
- { url = "https://files.pythonhosted.org/packages/48/b7/503c98092fb3b344a179579f55814b613c1fbb1c23b3ec14a7b008a66a6e/yarl-1.22.0-cp314-cp314t-win_arm64.whl", hash = "sha256:9f6d73c1436b934e3f01df1e1b21ff765cd1d28c77dfb9ace207f746d4610ee1", size = 85171, upload-time = "2025-10-06T14:12:16.935Z" },
- { url = "https://files.pythonhosted.org/packages/73/ae/b48f95715333080afb75a4504487cbe142cae1268afc482d06692d605ae6/yarl-1.22.0-py3-none-any.whl", hash = "sha256:1380560bdba02b6b6c90de54133c81c9f2a453dee9912fe58c1dcced1edb7cff", size = 46814, upload-time = "2025-10-06T14:12:53.872Z" },
+sdist = { url = "https://files.pythonhosted.org/packages/23/6e/beb1beec874a72f23815c1434518bfc4ed2175065173fb138c3705f658d4/yarl-1.23.0.tar.gz", hash = "sha256:53b1ea6ca88ebd4420379c330aea57e258408dd0df9af0992e5de2078dc9f5d5", size = 194676, upload-time = "2026-03-01T22:07:53.373Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/88/8a/94615bc31022f711add374097ad4144d569e95ff3c38d39215d07ac153a0/yarl-1.23.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:1932b6b8bba8d0160a9d1078aae5838a66039e8832d41d2992daa9a3a08f7860", size = 124737, upload-time = "2026-03-01T22:05:12.897Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/6f/c6554045d59d64052698add01226bc867b52fe4a12373415d7991fdca95d/yarl-1.23.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:411225bae281f114067578891bc75534cfb3d92a3b4dfef7a6ca78ba354e6069", size = 87029, upload-time = "2026-03-01T22:05:14.376Z" },
+ { url = "https://files.pythonhosted.org/packages/19/2a/725ecc166d53438bc88f76822ed4b1e3b10756e790bafd7b523fe97c322d/yarl-1.23.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:13a563739ae600a631c36ce096615fe307f131344588b0bc0daec108cdb47b25", size = 86310, upload-time = "2026-03-01T22:05:15.71Z" },
+ { url = "https://files.pythonhosted.org/packages/99/30/58260ed98e6ff7f90ba84442c1ddd758c9170d70327394a6227b310cd60f/yarl-1.23.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9cbf44c5cb4a7633d078788e1b56387e3d3cf2b8139a3be38040b22d6c3221c8", size = 97587, upload-time = "2026-03-01T22:05:17.384Z" },
+ { url = "https://files.pythonhosted.org/packages/76/0a/8b08aac08b50682e65759f7f8dde98ae8168f72487e7357a5d684c581ef9/yarl-1.23.0-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:53ad387048f6f09a8969631e4de3f1bf70c50e93545d64af4f751b2498755072", size = 92528, upload-time = "2026-03-01T22:05:18.804Z" },
+ { url = "https://files.pythonhosted.org/packages/52/07/0b7179101fe5f8385ec6c6bb5d0cb9f76bd9fb4a769591ab6fb5cdbfc69a/yarl-1.23.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4a59ba56f340334766f3a4442e0efd0af895fae9e2b204741ef885c446b3a1a8", size = 105339, upload-time = "2026-03-01T22:05:20.235Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/8a/36d82869ab5ec829ca8574dfcb92b51286fcfb1e9c7a73659616362dc880/yarl-1.23.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:803a3c3ce4acc62eaf01eaca1208dcf0783025ef27572c3336502b9c232005e7", size = 105061, upload-time = "2026-03-01T22:05:22.268Z" },
+ { url = "https://files.pythonhosted.org/packages/66/3e/868e5c3364b6cee19ff3e1a122194fa4ce51def02c61023970442162859e/yarl-1.23.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a3d2bff8f37f8d0f96c7ec554d16945050d54462d6e95414babaa18bfafc7f51", size = 100132, upload-time = "2026-03-01T22:05:23.638Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/26/9c89acf82f08a52cb52d6d39454f8d18af15f9d386a23795389d1d423823/yarl-1.23.0-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c75eb09e8d55bceb4367e83496ff8ef2bc7ea6960efb38e978e8073ea59ecb67", size = 99289, upload-time = "2026-03-01T22:05:25.749Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/54/5b0db00d2cb056922356104468019c0a132e89c8d3ab67d8ede9f4483d2a/yarl-1.23.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:877b0738624280e34c55680d6054a307aa94f7d52fa0e3034a9cc6e790871da7", size = 96950, upload-time = "2026-03-01T22:05:27.318Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/40/10fa93811fd439341fad7e0718a86aca0de9548023bbb403668d6555acab/yarl-1.23.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:b5405bb8f0e783a988172993cfc627e4d9d00432d6bbac65a923041edacf997d", size = 93960, upload-time = "2026-03-01T22:05:28.738Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/d2/8ae2e6cd77d0805f4526e30ec43b6f9a3dfc542d401ac4990d178e4bf0cf/yarl-1.23.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1c3a3598a832590c5a3ce56ab5576361b5688c12cb1d39429cf5dba30b510760", size = 104703, upload-time = "2026-03-01T22:05:30.438Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/0c/b3ceacf82c3fe21183ce35fa2acf5320af003d52bc1fcf5915077681142e/yarl-1.23.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:8419ebd326430d1cbb7efb5292330a2cf39114e82df5cc3d83c9a0d5ebeaf2f2", size = 98325, upload-time = "2026-03-01T22:05:31.835Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/e0/12900edd28bdab91a69bd2554b85ad7b151f64e8b521fe16f9ad2f56477a/yarl-1.23.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:be61f6fff406ca40e3b1d84716fde398fc08bc63dd96d15f3a14230a0973ed86", size = 105067, upload-time = "2026-03-01T22:05:33.358Z" },
+ { url = "https://files.pythonhosted.org/packages/15/61/74bb1182cf79c9bbe4eb6b1f14a57a22d7a0be5e9cedf8e2d5c2086474c3/yarl-1.23.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3ceb13c5c858d01321b5d9bb65e4cf37a92169ea470b70fec6f236b2c9dd7e34", size = 100285, upload-time = "2026-03-01T22:05:35.4Z" },
+ { url = "https://files.pythonhosted.org/packages/69/7f/cd5ef733f2550de6241bd8bd8c3febc78158b9d75f197d9c7baa113436af/yarl-1.23.0-cp312-cp312-win32.whl", hash = "sha256:fffc45637bcd6538de8b85f51e3df3223e4ad89bccbfca0481c08c7fc8b7ed7d", size = 82359, upload-time = "2026-03-01T22:05:36.811Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/be/25216a49daeeb7af2bec0db22d5e7df08ed1d7c9f65d78b14f3b74fd72fc/yarl-1.23.0-cp312-cp312-win_amd64.whl", hash = "sha256:f69f57305656a4852f2a7203efc661d8c042e6cc67f7acd97d8667fb448a426e", size = 87674, upload-time = "2026-03-01T22:05:38.171Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/35/aeab955d6c425b227d5b7247eafb24f2653fedc32f95373a001af5dfeb9e/yarl-1.23.0-cp312-cp312-win_arm64.whl", hash = "sha256:6e87a6e8735b44816e7db0b2fbc9686932df473c826b0d9743148432e10bb9b9", size = 81879, upload-time = "2026-03-01T22:05:40.006Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/4b/a0a6e5d0ee8a2f3a373ddef8a4097d74ac901ac363eea1440464ccbe0898/yarl-1.23.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:16c6994ac35c3e74fb0ae93323bf8b9c2a9088d55946109489667c510a7d010e", size = 123796, upload-time = "2026-03-01T22:05:41.412Z" },
+ { url = "https://files.pythonhosted.org/packages/67/b6/8925d68af039b835ae876db5838e82e76ec87b9782ecc97e192b809c4831/yarl-1.23.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:4a42e651629dafb64fd5b0286a3580613702b5809ad3f24934ea87595804f2c5", size = 86547, upload-time = "2026-03-01T22:05:42.841Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/50/06d511cc4b8e0360d3c94af051a768e84b755c5eb031b12adaaab6dec6e5/yarl-1.23.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7c6b9461a2a8b47c65eef63bb1c76a4f1c119618ffa99ea79bc5bb1e46c5821b", size = 85854, upload-time = "2026-03-01T22:05:44.85Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/f4/4e30b250927ffdab4db70da08b9b8d2194d7c7b400167b8fbeca1e4701ca/yarl-1.23.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2569b67d616eab450d262ca7cb9f9e19d2f718c70a8b88712859359d0ab17035", size = 98351, upload-time = "2026-03-01T22:05:46.836Z" },
+ { url = "https://files.pythonhosted.org/packages/86/fc/4118c5671ea948208bdb1492d8b76bdf1453d3e73df051f939f563e7dcc5/yarl-1.23.0-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:e9d9a4d06d3481eab79803beb4d9bd6f6a8e781ec078ac70d7ef2dcc29d1bea5", size = 92711, upload-time = "2026-03-01T22:05:48.316Z" },
+ { url = "https://files.pythonhosted.org/packages/56/11/1ed91d42bd9e73c13dc9e7eb0dd92298d75e7ac4dd7f046ad0c472e231cd/yarl-1.23.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f514f6474e04179d3d33175ed3f3e31434d3130d42ec153540d5b157deefd735", size = 106014, upload-time = "2026-03-01T22:05:50.028Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/c9/74e44e056a23fbc33aca71779ef450ca648a5bc472bdad7a82339918f818/yarl-1.23.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fda207c815b253e34f7e1909840fd14299567b1c0eb4908f8c2ce01a41265401", size = 105557, upload-time = "2026-03-01T22:05:51.416Z" },
+ { url = "https://files.pythonhosted.org/packages/66/fe/b1e10b08d287f518994f1e2ff9b6d26f0adeecd8dd7d533b01bab29a3eda/yarl-1.23.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:34b6cf500e61c90f305094911f9acc9c86da1a05a7a3f5be9f68817043f486e4", size = 101559, upload-time = "2026-03-01T22:05:52.872Z" },
+ { url = "https://files.pythonhosted.org/packages/72/59/c5b8d94b14e3d3c2a9c20cb100119fd534ab5a14b93673ab4cc4a4141ea5/yarl-1.23.0-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d7504f2b476d21653e4d143f44a175f7f751cd41233525312696c76aa3dbb23f", size = 100502, upload-time = "2026-03-01T22:05:54.954Z" },
+ { url = "https://files.pythonhosted.org/packages/77/4f/96976cb54cbfc5c9fd73ed4c51804f92f209481d1fb190981c0f8a07a1d7/yarl-1.23.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:578110dd426f0d209d1509244e6d4a3f1a3e9077655d98c5f22583d63252a08a", size = 98027, upload-time = "2026-03-01T22:05:56.409Z" },
+ { url = "https://files.pythonhosted.org/packages/63/6e/904c4f476471afdbad6b7e5b70362fb5810e35cd7466529a97322b6f5556/yarl-1.23.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:609d3614d78d74ebe35f54953c5bbd2ac647a7ddb9c30a5d877580f5e86b22f2", size = 95369, upload-time = "2026-03-01T22:05:58.141Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/40/acfcdb3b5f9d68ef499e39e04d25e141fe90661f9d54114556cf83be8353/yarl-1.23.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:4966242ec68afc74c122f8459abd597afd7d8a60dc93d695c1334c5fd25f762f", size = 105565, upload-time = "2026-03-01T22:06:00.286Z" },
+ { url = "https://files.pythonhosted.org/packages/5e/c6/31e28f3a6ba2869c43d124f37ea5260cac9c9281df803c354b31f4dd1f3c/yarl-1.23.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:e0fd068364a6759bc794459f0a735ab151d11304346332489c7972bacbe9e72b", size = 99813, upload-time = "2026-03-01T22:06:01.712Z" },
+ { url = "https://files.pythonhosted.org/packages/08/1f/6f65f59e72d54aa467119b63fc0b0b1762eff0232db1f4720cd89e2f4a17/yarl-1.23.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:39004f0ad156da43e86aa71f44e033de68a44e5a31fc53507b36dd253970054a", size = 105632, upload-time = "2026-03-01T22:06:03.188Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/c4/18b178a69935f9e7a338127d5b77d868fdc0f0e49becd286d51b3a18c61d/yarl-1.23.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e5723c01a56c5028c807c701aa66722916d2747ad737a046853f6c46f4875543", size = 101895, upload-time = "2026-03-01T22:06:04.651Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/54/f5b870b5505663911dba950a8e4776a0dbd51c9c54c0ae88e823e4b874a0/yarl-1.23.0-cp313-cp313-win32.whl", hash = "sha256:1b6b572edd95b4fa8df75de10b04bc81acc87c1c7d16bcdd2035b09d30acc957", size = 82356, upload-time = "2026-03-01T22:06:06.04Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/84/266e8da36879c6edcd37b02b547e2d9ecdfea776be49598e75696e3316e1/yarl-1.23.0-cp313-cp313-win_amd64.whl", hash = "sha256:baaf55442359053c7d62f6f8413a62adba3205119bcb6f49594894d8be47e5e3", size = 87515, upload-time = "2026-03-01T22:06:08.107Z" },
+ { url = "https://files.pythonhosted.org/packages/00/fd/7e1c66efad35e1649114fa13f17485f62881ad58edeeb7f49f8c5e748bf9/yarl-1.23.0-cp313-cp313-win_arm64.whl", hash = "sha256:fb4948814a2a98e3912505f09c9e7493b1506226afb1f881825368d6fb776ee3", size = 81785, upload-time = "2026-03-01T22:06:10.181Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/fc/119dd07004f17ea43bb91e3ece6587759edd7519d6b086d16bfbd3319982/yarl-1.23.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:aecfed0b41aa72b7881712c65cf764e39ce2ec352324f5e0837c7048d9e6daaa", size = 130719, upload-time = "2026-03-01T22:06:11.708Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/0d/9f2348502fbb3af409e8f47730282cd6bc80dec6630c1e06374d882d6eb2/yarl-1.23.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:a41bcf68efd19073376eb8cf948b8d9be0af26256403e512bb18f3966f1f9120", size = 89690, upload-time = "2026-03-01T22:06:13.429Z" },
+ { url = "https://files.pythonhosted.org/packages/50/93/e88f3c80971b42cfc83f50a51b9d165a1dbf154b97005f2994a79f212a07/yarl-1.23.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:cde9a2ecd91668bcb7f077c4966d8ceddb60af01b52e6e3e2680e4cf00ad1a59", size = 89851, upload-time = "2026-03-01T22:06:15.53Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/07/61c9dd8ba8f86473263b4036f70fb594c09e99c0d9737a799dfd8bc85651/yarl-1.23.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5023346c4ee7992febc0068e7593de5fa2bf611848c08404b35ebbb76b1b0512", size = 95874, upload-time = "2026-03-01T22:06:17.553Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/e9/f9ff8ceefba599eac6abddcfb0b3bee9b9e636e96dbf54342a8577252379/yarl-1.23.0-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:d1009abedb49ae95b136a8904a3f71b342f849ffeced2d3747bf29caeda218c4", size = 88710, upload-time = "2026-03-01T22:06:19.004Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/78/0231bfcc5d4c8eec220bc2f9ef82cb4566192ea867a7c5b4148f44f6cbcd/yarl-1.23.0-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a8d00f29b42f534cc8aa3931cfe773b13b23e561e10d2b26f27a8d309b0e82a1", size = 101033, upload-time = "2026-03-01T22:06:21.203Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/9b/30ea5239a61786f18fd25797151a17fbb3be176977187a48d541b5447dd4/yarl-1.23.0-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:95451e6ce06c3e104556d73b559f5da6c34a069b6b62946d3ad66afcd51642ea", size = 100817, upload-time = "2026-03-01T22:06:22.738Z" },
+ { url = "https://files.pythonhosted.org/packages/62/e2/a4980481071791bc83bce2b7a1a1f7adcabfa366007518b4b845e92eeee3/yarl-1.23.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:531ef597132086b6cf96faa7c6c1dcd0361dd5f1694e5cc30375907b9b7d3ea9", size = 97482, upload-time = "2026-03-01T22:06:24.21Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/1e/304a00cf5f6100414c4b5a01fc7ff9ee724b62158a08df2f8170dfc72a2d/yarl-1.23.0-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:88f9fb0116fbfcefcab70f85cf4b74a2b6ce5d199c41345296f49d974ddb4123", size = 95949, upload-time = "2026-03-01T22:06:25.697Z" },
+ { url = "https://files.pythonhosted.org/packages/68/03/093f4055ed4cae649ac53bca3d180bd37102e9e11d048588e9ab0c0108d0/yarl-1.23.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:e7b0460976dc75cb87ad9cc1f9899a4b97751e7d4e77ab840fc9b6d377b8fd24", size = 95839, upload-time = "2026-03-01T22:06:27.309Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/28/4c75ebb108f322aa8f917ae10a8ffa4f07cae10a8a627b64e578617df6a0/yarl-1.23.0-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:115136c4a426f9da976187d238e84139ff6b51a20839aa6e3720cd1026d768de", size = 90696, upload-time = "2026-03-01T22:06:29.048Z" },
+ { url = "https://files.pythonhosted.org/packages/23/9c/42c2e2dd91c1a570402f51bdf066bfdb1241c2240ba001967bad778e77b7/yarl-1.23.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:ead11956716a940c1abc816b7df3fa2b84d06eaed8832ca32f5c5e058c65506b", size = 100865, upload-time = "2026-03-01T22:06:30.525Z" },
+ { url = "https://files.pythonhosted.org/packages/74/05/1bcd60a8a0a914d462c305137246b6f9d167628d73568505fce3f1cb2e65/yarl-1.23.0-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:fe8f8f5e70e6dbdfca9882cd9deaac058729bcf323cf7a58660901e55c9c94f6", size = 96234, upload-time = "2026-03-01T22:06:32.692Z" },
+ { url = "https://files.pythonhosted.org/packages/90/b2/f52381aac396d6778ce516b7bc149c79e65bfc068b5de2857ab69eeea3b7/yarl-1.23.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:a0e317df055958a0c1e79e5d2aa5a5eaa4a6d05a20d4b0c9c3f48918139c9fc6", size = 100295, upload-time = "2026-03-01T22:06:34.268Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/e8/638bae5bbf1113a659b2435d8895474598afe38b4a837103764f603aba56/yarl-1.23.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:6f0fd84de0c957b2d280143522c4f91a73aada1923caee763e24a2b3fda9f8a5", size = 97784, upload-time = "2026-03-01T22:06:35.864Z" },
+ { url = "https://files.pythonhosted.org/packages/80/25/a3892b46182c586c202629fc2159aa13975d3741d52ebd7347fd501d48d5/yarl-1.23.0-cp313-cp313t-win32.whl", hash = "sha256:93a784271881035ab4406a172edb0faecb6e7d00f4b53dc2f55919d6c9688595", size = 88313, upload-time = "2026-03-01T22:06:37.39Z" },
+ { url = "https://files.pythonhosted.org/packages/43/68/8c5b36aa5178900b37387937bc2c2fe0e9505537f713495472dcf6f6fccc/yarl-1.23.0-cp313-cp313t-win_amd64.whl", hash = "sha256:dd00607bffbf30250fe108065f07453ec124dbf223420f57f5e749b04295e090", size = 94932, upload-time = "2026-03-01T22:06:39.579Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/cc/d79ba8292f51f81f4dc533a8ccfb9fc6992cabf0998ed3245de7589dc07c/yarl-1.23.0-cp313-cp313t-win_arm64.whl", hash = "sha256:ac09d42f48f80c9ee1635b2fcaa819496a44502737660d3c0f2ade7526d29144", size = 84786, upload-time = "2026-03-01T22:06:41.988Z" },
+ { url = "https://files.pythonhosted.org/packages/90/98/b85a038d65d1b92c3903ab89444f48d3cee490a883477b716d7a24b1a78c/yarl-1.23.0-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:21d1b7305a71a15b4794b5ff22e8eef96ff4a6d7f9657155e5aa419444b28912", size = 124455, upload-time = "2026-03-01T22:06:43.615Z" },
+ { url = "https://files.pythonhosted.org/packages/39/54/bc2b45559f86543d163b6e294417a107bb87557609007c007ad889afec18/yarl-1.23.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:85610b4f27f69984932a7abbe52703688de3724d9f72bceb1cca667deff27474", size = 86752, upload-time = "2026-03-01T22:06:45.425Z" },
+ { url = "https://files.pythonhosted.org/packages/24/f9/e8242b68362bffe6fb536c8db5076861466fc780f0f1b479fc4ffbebb128/yarl-1.23.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:23f371bd662cf44a7630d4d113101eafc0cfa7518a2760d20760b26021454719", size = 86291, upload-time = "2026-03-01T22:06:46.974Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/d8/d1cb2378c81dd729e98c716582b1ccb08357e8488e4c24714658cc6630e8/yarl-1.23.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c4a80f77dc1acaaa61f0934176fccca7096d9b1ff08c8ba9cddf5ae034a24319", size = 99026, upload-time = "2026-03-01T22:06:48.459Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/ff/7196790538f31debe3341283b5b0707e7feb947620fc5e8236ef28d44f72/yarl-1.23.0-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:bd654fad46d8d9e823afbb4f87c79160b5a374ed1ff5bde24e542e6ba8f41434", size = 92355, upload-time = "2026-03-01T22:06:50.306Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/56/25d58c3eddde825890a5fe6aa1866228377354a3c39262235234ab5f616b/yarl-1.23.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:682bae25f0a0dd23a056739f23a134db9f52a63e2afd6bfb37ddc76292bbd723", size = 106417, upload-time = "2026-03-01T22:06:52.1Z" },
+ { url = "https://files.pythonhosted.org/packages/51/8a/882c0e7bc8277eb895b31bce0138f51a1ba551fc2e1ec6753ffc1e7c1377/yarl-1.23.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a82836cab5f197a0514235aaf7ffccdc886ccdaa2324bc0aafdd4ae898103039", size = 106422, upload-time = "2026-03-01T22:06:54.424Z" },
+ { url = "https://files.pythonhosted.org/packages/42/2b/fef67d616931055bf3d6764885990a3ac647d68734a2d6a9e1d13de437a2/yarl-1.23.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1c57676bdedc94cd3bc37724cf6f8cd2779f02f6aba48de45feca073e714fe52", size = 101915, upload-time = "2026-03-01T22:06:55.895Z" },
+ { url = "https://files.pythonhosted.org/packages/18/6a/530e16aebce27c5937920f3431c628a29a4b6b430fab3fd1c117b26ff3f6/yarl-1.23.0-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c7f8dc16c498ff06497c015642333219871effba93e4a2e8604a06264aca5c5c", size = 100690, upload-time = "2026-03-01T22:06:58.21Z" },
+ { url = "https://files.pythonhosted.org/packages/88/08/93749219179a45e27b036e03260fda05190b911de8e18225c294ac95bbc9/yarl-1.23.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:5ee586fb17ff8f90c91cf73c6108a434b02d69925f44f5f8e0d7f2f260607eae", size = 98750, upload-time = "2026-03-01T22:06:59.794Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/cf/ea424a004969f5d81a362110a6ac1496d79efdc6d50c2c4b2e3ea0fc2519/yarl-1.23.0-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:17235362f580149742739cc3828b80e24029d08cbb9c4bda0242c7b5bc610a8e", size = 94685, upload-time = "2026-03-01T22:07:01.375Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/b7/14341481fe568e2b0408bcf1484c652accafe06a0ade9387b5d3fd9df446/yarl-1.23.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:0793e2bd0cf14234983bbb371591e6bea9e876ddf6896cdcc93450996b0b5c85", size = 106009, upload-time = "2026-03-01T22:07:03.151Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/e6/5c744a9b54f4e8007ad35bce96fbc9218338e84812d36f3390cea616881a/yarl-1.23.0-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:3650dc2480f94f7116c364096bc84b1d602f44224ef7d5c7208425915c0475dd", size = 100033, upload-time = "2026-03-01T22:07:04.701Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/23/e3bfc188d0b400f025bc49d99793d02c9abe15752138dcc27e4eaf0c4a9e/yarl-1.23.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:f40e782d49630ad384db66d4d8b73ff4f1b8955dc12e26b09a3e3af064b3b9d6", size = 106483, upload-time = "2026-03-01T22:07:06.231Z" },
+ { url = "https://files.pythonhosted.org/packages/72/42/f0505f949a90b3f8b7a363d6cbdf398f6e6c58946d85c6d3a3bc70595b26/yarl-1.23.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:94f8575fbdf81749008d980c17796097e645574a3b8c28ee313931068dad14fe", size = 102175, upload-time = "2026-03-01T22:07:08.4Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/65/b39290f1d892a9dd671d1c722014ca062a9c35d60885d57e5375db0404b5/yarl-1.23.0-cp314-cp314-win32.whl", hash = "sha256:c8aa34a5c864db1087d911a0b902d60d203ea3607d91f615acd3f3108ac32169", size = 83871, upload-time = "2026-03-01T22:07:09.968Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/5b/9b92f54c784c26e2a422e55a8d2607ab15b7ea3349e28359282f84f01d43/yarl-1.23.0-cp314-cp314-win_amd64.whl", hash = "sha256:63e92247f383c85ab00dd0091e8c3fa331a96e865459f5ee80353c70a4a42d70", size = 89093, upload-time = "2026-03-01T22:07:11.501Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/7d/8a84dc9381fd4412d5e7ff04926f9865f6372b4c2fd91e10092e65d29eb8/yarl-1.23.0-cp314-cp314-win_arm64.whl", hash = "sha256:70efd20be968c76ece7baa8dafe04c5be06abc57f754d6f36f3741f7aa7a208e", size = 83384, upload-time = "2026-03-01T22:07:13.069Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/8d/d2fad34b1c08aa161b74394183daa7d800141aaaee207317e82c790b418d/yarl-1.23.0-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:9a18d6f9359e45722c064c97464ec883eb0e0366d33eda61cb19a244bf222679", size = 131019, upload-time = "2026-03-01T22:07:14.903Z" },
+ { url = "https://files.pythonhosted.org/packages/19/ff/33009a39d3ccf4b94d7d7880dfe17fb5816c5a4fe0096d9b56abceea9ac7/yarl-1.23.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:2803ed8b21ca47a43da80a6fd1ed3019d30061f7061daa35ac54f63933409412", size = 89894, upload-time = "2026-03-01T22:07:17.372Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/f1/dab7ac5e7306fb79c0190766a3c00b4cb8d09a1f390ded68c85a5934faf5/yarl-1.23.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:394906945aa8b19fc14a61cf69743a868bb8c465efe85eee687109cc540b98f4", size = 89979, upload-time = "2026-03-01T22:07:19.361Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/b1/08e95f3caee1fad6e65017b9f26c1d79877b502622d60e517de01e72f95d/yarl-1.23.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:71d006bee8397a4a89f469b8deb22469fe7508132d3c17fa6ed871e79832691c", size = 95943, upload-time = "2026-03-01T22:07:21.266Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/cc/6409f9018864a6aa186c61175b977131f373f1988e198e031236916e87e4/yarl-1.23.0-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:62694e275c93d54f7ccedcfef57d42761b2aad5234b6be1f3e3026cae4001cd4", size = 88786, upload-time = "2026-03-01T22:07:23.129Z" },
+ { url = "https://files.pythonhosted.org/packages/76/40/cc22d1d7714b717fde2006fad2ced5efe5580606cb059ae42117542122f3/yarl-1.23.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a31de1613658308efdb21ada98cbc86a97c181aa050ba22a808120bb5be3ab94", size = 101307, upload-time = "2026-03-01T22:07:24.689Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/0d/476c38e85ddb4c6ec6b20b815bdd779aa386a013f3d8b85516feee55c8dc/yarl-1.23.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fb1e8b8d66c278b21d13b0a7ca22c41dd757a7c209c6b12c313e445c31dd3b28", size = 100904, upload-time = "2026-03-01T22:07:26.287Z" },
+ { url = "https://files.pythonhosted.org/packages/72/32/0abe4a76d59adf2081dcb0397168553ece4616ada1c54d1c49d8936c74f8/yarl-1.23.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:50f9d8d531dfb767c565f348f33dd5139a6c43f5cbdf3f67da40d54241df93f6", size = 97728, upload-time = "2026-03-01T22:07:27.906Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/35/7b30f4810fba112f60f5a43237545867504e15b1c7647a785fbaf588fac2/yarl-1.23.0-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:575aa4405a656e61a540f4a80eaa5260f2a38fff7bfdc4b5f611840d76e9e277", size = 95964, upload-time = "2026-03-01T22:07:30.198Z" },
+ { url = "https://files.pythonhosted.org/packages/2d/86/ed7a73ab85ef00e8bb70b0cb5421d8a2a625b81a333941a469a6f4022828/yarl-1.23.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:041b1a4cefacf65840b4e295c6985f334ba83c30607441ae3cf206a0eed1a2e4", size = 95882, upload-time = "2026-03-01T22:07:32.132Z" },
+ { url = "https://files.pythonhosted.org/packages/19/90/d56967f61a29d8498efb7afb651e0b2b422a1e9b47b0ab5f4e40a19b699b/yarl-1.23.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:d38c1e8231722c4ce40d7593f28d92b5fc72f3e9774fe73d7e800ec32299f63a", size = 90797, upload-time = "2026-03-01T22:07:34.404Z" },
+ { url = "https://files.pythonhosted.org/packages/72/00/8b8f76909259f56647adb1011d7ed8b321bcf97e464515c65016a47ecdf0/yarl-1.23.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:d53834e23c015ee83a99377db6e5e37d8484f333edb03bd15b4bc312cc7254fb", size = 101023, upload-time = "2026-03-01T22:07:35.953Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/e2/cab11b126fb7d440281b7df8e9ddbe4851e70a4dde47a202b6642586b8d9/yarl-1.23.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:2e27c8841126e017dd2a054a95771569e6070b9ee1b133366d8b31beb5018a41", size = 96227, upload-time = "2026-03-01T22:07:37.594Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/9b/2c893e16bfc50e6b2edf76c1a9eb6cb0c744346197e74c65e99ad8d634d0/yarl-1.23.0-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:76855800ac56f878847a09ce6dba727c93ca2d89c9e9d63002d26b916810b0a2", size = 100302, upload-time = "2026-03-01T22:07:39.334Z" },
+ { url = "https://files.pythonhosted.org/packages/28/ec/5498c4e3a6d5f1003beb23405671c2eb9cdbf3067d1c80f15eeafe301010/yarl-1.23.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:e09fd068c2e169a7070d83d3bde728a4d48de0549f975290be3c108c02e499b4", size = 98202, upload-time = "2026-03-01T22:07:41.717Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/c3/cd737e2d45e70717907f83e146f6949f20cc23cd4bf7b2688727763aa458/yarl-1.23.0-cp314-cp314t-win32.whl", hash = "sha256:73309162a6a571d4cbd3b6a1dcc703c7311843ae0d1578df6f09be4e98df38d4", size = 90558, upload-time = "2026-03-01T22:07:43.433Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/19/3774d162f6732d1cfb0b47b4140a942a35ca82bb19b6db1f80e9e7bdc8f8/yarl-1.23.0-cp314-cp314t-win_amd64.whl", hash = "sha256:4503053d296bc6e4cbd1fad61cf3b6e33b939886c4f249ba7c78b602214fabe2", size = 97610, upload-time = "2026-03-01T22:07:45.773Z" },
+ { url = "https://files.pythonhosted.org/packages/51/47/3fa2286c3cb162c71cdb34c4224d5745a1ceceb391b2bd9b19b668a8d724/yarl-1.23.0-cp314-cp314t-win_arm64.whl", hash = "sha256:44bb7bef4ea409384e3f8bc36c063d77ea1b8d4a5b2706956c0d6695f07dcc25", size = 86041, upload-time = "2026-03-01T22:07:49.026Z" },
+ { url = "https://files.pythonhosted.org/packages/69/68/c8739671f5699c7dc470580a4f821ef37c32c4cb0b047ce223a7f115757f/yarl-1.23.0-py3-none-any.whl", hash = "sha256:a2df6afe50dea8ae15fa34c9f824a3ee958d785fd5d089063d960bae1daa0a3f", size = 48288, upload-time = "2026-03-01T22:07:51.388Z" },
]
[[package]]
@@ -6521,3 +7197,60 @@ sdist = { url = "https://files.pythonhosted.org/packages/e3/02/0f2892c661036d50e
wheels = [
{ url = "https://files.pythonhosted.org/packages/2e/54/647ade08bf0db230bfea292f893923872fd20be6ac6f53b2b936ba839d75/zipp-3.23.0-py3-none-any.whl", hash = "sha256:071652d6115ed432f5ce1d34c336c0adfd6a884660d1e9712a256d3d3bd4b14e", size = 10276, upload-time = "2025-06-08T17:06:38.034Z" },
]
+
+[[package]]
+name = "zstandard"
+version = "0.25.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/fd/aa/3e0508d5a5dd96529cdc5a97011299056e14c6505b678fd58938792794b1/zstandard-0.25.0.tar.gz", hash = "sha256:7713e1179d162cf5c7906da876ec2ccb9c3a9dcbdffef0cc7f70c3667a205f0b", size = 711513, upload-time = "2025-09-14T22:15:54.002Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/82/fc/f26eb6ef91ae723a03e16eddb198abcfce2bc5a42e224d44cc8b6765e57e/zstandard-0.25.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7b3c3a3ab9daa3eed242d6ecceead93aebbb8f5f84318d82cee643e019c4b73b", size = 795738, upload-time = "2025-09-14T22:16:56.237Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/1c/d920d64b22f8dd028a8b90e2d756e431a5d86194caa78e3819c7bf53b4b3/zstandard-0.25.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:913cbd31a400febff93b564a23e17c3ed2d56c064006f54efec210d586171c00", size = 640436, upload-time = "2025-09-14T22:16:57.774Z" },
+ { url = "https://files.pythonhosted.org/packages/53/6c/288c3f0bd9fcfe9ca41e2c2fbfd17b2097f6af57b62a81161941f09afa76/zstandard-0.25.0-cp312-cp312-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:011d388c76b11a0c165374ce660ce2c8efa8e5d87f34996aa80f9c0816698b64", size = 5343019, upload-time = "2025-09-14T22:16:59.302Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/15/efef5a2f204a64bdb5571e6161d49f7ef0fffdbca953a615efbec045f60f/zstandard-0.25.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:6dffecc361d079bb48d7caef5d673c88c8988d3d33fb74ab95b7ee6da42652ea", size = 5063012, upload-time = "2025-09-14T22:17:01.156Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/37/a6ce629ffdb43959e92e87ebdaeebb5ac81c944b6a75c9c47e300f85abdf/zstandard-0.25.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:7149623bba7fdf7e7f24312953bcf73cae103db8cae49f8154dd1eadc8a29ecb", size = 5394148, upload-time = "2025-09-14T22:17:03.091Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/79/2bf870b3abeb5c070fe2d670a5a8d1057a8270f125ef7676d29ea900f496/zstandard-0.25.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:6a573a35693e03cf1d67799fd01b50ff578515a8aeadd4595d2a7fa9f3ec002a", size = 5451652, upload-time = "2025-09-14T22:17:04.979Z" },
+ { url = "https://files.pythonhosted.org/packages/53/60/7be26e610767316c028a2cbedb9a3beabdbe33e2182c373f71a1c0b88f36/zstandard-0.25.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5a56ba0db2d244117ed744dfa8f6f5b366e14148e00de44723413b2f3938a902", size = 5546993, upload-time = "2025-09-14T22:17:06.781Z" },
+ { url = "https://files.pythonhosted.org/packages/85/c7/3483ad9ff0662623f3648479b0380d2de5510abf00990468c286c6b04017/zstandard-0.25.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:10ef2a79ab8e2974e2075fb984e5b9806c64134810fac21576f0668e7ea19f8f", size = 5046806, upload-time = "2025-09-14T22:17:08.415Z" },
+ { url = "https://files.pythonhosted.org/packages/08/b3/206883dd25b8d1591a1caa44b54c2aad84badccf2f1de9e2d60a446f9a25/zstandard-0.25.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:aaf21ba8fb76d102b696781bddaa0954b782536446083ae3fdaa6f16b25a1c4b", size = 5576659, upload-time = "2025-09-14T22:17:10.164Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/31/76c0779101453e6c117b0ff22565865c54f48f8bd807df2b00c2c404b8e0/zstandard-0.25.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1869da9571d5e94a85a5e8d57e4e8807b175c9e4a6294e3b66fa4efb074d90f6", size = 4953933, upload-time = "2025-09-14T22:17:11.857Z" },
+ { url = "https://files.pythonhosted.org/packages/18/e1/97680c664a1bf9a247a280a053d98e251424af51f1b196c6d52f117c9720/zstandard-0.25.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:809c5bcb2c67cd0ed81e9229d227d4ca28f82d0f778fc5fea624a9def3963f91", size = 5268008, upload-time = "2025-09-14T22:17:13.627Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/73/316e4010de585ac798e154e88fd81bb16afc5c5cb1a72eeb16dd37e8024a/zstandard-0.25.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:f27662e4f7dbf9f9c12391cb37b4c4c3cb90ffbd3b1fb9284dadbbb8935fa708", size = 5433517, upload-time = "2025-09-14T22:17:16.103Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/60/dd0f8cfa8129c5a0ce3ea6b7f70be5b33d2618013a161e1ff26c2b39787c/zstandard-0.25.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:99c0c846e6e61718715a3c9437ccc625de26593fea60189567f0118dc9db7512", size = 5814292, upload-time = "2025-09-14T22:17:17.827Z" },
+ { url = "https://files.pythonhosted.org/packages/fc/5f/75aafd4b9d11b5407b641b8e41a57864097663699f23e9ad4dbb91dc6bfe/zstandard-0.25.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:474d2596a2dbc241a556e965fb76002c1ce655445e4e3bf38e5477d413165ffa", size = 5360237, upload-time = "2025-09-14T22:17:19.954Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/8d/0309daffea4fcac7981021dbf21cdb2e3427a9e76bafbcdbdf5392ff99a4/zstandard-0.25.0-cp312-cp312-win32.whl", hash = "sha256:23ebc8f17a03133b4426bcc04aabd68f8236eb78c3760f12783385171b0fd8bd", size = 436922, upload-time = "2025-09-14T22:17:24.398Z" },
+ { url = "https://files.pythonhosted.org/packages/79/3b/fa54d9015f945330510cb5d0b0501e8253c127cca7ebe8ba46a965df18c5/zstandard-0.25.0-cp312-cp312-win_amd64.whl", hash = "sha256:ffef5a74088f1e09947aecf91011136665152e0b4b359c42be3373897fb39b01", size = 506276, upload-time = "2025-09-14T22:17:21.429Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/6b/8b51697e5319b1f9ac71087b0af9a40d8a6288ff8025c36486e0c12abcc4/zstandard-0.25.0-cp312-cp312-win_arm64.whl", hash = "sha256:181eb40e0b6a29b3cd2849f825e0fa34397f649170673d385f3598ae17cca2e9", size = 462679, upload-time = "2025-09-14T22:17:23.147Z" },
+ { url = "https://files.pythonhosted.org/packages/35/0b/8df9c4ad06af91d39e94fa96cc010a24ac4ef1378d3efab9223cc8593d40/zstandard-0.25.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ec996f12524f88e151c339688c3897194821d7f03081ab35d31d1e12ec975e94", size = 795735, upload-time = "2025-09-14T22:17:26.042Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/06/9ae96a3e5dcfd119377ba33d4c42a7d89da1efabd5cb3e366b156c45ff4d/zstandard-0.25.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a1a4ae2dec3993a32247995bdfe367fc3266da832d82f8438c8570f989753de1", size = 640440, upload-time = "2025-09-14T22:17:27.366Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/14/933d27204c2bd404229c69f445862454dcc101cd69ef8c6068f15aaec12c/zstandard-0.25.0-cp313-cp313-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:e96594a5537722fdfb79951672a2a63aec5ebfb823e7560586f7484819f2a08f", size = 5343070, upload-time = "2025-09-14T22:17:28.896Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/db/ddb11011826ed7db9d0e485d13df79b58586bfdec56e5c84a928a9a78c1c/zstandard-0.25.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:bfc4e20784722098822e3eee42b8e576b379ed72cca4a7cb856ae733e62192ea", size = 5063001, upload-time = "2025-09-14T22:17:31.044Z" },
+ { url = "https://files.pythonhosted.org/packages/db/00/87466ea3f99599d02a5238498b87bf84a6348290c19571051839ca943777/zstandard-0.25.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:457ed498fc58cdc12fc48f7950e02740d4f7ae9493dd4ab2168a47c93c31298e", size = 5394120, upload-time = "2025-09-14T22:17:32.711Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/95/fc5531d9c618a679a20ff6c29e2b3ef1d1f4ad66c5e161ae6ff847d102a9/zstandard-0.25.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:fd7a5004eb1980d3cefe26b2685bcb0b17989901a70a1040d1ac86f1d898c551", size = 5451230, upload-time = "2025-09-14T22:17:34.41Z" },
+ { url = "https://files.pythonhosted.org/packages/63/4b/e3678b4e776db00f9f7b2fe58e547e8928ef32727d7a1ff01dea010f3f13/zstandard-0.25.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8e735494da3db08694d26480f1493ad2cf86e99bdd53e8e9771b2752a5c0246a", size = 5547173, upload-time = "2025-09-14T22:17:36.084Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/d5/ba05ed95c6b8ec30bd468dfeab20589f2cf709b5c940483e31d991f2ca58/zstandard-0.25.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:3a39c94ad7866160a4a46d772e43311a743c316942037671beb264e395bdd611", size = 5046736, upload-time = "2025-09-14T22:17:37.891Z" },
+ { url = "https://files.pythonhosted.org/packages/50/d5/870aa06b3a76c73eced65c044b92286a3c4e00554005ff51962deef28e28/zstandard-0.25.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:172de1f06947577d3a3005416977cce6168f2261284c02080e7ad0185faeced3", size = 5576368, upload-time = "2025-09-14T22:17:40.206Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/35/398dc2ffc89d304d59bc12f0fdd931b4ce455bddf7038a0a67733a25f550/zstandard-0.25.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3c83b0188c852a47cd13ef3bf9209fb0a77fa5374958b8c53aaa699398c6bd7b", size = 4954022, upload-time = "2025-09-14T22:17:41.879Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/5c/36ba1e5507d56d2213202ec2b05e8541734af5f2ce378c5d1ceaf4d88dc4/zstandard-0.25.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:1673b7199bbe763365b81a4f3252b8e80f44c9e323fc42940dc8843bfeaf9851", size = 5267889, upload-time = "2025-09-14T22:17:43.577Z" },
+ { url = "https://files.pythonhosted.org/packages/70/e8/2ec6b6fb7358b2ec0113ae202647ca7c0e9d15b61c005ae5225ad0995df5/zstandard-0.25.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:0be7622c37c183406f3dbf0cba104118eb16a4ea7359eeb5752f0794882fc250", size = 5433952, upload-time = "2025-09-14T22:17:45.271Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/01/b5f4d4dbc59ef193e870495c6f1275f5b2928e01ff5a81fecb22a06e22fb/zstandard-0.25.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:5f5e4c2a23ca271c218ac025bd7d635597048b366d6f31f420aaeb715239fc98", size = 5814054, upload-time = "2025-09-14T22:17:47.08Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/e5/fbd822d5c6f427cf158316d012c5a12f233473c2f9c5fe5ab1ae5d21f3d8/zstandard-0.25.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4f187a0bb61b35119d1926aee039524d1f93aaf38a9916b8c4b78ac8514a0aaf", size = 5360113, upload-time = "2025-09-14T22:17:48.893Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/e0/69a553d2047f9a2c7347caa225bb3a63b6d7704ad74610cb7823baa08ed7/zstandard-0.25.0-cp313-cp313-win32.whl", hash = "sha256:7030defa83eef3e51ff26f0b7bfb229f0204b66fe18e04359ce3474ac33cbc09", size = 436936, upload-time = "2025-09-14T22:17:52.658Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/82/b9c06c870f3bd8767c201f1edbdf9e8dc34be5b0fbc5682c4f80fe948475/zstandard-0.25.0-cp313-cp313-win_amd64.whl", hash = "sha256:1f830a0dac88719af0ae43b8b2d6aef487d437036468ef3c2ea59c51f9d55fd5", size = 506232, upload-time = "2025-09-14T22:17:50.402Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/57/60c3c01243bb81d381c9916e2a6d9e149ab8627c0c7d7abb2d73384b3c0c/zstandard-0.25.0-cp313-cp313-win_arm64.whl", hash = "sha256:85304a43f4d513f5464ceb938aa02c1e78c2943b29f44a750b48b25ac999a049", size = 462671, upload-time = "2025-09-14T22:17:51.533Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/5c/f8923b595b55fe49e30612987ad8bf053aef555c14f05bb659dd5dbe3e8a/zstandard-0.25.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:e29f0cf06974c899b2c188ef7f783607dbef36da4c242eb6c82dcd8b512855e3", size = 795887, upload-time = "2025-09-14T22:17:54.198Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/09/d0a2a14fc3439c5f874042dca72a79c70a532090b7ba0003be73fee37ae2/zstandard-0.25.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:05df5136bc5a011f33cd25bc9f506e7426c0c9b3f9954f056831ce68f3b6689f", size = 640658, upload-time = "2025-09-14T22:17:55.423Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/7c/8b6b71b1ddd517f68ffb55e10834388d4f793c49c6b83effaaa05785b0b4/zstandard-0.25.0-cp314-cp314-manylinux2010_i686.manylinux_2_12_i686.manylinux_2_28_i686.whl", hash = "sha256:f604efd28f239cc21b3adb53eb061e2a205dc164be408e553b41ba2ffe0ca15c", size = 5379849, upload-time = "2025-09-14T22:17:57.372Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/86/a48e56320d0a17189ab7a42645387334fba2200e904ee47fc5a26c1fd8ca/zstandard-0.25.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:223415140608d0f0da010499eaa8ccdb9af210a543fac54bce15babbcfc78439", size = 5058095, upload-time = "2025-09-14T22:17:59.498Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/ad/eb659984ee2c0a779f9d06dbfe45e2dc39d99ff40a319895df2d3d9a48e5/zstandard-0.25.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2e54296a283f3ab5a26fc9b8b5d4978ea0532f37b231644f367aa588930aa043", size = 5551751, upload-time = "2025-09-14T22:18:01.618Z" },
+ { url = "https://files.pythonhosted.org/packages/61/b3/b637faea43677eb7bd42ab204dfb7053bd5c4582bfe6b1baefa80ac0c47b/zstandard-0.25.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ca54090275939dc8ec5dea2d2afb400e0f83444b2fc24e07df7fdef677110859", size = 6364818, upload-time = "2025-09-14T22:18:03.769Z" },
+ { url = "https://files.pythonhosted.org/packages/31/dc/cc50210e11e465c975462439a492516a73300ab8caa8f5e0902544fd748b/zstandard-0.25.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e09bb6252b6476d8d56100e8147b803befa9a12cea144bbe629dd508800d1ad0", size = 5560402, upload-time = "2025-09-14T22:18:05.954Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/ae/56523ae9c142f0c08efd5e868a6da613ae76614eca1305259c3bf6a0ed43/zstandard-0.25.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a9ec8c642d1ec73287ae3e726792dd86c96f5681eb8df274a757bf62b750eae7", size = 4955108, upload-time = "2025-09-14T22:18:07.68Z" },
+ { url = "https://files.pythonhosted.org/packages/98/cf/c899f2d6df0840d5e384cf4c4121458c72802e8bda19691f3b16619f51e9/zstandard-0.25.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a4089a10e598eae6393756b036e0f419e8c1d60f44a831520f9af41c14216cf2", size = 5269248, upload-time = "2025-09-14T22:18:09.753Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/c0/59e912a531d91e1c192d3085fc0f6fb2852753c301a812d856d857ea03c6/zstandard-0.25.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:f67e8f1a324a900e75b5e28ffb152bcac9fbed1cc7b43f99cd90f395c4375344", size = 5430330, upload-time = "2025-09-14T22:18:11.966Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/1d/7e31db1240de2df22a58e2ea9a93fc6e38cc29353e660c0272b6735d6669/zstandard-0.25.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:9654dbc012d8b06fc3d19cc825af3f7bf8ae242226df5f83936cb39f5fdc846c", size = 5811123, upload-time = "2025-09-14T22:18:13.907Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/49/fac46df5ad353d50535e118d6983069df68ca5908d4d65b8c466150a4ff1/zstandard-0.25.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:4203ce3b31aec23012d3a4cf4a2ed64d12fea5269c49aed5e4c3611b938e4088", size = 5359591, upload-time = "2025-09-14T22:18:16.465Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/38/f249a2050ad1eea0bb364046153942e34abba95dd5520af199aed86fbb49/zstandard-0.25.0-cp314-cp314-win32.whl", hash = "sha256:da469dc041701583e34de852d8634703550348d5822e66a0c827d39b05365b12", size = 444513, upload-time = "2025-09-14T22:18:20.61Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/43/241f9615bcf8ba8903b3f0432da069e857fc4fd1783bd26183db53c4804b/zstandard-0.25.0-cp314-cp314-win_amd64.whl", hash = "sha256:c19bcdd826e95671065f8692b5a4aa95c52dc7a02a4c5a0cac46deb879a017a2", size = 516118, upload-time = "2025-09-14T22:18:17.849Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/ef/da163ce2450ed4febf6467d77ccb4cd52c4c30ab45624bad26ca0a27260c/zstandard-0.25.0-cp314-cp314-win_arm64.whl", hash = "sha256:d7541afd73985c630bafcd6338d2518ae96060075f9463d7dc14cfb33514383d", size = 476940, upload-time = "2025-09-14T22:18:19.088Z" },
+]